2023-10-30T12:11:12.906 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-10-30T12:11:12.907 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019 --verbose --name leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} --owner scheduled_leonidus@teuthology -- /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019/orig.config.yaml 2023-10-30T12:11:12.961 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T12:11:13.048 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019 branch: wip-lusov-balance-after-stop description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} email: null first_in_suite: false job_id: '7441019' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-lusov-balance-after-stop ceph: cephfs: max_mds: 3 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: false mds_sleep_rank_change: 5000000.0 mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds osd ms inject socket failures: 2500 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon client directed command retry: 5 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: client mds ms inject socket failures: 2500 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - Replacing daemon mds sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 kclient: syntax: v1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-balance-after-stop sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 owner: scheduled_leonidus@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 8397 sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 sleep_before_teardown: 0 subset: 464/1024 suite: fs suite_branch: wip-lusov-balance-after-stop suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi125.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAiFxaPW/fAIFYfAZ7S0QaSr7et4p16btxGCug/BeJSMAcBjUFWpEFIiQexh7R1wVQLHrWxl1wi22tuK+iNge+g= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHFv/5e/H1nftak/kNE7BpioBtAEeujqPjwXW0WI6M9KclRCZ1nNUcD4HU42sPUWOO2qfq1wGIW0kzp5YjMoVs4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - mds_thrash: null - workunit: clients: all: - fs/snaps timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-10-30_06:34:16 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.770726 2023-10-30T12:11:13.049 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa; will attempt to use it 2023-10-30T12:11:13.050 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks 2023-10-30T12:11:13.050 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-30T12:11:13.052 INFO:teuthology.task.internal:Checking packages... 2023-10-30T12:11:13.075 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '00c1c13cea21a686b046584596fbeac5315a7d71' 2023-10-30T12:11:13.075 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-30T12:11:13.076 INFO:teuthology.packaging:ref: None 2023-10-30T12:11:13.076 INFO:teuthology.packaging:tag: None 2023-10-30T12:11:13.076 INFO:teuthology.packaging:branch: wip-lusov-balance-after-stop 2023-10-30T12:11:13.077 INFO:teuthology.packaging:sha1: 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:11:13.077 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-lusov-balance-after-stop 2023-10-30T12:11:13.232 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:11:13.233 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-30T12:11:13.245 INFO:teuthology.task.internal:no buildpackages task found 2023-10-30T12:11:13.245 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-30T12:11:13.255 INFO:teuthology.task.internal:Saving configuration 2023-10-30T12:11:13.274 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-30T12:11:13.283 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T12:11:13.307 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi125.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 12:05:37.069272', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKwOATX/PtOOkJ4MqXDmGebF7ywXXAOxY0Am/5DaYkKj'} 2023-10-30T12:11:13.330 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 12:05:37.071199', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM0anWOxVBPZ/cmnA86n2ae5B8TKLJxLSnxXhiHelslr'} 2023-10-30T12:11:13.330 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-30T12:11:13.338 INFO:teuthology.task.internal:roles: ubuntu@smithi125.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2023-10-30T12:11:13.338 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2023-10-30T12:11:13.338 INFO:teuthology.run_tasks:Running task console_log... 2023-10-30T12:11:13.412 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd7744dfc10>, signals=[15]) 2023-10-30T12:11:13.413 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-30T12:11:13.422 INFO:teuthology.task.internal:Opening connections... 2023-10-30T12:11:13.422 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi125.front.sepia.ceph.com 2023-10-30T12:11:13.424 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:11:13.500 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2023-10-30T12:11:13.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:11:13.574 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-30T12:11:13.582 DEBUG:teuthology.orchestra.run.smithi125:> uname -m 2023-10-30T12:11:13.587 INFO:teuthology.orchestra.run.smithi125.stdout:x86_64 2023-10-30T12:11:13.587 DEBUG:teuthology.orchestra.run.smithi125:> cat /etc/os-release 2023-10-30T12:11:13.634 INFO:teuthology.orchestra.run.smithi125.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-30T12:11:13.635 INFO:teuthology.orchestra.run.smithi125.stdout:NAME="Ubuntu" 2023-10-30T12:11:13.635 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_ID="22.04" 2023-10-30T12:11:13.635 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-30T12:11:13.636 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_CODENAME=jammy 2023-10-30T12:11:13.636 INFO:teuthology.orchestra.run.smithi125.stdout:ID=ubuntu 2023-10-30T12:11:13.636 INFO:teuthology.orchestra.run.smithi125.stdout:ID_LIKE=debian 2023-10-30T12:11:13.636 INFO:teuthology.orchestra.run.smithi125.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-30T12:11:13.636 INFO:teuthology.orchestra.run.smithi125.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-30T12:11:13.637 INFO:teuthology.orchestra.run.smithi125.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-30T12:11:13.637 INFO:teuthology.orchestra.run.smithi125.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-30T12:11:13.637 INFO:teuthology.orchestra.run.smithi125.stdout:UBUNTU_CODENAME=jammy 2023-10-30T12:11:13.638 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server 2023-10-30T12:11:13.661 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2023-10-30T12:11:13.666 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2023-10-30T12:11:13.667 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2023-10-30T12:11:13.712 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-10-30T12:11:13.712 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="Ubuntu" 2023-10-30T12:11:13.713 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="22.04" 2023-10-30T12:11:13.713 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-10-30T12:11:13.713 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_CODENAME=jammy 2023-10-30T12:11:13.713 INFO:teuthology.orchestra.run.smithi136.stdout:ID=ubuntu 2023-10-30T12:11:13.713 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE=debian 2023-10-30T12:11:13.714 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://www.ubuntu.com/" 2023-10-30T12:11:13.714 INFO:teuthology.orchestra.run.smithi136.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-10-30T12:11:13.714 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-10-30T12:11:13.714 INFO:teuthology.orchestra.run.smithi136.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-10-30T12:11:13.714 INFO:teuthology.orchestra.run.smithi136.stdout:UBUNTU_CODENAME=jammy 2023-10-30T12:11:13.715 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2023-10-30T12:11:13.735 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-30T12:11:13.746 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-30T12:11:13.754 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-30T12:11:13.754 DEBUG:teuthology.orchestra.run.smithi125:> test '!' -e /home/ubuntu/cephtest 2023-10-30T12:11:13.757 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2023-10-30T12:11:13.762 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-30T12:11:13.772 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-30T12:11:13.772 DEBUG:teuthology.orchestra.run.smithi125:> test -z $(ls -A /var/lib/ceph) 2023-10-30T12:11:13.804 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2023-10-30T12:11:13.826 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-30T12:11:13.878 INFO:teuthology.run_tasks:Running task kernel... 2023-10-30T12:11:13.894 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-10-30T12:11:13.895 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2023-10-30T12:11:13.896 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2023-10-30T12:11:13.896 DEBUG:teuthology.orchestra.run.smithi125:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T12:11:13.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:11:13.902 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=testing 2023-10-30T12:11:14.084 DEBUG:teuthology.task.kernel:sha1 for client.0 is eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T12:11:14.085 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "eacf00cc51e204f401b4d5d4f84825a68f373c74"... 2023-10-30T12:11:14.085 DEBUG:teuthology.orchestra.run.smithi125:> uname -r 2023-10-30T12:11:14.091 INFO:teuthology.orchestra.run.smithi125.stdout:5.15.0-56-generic 2023-10-30T12:11:14.091 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-56-generic vs eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T12:11:14.091 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-30T12:11:14.092 INFO:teuthology.task.kernel:Installing linux-firmware on client.0... 2023-10-30T12:11:14.092 DEBUG:teuthology.orchestra.run.smithi125:> cd /lib/firmware/updates && test -d .git && test '!' -s .git/HEAD && sudo rm -rf .git ; sudo install -d -m0755 /lib/firmware/updates && cd /lib/firmware/updates && sudo git init 2023-10-30T12:11:14.152 INFO:teuthology.orchestra.run.smithi125.stderr:bash: line 1: cd: /lib/firmware/updates: No such file or directory 2023-10-30T12:11:14.300 INFO:teuthology.orchestra.run.smithi125.stderr:hint: Using 'master' as the name for the initial branch. This default branch name 2023-10-30T12:11:14.300 INFO:teuthology.orchestra.run.smithi125.stderr:hint: is subject to change. To configure the initial branch name to use in all 2023-10-30T12:11:14.300 INFO:teuthology.orchestra.run.smithi125.stderr:hint: of your new repositories, which will suppress this warning, call: 2023-10-30T12:11:14.301 INFO:teuthology.orchestra.run.smithi125.stderr:hint: 2023-10-30T12:11:14.301 INFO:teuthology.orchestra.run.smithi125.stderr:hint: git config --global init.defaultBranch 2023-10-30T12:11:14.301 INFO:teuthology.orchestra.run.smithi125.stderr:hint: 2023-10-30T12:11:14.301 INFO:teuthology.orchestra.run.smithi125.stderr:hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2023-10-30T12:11:14.301 INFO:teuthology.orchestra.run.smithi125.stderr:hint: 'development'. The just-created branch can be renamed via this command: 2023-10-30T12:11:14.302 INFO:teuthology.orchestra.run.smithi125.stderr:hint: 2023-10-30T12:11:14.302 INFO:teuthology.orchestra.run.smithi125.stderr:hint: git branch -m 2023-10-30T12:11:14.302 INFO:teuthology.orchestra.run.smithi125.stdout:Initialized empty Git repository in /usr/lib/firmware/updates/.git/ 2023-10-30T12:11:14.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo git --git-dir=/lib/firmware/updates/.git config --get remote.origin.url >/dev/null || sudo git --git-dir=/lib/firmware/updates/.git remote add origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git 2023-10-30T12:11:14.369 DEBUG:teuthology.orchestra.run.smithi125:> sudo git --git-dir=/lib/firmware/updates/.git remote set-url origin git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git >/dev/null 2023-10-30T12:11:14.424 DEBUG:teuthology.orchestra.run.smithi125:> cd /lib/firmware/updates && sudo git fetch origin && sudo git reset --hard origin/main 2023-10-30T12:13:57.711 INFO:teuthology.orchestra.run.smithi125.stderr:From git://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware 2023-10-30T12:13:57.712 INFO:teuthology.orchestra.run.smithi125.stderr: * [new branch] main -> origin/main 2023-10-30T12:13:57.742 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20190312 -> 20190312 2023-10-30T12:13:57.742 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20190416 -> 20190416 2023-10-30T12:13:57.742 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20190514 -> 20190514 2023-10-30T12:13:57.743 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20190618 -> 20190618 2023-10-30T12:13:57.743 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20190717 -> 20190717 2023-10-30T12:13:57.743 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20190815 -> 20190815 2023-10-30T12:13:57.743 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20190923 -> 20190923 2023-10-30T12:13:57.743 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20191022 -> 20191022 2023-10-30T12:13:57.744 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20191215 -> 20191215 2023-10-30T12:13:57.747 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200122 -> 20200122 2023-10-30T12:13:57.747 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200316 -> 20200316 2023-10-30T12:13:57.748 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200421 -> 20200421 2023-10-30T12:13:57.748 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200519 -> 20200519 2023-10-30T12:13:57.748 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200619 -> 20200619 2023-10-30T12:13:57.748 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200721 -> 20200721 2023-10-30T12:13:57.748 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200817 -> 20200817 2023-10-30T12:13:57.749 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20200918 -> 20200918 2023-10-30T12:13:57.749 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20201022 -> 20201022 2023-10-30T12:13:57.749 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20201118 -> 20201118 2023-10-30T12:13:57.749 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20201218 -> 20201218 2023-10-30T12:13:57.749 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20210208 -> 20210208 2023-10-30T12:13:57.750 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20210315 -> 20210315 2023-10-30T12:13:57.750 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20210511 -> 20210511 2023-10-30T12:13:57.750 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20210716 -> 20210716 2023-10-30T12:13:57.750 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20210818 -> 20210818 2023-10-30T12:13:57.750 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20210919 -> 20210919 2023-10-30T12:13:57.751 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20211027 -> 20211027 2023-10-30T12:13:57.751 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20211216 -> 20211216 2023-10-30T12:13:57.751 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220209 -> 20220209 2023-10-30T12:13:57.751 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220310 -> 20220310 2023-10-30T12:13:57.751 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220411 -> 20220411 2023-10-30T12:13:57.752 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220509 -> 20220509 2023-10-30T12:13:57.752 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220610 -> 20220610 2023-10-30T12:13:57.752 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220708 -> 20220708 2023-10-30T12:13:57.752 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220815 -> 20220815 2023-10-30T12:13:57.752 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20220913 -> 20220913 2023-10-30T12:13:57.753 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20221012 -> 20221012 2023-10-30T12:13:57.753 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20221109 -> 20221109 2023-10-30T12:13:57.753 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20221214 -> 20221214 2023-10-30T12:13:57.753 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230117 -> 20230117 2023-10-30T12:13:57.753 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230210 -> 20230210 2023-10-30T12:13:57.754 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230310 -> 20230310 2023-10-30T12:13:57.754 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230404 -> 20230404 2023-10-30T12:13:57.754 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230515 -> 20230515 2023-10-30T12:13:57.754 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230625 -> 20230625 2023-10-30T12:13:57.754 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230804 -> 20230804 2023-10-30T12:13:57.755 INFO:teuthology.orchestra.run.smithi125.stderr: * [new tag] 20230919 -> 20230919 2023-10-30T12:14:01.446 INFO:teuthology.orchestra.run.smithi125.stderr:Updating files: 51% (1643/3168) Updating files: 52% (1648/3168) Updating files: 53% (1680/3168) Updating files: 54% (1711/3168) Updating files: 55% (1743/3168) Updating files: 56% (1775/3168) Updating files: 57% (1806/3168) Updating files: 58% (1838/3168) Updating files: 59% (1870/3168) Updating files: 60% (1901/3168) Updating files: 61% (1933/3168) Updating files: 62% (1965/3168) Updating files: 63% (1996/3168) Updating files: 64% (2028/3168) Updating files: 65% (2060/3168) Updating files: 66% (2091/3168) Updating files: 66% (2113/3168) Updating files: 67% (2123/3168) Updating files: 68% (2155/3168) Updating files: 69% (2186/3168) Updating files: 70% (2218/3168) Updating files: 71% (2250/3168) Updating files: 72% (2281/3168) Updating files: 73% (2313/3168) Updating files: 74% (2345/3168) Updating files: 75% (2376/3168) Updating files: 76% (2408/3168) Updating files: 77% (2440/3168) Updating files: 78% (2472/3168) Updating files: 79% (2503/3168) Updating files: 80% (2535/3168) Updating files: 81% (2567/3168) Updating files: 81% (2596/3168) Updating files: 82% (2598/3168) Updating files: 83% (2630/3168) Updating files: 84% (2662/3168) Updating files: 85% (2693/3168) Updating files: 86% (2725/3168) Updating files: 87% (2757/3168) Updating files: 88% (2788/3168) Updating files: 89% (2820/3168) Updating files: 90% (2852/3168) Updating files: 91% (2883/3168) Updating files: 92% (2915/3168) Updating files: 93% (2947/3168) Updating files: 94% (2978/3168) Updating files: 95% (3010/3168) Updating files: 96% (3042/3168) Updating files: 97% (3073/3168) Updating files: 98% (3105/3168) Updating files: 99% (3137/3168) Updating files: 100% (3168/3168) Updating files: 100% (3168/3168), done. 2023-10-30T12:14:01.450 INFO:teuthology.orchestra.run.smithi125.stdout:HEAD is now at 2b304bfe Merge branch 'main' into 'main' 2023-10-30T12:14:01.471 INFO:teuthology.task.kernel:Downloading kernel eacf00cc51e204f401b4d5d4f84825a68f373c74 on client.0... 2023-10-30T12:14:01.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=eacf00cc51e204f401b4d5d4f84825a68f373c74 2023-10-30T12:14:01.849 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/ 2023-10-30T12:14:01.849 DEBUG:teuthology.orchestra.run.smithi125:> rm -f /tmp/linux-image.deb && echo linux-image-6.6.0-rc5-geacf00cc51e2_6.6.0-rc5-geacf00cc51e2-1_amd64.deb | wget -nv -O /tmp/linux-image.deb --base=https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/ --input-file=- 2023-10-30T12:14:01.852 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi125.front.sepia.ceph.com to complete... 2023-10-30T12:14:05.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30 12:14:05 URL:https://4.chacra.ceph.com/r/kernel/testing/eacf00cc51e204f401b4d5d4f84825a68f373c74/ubuntu/jammy/flavors/default/pool/main/l/linux-6.6.0-rc5-geacf00cc51e2/linux-image-6.6.0-rc5-geacf00cc51e2_6.6.0-rc5-geacf00cc51e2-1_amd64.deb [38909658/38909658] -> "/tmp/linux-image.deb" [1] 2023-10-30T12:14:05.103 INFO:teuthology.orchestra.run.smithi125.stderr:FINISHED --2023-10-30 12:14:05-- 2023-10-30T12:14:05.103 INFO:teuthology.orchestra.run.smithi125.stderr:Total wall clock time: 3.1s 2023-10-30T12:14:05.103 INFO:teuthology.orchestra.run.smithi125.stderr:Downloaded: 1 files, 37M in 2.8s (13.1 MB/s) 2023-10-30T12:14:05.106 INFO:teuthology.task.kernel:Installing kernel eacf00cc51e204f401b4d5d4f84825a68f373c74 on client.0... 2023-10-30T12:14:05.106 DEBUG:teuthology.orchestra.run.smithi125:> sudo dpkg -i /tmp/linux-image.deb 2023-10-30T12:14:05.418 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package linux-image-6.6.0-rc5-geacf00cc51e2. 2023-10-30T12:14:07.375 INFO:teuthology.orchestra.run.smithi125.stdout:(Reading database ... 129542 files and directories currently installed.) 2023-10-30T12:14:07.379 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack /tmp/linux-image.deb ... 2023-10-30T12:14:07.507 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking linux-image-6.6.0-rc5-geacf00cc51e2 (6.6.0-rc5-geacf00cc51e2-1) ... 2023-10-30T12:14:10.993 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up linux-image-6.6.0-rc5-geacf00cc51e2 (6.6.0-rc5-geacf00cc51e2-1) ... 2023-10-30T12:14:11.096 INFO:teuthology.orchestra.run.smithi125.stderr:update-initramfs: Generating /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:31.130 INFO:teuthology.orchestra.run.smithi125.stderr:Sourcing file `/etc/default/grub' 2023-10-30T12:14:31.140 INFO:teuthology.orchestra.run.smithi125.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-30T12:14:31.170 INFO:teuthology.orchestra.run.smithi125.stderr:Generating grub configuration file ... 2023-10-30T12:14:31.462 INFO:teuthology.orchestra.run.smithi125.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:31.479 INFO:teuthology.orchestra.run.smithi125.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:31.792 INFO:teuthology.orchestra.run.smithi125.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-30T12:14:31.804 INFO:teuthology.orchestra.run.smithi125.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-30T12:14:31.865 INFO:teuthology.orchestra.run.smithi125.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-30T12:14:31.870 INFO:teuthology.orchestra.run.smithi125.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-30T12:14:32.012 INFO:teuthology.orchestra.run.smithi125.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-30T12:14:32.012 INFO:teuthology.orchestra.run.smithi125.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-30T12:14:32.013 INFO:teuthology.orchestra.run.smithi125.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-30T12:14:32.065 INFO:teuthology.orchestra.run.smithi125.stderr:done 2023-10-30T12:14:32.248 DEBUG:teuthology.orchestra.run.smithi125:> dpkg-deb -c /tmp/linux-image.deb 2023-10-30T12:14:32.256 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./ 2023-10-30T12:14:32.257 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./boot/ 2023-10-30T12:14:32.257 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 4415097 2023-10-30 03:46 ./boot/System.map-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:32.269 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 169233 2023-10-30 03:46 ./boot/config-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:32.269 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11352928 2023-10-30 03:46 ./boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:32.288 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/ 2023-10-30T12:14:32.288 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/ 2023-10-30T12:14:32.289 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/postinst.d/ 2023-10-30T12:14:32.289 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/postrm.d/ 2023-10-30T12:14:32.289 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/preinst.d/ 2023-10-30T12:14:32.289 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./etc/kernel/prerm.d/ 2023-10-30T12:14:32.289 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/ 2023-10-30T12:14:32.290 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/ 2023-10-30T12:14:32.290 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/ 2023-10-30T12:14:32.290 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/ 2023-10-30T12:14:32.290 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/ 2023-10-30T12:14:32.290 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/ 2023-10-30T12:14:32.291 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/ 2023-10-30T12:14:32.291 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 490233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/aesni-intel.ko 2023-10-30T12:14:32.293 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-30T12:14:32.294 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-30T12:14:32.294 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-30T12:14:32.294 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 93113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-30T12:14:32.294 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 106729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-30T12:14:32.295 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 152241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-30T12:14:32.295 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-30T12:14:32.295 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-30T12:14:32.295 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 83889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-30T12:14:32.295 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-30T12:14:32.296 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-30T12:14:32.296 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-30T12:14:32.296 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-30T12:14:32.296 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-30T12:14:32.296 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-30T12:14:32.297 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 58585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-30T12:14:32.297 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-30T12:14:32.297 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-30T12:14:32.297 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-30T12:14:32.297 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/ 2023-10-30T12:14:32.297 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/ 2023-10-30T12:14:32.298 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/mce/ 2023-10-30T12:14:32.298 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-30T12:14:32.298 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/cpuid.ko 2023-10-30T12:14:32.299 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kernel/msr.ko 2023-10-30T12:14:32.299 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/ 2023-10-30T12:14:32.299 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 353761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-30T12:14:32.299 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 881713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-30T12:14:32.299 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 2276817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/arch/x86/kvm/kvm.ko 2023-10-30T12:14:32.342 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ 2023-10-30T12:14:32.342 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/af_alg.ko 2023-10-30T12:14:32.343 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/algif_hash.ko 2023-10-30T12:14:32.343 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/algif_skcipher.ko 2023-10-30T12:14:32.343 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ansi_cprng.ko 2023-10-30T12:14:32.343 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/anubis.ko 2023-10-30T12:14:32.343 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/arc4.ko 2023-10-30T12:14:32.344 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/asymmetric_keys/ 2023-10-30T12:14:32.344 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/asymmetric_keys/pkcs7_test_key.ko 2023-10-30T12:14:32.344 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/ 2023-10-30T12:14:32.344 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_memcpy.ko 2023-10-30T12:14:32.345 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_pq.ko 2023-10-30T12:14:32.345 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-30T12:14:32.345 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_tx.ko 2023-10-30T12:14:32.345 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/async_xor.ko 2023-10-30T12:14:32.346 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/async_tx/raid6test.ko 2023-10-30T12:14:32.346 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/authenc.ko 2023-10-30T12:14:32.346 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/authencesn.ko 2023-10-30T12:14:32.346 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blake2b_generic.ko 2023-10-30T12:14:32.347 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blowfish_common.ko 2023-10-30T12:14:32.347 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/blowfish_generic.ko 2023-10-30T12:14:32.347 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/camellia_generic.ko 2023-10-30T12:14:32.347 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast5_generic.ko 2023-10-30T12:14:32.347 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast6_generic.ko 2023-10-30T12:14:32.348 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cast_common.ko 2023-10-30T12:14:32.348 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/ccm.ko 2023-10-30T12:14:32.348 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cmac.ko 2023-10-30T12:14:32.348 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crc32_generic.ko 2023-10-30T12:14:32.348 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cryptd.ko 2023-10-30T12:14:32.349 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_engine.ko 2023-10-30T12:14:32.349 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_simd.ko 2023-10-30T12:14:32.349 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/crypto_user.ko 2023-10-30T12:14:32.349 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/cts.ko 2023-10-30T12:14:32.349 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/des_generic.ko 2023-10-30T12:14:32.350 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/dh_generic.ko 2023-10-30T12:14:32.350 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/echainiv.ko 2023-10-30T12:14:32.350 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/essiv.ko 2023-10-30T12:14:32.350 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/fcrypt.ko 2023-10-30T12:14:32.350 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/khazad.ko 2023-10-30T12:14:32.351 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lrw.ko 2023-10-30T12:14:32.351 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lz4.ko 2023-10-30T12:14:32.351 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/lz4hc.ko 2023-10-30T12:14:32.351 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/md4.ko 2023-10-30T12:14:32.351 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/michael_mic.ko 2023-10-30T12:14:32.352 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/pcbc.ko 2023-10-30T12:14:32.352 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/pcrypt.ko 2023-10-30T12:14:32.352 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/rmd160.ko 2023-10-30T12:14:32.352 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/seed.ko 2023-10-30T12:14:32.352 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/serpent_generic.ko 2023-10-30T12:14:32.353 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 149193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/tcrypt.ko 2023-10-30T12:14:32.353 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/tea.ko 2023-10-30T12:14:32.353 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/twofish_common.ko 2023-10-30T12:14:32.353 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/twofish_generic.ko 2023-10-30T12:14:32.353 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/vmac.ko 2023-10-30T12:14:32.353 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/wp512.ko 2023-10-30T12:14:32.354 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xcbc.ko 2023-10-30T12:14:32.354 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xor.ko 2023-10-30T12:14:32.354 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xts.ko 2023-10-30T12:14:32.354 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/crypto/xxhash_generic.ko 2023-10-30T12:14:32.354 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ 2023-10-30T12:14:32.355 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/ 2023-10-30T12:14:32.355 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_extlog.ko 2023-10-30T12:14:32.355 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-30T12:14:32.355 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/acpi_pad.ko 2023-10-30T12:14:32.355 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/apei/ 2023-10-30T12:14:32.356 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/apei/einj.ko 2023-10-30T12:14:32.356 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/ec_sys.ko 2023-10-30T12:14:32.356 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/platform_profile.ko 2023-10-30T12:14:32.356 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/sbs.ko 2023-10-30T12:14:32.357 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/sbshc.ko 2023-10-30T12:14:32.357 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 103025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/acpi/video.ko 2023-10-30T12:14:32.357 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ 2023-10-30T12:14:32.357 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/acard-ahci.ko 2023-10-30T12:14:32.358 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 88865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ahci.ko 2023-10-30T12:14:32.358 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/ahci_platform.ko 2023-10-30T12:14:32.358 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 88289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/libahci.ko 2023-10-30T12:14:32.358 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/libahci_platform.ko 2023-10-30T12:14:32.358 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_acpi.ko 2023-10-30T12:14:32.359 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ali.ko 2023-10-30T12:14:32.359 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_amd.ko 2023-10-30T12:14:32.359 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_artop.ko 2023-10-30T12:14:32.359 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_atiixp.ko 2023-10-30T12:14:32.359 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_atp867x.ko 2023-10-30T12:14:32.360 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cmd640.ko 2023-10-30T12:14:32.360 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cmd64x.ko 2023-10-30T12:14:32.360 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_cypress.ko 2023-10-30T12:14:32.360 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_efar.ko 2023-10-30T12:14:32.361 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt366.ko 2023-10-30T12:14:32.361 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt37x.ko 2023-10-30T12:14:32.361 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-30T12:14:32.361 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-30T12:14:32.362 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_it8213.ko 2023-10-30T12:14:32.362 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_it821x.ko 2023-10-30T12:14:32.362 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_jmicron.ko 2023-10-30T12:14:32.362 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_legacy.ko 2023-10-30T12:14:32.362 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_marvell.ko 2023-10-30T12:14:32.363 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_mpiix.ko 2023-10-30T12:14:32.363 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_netcell.ko 2023-10-30T12:14:32.363 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ninja32.ko 2023-10-30T12:14:32.363 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ns87410.ko 2023-10-30T12:14:32.363 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_ns87415.ko 2023-10-30T12:14:32.364 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_oldpiix.ko 2023-10-30T12:14:32.364 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_opti.ko 2023-10-30T12:14:32.364 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_optidma.ko 2023-10-30T12:14:32.364 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pcmcia.ko 2023-10-30T12:14:32.365 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-30T12:14:32.365 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-30T12:14:32.365 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_piccolo.ko 2023-10-30T12:14:32.365 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_radisys.ko 2023-10-30T12:14:32.366 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_rdc.ko 2023-10-30T12:14:32.366 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_rz1000.ko 2023-10-30T12:14:32.366 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sch.ko 2023-10-30T12:14:32.366 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_serverworks.ko 2023-10-30T12:14:32.367 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sil680.ko 2023-10-30T12:14:32.367 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_sl82c105.ko 2023-10-30T12:14:32.367 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_triflex.ko 2023-10-30T12:14:32.367 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pata_via.ko 2023-10-30T12:14:32.367 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/pdc_adma.ko 2023-10-30T12:14:32.368 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_inic162x.ko 2023-10-30T12:14:32.368 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 83345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_mv.ko 2023-10-30T12:14:32.368 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_nv.ko 2023-10-30T12:14:32.368 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_promise.ko 2023-10-30T12:14:32.369 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_qstor.ko 2023-10-30T12:14:32.369 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sil.ko 2023-10-30T12:14:32.369 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sil24.ko 2023-10-30T12:14:32.369 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sis.ko 2023-10-30T12:14:32.369 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_svw.ko 2023-10-30T12:14:32.369 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_sx4.ko 2023-10-30T12:14:32.370 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_uli.ko 2023-10-30T12:14:32.370 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_via.ko 2023-10-30T12:14:32.370 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ata/sata_vsc.ko 2023-10-30T12:14:32.370 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/auxdisplay/ 2023-10-30T12:14:32.370 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/auxdisplay/ks0108.ko 2023-10-30T12:14:32.371 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/bcma/ 2023-10-30T12:14:32.371 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 124473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/bcma/bcma.ko 2023-10-30T12:14:32.371 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/ 2023-10-30T12:14:32.371 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/aoe/ 2023-10-30T12:14:32.372 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 113665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/aoe/aoe.ko 2023-10-30T12:14:32.372 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/drbd/ 2023-10-30T12:14:32.372 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 965585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/drbd/drbd.ko 2023-10-30T12:14:32.372 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 173225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/floppy.ko 2023-10-30T12:14:32.372 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/mtip32xx/ 2023-10-30T12:14:32.373 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 94297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-30T12:14:32.373 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 115793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/nbd.ko 2023-10-30T12:14:32.373 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/null_blk/ 2023-10-30T12:14:32.373 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 110449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/null_blk/null_blk.ko 2023-10-30T12:14:32.373 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 98193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/pktcdvd.ko 2023-10-30T12:14:32.374 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 263769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/rbd.ko 2023-10-30T12:14:32.374 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/xen-blkback/ 2023-10-30T12:14:32.374 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 98145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-30T12:14:32.374 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/zram/ 2023-10-30T12:14:32.374 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/block/zram/zram.ko 2023-10-30T12:14:32.375 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ 2023-10-30T12:14:32.375 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/applicom.ko 2023-10-30T12:14:32.375 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hangcheck-timer.ko 2023-10-30T12:14:32.375 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/ 2023-10-30T12:14:32.375 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-30T12:14:32.376 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-30T12:14:32.376 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-30T12:14:32.376 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/via-rng.ko 2023-10-30T12:14:32.376 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-30T12:14:32.377 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ 2023-10-30T12:14:32.377 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-30T12:14:32.377 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 148601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-30T12:14:32.377 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-30T12:14:32.377 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 139313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-30T12:14:32.378 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-30T12:14:32.378 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-30T12:14:32.378 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/lp.ko 2023-10-30T12:14:32.378 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/mwave/ 2023-10-30T12:14:32.378 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 84097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/mwave/mwave.ko 2023-10-30T12:14:32.379 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/nvram.ko 2023-10-30T12:14:32.379 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/ppdev.ko 2023-10-30T12:14:32.379 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tlclk.ko 2023-10-30T12:14:32.379 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/ 2023-10-30T12:14:32.379 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-30T12:14:32.380 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_atmel.ko 2023-10-30T12:14:32.380 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_infineon.ko 2023-10-30T12:14:32.380 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_i2c_nuvoton.ko 2023-10-30T12:14:32.380 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-30T12:14:32.380 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-30T12:14:32.381 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/tpm/xen-tpmfront.ko 2023-10-30T12:14:32.381 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/ 2023-10-30T12:14:32.381 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_class.ko 2023-10-30T12:14:32.381 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_core.ko 2023-10-30T12:14:32.381 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/char/xillybus/xillybus_pcie.ko 2023-10-30T12:14:32.382 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/ 2023-10-30T12:14:32.382 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-palmas.ko 2023-10-30T12:14:32.382 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-si5351.ko 2023-10-30T12:14:32.382 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-twl6040.ko 2023-10-30T12:14:32.382 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/clk/clk-wm831x.ko 2023-10-30T12:14:32.382 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/ 2023-10-30T12:14:32.383 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-30T12:14:32.383 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-30T12:14:32.383 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-30T12:14:32.383 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ 2023-10-30T12:14:32.383 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ 2023-10-30T12:14:32.384 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-30T12:14:32.384 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 243025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/ccp/ccp.ko 2023-10-30T12:14:32.384 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/ 2023-10-30T12:14:32.384 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/ 2023-10-30T12:14:32.384 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_common/ 2023-10-30T12:14:32.385 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 438761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-30T12:14:32.385 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_dh895xcc/ 2023-10-30T12:14:32.385 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-30T12:14:32.385 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/padlock-aes.ko 2023-10-30T12:14:32.385 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/padlock-sha.ko 2023-10-30T12:14:32.386 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/virtio/ 2023-10-30T12:14:32.386 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 68857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-30T12:14:32.386 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dca/ 2023-10-30T12:14:32.386 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dca/dca.ko 2023-10-30T12:14:32.386 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ 2023-10-30T12:14:32.387 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/dw/ 2023-10-30T12:14:32.387 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-30T12:14:32.387 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ioat/ 2023-10-30T12:14:32.387 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 122569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-30T12:14:32.387 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/ 2023-10-30T12:14:32.387 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 79953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/amd64_edac.ko 2023-10-30T12:14:32.388 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/e752x_edac.ko 2023-10-30T12:14:32.388 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 63617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/edac_mce_amd.ko 2023-10-30T12:14:32.388 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i3000_edac.ko 2023-10-30T12:14:32.388 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i3200_edac.ko 2023-10-30T12:14:32.388 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i5100_edac.ko 2023-10-30T12:14:32.389 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i5400_edac.ko 2023-10-30T12:14:32.389 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i7300_edac.ko 2023-10-30T12:14:32.389 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 52041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i7core_edac.ko 2023-10-30T12:14:32.389 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/i82975x_edac.ko 2023-10-30T12:14:32.389 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/ie31200_edac.ko 2023-10-30T12:14:32.390 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 56825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/sb_edac.ko 2023-10-30T12:14:32.390 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/edac/x38_edac.ko 2023-10-30T12:14:32.390 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/ 2023-10-30T12:14:32.390 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-gpio.ko 2023-10-30T12:14:32.390 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max14577.ko 2023-10-30T12:14:32.391 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max77693.ko 2023-10-30T12:14:32.391 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-max8997.ko 2023-10-30T12:14:32.391 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-palmas.ko 2023-10-30T12:14:32.391 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-rt8973a.ko 2023-10-30T12:14:32.391 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/extcon/extcon-sm5502.ko 2023-10-30T12:14:32.391 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/ 2023-10-30T12:14:32.392 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/dmi-sysfs.ko 2023-10-30T12:14:32.392 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/efi/ 2023-10-30T12:14:32.392 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/efi/efi-pstore.ko 2023-10-30T12:14:32.392 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-30T12:14:32.392 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/ 2023-10-30T12:14:32.393 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-adp5520.ko 2023-10-30T12:14:32.393 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-amd8111.ko 2023-10-30T12:14:32.393 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-arizona.ko 2023-10-30T12:14:32.393 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-da9052.ko 2023-10-30T12:14:32.393 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-da9055.ko 2023-10-30T12:14:32.394 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-dln2.ko 2023-10-30T12:14:32.394 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 77849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-f7188x.ko 2023-10-30T12:14:32.394 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-generic.ko 2023-10-30T12:14:32.394 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-ich.ko 2023-10-30T12:14:32.394 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-janz-ttl.ko 2023-10-30T12:14:32.395 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-kempld.ko 2023-10-30T12:14:32.395 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-lp3943.ko 2023-10-30T12:14:32.395 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max7300.ko 2023-10-30T12:14:32.395 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max730x.ko 2023-10-30T12:14:32.396 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-max732x.ko 2023-10-30T12:14:32.396 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-ml-ioh.ko 2023-10-30T12:14:32.396 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-pca953x.ko 2023-10-30T12:14:32.396 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-pcf857x.ko 2023-10-30T12:14:32.396 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-rdc321x.ko 2023-10-30T12:14:32.397 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-sch.ko 2023-10-30T12:14:32.397 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-sch311x.ko 2023-10-30T12:14:32.397 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-tps65912.ko 2023-10-30T12:14:32.397 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-twl4030.ko 2023-10-30T12:14:32.397 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-twl6040.ko 2023-10-30T12:14:32.398 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-30T12:14:32.398 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-vx855.ko 2023-10-30T12:14:32.398 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm831x.ko 2023-10-30T12:14:32.398 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm8350.ko 2023-10-30T12:14:32.398 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/gpio/gpio-wm8994.ko 2023-10-30T12:14:32.399 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/ 2023-10-30T12:14:32.399 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-a4tech.ko 2023-10-30T12:14:32.399 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-apple.ko 2023-10-30T12:14:32.399 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-appleir.ko 2023-10-30T12:14:32.399 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-aureal.ko 2023-10-30T12:14:32.400 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-axff.ko 2023-10-30T12:14:32.400 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-belkin.ko 2023-10-30T12:14:32.400 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cherry.ko 2023-10-30T12:14:32.400 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-chicony.ko 2023-10-30T12:14:32.400 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cp2112.ko 2023-10-30T12:14:32.401 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-cypress.ko 2023-10-30T12:14:32.401 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-dr.ko 2023-10-30T12:14:32.401 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-elecom.ko 2023-10-30T12:14:32.401 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-elo.ko 2023-10-30T12:14:32.401 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-emsff.ko 2023-10-30T12:14:32.402 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ezkey.ko 2023-10-30T12:14:32.402 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gaff.ko 2023-10-30T12:14:32.402 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-generic.ko 2023-10-30T12:14:32.402 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gt683r.ko 2023-10-30T12:14:32.402 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-gyration.ko 2023-10-30T12:14:32.403 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-30T12:14:32.403 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-30T12:14:32.403 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-holtekff.ko 2023-10-30T12:14:32.403 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-hyperv.ko 2023-10-30T12:14:32.403 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-icade.ko 2023-10-30T12:14:32.404 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-kensington.ko 2023-10-30T12:14:32.404 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-keytouch.ko 2023-10-30T12:14:32.404 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-kye.ko 2023-10-30T12:14:32.404 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lcpower.ko 2023-10-30T12:14:32.404 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-led.ko 2023-10-30T12:14:32.405 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lenovo.ko 2023-10-30T12:14:32.405 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-lg-g15.ko 2023-10-30T12:14:32.405 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 58185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-30T12:14:32.405 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 101369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-30T12:14:32.405 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 80993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-logitech.ko 2023-10-30T12:14:32.405 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-magicmouse.ko 2023-10-30T12:14:32.406 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-microsoft.ko 2023-10-30T12:14:32.406 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-monterey.ko 2023-10-30T12:14:32.406 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-multitouch.ko 2023-10-30T12:14:32.406 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ntrig.ko 2023-10-30T12:14:32.406 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-ortek.ko 2023-10-30T12:14:32.407 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-penmount.ko 2023-10-30T12:14:32.407 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-petalynx.ko 2023-10-30T12:14:32.407 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 72937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-picolcd.ko 2023-10-30T12:14:32.407 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-pl.ko 2023-10-30T12:14:32.407 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-plantronics.ko 2023-10-30T12:14:32.408 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-primax.ko 2023-10-30T12:14:32.408 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-rmi.ko 2023-10-30T12:14:32.408 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-30T12:14:32.408 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-common.ko 2023-10-30T12:14:32.409 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-30T12:14:32.409 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-30T12:14:32.409 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-30T12:14:32.409 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-30T12:14:32.409 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-30T12:14:32.410 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-30T12:14:32.410 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-30T12:14:32.410 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-30T12:14:32.410 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-30T12:14:32.411 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-roccat.ko 2023-10-30T12:14:32.411 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-saitek.ko 2023-10-30T12:14:32.411 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-samsung.ko 2023-10-30T12:14:32.411 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-30T12:14:32.411 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sjoy.ko 2023-10-30T12:14:32.412 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sony.ko 2023-10-30T12:14:32.412 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-speedlink.ko 2023-10-30T12:14:32.412 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-steelseries.ko 2023-10-30T12:14:32.412 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-sunplus.ko 2023-10-30T12:14:32.412 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-30T12:14:32.413 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-tivo.ko 2023-10-30T12:14:32.413 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-tmff.ko 2023-10-30T12:14:32.413 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-topseed.ko 2023-10-30T12:14:32.413 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-twinhan.ko 2023-10-30T12:14:32.413 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 71073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-uclogic.ko 2023-10-30T12:14:32.414 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-waltop.ko 2023-10-30T12:14:32.414 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 143473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-wiimote.ko 2023-10-30T12:14:32.414 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-xinmo.ko 2023-10-30T12:14:32.414 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-zpff.ko 2023-10-30T12:14:32.414 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid-zydacron.ko 2023-10-30T12:14:32.415 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 267825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/hid.ko 2023-10-30T12:14:32.415 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/uhid.ko 2023-10-30T12:14:32.415 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/ 2023-10-30T12:14:32.415 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 125721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbhid.ko 2023-10-30T12:14:32.415 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbkbd.ko 2023-10-30T12:14:32.416 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/usbhid/usbmouse.ko 2023-10-30T12:14:32.416 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 217873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hid/wacom.ko 2023-10-30T12:14:32.416 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/ 2023-10-30T12:14:32.416 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 64201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_balloon.ko 2023-10-30T12:14:32.416 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_utils.ko 2023-10-30T12:14:32.417 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 305465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hv/hv_vmbus.ko 2023-10-30T12:14:32.417 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ 2023-10-30T12:14:32.417 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/abituguru.ko 2023-10-30T12:14:32.417 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 61393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/abituguru3.ko 2023-10-30T12:14:32.417 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-30T12:14:32.418 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ad7414.ko 2023-10-30T12:14:32.418 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ad7418.ko 2023-10-30T12:14:32.418 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adc128d818.ko 2023-10-30T12:14:32.418 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1025.ko 2023-10-30T12:14:32.418 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 97977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1026.ko 2023-10-30T12:14:32.419 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1029.ko 2023-10-30T12:14:32.419 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 45553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm1031.ko 2023-10-30T12:14:32.419 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adm9240.ko 2023-10-30T12:14:32.419 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ads7828.ko 2023-10-30T12:14:32.420 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7410.ko 2023-10-30T12:14:32.420 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7411.ko 2023-10-30T12:14:32.420 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 74433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7462.ko 2023-10-30T12:14:32.420 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7470.ko 2023-10-30T12:14:32.420 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7475.ko 2023-10-30T12:14:32.420 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/adt7x10.ko 2023-10-30T12:14:32.421 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/amc6821.ko 2023-10-30T12:14:32.421 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 55265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/applesmc.ko 2023-10-30T12:14:32.421 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asb100.ko 2023-10-30T12:14:32.421 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 58617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asc7621.ko 2023-10-30T12:14:32.421 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-30T12:14:32.422 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/atxp1.ko 2023-10-30T12:14:32.422 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/coretemp.ko 2023-10-30T12:14:32.422 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/da9052-hwmon.ko 2023-10-30T12:14:32.422 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/da9055-hwmon.ko 2023-10-30T12:14:32.422 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-30T12:14:32.423 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 99601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/dme1737.ko 2023-10-30T12:14:32.423 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ds1621.ko 2023-10-30T12:14:32.423 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ds620.ko 2023-10-30T12:14:32.423 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc1403.ko 2023-10-30T12:14:32.423 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc2103.ko 2023-10-30T12:14:32.424 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/emc6w201.ko 2023-10-30T12:14:32.424 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f71805f.ko 2023-10-30T12:14:32.424 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 94457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f71882fg.ko 2023-10-30T12:14:32.424 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/f75375s.ko 2023-10-30T12:14:32.424 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/fam15h_power.ko 2023-10-30T12:14:32.425 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 55449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/fschmd.ko 2023-10-30T12:14:32.425 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/g760a.ko 2023-10-30T12:14:32.425 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/g762.ko 2023-10-30T12:14:32.425 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/gl518sm.ko 2023-10-30T12:14:32.425 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 45529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/gl520sm.ko 2023-10-30T12:14:32.426 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/hih6130.ko 2023-10-30T12:14:32.426 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-30T12:14:32.426 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/i5500_temp.ko 2023-10-30T12:14:32.426 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/i5k_amb.ko 2023-10-30T12:14:32.426 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ibmaem.ko 2023-10-30T12:14:32.427 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ibmpex.ko 2023-10-30T12:14:32.427 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ina209.ko 2023-10-30T12:14:32.427 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ina2xx.ko 2023-10-30T12:14:32.427 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 130449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/it87.ko 2023-10-30T12:14:32.427 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/jc42.ko 2023-10-30T12:14:32.428 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/k10temp.ko 2023-10-30T12:14:32.428 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/k8temp.ko 2023-10-30T12:14:32.428 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lineage-pem.ko 2023-10-30T12:14:32.428 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 48601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm63.ko 2023-10-30T12:14:32.428 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm73.ko 2023-10-30T12:14:32.429 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm75.ko 2023-10-30T12:14:32.429 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm77.ko 2023-10-30T12:14:32.429 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm78.ko 2023-10-30T12:14:32.429 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm80.ko 2023-10-30T12:14:32.429 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm83.ko 2023-10-30T12:14:32.430 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 72833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm85.ko 2023-10-30T12:14:32.430 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm87.ko 2023-10-30T12:14:32.430 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm90.ko 2023-10-30T12:14:32.430 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm92.ko 2023-10-30T12:14:32.430 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 108841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm93.ko 2023-10-30T12:14:32.431 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95234.ko 2023-10-30T12:14:32.431 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95241.ko 2023-10-30T12:14:32.431 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/lm95245.ko 2023-10-30T12:14:32.431 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc2945.ko 2023-10-30T12:14:32.431 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4151.ko 2023-10-30T12:14:32.432 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4215.ko 2023-10-30T12:14:32.432 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4222.ko 2023-10-30T12:14:32.432 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4245.ko 2023-10-30T12:14:32.432 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4260.ko 2023-10-30T12:14:32.432 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/ltc4261.ko 2023-10-30T12:14:32.433 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max16065.ko 2023-10-30T12:14:32.433 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max1619.ko 2023-10-30T12:14:32.433 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max1668.ko 2023-10-30T12:14:32.433 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max197.ko 2023-10-30T12:14:32.433 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6639.ko 2023-10-30T12:14:32.434 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6650.ko 2023-10-30T12:14:32.434 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/max6697.ko 2023-10-30T12:14:32.434 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/mc13783-adc.ko 2023-10-30T12:14:32.434 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/mcp3021.ko 2023-10-30T12:14:32.434 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/menf21bmc_hwmon.ko 2023-10-30T12:14:32.435 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6683.ko 2023-10-30T12:14:32.435 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 119817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6775-core.ko 2023-10-30T12:14:32.435 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct6775.ko 2023-10-30T12:14:32.435 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/nct7802.ko 2023-10-30T12:14:32.435 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 82337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pc87360.ko 2023-10-30T12:14:32.436 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pc87427.ko 2023-10-30T12:14:32.436 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pcf8591.ko 2023-10-30T12:14:32.436 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ 2023-10-30T12:14:32.436 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-30T12:14:32.436 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-30T12:14:32.437 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-30T12:14:32.437 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-30T12:14:32.437 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-30T12:14:32.437 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-30T12:14:32.437 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-30T12:14:32.437 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 82577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-30T12:14:32.438 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-30T12:14:32.438 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-30T12:14:32.438 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-30T12:14:32.438 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-30T12:14:32.438 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/powr1220.ko 2023-10-30T12:14:32.439 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch5627.ko 2023-10-30T12:14:32.439 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch5636.ko 2023-10-30T12:14:32.439 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-30T12:14:32.439 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sht15.ko 2023-10-30T12:14:32.439 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sht21.ko 2023-10-30T12:14:32.440 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/shtc1.ko 2023-10-30T12:14:32.440 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/sis5595.ko 2023-10-30T12:14:32.440 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47b397.ko 2023-10-30T12:14:32.440 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47m1.ko 2023-10-30T12:14:32.441 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/smsc47m192.ko 2023-10-30T12:14:32.441 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/thmc50.ko 2023-10-30T12:14:32.441 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp102.ko 2023-10-30T12:14:32.441 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp103.ko 2023-10-30T12:14:32.441 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp401.ko 2023-10-30T12:14:32.441 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/tmp421.ko 2023-10-30T12:14:32.442 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/via-cputemp.ko 2023-10-30T12:14:32.442 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/via686a.ko 2023-10-30T12:14:32.442 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 52457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/vt1211.ko 2023-10-30T12:14:32.442 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/vt8231.ko 2023-10-30T12:14:32.442 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83627ehf.ko 2023-10-30T12:14:32.443 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83627hf.ko 2023-10-30T12:14:32.443 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 71345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83781d.ko 2023-10-30T12:14:32.443 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 63289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83791d.ko 2023-10-30T12:14:32.443 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 71321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83792d.ko 2023-10-30T12:14:32.443 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 101473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83793.ko 2023-10-30T12:14:32.444 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 119505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83795.ko 2023-10-30T12:14:32.444 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83l785ts.ko 2023-10-30T12:14:32.444 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/w83l786ng.ko 2023-10-30T12:14:32.444 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/wm831x-hwmon.ko 2023-10-30T12:14:32.444 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/hwmon/wm8350-hwmon.ko 2023-10-30T12:14:32.445 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/ 2023-10-30T12:14:32.445 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/ 2023-10-30T12:14:32.445 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-30T12:14:32.445 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-30T12:14:32.446 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/ 2023-10-30T12:14:32.446 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali1535.ko 2023-10-30T12:14:32.446 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali1563.ko 2023-10-30T12:14:32.446 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ali15x3.ko 2023-10-30T12:14:32.446 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-30T12:14:32.447 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-30T12:14:32.447 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-30T12:14:32.447 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-cbus-gpio.ko 2023-10-30T12:14:32.447 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 6929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-30T12:14:32.447 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-cros-ec-tunnel.ko 2023-10-30T12:14:32.448 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-30T12:14:32.448 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-30T12:14:32.448 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-30T12:14:32.448 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-30T12:14:32.448 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-dln2.ko 2023-10-30T12:14:32.448 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-gpio.ko 2023-10-30T12:14:32.449 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 62817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-30T12:14:32.449 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-30T12:14:32.449 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-30T12:14:32.449 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-kempld.ko 2023-10-30T12:14:32.449 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-30T12:14:32.450 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-30T12:14:32.450 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-ocores.ko 2023-10-30T12:14:32.450 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-30T12:14:32.450 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-30T12:14:32.450 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-30T12:14:32.451 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-robotfuzz-osif.ko 2023-10-30T12:14:32.451 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-30T12:14:32.451 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-30T12:14:32.451 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis5595.ko 2023-10-30T12:14:32.451 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis630.ko 2023-10-30T12:14:32.452 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-30T12:14:32.452 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-taos-evm.ko 2023-10-30T12:14:32.452 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-30T12:14:32.452 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-30T12:14:32.453 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-30T12:14:32.453 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-30T12:14:32.453 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/busses/i2c-xiic.ko 2023-10-30T12:14:32.453 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-mux.ko 2023-10-30T12:14:32.453 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-smbus.ko 2023-10-30T12:14:32.453 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/i2c-stub.ko 2023-10-30T12:14:32.454 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/ 2023-10-30T12:14:32.454 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-gpio.ko 2023-10-30T12:14:32.454 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-pca9541.ko 2023-10-30T12:14:32.454 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/i2c/muxes/i2c-mux-pca954x.ko 2023-10-30T12:14:32.454 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ 2023-10-30T12:14:32.455 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ 2023-10-30T12:14:32.455 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 273609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-30T12:14:32.455 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 879969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_core.ko 2023-10-30T12:14:32.455 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 69337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-30T12:14:32.455 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 324217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-30T12:14:32.456 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 97985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-30T12:14:32.456 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 282273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-30T12:14:32.456 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-30T12:14:32.456 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ 2023-10-30T12:14:32.456 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/cxgb4/ 2023-10-30T12:14:32.457 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 493073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-30T12:14:32.457 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx4/ 2023-10-30T12:14:32.457 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 419737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-30T12:14:32.457 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx5/ 2023-10-30T12:14:32.457 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 739025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-30T12:14:32.458 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mthca/ 2023-10-30T12:14:32.458 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 296625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-30T12:14:32.458 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ocrdma/ 2023-10-30T12:14:32.458 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 153697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/ocrdma/ocrdma.ko 2023-10-30T12:14:32.586 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/usnic/ 2023-10-30T12:14:32.587 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 168969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/hw/usnic/usnic_verbs.ko 2023-10-30T12:14:32.587 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ 2023-10-30T12:14:32.587 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ipoib/ 2023-10-30T12:14:32.587 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 263081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-30T12:14:32.587 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/iser/ 2023-10-30T12:14:32.588 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 100241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-30T12:14:32.588 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/isert/ 2023-10-30T12:14:32.588 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 111425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-30T12:14:32.588 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srp/ 2023-10-30T12:14:32.588 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 128753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-30T12:14:32.589 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srpt/ 2023-10-30T12:14:32.589 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 129529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-30T12:14:32.589 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/ 2023-10-30T12:14:32.589 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/evbug.ko 2023-10-30T12:14:32.589 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/ff-memless.ko 2023-10-30T12:14:32.590 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/ 2023-10-30T12:14:32.590 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-30T12:14:32.590 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-30T12:14:32.590 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/gameport/gameport.ko 2023-10-30T12:14:32.590 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/joydev.ko 2023-10-30T12:14:32.591 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/ 2023-10-30T12:14:32.591 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5520-keys.ko 2023-10-30T12:14:32.591 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5588-keys.ko 2023-10-30T12:14:32.591 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/adp5589-keys.ko 2023-10-30T12:14:32.591 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/cros_ec_keyb.ko 2023-10-30T12:14:32.591 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-30T12:14:32.592 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/gpio_keys_polled.ko 2023-10-30T12:14:32.592 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lkkbd.ko 2023-10-30T12:14:32.592 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lm8323.ko 2023-10-30T12:14:32.592 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/lm8333.ko 2023-10-30T12:14:32.592 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/matrix_keypad.ko 2023-10-30T12:14:32.593 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/max7359_keypad.ko 2023-10-30T12:14:32.593 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/mcs_touchkey.ko 2023-10-30T12:14:32.593 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/mpr121_touchkey.ko 2023-10-30T12:14:32.593 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/newtonkbd.ko 2023-10-30T12:14:32.593 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/opencores-kbd.ko 2023-10-30T12:14:32.594 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/qt1070.ko 2023-10-30T12:14:32.594 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/qt2160.ko 2023-10-30T12:14:32.594 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/samsung-keypad.ko 2023-10-30T12:14:32.594 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/stowaway.ko 2023-10-30T12:14:32.595 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/sunkbd.ko 2023-10-30T12:14:32.595 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/tca6416-keypad.ko 2023-10-30T12:14:32.595 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/tca8418_keypad.ko 2023-10-30T12:14:32.595 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/twl4030_keypad.ko 2023-10-30T12:14:32.595 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/keyboard/xtkbd.ko 2023-10-30T12:14:32.596 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/matrix-keymap.ko 2023-10-30T12:14:32.596 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/ 2023-10-30T12:14:32.596 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/appletouch.ko 2023-10-30T12:14:32.596 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/bcm5974.ko 2023-10-30T12:14:32.596 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 123681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/cyapatp.ko 2023-10-30T12:14:32.597 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-30T12:14:32.597 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/gpio_mouse.ko 2023-10-30T12:14:32.597 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 335977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/psmouse.ko 2023-10-30T12:14:32.597 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/sermouse.ko 2023-10-30T12:14:32.597 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-30T12:14:32.598 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-30T12:14:32.598 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-30T12:14:32.598 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/rmi4/ 2023-10-30T12:14:32.598 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 109873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-30T12:14:32.598 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ 2023-10-30T12:14:32.599 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/altera_ps2.ko 2023-10-30T12:14:32.599 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/arc_ps2.ko 2023-10-30T12:14:32.599 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ct82c710.ko 2023-10-30T12:14:32.599 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-30T12:14:32.599 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/parkbd.ko 2023-10-30T12:14:32.600 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/pcips2.ko 2023-10-30T12:14:32.600 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/ps2mult.ko 2023-10-30T12:14:32.600 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/serio_raw.ko 2023-10-30T12:14:32.600 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/serio/serport.ko 2023-10-30T12:14:32.601 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/input/sparse-keymap.ko 2023-10-30T12:14:32.601 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/ 2023-10-30T12:14:32.601 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/amd/ 2023-10-30T12:14:32.601 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-30T12:14:32.601 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/ 2023-10-30T12:14:32.602 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-88pm860x.ko 2023-10-30T12:14:32.602 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-adp5520.ko 2023-10-30T12:14:32.602 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-bd2802.ko 2023-10-30T12:14:32.602 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-blinkm.ko 2023-10-30T12:14:32.602 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-da903x.ko 2023-10-30T12:14:32.603 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-da9052.ko 2023-10-30T12:14:32.603 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-gpio.ko 2023-10-30T12:14:32.603 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3530.ko 2023-10-30T12:14:32.603 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3533.ko 2023-10-30T12:14:32.603 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm355x.ko 2023-10-30T12:14:32.603 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lm3642.ko 2023-10-30T12:14:32.604 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lp3944.ko 2023-10-30T12:14:32.604 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-lp8788.ko 2023-10-30T12:14:32.604 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-max8997.ko 2023-10-30T12:14:32.604 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-mc13783.ko 2023-10-30T12:14:32.604 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-menf21bmc.ko 2023-10-30T12:14:32.605 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca9532.ko 2023-10-30T12:14:32.605 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca955x.ko 2023-10-30T12:14:32.605 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-pca963x.ko 2023-10-30T12:14:32.605 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-ss4200.ko 2023-10-30T12:14:32.605 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-tca6507.ko 2023-10-30T12:14:32.606 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-wm831x-status.ko 2023-10-30T12:14:32.606 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/leds-wm8350.ko 2023-10-30T12:14:32.606 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ 2023-10-30T12:14:32.606 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-30T12:14:32.606 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-30T12:14:32.607 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-30T12:14:32.607 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-30T12:14:32.607 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-30T12:14:32.607 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-30T12:14:32.607 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-30T12:14:32.608 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-30T12:14:32.608 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/ 2023-10-30T12:14:32.608 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/bcache/ 2023-10-30T12:14:32.608 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 654969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/bcache/bcache.ko 2023-10-30T12:14:32.608 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-bio-prison.ko 2023-10-30T12:14:32.608 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-bufio.ko 2023-10-30T12:14:32.609 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-cache-smq.ko 2023-10-30T12:14:32.609 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 156009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-cache.ko 2023-10-30T12:14:32.609 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 106897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-crypt.ko 2023-10-30T12:14:32.609 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-delay.ko 2023-10-30T12:14:32.609 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-era.ko 2023-10-30T12:14:32.610 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-flakey.ko 2023-10-30T12:14:32.610 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-log-userspace.ko 2023-10-30T12:14:32.610 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-log.ko 2023-10-30T12:14:32.610 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-mirror.ko 2023-10-30T12:14:32.610 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-multipath.ko 2023-10-30T12:14:32.611 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-queue-length.ko 2023-10-30T12:14:32.611 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-raid.ko 2023-10-30T12:14:32.611 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-region-hash.ko 2023-10-30T12:14:32.611 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-round-robin.ko 2023-10-30T12:14:32.611 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-service-time.ko 2023-10-30T12:14:32.612 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 109593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-snapshot.ko 2023-10-30T12:14:32.612 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-switch.ko 2023-10-30T12:14:32.612 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 181865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-thin-pool.ko 2023-10-30T12:14:32.612 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-verity.ko 2023-10-30T12:14:32.612 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/dm-zero.ko 2023-10-30T12:14:32.613 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/faulty.ko 2023-10-30T12:14:32.613 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/linear.ko 2023-10-30T12:14:32.613 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/multipath.ko 2023-10-30T12:14:32.613 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/persistent-data/ 2023-10-30T12:14:32.613 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 199937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-30T12:14:32.614 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid0.ko 2023-10-30T12:14:32.614 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 113073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid1.ko 2023-10-30T12:14:32.614 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 158681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid10.ko 2023-10-30T12:14:32.614 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 395721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/md/raid456.ko 2023-10-30T12:14:32.614 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/ 2023-10-30T12:14:32.615 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/ 2023-10-30T12:14:32.615 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 178137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptbase.ko 2023-10-30T12:14:32.615 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 63385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptctl.ko 2023-10-30T12:14:32.615 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptfc.ko 2023-10-30T12:14:32.615 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptlan.ko 2023-10-30T12:14:32.616 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 123321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptsas.ko 2023-10-30T12:14:32.616 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 79041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptscsih.ko 2023-10-30T12:14:32.616 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/message/fusion/mptspi.ko 2023-10-30T12:14:32.616 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/ 2023-10-30T12:14:32.616 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm800.ko 2023-10-30T12:14:32.616 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm805.ko 2023-10-30T12:14:32.617 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/88pm80x.ko 2023-10-30T12:14:32.617 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/arizona-i2c.ko 2023-10-30T12:14:32.617 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 123721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/arizona.ko 2023-10-30T12:14:32.617 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/bcm590xx.ko 2023-10-30T12:14:32.617 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/cros_ec_dev.ko 2023-10-30T12:14:32.618 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/dln2.ko 2023-10-30T12:14:32.618 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/janz-cmodio.ko 2023-10-30T12:14:32.618 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/kempld-core.ko 2023-10-30T12:14:32.618 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lm3533-core.ko 2023-10-30T12:14:32.619 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lm3533-ctrlbank.ko 2023-10-30T12:14:32.619 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lp3943.ko 2023-10-30T12:14:32.619 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lpc_ich.ko 2023-10-30T12:14:32.619 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/lpc_sch.ko 2023-10-30T12:14:32.619 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/max8907.ko 2023-10-30T12:14:32.620 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/mc13xxx-core.ko 2023-10-30T12:14:32.620 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/mc13xxx-i2c.ko 2023-10-30T12:14:32.620 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/menf21bmc.ko 2023-10-30T12:14:32.620 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633-adc.ko 2023-10-30T12:14:32.620 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633-gpio.ko 2023-10-30T12:14:32.620 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/pcf50633.ko 2023-10-30T12:14:32.621 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/rdc321x-southbridge.ko 2023-10-30T12:14:32.621 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/retu-mfd.ko 2023-10-30T12:14:32.621 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 80425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/si476x-core.ko 2023-10-30T12:14:32.621 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 48457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/sm501.ko 2023-10-30T12:14:32.621 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/ti_am335x_tscadc.ko 2023-10-30T12:14:32.621 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/tps65010.ko 2023-10-30T12:14:32.622 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/tps6507x.ko 2023-10-30T12:14:32.622 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/viperboard.ko 2023-10-30T12:14:32.622 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/vx855.ko 2023-10-30T12:14:32.622 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/wl1273-core.ko 2023-10-30T12:14:32.622 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/mfd/wm8994.ko 2023-10-30T12:14:32.623 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ 2023-10-30T12:14:32.623 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ad525x_dpot-i2c.ko 2023-10-30T12:14:32.623 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 48489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ad525x_dpot.ko 2023-10-30T12:14:32.623 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/altera-stapl/ 2023-10-30T12:14:32.623 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-30T12:14:32.624 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/apds9802als.ko 2023-10-30T12:14:32.624 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/apds990x.ko 2023-10-30T12:14:32.624 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 45593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/bh1770glc.ko 2023-10-30T12:14:32.624 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/ 2023-10-30T12:14:32.624 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/c2port-duramar2150.ko 2023-10-30T12:14:32.625 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/c2port/core.ko 2023-10-30T12:14:32.625 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/cb710/ 2023-10-30T12:14:32.625 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/cb710/cb710.ko 2023-10-30T12:14:32.625 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ds1682.ko 2023-10-30T12:14:32.625 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/dummy-irq.ko 2023-10-30T12:14:32.625 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/echo/ 2023-10-30T12:14:32.626 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/echo/echo.ko 2023-10-30T12:14:32.626 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/ 2023-10-30T12:14:32.626 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/at24.ko 2023-10-30T12:14:32.626 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-30T12:14:32.626 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-30T12:14:32.627 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/eeprom/max6875.ko 2023-10-30T12:14:32.627 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/enclosure.ko 2023-10-30T12:14:32.627 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/genwqe/ 2023-10-30T12:14:32.627 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 152201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/genwqe/genwqe_card.ko 2023-10-30T12:14:32.627 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/hmc6352.ko 2023-10-30T12:14:32.628 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/hpilo.ko 2023-10-30T12:14:32.628 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ibmasm/ 2023-10-30T12:14:32.628 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 72129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ibmasm/ibmasm.ko 2023-10-30T12:14:32.628 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ics932s401.ko 2023-10-30T12:14:32.628 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/isl29003.ko 2023-10-30T12:14:32.629 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/isl29020.ko 2023-10-30T12:14:32.629 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/ 2023-10-30T12:14:32.629 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-30T12:14:32.629 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-30T12:14:32.630 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/ 2023-10-30T12:14:32.630 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 125209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei-me.ko 2023-10-30T12:14:32.630 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 56521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei-txe.ko 2023-10-30T12:14:32.630 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 310113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/mei/mei.ko 2023-10-30T12:14:32.630 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/phantom.ko 2023-10-30T12:14:32.631 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ti-st/ 2023-10-30T12:14:32.631 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 74265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/ti-st/st_drv.ko 2023-10-30T12:14:32.631 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tifm_7xx1.ko 2023-10-30T12:14:32.631 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tifm_core.ko 2023-10-30T12:14:32.631 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/tsl2550.ko 2023-10-30T12:14:32.631 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 48697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_balloon.ko 2023-10-30T12:14:32.632 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_vmci/ 2023-10-30T12:14:32.632 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 183609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-30T12:14:32.632 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ 2023-10-30T12:14:32.632 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/bonding/ 2023-10-30T12:14:32.632 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 490129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/bonding/bonding.ko 2023-10-30T12:14:32.632 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/ 2023-10-30T12:14:32.633 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/b53/ 2023-10-30T12:14:32.633 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 106033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-30T12:14:32.633 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-30T12:14:32.633 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-30T12:14:32.633 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6xxx/ 2023-10-30T12:14:32.633 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 453737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-30T12:14:32.634 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/dummy.ko 2023-10-30T12:14:32.634 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/eql.ko 2023-10-30T12:14:32.634 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ 2023-10-30T12:14:32.634 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/ 2023-10-30T12:14:32.634 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 45785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-30T12:14:32.635 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-30T12:14:32.635 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 130553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-30T12:14:32.635 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 68849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-30T12:14:32.635 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/ 2023-10-30T12:14:32.635 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-30T12:14:32.636 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-30T12:14:32.636 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-30T12:14:32.636 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 99113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-30T12:14:32.636 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/adaptec/ 2023-10-30T12:14:32.636 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-30T12:14:32.637 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/agere/ 2023-10-30T12:14:32.637 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 69201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-30T12:14:32.637 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/alteon/ 2023-10-30T12:14:32.637 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-30T12:14:32.637 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/altera/ 2023-10-30T12:14:32.638 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-30T12:14:32.638 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/ 2023-10-30T12:14:32.638 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-30T12:14:32.638 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-30T12:14:32.638 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 87297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-30T12:14:32.639 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/ 2023-10-30T12:14:32.639 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/alx/ 2023-10-30T12:14:32.639 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 97337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-30T12:14:32.639 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1c/ 2023-10-30T12:14:32.639 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 104673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-30T12:14:32.639 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1e/ 2023-10-30T12:14:32.640 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 90257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-30T12:14:32.640 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/ 2023-10-30T12:14:32.640 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 81569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-30T12:14:32.640 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 69729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-30T12:14:32.640 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/ 2023-10-30T12:14:32.641 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-30T12:14:32.641 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 176761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-30T12:14:32.641 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2x/ 2023-10-30T12:14:32.641 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1421497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-30T12:14:32.641 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 128569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-30T12:14:32.641 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/genet/ 2023-10-30T12:14:32.642 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 132089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-30T12:14:32.642 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 312081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-30T12:14:32.642 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/ 2023-10-30T12:14:32.642 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/bna/ 2023-10-30T12:14:32.642 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 323513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-30T12:14:32.643 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/ 2023-10-30T12:14:32.643 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb/ 2023-10-30T12:14:32.643 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 193497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-30T12:14:32.643 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb3/ 2023-10-30T12:14:32.643 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 347417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-30T12:14:32.643 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4/ 2023-10-30T12:14:32.644 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1044433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-30T12:14:32.644 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4vf/ 2023-10-30T12:14:32.644 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 166193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-30T12:14:32.644 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/libcxgb/ 2023-10-30T12:14:32.644 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-30T12:14:32.645 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/ 2023-10-30T12:14:32.645 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/enic/ 2023-10-30T12:14:32.645 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 204345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-30T12:14:32.645 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/ 2023-10-30T12:14:32.645 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/ 2023-10-30T12:14:32.645 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 69449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-30T12:14:32.646 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 61489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-30T12:14:32.646 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 141225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-30T12:14:32.646 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-30T12:14:32.646 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 62817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-30T12:14:32.646 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-30T12:14:32.647 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/ 2023-10-30T12:14:32.647 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-30T12:14:32.647 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 62833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-30T12:14:32.647 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 45721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/dnet.ko 2023-10-30T12:14:32.647 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ec_bhf.ko 2023-10-30T12:14:32.648 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/ 2023-10-30T12:14:32.648 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/benet/ 2023-10-30T12:14:32.648 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 281137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-30T12:14:32.648 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ethoc.ko 2023-10-30T12:14:32.648 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fealnx.ko 2023-10-30T12:14:32.649 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fujitsu/ 2023-10-30T12:14:32.649 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 45345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/fujitsu/fmvj18x_cs.ko 2023-10-30T12:14:32.649 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ 2023-10-30T12:14:32.649 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 90465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-30T12:14:32.650 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000/ 2023-10-30T12:14:32.650 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 295577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-30T12:14:32.650 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000e/ 2023-10-30T12:14:32.650 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 584497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-30T12:14:32.650 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/fm10k/ 2023-10-30T12:14:32.651 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 246209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-30T12:14:32.651 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/i40e/ 2023-10-30T12:14:32.651 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 925697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-30T12:14:32.651 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/iavf/ 2023-10-30T12:14:32.651 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 302849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-30T12:14:32.651 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igb/ 2023-10-30T12:14:32.652 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 514681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-30T12:14:32.652 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igbvf/ 2023-10-30T12:14:32.652 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 93593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-30T12:14:32.652 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbe/ 2023-10-30T12:14:32.652 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 734305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-30T12:14:32.653 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbevf/ 2023-10-30T12:14:32.653 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 151769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-30T12:14:32.653 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 86921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/jme.ko 2023-10-30T12:14:32.653 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/ 2023-10-30T12:14:32.653 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-30T12:14:32.653 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 102289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-30T12:14:32.654 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 122649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-30T12:14:32.654 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/ 2023-10-30T12:14:32.654 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/ 2023-10-30T12:14:32.654 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 727601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-30T12:14:32.654 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 278937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-30T12:14:32.655 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/ 2023-10-30T12:14:32.655 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/core/ 2023-10-30T12:14:32.655 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1113993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-30T12:14:32.655 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ 2023-10-30T12:14:32.655 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8842.ko 2023-10-30T12:14:32.655 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8851_common.ko 2023-10-30T12:14:32.656 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ks8851_par.ko 2023-10-30T12:14:32.656 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 95665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-30T12:14:32.656 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/ 2023-10-30T12:14:32.656 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/myri10ge/ 2023-10-30T12:14:32.656 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 108649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-30T12:14:32.657 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/ 2023-10-30T12:14:32.657 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 73441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-30T12:14:32.657 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-30T12:14:32.657 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/neterion/ 2023-10-30T12:14:32.657 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 150905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-30T12:14:32.657 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/nvidia/ 2023-10-30T12:14:32.658 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 136721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-30T12:14:32.658 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/ 2023-10-30T12:14:32.658 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-30T12:14:32.658 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-30T12:14:32.658 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/ 2023-10-30T12:14:32.659 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/netxen/ 2023-10-30T12:14:32.659 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 227897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-30T12:14:32.659 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 87081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-30T12:14:32.659 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qlcnic/ 2023-10-30T12:14:32.659 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 672049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-30T12:14:32.660 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/rdc/ 2023-10-30T12:14:32.660 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-30T12:14:32.660 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/ 2023-10-30T12:14:32.660 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-30T12:14:32.660 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 83985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-30T12:14:32.661 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-30T12:14:32.661 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 189801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-30T12:14:32.661 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/ 2023-10-30T12:14:32.661 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/sxgbe/ 2023-10-30T12:14:32.661 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 104241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/samsung/sxgbe/samsung-sxgbe.ko 2023-10-30T12:14:32.662 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sfc/ 2023-10-30T12:14:32.662 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1053921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-30T12:14:32.662 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/silan/ 2023-10-30T12:14:32.662 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-30T12:14:32.662 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/ 2023-10-30T12:14:32.663 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-30T12:14:32.663 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 63337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-30T12:14:32.663 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/ 2023-10-30T12:14:32.663 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 61601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-30T12:14:32.663 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-30T12:14:32.663 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 73409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-30T12:14:32.664 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 68177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-30T12:14:32.664 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/ 2023-10-30T12:14:32.664 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/ 2023-10-30T12:14:32.664 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-generic.ko 2023-10-30T12:14:32.664 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-30T12:14:32.664 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac-platform.ko 2023-10-30T12:14:32.665 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 486961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-30T12:14:32.665 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/ 2023-10-30T12:14:32.665 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 103337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-30T12:14:32.665 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 166161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-30T12:14:32.665 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-30T12:14:32.666 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-30T12:14:32.666 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/tehuti/ 2023-10-30T12:14:32.666 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-30T12:14:32.666 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ti/ 2023-10-30T12:14:32.666 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 86529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-30T12:14:32.667 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/ 2023-10-30T12:14:32.667 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 80273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-30T12:14:32.667 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 83633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-30T12:14:32.667 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/ 2023-10-30T12:14:32.667 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-30T12:14:32.668 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-30T12:14:32.668 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/xircom/ 2023-10-30T12:14:32.668 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-30T12:14:32.668 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/ 2023-10-30T12:14:32.668 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/defxx.ko 2023-10-30T12:14:32.668 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/skfp/ 2023-10-30T12:14:32.669 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 216985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/fddi/skfp/skfp.ko 2023-10-30T12:14:32.669 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 77345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/geneve.ko 2023-10-30T12:14:32.669 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/hyperv/ 2023-10-30T12:14:32.669 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 235521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-30T12:14:32.669 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ieee802154/ 2023-10-30T12:14:32.670 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-30T12:14:32.670 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ifb.ko 2023-10-30T12:14:32.670 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ipvlan/ 2023-10-30T12:14:32.670 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 83769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-30T12:14:32.670 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 62209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/macvlan.ko 2023-10-30T12:14:32.671 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/macvtap.ko 2023-10-30T12:14:32.671 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/ 2023-10-30T12:14:32.671 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-30T12:14:32.671 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-30T12:14:32.671 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-gpio.ko 2023-10-30T12:14:32.672 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-30T12:14:32.672 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mdio.ko 2023-10-30T12:14:32.672 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/mii.ko 2023-10-30T12:14:32.672 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/netconsole.ko 2023-10-30T12:14:32.672 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/nlmon.ko 2023-10-30T12:14:32.673 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/ 2023-10-30T12:14:32.673 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-30T12:14:32.673 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-30T12:14:32.673 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/ 2023-10-30T12:14:32.673 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/amd.ko 2023-10-30T12:14:32.674 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-30T12:14:32.674 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-30T12:14:32.674 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/bcm87xx.ko 2023-10-30T12:14:32.674 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/broadcom.ko 2023-10-30T12:14:32.675 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/cicada.ko 2023-10-30T12:14:32.675 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/davicom.ko 2023-10-30T12:14:32.675 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/et1011c.ko 2023-10-30T12:14:32.675 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/icplus.ko 2023-10-30T12:14:32.675 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/lxt.ko 2023-10-30T12:14:32.677 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 82049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/marvell.ko 2023-10-30T12:14:32.677 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 76177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/micrel.ko 2023-10-30T12:14:32.677 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/national.ko 2023-10-30T12:14:32.677 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 97857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/phylink.ko 2023-10-30T12:14:32.678 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/qsemi.ko 2023-10-30T12:14:32.678 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/realtek.ko 2023-10-30T12:14:32.678 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/smsc.ko 2023-10-30T12:14:32.678 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/ste10Xp.ko 2023-10-30T12:14:32.679 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/phy/vitesse.ko 2023-10-30T12:14:32.679 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/plip/ 2023-10-30T12:14:32.679 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/plip/plip.ko 2023-10-30T12:14:32.679 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ 2023-10-30T12:14:32.680 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-30T12:14:32.680 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_async.ko 2023-10-30T12:14:32.680 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-30T12:14:32.680 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-30T12:14:32.680 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-30T12:14:32.681 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pppoe.ko 2023-10-30T12:14:32.681 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pppox.ko 2023-10-30T12:14:32.681 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/ppp/pptp.ko 2023-10-30T12:14:32.681 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/sb1000.ko 2023-10-30T12:14:32.681 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/slip/ 2023-10-30T12:14:32.682 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/slip/slip.ko 2023-10-30T12:14:32.682 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/sungem_phy.ko 2023-10-30T12:14:32.682 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 64385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/tap.ko 2023-10-30T12:14:32.682 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/ 2023-10-30T12:14:32.682 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 106169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team.ko 2023-10-30T12:14:32.683 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-30T12:14:32.683 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-30T12:14:32.683 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-30T12:14:32.683 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_random.ko 2023-10-30T12:14:32.683 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-30T12:14:32.683 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 87017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/veth.ko 2023-10-30T12:14:32.684 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vmxnet3/ 2023-10-30T12:14:32.684 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 144297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-30T12:14:32.684 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vxlan/ 2023-10-30T12:14:32.684 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 297329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/vxlan/vxlan.ko 2023-10-30T12:14:32.684 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/xen-netback/ 2023-10-30T12:14:32.685 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 144649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-30T12:14:32.685 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ntb/ 2023-10-30T12:14:32.685 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ntb/ntb.ko 2023-10-30T12:14:32.685 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/ 2023-10-30T12:14:32.685 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/ 2023-10-30T12:14:32.686 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 264937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/nvme-core.ko 2023-10-30T12:14:32.686 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 102209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/nvme/host/nvme.ko 2023-10-30T12:14:32.686 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/ 2023-10-30T12:14:32.686 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 133841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport.ko 2023-10-30T12:14:32.686 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_cs.ko 2023-10-30T12:14:32.687 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 84465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_pc.ko 2023-10-30T12:14:32.687 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/parport/parport_serial.ko 2023-10-30T12:14:32.687 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/ 2023-10-30T12:14:32.687 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/ 2023-10-30T12:14:32.687 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-30T12:14:32.687 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/cpcihp_generic.ko 2023-10-30T12:14:32.688 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/hotplug/cpcihp_zt5550.ko 2023-10-30T12:14:32.688 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/pci-stub.ko 2023-10-30T12:14:32.688 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pci/xen-pcifront.ko 2023-10-30T12:14:32.688 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/ 2023-10-30T12:14:32.689 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/i82092.ko 2023-10-30T12:14:32.689 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 145809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia.ko 2023-10-30T12:14:32.689 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 62137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia_core.ko 2023-10-30T12:14:32.689 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pcmcia_rsrc.ko 2023-10-30T12:14:32.690 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/pd6729.ko 2023-10-30T12:14:32.690 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 112889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-30T12:14:32.690 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/ 2023-10-30T12:14:32.690 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/broadcom/ 2023-10-30T12:14:32.690 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/broadcom/phy-bcm-kona-usb2.ko 2023-10-30T12:14:32.691 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/samsung/ 2023-10-30T12:14:32.691 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/phy/samsung/phy-exynos-usb2.ko 2023-10-30T12:14:32.691 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/ 2023-10-30T12:14:32.691 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/intel/ 2023-10-30T12:14:32.691 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-30T12:14:32.692 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/ 2023-10-30T12:14:32.692 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/ 2023-10-30T12:14:32.692 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros-ec-sensorhub.ko 2023-10-30T12:14:32.692 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec.ko 2023-10-30T12:14:32.692 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_chardev.ko 2023-10-30T12:14:32.693 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_debugfs.ko 2023-10-30T12:14:32.693 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_lightbar.ko 2023-10-30T12:14:32.693 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_ec_sysfs.ko 2023-10-30T12:14:32.693 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/chrome/cros_usbpd_notify.ko 2023-10-30T12:14:32.693 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/ 2023-10-30T12:14:32.693 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-30T12:14:32.694 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/acerhdf.ko 2023-10-30T12:14:32.694 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-30T12:14:32.694 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-30T12:14:32.694 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-30T12:14:32.694 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 127569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-30T12:14:32.695 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-30T12:14:32.695 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-30T12:14:32.695 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-30T12:14:32.695 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-30T12:14:32.695 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-30T12:14:32.696 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/hdaps.ko 2023-10-30T12:14:32.696 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/ibm_rtl.ko 2023-10-30T12:14:32.696 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/ 2023-10-30T12:14:32.696 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-30T12:14:32.696 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel/intel-smartconnect.ko 2023-10-30T12:14:32.697 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/intel_ips.ko 2023-10-30T12:14:32.697 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-30T12:14:32.697 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-30T12:14:32.697 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-30T12:14:32.697 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-30T12:14:32.697 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-30T12:14:32.698 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-30T12:14:32.698 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-30T12:14:32.698 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-30T12:14:32.698 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-30T12:14:32.698 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/platform/x86/wmi.ko 2023-10-30T12:14:32.699 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/ 2023-10-30T12:14:32.699 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ 2023-10-30T12:14:32.699 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/88pm860x_battery.ko 2023-10-30T12:14:32.699 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/88pm860x_charger.ko 2023-10-30T12:14:32.699 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 55057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq2415x_charger.ko 2023-10-30T12:14:32.700 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq24190_charger.ko 2023-10-30T12:14:32.700 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/bq24735-charger.ko 2023-10-30T12:14:32.700 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/cros_peripheral_charger.ko 2023-10-30T12:14:32.700 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/da9030_battery.ko 2023-10-30T12:14:32.701 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/da9052-battery.ko 2023-10-30T12:14:32.701 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2760_battery.ko 2023-10-30T12:14:32.701 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2780_battery.ko 2023-10-30T12:14:32.701 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2781_battery.ko 2023-10-30T12:14:32.701 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/ds2782_battery.ko 2023-10-30T12:14:32.702 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/gpio-charger.ko 2023-10-30T12:14:32.702 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/isp1704_charger.ko 2023-10-30T12:14:32.702 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/lp8727_charger.ko 2023-10-30T12:14:32.702 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max14577_charger.ko 2023-10-30T12:14:32.702 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max17040_battery.ko 2023-10-30T12:14:32.703 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max17042_battery.ko 2023-10-30T12:14:32.703 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max8903_charger.ko 2023-10-30T12:14:32.703 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/max8925_power.ko 2023-10-30T12:14:32.703 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/pcf50633-charger.ko 2023-10-30T12:14:32.703 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/sbs-battery.ko 2023-10-30T12:14:32.704 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/test_power.ko 2023-10-30T12:14:32.704 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/tps65090-charger.ko 2023-10-30T12:14:32.704 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm831x_backup.ko 2023-10-30T12:14:32.704 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm831x_power.ko 2023-10-30T12:14:32.704 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/power/supply/wm8350_power.ko 2023-10-30T12:14:32.704 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/ 2023-10-30T12:14:32.705 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-30T12:14:32.705 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-30T12:14:32.705 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/ 2023-10-30T12:14:32.705 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/ 2023-10-30T12:14:32.706 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-30T12:14:32.706 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-30T12:14:32.706 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/clients/pps_parport.ko 2023-10-30T12:14:32.706 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/pps/pps_core.ko 2023-10-30T12:14:32.706 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ 2023-10-30T12:14:32.707 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ptp.ko 2023-10-30T12:14:32.707 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ptp/ptp_kvm.ko 2023-10-30T12:14:32.707 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/ 2023-10-30T12:14:32.707 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-88pm80x.ko 2023-10-30T12:14:32.707 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-88pm860x.ko 2023-10-30T12:14:32.708 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-30T12:14:32.708 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9052.ko 2023-10-30T12:14:32.708 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9055.ko 2023-10-30T12:14:32.708 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-da9063.ko 2023-10-30T12:14:32.708 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-30T12:14:32.709 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 68577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-30T12:14:32.709 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-30T12:14:32.709 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-30T12:14:32.709 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-30T12:14:32.709 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-30T12:14:32.710 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-30T12:14:32.710 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-30T12:14:32.710 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-30T12:14:32.710 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-em3027.ko 2023-10-30T12:14:32.710 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-30T12:14:32.711 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-30T12:14:32.711 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-30T12:14:32.711 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-lp8788.ko 2023-10-30T12:14:32.711 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-30T12:14:32.711 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-30T12:14:32.711 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-30T12:14:32.712 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-m48t86.ko 2023-10-30T12:14:32.712 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max6900.ko 2023-10-30T12:14:32.712 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8907.ko 2023-10-30T12:14:32.712 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8925.ko 2023-10-30T12:14:32.712 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8997.ko 2023-10-30T12:14:32.713 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-max8998.ko 2023-10-30T12:14:32.713 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-mc13xxx.ko 2023-10-30T12:14:32.713 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-30T12:14:32.713 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-palmas.ko 2023-10-30T12:14:32.714 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-30T12:14:32.714 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf50633.ko 2023-10-30T12:14:32.714 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-30T12:14:32.714 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-30T12:14:32.714 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-30T12:14:32.714 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-30T12:14:32.715 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rc5t583.ko 2023-10-30T12:14:32.715 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-30T12:14:32.715 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-30T12:14:32.715 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-30T12:14:32.715 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-30T12:14:32.716 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-30T12:14:32.716 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-s35390a.ko 2023-10-30T12:14:32.716 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-30T12:14:32.716 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-tps6586x.ko 2023-10-30T12:14:32.716 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-tps65910.ko 2023-10-30T12:14:32.717 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-wm831x.ko 2023-10-30T12:14:32.717 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-wm8350.ko 2023-10-30T12:14:32.717 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/rtc/rtc-x1205.ko 2023-10-30T12:14:32.717 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ 2023-10-30T12:14:32.717 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 77617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-9xxx.ko 2023-10-30T12:14:32.718 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-sas.ko 2023-10-30T12:14:32.718 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 55169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/3w-xxxx.ko 2023-10-30T12:14:32.718 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 135185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/BusLogic.ko 2023-10-30T12:14:32.718 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/a100u2w.ko 2023-10-30T12:14:32.718 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aacraid/ 2023-10-30T12:14:32.719 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 272305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-30T12:14:32.719 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 100617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/advansys.ko 2023-10-30T12:14:32.719 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/ 2023-10-30T12:14:32.719 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 356265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-30T12:14:32.719 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 295801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-30T12:14:32.720 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic94xx/ 2023-10-30T12:14:32.720 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 189185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/aic94xx/aic94xx.ko 2023-10-30T12:14:32.720 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/am53c974.ko 2023-10-30T12:14:32.720 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/arcmsr/ 2023-10-30T12:14:32.720 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 106881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-30T12:14:32.721 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/atp870u.ko 2023-10-30T12:14:32.721 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/be2iscsi/ 2023-10-30T12:14:32.721 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 246985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-30T12:14:32.721 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bfa/ 2023-10-30T12:14:32.721 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1171945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bfa/bfa.ko 2023-10-30T12:14:32.721 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2fc/ 2023-10-30T12:14:32.722 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 237273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-30T12:14:32.722 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2i/ 2023-10-30T12:14:32.722 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 115369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-30T12:14:32.722 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ch.ko 2023-10-30T12:14:32.722 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/csiostor/ 2023-10-30T12:14:32.723 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 308921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-30T12:14:32.723 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/ 2023-10-30T12:14:32.723 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb3i/ 2023-10-30T12:14:32.723 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 76145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-30T12:14:32.723 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb4i/ 2023-10-30T12:14:32.724 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 101449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-30T12:14:32.724 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 146689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-30T12:14:32.724 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 75553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/dc395x.ko 2023-10-30T12:14:32.724 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/dmx3191d.ko 2023-10-30T12:14:32.724 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esas2r/ 2023-10-30T12:14:32.725 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 212425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-30T12:14:32.725 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/esp_scsi.ko 2023-10-30T12:14:32.725 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/ 2023-10-30T12:14:32.725 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-30T12:14:32.725 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 158505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-30T12:14:32.725 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fdomain.ko 2023-10-30T12:14:32.726 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fnic/ 2023-10-30T12:14:32.726 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 263489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/fnic/fnic.ko 2023-10-30T12:14:32.726 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 201817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hpsa.ko 2023-10-30T12:14:32.726 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hptiop.ko 2023-10-30T12:14:32.726 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/hv_storvsc.ko 2023-10-30T12:14:32.727 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/imm.ko 2023-10-30T12:14:32.727 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/initio.ko 2023-10-30T12:14:32.727 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 266537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ipr.ko 2023-10-30T12:14:32.727 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 101881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ips.ko 2023-10-30T12:14:32.727 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/isci/ 2023-10-30T12:14:32.727 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 317153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/isci/isci.ko 2023-10-30T12:14:32.728 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-30T12:14:32.728 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 52961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-30T12:14:32.728 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libfc/ 2023-10-30T12:14:32.728 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 314865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libfc/libfc.ko 2023-10-30T12:14:32.728 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 145585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libiscsi.ko 2023-10-30T12:14:32.729 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-30T12:14:32.729 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libsas/ 2023-10-30T12:14:32.729 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 216665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/libsas/libsas.ko 2023-10-30T12:14:32.729 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/lpfc/ 2023-10-30T12:14:32.729 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 2197249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-30T12:14:32.730 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/ 2023-10-30T12:14:32.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 73569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-30T12:14:32.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-30T12:14:32.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 310617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-30T12:14:32.730 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 84721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/megaraid.ko 2023-10-30T12:14:32.731 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mpt3sas/ 2023-10-30T12:14:32.731 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 659777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-30T12:14:32.731 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvsas/ 2023-10-30T12:14:32.731 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 119561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-30T12:14:32.731 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/mvumi.ko 2023-10-30T12:14:32.732 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/ 2023-10-30T12:14:32.732 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 81625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/aha152x_cs.ko 2023-10-30T12:14:32.732 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/fdomain_cs.ko 2023-10-30T12:14:32.732 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/qlogic_cs.ko 2023-10-30T12:14:32.732 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pcmcia/sym53c500_cs.ko 2023-10-30T12:14:32.732 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pm8001/ 2023-10-30T12:14:32.733 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 364105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-30T12:14:32.733 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 123617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/pmcraid.ko 2023-10-30T12:14:32.733 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ppa.ko 2023-10-30T12:14:32.733 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 55361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla1280.ko 2023-10-30T12:14:32.733 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/ 2023-10-30T12:14:32.734 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1876625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-30T12:14:32.734 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 85289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-30T12:14:32.734 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla4xxx/ 2023-10-30T12:14:32.734 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 559033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-30T12:14:32.734 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/qlogicfas408.ko 2023-10-30T12:14:32.735 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/raid_class.ko 2023-10-30T12:14:32.735 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 238825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_debug.ko 2023-10-30T12:14:32.735 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 173937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-30T12:14:32.735 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 279313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-30T12:14:32.735 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 97729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-30T12:14:32.736 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 73881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-30T12:14:32.736 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-30T12:14:32.736 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/ses.ko 2023-10-30T12:14:32.736 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 115889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/st.ko 2023-10-30T12:14:32.736 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/stex.ko 2023-10-30T12:14:32.737 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/sym53c8xx_2/ 2023-10-30T12:14:32.737 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 162121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-30T12:14:32.737 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/virtio_scsi.ko 2023-10-30T12:14:32.737 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-30T12:14:32.737 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/wd719x.ko 2023-10-30T12:14:32.737 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/scsi/xen-scsifront.ko 2023-10-30T12:14:32.738 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ssb/ 2023-10-30T12:14:32.738 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 129441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ssb/ssb.ko 2023-10-30T12:14:32.738 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/ 2023-10-30T12:14:32.738 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/iscsi/ 2023-10-30T12:14:32.738 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 736601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-30T12:14:32.739 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/loopback/ 2023-10-30T12:14:32.739 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-30T12:14:32.739 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_file.ko 2023-10-30T12:14:32.739 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_iblock.ko 2023-10-30T12:14:32.739 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 930321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_mod.ko 2023-10-30T12:14:32.740 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_pscsi.ko 2023-10-30T12:14:32.740 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 122521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/target_core_user.ko 2023-10-30T12:14:32.740 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/tcm_fc/ 2023-10-30T12:14:32.740 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 75593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-30T12:14:32.740 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/ 2023-10-30T12:14:32.741 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/ 2023-10-30T12:14:32.741 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/ 2023-10-30T12:14:32.741 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-30T12:14:32.741 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-30T12:14:32.741 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-30T12:14:32.742 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-30T12:14:32.742 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-30T12:14:32.742 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-30T12:14:32.742 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-30T12:14:32.742 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-30T12:14:32.743 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-30T12:14:32.743 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-30T12:14:32.743 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-30T12:14:32.743 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-30T12:14:32.743 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_powerclamp.ko 2023-10-30T12:14:32.744 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-30T12:14:32.744 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-30T12:14:32.744 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-30T12:14:32.744 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ 2023-10-30T12:14:32.744 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ipwireless/ 2023-10-30T12:14:32.745 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-30T12:14:32.745 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/moxa.ko 2023-10-30T12:14:32.745 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 50953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/mxser.ko 2023-10-30T12:14:32.745 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 108601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/n_gsm.ko 2023-10-30T12:14:32.746 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/n_hdlc.ko 2023-10-30T12:14:32.746 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/nozomi.ko 2023-10-30T12:14:32.746 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/ 2023-10-30T12:14:32.746 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/ 2023-10-30T12:14:32.746 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/8250_dw.ko 2023-10-30T12:14:32.746 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-30T12:14:32.747 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/altera_jtaguart.ko 2023-10-30T12:14:32.747 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/altera_uart.ko 2023-10-30T12:14:32.747 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/arc_uart.ko 2023-10-30T12:14:32.747 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 72425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/fsl_lpuart.ko 2023-10-30T12:14:32.747 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/jsm/ 2023-10-30T12:14:32.748 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 103073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-30T12:14:32.748 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/rp2.ko 2023-10-30T12:14:32.748 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 52817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/tty/serial/sc16is7xx.ko 2023-10-30T12:14:32.748 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/ 2023-10-30T12:14:32.748 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/core/ 2023-10-30T12:14:32.749 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 531265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-30T12:14:32.749 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ 2023-10-30T12:14:32.749 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-30T12:14:32.749 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/ufs/host/ufshcd-pltfrm.ko 2023-10-30T12:14:32.749 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/ 2023-10-30T12:14:32.749 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio.ko 2023-10-30T12:14:32.750 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_aec.ko 2023-10-30T12:14:32.750 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_cif.ko 2023-10-30T12:14:32.750 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_dmem_genirq.ko 2023-10-30T12:14:32.750 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_mf624.ko 2023-10-30T12:14:32.750 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_netx.ko 2023-10-30T12:14:32.751 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_pci_generic.ko 2023-10-30T12:14:32.751 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_pdrv_genirq.ko 2023-10-30T12:14:32.751 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/uio/uio_sercos3.ko 2023-10-30T12:14:32.751 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/ 2023-10-30T12:14:32.751 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/c67x00/ 2023-10-30T12:14:32.752 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 73297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/c67x00/c67x00.ko 2023-10-30T12:14:32.752 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ 2023-10-30T12:14:32.752 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 144377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc.ko 2023-10-30T12:14:32.752 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_msm.ko 2023-10-30T12:14:32.752 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_pci.ko 2023-10-30T12:14:32.753 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/chipidea/ci_hdrc_usb2.ko 2023-10-30T12:14:32.753 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/ 2023-10-30T12:14:32.753 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 87089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/cdc-acm.ko 2023-10-30T12:14:32.753 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-30T12:14:32.753 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/usblp.ko 2023-10-30T12:14:32.753 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/class/usbtmc.ko 2023-10-30T12:14:32.754 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/common/ 2023-10-30T12:14:32.754 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/common/ulpi.ko 2023-10-30T12:14:32.754 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc2/ 2023-10-30T12:14:32.754 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc2/dwc2_pci.ko 2023-10-30T12:14:32.754 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/ 2023-10-30T12:14:32.755 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3-haps.ko 2023-10-30T12:14:32.755 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3-pci.ko 2023-10-30T12:14:32.755 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 444553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/dwc3/dwc3.ko 2023-10-30T12:14:32.755 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/ 2023-10-30T12:14:32.755 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/ 2023-10-30T12:14:32.756 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/u_ether.ko 2023-10-30T12:14:32.756 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/u_serial.ko 2023-10-30T12:14:32.756 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_acm.ko 2023-10-30T12:14:32.756 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ecm.ko 2023-10-30T12:14:32.757 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ecm_subset.ko 2023-10-30T12:14:32.757 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_eem.ko 2023-10-30T12:14:32.757 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 92489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_fs.ko 2023-10-30T12:14:32.757 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_hid.ko 2023-10-30T12:14:32.757 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 122081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_mass_storage.ko 2023-10-30T12:14:32.758 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ncm.ko 2023-10-30T12:14:32.758 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_obex.ko 2023-10-30T12:14:32.758 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_phonet.ko 2023-10-30T12:14:32.758 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_printer.ko 2023-10-30T12:14:32.758 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 85609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_rndis.ko 2023-10-30T12:14:32.759 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_serial.ko 2023-10-30T12:14:32.759 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 58113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_ss_lb.ko 2023-10-30T12:14:32.759 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/function/usb_f_tcm.ko 2023-10-30T12:14:32.759 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/ 2023-10-30T12:14:32.759 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_acm_ms.ko 2023-10-30T12:14:32.760 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_cdc.ko 2023-10-30T12:14:32.760 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_dbgp.ko 2023-10-30T12:14:32.760 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ether.ko 2023-10-30T12:14:32.760 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ffs.ko 2023-10-30T12:14:32.760 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_hid.ko 2023-10-30T12:14:32.761 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_mass_storage.ko 2023-10-30T12:14:32.761 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_ncm.ko 2023-10-30T12:14:32.761 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_nokia.ko 2023-10-30T12:14:32.761 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_printer.ko 2023-10-30T12:14:32.761 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_serial.ko 2023-10-30T12:14:32.761 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/g_zero.ko 2023-10-30T12:14:32.762 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 62185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/gadgetfs.ko 2023-10-30T12:14:32.762 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/legacy/tcm_usb_gadget.ko 2023-10-30T12:14:32.762 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 161225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/libcomposite.ko 2023-10-30T12:14:32.762 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/ 2023-10-30T12:14:32.762 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/amd5536udc_pci.ko 2023-10-30T12:14:32.763 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/bdc/ 2023-10-30T12:14:32.763 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 157681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/bdc/bdc.ko 2023-10-30T12:14:32.763 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/goku_udc.ko 2023-10-30T12:14:32.763 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 52721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/gr_udc.ko 2023-10-30T12:14:32.763 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 69705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/mv_u3d_core.ko 2023-10-30T12:14:32.764 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 56033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/mv_udc.ko 2023-10-30T12:14:32.764 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 76057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/net2272.ko 2023-10-30T12:14:32.764 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 75073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/net2280.ko 2023-10-30T12:14:32.764 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 73169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/pch_udc.ko 2023-10-30T12:14:32.764 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 65809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/pxa27x_udc.ko 2023-10-30T12:14:32.764 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/r8a66597-udc.ko 2023-10-30T12:14:32.765 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 81505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/snps_udc_core.ko 2023-10-30T12:14:32.765 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 156073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/gadget/udc/udc-core.ko 2023-10-30T12:14:32.765 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/ 2023-10-30T12:14:32.765 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/bcma-hcd.ko 2023-10-30T12:14:32.765 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 80113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/isp116x-hcd.ko 2023-10-30T12:14:32.766 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 87353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/oxu210hp-hcd.ko 2023-10-30T12:14:32.766 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/r8a66597-hcd.ko 2023-10-30T12:14:32.766 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-30T12:14:32.766 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/sl811_cs.ko 2023-10-30T12:14:32.766 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/ssb-hcd.ko 2023-10-30T12:14:32.767 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/host/xhci-plat-hcd.ko 2023-10-30T12:14:32.767 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/ 2023-10-30T12:14:32.767 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/mdc800.ko 2023-10-30T12:14:32.767 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/image/microtek.ko 2023-10-30T12:14:32.767 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ 2023-10-30T12:14:32.767 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/adutux.ko 2023-10-30T12:14:32.768 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/appledisplay.ko 2023-10-30T12:14:32.768 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/cypress_cy7c63.ko 2023-10-30T12:14:32.768 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/cytherm.ko 2023-10-30T12:14:32.768 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ehset.ko 2023-10-30T12:14:32.769 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/emi26.ko 2023-10-30T12:14:32.769 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/emi62.ko 2023-10-30T12:14:32.769 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ezusb.ko 2023-10-30T12:14:32.769 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/idmouse.ko 2023-10-30T12:14:32.769 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/iowarrior.ko 2023-10-30T12:14:32.769 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-30T12:14:32.770 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/ldusb.ko 2023-10-30T12:14:32.770 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/legousbtower.ko 2023-10-30T12:14:32.770 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/lvstest.ko 2023-10-30T12:14:32.770 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/sisusbvga/ 2023-10-30T12:14:32.771 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 64841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-30T12:14:32.771 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-30T12:14:32.771 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usb3503.ko 2023-10-30T12:14:32.771 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usblcd.ko 2023-10-30T12:14:32.771 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-30T12:14:32.771 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/usbtest.ko 2023-10-30T12:14:32.772 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/uss720.ko 2023-10-30T12:14:32.772 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/misc/yurex.ko 2023-10-30T12:14:32.772 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/mon/ 2023-10-30T12:14:32.772 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 72953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/mon/usbmon.ko 2023-10-30T12:14:32.772 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/musb/ 2023-10-30T12:14:32.772 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 301545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/musb/musb_hdrc.ko 2023-10-30T12:14:32.773 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/ 2023-10-30T12:14:32.773 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-generic.ko 2023-10-30T12:14:32.773 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-gpio-vbus-usb.ko 2023-10-30T12:14:32.773 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-isp1301.ko 2023-10-30T12:14:32.773 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/phy/phy-tahvo.ko 2023-10-30T12:14:32.774 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ 2023-10-30T12:14:32.774 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/aircable.ko 2023-10-30T12:14:32.774 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ark3116.ko 2023-10-30T12:14:32.774 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-30T12:14:32.774 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ch341.ko 2023-10-30T12:14:32.775 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cp210x.ko 2023-10-30T12:14:32.775 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cyberjack.ko 2023-10-30T12:14:32.775 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 58217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-30T12:14:32.775 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-30T12:14:32.775 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/empeg.ko 2023-10-30T12:14:32.776 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/f81232.ko 2023-10-30T12:14:32.776 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 175137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-30T12:14:32.776 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-30T12:14:32.776 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 137081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-30T12:14:32.776 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 124833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/io_ti.ko 2023-10-30T12:14:32.777 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ipaq.ko 2023-10-30T12:14:32.777 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ipw.ko 2023-10-30T12:14:32.777 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ir-usb.ko 2023-10-30T12:14:32.777 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 58233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-30T12:14:32.777 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 107521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/keyspan.ko 2023-10-30T12:14:32.778 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-30T12:14:32.778 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-30T12:14:32.778 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-30T12:14:32.778 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mct_u232.ko 2023-10-30T12:14:32.778 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/metro-usb.ko 2023-10-30T12:14:32.779 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mos7720.ko 2023-10-30T12:14:32.779 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 78217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mos7840.ko 2023-10-30T12:14:32.779 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 56337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/mxuport.ko 2023-10-30T12:14:32.779 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/navman.ko 2023-10-30T12:14:32.779 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/omninet.ko 2023-10-30T12:14:32.780 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/opticon.ko 2023-10-30T12:14:32.780 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 190633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/option.ko 2023-10-30T12:14:32.780 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/oti6858.ko 2023-10-30T12:14:32.780 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/pl2303.ko 2023-10-30T12:14:32.780 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/qcaux.ko 2023-10-30T12:14:32.781 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/qcserial.ko 2023-10-30T12:14:32.781 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/quatech2.ko 2023-10-30T12:14:32.781 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/safe_serial.ko 2023-10-30T12:14:32.781 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 51625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/sierra.ko 2023-10-30T12:14:32.781 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-30T12:14:32.782 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ssu100.ko 2023-10-30T12:14:32.782 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/symbolserial.ko 2023-10-30T12:14:32.782 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 61489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-30T12:14:32.782 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-30T12:14:32.782 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb_debug.ko 2023-10-30T12:14:32.783 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-30T12:14:32.783 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 125609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/usbserial.ko 2023-10-30T12:14:32.783 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/visor.ko 2023-10-30T12:14:32.783 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/whiteheat.ko 2023-10-30T12:14:32.783 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/wishbone-serial.ko 2023-10-30T12:14:32.784 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-30T12:14:32.784 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ 2023-10-30T12:14:32.784 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 52913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/uas.ko 2023-10-30T12:14:32.784 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-30T12:14:32.784 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-30T12:14:32.785 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-30T12:14:32.785 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-30T12:14:32.785 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-30T12:14:32.785 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-30T12:14:32.785 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-30T12:14:32.785 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-karma.ko 2023-10-30T12:14:32.786 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-30T12:14:32.786 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-30T12:14:32.786 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-30T12:14:32.786 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-30T12:14:32.786 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-30T12:14:32.787 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 179969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/storage/usb-storage.ko 2023-10-30T12:14:32.787 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/ 2023-10-30T12:14:32.787 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-30T12:14:32.787 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 72177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-30T12:14:32.787 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 107001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-30T12:14:32.788 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/ 2023-10-30T12:14:32.788 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/ 2023-10-30T12:14:32.788 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 154297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-30T12:14:32.788 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-30T12:14:32.788 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 108737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/vfio.ko 2023-10-30T12:14:32.789 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 77609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-30T12:14:32.789 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/ 2023-10-30T12:14:32.789 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 113873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost.ko 2023-10-30T12:14:32.789 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-30T12:14:32.789 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 58089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_net.ko 2023-10-30T12:14:32.789 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/vhost/vhost_scsi.ko 2023-10-30T12:14:32.790 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/ 2023-10-30T12:14:32.790 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/ 2023-10-30T12:14:32.790 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/88pm860x_bl.ko 2023-10-30T12:14:32.790 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/aat2870_bl.ko 2023-10-30T12:14:32.790 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp5520_bl.ko 2023-10-30T12:14:32.791 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp8860_bl.ko 2023-10-30T12:14:32.791 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/adp8870_bl.ko 2023-10-30T12:14:32.791 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/apple_bl.ko 2023-10-30T12:14:32.791 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/as3711_bl.ko 2023-10-30T12:14:32.791 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/bd6107.ko 2023-10-30T12:14:32.792 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/da903x_bl.ko 2023-10-30T12:14:32.792 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/da9052_bl.ko 2023-10-30T12:14:32.792 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/gpio_backlight.ko 2023-10-30T12:14:32.792 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/kb3886_bl.ko 2023-10-30T12:14:32.792 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lcd.ko 2023-10-30T12:14:32.793 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lm3533_bl.ko 2023-10-30T12:14:32.793 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lm3639_bl.ko 2023-10-30T12:14:32.793 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/lv5207lp.ko 2023-10-30T12:14:32.793 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/max8925_bl.ko 2023-10-30T12:14:32.794 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/pandora_bl.ko 2023-10-30T12:14:32.794 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/pcf50633-backlight.ko 2023-10-30T12:14:32.794 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-30T12:14:32.794 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/video/backlight/wm831x_bl.ko 2023-10-30T12:14:32.794 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/ 2023-10-30T12:14:32.795 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ 2023-10-30T12:14:32.795 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ds2482.ko 2023-10-30T12:14:32.795 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/ds2490.ko 2023-10-30T12:14:32.795 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/matrox_w1.ko 2023-10-30T12:14:32.795 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/masters/w1-gpio.ko 2023-10-30T12:14:32.796 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/ 2023-10-30T12:14:32.796 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-30T12:14:32.796 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-30T12:14:32.796 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-30T12:14:32.796 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-30T12:14:32.796 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-30T12:14:32.797 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-30T12:14:32.797 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-30T12:14:32.797 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-30T12:14:32.797 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-30T12:14:32.797 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-30T12:14:32.798 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 56425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-30T12:14:32.798 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 102673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/w1/wire.ko 2023-10-30T12:14:32.798 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ 2023-10-30T12:14:32.798 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/acquirewdt.ko 2023-10-30T12:14:32.799 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/advantechwdt.ko 2023-10-30T12:14:32.799 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-30T12:14:32.799 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-30T12:14:32.799 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/cpu5wdt.ko 2023-10-30T12:14:32.799 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9052_wdt.ko 2023-10-30T12:14:32.800 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9055_wdt.ko 2023-10-30T12:14:32.800 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/da9063_wdt.ko 2023-10-30T12:14:32.800 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/dw_wdt.ko 2023-10-30T12:14:32.800 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/eurotechwdt.ko 2023-10-30T12:14:32.800 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-30T12:14:32.801 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/hpwdt.ko 2023-10-30T12:14:32.801 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/i6300esb.ko 2023-10-30T12:14:32.801 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-30T12:14:32.801 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-30T12:14:32.801 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ib700wdt.ko 2023-10-30T12:14:32.802 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ibmasr.ko 2023-10-30T12:14:32.802 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-30T12:14:32.802 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-30T12:14:32.802 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/it87_wdt.ko 2023-10-30T12:14:32.802 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/kempld_wdt.ko 2023-10-30T12:14:32.803 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/machzwd.ko 2023-10-30T12:14:32.803 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/mena21_wdt.ko 2023-10-30T12:14:32.803 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/menf21bmc_wdt.ko 2023-10-30T12:14:32.803 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/nv_tco.ko 2023-10-30T12:14:32.803 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/of_xilinx_wdt.ko 2023-10-30T12:14:32.804 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22265 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pc87413_wdt.ko 2023-10-30T12:14:32.804 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-30T12:14:32.804 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-30T12:14:32.804 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/retu_wdt.ko 2023-10-30T12:14:32.804 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc60xxwdt.ko 2023-10-30T12:14:32.804 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc_epx_c3.ko 2023-10-30T12:14:32.805 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-30T12:14:32.805 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sc1200wdt.ko 2023-10-30T12:14:32.805 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-30T12:14:32.805 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/smsc37b787_wdt.ko 2023-10-30T12:14:32.805 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/softdog.ko 2023-10-30T12:14:32.806 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-30T12:14:32.806 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/twl4030_wdt.ko 2023-10-30T12:14:32.806 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/via_wdt.ko 2023-10-30T12:14:32.806 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-30T12:14:32.806 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-30T12:14:32.807 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-30T12:14:32.807 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wafer5823wdt.ko 2023-10-30T12:14:32.807 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wdt_pci.ko 2023-10-30T12:14:32.807 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wm831x_wdt.ko 2023-10-30T12:14:32.807 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/wm8350_wdt.ko 2023-10-30T12:14:32.808 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/watchdog/xen_wdt.ko 2023-10-30T12:14:32.808 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/ 2023-10-30T12:14:32.808 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-evtchn.ko 2023-10-30T12:14:32.808 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-gntalloc.ko 2023-10-30T12:14:32.808 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-gntdev.ko 2023-10-30T12:14:32.809 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-pciback/ 2023-10-30T12:14:32.809 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 176441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-30T12:14:32.809 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-privcmd.ko 2023-10-30T12:14:32.809 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 71249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xen-scsiback.ko 2023-10-30T12:14:32.809 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xenfs/ 2023-10-30T12:14:32.809 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-30T12:14:32.810 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ 2023-10-30T12:14:32.810 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/9p/ 2023-10-30T12:14:32.810 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 166425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/9p/9p.ko 2023-10-30T12:14:32.810 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/adfs/ 2023-10-30T12:14:32.810 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 61073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/adfs/adfs.ko 2023-10-30T12:14:32.811 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/affs/ 2023-10-30T12:14:32.811 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 163593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/affs/affs.ko 2023-10-30T12:14:32.811 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/afs/ 2023-10-30T12:14:32.811 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 932817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/afs/kafs.ko 2023-10-30T12:14:32.811 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/autofs/ 2023-10-30T12:14:32.812 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 101057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/autofs/autofs4.ko 2023-10-30T12:14:32.812 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/befs/ 2023-10-30T12:14:32.812 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 64545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/befs/befs.ko 2023-10-30T12:14:32.812 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/bfs/ 2023-10-30T12:14:32.812 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/bfs/bfs.ko 2023-10-30T12:14:32.813 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/binfmt_misc.ko 2023-10-30T12:14:32.813 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/btrfs/ 2023-10-30T12:14:32.813 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 3402585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/btrfs/btrfs.ko 2023-10-30T12:14:32.813 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cachefiles/ 2023-10-30T12:14:32.813 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 306073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cachefiles/cachefiles.ko 2023-10-30T12:14:32.814 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ceph/ 2023-10-30T12:14:32.814 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1323945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ceph/ceph.ko 2023-10-30T12:14:32.814 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/coda/ 2023-10-30T12:14:32.814 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 100801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/coda/coda.ko 2023-10-30T12:14:32.814 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/configfs/ 2023-10-30T12:14:32.815 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 108393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/configfs/configfs.ko 2023-10-30T12:14:32.815 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cramfs/ 2023-10-30T12:14:32.815 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 29625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/cramfs/cramfs.ko 2023-10-30T12:14:32.815 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/dlm/ 2023-10-30T12:14:32.815 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 642577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/dlm/dlm.ko 2023-10-30T12:14:32.815 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/efs/ 2023-10-30T12:14:32.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/efs/efs.ko 2023-10-30T12:14:32.816 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/f2fs/ 2023-10-30T12:14:32.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1505025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/f2fs/f2fs.ko 2023-10-30T12:14:32.816 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fat/ 2023-10-30T12:14:32.816 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fat/msdos.ko 2023-10-30T12:14:32.817 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/freevxfs/ 2023-10-30T12:14:32.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/freevxfs/freevxfs.ko 2023-10-30T12:14:32.817 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fscache/ 2023-10-30T12:14:32.817 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 226481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fscache/fscache.ko 2023-10-30T12:14:32.817 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fuse/ 2023-10-30T12:14:32.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/fuse/cuse.ko 2023-10-30T12:14:32.818 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/gfs2/ 2023-10-30T12:14:32.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1231529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/gfs2/gfs2.ko 2023-10-30T12:14:32.818 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfs/ 2023-10-30T12:14:32.818 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 141209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfs/hfs.ko 2023-10-30T12:14:32.819 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfsplus/ 2023-10-30T12:14:32.819 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 222817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hfsplus/hfsplus.ko 2023-10-30T12:14:32.819 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hpfs/ 2023-10-30T12:14:32.819 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 192329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/hpfs/hpfs.ko 2023-10-30T12:14:32.819 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/isofs/ 2023-10-30T12:14:32.819 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 74065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/isofs/isofs.ko 2023-10-30T12:14:32.820 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/jfs/ 2023-10-30T12:14:32.820 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 419497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/jfs/jfs.ko 2023-10-30T12:14:32.820 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/lockd/ 2023-10-30T12:14:32.820 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 280313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/lockd/lockd.ko 2023-10-30T12:14:32.820 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/minix/ 2023-10-30T12:14:32.821 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/minix/minix.ko 2023-10-30T12:14:32.821 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/netfs/ 2023-10-30T12:14:32.821 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 124649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/netfs/netfs.ko 2023-10-30T12:14:32.821 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/ 2023-10-30T12:14:32.821 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/blocklayout/ 2023-10-30T12:14:32.822 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 81449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-30T12:14:32.822 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/filelayout/ 2023-10-30T12:14:32.822 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 59449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-30T12:14:32.822 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/flexfilelayout/ 2023-10-30T12:14:32.822 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 117513 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-30T12:14:32.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1055649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfs.ko 2023-10-30T12:14:32.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv2.ko 2023-10-30T12:14:32.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 120177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv3.ko 2023-10-30T12:14:32.823 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 2041689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs/nfsv4.ko 2023-10-30T12:14:32.823 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/ 2023-10-30T12:14:32.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/grace.ko 2023-10-30T12:14:32.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfs_common/nfs_acl.ko 2023-10-30T12:14:32.824 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfsd/ 2023-10-30T12:14:32.824 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1516809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nfsd/nfsd.ko 2023-10-30T12:14:32.824 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nilfs2/ 2023-10-30T12:14:32.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 592833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nilfs2/nilfs2.ko 2023-10-30T12:14:32.825 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/ 2023-10-30T12:14:32.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-celtic.ko 2023-10-30T12:14:32.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-centeuro.ko 2023-10-30T12:14:32.825 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-croatian.ko 2023-10-30T12:14:32.826 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-cyrillic.ko 2023-10-30T12:14:32.826 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-gaelic.ko 2023-10-30T12:14:32.826 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-greek.ko 2023-10-30T12:14:32.826 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-iceland.ko 2023-10-30T12:14:32.826 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-inuit.ko 2023-10-30T12:14:32.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-roman.ko 2023-10-30T12:14:32.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-romanian.ko 2023-10-30T12:14:32.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/mac-turkish.ko 2023-10-30T12:14:32.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_ascii.ko 2023-10-30T12:14:32.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1250.ko 2023-10-30T12:14:32.827 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1251.ko 2023-10-30T12:14:32.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp1255.ko 2023-10-30T12:14:32.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp737.ko 2023-10-30T12:14:32.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp775.ko 2023-10-30T12:14:32.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp850.ko 2023-10-30T12:14:32.828 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp852.ko 2023-10-30T12:14:32.829 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp855.ko 2023-10-30T12:14:32.829 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp857.ko 2023-10-30T12:14:32.829 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp860.ko 2023-10-30T12:14:32.829 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp861.ko 2023-10-30T12:14:32.829 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp862.ko 2023-10-30T12:14:32.830 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp863.ko 2023-10-30T12:14:32.830 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp864.ko 2023-10-30T12:14:32.830 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp865.ko 2023-10-30T12:14:32.830 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp866.ko 2023-10-30T12:14:32.831 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp869.ko 2023-10-30T12:14:32.831 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp874.ko 2023-10-30T12:14:32.831 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 94137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp932.ko 2023-10-30T12:14:32.831 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 142673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp936.ko 2023-10-30T12:14:32.831 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 165745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp949.ko 2023-10-30T12:14:32.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 118121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_cp950.ko 2023-10-30T12:14:32.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_euc-jp.ko 2023-10-30T12:14:32.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-1.ko 2023-10-30T12:14:32.832 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-13.ko 2023-10-30T12:14:32.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-14.ko 2023-10-30T12:14:32.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-15.ko 2023-10-30T12:14:32.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-2.ko 2023-10-30T12:14:32.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-3.ko 2023-10-30T12:14:32.833 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-4.ko 2023-10-30T12:14:32.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-5.ko 2023-10-30T12:14:32.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-6.ko 2023-10-30T12:14:32.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-7.ko 2023-10-30T12:14:32.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_iso8859-9.ko 2023-10-30T12:14:32.834 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-r.ko 2023-10-30T12:14:32.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-ru.ko 2023-10-30T12:14:32.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_koi8-u.ko 2023-10-30T12:14:32.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-30T12:14:32.835 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/nls/nls_utf8.ko 2023-10-30T12:14:32.835 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ntfs/ 2023-10-30T12:14:32.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 219417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ntfs/ntfs.ko 2023-10-30T12:14:32.836 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ 2023-10-30T12:14:32.836 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/cluster/ 2023-10-30T12:14:32.836 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 244409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-30T12:14:32.836 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlm/ 2023-10-30T12:14:32.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 440737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-30T12:14:32.837 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlmfs/ 2023-10-30T12:14:32.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-30T12:14:32.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 2592433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2.ko 2023-10-30T12:14:32.837 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-30T12:14:32.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-30T12:14:32.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-30T12:14:32.838 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/omfs/ 2023-10-30T12:14:32.838 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/omfs/omfs.ko 2023-10-30T12:14:32.839 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/overlayfs/ 2023-10-30T12:14:32.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 352153 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/overlayfs/overlay.ko 2023-10-30T12:14:32.839 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/pstore/ 2023-10-30T12:14:32.839 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/pstore/ramoops.ko 2023-10-30T12:14:32.839 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx4/ 2023-10-30T12:14:32.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx4/qnx4.ko 2023-10-30T12:14:32.840 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx6/ 2023-10-30T12:14:32.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 46689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/qnx6/qnx6.ko 2023-10-30T12:14:32.840 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/ 2023-10-30T12:14:32.840 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_tree.ko 2023-10-30T12:14:32.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_v1.ko 2023-10-30T12:14:32.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/quota/quota_v2.ko 2023-10-30T12:14:32.841 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/reiserfs/ 2023-10-30T12:14:32.841 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 532673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/reiserfs/reiserfs.ko 2023-10-30T12:14:32.841 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/romfs/ 2023-10-30T12:14:32.842 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/romfs/romfs.ko 2023-10-30T12:14:32.842 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/ 2023-10-30T12:14:32.842 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/client/ 2023-10-30T12:14:32.842 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 2668121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/client/cifs.ko 2023-10-30T12:14:32.842 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/ 2023-10-30T12:14:32.842 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/cifs_arc4.ko 2023-10-30T12:14:32.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/smb/common/cifs_md4.ko 2023-10-30T12:14:32.843 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/squashfs/ 2023-10-30T12:14:32.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 134641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/squashfs/squashfs.ko 2023-10-30T12:14:32.843 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/sysv/ 2023-10-30T12:14:32.843 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 90721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/sysv/sysv.ko 2023-10-30T12:14:32.844 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/udf/ 2023-10-30T12:14:32.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 247737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/udf/udf.ko 2023-10-30T12:14:32.844 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ufs/ 2023-10-30T12:14:32.844 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 162497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/ufs/ufs.ko 2023-10-30T12:14:32.844 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/xfs/ 2023-10-30T12:14:32.845 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 4010729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/fs/xfs/xfs.ko 2023-10-30T12:14:32.845 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/ 2023-10-30T12:14:32.845 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/locking/ 2023-10-30T12:14:32.845 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 75489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/locking/locktorture.ko 2023-10-30T12:14:32.845 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/time/ 2023-10-30T12:14:32.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/time/test_udelay.ko 2023-10-30T12:14:32.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/kernel/torture.ko 2023-10-30T12:14:32.846 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ 2023-10-30T12:14:32.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 6833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc-itu-t.ko 2023-10-30T12:14:32.846 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 6529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc7.ko 2023-10-30T12:14:32.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crc8.ko 2023-10-30T12:14:32.847 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/ 2023-10-30T12:14:32.847 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/libarc4.ko 2023-10-30T12:14:32.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/crypto/libdes.ko 2023-10-30T12:14:32.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/interval_tree_test.ko 2023-10-30T12:14:32.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/libcrc32c.ko 2023-10-30T12:14:32.848 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lru_cache.ko 2023-10-30T12:14:32.848 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/ 2023-10-30T12:14:32.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/lz4_compress.ko 2023-10-30T12:14:32.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/lz4/lz4hc_compress.ko 2023-10-30T12:14:32.849 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/math/ 2023-10-30T12:14:32.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 6617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/math/cordic.ko 2023-10-30T12:14:32.849 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/memory-notifier-error-inject.ko 2023-10-30T12:14:32.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/notifier-error-inject.ko 2023-10-30T12:14:32.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/percpu_test.ko 2023-10-30T12:14:32.850 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/pm-notifier-error-inject.ko 2023-10-30T12:14:32.850 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/raid6/ 2023-10-30T12:14:32.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 143433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/raid6/raid6_pq.ko 2023-10-30T12:14:32.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/rbtree_test.ko 2023-10-30T12:14:32.851 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/reed_solomon/ 2023-10-30T12:14:32.851 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-30T12:14:32.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test-kstrtox.ko 2023-10-30T12:14:32.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test-string_helpers.ko 2023-10-30T12:14:32.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 4760425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_bpf.ko 2023-10-30T12:14:32.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_firmware.ko 2023-10-30T12:14:32.852 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 6369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_module.ko 2023-10-30T12:14:32.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/test_user_copy.ko 2023-10-30T12:14:32.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_bm.ko 2023-10-30T12:14:32.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_fsm.ko 2023-10-30T12:14:32.853 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/ts_kmp.ko 2023-10-30T12:14:32.853 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/xz/ 2023-10-30T12:14:32.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/lib/xz/xz_dec_test.ko 2023-10-30T12:14:32.854 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/mm/ 2023-10-30T12:14:32.854 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/mm/hwpoison-inject.ko 2023-10-30T12:14:32.854 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ 2023-10-30T12:14:32.855 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/ 2023-10-30T12:14:32.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 55737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/6lowpan.ko 2023-10-30T12:14:32.855 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_dest.ko 2023-10-30T12:14:32.856 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_fragment.ko 2023-10-30T12:14:32.856 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_hop.ko 2023-10-30T12:14:32.856 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 6985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-30T12:14:32.856 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_mobility.ko 2023-10-30T12:14:32.857 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_routing.ko 2023-10-30T12:14:32.857 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/6lowpan/nhc_udp.ko 2023-10-30T12:14:32.857 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/ 2023-10-30T12:14:32.857 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/garp.ko 2023-10-30T12:14:32.857 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/mrp.ko 2023-10-30T12:14:32.858 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/p8022.ko 2023-10-30T12:14:32.858 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/psnap.ko 2023-10-30T12:14:32.858 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/802/stp.ko 2023-10-30T12:14:32.858 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/8021q/ 2023-10-30T12:14:32.858 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 81737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/8021q/8021q.ko 2023-10-30T12:14:32.859 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/ 2023-10-30T12:14:32.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 198337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet.ko 2023-10-30T12:14:32.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_fd.ko 2023-10-30T12:14:32.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26345 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_rdma.ko 2023-10-30T12:14:32.859 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/9p/9pnet_virtio.ko 2023-10-30T12:14:32.860 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/batman-adv/ 2023-10-30T12:14:32.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 649297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/batman-adv/batman-adv.ko 2023-10-30T12:14:32.860 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/ 2023-10-30T12:14:32.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 63809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/br_netfilter.ko 2023-10-30T12:14:32.860 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 777689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/bridge.ko 2023-10-30T12:14:32.861 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ 2023-10-30T12:14:32.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-30T12:14:32.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-30T12:14:32.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-30T12:14:32.861 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-30T12:14:32.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-30T12:14:32.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-30T12:14:32.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-30T12:14:32.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-30T12:14:32.862 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13217 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-30T12:14:32.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-30T12:14:32.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-30T12:14:32.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8385 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-30T12:14:32.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-30T12:14:32.863 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-30T12:14:32.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-30T12:14:32.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-30T12:14:32.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-30T12:14:32.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-30T12:14:32.864 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-30T12:14:32.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-30T12:14:32.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 55929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/ebtables.ko 2023-10-30T12:14:32.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-30T12:14:32.865 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-30T12:14:32.866 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ceph/ 2023-10-30T12:14:32.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1047497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ceph/libceph.ko 2023-10-30T12:14:32.866 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/core/ 2023-10-30T12:14:32.866 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 125697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/core/pktgen.ko 2023-10-30T12:14:32.866 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/ 2023-10-30T12:14:32.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 194209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp.ko 2023-10-30T12:14:32.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_diag.ko 2023-10-30T12:14:32.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 57881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_ipv4.ko 2023-10-30T12:14:32.867 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dccp/dccp_ipv6.ko 2023-10-30T12:14:32.867 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/ 2023-10-30T12:14:32.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 390473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/dsa_core.ko 2023-10-30T12:14:32.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_brcm.ko 2023-10-30T12:14:32.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_dsa.ko 2023-10-30T12:14:32.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_none.ko 2023-10-30T12:14:32.868 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/dsa/tag_trailer.ko 2023-10-30T12:14:32.869 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/hsr/ 2023-10-30T12:14:32.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 99729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/hsr/hsr.ko 2023-10-30T12:14:32.869 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ 2023-10-30T12:14:32.869 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/6lowpan/ 2023-10-30T12:14:32.869 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 45233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-30T12:14:32.870 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 254625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ieee802154.ko 2023-10-30T12:14:32.870 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-30T12:14:32.870 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ 2023-10-30T12:14:32.870 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ah4.ko 2023-10-30T12:14:32.870 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/esp4.ko 2023-10-30T12:14:32.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 48401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/fou.ko 2023-10-30T12:14:32.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/gre.ko 2023-10-30T12:14:32.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 48657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/inet_diag.ko 2023-10-30T12:14:32.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_gre.ko 2023-10-30T12:14:32.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 61225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_tunnel.ko 2023-10-30T12:14:32.871 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ip_vti.ko 2023-10-30T12:14:32.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ipcomp.ko 2023-10-30T12:14:32.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/ipip.ko 2023-10-30T12:14:32.872 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ 2023-10-30T12:14:32.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-30T12:14:32.872 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-30T12:14:32.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-30T12:14:32.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ip_tables.ko 2023-10-30T12:14:32.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-30T12:14:32.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_REJECT.ko 2023-10-30T12:14:32.873 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-30T12:14:32.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-30T12:14:32.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-30T12:14:32.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11305 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_filter.ko 2023-10-30T12:14:32.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-30T12:14:32.874 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-30T12:14:32.875 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-30T12:14:32.875 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-30T12:14:32.875 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-30T12:14:32.875 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-30T12:14:32.875 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-30T12:14:32.876 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18225 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-30T12:14:32.876 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-30T12:14:32.876 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko 2023-10-30T12:14:32.876 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-30T12:14:32.876 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-30T12:14:32.877 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-30T12:14:32.877 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-30T12:14:32.877 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-30T12:14:32.877 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_bic.ko 2023-10-30T12:14:32.877 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_dctcp.ko 2023-10-30T12:14:32.877 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_diag.ko 2023-10-30T12:14:32.878 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_highspeed.ko 2023-10-30T12:14:32.878 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_htcp.ko 2023-10-30T12:14:32.878 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_hybla.ko 2023-10-30T12:14:32.878 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_illinois.ko 2023-10-30T12:14:32.878 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_lp.ko 2023-10-30T12:14:32.879 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_scalable.ko 2023-10-30T12:14:32.879 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_vegas.ko 2023-10-30T12:14:32.879 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_veno.ko 2023-10-30T12:14:32.881 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_westwood.ko 2023-10-30T12:14:32.881 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tcp_yeah.ko 2023-10-30T12:14:32.881 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/tunnel4.ko 2023-10-30T12:14:32.882 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/udp_diag.ko 2023-10-30T12:14:32.882 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 41665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/udp_tunnel.ko 2023-10-30T12:14:32.882 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-30T12:14:32.882 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ 2023-10-30T12:14:32.882 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ah6.ko 2023-10-30T12:14:32.883 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/esp6.ko 2023-10-30T12:14:32.883 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/fou6.ko 2023-10-30T12:14:32.883 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 80897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_gre.ko 2023-10-30T12:14:32.883 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 93577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_tunnel.ko 2023-10-30T12:14:32.883 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-30T12:14:32.884 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ip6_vti.ko 2023-10-30T12:14:32.884 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/ipcomp6.ko 2023-10-30T12:14:32.884 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19017 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/mip6.ko 2023-10-30T12:14:32.884 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ 2023-10-30T12:14:32.884 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-30T12:14:32.885 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_NPT.ko 2023-10-30T12:14:32.885 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-30T12:14:32.885 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-30T12:14:32.885 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-30T12:14:32.885 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-30T12:14:32.885 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-30T12:14:32.886 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-30T12:14:32.886 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-30T12:14:32.886 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-30T12:14:32.886 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-30T12:14:32.886 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-30T12:14:32.887 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-30T12:14:32.887 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11121 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-30T12:14:32.887 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-30T12:14:32.887 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-30T12:14:32.887 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-30T12:14:32.888 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-30T12:14:32.888 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-30T12:14:32.888 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25945 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-30T12:14:32.888 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-30T12:14:32.888 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-30T12:14:32.889 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-30T12:14:32.889 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-30T12:14:32.889 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9009 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-30T12:14:32.889 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 74073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/sit.ko 2023-10-30T12:14:32.889 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/tunnel6.ko 2023-10-30T12:14:32.889 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-30T12:14:32.890 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/key/ 2023-10-30T12:14:32.890 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 82833 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/key/af_key.ko 2023-10-30T12:14:32.890 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/ 2023-10-30T12:14:32.890 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 125281 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_core.ko 2023-10-30T12:14:32.890 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-30T12:14:32.891 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 25953 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_eth.ko 2023-10-30T12:14:32.891 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ip.ko 2023-10-30T12:14:32.891 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36713 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ip6.ko 2023-10-30T12:14:32.891 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_netlink.ko 2023-10-30T12:14:32.891 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 53241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/l2tp/l2tp_ppp.ko 2023-10-30T12:14:32.892 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/lapb/ 2023-10-30T12:14:32.892 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 47977 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/lapb/lapb.ko 2023-10-30T12:14:32.892 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/ 2023-10-30T12:14:32.892 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/llc.ko 2023-10-30T12:14:32.892 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 267777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/llc/llc2.ko 2023-10-30T12:14:32.893 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mac802154/ 2023-10-30T12:14:32.893 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 290417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mac802154/mac802154.ko 2023-10-30T12:14:32.893 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mpls/ 2023-10-30T12:14:32.893 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/mpls/mpls_gso.ko 2023-10-30T12:14:32.893 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ 2023-10-30T12:14:32.893 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ 2023-10-30T12:14:32.894 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 119585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set.ko 2023-10-30T12:14:32.894 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-30T12:14:32.894 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26913 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-30T12:14:32.894 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-30T12:14:32.894 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 100297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-30T12:14:32.895 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 101441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-30T12:14:32.895 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 102817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-30T12:14:32.895 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 103481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-30T12:14:32.895 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 112849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-30T12:14:32.895 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 60329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-30T12:14:32.896 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 109593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-30T12:14:32.896 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 113129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-30T12:14:32.896 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 112585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-30T12:14:32.896 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 111585 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-30T12:14:32.896 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 114465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-30T12:14:32.897 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-30T12:14:32.897 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ 2023-10-30T12:14:32.897 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 411729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-30T12:14:32.897 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-30T12:14:32.897 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-30T12:14:32.898 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-30T12:14:32.898 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-30T12:14:32.898 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-30T12:14:32.898 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-30T12:14:32.898 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7689 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-30T12:14:32.899 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10681 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-30T12:14:32.899 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-30T12:14:32.899 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-30T12:14:32.899 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-30T12:14:32.899 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-30T12:14:32.900 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10993 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-30T12:14:32.900 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conncount.ko 2023-10-30T12:14:32.900 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 408257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack.ko 2023-10-30T12:14:32.900 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-30T12:14:32.900 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-30T12:14:32.901 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33761 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-30T12:14:32.901 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 118145 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-30T12:14:32.901 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-30T12:14:32.901 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-30T12:14:32.901 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 107297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-30T12:14:32.901 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-30T12:14:32.902 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-30T12:14:32.902 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-30T12:14:32.902 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-30T12:14:32.902 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-30T12:14:32.902 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10113 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-30T12:14:32.903 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_flow_table.ko 2023-10-30T12:14:32.903 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9593 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-30T12:14:32.903 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 37129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_log_syslog.ko 2023-10-30T12:14:32.903 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 95417 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat.ko 2023-10-30T12:14:32.903 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-30T12:14:32.904 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-30T12:14:32.904 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11857 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_irc.ko 2023-10-30T12:14:32.904 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28929 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_sip.ko 2023-10-30T12:14:32.904 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-30T12:14:32.904 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 44065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-30T12:14:32.904 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 615313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nf_tables.ko 2023-10-30T12:14:32.905 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 42561 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink.ko 2023-10-30T12:14:32.905 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-30T12:14:32.905 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30569 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_cttimeout.ko 2023-10-30T12:14:32.905 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 39785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_log.ko 2023-10-30T12:14:32.905 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-30T12:14:32.906 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-30T12:14:32.906 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10457 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_chain_nat.ko 2023-10-30T12:14:32.906 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_compat.ko 2023-10-30T12:14:32.906 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 40721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_ct.ko 2023-10-30T12:14:32.906 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-30T12:14:32.907 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_fib.ko 2023-10-30T12:14:32.907 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-30T12:14:32.907 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14185 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_hash.ko 2023-10-30T12:14:32.907 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_limit.ko 2023-10-30T12:14:32.907 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_log.ko 2023-10-30T12:14:32.907 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_masq.ko 2023-10-30T12:14:32.908 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14329 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_nat.ko 2023-10-30T12:14:32.908 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13673 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_numgen.ko 2023-10-30T12:14:32.908 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11841 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_osf.ko 2023-10-30T12:14:32.908 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_queue.ko 2023-10-30T12:14:32.908 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_redir.ko 2023-10-30T12:14:32.909 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_reject.ko 2023-10-30T12:14:32.909 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_reject_inet.ko 2023-10-30T12:14:32.909 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_socket.ko 2023-10-30T12:14:32.909 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20369 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_synproxy.ko 2023-10-30T12:14:32.909 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/nft_tproxy.ko 2023-10-30T12:14:32.909 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 89313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/x_tables.ko 2023-10-30T12:14:32.910 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_AUDIT.ko 2023-10-30T12:14:32.910 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-30T12:14:32.910 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-30T12:14:32.910 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-30T12:14:32.910 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 18169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_CT.ko 2023-10-30T12:14:32.911 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_DSCP.ko 2023-10-30T12:14:32.911 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_HL.ko 2023-10-30T12:14:32.911 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_HMARK.ko 2023-10-30T12:14:32.911 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-30T12:14:32.911 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13889 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_LED.ko 2023-10-30T12:14:32.911 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_LOG.ko 2023-10-30T12:14:32.912 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11753 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-30T12:14:32.912 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NETMAP.ko 2023-10-30T12:14:32.912 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NFLOG.ko 2023-10-30T12:14:32.912 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-30T12:14:32.912 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_RATEEST.ko 2023-10-30T12:14:32.912 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-30T12:14:32.913 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 14337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_SECMARK.ko 2023-10-30T12:14:32.913 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-30T12:14:32.913 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-30T12:14:32.913 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17361 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TEE.ko 2023-10-30T12:14:32.913 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TPROXY.ko 2023-10-30T12:14:32.913 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_TRACE.ko 2023-10-30T12:14:32.914 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_addrtype.ko 2023-10-30T12:14:32.914 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_bpf.ko 2023-10-30T12:14:32.914 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cgroup.ko 2023-10-30T12:14:32.914 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11209 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cluster.ko 2023-10-30T12:14:32.914 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_comment.ko 2023-10-30T12:14:32.915 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10465 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connbytes.ko 2023-10-30T12:14:32.915 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11081 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connlabel.ko 2023-10-30T12:14:32.915 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connlimit.ko 2023-10-30T12:14:32.915 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_connmark.ko 2023-10-30T12:14:32.915 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_conntrack.ko 2023-10-30T12:14:32.916 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_cpu.ko 2023-10-30T12:14:32.916 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_dccp.ko 2023-10-30T12:14:32.916 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_devgroup.ko 2023-10-30T12:14:32.916 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8801 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_dscp.ko 2023-10-30T12:14:32.916 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ecn.ko 2023-10-30T12:14:32.917 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_esp.ko 2023-10-30T12:14:32.917 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_hashlimit.ko 2023-10-30T12:14:32.917 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_helper.ko 2023-10-30T12:14:32.917 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7865 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_hl.ko 2023-10-30T12:14:32.917 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11921 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ipcomp.ko 2023-10-30T12:14:32.918 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_iprange.ko 2023-10-30T12:14:32.918 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11785 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_ipvs.ko 2023-10-30T12:14:32.918 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16041 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_l2tp.ko 2023-10-30T12:14:32.918 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8001 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_length.ko 2023-10-30T12:14:32.918 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_limit.ko 2023-10-30T12:14:32.919 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7545 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_mac.ko 2023-10-30T12:14:32.919 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_mark.ko 2023-10-30T12:14:32.919 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11825 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_multiport.ko 2023-10-30T12:14:32.919 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13817 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_nat.ko 2023-10-30T12:14:32.919 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_nfacct.ko 2023-10-30T12:14:32.919 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_osf.ko 2023-10-30T12:14:32.920 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_owner.ko 2023-10-30T12:14:32.920 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_physdev.ko 2023-10-30T12:14:32.920 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_pkttype.ko 2023-10-30T12:14:32.920 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11985 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_policy.ko 2023-10-30T12:14:32.920 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_quota.ko 2023-10-30T12:14:32.921 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_rateest.ko 2023-10-30T12:14:32.921 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_realm.ko 2023-10-30T12:14:32.921 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_recent.ko 2023-10-30T12:14:32.921 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_sctp.ko 2023-10-30T12:14:32.921 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_set.ko 2023-10-30T12:14:32.922 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 16089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_socket.ko 2023-10-30T12:14:32.922 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9505 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_state.ko 2023-10-30T12:14:32.922 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9473 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_statistic.ko 2023-10-30T12:14:32.922 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8809 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_string.ko 2023-10-30T12:14:32.923 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 8553 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_tcpmss.ko 2023-10-30T12:14:32.923 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15777 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_tcpudp.ko 2023-10-30T12:14:32.923 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11129 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_time.ko 2023-10-30T12:14:32.923 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netfilter/xt_u32.ko 2023-10-30T12:14:32.923 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netlink/ 2023-10-30T12:14:32.923 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13969 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/netlink/netlink_diag.ko 2023-10-30T12:14:32.924 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/nsh/ 2023-10-30T12:14:32.924 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 12425 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/nsh/nsh.ko 2023-10-30T12:14:32.924 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/ 2023-10-30T12:14:32.924 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 380033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/openvswitch.ko 2023-10-30T12:14:32.925 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10321 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-geneve.ko 2023-10-30T12:14:32.925 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-gre.ko 2023-10-30T12:14:32.925 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 11793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/openvswitch/vport-vxlan.ko 2023-10-30T12:14:32.925 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/packet/ 2023-10-30T12:14:32.925 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13057 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/packet/af_packet_diag.ko 2023-10-30T12:14:32.925 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/ 2023-10-30T12:14:32.926 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 110201 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/phonet.ko 2023-10-30T12:14:32.926 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54697 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/phonet/pn_pep.ko 2023-10-30T12:14:32.926 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/ 2023-10-30T12:14:32.926 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 251481 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds.ko 2023-10-30T12:14:32.926 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 164937 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds_rdma.ko 2023-10-30T12:14:32.927 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 67377 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rds/rds_tcp.ko 2023-10-30T12:14:32.927 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rxrpc/ 2023-10-30T12:14:32.927 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 819657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/rxrpc/rxrpc.ko 2023-10-30T12:14:32.927 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/ 2023-10-30T12:14:32.928 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32241 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_csum.ko 2023-10-30T12:14:32.928 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23577 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_gact.ko 2023-10-30T12:14:32.928 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26873 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_ipt.ko 2023-10-30T12:14:32.928 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34097 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_mirred.ko 2023-10-30T12:14:32.928 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_nat.ko 2023-10-30T12:14:32.929 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 43449 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_pedit.ko 2023-10-30T12:14:32.929 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_police.ko 2023-10-30T12:14:32.929 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_simple.ko 2023-10-30T12:14:32.929 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35297 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_skbedit.ko 2023-10-30T12:14:32.929 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36665 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/act_vlan.ko 2023-10-30T12:14:32.929 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26169 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_basic.ko 2023-10-30T12:14:32.930 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33961 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_bpf.ko 2023-10-30T12:14:32.930 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19233 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_cgroup.ko 2023-10-30T12:14:32.930 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31401 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_flow.ko 2023-10-30T12:14:32.930 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31769 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_fw.ko 2023-10-30T12:14:32.930 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 38489 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_route.ko 2023-10-30T12:14:32.931 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 61409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/cls_u32.ko 2023-10-30T12:14:32.931 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7441 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_cmp.ko 2023-10-30T12:14:32.931 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 13409 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_ipset.ko 2023-10-30T12:14:32.931 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33289 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_meta.ko 2023-10-30T12:14:32.931 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_nbyte.ko 2023-10-30T12:14:32.932 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_text.ko 2023-10-30T12:14:32.932 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7249 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/em_u32.ko 2023-10-30T12:14:32.932 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 21649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_choke.ko 2023-10-30T12:14:32.932 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20849 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_codel.ko 2023-10-30T12:14:32.932 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 34633 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_drr.ko 2023-10-30T12:14:32.933 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 33609 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_fq.ko 2023-10-30T12:14:32.933 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31617 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_fq_codel.ko 2023-10-30T12:14:32.933 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 35433 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_gred.ko 2023-10-30T12:14:32.933 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 49601 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_hfsc.ko 2023-10-30T12:14:32.933 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22089 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_hhf.ko 2023-10-30T12:14:32.934 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 66393 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_htb.ko 2023-10-30T12:14:32.934 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26073 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_ingress.ko 2023-10-30T12:14:32.934 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 30273 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_mqprio.ko 2023-10-30T12:14:32.934 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_mqprio_lib.ko 2023-10-30T12:14:32.934 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26065 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_multiq.ko 2023-10-30T12:14:32.935 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 36137 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_netem.ko 2023-10-30T12:14:32.935 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 27497 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_pie.ko 2023-10-30T12:14:32.935 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10625 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_plug.ko 2023-10-30T12:14:32.935 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 24729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_prio.ko 2023-10-30T12:14:32.935 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 48793 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_qfq.ko 2023-10-30T12:14:32.935 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 32905 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_red.ko 2023-10-30T12:14:32.936 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 26881 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_sfb.ko 2023-10-30T12:14:32.936 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 31353 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_sfq.ko 2023-10-30T12:14:32.936 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28897 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_tbf.ko 2023-10-30T12:14:32.936 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 28105 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sched/sch_teql.ko 2023-10-30T12:14:32.936 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/ 2023-10-30T12:14:32.937 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 929745 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/sctp.ko 2023-10-30T12:14:32.937 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 19033 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sctp/sctp_diag.ko 2023-10-30T12:14:32.937 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/ 2023-10-30T12:14:32.937 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/ 2023-10-30T12:14:32.937 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 336529 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-30T12:14:32.938 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 54737 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-30T12:14:32.938 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 1520161 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/sunrpc.ko 2023-10-30T12:14:32.938 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/xprtrdma/ 2023-10-30T12:14:32.938 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 707721 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-30T12:14:32.938 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/ 2023-10-30T12:14:32.938 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10025 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/diag.ko 2023-10-30T12:14:32.939 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 862641 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/tipc/tipc.ko 2023-10-30T12:14:32.939 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/unix/ 2023-10-30T12:14:32.939 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15729 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/unix/unix_diag.ko 2023-10-30T12:14:32.939 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/ 2023-10-30T12:14:32.939 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 85337 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-30T12:14:32.940 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 68049 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-30T12:14:32.940 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 96193 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock.ko 2023-10-30T12:14:32.940 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 10705 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-30T12:14:32.940 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 17521 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-30T12:14:32.940 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/x25/ 2023-10-30T12:14:32.940 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 125177 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/x25/x25.ko 2023-10-30T12:14:32.941 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/ 2023-10-30T12:14:32.941 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 22657 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_algo.ko 2023-10-30T12:14:32.941 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 20649 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-30T12:14:32.941 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 102257 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/net/xfrm/xfrm_user.ko 2023-10-30T12:14:32.941 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/ 2023-10-30T12:14:32.942 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/lib/ 2023-10-30T12:14:32.942 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 15537 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/kernel/virt/lib/irqbypass.ko 2023-10-30T12:14:32.942 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 639863 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.alias 2023-10-30T12:14:32.942 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 623919 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.alias.bin 2023-10-30T12:14:32.942 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 7663 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin 2023-10-30T12:14:32.943 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 23428 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.alias.bin 2023-10-30T12:14:32.943 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 9821 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.bin 2023-10-30T12:14:32.943 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 70004 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.builtin.modinfo 2023-10-30T12:14:32.943 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 146313 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.dep 2023-10-30T12:14:32.943 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 232548 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.dep.bin 2023-10-30T12:14:32.943 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 210 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.devname 2023-10-30T12:14:32.944 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 72606 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.order 2023-10-30T12:14:32.944 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 879 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.softdep 2023-10-30T12:14:32.944 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 254438 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.symbols 2023-10-30T12:14:32.944 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 309363 2023-10-30 03:46 ./lib/modules/6.6.0-rc5-geacf00cc51e2/modules.symbols.bin 2023-10-30T12:14:32.944 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/ 2023-10-30T12:14:32.944 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/ 2023-10-30T12:14:32.945 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/doc/ 2023-10-30T12:14:32.945 INFO:teuthology.orchestra.run.smithi125.stdout:drwxr-xr-x root/root 0 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/ 2023-10-30T12:14:32.945 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 173 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/changelog.Debian.gz 2023-10-30T12:14:32.945 INFO:teuthology.orchestra.run.smithi125.stdout:-rw-r--r-- root/root 691 2023-10-30 03:46 ./usr/share/doc/linux-image-6.6.0-rc5-geacf00cc51e2/copyright 2023-10-30T12:14:32.946 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:32.946 INFO:teuthology.task.kernel:searching for kernel 6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:32.947 DEBUG:teuthology.orchestra.run.smithi125:> egrep '(submenu|menuentry.*6.6.0-rc5-geacf00cc51e2).*{' /boot/grub/grub.cfg 2023-10-30T12:14:32.954 INFO:teuthology.orchestra.run.smithi125.stdout:submenu 'Advanced options for Ubuntu' $menuentry_id_option 'gnulinux-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T12:14:32.954 INFO:teuthology.orchestra.run.smithi125.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-geacf00cc51e2-advanced-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T12:14:32.954 INFO:teuthology.orchestra.run.smithi125.stdout: menuentry 'Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2 (recovery mode)' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-6.6.0-rc5-geacf00cc51e2-recovery-7e9e2103-8bb7-4452-b35a-3b1230743254' { 2023-10-30T12:14:32.955 INFO:teuthology.task.kernel:submenu_title:Advanced options for Ubuntu> 2023-10-30T12:14:32.955 INFO:teuthology.task.kernel:default_title:Ubuntu, with Linux 6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:32.956 DEBUG:teuthology.orchestra.run.smithi125:> /bin/echo -e 'cat </dev/null && sudo chmod a+x -- '/etc/grub.d/01_ceph_kernel.tmp~' && sudo mv -- '/etc/grub.d/01_ceph_kernel.tmp~' /etc/grub.d/01_ceph_kernel && sudo update-grub && rm /tmp/linux-image.deb && ( sleep 1 && sudo shutdown -r now & ) 2023-10-30T12:14:32.999 DEBUG:teuthology.task.kernel:Waiting for install on ubuntu@smithi125.front.sepia.ceph.com to complete... 2023-10-30T12:14:33.366 INFO:teuthology.orchestra.run.smithi125.stderr:Sourcing file `/etc/default/grub' 2023-10-30T12:14:33.367 INFO:teuthology.orchestra.run.smithi125.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-10-30T12:14:33.388 INFO:teuthology.orchestra.run.smithi125.stderr:Generating grub configuration file ... 2023-10-30T12:14:33.597 INFO:teuthology.orchestra.run.smithi125.stderr:Found linux image: /boot/vmlinuz-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:33.603 INFO:teuthology.orchestra.run.smithi125.stderr:Found initrd image: /boot/initrd.img-6.6.0-rc5-geacf00cc51e2 2023-10-30T12:14:33.824 INFO:teuthology.orchestra.run.smithi125.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-10-30T12:14:33.829 INFO:teuthology.orchestra.run.smithi125.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-10-30T12:14:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-10-30T12:14:33.874 INFO:teuthology.orchestra.run.smithi125.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-10-30T12:14:34.018 INFO:teuthology.orchestra.run.smithi125.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-10-30T12:14:34.019 INFO:teuthology.orchestra.run.smithi125.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-10-30T12:14:34.019 INFO:teuthology.orchestra.run.smithi125.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-10-30T12:14:34.029 INFO:teuthology.orchestra.run.smithi125.stderr:done 2023-10-30T12:15:05.123 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-30T12:15:05.124 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-10-30T12:15:05.125 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:15:39.646 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-10-30T12:15:48.648 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-10-30T12:15:48.649 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:16:07.034 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.125 2023-10-30T12:16:19.036 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-10-30T12:16:19.037 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:16:19.546 DEBUG:teuthology.orchestra.run.smithi125:> true 2023-10-30T12:16:20.490 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-10-30T12:16:20.491 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-rc5-geacf00cc51e2"... 2023-10-30T12:16:20.491 DEBUG:teuthology.orchestra.run.smithi125:> uname -r 2023-10-30T12:16:20.545 INFO:teuthology.orchestra.run.smithi125.stdout:6.6.0-rc5-geacf00cc51e2 2023-10-30T12:16:20.546 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-geacf00cc51e2 vs 6.6.0-rc5-geacf00cc51e2 2023-10-30T12:16:20.546 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-30T12:16:20.546 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-10-30T12:16:21.547 DEBUG:teuthology.parallel:result is None 2023-10-30T12:16:21.548 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-30T12:16:21.556 INFO:teuthology.task.internal:Creating test directory... 2023-10-30T12:16:21.556 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T12:16:21.560 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-30T12:16:21.566 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-30T12:16:21.573 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-30T12:16:21.583 INFO:teuthology.task.internal:Creating archive directory... 2023-10-30T12:16:21.583 DEBUG:teuthology.orchestra.run.smithi125:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T12:16:21.613 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-30T12:16:21.809 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-30T12:16:21.816 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-30T12:16:21.816 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-30T12:16:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:16:21.822 DEBUG:teuthology.orchestra.run.smithi125:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T12:16:21.825 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-30T12:16:21.900 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T12:16:21.911 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T12:16:22.124 INFO:teuthology.orchestra.run.smithi125.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T12:16:22.165 INFO:teuthology.orchestra.run.smithi125.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-30T12:16:22.168 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-30T12:16:22.175 INFO:teuthology.task.internal:Configuring sudo... 2023-10-30T12:16:22.175 DEBUG:teuthology.orchestra.run.smithi125:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T12:16:22.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-30T12:16:22.197 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-30T12:16:22.206 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-30T12:16:22.207 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T12:16:22.245 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-30T12:16:22.251 DEBUG:teuthology.orchestra.run.smithi125:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T12:16:22.303 DEBUG:teuthology.orchestra.run.smithi125:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T12:16:22.358 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:16:22.358 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T12:16:22.435 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-30T12:16:22.440 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-30T12:16:22.489 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:16:22.489 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-30T12:16:22.561 DEBUG:teuthology.orchestra.run.smithi125:> sudo service rsyslog restart 2023-10-30T12:16:22.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2023-10-30T12:16:22.728 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-30T12:16:22.738 INFO:teuthology.task.internal:Starting timer... 2023-10-30T12:16:22.739 INFO:teuthology.run_tasks:Running task pcp... 2023-10-30T12:16:22.805 INFO:teuthology.run_tasks:Running task selinux... 2023-10-30T12:16:22.814 DEBUG:teuthology.task.selinux:Excluding smithi125: OS 'ubuntu' does not support SELinux 2023-10-30T12:16:22.814 DEBUG:teuthology.task.selinux:Excluding smithi136: OS 'ubuntu' does not support SELinux 2023-10-30T12:16:22.815 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-30T12:16:22.815 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-10-30T12:16:22.815 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-30T12:16:22.815 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-30T12:16:22.825 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-10-30T12:16:22.825 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-30T12:16:22.841 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-30T12:16:22.843 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi125.front.sepia.ceph.com,smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-30T12:24:56.064 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi125.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2023-10-30T12:24:56.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-10-30T12:24:56.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:24:56.144 DEBUG:teuthology.orchestra.run.smithi125:> true 2023-10-30T12:24:56.261 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi125.front.sepia.ceph.com' 2023-10-30T12:24:56.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-10-30T12:24:56.262 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-30T12:24:56.344 DEBUG:teuthology.orchestra.run.smithi136:> true 2023-10-30T12:24:56.454 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2023-10-30T12:24:56.455 INFO:teuthology.run_tasks:Running task clock... 2023-10-30T12:24:56.467 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-30T12:24:56.468 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T12:24:56.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T12:24:56.472 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-30T12:24:56.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T12:24:56.508 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-30T12:24:56.509 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: Command line: ntpd -gq 2023-10-30T12:24:56.509 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: ---------------------------------------------------- 2023-10-30T12:24:56.509 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: ntp-4 is maintained by Network Time Foundation, 2023-10-30T12:24:56.509 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-30T12:24:56.510 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: corporation. Support and training for ntp-4 are 2023-10-30T12:24:56.510 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: available at https://www.nwtime.org/support 2023-10-30T12:24:56.510 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: ---------------------------------------------------- 2023-10-30T12:24:56.511 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: proto: precision = 0.042 usec (-24) 2023-10-30T12:24:56.511 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: basedate set to 2022-02-04 2023-10-30T12:24:56.511 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: gps base set to 2022-02-06 (week 2196) 2023-10-30T12:24:56.512 INFO:teuthology.orchestra.run.smithi125.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T12:24:56.512 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T12:24:56.513 INFO:teuthology.orchestra.run.smithi125.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-30T12:24:56.513 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: restrict ::: KOD does nothing without LIMITED. 2023-10-30T12:24:56.514 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: Listen and drop on 0 v6wildcard [::]:123 2023-10-30T12:24:56.514 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-30T12:24:56.514 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: Listen normally on 2 lo 127.0.0.1:123 2023-10-30T12:24:56.515 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: Listen normally on 3 enp3s0f1 172.21.15.125:123 2023-10-30T12:24:56.515 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: Listen normally on 4 lo [::1]:123 2023-10-30T12:24:56.515 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d739%5]:123 2023-10-30T12:24:56.515 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:24:56 ntpd[27992]: Listening on routing socket on fd #22 for interface updates 2023-10-30T12:24:56.522 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-10-30T12:24:56.523 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: Command line: ntpd -gq 2023-10-30T12:24:56.523 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: ---------------------------------------------------- 2023-10-30T12:24:56.523 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: ntp-4 is maintained by Network Time Foundation, 2023-10-30T12:24:56.523 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-10-30T12:24:56.523 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: corporation. Support and training for ntp-4 are 2023-10-30T12:24:56.524 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: available at https://www.nwtime.org/support 2023-10-30T12:24:56.524 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: ---------------------------------------------------- 2023-10-30T12:24:56.525 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: proto: precision = 0.045 usec (-24) 2023-10-30T12:24:56.525 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: basedate set to 2022-02-04 2023-10-30T12:24:56.525 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: gps base set to 2022-02-06 (week 2196) 2023-10-30T12:24:56.525 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T12:24:56.526 INFO:teuthology.orchestra.run.smithi136.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-10-30T12:24:56.526 INFO:teuthology.orchestra.run.smithi136.stderr:restrict ::: KOD does nothing without LIMITED. 2023-10-30T12:24:56.527 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: restrict ::: KOD does nothing without LIMITED. 2023-10-30T12:24:56.527 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: Listen and drop on 0 v6wildcard [::]:123 2023-10-30T12:24:56.528 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-10-30T12:24:56.528 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: Listen normally on 2 lo 127.0.0.1:123 2023-10-30T12:24:56.528 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: Listen normally on 3 enp3s0f1 172.21.15.136:123 2023-10-30T12:24:56.528 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: Listen normally on 4 lo [::1]:123 2023-10-30T12:24:56.528 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e43%5]:123 2023-10-30T12:24:56.529 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:24:56 ntpd[28426]: Listening on routing socket on fd #22 for interface updates 2023-10-30T12:24:57.509 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:57 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:57.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:57 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:57.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:57 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:57.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:57 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:57.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:57 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:57.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:57 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:57.523 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:57 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:57.523 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:57 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:57.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:57 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:57.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:57 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:57.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:57 ntpd[28426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:57.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:57 ntpd[28426]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:58.509 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:58 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:58.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:58 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:58.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:58 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:58.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:58 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:58.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:58 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:58.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:58 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:58.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:58 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:58.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:58 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:58.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:58 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:58.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:58 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:58.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:58 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:58.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:58 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:58.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:58 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:58.526 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:58 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:58.526 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:58 ntpd[28426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:58.526 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:58 ntpd[28426]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:58.526 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:58 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:58.526 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:58 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:58.527 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:58 ntpd[28426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:58.527 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:58 ntpd[28426]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:59.509 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:59 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:59.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:59 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:59.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:59 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:59.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:59 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:59.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:59 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:59.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:59 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:59.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:59 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:59.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:24:59 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:24:59.523 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:59 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:59.523 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:59 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:59.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:59 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:24:59.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:59 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:24:59.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:59 ntpd[28426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:24:59.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:24:59 ntpd[28426]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:25:00.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:00.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:00 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:00.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:00.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:00 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:00.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:00.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:00 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:00.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:25:00.512 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:00 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:25:00.512 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:00.512 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:00 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:00.512 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:00 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:25:00.512 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:00 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:25:00.523 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:00 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:00.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:00 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:00.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:00 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:00.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:00 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:00.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:00 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:00.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:00 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:00.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:00 ntpd[28426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:25:00.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:00 ntpd[28426]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:25:00.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:00 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:00.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:00 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:00.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:00 ntpd[28426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:25:00.526 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:00 ntpd[28426]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:25:01.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:01 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:01.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:01 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:01.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:01 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:01.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:01 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:01.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:01 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:25:01.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:01 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:25:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:01 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:01.523 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:01 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:01 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:01 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:01 ntpd[28426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:25:01.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:01 ntpd[28426]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:25:02.509 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:02 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:02.509 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:02 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:02.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:02 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:02.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:02 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:02.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:02 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:02.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:02 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:02.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:02 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:25:02.512 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:02 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:25:02.512 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:02 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:02.512 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:02 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:02.512 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:02 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:25:02.512 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:02 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:25:02.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:02 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:02.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:02 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:02.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:02 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:02.524 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:02 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:02.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:02 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:02.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:02 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:02.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:02 ntpd[28426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:25:02.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:02 ntpd[28426]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:25:02.525 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:02 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:02.526 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:02 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:02.526 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:02 ntpd[28426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:25:02.526 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:02 ntpd[28426]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:25:03.509 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:03 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:03.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:03 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:03.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:03 ntpd[27992]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:03.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:03 ntpd[27992]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:03.510 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:03 ntpd[27992]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:25:03.511 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:03 ntpd[27992]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:25:03.511 INFO:teuthology.orchestra.run.smithi125.stdout:30 Oct 12:25:03 ntpd[27992]: ntpd: time slew -0.001990 s 2023-10-30T12:25:03.511 INFO:teuthology.orchestra.run.smithi125.stdout:ntpd: time slew -0.001990s 2023-10-30T12:25:03.512 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:03 ntpd[27992]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-30T12:25:03.512 INFO:teuthology.orchestra.run.smithi125.stderr:30 Oct 12:25:03 ntpd[27992]: can't open /var/log/ntpstats/loopstats.20231030: Permission denied 2023-10-30T12:25:03.526 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:03 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:03.526 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:03 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:03.526 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:03 ntpd[28426]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-10-30T12:25:03.526 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:03 ntpd[28426]: can't open /var/log/ntpstats/rawstats.20231030: Permission denied 2023-10-30T12:25:03.527 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:03 ntpd[28426]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-10-30T12:25:03.527 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:03 ntpd[28426]: can't open /var/log/ntpstats/peerstats.20231030: Permission denied 2023-10-30T12:25:03.527 INFO:teuthology.orchestra.run.smithi136.stdout:30 Oct 12:25:03 ntpd[28426]: ntpd: time slew -0.000304 s 2023-10-30T12:25:03.527 INFO:teuthology.orchestra.run.smithi136.stdout:ntpd: time slew -0.000304s 2023-10-30T12:25:03.528 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:03 ntpd[28426]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-10-30T12:25:03.528 INFO:teuthology.orchestra.run.smithi136.stderr:30 Oct 12:25:03 ntpd[28426]: can't open /var/log/ntpstats/loopstats.20231030: Permission denied 2023-10-30T12:25:03.606 INFO:teuthology.orchestra.run.smithi136.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T12:25:03.607 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================== 2023-10-30T12:25:03.607 INFO:teuthology.orchestra.run.smithi136.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:25:03.607 INFO:teuthology.orchestra.run.smithi136.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:25:03.607 INFO:teuthology.orchestra.run.smithi136.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:25:03.607 INFO:teuthology.orchestra.run.smithi136.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:25:03.608 INFO:teuthology.orchestra.run.smithi125.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T12:25:03.608 INFO:teuthology.orchestra.run.smithi125.stdout:============================================================================== 2023-10-30T12:25:03.608 INFO:teuthology.orchestra.run.smithi125.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:25:03.609 INFO:teuthology.orchestra.run.smithi125.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:25:03.609 INFO:teuthology.orchestra.run.smithi125.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:25:03.609 INFO:teuthology.orchestra.run.smithi125.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-10-30T12:25:03.611 INFO:teuthology.run_tasks:Running task install... 2023-10-30T12:25:03.619 DEBUG:teuthology.task.install:project ceph 2023-10-30T12:25:03.620 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}} 2023-10-30T12:25:03.620 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'} 2023-10-30T12:25:03.621 INFO:teuthology.task.install:Using flavor: default 2023-10-30T12:25:03.626 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-30T12:25:03.627 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-30T12:25:03.627 DEBUG:teuthology.orchestra.run.smithi125:> sudo apt-key list | grep Ceph 2023-10-30T12:25:03.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-key list | grep Ceph 2023-10-30T12:25:03.708 INFO:teuthology.orchestra.run.smithi136.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-30T12:25:03.736 INFO:teuthology.orchestra.run.smithi136.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-30T12:25:03.737 INFO:teuthology.orchestra.run.smithi136.stdout:uid [ unknown] Ceph.com (release key) 2023-10-30T12:25:03.737 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-30T12:25:03.738 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-30T12:25:03.738 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:25:03.762 INFO:teuthology.orchestra.run.smithi125.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-10-30T12:25:03.815 INFO:teuthology.orchestra.run.smithi125.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-10-30T12:25:03.815 INFO:teuthology.orchestra.run.smithi125.stdout:uid [ unknown] Ceph.com (release key) 2023-10-30T12:25:03.816 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2023-10-30T12:25:03.816 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2023-10-30T12:25:03.817 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:25:03.912 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default/ 2023-10-30T12:25:03.912 INFO:teuthology.task.install.deb:Package version is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:25:03.964 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default/ 2023-10-30T12:25:03.965 INFO:teuthology.task.install.deb:Package version is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:25:04.244 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:25:04.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-30T12:25:04.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2023-10-30T12:25:04.355 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:25:04.356 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-10-30T12:25:04.377 DEBUG:teuthology.orchestra.run.smithi125:> sudo apt-get update 2023-10-30T12:25:04.432 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-30T12:25:04.432 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-30T12:25:04.447 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-30T12:25:04.466 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-30T12:25:04.567 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-10-30T12:25:04.567 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-10-30T12:25:04.580 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-10-30T12:25:04.600 INFO:teuthology.orchestra.run.smithi125.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-10-30T12:25:04.801 INFO:teuthology.orchestra.run.smithi125.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy InRelease 2023-10-30T12:25:04.818 INFO:teuthology.orchestra.run.smithi136.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy InRelease 2023-10-30T12:25:04.853 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-30T12:25:04.866 INFO:teuthology.orchestra.run.smithi125.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-10-30T12:25:04.881 INFO:teuthology.orchestra.run.smithi136.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-30T12:25:04.907 INFO:teuthology.orchestra.run.smithi125.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy Release.gpg 2023-10-30T12:25:04.944 INFO:teuthology.orchestra.run.smithi125.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-30T12:25:04.992 INFO:teuthology.orchestra.run.smithi125.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-30T12:25:04.994 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2023-10-30T12:25:05.026 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.5 kB] 2023-10-30T12:25:05.121 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 28.5 kB in 1s (43.1 kB/s) 2023-10-30T12:25:05.170 INFO:teuthology.orchestra.run.smithi125.stdout:Fetched 28.5 kB in 1s (50.2 kB/s) 2023-10-30T12:25:06.019 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2023-10-30T12:25:06.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6835-g00c1c13c-1jammy cephadm=18.0.0-6835-g00c1c13c-1jammy ceph-mds=18.0.0-6835-g00c1c13c-1jammy ceph-mgr=18.0.0-6835-g00c1c13c-1jammy ceph-common=18.0.0-6835-g00c1c13c-1jammy ceph-fuse=18.0.0-6835-g00c1c13c-1jammy ceph-test=18.0.0-6835-g00c1c13c-1jammy ceph-volume=18.0.0-6835-g00c1c13c-1jammy radosgw=18.0.0-6835-g00c1c13c-1jammy python3-rados=18.0.0-6835-g00c1c13c-1jammy python3-rgw=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy python3-rbd=18.0.0-6835-g00c1c13c-1jammy libcephfs2=18.0.0-6835-g00c1c13c-1jammy libcephfs-dev=18.0.0-6835-g00c1c13c-1jammy librados2=18.0.0-6835-g00c1c13c-1jammy librbd1=18.0.0-6835-g00c1c13c-1jammy rbd-fuse=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy cephfs-shell=18.0.0-6835-g00c1c13c-1jammy cephfs-top=18.0.0-6835-g00c1c13c-1jammy cephfs-mirror=18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:25:06.089 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2023-10-30T12:25:06.096 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2023-10-30T12:25:06.103 DEBUG:teuthology.orchestra.run.smithi125:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.0.0-6835-g00c1c13c-1jammy cephadm=18.0.0-6835-g00c1c13c-1jammy ceph-mds=18.0.0-6835-g00c1c13c-1jammy ceph-mgr=18.0.0-6835-g00c1c13c-1jammy ceph-common=18.0.0-6835-g00c1c13c-1jammy ceph-fuse=18.0.0-6835-g00c1c13c-1jammy ceph-test=18.0.0-6835-g00c1c13c-1jammy ceph-volume=18.0.0-6835-g00c1c13c-1jammy radosgw=18.0.0-6835-g00c1c13c-1jammy python3-rados=18.0.0-6835-g00c1c13c-1jammy python3-rgw=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy python3-rbd=18.0.0-6835-g00c1c13c-1jammy libcephfs2=18.0.0-6835-g00c1c13c-1jammy libcephfs-dev=18.0.0-6835-g00c1c13c-1jammy librados2=18.0.0-6835-g00c1c13c-1jammy librbd1=18.0.0-6835-g00c1c13c-1jammy rbd-fuse=18.0.0-6835-g00c1c13c-1jammy python3-cephfs=18.0.0-6835-g00c1c13c-1jammy cephfs-shell=18.0.0-6835-g00c1c13c-1jammy cephfs-top=18.0.0-6835-g00c1c13c-1jammy cephfs-mirror=18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:25:06.179 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2023-10-30T12:25:06.281 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2023-10-30T12:25:06.282 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2023-10-30T12:25:06.350 INFO:teuthology.orchestra.run.smithi125.stdout:Building dependency tree... 2023-10-30T12:25:06.350 INFO:teuthology.orchestra.run.smithi125.stdout:Reading state information... 2023-10-30T12:25:06.448 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T12:25:06.449 INFO:teuthology.orchestra.run.smithi136.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T12:25:06.449 INFO:teuthology.orchestra.run.smithi136.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T12:25:06.450 INFO:teuthology.orchestra.run.smithi136.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T12:25:06.450 INFO:teuthology.orchestra.run.smithi136.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-30T12:25:06.450 INFO:teuthology.orchestra.run.smithi136.stdout: python2.7-minimal 2023-10-30T12:25:06.450 INFO:teuthology.orchestra.run.smithi136.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T12:25:06.451 INFO:teuthology.orchestra.run.smithi136.stdout:The following additional packages will be installed: 2023-10-30T12:25:06.451 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-30T12:25:06.451 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-30T12:25:06.451 INFO:teuthology.orchestra.run.smithi136.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T12:25:06.451 INFO:teuthology.orchestra.run.smithi136.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-30T12:25:06.452 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-30T12:25:06.453 INFO:teuthology.orchestra.run.smithi136.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-30T12:25:06.453 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-30T12:25:06.453 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-30T12:25:06.453 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-30T12:25:06.453 INFO:teuthology.orchestra.run.smithi136.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-30T12:25:06.454 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-30T12:25:06.454 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-30T12:25:06.454 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-30T12:25:06.454 INFO:teuthology.orchestra.run.smithi136.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-30T12:25:06.454 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-30T12:25:06.455 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-30T12:25:06.455 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-30T12:25:06.455 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-30T12:25:06.455 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-30T12:25:06.455 INFO:teuthology.orchestra.run.smithi136.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-30T12:25:06.456 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-30T12:25:06.456 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-30T12:25:06.456 INFO:teuthology.orchestra.run.smithi136.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-30T12:25:06.456 INFO:teuthology.orchestra.run.smithi136.stdout:Suggested packages: 2023-10-30T12:25:06.456 INFO:teuthology.orchestra.run.smithi136.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-30T12:25:06.456 INFO:teuthology.orchestra.run.smithi136.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-30T12:25:06.457 INFO:teuthology.orchestra.run.smithi136.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-30T12:25:06.457 INFO:teuthology.orchestra.run.smithi136.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-30T12:25:06.457 INFO:teuthology.orchestra.run.smithi136.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-30T12:25:06.457 INFO:teuthology.orchestra.run.smithi136.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-30T12:25:06.457 INFO:teuthology.orchestra.run.smithi136.stdout: python3-watchdog 2023-10-30T12:25:06.458 INFO:teuthology.orchestra.run.smithi136.stdout:Recommended packages: 2023-10-30T12:25:06.458 INFO:teuthology.orchestra.run.smithi136.stdout: btrfs-tools 2023-10-30T12:25:06.505 INFO:teuthology.orchestra.run.smithi125.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T12:25:06.506 INFO:teuthology.orchestra.run.smithi125.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T12:25:06.506 INFO:teuthology.orchestra.run.smithi125.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T12:25:06.506 INFO:teuthology.orchestra.run.smithi125.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T12:25:06.508 INFO:teuthology.orchestra.run.smithi125.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-30T12:25:06.508 INFO:teuthology.orchestra.run.smithi125.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-30T12:25:06.508 INFO:teuthology.orchestra.run.smithi125.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T12:25:06.509 INFO:teuthology.orchestra.run.smithi125.stdout:The following additional packages will be installed: 2023-10-30T12:25:06.509 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-10-30T12:25:06.509 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-10-30T12:25:06.509 INFO:teuthology.orchestra.run.smithi125.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T12:25:06.509 INFO:teuthology.orchestra.run.smithi125.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-10-30T12:25:06.510 INFO:teuthology.orchestra.run.smithi125.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2023-10-30T12:25:06.510 INFO:teuthology.orchestra.run.smithi125.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-10-30T12:25:06.510 INFO:teuthology.orchestra.run.smithi125.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-10-30T12:25:06.510 INFO:teuthology.orchestra.run.smithi125.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-10-30T12:25:06.510 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-google-auth 2023-10-30T12:25:06.511 INFO:teuthology.orchestra.run.smithi125.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2023-10-30T12:25:06.511 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2023-10-30T12:25:06.511 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2023-10-30T12:25:06.511 INFO:teuthology.orchestra.run.smithi125.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2023-10-30T12:25:06.511 INFO:teuthology.orchestra.run.smithi125.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2023-10-30T12:25:06.512 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2023-10-30T12:25:06.512 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2023-10-30T12:25:06.512 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyperclip python3-pytest python3-repoze.lru python3-requests 2023-10-30T12:25:06.512 INFO:teuthology.orchestra.run.smithi125.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-simplegeneric 2023-10-30T12:25:06.512 INFO:teuthology.orchestra.run.smithi125.stdout: python3-simplejson python3-singledispatch python3-sklearn 2023-10-30T12:25:06.513 INFO:teuthology.orchestra.run.smithi125.stdout: python3-sklearn-lib python3-tempita python3-tempora python3-threadpoolctl 2023-10-30T12:25:06.513 INFO:teuthology.orchestra.run.smithi125.stdout: python3-toml python3-urllib3 python3-waitress python3-wcwidth python3-webob 2023-10-30T12:25:06.513 INFO:teuthology.orchestra.run.smithi125.stdout: python3-websocket python3-webtest python3-werkzeug python3-zc.lockfile 2023-10-30T12:25:06.513 INFO:teuthology.orchestra.run.smithi125.stdout: qttranslations5-l10n socat xmlstarlet 2023-10-30T12:25:06.514 INFO:teuthology.orchestra.run.smithi125.stdout:Suggested packages: 2023-10-30T12:25:06.514 INFO:teuthology.orchestra.run.smithi125.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2023-10-30T12:25:06.514 INFO:teuthology.orchestra.run.smithi125.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2023-10-30T12:25:06.514 INFO:teuthology.orchestra.run.smithi125.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2023-10-30T12:25:06.514 INFO:teuthology.orchestra.run.smithi125.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2023-10-30T12:25:06.515 INFO:teuthology.orchestra.run.smithi125.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2023-10-30T12:25:06.515 INFO:teuthology.orchestra.run.smithi125.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2023-10-30T12:25:06.515 INFO:teuthology.orchestra.run.smithi125.stdout: python3-watchdog 2023-10-30T12:25:06.515 INFO:teuthology.orchestra.run.smithi125.stdout:Recommended packages: 2023-10-30T12:25:06.515 INFO:teuthology.orchestra.run.smithi125.stdout: btrfs-tools 2023-10-30T12:25:06.559 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2023-10-30T12:25:06.560 INFO:teuthology.orchestra.run.smithi136.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-30T12:25:06.560 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-30T12:25:06.560 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-30T12:25:06.560 INFO:teuthology.orchestra.run.smithi136.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-30T12:25:06.561 INFO:teuthology.orchestra.run.smithi136.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T12:25:06.561 INFO:teuthology.orchestra.run.smithi136.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-30T12:25:06.561 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-30T12:25:06.562 INFO:teuthology.orchestra.run.smithi136.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-30T12:25:06.562 INFO:teuthology.orchestra.run.smithi136.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-30T12:25:06.562 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-30T12:25:06.563 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-30T12:25:06.563 INFO:teuthology.orchestra.run.smithi136.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-30T12:25:06.563 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-30T12:25:06.563 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-30T12:25:06.563 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-30T12:25:06.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-30T12:25:06.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-30T12:25:06.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-30T12:25:06.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-30T12:25:06.564 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-30T12:25:06.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-30T12:25:06.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-30T12:25:06.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-30T12:25:06.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-30T12:25:06.565 INFO:teuthology.orchestra.run.smithi136.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-30T12:25:06.566 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-30T12:25:06.566 INFO:teuthology.orchestra.run.smithi136.stdout: socat xmlstarlet 2023-10-30T12:25:06.651 INFO:teuthology.orchestra.run.smithi125.stdout:The following NEW packages will be installed: 2023-10-30T12:25:06.652 INFO:teuthology.orchestra.run.smithi125.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-10-30T12:25:06.652 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-10-30T12:25:06.652 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2023-10-30T12:25:06.652 INFO:teuthology.orchestra.run.smithi125.stdout: cephfs-mirror cephfs-shell cephfs-top jq libcephfs-dev libcephfs2 2023-10-30T12:25:06.652 INFO:teuthology.orchestra.run.smithi125.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2023-10-30T12:25:06.653 INFO:teuthology.orchestra.run.smithi125.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2023-10-30T12:25:06.653 INFO:teuthology.orchestra.run.smithi125.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2023-10-30T12:25:06.654 INFO:teuthology.orchestra.run.smithi125.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2023-10-30T12:25:06.654 INFO:teuthology.orchestra.run.smithi125.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2023-10-30T12:25:06.654 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2023-10-30T12:25:06.655 INFO:teuthology.orchestra.run.smithi125.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-cmd2 2023-10-30T12:25:06.655 INFO:teuthology.orchestra.run.smithi125.stdout: python3-colorama python3-google-auth python3-idna python3-iniconfig 2023-10-30T12:25:06.655 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-10-30T12:25:06.655 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-10-30T12:25:06.655 INFO:teuthology.orchestra.run.smithi125.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-10-30T12:25:06.656 INFO:teuthology.orchestra.run.smithi125.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-10-30T12:25:06.656 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-10-30T12:25:06.656 INFO:teuthology.orchestra.run.smithi125.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-10-30T12:25:06.656 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pygments python3-pyinotify python3-pyperclip python3-pytest 2023-10-30T12:25:06.656 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests 2023-10-30T12:25:06.657 INFO:teuthology.orchestra.run.smithi125.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2023-10-30T12:25:06.657 INFO:teuthology.orchestra.run.smithi125.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-10-30T12:25:06.657 INFO:teuthology.orchestra.run.smithi125.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-10-30T12:25:06.657 INFO:teuthology.orchestra.run.smithi125.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-10-30T12:25:06.657 INFO:teuthology.orchestra.run.smithi125.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-10-30T12:25:06.658 INFO:teuthology.orchestra.run.smithi125.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2023-10-30T12:25:06.658 INFO:teuthology.orchestra.run.smithi125.stdout: socat xmlstarlet 2023-10-30T12:25:06.723 INFO:teuthology.orchestra.run.smithi125.stdout:0 upgraded, 115 newly installed, 0 to remove and 260 not upgraded. 2023-10-30T12:25:06.724 INFO:teuthology.orchestra.run.smithi125.stdout:Need to get 184 MB of archives. 2023-10-30T12:25:06.724 INFO:teuthology.orchestra.run.smithi125.stdout:After this operation, 826 MB of additional disk space will be used. 2023-10-30T12:25:06.724 INFO:teuthology.orchestra.run.smithi125.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-30T12:25:06.744 INFO:teuthology.orchestra.run.smithi125.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-30T12:25:06.746 INFO:teuthology.orchestra.run.smithi136.stdout:0 upgraded, 115 newly installed, 0 to remove and 260 not upgraded. 2023-10-30T12:25:06.746 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 184 MB of archives. 2023-10-30T12:25:06.747 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 826 MB of additional disk space will be used. 2023-10-30T12:25:06.747 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyperclip all 1.8.2-2 [11.5 kB] 2023-10-30T12:25:06.754 INFO:teuthology.orchestra.run.smithi125.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-30T12:25:06.779 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6835-g00c1c13c-1jammy [3,450 kB] 2023-10-30T12:25:06.802 INFO:teuthology.orchestra.run.smithi125.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-30T12:25:06.811 INFO:teuthology.orchestra.run.smithi125.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-30T12:25:06.829 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-10-30T12:25:06.830 INFO:teuthology.orchestra.run.smithi125.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-30T12:25:06.833 INFO:teuthology.orchestra.run.smithi125.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-30T12:25:06.844 INFO:teuthology.orchestra.run.smithi125.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-30T12:25:06.874 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cmd2 all 2.3.3-1 [124 kB] 2023-10-30T12:25:06.879 INFO:teuthology.orchestra.run.smithi125.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.0.0-6835-g00c1c13c-1jammy [3,450 kB] 2023-10-30T12:25:06.892 INFO:teuthology.orchestra.run.smithi125.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-30T12:25:06.894 INFO:teuthology.orchestra.run.smithi125.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-30T12:25:06.906 INFO:teuthology.orchestra.run.smithi125.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-30T12:25:06.911 INFO:teuthology.orchestra.run.smithi125.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-30T12:25:06.912 INFO:teuthology.orchestra.run.smithi125.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-30T12:25:06.919 INFO:teuthology.orchestra.run.smithi125.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-30T12:25:06.919 INFO:teuthology.orchestra.run.smithi125.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-30T12:25:06.920 INFO:teuthology.orchestra.run.smithi125.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-30T12:25:06.921 INFO:teuthology.orchestra.run.smithi125.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-30T12:25:06.923 INFO:teuthology.orchestra.run.smithi125.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-30T12:25:06.941 INFO:teuthology.orchestra.run.smithi125.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-30T12:25:06.942 INFO:teuthology.orchestra.run.smithi125.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-30T12:25:06.942 INFO:teuthology.orchestra.run.smithi125.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-30T12:25:06.942 INFO:teuthology.orchestra.run.smithi125.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-30T12:25:06.943 INFO:teuthology.orchestra.run.smithi125.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-30T12:25:06.945 INFO:teuthology.orchestra.run.smithi125.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-30T12:25:06.946 INFO:teuthology.orchestra.run.smithi125.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-30T12:25:06.946 INFO:teuthology.orchestra.run.smithi125.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-30T12:25:06.947 INFO:teuthology.orchestra.run.smithi125.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-30T12:25:06.964 INFO:teuthology.orchestra.run.smithi125.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-30T12:25:06.965 INFO:teuthology.orchestra.run.smithi125.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-30T12:25:06.965 INFO:teuthology.orchestra.run.smithi125.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-30T12:25:06.966 INFO:teuthology.orchestra.run.smithi125.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-30T12:25:06.983 INFO:teuthology.orchestra.run.smithi125.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-30T12:25:06.984 INFO:teuthology.orchestra.run.smithi125.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-30T12:25:06.989 INFO:teuthology.orchestra.run.smithi125.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-30T12:25:06.989 INFO:teuthology.orchestra.run.smithi125.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-30T12:25:06.989 INFO:teuthology.orchestra.run.smithi125.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-30T12:25:06.990 INFO:teuthology.orchestra.run.smithi125.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-30T12:25:07.003 INFO:teuthology.orchestra.run.smithi125.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-30T12:25:07.005 INFO:teuthology.orchestra.run.smithi125.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-30T12:25:07.006 INFO:teuthology.orchestra.run.smithi125.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-30T12:25:07.007 INFO:teuthology.orchestra.run.smithi125.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-30T12:25:07.023 INFO:teuthology.orchestra.run.smithi125.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-30T12:25:07.025 INFO:teuthology.orchestra.run.smithi125.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-30T12:25:07.026 INFO:teuthology.orchestra.run.smithi125.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-30T12:25:07.028 INFO:teuthology.orchestra.run.smithi125.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-30T12:25:07.069 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2023-10-30T12:25:07.097 INFO:teuthology.orchestra.run.smithi125.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-30T12:25:07.098 INFO:teuthology.orchestra.run.smithi125.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-30T12:25:07.099 INFO:teuthology.orchestra.run.smithi125.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-30T12:25:07.100 INFO:teuthology.orchestra.run.smithi125.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-30T12:25:07.100 INFO:teuthology.orchestra.run.smithi125.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-30T12:25:07.101 INFO:teuthology.orchestra.run.smithi125.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-30T12:25:07.112 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-10-30T12:25:07.129 INFO:teuthology.orchestra.run.smithi125.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-30T12:25:07.131 INFO:teuthology.orchestra.run.smithi125.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-30T12:25:07.132 INFO:teuthology.orchestra.run.smithi125.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-30T12:25:07.162 INFO:teuthology.orchestra.run.smithi125.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-30T12:25:07.162 INFO:teuthology.orchestra.run.smithi125.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-30T12:25:07.163 INFO:teuthology.orchestra.run.smithi125.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-30T12:25:07.163 INFO:teuthology.orchestra.run.smithi125.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-30T12:25:07.164 INFO:teuthology.orchestra.run.smithi125.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-30T12:25:07.164 INFO:teuthology.orchestra.run.smithi125.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-30T12:25:07.184 INFO:teuthology.orchestra.run.smithi125.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-30T12:25:07.184 INFO:teuthology.orchestra.run.smithi125.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-30T12:25:07.186 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-10-30T12:25:07.188 INFO:teuthology.orchestra.run.smithi125.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-30T12:25:07.198 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-10-30T12:25:07.204 INFO:teuthology.orchestra.run.smithi125.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-30T12:25:07.206 INFO:teuthology.orchestra.run.smithi125.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-30T12:25:07.207 INFO:teuthology.orchestra.run.smithi125.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-30T12:25:07.211 INFO:teuthology.orchestra.run.smithi125.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-30T12:25:07.214 INFO:teuthology.orchestra.run.smithi125.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-30T12:25:07.215 INFO:teuthology.orchestra.run.smithi125.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-30T12:25:07.220 INFO:teuthology.orchestra.run.smithi125.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-30T12:25:07.231 INFO:teuthology.orchestra.run.smithi125.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-30T12:25:07.232 INFO:teuthology.orchestra.run.smithi125.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-30T12:25:07.232 INFO:teuthology.orchestra.run.smithi125.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-30T12:25:07.243 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-10-30T12:25:07.251 INFO:teuthology.orchestra.run.smithi125.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-30T12:25:07.253 INFO:teuthology.orchestra.run.smithi125.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-30T12:25:07.254 INFO:teuthology.orchestra.run.smithi125.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-30T12:25:07.262 INFO:teuthology.orchestra.run.smithi125.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-30T12:25:07.263 INFO:teuthology.orchestra.run.smithi125.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-30T12:25:07.263 INFO:teuthology.orchestra.run.smithi125.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-30T12:25:07.265 INFO:teuthology.orchestra.run.smithi125.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-30T12:25:07.276 INFO:teuthology.orchestra.run.smithi125.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-30T12:25:07.381 INFO:teuthology.orchestra.run.smithi125.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6835-g00c1c13c-1jammy [3,131 kB] 2023-10-30T12:25:07.402 INFO:teuthology.orchestra.run.smithi136.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.0.0-6835-g00c1c13c-1jammy [3,131 kB] 2023-10-30T12:25:07.409 INFO:teuthology.orchestra.run.smithi136.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-10-30T12:25:07.418 INFO:teuthology.orchestra.run.smithi136.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-10-30T12:25:07.445 INFO:teuthology.orchestra.run.smithi136.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-10-30T12:25:07.602 INFO:teuthology.orchestra.run.smithi125.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6835-g00c1c13c-1jammy [689 kB] 2023-10-30T12:25:07.617 INFO:teuthology.orchestra.run.smithi136.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-10-30T12:25:07.656 INFO:teuthology.orchestra.run.smithi125.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6835-g00c1c13c-1jammy [356 kB] 2023-10-30T12:25:07.696 INFO:teuthology.orchestra.run.smithi125.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6835-g00c1c13c-1jammy [32.8 kB] 2023-10-30T12:25:07.697 INFO:teuthology.orchestra.run.smithi125.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6835-g00c1c13c-1jammy [180 kB] 2023-10-30T12:25:07.705 INFO:teuthology.orchestra.run.smithi125.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6835-g00c1c13c-1jammy [62.4 kB] 2023-10-30T12:25:07.707 INFO:teuthology.orchestra.run.smithi125.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6835-g00c1c13c-1jammy [330 kB] 2023-10-30T12:25:07.721 INFO:teuthology.orchestra.run.smithi125.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6835-g00c1c13c-1jammy [6,226 kB] 2023-10-30T12:25:07.741 INFO:teuthology.orchestra.run.smithi136.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.0.0-6835-g00c1c13c-1jammy [689 kB] 2023-10-30T12:25:07.808 INFO:teuthology.orchestra.run.smithi136.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.0.0-6835-g00c1c13c-1jammy [356 kB] 2023-10-30T12:25:07.812 INFO:teuthology.orchestra.run.smithi136.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-10-30T12:25:07.862 INFO:teuthology.orchestra.run.smithi136.stdout:Get:18 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.0.0-6835-g00c1c13c-1jammy [32.8 kB] 2023-10-30T12:25:07.863 INFO:teuthology.orchestra.run.smithi136.stdout:Get:19 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.0.0-6835-g00c1c13c-1jammy [180 kB] 2023-10-30T12:25:07.870 INFO:teuthology.orchestra.run.smithi136.stdout:Get:20 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.0.0-6835-g00c1c13c-1jammy [62.4 kB] 2023-10-30T12:25:07.873 INFO:teuthology.orchestra.run.smithi136.stdout:Get:21 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.0.0-6835-g00c1c13c-1jammy [330 kB] 2023-10-30T12:25:07.936 INFO:teuthology.orchestra.run.smithi136.stdout:Get:22 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.0.0-6835-g00c1c13c-1jammy [6,226 kB] 2023-10-30T12:25:08.181 INFO:teuthology.orchestra.run.smithi136.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-10-30T12:25:08.187 INFO:teuthology.orchestra.run.smithi136.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-10-30T12:25:08.189 INFO:teuthology.orchestra.run.smithi136.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-10-30T12:25:08.201 INFO:teuthology.orchestra.run.smithi136.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-10-30T12:25:08.202 INFO:teuthology.orchestra.run.smithi136.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-10-30T12:25:08.203 INFO:teuthology.orchestra.run.smithi136.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-10-30T12:25:08.205 INFO:teuthology.orchestra.run.smithi136.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-10-30T12:25:08.208 INFO:teuthology.orchestra.run.smithi136.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-10-30T12:25:08.208 INFO:teuthology.orchestra.run.smithi136.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-10-30T12:25:08.215 INFO:teuthology.orchestra.run.smithi125.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6835-g00c1c13c-1jammy [112 kB] 2023-10-30T12:25:08.217 INFO:teuthology.orchestra.run.smithi125.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6835-g00c1c13c-1jammy [439 kB] 2023-10-30T12:25:08.234 INFO:teuthology.orchestra.run.smithi125.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6835-g00c1c13c-1jammy [22.6 MB] 2023-10-30T12:25:08.374 INFO:teuthology.orchestra.run.smithi136.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-10-30T12:25:08.641 INFO:teuthology.orchestra.run.smithi136.stdout:Get:33 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.0.0-6835-g00c1c13c-1jammy [112 kB] 2023-10-30T12:25:08.643 INFO:teuthology.orchestra.run.smithi136.stdout:Get:34 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.0.0-6835-g00c1c13c-1jammy [439 kB] 2023-10-30T12:25:08.708 INFO:teuthology.orchestra.run.smithi136.stdout:Get:35 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.0.0-6835-g00c1c13c-1jammy [22.6 MB] 2023-10-30T12:25:08.805 INFO:teuthology.orchestra.run.smithi136.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-10-30T12:25:08.815 INFO:teuthology.orchestra.run.smithi136.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-10-30T12:25:08.820 INFO:teuthology.orchestra.run.smithi136.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-10-30T12:25:08.824 INFO:teuthology.orchestra.run.smithi136.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-10-30T12:25:08.843 INFO:teuthology.orchestra.run.smithi136.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-10-30T12:25:08.846 INFO:teuthology.orchestra.run.smithi136.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-10-30T12:25:08.848 INFO:teuthology.orchestra.run.smithi136.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-10-30T12:25:08.889 INFO:teuthology.orchestra.run.smithi136.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-10-30T12:25:08.903 INFO:teuthology.orchestra.run.smithi136.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-10-30T12:25:08.908 INFO:teuthology.orchestra.run.smithi136.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-10-30T12:25:09.162 INFO:teuthology.orchestra.run.smithi136.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-10-30T12:25:09.245 INFO:teuthology.orchestra.run.smithi136.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-10-30T12:25:09.335 INFO:teuthology.orchestra.run.smithi136.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-10-30T12:25:09.392 INFO:teuthology.orchestra.run.smithi136.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-10-30T12:25:09.514 INFO:teuthology.orchestra.run.smithi136.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-10-30T12:25:09.602 INFO:teuthology.orchestra.run.smithi136.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-10-30T12:25:09.620 INFO:teuthology.orchestra.run.smithi136.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-urllib3 all 1.26.5-1~exp1 [96.4 kB] 2023-10-30T12:25:09.649 INFO:teuthology.orchestra.run.smithi136.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-10-30T12:25:09.662 INFO:teuthology.orchestra.run.smithi136.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-10-30T12:25:09.700 INFO:teuthology.orchestra.run.smithi136.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-10-30T12:25:09.708 INFO:teuthology.orchestra.run.smithi136.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-10-30T12:25:09.899 INFO:teuthology.orchestra.run.smithi136.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-10-30T12:25:10.168 INFO:teuthology.orchestra.run.smithi125.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6835-g00c1c13c-1jammy [6,858 kB] 2023-10-30T12:25:10.714 INFO:teuthology.orchestra.run.smithi136.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-10-30T12:25:10.725 INFO:teuthology.orchestra.run.smithi136.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-10-30T12:25:10.758 INFO:teuthology.orchestra.run.smithi136.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-10-30T12:25:10.759 INFO:teuthology.orchestra.run.smithi136.stdout:Get:61 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.0.0-6835-g00c1c13c-1jammy [6,858 kB] 2023-10-30T12:25:10.760 INFO:teuthology.orchestra.run.smithi136.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-10-30T12:25:10.761 INFO:teuthology.orchestra.run.smithi136.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-10-30T12:25:10.764 INFO:teuthology.orchestra.run.smithi136.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-10-30T12:25:10.764 INFO:teuthology.orchestra.run.smithi125.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6835-g00c1c13c-1jammy [238 kB] 2023-10-30T12:25:10.776 INFO:teuthology.orchestra.run.smithi125.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6835-g00c1c13c-1jammy [128 kB] 2023-10-30T12:25:10.782 INFO:teuthology.orchestra.run.smithi125.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6835-g00c1c13c-1jammy [1,496 kB] 2023-10-30T12:25:10.809 INFO:teuthology.orchestra.run.smithi136.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-10-30T12:25:10.813 INFO:teuthology.orchestra.run.smithi136.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-10-30T12:25:10.814 INFO:teuthology.orchestra.run.smithi136.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-10-30T12:25:10.878 INFO:teuthology.orchestra.run.smithi136.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-10-30T12:25:10.914 INFO:teuthology.orchestra.run.smithi125.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6835-g00c1c13c-1jammy [9,892 kB] 2023-10-30T12:25:11.042 INFO:teuthology.orchestra.run.smithi136.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-10-30T12:25:11.211 INFO:teuthology.orchestra.run.smithi136.stdout:Get:70 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.0.0-6835-g00c1c13c-1jammy [238 kB] 2023-10-30T12:25:11.220 INFO:teuthology.orchestra.run.smithi136.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.0.0-6835-g00c1c13c-1jammy [128 kB] 2023-10-30T12:25:11.239 INFO:teuthology.orchestra.run.smithi136.stdout:Get:72 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.0.0-6835-g00c1c13c-1jammy [1,496 kB] 2023-10-30T12:25:11.299 INFO:teuthology.orchestra.run.smithi136.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-10-30T12:25:11.332 INFO:teuthology.orchestra.run.smithi136.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-10-30T12:25:11.357 INFO:teuthology.orchestra.run.smithi136.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.0.0-6835-g00c1c13c-1jammy [9,892 kB] 2023-10-30T12:25:11.359 INFO:teuthology.orchestra.run.smithi136.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-10-30T12:25:11.412 INFO:teuthology.orchestra.run.smithi136.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-10-30T12:25:11.430 INFO:teuthology.orchestra.run.smithi136.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-10-30T12:25:11.460 INFO:teuthology.orchestra.run.smithi136.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-10-30T12:25:11.588 INFO:teuthology.orchestra.run.smithi136.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-10-30T12:25:11.619 INFO:teuthology.orchestra.run.smithi136.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-10-30T12:25:11.641 INFO:teuthology.orchestra.run.smithi136.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-10-30T12:25:11.648 INFO:teuthology.orchestra.run.smithi136.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-10-30T12:25:11.806 INFO:teuthology.orchestra.run.smithi125.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6835-g00c1c13c-1jammy [31.7 MB] 2023-10-30T12:25:11.852 INFO:teuthology.orchestra.run.smithi136.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-10-30T12:25:12.211 INFO:teuthology.orchestra.run.smithi136.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.0.0-6835-g00c1c13c-1jammy [31.7 MB] 2023-10-30T12:25:12.313 INFO:teuthology.orchestra.run.smithi136.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-colorama all 0.4.4-1 [24.5 kB] 2023-10-30T12:25:12.319 INFO:teuthology.orchestra.run.smithi136.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-10-30T12:25:12.446 INFO:teuthology.orchestra.run.smithi136.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-10-30T12:25:12.493 INFO:teuthology.orchestra.run.smithi136.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-10-30T12:25:12.493 INFO:teuthology.orchestra.run.smithi136.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-10-30T12:25:12.499 INFO:teuthology.orchestra.run.smithi136.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-10-30T12:25:12.500 INFO:teuthology.orchestra.run.smithi136.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-10-30T12:25:12.516 INFO:teuthology.orchestra.run.smithi136.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-10-30T12:25:12.524 INFO:teuthology.orchestra.run.smithi136.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-10-30T12:25:12.584 INFO:teuthology.orchestra.run.smithi136.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-10-30T12:25:12.750 INFO:teuthology.orchestra.run.smithi136.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-10-30T12:25:12.916 INFO:teuthology.orchestra.run.smithi136.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-10-30T12:25:13.193 INFO:teuthology.orchestra.run.smithi136.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-10-30T12:25:13.218 INFO:teuthology.orchestra.run.smithi136.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-10-30T12:25:14.666 INFO:teuthology.orchestra.run.smithi125.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6835-g00c1c13c-1jammy [14.1 kB] 2023-10-30T12:25:14.666 INFO:teuthology.orchestra.run.smithi125.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [839 kB] 2023-10-30T12:25:14.741 INFO:teuthology.orchestra.run.smithi125.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6835-g00c1c13c-1jammy [2,148 kB] 2023-10-30T12:25:14.872 INFO:teuthology.orchestra.run.smithi136.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.0.0-6835-g00c1c13c-1jammy [14.1 kB] 2023-10-30T12:25:14.872 INFO:teuthology.orchestra.run.smithi136.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [839 kB] 2023-10-30T12:25:14.889 INFO:teuthology.orchestra.run.smithi136.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.0.0-6835-g00c1c13c-1jammy [2,148 kB] 2023-10-30T12:25:14.923 INFO:teuthology.orchestra.run.smithi125.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6835-g00c1c13c-1jammy [284 kB] 2023-10-30T12:25:14.954 INFO:teuthology.orchestra.run.smithi125.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6835-g00c1c13c-1jammy [138 kB] 2023-10-30T12:25:14.980 INFO:teuthology.orchestra.run.smithi125.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6835-g00c1c13c-1jammy [1,907 kB] 2023-10-30T12:25:15.024 INFO:teuthology.orchestra.run.smithi136.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.0.0-6835-g00c1c13c-1jammy [284 kB] 2023-10-30T12:25:15.041 INFO:teuthology.orchestra.run.smithi136.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.0.0-6835-g00c1c13c-1jammy [138 kB] 2023-10-30T12:25:15.047 INFO:teuthology.orchestra.run.smithi136.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.0.0-6835-g00c1c13c-1jammy [1,907 kB] 2023-10-30T12:25:15.144 INFO:teuthology.orchestra.run.smithi125.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6835-g00c1c13c-1jammy [8,625 kB] 2023-10-30T12:25:15.202 INFO:teuthology.orchestra.run.smithi136.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.0.0-6835-g00c1c13c-1jammy [8,625 kB] 2023-10-30T12:25:15.919 INFO:teuthology.orchestra.run.smithi136.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6835-g00c1c13c-1jammy [14.2 kB] 2023-10-30T12:25:15.919 INFO:teuthology.orchestra.run.smithi136.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6835-g00c1c13c-1jammy [47.8 MB] 2023-10-30T12:25:15.956 INFO:teuthology.orchestra.run.smithi125.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.0.0-6835-g00c1c13c-1jammy [14.2 kB] 2023-10-30T12:25:15.956 INFO:teuthology.orchestra.run.smithi125.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.0.0-6835-g00c1c13c-1jammy [47.8 MB] 2023-10-30T12:25:20.614 INFO:teuthology.orchestra.run.smithi136.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6835-g00c1c13c-1jammy [120 kB] 2023-10-30T12:25:20.616 INFO:teuthology.orchestra.run.smithi136.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-6835-g00c1c13c-1jammy [227 kB] 2023-10-30T12:25:20.620 INFO:teuthology.orchestra.run.smithi136.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-6835-g00c1c13c-1jammy [34.0 kB] 2023-10-30T12:25:20.621 INFO:teuthology.orchestra.run.smithi136.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-6835-g00c1c13c-1jammy [23.4 kB] 2023-10-30T12:25:20.622 INFO:teuthology.orchestra.run.smithi136.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6835-g00c1c13c-1jammy [38.3 kB] 2023-10-30T12:25:20.623 INFO:teuthology.orchestra.run.smithi136.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6835-g00c1c13c-1jammy [12.3 MB] 2023-10-30T12:25:20.918 INFO:teuthology.orchestra.run.smithi125.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.0.0-6835-g00c1c13c-1jammy [120 kB] 2023-10-30T12:25:20.935 INFO:teuthology.orchestra.run.smithi125.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-mirror amd64 18.0.0-6835-g00c1c13c-1jammy [227 kB] 2023-10-30T12:25:20.946 INFO:teuthology.orchestra.run.smithi125.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-shell all 18.0.0-6835-g00c1c13c-1jammy [34.0 kB] 2023-10-30T12:25:20.948 INFO:teuthology.orchestra.run.smithi125.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 cephfs-top all 18.0.0-6835-g00c1c13c-1jammy [23.4 kB] 2023-10-30T12:25:20.949 INFO:teuthology.orchestra.run.smithi125.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.0.0-6835-g00c1c13c-1jammy [38.3 kB] 2023-10-30T12:25:20.951 INFO:teuthology.orchestra.run.smithi125.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.0.0-6835-g00c1c13c-1jammy [12.3 MB] 2023-10-30T12:25:21.515 INFO:teuthology.orchestra.run.smithi136.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [91.6 kB] 2023-10-30T12:25:21.958 INFO:teuthology.orchestra.run.smithi125.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-lusov-balance-after-stop/00c1c13cea21a686b046584596fbeac5315a7d71/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.0.0-6835-g00c1c13c-1jammy [91.6 kB] 2023-10-30T12:25:22.068 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 184 MB in 15s (12.3 MB/s) 2023-10-30T12:25:22.230 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-30T12:25:22.305 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 130506 files and directories currently installed.) 2023-10-30T12:25:22.309 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-30T12:25:22.353 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-30T12:25:22.540 INFO:teuthology.orchestra.run.smithi125.stdout:Fetched 184 MB in 15s (12.0 MB/s) 2023-10-30T12:25:22.640 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-30T12:25:22.661 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-30T12:25:22.692 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pyperclip. 2023-10-30T12:25:22.698 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T12:25:22.762 INFO:teuthology.orchestra.run.smithi125.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 132932 files and directories currently installed.) 2023-10-30T12:25:22.765 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../000-python3-pyperclip_1.8.2-2_all.deb ... 2023-10-30T12:25:22.808 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pyperclip (1.8.2-2) ... 2023-10-30T12:25:23.102 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-wcwidth. 2023-10-30T12:25:23.116 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../001-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-10-30T12:25:23.152 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T12:25:23.318 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cmd2. 2023-10-30T12:25:23.339 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-30T12:25:23.376 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-30T12:25:23.472 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-cmd2. 2023-10-30T12:25:23.486 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../002-python3-cmd2_2.3.3-1_all.deb ... 2023-10-30T12:25:23.521 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-cmd2 (2.3.3-1) ... 2023-10-30T12:25:23.721 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-30T12:25:23.742 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-30T12:25:23.821 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T12:25:23.824 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libfmt8:amd64. 2023-10-30T12:25:23.839 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../003-libfmt8_8.1.1+ds1-2_amd64.deb ... 2023-10-30T12:25:23.908 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T12:25:24.165 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-30T12:25:24.186 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-30T12:25:24.224 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T12:25:24.261 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-10-30T12:25:24.275 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../004-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-10-30T12:25:24.319 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T12:25:24.569 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-30T12:25:24.590 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-30T12:25:24.627 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T12:25:24.891 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-10-30T12:25:24.906 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../005-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-10-30T12:25:24.939 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-30T12:25:24.960 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-30T12:25:24.997 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T12:25:25.089 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T12:25:25.409 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-10-30T12:25:25.423 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../006-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-10-30T12:25:25.459 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T12:25:25.492 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-30T12:25:25.512 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T12:25:25.558 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:25:25.862 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-10-30T12:25:25.877 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../007-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T12:25:25.928 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:25:25.970 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-30T12:25:25.982 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T12:25:26.012 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:25:26.298 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-10-30T12:25:26.313 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../008-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T12:25:26.348 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-30T12:25:26.357 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:25:26.369 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T12:25:26.406 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:25:26.701 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-10-30T12:25:26.716 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../009-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-10-30T12:25:26.751 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:25:26.793 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-30T12:25:26.814 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-30T12:25:26.852 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T12:25:27.062 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-10-30T12:25:27.077 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../010-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-10-30T12:25:27.121 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T12:25:27.188 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librados2. 2023-10-30T12:25:27.219 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../011-librados2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:27.246 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:27.440 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package librados2. 2023-10-30T12:25:27.455 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../011-librados2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:27.501 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:27.682 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librbd1. 2023-10-30T12:25:27.702 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../012-librbd1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:27.740 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:27.935 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package librbd1. 2023-10-30T12:25:27.949 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../012-librbd1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:27.992 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:28.235 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libcephfs2. 2023-10-30T12:25:28.255 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:28.293 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:28.395 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libcephfs2. 2023-10-30T12:25:28.410 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../013-libcephfs2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:28.445 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:28.579 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rados. 2023-10-30T12:25:28.591 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../014-python3-rados_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:28.629 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:28.706 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-rados. 2023-10-30T12:25:28.721 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../014-python3-rados_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:28.764 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:28.932 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-30T12:25:28.952 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:28.982 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:29.092 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-10-30T12:25:29.106 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../015-python3-ceph-argparse_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:29.150 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:29.218 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cephfs. 2023-10-30T12:25:29.230 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:29.268 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:29.427 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-cephfs. 2023-10-30T12:25:29.442 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../016-python3-cephfs_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:29.486 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:29.620 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-30T12:25:29.641 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:29.679 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:29.797 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-ceph-common. 2023-10-30T12:25:29.812 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../017-python3-ceph-common_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:29.847 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:29.990 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-prettytable. 2023-10-30T12:25:30.010 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-30T12:25:30.048 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-30T12:25:30.133 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-prettytable. 2023-10-30T12:25:30.148 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../018-python3-prettytable_2.5.0-2_all.deb ... 2023-10-30T12:25:30.191 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-10-30T12:25:30.326 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rbd. 2023-10-30T12:25:30.338 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:30.376 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:30.444 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-rbd. 2023-10-30T12:25:30.458 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../019-python3-rbd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:30.502 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:30.704 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-30T12:25:30.724 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-30T12:25:30.762 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T12:25:30.871 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-10-30T12:25:30.886 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../020-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-10-30T12:25:30.938 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T12:25:31.131 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librgw2. 2023-10-30T12:25:31.152 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../021-librgw2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:31.190 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:31.233 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package librgw2. 2023-10-30T12:25:31.247 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../021-librgw2_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:31.282 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:31.710 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-rgw. 2023-10-30T12:25:31.724 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:31.759 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rgw. 2023-10-30T12:25:31.768 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:31.779 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../022-python3-rgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:31.817 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:32.087 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-30T12:25:32.102 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-30T12:25:32.112 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liboath0:amd64. 2023-10-30T12:25:32.132 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../023-liboath0_2.6.7-3build1_amd64.deb ... 2023-10-30T12:25:32.162 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T12:25:32.170 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T12:25:32.482 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libradosstriper1. 2023-10-30T12:25:32.497 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:32.540 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libradosstriper1. 2023-10-30T12:25:32.540 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:32.560 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../024-libradosstriper1_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:32.598 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:32.943 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-common. 2023-10-30T12:25:32.958 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../025-ceph-common_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:32.993 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:33.051 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-common. 2023-10-30T12:25:33.072 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../025-ceph-common_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:33.109 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:34.074 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-base. 2023-10-30T12:25:34.088 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../026-ceph-base_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:34.141 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:34.440 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-base. 2023-10-30T12:25:34.461 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../026-ceph-base_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:34.507 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:34.819 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-30T12:25:34.833 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-30T12:25:34.885 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-30T12:25:35.228 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-10-30T12:25:35.248 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../027-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-10-30T12:25:35.286 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-10-30T12:25:35.339 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-cheroot. 2023-10-30T12:25:35.353 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-30T12:25:35.397 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T12:25:35.614 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cheroot. 2023-10-30T12:25:35.634 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../028-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-10-30T12:25:35.672 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T12:25:35.725 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-30T12:25:35.739 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-30T12:25:35.783 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-30T12:25:36.009 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-10-30T12:25:36.029 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../029-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-10-30T12:25:36.067 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-10-30T12:25:36.119 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-30T12:25:36.133 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-30T12:25:36.194 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-30T12:25:36.395 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.text. 2023-10-30T12:25:36.415 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../030-python3-jaraco.text_3.6.0-2_all.deb ... 2023-10-30T12:25:36.445 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-10-30T12:25:36.522 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-30T12:25:36.535 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-30T12:25:36.589 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-30T12:25:36.731 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-10-30T12:25:36.751 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../031-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-10-30T12:25:36.789 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-10-30T12:25:36.875 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-tempora. 2023-10-30T12:25:36.888 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-30T12:25:36.933 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-30T12:25:37.117 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-tempora. 2023-10-30T12:25:37.138 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../032-python3-tempora_4.1.2-1_all.deb ... 2023-10-30T12:25:37.175 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-10-30T12:25:37.210 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-portend. 2023-10-30T12:25:37.224 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-30T12:25:37.268 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-30T12:25:37.478 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-portend. 2023-10-30T12:25:37.499 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../033-python3-portend_3.0.0-1_all.deb ... 2023-10-30T12:25:37.536 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-10-30T12:25:37.571 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-30T12:25:37.586 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-30T12:25:37.646 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-30T12:25:37.856 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-10-30T12:25:37.877 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../034-python3-zc.lockfile_2.0-1_all.deb ... 2023-10-30T12:25:37.923 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-10-30T12:25:37.966 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-30T12:25:37.981 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-30T12:25:38.024 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-30T12:25:38.226 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cherrypy3. 2023-10-30T12:25:38.246 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../035-python3-cherrypy3_18.6.1-4_all.deb ... 2023-10-30T12:25:38.284 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-10-30T12:25:38.377 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-natsort. 2023-10-30T12:25:38.392 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-30T12:25:38.435 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-30T12:25:38.629 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-natsort. 2023-10-30T12:25:38.650 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../036-python3-natsort_8.0.2-1_all.deb ... 2023-10-30T12:25:38.687 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-10-30T12:25:38.730 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-logutils. 2023-10-30T12:25:38.745 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-30T12:25:38.797 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-30T12:25:38.999 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-logutils. 2023-10-30T12:25:39.019 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../037-python3-logutils_0.3.3-8_all.deb ... 2023-10-30T12:25:39.057 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-10-30T12:25:39.099 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-30T12:25:39.114 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-30T12:25:39.149 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-30T12:25:39.345 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-markupsafe. 2023-10-30T12:25:39.366 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../038-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-10-30T12:25:39.403 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-10-30T12:25:39.436 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-mako. 2023-10-30T12:25:39.450 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-30T12:25:39.493 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T12:25:40.424 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-mako. 2023-10-30T12:25:40.444 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../039-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-10-30T12:25:40.482 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T12:25:40.514 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-30T12:25:40.529 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-30T12:25:40.572 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-30T12:25:40.802 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-simplegeneric. 2023-10-30T12:25:40.822 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../040-python3-simplegeneric_0.8.1-3_all.deb ... 2023-10-30T12:25:40.858 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-30T12:25:40.860 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-10-30T12:25:40.873 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-30T12:25:40.916 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-30T12:25:41.146 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-singledispatch. 2023-10-30T12:25:41.167 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../041-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-10-30T12:25:41.202 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-webob. 2023-10-30T12:25:41.204 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-10-30T12:25:41.217 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-30T12:25:41.260 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-30T12:25:41.524 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-webob. 2023-10-30T12:25:41.545 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../042-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-10-30T12:25:41.582 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-10-30T12:25:41.613 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-waitress. 2023-10-30T12:25:41.628 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-30T12:25:41.674 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T12:25:41.960 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-waitress. 2023-10-30T12:25:41.981 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../043-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-10-30T12:25:42.008 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-tempita. 2023-10-30T12:25:42.021 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T12:25:42.022 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-30T12:25:42.066 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T12:25:42.336 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-paste. 2023-10-30T12:25:42.351 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-30T12:25:42.371 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-tempita. 2023-10-30T12:25:42.385 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T12:25:42.392 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../044-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-10-30T12:25:42.430 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T12:25:42.696 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-30T12:25:42.711 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-30T12:25:42.732 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-paste. 2023-10-30T12:25:42.753 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../045-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-10-30T12:25:42.754 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T12:25:42.790 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T12:25:43.041 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-30T12:25:43.056 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-30T12:25:43.124 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-30T12:25:43.152 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-10-30T12:25:43.164 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../046-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-10-30T12:25:43.202 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T12:25:43.394 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-webtest. 2023-10-30T12:25:43.409 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-30T12:25:43.452 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-30T12:25:43.505 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pastedeploy. 2023-10-30T12:25:43.526 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../047-python3-pastedeploy_2.1.1-1_all.deb ... 2023-10-30T12:25:43.558 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-10-30T12:25:43.746 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pecan. 2023-10-30T12:25:43.761 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-30T12:25:43.804 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T12:25:44.083 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-webtest. 2023-10-30T12:25:44.103 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../048-python3-webtest_2.0.35-1_all.deb ... 2023-10-30T12:25:44.141 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-10-30T12:25:44.142 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-certifi. 2023-10-30T12:25:44.157 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-30T12:25:44.201 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-30T12:25:44.453 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pecan. 2023-10-30T12:25:44.473 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../049-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-10-30T12:25:44.511 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T12:25:44.570 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-idna. 2023-10-30T12:25:44.585 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-30T12:25:44.628 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-30T12:25:44.891 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-certifi. 2023-10-30T12:25:44.898 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-urllib3. 2023-10-30T12:25:44.913 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../050-python3-certifi_2020.6.20-1_all.deb ... 2023-10-30T12:25:44.914 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-30T12:25:44.948 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T12:25:44.948 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-10-30T12:25:45.242 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-requests. 2023-10-30T12:25:45.258 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-30T12:25:45.292 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T12:25:45.318 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-idna. 2023-10-30T12:25:45.339 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../051-python3-idna_3.3-1_all.deb ... 2023-10-30T12:25:45.385 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-idna (3.3-1) ... 2023-10-30T12:25:45.603 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-30T12:25:45.618 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-30T12:25:45.661 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T12:25:45.671 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-urllib3. 2023-10-30T12:25:45.692 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../052-python3-urllib3_1.26.5-1~exp1_all.deb ... 2023-10-30T12:25:45.729 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T12:25:45.964 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-30T12:25:45.979 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:46.023 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:46.058 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-requests. 2023-10-30T12:25:46.079 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../053-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-10-30T12:25:46.116 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T12:25:46.394 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-30T12:25:46.408 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:46.436 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-werkzeug. 2023-10-30T12:25:46.452 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:46.456 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../054-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-10-30T12:25:46.494 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T12:25:46.746 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-openssl. 2023-10-30T12:25:46.761 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-30T12:25:46.805 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-30T12:25:46.814 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-10-30T12:25:46.834 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../055-ceph-mgr-modules-core_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:46.872 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:47.091 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr. 2023-10-30T12:25:47.106 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:47.141 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:47.252 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-10-30T12:25:47.273 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../056-libsqlite3-mod-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:47.310 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:47.535 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mon. 2023-10-30T12:25:47.550 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:47.594 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:47.646 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-openssl. 2023-10-30T12:25:47.667 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../057-python3-openssl_21.0.0-1_all.deb ... 2023-10-30T12:25:47.704 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-10-30T12:25:48.058 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr. 2023-10-30T12:25:48.078 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../058-ceph-mgr_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:48.116 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:48.364 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-osd. 2023-10-30T12:25:48.378 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:48.422 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:48.561 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mon. 2023-10-30T12:25:48.582 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../059-ceph-mon_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:48.620 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:49.506 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-osd. 2023-10-30T12:25:49.526 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../060-ceph-osd_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:49.527 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph. 2023-10-30T12:25:49.542 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../061-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:49.564 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:49.577 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:49.921 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-fuse. 2023-10-30T12:25:49.936 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:49.980 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:50.325 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mds. 2023-10-30T12:25:50.339 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:50.374 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:50.894 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package cephadm. 2023-10-30T12:25:50.909 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../064-cephadm_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:50.952 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:51.289 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-30T12:25:51.303 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-30T12:25:51.347 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-30T12:25:51.476 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph. 2023-10-30T12:25:51.496 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../061-ceph_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:51.534 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:51.658 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-30T12:25:51.672 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T12:25:51.716 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T12:25:51.870 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-fuse. 2023-10-30T12:25:51.891 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../062-ceph-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:51.953 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:52.290 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mds. 2023-10-30T12:25:52.311 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../063-ceph-mds_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:52.348 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:52.729 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-babel. 2023-10-30T12:25:52.744 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T12:25:52.759 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package cephadm. 2023-10-30T12:25:52.780 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../064-cephadm_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:25:52.787 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T12:25:52.817 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:53.107 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-jinja2. 2023-10-30T12:25:53.121 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-30T12:25:53.154 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-asyncssh. 2023-10-30T12:25:53.166 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-30T12:25:53.175 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../065-python3-asyncssh_2.5.0-1_all.deb ... 2023-10-30T12:25:53.212 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-10-30T12:25:53.502 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-30T12:25:53.517 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:53.552 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:53.556 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python-babel-localedata. 2023-10-30T12:25:53.569 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../066-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T12:25:53.607 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T12:25:53.872 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-30T12:25:53.887 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-30T12:25:53.922 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-30T12:25:54.225 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-30T12:25:54.240 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../071-python3-repoze.lru_0.7-2_all.deb ... 2023-10-30T12:25:54.283 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-30T12:25:54.569 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-routes. 2023-10-30T12:25:54.584 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../072-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-30T12:25:54.628 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T12:25:54.712 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-babel. 2023-10-30T12:25:54.733 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../067-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-10-30T12:25:54.770 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T12:25:54.964 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-30T12:25:54.979 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../073-ceph-mgr-dashboard_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:55.022 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:55.157 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jinja2. 2023-10-30T12:25:55.178 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../068-python3-jinja2_3.0.3-1_all.deb ... 2023-10-30T12:25:55.206 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-10-30T12:25:55.442 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-30T12:25:55.457 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-30T12:25:55.492 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T12:25:55.526 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-10-30T12:25:55.538 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../069-ceph-mgr-cephadm_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:55.576 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:55.964 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-bcrypt. 2023-10-30T12:25:55.985 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../070-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-10-30T12:25:55.988 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-joblib. 2023-10-30T12:25:56.002 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../075-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-30T12:25:56.022 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-10-30T12:25:56.046 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T12:25:56.324 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-repoze.lru. 2023-10-30T12:25:56.346 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../071-python3-repoze.lru_0.7-2_all.deb ... 2023-10-30T12:25:56.383 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-30T12:25:56.391 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-10-30T12:25:56.397 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../076-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-30T12:25:56.441 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-30T12:25:56.703 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-routes. 2023-10-30T12:25:56.724 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../072-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-10-30T12:25:56.736 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-sklearn. 2023-10-30T12:25:56.751 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../077-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-30T12:25:56.761 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T12:25:56.794 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T12:25:57.139 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-10-30T12:25:57.161 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../073-ceph-mgr-dashboard_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:57.206 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:57.469 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-30T12:25:57.483 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../078-ceph-mgr-diskprediction-local_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:57.527 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:57.651 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-10-30T12:25:57.672 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../074-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-10-30T12:25:57.717 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T12:25:58.271 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-joblib. 2023-10-30T12:25:58.292 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../075-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-10-30T12:25:58.329 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T12:25:58.472 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-cachetools. 2023-10-30T12:25:58.487 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../079-python3-cachetools_5.0.0-1_all.deb ... 2023-10-30T12:25:58.530 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-30T12:25:58.700 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-10-30T12:25:58.721 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../076-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-10-30T12:25:58.766 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-10-30T12:25:58.808 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-30T12:25:58.823 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../080-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-30T12:25:58.858 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-30T12:25:59.086 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-sklearn. 2023-10-30T12:25:59.107 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../077-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-10-30T12:25:59.152 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T12:25:59.153 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-30T12:25:59.168 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../081-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-30T12:25:59.211 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T12:25:59.505 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-rsa. 2023-10-30T12:25:59.520 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../082-python3-rsa_4.8-1_all.deb ... 2023-10-30T12:25:59.588 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-30T12:25:59.878 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-10-30T12:25:59.900 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../078-ceph-mgr-diskprediction-local_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:25:59.944 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:25:59.977 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-google-auth. 2023-10-30T12:25:59.992 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../083-python3-google-auth_1.5.1-3_all.deb ... 2023-10-30T12:26:00.035 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-30T12:26:00.313 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-30T12:26:00.327 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../084-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-30T12:26:00.363 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T12:26:00.658 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-websocket. 2023-10-30T12:26:00.673 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../085-python3-websocket_1.2.3-1_all.deb ... 2023-10-30T12:26:00.716 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-30T12:26:01.149 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cachetools. 2023-10-30T12:26:01.170 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../079-python3-cachetools_5.0.0-1_all.deb ... 2023-10-30T12:26:01.214 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-10-30T12:26:01.411 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-30T12:26:01.427 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../086-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-30T12:26:01.467 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyasn1. 2023-10-30T12:26:01.469 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T12:26:01.488 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../080-python3-pyasn1_0.4.8-1_all.deb ... 2023-10-30T12:26:01.542 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-10-30T12:26:01.829 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-10-30T12:26:01.850 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../081-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-10-30T12:26:01.887 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T12:26:02.116 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-30T12:26:02.130 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../087-ceph-mgr-k8sevents_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:26:02.156 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rsa. 2023-10-30T12:26:02.177 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../082-python3-rsa_4.8-1_all.deb ... 2023-10-30T12:26:02.182 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:02.214 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rsa (4.8-1) ... 2023-10-30T12:26:02.527 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-30T12:26:02.541 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../088-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-30T12:26:02.585 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T12:26:02.611 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-google-auth. 2023-10-30T12:26:02.624 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../083-python3-google-auth_1.5.1-3_all.deb ... 2023-10-30T12:26:02.653 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-10-30T12:26:02.922 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-30T12:26:02.936 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../089-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T12:26:02.948 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-10-30T12:26:02.969 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../084-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-10-30T12:26:02.979 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T12:26:03.023 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T12:26:03.216 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package jq. 2023-10-30T12:26:03.231 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../090-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T12:26:03.265 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-30T12:26:03.301 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-websocket. 2023-10-30T12:26:03.322 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../085-python3-websocket_1.2.3-1_all.deb ... 2023-10-30T12:26:03.359 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-10-30T12:26:03.493 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package socat. 2023-10-30T12:26:03.508 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../091-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-30T12:26:03.551 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-30T12:26:03.688 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-kubernetes. 2023-10-30T12:26:03.709 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../086-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-10-30T12:26:03.762 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T12:26:03.863 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package xmlstarlet. 2023-10-30T12:26:03.878 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../092-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-30T12:26:03.921 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-30T12:26:04.283 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-test. 2023-10-30T12:26:04.298 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../093-ceph-test_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:04.342 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:04.410 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-10-30T12:26:04.422 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../087-ceph-mgr-k8sevents_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:26:04.493 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:04.813 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libonig5:amd64. 2023-10-30T12:26:04.834 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../088-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-10-30T12:26:04.871 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T12:26:05.224 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libjq1:amd64. 2023-10-30T12:26:05.245 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../089-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T12:26:05.282 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T12:26:05.526 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package jq. 2023-10-30T12:26:05.538 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../090-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-10-30T12:26:05.576 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-10-30T12:26:05.812 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package socat. 2023-10-30T12:26:05.834 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../091-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-10-30T12:26:05.870 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-10-30T12:26:06.079 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ceph-volume. 2023-10-30T12:26:06.094 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../094-ceph-volume_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:26:06.145 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:06.257 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package xmlstarlet. 2023-10-30T12:26:06.278 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../092-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-10-30T12:26:06.315 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-10-30T12:26:06.493 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-30T12:26:06.509 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../095-cephfs-mirror_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:06.558 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:06.745 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-test. 2023-10-30T12:26:06.766 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../093-ceph-test_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:06.820 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:06.870 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-colorama. 2023-10-30T12:26:06.886 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../096-python3-colorama_0.4.4-1_all.deb ... 2023-10-30T12:26:06.927 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-30T12:26:07.172 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package cephfs-shell. 2023-10-30T12:26:07.187 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../097-cephfs-shell_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:26:07.230 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:07.491 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package cephfs-top. 2023-10-30T12:26:07.507 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../098-cephfs-top_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:26:07.558 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:07.811 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-30T12:26:07.826 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../099-libcephfs-dev_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:07.894 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:08.272 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package nvme-cli. 2023-10-30T12:26:08.287 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../100-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-30T12:26:08.330 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T12:26:08.660 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-30T12:26:08.675 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../101-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-30T12:26:08.718 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-30T12:26:09.121 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-30T12:26:09.136 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../102-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-30T12:26:09.179 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-30T12:26:09.490 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pastescript. 2023-10-30T12:26:09.506 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-4_all.deb ... 2023-10-30T12:26:09.548 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-30T12:26:09.557 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-volume. 2023-10-30T12:26:09.579 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../094-ceph-volume_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:26:09.648 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:09.869 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pluggy. 2023-10-30T12:26:09.884 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../104-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-30T12:26:09.927 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-30T12:26:10.011 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package cephfs-mirror. 2023-10-30T12:26:10.032 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../095-cephfs-mirror_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:10.069 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:10.238 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-psutil. 2023-10-30T12:26:10.253 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../105-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-30T12:26:10.312 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-30T12:26:10.431 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-colorama. 2023-10-30T12:26:10.453 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../096-python3-colorama_0.4.4-1_all.deb ... 2023-10-30T12:26:10.497 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-colorama (0.4.4-1) ... 2023-10-30T12:26:10.691 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-py. 2023-10-30T12:26:10.706 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../106-python3-py_1.10.0-1_all.deb ... 2023-10-30T12:26:10.742 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package cephfs-shell. 2023-10-30T12:26:10.749 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-30T12:26:10.754 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../097-cephfs-shell_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:26:10.791 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:11.061 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package cephfs-top. 2023-10-30T12:26:11.074 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../098-cephfs-top_18.0.0-6835-g00c1c13c-1jammy_all.deb ... 2023-10-30T12:26:11.102 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pygments. 2023-10-30T12:26:11.118 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../107-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-30T12:26:11.136 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:11.160 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T12:26:11.364 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libcephfs-dev. 2023-10-30T12:26:11.376 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../099-libcephfs-dev_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:11.405 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:11.764 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-30T12:26:11.780 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-30T12:26:11.792 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package nvme-cli. 2023-10-30T12:26:11.814 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../100-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-10-30T12:26:11.822 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-30T12:26:11.883 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T12:26:12.142 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-toml. 2023-10-30T12:26:12.157 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../109-python3-toml_0.10.2-1_all.deb ... 2023-10-30T12:26:12.200 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-30T12:26:12.239 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-10-30T12:26:12.260 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../101-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-10-30T12:26:12.305 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-10-30T12:26:12.486 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-pytest. 2023-10-30T12:26:12.501 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../110-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-30T12:26:12.535 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T12:26:12.792 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-iniconfig. 2023-10-30T12:26:12.813 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../102-python3-iniconfig_1.1.1-2_all.deb ... 2023-10-30T12:26:12.858 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-10-30T12:26:12.873 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package python3-simplejson. 2023-10-30T12:26:12.888 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-30T12:26:12.931 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-30T12:26:13.153 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pastescript. 2023-10-30T12:26:13.174 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-4_all.deb ... 2023-10-30T12:26:13.191 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-30T12:26:13.207 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-30T12:26:13.211 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-10-30T12:26:13.249 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-30T12:26:13.531 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pluggy. 2023-10-30T12:26:13.553 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../104-python3-pluggy_0.13.0-7.1_all.deb ... 2023-10-30T12:26:13.597 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-10-30T12:26:13.909 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-psutil. 2023-10-30T12:26:13.930 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../105-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-10-30T12:26:13.937 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package radosgw. 2023-10-30T12:26:13.951 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../113-radosgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:13.967 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-10-30T12:26:13.995 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:14.345 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-py. 2023-10-30T12:26:14.367 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../106-python3-py_1.10.0-1_all.deb ... 2023-10-30T12:26:14.420 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-py (1.10.0-1) ... 2023-10-30T12:26:14.565 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package rbd-fuse. 2023-10-30T12:26:14.579 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:14.623 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:14.948 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-30T12:26:15.015 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pygments. 2023-10-30T12:26:15.037 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../107-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-10-30T12:26:15.082 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T12:26:15.211 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T12:26:15.337 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T12:26:15.586 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyinotify. 2023-10-30T12:26:15.587 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-30T12:26:15.608 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../108-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-10-30T12:26:15.653 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-10-30T12:26:15.956 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-toml. 2023-10-30T12:26:15.962 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-30T12:26:15.977 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../109-python3-toml_0.10.2-1_all.deb ... 2023-10-30T12:26:16.022 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-10-30T12:26:16.350 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pytest. 2023-10-30T12:26:16.371 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../110-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-10-30T12:26:16.416 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T12:26:16.607 INFO:teuthology.orchestra.run.smithi125.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-30T12:26:16.647 INFO:teuthology.orchestra.run.smithi125.stdout:Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 142. 2023-10-30T12:26:16.689 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:16.779 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-simplejson. 2023-10-30T12:26:16.800 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../111-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-10-30T12:26:16.845 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-10-30T12:26:17.132 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-10-30T12:26:17.153 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../112-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-10-30T12:26:17.198 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-10-30T12:26:17.225 INFO:teuthology.orchestra.run.smithi125.stdout:Adding system user cephadm....done 2023-10-30T12:26:17.387 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T12:26:17.671 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-30T12:26:17.939 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-30T12:26:17.961 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package radosgw. 2023-10-30T12:26:17.982 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../113-radosgw_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:18.027 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:18.065 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-30T12:26:18.325 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-30T12:26:18.576 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T12:26:18.702 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-30T12:26:18.813 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package rbd-fuse. 2023-10-30T12:26:18.826 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../114-rbd-fuse_18.0.0-6835-g00c1c13c-1jammy_amd64.deb ... 2023-10-30T12:26:18.863 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:19.012 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T12:26:19.190 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-10-30T12:26:19.372 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-30T12:26:19.469 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-10-30T12:26:19.570 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-10-30T12:26:19.632 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-30T12:26:19.803 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-10-30T12:26:19.891 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-30T12:26:20.135 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-10-30T12:26:20.201 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-30T12:26:20.452 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-30T12:26:20.601 INFO:teuthology.orchestra.run.smithi136.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-10-30T12:26:20.671 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:20.719 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-30T12:26:20.987 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:21.254 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T12:26:21.281 INFO:teuthology.orchestra.run.smithi136.stdout:Adding system user cephadm....done 2023-10-30T12:26:21.356 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-30T12:26:21.436 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-10-30T12:26:21.607 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-30T12:26:21.745 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-10-30T12:26:21.883 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T12:26:21.988 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-10-30T12:26:22.000 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-30T12:26:22.114 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-10-30T12:26:22.360 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T12:26:22.391 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-10-30T12:26:22.486 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-30T12:26:22.650 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-10-30T12:26:22.762 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T12:26:22.770 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-py (1.10.0-1) ... 2023-10-30T12:26:23.021 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-30T12:26:23.071 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-10-30T12:26:23.323 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T12:26:23.431 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-10-30T12:26:23.716 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-colorama (0.4.4-1) ... 2023-10-30T12:26:23.958 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-10-30T12:26:24.016 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T12:26:24.218 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-10-30T12:26:24.292 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T12:26:24.418 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-30T12:26:24.469 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-10-30T12:26:24.544 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T12:26:24.720 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-10-30T12:26:24.853 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-30T12:26:24.971 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-ceph-argparse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:25.238 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-30T12:26:25.263 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-10-30T12:26:25.406 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyperclip (1.8.2-2) ... 2023-10-30T12:26:25.489 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T12:26:25.666 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-10-30T12:26:25.874 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T12:26:25.925 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-10-30T12:26:26.051 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-10-30T12:26:26.184 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-30T12:26:26.410 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-10-30T12:26:26.502 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T12:26:26.545 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-10-30T12:26:26.838 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-10-30T12:26:26.853 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-30T12:26:27.113 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-30T12:26:27.113 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-10-30T12:26:27.213 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:27.432 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-10-30T12:26:27.498 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-idna (3.3-1) ... 2023-10-30T12:26:27.840 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T12:26:28.125 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-10-30T12:26:28.376 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-10-30T12:26:28.502 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-10-30T12:26:28.653 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-10-30T12:26:28.946 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-10-30T12:26:29.063 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:26:29.172 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-30T12:26:29.322 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-10-30T12:26:29.432 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T12:26:29.549 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-30T12:26:29.590 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-10-30T12:26:29.683 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T12:26:29.958 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-10-30T12:26:29.993 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-30T12:26:30.225 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cmd2 (2.3.3-1) ... 2023-10-30T12:26:30.261 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-30T12:26:30.520 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:26:30.568 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-10-30T12:26:30.646 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T12:26:30.772 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-30T12:26:30.920 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-10-30T12:26:31.082 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-30T12:26:31.170 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-10-30T12:26:31.297 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:31.343 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-30T12:26:31.589 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-idna (3.3-1) ... 2023-10-30T12:26:31.611 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-30T12:26:31.870 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-30T12:26:31.907 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-10-30T12:26:32.121 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-30T12:26:32.397 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T12:26:32.523 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-30T12:26:32.808 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T12:26:32.909 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T12:26:33.087 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:26:33.210 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-30T12:26:33.221 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-toml (0.10.2-1) ... 2023-10-30T12:26:33.506 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-10-30T12:26:33.519 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-30T12:26:33.632 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-10-30T12:26:33.750 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-urllib3 (1.26.5-1~exp1) ... 2023-10-30T12:26:33.779 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T12:26:34.042 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-10-30T12:26:34.189 INFO:teuthology.orchestra.run.smithi125.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-30T12:26:34.302 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-10-30T12:26:34.432 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T12:26:34.553 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:26:34.557 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T12:26:34.679 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2023-10-30T12:26:34.830 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-10-30T12:26:34.934 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-30T12:26:35.123 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-10-30T12:26:35.211 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-30T12:26:35.382 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-10-30T12:26:35.470 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:26:35.596 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-30T12:26:35.666 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-10-30T12:26:35.913 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T12:26:35.917 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-10-30T12:26:36.210 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-10-30T12:26:36.223 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T12:26:36.470 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-10-30T12:26:36.498 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-30T12:26:36.598 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-10-30T12:26:36.624 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-30T12:26:36.880 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-10-30T12:26:36.909 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-30T12:26:37.056 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-10-30T12:26:37.194 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T12:26:37.342 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-10-30T12:26:37.461 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-30T12:26:37.661 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-10-30T12:26:37.846 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-30T12:26:37.926 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-10-30T12:26:38.139 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T12:26:38.333 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-10-30T12:26:38.499 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T12:26:38.570 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-10-30T12:26:38.608 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:38.696 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-10-30T12:26:38.743 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:38.946 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:39.072 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:39.080 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-10-30T12:26:39.215 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:39.332 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-30T12:26:39.365 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-portend (3.0.0-1) ... 2023-10-30T12:26:39.601 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:39.616 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-10-30T12:26:39.727 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:39.742 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-10-30T12:26:39.828 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:40.020 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T12:26:40.020 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T12:26:40.077 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-10-30T12:26:40.369 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-10-30T12:26:40.655 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:40.662 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-10-30T12:26:40.780 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:40.788 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rsa (4.8-1) ... 2023-10-30T12:26:40.906 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:41.024 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T12:26:41.056 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-10-30T12:26:41.344 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-10-30T12:26:41.608 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-10-30T12:26:42.001 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-10-30T12:26:42.304 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.310 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-10-30T12:26:42.430 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.556 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.646 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-10-30T12:26:42.673 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.772 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librados2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.791 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.906 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librgw2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:42.917 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:43.117 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libsqlite3-mod-ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:43.244 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libcephfs2 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:43.369 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libradosstriper1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:43.478 INFO:teuthology.orchestra.run.smithi125.stdout:Adding group ceph....done 2023-10-30T12:26:43.495 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-10-30T12:26:43.814 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librbd1 (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:43.838 INFO:teuthology.orchestra.run.smithi125.stdout:Adding system user ceph....done 2023-10-30T12:26:43.915 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-modules-core (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:44.026 INFO:teuthology.orchestra.run.smithi125.stdout:Setting system user ceph properties....done 2023-10-30T12:26:44.032 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:44.047 INFO:teuthology.orchestra.run.smithi125.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-30T12:26:44.186 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-30T12:26:44.244 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T12:26:44.244 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-10-30T12:26:44.541 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-30T12:26:44.734 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libcephfs-dev (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:44.860 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rados (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:44.986 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up cephfs-top (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:45.137 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-10-30T12:26:45.185 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:45.392 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T12:26:45.393 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T12:26:45.986 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:46.112 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:46.399 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rbd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:46.559 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up rbd-fuse (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:46.648 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T12:26:46.649 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T12:26:46.685 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:46.811 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cephfs (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:46.953 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up cephfs-shell (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:47.079 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-common (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:47.280 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:47.683 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-30T12:26:47.705 INFO:teuthology.orchestra.run.smithi136.stdout:Adding group ceph....done 2023-10-30T12:26:48.123 INFO:teuthology.orchestra.run.smithi136.stdout:Adding system user ceph....done 2023-10-30T12:26:48.300 INFO:teuthology.orchestra.run.smithi136.stdout:Setting system user ceph properties....done 2023-10-30T12:26:48.314 INFO:teuthology.orchestra.run.smithi136.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-10-30T12:26:48.324 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:48.452 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-10-30T12:26:48.530 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T12:26:48.530 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T12:26:48.832 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-10-30T12:26:49.151 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:49.364 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T12:26:49.365 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T12:26:49.371 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up cephfs-mirror (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:49.600 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T12:26:49.600 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2023-10-30T12:26:49.993 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:50.114 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-test (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:50.265 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up radosgw (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:50.329 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T12:26:50.329 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T12:26:50.750 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T12:26:50.750 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-10-30T12:26:50.962 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:51.088 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:51.230 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:51.250 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-base (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:51.439 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T12:26:51.439 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T12:26:51.605 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-10-30T12:26:52.115 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:52.135 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mds (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:52.258 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:52.358 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T12:26:52.358 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-10-30T12:26:52.727 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:52.862 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:52.886 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:53.081 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T12:26:53.081 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-10-30T12:26:53.239 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T12:26:53.580 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-osd (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:53.883 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T12:26:53.884 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-10-30T12:26:54.406 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-k8sevents (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:54.532 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-diskprediction-local (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:54.573 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T12:26:54.708 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mon (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:54.923 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T12:26:54.923 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-10-30T12:26:55.050 INFO:teuthology.orchestra.run.smithi125.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T12:26:55.054 DEBUG:teuthology.orchestra.run.smithi125:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-30T12:26:55.130 INFO:teuthology.orchestra.run.smithi125.stdout:Reading package lists... 2023-10-30T12:26:55.295 INFO:teuthology.orchestra.run.smithi125.stdout:Building dependency tree... 2023-10-30T12:26:55.295 INFO:teuthology.orchestra.run.smithi125.stdout:Reading state information... 2023-10-30T12:26:55.443 INFO:teuthology.orchestra.run.smithi125.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-30T12:26:55.443 INFO:teuthology.orchestra.run.smithi125.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-30T12:26:55.443 INFO:teuthology.orchestra.run.smithi125.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-30T12:26:55.444 INFO:teuthology.orchestra.run.smithi125.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-30T12:26:55.444 INFO:teuthology.orchestra.run.smithi125.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-30T12:26:55.444 INFO:teuthology.orchestra.run.smithi125.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T12:26:55.444 INFO:teuthology.orchestra.run.smithi125.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T12:26:55.444 INFO:teuthology.orchestra.run.smithi125.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T12:26:55.444 INFO:teuthology.orchestra.run.smithi125.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T12:26:55.445 INFO:teuthology.orchestra.run.smithi125.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-10-30T12:26:55.445 INFO:teuthology.orchestra.run.smithi125.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-10-30T12:26:55.445 INFO:teuthology.orchestra.run.smithi125.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T12:26:55.446 INFO:teuthology.orchestra.run.smithi125.stdout:The following additional packages will be installed: 2023-10-30T12:26:55.446 INFO:teuthology.orchestra.run.smithi125.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-30T12:26:55.446 INFO:teuthology.orchestra.run.smithi125.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-30T12:26:55.447 INFO:teuthology.orchestra.run.smithi125.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-30T12:26:55.447 INFO:teuthology.orchestra.run.smithi125.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-30T12:26:55.447 INFO:teuthology.orchestra.run.smithi125.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T12:26:55.447 INFO:teuthology.orchestra.run.smithi125.stdout:Suggested packages: 2023-10-30T12:26:55.447 INFO:teuthology.orchestra.run.smithi125.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-30T12:26:55.447 INFO:teuthology.orchestra.run.smithi125.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-30T12:26:55.448 INFO:teuthology.orchestra.run.smithi125.stdout: libengine-pkcs11-openssl 2023-10-30T12:26:55.468 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-cephadm (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:55.494 INFO:teuthology.orchestra.run.smithi125.stdout:The following NEW packages will be installed: 2023-10-30T12:26:55.495 INFO:teuthology.orchestra.run.smithi125.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-30T12:26:55.495 INFO:teuthology.orchestra.run.smithi125.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-30T12:26:55.496 INFO:teuthology.orchestra.run.smithi125.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-30T12:26:55.496 INFO:teuthology.orchestra.run.smithi125.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-30T12:26:55.496 INFO:teuthology.orchestra.run.smithi125.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-30T12:26:55.497 INFO:teuthology.orchestra.run.smithi125.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-30T12:26:55.497 INFO:teuthology.orchestra.run.smithi125.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T12:26:55.497 INFO:teuthology.orchestra.run.smithi125.stdout:The following packages will be upgraded: 2023-10-30T12:26:55.498 INFO:teuthology.orchestra.run.smithi125.stdout: libssl3 2023-10-30T12:26:55.574 INFO:teuthology.orchestra.run.smithi125.stdout:1 upgraded, 39 newly installed, 0 to remove and 259 not upgraded. 2023-10-30T12:26:55.574 INFO:teuthology.orchestra.run.smithi125.stdout:Need to get 54.6 MB of archives. 2023-10-30T12:26:55.575 INFO:teuthology.orchestra.run.smithi125.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-30T12:26:55.575 INFO:teuthology.orchestra.run.smithi125.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-30T12:26:55.619 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:55.677 INFO:teuthology.orchestra.run.smithi125.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-10-30T12:26:55.731 INFO:teuthology.orchestra.run.smithi125.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-30T12:26:55.738 INFO:teuthology.orchestra.run.smithi125.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-30T12:26:55.739 INFO:teuthology.orchestra.run.smithi125.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-30T12:26:55.740 INFO:teuthology.orchestra.run.smithi125.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-30T12:26:55.742 INFO:teuthology.orchestra.run.smithi125.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-30T12:26:55.742 INFO:teuthology.orchestra.run.smithi125.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-30T12:26:55.743 INFO:teuthology.orchestra.run.smithi125.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-30T12:26:55.744 INFO:teuthology.orchestra.run.smithi125.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-30T12:26:55.745 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-dashboard (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:55.745 INFO:teuthology.orchestra.run.smithi125.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-30T12:26:55.750 INFO:teuthology.orchestra.run.smithi125.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-30T12:26:55.762 INFO:teuthology.orchestra.run.smithi125.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-30T12:26:55.887 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-volume (18.0.0-6835-g00c1c13c-1jammy) ... 2023-10-30T12:26:55.976 INFO:teuthology.orchestra.run.smithi125.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-30T12:26:55.978 INFO:teuthology.orchestra.run.smithi125.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-30T12:26:55.978 INFO:teuthology.orchestra.run.smithi125.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-30T12:26:55.981 INFO:teuthology.orchestra.run.smithi125.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-30T12:26:55.982 INFO:teuthology.orchestra.run.smithi125.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-30T12:26:55.986 INFO:teuthology.orchestra.run.smithi125.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-30T12:26:55.987 INFO:teuthology.orchestra.run.smithi125.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-30T12:26:55.995 INFO:teuthology.orchestra.run.smithi125.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-30T12:26:55.995 INFO:teuthology.orchestra.run.smithi125.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-10-30T12:26:56.016 INFO:teuthology.orchestra.run.smithi125.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-30T12:26:56.016 INFO:teuthology.orchestra.run.smithi125.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-30T12:26:56.026 INFO:teuthology.orchestra.run.smithi125.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-30T12:26:56.047 INFO:teuthology.orchestra.run.smithi125.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-30T12:26:56.051 INFO:teuthology.orchestra.run.smithi125.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-30T12:26:56.051 INFO:teuthology.orchestra.run.smithi125.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-30T12:26:56.052 INFO:teuthology.orchestra.run.smithi125.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-30T12:26:56.052 INFO:teuthology.orchestra.run.smithi125.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-30T12:26:56.064 INFO:teuthology.orchestra.run.smithi125.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-30T12:26:56.066 INFO:teuthology.orchestra.run.smithi125.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-30T12:26:56.212 INFO:teuthology.orchestra.run.smithi125.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-30T12:26:56.213 INFO:teuthology.orchestra.run.smithi125.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-30T12:26:56.213 INFO:teuthology.orchestra.run.smithi125.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-30T12:26:56.213 INFO:teuthology.orchestra.run.smithi125.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-30T12:26:56.214 INFO:teuthology.orchestra.run.smithi125.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-30T12:26:56.214 INFO:teuthology.orchestra.run.smithi125.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-30T12:26:56.226 INFO:teuthology.orchestra.run.smithi125.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-30T12:26:56.227 INFO:teuthology.orchestra.run.smithi125.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-30T12:26:56.248 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T12:26:56.605 INFO:teuthology.orchestra.run.smithi125.stdout:Preconfiguring packages ... 2023-10-30T12:26:56.763 INFO:teuthology.orchestra.run.smithi125.stdout:Fetched 54.6 MB in 1s (75.7 MB/s) 2023-10-30T12:26:56.947 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package flex. 2023-10-30T12:26:57.019 INFO:teuthology.orchestra.run.smithi125.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139851 files and directories currently installed.) 2023-10-30T12:26:57.023 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-30T12:26:57.071 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-30T12:26:57.457 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T12:26:57.816 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T12:26:57.959 INFO:teuthology.orchestra.run.smithi136.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T12:26:57.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2023-10-30T12:26:58.011 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-10-30T12:26:58.017 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2023-10-30T12:26:58.204 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2023-10-30T12:26:58.205 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2023-10-30T12:26:58.367 INFO:teuthology.orchestra.run.smithi136.stdout:iproute2 is already the newest version (5.15.0-1ubuntu2). 2023-10-30T12:26:58.368 INFO:teuthology.orchestra.run.smithi136.stdout:libtool-bin is already the newest version (2.4.6-15build2). 2023-10-30T12:26:58.368 INFO:teuthology.orchestra.run.smithi136.stdout:util-linux is already the newest version (2.37.2-4ubuntu3). 2023-10-30T12:26:58.368 INFO:teuthology.orchestra.run.smithi136.stdout:uuid-dev is already the newest version (2.37.2-4ubuntu3). 2023-10-30T12:26:58.368 INFO:teuthology.orchestra.run.smithi136.stdout:xfslibs-dev is already the newest version (5.13.0-1ubuntu2). 2023-10-30T12:26:58.368 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages were automatically installed and are no longer required: 2023-10-30T12:26:58.369 INFO:teuthology.orchestra.run.smithi136.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-10-30T12:26:58.369 INFO:teuthology.orchestra.run.smithi136.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-10-30T12:26:58.370 INFO:teuthology.orchestra.run.smithi136.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-10-30T12:26:58.370 INFO:teuthology.orchestra.run.smithi136.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-10-30T12:26:58.371 INFO:teuthology.orchestra.run.smithi136.stdout: python2.7-minimal 2023-10-30T12:26:58.371 INFO:teuthology.orchestra.run.smithi136.stdout:Use 'sudo apt autoremove' to remove them. 2023-10-30T12:26:58.371 INFO:teuthology.orchestra.run.smithi136.stdout:The following additional packages will be installed: 2023-10-30T12:26:58.371 INFO:teuthology.orchestra.run.smithi136.stdout: libbluetooth3 libfl-dev libfl2 libjson-perl libllvm14 liblzo2-2 2023-10-30T12:26:58.371 INFO:teuthology.orchestra.run.smithi136.stdout: libmbim-glib4 libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 2023-10-30T12:26:58.372 INFO:teuthology.orchestra.run.smithi136.stdout: libqmi-proxy libssl3 libteamdctl0 modemmanager network-manager-pptp 2023-10-30T12:26:58.372 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-14 postgresql-client-14 postgresql-client-common ppp pptp-linux 2023-10-30T12:26:58.372 INFO:teuthology.orchestra.run.smithi136.stdout: tcl tcl8.6 usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T12:26:58.372 INFO:teuthology.orchestra.run.smithi136.stdout:Suggested packages: 2023-10-30T12:26:58.372 INFO:teuthology.orchestra.run.smithi136.stdout: bison-doc flex-doc indent-doc libssl-doc avahi-autoipd libteam-utils 2023-10-30T12:26:58.373 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-doc postgresql-doc-14 tcl-tclreadline comgt wvdial wpagui 2023-10-30T12:26:58.373 INFO:teuthology.orchestra.run.smithi136.stdout: libengine-pkcs11-openssl 2023-10-30T12:26:58.407 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2023-10-30T12:26:58.408 INFO:teuthology.orchestra.run.smithi136.stdout: bison dump flex indent libaio-dev libbluetooth3 libelf-dev libfl-dev libfl2 2023-10-30T12:26:58.408 INFO:teuthology.orchestra.run.smithi136.stdout: libjson-perl libllvm14 liblzo2-2 libmbim-glib4 libmbim-proxy libmm-glib0 2023-10-30T12:26:58.409 INFO:teuthology.orchestra.run.smithi136.stdout: libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl-dev libteamdctl0 2023-10-30T12:26:58.409 INFO:teuthology.orchestra.run.smithi136.stdout: modemmanager network-manager network-manager-pptp postgresql postgresql-14 2023-10-30T12:26:58.409 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-client postgresql-client-14 postgresql-client-common 2023-10-30T12:26:58.409 INFO:teuthology.orchestra.run.smithi136.stdout: postgresql-common postgresql-contrib ppp pptp-linux tcl tcl8.6 2023-10-30T12:26:58.409 INFO:teuthology.orchestra.run.smithi136.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2023-10-30T12:26:58.410 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages will be upgraded: 2023-10-30T12:26:58.410 INFO:teuthology.orchestra.run.smithi136.stdout: libssl3 2023-10-30T12:26:58.457 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T12:26:58.462 INFO:teuthology.orchestra.run.smithi136.stdout:1 upgraded, 39 newly installed, 0 to remove and 259 not upgraded. 2023-10-30T12:26:58.462 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 54.6 MB of archives. 2023-10-30T12:26:58.462 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 202 MB of additional disk space will be used. 2023-10-30T12:26:58.462 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 flex amd64 2.6.4-8build2 [307 kB] 2023-10-30T12:26:58.562 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl3 amd64 3.0.2-0ubuntu1.12 [1,901 kB] 2023-10-30T12:26:58.623 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/main amd64 bison amd64 2:3.8.2+dfsg-1build1 [748 kB] 2023-10-30T12:26:58.636 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblzo2-2 amd64 2.10-2build3 [53.7 kB] 2023-10-30T12:26:58.638 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 dump amd64 0.4b47-3 [133 kB] 2023-10-30T12:26:58.640 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 indent amd64 2.2.12-1ubuntu0.22.04.1 [90.8 kB] 2023-10-30T12:26:58.642 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy/main amd64 libaio-dev amd64 0.3.112-13build1 [21.2 kB] 2023-10-30T12:26:58.642 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/main amd64 libbluetooth3 amd64 5.64-0ubuntu1 [86.9 kB] 2023-10-30T12:26:58.644 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 libelf-dev amd64 0.186-1build1 [64.4 kB] 2023-10-30T12:26:58.646 INFO:teuthology.orchestra.run.smithi136.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl2 amd64 2.6.4-8build2 [10.7 kB] 2023-10-30T12:26:58.646 INFO:teuthology.orchestra.run.smithi136.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 libfl-dev amd64 2.6.4-8build2 [6,236 B] 2023-10-30T12:26:58.646 INFO:teuthology.orchestra.run.smithi136.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjson-perl all 4.04000-1 [81.8 kB] 2023-10-30T12:26:58.661 INFO:teuthology.orchestra.run.smithi136.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libllvm14 amd64 1:14.0.0-1ubuntu1.1 [24.0 MB] 2023-10-30T12:26:58.995 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package bison. 2023-10-30T12:26:59.019 INFO:teuthology.orchestra.run.smithi136.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-glib4 amd64 1.28.0-1~ubuntu20.04.1 [191 kB] 2023-10-30T12:26:59.022 INFO:teuthology.orchestra.run.smithi136.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmbim-proxy amd64 1.28.0-1~ubuntu20.04.1 [6,130 B] 2023-10-30T12:26:59.022 INFO:teuthology.orchestra.run.smithi136.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libmm-glib0 amd64 1.20.0-1~ubuntu22.04.2 [263 kB] 2023-10-30T12:26:59.025 INFO:teuthology.orchestra.run.smithi136.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 libndp0 amd64 1.8-0ubuntu3 [12.3 kB] 2023-10-30T12:26:59.025 INFO:teuthology.orchestra.run.smithi136.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnm0 amd64 1.36.6-0ubuntu2 [456 kB] 2023-10-30T12:26:59.031 INFO:teuthology.orchestra.run.smithi136.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpq5 amd64 14.9-0ubuntu0.22.04.1 [141 kB] 2023-10-30T12:26:59.033 INFO:teuthology.orchestra.run.smithi136.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-glib5 amd64 1.32.0-1ubuntu0.22.04.1 [772 kB] 2023-10-30T12:26:59.042 INFO:teuthology.orchestra.run.smithi136.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libqmi-proxy amd64 1.32.0-1ubuntu0.22.04.1 [6,072 B] 2023-10-30T12:26:59.042 INFO:teuthology.orchestra.run.smithi136.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libssl-dev amd64 3.0.2-0ubuntu1.12 [2,373 kB] 2023-10-30T12:26:59.067 INFO:teuthology.orchestra.run.smithi125.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 139924 files and directories currently installed.) 2023-10-30T12:26:59.070 INFO:teuthology.orchestra.run.smithi136.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libteamdctl0 amd64 1.31-1build2 [12.6 kB] 2023-10-30T12:26:59.070 INFO:teuthology.orchestra.run.smithi136.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 modemmanager amd64 1.20.0-1~ubuntu22.04.2 [1,094 kB] 2023-10-30T12:26:59.071 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-30T12:26:59.081 INFO:teuthology.orchestra.run.smithi136.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 network-manager amd64 1.36.6-0ubuntu2 [2,203 kB] 2023-10-30T12:26:59.101 INFO:teuthology.orchestra.run.smithi136.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 ppp amd64 2.4.9-1+1ubuntu3 [363 kB] 2023-10-30T12:26:59.104 INFO:teuthology.orchestra.run.smithi136.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 pptp-linux amd64 1.10.0-1build3 [42.5 kB] 2023-10-30T12:26:59.105 INFO:teuthology.orchestra.run.smithi136.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 network-manager-pptp amd64 1.2.10-1 [30.9 kB] 2023-10-30T12:26:59.105 INFO:teuthology.orchestra.run.smithi136.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client-common all 238 [29.6 kB] 2023-10-30T12:26:59.106 INFO:teuthology.orchestra.run.smithi136.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-client-14 amd64 14.9-0ubuntu0.22.04.1 [1,222 kB] 2023-10-30T12:26:59.114 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T12:26:59.115 INFO:teuthology.orchestra.run.smithi136.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-common all 238 [169 kB] 2023-10-30T12:26:59.117 INFO:teuthology.orchestra.run.smithi136.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 postgresql-14 amd64 14.9-0ubuntu0.22.04.1 [16.1 MB] 2023-10-30T12:26:59.345 INFO:teuthology.orchestra.run.smithi136.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql all 14+238 [3,288 B] 2023-10-30T12:26:59.345 INFO:teuthology.orchestra.run.smithi136.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-client all 14+238 [3,292 B] 2023-10-30T12:26:59.345 INFO:teuthology.orchestra.run.smithi136.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 postgresql-contrib all 14+238 [3,292 B] 2023-10-30T12:26:59.346 INFO:teuthology.orchestra.run.smithi136.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl8.6 amd64 8.6.12+dfsg-1build1 [15.0 kB] 2023-10-30T12:26:59.346 INFO:teuthology.orchestra.run.smithi136.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 tcl amd64 8.6.11+1build2 [4,678 B] 2023-10-30T12:26:59.346 INFO:teuthology.orchestra.run.smithi136.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 wpasupplicant amd64 2:2.10-6ubuntu2 [1,481 kB] 2023-10-30T12:26:59.357 INFO:teuthology.orchestra.run.smithi136.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch-data all 20191128-4 [33.2 kB] 2023-10-30T12:26:59.357 INFO:teuthology.orchestra.run.smithi136.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 usb-modeswitch amd64 2.6.1-3ubuntu2 [46.0 kB] 2023-10-30T12:26:59.506 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-30T12:26:59.521 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-30T12:26:59.564 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T12:26:59.753 INFO:teuthology.orchestra.run.smithi136.stdout:Preconfiguring packages ... 2023-10-30T12:26:59.916 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 54.6 MB in 1s (58.0 MB/s) 2023-10-30T12:26:59.942 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package dump. 2023-10-30T12:26:59.956 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-30T12:27:00.001 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking dump (0.4b47-3) ... 2023-10-30T12:27:00.056 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package flex. 2023-10-30T12:27:00.133 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137425 files and directories currently installed.) 2023-10-30T12:27:00.138 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../flex_2.6.4-8build2_amd64.deb ... 2023-10-30T12:27:00.179 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking flex (2.6.4-8build2) ... 2023-10-30T12:27:00.279 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package indent. 2023-10-30T12:27:00.293 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:00.337 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T12:27:00.573 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-30T12:27:00.587 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-30T12:27:00.631 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T12:27:00.798 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../libssl3_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T12:27:00.951 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-30T12:27:00.966 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-30T12:27:00.977 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libssl3:amd64 (3.0.2-0ubuntu1.12) over (3.0.2-0ubuntu1.7) ... 2023-10-30T12:27:01.026 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T12:27:01.287 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-30T12:27:01.301 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-30T12:27:01.345 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T12:27:01.609 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libssl3:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T12:27:01.682 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-30T12:27:01.696 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-30T12:27:01.740 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T12:27:01.976 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-30T12:27:01.990 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-30T12:27:02.034 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T12:27:02.145 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package bison. 2023-10-30T12:27:02.234 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137498 files and directories currently installed.) 2023-10-30T12:27:02.239 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../00-bison_2%3a3.8.2+dfsg-1build1_amd64.deb ... 2023-10-30T12:27:02.286 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libjson-perl. 2023-10-30T12:27:02.289 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T12:27:02.301 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-30T12:27:02.362 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-30T12:27:02.656 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-30T12:27:02.671 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-30T12:27:02.697 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblzo2-2:amd64. 2023-10-30T12:27:02.720 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../01-liblzo2-2_2.10-2build3_amd64.deb ... 2023-10-30T12:27:02.789 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T12:27:02.832 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T12:27:03.101 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package dump. 2023-10-30T12:27:03.123 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../02-dump_0.4b47-3_amd64.deb ... 2023-10-30T12:27:03.167 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking dump (0.4b47-3) ... 2023-10-30T12:27:03.496 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package indent. 2023-10-30T12:27:03.518 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../03-indent_2.2.12-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:03.562 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T12:27:03.831 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libaio-dev:amd64. 2023-10-30T12:27:03.843 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../04-libaio-dev_0.3.112-13build1_amd64.deb ... 2023-10-30T12:27:03.880 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T12:27:04.169 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-30T12:27:04.185 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T12:27:04.227 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:27:04.243 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libbluetooth3:amd64. 2023-10-30T12:27:04.264 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../05-libbluetooth3_5.64-0ubuntu1_amd64.deb ... 2023-10-30T12:27:04.309 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T12:27:04.463 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-30T12:27:04.478 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T12:27:04.521 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:27:04.536 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libelf-dev:amd64. 2023-10-30T12:27:04.549 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../06-libelf-dev_0.186-1build1_amd64.deb ... 2023-10-30T12:27:04.578 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T12:27:04.815 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-30T12:27:04.831 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T12:27:04.874 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:27:04.932 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libfl2:amd64. 2023-10-30T12:27:04.953 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../07-libfl2_2.6.4-8build2_amd64.deb ... 2023-10-30T12:27:05.006 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T12:27:05.210 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-30T12:27:05.225 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-30T12:27:05.267 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libfl-dev:amd64. 2023-10-30T12:27:05.280 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../08-libfl-dev_2.6.4-8build2_amd64.deb ... 2023-10-30T12:27:05.284 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T12:27:05.317 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T12:27:05.561 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libjson-perl. 2023-10-30T12:27:05.574 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../09-libjson-perl_4.04000-1_all.deb ... 2023-10-30T12:27:05.611 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libjson-perl (4.04000-1) ... 2023-10-30T12:27:05.612 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-30T12:27:05.627 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T12:27:05.670 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T12:27:05.931 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libllvm14:amd64. 2023-10-30T12:27:05.943 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../10-libllvm14_1%3a14.0.0-1ubuntu1.1_amd64.deb ... 2023-10-30T12:27:05.972 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T12:27:05.973 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-30T12:27:05.988 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:06.023 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:06.376 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-30T12:27:06.392 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:06.434 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:27:06.737 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-30T12:27:06.752 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:06.795 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:27:07.039 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-30T12:27:07.054 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T12:27:07.089 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T12:27:07.559 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-30T12:27:07.573 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-30T12:27:07.617 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T12:27:07.734 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2023-10-30T12:27:07.756 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../11-libmbim-glib4_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T12:27:07.801 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:27:08.046 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package modemmanager. 2023-10-30T12:27:08.060 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T12:27:08.103 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libmbim-proxy. 2023-10-30T12:27:08.104 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:27:08.115 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../12-libmbim-proxy_1.28.0-1~ubuntu20.04.1_amd64.deb ... 2023-10-30T12:27:08.153 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:27:08.480 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libmm-glib0:amd64. 2023-10-30T12:27:08.502 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../13-libmm-glib0_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T12:27:08.547 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:27:08.710 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package network-manager. 2023-10-30T12:27:08.724 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T12:27:08.776 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-30T12:27:08.916 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libndp0:amd64. 2023-10-30T12:27:08.938 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../14-libndp0_1.8-0ubuntu3_amd64.deb ... 2023-10-30T12:27:08.983 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T12:27:09.311 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libnm0:amd64. 2023-10-30T12:27:09.332 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../15-libnm0_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T12:27:09.377 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T12:27:09.408 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package ppp. 2023-10-30T12:27:09.422 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-30T12:27:09.755 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libpq5:amd64. 2023-10-30T12:27:09.770 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T12:27:09.777 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../16-libpq5_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:09.821 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:10.080 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package pptp-linux. 2023-10-30T12:27:10.093 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-30T12:27:10.129 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-30T12:27:10.141 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2023-10-30T12:27:10.163 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../17-libqmi-glib5_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:10.215 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:27:10.349 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-30T12:27:10.363 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-30T12:27:10.407 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-30T12:27:10.526 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqmi-proxy. 2023-10-30T12:27:10.549 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../18-libqmi-proxy_1.32.0-1ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:10.593 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:27:10.726 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-30T12:27:10.740 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-30T12:27:10.785 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking postgresql-client-common (238) ... 2023-10-30T12:27:10.837 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libssl-dev:amd64. 2023-10-30T12:27:10.855 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../19-libssl-dev_3.0.2-0ubuntu1.12_amd64.deb ... 2023-10-30T12:27:10.917 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T12:27:11.097 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-30T12:27:11.111 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:11.155 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:11.440 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libteamdctl0:amd64. 2023-10-30T12:27:11.462 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../20-libteamdctl0_1.31-1build2_amd64.deb ... 2023-10-30T12:27:11.507 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T12:27:11.727 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package postgresql-common. 2023-10-30T12:27:11.741 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-30T12:27:11.802 INFO:teuthology.orchestra.run.smithi125.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-30T12:27:11.833 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking postgresql-common (238) ... 2023-10-30T12:27:11.944 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package modemmanager. 2023-10-30T12:27:11.966 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../21-modemmanager_1.20.0-1~ubuntu22.04.2_amd64.deb ... 2023-10-30T12:27:12.010 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:27:12.334 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package postgresql-14. 2023-10-30T12:27:12.348 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:12.394 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:12.691 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package network-manager. 2023-10-30T12:27:12.714 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../22-network-manager_1.36.6-0ubuntu2_amd64.deb ... 2023-10-30T12:27:12.765 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking network-manager (1.36.6-0ubuntu2) ... 2023-10-30T12:27:13.381 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ppp. 2023-10-30T12:27:13.404 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../23-ppp_2.4.9-1+1ubuntu3_amd64.deb ... 2023-10-30T12:27:13.724 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T12:27:14.053 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package pptp-linux. 2023-10-30T12:27:14.074 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../24-pptp-linux_1.10.0-1build3_amd64.deb ... 2023-10-30T12:27:14.119 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking pptp-linux (1.10.0-1build3) ... 2023-10-30T12:27:14.155 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package postgresql. 2023-10-30T12:27:14.170 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-30T12:27:14.227 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking postgresql (14+238) ... 2023-10-30T12:27:14.371 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package network-manager-pptp. 2023-10-30T12:27:14.384 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../25-network-manager-pptp_1.2.10-1_amd64.deb ... 2023-10-30T12:27:14.421 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking network-manager-pptp (1.2.10-1) ... 2023-10-30T12:27:14.574 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package postgresql-client. 2023-10-30T12:27:14.588 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-30T12:27:14.632 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking postgresql-client (14+238) ... 2023-10-30T12:27:14.774 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package postgresql-client-common. 2023-10-30T12:27:14.796 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../26-postgresql-client-common_238_all.deb ... 2023-10-30T12:27:14.849 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking postgresql-client-common (238) ... 2023-10-30T12:27:14.876 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-30T12:27:14.890 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-30T12:27:14.934 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-30T12:27:15.153 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package postgresql-client-14. 2023-10-30T12:27:15.174 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../27-postgresql-client-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:15.219 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:15.245 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package tcl8.6. 2023-10-30T12:27:15.259 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-30T12:27:15.295 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T12:27:15.573 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package tcl. 2023-10-30T12:27:15.587 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-30T12:27:15.635 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-30T12:27:15.841 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package postgresql-common. 2023-10-30T12:27:15.854 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../28-postgresql-common_238_all.deb ... 2023-10-30T12:27:15.908 INFO:teuthology.orchestra.run.smithi136.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2023-10-30T12:27:15.932 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking postgresql-common (238) ... 2023-10-30T12:27:15.984 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package wpasupplicant. 2023-10-30T12:27:15.998 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-30T12:27:16.042 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T12:27:16.371 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-30T12:27:16.385 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-30T12:27:16.417 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package postgresql-14. 2023-10-30T12:27:16.438 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../29-postgresql-14_14.9-0ubuntu0.22.04.1_amd64.deb ... 2023-10-30T12:27:16.446 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-30T12:27:16.486 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:16.808 INFO:teuthology.orchestra.run.smithi125.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-30T12:27:16.822 INFO:teuthology.orchestra.run.smithi125.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-30T12:27:16.865 INFO:teuthology.orchestra.run.smithi125.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T12:27:17.134 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up postgresql-client-common (238) ... 2023-10-30T12:27:17.341 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-30T12:27:17.467 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T12:27:17.584 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:17.685 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T12:27:17.836 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T12:27:17.971 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:18.656 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package postgresql. 2023-10-30T12:27:18.669 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../30-postgresql_14+238_all.deb ... 2023-10-30T12:27:18.706 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking postgresql (14+238) ... 2023-10-30T12:27:18.950 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package postgresql-client. 2023-10-30T12:27:18.962 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../31-postgresql-client_14+238_all.deb ... 2023-10-30T12:27:18.967 INFO:teuthology.orchestra.run.smithi125.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-30T12:27:19.025 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking postgresql-client (14+238) ... 2023-10-30T12:27:19.059 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T12:27:19.244 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package postgresql-contrib. 2023-10-30T12:27:19.257 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../32-postgresql-contrib_14+238_all.deb ... 2023-10-30T12:27:19.294 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking postgresql-contrib (14+238) ... 2023-10-30T12:27:19.454 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T12:27:19.455 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T12:27:19.613 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package tcl8.6. 2023-10-30T12:27:19.635 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../33-tcl8.6_8.6.12+dfsg-1build1_amd64.deb ... 2023-10-30T12:27:19.680 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T12:27:19.999 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package tcl. 2023-10-30T12:27:20.014 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../34-tcl_8.6.11+1build2_amd64.deb ... 2023-10-30T12:27:20.062 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking tcl (8.6.11+1build2) ... 2023-10-30T12:27:20.154 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T12:27:20.272 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T12:27:20.381 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-30T12:27:20.386 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package wpasupplicant. 2023-10-30T12:27:20.408 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../35-wpasupplicant_2%3a2.10-6ubuntu2_amd64.deb ... 2023-10-30T12:27:20.452 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T12:27:20.507 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T12:27:20.633 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T12:27:20.759 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T12:27:20.832 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package usb-modeswitch-data. 2023-10-30T12:27:20.854 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../36-usb-modeswitch-data_20191128-4_all.deb ... 2023-10-30T12:27:20.885 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T12:27:20.899 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking usb-modeswitch-data (20191128-4) ... 2023-10-30T12:27:21.294 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package usb-modeswitch. 2023-10-30T12:27:21.316 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../37-usb-modeswitch_2.6.1-3ubuntu2_amd64.deb ... 2023-10-30T12:27:21.384 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T12:27:21.515 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T12:27:21.632 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up dump (0.4b47-3) ... 2023-10-30T12:27:21.669 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up postgresql-client-common (238) ... 2023-10-30T12:27:21.727 INFO:teuthology.orchestra.run.smithi125.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-30T12:27:21.792 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:27:21.867 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up flex (2.6.4-8build2) ... 2023-10-30T12:27:21.943 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T12:27:22.035 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up tcl8.6 (8.6.12+dfsg-1build1) ... 2023-10-30T12:27:22.069 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-30T12:27:22.161 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libpq5:amd64 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:22.204 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-30T12:27:22.304 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblzo2-2:amd64 (2.10-2build3) ... 2023-10-30T12:27:22.330 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T12:27:22.411 INFO:teuthology.orchestra.run.smithi125.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-30T12:27:22.455 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libaio-dev:amd64 (0.3.112-13build1) ... 2023-10-30T12:27:22.473 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T12:27:22.574 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T12:27:22.581 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up postgresql-client-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:22.775 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T12:27:22.901 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:27:23.010 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2023-10-30T12:27:23.027 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-30T12:27:23.087 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up wpasupplicant (2:2.10-6ubuntu2) ... 2023-10-30T12:27:23.195 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up postgresql-client (14+238) ... 2023-10-30T12:27:23.338 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-30T12:27:23.465 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T12:27:23.465 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2023-10-30T12:27:23.891 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:23.891 INFO:teuthology.orchestra.run.smithi125.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-30T12:27:23.891 INFO:teuthology.orchestra.run.smithi125.stdout:which means they are currently configured by ifupdown: 2023-10-30T12:27:23.891 INFO:teuthology.orchestra.run.smithi125.stdout:- enp3s0f1 2023-10-30T12:27:23.892 INFO:teuthology.orchestra.run.smithi125.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-30T12:27:23.892 INFO:teuthology.orchestra.run.smithi125.stdout:remove their configuration from /etc/network/interfaces. 2023-10-30T12:27:23.892 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:23.973 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libteamdctl0:amd64 (1.31-1build2) ... 2023-10-30T12:27:24.026 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-30T12:27:24.100 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libssl-dev:amd64 (3.0.2-0ubuntu1.12) ... 2023-10-30T12:27:24.251 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up usb-modeswitch-data (20191128-4) ... 2023-10-30T12:27:24.377 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libelf-dev:amd64 (0.186-1build1) ... 2023-10-30T12:27:24.418 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-30T12:27:24.503 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libfl2:amd64 (2.6.4-8build2) ... 2023-10-30T12:27:24.629 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libnm0:amd64 (1.36.6-0ubuntu2) ... 2023-10-30T12:27:24.755 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ppp (2.4.9-1+1ubuntu3) ... 2023-10-30T12:27:24.822 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-30T12:27:25.445 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libllvm14:amd64 (1:14.0.0-1ubuntu1.1) ... 2023-10-30T12:27:25.524 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:27:25.571 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up dump (0.4b47-3) ... 2023-10-30T12:27:25.658 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T12:27:25.690 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2023-10-30T12:27:25.764 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libmm-glib0:amd64 (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:27:25.767 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-30T12:27:25.865 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libbluetooth3:amd64 (5.64-0ubuntu1) ... 2023-10-30T12:27:25.885 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up postgresql-common (238) ... 2023-10-30T12:27:25.992 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up tcl (8.6.11+1build2) ... 2023-10-30T12:27:26.126 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libjson-perl (4.04000-1) ... 2023-10-30T12:27:26.252 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up bison (2:3.8.2+dfsg-1build1) ... 2023-10-30T12:27:26.342 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2023-10-30T12:27:26.437 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up indent (2.2.12-1ubuntu0.22.04.1) ... 2023-10-30T12:27:26.563 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up usb-modeswitch (2.6.1-3ubuntu2) ... 2023-10-30T12:27:26.731 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libndp0:amd64 (1.8-0ubuntu3) ... 2023-10-30T12:27:26.798 INFO:teuthology.orchestra.run.smithi125.stdout:Adding user postgres to group ssl-cert 2023-10-30T12:27:26.841 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libmbim-glib4:amd64 (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:27:26.967 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up pptp-linux (1.10.0-1build3) ... 2023-10-30T12:27:27.054 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:27.054 INFO:teuthology.orchestra.run.smithi125.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-30T12:27:27.160 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up postgresql-client (14+238) ... 2023-10-30T12:27:27.287 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up network-manager (1.36.6-0ubuntu2) ... 2023-10-30T12:27:27.333 INFO:teuthology.orchestra.run.smithi125.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-30T12:27:27.333 INFO:teuthology.orchestra.run.smithi125.stdout:Removing obsolete dictionary files: 2023-10-30T12:27:27.798 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:27.799 INFO:teuthology.orchestra.run.smithi136.stdout:The following network interfaces were found in /etc/network/interfaces 2023-10-30T12:27:27.799 INFO:teuthology.orchestra.run.smithi136.stdout:which means they are currently configured by ifupdown: 2023-10-30T12:27:27.799 INFO:teuthology.orchestra.run.smithi136.stdout:- enp3s0f1 2023-10-30T12:27:27.799 INFO:teuthology.orchestra.run.smithi136.stdout:If you want to manage those interfaces with NetworkManager instead 2023-10-30T12:27:27.799 INFO:teuthology.orchestra.run.smithi136.stdout:remove their configuration from /etc/network/interfaces. 2023-10-30T12:27:27.800 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:27.928 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2023-10-30T12:27:28.066 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-30T12:27:28.259 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2023-10-30T12:27:28.519 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:28.616 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2023-10-30T12:27:28.860 INFO:teuthology.orchestra.run.smithi125.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-30T12:27:28.895 INFO:teuthology.orchestra.run.smithi125.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-30T12:27:28.955 INFO:teuthology.orchestra.run.smithi125.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-30T12:27:28.958 INFO:teuthology.orchestra.run.smithi125.stdout:This user must also own the server process. 2023-10-30T12:27:28.958 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:28.958 INFO:teuthology.orchestra.run.smithi125.stdout:The database cluster will be initialized with locale "en_US". 2023-10-30T12:27:28.958 INFO:teuthology.orchestra.run.smithi125.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-30T12:27:28.959 INFO:teuthology.orchestra.run.smithi125.stdout:The default text search configuration will be set to "english". 2023-10-30T12:27:28.959 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:28.959 INFO:teuthology.orchestra.run.smithi125.stdout:Data page checksums are disabled. 2023-10-30T12:27:28.959 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:28.959 INFO:teuthology.orchestra.run.smithi125.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-30T12:27:28.960 INFO:teuthology.orchestra.run.smithi125.stdout:creating subdirectories ... ok 2023-10-30T12:27:28.961 INFO:teuthology.orchestra.run.smithi125.stdout:selecting dynamic shared memory implementation ... posix 2023-10-30T12:27:28.980 INFO:teuthology.orchestra.run.smithi125.stdout:selecting default max_connections ... 100 2023-10-30T12:27:29.015 INFO:teuthology.orchestra.run.smithi125.stdout:selecting default shared_buffers ... 128MB 2023-10-30T12:27:29.029 INFO:teuthology.orchestra.run.smithi125.stdout:selecting default time zone ... Etc/UTC 2023-10-30T12:27:29.030 INFO:teuthology.orchestra.run.smithi125.stdout:creating configuration files ... ok 2023-10-30T12:27:29.181 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libmbim-proxy (1.28.0-1~ubuntu20.04.1) ... 2023-10-30T12:27:29.199 INFO:teuthology.orchestra.run.smithi125.stdout:running bootstrap script ... ok 2023-10-30T12:27:29.366 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libfl-dev:amd64 (2.6.4-8build2) ... 2023-10-30T12:27:29.492 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up network-manager-pptp (1.2.10-1) ... 2023-10-30T12:27:29.627 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up postgresql-common (238) ... 2023-10-30T12:27:29.851 INFO:teuthology.orchestra.run.smithi125.stdout:performing post-bootstrap initialization ... ok 2023-10-30T12:27:30.395 INFO:teuthology.orchestra.run.smithi125.stdout:syncing data to disk ... ok 2023-10-30T12:27:30.496 INFO:teuthology.orchestra.run.smithi136.stdout:Adding user postgres to group ssl-cert 2023-10-30T12:27:30.685 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:30.685 INFO:teuthology.orchestra.run.smithi136.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2023-10-30T12:27:30.847 INFO:teuthology.orchestra.run.smithi136.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2023-10-30T12:27:30.847 INFO:teuthology.orchestra.run.smithi136.stdout:Removing obsolete dictionary files: 2023-10-30T12:27:30.937 INFO:teuthology.orchestra.run.smithi125.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-30T12:27:31.408 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2023-10-30T12:27:31.804 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up postgresql-14 (14.9-0ubuntu0.22.04.1) ... 2023-10-30T12:27:32.141 INFO:teuthology.orchestra.run.smithi136.stdout:Creating new PostgreSQL cluster 14/main ... 2023-10-30T12:27:32.154 INFO:teuthology.orchestra.run.smithi136.stdout:/usr/lib/postgresql/14/bin/initdb -D /var/lib/postgresql/14/main --auth-local peer --auth-host scram-sha-256 --no-instructions 2023-10-30T12:27:32.203 INFO:teuthology.orchestra.run.smithi136.stdout:The files belonging to this database system will be owned by user "postgres". 2023-10-30T12:27:32.203 INFO:teuthology.orchestra.run.smithi136.stdout:This user must also own the server process. 2023-10-30T12:27:32.203 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:32.203 INFO:teuthology.orchestra.run.smithi136.stdout:The database cluster will be initialized with locale "en_US". 2023-10-30T12:27:32.203 INFO:teuthology.orchestra.run.smithi136.stdout:The default database encoding has accordingly been set to "LATIN1". 2023-10-30T12:27:32.204 INFO:teuthology.orchestra.run.smithi136.stdout:The default text search configuration will be set to "english". 2023-10-30T12:27:32.204 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:32.204 INFO:teuthology.orchestra.run.smithi136.stdout:Data page checksums are disabled. 2023-10-30T12:27:32.204 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:32.204 INFO:teuthology.orchestra.run.smithi136.stdout:fixing permissions on existing directory /var/lib/postgresql/14/main ... ok 2023-10-30T12:27:32.205 INFO:teuthology.orchestra.run.smithi136.stdout:creating subdirectories ... ok 2023-10-30T12:27:32.205 INFO:teuthology.orchestra.run.smithi136.stdout:selecting dynamic shared memory implementation ... posix 2023-10-30T12:27:32.219 INFO:teuthology.orchestra.run.smithi136.stdout:selecting default max_connections ... 100 2023-10-30T12:27:32.240 INFO:teuthology.orchestra.run.smithi136.stdout:selecting default shared_buffers ... 128MB 2023-10-30T12:27:32.249 INFO:teuthology.orchestra.run.smithi136.stdout:selecting default time zone ... Etc/UTC 2023-10-30T12:27:32.250 INFO:teuthology.orchestra.run.smithi136.stdout:creating configuration files ... ok 2023-10-30T12:27:32.365 INFO:teuthology.orchestra.run.smithi136.stdout:running bootstrap script ... ok 2023-10-30T12:27:33.019 INFO:teuthology.orchestra.run.smithi136.stdout:performing post-bootstrap initialization ... ok 2023-10-30T12:27:33.829 INFO:teuthology.orchestra.run.smithi136.stdout:syncing data to disk ... ok 2023-10-30T12:27:34.010 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:27:34.144 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-30T12:27:34.245 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/share/postgresql/14/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2023-10-30T12:27:34.270 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up postgresql (14+238) ... 2023-10-30T12:27:34.413 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:27:34.539 INFO:teuthology.orchestra.run.smithi125.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:27:34.797 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T12:27:34.797 INFO:teuthology.orchestra.run.smithi125.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T12:27:35.524 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-30T12:27:35.843 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T12:27:36.020 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T12:27:37.161 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqmi-glib5:amd64 (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:27:37.304 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up postgresql-contrib (14+238) ... 2023-10-30T12:27:37.447 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-30T12:27:37.463 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up postgresql (14+238) ... 2023-10-30T12:27:37.547 INFO:teuthology.orchestra.run.smithi125.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-30T12:27:37.589 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqmi-proxy (1.32.0-1ubuntu0.22.04.1) ... 2023-10-30T12:27:37.715 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up modemmanager (1.20.0-1~ubuntu22.04.2) ... 2023-10-30T12:27:37.969 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T12:27:37.970 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2023-10-30T12:27:38.215 INFO:teuthology.orchestra.run.smithi125.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T12:27:38.220 DEBUG:teuthology.parallel:result is None 2023-10-30T12:27:38.576 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2023-10-30T12:27:38.920 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-10-30T12:27:39.072 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-10-30T12:27:40.435 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for dbus (1.12.20-2ubuntu4.1) ... 2023-10-30T12:27:40.534 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-10-30T12:27:41.176 INFO:teuthology.orchestra.run.smithi136.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-10-30T12:27:41.178 DEBUG:teuthology.parallel:result is None 2023-10-30T12:27:41.179 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:27:41.428 DEBUG:teuthology.orchestra.run.smithi125:> dpkg-query -W -f '${Version}' ceph 2023-10-30T12:27:41.452 INFO:teuthology.orchestra.run.smithi125.stdout:18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:27:41.452 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:27:41.453 INFO:teuthology.task.install:The correct ceph version 18.0.0-6835-g00c1c13c-1jammy is installed. 2023-10-30T12:27:41.453 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:27:41.687 DEBUG:teuthology.orchestra.run.smithi136:> dpkg-query -W -f '${Version}' ceph 2023-10-30T12:27:41.707 INFO:teuthology.orchestra.run.smithi136.stdout:18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:27:41.707 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6835-g00c1c13c-1jammy 2023-10-30T12:27:41.707 INFO:teuthology.task.install:The correct ceph version 18.0.0-6835-g00c1c13c-1jammy is installed. 2023-10-30T12:27:41.708 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-30T12:27:41.709 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:27:41.709 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T12:27:41.731 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:27:41.732 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-30T12:27:41.761 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-30T12:27:41.761 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:27:41.761 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T12:27:41.796 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T12:27:41.863 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:27:41.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2023-10-30T12:27:41.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-30T12:27:41.931 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-30T12:27:41.932 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:27:41.932 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T12:27:41.953 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T12:27:42.015 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:27:42.015 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-30T12:27:42.028 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-30T12:27:42.083 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-30T12:27:42.083 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:27:42.084 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T12:27:42.104 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T12:27:42.167 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:27:42.167 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2023-10-30T12:27:42.181 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-30T12:27:42.235 INFO:teuthology.run_tasks:Running task ceph... 2023-10-30T12:27:42.294 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-30T12:27:42.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 777 /var/log/ceph 2023-10-30T12:27:42.299 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /var/log/ceph 2023-10-30T12:27:42.316 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-30T12:27:42.317 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T12:27:42.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-30T12:27:42.379 INFO:tasks.ceph:Creating extra log directories... 2023-10-30T12:27:42.379 DEBUG:teuthology.orchestra.run.smithi125:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T12:27:42.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-30T12:27:42.450 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-30T12:27:42.450 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-30T12:27:42.450 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-30T12:27:42.451 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:27:42.451 DEBUG:teuthology.orchestra.run.smithi125:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-30T12:27:42.500 DEBUG:teuthology.orchestra.run.smithi125:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-30T12:27:42.593 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:27:42.593 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-30T12:27:42.599 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-30T12:27:42.677 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-30T12:27:42.678 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-30T12:27:42.678 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019', 'branch': 'wip-lusov-balance-after-stop', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}}', 'email': None, 'first_in_suite': False, 'job_id': '7441019', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-lusov-balance-after-stop'}, 'ceph': {'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, 'kclient': {'syntax': 'v1'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-lusov-balance-after-stop', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, 'owner': 'scheduled_leonidus@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 8397, 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'sleep_before_teardown': 0, 'subset': '464/1024', 'suite': 'fs', 'suite_branch': 'wip-lusov-balance-after-stop', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi125.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAiFxaPW/fAIFYfAZ7S0QaSr7et4p16btxGCug/BeJSMAcBjUFWpEFIiQexh7R1wVQLHrWxl1wi22tuK+iNge+g=', 'smithi136.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHFv/5e/H1nftak/kNE7BpioBtAEeujqPjwXW0WI6M9KclRCZ1nNUcD4HU42sPUWOO2qfq1wGIW0kzp5YjMoVs4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'Replacing daemon mds'], 'sha1': '00c1c13cea21a686b046584596fbeac5315a7d71', 'cluster': 'ceph'}}, {'kclient': None}, {'mds_thrash': None}, {'workunit': {'clients': {'all': ['fs/snaps']}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '6899cd26fceddb2fec83dc1a1349394b28c8998e', 'timestamp': '2023-10-30_06:34:16', 'tube': 'smithi', 'user': 'leonidus', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.770726'} 2023-10-30T12:27:42.679 DEBUG:teuthology.orchestra.run.smithi125:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T12:27:42.683 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-30T12:27:42.720 DEBUG:teuthology.orchestra.run.smithi125:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T12:27:42.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/run/ceph 2023-10-30T12:27:42.776 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:27:42.776 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T12:27:42.807 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T12:27:42.807 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_1 2023-10-30T12:27:42.864 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T12:27:42.864 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:42.864 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 5h/5d Inode: 891 Links: 1 2023-10-30T12:27:42.864 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:42.864 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2023-10-30 12:21:26.563170853 +0000 2023-10-30T12:27:42.865 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2023-10-30 12:21:26.175174319 +0000 2023-10-30T12:27:42.865 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2023-10-30 12:21:26.175174319 +0000 2023-10-30T12:27:42.865 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: 2023-10-30 12:21:26.175174319 +0000 2023-10-30T12:27:42.865 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T12:27:42.929 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2023-10-30T12:27:42.930 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2023-10-30T12:27:42.930 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000368681 s, 1.4 MB/s 2023-10-30T12:27:42.931 DEBUG:teuthology.orchestra.run.smithi125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T12:27:42.985 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_2 2023-10-30T12:27:43.039 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T12:27:43.039 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:43.040 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 5h/5d Inode: 906 Links: 1 2023-10-30T12:27:43.040 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:43.040 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2023-10-30 12:21:27.203165137 +0000 2023-10-30T12:27:43.040 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2023-10-30 12:21:26.815168603 +0000 2023-10-30T12:27:43.040 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2023-10-30 12:21:26.815168603 +0000 2023-10-30T12:27:43.041 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: 2023-10-30 12:21:26.815168603 +0000 2023-10-30T12:27:43.041 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T12:27:43.106 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2023-10-30T12:27:43.106 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2023-10-30T12:27:43.106 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000638798 s, 802 kB/s 2023-10-30T12:27:43.108 DEBUG:teuthology.orchestra.run.smithi125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T12:27:43.165 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_3 2023-10-30T12:27:43.216 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T12:27:43.216 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:43.216 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 5h/5d Inode: 924 Links: 1 2023-10-30T12:27:43.216 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:43.217 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2023-10-30 12:21:27.847159384 +0000 2023-10-30T12:27:43.217 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2023-10-30 12:21:27.463162815 +0000 2023-10-30T12:27:43.217 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2023-10-30 12:21:27.463162815 +0000 2023-10-30T12:27:43.217 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: 2023-10-30 12:21:27.463162815 +0000 2023-10-30T12:27:43.217 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T12:27:43.278 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2023-10-30T12:27:43.278 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2023-10-30T12:27:43.278 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000390423 s, 1.3 MB/s 2023-10-30T12:27:43.280 DEBUG:teuthology.orchestra.run.smithi125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T12:27:43.333 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_4 2023-10-30T12:27:43.383 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T12:27:43.384 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:43.384 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 5h/5d Inode: 942 Links: 1 2023-10-30T12:27:43.384 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:43.384 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2023-10-30 12:21:28.583152811 +0000 2023-10-30T12:27:43.384 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2023-10-30 12:21:28.083157277 +0000 2023-10-30T12:27:43.385 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2023-10-30 12:21:28.083157277 +0000 2023-10-30T12:27:43.385 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: 2023-10-30 12:21:28.083157277 +0000 2023-10-30T12:27:43.385 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T12:27:43.446 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2023-10-30T12:27:43.446 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2023-10-30T12:27:43.446 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000419448 s, 1.2 MB/s 2023-10-30T12:27:43.448 DEBUG:teuthology.orchestra.run.smithi125:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T12:27:43.505 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:27:43.506 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:27:43.506 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2023-10-30T12:27:43.512 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-30T12:27:43.512 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2023-10-30T12:27:43.560 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-30T12:27:43.561 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:43.561 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 878 Links: 1 2023-10-30T12:27:43.561 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:43.561 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-10-30 12:20:56.579441840 +0000 2023-10-30T12:27:43.562 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-10-30 12:20:56.207445163 +0000 2023-10-30T12:27:43.562 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-10-30 12:20:56.207445163 +0000 2023-10-30T12:27:43.562 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-10-30T12:27:43.562 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-30T12:27:43.615 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-10-30T12:27:43.615 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-10-30T12:27:43.615 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000604315 s, 847 kB/s 2023-10-30T12:27:43.616 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-30T12:27:43.668 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2023-10-30T12:27:43.717 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-30T12:27:43.717 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:43.717 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 898 Links: 1 2023-10-30T12:27:43.717 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:43.717 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-10-30 12:20:57.203436265 +0000 2023-10-30T12:27:43.718 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-10-30 12:20:56.851439411 +0000 2023-10-30T12:27:43.718 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-10-30 12:20:56.851439411 +0000 2023-10-30T12:27:43.718 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-10-30T12:27:43.718 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-30T12:27:43.770 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-10-30T12:27:43.770 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-10-30T12:27:43.771 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000596159 s, 859 kB/s 2023-10-30T12:27:43.771 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-30T12:27:43.821 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2023-10-30T12:27:43.870 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-30T12:27:43.870 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:43.870 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 914 Links: 1 2023-10-30T12:27:43.870 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:43.870 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-10-30 12:20:57.775431157 +0000 2023-10-30T12:27:43.871 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-10-30 12:20:57.443434123 +0000 2023-10-30T12:27:43.871 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-10-30 12:20:57.443434123 +0000 2023-10-30T12:27:43.871 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-10-30T12:27:43.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-30T12:27:43.922 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-10-30T12:27:43.922 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-10-30T12:27:43.923 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000602445 s, 850 kB/s 2023-10-30T12:27:43.924 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-30T12:27:43.973 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2023-10-30T12:27:44.020 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-30T12:27:44.021 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-30T12:27:44.021 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 934 Links: 1 2023-10-30T12:27:44.021 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:27:44.021 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2023-10-30 12:20:58.011429049 +0000 2023-10-30T12:27:44.022 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2023-10-30 12:20:58.007429085 +0000 2023-10-30T12:27:44.022 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2023-10-30 12:20:58.007429085 +0000 2023-10-30T12:27:44.022 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2023-10-30T12:27:44.022 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-30T12:27:44.074 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2023-10-30T12:27:44.074 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2023-10-30T12:27:44.074 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000420775 s, 1.2 MB/s 2023-10-30T12:27:44.075 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-30T12:27:44.121 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:27:44.122 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi125.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T12:27:44.122 INFO:tasks.ceph:Generating config... 2023-10-30T12:27:44.124 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-30T12:27:44.124 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2023-10-30T12:27:44.124 INFO:tasks.ceph:[client] debug client = 20 2023-10-30T12:27:44.124 INFO:tasks.ceph:[client] debug ms = 1 2023-10-30T12:27:44.125 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-30T12:27:44.125 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-30T12:27:44.125 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-30T12:27:44.126 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-30T12:27:44.126 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-10-30T12:27:44.127 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-30T12:27:44.127 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2023-10-30T12:27:44.127 INFO:tasks.ceph:[mds] mds bal merge size = 5 2023-10-30T12:27:44.127 INFO:tasks.ceph:[mds] mds bal split bits = 3 2023-10-30T12:27:44.128 INFO:tasks.ceph:[mds] mds bal split size = 100 2023-10-30T12:27:44.128 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-30T12:27:44.128 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-30T12:27:44.128 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2023-10-30T12:27:44.129 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-30T12:27:44.129 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-30T12:27:44.129 INFO:tasks.ceph:[mds] mds_dir_prefetch = False 2023-10-30T12:27:44.129 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2023-10-30T12:27:44.130 INFO:tasks.ceph:[mds] mon client directed command retry = 5 2023-10-30T12:27:44.130 INFO:tasks.ceph:[mds] ms inject delay max = 1 2023-10-30T12:27:44.130 INFO:tasks.ceph:[mds] ms inject delay probability = 0.005 2023-10-30T12:27:44.130 INFO:tasks.ceph:[mds] ms inject delay type = client mds osd 2023-10-30T12:27:44.131 INFO:tasks.ceph:[mds] ms inject socket failures = 2500 2023-10-30T12:27:44.131 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-30T12:27:44.131 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-30T12:27:44.131 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-30T12:27:44.131 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-30T12:27:44.132 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-30T12:27:44.132 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-30T12:27:44.132 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-30T12:27:44.132 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-30T12:27:44.133 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-30T12:27:44.133 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-30T12:27:44.133 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-30T12:27:44.133 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-30T12:27:44.134 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-30T12:27:44.134 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-30T12:27:44.134 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-30T12:27:44.135 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-30T12:27:44.135 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-30T12:27:44.135 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-30T12:27:44.135 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-30T12:27:44.135 INFO:tasks.ceph:[osd] mon client directed command retry = 5 2023-10-30T12:27:44.136 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-30T12:27:44.136 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-30T12:27:44.136 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-30T12:27:44.136 INFO:tasks.ceph:[osd] ms inject delay max = 1 2023-10-30T12:27:44.137 INFO:tasks.ceph:[osd] ms inject delay probability = 0.005 2023-10-30T12:27:44.137 INFO:tasks.ceph:[osd] ms inject delay type = client mds 2023-10-30T12:27:44.137 INFO:tasks.ceph:[osd] ms inject socket failures = 2500 2023-10-30T12:27:44.137 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2023-10-30T12:27:44.138 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-30T12:27:44.138 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2023-10-30T12:27:44.138 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-30T12:27:44.138 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-30T12:27:44.139 INFO:tasks.ceph:Setting up mon.a... 2023-10-30T12:27:44.139 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-30T12:27:44.184 INFO:teuthology.orchestra.run.smithi125.stdout:creating /etc/ceph/ceph.keyring 2023-10-30T12:27:44.212 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-30T12:27:44.302 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:27:44.367 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.125'), ('mon.b', '172.21.15.136'), ('mon.c', '[v2:172.21.15.136:3301,v1:172.21.15.136:6790]')] 2023-10-30T12:27:44.368 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.125,172.21.15.136,[v2:172.21.15.136:3301,v1:172.21.15.136:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon client directed command retry': 5, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds', 'ms inject socket failures': 2500, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': False, 'mds_sleep_rank_change': 5000000.0, 'mon client directed command retry': 5, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'client mds osd', 'ms inject socket failures': 2500, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-30T12:27:44.369 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:27:44.369 DEBUG:teuthology.orchestra.run.smithi125:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T12:27:44.419 DEBUG:teuthology.orchestra.run.smithi125:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.125 --add b 172.21.15.136 --addv c '[v2:172.21.15.136:3301,v1:172.21.15.136:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:44.510 INFO:teuthology.orchestra.run.smithi125.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:44.511 INFO:teuthology.orchestra.run.smithi125.stdout:monmaptool: generated fsid 714f08a3-69cf-4c1b-8fab-64b3ba5e42e9 2023-10-30T12:27:44.511 INFO:teuthology.orchestra.run.smithi125.stdout:setting min_mon_release = pacific 2023-10-30T12:27:44.511 INFO:teuthology.orchestra.run.smithi125.stdout:epoch 0 2023-10-30T12:27:44.511 INFO:teuthology.orchestra.run.smithi125.stdout:fsid 714f08a3-69cf-4c1b-8fab-64b3ba5e42e9 2023-10-30T12:27:44.511 INFO:teuthology.orchestra.run.smithi125.stdout:last_changed 2023-10-30T12:27:44.511128+0000 2023-10-30T12:27:44.512 INFO:teuthology.orchestra.run.smithi125.stdout:created 2023-10-30T12:27:44.511128+0000 2023-10-30T12:27:44.512 INFO:teuthology.orchestra.run.smithi125.stdout:min_mon_release 16 (pacific) 2023-10-30T12:27:44.512 INFO:teuthology.orchestra.run.smithi125.stdout:election_strategy: 1 2023-10-30T12:27:44.512 INFO:teuthology.orchestra.run.smithi125.stdout:0: [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] mon.a 2023-10-30T12:27:44.512 INFO:teuthology.orchestra.run.smithi125.stdout:1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.b 2023-10-30T12:27:44.512 INFO:teuthology.orchestra.run.smithi125.stdout:2: [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] mon.c 2023-10-30T12:27:44.513 INFO:teuthology.orchestra.run.smithi125.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-30T12:27:44.514 DEBUG:teuthology.orchestra.run.smithi125:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-30T12:27:44.522 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 714f08a3-69cf-4c1b-8fab-64b3ba5e42e9... 2023-10-30T12:27:44.523 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T12:27:44.566 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-30T12:27:44.595 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2023-10-30T12:27:44.595 INFO:teuthology.orchestra.run.smithi136.stdout: chdir = "" 2023-10-30T12:27:44.595 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T12:27:44.596 INFO:teuthology.orchestra.run.smithi136.stdout: auth supported = cephx 2023-10-30T12:27:44.596 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.596 INFO:teuthology.orchestra.run.smithi136.stdout: filestore xattr use omap = true 2023-10-30T12:27:44.596 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.596 INFO:teuthology.orchestra.run.smithi136.stdout: mon clock drift allowed = 1.000 2023-10-30T12:27:44.597 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.597 INFO:teuthology.orchestra.run.smithi136.stdout: osd crush chooseleaf type = 0 2023-10-30T12:27:44.597 INFO:teuthology.orchestra.run.smithi136.stdout: auth debug = true 2023-10-30T12:27:44.597 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.597 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on old message = true 2023-10-30T12:27:44.597 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on bug = true 2023-10-30T12:27:44.598 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.598 INFO:teuthology.orchestra.run.smithi136.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T12:27:44.598 INFO:teuthology.orchestra.run.smithi136.stdout: mon pg warn max object skew = 0 2023-10-30T12:27:44.598 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.598 INFO:teuthology.orchestra.run.smithi136.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T12:27:44.599 INFO:teuthology.orchestra.run.smithi136.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T12:27:44.599 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.599 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default size = 2 2023-10-30T12:27:44.599 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.599 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow primary affinity = true 2023-10-30T12:27:44.599 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow pg remap = true 2023-10-30T12:27:44.600 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on legacy crush tunables = false 2023-10-30T12:27:44.600 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on crush straw calc version zero = false 2023-10-30T12:27:44.600 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on no sortbitwise = false 2023-10-30T12:27:44.600 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on osd down out interval zero = false 2023-10-30T12:27:44.600 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on too few osds = false 2023-10-30T12:27:44.600 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T12:27:44.601 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T12:27:44.601 INFO:teuthology.orchestra.run.smithi136.stdout: mon_allow_pool_size_one = true 2023-10-30T12:27:44.601 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.601 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T12:27:44.601 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.602 INFO:teuthology.orchestra.run.smithi136.stdout: osd default data pool replay window = 5 2023-10-30T12:27:44.602 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.602 INFO:teuthology.orchestra.run.smithi136.stdout: mon allow pool delete = true 2023-10-30T12:27:44.602 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.602 INFO:teuthology.orchestra.run.smithi136.stdout: mon cluster log file level = debug 2023-10-30T12:27:44.603 INFO:teuthology.orchestra.run.smithi136.stdout: debug asserts on shutdown = true 2023-10-30T12:27:44.603 INFO:teuthology.orchestra.run.smithi136.stdout: mon health detail to clog = false 2023-10-30T12:27:44.603 INFO:teuthology.orchestra.run.smithi136.stdout: mon host = "172.21.15.125,172.21.15.136,[v2:172.21.15.136:3301,v1:172.21.15.136:6790]" 2023-10-30T12:27:44.603 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on skipped message = False 2023-10-30T12:27:44.603 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = 714f08a3-69cf-4c1b-8fab-64b3ba5e42e9 2023-10-30T12:27:44.603 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.604 INFO:teuthology.orchestra.run.smithi136.stdout:[osd] 2023-10-30T12:27:44.604 INFO:teuthology.orchestra.run.smithi136.stdout: osd journal size = 100 2023-10-30T12:27:44.604 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.604 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub load threshold = 5.0 2023-10-30T12:27:44.604 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub max interval = 600 2023-10-30T12:27:44.604 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock profile = high_recovery_ops 2023-10-30T12:27:44.605 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.605 INFO:teuthology.orchestra.run.smithi136.stdout: osd recover clone overlap = true 2023-10-30T12:27:44.605 INFO:teuthology.orchestra.run.smithi136.stdout: osd recovery max chunk = 1048576 2023-10-30T12:27:44.605 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.605 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug shutdown = true 2023-10-30T12:27:44.605 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug op order = true 2023-10-30T12:27:44.606 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify stray on activate = true 2023-10-30T12:27:44.606 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.606 INFO:teuthology.orchestra.run.smithi136.stdout: osd open classes on start = true 2023-10-30T12:27:44.606 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug pg log writeout = true 2023-10-30T12:27:44.606 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.607 INFO:teuthology.orchestra.run.smithi136.stdout: osd deep scrub update digest min age = 30 2023-10-30T12:27:44.607 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.607 INFO:teuthology.orchestra.run.smithi136.stdout: osd map max advance = 10 2023-10-30T12:27:44.607 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.607 INFO:teuthology.orchestra.run.smithi136.stdout: journal zero on create = true 2023-10-30T12:27:44.607 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.608 INFO:teuthology.orchestra.run.smithi136.stdout: filestore ondisk finisher threads = 3 2023-10-30T12:27:44.608 INFO:teuthology.orchestra.run.smithi136.stdout: filestore apply finisher threads = 3 2023-10-30T12:27:44.608 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.608 INFO:teuthology.orchestra.run.smithi136.stdout: bdev debug aio = true 2023-10-30T12:27:44.608 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug misdirected ops = true 2023-10-30T12:27:44.608 INFO:teuthology.orchestra.run.smithi136.stdout: bdev async discard = True 2023-10-30T12:27:44.609 INFO:teuthology.orchestra.run.smithi136.stdout: bdev enable discard = True 2023-10-30T12:27:44.609 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore allocator = bitmap 2023-10-30T12:27:44.609 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore block size = 96636764160 2023-10-30T12:27:44.609 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore fsck on mount = True 2023-10-30T12:27:44.609 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluefs = 1/20 2023-10-30T12:27:44.610 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluestore = 1/20 2023-10-30T12:27:44.610 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2023-10-30T12:27:44.610 INFO:teuthology.orchestra.run.smithi136.stdout: debug osd = 20 2023-10-30T12:27:44.610 INFO:teuthology.orchestra.run.smithi136.stdout: debug rocksdb = 4/10 2023-10-30T12:27:44.610 INFO:teuthology.orchestra.run.smithi136.stdout: mon client directed command retry = 5 2023-10-30T12:27:44.610 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T12:27:44.611 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd full ratio = 0.9 2023-10-30T12:27:44.611 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd nearfull ratio = 0.8 2023-10-30T12:27:44.611 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject delay max = 1 2023-10-30T12:27:44.611 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject delay probability = 0.005 2023-10-30T12:27:44.611 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject delay type = client mds 2023-10-30T12:27:44.611 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject socket failures = 2500 2023-10-30T12:27:44.612 INFO:teuthology.orchestra.run.smithi136.stdout: osd blocked scrub grace period = 3600 2023-10-30T12:27:44.612 INFO:teuthology.orchestra.run.smithi136.stdout: osd failsafe full ratio = 0.95 2023-10-30T12:27:44.612 INFO:teuthology.orchestra.run.smithi136.stdout: osd max markdown count = 1000 2023-10-30T12:27:44.612 INFO:teuthology.orchestra.run.smithi136.stdout: osd objectstore = bluestore 2023-10-30T12:27:44.612 INFO:teuthology.orchestra.run.smithi136.stdout: osd op complaint time = 180 2023-10-30T12:27:44.613 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.613 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr] 2023-10-30T12:27:44.613 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2023-10-30T12:27:44.613 INFO:teuthology.orchestra.run.smithi136.stdout: debug mgr = 20 2023-10-30T12:27:44.613 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2023-10-30T12:27:44.613 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2023-10-30T12:27:44.614 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:27:44.614 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:27:44.614 INFO:teuthology.orchestra.run.smithi136.stdout: mgr/telemetry/nag = false 2023-10-30T12:27:44.614 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.614 INFO:teuthology.orchestra.run.smithi136.stdout:[mon] 2023-10-30T12:27:44.614 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2023-10-30T12:27:44.615 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2023-10-30T12:27:44.615 INFO:teuthology.orchestra.run.smithi136.stdout: debug paxos = 20 2023-10-30T12:27:44.615 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2023-10-30T12:27:44.615 INFO:teuthology.orchestra.run.smithi136.stdout: mon data avail warn = 5 2023-10-30T12:27:44.615 INFO:teuthology.orchestra.run.smithi136.stdout: mon mgr mkfs grace = 240 2023-10-30T12:27:44.616 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:27:44.616 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd reporter subtree level = osd 2023-10-30T12:27:44.616 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd prime pg temp = true 2023-10-30T12:27:44.616 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:27:44.616 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.616 INFO:teuthology.orchestra.run.smithi136.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T12:27:44.617 INFO:teuthology.orchestra.run.smithi136.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T12:27:44.617 INFO:teuthology.orchestra.run.smithi136.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T12:27:44.617 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.617 INFO:teuthology.orchestra.run.smithi136.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T12:27:44.617 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T12:27:44.617 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T12:27:44.618 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.618 INFO:teuthology.orchestra.run.smithi136.stdout: # 1m isn't quite enough 2023-10-30T12:27:44.618 INFO:teuthology.orchestra.run.smithi136.stdout: mon_down_mkfs_grace = 2m 2023-10-30T12:27:44.618 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.618 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_filestore_osds = false 2023-10-30T12:27:44.618 INFO:teuthology.orchestra.run.smithi136.stdout: mon op complaint time = 120 2023-10-30T12:27:44.619 INFO:teuthology.orchestra.run.smithi136.stdout: 2023-10-30T12:27:44.619 INFO:teuthology.orchestra.run.smithi136.stdout:[client] 2023-10-30T12:27:44.619 INFO:teuthology.orchestra.run.smithi136.stdout: rgw cache enabled = true 2023-10-30T12:27:44.619 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable ops log = true 2023-10-30T12:27:44.619 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable usage log = true 2023-10-30T12:27:44.620 INFO:teuthology.orchestra.run.smithi136.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T12:27:44.620 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T12:27:44.620 INFO:teuthology.orchestra.run.smithi136.stdout: client mount timeout = 600 2023-10-30T12:27:44.620 INFO:teuthology.orchestra.run.smithi136.stdout: client_shutdown_timeout = 120 2023-10-30T12:27:44.620 INFO:teuthology.orchestra.run.smithi136.stdout: debug client = 20 2023-10-30T12:27:44.620 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2023-10-30T12:27:44.621 INFO:teuthology.orchestra.run.smithi136.stdout: rados mon op timeout = 900 2023-10-30T12:27:44.621 INFO:teuthology.orchestra.run.smithi136.stdout: rados osd op timeout = 900 2023-10-30T12:27:44.621 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.a] 2023-10-30T12:27:44.621 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.b] 2023-10-30T12:27:44.621 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.c] 2023-10-30T12:27:44.621 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.a] 2023-10-30T12:27:44.622 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.c] 2023-10-30T12:27:44.622 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.e] 2023-10-30T12:27:44.622 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.b] 2023-10-30T12:27:44.622 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.d] 2023-10-30T12:27:44.622 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.f] 2023-10-30T12:27:44.622 INFO:teuthology.orchestra.run.smithi136.stdout:[mds] 2023-10-30T12:27:44.623 INFO:teuthology.orchestra.run.smithi136.stdout: debug mds = 20 2023-10-30T12:27:44.623 INFO:teuthology.orchestra.run.smithi136.stdout: debug mds balancer = 20 2023-10-30T12:27:44.623 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2023-10-30T12:27:44.623 INFO:teuthology.orchestra.run.smithi136.stdout: mds bal fragment size max = 10000 2023-10-30T12:27:44.623 INFO:teuthology.orchestra.run.smithi136.stdout: mds bal merge size = 5 2023-10-30T12:27:44.624 INFO:teuthology.orchestra.run.smithi136.stdout: mds bal split bits = 3 2023-10-30T12:27:44.624 INFO:teuthology.orchestra.run.smithi136.stdout: mds bal split size = 100 2023-10-30T12:27:44.624 INFO:teuthology.orchestra.run.smithi136.stdout: mds debug frag = True 2023-10-30T12:27:44.624 INFO:teuthology.orchestra.run.smithi136.stdout: mds debug scatterstat = True 2023-10-30T12:27:44.624 INFO:teuthology.orchestra.run.smithi136.stdout: mds oft prefetch dirfrags = True 2023-10-30T12:27:44.624 INFO:teuthology.orchestra.run.smithi136.stdout: mds op complaint time = 180 2023-10-30T12:27:44.625 INFO:teuthology.orchestra.run.smithi136.stdout: mds verify scatter = True 2023-10-30T12:27:44.625 INFO:teuthology.orchestra.run.smithi136.stdout: mds_dir_prefetch = False 2023-10-30T12:27:44.625 INFO:teuthology.orchestra.run.smithi136.stdout: mds_sleep_rank_change = 5000000.0 2023-10-30T12:27:44.625 INFO:teuthology.orchestra.run.smithi136.stdout: mon client directed command retry = 5 2023-10-30T12:27:44.625 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject delay max = 1 2023-10-30T12:27:44.626 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject delay probability = 0.005 2023-10-30T12:27:44.626 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject delay type = client mds osd 2023-10-30T12:27:44.626 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject socket failures = 2500 2023-10-30T12:27:44.626 INFO:teuthology.orchestra.run.smithi136.stdout: osd op complaint time = 180 2023-10-30T12:27:44.626 INFO:teuthology.orchestra.run.smithi136.stdout: rados mon op timeout = 900 2023-10-30T12:27:44.626 INFO:teuthology.orchestra.run.smithi136.stdout: rados osd op timeout = 900 2023-10-30T12:27:44.628 INFO:teuthology.orchestra.run.smithi125.stdout:[global] 2023-10-30T12:27:44.628 INFO:teuthology.orchestra.run.smithi125.stdout: chdir = "" 2023-10-30T12:27:44.628 INFO:teuthology.orchestra.run.smithi125.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-30T12:27:44.628 INFO:teuthology.orchestra.run.smithi125.stdout: auth supported = cephx 2023-10-30T12:27:44.628 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.629 INFO:teuthology.orchestra.run.smithi125.stdout: filestore xattr use omap = true 2023-10-30T12:27:44.629 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.629 INFO:teuthology.orchestra.run.smithi125.stdout: mon clock drift allowed = 1.000 2023-10-30T12:27:44.629 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.629 INFO:teuthology.orchestra.run.smithi125.stdout: osd crush chooseleaf type = 0 2023-10-30T12:27:44.630 INFO:teuthology.orchestra.run.smithi125.stdout: auth debug = true 2023-10-30T12:27:44.630 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.630 INFO:teuthology.orchestra.run.smithi125.stdout: ms die on old message = true 2023-10-30T12:27:44.630 INFO:teuthology.orchestra.run.smithi125.stdout: ms die on bug = true 2023-10-30T12:27:44.630 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.631 INFO:teuthology.orchestra.run.smithi125.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-30T12:27:44.631 INFO:teuthology.orchestra.run.smithi125.stdout: mon pg warn max object skew = 0 2023-10-30T12:27:44.631 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.631 INFO:teuthology.orchestra.run.smithi125.stdout: # disable pg_autoscaler by default for new pools 2023-10-30T12:27:44.631 INFO:teuthology.orchestra.run.smithi125.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-30T12:27:44.631 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.632 INFO:teuthology.orchestra.run.smithi125.stdout: osd pool default size = 2 2023-10-30T12:27:44.632 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.632 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd allow primary affinity = true 2023-10-30T12:27:44.632 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd allow pg remap = true 2023-10-30T12:27:44.632 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on legacy crush tunables = false 2023-10-30T12:27:44.632 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on crush straw calc version zero = false 2023-10-30T12:27:44.633 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on no sortbitwise = false 2023-10-30T12:27:44.633 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on osd down out interval zero = false 2023-10-30T12:27:44.633 INFO:teuthology.orchestra.run.smithi125.stdout: mon warn on too few osds = false 2023-10-30T12:27:44.633 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-30T12:27:44.633 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-30T12:27:44.633 INFO:teuthology.orchestra.run.smithi125.stdout: mon_allow_pool_size_one = true 2023-10-30T12:27:44.634 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.634 INFO:teuthology.orchestra.run.smithi125.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-30T12:27:44.634 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.634 INFO:teuthology.orchestra.run.smithi125.stdout: osd default data pool replay window = 5 2023-10-30T12:27:44.634 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.635 INFO:teuthology.orchestra.run.smithi125.stdout: mon allow pool delete = true 2023-10-30T12:27:44.635 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.635 INFO:teuthology.orchestra.run.smithi125.stdout: mon cluster log file level = debug 2023-10-30T12:27:44.635 INFO:teuthology.orchestra.run.smithi125.stdout: debug asserts on shutdown = true 2023-10-30T12:27:44.635 INFO:teuthology.orchestra.run.smithi125.stdout: mon health detail to clog = false 2023-10-30T12:27:44.635 INFO:teuthology.orchestra.run.smithi125.stdout: mon host = "172.21.15.125,172.21.15.136,[v2:172.21.15.136:3301,v1:172.21.15.136:6790]" 2023-10-30T12:27:44.636 INFO:teuthology.orchestra.run.smithi125.stdout: ms die on skipped message = False 2023-10-30T12:27:44.636 INFO:teuthology.orchestra.run.smithi125.stdout: fsid = 714f08a3-69cf-4c1b-8fab-64b3ba5e42e9 2023-10-30T12:27:44.636 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.636 INFO:teuthology.orchestra.run.smithi125.stdout:[osd] 2023-10-30T12:27:44.636 INFO:teuthology.orchestra.run.smithi125.stdout: osd journal size = 100 2023-10-30T12:27:44.637 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.637 INFO:teuthology.orchestra.run.smithi125.stdout: osd scrub load threshold = 5.0 2023-10-30T12:27:44.637 INFO:teuthology.orchestra.run.smithi125.stdout: osd scrub max interval = 600 2023-10-30T12:27:44.637 INFO:teuthology.orchestra.run.smithi125.stdout: osd mclock profile = high_recovery_ops 2023-10-30T12:27:44.637 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.638 INFO:teuthology.orchestra.run.smithi125.stdout: osd recover clone overlap = true 2023-10-30T12:27:44.638 INFO:teuthology.orchestra.run.smithi125.stdout: osd recovery max chunk = 1048576 2023-10-30T12:27:44.638 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.638 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug shutdown = true 2023-10-30T12:27:44.638 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug op order = true 2023-10-30T12:27:44.638 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug verify stray on activate = true 2023-10-30T12:27:44.639 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.639 INFO:teuthology.orchestra.run.smithi125.stdout: osd open classes on start = true 2023-10-30T12:27:44.639 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug pg log writeout = true 2023-10-30T12:27:44.639 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.639 INFO:teuthology.orchestra.run.smithi125.stdout: osd deep scrub update digest min age = 30 2023-10-30T12:27:44.640 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.640 INFO:teuthology.orchestra.run.smithi125.stdout: osd map max advance = 10 2023-10-30T12:27:44.640 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.640 INFO:teuthology.orchestra.run.smithi125.stdout: journal zero on create = true 2023-10-30T12:27:44.640 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.640 INFO:teuthology.orchestra.run.smithi125.stdout: filestore ondisk finisher threads = 3 2023-10-30T12:27:44.641 INFO:teuthology.orchestra.run.smithi125.stdout: filestore apply finisher threads = 3 2023-10-30T12:27:44.641 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.641 INFO:teuthology.orchestra.run.smithi125.stdout: bdev debug aio = true 2023-10-30T12:27:44.641 INFO:teuthology.orchestra.run.smithi125.stdout: osd debug misdirected ops = true 2023-10-30T12:27:44.641 INFO:teuthology.orchestra.run.smithi125.stdout: bdev async discard = True 2023-10-30T12:27:44.642 INFO:teuthology.orchestra.run.smithi125.stdout: bdev enable discard = True 2023-10-30T12:27:44.642 INFO:teuthology.orchestra.run.smithi125.stdout: bluestore allocator = bitmap 2023-10-30T12:27:44.642 INFO:teuthology.orchestra.run.smithi125.stdout: bluestore block size = 96636764160 2023-10-30T12:27:44.642 INFO:teuthology.orchestra.run.smithi125.stdout: bluestore fsck on mount = True 2023-10-30T12:27:44.642 INFO:teuthology.orchestra.run.smithi125.stdout: debug bluefs = 1/20 2023-10-30T12:27:44.642 INFO:teuthology.orchestra.run.smithi125.stdout: debug bluestore = 1/20 2023-10-30T12:27:44.643 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-10-30T12:27:44.643 INFO:teuthology.orchestra.run.smithi125.stdout: debug osd = 20 2023-10-30T12:27:44.643 INFO:teuthology.orchestra.run.smithi125.stdout: debug rocksdb = 4/10 2023-10-30T12:27:44.643 INFO:teuthology.orchestra.run.smithi125.stdout: mon client directed command retry = 5 2023-10-30T12:27:44.643 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd backfillfull_ratio = 0.85 2023-10-30T12:27:44.644 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd full ratio = 0.9 2023-10-30T12:27:44.644 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd nearfull ratio = 0.8 2023-10-30T12:27:44.644 INFO:teuthology.orchestra.run.smithi125.stdout: ms inject delay max = 1 2023-10-30T12:27:44.644 INFO:teuthology.orchestra.run.smithi125.stdout: ms inject delay probability = 0.005 2023-10-30T12:27:44.644 INFO:teuthology.orchestra.run.smithi125.stdout: ms inject delay type = client mds 2023-10-30T12:27:44.644 INFO:teuthology.orchestra.run.smithi125.stdout: ms inject socket failures = 2500 2023-10-30T12:27:44.645 INFO:teuthology.orchestra.run.smithi125.stdout: osd blocked scrub grace period = 3600 2023-10-30T12:27:44.645 INFO:teuthology.orchestra.run.smithi125.stdout: osd failsafe full ratio = 0.95 2023-10-30T12:27:44.645 INFO:teuthology.orchestra.run.smithi125.stdout: osd max markdown count = 1000 2023-10-30T12:27:44.645 INFO:teuthology.orchestra.run.smithi125.stdout: osd objectstore = bluestore 2023-10-30T12:27:44.645 INFO:teuthology.orchestra.run.smithi125.stdout: osd op complaint time = 180 2023-10-30T12:27:44.645 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.646 INFO:teuthology.orchestra.run.smithi125.stdout:[mgr] 2023-10-30T12:27:44.646 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-10-30T12:27:44.646 INFO:teuthology.orchestra.run.smithi125.stdout: debug mgr = 20 2023-10-30T12:27:44.646 INFO:teuthology.orchestra.run.smithi125.stdout: debug mon = 20 2023-10-30T12:27:44.646 INFO:teuthology.orchestra.run.smithi125.stdout: debug auth = 20 2023-10-30T12:27:44.647 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:27:44.647 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:27:44.647 INFO:teuthology.orchestra.run.smithi125.stdout: mgr/telemetry/nag = false 2023-10-30T12:27:44.647 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.647 INFO:teuthology.orchestra.run.smithi125.stdout:[mon] 2023-10-30T12:27:44.648 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-10-30T12:27:44.648 INFO:teuthology.orchestra.run.smithi125.stdout: debug mon = 20 2023-10-30T12:27:44.648 INFO:teuthology.orchestra.run.smithi125.stdout: debug paxos = 20 2023-10-30T12:27:44.648 INFO:teuthology.orchestra.run.smithi125.stdout: debug auth = 20 2023-10-30T12:27:44.648 INFO:teuthology.orchestra.run.smithi125.stdout: mon data avail warn = 5 2023-10-30T12:27:44.648 INFO:teuthology.orchestra.run.smithi125.stdout: mon mgr mkfs grace = 240 2023-10-30T12:27:44.649 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min pgs per osd = 4 2023-10-30T12:27:44.649 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd reporter subtree level = osd 2023-10-30T12:27:44.649 INFO:teuthology.orchestra.run.smithi125.stdout: mon osd prime pg temp = true 2023-10-30T12:27:44.649 INFO:teuthology.orchestra.run.smithi125.stdout: mon reweight min bytes per osd = 10 2023-10-30T12:27:44.649 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.649 INFO:teuthology.orchestra.run.smithi125.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-30T12:27:44.650 INFO:teuthology.orchestra.run.smithi125.stdout: auth mon ticket ttl = 660 # 11m 2023-10-30T12:27:44.650 INFO:teuthology.orchestra.run.smithi125.stdout: auth service ticket ttl = 240 # 4m 2023-10-30T12:27:44.650 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.650 INFO:teuthology.orchestra.run.smithi125.stdout: # don't complain about insecure global_id in the test suite 2023-10-30T12:27:44.650 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-30T12:27:44.651 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-30T12:27:44.651 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.651 INFO:teuthology.orchestra.run.smithi125.stdout: # 1m isn't quite enough 2023-10-30T12:27:44.651 INFO:teuthology.orchestra.run.smithi125.stdout: mon_down_mkfs_grace = 2m 2023-10-30T12:27:44.651 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.651 INFO:teuthology.orchestra.run.smithi125.stdout: mon_warn_on_filestore_osds = false 2023-10-30T12:27:44.652 INFO:teuthology.orchestra.run.smithi125.stdout: mon op complaint time = 120 2023-10-30T12:27:44.652 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:27:44.652 INFO:teuthology.orchestra.run.smithi125.stdout:[client] 2023-10-30T12:27:44.652 INFO:teuthology.orchestra.run.smithi125.stdout: rgw cache enabled = true 2023-10-30T12:27:44.652 INFO:teuthology.orchestra.run.smithi125.stdout: rgw enable ops log = true 2023-10-30T12:27:44.652 INFO:teuthology.orchestra.run.smithi125.stdout: rgw enable usage log = true 2023-10-30T12:27:44.653 INFO:teuthology.orchestra.run.smithi125.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-30T12:27:44.653 INFO:teuthology.orchestra.run.smithi125.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-30T12:27:44.653 INFO:teuthology.orchestra.run.smithi125.stdout: client mount timeout = 600 2023-10-30T12:27:44.653 INFO:teuthology.orchestra.run.smithi125.stdout: client_shutdown_timeout = 120 2023-10-30T12:27:44.653 INFO:teuthology.orchestra.run.smithi125.stdout: debug client = 20 2023-10-30T12:27:44.653 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-10-30T12:27:44.654 INFO:teuthology.orchestra.run.smithi125.stdout: rados mon op timeout = 900 2023-10-30T12:27:44.654 INFO:teuthology.orchestra.run.smithi125.stdout: rados osd op timeout = 900 2023-10-30T12:27:44.654 INFO:teuthology.orchestra.run.smithi125.stdout:[mon.a] 2023-10-30T12:27:44.654 INFO:teuthology.orchestra.run.smithi125.stdout:[mon.b] 2023-10-30T12:27:44.654 INFO:teuthology.orchestra.run.smithi125.stdout:[mon.c] 2023-10-30T12:27:44.654 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.a] 2023-10-30T12:27:44.655 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.c] 2023-10-30T12:27:44.655 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.e] 2023-10-30T12:27:44.655 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.b] 2023-10-30T12:27:44.655 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.d] 2023-10-30T12:27:44.655 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.f] 2023-10-30T12:27:44.656 INFO:teuthology.orchestra.run.smithi125.stdout:[mds] 2023-10-30T12:27:44.656 INFO:teuthology.orchestra.run.smithi125.stdout: debug mds = 20 2023-10-30T12:27:44.656 INFO:teuthology.orchestra.run.smithi125.stdout: debug mds balancer = 20 2023-10-30T12:27:44.656 INFO:teuthology.orchestra.run.smithi125.stdout: debug ms = 1 2023-10-30T12:27:44.656 INFO:teuthology.orchestra.run.smithi125.stdout: mds bal fragment size max = 10000 2023-10-30T12:27:44.656 INFO:teuthology.orchestra.run.smithi125.stdout: mds bal merge size = 5 2023-10-30T12:27:44.657 INFO:teuthology.orchestra.run.smithi125.stdout: mds bal split bits = 3 2023-10-30T12:27:44.657 INFO:teuthology.orchestra.run.smithi125.stdout: mds bal split size = 100 2023-10-30T12:27:44.657 INFO:teuthology.orchestra.run.smithi125.stdout: mds debug frag = True 2023-10-30T12:27:44.657 INFO:teuthology.orchestra.run.smithi125.stdout: mds debug scatterstat = True 2023-10-30T12:27:44.657 INFO:teuthology.orchestra.run.smithi125.stdout: mds oft prefetch dirfrags = True 2023-10-30T12:27:44.657 INFO:teuthology.orchestra.run.smithi125.stdout: mds op complaint time = 180 2023-10-30T12:27:44.658 INFO:teuthology.orchestra.run.smithi125.stdout: mds verify scatter = True 2023-10-30T12:27:44.658 INFO:teuthology.orchestra.run.smithi125.stdout: mds_dir_prefetch = False 2023-10-30T12:27:44.658 INFO:teuthology.orchestra.run.smithi125.stdout: mds_sleep_rank_change = 5000000.0 2023-10-30T12:27:44.658 INFO:teuthology.orchestra.run.smithi125.stdout: mon client directed command retry = 5 2023-10-30T12:27:44.658 INFO:teuthology.orchestra.run.smithi125.stdout: ms inject delay max = 1 2023-10-30T12:27:44.659 INFO:teuthology.orchestra.run.smithi125.stdout: ms inject delay probability = 0.005 2023-10-30T12:27:44.659 INFO:teuthology.orchestra.run.smithi125.stdout: ms inject delay type = client mds osd 2023-10-30T12:27:44.659 INFO:teuthology.orchestra.run.smithi125.stdout: ms inject socket failures = 2500 2023-10-30T12:27:44.659 INFO:teuthology.orchestra.run.smithi125.stdout: osd op complaint time = 180 2023-10-30T12:27:44.659 INFO:teuthology.orchestra.run.smithi125.stdout: rados mon op timeout = 900 2023-10-30T12:27:44.659 INFO:teuthology.orchestra.run.smithi125.stdout: rados osd op timeout = 900 2023-10-30T12:27:44.660 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-30T12:27:44.660 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-30T12:27:44.732 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-30T12:27:44.733 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:27:44.733 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-30T12:27:44.742 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:27:44.742 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-30T12:27:44.791 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi125.front.sepia.ceph.com 2023-10-30T12:27:44.791 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:27:44.791 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T12:27:44.792 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:27:44.867 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:27:44.867 DEBUG:teuthology.orchestra.run.smithi125:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:44.919 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi136.front.sepia.ceph.com 2023-10-30T12:27:44.919 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:27:44.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-30T12:27:44.920 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-30T12:27:44.944 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:27:44.944 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:44.992 INFO:tasks.ceph:Setting up mon nodes... 2023-10-30T12:27:44.993 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-30T12:27:44.993 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T12:27:45.049 INFO:teuthology.orchestra.run.smithi125.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-30T12:27:45.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T12:27:45.097 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-30T12:27:45.112 INFO:tasks.ceph:Setting up mds nodes... 2023-10-30T12:27:45.112 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-30T12:27:45.169 INFO:teuthology.orchestra.run.smithi125.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-30T12:27:45.173 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-30T12:27:45.194 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-30T12:27:45.289 INFO:teuthology.orchestra.run.smithi125.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-30T12:27:45.294 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-30T12:27:45.314 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2023-10-30T12:27:45.409 INFO:teuthology.orchestra.run.smithi125.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2023-10-30T12:27:45.414 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2023-10-30T12:27:45.434 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-30T12:27:45.477 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-30T12:27:45.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-30T12:27:45.541 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-30T12:27:45.625 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-30T12:27:45.628 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-30T12:27:45.684 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2023-10-30T12:27:45.767 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2023-10-30T12:27:45.770 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2023-10-30T12:27:45.823 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-30T12:27:45.824 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-30T12:27:45.867 INFO:teuthology.orchestra.run.smithi125.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-30T12:27:45.883 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-30T12:27:45.884 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi125.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-30T12:27:45.884 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-30T12:27:45.946 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:27:45.946 INFO:tasks.ceph:role: osd.0 2023-10-30T12:27:45.947 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi125.front.sepia.ceph.com 2023-10-30T12:27:45.947 DEBUG:teuthology.orchestra.run.smithi125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T12:27:46.014 INFO:teuthology.orchestra.run.smithi125.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:27:46.014 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:27:46.014 INFO:teuthology.orchestra.run.smithi125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:27:46.014 INFO:teuthology.orchestra.run.smithi125.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:27:46.015 INFO:teuthology.orchestra.run.smithi125.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:27:46.015 INFO:teuthology.orchestra.run.smithi125.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:27:46.015 INFO:teuthology.orchestra.run.smithi125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:27:46.015 INFO:teuthology.orchestra.run.smithi125.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:27:46.015 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:27:46.016 INFO:teuthology.orchestra.run.smithi125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:27:46.473 INFO:teuthology.orchestra.run.smithi125.stdout:Discarding blocks...Done. 2023-10-30T12:27:46.475 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi125.front.sepia.ceph.com -o noatime 2023-10-30T12:27:46.476 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-30T12:27:46.580 DEBUG:teuthology.orchestra.run.smithi125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-30T12:27:46.596 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:27:46.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:27:46.598 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-30T12:27:46.659 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:27:46.659 INFO:tasks.ceph:role: osd.1 2023-10-30T12:27:46.660 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi125.front.sepia.ceph.com 2023-10-30T12:27:46.660 DEBUG:teuthology.orchestra.run.smithi125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T12:27:46.730 INFO:teuthology.orchestra.run.smithi125.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:27:46.731 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:27:46.731 INFO:teuthology.orchestra.run.smithi125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:27:46.731 INFO:teuthology.orchestra.run.smithi125.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:27:46.731 INFO:teuthology.orchestra.run.smithi125.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:27:46.732 INFO:teuthology.orchestra.run.smithi125.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:27:46.732 INFO:teuthology.orchestra.run.smithi125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:27:46.732 INFO:teuthology.orchestra.run.smithi125.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:27:46.732 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:27:46.732 INFO:teuthology.orchestra.run.smithi125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:27:47.240 INFO:teuthology.orchestra.run.smithi125.stdout:Discarding blocks...Done. 2023-10-30T12:27:47.243 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi125.front.sepia.ceph.com -o noatime 2023-10-30T12:27:47.244 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-30T12:27:47.378 DEBUG:teuthology.orchestra.run.smithi125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-30T12:27:47.396 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:27:47.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:27:47.397 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-30T12:27:47.460 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:27:47.460 INFO:tasks.ceph:role: osd.2 2023-10-30T12:27:47.460 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi125.front.sepia.ceph.com 2023-10-30T12:27:47.461 DEBUG:teuthology.orchestra.run.smithi125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T12:27:47.536 INFO:teuthology.orchestra.run.smithi125.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:27:47.536 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:27:47.536 INFO:teuthology.orchestra.run.smithi125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:27:47.537 INFO:teuthology.orchestra.run.smithi125.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:27:47.537 INFO:teuthology.orchestra.run.smithi125.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:27:47.537 INFO:teuthology.orchestra.run.smithi125.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:27:47.537 INFO:teuthology.orchestra.run.smithi125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:27:47.537 INFO:teuthology.orchestra.run.smithi125.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:27:47.537 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:27:47.538 INFO:teuthology.orchestra.run.smithi125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:27:48.011 INFO:teuthology.orchestra.run.smithi125.stdout:Discarding blocks...Done. 2023-10-30T12:27:48.014 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi125.front.sepia.ceph.com -o noatime 2023-10-30T12:27:48.015 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-30T12:27:48.132 DEBUG:teuthology.orchestra.run.smithi125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-30T12:27:48.150 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:27:48.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:27:48.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-30T12:27:48.219 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-30T12:27:48.220 INFO:tasks.ceph:role: osd.3 2023-10-30T12:27:48.220 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi125.front.sepia.ceph.com 2023-10-30T12:27:48.220 DEBUG:teuthology.orchestra.run.smithi125:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T12:27:48.296 INFO:teuthology.orchestra.run.smithi125.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:27:48.296 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:27:48.296 INFO:teuthology.orchestra.run.smithi125.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:27:48.297 INFO:teuthology.orchestra.run.smithi125.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:27:48.297 INFO:teuthology.orchestra.run.smithi125.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:27:48.297 INFO:teuthology.orchestra.run.smithi125.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:27:48.297 INFO:teuthology.orchestra.run.smithi125.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:27:48.297 INFO:teuthology.orchestra.run.smithi125.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:27:48.298 INFO:teuthology.orchestra.run.smithi125.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:27:48.298 INFO:teuthology.orchestra.run.smithi125.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:27:48.759 INFO:teuthology.orchestra.run.smithi125.stdout:Discarding blocks...Done. 2023-10-30T12:27:48.761 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi125.front.sepia.ceph.com -o noatime 2023-10-30T12:27:48.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-30T12:27:48.866 DEBUG:teuthology.orchestra.run.smithi125:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-30T12:27:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:27:48.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:27:48.887 DEBUG:teuthology.orchestra.run.smithi125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:49.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:49.005+0000 7fa8355d3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-30T12:27:49.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:49.009+0000 7fa8355d3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-30T12:27:49.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:49.009+0000 7fa8355d3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:27:49.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:49.009+0000 7fa8355d3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:27:49.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:49.009+0000 7fa8355d3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-30T12:27:49.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:49.013+0000 7fa8355d3640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-30T12:27:52.000 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-30T12:27:52.022 DEBUG:teuthology.orchestra.run.smithi125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:52.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:52.117+0000 7fca96d75640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-30T12:27:52.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:52.117+0000 7fca96d75640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-30T12:27:52.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:52.117+0000 7fca96d75640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:27:52.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:52.117+0000 7fca96d75640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:27:52.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:52.117+0000 7fca96d75640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-30T12:27:52.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:52.117+0000 7fca96d75640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-30T12:27:55.107 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-30T12:27:55.129 DEBUG:teuthology.orchestra.run.smithi125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:55.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:55.225+0000 7f495d703640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-30T12:27:55.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:55.225+0000 7f495d703640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-30T12:27:55.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:55.225+0000 7f495d703640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:27:55.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:55.225+0000 7f495d703640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:27:55.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:55.225+0000 7f495d703640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-30T12:27:55.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:55.229+0000 7f495d703640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-30T12:27:58.218 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-30T12:27:58.240 DEBUG:teuthology.orchestra.run.smithi125:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:27:58.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:58.337+0000 7f354efa4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-30T12:27:58.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:58.337+0000 7f354efa4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-30T12:27:58.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:58.337+0000 7f354efa4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:27:58.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:58.337+0000 7f354efa4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:27:58.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:58.337+0000 7f354efa4640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-30T12:27:58.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:27:58.337+0000 7f354efa4640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-30T12:28:01.349 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-30T12:28:01.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-30T12:28:01.387 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:28:01.388 INFO:tasks.ceph:role: osd.4 2023-10-30T12:28:01.388 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com 2023-10-30T12:28:01.388 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-30T12:28:01.451 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:28:01.452 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:28:01.452 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:28:01.452 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:28:01.452 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:28:01.453 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:28:01.453 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:28:01.453 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:28:01.453 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:28:01.453 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:28:01.969 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2023-10-30T12:28:01.971 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2023-10-30T12:28:01.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-30T12:28:02.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-30T12:28:02.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:28:02.085 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:28:02.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-30T12:28:02.137 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:28:02.138 INFO:tasks.ceph:role: osd.5 2023-10-30T12:28:02.138 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com 2023-10-30T12:28:02.138 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-30T12:28:02.203 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:28:02.203 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:28:02.204 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:28:02.204 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:28:02.204 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:28:02.204 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:28:02.204 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:28:02.205 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:28:02.205 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:28:02.205 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:28:02.667 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2023-10-30T12:28:02.668 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2023-10-30T12:28:02.669 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-30T12:28:02.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-30T12:28:02.782 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:28:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:28:02.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-30T12:28:02.843 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:28:02.844 INFO:tasks.ceph:role: osd.6 2023-10-30T12:28:02.844 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com 2023-10-30T12:28:02.844 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-30T12:28:02.907 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:28:02.907 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:28:02.907 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:28:02.907 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:28:02.907 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:28:02.908 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:28:02.908 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:28:02.908 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:28:02.908 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:28:02.908 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:28:03.392 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2023-10-30T12:28:03.395 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2023-10-30T12:28:03.395 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-30T12:28:03.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-30T12:28:03.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:28:03.501 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:28:03.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-30T12:28:03.555 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-30T12:28:03.556 INFO:tasks.ceph:role: osd.7 2023-10-30T12:28:03.556 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi136.front.sepia.ceph.com 2023-10-30T12:28:03.556 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-30T12:28:03.620 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-30T12:28:03.620 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-30T12:28:03.620 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-30T12:28:03.620 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-10-30T12:28:03.621 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-30T12:28:03.621 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2023-10-30T12:28:03.621 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-30T12:28:03.621 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-10-30T12:28:03.621 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-30T12:28:03.621 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-30T12:28:04.125 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2023-10-30T12:28:04.127 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2023-10-30T12:28:04.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-30T12:28:04.259 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-30T12:28:04.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:28:04.271 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2023-10-30T12:28:04.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:28:04.387 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:04.386+0000 7fdcd5268640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-30T12:28:04.394 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:04.394+0000 7fdcd5268640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-30T12:28:04.394 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:04.394+0000 7fdcd5268640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:28:04.394 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:04.394+0000 7fdcd5268640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:28:04.394 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:04.394+0000 7fdcd5268640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-30T12:28:04.397 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:04.398+0000 7fdcd5268640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-30T12:28:07.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-30T12:28:07.454 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:28:07.536 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:07.538+0000 7fb1ad150640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-30T12:28:07.537 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:07.538+0000 7fb1ad150640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-30T12:28:07.537 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:07.538+0000 7fb1ad150640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:28:07.537 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:07.538+0000 7fb1ad150640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:28:07.538 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:07.538+0000 7fb1ad150640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-30T12:28:07.538 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:07.538+0000 7fb1ad150640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-30T12:28:10.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-30T12:28:10.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:28:10.691 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:10.690+0000 7f8643e8d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-30T12:28:10.692 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:10.694+0000 7f8643e8d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-30T12:28:10.692 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:10.694+0000 7f8643e8d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T12:28:10.692 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:10.694+0000 7f8643e8d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T12:28:10.693 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:10.694+0000 7f8643e8d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-30T12:28:10.693 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:10.694+0000 7f8643e8d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-30T12:28:12.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:28:12.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:28:13.728 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-30T12:28:13.743 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:28:13.826 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:13.826+0000 7f32bd341640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-30T12:28:13.827 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:13.826+0000 7f32bd341640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-30T12:28:13.827 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:13.826+0000 7f32bd341640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T12:28:13.827 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:13.826+0000 7f32bd341640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T12:28:13.828 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:13.826+0000 7f32bd341640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-30T12:28:13.828 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:13.830+0000 7f32bd341640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-30T12:28:16.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-30T12:28:16.900 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-30T12:28:16.901 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:28:16.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-30T12:28:16.924 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:28:16.924 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-30T12:28:16.988 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:28:16.988 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-30T12:28:17.052 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:28:17.052 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2023-10-30T12:28:17.115 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:28:17.116 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-30T12:28:17.179 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:28:17.179 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-30T12:28:17.242 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:28:17.243 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-30T12:28:17.307 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:28:17.307 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-30T12:28:17.370 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:28:17.371 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-30T12:28:17.384 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:28:17.385 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-30T12:28:17.438 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:28:17.438 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-30T12:28:17.491 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:28:17.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2023-10-30T12:28:17.547 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:28:17.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-30T12:28:17.602 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:28:17.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-30T12:28:17.655 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:28:17.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-30T12:28:17.710 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:28:17.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-30T12:28:17.762 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:28:17.763 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-30T12:28:17.772 INFO:tasks.ceph:Adding keys to all mons... 2023-10-30T12:28:17.772 DEBUG:teuthology.orchestra.run.smithi125:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T12:28:17.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-30T12:28:17.833 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.y] 2023-10-30T12:28:17.833 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBBoT9lgjD+AhAAr4Tw0VM4lhd2OxWF4idfRQ== 2023-10-30T12:28:17.833 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.a] 2023-10-30T12:28:17.833 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBBoT9lOGAkChAAdqxdXODVgkp7eYeg2jtySQ== 2023-10-30T12:28:17.834 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.c] 2023-10-30T12:28:17.834 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBBoT9lpZdPERAAYhf12wBPLfEgbdzrSbkP9Q== 2023-10-30T12:28:17.834 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.e] 2023-10-30T12:28:17.834 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBBoT9lxw55GBAA0komH/xohfzodpxioqkyTA== 2023-10-30T12:28:17.834 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.0] 2023-10-30T12:28:17.835 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBFoT9lilmhABAA+WicrAG4/U7O0/EuBo/kRA== 2023-10-30T12:28:17.835 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.1] 2023-10-30T12:28:17.835 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBIoT9liyY2BxAAn7y4P2dB+V07RCAKS5YsKQ== 2023-10-30T12:28:17.835 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.2] 2023-10-30T12:28:17.835 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBLoT9ls6u+DRAAWgqjqePbRNwq04cbOPqtBA== 2023-10-30T12:28:17.836 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.3] 2023-10-30T12:28:17.836 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBOoT9lmeJNFBAA/QVZ2C7KTnatLObEEwFeIw== 2023-10-30T12:28:17.836 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.x] 2023-10-30T12:28:17.836 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBBoT9l4h8FBhAAx42Q0A1TwUu5OjhPK8PkBA== 2023-10-30T12:28:17.836 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.b] 2023-10-30T12:28:17.836 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBBoT9lxW+vHBAA6wQR6fcMk4WbgQes8HZyqQ== 2023-10-30T12:28:17.837 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.d] 2023-10-30T12:28:17.837 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBBoT9lioV7JRAAkOfb5JP9CkWsGl06NOQyIQ== 2023-10-30T12:28:17.837 INFO:teuthology.orchestra.run.smithi136.stdout:[mds.f] 2023-10-30T12:28:17.837 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBBoT9ljM31LRAAi+QvFJHskWN4DQ/j1qhJSA== 2023-10-30T12:28:17.837 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.4] 2023-10-30T12:28:17.837 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBUoT9lD25RFxAAJlsVLu0GmWwRW/ug0NhCug== 2023-10-30T12:28:17.838 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.5] 2023-10-30T12:28:17.838 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBXoT9ldxcxIBAA4fFnSdmzcflaArxgcsYpKA== 2023-10-30T12:28:17.838 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.6] 2023-10-30T12:28:17.838 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBaoT9ldwdxKRAAVPW6IbLFbic9/PEoboezfA== 2023-10-30T12:28:17.838 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.7] 2023-10-30T12:28:17.839 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBdoT9ldIJ5MRAAQjLPHCw1MeGjhOKM7ptKCg== 2023-10-30T12:28:17.839 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2023-10-30T12:28:17.839 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBBoT9lSAW6MxAAa/u+qvv3IHIX0krh1UIuag== 2023-10-30T12:28:17.839 INFO:teuthology.orchestra.run.smithi125.stdout:[mgr.y] 2023-10-30T12:28:17.840 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBBoT9lgjD+AhAAr4Tw0VM4lhd2OxWF4idfRQ== 2023-10-30T12:28:17.840 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.a] 2023-10-30T12:28:17.840 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBBoT9lOGAkChAAdqxdXODVgkp7eYeg2jtySQ== 2023-10-30T12:28:17.840 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.c] 2023-10-30T12:28:17.840 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBBoT9lpZdPERAAYhf12wBPLfEgbdzrSbkP9Q== 2023-10-30T12:28:17.841 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.e] 2023-10-30T12:28:17.841 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBBoT9lxw55GBAA0komH/xohfzodpxioqkyTA== 2023-10-30T12:28:17.841 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.0] 2023-10-30T12:28:17.841 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBFoT9lilmhABAA+WicrAG4/U7O0/EuBo/kRA== 2023-10-30T12:28:17.841 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.1] 2023-10-30T12:28:17.841 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBIoT9liyY2BxAAn7y4P2dB+V07RCAKS5YsKQ== 2023-10-30T12:28:17.842 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.2] 2023-10-30T12:28:17.842 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBLoT9ls6u+DRAAWgqjqePbRNwq04cbOPqtBA== 2023-10-30T12:28:17.842 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.3] 2023-10-30T12:28:17.842 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBOoT9lmeJNFBAA/QVZ2C7KTnatLObEEwFeIw== 2023-10-30T12:28:17.842 INFO:teuthology.orchestra.run.smithi125.stdout:[mgr.x] 2023-10-30T12:28:17.843 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBBoT9l4h8FBhAAx42Q0A1TwUu5OjhPK8PkBA== 2023-10-30T12:28:17.843 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.b] 2023-10-30T12:28:17.843 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBBoT9lxW+vHBAA6wQR6fcMk4WbgQes8HZyqQ== 2023-10-30T12:28:17.843 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.d] 2023-10-30T12:28:17.843 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBBoT9lioV7JRAAkOfb5JP9CkWsGl06NOQyIQ== 2023-10-30T12:28:17.843 INFO:teuthology.orchestra.run.smithi125.stdout:[mds.f] 2023-10-30T12:28:17.844 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBBoT9ljM31LRAAi+QvFJHskWN4DQ/j1qhJSA== 2023-10-30T12:28:17.844 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.4] 2023-10-30T12:28:17.844 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBUoT9lD25RFxAAJlsVLu0GmWwRW/ug0NhCug== 2023-10-30T12:28:17.844 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.5] 2023-10-30T12:28:17.844 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBXoT9ldxcxIBAA4fFnSdmzcflaArxgcsYpKA== 2023-10-30T12:28:17.844 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.6] 2023-10-30T12:28:17.845 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBaoT9ldwdxKRAAVPW6IbLFbic9/PEoboezfA== 2023-10-30T12:28:17.845 INFO:teuthology.orchestra.run.smithi125.stdout:[osd.7] 2023-10-30T12:28:17.845 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBdoT9ldIJ5MRAAQjLPHCw1MeGjhOKM7ptKCg== 2023-10-30T12:28:17.845 INFO:teuthology.orchestra.run.smithi125.stdout:[client.0] 2023-10-30T12:28:17.845 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQBBoT9lSAW6MxAAa/u+qvv3IHIX0krh1UIuag== 2023-10-30T12:28:17.847 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:28:17.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:28:17.948 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:17.953 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:18.028 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:18.032 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:18.086 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:18.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:18.162 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:18.165 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:18.243 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:18.246 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:18.326 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:18.330 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:18.406 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:18.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:18.491 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:28:18.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-30T12:28:18.576 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:18.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:18.665 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:18.669 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:18.722 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:18.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-30T12:28:18.799 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:18.803 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:18.888 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:18.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:18.971 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:18.975 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:19.055 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:19.058 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-30T12:28:19.140 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T12:28:19.144 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-30T12:28:19.229 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-30T12:28:19.230 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-30T12:28:19.250 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:28:19.362 DEBUG:teuthology.orchestra.run.smithi125:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-30T12:28:19.382 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-30T12:28:19.396 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:28:19.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-30T12:28:19.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-30T12:28:19.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-30T12:28:19.652 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-30T12:28:19.665 DEBUG:teuthology.orchestra.run.smithi125:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:28:19.668 DEBUG:teuthology.orchestra.run.smithi136:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-30T12:28:19.715 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-30T12:28:19.715 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-30T12:28:19.716 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-30T12:28:19.722 INFO:tasks.ceph.mon.a:Started 2023-10-30T12:28:19.722 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-30T12:28:19.722 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-30T12:28:19.761 INFO:tasks.ceph.mon.b:Started 2023-10-30T12:28:19.761 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-30T12:28:19.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-30T12:28:19.764 INFO:tasks.ceph.mon.c:Started 2023-10-30T12:28:19.764 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-30T12:28:19.764 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-30T12:28:19.765 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-30T12:28:19.768 INFO:tasks.ceph.mgr.y:Started 2023-10-30T12:28:19.769 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-30T12:28:19.769 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-30T12:28:19.771 INFO:tasks.ceph.mgr.x:Started 2023-10-30T12:28:19.772 INFO:tasks.ceph:Setting crush tunables to default 2023-10-30T12:28:19.772 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --cluster ceph osd crush tunables default 2023-10-30T12:28:19.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:19.908+0000 7f9e72a32640 1 Processor -- start 2023-10-30T12:28:19.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:19.908+0000 7f9e72a32640 1 -- start start 2023-10-30T12:28:19.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:19.912+0000 7f9e72a32640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 0x7f9e6c15da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:19.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:19.912+0000 7f9e72a32640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c15a920 con 0x7f9e6c159930 2023-10-30T12:28:19.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:19.912+0000 7f9e72a32640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6c0cb930 con 0x7f9e6c153980 2023-10-30T12:28:19.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:19.912+0000 7f9e72a32640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c159dc0 con 0x7f9e6c15d570 2023-10-30T12:28:19.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:19.912+0000 7f9e6b7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9e6c159930 0x7f9e6c0cf1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52760/0 (socket says 172.21.15.125:52760) 2023-10-30T12:28:19.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:19.912+0000 7f9e6b7fe640 1 -- 172.21.15.125:0/1296557704 learned_addr learned my addr 172.21.15.125:0/1296557704 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:19.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:19.912+0000 7f9e6bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9e6c153980 0x7f9e6c0caca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:55984/0 (socket says 172.21.15.125:55984) 2023-10-30T12:28:19.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:19.912+0000 7f9e70fcf640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 0x7f9e6c15da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:19.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:19.912+0000 7f9e70fcf640 1 -- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 msgr2=0x7f9e6c15da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:28:19.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:19.912+0000 7f9e70fcf640 1 -- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 msgr2=0x7f9e6c15da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:28:19.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:19.912+0000 7f9e70fcf640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 0x7f9e6c15da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:28:19.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:19.912+0000 7f9e70fcf640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 0x7f9e6c15da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T12:28:20.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.116+0000 7f9e70fcf640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 0x7f9e6c15da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.116+0000 7f9e70fcf640 1 -- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 msgr2=0x7f9e6c15da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-10-30T12:28:20.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.116+0000 7f9e70fcf640 1 -- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 msgr2=0x7f9e6c15da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-30T12:28:20.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.116+0000 7f9e70fcf640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 0x7f9e6c15da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T12:28:20.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.116+0000 7f9e70fcf640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 0x7f9e6c15da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-30T12:28:20.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 -- 172.21.15.125:0/1296557704 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211632522 0 0) 0x7f9e6c15a920 con 0x7f9e6c159930 2023-10-30T12:28:20.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 -- 172.21.15.125:0/1296557704 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e54003530 con 0x7f9e6c159930 2023-10-30T12:28:20.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 -- 172.21.15.125:0/1296557704 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2819081319 0 0) 0x7f9e54003530 con 0x7f9e6c159930 2023-10-30T12:28:20.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 -- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 msgr2=0x7f9e6c15da00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:28:20.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 0x7f9e6c15da00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 -- 172.21.15.125:0/1296557704 >> v1:172.21.15.136:6789/0 conn(0x7f9e6c153980 legacy=0x7f9e6c0caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:20.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 -- 172.21.15.125:0/1296557704 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c15a080 con 0x7f9e6c159930 2023-10-30T12:28:20.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 -- 172.21.15.125:0/1296557704 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f9e600015b0 con 0x7f9e6c159930 2023-10-30T12:28:20.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 -- 172.21.15.125:0/1296557704 >> v1:172.21.15.125:6789/0 conn(0x7f9e6c159930 legacy=0x7f9e6c0cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:20.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e540039b0 0x7f9e54003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e540047d0 0x7f9e54004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 0x7f9e54005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 -- 172.21.15.125:0/1296557704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c159dc0 con 0x7f9e540047d0 2023-10-30T12:28:20.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 -- 172.21.15.125:0/1296557704 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5400bb20 con 0x7f9e540039b0 2023-10-30T12:28:20.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6affd640 1 -- 172.21.15.125:0/1296557704 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5400bce0 con 0x7f9e6c15d570 2023-10-30T12:28:20.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6bfff640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e540039b0 0x7f9e54003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e70fcf640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 0x7f9e54005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6b7fe640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e540047d0 0x7f9e54004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6b7fe640 1 -- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 msgr2=0x7f9e54005220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:20.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6b7fe640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c15d570 0x7f9e54005220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6b7fe640 1 -- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e540039b0 msgr2=0x7f9e54003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:20.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6b7fe640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e540039b0 0x7f9e54003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.132+0000 7f9e6b7fe640 1 -- 172.21.15.125:0/1296557704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54003530 con 0x7f9e540047d0 2023-10-30T12:28:20.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.136+0000 7f9e6b7fe640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e540047d0 0x7f9e54004c50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000c0e0 tx=0x7f9e6001f2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b2ca19fb1c0a19 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:20.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.136+0000 7f9e6affd640 1 -- 172.21.15.125:0/1296557704 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e600058e0 con 0x7f9e540047d0 2023-10-30T12:28:20.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.136+0000 7f9e6affd640 1 -- 172.21.15.125:0/1296557704 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e60005a60 con 0x7f9e540047d0 2023-10-30T12:28:20.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.136+0000 7f9e6affd640 1 -- 172.21.15.125:0/1296557704 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60005e10 con 0x7f9e540047d0 2023-10-30T12:28:20.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.136+0000 7f9e72a32640 1 -- 172.21.15.125:0/1296557704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e540047d0 msgr2=0x7f9e54004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:20.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.136+0000 7f9e72a32640 1 --2- 172.21.15.125:0/1296557704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e540047d0 0x7f9e54004c50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9e6000c0e0 tx=0x7f9e6001f2d0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.136+0000 7f9e70fcf640 1 -- 172.21.15.125:0/1296557704 reap_dead start 2023-10-30T12:28:20.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.136+0000 7f9e72a32640 1 -- 172.21.15.125:0/1296557704 shutdown_connections 2023-10-30T12:28:20.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.136+0000 7f9e72a32640 1 -- 172.21.15.125:0/1296557704 >> 172.21.15.125:0/1296557704 conn(0x7f9e6c06c0f0 msgr2=0x7f9e6c06c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:20.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.136+0000 7f9e72a32640 1 -- 172.21.15.125:0/1296557704 shutdown_connections 2023-10-30T12:28:20.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.136+0000 7f9e72a32640 1 -- 172.21.15.125:0/1296557704 wait complete. 2023-10-30T12:28:20.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e72a32640 1 Processor -- start 2023-10-30T12:28:20.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e72a32640 1 -- start start 2023-10-30T12:28:20.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e72a32640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e540047d0 0x7f9e6c1511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e72a32640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c1e2930 0x7f9e6c0ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e72a32640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e6c153240 0x7f9e6c157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e72a32640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c152460 con 0x7f9e6c153240 2023-10-30T12:28:20.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e72a32640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c1ef530 con 0x7f9e540047d0 2023-10-30T12:28:20.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e72a32640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6c1eec20 con 0x7f9e6c1e2930 2023-10-30T12:28:20.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e70fcf640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e6c153240 0x7f9e6c157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e70fcf640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e6c153240 0x7f9e6c157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48068/0 (socket says 172.21.15.125:48068) 2023-10-30T12:28:20.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e70fcf640 1 -- 172.21.15.125:0/3978220628 learned_addr learned my addr 172.21.15.125:0/3978220628 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:20.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e6bfff640 1 --2- 172.21.15.125:0/3978220628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e540047d0 0x7f9e6c1511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e6b7fe640 1 --2- 172.21.15.125:0/3978220628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c1e2930 0x7f9e6c0ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e70fcf640 1 -- 172.21.15.125:0/3978220628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c1e2930 msgr2=0x7f9e6c0ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:20.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e70fcf640 1 --2- 172.21.15.125:0/3978220628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c1e2930 0x7f9e6c0ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e70fcf640 1 -- 172.21.15.125:0/3978220628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e540047d0 msgr2=0x7f9e6c1511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:20.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e70fcf640 1 --2- 172.21.15.125:0/3978220628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e540047d0 0x7f9e6c1511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.140+0000 7f9e70fcf640 1 -- 172.21.15.125:0/3978220628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c1e31c0 con 0x7f9e6c153240 2023-10-30T12:28:20.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.144+0000 7f9e70fcf640 1 --2- 172.21.15.125:0/3978220628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e6c153240 0x7f9e6c157420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0caa50 tx=0x7f9e64007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:20.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.144+0000 7f9e68ff9640 1 -- 172.21.15.125:0/3978220628 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e640048a0 con 0x7f9e6c153240 2023-10-30T12:28:20.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.144+0000 7f9e68ff9640 1 -- 172.21.15.125:0/3978220628 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6400f040 con 0x7f9e6c153240 2023-10-30T12:28:20.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.144+0000 7f9e72a32640 1 -- 172.21.15.125:0/3978220628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6c1e34a0 con 0x7f9e6c153240 2023-10-30T12:28:20.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.144+0000 7f9e68ff9640 1 -- 172.21.15.125:0/3978220628 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e640166a0 con 0x7f9e6c153240 2023-10-30T12:28:20.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.144+0000 7f9e72a32640 1 -- 172.21.15.125:0/3978220628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6c1ef1c0 con 0x7f9e6c153240 2023-10-30T12:28:20.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.144+0000 7f9e68ff9640 1 -- 172.21.15.125:0/3978220628 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9e64016b00 con 0x7f9e6c153240 2023-10-30T12:28:20.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.144+0000 7f9e68ff9640 1 -- 172.21.15.125:0/3978220628 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9e64020ce0 con 0x7f9e6c153240 2023-10-30T12:28:20.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.144+0000 7f9e72a32640 1 -- 172.21.15.125:0/3978220628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6c1ef380 con 0x7f9e6c153240 2023-10-30T12:28:20.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.148+0000 7f9e68ff9640 1 -- 172.21.15.125:0/3978220628 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f9e64020400 con 0x7f9e6c153240 2023-10-30T12:28:20.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.208+0000 7f9e72a32640 1 -- 172.21.15.125:0/3978220628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f9e6c04e170 con 0x7f9e6c153240 2023-10-30T12:28:20.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.212+0000 7f9e68ff9640 1 -- 172.21.15.125:0/3978220628 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9e640205a0 con 0x7f9e6c153240 2023-10-30T12:28:20.216 INFO:teuthology.orchestra.run.smithi125.stderr:adjusted tunables profile to default 2023-10-30T12:28:20.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.212+0000 7f9e72a32640 1 -- 172.21.15.125:0/3978220628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e6c153240 msgr2=0x7f9e6c157420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:20.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.212+0000 7f9e72a32640 1 --2- 172.21.15.125:0/3978220628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e6c153240 0x7f9e6c157420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0caa50 tx=0x7f9e64007700 comp rx=0 tx=0).stop 2023-10-30T12:28:20.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.212+0000 7f9e72a32640 1 -- 172.21.15.125:0/3978220628 shutdown_connections 2023-10-30T12:28:20.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.212+0000 7f9e72a32640 1 --2- 172.21.15.125:0/3978220628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e6c153240 0x7f9e6c157420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.212+0000 7f9e72a32640 1 --2- 172.21.15.125:0/3978220628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e6c1e2930 0x7f9e6c0ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.212+0000 7f9e72a32640 1 --2- 172.21.15.125:0/3978220628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e540047d0 0x7f9e6c1511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.212+0000 7f9e72a32640 1 -- 172.21.15.125:0/3978220628 >> 172.21.15.125:0/3978220628 conn(0x7f9e6c06c0f0 msgr2=0x7f9e6c1590e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:20.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.212+0000 7f9e72a32640 1 -- 172.21.15.125:0/3978220628 shutdown_connections 2023-10-30T12:28:20.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:20.212+0000 7f9e72a32640 1 -- 172.21.15.125:0/3978220628 wait complete. 2023-10-30T12:28:20.236 INFO:tasks.ceph:check_enable_crimson: False 2023-10-30T12:28:20.237 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-30T12:28:20.237 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:28:20.238 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-30T12:28:20.258 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:28:20.258 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-30T12:28:20.322 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:28:20.323 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-30T12:28:20.342 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:20.336+0000 7f6caba49180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T12:28:20.362 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:28:20.362 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-30T12:28:20.407 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:20.406+0000 7f0f681bb180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-30T12:28:20.422 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:28:20.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-30T12:28:20.436 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:28:20.436 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-30T12:28:20.490 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:28:20.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-30T12:28:20.546 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2023-10-30T12:28:20.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-30T12:28:20.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf 0 2023-10-30T12:28:20.758 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.758+0000 7fa03241e640 1 Processor -- start 2023-10-30T12:28:20.758 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.758+0000 7fa03241e640 1 -- start start 2023-10-30T12:28:20.759 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.758+0000 7fa03241e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa02c0c4080 0x7fa02c06fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.759 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.758+0000 7fa03241e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa02c0c7110 0x7fa02c0c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.760 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.758+0000 7fa03241e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02c070580 con 0x7fa02c150810 2023-10-30T12:28:20.760 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.758+0000 7fa03241e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa02c0c37d0 con 0x7fa02c0c4080 2023-10-30T12:28:20.760 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.758+0000 7fa03241e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c150bf0 con 0x7fa02c0c7110 2023-10-30T12:28:20.760 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa02bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa02c150810 0x7fa02c15a3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52916/0 (socket says 172.21.15.136:52916) 2023-10-30T12:28:20.760 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa0309bb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa02c0c7110 0x7fa02c0c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.761 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa02bfff640 1 -- 172.21.15.136:0/2508189333 learned_addr learned my addr 172.21.15.136:0/2508189333 (peer_addr_for_me v1:172.21.15.136:0/0) 2023-10-30T12:28:20.761 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa02b7fe640 1 --2- 172.21.15.136:0/2508189333 >> v2:172.21.15.125:3300/0 conn(0x7fa02c0c4080 0x7fa02c06fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.761 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa02affd640 1 -- 172.21.15.136:0/2508189333 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636644054 0 0) 0x7fa02c070580 con 0x7fa02c150810 2023-10-30T12:28:20.761 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa02affd640 1 -- 172.21.15.136:0/2508189333 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa014003530 con 0x7fa02c150810 2023-10-30T12:28:20.763 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa02affd640 1 -- 172.21.15.136:0/2508189333 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1667450567 0 0) 0x7fa014003530 con 0x7fa02c150810 2023-10-30T12:28:20.763 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa0309bb640 1 -- 172.21.15.136:0/2508189333 >> v2:172.21.15.125:3300/0 conn(0x7fa02c0c4080 msgr2=0x7fa02c06fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:20.763 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa0309bb640 1 --2- 172.21.15.136:0/2508189333 >> v2:172.21.15.125:3300/0 conn(0x7fa02c0c4080 0x7fa02c06fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.764 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa0309bb640 1 -- 172.21.15.136:0/2508189333 >> v1:172.21.15.136:6789/0 conn(0x7fa02c150810 legacy=0x7fa02c15a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:20.764 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa0309bb640 1 -- 172.21.15.136:0/2508189333 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02c0c6070 con 0x7fa02c0c7110 2023-10-30T12:28:20.764 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa0309bb640 1 --2- 172.21.15.136:0/2508189333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa02c0c7110 0x7fa02c0c7fa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0097c0 tx=0x7fa01c01c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11a83ed732ca82e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:20.764 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa02affd640 1 -- 172.21.15.136:0/2508189333 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa024001a00 con 0x7fa02c0c7110 2023-10-30T12:28:20.764 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa02affd640 1 -- 172.21.15.136:0/2508189333 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01c0a7050 con 0x7fa02c0c7110 2023-10-30T12:28:20.765 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa02affd640 1 -- 172.21.15.136:0/2508189333 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01c0ab600 con 0x7fa02c0c7110 2023-10-30T12:28:20.765 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa03241e640 1 -- 172.21.15.136:0/2508189333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa02c0c7110 msgr2=0x7fa02c0c7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:20.765 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa03241e640 1 --2- 172.21.15.136:0/2508189333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa02c0c7110 0x7fa02c0c7fa0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa01c0097c0 tx=0x7fa01c01c3a0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.765 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa03241e640 1 -- 172.21.15.136:0/2508189333 shutdown_connections 2023-10-30T12:28:20.765 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa03241e640 1 --2- 172.21.15.136:0/2508189333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa02c0c7110 0x7fa02c0c7fa0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.765 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa03241e640 1 --2- 172.21.15.136:0/2508189333 >> v2:172.21.15.125:3300/0 conn(0x7fa02c0c4080 0x7fa02c06fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.766 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa03241e640 1 -- 172.21.15.136:0/2508189333 >> 172.21.15.136:0/2508189333 conn(0x7fa02c06c620 msgr2=0x7fa02c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:20.766 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa03241e640 1 -- 172.21.15.136:0/2508189333 shutdown_connections 2023-10-30T12:28:20.766 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.762+0000 7fa03241e640 1 -- 172.21.15.136:0/2508189333 wait complete. 2023-10-30T12:28:20.768 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa03241e640 1 Processor -- start 2023-10-30T12:28:20.768 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa03241e640 1 -- start start 2023-10-30T12:28:20.769 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa03241e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa02c0c4080 0x7fa02c14fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.769 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa03241e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa02c0c7110 0x7fa02c137c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.769 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa03241e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa02c150810 0x7fa02c14ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.769 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa03241e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c1efd90 con 0x7fa02c0c7110 2023-10-30T12:28:20.769 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa03241e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c1f0d20 con 0x7fa02c150810 2023-10-30T12:28:20.770 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa03241e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02c1f4270 con 0x7fa02c0c4080 2023-10-30T12:28:20.770 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa02bfff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa02c0c4080 0x7fa02c14fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.770 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa02bfff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa02c0c4080 0x7fa02c14fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:55624/0 (socket says 172.21.15.136:55624) 2023-10-30T12:28:20.770 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa02bfff640 1 -- 172.21.15.136:0/4092305653 learned_addr learned my addr 172.21.15.136:0/4092305653 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-10-30T12:28:20.770 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa0309bb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa02c150810 0x7fa02c14ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.771 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa02bfff640 1 -- 172.21.15.136:0/4092305653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa02c150810 msgr2=0x7fa02c14ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:20.771 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa02b7fe640 1 --2- 172.21.15.136:0/4092305653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa02c0c7110 0x7fa02c137c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.771 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa02bfff640 1 --2- 172.21.15.136:0/4092305653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa02c150810 0x7fa02c14ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.771 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa02bfff640 1 -- 172.21.15.136:0/4092305653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa02c0c7110 msgr2=0x7fa02c137c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:20.771 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa02bfff640 1 --2- 172.21.15.136:0/4092305653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa02c0c7110 0x7fa02c137c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.772 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa02bfff640 1 -- 172.21.15.136:0/4092305653 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa01c009420 con 0x7fa02c0c4080 2023-10-30T12:28:20.772 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa02b7fe640 1 --2- 172.21.15.136:0/4092305653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa02c0c7110 0x7fa02c137c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:20.772 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa02bfff640 1 --2- 172.21.15.136:0/4092305653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa02c0c4080 0x7fa02c14fcc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa0240018a0 tx=0x7fa02400a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:20.772 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa028ff9640 1 -- 172.21.15.136:0/4092305653 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa024012400 con 0x7fa02c0c4080 2023-10-30T12:28:20.772 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa028ff9640 1 -- 172.21.15.136:0/4092305653 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa024012560 con 0x7fa02c0c4080 2023-10-30T12:28:20.773 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa028ff9640 1 -- 172.21.15.136:0/4092305653 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa024012910 con 0x7fa02c0c4080 2023-10-30T12:28:20.773 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa03241e640 1 -- 172.21.15.136:0/4092305653 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa02c14e8a0 con 0x7fa02c0c4080 2023-10-30T12:28:20.773 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa03241e640 1 -- 172.21.15.136:0/4092305653 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02c1effa0 con 0x7fa02c0c4080 2023-10-30T12:28:20.773 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa028ff9640 1 -- 172.21.15.136:0/4092305653 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa02401b400 con 0x7fa02c0c4080 2023-10-30T12:28:20.773 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa028ff9640 1 -- 172.21.15.136:0/4092305653 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa0240073b0 con 0x7fa02c0c4080 2023-10-30T12:28:20.774 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.766+0000 7fa03241e640 1 -- 172.21.15.136:0/4092305653 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa02c0c6ba0 con 0x7fa02c0c4080 2023-10-30T12:28:20.774 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.774+0000 7fa028ff9640 1 -- 172.21.15.136:0/4092305653 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fa024012e20 con 0x7fa02c0c4080 2023-10-30T12:28:20.841 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.842+0000 7fa03241e640 1 -- 172.21.15.136:0/4092305653 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf", "id": 0} v 0) v1 -- 0x7fa02c04e170 con 0x7fa02c0c4080 2023-10-30T12:28:20.850 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.850+0000 7fa028ff9640 1 -- 172.21.15.136:0/4092305653 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa024007690 con 0x7fa02c0c4080 2023-10-30T12:28:20.851 INFO:teuthology.orchestra.run.smithi136.stdout:0 2023-10-30T12:28:20.852 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.854+0000 7fa03241e640 1 -- 172.21.15.136:0/4092305653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa02c0c4080 msgr2=0x7fa02c14fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:20.852 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.854+0000 7fa03241e640 1 --2- 172.21.15.136:0/4092305653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa02c0c4080 0x7fa02c14fcc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa0240018a0 tx=0x7fa02400a430 comp rx=0 tx=0).stop 2023-10-30T12:28:20.852 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.854+0000 7fa03241e640 1 -- 172.21.15.136:0/4092305653 shutdown_connections 2023-10-30T12:28:20.853 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.854+0000 7fa03241e640 1 --2- 172.21.15.136:0/4092305653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa02c150810 0x7fa02c14ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.853 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.854+0000 7fa03241e640 1 --2- 172.21.15.136:0/4092305653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa02c0c7110 0x7fa02c137c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.853 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.854+0000 7fa03241e640 1 --2- 172.21.15.136:0/4092305653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa02c0c4080 0x7fa02c14fcc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.853 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.854+0000 7fa03241e640 1 -- 172.21.15.136:0/4092305653 >> 172.21.15.136:0/4092305653 conn(0x7fa02c06c620 msgr2=0x7fa02c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:20.853 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.854+0000 7fa03241e640 1 -- 172.21.15.136:0/4092305653 shutdown_connections 2023-10-30T12:28:20.854 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.854+0000 7fa03241e640 1 -- 172.21.15.136:0/4092305653 wait complete. 2023-10-30T12:28:20.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 5470ed31-d4ff-483d-a06f-82dc6ac62cce 1 2023-10-30T12:28:20.878 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:20.872+0000 7f6caba49180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T12:28:20.984 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:20.986+0000 7f0f681bb180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-30T12:28:20.989 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f6190387640 1 Processor -- start 2023-10-30T12:28:20.989 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f6190387640 1 -- start start 2023-10-30T12:28:20.991 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f6190387640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6188151650 0x7f61880c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.991 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f6190387640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f61880c1f50 0x7f618806f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:20.991 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f6190387640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61880c9200 con 0x7f618806fe40 2023-10-30T12:28:20.992 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f6190387640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6188157d80 con 0x7f61880c1f50 2023-10-30T12:28:20.992 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f6190387640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61880c8ed0 con 0x7f6188151650 2023-10-30T12:28:20.992 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f618e123640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6188151650 0x7f61880c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.992 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f618e123640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6188151650 0x7f61880c7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:55632/0 (socket says 172.21.15.136:55632) 2023-10-30T12:28:20.992 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f618e123640 1 -- 172.21.15.136:0/859618331 learned_addr learned my addr 172.21.15.136:0/859618331 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-10-30T12:28:20.993 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f618e924640 1 --2- 172.21.15.136:0/859618331 >> v2:172.21.15.125:3300/0 conn(0x7f61880c1f50 0x7f618806f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:20.993 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f618e123640 1 -- 172.21.15.136:0/859618331 >> v2:172.21.15.125:3300/0 conn(0x7f61880c1f50 msgr2=0x7f618806f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:20.993 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f618e123640 1 --2- 172.21.15.136:0/859618331 >> v2:172.21.15.125:3300/0 conn(0x7f61880c1f50 0x7f618806f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.993 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f618e123640 1 -- 172.21.15.136:0/859618331 >> v1:172.21.15.125:6789/0 conn(0x7f618806fe40 legacy=0x7f6188157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:20.993 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.990+0000 7f618e123640 1 -- 172.21.15.136:0/859618331 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61880c3b10 con 0x7f6188151650 2023-10-30T12:28:20.994 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.994+0000 7f618e123640 1 --2- 172.21.15.136:0/859618331 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6188151650 0x7f61880c7990 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6184009e50 tx=0x7f618401f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c862121e1bb5f55 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:20.994 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.994+0000 7f618d121640 1 -- 172.21.15.136:0/859618331 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61840af400 con 0x7f6188151650 2023-10-30T12:28:20.995 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.994+0000 7f618d121640 1 -- 172.21.15.136:0/859618331 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61840ab050 con 0x7f6188151650 2023-10-30T12:28:20.995 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.994+0000 7f618d121640 1 -- 172.21.15.136:0/859618331 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61840af780 con 0x7f6188151650 2023-10-30T12:28:20.996 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.994+0000 7f6190387640 1 -- 172.21.15.136:0/859618331 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6188151650 msgr2=0x7f61880c7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:20.996 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.994+0000 7f6190387640 1 --2- 172.21.15.136:0/859618331 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6188151650 0x7f61880c7990 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6184009e50 tx=0x7f618401f1f0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.996 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.994+0000 7f6190387640 1 -- 172.21.15.136:0/859618331 shutdown_connections 2023-10-30T12:28:20.997 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.994+0000 7f6190387640 1 --2- 172.21.15.136:0/859618331 >> v2:172.21.15.125:3300/0 conn(0x7f61880c1f50 0x7f618806f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.997 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.994+0000 7f6190387640 1 --2- 172.21.15.136:0/859618331 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6188151650 0x7f61880c7990 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:20.997 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.994+0000 7f6190387640 1 -- 172.21.15.136:0/859618331 >> 172.21.15.136:0/859618331 conn(0x7f6188069e90 msgr2=0x7f618806a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:20.997 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.994+0000 7f6190387640 1 -- 172.21.15.136:0/859618331 shutdown_connections 2023-10-30T12:28:20.997 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.994+0000 7f6190387640 1 -- 172.21.15.136:0/859618331 wait complete. 2023-10-30T12:28:20.998 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f6190387640 1 Processor -- start 2023-10-30T12:28:20.998 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f6190387640 1 -- start start 2023-10-30T12:28:21.000 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f6190387640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f618806fe40 0x7f6188065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.000 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f6190387640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61880c1f50 0x7f6188151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.000 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f6190387640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6188151650 0x7f61881ed860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.000 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f6190387640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61881476b0 con 0x7f618806fe40 2023-10-30T12:28:21.001 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f6190387640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61881473d0 con 0x7f61880c1f50 2023-10-30T12:28:21.001 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f6190387640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6188146500 con 0x7f6188151650 2023-10-30T12:28:21.001 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f618e924640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6188151650 0x7f61881ed860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.001 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f618d922640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61880c1f50 0x7f6188151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.001 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f618e924640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6188151650 0x7f61881ed860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:55638/0 (socket says 172.21.15.136:55638) 2023-10-30T12:28:21.002 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f618e924640 1 -- 172.21.15.136:0/3242175422 learned_addr learned my addr 172.21.15.136:0/3242175422 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-10-30T12:28:21.002 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f618d922640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61880c1f50 0x7f6188151f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58894/0 (socket says 172.21.15.136:58894) 2023-10-30T12:28:21.002 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f618e123640 1 --2- 172.21.15.136:0/3242175422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f618806fe40 0x7f6188065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.002 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f618d922640 1 -- 172.21.15.136:0/3242175422 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6188151650 msgr2=0x7f61881ed860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.002 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f618d922640 1 --2- 172.21.15.136:0/3242175422 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6188151650 0x7f61881ed860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.003 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f618d922640 1 -- 172.21.15.136:0/3242175422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f618806fe40 msgr2=0x7f6188065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.003 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f618d922640 1 --2- 172.21.15.136:0/3242175422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f618806fe40 0x7f6188065ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.003 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f618d922640 1 -- 172.21.15.136:0/3242175422 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6184009b30 con 0x7f61880c1f50 2023-10-30T12:28:21.003 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f618e924640 1 --2- 172.21.15.136:0/3242175422 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6188151650 0x7f61881ed860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:21.003 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f618d922640 1 --2- 172.21.15.136:0/3242175422 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61880c1f50 0x7f6188151f60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6178000f70 tx=0x7f617800c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:21.004 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f617effd640 1 -- 172.21.15.136:0/3242175422 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61780052d0 con 0x7f61880c1f50 2023-10-30T12:28:21.004 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f617effd640 1 -- 172.21.15.136:0/3242175422 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6178005430 con 0x7f61880c1f50 2023-10-30T12:28:21.004 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f617effd640 1 -- 172.21.15.136:0/3242175422 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6178003840 con 0x7f61880c1f50 2023-10-30T12:28:21.004 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f6190387640 1 -- 172.21.15.136:0/3242175422 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61881467e0 con 0x7f61880c1f50 2023-10-30T12:28:21.004 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f6190387640 1 -- 172.21.15.136:0/3242175422 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61881ed2f0 con 0x7f61880c1f50 2023-10-30T12:28:21.005 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f617effd640 1 -- 172.21.15.136:0/3242175422 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6178003ca0 con 0x7f61880c1f50 2023-10-30T12:28:21.005 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f617effd640 1 -- 172.21.15.136:0/3242175422 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f617801c440 con 0x7f61880c1f50 2023-10-30T12:28:21.005 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:20.998+0000 7f6190387640 1 -- 172.21.15.136:0/3242175422 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61880c3dd0 con 0x7f61880c1f50 2023-10-30T12:28:21.005 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.002+0000 7f617effd640 1 -- 172.21.15.136:0/3242175422 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f61880c3dd0 con 0x7f61880c1f50 2023-10-30T12:28:21.069 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.070+0000 7f6190387640 1 -- 172.21.15.136:0/3242175422 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5470ed31-d4ff-483d-a06f-82dc6ac62cce", "id": 1} v 0) v1 -- 0x7f618804e170 con 0x7f61880c1f50 2023-10-30T12:28:21.081 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.082+0000 7f617effd640 1 -- 172.21.15.136:0/3242175422 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5470ed31-d4ff-483d-a06f-82dc6ac62cce", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6178017070 con 0x7f61880c1f50 2023-10-30T12:28:21.081 INFO:teuthology.orchestra.run.smithi136.stdout:1 2023-10-30T12:28:21.082 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.082+0000 7f6190387640 1 -- 172.21.15.136:0/3242175422 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61880c1f50 msgr2=0x7f6188151f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.082 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.082+0000 7f6190387640 1 --2- 172.21.15.136:0/3242175422 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61880c1f50 0x7f6188151f60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6178000f70 tx=0x7f617800c820 comp rx=0 tx=0).stop 2023-10-30T12:28:21.083 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.082+0000 7f6190387640 1 -- 172.21.15.136:0/3242175422 shutdown_connections 2023-10-30T12:28:21.083 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.082+0000 7f6190387640 1 --2- 172.21.15.136:0/3242175422 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6188151650 0x7f61881ed860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.083 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.082+0000 7f6190387640 1 --2- 172.21.15.136:0/3242175422 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61880c1f50 0x7f6188151f60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.083 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.082+0000 7f6190387640 1 --2- 172.21.15.136:0/3242175422 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f618806fe40 0x7f6188065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.083 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.082+0000 7f6190387640 1 -- 172.21.15.136:0/3242175422 >> 172.21.15.136:0/3242175422 conn(0x7f6188069e90 msgr2=0x7f618814ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:21.084 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.082+0000 7f6190387640 1 -- 172.21.15.136:0/3242175422 shutdown_connections 2023-10-30T12:28:21.084 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.082+0000 7f6190387640 1 -- 172.21.15.136:0/3242175422 wait complete. 2023-10-30T12:28:21.092 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:21.088+0000 7f6caba49180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T12:28:21.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 74491459-06a1-488c-8a93-b04224b25521 2 2023-10-30T12:28:21.188 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:21.190+0000 7f0f681bb180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-30T12:28:21.220 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f083155e640 1 Processor -- start 2023-10-30T12:28:21.221 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f083155e640 1 -- start start 2023-10-30T12:28:21.222 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f083155e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f082c0c2e20 0x7f082c0c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.222 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f083155e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f082c15a070 0x7f082c15a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.223 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f083155e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f082c159be0 con 0x7f082c150800 2023-10-30T12:28:21.223 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f083155e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f082c0c7020 con 0x7f082c0c2e20 2023-10-30T12:28:21.223 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f083155e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f082c15aa40 con 0x7f082c15a070 2023-10-30T12:28:21.223 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f082b7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f082c15a070 0x7f082c15a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.223 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f082b7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f082c15a070 0x7f082c15a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58910/0 (socket says 172.21.15.136:58910) 2023-10-30T12:28:21.224 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f082b7fe640 1 -- 172.21.15.136:0/3059912506 learned_addr learned my addr 172.21.15.136:0/3059912506 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-10-30T12:28:21.224 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f082a7fc640 1 --2- 172.21.15.136:0/3059912506 >> v2:172.21.15.125:3300/0 conn(0x7f082c0c2e20 0x7f082c0c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.224 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f082b7fe640 1 -- 172.21.15.136:0/3059912506 >> v2:172.21.15.125:3300/0 conn(0x7f082c0c2e20 msgr2=0x7f082c0c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:21.224 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f082b7fe640 1 --2- 172.21.15.136:0/3059912506 >> v2:172.21.15.125:3300/0 conn(0x7f082c0c2e20 0x7f082c0c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.224 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f082b7fe640 1 -- 172.21.15.136:0/3059912506 >> v1:172.21.15.125:6789/0 conn(0x7f082c150800 legacy=0x7f082c0cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.225 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.222+0000 7f082b7fe640 1 -- 172.21.15.136:0/3059912506 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c156850 con 0x7f082c15a070 2023-10-30T12:28:21.226 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.226+0000 7f082b7fe640 1 --2- 172.21.15.136:0/3059912506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f082c15a070 0x7f082c15a500 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f082c0c1ac0 tx=0x7f081c0213c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffb8343aadfd227f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:21.226 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.226+0000 7f0829ffb640 1 -- 172.21.15.136:0/3059912506 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c00b2d0 con 0x7f082c15a070 2023-10-30T12:28:21.226 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.226+0000 7f0829ffb640 1 -- 172.21.15.136:0/3059912506 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081c0ad050 con 0x7f082c15a070 2023-10-30T12:28:21.226 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.226+0000 7f0829ffb640 1 -- 172.21.15.136:0/3059912506 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081c009820 con 0x7f082c15a070 2023-10-30T12:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.226+0000 7f083155e640 1 -- 172.21.15.136:0/3059912506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f082c15a070 msgr2=0x7f082c15a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.226+0000 7f083155e640 1 --2- 172.21.15.136:0/3059912506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f082c15a070 0x7f082c15a500 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f082c0c1ac0 tx=0x7f081c0213c0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.226+0000 7f083155e640 1 -- 172.21.15.136:0/3059912506 shutdown_connections 2023-10-30T12:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.226+0000 7f083155e640 1 --2- 172.21.15.136:0/3059912506 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f082c15a070 0x7f082c15a500 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.227 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.226+0000 7f083155e640 1 --2- 172.21.15.136:0/3059912506 >> v2:172.21.15.125:3300/0 conn(0x7f082c0c2e20 0x7f082c0c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.228 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.226+0000 7f083155e640 1 -- 172.21.15.136:0/3059912506 >> 172.21.15.136:0/3059912506 conn(0x7f082c06c850 msgr2=0x7f082c06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:21.228 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.226+0000 7f083155e640 1 -- 172.21.15.136:0/3059912506 shutdown_connections 2023-10-30T12:28:21.228 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.226+0000 7f083155e640 1 -- 172.21.15.136:0/3059912506 wait complete. 2023-10-30T12:28:21.230 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.226+0000 7f083155e640 1 Processor -- start 2023-10-30T12:28:21.230 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.226+0000 7f083155e640 1 -- start start 2023-10-30T12:28:21.231 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f083155e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f082c0c2e20 0x7f082c158b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.231 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f083155e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f082c150800 0x7f082c1eac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.231 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f083155e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f082c15a070 0x7f082c141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.231 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f083155e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c1549a0 con 0x7f082c15a070 2023-10-30T12:28:21.231 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f083155e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f082c06bf00 con 0x7f082c0c2e20 2023-10-30T12:28:21.231 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f083155e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f082c069e90 con 0x7f082c150800 2023-10-30T12:28:21.232 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f082affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f082c0c2e20 0x7f082c158b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.232 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f082a7fc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f082c150800 0x7f082c1eac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.232 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f082affd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f082c0c2e20 0x7f082c158b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58914/0 (socket says 172.21.15.136:58914) 2023-10-30T12:28:21.232 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f082affd640 1 -- 172.21.15.136:0/3689217098 learned_addr learned my addr 172.21.15.136:0/3689217098 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-10-30T12:28:21.232 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f082b7fe640 1 --2- 172.21.15.136:0/3689217098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f082c15a070 0x7f082c141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.233 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f082a7fc640 1 -- 172.21.15.136:0/3689217098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f082c0c2e20 msgr2=0x7f082c158b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.233 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f082a7fc640 1 --2- 172.21.15.136:0/3689217098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f082c0c2e20 0x7f082c158b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.233 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f082a7fc640 1 -- 172.21.15.136:0/3689217098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f082c15a070 msgr2=0x7f082c141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.233 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f082a7fc640 1 --2- 172.21.15.136:0/3689217098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f082c15a070 0x7f082c141740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.233 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f082a7fc640 1 -- 172.21.15.136:0/3689217098 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081c00bd40 con 0x7f082c150800 2023-10-30T12:28:21.234 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f082affd640 1 --2- 172.21.15.136:0/3689217098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f082c0c2e20 0x7f082c158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:21.234 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f082a7fc640 1 --2- 172.21.15.136:0/3689217098 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f082c150800 0x7f082c1eac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f081400e870 tx=0x7f081400ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:21.234 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f08237fe640 1 -- 172.21.15.136:0/3689217098 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08140095d0 con 0x7f082c150800 2023-10-30T12:28:21.234 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f08237fe640 1 -- 172.21.15.136:0/3689217098 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081400c8e0 con 0x7f082c150800 2023-10-30T12:28:21.234 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f08237fe640 1 -- 172.21.15.136:0/3689217098 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f081400cc50 con 0x7f082c150800 2023-10-30T12:28:21.235 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f083155e640 1 -- 172.21.15.136:0/3689217098 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f082c1ea780 con 0x7f082c150800 2023-10-30T12:28:21.235 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f083155e640 1 -- 172.21.15.136:0/3689217098 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f082c06b660 con 0x7f082c150800 2023-10-30T12:28:21.235 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f08237fe640 1 -- 172.21.15.136:0/3689217098 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0814013770 con 0x7f082c150800 2023-10-30T12:28:21.235 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f08237fe640 1 -- 172.21.15.136:0/3689217098 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f081401cbe0 con 0x7f082c150800 2023-10-30T12:28:21.235 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.230+0000 7f083155e640 1 -- 172.21.15.136:0/3689217098 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f082c1ea910 con 0x7f082c150800 2023-10-30T12:28:21.236 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.234+0000 7f08237fe640 1 -- 172.21.15.136:0/3689217098 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f081403bb40 con 0x7f082c150800 2023-10-30T12:28:21.300 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.302+0000 7f083155e640 1 -- 172.21.15.136:0/3689217098 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd new", "uuid": "74491459-06a1-488c-8a93-b04224b25521", "id": 2} v 0) v1 -- 0x7f082c04e170 con 0x7f082c150800 2023-10-30T12:28:21.308 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.310+0000 7f08237fe640 1 -- 172.21.15.136:0/3689217098 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "74491459-06a1-488c-8a93-b04224b25521", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f081402a030 con 0x7f082c150800 2023-10-30T12:28:21.308 INFO:teuthology.orchestra.run.smithi136.stdout:2 2023-10-30T12:28:21.309 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.310+0000 7f083155e640 1 -- 172.21.15.136:0/3689217098 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f082c150800 msgr2=0x7f082c1eac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.310 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.310+0000 7f083155e640 1 --2- 172.21.15.136:0/3689217098 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f082c150800 0x7f082c1eac10 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f081400e870 tx=0x7f081400ed30 comp rx=0 tx=0).stop 2023-10-30T12:28:21.310 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.310+0000 7f083155e640 1 -- 172.21.15.136:0/3689217098 shutdown_connections 2023-10-30T12:28:21.310 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.310+0000 7f083155e640 1 --2- 172.21.15.136:0/3689217098 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f082c15a070 0x7f082c141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.310 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.310+0000 7f083155e640 1 --2- 172.21.15.136:0/3689217098 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f082c150800 0x7f082c1eac10 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.310 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.310+0000 7f083155e640 1 --2- 172.21.15.136:0/3689217098 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f082c0c2e20 0x7f082c158b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.311 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.310+0000 7f083155e640 1 -- 172.21.15.136:0/3689217098 >> 172.21.15.136:0/3689217098 conn(0x7f082c06c850 msgr2=0x7f082c0c7170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:21.311 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.310+0000 7f083155e640 1 -- 172.21.15.136:0/3689217098 shutdown_connections 2023-10-30T12:28:21.311 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.310+0000 7f083155e640 1 -- 172.21.15.136:0/3689217098 wait complete. 2023-10-30T12:28:21.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 0290df24-0fff-4968-a73f-df7f903afb35 3 2023-10-30T12:28:21.438 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.438+0000 7f93db5a6640 1 Processor -- start 2023-10-30T12:28:21.438 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.438+0000 7f93db5a6640 1 -- start start 2023-10-30T12:28:21.440 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.438+0000 7f93db5a6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93d4151900 0x7f93d40c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.440 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.438+0000 7f93db5a6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f93d406e570 0x7f93d406d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.440 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.438+0000 7f93db5a6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f93d40c79a0 0x7f93d406f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.440 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.438+0000 7f93db5a6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d406eab0 con 0x7f93d406e570 2023-10-30T12:28:21.440 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.438+0000 7f93db5a6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93d406e360 con 0x7f93d40c79a0 2023-10-30T12:28:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.438+0000 7f93db5a6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d40c7480 con 0x7f93d4151900 2023-10-30T12:28:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93d9b43640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f93d40c79a0 0x7f93d406f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93d9342640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93d4151900 0x7f93d40c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93d9b43640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f93d40c79a0 0x7f93d406f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58916/0 (socket says 172.21.15.136:58916) 2023-10-30T12:28:21.441 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93d9342640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93d4151900 0x7f93d40c22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:55660/0 (socket says 172.21.15.136:55660) 2023-10-30T12:28:21.442 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93d9b43640 1 -- 172.21.15.136:0/3117710430 learned_addr learned my addr 172.21.15.136:0/3117710430 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-10-30T12:28:21.442 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93d8b41640 1 --2- 172.21.15.136:0/3117710430 >> v2:172.21.15.125:3300/0 conn(0x7f93d406e570 0x7f93d406d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.442 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93d9b43640 1 -- 172.21.15.136:0/3117710430 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93d4151900 msgr2=0x7f93d40c22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:21.442 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93d9b43640 1 --2- 172.21.15.136:0/3117710430 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93d4151900 0x7f93d40c22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.443 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93d9b43640 1 -- 172.21.15.136:0/3117710430 >> v2:172.21.15.125:3300/0 conn(0x7f93d406e570 msgr2=0x7f93d406d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:21.443 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93d9b43640 1 --2- 172.21.15.136:0/3117710430 >> v2:172.21.15.125:3300/0 conn(0x7f93d406e570 0x7f93d406d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.443 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93d9b43640 1 -- 172.21.15.136:0/3117710430 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d406d6f0 con 0x7f93d40c79a0 2023-10-30T12:28:21.443 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93d8b41640 1 --2- 172.21.15.136:0/3117710430 >> v2:172.21.15.125:3300/0 conn(0x7f93d406e570 0x7f93d406d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:21.443 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93d9342640 1 --2- 172.21.15.136:0/3117710430 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93d4151900 0x7f93d40c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:21.447 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93d9b43640 1 --2- 172.21.15.136:0/3117710430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d40c79a0 0x7f93d406f1c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f93d40b2700 tx=0x7f93cc01c9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=410a9164b575bddd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:21.447 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93cbfff640 1 -- 172.21.15.136:0/3117710430 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc0a5730 con 0x7f93d40c79a0 2023-10-30T12:28:21.447 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93cbfff640 1 -- 172.21.15.136:0/3117710430 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93cc0a5890 con 0x7f93d40c79a0 2023-10-30T12:28:21.447 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93cbfff640 1 -- 172.21.15.136:0/3117710430 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc0a5c20 con 0x7f93d40c79a0 2023-10-30T12:28:21.447 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93db5a6640 1 -- 172.21.15.136:0/3117710430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d40c79a0 msgr2=0x7f93d406f1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.448 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93db5a6640 1 --2- 172.21.15.136:0/3117710430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d40c79a0 0x7f93d406f1c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f93d40b2700 tx=0x7f93cc01c9a0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.448 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93db5a6640 1 -- 172.21.15.136:0/3117710430 shutdown_connections 2023-10-30T12:28:21.448 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93db5a6640 1 --2- 172.21.15.136:0/3117710430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d40c79a0 0x7f93d406f1c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.448 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93db5a6640 1 --2- 172.21.15.136:0/3117710430 >> v2:172.21.15.125:3300/0 conn(0x7f93d406e570 0x7f93d406d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.448 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93db5a6640 1 --2- 172.21.15.136:0/3117710430 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93d4151900 0x7f93d40c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.449 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93db5a6640 1 -- 172.21.15.136:0/3117710430 >> 172.21.15.136:0/3117710430 conn(0x7f93d40ccfe0 msgr2=0x7f93d40cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:21.449 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93db5a6640 1 -- 172.21.15.136:0/3117710430 shutdown_connections 2023-10-30T12:28:21.449 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.442+0000 7f93db5a6640 1 -- 172.21.15.136:0/3117710430 wait complete. 2023-10-30T12:28:21.449 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93db5a6640 1 Processor -- start 2023-10-30T12:28:21.449 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93db5a6640 1 -- start start 2023-10-30T12:28:21.450 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93db5a6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d406e570 0x7f93d41605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.450 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93db5a6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93d40c79a0 0x7f93d40c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.450 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93db5a6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93d4151900 0x7f93d4149c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.450 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93db5a6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d4157800 con 0x7f93d40c79a0 2023-10-30T12:28:21.450 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93db5a6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93d41f1830 con 0x7f93d406e570 2023-10-30T12:28:21.451 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93db5a6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93d41f1af0 con 0x7f93d4151900 2023-10-30T12:28:21.451 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93d9b43640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93d4151900 0x7f93d4149c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.451 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93d9342640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d406e570 0x7f93d41605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.451 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93d9b43640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93d4151900 0x7f93d4149c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:55666/0 (socket says 172.21.15.136:55666) 2023-10-30T12:28:21.451 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93d9b43640 1 -- 172.21.15.136:0/1367701748 learned_addr learned my addr 172.21.15.136:0/1367701748 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-10-30T12:28:21.451 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93d9342640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d406e570 0x7f93d41605c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58930/0 (socket says 172.21.15.136:58930) 2023-10-30T12:28:21.452 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93d8b41640 1 --2- 172.21.15.136:0/1367701748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93d40c79a0 0x7f93d40c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.452 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93d9b43640 1 -- 172.21.15.136:0/1367701748 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d406e570 msgr2=0x7f93d41605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.452 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93d9b43640 1 --2- 172.21.15.136:0/1367701748 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d406e570 0x7f93d41605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.452 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93d9b43640 1 -- 172.21.15.136:0/1367701748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93d40c79a0 msgr2=0x7f93d40c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.452 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93d9b43640 1 --2- 172.21.15.136:0/1367701748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93d40c79a0 0x7f93d40c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.453 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93d9b43640 1 -- 172.21.15.136:0/1367701748 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d0009420 con 0x7f93d4151900 2023-10-30T12:28:21.453 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93d9342640 1 --2- 172.21.15.136:0/1367701748 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d406e570 0x7f93d41605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:21.453 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93d9b43640 1 --2- 172.21.15.136:0/1367701748 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93d4151900 0x7f93d4149c00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f93cc0af040 tx=0x7f93cc019740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:21.453 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93c9ffb640 1 -- 172.21.15.136:0/1367701748 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc0b8670 con 0x7f93d4151900 2023-10-30T12:28:21.453 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93c9ffb640 1 -- 172.21.15.136:0/1367701748 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93cc0b87d0 con 0x7f93d4151900 2023-10-30T12:28:21.454 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93c9ffb640 1 -- 172.21.15.136:0/1367701748 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93cc0b8b80 con 0x7f93d4151900 2023-10-30T12:28:21.454 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93db5a6640 1 -- 172.21.15.136:0/1367701748 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93cc009b30 con 0x7f93d4151900 2023-10-30T12:28:21.454 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93db5a6640 1 -- 172.21.15.136:0/1367701748 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93d41f20b0 con 0x7f93d4151900 2023-10-30T12:28:21.454 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93c9ffb640 1 -- 172.21.15.136:0/1367701748 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f93cc0c3460 con 0x7f93d4151900 2023-10-30T12:28:21.454 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93c9ffb640 1 -- 172.21.15.136:0/1367701748 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f93cc0b68d0 con 0x7f93d4151900 2023-10-30T12:28:21.455 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.446+0000 7f93db5a6640 1 -- 172.21.15.136:0/1367701748 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93d404e170 con 0x7f93d4151900 2023-10-30T12:28:21.455 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.450+0000 7f93c9ffb640 1 -- 172.21.15.136:0/1367701748 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f93cc0b6440 con 0x7f93d4151900 2023-10-30T12:28:21.515 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.514+0000 7f93db5a6640 1 -- 172.21.15.136:0/1367701748 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0290df24-0fff-4968-a73f-df7f903afb35", "id": 3} v 0) v1 -- 0x7f93d40c8ac0 con 0x7f93d4151900 2023-10-30T12:28:21.521 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.522+0000 7f93c9ffb640 1 -- 172.21.15.136:0/1367701748 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0290df24-0fff-4968-a73f-df7f903afb35", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f93cc0b66e0 con 0x7f93d4151900 2023-10-30T12:28:21.521 INFO:teuthology.orchestra.run.smithi136.stdout:3 2023-10-30T12:28:21.523 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.522+0000 7f93db5a6640 1 -- 172.21.15.136:0/1367701748 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93d4151900 msgr2=0x7f93d4149c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.523 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.522+0000 7f93db5a6640 1 --2- 172.21.15.136:0/1367701748 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93d4151900 0x7f93d4149c00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f93cc0af040 tx=0x7f93cc019740 comp rx=0 tx=0).stop 2023-10-30T12:28:21.523 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.522+0000 7f93db5a6640 1 -- 172.21.15.136:0/1367701748 shutdown_connections 2023-10-30T12:28:21.523 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.522+0000 7f93db5a6640 1 --2- 172.21.15.136:0/1367701748 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93d4151900 0x7f93d4149c00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.523 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.522+0000 7f93db5a6640 1 --2- 172.21.15.136:0/1367701748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93d40c79a0 0x7f93d40c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.524 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.522+0000 7f93db5a6640 1 --2- 172.21.15.136:0/1367701748 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93d406e570 0x7f93d41605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.524 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.522+0000 7f93db5a6640 1 -- 172.21.15.136:0/1367701748 >> 172.21.15.136:0/1367701748 conn(0x7f93d40ccfe0 msgr2=0x7f93d40c4180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:21.524 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.522+0000 7f93db5a6640 1 -- 172.21.15.136:0/1367701748 shutdown_connections 2023-10-30T12:28:21.524 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.522+0000 7f93db5a6640 1 -- 172.21.15.136:0/1367701748 wait complete. 2023-10-30T12:28:21.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new be856cf6-9dc7-479a-83b5-14f05dd3de02 4 2023-10-30T12:28:21.649 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.650+0000 7fce6c877640 1 Processor -- start 2023-10-30T12:28:21.650 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.650+0000 7fce6c877640 1 -- start start 2023-10-30T12:28:21.651 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.650+0000 7fce6c877640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fce6806e570 0x7fce680c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.651 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.650+0000 7fce6c877640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce6806e360 con 0x7fce68151900 2023-10-30T12:28:21.651 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.650+0000 7fce6c877640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce680c7480 con 0x7fce680c79a0 2023-10-30T12:28:21.651 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.650+0000 7fce6c877640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce6806eab0 con 0x7fce6806e570 2023-10-30T12:28:21.652 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.650+0000 7fce5dd9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fce6806e570 0x7fce680c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.652 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.650+0000 7fce66d9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fce680c79a0 0x7fce6806f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52932/0 (socket says 172.21.15.136:52932) 2023-10-30T12:28:21.652 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.650+0000 7fce5dd9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fce6806e570 0x7fce680c22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58934/0 (socket says 172.21.15.136:58934) 2023-10-30T12:28:21.652 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.650+0000 7fce66d9d640 1 -- 172.21.15.136:0/1544220716 learned_addr learned my addr 172.21.15.136:0/1544220716 (peer_addr_for_me v1:172.21.15.136:0/0) 2023-10-30T12:28:21.652 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.650+0000 7fce5dd9b640 1 -- 172.21.15.136:0/1544220716 >> v1:172.21.15.136:6789/0 conn(0x7fce680c79a0 legacy=0x7fce6806f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.653 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.650+0000 7fce5dd9b640 1 -- 172.21.15.136:0/1544220716 >> v1:172.21.15.125:6789/0 conn(0x7fce68151900 legacy=0x7fce6806d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:21.653 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.650+0000 7fce5dd9b640 1 -- 172.21.15.136:0/1544220716 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce68158320 con 0x7fce6806e570 2023-10-30T12:28:21.654 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.654+0000 7fce5dd9b640 1 --2- 172.21.15.136:0/1544220716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce6806e570 0x7fce680c22f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fce680c7610 tx=0x7fce5001c410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94a59ebcb81cfda7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:21.654 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.654+0000 7fce65d9b640 1 -- 172.21.15.136:0/1544220716 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce500ab610 con 0x7fce6806e570 2023-10-30T12:28:21.654 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.654+0000 7fce65d9b640 1 -- 172.21.15.136:0/1544220716 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce500ab770 con 0x7fce6806e570 2023-10-30T12:28:21.654 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.654+0000 7fce65d9b640 1 -- 172.21.15.136:0/1544220716 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce500abb00 con 0x7fce6806e570 2023-10-30T12:28:21.655 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.654+0000 7fce6c877640 1 -- 172.21.15.136:0/1544220716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce6806e570 msgr2=0x7fce680c22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.655 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.654+0000 7fce6c877640 1 --2- 172.21.15.136:0/1544220716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce6806e570 0x7fce680c22f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fce680c7610 tx=0x7fce5001c410 comp rx=0 tx=0).stop 2023-10-30T12:28:21.655 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.654+0000 7fce6c877640 1 -- 172.21.15.136:0/1544220716 shutdown_connections 2023-10-30T12:28:21.655 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.654+0000 7fce6c877640 1 --2- 172.21.15.136:0/1544220716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce6806e570 0x7fce680c22f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.655 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.654+0000 7fce6c877640 1 -- 172.21.15.136:0/1544220716 >> 172.21.15.136:0/1544220716 conn(0x7fce680ccfe0 msgr2=0x7fce680cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:21.656 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.654+0000 7fce6c877640 1 -- 172.21.15.136:0/1544220716 shutdown_connections 2023-10-30T12:28:21.656 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.654+0000 7fce6c877640 1 -- 172.21.15.136:0/1544220716 wait complete. 2023-10-30T12:28:21.658 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce6c877640 1 Processor -- start 2023-10-30T12:28:21.658 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce6c877640 1 -- start start 2023-10-30T12:28:21.659 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce6c877640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce6806e570 0x7fce6814a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.659 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce6c877640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce680c79a0 0x7fce681f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.659 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce6c877640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce68151900 0x7fce681480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.659 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce6c877640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce68142db0 con 0x7fce6806e570 2023-10-30T12:28:21.659 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce6c877640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce68149ea0 con 0x7fce680c79a0 2023-10-30T12:28:21.660 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce6c877640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce68143050 con 0x7fce68151900 2023-10-30T12:28:21.660 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce5dd9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce680c79a0 0x7fce681f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.660 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce66d9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce68151900 0x7fce681480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.660 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce5dd9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce680c79a0 0x7fce681f3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58948/0 (socket says 172.21.15.136:58948) 2023-10-30T12:28:21.660 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce5dd9b640 1 -- 172.21.15.136:0/2404737092 learned_addr learned my addr 172.21.15.136:0/2404737092 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-10-30T12:28:21.660 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce6659c640 1 --2- 172.21.15.136:0/2404737092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce6806e570 0x7fce6814a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.661 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce5dd9b640 1 -- 172.21.15.136:0/2404737092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce68151900 msgr2=0x7fce681480f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.661 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce5dd9b640 1 --2- 172.21.15.136:0/2404737092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce68151900 0x7fce681480f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.661 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce5dd9b640 1 -- 172.21.15.136:0/2404737092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce6806e570 msgr2=0x7fce6814a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.661 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce5dd9b640 1 --2- 172.21.15.136:0/2404737092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce6806e570 0x7fce6814a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.661 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce5dd9b640 1 -- 172.21.15.136:0/2404737092 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce50009420 con 0x7fce680c79a0 2023-10-30T12:28:21.662 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce66d9d640 1 --2- 172.21.15.136:0/2404737092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce68151900 0x7fce681480f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:21.662 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce5dd9b640 1 --2- 172.21.15.136:0/2404737092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce680c79a0 0x7fce681f3900 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fce500ad040 tx=0x7fce5001ac50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:21.662 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce5f7fe640 1 -- 172.21.15.136:0/2404737092 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce500b5810 con 0x7fce680c79a0 2023-10-30T12:28:21.662 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce5f7fe640 1 -- 172.21.15.136:0/2404737092 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce500b5990 con 0x7fce680c79a0 2023-10-30T12:28:21.662 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce6c877640 1 -- 172.21.15.136:0/2404737092 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce68151e60 con 0x7fce680c79a0 2023-10-30T12:28:21.663 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce5f7fe640 1 -- 172.21.15.136:0/2404737092 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce500b5d40 con 0x7fce680c79a0 2023-10-30T12:28:21.663 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce6c877640 1 -- 172.21.15.136:0/2404737092 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce68143330 con 0x7fce680c79a0 2023-10-30T12:28:21.663 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce5f7fe640 1 -- 172.21.15.136:0/2404737092 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fce500bf410 con 0x7fce680c79a0 2023-10-30T12:28:21.663 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce5f7fe640 1 -- 172.21.15.136:0/2404737092 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fce500bfbd0 con 0x7fce680c79a0 2023-10-30T12:28:21.663 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.658+0000 7fce6c877640 1 -- 172.21.15.136:0/2404737092 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce6804e170 con 0x7fce680c79a0 2023-10-30T12:28:21.664 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.662+0000 7fce5f7fe640 1 -- 172.21.15.136:0/2404737092 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fce500bfd90 con 0x7fce680c79a0 2023-10-30T12:28:21.727 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.730+0000 7fce6c877640 1 -- 172.21.15.136:0/2404737092 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd new", "uuid": "be856cf6-9dc7-479a-83b5-14f05dd3de02", "id": 4} v 0) v1 -- 0x7fce68147e50 con 0x7fce680c79a0 2023-10-30T12:28:21.733 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.734+0000 7fce5f7fe640 1 -- 172.21.15.136:0/2404737092 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "be856cf6-9dc7-479a-83b5-14f05dd3de02", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fce500bf6c0 con 0x7fce680c79a0 2023-10-30T12:28:21.733 INFO:teuthology.orchestra.run.smithi136.stdout:4 2023-10-30T12:28:21.734 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.734+0000 7fce6c877640 1 -- 172.21.15.136:0/2404737092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce680c79a0 msgr2=0x7fce681f3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.734 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.734+0000 7fce6c877640 1 --2- 172.21.15.136:0/2404737092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce680c79a0 0x7fce681f3900 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fce500ad040 tx=0x7fce5001ac50 comp rx=0 tx=0).stop 2023-10-30T12:28:21.734 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.734+0000 7fce6c877640 1 -- 172.21.15.136:0/2404737092 shutdown_connections 2023-10-30T12:28:21.735 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.734+0000 7fce6c877640 1 --2- 172.21.15.136:0/2404737092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce68151900 0x7fce681480f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.735 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.734+0000 7fce6c877640 1 --2- 172.21.15.136:0/2404737092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce680c79a0 0x7fce681f3900 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.735 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.734+0000 7fce6c877640 1 --2- 172.21.15.136:0/2404737092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce6806e570 0x7fce6814a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.735 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.734+0000 7fce6c877640 1 -- 172.21.15.136:0/2404737092 >> 172.21.15.136:0/2404737092 conn(0x7fce680ccfe0 msgr2=0x7fce680c22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:21.735 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.734+0000 7fce6c877640 1 -- 172.21.15.136:0/2404737092 shutdown_connections 2023-10-30T12:28:21.736 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.734+0000 7fce6c877640 1 -- 172.21.15.136:0/2404737092 wait complete. 2023-10-30T12:28:21.736 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:21.732+0000 7f6caba49180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T12:28:21.749 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 906dc735-5893-48ac-9c97-f02ae207d1fb 5 2023-10-30T12:28:21.789 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:21.790+0000 7f0f681bb180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-30T12:28:21.798 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:21.792+0000 7f6caba49180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T12:28:21.845 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:21.846+0000 7f0f681bb180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-30T12:28:21.859 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:21.856+0000 7f6caba49180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T12:28:21.862 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.862+0000 7fa11ec5c640 1 Processor -- start 2023-10-30T12:28:21.862 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.862+0000 7fa11ec5c640 1 -- start start 2023-10-30T12:28:21.864 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.862+0000 7fa11ec5c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa118151900 0x7fa1180c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.864 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.862+0000 7fa11ec5c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa11806e570 0x7fa11806d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.864 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.862+0000 7fa11ec5c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1180c79a0 0x7fa11806f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.864 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.862+0000 7fa11ec5c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11806eab0 con 0x7fa11806e570 2023-10-30T12:28:21.864 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.862+0000 7fa11ec5c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa11806e360 con 0x7fa118151900 2023-10-30T12:28:21.864 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.862+0000 7fa11ec5c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1180c7480 con 0x7fa1180c79a0 2023-10-30T12:28:21.865 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.862+0000 7fa11c9f8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa118151900 0x7fa1180c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.865 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.862+0000 7fa11d1f9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1180c79a0 0x7fa11806f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.865 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.862+0000 7fa11c9f8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa118151900 0x7fa1180c22f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58960/0 (socket says 172.21.15.136:58960) 2023-10-30T12:28:21.865 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.862+0000 7fa11d1f9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1180c79a0 0x7fa11806f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:55688/0 (socket says 172.21.15.136:55688) 2023-10-30T12:28:21.865 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.862+0000 7fa11c9f8640 1 -- 172.21.15.136:0/2775316850 learned_addr learned my addr 172.21.15.136:0/2775316850 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-10-30T12:28:21.866 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa117fff640 1 --2- 172.21.15.136:0/2775316850 >> v2:172.21.15.125:3300/0 conn(0x7fa11806e570 0x7fa11806d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.866 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11c9f8640 1 -- 172.21.15.136:0/2775316850 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1180c79a0 msgr2=0x7fa11806f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:21.866 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11c9f8640 1 --2- 172.21.15.136:0/2775316850 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1180c79a0 0x7fa11806f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.866 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11c9f8640 1 -- 172.21.15.136:0/2775316850 >> v2:172.21.15.125:3300/0 conn(0x7fa11806e570 msgr2=0x7fa11806d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:21.866 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11c9f8640 1 --2- 172.21.15.136:0/2775316850 >> v2:172.21.15.125:3300/0 conn(0x7fa11806e570 0x7fa11806d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.867 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11c9f8640 1 -- 172.21.15.136:0/2775316850 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11806d6f0 con 0x7fa118151900 2023-10-30T12:28:21.867 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11d1f9640 1 --2- 172.21.15.136:0/2775316850 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1180c79a0 0x7fa11806f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:21.870 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11c9f8640 1 --2- 172.21.15.136:0/2775316850 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa118151900 0x7fa1180c22f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa118070990 tx=0x7fa10401c4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21968721bcacac2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:21.870 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa1177fe640 1 -- 172.21.15.136:0/2775316850 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1040aa5f0 con 0x7fa118151900 2023-10-30T12:28:21.870 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa1177fe640 1 -- 172.21.15.136:0/2775316850 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1040aa750 con 0x7fa118151900 2023-10-30T12:28:21.870 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa1177fe640 1 -- 172.21.15.136:0/2775316850 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1040aaae0 con 0x7fa118151900 2023-10-30T12:28:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11ec5c640 1 -- 172.21.15.136:0/2775316850 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa118151900 msgr2=0x7fa1180c22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11ec5c640 1 --2- 172.21.15.136:0/2775316850 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa118151900 0x7fa1180c22f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fa118070990 tx=0x7fa10401c4d0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11ec5c640 1 -- 172.21.15.136:0/2775316850 shutdown_connections 2023-10-30T12:28:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11ec5c640 1 --2- 172.21.15.136:0/2775316850 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1180c79a0 0x7fa11806f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.871 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11ec5c640 1 --2- 172.21.15.136:0/2775316850 >> v2:172.21.15.125:3300/0 conn(0x7fa11806e570 0x7fa11806d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11ec5c640 1 --2- 172.21.15.136:0/2775316850 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa118151900 0x7fa1180c22f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11ec5c640 1 -- 172.21.15.136:0/2775316850 >> 172.21.15.136:0/2775316850 conn(0x7fa1180ccfe0 msgr2=0x7fa1180cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11ec5c640 1 -- 172.21.15.136:0/2775316850 shutdown_connections 2023-10-30T12:28:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.866+0000 7fa11ec5c640 1 -- 172.21.15.136:0/2775316850 wait complete. 2023-10-30T12:28:21.872 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11ec5c640 1 Processor -- start 2023-10-30T12:28:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11ec5c640 1 -- start start 2023-10-30T12:28:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11ec5c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa11806e570 0x7fa1181f17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11ec5c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1180c79a0 0x7fa118157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11ec5c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa118151900 0x7fa118161640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11ec5c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1181f2710 con 0x7fa11806e570 2023-10-30T12:28:21.873 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11ec5c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1181f1e70 con 0x7fa118151900 2023-10-30T12:28:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11ec5c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1181f2150 con 0x7fa1180c79a0 2023-10-30T12:28:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa117fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1180c79a0 0x7fa118157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11d1f9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa118151900 0x7fa118161640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa117fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1180c79a0 0x7fa118157050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:55692/0 (socket says 172.21.15.136:55692) 2023-10-30T12:28:21.874 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa117fff640 1 -- 172.21.15.136:0/651707319 learned_addr learned my addr 172.21.15.136:0/651707319 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-10-30T12:28:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11c9f8640 1 --2- 172.21.15.136:0/651707319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa11806e570 0x7fa1181f17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11d1f9640 1 -- 172.21.15.136:0/651707319 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1180c79a0 msgr2=0x7fa118157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11d1f9640 1 --2- 172.21.15.136:0/651707319 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1180c79a0 0x7fa118157050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11d1f9640 1 -- 172.21.15.136:0/651707319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa11806e570 msgr2=0x7fa1181f17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.875 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11d1f9640 1 --2- 172.21.15.136:0/651707319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa11806e570 0x7fa1181f17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11d1f9640 1 -- 172.21.15.136:0/651707319 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa110009a70 con 0x7fa118151900 2023-10-30T12:28:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa117fff640 1 --2- 172.21.15.136:0/651707319 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1180c79a0 0x7fa118157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11c9f8640 1 --2- 172.21.15.136:0/651707319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa11806e570 0x7fa1181f17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11d1f9640 1 --2- 172.21.15.136:0/651707319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa118151900 0x7fa118161640 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa11000efc0 tx=0x7fa11000c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa1157fa640 1 -- 172.21.15.136:0/651707319 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11000cd30 con 0x7fa118151900 2023-10-30T12:28:21.876 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa1157fa640 1 -- 172.21.15.136:0/651707319 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11000f040 con 0x7fa118151900 2023-10-30T12:28:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11ec5c640 1 -- 172.21.15.136:0/651707319 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1040094d0 con 0x7fa118151900 2023-10-30T12:28:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa1157fa640 1 -- 172.21.15.136:0/651707319 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110003be0 con 0x7fa118151900 2023-10-30T12:28:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11ec5c640 1 -- 172.21.15.136:0/651707319 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa118156850 con 0x7fa118151900 2023-10-30T12:28:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa1157fa640 1 -- 172.21.15.136:0/651707319 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fa110014780 con 0x7fa118151900 2023-10-30T12:28:21.877 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa1157fa640 1 -- 172.21.15.136:0/651707319 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fa11001dda0 con 0x7fa118151900 2023-10-30T12:28:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.870+0000 7fa11ec5c640 1 -- 172.21.15.136:0/651707319 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa11804e170 con 0x7fa118151900 2023-10-30T12:28:21.878 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.874+0000 7fa1157fa640 1 -- 172.21.15.136:0/651707319 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fa110018030 con 0x7fa118151900 2023-10-30T12:28:21.900 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:21.902+0000 7f0f681bb180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-30T12:28:21.939 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.938+0000 7fa11ec5c640 1 -- 172.21.15.136:0/651707319 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd new", "uuid": "906dc735-5893-48ac-9c97-f02ae207d1fb", "id": 5} v 0) v1 -- 0x7fa118151460 con 0x7fa118151900 2023-10-30T12:28:21.947 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.946+0000 7fa1157fa640 1 -- 172.21.15.136:0/651707319 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "906dc735-5893-48ac-9c97-f02ae207d1fb", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa110014c10 con 0x7fa118151900 2023-10-30T12:28:21.947 INFO:teuthology.orchestra.run.smithi136.stdout:5 2023-10-30T12:28:21.948 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.950+0000 7fa11ec5c640 1 -- 172.21.15.136:0/651707319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa118151900 msgr2=0x7fa118161640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:21.948 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.950+0000 7fa11ec5c640 1 --2- 172.21.15.136:0/651707319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa118151900 0x7fa118161640 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa11000efc0 tx=0x7fa11000c490 comp rx=0 tx=0).stop 2023-10-30T12:28:21.949 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.950+0000 7fa11ec5c640 1 -- 172.21.15.136:0/651707319 shutdown_connections 2023-10-30T12:28:21.949 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.950+0000 7fa11ec5c640 1 --2- 172.21.15.136:0/651707319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa118151900 0x7fa118161640 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.949 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.950+0000 7fa11ec5c640 1 --2- 172.21.15.136:0/651707319 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1180c79a0 0x7fa118157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.949 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.950+0000 7fa11ec5c640 1 --2- 172.21.15.136:0/651707319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa11806e570 0x7fa1181f17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:21.949 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.950+0000 7fa11ec5c640 1 -- 172.21.15.136:0/651707319 >> 172.21.15.136:0/651707319 conn(0x7fa1180ccfe0 msgr2=0x7fa1180c4180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:21.950 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.950+0000 7fa11ec5c640 1 -- 172.21.15.136:0/651707319 shutdown_connections 2023-10-30T12:28:21.950 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:21.950+0000 7fa11ec5c640 1 -- 172.21.15.136:0/651707319 wait complete. 2023-10-30T12:28:21.950 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:21.944+0000 7f6caba49180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T12:28:21.964 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 69ecbffa-5402-4e0e-8833-f4334bd95935 6 2023-10-30T12:28:21.981 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:21.982+0000 7f0f681bb180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-30T12:28:22.008 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:22.004+0000 7f6caba49180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T12:28:22.035 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:22.038+0000 7f0f681bb180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-30T12:28:22.096 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:22.092+0000 7f6caba49180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T12:28:22.098 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.098+0000 7f8916728640 1 Processor -- start 2023-10-30T12:28:22.098 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.098+0000 7f8916728640 1 -- start start 2023-10-30T12:28:22.100 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8916728640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f89101519c0 0x7f891006d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.100 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8916728640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89100c8470 0x7f89100c6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.101 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8916728640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f891015ae60 con 0x7f89100c7300 2023-10-30T12:28:22.101 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8916728640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f891006a030 con 0x7f89101519c0 2023-10-30T12:28:22.101 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8916728640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89100c6a60 con 0x7f89100c8470 2023-10-30T12:28:22.101 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f89077fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89100c8470 0x7f89100c6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.101 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8914cc5640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f89100c7300 0x7f891006a450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52948/0 (socket says 172.21.15.136:52948) 2023-10-30T12:28:22.102 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8914cc5640 1 -- 172.21.15.136:0/2632673368 learned_addr learned my addr 172.21.15.136:0/2632673368 (peer_addr_for_me v1:172.21.15.136:0/0) 2023-10-30T12:28:22.102 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f890ffff640 1 --2- 172.21.15.136:0/2632673368 >> v2:172.21.15.125:3300/0 conn(0x7f89101519c0 0x7f891006d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.102 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f890f7fe640 1 -- 172.21.15.136:0/2632673368 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978451389 0 0) 0x7f891015ae60 con 0x7f89100c7300 2023-10-30T12:28:22.102 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f890f7fe640 1 -- 172.21.15.136:0/2632673368 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88fc003530 con 0x7f89100c7300 2023-10-30T12:28:22.102 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f89077fe640 1 -- 172.21.15.136:0/2632673368 >> v2:172.21.15.125:3300/0 conn(0x7f89101519c0 msgr2=0x7f891006d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:22.103 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f89077fe640 1 --2- 172.21.15.136:0/2632673368 >> v2:172.21.15.125:3300/0 conn(0x7f89101519c0 0x7f891006d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.103 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f89077fe640 1 -- 172.21.15.136:0/2632673368 >> v1:172.21.15.136:6789/0 conn(0x7f89100c7300 legacy=0x7f891006a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.103 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f89077fe640 1 -- 172.21.15.136:0/2632673368 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891015b140 con 0x7f89100c8470 2023-10-30T12:28:22.106 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f89077fe640 1 --2- 172.21.15.136:0/2632673368 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89100c8470 0x7f89100c6dc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f88f80097c0 tx=0x7f88f801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e10867a5597b1da8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:22.106 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f890f7fe640 1 -- 172.21.15.136:0/2632673368 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f80ad640 con 0x7f89100c8470 2023-10-30T12:28:22.107 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f890f7fe640 1 -- 172.21.15.136:0/2632673368 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f80ad7a0 con 0x7f89100c8470 2023-10-30T12:28:22.107 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f890f7fe640 1 -- 172.21.15.136:0/2632673368 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f80adb30 con 0x7f89100c8470 2023-10-30T12:28:22.107 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8916728640 1 -- 172.21.15.136:0/2632673368 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89100c8470 msgr2=0x7f89100c6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.107 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8916728640 1 --2- 172.21.15.136:0/2632673368 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89100c8470 0x7f89100c6dc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f88f80097c0 tx=0x7f88f801c480 comp rx=0 tx=0).stop 2023-10-30T12:28:22.108 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8916728640 1 -- 172.21.15.136:0/2632673368 shutdown_connections 2023-10-30T12:28:22.108 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8916728640 1 --2- 172.21.15.136:0/2632673368 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89100c8470 0x7f89100c6dc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.108 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8916728640 1 --2- 172.21.15.136:0/2632673368 >> v2:172.21.15.125:3300/0 conn(0x7f89101519c0 0x7f891006d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.108 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8916728640 1 -- 172.21.15.136:0/2632673368 >> 172.21.15.136:0/2632673368 conn(0x7f891006c850 msgr2=0x7f891006cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:22.108 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8916728640 1 -- 172.21.15.136:0/2632673368 shutdown_connections 2023-10-30T12:28:22.109 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.102+0000 7f8916728640 1 -- 172.21.15.136:0/2632673368 wait complete. 2023-10-30T12:28:22.109 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f8916728640 1 Processor -- start 2023-10-30T12:28:22.109 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f8916728640 1 -- start start 2023-10-30T12:28:22.109 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f8916728640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89100c7300 0x7f891014dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.109 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f8916728640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89100c8470 0x7f89101f2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.109 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f8916728640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89101519c0 0x7f89101f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.110 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f8916728640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89101f2be0 con 0x7f89101519c0 2023-10-30T12:28:22.110 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f8916728640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8910071ba0 con 0x7f89100c8470 2023-10-30T12:28:22.110 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f8916728640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8910160a50 con 0x7f89100c7300 2023-10-30T12:28:22.110 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f89077fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89100c8470 0x7f89101f2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.111 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f890ffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89100c7300 0x7f891014dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.111 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f89077fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89100c8470 0x7f89101f2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.136:58984/0 (socket says 172.21.15.136:58984) 2023-10-30T12:28:22.111 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f89077fe640 1 -- 172.21.15.136:0/4044675445 learned_addr learned my addr 172.21.15.136:0/4044675445 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-10-30T12:28:22.111 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f8914cc5640 1 --2- 172.21.15.136:0/4044675445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89101519c0 0x7f89101f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.111 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f89077fe640 1 -- 172.21.15.136:0/4044675445 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89100c7300 msgr2=0x7f891014dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.112 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f89077fe640 1 --2- 172.21.15.136:0/4044675445 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89100c7300 0x7f891014dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.112 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f89077fe640 1 -- 172.21.15.136:0/4044675445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89101519c0 msgr2=0x7f89101f26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.112 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f89077fe640 1 --2- 172.21.15.136:0/4044675445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89101519c0 0x7f89101f26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.112 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f89077fe640 1 -- 172.21.15.136:0/4044675445 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f8009420 con 0x7f89100c8470 2023-10-30T12:28:22.112 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f890ffff640 1 --2- 172.21.15.136:0/4044675445 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89100c7300 0x7f891014dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:22.113 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f8914cc5640 1 --2- 172.21.15.136:0/4044675445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89101519c0 0x7f89101f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:22.113 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f89077fe640 1 --2- 172.21.15.136:0/4044675445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89100c8470 0x7f89101f2160 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f88f80b0040 tx=0x7f88f8019220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:22.113 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f890d7fa640 1 -- 172.21.15.136:0/4044675445 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f80b6610 con 0x7f89100c8470 2023-10-30T12:28:22.113 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f890d7fa640 1 -- 172.21.15.136:0/4044675445 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88f80a6040 con 0x7f89100c8470 2023-10-30T12:28:22.113 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f890d7fa640 1 -- 172.21.15.136:0/4044675445 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f80b69d0 con 0x7f89100c8470 2023-10-30T12:28:22.113 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f8916728640 1 -- 172.21.15.136:0/4044675445 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8910160cf0 con 0x7f89100c8470 2023-10-30T12:28:22.114 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f8916728640 1 -- 172.21.15.136:0/4044675445 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89101f5a50 con 0x7f89100c8470 2023-10-30T12:28:22.114 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f890d7fa640 1 -- 172.21.15.136:0/4044675445 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f88f80b4040 con 0x7f89100c8470 2023-10-30T12:28:22.114 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f890d7fa640 1 -- 172.21.15.136:0/4044675445 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f88f80c2aa0 con 0x7f89100c8470 2023-10-30T12:28:22.115 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.106+0000 7f8916728640 1 -- 172.21.15.136:0/4044675445 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f891004e170 con 0x7f89100c8470 2023-10-30T12:28:22.115 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.110+0000 7f890d7fa640 1 -- 172.21.15.136:0/4044675445 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f88f80b2070 con 0x7f89100c8470 2023-10-30T12:28:22.117 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:22.118+0000 7f0f681bb180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-30T12:28:22.157 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:22.152+0000 7f6caba49180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T12:28:22.173 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:22.174+0000 7f0f681bb180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-30T12:28:22.178 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.178+0000 7f8916728640 1 -- 172.21.15.136:0/4044675445 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd new", "uuid": "69ecbffa-5402-4e0e-8833-f4334bd95935", "id": 6} v 0) v1 -- 0x7f891006d4c0 con 0x7f89100c8470 2023-10-30T12:28:22.182 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.182+0000 7f890d7fa640 1 -- 172.21.15.136:0/4044675445 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "69ecbffa-5402-4e0e-8833-f4334bd95935", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f88f80c2460 con 0x7f89100c8470 2023-10-30T12:28:22.183 INFO:teuthology.orchestra.run.smithi136.stdout:6 2023-10-30T12:28:22.184 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.186+0000 7f8916728640 1 -- 172.21.15.136:0/4044675445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89100c8470 msgr2=0x7f89101f2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.184 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.186+0000 7f8916728640 1 --2- 172.21.15.136:0/4044675445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89100c8470 0x7f89101f2160 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f88f80b0040 tx=0x7f88f8019220 comp rx=0 tx=0).stop 2023-10-30T12:28:22.184 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.186+0000 7f8916728640 1 -- 172.21.15.136:0/4044675445 shutdown_connections 2023-10-30T12:28:22.184 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.186+0000 7f8916728640 1 --2- 172.21.15.136:0/4044675445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89101519c0 0x7f89101f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.184 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.186+0000 7f8916728640 1 --2- 172.21.15.136:0/4044675445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89100c8470 0x7f89101f2160 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.185 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.186+0000 7f8916728640 1 --2- 172.21.15.136:0/4044675445 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89100c7300 0x7f891014dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.185 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.186+0000 7f8916728640 1 -- 172.21.15.136:0/4044675445 >> 172.21.15.136:0/4044675445 conn(0x7f891006c850 msgr2=0x7f891006a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:22.185 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.186+0000 7f8916728640 1 -- 172.21.15.136:0/4044675445 shutdown_connections 2023-10-30T12:28:22.185 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.186+0000 7f8916728640 1 -- 172.21.15.136:0/4044675445 wait complete. 2023-10-30T12:28:22.199 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new ff4daea0-1315-43b4-83ed-d32535b951e0 7 2023-10-30T12:28:22.314 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:22.308+0000 7f6caba49180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T12:28:22.317 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a58833640 1 Processor -- start 2023-10-30T12:28:22.318 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a58833640 1 -- start start 2023-10-30T12:28:22.319 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a58833640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2a54151900 0x7f2a540c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.319 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a58833640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a5406e360 con 0x7f2a5406e570 2023-10-30T12:28:22.319 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a58833640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a540c7480 con 0x7f2a540c79a0 2023-10-30T12:28:22.319 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a58833640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a5406eab0 con 0x7f2a54151900 2023-10-30T12:28:22.319 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a52d9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2a540c79a0 0x7f2a5406f1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:52962/0 (socket says 172.21.15.136:52962) 2023-10-30T12:28:22.320 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a52d9d640 1 -- 172.21.15.136:0/2229798548 learned_addr learned my addr 172.21.15.136:0/2229798548 (peer_addr_for_me v1:172.21.15.136:0/0) 2023-10-30T12:28:22.320 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a5259c640 1 --2- 172.21.15.136:0/2229798548 >> v2:172.21.15.125:3300/0 conn(0x7f2a54151900 0x7f2a540c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.320 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a5159a640 1 -- 172.21.15.136:0/2229798548 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188255737 0 0) 0x7f2a540c7480 con 0x7f2a540c79a0 2023-10-30T12:28:22.320 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a5159a640 1 -- 172.21.15.136:0/2229798548 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a3c003530 con 0x7f2a540c79a0 2023-10-30T12:28:22.321 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f0f681bb180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-30T12:28:22.323 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a5159a640 1 -- 172.21.15.136:0/2229798548 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3857523870 0 0) 0x7f2a3c003530 con 0x7f2a540c79a0 2023-10-30T12:28:22.323 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a5159a640 1 -- 172.21.15.136:0/2229798548 >> v2:172.21.15.125:3300/0 conn(0x7f2a54151900 msgr2=0x7f2a540c22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:22.323 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a5159a640 1 --2- 172.21.15.136:0/2229798548 >> v2:172.21.15.125:3300/0 conn(0x7f2a54151900 0x7f2a540c22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.324 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a5159a640 1 -- 172.21.15.136:0/2229798548 >> v1:172.21.15.125:6789/0 conn(0x7f2a5406e570 legacy=0x7f2a5406d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.324 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a5159a640 1 -- 172.21.15.136:0/2229798548 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a54158320 con 0x7f2a540c79a0 2023-10-30T12:28:22.324 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.318+0000 7f2a5159a640 1 -- 172.21.15.136:0/2229798548 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f2a44001680 con 0x7f2a540c79a0 2023-10-30T12:28:22.324 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a5159a640 1 -- 172.21.15.136:0/2229798548 >> v1:172.21.15.136:6789/0 conn(0x7f2a540c79a0 legacy=0x7f2a5406f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.324 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a5159a640 1 --2- 172.21.15.136:0/2229798548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a3c003980 0x7f2a3c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.325 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a5159a640 1 --2- 172.21.15.136:0/2229798548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a3c0047d0 0x7f2a3c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.325 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a5159a640 1 --2- 172.21.15.136:0/2229798548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a3c005190 0x7f2a3c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.325 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a5159a640 1 -- 172.21.15.136:0/2229798548 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5406eab0 con 0x7f2a3c005190 2023-10-30T12:28:22.325 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a5159a640 1 -- 172.21.15.136:0/2229798548 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c00c060 con 0x7f2a3c0047d0 2023-10-30T12:28:22.325 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a5159a640 1 -- 172.21.15.136:0/2229798548 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c00c220 con 0x7f2a3c003980 2023-10-30T12:28:22.325 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a51d9b640 1 --2- 172.21.15.136:0/2229798548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a3c0047d0 0x7f2a3c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.326 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a5259c640 1 --2- 172.21.15.136:0/2229798548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a3c003980 0x7f2a3c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.326 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a52d9d640 1 --2- 172.21.15.136:0/2229798548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a3c005190 0x7f2a3c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.326 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a51d9b640 1 -- 172.21.15.136:0/2229798548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a3c003980 msgr2=0x7f2a3c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:22.326 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a51d9b640 1 --2- 172.21.15.136:0/2229798548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a3c003980 0x7f2a3c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.327 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a51d9b640 1 -- 172.21.15.136:0/2229798548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a3c005190 msgr2=0x7f2a3c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:22.327 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a51d9b640 1 --2- 172.21.15.136:0/2229798548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a3c005190 0x7f2a3c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.327 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a51d9b640 1 -- 172.21.15.136:0/2229798548 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3c003530 con 0x7f2a3c0047d0 2023-10-30T12:28:22.327 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a52d9d640 1 --2- 172.21.15.136:0/2229798548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a3c005190 0x7f2a3c00b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:22.327 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a52d9d640 1 -- 172.21.15.136:0/2229798548 reap_dead start 2023-10-30T12:28:22.327 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a51d9b640 1 --2- 172.21.15.136:0/2229798548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a3c0047d0 0x7f2a3c004c50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2a480017b0 tx=0x7f2a4801c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=279b6f227ccd7366 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:22.328 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a5159a640 1 -- 172.21.15.136:0/2229798548 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a480052d0 con 0x7f2a3c0047d0 2023-10-30T12:28:22.328 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a5159a640 1 -- 172.21.15.136:0/2229798548 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a480a7050 con 0x7f2a3c0047d0 2023-10-30T12:28:22.328 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a5159a640 1 -- 172.21.15.136:0/2229798548 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a48003600 con 0x7f2a3c0047d0 2023-10-30T12:28:22.328 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a58833640 1 -- 172.21.15.136:0/2229798548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a3c0047d0 msgr2=0x7f2a3c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.328 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a58833640 1 --2- 172.21.15.136:0/2229798548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a3c0047d0 0x7f2a3c004c50 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2a480017b0 tx=0x7f2a4801c3a0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.329 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a58833640 1 -- 172.21.15.136:0/2229798548 shutdown_connections 2023-10-30T12:28:22.329 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a58833640 1 --2- 172.21.15.136:0/2229798548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a3c0047d0 0x7f2a3c004c50 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.329 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a58833640 1 -- 172.21.15.136:0/2229798548 >> 172.21.15.136:0/2229798548 conn(0x7f2a540ccfe0 msgr2=0x7f2a540cd3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:22.329 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a58833640 1 -- 172.21.15.136:0/2229798548 shutdown_connections 2023-10-30T12:28:22.329 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.322+0000 7f2a58833640 1 -- 172.21.15.136:0/2229798548 wait complete. 2023-10-30T12:28:22.332 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a58833640 1 Processor -- start 2023-10-30T12:28:22.332 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a58833640 1 -- start start 2023-10-30T12:28:22.332 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a58833640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a3c0047d0 0x7f2a541f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.332 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a58833640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a541588c0 0x7f2a541f2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.332 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a58833640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a54156680 0x7f2a54156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.333 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a58833640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a54155100 con 0x7f2a541588c0 2023-10-30T12:28:22.333 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a58833640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a54157040 con 0x7f2a54156680 2023-10-30T12:28:22.333 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a58833640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a54159e90 con 0x7f2a3c0047d0 2023-10-30T12:28:22.333 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a5259c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a3c0047d0 0x7f2a541f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.333 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a5259c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a3c0047d0 0x7f2a541f28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.136:55716/0 (socket says 172.21.15.136:55716) 2023-10-30T12:28:22.334 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a5259c640 1 -- 172.21.15.136:0/2310593261 learned_addr learned my addr 172.21.15.136:0/2310593261 (peer_addr_for_me v2:172.21.15.136:0/0) 2023-10-30T12:28:22.334 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a52d9d640 1 --2- 172.21.15.136:0/2310593261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a54156680 0x7f2a54156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.334 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a51d9b640 1 --2- 172.21.15.136:0/2310593261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a541588c0 0x7f2a541f2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.334 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a52d9d640 1 -- 172.21.15.136:0/2310593261 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a3c0047d0 msgr2=0x7f2a541f28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.334 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a52d9d640 1 --2- 172.21.15.136:0/2310593261 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a3c0047d0 0x7f2a541f28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.335 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a52d9d640 1 -- 172.21.15.136:0/2310593261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a541588c0 msgr2=0x7f2a541f2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.335 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a52d9d640 1 --2- 172.21.15.136:0/2310593261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a541588c0 0x7f2a541f2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.335 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a52d9d640 1 -- 172.21.15.136:0/2310593261 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5415a170 con 0x7f2a54156680 2023-10-30T12:28:22.335 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a51d9b640 1 --2- 172.21.15.136:0/2310593261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a541588c0 0x7f2a541f2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:22.335 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a5259c640 1 --2- 172.21.15.136:0/2310593261 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a3c0047d0 0x7f2a541f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:22.336 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a52d9d640 1 --2- 172.21.15.136:0/2310593261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a54156680 0x7f2a54156b00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2a54066a60 tx=0x7f2a4400eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:22.336 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a42ffd640 1 -- 172.21.15.136:0/2310593261 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4400eed0 con 0x7f2a54156680 2023-10-30T12:28:22.336 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a42ffd640 1 -- 172.21.15.136:0/2310593261 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4400c610 con 0x7f2a54156680 2023-10-30T12:28:22.336 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a42ffd640 1 -- 172.21.15.136:0/2310593261 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4400c9a0 con 0x7f2a54156680 2023-10-30T12:28:22.336 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a58833640 1 -- 172.21.15.136:0/2310593261 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a54154ab0 con 0x7f2a54156680 2023-10-30T12:28:22.336 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a58833640 1 -- 172.21.15.136:0/2310593261 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a540cc930 con 0x7f2a54156680 2023-10-30T12:28:22.337 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a42ffd640 1 -- 172.21.15.136:0/2310593261 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2a4401d460 con 0x7f2a54156680 2023-10-30T12:28:22.337 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a42ffd640 1 -- 172.21.15.136:0/2310593261 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f2a44013a80 con 0x7f2a54156680 2023-10-30T12:28:22.337 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.326+0000 7f2a58833640 1 -- 172.21.15.136:0/2310593261 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a54157840 con 0x7f2a54156680 2023-10-30T12:28:22.337 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.330+0000 7f2a42ffd640 1 -- 172.21.15.136:0/2310593261 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f2a44013c40 con 0x7f2a54156680 2023-10-30T12:28:22.382 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:22.382+0000 7f0f681bb180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T12:28:22.383 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:22.380+0000 7f6caba49180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-30T12:28:22.398 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.398+0000 7f2a58833640 1 -- 172.21.15.136:0/2310593261 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ff4daea0-1315-43b4-83ed-d32535b951e0", "id": 7} v 0) v1 -- 0x7f2a540c6750 con 0x7f2a54156680 2023-10-30T12:28:22.404 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.406+0000 7f2a42ffd640 1 -- 172.21.15.136:0/2310593261 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ff4daea0-1315-43b4-83ed-d32535b951e0", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2a44013de0 con 0x7f2a54156680 2023-10-30T12:28:22.404 INFO:teuthology.orchestra.run.smithi136.stdout:7 2023-10-30T12:28:22.405 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.406+0000 7f2a58833640 1 -- 172.21.15.136:0/2310593261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a54156680 msgr2=0x7f2a54156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.406 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.406+0000 7f2a58833640 1 --2- 172.21.15.136:0/2310593261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a54156680 0x7f2a54156b00 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2a54066a60 tx=0x7f2a4400eb90 comp rx=0 tx=0).stop 2023-10-30T12:28:22.406 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.406+0000 7f2a58833640 1 -- 172.21.15.136:0/2310593261 shutdown_connections 2023-10-30T12:28:22.406 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.406+0000 7f2a58833640 1 --2- 172.21.15.136:0/2310593261 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a54156680 0x7f2a54156b00 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.406 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.406+0000 7f2a58833640 1 --2- 172.21.15.136:0/2310593261 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a541588c0 0x7f2a541f2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.407 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.406+0000 7f2a58833640 1 --2- 172.21.15.136:0/2310593261 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a3c0047d0 0x7f2a541f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.407 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.406+0000 7f2a58833640 1 -- 172.21.15.136:0/2310593261 >> 172.21.15.136:0/2310593261 conn(0x7f2a540ccfe0 msgr2=0x7f2a54150c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:22.407 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.406+0000 7f2a58833640 1 -- 172.21.15.136:0/2310593261 shutdown_connections 2023-10-30T12:28:22.407 INFO:teuthology.orchestra.run.smithi136.stderr:2023-10-30T12:28:22.406+0000 7f2a58833640 1 -- 172.21.15.136:0/2310593261 wait complete. 2023-10-30T12:28:22.421 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-30T12:28:22.421 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-30T12:28:22.425 INFO:tasks.ceph.osd.0:Started 2023-10-30T12:28:22.425 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-30T12:28:22.426 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-30T12:28:22.429 INFO:tasks.ceph.osd.1:Started 2023-10-30T12:28:22.429 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-30T12:28:22.430 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-30T12:28:22.433 INFO:tasks.ceph.osd.2:Started 2023-10-30T12:28:22.434 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-30T12:28:22.434 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-30T12:28:22.439 INFO:tasks.ceph.osd.3:Started 2023-10-30T12:28:22.440 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-30T12:28:22.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-30T12:28:22.442 INFO:tasks.ceph.osd.4:Started 2023-10-30T12:28:22.443 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-30T12:28:22.443 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-30T12:28:22.445 INFO:tasks.ceph.osd.5:Started 2023-10-30T12:28:22.447 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-30T12:28:22.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-30T12:28:22.449 INFO:tasks.ceph.osd.6:Started 2023-10-30T12:28:22.450 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-30T12:28:22.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-30T12:28:22.452 INFO:tasks.ceph.osd.7:Started 2023-10-30T12:28:22.452 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:22.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff87171640 1 Processor -- start 2023-10-30T12:28:22.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff87171640 1 -- start start 2023-10-30T12:28:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff87171640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7eff801538f0 0x7eff80159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff87171640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff800cefa0 0x7eff800cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff87171640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff800ca6a0 con 0x7eff8015cd50 2023-10-30T12:28:22.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff87171640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff8015d300 con 0x7eff801538f0 2023-10-30T12:28:22.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff87171640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff8015d5e0 con 0x7eff800cefa0 2023-10-30T12:28:22.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff8596e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7eff8015cd50 0x7eff800c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44714/0 (socket says 172.21.15.125:44714) 2023-10-30T12:28:22.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff8596e640 1 -- 172.21.15.125:0/3338595538 learned_addr learned my addr 172.21.15.125:0/3338595538 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:22.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff8616f640 1 --2- 172.21.15.125:0/3338595538 >> v2:172.21.15.136:3300/0 conn(0x7eff801538f0 0x7eff80159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff86970640 1 --2- 172.21.15.125:0/3338595538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff800cefa0 0x7eff800cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff8516d640 1 -- 172.21.15.125:0/3338595538 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 716986914 0 0) 0x7eff800ca6a0 con 0x7eff8015cd50 2023-10-30T12:28:22.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff8516d640 1 -- 172.21.15.125:0/3338595538 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff6c003530 con 0x7eff8015cd50 2023-10-30T12:28:22.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff8616f640 1 -- 172.21.15.125:0/3338595538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff800cefa0 msgr2=0x7eff800cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:22.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff8616f640 1 --2- 172.21.15.125:0/3338595538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff800cefa0 0x7eff800cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff8616f640 1 -- 172.21.15.125:0/3338595538 >> v1:172.21.15.125:6789/0 conn(0x7eff8015cd50 legacy=0x7eff800c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.616+0000 7eff8616f640 1 -- 172.21.15.125:0/3338595538 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff8015d8c0 con 0x7eff801538f0 2023-10-30T12:28:22.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.620+0000 7eff8616f640 1 --2- 172.21.15.125:0/3338595538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff801538f0 0x7eff80159fb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7eff7c0097c0 tx=0x7eff7c01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d01ccbace48fccf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:22.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.620+0000 7eff8516d640 1 -- 172.21.15.125:0/3338595538 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c0a8970 con 0x7eff801538f0 2023-10-30T12:28:22.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.620+0000 7eff8516d640 1 -- 172.21.15.125:0/3338595538 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7c0a8ad0 con 0x7eff801538f0 2023-10-30T12:28:22.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.620+0000 7eff8516d640 1 -- 172.21.15.125:0/3338595538 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c0a8e60 con 0x7eff801538f0 2023-10-30T12:28:22.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.620+0000 7eff87171640 1 -- 172.21.15.125:0/3338595538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff801538f0 msgr2=0x7eff80159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.620+0000 7eff87171640 1 --2- 172.21.15.125:0/3338595538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff801538f0 0x7eff80159fb0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7eff7c0097c0 tx=0x7eff7c01ec70 comp rx=0 tx=0).stop 2023-10-30T12:28:22.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.620+0000 7eff87171640 1 -- 172.21.15.125:0/3338595538 shutdown_connections 2023-10-30T12:28:22.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.620+0000 7eff87171640 1 --2- 172.21.15.125:0/3338595538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff800cefa0 0x7eff800cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.620+0000 7eff87171640 1 --2- 172.21.15.125:0/3338595538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff801538f0 0x7eff80159fb0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.620+0000 7eff87171640 1 -- 172.21.15.125:0/3338595538 >> 172.21.15.125:0/3338595538 conn(0x7eff8006a640 msgr2=0x7eff80065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:22.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.620+0000 7eff87171640 1 -- 172.21.15.125:0/3338595538 shutdown_connections 2023-10-30T12:28:22.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.620+0000 7eff87171640 1 -- 172.21.15.125:0/3338595538 wait complete. 2023-10-30T12:28:22.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff87171640 1 Processor -- start 2023-10-30T12:28:22.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff87171640 1 -- start start 2023-10-30T12:28:22.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff87171640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff800cefa0 0x7eff801f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff87171640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff801538f0 0x7eff801513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff87171640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff8015cd50 0x7eff801eee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:22.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff87171640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff80151bc0 con 0x7eff8015cd50 2023-10-30T12:28:22.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff87171640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff80153100 con 0x7eff800cefa0 2023-10-30T12:28:22.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff87171640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff801efe50 con 0x7eff801538f0 2023-10-30T12:28:22.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff86970640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff8015cd50 0x7eff801eee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff86970640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff8015cd50 0x7eff801eee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46526/0 (socket says 172.21.15.125:46526) 2023-10-30T12:28:22.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff86970640 1 -- 172.21.15.125:0/4037597917 learned_addr learned my addr 172.21.15.125:0/4037597917 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:22.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff8616f640 1 --2- 172.21.15.125:0/4037597917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff800cefa0 0x7eff801f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff8596e640 1 --2- 172.21.15.125:0/4037597917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff801538f0 0x7eff801513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:22.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff86970640 1 -- 172.21.15.125:0/4037597917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff801538f0 msgr2=0x7eff801513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff86970640 1 --2- 172.21.15.125:0/4037597917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff801538f0 0x7eff801513f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff86970640 1 -- 172.21.15.125:0/4037597917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff800cefa0 msgr2=0x7eff801f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff86970640 1 --2- 172.21.15.125:0/4037597917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff800cefa0 0x7eff801f2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff86970640 1 -- 172.21.15.125:0/4037597917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c009420 con 0x7eff8015cd50 2023-10-30T12:28:22.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff86970640 1 --2- 172.21.15.125:0/4037597917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff8015cd50 0x7eff801eee50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7eff78009dd0 tx=0x7eff7800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:22.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff76ffd640 1 -- 172.21.15.125:0/4037597917 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff78014480 con 0x7eff8015cd50 2023-10-30T12:28:22.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff76ffd640 1 -- 172.21.15.125:0/4037597917 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7800f040 con 0x7eff8015cd50 2023-10-30T12:28:22.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff76ffd640 1 -- 172.21.15.125:0/4037597917 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff78014840 con 0x7eff8015cd50 2023-10-30T12:28:22.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.624+0000 7eff87171640 1 -- 172.21.15.125:0/4037597917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff801ef390 con 0x7eff8015cd50 2023-10-30T12:28:22.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.628+0000 7eff87171640 1 -- 172.21.15.125:0/4037597917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8015ada0 con 0x7eff8015cd50 2023-10-30T12:28:22.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.628+0000 7eff76ffd640 1 -- 172.21.15.125:0/4037597917 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7eff78014ca0 con 0x7eff8015cd50 2023-10-30T12:28:22.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.628+0000 7eff76ffd640 1 -- 172.21.15.125:0/4037597917 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7eff780133e0 con 0x7eff8015cd50 2023-10-30T12:28:22.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.628+0000 7eff87171640 1 -- 172.21.15.125:0/4037597917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff801597a0 con 0x7eff8015cd50 2023-10-30T12:28:22.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.628+0000 7eff76ffd640 1 -- 172.21.15.125:0/4037597917 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7eff78018020 con 0x7eff8015cd50 2023-10-30T12:28:22.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.684+0000 7eff87171640 1 -- 172.21.15.125:0/4037597917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff8004e170 con 0x7eff8015cd50 2023-10-30T12:28:22.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.684+0000 7eff76ffd640 1 -- 172.21.15.125:0/4037597917 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7eff7802b090 con 0x7eff8015cd50 2023-10-30T12:28:22.690 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:22.690 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":10,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:22.401009+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:22.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.684+0000 7eff87171640 1 -- 172.21.15.125:0/4037597917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff8015cd50 msgr2=0x7eff801eee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:22.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.684+0000 7eff87171640 1 --2- 172.21.15.125:0/4037597917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff8015cd50 0x7eff801eee50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7eff78009dd0 tx=0x7eff7800c490 comp rx=0 tx=0).stop 2023-10-30T12:28:22.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.684+0000 7eff87171640 1 -- 172.21.15.125:0/4037597917 shutdown_connections 2023-10-30T12:28:22.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.684+0000 7eff87171640 1 --2- 172.21.15.125:0/4037597917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff8015cd50 0x7eff801eee50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.684+0000 7eff87171640 1 --2- 172.21.15.125:0/4037597917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff801538f0 0x7eff801513f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.684+0000 7eff87171640 1 --2- 172.21.15.125:0/4037597917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff800cefa0 0x7eff801f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:22.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.684+0000 7eff87171640 1 -- 172.21.15.125:0/4037597917 >> 172.21.15.125:0/4037597917 conn(0x7eff8006a640 msgr2=0x7eff800c4740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:22.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.684+0000 7eff87171640 1 -- 172.21.15.125:0/4037597917 shutdown_connections 2023-10-30T12:28:22.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:22.684+0000 7eff87171640 1 -- 172.21.15.125:0/4037597917 wait complete. 2023-10-30T12:28:22.709 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-30T12:28:22.709 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-30T12:28:23.047 INFO:tasks.ceph.mgr.x.smithi136.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-30T12:28:23.047 INFO:tasks.ceph.mgr.x.smithi136.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-30T12:28:23.047 INFO:tasks.ceph.mgr.x.smithi136.stderr: from numpy import show_config as show_numpy_config 2023-10-30T12:28:23.057 INFO:tasks.ceph.osd.5.smithi136.stderr:2023-10-30T12:28:23.058+0000 7fb008737640 -1 Falling back to public interface 2023-10-30T12:28:23.061 INFO:tasks.ceph.osd.7.smithi136.stderr:2023-10-30T12:28:23.062+0000 7ff8d44ba640 -1 Falling back to public interface 2023-10-30T12:28:23.084 INFO:tasks.ceph.osd.0.smithi125.stderr:2023-10-30T12:28:23.080+0000 7fbf55982640 -1 Falling back to public interface 2023-10-30T12:28:23.092 INFO:tasks.ceph.osd.6.smithi136.stderr:2023-10-30T12:28:23.094+0000 7f0e22171640 -1 Falling back to public interface 2023-10-30T12:28:23.092 INFO:tasks.ceph.osd.4.smithi136.stderr:2023-10-30T12:28:23.094+0000 7f5342883640 -1 Falling back to public interface 2023-10-30T12:28:23.101 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:23.102+0000 7f0f681bb180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T12:28:23.127 INFO:tasks.ceph.mgr.y.smithi125.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-10-30T12:28:23.127 INFO:tasks.ceph.mgr.y.smithi125.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-10-30T12:28:23.128 INFO:tasks.ceph.mgr.y.smithi125.stderr: from numpy import show_config as show_numpy_config 2023-10-30T12:28:23.136 INFO:tasks.ceph.osd.2.smithi125.stderr:2023-10-30T12:28:23.132+0000 7f189502b640 -1 Falling back to public interface 2023-10-30T12:28:23.140 INFO:tasks.ceph.osd.1.smithi125.stderr:2023-10-30T12:28:23.136+0000 7fb6baed9640 -1 Falling back to public interface 2023-10-30T12:28:23.148 INFO:tasks.ceph.osd.3.smithi125.stderr:2023-10-30T12:28:23.144+0000 7f91cca95640 -1 Falling back to public interface 2023-10-30T12:28:23.169 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:23.164+0000 7f6caba49180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-30T12:28:23.254 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:23.254+0000 7f0f681bb180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:28:23.304 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:23.300+0000 7f6caba49180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:28:23.420 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:23.422+0000 7f0f681bb180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T12:28:23.477 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:23.472+0000 7f6caba49180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-30T12:28:23.484 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:23.486+0000 7f0f681bb180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T12:28:23.514 DEBUG:teuthology.orchestra.run.smithi125:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:23.552 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:23.548+0000 7f6caba49180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-30T12:28:23.619 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:23.618+0000 7f0f681bb180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:28:23.643 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.636+0000 7fba61dc5640 1 Processor -- start 2023-10-30T12:28:23.643 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.636+0000 7fba61dc5640 1 -- start start 2023-10-30T12:28:23.644 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba61dc5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fba5c1539f0 0x7fba5c0c5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:23.644 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba61dc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c0c9f90 0x7fba5c159740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:23.645 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba61dc5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba5c0c5390 con 0x7fba5c159c80 2023-10-30T12:28:23.645 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba61dc5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba5c0ca980 con 0x7fba5c1539f0 2023-10-30T12:28:23.645 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba61dc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c15a240 con 0x7fba5c0c9f90 2023-10-30T12:28:23.645 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5bfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fba5c159c80 0x7fba5c0cf080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44722/0 (socket says 172.21.15.125:44722) 2023-10-30T12:28:23.646 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5bfff640 1 -- 172.21.15.125:0/3759585799 learned_addr learned my addr 172.21.15.125:0/3759585799 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:23.646 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5affd640 1 --2- 172.21.15.125:0/3759585799 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c0c9f90 0x7fba5c159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:23.646 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5b7fe640 1 --2- 172.21.15.125:0/3759585799 >> v2:172.21.15.136:3300/0 conn(0x7fba5c1539f0 0x7fba5c0c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:23.648 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 -- 172.21.15.125:0/3759585799 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444395022 0 0) 0x7fba5c0c5390 con 0x7fba5c159c80 2023-10-30T12:28:23.648 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 -- 172.21.15.125:0/3759585799 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba44003530 con 0x7fba5c159c80 2023-10-30T12:28:23.649 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 -- 172.21.15.125:0/3759585799 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1182579786 0 0) 0x7fba44003530 con 0x7fba5c159c80 2023-10-30T12:28:23.649 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 -- 172.21.15.125:0/3759585799 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c0c9f90 msgr2=0x7fba5c159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:23.649 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 --2- 172.21.15.125:0/3759585799 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c0c9f90 0x7fba5c159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:23.649 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 -- 172.21.15.125:0/3759585799 >> v2:172.21.15.136:3300/0 conn(0x7fba5c1539f0 msgr2=0x7fba5c0c5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:23.649 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 --2- 172.21.15.125:0/3759585799 >> v2:172.21.15.136:3300/0 conn(0x7fba5c1539f0 0x7fba5c0c5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:23.650 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 -- 172.21.15.125:0/3759585799 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c15a520 con 0x7fba5c159c80 2023-10-30T12:28:23.650 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 -- 172.21.15.125:0/3759585799 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fba54001a00 con 0x7fba5c159c80 2023-10-30T12:28:23.650 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5b7fe640 1 --2- 172.21.15.125:0/3759585799 >> v2:172.21.15.136:3300/0 conn(0x7fba5c1539f0 0x7fba5c0c5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:23.650 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 -- 172.21.15.125:0/3759585799 >> v1:172.21.15.125:6789/0 conn(0x7fba5c159c80 legacy=0x7fba5c0cf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:23.650 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 --2- 172.21.15.125:0/3759585799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba44003980 0x7fba44003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:23.650 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 --2- 172.21.15.125:0/3759585799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba440047d0 0x7fba44004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:23.651 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 --2- 172.21.15.125:0/3759585799 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c0c9f90 0x7fba44005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:23.651 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 -- 172.21.15.125:0/3759585799 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c15a240 con 0x7fba440047d0 2023-10-30T12:28:23.651 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 -- 172.21.15.125:0/3759585799 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba4400b990 con 0x7fba44003980 2023-10-30T12:28:23.651 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5a7fc640 1 -- 172.21.15.125:0/3759585799 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba4400bad0 con 0x7fba5c0c9f90 2023-10-30T12:28:23.651 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5affd640 1 --2- 172.21.15.125:0/3759585799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba440047d0 0x7fba44004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:23.652 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5b7fe640 1 --2- 172.21.15.125:0/3759585799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba44003980 0x7fba44003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:23.652 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5bfff640 1 --2- 172.21.15.125:0/3759585799 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c0c9f90 0x7fba44005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:23.652 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5affd640 1 -- 172.21.15.125:0/3759585799 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c0c9f90 msgr2=0x7fba44005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:23.652 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5affd640 1 --2- 172.21.15.125:0/3759585799 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c0c9f90 0x7fba44005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:23.652 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5affd640 1 -- 172.21.15.125:0/3759585799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba44003980 msgr2=0x7fba44003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:23.653 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5affd640 1 --2- 172.21.15.125:0/3759585799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba44003980 0x7fba44003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:23.653 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5affd640 1 -- 172.21.15.125:0/3759585799 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba44003530 con 0x7fba440047d0 2023-10-30T12:28:23.653 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.640+0000 7fba5bfff640 1 --2- 172.21.15.125:0/3759585799 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c0c9f90 0x7fba44005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:23.656 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.644+0000 7fba5affd640 1 --2- 172.21.15.125:0/3759585799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba440047d0 0x7fba44004c50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba50009b10 tx=0x7fba5001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd2109c733910b6c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:23.656 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.644+0000 7fba5a7fc640 1 -- 172.21.15.125:0/3759585799 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba500a9450 con 0x7fba440047d0 2023-10-30T12:28:23.656 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.644+0000 7fba5a7fc640 1 -- 172.21.15.125:0/3759585799 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba500a95b0 con 0x7fba440047d0 2023-10-30T12:28:23.657 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.644+0000 7fba5a7fc640 1 -- 172.21.15.125:0/3759585799 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba500a9960 con 0x7fba440047d0 2023-10-30T12:28:23.657 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.644+0000 7fba61dc5640 1 -- 172.21.15.125:0/3759585799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba440047d0 msgr2=0x7fba44004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:23.657 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.644+0000 7fba61dc5640 1 --2- 172.21.15.125:0/3759585799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba440047d0 0x7fba44004c50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fba50009b10 tx=0x7fba5001ca50 comp rx=0 tx=0).stop 2023-10-30T12:28:23.657 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.644+0000 7fba5bfff640 1 -- 172.21.15.125:0/3759585799 reap_dead start 2023-10-30T12:28:23.657 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.644+0000 7fba61dc5640 1 -- 172.21.15.125:0/3759585799 shutdown_connections 2023-10-30T12:28:23.658 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.644+0000 7fba61dc5640 1 -- 172.21.15.125:0/3759585799 >> 172.21.15.125:0/3759585799 conn(0x7fba5c065bf0 msgr2=0x7fba5c066000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:23.658 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.644+0000 7fba61dc5640 1 -- 172.21.15.125:0/3759585799 shutdown_connections 2023-10-30T12:28:23.658 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.644+0000 7fba61dc5640 1 -- 172.21.15.125:0/3759585799 wait complete. 2023-10-30T12:28:23.658 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba61dc5640 1 Processor -- start 2023-10-30T12:28:23.658 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba61dc5640 1 -- start start 2023-10-30T12:28:23.659 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba61dc5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba440047d0 0x7fba5c15de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:23.659 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba61dc5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba5c1576e0 0x7fba5c154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:23.659 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba61dc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c1f2740 0x7fba5c1547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:23.659 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba61dc5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c14fc40 con 0x7fba5c1576e0 2023-10-30T12:28:23.659 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba61dc5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c1f1a50 con 0x7fba440047d0 2023-10-30T12:28:23.659 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba61dc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba5c1f22b0 con 0x7fba5c1f2740 2023-10-30T12:28:23.660 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba5affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba5c1576e0 0x7fba5c154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:23.660 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba5affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba5c1576e0 0x7fba5c154270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46548/0 (socket says 172.21.15.125:46548) 2023-10-30T12:28:23.660 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba5affd640 1 -- 172.21.15.125:0/1420792125 learned_addr learned my addr 172.21.15.125:0/1420792125 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:23.660 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba5b7fe640 1 --2- 172.21.15.125:0/1420792125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba440047d0 0x7fba5c15de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:23.660 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba5bfff640 1 --2- 172.21.15.125:0/1420792125 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c1f2740 0x7fba5c1547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:23.661 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba5affd640 1 -- 172.21.15.125:0/1420792125 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c1f2740 msgr2=0x7fba5c1547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:23.661 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba5affd640 1 --2- 172.21.15.125:0/1420792125 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c1f2740 0x7fba5c1547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:23.661 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba5affd640 1 -- 172.21.15.125:0/1420792125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba440047d0 msgr2=0x7fba5c15de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:23.661 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba5affd640 1 --2- 172.21.15.125:0/1420792125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba440047d0 0x7fba5c15de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:23.661 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba5affd640 1 -- 172.21.15.125:0/1420792125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5c14f960 con 0x7fba5c1576e0 2023-10-30T12:28:23.661 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba5bfff640 1 --2- 172.21.15.125:0/1420792125 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c1f2740 0x7fba5c1547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:23.662 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba5affd640 1 --2- 172.21.15.125:0/1420792125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba5c1576e0 0x7fba5c154270 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fba500ab040 tx=0x7fba500197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:23.662 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba43fff640 1 -- 172.21.15.125:0/1420792125 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba500bf070 con 0x7fba5c1576e0 2023-10-30T12:28:23.662 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba43fff640 1 -- 172.21.15.125:0/1420792125 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba5000f000 con 0x7fba5c1576e0 2023-10-30T12:28:23.662 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba61dc5640 1 -- 172.21.15.125:0/1420792125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba5c1f1c60 con 0x7fba5c1576e0 2023-10-30T12:28:23.662 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba43fff640 1 -- 172.21.15.125:0/1420792125 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba50004020 con 0x7fba5c1576e0 2023-10-30T12:28:23.662 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba61dc5640 1 -- 172.21.15.125:0/1420792125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba5c15e3a0 con 0x7fba5c1576e0 2023-10-30T12:28:23.663 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba43fff640 1 -- 172.21.15.125:0/1420792125 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fba50003620 con 0x7fba5c1576e0 2023-10-30T12:28:23.663 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.648+0000 7fba43fff640 1 -- 172.21.15.125:0/1420792125 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fba500b6070 con 0x7fba5c1576e0 2023-10-30T12:28:23.663 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.652+0000 7fba61dc5640 1 -- 172.21.15.125:0/1420792125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba5c04e170 con 0x7fba5c1576e0 2023-10-30T12:28:23.664 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.652+0000 7fba43fff640 1 -- 172.21.15.125:0/1420792125 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fba500096b0 con 0x7fba5c1576e0 2023-10-30T12:28:23.696 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:23.692+0000 7f6caba49180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-30T12:28:23.714 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.708+0000 7fba61dc5640 1 -- 172.21.15.125:0/1420792125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba5c157160 con 0x7fba5c1576e0 2023-10-30T12:28:23.715 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.712+0000 7fba43fff640 1 -- 172.21.15.125:0/1420792125 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fba50004480 con 0x7fba5c1576e0 2023-10-30T12:28:23.716 INFO:teuthology.misc.health.smithi125.stdout: 2023-10-30T12:28:23.716 INFO:teuthology.misc.health.smithi125.stdout:{"epoch":10,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:22.401009+0000","last_up_change":"0.000000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:23.717 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.712+0000 7fba61dc5640 1 -- 172.21.15.125:0/1420792125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba5c1576e0 msgr2=0x7fba5c154270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:23.717 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.712+0000 7fba61dc5640 1 --2- 172.21.15.125:0/1420792125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba5c1576e0 0x7fba5c154270 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fba500ab040 tx=0x7fba500197f0 comp rx=0 tx=0).stop 2023-10-30T12:28:23.717 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.712+0000 7fba61dc5640 1 -- 172.21.15.125:0/1420792125 shutdown_connections 2023-10-30T12:28:23.717 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.712+0000 7fba61dc5640 1 --2- 172.21.15.125:0/1420792125 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba5c1f2740 0x7fba5c1547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:23.717 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.712+0000 7fba61dc5640 1 --2- 172.21.15.125:0/1420792125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba5c1576e0 0x7fba5c154270 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:23.718 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.712+0000 7fba61dc5640 1 --2- 172.21.15.125:0/1420792125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba440047d0 0x7fba5c15de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:23.718 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.712+0000 7fba61dc5640 1 -- 172.21.15.125:0/1420792125 >> 172.21.15.125:0/1420792125 conn(0x7fba5c065bf0 msgr2=0x7fba5c157b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:23.718 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.712+0000 7fba61dc5640 1 -- 172.21.15.125:0/1420792125 shutdown_connections 2023-10-30T12:28:23.718 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:23.712+0000 7fba61dc5640 1 -- 172.21.15.125:0/1420792125 wait complete. 2023-10-30T12:28:23.734 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-30T12:28:24.248 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:24.250+0000 7f0f681bb180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T12:28:24.324 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:24.326+0000 7f0f681bb180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T12:28:24.395 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:24.388+0000 7f6caba49180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-30T12:28:24.482 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:24.476+0000 7f6caba49180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-30T12:28:24.495 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:24.494+0000 7f0f681bb180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T12:28:24.569 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:24.570+0000 7f0f681bb180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T12:28:24.675 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:24.668+0000 7f6caba49180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-30T12:28:24.760 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:24.756+0000 7f6caba49180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-30T12:28:24.859 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:24.862+0000 7f0f681bb180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T12:28:24.919 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:24.922+0000 7f0f681bb180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T12:28:25.097 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:25.092+0000 7f6caba49180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-30T12:28:25.166 INFO:tasks.ceph.mgr.y.smithi125.stderr:2023-10-30T12:28:25.160+0000 7f6caba49180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-30T12:28:25.970 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:25.970+0000 7f0f0f931640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-30T12:28:25.971 INFO:tasks.ceph.mgr.x.smithi136.stderr:2023-10-30T12:28:25.974+0000 7f0f0f931640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-30T12:28:26.833 INFO:tasks.ceph.osd.7.smithi136.stderr:2023-10-30T12:28:26.834+0000 7ff8d44ba640 -1 osd.7 0 log_to_monitors true 2023-10-30T12:28:26.923 INFO:tasks.ceph.osd.5.smithi136.stderr:2023-10-30T12:28:26.922+0000 7fb008737640 -1 osd.5 0 log_to_monitors true 2023-10-30T12:28:26.993 INFO:tasks.ceph.osd.1.smithi125.stderr:2023-10-30T12:28:26.988+0000 7fb6baed9640 -1 osd.1 0 log_to_monitors true 2023-10-30T12:28:26.999 INFO:tasks.ceph.osd.2.smithi125.stderr:2023-10-30T12:28:26.996+0000 7f189502b640 -1 osd.2 0 log_to_monitors true 2023-10-30T12:28:27.028 INFO:tasks.ceph.osd.6.smithi136.stderr:2023-10-30T12:28:27.029+0000 7f0e22171640 -1 osd.6 0 log_to_monitors true 2023-10-30T12:28:27.045 INFO:tasks.ceph.osd.4.smithi136.stderr:2023-10-30T12:28:27.045+0000 7f5342883640 -1 osd.4 0 log_to_monitors true 2023-10-30T12:28:27.076 INFO:tasks.ceph.osd.0.smithi125.stderr:2023-10-30T12:28:27.072+0000 7fbf55982640 -1 osd.0 0 log_to_monitors true 2023-10-30T12:28:27.124 INFO:tasks.ceph.osd.3.smithi125.stderr:2023-10-30T12:28:27.120+0000 7f91cca95640 -1 osd.3 0 log_to_monitors true 2023-10-30T12:28:29.151 INFO:tasks.ceph.osd.5.smithi136.stderr:2023-10-30T12:28:29.149+0000 7fb003ebc640 -1 osd.5 0 waiting for initial osdmap 2023-10-30T12:28:29.171 INFO:tasks.ceph.osd.7.smithi136.stderr:2023-10-30T12:28:29.169+0000 7ff8cfc3f640 -1 osd.7 0 waiting for initial osdmap 2023-10-30T12:28:29.184 INFO:tasks.ceph.osd.5.smithi136.stderr:2023-10-30T12:28:29.181+0000 7faffc503640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:29.222 INFO:tasks.ceph.osd.7.smithi136.stderr:2023-10-30T12:28:29.221+0000 7ff8c8286640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:30.162 INFO:tasks.ceph.osd.6.smithi136.stderr:2023-10-30T12:28:30.161+0000 7f0e1d8f6640 -1 osd.6 0 waiting for initial osdmap 2023-10-30T12:28:30.169 INFO:tasks.ceph.osd.4.smithi136.stderr:2023-10-30T12:28:30.169+0000 7f533e008640 -1 osd.4 0 waiting for initial osdmap 2023-10-30T12:28:30.193 INFO:tasks.ceph.osd.6.smithi136.stderr:2023-10-30T12:28:30.193+0000 7f0e15f3d640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:30.205 INFO:tasks.ceph.osd.4.smithi136.stderr:2023-10-30T12:28:30.205+0000 7f533664f640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:30.540 DEBUG:teuthology.orchestra.run.smithi125:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:30.751 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.744+0000 7f6e78479640 1 Processor -- start 2023-10-30T12:28:30.751 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.744+0000 7f6e78479640 1 -- start start 2023-10-30T12:28:30.752 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.748+0000 7f6e78479640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6e700c4af0 0x7f6e70070330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:30.752 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.748+0000 7f6e78479640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6e700b0750 0x7f6e700b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:30.753 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.748+0000 7f6e78479640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e70070a60 con 0x7f6e700c4ed0 2023-10-30T12:28:30.754 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.748+0000 7f6e78479640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e70070870 con 0x7f6e700b0750 2023-10-30T12:28:30.754 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.748+0000 7f6e78479640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e701570e0 con 0x7f6e700c4af0 2023-10-30T12:28:30.754 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.748+0000 7f6e76a16640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6e700b0750 0x7f6e700b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:30.754 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.748+0000 7f6e76a16640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6e700b0750 0x7f6e700b0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56322/0 (socket says 172.21.15.125:56322) 2023-10-30T12:28:30.754 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.748+0000 7f6e76a16640 1 -- 172.21.15.125:0/44121834 learned_addr learned my addr 172.21.15.125:0/44121834 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:30.755 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.748+0000 7f6e75a14640 1 --1- 172.21.15.125:0/44121834 >> v1:172.21.15.125:6789/0 conn(0x7f6e700c4ed0 0x7f6e70156630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40718/0 (socket says 172.21.15.125:40718) 2023-10-30T12:28:30.755 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.748+0000 7f6e76215640 1 --2- 172.21.15.125:0/44121834 >> v2:172.21.15.136:3300/0 conn(0x7f6e700c4af0 0x7f6e70070330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:30.755 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.748+0000 7f6e76a16640 1 -- 172.21.15.125:0/44121834 >> v2:172.21.15.136:3300/0 conn(0x7f6e700c4af0 msgr2=0x7f6e70070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:30.755 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.748+0000 7f6e76a16640 1 --2- 172.21.15.125:0/44121834 >> v2:172.21.15.136:3300/0 conn(0x7f6e700c4af0 0x7f6e70070330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:30.755 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.748+0000 7f6e76a16640 1 -- 172.21.15.125:0/44121834 >> v1:172.21.15.125:6789/0 conn(0x7f6e700c4ed0 legacy=0x7f6e70156630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:30.756 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.748+0000 7f6e76a16640 1 -- 172.21.15.125:0/44121834 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e701573c0 con 0x7f6e700b0750 2023-10-30T12:28:30.758 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.752+0000 7f6e76a16640 1 --2- 172.21.15.125:0/44121834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e700b0750 0x7f6e700b0bc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6e68009870 tx=0x7f6e6801e6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40697b1098784cf0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:30.759 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.752+0000 7f6e75213640 1 -- 172.21.15.125:0/44121834 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e680b2600 con 0x7f6e700b0750 2023-10-30T12:28:30.759 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.752+0000 7f6e75213640 1 -- 172.21.15.125:0/44121834 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e680b2760 con 0x7f6e700b0750 2023-10-30T12:28:30.759 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.752+0000 7f6e75213640 1 -- 172.21.15.125:0/44121834 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e680b2af0 con 0x7f6e700b0750 2023-10-30T12:28:30.760 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.756+0000 7f6e78479640 1 -- 172.21.15.125:0/44121834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e700b0750 msgr2=0x7f6e700b0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:30.760 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.756+0000 7f6e78479640 1 --2- 172.21.15.125:0/44121834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e700b0750 0x7f6e700b0bc0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6e68009870 tx=0x7f6e6801e6f0 comp rx=0 tx=0).stop 2023-10-30T12:28:30.761 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.756+0000 7f6e78479640 1 -- 172.21.15.125:0/44121834 shutdown_connections 2023-10-30T12:28:30.762 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.756+0000 7f6e78479640 1 --2- 172.21.15.125:0/44121834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e700b0750 0x7f6e700b0bc0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:30.762 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.756+0000 7f6e78479640 1 --2- 172.21.15.125:0/44121834 >> v2:172.21.15.136:3300/0 conn(0x7f6e700c4af0 0x7f6e70070330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:30.762 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.756+0000 7f6e78479640 1 -- 172.21.15.125:0/44121834 >> 172.21.15.125:0/44121834 conn(0x7f6e700cb0b0 msgr2=0x7f6e700cb4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:30.762 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.756+0000 7f6e78479640 1 -- 172.21.15.125:0/44121834 shutdown_connections 2023-10-30T12:28:30.762 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.756+0000 7f6e78479640 1 -- 172.21.15.125:0/44121834 wait complete. 2023-10-30T12:28:30.763 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e78479640 1 Processor -- start 2023-10-30T12:28:30.764 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e78479640 1 -- start start 2023-10-30T12:28:30.765 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e78479640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e700b0750 0x7f6e701556e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:30.766 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e78479640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e700c4af0 0x7f6e701ea100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:30.766 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e78479640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e700c4ed0 0x7f6e701ea640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:30.767 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e76215640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e700b0750 0x7f6e701556e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:30.767 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e76215640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e700b0750 0x7f6e701556e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:33936/0 (socket says 172.21.15.125:33936) 2023-10-30T12:28:30.767 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e78479640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e70156380 con 0x7f6e700c4ed0 2023-10-30T12:28:30.767 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e78479640 1 -- 172.21.15.125:0/2231992618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e701e9c70 con 0x7f6e700b0750 2023-10-30T12:28:30.768 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e76215640 1 -- 172.21.15.125:0/2231992618 learned_addr learned my addr 172.21.15.125:0/2231992618 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:30.768 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e78479640 1 -- 172.21.15.125:0/2231992618 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e701e96d0 con 0x7f6e700c4af0 2023-10-30T12:28:30.768 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e75a14640 1 --2- 172.21.15.125:0/2231992618 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e700c4af0 0x7f6e701ea100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:30.768 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e76a16640 1 --2- 172.21.15.125:0/2231992618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e700c4ed0 0x7f6e701ea640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:30.768 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e76215640 1 -- 172.21.15.125:0/2231992618 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e700c4af0 msgr2=0x7f6e701ea100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:30.769 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e76215640 1 --2- 172.21.15.125:0/2231992618 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e700c4af0 0x7f6e701ea100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:30.769 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e76215640 1 -- 172.21.15.125:0/2231992618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e700c4ed0 msgr2=0x7f6e701ea640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:30.769 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e76215640 1 --2- 172.21.15.125:0/2231992618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e700c4ed0 0x7f6e701ea640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:30.769 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.760+0000 7f6e76215640 1 -- 172.21.15.125:0/2231992618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e680094d0 con 0x7f6e700b0750 2023-10-30T12:28:30.769 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.764+0000 7f6e76215640 1 --2- 172.21.15.125:0/2231992618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e700b0750 0x7f6e701556e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00efc0 tx=0x7f6e6c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:30.770 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.764+0000 7f6e62ffd640 1 -- 172.21.15.125:0/2231992618 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c014480 con 0x7f6e700b0750 2023-10-30T12:28:30.770 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.764+0000 7f6e62ffd640 1 -- 172.21.15.125:0/2231992618 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c00f040 con 0x7f6e700b0750 2023-10-30T12:28:30.770 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.764+0000 7f6e62ffd640 1 -- 172.21.15.125:0/2231992618 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c014800 con 0x7f6e700b0750 2023-10-30T12:28:30.771 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.764+0000 7f6e78479640 1 -- 172.21.15.125:0/2231992618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e70156590 con 0x7f6e700b0750 2023-10-30T12:28:30.771 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.764+0000 7f6e78479640 1 -- 172.21.15.125:0/2231992618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7006a510 con 0x7f6e700b0750 2023-10-30T12:28:30.771 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.764+0000 7f6e78479640 1 -- 172.21.15.125:0/2231992618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e58005260 con 0x7f6e700b0750 2023-10-30T12:28:30.773 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.768+0000 7f6e62ffd640 1 -- 172.21.15.125:0/2231992618 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 4) v1 ==== 85794+0+0 (secure 0 0 0) 0x7f6e6c014960 con 0x7f6e700b0750 2023-10-30T12:28:30.774 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.768+0000 7f6e62ffd640 1 --2- 172.21.15.125:0/2231992618 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e540664e0 0x7f6e540689a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:30.775 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.768+0000 7f6e62ffd640 1 -- 172.21.15.125:0/2231992618 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2791+0+0 (secure 0 0 0) 0x7f6e6c084f10 con 0x7f6e700b0750 2023-10-30T12:28:30.775 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.768+0000 7f6e75a14640 1 --2- 172.21.15.125:0/2231992618 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e540664e0 0x7f6e540689a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:30.776 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.772+0000 7f6e75a14640 1 --2- 172.21.15.125:0/2231992618 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e540664e0 0x7f6e540689a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6e700af7c0 tx=0x7f6e64009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:30.786 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:30.780+0000 7f6e62ffd640 1 -- 172.21.15.125:0/2231992618 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6e6c052fb0 con 0x7f6e700b0750 2023-10-30T12:28:31.066 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:31.060+0000 7f6e78479640 1 -- 172.21.15.125:0/2231992618 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e580057e0 con 0x7f6e700b0750 2023-10-30T12:28:31.077 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:31.072+0000 7f6e62ffd640 1 -- 172.21.15.125:0/2231992618 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+8052 (secure 0 0 0) 0x7f6e6c018070 con 0x7f6e700b0750 2023-10-30T12:28:31.077 INFO:teuthology.misc.health.smithi125.stdout: 2023-10-30T12:28:31.078 INFO:teuthology.misc.health.smithi125.stdout:{"epoch":14,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:30.215316+0000","last_up_change":"2023-10-30T12:28:30.215316+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:31.080 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:31.076+0000 7f6e78479640 1 -- 172.21.15.125:0/2231992618 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e540664e0 msgr2=0x7f6e540689a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:31.080 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:31.076+0000 7f6e78479640 1 --2- 172.21.15.125:0/2231992618 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e540664e0 0x7f6e540689a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f6e700af7c0 tx=0x7f6e64009060 comp rx=0 tx=0).stop 2023-10-30T12:28:31.080 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:31.076+0000 7f6e78479640 1 -- 172.21.15.125:0/2231992618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e700b0750 msgr2=0x7f6e701556e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:31.080 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:31.076+0000 7f6e78479640 1 --2- 172.21.15.125:0/2231992618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e700b0750 0x7f6e701556e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c00efc0 tx=0x7f6e6c00c490 comp rx=0 tx=0).stop 2023-10-30T12:28:31.081 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:31.076+0000 7f6e78479640 1 -- 172.21.15.125:0/2231992618 shutdown_connections 2023-10-30T12:28:31.081 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:31.076+0000 7f6e78479640 1 --2- 172.21.15.125:0/2231992618 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e540664e0 0x7f6e540689a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:31.081 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:31.076+0000 7f6e78479640 1 --2- 172.21.15.125:0/2231992618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e700c4ed0 0x7f6e701ea640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:31.081 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:31.076+0000 7f6e78479640 1 --2- 172.21.15.125:0/2231992618 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e700c4af0 0x7f6e701ea100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:31.082 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:31.076+0000 7f6e78479640 1 --2- 172.21.15.125:0/2231992618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e700b0750 0x7f6e701556e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:31.082 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:31.076+0000 7f6e78479640 1 -- 172.21.15.125:0/2231992618 >> 172.21.15.125:0/2231992618 conn(0x7f6e700cb0b0 msgr2=0x7f6e700cb490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:31.082 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:31.076+0000 7f6e78479640 1 -- 172.21.15.125:0/2231992618 shutdown_connections 2023-10-30T12:28:31.082 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:31.076+0000 7f6e78479640 1 -- 172.21.15.125:0/2231992618 wait complete. 2023-10-30T12:28:31.111 DEBUG:teuthology.misc:2 of 8 OSDs are up 2023-10-30T12:28:31.471 INFO:tasks.ceph.osd.0.smithi125.stderr:2023-10-30T12:28:31.464+0000 7fbf5293f640 -1 osd.0 0 waiting for initial osdmap 2023-10-30T12:28:31.471 INFO:tasks.ceph.osd.3.smithi125.stderr:2023-10-30T12:28:31.468+0000 7f91c821c640 -1 osd.3 0 waiting for initial osdmap 2023-10-30T12:28:31.497 INFO:tasks.ceph.osd.3.smithi125.stderr:2023-10-30T12:28:31.492+0000 7f91c0863640 -1 osd.3 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:31.513 INFO:tasks.ceph.osd.0.smithi125.stderr:2023-10-30T12:28:31.508+0000 7fbf4974e640 -1 osd.0 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:31.524 INFO:tasks.ceph.osd.1.smithi125.stderr:2023-10-30T12:28:31.520+0000 7fb6b7e96640 -1 osd.1 0 waiting for initial osdmap 2023-10-30T12:28:31.543 INFO:tasks.ceph.osd.2.smithi125.stderr:2023-10-30T12:28:31.540+0000 7f188fd7a640 -1 osd.2 0 waiting for initial osdmap 2023-10-30T12:28:31.560 INFO:tasks.ceph.osd.1.smithi125.stderr:2023-10-30T12:28:31.556+0000 7fb6aeca5640 -1 osd.1 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:31.572 INFO:tasks.ceph.osd.2.smithi125.stderr:2023-10-30T12:28:31.568+0000 7f18883c1640 -1 osd.2 15 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-30T12:28:37.918 DEBUG:teuthology.orchestra.run.smithi125:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:38.056 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82de7c5640 1 Processor -- start 2023-10-30T12:28:38.056 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82de7c5640 1 -- start start 2023-10-30T12:28:38.057 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82de7c5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f82d81539f0 0x7f82d80c5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.058 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82de7c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82d80cf080 0x7f82d80cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.058 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82de7c5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82d80c5390 con 0x7f82d80c9f90 2023-10-30T12:28:38.058 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82de7c5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82d80ca980 con 0x7f82d81539f0 2023-10-30T12:28:38.058 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82de7c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d815a240 con 0x7f82d80cf080 2023-10-30T12:28:38.059 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82d7fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f82d81539f0 0x7f82d80c5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.059 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82d7fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f82d81539f0 0x7f82d80c5820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56344/0 (socket says 172.21.15.125:56344) 2023-10-30T12:28:38.059 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82d7fff640 1 -- 172.21.15.125:0/1007895585 learned_addr learned my addr 172.21.15.125:0/1007895585 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:38.059 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82dcd62640 1 --2- 172.21.15.125:0/1007895585 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82d80cf080 0x7f82d80cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.059 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82d7fff640 1 -- 172.21.15.125:0/1007895585 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82d80cf080 msgr2=0x7f82d80cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:38.059 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82d7fff640 1 --2- 172.21.15.125:0/1007895585 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82d80cf080 0x7f82d80cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.060 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82d7fff640 1 -- 172.21.15.125:0/1007895585 >> v1:172.21.15.125:6789/0 conn(0x7f82d80c9f90 legacy=0x7f82d8159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.060 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.056+0000 7f82d7fff640 1 -- 172.21.15.125:0/1007895585 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d815a520 con 0x7f82d81539f0 2023-10-30T12:28:38.061 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.060+0000 7f82d7fff640 1 --2- 172.21.15.125:0/1007895585 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82d81539f0 0x7f82d80c5820 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f82c8009870 tx=0x7f82c801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc57345e31049f43 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:38.061 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.060+0000 7f82d6ffd640 1 -- 172.21.15.125:0/1007895585 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c80a7070 con 0x7f82d81539f0 2023-10-30T12:28:38.061 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.060+0000 7f82d6ffd640 1 -- 172.21.15.125:0/1007895585 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c80ab430 con 0x7f82d81539f0 2023-10-30T12:28:38.061 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.060+0000 7f82d6ffd640 1 -- 172.21.15.125:0/1007895585 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c80ab7c0 con 0x7f82d81539f0 2023-10-30T12:28:38.062 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.060+0000 7f82de7c5640 1 -- 172.21.15.125:0/1007895585 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82d81539f0 msgr2=0x7f82d80c5820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.062 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.060+0000 7f82de7c5640 1 --2- 172.21.15.125:0/1007895585 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82d81539f0 0x7f82d80c5820 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f82c8009870 tx=0x7f82c801c540 comp rx=0 tx=0).stop 2023-10-30T12:28:38.063 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.060+0000 7f82de7c5640 1 -- 172.21.15.125:0/1007895585 shutdown_connections 2023-10-30T12:28:38.063 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.060+0000 7f82de7c5640 1 --2- 172.21.15.125:0/1007895585 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82d80cf080 0x7f82d80cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.063 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.060+0000 7f82de7c5640 1 --2- 172.21.15.125:0/1007895585 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82d81539f0 0x7f82d80c5820 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.063 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.060+0000 7f82de7c5640 1 -- 172.21.15.125:0/1007895585 >> 172.21.15.125:0/1007895585 conn(0x7f82d8065bf0 msgr2=0x7f82d8066000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:38.064 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.060+0000 7f82de7c5640 1 -- 172.21.15.125:0/1007895585 shutdown_connections 2023-10-30T12:28:38.064 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.060+0000 7f82de7c5640 1 -- 172.21.15.125:0/1007895585 wait complete. 2023-10-30T12:28:38.066 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.060+0000 7f82de7c5640 1 Processor -- start 2023-10-30T12:28:38.066 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82de7c5640 1 -- start start 2023-10-30T12:28:38.066 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82de7c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82d80c9f90 0x7f82d8152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.067 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82de7c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82d80cf080 0x7f82d815a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.067 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82de7c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82d81539f0 0x7f82d81f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.067 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82de7c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d8158600 con 0x7f82d81539f0 2023-10-30T12:28:38.067 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82de7c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82d81588e0 con 0x7f82d80c9f90 2023-10-30T12:28:38.067 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82de7c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82d8158bc0 con 0x7f82d80cf080 2023-10-30T12:28:38.068 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82dcd62640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82d81539f0 0x7f82d81f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.068 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82dcd62640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82d81539f0 0x7f82d81f54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56350/0 (socket says 172.21.15.125:56350) 2023-10-30T12:28:38.068 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82dcd62640 1 -- 172.21.15.125:0/3670574768 learned_addr learned my addr 172.21.15.125:0/3670574768 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:38.068 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82d7fff640 1 --2- 172.21.15.125:0/3670574768 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82d80c9f90 0x7f82d8152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.068 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82d77fe640 1 --2- 172.21.15.125:0/3670574768 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82d80cf080 0x7f82d815a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.069 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82dcd62640 1 -- 172.21.15.125:0/3670574768 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82d80cf080 msgr2=0x7f82d815a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.069 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82dcd62640 1 --2- 172.21.15.125:0/3670574768 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82d80cf080 0x7f82d815a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.069 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82dcd62640 1 -- 172.21.15.125:0/3670574768 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82d80c9f90 msgr2=0x7f82d8152b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.069 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82dcd62640 1 --2- 172.21.15.125:0/3670574768 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82d80c9f90 0x7f82d8152b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.069 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82dcd62640 1 -- 172.21.15.125:0/3670574768 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82c80094d0 con 0x7f82d81539f0 2023-10-30T12:28:38.070 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82dcd62640 1 --2- 172.21.15.125:0/3670574768 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82d81539f0 0x7f82d81f54e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f82d0007e40 tx=0x7f82d000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:38.070 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82d4ff9640 1 -- 172.21.15.125:0/3670574768 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d0014620 con 0x7f82d81539f0 2023-10-30T12:28:38.070 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82d4ff9640 1 -- 172.21.15.125:0/3670574768 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d000ee90 con 0x7f82d81539f0 2023-10-30T12:28:38.070 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82de7c5640 1 -- 172.21.15.125:0/3670574768 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82d8158ea0 con 0x7f82d81539f0 2023-10-30T12:28:38.070 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82d4ff9640 1 -- 172.21.15.125:0/3670574768 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d00149e0 con 0x7f82d81539f0 2023-10-30T12:28:38.071 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82de7c5640 1 -- 172.21.15.125:0/3670574768 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82d8159960 con 0x7f82d81539f0 2023-10-30T12:28:38.071 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82de7c5640 1 -- 172.21.15.125:0/3670574768 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82d8159350 con 0x7f82d81539f0 2023-10-30T12:28:38.072 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.064+0000 7f82d4ff9640 1 -- 172.21.15.125:0/3670574768 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82d0014b40 con 0x7f82d81539f0 2023-10-30T12:28:38.072 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.068+0000 7f82d4ff9640 1 --2- 172.21.15.125:0/3670574768 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82c40664a0 0x7f82c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.073 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.068+0000 7f82d4ff9640 1 -- 172.21.15.125:0/3670574768 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f82d0084670 con 0x7f82d81539f0 2023-10-30T12:28:38.073 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.068+0000 7f82d7fff640 1 --2- 172.21.15.125:0/3670574768 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82c40664a0 0x7f82c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.073 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.068+0000 7f82d7fff640 1 --2- 172.21.15.125:0/3670574768 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82c40664a0 0x7f82c4068960 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f82c80af000 tx=0x7f82c8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:38.085 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.084+0000 7f82d4ff9640 1 -- 172.21.15.125:0/3670574768 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82d0051d10 con 0x7f82d81539f0 2023-10-30T12:28:38.237 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.236+0000 7f82de7c5640 1 -- 172.21.15.125:0/3670574768 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82d804e170 con 0x7f82d81539f0 2023-10-30T12:28:38.238 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.236+0000 7f82d4ff9640 1 -- 172.21.15.125:0/3670574768 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12899 (secure 0 0 0) 0x7f82d0055330 con 0x7f82d81539f0 2023-10-30T12:28:38.238 INFO:teuthology.misc.health.smithi125.stdout: 2023-10-30T12:28:38.239 INFO:teuthology.misc.health.smithi125.stdout:{"epoch":18,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:34.424776+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:38.240 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.240+0000 7f82de7c5640 1 -- 172.21.15.125:0/3670574768 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82c40664a0 msgr2=0x7f82c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.240 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.240+0000 7f82de7c5640 1 --2- 172.21.15.125:0/3670574768 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82c40664a0 0x7f82c4068960 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f82c80af000 tx=0x7f82c8008040 comp rx=0 tx=0).stop 2023-10-30T12:28:38.240 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.240+0000 7f82de7c5640 1 -- 172.21.15.125:0/3670574768 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82d81539f0 msgr2=0x7f82d81f54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.241 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.240+0000 7f82de7c5640 1 --2- 172.21.15.125:0/3670574768 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82d81539f0 0x7f82d81f54e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f82d0007e40 tx=0x7f82d000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.241 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.240+0000 7f82de7c5640 1 -- 172.21.15.125:0/3670574768 shutdown_connections 2023-10-30T12:28:38.241 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.240+0000 7f82de7c5640 1 --2- 172.21.15.125:0/3670574768 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82c40664a0 0x7f82c4068960 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.241 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.240+0000 7f82de7c5640 1 --2- 172.21.15.125:0/3670574768 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82d81539f0 0x7f82d81f54e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.241 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.240+0000 7f82de7c5640 1 --2- 172.21.15.125:0/3670574768 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82d80cf080 0x7f82d815a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.242 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.240+0000 7f82de7c5640 1 --2- 172.21.15.125:0/3670574768 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82d80c9f90 0x7f82d8152b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.242 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.240+0000 7f82de7c5640 1 -- 172.21.15.125:0/3670574768 >> 172.21.15.125:0/3670574768 conn(0x7f82d8065bf0 msgr2=0x7f82d80c4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:38.242 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.240+0000 7f82de7c5640 1 -- 172.21.15.125:0/3670574768 shutdown_connections 2023-10-30T12:28:38.242 INFO:teuthology.misc.health.smithi125.stderr:2023-10-30T12:28:38.240+0000 7f82de7c5640 1 -- 172.21.15.125:0/3670574768 wait complete. 2023-10-30T12:28:38.257 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-30T12:28:38.258 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-30T12:28:38.258 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-30T12:28:38.259 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-30T12:28:38.262 INFO:tasks.ceph.mds.a:Started 2023-10-30T12:28:38.263 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T12:28:38.263 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T12:28:38.266 INFO:tasks.ceph.mds.c:Started 2023-10-30T12:28:38.266 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T12:28:38.267 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T12:28:38.270 INFO:tasks.ceph.mds.e:Started 2023-10-30T12:28:38.271 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-30T12:28:38.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-30T12:28:38.274 INFO:tasks.ceph.mds.b:Started 2023-10-30T12:28:38.274 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T12:28:38.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T12:28:38.277 INFO:tasks.ceph.mds.d:Started 2023-10-30T12:28:38.277 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T12:28:38.277 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T12:28:38.280 INFO:tasks.ceph.mds.f:Started 2023-10-30T12:28:38.280 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-30T12:28:38.281 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:38.357 INFO:tasks.ceph.mds.d.smithi136.stdout:starting mds.d at 2023-10-30T12:28:38.358 INFO:tasks.ceph.mds.b.smithi136.stdout:starting mds.b at 2023-10-30T12:28:38.359 INFO:tasks.ceph.mds.f.smithi136.stdout:starting mds.f at 2023-10-30T12:28:38.360 INFO:tasks.ceph.mds.a.smithi125.stdout:starting mds.a at 2023-10-30T12:28:38.361 INFO:tasks.ceph.mds.e.smithi125.stdout:starting mds.e at 2023-10-30T12:28:38.361 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2023-10-30T12:28:38.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9bc99d640 1 Processor -- start 2023-10-30T12:28:38.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9bc99d640 1 -- start start 2023-10-30T12:28:38.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9bc99d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9b8069e90 0x7ff9b815a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9bc99d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff9b806a270 0x7ff9b80c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9bc99d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9b80c6c10 con 0x7ff9b806cb80 2023-10-30T12:28:38.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9bc99d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9b80c8570 con 0x7ff9b806a270 2023-10-30T12:28:38.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9bc99d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b80c6ef0 con 0x7ff9b8069e90 2023-10-30T12:28:38.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9b6d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff9b806cb80 0x7ff9b806d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40764/0 (socket says 172.21.15.125:40764) 2023-10-30T12:28:38.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9b6d9d640 1 -- 172.21.15.125:0/190394469 learned_addr learned my addr 172.21.15.125:0/190394469 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:38.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9b5d9b640 1 --2- 172.21.15.125:0/190394469 >> v2:172.21.15.136:3300/0 conn(0x7ff9b806a270 0x7ff9b80c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9b659c640 1 --2- 172.21.15.125:0/190394469 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9b8069e90 0x7ff9b815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9b559a640 1 -- 172.21.15.125:0/190394469 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317734464 0 0) 0x7ff9b80c6c10 con 0x7ff9b806cb80 2023-10-30T12:28:38.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9b559a640 1 -- 172.21.15.125:0/190394469 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9a0003530 con 0x7ff9b806cb80 2023-10-30T12:28:38.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9b5d9b640 1 -- 172.21.15.125:0/190394469 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9b8069e90 msgr2=0x7ff9b815a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:38.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9b5d9b640 1 --2- 172.21.15.125:0/190394469 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9b8069e90 0x7ff9b815a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9b5d9b640 1 -- 172.21.15.125:0/190394469 >> v1:172.21.15.125:6789/0 conn(0x7ff9b806cb80 legacy=0x7ff9b806d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.428+0000 7ff9b5d9b640 1 -- 172.21.15.125:0/190394469 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9b80c71d0 con 0x7ff9b806a270 2023-10-30T12:28:38.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.432+0000 7ff9b5d9b640 1 --2- 172.21.15.125:0/190394469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b806a270 0x7ff9b80c9cb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0097c0 tx=0x7ff9ac01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ae254b036c33a29 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:38.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.432+0000 7ff9b559a640 1 -- 172.21.15.125:0/190394469 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac0ab600 con 0x7ff9b806a270 2023-10-30T12:28:38.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.432+0000 7ff9b559a640 1 -- 172.21.15.125:0/190394469 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ac0a7050 con 0x7ff9b806a270 2023-10-30T12:28:38.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.432+0000 7ff9b559a640 1 -- 172.21.15.125:0/190394469 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ac0ab980 con 0x7ff9b806a270 2023-10-30T12:28:38.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.432+0000 7ff9bc99d640 1 -- 172.21.15.125:0/190394469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b806a270 msgr2=0x7ff9b80c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.432+0000 7ff9bc99d640 1 --2- 172.21.15.125:0/190394469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b806a270 0x7ff9b80c9cb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0097c0 tx=0x7ff9ac01c480 comp rx=0 tx=0).stop 2023-10-30T12:28:38.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.432+0000 7ff9bc99d640 1 -- 172.21.15.125:0/190394469 shutdown_connections 2023-10-30T12:28:38.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.432+0000 7ff9bc99d640 1 --2- 172.21.15.125:0/190394469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b806a270 0x7ff9b80c9cb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.432+0000 7ff9bc99d640 1 --2- 172.21.15.125:0/190394469 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9b8069e90 0x7ff9b815a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.432+0000 7ff9bc99d640 1 -- 172.21.15.125:0/190394469 >> 172.21.15.125:0/190394469 conn(0x7ff9b806dfd0 msgr2=0x7ff9b806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:38.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.432+0000 7ff9bc99d640 1 -- 172.21.15.125:0/190394469 shutdown_connections 2023-10-30T12:28:38.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.432+0000 7ff9bc99d640 1 -- 172.21.15.125:0/190394469 wait complete. 2023-10-30T12:28:38.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9bc99d640 1 Processor -- start 2023-10-30T12:28:38.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9bc99d640 1 -- start start 2023-10-30T12:28:38.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9bc99d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9b8069e90 0x7ff9b8154b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9bc99d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b806a270 0x7ff9b80cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9bc99d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9b806cb80 0x7ff9b814c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9bc99d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b81552a0 con 0x7ff9b8069e90 2023-10-30T12:28:38.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9bc99d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b814a810 con 0x7ff9b806a270 2023-10-30T12:28:38.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9bc99d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9b80cbb20 con 0x7ff9b806cb80 2023-10-30T12:28:38.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9b659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9b8069e90 0x7ff9b8154b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9b659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9b8069e90 0x7ff9b8154b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56402/0 (socket says 172.21.15.125:56402) 2023-10-30T12:28:38.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9b659c640 1 -- 172.21.15.125:0/3252198112 learned_addr learned my addr 172.21.15.125:0/3252198112 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:38.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9b5d9b640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b806a270 0x7ff9b80cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9b6d9d640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9b806cb80 0x7ff9b814c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9b659c640 1 -- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9b806cb80 msgr2=0x7ff9b814c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9b659c640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9b806cb80 0x7ff9b814c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9b659c640 1 -- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b806a270 msgr2=0x7ff9b80cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9b659c640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b806a270 0x7ff9b80cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9b659c640 1 -- 172.21.15.125:0/3252198112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac009420 con 0x7ff9b8069e90 2023-10-30T12:28:38.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9b6d9d640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9b806cb80 0x7ff9b814c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:38.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9b659c640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9b8069e90 0x7ff9b8154b00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800e930 tx=0x7ff9a800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:38.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9a6ffd640 1 -- 172.21.15.125:0/3252198112 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a8009690 con 0x7ff9b8069e90 2023-10-30T12:28:38.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9a6ffd640 1 -- 172.21.15.125:0/3252198112 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a800ca00 con 0x7ff9b8069e90 2023-10-30T12:28:38.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9a6ffd640 1 -- 172.21.15.125:0/3252198112 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a800cd70 con 0x7ff9b8069e90 2023-10-30T12:28:38.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9bc99d640 1 -- 172.21.15.125:0/3252198112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b8072080 con 0x7ff9b8069e90 2023-10-30T12:28:38.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.436+0000 7ff9bc99d640 1 -- 172.21.15.125:0/3252198112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9b8156890 con 0x7ff9b8069e90 2023-10-30T12:28:38.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.440+0000 7ff9bc99d640 1 -- 172.21.15.125:0/3252198112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9b804e170 con 0x7ff9b8069e90 2023-10-30T12:28:38.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.440+0000 7ff9a6ffd640 1 -- 172.21.15.125:0/3252198112 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff9a8013470 con 0x7ff9b8069e90 2023-10-30T12:28:38.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.440+0000 7ff9a6ffd640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9980664a0 0x7ff998068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.440+0000 7ff9a6ffd640 1 -- 172.21.15.125:0/3252198112 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff9a8083dc0 con 0x7ff9b8069e90 2023-10-30T12:28:38.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.440+0000 7ff9b5d9b640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9980664a0 0x7ff998068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.440+0000 7ff9b5d9b640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9980664a0 0x7ff998068960 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0ad000 tx=0x7ff9ac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:38.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.444+0000 7ff9a6ffd640 1 -- 172.21.15.125:0/3252198112 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff9a8051660 con 0x7ff9b8069e90 2023-10-30T12:28:38.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.592+0000 7ff9bc99d640 1 -- 172.21.15.125:0/3252198112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff9b806ef60 con 0x7ff9b8069e90 2023-10-30T12:28:38.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.596+0000 7ff9a6ffd640 1 -- 172.21.15.125:0/3252198112 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12899 (secure 0 0 0) 0x7ff9a8017090 con 0x7ff9b8069e90 2023-10-30T12:28:38.597 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:38.597 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":18,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:34.424776+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:38.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.596+0000 7ff9bc99d640 1 -- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9980664a0 msgr2=0x7ff998068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.596+0000 7ff9bc99d640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9980664a0 0x7ff998068960 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0ad000 tx=0x7ff9ac008040 comp rx=0 tx=0).stop 2023-10-30T12:28:38.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.596+0000 7ff9bc99d640 1 -- 172.21.15.125:0/3252198112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9b8069e90 msgr2=0x7ff9b8154b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.596+0000 7ff9bc99d640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9b8069e90 0x7ff9b8154b00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff9a800e930 tx=0x7ff9a800edf0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.596+0000 7ff9bc99d640 1 -- 172.21.15.125:0/3252198112 shutdown_connections 2023-10-30T12:28:38.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.596+0000 7ff9bc99d640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9980664a0 0x7ff998068960 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.596+0000 7ff9bc99d640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9b806cb80 0x7ff9b814c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.596+0000 7ff9bc99d640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9b806a270 0x7ff9b80cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.596+0000 7ff9bc99d640 1 --2- 172.21.15.125:0/3252198112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9b8069e90 0x7ff9b8154b00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.596+0000 7ff9bc99d640 1 -- 172.21.15.125:0/3252198112 >> 172.21.15.125:0/3252198112 conn(0x7ff9b806dfd0 msgr2=0x7ff9b80712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:38.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.596+0000 7ff9bc99d640 1 -- 172.21.15.125:0/3252198112 shutdown_connections 2023-10-30T12:28:38.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.596+0000 7ff9bc99d640 1 -- 172.21.15.125:0/3252198112 wait complete. 2023-10-30T12:28:38.617 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:31.063403+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:28:38.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:28:38.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.756+0000 7f44f011d640 1 Processor -- start 2023-10-30T12:28:38.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.756+0000 7f44f011d640 1 -- start start 2023-10-30T12:28:38.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.756+0000 7f44f011d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f44e80cd6b0 0x7f44e806afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.756+0000 7f44f011d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e806b4e0 con 0x7f44e81570d0 2023-10-30T12:28:38.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.756+0000 7f44f011d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e81599b0 con 0x7f44e806c9e0 2023-10-30T12:28:38.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.756+0000 7f44f011d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e815a630 con 0x7f44e80cd6b0 2023-10-30T12:28:38.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.756+0000 7f44edeb9640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f44e81570d0 0x7f44e8071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40774/0 (socket says 172.21.15.125:40774) 2023-10-30T12:28:38.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.760+0000 7f44edeb9640 1 -- 172.21.15.125:0/56735391 learned_addr learned my addr 172.21.15.125:0/56735391 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:38.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.760+0000 7f44ee6ba640 1 --2- 172.21.15.125:0/56735391 >> v2:172.21.15.125:3300/0 conn(0x7f44e80cd6b0 0x7f44e806afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.760+0000 7f44eceb7640 1 -- 172.21.15.125:0/56735391 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537339102 0 0) 0x7f44e4001120 con 0x7f44e81570d0 2023-10-30T12:28:38.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.760+0000 7f44eceb7640 1 -- 172.21.15.125:0/56735391 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d4003530 con 0x7f44e81570d0 2023-10-30T12:28:38.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.760+0000 7f44ee6ba640 1 -- 172.21.15.125:0/56735391 >> v1:172.21.15.136:6789/0 conn(0x7f44e806c9e0 legacy=0x7f44e80ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.760+0000 7f44ee6ba640 1 -- 172.21.15.125:0/56735391 >> v1:172.21.15.125:6789/0 conn(0x7f44e81570d0 legacy=0x7f44e8071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.760+0000 7f44ee6ba640 1 -- 172.21.15.125:0/56735391 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e8159c90 con 0x7f44e80cd6b0 2023-10-30T12:28:38.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.760+0000 7f44ee6ba640 1 --2- 172.21.15.125:0/56735391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44e80cd6b0 0x7f44e806afa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009d90 tx=0x7f44e001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71da4ceb9e79824 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:38.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.760+0000 7f44eceb7640 1 -- 172.21.15.125:0/56735391 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e00af600 con 0x7f44e80cd6b0 2023-10-30T12:28:38.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.760+0000 7f44eceb7640 1 -- 172.21.15.125:0/56735391 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e00ab050 con 0x7f44e80cd6b0 2023-10-30T12:28:38.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.760+0000 7f44eceb7640 1 -- 172.21.15.125:0/56735391 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e00af980 con 0x7f44e80cd6b0 2023-10-30T12:28:38.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 -- 172.21.15.125:0/56735391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44e80cd6b0 msgr2=0x7f44e806afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 --2- 172.21.15.125:0/56735391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44e80cd6b0 0x7f44e806afa0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009d90 tx=0x7f44e001f130 comp rx=0 tx=0).stop 2023-10-30T12:28:38.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 -- 172.21.15.125:0/56735391 shutdown_connections 2023-10-30T12:28:38.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 --2- 172.21.15.125:0/56735391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44e80cd6b0 0x7f44e806afa0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 -- 172.21.15.125:0/56735391 >> 172.21.15.125:0/56735391 conn(0x7f44e8069e90 msgr2=0x7f44e806a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:38.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 -- 172.21.15.125:0/56735391 shutdown_connections 2023-10-30T12:28:38.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 -- 172.21.15.125:0/56735391 wait complete. 2023-10-30T12:28:38.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 Processor -- start 2023-10-30T12:28:38.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 -- start start 2023-10-30T12:28:38.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44e806c9e0 0x7f44e81490f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f44e80cd6b0 0x7f44e81439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44e81570d0 0x7f44e8143ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e81ef740 con 0x7f44e806c9e0 2023-10-30T12:28:38.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80c3730 con 0x7f44e81570d0 2023-10-30T12:28:38.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44f011d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e81efa00 con 0x7f44e80cd6b0 2023-10-30T12:28:38.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44edeb9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44e806c9e0 0x7f44e81490f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44edeb9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44e806c9e0 0x7f44e81490f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56426/0 (socket says 172.21.15.125:56426) 2023-10-30T12:28:38.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44edeb9640 1 -- 172.21.15.125:0/3475465155 learned_addr learned my addr 172.21.15.125:0/3475465155 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:38.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44ee6ba640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44e81570d0 0x7f44e8143ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44ed6b8640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f44e80cd6b0 0x7f44e81439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44edeb9640 1 -- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f44e80cd6b0 msgr2=0x7f44e81439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44edeb9640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f44e80cd6b0 0x7f44e81439b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44edeb9640 1 -- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44e81570d0 msgr2=0x7f44e8143ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44edeb9640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44e81570d0 0x7f44e8143ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44edeb9640 1 -- 172.21.15.125:0/3475465155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e0009a70 con 0x7f44e806c9e0 2023-10-30T12:28:38.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44ed6b8640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f44e80cd6b0 0x7f44e81439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:38.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.764+0000 7f44edeb9640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44e806c9e0 0x7f44e81490f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f44e400c4d0 tx=0x7f44e400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:38.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.768+0000 7f44de7fc640 1 -- 172.21.15.125:0/3475465155 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4005cf0 con 0x7f44e806c9e0 2023-10-30T12:28:38.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.768+0000 7f44de7fc640 1 -- 172.21.15.125:0/3475465155 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e4005e50 con 0x7f44e806c9e0 2023-10-30T12:28:38.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.768+0000 7f44de7fc640 1 -- 172.21.15.125:0/3475465155 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e4007290 con 0x7f44e806c9e0 2023-10-30T12:28:38.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.768+0000 7f44f011d640 1 -- 172.21.15.125:0/3475465155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e81efce0 con 0x7f44e806c9e0 2023-10-30T12:28:38.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.768+0000 7f44f011d640 1 -- 172.21.15.125:0/3475465155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e81f02a0 con 0x7f44e806c9e0 2023-10-30T12:28:38.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.768+0000 7f44f011d640 1 -- 172.21.15.125:0/3475465155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e804e170 con 0x7f44e806c9e0 2023-10-30T12:28:38.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.768+0000 7f44de7fc640 1 -- 172.21.15.125:0/3475465155 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f44e40073f0 con 0x7f44e806c9e0 2023-10-30T12:28:38.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.768+0000 7f44de7fc640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f44cc0664a0 0x7f44cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:38.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.768+0000 7f44de7fc640 1 -- 172.21.15.125:0/3475465155 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f44e4083100 con 0x7f44e806c9e0 2023-10-30T12:28:38.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.768+0000 7f44ed6b8640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f44cc0664a0 0x7f44cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:38.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.772+0000 7f44ed6b8640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f44cc0664a0 0x7f44cc068960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f44e806b150 tx=0x7f44d8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:38.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.772+0000 7f44de7fc640 1 -- 172.21.15.125:0/3475465155 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f44e40507a0 con 0x7f44e806c9e0 2023-10-30T12:28:38.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.920+0000 7f44f011d640 1 -- 172.21.15.125:0/3475465155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f44e806b4e0 con 0x7f44e806c9e0 2023-10-30T12:28:38.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.920+0000 7f44de7fc640 1 -- 172.21.15.125:0/3475465155 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f44e806b4e0 con 0x7f44e806c9e0 2023-10-30T12:28:38.922 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2023-10-30T12:28:38.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.924+0000 7f44f011d640 1 -- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f44cc0664a0 msgr2=0x7f44cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.924+0000 7f44f011d640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f44cc0664a0 0x7f44cc068960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f44e806b150 tx=0x7f44d8009040 comp rx=0 tx=0).stop 2023-10-30T12:28:38.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.924+0000 7f44f011d640 1 -- 172.21.15.125:0/3475465155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44e806c9e0 msgr2=0x7f44e81490f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:38.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.924+0000 7f44f011d640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44e806c9e0 0x7f44e81490f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f44e400c4d0 tx=0x7f44e400c990 comp rx=0 tx=0).stop 2023-10-30T12:28:38.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.924+0000 7f44f011d640 1 -- 172.21.15.125:0/3475465155 shutdown_connections 2023-10-30T12:28:38.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.924+0000 7f44f011d640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f44cc0664a0 0x7f44cc068960 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.924+0000 7f44f011d640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f44e81570d0 0x7f44e8143ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.924+0000 7f44f011d640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f44e80cd6b0 0x7f44e81439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.924+0000 7f44f011d640 1 --2- 172.21.15.125:0/3475465155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f44e806c9e0 0x7f44e81490f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:38.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.924+0000 7f44f011d640 1 -- 172.21.15.125:0/3475465155 >> 172.21.15.125:0/3475465155 conn(0x7f44e8069e90 msgr2=0x7f44e806ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:38.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.924+0000 7f44f011d640 1 -- 172.21.15.125:0/3475465155 shutdown_connections 2023-10-30T12:28:38.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:38.924+0000 7f44f011d640 1 -- 172.21.15.125:0/3475465155 wait complete. 2023-10-30T12:28:38.942 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:28:39.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.080+0000 7f9ac8b1d640 1 Processor -- start 2023-10-30T12:28:39.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.080+0000 7f9ac8b1d640 1 -- start start 2023-10-30T12:28:39.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.080+0000 7f9ac8b1d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9ac40cf160 0x7f9ac40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:39.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.080+0000 7f9ac8b1d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac40cb800 con 0x7f9ac41538a0 2023-10-30T12:28:39.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.080+0000 7f9ac8b1d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac40cb330 con 0x7f9ac415d6f0 2023-10-30T12:28:39.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.080+0000 7f9ac8b1d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac40cad70 con 0x7f9ac40cf160 2023-10-30T12:28:39.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.080+0000 7f9ac259c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9ac41538a0 0x7f9ac40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40786/0 (socket says 172.21.15.125:40786) 2023-10-30T12:28:39.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.080+0000 7f9ac259c640 1 -- 172.21.15.125:0/2036012888 learned_addr learned my addr 172.21.15.125:0/2036012888 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:39.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.080+0000 7f9ac2d9d640 1 --2- 172.21.15.125:0/2036012888 >> v2:172.21.15.125:3300/0 conn(0x7f9ac40cf160 0x7f9ac40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:39.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.080+0000 7f9ac1d9b640 1 -- 172.21.15.125:0/2036012888 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616489602 0 0) 0x7f9ac40cb800 con 0x7f9ac41538a0 2023-10-30T12:28:39.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.080+0000 7f9ac1d9b640 1 -- 172.21.15.125:0/2036012888 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab0003530 con 0x7f9ac41538a0 2023-10-30T12:28:39.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.080+0000 7f9ac2d9d640 1 -- 172.21.15.125:0/2036012888 >> v1:172.21.15.136:6789/0 conn(0x7f9ac415d6f0 legacy=0x7f9ac415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:39.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.080+0000 7f9ac2d9d640 1 -- 172.21.15.125:0/2036012888 >> v1:172.21.15.125:6789/0 conn(0x7f9ac41538a0 legacy=0x7f9ac40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:39.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.080+0000 7f9ac2d9d640 1 -- 172.21.15.125:0/2036012888 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac40cb050 con 0x7f9ac40cf160 2023-10-30T12:28:39.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.084+0000 7f9ac2d9d640 1 --2- 172.21.15.125:0/2036012888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ac40cf160 0x7f9ac40cf590 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9abc009d90 tx=0x7f9abc01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63271081a6f201a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:39.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.084+0000 7f9ac1d9b640 1 -- 172.21.15.125:0/2036012888 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9abc0af600 con 0x7f9ac40cf160 2023-10-30T12:28:39.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.084+0000 7f9ac1d9b640 1 -- 172.21.15.125:0/2036012888 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9abc0ab050 con 0x7f9ac40cf160 2023-10-30T12:28:39.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.084+0000 7f9ac1d9b640 1 -- 172.21.15.125:0/2036012888 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9abc0af980 con 0x7f9ac40cf160 2023-10-30T12:28:39.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.084+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/2036012888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ac40cf160 msgr2=0x7f9ac40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:39.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.084+0000 7f9ac8b1d640 1 --2- 172.21.15.125:0/2036012888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ac40cf160 0x7f9ac40cf590 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9abc009d90 tx=0x7f9abc01f130 comp rx=0 tx=0).stop 2023-10-30T12:28:39.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.084+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/2036012888 shutdown_connections 2023-10-30T12:28:39.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.084+0000 7f9ac8b1d640 1 --2- 172.21.15.125:0/2036012888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ac40cf160 0x7f9ac40cf590 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:39.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.084+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/2036012888 >> 172.21.15.125:0/2036012888 conn(0x7f9ac406bfe0 msgr2=0x7f9ac406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:39.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.084+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/2036012888 shutdown_connections 2023-10-30T12:28:39.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.084+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/2036012888 wait complete. 2023-10-30T12:28:39.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac8b1d640 1 Processor -- start 2023-10-30T12:28:39.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac8b1d640 1 -- start start 2023-10-30T12:28:39.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac8b1d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ac40cf160 0x7f9ac40ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:39.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac8b1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ac41538a0 0x7f9ac40b3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:39.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac8b1d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ac415d6f0 0x7f9ac4148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:39.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac8b1d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac4149a30 con 0x7f9ac415d6f0 2023-10-30T12:28:39.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac8b1d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac40ce3a0 con 0x7f9ac40cf160 2023-10-30T12:28:39.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac8b1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac4149cd0 con 0x7f9ac41538a0 2023-10-30T12:28:39.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac2d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ac415d6f0 0x7f9ac4148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:39.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac2d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ac415d6f0 0x7f9ac4148690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56434/0 (socket says 172.21.15.125:56434) 2023-10-30T12:28:39.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac2d9d640 1 -- 172.21.15.125:0/1544619397 learned_addr learned my addr 172.21.15.125:0/1544619397 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:39.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac259c640 1 --2- 172.21.15.125:0/1544619397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ac40cf160 0x7f9ac40ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:39.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9abbfff640 1 --2- 172.21.15.125:0/1544619397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ac41538a0 0x7f9ac40b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:39.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac2d9d640 1 -- 172.21.15.125:0/1544619397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ac41538a0 msgr2=0x7f9ac40b3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:39.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac2d9d640 1 --2- 172.21.15.125:0/1544619397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ac41538a0 0x7f9ac40b3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:39.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac2d9d640 1 -- 172.21.15.125:0/1544619397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ac40cf160 msgr2=0x7f9ac40ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:39.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac2d9d640 1 --2- 172.21.15.125:0/1544619397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ac40cf160 0x7f9ac40ce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:39.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac2d9d640 1 -- 172.21.15.125:0/1544619397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9abc009a70 con 0x7f9ac415d6f0 2023-10-30T12:28:39.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac2d9d640 1 --2- 172.21.15.125:0/1544619397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ac415d6f0 0x7f9ac4148690 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9abc0b2040 tx=0x7f9abc01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:39.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9abaffd640 1 -- 172.21.15.125:0/1544619397 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9abc009450 con 0x7f9ac415d6f0 2023-10-30T12:28:39.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9abaffd640 1 -- 172.21.15.125:0/1544619397 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9abc0b9410 con 0x7f9ac415d6f0 2023-10-30T12:28:39.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/1544619397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac4149fb0 con 0x7f9ac415d6f0 2023-10-30T12:28:39.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9abaffd640 1 -- 172.21.15.125:0/1544619397 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9abc0b97c0 con 0x7f9ac415d6f0 2023-10-30T12:28:39.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.088+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/1544619397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac40ce680 con 0x7f9ac415d6f0 2023-10-30T12:28:39.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.092+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/1544619397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac41483b0 con 0x7f9ac415d6f0 2023-10-30T12:28:39.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.092+0000 7f9abaffd640 1 -- 172.21.15.125:0/1544619397 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9abc0b6050 con 0x7f9ac415d6f0 2023-10-30T12:28:39.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.092+0000 7f9abaffd640 1 --2- 172.21.15.125:0/1544619397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9aa40664a0 0x7f9aa4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:39.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.092+0000 7f9abaffd640 1 -- 172.21.15.125:0/1544619397 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9abc121ff0 con 0x7f9ac415d6f0 2023-10-30T12:28:39.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.092+0000 7f9ac259c640 1 --2- 172.21.15.125:0/1544619397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9aa40664a0 0x7f9aa4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:39.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.092+0000 7f9ac259c640 1 --2- 172.21.15.125:0/1544619397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9aa40664a0 0x7f9aa4068960 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9ab40011f0 tx=0x7f9ab4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:39.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.096+0000 7f9abaffd640 1 -- 172.21.15.125:0/1544619397 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9ac41483b0 con 0x7f9ac415d6f0 2023-10-30T12:28:39.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.280+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/1544619397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ac41483b0 con 0x7f9ac415d6f0 2023-10-30T12:28:39.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.280+0000 7f9abaffd640 1 -- 172.21.15.125:0/1544619397 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1327 (secure 0 0 0) 0x7f9ac41483b0 con 0x7f9ac415d6f0 2023-10-30T12:28:39.281 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:39.281 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4285,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2023-10-30T12:28:39.282 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 2 2023-10-30T12:28:39.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.280+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/1544619397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9aa40664a0 msgr2=0x7f9aa4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:39.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.280+0000 7f9ac8b1d640 1 --2- 172.21.15.125:0/1544619397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9aa40664a0 0x7f9aa4068960 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9ab40011f0 tx=0x7f9ab4009040 comp rx=0 tx=0).stop 2023-10-30T12:28:39.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.280+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/1544619397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ac415d6f0 msgr2=0x7f9ac4148690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:39.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.280+0000 7f9ac8b1d640 1 --2- 172.21.15.125:0/1544619397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ac415d6f0 0x7f9ac4148690 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9abc0b2040 tx=0x7f9abc01bed0 comp rx=0 tx=0).stop 2023-10-30T12:28:39.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.280+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/1544619397 shutdown_connections 2023-10-30T12:28:39.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.280+0000 7f9ac8b1d640 1 --2- 172.21.15.125:0/1544619397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9aa40664a0 0x7f9aa4068960 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:39.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.280+0000 7f9ac8b1d640 1 --2- 172.21.15.125:0/1544619397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ac415d6f0 0x7f9ac4148690 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:39.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.280+0000 7f9ac8b1d640 1 --2- 172.21.15.125:0/1544619397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ac41538a0 0x7f9ac40b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:39.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.280+0000 7f9ac8b1d640 1 --2- 172.21.15.125:0/1544619397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ac40cf160 0x7f9ac40ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:39.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.280+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/1544619397 >> 172.21.15.125:0/1544619397 conn(0x7f9ac406bfe0 msgr2=0x7f9ac40c5610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:39.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.280+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/1544619397 shutdown_connections 2023-10-30T12:28:39.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:39.280+0000 7f9ac8b1d640 1 -- 172.21.15.125:0/1544619397 wait complete. 2023-10-30T12:28:41.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:28:41.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.448+0000 7fb826009640 1 Processor -- start 2023-10-30T12:28:41.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.448+0000 7fb826009640 1 -- start start 2023-10-30T12:28:41.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb826009640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb820153d70 0x7fb82006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:41.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb826009640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb82006e470 0x7fb8200cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:41.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb826009640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb8200c8490 0x7fb8200cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:41.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb826009640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82006d780 con 0x7fb82006e470 2023-10-30T12:28:41.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb826009640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb82006afa0 con 0x7fb8200c8490 2023-10-30T12:28:41.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb826009640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb820154150 con 0x7fb820153d70 2023-10-30T12:28:41.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb81effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb82006e470 0x7fb8200cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:41.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb81effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb82006e470 0x7fb8200cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47324/0 (socket says 172.21.15.125:47324) 2023-10-30T12:28:41.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb81effd640 1 -- 172.21.15.125:0/3492670242 learned_addr learned my addr 172.21.15.125:0/3492670242 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:41.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb81ffff640 1 --2- 172.21.15.125:0/3492670242 >> v2:172.21.15.136:3300/0 conn(0x7fb8200c8490 0x7fb8200cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:41.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb81f7fe640 1 --2- 172.21.15.125:0/3492670242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb820153d70 0x7fb82006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:41.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb81effd640 1 -- 172.21.15.125:0/3492670242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb820153d70 msgr2=0x7fb82006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:41.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb81effd640 1 --2- 172.21.15.125:0/3492670242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb820153d70 0x7fb82006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb81effd640 1 -- 172.21.15.125:0/3492670242 >> v2:172.21.15.136:3300/0 conn(0x7fb8200c8490 msgr2=0x7fb8200cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:41.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb81effd640 1 --2- 172.21.15.125:0/3492670242 >> v2:172.21.15.136:3300/0 conn(0x7fb8200c8490 0x7fb8200cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb81effd640 1 -- 172.21.15.125:0/3492670242 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb82006e030 con 0x7fb82006e470 2023-10-30T12:28:41.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.452+0000 7fb81effd640 1 --2- 172.21.15.125:0/3492670242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb82006e470 0x7fb8200cce00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb8200b1c40 tx=0x7fb81401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=728cb71a9e613ae7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:41.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb81e7fc640 1 -- 172.21.15.125:0/3492670242 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8140aa5f0 con 0x7fb82006e470 2023-10-30T12:28:41.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb81e7fc640 1 -- 172.21.15.125:0/3492670242 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8140aa750 con 0x7fb82006e470 2023-10-30T12:28:41.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb81e7fc640 1 -- 172.21.15.125:0/3492670242 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8140aaae0 con 0x7fb82006e470 2023-10-30T12:28:41.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 -- 172.21.15.125:0/3492670242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb82006e470 msgr2=0x7fb8200cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:41.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 --2- 172.21.15.125:0/3492670242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb82006e470 0x7fb8200cce00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb8200b1c40 tx=0x7fb81401c4d0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 -- 172.21.15.125:0/3492670242 shutdown_connections 2023-10-30T12:28:41.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 --2- 172.21.15.125:0/3492670242 >> v2:172.21.15.136:3300/0 conn(0x7fb8200c8490 0x7fb8200cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 --2- 172.21.15.125:0/3492670242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb82006e470 0x7fb8200cce00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 --2- 172.21.15.125:0/3492670242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb820153d70 0x7fb82006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 -- 172.21.15.125:0/3492670242 >> 172.21.15.125:0/3492670242 conn(0x7fb820071ea0 msgr2=0x7fb820065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:41.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 -- 172.21.15.125:0/3492670242 shutdown_connections 2023-10-30T12:28:41.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 -- 172.21.15.125:0/3492670242 wait complete. 2023-10-30T12:28:41.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 Processor -- start 2023-10-30T12:28:41.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 -- start start 2023-10-30T12:28:41.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb82006e470 0x7fb82015b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:41.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8200c8490 0x7fb820156cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:41.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb820153d70 0x7fb8201f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:41.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8201f6ce0 con 0x7fb8200c8490 2023-10-30T12:28:41.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8200ccb60 con 0x7fb820153d70 2023-10-30T12:28:41.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb826009640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb820154730 con 0x7fb82006e470 2023-10-30T12:28:41.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.456+0000 7fb81effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8200c8490 0x7fb820156cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:41.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb81effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8200c8490 0x7fb820156cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47330/0 (socket says 172.21.15.125:47330) 2023-10-30T12:28:41.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb81effd640 1 -- 172.21.15.125:0/988166333 learned_addr learned my addr 172.21.15.125:0/988166333 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:41.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb81ffff640 1 --2- 172.21.15.125:0/988166333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb820153d70 0x7fb8201f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:41.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb81f7fe640 1 --2- 172.21.15.125:0/988166333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb82006e470 0x7fb82015b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:41.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb81effd640 1 -- 172.21.15.125:0/988166333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb82006e470 msgr2=0x7fb82015b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:41.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb81effd640 1 --2- 172.21.15.125:0/988166333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb82006e470 0x7fb82015b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb81effd640 1 -- 172.21.15.125:0/988166333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb820153d70 msgr2=0x7fb8201f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:41.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb81effd640 1 --2- 172.21.15.125:0/988166333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb820153d70 0x7fb8201f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb81effd640 1 -- 172.21.15.125:0/988166333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8140094d0 con 0x7fb8200c8490 2023-10-30T12:28:41.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb81effd640 1 --2- 172.21.15.125:0/988166333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8200c8490 0x7fb820156cf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb8140ac040 tx=0x7fb81401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:41.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb807fff640 1 -- 172.21.15.125:0/988166333 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8140aa5f0 con 0x7fb8200c8490 2023-10-30T12:28:41.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb807fff640 1 -- 172.21.15.125:0/988166333 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8140b3800 con 0x7fb8200c8490 2023-10-30T12:28:41.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb826009640 1 -- 172.21.15.125:0/988166333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8201f6a00 con 0x7fb8200c8490 2023-10-30T12:28:41.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb807fff640 1 -- 172.21.15.125:0/988166333 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8140b3b90 con 0x7fb8200c8490 2023-10-30T12:28:41.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb826009640 1 -- 172.21.15.125:0/988166333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb82015b530 con 0x7fb8200c8490 2023-10-30T12:28:41.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb826009640 1 -- 172.21.15.125:0/988166333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb82004e170 con 0x7fb8200c8490 2023-10-30T12:28:41.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.460+0000 7fb807fff640 1 -- 172.21.15.125:0/988166333 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb8140b3cf0 con 0x7fb8200c8490 2023-10-30T12:28:41.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.464+0000 7fb807fff640 1 --2- 172.21.15.125:0/988166333 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb80c0664a0 0x7fb80c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:41.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.464+0000 7fb807fff640 1 -- 172.21.15.125:0/988166333 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb8140a7070 con 0x7fb8200c8490 2023-10-30T12:28:41.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.464+0000 7fb81f7fe640 1 --2- 172.21.15.125:0/988166333 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb80c0664a0 0x7fb80c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:41.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.464+0000 7fb81f7fe640 1 --2- 172.21.15.125:0/988166333 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb80c0664a0 0x7fb80c068960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb81000d8d0 tx=0x7fb810008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:41.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.467+0000 7fb807fff640 1 -- 172.21.15.125:0/988166333 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb8140ebe20 con 0x7fb8200c8490 2023-10-30T12:28:41.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.643+0000 7fb826009640 1 -- 172.21.15.125:0/988166333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb820157360 con 0x7fb8200c8490 2023-10-30T12:28:41.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.647+0000 7fb807fff640 1 -- 172.21.15.125:0/988166333 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+5292 (secure 0 0 0) 0x7fb8140ef440 con 0x7fb8200c8490 2023-10-30T12:28:41.648 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:41.649 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3346501886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3346501886},{"type":"v1","addr":"172.21.15.136:6835","nonce":3346501886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4285,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4294,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-10-30T12:28:41.649 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 3 2023-10-30T12:28:41.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.647+0000 7fb826009640 1 -- 172.21.15.125:0/988166333 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb80c0664a0 msgr2=0x7fb80c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:41.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.647+0000 7fb826009640 1 --2- 172.21.15.125:0/988166333 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb80c0664a0 0x7fb80c068960 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb81000d8d0 tx=0x7fb810008040 comp rx=0 tx=0).stop 2023-10-30T12:28:41.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.647+0000 7fb826009640 1 -- 172.21.15.125:0/988166333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8200c8490 msgr2=0x7fb820156cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:41.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.647+0000 7fb826009640 1 --2- 172.21.15.125:0/988166333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8200c8490 0x7fb820156cf0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb8140ac040 tx=0x7fb81401ad10 comp rx=0 tx=0).stop 2023-10-30T12:28:41.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.647+0000 7fb826009640 1 -- 172.21.15.125:0/988166333 shutdown_connections 2023-10-30T12:28:41.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.647+0000 7fb826009640 1 --2- 172.21.15.125:0/988166333 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb80c0664a0 0x7fb80c068960 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.647+0000 7fb826009640 1 --2- 172.21.15.125:0/988166333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb820153d70 0x7fb8201f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.647+0000 7fb826009640 1 --2- 172.21.15.125:0/988166333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8200c8490 0x7fb820156cf0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.647+0000 7fb826009640 1 --2- 172.21.15.125:0/988166333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb82006e470 0x7fb82015b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.647+0000 7fb826009640 1 -- 172.21.15.125:0/988166333 >> 172.21.15.125:0/988166333 conn(0x7fb820071ea0 msgr2=0x7fb8200c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:41.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.647+0000 7fb826009640 1 -- 172.21.15.125:0/988166333 shutdown_connections 2023-10-30T12:28:41.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.647+0000 7fb826009640 1 -- 172.21.15.125:0/988166333 wait complete. 2023-10-30T12:28:41.669 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:41.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.803+0000 7f988833e640 1 Processor -- start 2023-10-30T12:28:41.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.803+0000 7f988833e640 1 -- start start 2023-10-30T12:28:41.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f988833e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f98801539a0 0x7f988015a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:41.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f988833e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9880070a00 0x7f9880070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:41.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f988833e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f988015a8c0 con 0x7f9880069e90 2023-10-30T12:28:41.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f988833e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988006fbb0 con 0x7f98801539a0 2023-10-30T12:28:41.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f988833e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98800713b0 con 0x7f9880070a00 2023-10-30T12:28:41.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f98860da640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f98801539a0 0x7f988015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:41.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f98868db640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9880069e90 0x7f988006a2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43232/0 (socket says 172.21.15.125:43232) 2023-10-30T12:28:41.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f98860da640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f98801539a0 0x7f988015a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47342/0 (socket says 172.21.15.125:47342) 2023-10-30T12:28:41.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f98868db640 1 -- 172.21.15.125:0/1051685065 learned_addr learned my addr 172.21.15.125:0/1051685065 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:41.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f98858d9640 1 --2- 172.21.15.125:0/1051685065 >> v2:172.21.15.136:3300/0 conn(0x7f9880070a00 0x7f9880070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:41.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f98860da640 1 -- 172.21.15.125:0/1051685065 >> v2:172.21.15.136:3300/0 conn(0x7f9880070a00 msgr2=0x7f9880070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:41.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f98860da640 1 --2- 172.21.15.125:0/1051685065 >> v2:172.21.15.136:3300/0 conn(0x7f9880070a00 0x7f9880070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f98860da640 1 -- 172.21.15.125:0/1051685065 >> v1:172.21.15.125:6789/0 conn(0x7f9880069e90 legacy=0x7f988006a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:41.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f98860da640 1 -- 172.21.15.125:0/1051685065 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9880153d80 con 0x7f98801539a0 2023-10-30T12:28:41.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.807+0000 7f98850d8640 1 -- 172.21.15.125:0/1051685065 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302962748 0 0) 0x7f988015a8c0 con 0x7f9880069e90 2023-10-30T12:28:41.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f98860da640 1 --2- 172.21.15.125:0/1051685065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98801539a0 0x7f988015a380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9880072640 tx=0x7f987c01c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb430285ebad209a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:41.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f98850d8640 1 -- 172.21.15.125:0/1051685065 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c0ac600 con 0x7f98801539a0 2023-10-30T12:28:41.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f98850d8640 1 -- 172.21.15.125:0/1051685065 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987c0a8050 con 0x7f98801539a0 2023-10-30T12:28:41.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f98850d8640 1 -- 172.21.15.125:0/1051685065 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c0ac980 con 0x7f98801539a0 2023-10-30T12:28:41.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 -- 172.21.15.125:0/1051685065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98801539a0 msgr2=0x7f988015a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:41.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 --2- 172.21.15.125:0/1051685065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98801539a0 0x7f988015a380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9880072640 tx=0x7f987c01c3a0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 -- 172.21.15.125:0/1051685065 shutdown_connections 2023-10-30T12:28:41.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 --2- 172.21.15.125:0/1051685065 >> v2:172.21.15.136:3300/0 conn(0x7f9880070a00 0x7f9880070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 --2- 172.21.15.125:0/1051685065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98801539a0 0x7f988015a380 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 -- 172.21.15.125:0/1051685065 >> 172.21.15.125:0/1051685065 conn(0x7f98800c7910 msgr2=0x7f98800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:41.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 -- 172.21.15.125:0/1051685065 shutdown_connections 2023-10-30T12:28:41.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 -- 172.21.15.125:0/1051685065 wait complete. 2023-10-30T12:28:41.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 Processor -- start 2023-10-30T12:28:41.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 -- start start 2023-10-30T12:28:41.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9880069e90 0x7f9880152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:41.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9880070a00 0x7f98801563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:41.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98801539a0 0x7f9880156910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:41.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9880158080 con 0x7f9880069e90 2023-10-30T12:28:41.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98800c1a00 con 0x7f98801539a0 2023-10-30T12:28:41.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f988833e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98801599e0 con 0x7f9880070a00 2023-10-30T12:28:41.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.811+0000 7f98860da640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9880069e90 0x7f9880152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:41.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f98860da640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9880069e90 0x7f9880152350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47358/0 (socket says 172.21.15.125:47358) 2023-10-30T12:28:41.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f98860da640 1 -- 172.21.15.125:0/4028290247 learned_addr learned my addr 172.21.15.125:0/4028290247 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:41.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f98858d9640 1 --2- 172.21.15.125:0/4028290247 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9880070a00 0x7f98801563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:41.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f98860da640 1 -- 172.21.15.125:0/4028290247 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9880070a00 msgr2=0x7f98801563d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:41.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f98860da640 1 --2- 172.21.15.125:0/4028290247 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9880070a00 0x7f98801563d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f98860da640 1 -- 172.21.15.125:0/4028290247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98801539a0 msgr2=0x7f9880156910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:41.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f98860da640 1 --2- 172.21.15.125:0/4028290247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98801539a0 0x7f9880156910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f98860da640 1 -- 172.21.15.125:0/4028290247 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c009420 con 0x7f9880069e90 2023-10-30T12:28:41.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f98860da640 1 --2- 172.21.15.125:0/4028290247 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9880069e90 0x7f9880152350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f987c019fd0 tx=0x7f987c01abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:41.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f9876ffd640 1 -- 172.21.15.125:0/4028290247 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c0b5830 con 0x7f9880069e90 2023-10-30T12:28:41.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f9876ffd640 1 -- 172.21.15.125:0/4028290247 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987c0b5990 con 0x7f9880069e90 2023-10-30T12:28:41.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f988833e640 1 -- 172.21.15.125:0/4028290247 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98800c1bf0 con 0x7f9880069e90 2023-10-30T12:28:41.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f9876ffd640 1 -- 172.21.15.125:0/4028290247 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c0b5d00 con 0x7f9880069e90 2023-10-30T12:28:41.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f988833e640 1 -- 172.21.15.125:0/4028290247 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9880159390 con 0x7f9880069e90 2023-10-30T12:28:41.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f988833e640 1 -- 172.21.15.125:0/4028290247 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9880159b70 con 0x7f9880069e90 2023-10-30T12:28:41.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f9876ffd640 1 -- 172.21.15.125:0/4028290247 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f987c0b3070 con 0x7f9880069e90 2023-10-30T12:28:41.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f9876ffd640 1 --2- 172.21.15.125:0/4028290247 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98640664d0 0x7f9864068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:41.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.815+0000 7f9876ffd640 1 -- 172.21.15.125:0/4028290247 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f987c11e540 con 0x7f9880069e90 2023-10-30T12:28:41.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.819+0000 7f98858d9640 1 --2- 172.21.15.125:0/4028290247 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98640664d0 0x7f9864068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:41.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.819+0000 7f98858d9640 1 --2- 172.21.15.125:0/4028290247 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98640664d0 0x7f9864068990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9880158ca0 tx=0x7f9870008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:41.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.819+0000 7f9876ffd640 1 -- 172.21.15.125:0/4028290247 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f987c0ebd00 con 0x7f9880069e90 2023-10-30T12:28:41.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.971+0000 7f988833e640 1 -- 172.21.15.125:0/4028290247 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f988004e170 con 0x7f9880069e90 2023-10-30T12:28:41.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.971+0000 7f9876ffd640 1 -- 172.21.15.125:0/4028290247 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12899 (secure 0 0 0) 0x7f987c0ef320 con 0x7f9880069e90 2023-10-30T12:28:41.974 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:41.974 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":18,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:34.424776+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:41.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.971+0000 7f988833e640 1 -- 172.21.15.125:0/4028290247 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98640664d0 msgr2=0x7f9864068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:41.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.971+0000 7f988833e640 1 --2- 172.21.15.125:0/4028290247 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98640664d0 0x7f9864068990 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9880158ca0 tx=0x7f9870008040 comp rx=0 tx=0).stop 2023-10-30T12:28:41.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.971+0000 7f988833e640 1 -- 172.21.15.125:0/4028290247 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9880069e90 msgr2=0x7f9880152350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:41.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.971+0000 7f988833e640 1 --2- 172.21.15.125:0/4028290247 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9880069e90 0x7f9880152350 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f987c019fd0 tx=0x7f987c01abe0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.971+0000 7f988833e640 1 -- 172.21.15.125:0/4028290247 shutdown_connections 2023-10-30T12:28:41.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.971+0000 7f988833e640 1 --2- 172.21.15.125:0/4028290247 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98640664d0 0x7f9864068990 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.971+0000 7f988833e640 1 --2- 172.21.15.125:0/4028290247 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98801539a0 0x7f9880156910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.971+0000 7f988833e640 1 --2- 172.21.15.125:0/4028290247 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9880070a00 0x7f98801563d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.971+0000 7f988833e640 1 --2- 172.21.15.125:0/4028290247 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9880069e90 0x7f9880152350 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:41.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.971+0000 7f988833e640 1 -- 172.21.15.125:0/4028290247 >> 172.21.15.125:0/4028290247 conn(0x7f98800c7910 msgr2=0x7f9880149b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:41.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.971+0000 7f988833e640 1 -- 172.21.15.125:0/4028290247 shutdown_connections 2023-10-30T12:28:41.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:41.975+0000 7f988833e640 1 -- 172.21.15.125:0/4028290247 wait complete. 2023-10-30T12:28:41.995 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:31.063403+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:28:41.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:28:42.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0ff7d3640 1 Processor -- start 2023-10-30T12:28:42.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0ff7d3640 1 -- start start 2023-10-30T12:28:42.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0ff7d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0f81508c0 0x7fa0f80c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0ff7d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa0f80c6e60 0x7fa0f8156930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0ff7d3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f80c0a30 con 0x7fa0f80722a0 2023-10-30T12:28:42.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0ff7d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0f80c85b0 con 0x7fa0f80c6e60 2023-10-30T12:28:42.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0ff7d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f806f650 con 0x7fa0f81508c0 2023-10-30T12:28:42.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0fdd70640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa0f80c6e60 0x7fa0f8156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0fd56f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0f81508c0 0x7fa0f80c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0fd56f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0f81508c0 0x7fa0f80c6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:48948/0 (socket says 172.21.15.125:48948) 2023-10-30T12:28:42.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0fdd70640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa0f80c6e60 0x7fa0f8156930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:34126/0 (socket says 172.21.15.125:34126) 2023-10-30T12:28:42.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0fcd6e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa0f80722a0 0x7fa0f80bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:40392/0 (socket says 172.21.15.125:40392) 2023-10-30T12:28:42.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0fd56f640 1 -- 172.21.15.125:0/219160931 learned_addr learned my addr 172.21.15.125:0/219160931 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:42.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0fd56f640 1 -- 172.21.15.125:0/219160931 >> v2:172.21.15.136:3300/0 conn(0x7fa0f80c6e60 msgr2=0x7fa0f8156930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:42.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0fd56f640 1 --2- 172.21.15.125:0/219160931 >> v2:172.21.15.136:3300/0 conn(0x7fa0f80c6e60 0x7fa0f8156930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0fd56f640 1 -- 172.21.15.125:0/219160931 >> v1:172.21.15.136:6789/0 conn(0x7fa0f80722a0 legacy=0x7fa0f80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.135+0000 7fa0fd56f640 1 -- 172.21.15.125:0/219160931 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f806f930 con 0x7fa0f81508c0 2023-10-30T12:28:42.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.139+0000 7fa0fd56f640 1 --2- 172.21.15.125:0/219160931 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0f81508c0 0x7fa0f80c6920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4006fd0 tx=0x7fa0f401c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=184454cf7b270d06 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:42.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.139+0000 7fa0effff640 1 -- 172.21.15.125:0/219160931 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f40ab610 con 0x7fa0f81508c0 2023-10-30T12:28:42.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.139+0000 7fa0effff640 1 -- 172.21.15.125:0/219160931 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f40ab770 con 0x7fa0f81508c0 2023-10-30T12:28:42.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.139+0000 7fa0effff640 1 -- 172.21.15.125:0/219160931 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f40abb00 con 0x7fa0f81508c0 2023-10-30T12:28:42.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.139+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/219160931 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0f81508c0 msgr2=0x7fa0f80c6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.139+0000 7fa0ff7d3640 1 --2- 172.21.15.125:0/219160931 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0f81508c0 0x7fa0f80c6920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa0f4006fd0 tx=0x7fa0f401c520 comp rx=0 tx=0).stop 2023-10-30T12:28:42.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.139+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/219160931 shutdown_connections 2023-10-30T12:28:42.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.139+0000 7fa0ff7d3640 1 --2- 172.21.15.125:0/219160931 >> v2:172.21.15.136:3300/0 conn(0x7fa0f80c6e60 0x7fa0f8156930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.139+0000 7fa0ff7d3640 1 --2- 172.21.15.125:0/219160931 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0f81508c0 0x7fa0f80c6920 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.139+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/219160931 >> 172.21.15.125:0/219160931 conn(0x7fa0f80c7910 msgr2=0x7fa0f80c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:42.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.139+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/219160931 shutdown_connections 2023-10-30T12:28:42.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.139+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/219160931 wait complete. 2023-10-30T12:28:42.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0ff7d3640 1 Processor -- start 2023-10-30T12:28:42.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0ff7d3640 1 -- start start 2023-10-30T12:28:42.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0ff7d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0f80722a0 0x7fa0f80c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0ff7d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0f80c6e60 0x7fa0f8141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0ff7d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f81508c0 0x7fa0f8141b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0ff7d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f814b230 con 0x7fa0f80722a0 2023-10-30T12:28:42.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0ff7d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f81f1c20 con 0x7fa0f81508c0 2023-10-30T12:28:42.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0ff7d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0f806d250 con 0x7fa0f80c6e60 2023-10-30T12:28:42.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0fd56f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0f80722a0 0x7fa0f80c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0fd56f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0f80722a0 0x7fa0f80c9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47364/0 (socket says 172.21.15.125:47364) 2023-10-30T12:28:42.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0fd56f640 1 -- 172.21.15.125:0/923153118 learned_addr learned my addr 172.21.15.125:0/923153118 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:42.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0fdd70640 1 --2- 172.21.15.125:0/923153118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f81508c0 0x7fa0f8141b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0fcd6e640 1 --2- 172.21.15.125:0/923153118 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0f80c6e60 0x7fa0f8141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0fd56f640 1 -- 172.21.15.125:0/923153118 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0f80c6e60 msgr2=0x7fa0f8141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0fd56f640 1 --2- 172.21.15.125:0/923153118 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0f80c6e60 0x7fa0f8141620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0fd56f640 1 -- 172.21.15.125:0/923153118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f81508c0 msgr2=0x7fa0f8141b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0fd56f640 1 --2- 172.21.15.125:0/923153118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f81508c0 0x7fa0f8141b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0fd56f640 1 -- 172.21.15.125:0/923153118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f4009480 con 0x7fa0f80722a0 2023-10-30T12:28:42.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0fd56f640 1 --2- 172.21.15.125:0/923153118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0f80722a0 0x7fa0f80c9ff0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0f401ca00 tx=0x7fa0f40192c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:42.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0edffb640 1 -- 172.21.15.125:0/923153118 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f4005ba0 con 0x7fa0f80722a0 2023-10-30T12:28:42.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0edffb640 1 -- 172.21.15.125:0/923153118 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f4005d00 con 0x7fa0f80722a0 2023-10-30T12:28:42.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0edffb640 1 -- 172.21.15.125:0/923153118 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f40abb00 con 0x7fa0f80722a0 2023-10-30T12:28:42.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/923153118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f81f5060 con 0x7fa0f80722a0 2023-10-30T12:28:42.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.143+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/923153118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f81f5600 con 0x7fa0f80722a0 2023-10-30T12:28:42.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.147+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/923153118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f804e170 con 0x7fa0f80722a0 2023-10-30T12:28:42.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.147+0000 7fa0edffb640 1 -- 172.21.15.125:0/923153118 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa0f4003620 con 0x7fa0f80722a0 2023-10-30T12:28:42.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.147+0000 7fa0edffb640 1 --2- 172.21.15.125:0/923153118 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0dc0664d0 0x7fa0dc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.147+0000 7fa0edffb640 1 -- 172.21.15.125:0/923153118 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa0f411de20 con 0x7fa0f80722a0 2023-10-30T12:28:42.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.147+0000 7fa0fcd6e640 1 --2- 172.21.15.125:0/923153118 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0dc0664d0 0x7fa0dc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.151+0000 7fa0fcd6e640 1 --2- 172.21.15.125:0/923153118 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0dc0664d0 0x7fa0dc068990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8156b70 tx=0x7fa0e8007e50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:42.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.151+0000 7fa0edffb640 1 -- 172.21.15.125:0/923153118 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa0f40eb610 con 0x7fa0f80722a0 2023-10-30T12:28:42.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.299+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/923153118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa0f80c86f0 con 0x7fa0f80722a0 2023-10-30T12:28:42.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.299+0000 7fa0edffb640 1 -- 172.21.15.125:0/923153118 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa0f40eec30 con 0x7fa0f80722a0 2023-10-30T12:28:42.303 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2023-10-30T12:28:42.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.303+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/923153118 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0dc0664d0 msgr2=0x7fa0dc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.303+0000 7fa0ff7d3640 1 --2- 172.21.15.125:0/923153118 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0dc0664d0 0x7fa0dc068990 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8156b70 tx=0x7fa0e8007e50 comp rx=0 tx=0).stop 2023-10-30T12:28:42.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.303+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/923153118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0f80722a0 msgr2=0x7fa0f80c9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.303+0000 7fa0ff7d3640 1 --2- 172.21.15.125:0/923153118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0f80722a0 0x7fa0f80c9ff0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa0f401ca00 tx=0x7fa0f40192c0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.303+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/923153118 shutdown_connections 2023-10-30T12:28:42.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.303+0000 7fa0ff7d3640 1 --2- 172.21.15.125:0/923153118 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0dc0664d0 0x7fa0dc068990 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.303+0000 7fa0ff7d3640 1 --2- 172.21.15.125:0/923153118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0f81508c0 0x7fa0f8141b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.303+0000 7fa0ff7d3640 1 --2- 172.21.15.125:0/923153118 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0f80c6e60 0x7fa0f8141620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.303+0000 7fa0ff7d3640 1 --2- 172.21.15.125:0/923153118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0f80722a0 0x7fa0f80c9ff0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.303+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/923153118 >> 172.21.15.125:0/923153118 conn(0x7fa0f80c7910 msgr2=0x7fa0f80bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:42.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.303+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/923153118 shutdown_connections 2023-10-30T12:28:42.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.303+0000 7fa0ff7d3640 1 -- 172.21.15.125:0/923153118 wait complete. 2023-10-30T12:28:42.323 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-30T12:28:42.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2e5846640 1 Processor -- start 2023-10-30T12:28:42.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2e5846640 1 -- start start 2023-10-30T12:28:42.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2e5846640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe2e01538a0 0x7fe2e015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2e5846640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e00cb800 con 0x7fe2e00ca0e0 2023-10-30T12:28:42.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2e5846640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2e015dc30 con 0x7fe2e00cf160 2023-10-30T12:28:42.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2e5846640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2e00cb330 con 0x7fe2e01538a0 2023-10-30T12:28:42.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2d7fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe2e00ca0e0 0x7fe2e015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43248/0 (socket says 172.21.15.125:43248) 2023-10-30T12:28:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2d7fff640 1 -- 172.21.15.125:0/2710530192 learned_addr learned my addr 172.21.15.125:0/2710530192 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2deffd640 1 --2- 172.21.15.125:0/2710530192 >> v2:172.21.15.136:3300/0 conn(0x7fe2e01538a0 0x7fe2e015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 -- 172.21.15.125:0/2710530192 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420195584 0 0) 0x7fe2e00cb800 con 0x7fe2e00ca0e0 2023-10-30T12:28:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 -- 172.21.15.125:0/2710530192 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2cc003530 con 0x7fe2e00ca0e0 2023-10-30T12:28:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 -- 172.21.15.125:0/2710530192 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1464107545 0 0) 0x7fe2cc003530 con 0x7fe2e00ca0e0 2023-10-30T12:28:42.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 -- 172.21.15.125:0/2710530192 >> v2:172.21.15.136:3300/0 conn(0x7fe2e01538a0 msgr2=0x7fe2e015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 --2- 172.21.15.125:0/2710530192 >> v2:172.21.15.136:3300/0 conn(0x7fe2e01538a0 0x7fe2e015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 -- 172.21.15.125:0/2710530192 >> v1:172.21.15.136:6789/0 conn(0x7fe2e00cf160 legacy=0x7fe2e0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 -- 172.21.15.125:0/2710530192 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e00cad70 con 0x7fe2e00ca0e0 2023-10-30T12:28:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 -- 172.21.15.125:0/2710530192 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fe2c8001680 con 0x7fe2e00ca0e0 2023-10-30T12:28:42.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 -- 172.21.15.125:0/2710530192 >> v1:172.21.15.125:6789/0 conn(0x7fe2e00ca0e0 legacy=0x7fe2e015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 --2- 172.21.15.125:0/2710530192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2cc003980 0x7fe2cc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 --2- 172.21.15.125:0/2710530192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2cc0047d0 0x7fe2cc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 --2- 172.21.15.125:0/2710530192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2cc005190 0x7fe2cc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 -- 172.21.15.125:0/2710530192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e00cb330 con 0x7fe2cc005190 2023-10-30T12:28:42.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 -- 172.21.15.125:0/2710530192 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc00c040 con 0x7fe2cc0047d0 2023-10-30T12:28:42.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2de7fc640 1 -- 172.21.15.125:0/2710530192 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2cc00c200 con 0x7fe2cc003980 2023-10-30T12:28:42.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.459+0000 7fe2df7fe640 1 --2- 172.21.15.125:0/2710530192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2cc005190 0x7fe2cc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.463+0000 7fe2d7fff640 1 --2- 172.21.15.125:0/2710530192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2cc0047d0 0x7fe2cc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.463+0000 7fe2deffd640 1 --2- 172.21.15.125:0/2710530192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2cc003980 0x7fe2cc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.463+0000 7fe2df7fe640 1 -- 172.21.15.125:0/2710530192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2cc003980 msgr2=0x7fe2cc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:42.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.463+0000 7fe2df7fe640 1 --2- 172.21.15.125:0/2710530192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2cc003980 0x7fe2cc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.463+0000 7fe2df7fe640 1 -- 172.21.15.125:0/2710530192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2cc0047d0 msgr2=0x7fe2cc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:42.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.463+0000 7fe2df7fe640 1 --2- 172.21.15.125:0/2710530192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2cc0047d0 0x7fe2cc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.463+0000 7fe2df7fe640 1 -- 172.21.15.125:0/2710530192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2cc003530 con 0x7fe2cc005190 2023-10-30T12:28:42.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.463+0000 7fe2df7fe640 1 -- 172.21.15.125:0/2710530192 reap_dead start 2023-10-30T12:28:42.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.463+0000 7fe2df7fe640 1 --2- 172.21.15.125:0/2710530192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2cc005190 0x7fe2cc00b990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8001cd0 tx=0x7fe2d801c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9351b2c5644bd98 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:42.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.463+0000 7fe2de7fc640 1 -- 172.21.15.125:0/2710530192 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8003b80 con 0x7fe2cc005190 2023-10-30T12:28:42.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.463+0000 7fe2de7fc640 1 -- 172.21.15.125:0/2710530192 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d8003d20 con 0x7fe2cc005190 2023-10-30T12:28:42.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.463+0000 7fe2de7fc640 1 -- 172.21.15.125:0/2710530192 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d8005310 con 0x7fe2cc005190 2023-10-30T12:28:42.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.463+0000 7fe2e5846640 1 -- 172.21.15.125:0/2710530192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2cc005190 msgr2=0x7fe2cc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.463+0000 7fe2e5846640 1 --2- 172.21.15.125:0/2710530192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2cc005190 0x7fe2cc00b990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe2d8001cd0 tx=0x7fe2d801c5c0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2e5846640 1 -- 172.21.15.125:0/2710530192 shutdown_connections 2023-10-30T12:28:42.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2e5846640 1 --2- 172.21.15.125:0/2710530192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2cc005190 0x7fe2cc00b990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2e5846640 1 -- 172.21.15.125:0/2710530192 >> 172.21.15.125:0/2710530192 conn(0x7fe2e006bfe0 msgr2=0x7fe2e006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:42.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2e5846640 1 -- 172.21.15.125:0/2710530192 shutdown_connections 2023-10-30T12:28:42.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2e5846640 1 -- 172.21.15.125:0/2710530192 wait complete. 2023-10-30T12:28:42.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2e5846640 1 Processor -- start 2023-10-30T12:28:42.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2e5846640 1 -- start start 2023-10-30T12:28:42.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2e5846640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2cc005190 0x7fe2e01510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2e5846640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e01525a0 0x7fe2e00b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2e5846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2e01efe20 0x7fe2e01459d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2e5846640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e015d340 con 0x7fe2cc005190 2023-10-30T12:28:42.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2e5846640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e01532a0 con 0x7fe2e01525a0 2023-10-30T12:28:42.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2e5846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e0152b10 con 0x7fe2e01efe20 2023-10-30T12:28:42.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2deffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2cc005190 0x7fe2e01510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2deffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2cc005190 0x7fe2e01510d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47368/0 (socket says 172.21.15.125:47368) 2023-10-30T12:28:42.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2deffd640 1 -- 172.21.15.125:0/1466095 learned_addr learned my addr 172.21.15.125:0/1466095 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:42.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2df7fe640 1 --2- 172.21.15.125:0/1466095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2e01efe20 0x7fe2e01459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2d7fff640 1 --2- 172.21.15.125:0/1466095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e01525a0 0x7fe2e00b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2deffd640 1 -- 172.21.15.125:0/1466095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2e01efe20 msgr2=0x7fe2e01459d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2deffd640 1 --2- 172.21.15.125:0/1466095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2e01efe20 0x7fe2e01459d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2deffd640 1 -- 172.21.15.125:0/1466095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e01525a0 msgr2=0x7fe2e00b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2deffd640 1 --2- 172.21.15.125:0/1466095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e01525a0 0x7fe2e00b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2deffd640 1 -- 172.21.15.125:0/1466095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2e0152d00 con 0x7fe2cc005190 2023-10-30T12:28:42.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2deffd640 1 --2- 172.21.15.125:0/1466095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2cc005190 0x7fe2e01510d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0011590 tx=0x7fe2d0011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:42.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2dcff9640 1 -- 172.21.15.125:0/1466095 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0015930 con 0x7fe2cc005190 2023-10-30T12:28:42.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2dcff9640 1 -- 172.21.15.125:0/1466095 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2d0015a90 con 0x7fe2cc005190 2023-10-30T12:28:42.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2dcff9640 1 -- 172.21.15.125:0/1466095 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2d0015e20 con 0x7fe2cc005190 2023-10-30T12:28:42.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.467+0000 7fe2e5846640 1 -- 172.21.15.125:0/1466095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e0152fc0 con 0x7fe2cc005190 2023-10-30T12:28:42.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.471+0000 7fe2e5846640 1 -- 172.21.15.125:0/1466095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e01ef620 con 0x7fe2cc005190 2023-10-30T12:28:42.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.471+0000 7fe2e5846640 1 -- 172.21.15.125:0/1466095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e01ef130 con 0x7fe2cc005190 2023-10-30T12:28:42.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.471+0000 7fe2dcff9640 1 -- 172.21.15.125:0/1466095 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe2d000f460 con 0x7fe2cc005190 2023-10-30T12:28:42.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.471+0000 7fe2dcff9640 1 --2- 172.21.15.125:0/1466095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2c00664a0 0x7fe2c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.471+0000 7fe2dcff9640 1 -- 172.21.15.125:0/1466095 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe2d00843c0 con 0x7fe2cc005190 2023-10-30T12:28:42.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.471+0000 7fe2d7fff640 1 --2- 172.21.15.125:0/1466095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2c00664a0 0x7fe2c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.471+0000 7fe2d7fff640 1 --2- 172.21.15.125:0/1466095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2c00664a0 0x7fe2c0068960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0152550 tx=0x7fe2c8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:42.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.475+0000 7fe2dcff9640 1 -- 172.21.15.125:0/1466095 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe2d0051af0 con 0x7fe2cc005190 2023-10-30T12:28:42.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.627+0000 7fe2e5846640 1 -- 172.21.15.125:0/1466095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe2e004e170 con 0x7fe2cc005190 2023-10-30T12:28:42.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.627+0000 7fe2dcff9640 1 -- 172.21.15.125:0/1466095 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe2d0055110 con 0x7fe2cc005190 2023-10-30T12:28:42.630 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:42.630 INFO:teuthology.orchestra.run.smithi125.stdout:[ 2023-10-30T12:28:42.630 INFO:teuthology.orchestra.run.smithi125.stdout: { 2023-10-30T12:28:42.631 INFO:teuthology.orchestra.run.smithi125.stdout: "poolnum": 1, 2023-10-30T12:28:42.631 INFO:teuthology.orchestra.run.smithi125.stdout: "poolname": ".mgr" 2023-10-30T12:28:42.631 INFO:teuthology.orchestra.run.smithi125.stdout: } 2023-10-30T12:28:42.631 INFO:teuthology.orchestra.run.smithi125.stdout:] 2023-10-30T12:28:42.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.627+0000 7fe2e5846640 1 -- 172.21.15.125:0/1466095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2c00664a0 msgr2=0x7fe2c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.627+0000 7fe2e5846640 1 --2- 172.21.15.125:0/1466095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2c00664a0 0x7fe2c0068960 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fe2e0152550 tx=0x7fe2c8009040 comp rx=0 tx=0).stop 2023-10-30T12:28:42.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.627+0000 7fe2e5846640 1 -- 172.21.15.125:0/1466095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2cc005190 msgr2=0x7fe2e01510d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.627+0000 7fe2e5846640 1 --2- 172.21.15.125:0/1466095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2cc005190 0x7fe2e01510d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe2d0011590 tx=0x7fe2d0011a50 comp rx=0 tx=0).stop 2023-10-30T12:28:42.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.627+0000 7fe2e5846640 1 -- 172.21.15.125:0/1466095 shutdown_connections 2023-10-30T12:28:42.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.627+0000 7fe2e5846640 1 --2- 172.21.15.125:0/1466095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2c00664a0 0x7fe2c0068960 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.631+0000 7fe2e5846640 1 --2- 172.21.15.125:0/1466095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2e01efe20 0x7fe2e01459d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.631+0000 7fe2e5846640 1 --2- 172.21.15.125:0/1466095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2e01525a0 0x7fe2e00b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.631+0000 7fe2e5846640 1 --2- 172.21.15.125:0/1466095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2cc005190 0x7fe2e01510d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.631+0000 7fe2e5846640 1 -- 172.21.15.125:0/1466095 >> 172.21.15.125:0/1466095 conn(0x7fe2e006bfe0 msgr2=0x7fe2e0157ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:42.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.631+0000 7fe2e5846640 1 -- 172.21.15.125:0/1466095 shutdown_connections 2023-10-30T12:28:42.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.631+0000 7fe2e5846640 1 -- 172.21.15.125:0/1466095 wait complete. 2023-10-30T12:28:42.650 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-30T12:28:42.651 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-30T12:28:42.766 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:28:42.772 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:28:42.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.791+0000 7fd927017640 1 Processor -- start 2023-10-30T12:28:42.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.791+0000 7fd927017640 1 -- start start 2023-10-30T12:28:42.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.791+0000 7fd927017640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9201536d0 0x7fd9200cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.791+0000 7fd927017640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd9200c4000 0x7fd9200cf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.791+0000 7fd927017640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92015a210 con 0x7fd92015d9d0 2023-10-30T12:28:42.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.791+0000 7fd927017640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd92015a4f0 con 0x7fd9200c4000 2023-10-30T12:28:42.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.791+0000 7fd927017640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92015cef0 con 0x7fd9201536d0 2023-10-30T12:28:42.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91ffff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd92015d9d0 0x7fd920159760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43258/0 (socket says 172.21.15.125:43258) 2023-10-30T12:28:42.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91ffff640 1 -- 172.21.15.125:0/484911795 learned_addr learned my addr 172.21.15.125:0/484911795 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:42.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd9255b4640 1 --2- 172.21.15.125:0/484911795 >> v2:172.21.15.136:3300/0 conn(0x7fd9200c4000 0x7fd9200cf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd924db3640 1 --2- 172.21.15.125:0/484911795 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9201536d0 0x7fd9200cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 -- 172.21.15.125:0/484911795 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 605371657 0 0) 0x7fd92015a210 con 0x7fd92015d9d0 2023-10-30T12:28:42.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 -- 172.21.15.125:0/484911795 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd910003530 con 0x7fd92015d9d0 2023-10-30T12:28:42.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 -- 172.21.15.125:0/484911795 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1988287795 0 0) 0x7fd910003530 con 0x7fd92015d9d0 2023-10-30T12:28:42.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 -- 172.21.15.125:0/484911795 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9201536d0 msgr2=0x7fd9200cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:42.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 --2- 172.21.15.125:0/484911795 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9201536d0 0x7fd9200cb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 -- 172.21.15.125:0/484911795 >> v2:172.21.15.136:3300/0 conn(0x7fd9200c4000 msgr2=0x7fd9200cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:42.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 --2- 172.21.15.125:0/484911795 >> v2:172.21.15.136:3300/0 conn(0x7fd9200c4000 0x7fd9200cf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 -- 172.21.15.125:0/484911795 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92015d1d0 con 0x7fd92015d9d0 2023-10-30T12:28:42.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 -- 172.21.15.125:0/484911795 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fd90c001680 con 0x7fd92015d9d0 2023-10-30T12:28:42.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 -- 172.21.15.125:0/484911795 >> v1:172.21.15.125:6789/0 conn(0x7fd92015d9d0 legacy=0x7fd920159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 --2- 172.21.15.125:0/484911795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd910003980 0x7fd910003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 --2- 172.21.15.125:0/484911795 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9201536d0 0x7fd9100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 --2- 172.21.15.125:0/484911795 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd910004d10 0x7fd910005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 -- 172.21.15.125:0/484911795 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92015a4f0 con 0x7fd910004d10 2023-10-30T12:28:42.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 -- 172.21.15.125:0/484911795 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92015cef0 con 0x7fd910003980 2023-10-30T12:28:42.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91f7fe640 1 -- 172.21.15.125:0/484911795 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd91000b990 con 0x7fd9201536d0 2023-10-30T12:28:42.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd9255b4640 1 --2- 172.21.15.125:0/484911795 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd910004d10 0x7fd910005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd91ffff640 1 --2- 172.21.15.125:0/484911795 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9201536d0 0x7fd9100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd924db3640 1 --2- 172.21.15.125:0/484911795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd910003980 0x7fd910003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd9255b4640 1 -- 172.21.15.125:0/484911795 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9201536d0 msgr2=0x7fd9100047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:42.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd9255b4640 1 --2- 172.21.15.125:0/484911795 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9201536d0 0x7fd9100047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd9255b4640 1 -- 172.21.15.125:0/484911795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd910003980 msgr2=0x7fd910003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:42.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd9255b4640 1 --2- 172.21.15.125:0/484911795 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd910003980 0x7fd910003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.795+0000 7fd9255b4640 1 -- 172.21.15.125:0/484911795 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd910003530 con 0x7fd910004d10 2023-10-30T12:28:42.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.799+0000 7fd9255b4640 1 --2- 172.21.15.125:0/484911795 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd910004d10 0x7fd910005190 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd918009ae0 tx=0x7fd91801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24a9baaef3982c7e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:42.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.799+0000 7fd91f7fe640 1 -- 172.21.15.125:0/484911795 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9180a5680 con 0x7fd910004d10 2023-10-30T12:28:42.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.799+0000 7fd91f7fe640 1 -- 172.21.15.125:0/484911795 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9180a57e0 con 0x7fd910004d10 2023-10-30T12:28:42.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.799+0000 7fd91f7fe640 1 -- 172.21.15.125:0/484911795 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9180a5b90 con 0x7fd910004d10 2023-10-30T12:28:42.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.799+0000 7fd927017640 1 -- 172.21.15.125:0/484911795 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd910004d10 msgr2=0x7fd910005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.799+0000 7fd927017640 1 --2- 172.21.15.125:0/484911795 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd910004d10 0x7fd910005190 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd918009ae0 tx=0x7fd91801ca90 comp rx=0 tx=0).stop 2023-10-30T12:28:42.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.799+0000 7fd9255b4640 1 -- 172.21.15.125:0/484911795 reap_dead start 2023-10-30T12:28:42.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.799+0000 7fd927017640 1 -- 172.21.15.125:0/484911795 shutdown_connections 2023-10-30T12:28:42.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.799+0000 7fd927017640 1 -- 172.21.15.125:0/484911795 >> 172.21.15.125:0/484911795 conn(0x7fd92006bf70 msgr2=0x7fd92006c380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:42.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.799+0000 7fd927017640 1 -- 172.21.15.125:0/484911795 shutdown_connections 2023-10-30T12:28:42.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.799+0000 7fd927017640 1 -- 172.21.15.125:0/484911795 wait complete. 2023-10-30T12:28:42.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.799+0000 7fd927017640 1 Processor -- start 2023-10-30T12:28:42.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.799+0000 7fd927017640 1 -- start start 2023-10-30T12:28:42.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd927017640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd910004d10 0x7fd9200ce7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd927017640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9200b2260 0x7fd9201f2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd927017640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd92015a920 0x7fd920145800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd927017640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92015b820 con 0x7fd910004d10 2023-10-30T12:28:42.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd927017640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9201f1e70 con 0x7fd9200b2260 2023-10-30T12:28:42.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd927017640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd92015c1a0 con 0x7fd92015a920 2023-10-30T12:28:42.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd924db3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd910004d10 0x7fd9200ce7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd924db3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd910004d10 0x7fd9200ce7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47384/0 (socket says 172.21.15.125:47384) 2023-10-30T12:28:42.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd924db3640 1 -- 172.21.15.125:0/1258288542 learned_addr learned my addr 172.21.15.125:0/1258288542 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:42.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd91ffff640 1 --2- 172.21.15.125:0/1258288542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9200b2260 0x7fd9201f2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd9255b4640 1 --2- 172.21.15.125:0/1258288542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd92015a920 0x7fd920145800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd924db3640 1 -- 172.21.15.125:0/1258288542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd92015a920 msgr2=0x7fd920145800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd924db3640 1 --2- 172.21.15.125:0/1258288542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd92015a920 0x7fd920145800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd924db3640 1 -- 172.21.15.125:0/1258288542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9200b2260 msgr2=0x7fd9201f2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:42.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd924db3640 1 --2- 172.21.15.125:0/1258288542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9200b2260 0x7fd9201f2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:42.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd924db3640 1 -- 172.21.15.125:0/1258288542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92015c480 con 0x7fd910004d10 2023-10-30T12:28:42.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd924db3640 1 --2- 172.21.15.125:0/1258288542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd910004d10 0x7fd9200ce7b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd914007f00 tx=0x7fd9140115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:42.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd91d7fa640 1 -- 172.21.15.125:0/1258288542 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91400e030 con 0x7fd910004d10 2023-10-30T12:28:42.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd91d7fa640 1 -- 172.21.15.125:0/1258288542 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd914016470 con 0x7fd910004d10 2023-10-30T12:28:42.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd91d7fa640 1 -- 172.21.15.125:0/1258288542 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914016820 con 0x7fd910004d10 2023-10-30T12:28:42.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd927017640 1 -- 172.21.15.125:0/1258288542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd92015c760 con 0x7fd910004d10 2023-10-30T12:28:42.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd927017640 1 -- 172.21.15.125:0/1258288542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9201f1b00 con 0x7fd910004d10 2023-10-30T12:28:42.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd927017640 1 -- 172.21.15.125:0/1258288542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd92004e170 con 0x7fd910004d10 2023-10-30T12:28:42.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.803+0000 7fd91d7fa640 1 -- 172.21.15.125:0/1258288542 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd914016980 con 0x7fd910004d10 2023-10-30T12:28:42.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.807+0000 7fd91d7fa640 1 --2- 172.21.15.125:0/1258288542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9040664a0 0x7fd904068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:42.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.807+0000 7fd91d7fa640 1 -- 172.21.15.125:0/1258288542 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd91400f070 con 0x7fd910004d10 2023-10-30T12:28:42.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.807+0000 7fd91ffff640 1 --2- 172.21.15.125:0/1258288542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9040664a0 0x7fd904068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:42.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.807+0000 7fd91ffff640 1 --2- 172.21.15.125:0/1258288542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9040664a0 0x7fd904068960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd9200c96a0 tx=0x7fd90c007900 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:42.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.811+0000 7fd91d7fa640 1 -- 172.21.15.125:0/1258288542 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd9140520d0 con 0x7fd910004d10 2023-10-30T12:28:42.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:42.963+0000 7fd927017640 1 -- 172.21.15.125:0/1258288542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd9200cab70 con 0x7fd910004d10 2023-10-30T12:28:43.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.019+0000 7fd91d7fa640 1 -- 172.21.15.125:0/1258288542 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd9140556f0 con 0x7fd910004d10 2023-10-30T12:28:43.020 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_metadata' created 2023-10-30T12:28:43.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.019+0000 7fd927017640 1 -- 172.21.15.125:0/1258288542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9040664a0 msgr2=0x7fd904068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:43.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.019+0000 7fd927017640 1 --2- 172.21.15.125:0/1258288542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9040664a0 0x7fd904068960 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd9200c96a0 tx=0x7fd90c007900 comp rx=0 tx=0).stop 2023-10-30T12:28:43.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.019+0000 7fd927017640 1 -- 172.21.15.125:0/1258288542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd910004d10 msgr2=0x7fd9200ce7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:43.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.019+0000 7fd927017640 1 --2- 172.21.15.125:0/1258288542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd910004d10 0x7fd9200ce7b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd914007f00 tx=0x7fd9140115a0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.019+0000 7fd927017640 1 -- 172.21.15.125:0/1258288542 shutdown_connections 2023-10-30T12:28:43.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.019+0000 7fd927017640 1 --2- 172.21.15.125:0/1258288542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9040664a0 0x7fd904068960 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.019+0000 7fd927017640 1 --2- 172.21.15.125:0/1258288542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd92015a920 0x7fd920145800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.019+0000 7fd927017640 1 --2- 172.21.15.125:0/1258288542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9200b2260 0x7fd9201f2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.019+0000 7fd927017640 1 --2- 172.21.15.125:0/1258288542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd910004d10 0x7fd9200ce7b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.019+0000 7fd927017640 1 -- 172.21.15.125:0/1258288542 >> 172.21.15.125:0/1258288542 conn(0x7fd92006bf70 msgr2=0x7fd9200c9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:43.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.019+0000 7fd927017640 1 -- 172.21.15.125:0/1258288542 shutdown_connections 2023-10-30T12:28:43.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.019+0000 7fd927017640 1 -- 172.21.15.125:0/1258288542 wait complete. 2023-10-30T12:28:43.047 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-30T12:28:43.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.187+0000 7fc7e68ea640 1 Processor -- start 2023-10-30T12:28:43.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.187+0000 7fc7e68ea640 1 -- start start 2023-10-30T12:28:43.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7e68ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7e006f880 0x7fc7e006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:43.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7e68ea640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc7e006a7b0 0x7fc7e0161f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:43.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7e68ea640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7e00c6cb0 con 0x7fc7e00c5810 2023-10-30T12:28:43.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7e68ea640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e0071d90 con 0x7fc7e006a7b0 2023-10-30T12:28:43.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7e68ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e01585d0 con 0x7fc7e006f880 2023-10-30T12:28:43.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7dffff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc7e00c5810 0x7fc7e00c5bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43274/0 (socket says 172.21.15.125:43274) 2023-10-30T12:28:43.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7dffff640 1 -- 172.21.15.125:0/1826493272 learned_addr learned my addr 172.21.15.125:0/1826493272 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:43.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7df7fe640 1 --2- 172.21.15.125:0/1826493272 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7e006f880 0x7fc7e006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:43.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7e4e87640 1 --2- 172.21.15.125:0/1826493272 >> v2:172.21.15.136:3300/0 conn(0x7fc7e006a7b0 0x7fc7e0161f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:43.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7deffd640 1 -- 172.21.15.125:0/1826493272 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1332573977 0 0) 0x7fc7e00c6cb0 con 0x7fc7e00c5810 2023-10-30T12:28:43.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7deffd640 1 -- 172.21.15.125:0/1826493272 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7c8003530 con 0x7fc7e00c5810 2023-10-30T12:28:43.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7df7fe640 1 -- 172.21.15.125:0/1826493272 >> v2:172.21.15.136:3300/0 conn(0x7fc7e006a7b0 msgr2=0x7fc7e0161f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:43.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7df7fe640 1 --2- 172.21.15.125:0/1826493272 >> v2:172.21.15.136:3300/0 conn(0x7fc7e006a7b0 0x7fc7e0161f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7df7fe640 1 -- 172.21.15.125:0/1826493272 >> v1:172.21.15.125:6789/0 conn(0x7fc7e00c5810 legacy=0x7fc7e00c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:43.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.191+0000 7fc7df7fe640 1 -- 172.21.15.125:0/1826493272 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e00c6620 con 0x7fc7e006f880 2023-10-30T12:28:43.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.195+0000 7fc7df7fe640 1 --2- 172.21.15.125:0/1826493272 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7e006f880 0x7fc7e006a270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40097c0 tx=0x7fc7d401c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f2826704d47721a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:43.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.195+0000 7fc7deffd640 1 -- 172.21.15.125:0/1826493272 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40ab600 con 0x7fc7e006f880 2023-10-30T12:28:43.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.195+0000 7fc7deffd640 1 -- 172.21.15.125:0/1826493272 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d40a7050 con 0x7fc7e006f880 2023-10-30T12:28:43.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.195+0000 7fc7deffd640 1 -- 172.21.15.125:0/1826493272 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40ab960 con 0x7fc7e006f880 2023-10-30T12:28:43.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.195+0000 7fc7e68ea640 1 -- 172.21.15.125:0/1826493272 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7e006f880 msgr2=0x7fc7e006a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:43.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.195+0000 7fc7e68ea640 1 --2- 172.21.15.125:0/1826493272 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7e006f880 0x7fc7e006a270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40097c0 tx=0x7fc7d401c360 comp rx=0 tx=0).stop 2023-10-30T12:28:43.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.195+0000 7fc7e68ea640 1 -- 172.21.15.125:0/1826493272 shutdown_connections 2023-10-30T12:28:43.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.195+0000 7fc7e68ea640 1 --2- 172.21.15.125:0/1826493272 >> v2:172.21.15.136:3300/0 conn(0x7fc7e006a7b0 0x7fc7e0161f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.195+0000 7fc7e68ea640 1 --2- 172.21.15.125:0/1826493272 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7e006f880 0x7fc7e006a270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.195+0000 7fc7e68ea640 1 -- 172.21.15.125:0/1826493272 >> 172.21.15.125:0/1826493272 conn(0x7fc7e00c7910 msgr2=0x7fc7e00c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:43.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.195+0000 7fc7e68ea640 1 -- 172.21.15.125:0/1826493272 shutdown_connections 2023-10-30T12:28:43.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.195+0000 7fc7e68ea640 1 -- 172.21.15.125:0/1826493272 wait complete. 2023-10-30T12:28:43.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.195+0000 7fc7e68ea640 1 Processor -- start 2023-10-30T12:28:43.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.195+0000 7fc7e68ea640 1 -- start start 2023-10-30T12:28:43.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7e68ea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7e006a7b0 0x7fc7e01555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:43.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7e68ea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7e006f880 0x7fc7e014b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:43.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7e68ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7e00c5810 0x7fc7e01f0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:43.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7e68ea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e0158e50 con 0x7fc7e006a7b0 2023-10-30T12:28:43.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7e68ea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e015ba10 con 0x7fc7e006f880 2023-10-30T12:28:43.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7e68ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e0155e30 con 0x7fc7e00c5810 2023-10-30T12:28:43.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7dffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7e006a7b0 0x7fc7e01555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:43.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7dffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7e006a7b0 0x7fc7e01555b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47396/0 (socket says 172.21.15.125:47396) 2023-10-30T12:28:43.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7dffff640 1 -- 172.21.15.125:0/2120389298 learned_addr learned my addr 172.21.15.125:0/2120389298 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:43.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7df7fe640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7e006f880 0x7fc7e014b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:43.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7e4e87640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7e00c5810 0x7fc7e01f0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:43.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7dffff640 1 -- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7e00c5810 msgr2=0x7fc7e01f0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:43.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7dffff640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7e00c5810 0x7fc7e01f0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7dffff640 1 -- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7e006f880 msgr2=0x7fc7e014b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:43.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7dffff640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7e006f880 0x7fc7e014b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:43.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7dffff640 1 -- 172.21.15.125:0/2120389298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d4009420 con 0x7fc7e006a7b0 2023-10-30T12:28:43.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7e4e87640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7e00c5810 0x7fc7e01f0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:43.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7dffff640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7e006a7b0 0x7fc7e01555b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000c730 tx=0x7fc7d000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:43.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7dcff9640 1 -- 172.21.15.125:0/2120389298 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0007270 con 0x7fc7e006a7b0 2023-10-30T12:28:43.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7dcff9640 1 -- 172.21.15.125:0/2120389298 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d00073d0 con 0x7fc7e006a7b0 2023-10-30T12:28:43.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7dcff9640 1 -- 172.21.15.125:0/2120389298 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d0001d50 con 0x7fc7e006a7b0 2023-10-30T12:28:43.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7e68ea640 1 -- 172.21.15.125:0/2120389298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e015b200 con 0x7fc7e006a7b0 2023-10-30T12:28:43.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7e68ea640 1 -- 172.21.15.125:0/2120389298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e01f0070 con 0x7fc7e006a7b0 2023-10-30T12:28:43.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.199+0000 7fc7e68ea640 1 -- 172.21.15.125:0/2120389298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e00c75d0 con 0x7fc7e006a7b0 2023-10-30T12:28:43.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.203+0000 7fc7dcff9640 1 -- 172.21.15.125:0/2120389298 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7d00057b0 con 0x7fc7e006a7b0 2023-10-30T12:28:43.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.203+0000 7fc7dcff9640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7cc0664a0 0x7fc7cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:43.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.203+0000 7fc7dcff9640 1 -- 172.21.15.125:0/2120389298 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fc7d00832c0 con 0x7fc7e006a7b0 2023-10-30T12:28:43.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.203+0000 7fc7df7fe640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7cc0664a0 0x7fc7cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:43.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.203+0000 7fc7df7fe640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7cc0664a0 0x7fc7cc068960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40097c0 tx=0x7fc7d40a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:43.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.207+0000 7fc7dcff9640 1 -- 172.21.15.125:0/2120389298 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7d00508a0 con 0x7fc7e006a7b0 2023-10-30T12:28:43.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:43.355+0000 7fc7e68ea640 1 -- 172.21.15.125:0/2120389298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc7e01efdd0 con 0x7fc7e006a7b0 2023-10-30T12:28:44.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.023+0000 7fc7dcff9640 1 -- 172.21.15.125:0/2120389298 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc7d0017020 con 0x7fc7e006a7b0 2023-10-30T12:28:44.025 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_data' created 2023-10-30T12:28:44.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.023+0000 7fc7e68ea640 1 -- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7cc0664a0 msgr2=0x7fc7cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.023+0000 7fc7e68ea640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7cc0664a0 0x7fc7cc068960 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40097c0 tx=0x7fc7d40a6000 comp rx=0 tx=0).stop 2023-10-30T12:28:44.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.023+0000 7fc7e68ea640 1 -- 172.21.15.125:0/2120389298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7e006a7b0 msgr2=0x7fc7e01555b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.023+0000 7fc7e68ea640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7e006a7b0 0x7fc7e01555b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000c730 tx=0x7fc7d000cbf0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.023+0000 7fc7e68ea640 1 -- 172.21.15.125:0/2120389298 shutdown_connections 2023-10-30T12:28:44.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.023+0000 7fc7e68ea640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7cc0664a0 0x7fc7cc068960 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.023+0000 7fc7e68ea640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7e00c5810 0x7fc7e01f0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.023+0000 7fc7e68ea640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7e006f880 0x7fc7e014b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.023+0000 7fc7e68ea640 1 --2- 172.21.15.125:0/2120389298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7e006a7b0 0x7fc7e01555b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.023+0000 7fc7e68ea640 1 -- 172.21.15.125:0/2120389298 >> 172.21.15.125:0/2120389298 conn(0x7fc7e00c7910 msgr2=0x7fc7e00c9de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:44.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.027+0000 7fc7e68ea640 1 -- 172.21.15.125:0/2120389298 shutdown_connections 2023-10-30T12:28:44.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.027+0000 7fc7e68ea640 1 -- 172.21.15.125:0/2120389298 wait complete. 2023-10-30T12:28:44.068 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-30T12:28:44.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.207+0000 7fc2e07f8640 1 Processor -- start 2023-10-30T12:28:44.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.207+0000 7fc2e07f8640 1 -- start start 2023-10-30T12:28:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2e07f8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc2d806a980 0x7fc2d8162f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2e07f8640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d80c6cb0 con 0x7fc2d80c5810 2023-10-30T12:28:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2e07f8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2d80c6620 con 0x7fc2d806f880 2023-10-30T12:28:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2e07f8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2d8071d90 con 0x7fc2d806a980 2023-10-30T12:28:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2de594640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc2d80c5810 0x7fc2d80c5bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43286/0 (socket says 172.21.15.125:43286) 2023-10-30T12:28:44.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2de594640 1 -- 172.21.15.125:0/1128682730 learned_addr learned my addr 172.21.15.125:0/1128682730 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:44.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2ded95640 1 --2- 172.21.15.125:0/1128682730 >> v2:172.21.15.136:3300/0 conn(0x7fc2d806a980 0x7fc2d8162f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 -- 172.21.15.125:0/1128682730 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4148153537 0 0) 0x7fc2d80c6cb0 con 0x7fc2d80c5810 2023-10-30T12:28:44.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 -- 172.21.15.125:0/1128682730 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c4003530 con 0x7fc2d80c5810 2023-10-30T12:28:44.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 -- 172.21.15.125:0/1128682730 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1456823210 0 0) 0x7fc2c4003530 con 0x7fc2d80c5810 2023-10-30T12:28:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 -- 172.21.15.125:0/1128682730 >> v2:172.21.15.136:3300/0 conn(0x7fc2d806a980 msgr2=0x7fc2d8162f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 --2- 172.21.15.125:0/1128682730 >> v2:172.21.15.136:3300/0 conn(0x7fc2d806a980 0x7fc2d8162f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 -- 172.21.15.125:0/1128682730 >> v1:172.21.15.136:6789/0 conn(0x7fc2d806f880 legacy=0x7fc2d806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 -- 172.21.15.125:0/1128682730 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d81582f0 con 0x7fc2d80c5810 2023-10-30T12:28:44.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 -- 172.21.15.125:0/1128682730 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fc2d4001800 con 0x7fc2d80c5810 2023-10-30T12:28:44.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2ded95640 1 --2- 172.21.15.125:0/1128682730 >> v2:172.21.15.136:3300/0 conn(0x7fc2d806a980 0x7fc2d8162f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:44.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 -- 172.21.15.125:0/1128682730 >> v1:172.21.15.125:6789/0 conn(0x7fc2d80c5810 legacy=0x7fc2d80c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 --2- 172.21.15.125:0/1128682730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4003980 0x7fc2c4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 --2- 172.21.15.125:0/1128682730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2c40047d0 0x7fc2c4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 --2- 172.21.15.125:0/1128682730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c40052b0 0x7fc2c400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 -- 172.21.15.125:0/1128682730 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c400c150 con 0x7fc2c4003980 2023-10-30T12:28:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 -- 172.21.15.125:0/1128682730 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c400c2c0 con 0x7fc2c40047d0 2023-10-30T12:28:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2dd592640 1 -- 172.21.15.125:0/1128682730 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c400c480 con 0x7fc2c40052b0 2023-10-30T12:28:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2de594640 1 --2- 172.21.15.125:0/1128682730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4003980 0x7fc2c4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2ddd93640 1 --2- 172.21.15.125:0/1128682730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2c40047d0 0x7fc2c4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2ded95640 1 --2- 172.21.15.125:0/1128682730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c40052b0 0x7fc2c400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2de594640 1 -- 172.21.15.125:0/1128682730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c40052b0 msgr2=0x7fc2c400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:44.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2de594640 1 --2- 172.21.15.125:0/1128682730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c40052b0 0x7fc2c400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2de594640 1 -- 172.21.15.125:0/1128682730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2c40047d0 msgr2=0x7fc2c4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:44.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2de594640 1 --2- 172.21.15.125:0/1128682730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2c40047d0 0x7fc2c4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2de594640 1 -- 172.21.15.125:0/1128682730 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c4003530 con 0x7fc2c4003980 2023-10-30T12:28:44.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2ddd93640 1 --2- 172.21.15.125:0/1128682730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2c40047d0 0x7fc2c4004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:44.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.211+0000 7fc2ded95640 1 -- 172.21.15.125:0/1128682730 reap_dead start 2023-10-30T12:28:44.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.215+0000 7fc2de594640 1 --2- 172.21.15.125:0/1128682730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4003980 0x7fc2c4003da0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400c210 tx=0x7fc2d401cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e30e8690111ac71 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:44.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.215+0000 7fc2dd592640 1 -- 172.21.15.125:0/1128682730 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4007270 con 0x7fc2c4003980 2023-10-30T12:28:44.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.215+0000 7fc2dd592640 1 -- 172.21.15.125:0/1128682730 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d40073f0 con 0x7fc2c4003980 2023-10-30T12:28:44.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.215+0000 7fc2dd592640 1 -- 172.21.15.125:0/1128682730 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d40057b0 con 0x7fc2c4003980 2023-10-30T12:28:44.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.215+0000 7fc2e07f8640 1 -- 172.21.15.125:0/1128682730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4003980 msgr2=0x7fc2c4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.215+0000 7fc2e07f8640 1 --2- 172.21.15.125:0/1128682730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4003980 0x7fc2c4003da0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400c210 tx=0x7fc2d401cb60 comp rx=0 tx=0).stop 2023-10-30T12:28:44.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.215+0000 7fc2e07f8640 1 -- 172.21.15.125:0/1128682730 shutdown_connections 2023-10-30T12:28:44.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.215+0000 7fc2e07f8640 1 --2- 172.21.15.125:0/1128682730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4003980 0x7fc2c4003da0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.215+0000 7fc2e07f8640 1 -- 172.21.15.125:0/1128682730 >> 172.21.15.125:0/1128682730 conn(0x7fc2d80c7910 msgr2=0x7fc2d80c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:44.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.215+0000 7fc2e07f8640 1 -- 172.21.15.125:0/1128682730 shutdown_connections 2023-10-30T12:28:44.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.215+0000 7fc2e07f8640 1 -- 172.21.15.125:0/1128682730 wait complete. 2023-10-30T12:28:44.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2e07f8640 1 Processor -- start 2023-10-30T12:28:44.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2e07f8640 1 -- start start 2023-10-30T12:28:44.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2e07f8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4003980 0x7fc2d814b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2e07f8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2d814dc20 0x7fc2d815b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2e07f8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2d8158750 0x7fc2d8152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2e07f8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d81594b0 con 0x7fc2c4003980 2023-10-30T12:28:44.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2e07f8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d81548d0 con 0x7fc2d814dc20 2023-10-30T12:28:44.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2e07f8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2d815b7d0 con 0x7fc2d8158750 2023-10-30T12:28:44.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2de594640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4003980 0x7fc2d814b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2de594640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4003980 0x7fc2d814b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47408/0 (socket says 172.21.15.125:47408) 2023-10-30T12:28:44.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2de594640 1 -- 172.21.15.125:0/3416225001 learned_addr learned my addr 172.21.15.125:0/3416225001 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:44.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2ddd93640 1 --2- 172.21.15.125:0/3416225001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2d814dc20 0x7fc2d815b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2ded95640 1 --2- 172.21.15.125:0/3416225001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2d8158750 0x7fc2d8152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2de594640 1 -- 172.21.15.125:0/3416225001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2d8158750 msgr2=0x7fc2d8152420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2de594640 1 --2- 172.21.15.125:0/3416225001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2d8158750 0x7fc2d8152420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2de594640 1 -- 172.21.15.125:0/3416225001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2d814dc20 msgr2=0x7fc2d815b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:44.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2de594640 1 --2- 172.21.15.125:0/3416225001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2d814dc20 0x7fc2d815b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:44.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2de594640 1 -- 172.21.15.125:0/3416225001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2d8159160 con 0x7fc2c4003980 2023-10-30T12:28:44.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2de594640 1 --2- 172.21.15.125:0/3416225001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4003980 0x7fc2d814b090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4000f00 tx=0x7fc2c4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:44.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2ceffd640 1 -- 172.21.15.125:0/3416225001 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d40b4070 con 0x7fc2c4003980 2023-10-30T12:28:44.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2ceffd640 1 -- 172.21.15.125:0/3416225001 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d40a5dd0 con 0x7fc2c4003980 2023-10-30T12:28:44.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2ceffd640 1 -- 172.21.15.125:0/3416225001 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4000f80 con 0x7fc2c4003980 2023-10-30T12:28:44.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2e07f8640 1 -- 172.21.15.125:0/3416225001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2d815b9d0 con 0x7fc2c4003980 2023-10-30T12:28:44.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2e07f8640 1 -- 172.21.15.125:0/3416225001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2d8153cb0 con 0x7fc2c4003980 2023-10-30T12:28:44.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.219+0000 7fc2e07f8640 1 -- 172.21.15.125:0/3416225001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2d804e170 con 0x7fc2c4003980 2023-10-30T12:28:44.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.223+0000 7fc2ceffd640 1 -- 172.21.15.125:0/3416225001 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc2d40070d0 con 0x7fc2c4003980 2023-10-30T12:28:44.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.223+0000 7fc2ceffd640 1 --2- 172.21.15.125:0/3416225001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2bc0664d0 0x7fc2bc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:44.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.223+0000 7fc2ceffd640 1 -- 172.21.15.125:0/3416225001 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fc2d411d540 con 0x7fc2c4003980 2023-10-30T12:28:44.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.223+0000 7fc2ddd93640 1 --2- 172.21.15.125:0/3416225001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2bc0664d0 0x7fc2bc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:44.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.223+0000 7fc2ddd93640 1 --2- 172.21.15.125:0/3416225001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2bc0664d0 0x7fc2bc068990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8000c10 tx=0x7fc2c8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:44.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.227+0000 7fc2ceffd640 1 -- 172.21.15.125:0/3416225001 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc2d40eaa90 con 0x7fc2c4003980 2023-10-30T12:28:44.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:44.407+0000 7fc2e07f8640 1 -- 172.21.15.125:0/3416225001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc2d8159160 con 0x7fc2c4003980 2023-10-30T12:28:45.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.039+0000 7fc2ceffd640 1 -- 172.21.15.125:0/3416225001 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fc2d8159160 con 0x7fc2c4003980 2023-10-30T12:28:45.044 INFO:teuthology.orchestra.run.smithi125.stderr:new fs with metadata pool 2 and data pool 3 2023-10-30T12:28:45.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.043+0000 7fc2e07f8640 1 -- 172.21.15.125:0/3416225001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2bc0664d0 msgr2=0x7fc2bc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.043+0000 7fc2e07f8640 1 --2- 172.21.15.125:0/3416225001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2bc0664d0 0x7fc2bc068990 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc2c8000c10 tx=0x7fc2c8009040 comp rx=0 tx=0).stop 2023-10-30T12:28:45.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.043+0000 7fc2e07f8640 1 -- 172.21.15.125:0/3416225001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4003980 msgr2=0x7fc2d814b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.043+0000 7fc2e07f8640 1 --2- 172.21.15.125:0/3416225001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4003980 0x7fc2d814b090 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc2d4000f00 tx=0x7fc2c4005800 comp rx=0 tx=0).stop 2023-10-30T12:28:45.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.043+0000 7fc2e07f8640 1 -- 172.21.15.125:0/3416225001 shutdown_connections 2023-10-30T12:28:45.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.043+0000 7fc2e07f8640 1 --2- 172.21.15.125:0/3416225001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2bc0664d0 0x7fc2bc068990 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.043+0000 7fc2e07f8640 1 --2- 172.21.15.125:0/3416225001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2d8158750 0x7fc2d8152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.043+0000 7fc2e07f8640 1 --2- 172.21.15.125:0/3416225001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2d814dc20 0x7fc2d815b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.043+0000 7fc2e07f8640 1 --2- 172.21.15.125:0/3416225001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4003980 0x7fc2d814b090 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.043+0000 7fc2e07f8640 1 -- 172.21.15.125:0/3416225001 >> 172.21.15.125:0/3416225001 conn(0x7fc2d80c7910 msgr2=0x7fc2d80c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:45.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.043+0000 7fc2e07f8640 1 -- 172.21.15.125:0/3416225001 shutdown_connections 2023-10-30T12:28:45.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.043+0000 7fc2e07f8640 1 -- 172.21.15.125:0/3416225001 wait complete. 2023-10-30T12:28:45.067 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:45.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.203+0000 7f61bfd65640 1 Processor -- start 2023-10-30T12:28:45.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.203+0000 7f61bfd65640 1 -- start start 2023-10-30T12:28:45.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.203+0000 7f61bfd65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61b8071110 0x7f61b806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.203+0000 7f61bfd65640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b8066cf0 con 0x7f61b80c6bd0 2023-10-30T12:28:45.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.203+0000 7f61bfd65640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b80c70d0 con 0x7f61b806f7a0 2023-10-30T12:28:45.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.203+0000 7f61bfd65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80700c0 con 0x7f61b8071110 2023-10-30T12:28:45.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.203+0000 7f61bd300640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f61b80c6bd0 0x7f61b8158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43302/0 (socket says 172.21.15.125:43302) 2023-10-30T12:28:45.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.203+0000 7f61bd300640 1 -- 172.21.15.125:0/421736088 learned_addr learned my addr 172.21.15.125:0/421736088 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:45.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.203+0000 7f61be302640 1 --2- 172.21.15.125:0/421736088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61b8071110 0x7f61b806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.203+0000 7f61bcaff640 1 -- 172.21.15.125:0/421736088 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1885462984 0 0) 0x7f61b8066cf0 con 0x7f61b80c6bd0 2023-10-30T12:28:45.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.203+0000 7f61bcaff640 1 -- 172.21.15.125:0/421736088 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61a4003530 con 0x7f61b80c6bd0 2023-10-30T12:28:45.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 -- 172.21.15.125:0/421736088 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3571429717 0 0) 0x7f61a4003530 con 0x7f61b80c6bd0 2023-10-30T12:28:45.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 -- 172.21.15.125:0/421736088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61b8071110 msgr2=0x7f61b806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:45.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 --2- 172.21.15.125:0/421736088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61b8071110 0x7f61b806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 -- 172.21.15.125:0/421736088 >> v1:172.21.15.136:6789/0 conn(0x7f61b806f7a0 legacy=0x7f61b8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 -- 172.21.15.125:0/421736088 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b80c7390 con 0x7f61b80c6bd0 2023-10-30T12:28:45.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 -- 172.21.15.125:0/421736088 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f61a8001680 con 0x7f61b80c6bd0 2023-10-30T12:28:45.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 -- 172.21.15.125:0/421736088 >> v1:172.21.15.125:6789/0 conn(0x7f61b80c6bd0 legacy=0x7f61b8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 --2- 172.21.15.125:0/421736088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61a40039b0 0x7f61a4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 --2- 172.21.15.125:0/421736088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61a4004360 0x7f61a4004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 --2- 172.21.15.125:0/421736088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61b8071110 0x7f61a4005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 -- 172.21.15.125:0/421736088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b80700c0 con 0x7f61a40039b0 2023-10-30T12:28:45.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 -- 172.21.15.125:0/421736088 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a400b990 con 0x7f61a4004360 2023-10-30T12:28:45.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 -- 172.21.15.125:0/421736088 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a400bb50 con 0x7f61b8071110 2023-10-30T12:28:45.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bdb01640 1 --2- 172.21.15.125:0/421736088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61a40039b0 0x7f61a4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bd300640 1 --2- 172.21.15.125:0/421736088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61a4004360 0x7f61a4004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61be302640 1 --2- 172.21.15.125:0/421736088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61b8071110 0x7f61a4005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bdb01640 1 -- 172.21.15.125:0/421736088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61b8071110 msgr2=0x7f61a4005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:45.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bdb01640 1 --2- 172.21.15.125:0/421736088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61b8071110 0x7f61a4005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bdb01640 1 -- 172.21.15.125:0/421736088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61a4004360 msgr2=0x7f61a4004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:45.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bdb01640 1 --2- 172.21.15.125:0/421736088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61a4004360 0x7f61a4004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bdb01640 1 -- 172.21.15.125:0/421736088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a4003530 con 0x7f61a40039b0 2023-10-30T12:28:45.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bdb01640 1 --2- 172.21.15.125:0/421736088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61a40039b0 0x7f61a4003d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f61b4001910 tx=0x7f61b401ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338bfd94893b0299 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:45.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 -- 172.21.15.125:0/421736088 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b40a88f0 con 0x7f61a40039b0 2023-10-30T12:28:45.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 -- 172.21.15.125:0/421736088 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b40a8a70 con 0x7f61a40039b0 2023-10-30T12:28:45.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.207+0000 7f61bcaff640 1 -- 172.21.15.125:0/421736088 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b40a8e20 con 0x7f61a40039b0 2023-10-30T12:28:45.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bfd65640 1 -- 172.21.15.125:0/421736088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61a40039b0 msgr2=0x7f61a4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bfd65640 1 --2- 172.21.15.125:0/421736088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61a40039b0 0x7f61a4003d90 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f61b4001910 tx=0x7f61b401ecd0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61be302640 1 -- 172.21.15.125:0/421736088 reap_dead start 2023-10-30T12:28:45.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bfd65640 1 -- 172.21.15.125:0/421736088 shutdown_connections 2023-10-30T12:28:45.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bfd65640 1 -- 172.21.15.125:0/421736088 >> 172.21.15.125:0/421736088 conn(0x7f61b80c7910 msgr2=0x7f61b80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:45.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bfd65640 1 -- 172.21.15.125:0/421736088 shutdown_connections 2023-10-30T12:28:45.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bfd65640 1 -- 172.21.15.125:0/421736088 wait complete. 2023-10-30T12:28:45.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bfd65640 1 Processor -- start 2023-10-30T12:28:45.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bfd65640 1 -- start start 2023-10-30T12:28:45.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bfd65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61a40039b0 0x7f61b81f1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bfd65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61b81553e0 0x7f61b80c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bfd65640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61b8152b60 0x7f61b8152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bfd65640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b81526d0 con 0x7f61a40039b0 2023-10-30T12:28:45.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bfd65640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b81550c0 con 0x7f61b8152b60 2023-10-30T12:28:45.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bfd65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b8152110 con 0x7f61b81553e0 2023-10-30T12:28:45.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bdb01640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61a40039b0 0x7f61b81f1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bdb01640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61a40039b0 0x7f61b81f1920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47430/0 (socket says 172.21.15.125:47430) 2023-10-30T12:28:45.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bdb01640 1 -- 172.21.15.125:0/163291170 learned_addr learned my addr 172.21.15.125:0/163291170 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:45.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61be302640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61b8152b60 0x7f61b8152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bd300640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61b81553e0 0x7f61b80c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bdb01640 1 -- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61b81553e0 msgr2=0x7f61b80c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bdb01640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61b81553e0 0x7f61b80c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bdb01640 1 -- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61b8152b60 msgr2=0x7f61b8152fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bdb01640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61b8152b60 0x7f61b8152fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bdb01640 1 -- 172.21.15.125:0/163291170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b8153520 con 0x7f61a40039b0 2023-10-30T12:28:45.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61be302640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61b8152b60 0x7f61b8152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:45.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.211+0000 7f61bd300640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61b81553e0 0x7f61b80c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:45.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.215+0000 7f61bdb01640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61a40039b0 0x7f61b81f1920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f61b40b3040 tx=0x7f61a4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:45.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.215+0000 7f61ae7fc640 1 -- 172.21.15.125:0/163291170 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b40052c0 con 0x7f61a40039b0 2023-10-30T12:28:45.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.215+0000 7f61ae7fc640 1 -- 172.21.15.125:0/163291170 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b4005420 con 0x7f61a40039b0 2023-10-30T12:28:45.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.215+0000 7f61ae7fc640 1 -- 172.21.15.125:0/163291170 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b4006a00 con 0x7f61a40039b0 2023-10-30T12:28:45.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.215+0000 7f61bfd65640 1 -- 172.21.15.125:0/163291170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b81f1e60 con 0x7f61a40039b0 2023-10-30T12:28:45.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.215+0000 7f61bfd65640 1 -- 172.21.15.125:0/163291170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b81f23d0 con 0x7f61a40039b0 2023-10-30T12:28:45.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.215+0000 7f61bfd65640 1 -- 172.21.15.125:0/163291170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b804e170 con 0x7f61a40039b0 2023-10-30T12:28:45.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.215+0000 7f61ae7fc640 1 -- 172.21.15.125:0/163291170 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f61b40b7040 con 0x7f61a40039b0 2023-10-30T12:28:45.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.215+0000 7f61ae7fc640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f619c0664a0 0x7f619c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.215+0000 7f61ae7fc640 1 -- 172.21.15.125:0/163291170 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61b4120b60 con 0x7f61a40039b0 2023-10-30T12:28:45.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.215+0000 7f61bd300640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f619c0664a0 0x7f619c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.219+0000 7f61bd300640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f619c0664a0 0x7f619c068960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f61a8001ec0 tx=0x7f61a8001140 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:45.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.219+0000 7f61ae7fc640 1 -- 172.21.15.125:0/163291170 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f61b40edf40 con 0x7f61a40039b0 2023-10-30T12:28:45.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.371+0000 7f61bfd65640 1 -- 172.21.15.125:0/163291170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61b80708e0 con 0x7f61a40039b0 2023-10-30T12:28:45.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.371+0000 7f61ae7fc640 1 -- 172.21.15.125:0/163291170 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f61b40b5090 con 0x7f61a40039b0 2023-10-30T12:28:45.375 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:45.376 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":21,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:44.410415+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:45.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.375+0000 7f61bfd65640 1 -- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f619c0664a0 msgr2=0x7f619c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.375+0000 7f61bfd65640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f619c0664a0 0x7f619c068960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f61a8001ec0 tx=0x7f61a8001140 comp rx=0 tx=0).stop 2023-10-30T12:28:45.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.375+0000 7f61bfd65640 1 -- 172.21.15.125:0/163291170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61a40039b0 msgr2=0x7f61b81f1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.375+0000 7f61bfd65640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61a40039b0 0x7f61b81f1920 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f61b40b3040 tx=0x7f61a4005800 comp rx=0 tx=0).stop 2023-10-30T12:28:45.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.375+0000 7f61bfd65640 1 -- 172.21.15.125:0/163291170 shutdown_connections 2023-10-30T12:28:45.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.375+0000 7f61bfd65640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f619c0664a0 0x7f619c068960 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.375+0000 7f61bfd65640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61b8152b60 0x7f61b8152fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.375+0000 7f61bfd65640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61b81553e0 0x7f61b80c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.375+0000 7f61bfd65640 1 --2- 172.21.15.125:0/163291170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61a40039b0 0x7f61b81f1920 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.375+0000 7f61bfd65640 1 -- 172.21.15.125:0/163291170 >> 172.21.15.125:0/163291170 conn(0x7f61b80c7910 msgr2=0x7f61b80c63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:45.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.375+0000 7f61bfd65640 1 -- 172.21.15.125:0/163291170 shutdown_connections 2023-10-30T12:28:45.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.375+0000 7f61bfd65640 1 -- 172.21.15.125:0/163291170 wait complete. 2023-10-30T12:28:45.396 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:45.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.531+0000 7f65b5594640 1 Processor -- start 2023-10-30T12:28:45.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.531+0000 7f65b5594640 1 -- start start 2023-10-30T12:28:45.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.531+0000 7f65b5594640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f65b01538a0 0x7f65b015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.531+0000 7f65b5594640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65b00ca0e0 0x7f65b015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.531+0000 7f65b5594640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65b00cb800 con 0x7f65b00cf160 2023-10-30T12:28:45.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.531+0000 7f65b5594640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b015dc30 con 0x7f65b00ca0e0 2023-10-30T12:28:45.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.531+0000 7f65b5594640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b00cb330 con 0x7f65b01538a0 2023-10-30T12:28:45.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.531+0000 7f65ae7fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65b00ca0e0 0x7f65b015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.531+0000 7f65ae7fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65b00ca0e0 0x7f65b015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47432/0 (socket says 172.21.15.125:47432) 2023-10-30T12:28:45.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.531+0000 7f65ae7fc640 1 -- 172.21.15.125:0/3935159736 learned_addr learned my addr 172.21.15.125:0/3935159736 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:45.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.535+0000 7f65aeffd640 1 --2- 172.21.15.125:0/3935159736 >> v2:172.21.15.136:3300/0 conn(0x7f65b01538a0 0x7f65b015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.535+0000 7f65ae7fc640 1 -- 172.21.15.125:0/3935159736 >> v2:172.21.15.136:3300/0 conn(0x7f65b01538a0 msgr2=0x7f65b015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:45.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.535+0000 7f65ae7fc640 1 --2- 172.21.15.125:0/3935159736 >> v2:172.21.15.136:3300/0 conn(0x7f65b01538a0 0x7f65b015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.535+0000 7f65ae7fc640 1 -- 172.21.15.125:0/3935159736 >> v1:172.21.15.136:6789/0 conn(0x7f65b00cf160 legacy=0x7f65b00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.535+0000 7f65ae7fc640 1 -- 172.21.15.125:0/3935159736 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b00cad70 con 0x7f65b00ca0e0 2023-10-30T12:28:45.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.535+0000 7f65ae7fc640 1 --2- 172.21.15.125:0/3935159736 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b00ca0e0 0x7f65b015a090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f65b00cfd10 tx=0x7f65a401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5a63d2e735f320c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:45.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.535+0000 7f65adffb640 1 -- 172.21.15.125:0/3935159736 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a40a89f0 con 0x7f65b00ca0e0 2023-10-30T12:28:45.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.535+0000 7f65adffb640 1 -- 172.21.15.125:0/3935159736 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a40a8b50 con 0x7f65b00ca0e0 2023-10-30T12:28:45.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.535+0000 7f65adffb640 1 -- 172.21.15.125:0/3935159736 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a40b0440 con 0x7f65b00ca0e0 2023-10-30T12:28:45.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 -- 172.21.15.125:0/3935159736 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b00ca0e0 msgr2=0x7f65b015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 --2- 172.21.15.125:0/3935159736 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b00ca0e0 0x7f65b015a090 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f65b00cfd10 tx=0x7f65a401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 -- 172.21.15.125:0/3935159736 shutdown_connections 2023-10-30T12:28:45.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 --2- 172.21.15.125:0/3935159736 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b00ca0e0 0x7f65b015a090 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 --2- 172.21.15.125:0/3935159736 >> v2:172.21.15.136:3300/0 conn(0x7f65b01538a0 0x7f65b015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 -- 172.21.15.125:0/3935159736 >> 172.21.15.125:0/3935159736 conn(0x7f65b006bfe0 msgr2=0x7f65b006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:45.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 -- 172.21.15.125:0/3935159736 shutdown_connections 2023-10-30T12:28:45.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 -- 172.21.15.125:0/3935159736 wait complete. 2023-10-30T12:28:45.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 Processor -- start 2023-10-30T12:28:45.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 -- start start 2023-10-30T12:28:45.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b00ca0e0 0x7f65b00ce630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65b00cf160 0x7f65b0142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b01538a0 0x7f65b01ef7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b01584d0 con 0x7f65b01538a0 2023-10-30T12:28:45.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b01ef390 con 0x7f65b00cf160 2023-10-30T12:28:45.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65b5594640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b00cec20 con 0x7f65b00ca0e0 2023-10-30T12:28:45.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65af7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b01538a0 0x7f65b01ef7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65af7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b01538a0 0x7f65b01ef7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47444/0 (socket says 172.21.15.125:47444) 2023-10-30T12:28:45.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65af7fe640 1 -- 172.21.15.125:0/1277646465 learned_addr learned my addr 172.21.15.125:0/1277646465 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:45.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65aeffd640 1 --2- 172.21.15.125:0/1277646465 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b00ca0e0 0x7f65b00ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65ae7fc640 1 --2- 172.21.15.125:0/1277646465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65b00cf160 0x7f65b0142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65af7fe640 1 -- 172.21.15.125:0/1277646465 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b00ca0e0 msgr2=0x7f65b00ce630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65af7fe640 1 --2- 172.21.15.125:0/1277646465 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b00ca0e0 0x7f65b00ce630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65af7fe640 1 -- 172.21.15.125:0/1277646465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65b00cf160 msgr2=0x7f65b0142550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65af7fe640 1 --2- 172.21.15.125:0/1277646465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65b00cf160 0x7f65b0142550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.539+0000 7f65af7fe640 1 -- 172.21.15.125:0/1277646465 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a40094d0 con 0x7f65b01538a0 2023-10-30T12:28:45.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.543+0000 7f65af7fe640 1 --2- 172.21.15.125:0/1277646465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b01538a0 0x7f65b01ef7c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f65a800c6c0 tx=0x7f65a800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:45.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.543+0000 7f65977fe640 1 -- 172.21.15.125:0/1277646465 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a8003b80 con 0x7f65b01538a0 2023-10-30T12:28:45.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.543+0000 7f65977fe640 1 -- 172.21.15.125:0/1277646465 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65a8003ce0 con 0x7f65b01538a0 2023-10-30T12:28:45.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.543+0000 7f65977fe640 1 -- 172.21.15.125:0/1277646465 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65a80052f0 con 0x7f65b01538a0 2023-10-30T12:28:45.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.543+0000 7f65b5594640 1 -- 172.21.15.125:0/1277646465 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b00ceec0 con 0x7f65b01538a0 2023-10-30T12:28:45.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.543+0000 7f65b5594640 1 -- 172.21.15.125:0/1277646465 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b0149080 con 0x7f65b01538a0 2023-10-30T12:28:45.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.543+0000 7f65b5594640 1 -- 172.21.15.125:0/1277646465 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b00b2a50 con 0x7f65b01538a0 2023-10-30T12:28:45.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.543+0000 7f65977fe640 1 -- 172.21.15.125:0/1277646465 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f65a8005450 con 0x7f65b01538a0 2023-10-30T12:28:45.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.543+0000 7f65977fe640 1 --2- 172.21.15.125:0/1277646465 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f659c0664a0 0x7f659c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.543+0000 7f65977fe640 1 -- 172.21.15.125:0/1277646465 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f65a8083c80 con 0x7f65b01538a0 2023-10-30T12:28:45.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.543+0000 7f65aeffd640 1 --2- 172.21.15.125:0/1277646465 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f659c0664a0 0x7f659c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.547+0000 7f65aeffd640 1 --2- 172.21.15.125:0/1277646465 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f659c0664a0 0x7f659c068960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f65a000b930 tx=0x7f65a0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:45.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.547+0000 7f65977fe640 1 -- 172.21.15.125:0/1277646465 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f65a808a080 con 0x7f65b01538a0 2023-10-30T12:28:45.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.703+0000 7f65b5594640 1 -- 172.21.15.125:0/1277646465 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65b01ef4e0 con 0x7f65b01538a0 2023-10-30T12:28:45.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.703+0000 7f65977fe640 1 -- 172.21.15.125:0/1277646465 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f65b01ef4e0 con 0x7f65b01538a0 2023-10-30T12:28:45.706 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:45.707 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":21,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:44.410415+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:45.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.707+0000 7f65b5594640 1 -- 172.21.15.125:0/1277646465 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f659c0664a0 msgr2=0x7f659c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.707+0000 7f65b5594640 1 --2- 172.21.15.125:0/1277646465 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f659c0664a0 0x7f659c068960 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f65a000b930 tx=0x7f65a0008040 comp rx=0 tx=0).stop 2023-10-30T12:28:45.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.707+0000 7f65b5594640 1 -- 172.21.15.125:0/1277646465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b01538a0 msgr2=0x7f65b01ef7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.707+0000 7f65b5594640 1 --2- 172.21.15.125:0/1277646465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b01538a0 0x7f65b01ef7c0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f65a800c6c0 tx=0x7f65a800cb80 comp rx=0 tx=0).stop 2023-10-30T12:28:45.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.707+0000 7f65b5594640 1 -- 172.21.15.125:0/1277646465 shutdown_connections 2023-10-30T12:28:45.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.707+0000 7f65b5594640 1 --2- 172.21.15.125:0/1277646465 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f659c0664a0 0x7f659c068960 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.707+0000 7f65b5594640 1 --2- 172.21.15.125:0/1277646465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b01538a0 0x7f65b01ef7c0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.707+0000 7f65b5594640 1 --2- 172.21.15.125:0/1277646465 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65b00cf160 0x7f65b0142550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.707+0000 7f65b5594640 1 --2- 172.21.15.125:0/1277646465 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b00ca0e0 0x7f65b00ce630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.707+0000 7f65b5594640 1 -- 172.21.15.125:0/1277646465 >> 172.21.15.125:0/1277646465 conn(0x7f65b006bfe0 msgr2=0x7f65b0152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:45.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.707+0000 7f65b5594640 1 -- 172.21.15.125:0/1277646465 shutdown_connections 2023-10-30T12:28:45.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.707+0000 7f65b5594640 1 -- 172.21.15.125:0/1277646465 wait complete. 2023-10-30T12:28:45.727 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-30T12:28:45.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.863+0000 7ff63b288640 1 Processor -- start 2023-10-30T12:28:45.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.863+0000 7ff63b288640 1 -- start start 2023-10-30T12:28:45.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.867+0000 7ff63b288640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff634153c20 0x7ff63415a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.867+0000 7ff63b288640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6340cb4c0 0x7ff6340cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.867+0000 7ff63b288640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff63415a6b0 con 0x7ff634153840 2023-10-30T12:28:45.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.867+0000 7ff63b288640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff63415dbc0 con 0x7ff634153c20 2023-10-30T12:28:45.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.867+0000 7ff63b288640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63415d730 con 0x7ff6340cb4c0 2023-10-30T12:28:45.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.867+0000 7ff639a85640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff634153c20 0x7ff63415a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.867+0000 7ff639a85640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff634153c20 0x7ff63415a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47448/0 (socket says 172.21.15.125:47448) 2023-10-30T12:28:45.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.867+0000 7ff639a85640 1 -- 172.21.15.125:0/3152117560 learned_addr learned my addr 172.21.15.125:0/3152117560 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:45.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.867+0000 7ff63a286640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff634153840 0x7ff6340c3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43314/0 (socket says 172.21.15.125:43314) 2023-10-30T12:28:45.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.867+0000 7ff63aa87640 1 --2- 172.21.15.125:0/3152117560 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6340cb4c0 0x7ff6340cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.867+0000 7ff639a85640 1 -- 172.21.15.125:0/3152117560 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6340cb4c0 msgr2=0x7ff6340cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:45.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.867+0000 7ff639a85640 1 --2- 172.21.15.125:0/3152117560 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6340cb4c0 0x7ff6340cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.867+0000 7ff639a85640 1 -- 172.21.15.125:0/3152117560 >> v1:172.21.15.125:6789/0 conn(0x7ff634153840 legacy=0x7ff6340c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.867+0000 7ff639a85640 1 -- 172.21.15.125:0/3152117560 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6340ca8f0 con 0x7ff634153c20 2023-10-30T12:28:45.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff639a85640 1 --2- 172.21.15.125:0/3152117560 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff634153c20 0x7ff63415a170 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff624009870 tx=0x7ff62401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5235e9f95a8534f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:45.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff639284640 1 -- 172.21.15.125:0/3152117560 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6240a8a30 con 0x7ff634153c20 2023-10-30T12:28:45.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff639284640 1 -- 172.21.15.125:0/3152117560 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6240a8b90 con 0x7ff634153c20 2023-10-30T12:28:45.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff639284640 1 -- 172.21.15.125:0/3152117560 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6240b0440 con 0x7ff634153c20 2023-10-30T12:28:45.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 -- 172.21.15.125:0/3152117560 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff634153c20 msgr2=0x7ff63415a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 --2- 172.21.15.125:0/3152117560 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff634153c20 0x7ff63415a170 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff624009870 tx=0x7ff62401ed30 comp rx=0 tx=0).stop 2023-10-30T12:28:45.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 -- 172.21.15.125:0/3152117560 shutdown_connections 2023-10-30T12:28:45.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 --2- 172.21.15.125:0/3152117560 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff634153c20 0x7ff63415a170 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 --2- 172.21.15.125:0/3152117560 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6340cb4c0 0x7ff6340cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 -- 172.21.15.125:0/3152117560 >> 172.21.15.125:0/3152117560 conn(0x7ff63406be40 msgr2=0x7ff63406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:45.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 -- 172.21.15.125:0/3152117560 shutdown_connections 2023-10-30T12:28:45.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 -- 172.21.15.125:0/3152117560 wait complete. 2023-10-30T12:28:45.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 Processor -- start 2023-10-30T12:28:45.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 -- start start 2023-10-30T12:28:45.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6340cb4c0 0x7ff634149370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff634153840 0x7ff6340ce040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634153c20 0x7ff6340ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6341502c0 con 0x7ff6340cb4c0 2023-10-30T12:28:45.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6341f7310 con 0x7ff634153840 2023-10-30T12:28:45.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.871+0000 7ff63b288640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6340c8ff0 con 0x7ff634153c20 2023-10-30T12:28:45.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff63a286640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6340cb4c0 0x7ff634149370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff63a286640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6340cb4c0 0x7ff634149370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47458/0 (socket says 172.21.15.125:47458) 2023-10-30T12:28:45.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff63a286640 1 -- 172.21.15.125:0/2692098425 learned_addr learned my addr 172.21.15.125:0/2692098425 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:45.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff63aa87640 1 --2- 172.21.15.125:0/2692098425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634153c20 0x7ff6340ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff639a85640 1 --2- 172.21.15.125:0/2692098425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff634153840 0x7ff6340ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff63a286640 1 -- 172.21.15.125:0/2692098425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634153c20 msgr2=0x7ff6340ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff63a286640 1 --2- 172.21.15.125:0/2692098425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634153c20 0x7ff6340ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff63a286640 1 -- 172.21.15.125:0/2692098425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff634153840 msgr2=0x7ff6340ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:45.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff63a286640 1 --2- 172.21.15.125:0/2692098425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff634153840 0x7ff6340ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:45.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff63a286640 1 -- 172.21.15.125:0/2692098425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6240094d0 con 0x7ff6340cb4c0 2023-10-30T12:28:45.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff63a286640 1 --2- 172.21.15.125:0/2692098425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6340cb4c0 0x7ff634149370 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff63000c490 tx=0x7ff63000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:45.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff62affd640 1 -- 172.21.15.125:0/2692098425 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff630007480 con 0x7ff6340cb4c0 2023-10-30T12:28:45.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff62affd640 1 -- 172.21.15.125:0/2692098425 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6300075e0 con 0x7ff6340cb4c0 2023-10-30T12:28:45.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff63b288640 1 -- 172.21.15.125:0/2692098425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6341f6d90 con 0x7ff6340cb4c0 2023-10-30T12:28:45.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff62affd640 1 -- 172.21.15.125:0/2692098425 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff630004a10 con 0x7ff6340cb4c0 2023-10-30T12:28:45.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff63b288640 1 -- 172.21.15.125:0/2692098425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6340c9290 con 0x7ff6340cb4c0 2023-10-30T12:28:45.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff63b288640 1 -- 172.21.15.125:0/2692098425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63415cdc0 con 0x7ff6340cb4c0 2023-10-30T12:28:45.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff62affd640 1 -- 172.21.15.125:0/2692098425 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff630004b70 con 0x7ff6340cb4c0 2023-10-30T12:28:45.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff62affd640 1 --2- 172.21.15.125:0/2692098425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6180664d0 0x7ff618068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:45.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.875+0000 7ff62affd640 1 -- 172.21.15.125:0/2692098425 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff630083180 con 0x7ff6340cb4c0 2023-10-30T12:28:45.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.879+0000 7ff639a85640 1 --2- 172.21.15.125:0/2692098425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6180664d0 0x7ff618068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:45.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.879+0000 7ff639a85640 1 --2- 172.21.15.125:0/2692098425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6180664d0 0x7ff618068990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff6240b2000 tx=0x7ff624008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:45.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:45.879+0000 7ff62affd640 1 -- 172.21.15.125:0/2692098425 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff630050600 con 0x7ff6340cb4c0 2023-10-30T12:28:46.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:46.071+0000 7ff63b288640 1 -- 172.21.15.125:0/2692098425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff63415a6b0 con 0x7ff6340cb4c0 2023-10-30T12:28:47.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.063+0000 7ff62affd640 1 -- 172.21.15.125:0/2692098425 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff63415a6b0 con 0x7ff6340cb4c0 2023-10-30T12:28:47.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.067+0000 7ff63b288640 1 -- 172.21.15.125:0/2692098425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6180664d0 msgr2=0x7ff618068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.067+0000 7ff63b288640 1 --2- 172.21.15.125:0/2692098425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6180664d0 0x7ff618068990 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff6240b2000 tx=0x7ff624008040 comp rx=0 tx=0).stop 2023-10-30T12:28:47.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.067+0000 7ff63b288640 1 -- 172.21.15.125:0/2692098425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6340cb4c0 msgr2=0x7ff634149370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.067+0000 7ff63b288640 1 --2- 172.21.15.125:0/2692098425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6340cb4c0 0x7ff634149370 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff63000c490 tx=0x7ff63000c950 comp rx=0 tx=0).stop 2023-10-30T12:28:47.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.067+0000 7ff63b288640 1 -- 172.21.15.125:0/2692098425 shutdown_connections 2023-10-30T12:28:47.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.067+0000 7ff63b288640 1 --2- 172.21.15.125:0/2692098425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6180664d0 0x7ff618068990 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.067+0000 7ff63b288640 1 --2- 172.21.15.125:0/2692098425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634153c20 0x7ff6340ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.067+0000 7ff63b288640 1 --2- 172.21.15.125:0/2692098425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff634153840 0x7ff6340ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.067+0000 7ff63b288640 1 --2- 172.21.15.125:0/2692098425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6340cb4c0 0x7ff634149370 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.067+0000 7ff63b288640 1 -- 172.21.15.125:0/2692098425 >> 172.21.15.125:0/2692098425 conn(0x7ff63406be40 msgr2=0x7ff634159350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:47.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.067+0000 7ff63b288640 1 -- 172.21.15.125:0/2692098425 shutdown_connections 2023-10-30T12:28:47.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.067+0000 7ff63b288640 1 -- 172.21.15.125:0/2692098425 wait complete. 2023-10-30T12:28:47.091 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300} 2023-10-30T12:28:47.091 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-10-30T12:28:47.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.231+0000 7f3ce0e79640 1 Processor -- start 2023-10-30T12:28:47.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.231+0000 7f3ce0e79640 1 -- start start 2023-10-30T12:28:47.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3ce0e79640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3cdc150170 0x7f3cdc155d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3ce0e79640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cdc0b0cb0 0x7f3cdc06a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3ce0e79640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3cdc156260 0x7f3cdc06ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3ce0e79640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cdc1592c0 con 0x7f3cdc156260 2023-10-30T12:28:47.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3ce0e79640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cdc15a040 con 0x7f3cdc150170 2023-10-30T12:28:47.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3ce0e79640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cdc072170 con 0x7f3cdc0b0cb0 2023-10-30T12:28:47.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3cdad9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3cdc156260 0x7f3cdc06ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3cdad9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3cdc156260 0x7f3cdc06ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47470/0 (socket says 172.21.15.125:47470) 2023-10-30T12:28:47.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3cdad9d640 1 -- 172.21.15.125:0/670548111 learned_addr learned my addr 172.21.15.125:0/670548111 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:47.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3cda59c640 1 --2- 172.21.15.125:0/670548111 >> v2:172.21.15.136:3300/0 conn(0x7f3cdc150170 0x7f3cdc155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3cd3fff640 1 --2- 172.21.15.125:0/670548111 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cdc0b0cb0 0x7f3cdc06a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3cdad9d640 1 -- 172.21.15.125:0/670548111 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cdc0b0cb0 msgr2=0x7f3cdc06a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:47.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3cdad9d640 1 --2- 172.21.15.125:0/670548111 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cdc0b0cb0 0x7f3cdc06a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3cdad9d640 1 -- 172.21.15.125:0/670548111 >> v2:172.21.15.136:3300/0 conn(0x7f3cdc150170 msgr2=0x7f3cdc155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:47.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3cdad9d640 1 --2- 172.21.15.125:0/670548111 >> v2:172.21.15.136:3300/0 conn(0x7f3cdc150170 0x7f3cdc155d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.235+0000 7f3cdad9d640 1 -- 172.21.15.125:0/670548111 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cdc159560 con 0x7f3cdc156260 2023-10-30T12:28:47.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3cdad9d640 1 --2- 172.21.15.125:0/670548111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cdc156260 0x7f3cdc06ac60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc0c89d0 tx=0x7f3cd401daa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33be7c9b6f0c78e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:47.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3cd9d9b640 1 -- 172.21.15.125:0/670548111 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd400a2d0 con 0x7f3cdc156260 2023-10-30T12:28:47.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3cd9d9b640 1 -- 172.21.15.125:0/670548111 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd40a68a0 con 0x7f3cdc156260 2023-10-30T12:28:47.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3cd9d9b640 1 -- 172.21.15.125:0/670548111 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd40a6c30 con 0x7f3cdc156260 2023-10-30T12:28:47.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3ce0e79640 1 -- 172.21.15.125:0/670548111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cdc156260 msgr2=0x7f3cdc06ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3ce0e79640 1 --2- 172.21.15.125:0/670548111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cdc156260 0x7f3cdc06ac60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc0c89d0 tx=0x7f3cd401daa0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3ce0e79640 1 -- 172.21.15.125:0/670548111 shutdown_connections 2023-10-30T12:28:47.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3ce0e79640 1 --2- 172.21.15.125:0/670548111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cdc156260 0x7f3cdc06ac60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3ce0e79640 1 --2- 172.21.15.125:0/670548111 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cdc0b0cb0 0x7f3cdc06a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3ce0e79640 1 --2- 172.21.15.125:0/670548111 >> v2:172.21.15.136:3300/0 conn(0x7f3cdc150170 0x7f3cdc155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3ce0e79640 1 -- 172.21.15.125:0/670548111 >> 172.21.15.125:0/670548111 conn(0x7f3cdc0c7910 msgr2=0x7f3cdc0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:47.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3ce0e79640 1 -- 172.21.15.125:0/670548111 shutdown_connections 2023-10-30T12:28:47.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3ce0e79640 1 -- 172.21.15.125:0/670548111 wait complete. 2023-10-30T12:28:47.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3ce0e79640 1 Processor -- start 2023-10-30T12:28:47.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.239+0000 7f3ce0e79640 1 -- start start 2023-10-30T12:28:47.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3ce0e79640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cdc0b0cb0 0x7f3cdc14adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3ce0e79640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cdc150170 0x7f3cdc14e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3ce0e79640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3cdc156260 0x7f3cdc1f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3ce0e79640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cdc14f9f0 con 0x7f3cdc0b0cb0 2023-10-30T12:28:47.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3ce0e79640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cdc14a1b0 con 0x7f3cdc156260 2023-10-30T12:28:47.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3ce0e79640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cdc14e0a0 con 0x7f3cdc150170 2023-10-30T12:28:47.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cda59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cdc0b0cb0 0x7f3cdc14adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cda59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cdc0b0cb0 0x7f3cdc14adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47484/0 (socket says 172.21.15.125:47484) 2023-10-30T12:28:47.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cda59c640 1 -- 172.21.15.125:0/1988127416 learned_addr learned my addr 172.21.15.125:0/1988127416 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:47.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cd3fff640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cdc150170 0x7f3cdc14e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cdad9d640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3cdc156260 0x7f3cdc1f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cda59c640 1 -- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cdc150170 msgr2=0x7f3cdc14e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cda59c640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cdc150170 0x7f3cdc14e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cda59c640 1 -- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3cdc156260 msgr2=0x7f3cdc1f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:47.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cda59c640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3cdc156260 0x7f3cdc1f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:47.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cda59c640 1 -- 172.21.15.125:0/1988127416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd400ac40 con 0x7f3cdc0b0cb0 2023-10-30T12:28:47.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cdad9d640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3cdc156260 0x7f3cdc1f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:47.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cda59c640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cdc0b0cb0 0x7f3cdc14adc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00e9b0 tx=0x7f3ccc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:47.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cd2ffd640 1 -- 172.21.15.125:0/1988127416 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc009670 con 0x7f3cdc0b0cb0 2023-10-30T12:28:47.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cd2ffd640 1 -- 172.21.15.125:0/1988127416 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ccc00c9a0 con 0x7f3cdc0b0cb0 2023-10-30T12:28:47.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cd2ffd640 1 -- 172.21.15.125:0/1988127416 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ccc00cd10 con 0x7f3cdc0b0cb0 2023-10-30T12:28:47.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3ce0e79640 1 -- 172.21.15.125:0/1988127416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cdc1f2af0 con 0x7f3cdc0b0cb0 2023-10-30T12:28:47.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3ce0e79640 1 -- 172.21.15.125:0/1988127416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cdc14f270 con 0x7f3cdc0b0cb0 2023-10-30T12:28:47.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3ce0e79640 1 -- 172.21.15.125:0/1988127416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cdc04e170 con 0x7f3cdc0b0cb0 2023-10-30T12:28:47.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.243+0000 7f3cd2ffd640 1 -- 172.21.15.125:0/1988127416 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3ccc013470 con 0x7f3cdc0b0cb0 2023-10-30T12:28:47.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.247+0000 7f3cd2ffd640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cbc0664d0 0x7f3cbc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:47.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.247+0000 7f3cd2ffd640 1 -- 172.21.15.125:0/1988127416 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ccc084410 con 0x7f3cdc0b0cb0 2023-10-30T12:28:47.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.247+0000 7f3cd3fff640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cbc0664d0 0x7f3cbc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:47.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.247+0000 7f3cd3fff640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cbc0664d0 0x7f3cbc068990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc14c830 tx=0x7f3cc4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:47.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.251+0000 7f3cd2ffd640 1 -- 172.21.15.125:0/1988127416 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3ccc051880 con 0x7f3cdc0b0cb0 2023-10-30T12:28:47.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:47.431+0000 7f3ce0e79640 1 -- 172.21.15.125:0/1988127416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f3cdc0c7200 con 0x7f3cdc0b0cb0 2023-10-30T12:28:48.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.067+0000 7f3cd2ffd640 1 -- 172.21.15.125:0/1988127416 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v6) v1 ==== 105+0+0 (secure 0 0 0) 0x7f3ccc054ea0 con 0x7f3cdc0b0cb0 2023-10-30T12:28:48.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.067+0000 7f3ce0e79640 1 -- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cbc0664d0 msgr2=0x7f3cbc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.067+0000 7f3ce0e79640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cbc0664d0 0x7f3cbc068990 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc14c830 tx=0x7f3cc4008040 comp rx=0 tx=0).stop 2023-10-30T12:28:48.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.067+0000 7f3ce0e79640 1 -- 172.21.15.125:0/1988127416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cdc0b0cb0 msgr2=0x7f3cdc14adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.067+0000 7f3ce0e79640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cdc0b0cb0 0x7f3cdc14adc0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc00e9b0 tx=0x7f3ccc00ee70 comp rx=0 tx=0).stop 2023-10-30T12:28:48.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.067+0000 7f3ce0e79640 1 -- 172.21.15.125:0/1988127416 shutdown_connections 2023-10-30T12:28:48.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.067+0000 7f3ce0e79640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cbc0664d0 0x7f3cbc068990 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.067+0000 7f3ce0e79640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3cdc156260 0x7f3cdc1f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.067+0000 7f3ce0e79640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cdc150170 0x7f3cdc14e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.067+0000 7f3ce0e79640 1 --2- 172.21.15.125:0/1988127416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cdc0b0cb0 0x7f3cdc14adc0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.067+0000 7f3ce0e79640 1 -- 172.21.15.125:0/1988127416 >> 172.21.15.125:0/1988127416 conn(0x7f3cdc0c7910 msgr2=0x7f3cdc06a720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:48.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.067+0000 7f3ce0e79640 1 -- 172.21.15.125:0/1988127416 shutdown_connections 2023-10-30T12:28:48.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.067+0000 7f3ce0e79640 1 -- 172.21.15.125:0/1988127416 wait complete. 2023-10-30T12:28:48.093 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-30T12:28:48.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.231+0000 7f3271980640 1 Processor -- start 2023-10-30T12:28:48.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.231+0000 7f3271980640 1 -- start start 2023-10-30T12:28:48.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.235+0000 7f3271980640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f326c159930 0x7f326c0cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.235+0000 7f3271980640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f326c0cf6e0 0x7f326c15d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.235+0000 7f3271980640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f326c15a920 con 0x7f326c153980 2023-10-30T12:28:48.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.235+0000 7f3271980640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326c0cb930 con 0x7f326c159930 2023-10-30T12:28:48.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.235+0000 7f3271980640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f326c159dc0 con 0x7f326c0cf6e0 2023-10-30T12:28:48.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.235+0000 7f32627fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f326c159930 0x7f326c0cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.235+0000 7f326affd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f326c153980 0x7f326c0caca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43326/0 (socket says 172.21.15.125:43326) 2023-10-30T12:28:48.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.235+0000 7f326affd640 1 -- 172.21.15.125:0/3440331344 learned_addr learned my addr 172.21.15.125:0/3440331344 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:48.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.235+0000 7f326b7fe640 1 --2- 172.21.15.125:0/3440331344 >> v2:172.21.15.136:3300/0 conn(0x7f326c0cf6e0 0x7f326c15d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.235+0000 7f32627fc640 1 -- 172.21.15.125:0/3440331344 >> v2:172.21.15.136:3300/0 conn(0x7f326c0cf6e0 msgr2=0x7f326c15d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:48.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.235+0000 7f32627fc640 1 --2- 172.21.15.125:0/3440331344 >> v2:172.21.15.136:3300/0 conn(0x7f326c0cf6e0 0x7f326c15d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.235+0000 7f32627fc640 1 -- 172.21.15.125:0/3440331344 >> v1:172.21.15.125:6789/0 conn(0x7f326c153980 legacy=0x7f326c0caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.235+0000 7f32627fc640 1 -- 172.21.15.125:0/3440331344 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f326c15a080 con 0x7f326c159930 2023-10-30T12:28:48.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.235+0000 7f326a7fc640 1 -- 172.21.15.125:0/3440331344 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63376438 0 0) 0x7f326c15a920 con 0x7f326c153980 2023-10-30T12:28:48.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f32627fc640 1 --2- 172.21.15.125:0/3440331344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f326c159930 0x7f326c0cf1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f326c0b1cf0 tx=0x7f325401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=523c958bc60b396c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:48.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f326a7fc640 1 -- 172.21.15.125:0/3440331344 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32540ab600 con 0x7f326c159930 2023-10-30T12:28:48.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f326a7fc640 1 -- 172.21.15.125:0/3440331344 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32540a7050 con 0x7f326c159930 2023-10-30T12:28:48.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f326a7fc640 1 -- 172.21.15.125:0/3440331344 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32540ab980 con 0x7f326c159930 2023-10-30T12:28:48.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 -- 172.21.15.125:0/3440331344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f326c159930 msgr2=0x7f326c0cf1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 --2- 172.21.15.125:0/3440331344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f326c159930 0x7f326c0cf1a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f326c0b1cf0 tx=0x7f325401c430 comp rx=0 tx=0).stop 2023-10-30T12:28:48.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 -- 172.21.15.125:0/3440331344 shutdown_connections 2023-10-30T12:28:48.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 --2- 172.21.15.125:0/3440331344 >> v2:172.21.15.136:3300/0 conn(0x7f326c0cf6e0 0x7f326c15d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 --2- 172.21.15.125:0/3440331344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f326c159930 0x7f326c0cf1a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 -- 172.21.15.125:0/3440331344 >> 172.21.15.125:0/3440331344 conn(0x7f326c06c0f0 msgr2=0x7f326c06c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:48.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 -- 172.21.15.125:0/3440331344 shutdown_connections 2023-10-30T12:28:48.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 -- 172.21.15.125:0/3440331344 wait complete. 2023-10-30T12:28:48.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 Processor -- start 2023-10-30T12:28:48.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 -- start start 2023-10-30T12:28:48.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f326c0cf6e0 0x7f326c1f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f326c153980 0x7f326c1454a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f326c1459e0 0x7f326c0c4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c152ad0 con 0x7f326c1459e0 2023-10-30T12:28:48.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f326c151c40 con 0x7f326c0cf6e0 2023-10-30T12:28:48.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.239+0000 7f3271980640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f326c1f2900 con 0x7f326c153980 2023-10-30T12:28:48.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f326b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f326c1459e0 0x7f326c0c4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f326b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f326c1459e0 0x7f326c0c4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47512/0 (socket says 172.21.15.125:47512) 2023-10-30T12:28:48.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f326b7fe640 1 -- 172.21.15.125:0/1002088656 learned_addr learned my addr 172.21.15.125:0/1002088656 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:48.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f326affd640 1 --2- 172.21.15.125:0/1002088656 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f326c0cf6e0 0x7f326c1f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f32627fc640 1 --2- 172.21.15.125:0/1002088656 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f326c153980 0x7f326c1454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f326b7fe640 1 -- 172.21.15.125:0/1002088656 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f326c153980 msgr2=0x7f326c1454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f326b7fe640 1 --2- 172.21.15.125:0/1002088656 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f326c153980 0x7f326c1454a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f326b7fe640 1 -- 172.21.15.125:0/1002088656 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f326c0cf6e0 msgr2=0x7f326c1f23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:48.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f326b7fe640 1 --2- 172.21.15.125:0/1002088656 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f326c0cf6e0 0x7f326c1f23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:48.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f326b7fe640 1 -- 172.21.15.125:0/1002088656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3254009420 con 0x7f326c1459e0 2023-10-30T12:28:48.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f326b7fe640 1 --2- 172.21.15.125:0/1002088656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f326c1459e0 0x7f326c0c4150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3264007e40 tx=0x7f326400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:48.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f3268ff9640 1 -- 172.21.15.125:0/1002088656 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3264014620 con 0x7f326c1459e0 2023-10-30T12:28:48.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f3268ff9640 1 -- 172.21.15.125:0/1002088656 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f326400ee90 con 0x7f326c1459e0 2023-10-30T12:28:48.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f3268ff9640 1 -- 172.21.15.125:0/1002088656 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32640149e0 con 0x7f326c1459e0 2023-10-30T12:28:48.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f3271980640 1 -- 172.21.15.125:0/1002088656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f326c147010 con 0x7f326c1459e0 2023-10-30T12:28:48.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f3271980640 1 -- 172.21.15.125:0/1002088656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f326c1472d0 con 0x7f326c1459e0 2023-10-30T12:28:48.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f3271980640 1 -- 172.21.15.125:0/1002088656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f326c151df0 con 0x7f326c1459e0 2023-10-30T12:28:48.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f3268ff9640 1 -- 172.21.15.125:0/1002088656 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3264014b40 con 0x7f326c1459e0 2023-10-30T12:28:48.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.243+0000 7f3268ff9640 1 --2- 172.21.15.125:0/1002088656 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f324c0664d0 0x7f324c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:48.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.247+0000 7f3268ff9640 1 -- 172.21.15.125:0/1002088656 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32640848c0 con 0x7f326c1459e0 2023-10-30T12:28:48.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.247+0000 7f326affd640 1 --2- 172.21.15.125:0/1002088656 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f324c0664d0 0x7f324c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:48.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.247+0000 7f326affd640 1 --2- 172.21.15.125:0/1002088656 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f324c0664d0 0x7f324c068990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f325c000f70 tx=0x7f325c007b60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:48.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.247+0000 7f3268ff9640 1 -- 172.21.15.125:0/1002088656 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f326c151df0 con 0x7f326c1459e0 2023-10-30T12:28:48.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:48.431+0000 7f3271980640 1 -- 172.21.15.125:0/1002088656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f326c151df0 con 0x7f326c1459e0 2023-10-30T12:28:49.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.083+0000 7f3268ff9640 1 -- 172.21.15.125:0/1002088656 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7f326c151df0 con 0x7f326c1459e0 2023-10-30T12:28:49.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.087+0000 7f3271980640 1 -- 172.21.15.125:0/1002088656 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f324c0664d0 msgr2=0x7f324c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:49.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.087+0000 7f3271980640 1 --2- 172.21.15.125:0/1002088656 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f324c0664d0 0x7f324c068990 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f325c000f70 tx=0x7f325c007b60 comp rx=0 tx=0).stop 2023-10-30T12:28:49.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.087+0000 7f3271980640 1 -- 172.21.15.125:0/1002088656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f326c1459e0 msgr2=0x7f326c0c4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:49.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.087+0000 7f3271980640 1 --2- 172.21.15.125:0/1002088656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f326c1459e0 0x7f326c0c4150 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3264007e40 tx=0x7f326400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.087+0000 7f3271980640 1 -- 172.21.15.125:0/1002088656 shutdown_connections 2023-10-30T12:28:49.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.087+0000 7f3271980640 1 --2- 172.21.15.125:0/1002088656 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f324c0664d0 0x7f324c068990 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.087+0000 7f3271980640 1 --2- 172.21.15.125:0/1002088656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f326c1459e0 0x7f326c0c4150 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.087+0000 7f3271980640 1 --2- 172.21.15.125:0/1002088656 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f326c153980 0x7f326c1454a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.087+0000 7f3271980640 1 --2- 172.21.15.125:0/1002088656 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f326c0cf6e0 0x7f326c1f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.087+0000 7f3271980640 1 -- 172.21.15.125:0/1002088656 >> 172.21.15.125:0/1002088656 conn(0x7f326c06c0f0 msgr2=0x7f326c066310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:49.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.087+0000 7f3271980640 1 -- 172.21.15.125:0/1002088656 shutdown_connections 2023-10-30T12:28:49.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.087+0000 7f3271980640 1 -- 172.21.15.125:0/1002088656 wait complete. 2023-10-30T12:28:49.111 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-10-30T12:28:49.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.251+0000 7f01eaac9640 1 Processor -- start 2023-10-30T12:28:49.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.251+0000 7f01eaac9640 1 -- start start 2023-10-30T12:28:49.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.251+0000 7f01eaac9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f01e406a450 0x7f01e406a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:49.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.251+0000 7f01eaac9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e415ae60 con 0x7f01e40c8470 2023-10-30T12:28:49.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.251+0000 7f01eaac9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e415b140 con 0x7f01e41519c0 2023-10-30T12:28:49.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.251+0000 7f01eaac9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e406a030 con 0x7f01e406a450 2023-10-30T12:28:49.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.251+0000 7f01dbfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f01e40c8470 0x7f01e40c6dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43332/0 (socket says 172.21.15.125:43332) 2023-10-30T12:28:49.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.251+0000 7f01dbfff640 1 -- 172.21.15.125:0/3106930761 learned_addr learned my addr 172.21.15.125:0/3106930761 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:49.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01e9066640 1 --2- 172.21.15.125:0/3106930761 >> v2:172.21.15.136:3300/0 conn(0x7f01e406a450 0x7f01e406a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:49.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 -- 172.21.15.125:0/3106930761 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652796349 0 0) 0x7f01e415ae60 con 0x7f01e40c8470 2023-10-30T12:28:49.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 -- 172.21.15.125:0/3106930761 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01cc003530 con 0x7f01e40c8470 2023-10-30T12:28:49.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 -- 172.21.15.125:0/3106930761 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1250667172 0 0) 0x7f01cc003530 con 0x7f01e40c8470 2023-10-30T12:28:49.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 -- 172.21.15.125:0/3106930761 >> v2:172.21.15.136:3300/0 conn(0x7f01e406a450 msgr2=0x7f01e406a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:49.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 --2- 172.21.15.125:0/3106930761 >> v2:172.21.15.136:3300/0 conn(0x7f01e406a450 0x7f01e406a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 -- 172.21.15.125:0/3106930761 >> v1:172.21.15.136:6789/0 conn(0x7f01e41519c0 legacy=0x7f01e406d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:49.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 -- 172.21.15.125:0/3106930761 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e415b420 con 0x7f01e40c8470 2023-10-30T12:28:49.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 -- 172.21.15.125:0/3106930761 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f01d4001680 con 0x7f01e40c8470 2023-10-30T12:28:49.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 -- 172.21.15.125:0/3106930761 >> v1:172.21.15.125:6789/0 conn(0x7f01e40c8470 legacy=0x7f01e40c6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:49.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 --2- 172.21.15.125:0/3106930761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01cc003980 0x7f01cc003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:49.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 --2- 172.21.15.125:0/3106930761 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f01cc004380 0x7f01cc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:49.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 --2- 172.21.15.125:0/3106930761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01cc005050 0x7f01cc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:49.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 -- 172.21.15.125:0/3106930761 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e406a030 con 0x7f01cc003980 2023-10-30T12:28:49.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 -- 172.21.15.125:0/3106930761 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc00bfb0 con 0x7f01cc005050 2023-10-30T12:28:49.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01db7fe640 1 -- 172.21.15.125:0/3106930761 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc00c170 con 0x7f01cc004380 2023-10-30T12:28:49.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01e8865640 1 --2- 172.21.15.125:0/3106930761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01cc003980 0x7f01cc003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:49.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01dbfff640 1 --2- 172.21.15.125:0/3106930761 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f01cc004380 0x7f01cc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:49.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01e9066640 1 --2- 172.21.15.125:0/3106930761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01cc005050 0x7f01cc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:49.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01e8865640 1 -- 172.21.15.125:0/3106930761 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f01cc004380 msgr2=0x7f01cc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:49.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01e8865640 1 --2- 172.21.15.125:0/3106930761 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f01cc004380 0x7f01cc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01e8865640 1 -- 172.21.15.125:0/3106930761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01cc005050 msgr2=0x7f01cc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:49.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01e8865640 1 --2- 172.21.15.125:0/3106930761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01cc005050 0x7f01cc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01e8865640 1 -- 172.21.15.125:0/3106930761 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc003530 con 0x7f01cc003980 2023-10-30T12:28:49.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.255+0000 7f01e9066640 1 -- 172.21.15.125:0/3106930761 reap_dead start 2023-10-30T12:28:49.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01e8865640 1 --2- 172.21.15.125:0/3106930761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01cc003980 0x7f01cc003d60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f01e0001860 tx=0x7f01e406db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e91dd1a224e84015 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:49.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01db7fe640 1 -- 172.21.15.125:0/3106930761 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e00ac640 con 0x7f01cc003980 2023-10-30T12:28:49.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01db7fe640 1 -- 172.21.15.125:0/3106930761 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01e00ac7a0 con 0x7f01cc003980 2023-10-30T12:28:49.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01db7fe640 1 -- 172.21.15.125:0/3106930761 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01e00acb90 con 0x7f01cc003980 2023-10-30T12:28:49.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01eaac9640 1 -- 172.21.15.125:0/3106930761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01cc003980 msgr2=0x7f01cc003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:49.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01eaac9640 1 --2- 172.21.15.125:0/3106930761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01cc003980 0x7f01cc003d60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f01e0001860 tx=0x7f01e406db80 comp rx=0 tx=0).stop 2023-10-30T12:28:49.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01eaac9640 1 -- 172.21.15.125:0/3106930761 shutdown_connections 2023-10-30T12:28:49.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01eaac9640 1 --2- 172.21.15.125:0/3106930761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01cc003980 0x7f01cc003d60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01eaac9640 1 -- 172.21.15.125:0/3106930761 >> 172.21.15.125:0/3106930761 conn(0x7f01e406c850 msgr2=0x7f01e406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:49.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01eaac9640 1 -- 172.21.15.125:0/3106930761 shutdown_connections 2023-10-30T12:28:49.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01eaac9640 1 -- 172.21.15.125:0/3106930761 wait complete. 2023-10-30T12:28:49.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01eaac9640 1 Processor -- start 2023-10-30T12:28:49.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01eaac9640 1 -- start start 2023-10-30T12:28:49.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01eaac9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01cc003980 0x7f01e414d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:49.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01eaac9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01e415a470 0x7f01e41f17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:49.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.259+0000 7f01eaac9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f01e41f1cf0 0x7f01e41466b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:49.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01eaac9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e414ddf0 con 0x7f01e415a470 2023-10-30T12:28:49.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01eaac9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e41f0d60 con 0x7f01cc003980 2023-10-30T12:28:49.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01eaac9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e414c430 con 0x7f01e41f1cf0 2023-10-30T12:28:49.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01dbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01e415a470 0x7f01e41f17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:49.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01dbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01e415a470 0x7f01e41f17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47534/0 (socket says 172.21.15.125:47534) 2023-10-30T12:28:49.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01dbfff640 1 -- 172.21.15.125:0/315704912 learned_addr learned my addr 172.21.15.125:0/315704912 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:49.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01e8865640 1 --2- 172.21.15.125:0/315704912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01cc003980 0x7f01e414d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:49.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01e9066640 1 --2- 172.21.15.125:0/315704912 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f01e41f1cf0 0x7f01e41466b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:49.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01dbfff640 1 -- 172.21.15.125:0/315704912 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f01e41f1cf0 msgr2=0x7f01e41466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:49.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01dbfff640 1 --2- 172.21.15.125:0/315704912 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f01e41f1cf0 0x7f01e41466b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01dbfff640 1 -- 172.21.15.125:0/315704912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01cc003980 msgr2=0x7f01e414d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:49.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01dbfff640 1 --2- 172.21.15.125:0/315704912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01cc003980 0x7f01e414d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:49.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01dbfff640 1 -- 172.21.15.125:0/315704912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e41f2170 con 0x7f01e415a470 2023-10-30T12:28:49.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01dbfff640 1 --2- 172.21.15.125:0/315704912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01e415a470 0x7f01e41f17b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f01d400e7b0 tx=0x7f01d400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:49.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01d97fa640 1 -- 172.21.15.125:0/315704912 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d400eec0 con 0x7f01e415a470 2023-10-30T12:28:49.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01d97fa640 1 -- 172.21.15.125:0/315704912 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d400c8d0 con 0x7f01e415a470 2023-10-30T12:28:49.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01eaac9640 1 -- 172.21.15.125:0/315704912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e414cc60 con 0x7f01e415a470 2023-10-30T12:28:49.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01d97fa640 1 -- 172.21.15.125:0/315704912 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d400cc60 con 0x7f01e415a470 2023-10-30T12:28:49.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01eaac9640 1 -- 172.21.15.125:0/315704912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e414c980 con 0x7f01e415a470 2023-10-30T12:28:49.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01eaac9640 1 -- 172.21.15.125:0/315704912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e404e170 con 0x7f01e415a470 2023-10-30T12:28:49.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01d97fa640 1 -- 172.21.15.125:0/315704912 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f01d401d460 con 0x7f01e415a470 2023-10-30T12:28:49.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01d97fa640 1 --2- 172.21.15.125:0/315704912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01d00664a0 0x7f01d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:49.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.263+0000 7f01d97fa640 1 -- 172.21.15.125:0/315704912 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01d4083e90 con 0x7f01e415a470 2023-10-30T12:28:49.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.267+0000 7f01e8865640 1 --2- 172.21.15.125:0/315704912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01d00664a0 0x7f01d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:49.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.267+0000 7f01e8865640 1 --2- 172.21.15.125:0/315704912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01d00664a0 0x7f01d0068960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f01e00ae000 tx=0x7f01e0001240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:49.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.267+0000 7f01d97fa640 1 -- 172.21.15.125:0/315704912 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f01d4051280 con 0x7f01e415a470 2023-10-30T12:28:49.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:49.447+0000 7f01eaac9640 1 -- 172.21.15.125:0/315704912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f01e4071bd0 con 0x7f01e415a470 2023-10-30T12:28:50.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.095+0000 7f01d97fa640 1 -- 172.21.15.125:0/315704912 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) v1 ==== 115+0+0 (secure 0 0 0) 0x7f01d40548a0 con 0x7f01e415a470 2023-10-30T12:28:50.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.099+0000 7f01eaac9640 1 -- 172.21.15.125:0/315704912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01d00664a0 msgr2=0x7f01d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.099+0000 7f01eaac9640 1 --2- 172.21.15.125:0/315704912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01d00664a0 0x7f01d0068960 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f01e00ae000 tx=0x7f01e0001240 comp rx=0 tx=0).stop 2023-10-30T12:28:50.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.099+0000 7f01eaac9640 1 -- 172.21.15.125:0/315704912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01e415a470 msgr2=0x7f01e41f17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.099+0000 7f01eaac9640 1 --2- 172.21.15.125:0/315704912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01e415a470 0x7f01e41f17b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f01d400e7b0 tx=0x7f01d400ec70 comp rx=0 tx=0).stop 2023-10-30T12:28:50.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.099+0000 7f01eaac9640 1 -- 172.21.15.125:0/315704912 shutdown_connections 2023-10-30T12:28:50.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.099+0000 7f01eaac9640 1 --2- 172.21.15.125:0/315704912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01d00664a0 0x7f01d0068960 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.099+0000 7f01eaac9640 1 --2- 172.21.15.125:0/315704912 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f01e41f1cf0 0x7f01e41466b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.099+0000 7f01eaac9640 1 --2- 172.21.15.125:0/315704912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01e415a470 0x7f01e41f17b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.099+0000 7f01eaac9640 1 --2- 172.21.15.125:0/315704912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f01cc003980 0x7f01e414d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.099+0000 7f01eaac9640 1 -- 172.21.15.125:0/315704912 >> 172.21.15.125:0/315704912 conn(0x7f01e406c850 msgr2=0x7f01e40706c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:50.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.099+0000 7f01eaac9640 1 -- 172.21.15.125:0/315704912 shutdown_connections 2023-10-30T12:28:50.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.099+0000 7f01eaac9640 1 -- 172.21.15.125:0/315704912 wait complete. 2023-10-30T12:28:50.124 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-30T12:28:50.125 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:28:50.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d453e640 1 Processor -- start 2023-10-30T12:28:50.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d453e640 1 -- start start 2023-10-30T12:28:50.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d453e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f35cc1538a0 0x7f35cc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d453e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f35cc0ca0e0 0x7f35cc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d453e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35cc0cb800 con 0x7f35cc0cf160 2023-10-30T12:28:50.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d453e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35cc15dc30 con 0x7f35cc0ca0e0 2023-10-30T12:28:50.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d453e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35cc0cb330 con 0x7f35cc1538a0 2023-10-30T12:28:50.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d1ad9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f35cc0ca0e0 0x7f35cc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d1ad9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f35cc0ca0e0 0x7f35cc15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47538/0 (socket says 172.21.15.125:47538) 2023-10-30T12:28:50.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d1ad9640 1 -- 172.21.15.125:0/810047777 learned_addr learned my addr 172.21.15.125:0/810047777 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:50.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d22da640 1 --2- 172.21.15.125:0/810047777 >> v2:172.21.15.136:3300/0 conn(0x7f35cc1538a0 0x7f35cc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d1ad9640 1 -- 172.21.15.125:0/810047777 >> v2:172.21.15.136:3300/0 conn(0x7f35cc1538a0 msgr2=0x7f35cc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:50.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d1ad9640 1 --2- 172.21.15.125:0/810047777 >> v2:172.21.15.136:3300/0 conn(0x7f35cc1538a0 0x7f35cc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d1ad9640 1 -- 172.21.15.125:0/810047777 >> v1:172.21.15.136:6789/0 conn(0x7f35cc0cf160 legacy=0x7f35cc0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d1ad9640 1 -- 172.21.15.125:0/810047777 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc0cad70 con 0x7f35cc0ca0e0 2023-10-30T12:28:50.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.263+0000 7f35d22da640 1 --2- 172.21.15.125:0/810047777 >> v2:172.21.15.136:3300/0 conn(0x7f35cc1538a0 0x7f35cc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:50.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.267+0000 7f35d1ad9640 1 --2- 172.21.15.125:0/810047777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35cc0ca0e0 0x7f35cc15a090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0cfd10 tx=0x7f35bc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e792dc0717274b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:50.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.267+0000 7f35d12d8640 1 -- 172.21.15.125:0/810047777 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35bc0a89f0 con 0x7f35cc0ca0e0 2023-10-30T12:28:50.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.267+0000 7f35d12d8640 1 -- 172.21.15.125:0/810047777 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35bc0a8b50 con 0x7f35cc0ca0e0 2023-10-30T12:28:50.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.267+0000 7f35d12d8640 1 -- 172.21.15.125:0/810047777 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35bc0b0440 con 0x7f35cc0ca0e0 2023-10-30T12:28:50.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.267+0000 7f35d453e640 1 -- 172.21.15.125:0/810047777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35cc0ca0e0 msgr2=0x7f35cc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.267+0000 7f35d453e640 1 --2- 172.21.15.125:0/810047777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35cc0ca0e0 0x7f35cc15a090 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0cfd10 tx=0x7f35bc01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.267+0000 7f35d453e640 1 -- 172.21.15.125:0/810047777 shutdown_connections 2023-10-30T12:28:50.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.267+0000 7f35d453e640 1 --2- 172.21.15.125:0/810047777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35cc0ca0e0 0x7f35cc15a090 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.267+0000 7f35d453e640 1 --2- 172.21.15.125:0/810047777 >> v2:172.21.15.136:3300/0 conn(0x7f35cc1538a0 0x7f35cc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.267+0000 7f35d453e640 1 -- 172.21.15.125:0/810047777 >> 172.21.15.125:0/810047777 conn(0x7f35cc06bfe0 msgr2=0x7f35cc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:50.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.267+0000 7f35d453e640 1 -- 172.21.15.125:0/810047777 shutdown_connections 2023-10-30T12:28:50.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.267+0000 7f35d453e640 1 -- 172.21.15.125:0/810047777 wait complete. 2023-10-30T12:28:50.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d453e640 1 Processor -- start 2023-10-30T12:28:50.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d453e640 1 -- start start 2023-10-30T12:28:50.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d453e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35cc0ca0e0 0x7f35cc140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d453e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35cc0cf160 0x7f35cc0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d453e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35cc1538a0 0x7f35cc0cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d453e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc1e4380 con 0x7f35cc0cf160 2023-10-30T12:28:50.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d453e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc158620 con 0x7f35cc0ca0e0 2023-10-30T12:28:50.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d453e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc159690 con 0x7f35cc1538a0 2023-10-30T12:28:50.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d1ad9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35cc0cf160 0x7f35cc0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d22da640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35cc0ca0e0 0x7f35cc140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d1ad9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35cc0cf160 0x7f35cc0ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47550/0 (socket says 172.21.15.125:47550) 2023-10-30T12:28:50.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d1ad9640 1 -- 172.21.15.125:0/4005953777 learned_addr learned my addr 172.21.15.125:0/4005953777 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:50.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d2adb640 1 --2- 172.21.15.125:0/4005953777 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35cc1538a0 0x7f35cc0cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d1ad9640 1 -- 172.21.15.125:0/4005953777 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35cc1538a0 msgr2=0x7f35cc0cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d1ad9640 1 --2- 172.21.15.125:0/4005953777 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35cc1538a0 0x7f35cc0cd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d1ad9640 1 -- 172.21.15.125:0/4005953777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35cc0ca0e0 msgr2=0x7f35cc140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d1ad9640 1 --2- 172.21.15.125:0/4005953777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35cc0ca0e0 0x7f35cc140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d1ad9640 1 -- 172.21.15.125:0/4005953777 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc0094d0 con 0x7f35cc0cf160 2023-10-30T12:28:50.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d1ad9640 1 --2- 172.21.15.125:0/4005953777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35cc0cf160 0x7f35cc0ccfb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f35bc01b900 tx=0x7f35bc01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:50.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35c2ffd640 1 -- 172.21.15.125:0/4005953777 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35bc0ba600 con 0x7f35cc0cf160 2023-10-30T12:28:50.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35c2ffd640 1 -- 172.21.15.125:0/4005953777 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35bc0ba760 con 0x7f35cc0cf160 2023-10-30T12:28:50.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35c2ffd640 1 -- 172.21.15.125:0/4005953777 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35bc0bab10 con 0x7f35cc0cf160 2023-10-30T12:28:50.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d453e640 1 -- 172.21.15.125:0/4005953777 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35cc1e51e0 con 0x7f35cc0cf160 2023-10-30T12:28:50.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.271+0000 7f35d453e640 1 -- 172.21.15.125:0/4005953777 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35cc1e5610 con 0x7f35cc0cf160 2023-10-30T12:28:50.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.275+0000 7f35d453e640 1 -- 172.21.15.125:0/4005953777 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35cc0b2a50 con 0x7f35cc0cf160 2023-10-30T12:28:50.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.275+0000 7f35c2ffd640 1 -- 172.21.15.125:0/4005953777 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f35bc0bac70 con 0x7f35cc0cf160 2023-10-30T12:28:50.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.275+0000 7f35c2ffd640 1 --2- 172.21.15.125:0/4005953777 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35b00664d0 0x7f35b0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.275+0000 7f35c2ffd640 1 -- 172.21.15.125:0/4005953777 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35bc122300 con 0x7f35cc0cf160 2023-10-30T12:28:50.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.275+0000 7f35d22da640 1 --2- 172.21.15.125:0/4005953777 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35b00664d0 0x7f35b0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.275+0000 7f35d22da640 1 --2- 172.21.15.125:0/4005953777 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35b00664d0 0x7f35b0068990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0cf070 tx=0x7f35c8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:50.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.279+0000 7f35c2ffd640 1 -- 172.21.15.125:0/4005953777 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f35bc0ef770 con 0x7f35cc0cf160 2023-10-30T12:28:50.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.463+0000 7f35d453e640 1 -- 172.21.15.125:0/4005953777 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35cc158790 con 0x7f35cc0cf160 2023-10-30T12:28:50.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.463+0000 7f35c2ffd640 1 -- 172.21.15.125:0/4005953777 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6522 (secure 0 0 0) 0x7f35bc0f2d90 con 0x7f35cc0cf160 2023-10-30T12:28:50.466 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:50.466 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3346501886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3346501886},{"type":"v1","addr":"172.21.15.136:6835","nonce":3346501886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:28:50.102001+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":4294,"mds_1":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:creating","state_seq":1,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:28:50.467 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 9 2023-10-30T12:28:50.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.463+0000 7f35d453e640 1 -- 172.21.15.125:0/4005953777 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35b00664d0 msgr2=0x7f35b0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.463+0000 7f35d453e640 1 --2- 172.21.15.125:0/4005953777 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35b00664d0 0x7f35b0068990 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f35cc0cf070 tx=0x7f35c8008040 comp rx=0 tx=0).stop 2023-10-30T12:28:50.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.463+0000 7f35d453e640 1 -- 172.21.15.125:0/4005953777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35cc0cf160 msgr2=0x7f35cc0ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.463+0000 7f35d453e640 1 --2- 172.21.15.125:0/4005953777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35cc0cf160 0x7f35cc0ccfb0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f35bc01b900 tx=0x7f35bc01ba90 comp rx=0 tx=0).stop 2023-10-30T12:28:50.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.463+0000 7f35d453e640 1 -- 172.21.15.125:0/4005953777 shutdown_connections 2023-10-30T12:28:50.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.463+0000 7f35d453e640 1 --2- 172.21.15.125:0/4005953777 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35b00664d0 0x7f35b0068990 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.463+0000 7f35d453e640 1 --2- 172.21.15.125:0/4005953777 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35cc1538a0 0x7f35cc0cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.463+0000 7f35d453e640 1 --2- 172.21.15.125:0/4005953777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35cc0cf160 0x7f35cc0ccfb0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.463+0000 7f35d453e640 1 --2- 172.21.15.125:0/4005953777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35cc0ca0e0 0x7f35cc140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.463+0000 7f35d453e640 1 -- 172.21.15.125:0/4005953777 >> 172.21.15.125:0/4005953777 conn(0x7f35cc06bfe0 msgr2=0x7f35cc0c5210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:50.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.463+0000 7f35d453e640 1 -- 172.21.15.125:0/4005953777 shutdown_connections 2023-10-30T12:28:50.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.463+0000 7f35d453e640 1 -- 172.21.15.125:0/4005953777 wait complete. 2023-10-30T12:28:50.486 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:50.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fefa23f8640 1 Processor -- start 2023-10-30T12:28:50.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fefa23f8640 1 -- start start 2023-10-30T12:28:50.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fefa23f8640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fef9c0c6b80 0x7fef9c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fefa23f8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fef9c13f4a0 0x7fef9c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fefa23f8640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef9c0c0a50 con 0x7fef9c0c6f60 2023-10-30T12:28:50.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fefa23f8640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9c06f6e0 con 0x7fef9c0c6b80 2023-10-30T12:28:50.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fefa23f8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef9c06f9c0 con 0x7fef9c13f4a0 2023-10-30T12:28:50.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fef9bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fef9c0c6b80 0x7fef9c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fef9bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fef9c0c6b80 0x7fef9c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47566/0 (socket says 172.21.15.125:47566) 2023-10-30T12:28:50.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fef9bfff640 1 -- 172.21.15.125:0/4117885223 learned_addr learned my addr 172.21.15.125:0/4117885223 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:50.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fefa0995640 1 --2- 172.21.15.125:0/4117885223 >> v2:172.21.15.136:3300/0 conn(0x7fef9c13f4a0 0x7fef9c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fef9bfff640 1 -- 172.21.15.125:0/4117885223 >> v2:172.21.15.136:3300/0 conn(0x7fef9c13f4a0 msgr2=0x7fef9c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:50.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fef9bfff640 1 --2- 172.21.15.125:0/4117885223 >> v2:172.21.15.136:3300/0 conn(0x7fef9c13f4a0 0x7fef9c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fef9bfff640 1 -- 172.21.15.125:0/4117885223 >> v1:172.21.15.125:6789/0 conn(0x7fef9c0c6f60 legacy=0x7fef9c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fef9bfff640 1 -- 172.21.15.125:0/4117885223 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef9c06fca0 con 0x7fef9c0c6b80 2023-10-30T12:28:50.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.623+0000 7fef9affd640 1 -- 172.21.15.125:0/4117885223 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601339401 0 0) 0x7fef9c0c0a50 con 0x7fef9c0c6f60 2023-10-30T12:28:50.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.627+0000 7fef9bfff640 1 --2- 172.21.15.125:0/4117885223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef9c0c6b80 0x7fef9c0bfec0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0c8980 tx=0x7fef8c01ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8750e14700aac9de server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:50.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.627+0000 7fef9affd640 1 -- 172.21.15.125:0/4117885223 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c0a7bb0 con 0x7fef9c0c6b80 2023-10-30T12:28:50.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.627+0000 7fef9affd640 1 -- 172.21.15.125:0/4117885223 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8c0a7d10 con 0x7fef9c0c6b80 2023-10-30T12:28:50.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.627+0000 7fef9affd640 1 -- 172.21.15.125:0/4117885223 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c0af660 con 0x7fef9c0c6b80 2023-10-30T12:28:50.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.627+0000 7fefa23f8640 1 -- 172.21.15.125:0/4117885223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef9c0c6b80 msgr2=0x7fef9c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.627+0000 7fefa23f8640 1 --2- 172.21.15.125:0/4117885223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef9c0c6b80 0x7fef9c0bfec0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fef9c0c8980 tx=0x7fef8c01ebd0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.627+0000 7fefa23f8640 1 -- 172.21.15.125:0/4117885223 shutdown_connections 2023-10-30T12:28:50.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.627+0000 7fefa23f8640 1 --2- 172.21.15.125:0/4117885223 >> v2:172.21.15.136:3300/0 conn(0x7fef9c13f4a0 0x7fef9c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.627+0000 7fefa23f8640 1 --2- 172.21.15.125:0/4117885223 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef9c0c6b80 0x7fef9c0bfec0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.627+0000 7fefa23f8640 1 -- 172.21.15.125:0/4117885223 >> 172.21.15.125:0/4117885223 conn(0x7fef9c0c7910 msgr2=0x7fef9c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:50.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.627+0000 7fefa23f8640 1 -- 172.21.15.125:0/4117885223 shutdown_connections 2023-10-30T12:28:50.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.627+0000 7fefa23f8640 1 -- 172.21.15.125:0/4117885223 wait complete. 2023-10-30T12:28:50.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fefa23f8640 1 Processor -- start 2023-10-30T12:28:50.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fefa23f8640 1 -- start start 2023-10-30T12:28:50.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fefa23f8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef9c0c6b80 0x7fef9c0c3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fefa23f8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fef9c0c6f60 0x7fef9c0c4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fefa23f8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef9c13f4a0 0x7fef9c138300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fefa23f8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c1f6510 con 0x7fef9c0c6b80 2023-10-30T12:28:50.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fefa23f8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c1f67b0 con 0x7fef9c13f4a0 2023-10-30T12:28:50.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fefa23f8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9c154290 con 0x7fef9c0c6f60 2023-10-30T12:28:50.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fef9bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef9c0c6b80 0x7fef9c0c3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fef9bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef9c0c6b80 0x7fef9c0c3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47572/0 (socket says 172.21.15.125:47572) 2023-10-30T12:28:50.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fef9bfff640 1 -- 172.21.15.125:0/3478531349 learned_addr learned my addr 172.21.15.125:0/3478531349 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:50.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fefa0995640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef9c13f4a0 0x7fef9c138300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fef9b7fe640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fef9c0c6f60 0x7fef9c0c4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fef9bfff640 1 -- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fef9c0c6f60 msgr2=0x7fef9c0c4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fef9bfff640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fef9c0c6f60 0x7fef9c0c4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fef9bfff640 1 -- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef9c13f4a0 msgr2=0x7fef9c138300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fef9bfff640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef9c13f4a0 0x7fef9c138300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fef9bfff640 1 -- 172.21.15.125:0/3478531349 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef8c00b770 con 0x7fef9c0c6b80 2023-10-30T12:28:50.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fef9b7fe640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fef9c0c6f60 0x7fef9c0c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:50.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fef9bfff640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef9c0c6b80 0x7fef9c0c3ad0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fef8c01b7e0 tx=0x7fef8c01b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:50.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fef98ff9640 1 -- 172.21.15.125:0/3478531349 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c0ba660 con 0x7fef9c0c6b80 2023-10-30T12:28:50.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fef98ff9640 1 -- 172.21.15.125:0/3478531349 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef8c0ba7c0 con 0x7fef9c0c6b80 2023-10-30T12:28:50.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fef98ff9640 1 -- 172.21.15.125:0/3478531349 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef8c0bab70 con 0x7fef9c0c6b80 2023-10-30T12:28:50.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.631+0000 7fefa23f8640 1 -- 172.21.15.125:0/3478531349 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9c154570 con 0x7fef9c0c6b80 2023-10-30T12:28:50.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.635+0000 7fefa23f8640 1 -- 172.21.15.125:0/3478531349 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9c154b30 con 0x7fef9c0c6b80 2023-10-30T12:28:50.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.635+0000 7fefa23f8640 1 -- 172.21.15.125:0/3478531349 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef9c04e170 con 0x7fef9c0c6b80 2023-10-30T12:28:50.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.635+0000 7fef98ff9640 1 -- 172.21.15.125:0/3478531349 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fef8c0c2360 con 0x7fef9c0c6b80 2023-10-30T12:28:50.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.635+0000 7fef98ff9640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fef880664a0 0x7fef88068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.635+0000 7fef98ff9640 1 -- 172.21.15.125:0/3478531349 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef8c11edc0 con 0x7fef9c0c6b80 2023-10-30T12:28:50.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.635+0000 7fef9b7fe640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fef880664a0 0x7fef88068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.639+0000 7fef9b7fe640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fef880664a0 0x7fef88068960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fef9c1ed9d0 tx=0x7fef90009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:50.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.639+0000 7fef98ff9640 1 -- 172.21.15.125:0/3478531349 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fef8c0ec230 con 0x7fef9c0c6b80 2023-10-30T12:28:50.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.791+0000 7fefa23f8640 1 -- 172.21.15.125:0/3478531349 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef9c0c1680 con 0x7fef9c0c6b80 2023-10-30T12:28:50.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.791+0000 7fef98ff9640 1 -- 172.21.15.125:0/3478531349 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16557 (secure 0 0 0) 0x7fef8c0bf090 con 0x7fef9c0c6b80 2023-10-30T12:28:50.795 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:50.795 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:45.927862+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:50.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.795+0000 7fefa23f8640 1 -- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fef880664a0 msgr2=0x7fef88068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.795+0000 7fefa23f8640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fef880664a0 0x7fef88068960 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fef9c1ed9d0 tx=0x7fef90009040 comp rx=0 tx=0).stop 2023-10-30T12:28:50.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.795+0000 7fefa23f8640 1 -- 172.21.15.125:0/3478531349 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef9c0c6b80 msgr2=0x7fef9c0c3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.795+0000 7fefa23f8640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef9c0c6b80 0x7fef9c0c3ad0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fef8c01b7e0 tx=0x7fef8c01b970 comp rx=0 tx=0).stop 2023-10-30T12:28:50.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.795+0000 7fefa23f8640 1 -- 172.21.15.125:0/3478531349 shutdown_connections 2023-10-30T12:28:50.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.795+0000 7fefa23f8640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fef880664a0 0x7fef88068960 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.795+0000 7fefa23f8640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fef9c13f4a0 0x7fef9c138300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.795+0000 7fefa23f8640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fef9c0c6f60 0x7fef9c0c4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.795+0000 7fefa23f8640 1 --2- 172.21.15.125:0/3478531349 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fef9c0c6b80 0x7fef9c0c3ad0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.795+0000 7fefa23f8640 1 -- 172.21.15.125:0/3478531349 >> 172.21.15.125:0/3478531349 conn(0x7fef9c0c7910 msgr2=0x7fef9c0b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:50.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.795+0000 7fefa23f8640 1 -- 172.21.15.125:0/3478531349 shutdown_connections 2023-10-30T12:28:50.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.795+0000 7fefa23f8640 1 -- 172.21.15.125:0/3478531349 wait complete. 2023-10-30T12:28:50.816 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-30T12:28:50.816 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:28:50.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.951+0000 7f5b3fc9c640 1 Processor -- start 2023-10-30T12:28:50.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.951+0000 7f5b3fc9c640 1 -- start start 2023-10-30T12:28:50.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3fc9c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b380cb7e0 0x7f5b38156680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3fc9c640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b380c68b0 con 0x7f5b380c2570 2023-10-30T12:28:50.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3fc9c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b38150d00 con 0x7f5b38150920 2023-10-30T12:28:50.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3fc9c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b380c2e90 con 0x7f5b380cb7e0 2023-10-30T12:28:50.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3d237640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5b380c2570 0x7f5b380cb0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54048/0 (socket says 172.21.15.125:54048) 2023-10-30T12:28:50.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3d237640 1 -- 172.21.15.125:0/1048273307 learned_addr learned my addr 172.21.15.125:0/1048273307 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:50.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3e239640 1 --2- 172.21.15.125:0/1048273307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b380cb7e0 0x7f5b38156680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989986423 0 0) 0x7f5b380c68b0 con 0x7f5b380c2570 2023-10-30T12:28:50.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b24003530 con 0x7f5b380c2570 2023-10-30T12:28:50.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577303219 0 0) 0x7f5b38150d00 con 0x7f5b38150920 2023-10-30T12:28:50.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b380c68b0 con 0x7f5b38150920 2023-10-30T12:28:50.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2186735411 0 0) 0x7f5b24003530 con 0x7f5b380c2570 2023-10-30T12:28:50.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b380cb7e0 msgr2=0x7f5b38156680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:50.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 --2- 172.21.15.125:0/1048273307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b380cb7e0 0x7f5b38156680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 >> v1:172.21.15.136:6789/0 conn(0x7f5b38150920 legacy=0x7f5b3806d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b380c6b00 con 0x7f5b380c2570 2023-10-30T12:28:50.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f5b28001680 con 0x7f5b380c2570 2023-10-30T12:28:50.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 >> v1:172.21.15.125:6789/0 conn(0x7f5b380c2570 legacy=0x7f5b380cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 --2- 172.21.15.125:0/1048273307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b24003b40 0x7f5b24003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 --2- 172.21.15.125:0/1048273307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b380cb7e0 0x7f5b240047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 --2- 172.21.15.125:0/1048273307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b24004d10 0x7f5b24005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b380c2e90 con 0x7f5b24004d10 2023-10-30T12:28:50.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2400dc50 con 0x7f5b24003b40 2023-10-30T12:28:50.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2400de10 con 0x7f5b380cb7e0 2023-10-30T12:28:50.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3e239640 1 --2- 172.21.15.125:0/1048273307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b24004d10 0x7f5b24005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3d237640 1 --2- 172.21.15.125:0/1048273307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b380cb7e0 0x7f5b240047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3da38640 1 --2- 172.21.15.125:0/1048273307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b24003b40 0x7f5b24003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3e239640 1 -- 172.21.15.125:0/1048273307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b380cb7e0 msgr2=0x7f5b240047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:50.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3e239640 1 --2- 172.21.15.125:0/1048273307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b380cb7e0 0x7f5b240047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3e239640 1 -- 172.21.15.125:0/1048273307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b24003b40 msgr2=0x7f5b24003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:50.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3e239640 1 --2- 172.21.15.125:0/1048273307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b24003b40 0x7f5b24003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3e239640 1 -- 172.21.15.125:0/1048273307 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b24003530 con 0x7f5b24004d10 2023-10-30T12:28:50.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.955+0000 7f5b3da38640 1 --2- 172.21.15.125:0/1048273307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b24003b40 0x7f5b24003f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:50.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.959+0000 7f5b3e239640 1 --2- 172.21.15.125:0/1048273307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b24004d10 0x7f5b24005190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5b380c8940 tx=0x7f5b3001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d99c488de059cad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:50.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.959+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b300a5c10 con 0x7f5b24004d10 2023-10-30T12:28:50.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.959+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b3000f040 con 0x7f5b24004d10 2023-10-30T12:28:50.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.959+0000 7f5b3ca36640 1 -- 172.21.15.125:0/1048273307 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b300ad440 con 0x7f5b24004d10 2023-10-30T12:28:50.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.959+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/1048273307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b24004d10 msgr2=0x7f5b24005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.959+0000 7f5b3fc9c640 1 --2- 172.21.15.125:0/1048273307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b24004d10 0x7f5b24005190 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5b380c8940 tx=0x7f5b3001cc00 comp rx=0 tx=0).stop 2023-10-30T12:28:50.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.959+0000 7f5b3e239640 1 -- 172.21.15.125:0/1048273307 reap_dead start 2023-10-30T12:28:50.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.959+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/1048273307 shutdown_connections 2023-10-30T12:28:50.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.959+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/1048273307 >> 172.21.15.125:0/1048273307 conn(0x7f5b3806dfd0 msgr2=0x7f5b3806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:50.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.959+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/1048273307 shutdown_connections 2023-10-30T12:28:50.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.959+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/1048273307 wait complete. 2023-10-30T12:28:50.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.959+0000 7f5b3fc9c640 1 Processor -- start 2023-10-30T12:28:50.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3fc9c640 1 -- start start 2023-10-30T12:28:50.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3fc9c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b24004d10 0x7f5b381534d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3fc9c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b38152c30 0x7f5b3814f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3fc9c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b381eeaa0 0x7f5b381eef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3fc9c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b38152380 con 0x7f5b381eeaa0 2023-10-30T12:28:50.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3fc9c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b381f2c00 con 0x7f5b38152c30 2023-10-30T12:28:50.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3fc9c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b381ef460 con 0x7f5b24004d10 2023-10-30T12:28:50.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3e239640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b381eeaa0 0x7f5b381eef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3e239640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b381eeaa0 0x7f5b381eef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:32816/0 (socket says 172.21.15.125:32816) 2023-10-30T12:28:50.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3e239640 1 -- 172.21.15.125:0/3376981252 learned_addr learned my addr 172.21.15.125:0/3376981252 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:50.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3da38640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b24004d10 0x7f5b381534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3d237640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b38152c30 0x7f5b3814f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3e239640 1 -- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b24004d10 msgr2=0x7f5b381534d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3e239640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b24004d10 0x7f5b381534d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3e239640 1 -- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b38152c30 msgr2=0x7f5b3814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:50.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3e239640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b38152c30 0x7f5b3814f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:50.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3e239640 1 -- 172.21.15.125:0/3376981252 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b381f2010 con 0x7f5b381eeaa0 2023-10-30T12:28:50.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3da38640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b24004d10 0x7f5b381534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:50.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3e239640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b381eeaa0 0x7f5b381eef20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5b30002240 tx=0x7f5b30009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:50.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b2e7fc640 1 -- 172.21.15.125:0/3376981252 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b300ad650 con 0x7f5b381eeaa0 2023-10-30T12:28:50.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b2e7fc640 1 -- 172.21.15.125:0/3376981252 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b300ad7b0 con 0x7f5b381eeaa0 2023-10-30T12:28:50.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b2e7fc640 1 -- 172.21.15.125:0/3376981252 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b300adb40 con 0x7f5b381eeaa0 2023-10-30T12:28:50.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/3376981252 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b3806bef0 con 0x7f5b381eeaa0 2023-10-30T12:28:50.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/3376981252 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b38153090 con 0x7f5b381eeaa0 2023-10-30T12:28:50.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.963+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/3376981252 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b3804e170 con 0x7f5b381eeaa0 2023-10-30T12:28:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.967+0000 7f5b2e7fc640 1 -- 172.21.15.125:0/3376981252 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5b300b4040 con 0x7f5b381eeaa0 2023-10-30T12:28:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.967+0000 7f5b2e7fc640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5b1c0664a0 0x7f5b1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.967+0000 7f5b2e7fc640 1 -- 172.21.15.125:0/3376981252 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5b3011e640 con 0x7f5b381eeaa0 2023-10-30T12:28:50.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.967+0000 7f5b3da38640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5b1c0664a0 0x7f5b1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.967+0000 7f5b3da38640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5b1c0664a0 0x7f5b1c068960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5b34000f70 tx=0x7f5b34009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:50.971+0000 7f5b2e7fc640 1 -- 172.21.15.125:0/3376981252 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5b300eb990 con 0x7f5b381eeaa0 2023-10-30T12:28:51.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.119+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/3376981252 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b380c70c0 con 0x7f5b1c0664a0 2023-10-30T12:28:51.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.139+0000 7f5b2e7fc640 1 -- 172.21.15.125:0/3376981252 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264553 (secure 0 0 0) 0x7f5b34005220 con 0x7f5b1c0664a0 2023-10-30T12:28:51.145 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T12:28:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.143+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5b1c0664a0 msgr2=0x7f5b1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.143+0000 7f5b3fc9c640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5b1c0664a0 0x7f5b1c068960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5b34000f70 tx=0x7f5b34009040 comp rx=0 tx=0).stop 2023-10-30T12:28:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.143+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/3376981252 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b381eeaa0 msgr2=0x7f5b381eef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.143+0000 7f5b3fc9c640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b381eeaa0 0x7f5b381eef20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5b30002240 tx=0x7f5b30009750 comp rx=0 tx=0).stop 2023-10-30T12:28:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.143+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/3376981252 shutdown_connections 2023-10-30T12:28:51.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.143+0000 7f5b3fc9c640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5b1c0664a0 0x7f5b1c068960 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.143+0000 7f5b3fc9c640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b381eeaa0 0x7f5b381eef20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.143+0000 7f5b3fc9c640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b38152c30 0x7f5b3814f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.143+0000 7f5b3fc9c640 1 --2- 172.21.15.125:0/3376981252 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b24004d10 0x7f5b381534d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.143+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/3376981252 >> 172.21.15.125:0/3376981252 conn(0x7f5b3806dfd0 msgr2=0x7f5b38071b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:51.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.143+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/3376981252 shutdown_connections 2023-10-30T12:28:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.143+0000 7f5b3fc9c640 1 -- 172.21.15.125:0/3376981252 wait complete. 2023-10-30T12:28:51.165 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-30T12:28:50.961435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":175,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223156,"kb_used_data":2832,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751564,"statfs":{"total":773094113280,"available":772865601536,"internally_reserved":0,"allocated":2899968,"data_stored":1775080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":225496633},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.932371"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104854+0000","last_change":"2023-10-30T12:28:44.957488+0000","last_active":"2023-10-30T12:28:46.104854+0000","last_peered":"2023-10-30T12:28:46.104854+0000","last_clean":"2023-10-30T12:28:46.104854+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T12:28:46.104854+0000","last_undegraded":"2023-10-30T12:28:46.104854+0000","last_fullsized":"2023-10-30T12:28:46.104854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:30.240483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.984223+0000","last_change":"2023-10-30T12:28:44.978898+0000","last_active":"2023-10-30T12:28:44.984223+0000","last_peered":"2023-10-30T12:28:44.984223+0000","last_clean":"2023-10-30T12:28:44.984223+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T12:28:44.984223+0000","last_undegraded":"2023-10-30T12:28:44.984223+0000","last_fullsized":"2023-10-30T12:28:44.984223+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:55:54.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.985470+0000","last_change":"2023-10-30T12:28:44.980722+0000","last_active":"2023-10-30T12:28:44.985470+0000","last_peered":"2023-10-30T12:28:44.985470+0000","last_clean":"2023-10-30T12:28:44.985470+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T12:28:44.985470+0000","last_undegraded":"2023-10-30T12:28:44.985470+0000","last_fullsized":"2023-10-30T12:28:44.985470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:00:50.143485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103812+0000","last_change":"2023-10-30T12:28:44.965061+0000","last_active":"2023-10-30T12:28:46.103812+0000","last_peered":"2023-10-30T12:28:46.103812+0000","last_clean":"2023-10-30T12:28:46.103812+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T12:28:46.103812+0000","last_undegraded":"2023-10-30T12:28:46.103812+0000","last_fullsized":"2023-10-30T12:28:46.103812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:23:43.749064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104285+0000","last_change":"2023-10-30T12:28:44.976985+0000","last_active":"2023-10-30T12:28:46.104285+0000","last_peered":"2023-10-30T12:28:46.104285+0000","last_clean":"2023-10-30T12:28:46.104285+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T12:28:46.104285+0000","last_undegraded":"2023-10-30T12:28:46.104285+0000","last_fullsized":"2023-10-30T12:28:46.104285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:08.758140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103030+0000","last_change":"2023-10-30T12:28:44.965453+0000","last_active":"2023-10-30T12:28:46.103030+0000","last_peered":"2023-10-30T12:28:46.103030+0000","last_clean":"2023-10-30T12:28:46.103030+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T12:28:46.103030+0000","last_undegraded":"2023-10-30T12:28:46.103030+0000","last_fullsized":"2023-10-30T12:28:46.103030+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:19.975226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103434+0000","last_change":"2023-10-30T12:28:44.968493+0000","last_active":"2023-10-30T12:28:46.103434+0000","last_peered":"2023-10-30T12:28:46.103434+0000","last_clean":"2023-10-30T12:28:46.103434+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T12:28:46.103434+0000","last_undegraded":"2023-10-30T12:28:46.103434+0000","last_fullsized":"2023-10-30T12:28:46.103434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:58.895635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.983694+0000","last_change":"2023-10-30T12:28:44.972695+0000","last_active":"2023-10-30T12:28:44.983694+0000","last_peered":"2023-10-30T12:28:44.983694+0000","last_clean":"2023-10-30T12:28:44.983694+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T12:28:44.983694+0000","last_undegraded":"2023-10-30T12:28:44.983694+0000","last_fullsized":"2023-10-30T12:28:44.983694+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:52.580801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939624+0000","last_change":"2023-10-30T12:28:44.946156+0000","last_active":"2023-10-30T12:28:45.939624+0000","last_peered":"2023-10-30T12:28:45.939624+0000","last_clean":"2023-10-30T12:28:45.939624+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T12:28:45.939624+0000","last_undegraded":"2023-10-30T12:28:45.939624+0000","last_fullsized":"2023-10-30T12:28:45.939624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:42:36.408944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104638+0000","last_change":"2023-10-30T12:28:44.957225+0000","last_active":"2023-10-30T12:28:46.104638+0000","last_peered":"2023-10-30T12:28:46.104638+0000","last_clean":"2023-10-30T12:28:46.104638+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T12:28:46.104638+0000","last_undegraded":"2023-10-30T12:28:46.104638+0000","last_fullsized":"2023-10-30T12:28:46.104638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:55:20.194553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104526+0000","last_change":"2023-10-30T12:28:44.956025+0000","last_active":"2023-10-30T12:28:46.104526+0000","last_peered":"2023-10-30T12:28:46.104526+0000","last_clean":"2023-10-30T12:28:46.104526+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T12:28:46.104526+0000","last_undegraded":"2023-10-30T12:28:46.104526+0000","last_fullsized":"2023-10-30T12:28:46.104526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:12.113264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104464+0000","last_change":"2023-10-30T12:28:44.956832+0000","last_active":"2023-10-30T12:28:46.104464+0000","last_peered":"2023-10-30T12:28:46.104464+0000","last_clean":"2023-10-30T12:28:46.104464+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T12:28:46.104464+0000","last_undegraded":"2023-10-30T12:28:46.104464+0000","last_fullsized":"2023-10-30T12:28:46.104464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:36.932024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945824+0000","last_change":"2023-10-30T12:28:44.962961+0000","last_active":"2023-10-30T12:28:45.945824+0000","last_peered":"2023-10-30T12:28:45.945824+0000","last_clean":"2023-10-30T12:28:45.945824+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T12:28:45.945824+0000","last_undegraded":"2023-10-30T12:28:45.945824+0000","last_fullsized":"2023-10-30T12:28:45.945824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:13.200996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945413+0000","last_change":"2023-10-30T12:28:44.968005+0000","last_active":"2023-10-30T12:28:45.945413+0000","last_peered":"2023-10-30T12:28:45.945413+0000","last_clean":"2023-10-30T12:28:45.945413+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T12:28:45.945413+0000","last_undegraded":"2023-10-30T12:28:45.945413+0000","last_fullsized":"2023-10-30T12:28:45.945413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:53:25.168553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.984122+0000","last_change":"2023-10-30T12:28:44.977064+0000","last_active":"2023-10-30T12:28:44.984122+0000","last_peered":"2023-10-30T12:28:44.984122+0000","last_clean":"2023-10-30T12:28:44.984122+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T12:28:44.984122+0000","last_undegraded":"2023-10-30T12:28:44.984122+0000","last_fullsized":"2023-10-30T12:28:44.984122+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:29.883626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102782+0000","last_change":"2023-10-30T12:28:44.950169+0000","last_active":"2023-10-30T12:28:46.102782+0000","last_peered":"2023-10-30T12:28:46.102782+0000","last_clean":"2023-10-30T12:28:46.102782+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T12:28:46.102782+0000","last_undegraded":"2023-10-30T12:28:46.102782+0000","last_fullsized":"2023-10-30T12:28:46.102782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:35.297525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103374+0000","last_change":"2023-10-30T12:28:44.960467+0000","last_active":"2023-10-30T12:28:46.103374+0000","last_peered":"2023-10-30T12:28:46.103374+0000","last_clean":"2023-10-30T12:28:46.103374+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T12:28:46.103374+0000","last_undegraded":"2023-10-30T12:28:46.103374+0000","last_fullsized":"2023-10-30T12:28:46.103374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:33.495256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.972976+0000","last_change":"2023-10-30T12:28:44.968948+0000","last_active":"2023-10-30T12:28:44.972976+0000","last_peered":"2023-10-30T12:28:44.972976+0000","last_clean":"2023-10-30T12:28:44.972976+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T12:28:44.972976+0000","last_undegraded":"2023-10-30T12:28:44.972976+0000","last_fullsized":"2023-10-30T12:28:44.972976+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:37:20.018128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103539+0000","last_change":"2023-10-30T12:28:44.966743+0000","last_active":"2023-10-30T12:28:46.103539+0000","last_peered":"2023-10-30T12:28:46.103539+0000","last_clean":"2023-10-30T12:28:46.103539+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T12:28:46.103539+0000","last_undegraded":"2023-10-30T12:28:46.103539+0000","last_fullsized":"2023-10-30T12:28:46.103539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:30.077107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104499+0000","last_change":"2023-10-30T12:28:44.955323+0000","last_active":"2023-10-30T12:28:46.104499+0000","last_peered":"2023-10-30T12:28:46.104499+0000","last_clean":"2023-10-30T12:28:46.104499+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T12:28:46.104499+0000","last_undegraded":"2023-10-30T12:28:46.104499+0000","last_fullsized":"2023-10-30T12:28:46.104499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:59:22.424571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945425+0000","last_change":"2023-10-30T12:28:44.946048+0000","last_active":"2023-10-30T12:28:45.945425+0000","last_peered":"2023-10-30T12:28:45.945425+0000","last_clean":"2023-10-30T12:28:45.945425+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T12:28:45.945425+0000","last_undegraded":"2023-10-30T12:28:45.945425+0000","last_fullsized":"2023-10-30T12:28:45.945425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:20:21.145943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939444+0000","last_change":"2023-10-30T12:28:44.954167+0000","last_active":"2023-10-30T12:28:45.939444+0000","last_peered":"2023-10-30T12:28:45.939444+0000","last_clean":"2023-10-30T12:28:45.939444+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T12:28:45.939444+0000","last_undegraded":"2023-10-30T12:28:45.939444+0000","last_fullsized":"2023-10-30T12:28:45.939444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:08.457457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.985640+0000","last_change":"2023-10-30T12:28:44.982724+0000","last_active":"2023-10-30T12:28:44.985640+0000","last_peered":"2023-10-30T12:28:44.985640+0000","last_clean":"2023-10-30T12:28:44.985640+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T12:28:44.985640+0000","last_undegraded":"2023-10-30T12:28:44.985640+0000","last_fullsized":"2023-10-30T12:28:44.985640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:56:36.579208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103197+0000","last_change":"2023-10-30T12:28:44.981034+0000","last_active":"2023-10-30T12:28:46.103197+0000","last_peered":"2023-10-30T12:28:46.103197+0000","last_clean":"2023-10-30T12:28:46.103197+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T12:28:46.103197+0000","last_undegraded":"2023-10-30T12:28:46.103197+0000","last_fullsized":"2023-10-30T12:28:46.103197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:39:25.263228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.983827+0000","last_change":"2023-10-30T12:28:44.975010+0000","last_active":"2023-10-30T12:28:44.983827+0000","last_peered":"2023-10-30T12:28:44.983827+0000","last_clean":"2023-10-30T12:28:44.983827+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T12:28:44.983827+0000","last_undegraded":"2023-10-30T12:28:44.983827+0000","last_fullsized":"2023-10-30T12:28:44.983827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:51:26.995049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104243+0000","last_change":"2023-10-30T12:28:44.951099+0000","last_active":"2023-10-30T12:28:46.104243+0000","last_peered":"2023-10-30T12:28:46.104243+0000","last_clean":"2023-10-30T12:28:46.104243+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T12:28:46.104243+0000","last_undegraded":"2023-10-30T12:28:46.104243+0000","last_fullsized":"2023-10-30T12:28:46.104243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:21:27.902671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946065+0000","last_change":"2023-10-30T12:28:44.959915+0000","last_active":"2023-10-30T12:28:45.946065+0000","last_peered":"2023-10-30T12:28:45.946065+0000","last_clean":"2023-10-30T12:28:45.946065+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T12:28:45.946065+0000","last_undegraded":"2023-10-30T12:28:45.946065+0000","last_fullsized":"2023-10-30T12:28:45.946065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:57:49.711309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946417+0000","last_change":"2023-10-30T12:28:44.972677+0000","last_active":"2023-10-30T12:28:45.946417+0000","last_peered":"2023-10-30T12:28:45.946417+0000","last_clean":"2023-10-30T12:28:45.946417+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T12:28:45.946417+0000","last_undegraded":"2023-10-30T12:28:45.946417+0000","last_fullsized":"2023-10-30T12:28:45.946417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:12.072355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.983977+0000","last_change":"2023-10-30T12:28:44.975973+0000","last_active":"2023-10-30T12:28:44.983977+0000","last_peered":"2023-10-30T12:28:44.983977+0000","last_clean":"2023-10-30T12:28:44.983977+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T12:28:44.983977+0000","last_undegraded":"2023-10-30T12:28:44.983977+0000","last_fullsized":"2023-10-30T12:28:44.983977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:51.250163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103531+0000","last_change":"2023-10-30T12:28:44.980814+0000","last_active":"2023-10-30T12:28:46.103531+0000","last_peered":"2023-10-30T12:28:46.103531+0000","last_clean":"2023-10-30T12:28:46.103531+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T12:28:46.103531+0000","last_undegraded":"2023-10-30T12:28:46.103531+0000","last_fullsized":"2023-10-30T12:28:46.103531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:51:22.453883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.970010+0000","last_change":"2023-10-30T12:28:44.965026+0000","last_active":"2023-10-30T12:28:44.970010+0000","last_peered":"2023-10-30T12:28:44.970010+0000","last_clean":"2023-10-30T12:28:44.970010+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T12:28:44.970010+0000","last_undegraded":"2023-10-30T12:28:44.970010+0000","last_fullsized":"2023-10-30T12:28:44.970010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:19:42.514601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102782+0000","last_change":"2023-10-30T12:28:44.973401+0000","last_active":"2023-10-30T12:28:46.102782+0000","last_peered":"2023-10-30T12:28:46.102782+0000","last_clean":"2023-10-30T12:28:46.102782+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T12:28:46.102782+0000","last_undegraded":"2023-10-30T12:28:46.102782+0000","last_fullsized":"2023-10-30T12:28:46.102782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:34.198424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939578+0000","last_change":"2023-10-30T12:28:44.971817+0000","last_active":"2023-10-30T12:28:45.939578+0000","last_peered":"2023-10-30T12:28:45.939578+0000","last_clean":"2023-10-30T12:28:45.939578+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T12:28:45.939578+0000","last_undegraded":"2023-10-30T12:28:45.939578+0000","last_fullsized":"2023-10-30T12:28:45.939578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:45:35.205305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.956825+0000","last_change":"2023-10-30T12:28:44.074842+0000","last_active":"2023-10-30T12:28:44.956825+0000","last_peered":"2023-10-30T12:28:44.956825+0000","last_clean":"2023-10-30T12:28:44.956825+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T12:28:44.956825+0000","last_undegraded":"2023-10-30T12:28:44.956825+0000","last_fullsized":"2023-10-30T12:28:44.956825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:01.428785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945765+0000","last_change":"2023-10-30T12:28:44.958560+0000","last_active":"2023-10-30T12:28:45.945765+0000","last_peered":"2023-10-30T12:28:45.945765+0000","last_clean":"2023-10-30T12:28:45.945765+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T12:28:45.945765+0000","last_undegraded":"2023-10-30T12:28:45.945765+0000","last_fullsized":"2023-10-30T12:28:45.945765+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:45.564493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103724+0000","last_change":"2023-10-30T12:28:44.060621+0000","last_active":"2023-10-30T12:28:46.103724+0000","last_peered":"2023-10-30T12:28:46.103724+0000","last_clean":"2023-10-30T12:28:46.103724+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T12:28:46.103724+0000","last_undegraded":"2023-10-30T12:28:46.103724+0000","last_fullsized":"2023-10-30T12:28:46.103724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:53.741238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945157+0000","last_change":"2023-10-30T12:28:44.060171+0000","last_active":"2023-10-30T12:28:45.945157+0000","last_peered":"2023-10-30T12:28:45.945157+0000","last_clean":"2023-10-30T12:28:45.945157+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T12:28:45.945157+0000","last_undegraded":"2023-10-30T12:28:45.945157+0000","last_fullsized":"2023-10-30T12:28:45.945157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:18.181418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.984310+0000","last_change":"2023-10-30T12:28:44.978631+0000","last_active":"2023-10-30T12:28:44.984310+0000","last_peered":"2023-10-30T12:28:44.984310+0000","last_clean":"2023-10-30T12:28:44.984310+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T12:28:44.984310+0000","last_undegraded":"2023-10-30T12:28:44.984310+0000","last_fullsized":"2023-10-30T12:28:44.984310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:02:55.297052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103283+0000","last_change":"2023-10-30T12:28:44.058798+0000","last_active":"2023-10-30T12:28:46.103283+0000","last_peered":"2023-10-30T12:28:46.103283+0000","last_clean":"2023-10-30T12:28:46.103283+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T12:28:46.103283+0000","last_undegraded":"2023-10-30T12:28:46.103283+0000","last_fullsized":"2023-10-30T12:28:46.103283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:08.643080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945025+0000","last_change":"2023-10-30T12:28:44.949118+0000","last_active":"2023-10-30T12:28:45.945025+0000","last_peered":"2023-10-30T12:28:45.945025+0000","last_clean":"2023-10-30T12:28:45.945025+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T12:28:45.945025+0000","last_undegraded":"2023-10-30T12:28:45.945025+0000","last_fullsized":"2023-10-30T12:28:45.945025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:11.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939606+0000","last_change":"2023-10-30T12:28:44.063439+0000","last_active":"2023-10-30T12:28:45.939606+0000","last_peered":"2023-10-30T12:28:45.939606+0000","last_clean":"2023-10-30T12:28:45.939606+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T12:28:45.939606+0000","last_undegraded":"2023-10-30T12:28:45.939606+0000","last_fullsized":"2023-10-30T12:28:45.939606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:08.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104375+0000","last_change":"2023-10-30T12:28:44.951319+0000","last_active":"2023-10-30T12:28:46.104375+0000","last_peered":"2023-10-30T12:28:46.104375+0000","last_clean":"2023-10-30T12:28:46.104375+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T12:28:46.104375+0000","last_undegraded":"2023-10-30T12:28:46.104375+0000","last_fullsized":"2023-10-30T12:28:46.104375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:20.108360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104752+0000","last_change":"2023-10-30T12:28:44.052641+0000","last_active":"2023-10-30T12:28:46.104752+0000","last_peered":"2023-10-30T12:28:46.104752+0000","last_clean":"2023-10-30T12:28:46.104752+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T12:28:46.104752+0000","last_undegraded":"2023-10-30T12:28:46.104752+0000","last_fullsized":"2023-10-30T12:28:46.104752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.985551+0000","last_change":"2023-10-30T12:28:44.981614+0000","last_active":"2023-10-30T12:28:44.985551+0000","last_peered":"2023-10-30T12:28:44.985551+0000","last_clean":"2023-10-30T12:28:44.985551+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T12:28:44.985551+0000","last_undegraded":"2023-10-30T12:28:44.985551+0000","last_fullsized":"2023-10-30T12:28:44.985551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:00.812658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.962329+0000","last_change":"2023-10-30T12:28:44.075783+0000","last_active":"2023-10-30T12:28:44.962329+0000","last_peered":"2023-10-30T12:28:44.962329+0000","last_clean":"2023-10-30T12:28:44.962329+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T12:28:44.962329+0000","last_undegraded":"2023-10-30T12:28:44.962329+0000","last_fullsized":"2023-10-30T12:28:44.962329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:01.094929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103142+0000","last_change":"2023-10-30T12:28:44.950899+0000","last_active":"2023-10-30T12:28:46.103142+0000","last_peered":"2023-10-30T12:28:46.103142+0000","last_clean":"2023-10-30T12:28:46.103142+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T12:28:46.103142+0000","last_undegraded":"2023-10-30T12:28:46.103142+0000","last_fullsized":"2023-10-30T12:28:46.103142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:34.005863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103869+0000","last_change":"2023-10-30T12:28:44.063367+0000","last_active":"2023-10-30T12:28:46.103869+0000","last_peered":"2023-10-30T12:28:46.103869+0000","last_clean":"2023-10-30T12:28:46.103869+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T12:28:46.103869+0000","last_undegraded":"2023-10-30T12:28:46.103869+0000","last_fullsized":"2023-10-30T12:28:46.103869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:45.585121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104587+0000","last_change":"2023-10-30T12:28:44.972056+0000","last_active":"2023-10-30T12:28:46.104587+0000","last_peered":"2023-10-30T12:28:46.104587+0000","last_clean":"2023-10-30T12:28:46.104587+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T12:28:46.104587+0000","last_undegraded":"2023-10-30T12:28:46.104587+0000","last_fullsized":"2023-10-30T12:28:46.104587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:06:08.345856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.959808+0000","last_change":"2023-10-30T12:28:44.065545+0000","last_active":"2023-10-30T12:28:44.959808+0000","last_peered":"2023-10-30T12:28:44.959808+0000","last_clean":"2023-10-30T12:28:44.959808+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T12:28:44.959808+0000","last_undegraded":"2023-10-30T12:28:44.959808+0000","last_fullsized":"2023-10-30T12:28:44.959808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:09.900189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945563+0000","last_change":"2023-10-30T12:28:44.961237+0000","last_active":"2023-10-30T12:28:45.945563+0000","last_peered":"2023-10-30T12:28:45.945563+0000","last_clean":"2023-10-30T12:28:45.945563+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T12:28:45.945563+0000","last_undegraded":"2023-10-30T12:28:45.945563+0000","last_fullsized":"2023-10-30T12:28:45.945563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:17.160724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103797+0000","last_change":"2023-10-30T12:28:44.061922+0000","last_active":"2023-10-30T12:28:46.103797+0000","last_peered":"2023-10-30T12:28:46.103797+0000","last_clean":"2023-10-30T12:28:46.103797+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T12:28:46.103797+0000","last_undegraded":"2023-10-30T12:28:46.103797+0000","last_fullsized":"2023-10-30T12:28:46.103797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:34.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103258+0000","last_change":"2023-10-30T12:28:44.958245+0000","last_active":"2023-10-30T12:28:46.103258+0000","last_peered":"2023-10-30T12:28:46.103258+0000","last_clean":"2023-10-30T12:28:46.103258+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T12:28:46.103258+0000","last_undegraded":"2023-10-30T12:28:46.103258+0000","last_fullsized":"2023-10-30T12:28:46.103258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:42:10.840490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103980+0000","last_change":"2023-10-30T12:28:44.059784+0000","last_active":"2023-10-30T12:28:46.103980+0000","last_peered":"2023-10-30T12:28:46.103980+0000","last_clean":"2023-10-30T12:28:46.103980+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T12:28:46.103980+0000","last_undegraded":"2023-10-30T12:28:46.103980+0000","last_fullsized":"2023-10-30T12:28:46.103980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:11.468221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939230+0000","last_change":"2023-10-30T12:28:44.980813+0000","last_active":"2023-10-30T12:28:45.939230+0000","last_peered":"2023-10-30T12:28:45.939230+0000","last_clean":"2023-10-30T12:28:45.939230+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T12:28:45.939230+0000","last_undegraded":"2023-10-30T12:28:45.939230+0000","last_fullsized":"2023-10-30T12:28:45.939230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:10.858970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944535+0000","last_change":"2023-10-30T12:28:44.065885+0000","last_active":"2023-10-30T12:28:45.944535+0000","last_peered":"2023-10-30T12:28:45.944535+0000","last_clean":"2023-10-30T12:28:45.944535+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T12:28:45.944535+0000","last_undegraded":"2023-10-30T12:28:45.944535+0000","last_fullsized":"2023-10-30T12:28:45.944535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:26.313595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103080+0000","last_change":"2023-10-30T12:28:44.952458+0000","last_active":"2023-10-30T12:28:46.103080+0000","last_peered":"2023-10-30T12:28:46.103080+0000","last_clean":"2023-10-30T12:28:46.103080+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T12:28:46.103080+0000","last_undegraded":"2023-10-30T12:28:46.103080+0000","last_fullsized":"2023-10-30T12:28:46.103080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:48:19.752492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104898+0000","last_change":"2023-10-30T12:28:44.080011+0000","last_active":"2023-10-30T12:28:46.104898+0000","last_peered":"2023-10-30T12:28:46.104898+0000","last_clean":"2023-10-30T12:28:46.104898+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T12:28:46.104898+0000","last_undegraded":"2023-10-30T12:28:46.104898+0000","last_fullsized":"2023-10-30T12:28:46.104898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104494+0000","last_change":"2023-10-30T12:28:44.961296+0000","last_active":"2023-10-30T12:28:46.104494+0000","last_peered":"2023-10-30T12:28:46.104494+0000","last_clean":"2023-10-30T12:28:46.104494+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T12:28:46.104494+0000","last_undegraded":"2023-10-30T12:28:46.104494+0000","last_fullsized":"2023-10-30T12:28:46.104494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:56.540392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946744+0000","last_change":"2023-10-30T12:28:44.076352+0000","last_active":"2023-10-30T12:28:45.946744+0000","last_peered":"2023-10-30T12:28:45.946744+0000","last_clean":"2023-10-30T12:28:45.946744+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T12:28:45.946744+0000","last_undegraded":"2023-10-30T12:28:45.946744+0000","last_fullsized":"2023-10-30T12:28:45.946744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104918+0000","last_change":"2023-10-30T12:28:44.956918+0000","last_active":"2023-10-30T12:28:46.104918+0000","last_peered":"2023-10-30T12:28:46.104918+0000","last_clean":"2023-10-30T12:28:46.104918+0000","last_became_active":"2023-10-30T12:28:44.956397+0000","last_became_peered":"2023-10-30T12:28:44.956397+0000","last_unstale":"2023-10-30T12:28:46.104918+0000","last_undegraded":"2023-10-30T12:28:46.104918+0000","last_fullsized":"2023-10-30T12:28:46.104918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:26.252196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.957053+0000","last_change":"2023-10-30T12:28:44.075684+0000","last_active":"2023-10-30T12:28:44.957053+0000","last_peered":"2023-10-30T12:28:44.957053+0000","last_clean":"2023-10-30T12:28:44.957053+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T12:28:44.957053+0000","last_undegraded":"2023-10-30T12:28:44.957053+0000","last_fullsized":"2023-10-30T12:28:44.957053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:07.375795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104588+0000","last_change":"2023-10-30T12:28:44.956047+0000","last_active":"2023-10-30T12:28:46.104588+0000","last_peered":"2023-10-30T12:28:46.104588+0000","last_clean":"2023-10-30T12:28:46.104588+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T12:28:46.104588+0000","last_undegraded":"2023-10-30T12:28:46.104588+0000","last_fullsized":"2023-10-30T12:28:46.104588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:49.924947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946632+0000","last_change":"2023-10-30T12:28:44.060686+0000","last_active":"2023-10-30T12:28:45.946632+0000","last_peered":"2023-10-30T12:28:45.946632+0000","last_clean":"2023-10-30T12:28:45.946632+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T12:28:45.946632+0000","last_undegraded":"2023-10-30T12:28:45.946632+0000","last_fullsized":"2023-10-30T12:28:45.946632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:23.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103562+0000","last_change":"2023-10-30T12:28:44.946758+0000","last_active":"2023-10-30T12:28:46.103562+0000","last_peered":"2023-10-30T12:28:46.103562+0000","last_clean":"2023-10-30T12:28:46.103562+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T12:28:46.103562+0000","last_undegraded":"2023-10-30T12:28:46.103562+0000","last_fullsized":"2023-10-30T12:28:46.103562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:20.520148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.932509+0000","last_change":"2023-10-30T12:28:33.433824+0000","last_active":"2023-10-30T12:28:44.932509+0000","last_peered":"2023-10-30T12:28:44.932509+0000","last_clean":"2023-10-30T12:28:44.932509+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T12:28:44.932509+0000","last_undegraded":"2023-10-30T12:28:44.932509+0000","last_fullsized":"2023-10-30T12:28:44.932509+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_clean_scrub_stamp":"2023-10-30T12:28:31.408255+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:17.624179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103427+0000","last_change":"2023-10-30T12:28:44.059051+0000","last_active":"2023-10-30T12:28:46.103427+0000","last_peered":"2023-10-30T12:28:46.103427+0000","last_clean":"2023-10-30T12:28:46.103427+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T12:28:46.103427+0000","last_undegraded":"2023-10-30T12:28:46.103427+0000","last_fullsized":"2023-10-30T12:28:46.103427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:54.810636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945062+0000","last_change":"2023-10-30T12:28:44.965183+0000","last_active":"2023-10-30T12:28:45.945062+0000","last_peered":"2023-10-30T12:28:45.945062+0000","last_clean":"2023-10-30T12:28:45.945062+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T12:28:45.945062+0000","last_undegraded":"2023-10-30T12:28:45.945062+0000","last_fullsized":"2023-10-30T12:28:45.945062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:57.655687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103361+0000","last_change":"2023-10-30T12:28:44.080378+0000","last_active":"2023-10-30T12:28:46.103361+0000","last_peered":"2023-10-30T12:28:46.103361+0000","last_clean":"2023-10-30T12:28:46.103361+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T12:28:46.103361+0000","last_undegraded":"2023-10-30T12:28:46.103361+0000","last_fullsized":"2023-10-30T12:28:46.103361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:22.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944927+0000","last_change":"2023-10-30T12:28:44.961659+0000","last_active":"2023-10-30T12:28:45.944927+0000","last_peered":"2023-10-30T12:28:45.944927+0000","last_clean":"2023-10-30T12:28:45.944927+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T12:28:45.944927+0000","last_undegraded":"2023-10-30T12:28:45.944927+0000","last_fullsized":"2023-10-30T12:28:45.944927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:34:00.000084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.954829+0000","last_change":"2023-10-30T12:28:44.082480+0000","last_active":"2023-10-30T12:28:44.954829+0000","last_peered":"2023-10-30T12:28:44.954829+0000","last_clean":"2023-10-30T12:28:44.954829+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T12:28:44.954829+0000","last_undegraded":"2023-10-30T12:28:44.954829+0000","last_fullsized":"2023-10-30T12:28:44.954829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:20.126782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.983963+0000","last_change":"2023-10-30T12:28:44.973906+0000","last_active":"2023-10-30T12:28:44.983963+0000","last_peered":"2023-10-30T12:28:44.983963+0000","last_clean":"2023-10-30T12:28:44.983963+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T12:28:44.983963+0000","last_undegraded":"2023-10-30T12:28:44.983963+0000","last_fullsized":"2023-10-30T12:28:44.983963+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:50.843265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103873+0000","last_change":"2023-10-30T12:28:44.059806+0000","last_active":"2023-10-30T12:28:46.103873+0000","last_peered":"2023-10-30T12:28:46.103873+0000","last_clean":"2023-10-30T12:28:46.103873+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T12:28:46.103873+0000","last_undegraded":"2023-10-30T12:28:46.103873+0000","last_fullsized":"2023-10-30T12:28:46.103873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:50.939946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104302+0000","last_change":"2023-10-30T12:28:44.955423+0000","last_active":"2023-10-30T12:28:46.104302+0000","last_peered":"2023-10-30T12:28:46.104302+0000","last_clean":"2023-10-30T12:28:46.104302+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T12:28:46.104302+0000","last_undegraded":"2023-10-30T12:28:46.104302+0000","last_fullsized":"2023-10-30T12:28:46.104302+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:30.054048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.956825+0000","last_change":"2023-10-30T12:28:44.074970+0000","last_active":"2023-10-30T12:28:44.956825+0000","last_peered":"2023-10-30T12:28:44.956825+0000","last_clean":"2023-10-30T12:28:44.956825+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T12:28:44.956825+0000","last_undegraded":"2023-10-30T12:28:44.956825+0000","last_fullsized":"2023-10-30T12:28:44.956825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:33.562703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103885+0000","last_change":"2023-10-30T12:28:44.973482+0000","last_active":"2023-10-30T12:28:46.103885+0000","last_peered":"2023-10-30T12:28:46.103885+0000","last_clean":"2023-10-30T12:28:46.103885+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T12:28:46.103885+0000","last_undegraded":"2023-10-30T12:28:46.103885+0000","last_fullsized":"2023-10-30T12:28:46.103885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:25.393161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103579+0000","last_change":"2023-10-30T12:28:44.063256+0000","last_active":"2023-10-30T12:28:46.103579+0000","last_peered":"2023-10-30T12:28:46.103579+0000","last_clean":"2023-10-30T12:28:46.103579+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T12:28:46.103579+0000","last_undegraded":"2023-10-30T12:28:46.103579+0000","last_fullsized":"2023-10-30T12:28:46.103579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:01:36.144966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946945+0000","last_change":"2023-10-30T12:28:44.975443+0000","last_active":"2023-10-30T12:28:45.946945+0000","last_peered":"2023-10-30T12:28:45.946945+0000","last_clean":"2023-10-30T12:28:45.946945+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T12:28:45.946945+0000","last_undegraded":"2023-10-30T12:28:45.946945+0000","last_fullsized":"2023-10-30T12:28:45.946945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:23.765723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946972+0000","last_change":"2023-10-30T12:28:44.055374+0000","last_active":"2023-10-30T12:28:45.946972+0000","last_peered":"2023-10-30T12:28:45.946972+0000","last_clean":"2023-10-30T12:28:45.946972+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T12:28:45.946972+0000","last_undegraded":"2023-10-30T12:28:45.946972+0000","last_fullsized":"2023-10-30T12:28:45.946972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946991+0000","last_change":"2023-10-30T12:28:44.974359+0000","last_active":"2023-10-30T12:28:45.946991+0000","last_peered":"2023-10-30T12:28:45.946991+0000","last_clean":"2023-10-30T12:28:45.946991+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T12:28:45.946991+0000","last_undegraded":"2023-10-30T12:28:45.946991+0000","last_fullsized":"2023-10-30T12:28:45.946991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:48.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103361+0000","last_change":"2023-10-30T12:28:44.058663+0000","last_active":"2023-10-30T12:28:46.103361+0000","last_peered":"2023-10-30T12:28:46.103361+0000","last_clean":"2023-10-30T12:28:46.103361+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T12:28:46.103361+0000","last_undegraded":"2023-10-30T12:28:46.103361+0000","last_fullsized":"2023-10-30T12:28:46.103361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:42:22.279653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946815+0000","last_change":"2023-10-30T12:28:44.957471+0000","last_active":"2023-10-30T12:28:45.946815+0000","last_peered":"2023-10-30T12:28:45.946815+0000","last_clean":"2023-10-30T12:28:45.946815+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T12:28:45.946815+0000","last_undegraded":"2023-10-30T12:28:45.946815+0000","last_fullsized":"2023-10-30T12:28:45.946815+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:27:38.669452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102847+0000","last_change":"2023-10-30T12:28:44.054338+0000","last_active":"2023-10-30T12:28:46.102847+0000","last_peered":"2023-10-30T12:28:46.102847+0000","last_clean":"2023-10-30T12:28:46.102847+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T12:28:46.102847+0000","last_undegraded":"2023-10-30T12:28:46.102847+0000","last_fullsized":"2023-10-30T12:28:46.102847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:55:35.370652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103200+0000","last_change":"2023-10-30T12:28:44.965759+0000","last_active":"2023-10-30T12:28:46.103200+0000","last_peered":"2023-10-30T12:28:46.103200+0000","last_clean":"2023-10-30T12:28:46.103200+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T12:28:46.103200+0000","last_undegraded":"2023-10-30T12:28:46.103200+0000","last_fullsized":"2023-10-30T12:28:46.103200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:13:52.965500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944989+0000","last_change":"2023-10-30T12:28:44.064606+0000","last_active":"2023-10-30T12:28:45.944989+0000","last_peered":"2023-10-30T12:28:45.944989+0000","last_clean":"2023-10-30T12:28:45.944989+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T12:28:45.944989+0000","last_undegraded":"2023-10-30T12:28:45.944989+0000","last_fullsized":"2023-10-30T12:28:45.944989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944678+0000","last_change":"2023-10-30T12:28:44.958947+0000","last_active":"2023-10-30T12:28:45.944678+0000","last_peered":"2023-10-30T12:28:45.944678+0000","last_clean":"2023-10-30T12:28:45.944678+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T12:28:45.944678+0000","last_undegraded":"2023-10-30T12:28:45.944678+0000","last_fullsized":"2023-10-30T12:28:45.944678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:01:16.329994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945603+0000","last_change":"2023-10-30T12:28:44.072754+0000","last_active":"2023-10-30T12:28:45.945603+0000","last_peered":"2023-10-30T12:28:45.945603+0000","last_clean":"2023-10-30T12:28:45.945603+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T12:28:45.945603+0000","last_undegraded":"2023-10-30T12:28:45.945603+0000","last_fullsized":"2023-10-30T12:28:45.945603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:51:55.865069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.982845+0000","last_change":"2023-10-30T12:28:44.978376+0000","last_active":"2023-10-30T12:28:44.982845+0000","last_peered":"2023-10-30T12:28:44.982845+0000","last_clean":"2023-10-30T12:28:44.982845+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T12:28:44.982845+0000","last_undegraded":"2023-10-30T12:28:44.982845+0000","last_fullsized":"2023-10-30T12:28:44.982845+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:34.418948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103349+0000","last_change":"2023-10-30T12:28:44.058801+0000","last_active":"2023-10-30T12:28:46.103349+0000","last_peered":"2023-10-30T12:28:46.103349+0000","last_clean":"2023-10-30T12:28:46.103349+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T12:28:46.103349+0000","last_undegraded":"2023-10-30T12:28:46.103349+0000","last_fullsized":"2023-10-30T12:28:46.103349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:11:24.706813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103647+0000","last_change":"2023-10-30T12:28:44.943292+0000","last_active":"2023-10-30T12:28:46.103647+0000","last_peered":"2023-10-30T12:28:46.103647+0000","last_clean":"2023-10-30T12:28:46.103647+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T12:28:46.103647+0000","last_undegraded":"2023-10-30T12:28:46.103647+0000","last_fullsized":"2023-10-30T12:28:46.103647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:37.538525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103472+0000","last_change":"2023-10-30T12:28:44.958607+0000","last_active":"2023-10-30T12:28:46.103472+0000","last_peered":"2023-10-30T12:28:46.103472+0000","last_clean":"2023-10-30T12:28:46.103472+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T12:28:46.103472+0000","last_undegraded":"2023-10-30T12:28:46.103472+0000","last_fullsized":"2023-10-30T12:28:46.103472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:28:55.169198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104582+0000","last_change":"2023-10-30T12:28:44.054652+0000","last_active":"2023-10-30T12:28:46.104582+0000","last_peered":"2023-10-30T12:28:46.104582+0000","last_clean":"2023-10-30T12:28:46.104582+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T12:28:46.104582+0000","last_undegraded":"2023-10-30T12:28:46.104582+0000","last_fullsized":"2023-10-30T12:28:46.104582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:05.023374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939383+0000","last_change":"2023-10-30T12:28:44.958246+0000","last_active":"2023-10-30T12:28:45.939383+0000","last_peered":"2023-10-30T12:28:45.939383+0000","last_clean":"2023-10-30T12:28:45.939383+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T12:28:45.939383+0000","last_undegraded":"2023-10-30T12:28:45.939383+0000","last_fullsized":"2023-10-30T12:28:45.939383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:36.947398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104619+0000","last_change":"2023-10-30T12:28:44.052633+0000","last_active":"2023-10-30T12:28:46.104619+0000","last_peered":"2023-10-30T12:28:46.104619+0000","last_clean":"2023-10-30T12:28:46.104619+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T12:28:46.104619+0000","last_undegraded":"2023-10-30T12:28:46.104619+0000","last_fullsized":"2023-10-30T12:28:46.104619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.983794+0000","last_change":"2023-10-30T12:28:44.974457+0000","last_active":"2023-10-30T12:28:44.983794+0000","last_peered":"2023-10-30T12:28:44.983794+0000","last_clean":"2023-10-30T12:28:44.983794+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T12:28:44.983794+0000","last_undegraded":"2023-10-30T12:28:44.983794+0000","last_fullsized":"2023-10-30T12:28:44.983794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:30:35.231889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.957613+0000","last_change":"2023-10-30T12:28:44.075493+0000","last_active":"2023-10-30T12:28:44.957613+0000","last_peered":"2023-10-30T12:28:44.957613+0000","last_clean":"2023-10-30T12:28:44.957613+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T12:28:44.957613+0000","last_undegraded":"2023-10-30T12:28:44.957613+0000","last_fullsized":"2023-10-30T12:28:44.957613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:00:11.718669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.983743+0000","last_change":"2023-10-30T12:28:44.974424+0000","last_active":"2023-10-30T12:28:44.983743+0000","last_peered":"2023-10-30T12:28:44.983743+0000","last_clean":"2023-10-30T12:28:44.983743+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T12:28:44.983743+0000","last_undegraded":"2023-10-30T12:28:44.983743+0000","last_fullsized":"2023-10-30T12:28:44.983743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:56:01.484533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.947065+0000","last_change":"2023-10-30T12:28:44.065843+0000","last_active":"2023-10-30T12:28:45.947065+0000","last_peered":"2023-10-30T12:28:45.947065+0000","last_clean":"2023-10-30T12:28:45.947065+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T12:28:45.947065+0000","last_undegraded":"2023-10-30T12:28:45.947065+0000","last_fullsized":"2023-10-30T12:28:45.947065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:32.405199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476743,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]}]},{"osd":2,"up_from":16,"seq":68719476743,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]}]},{"osd":1,"up_from":16,"seq":68719476741,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":666236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]}]},{"osd":0,"up_from":16,"seq":68719476742,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]}]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":75868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]}]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":798720,"data_stored":659936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.222}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:28:51.170 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:28:51.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.343+0000 7f590759e640 1 Processor -- start 2023-10-30T12:28:51.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.343+0000 7f590759e640 1 -- start start 2023-10-30T12:28:51.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.343+0000 7f590759e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f59080c6840 0x7f590815a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.347+0000 7f590759e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59080c6c20 0x7f5908156590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.347+0000 7f590759e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59080c3b30 con 0x7f59080c3640 2023-10-30T12:28:51.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.347+0000 7f590759e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59080c2aa0 con 0x7f59080c6840 2023-10-30T12:28:51.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.347+0000 7f590759e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59080c3df0 con 0x7f59080c6c20 2023-10-30T12:28:51.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.347+0000 7f5906d9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59080c6c20 0x7f5908156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.347+0000 7f590659c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f59080c6840 0x7f590815a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.347+0000 7f58fdd9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f59080c3640 0x7f5908153730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:36886/0 (socket says 172.21.15.125:36886) 2023-10-30T12:28:51.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.347+0000 7f58fdd9b640 1 -- 172.21.15.125:0/900022902 learned_addr learned my addr 172.21.15.125:0/900022902 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:51.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.347+0000 7f590659c640 1 -- 172.21.15.125:0/900022902 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59080c6c20 msgr2=0x7f5908156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:51.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.347+0000 7f590659c640 1 --2- 172.21.15.125:0/900022902 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59080c6c20 0x7f5908156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.347+0000 7f590659c640 1 -- 172.21.15.125:0/900022902 >> v1:172.21.15.136:6789/0 conn(0x7f59080c3640 legacy=0x7f5908153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.347+0000 7f590659c640 1 -- 172.21.15.125:0/900022902 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5908159ba0 con 0x7f59080c6840 2023-10-30T12:28:51.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.347+0000 7f5906d9d640 1 --2- 172.21.15.125:0/900022902 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59080c6c20 0x7f5908156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:51.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.351+0000 7f590659c640 1 --2- 172.21.15.125:0/900022902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59080c6840 0x7f590815a680 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f58f80097a0 tx=0x7f58f801c460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95e814e397c8d992 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:51.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.351+0000 7f5905d9b640 1 -- 172.21.15.125:0/900022902 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f80ab600 con 0x7f59080c6840 2023-10-30T12:28:51.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.351+0000 7f5905d9b640 1 -- 172.21.15.125:0/900022902 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f80a7050 con 0x7f59080c6840 2023-10-30T12:28:51.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.351+0000 7f5905d9b640 1 -- 172.21.15.125:0/900022902 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f80ab980 con 0x7f59080c6840 2023-10-30T12:28:51.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.351+0000 7f590759e640 1 -- 172.21.15.125:0/900022902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59080c6840 msgr2=0x7f590815a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.351+0000 7f590759e640 1 --2- 172.21.15.125:0/900022902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59080c6840 0x7f590815a680 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f58f80097a0 tx=0x7f58f801c460 comp rx=0 tx=0).stop 2023-10-30T12:28:51.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.351+0000 7f590759e640 1 -- 172.21.15.125:0/900022902 shutdown_connections 2023-10-30T12:28:51.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.351+0000 7f590759e640 1 --2- 172.21.15.125:0/900022902 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59080c6c20 0x7f5908156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.351+0000 7f590759e640 1 --2- 172.21.15.125:0/900022902 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59080c6840 0x7f590815a680 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.351+0000 7f590759e640 1 -- 172.21.15.125:0/900022902 >> 172.21.15.125:0/900022902 conn(0x7f59080c75d0 msgr2=0x7f59080c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:51.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.351+0000 7f590759e640 1 -- 172.21.15.125:0/900022902 shutdown_connections 2023-10-30T12:28:51.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.351+0000 7f590759e640 1 -- 172.21.15.125:0/900022902 wait complete. 2023-10-30T12:28:51.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.351+0000 7f590759e640 1 Processor -- start 2023-10-30T12:28:51.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.351+0000 7f590759e640 1 -- start start 2023-10-30T12:28:51.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f590759e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59080c3640 0x7f59080c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f590759e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59080c6840 0x7f59080ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f590759e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080c6c20 0x7f590814c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f590759e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59081420c0 con 0x7f59080c6c20 2023-10-30T12:28:51.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f590759e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5908070740 con 0x7f59080c3640 2023-10-30T12:28:51.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f590759e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59080c5760 con 0x7f59080c6840 2023-10-30T12:28:51.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f5906d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080c6c20 0x7f590814c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f5906d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080c6c20 0x7f590814c0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:32832/0 (socket says 172.21.15.125:32832) 2023-10-30T12:28:51.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f5906d9d640 1 -- 172.21.15.125:0/676429192 learned_addr learned my addr 172.21.15.125:0/676429192 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:51.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f590659c640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59080c3640 0x7f59080c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f58fdd9b640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59080c6840 0x7f59080ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f5906d9d640 1 -- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59080c6840 msgr2=0x7f59080ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f5906d9d640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59080c6840 0x7f59080ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f5906d9d640 1 -- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59080c3640 msgr2=0x7f59080c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f5906d9d640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59080c3640 0x7f59080c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f5906d9d640 1 -- 172.21.15.125:0/676429192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5900009a70 con 0x7f59080c6c20 2023-10-30T12:28:51.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f58fdd9b640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59080c6840 0x7f59080ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:51.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f5906d9d640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080c6c20 0x7f590814c0f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f590000efc0 tx=0x7f590000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:51.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f58ff7fe640 1 -- 172.21.15.125:0/676429192 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900009cd0 con 0x7f59080c6c20 2023-10-30T12:28:51.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f58ff7fe640 1 -- 172.21.15.125:0/676429192 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590000f040 con 0x7f59080c6c20 2023-10-30T12:28:51.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f58ff7fe640 1 -- 172.21.15.125:0/676429192 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59000039a0 con 0x7f59080c6c20 2023-10-30T12:28:51.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f590759e640 1 -- 172.21.15.125:0/676429192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f80093b0 con 0x7f59080c6c20 2023-10-30T12:28:51.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f590759e640 1 -- 172.21.15.125:0/676429192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59081f4f30 con 0x7f59080c6c20 2023-10-30T12:28:51.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f590759e640 1 -- 172.21.15.125:0/676429192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f590804e170 con 0x7f59080c6c20 2023-10-30T12:28:51.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.355+0000 7f58ff7fe640 1 -- 172.21.15.125:0/676429192 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5900014480 con 0x7f59080c6c20 2023-10-30T12:28:51.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.359+0000 7f58ff7fe640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58e80664d0 0x7f58e8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.359+0000 7f58ff7fe640 1 -- 172.21.15.125:0/676429192 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5900084ac0 con 0x7f59080c6c20 2023-10-30T12:28:51.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.359+0000 7f590659c640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58e80664d0 0x7f58e8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.359+0000 7f590659c640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58e80664d0 0x7f58e8068990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f58f80ad000 tx=0x7f58f80a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:51.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.363+0000 7f58ff7fe640 1 -- 172.21.15.125:0/676429192 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5900051ea0 con 0x7f59080c6c20 2023-10-30T12:28:51.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.507+0000 7f590759e640 1 -- 172.21.15.125:0/676429192 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f59080c1f50 con 0x7f58e80664d0 2023-10-30T12:28:51.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.527+0000 7f58ff7fe640 1 -- 172.21.15.125:0/676429192 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264553 (secure 0 0 0) 0x7f58f80c14a0 con 0x7f58e80664d0 2023-10-30T12:28:51.531 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:51.533 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T12:28:51.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.531+0000 7f590759e640 1 -- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58e80664d0 msgr2=0x7f58e8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.531+0000 7f590759e640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58e80664d0 0x7f58e8068990 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f58f80ad000 tx=0x7f58f80a6000 comp rx=0 tx=0).stop 2023-10-30T12:28:51.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.531+0000 7f590759e640 1 -- 172.21.15.125:0/676429192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080c6c20 msgr2=0x7f590814c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.531+0000 7f590759e640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080c6c20 0x7f590814c0f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f590000efc0 tx=0x7f590000c490 comp rx=0 tx=0).stop 2023-10-30T12:28:51.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.531+0000 7f590759e640 1 -- 172.21.15.125:0/676429192 shutdown_connections 2023-10-30T12:28:51.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.531+0000 7f590759e640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58e80664d0 0x7f58e8068990 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.531+0000 7f590759e640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59080c6c20 0x7f590814c0f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.531+0000 7f590759e640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59080c6840 0x7f59080ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.531+0000 7f590759e640 1 --2- 172.21.15.125:0/676429192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59080c3640 0x7f59080c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.531+0000 7f590759e640 1 -- 172.21.15.125:0/676429192 >> 172.21.15.125:0/676429192 conn(0x7f59080c75d0 msgr2=0x7f590806d430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:51.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.531+0000 7f590759e640 1 -- 172.21.15.125:0/676429192 shutdown_connections 2023-10-30T12:28:51.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.531+0000 7f590759e640 1 -- 172.21.15.125:0/676429192 wait complete. 2023-10-30T12:28:51.552 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-30T12:28:50.961435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":175,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223156,"kb_used_data":2832,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751564,"statfs":{"total":773094113280,"available":772865601536,"internally_reserved":0,"allocated":2899968,"data_stored":1775080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":225496633},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":62,"apply_latency_ms":62,"commit_latency_ns":62000000,"apply_latency_ns":62000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.932371"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104854+0000","last_change":"2023-10-30T12:28:44.957488+0000","last_active":"2023-10-30T12:28:46.104854+0000","last_peered":"2023-10-30T12:28:46.104854+0000","last_clean":"2023-10-30T12:28:46.104854+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T12:28:46.104854+0000","last_undegraded":"2023-10-30T12:28:46.104854+0000","last_fullsized":"2023-10-30T12:28:46.104854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:30.240483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.984223+0000","last_change":"2023-10-30T12:28:44.978898+0000","last_active":"2023-10-30T12:28:44.984223+0000","last_peered":"2023-10-30T12:28:44.984223+0000","last_clean":"2023-10-30T12:28:44.984223+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T12:28:44.984223+0000","last_undegraded":"2023-10-30T12:28:44.984223+0000","last_fullsized":"2023-10-30T12:28:44.984223+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:55:54.286491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.985470+0000","last_change":"2023-10-30T12:28:44.980722+0000","last_active":"2023-10-30T12:28:44.985470+0000","last_peered":"2023-10-30T12:28:44.985470+0000","last_clean":"2023-10-30T12:28:44.985470+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T12:28:44.985470+0000","last_undegraded":"2023-10-30T12:28:44.985470+0000","last_fullsized":"2023-10-30T12:28:44.985470+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:00:50.143485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103812+0000","last_change":"2023-10-30T12:28:44.965061+0000","last_active":"2023-10-30T12:28:46.103812+0000","last_peered":"2023-10-30T12:28:46.103812+0000","last_clean":"2023-10-30T12:28:46.103812+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T12:28:46.103812+0000","last_undegraded":"2023-10-30T12:28:46.103812+0000","last_fullsized":"2023-10-30T12:28:46.103812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:23:43.749064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104285+0000","last_change":"2023-10-30T12:28:44.976985+0000","last_active":"2023-10-30T12:28:46.104285+0000","last_peered":"2023-10-30T12:28:46.104285+0000","last_clean":"2023-10-30T12:28:46.104285+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T12:28:46.104285+0000","last_undegraded":"2023-10-30T12:28:46.104285+0000","last_fullsized":"2023-10-30T12:28:46.104285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:08.758140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103030+0000","last_change":"2023-10-30T12:28:44.965453+0000","last_active":"2023-10-30T12:28:46.103030+0000","last_peered":"2023-10-30T12:28:46.103030+0000","last_clean":"2023-10-30T12:28:46.103030+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T12:28:46.103030+0000","last_undegraded":"2023-10-30T12:28:46.103030+0000","last_fullsized":"2023-10-30T12:28:46.103030+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:19.975226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103434+0000","last_change":"2023-10-30T12:28:44.968493+0000","last_active":"2023-10-30T12:28:46.103434+0000","last_peered":"2023-10-30T12:28:46.103434+0000","last_clean":"2023-10-30T12:28:46.103434+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T12:28:46.103434+0000","last_undegraded":"2023-10-30T12:28:46.103434+0000","last_fullsized":"2023-10-30T12:28:46.103434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:58.895635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.983694+0000","last_change":"2023-10-30T12:28:44.972695+0000","last_active":"2023-10-30T12:28:44.983694+0000","last_peered":"2023-10-30T12:28:44.983694+0000","last_clean":"2023-10-30T12:28:44.983694+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T12:28:44.983694+0000","last_undegraded":"2023-10-30T12:28:44.983694+0000","last_fullsized":"2023-10-30T12:28:44.983694+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:52.580801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939624+0000","last_change":"2023-10-30T12:28:44.946156+0000","last_active":"2023-10-30T12:28:45.939624+0000","last_peered":"2023-10-30T12:28:45.939624+0000","last_clean":"2023-10-30T12:28:45.939624+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T12:28:45.939624+0000","last_undegraded":"2023-10-30T12:28:45.939624+0000","last_fullsized":"2023-10-30T12:28:45.939624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:42:36.408944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104638+0000","last_change":"2023-10-30T12:28:44.957225+0000","last_active":"2023-10-30T12:28:46.104638+0000","last_peered":"2023-10-30T12:28:46.104638+0000","last_clean":"2023-10-30T12:28:46.104638+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T12:28:46.104638+0000","last_undegraded":"2023-10-30T12:28:46.104638+0000","last_fullsized":"2023-10-30T12:28:46.104638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:55:20.194553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104526+0000","last_change":"2023-10-30T12:28:44.956025+0000","last_active":"2023-10-30T12:28:46.104526+0000","last_peered":"2023-10-30T12:28:46.104526+0000","last_clean":"2023-10-30T12:28:46.104526+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T12:28:46.104526+0000","last_undegraded":"2023-10-30T12:28:46.104526+0000","last_fullsized":"2023-10-30T12:28:46.104526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:12.113264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104464+0000","last_change":"2023-10-30T12:28:44.956832+0000","last_active":"2023-10-30T12:28:46.104464+0000","last_peered":"2023-10-30T12:28:46.104464+0000","last_clean":"2023-10-30T12:28:46.104464+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T12:28:46.104464+0000","last_undegraded":"2023-10-30T12:28:46.104464+0000","last_fullsized":"2023-10-30T12:28:46.104464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:36.932024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945824+0000","last_change":"2023-10-30T12:28:44.962961+0000","last_active":"2023-10-30T12:28:45.945824+0000","last_peered":"2023-10-30T12:28:45.945824+0000","last_clean":"2023-10-30T12:28:45.945824+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T12:28:45.945824+0000","last_undegraded":"2023-10-30T12:28:45.945824+0000","last_fullsized":"2023-10-30T12:28:45.945824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:13.200996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945413+0000","last_change":"2023-10-30T12:28:44.968005+0000","last_active":"2023-10-30T12:28:45.945413+0000","last_peered":"2023-10-30T12:28:45.945413+0000","last_clean":"2023-10-30T12:28:45.945413+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T12:28:45.945413+0000","last_undegraded":"2023-10-30T12:28:45.945413+0000","last_fullsized":"2023-10-30T12:28:45.945413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:53:25.168553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.984122+0000","last_change":"2023-10-30T12:28:44.977064+0000","last_active":"2023-10-30T12:28:44.984122+0000","last_peered":"2023-10-30T12:28:44.984122+0000","last_clean":"2023-10-30T12:28:44.984122+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T12:28:44.984122+0000","last_undegraded":"2023-10-30T12:28:44.984122+0000","last_fullsized":"2023-10-30T12:28:44.984122+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:29.883626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102782+0000","last_change":"2023-10-30T12:28:44.950169+0000","last_active":"2023-10-30T12:28:46.102782+0000","last_peered":"2023-10-30T12:28:46.102782+0000","last_clean":"2023-10-30T12:28:46.102782+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T12:28:46.102782+0000","last_undegraded":"2023-10-30T12:28:46.102782+0000","last_fullsized":"2023-10-30T12:28:46.102782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:35.297525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103374+0000","last_change":"2023-10-30T12:28:44.960467+0000","last_active":"2023-10-30T12:28:46.103374+0000","last_peered":"2023-10-30T12:28:46.103374+0000","last_clean":"2023-10-30T12:28:46.103374+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T12:28:46.103374+0000","last_undegraded":"2023-10-30T12:28:46.103374+0000","last_fullsized":"2023-10-30T12:28:46.103374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:33.495256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.972976+0000","last_change":"2023-10-30T12:28:44.968948+0000","last_active":"2023-10-30T12:28:44.972976+0000","last_peered":"2023-10-30T12:28:44.972976+0000","last_clean":"2023-10-30T12:28:44.972976+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T12:28:44.972976+0000","last_undegraded":"2023-10-30T12:28:44.972976+0000","last_fullsized":"2023-10-30T12:28:44.972976+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:37:20.018128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103539+0000","last_change":"2023-10-30T12:28:44.966743+0000","last_active":"2023-10-30T12:28:46.103539+0000","last_peered":"2023-10-30T12:28:46.103539+0000","last_clean":"2023-10-30T12:28:46.103539+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T12:28:46.103539+0000","last_undegraded":"2023-10-30T12:28:46.103539+0000","last_fullsized":"2023-10-30T12:28:46.103539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:30.077107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104499+0000","last_change":"2023-10-30T12:28:44.955323+0000","last_active":"2023-10-30T12:28:46.104499+0000","last_peered":"2023-10-30T12:28:46.104499+0000","last_clean":"2023-10-30T12:28:46.104499+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T12:28:46.104499+0000","last_undegraded":"2023-10-30T12:28:46.104499+0000","last_fullsized":"2023-10-30T12:28:46.104499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:59:22.424571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945425+0000","last_change":"2023-10-30T12:28:44.946048+0000","last_active":"2023-10-30T12:28:45.945425+0000","last_peered":"2023-10-30T12:28:45.945425+0000","last_clean":"2023-10-30T12:28:45.945425+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T12:28:45.945425+0000","last_undegraded":"2023-10-30T12:28:45.945425+0000","last_fullsized":"2023-10-30T12:28:45.945425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:20:21.145943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939444+0000","last_change":"2023-10-30T12:28:44.954167+0000","last_active":"2023-10-30T12:28:45.939444+0000","last_peered":"2023-10-30T12:28:45.939444+0000","last_clean":"2023-10-30T12:28:45.939444+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T12:28:45.939444+0000","last_undegraded":"2023-10-30T12:28:45.939444+0000","last_fullsized":"2023-10-30T12:28:45.939444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:08.457457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.985640+0000","last_change":"2023-10-30T12:28:44.982724+0000","last_active":"2023-10-30T12:28:44.985640+0000","last_peered":"2023-10-30T12:28:44.985640+0000","last_clean":"2023-10-30T12:28:44.985640+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T12:28:44.985640+0000","last_undegraded":"2023-10-30T12:28:44.985640+0000","last_fullsized":"2023-10-30T12:28:44.985640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:56:36.579208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103197+0000","last_change":"2023-10-30T12:28:44.981034+0000","last_active":"2023-10-30T12:28:46.103197+0000","last_peered":"2023-10-30T12:28:46.103197+0000","last_clean":"2023-10-30T12:28:46.103197+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T12:28:46.103197+0000","last_undegraded":"2023-10-30T12:28:46.103197+0000","last_fullsized":"2023-10-30T12:28:46.103197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:39:25.263228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.983827+0000","last_change":"2023-10-30T12:28:44.975010+0000","last_active":"2023-10-30T12:28:44.983827+0000","last_peered":"2023-10-30T12:28:44.983827+0000","last_clean":"2023-10-30T12:28:44.983827+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T12:28:44.983827+0000","last_undegraded":"2023-10-30T12:28:44.983827+0000","last_fullsized":"2023-10-30T12:28:44.983827+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:51:26.995049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104243+0000","last_change":"2023-10-30T12:28:44.951099+0000","last_active":"2023-10-30T12:28:46.104243+0000","last_peered":"2023-10-30T12:28:46.104243+0000","last_clean":"2023-10-30T12:28:46.104243+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T12:28:46.104243+0000","last_undegraded":"2023-10-30T12:28:46.104243+0000","last_fullsized":"2023-10-30T12:28:46.104243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:21:27.902671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946065+0000","last_change":"2023-10-30T12:28:44.959915+0000","last_active":"2023-10-30T12:28:45.946065+0000","last_peered":"2023-10-30T12:28:45.946065+0000","last_clean":"2023-10-30T12:28:45.946065+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T12:28:45.946065+0000","last_undegraded":"2023-10-30T12:28:45.946065+0000","last_fullsized":"2023-10-30T12:28:45.946065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:57:49.711309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946417+0000","last_change":"2023-10-30T12:28:44.972677+0000","last_active":"2023-10-30T12:28:45.946417+0000","last_peered":"2023-10-30T12:28:45.946417+0000","last_clean":"2023-10-30T12:28:45.946417+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T12:28:45.946417+0000","last_undegraded":"2023-10-30T12:28:45.946417+0000","last_fullsized":"2023-10-30T12:28:45.946417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:12.072355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.983977+0000","last_change":"2023-10-30T12:28:44.975973+0000","last_active":"2023-10-30T12:28:44.983977+0000","last_peered":"2023-10-30T12:28:44.983977+0000","last_clean":"2023-10-30T12:28:44.983977+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T12:28:44.983977+0000","last_undegraded":"2023-10-30T12:28:44.983977+0000","last_fullsized":"2023-10-30T12:28:44.983977+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:51.250163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103531+0000","last_change":"2023-10-30T12:28:44.980814+0000","last_active":"2023-10-30T12:28:46.103531+0000","last_peered":"2023-10-30T12:28:46.103531+0000","last_clean":"2023-10-30T12:28:46.103531+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T12:28:46.103531+0000","last_undegraded":"2023-10-30T12:28:46.103531+0000","last_fullsized":"2023-10-30T12:28:46.103531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:51:22.453883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.970010+0000","last_change":"2023-10-30T12:28:44.965026+0000","last_active":"2023-10-30T12:28:44.970010+0000","last_peered":"2023-10-30T12:28:44.970010+0000","last_clean":"2023-10-30T12:28:44.970010+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T12:28:44.970010+0000","last_undegraded":"2023-10-30T12:28:44.970010+0000","last_fullsized":"2023-10-30T12:28:44.970010+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:19:42.514601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102782+0000","last_change":"2023-10-30T12:28:44.973401+0000","last_active":"2023-10-30T12:28:46.102782+0000","last_peered":"2023-10-30T12:28:46.102782+0000","last_clean":"2023-10-30T12:28:46.102782+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T12:28:46.102782+0000","last_undegraded":"2023-10-30T12:28:46.102782+0000","last_fullsized":"2023-10-30T12:28:46.102782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:34.198424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939578+0000","last_change":"2023-10-30T12:28:44.971817+0000","last_active":"2023-10-30T12:28:45.939578+0000","last_peered":"2023-10-30T12:28:45.939578+0000","last_clean":"2023-10-30T12:28:45.939578+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T12:28:45.939578+0000","last_undegraded":"2023-10-30T12:28:45.939578+0000","last_fullsized":"2023-10-30T12:28:45.939578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:45:35.205305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.956825+0000","last_change":"2023-10-30T12:28:44.074842+0000","last_active":"2023-10-30T12:28:44.956825+0000","last_peered":"2023-10-30T12:28:44.956825+0000","last_clean":"2023-10-30T12:28:44.956825+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T12:28:44.956825+0000","last_undegraded":"2023-10-30T12:28:44.956825+0000","last_fullsized":"2023-10-30T12:28:44.956825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:01.428785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945765+0000","last_change":"2023-10-30T12:28:44.958560+0000","last_active":"2023-10-30T12:28:45.945765+0000","last_peered":"2023-10-30T12:28:45.945765+0000","last_clean":"2023-10-30T12:28:45.945765+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T12:28:45.945765+0000","last_undegraded":"2023-10-30T12:28:45.945765+0000","last_fullsized":"2023-10-30T12:28:45.945765+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:45.564493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103724+0000","last_change":"2023-10-30T12:28:44.060621+0000","last_active":"2023-10-30T12:28:46.103724+0000","last_peered":"2023-10-30T12:28:46.103724+0000","last_clean":"2023-10-30T12:28:46.103724+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T12:28:46.103724+0000","last_undegraded":"2023-10-30T12:28:46.103724+0000","last_fullsized":"2023-10-30T12:28:46.103724+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:53.741238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945157+0000","last_change":"2023-10-30T12:28:44.060171+0000","last_active":"2023-10-30T12:28:45.945157+0000","last_peered":"2023-10-30T12:28:45.945157+0000","last_clean":"2023-10-30T12:28:45.945157+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T12:28:45.945157+0000","last_undegraded":"2023-10-30T12:28:45.945157+0000","last_fullsized":"2023-10-30T12:28:45.945157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:18.181418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.984310+0000","last_change":"2023-10-30T12:28:44.978631+0000","last_active":"2023-10-30T12:28:44.984310+0000","last_peered":"2023-10-30T12:28:44.984310+0000","last_clean":"2023-10-30T12:28:44.984310+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T12:28:44.984310+0000","last_undegraded":"2023-10-30T12:28:44.984310+0000","last_fullsized":"2023-10-30T12:28:44.984310+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:02:55.297052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103283+0000","last_change":"2023-10-30T12:28:44.058798+0000","last_active":"2023-10-30T12:28:46.103283+0000","last_peered":"2023-10-30T12:28:46.103283+0000","last_clean":"2023-10-30T12:28:46.103283+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T12:28:46.103283+0000","last_undegraded":"2023-10-30T12:28:46.103283+0000","last_fullsized":"2023-10-30T12:28:46.103283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:08.643080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945025+0000","last_change":"2023-10-30T12:28:44.949118+0000","last_active":"2023-10-30T12:28:45.945025+0000","last_peered":"2023-10-30T12:28:45.945025+0000","last_clean":"2023-10-30T12:28:45.945025+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T12:28:45.945025+0000","last_undegraded":"2023-10-30T12:28:45.945025+0000","last_fullsized":"2023-10-30T12:28:45.945025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:11.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939606+0000","last_change":"2023-10-30T12:28:44.063439+0000","last_active":"2023-10-30T12:28:45.939606+0000","last_peered":"2023-10-30T12:28:45.939606+0000","last_clean":"2023-10-30T12:28:45.939606+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T12:28:45.939606+0000","last_undegraded":"2023-10-30T12:28:45.939606+0000","last_fullsized":"2023-10-30T12:28:45.939606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:08.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104375+0000","last_change":"2023-10-30T12:28:44.951319+0000","last_active":"2023-10-30T12:28:46.104375+0000","last_peered":"2023-10-30T12:28:46.104375+0000","last_clean":"2023-10-30T12:28:46.104375+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T12:28:46.104375+0000","last_undegraded":"2023-10-30T12:28:46.104375+0000","last_fullsized":"2023-10-30T12:28:46.104375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:20.108360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104752+0000","last_change":"2023-10-30T12:28:44.052641+0000","last_active":"2023-10-30T12:28:46.104752+0000","last_peered":"2023-10-30T12:28:46.104752+0000","last_clean":"2023-10-30T12:28:46.104752+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T12:28:46.104752+0000","last_undegraded":"2023-10-30T12:28:46.104752+0000","last_fullsized":"2023-10-30T12:28:46.104752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.985551+0000","last_change":"2023-10-30T12:28:44.981614+0000","last_active":"2023-10-30T12:28:44.985551+0000","last_peered":"2023-10-30T12:28:44.985551+0000","last_clean":"2023-10-30T12:28:44.985551+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T12:28:44.985551+0000","last_undegraded":"2023-10-30T12:28:44.985551+0000","last_fullsized":"2023-10-30T12:28:44.985551+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:19:00.812658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.962329+0000","last_change":"2023-10-30T12:28:44.075783+0000","last_active":"2023-10-30T12:28:44.962329+0000","last_peered":"2023-10-30T12:28:44.962329+0000","last_clean":"2023-10-30T12:28:44.962329+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T12:28:44.962329+0000","last_undegraded":"2023-10-30T12:28:44.962329+0000","last_fullsized":"2023-10-30T12:28:44.962329+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:01.094929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103142+0000","last_change":"2023-10-30T12:28:44.950899+0000","last_active":"2023-10-30T12:28:46.103142+0000","last_peered":"2023-10-30T12:28:46.103142+0000","last_clean":"2023-10-30T12:28:46.103142+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T12:28:46.103142+0000","last_undegraded":"2023-10-30T12:28:46.103142+0000","last_fullsized":"2023-10-30T12:28:46.103142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:34.005863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103869+0000","last_change":"2023-10-30T12:28:44.063367+0000","last_active":"2023-10-30T12:28:46.103869+0000","last_peered":"2023-10-30T12:28:46.103869+0000","last_clean":"2023-10-30T12:28:46.103869+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T12:28:46.103869+0000","last_undegraded":"2023-10-30T12:28:46.103869+0000","last_fullsized":"2023-10-30T12:28:46.103869+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:45.585121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104587+0000","last_change":"2023-10-30T12:28:44.972056+0000","last_active":"2023-10-30T12:28:46.104587+0000","last_peered":"2023-10-30T12:28:46.104587+0000","last_clean":"2023-10-30T12:28:46.104587+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T12:28:46.104587+0000","last_undegraded":"2023-10-30T12:28:46.104587+0000","last_fullsized":"2023-10-30T12:28:46.104587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:06:08.345856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.959808+0000","last_change":"2023-10-30T12:28:44.065545+0000","last_active":"2023-10-30T12:28:44.959808+0000","last_peered":"2023-10-30T12:28:44.959808+0000","last_clean":"2023-10-30T12:28:44.959808+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T12:28:44.959808+0000","last_undegraded":"2023-10-30T12:28:44.959808+0000","last_fullsized":"2023-10-30T12:28:44.959808+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:09.900189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945563+0000","last_change":"2023-10-30T12:28:44.961237+0000","last_active":"2023-10-30T12:28:45.945563+0000","last_peered":"2023-10-30T12:28:45.945563+0000","last_clean":"2023-10-30T12:28:45.945563+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T12:28:45.945563+0000","last_undegraded":"2023-10-30T12:28:45.945563+0000","last_fullsized":"2023-10-30T12:28:45.945563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:17.160724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103797+0000","last_change":"2023-10-30T12:28:44.061922+0000","last_active":"2023-10-30T12:28:46.103797+0000","last_peered":"2023-10-30T12:28:46.103797+0000","last_clean":"2023-10-30T12:28:46.103797+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T12:28:46.103797+0000","last_undegraded":"2023-10-30T12:28:46.103797+0000","last_fullsized":"2023-10-30T12:28:46.103797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:34.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103258+0000","last_change":"2023-10-30T12:28:44.958245+0000","last_active":"2023-10-30T12:28:46.103258+0000","last_peered":"2023-10-30T12:28:46.103258+0000","last_clean":"2023-10-30T12:28:46.103258+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T12:28:46.103258+0000","last_undegraded":"2023-10-30T12:28:46.103258+0000","last_fullsized":"2023-10-30T12:28:46.103258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:42:10.840490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103980+0000","last_change":"2023-10-30T12:28:44.059784+0000","last_active":"2023-10-30T12:28:46.103980+0000","last_peered":"2023-10-30T12:28:46.103980+0000","last_clean":"2023-10-30T12:28:46.103980+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T12:28:46.103980+0000","last_undegraded":"2023-10-30T12:28:46.103980+0000","last_fullsized":"2023-10-30T12:28:46.103980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:11.468221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939230+0000","last_change":"2023-10-30T12:28:44.980813+0000","last_active":"2023-10-30T12:28:45.939230+0000","last_peered":"2023-10-30T12:28:45.939230+0000","last_clean":"2023-10-30T12:28:45.939230+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T12:28:45.939230+0000","last_undegraded":"2023-10-30T12:28:45.939230+0000","last_fullsized":"2023-10-30T12:28:45.939230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:10.858970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944535+0000","last_change":"2023-10-30T12:28:44.065885+0000","last_active":"2023-10-30T12:28:45.944535+0000","last_peered":"2023-10-30T12:28:45.944535+0000","last_clean":"2023-10-30T12:28:45.944535+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T12:28:45.944535+0000","last_undegraded":"2023-10-30T12:28:45.944535+0000","last_fullsized":"2023-10-30T12:28:45.944535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:26.313595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103080+0000","last_change":"2023-10-30T12:28:44.952458+0000","last_active":"2023-10-30T12:28:46.103080+0000","last_peered":"2023-10-30T12:28:46.103080+0000","last_clean":"2023-10-30T12:28:46.103080+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T12:28:46.103080+0000","last_undegraded":"2023-10-30T12:28:46.103080+0000","last_fullsized":"2023-10-30T12:28:46.103080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:48:19.752492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104898+0000","last_change":"2023-10-30T12:28:44.080011+0000","last_active":"2023-10-30T12:28:46.104898+0000","last_peered":"2023-10-30T12:28:46.104898+0000","last_clean":"2023-10-30T12:28:46.104898+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T12:28:46.104898+0000","last_undegraded":"2023-10-30T12:28:46.104898+0000","last_fullsized":"2023-10-30T12:28:46.104898+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104494+0000","last_change":"2023-10-30T12:28:44.961296+0000","last_active":"2023-10-30T12:28:46.104494+0000","last_peered":"2023-10-30T12:28:46.104494+0000","last_clean":"2023-10-30T12:28:46.104494+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T12:28:46.104494+0000","last_undegraded":"2023-10-30T12:28:46.104494+0000","last_fullsized":"2023-10-30T12:28:46.104494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:56.540392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946744+0000","last_change":"2023-10-30T12:28:44.076352+0000","last_active":"2023-10-30T12:28:45.946744+0000","last_peered":"2023-10-30T12:28:45.946744+0000","last_clean":"2023-10-30T12:28:45.946744+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T12:28:45.946744+0000","last_undegraded":"2023-10-30T12:28:45.946744+0000","last_fullsized":"2023-10-30T12:28:45.946744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104918+0000","last_change":"2023-10-30T12:28:44.956918+0000","last_active":"2023-10-30T12:28:46.104918+0000","last_peered":"2023-10-30T12:28:46.104918+0000","last_clean":"2023-10-30T12:28:46.104918+0000","last_became_active":"2023-10-30T12:28:44.956397+0000","last_became_peered":"2023-10-30T12:28:44.956397+0000","last_unstale":"2023-10-30T12:28:46.104918+0000","last_undegraded":"2023-10-30T12:28:46.104918+0000","last_fullsized":"2023-10-30T12:28:46.104918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:26.252196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.957053+0000","last_change":"2023-10-30T12:28:44.075684+0000","last_active":"2023-10-30T12:28:44.957053+0000","last_peered":"2023-10-30T12:28:44.957053+0000","last_clean":"2023-10-30T12:28:44.957053+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T12:28:44.957053+0000","last_undegraded":"2023-10-30T12:28:44.957053+0000","last_fullsized":"2023-10-30T12:28:44.957053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:07.375795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104588+0000","last_change":"2023-10-30T12:28:44.956047+0000","last_active":"2023-10-30T12:28:46.104588+0000","last_peered":"2023-10-30T12:28:46.104588+0000","last_clean":"2023-10-30T12:28:46.104588+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T12:28:46.104588+0000","last_undegraded":"2023-10-30T12:28:46.104588+0000","last_fullsized":"2023-10-30T12:28:46.104588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:49.924947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946632+0000","last_change":"2023-10-30T12:28:44.060686+0000","last_active":"2023-10-30T12:28:45.946632+0000","last_peered":"2023-10-30T12:28:45.946632+0000","last_clean":"2023-10-30T12:28:45.946632+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T12:28:45.946632+0000","last_undegraded":"2023-10-30T12:28:45.946632+0000","last_fullsized":"2023-10-30T12:28:45.946632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:23.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103562+0000","last_change":"2023-10-30T12:28:44.946758+0000","last_active":"2023-10-30T12:28:46.103562+0000","last_peered":"2023-10-30T12:28:46.103562+0000","last_clean":"2023-10-30T12:28:46.103562+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T12:28:46.103562+0000","last_undegraded":"2023-10-30T12:28:46.103562+0000","last_fullsized":"2023-10-30T12:28:46.103562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:20.520148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.932509+0000","last_change":"2023-10-30T12:28:33.433824+0000","last_active":"2023-10-30T12:28:44.932509+0000","last_peered":"2023-10-30T12:28:44.932509+0000","last_clean":"2023-10-30T12:28:44.932509+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T12:28:44.932509+0000","last_undegraded":"2023-10-30T12:28:44.932509+0000","last_fullsized":"2023-10-30T12:28:44.932509+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_clean_scrub_stamp":"2023-10-30T12:28:31.408255+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:17.624179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103427+0000","last_change":"2023-10-30T12:28:44.059051+0000","last_active":"2023-10-30T12:28:46.103427+0000","last_peered":"2023-10-30T12:28:46.103427+0000","last_clean":"2023-10-30T12:28:46.103427+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T12:28:46.103427+0000","last_undegraded":"2023-10-30T12:28:46.103427+0000","last_fullsized":"2023-10-30T12:28:46.103427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:54.810636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945062+0000","last_change":"2023-10-30T12:28:44.965183+0000","last_active":"2023-10-30T12:28:45.945062+0000","last_peered":"2023-10-30T12:28:45.945062+0000","last_clean":"2023-10-30T12:28:45.945062+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T12:28:45.945062+0000","last_undegraded":"2023-10-30T12:28:45.945062+0000","last_fullsized":"2023-10-30T12:28:45.945062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:57.655687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103361+0000","last_change":"2023-10-30T12:28:44.080378+0000","last_active":"2023-10-30T12:28:46.103361+0000","last_peered":"2023-10-30T12:28:46.103361+0000","last_clean":"2023-10-30T12:28:46.103361+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T12:28:46.103361+0000","last_undegraded":"2023-10-30T12:28:46.103361+0000","last_fullsized":"2023-10-30T12:28:46.103361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:22.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944927+0000","last_change":"2023-10-30T12:28:44.961659+0000","last_active":"2023-10-30T12:28:45.944927+0000","last_peered":"2023-10-30T12:28:45.944927+0000","last_clean":"2023-10-30T12:28:45.944927+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T12:28:45.944927+0000","last_undegraded":"2023-10-30T12:28:45.944927+0000","last_fullsized":"2023-10-30T12:28:45.944927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:34:00.000084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.954829+0000","last_change":"2023-10-30T12:28:44.082480+0000","last_active":"2023-10-30T12:28:44.954829+0000","last_peered":"2023-10-30T12:28:44.954829+0000","last_clean":"2023-10-30T12:28:44.954829+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T12:28:44.954829+0000","last_undegraded":"2023-10-30T12:28:44.954829+0000","last_fullsized":"2023-10-30T12:28:44.954829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:20.126782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.983963+0000","last_change":"2023-10-30T12:28:44.973906+0000","last_active":"2023-10-30T12:28:44.983963+0000","last_peered":"2023-10-30T12:28:44.983963+0000","last_clean":"2023-10-30T12:28:44.983963+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T12:28:44.983963+0000","last_undegraded":"2023-10-30T12:28:44.983963+0000","last_fullsized":"2023-10-30T12:28:44.983963+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:50.843265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103873+0000","last_change":"2023-10-30T12:28:44.059806+0000","last_active":"2023-10-30T12:28:46.103873+0000","last_peered":"2023-10-30T12:28:46.103873+0000","last_clean":"2023-10-30T12:28:46.103873+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T12:28:46.103873+0000","last_undegraded":"2023-10-30T12:28:46.103873+0000","last_fullsized":"2023-10-30T12:28:46.103873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:50.939946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104302+0000","last_change":"2023-10-30T12:28:44.955423+0000","last_active":"2023-10-30T12:28:46.104302+0000","last_peered":"2023-10-30T12:28:46.104302+0000","last_clean":"2023-10-30T12:28:46.104302+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T12:28:46.104302+0000","last_undegraded":"2023-10-30T12:28:46.104302+0000","last_fullsized":"2023-10-30T12:28:46.104302+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:30.054048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.956825+0000","last_change":"2023-10-30T12:28:44.074970+0000","last_active":"2023-10-30T12:28:44.956825+0000","last_peered":"2023-10-30T12:28:44.956825+0000","last_clean":"2023-10-30T12:28:44.956825+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T12:28:44.956825+0000","last_undegraded":"2023-10-30T12:28:44.956825+0000","last_fullsized":"2023-10-30T12:28:44.956825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:33.562703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103885+0000","last_change":"2023-10-30T12:28:44.973482+0000","last_active":"2023-10-30T12:28:46.103885+0000","last_peered":"2023-10-30T12:28:46.103885+0000","last_clean":"2023-10-30T12:28:46.103885+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T12:28:46.103885+0000","last_undegraded":"2023-10-30T12:28:46.103885+0000","last_fullsized":"2023-10-30T12:28:46.103885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:25.393161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103579+0000","last_change":"2023-10-30T12:28:44.063256+0000","last_active":"2023-10-30T12:28:46.103579+0000","last_peered":"2023-10-30T12:28:46.103579+0000","last_clean":"2023-10-30T12:28:46.103579+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T12:28:46.103579+0000","last_undegraded":"2023-10-30T12:28:46.103579+0000","last_fullsized":"2023-10-30T12:28:46.103579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:01:36.144966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946945+0000","last_change":"2023-10-30T12:28:44.975443+0000","last_active":"2023-10-30T12:28:45.946945+0000","last_peered":"2023-10-30T12:28:45.946945+0000","last_clean":"2023-10-30T12:28:45.946945+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T12:28:45.946945+0000","last_undegraded":"2023-10-30T12:28:45.946945+0000","last_fullsized":"2023-10-30T12:28:45.946945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:23.765723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946972+0000","last_change":"2023-10-30T12:28:44.055374+0000","last_active":"2023-10-30T12:28:45.946972+0000","last_peered":"2023-10-30T12:28:45.946972+0000","last_clean":"2023-10-30T12:28:45.946972+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T12:28:45.946972+0000","last_undegraded":"2023-10-30T12:28:45.946972+0000","last_fullsized":"2023-10-30T12:28:45.946972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946991+0000","last_change":"2023-10-30T12:28:44.974359+0000","last_active":"2023-10-30T12:28:45.946991+0000","last_peered":"2023-10-30T12:28:45.946991+0000","last_clean":"2023-10-30T12:28:45.946991+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T12:28:45.946991+0000","last_undegraded":"2023-10-30T12:28:45.946991+0000","last_fullsized":"2023-10-30T12:28:45.946991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:48.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103361+0000","last_change":"2023-10-30T12:28:44.058663+0000","last_active":"2023-10-30T12:28:46.103361+0000","last_peered":"2023-10-30T12:28:46.103361+0000","last_clean":"2023-10-30T12:28:46.103361+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T12:28:46.103361+0000","last_undegraded":"2023-10-30T12:28:46.103361+0000","last_fullsized":"2023-10-30T12:28:46.103361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:42:22.279653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946815+0000","last_change":"2023-10-30T12:28:44.957471+0000","last_active":"2023-10-30T12:28:45.946815+0000","last_peered":"2023-10-30T12:28:45.946815+0000","last_clean":"2023-10-30T12:28:45.946815+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T12:28:45.946815+0000","last_undegraded":"2023-10-30T12:28:45.946815+0000","last_fullsized":"2023-10-30T12:28:45.946815+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:27:38.669452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102847+0000","last_change":"2023-10-30T12:28:44.054338+0000","last_active":"2023-10-30T12:28:46.102847+0000","last_peered":"2023-10-30T12:28:46.102847+0000","last_clean":"2023-10-30T12:28:46.102847+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T12:28:46.102847+0000","last_undegraded":"2023-10-30T12:28:46.102847+0000","last_fullsized":"2023-10-30T12:28:46.102847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:55:35.370652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103200+0000","last_change":"2023-10-30T12:28:44.965759+0000","last_active":"2023-10-30T12:28:46.103200+0000","last_peered":"2023-10-30T12:28:46.103200+0000","last_clean":"2023-10-30T12:28:46.103200+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T12:28:46.103200+0000","last_undegraded":"2023-10-30T12:28:46.103200+0000","last_fullsized":"2023-10-30T12:28:46.103200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:13:52.965500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944989+0000","last_change":"2023-10-30T12:28:44.064606+0000","last_active":"2023-10-30T12:28:45.944989+0000","last_peered":"2023-10-30T12:28:45.944989+0000","last_clean":"2023-10-30T12:28:45.944989+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T12:28:45.944989+0000","last_undegraded":"2023-10-30T12:28:45.944989+0000","last_fullsized":"2023-10-30T12:28:45.944989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944678+0000","last_change":"2023-10-30T12:28:44.958947+0000","last_active":"2023-10-30T12:28:45.944678+0000","last_peered":"2023-10-30T12:28:45.944678+0000","last_clean":"2023-10-30T12:28:45.944678+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T12:28:45.944678+0000","last_undegraded":"2023-10-30T12:28:45.944678+0000","last_fullsized":"2023-10-30T12:28:45.944678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:01:16.329994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945603+0000","last_change":"2023-10-30T12:28:44.072754+0000","last_active":"2023-10-30T12:28:45.945603+0000","last_peered":"2023-10-30T12:28:45.945603+0000","last_clean":"2023-10-30T12:28:45.945603+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T12:28:45.945603+0000","last_undegraded":"2023-10-30T12:28:45.945603+0000","last_fullsized":"2023-10-30T12:28:45.945603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:51:55.865069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.982845+0000","last_change":"2023-10-30T12:28:44.978376+0000","last_active":"2023-10-30T12:28:44.982845+0000","last_peered":"2023-10-30T12:28:44.982845+0000","last_clean":"2023-10-30T12:28:44.982845+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T12:28:44.982845+0000","last_undegraded":"2023-10-30T12:28:44.982845+0000","last_fullsized":"2023-10-30T12:28:44.982845+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:22:34.418948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103349+0000","last_change":"2023-10-30T12:28:44.058801+0000","last_active":"2023-10-30T12:28:46.103349+0000","last_peered":"2023-10-30T12:28:46.103349+0000","last_clean":"2023-10-30T12:28:46.103349+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T12:28:46.103349+0000","last_undegraded":"2023-10-30T12:28:46.103349+0000","last_fullsized":"2023-10-30T12:28:46.103349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:11:24.706813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103647+0000","last_change":"2023-10-30T12:28:44.943292+0000","last_active":"2023-10-30T12:28:46.103647+0000","last_peered":"2023-10-30T12:28:46.103647+0000","last_clean":"2023-10-30T12:28:46.103647+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T12:28:46.103647+0000","last_undegraded":"2023-10-30T12:28:46.103647+0000","last_fullsized":"2023-10-30T12:28:46.103647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:37.538525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103472+0000","last_change":"2023-10-30T12:28:44.958607+0000","last_active":"2023-10-30T12:28:46.103472+0000","last_peered":"2023-10-30T12:28:46.103472+0000","last_clean":"2023-10-30T12:28:46.103472+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T12:28:46.103472+0000","last_undegraded":"2023-10-30T12:28:46.103472+0000","last_fullsized":"2023-10-30T12:28:46.103472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:28:55.169198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104582+0000","last_change":"2023-10-30T12:28:44.054652+0000","last_active":"2023-10-30T12:28:46.104582+0000","last_peered":"2023-10-30T12:28:46.104582+0000","last_clean":"2023-10-30T12:28:46.104582+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T12:28:46.104582+0000","last_undegraded":"2023-10-30T12:28:46.104582+0000","last_fullsized":"2023-10-30T12:28:46.104582+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:05.023374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939383+0000","last_change":"2023-10-30T12:28:44.958246+0000","last_active":"2023-10-30T12:28:45.939383+0000","last_peered":"2023-10-30T12:28:45.939383+0000","last_clean":"2023-10-30T12:28:45.939383+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T12:28:45.939383+0000","last_undegraded":"2023-10-30T12:28:45.939383+0000","last_fullsized":"2023-10-30T12:28:45.939383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:36.947398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104619+0000","last_change":"2023-10-30T12:28:44.052633+0000","last_active":"2023-10-30T12:28:46.104619+0000","last_peered":"2023-10-30T12:28:46.104619+0000","last_clean":"2023-10-30T12:28:46.104619+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T12:28:46.104619+0000","last_undegraded":"2023-10-30T12:28:46.104619+0000","last_fullsized":"2023-10-30T12:28:46.104619+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.983794+0000","last_change":"2023-10-30T12:28:44.974457+0000","last_active":"2023-10-30T12:28:44.983794+0000","last_peered":"2023-10-30T12:28:44.983794+0000","last_clean":"2023-10-30T12:28:44.983794+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T12:28:44.983794+0000","last_undegraded":"2023-10-30T12:28:44.983794+0000","last_fullsized":"2023-10-30T12:28:44.983794+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:30:35.231889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.957613+0000","last_change":"2023-10-30T12:28:44.075493+0000","last_active":"2023-10-30T12:28:44.957613+0000","last_peered":"2023-10-30T12:28:44.957613+0000","last_clean":"2023-10-30T12:28:44.957613+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T12:28:44.957613+0000","last_undegraded":"2023-10-30T12:28:44.957613+0000","last_fullsized":"2023-10-30T12:28:44.957613+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:00:11.718669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.983743+0000","last_change":"2023-10-30T12:28:44.974424+0000","last_active":"2023-10-30T12:28:44.983743+0000","last_peered":"2023-10-30T12:28:44.983743+0000","last_clean":"2023-10-30T12:28:44.983743+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T12:28:44.983743+0000","last_undegraded":"2023-10-30T12:28:44.983743+0000","last_fullsized":"2023-10-30T12:28:44.983743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:56:01.484533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.947065+0000","last_change":"2023-10-30T12:28:44.065843+0000","last_active":"2023-10-30T12:28:45.947065+0000","last_peered":"2023-10-30T12:28:45.947065+0000","last_clean":"2023-10-30T12:28:45.947065+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T12:28:45.947065+0000","last_undegraded":"2023-10-30T12:28:45.947065+0000","last_fullsized":"2023-10-30T12:28:45.947065+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:32.405199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476743,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1890000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.089}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]}]},{"osd":2,"up_from":16,"seq":68719476743,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54500000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]}]},{"osd":1,"up_from":16,"seq":68719476741,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":811008,"data_stored":666236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0069999999999999}]}]},{"osd":0,"up_from":16,"seq":68719476742,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77800000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.157}]}]},{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":75868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92200000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0249999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":75868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]}]},{"osd":7,"up_from":14,"seq":60129542149,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":204800,"data_stored":69568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58499999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80600000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.085}]}]},{"osd":5,"up_from":14,"seq":60129542148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":798720,"data_stored":659936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81200000000000006}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.054}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.222}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:28:51.557 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-30T12:28:51.557 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-30T12:28:51.558 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-30T12:28:51.558 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-30T12:28:51.558 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-30T12:28:51.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.731+0000 7ff57e817640 1 Processor -- start 2023-10-30T12:28:51.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff57e817640 1 -- start start 2023-10-30T12:28:51.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff57e817640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff57815d6f0 0x7ff57815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff57e817640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5780cf160 0x7ff5780cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff57e817640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5780cb800 con 0x7ff5781538a0 2023-10-30T12:28:51.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff57e817640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5780cb330 con 0x7ff57815d6f0 2023-10-30T12:28:51.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff57e817640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5780cad70 con 0x7ff5780cf160 2023-10-30T12:28:51.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff57cdb4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5780cf160 0x7ff5780cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff5777fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff57815d6f0 0x7ff57815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff57cdb4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5780cf160 0x7ff5780cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38582/0 (socket says 172.21.15.125:38582) 2023-10-30T12:28:51.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff57cdb4640 1 -- 172.21.15.125:0/3413813894 learned_addr learned my addr 172.21.15.125:0/3413813894 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:51.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff57cdb4640 1 -- 172.21.15.125:0/3413813894 >> v2:172.21.15.136:3300/0 conn(0x7ff57815d6f0 msgr2=0x7ff57815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:51.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff57cdb4640 1 --2- 172.21.15.125:0/3413813894 >> v2:172.21.15.136:3300/0 conn(0x7ff57815d6f0 0x7ff57815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff57cdb4640 1 -- 172.21.15.125:0/3413813894 >> v1:172.21.15.136:6789/0 conn(0x7ff5781538a0 legacy=0x7ff5780ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff57cdb4640 1 -- 172.21.15.125:0/3413813894 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5780cb050 con 0x7ff5780cf160 2023-10-30T12:28:51.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.735+0000 7ff5777fe640 1 --2- 172.21.15.125:0/3413813894 >> v2:172.21.15.136:3300/0 conn(0x7ff57815d6f0 0x7ff57815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.739+0000 7ff57cdb4640 1 --2- 172.21.15.125:0/3413813894 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5780cf160 0x7ff5780cf590 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff570009e50 tx=0x7ff57001f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28df466b91940acc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.739+0000 7ff576ffd640 1 -- 172.21.15.125:0/3413813894 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5700af400 con 0x7ff5780cf160 2023-10-30T12:28:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.739+0000 7ff576ffd640 1 -- 172.21.15.125:0/3413813894 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5700ab050 con 0x7ff5780cf160 2023-10-30T12:28:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.739+0000 7ff576ffd640 1 -- 172.21.15.125:0/3413813894 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5700af780 con 0x7ff5780cf160 2023-10-30T12:28:51.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.739+0000 7ff57e817640 1 -- 172.21.15.125:0/3413813894 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5780cf160 msgr2=0x7ff5780cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.739+0000 7ff57e817640 1 --2- 172.21.15.125:0/3413813894 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5780cf160 0x7ff5780cf590 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff570009e50 tx=0x7ff57001f1f0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.739+0000 7ff57e817640 1 -- 172.21.15.125:0/3413813894 shutdown_connections 2023-10-30T12:28:51.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.739+0000 7ff57e817640 1 --2- 172.21.15.125:0/3413813894 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5780cf160 0x7ff5780cf590 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.739+0000 7ff57e817640 1 --2- 172.21.15.125:0/3413813894 >> v2:172.21.15.136:3300/0 conn(0x7ff57815d6f0 0x7ff57815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.739+0000 7ff57e817640 1 -- 172.21.15.125:0/3413813894 >> 172.21.15.125:0/3413813894 conn(0x7ff57806bfe0 msgr2=0x7ff57806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:51.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.739+0000 7ff57e817640 1 -- 172.21.15.125:0/3413813894 shutdown_connections 2023-10-30T12:28:51.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.739+0000 7ff57e817640 1 -- 172.21.15.125:0/3413813894 wait complete. 2023-10-30T12:28:51.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff57e817640 1 Processor -- start 2023-10-30T12:28:51.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff57e817640 1 -- start start 2023-10-30T12:28:51.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff57e817640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5780cf160 0x7ff57814e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff57e817640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5781538a0 0x7ff5780b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff57e817640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff57815d6f0 0x7ff5780cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff57e817640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5781efd70 con 0x7ff5780cf160 2023-10-30T12:28:51.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff57e817640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5781f0260 con 0x7ff57815d6f0 2023-10-30T12:28:51.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff57e817640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5780ce7b0 con 0x7ff5781538a0 2023-10-30T12:28:51.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff577fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5780cf160 0x7ff57814e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff577fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5780cf160 0x7ff57814e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:32846/0 (socket says 172.21.15.125:32846) 2023-10-30T12:28:51.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff577fff640 1 -- 172.21.15.125:0/1604939471 learned_addr learned my addr 172.21.15.125:0/1604939471 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:51.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff57cdb4640 1 --2- 172.21.15.125:0/1604939471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff57815d6f0 0x7ff5780cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff5777fe640 1 --2- 172.21.15.125:0/1604939471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5781538a0 0x7ff5780b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff577fff640 1 -- 172.21.15.125:0/1604939471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5781538a0 msgr2=0x7ff5780b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff577fff640 1 --2- 172.21.15.125:0/1604939471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5781538a0 0x7ff5780b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff577fff640 1 -- 172.21.15.125:0/1604939471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff57815d6f0 msgr2=0x7ff5780cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff577fff640 1 --2- 172.21.15.125:0/1604939471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff57815d6f0 0x7ff5780cd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff577fff640 1 -- 172.21.15.125:0/1604939471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56c009420 con 0x7ff5780cf160 2023-10-30T12:28:51.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff577fff640 1 --2- 172.21.15.125:0/1604939471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5780cf160 0x7ff57814e220 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff56800c500 tx=0x7ff56800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:51.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff574ff9640 1 -- 172.21.15.125:0/1604939471 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff568005220 con 0x7ff5780cf160 2023-10-30T12:28:51.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff574ff9640 1 -- 172.21.15.125:0/1604939471 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff568005380 con 0x7ff5780cf160 2023-10-30T12:28:51.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff57e817640 1 -- 172.21.15.125:0/1604939471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff570009b30 con 0x7ff5780cf160 2023-10-30T12:28:51.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff574ff9640 1 -- 172.21.15.125:0/1604939471 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff568007270 con 0x7ff5780cf160 2023-10-30T12:28:51.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff57e817640 1 -- 172.21.15.125:0/1604939471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5781eff80 con 0x7ff5780cf160 2023-10-30T12:28:51.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.743+0000 7ff57e817640 1 -- 172.21.15.125:0/1604939471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5780b2a50 con 0x7ff5780cf160 2023-10-30T12:28:51.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.747+0000 7ff574ff9640 1 -- 172.21.15.125:0/1604939471 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5680073d0 con 0x7ff5780cf160 2023-10-30T12:28:51.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.747+0000 7ff574ff9640 1 --2- 172.21.15.125:0/1604939471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5640664d0 0x7ff564068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:51.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.747+0000 7ff574ff9640 1 -- 172.21.15.125:0/1604939471 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff568083730 con 0x7ff5780cf160 2023-10-30T12:28:51.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.747+0000 7ff5777fe640 1 --2- 172.21.15.125:0/1604939471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5640664d0 0x7ff564068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:51.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.747+0000 7ff5777fe640 1 --2- 172.21.15.125:0/1604939471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5640664d0 0x7ff564068990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff5780c98a0 tx=0x7ff56c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:51.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.751+0000 7ff574ff9640 1 -- 172.21.15.125:0/1604939471 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff568050b20 con 0x7ff5780cf160 2023-10-30T12:28:51.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.935+0000 7ff57e817640 1 -- 172.21.15.125:0/1604939471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff57804e170 con 0x7ff5780cf160 2023-10-30T12:28:51.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.939+0000 7ff574ff9640 1 -- 172.21.15.125:0/1604939471 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+164434 (secure 0 0 0) 0x7ff568054140 con 0x7ff5780cf160 2023-10-30T12:28:51.942 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:51.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.939+0000 7ff57e817640 1 -- 172.21.15.125:0/1604939471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5640664d0 msgr2=0x7ff564068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.939+0000 7ff57e817640 1 --2- 172.21.15.125:0/1604939471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5640664d0 0x7ff564068990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff5780c98a0 tx=0x7ff56c008040 comp rx=0 tx=0).stop 2023-10-30T12:28:51.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.939+0000 7ff57e817640 1 -- 172.21.15.125:0/1604939471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5780cf160 msgr2=0x7ff57814e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:51.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.939+0000 7ff57e817640 1 --2- 172.21.15.125:0/1604939471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5780cf160 0x7ff57814e220 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff56800c500 tx=0x7ff56800c9c0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.939+0000 7ff57e817640 1 -- 172.21.15.125:0/1604939471 shutdown_connections 2023-10-30T12:28:51.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.939+0000 7ff57e817640 1 --2- 172.21.15.125:0/1604939471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5640664d0 0x7ff564068990 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.939+0000 7ff57e817640 1 --2- 172.21.15.125:0/1604939471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff57815d6f0 0x7ff5780cd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.939+0000 7ff57e817640 1 --2- 172.21.15.125:0/1604939471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5781538a0 0x7ff5780b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.939+0000 7ff57e817640 1 --2- 172.21.15.125:0/1604939471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5780cf160 0x7ff57814e220 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:51.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.939+0000 7ff57e817640 1 -- 172.21.15.125:0/1604939471 >> 172.21.15.125:0/1604939471 conn(0x7ff57806bfe0 msgr2=0x7ff5780c48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:51.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.943+0000 7ff57e817640 1 -- 172.21.15.125:0/1604939471 shutdown_connections 2023-10-30T12:28:51.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:51.943+0000 7ff57e817640 1 -- 172.21.15.125:0/1604939471 wait complete. 2023-10-30T12:28:51.962 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":5,"active_gid":4108,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6832","nonce":999226929},{"type":"v1","addr":"172.21.15.136:6833","nonce":999226929}]},"active_addr":"172.21.15.136:6833/999226929","active_change":"2023-10-30T12:28:24.942359+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":1386414757}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":1917784796}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":2780885842}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.136:0","nonce":1265890078}]}]} 2023-10-30T12:28:51.966 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-30T12:28:51.966 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-30T12:28:51.967 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:52.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.103+0000 7fcb81f9d640 1 Processor -- start 2023-10-30T12:28:52.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.103+0000 7fcb81f9d640 1 -- start start 2023-10-30T12:28:52.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.103+0000 7fcb81f9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb7c151800 0x7fcb7c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.103+0000 7fcb81f9d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c06e090 con 0x7fcb7c0c1ee0 2023-10-30T12:28:52.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.103+0000 7fcb81f9d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb7c0c8a50 con 0x7fcb7c0c8260 2023-10-30T12:28:52.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.103+0000 7fcb81f9d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb7c06dc70 con 0x7fcb7c151800 2023-10-30T12:28:52.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.103+0000 7fcb7b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb7c151800 0x7fcb7c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.107+0000 7fcb7b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb7c151800 0x7fcb7c06e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:32850/0 (socket says 172.21.15.125:32850) 2023-10-30T12:28:52.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.107+0000 7fcb7affd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcb7c0c1ee0 0x7fcb7c0c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54058/0 (socket says 172.21.15.125:54058) 2023-10-30T12:28:52.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.107+0000 7fcb7b7fe640 1 -- 172.21.15.125:0/3160632368 learned_addr learned my addr 172.21.15.125:0/3160632368 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:52.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.107+0000 7fcb7b7fe640 1 -- 172.21.15.125:0/3160632368 >> v1:172.21.15.136:6789/0 conn(0x7fcb7c0c8260 legacy=0x7fcb7c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.107+0000 7fcb7b7fe640 1 -- 172.21.15.125:0/3160632368 >> v1:172.21.15.125:6789/0 conn(0x7fcb7c0c1ee0 legacy=0x7fcb7c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.107+0000 7fcb7a7fc640 1 -- 172.21.15.125:0/3160632368 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425396603 0 0) 0x7fcb7c06e090 con 0x7fcb7c0c1ee0 2023-10-30T12:28:52.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.107+0000 7fcb7b7fe640 1 -- 172.21.15.125:0/3160632368 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb7c0c3b10 con 0x7fcb7c151800 2023-10-30T12:28:52.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.107+0000 7fcb7b7fe640 1 --2- 172.21.15.125:0/3160632368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7c151800 0x7fcb7c06e470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009870 tx=0x7fcb6c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34248533c2f6e4c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.107+0000 7fcb7a7fc640 1 -- 172.21.15.125:0/3160632368 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c0aa670 con 0x7fcb7c151800 2023-10-30T12:28:52.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.107+0000 7fcb7a7fc640 1 -- 172.21.15.125:0/3160632368 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb6c0aa7d0 con 0x7fcb7c151800 2023-10-30T12:28:52.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.107+0000 7fcb7a7fc640 1 -- 172.21.15.125:0/3160632368 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb6c0aab60 con 0x7fcb7c151800 2023-10-30T12:28:52.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.111+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3160632368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7c151800 msgr2=0x7fcb7c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.111+0000 7fcb81f9d640 1 --2- 172.21.15.125:0/3160632368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7c151800 0x7fcb7c06e470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c009870 tx=0x7fcb6c01c510 comp rx=0 tx=0).stop 2023-10-30T12:28:52.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.111+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3160632368 shutdown_connections 2023-10-30T12:28:52.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.111+0000 7fcb81f9d640 1 --2- 172.21.15.125:0/3160632368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7c151800 0x7fcb7c06e470 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.111+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3160632368 >> 172.21.15.125:0/3160632368 conn(0x7fcb7c071ea0 msgr2=0x7fcb7c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:52.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.111+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3160632368 shutdown_connections 2023-10-30T12:28:52.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.111+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3160632368 wait complete. 2023-10-30T12:28:52.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.111+0000 7fcb81f9d640 1 Processor -- start 2023-10-30T12:28:52.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.111+0000 7fcb81f9d640 1 -- start start 2023-10-30T12:28:52.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb81f9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb7c0c1ee0 0x7fcb7c14cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb81f9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb7c0c8260 0x7fcb7c06afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb81f9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7c151800 0x7fcb7c1f5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb81f9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0c45e0 con 0x7fcb7c151800 2023-10-30T12:28:52.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb81f9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0c48c0 con 0x7fcb7c0c1ee0 2023-10-30T12:28:52.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb81f9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb7c0c4ba0 con 0x7fcb7c0c8260 2023-10-30T12:28:52.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb7bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7c151800 0x7fcb7c1f5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb7bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7c151800 0x7fcb7c1f5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:32860/0 (socket says 172.21.15.125:32860) 2023-10-30T12:28:52.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb7bfff640 1 -- 172.21.15.125:0/3592752325 learned_addr learned my addr 172.21.15.125:0/3592752325 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:52.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb7affd640 1 --2- 172.21.15.125:0/3592752325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb7c0c8260 0x7fcb7c06afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb7b7fe640 1 --2- 172.21.15.125:0/3592752325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb7c0c1ee0 0x7fcb7c14cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb7bfff640 1 -- 172.21.15.125:0/3592752325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb7c0c8260 msgr2=0x7fcb7c06afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb7bfff640 1 --2- 172.21.15.125:0/3592752325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb7c0c8260 0x7fcb7c06afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb7bfff640 1 -- 172.21.15.125:0/3592752325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb7c0c1ee0 msgr2=0x7fcb7c14cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb7bfff640 1 --2- 172.21.15.125:0/3592752325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb7c0c1ee0 0x7fcb7c14cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb7bfff640 1 -- 172.21.15.125:0/3592752325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c0094d0 con 0x7fcb7c151800 2023-10-30T12:28:52.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb7bfff640 1 --2- 172.21.15.125:0/3592752325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7c151800 0x7fcb7c1f5790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400c730 tx=0x7fcb7400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb63fff640 1 -- 172.21.15.125:0/3592752325 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb74003b80 con 0x7fcb7c151800 2023-10-30T12:28:52.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb63fff640 1 -- 172.21.15.125:0/3592752325 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb74003ce0 con 0x7fcb7c151800 2023-10-30T12:28:52.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb63fff640 1 -- 172.21.15.125:0/3592752325 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb7400aa30 con 0x7fcb7c151800 2023-10-30T12:28:52.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3592752325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb7c0c4e80 con 0x7fcb7c151800 2023-10-30T12:28:52.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3592752325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb7c0c5440 con 0x7fcb7c151800 2023-10-30T12:28:52.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3592752325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7c04e170 con 0x7fcb7c151800 2023-10-30T12:28:52.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.115+0000 7fcb63fff640 1 -- 172.21.15.125:0/3592752325 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcb7400ab90 con 0x7fcb7c151800 2023-10-30T12:28:52.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.119+0000 7fcb63fff640 1 --2- 172.21.15.125:0/3592752325 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb680664d0 0x7fcb68068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.119+0000 7fcb63fff640 1 -- 172.21.15.125:0/3592752325 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcb74083d50 con 0x7fcb7c151800 2023-10-30T12:28:52.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.119+0000 7fcb7b7fe640 1 --2- 172.21.15.125:0/3592752325 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb680664d0 0x7fcb68068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.119+0000 7fcb7b7fe640 1 --2- 172.21.15.125:0/3592752325 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb680664d0 0x7fcb68068990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c0ae000 tx=0x7fcb6c0a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.123+0000 7fcb63fff640 1 -- 172.21.15.125:0/3592752325 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcb740510a0 con 0x7fcb7c151800 2023-10-30T12:28:52.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.279+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3592752325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcb7c151030 con 0x7fcb7c151800 2023-10-30T12:28:52.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.279+0000 7fcb63fff640 1 -- 172.21.15.125:0/3592752325 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16557 (secure 0 0 0) 0x7fcb740546c0 con 0x7fcb7c151800 2023-10-30T12:28:52.284 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:52.284 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:45.927862+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:52.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.283+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3592752325 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb680664d0 msgr2=0x7fcb68068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.283+0000 7fcb81f9d640 1 --2- 172.21.15.125:0/3592752325 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb680664d0 0x7fcb68068990 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c0ae000 tx=0x7fcb6c0a7000 comp rx=0 tx=0).stop 2023-10-30T12:28:52.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.283+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3592752325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7c151800 msgr2=0x7fcb7c1f5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.283+0000 7fcb81f9d640 1 --2- 172.21.15.125:0/3592752325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7c151800 0x7fcb7c1f5790 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcb7400c730 tx=0x7fcb7400cbf0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.283+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3592752325 shutdown_connections 2023-10-30T12:28:52.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.283+0000 7fcb81f9d640 1 --2- 172.21.15.125:0/3592752325 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb680664d0 0x7fcb68068990 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.283+0000 7fcb81f9d640 1 --2- 172.21.15.125:0/3592752325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb7c151800 0x7fcb7c1f5790 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.283+0000 7fcb81f9d640 1 --2- 172.21.15.125:0/3592752325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb7c0c8260 0x7fcb7c06afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.283+0000 7fcb81f9d640 1 --2- 172.21.15.125:0/3592752325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb7c0c1ee0 0x7fcb7c14cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.283+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3592752325 >> 172.21.15.125:0/3592752325 conn(0x7fcb7c071ea0 msgr2=0x7fcb7c0cc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:52.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.283+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3592752325 shutdown_connections 2023-10-30T12:28:52.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.283+0000 7fcb81f9d640 1 -- 172.21.15.125:0/3592752325 wait complete. 2023-10-30T12:28:52.305 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-30T12:28:52.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:52.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb45c4640 1 Processor -- start 2023-10-30T12:28:52.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb45c4640 1 -- start start 2023-10-30T12:28:52.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb45c4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7feeac0b2300 0x7feeac071b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb45c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feeac06f370 0x7feeac072070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb45c4640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac06b3b0 con 0x7feeac1570f0 2023-10-30T12:28:52.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb45c4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feeac0c38a0 con 0x7feeac0b2300 2023-10-30T12:28:52.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb45c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeac0c3b20 con 0x7feeac06f370 2023-10-30T12:28:52.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb2360640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7feeac1570f0 0x7feeac06ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54072/0 (socket says 172.21.15.125:54072) 2023-10-30T12:28:52.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb2360640 1 -- 172.21.15.125:0/2920162208 learned_addr learned my addr 172.21.15.125:0/2920162208 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:52.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb1b5f640 1 --2- 172.21.15.125:0/2920162208 >> v2:172.21.15.136:3300/0 conn(0x7feeac0b2300 0x7feeac071b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb2b61640 1 --2- 172.21.15.125:0/2920162208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feeac06f370 0x7feeac072070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb135e640 1 -- 172.21.15.125:0/2920162208 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222242186 0 0) 0x7feeac06b3b0 con 0x7feeac1570f0 2023-10-30T12:28:52.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb135e640 1 -- 172.21.15.125:0/2920162208 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee98003530 con 0x7feeac1570f0 2023-10-30T12:28:52.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb135e640 1 -- 172.21.15.125:0/2920162208 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1711289988 0 0) 0x7fee98003530 con 0x7feeac1570f0 2023-10-30T12:28:52.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb135e640 1 -- 172.21.15.125:0/2920162208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feeac06f370 msgr2=0x7feeac072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb135e640 1 --2- 172.21.15.125:0/2920162208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feeac06f370 0x7feeac072070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb135e640 1 -- 172.21.15.125:0/2920162208 >> v2:172.21.15.136:3300/0 conn(0x7feeac0b2300 msgr2=0x7feeac071b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb135e640 1 --2- 172.21.15.125:0/2920162208 >> v2:172.21.15.136:3300/0 conn(0x7feeac0b2300 0x7feeac071b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb135e640 1 -- 172.21.15.125:0/2920162208 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac0ca9f0 con 0x7feeac1570f0 2023-10-30T12:28:52.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.439+0000 7feeb135e640 1 -- 172.21.15.125:0/2920162208 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7feea8001680 con 0x7feeac1570f0 2023-10-30T12:28:52.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb135e640 1 -- 172.21.15.125:0/2920162208 >> v1:172.21.15.125:6789/0 conn(0x7feeac1570f0 legacy=0x7feeac06ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb135e640 1 --2- 172.21.15.125:0/2920162208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee98003980 0x7fee98003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb135e640 1 --2- 172.21.15.125:0/2920162208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee980047d0 0x7fee98004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb135e640 1 --2- 172.21.15.125:0/2920162208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feeac06f370 0x7fee98005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb135e640 1 -- 172.21.15.125:0/2920162208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac0c38a0 con 0x7fee980047d0 2023-10-30T12:28:52.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb135e640 1 -- 172.21.15.125:0/2920162208 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac0c3b20 con 0x7fee98003980 2023-10-30T12:28:52.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb135e640 1 -- 172.21.15.125:0/2920162208 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9800b990 con 0x7feeac06f370 2023-10-30T12:28:52.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb1b5f640 1 --2- 172.21.15.125:0/2920162208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee980047d0 0x7fee98004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb2360640 1 --2- 172.21.15.125:0/2920162208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee98003980 0x7fee98003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb2b61640 1 --2- 172.21.15.125:0/2920162208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feeac06f370 0x7fee98005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb1b5f640 1 -- 172.21.15.125:0/2920162208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feeac06f370 msgr2=0x7fee98005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb1b5f640 1 --2- 172.21.15.125:0/2920162208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feeac06f370 0x7fee98005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb1b5f640 1 -- 172.21.15.125:0/2920162208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee98003980 msgr2=0x7fee98003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb1b5f640 1 --2- 172.21.15.125:0/2920162208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee98003980 0x7fee98003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb1b5f640 1 -- 172.21.15.125:0/2920162208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee98003530 con 0x7fee980047d0 2023-10-30T12:28:52.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb1b5f640 1 --2- 172.21.15.125:0/2920162208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee980047d0 0x7fee98004c50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fee9c009ae0 tx=0x7fee9c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588d7512fbcdf1ec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb135e640 1 -- 172.21.15.125:0/2920162208 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9c0a55a0 con 0x7fee980047d0 2023-10-30T12:28:52.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb135e640 1 -- 172.21.15.125:0/2920162208 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9c0a5700 con 0x7fee980047d0 2023-10-30T12:28:52.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.443+0000 7feeb135e640 1 -- 172.21.15.125:0/2920162208 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9c0a5ab0 con 0x7fee980047d0 2023-10-30T12:28:52.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb45c4640 1 -- 172.21.15.125:0/2920162208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee980047d0 msgr2=0x7fee98004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb45c4640 1 --2- 172.21.15.125:0/2920162208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee980047d0 0x7fee98004c50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fee9c009ae0 tx=0x7fee9c01ca90 comp rx=0 tx=0).stop 2023-10-30T12:28:52.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb2b61640 1 -- 172.21.15.125:0/2920162208 reap_dead start 2023-10-30T12:28:52.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb45c4640 1 -- 172.21.15.125:0/2920162208 shutdown_connections 2023-10-30T12:28:52.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb45c4640 1 -- 172.21.15.125:0/2920162208 >> 172.21.15.125:0/2920162208 conn(0x7feeac069e90 msgr2=0x7feeac06a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:52.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb45c4640 1 -- 172.21.15.125:0/2920162208 shutdown_connections 2023-10-30T12:28:52.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb45c4640 1 -- 172.21.15.125:0/2920162208 wait complete. 2023-10-30T12:28:52.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb45c4640 1 Processor -- start 2023-10-30T12:28:52.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb45c4640 1 -- start start 2023-10-30T12:28:52.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb45c4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee980047d0 0x7feeac1f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb45c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feeac15c7c0 0x7feeac0cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb45c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feeac1ef6a0 0x7feeac14b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb45c4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac1f3380 con 0x7fee980047d0 2023-10-30T12:28:52.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb45c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac15d240 con 0x7feeac1ef6a0 2023-10-30T12:28:52.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb45c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeac15a990 con 0x7feeac15c7c0 2023-10-30T12:28:52.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb2360640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee980047d0 0x7feeac1f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb2360640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee980047d0 0x7feeac1f2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:32882/0 (socket says 172.21.15.125:32882) 2023-10-30T12:28:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb2360640 1 -- 172.21.15.125:0/1082262400 learned_addr learned my addr 172.21.15.125:0/1082262400 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb2b61640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feeac1ef6a0 0x7feeac14b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb1b5f640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feeac15c7c0 0x7feeac0cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb2360640 1 -- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feeac15c7c0 msgr2=0x7feeac0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb2360640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feeac15c7c0 0x7feeac0cbbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb2360640 1 -- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feeac1ef6a0 msgr2=0x7feeac14b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb2360640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feeac1ef6a0 0x7feeac14b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.447+0000 7feeb2360640 1 -- 172.21.15.125:0/1082262400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac1eeec0 con 0x7fee980047d0 2023-10-30T12:28:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.451+0000 7feeb2b61640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feeac1ef6a0 0x7feeac14b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.451+0000 7feeb2360640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee980047d0 0x7feeac1f2e40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feea8012540 tx=0x7feea8012a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.451+0000 7feea2ffd640 1 -- 172.21.15.125:0/1082262400 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea8014810 con 0x7fee980047d0 2023-10-30T12:28:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.451+0000 7feea2ffd640 1 -- 172.21.15.125:0/1082262400 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea8014970 con 0x7fee980047d0 2023-10-30T12:28:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.451+0000 7feea2ffd640 1 -- 172.21.15.125:0/1082262400 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea8014d20 con 0x7fee980047d0 2023-10-30T12:28:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.451+0000 7feeb45c4640 1 -- 172.21.15.125:0/1082262400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeac15d4c0 con 0x7fee980047d0 2023-10-30T12:28:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.451+0000 7feeb45c4640 1 -- 172.21.15.125:0/1082262400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeac15da80 con 0x7fee980047d0 2023-10-30T12:28:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.451+0000 7feeb45c4640 1 -- 172.21.15.125:0/1082262400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeac04e170 con 0x7fee980047d0 2023-10-30T12:28:52.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.451+0000 7feea2ffd640 1 -- 172.21.15.125:0/1082262400 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feea8016480 con 0x7fee980047d0 2023-10-30T12:28:52.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.451+0000 7feea2ffd640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fee900664d0 0x7fee90068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.451+0000 7feea2ffd640 1 -- 172.21.15.125:0/1082262400 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feea8087280 con 0x7fee980047d0 2023-10-30T12:28:52.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.451+0000 7feeb1b5f640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fee900664d0 0x7fee90068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.455+0000 7feeb1b5f640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fee900664d0 0x7fee90068990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0af000 tx=0x7fee9c0046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.455+0000 7feea2ffd640 1 -- 172.21.15.125:0/1082262400 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feea80547c0 con 0x7fee980047d0 2023-10-30T12:28:52.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.607+0000 7feeb45c4640 1 -- 172.21.15.125:0/1082262400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feeac06fbd0 con 0x7fee980047d0 2023-10-30T12:28:52.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.607+0000 7feea2ffd640 1 -- 172.21.15.125:0/1082262400 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16557 (secure 0 0 0) 0x7feea8057de0 con 0x7fee980047d0 2023-10-30T12:28:52.612 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:52.612 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:45.927862+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:52.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.611+0000 7feeb45c4640 1 -- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fee900664d0 msgr2=0x7fee90068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.611+0000 7feeb45c4640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fee900664d0 0x7fee90068990 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fee9c0af000 tx=0x7fee9c0046d0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.611+0000 7feeb45c4640 1 -- 172.21.15.125:0/1082262400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee980047d0 msgr2=0x7feeac1f2e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.611+0000 7feeb45c4640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee980047d0 0x7feeac1f2e40 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feea8012540 tx=0x7feea8012a00 comp rx=0 tx=0).stop 2023-10-30T12:28:52.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.611+0000 7feeb45c4640 1 -- 172.21.15.125:0/1082262400 shutdown_connections 2023-10-30T12:28:52.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.611+0000 7feeb45c4640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fee900664d0 0x7fee90068990 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.611+0000 7feeb45c4640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feeac1ef6a0 0x7feeac14b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.611+0000 7feeb45c4640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feeac15c7c0 0x7feeac0cbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.611+0000 7feeb45c4640 1 --2- 172.21.15.125:0/1082262400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee980047d0 0x7feeac1f2e40 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.611+0000 7feeb45c4640 1 -- 172.21.15.125:0/1082262400 >> 172.21.15.125:0/1082262400 conn(0x7feeac069e90 msgr2=0x7feeac0c28c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:52.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.611+0000 7feeb45c4640 1 -- 172.21.15.125:0/1082262400 shutdown_connections 2023-10-30T12:28:52.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.611+0000 7feeb45c4640 1 -- 172.21.15.125:0/1082262400 wait complete. 2023-10-30T12:28:52.633 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-30T12:28:52.634 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-30T12:28:52.634 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-30T12:28:52.635 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-30T12:28:52.635 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-30T12:28:52.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-30T12:28:52.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-30T12:28:52.637 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-30T12:28:52.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b9679640 1 Processor -- start 2023-10-30T12:28:52.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b9679640 1 -- start start 2023-10-30T12:28:52.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f94305d3640 1 Processor -- start 2023-10-30T12:28:52.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f94305d3640 1 -- start start 2023-10-30T12:28:52.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b9679640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f52b406ad90 0x7f52b4152de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b9679640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f52b40c8430 0x7f52b4156280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b9679640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b40c6670 0x7f52b4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b9679640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b4070820 con 0x7f52b406ad90 2023-10-30T12:28:52.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b9679640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b41567c0 con 0x7f52b40c8430 2023-10-30T12:28:52.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b9679640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b40c7400 con 0x7f52b40c6670 2023-10-30T12:28:52.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b2ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f52b406ad90 0x7f52b4152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b27fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f52b40c8430 0x7f52b4156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b27fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f52b40c8430 0x7f52b4156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38730/0 (socket says 172.21.15.125:38730) 2023-10-30T12:28:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b27fc640 1 -- 172.21.15.125:0/2940805855 learned_addr learned my addr 172.21.15.125:0/2940805855 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:52.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b37fe640 1 --2- 172.21.15.125:0/2940805855 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b40c6670 0x7f52b4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f94305d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9428153c60 0x7f94280ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f94305d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94280c4430 0x7f94280cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f94305d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9428159830 0x7f9428159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f94305d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9428154040 con 0x7f9428153c60 2023-10-30T12:28:52.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f94305d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94280cb780 con 0x7f9428159830 2023-10-30T12:28:52.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f94305d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94280cc9f0 con 0x7f94280c4430 2023-10-30T12:28:52.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f942eb70640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9428159830 0x7f9428159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f942db6e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94280c4430 0x7f94280cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f942db6e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94280c4430 0x7f94280cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38640/0 (socket says 172.21.15.125:38640) 2023-10-30T12:28:52.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f942e36f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9428153c60 0x7f94280ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f942e36f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9428153c60 0x7f94280ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:32906/0 (socket says 172.21.15.125:32906) 2023-10-30T12:28:52.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f942e36f640 1 -- 172.21.15.125:0/2119714763 learned_addr learned my addr 172.21.15.125:0/2119714763 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:52.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f942eb70640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9428159830 0x7f9428159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38736/0 (socket says 172.21.15.125:38736) 2023-10-30T12:28:52.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b27fc640 1 -- 172.21.15.125:0/2940805855 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b40c6670 msgr2=0x7f52b4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b27fc640 1 --2- 172.21.15.125:0/2940805855 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b40c6670 0x7f52b4069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b27fc640 1 -- 172.21.15.125:0/2940805855 >> v2:172.21.15.125:3300/0 conn(0x7f52b406ad90 msgr2=0x7f52b4152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b27fc640 1 --2- 172.21.15.125:0/2940805855 >> v2:172.21.15.125:3300/0 conn(0x7f52b406ad90 0x7f52b4152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.871+0000 7f52b27fc640 1 -- 172.21.15.125:0/2940805855 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b40c76c0 con 0x7f52b40c8430 2023-10-30T12:28:52.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f942eb70640 1 -- 172.21.15.125:0/2119714763 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94280c4430 msgr2=0x7f94280cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f942eb70640 1 --2- 172.21.15.125:0/2119714763 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94280c4430 0x7f94280cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f942eb70640 1 -- 172.21.15.125:0/2119714763 >> v2:172.21.15.125:3300/0 conn(0x7f9428153c60 msgr2=0x7f94280ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f942eb70640 1 --2- 172.21.15.125:0/2119714763 >> v2:172.21.15.125:3300/0 conn(0x7f9428153c60 0x7f94280ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f942eb70640 1 -- 172.21.15.125:0/2119714763 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f942815a2e0 con 0x7f9428159830 2023-10-30T12:28:52.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.875+0000 7f942e36f640 1 --2- 172.21.15.125:0/2119714763 >> v2:172.21.15.125:3300/0 conn(0x7f9428153c60 0x7f94280ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:52.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.879+0000 7f52b27fc640 1 --2- 172.21.15.125:0/2940805855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b40c8430 0x7f52b4156280 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f52b4072e50 tx=0x7f52a8020950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d645ad3797e3417 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.879+0000 7f52b1ffb640 1 -- 172.21.15.125:0/2940805855 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a80a97a0 con 0x7f52b40c8430 2023-10-30T12:28:52.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.879+0000 7f52b1ffb640 1 -- 172.21.15.125:0/2940805855 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a80a9900 con 0x7f52b40c8430 2023-10-30T12:28:52.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.879+0000 7f52b1ffb640 1 -- 172.21.15.125:0/2940805855 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a80a9c90 con 0x7f52b40c8430 2023-10-30T12:28:52.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.879+0000 7f52b9679640 1 -- 172.21.15.125:0/2940805855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b40c8430 msgr2=0x7f52b4156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.879+0000 7f52b9679640 1 --2- 172.21.15.125:0/2940805855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b40c8430 0x7f52b4156280 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f52b4072e50 tx=0x7f52a8020950 comp rx=0 tx=0).stop 2023-10-30T12:28:52.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f52b9679640 1 -- 172.21.15.125:0/2940805855 shutdown_connections 2023-10-30T12:28:52.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f52b9679640 1 --2- 172.21.15.125:0/2940805855 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b40c6670 0x7f52b4069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f52b9679640 1 --2- 172.21.15.125:0/2940805855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b40c8430 0x7f52b4156280 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f52b9679640 1 --2- 172.21.15.125:0/2940805855 >> v2:172.21.15.125:3300/0 conn(0x7f52b406ad90 0x7f52b4152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f52b9679640 1 -- 172.21.15.125:0/2940805855 >> 172.21.15.125:0/2940805855 conn(0x7f52b40c7910 msgr2=0x7f52b40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:52.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f52b9679640 1 -- 172.21.15.125:0/2940805855 shutdown_connections 2023-10-30T12:28:52.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f52b9679640 1 -- 172.21.15.125:0/2940805855 wait complete. 2023-10-30T12:28:52.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f3ba7b2a640 1 Processor -- start 2023-10-30T12:28:52.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f942eb70640 1 --2- 172.21.15.125:0/2119714763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9428159830 0x7f9428159ce0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f942808d1a0 tx=0x7f942001c930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de2cbc0e9d2d7db9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f3ba7b2a640 1 -- start start 2023-10-30T12:28:52.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f942d36d640 1 -- 172.21.15.125:0/2119714763 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94200a5880 con 0x7f9428159830 2023-10-30T12:28:52.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f942d36d640 1 -- 172.21.15.125:0/2119714763 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94200a59e0 con 0x7f9428159830 2023-10-30T12:28:52.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f942d36d640 1 -- 172.21.15.125:0/2119714763 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94200a5d70 con 0x7f9428159830 2023-10-30T12:28:52.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f52b9679640 1 Processor -- start 2023-10-30T12:28:52.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f3ba7b2a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3ba0153c60 0x7f3ba00ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f3ba7b2a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3ba00c4430 0x7f3ba00cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f3ba7b2a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ba00cb780 con 0x7f3ba0159830 2023-10-30T12:28:52.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f3ba7b2a640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba0154040 con 0x7f3ba00c4430 2023-10-30T12:28:52.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f3ba7b2a640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ba00cc9f0 con 0x7f3ba0153c60 2023-10-30T12:28:52.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.883+0000 7f52b9679640 1 -- start start 2023-10-30T12:28:52.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba58c6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3ba0153c60 0x7f3ba00ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba58c6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3ba0153c60 0x7f3ba00ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38742/0 (socket says 172.21.15.125:38742) 2023-10-30T12:28:52.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba58c6640 1 -- 172.21.15.125:0/584044531 learned_addr learned my addr 172.21.15.125:0/584044531 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:52.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f94305d3640 1 -- 172.21.15.125:0/2119714763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9428159830 msgr2=0x7f9428159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f94305d3640 1 --2- 172.21.15.125:0/2119714763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9428159830 0x7f9428159ce0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f942808d1a0 tx=0x7f942001c930 comp rx=0 tx=0).stop 2023-10-30T12:28:52.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b9679640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b406ad90 0x7f52b414c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b9679640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b40c6670 0x7f52b41f97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b9679640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b40c8430 0x7f52b40c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b9679640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b414b380 con 0x7f52b40c8430 2023-10-30T12:28:52.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b9679640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b40ca440 con 0x7f52b40c6670 2023-10-30T12:28:52.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b9679640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b40cac00 con 0x7f52b406ad90 2023-10-30T12:28:52.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f94305d3640 1 -- 172.21.15.125:0/2119714763 shutdown_connections 2023-10-30T12:28:52.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f94305d3640 1 --2- 172.21.15.125:0/2119714763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9428159830 0x7f9428159ce0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b27fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b40c6670 0x7f52b41f97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b2ffd640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b406ad90 0x7f52b414c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b27fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b40c6670 0x7f52b41f97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38750/0 (socket says 172.21.15.125:38750) 2023-10-30T12:28:52.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b27fc640 1 -- 172.21.15.125:0/2151436439 learned_addr learned my addr 172.21.15.125:0/2151436439 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:52.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b37fe640 1 --2- 172.21.15.125:0/2151436439 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b40c8430 0x7f52b40c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f94305d3640 1 --2- 172.21.15.125:0/2119714763 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94280c4430 0x7f94280cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f94305d3640 1 --2- 172.21.15.125:0/2119714763 >> v2:172.21.15.125:3300/0 conn(0x7f9428153c60 0x7f94280ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f94305d3640 1 -- 172.21.15.125:0/2119714763 >> 172.21.15.125:0/2119714763 conn(0x7f942806bfe0 msgr2=0x7f942806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:52.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f94305d3640 1 -- 172.21.15.125:0/2119714763 shutdown_connections 2023-10-30T12:28:52.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f94305d3640 1 -- 172.21.15.125:0/2119714763 wait complete. 2023-10-30T12:28:52.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b27fc640 1 -- 172.21.15.125:0/2151436439 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b406ad90 msgr2=0x7f52b414c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b27fc640 1 --2- 172.21.15.125:0/2151436439 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b406ad90 0x7f52b414c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b27fc640 1 -- 172.21.15.125:0/2151436439 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b40c8430 msgr2=0x7f52b40c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b27fc640 1 --2- 172.21.15.125:0/2151436439 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b40c8430 0x7f52b40c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b27fc640 1 -- 172.21.15.125:0/2151436439 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a80094d0 con 0x7f52b40c6670 2023-10-30T12:28:52.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f1836646640 1 Processor -- start 2023-10-30T12:28:52.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f52b27fc640 1 --2- 172.21.15.125:0/2151436439 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b40c6670 0x7f52b41f97f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f52a80211f0 tx=0x7f52a801d6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f1836646640 1 -- start start 2023-10-30T12:28:52.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 -- 172.21.15.125:0/584044531 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3402229894 0 0) 0x7f3ba00cb780 con 0x7f3ba0159830 2023-10-30T12:28:52.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 -- 172.21.15.125:0/584044531 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b88003530 con 0x7f3ba0159830 2023-10-30T12:28:52.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba50c5640 1 --2- 172.21.15.125:0/584044531 >> v2:172.21.15.125:3300/0 conn(0x7f3ba00c4430 0x7f3ba00cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 -- 172.21.15.125:0/584044531 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1850916004 0 0) 0x7f3b88003530 con 0x7f3ba0159830 2023-10-30T12:28:52.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 -- 172.21.15.125:0/584044531 >> v2:172.21.15.136:3300/0 conn(0x7f3ba0153c60 msgr2=0x7f3ba00ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 --2- 172.21.15.125:0/584044531 >> v2:172.21.15.136:3300/0 conn(0x7f3ba0153c60 0x7f3ba00ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 -- 172.21.15.125:0/584044531 >> v2:172.21.15.125:3300/0 conn(0x7f3ba00c4430 msgr2=0x7f3ba00cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 --2- 172.21.15.125:0/584044531 >> v2:172.21.15.125:3300/0 conn(0x7f3ba00c4430 0x7f3ba00cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 -- 172.21.15.125:0/584044531 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba015a2e0 con 0x7f3ba0159830 2023-10-30T12:28:52.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 -- 172.21.15.125:0/584044531 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f3b90001680 con 0x7f3ba0159830 2023-10-30T12:28:52.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 -- 172.21.15.125:0/584044531 >> v1:172.21.15.136:6789/0 conn(0x7f3ba0159830 legacy=0x7f3ba00cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 --2- 172.21.15.125:0/584044531 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b880039d0 0x7f3b88003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 --2- 172.21.15.125:0/584044531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b880047d0 0x7f3b88004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 --2- 172.21.15.125:0/584044531 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b88005190 0x7f3b8800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 -- 172.21.15.125:0/584044531 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba0154040 con 0x7f3b88005190 2023-10-30T12:28:52.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 -- 172.21.15.125:0/584044531 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00cc9f0 con 0x7f3b880047d0 2023-10-30T12:28:52.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba48c4640 1 -- 172.21.15.125:0/584044531 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8800bf80 con 0x7f3b880039d0 2023-10-30T12:28:52.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba50c5640 1 --2- 172.21.15.125:0/584044531 >> v2:172.21.15.125:3300/0 conn(0x7f3ba00c4430 0x7f3ba00cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:52.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba58c6640 1 --2- 172.21.15.125:0/584044531 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b880039d0 0x7f3b88003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba50c5640 1 --2- 172.21.15.125:0/584044531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b880047d0 0x7f3b88004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba58c6640 1 -- 172.21.15.125:0/584044531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b880047d0 msgr2=0x7f3b88004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba58c6640 1 --2- 172.21.15.125:0/584044531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b880047d0 0x7f3b88004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba58c6640 1 -- 172.21.15.125:0/584044531 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b88005190 msgr2=0x7f3b8800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba58c6640 1 --2- 172.21.15.125:0/584044531 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b88005190 0x7f3b8800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba58c6640 1 -- 172.21.15.125:0/584044531 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b88003530 con 0x7f3b880039d0 2023-10-30T12:28:52.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.887+0000 7f3ba60c7640 1 -- 172.21.15.125:0/584044531 reap_dead start 2023-10-30T12:28:52.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f94305d3640 1 Processor -- start 2023-10-30T12:28:52.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f94305d3640 1 -- start start 2023-10-30T12:28:52.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f1836646640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f18300c38e0 0x7f18300c4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f529b7fe640 1 -- 172.21.15.125:0/2151436439 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a80b1640 con 0x7f52b40c6670 2023-10-30T12:28:52.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f529b7fe640 1 -- 172.21.15.125:0/2151436439 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a80b17a0 con 0x7f52b40c6670 2023-10-30T12:28:52.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f529b7fe640 1 -- 172.21.15.125:0/2151436439 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a80b1b50 con 0x7f52b40c6670 2023-10-30T12:28:52.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f1836646640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f183015a150 con 0x7f18300c46c0 2023-10-30T12:28:52.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f1836646640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18300bd000 con 0x7f183015d1f0 2023-10-30T12:28:52.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f1836646640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1830159820 con 0x7f18300c38e0 2023-10-30T12:28:52.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f52b9679640 1 -- 172.21.15.125:0/2151436439 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b41fa1a0 con 0x7f52b40c6670 2023-10-30T12:28:52.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f1834be3640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f183015d1f0 0x7f18300bc340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:36916/0 (socket says 172.21.15.125:36916) 2023-10-30T12:28:52.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f1834be3640 1 -- 172.21.15.125:0/2684415843 learned_addr learned my addr 172.21.15.125:0/2684415843 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:52.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f182ffff640 1 --2- 172.21.15.125:0/2684415843 >> v2:172.21.15.125:3300/0 conn(0x7f18300c38e0 0x7f18300c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f5c1e6ea640 1 Processor -- start 2023-10-30T12:28:52.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f94305d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94280c4430 0x7f942806f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f52b9679640 1 -- 172.21.15.125:0/2151436439 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b414b870 con 0x7f52b40c6670 2023-10-30T12:28:52.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f5c1e6ea640 1 -- start start 2023-10-30T12:28:52.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f94305d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9428153c60 0x7f942806fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.891+0000 7f3ba58c6640 1 --2- 172.21.15.125:0/584044531 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b880039d0 0x7f3b88003df0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b9800e1c0 tx=0x7f3b9803ecf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4c0b92abe7a0290 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f94305d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9428159830 0x7f94280b3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f942e36f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94280c4430 0x7f942806f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f942db6e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9428153c60 0x7f942806fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f942db6e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9428153c60 0x7f942806fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38656/0 (socket says 172.21.15.125:38656) 2023-10-30T12:28:52.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f94305d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94280b49b0 con 0x7f9428159830 2023-10-30T12:28:52.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f94305d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94280b5430 con 0x7f94280c4430 2023-10-30T12:28:52.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f94305d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f942814f930 con 0x7f9428153c60 2023-10-30T12:28:52.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f942db6e640 1 -- 172.21.15.125:0/1937310020 learned_addr learned my addr 172.21.15.125:0/1937310020 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:52.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f52b9679640 1 -- 172.21.15.125:0/2151436439 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f52b40c6150 con 0x7f52b40c6670 2023-10-30T12:28:52.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f182effd640 1 -- 172.21.15.125:0/2684415843 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994505372 0 0) 0x7f18300bd000 con 0x7f183015d1f0 2023-10-30T12:28:52.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f182effd640 1 -- 172.21.15.125:0/2684415843 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1818003530 con 0x7f183015d1f0 2023-10-30T12:28:52.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f182ffff640 1 -- 172.21.15.125:0/2684415843 >> v1:172.21.15.136:6789/0 conn(0x7f183015d1f0 legacy=0x7f18300bc340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f3ba48c4640 1 -- 172.21.15.125:0/584044531 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b980c7b00 con 0x7f3b880039d0 2023-10-30T12:28:52.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f3ba48c4640 1 -- 172.21.15.125:0/584044531 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b9800f040 con 0x7f3b880039d0 2023-10-30T12:28:52.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f3ba48c4640 1 -- 172.21.15.125:0/584044531 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b980c7e80 con 0x7f3b880039d0 2023-10-30T12:28:52.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f942eb70640 1 --2- 172.21.15.125:0/1937310020 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9428159830 0x7f94280b3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/584044531 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b880039d0 msgr2=0x7f3b88003df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f3ba7b2a640 1 --2- 172.21.15.125:0/584044531 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b880039d0 0x7f3b88003df0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b9800e1c0 tx=0x7f3b9803ecf0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f942db6e640 1 -- 172.21.15.125:0/1937310020 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94280c4430 msgr2=0x7f942806f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f942db6e640 1 --2- 172.21.15.125:0/1937310020 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94280c4430 0x7f942806f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f942db6e640 1 -- 172.21.15.125:0/1937310020 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9428159830 msgr2=0x7f94280b3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f942db6e640 1 --2- 172.21.15.125:0/1937310020 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9428159830 0x7f94280b3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f942db6e640 1 -- 172.21.15.125:0/1937310020 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94180094d0 con 0x7f9428153c60 2023-10-30T12:28:52.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/584044531 shutdown_connections 2023-10-30T12:28:52.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f3ba7b2a640 1 --2- 172.21.15.125:0/584044531 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b880039d0 0x7f3b88003df0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/584044531 >> 172.21.15.125:0/584044531 conn(0x7f3ba006bfe0 msgr2=0x7f3ba006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:52.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f182ffff640 1 -- 172.21.15.125:0/2684415843 >> v1:172.21.15.125:6789/0 conn(0x7f18300c46c0 legacy=0x7f183015cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f182ffff640 1 -- 172.21.15.125:0/2684415843 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1830159b00 con 0x7f18300c38e0 2023-10-30T12:28:52.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f942db6e640 1 --2- 172.21.15.125:0/1937310020 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9428153c60 0x7f942806fdb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f942400e8e0 tx=0x7f942400eda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f941effd640 1 -- 172.21.15.125:0/1937310020 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942400ca00 con 0x7f9428153c60 2023-10-30T12:28:52.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f941effd640 1 -- 172.21.15.125:0/1937310020 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942400cb60 con 0x7f9428153c60 2023-10-30T12:28:52.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f941effd640 1 -- 172.21.15.125:0/1937310020 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9424013470 con 0x7f9428153c60 2023-10-30T12:28:52.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f5c1e6ea640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5c18153810 0x7f5c1815a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f5c1e6ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c18069e90 0x7f5c1806a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f5c1e6ea640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c180c6880 con 0x7f5c18070840 2023-10-30T12:28:52.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f5c1e6ea640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c1815a820 con 0x7f5c18153810 2023-10-30T12:28:52.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f5c1e6ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c180c5990 con 0x7f5c18069e90 2023-10-30T12:28:52.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f94305d3640 1 -- 172.21.15.125:0/1937310020 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9420009a70 con 0x7f9428153c60 2023-10-30T12:28:52.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f94305d3640 1 -- 172.21.15.125:0/1937310020 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942814fef0 con 0x7f9428153c60 2023-10-30T12:28:52.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/584044531 shutdown_connections 2023-10-30T12:28:52.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f94305d3640 1 -- 172.21.15.125:0/1937310020 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f94280cbf90 con 0x7f9428153c60 2023-10-30T12:28:52.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f529b7fe640 1 -- 172.21.15.125:0/2151436439 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f52a80b1cb0 con 0x7f52b40c6670 2023-10-30T12:28:52.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f529b7fe640 1 --2- 172.21.15.125:0/2151436439 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f52a00664f0 0x7f52a00689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7fe57e580640 1 Processor -- start 2023-10-30T12:28:52.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7fe57e580640 1 -- start start 2023-10-30T12:28:52.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f529b7fe640 1 -- 172.21.15.125:0/2151436439 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f52a80ef7f0 con 0x7f52b40c6670 2023-10-30T12:28:52.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f52b2ffd640 1 --2- 172.21.15.125:0/2151436439 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f52a00664f0 0x7f52a00689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f529b7fe640 1 --2- 172.21.15.125:0/2151436439 >> [v2:172.21.15.125:6816/3694357726,v1:172.21.15.125:6817/3694357726] conn(0x7f52a006f320 0x7f52a00717c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f529b7fe640 1 -- 172.21.15.125:0/2151436439 --> [v2:172.21.15.125:6816/3694357726,v1:172.21.15.125:6817/3694357726] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52a0071e90 con 0x7f52a006f320 2023-10-30T12:28:52.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f529b7fe640 1 -- 172.21.15.125:0/2151436439 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52a80ef5c0 con 0x7f52b40c6670 2023-10-30T12:28:52.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/584044531 wait complete. 2023-10-30T12:28:52.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f182ffff640 1 --2- 172.21.15.125:0/2684415843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18300c38e0 0x7f18300c4180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f18200097c0 tx=0x7f182001ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24655a546285c0e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7fa7fade4640 1 Processor -- start 2023-10-30T12:28:52.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7fa7fade4640 1 -- start start 2023-10-30T12:28:52.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f3ba7b2a640 1 Processor -- start 2023-10-30T12:28:52.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.895+0000 7f5c1cc87640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c18069e90 0x7f5c1806a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f5c1cc87640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c18069e90 0x7f5c1806a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38660/0 (socket says 172.21.15.125:38660) 2023-10-30T12:28:52.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f5c1cc87640 1 -- 172.21.15.125:0/709380910 learned_addr learned my addr 172.21.15.125:0/709380910 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:52.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f5c1cc87640 1 -- 172.21.15.125:0/709380910 >> v2:172.21.15.125:3300/0 conn(0x7f5c18153810 msgr2=0x7f5c1815a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f5c1cc87640 1 --2- 172.21.15.125:0/709380910 >> v2:172.21.15.125:3300/0 conn(0x7f5c18153810 0x7f5c1815a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f5c1cc87640 1 -- 172.21.15.125:0/709380910 >> v1:172.21.15.125:6789/0 conn(0x7f5c18070840 legacy=0x7f5c18070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f5c1cc87640 1 -- 172.21.15.125:0/709380910 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c180c5c70 con 0x7f5c18069e90 2023-10-30T12:28:52.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f52b37fe640 1 --2- 172.21.15.125:0/2151436439 >> [v2:172.21.15.125:6816/3694357726,v1:172.21.15.125:6817/3694357726] conn(0x7f52a006f320 0x7f52a00717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f182effd640 1 -- 172.21.15.125:0/2684415843 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18200a8970 con 0x7f18300c38e0 2023-10-30T12:28:52.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f182effd640 1 -- 172.21.15.125:0/2684415843 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18200a8ad0 con 0x7f18300c38e0 2023-10-30T12:28:52.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f182effd640 1 -- 172.21.15.125:0/2684415843 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18200a8e60 con 0x7f18300c38e0 2023-10-30T12:28:52.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f3ba7b2a640 1 -- start start 2023-10-30T12:28:52.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f52b2ffd640 1 --2- 172.21.15.125:0/2151436439 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f52a00664f0 0x7f52a00689b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f52ac003b90 tx=0x7f52ac0098d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f1836646640 1 -- 172.21.15.125:0/2684415843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18300c38e0 msgr2=0x7f18300c4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7f1836646640 1 --2- 172.21.15.125:0/2684415843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18300c38e0 0x7f18300c4180 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f18200097c0 tx=0x7f182001ec90 comp rx=0 tx=0).stop 2023-10-30T12:28:52.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f1836646640 1 -- 172.21.15.125:0/2684415843 shutdown_connections 2023-10-30T12:28:52.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f1836646640 1 --2- 172.21.15.125:0/2684415843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18300c38e0 0x7f18300c4180 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f1836646640 1 -- 172.21.15.125:0/2684415843 >> 172.21.15.125:0/2684415843 conn(0x7f18300cac50 msgr2=0x7f18300cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:52.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7fa7fade4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa7ec0d1760 0x7fa7ec0dade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fa7f95e1640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa7ec00e400 0x7fa7ec045280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54086/0 (socket says 172.21.15.125:54086) 2023-10-30T12:28:52.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fa7f95e1640 1 -- 172.21.15.125:0/3345919528 learned_addr learned my addr 172.21.15.125:0/3345919528 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:52.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fa7f9de2640 1 --2- 172.21.15.125:0/3345919528 >> v2:172.21.15.136:3300/0 conn(0x7fa7ec0d1760 0x7fa7ec0dade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fa7fade4640 1 -- 172.21.15.125:0/3345919528 --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ec007400 con 0x7fa7ec00e400 2023-10-30T12:28:52.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fa7fade4640 1 -- 172.21.15.125:0/3345919528 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7ec006530 con 0x7fa7ec005ff0 2023-10-30T12:28:52.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fa7fade4640 1 -- 172.21.15.125:0/3345919528 --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7ec007890 con 0x7fa7ec0d1760 2023-10-30T12:28:52.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7fe57e580640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe57806a0f0 0x7fe57806a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7fe57e580640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe57806aa10 0x7fe5780718c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7fe57e580640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe578164060 con 0x7fe57808b5a0 2023-10-30T12:28:52.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7fe57e580640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe57806fbb0 con 0x7fe57806a0f0 2023-10-30T12:28:52.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.899+0000 7fe57e580640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5781670b0 con 0x7fe57806aa10 2023-10-30T12:28:52.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fe577fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe57806a0f0 0x7fe57806a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fe577fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe57806a0f0 0x7fe57806a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:32986/0 (socket says 172.21.15.125:32986) 2023-10-30T12:28:52.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fe577fff640 1 -- 172.21.15.125:0/847636659 learned_addr learned my addr 172.21.15.125:0/847636659 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:52.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fe5777fe640 1 --2- 172.21.15.125:0/847636659 >> v2:172.21.15.136:3300/0 conn(0x7fe57806aa10 0x7fe5780718c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fe576ffd640 1 -- 172.21.15.125:0/847636659 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685474382 0 0) 0x7fe578164060 con 0x7fe57808b5a0 2023-10-30T12:28:52.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fe576ffd640 1 -- 172.21.15.125:0/847636659 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe560003530 con 0x7fe57808b5a0 2023-10-30T12:28:52.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f941effd640 1 -- 172.21.15.125:0/1937310020 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94240135d0 con 0x7f9428153c60 2023-10-30T12:28:52.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba7b2a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b880039d0 0x7f3ba0153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba7b2a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ba01582c0 0x7f3ba0156cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba7b2a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3ba00cd0b0 0x7f3ba013b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f1836646640 1 -- 172.21.15.125:0/2684415843 shutdown_connections 2023-10-30T12:28:52.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fe577fff640 1 -- 172.21.15.125:0/847636659 >> v2:172.21.15.136:3300/0 conn(0x7fe57806aa10 msgr2=0x7fe5780718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fe577fff640 1 --2- 172.21.15.125:0/847636659 >> v2:172.21.15.136:3300/0 conn(0x7fe57806aa10 0x7fe5780718c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fe577fff640 1 -- 172.21.15.125:0/847636659 >> v1:172.21.15.136:6789/0 conn(0x7fe57808b5a0 legacy=0x7fe578169b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fe577fff640 1 -- 172.21.15.125:0/847636659 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe578167390 con 0x7fe57806a0f0 2023-10-30T12:28:52.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba50c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ba01582c0 0x7f3ba0156cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba50c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ba01582c0 0x7f3ba0156cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38804/0 (socket says 172.21.15.125:38804) 2023-10-30T12:28:52.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba50c5640 1 -- 172.21.15.125:0/3245473633 learned_addr learned my addr 172.21.15.125:0/3245473633 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:52.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/3245473633 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00cde00 con 0x7f3b880039d0 2023-10-30T12:28:52.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f1836646640 1 -- 172.21.15.125:0/2684415843 wait complete. 2023-10-30T12:28:52.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f941effd640 1 --2- 172.21.15.125:0/1937310020 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f940c0664a0 0x7f940c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f941effd640 1 -- 172.21.15.125:0/1937310020 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94240849e0 con 0x7f9428153c60 2023-10-30T12:28:52.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f942e36f640 1 --2- 172.21.15.125:0/1937310020 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f940c0664a0 0x7f940c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f941effd640 1 --2- 172.21.15.125:0/1937310020 >> [v2:172.21.15.125:6800/1302652123,v1:172.21.15.125:6801/1302652123] conn(0x7f940c06e6c0 0x7f940c070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f941effd640 1 -- 172.21.15.125:0/1937310020 --> [v2:172.21.15.125:6800/1302652123,v1:172.21.15.125:6801/1302652123] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f940c071190 con 0x7f940c06e6c0 2023-10-30T12:28:52.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f941effd640 1 -- 172.21.15.125:0/1937310020 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9424084d90 con 0x7f9428153c60 2023-10-30T12:28:52.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f52b37fe640 1 --2- 172.21.15.125:0/2151436439 >> [v2:172.21.15.125:6816/3694357726,v1:172.21.15.125:6817/3694357726] conn(0x7f52a006f320 0x7f52a00717c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/3245473633 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba0152370 con 0x7f3ba01582c0 2023-10-30T12:28:52.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba60c7640 1 --2- 172.21.15.125:0/3245473633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3ba00cd0b0 0x7f3ba013b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/3245473633 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ba00cd530 con 0x7f3ba00cd0b0 2023-10-30T12:28:52.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fa7f9de2640 1 -- 172.21.15.125:0/3345919528 >> v1:172.21.15.136:6789/0 conn(0x7fa7ec005ff0 legacy=0x7fa7ec04b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fa7f9de2640 1 -- 172.21.15.125:0/3345919528 >> v1:172.21.15.125:6789/0 conn(0x7fa7ec00e400 legacy=0x7fa7ec045280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7fa7f9de2640 1 -- 172.21.15.125:0/3345919528 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7ec0067f0 con 0x7fa7ec0d1760 2023-10-30T12:28:52.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f942e36f640 1 --2- 172.21.15.125:0/1937310020 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f940c0664a0 0x7f940c068960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9418009b90 tx=0x7f9418008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f942eb70640 1 --2- 172.21.15.125:0/1937310020 >> [v2:172.21.15.125:6800/1302652123,v1:172.21.15.125:6801/1302652123] conn(0x7f940c06e6c0 0x7f940c070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba58c6640 1 --2- 172.21.15.125:0/3245473633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b880039d0 0x7f3ba0153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba60c7640 1 -- 172.21.15.125:0/3245473633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ba01582c0 msgr2=0x7f3ba0156cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba60c7640 1 --2- 172.21.15.125:0/3245473633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3ba01582c0 0x7f3ba0156cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba60c7640 1 -- 172.21.15.125:0/3245473633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b880039d0 msgr2=0x7f3ba0153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba60c7640 1 --2- 172.21.15.125:0/3245473633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b880039d0 0x7f3ba0153140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f3ba60c7640 1 -- 172.21.15.125:0/3245473633 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba0152d10 con 0x7f3ba00cd0b0 2023-10-30T12:28:52.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.903+0000 7f5c1cc87640 1 --2- 172.21.15.125:0/709380910 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c18069e90 0x7f5c1806a320 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5c10009e50 tx=0x7f5c100279f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8a1e316660041459 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f5c177fe640 1 -- 172.21.15.125:0/709380910 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c100b15a0 con 0x7f5c18069e90 2023-10-30T12:28:52.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f5c177fe640 1 -- 172.21.15.125:0/709380910 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c100b1700 con 0x7f5c18069e90 2023-10-30T12:28:52.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f5c177fe640 1 -- 172.21.15.125:0/709380910 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c100b1a90 con 0x7f5c18069e90 2023-10-30T12:28:52.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f3ba60c7640 1 --2- 172.21.15.125:0/3245473633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3ba00cd0b0 0x7f3ba013b4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3ba008d1c0 tx=0x7f3b9000c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f1836646640 1 Processor -- start 2023-10-30T12:28:52.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f1836646640 1 -- start start 2023-10-30T12:28:52.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f3b9e7fc640 1 -- 172.21.15.125:0/3245473633 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b900074e0 con 0x7f3ba00cd0b0 2023-10-30T12:28:52.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f529b7fe640 1 -- 172.21.15.125:0/2151436439 <== osd.1 v2:172.21.15.125:6816/3694357726 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f52a0071e90 con 0x7f52a006f320 2023-10-30T12:28:52.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f3b9e7fc640 1 -- 172.21.15.125:0/3245473633 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b90007640 con 0x7f3ba00cd0b0 2023-10-30T12:28:52.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f3b9e7fc640 1 -- 172.21.15.125:0/3245473633 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b90005a10 con 0x7f3ba00cd0b0 2023-10-30T12:28:52.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/3245473633 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ba0152040 con 0x7f3ba00cd0b0 2023-10-30T12:28:52.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/3245473633 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ba0152750 con 0x7f3ba00cd0b0 2023-10-30T12:28:52.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f1836646640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18300c38e0 0x7f183014ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7efdc3fff640 1 Processor -- start 2023-10-30T12:28:52.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f1836646640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f18300c46c0 0x7f18301494a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f1836646640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f183015d1f0 0x7f18301499e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7efdc3fff640 1 -- start start 2023-10-30T12:28:52.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f942eb70640 1 --2- 172.21.15.125:0/1937310020 >> [v2:172.21.15.125:6800/1302652123,v1:172.21.15.125:6801/1302652123] conn(0x7f940c06e6c0 0x7f940c070ae0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/3245473633 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3ba00cb9d0 con 0x7f3ba00cd0b0 2023-10-30T12:28:52.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7fa7f9de2640 1 --2- 172.21.15.125:0/3345919528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7ec0d1760 0x7fa7ec0dade0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa7f00097f0 tx=0x7fa7f001ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88b1f036915dab03 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f182ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18300c38e0 0x7f183014ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f1836646640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f183015aef0 con 0x7f183015d1f0 2023-10-30T12:28:52.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f1836646640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18301ef010 con 0x7f18300c38e0 2023-10-30T12:28:52.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f1836646640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18301ef2f0 con 0x7f18300c46c0 2023-10-30T12:28:52.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f182ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18300c38e0 0x7f183014ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38808/0 (socket says 172.21.15.125:38808) 2023-10-30T12:28:52.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f182ffff640 1 -- 172.21.15.125:0/3578200554 learned_addr learned my addr 172.21.15.125:0/3578200554 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:52.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f182f7fe640 1 --2- 172.21.15.125:0/3578200554 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f18300c46c0 0x7f18301494a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7fa7f8de0640 1 -- 172.21.15.125:0/3345919528 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f00a8930 con 0x7fa7ec0d1760 2023-10-30T12:28:52.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7fa7f8de0640 1 -- 172.21.15.125:0/3345919528 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f00a8a90 con 0x7fa7ec0d1760 2023-10-30T12:28:52.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7fa7f8de0640 1 -- 172.21.15.125:0/3345919528 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f00a8e20 con 0x7fa7ec0d1760 2023-10-30T12:28:52.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7fe577fff640 1 --2- 172.21.15.125:0/847636659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe57806a0f0 0x7fe57806a4d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe57808bcf0 tx=0x7fe56801c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca742ffffa415b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f1834be3640 1 --2- 172.21.15.125:0/3578200554 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f183015d1f0 0x7f18301499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f182f7fe640 1 -- 172.21.15.125:0/3578200554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18300c38e0 msgr2=0x7f183014ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f182f7fe640 1 --2- 172.21.15.125:0/3578200554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18300c38e0 0x7f183014ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f182f7fe640 1 -- 172.21.15.125:0/3578200554 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f183015d1f0 msgr2=0x7f18301499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f182f7fe640 1 --2- 172.21.15.125:0/3578200554 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f183015d1f0 0x7f18301499e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7f182f7fe640 1 -- 172.21.15.125:0/3578200554 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1820009420 con 0x7f18300c46c0 2023-10-30T12:28:52.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7efdc3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efdbc04f420 0x7efdbc0dec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7efdc3fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdbc050a30 0x7efdbc0df1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f941effd640 1 -- 172.21.15.125:0/1937310020 <== osd.0 v2:172.21.15.125:6800/1302652123 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f940c071190 con 0x7f940c06e6c0 2023-10-30T12:28:52.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f182f7fe640 1 --2- 172.21.15.125:0/3578200554 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f18300c46c0 0x7f18301494a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1824014bf0 tx=0x7f1824012430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f182cff9640 1 -- 172.21.15.125:0/3578200554 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824012cf0 con 0x7f18300c46c0 2023-10-30T12:28:52.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fa7fade4640 1 -- 172.21.15.125:0/3345919528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7ec0d1760 msgr2=0x7fa7ec0dade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fa7fade4640 1 --2- 172.21.15.125:0/3345919528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7ec0d1760 0x7fa7ec0dade0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa7f00097f0 tx=0x7fa7f001ec50 comp rx=0 tx=0).stop 2023-10-30T12:28:52.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/709380910 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c18069e90 msgr2=0x7f5c1806a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f5c1e6ea640 1 --2- 172.21.15.125:0/709380910 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c18069e90 0x7f5c1806a320 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5c10009e50 tx=0x7f5c100279f0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7efdc3fff640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdbc04af40 con 0x7efdbc0d8e50 2023-10-30T12:28:52.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7efdc3fff640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdbc04f8f0 con 0x7efdbc04f420 2023-10-30T12:28:52.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.907+0000 7efdc3fff640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc049a70 con 0x7efdbc050a30 2023-10-30T12:28:52.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7efdc37fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdbc050a30 0x7efdbc0df1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7efdc27fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efdbc04f420 0x7efdbc0dec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7efdc37fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdbc050a30 0x7efdbc0df1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38680/0 (socket says 172.21.15.125:38680) 2023-10-30T12:28:52.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7efdc37fe640 1 -- 172.21.15.125:0/1211381332 learned_addr learned my addr 172.21.15.125:0/1211381332 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:52.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f182cff9640 1 -- 172.21.15.125:0/3578200554 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1824012e50 con 0x7f18300c46c0 2023-10-30T12:28:52.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f182cff9640 1 -- 172.21.15.125:0/3578200554 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824005310 con 0x7f18300c46c0 2023-10-30T12:28:52.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fa7fade4640 1 -- 172.21.15.125:0/3345919528 shutdown_connections 2023-10-30T12:28:52.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fa7fade4640 1 --2- 172.21.15.125:0/3345919528 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7ec0d1760 0x7fa7ec0dade0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fa7fade4640 1 -- 172.21.15.125:0/3345919528 >> 172.21.15.125:0/3345919528 conn(0x7fa7ec0502e0 msgr2=0x7fa7ec0506f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:52.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f1836646640 1 -- 172.21.15.125:0/3578200554 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183015a830 con 0x7f18300c46c0 2023-10-30T12:28:52.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fa7fade4640 1 -- 172.21.15.125:0/3345919528 shutdown_connections 2023-10-30T12:28:52.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fa7fade4640 1 -- 172.21.15.125:0/3345919528 wait complete. 2023-10-30T12:28:52.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/709380910 shutdown_connections 2023-10-30T12:28:52.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f5c1e6ea640 1 --2- 172.21.15.125:0/709380910 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c18069e90 0x7f5c1806a320 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f5c1e6ea640 1 --2- 172.21.15.125:0/709380910 >> v2:172.21.15.125:3300/0 conn(0x7f5c18153810 0x7f5c1815a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f1836646640 1 -- 172.21.15.125:0/3578200554 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1830158c60 con 0x7f18300c46c0 2023-10-30T12:28:52.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/709380910 >> 172.21.15.125:0/709380910 conn(0x7f5c180c7910 msgr2=0x7f5c180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:52.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fe576ffd640 1 -- 172.21.15.125:0/847636659 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5680ab600 con 0x7fe57806a0f0 2023-10-30T12:28:52.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fe576ffd640 1 -- 172.21.15.125:0/847636659 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5680a7050 con 0x7fe57806a0f0 2023-10-30T12:28:52.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fe576ffd640 1 -- 172.21.15.125:0/847636659 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5680ab980 con 0x7fe57806a0f0 2023-10-30T12:28:52.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/709380910 shutdown_connections 2023-10-30T12:28:52.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7efdc37fe640 1 -- 172.21.15.125:0/1211381332 >> v2:172.21.15.125:3300/0 conn(0x7efdbc04f420 msgr2=0x7efdbc0dec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7efdc37fe640 1 --2- 172.21.15.125:0/1211381332 >> v2:172.21.15.125:3300/0 conn(0x7efdbc04f420 0x7efdbc0dec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7efdc37fe640 1 -- 172.21.15.125:0/1211381332 >> v1:172.21.15.125:6789/0 conn(0x7efdbc0d8e50 legacy=0x7efdbc04a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:52.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f1836646640 1 -- 172.21.15.125:0/3578200554 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f183015a6b0 con 0x7f18300c46c0 2023-10-30T12:28:52.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fe57e580640 1 -- 172.21.15.125:0/847636659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe57806a0f0 msgr2=0x7fe57806a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fe57e580640 1 --2- 172.21.15.125:0/847636659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe57806a0f0 0x7fe57806a4d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe57808bcf0 tx=0x7fe56801c4f0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/709380910 wait complete. 2023-10-30T12:28:52.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7efdc37fe640 1 -- 172.21.15.125:0/1211381332 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc049d50 con 0x7efdbc050a30 2023-10-30T12:28:52.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fe57e580640 1 -- 172.21.15.125:0/847636659 shutdown_connections 2023-10-30T12:28:52.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fe57e580640 1 --2- 172.21.15.125:0/847636659 >> v2:172.21.15.136:3300/0 conn(0x7fe57806aa10 0x7fe5780718c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fe57e580640 1 --2- 172.21.15.125:0/847636659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe57806a0f0 0x7fe57806a4d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fe57e580640 1 -- 172.21.15.125:0/847636659 >> 172.21.15.125:0/847636659 conn(0x7fe5780c7910 msgr2=0x7fe5780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:52.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fe57e580640 1 -- 172.21.15.125:0/847636659 shutdown_connections 2023-10-30T12:28:52.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f182cff9640 1 -- 172.21.15.125:0/3578200554 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1824003600 con 0x7f18300c46c0 2023-10-30T12:28:52.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7f3b9e7fc640 1 -- 172.21.15.125:0/3245473633 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3b90005b70 con 0x7f3ba00cd0b0 2023-10-30T12:28:52.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.911+0000 7fa7fade4640 1 Processor -- start 2023-10-30T12:28:52.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fa7fade4640 1 -- start start 2023-10-30T12:28:52.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7f182cff9640 1 --2- 172.21.15.125:0/3578200554 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f181c0664a0 0x7f181c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7f182cff9640 1 -- 172.21.15.125:0/3578200554 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f182408cac0 con 0x7f18300c46c0 2023-10-30T12:28:52.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7f5c1e6ea640 1 Processor -- start 2023-10-30T12:28:52.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fe57e580640 1 -- 172.21.15.125:0/847636659 wait complete. 2023-10-30T12:28:52.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7f182cff9640 1 --2- 172.21.15.125:0/3578200554 >> [v2:172.21.15.125:6824/3730905694,v1:172.21.15.125:6825/3730905694] conn(0x7f181c06e640 0x7f181c070a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7f182cff9640 1 -- 172.21.15.125:0/3578200554 --> [v2:172.21.15.125:6824/3730905694,v1:172.21.15.125:6825/3730905694] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f181c071110 con 0x7f181c06e640 2023-10-30T12:28:52.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7f182cff9640 1 -- 172.21.15.125:0/3578200554 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f182408cf10 con 0x7f18300c46c0 2023-10-30T12:28:52.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7f182ffff640 1 --2- 172.21.15.125:0/3578200554 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f181c0664a0 0x7f181c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7f1834be3640 1 --2- 172.21.15.125:0/3578200554 >> [v2:172.21.15.125:6824/3730905694,v1:172.21.15.125:6825/3730905694] conn(0x7f181c06e640 0x7f181c070a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fa7fade4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7ec005ff0 0x7fa7ec0dc0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7f1834be3640 1 --2- 172.21.15.125:0/3578200554 >> [v2:172.21.15.125:6824/3730905694,v1:172.21.15.125:6825/3730905694] conn(0x7f181c06e640 0x7f181c070a60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fa7fade4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7ec00e400 0x7fa7ec0464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fa7fade4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7ec0d1760 0x7fa7ec0c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fa7f9de2640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7ec005ff0 0x7fa7ec0dc0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fa7fa5e3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7ec0d1760 0x7fa7ec0c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fa7fa5e3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7ec0d1760 0x7fa7ec0c6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38814/0 (socket says 172.21.15.125:38814) 2023-10-30T12:28:52.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fa7fa5e3640 1 -- 172.21.15.125:0/1415221079 learned_addr learned my addr 172.21.15.125:0/1415221079 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:52.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fa7f9de2640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7ec005ff0 0x7fa7ec0dc0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38684/0 (socket says 172.21.15.125:38684) 2023-10-30T12:28:52.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fa7fade4640 1 -- 172.21.15.125:0/1415221079 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec048e90 con 0x7fa7ec00e400 2023-10-30T12:28:52.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fa7fade4640 1 -- 172.21.15.125:0/1415221079 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec0495c0 con 0x7fa7ec0d1760 2023-10-30T12:28:52.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fa7fade4640 1 -- 172.21.15.125:0/1415221079 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ec1757c0 con 0x7fa7ec005ff0 2023-10-30T12:28:52.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fa7f95e1640 1 --2- 172.21.15.125:0/1415221079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7ec00e400 0x7fa7ec0464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7f5c1e6ea640 1 -- start start 2023-10-30T12:28:52.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fe57e580640 1 Processor -- start 2023-10-30T12:28:52.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7fe57e580640 1 -- start start 2023-10-30T12:28:52.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7efdc37fe640 1 --2- 172.21.15.125:0/1211381332 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdbc050a30 0x7efdbc0df1d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efdc4063b60 tx=0x7efdc4099c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab202c986e21aa4e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7efdc1ffb640 1 -- 172.21.15.125:0/1211381332 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc4123920 con 0x7efdbc050a30 2023-10-30T12:28:52.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7efdc1ffb640 1 -- 172.21.15.125:0/1211381332 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdc4123ac0 con 0x7efdbc050a30 2023-10-30T12:28:52.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.915+0000 7efdc1ffb640 1 -- 172.21.15.125:0/1211381332 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc412b440 con 0x7efdbc050a30 2023-10-30T12:28:52.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fa7fa5e3640 1 -- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7ec005ff0 msgr2=0x7fa7ec0dc0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fa7fa5e3640 1 --2- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7ec005ff0 0x7fa7ec0dc0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fa7fa5e3640 1 -- 172.21.15.125:0/1415221079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7ec00e400 msgr2=0x7fa7ec0464a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fa7fa5e3640 1 --2- 172.21.15.125:0/1415221079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7ec00e400 0x7fa7ec0464a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fa7fa5e3640 1 -- 172.21.15.125:0/1415221079 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f0009420 con 0x7fa7ec0d1760 2023-10-30T12:28:52.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7f182ffff640 1 --2- 172.21.15.125:0/3578200554 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f181c0664a0 0x7f181c068960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f182001ecd0 tx=0x7f18200aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7f3b9e7fc640 1 --2- 172.21.15.125:0/3245473633 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b8c0664a0 0x7f3b8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7efdc3fff640 1 -- 172.21.15.125:0/1211381332 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdbc050a30 msgr2=0x7efdbc0df1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:52.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fa7fa5e3640 1 --2- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7ec0d1760 0x7fa7ec0c6a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa7f40683f0 tx=0x7fa7f4066e60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:52.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7f3b9e7fc640 1 -- 172.21.15.125:0/3245473633 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b90083a20 con 0x7f3ba00cd0b0 2023-10-30T12:28:52.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe57e580640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe57806a0f0 0x7fe578161820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe57e580640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe57806aa10 0x7fe5780c10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7efdc3fff640 1 --2- 172.21.15.125:0/1211381332 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdbc050a30 0x7efdbc0df1d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efdc4063b60 tx=0x7efdc4099c40 comp rx=0 tx=0).stop 2023-10-30T12:28:52.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7f182cff9640 1 -- 172.21.15.125:0/3578200554 <== osd.3 v2:172.21.15.125:6824/3730905694 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f181c071110 con 0x7f181c06e640 2023-10-30T12:28:52.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fa7ea7fc640 1 -- 172.21.15.125:0/1415221079 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4072770 con 0x7fa7ec0d1760 2023-10-30T12:28:52.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7efdc3fff640 1 -- 172.21.15.125:0/1211381332 shutdown_connections 2023-10-30T12:28:52.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7efdc3fff640 1 --2- 172.21.15.125:0/1211381332 >> v2:172.21.15.125:3300/0 conn(0x7efdbc04f420 0x7efdbc0dec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7efdc3fff640 1 --2- 172.21.15.125:0/1211381332 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdbc050a30 0x7efdbc0df1d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:52.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7efdc3fff640 1 -- 172.21.15.125:0/1211381332 >> 172.21.15.125:0/1211381332 conn(0x7efdbc051040 msgr2=0x7efdbc051450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:52.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7efdc3fff640 1 -- 172.21.15.125:0/1211381332 shutdown_connections 2023-10-30T12:28:52.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7efdc3fff640 1 -- 172.21.15.125:0/1211381332 wait complete. 2023-10-30T12:28:52.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe57e580640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe57808b5a0 0x7fe5781688b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:52.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe577fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe57806a0f0 0x7fe578161820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:52.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe577fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe57806a0f0 0x7fe578161820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38818/0 (socket says 172.21.15.125:38818) 2023-10-30T12:28:53.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe577fff640 1 -- 172.21.15.125:0/1157019643 learned_addr learned my addr 172.21.15.125:0/1157019643 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe57e580640 1 -- 172.21.15.125:0/1157019643 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe578163bb0 con 0x7fe57806aa10 2023-10-30T12:28:53.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe57e580640 1 -- 172.21.15.125:0/1157019643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5782044b0 con 0x7fe57806a0f0 2023-10-30T12:28:53.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe57e580640 1 -- 172.21.15.125:0/1157019643 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5782035e0 con 0x7fe57808b5a0 2023-10-30T12:28:53.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7f3b9e7fc640 1 --2- 172.21.15.125:0/3245473633 >> [v2:172.21.15.125:6808/456280330,v1:172.21.15.125:6809/456280330] conn(0x7f3b8c06e6c0 0x7f3b8c070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fa7ea7fc640 1 -- 172.21.15.125:0/1415221079 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f40728f0 con 0x7fa7ec0d1760 2023-10-30T12:28:53.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fa7ea7fc640 1 -- 172.21.15.125:0/1415221079 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4072ca0 con 0x7fa7ec0d1760 2023-10-30T12:28:53.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe5777fe640 1 --2- 172.21.15.125:0/1157019643 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe57806aa10 0x7fe5780c10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe57cb1d640 1 --2- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe57808b5a0 0x7fe5781688b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7f3b9e7fc640 1 -- 172.21.15.125:0/3245473633 --> [v2:172.21.15.125:6808/456280330,v1:172.21.15.125:6809/456280330] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3b8c071190 con 0x7f3b8c06e6c0 2023-10-30T12:28:53.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fa7fade4640 1 -- 172.21.15.125:0/1415221079 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7ec0488d0 con 0x7fa7ec0d1760 2023-10-30T12:28:53.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fa7fade4640 1 -- 172.21.15.125:0/1415221079 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7ec0491e0 con 0x7fa7ec0d1760 2023-10-30T12:28:53.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7f3b9e7fc640 1 -- 172.21.15.125:0/3245473633 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b90083dd0 con 0x7f3ba00cd0b0 2023-10-30T12:28:53.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe577fff640 1 -- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe57808b5a0 msgr2=0x7fe5781688b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe577fff640 1 --2- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe57808b5a0 0x7fe5781688b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe577fff640 1 -- 172.21.15.125:0/1157019643 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe57806aa10 msgr2=0x7fe5780c10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe577fff640 1 --2- 172.21.15.125:0/1157019643 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe57806aa10 0x7fe5780c10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fe577fff640 1 -- 172.21.15.125:0/1157019643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5680094d0 con 0x7fe57806a0f0 2023-10-30T12:28:53.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7f3ba58c6640 1 --2- 172.21.15.125:0/3245473633 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b8c0664a0 0x7f3b8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7f5c1e6ea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c18069e90 0x7f5c181e1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7fa7fade4640 1 -- 172.21.15.125:0/1415221079 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa7ec006db0 con 0x7fa7ec0d1760 2023-10-30T12:28:53.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7f5c1e6ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c18070840 0x7f5c18148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.919+0000 7f5c1e6ea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c18153810 0x7f5c181490b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7f5c17fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c18069e90 0x7f5c181e1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7f5c17fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c18069e90 0x7f5c181e1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38822/0 (socket says 172.21.15.125:38822) 2023-10-30T12:28:53.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7f5c0f7fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c18070840 0x7f5c18148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7f5c0f7fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c18070840 0x7f5c18148b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38710/0 (socket says 172.21.15.125:38710) 2023-10-30T12:28:53.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7efdc3fff640 1 Processor -- start 2023-10-30T12:28:53.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7f5c1cc87640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c18153810 0x7f5c181490b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7f5c1cc87640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c18153810 0x7f5c181490b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33062/0 (socket says 172.21.15.125:33062) 2023-10-30T12:28:53.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7fe577fff640 1 --2- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe57806a0f0 0x7fe578161820 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe56801a120 tx=0x7fe568019290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7f3ba50c5640 1 --2- 172.21.15.125:0/3245473633 >> [v2:172.21.15.125:6808/456280330,v1:172.21.15.125:6809/456280330] conn(0x7f3b8c06e6c0 0x7f3b8c070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7f5c1e6ea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c181518f0 con 0x7f5c18153810 2023-10-30T12:28:53.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7f5c1e6ea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c18154170 con 0x7f5c18069e90 2023-10-30T12:28:53.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7f5c1e6ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c18151c40 con 0x7f5c18070840 2023-10-30T12:28:53.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7f5c17fff640 1 -- 172.21.15.125:0/3214841482 learned_addr learned my addr 172.21.15.125:0/3214841482 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7fe574ff9640 1 -- 172.21.15.125:0/1157019643 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5680b5440 con 0x7fe57806a0f0 2023-10-30T12:28:53.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7fe57e580640 1 -- 172.21.15.125:0/1157019643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5782038a0 con 0x7fe57806a0f0 2023-10-30T12:28:53.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7efdc3fff640 1 -- start start 2023-10-30T12:28:53.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7f3ba50c5640 1 --2- 172.21.15.125:0/3245473633 >> [v2:172.21.15.125:6808/456280330,v1:172.21.15.125:6809/456280330] conn(0x7f3b8c06e6c0 0x7f3b8c070ae0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7fa7ea7fc640 1 -- 172.21.15.125:0/1415221079 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa7f409e3f0 con 0x7fa7ec0d1760 2023-10-30T12:28:53.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7fa7ea7fc640 1 --2- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa7d8066570 0x7fa7d8068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7efdc3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdbc04f420 0x7efdbc04e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7f3ba58c6640 1 --2- 172.21.15.125:0/3245473633 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b8c0664a0 0x7f3b8c068960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3b980d0000 tx=0x7f3b980c9040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7fa7ea7fc640 1 -- 172.21.15.125:0/1415221079 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7f40fde90 con 0x7fa7ec0d1760 2023-10-30T12:28:53.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7fa7ea7fc640 1 --2- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7fa7d806e790 0x7fa7d8070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7fa7ea7fc640 1 -- 172.21.15.125:0/1415221079 --> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa7d8071260 con 0x7fa7d806e790 2023-10-30T12:28:53.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7fa7ea7fc640 1 -- 172.21.15.125:0/1415221079 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7f41004b0 con 0x7fa7ec0d1760 2023-10-30T12:28:53.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7fe57e580640 1 -- 172.21.15.125:0/1157019643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe578203e60 con 0x7fe57806a0f0 2023-10-30T12:28:53.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7fe574ff9640 1 -- 172.21.15.125:0/1157019643 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5680b55a0 con 0x7fe57806a0f0 2023-10-30T12:28:53.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7fe574ff9640 1 -- 172.21.15.125:0/1157019643 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5680b5440 con 0x7fe57806a0f0 2023-10-30T12:28:53.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdbc050a30 0x7efdbc0533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc3fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdbc0d8e50 0x7efdbc053930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc2ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdbc04f420 0x7efdbc04e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc27fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdbc050a30 0x7efdbc0533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc27fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdbc050a30 0x7efdbc0533f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38828/0 (socket says 172.21.15.125:38828) 2023-10-30T12:28:53.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7fa7f9de2640 1 --2- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa7d8066570 0x7fa7d8068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7f5c0f7fe640 1 -- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c18069e90 msgr2=0x7f5c181e1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc27fc640 1 -- 172.21.15.125:0/2910377160 learned_addr learned my addr 172.21.15.125:0/2910377160 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc3fff640 1 -- 172.21.15.125:0/2910377160 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc04b950 con 0x7efdbc04f420 2023-10-30T12:28:53.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc3fff640 1 -- 172.21.15.125:0/2910377160 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc039050 con 0x7efdbc050a30 2023-10-30T12:28:53.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc3fff640 1 -- 172.21.15.125:0/2910377160 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdbc1755c0 con 0x7efdbc0d8e50 2023-10-30T12:28:53.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7f3b9e7fc640 1 -- 172.21.15.125:0/3245473633 <== osd.2 v2:172.21.15.125:6808/456280330 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f3b8c071190 con 0x7f3b8c06e6c0 2023-10-30T12:28:53.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc37fe640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdbc0d8e50 0x7efdbc053930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7fa7f95e1640 1 --2- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7fa7d806e790 0x7fa7d8070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc27fc640 1 -- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdbc0d8e50 msgr2=0x7efdbc053930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc27fc640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdbc0d8e50 0x7efdbc053930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.923+0000 7fe57e580640 1 -- 172.21.15.125:0/1157019643 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe558000fd0 con 0x7fe57806a0f0 2023-10-30T12:28:53.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7f5c0f7fe640 1 --2- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c18069e90 0x7f5c181e1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7f5c0f7fe640 1 -- 172.21.15.125:0/3214841482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c18153810 msgr2=0x7f5c181490b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7f5c0f7fe640 1 --2- 172.21.15.125:0/3214841482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c18153810 0x7f5c181490b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7f5c0f7fe640 1 -- 172.21.15.125:0/3214841482 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c10009b30 con 0x7f5c18070840 2023-10-30T12:28:53.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7f5c1cc87640 1 --2- 172.21.15.125:0/3214841482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c18153810 0x7f5c181490b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:53.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc27fc640 1 -- 172.21.15.125:0/2910377160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdbc04f420 msgr2=0x7efdbc04e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc27fc640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdbc04f420 0x7efdbc04e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc27fc640 1 -- 172.21.15.125:0/2910377160 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdbc03a480 con 0x7efdbc050a30 2023-10-30T12:28:53.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc37fe640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdbc0d8e50 0x7efdbc053930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:53.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7f5c0f7fe640 1 --2- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c18070840 0x7f5c18148b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000b440 tx=0x7f5c0000b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc2ffd640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdbc04f420 0x7efdbc04e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:53.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc27fc640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdbc050a30 0x7efdbc0533f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efdb800d5f0 tx=0x7efdb800dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7fa7f9de2640 1 --2- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa7d8066570 0x7fa7d8068a30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa7f00b2000 tx=0x7fa7f00aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7fa7f95e1640 1 --2- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7fa7d806e790 0x7fa7d8070bb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7f5c157fa640 1 -- 172.21.15.125:0/3214841482 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c00011830 con 0x7f5c18070840 2023-10-30T12:28:53.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdab7fe640 1 -- 172.21.15.125:0/2910377160 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb80125f0 con 0x7efdbc050a30 2023-10-30T12:28:53.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc3fff640 1 -- 172.21.15.125:0/2910377160 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdbc0392f0 con 0x7efdbc050a30 2023-10-30T12:28:53.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdab7fe640 1 -- 172.21.15.125:0/2910377160 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdb8012750 con 0x7efdbc050a30 2023-10-30T12:28:53.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdab7fe640 1 -- 172.21.15.125:0/2910377160 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdb8012ae0 con 0x7efdbc050a30 2023-10-30T12:28:53.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc3fff640 1 -- 172.21.15.125:0/2910377160 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdbc175880 con 0x7efdbc050a30 2023-10-30T12:28:53.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/3214841482 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c18154d30 con 0x7f5c18070840 2023-10-30T12:28:53.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/3214841482 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c18150df0 con 0x7f5c18070840 2023-10-30T12:28:53.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7f5c157fa640 1 -- 172.21.15.125:0/3214841482 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c000119b0 con 0x7f5c18070840 2023-10-30T12:28:53.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7f5c157fa640 1 -- 172.21.15.125:0/3214841482 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c00011d60 con 0x7f5c18070840 2023-10-30T12:28:53.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7efdc3fff640 1 -- 172.21.15.125:0/2910377160 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efdbc04a310 con 0x7efdbc050a30 2023-10-30T12:28:53.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.927+0000 7fe574ff9640 1 -- 172.21.15.125:0/1157019643 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe5680b5710 con 0x7fe57806a0f0 2023-10-30T12:28:53.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.931+0000 7fa7ea7fc640 1 -- 172.21.15.125:0/1415221079 <== osd.4 v2:172.21.15.136:6817/4208991530 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fa7d8071260 con 0x7fa7d806e790 2023-10-30T12:28:53.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.931+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/3214841482 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5bfc000fd0 con 0x7f5c18070840 2023-10-30T12:28:53.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.931+0000 7fe574ff9640 1 --2- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe564066530 0x7fe5640689f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.931+0000 7fe574ff9640 1 -- 172.21.15.125:0/1157019643 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe56811dab0 con 0x7fe57806a0f0 2023-10-30T12:28:53.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.931+0000 7fe574ff9640 1 --2- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:6816/3758986733,v1:172.21.15.136:6818/3758986733] conn(0x7fe56406e750 0x7fe564070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.931+0000 7fe574ff9640 1 -- 172.21.15.125:0/1157019643 --> [v2:172.21.15.136:6816/3758986733,v1:172.21.15.136:6818/3758986733] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe564071220 con 0x7fe56406e750 2023-10-30T12:28:53.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.931+0000 7fe574ff9640 1 -- 172.21.15.125:0/1157019643 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe5681222a0 con 0x7fe57806a0f0 2023-10-30T12:28:53.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.931+0000 7fe57cb1d640 1 --2- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:6816/3758986733,v1:172.21.15.136:6818/3758986733] conn(0x7fe56406e750 0x7fe564070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.931+0000 7fe5777fe640 1 --2- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe564066530 0x7fe5640689f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.931+0000 7f5c157fa640 1 -- 172.21.15.125:0/3214841482 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5c00004020 con 0x7f5c18070840 2023-10-30T12:28:53.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.931+0000 7efdab7fe640 1 -- 172.21.15.125:0/2910377160 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efdb8012c40 con 0x7efdbc050a30 2023-10-30T12:28:53.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7fe57cb1d640 1 --2- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:6816/3758986733,v1:172.21.15.136:6818/3758986733] conn(0x7fe56406e750 0x7fe564070b70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7efdab7fe640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efdb00664a0 0x7efdb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7f5c157fa640 1 --2- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5bf80664d0 0x7f5bf8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7f5c157fa640 1 -- 172.21.15.125:0/3214841482 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5c000836a0 con 0x7f5c18070840 2023-10-30T12:28:53.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7efdab7fe640 1 -- 172.21.15.125:0/2910377160 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efdb8084e90 con 0x7efdbc050a30 2023-10-30T12:28:53.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7f5c157fa640 1 --2- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:6800/1849732357,v1:172.21.15.136:6801/1849732357] conn(0x7f5bf806e6f0 0x7f5bf8070b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7fe5777fe640 1 --2- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe564066530 0x7fe5640689f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe5780ca7f0 tx=0x7fe56c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7efdc2ffd640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efdb00664a0 0x7efdb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7efdab7fe640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:6808/3315858521,v1:172.21.15.136:6809/3315858521] conn(0x7efdb006e6c0 0x7efdb0070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7efdab7fe640 1 -- 172.21.15.125:0/2910377160 --> [v2:172.21.15.136:6808/3315858521,v1:172.21.15.136:6809/3315858521] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efdb0071190 con 0x7efdb006e6c0 2023-10-30T12:28:53.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7efdab7fe640 1 -- 172.21.15.125:0/2910377160 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdb8085240 con 0x7efdbc050a30 2023-10-30T12:28:53.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7f5c17fff640 1 --2- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5bf80664d0 0x7f5bf8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7f5c157fa640 1 -- 172.21.15.125:0/3214841482 --> [v2:172.21.15.136:6800/1849732357,v1:172.21.15.136:6801/1849732357] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5bf80711c0 con 0x7f5bf806e6f0 2023-10-30T12:28:53.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7f5c157fa640 1 -- 172.21.15.125:0/3214841482 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5c00085c00 con 0x7f5c18070840 2023-10-30T12:28:53.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7efdc37fe640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:6808/3315858521,v1:172.21.15.136:6809/3315858521] conn(0x7efdb006e6c0 0x7efdb0070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7f5c1cc87640 1 --2- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:6800/1849732357,v1:172.21.15.136:6801/1849732357] conn(0x7f5bf806e6f0 0x7f5bf8070b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7f52b9679640 1 -- 172.21.15.125:0/2151436439 --> [v2:172.21.15.125:6816/3694357726,v1:172.21.15.125:6817/3694357726] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f52b40c5e70 con 0x7f52a006f320 2023-10-30T12:28:53.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7efdc2ffd640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efdb00664a0 0x7efdb0068960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efdb4010f10 tx=0x7efdb4007cc0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7fe574ff9640 1 -- 172.21.15.125:0/1157019643 <== osd.6 v2:172.21.15.136:6816/3758986733 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fe564071220 con 0x7fe56406e750 2023-10-30T12:28:53.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7efdc37fe640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:6808/3315858521,v1:172.21.15.136:6809/3315858521] conn(0x7efdb006e6c0 0x7efdb0070ae0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7f529b7fe640 1 -- 172.21.15.125:0/2151436439 <== osd.1 v2:172.21.15.125:6816/3694357726 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f52a0071e90 con 0x7f52a006f320 2023-10-30T12:28:53.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7f5c17fff640 1 --2- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5bf80664d0 0x7f5bf8068990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5c080099b0 tx=0x7f5c08009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7f52b9679640 1 -- 172.21.15.125:0/2151436439 >> [v2:172.21.15.125:6816/3694357726,v1:172.21.15.125:6817/3694357726] conn(0x7f52a006f320 msgr2=0x7f52a00717c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.939+0000 7f52b9679640 1 --2- 172.21.15.125:0/2151436439 >> [v2:172.21.15.125:6816/3694357726,v1:172.21.15.125:6817/3694357726] conn(0x7f52a006f320 0x7f52a00717c0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.935+0000 7f5c1cc87640 1 --2- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:6800/1849732357,v1:172.21.15.136:6801/1849732357] conn(0x7f5bf806e6f0 0x7f5bf8070b10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.939+0000 7f52b9679640 1 -- 172.21.15.125:0/2151436439 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f52a00664f0 msgr2=0x7f52a00689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.939+0000 7f52b9679640 1 --2- 172.21.15.125:0/2151436439 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f52a00664f0 0x7f52a00689b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f52ac003b90 tx=0x7f52ac0098d0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.939+0000 7f52b9679640 1 -- 172.21.15.125:0/2151436439 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b40c6670 msgr2=0x7f52b41f97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.939+0000 7f52b9679640 1 --2- 172.21.15.125:0/2151436439 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b40c6670 0x7f52b41f97f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f52a80211f0 tx=0x7f52a801d6f0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.939+0000 7f52b37fe640 1 -- 172.21.15.125:0/2151436439 reap_dead start 2023-10-30T12:28:53.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.939+0000 7f52b9679640 1 -- 172.21.15.125:0/2151436439 shutdown_connections 2023-10-30T12:28:53.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.939+0000 7f52b9679640 1 -- 172.21.15.125:0/2151436439 >> 172.21.15.125:0/2151436439 conn(0x7f52b40c7910 msgr2=0x7f52b40c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.939+0000 7f52b9679640 1 -- 172.21.15.125:0/2151436439 shutdown_connections 2023-10-30T12:28:53.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.939+0000 7f52b9679640 1 -- 172.21.15.125:0/2151436439 wait complete. 2023-10-30T12:28:53.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.939+0000 7efdab7fe640 1 -- 172.21.15.125:0/2910377160 <== osd.7 v2:172.21.15.136:6808/3315858521 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7efdb0071190 con 0x7efdb006e6c0 2023-10-30T12:28:53.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.939+0000 7f5c157fa640 1 -- 172.21.15.125:0/3214841482 <== osd.5 v2:172.21.15.136:6800/1849732357 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f5bf80711c0 con 0x7f5bf806e6f0 2023-10-30T12:28:53.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.939+0000 7f94305d3640 1 -- 172.21.15.125:0/1937310020 --> [v2:172.21.15.125:6800/1302652123,v1:172.21.15.125:6801/1302652123] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f94280cbcb0 con 0x7f940c06e6c0 2023-10-30T12:28:53.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.939+0000 7f1836646640 1 -- 172.21.15.125:0/3578200554 --> [v2:172.21.15.125:6824/3730905694,v1:172.21.15.125:6825/3730905694] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f18301594f0 con 0x7f181c06e640 2023-10-30T12:28:53.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f182cff9640 1 -- 172.21.15.125:0/3578200554 <== osd.3 v2:172.21.15.125:6824/3730905694 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1828001c50 con 0x7f181c06e640 2023-10-30T12:28:53.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f941effd640 1 -- 172.21.15.125:0/1937310020 <== osd.0 v2:172.21.15.125:6800/1302652123 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f94200a5b50 con 0x7f940c06e6c0 2023-10-30T12:28:53.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f94305d3640 1 -- 172.21.15.125:0/1937310020 >> [v2:172.21.15.125:6800/1302652123,v1:172.21.15.125:6801/1302652123] conn(0x7f940c06e6c0 msgr2=0x7f940c070ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f94305d3640 1 --2- 172.21.15.125:0/1937310020 >> [v2:172.21.15.125:6800/1302652123,v1:172.21.15.125:6801/1302652123] conn(0x7f940c06e6c0 0x7f940c070ae0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f94305d3640 1 -- 172.21.15.125:0/1937310020 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f940c0664a0 msgr2=0x7f940c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f94305d3640 1 --2- 172.21.15.125:0/1937310020 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f940c0664a0 0x7f940c068960 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f9418009b90 tx=0x7f9418008040 comp rx=0 tx=0).stop 2023-10-30T12:28:53.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f94305d3640 1 -- 172.21.15.125:0/1937310020 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9428153c60 msgr2=0x7f942806fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f94305d3640 1 --2- 172.21.15.125:0/1937310020 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9428153c60 0x7f942806fdb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f942400e8e0 tx=0x7f942400eda0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f942eb70640 1 -- 172.21.15.125:0/1937310020 reap_dead start 2023-10-30T12:28:53.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f94305d3640 1 -- 172.21.15.125:0/1937310020 shutdown_connections 2023-10-30T12:28:53.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f94305d3640 1 -- 172.21.15.125:0/1937310020 >> 172.21.15.125:0/1937310020 conn(0x7f942806bfe0 msgr2=0x7f94280b2080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f1836646640 1 -- 172.21.15.125:0/3578200554 >> [v2:172.21.15.125:6824/3730905694,v1:172.21.15.125:6825/3730905694] conn(0x7f181c06e640 msgr2=0x7f181c070a60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f1836646640 1 --2- 172.21.15.125:0/3578200554 >> [v2:172.21.15.125:6824/3730905694,v1:172.21.15.125:6825/3730905694] conn(0x7f181c06e640 0x7f181c070a60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f94305d3640 1 -- 172.21.15.125:0/1937310020 shutdown_connections 2023-10-30T12:28:53.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f1836646640 1 -- 172.21.15.125:0/3578200554 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f181c0664a0 msgr2=0x7f181c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f1836646640 1 --2- 172.21.15.125:0/3578200554 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f181c0664a0 0x7f181c068960 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f182001ecd0 tx=0x7f18200aa070 comp rx=0 tx=0).stop 2023-10-30T12:28:53.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f1836646640 1 -- 172.21.15.125:0/3578200554 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f18300c46c0 msgr2=0x7f18301494a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.943+0000 7f1836646640 1 --2- 172.21.15.125:0/3578200554 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f18300c46c0 0x7f18301494a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1824014bf0 tx=0x7f1824012430 comp rx=0 tx=0).stop 2023-10-30T12:28:53.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.947+0000 7f94305d3640 1 -- 172.21.15.125:0/1937310020 wait complete. 2023-10-30T12:28:53.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.947+0000 7f1834be3640 1 -- 172.21.15.125:0/3578200554 reap_dead start 2023-10-30T12:28:53.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.947+0000 7f1836646640 1 -- 172.21.15.125:0/3578200554 shutdown_connections 2023-10-30T12:28:53.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.947+0000 7f1836646640 1 -- 172.21.15.125:0/3578200554 >> 172.21.15.125:0/3578200554 conn(0x7f18300cac50 msgr2=0x7f1830157810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.947+0000 7f1836646640 1 -- 172.21.15.125:0/3578200554 shutdown_connections 2023-10-30T12:28:53.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.947+0000 7f1836646640 1 -- 172.21.15.125:0/3578200554 wait complete. 2023-10-30T12:28:53.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.959+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/3245473633 --> [v2:172.21.15.125:6808/456280330,v1:172.21.15.125:6809/456280330] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3ba015a5c0 con 0x7f3b8c06e6c0 2023-10-30T12:28:53.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.959+0000 7fa7fade4640 1 -- 172.21.15.125:0/1415221079 --> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa7ec006ad0 con 0x7fa7d806e790 2023-10-30T12:28:53.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.959+0000 7f3b9e7fc640 1 -- 172.21.15.125:0/3245473633 <== osd.2 v2:172.21.15.125:6808/456280330 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3b94009a30 con 0x7f3b8c06e6c0 2023-10-30T12:28:53.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.959+0000 7fa7ea7fc640 1 -- 172.21.15.125:0/1415221079 <== osd.4 v2:172.21.15.136:6817/4208991530 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa7d8071260 con 0x7fa7d806e790 2023-10-30T12:28:53.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.959+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/3245473633 >> [v2:172.21.15.125:6808/456280330,v1:172.21.15.125:6809/456280330] conn(0x7f3b8c06e6c0 msgr2=0x7f3b8c070ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.959+0000 7f3ba7b2a640 1 --2- 172.21.15.125:0/3245473633 >> [v2:172.21.15.125:6808/456280330,v1:172.21.15.125:6809/456280330] conn(0x7f3b8c06e6c0 0x7f3b8c070ae0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.959+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/3245473633 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b8c0664a0 msgr2=0x7f3b8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.959+0000 7f3ba7b2a640 1 --2- 172.21.15.125:0/3245473633 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b8c0664a0 0x7f3b8c068960 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3b980d0000 tx=0x7f3b980c9040 comp rx=0 tx=0).stop 2023-10-30T12:28:53.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.963+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/3245473633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3ba00cd0b0 msgr2=0x7f3ba013b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.963+0000 7f3ba7b2a640 1 --2- 172.21.15.125:0/3245473633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3ba00cd0b0 0x7f3ba013b4e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3ba008d1c0 tx=0x7f3b9000c810 comp rx=0 tx=0).stop 2023-10-30T12:28:53.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.963+0000 7f3ba60c7640 1 -- 172.21.15.125:0/3245473633 reap_dead start 2023-10-30T12:28:53.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.963+0000 7fa7fade4640 1 -- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7fa7d806e790 msgr2=0x7fa7d8070bb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.963+0000 7fa7fade4640 1 --2- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7fa7d806e790 0x7fa7d8070bb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.963+0000 7fa7fade4640 1 -- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa7d8066570 msgr2=0x7fa7d8068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.963+0000 7fa7fade4640 1 --2- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa7d8066570 0x7fa7d8068a30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa7f00b2000 tx=0x7fa7f00aa070 comp rx=0 tx=0).stop 2023-10-30T12:28:53.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.963+0000 7fa7fade4640 1 -- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7ec0d1760 msgr2=0x7fa7ec0c6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.963+0000 7fa7fade4640 1 --2- 172.21.15.125:0/1415221079 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7ec0d1760 0x7fa7ec0c6a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa7f40683f0 tx=0x7fa7f4066e60 comp rx=0 tx=0).stop 2023-10-30T12:28:53.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.963+0000 7fa7fa5e3640 1 -- 172.21.15.125:0/1415221079 reap_dead start 2023-10-30T12:28:53.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.963+0000 7fa7fade4640 1 -- 172.21.15.125:0/1415221079 shutdown_connections 2023-10-30T12:28:53.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.963+0000 7fa7fade4640 1 -- 172.21.15.125:0/1415221079 >> 172.21.15.125:0/1415221079 conn(0x7fa7ec0502e0 msgr2=0x7fa7ec0344b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.963+0000 7fa7fade4640 1 -- 172.21.15.125:0/1415221079 shutdown_connections 2023-10-30T12:28:53.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.963+0000 7fa7fade4640 1 -- 172.21.15.125:0/1415221079 wait complete. 2023-10-30T12:28:53.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/3245473633 shutdown_connections 2023-10-30T12:28:53.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/3245473633 >> 172.21.15.125:0/3245473633 conn(0x7f3ba006bfe0 msgr2=0x7f3ba008b380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/3245473633 shutdown_connections 2023-10-30T12:28:53.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7f3ba7b2a640 1 -- 172.21.15.125:0/3245473633 wait complete. 2023-10-30T12:28:53.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7efdc3fff640 1 -- 172.21.15.125:0/2910377160 --> [v2:172.21.15.136:6808/3315858521,v1:172.21.15.136:6809/3315858521] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efdbc04a030 con 0x7efdb006e6c0 2023-10-30T12:28:53.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7efdab7fe640 1 -- 172.21.15.125:0/2910377160 <== osd.7 v2:172.21.15.136:6808/3315858521 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efdc4123840 con 0x7efdb006e6c0 2023-10-30T12:28:53.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/3214841482 --> [v2:172.21.15.136:6800/1849732357,v1:172.21.15.136:6801/1849732357] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5bfc002cc0 con 0x7f5bf806e6f0 2023-10-30T12:28:53.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7efdc3fff640 1 -- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:6808/3315858521,v1:172.21.15.136:6809/3315858521] conn(0x7efdb006e6c0 msgr2=0x7efdb0070ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7efdc3fff640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:6808/3315858521,v1:172.21.15.136:6809/3315858521] conn(0x7efdb006e6c0 0x7efdb0070ae0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7efdc3fff640 1 -- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efdb00664a0 msgr2=0x7efdb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7efdc3fff640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efdb00664a0 0x7efdb0068960 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7efdb4010f10 tx=0x7efdb4007cc0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7efdc3fff640 1 -- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdbc050a30 msgr2=0x7efdbc0533f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7efdc3fff640 1 --2- 172.21.15.125:0/2910377160 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdbc050a30 0x7efdbc0533f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efdb800d5f0 tx=0x7efdb800dab0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7efdc37fe640 1 -- 172.21.15.125:0/2910377160 reap_dead start 2023-10-30T12:28:53.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7efdc3fff640 1 -- 172.21.15.125:0/2910377160 shutdown_connections 2023-10-30T12:28:53.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.967+0000 7efdc3fff640 1 -- 172.21.15.125:0/2910377160 >> 172.21.15.125:0/2910377160 conn(0x7efdbc051040 msgr2=0x7efdbc16af20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7f5c157fa640 1 -- 172.21.15.125:0/3214841482 <== osd.5 v2:172.21.15.136:6800/1849732357 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5bfc002cc0 con 0x7f5bf806e6f0 2023-10-30T12:28:53.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7fe57e580640 1 -- 172.21.15.125:0/1157019643 --> [v2:172.21.15.136:6816/3758986733,v1:172.21.15.136:6818/3758986733] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe558002cc0 con 0x7fe56406e750 2023-10-30T12:28:53.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:6800/1849732357,v1:172.21.15.136:6801/1849732357] conn(0x7f5bf806e6f0 msgr2=0x7f5bf8070b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7f5c1e6ea640 1 --2- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:6800/1849732357,v1:172.21.15.136:6801/1849732357] conn(0x7f5bf806e6f0 0x7f5bf8070b10 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5bf80664d0 msgr2=0x7f5bf8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7f5c1e6ea640 1 --2- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5bf80664d0 0x7f5bf8068990 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5c080099b0 tx=0x7f5c08009190 comp rx=0 tx=0).stop 2023-10-30T12:28:53.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c18070840 msgr2=0x7f5c18148b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7f5c1e6ea640 1 --2- 172.21.15.125:0/3214841482 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c18070840 0x7f5c18148b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000b440 tx=0x7f5c0000b900 comp rx=0 tx=0).stop 2023-10-30T12:28:53.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7f5c1cc87640 1 -- 172.21.15.125:0/3214841482 reap_dead start 2023-10-30T12:28:53.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7fe574ff9640 1 -- 172.21.15.125:0/1157019643 <== osd.6 v2:172.21.15.136:6816/3758986733 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe558002cc0 con 0x7fe56406e750 2023-10-30T12:28:53.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/3214841482 shutdown_connections 2023-10-30T12:28:53.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/3214841482 >> 172.21.15.125:0/3214841482 conn(0x7f5c180c7910 msgr2=0x7f5c18157c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7efdc3fff640 1 -- 172.21.15.125:0/2910377160 shutdown_connections 2023-10-30T12:28:53.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7efdc3fff640 1 -- 172.21.15.125:0/2910377160 wait complete. 2023-10-30T12:28:53.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7fe57e580640 1 -- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:6816/3758986733,v1:172.21.15.136:6818/3758986733] conn(0x7fe56406e750 msgr2=0x7fe564070b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7fe57e580640 1 --2- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:6816/3758986733,v1:172.21.15.136:6818/3758986733] conn(0x7fe56406e750 0x7fe564070b70 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/3214841482 shutdown_connections 2023-10-30T12:28:53.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7f5c1e6ea640 1 -- 172.21.15.125:0/3214841482 wait complete. 2023-10-30T12:28:53.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7fe57e580640 1 -- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe564066530 msgr2=0x7fe5640689f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7fe57e580640 1 --2- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe564066530 0x7fe5640689f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fe5780ca7f0 tx=0x7fe56c008040 comp rx=0 tx=0).stop 2023-10-30T12:28:53.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7fe57e580640 1 -- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe57806a0f0 msgr2=0x7fe578161820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7fe57e580640 1 --2- 172.21.15.125:0/1157019643 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe57806a0f0 0x7fe578161820 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe56801a120 tx=0x7fe568019290 comp rx=0 tx=0).stop 2023-10-30T12:28:53.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.971+0000 7fe57cb1d640 1 -- 172.21.15.125:0/1157019643 reap_dead start 2023-10-30T12:28:53.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.975+0000 7fe57e580640 1 -- 172.21.15.125:0/1157019643 shutdown_connections 2023-10-30T12:28:53.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.975+0000 7fe57e580640 1 -- 172.21.15.125:0/1157019643 >> 172.21.15.125:0/1157019643 conn(0x7fe5780c7910 msgr2=0x7fe578162640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.065 INFO:teuthology.orchestra.run.smithi125.stdout:68719476742 2023-10-30T12:28:53.066 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-30T12:28:53.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.979+0000 7fe57e580640 1 -- 172.21.15.125:0/1157019643 shutdown_connections 2023-10-30T12:28:53.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:52.983+0000 7fe57e580640 1 -- 172.21.15.125:0/1157019643 wait complete. 2023-10-30T12:28:53.067 INFO:teuthology.orchestra.run.smithi125.stdout:68719476743 2023-10-30T12:28:53.068 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-30T12:28:53.068 INFO:teuthology.orchestra.run.smithi125.stdout:68719476744 2023-10-30T12:28:53.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-30T12:28:53.069 INFO:teuthology.orchestra.run.smithi125.stdout:64424509449 2023-10-30T12:28:53.070 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T12:28:53.071 INFO:teuthology.orchestra.run.smithi125.stdout:60129542150 2023-10-30T12:28:53.071 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T12:28:53.072 INFO:teuthology.orchestra.run.smithi125.stdout:68719476744 2023-10-30T12:28:53.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T12:28:53.073 INFO:teuthology.orchestra.run.smithi125.stdout:60129542151 2023-10-30T12:28:53.073 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T12:28:53.074 INFO:teuthology.orchestra.run.smithi125.stdout:64424509448 2023-10-30T12:28:53.075 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T12:28:53.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.351+0000 7f5120d65640 1 Processor -- start 2023-10-30T12:28:53.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.351+0000 7f5120d65640 1 -- start start 2023-10-30T12:28:53.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.351+0000 7f5120d65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f511c0c90d0 0x7f511c0cef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.351+0000 7f5120d65640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f511c0cf4d0 0x7f511c15d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.351+0000 7f5120d65640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f511c15ce30 con 0x7f511c0c8cf0 2023-10-30T12:28:53.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.351+0000 7f5120d65640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f511c159880 con 0x7f511c0cf4d0 2023-10-30T12:28:53.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.351+0000 7f5120d65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511c159ae0 con 0x7f511c0c90d0 2023-10-30T12:28:53.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f511a59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f511c0c8cf0 0x7f511c0c9e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:36944/0 (socket says 172.21.15.125:36944) 2023-10-30T12:28:53.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f511a59c640 1 -- 172.21.15.125:0/645945895 learned_addr learned my addr 172.21.15.125:0/645945895 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:53.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5119d9b640 1 --2- 172.21.15.125:0/645945895 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f511c0c90d0 0x7f511c0cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f511ad9d640 1 --2- 172.21.15.125:0/645945895 >> v2:172.21.15.125:3300/0 conn(0x7f511c0cf4d0 0x7f511c15d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f511ad9d640 1 -- 172.21.15.125:0/645945895 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f511c0c90d0 msgr2=0x7f511c0cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:53.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f511ad9d640 1 --2- 172.21.15.125:0/645945895 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f511c0c90d0 0x7f511c0cef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f511ad9d640 1 -- 172.21.15.125:0/645945895 >> v1:172.21.15.136:6789/0 conn(0x7f511c0c8cf0 legacy=0x7f511c0c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f511ad9d640 1 -- 172.21.15.125:0/645945895 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511c159dc0 con 0x7f511c0cf4d0 2023-10-30T12:28:53.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.351+0000 7f5eb5a17640 1 Processor -- start 2023-10-30T12:28:53.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.351+0000 7f5eb5a17640 1 -- start start 2023-10-30T12:28:53.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eb5a17640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ea80dd270 0x7f5ea8045170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eb5a17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ea8007390 0x7f5ea80e3b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eaf7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5ea804c370 0x7f5ea804b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54102/0 (socket says 172.21.15.125:54102) 2023-10-30T12:28:53.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eaf7fe640 1 -- 172.21.15.125:0/975632535 learned_addr learned my addr 172.21.15.125:0/975632535 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:53.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eb5a17640 1 -- 172.21.15.125:0/975632535 --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea8006f00 con 0x7f5ea804c370 2023-10-30T12:28:53.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 -- 172.21.15.125:0/975632535 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652004884 0 0) 0x7f5ea8006f00 con 0x7f5ea804c370 2023-10-30T12:28:53.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eb5a17640 1 -- 172.21.15.125:0/975632535 --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea8033510 con 0x7f5ea80dd270 2023-10-30T12:28:53.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eb5a17640 1 -- 172.21.15.125:0/975632535 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea8034010 con 0x7f5ea8007390 2023-10-30T12:28:53.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eaeffd640 1 --2- 172.21.15.125:0/975632535 >> v2:172.21.15.125:3300/0 conn(0x7f5ea80dd270 0x7f5ea8045170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 -- 172.21.15.125:0/975632535 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e98003530 con 0x7f5ea804c370 2023-10-30T12:28:53.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eae7fc640 1 --2- 172.21.15.125:0/975632535 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ea8007390 0x7f5ea80e3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 -- 172.21.15.125:0/975632535 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1063907871 0 0) 0x7f5e98003530 con 0x7f5ea804c370 2023-10-30T12:28:53.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 -- 172.21.15.125:0/975632535 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ea8007390 msgr2=0x7f5ea80e3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:53.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 --2- 172.21.15.125:0/975632535 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ea8007390 0x7f5ea80e3b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 -- 172.21.15.125:0/975632535 >> v2:172.21.15.125:3300/0 conn(0x7f5ea80dd270 msgr2=0x7f5ea8045170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:53.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 --2- 172.21.15.125:0/975632535 >> v2:172.21.15.125:3300/0 conn(0x7f5ea80dd270 0x7f5ea8045170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 -- 172.21.15.125:0/975632535 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea80077d0 con 0x7f5ea804c370 2023-10-30T12:28:53.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 -- 172.21.15.125:0/975632535 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f5eb0069ab0 con 0x7f5ea804c370 2023-10-30T12:28:53.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 -- 172.21.15.125:0/975632535 >> v1:172.21.15.125:6789/0 conn(0x7f5ea804c370 legacy=0x7f5ea804b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 --2- 172.21.15.125:0/975632535 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ea8007390 0x7f5e98003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 --2- 172.21.15.125:0/975632535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e98003fb0 0x7f5e980047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 --2- 172.21.15.125:0/975632535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e98004d10 0x7f5e98005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 -- 172.21.15.125:0/975632535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea8033510 con 0x7f5e98004d10 2023-10-30T12:28:53.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 -- 172.21.15.125:0/975632535 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9802db00 con 0x7f5e98003fb0 2023-10-30T12:28:53.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.355+0000 7f5eadffb640 1 -- 172.21.15.125:0/975632535 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e9802dcc0 con 0x7f5ea8007390 2023-10-30T12:28:53.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.359+0000 7f5eaf7fe640 1 --2- 172.21.15.125:0/975632535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e98004d10 0x7f5e98005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.359+0000 7f5eaf7fe640 1 -- 172.21.15.125:0/975632535 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ea8007390 msgr2=0x7f5e98003980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T12:28:53.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.359+0000 7f5eaf7fe640 1 --2- 172.21.15.125:0/975632535 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ea8007390 0x7f5e98003980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.359+0000 7f5eaf7fe640 1 -- 172.21.15.125:0/975632535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e98003fb0 msgr2=0x7f5e980047d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-30T12:28:53.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.359+0000 7f5eaf7fe640 1 --2- 172.21.15.125:0/975632535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e98003fb0 0x7f5e980047d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.359+0000 7f5eaf7fe640 1 -- 172.21.15.125:0/975632535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e98003530 con 0x7f5e98004d10 2023-10-30T12:28:53.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.359+0000 7fa3b411d640 1 Processor -- start 2023-10-30T12:28:53.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.359+0000 7fa3b411d640 1 -- start start 2023-10-30T12:28:53.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.359+0000 7f511ad9d640 1 --2- 172.21.15.125:0/645945895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f511c0cf4d0 0x7f511c15d2c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5110009a20 tx=0x7f511001c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5beae6d84608e8f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.359+0000 7fa3b411d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa3ac072040 0x7fa3ac16b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.359+0000 7fa3b411d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3ac0c5810 0x7fa3ac070bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7fa3b411d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3ac16b8f0 con 0x7fa3ac06aa00 2023-10-30T12:28:53.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7fa3b411d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3ac0707d0 con 0x7fa3ac072040 2023-10-30T12:28:53.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7fa3b411d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac072420 con 0x7fa3ac0c5810 2023-10-30T12:28:53.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7fa3b1eb9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa3ac072040 0x7fa3ac16b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f511959a640 1 -- 172.21.15.125:0/645945895 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51100a6670 con 0x7f511c0cf4d0 2023-10-30T12:28:53.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f511959a640 1 -- 172.21.15.125:0/645945895 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51100a67d0 con 0x7f511c0cf4d0 2023-10-30T12:28:53.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f511959a640 1 -- 172.21.15.125:0/645945895 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51100a6b60 con 0x7f511c0cf4d0 2023-10-30T12:28:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7fa3b1eb9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa3ac072040 0x7fa3ac16b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38840/0 (socket says 172.21.15.125:38840) 2023-10-30T12:28:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7fa3b1eb9640 1 -- 172.21.15.125:0/2496327145 learned_addr learned my addr 172.21.15.125:0/2496327145 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7fa3b26ba640 1 --2- 172.21.15.125:0/2496327145 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3ac0c5810 0x7fa3ac070bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7fa3b1eb9640 1 -- 172.21.15.125:0/2496327145 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3ac0c5810 msgr2=0x7fa3ac070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7fa3b1eb9640 1 --2- 172.21.15.125:0/2496327145 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3ac0c5810 0x7fa3ac070bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7fa3b1eb9640 1 -- 172.21.15.125:0/2496327145 >> v1:172.21.15.136:6789/0 conn(0x7fa3ac06aa00 legacy=0x7fa3ac069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7fa3b1eb9640 1 -- 172.21.15.125:0/2496327145 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3ac0c5de0 con 0x7fa3ac072040 2023-10-30T12:28:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f5120d65640 1 -- 172.21.15.125:0/645945895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f511c0cf4d0 msgr2=0x7f511c15d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f8b4a11e640 1 Processor -- start 2023-10-30T12:28:53.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f5120d65640 1 --2- 172.21.15.125:0/645945895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f511c0cf4d0 0x7f511c15d2c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5110009a20 tx=0x7f511001c6f0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f8b4a11e640 1 -- start start 2023-10-30T12:28:53.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f5120d65640 1 -- 172.21.15.125:0/645945895 shutdown_connections 2023-10-30T12:28:53.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f5120d65640 1 --2- 172.21.15.125:0/645945895 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f511c0cf4d0 0x7f511c15d2c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f5120d65640 1 --2- 172.21.15.125:0/645945895 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f511c0c90d0 0x7f511c0cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f5120d65640 1 -- 172.21.15.125:0/645945895 >> 172.21.15.125:0/645945895 conn(0x7f511c06be40 msgr2=0x7f511c06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f5eaf7fe640 1 --2- 172.21.15.125:0/975632535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e98004d10 0x7f5e98005190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0069490 tx=0x7f5eb0097eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb425e31b765efca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f5120d65640 1 -- 172.21.15.125:0/645945895 shutdown_connections 2023-10-30T12:28:53.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f5eadffb640 1 -- 172.21.15.125:0/975632535 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb01209b0 con 0x7f5e98004d10 2023-10-30T12:28:53.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f5120d65640 1 -- 172.21.15.125:0/645945895 wait complete. 2023-10-30T12:28:53.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.363+0000 7f5eadffb640 1 -- 172.21.15.125:0/975632535 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb0069e90 con 0x7f5e98004d10 2023-10-30T12:28:53.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f5eadffb640 1 -- 172.21.15.125:0/975632535 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0120e10 con 0x7f5e98004d10 2023-10-30T12:28:53.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f8b4a11e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8b4406fef0 0x7f8b44161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f8b4a11e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8b4406ab20 0x7f8b44070c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f8b4a11e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b44161f60 con 0x7f8b44157c80 2023-10-30T12:28:53.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f8b4a11e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b4415ef30 con 0x7f8b4406ab20 2023-10-30T12:28:53.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f8b4a11e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b44160e50 con 0x7f8b4406fef0 2023-10-30T12:28:53.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f1675279640 1 Processor -- start 2023-10-30T12:28:53.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f8b4891b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8b4406fef0 0x7f8b44161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f8b4891b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8b4406fef0 0x7f8b44161030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38848/0 (socket says 172.21.15.125:38848) 2023-10-30T12:28:53.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f8b4891b640 1 -- 172.21.15.125:0/79501558 learned_addr learned my addr 172.21.15.125:0/79501558 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f5eb5a17640 1 -- 172.21.15.125:0/975632535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e98004d10 msgr2=0x7f5e98005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f5eb5a17640 1 --2- 172.21.15.125:0/975632535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e98004d10 0x7f5e98005190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5eb0069490 tx=0x7f5eb0097eb0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f5eaf7fe640 1 -- 172.21.15.125:0/975632535 reap_dead start 2023-10-30T12:28:53.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f5eb5a17640 1 -- 172.21.15.125:0/975632535 shutdown_connections 2023-10-30T12:28:53.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f5eb5a17640 1 -- 172.21.15.125:0/975632535 >> 172.21.15.125:0/975632535 conn(0x7f5ea80501d0 msgr2=0x7f5ea80505e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f1675279640 1 -- start start 2023-10-30T12:28:53.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f5eb5a17640 1 -- 172.21.15.125:0/975632535 shutdown_connections 2023-10-30T12:28:53.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f5eb5a17640 1 -- 172.21.15.125:0/975632535 wait complete. 2023-10-30T12:28:53.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f8b4891b640 1 -- 172.21.15.125:0/79501558 >> v2:172.21.15.125:3300/0 conn(0x7f8b4406ab20 msgr2=0x7f8b44070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:53.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f8b4891b640 1 --2- 172.21.15.125:0/79501558 >> v2:172.21.15.125:3300/0 conn(0x7f8b4406ab20 0x7f8b44070c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f8b4891b640 1 -- 172.21.15.125:0/79501558 >> v1:172.21.15.125:6789/0 conn(0x7f8b44157c80 legacy=0x7f8b4406f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:53.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f8b4891b640 1 -- 172.21.15.125:0/79501558 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b441616c0 con 0x7f8b4406fef0 2023-10-30T12:28:53.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7fa3b1eb9640 1 --2- 172.21.15.125:0/2496327145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3ac072040 0x7fa3ac16b3b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009e50 tx=0x7fa3a401f210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfcac48bd333da1a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f5120d65640 1 Processor -- start 2023-10-30T12:28:53.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f5120d65640 1 -- start start 2023-10-30T12:28:53.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f1675279640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1670153c60 0x7f16700ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f1675279640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1670159830 0x7f1670159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f1675279640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16700cb780 con 0x7f16700c4430 2023-10-30T12:28:53.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f1675279640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1670154040 con 0x7f1670159830 2023-10-30T12:28:53.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f1675279640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167015a2e0 con 0x7f1670153c60 2023-10-30T12:28:53.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7fa3b0eb7640 1 -- 172.21.15.125:0/2496327145 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a40af400 con 0x7fa3ac072040 2023-10-30T12:28:53.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f127913e640 1 Processor -- start 2023-10-30T12:28:53.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7fa3b0eb7640 1 -- 172.21.15.125:0/2496327145 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a40ab050 con 0x7fa3ac072040 2023-10-30T12:28:53.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7fa3b0eb7640 1 -- 172.21.15.125:0/2496327145 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a40af400 con 0x7fa3ac072040 2023-10-30T12:28:53.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f127913e640 1 -- start start 2023-10-30T12:28:53.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.367+0000 7f166f7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1670159830 0x7f1670159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f166effd640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1670153c60 0x7f16700ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f166f7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1670159830 0x7f1670159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38862/0 (socket says 172.21.15.125:38862) 2023-10-30T12:28:53.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f166f7fe640 1 -- 172.21.15.125:0/690227835 learned_addr learned my addr 172.21.15.125:0/690227835 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5120d65640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f511c0c8cf0 0x7f511c0c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5120d65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f511c0c90d0 0x7f511c147fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5120d65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f511c0cf4d0 0x7f511c1484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5120d65640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c15c920 con 0x7f511c0c90d0 2023-10-30T12:28:53.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5120d65640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c15aa70 con 0x7f511c0c8cf0 2023-10-30T12:28:53.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5120d65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511c0b3780 con 0x7f511c0cf4d0 2023-10-30T12:28:53.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f166f7fe640 1 -- 172.21.15.125:0/690227835 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1670153c60 msgr2=0x7f16700ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:53.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f166f7fe640 1 --2- 172.21.15.125:0/690227835 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1670153c60 0x7f16700ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f166f7fe640 1 -- 172.21.15.125:0/690227835 >> v1:172.21.15.125:6789/0 conn(0x7f16700c4430 legacy=0x7f16700cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:53.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5eb5a17640 1 Processor -- start 2023-10-30T12:28:53.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f511ad9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f511c0cf4d0 0x7f511c1484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f511a59c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f511c0c8cf0 0x7f511c0c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f511ad9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f511c0cf4d0 0x7f511c1484f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38788/0 (socket says 172.21.15.125:38788) 2023-10-30T12:28:53.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f511ad9d640 1 -- 172.21.15.125:0/3818127730 learned_addr learned my addr 172.21.15.125:0/3818127730 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5eb5a17640 1 -- start start 2023-10-30T12:28:53.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5119d9b640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f511c0c90d0 0x7f511c147fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f166f7fe640 1 -- 172.21.15.125:0/690227835 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167015a5c0 con 0x7f1670159830 2023-10-30T12:28:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7fa3b411d640 1 -- 172.21.15.125:0/2496327145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3ac072040 msgr2=0x7fa3ac16b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f511ad9d640 1 -- 172.21.15.125:0/3818127730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f511c0c8cf0 msgr2=0x7f511c0c77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f511ad9d640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f511c0c8cf0 0x7f511c0c77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f511ad9d640 1 -- 172.21.15.125:0/3818127730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f511c0c90d0 msgr2=0x7f511c147fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f511ad9d640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f511c0c90d0 0x7f511c147fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f511ad9d640 1 -- 172.21.15.125:0/3818127730 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5110009420 con 0x7f511c0cf4d0 2023-10-30T12:28:53.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f511a59c640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f511c0c8cf0 0x7f511c0c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:53.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5119d9b640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f511c0c90d0 0x7f511c147fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:53.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7fa3b411d640 1 --2- 172.21.15.125:0/2496327145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3ac072040 0x7fa3ac16b3b0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa3a4009e50 tx=0x7fa3a401f210 comp rx=0 tx=0).stop 2023-10-30T12:28:53.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f511ad9d640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f511c0cf4d0 0x7f511c1484f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f511001a320 tx=0x7f511001af30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f8b4891b640 1 --2- 172.21.15.125:0/79501558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b4406fef0 0x7f8b44161030 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8b4406b250 tx=0x7f8b3801ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19a43641713b8f63 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f8b37fff640 1 -- 172.21.15.125:0/79501558 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b380a89b0 con 0x7f8b4406fef0 2023-10-30T12:28:53.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5eb5a17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e98004d10 0x7f5ea80cc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5eb5a17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea80ca7f0 0x7f5ea80cb340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5eb5a17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ea80cb880 0x7f5ea80d8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5eb5a17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80d9410 con 0x7f5e98004d10 2023-10-30T12:28:53.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5eb5a17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80da2d0 con 0x7f5ea80ca7f0 2023-10-30T12:28:53.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5eb5a17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea80da560 con 0x7f5ea80cb880 2023-10-30T12:28:53.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5eae7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea80ca7f0 0x7f5ea80cb340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f8b37fff640 1 -- 172.21.15.125:0/79501558 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b380a8b10 con 0x7f8b4406fef0 2023-10-30T12:28:53.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f8b37fff640 1 -- 172.21.15.125:0/79501558 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b380a8ea0 con 0x7f8b4406fef0 2023-10-30T12:28:53.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5eae7fc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea80ca7f0 0x7f5ea80cb340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38868/0 (socket says 172.21.15.125:38868) 2023-10-30T12:28:53.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5eae7fc640 1 -- 172.21.15.125:0/1513965327 learned_addr learned my addr 172.21.15.125:0/1513965327 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f510affd640 1 -- 172.21.15.125:0/3818127730 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51100ba680 con 0x7f511c0cf4d0 2023-10-30T12:28:53.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f510affd640 1 -- 172.21.15.125:0/3818127730 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51100ba820 con 0x7f511c0cf4d0 2023-10-30T12:28:53.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f510affd640 1 -- 172.21.15.125:0/3818127730 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51100babd0 con 0x7f511c0cf4d0 2023-10-30T12:28:53.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5120d65640 1 -- 172.21.15.125:0/3818127730 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511c0b3a20 con 0x7f511c0cf4d0 2023-10-30T12:28:53.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5eaeffd640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e98004d10 0x7f5ea80cc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.371+0000 7f5120d65640 1 -- 172.21.15.125:0/3818127730 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f511c14f360 con 0x7f511c0cf4d0 2023-10-30T12:28:53.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f5eaf7fe640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ea80cb880 0x7f5ea80d8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f5eae7fc640 1 -- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ea80cb880 msgr2=0x7f5ea80d8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f5eae7fc640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ea80cb880 0x7f5ea80d8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f5eae7fc640 1 -- 172.21.15.125:0/1513965327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e98004d10 msgr2=0x7f5ea80cc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f5eae7fc640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e98004d10 0x7f5ea80cc280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f5eae7fc640 1 -- 172.21.15.125:0/1513965327 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea8036330 con 0x7f5ea80ca7f0 2023-10-30T12:28:53.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f5eaf7fe640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ea80cb880 0x7f5ea80d8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:53.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa3b411d640 1 -- 172.21.15.125:0/2496327145 shutdown_connections 2023-10-30T12:28:53.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa3b411d640 1 --2- 172.21.15.125:0/2496327145 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3ac0c5810 0x7fa3ac070bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa3b411d640 1 --2- 172.21.15.125:0/2496327145 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3ac072040 0x7fa3ac16b3b0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa3b411d640 1 -- 172.21.15.125:0/2496327145 >> 172.21.15.125:0/2496327145 conn(0x7fa3ac0c7910 msgr2=0x7fa3ac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f8b4a11e640 1 -- 172.21.15.125:0/79501558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b4406fef0 msgr2=0x7f8b44161030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f8b4a11e640 1 --2- 172.21.15.125:0/79501558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b4406fef0 0x7f8b44161030 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8b4406b250 tx=0x7f8b3801ecb0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa3b411d640 1 -- 172.21.15.125:0/2496327145 shutdown_connections 2023-10-30T12:28:53.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f127913e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12740c4430 0x7f12740ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f127913e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1274159830 0x7f1274159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa3b411d640 1 -- 172.21.15.125:0/2496327145 wait complete. 2023-10-30T12:28:53.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f5eae7fc640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea80ca7f0 0x7f5ea80cb340 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000c5b0 tx=0x7f5ea000ca70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f1272d9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1274153c60 0x7f12740cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:36958/0 (socket says 172.21.15.125:36958) 2023-10-30T12:28:53.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa39a4a9640 1 Processor -- start 2023-10-30T12:28:53.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f8b4a11e640 1 -- 172.21.15.125:0/79501558 shutdown_connections 2023-10-30T12:28:53.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f8b4a11e640 1 --2- 172.21.15.125:0/79501558 >> v2:172.21.15.125:3300/0 conn(0x7f8b4406ab20 0x7f8b44070c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f8b4a11e640 1 --2- 172.21.15.125:0/79501558 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b4406fef0 0x7f8b44161030 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f8b4a11e640 1 -- 172.21.15.125:0/79501558 >> 172.21.15.125:0/79501558 conn(0x7f8b440c75d0 msgr2=0x7f8b440c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f127259c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12740c4430 0x7f12740ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f127259c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12740c4430 0x7f12740ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38808/0 (socket says 172.21.15.125:38808) 2023-10-30T12:28:53.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f8b4a11e640 1 -- 172.21.15.125:0/79501558 shutdown_connections 2023-10-30T12:28:53.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa39a4a9640 1 -- start start 2023-10-30T12:28:53.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f5e97fff640 1 -- 172.21.15.125:0/1513965327 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0011800 con 0x7f5ea80ca7f0 2023-10-30T12:28:53.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f5eb5a17640 1 -- 172.21.15.125:0/1513965327 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea80d9660 con 0x7f5ea80ca7f0 2023-10-30T12:28:53.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f5120d65640 1 -- 172.21.15.125:0/3818127730 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f511c14f1c0 con 0x7f511c0cf4d0 2023-10-30T12:28:53.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f5e97fff640 1 -- 172.21.15.125:0/1513965327 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea0011960 con 0x7f5ea80ca7f0 2023-10-30T12:28:53.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f8b4a11e640 1 -- 172.21.15.125:0/79501558 wait complete. 2023-10-30T12:28:53.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f1272d9d640 1 -- 172.21.15.125:0/166125011 learned_addr learned my addr 172.21.15.125:0/166125011 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:53.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f127913e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12740cb780 con 0x7f1274153c60 2023-10-30T12:28:53.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f127913e640 1 -- 172.21.15.125:0/166125011 --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1274154040 con 0x7f1274159830 2023-10-30T12:28:53.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f127913e640 1 -- 172.21.15.125:0/166125011 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f127415a2e0 con 0x7f12740c4430 2023-10-30T12:28:53.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f127359e640 1 --2- 172.21.15.125:0/166125011 >> v2:172.21.15.136:3300/0 conn(0x7f1274159830 0x7f1274159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f1271d9b640 1 -- 172.21.15.125:0/166125011 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871779380 0 0) 0x7f12740cb780 con 0x7f1274153c60 2023-10-30T12:28:53.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f1271d9b640 1 -- 172.21.15.125:0/166125011 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1260003530 con 0x7f1274153c60 2023-10-30T12:28:53.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa3b411d640 1 Processor -- start 2023-10-30T12:28:53.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa3b411d640 1 -- start start 2023-10-30T12:28:53.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f5eb5a17640 1 -- 172.21.15.125:0/1513965327 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea80d7ee0 con 0x7f5ea80ca7f0 2023-10-30T12:28:53.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f5e97fff640 1 -- 172.21.15.125:0/1513965327 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0011800 con 0x7f5ea80ca7f0 2023-10-30T12:28:53.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa39a4a9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa394153c60 0x7fa3940ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa39a4a9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa3940c4430 0x7fa3940cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa39a4a9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3940cb780 con 0x7fa394159830 2023-10-30T12:28:53.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa39a4a9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa394154040 con 0x7fa3940c4430 2023-10-30T12:28:53.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7fa39a4a9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3940cc9f0 con 0x7fa394153c60 2023-10-30T12:28:53.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f5eb5a17640 1 -- 172.21.15.125:0/1513965327 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea8005350 con 0x7f5ea80ca7f0 2023-10-30T12:28:53.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f510affd640 1 -- 172.21.15.125:0/3818127730 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51100bad30 con 0x7f511c0cf4d0 2023-10-30T12:28:53.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f127259c640 1 -- 172.21.15.125:0/166125011 >> v2:172.21.15.136:3300/0 conn(0x7f1274159830 msgr2=0x7f1274159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:53.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f127259c640 1 --2- 172.21.15.125:0/166125011 >> v2:172.21.15.136:3300/0 conn(0x7f1274159830 0x7f1274159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f127259c640 1 -- 172.21.15.125:0/166125011 >> v1:172.21.15.136:6789/0 conn(0x7f1274153c60 legacy=0x7f12740cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.375+0000 7f127259c640 1 -- 172.21.15.125:0/166125011 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f127415a5c0 con 0x7f12740c4430 2023-10-30T12:28:53.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f8b4a11e640 1 Processor -- start 2023-10-30T12:28:53.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f510affd640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f50fc0665a0 0x7f50fc068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa393fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa394153c60 0x7fa3940ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f8b4a11e640 1 -- start start 2023-10-30T12:28:53.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa393fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa394153c60 0x7fa3940ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38818/0 (socket says 172.21.15.125:38818) 2023-10-30T12:28:53.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa393fff640 1 -- 172.21.15.125:0/2897825713 learned_addr learned my addr 172.21.15.125:0/2897825713 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fc399c09640 1 Processor -- start 2023-10-30T12:28:53.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f510affd640 1 -- 172.21.15.125:0/3818127730 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f511011e850 con 0x7f511c0cf4d0 2023-10-30T12:28:53.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa38bfff640 1 --2- 172.21.15.125:0/2897825713 >> v2:172.21.15.125:3300/0 conn(0x7fa3940c4430 0x7fa3940cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fc399c09640 1 -- start start 2023-10-30T12:28:53.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f511a59c640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f50fc0665a0 0x7f50fc068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa3b411d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3ac06aa00 0x7fa3ac0adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa3b411d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3ac072040 0x7fa3ac0aff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa3b411d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3ac0c5810 0x7fa3ac0b04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f8b4a11e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b4406ab20 0x7f8b441e6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f8b4a11e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b4406fef0 0x7f8b440c5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f8b4a11e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b44157c80 0x7f8b44154e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f8b4a11e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b441e7630 con 0x7f8b4406fef0 2023-10-30T12:28:53.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f8b4a11e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b440c7300 con 0x7f8b44157c80 2023-10-30T12:28:53.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f8b4a11e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b441e84c0 con 0x7f8b4406ab20 2023-10-30T12:28:53.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f511a59c640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f50fc0665a0 0x7f50fc068a60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f510c005890 tx=0x7f510c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f8b4911c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b4406ab20 0x7f8b441e6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f8b4911c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b4406ab20 0x7f8b441e6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38832/0 (socket says 172.21.15.125:38832) 2023-10-30T12:28:53.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f8b4911c640 1 -- 172.21.15.125:0/1223143371 learned_addr learned my addr 172.21.15.125:0/1223143371 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f8b4991d640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b44157c80 0x7f8b44154e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f8b4891b640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b4406fef0 0x7f8b440c5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f166f7fe640 1 --2- 172.21.15.125:0/690227835 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1670159830 0x7f1670159ce0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1668009e50 tx=0x7f166801c9d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29335429ace83e0e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa393fff640 1 -- 172.21.15.125:0/2897825713 >> v2:172.21.15.125:3300/0 conn(0x7fa3940c4430 msgr2=0x7fa3940cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:53.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa393fff640 1 --2- 172.21.15.125:0/2897825713 >> v2:172.21.15.125:3300/0 conn(0x7fa3940c4430 0x7fa3940cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa393fff640 1 -- 172.21.15.125:0/2897825713 >> v1:172.21.15.136:6789/0 conn(0x7fa394159830 legacy=0x7fa3940cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa393fff640 1 -- 172.21.15.125:0/2897825713 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39415a2e0 con 0x7fa394153c60 2023-10-30T12:28:53.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7f127259c640 1 --2- 172.21.15.125:0/166125011 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12740c4430 0x7f12740ca360 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f126c0097c0 tx=0x7f126c01c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9ef1383161a78c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7f5e97fff640 1 -- 172.21.15.125:0/1513965327 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5ea0011ad0 con 0x7f5ea80ca7f0 2023-10-30T12:28:53.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7f8b4911c640 1 -- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b44157c80 msgr2=0x7f8b44154e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7f8b4911c640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b44157c80 0x7f8b44154e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7f8b4911c640 1 -- 172.21.15.125:0/1223143371 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b4406fef0 msgr2=0x7f8b440c5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7f8b4911c640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b4406fef0 0x7f8b440c5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7f8b4911c640 1 -- 172.21.15.125:0/1223143371 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b380094d0 con 0x7f8b4406ab20 2023-10-30T12:28:53.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7f8b4891b640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b4406fef0 0x7f8b440c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fc399c09640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc3940c4430 0x7fc3940ca350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fc399c09640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc3940cb280 0x7fc394159830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fc399c09640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3940cb780 con 0x7fc3940caea0 2023-10-30T12:28:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fc399c09640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc39415a2e0 con 0x7fc3940c4430 2023-10-30T12:28:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fc399c09640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc39415a5c0 con 0x7fc3940cb280 2023-10-30T12:28:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fc38bfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc3940caea0 0x7fc3940cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54124/0 (socket says 172.21.15.125:54124) 2023-10-30T12:28:53.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fc38bfff640 1 -- 172.21.15.125:0/2106821635 learned_addr learned my addr 172.21.15.125:0/2106821635 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:53.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa3b411d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac0af850 con 0x7fa3ac072040 2023-10-30T12:28:53.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa3b411d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac0afae0 con 0x7fa3ac06aa00 2023-10-30T12:28:53.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa3b411d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3ac1ffbd0 con 0x7fa3ac0c5810 2023-10-30T12:28:53.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa3b1eb9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3ac06aa00 0x7fa3ac0adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7f8b4991d640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b44157c80 0x7f8b44154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:53.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa3b26ba640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3ac0c5810 0x7fa3ac0b04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa3b16b8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3ac072040 0x7fa3ac0aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa3b16b8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3ac072040 0x7fa3ac0aff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33158/0 (socket says 172.21.15.125:33158) 2023-10-30T12:28:53.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.379+0000 7fa3b16b8640 1 -- 172.21.15.125:0/934405489 learned_addr learned my addr 172.21.15.125:0/934405489 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fa3b1eb9640 1 -- 172.21.15.125:0/934405489 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3ac0c5810 msgr2=0x7fa3ac0b04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fa3b1eb9640 1 --2- 172.21.15.125:0/934405489 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3ac0c5810 0x7fa3ac0b04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fa3b1eb9640 1 -- 172.21.15.125:0/934405489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3ac072040 msgr2=0x7fa3ac0aff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fa3b1eb9640 1 --2- 172.21.15.125:0/934405489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3ac072040 0x7fa3ac0aff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fa3b1eb9640 1 -- 172.21.15.125:0/934405489 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a4009b30 con 0x7fa3ac06aa00 2023-10-30T12:28:53.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fa3b16b8640 1 --2- 172.21.15.125:0/934405489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3ac072040 0x7fa3ac0aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fc393fff640 1 --2- 172.21.15.125:0/2106821635 >> v2:172.21.15.136:3300/0 conn(0x7fc3940cb280 0x7fc394159830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fc3937fe640 1 --2- 172.21.15.125:0/2106821635 >> v2:172.21.15.125:3300/0 conn(0x7fc3940c4430 0x7fc3940ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fa3b1eb9640 1 --2- 172.21.15.125:0/934405489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3ac06aa00 0x7fa3ac0adb30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa3a40ae000 tx=0x7fa3a401bfb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7f5e97fff640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e9c066500 0x7f5e9c0689c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fc392ffd640 1 -- 172.21.15.125:0/2106821635 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857170681 0 0) 0x7fc3940cb780 con 0x7fc3940caea0 2023-10-30T12:28:53.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fc392ffd640 1 -- 172.21.15.125:0/2106821635 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc380003530 con 0x7fc3940caea0 2023-10-30T12:28:53.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7f5e97fff640 1 -- 172.21.15.125:0/1513965327 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5ea0084060 con 0x7f5ea80ca7f0 2023-10-30T12:28:53.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7f8b4911c640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b4406ab20 0x7f8b441e6e90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000c490 tx=0x7f8b4000c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fc3937fe640 1 -- 172.21.15.125:0/2106821635 >> v2:172.21.15.136:3300/0 conn(0x7fc3940cb280 msgr2=0x7fc394159830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:53.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fc3937fe640 1 --2- 172.21.15.125:0/2106821635 >> v2:172.21.15.136:3300/0 conn(0x7fc3940cb280 0x7fc394159830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fc3937fe640 1 -- 172.21.15.125:0/2106821635 >> v1:172.21.15.125:6789/0 conn(0x7fc3940caea0 legacy=0x7fc3940cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fc3937fe640 1 -- 172.21.15.125:0/2106821635 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3940cb9d0 con 0x7fc3940c4430 2023-10-30T12:28:53.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fc393fff640 1 --2- 172.21.15.125:0/2106821635 >> v2:172.21.15.136:3300/0 conn(0x7fc3940cb280 0x7fc394159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:53.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7f5eaeffd640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e9c066500 0x7f5e9c0689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7fa393fff640 1 --2- 172.21.15.125:0/2897825713 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa394153c60 0x7fa3940ca360 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa384009870 tx=0x7fa38401c560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dad773678260b68 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.383+0000 7f5eaeffd640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e9c066500 0x7f5e9c0689c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400b950 tx=0x7f5ea4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.387+0000 7fc3937fe640 1 --2- 172.21.15.125:0/2106821635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3940c4430 0x7fc3940ca350 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc39408d1a0 tx=0x7fc37c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb469fda263a867c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.387+0000 7fc392ffd640 1 -- 172.21.15.125:0/2106821635 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c004230 con 0x7fc3940c4430 2023-10-30T12:28:53.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.387+0000 7fc392ffd640 1 -- 172.21.15.125:0/2106821635 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37c0a7050 con 0x7fc3940c4430 2023-10-30T12:28:53.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.387+0000 7fc392ffd640 1 -- 172.21.15.125:0/2106821635 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37c003620 con 0x7fc3940c4430 2023-10-30T12:28:53.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.387+0000 7fc399c09640 1 -- 172.21.15.125:0/2106821635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3940c4430 msgr2=0x7fc3940ca350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.387+0000 7fc399c09640 1 --2- 172.21.15.125:0/2106821635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3940c4430 0x7fc3940ca350 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc39408d1a0 tx=0x7fc37c01c460 comp rx=0 tx=0).stop 2023-10-30T12:28:53.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.387+0000 7fc399c09640 1 -- 172.21.15.125:0/2106821635 shutdown_connections 2023-10-30T12:28:53.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.387+0000 7fc399c09640 1 --2- 172.21.15.125:0/2106821635 >> v2:172.21.15.136:3300/0 conn(0x7fc3940cb280 0x7fc394159830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.387+0000 7fc399c09640 1 --2- 172.21.15.125:0/2106821635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3940c4430 0x7fc3940ca350 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.387+0000 7fc399c09640 1 -- 172.21.15.125:0/2106821635 >> 172.21.15.125:0/2106821635 conn(0x7fc39406bfe0 msgr2=0x7fc39406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.387+0000 7fc399c09640 1 -- 172.21.15.125:0/2106821635 shutdown_connections 2023-10-30T12:28:53.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.387+0000 7fc399c09640 1 -- 172.21.15.125:0/2106821635 wait complete. 2023-10-30T12:28:53.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.387+0000 7fc399c09640 1 Processor -- start 2023-10-30T12:28:53.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.387+0000 7fc399c09640 1 -- start start 2023-10-30T12:28:53.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc399c09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3940c4430 0x7fc394168650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc399c09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3940caea0 0x7fc394168b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc399c09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3940cb280 0x7fc394148100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc399c09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc394159350 con 0x7fc3940c4430 2023-10-30T12:28:53.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc399c09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3941681c0 con 0x7fc3940caea0 2023-10-30T12:28:53.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc399c09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc394167be0 con 0x7fc3940cb280 2023-10-30T12:28:53.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc38bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3940c4430 0x7fc394168650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc38bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3940c4430 0x7fc394168650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33180/0 (socket says 172.21.15.125:33180) 2023-10-30T12:28:53.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc38bfff640 1 -- 172.21.15.125:0/286914669 learned_addr learned my addr 172.21.15.125:0/286914669 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc3937fe640 1 --2- 172.21.15.125:0/286914669 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3940caea0 0x7fc394168b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc393fff640 1 --2- 172.21.15.125:0/286914669 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3940cb280 0x7fc394148100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc38bfff640 1 -- 172.21.15.125:0/286914669 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3940cb280 msgr2=0x7fc394148100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc38bfff640 1 --2- 172.21.15.125:0/286914669 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3940cb280 0x7fc394148100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc38bfff640 1 -- 172.21.15.125:0/286914669 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3940caea0 msgr2=0x7fc394168b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc38bfff640 1 --2- 172.21.15.125:0/286914669 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3940caea0 0x7fc394168b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc38bfff640 1 -- 172.21.15.125:0/286914669 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c009a70 con 0x7fc3940c4430 2023-10-30T12:28:53.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc38bfff640 1 --2- 172.21.15.125:0/286914669 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3940c4430 0x7fc394168650 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc38400c4d0 tx=0x7fc38400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc390ff9640 1 -- 172.21.15.125:0/286914669 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384005cf0 con 0x7fc3940c4430 2023-10-30T12:28:53.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc390ff9640 1 -- 172.21.15.125:0/286914669 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc384005e50 con 0x7fc3940c4430 2023-10-30T12:28:53.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc390ff9640 1 -- 172.21.15.125:0/286914669 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc384007270 con 0x7fc3940c4430 2023-10-30T12:28:53.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc399c09640 1 -- 172.21.15.125:0/286914669 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37c009420 con 0x7fc3940c4430 2023-10-30T12:28:53.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc399c09640 1 -- 172.21.15.125:0/286914669 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc394159880 con 0x7fc3940c4430 2023-10-30T12:28:53.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7f1271d9b640 1 -- 172.21.15.125:0/166125011 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c0ab600 con 0x7f12740c4430 2023-10-30T12:28:53.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7f1271d9b640 1 -- 172.21.15.125:0/166125011 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126c0a7050 con 0x7f12740c4430 2023-10-30T12:28:53.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7f1271d9b640 1 -- 172.21.15.125:0/166125011 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c0ab960 con 0x7f12740c4430 2023-10-30T12:28:53.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fa3937fe640 1 -- 172.21.15.125:0/2897825713 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3840ab600 con 0x7fa394153c60 2023-10-30T12:28:53.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fc399c09640 1 -- 172.21.15.125:0/286914669 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc394159a70 con 0x7fc3940c4430 2023-10-30T12:28:53.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fa3937fe640 1 -- 172.21.15.125:0/2897825713 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3840a7050 con 0x7fa394153c60 2023-10-30T12:28:53.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7f8b35ffb640 1 -- 172.21.15.125:0/1223143371 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40003600 con 0x7f8b4406ab20 2023-10-30T12:28:53.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7f8b35ffb640 1 -- 172.21.15.125:0/1223143371 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b40003780 con 0x7f8b4406ab20 2023-10-30T12:28:53.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7f8b35ffb640 1 -- 172.21.15.125:0/1223143371 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b40003b70 con 0x7f8b4406ab20 2023-10-30T12:28:53.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7fa3937fe640 1 -- 172.21.15.125:0/2897825713 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3840ab980 con 0x7fa394153c60 2023-10-30T12:28:53.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7f510affd640 1 -- 172.21.15.125:0/3818127730 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f511c14f1c0 con 0x7f511c0cf4d0 2023-10-30T12:28:53.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.391+0000 7f8b4a11e640 1 -- 172.21.15.125:0/1223143371 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b440c67c0 con 0x7f8b4406ab20 2023-10-30T12:28:53.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f8b4a11e640 1 -- 172.21.15.125:0/1223143371 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b440c56b0 con 0x7f8b4406ab20 2023-10-30T12:28:53.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2897825713 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa394153c60 msgr2=0x7fa3940ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fa39a4a9640 1 --2- 172.21.15.125:0/2897825713 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa394153c60 0x7fa3940ca360 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa384009870 tx=0x7fa38401c560 comp rx=0 tx=0).stop 2023-10-30T12:28:53.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f127913e640 1 -- 172.21.15.125:0/166125011 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12740c4430 msgr2=0x7f12740ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f127913e640 1 --2- 172.21.15.125:0/166125011 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12740c4430 0x7f12740ca360 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f126c0097c0 tx=0x7f126c01c480 comp rx=0 tx=0).stop 2023-10-30T12:28:53.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fc390ff9640 1 -- 172.21.15.125:0/286914669 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3840073d0 con 0x7fc3940c4430 2023-10-30T12:28:53.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fa3a27fc640 1 -- 172.21.15.125:0/934405489 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a4004450 con 0x7fa3ac06aa00 2023-10-30T12:28:53.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f127913e640 1 -- 172.21.15.125:0/166125011 shutdown_connections 2023-10-30T12:28:53.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f127913e640 1 --2- 172.21.15.125:0/166125011 >> v2:172.21.15.136:3300/0 conn(0x7f1274159830 0x7f1274159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f127913e640 1 --2- 172.21.15.125:0/166125011 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12740c4430 0x7f12740ca360 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f127913e640 1 -- 172.21.15.125:0/166125011 >> 172.21.15.125:0/166125011 conn(0x7f127406bfe0 msgr2=0x7f127406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fa3a27fc640 1 -- 172.21.15.125:0/934405489 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a40045d0 con 0x7fa3ac06aa00 2023-10-30T12:28:53.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fa3a27fc640 1 -- 172.21.15.125:0/934405489 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a40ba440 con 0x7fa3ac06aa00 2023-10-30T12:28:53.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f127913e640 1 -- 172.21.15.125:0/166125011 shutdown_connections 2023-10-30T12:28:53.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f127913e640 1 -- 172.21.15.125:0/166125011 wait complete. 2023-10-30T12:28:53.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f5e97fff640 1 -- 172.21.15.125:0/1513965327 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5ea0051450 con 0x7f5ea80ca7f0 2023-10-30T12:28:53.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fa3b411d640 1 -- 172.21.15.125:0/934405489 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3ac1ffeb0 con 0x7fa3ac06aa00 2023-10-30T12:28:53.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fa3b411d640 1 -- 172.21.15.125:0/934405489 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3ac200470 con 0x7fa3ac06aa00 2023-10-30T12:28:53.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f8b4a11e640 1 -- 172.21.15.125:0/1223143371 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4404e170 con 0x7f8b4406ab20 2023-10-30T12:28:53.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f166e7fc640 1 -- 172.21.15.125:0/690227835 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16680a5740 con 0x7f1670159830 2023-10-30T12:28:53.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f166e7fc640 1 -- 172.21.15.125:0/690227835 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16680a58a0 con 0x7f1670159830 2023-10-30T12:28:53.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f166e7fc640 1 -- 172.21.15.125:0/690227835 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16680a5c30 con 0x7f1670159830 2023-10-30T12:28:53.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2897825713 shutdown_connections 2023-10-30T12:28:53.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fa39a4a9640 1 --2- 172.21.15.125:0/2897825713 >> v2:172.21.15.125:3300/0 conn(0x7fa3940c4430 0x7fa3940cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fa39a4a9640 1 --2- 172.21.15.125:0/2897825713 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa394153c60 0x7fa3940ca360 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2897825713 >> 172.21.15.125:0/2897825713 conn(0x7fa39406bfe0 msgr2=0x7fa39406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fc390ff9640 1 --2- 172.21.15.125:0/286914669 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc3740664a0 0x7fc374068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fc390ff9640 1 -- 172.21.15.125:0/286914669 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc3840833d0 con 0x7fc3940c4430 2023-10-30T12:28:53.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f1675279640 1 -- 172.21.15.125:0/690227835 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1670159830 msgr2=0x7f1670159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f1675279640 1 --2- 172.21.15.125:0/690227835 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1670159830 0x7f1670159ce0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1668009e50 tx=0x7f166801c9d0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f1675279640 1 -- 172.21.15.125:0/690227835 shutdown_connections 2023-10-30T12:28:53.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f1675279640 1 --2- 172.21.15.125:0/690227835 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1670159830 0x7f1670159ce0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f1675279640 1 --2- 172.21.15.125:0/690227835 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1670153c60 0x7f16700ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7f1675279640 1 -- 172.21.15.125:0/690227835 >> 172.21.15.125:0/690227835 conn(0x7f167006bfe0 msgr2=0x7f167006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.395+0000 7fa3b411d640 1 -- 172.21.15.125:0/934405489 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3ac200660 con 0x7fa3ac06aa00 2023-10-30T12:28:53.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7fc3937fe640 1 --2- 172.21.15.125:0/286914669 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc3740664a0 0x7fc374068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7f127913e640 1 Processor -- start 2023-10-30T12:28:53.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2897825713 shutdown_connections 2023-10-30T12:28:53.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7fa3a27fc640 1 -- 172.21.15.125:0/934405489 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3a40b6040 con 0x7fa3ac06aa00 2023-10-30T12:28:53.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7f8b35ffb640 1 -- 172.21.15.125:0/1223143371 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8b40003cd0 con 0x7f8b4406ab20 2023-10-30T12:28:53.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7f1675279640 1 -- 172.21.15.125:0/690227835 shutdown_connections 2023-10-30T12:28:53.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7f1675279640 1 -- 172.21.15.125:0/690227835 wait complete. 2023-10-30T12:28:53.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2897825713 wait complete. 2023-10-30T12:28:53.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7fa3a27fc640 1 --2- 172.21.15.125:0/934405489 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa3900665a0 0x7fa390068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7fa3a27fc640 1 -- 172.21.15.125:0/934405489 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3a41211e0 con 0x7fa3ac06aa00 2023-10-30T12:28:53.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7f8b35ffb640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b280665a0 0x7f8b28068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7f8b35ffb640 1 -- 172.21.15.125:0/1223143371 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b40083320 con 0x7f8b4406ab20 2023-10-30T12:28:53.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7fa3b16b8640 1 --2- 172.21.15.125:0/934405489 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa3900665a0 0x7fa390068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7f8b4891b640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b280665a0 0x7f8b28068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7f127913e640 1 -- start start 2023-10-30T12:28:53.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7fa39a4a9640 1 Processor -- start 2023-10-30T12:28:53.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7f1675279640 1 Processor -- start 2023-10-30T12:28:53.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7fa3b16b8640 1 --2- 172.21.15.125:0/934405489 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa3900665a0 0x7fa390068a60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa39c012fd0 tx=0x7fa39c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7f8b4891b640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b280665a0 0x7f8b28068a60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8b380b2000 tx=0x7f8b38008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7fa39a4a9640 1 -- start start 2023-10-30T12:28:53.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7f127913e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12740c4430 0x7f1274143db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.399+0000 7f1675279640 1 -- start start 2023-10-30T12:28:53.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f127913e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1274153c60 0x7f12741633b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f127913e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1274159830 0x7f12741638f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f1272d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12740c4430 0x7f1274143db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f1272d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12740c4430 0x7f1274143db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38916/0 (socket says 172.21.15.125:38916) 2023-10-30T12:28:53.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f127913e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f127414e510 con 0x7f1274159830 2023-10-30T12:28:53.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f127913e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12740ce3d0 con 0x7f12740c4430 2023-10-30T12:28:53.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f127913e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12741f5be0 con 0x7f1274153c60 2023-10-30T12:28:53.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f127259c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1274153c60 0x7f12741633b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f127259c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1274153c60 0x7f12741633b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38852/0 (socket says 172.21.15.125:38852) 2023-10-30T12:28:53.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f1675279640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16700c4430 0x7f16700c6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fc3937fe640 1 --2- 172.21.15.125:0/286914669 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc3740664a0 0x7fc374068960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0ad000 tx=0x7fc37c0a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f1272d9d640 1 -- 172.21.15.125:0/1944020790 learned_addr learned my addr 172.21.15.125:0/1944020790 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa39a4a9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3940c4430 0x7fa3941f9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa39a4a9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa394153c60 0x7fa394152740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa39a4a9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa394159830 0x7fa39413eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa39a4a9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3941409d0 con 0x7fa3940c4430 2023-10-30T12:28:53.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa39a4a9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3941f9f70 con 0x7fa394159830 2023-10-30T12:28:53.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa39a4a9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39406f260 con 0x7fa394153c60 2023-10-30T12:28:53.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f166effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16700c4430 0x7f16700c6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f166effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16700c4430 0x7f16700c6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33196/0 (socket says 172.21.15.125:33196) 2023-10-30T12:28:53.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f1675279640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1670153c60 0x7f16701443e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f1675279640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1670159830 0x7f16700c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa38bfff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa394153c60 0x7fa394152740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa38bfff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa394153c60 0x7fa394152740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38862/0 (socket says 172.21.15.125:38862) 2023-10-30T12:28:53.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa38bfff640 1 -- 172.21.15.125:0/2581999175 learned_addr learned my addr 172.21.15.125:0/2581999175 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f1675279640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16700b21a0 con 0x7f16700c4430 2023-10-30T12:28:53.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f1675279640 1 -- 172.21.15.125:0/3930509351 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16700c74d0 con 0x7f1670153c60 2023-10-30T12:28:53.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f166effd640 1 -- 172.21.15.125:0/3930509351 learned_addr learned my addr 172.21.15.125:0/3930509351 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:53.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f1675279640 1 -- 172.21.15.125:0/3930509351 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1670145bb0 con 0x7f1670159830 2023-10-30T12:28:53.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa398a46640 1 --2- 172.21.15.125:0/2581999175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa394159830 0x7fa39413eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa393fff640 1 --2- 172.21.15.125:0/2581999175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3940c4430 0x7fa3941f9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f127259c640 1 -- 172.21.15.125:0/1944020790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12740c4430 msgr2=0x7f1274143db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f127259c640 1 --2- 172.21.15.125:0/1944020790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12740c4430 0x7f1274143db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f127259c640 1 -- 172.21.15.125:0/1944020790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1274159830 msgr2=0x7f12741638f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f127259c640 1 --2- 172.21.15.125:0/1944020790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1274159830 0x7f12741638f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f127259c640 1 -- 172.21.15.125:0/1944020790 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c009420 con 0x7f1274153c60 2023-10-30T12:28:53.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f1667fff640 1 --2- 172.21.15.125:0/3930509351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1670153c60 0x7f16701443e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f166f7fe640 1 --2- 172.21.15.125:0/3930509351 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1670159830 0x7f16700c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa393fff640 1 -- 172.21.15.125:0/2581999175 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa394153c60 msgr2=0x7fa394152740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa393fff640 1 --2- 172.21.15.125:0/2581999175 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa394153c60 0x7fa394152740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa393fff640 1 -- 172.21.15.125:0/2581999175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa394159830 msgr2=0x7fa39413eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa393fff640 1 --2- 172.21.15.125:0/2581999175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa394159830 0x7fa39413eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fa393fff640 1 -- 172.21.15.125:0/2581999175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3840094d0 con 0x7fa3940c4430 2023-10-30T12:28:53.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7fc390ff9640 1 -- 172.21.15.125:0/286914669 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc384050860 con 0x7fc3940c4430 2023-10-30T12:28:53.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f166effd640 1 -- 172.21.15.125:0/3930509351 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1670159830 msgr2=0x7f16700c86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f166effd640 1 --2- 172.21.15.125:0/3930509351 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1670159830 0x7f16700c86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f166effd640 1 -- 172.21.15.125:0/3930509351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1670153c60 msgr2=0x7f16701443e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f166effd640 1 --2- 172.21.15.125:0/3930509351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1670153c60 0x7f16701443e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f166effd640 1 -- 172.21.15.125:0/3930509351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1668009b30 con 0x7f16700c4430 2023-10-30T12:28:53.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.403+0000 7f127259c640 1 --2- 172.21.15.125:0/1944020790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1274153c60 0x7f12741633b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f126c0ab000 tx=0x7f126c019240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.407+0000 7f166effd640 1 --2- 172.21.15.125:0/3930509351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16700c4430 0x7f16700c6a50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f165800e960 tx=0x7f165800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.407+0000 7fa393fff640 1 --2- 172.21.15.125:0/2581999175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3940c4430 0x7fa3941f9a30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa3940ccb30 tx=0x7fa384019300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.407+0000 7fa3917fa640 1 -- 172.21.15.125:0/2581999175 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3840b4460 con 0x7fa3940c4430 2023-10-30T12:28:53.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.407+0000 7f16677fe640 1 -- 172.21.15.125:0/3930509351 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1658009690 con 0x7f16700c4430 2023-10-30T12:28:53.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.407+0000 7f16677fe640 1 -- 172.21.15.125:0/3930509351 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165800ca00 con 0x7f16700c4430 2023-10-30T12:28:53.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.407+0000 7f16677fe640 1 -- 172.21.15.125:0/3930509351 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165800cd70 con 0x7f16700c4430 2023-10-30T12:28:53.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.407+0000 7fa3917fa640 1 -- 172.21.15.125:0/2581999175 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3840b45e0 con 0x7fa3940c4430 2023-10-30T12:28:53.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.407+0000 7fa3917fa640 1 -- 172.21.15.125:0/2581999175 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3840b4990 con 0x7fa3940c4430 2023-10-30T12:28:53.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.407+0000 7f1675279640 1 -- 172.21.15.125:0/3930509351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16700b31e0 con 0x7f16700c4430 2023-10-30T12:28:53.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.407+0000 7f1675279640 1 -- 172.21.15.125:0/3930509351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16700c6f90 con 0x7f16700c4430 2023-10-30T12:28:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.407+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2581999175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa394070da0 con 0x7fa3940c4430 2023-10-30T12:28:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.407+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2581999175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39406f500 con 0x7fa3940c4430 2023-10-30T12:28:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.407+0000 7f1675279640 1 -- 172.21.15.125:0/3930509351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16700c8300 con 0x7f16700c4430 2023-10-30T12:28:53.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.407+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2581999175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39404e170 con 0x7fa3940c4430 2023-10-30T12:28:53.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.411+0000 7fa3917fa640 1 -- 172.21.15.125:0/2581999175 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3840b2070 con 0x7fa3940c4430 2023-10-30T12:28:53.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.411+0000 7fa3917fa640 1 --2- 172.21.15.125:0/2581999175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa374066520 0x7fa3740689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.411+0000 7f16677fe640 1 -- 172.21.15.125:0/3930509351 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1658013470 con 0x7f16700c4430 2023-10-30T12:28:53.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.411+0000 7f126b7fe640 1 -- 172.21.15.125:0/1944020790 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c003ab0 con 0x7f1274153c60 2023-10-30T12:28:53.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.411+0000 7f126b7fe640 1 -- 172.21.15.125:0/1944020790 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f126c003c10 con 0x7f1274153c60 2023-10-30T12:28:53.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.411+0000 7f126b7fe640 1 -- 172.21.15.125:0/1944020790 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f126c0ab400 con 0x7f1274153c60 2023-10-30T12:28:53.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.411+0000 7fa38bfff640 1 --2- 172.21.15.125:0/2581999175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa374066520 0x7fa3740689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.411+0000 7f127913e640 1 -- 172.21.15.125:0/1944020790 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12740ce810 con 0x7f1274153c60 2023-10-30T12:28:53.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.411+0000 7f127913e640 1 -- 172.21.15.125:0/1944020790 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12740cedd0 con 0x7f1274153c60 2023-10-30T12:28:53.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.415+0000 7f12697fa640 1 -- 172.21.15.125:0/1944020790 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12741537b0 con 0x7f1274153c60 2023-10-30T12:28:53.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.411+0000 7f16677fe640 1 --2- 172.21.15.125:0/3930509351 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f16500664f0 0x7f16500689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.411+0000 7f16677fe640 1 -- 172.21.15.125:0/3930509351 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1658084830 con 0x7f16700c4430 2023-10-30T12:28:53.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.415+0000 7fa3a27fc640 1 -- 172.21.15.125:0/934405489 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa3a40ee670 con 0x7fa3ac06aa00 2023-10-30T12:28:53.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.415+0000 7f8b35ffb640 1 -- 172.21.15.125:0/1223143371 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8b400508c0 con 0x7f8b4406ab20 2023-10-30T12:28:53.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.415+0000 7f1667fff640 1 --2- 172.21.15.125:0/3930509351 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f16500664f0 0x7f16500689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.415+0000 7f1667fff640 1 --2- 172.21.15.125:0/3930509351 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f16500664f0 0x7f16500689b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1670157430 tx=0x7f1660008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.419+0000 7f16677fe640 1 -- 172.21.15.125:0/3930509351 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1658051d40 con 0x7f16700c4430 2023-10-30T12:28:53.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.415+0000 7fa38bfff640 1 --2- 172.21.15.125:0/2581999175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa374066520 0x7fa3740689e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa3940b43d0 tx=0x7fa37c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.415+0000 7f126b7fe640 1 -- 172.21.15.125:0/1944020790 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f126c0b2040 con 0x7f1274153c60 2023-10-30T12:28:53.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.415+0000 7f126b7fe640 1 --2- 172.21.15.125:0/1944020790 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12540664a0 0x7f1254068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:53.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.415+0000 7f126b7fe640 1 -- 172.21.15.125:0/1944020790 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f126c11e890 con 0x7f1274153c60 2023-10-30T12:28:53.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.419+0000 7f1272d9d640 1 --2- 172.21.15.125:0/1944020790 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12540664a0 0x7f1254068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:53.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.419+0000 7fa3917fa640 1 -- 172.21.15.125:0/2581999175 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3840efe20 con 0x7fa3940c4430 2023-10-30T12:28:53.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.423+0000 7f1272d9d640 1 --2- 172.21.15.125:0/1944020790 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12540664a0 0x7f1254068960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1264007fd0 tx=0x7f1264009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:53.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.427+0000 7fa3917fa640 1 -- 172.21.15.125:0/2581999175 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa3840ec1f0 con 0x7fa3940c4430 2023-10-30T12:28:53.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.427+0000 7f126b7fe640 1 -- 172.21.15.125:0/1944020790 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f126c0ebcf0 con 0x7f1274153c60 2023-10-30T12:28:53.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.719+0000 7f5eb5a17640 1 -- 172.21.15.125:0/1513965327 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5ea804bc20 con 0x7f5ea80ca7f0 2023-10-30T12:28:53.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.719+0000 7f5120d65640 1 -- 172.21.15.125:0/3818127730 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f511c147cd0 con 0x7f511c0cf4d0 2023-10-30T12:28:53.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.723+0000 7f510affd640 1 -- 172.21.15.125:0/3818127730 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f511c147cd0 con 0x7f511c0cf4d0 2023-10-30T12:28:53.725 INFO:teuthology.orchestra.run.smithi125.stdout:68719476743 2023-10-30T12:28:53.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.723+0000 7f5e97fff640 1 -- 172.21.15.125:0/1513965327 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5ea0054a70 con 0x7f5ea80ca7f0 2023-10-30T12:28:53.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.723+0000 7fc399c09640 1 -- 172.21.15.125:0/286914669 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc39404e170 con 0x7fc3940c4430 2023-10-30T12:28:53.726 INFO:teuthology.orchestra.run.smithi125.stdout:68719476742 2023-10-30T12:28:53.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.723+0000 7fc390ff9640 1 -- 172.21.15.125:0/286914669 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc384053e80 con 0x7fc3940c4430 2023-10-30T12:28:53.727 INFO:teuthology.orchestra.run.smithi125.stdout:64424509447 2023-10-30T12:28:53.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5120d65640 1 -- 172.21.15.125:0/3818127730 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f50fc0665a0 msgr2=0x7f50fc068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5120d65640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f50fc0665a0 0x7f50fc068a60 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f510c005890 tx=0x7f510c009040 comp rx=0 tx=0).stop 2023-10-30T12:28:53.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5120d65640 1 -- 172.21.15.125:0/3818127730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f511c0cf4d0 msgr2=0x7f511c1484f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5120d65640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f511c0cf4d0 0x7f511c1484f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f511001a320 tx=0x7f511001af30 comp rx=0 tx=0).stop 2023-10-30T12:28:53.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5120d65640 1 -- 172.21.15.125:0/3818127730 shutdown_connections 2023-10-30T12:28:53.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5120d65640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f50fc0665a0 0x7f50fc068a60 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f510c005890 tx=0x7f510c009040 comp rx=0 tx=0).stop 2023-10-30T12:28:53.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5120d65640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f511c0cf4d0 0x7f511c1484f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5120d65640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f511c0c90d0 0x7f511c147fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5120d65640 1 --2- 172.21.15.125:0/3818127730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f511c0c8cf0 0x7f511c0c77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5120d65640 1 -- 172.21.15.125:0/3818127730 >> 172.21.15.125:0/3818127730 conn(0x7f511c06be40 msgr2=0x7f511c1576d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5120d65640 1 -- 172.21.15.125:0/3818127730 shutdown_connections 2023-10-30T12:28:53.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5120d65640 1 -- 172.21.15.125:0/3818127730 wait complete. 2023-10-30T12:28:53.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7fc399c09640 1 -- 172.21.15.125:0/286914669 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc3740664a0 msgr2=0x7fc374068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7fc399c09640 1 --2- 172.21.15.125:0/286914669 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc3740664a0 0x7fc374068960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc37c0ad000 tx=0x7fc37c0a6000 comp rx=0 tx=0).stop 2023-10-30T12:28:53.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7fc399c09640 1 -- 172.21.15.125:0/286914669 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3940c4430 msgr2=0x7fc394168650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7fc399c09640 1 --2- 172.21.15.125:0/286914669 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3940c4430 0x7fc394168650 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc38400c4d0 tx=0x7fc38400c990 comp rx=0 tx=0).stop 2023-10-30T12:28:53.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7fc399c09640 1 -- 172.21.15.125:0/286914669 shutdown_connections 2023-10-30T12:28:53.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7fc399c09640 1 --2- 172.21.15.125:0/286914669 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc3740664a0 0x7fc374068960 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7fc399c09640 1 --2- 172.21.15.125:0/286914669 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3940cb280 0x7fc394148100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7fc399c09640 1 --2- 172.21.15.125:0/286914669 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3940caea0 0x7fc394168b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7fc399c09640 1 --2- 172.21.15.125:0/286914669 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3940c4430 0x7fc394168650 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7fc399c09640 1 -- 172.21.15.125:0/286914669 >> 172.21.15.125:0/286914669 conn(0x7fc39406bfe0 msgr2=0x7fc394149a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5eb5a17640 1 -- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e9c066500 msgr2=0x7f5e9c0689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5eb5a17640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e9c066500 0x7f5e9c0689c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400b950 tx=0x7f5ea4008040 comp rx=0 tx=0).stop 2023-10-30T12:28:53.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5eb5a17640 1 -- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea80ca7f0 msgr2=0x7f5ea80cb340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5eb5a17640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea80ca7f0 0x7f5ea80cb340 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5ea000c5b0 tx=0x7f5ea000ca70 comp rx=0 tx=0).stop 2023-10-30T12:28:53.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5eb5a17640 1 -- 172.21.15.125:0/1513965327 shutdown_connections 2023-10-30T12:28:53.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5eb5a17640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e9c066500 0x7f5e9c0689c0 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400b950 tx=0x7f5ea4008040 comp rx=0 tx=0).stop 2023-10-30T12:28:53.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5eb5a17640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ea80cb880 0x7f5ea80d8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5eb5a17640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea80ca7f0 0x7f5ea80cb340 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7fc399c09640 1 -- 172.21.15.125:0/286914669 shutdown_connections 2023-10-30T12:28:53.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5eb5a17640 1 --2- 172.21.15.125:0/1513965327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e98004d10 0x7f5ea80cc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7f5eb5a17640 1 -- 172.21.15.125:0/1513965327 >> 172.21.15.125:0/1513965327 conn(0x7f5ea80501d0 msgr2=0x7f5ea80cf670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.727+0000 7fc399c09640 1 -- 172.21.15.125:0/286914669 wait complete. 2023-10-30T12:28:53.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.731+0000 7f5eb5a17640 1 -- 172.21.15.125:0/1513965327 shutdown_connections 2023-10-30T12:28:53.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.731+0000 7f5eb5a17640 1 -- 172.21.15.125:0/1513965327 wait complete. 2023-10-30T12:28:53.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.735+0000 7fa3b411d640 1 -- 172.21.15.125:0/934405489 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fa3ac066cf0 con 0x7fa3ac06aa00 2023-10-30T12:28:53.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.735+0000 7fa3a27fc640 1 -- 172.21.15.125:0/934405489 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa3a40f1c90 con 0x7fa3ac06aa00 2023-10-30T12:28:53.739 INFO:teuthology.orchestra.run.smithi125.stdout:68719476744 2023-10-30T12:28:53.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.735+0000 7f8b4a11e640 1 -- 172.21.15.125:0/1223143371 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f8b440c17b0 con 0x7f8b4406ab20 2023-10-30T12:28:53.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.735+0000 7f8b35ffb640 1 -- 172.21.15.125:0/1223143371 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8b40017020 con 0x7f8b4406ab20 2023-10-30T12:28:53.741 INFO:teuthology.orchestra.run.smithi125.stdout:64424509448 2023-10-30T12:28:53.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7fa3b411d640 1 -- 172.21.15.125:0/934405489 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa3900665a0 msgr2=0x7fa390068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7fa3b411d640 1 --2- 172.21.15.125:0/934405489 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa3900665a0 0x7fa390068a60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa39c012fd0 tx=0x7fa39c009040 comp rx=0 tx=0).stop 2023-10-30T12:28:53.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7fa3b411d640 1 -- 172.21.15.125:0/934405489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3ac06aa00 msgr2=0x7fa3ac0adb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7fa3b411d640 1 --2- 172.21.15.125:0/934405489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3ac06aa00 0x7fa3ac0adb30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa3a40ae000 tx=0x7fa3a401bfb0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7f1675279640 1 -- 172.21.15.125:0/3930509351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f167004e170 con 0x7f16700c4430 2023-10-30T12:28:53.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7fa3b411d640 1 -- 172.21.15.125:0/934405489 shutdown_connections 2023-10-30T12:28:53.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7fa3b411d640 1 --2- 172.21.15.125:0/934405489 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa3900665a0 0x7fa390068a60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7fa3b411d640 1 --2- 172.21.15.125:0/934405489 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3ac0c5810 0x7fa3ac0b04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7fa3b411d640 1 --2- 172.21.15.125:0/934405489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3ac072040 0x7fa3ac0aff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7fa3b411d640 1 --2- 172.21.15.125:0/934405489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3ac06aa00 0x7fa3ac0adb30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7fa3b411d640 1 -- 172.21.15.125:0/934405489 >> 172.21.15.125:0/934405489 conn(0x7fa3ac0c7910 msgr2=0x7fa3ac0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7fa3b411d640 1 -- 172.21.15.125:0/934405489 shutdown_connections 2023-10-30T12:28:53.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7fa3b411d640 1 -- 172.21.15.125:0/934405489 wait complete. 2023-10-30T12:28:53.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7f16677fe640 1 -- 172.21.15.125:0/3930509351 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1658055360 con 0x7f16700c4430 2023-10-30T12:28:53.745 INFO:teuthology.orchestra.run.smithi125.stdout:68719476743 2023-10-30T12:28:53.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7f8b4a11e640 1 -- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b280665a0 msgr2=0x7f8b28068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7f8b4a11e640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b280665a0 0x7f8b28068a60 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8b380b2000 tx=0x7f8b38008040 comp rx=0 tx=0).stop 2023-10-30T12:28:53.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7f8b4a11e640 1 -- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b4406ab20 msgr2=0x7f8b441e6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7f8b4a11e640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b4406ab20 0x7f8b441e6e90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8b4000c490 tx=0x7f8b4000c950 comp rx=0 tx=0).stop 2023-10-30T12:28:53.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7f8b4a11e640 1 -- 172.21.15.125:0/1223143371 shutdown_connections 2023-10-30T12:28:53.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.739+0000 7f8b4a11e640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b280665a0 0x7f8b28068a60 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f8b4a11e640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b44157c80 0x7f8b44154e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f8b4a11e640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b4406fef0 0x7f8b440c5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f8b4a11e640 1 --2- 172.21.15.125:0/1223143371 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b4406ab20 0x7f8b441e6e90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f8b4a11e640 1 -- 172.21.15.125:0/1223143371 >> 172.21.15.125:0/1223143371 conn(0x7f8b440c75d0 msgr2=0x7f8b440c79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f8b4a11e640 1 -- 172.21.15.125:0/1223143371 shutdown_connections 2023-10-30T12:28:53.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f8b4a11e640 1 -- 172.21.15.125:0/1223143371 wait complete. 2023-10-30T12:28:53.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f1675279640 1 -- 172.21.15.125:0/3930509351 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f16500664f0 msgr2=0x7f16500689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f1675279640 1 --2- 172.21.15.125:0/3930509351 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f16500664f0 0x7f16500689b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1670157430 tx=0x7f1660008040 comp rx=0 tx=0).stop 2023-10-30T12:28:53.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f1675279640 1 -- 172.21.15.125:0/3930509351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16700c4430 msgr2=0x7f16700c6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f1675279640 1 --2- 172.21.15.125:0/3930509351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16700c4430 0x7f16700c6a50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f165800e960 tx=0x7f165800ee20 comp rx=0 tx=0).stop 2023-10-30T12:28:53.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f1675279640 1 -- 172.21.15.125:0/3930509351 shutdown_connections 2023-10-30T12:28:53.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f1675279640 1 --2- 172.21.15.125:0/3930509351 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f16500664f0 0x7f16500689b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f1675279640 1 --2- 172.21.15.125:0/3930509351 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1670159830 0x7f16700c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f1675279640 1 --2- 172.21.15.125:0/3930509351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1670153c60 0x7f16701443e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f1675279640 1 --2- 172.21.15.125:0/3930509351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16700c4430 0x7f16700c6a50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.743+0000 7f1675279640 1 -- 172.21.15.125:0/3930509351 >> 172.21.15.125:0/3930509351 conn(0x7f167006bfe0 msgr2=0x7f1670151b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.747+0000 7f1675279640 1 -- 172.21.15.125:0/3930509351 shutdown_connections 2023-10-30T12:28:53.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.747+0000 7f1675279640 1 -- 172.21.15.125:0/3930509351 wait complete. 2023-10-30T12:28:53.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.751+0000 7f12697fa640 1 -- 172.21.15.125:0/1944020790 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f12740ce560 con 0x7f1274153c60 2023-10-30T12:28:53.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.751+0000 7f126b7fe640 1 -- 172.21.15.125:0/1944020790 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f126c0b0090 con 0x7f1274153c60 2023-10-30T12:28:53.756 INFO:teuthology.orchestra.run.smithi125.stdout:60129542149 2023-10-30T12:28:53.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.755+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2581999175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fa3940caa20 con 0x7fa3940c4430 2023-10-30T12:28:53.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.755+0000 7fa3917fa640 1 -- 172.21.15.125:0/2581999175 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa3840b0070 con 0x7fa3940c4430 2023-10-30T12:28:53.758 INFO:teuthology.orchestra.run.smithi125.stdout:60129542150 2023-10-30T12:28:53.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.759+0000 7f12697fa640 1 -- 172.21.15.125:0/1944020790 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12540664a0 msgr2=0x7f1254068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.759+0000 7f12697fa640 1 --2- 172.21.15.125:0/1944020790 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12540664a0 0x7f1254068960 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1264007fd0 tx=0x7f1264009040 comp rx=0 tx=0).stop 2023-10-30T12:28:53.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.759+0000 7f12697fa640 1 -- 172.21.15.125:0/1944020790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1274153c60 msgr2=0x7f12741633b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.759+0000 7f12697fa640 1 --2- 172.21.15.125:0/1944020790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1274153c60 0x7f12741633b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f126c0ab000 tx=0x7f126c019240 comp rx=0 tx=0).stop 2023-10-30T12:28:53.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.759+0000 7f12697fa640 1 -- 172.21.15.125:0/1944020790 shutdown_connections 2023-10-30T12:28:53.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.759+0000 7f12697fa640 1 --2- 172.21.15.125:0/1944020790 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12540664a0 0x7f1254068960 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.759+0000 7f12697fa640 1 --2- 172.21.15.125:0/1944020790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1274159830 0x7f12741638f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.759+0000 7f12697fa640 1 --2- 172.21.15.125:0/1944020790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1274153c60 0x7f12741633b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.759+0000 7f12697fa640 1 --2- 172.21.15.125:0/1944020790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12740c4430 0x7f1274143db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.759+0000 7f12697fa640 1 -- 172.21.15.125:0/1944020790 >> 172.21.15.125:0/1944020790 conn(0x7f127406bfe0 msgr2=0x7f1274149fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.759+0000 7f12697fa640 1 -- 172.21.15.125:0/1944020790 shutdown_connections 2023-10-30T12:28:53.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.759+0000 7f12697fa640 1 -- 172.21.15.125:0/1944020790 wait complete. 2023-10-30T12:28:53.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.763+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2581999175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa374066520 msgr2=0x7fa3740689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.763+0000 7fa39a4a9640 1 --2- 172.21.15.125:0/2581999175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa374066520 0x7fa3740689e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa3940b43d0 tx=0x7fa37c009190 comp rx=0 tx=0).stop 2023-10-30T12:28:53.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.763+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2581999175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3940c4430 msgr2=0x7fa3941f9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:53.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.763+0000 7fa39a4a9640 1 --2- 172.21.15.125:0/2581999175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3940c4430 0x7fa3941f9a30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa3940ccb30 tx=0x7fa384019300 comp rx=0 tx=0).stop 2023-10-30T12:28:53.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.763+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2581999175 shutdown_connections 2023-10-30T12:28:53.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.763+0000 7fa39a4a9640 1 --2- 172.21.15.125:0/2581999175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa374066520 0x7fa3740689e0 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa3940b43d0 tx=0x7fa37c009190 comp rx=0 tx=0).stop 2023-10-30T12:28:53.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.763+0000 7fa39a4a9640 1 --2- 172.21.15.125:0/2581999175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa394159830 0x7fa39413eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.763+0000 7fa39a4a9640 1 --2- 172.21.15.125:0/2581999175 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa394153c60 0x7fa394152740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.763+0000 7fa39a4a9640 1 --2- 172.21.15.125:0/2581999175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3940c4430 0x7fa3941f9a30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:53.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.763+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2581999175 >> 172.21.15.125:0/2581999175 conn(0x7fa39406bfe0 msgr2=0x7fa39406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:53.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.763+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2581999175 shutdown_connections 2023-10-30T12:28:53.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:53.763+0000 7fa39a4a9640 1 -- 172.21.15.125:0/2581999175 wait complete. 2023-10-30T12:28:53.774 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.6 2023-10-30T12:28:53.775 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476743 got 68719476743 for osd.0 2023-10-30T12:28:53.776 DEBUG:teuthology.parallel:result is None 2023-10-30T12:28:53.778 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476742 got 68719476742 for osd.1 2023-10-30T12:28:53.778 DEBUG:teuthology.parallel:result is None 2023-10-30T12:28:53.786 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476744 got 68719476744 for osd.3 2023-10-30T12:28:53.786 DEBUG:teuthology.parallel:result is None 2023-10-30T12:28:53.788 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.4 2023-10-30T12:28:53.790 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476744 got 68719476743 for osd.2 2023-10-30T12:28:53.793 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.5 2023-10-30T12:28:53.796 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.7 2023-10-30T12:28:54.774 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T12:28:54.788 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T12:28:54.791 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T12:28:54.794 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T12:28:54.798 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T12:28:54.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.975+0000 7fd5ca817640 1 Processor -- start 2023-10-30T12:28:54.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.975+0000 7fd5ca817640 1 -- start start 2023-10-30T12:28:54.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7fd5ca817640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd5c4153c60 0x7fd5c40ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:54.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7fd5ca817640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd5c40c4430 0x7fd5c40cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:54.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7fd5ca817640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5c40cb780 con 0x7fd5c4159830 2023-10-30T12:28:54.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7fd5ca817640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5c4154040 con 0x7fd5c40c4430 2023-10-30T12:28:54.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7fd5ca817640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5c40cc9f0 con 0x7fd5c4153c60 2023-10-30T12:28:54.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7fd5c37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd5c40c4430 0x7fd5c40cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:54.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7f7c86b1d640 1 Processor -- start 2023-10-30T12:28:54.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7fd5c37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd5c40c4430 0x7fd5c40cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33210/0 (socket says 172.21.15.125:33210) 2023-10-30T12:28:54.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7fd5c37fe640 1 -- 172.21.15.125:0/2498394469 learned_addr learned my addr 172.21.15.125:0/2498394469 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:54.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7fd5c3fff640 1 --2- 172.21.15.125:0/2498394469 >> v2:172.21.15.136:3300/0 conn(0x7fd5c4153c60 0x7fd5c40ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:54.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7f7c86b1d640 1 -- start start 2023-10-30T12:28:54.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7fd5c37fe640 1 -- 172.21.15.125:0/2498394469 >> v2:172.21.15.136:3300/0 conn(0x7fd5c4153c60 msgr2=0x7fd5c40ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:54.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7fd5c37fe640 1 --2- 172.21.15.125:0/2498394469 >> v2:172.21.15.136:3300/0 conn(0x7fd5c4153c60 0x7fd5c40ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7fd5c37fe640 1 -- 172.21.15.125:0/2498394469 >> v1:172.21.15.136:6789/0 conn(0x7fd5c4159830 legacy=0x7fd5c40cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:54.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7fd5c37fe640 1 -- 172.21.15.125:0/2498394469 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c415a2e0 con 0x7fd5c40c4430 2023-10-30T12:28:54.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7f7c86b1d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7c80153c60 0x7f7c800ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:54.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7f7c86b1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c800c4430 0x7f7c800cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:54.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7f7c86b1d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7c80159830 0x7f7c80159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:54.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7f7c86b1d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c80154040 con 0x7f7c80159830 2023-10-30T12:28:54.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7f7c86b1d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c800cb780 con 0x7f7c80153c60 2023-10-30T12:28:54.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7f7c86b1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c800cc9f0 con 0x7f7c800c4430 2023-10-30T12:28:54.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7f7c850ba640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7c80159830 0x7f7c80159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:54.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7f7c7ffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c800c4430 0x7f7c800cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:54.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7f7c848b9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7c80153c60 0x7f7c800ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:54.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.983+0000 7f7c7ffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c800c4430 0x7f7c800cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38888/0 (socket says 172.21.15.125:38888) 2023-10-30T12:28:54.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.983+0000 7f7c7ffff640 1 -- 172.21.15.125:0/3970791577 learned_addr learned my addr 172.21.15.125:0/3970791577 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:54.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.979+0000 7f7c850ba640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7c80159830 0x7f7c80159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33220/0 (socket says 172.21.15.125:33220) 2023-10-30T12:28:54.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.983+0000 7f7c850ba640 1 -- 172.21.15.125:0/3970791577 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c800c4430 msgr2=0x7f7c800cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:54.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.983+0000 7f7c850ba640 1 --2- 172.21.15.125:0/3970791577 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c800c4430 0x7f7c800cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.983+0000 7f7c850ba640 1 -- 172.21.15.125:0/3970791577 >> v2:172.21.15.136:3300/0 conn(0x7f7c80153c60 msgr2=0x7f7c800ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:54.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.983+0000 7f7c850ba640 1 --2- 172.21.15.125:0/3970791577 >> v2:172.21.15.136:3300/0 conn(0x7f7c80153c60 0x7f7c800ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.983+0000 7f7c850ba640 1 -- 172.21.15.125:0/3970791577 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8015a2e0 con 0x7f7c80159830 2023-10-30T12:28:54.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.983+0000 7f7c848b9640 1 --2- 172.21.15.125:0/3970791577 >> v2:172.21.15.136:3300/0 conn(0x7f7c80153c60 0x7f7c800ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:54.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.983+0000 7fd5c37fe640 1 --2- 172.21.15.125:0/2498394469 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5c40c4430 0x7fd5c40cc4b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4072ce0 tx=0x7fd5b801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491e2c071341a09f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:54.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.983+0000 7fd5c2ffd640 1 -- 172.21.15.125:0/2498394469 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b80ab600 con 0x7fd5c40c4430 2023-10-30T12:28:54.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.983+0000 7fd5c2ffd640 1 -- 172.21.15.125:0/2498394469 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b80a7050 con 0x7fd5c40c4430 2023-10-30T12:28:54.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.983+0000 7fd5c2ffd640 1 -- 172.21.15.125:0/2498394469 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b80ab980 con 0x7fd5c40c4430 2023-10-30T12:28:54.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.983+0000 7f09f6409640 1 Processor -- start 2023-10-30T12:28:54.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 -- 172.21.15.125:0/2498394469 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5c40c4430 msgr2=0x7fd5c40cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:54.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 --2- 172.21.15.125:0/2498394469 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5c40c4430 0x7fd5c40cc4b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4072ce0 tx=0x7fd5b801c500 comp rx=0 tx=0).stop 2023-10-30T12:28:54.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f7c850ba640 1 --2- 172.21.15.125:0/3970791577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c80159830 0x7f7c80159ce0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7c80072ce0 tx=0x7f7c7801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=858727f7428746e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:54.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 -- 172.21.15.125:0/2498394469 shutdown_connections 2023-10-30T12:28:54.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 --2- 172.21.15.125:0/2498394469 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5c40c4430 0x7fd5c40cc4b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 --2- 172.21.15.125:0/2498394469 >> v2:172.21.15.136:3300/0 conn(0x7fd5c4153c60 0x7fd5c40ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:54.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 -- 172.21.15.125:0/2498394469 >> 172.21.15.125:0/2498394469 conn(0x7fd5c406bfe0 msgr2=0x7fd5c406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:54.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09f6409640 1 -- start start 2023-10-30T12:28:54.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 -- 172.21.15.125:0/2498394469 shutdown_connections 2023-10-30T12:28:54.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 -- 172.21.15.125:0/2498394469 wait complete. 2023-10-30T12:28:54.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f7c7f7fe640 1 -- 172.21.15.125:0/3970791577 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c780a5880 con 0x7f7c80159830 2023-10-30T12:28:54.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f7c7f7fe640 1 -- 172.21.15.125:0/3970791577 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c780a59e0 con 0x7f7c80159830 2023-10-30T12:28:54.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f7c7f7fe640 1 -- 172.21.15.125:0/3970791577 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c780a5d70 con 0x7f7c80159830 2023-10-30T12:28:54.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09f6409640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09f00c4430 0x7f09f00ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:54.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f7c86b1d640 1 -- 172.21.15.125:0/3970791577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c80159830 msgr2=0x7f7c80159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:54.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f7c86b1d640 1 --2- 172.21.15.125:0/3970791577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c80159830 0x7f7c80159ce0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7c80072ce0 tx=0x7f7c7801c910 comp rx=0 tx=0).stop 2023-10-30T12:28:54.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09f6409640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f09f0159830 0x7f09f0159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:54.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09f6409640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f00cb780 con 0x7f09f0153c60 2023-10-30T12:28:54.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09f6409640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09f0154040 con 0x7f09f0159830 2023-10-30T12:28:54.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09f6409640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f015a2e0 con 0x7f09f00c4430 2023-10-30T12:28:54.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09effff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f09f0153c60 0x7f09f00cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:36984/0 (socket says 172.21.15.125:36984) 2023-10-30T12:28:54.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09effff640 1 -- 172.21.15.125:0/1862528933 learned_addr learned my addr 172.21.15.125:0/1862528933 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:54.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 Processor -- start 2023-10-30T12:28:54.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09f49a6640 1 --2- 172.21.15.125:0/1862528933 >> v2:172.21.15.136:3300/0 conn(0x7f09f0159830 0x7f09f0159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09e7fff640 1 --2- 172.21.15.125:0/1862528933 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09f00c4430 0x7f09f00ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f7c86b1d640 1 -- 172.21.15.125:0/3970791577 shutdown_connections 2023-10-30T12:28:55.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f7c86b1d640 1 --2- 172.21.15.125:0/3970791577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c80159830 0x7f7c80159ce0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f7c86b1d640 1 --2- 172.21.15.125:0/3970791577 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c800c4430 0x7f7c800cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f7c86b1d640 1 --2- 172.21.15.125:0/3970791577 >> v2:172.21.15.136:3300/0 conn(0x7f7c80153c60 0x7f7c800ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f7c86b1d640 1 -- 172.21.15.125:0/3970791577 >> 172.21.15.125:0/3970791577 conn(0x7f7c8006bfe0 msgr2=0x7f7c8006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:55.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f7c86b1d640 1 -- 172.21.15.125:0/3970791577 shutdown_connections 2023-10-30T12:28:55.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f7c86b1d640 1 -- 172.21.15.125:0/3970791577 wait complete. 2023-10-30T12:28:55.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 -- start start 2023-10-30T12:28:55.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09e7fff640 1 -- 172.21.15.125:0/1862528933 >> v2:172.21.15.136:3300/0 conn(0x7f09f0159830 msgr2=0x7f09f0159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:55.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09e7fff640 1 --2- 172.21.15.125:0/1862528933 >> v2:172.21.15.136:3300/0 conn(0x7f09f0159830 0x7f09f0159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09e7fff640 1 -- 172.21.15.125:0/1862528933 >> v1:172.21.15.136:6789/0 conn(0x7f09f0153c60 legacy=0x7f09f00cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09ef7fe640 1 -- 172.21.15.125:0/1862528933 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989576168 0 0) 0x7f09f00cb780 con 0x7f09f0153c60 2023-10-30T12:28:55.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7f09e7fff640 1 -- 172.21.15.125:0/1862528933 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f015a5c0 con 0x7f09f00c4430 2023-10-30T12:28:55.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5c40c4430 0x7fd5c4154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd5c4153c60 0x7fd5c41489a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c4159830 0x7fd5c4155ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c4148580 con 0x7fd5c40c4430 2023-10-30T12:28:55.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c41f7d00 con 0x7fd5c4159830 2023-10-30T12:28:55.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5ca817640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5c4149730 con 0x7fd5c4153c60 2023-10-30T12:28:55.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.987+0000 7fd5c3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5c40c4430 0x7fd5c4154df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5c3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5c40c4430 0x7fd5c4154df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33230/0 (socket says 172.21.15.125:33230) 2023-10-30T12:28:55.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5c3fff640 1 -- 172.21.15.125:0/2449332756 learned_addr learned my addr 172.21.15.125:0/2449332756 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:55.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5c8db4640 1 --2- 172.21.15.125:0/2449332756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c4159830 0x7fd5c4155ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5c3fff640 1 -- 172.21.15.125:0/2449332756 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd5c4153c60 msgr2=0x7fd5c41489a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5c3fff640 1 --2- 172.21.15.125:0/2449332756 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd5c4153c60 0x7fd5c41489a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5c3fff640 1 -- 172.21.15.125:0/2449332756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c4159830 msgr2=0x7fd5c4155ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5c3fff640 1 --2- 172.21.15.125:0/2449332756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c4159830 0x7fd5c4155ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5c3fff640 1 -- 172.21.15.125:0/2449332756 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b80094d0 con 0x7fd5c40c4430 2023-10-30T12:28:55.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5c3fff640 1 --2- 172.21.15.125:0/2449332756 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5c40c4430 0x7fd5c4154df0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400e930 tx=0x7fd5b400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5c0ff9640 1 -- 172.21.15.125:0/2449332756 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b400ca00 con 0x7fd5c40c4430 2023-10-30T12:28:55.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5c0ff9640 1 -- 172.21.15.125:0/2449332756 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b400cb60 con 0x7fd5c40c4430 2023-10-30T12:28:55.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5c0ff9640 1 -- 172.21.15.125:0/2449332756 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b4013470 con 0x7fd5c40c4430 2023-10-30T12:28:55.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7f7c86b1d640 1 Processor -- start 2023-10-30T12:28:55.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5ca817640 1 -- 172.21.15.125:0/2449332756 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5c41eb7c0 con 0x7fd5c40c4430 2023-10-30T12:28:55.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5ca817640 1 -- 172.21.15.125:0/2449332756 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5c4150fd0 con 0x7fd5c40c4430 2023-10-30T12:28:55.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7f7c86b1d640 1 -- start start 2023-10-30T12:28:55.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7fd5ca817640 1 -- 172.21.15.125:0/2449332756 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5c41ebce0 con 0x7fd5c40c4430 2023-10-30T12:28:55.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7f09e7fff640 1 --2- 172.21.15.125:0/1862528933 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09f00c4430 0x7f09f00ca360 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f09d80097c0 tx=0x7f09d801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8c5fe81da7b345d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7f09ef7fe640 1 -- 172.21.15.125:0/1862528933 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d80ab600 con 0x7f09f00c4430 2023-10-30T12:28:55.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7f09ef7fe640 1 -- 172.21.15.125:0/1862528933 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09d80a7050 con 0x7f09f00c4430 2023-10-30T12:28:55.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7f09ef7fe640 1 -- 172.21.15.125:0/1862528933 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d80ab980 con 0x7f09f00c4430 2023-10-30T12:28:55.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7f7c86b1d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c800c4430 0x7f7c800c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7f7c86b1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c80153c60 0x7f7c8014a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7f7c86b1d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c80159830 0x7f7c80152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7f7c86b1d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8014ab80 con 0x7f7c80159830 2023-10-30T12:28:55.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7f7c86b1d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c800b4890 con 0x7f7c800c4430 2023-10-30T12:28:55.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.991+0000 7f7c86b1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c80130640 con 0x7f7c80153c60 2023-10-30T12:28:55.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f7c850ba640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c80159830 0x7f7c80152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f7c848b9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c800c4430 0x7f7c800c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f7c848b9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c800c4430 0x7f7c800c8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38996/0 (socket says 172.21.15.125:38996) 2023-10-30T12:28:55.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f7c850ba640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c80159830 0x7f7c80152260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33240/0 (socket says 172.21.15.125:33240) 2023-10-30T12:28:55.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f7c848b9640 1 -- 172.21.15.125:0/3428995700 learned_addr learned my addr 172.21.15.125:0/3428995700 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:55.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f7c7ffff640 1 --2- 172.21.15.125:0/3428995700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c80153c60 0x7f7c8014a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f09f6409640 1 -- 172.21.15.125:0/1862528933 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09f00c4430 msgr2=0x7f09f00ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f09f6409640 1 --2- 172.21.15.125:0/1862528933 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09f00c4430 0x7f09f00ca360 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f09d80097c0 tx=0x7f09d801c480 comp rx=0 tx=0).stop 2023-10-30T12:28:55.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fdbaa89e640 1 Processor -- start 2023-10-30T12:28:55.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f7c850ba640 1 -- 172.21.15.125:0/3428995700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c80153c60 msgr2=0x7f7c8014a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f7c850ba640 1 --2- 172.21.15.125:0/3428995700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c80153c60 0x7f7c8014a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f7c850ba640 1 -- 172.21.15.125:0/3428995700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c800c4430 msgr2=0x7f7c800c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f7c850ba640 1 --2- 172.21.15.125:0/3428995700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c800c4430 0x7f7c800c8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f7c850ba640 1 -- 172.21.15.125:0/3428995700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c74009420 con 0x7f7c80159830 2023-10-30T12:28:55.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fdbaa89e640 1 -- start start 2023-10-30T12:28:55.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fd5c0ff9640 1 -- 172.21.15.125:0/2449332756 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd5b40135d0 con 0x7fd5c40c4430 2023-10-30T12:28:55.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f09f6409640 1 -- 172.21.15.125:0/1862528933 shutdown_connections 2023-10-30T12:28:55.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f09f6409640 1 --2- 172.21.15.125:0/1862528933 >> v2:172.21.15.136:3300/0 conn(0x7f09f0159830 0x7f09f0159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f09f6409640 1 --2- 172.21.15.125:0/1862528933 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09f00c4430 0x7f09f00ca360 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f09f6409640 1 -- 172.21.15.125:0/1862528933 >> 172.21.15.125:0/1862528933 conn(0x7f09f006bfe0 msgr2=0x7f09f006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:55.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f7c850ba640 1 --2- 172.21.15.125:0/3428995700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c80159830 0x7f7c80152260 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7c800ccd50 tx=0x7f7c780196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f09f6409640 1 -- 172.21.15.125:0/1862528933 shutdown_connections 2023-10-30T12:28:55.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f09f6409640 1 -- 172.21.15.125:0/1862528933 wait complete. 2023-10-30T12:28:55.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fdbaa89e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdba4153c60 0x7fdba40ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fdbaa89e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdba4159830 0x7fdba4159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fdbaa89e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdba40cb780 con 0x7fdba40c4430 2023-10-30T12:28:55.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fdbaa89e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdba4154040 con 0x7fdba4153c60 2023-10-30T12:28:55.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fdbaa89e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba415a2e0 con 0x7fdba4159830 2023-10-30T12:28:55.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fd5c0ff9640 1 --2- 172.21.15.125:0/2449332756 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd5b00664a0 0x7fd5b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fdba3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdba4153c60 0x7fdba40ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fdba37fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdba40c4430 0x7fdba40cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54134/0 (socket says 172.21.15.125:54134) 2023-10-30T12:28:55.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fdba37fe640 1 -- 172.21.15.125:0/3606208412 learned_addr learned my addr 172.21.15.125:0/3606208412 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:55.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fdba3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdba4153c60 0x7fdba40ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33250/0 (socket says 172.21.15.125:33250) 2023-10-30T12:28:55.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fdba8e3b640 1 --2- 172.21.15.125:0/3606208412 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdba4159830 0x7fdba4159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7fd5c0ff9640 1 -- 172.21.15.125:0/2449332756 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd5b4084070 con 0x7fd5c40c4430 2023-10-30T12:28:55.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f09f6409640 1 Processor -- start 2023-10-30T12:28:55.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.995+0000 7f09f6409640 1 -- start start 2023-10-30T12:28:55.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7fd5c37fe640 1 --2- 172.21.15.125:0/2449332756 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd5b00664a0 0x7fd5b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7fd5c37fe640 1 --2- 172.21.15.125:0/2449332756 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd5b00664a0 0x7fd5b0068960 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd5b80ad000 tx=0x7fd5b8004740 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7fdba3fff640 1 -- 172.21.15.125:0/3606208412 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdba4159830 msgr2=0x7fdba4159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:55.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7fdba3fff640 1 --2- 172.21.15.125:0/3606208412 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdba4159830 0x7fdba4159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7fdba3fff640 1 -- 172.21.15.125:0/3606208412 >> v1:172.21.15.125:6789/0 conn(0x7fdba40c4430 legacy=0x7fdba40cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7fdba3fff640 1 -- 172.21.15.125:0/3606208412 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdba415a5c0 con 0x7fdba4153c60 2023-10-30T12:28:55.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f6409640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09f00c4430 0x7f09f00ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f6409640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09f0153c60 0x7f09f0145ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f6409640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09f0159830 0x7f09f0155e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f6409640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f00b52b0 con 0x7f09f0159830 2023-10-30T12:28:55.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f6409640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f01559e0 con 0x7f09f0153c60 2023-10-30T12:28:55.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f6409640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09f00ce830 con 0x7f09f00c4430 2023-10-30T12:28:55.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f49a6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09f0159830 0x7f09f0155e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f49a6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09f0159830 0x7f09f0155e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33256/0 (socket says 172.21.15.125:33256) 2023-10-30T12:28:55.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f49a6640 1 -- 172.21.15.125:0/2588529362 learned_addr learned my addr 172.21.15.125:0/2588529362 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:55.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09e7fff640 1 --2- 172.21.15.125:0/2588529362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09f0153c60 0x7f09f0145ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f49a6640 1 -- 172.21.15.125:0/2588529362 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09f00c4430 msgr2=0x7f09f00ce2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-30T12:28:55.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f49a6640 1 --2- 172.21.15.125:0/2588529362 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09f00c4430 0x7f09f00ce2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f49a6640 1 -- 172.21.15.125:0/2588529362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09f0153c60 msgr2=0x7f09f0145ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f49a6640 1 --2- 172.21.15.125:0/2588529362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09f0153c60 0x7f09f0145ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f49a6640 1 -- 172.21.15.125:0/2588529362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09d8009420 con 0x7f09f0159830 2023-10-30T12:28:55.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f49a6640 1 --2- 172.21.15.125:0/2588529362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09f0159830 0x7f09f0155e00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009d30 tx=0x7f09e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f7c7d7fa640 1 -- 172.21.15.125:0/3428995700 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c780ad640 con 0x7f7c80159830 2023-10-30T12:28:55.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f7c7d7fa640 1 -- 172.21.15.125:0/3428995700 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c780ad7a0 con 0x7f7c80159830 2023-10-30T12:28:55.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f7c7d7fa640 1 -- 172.21.15.125:0/3428995700 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c780adb10 con 0x7f7c80159830 2023-10-30T12:28:55.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09ed7fa640 1 -- 172.21.15.125:0/2588529362 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8014480 con 0x7f09f0159830 2023-10-30T12:28:55.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f7c86b1d640 1 -- 172.21.15.125:0/3428995700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c78009a70 con 0x7f7c80159830 2023-10-30T12:28:55.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09ed7fa640 1 -- 172.21.15.125:0/2588529362 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09e800f040 con 0x7f09f0159830 2023-10-30T12:28:55.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09ed7fa640 1 -- 172.21.15.125:0/2588529362 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09e8014800 con 0x7f09f0159830 2023-10-30T12:28:55.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7fd5c0ff9640 1 -- 172.21.15.125:0/2449332756 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd5c41ebce0 con 0x7fd5c40c4430 2023-10-30T12:28:55.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f6409640 1 -- 172.21.15.125:0/2588529362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f00cdca0 con 0x7f09f0159830 2023-10-30T12:28:55.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f09f6409640 1 -- 172.21.15.125:0/2588529362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f0155320 con 0x7f09f0159830 2023-10-30T12:28:55.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:54.999+0000 7f7c86b1d640 1 -- 172.21.15.125:0/3428995700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c800b43a0 con 0x7f7c80159830 2023-10-30T12:28:55.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7fdba3fff640 1 --2- 172.21.15.125:0/3606208412 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdba4153c60 0x7fdba40ca360 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdb940097c0 tx=0x7fdb9401c4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f17afb99bc661e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7f09f6409640 1 -- 172.21.15.125:0/2588529362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f004e170 con 0x7f09f0159830 2023-10-30T12:28:55.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7fdba2ffd640 1 -- 172.21.15.125:0/3606208412 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb940ab600 con 0x7fdba4153c60 2023-10-30T12:28:55.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7f7c677fe640 1 -- 172.21.15.125:0/3428995700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8004e170 con 0x7f7c80159830 2023-10-30T12:28:55.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7fdba2ffd640 1 -- 172.21.15.125:0/3606208412 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb940a7050 con 0x7fdba4153c60 2023-10-30T12:28:55.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7fdba2ffd640 1 -- 172.21.15.125:0/3606208412 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb940ab980 con 0x7fdba4153c60 2023-10-30T12:28:55.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3606208412 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdba4153c60 msgr2=0x7fdba40ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7fdbaa89e640 1 --2- 172.21.15.125:0/3606208412 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdba4153c60 0x7fdba40ca360 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdb940097c0 tx=0x7fdb9401c4a0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3606208412 shutdown_connections 2023-10-30T12:28:55.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7fdbaa89e640 1 --2- 172.21.15.125:0/3606208412 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdba4159830 0x7fdba4159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7fdbaa89e640 1 --2- 172.21.15.125:0/3606208412 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdba4153c60 0x7fdba40ca360 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3606208412 >> 172.21.15.125:0/3606208412 conn(0x7fdba406bfe0 msgr2=0x7fdba406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:55.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3606208412 shutdown_connections 2023-10-30T12:28:55.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3606208412 wait complete. 2023-10-30T12:28:55.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7f7c7d7fa640 1 -- 172.21.15.125:0/3428995700 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7c780b4050 con 0x7f7c80159830 2023-10-30T12:28:55.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.003+0000 7f09ed7fa640 1 -- 172.21.15.125:0/2588529362 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f09e8014960 con 0x7f09f0159830 2023-10-30T12:28:55.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7f09ed7fa640 1 --2- 172.21.15.125:0/2588529362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f09d00664f0 0x7f09d00689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7f7c7d7fa640 1 --2- 172.21.15.125:0/3428995700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7c6c0665a0 0x7f7c6c068a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7f09ed7fa640 1 -- 172.21.15.125:0/2588529362 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09e80850a0 con 0x7f09f0159830 2023-10-30T12:28:55.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7f7c7d7fa640 1 -- 172.21.15.125:0/3428995700 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c7811e2e0 con 0x7f7c80159830 2023-10-30T12:28:55.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7f09effff640 1 --2- 172.21.15.125:0/2588529362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f09d00664f0 0x7f09d00689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7f7c848b9640 1 --2- 172.21.15.125:0/3428995700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7c6c0665a0 0x7f7c6c068a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7f09effff640 1 --2- 172.21.15.125:0/2588529362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f09d00664f0 0x7f09d00689b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f09e0000c10 tx=0x7f09e00078a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdbaa89e640 1 Processor -- start 2023-10-30T12:28:55.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7f7c848b9640 1 --2- 172.21.15.125:0/3428995700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7c6c0665a0 0x7f7c6c068a60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7c74009df0 tx=0x7f7c74009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdbaa89e640 1 -- start start 2023-10-30T12:28:55.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdbaa89e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdba40c4430 0x7fdba4070470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdbaa89e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdba4153c60 0x7fdba41513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdbaa89e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdba4159830 0x7fdba413b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdbaa89e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba4152d20 con 0x7fdba4159830 2023-10-30T12:28:55.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdbaa89e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdba414df10 con 0x7fdba4153c60 2023-10-30T12:28:55.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdbaa89e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdba4152fc0 con 0x7fdba40c4430 2023-10-30T12:28:55.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdba8e3b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdba4159830 0x7fdba413b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdba37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdba4153c60 0x7fdba41513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdba3fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdba40c4430 0x7fdba4070470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdba3fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdba40c4430 0x7fdba4070470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38930/0 (socket says 172.21.15.125:38930) 2023-10-30T12:28:55.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdba8e3b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdba4159830 0x7fdba413b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33258/0 (socket says 172.21.15.125:33258) 2023-10-30T12:28:55.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdba3fff640 1 -- 172.21.15.125:0/3736708712 learned_addr learned my addr 172.21.15.125:0/3736708712 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:55.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.007+0000 7fdba37fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdba4153c60 0x7fdba41513b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:39012/0 (socket says 172.21.15.125:39012) 2023-10-30T12:28:55.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.011+0000 7fdba8e3b640 1 -- 172.21.15.125:0/3736708712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdba40c4430 msgr2=0x7fdba4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.011+0000 7fdba8e3b640 1 --2- 172.21.15.125:0/3736708712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdba40c4430 0x7fdba4070470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.011+0000 7fdba8e3b640 1 -- 172.21.15.125:0/3736708712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdba4153c60 msgr2=0x7fdba41513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.011+0000 7fdba8e3b640 1 --2- 172.21.15.125:0/3736708712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdba4153c60 0x7fdba41513b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.011+0000 7fdba8e3b640 1 -- 172.21.15.125:0/3736708712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb94009420 con 0x7fdba4159830 2023-10-30T12:28:55.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.011+0000 7fdba8e3b640 1 --2- 172.21.15.125:0/3736708712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdba4159830 0x7fdba413b9c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00eba0 tx=0x7fdb9c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.011+0000 7f09ed7fa640 1 -- 172.21.15.125:0/2588529362 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f09e8052690 con 0x7f09f0159830 2023-10-30T12:28:55.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.019+0000 7fdba0ff9640 1 -- 172.21.15.125:0/3736708712 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c00cd20 con 0x7fdba4159830 2023-10-30T12:28:55.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.019+0000 7fdba0ff9640 1 -- 172.21.15.125:0/3736708712 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb9c00f040 con 0x7fdba4159830 2023-10-30T12:28:55.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.019+0000 7fdba0ff9640 1 -- 172.21.15.125:0/3736708712 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb9c0146c0 con 0x7fdba4159830 2023-10-30T12:28:55.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.019+0000 7f7c7d7fa640 1 -- 172.21.15.125:0/3428995700 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7c780eb6b0 con 0x7f7c80159830 2023-10-30T12:28:55.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.023+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3736708712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdba414e140 con 0x7fdba4159830 2023-10-30T12:28:55.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.023+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3736708712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdba4153450 con 0x7fdba4159830 2023-10-30T12:28:55.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.023+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3736708712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdba414dd40 con 0x7fdba4159830 2023-10-30T12:28:55.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.027+0000 7fdba0ff9640 1 -- 172.21.15.125:0/3736708712 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdb9c014820 con 0x7fdba4159830 2023-10-30T12:28:55.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.027+0000 7fdba0ff9640 1 --2- 172.21.15.125:0/3736708712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb90066520 0x7fdb900689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.027+0000 7fdba0ff9640 1 -- 172.21.15.125:0/3736708712 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdb9c056280 con 0x7fdba4159830 2023-10-30T12:28:55.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.031+0000 7fdba3fff640 1 --2- 172.21.15.125:0/3736708712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb90066520 0x7fdb900689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.031+0000 7fdba3fff640 1 --2- 172.21.15.125:0/3736708712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb90066520 0x7fdb900689e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdb9401ab50 tx=0x7fdb94008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.031+0000 7f002f10e640 1 Processor -- start 2023-10-30T12:28:55.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.031+0000 7f002f10e640 1 -- start start 2023-10-30T12:28:55.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.035+0000 7fdba0ff9640 1 -- 172.21.15.125:0/3736708712 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdb9c052d50 con 0x7fdba4159830 2023-10-30T12:28:55.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.035+0000 7f002f10e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00280c7830 0x7f002806ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.035+0000 7f002f10e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002806d710 con 0x7f002806e430 2023-10-30T12:28:55.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.035+0000 7f002f10e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f002806b560 con 0x7f00280cce00 2023-10-30T12:28:55.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.035+0000 7f002f10e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002806dd30 con 0x7f00280c7830 2023-10-30T12:28:55.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.035+0000 7f002ceaa640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f00280cce00 0x7f0028159850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:36990/0 (socket says 172.21.15.125:36990) 2023-10-30T12:28:55.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.035+0000 7f002ceaa640 1 -- 172.21.15.125:0/3685704679 learned_addr learned my addr 172.21.15.125:0/3685704679 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:55.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.035+0000 7f001ffff640 1 --2- 172.21.15.125:0/3685704679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00280c7830 0x7f002806ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.035+0000 7f0027fff640 1 -- 172.21.15.125:0/3685704679 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1790298562 0 0) 0x7f002806b560 con 0x7f00280cce00 2023-10-30T12:28:55.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.035+0000 7f0027fff640 1 -- 172.21.15.125:0/3685704679 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0014003530 con 0x7f00280cce00 2023-10-30T12:28:55.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.035+0000 7f0027fff640 1 -- 172.21.15.125:0/3685704679 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 714398614 0 0) 0x7f002806d710 con 0x7f002806e430 2023-10-30T12:28:55.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.035+0000 7f001ffff640 1 -- 172.21.15.125:0/3685704679 >> v1:172.21.15.136:6789/0 conn(0x7f00280cce00 legacy=0x7f0028159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.035+0000 7f001ffff640 1 -- 172.21.15.125:0/3685704679 >> v1:172.21.15.125:6789/0 conn(0x7f002806e430 legacy=0x7f0028159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.035+0000 7f001ffff640 1 -- 172.21.15.125:0/3685704679 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00280c3c80 con 0x7f00280c7830 2023-10-30T12:28:55.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.039+0000 7f001ffff640 1 --2- 172.21.15.125:0/3685704679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00280c7830 0x7f002806ca50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f00100097f0 tx=0x7f001001c4c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0471c989833b3b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.039+0000 7f0027fff640 1 -- 172.21.15.125:0/3685704679 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00100ab600 con 0x7f00280c7830 2023-10-30T12:28:55.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.039+0000 7f0027fff640 1 -- 172.21.15.125:0/3685704679 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00100a7050 con 0x7f00280c7830 2023-10-30T12:28:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.039+0000 7f0027fff640 1 -- 172.21.15.125:0/3685704679 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00100ab980 con 0x7f00280c7830 2023-10-30T12:28:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.039+0000 7f002f10e640 1 -- 172.21.15.125:0/3685704679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00280c7830 msgr2=0x7f002806ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.039+0000 7f002f10e640 1 --2- 172.21.15.125:0/3685704679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00280c7830 0x7f002806ca50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f00100097f0 tx=0x7f001001c4c0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002f10e640 1 -- 172.21.15.125:0/3685704679 shutdown_connections 2023-10-30T12:28:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002f10e640 1 --2- 172.21.15.125:0/3685704679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00280c7830 0x7f002806ca50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002f10e640 1 -- 172.21.15.125:0/3685704679 >> 172.21.15.125:0/3685704679 conn(0x7f0028071ea0 msgr2=0x7f0028065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002f10e640 1 -- 172.21.15.125:0/3685704679 shutdown_connections 2023-10-30T12:28:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002f10e640 1 -- 172.21.15.125:0/3685704679 wait complete. 2023-10-30T12:28:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002f10e640 1 Processor -- start 2023-10-30T12:28:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002f10e640 1 -- start start 2023-10-30T12:28:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002f10e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f002806e430 0x7f0028153170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002f10e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00280c7830 0x7f00281528a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002f10e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00280cce00 0x7f0028151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002f10e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00280cb020 con 0x7f00280cce00 2023-10-30T12:28:55.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002f10e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00280c7180 con 0x7f00280c7830 2023-10-30T12:28:55.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002f10e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00281536b0 con 0x7f002806e430 2023-10-30T12:28:55.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002d6ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00280cce00 0x7f0028151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002d6ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00280cce00 0x7f0028151320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33272/0 (socket says 172.21.15.125:33272) 2023-10-30T12:28:55.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002d6ab640 1 -- 172.21.15.125:0/4252717433 learned_addr learned my addr 172.21.15.125:0/4252717433 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:55.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f002ceaa640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f002806e430 0x7f0028153170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.043+0000 7f001ffff640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00280c7830 0x7f00281528a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.047+0000 7f001ffff640 1 -- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f002806e430 msgr2=0x7f0028153170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.047+0000 7f001ffff640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f002806e430 0x7f0028153170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.047+0000 7f001ffff640 1 -- 172.21.15.125:0/4252717433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00280cce00 msgr2=0x7f0028151320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.047+0000 7f001ffff640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00280cce00 0x7f0028151320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.047+0000 7f001ffff640 1 -- 172.21.15.125:0/4252717433 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0010009420 con 0x7f00280c7830 2023-10-30T12:28:55.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.047+0000 7f002d6ab640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00280cce00 0x7f0028151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:55.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.047+0000 7f001ffff640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00280c7830 0x7f00281528a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f00280cdae0 tx=0x7f0010019260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.047+0000 7f0025ffb640 1 -- 172.21.15.125:0/4252717433 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010004030 con 0x7f00280c7830 2023-10-30T12:28:55.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.047+0000 7f0025ffb640 1 -- 172.21.15.125:0/4252717433 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0010004190 con 0x7f00280c7830 2023-10-30T12:28:55.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.047+0000 7f0025ffb640 1 -- 172.21.15.125:0/4252717433 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010003620 con 0x7f00280c7830 2023-10-30T12:28:55.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.047+0000 7f002f10e640 1 -- 172.21.15.125:0/4252717433 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00280c6bc0 con 0x7f00280c7830 2023-10-30T12:28:55.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.047+0000 7f002f10e640 1 -- 172.21.15.125:0/4252717433 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00281f6100 con 0x7f00280c7830 2023-10-30T12:28:55.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.047+0000 7f002f10e640 1 -- 172.21.15.125:0/4252717433 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00281f5f60 con 0x7f00280c7830 2023-10-30T12:28:55.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.051+0000 7f0025ffb640 1 -- 172.21.15.125:0/4252717433 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f00100b2040 con 0x7f00280c7830 2023-10-30T12:28:55.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.051+0000 7f0025ffb640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00080664a0 0x7f0008068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:55.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.051+0000 7f0025ffb640 1 -- 172.21.15.125:0/4252717433 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f001011d1a0 con 0x7f00280c7830 2023-10-30T12:28:55.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.051+0000 7f002ceaa640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00080664a0 0x7f0008068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:55.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.055+0000 7f002ceaa640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00080664a0 0x7f0008068960 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0018005c10 tx=0x7f0018009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:55.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.063+0000 7f0025ffb640 1 -- 172.21.15.125:0/4252717433 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f00100ea620 con 0x7f00280c7830 2023-10-30T12:28:55.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.187+0000 7f09f6409640 1 -- 172.21.15.125:0/2588529362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f09f0157ad0 con 0x7f09f0159830 2023-10-30T12:28:55.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.187+0000 7f09ed7fa640 1 -- 172.21.15.125:0/2588529362 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f09e8055cb0 con 0x7f09f0159830 2023-10-30T12:28:55.190 INFO:teuthology.orchestra.run.smithi125.stdout:60129542149 2023-10-30T12:28:55.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.191+0000 7f09f6409640 1 -- 172.21.15.125:0/2588529362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f09d00664f0 msgr2=0x7f09d00689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.191+0000 7f09f6409640 1 --2- 172.21.15.125:0/2588529362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f09d00664f0 0x7f09d00689b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f09e0000c10 tx=0x7f09e00078a0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.191+0000 7f09f6409640 1 -- 172.21.15.125:0/2588529362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09f0159830 msgr2=0x7f09f0155e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.191+0000 7f09f6409640 1 --2- 172.21.15.125:0/2588529362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09f0159830 0x7f09f0155e00 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f09e8009d30 tx=0x7f09e800c490 comp rx=0 tx=0).stop 2023-10-30T12:28:55.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.191+0000 7f09f6409640 1 -- 172.21.15.125:0/2588529362 shutdown_connections 2023-10-30T12:28:55.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.191+0000 7f09f6409640 1 --2- 172.21.15.125:0/2588529362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f09d00664f0 0x7f09d00689b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.191+0000 7f09f6409640 1 --2- 172.21.15.125:0/2588529362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f09f0159830 0x7f09f0155e00 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.191+0000 7f09f6409640 1 --2- 172.21.15.125:0/2588529362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f09f0153c60 0x7f09f0145ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.191+0000 7f09f6409640 1 --2- 172.21.15.125:0/2588529362 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09f00c4430 0x7f09f00ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.191+0000 7f09f6409640 1 -- 172.21.15.125:0/2588529362 >> 172.21.15.125:0/2588529362 conn(0x7f09f006bfe0 msgr2=0x7f09f006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:55.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.191+0000 7f09f6409640 1 -- 172.21.15.125:0/2588529362 shutdown_connections 2023-10-30T12:28:55.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.191+0000 7f09f6409640 1 -- 172.21.15.125:0/2588529362 wait complete. 2023-10-30T12:28:55.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.195+0000 7fd5ca817640 1 -- 172.21.15.125:0/2449332756 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd5c41482e0 con 0x7fd5c40c4430 2023-10-30T12:28:55.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.195+0000 7fd5c0ff9640 1 -- 172.21.15.125:0/2449332756 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd5c41482e0 con 0x7fd5c40c4430 2023-10-30T12:28:55.198 INFO:teuthology.orchestra.run.smithi125.stdout:68719476743 2023-10-30T12:28:55.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.195+0000 7fd5ca817640 1 -- 172.21.15.125:0/2449332756 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd5b00664a0 msgr2=0x7fd5b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.195+0000 7fd5ca817640 1 --2- 172.21.15.125:0/2449332756 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd5b00664a0 0x7fd5b0068960 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd5b80ad000 tx=0x7fd5b8004740 comp rx=0 tx=0).stop 2023-10-30T12:28:55.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.195+0000 7fd5ca817640 1 -- 172.21.15.125:0/2449332756 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5c40c4430 msgr2=0x7fd5c4154df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.195+0000 7fd5ca817640 1 --2- 172.21.15.125:0/2449332756 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5c40c4430 0x7fd5c4154df0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd5b400e930 tx=0x7fd5b400edf0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.195+0000 7fd5ca817640 1 -- 172.21.15.125:0/2449332756 shutdown_connections 2023-10-30T12:28:55.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.195+0000 7fd5ca817640 1 --2- 172.21.15.125:0/2449332756 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd5b00664a0 0x7fd5b0068960 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.195+0000 7fd5ca817640 1 --2- 172.21.15.125:0/2449332756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd5c4159830 0x7fd5c4155ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.195+0000 7fd5ca817640 1 --2- 172.21.15.125:0/2449332756 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd5c4153c60 0x7fd5c41489a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.195+0000 7fd5ca817640 1 --2- 172.21.15.125:0/2449332756 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd5c40c4430 0x7fd5c4154df0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.195+0000 7fd5ca817640 1 -- 172.21.15.125:0/2449332756 >> 172.21.15.125:0/2449332756 conn(0x7fd5c406bfe0 msgr2=0x7fd5c4158e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:55.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.199+0000 7fd5ca817640 1 -- 172.21.15.125:0/2449332756 shutdown_connections 2023-10-30T12:28:55.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.199+0000 7fd5ca817640 1 -- 172.21.15.125:0/2449332756 wait complete. 2023-10-30T12:28:55.221 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476744 got 68719476743 for osd.2 2023-10-30T12:28:55.226 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.5 2023-10-30T12:28:55.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.247+0000 7f7c677fe640 1 -- 172.21.15.125:0/3428995700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f7c8006c5e0 con 0x7f7c80159830 2023-10-30T12:28:55.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.247+0000 7f7c7d7fa640 1 -- 172.21.15.125:0/3428995700 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7c780b2090 con 0x7f7c80159830 2023-10-30T12:28:55.251 INFO:teuthology.orchestra.run.smithi125.stdout:64424509447 2023-10-30T12:28:55.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.251+0000 7f7c677fe640 1 -- 172.21.15.125:0/3428995700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7c6c0665a0 msgr2=0x7f7c6c068a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.251+0000 7f7c677fe640 1 --2- 172.21.15.125:0/3428995700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7c6c0665a0 0x7f7c6c068a60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7c74009df0 tx=0x7f7c74009190 comp rx=0 tx=0).stop 2023-10-30T12:28:55.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.251+0000 7f7c677fe640 1 -- 172.21.15.125:0/3428995700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c80159830 msgr2=0x7f7c80152260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.251+0000 7f7c677fe640 1 --2- 172.21.15.125:0/3428995700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c80159830 0x7f7c80152260 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7c800ccd50 tx=0x7f7c780196b0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.251+0000 7f7c677fe640 1 -- 172.21.15.125:0/3428995700 shutdown_connections 2023-10-30T12:28:55.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.251+0000 7f7c677fe640 1 --2- 172.21.15.125:0/3428995700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7c6c0665a0 0x7f7c6c068a60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.251+0000 7f7c677fe640 1 --2- 172.21.15.125:0/3428995700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c80159830 0x7f7c80152260 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.251+0000 7f7c677fe640 1 --2- 172.21.15.125:0/3428995700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c80153c60 0x7f7c8014a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.251+0000 7f7c677fe640 1 --2- 172.21.15.125:0/3428995700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c800c4430 0x7f7c800c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.251+0000 7f7c677fe640 1 -- 172.21.15.125:0/3428995700 >> 172.21.15.125:0/3428995700 conn(0x7f7c8006bfe0 msgr2=0x7f7c801512a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:55.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.251+0000 7f7c677fe640 1 -- 172.21.15.125:0/3428995700 shutdown_connections 2023-10-30T12:28:55.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.251+0000 7f7c677fe640 1 -- 172.21.15.125:0/3428995700 wait complete. 2023-10-30T12:28:55.273 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509447 for osd.6 2023-10-30T12:28:55.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.287+0000 7f002f10e640 1 -- 172.21.15.125:0/4252717433 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f00281f5f60 con 0x7f00280c7830 2023-10-30T12:28:55.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.287+0000 7f0025ffb640 1 -- 172.21.15.125:0/4252717433 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f00281f5f60 con 0x7f00280c7830 2023-10-30T12:28:55.292 INFO:teuthology.orchestra.run.smithi125.stdout:64424509448 2023-10-30T12:28:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.291+0000 7f002f10e640 1 -- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00080664a0 msgr2=0x7f0008068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.291+0000 7f002f10e640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00080664a0 0x7f0008068960 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0018005c10 tx=0x7f0018009040 comp rx=0 tx=0).stop 2023-10-30T12:28:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.291+0000 7f002f10e640 1 -- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00280c7830 msgr2=0x7f00281528a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.291+0000 7f002f10e640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00280c7830 0x7f00281528a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f00280cdae0 tx=0x7f0010019260 comp rx=0 tx=0).stop 2023-10-30T12:28:55.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.291+0000 7f002f10e640 1 -- 172.21.15.125:0/4252717433 shutdown_connections 2023-10-30T12:28:55.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.291+0000 7f002f10e640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00080664a0 0x7f0008068960 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.291+0000 7f002f10e640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00280cce00 0x7f0028151320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.291+0000 7f002f10e640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00280c7830 0x7f00281528a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.291+0000 7f002f10e640 1 --2- 172.21.15.125:0/4252717433 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f002806e430 0x7f0028153170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.291+0000 7f002f10e640 1 -- 172.21.15.125:0/4252717433 >> 172.21.15.125:0/4252717433 conn(0x7f0028071ea0 msgr2=0x7f00280c3440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:55.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.291+0000 7f002f10e640 1 -- 172.21.15.125:0/4252717433 shutdown_connections 2023-10-30T12:28:55.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.291+0000 7f002f10e640 1 -- 172.21.15.125:0/4252717433 wait complete. 2023-10-30T12:28:55.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.307+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3736708712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fdba4153130 con 0x7fdba4159830 2023-10-30T12:28:55.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.307+0000 7fdba0ff9640 1 -- 172.21.15.125:0/3736708712 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdb9c018020 con 0x7fdba4159830 2023-10-30T12:28:55.311 INFO:teuthology.orchestra.run.smithi125.stdout:60129542150 2023-10-30T12:28:55.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.311+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3736708712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb90066520 msgr2=0x7fdb900689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.311+0000 7fdbaa89e640 1 --2- 172.21.15.125:0/3736708712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb90066520 0x7fdb900689e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdb9401ab50 tx=0x7fdb94008040 comp rx=0 tx=0).stop 2023-10-30T12:28:55.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.311+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3736708712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdba4159830 msgr2=0x7fdba413b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:55.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.311+0000 7fdbaa89e640 1 --2- 172.21.15.125:0/3736708712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdba4159830 0x7fdba413b9c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdb9c00eba0 tx=0x7fdb9c00c490 comp rx=0 tx=0).stop 2023-10-30T12:28:55.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.311+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3736708712 shutdown_connections 2023-10-30T12:28:55.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.311+0000 7fdbaa89e640 1 --2- 172.21.15.125:0/3736708712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb90066520 0x7fdb900689e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.311+0000 7fdbaa89e640 1 --2- 172.21.15.125:0/3736708712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdba4159830 0x7fdba413b9c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.311+0000 7fdbaa89e640 1 --2- 172.21.15.125:0/3736708712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdba4153c60 0x7fdba41513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.311+0000 7fdbaa89e640 1 --2- 172.21.15.125:0/3736708712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdba40c4430 0x7fdba4070470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:55.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.311+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3736708712 >> 172.21.15.125:0/3736708712 conn(0x7fdba406bfe0 msgr2=0x7fdba40c4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:55.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.311+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3736708712 shutdown_connections 2023-10-30T12:28:55.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:55.311+0000 7fdbaa89e640 1 -- 172.21.15.125:0/3736708712 wait complete. 2023-10-30T12:28:55.318 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509448 for osd.4 2023-10-30T12:28:55.332 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.7 2023-10-30T12:28:56.223 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-30T12:28:56.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-30T12:28:56.274 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-30T12:28:56.319 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-30T12:28:56.333 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-30T12:28:56.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f1b5b846640 1 Processor -- start 2023-10-30T12:28:56.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f1b5b846640 1 -- start start 2023-10-30T12:28:56.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f4890f1d640 1 Processor -- start 2023-10-30T12:28:56.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f4890f1d640 1 -- start start 2023-10-30T12:28:56.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f1b5b846640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1b540c4430 0x7f1b540ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f1b5b846640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b540cb780 con 0x7f1b54159830 2023-10-30T12:28:56.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f1b5b846640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b5415a2e0 con 0x7f1b54153c60 2023-10-30T12:28:56.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f1b5b846640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b54154040 con 0x7f1b540c4430 2023-10-30T12:28:56.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f1b59de3640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1b54159830 0x7f1b540cac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54150/0 (socket says 172.21.15.125:54150) 2023-10-30T12:28:56.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f1b59de3640 1 -- 172.21.15.125:0/3469381717 learned_addr learned my addr 172.21.15.125:0/3469381717 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:56.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f1b58de1640 1 --2- 172.21.15.125:0/3469381717 >> v2:172.21.15.125:3300/0 conn(0x7f1b540c4430 0x7f1b540ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f1b4bfff640 1 -- 172.21.15.125:0/3469381717 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464791621 0 0) 0x7f1b540cb780 con 0x7f1b54159830 2023-10-30T12:28:56.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f1b4bfff640 1 -- 172.21.15.125:0/3469381717 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b40003530 con 0x7f1b54159830 2023-10-30T12:28:56.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f1b58de1640 1 -- 172.21.15.125:0/3469381717 >> v1:172.21.15.136:6789/0 conn(0x7f1b54153c60 legacy=0x7f1b540cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:56.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f1b58de1640 1 -- 172.21.15.125:0/3469381717 >> v1:172.21.15.125:6789/0 conn(0x7f1b54159830 legacy=0x7f1b540cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.379+0000 7f1b58de1640 1 -- 172.21.15.125:0/3469381717 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b5415a5c0 con 0x7f1b540c4430 2023-10-30T12:28:56.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f4890f1d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f488c0c4430 0x7f488c0ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f4890f1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488c159830 0x7f488c159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f4890f1d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488c0cb780 con 0x7f488c153c60 2023-10-30T12:28:56.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f4890f1d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488c154040 con 0x7f488c0c4430 2023-10-30T12:28:56.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f4890f1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488c15a2e0 con 0x7f488c159830 2023-10-30T12:28:56.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f488a59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f488c153c60 0x7f488c0cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:37004/0 (socket says 172.21.15.125:37004) 2023-10-30T12:28:56.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f488a59c640 1 -- 172.21.15.125:0/4050357596 learned_addr learned my addr 172.21.15.125:0/4050357596 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:56.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f4889d9b640 1 --2- 172.21.15.125:0/4050357596 >> v2:172.21.15.136:3300/0 conn(0x7f488c0c4430 0x7f488c0ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f488ad9d640 1 --2- 172.21.15.125:0/4050357596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488c159830 0x7f488c159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f488959a640 1 -- 172.21.15.125:0/4050357596 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616584683 0 0) 0x7f488c0cb780 con 0x7f488c153c60 2023-10-30T12:28:56.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f488959a640 1 -- 172.21.15.125:0/4050357596 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4874003530 con 0x7f488c153c60 2023-10-30T12:28:56.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f4889d9b640 1 -- 172.21.15.125:0/4050357596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488c159830 msgr2=0x7f488c159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:56.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f4889d9b640 1 --2- 172.21.15.125:0/4050357596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488c159830 0x7f488c159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f4889d9b640 1 -- 172.21.15.125:0/4050357596 >> v1:172.21.15.136:6789/0 conn(0x7f488c153c60 legacy=0x7f488c0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f4889d9b640 1 -- 172.21.15.125:0/4050357596 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c15a5c0 con 0x7f488c0c4430 2023-10-30T12:28:56.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f1b58de1640 1 --2- 172.21.15.125:0/3469381717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b540c4430 0x7f1b540ca360 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1b440097c0 tx=0x7f1b4401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd361e35e31d2ce2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f1b4bfff640 1 -- 172.21.15.125:0/3469381717 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b440ab600 con 0x7f1b540c4430 2023-10-30T12:28:56.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f1b4bfff640 1 -- 172.21.15.125:0/3469381717 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b440a7050 con 0x7f1b540c4430 2023-10-30T12:28:56.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.383+0000 7f1b4bfff640 1 -- 172.21.15.125:0/3469381717 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b440ab980 con 0x7f1b540c4430 2023-10-30T12:28:56.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 -- 172.21.15.125:0/3469381717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b540c4430 msgr2=0x7f1b540ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 --2- 172.21.15.125:0/3469381717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b540c4430 0x7f1b540ca360 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1b440097c0 tx=0x7f1b4401c440 comp rx=0 tx=0).stop 2023-10-30T12:28:56.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 -- 172.21.15.125:0/3469381717 shutdown_connections 2023-10-30T12:28:56.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 --2- 172.21.15.125:0/3469381717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b540c4430 0x7f1b540ca360 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 -- 172.21.15.125:0/3469381717 >> 172.21.15.125:0/3469381717 conn(0x7f1b5406bfe0 msgr2=0x7f1b5406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 -- 172.21.15.125:0/3469381717 shutdown_connections 2023-10-30T12:28:56.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 -- 172.21.15.125:0/3469381717 wait complete. 2023-10-30T12:28:56.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f4889d9b640 1 --2- 172.21.15.125:0/4050357596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f488c0c4430 0x7f488c0ca360 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f48800097c0 tx=0x7f488001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bde4f89d0f90751 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f488959a640 1 -- 172.21.15.125:0/4050357596 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48800ab600 con 0x7f488c0c4430 2023-10-30T12:28:56.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f488959a640 1 -- 172.21.15.125:0/4050357596 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48800a7050 con 0x7f488c0c4430 2023-10-30T12:28:56.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 Processor -- start 2023-10-30T12:28:56.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f488959a640 1 -- 172.21.15.125:0/4050357596 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48800ab960 con 0x7f488c0c4430 2023-10-30T12:28:56.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f4890f1d640 1 -- 172.21.15.125:0/4050357596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f488c0c4430 msgr2=0x7f488c0ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f4890f1d640 1 --2- 172.21.15.125:0/4050357596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f488c0c4430 0x7f488c0ca360 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f48800097c0 tx=0x7f488001c480 comp rx=0 tx=0).stop 2023-10-30T12:28:56.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f4890f1d640 1 -- 172.21.15.125:0/4050357596 shutdown_connections 2023-10-30T12:28:56.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f4890f1d640 1 --2- 172.21.15.125:0/4050357596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488c159830 0x7f488c159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f4890f1d640 1 --2- 172.21.15.125:0/4050357596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f488c0c4430 0x7f488c0ca360 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f4890f1d640 1 -- 172.21.15.125:0/4050357596 >> 172.21.15.125:0/4050357596 conn(0x7f488c06bfe0 msgr2=0x7f488c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 -- start start 2023-10-30T12:28:56.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b540c4430 0x7f1b541f5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b54153c60 0x7f1b540cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b54159830 0x7f1b540cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5414b0c0 con 0x7f1b540c4430 2023-10-30T12:28:56.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b541f5c30 con 0x7f1b54153c60 2023-10-30T12:28:56.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.387+0000 7f1b5b846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b541f5ed0 con 0x7f1b54159830 2023-10-30T12:28:56.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f4890f1d640 1 -- 172.21.15.125:0/4050357596 shutdown_connections 2023-10-30T12:28:56.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b595e2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b540c4430 0x7f1b541f5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f4890f1d640 1 -- 172.21.15.125:0/4050357596 wait complete. 2023-10-30T12:28:56.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b58de1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b54153c60 0x7f1b540cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b595e2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b540c4430 0x7f1b541f5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33288/0 (socket says 172.21.15.125:33288) 2023-10-30T12:28:56.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b595e2640 1 -- 172.21.15.125:0/3821842393 learned_addr learned my addr 172.21.15.125:0/3821842393 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:56.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b59de3640 1 --2- 172.21.15.125:0/3821842393 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b54159830 0x7f1b540cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b595e2640 1 -- 172.21.15.125:0/3821842393 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b54159830 msgr2=0x7f1b540cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b595e2640 1 --2- 172.21.15.125:0/3821842393 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b54159830 0x7f1b540cdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b595e2640 1 -- 172.21.15.125:0/3821842393 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b54153c60 msgr2=0x7f1b540cd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b595e2640 1 --2- 172.21.15.125:0/3821842393 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b54153c60 0x7f1b540cd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b595e2640 1 -- 172.21.15.125:0/3821842393 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b44009420 con 0x7f1b540c4430 2023-10-30T12:28:56.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b595e2640 1 --2- 172.21.15.125:0/3821842393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b540c4430 0x7f1b541f5360 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000c450 tx=0x7f1b5000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b49ffb640 1 -- 172.21.15.125:0/3821842393 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b50012830 con 0x7f1b540c4430 2023-10-30T12:28:56.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b49ffb640 1 -- 172.21.15.125:0/3821842393 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b500129b0 con 0x7f1b540c4430 2023-10-30T12:28:56.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b5b846640 1 -- 172.21.15.125:0/3821842393 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b54148aa0 con 0x7f1b540c4430 2023-10-30T12:28:56.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b49ffb640 1 -- 172.21.15.125:0/3821842393 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b50012d60 con 0x7f1b540c4430 2023-10-30T12:28:56.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f1b5b846640 1 -- 172.21.15.125:0/3821842393 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b540cd050 con 0x7f1b540c4430 2023-10-30T12:28:56.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f4890f1d640 1 Processor -- start 2023-10-30T12:28:56.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f4890f1d640 1 -- start start 2023-10-30T12:28:56.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f1b3f7fe640 1 -- 172.21.15.125:0/3821842393 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5404e170 con 0x7f1b540c4430 2023-10-30T12:28:56.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f1b49ffb640 1 -- 172.21.15.125:0/3821842393 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1b5001c460 con 0x7f1b540c4430 2023-10-30T12:28:56.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f4890f1d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f488c0c4430 0x7f488c1f17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f4890f1d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f488c153c60 0x7f488c1f5790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f4890f1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488c159830 0x7f488c146b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f4890f1d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c1f4d40 con 0x7f488c153c60 2023-10-30T12:28:56.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f4890f1d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c1f5300 con 0x7f488c0c4430 2023-10-30T12:28:56.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f4890f1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488c0c6570 con 0x7f488c159830 2023-10-30T12:28:56.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.391+0000 7f4889d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f488c153c60 0x7f488c1f5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f488ad9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488c159830 0x7f488c146b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f4889d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f488c153c60 0x7f488c1f5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33292/0 (socket says 172.21.15.125:33292) 2023-10-30T12:28:56.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f4889d9b640 1 -- 172.21.15.125:0/4287666699 learned_addr learned my addr 172.21.15.125:0/4287666699 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:56.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f4889d9b640 1 -- 172.21.15.125:0/4287666699 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488c159830 msgr2=0x7f488c146b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f4889d9b640 1 --2- 172.21.15.125:0/4287666699 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488c159830 0x7f488c146b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f4889d9b640 1 -- 172.21.15.125:0/4287666699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f488c0c4430 msgr2=0x7f488c1f17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f4889d9b640 1 --2- 172.21.15.125:0/4287666699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f488c0c4430 0x7f488c1f17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f4889d9b640 1 -- 172.21.15.125:0/4287666699 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4880009420 con 0x7f488c153c60 2023-10-30T12:28:56.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f4889d9b640 1 --2- 172.21.15.125:0/4287666699 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f488c153c60 0x7f488c1f5790 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f488001a0b0 tx=0x7f4880019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f487affd640 1 -- 172.21.15.125:0/4287666699 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48800b4400 con 0x7f488c153c60 2023-10-30T12:28:56.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f487affd640 1 -- 172.21.15.125:0/4287666699 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48800b4560 con 0x7f488c153c60 2023-10-30T12:28:56.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f487affd640 1 -- 172.21.15.125:0/4287666699 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48800b48f0 con 0x7f488c153c60 2023-10-30T12:28:56.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f1b49ffb640 1 --2- 172.21.15.125:0/3821842393 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1b380664a0 0x7f1b38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f1b49ffb640 1 -- 172.21.15.125:0/3821842393 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b50083cf0 con 0x7f1b540c4430 2023-10-30T12:28:56.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f4890f1d640 1 -- 172.21.15.125:0/4287666699 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488c0c6810 con 0x7f488c153c60 2023-10-30T12:28:56.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f4890f1d640 1 -- 172.21.15.125:0/4287666699 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488c1f4f90 con 0x7f488c153c60 2023-10-30T12:28:56.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.395+0000 7f1b58de1640 1 --2- 172.21.15.125:0/3821842393 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1b380664a0 0x7f1b38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.399+0000 7f4890f1d640 1 -- 172.21.15.125:0/4287666699 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488c1f5120 con 0x7f488c153c60 2023-10-30T12:28:56.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.399+0000 7f1b58de1640 1 --2- 172.21.15.125:0/3821842393 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1b380664a0 0x7f1b38068960 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1b440ad000 tx=0x7f1b440a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.399+0000 7f1b49ffb640 1 -- 172.21.15.125:0/3821842393 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1b500510f0 con 0x7f1b540c4430 2023-10-30T12:28:56.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.403+0000 7f487affd640 1 -- 172.21.15.125:0/4287666699 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f48800b2040 con 0x7f488c153c60 2023-10-30T12:28:56.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.403+0000 7f487affd640 1 --2- 172.21.15.125:0/4287666699 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f486c066520 0x7f486c0689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.403+0000 7f487affd640 1 -- 172.21.15.125:0/4287666699 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f488011e740 con 0x7f488c153c60 2023-10-30T12:28:56.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.403+0000 7f488a59c640 1 --2- 172.21.15.125:0/4287666699 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f486c066520 0x7f486c0689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.407+0000 7f488a59c640 1 --2- 172.21.15.125:0/4287666699 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f486c066520 0x7f486c0689e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f487c007fd0 tx=0x7f487c007940 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.407+0000 7f487affd640 1 -- 172.21.15.125:0/4287666699 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f48800ebb10 con 0x7f488c153c60 2023-10-30T12:28:56.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.435+0000 7fc2cac68640 1 Processor -- start 2023-10-30T12:28:56.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.435+0000 7fc2cac68640 1 -- start start 2023-10-30T12:28:56.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.435+0000 7fc2cac68640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc2c4153c60 0x7fc2c40ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.435+0000 7fc2cac68640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c40c4430 0x7fc2c40cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.435+0000 7fc2cac68640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc2c4159830 0x7fc2c4159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.435+0000 7fc2cac68640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c4154040 con 0x7fc2c4153c60 2023-10-30T12:28:56.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.435+0000 7fc2cac68640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2c40cb780 con 0x7fc2c4159830 2023-10-30T12:28:56.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.435+0000 7fc2cac68640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c40cc9f0 con 0x7fc2c40c4430 2023-10-30T12:28:56.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.435+0000 7fc2c8a04640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc2c4153c60 0x7fc2c40ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.435+0000 7fc2c8a04640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc2c4153c60 0x7fc2c40ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33296/0 (socket says 172.21.15.125:33296) 2023-10-30T12:28:56.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.435+0000 7fc2c3fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c40c4430 0x7fc2c40cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.435+0000 7fc2c8a04640 1 -- 172.21.15.125:0/2119515731 learned_addr learned my addr 172.21.15.125:0/2119515731 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:56.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.439+0000 7fc2c9205640 1 --2- 172.21.15.125:0/2119515731 >> v2:172.21.15.136:3300/0 conn(0x7fc2c4159830 0x7fc2c4159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.439+0000 7fc2c3fff640 1 -- 172.21.15.125:0/2119515731 >> v2:172.21.15.136:3300/0 conn(0x7fc2c4159830 msgr2=0x7fc2c4159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:56.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.439+0000 7fc2c3fff640 1 --2- 172.21.15.125:0/2119515731 >> v2:172.21.15.136:3300/0 conn(0x7fc2c4159830 0x7fc2c4159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.439+0000 7fc2c3fff640 1 -- 172.21.15.125:0/2119515731 >> v2:172.21.15.125:3300/0 conn(0x7fc2c4153c60 msgr2=0x7fc2c40ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:56.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.439+0000 7fc2c3fff640 1 --2- 172.21.15.125:0/2119515731 >> v2:172.21.15.125:3300/0 conn(0x7fc2c4153c60 0x7fc2c40ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.439+0000 7fc2c3fff640 1 -- 172.21.15.125:0/2119515731 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2c415a2e0 con 0x7fc2c40c4430 2023-10-30T12:28:56.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.439+0000 7fc2c9205640 1 --2- 172.21.15.125:0/2119515731 >> v2:172.21.15.136:3300/0 conn(0x7fc2c4159830 0x7fc2c4159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:28:56.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.439+0000 7fc2c3fff640 1 --2- 172.21.15.125:0/2119515731 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c40c4430 0x7fc2c40cc4b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc2c408d1a0 tx=0x7fc2b001c460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21bc8f46bdd323c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.443+0000 7fc2c37fe640 1 -- 172.21.15.125:0/2119515731 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b00ac640 con 0x7fc2c40c4430 2023-10-30T12:28:56.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.443+0000 7fc2c37fe640 1 -- 172.21.15.125:0/2119515731 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b00ac7a0 con 0x7fc2c40c4430 2023-10-30T12:28:56.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.443+0000 7fc2c37fe640 1 -- 172.21.15.125:0/2119515731 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b00acb30 con 0x7fc2c40c4430 2023-10-30T12:28:56.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.443+0000 7fc2cac68640 1 -- 172.21.15.125:0/2119515731 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c40c4430 msgr2=0x7fc2c40cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.443+0000 7fc2cac68640 1 --2- 172.21.15.125:0/2119515731 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c40c4430 0x7fc2c40cc4b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc2c408d1a0 tx=0x7fc2b001c460 comp rx=0 tx=0).stop 2023-10-30T12:28:56.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.443+0000 7fc2cac68640 1 -- 172.21.15.125:0/2119515731 shutdown_connections 2023-10-30T12:28:56.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.443+0000 7fc2cac68640 1 --2- 172.21.15.125:0/2119515731 >> v2:172.21.15.136:3300/0 conn(0x7fc2c4159830 0x7fc2c4159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.443+0000 7fc2cac68640 1 --2- 172.21.15.125:0/2119515731 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c40c4430 0x7fc2c40cc4b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.443+0000 7fc2cac68640 1 --2- 172.21.15.125:0/2119515731 >> v2:172.21.15.125:3300/0 conn(0x7fc2c4153c60 0x7fc2c40ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.443+0000 7fc2cac68640 1 -- 172.21.15.125:0/2119515731 >> 172.21.15.125:0/2119515731 conn(0x7fc2c406bfe0 msgr2=0x7fc2c406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.443+0000 7fc2cac68640 1 -- 172.21.15.125:0/2119515731 shutdown_connections 2023-10-30T12:28:56.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.443+0000 7fc2cac68640 1 -- 172.21.15.125:0/2119515731 wait complete. 2023-10-30T12:28:56.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.443+0000 7fc2cac68640 1 Processor -- start 2023-10-30T12:28:56.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.443+0000 7fc2cac68640 1 -- start start 2023-10-30T12:28:56.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2cac68640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2c40c4430 0x7fc2c40c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2cac68640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4153c60 0x7fc2c40b4310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2cac68640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c4159830 0x7fc2c4148b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2cac68640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c4156f00 con 0x7fc2c4153c60 2023-10-30T12:28:56.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2cac68640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c40c9410 con 0x7fc2c40c4430 2023-10-30T12:28:56.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2cac68640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c41630d0 con 0x7fc2c4159830 2023-10-30T12:28:56.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2c8a04640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2c40c4430 0x7fc2c40c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2c9205640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c4159830 0x7fc2c4148b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2c8a04640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2c40c4430 0x7fc2c40c8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:39062/0 (socket says 172.21.15.125:39062) 2023-10-30T12:28:56.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2c8a04640 1 -- 172.21.15.125:0/2713653021 learned_addr learned my addr 172.21.15.125:0/2713653021 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:56.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2c3fff640 1 --2- 172.21.15.125:0/2713653021 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4153c60 0x7fc2c40b4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2c8a04640 1 -- 172.21.15.125:0/2713653021 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c4159830 msgr2=0x7fc2c4148b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2c8a04640 1 --2- 172.21.15.125:0/2713653021 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c4159830 0x7fc2c4148b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2c8a04640 1 -- 172.21.15.125:0/2713653021 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4153c60 msgr2=0x7fc2c40b4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2c8a04640 1 --2- 172.21.15.125:0/2713653021 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4153c60 0x7fc2c40b4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2c8a04640 1 -- 172.21.15.125:0/2713653021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b0009420 con 0x7fc2c40c4430 2023-10-30T12:28:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2c8a04640 1 --2- 172.21.15.125:0/2713653021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2c40c4430 0x7fc2c40c8ed0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800e8e0 tx=0x7fc2b800eda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2c17fa640 1 -- 172.21.15.125:0/2713653021 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b800ca00 con 0x7fc2c40c4430 2023-10-30T12:28:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2c17fa640 1 -- 172.21.15.125:0/2713653021 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2b800cb60 con 0x7fc2c40c4430 2023-10-30T12:28:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2c17fa640 1 -- 172.21.15.125:0/2713653021 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2b800ced0 con 0x7fc2c40c4430 2023-10-30T12:28:56.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2cac68640 1 -- 172.21.15.125:0/2713653021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c41566c0 con 0x7fc2c40c4430 2023-10-30T12:28:56.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2cac68640 1 -- 172.21.15.125:0/2713653021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c4162df0 con 0x7fc2c40c4430 2023-10-30T12:28:56.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.447+0000 7fc2cac68640 1 -- 172.21.15.125:0/2713653021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c404e170 con 0x7fc2c40c4430 2023-10-30T12:28:56.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.451+0000 7fc2c17fa640 1 -- 172.21.15.125:0/2713653021 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc2b8013470 con 0x7fc2c40c4430 2023-10-30T12:28:56.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.451+0000 7fc2c17fa640 1 --2- 172.21.15.125:0/2713653021 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2a80664a0 0x7fc2a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.451+0000 7fc2c17fa640 1 -- 172.21.15.125:0/2713653021 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc2b80847f0 con 0x7fc2c40c4430 2023-10-30T12:28:56.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.451+0000 7fc2c3fff640 1 --2- 172.21.15.125:0/2713653021 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2a80664a0 0x7fc2a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.455+0000 7fc2c3fff640 1 --2- 172.21.15.125:0/2713653021 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2a80664a0 0x7fc2a8068960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc2b00ae000 tx=0x7fc2b00a6070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.463+0000 7fc2c17fa640 1 -- 172.21.15.125:0/2713653021 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc2b8051e00 con 0x7fc2c40c4430 2023-10-30T12:28:56.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.519+0000 7fa0af93e640 1 Processor -- start 2023-10-30T12:28:56.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.519+0000 7fa0af93e640 1 -- start start 2023-10-30T12:28:56.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.519+0000 7fa0af93e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a8153c60 0x7fa0a80ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.519+0000 7fa0af93e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa0a80c4430 0x7fa0a80cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.519+0000 7fa0af93e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a80cb780 con 0x7fa0a8159830 2023-10-30T12:28:56.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.519+0000 7fa0af93e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a8154040 con 0x7fa0a80c4430 2023-10-30T12:28:56.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.519+0000 7fa0af93e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80cc9f0 con 0x7fa0a8153c60 2023-10-30T12:28:56.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.523+0000 7fa0adedb640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa0a8159830 0x7fa0a80cac40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54160/0 (socket says 172.21.15.125:54160) 2023-10-30T12:28:56.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.523+0000 7fa0adedb640 1 -- 172.21.15.125:0/3345171268 learned_addr learned my addr 172.21.15.125:0/3345171268 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:56.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.523+0000 7fa0ad6da640 1 --2- 172.21.15.125:0/3345171268 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a8153c60 0x7fa0a80ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.523+0000 7fa0aced9640 1 --2- 172.21.15.125:0/3345171268 >> v2:172.21.15.136:3300/0 conn(0x7fa0a80c4430 0x7fa0a80cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.523+0000 7fa0aced9640 1 -- 172.21.15.125:0/3345171268 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a8153c60 msgr2=0x7fa0a80ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:56.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.523+0000 7fa0aced9640 1 --2- 172.21.15.125:0/3345171268 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a8153c60 0x7fa0a80ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.523+0000 7fa0aced9640 1 -- 172.21.15.125:0/3345171268 >> v1:172.21.15.125:6789/0 conn(0x7fa0a8159830 legacy=0x7fa0a80cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.523+0000 7fa09ffff640 1 -- 172.21.15.125:0/3345171268 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2563565181 0 0) 0x7fa0a80cb780 con 0x7fa0a8159830 2023-10-30T12:28:56.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.523+0000 7fa0aced9640 1 -- 172.21.15.125:0/3345171268 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a815a2e0 con 0x7fa0a80c4430 2023-10-30T12:28:56.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.523+0000 7fa0ad6da640 1 --2- 172.21.15.125:0/3345171268 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a8153c60 0x7fa0a80ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:56.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.527+0000 7fa0aced9640 1 --2- 172.21.15.125:0/3345171268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a80c4430 0x7fa0a80cc4b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa0980097c0 tx=0x7fa09801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cbac39f901768e62 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.527+0000 7fa09ffff640 1 -- 172.21.15.125:0/3345171268 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0980ab600 con 0x7fa0a80c4430 2023-10-30T12:28:56.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.527+0000 7fa09ffff640 1 -- 172.21.15.125:0/3345171268 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0980a7050 con 0x7fa0a80c4430 2023-10-30T12:28:56.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.527+0000 7fa09ffff640 1 -- 172.21.15.125:0/3345171268 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0980ab980 con 0x7fa0a80c4430 2023-10-30T12:28:56.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.527+0000 7fa0af93e640 1 -- 172.21.15.125:0/3345171268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a80c4430 msgr2=0x7fa0a80cc4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.527+0000 7fa0af93e640 1 --2- 172.21.15.125:0/3345171268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a80c4430 0x7fa0a80cc4b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa0980097c0 tx=0x7fa09801c480 comp rx=0 tx=0).stop 2023-10-30T12:28:56.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.527+0000 7fa0af93e640 1 -- 172.21.15.125:0/3345171268 shutdown_connections 2023-10-30T12:28:56.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.527+0000 7fa0af93e640 1 --2- 172.21.15.125:0/3345171268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a80c4430 0x7fa0a80cc4b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.527+0000 7fa0af93e640 1 --2- 172.21.15.125:0/3345171268 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a8153c60 0x7fa0a80ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.527+0000 7fa0af93e640 1 -- 172.21.15.125:0/3345171268 >> 172.21.15.125:0/3345171268 conn(0x7fa0a806bfe0 msgr2=0x7fa0a806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.527+0000 7fa0af93e640 1 -- 172.21.15.125:0/3345171268 shutdown_connections 2023-10-30T12:28:56.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.527+0000 7fa0af93e640 1 -- 172.21.15.125:0/3345171268 wait complete. 2023-10-30T12:28:56.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0af93e640 1 Processor -- start 2023-10-30T12:28:56.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0af93e640 1 -- start start 2023-10-30T12:28:56.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0af93e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a80c4430 0x7fa0a80ce850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0af93e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a8153c60 0x7fa0a81f5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0af93e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a8159830 0x7fa0a80b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0af93e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a8143610 con 0x7fa0a8159830 2023-10-30T12:28:56.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0af93e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80b53f0 con 0x7fa0a80c4430 2023-10-30T12:28:56.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0af93e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80c7f60 con 0x7fa0a8153c60 2023-10-30T12:28:56.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0adedb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a8159830 0x7fa0a80b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0ad6da640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a80c4430 0x7fa0a80ce850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0adedb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a8159830 0x7fa0a80b4160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33328/0 (socket says 172.21.15.125:33328) 2023-10-30T12:28:56.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0adedb640 1 -- 172.21.15.125:0/2270724904 learned_addr learned my addr 172.21.15.125:0/2270724904 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:56.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0ad6da640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a80c4430 0x7fa0a80ce850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:39086/0 (socket says 172.21.15.125:39086) 2023-10-30T12:28:56.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0aced9640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a8153c60 0x7fa0a81f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0aced9640 1 -- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a80c4430 msgr2=0x7fa0a80ce850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0aced9640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a80c4430 0x7fa0a80ce850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0aced9640 1 -- 172.21.15.125:0/2270724904 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a8159830 msgr2=0x7fa0a80b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0aced9640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a8159830 0x7fa0a80b4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0aced9640 1 -- 172.21.15.125:0/2270724904 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a4009420 con 0x7fa0a8153c60 2023-10-30T12:28:56.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0adedb640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a8159830 0x7fa0a80b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:56.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.531+0000 7fa0ad6da640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a80c4430 0x7fa0a80ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:56.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.535+0000 7fa0aced9640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a8153c60 0x7fa0a81f5150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa0980ab000 tx=0x7fa098019220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.535+0000 7fa09dffb640 1 -- 172.21.15.125:0/2270724904 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098006e20 con 0x7fa0a8153c60 2023-10-30T12:28:56.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.535+0000 7fa09dffb640 1 -- 172.21.15.125:0/2270724904 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa098004030 con 0x7fa0a8153c60 2023-10-30T12:28:56.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.535+0000 7fa09dffb640 1 -- 172.21.15.125:0/2270724904 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0980043c0 con 0x7fa0a8153c60 2023-10-30T12:28:56.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.535+0000 7fa0af93e640 1 -- 172.21.15.125:0/2270724904 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa098009420 con 0x7fa0a8153c60 2023-10-30T12:28:56.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.535+0000 7fa0af93e640 1 -- 172.21.15.125:0/2270724904 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a8142b60 con 0x7fa0a8153c60 2023-10-30T12:28:56.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.535+0000 7fa0af93e640 1 -- 172.21.15.125:0/2270724904 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a80c6910 con 0x7fa0a8153c60 2023-10-30T12:28:56.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.535+0000 7fa09dffb640 1 -- 172.21.15.125:0/2270724904 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa0980b2040 con 0x7fa0a8153c60 2023-10-30T12:28:56.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.539+0000 7fa09dffb640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa08c0664d0 0x7fa08c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.539+0000 7fa09dffb640 1 -- 172.21.15.125:0/2270724904 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa09811cec0 con 0x7fa0a8153c60 2023-10-30T12:28:56.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.539+0000 7fa0ad6da640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa08c0664d0 0x7fa08c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.539+0000 7fa0ad6da640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa08c0664d0 0x7fa08c068990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009fd0 tx=0x7fa0a4009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.539+0000 7f2638a09640 1 Processor -- start 2023-10-30T12:28:56.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.539+0000 7f2638a09640 1 -- start start 2023-10-30T12:28:56.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f2638a09640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f26340c6b80 0x7f2634071920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f2638a09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f263413fd10 0x7f26341565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f2638a09640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2634070010 con 0x7f26340c6f60 2023-10-30T12:28:56.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f2638a09640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2634072260 con 0x7f26340c6b80 2023-10-30T12:28:56.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f2638a09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263406f6e0 con 0x7f263413fd10 2023-10-30T12:28:56.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f2631d9b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f26340c6f60 0x7f263413f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54166/0 (socket says 172.21.15.125:54166) 2023-10-30T12:28:56.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f2631d9b640 1 -- 172.21.15.125:0/1367213242 learned_addr learned my addr 172.21.15.125:0/1367213242 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:56.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f263259c640 1 --2- 172.21.15.125:0/1367213242 >> v2:172.21.15.136:3300/0 conn(0x7f26340c6b80 0x7f2634071920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f2632d9d640 1 --2- 172.21.15.125:0/1367213242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f263413fd10 0x7f26341565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f263159a640 1 -- 172.21.15.125:0/1367213242 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501516415 0 0) 0x7f2634070010 con 0x7f26340c6f60 2023-10-30T12:28:56.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f263159a640 1 -- 172.21.15.125:0/1367213242 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f261c005740 con 0x7f26340c6f60 2023-10-30T12:28:56.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f263259c640 1 -- 172.21.15.125:0/1367213242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f263413fd10 msgr2=0x7f26341565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:56.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f263259c640 1 --2- 172.21.15.125:0/1367213242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f263413fd10 0x7f26341565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f263259c640 1 -- 172.21.15.125:0/1367213242 >> v1:172.21.15.125:6789/0 conn(0x7f26340c6f60 legacy=0x7f263413f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.543+0000 7f263259c640 1 -- 172.21.15.125:0/1367213242 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f263406f9c0 con 0x7f26340c6b80 2023-10-30T12:28:56.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.547+0000 7f263259c640 1 --2- 172.21.15.125:0/1367213242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26340c6b80 0x7f2634071920 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f26240097c0 tx=0x7f262401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e35bffabcc75032 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.547+0000 7f263159a640 1 -- 172.21.15.125:0/1367213242 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26240ab600 con 0x7f26340c6b80 2023-10-30T12:28:56.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.547+0000 7f263159a640 1 -- 172.21.15.125:0/1367213242 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26240a7050 con 0x7f26340c6b80 2023-10-30T12:28:56.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.547+0000 7f263159a640 1 -- 172.21.15.125:0/1367213242 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26240ab960 con 0x7f26340c6b80 2023-10-30T12:28:56.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.547+0000 7f2638a09640 1 -- 172.21.15.125:0/1367213242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26340c6b80 msgr2=0x7f2634071920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.547+0000 7f2638a09640 1 --2- 172.21.15.125:0/1367213242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26340c6b80 0x7f2634071920 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f26240097c0 tx=0x7f262401c450 comp rx=0 tx=0).stop 2023-10-30T12:28:56.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.547+0000 7f2638a09640 1 -- 172.21.15.125:0/1367213242 shutdown_connections 2023-10-30T12:28:56.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.547+0000 7f2638a09640 1 --2- 172.21.15.125:0/1367213242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f263413fd10 0x7f26341565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.547+0000 7f2638a09640 1 --2- 172.21.15.125:0/1367213242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26340c6b80 0x7f2634071920 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.547+0000 7f2638a09640 1 -- 172.21.15.125:0/1367213242 >> 172.21.15.125:0/1367213242 conn(0x7f26340c7910 msgr2=0x7f26340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.547+0000 7f2638a09640 1 -- 172.21.15.125:0/1367213242 shutdown_connections 2023-10-30T12:28:56.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.547+0000 7f2638a09640 1 -- 172.21.15.125:0/1367213242 wait complete. 2023-10-30T12:28:56.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7fa09dffb640 1 -- 172.21.15.125:0/2270724904 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa0980ea3d0 con 0x7fa0a8153c60 2023-10-30T12:28:56.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f2638a09640 1 Processor -- start 2023-10-30T12:28:56.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f2638a09640 1 -- start start 2023-10-30T12:28:56.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f2638a09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26340c6b80 0x7f26341f6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f2638a09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26340c6f60 0x7f26341585f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f2638a09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26341596b0 0x7f26341557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f2638a09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263413df20 con 0x7f26341596b0 2023-10-30T12:28:56.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f2638a09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26341f6980 con 0x7f26340c6b80 2023-10-30T12:28:56.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f2638a09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26341f6c60 con 0x7f26340c6f60 2023-10-30T12:28:56.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f263259c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26340c6b80 0x7f26341f6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f263259c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26340c6b80 0x7f26341f6380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:39116/0 (socket says 172.21.15.125:39116) 2023-10-30T12:28:56.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f263259c640 1 -- 172.21.15.125:0/1778419471 learned_addr learned my addr 172.21.15.125:0/1778419471 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:56.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f2632d9d640 1 --2- 172.21.15.125:0/1778419471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26341596b0 0x7f26341557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f263259c640 1 -- 172.21.15.125:0/1778419471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26340c6f60 msgr2=0x7f26341585f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-30T12:28:56.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f263259c640 1 --2- 172.21.15.125:0/1778419471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26340c6f60 0x7f26341585f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f263259c640 1 -- 172.21.15.125:0/1778419471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26341596b0 msgr2=0x7f26341557c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f263259c640 1 --2- 172.21.15.125:0/1778419471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26341596b0 0x7f26341557c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f263259c640 1 -- 172.21.15.125:0/1778419471 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2624009420 con 0x7f26340c6b80 2023-10-30T12:28:56.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f263259c640 1 --2- 172.21.15.125:0/1778419471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26340c6b80 0x7f26341f6380 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2624000c10 tx=0x7f26240191f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f2622ffd640 1 -- 172.21.15.125:0/1778419471 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26240b9070 con 0x7f26340c6b80 2023-10-30T12:28:56.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f2622ffd640 1 -- 172.21.15.125:0/1778419471 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26240038b0 con 0x7f26340c6b80 2023-10-30T12:28:56.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f2622ffd640 1 -- 172.21.15.125:0/1778419471 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2624003c20 con 0x7f26340c6b80 2023-10-30T12:28:56.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f2638a09640 1 -- 172.21.15.125:0/1778419471 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26341f6f40 con 0x7f26340c6b80 2023-10-30T12:28:56.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.551+0000 7f2638a09640 1 -- 172.21.15.125:0/1778419471 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26341552a0 con 0x7f26340c6b80 2023-10-30T12:28:56.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.555+0000 7f2638a09640 1 -- 172.21.15.125:0/1778419471 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26341554c0 con 0x7f26340c6b80 2023-10-30T12:28:56.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.555+0000 7f2622ffd640 1 -- 172.21.15.125:0/1778419471 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f26240b2040 con 0x7f26340c6b80 2023-10-30T12:28:56.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.555+0000 7f2622ffd640 1 --2- 172.21.15.125:0/1778419471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26140664d0 0x7f2614068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.555+0000 7f2622ffd640 1 -- 172.21.15.125:0/1778419471 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f262411d500 con 0x7f26340c6b80 2023-10-30T12:28:56.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.559+0000 7f2631d9b640 1 --2- 172.21.15.125:0/1778419471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26140664d0 0x7f2614068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.559+0000 7f2631d9b640 1 --2- 172.21.15.125:0/1778419471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26140664d0 0x7f2614068990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2628000f70 tx=0x7f26280010b0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.571+0000 7f2622ffd640 1 -- 172.21.15.125:0/1778419471 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f26240ea960 con 0x7f26340c6b80 2023-10-30T12:28:56.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.607+0000 7f4890f1d640 1 -- 172.21.15.125:0/4287666699 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f488c04e170 con 0x7f488c153c60 2023-10-30T12:28:56.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.607+0000 7f487affd640 1 -- 172.21.15.125:0/4287666699 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f48800b0070 con 0x7f488c153c60 2023-10-30T12:28:56.610 INFO:teuthology.orchestra.run.smithi125.stdout:60129542150 2023-10-30T12:28:56.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.611+0000 7f4890f1d640 1 -- 172.21.15.125:0/4287666699 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f486c066520 msgr2=0x7f486c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.611+0000 7f4890f1d640 1 --2- 172.21.15.125:0/4287666699 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f486c066520 0x7f486c0689e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f487c007fd0 tx=0x7f487c007940 comp rx=0 tx=0).stop 2023-10-30T12:28:56.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.611+0000 7f4890f1d640 1 -- 172.21.15.125:0/4287666699 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f488c153c60 msgr2=0x7f488c1f5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.611+0000 7f4890f1d640 1 --2- 172.21.15.125:0/4287666699 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f488c153c60 0x7f488c1f5790 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f488001a0b0 tx=0x7f4880019220 comp rx=0 tx=0).stop 2023-10-30T12:28:56.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.611+0000 7f4890f1d640 1 -- 172.21.15.125:0/4287666699 shutdown_connections 2023-10-30T12:28:56.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.611+0000 7f4890f1d640 1 --2- 172.21.15.125:0/4287666699 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f486c066520 0x7f486c0689e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.611+0000 7f4890f1d640 1 --2- 172.21.15.125:0/4287666699 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488c159830 0x7f488c146b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.611+0000 7f4890f1d640 1 --2- 172.21.15.125:0/4287666699 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f488c153c60 0x7f488c1f5790 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f488001a0b0 tx=0x7f4880019220 comp rx=0 tx=0).stop 2023-10-30T12:28:56.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.611+0000 7f4890f1d640 1 --2- 172.21.15.125:0/4287666699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f488c0c4430 0x7f488c1f17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.611+0000 7f4890f1d640 1 -- 172.21.15.125:0/4287666699 >> 172.21.15.125:0/4287666699 conn(0x7f488c06bfe0 msgr2=0x7f488c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.611+0000 7f4890f1d640 1 -- 172.21.15.125:0/4287666699 shutdown_connections 2023-10-30T12:28:56.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.611+0000 7f4890f1d640 1 -- 172.21.15.125:0/4287666699 wait complete. 2023-10-30T12:28:56.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.635+0000 7fc2cac68640 1 -- 172.21.15.125:0/2713653021 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fc2c40caa20 con 0x7fc2c40c4430 2023-10-30T12:28:56.640 INFO:teuthology.orchestra.run.smithi125.stdout:64424509448 2023-10-30T12:28:56.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.635+0000 7fc2c17fa640 1 -- 172.21.15.125:0/2713653021 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc2b8055420 con 0x7fc2c40c4430 2023-10-30T12:28:56.642 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542150 for osd.5 2023-10-30T12:28:56.642 DEBUG:teuthology.parallel:result is None 2023-10-30T12:28:56.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.639+0000 7fc2cac68640 1 -- 172.21.15.125:0/2713653021 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2a80664a0 msgr2=0x7fc2a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.639+0000 7fc2cac68640 1 --2- 172.21.15.125:0/2713653021 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2a80664a0 0x7fc2a8068960 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fc2b00ae000 tx=0x7fc2b00a6070 comp rx=0 tx=0).stop 2023-10-30T12:28:56.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.639+0000 7fc2cac68640 1 -- 172.21.15.125:0/2713653021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2c40c4430 msgr2=0x7fc2c40c8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.639+0000 7fc2cac68640 1 --2- 172.21.15.125:0/2713653021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2c40c4430 0x7fc2c40c8ed0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc2b800e8e0 tx=0x7fc2b800eda0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.639+0000 7fc2cac68640 1 -- 172.21.15.125:0/2713653021 shutdown_connections 2023-10-30T12:28:56.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.639+0000 7fc2cac68640 1 --2- 172.21.15.125:0/2713653021 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2a80664a0 0x7fc2a8068960 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.639+0000 7fc2cac68640 1 --2- 172.21.15.125:0/2713653021 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2c4159830 0x7fc2c4148b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.639+0000 7fc2cac68640 1 --2- 172.21.15.125:0/2713653021 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2c4153c60 0x7fc2c40b4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.639+0000 7fc2cac68640 1 --2- 172.21.15.125:0/2713653021 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2c40c4430 0x7fc2c40c8ed0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.639+0000 7fc2cac68640 1 -- 172.21.15.125:0/2713653021 >> 172.21.15.125:0/2713653021 conn(0x7fc2c406bfe0 msgr2=0x7fc2c41520f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.639+0000 7fc2cac68640 1 -- 172.21.15.125:0/2713653021 shutdown_connections 2023-10-30T12:28:56.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.639+0000 7fc2cac68640 1 -- 172.21.15.125:0/2713653021 wait complete. 2023-10-30T12:28:56.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.659+0000 7f1b3f7fe640 1 -- 172.21.15.125:0/3821842393 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1b540caa20 con 0x7f1b540c4430 2023-10-30T12:28:56.662 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509448 got 64424509448 for osd.6 2023-10-30T12:28:56.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.659+0000 7f1b49ffb640 1 -- 172.21.15.125:0/3821842393 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1b50054710 con 0x7f1b540c4430 2023-10-30T12:28:56.663 INFO:teuthology.orchestra.run.smithi125.stdout:68719476745 2023-10-30T12:28:56.663 DEBUG:teuthology.parallel:result is None 2023-10-30T12:28:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.663+0000 7f1b3f7fe640 1 -- 172.21.15.125:0/3821842393 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1b380664a0 msgr2=0x7f1b38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.663+0000 7f1b3f7fe640 1 --2- 172.21.15.125:0/3821842393 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1b380664a0 0x7f1b38068960 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1b440ad000 tx=0x7f1b440a6000 comp rx=0 tx=0).stop 2023-10-30T12:28:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.663+0000 7f1b3f7fe640 1 -- 172.21.15.125:0/3821842393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b540c4430 msgr2=0x7f1b541f5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.663+0000 7f1b3f7fe640 1 --2- 172.21.15.125:0/3821842393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b540c4430 0x7f1b541f5360 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1b5000c450 tx=0x7f1b5000c910 comp rx=0 tx=0).stop 2023-10-30T12:28:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.663+0000 7f1b3f7fe640 1 -- 172.21.15.125:0/3821842393 shutdown_connections 2023-10-30T12:28:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.663+0000 7f1b3f7fe640 1 --2- 172.21.15.125:0/3821842393 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1b380664a0 0x7f1b38068960 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.663+0000 7f1b3f7fe640 1 --2- 172.21.15.125:0/3821842393 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b54159830 0x7f1b540cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.663+0000 7f1b3f7fe640 1 --2- 172.21.15.125:0/3821842393 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b54153c60 0x7f1b540cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.663+0000 7f1b3f7fe640 1 --2- 172.21.15.125:0/3821842393 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b540c4430 0x7f1b541f5360 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.663+0000 7f1b3f7fe640 1 -- 172.21.15.125:0/3821842393 >> 172.21.15.125:0/3821842393 conn(0x7f1b5406bfe0 msgr2=0x7f1b5406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.663+0000 7f1b3f7fe640 1 -- 172.21.15.125:0/3821842393 shutdown_connections 2023-10-30T12:28:56.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.663+0000 7f1b3f7fe640 1 -- 172.21.15.125:0/3821842393 wait complete. 2023-10-30T12:28:56.685 INFO:tasks.ceph.ceph_manager.ceph:need seq 68719476744 got 68719476745 for osd.2 2023-10-30T12:28:56.685 DEBUG:teuthology.parallel:result is None 2023-10-30T12:28:56.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.751+0000 7fa0af93e640 1 -- 172.21.15.125:0/2270724904 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa0a80c6910 con 0x7fa0a8153c60 2023-10-30T12:28:56.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.751+0000 7fa09dffb640 1 -- 172.21.15.125:0/2270724904 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa0980bc020 con 0x7fa0a8153c60 2023-10-30T12:28:56.753 INFO:teuthology.orchestra.run.smithi125.stdout:64424509449 2023-10-30T12:28:56.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.751+0000 7fa0af93e640 1 -- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa08c0664d0 msgr2=0x7fa08c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.751+0000 7fa0af93e640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa08c0664d0 0x7fa08c068990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009fd0 tx=0x7fa0a4009190 comp rx=0 tx=0).stop 2023-10-30T12:28:56.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.751+0000 7fa0af93e640 1 -- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a8153c60 msgr2=0x7fa0a81f5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.751+0000 7fa0af93e640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a8153c60 0x7fa0a81f5150 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa0980ab000 tx=0x7fa098019220 comp rx=0 tx=0).stop 2023-10-30T12:28:56.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.751+0000 7fa0af93e640 1 -- 172.21.15.125:0/2270724904 shutdown_connections 2023-10-30T12:28:56.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.751+0000 7fa0af93e640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa08c0664d0 0x7fa08c068990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.751+0000 7fa0af93e640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a8159830 0x7fa0a80b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.751+0000 7fa0af93e640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a8153c60 0x7fa0a81f5150 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.751+0000 7fa0af93e640 1 --2- 172.21.15.125:0/2270724904 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a80c4430 0x7fa0a80ce850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.751+0000 7fa0af93e640 1 -- 172.21.15.125:0/2270724904 >> 172.21.15.125:0/2270724904 conn(0x7fa0a806bfe0 msgr2=0x7fa0a8151d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.751+0000 7fa0af93e640 1 -- 172.21.15.125:0/2270724904 shutdown_connections 2023-10-30T12:28:56.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.751+0000 7fa0af93e640 1 -- 172.21.15.125:0/2270724904 wait complete. 2023-10-30T12:28:56.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.763+0000 7f2638a09640 1 -- 172.21.15.125:0/1778419471 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f263404e170 con 0x7f26340c6b80 2023-10-30T12:28:56.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.763+0000 7f2622ffd640 1 -- 172.21.15.125:0/1778419471 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f26240edf80 con 0x7f26340c6b80 2023-10-30T12:28:56.766 INFO:teuthology.orchestra.run.smithi125.stdout:60129542151 2023-10-30T12:28:56.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.767+0000 7f2638a09640 1 -- 172.21.15.125:0/1778419471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26140664d0 msgr2=0x7f2614068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.767+0000 7f2638a09640 1 --2- 172.21.15.125:0/1778419471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26140664d0 0x7f2614068990 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2628000f70 tx=0x7f26280010b0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.767+0000 7f2638a09640 1 -- 172.21.15.125:0/1778419471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26340c6b80 msgr2=0x7f26341f6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.767+0000 7f2638a09640 1 --2- 172.21.15.125:0/1778419471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26340c6b80 0x7f26341f6380 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2624000c10 tx=0x7f26240191f0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.767+0000 7f2638a09640 1 -- 172.21.15.125:0/1778419471 shutdown_connections 2023-10-30T12:28:56.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.767+0000 7f2638a09640 1 --2- 172.21.15.125:0/1778419471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26140664d0 0x7f2614068990 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.767+0000 7f2638a09640 1 --2- 172.21.15.125:0/1778419471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26341596b0 0x7f26341557c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.767+0000 7f2638a09640 1 --2- 172.21.15.125:0/1778419471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26340c6f60 0x7f26341585f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.767+0000 7f2638a09640 1 --2- 172.21.15.125:0/1778419471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26340c6b80 0x7f26341f6380 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.767+0000 7f2638a09640 1 -- 172.21.15.125:0/1778419471 >> 172.21.15.125:0/1778419471 conn(0x7f26340c7910 msgr2=0x7f2634069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.767+0000 7f2638a09640 1 -- 172.21.15.125:0/1778419471 shutdown_connections 2023-10-30T12:28:56.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.767+0000 7f2638a09640 1 -- 172.21.15.125:0/1778419471 wait complete. 2023-10-30T12:28:56.774 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509449 got 64424509449 for osd.4 2023-10-30T12:28:56.775 DEBUG:teuthology.parallel:result is None 2023-10-30T12:28:56.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.7 2023-10-30T12:28:56.787 DEBUG:teuthology.parallel:result is None 2023-10-30T12:28:56.788 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T12:28:56.788 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:28:56.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.959+0000 7fe32b59e640 1 Processor -- start 2023-10-30T12:28:56.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32b59e640 1 -- start start 2023-10-30T12:28:56.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32b59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe32c1538f0 0x7fe32c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32b59e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe32c15cd50 0x7fe32c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32b59e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe32c0cefa0 0x7fe32c0cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32b59e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe32c15d300 con 0x7fe32c0cefa0 2023-10-30T12:28:56.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32b59e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe32c15d5e0 con 0x7fe32c15cd50 2023-10-30T12:28:56.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32b59e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c15d8c0 con 0x7fe32c1538f0 2023-10-30T12:28:56.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32ad9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe32c0cefa0 0x7fe32c0cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32ad9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe32c0cefa0 0x7fe32c0cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33358/0 (socket says 172.21.15.125:33358) 2023-10-30T12:28:56.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32ad9d640 1 -- 172.21.15.125:0/3643950607 learned_addr learned my addr 172.21.15.125:0/3643950607 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:56.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe329d9b640 1 --2- 172.21.15.125:0/3643950607 >> v2:172.21.15.136:3300/0 conn(0x7fe32c15cd50 0x7fe32c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32a59c640 1 --2- 172.21.15.125:0/3643950607 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe32c1538f0 0x7fe32c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32ad9d640 1 -- 172.21.15.125:0/3643950607 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe32c1538f0 msgr2=0x7fe32c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:56.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32ad9d640 1 --2- 172.21.15.125:0/3643950607 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe32c1538f0 0x7fe32c159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32ad9d640 1 -- 172.21.15.125:0/3643950607 >> v2:172.21.15.136:3300/0 conn(0x7fe32c15cd50 msgr2=0x7fe32c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:56.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32ad9d640 1 --2- 172.21.15.125:0/3643950607 >> v2:172.21.15.136:3300/0 conn(0x7fe32c15cd50 0x7fe32c0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe32ad9d640 1 -- 172.21.15.125:0/3643950607 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe32c15dba0 con 0x7fe32c0cefa0 2023-10-30T12:28:56.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.963+0000 7fe329d9b640 1 --2- 172.21.15.125:0/3643950607 >> v2:172.21.15.136:3300/0 conn(0x7fe32c15cd50 0x7fe32c0c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:56.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32ad9d640 1 --2- 172.21.15.125:0/3643950607 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cefa0 0x7fe32c0cf3f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe32c066410 tx=0x7fe32401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9a3005cf25173ac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32959a640 1 -- 172.21.15.125:0/3643950607 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3240af400 con 0x7fe32c0cefa0 2023-10-30T12:28:56.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32959a640 1 -- 172.21.15.125:0/3643950607 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3240ab050 con 0x7fe32c0cefa0 2023-10-30T12:28:56.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32959a640 1 -- 172.21.15.125:0/3643950607 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3240af780 con 0x7fe32c0cefa0 2023-10-30T12:28:56.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32b59e640 1 -- 172.21.15.125:0/3643950607 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cefa0 msgr2=0x7fe32c0cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32b59e640 1 --2- 172.21.15.125:0/3643950607 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cefa0 0x7fe32c0cf3f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe32c066410 tx=0x7fe32401f1c0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32b59e640 1 -- 172.21.15.125:0/3643950607 shutdown_connections 2023-10-30T12:28:56.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32b59e640 1 --2- 172.21.15.125:0/3643950607 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cefa0 0x7fe32c0cf3f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32b59e640 1 --2- 172.21.15.125:0/3643950607 >> v2:172.21.15.136:3300/0 conn(0x7fe32c15cd50 0x7fe32c0c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32b59e640 1 --2- 172.21.15.125:0/3643950607 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe32c1538f0 0x7fe32c159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32b59e640 1 -- 172.21.15.125:0/3643950607 >> 172.21.15.125:0/3643950607 conn(0x7fe32c06a640 msgr2=0x7fe32c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:56.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32b59e640 1 -- 172.21.15.125:0/3643950607 shutdown_connections 2023-10-30T12:28:56.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32b59e640 1 -- 172.21.15.125:0/3643950607 wait complete. 2023-10-30T12:28:56.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32b59e640 1 Processor -- start 2023-10-30T12:28:56.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.967+0000 7fe32b59e640 1 -- start start 2023-10-30T12:28:56.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32b59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cefa0 0x7fe32c13c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32b59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe32c1538f0 0x7fe32c142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32b59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe32c15cd50 0x7fe32c147a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32b59e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c152ed0 con 0x7fe32c0cefa0 2023-10-30T12:28:56.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32b59e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c0c5270 con 0x7fe32c1538f0 2023-10-30T12:28:56.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32b59e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c142790 con 0x7fe32c15cd50 2023-10-30T12:28:56.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cefa0 0x7fe32c13c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cefa0 0x7fe32c13c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33370/0 (socket says 172.21.15.125:33370) 2023-10-30T12:28:56.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32a59c640 1 -- 172.21.15.125:0/806029529 learned_addr learned my addr 172.21.15.125:0/806029529 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:56.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32ad9d640 1 --2- 172.21.15.125:0/806029529 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe32c15cd50 0x7fe32c147a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe329d9b640 1 --2- 172.21.15.125:0/806029529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe32c1538f0 0x7fe32c142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32a59c640 1 -- 172.21.15.125:0/806029529 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe32c15cd50 msgr2=0x7fe32c147a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32a59c640 1 --2- 172.21.15.125:0/806029529 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe32c15cd50 0x7fe32c147a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32a59c640 1 -- 172.21.15.125:0/806029529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe32c1538f0 msgr2=0x7fe32c142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:56.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32a59c640 1 --2- 172.21.15.125:0/806029529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe32c1538f0 0x7fe32c142250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:56.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32a59c640 1 -- 172.21.15.125:0/806029529 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe324009b30 con 0x7fe32c0cefa0 2023-10-30T12:28:56.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32a59c640 1 --2- 172.21.15.125:0/806029529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cefa0 0x7fe32c13c590 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00d800 tx=0x7fe31c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe31affd640 1 -- 172.21.15.125:0/806029529 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c0096a0 con 0x7fe32c0cefa0 2023-10-30T12:28:56.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe31affd640 1 -- 172.21.15.125:0/806029529 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe31c009800 con 0x7fe32c0cefa0 2023-10-30T12:28:56.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe31affd640 1 -- 172.21.15.125:0/806029529 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe31c009b70 con 0x7fe32c0cefa0 2023-10-30T12:28:56.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32b59e640 1 -- 172.21.15.125:0/806029529 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32c140c00 con 0x7fe32c0cefa0 2023-10-30T12:28:56.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32b59e640 1 -- 172.21.15.125:0/806029529 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32c141180 con 0x7fe32c0cefa0 2023-10-30T12:28:56.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe32b59e640 1 -- 172.21.15.125:0/806029529 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32c04e170 con 0x7fe32c0cefa0 2023-10-30T12:28:56.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.971+0000 7fe31affd640 1 -- 172.21.15.125:0/806029529 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe31c009cd0 con 0x7fe32c0cefa0 2023-10-30T12:28:56.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.975+0000 7fe31affd640 1 --2- 172.21.15.125:0/806029529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe30c0664a0 0x7fe30c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:56.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.975+0000 7fe31affd640 1 -- 172.21.15.125:0/806029529 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe31c084700 con 0x7fe32c0cefa0 2023-10-30T12:28:56.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.975+0000 7fe329d9b640 1 --2- 172.21.15.125:0/806029529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe30c0664a0 0x7fe30c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:56.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.975+0000 7fe329d9b640 1 --2- 172.21.15.125:0/806029529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe30c0664a0 0x7fe30c068960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0cfb70 tx=0x7fe320008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:56.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:56.979+0000 7fe31affd640 1 -- 172.21.15.125:0/806029529 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe31c051a50 con 0x7fe32c0cefa0 2023-10-30T12:28:57.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.123+0000 7fe32b59e640 1 -- 172.21.15.125:0/806029529 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe32c0cb750 con 0x7fe30c0664a0 2023-10-30T12:28:57.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.143+0000 7fe31affd640 1 -- 172.21.15.125:0/806029529 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264584 (secure 0 0 0) 0x7fe32000b6b0 con 0x7fe30c0664a0 2023-10-30T12:28:57.191 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:57.193 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T12:28:57.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.147+0000 7fe32b59e640 1 -- 172.21.15.125:0/806029529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe30c0664a0 msgr2=0x7fe30c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.147+0000 7fe32b59e640 1 --2- 172.21.15.125:0/806029529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe30c0664a0 0x7fe30c068960 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe32c0cfb70 tx=0x7fe320008040 comp rx=0 tx=0).stop 2023-10-30T12:28:57.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.147+0000 7fe32b59e640 1 -- 172.21.15.125:0/806029529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cefa0 msgr2=0x7fe32c13c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.147+0000 7fe32b59e640 1 --2- 172.21.15.125:0/806029529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cefa0 0x7fe32c13c590 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe31c00d800 tx=0x7fe31c00dcc0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.147+0000 7fe32b59e640 1 -- 172.21.15.125:0/806029529 shutdown_connections 2023-10-30T12:28:57.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.147+0000 7fe32b59e640 1 --2- 172.21.15.125:0/806029529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe30c0664a0 0x7fe30c068960 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.147+0000 7fe32b59e640 1 --2- 172.21.15.125:0/806029529 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe32c15cd50 0x7fe32c147a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.147+0000 7fe32b59e640 1 --2- 172.21.15.125:0/806029529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe32c1538f0 0x7fe32c142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.147+0000 7fe32b59e640 1 --2- 172.21.15.125:0/806029529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe32c0cefa0 0x7fe32c13c590 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.147+0000 7fe32b59e640 1 -- 172.21.15.125:0/806029529 >> 172.21.15.125:0/806029529 conn(0x7fe32c06a640 msgr2=0x7fe32c0caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:57.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.147+0000 7fe32b59e640 1 -- 172.21.15.125:0/806029529 shutdown_connections 2023-10-30T12:28:57.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.147+0000 7fe32b59e640 1 -- 172.21.15.125:0/806029529 wait complete. 2023-10-30T12:28:57.197 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-30T12:28:56.963919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593982,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":209,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163,"ondisk_log_size":163,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223224,"kb_used_data":2932,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751496,"statfs":{"total":773094113280,"available":772865531904,"internally_reserved":0,"allocated":3002368,"data_stored":1793704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":225496633},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3614,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.023323"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104854+0000","last_change":"2023-10-30T12:28:44.957488+0000","last_active":"2023-10-30T12:28:46.104854+0000","last_peered":"2023-10-30T12:28:46.104854+0000","last_clean":"2023-10-30T12:28:46.104854+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T12:28:46.104854+0000","last_undegraded":"2023-10-30T12:28:46.104854+0000","last_fullsized":"2023-10-30T12:28:46.104854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:30.240483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106772+0000","last_change":"2023-10-30T12:28:44.978898+0000","last_active":"2023-10-30T12:28:46.106772+0000","last_peered":"2023-10-30T12:28:46.106772+0000","last_clean":"2023-10-30T12:28:46.106772+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T12:28:46.106772+0000","last_undegraded":"2023-10-30T12:28:46.106772+0000","last_fullsized":"2023-10-30T12:28:46.106772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:57:21.463798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106769+0000","last_change":"2023-10-30T12:28:44.980722+0000","last_active":"2023-10-30T12:28:46.106769+0000","last_peered":"2023-10-30T12:28:46.106769+0000","last_clean":"2023-10-30T12:28:46.106769+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T12:28:46.106769+0000","last_undegraded":"2023-10-30T12:28:46.106769+0000","last_fullsized":"2023-10-30T12:28:46.106769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:25.009946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103812+0000","last_change":"2023-10-30T12:28:44.965061+0000","last_active":"2023-10-30T12:28:46.103812+0000","last_peered":"2023-10-30T12:28:46.103812+0000","last_clean":"2023-10-30T12:28:46.103812+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T12:28:46.103812+0000","last_undegraded":"2023-10-30T12:28:46.103812+0000","last_fullsized":"2023-10-30T12:28:46.103812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:23:43.749064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104285+0000","last_change":"2023-10-30T12:28:44.976985+0000","last_active":"2023-10-30T12:28:46.104285+0000","last_peered":"2023-10-30T12:28:46.104285+0000","last_clean":"2023-10-30T12:28:46.104285+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T12:28:46.104285+0000","last_undegraded":"2023-10-30T12:28:46.104285+0000","last_fullsized":"2023-10-30T12:28:46.104285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:08.758140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103030+0000","last_change":"2023-10-30T12:28:44.965453+0000","last_active":"2023-10-30T12:28:46.103030+0000","last_peered":"2023-10-30T12:28:46.103030+0000","last_clean":"2023-10-30T12:28:46.103030+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T12:28:46.103030+0000","last_undegraded":"2023-10-30T12:28:46.103030+0000","last_fullsized":"2023-10-30T12:28:46.103030+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:19.975226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103434+0000","last_change":"2023-10-30T12:28:44.968493+0000","last_active":"2023-10-30T12:28:46.103434+0000","last_peered":"2023-10-30T12:28:46.103434+0000","last_clean":"2023-10-30T12:28:46.103434+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T12:28:46.103434+0000","last_undegraded":"2023-10-30T12:28:46.103434+0000","last_fullsized":"2023-10-30T12:28:46.103434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:58.895635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106402+0000","last_change":"2023-10-30T12:28:44.972695+0000","last_active":"2023-10-30T12:28:46.106402+0000","last_peered":"2023-10-30T12:28:46.106402+0000","last_clean":"2023-10-30T12:28:46.106402+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T12:28:46.106402+0000","last_undegraded":"2023-10-30T12:28:46.106402+0000","last_fullsized":"2023-10-30T12:28:46.106402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:45.353683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939624+0000","last_change":"2023-10-30T12:28:44.946156+0000","last_active":"2023-10-30T12:28:45.939624+0000","last_peered":"2023-10-30T12:28:45.939624+0000","last_clean":"2023-10-30T12:28:45.939624+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T12:28:45.939624+0000","last_undegraded":"2023-10-30T12:28:45.939624+0000","last_fullsized":"2023-10-30T12:28:45.939624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:42:36.408944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104638+0000","last_change":"2023-10-30T12:28:44.957225+0000","last_active":"2023-10-30T12:28:46.104638+0000","last_peered":"2023-10-30T12:28:46.104638+0000","last_clean":"2023-10-30T12:28:46.104638+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T12:28:46.104638+0000","last_undegraded":"2023-10-30T12:28:46.104638+0000","last_fullsized":"2023-10-30T12:28:46.104638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:55:20.194553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104526+0000","last_change":"2023-10-30T12:28:44.956025+0000","last_active":"2023-10-30T12:28:46.104526+0000","last_peered":"2023-10-30T12:28:46.104526+0000","last_clean":"2023-10-30T12:28:46.104526+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T12:28:46.104526+0000","last_undegraded":"2023-10-30T12:28:46.104526+0000","last_fullsized":"2023-10-30T12:28:46.104526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:12.113264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104464+0000","last_change":"2023-10-30T12:28:44.956832+0000","last_active":"2023-10-30T12:28:46.104464+0000","last_peered":"2023-10-30T12:28:46.104464+0000","last_clean":"2023-10-30T12:28:46.104464+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T12:28:46.104464+0000","last_undegraded":"2023-10-30T12:28:46.104464+0000","last_fullsized":"2023-10-30T12:28:46.104464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:36.932024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945824+0000","last_change":"2023-10-30T12:28:44.962961+0000","last_active":"2023-10-30T12:28:45.945824+0000","last_peered":"2023-10-30T12:28:45.945824+0000","last_clean":"2023-10-30T12:28:45.945824+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T12:28:45.945824+0000","last_undegraded":"2023-10-30T12:28:45.945824+0000","last_fullsized":"2023-10-30T12:28:45.945824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:13.200996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945413+0000","last_change":"2023-10-30T12:28:44.968005+0000","last_active":"2023-10-30T12:28:45.945413+0000","last_peered":"2023-10-30T12:28:45.945413+0000","last_clean":"2023-10-30T12:28:45.945413+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T12:28:45.945413+0000","last_undegraded":"2023-10-30T12:28:45.945413+0000","last_fullsized":"2023-10-30T12:28:45.945413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:53:25.168553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106306+0000","last_change":"2023-10-30T12:28:44.977064+0000","last_active":"2023-10-30T12:28:46.106306+0000","last_peered":"2023-10-30T12:28:46.106306+0000","last_clean":"2023-10-30T12:28:46.106306+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T12:28:46.106306+0000","last_undegraded":"2023-10-30T12:28:46.106306+0000","last_fullsized":"2023-10-30T12:28:46.106306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:27:44.043189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102782+0000","last_change":"2023-10-30T12:28:44.950169+0000","last_active":"2023-10-30T12:28:46.102782+0000","last_peered":"2023-10-30T12:28:46.102782+0000","last_clean":"2023-10-30T12:28:46.102782+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T12:28:46.102782+0000","last_undegraded":"2023-10-30T12:28:46.102782+0000","last_fullsized":"2023-10-30T12:28:46.102782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:35.297525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103374+0000","last_change":"2023-10-30T12:28:44.960467+0000","last_active":"2023-10-30T12:28:46.103374+0000","last_peered":"2023-10-30T12:28:46.103374+0000","last_clean":"2023-10-30T12:28:46.103374+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T12:28:46.103374+0000","last_undegraded":"2023-10-30T12:28:46.103374+0000","last_fullsized":"2023-10-30T12:28:46.103374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:33.495256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.108582+0000","last_change":"2023-10-30T12:28:44.968948+0000","last_active":"2023-10-30T12:28:46.108582+0000","last_peered":"2023-10-30T12:28:46.108582+0000","last_clean":"2023-10-30T12:28:46.108582+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T12:28:46.108582+0000","last_undegraded":"2023-10-30T12:28:46.108582+0000","last_fullsized":"2023-10-30T12:28:46.108582+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:26:14.661648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103539+0000","last_change":"2023-10-30T12:28:44.966743+0000","last_active":"2023-10-30T12:28:46.103539+0000","last_peered":"2023-10-30T12:28:46.103539+0000","last_clean":"2023-10-30T12:28:46.103539+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T12:28:46.103539+0000","last_undegraded":"2023-10-30T12:28:46.103539+0000","last_fullsized":"2023-10-30T12:28:46.103539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:30.077107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104499+0000","last_change":"2023-10-30T12:28:44.955323+0000","last_active":"2023-10-30T12:28:46.104499+0000","last_peered":"2023-10-30T12:28:46.104499+0000","last_clean":"2023-10-30T12:28:46.104499+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T12:28:46.104499+0000","last_undegraded":"2023-10-30T12:28:46.104499+0000","last_fullsized":"2023-10-30T12:28:46.104499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:59:22.424571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945425+0000","last_change":"2023-10-30T12:28:44.946048+0000","last_active":"2023-10-30T12:28:45.945425+0000","last_peered":"2023-10-30T12:28:45.945425+0000","last_clean":"2023-10-30T12:28:45.945425+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T12:28:45.945425+0000","last_undegraded":"2023-10-30T12:28:45.945425+0000","last_fullsized":"2023-10-30T12:28:45.945425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:20:21.145943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939444+0000","last_change":"2023-10-30T12:28:44.954167+0000","last_active":"2023-10-30T12:28:45.939444+0000","last_peered":"2023-10-30T12:28:45.939444+0000","last_clean":"2023-10-30T12:28:45.939444+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T12:28:45.939444+0000","last_undegraded":"2023-10-30T12:28:45.939444+0000","last_fullsized":"2023-10-30T12:28:45.939444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:08.457457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106494+0000","last_change":"2023-10-30T12:28:44.982724+0000","last_active":"2023-10-30T12:28:46.106494+0000","last_peered":"2023-10-30T12:28:46.106494+0000","last_clean":"2023-10-30T12:28:46.106494+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T12:28:46.106494+0000","last_undegraded":"2023-10-30T12:28:46.106494+0000","last_fullsized":"2023-10-30T12:28:46.106494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:13.873943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103197+0000","last_change":"2023-10-30T12:28:44.981034+0000","last_active":"2023-10-30T12:28:46.103197+0000","last_peered":"2023-10-30T12:28:46.103197+0000","last_clean":"2023-10-30T12:28:46.103197+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T12:28:46.103197+0000","last_undegraded":"2023-10-30T12:28:46.103197+0000","last_fullsized":"2023-10-30T12:28:46.103197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:39:25.263228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107211+0000","last_change":"2023-10-30T12:28:44.975010+0000","last_active":"2023-10-30T12:28:46.107211+0000","last_peered":"2023-10-30T12:28:46.107211+0000","last_clean":"2023-10-30T12:28:46.107211+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T12:28:46.107211+0000","last_undegraded":"2023-10-30T12:28:46.107211+0000","last_fullsized":"2023-10-30T12:28:46.107211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:13.768876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104243+0000","last_change":"2023-10-30T12:28:44.951099+0000","last_active":"2023-10-30T12:28:46.104243+0000","last_peered":"2023-10-30T12:28:46.104243+0000","last_clean":"2023-10-30T12:28:46.104243+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T12:28:46.104243+0000","last_undegraded":"2023-10-30T12:28:46.104243+0000","last_fullsized":"2023-10-30T12:28:46.104243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:21:27.902671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946065+0000","last_change":"2023-10-30T12:28:44.959915+0000","last_active":"2023-10-30T12:28:45.946065+0000","last_peered":"2023-10-30T12:28:45.946065+0000","last_clean":"2023-10-30T12:28:45.946065+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T12:28:45.946065+0000","last_undegraded":"2023-10-30T12:28:45.946065+0000","last_fullsized":"2023-10-30T12:28:45.946065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:57:49.711309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946417+0000","last_change":"2023-10-30T12:28:44.972677+0000","last_active":"2023-10-30T12:28:45.946417+0000","last_peered":"2023-10-30T12:28:45.946417+0000","last_clean":"2023-10-30T12:28:45.946417+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T12:28:45.946417+0000","last_undegraded":"2023-10-30T12:28:45.946417+0000","last_fullsized":"2023-10-30T12:28:45.946417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:12.072355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106685+0000","last_change":"2023-10-30T12:28:44.975973+0000","last_active":"2023-10-30T12:28:46.106685+0000","last_peered":"2023-10-30T12:28:46.106685+0000","last_clean":"2023-10-30T12:28:46.106685+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T12:28:46.106685+0000","last_undegraded":"2023-10-30T12:28:46.106685+0000","last_fullsized":"2023-10-30T12:28:46.106685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:48.152823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103531+0000","last_change":"2023-10-30T12:28:44.980814+0000","last_active":"2023-10-30T12:28:46.103531+0000","last_peered":"2023-10-30T12:28:46.103531+0000","last_clean":"2023-10-30T12:28:46.103531+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T12:28:46.103531+0000","last_undegraded":"2023-10-30T12:28:46.103531+0000","last_fullsized":"2023-10-30T12:28:46.103531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:51:22.453883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107798+0000","last_change":"2023-10-30T12:28:44.965026+0000","last_active":"2023-10-30T12:28:46.107798+0000","last_peered":"2023-10-30T12:28:46.107798+0000","last_clean":"2023-10-30T12:28:46.107798+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T12:28:46.107798+0000","last_undegraded":"2023-10-30T12:28:46.107798+0000","last_fullsized":"2023-10-30T12:28:46.107798+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:04:08.680744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102782+0000","last_change":"2023-10-30T12:28:44.973401+0000","last_active":"2023-10-30T12:28:46.102782+0000","last_peered":"2023-10-30T12:28:46.102782+0000","last_clean":"2023-10-30T12:28:46.102782+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T12:28:46.102782+0000","last_undegraded":"2023-10-30T12:28:46.102782+0000","last_fullsized":"2023-10-30T12:28:46.102782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:34.198424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939578+0000","last_change":"2023-10-30T12:28:44.971817+0000","last_active":"2023-10-30T12:28:45.939578+0000","last_peered":"2023-10-30T12:28:45.939578+0000","last_clean":"2023-10-30T12:28:45.939578+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T12:28:45.939578+0000","last_undegraded":"2023-10-30T12:28:45.939578+0000","last_fullsized":"2023-10-30T12:28:45.939578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:45:35.205305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.130160+0000","last_change":"2023-10-30T12:28:44.074842+0000","last_active":"2023-10-30T12:28:55.130160+0000","last_peered":"2023-10-30T12:28:55.130160+0000","last_clean":"2023-10-30T12:28:55.130160+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T12:28:55.130160+0000","last_undegraded":"2023-10-30T12:28:55.130160+0000","last_fullsized":"2023-10-30T12:28:55.130160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:01.428785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945765+0000","last_change":"2023-10-30T12:28:44.958560+0000","last_active":"2023-10-30T12:28:45.945765+0000","last_peered":"2023-10-30T12:28:45.945765+0000","last_clean":"2023-10-30T12:28:45.945765+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T12:28:45.945765+0000","last_undegraded":"2023-10-30T12:28:45.945765+0000","last_fullsized":"2023-10-30T12:28:45.945765+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:45.564493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:47.984730+0000","last_change":"2023-10-30T12:28:44.060621+0000","last_active":"2023-10-30T12:28:47.984730+0000","last_peered":"2023-10-30T12:28:47.984730+0000","last_clean":"2023-10-30T12:28:47.984730+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T12:28:47.984730+0000","last_undegraded":"2023-10-30T12:28:47.984730+0000","last_fullsized":"2023-10-30T12:28:47.984730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:53.741238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945157+0000","last_change":"2023-10-30T12:28:44.060171+0000","last_active":"2023-10-30T12:28:45.945157+0000","last_peered":"2023-10-30T12:28:45.945157+0000","last_clean":"2023-10-30T12:28:45.945157+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T12:28:45.945157+0000","last_undegraded":"2023-10-30T12:28:45.945157+0000","last_fullsized":"2023-10-30T12:28:45.945157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:18.181418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106893+0000","last_change":"2023-10-30T12:28:44.978631+0000","last_active":"2023-10-30T12:28:46.106893+0000","last_peered":"2023-10-30T12:28:46.106893+0000","last_clean":"2023-10-30T12:28:46.106893+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T12:28:46.106893+0000","last_undegraded":"2023-10-30T12:28:46.106893+0000","last_fullsized":"2023-10-30T12:28:46.106893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:37.006427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103283+0000","last_change":"2023-10-30T12:28:44.058798+0000","last_active":"2023-10-30T12:28:46.103283+0000","last_peered":"2023-10-30T12:28:46.103283+0000","last_clean":"2023-10-30T12:28:46.103283+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T12:28:46.103283+0000","last_undegraded":"2023-10-30T12:28:46.103283+0000","last_fullsized":"2023-10-30T12:28:46.103283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:08.643080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945025+0000","last_change":"2023-10-30T12:28:44.949118+0000","last_active":"2023-10-30T12:28:45.945025+0000","last_peered":"2023-10-30T12:28:45.945025+0000","last_clean":"2023-10-30T12:28:45.945025+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T12:28:45.945025+0000","last_undegraded":"2023-10-30T12:28:45.945025+0000","last_fullsized":"2023-10-30T12:28:45.945025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:11.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939606+0000","last_change":"2023-10-30T12:28:44.063439+0000","last_active":"2023-10-30T12:28:45.939606+0000","last_peered":"2023-10-30T12:28:45.939606+0000","last_clean":"2023-10-30T12:28:45.939606+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T12:28:45.939606+0000","last_undegraded":"2023-10-30T12:28:45.939606+0000","last_fullsized":"2023-10-30T12:28:45.939606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:08.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104375+0000","last_change":"2023-10-30T12:28:44.951319+0000","last_active":"2023-10-30T12:28:46.104375+0000","last_peered":"2023-10-30T12:28:46.104375+0000","last_clean":"2023-10-30T12:28:46.104375+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T12:28:46.104375+0000","last_undegraded":"2023-10-30T12:28:46.104375+0000","last_fullsized":"2023-10-30T12:28:46.104375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:20.108360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104752+0000","last_change":"2023-10-30T12:28:44.052641+0000","last_active":"2023-10-30T12:28:46.104752+0000","last_peered":"2023-10-30T12:28:46.104752+0000","last_clean":"2023-10-30T12:28:46.104752+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T12:28:46.104752+0000","last_undegraded":"2023-10-30T12:28:46.104752+0000","last_fullsized":"2023-10-30T12:28:46.104752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107640+0000","last_change":"2023-10-30T12:28:44.981614+0000","last_active":"2023-10-30T12:28:46.107640+0000","last_peered":"2023-10-30T12:28:46.107640+0000","last_clean":"2023-10-30T12:28:46.107640+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T12:28:46.107640+0000","last_undegraded":"2023-10-30T12:28:46.107640+0000","last_fullsized":"2023-10-30T12:28:46.107640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:15:11.750243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.068083+0000","last_change":"2023-10-30T12:28:44.075783+0000","last_active":"2023-10-30T12:28:50.068083+0000","last_peered":"2023-10-30T12:28:50.068083+0000","last_clean":"2023-10-30T12:28:50.068083+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T12:28:50.068083+0000","last_undegraded":"2023-10-30T12:28:50.068083+0000","last_fullsized":"2023-10-30T12:28:50.068083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:01.094929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103142+0000","last_change":"2023-10-30T12:28:44.950899+0000","last_active":"2023-10-30T12:28:46.103142+0000","last_peered":"2023-10-30T12:28:46.103142+0000","last_clean":"2023-10-30T12:28:46.103142+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T12:28:46.103142+0000","last_undegraded":"2023-10-30T12:28:46.103142+0000","last_fullsized":"2023-10-30T12:28:46.103142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:34.005863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.065815+0000","last_change":"2023-10-30T12:28:44.063367+0000","last_active":"2023-10-30T12:28:50.065815+0000","last_peered":"2023-10-30T12:28:50.065815+0000","last_clean":"2023-10-30T12:28:50.065815+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T12:28:50.065815+0000","last_undegraded":"2023-10-30T12:28:50.065815+0000","last_fullsized":"2023-10-30T12:28:50.065815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:45.585121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104587+0000","last_change":"2023-10-30T12:28:44.972056+0000","last_active":"2023-10-30T12:28:46.104587+0000","last_peered":"2023-10-30T12:28:46.104587+0000","last_clean":"2023-10-30T12:28:46.104587+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T12:28:46.104587+0000","last_undegraded":"2023-10-30T12:28:46.104587+0000","last_fullsized":"2023-10-30T12:28:46.104587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:06:08.345856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.132476+0000","last_change":"2023-10-30T12:28:44.065545+0000","last_active":"2023-10-30T12:28:55.132476+0000","last_peered":"2023-10-30T12:28:55.132476+0000","last_clean":"2023-10-30T12:28:55.132476+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T12:28:55.132476+0000","last_undegraded":"2023-10-30T12:28:55.132476+0000","last_fullsized":"2023-10-30T12:28:55.132476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:09.900189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945563+0000","last_change":"2023-10-30T12:28:44.961237+0000","last_active":"2023-10-30T12:28:45.945563+0000","last_peered":"2023-10-30T12:28:45.945563+0000","last_clean":"2023-10-30T12:28:45.945563+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T12:28:45.945563+0000","last_undegraded":"2023-10-30T12:28:45.945563+0000","last_fullsized":"2023-10-30T12:28:45.945563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:17.160724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103797+0000","last_change":"2023-10-30T12:28:44.061922+0000","last_active":"2023-10-30T12:28:46.103797+0000","last_peered":"2023-10-30T12:28:46.103797+0000","last_clean":"2023-10-30T12:28:46.103797+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T12:28:46.103797+0000","last_undegraded":"2023-10-30T12:28:46.103797+0000","last_fullsized":"2023-10-30T12:28:46.103797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:34.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103258+0000","last_change":"2023-10-30T12:28:44.958245+0000","last_active":"2023-10-30T12:28:46.103258+0000","last_peered":"2023-10-30T12:28:46.103258+0000","last_clean":"2023-10-30T12:28:46.103258+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T12:28:46.103258+0000","last_undegraded":"2023-10-30T12:28:46.103258+0000","last_fullsized":"2023-10-30T12:28:46.103258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:42:10.840490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103980+0000","last_change":"2023-10-30T12:28:44.059784+0000","last_active":"2023-10-30T12:28:46.103980+0000","last_peered":"2023-10-30T12:28:46.103980+0000","last_clean":"2023-10-30T12:28:46.103980+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T12:28:46.103980+0000","last_undegraded":"2023-10-30T12:28:46.103980+0000","last_fullsized":"2023-10-30T12:28:46.103980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:11.468221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939230+0000","last_change":"2023-10-30T12:28:44.980813+0000","last_active":"2023-10-30T12:28:45.939230+0000","last_peered":"2023-10-30T12:28:45.939230+0000","last_clean":"2023-10-30T12:28:45.939230+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T12:28:45.939230+0000","last_undegraded":"2023-10-30T12:28:45.939230+0000","last_fullsized":"2023-10-30T12:28:45.939230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:10.858970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.070290+0000","last_change":"2023-10-30T12:28:44.065885+0000","last_active":"2023-10-30T12:28:50.070290+0000","last_peered":"2023-10-30T12:28:50.070290+0000","last_clean":"2023-10-30T12:28:50.070290+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T12:28:50.070290+0000","last_undegraded":"2023-10-30T12:28:50.070290+0000","last_fullsized":"2023-10-30T12:28:50.070290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:26.313595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103080+0000","last_change":"2023-10-30T12:28:44.952458+0000","last_active":"2023-10-30T12:28:46.103080+0000","last_peered":"2023-10-30T12:28:46.103080+0000","last_clean":"2023-10-30T12:28:46.103080+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T12:28:46.103080+0000","last_undegraded":"2023-10-30T12:28:46.103080+0000","last_fullsized":"2023-10-30T12:28:46.103080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:48:19.752492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:47.984275+0000","last_change":"2023-10-30T12:28:44.080011+0000","last_active":"2023-10-30T12:28:47.984275+0000","last_peered":"2023-10-30T12:28:47.984275+0000","last_clean":"2023-10-30T12:28:47.984275+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T12:28:47.984275+0000","last_undegraded":"2023-10-30T12:28:47.984275+0000","last_fullsized":"2023-10-30T12:28:47.984275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104494+0000","last_change":"2023-10-30T12:28:44.961296+0000","last_active":"2023-10-30T12:28:46.104494+0000","last_peered":"2023-10-30T12:28:46.104494+0000","last_clean":"2023-10-30T12:28:46.104494+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T12:28:46.104494+0000","last_undegraded":"2023-10-30T12:28:46.104494+0000","last_fullsized":"2023-10-30T12:28:46.104494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:56.540392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.066004+0000","last_change":"2023-10-30T12:28:44.076352+0000","last_active":"2023-10-30T12:28:50.066004+0000","last_peered":"2023-10-30T12:28:50.066004+0000","last_clean":"2023-10-30T12:28:50.066004+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T12:28:50.066004+0000","last_undegraded":"2023-10-30T12:28:50.066004+0000","last_fullsized":"2023-10-30T12:28:50.066004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104918+0000","last_change":"2023-10-30T12:28:44.956918+0000","last_active":"2023-10-30T12:28:46.104918+0000","last_peered":"2023-10-30T12:28:46.104918+0000","last_clean":"2023-10-30T12:28:46.104918+0000","last_became_active":"2023-10-30T12:28:44.956397+0000","last_became_peered":"2023-10-30T12:28:44.956397+0000","last_unstale":"2023-10-30T12:28:46.104918+0000","last_undegraded":"2023-10-30T12:28:46.104918+0000","last_fullsized":"2023-10-30T12:28:46.104918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:26.252196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106813+0000","last_change":"2023-10-30T12:28:44.075684+0000","last_active":"2023-10-30T12:28:46.106813+0000","last_peered":"2023-10-30T12:28:46.106813+0000","last_clean":"2023-10-30T12:28:46.106813+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T12:28:46.106813+0000","last_undegraded":"2023-10-30T12:28:46.106813+0000","last_fullsized":"2023-10-30T12:28:46.106813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:07.375795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104588+0000","last_change":"2023-10-30T12:28:44.956047+0000","last_active":"2023-10-30T12:28:46.104588+0000","last_peered":"2023-10-30T12:28:46.104588+0000","last_clean":"2023-10-30T12:28:46.104588+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T12:28:46.104588+0000","last_undegraded":"2023-10-30T12:28:46.104588+0000","last_fullsized":"2023-10-30T12:28:46.104588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:49.924947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946632+0000","last_change":"2023-10-30T12:28:44.060686+0000","last_active":"2023-10-30T12:28:45.946632+0000","last_peered":"2023-10-30T12:28:45.946632+0000","last_clean":"2023-10-30T12:28:45.946632+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T12:28:45.946632+0000","last_undegraded":"2023-10-30T12:28:45.946632+0000","last_fullsized":"2023-10-30T12:28:45.946632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:23.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103562+0000","last_change":"2023-10-30T12:28:44.946758+0000","last_active":"2023-10-30T12:28:46.103562+0000","last_peered":"2023-10-30T12:28:46.103562+0000","last_clean":"2023-10-30T12:28:46.103562+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T12:28:46.103562+0000","last_undegraded":"2023-10-30T12:28:46.103562+0000","last_fullsized":"2023-10-30T12:28:46.103562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:20.520148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.932509+0000","last_change":"2023-10-30T12:28:33.433824+0000","last_active":"2023-10-30T12:28:44.932509+0000","last_peered":"2023-10-30T12:28:44.932509+0000","last_clean":"2023-10-30T12:28:44.932509+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T12:28:44.932509+0000","last_undegraded":"2023-10-30T12:28:44.932509+0000","last_fullsized":"2023-10-30T12:28:44.932509+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_clean_scrub_stamp":"2023-10-30T12:28:31.408255+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:17.624179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103427+0000","last_change":"2023-10-30T12:28:44.059051+0000","last_active":"2023-10-30T12:28:46.103427+0000","last_peered":"2023-10-30T12:28:46.103427+0000","last_clean":"2023-10-30T12:28:46.103427+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T12:28:46.103427+0000","last_undegraded":"2023-10-30T12:28:46.103427+0000","last_fullsized":"2023-10-30T12:28:46.103427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:54.810636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945062+0000","last_change":"2023-10-30T12:28:44.965183+0000","last_active":"2023-10-30T12:28:45.945062+0000","last_peered":"2023-10-30T12:28:45.945062+0000","last_clean":"2023-10-30T12:28:45.945062+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T12:28:45.945062+0000","last_undegraded":"2023-10-30T12:28:45.945062+0000","last_fullsized":"2023-10-30T12:28:45.945062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:57.655687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103361+0000","last_change":"2023-10-30T12:28:44.080378+0000","last_active":"2023-10-30T12:28:46.103361+0000","last_peered":"2023-10-30T12:28:46.103361+0000","last_clean":"2023-10-30T12:28:46.103361+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T12:28:46.103361+0000","last_undegraded":"2023-10-30T12:28:46.103361+0000","last_fullsized":"2023-10-30T12:28:46.103361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:22.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944927+0000","last_change":"2023-10-30T12:28:44.961659+0000","last_active":"2023-10-30T12:28:45.944927+0000","last_peered":"2023-10-30T12:28:45.944927+0000","last_clean":"2023-10-30T12:28:45.944927+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T12:28:45.944927+0000","last_undegraded":"2023-10-30T12:28:45.944927+0000","last_fullsized":"2023-10-30T12:28:45.944927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:34:00.000084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107831+0000","last_change":"2023-10-30T12:28:44.082480+0000","last_active":"2023-10-30T12:28:46.107831+0000","last_peered":"2023-10-30T12:28:46.107831+0000","last_clean":"2023-10-30T12:28:46.107831+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T12:28:46.107831+0000","last_undegraded":"2023-10-30T12:28:46.107831+0000","last_fullsized":"2023-10-30T12:28:46.107831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:20.126782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106385+0000","last_change":"2023-10-30T12:28:44.973906+0000","last_active":"2023-10-30T12:28:46.106385+0000","last_peered":"2023-10-30T12:28:46.106385+0000","last_clean":"2023-10-30T12:28:46.106385+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T12:28:46.106385+0000","last_undegraded":"2023-10-30T12:28:46.106385+0000","last_fullsized":"2023-10-30T12:28:46.106385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:31.845894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103873+0000","last_change":"2023-10-30T12:28:44.059806+0000","last_active":"2023-10-30T12:28:46.103873+0000","last_peered":"2023-10-30T12:28:46.103873+0000","last_clean":"2023-10-30T12:28:46.103873+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T12:28:46.103873+0000","last_undegraded":"2023-10-30T12:28:46.103873+0000","last_fullsized":"2023-10-30T12:28:46.103873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:50.939946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104302+0000","last_change":"2023-10-30T12:28:44.955423+0000","last_active":"2023-10-30T12:28:46.104302+0000","last_peered":"2023-10-30T12:28:46.104302+0000","last_clean":"2023-10-30T12:28:46.104302+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T12:28:46.104302+0000","last_undegraded":"2023-10-30T12:28:46.104302+0000","last_fullsized":"2023-10-30T12:28:46.104302+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:30.054048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.982937+0000","last_change":"2023-10-30T12:28:44.074970+0000","last_active":"2023-10-30T12:28:45.982937+0000","last_peered":"2023-10-30T12:28:45.982937+0000","last_clean":"2023-10-30T12:28:45.982937+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T12:28:45.982937+0000","last_undegraded":"2023-10-30T12:28:45.982937+0000","last_fullsized":"2023-10-30T12:28:45.982937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:33.562703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103885+0000","last_change":"2023-10-30T12:28:44.973482+0000","last_active":"2023-10-30T12:28:46.103885+0000","last_peered":"2023-10-30T12:28:46.103885+0000","last_clean":"2023-10-30T12:28:46.103885+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T12:28:46.103885+0000","last_undegraded":"2023-10-30T12:28:46.103885+0000","last_fullsized":"2023-10-30T12:28:46.103885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:25.393161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103579+0000","last_change":"2023-10-30T12:28:44.063256+0000","last_active":"2023-10-30T12:28:46.103579+0000","last_peered":"2023-10-30T12:28:46.103579+0000","last_clean":"2023-10-30T12:28:46.103579+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T12:28:46.103579+0000","last_undegraded":"2023-10-30T12:28:46.103579+0000","last_fullsized":"2023-10-30T12:28:46.103579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:01:36.144966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946945+0000","last_change":"2023-10-30T12:28:44.975443+0000","last_active":"2023-10-30T12:28:45.946945+0000","last_peered":"2023-10-30T12:28:45.946945+0000","last_clean":"2023-10-30T12:28:45.946945+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T12:28:45.946945+0000","last_undegraded":"2023-10-30T12:28:45.946945+0000","last_fullsized":"2023-10-30T12:28:45.946945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:23.765723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946972+0000","last_change":"2023-10-30T12:28:44.055374+0000","last_active":"2023-10-30T12:28:45.946972+0000","last_peered":"2023-10-30T12:28:45.946972+0000","last_clean":"2023-10-30T12:28:45.946972+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T12:28:45.946972+0000","last_undegraded":"2023-10-30T12:28:45.946972+0000","last_fullsized":"2023-10-30T12:28:45.946972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946991+0000","last_change":"2023-10-30T12:28:44.974359+0000","last_active":"2023-10-30T12:28:45.946991+0000","last_peered":"2023-10-30T12:28:45.946991+0000","last_clean":"2023-10-30T12:28:45.946991+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T12:28:45.946991+0000","last_undegraded":"2023-10-30T12:28:45.946991+0000","last_fullsized":"2023-10-30T12:28:45.946991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:48.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.063134+0000","last_change":"2023-10-30T12:28:44.058663+0000","last_active":"2023-10-30T12:28:50.063134+0000","last_peered":"2023-10-30T12:28:50.063134+0000","last_clean":"2023-10-30T12:28:50.063134+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T12:28:50.063134+0000","last_undegraded":"2023-10-30T12:28:50.063134+0000","last_fullsized":"2023-10-30T12:28:50.063134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:42:22.279653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946815+0000","last_change":"2023-10-30T12:28:44.957471+0000","last_active":"2023-10-30T12:28:45.946815+0000","last_peered":"2023-10-30T12:28:45.946815+0000","last_clean":"2023-10-30T12:28:45.946815+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T12:28:45.946815+0000","last_undegraded":"2023-10-30T12:28:45.946815+0000","last_fullsized":"2023-10-30T12:28:45.946815+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:27:38.669452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102847+0000","last_change":"2023-10-30T12:28:44.054338+0000","last_active":"2023-10-30T12:28:46.102847+0000","last_peered":"2023-10-30T12:28:46.102847+0000","last_clean":"2023-10-30T12:28:46.102847+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T12:28:46.102847+0000","last_undegraded":"2023-10-30T12:28:46.102847+0000","last_fullsized":"2023-10-30T12:28:46.102847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:55:35.370652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103200+0000","last_change":"2023-10-30T12:28:44.965759+0000","last_active":"2023-10-30T12:28:46.103200+0000","last_peered":"2023-10-30T12:28:46.103200+0000","last_clean":"2023-10-30T12:28:46.103200+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T12:28:46.103200+0000","last_undegraded":"2023-10-30T12:28:46.103200+0000","last_fullsized":"2023-10-30T12:28:46.103200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:13:52.965500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944989+0000","last_change":"2023-10-30T12:28:44.064606+0000","last_active":"2023-10-30T12:28:45.944989+0000","last_peered":"2023-10-30T12:28:45.944989+0000","last_clean":"2023-10-30T12:28:45.944989+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T12:28:45.944989+0000","last_undegraded":"2023-10-30T12:28:45.944989+0000","last_fullsized":"2023-10-30T12:28:45.944989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944678+0000","last_change":"2023-10-30T12:28:44.958947+0000","last_active":"2023-10-30T12:28:45.944678+0000","last_peered":"2023-10-30T12:28:45.944678+0000","last_clean":"2023-10-30T12:28:45.944678+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T12:28:45.944678+0000","last_undegraded":"2023-10-30T12:28:45.944678+0000","last_fullsized":"2023-10-30T12:28:45.944678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:01:16.329994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945603+0000","last_change":"2023-10-30T12:28:44.072754+0000","last_active":"2023-10-30T12:28:45.945603+0000","last_peered":"2023-10-30T12:28:45.945603+0000","last_clean":"2023-10-30T12:28:45.945603+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T12:28:45.945603+0000","last_undegraded":"2023-10-30T12:28:45.945603+0000","last_fullsized":"2023-10-30T12:28:45.945603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:51:55.865069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.108063+0000","last_change":"2023-10-30T12:28:44.978376+0000","last_active":"2023-10-30T12:28:46.108063+0000","last_peered":"2023-10-30T12:28:46.108063+0000","last_clean":"2023-10-30T12:28:46.108063+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T12:28:46.108063+0000","last_undegraded":"2023-10-30T12:28:46.108063+0000","last_fullsized":"2023-10-30T12:28:46.108063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:59.643447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.057363+0000","last_change":"2023-10-30T12:28:44.058801+0000","last_active":"2023-10-30T12:28:50.057363+0000","last_peered":"2023-10-30T12:28:50.057363+0000","last_clean":"2023-10-30T12:28:50.057363+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T12:28:50.057363+0000","last_undegraded":"2023-10-30T12:28:50.057363+0000","last_fullsized":"2023-10-30T12:28:50.057363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:11:24.706813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103647+0000","last_change":"2023-10-30T12:28:44.943292+0000","last_active":"2023-10-30T12:28:46.103647+0000","last_peered":"2023-10-30T12:28:46.103647+0000","last_clean":"2023-10-30T12:28:46.103647+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T12:28:46.103647+0000","last_undegraded":"2023-10-30T12:28:46.103647+0000","last_fullsized":"2023-10-30T12:28:46.103647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:37.538525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103472+0000","last_change":"2023-10-30T12:28:44.958607+0000","last_active":"2023-10-30T12:28:46.103472+0000","last_peered":"2023-10-30T12:28:46.103472+0000","last_clean":"2023-10-30T12:28:46.103472+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T12:28:46.103472+0000","last_undegraded":"2023-10-30T12:28:46.103472+0000","last_fullsized":"2023-10-30T12:28:46.103472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:28:55.169198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.059856+0000","last_change":"2023-10-30T12:28:44.054652+0000","last_active":"2023-10-30T12:28:50.059856+0000","last_peered":"2023-10-30T12:28:50.059856+0000","last_clean":"2023-10-30T12:28:50.059856+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T12:28:50.059856+0000","last_undegraded":"2023-10-30T12:28:50.059856+0000","last_fullsized":"2023-10-30T12:28:50.059856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:05.023374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939383+0000","last_change":"2023-10-30T12:28:44.958246+0000","last_active":"2023-10-30T12:28:45.939383+0000","last_peered":"2023-10-30T12:28:45.939383+0000","last_clean":"2023-10-30T12:28:45.939383+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T12:28:45.939383+0000","last_undegraded":"2023-10-30T12:28:45.939383+0000","last_fullsized":"2023-10-30T12:28:45.939383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:36.947398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.057546+0000","last_change":"2023-10-30T12:28:44.052633+0000","last_active":"2023-10-30T12:28:50.057546+0000","last_peered":"2023-10-30T12:28:50.057546+0000","last_clean":"2023-10-30T12:28:50.057546+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T12:28:50.057546+0000","last_undegraded":"2023-10-30T12:28:50.057546+0000","last_fullsized":"2023-10-30T12:28:50.057546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107324+0000","last_change":"2023-10-30T12:28:44.974457+0000","last_active":"2023-10-30T12:28:46.107324+0000","last_peered":"2023-10-30T12:28:46.107324+0000","last_clean":"2023-10-30T12:28:46.107324+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T12:28:46.107324+0000","last_undegraded":"2023-10-30T12:28:46.107324+0000","last_fullsized":"2023-10-30T12:28:46.107324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:25:56.791053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107384+0000","last_change":"2023-10-30T12:28:44.075493+0000","last_active":"2023-10-30T12:28:46.107384+0000","last_peered":"2023-10-30T12:28:46.107384+0000","last_clean":"2023-10-30T12:28:46.107384+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T12:28:46.107384+0000","last_undegraded":"2023-10-30T12:28:46.107384+0000","last_fullsized":"2023-10-30T12:28:46.107384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:00:11.718669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107054+0000","last_change":"2023-10-30T12:28:44.974424+0000","last_active":"2023-10-30T12:28:46.107054+0000","last_peered":"2023-10-30T12:28:46.107054+0000","last_clean":"2023-10-30T12:28:46.107054+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T12:28:46.107054+0000","last_undegraded":"2023-10-30T12:28:46.107054+0000","last_fullsized":"2023-10-30T12:28:46.107054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:14.375671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.066270+0000","last_change":"2023-10-30T12:28:44.065843+0000","last_active":"2023-10-30T12:28:50.066270+0000","last_peered":"2023-10-30T12:28:50.066270+0000","last_clean":"2023-10-30T12:28:50.066270+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T12:28:50.066270+0000","last_undegraded":"2023-10-30T12:28:50.066270+0000","last_fullsized":"2023-10-30T12:28:50.066270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:32.405199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3614,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":6024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476745,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":76450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]}]},{"osd":2,"up_from":16,"seq":68719476745,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]}]},{"osd":1,"up_from":16,"seq":68719476743,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":815104,"data_stored":666306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]}]},{"osd":0,"up_from":16,"seq":68719476744,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509448,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":75868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]}]},{"osd":4,"up_from":15,"seq":64424509449,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":78152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]}]},{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":76574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6859999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5860000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":827392,"data_stored":666950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:28:57.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:28:57.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.335+0000 7fecd73aa640 1 Processor -- start 2023-10-30T12:28:57.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.335+0000 7fecd73aa640 1 -- start start 2023-10-30T12:28:57.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.339+0000 7fecd73aa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fecd015a410 0x7fecd006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.339+0000 7fecd73aa640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fecd006cb80 0x7fecd006cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.339+0000 7fecd73aa640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecd006d530 con 0x7fecd0069e90 2023-10-30T12:28:57.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.339+0000 7fecd73aa640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd00c8570 con 0x7fecd015a410 2023-10-30T12:28:57.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.339+0000 7fecd73aa640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fecd015a850 con 0x7fecd006cb80 2023-10-30T12:28:57.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.339+0000 7fecd4945640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fecd015a410 0x7fecd006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.339+0000 7fecd4945640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fecd015a410 0x7fecd006a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33386/0 (socket says 172.21.15.125:33386) 2023-10-30T12:28:57.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.339+0000 7fecd4945640 1 -- 172.21.15.125:0/4223872225 learned_addr learned my addr 172.21.15.125:0/4223872225 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:57.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.339+0000 7fecd5146640 1 --1- 172.21.15.125:0/4223872225 >> v1:172.21.15.125:6789/0 conn(0x7fecd0069e90 0x7fecd00c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54174/0 (socket says 172.21.15.125:54174) 2023-10-30T12:28:57.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.339+0000 7fecd5947640 1 --2- 172.21.15.125:0/4223872225 >> v2:172.21.15.136:3300/0 conn(0x7fecd006cb80 0x7fecd006cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.339+0000 7fecd4945640 1 -- 172.21.15.125:0/4223872225 >> v2:172.21.15.136:3300/0 conn(0x7fecd006cb80 msgr2=0x7fecd006cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:57.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.339+0000 7fecd4945640 1 --2- 172.21.15.125:0/4223872225 >> v2:172.21.15.136:3300/0 conn(0x7fecd006cb80 0x7fecd006cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.339+0000 7fecd4945640 1 -- 172.21.15.125:0/4223872225 >> v1:172.21.15.125:6789/0 conn(0x7fecd0069e90 legacy=0x7fecd00c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.339+0000 7fecd4945640 1 -- 172.21.15.125:0/4223872225 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd00c6c10 con 0x7fecd015a410 2023-10-30T12:28:57.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd4945640 1 --2- 172.21.15.125:0/4223872225 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fecd015a410 0x7fecd006a270 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fecd00ca5c0 tx=0x7fecc001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0158f87eeb1ddb7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:57.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecc7fff640 1 -- 172.21.15.125:0/4223872225 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc00ab600 con 0x7fecd015a410 2023-10-30T12:28:57.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecc7fff640 1 -- 172.21.15.125:0/4223872225 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecc00a7050 con 0x7fecd015a410 2023-10-30T12:28:57.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecc7fff640 1 -- 172.21.15.125:0/4223872225 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc00ab980 con 0x7fecd015a410 2023-10-30T12:28:57.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 -- 172.21.15.125:0/4223872225 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fecd015a410 msgr2=0x7fecd006a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 --2- 172.21.15.125:0/4223872225 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fecd015a410 0x7fecd006a270 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fecd00ca5c0 tx=0x7fecc001c500 comp rx=0 tx=0).stop 2023-10-30T12:28:57.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 -- 172.21.15.125:0/4223872225 shutdown_connections 2023-10-30T12:28:57.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 --2- 172.21.15.125:0/4223872225 >> v2:172.21.15.136:3300/0 conn(0x7fecd006cb80 0x7fecd006cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 --2- 172.21.15.125:0/4223872225 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fecd015a410 0x7fecd006a270 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 -- 172.21.15.125:0/4223872225 >> 172.21.15.125:0/4223872225 conn(0x7fecd006dfd0 msgr2=0x7fecd006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:57.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 -- 172.21.15.125:0/4223872225 shutdown_connections 2023-10-30T12:28:57.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 -- 172.21.15.125:0/4223872225 wait complete. 2023-10-30T12:28:57.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 Processor -- start 2023-10-30T12:28:57.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 -- start start 2023-10-30T12:28:57.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fecd0069e90 0x7fecd015a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fecd006cb80 0x7fecd0152110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fecd015a410 0x7fecd01f4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd00cb1e0 con 0x7fecd0069e90 2023-10-30T12:28:57.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd00cb9f0 con 0x7fecd006cb80 2023-10-30T12:28:57.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.343+0000 7fecd73aa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecd015bf50 con 0x7fecd015a410 2023-10-30T12:28:57.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecd5146640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fecd0069e90 0x7fecd015a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecd5146640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fecd0069e90 0x7fecd015a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33392/0 (socket says 172.21.15.125:33392) 2023-10-30T12:28:57.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecd5146640 1 -- 172.21.15.125:0/2464660645 learned_addr learned my addr 172.21.15.125:0/2464660645 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:57.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecd4945640 1 --2- 172.21.15.125:0/2464660645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fecd006cb80 0x7fecd0152110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecd5947640 1 --2- 172.21.15.125:0/2464660645 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fecd015a410 0x7fecd01f4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecd5146640 1 -- 172.21.15.125:0/2464660645 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fecd015a410 msgr2=0x7fecd01f4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecd5146640 1 --2- 172.21.15.125:0/2464660645 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fecd015a410 0x7fecd01f4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecd5146640 1 -- 172.21.15.125:0/2464660645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fecd006cb80 msgr2=0x7fecd0152110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecd5146640 1 --2- 172.21.15.125:0/2464660645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fecd006cb80 0x7fecd0152110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecd5146640 1 -- 172.21.15.125:0/2464660645 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc00094d0 con 0x7fecd0069e90 2023-10-30T12:28:57.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecd5146640 1 --2- 172.21.15.125:0/2464660645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fecd0069e90 0x7fecd015a9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feccc00c4c0 tx=0x7feccc00c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:57.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecc5ffb640 1 -- 172.21.15.125:0/2464660645 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feccc007480 con 0x7fecd0069e90 2023-10-30T12:28:57.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecc5ffb640 1 -- 172.21.15.125:0/2464660645 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feccc0075e0 con 0x7fecd0069e90 2023-10-30T12:28:57.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecc5ffb640 1 -- 172.21.15.125:0/2464660645 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feccc004a10 con 0x7fecd0069e90 2023-10-30T12:28:57.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecd73aa640 1 -- 172.21.15.125:0/2464660645 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd00cb750 con 0x7fecd0069e90 2023-10-30T12:28:57.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecd73aa640 1 -- 172.21.15.125:0/2464660645 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd0159400 con 0x7fecd0069e90 2023-10-30T12:28:57.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecd73aa640 1 -- 172.21.15.125:0/2464660645 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecd00cb390 con 0x7fecd0069e90 2023-10-30T12:28:57.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.347+0000 7fecc5ffb640 1 -- 172.21.15.125:0/2464660645 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feccc004b70 con 0x7fecd0069e90 2023-10-30T12:28:57.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.351+0000 7fecc5ffb640 1 --2- 172.21.15.125:0/2464660645 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fecb40664a0 0x7fecb4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.351+0000 7fecc5ffb640 1 -- 172.21.15.125:0/2464660645 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feccc082fd0 con 0x7fecd0069e90 2023-10-30T12:28:57.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.351+0000 7fecd4945640 1 --2- 172.21.15.125:0/2464660645 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fecb40664a0 0x7fecb4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.351+0000 7fecd4945640 1 --2- 172.21.15.125:0/2464660645 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fecb40664a0 0x7fecb4068960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fecc00ad000 tx=0x7fecc0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:57.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.351+0000 7fecc5ffb640 1 -- 172.21.15.125:0/2464660645 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feccc0504e0 con 0x7fecd0069e90 2023-10-30T12:28:57.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.499+0000 7fecd73aa640 1 -- 172.21.15.125:0/2464660645 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fecd00c7790 con 0x7fecb40664a0 2023-10-30T12:28:57.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.519+0000 7fecc5ffb640 1 -- 172.21.15.125:0/2464660645 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264584 (secure 0 0 0) 0x7fecc00ab6a0 con 0x7fecb40664a0 2023-10-30T12:28:57.523 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:57.525 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T12:28:57.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.523+0000 7fecd73aa640 1 -- 172.21.15.125:0/2464660645 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fecb40664a0 msgr2=0x7fecb4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.523+0000 7fecd73aa640 1 --2- 172.21.15.125:0/2464660645 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fecb40664a0 0x7fecb4068960 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fecc00ad000 tx=0x7fecc0008040 comp rx=0 tx=0).stop 2023-10-30T12:28:57.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.523+0000 7fecd73aa640 1 -- 172.21.15.125:0/2464660645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fecd0069e90 msgr2=0x7fecd015a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.523+0000 7fecd73aa640 1 --2- 172.21.15.125:0/2464660645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fecd0069e90 0x7fecd015a9c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7feccc00c4c0 tx=0x7feccc00c980 comp rx=0 tx=0).stop 2023-10-30T12:28:57.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.523+0000 7fecd73aa640 1 -- 172.21.15.125:0/2464660645 shutdown_connections 2023-10-30T12:28:57.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.523+0000 7fecd73aa640 1 --2- 172.21.15.125:0/2464660645 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fecb40664a0 0x7fecb4068960 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.523+0000 7fecd73aa640 1 --2- 172.21.15.125:0/2464660645 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fecd015a410 0x7fecd01f4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.523+0000 7fecd73aa640 1 --2- 172.21.15.125:0/2464660645 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fecd006cb80 0x7fecd0152110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.523+0000 7fecd73aa640 1 --2- 172.21.15.125:0/2464660645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fecd0069e90 0x7fecd015a9c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.523+0000 7fecd73aa640 1 -- 172.21.15.125:0/2464660645 >> 172.21.15.125:0/2464660645 conn(0x7fecd006dfd0 msgr2=0x7fecd0153110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:57.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.523+0000 7fecd73aa640 1 -- 172.21.15.125:0/2464660645 shutdown_connections 2023-10-30T12:28:57.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.523+0000 7fecd73aa640 1 -- 172.21.15.125:0/2464660645 wait complete. 2023-10-30T12:28:57.544 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-30T12:28:56.963919+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":593982,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":209,"num_write_kb":2433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":163,"ondisk_log_size":163,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":223224,"kb_used_data":2932,"kb_used_omap":12,"kb_used_meta":220211,"kb_avail":754751496,"statfs":{"total":773094113280,"available":772865531904,"internally_reserved":0,"allocated":3002368,"data_stored":1793704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":225496633},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3614,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.023323"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104854+0000","last_change":"2023-10-30T12:28:44.957488+0000","last_active":"2023-10-30T12:28:46.104854+0000","last_peered":"2023-10-30T12:28:46.104854+0000","last_clean":"2023-10-30T12:28:46.104854+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T12:28:46.104854+0000","last_undegraded":"2023-10-30T12:28:46.104854+0000","last_fullsized":"2023-10-30T12:28:46.104854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:30.240483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106772+0000","last_change":"2023-10-30T12:28:44.978898+0000","last_active":"2023-10-30T12:28:46.106772+0000","last_peered":"2023-10-30T12:28:46.106772+0000","last_clean":"2023-10-30T12:28:46.106772+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T12:28:46.106772+0000","last_undegraded":"2023-10-30T12:28:46.106772+0000","last_fullsized":"2023-10-30T12:28:46.106772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:57:21.463798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106769+0000","last_change":"2023-10-30T12:28:44.980722+0000","last_active":"2023-10-30T12:28:46.106769+0000","last_peered":"2023-10-30T12:28:46.106769+0000","last_clean":"2023-10-30T12:28:46.106769+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T12:28:46.106769+0000","last_undegraded":"2023-10-30T12:28:46.106769+0000","last_fullsized":"2023-10-30T12:28:46.106769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:25.009946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103812+0000","last_change":"2023-10-30T12:28:44.965061+0000","last_active":"2023-10-30T12:28:46.103812+0000","last_peered":"2023-10-30T12:28:46.103812+0000","last_clean":"2023-10-30T12:28:46.103812+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T12:28:46.103812+0000","last_undegraded":"2023-10-30T12:28:46.103812+0000","last_fullsized":"2023-10-30T12:28:46.103812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:23:43.749064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104285+0000","last_change":"2023-10-30T12:28:44.976985+0000","last_active":"2023-10-30T12:28:46.104285+0000","last_peered":"2023-10-30T12:28:46.104285+0000","last_clean":"2023-10-30T12:28:46.104285+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T12:28:46.104285+0000","last_undegraded":"2023-10-30T12:28:46.104285+0000","last_fullsized":"2023-10-30T12:28:46.104285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:08.758140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103030+0000","last_change":"2023-10-30T12:28:44.965453+0000","last_active":"2023-10-30T12:28:46.103030+0000","last_peered":"2023-10-30T12:28:46.103030+0000","last_clean":"2023-10-30T12:28:46.103030+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T12:28:46.103030+0000","last_undegraded":"2023-10-30T12:28:46.103030+0000","last_fullsized":"2023-10-30T12:28:46.103030+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:19.975226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103434+0000","last_change":"2023-10-30T12:28:44.968493+0000","last_active":"2023-10-30T12:28:46.103434+0000","last_peered":"2023-10-30T12:28:46.103434+0000","last_clean":"2023-10-30T12:28:46.103434+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T12:28:46.103434+0000","last_undegraded":"2023-10-30T12:28:46.103434+0000","last_fullsized":"2023-10-30T12:28:46.103434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:58.895635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106402+0000","last_change":"2023-10-30T12:28:44.972695+0000","last_active":"2023-10-30T12:28:46.106402+0000","last_peered":"2023-10-30T12:28:46.106402+0000","last_clean":"2023-10-30T12:28:46.106402+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T12:28:46.106402+0000","last_undegraded":"2023-10-30T12:28:46.106402+0000","last_fullsized":"2023-10-30T12:28:46.106402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:45.353683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939624+0000","last_change":"2023-10-30T12:28:44.946156+0000","last_active":"2023-10-30T12:28:45.939624+0000","last_peered":"2023-10-30T12:28:45.939624+0000","last_clean":"2023-10-30T12:28:45.939624+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T12:28:45.939624+0000","last_undegraded":"2023-10-30T12:28:45.939624+0000","last_fullsized":"2023-10-30T12:28:45.939624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:42:36.408944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104638+0000","last_change":"2023-10-30T12:28:44.957225+0000","last_active":"2023-10-30T12:28:46.104638+0000","last_peered":"2023-10-30T12:28:46.104638+0000","last_clean":"2023-10-30T12:28:46.104638+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T12:28:46.104638+0000","last_undegraded":"2023-10-30T12:28:46.104638+0000","last_fullsized":"2023-10-30T12:28:46.104638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:55:20.194553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104526+0000","last_change":"2023-10-30T12:28:44.956025+0000","last_active":"2023-10-30T12:28:46.104526+0000","last_peered":"2023-10-30T12:28:46.104526+0000","last_clean":"2023-10-30T12:28:46.104526+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T12:28:46.104526+0000","last_undegraded":"2023-10-30T12:28:46.104526+0000","last_fullsized":"2023-10-30T12:28:46.104526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:12.113264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104464+0000","last_change":"2023-10-30T12:28:44.956832+0000","last_active":"2023-10-30T12:28:46.104464+0000","last_peered":"2023-10-30T12:28:46.104464+0000","last_clean":"2023-10-30T12:28:46.104464+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T12:28:46.104464+0000","last_undegraded":"2023-10-30T12:28:46.104464+0000","last_fullsized":"2023-10-30T12:28:46.104464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:36.932024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945824+0000","last_change":"2023-10-30T12:28:44.962961+0000","last_active":"2023-10-30T12:28:45.945824+0000","last_peered":"2023-10-30T12:28:45.945824+0000","last_clean":"2023-10-30T12:28:45.945824+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T12:28:45.945824+0000","last_undegraded":"2023-10-30T12:28:45.945824+0000","last_fullsized":"2023-10-30T12:28:45.945824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:13.200996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945413+0000","last_change":"2023-10-30T12:28:44.968005+0000","last_active":"2023-10-30T12:28:45.945413+0000","last_peered":"2023-10-30T12:28:45.945413+0000","last_clean":"2023-10-30T12:28:45.945413+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T12:28:45.945413+0000","last_undegraded":"2023-10-30T12:28:45.945413+0000","last_fullsized":"2023-10-30T12:28:45.945413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:53:25.168553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106306+0000","last_change":"2023-10-30T12:28:44.977064+0000","last_active":"2023-10-30T12:28:46.106306+0000","last_peered":"2023-10-30T12:28:46.106306+0000","last_clean":"2023-10-30T12:28:46.106306+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T12:28:46.106306+0000","last_undegraded":"2023-10-30T12:28:46.106306+0000","last_fullsized":"2023-10-30T12:28:46.106306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:27:44.043189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102782+0000","last_change":"2023-10-30T12:28:44.950169+0000","last_active":"2023-10-30T12:28:46.102782+0000","last_peered":"2023-10-30T12:28:46.102782+0000","last_clean":"2023-10-30T12:28:46.102782+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T12:28:46.102782+0000","last_undegraded":"2023-10-30T12:28:46.102782+0000","last_fullsized":"2023-10-30T12:28:46.102782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:35.297525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103374+0000","last_change":"2023-10-30T12:28:44.960467+0000","last_active":"2023-10-30T12:28:46.103374+0000","last_peered":"2023-10-30T12:28:46.103374+0000","last_clean":"2023-10-30T12:28:46.103374+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T12:28:46.103374+0000","last_undegraded":"2023-10-30T12:28:46.103374+0000","last_fullsized":"2023-10-30T12:28:46.103374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:33.495256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.108582+0000","last_change":"2023-10-30T12:28:44.968948+0000","last_active":"2023-10-30T12:28:46.108582+0000","last_peered":"2023-10-30T12:28:46.108582+0000","last_clean":"2023-10-30T12:28:46.108582+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T12:28:46.108582+0000","last_undegraded":"2023-10-30T12:28:46.108582+0000","last_fullsized":"2023-10-30T12:28:46.108582+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:26:14.661648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103539+0000","last_change":"2023-10-30T12:28:44.966743+0000","last_active":"2023-10-30T12:28:46.103539+0000","last_peered":"2023-10-30T12:28:46.103539+0000","last_clean":"2023-10-30T12:28:46.103539+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T12:28:46.103539+0000","last_undegraded":"2023-10-30T12:28:46.103539+0000","last_fullsized":"2023-10-30T12:28:46.103539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:30.077107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104499+0000","last_change":"2023-10-30T12:28:44.955323+0000","last_active":"2023-10-30T12:28:46.104499+0000","last_peered":"2023-10-30T12:28:46.104499+0000","last_clean":"2023-10-30T12:28:46.104499+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T12:28:46.104499+0000","last_undegraded":"2023-10-30T12:28:46.104499+0000","last_fullsized":"2023-10-30T12:28:46.104499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:59:22.424571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945425+0000","last_change":"2023-10-30T12:28:44.946048+0000","last_active":"2023-10-30T12:28:45.945425+0000","last_peered":"2023-10-30T12:28:45.945425+0000","last_clean":"2023-10-30T12:28:45.945425+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T12:28:45.945425+0000","last_undegraded":"2023-10-30T12:28:45.945425+0000","last_fullsized":"2023-10-30T12:28:45.945425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:20:21.145943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939444+0000","last_change":"2023-10-30T12:28:44.954167+0000","last_active":"2023-10-30T12:28:45.939444+0000","last_peered":"2023-10-30T12:28:45.939444+0000","last_clean":"2023-10-30T12:28:45.939444+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T12:28:45.939444+0000","last_undegraded":"2023-10-30T12:28:45.939444+0000","last_fullsized":"2023-10-30T12:28:45.939444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:08.457457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106494+0000","last_change":"2023-10-30T12:28:44.982724+0000","last_active":"2023-10-30T12:28:46.106494+0000","last_peered":"2023-10-30T12:28:46.106494+0000","last_clean":"2023-10-30T12:28:46.106494+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T12:28:46.106494+0000","last_undegraded":"2023-10-30T12:28:46.106494+0000","last_fullsized":"2023-10-30T12:28:46.106494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:13.873943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103197+0000","last_change":"2023-10-30T12:28:44.981034+0000","last_active":"2023-10-30T12:28:46.103197+0000","last_peered":"2023-10-30T12:28:46.103197+0000","last_clean":"2023-10-30T12:28:46.103197+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T12:28:46.103197+0000","last_undegraded":"2023-10-30T12:28:46.103197+0000","last_fullsized":"2023-10-30T12:28:46.103197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:39:25.263228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107211+0000","last_change":"2023-10-30T12:28:44.975010+0000","last_active":"2023-10-30T12:28:46.107211+0000","last_peered":"2023-10-30T12:28:46.107211+0000","last_clean":"2023-10-30T12:28:46.107211+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T12:28:46.107211+0000","last_undegraded":"2023-10-30T12:28:46.107211+0000","last_fullsized":"2023-10-30T12:28:46.107211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:13.768876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104243+0000","last_change":"2023-10-30T12:28:44.951099+0000","last_active":"2023-10-30T12:28:46.104243+0000","last_peered":"2023-10-30T12:28:46.104243+0000","last_clean":"2023-10-30T12:28:46.104243+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T12:28:46.104243+0000","last_undegraded":"2023-10-30T12:28:46.104243+0000","last_fullsized":"2023-10-30T12:28:46.104243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:21:27.902671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946065+0000","last_change":"2023-10-30T12:28:44.959915+0000","last_active":"2023-10-30T12:28:45.946065+0000","last_peered":"2023-10-30T12:28:45.946065+0000","last_clean":"2023-10-30T12:28:45.946065+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T12:28:45.946065+0000","last_undegraded":"2023-10-30T12:28:45.946065+0000","last_fullsized":"2023-10-30T12:28:45.946065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:57:49.711309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946417+0000","last_change":"2023-10-30T12:28:44.972677+0000","last_active":"2023-10-30T12:28:45.946417+0000","last_peered":"2023-10-30T12:28:45.946417+0000","last_clean":"2023-10-30T12:28:45.946417+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T12:28:45.946417+0000","last_undegraded":"2023-10-30T12:28:45.946417+0000","last_fullsized":"2023-10-30T12:28:45.946417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:12.072355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106685+0000","last_change":"2023-10-30T12:28:44.975973+0000","last_active":"2023-10-30T12:28:46.106685+0000","last_peered":"2023-10-30T12:28:46.106685+0000","last_clean":"2023-10-30T12:28:46.106685+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T12:28:46.106685+0000","last_undegraded":"2023-10-30T12:28:46.106685+0000","last_fullsized":"2023-10-30T12:28:46.106685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:48.152823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103531+0000","last_change":"2023-10-30T12:28:44.980814+0000","last_active":"2023-10-30T12:28:46.103531+0000","last_peered":"2023-10-30T12:28:46.103531+0000","last_clean":"2023-10-30T12:28:46.103531+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T12:28:46.103531+0000","last_undegraded":"2023-10-30T12:28:46.103531+0000","last_fullsized":"2023-10-30T12:28:46.103531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:51:22.453883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107798+0000","last_change":"2023-10-30T12:28:44.965026+0000","last_active":"2023-10-30T12:28:46.107798+0000","last_peered":"2023-10-30T12:28:46.107798+0000","last_clean":"2023-10-30T12:28:46.107798+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T12:28:46.107798+0000","last_undegraded":"2023-10-30T12:28:46.107798+0000","last_fullsized":"2023-10-30T12:28:46.107798+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:04:08.680744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102782+0000","last_change":"2023-10-30T12:28:44.973401+0000","last_active":"2023-10-30T12:28:46.102782+0000","last_peered":"2023-10-30T12:28:46.102782+0000","last_clean":"2023-10-30T12:28:46.102782+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T12:28:46.102782+0000","last_undegraded":"2023-10-30T12:28:46.102782+0000","last_fullsized":"2023-10-30T12:28:46.102782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:34.198424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939578+0000","last_change":"2023-10-30T12:28:44.971817+0000","last_active":"2023-10-30T12:28:45.939578+0000","last_peered":"2023-10-30T12:28:45.939578+0000","last_clean":"2023-10-30T12:28:45.939578+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T12:28:45.939578+0000","last_undegraded":"2023-10-30T12:28:45.939578+0000","last_fullsized":"2023-10-30T12:28:45.939578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:45:35.205305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.130160+0000","last_change":"2023-10-30T12:28:44.074842+0000","last_active":"2023-10-30T12:28:55.130160+0000","last_peered":"2023-10-30T12:28:55.130160+0000","last_clean":"2023-10-30T12:28:55.130160+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T12:28:55.130160+0000","last_undegraded":"2023-10-30T12:28:55.130160+0000","last_fullsized":"2023-10-30T12:28:55.130160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:01.428785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945765+0000","last_change":"2023-10-30T12:28:44.958560+0000","last_active":"2023-10-30T12:28:45.945765+0000","last_peered":"2023-10-30T12:28:45.945765+0000","last_clean":"2023-10-30T12:28:45.945765+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T12:28:45.945765+0000","last_undegraded":"2023-10-30T12:28:45.945765+0000","last_fullsized":"2023-10-30T12:28:45.945765+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:45.564493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:47.984730+0000","last_change":"2023-10-30T12:28:44.060621+0000","last_active":"2023-10-30T12:28:47.984730+0000","last_peered":"2023-10-30T12:28:47.984730+0000","last_clean":"2023-10-30T12:28:47.984730+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T12:28:47.984730+0000","last_undegraded":"2023-10-30T12:28:47.984730+0000","last_fullsized":"2023-10-30T12:28:47.984730+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:53.741238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945157+0000","last_change":"2023-10-30T12:28:44.060171+0000","last_active":"2023-10-30T12:28:45.945157+0000","last_peered":"2023-10-30T12:28:45.945157+0000","last_clean":"2023-10-30T12:28:45.945157+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T12:28:45.945157+0000","last_undegraded":"2023-10-30T12:28:45.945157+0000","last_fullsized":"2023-10-30T12:28:45.945157+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:18.181418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106893+0000","last_change":"2023-10-30T12:28:44.978631+0000","last_active":"2023-10-30T12:28:46.106893+0000","last_peered":"2023-10-30T12:28:46.106893+0000","last_clean":"2023-10-30T12:28:46.106893+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T12:28:46.106893+0000","last_undegraded":"2023-10-30T12:28:46.106893+0000","last_fullsized":"2023-10-30T12:28:46.106893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:37.006427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103283+0000","last_change":"2023-10-30T12:28:44.058798+0000","last_active":"2023-10-30T12:28:46.103283+0000","last_peered":"2023-10-30T12:28:46.103283+0000","last_clean":"2023-10-30T12:28:46.103283+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T12:28:46.103283+0000","last_undegraded":"2023-10-30T12:28:46.103283+0000","last_fullsized":"2023-10-30T12:28:46.103283+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:08.643080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945025+0000","last_change":"2023-10-30T12:28:44.949118+0000","last_active":"2023-10-30T12:28:45.945025+0000","last_peered":"2023-10-30T12:28:45.945025+0000","last_clean":"2023-10-30T12:28:45.945025+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T12:28:45.945025+0000","last_undegraded":"2023-10-30T12:28:45.945025+0000","last_fullsized":"2023-10-30T12:28:45.945025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:11.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939606+0000","last_change":"2023-10-30T12:28:44.063439+0000","last_active":"2023-10-30T12:28:45.939606+0000","last_peered":"2023-10-30T12:28:45.939606+0000","last_clean":"2023-10-30T12:28:45.939606+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T12:28:45.939606+0000","last_undegraded":"2023-10-30T12:28:45.939606+0000","last_fullsized":"2023-10-30T12:28:45.939606+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:08.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104375+0000","last_change":"2023-10-30T12:28:44.951319+0000","last_active":"2023-10-30T12:28:46.104375+0000","last_peered":"2023-10-30T12:28:46.104375+0000","last_clean":"2023-10-30T12:28:46.104375+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T12:28:46.104375+0000","last_undegraded":"2023-10-30T12:28:46.104375+0000","last_fullsized":"2023-10-30T12:28:46.104375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:20.108360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104752+0000","last_change":"2023-10-30T12:28:44.052641+0000","last_active":"2023-10-30T12:28:46.104752+0000","last_peered":"2023-10-30T12:28:46.104752+0000","last_clean":"2023-10-30T12:28:46.104752+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T12:28:46.104752+0000","last_undegraded":"2023-10-30T12:28:46.104752+0000","last_fullsized":"2023-10-30T12:28:46.104752+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107640+0000","last_change":"2023-10-30T12:28:44.981614+0000","last_active":"2023-10-30T12:28:46.107640+0000","last_peered":"2023-10-30T12:28:46.107640+0000","last_clean":"2023-10-30T12:28:46.107640+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T12:28:46.107640+0000","last_undegraded":"2023-10-30T12:28:46.107640+0000","last_fullsized":"2023-10-30T12:28:46.107640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:15:11.750243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.068083+0000","last_change":"2023-10-30T12:28:44.075783+0000","last_active":"2023-10-30T12:28:50.068083+0000","last_peered":"2023-10-30T12:28:50.068083+0000","last_clean":"2023-10-30T12:28:50.068083+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T12:28:50.068083+0000","last_undegraded":"2023-10-30T12:28:50.068083+0000","last_fullsized":"2023-10-30T12:28:50.068083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:01.094929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103142+0000","last_change":"2023-10-30T12:28:44.950899+0000","last_active":"2023-10-30T12:28:46.103142+0000","last_peered":"2023-10-30T12:28:46.103142+0000","last_clean":"2023-10-30T12:28:46.103142+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T12:28:46.103142+0000","last_undegraded":"2023-10-30T12:28:46.103142+0000","last_fullsized":"2023-10-30T12:28:46.103142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:34.005863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.065815+0000","last_change":"2023-10-30T12:28:44.063367+0000","last_active":"2023-10-30T12:28:50.065815+0000","last_peered":"2023-10-30T12:28:50.065815+0000","last_clean":"2023-10-30T12:28:50.065815+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T12:28:50.065815+0000","last_undegraded":"2023-10-30T12:28:50.065815+0000","last_fullsized":"2023-10-30T12:28:50.065815+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:45.585121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104587+0000","last_change":"2023-10-30T12:28:44.972056+0000","last_active":"2023-10-30T12:28:46.104587+0000","last_peered":"2023-10-30T12:28:46.104587+0000","last_clean":"2023-10-30T12:28:46.104587+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T12:28:46.104587+0000","last_undegraded":"2023-10-30T12:28:46.104587+0000","last_fullsized":"2023-10-30T12:28:46.104587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:06:08.345856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.132476+0000","last_change":"2023-10-30T12:28:44.065545+0000","last_active":"2023-10-30T12:28:55.132476+0000","last_peered":"2023-10-30T12:28:55.132476+0000","last_clean":"2023-10-30T12:28:55.132476+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T12:28:55.132476+0000","last_undegraded":"2023-10-30T12:28:55.132476+0000","last_fullsized":"2023-10-30T12:28:55.132476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:09.900189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945563+0000","last_change":"2023-10-30T12:28:44.961237+0000","last_active":"2023-10-30T12:28:45.945563+0000","last_peered":"2023-10-30T12:28:45.945563+0000","last_clean":"2023-10-30T12:28:45.945563+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T12:28:45.945563+0000","last_undegraded":"2023-10-30T12:28:45.945563+0000","last_fullsized":"2023-10-30T12:28:45.945563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:17.160724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103797+0000","last_change":"2023-10-30T12:28:44.061922+0000","last_active":"2023-10-30T12:28:46.103797+0000","last_peered":"2023-10-30T12:28:46.103797+0000","last_clean":"2023-10-30T12:28:46.103797+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T12:28:46.103797+0000","last_undegraded":"2023-10-30T12:28:46.103797+0000","last_fullsized":"2023-10-30T12:28:46.103797+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:34.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103258+0000","last_change":"2023-10-30T12:28:44.958245+0000","last_active":"2023-10-30T12:28:46.103258+0000","last_peered":"2023-10-30T12:28:46.103258+0000","last_clean":"2023-10-30T12:28:46.103258+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T12:28:46.103258+0000","last_undegraded":"2023-10-30T12:28:46.103258+0000","last_fullsized":"2023-10-30T12:28:46.103258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:42:10.840490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103980+0000","last_change":"2023-10-30T12:28:44.059784+0000","last_active":"2023-10-30T12:28:46.103980+0000","last_peered":"2023-10-30T12:28:46.103980+0000","last_clean":"2023-10-30T12:28:46.103980+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T12:28:46.103980+0000","last_undegraded":"2023-10-30T12:28:46.103980+0000","last_fullsized":"2023-10-30T12:28:46.103980+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:11.468221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939230+0000","last_change":"2023-10-30T12:28:44.980813+0000","last_active":"2023-10-30T12:28:45.939230+0000","last_peered":"2023-10-30T12:28:45.939230+0000","last_clean":"2023-10-30T12:28:45.939230+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T12:28:45.939230+0000","last_undegraded":"2023-10-30T12:28:45.939230+0000","last_fullsized":"2023-10-30T12:28:45.939230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:10.858970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.070290+0000","last_change":"2023-10-30T12:28:44.065885+0000","last_active":"2023-10-30T12:28:50.070290+0000","last_peered":"2023-10-30T12:28:50.070290+0000","last_clean":"2023-10-30T12:28:50.070290+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T12:28:50.070290+0000","last_undegraded":"2023-10-30T12:28:50.070290+0000","last_fullsized":"2023-10-30T12:28:50.070290+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:26.313595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103080+0000","last_change":"2023-10-30T12:28:44.952458+0000","last_active":"2023-10-30T12:28:46.103080+0000","last_peered":"2023-10-30T12:28:46.103080+0000","last_clean":"2023-10-30T12:28:46.103080+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T12:28:46.103080+0000","last_undegraded":"2023-10-30T12:28:46.103080+0000","last_fullsized":"2023-10-30T12:28:46.103080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:48:19.752492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:47.984275+0000","last_change":"2023-10-30T12:28:44.080011+0000","last_active":"2023-10-30T12:28:47.984275+0000","last_peered":"2023-10-30T12:28:47.984275+0000","last_clean":"2023-10-30T12:28:47.984275+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T12:28:47.984275+0000","last_undegraded":"2023-10-30T12:28:47.984275+0000","last_fullsized":"2023-10-30T12:28:47.984275+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104494+0000","last_change":"2023-10-30T12:28:44.961296+0000","last_active":"2023-10-30T12:28:46.104494+0000","last_peered":"2023-10-30T12:28:46.104494+0000","last_clean":"2023-10-30T12:28:46.104494+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T12:28:46.104494+0000","last_undegraded":"2023-10-30T12:28:46.104494+0000","last_fullsized":"2023-10-30T12:28:46.104494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:56.540392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.066004+0000","last_change":"2023-10-30T12:28:44.076352+0000","last_active":"2023-10-30T12:28:50.066004+0000","last_peered":"2023-10-30T12:28:50.066004+0000","last_clean":"2023-10-30T12:28:50.066004+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T12:28:50.066004+0000","last_undegraded":"2023-10-30T12:28:50.066004+0000","last_fullsized":"2023-10-30T12:28:50.066004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104918+0000","last_change":"2023-10-30T12:28:44.956918+0000","last_active":"2023-10-30T12:28:46.104918+0000","last_peered":"2023-10-30T12:28:46.104918+0000","last_clean":"2023-10-30T12:28:46.104918+0000","last_became_active":"2023-10-30T12:28:44.956397+0000","last_became_peered":"2023-10-30T12:28:44.956397+0000","last_unstale":"2023-10-30T12:28:46.104918+0000","last_undegraded":"2023-10-30T12:28:46.104918+0000","last_fullsized":"2023-10-30T12:28:46.104918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:26.252196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106813+0000","last_change":"2023-10-30T12:28:44.075684+0000","last_active":"2023-10-30T12:28:46.106813+0000","last_peered":"2023-10-30T12:28:46.106813+0000","last_clean":"2023-10-30T12:28:46.106813+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T12:28:46.106813+0000","last_undegraded":"2023-10-30T12:28:46.106813+0000","last_fullsized":"2023-10-30T12:28:46.106813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:07.375795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104588+0000","last_change":"2023-10-30T12:28:44.956047+0000","last_active":"2023-10-30T12:28:46.104588+0000","last_peered":"2023-10-30T12:28:46.104588+0000","last_clean":"2023-10-30T12:28:46.104588+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T12:28:46.104588+0000","last_undegraded":"2023-10-30T12:28:46.104588+0000","last_fullsized":"2023-10-30T12:28:46.104588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:49.924947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946632+0000","last_change":"2023-10-30T12:28:44.060686+0000","last_active":"2023-10-30T12:28:45.946632+0000","last_peered":"2023-10-30T12:28:45.946632+0000","last_clean":"2023-10-30T12:28:45.946632+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T12:28:45.946632+0000","last_undegraded":"2023-10-30T12:28:45.946632+0000","last_fullsized":"2023-10-30T12:28:45.946632+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:23.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103562+0000","last_change":"2023-10-30T12:28:44.946758+0000","last_active":"2023-10-30T12:28:46.103562+0000","last_peered":"2023-10-30T12:28:46.103562+0000","last_clean":"2023-10-30T12:28:46.103562+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T12:28:46.103562+0000","last_undegraded":"2023-10-30T12:28:46.103562+0000","last_fullsized":"2023-10-30T12:28:46.103562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:20.520148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.932509+0000","last_change":"2023-10-30T12:28:33.433824+0000","last_active":"2023-10-30T12:28:44.932509+0000","last_peered":"2023-10-30T12:28:44.932509+0000","last_clean":"2023-10-30T12:28:44.932509+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T12:28:44.932509+0000","last_undegraded":"2023-10-30T12:28:44.932509+0000","last_fullsized":"2023-10-30T12:28:44.932509+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_clean_scrub_stamp":"2023-10-30T12:28:31.408255+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:17.624179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103427+0000","last_change":"2023-10-30T12:28:44.059051+0000","last_active":"2023-10-30T12:28:46.103427+0000","last_peered":"2023-10-30T12:28:46.103427+0000","last_clean":"2023-10-30T12:28:46.103427+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T12:28:46.103427+0000","last_undegraded":"2023-10-30T12:28:46.103427+0000","last_fullsized":"2023-10-30T12:28:46.103427+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:54.810636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945062+0000","last_change":"2023-10-30T12:28:44.965183+0000","last_active":"2023-10-30T12:28:45.945062+0000","last_peered":"2023-10-30T12:28:45.945062+0000","last_clean":"2023-10-30T12:28:45.945062+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T12:28:45.945062+0000","last_undegraded":"2023-10-30T12:28:45.945062+0000","last_fullsized":"2023-10-30T12:28:45.945062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:57.655687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103361+0000","last_change":"2023-10-30T12:28:44.080378+0000","last_active":"2023-10-30T12:28:46.103361+0000","last_peered":"2023-10-30T12:28:46.103361+0000","last_clean":"2023-10-30T12:28:46.103361+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T12:28:46.103361+0000","last_undegraded":"2023-10-30T12:28:46.103361+0000","last_fullsized":"2023-10-30T12:28:46.103361+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:22.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944927+0000","last_change":"2023-10-30T12:28:44.961659+0000","last_active":"2023-10-30T12:28:45.944927+0000","last_peered":"2023-10-30T12:28:45.944927+0000","last_clean":"2023-10-30T12:28:45.944927+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T12:28:45.944927+0000","last_undegraded":"2023-10-30T12:28:45.944927+0000","last_fullsized":"2023-10-30T12:28:45.944927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:34:00.000084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107831+0000","last_change":"2023-10-30T12:28:44.082480+0000","last_active":"2023-10-30T12:28:46.107831+0000","last_peered":"2023-10-30T12:28:46.107831+0000","last_clean":"2023-10-30T12:28:46.107831+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T12:28:46.107831+0000","last_undegraded":"2023-10-30T12:28:46.107831+0000","last_fullsized":"2023-10-30T12:28:46.107831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:20.126782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106385+0000","last_change":"2023-10-30T12:28:44.973906+0000","last_active":"2023-10-30T12:28:46.106385+0000","last_peered":"2023-10-30T12:28:46.106385+0000","last_clean":"2023-10-30T12:28:46.106385+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T12:28:46.106385+0000","last_undegraded":"2023-10-30T12:28:46.106385+0000","last_fullsized":"2023-10-30T12:28:46.106385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:31.845894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103873+0000","last_change":"2023-10-30T12:28:44.059806+0000","last_active":"2023-10-30T12:28:46.103873+0000","last_peered":"2023-10-30T12:28:46.103873+0000","last_clean":"2023-10-30T12:28:46.103873+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T12:28:46.103873+0000","last_undegraded":"2023-10-30T12:28:46.103873+0000","last_fullsized":"2023-10-30T12:28:46.103873+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:50.939946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104302+0000","last_change":"2023-10-30T12:28:44.955423+0000","last_active":"2023-10-30T12:28:46.104302+0000","last_peered":"2023-10-30T12:28:46.104302+0000","last_clean":"2023-10-30T12:28:46.104302+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T12:28:46.104302+0000","last_undegraded":"2023-10-30T12:28:46.104302+0000","last_fullsized":"2023-10-30T12:28:46.104302+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:30.054048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.982937+0000","last_change":"2023-10-30T12:28:44.074970+0000","last_active":"2023-10-30T12:28:45.982937+0000","last_peered":"2023-10-30T12:28:45.982937+0000","last_clean":"2023-10-30T12:28:45.982937+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T12:28:45.982937+0000","last_undegraded":"2023-10-30T12:28:45.982937+0000","last_fullsized":"2023-10-30T12:28:45.982937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:33.562703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103885+0000","last_change":"2023-10-30T12:28:44.973482+0000","last_active":"2023-10-30T12:28:46.103885+0000","last_peered":"2023-10-30T12:28:46.103885+0000","last_clean":"2023-10-30T12:28:46.103885+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T12:28:46.103885+0000","last_undegraded":"2023-10-30T12:28:46.103885+0000","last_fullsized":"2023-10-30T12:28:46.103885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:25.393161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103579+0000","last_change":"2023-10-30T12:28:44.063256+0000","last_active":"2023-10-30T12:28:46.103579+0000","last_peered":"2023-10-30T12:28:46.103579+0000","last_clean":"2023-10-30T12:28:46.103579+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T12:28:46.103579+0000","last_undegraded":"2023-10-30T12:28:46.103579+0000","last_fullsized":"2023-10-30T12:28:46.103579+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:01:36.144966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946945+0000","last_change":"2023-10-30T12:28:44.975443+0000","last_active":"2023-10-30T12:28:45.946945+0000","last_peered":"2023-10-30T12:28:45.946945+0000","last_clean":"2023-10-30T12:28:45.946945+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T12:28:45.946945+0000","last_undegraded":"2023-10-30T12:28:45.946945+0000","last_fullsized":"2023-10-30T12:28:45.946945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:23.765723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946972+0000","last_change":"2023-10-30T12:28:44.055374+0000","last_active":"2023-10-30T12:28:45.946972+0000","last_peered":"2023-10-30T12:28:45.946972+0000","last_clean":"2023-10-30T12:28:45.946972+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T12:28:45.946972+0000","last_undegraded":"2023-10-30T12:28:45.946972+0000","last_fullsized":"2023-10-30T12:28:45.946972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946991+0000","last_change":"2023-10-30T12:28:44.974359+0000","last_active":"2023-10-30T12:28:45.946991+0000","last_peered":"2023-10-30T12:28:45.946991+0000","last_clean":"2023-10-30T12:28:45.946991+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T12:28:45.946991+0000","last_undegraded":"2023-10-30T12:28:45.946991+0000","last_fullsized":"2023-10-30T12:28:45.946991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:48.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.063134+0000","last_change":"2023-10-30T12:28:44.058663+0000","last_active":"2023-10-30T12:28:50.063134+0000","last_peered":"2023-10-30T12:28:50.063134+0000","last_clean":"2023-10-30T12:28:50.063134+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T12:28:50.063134+0000","last_undegraded":"2023-10-30T12:28:50.063134+0000","last_fullsized":"2023-10-30T12:28:50.063134+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:42:22.279653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946815+0000","last_change":"2023-10-30T12:28:44.957471+0000","last_active":"2023-10-30T12:28:45.946815+0000","last_peered":"2023-10-30T12:28:45.946815+0000","last_clean":"2023-10-30T12:28:45.946815+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T12:28:45.946815+0000","last_undegraded":"2023-10-30T12:28:45.946815+0000","last_fullsized":"2023-10-30T12:28:45.946815+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:27:38.669452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102847+0000","last_change":"2023-10-30T12:28:44.054338+0000","last_active":"2023-10-30T12:28:46.102847+0000","last_peered":"2023-10-30T12:28:46.102847+0000","last_clean":"2023-10-30T12:28:46.102847+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T12:28:46.102847+0000","last_undegraded":"2023-10-30T12:28:46.102847+0000","last_fullsized":"2023-10-30T12:28:46.102847+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:55:35.370652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103200+0000","last_change":"2023-10-30T12:28:44.965759+0000","last_active":"2023-10-30T12:28:46.103200+0000","last_peered":"2023-10-30T12:28:46.103200+0000","last_clean":"2023-10-30T12:28:46.103200+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T12:28:46.103200+0000","last_undegraded":"2023-10-30T12:28:46.103200+0000","last_fullsized":"2023-10-30T12:28:46.103200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:13:52.965500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944989+0000","last_change":"2023-10-30T12:28:44.064606+0000","last_active":"2023-10-30T12:28:45.944989+0000","last_peered":"2023-10-30T12:28:45.944989+0000","last_clean":"2023-10-30T12:28:45.944989+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T12:28:45.944989+0000","last_undegraded":"2023-10-30T12:28:45.944989+0000","last_fullsized":"2023-10-30T12:28:45.944989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944678+0000","last_change":"2023-10-30T12:28:44.958947+0000","last_active":"2023-10-30T12:28:45.944678+0000","last_peered":"2023-10-30T12:28:45.944678+0000","last_clean":"2023-10-30T12:28:45.944678+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T12:28:45.944678+0000","last_undegraded":"2023-10-30T12:28:45.944678+0000","last_fullsized":"2023-10-30T12:28:45.944678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:01:16.329994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945603+0000","last_change":"2023-10-30T12:28:44.072754+0000","last_active":"2023-10-30T12:28:45.945603+0000","last_peered":"2023-10-30T12:28:45.945603+0000","last_clean":"2023-10-30T12:28:45.945603+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T12:28:45.945603+0000","last_undegraded":"2023-10-30T12:28:45.945603+0000","last_fullsized":"2023-10-30T12:28:45.945603+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:51:55.865069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.108063+0000","last_change":"2023-10-30T12:28:44.978376+0000","last_active":"2023-10-30T12:28:46.108063+0000","last_peered":"2023-10-30T12:28:46.108063+0000","last_clean":"2023-10-30T12:28:46.108063+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T12:28:46.108063+0000","last_undegraded":"2023-10-30T12:28:46.108063+0000","last_fullsized":"2023-10-30T12:28:46.108063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:59.643447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.057363+0000","last_change":"2023-10-30T12:28:44.058801+0000","last_active":"2023-10-30T12:28:50.057363+0000","last_peered":"2023-10-30T12:28:50.057363+0000","last_clean":"2023-10-30T12:28:50.057363+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T12:28:50.057363+0000","last_undegraded":"2023-10-30T12:28:50.057363+0000","last_fullsized":"2023-10-30T12:28:50.057363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:11:24.706813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103647+0000","last_change":"2023-10-30T12:28:44.943292+0000","last_active":"2023-10-30T12:28:46.103647+0000","last_peered":"2023-10-30T12:28:46.103647+0000","last_clean":"2023-10-30T12:28:46.103647+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T12:28:46.103647+0000","last_undegraded":"2023-10-30T12:28:46.103647+0000","last_fullsized":"2023-10-30T12:28:46.103647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:37.538525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103472+0000","last_change":"2023-10-30T12:28:44.958607+0000","last_active":"2023-10-30T12:28:46.103472+0000","last_peered":"2023-10-30T12:28:46.103472+0000","last_clean":"2023-10-30T12:28:46.103472+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T12:28:46.103472+0000","last_undegraded":"2023-10-30T12:28:46.103472+0000","last_fullsized":"2023-10-30T12:28:46.103472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:28:55.169198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.059856+0000","last_change":"2023-10-30T12:28:44.054652+0000","last_active":"2023-10-30T12:28:50.059856+0000","last_peered":"2023-10-30T12:28:50.059856+0000","last_clean":"2023-10-30T12:28:50.059856+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T12:28:50.059856+0000","last_undegraded":"2023-10-30T12:28:50.059856+0000","last_fullsized":"2023-10-30T12:28:50.059856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:05.023374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939383+0000","last_change":"2023-10-30T12:28:44.958246+0000","last_active":"2023-10-30T12:28:45.939383+0000","last_peered":"2023-10-30T12:28:45.939383+0000","last_clean":"2023-10-30T12:28:45.939383+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T12:28:45.939383+0000","last_undegraded":"2023-10-30T12:28:45.939383+0000","last_fullsized":"2023-10-30T12:28:45.939383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:36.947398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.057546+0000","last_change":"2023-10-30T12:28:44.052633+0000","last_active":"2023-10-30T12:28:50.057546+0000","last_peered":"2023-10-30T12:28:50.057546+0000","last_clean":"2023-10-30T12:28:50.057546+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T12:28:50.057546+0000","last_undegraded":"2023-10-30T12:28:50.057546+0000","last_fullsized":"2023-10-30T12:28:50.057546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107324+0000","last_change":"2023-10-30T12:28:44.974457+0000","last_active":"2023-10-30T12:28:46.107324+0000","last_peered":"2023-10-30T12:28:46.107324+0000","last_clean":"2023-10-30T12:28:46.107324+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T12:28:46.107324+0000","last_undegraded":"2023-10-30T12:28:46.107324+0000","last_fullsized":"2023-10-30T12:28:46.107324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:25:56.791053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107384+0000","last_change":"2023-10-30T12:28:44.075493+0000","last_active":"2023-10-30T12:28:46.107384+0000","last_peered":"2023-10-30T12:28:46.107384+0000","last_clean":"2023-10-30T12:28:46.107384+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T12:28:46.107384+0000","last_undegraded":"2023-10-30T12:28:46.107384+0000","last_fullsized":"2023-10-30T12:28:46.107384+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:00:11.718669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107054+0000","last_change":"2023-10-30T12:28:44.974424+0000","last_active":"2023-10-30T12:28:46.107054+0000","last_peered":"2023-10-30T12:28:46.107054+0000","last_clean":"2023-10-30T12:28:46.107054+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T12:28:46.107054+0000","last_undegraded":"2023-10-30T12:28:46.107054+0000","last_fullsized":"2023-10-30T12:28:46.107054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:14.375671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.066270+0000","last_change":"2023-10-30T12:28:44.065843+0000","last_active":"2023-10-30T12:28:50.066270+0000","last_peered":"2023-10-30T12:28:50.066270+0000","last_clean":"2023-10-30T12:28:50.066270+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T12:28:50.066270+0000","last_undegraded":"2023-10-30T12:28:50.066270+0000","last_fullsized":"2023-10-30T12:28:50.066270+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:32.405199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":3614,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":50,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":6024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":46,"ondisk_log_size":46,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476745,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":76450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]}]},{"osd":2,"up_from":16,"seq":68719476745,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":77480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]}]},{"osd":1,"up_from":16,"seq":68719476743,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":815104,"data_stored":666306,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]}]},{"osd":0,"up_from":16,"seq":68719476744,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":225280,"data_stored":75924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82499999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509448,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":75868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81000000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2130000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.306}]}]},{"osd":4,"up_from":15,"seq":64424509449,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":78152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77900000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.056}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]}]},{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":229376,"data_stored":76574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2270000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64700000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6859999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5860000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3240000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":827392,"data_stored":666950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.617}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:28:57.549 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T12:28:57.549 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-30T12:28:57.549 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-30T12:28:57.550 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-30T12:28:57.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.723+0000 7f7fbb491640 1 Processor -- start 2023-10-30T12:28:57.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.723+0000 7f7fbb491640 1 -- start start 2023-10-30T12:28:57.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.723+0000 7f7fbb491640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7fb41570e0 0x7f7fb4159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.723+0000 7f7fbb491640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fb406d320 0x7f7fb406c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.723+0000 7f7fbb491640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fb4150c90 con 0x7f7fb41508b0 2023-10-30T12:28:57.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.723+0000 7f7fbb491640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fb40c8760 con 0x7f7fb41570e0 2023-10-30T12:28:57.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.723+0000 7f7fbb491640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb40c81a0 con 0x7f7fb406d320 2023-10-30T12:28:57.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.727+0000 7f7fb8a2c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7fb41570e0 0x7f7fb4159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.727+0000 7f7fb8a2c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7fb41570e0 0x7f7fb4159ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33400/0 (socket says 172.21.15.125:33400) 2023-10-30T12:28:57.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.727+0000 7f7fb8a2c640 1 -- 172.21.15.125:0/3983011359 learned_addr learned my addr 172.21.15.125:0/3983011359 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:57.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.727+0000 7f7fb9a2e640 1 --2- 172.21.15.125:0/3983011359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fb406d320 0x7f7fb406c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.727+0000 7f7fb8a2c640 1 -- 172.21.15.125:0/3983011359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fb406d320 msgr2=0x7f7fb406c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:57.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.727+0000 7f7fb8a2c640 1 --2- 172.21.15.125:0/3983011359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fb406d320 0x7f7fb406c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.727+0000 7f7fb8a2c640 1 -- 172.21.15.125:0/3983011359 >> v1:172.21.15.125:6789/0 conn(0x7f7fb41508b0 legacy=0x7f7fb40c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.727+0000 7f7fb8a2c640 1 -- 172.21.15.125:0/3983011359 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fb40c8480 con 0x7f7fb41570e0 2023-10-30T12:28:57.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.727+0000 7f7fb8a2c640 1 --2- 172.21.15.125:0/3983011359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fb41570e0 0x7f7fb4159ac0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4009870 tx=0x7f7fa401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=645a7e62d0d69f27 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:57.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.727+0000 7f7fabfff640 1 -- 172.21.15.125:0/3983011359 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa40a8a30 con 0x7f7fb41570e0 2023-10-30T12:28:57.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.727+0000 7f7fabfff640 1 -- 172.21.15.125:0/3983011359 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa40a8b90 con 0x7f7fb41570e0 2023-10-30T12:28:57.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.727+0000 7f7fabfff640 1 -- 172.21.15.125:0/3983011359 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa40b0440 con 0x7f7fb41570e0 2023-10-30T12:28:57.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 -- 172.21.15.125:0/3983011359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fb41570e0 msgr2=0x7f7fb4159ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 --2- 172.21.15.125:0/3983011359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fb41570e0 0x7f7fb4159ac0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7fa4009870 tx=0x7f7fa401ed30 comp rx=0 tx=0).stop 2023-10-30T12:28:57.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 -- 172.21.15.125:0/3983011359 shutdown_connections 2023-10-30T12:28:57.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 --2- 172.21.15.125:0/3983011359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fb41570e0 0x7f7fb4159ac0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 --2- 172.21.15.125:0/3983011359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fb406d320 0x7f7fb406c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 -- 172.21.15.125:0/3983011359 >> 172.21.15.125:0/3983011359 conn(0x7f7fb406dfd0 msgr2=0x7f7fb406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:57.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 -- 172.21.15.125:0/3983011359 shutdown_connections 2023-10-30T12:28:57.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 -- 172.21.15.125:0/3983011359 wait complete. 2023-10-30T12:28:57.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 Processor -- start 2023-10-30T12:28:57.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 -- start start 2023-10-30T12:28:57.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fb406d320 0x7f7fb4154300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fb41508b0 0x7f7fb40711d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fb41570e0 0x7f7fb41eef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb4156080 con 0x7f7fb41570e0 2023-10-30T12:28:57.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb4156360 con 0x7f7fb41508b0 2023-10-30T12:28:57.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fbb491640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fb41f2580 con 0x7f7fb406d320 2023-10-30T12:28:57.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fb9a2e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fb41570e0 0x7f7fb41eef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fb9a2e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fb41570e0 0x7f7fb41eef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33404/0 (socket says 172.21.15.125:33404) 2023-10-30T12:28:57.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fb9a2e640 1 -- 172.21.15.125:0/2006683681 learned_addr learned my addr 172.21.15.125:0/2006683681 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:57.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fb922d640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fb406d320 0x7f7fb4154300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fb8a2c640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fb41508b0 0x7f7fb40711d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fb9a2e640 1 -- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fb406d320 msgr2=0x7f7fb4154300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fb9a2e640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fb406d320 0x7f7fb4154300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fb9a2e640 1 -- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fb41508b0 msgr2=0x7f7fb40711d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fb9a2e640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fb41508b0 0x7f7fb40711d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fb9a2e640 1 -- 172.21.15.125:0/2006683681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa40094d0 con 0x7f7fb41570e0 2023-10-30T12:28:57.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fb922d640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fb406d320 0x7f7fb4154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:57.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.731+0000 7f7fb8a2c640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fb41508b0 0x7f7fb40711d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:57.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.735+0000 7f7fb9a2e640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fb41570e0 0x7f7fb41eef70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7fac007e40 tx=0x7f7fac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:57.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.735+0000 7f7fa9ffb640 1 -- 172.21.15.125:0/2006683681 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac014620 con 0x7f7fb41570e0 2023-10-30T12:28:57.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.735+0000 7f7fa9ffb640 1 -- 172.21.15.125:0/2006683681 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fac00ee90 con 0x7f7fb41570e0 2023-10-30T12:28:57.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.735+0000 7f7fbb491640 1 -- 172.21.15.125:0/2006683681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fb41f2860 con 0x7f7fb41570e0 2023-10-30T12:28:57.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.735+0000 7f7fa9ffb640 1 -- 172.21.15.125:0/2006683681 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fac0149e0 con 0x7f7fb41570e0 2023-10-30T12:28:57.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.735+0000 7f7fbb491640 1 -- 172.21.15.125:0/2006683681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fb41f2e20 con 0x7f7fb41570e0 2023-10-30T12:28:57.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.735+0000 7f7fbb491640 1 -- 172.21.15.125:0/2006683681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fb41f3010 con 0x7f7fb41570e0 2023-10-30T12:28:57.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.735+0000 7f7fa9ffb640 1 -- 172.21.15.125:0/2006683681 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7fac014b40 con 0x7f7fb41570e0 2023-10-30T12:28:57.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.735+0000 7f7fa9ffb640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f980664a0 0x7f7f98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:57.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.735+0000 7f7fa9ffb640 1 -- 172.21.15.125:0/2006683681 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7fac084930 con 0x7f7fb41570e0 2023-10-30T12:28:57.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.735+0000 7f7fb922d640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f980664a0 0x7f7f98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:57.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.739+0000 7f7fb922d640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f980664a0 0x7f7f98068960 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0000f70 tx=0x7f7fb0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:57.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.739+0000 7f7fa9ffb640 1 -- 172.21.15.125:0/2006683681 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7fac051d10 con 0x7f7fb41570e0 2023-10-30T12:28:57.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.923+0000 7f7fbb491640 1 -- 172.21.15.125:0/2006683681 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7fb4153eb0 con 0x7f7fb41570e0 2023-10-30T12:28:57.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.923+0000 7f7fa9ffb640 1 -- 172.21.15.125:0/2006683681 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f7fac055330 con 0x7f7fb41570e0 2023-10-30T12:28:57.926 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:57.926 INFO:teuthology.orchestra.run.smithi125.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-30T12:28:57.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.923+0000 7f7fbb491640 1 -- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f980664a0 msgr2=0x7f7f98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.923+0000 7f7fbb491640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f980664a0 0x7f7f98068960 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7fb0000f70 tx=0x7f7fb0009040 comp rx=0 tx=0).stop 2023-10-30T12:28:57.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.923+0000 7f7fbb491640 1 -- 172.21.15.125:0/2006683681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fb41570e0 msgr2=0x7f7fb41eef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:57.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.923+0000 7f7fbb491640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fb41570e0 0x7f7fb41eef70 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7fac007e40 tx=0x7f7fac00e4b0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.923+0000 7f7fbb491640 1 -- 172.21.15.125:0/2006683681 shutdown_connections 2023-10-30T12:28:57.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.923+0000 7f7fbb491640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f980664a0 0x7f7f98068960 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.923+0000 7f7fbb491640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fb41570e0 0x7f7fb41eef70 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.923+0000 7f7fbb491640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fb41508b0 0x7f7fb40711d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.923+0000 7f7fbb491640 1 --2- 172.21.15.125:0/2006683681 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fb406d320 0x7f7fb4154300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:57.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.923+0000 7f7fbb491640 1 -- 172.21.15.125:0/2006683681 >> 172.21.15.125:0/2006683681 conn(0x7f7fb406dfd0 msgr2=0x7f7fb40c7750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:57.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.923+0000 7f7fbb491640 1 -- 172.21.15.125:0/2006683681 shutdown_connections 2023-10-30T12:28:57.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:57.927+0000 7f7fbb491640 1 -- 172.21.15.125:0/2006683681 wait complete. 2023-10-30T12:28:57.947 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-30T12:28:57.947 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:58.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.083+0000 7fa1f546b640 1 Processor -- start 2023-10-30T12:28:58.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.083+0000 7fa1f546b640 1 -- start start 2023-10-30T12:28:58.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.083+0000 7fa1f546b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa1f01538a0 0x7fa1f015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.083+0000 7fa1f546b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa1f00cf160 0x7fa1f00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.083+0000 7fa1f546b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1f00cb800 con 0x7fa1f00ca0e0 2023-10-30T12:28:58.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.083+0000 7fa1f546b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f015dc30 con 0x7fa1f00cf160 2023-10-30T12:28:58.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.083+0000 7fa1f546b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1f00cb330 con 0x7fa1f01538a0 2023-10-30T12:28:58.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.083+0000 7fa1ef7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa1f00cf160 0x7fa1f00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.083+0000 7fa1ef7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa1f00cf160 0x7fa1f00cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33408/0 (socket says 172.21.15.125:33408) 2023-10-30T12:28:58.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.083+0000 7fa1ef7fe640 1 -- 172.21.15.125:0/244542591 learned_addr learned my addr 172.21.15.125:0/244542591 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:58.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.083+0000 7fa1e7fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa1f00ca0e0 0x7fa1f015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54200/0 (socket says 172.21.15.125:54200) 2023-10-30T12:28:58.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.087+0000 7fa1eeffd640 1 --2- 172.21.15.125:0/244542591 >> v2:172.21.15.136:3300/0 conn(0x7fa1f01538a0 0x7fa1f015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.087+0000 7fa1ef7fe640 1 -- 172.21.15.125:0/244542591 >> v2:172.21.15.136:3300/0 conn(0x7fa1f01538a0 msgr2=0x7fa1f015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:58.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.087+0000 7fa1ef7fe640 1 --2- 172.21.15.125:0/244542591 >> v2:172.21.15.136:3300/0 conn(0x7fa1f01538a0 0x7fa1f015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.087+0000 7fa1ef7fe640 1 -- 172.21.15.125:0/244542591 >> v1:172.21.15.125:6789/0 conn(0x7fa1f00ca0e0 legacy=0x7fa1f015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.087+0000 7fa1ef7fe640 1 -- 172.21.15.125:0/244542591 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f00cad70 con 0x7fa1f00cf160 2023-10-30T12:28:58.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.087+0000 7fa1eeffd640 1 --2- 172.21.15.125:0/244542591 >> v2:172.21.15.136:3300/0 conn(0x7fa1f01538a0 0x7fa1f015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:58.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.087+0000 7fa1ee7fc640 1 -- 172.21.15.125:0/244542591 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575854319 0 0) 0x7fa1f00cb800 con 0x7fa1f00ca0e0 2023-10-30T12:28:58.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.087+0000 7fa1ef7fe640 1 --2- 172.21.15.125:0/244542591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1f00cf160 0x7fa1f00cf590 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00cfd10 tx=0x7fa1e801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b871684159f802 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.087+0000 7fa1ee7fc640 1 -- 172.21.15.125:0/244542591 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e80af400 con 0x7fa1f00cf160 2023-10-30T12:28:58.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.087+0000 7fa1ee7fc640 1 -- 172.21.15.125:0/244542591 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1e80ab050 con 0x7fa1f00cf160 2023-10-30T12:28:58.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.087+0000 7fa1ee7fc640 1 -- 172.21.15.125:0/244542591 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e80af780 con 0x7fa1f00cf160 2023-10-30T12:28:58.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 -- 172.21.15.125:0/244542591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1f00cf160 msgr2=0x7fa1f00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 --2- 172.21.15.125:0/244542591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1f00cf160 0x7fa1f00cf590 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00cfd10 tx=0x7fa1e801f1c0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 -- 172.21.15.125:0/244542591 shutdown_connections 2023-10-30T12:28:58.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 --2- 172.21.15.125:0/244542591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1f00cf160 0x7fa1f00cf590 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 --2- 172.21.15.125:0/244542591 >> v2:172.21.15.136:3300/0 conn(0x7fa1f01538a0 0x7fa1f015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 -- 172.21.15.125:0/244542591 >> 172.21.15.125:0/244542591 conn(0x7fa1f006bfe0 msgr2=0x7fa1f006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:58.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 -- 172.21.15.125:0/244542591 shutdown_connections 2023-10-30T12:28:58.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 -- 172.21.15.125:0/244542591 wait complete. 2023-10-30T12:28:58.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 Processor -- start 2023-10-30T12:28:58.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 -- start start 2023-10-30T12:28:58.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1f00ca0e0 0x7fa1f00cdc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1f00cf160 0x7fa1f01e3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1f01538a0 0x7fa1f01e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f01e31d0 con 0x7fa1f00cf160 2023-10-30T12:28:58.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f0148450 con 0x7fa1f00ca0e0 2023-10-30T12:28:58.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1f546b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f0148730 con 0x7fa1f01538a0 2023-10-30T12:28:58.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1e7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1f00cf160 0x7fa1f01e3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1e7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1f00cf160 0x7fa1f01e3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33412/0 (socket says 172.21.15.125:33412) 2023-10-30T12:28:58.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1e7fff640 1 -- 172.21.15.125:0/3761860048 learned_addr learned my addr 172.21.15.125:0/3761860048 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:58.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1eeffd640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1f00ca0e0 0x7fa1f00cdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1ef7fe640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1f01538a0 0x7fa1f01e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1e7fff640 1 -- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1f01538a0 msgr2=0x7fa1f01e5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1e7fff640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1f01538a0 0x7fa1f01e5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1e7fff640 1 -- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1f00ca0e0 msgr2=0x7fa1f00cdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1e7fff640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1f00ca0e0 0x7fa1f00cdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1e7fff640 1 -- 172.21.15.125:0/3761860048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e8009b30 con 0x7fa1f00cf160 2023-10-30T12:28:58.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1ef7fe640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1f01538a0 0x7fa1f01e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:58.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.091+0000 7fa1e7fff640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1f00cf160 0x7fa1f01e3660 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800c480 tx=0x7fa1d800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.095+0000 7fa1e77fe640 1 -- 172.21.15.125:0/3761860048 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d80074e0 con 0x7fa1f00cf160 2023-10-30T12:28:58.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.095+0000 7fa1e77fe640 1 -- 172.21.15.125:0/3761860048 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d8007660 con 0x7fa1f00cf160 2023-10-30T12:28:58.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.095+0000 7fa1f546b640 1 -- 172.21.15.125:0/3761860048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f01e2b80 con 0x7fa1f00cf160 2023-10-30T12:28:58.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.095+0000 7fa1e77fe640 1 -- 172.21.15.125:0/3761860048 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d8004a30 con 0x7fa1f00cf160 2023-10-30T12:28:58.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.095+0000 7fa1f546b640 1 -- 172.21.15.125:0/3761860048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f01e57a0 con 0x7fa1f00cf160 2023-10-30T12:28:58.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.095+0000 7fa1f546b640 1 -- 172.21.15.125:0/3761860048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f00b2a50 con 0x7fa1f00cf160 2023-10-30T12:28:58.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.095+0000 7fa1e77fe640 1 -- 172.21.15.125:0/3761860048 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa1d8004b90 con 0x7fa1f00cf160 2023-10-30T12:28:58.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.095+0000 7fa1e77fe640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1d00664a0 0x7fa1d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.095+0000 7fa1e77fe640 1 -- 172.21.15.125:0/3761860048 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa1d8082fd0 con 0x7fa1f00cf160 2023-10-30T12:28:58.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.095+0000 7fa1eeffd640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1d00664a0 0x7fa1d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.099+0000 7fa1eeffd640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1d00664a0 0x7fa1d0068960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009950 tx=0x7fa1e0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.099+0000 7fa1e77fe640 1 -- 172.21.15.125:0/3761860048 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa1d8053f80 con 0x7fa1f00cf160 2023-10-30T12:28:58.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.251+0000 7fa1f546b640 1 -- 172.21.15.125:0/3761860048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1f0159850 con 0x7fa1f00cf160 2023-10-30T12:28:58.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.251+0000 7fa1e77fe640 1 -- 172.21.15.125:0/3761860048 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16557 (secure 0 0 0) 0x7fa1d8018090 con 0x7fa1f00cf160 2023-10-30T12:28:58.254 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:58.255 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:45.927862+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:58.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.255+0000 7fa1f546b640 1 -- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1d00664a0 msgr2=0x7fa1d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.255+0000 7fa1f546b640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1d00664a0 0x7fa1d0068960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009950 tx=0x7fa1e0008040 comp rx=0 tx=0).stop 2023-10-30T12:28:58.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.255+0000 7fa1f546b640 1 -- 172.21.15.125:0/3761860048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1f00cf160 msgr2=0x7fa1f01e3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.255+0000 7fa1f546b640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1f00cf160 0x7fa1f01e3660 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa1d800c480 tx=0x7fa1d800c940 comp rx=0 tx=0).stop 2023-10-30T12:28:58.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.255+0000 7fa1f546b640 1 -- 172.21.15.125:0/3761860048 shutdown_connections 2023-10-30T12:28:58.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.255+0000 7fa1f546b640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1d00664a0 0x7fa1d0068960 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.255+0000 7fa1f546b640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1f01538a0 0x7fa1f01e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.255+0000 7fa1f546b640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1f00cf160 0x7fa1f01e3660 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.255+0000 7fa1f546b640 1 --2- 172.21.15.125:0/3761860048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1f00ca0e0 0x7fa1f00cdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.255+0000 7fa1f546b640 1 -- 172.21.15.125:0/3761860048 >> 172.21.15.125:0/3761860048 conn(0x7fa1f006bfe0 msgr2=0x7fa1f0066390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:58.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.255+0000 7fa1f546b640 1 -- 172.21.15.125:0/3761860048 shutdown_connections 2023-10-30T12:28:58.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.255+0000 7fa1f546b640 1 -- 172.21.15.125:0/3761860048 wait complete. 2023-10-30T12:28:58.275 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:31.063403+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T12:28:43.007849+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T12:28:43.401231+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:28:58.276 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:28:58.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.415+0000 7f3b8be9e640 1 Processor -- start 2023-10-30T12:28:58.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.415+0000 7f3b8be9e640 1 -- start start 2023-10-30T12:28:58.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.415+0000 7f3b8be9e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3b840c5f20 0x7f3b840c02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.415+0000 7f3b8be9e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b84069e90 con 0x7f3b840bfec0 2023-10-30T12:28:58.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.415+0000 7f3b8be9e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b8406aa10 con 0x7f3b840c6bf0 2023-10-30T12:28:58.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.415+0000 7f3b8be9e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b840c07e0 con 0x7f3b840c5f20 2023-10-30T12:28:58.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.415+0000 7f3b89439640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3b840c5f20 0x7f3b840c02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.415+0000 7f3b89c3a640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3b840bfec0 0x7f3b840c5810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54206/0 (socket says 172.21.15.125:54206) 2023-10-30T12:28:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.415+0000 7f3b89c3a640 1 -- 172.21.15.125:0/166886517 learned_addr learned my addr 172.21.15.125:0/166886517 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.415+0000 7f3b89439640 1 -- 172.21.15.125:0/166886517 >> v1:172.21.15.136:6789/0 conn(0x7f3b840c6bf0 legacy=0x7f3b84168500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.415+0000 7f3b89439640 1 -- 172.21.15.125:0/166886517 >> v1:172.21.15.125:6789/0 conn(0x7f3b840bfec0 legacy=0x7f3b840c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.415+0000 7f3b89439640 1 -- 172.21.15.125:0/166886517 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b840c09c0 con 0x7f3b840c5f20 2023-10-30T12:28:58.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.415+0000 7f3b88c38640 1 -- 172.21.15.125:0/166886517 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763795390 0 0) 0x7f3b84069e90 con 0x7f3b840bfec0 2023-10-30T12:28:58.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.419+0000 7f3b89439640 1 --2- 172.21.15.125:0/166886517 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b840c5f20 0x7f3b840c02a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3b740097c0 tx=0x7f3b7401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f39ce049b3adac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.419+0000 7f3b88c38640 1 -- 172.21.15.125:0/166886517 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b740ab680 con 0x7f3b840c5f20 2023-10-30T12:28:58.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.419+0000 7f3b88c38640 1 -- 172.21.15.125:0/166886517 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b740ab7e0 con 0x7f3b840c5f20 2023-10-30T12:28:58.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.419+0000 7f3b88c38640 1 -- 172.21.15.125:0/166886517 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b740abb70 con 0x7f3b840c5f20 2023-10-30T12:28:58.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.419+0000 7f3b8be9e640 1 -- 172.21.15.125:0/166886517 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b840c5f20 msgr2=0x7f3b840c02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.419+0000 7f3b8be9e640 1 --2- 172.21.15.125:0/166886517 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b840c5f20 0x7f3b840c02a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3b740097c0 tx=0x7f3b7401c480 comp rx=0 tx=0).stop 2023-10-30T12:28:58.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.419+0000 7f3b8be9e640 1 -- 172.21.15.125:0/166886517 shutdown_connections 2023-10-30T12:28:58.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.419+0000 7f3b8be9e640 1 --2- 172.21.15.125:0/166886517 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b840c5f20 0x7f3b840c02a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.419+0000 7f3b8be9e640 1 -- 172.21.15.125:0/166886517 >> 172.21.15.125:0/166886517 conn(0x7f3b840c7910 msgr2=0x7f3b840c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:58.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.419+0000 7f3b8be9e640 1 -- 172.21.15.125:0/166886517 shutdown_connections 2023-10-30T12:28:58.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.419+0000 7f3b8be9e640 1 -- 172.21.15.125:0/166886517 wait complete. 2023-10-30T12:28:58.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b8be9e640 1 Processor -- start 2023-10-30T12:28:58.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b8be9e640 1 -- start start 2023-10-30T12:28:58.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b8be9e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b840bfec0 0x7f3b8415a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b8be9e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b840c6bf0 0x7f3b84158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b8be9e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b841558e0 0x7f3b84143910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b8be9e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8415a650 con 0x7f3b840bfec0 2023-10-30T12:28:58.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b8be9e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8415b6b0 con 0x7f3b841558e0 2023-10-30T12:28:58.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b8be9e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b841f6670 con 0x7f3b840c6bf0 2023-10-30T12:28:58.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b89c3a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b840bfec0 0x7f3b8415a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b89c3a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b840bfec0 0x7f3b8415a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33430/0 (socket says 172.21.15.125:33430) 2023-10-30T12:28:58.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b89c3a640 1 -- 172.21.15.125:0/1963143568 learned_addr learned my addr 172.21.15.125:0/1963143568 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:58.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b89439640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b840c6bf0 0x7f3b84158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b8a43b640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b841558e0 0x7f3b84143910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b89c3a640 1 -- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b840c6bf0 msgr2=0x7f3b84158f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b89c3a640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b840c6bf0 0x7f3b84158f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b89c3a640 1 -- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b841558e0 msgr2=0x7f3b84143910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b89c3a640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b841558e0 0x7f3b84143910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b89c3a640 1 -- 172.21.15.125:0/1963143568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b74009420 con 0x7f3b840bfec0 2023-10-30T12:28:58.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b8a43b640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b841558e0 0x7f3b84143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:58.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b89c3a640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b840bfec0 0x7f3b8415a110 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000c420 tx=0x7f3b8000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b7a7fc640 1 -- 172.21.15.125:0/1963143568 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b800074e0 con 0x7f3b840bfec0 2023-10-30T12:28:58.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b7a7fc640 1 -- 172.21.15.125:0/1963143568 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b80007640 con 0x7f3b840bfec0 2023-10-30T12:28:58.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b7a7fc640 1 -- 172.21.15.125:0/1963143568 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b800059f0 con 0x7f3b840bfec0 2023-10-30T12:28:58.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b8be9e640 1 -- 172.21.15.125:0/1963143568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b8415ab70 con 0x7f3b840bfec0 2023-10-30T12:28:58.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b8be9e640 1 -- 172.21.15.125:0/1963143568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b841594b0 con 0x7f3b840bfec0 2023-10-30T12:28:58.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.423+0000 7f3b8be9e640 1 -- 172.21.15.125:0/1963143568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8404e170 con 0x7f3b840bfec0 2023-10-30T12:28:58.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.427+0000 7f3b7a7fc640 1 -- 172.21.15.125:0/1963143568 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3b80005b50 con 0x7f3b840bfec0 2023-10-30T12:28:58.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.427+0000 7f3b7a7fc640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b68066470 0x7f3b68068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.427+0000 7f3b7a7fc640 1 -- 172.21.15.125:0/1963143568 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3b80083490 con 0x7f3b840bfec0 2023-10-30T12:28:58.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.427+0000 7f3b89439640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b68066470 0x7f3b68068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.427+0000 7f3b89439640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b68066470 0x7f3b68068930 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3b740af000 tx=0x7f3b740a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.431+0000 7f3b7a7fc640 1 -- 172.21.15.125:0/1963143568 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3b80050890 con 0x7f3b840bfec0 2023-10-30T12:28:58.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.579+0000 7f3b8be9e640 1 -- 172.21.15.125:0/1963143568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3b840c76f0 con 0x7f3b840bfec0 2023-10-30T12:28:58.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.579+0000 7f3b7a7fc640 1 -- 172.21.15.125:0/1963143568 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3b800170a0 con 0x7f3b840bfec0 2023-10-30T12:28:58.584 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2023-10-30T12:28:58.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.583+0000 7f3b8be9e640 1 -- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b68066470 msgr2=0x7f3b68068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.583+0000 7f3b8be9e640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b68066470 0x7f3b68068930 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f3b740af000 tx=0x7f3b740a7000 comp rx=0 tx=0).stop 2023-10-30T12:28:58.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.583+0000 7f3b8be9e640 1 -- 172.21.15.125:0/1963143568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b840bfec0 msgr2=0x7f3b8415a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.583+0000 7f3b8be9e640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b840bfec0 0x7f3b8415a110 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3b8000c420 tx=0x7f3b8000c8e0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.583+0000 7f3b8be9e640 1 -- 172.21.15.125:0/1963143568 shutdown_connections 2023-10-30T12:28:58.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.583+0000 7f3b8be9e640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b68066470 0x7f3b68068930 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.583+0000 7f3b8be9e640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b841558e0 0x7f3b84143910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.583+0000 7f3b8be9e640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b840c6bf0 0x7f3b84158f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.583+0000 7f3b8be9e640 1 --2- 172.21.15.125:0/1963143568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b840bfec0 0x7f3b8415a110 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.583+0000 7f3b8be9e640 1 -- 172.21.15.125:0/1963143568 >> 172.21.15.125:0/1963143568 conn(0x7f3b840c7910 msgr2=0x7f3b84071de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:58.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.583+0000 7f3b8be9e640 1 -- 172.21.15.125:0/1963143568 shutdown_connections 2023-10-30T12:28:58.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.583+0000 7f3b8be9e640 1 -- 172.21.15.125:0/1963143568 wait complete. 2023-10-30T12:28:58.604 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T12:28:58.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.739+0000 7efc3e605640 1 Processor -- start 2023-10-30T12:28:58.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.739+0000 7efc3e605640 1 -- start start 2023-10-30T12:28:58.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.743+0000 7efc3e605640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7efc381508c0 0x7efc380c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.743+0000 7efc3e605640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efc380c6e60 0x7efc38156930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.743+0000 7efc3e605640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc380c0a30 con 0x7efc380722a0 2023-10-30T12:28:58.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.743+0000 7efc3e605640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc380c85b0 con 0x7efc380c6e60 2023-10-30T12:28:58.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.743+0000 7efc3e605640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc3806f650 con 0x7efc381508c0 2023-10-30T12:28:58.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.743+0000 7efc3cba2640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efc380c6e60 0x7efc38156930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.743+0000 7efc377fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7efc380722a0 0x7efc380bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54216/0 (socket says 172.21.15.125:54216) 2023-10-30T12:28:58.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.743+0000 7efc377fe640 1 -- 172.21.15.125:0/2366709950 learned_addr learned my addr 172.21.15.125:0/2366709950 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:58.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.743+0000 7efc37fff640 1 --2- 172.21.15.125:0/2366709950 >> v2:172.21.15.136:3300/0 conn(0x7efc381508c0 0x7efc380c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.743+0000 7efc3cba2640 1 -- 172.21.15.125:0/2366709950 >> v2:172.21.15.136:3300/0 conn(0x7efc381508c0 msgr2=0x7efc380c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:58.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.743+0000 7efc3cba2640 1 --2- 172.21.15.125:0/2366709950 >> v2:172.21.15.136:3300/0 conn(0x7efc381508c0 0x7efc380c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.743+0000 7efc3cba2640 1 -- 172.21.15.125:0/2366709950 >> v1:172.21.15.125:6789/0 conn(0x7efc380722a0 legacy=0x7efc380bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.743+0000 7efc3cba2640 1 -- 172.21.15.125:0/2366709950 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3806f930 con 0x7efc380c6e60 2023-10-30T12:28:58.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.747+0000 7efc3cba2640 1 --2- 172.21.15.125:0/2366709950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc380c6e60 0x7efc38156930 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efc380c7540 tx=0x7efc3001c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f430481acade756b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.747+0000 7efc36ffd640 1 -- 172.21.15.125:0/2366709950 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc300a5830 con 0x7efc380c6e60 2023-10-30T12:28:58.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.747+0000 7efc36ffd640 1 -- 172.21.15.125:0/2366709950 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc300a5990 con 0x7efc380c6e60 2023-10-30T12:28:58.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.747+0000 7efc36ffd640 1 -- 172.21.15.125:0/2366709950 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc300a5d20 con 0x7efc380c6e60 2023-10-30T12:28:58.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.747+0000 7efc3e605640 1 -- 172.21.15.125:0/2366709950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc380c6e60 msgr2=0x7efc38156930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.747+0000 7efc3e605640 1 --2- 172.21.15.125:0/2366709950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc380c6e60 0x7efc38156930 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7efc380c7540 tx=0x7efc3001c880 comp rx=0 tx=0).stop 2023-10-30T12:28:58.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.747+0000 7efc3e605640 1 -- 172.21.15.125:0/2366709950 shutdown_connections 2023-10-30T12:28:58.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.747+0000 7efc3e605640 1 --2- 172.21.15.125:0/2366709950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc380c6e60 0x7efc38156930 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.747+0000 7efc3e605640 1 --2- 172.21.15.125:0/2366709950 >> v2:172.21.15.136:3300/0 conn(0x7efc381508c0 0x7efc380c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.747+0000 7efc3e605640 1 -- 172.21.15.125:0/2366709950 >> 172.21.15.125:0/2366709950 conn(0x7efc380c7910 msgr2=0x7efc380c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:58.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.747+0000 7efc3e605640 1 -- 172.21.15.125:0/2366709950 shutdown_connections 2023-10-30T12:28:58.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.747+0000 7efc3e605640 1 -- 172.21.15.125:0/2366709950 wait complete. 2023-10-30T12:28:58.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.747+0000 7efc3e605640 1 Processor -- start 2023-10-30T12:28:58.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.747+0000 7efc3e605640 1 -- start start 2023-10-30T12:28:58.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc3e605640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efc380722a0 0x7efc381f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc3e605640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc380c6e60 0x7efc3806a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc3e605640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efc381508c0 0x7efc3814b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc3e605640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc381f3270 con 0x7efc380c6e60 2023-10-30T12:28:58.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc3e605640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3814ab70 con 0x7efc381508c0 2023-10-30T12:28:58.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc3e605640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3814ae50 con 0x7efc380722a0 2023-10-30T12:28:58.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc377fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc380c6e60 0x7efc3806a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc377fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc380c6e60 0x7efc3806a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33450/0 (socket says 172.21.15.125:33450) 2023-10-30T12:28:58.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc377fe640 1 -- 172.21.15.125:0/2244388885 learned_addr learned my addr 172.21.15.125:0/2244388885 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:58.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc3cba2640 1 --2- 172.21.15.125:0/2244388885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efc381508c0 0x7efc3814b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc37fff640 1 --2- 172.21.15.125:0/2244388885 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efc380722a0 0x7efc381f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc377fe640 1 -- 172.21.15.125:0/2244388885 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efc380722a0 msgr2=0x7efc381f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc377fe640 1 --2- 172.21.15.125:0/2244388885 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efc380722a0 0x7efc381f2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc377fe640 1 -- 172.21.15.125:0/2244388885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efc381508c0 msgr2=0x7efc3814b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc377fe640 1 --2- 172.21.15.125:0/2244388885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efc381508c0 0x7efc3814b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc377fe640 1 -- 172.21.15.125:0/2244388885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc30009a70 con 0x7efc380c6e60 2023-10-30T12:28:58.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc377fe640 1 --2- 172.21.15.125:0/2244388885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc380c6e60 0x7efc3806a0f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efc2c00c400 tx=0x7efc2c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc34ff9640 1 -- 172.21.15.125:0/2244388885 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c005cf0 con 0x7efc380c6e60 2023-10-30T12:28:58.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc34ff9640 1 -- 172.21.15.125:0/2244388885 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc2c005e50 con 0x7efc380c6e60 2023-10-30T12:28:58.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc3e605640 1 -- 172.21.15.125:0/2244388885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc3814c790 con 0x7efc380c6e60 2023-10-30T12:28:58.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc34ff9640 1 -- 172.21.15.125:0/2244388885 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c0072d0 con 0x7efc380c6e60 2023-10-30T12:28:58.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc3e605640 1 -- 172.21.15.125:0/2244388885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc38156100 con 0x7efc380c6e60 2023-10-30T12:28:58.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc3e605640 1 -- 172.21.15.125:0/2244388885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc3804e170 con 0x7efc380c6e60 2023-10-30T12:28:58.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.751+0000 7efc34ff9640 1 -- 172.21.15.125:0/2244388885 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efc2c007430 con 0x7efc380c6e60 2023-10-30T12:28:58.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.755+0000 7efc34ff9640 1 --2- 172.21.15.125:0/2244388885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efc240664a0 0x7efc24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:58.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.755+0000 7efc34ff9640 1 -- 172.21.15.125:0/2244388885 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc2c0833f0 con 0x7efc380c6e60 2023-10-30T12:28:58.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.755+0000 7efc37fff640 1 --2- 172.21.15.125:0/2244388885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efc240664a0 0x7efc24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:58.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.755+0000 7efc37fff640 1 --2- 172.21.15.125:0/2244388885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efc240664a0 0x7efc24068960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efc28009ad0 tx=0x7efc28008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:58.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.759+0000 7efc34ff9640 1 -- 172.21.15.125:0/2244388885 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efc2c050880 con 0x7efc380c6e60 2023-10-30T12:28:58.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.903+0000 7efc3e605640 1 -- 172.21.15.125:0/2244388885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efc380b0bb0 con 0x7efc380c6e60 2023-10-30T12:28:58.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.907+0000 7efc34ff9640 1 -- 172.21.15.125:0/2244388885 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7efc2c053ea0 con 0x7efc380c6e60 2023-10-30T12:28:58.909 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2023-10-30T12:28:58.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.907+0000 7efc3e605640 1 -- 172.21.15.125:0/2244388885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efc240664a0 msgr2=0x7efc24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.907+0000 7efc3e605640 1 --2- 172.21.15.125:0/2244388885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efc240664a0 0x7efc24068960 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7efc28009ad0 tx=0x7efc28008040 comp rx=0 tx=0).stop 2023-10-30T12:28:58.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.907+0000 7efc3e605640 1 -- 172.21.15.125:0/2244388885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc380c6e60 msgr2=0x7efc3806a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:58.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.907+0000 7efc3e605640 1 --2- 172.21.15.125:0/2244388885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc380c6e60 0x7efc3806a0f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7efc2c00c400 tx=0x7efc2c00c8c0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.907+0000 7efc3e605640 1 -- 172.21.15.125:0/2244388885 shutdown_connections 2023-10-30T12:28:58.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.907+0000 7efc3e605640 1 --2- 172.21.15.125:0/2244388885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efc240664a0 0x7efc24068960 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.907+0000 7efc3e605640 1 --2- 172.21.15.125:0/2244388885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efc381508c0 0x7efc3814b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.907+0000 7efc3e605640 1 --2- 172.21.15.125:0/2244388885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efc380c6e60 0x7efc3806a0f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.907+0000 7efc3e605640 1 --2- 172.21.15.125:0/2244388885 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efc380722a0 0x7efc381f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:58.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.907+0000 7efc3e605640 1 -- 172.21.15.125:0/2244388885 >> 172.21.15.125:0/2244388885 conn(0x7efc380c7910 msgr2=0x7efc3806d6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:58.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.907+0000 7efc3e605640 1 -- 172.21.15.125:0/2244388885 shutdown_connections 2023-10-30T12:28:58.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:58.907+0000 7efc3e605640 1 -- 172.21.15.125:0/2244388885 wait complete. 2023-10-30T12:28:58.929 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T12:28:59.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.067+0000 7f0985b63640 1 Processor -- start 2023-10-30T12:28:59.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.067+0000 7f0985b63640 1 -- start start 2023-10-30T12:28:59.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.071+0000 7f0985b63640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0980153c20 0x7f098015a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.071+0000 7f0985b63640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09800cb4c0 0x7f09800cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.071+0000 7f0985b63640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f098015a6b0 con 0x7f0980153840 2023-10-30T12:28:59.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.071+0000 7f0985b63640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f098015dbc0 con 0x7f0980153c20 2023-10-30T12:28:59.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.071+0000 7f0985b63640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098015d730 con 0x7f09800cb4c0 2023-10-30T12:28:59.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.071+0000 7f0977fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0980153c20 0x7f098015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.071+0000 7f0977fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0980153c20 0x7f098015a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33458/0 (socket says 172.21.15.125:33458) 2023-10-30T12:28:59.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.071+0000 7f0977fff640 1 -- 172.21.15.125:0/3138635594 learned_addr learned my addr 172.21.15.125:0/3138635594 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:59.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.071+0000 7f0985362640 1 --2- 172.21.15.125:0/3138635594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09800cb4c0 0x7f09800cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.071+0000 7f0977fff640 1 -- 172.21.15.125:0/3138635594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09800cb4c0 msgr2=0x7f09800cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.071+0000 7f0977fff640 1 --2- 172.21.15.125:0/3138635594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09800cb4c0 0x7f09800cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.071+0000 7f0977fff640 1 -- 172.21.15.125:0/3138635594 >> v1:172.21.15.136:6789/0 conn(0x7f0980153840 legacy=0x7f09800c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.071+0000 7f0977fff640 1 -- 172.21.15.125:0/3138635594 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09800ca8f0 con 0x7f0980153c20 2023-10-30T12:28:59.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.071+0000 7f0985362640 1 --2- 172.21.15.125:0/3138635594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09800cb4c0 0x7f09800cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:59.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0977fff640 1 --2- 172.21.15.125:0/3138635594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980153c20 0x7f098015a170 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0970009870 tx=0x7f097001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7badf9cf47212b97 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f09777fe640 1 -- 172.21.15.125:0/3138635594 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700a8a30 con 0x7f0980153c20 2023-10-30T12:28:59.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f09777fe640 1 -- 172.21.15.125:0/3138635594 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09700a8b90 con 0x7f0980153c20 2023-10-30T12:28:59.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f09777fe640 1 -- 172.21.15.125:0/3138635594 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700b0440 con 0x7f0980153c20 2023-10-30T12:28:59.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 -- 172.21.15.125:0/3138635594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980153c20 msgr2=0x7f098015a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 --2- 172.21.15.125:0/3138635594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980153c20 0x7f098015a170 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0970009870 tx=0x7f097001ed30 comp rx=0 tx=0).stop 2023-10-30T12:28:59.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 -- 172.21.15.125:0/3138635594 shutdown_connections 2023-10-30T12:28:59.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 --2- 172.21.15.125:0/3138635594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09800cb4c0 0x7f09800cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 --2- 172.21.15.125:0/3138635594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980153c20 0x7f098015a170 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 -- 172.21.15.125:0/3138635594 >> 172.21.15.125:0/3138635594 conn(0x7f098006be40 msgr2=0x7f098006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 -- 172.21.15.125:0/3138635594 shutdown_connections 2023-10-30T12:28:59.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 -- 172.21.15.125:0/3138635594 wait complete. 2023-10-30T12:28:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 Processor -- start 2023-10-30T12:28:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 -- start start 2023-10-30T12:28:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09800cb4c0 0x7f09801e59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980153840 0x7f09801f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0980153c20 0x7f09801f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09801e3590 con 0x7f0980153840 2023-10-30T12:28:59.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09801e5f20 con 0x7f0980153c20 2023-10-30T12:28:59.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.075+0000 7f0985b63640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09801f31d0 con 0x7f09800cb4c0 2023-10-30T12:28:59.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f0977fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980153840 0x7f09801f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f0977fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980153840 0x7f09801f2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33462/0 (socket says 172.21.15.125:33462) 2023-10-30T12:28:59.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f0977fff640 1 -- 172.21.15.125:0/1553785087 learned_addr learned my addr 172.21.15.125:0/1553785087 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:59.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f0985362640 1 --2- 172.21.15.125:0/1553785087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0980153c20 0x7f09801f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f0984b61640 1 --2- 172.21.15.125:0/1553785087 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09800cb4c0 0x7f09801e59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f0977fff640 1 -- 172.21.15.125:0/1553785087 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09800cb4c0 msgr2=0x7f09801e59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f0977fff640 1 --2- 172.21.15.125:0/1553785087 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09800cb4c0 0x7f09801e59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f0977fff640 1 -- 172.21.15.125:0/1553785087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0980153c20 msgr2=0x7f09801f2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f0977fff640 1 --2- 172.21.15.125:0/1553785087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0980153c20 0x7f09801f2ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f0977fff640 1 -- 172.21.15.125:0/1553785087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09700094d0 con 0x7f0980153840 2023-10-30T12:28:59.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f0977fff640 1 --2- 172.21.15.125:0/1553785087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980153840 0x7f09801f2660 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f09700b3040 tx=0x7f097001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f09757fa640 1 -- 172.21.15.125:0/1553785087 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700be070 con 0x7f0980153840 2023-10-30T12:28:59.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f09757fa640 1 -- 172.21.15.125:0/1553785087 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09700ba400 con 0x7f0980153840 2023-10-30T12:28:59.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f0985b63640 1 -- 172.21.15.125:0/1553785087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09801e5020 con 0x7f0980153840 2023-10-30T12:28:59.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f09757fa640 1 -- 172.21.15.125:0/1553785087 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700ba7d0 con 0x7f0980153840 2023-10-30T12:28:59.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f0985b63640 1 -- 172.21.15.125:0/1553785087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09801e4d40 con 0x7f0980153840 2023-10-30T12:28:59.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f0985b63640 1 -- 172.21.15.125:0/1553785087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f098004e9c0 con 0x7f0980153840 2023-10-30T12:28:59.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.079+0000 7f09757fa640 1 -- 172.21.15.125:0/1553785087 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f09700c3410 con 0x7f0980153840 2023-10-30T12:28:59.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.083+0000 7f09757fa640 1 --2- 172.21.15.125:0/1553785087 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f096c0664a0 0x7f096c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.083+0000 7f09757fa640 1 -- 172.21.15.125:0/1553785087 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0970121940 con 0x7f0980153840 2023-10-30T12:28:59.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.083+0000 7f0984b61640 1 --2- 172.21.15.125:0/1553785087 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f096c0664a0 0x7f096c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.083+0000 7f0984b61640 1 --2- 172.21.15.125:0/1553785087 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f096c0664a0 0x7f096c068960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f097c000f70 tx=0x7f097c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.087+0000 7f09757fa640 1 -- 172.21.15.125:0/1553785087 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f09700eee50 con 0x7f0980153840 2023-10-30T12:28:59.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.235+0000 7f0985b63640 1 -- 172.21.15.125:0/1553785087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f09800c48c0 con 0x7f0980153840 2023-10-30T12:28:59.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.235+0000 7f09757fa640 1 -- 172.21.15.125:0/1553785087 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f09700f2470 con 0x7f0980153840 2023-10-30T12:28:59.240 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2023-10-30T12:28:59.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.239+0000 7f0985b63640 1 -- 172.21.15.125:0/1553785087 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f096c0664a0 msgr2=0x7f096c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.239+0000 7f0985b63640 1 --2- 172.21.15.125:0/1553785087 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f096c0664a0 0x7f096c068960 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f097c000f70 tx=0x7f097c009040 comp rx=0 tx=0).stop 2023-10-30T12:28:59.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.239+0000 7f0985b63640 1 -- 172.21.15.125:0/1553785087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980153840 msgr2=0x7f09801f2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.239+0000 7f0985b63640 1 --2- 172.21.15.125:0/1553785087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980153840 0x7f09801f2660 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f09700b3040 tx=0x7f097001bad0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.239+0000 7f0985b63640 1 -- 172.21.15.125:0/1553785087 shutdown_connections 2023-10-30T12:28:59.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.239+0000 7f0985b63640 1 --2- 172.21.15.125:0/1553785087 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f096c0664a0 0x7f096c068960 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.239+0000 7f0985b63640 1 --2- 172.21.15.125:0/1553785087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0980153c20 0x7f09801f2ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.239+0000 7f0985b63640 1 --2- 172.21.15.125:0/1553785087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0980153840 0x7f09801f2660 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.239+0000 7f0985b63640 1 --2- 172.21.15.125:0/1553785087 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f09800cb4c0 0x7f09801e59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.239+0000 7f0985b63640 1 -- 172.21.15.125:0/1553785087 >> 172.21.15.125:0/1553785087 conn(0x7f098006be40 msgr2=0x7f09800b3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.239+0000 7f0985b63640 1 -- 172.21.15.125:0/1553785087 shutdown_connections 2023-10-30T12:28:59.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.239+0000 7f0985b63640 1 -- 172.21.15.125:0/1553785087 wait complete. 2023-10-30T12:28:59.260 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:28:59.385 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:28:59.383+0000 7f6a6fdf8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:28:59.386 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:28:59.383+0000 7f6a6fdf8640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:28:59.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.395+0000 7faffd780640 1 Processor -- start 2023-10-30T12:28:59.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.395+0000 7faffd780640 1 -- start start 2023-10-30T12:28:59.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.395+0000 7faffd780640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faff80c05d0 0x7faff813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.395+0000 7faffd780640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7faff80c6f60 0x7faff813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.395+0000 7faffd780640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff80c0a50 con 0x7faff80c6b80 2023-10-30T12:28:59.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.395+0000 7faffd780640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff806f6e0 con 0x7faff80c6f60 2023-10-30T12:28:59.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.395+0000 7faffd780640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff806f9c0 con 0x7faff80c05d0 2023-10-30T12:28:59.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.395+0000 7fafee7fc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faff80c05d0 0x7faff813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.395+0000 7faff77fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7faff80c6f60 0x7faff813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.395+0000 7faff6ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7faff80c6b80 0x7faff80bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:37026/0 (socket says 172.21.15.125:37026) 2023-10-30T12:28:59.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.395+0000 7faff6ffd640 1 -- 172.21.15.125:0/1942774924 learned_addr learned my addr 172.21.15.125:0/1942774924 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:59.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.395+0000 7faff67fc640 1 -- 172.21.15.125:0/1942774924 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749168649 0 0) 0x7faff80c0a50 con 0x7faff80c6b80 2023-10-30T12:28:59.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.399+0000 7faff67fc640 1 -- 172.21.15.125:0/1942774924 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe4003530 con 0x7faff80c6b80 2023-10-30T12:28:59.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.399+0000 7fafee7fc640 1 -- 172.21.15.125:0/1942774924 >> v2:172.21.15.136:3300/0 conn(0x7faff80c6f60 msgr2=0x7faff813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:28:59.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.399+0000 7fafee7fc640 1 --2- 172.21.15.125:0/1942774924 >> v2:172.21.15.136:3300/0 conn(0x7faff80c6f60 0x7faff813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.399+0000 7fafee7fc640 1 -- 172.21.15.125:0/1942774924 >> v1:172.21.15.136:6789/0 conn(0x7faff80c6b80 legacy=0x7faff80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.399+0000 7fafee7fc640 1 -- 172.21.15.125:0/1942774924 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff806fca0 con 0x7faff80c05d0 2023-10-30T12:28:59.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.399+0000 7faff77fe640 1 --2- 172.21.15.125:0/1942774924 >> v2:172.21.15.136:3300/0 conn(0x7faff80c6f60 0x7faff813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:28:59.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.399+0000 7fafee7fc640 1 --2- 172.21.15.125:0/1942774924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faff80c05d0 0x7faff813ed90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fafe00097c0 tx=0x7fafe001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38ceea33cda90c9a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.399+0000 7faff67fc640 1 -- 172.21.15.125:0/1942774924 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe00ad640 con 0x7faff80c05d0 2023-10-30T12:28:59.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.399+0000 7faff67fc640 1 -- 172.21.15.125:0/1942774924 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe00ad7a0 con 0x7faff80c05d0 2023-10-30T12:28:59.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.399+0000 7faff67fc640 1 -- 172.21.15.125:0/1942774924 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe00adb10 con 0x7faff80c05d0 2023-10-30T12:28:59.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.399+0000 7faffd780640 1 -- 172.21.15.125:0/1942774924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faff80c05d0 msgr2=0x7faff813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.399+0000 7faffd780640 1 --2- 172.21.15.125:0/1942774924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faff80c05d0 0x7faff813ed90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fafe00097c0 tx=0x7fafe001c480 comp rx=0 tx=0).stop 2023-10-30T12:28:59.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faffd780640 1 -- 172.21.15.125:0/1942774924 shutdown_connections 2023-10-30T12:28:59.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faffd780640 1 --2- 172.21.15.125:0/1942774924 >> v2:172.21.15.136:3300/0 conn(0x7faff80c6f60 0x7faff813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faffd780640 1 --2- 172.21.15.125:0/1942774924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faff80c05d0 0x7faff813ed90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faffd780640 1 -- 172.21.15.125:0/1942774924 >> 172.21.15.125:0/1942774924 conn(0x7faff80c7910 msgr2=0x7faff80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faffd780640 1 -- 172.21.15.125:0/1942774924 shutdown_connections 2023-10-30T12:28:59.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faffd780640 1 -- 172.21.15.125:0/1942774924 wait complete. 2023-10-30T12:28:59.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faffd780640 1 Processor -- start 2023-10-30T12:28:59.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faffd780640 1 -- start start 2023-10-30T12:28:59.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faffd780640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faff80c05d0 0x7faff81f94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faffd780640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faff80c6b80 0x7faff81f9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faffd780640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faff80c6f60 0x7faff8158f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faffd780640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff8154e90 con 0x7faff80c6f60 2023-10-30T12:28:59.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faffd780640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff8158ae0 con 0x7faff80c6b80 2023-10-30T12:28:59.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faffd780640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff81fa060 con 0x7faff80c05d0 2023-10-30T12:28:59.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faff77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faff80c6f60 0x7faff8158f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faff77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faff80c6f60 0x7faff8158f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33474/0 (socket says 172.21.15.125:33474) 2023-10-30T12:28:59.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faff77fe640 1 -- 172.21.15.125:0/192828254 learned_addr learned my addr 172.21.15.125:0/192828254 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:59.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7fafee7fc640 1 --2- 172.21.15.125:0/192828254 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faff80c6b80 0x7faff81f9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faff6ffd640 1 --2- 172.21.15.125:0/192828254 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faff80c05d0 0x7faff81f94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faff77fe640 1 -- 172.21.15.125:0/192828254 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faff80c05d0 msgr2=0x7faff81f94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faff77fe640 1 --2- 172.21.15.125:0/192828254 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faff80c05d0 0x7faff81f94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faff77fe640 1 -- 172.21.15.125:0/192828254 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faff80c6b80 msgr2=0x7faff81f9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faff77fe640 1 --2- 172.21.15.125:0/192828254 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faff80c6b80 0x7faff81f9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faff77fe640 1 -- 172.21.15.125:0/192828254 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff0009420 con 0x7faff80c6f60 2023-10-30T12:28:59.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.403+0000 7faff77fe640 1 --2- 172.21.15.125:0/192828254 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faff80c6f60 0x7faff8158f70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7faff000e960 tx=0x7faff000ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.407+0000 7fafeffff640 1 -- 172.21.15.125:0/192828254 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0009680 con 0x7faff80c6f60 2023-10-30T12:28:59.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.407+0000 7fafeffff640 1 -- 172.21.15.125:0/192828254 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff000ca00 con 0x7faff80c6f60 2023-10-30T12:28:59.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.407+0000 7faffd780640 1 -- 172.21.15.125:0/192828254 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe0009420 con 0x7faff80c6f60 2023-10-30T12:28:59.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.407+0000 7fafeffff640 1 -- 172.21.15.125:0/192828254 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff000cdd0 con 0x7faff80c6f60 2023-10-30T12:28:59.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.407+0000 7faffd780640 1 -- 172.21.15.125:0/192828254 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff81550e0 con 0x7faff80c6f60 2023-10-30T12:28:59.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.407+0000 7faffd780640 1 -- 172.21.15.125:0/192828254 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff804e170 con 0x7faff80c6f60 2023-10-30T12:28:59.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.407+0000 7fafeffff640 1 -- 172.21.15.125:0/192828254 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faff0013470 con 0x7faff80c6f60 2023-10-30T12:28:59.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.407+0000 7fafeffff640 1 --2- 172.21.15.125:0/192828254 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fafd80664a0 0x7fafd8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.407+0000 7fafeffff640 1 -- 172.21.15.125:0/192828254 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faff00839e0 con 0x7faff80c6f60 2023-10-30T12:28:59.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.407+0000 7faff6ffd640 1 --2- 172.21.15.125:0/192828254 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fafd80664a0 0x7fafd8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.411+0000 7faff6ffd640 1 --2- 172.21.15.125:0/192828254 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fafd80664a0 0x7fafd8068960 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fafe8001570 tx=0x7fafe8001aa0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.411+0000 7fafeffff640 1 -- 172.21.15.125:0/192828254 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faff0050e70 con 0x7faff80c6f60 2023-10-30T12:28:59.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.591+0000 7faffd780640 1 -- 172.21.15.125:0/192828254 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faff8158c20 con 0x7faff80c6f60 2023-10-30T12:28:59.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.591+0000 7fafeffff640 1 -- 172.21.15.125:0/192828254 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+6539 (secure 0 0 0) 0x7faff0054490 con 0x7faff80c6f60 2023-10-30T12:28:59.595 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:59.595 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:28:55.942137+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.136:6835/3346501886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3346501886},{"type":"v1","addr":"172.21.15.136:6835","nonce":3346501886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:28:59.596 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 11 2023-10-30T12:28:59.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.595+0000 7faffd780640 1 -- 172.21.15.125:0/192828254 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fafd80664a0 msgr2=0x7fafd8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.595+0000 7faffd780640 1 --2- 172.21.15.125:0/192828254 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fafd80664a0 0x7fafd8068960 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fafe8001570 tx=0x7fafe8001aa0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.595+0000 7faffd780640 1 -- 172.21.15.125:0/192828254 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faff80c6f60 msgr2=0x7faff8158f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.595+0000 7faffd780640 1 --2- 172.21.15.125:0/192828254 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faff80c6f60 0x7faff8158f70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7faff000e960 tx=0x7faff000ee20 comp rx=0 tx=0).stop 2023-10-30T12:28:59.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.595+0000 7faffd780640 1 -- 172.21.15.125:0/192828254 shutdown_connections 2023-10-30T12:28:59.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.595+0000 7faffd780640 1 --2- 172.21.15.125:0/192828254 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fafd80664a0 0x7fafd8068960 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.595+0000 7faffd780640 1 --2- 172.21.15.125:0/192828254 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faff80c6f60 0x7faff8158f70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.595+0000 7faffd780640 1 --2- 172.21.15.125:0/192828254 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faff80c6b80 0x7faff81f9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.595+0000 7faffd780640 1 --2- 172.21.15.125:0/192828254 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faff80c05d0 0x7faff81f94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.595+0000 7faffd780640 1 -- 172.21.15.125:0/192828254 >> 172.21.15.125:0/192828254 conn(0x7faff80c7910 msgr2=0x7faff80c16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.595+0000 7faffd780640 1 -- 172.21.15.125:0/192828254 shutdown_connections 2023-10-30T12:28:59.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.595+0000 7faffd780640 1 -- 172.21.15.125:0/192828254 wait complete. 2023-10-30T12:28:59.616 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:28:59.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.751+0000 7fb02286b640 1 Processor -- start 2023-10-30T12:28:59.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.751+0000 7fb02286b640 1 -- start start 2023-10-30T12:28:59.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.751+0000 7fb02286b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb01c1538a0 0x7fb01c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.751+0000 7fb02286b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c0cb800 con 0x7fb01c0cf160 2023-10-30T12:28:59.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.751+0000 7fb02286b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb01c15dc30 con 0x7fb01c0ca0e0 2023-10-30T12:28:59.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.751+0000 7fb02286b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c0cb330 con 0x7fb01c1538a0 2023-10-30T12:28:59.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.751+0000 7fb020e08640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb01c0cf160 0x7fb01c164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54232/0 (socket says 172.21.15.125:54232) 2023-10-30T12:28:59.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.751+0000 7fb020e08640 1 -- 172.21.15.125:0/1753495073 learned_addr learned my addr 172.21.15.125:0/1753495073 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:28:59.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.751+0000 7fb01bfff640 1 --2- 172.21.15.125:0/1753495073 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb01c1538a0 0x7fb01c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.751+0000 7fb01affd640 1 -- 172.21.15.125:0/1753495073 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906416032 0 0) 0x7fb01c0cb800 con 0x7fb01c0cf160 2023-10-30T12:28:59.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.751+0000 7fb01affd640 1 -- 172.21.15.125:0/1753495073 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb004003530 con 0x7fb01c0cf160 2023-10-30T12:28:59.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.751+0000 7fb01bfff640 1 -- 172.21.15.125:0/1753495073 >> v1:172.21.15.136:6789/0 conn(0x7fb01c0ca0e0 legacy=0x7fb01c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.751+0000 7fb01bfff640 1 -- 172.21.15.125:0/1753495073 >> v1:172.21.15.125:6789/0 conn(0x7fb01c0cf160 legacy=0x7fb01c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.751+0000 7fb01bfff640 1 -- 172.21.15.125:0/1753495073 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c0cad70 con 0x7fb01c1538a0 2023-10-30T12:28:59.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.755+0000 7fb01bfff640 1 --2- 172.21.15.125:0/1753495073 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb01c1538a0 0x7fb01c15d6f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb00c0097f0 tx=0x7fb00c01ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3517aefd77650789 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb01affd640 1 -- 172.21.15.125:0/1753495073 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c0a89b0 con 0x7fb01c1538a0 2023-10-30T12:28:59.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb01affd640 1 -- 172.21.15.125:0/1753495073 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00c0a8b10 con 0x7fb01c1538a0 2023-10-30T12:28:59.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb01affd640 1 -- 172.21.15.125:0/1753495073 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00c0a8ea0 con 0x7fb01c1538a0 2023-10-30T12:28:59.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 -- 172.21.15.125:0/1753495073 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb01c1538a0 msgr2=0x7fb01c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 --2- 172.21.15.125:0/1753495073 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb01c1538a0 0x7fb01c15d6f0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb00c0097f0 tx=0x7fb00c01ecb0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 -- 172.21.15.125:0/1753495073 shutdown_connections 2023-10-30T12:28:59.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 --2- 172.21.15.125:0/1753495073 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb01c1538a0 0x7fb01c15d6f0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 -- 172.21.15.125:0/1753495073 >> 172.21.15.125:0/1753495073 conn(0x7fb01c06bfe0 msgr2=0x7fb01c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 -- 172.21.15.125:0/1753495073 shutdown_connections 2023-10-30T12:28:59.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 -- 172.21.15.125:0/1753495073 wait complete. 2023-10-30T12:28:59.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 Processor -- start 2023-10-30T12:28:59.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 -- start start 2023-10-30T12:28:59.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb01c0ca0e0 0x7fb01c1e4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb01c0cf160 0x7fb01c0ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb01c1538a0 0x7fb01c145d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c1e3480 con 0x7fb01c1538a0 2023-10-30T12:28:59.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c1e3cc0 con 0x7fb01c0cf160 2023-10-30T12:28:59.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb02286b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb01c0b1c00 con 0x7fb01c0ca0e0 2023-10-30T12:28:59.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb020e08640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb01c1538a0 0x7fb01c145d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb020e08640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb01c1538a0 0x7fb01c145d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33490/0 (socket says 172.21.15.125:33490) 2023-10-30T12:28:59.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.759+0000 7fb020e08640 1 -- 172.21.15.125:0/622576492 learned_addr learned my addr 172.21.15.125:0/622576492 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:28:59.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb01bfff640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb01c0ca0e0 0x7fb01c1e4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb01b7fe640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb01c0cf160 0x7fb01c0ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb020e08640 1 -- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb01c0ca0e0 msgr2=0x7fb01c1e4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb020e08640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb01c0ca0e0 0x7fb01c1e4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb020e08640 1 -- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb01c0cf160 msgr2=0x7fb01c0ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb020e08640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb01c0cf160 0x7fb01c0ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb020e08640 1 -- 172.21.15.125:0/622576492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb00c009420 con 0x7fb01c1538a0 2023-10-30T12:28:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb01b7fe640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb01c0cf160 0x7fb01c0ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:28:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb020e08640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb01c1538a0 0x7fb01c145d00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb01400caa0 tx=0x7fb01400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb018ff9640 1 -- 172.21.15.125:0/622576492 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01400ac00 con 0x7fb01c1538a0 2023-10-30T12:28:59.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb018ff9640 1 -- 172.21.15.125:0/622576492 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01400ad60 con 0x7fb01c1538a0 2023-10-30T12:28:59.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb018ff9640 1 -- 172.21.15.125:0/622576492 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb014012620 con 0x7fb01c1538a0 2023-10-30T12:28:59.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb02286b640 1 -- 172.21.15.125:0/622576492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01c0b1960 con 0x7fb01c1538a0 2023-10-30T12:28:59.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb02286b640 1 -- 172.21.15.125:0/622576492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01c1e3670 con 0x7fb01c1538a0 2023-10-30T12:28:59.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb02286b640 1 -- 172.21.15.125:0/622576492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01c1e3e70 con 0x7fb01c1538a0 2023-10-30T12:28:59.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.763+0000 7fb018ff9640 1 -- 172.21.15.125:0/622576492 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb014007070 con 0x7fb01c1538a0 2023-10-30T12:28:59.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.767+0000 7fb018ff9640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0080664a0 0x7fb008068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:28:59.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.767+0000 7fb018ff9640 1 -- 172.21.15.125:0/622576492 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb014083e00 con 0x7fb01c1538a0 2023-10-30T12:28:59.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.767+0000 7fb01bfff640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0080664a0 0x7fb008068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:28:59.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.767+0000 7fb01bfff640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0080664a0 0x7fb008068960 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb00c0b2000 tx=0x7fb00c0aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:28:59.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.771+0000 7fb018ff9640 1 -- 172.21.15.125:0/622576492 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb01c1e3e70 con 0x7fb01c1538a0 2023-10-30T12:28:59.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.919+0000 7fb02286b640 1 -- 172.21.15.125:0/622576492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb01c1e3e30 con 0x7fb01c1538a0 2023-10-30T12:28:59.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.919+0000 7fb018ff9640 1 -- 172.21.15.125:0/622576492 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16557 (secure 0 0 0) 0x7fb014017070 con 0x7fb01c1538a0 2023-10-30T12:28:59.924 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:28:59.924 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:45.927862+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:28:59.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.923+0000 7fb02286b640 1 -- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0080664a0 msgr2=0x7fb008068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.923+0000 7fb02286b640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0080664a0 0x7fb008068960 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb00c0b2000 tx=0x7fb00c0aa070 comp rx=0 tx=0).stop 2023-10-30T12:28:59.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.923+0000 7fb02286b640 1 -- 172.21.15.125:0/622576492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb01c1538a0 msgr2=0x7fb01c145d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:28:59.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.923+0000 7fb02286b640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb01c1538a0 0x7fb01c145d00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb01400caa0 tx=0x7fb01400cf60 comp rx=0 tx=0).stop 2023-10-30T12:28:59.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.923+0000 7fb02286b640 1 -- 172.21.15.125:0/622576492 shutdown_connections 2023-10-30T12:28:59.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.923+0000 7fb02286b640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0080664a0 0x7fb008068960 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.923+0000 7fb02286b640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb01c1538a0 0x7fb01c145d00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.923+0000 7fb02286b640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb01c0cf160 0x7fb01c0ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.923+0000 7fb02286b640 1 --2- 172.21.15.125:0/622576492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb01c0ca0e0 0x7fb01c1e4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:28:59.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.923+0000 7fb02286b640 1 -- 172.21.15.125:0/622576492 >> 172.21.15.125:0/622576492 conn(0x7fb01c06bfe0 msgr2=0x7fb01c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:28:59.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.923+0000 7fb02286b640 1 -- 172.21.15.125:0/622576492 shutdown_connections 2023-10-30T12:28:59.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:28:59.923+0000 7fb02286b640 1 -- 172.21.15.125:0/622576492 wait complete. 2023-10-30T12:28:59.946 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:31.063403+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T12:28:43.007849+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T12:28:43.401231+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:28:59.947 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:29:00.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.087+0000 7f9aae562640 1 Processor -- start 2023-10-30T12:29:00.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.087+0000 7f9aae562640 1 -- start start 2023-10-30T12:29:00.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aae562640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9aa806c200 0x7f9aa80c9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aae562640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9aa8159620 0x7f9aa8159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aae562640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa815d760 con 0x7f9aa8153ae0 2023-10-30T12:29:00.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aae562640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aa80c5df0 con 0x7f9aa806c200 2023-10-30T12:29:00.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aae562640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa80ca7c0 con 0x7f9aa8159620 2023-10-30T12:29:00.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aadd61640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9aa8159620 0x7f9aa8159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aacd5f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9aa806c200 0x7f9aa80c9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aadd61640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9aa8159620 0x7f9aa8159ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:39206/0 (socket says 172.21.15.125:39206) 2023-10-30T12:29:00.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aadd61640 1 -- 172.21.15.125:0/197443966 learned_addr learned my addr 172.21.15.125:0/197443966 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:00.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aad560640 1 --1- 172.21.15.125:0/197443966 >> v1:172.21.15.136:6789/0 conn(0x7f9aa8153ae0 0x7f9aa806baf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:37046/0 (socket says 172.21.15.125:37046) 2023-10-30T12:29:00.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aacd5f640 1 --2- 172.21.15.125:0/197443966 >> v2:172.21.15.136:3300/0 conn(0x7f9aa806c200 0x7f9aa80c9940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:39244/0 (socket says 172.21.15.125:39244) 2023-10-30T12:29:00.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aacd5f640 1 -- 172.21.15.125:0/197443966 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9aa8159620 msgr2=0x7f9aa8159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:00.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aacd5f640 1 --2- 172.21.15.125:0/197443966 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9aa8159620 0x7f9aa8159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aacd5f640 1 -- 172.21.15.125:0/197443966 >> v1:172.21.15.136:6789/0 conn(0x7f9aa8153ae0 legacy=0x7f9aa806baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aacd5f640 1 -- 172.21.15.125:0/197443966 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa80caaa0 con 0x7f9aa806c200 2023-10-30T12:29:00.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9aadd61640 1 --2- 172.21.15.125:0/197443966 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9aa8159620 0x7f9aa8159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:29:00.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.091+0000 7f9a9ffff640 1 -- 172.21.15.125:0/197443966 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1371871445 0 0) 0x7f9aa815d760 con 0x7f9aa8153ae0 2023-10-30T12:29:00.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.095+0000 7f9aacd5f640 1 --2- 172.21.15.125:0/197443966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9aa806c200 0x7f9aa80c9940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9a980097c0 tx=0x7f9a9801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58ef4f992ae9f0b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:00.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.095+0000 7f9a9ffff640 1 -- 172.21.15.125:0/197443966 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a980a8970 con 0x7f9aa806c200 2023-10-30T12:29:00.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.095+0000 7f9a9ffff640 1 -- 172.21.15.125:0/197443966 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a980a8ad0 con 0x7f9aa806c200 2023-10-30T12:29:00.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.095+0000 7f9a9ffff640 1 -- 172.21.15.125:0/197443966 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a980a8e60 con 0x7f9aa806c200 2023-10-30T12:29:00.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.095+0000 7f9aae562640 1 -- 172.21.15.125:0/197443966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9aa806c200 msgr2=0x7f9aa80c9940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.095+0000 7f9aae562640 1 --2- 172.21.15.125:0/197443966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9aa806c200 0x7f9aa80c9940 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9a980097c0 tx=0x7f9a9801ec70 comp rx=0 tx=0).stop 2023-10-30T12:29:00.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.095+0000 7f9aae562640 1 -- 172.21.15.125:0/197443966 shutdown_connections 2023-10-30T12:29:00.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.095+0000 7f9aae562640 1 --2- 172.21.15.125:0/197443966 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9aa8159620 0x7f9aa8159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.095+0000 7f9aae562640 1 --2- 172.21.15.125:0/197443966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9aa806c200 0x7f9aa80c9940 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.095+0000 7f9aae562640 1 -- 172.21.15.125:0/197443966 >> 172.21.15.125:0/197443966 conn(0x7f9aa80664a0 msgr2=0x7f9aa80cefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:00.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.095+0000 7f9aae562640 1 -- 172.21.15.125:0/197443966 shutdown_connections 2023-10-30T12:29:00.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.095+0000 7f9aae562640 1 -- 172.21.15.125:0/197443966 wait complete. 2023-10-30T12:29:00.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.095+0000 7f9aae562640 1 Processor -- start 2023-10-30T12:29:00.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.095+0000 7f9aae562640 1 -- start start 2023-10-30T12:29:00.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aae562640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9aa806c200 0x7f9aa815ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aae562640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9aa8153ae0 0x7f9aa815b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aae562640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9aa8159620 0x7f9aa80c87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aae562640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa814bbc0 con 0x7f9aa806c200 2023-10-30T12:29:00.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aae562640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa815a8b0 con 0x7f9aa8159620 2023-10-30T12:29:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aae562640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa80c6ef0 con 0x7f9aa8153ae0 2023-10-30T12:29:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aad560640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9aa806c200 0x7f9aa815ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aad560640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9aa806c200 0x7f9aa815ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33502/0 (socket says 172.21.15.125:33502) 2023-10-30T12:29:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aad560640 1 -- 172.21.15.125:0/4178383473 learned_addr learned my addr 172.21.15.125:0/4178383473 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:00.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aadd61640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9aa8159620 0x7f9aa80c87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aacd5f640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9aa8153ae0 0x7f9aa815b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aad560640 1 -- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9aa8153ae0 msgr2=0x7f9aa815b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aad560640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9aa8153ae0 0x7f9aa815b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aad560640 1 -- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9aa8159620 msgr2=0x7f9aa80c87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aad560640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9aa8159620 0x7f9aa80c87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aad560640 1 -- 172.21.15.125:0/4178383473 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa0009b30 con 0x7f9aa806c200 2023-10-30T12:29:00.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aacd5f640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9aa8153ae0 0x7f9aa815b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:00.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aad560640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9aa806c200 0x7f9aa815ad40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400c4a0 tx=0x7f9aa400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:00.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9a9dffb640 1 -- 172.21.15.125:0/4178383473 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa40074e0 con 0x7f9aa806c200 2023-10-30T12:29:00.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9a9dffb640 1 -- 172.21.15.125:0/4178383473 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa4007640 con 0x7f9aa806c200 2023-10-30T12:29:00.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9a9dffb640 1 -- 172.21.15.125:0/4178383473 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4004a30 con 0x7f9aa806c200 2023-10-30T12:29:00.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aae562640 1 -- 172.21.15.125:0/4178383473 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a98009420 con 0x7f9aa806c200 2023-10-30T12:29:00.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aae562640 1 -- 172.21.15.125:0/4178383473 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa80c74b0 con 0x7f9aa806c200 2023-10-30T12:29:00.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.099+0000 7f9aae562640 1 -- 172.21.15.125:0/4178383473 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa804e9c0 con 0x7f9aa806c200 2023-10-30T12:29:00.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.103+0000 7f9a9dffb640 1 -- 172.21.15.125:0/4178383473 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9aa4004b90 con 0x7f9aa806c200 2023-10-30T12:29:00.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.103+0000 7f9a9dffb640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9a8c0664d0 0x7f9a8c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.103+0000 7f9a9dffb640 1 -- 172.21.15.125:0/4178383473 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9aa40830c0 con 0x7f9aa806c200 2023-10-30T12:29:00.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.103+0000 7f9aacd5f640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9a8c0664d0 0x7f9a8c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.103+0000 7f9aacd5f640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9a8c0664d0 0x7f9a8c068990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9a980b2000 tx=0x7f9a980aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:00.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.107+0000 7f9a9dffb640 1 -- 172.21.15.125:0/4178383473 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9aa40504c0 con 0x7f9aa806c200 2023-10-30T12:29:00.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.259+0000 7f9aae562640 1 -- 172.21.15.125:0/4178383473 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9aa806cc50 con 0x7f9aa806c200 2023-10-30T12:29:00.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.259+0000 7f9a9dffb640 1 -- 172.21.15.125:0/4178383473 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9aa4053ae0 con 0x7f9aa806c200 2023-10-30T12:29:00.263 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2023-10-30T12:29:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.263+0000 7f9aae562640 1 -- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9a8c0664d0 msgr2=0x7f9a8c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.263+0000 7f9aae562640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9a8c0664d0 0x7f9a8c068990 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9a980b2000 tx=0x7f9a980aa070 comp rx=0 tx=0).stop 2023-10-30T12:29:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.263+0000 7f9aae562640 1 -- 172.21.15.125:0/4178383473 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9aa806c200 msgr2=0x7f9aa815ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.263+0000 7f9aae562640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9aa806c200 0x7f9aa815ad40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9aa400c4a0 tx=0x7f9aa400c960 comp rx=0 tx=0).stop 2023-10-30T12:29:00.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.263+0000 7f9aae562640 1 -- 172.21.15.125:0/4178383473 shutdown_connections 2023-10-30T12:29:00.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.263+0000 7f9aae562640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9a8c0664d0 0x7f9a8c068990 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.263+0000 7f9aae562640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9aa8159620 0x7f9aa80c87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.263+0000 7f9aae562640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9aa8153ae0 0x7f9aa815b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.263+0000 7f9aae562640 1 --2- 172.21.15.125:0/4178383473 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9aa806c200 0x7f9aa815ad40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.263+0000 7f9aae562640 1 -- 172.21.15.125:0/4178383473 >> 172.21.15.125:0/4178383473 conn(0x7f9aa80664a0 msgr2=0x7f9aa8159a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:00.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.263+0000 7f9aae562640 1 -- 172.21.15.125:0/4178383473 shutdown_connections 2023-10-30T12:29:00.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.263+0000 7f9aae562640 1 -- 172.21.15.125:0/4178383473 wait complete. 2023-10-30T12:29:00.284 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T12:29:00.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.423+0000 7f7c6fbd3640 1 Processor -- start 2023-10-30T12:29:00.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.423+0000 7f7c6fbd3640 1 -- start start 2023-10-30T12:29:00.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.423+0000 7f7c6fbd3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7c68159930 0x7f7c680cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.423+0000 7f7c6fbd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c680cf6e0 0x7f7c6815d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.423+0000 7f7c6fbd3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c6815a920 con 0x7f7c68153980 2023-10-30T12:29:00.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.423+0000 7f7c6fbd3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c680cb930 con 0x7f7c68159930 2023-10-30T12:29:00.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.423+0000 7f7c6fbd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68159dc0 con 0x7f7c680cf6e0 2023-10-30T12:29:00.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.423+0000 7f7c6d96f640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7c68153980 0x7f7c680caca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54234/0 (socket says 172.21.15.125:54234) 2023-10-30T12:29:00.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.423+0000 7f7c6d96f640 1 -- 172.21.15.125:0/278251027 learned_addr learned my addr 172.21.15.125:0/278251027 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:29:00.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.423+0000 7f7c6e170640 1 --2- 172.21.15.125:0/278251027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c680cf6e0 0x7f7c6815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.423+0000 7f7c6d16e640 1 --2- 172.21.15.125:0/278251027 >> v2:172.21.15.136:3300/0 conn(0x7f7c68159930 0x7f7c680cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.423+0000 7f7c6c96d640 1 -- 172.21.15.125:0/278251027 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1983779688 0 0) 0x7f7c6815a920 con 0x7f7c68153980 2023-10-30T12:29:00.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.423+0000 7f7c6c96d640 1 -- 172.21.15.125:0/278251027 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c54003530 con 0x7f7c68153980 2023-10-30T12:29:00.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6c96d640 1 -- 172.21.15.125:0/278251027 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 201447883 0 0) 0x7f7c54003530 con 0x7f7c68153980 2023-10-30T12:29:00.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6c96d640 1 -- 172.21.15.125:0/278251027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c680cf6e0 msgr2=0x7f7c6815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:00.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6c96d640 1 --2- 172.21.15.125:0/278251027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c680cf6e0 0x7f7c6815d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6c96d640 1 -- 172.21.15.125:0/278251027 >> v2:172.21.15.136:3300/0 conn(0x7f7c68159930 msgr2=0x7f7c680cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:00.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6c96d640 1 --2- 172.21.15.125:0/278251027 >> v2:172.21.15.136:3300/0 conn(0x7f7c68159930 0x7f7c680cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6c96d640 1 -- 172.21.15.125:0/278251027 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6815a080 con 0x7f7c68153980 2023-10-30T12:29:00.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6c96d640 1 -- 172.21.15.125:0/278251027 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f7c64001680 con 0x7f7c68153980 2023-10-30T12:29:00.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6c96d640 1 -- 172.21.15.125:0/278251027 >> v1:172.21.15.125:6789/0 conn(0x7f7c68153980 legacy=0x7f7c680caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6c96d640 1 --2- 172.21.15.125:0/278251027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c54003980 0x7f7c54003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6c96d640 1 --2- 172.21.15.125:0/278251027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c540047d0 0x7f7c54004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6c96d640 1 --2- 172.21.15.125:0/278251027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c680cf6e0 0x7f7c54005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6c96d640 1 -- 172.21.15.125:0/278251027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c680cb930 con 0x7f7c540047d0 2023-10-30T12:29:00.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6c96d640 1 -- 172.21.15.125:0/278251027 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68159dc0 con 0x7f7c54003980 2023-10-30T12:29:00.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6c96d640 1 -- 172.21.15.125:0/278251027 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5400b990 con 0x7f7c680cf6e0 2023-10-30T12:29:00.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6d16e640 1 --2- 172.21.15.125:0/278251027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c540047d0 0x7f7c54004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6d96f640 1 --2- 172.21.15.125:0/278251027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c54003980 0x7f7c54003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6e170640 1 --2- 172.21.15.125:0/278251027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c680cf6e0 0x7f7c54005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6d16e640 1 -- 172.21.15.125:0/278251027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c680cf6e0 msgr2=0x7f7c54005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:00.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6d16e640 1 --2- 172.21.15.125:0/278251027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c680cf6e0 0x7f7c54005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6d16e640 1 -- 172.21.15.125:0/278251027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c54003980 msgr2=0x7f7c54003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:00.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6d16e640 1 --2- 172.21.15.125:0/278251027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c54003980 0x7f7c54003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6d16e640 1 -- 172.21.15.125:0/278251027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c54003530 con 0x7f7c540047d0 2023-10-30T12:29:00.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6e170640 1 --2- 172.21.15.125:0/278251027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c680cf6e0 0x7f7c54005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:00.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.427+0000 7f7c6d16e640 1 --2- 172.21.15.125:0/278251027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c540047d0 0x7f7c54004c50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7c58009ae0 tx=0x7f7c5801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f65782817d17ccc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:00.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.431+0000 7f7c6c96d640 1 -- 172.21.15.125:0/278251027 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c580a64a0 con 0x7f7c540047d0 2023-10-30T12:29:00.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.431+0000 7f7c6c96d640 1 -- 172.21.15.125:0/278251027 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c580a6600 con 0x7f7c540047d0 2023-10-30T12:29:00.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.431+0000 7f7c6c96d640 1 -- 172.21.15.125:0/278251027 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c580a69b0 con 0x7f7c540047d0 2023-10-30T12:29:00.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.431+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/278251027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c540047d0 msgr2=0x7f7c54004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.431+0000 7f7c6fbd3640 1 --2- 172.21.15.125:0/278251027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c540047d0 0x7f7c54004c50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7c58009ae0 tx=0x7f7c5801ca90 comp rx=0 tx=0).stop 2023-10-30T12:29:00.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.431+0000 7f7c6e170640 1 -- 172.21.15.125:0/278251027 reap_dead start 2023-10-30T12:29:00.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.431+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/278251027 shutdown_connections 2023-10-30T12:29:00.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.431+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/278251027 >> 172.21.15.125:0/278251027 conn(0x7f7c6806c0f0 msgr2=0x7f7c6806c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:00.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.431+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/278251027 shutdown_connections 2023-10-30T12:29:00.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.431+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/278251027 wait complete. 2023-10-30T12:29:00.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.431+0000 7f7c6fbd3640 1 Processor -- start 2023-10-30T12:29:00.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6fbd3640 1 -- start start 2023-10-30T12:29:00.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6fbd3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c540047d0 0x7f7c680cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6fbd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c680cec90 0x7f7c681f2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6fbd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c68151030 0x7f7c6815c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6fbd3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c681f2820 con 0x7f7c540047d0 2023-10-30T12:29:00.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6fbd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6814cd30 con 0x7f7c68151030 2023-10-30T12:29:00.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6fbd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c68152d10 con 0x7f7c680cec90 2023-10-30T12:29:00.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6d96f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c540047d0 0x7f7c680cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6d96f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c540047d0 0x7f7c680cd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33520/0 (socket says 172.21.15.125:33520) 2023-10-30T12:29:00.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6d96f640 1 -- 172.21.15.125:0/706434900 learned_addr learned my addr 172.21.15.125:0/706434900 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:00.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6d16e640 1 --2- 172.21.15.125:0/706434900 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c680cec90 0x7f7c681f2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6e170640 1 --2- 172.21.15.125:0/706434900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c68151030 0x7f7c6815c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6d96f640 1 -- 172.21.15.125:0/706434900 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c680cec90 msgr2=0x7f7c681f2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6d96f640 1 --2- 172.21.15.125:0/706434900 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c680cec90 0x7f7c681f2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6d96f640 1 -- 172.21.15.125:0/706434900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c68151030 msgr2=0x7f7c6815c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6d96f640 1 --2- 172.21.15.125:0/706434900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c68151030 0x7f7c6815c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6d96f640 1 -- 172.21.15.125:0/706434900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c68152ff0 con 0x7f7c540047d0 2023-10-30T12:29:00.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6d96f640 1 --2- 172.21.15.125:0/706434900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c540047d0 0x7f7c680cd980 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7c64011790 tx=0x7f7c64011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:00.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c5e7fc640 1 -- 172.21.15.125:0/706434900 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6400fae0 con 0x7f7c540047d0 2023-10-30T12:29:00.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c5e7fc640 1 -- 172.21.15.125:0/706434900 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6400fc80 con 0x7f7c540047d0 2023-10-30T12:29:00.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c5e7fc640 1 -- 172.21.15.125:0/706434900 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c64011e40 con 0x7f7c540047d0 2023-10-30T12:29:00.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/706434900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c681f2440 con 0x7f7c540047d0 2023-10-30T12:29:00.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.435+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/706434900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c681f3180 con 0x7f7c540047d0 2023-10-30T12:29:00.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.439+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/706434900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c68153230 con 0x7f7c540047d0 2023-10-30T12:29:00.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.439+0000 7f7c5e7fc640 1 -- 172.21.15.125:0/706434900 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7c6401f460 con 0x7f7c540047d0 2023-10-30T12:29:00.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.439+0000 7f7c5e7fc640 1 --2- 172.21.15.125:0/706434900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7c4c0664a0 0x7f7c4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.439+0000 7f7c5e7fc640 1 -- 172.21.15.125:0/706434900 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7c64086f30 con 0x7f7c540047d0 2023-10-30T12:29:00.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.439+0000 7f7c6d16e640 1 --2- 172.21.15.125:0/706434900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7c4c0664a0 0x7f7c4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.439+0000 7f7c6d16e640 1 --2- 172.21.15.125:0/706434900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7c4c0664a0 0x7f7c4c068960 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7c580ad000 tx=0x7f7c580b3040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:00.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.443+0000 7f7c5e7fc640 1 -- 172.21.15.125:0/706434900 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7c68153230 con 0x7f7c540047d0 2023-10-30T12:29:00.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.591+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/706434900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7c68153230 con 0x7f7c540047d0 2023-10-30T12:29:00.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.595+0000 7f7c5e7fc640 1 -- 172.21.15.125:0/706434900 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7c68153230 con 0x7f7c540047d0 2023-10-30T12:29:00.597 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2023-10-30T12:29:00.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.595+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/706434900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7c4c0664a0 msgr2=0x7f7c4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.595+0000 7f7c6fbd3640 1 --2- 172.21.15.125:0/706434900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7c4c0664a0 0x7f7c4c068960 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7c580ad000 tx=0x7f7c580b3040 comp rx=0 tx=0).stop 2023-10-30T12:29:00.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.595+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/706434900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c540047d0 msgr2=0x7f7c680cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.595+0000 7f7c6fbd3640 1 --2- 172.21.15.125:0/706434900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c540047d0 0x7f7c680cd980 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7c64011790 tx=0x7f7c64011c50 comp rx=0 tx=0).stop 2023-10-30T12:29:00.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.595+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/706434900 shutdown_connections 2023-10-30T12:29:00.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.595+0000 7f7c6fbd3640 1 --2- 172.21.15.125:0/706434900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7c4c0664a0 0x7f7c4c068960 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.595+0000 7f7c6fbd3640 1 --2- 172.21.15.125:0/706434900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7c68151030 0x7f7c6815c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.595+0000 7f7c6fbd3640 1 --2- 172.21.15.125:0/706434900 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7c680cec90 0x7f7c681f2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.595+0000 7f7c6fbd3640 1 --2- 172.21.15.125:0/706434900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7c540047d0 0x7f7c680cd980 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.595+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/706434900 >> 172.21.15.125:0/706434900 conn(0x7f7c6806c0f0 msgr2=0x7f7c680caca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:00.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.595+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/706434900 shutdown_connections 2023-10-30T12:29:00.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.595+0000 7f7c6fbd3640 1 -- 172.21.15.125:0/706434900 wait complete. 2023-10-30T12:29:00.617 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T12:29:00.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.755+0000 7fb575e09640 1 Processor -- start 2023-10-30T12:29:00.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.755+0000 7fb575e09640 1 -- start start 2023-10-30T12:29:00.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.759+0000 7fb575e09640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb570159930 0x7fb5700cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.759+0000 7fb575e09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5700cf6e0 0x7fb57015d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.759+0000 7fb575e09640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57015a920 con 0x7fb570153980 2023-10-30T12:29:00.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.759+0000 7fb575e09640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5700cb930 con 0x7fb570159930 2023-10-30T12:29:00.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.759+0000 7fb575e09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb570159dc0 con 0x7fb5700cf6e0 2023-10-30T12:29:00.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.759+0000 7fb56effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb570159930 0x7fb5700cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.759+0000 7fb56effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb570159930 0x7fb5700cf1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54538/0 (socket says 172.21.15.125:54538) 2023-10-30T12:29:00.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.759+0000 7fb56effd640 1 -- 172.21.15.125:0/3029951989 learned_addr learned my addr 172.21.15.125:0/3029951989 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:00.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.759+0000 7fb56ffff640 1 --2- 172.21.15.125:0/3029951989 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5700cf6e0 0x7fb57015d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.759+0000 7fb56effd640 1 -- 172.21.15.125:0/3029951989 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5700cf6e0 msgr2=0x7fb57015d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:00.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.759+0000 7fb56effd640 1 --2- 172.21.15.125:0/3029951989 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5700cf6e0 0x7fb57015d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.759+0000 7fb56effd640 1 -- 172.21.15.125:0/3029951989 >> v1:172.21.15.136:6789/0 conn(0x7fb570153980 legacy=0x7fb5700caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.759+0000 7fb56effd640 1 -- 172.21.15.125:0/3029951989 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57015a080 con 0x7fb570159930 2023-10-30T12:29:00.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb56effd640 1 --2- 172.21.15.125:0/3029951989 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb570159930 0x7fb5700cf1a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb564009870 tx=0x7fb56401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc14b3ac15f0558b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:00.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb56e7fc640 1 -- 172.21.15.125:0/3029951989 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5640ab620 con 0x7fb570159930 2023-10-30T12:29:00.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb56e7fc640 1 -- 172.21.15.125:0/3029951989 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5640ab780 con 0x7fb570159930 2023-10-30T12:29:00.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb56e7fc640 1 -- 172.21.15.125:0/3029951989 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5640abb10 con 0x7fb570159930 2023-10-30T12:29:00.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb575e09640 1 -- 172.21.15.125:0/3029951989 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb570159930 msgr2=0x7fb5700cf1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb575e09640 1 --2- 172.21.15.125:0/3029951989 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb570159930 0x7fb5700cf1a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fb564009870 tx=0x7fb56401c510 comp rx=0 tx=0).stop 2023-10-30T12:29:00.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb575e09640 1 -- 172.21.15.125:0/3029951989 shutdown_connections 2023-10-30T12:29:00.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb575e09640 1 --2- 172.21.15.125:0/3029951989 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5700cf6e0 0x7fb57015d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb575e09640 1 --2- 172.21.15.125:0/3029951989 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb570159930 0x7fb5700cf1a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb575e09640 1 -- 172.21.15.125:0/3029951989 >> 172.21.15.125:0/3029951989 conn(0x7fb57006c0f0 msgr2=0x7fb57006c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:00.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb575e09640 1 -- 172.21.15.125:0/3029951989 shutdown_connections 2023-10-30T12:29:00.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb575e09640 1 -- 172.21.15.125:0/3029951989 wait complete. 2023-10-30T12:29:00.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb575e09640 1 Processor -- start 2023-10-30T12:29:00.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb575e09640 1 -- start start 2023-10-30T12:29:00.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.763+0000 7fb575e09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5700cf6e0 0x7fb570152460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb575e09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb570153980 0x7fb5700b4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb575e09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5700b4e90 0x7fb5700b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb575e09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57015be70 con 0x7fb5700cf6e0 2023-10-30T12:29:00.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb575e09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb570145690 con 0x7fb570153980 2023-10-30T12:29:00.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb575e09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57014e880 con 0x7fb5700b4e90 2023-10-30T12:29:00.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb56f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5700cf6e0 0x7fb570152460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb56f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5700cf6e0 0x7fb570152460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54540/0 (socket says 172.21.15.125:54540) 2023-10-30T12:29:00.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb56f7fe640 1 -- 172.21.15.125:0/1705406143 learned_addr learned my addr 172.21.15.125:0/1705406143 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:00.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb56effd640 1 --2- 172.21.15.125:0/1705406143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb570153980 0x7fb5700b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb56ffff640 1 --2- 172.21.15.125:0/1705406143 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5700b4e90 0x7fb5700b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb56f7fe640 1 -- 172.21.15.125:0/1705406143 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5700b4e90 msgr2=0x7fb5700b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb56f7fe640 1 --2- 172.21.15.125:0/1705406143 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5700b4e90 0x7fb5700b38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb56f7fe640 1 -- 172.21.15.125:0/1705406143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb570153980 msgr2=0x7fb5700b4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb56f7fe640 1 --2- 172.21.15.125:0/1705406143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb570153980 0x7fb5700b4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb56f7fe640 1 -- 172.21.15.125:0/1705406143 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5640094d0 con 0x7fb5700cf6e0 2023-10-30T12:29:00.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb56f7fe640 1 --2- 172.21.15.125:0/1705406143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5700cf6e0 0x7fb570152460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb56000c500 tx=0x7fb56000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:00.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb557fff640 1 -- 172.21.15.125:0/1705406143 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560005220 con 0x7fb5700cf6e0 2023-10-30T12:29:00.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb557fff640 1 -- 172.21.15.125:0/1705406143 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb560005380 con 0x7fb5700cf6e0 2023-10-30T12:29:00.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb557fff640 1 -- 172.21.15.125:0/1705406143 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb560007270 con 0x7fb5700cf6e0 2023-10-30T12:29:00.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb575e09640 1 -- 172.21.15.125:0/1705406143 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57015cbf0 con 0x7fb5700cf6e0 2023-10-30T12:29:00.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb575e09640 1 -- 172.21.15.125:0/1705406143 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5701529a0 con 0x7fb5700cf6e0 2023-10-30T12:29:00.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb575e09640 1 -- 172.21.15.125:0/1705406143 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57015ca50 con 0x7fb5700cf6e0 2023-10-30T12:29:00.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.767+0000 7fb557fff640 1 -- 172.21.15.125:0/1705406143 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb5600073d0 con 0x7fb5700cf6e0 2023-10-30T12:29:00.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.771+0000 7fb557fff640 1 --2- 172.21.15.125:0/1705406143 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb55c0664a0 0x7fb55c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:00.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.771+0000 7fb557fff640 1 -- 172.21.15.125:0/1705406143 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb560083760 con 0x7fb5700cf6e0 2023-10-30T12:29:00.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.771+0000 7fb56effd640 1 --2- 172.21.15.125:0/1705406143 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb55c0664a0 0x7fb55c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:00.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.771+0000 7fb56effd640 1 --2- 172.21.15.125:0/1705406143 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb55c0664a0 0x7fb55c068960 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb5640af000 tx=0x7fb5640a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:00.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.775+0000 7fb557fff640 1 -- 172.21.15.125:0/1705406143 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb57015ca50 con 0x7fb5700cf6e0 2023-10-30T12:29:00.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.923+0000 7fb575e09640 1 -- 172.21.15.125:0/1705406143 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb57015ca50 con 0x7fb5700cf6e0 2023-10-30T12:29:00.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.923+0000 7fb557fff640 1 -- 172.21.15.125:0/1705406143 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb57015ca50 con 0x7fb5700cf6e0 2023-10-30T12:29:00.926 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2023-10-30T12:29:00.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.923+0000 7fb575e09640 1 -- 172.21.15.125:0/1705406143 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb55c0664a0 msgr2=0x7fb55c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.923+0000 7fb575e09640 1 --2- 172.21.15.125:0/1705406143 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb55c0664a0 0x7fb55c068960 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb5640af000 tx=0x7fb5640a7000 comp rx=0 tx=0).stop 2023-10-30T12:29:00.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.923+0000 7fb575e09640 1 -- 172.21.15.125:0/1705406143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5700cf6e0 msgr2=0x7fb570152460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:00.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.923+0000 7fb575e09640 1 --2- 172.21.15.125:0/1705406143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5700cf6e0 0x7fb570152460 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb56000c500 tx=0x7fb56000c9c0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.923+0000 7fb575e09640 1 -- 172.21.15.125:0/1705406143 shutdown_connections 2023-10-30T12:29:00.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.923+0000 7fb575e09640 1 --2- 172.21.15.125:0/1705406143 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb55c0664a0 0x7fb55c068960 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.923+0000 7fb575e09640 1 --2- 172.21.15.125:0/1705406143 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5700b4e90 0x7fb5700b38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.923+0000 7fb575e09640 1 --2- 172.21.15.125:0/1705406143 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb570153980 0x7fb5700b4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.923+0000 7fb575e09640 1 --2- 172.21.15.125:0/1705406143 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5700cf6e0 0x7fb570152460 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:00.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.923+0000 7fb575e09640 1 -- 172.21.15.125:0/1705406143 >> 172.21.15.125:0/1705406143 conn(0x7fb57006c0f0 msgr2=0x7fb570065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:00.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.923+0000 7fb575e09640 1 -- 172.21.15.125:0/1705406143 shutdown_connections 2023-10-30T12:29:00.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:00.923+0000 7fb575e09640 1 -- 172.21.15.125:0/1705406143 wait complete. 2023-10-30T12:29:00.948 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:29:01.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e9aa46640 1 Processor -- start 2023-10-30T12:29:01.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e9aa46640 1 -- start start 2023-10-30T12:29:01.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e9aa46640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7e94071110 0x7f7e9406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e9aa46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e94066cf0 con 0x7f7e940c6bd0 2023-10-30T12:29:01.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e9aa46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e940c70d0 con 0x7f7e9406f7a0 2023-10-30T12:29:01.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e9aa46640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e940700c0 con 0x7f7e94071110 2023-10-30T12:29:01.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e937fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7e940c6bd0 0x7f7e94158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35132/0 (socket says 172.21.15.125:35132) 2023-10-30T12:29:01.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e937fe640 1 -- 172.21.15.125:0/1339810080 learned_addr learned my addr 172.21.15.125:0/1339810080 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:29:01.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e98fe3640 1 --2- 172.21.15.125:0/1339810080 >> v2:172.21.15.136:3300/0 conn(0x7f7e94071110 0x7f7e9406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666910611 0 0) 0x7f7e94066cf0 con 0x7f7e940c6bd0 2023-10-30T12:29:01.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e7c003530 con 0x7f7e940c6bd0 2023-10-30T12:29:01.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1229930181 0 0) 0x7f7e940c70d0 con 0x7f7e9406f7a0 2023-10-30T12:29:01.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e94066cf0 con 0x7f7e9406f7a0 2023-10-30T12:29:01.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3697324324 0 0) 0x7f7e7c003530 con 0x7f7e940c6bd0 2023-10-30T12:29:01.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 >> v2:172.21.15.136:3300/0 conn(0x7f7e94071110 msgr2=0x7f7e9406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 --2- 172.21.15.125:0/1339810080 >> v2:172.21.15.136:3300/0 conn(0x7f7e94071110 0x7f7e9406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 >> v1:172.21.15.136:6789/0 conn(0x7f7e9406f7a0 legacy=0x7f7e94069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e940c7390 con 0x7f7e940c6bd0 2023-10-30T12:29:01.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f7e88001680 con 0x7f7e940c6bd0 2023-10-30T12:29:01.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 >> v1:172.21.15.125:6789/0 conn(0x7f7e940c6bd0 legacy=0x7f7e94158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 --2- 172.21.15.125:0/1339810080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e7c003b10 0x7f7e7c003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 --2- 172.21.15.125:0/1339810080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e7c0047d0 0x7f7e7c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 --2- 172.21.15.125:0/1339810080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e7c005190 0x7f7e7c00fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e940700c0 con 0x7f7e7c005190 2023-10-30T12:29:01.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c010390 con 0x7f7e7c0047d0 2023-10-30T12:29:01.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7c010550 con 0x7f7e7c003b10 2023-10-30T12:29:01.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.091+0000 7f7e98fe3640 1 --2- 172.21.15.125:0/1339810080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e7c005190 0x7f7e7c00fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e93fff640 1 --2- 172.21.15.125:0/1339810080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e7c003b10 0x7f7e7c003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e937fe640 1 --2- 172.21.15.125:0/1339810080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e7c0047d0 0x7f7e7c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e98fe3640 1 -- 172.21.15.125:0/1339810080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e7c003b10 msgr2=0x7f7e7c003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e98fe3640 1 --2- 172.21.15.125:0/1339810080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e7c003b10 0x7f7e7c003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e98fe3640 1 -- 172.21.15.125:0/1339810080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e7c0047d0 msgr2=0x7f7e7c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e98fe3640 1 --2- 172.21.15.125:0/1339810080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e7c0047d0 0x7f7e7c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e98fe3640 1 -- 172.21.15.125:0/1339810080 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c003530 con 0x7f7e7c005190 2023-10-30T12:29:01.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e937fe640 1 --2- 172.21.15.125:0/1339810080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e7c0047d0 0x7f7e7c004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:01.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e98fe3640 1 -- 172.21.15.125:0/1339810080 reap_dead start 2023-10-30T12:29:01.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e98fe3640 1 --2- 172.21.15.125:0/1339810080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e7c005190 0x7f7e7c00fda0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009f90 tx=0x7f7e8c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9ecb2b25dec1a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c0a5c70 con 0x7f7e7c005190 2023-10-30T12:29:01.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8c00f040 con 0x7f7e7c005190 2023-10-30T12:29:01.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e92ffd640 1 -- 172.21.15.125:0/1339810080 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8c0039a0 con 0x7f7e7c005190 2023-10-30T12:29:01.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e9aa46640 1 -- 172.21.15.125:0/1339810080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e7c005190 msgr2=0x7f7e7c00fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e9aa46640 1 --2- 172.21.15.125:0/1339810080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e7c005190 0x7f7e7c00fda0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7e8c009f90 tx=0x7f7e8c01cc60 comp rx=0 tx=0).stop 2023-10-30T12:29:01.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e9aa46640 1 -- 172.21.15.125:0/1339810080 shutdown_connections 2023-10-30T12:29:01.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e9aa46640 1 --2- 172.21.15.125:0/1339810080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e7c005190 0x7f7e7c00fda0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.095+0000 7f7e9aa46640 1 -- 172.21.15.125:0/1339810080 >> 172.21.15.125:0/1339810080 conn(0x7f7e940c7910 msgr2=0x7f7e940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e9aa46640 1 -- 172.21.15.125:0/1339810080 shutdown_connections 2023-10-30T12:29:01.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e9aa46640 1 -- 172.21.15.125:0/1339810080 wait complete. 2023-10-30T12:29:01.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e9aa46640 1 Processor -- start 2023-10-30T12:29:01.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e9aa46640 1 -- start start 2023-10-30T12:29:01.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e9aa46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e7c005190 0x7f7e9414ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e9aa46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e94153bd0 0x7f7e9414bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e9aa46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e9414c420 0x7f7e940c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e9aa46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e941488d0 con 0x7f7e94153bd0 2023-10-30T12:29:01.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e9aa46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e941edb30 con 0x7f7e9414c420 2023-10-30T12:29:01.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e9aa46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e941eeb00 con 0x7f7e7c005190 2023-10-30T12:29:01.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e937fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e94153bd0 0x7f7e9414bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e937fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e94153bd0 0x7f7e9414bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54566/0 (socket says 172.21.15.125:54566) 2023-10-30T12:29:01.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e937fe640 1 -- 172.21.15.125:0/2401182705 learned_addr learned my addr 172.21.15.125:0/2401182705 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:01.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e98fe3640 1 --2- 172.21.15.125:0/2401182705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e9414c420 0x7f7e940c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e93fff640 1 --2- 172.21.15.125:0/2401182705 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e7c005190 0x7f7e9414ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e937fe640 1 -- 172.21.15.125:0/2401182705 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e7c005190 msgr2=0x7f7e9414ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e937fe640 1 --2- 172.21.15.125:0/2401182705 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e7c005190 0x7f7e9414ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e937fe640 1 -- 172.21.15.125:0/2401182705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e9414c420 msgr2=0x7f7e940c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e937fe640 1 --2- 172.21.15.125:0/2401182705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e9414c420 0x7f7e940c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e937fe640 1 -- 172.21.15.125:0/2401182705 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e941eb370 con 0x7f7e94153bd0 2023-10-30T12:29:01.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.099+0000 7f7e937fe640 1 --2- 172.21.15.125:0/2401182705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e94153bd0 0x7f7e9414bee0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800e7e0 tx=0x7f7e8800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.103+0000 7f7e90ff9640 1 -- 172.21.15.125:0/2401182705 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8800c660 con 0x7f7e94153bd0 2023-10-30T12:29:01.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.103+0000 7f7e90ff9640 1 -- 172.21.15.125:0/2401182705 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e8800c7c0 con 0x7f7e94153bd0 2023-10-30T12:29:01.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.103+0000 7f7e9aa46640 1 -- 172.21.15.125:0/2401182705 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e941edd40 con 0x7f7e94153bd0 2023-10-30T12:29:01.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.103+0000 7f7e90ff9640 1 -- 172.21.15.125:0/2401182705 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e8800cb30 con 0x7f7e94153bd0 2023-10-30T12:29:01.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.103+0000 7f7e9aa46640 1 -- 172.21.15.125:0/2401182705 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e941ee300 con 0x7f7e94153bd0 2023-10-30T12:29:01.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.103+0000 7f7e9aa46640 1 -- 172.21.15.125:0/2401182705 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e9404e170 con 0x7f7e94153bd0 2023-10-30T12:29:01.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.103+0000 7f7e90ff9640 1 -- 172.21.15.125:0/2401182705 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7e8800cc90 con 0x7f7e94153bd0 2023-10-30T12:29:01.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.103+0000 7f7e90ff9640 1 --2- 172.21.15.125:0/2401182705 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7e800664d0 0x7f7e80068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.103+0000 7f7e90ff9640 1 -- 172.21.15.125:0/2401182705 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7e88083a90 con 0x7f7e94153bd0 2023-10-30T12:29:01.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.103+0000 7f7e93fff640 1 --2- 172.21.15.125:0/2401182705 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7e800664d0 0x7f7e80068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.107+0000 7f7e93fff640 1 --2- 172.21.15.125:0/2401182705 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7e800664d0 0x7f7e80068990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7e84007860 tx=0x7f7e84009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.107+0000 7f7e90ff9640 1 -- 172.21.15.125:0/2401182705 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7e88050de0 con 0x7f7e94153bd0 2023-10-30T12:29:01.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.287+0000 7f7e9aa46640 1 -- 172.21.15.125:0/2401182705 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e940c3a00 con 0x7f7e94153bd0 2023-10-30T12:29:01.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.287+0000 7f7e90ff9640 1 -- 172.21.15.125:0/2401182705 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+6539 (secure 0 0 0) 0x7f7e88054400 con 0x7f7e94153bd0 2023-10-30T12:29:01.290 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:29:01.290 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:28:55.942137+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.136:6835/3346501886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3346501886},{"type":"v1","addr":"172.21.15.136:6835","nonce":3346501886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:29:01.291 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 11 2023-10-30T12:29:01.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.287+0000 7f7e9aa46640 1 -- 172.21.15.125:0/2401182705 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7e800664d0 msgr2=0x7f7e80068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.287+0000 7f7e9aa46640 1 --2- 172.21.15.125:0/2401182705 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7e800664d0 0x7f7e80068990 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7e84007860 tx=0x7f7e84009040 comp rx=0 tx=0).stop 2023-10-30T12:29:01.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.287+0000 7f7e9aa46640 1 -- 172.21.15.125:0/2401182705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e94153bd0 msgr2=0x7f7e9414bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.287+0000 7f7e9aa46640 1 --2- 172.21.15.125:0/2401182705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e94153bd0 0x7f7e9414bee0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7e8800e7e0 tx=0x7f7e8800eca0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.287+0000 7f7e9aa46640 1 -- 172.21.15.125:0/2401182705 shutdown_connections 2023-10-30T12:29:01.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.287+0000 7f7e9aa46640 1 --2- 172.21.15.125:0/2401182705 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7e800664d0 0x7f7e80068990 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.287+0000 7f7e9aa46640 1 --2- 172.21.15.125:0/2401182705 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e9414c420 0x7f7e940c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.287+0000 7f7e9aa46640 1 --2- 172.21.15.125:0/2401182705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e94153bd0 0x7f7e9414bee0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.287+0000 7f7e9aa46640 1 --2- 172.21.15.125:0/2401182705 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e7c005190 0x7f7e9414ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.287+0000 7f7e9aa46640 1 -- 172.21.15.125:0/2401182705 >> 172.21.15.125:0/2401182705 conn(0x7f7e940c7910 msgr2=0x7f7e940c41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.287+0000 7f7e9aa46640 1 -- 172.21.15.125:0/2401182705 shutdown_connections 2023-10-30T12:29:01.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.287+0000 7f7e9aa46640 1 -- 172.21.15.125:0/2401182705 wait complete. 2023-10-30T12:29:01.311 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:29:01.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927da3c640 1 Processor -- start 2023-10-30T12:29:01.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927da3c640 1 -- start start 2023-10-30T12:29:01.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927da3c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f92781538f0 0x7f9278159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927da3c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f92780cefa0 0x7f92780cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927da3c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92780ca6a0 con 0x7f927815cd50 2023-10-30T12:29:01.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927da3c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92780cf8f0 con 0x7f92781538f0 2023-10-30T12:29:01.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927da3c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f927815d300 con 0x7f92780cefa0 2023-10-30T12:29:01.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927ca3a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f92781538f0 0x7f9278159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927ca3a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f92781538f0 0x7f9278159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54574/0 (socket says 172.21.15.125:54574) 2023-10-30T12:29:01.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927ca3a640 1 -- 172.21.15.125:0/329441249 learned_addr learned my addr 172.21.15.125:0/329441249 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:01.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927d23b640 1 --2- 172.21.15.125:0/329441249 >> v2:172.21.15.136:3300/0 conn(0x7f92780cefa0 0x7f92780cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927ca3a640 1 -- 172.21.15.125:0/329441249 >> v2:172.21.15.136:3300/0 conn(0x7f92780cefa0 msgr2=0x7f92780cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927ca3a640 1 --2- 172.21.15.125:0/329441249 >> v2:172.21.15.136:3300/0 conn(0x7f92780cefa0 0x7f92780cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927ca3a640 1 -- 172.21.15.125:0/329441249 >> v1:172.21.15.136:6789/0 conn(0x7f927815cd50 legacy=0x7f92780c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.447+0000 7f927ca3a640 1 -- 172.21.15.125:0/329441249 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f927815d5e0 con 0x7f92781538f0 2023-10-30T12:29:01.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.451+0000 7f927ca3a640 1 --2- 172.21.15.125:0/329441249 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92781538f0 0x7f9278159fb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9278066a40 tx=0x7f926c01ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8855798d29f977 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.451+0000 7f92777fe640 1 -- 172.21.15.125:0/329441249 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c0a89b0 con 0x7f92781538f0 2023-10-30T12:29:01.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.451+0000 7f92777fe640 1 -- 172.21.15.125:0/329441249 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f926c0a8b10 con 0x7f92781538f0 2023-10-30T12:29:01.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.451+0000 7f92777fe640 1 -- 172.21.15.125:0/329441249 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f926c0a8ea0 con 0x7f92781538f0 2023-10-30T12:29:01.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.451+0000 7f927da3c640 1 -- 172.21.15.125:0/329441249 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92781538f0 msgr2=0x7f9278159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.451+0000 7f927da3c640 1 --2- 172.21.15.125:0/329441249 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92781538f0 0x7f9278159fb0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f9278066a40 tx=0x7f926c01ecb0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.451+0000 7f927da3c640 1 -- 172.21.15.125:0/329441249 shutdown_connections 2023-10-30T12:29:01.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.451+0000 7f927da3c640 1 --2- 172.21.15.125:0/329441249 >> v2:172.21.15.136:3300/0 conn(0x7f92780cefa0 0x7f92780cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.451+0000 7f927da3c640 1 --2- 172.21.15.125:0/329441249 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92781538f0 0x7f9278159fb0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.451+0000 7f927da3c640 1 -- 172.21.15.125:0/329441249 >> 172.21.15.125:0/329441249 conn(0x7f927806a640 msgr2=0x7f9278065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.451+0000 7f927da3c640 1 -- 172.21.15.125:0/329441249 shutdown_connections 2023-10-30T12:29:01.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.451+0000 7f927da3c640 1 -- 172.21.15.125:0/329441249 wait complete. 2023-10-30T12:29:01.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.451+0000 7f927da3c640 1 Processor -- start 2023-10-30T12:29:01.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.451+0000 7f927da3c640 1 -- start start 2023-10-30T12:29:01.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f927da3c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92780cefa0 0x7f9278149660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f927da3c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92781538f0 0x7f92781583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f927da3c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f927815cd50 0x7f9278158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f927da3c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9278152ae0 con 0x7f92781538f0 2023-10-30T12:29:01.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f927da3c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92781efbc0 con 0x7f92780cefa0 2023-10-30T12:29:01.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f927da3c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92781f00f0 con 0x7f927815cd50 2023-10-30T12:29:01.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f9277fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92781538f0 0x7f92781583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f9277fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92781538f0 0x7f92781583e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54588/0 (socket says 172.21.15.125:54588) 2023-10-30T12:29:01.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f9277fff640 1 -- 172.21.15.125:0/2731835381 learned_addr learned my addr 172.21.15.125:0/2731835381 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:01.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f927ca3a640 1 --2- 172.21.15.125:0/2731835381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92780cefa0 0x7f9278149660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f927d23b640 1 --2- 172.21.15.125:0/2731835381 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f927815cd50 0x7f9278158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f9277fff640 1 -- 172.21.15.125:0/2731835381 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f927815cd50 msgr2=0x7f9278158920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f9277fff640 1 --2- 172.21.15.125:0/2731835381 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f927815cd50 0x7f9278158920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f9277fff640 1 -- 172.21.15.125:0/2731835381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92780cefa0 msgr2=0x7f9278149660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f9277fff640 1 --2- 172.21.15.125:0/2731835381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92780cefa0 0x7f9278149660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f9277fff640 1 -- 172.21.15.125:0/2731835381 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f926c0094d0 con 0x7f92781538f0 2023-10-30T12:29:01.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f9277fff640 1 --2- 172.21.15.125:0/2731835381 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92781538f0 0x7f92781583e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f926400b410 tx=0x7f926400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f92757fa640 1 -- 172.21.15.125:0/2731835381 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264004020 con 0x7f92781538f0 2023-10-30T12:29:01.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f92757fa640 1 -- 172.21.15.125:0/2731835381 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9264004180 con 0x7f92781538f0 2023-10-30T12:29:01.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f92757fa640 1 -- 172.21.15.125:0/2731835381 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264011630 con 0x7f92781538f0 2023-10-30T12:29:01.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f927da3c640 1 -- 172.21.15.125:0/2731835381 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9278157f50 con 0x7f92781538f0 2023-10-30T12:29:01.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f927da3c640 1 -- 172.21.15.125:0/2731835381 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9278157900 con 0x7f92781538f0 2023-10-30T12:29:01.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f927da3c640 1 -- 172.21.15.125:0/2731835381 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92781580e0 con 0x7f92781538f0 2023-10-30T12:29:01.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.455+0000 7f92757fa640 1 -- 172.21.15.125:0/2731835381 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9264011790 con 0x7f92781538f0 2023-10-30T12:29:01.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.459+0000 7f92757fa640 1 --2- 172.21.15.125:0/2731835381 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f925c0664a0 0x7f925c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.459+0000 7f92757fa640 1 -- 172.21.15.125:0/2731835381 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9264083bc0 con 0x7f92781538f0 2023-10-30T12:29:01.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.459+0000 7f927ca3a640 1 --2- 172.21.15.125:0/2731835381 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f925c0664a0 0x7f925c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.459+0000 7f927ca3a640 1 --2- 172.21.15.125:0/2731835381 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f925c0664a0 0x7f925c068960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f926c0b2000 tx=0x7f926c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.463+0000 7f92757fa640 1 -- 172.21.15.125:0/2731835381 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9264050fa0 con 0x7f92781538f0 2023-10-30T12:29:01.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.611+0000 7f927da3c640 1 -- 172.21.15.125:0/2731835381 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f927804e170 con 0x7f92781538f0 2023-10-30T12:29:01.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.615+0000 7f92757fa640 1 -- 172.21.15.125:0/2731835381 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16557 (secure 0 0 0) 0x7f9264015090 con 0x7f92781538f0 2023-10-30T12:29:01.616 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:29:01.616 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:45.927862+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:29:01.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.615+0000 7f927da3c640 1 -- 172.21.15.125:0/2731835381 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f925c0664a0 msgr2=0x7f925c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.615+0000 7f927da3c640 1 --2- 172.21.15.125:0/2731835381 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f925c0664a0 0x7f925c068960 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f926c0b2000 tx=0x7f926c009040 comp rx=0 tx=0).stop 2023-10-30T12:29:01.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.615+0000 7f927da3c640 1 -- 172.21.15.125:0/2731835381 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92781538f0 msgr2=0x7f92781583e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.615+0000 7f927da3c640 1 --2- 172.21.15.125:0/2731835381 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92781538f0 0x7f92781583e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f926400b410 tx=0x7f926400b8d0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.615+0000 7f927da3c640 1 -- 172.21.15.125:0/2731835381 shutdown_connections 2023-10-30T12:29:01.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.615+0000 7f927da3c640 1 --2- 172.21.15.125:0/2731835381 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f925c0664a0 0x7f925c068960 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.615+0000 7f927da3c640 1 --2- 172.21.15.125:0/2731835381 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f927815cd50 0x7f9278158920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.615+0000 7f927da3c640 1 --2- 172.21.15.125:0/2731835381 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92781538f0 0x7f92781583e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.615+0000 7f927da3c640 1 --2- 172.21.15.125:0/2731835381 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92780cefa0 0x7f9278149660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.615+0000 7f927da3c640 1 -- 172.21.15.125:0/2731835381 >> 172.21.15.125:0/2731835381 conn(0x7f927806a640 msgr2=0x7f92780ce610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.615+0000 7f927da3c640 1 -- 172.21.15.125:0/2731835381 shutdown_connections 2023-10-30T12:29:01.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.615+0000 7f927da3c640 1 -- 172.21.15.125:0/2731835381 wait complete. 2023-10-30T12:29:01.637 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:29:01.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.771+0000 7fda93372640 1 Processor -- start 2023-10-30T12:29:01.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.771+0000 7fda93372640 1 -- start start 2023-10-30T12:29:01.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.771+0000 7fda93372640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fda8c1538a0 0x7fda8c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda93372640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda8c0ca0e0 0x7fda8c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda93372640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fda8c0cf160 0x7fda8c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda93372640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda8c0cb800 con 0x7fda8c1538a0 2023-10-30T12:29:01.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda93372640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda8c15dc30 con 0x7fda8c0cf160 2023-10-30T12:29:01.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda93372640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c0cb330 con 0x7fda8c0ca0e0 2023-10-30T12:29:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda9110e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fda8c1538a0 0x7fda8c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda9110e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fda8c1538a0 0x7fda8c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54600/0 (socket says 172.21.15.125:54600) 2023-10-30T12:29:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda9110e640 1 -- 172.21.15.125:0/619064946 learned_addr learned my addr 172.21.15.125:0/619064946 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda9190f640 1 --2- 172.21.15.125:0/619064946 >> v2:172.21.15.136:3300/0 conn(0x7fda8c0cf160 0x7fda8c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda9090d640 1 --2- 172.21.15.125:0/619064946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda8c0ca0e0 0x7fda8c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda9110e640 1 -- 172.21.15.125:0/619064946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda8c0ca0e0 msgr2=0x7fda8c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda9110e640 1 --2- 172.21.15.125:0/619064946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda8c0ca0e0 0x7fda8c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda9110e640 1 -- 172.21.15.125:0/619064946 >> v2:172.21.15.136:3300/0 conn(0x7fda8c0cf160 msgr2=0x7fda8c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:01.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda9110e640 1 --2- 172.21.15.125:0/619064946 >> v2:172.21.15.136:3300/0 conn(0x7fda8c0cf160 0x7fda8c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda9110e640 1 -- 172.21.15.125:0/619064946 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c0cad70 con 0x7fda8c1538a0 2023-10-30T12:29:01.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda9110e640 1 --2- 172.21.15.125:0/619064946 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda8c1538a0 0x7fda8c15d6f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fda8c06cb00 tx=0x7fda8801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fa017b9506e5ee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda83fff640 1 -- 172.21.15.125:0/619064946 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda880a89b0 con 0x7fda8c1538a0 2023-10-30T12:29:01.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda83fff640 1 -- 172.21.15.125:0/619064946 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda880a8b10 con 0x7fda8c1538a0 2023-10-30T12:29:01.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.775+0000 7fda83fff640 1 -- 172.21.15.125:0/619064946 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda880a8ea0 con 0x7fda8c1538a0 2023-10-30T12:29:01.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 -- 172.21.15.125:0/619064946 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda8c1538a0 msgr2=0x7fda8c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 --2- 172.21.15.125:0/619064946 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda8c1538a0 0x7fda8c15d6f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fda8c06cb00 tx=0x7fda8801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 -- 172.21.15.125:0/619064946 shutdown_connections 2023-10-30T12:29:01.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 --2- 172.21.15.125:0/619064946 >> v2:172.21.15.136:3300/0 conn(0x7fda8c0cf160 0x7fda8c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 --2- 172.21.15.125:0/619064946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda8c0ca0e0 0x7fda8c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 --2- 172.21.15.125:0/619064946 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda8c1538a0 0x7fda8c15d6f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 -- 172.21.15.125:0/619064946 >> 172.21.15.125:0/619064946 conn(0x7fda8c06bfe0 msgr2=0x7fda8c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 -- 172.21.15.125:0/619064946 shutdown_connections 2023-10-30T12:29:01.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 -- 172.21.15.125:0/619064946 wait complete. 2023-10-30T12:29:01.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 Processor -- start 2023-10-30T12:29:01.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 -- start start 2023-10-30T12:29:01.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda8c0ca0e0 0x7fda8c0b4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda8c0cf160 0x7fda8c14fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda8c1538a0 0x7fda8c0b35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c1f36d0 con 0x7fda8c0cf160 2023-10-30T12:29:01.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c1efd60 con 0x7fda8c0ca0e0 2023-10-30T12:29:01.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda93372640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda8c15cc90 con 0x7fda8c1538a0 2023-10-30T12:29:01.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda9090d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda8c0cf160 0x7fda8c14fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda9090d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda8c0cf160 0x7fda8c14fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54610/0 (socket says 172.21.15.125:54610) 2023-10-30T12:29:01.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda9090d640 1 -- 172.21.15.125:0/1535804893 learned_addr learned my addr 172.21.15.125:0/1535804893 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:01.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda9110e640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda8c0ca0e0 0x7fda8c0b4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.779+0000 7fda9190f640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda8c1538a0 0x7fda8c0b35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda9090d640 1 -- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda8c1538a0 msgr2=0x7fda8c0b35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda9090d640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda8c1538a0 0x7fda8c0b35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda9090d640 1 -- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda8c0ca0e0 msgr2=0x7fda8c0b4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda9090d640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda8c0ca0e0 0x7fda8c0b4df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda9090d640 1 -- 172.21.15.125:0/1535804893 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda880094d0 con 0x7fda8c0cf160 2023-10-30T12:29:01.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda9190f640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda8c1538a0 0x7fda8c0b35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:01.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda9090d640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda8c0cf160 0x7fda8c14fde0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00d830 tx=0x7fda7c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda81ffb640 1 -- 172.21.15.125:0/1535804893 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c009640 con 0x7fda8c0cf160 2023-10-30T12:29:01.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda81ffb640 1 -- 172.21.15.125:0/1535804893 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda7c0097a0 con 0x7fda8c0cf160 2023-10-30T12:29:01.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda81ffb640 1 -- 172.21.15.125:0/1535804893 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda7c009b10 con 0x7fda8c0cf160 2023-10-30T12:29:01.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda93372640 1 -- 172.21.15.125:0/1535804893 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8c1f0180 con 0x7fda8c0cf160 2023-10-30T12:29:01.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda93372640 1 -- 172.21.15.125:0/1535804893 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda8c1ef3c0 con 0x7fda8c0cf160 2023-10-30T12:29:01.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda93372640 1 -- 172.21.15.125:0/1535804893 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda8c1effd0 con 0x7fda8c0cf160 2023-10-30T12:29:01.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda81ffb640 1 -- 172.21.15.125:0/1535804893 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fda7c009c70 con 0x7fda8c0cf160 2023-10-30T12:29:01.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda81ffb640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda700664a0 0x7fda70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:01.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.783+0000 7fda81ffb640 1 -- 172.21.15.125:0/1535804893 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda7c084520 con 0x7fda8c0cf160 2023-10-30T12:29:01.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.787+0000 7fda9110e640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda700664a0 0x7fda70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:01.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.787+0000 7fda9110e640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda700664a0 0x7fda70068960 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fda880b2000 tx=0x7fda88009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:01.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.787+0000 7fda81ffb640 1 -- 172.21.15.125:0/1535804893 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fda7c0519a0 con 0x7fda8c0cf160 2023-10-30T12:29:01.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.971+0000 7fda93372640 1 -- 172.21.15.125:0/1535804893 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda8c0b2a50 con 0x7fda8c0cf160 2023-10-30T12:29:01.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.971+0000 7fda81ffb640 1 -- 172.21.15.125:0/1535804893 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+6539 (secure 0 0 0) 0x7fda7c054fc0 con 0x7fda8c0cf160 2023-10-30T12:29:01.975 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:29:01.975 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":11,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":11,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:28:55.942137+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":2,"incarnation":11,"state":"up:creating","state_seq":1,"addr":"172.21.15.136:6835/3346501886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3346501886},{"type":"v1","addr":"172.21.15.136:6835","nonce":3346501886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:29:01.976 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 11 2023-10-30T12:29:01.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.975+0000 7fda93372640 1 -- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda700664a0 msgr2=0x7fda70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.975+0000 7fda93372640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda700664a0 0x7fda70068960 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fda880b2000 tx=0x7fda88009040 comp rx=0 tx=0).stop 2023-10-30T12:29:01.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.975+0000 7fda93372640 1 -- 172.21.15.125:0/1535804893 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda8c0cf160 msgr2=0x7fda8c14fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:01.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.975+0000 7fda93372640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda8c0cf160 0x7fda8c14fde0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fda7c00d830 tx=0x7fda7c00dcf0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.975+0000 7fda93372640 1 -- 172.21.15.125:0/1535804893 shutdown_connections 2023-10-30T12:29:01.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.975+0000 7fda93372640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda700664a0 0x7fda70068960 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.975+0000 7fda93372640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda8c1538a0 0x7fda8c0b35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.975+0000 7fda93372640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda8c0cf160 0x7fda8c14fde0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.975+0000 7fda93372640 1 --2- 172.21.15.125:0/1535804893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda8c0ca0e0 0x7fda8c0b4df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:01.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.975+0000 7fda93372640 1 -- 172.21.15.125:0/1535804893 >> 172.21.15.125:0/1535804893 conn(0x7fda8c06bfe0 msgr2=0x7fda8c151f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:01.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.975+0000 7fda93372640 1 -- 172.21.15.125:0/1535804893 shutdown_connections 2023-10-30T12:29:01.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:01.975+0000 7fda93372640 1 -- 172.21.15.125:0/1535804893 wait complete. 2023-10-30T12:29:02.599 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 11, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-30T12:28:44.410746+0000', 'modified': '2023-10-30T12:28:55.942137+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4294, 'mds_1': 4285, 'mds_2': 4279}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4279': {'gid': 4279, 'name': 'd', 'rank': 2, 'incarnation': 11, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.136:6835/3346501886', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6834', 'nonce': 3346501886}, {'type': 'v1', 'addr': '172.21.15.136:6835', 'nonce': 3346501886}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4285': {'gid': 4285, 'name': 'f', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.136:6839/2250748765', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6838', 'nonce': 2250748765}, {'type': 'v1', 'addr': '172.21.15.136:6839', 'nonce': 2250748765}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4294': {'gid': 4294, 'name': 'e', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.125:6836/2017382874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6833', 'nonce': 2017382874}, {'type': 'v1', 'addr': '172.21.15.125:6836', 'nonce': 2017382874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-30T12:29:02.599 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4279:up:creating 2023-10-30T12:29:03.600 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:29:03.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.743+0000 7fce5fd2a640 1 Processor -- start 2023-10-30T12:29:03.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.743+0000 7fce5fd2a640 1 -- start start 2023-10-30T12:29:03.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.743+0000 7fce5fd2a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fce581538a0 0x7fce5815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.743+0000 7fce5fd2a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce580cf160 0x7fce580cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.743+0000 7fce5fd2a640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce580cb800 con 0x7fce580ca0e0 2023-10-30T12:29:03.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.743+0000 7fce5fd2a640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce5815dc30 con 0x7fce581538a0 2023-10-30T12:29:03.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.743+0000 7fce5fd2a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce580cb330 con 0x7fce580cf160 2023-10-30T12:29:03.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.743+0000 7fce5d2c5640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fce580ca0e0 0x7fce5815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35148/0 (socket says 172.21.15.125:35148) 2023-10-30T12:29:03.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.743+0000 7fce5d2c5640 1 -- 172.21.15.125:0/4142810890 learned_addr learned my addr 172.21.15.125:0/4142810890 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:29:03.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.747+0000 7fce5dac6640 1 --2- 172.21.15.125:0/4142810890 >> v2:172.21.15.136:3300/0 conn(0x7fce581538a0 0x7fce5815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.747+0000 7fce5e2c7640 1 --2- 172.21.15.125:0/4142810890 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce580cf160 0x7fce580cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.747+0000 7fce5cac4640 1 -- 172.21.15.125:0/4142810890 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000546487 0 0) 0x7fce580cb800 con 0x7fce580ca0e0 2023-10-30T12:29:03.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.747+0000 7fce5cac4640 1 -- 172.21.15.125:0/4142810890 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce44003530 con 0x7fce580ca0e0 2023-10-30T12:29:03.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.747+0000 7fce5dac6640 1 -- 172.21.15.125:0/4142810890 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce580cf160 msgr2=0x7fce580cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:03.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.747+0000 7fce5dac6640 1 --2- 172.21.15.125:0/4142810890 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce580cf160 0x7fce580cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.747+0000 7fce5dac6640 1 -- 172.21.15.125:0/4142810890 >> v1:172.21.15.125:6789/0 conn(0x7fce580ca0e0 legacy=0x7fce5815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.747+0000 7fce5cac4640 1 -- 172.21.15.125:0/4142810890 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 242760515 0 0) 0x7fce44003530 con 0x7fce580ca0e0 2023-10-30T12:29:03.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.747+0000 7fce5dac6640 1 -- 172.21.15.125:0/4142810890 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce580cad70 con 0x7fce581538a0 2023-10-30T12:29:03.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.747+0000 7fce5dac6640 1 --2- 172.21.15.125:0/4142810890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce581538a0 0x7fce5815d6f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce540097c0 tx=0x7fce5401eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f5dbbcce8338844 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.747+0000 7fce5cac4640 1 -- 172.21.15.125:0/4142810890 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce48001680 con 0x7fce581538a0 2023-10-30T12:29:03.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5cac4640 1 -- 172.21.15.125:0/4142810890 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce540a85d0 con 0x7fce581538a0 2023-10-30T12:29:03.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5cac4640 1 -- 172.21.15.125:0/4142810890 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce540a8940 con 0x7fce581538a0 2023-10-30T12:29:03.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 -- 172.21.15.125:0/4142810890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce581538a0 msgr2=0x7fce5815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 --2- 172.21.15.125:0/4142810890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce581538a0 0x7fce5815d6f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce540097c0 tx=0x7fce5401eb90 comp rx=0 tx=0).stop 2023-10-30T12:29:03.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 -- 172.21.15.125:0/4142810890 shutdown_connections 2023-10-30T12:29:03.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 --2- 172.21.15.125:0/4142810890 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce580cf160 0x7fce580cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 --2- 172.21.15.125:0/4142810890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce581538a0 0x7fce5815d6f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 -- 172.21.15.125:0/4142810890 >> 172.21.15.125:0/4142810890 conn(0x7fce5806bfe0 msgr2=0x7fce5806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 -- 172.21.15.125:0/4142810890 shutdown_connections 2023-10-30T12:29:03.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 -- 172.21.15.125:0/4142810890 wait complete. 2023-10-30T12:29:03.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 Processor -- start 2023-10-30T12:29:03.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 -- start start 2023-10-30T12:29:03.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce580ca0e0 0x7fce58065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce580cf160 0x7fce580b33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce581538a0 0x7fce58145d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce580ccfb0 con 0x7fce580cf160 2023-10-30T12:29:03.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce581596a0 con 0x7fce580ca0e0 2023-10-30T12:29:03.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5fd2a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce580b5120 con 0x7fce581538a0 2023-10-30T12:29:03.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5d2c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce580cf160 0x7fce580b33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5d2c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce580cf160 0x7fce580b33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54618/0 (socket says 172.21.15.125:54618) 2023-10-30T12:29:03.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5d2c5640 1 -- 172.21.15.125:0/833327503 learned_addr learned my addr 172.21.15.125:0/833327503 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:03.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5dac6640 1 --2- 172.21.15.125:0/833327503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce580ca0e0 0x7fce58065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.751+0000 7fce5e2c7640 1 --2- 172.21.15.125:0/833327503 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce581538a0 0x7fce58145d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce5d2c5640 1 -- 172.21.15.125:0/833327503 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce581538a0 msgr2=0x7fce58145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce5d2c5640 1 --2- 172.21.15.125:0/833327503 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce581538a0 0x7fce58145d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce5d2c5640 1 -- 172.21.15.125:0/833327503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce580ca0e0 msgr2=0x7fce58065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce5d2c5640 1 --2- 172.21.15.125:0/833327503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce580ca0e0 0x7fce58065ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce5d2c5640 1 -- 172.21.15.125:0/833327503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce54009420 con 0x7fce580cf160 2023-10-30T12:29:03.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce5d2c5640 1 --2- 172.21.15.125:0/833327503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce580cf160 0x7fce580b33e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fce4800c750 tx=0x7fce4800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce4e7fc640 1 -- 172.21.15.125:0/833327503 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce48005ae0 con 0x7fce580cf160 2023-10-30T12:29:03.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce4e7fc640 1 -- 172.21.15.125:0/833327503 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce48005c40 con 0x7fce580cf160 2023-10-30T12:29:03.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce5fd2a640 1 -- 172.21.15.125:0/833327503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce580ce4f0 con 0x7fce580cf160 2023-10-30T12:29:03.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce4e7fc640 1 -- 172.21.15.125:0/833327503 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce48007070 con 0x7fce580cf160 2023-10-30T12:29:03.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce5fd2a640 1 -- 172.21.15.125:0/833327503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce581593a0 con 0x7fce580cf160 2023-10-30T12:29:03.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce5fd2a640 1 -- 172.21.15.125:0/833327503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce580b2a50 con 0x7fce580cf160 2023-10-30T12:29:03.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce4e7fc640 1 -- 172.21.15.125:0/833327503 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fce480071f0 con 0x7fce580cf160 2023-10-30T12:29:03.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce4e7fc640 1 --2- 172.21.15.125:0/833327503 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fce3c0664a0 0x7fce3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:03.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce4e7fc640 1 -- 172.21.15.125:0/833327503 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce480834e0 con 0x7fce580cf160 2023-10-30T12:29:03.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.755+0000 7fce5dac6640 1 --2- 172.21.15.125:0/833327503 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fce3c0664a0 0x7fce3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:03.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.759+0000 7fce5dac6640 1 --2- 172.21.15.125:0/833327503 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fce3c0664a0 0x7fce3c068960 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fce540b2000 tx=0x7fce540aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:03.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.759+0000 7fce4e7fc640 1 -- 172.21.15.125:0/833327503 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fce48050a40 con 0x7fce580cf160 2023-10-30T12:29:03.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.943+0000 7fce5fd2a640 1 -- 172.21.15.125:0/833327503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce580b3180 con 0x7fce580cf160 2023-10-30T12:29:03.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.943+0000 7fce4e7fc640 1 -- 172.21.15.125:0/833327503 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6537 (secure 0 0 0) 0x7fce48054060 con 0x7fce580cf160 2023-10-30T12:29:03.946 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:29:03.946 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:29:01.095107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.136:6835/3346501886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3346501886},{"type":"v1","addr":"172.21.15.136:6835","nonce":3346501886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:29:03.947 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 12 2023-10-30T12:29:03.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.943+0000 7fce5fd2a640 1 -- 172.21.15.125:0/833327503 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fce3c0664a0 msgr2=0x7fce3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.943+0000 7fce5fd2a640 1 --2- 172.21.15.125:0/833327503 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fce3c0664a0 0x7fce3c068960 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fce540b2000 tx=0x7fce540aa070 comp rx=0 tx=0).stop 2023-10-30T12:29:03.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.943+0000 7fce5fd2a640 1 -- 172.21.15.125:0/833327503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce580cf160 msgr2=0x7fce580b33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:03.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.943+0000 7fce5fd2a640 1 --2- 172.21.15.125:0/833327503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce580cf160 0x7fce580b33e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fce4800c750 tx=0x7fce4800cc10 comp rx=0 tx=0).stop 2023-10-30T12:29:03.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.943+0000 7fce5fd2a640 1 -- 172.21.15.125:0/833327503 shutdown_connections 2023-10-30T12:29:03.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.943+0000 7fce5fd2a640 1 --2- 172.21.15.125:0/833327503 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fce3c0664a0 0x7fce3c068960 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.943+0000 7fce5fd2a640 1 --2- 172.21.15.125:0/833327503 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fce581538a0 0x7fce58145d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.943+0000 7fce5fd2a640 1 --2- 172.21.15.125:0/833327503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fce580cf160 0x7fce580b33e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.943+0000 7fce5fd2a640 1 --2- 172.21.15.125:0/833327503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fce580ca0e0 0x7fce58065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:03.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.943+0000 7fce5fd2a640 1 -- 172.21.15.125:0/833327503 >> 172.21.15.125:0/833327503 conn(0x7fce5806bfe0 msgr2=0x7fce581e2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:03.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.943+0000 7fce5fd2a640 1 -- 172.21.15.125:0/833327503 shutdown_connections 2023-10-30T12:29:03.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:03.943+0000 7fce5fd2a640 1 -- 172.21.15.125:0/833327503 wait complete. 2023-10-30T12:29:04.386 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:29:04.383+0000 7f6a6fdf8640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T12:29:04.569 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 12, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-30T12:28:44.410746+0000', 'modified': '2023-10-30T12:29:01.095107+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 4294, 'mds_1': 4285, 'mds_2': 4279}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4279': {'gid': 4279, 'name': 'd', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.136:6835/3346501886', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6834', 'nonce': 3346501886}, {'type': 'v1', 'addr': '172.21.15.136:6835', 'nonce': 3346501886}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4285': {'gid': 4285, 'name': 'f', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.136:6839/2250748765', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6838', 'nonce': 2250748765}, {'type': 'v1', 'addr': '172.21.15.136:6839', 'nonce': 2250748765}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_4294': {'gid': 4294, 'name': 'e', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.125:6836/2017382874', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6833', 'nonce': 2017382874}, {'type': 'v1', 'addr': '172.21.15.125:6836', 'nonce': 2017382874}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-30T12:29:04.570 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-10-30T12:29:04.570 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-30T12:29:04.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.707+0000 7fde99091640 1 Processor -- start 2023-10-30T12:29:04.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.707+0000 7fde99091640 1 -- start start 2023-10-30T12:29:04.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde99091640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fde940c7830 0x7fde9406ca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde99091640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9406d710 con 0x7fde940cce00 2023-10-30T12:29:04.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde99091640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9406b560 con 0x7fde9406e430 2023-10-30T12:29:04.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde99091640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde9406dd30 con 0x7fde940c7830 2023-10-30T12:29:04.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde92d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fde940cce00 0x7fde94159850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35150/0 (socket says 172.21.15.125:35150) 2023-10-30T12:29:04.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde92d9d640 1 -- 172.21.15.125:0/171483586 learned_addr learned my addr 172.21.15.125:0/171483586 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:29:04.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde8bfff640 1 --2- 172.21.15.125:0/171483586 >> v2:172.21.15.136:3300/0 conn(0x7fde940c7830 0x7fde9406ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 -- 172.21.15.125:0/171483586 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811734934 0 0) 0x7fde9406d710 con 0x7fde940cce00 2023-10-30T12:29:04.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 -- 172.21.15.125:0/171483586 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde80003530 con 0x7fde940cce00 2023-10-30T12:29:04.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 -- 172.21.15.125:0/171483586 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1896999537 0 0) 0x7fde80003530 con 0x7fde940cce00 2023-10-30T12:29:04.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 -- 172.21.15.125:0/171483586 >> v2:172.21.15.136:3300/0 conn(0x7fde940c7830 msgr2=0x7fde9406ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:04.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 --2- 172.21.15.125:0/171483586 >> v2:172.21.15.136:3300/0 conn(0x7fde940c7830 0x7fde9406ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 -- 172.21.15.125:0/171483586 >> v1:172.21.15.136:6789/0 conn(0x7fde9406e430 legacy=0x7fde94159f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 -- 172.21.15.125:0/171483586 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde940c3c80 con 0x7fde940cce00 2023-10-30T12:29:04.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 -- 172.21.15.125:0/171483586 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fde84001680 con 0x7fde940cce00 2023-10-30T12:29:04.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 -- 172.21.15.125:0/171483586 >> v1:172.21.15.125:6789/0 conn(0x7fde940cce00 legacy=0x7fde94159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 --2- 172.21.15.125:0/171483586 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde80003980 0x7fde80003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 --2- 172.21.15.125:0/171483586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde800047d0 0x7fde80004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 --2- 172.21.15.125:0/171483586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde80005190 0x7fde8000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 -- 172.21.15.125:0/171483586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9406dd30 con 0x7fde800047d0 2023-10-30T12:29:04.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 -- 172.21.15.125:0/171483586 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde8000c130 con 0x7fde80005190 2023-10-30T12:29:04.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9259c640 1 -- 172.21.15.125:0/171483586 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde8000c2f0 con 0x7fde80003980 2023-10-30T12:29:04.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde8bfff640 1 --2- 172.21.15.125:0/171483586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde800047d0 0x7fde80004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9359e640 1 --2- 172.21.15.125:0/171483586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde80005190 0x7fde8000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde92d9d640 1 --2- 172.21.15.125:0/171483586 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde80003980 0x7fde80003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde8bfff640 1 -- 172.21.15.125:0/171483586 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde80003980 msgr2=0x7fde80003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:04.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde8bfff640 1 --2- 172.21.15.125:0/171483586 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde80003980 0x7fde80003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde8bfff640 1 -- 172.21.15.125:0/171483586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde80005190 msgr2=0x7fde8000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:04.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde8bfff640 1 --2- 172.21.15.125:0/171483586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde80005190 0x7fde8000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde8bfff640 1 -- 172.21.15.125:0/171483586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde80003530 con 0x7fde800047d0 2023-10-30T12:29:04.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.711+0000 7fde9359e640 1 -- 172.21.15.125:0/171483586 reap_dead start 2023-10-30T12:29:04.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.715+0000 7fde8bfff640 1 --2- 172.21.15.125:0/171483586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde800047d0 0x7fde80004c50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009ae0 tx=0x7fde7c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b33b85ab44bf3fd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:04.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.715+0000 7fde9259c640 1 -- 172.21.15.125:0/171483586 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c0a6430 con 0x7fde800047d0 2023-10-30T12:29:04.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.715+0000 7fde9259c640 1 -- 172.21.15.125:0/171483586 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c0a6590 con 0x7fde800047d0 2023-10-30T12:29:04.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.715+0000 7fde9259c640 1 -- 172.21.15.125:0/171483586 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c0a6940 con 0x7fde800047d0 2023-10-30T12:29:04.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.715+0000 7fde99091640 1 -- 172.21.15.125:0/171483586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde800047d0 msgr2=0x7fde80004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.715+0000 7fde99091640 1 --2- 172.21.15.125:0/171483586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde800047d0 0x7fde80004c50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009ae0 tx=0x7fde7c01ca50 comp rx=0 tx=0).stop 2023-10-30T12:29:04.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.715+0000 7fde99091640 1 -- 172.21.15.125:0/171483586 shutdown_connections 2023-10-30T12:29:04.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.715+0000 7fde99091640 1 --2- 172.21.15.125:0/171483586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde800047d0 0x7fde80004c50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.715+0000 7fde99091640 1 -- 172.21.15.125:0/171483586 >> 172.21.15.125:0/171483586 conn(0x7fde94071ea0 msgr2=0x7fde94065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:04.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.715+0000 7fde99091640 1 -- 172.21.15.125:0/171483586 shutdown_connections 2023-10-30T12:29:04.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.715+0000 7fde99091640 1 -- 172.21.15.125:0/171483586 wait complete. 2023-10-30T12:29:04.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.715+0000 7fde99091640 1 Processor -- start 2023-10-30T12:29:04.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde99091640 1 -- start start 2023-10-30T12:29:04.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde99091640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde800047d0 0x7fde94152d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde99091640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde941f2c70 0x7fde9415c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde99091640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde94153a30 0x7fde9414e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde99091640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde94155e90 con 0x7fde941f2c70 2023-10-30T12:29:04.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde99091640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde941f30d0 con 0x7fde94153a30 2023-10-30T12:29:04.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde99091640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde941558d0 con 0x7fde800047d0 2023-10-30T12:29:04.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde8bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde941f2c70 0x7fde9415c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde8bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde941f2c70 0x7fde9415c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54638/0 (socket says 172.21.15.125:54638) 2023-10-30T12:29:04.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde8bfff640 1 -- 172.21.15.125:0/966886890 learned_addr learned my addr 172.21.15.125:0/966886890 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:04.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde9359e640 1 --2- 172.21.15.125:0/966886890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde94153a30 0x7fde9414e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde92d9d640 1 --2- 172.21.15.125:0/966886890 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde800047d0 0x7fde94152d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde8bfff640 1 -- 172.21.15.125:0/966886890 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde800047d0 msgr2=0x7fde94152d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde8bfff640 1 --2- 172.21.15.125:0/966886890 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde800047d0 0x7fde94152d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde8bfff640 1 -- 172.21.15.125:0/966886890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde94153a30 msgr2=0x7fde9414e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde8bfff640 1 --2- 172.21.15.125:0/966886890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde94153a30 0x7fde9414e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde8bfff640 1 -- 172.21.15.125:0/966886890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde94155bb0 con 0x7fde941f2c70 2023-10-30T12:29:04.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde8bfff640 1 --2- 172.21.15.125:0/966886890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde941f2c70 0x7fde9415c120 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009ae0 tx=0x7fde7c0197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:04.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde8b7fe640 1 -- 172.21.15.125:0/966886890 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c0a6e00 con 0x7fde941f2c70 2023-10-30T12:29:04.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde8b7fe640 1 -- 172.21.15.125:0/966886890 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c0b7420 con 0x7fde941f2c70 2023-10-30T12:29:04.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde8b7fe640 1 -- 172.21.15.125:0/966886890 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c0b7790 con 0x7fde941f2c70 2023-10-30T12:29:04.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde99091640 1 -- 172.21.15.125:0/966886890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde94152250 con 0x7fde941f2c70 2023-10-30T12:29:04.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.719+0000 7fde99091640 1 -- 172.21.15.125:0/966886890 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde94152810 con 0x7fde941f2c70 2023-10-30T12:29:04.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde8b7fe640 1 -- 172.21.15.125:0/966886890 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fde7c0b78f0 con 0x7fde941f2c70 2023-10-30T12:29:04.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde8b7fe640 1 --2- 172.21.15.125:0/966886890 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fde740664d0 0x7fde74068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde8b7fe640 1 -- 172.21.15.125:0/966886890 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde7c11e600 con 0x7fde941f2c70 2023-10-30T12:29:04.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde92d9d640 1 --2- 172.21.15.125:0/966886890 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fde740664d0 0x7fde74068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde9c411000 1 Processor -- start 2023-10-30T12:29:04.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde9c411000 1 -- start start 2023-10-30T12:29:04.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde92d9d640 1 --2- 172.21.15.125:0/966886890 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fde740664d0 0x7fde74068990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fde84001ec0 tx=0x7fde84009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:04.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde9c411000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a619952bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde9c411000 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953130 0x55a619953510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde9c411000 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953a50 0x55a61995daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde9c411000 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a619706cc0 con 0x55a619953130 2023-10-30T12:29:04.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde9c411000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a619769920 con 0x55a619952810 2023-10-30T12:29:04.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde9c411000 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a619766e50 con 0x55a619953a50 2023-10-30T12:29:04.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde8bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953130 0x55a619953510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde8bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953130 0x55a619953510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54652/0 (socket says 172.21.15.125:54652) 2023-10-30T12:29:04.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde8bfff640 1 -- 172.21.15.125:0/3665057946 learned_addr learned my addr 172.21.15.125:0/3665057946 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:04.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde92d9d640 1 --2- 172.21.15.125:0/3665057946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a619952bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde9359e640 1 --2- 172.21.15.125:0/3665057946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953a50 0x55a61995daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde8bfff640 1 -- 172.21.15.125:0/3665057946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953a50 msgr2=0x55a61995daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:04.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde8bfff640 1 --2- 172.21.15.125:0/3665057946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953a50 0x55a61995daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde8bfff640 1 -- 172.21.15.125:0/3665057946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 msgr2=0x55a619952bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:04.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde8bfff640 1 --2- 172.21.15.125:0/3665057946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a619952bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde8bfff640 1 -- 172.21.15.125:0/3665057946 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a61945b890 con 0x55a619953130 2023-10-30T12:29:04.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde8bfff640 1 --2- 172.21.15.125:0/3665057946 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953130 0x55a619953510 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0f7590 tx=0x7fde7c0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd117bcf596c6bbb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:04.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde88ff9640 1 -- 172.21.15.125:0/3665057946 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c0f7ac0 con 0x55a619953130 2023-10-30T12:29:04.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde88ff9640 1 -- 172.21.15.125:0/3665057946 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c0edb60 con 0x55a619953130 2023-10-30T12:29:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.723+0000 7fde88ff9640 1 -- 172.21.15.125:0/3665057946 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c0ebed0 con 0x55a619953130 2023-10-30T12:29:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 -- 172.21.15.125:0/3665057946 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953130 msgr2=0x55a619953510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 --2- 172.21.15.125:0/3665057946 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953130 0x55a619953510 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0f7590 tx=0x7fde7c0046d0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 -- 172.21.15.125:0/3665057946 shutdown_connections 2023-10-30T12:29:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 --2- 172.21.15.125:0/3665057946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953a50 0x55a61995daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 --2- 172.21.15.125:0/3665057946 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953130 0x55a619953510 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 --2- 172.21.15.125:0/3665057946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a619952bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 -- 172.21.15.125:0/3665057946 >> 172.21.15.125:0/3665057946 conn(0x55a619908fc0 msgr2=0x55a6199505d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:04.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 -- 172.21.15.125:0/3665057946 shutdown_connections 2023-10-30T12:29:04.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 -- 172.21.15.125:0/3665057946 wait complete. 2023-10-30T12:29:04.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 -- start start 2023-10-30T12:29:04.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 Processor -- start 2023-10-30T12:29:04.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a619914240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953130 0x55a619914780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953a50 0x55a619914cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a619762740 con 0x55a619953a50 2023-10-30T12:29:04.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a619721d10 con 0x55a619952810 2023-10-30T12:29:04.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6197219b0 con 0x55a619953130 2023-10-30T12:29:04.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 10 client.0 _ll_register_callbacks cb 0x55a6198fd390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T12:29:04.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9359e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953a50 0x55a619914cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9359e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953a50 0x55a619914cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54654/0 (socket says 172.21.15.125:54654) 2023-10-30T12:29:04.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9359e640 1 -- 172.21.15.125:0/2679281286 learned_addr learned my addr 172.21.15.125:0/2679281286 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:04.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde92d9d640 1 --2- 172.21.15.125:0/2679281286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a619914240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde8bfff640 1 --2- 172.21.15.125:0/2679281286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953130 0x55a619914780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9359e640 1 -- 172.21.15.125:0/2679281286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953130 msgr2=0x55a619914780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:04.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9359e640 1 --2- 172.21.15.125:0/2679281286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953130 0x55a619914780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9359e640 1 -- 172.21.15.125:0/2679281286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 msgr2=0x55a619914240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9359e640 1 --2- 172.21.15.125:0/2679281286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a619914240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9359e640 1 -- 172.21.15.125:0/2679281286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde94152810 con 0x55a619953a50 2023-10-30T12:29:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9359e640 1 --2- 172.21.15.125:0/2679281286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953a50 0x55a619914cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fde94070090 tx=0x7fde8c005820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1bb4285132f0320 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2023-10-30T12:29:04.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde71ffb640 1 -- 172.21.15.125:0/2679281286 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c001f70 con 0x55a619953a50 2023-10-30T12:29:04.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde71ffb640 1 -- 172.21.15.125:0/2679281286 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde8c0020d0 con 0x55a619953a50 2023-10-30T12:29:04.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.727+0000 7fde9c411000 1 -- 172.21.15.125:0/2679281286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a619716370 con 0x55a619953a50 2023-10-30T12:29:04.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.731+0000 7fde71ffb640 1 -- 172.21.15.125:0/2679281286 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde8c016480 con 0x55a619953a50 2023-10-30T12:29:04.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.731+0000 7fde9c411000 10 client.4680 fetch_fsmap learned FSMap version 12 2023-10-30T12:29:04.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.731+0000 7fde9c411000 1 -- 172.21.15.125:0/2679281286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55a619724460 con 0x55a619953a50 2023-10-30T12:29:04.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.731+0000 7fde71ffb640 1 -- 172.21.15.125:0/2679281286 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a619716370 con 0x55a619953a50 2023-10-30T12:29:04.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.731+0000 7fde71ffb640 1 -- 172.21.15.125:0/2679281286 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fde8c0165e0 con 0x55a619953a50 2023-10-30T12:29:04.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.731+0000 7fde9c411000 10 client.4680 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T12:29:04.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.731+0000 7fde9c411000 10 client.4680 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-30T12:29:04.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.731+0000 7fde9c411000 20 client.4680 populate_metadata read hostname 'smithi125' 2023-10-30T12:29:04.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.731+0000 7fde9c411000 1 --2- 172.21.15.125:0/2679281286 >> [v2:172.21.15.136:6834/3346501886,v1:172.21.15.136:6835/3346501886] conn(0x55a6199156d0 0x55a619967900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:04.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.731+0000 7fde9c411000 4 client.4680 mds_command: new command op to 4279 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:29:04.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.731+0000 7fde9c411000 1 -- 172.21.15.125:0/2679281286 --> [v2:172.21.15.136:6834/3346501886,v1:172.21.15.136:6835/3346501886] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a6196fc2d0 con 0x55a6199156d0 2023-10-30T12:29:04.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.731+0000 7fde92d9d640 1 --2- 172.21.15.125:0/2679281286 >> [v2:172.21.15.136:6834/3346501886,v1:172.21.15.136:6835/3346501886] conn(0x55a6199156d0 0x55a619967900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:04.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.731+0000 7fde71ffb640 10 client.4680 ms_handle_connect on v2:172.21.15.136:6834/3346501886 2023-10-30T12:29:04.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:04.731+0000 7fde92d9d640 1 --2- 172.21.15.125:0/2679281286 >> [v2:172.21.15.136:6834/3346501886,v1:172.21.15.136:6835/3346501886] conn(0x55a6199156d0 0x55a619967900 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ede13a2303de7d42 server_cookie=250803e34e4ef1e6 in_seq=0 out_seq=0 2023-10-30T12:29:07.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.035+0000 7fde71ffb640 1 -- 172.21.15.125:0/2679281286 <== mds.2 v2:172.21.15.136:6834/3346501886 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a6196fc2d0 con 0x55a6199156d0 2023-10-30T12:29:07.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.035+0000 7fde71ffb640 10 client.4680 handle_command_reply: tid=0 2023-10-30T12:29:07.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.035+0000 7fde9c411000 1 client.4680 shutdown 2023-10-30T12:29:07.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.039+0000 7fde9c411000 1 -- 172.21.15.125:0/2679281286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953a50 msgr2=0x55a619914cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.039+0000 7fde9c411000 1 --2- 172.21.15.125:0/2679281286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953a50 0x55a619914cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fde94070090 tx=0x7fde8c005820 comp rx=0 tx=0).stop 2023-10-30T12:29:07.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.039+0000 7fde9c411000 1 -- 172.21.15.125:0/2679281286 shutdown_connections 2023-10-30T12:29:07.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.039+0000 7fde9c411000 1 --2- 172.21.15.125:0/2679281286 >> [v2:172.21.15.136:6834/3346501886,v1:172.21.15.136:6835/3346501886] conn(0x55a6199156d0 0x55a619967900 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.039+0000 7fde9c411000 1 --2- 172.21.15.125:0/2679281286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953a50 0x55a619914cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.039+0000 7fde9c411000 1 --2- 172.21.15.125:0/2679281286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953130 0x55a619914780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.039+0000 7fde9c411000 1 --2- 172.21.15.125:0/2679281286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a619914240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.039+0000 7fde71ffb640 0 client.4680 ms_handle_reset on v2:172.21.15.136:6834/3346501886 2023-10-30T12:29:07.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.039+0000 7fde9c411000 1 -- 172.21.15.125:0/2679281286 >> 172.21.15.125:0/2679281286 conn(0x55a619908fc0 msgr2=0x55a619952090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.039+0000 7fde9c411000 1 -- 172.21.15.125:0/2679281286 shutdown_connections 2023-10-30T12:29:07.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.039+0000 7fde9c411000 1 -- 172.21.15.125:0/2679281286 wait complete. 2023-10-30T12:29:07.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.039+0000 7fde9c411000 20 client.4680 trim_cache size 0 max 16384 2023-10-30T12:29:07.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde9c411000 1 Processor -- start 2023-10-30T12:29:07.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde9c411000 1 -- start start 2023-10-30T12:29:07.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde9c411000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a619910770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde9c411000 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953130 0x55a619910cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde9c411000 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953a50 0x55a6199111f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde9c411000 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a61941c940 con 0x55a619953130 2023-10-30T12:29:07.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde9c411000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a61945c1b0 con 0x55a619952810 2023-10-30T12:29:07.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde9c411000 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a619766e50 con 0x55a619953a50 2023-10-30T12:29:07.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde8bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953130 0x55a619910cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde8bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953130 0x55a619910cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54666/0 (socket says 172.21.15.125:54666) 2023-10-30T12:29:07.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde8bfff640 1 -- 172.21.15.125:0/1676494235 learned_addr learned my addr 172.21.15.125:0/1676494235 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde92d9d640 1 --2- 172.21.15.125:0/1676494235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a619910770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde9359e640 1 --2- 172.21.15.125:0/1676494235 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953a50 0x55a6199111f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde8bfff640 1 -- 172.21.15.125:0/1676494235 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953a50 msgr2=0x55a6199111f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde8bfff640 1 --2- 172.21.15.125:0/1676494235 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953a50 0x55a6199111f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde8bfff640 1 -- 172.21.15.125:0/1676494235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 msgr2=0x55a619910770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde8bfff640 1 --2- 172.21.15.125:0/1676494235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a619910770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde8bfff640 1 -- 172.21.15.125:0/1676494235 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a61970a760 con 0x55a619953130 2023-10-30T12:29:07.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde92d9d640 1 --2- 172.21.15.125:0/1676494235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a619910770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:07.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde8bfff640 1 --2- 172.21.15.125:0/1676494235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953130 0x55a619910cb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0ef750 tx=0x7fde7c11e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9726a1a476158a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde897fa640 1 -- 172.21.15.125:0/1676494235 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c0f7ac0 con 0x55a619953130 2023-10-30T12:29:07.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde897fa640 1 -- 172.21.15.125:0/1676494235 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde7c107030 con 0x55a619953130 2023-10-30T12:29:07.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.055+0000 7fde897fa640 1 -- 172.21.15.125:0/1676494235 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde7c0096b0 con 0x55a619953130 2023-10-30T12:29:07.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 -- 172.21.15.125:0/1676494235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953130 msgr2=0x55a619910cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 --2- 172.21.15.125:0/1676494235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953130 0x55a619910cb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fde7c0ef750 tx=0x7fde7c11e000 comp rx=0 tx=0).stop 2023-10-30T12:29:07.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 -- 172.21.15.125:0/1676494235 shutdown_connections 2023-10-30T12:29:07.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 --2- 172.21.15.125:0/1676494235 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953a50 0x55a6199111f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 --2- 172.21.15.125:0/1676494235 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953130 0x55a619910cb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 --2- 172.21.15.125:0/1676494235 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a619910770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 -- 172.21.15.125:0/1676494235 >> 172.21.15.125:0/1676494235 conn(0x55a619908fc0 msgr2=0x55a61994f280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 -- 172.21.15.125:0/1676494235 shutdown_connections 2023-10-30T12:29:07.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 -- 172.21.15.125:0/1676494235 wait complete. 2023-10-30T12:29:07.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 -- start start 2023-10-30T12:29:07.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 Processor -- start 2023-10-30T12:29:07.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a6199aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953130 0x55a6199af320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953a50 0x55a6199afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a61975ca40 con 0x55a619953a50 2023-10-30T12:29:07.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6197615d0 con 0x55a619952810 2023-10-30T12:29:07.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a6197fe810 con 0x55a619953130 2023-10-30T12:29:07.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9c411000 10 client.0 _ll_register_callbacks cb 0x7fde7c0ebb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T12:29:07.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9359e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953a50 0x55a6199afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9359e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953a50 0x55a6199afcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54668/0 (socket says 172.21.15.125:54668) 2023-10-30T12:29:07.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9359e640 1 -- 172.21.15.125:0/2419933732 learned_addr learned my addr 172.21.15.125:0/2419933732 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde8bfff640 1 --2- 172.21.15.125:0/2419933732 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953130 0x55a6199af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde92d9d640 1 --2- 172.21.15.125:0/2419933732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a6199aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9359e640 1 -- 172.21.15.125:0/2419933732 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953130 msgr2=0x55a6199af320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9359e640 1 --2- 172.21.15.125:0/2419933732 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953130 0x55a6199af320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9359e640 1 -- 172.21.15.125:0/2419933732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 msgr2=0x55a6199aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9359e640 1 --2- 172.21.15.125:0/2419933732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a6199aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9359e640 1 -- 172.21.15.125:0/2419933732 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a61945b890 con 0x55a619953a50 2023-10-30T12:29:07.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.059+0000 7fde9359e640 1 --2- 172.21.15.125:0/2419933732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953a50 0x55a6199afcc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fde8c001230 tx=0x7fde8c001500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3451f3e30fb8d97 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde71ffb640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2023-10-30T12:29:07.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde71ffb640 1 -- 172.21.15.125:0/2419933732 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8c0012b0 con 0x55a619953a50 2023-10-30T12:29:07.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde71ffb640 1 -- 172.21.15.125:0/2419933732 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde8c016b40 con 0x55a619953a50 2023-10-30T12:29:07.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 -- 172.21.15.125:0/2419933732 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a619915090 con 0x55a619953a50 2023-10-30T12:29:07.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde71ffb640 1 -- 172.21.15.125:0/2419933732 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7fde8c016690 con 0x55a619953a50 2023-10-30T12:29:07.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 10 client.4686 fetch_fsmap learned FSMap version 12 2023-10-30T12:29:07.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 -- 172.21.15.125:0/2419933732 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55a61945bcc0 con 0x55a619953a50 2023-10-30T12:29:07.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde71ffb640 1 -- 172.21.15.125:0/2419933732 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a619915090 con 0x55a619953a50 2023-10-30T12:29:07.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde71ffb640 1 -- 172.21.15.125:0/2419933732 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7fde8c02f420 con 0x55a619953a50 2023-10-30T12:29:07.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 10 client.4686 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T12:29:07.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 10 client.4686 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-30T12:29:07.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 20 client.4686 populate_metadata read hostname 'smithi125' 2023-10-30T12:29:07.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 --2- 172.21.15.125:0/2419933732 >> [v2:172.21.15.136:6834/3346501886,v1:172.21.15.136:6835/3346501886] conn(0x55a6199bc380 0x55a6199b0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 4 client.4686 mds_command: new command op to 4279 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T12:29:07.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 -- 172.21.15.125:0/2419933732 --> [v2:172.21.15.136:6834/3346501886,v1:172.21.15.136:6835/3346501886] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a6196fa6f0 con 0x55a6199bc380 2023-10-30T12:29:07.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde92d9d640 1 --2- 172.21.15.125:0/2419933732 >> [v2:172.21.15.136:6834/3346501886,v1:172.21.15.136:6835/3346501886] conn(0x55a6199bc380 0x55a6199b0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde92d9d640 1 --2- 172.21.15.125:0/2419933732 >> [v2:172.21.15.136:6834/3346501886,v1:172.21.15.136:6835/3346501886] conn(0x55a6199bc380 0x55a6199b0bf0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=333a9b45cc666c15 server_cookie=ea5fa29f2fe02181 in_seq=0 out_seq=0 2023-10-30T12:29:07.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde71ffb640 10 client.4686 ms_handle_connect on v2:172.21.15.136:6834/3346501886 2023-10-30T12:29:07.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde71ffb640 1 -- 172.21.15.125:0/2419933732 <== mds.2 v2:172.21.15.136:6834/3346501886 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55a6196fa6f0 con 0x55a6199bc380 2023-10-30T12:29:07.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde71ffb640 10 client.4686 handle_command_reply: tid=0 2023-10-30T12:29:07.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 client.4686 shutdown 2023-10-30T12:29:07.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 -- 172.21.15.125:0/2419933732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953a50 msgr2=0x55a6199afcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 --2- 172.21.15.125:0/2419933732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953a50 0x55a6199afcc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fde8c001230 tx=0x7fde8c001500 comp rx=0 tx=0).stop 2023-10-30T12:29:07.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 -- 172.21.15.125:0/2419933732 shutdown_connections 2023-10-30T12:29:07.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 --2- 172.21.15.125:0/2419933732 >> [v2:172.21.15.136:6834/3346501886,v1:172.21.15.136:6835/3346501886] conn(0x55a6199bc380 0x55a6199b0bf0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 --2- 172.21.15.125:0/2419933732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55a619953a50 0x55a6199afcc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 --2- 172.21.15.125:0/2419933732 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55a619953130 0x55a6199af320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde71ffb640 0 client.4686 ms_handle_reset on v2:172.21.15.136:6834/3346501886 2023-10-30T12:29:07.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 --2- 172.21.15.125:0/2419933732 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55a619952810 0x55a6199aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 -- 172.21.15.125:0/2419933732 >> 172.21.15.125:0/2419933732 conn(0x55a619908fc0 msgr2=0x55a61994fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 -- 172.21.15.125:0/2419933732 shutdown_connections 2023-10-30T12:29:07.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 1 -- 172.21.15.125:0/2419933732 wait complete. 2023-10-30T12:29:07.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.063+0000 7fde9c411000 20 client.4686 trim_cache size 0 max 16384 2023-10-30T12:29:07.078 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2023-10-30T12:29:07.078 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_fsid": "714f08a3-69cf-4c1b-8fab-64b3ba5e42e9", 2023-10-30T12:29:07.078 INFO:teuthology.orchestra.run.smithi125.stdout: "whoami": 2, 2023-10-30T12:29:07.079 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4279, 2023-10-30T12:29:07.079 INFO:teuthology.orchestra.run.smithi125.stdout: "want_state": "up:active", 2023-10-30T12:29:07.079 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "up:active", 2023-10-30T12:29:07.079 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs", 2023-10-30T12:29:07.079 INFO:teuthology.orchestra.run.smithi125.stdout: "rank_uptime": 11.115994625000001, 2023-10-30T12:29:07.079 INFO:teuthology.orchestra.run.smithi125.stdout: "mdsmap_epoch": 12, 2023-10-30T12:29:07.080 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch": 22, 2023-10-30T12:29:07.080 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch_barrier": 22, 2023-10-30T12:29:07.080 INFO:teuthology.orchestra.run.smithi125.stdout: "uptime": 28.706088397999999 2023-10-30T12:29:07.080 INFO:teuthology.orchestra.run.smithi125.stdout:} 2023-10-30T12:29:07.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.067+0000 7fde99091640 1 -- 172.21.15.125:0/966886890 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fde740664d0 msgr2=0x7fde74068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.067+0000 7fde99091640 1 --2- 172.21.15.125:0/966886890 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fde740664d0 0x7fde74068990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fde84001ec0 tx=0x7fde84009040 comp rx=0 tx=0).stop 2023-10-30T12:29:07.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.067+0000 7fde99091640 1 -- 172.21.15.125:0/966886890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde941f2c70 msgr2=0x7fde9415c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.067+0000 7fde99091640 1 --2- 172.21.15.125:0/966886890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde941f2c70 0x7fde9415c120 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fde7c009ae0 tx=0x7fde7c0197f0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.067+0000 7fde99091640 1 -- 172.21.15.125:0/966886890 shutdown_connections 2023-10-30T12:29:07.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.067+0000 7fde99091640 1 --2- 172.21.15.125:0/966886890 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fde740664d0 0x7fde74068990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.067+0000 7fde99091640 1 --2- 172.21.15.125:0/966886890 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde94153a30 0x7fde9414e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.067+0000 7fde99091640 1 --2- 172.21.15.125:0/966886890 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde941f2c70 0x7fde9415c120 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.067+0000 7fde99091640 1 --2- 172.21.15.125:0/966886890 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde800047d0 0x7fde94152d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.067+0000 7fde99091640 1 -- 172.21.15.125:0/966886890 >> 172.21.15.125:0/966886890 conn(0x7fde94071ea0 msgr2=0x7fde94157f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.067+0000 7fde99091640 1 -- 172.21.15.125:0/966886890 shutdown_connections 2023-10-30T12:29:07.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.067+0000 7fde99091640 1 -- 172.21.15.125:0/966886890 wait complete. 2023-10-30T12:29:07.091 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2023-10-30T12:29:07.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d76c9640 1 Processor -- start 2023-10-30T12:29:07.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d76c9640 1 -- start start 2023-10-30T12:29:07.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d76c9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42d0153c60 0x7f42d00ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d76c9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f42d00c4430 0x7f42d00cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d76c9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d00cb780 con 0x7f42d0159830 2023-10-30T12:29:07.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d76c9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42d0154040 con 0x7f42d0153c60 2023-10-30T12:29:07.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d76c9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42d00cc9f0 con 0x7f42d00c4430 2023-10-30T12:29:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d5465640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42d0153c60 0x7f42d00ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d5465640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42d0153c60 0x7f42d00ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54680/0 (socket says 172.21.15.125:54680) 2023-10-30T12:29:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d5465640 1 -- 172.21.15.125:0/1710060506 learned_addr learned my addr 172.21.15.125:0/1710060506 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d4c64640 1 --2- 172.21.15.125:0/1710060506 >> v2:172.21.15.136:3300/0 conn(0x7f42d00c4430 0x7f42d00cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d5465640 1 -- 172.21.15.125:0/1710060506 >> v2:172.21.15.136:3300/0 conn(0x7f42d00c4430 msgr2=0x7f42d00cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d5465640 1 --2- 172.21.15.125:0/1710060506 >> v2:172.21.15.136:3300/0 conn(0x7f42d00c4430 0x7f42d00cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d5465640 1 -- 172.21.15.125:0/1710060506 >> v1:172.21.15.125:6789/0 conn(0x7f42d0159830 legacy=0x7f42d00cac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42d5465640 1 -- 172.21.15.125:0/1710060506 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d015a2e0 con 0x7f42d0153c60 2023-10-30T12:29:07.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.231+0000 7f42c7fff640 1 -- 172.21.15.125:0/1710060506 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826730141 0 0) 0x7f42d00cb780 con 0x7f42d0159830 2023-10-30T12:29:07.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.235+0000 7f42d5465640 1 --2- 172.21.15.125:0/1710060506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d0153c60 0x7f42d00ca360 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f42d00b1950 tx=0x7f42cc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ee0b77a6ad91a3c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.235+0000 7f42c7fff640 1 -- 172.21.15.125:0/1710060506 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0ab600 con 0x7f42d0153c60 2023-10-30T12:29:07.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.235+0000 7f42c7fff640 1 -- 172.21.15.125:0/1710060506 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42cc0a7050 con 0x7f42d0153c60 2023-10-30T12:29:07.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.235+0000 7f42c7fff640 1 -- 172.21.15.125:0/1710060506 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0ab980 con 0x7f42d0153c60 2023-10-30T12:29:07.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.235+0000 7f42d76c9640 1 -- 172.21.15.125:0/1710060506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d0153c60 msgr2=0x7f42d00ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.235+0000 7f42d76c9640 1 --2- 172.21.15.125:0/1710060506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d0153c60 0x7f42d00ca360 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f42d00b1950 tx=0x7f42cc01c500 comp rx=0 tx=0).stop 2023-10-30T12:29:07.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.235+0000 7f42d76c9640 1 -- 172.21.15.125:0/1710060506 shutdown_connections 2023-10-30T12:29:07.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.235+0000 7f42d76c9640 1 --2- 172.21.15.125:0/1710060506 >> v2:172.21.15.136:3300/0 conn(0x7f42d00c4430 0x7f42d00cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.235+0000 7f42d76c9640 1 --2- 172.21.15.125:0/1710060506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d0153c60 0x7f42d00ca360 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.235+0000 7f42d76c9640 1 -- 172.21.15.125:0/1710060506 >> 172.21.15.125:0/1710060506 conn(0x7f42d006bfe0 msgr2=0x7f42d006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.235+0000 7f42d76c9640 1 -- 172.21.15.125:0/1710060506 shutdown_connections 2023-10-30T12:29:07.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.235+0000 7f42d76c9640 1 -- 172.21.15.125:0/1710060506 wait complete. 2023-10-30T12:29:07.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d76c9640 1 Processor -- start 2023-10-30T12:29:07.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d76c9640 1 -- start start 2023-10-30T12:29:07.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d76c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d00c4430 0x7f42d00ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d76c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d0153c60 0x7f42d00b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d76c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d0159830 0x7f42d01f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d76c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d00b4af0 con 0x7f42d0159830 2023-10-30T12:29:07.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d76c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d01f18b0 con 0x7f42d00c4430 2023-10-30T12:29:07.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d76c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d01512d0 con 0x7f42d0153c60 2023-10-30T12:29:07.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d5c66640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d0159830 0x7f42d01f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d5c66640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d0159830 0x7f42d01f22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54682/0 (socket says 172.21.15.125:54682) 2023-10-30T12:29:07.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d5c66640 1 -- 172.21.15.125:0/735117121 learned_addr learned my addr 172.21.15.125:0/735117121 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d5465640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d00c4430 0x7f42d00ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d4c64640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d0153c60 0x7f42d00b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d5c66640 1 -- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d0153c60 msgr2=0x7f42d00b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d5c66640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d0153c60 0x7f42d00b3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d5c66640 1 -- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d00c4430 msgr2=0x7f42d00ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d5c66640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d00c4430 0x7f42d00ce580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d5c66640 1 -- 172.21.15.125:0/735117121 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc0094d0 con 0x7f42d0159830 2023-10-30T12:29:07.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d4c64640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d0153c60 0x7f42d00b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:07.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d5c66640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d0159830 0x7f42d01f22c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f42c800c750 tx=0x7f42c800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42c5ffb640 1 -- 172.21.15.125:0/735117121 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8003db0 con 0x7f42d0159830 2023-10-30T12:29:07.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42c5ffb640 1 -- 172.21.15.125:0/735117121 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c800ce60 con 0x7f42d0159830 2023-10-30T12:29:07.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42c5ffb640 1 -- 172.21.15.125:0/735117121 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8005270 con 0x7f42d0159830 2023-10-30T12:29:07.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d76c9640 1 -- 172.21.15.125:0/735117121 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d01f1e30 con 0x7f42d0159830 2023-10-30T12:29:07.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.239+0000 7f42d76c9640 1 -- 172.21.15.125:0/735117121 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d01f2800 con 0x7f42d0159830 2023-10-30T12:29:07.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.243+0000 7f42c5ffb640 1 -- 172.21.15.125:0/735117121 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f42c80053d0 con 0x7f42d0159830 2023-10-30T12:29:07.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.243+0000 7f42c5ffb640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42b40664d0 0x7f42b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.243+0000 7f42c5ffb640 1 -- 172.21.15.125:0/735117121 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42c8083590 con 0x7f42d0159830 2023-10-30T12:29:07.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.243+0000 7f42d5465640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42b40664d0 0x7f42b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.243+0000 7f42daa49000 1 Processor -- start 2023-10-30T12:29:07.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.243+0000 7f42daa49000 1 -- start start 2023-10-30T12:29:07.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42d5465640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42b40664d0 0x7f42b4068990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0ad000 tx=0x7f42cc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88ab5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6130 0x559a88ab6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6a50 0x559a88ac0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a88869cc0 con 0x559a88ab5810 2023-10-30T12:29:07.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a888cc920 con 0x559a88ab6a50 2023-10-30T12:29:07.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x559a888c9e50 con 0x559a88ab6130 2023-10-30T12:29:07.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42d5465640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88ab5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42d5465640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88ab5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54686/0 (socket says 172.21.15.125:54686) 2023-10-30T12:29:07.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42d5465640 1 -- 172.21.15.125:0/95624739 learned_addr learned my addr 172.21.15.125:0/95624739 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42d4c64640 1 --2- 172.21.15.125:0/95624739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6130 0x559a88ab6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42d5c66640 1 --2- 172.21.15.125:0/95624739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6a50 0x559a88ac0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42d5465640 1 -- 172.21.15.125:0/95624739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6130 msgr2=0x559a88ab6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42d5465640 1 --2- 172.21.15.125:0/95624739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6130 0x559a88ab6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42d5465640 1 -- 172.21.15.125:0/95624739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6a50 msgr2=0x559a88ac0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42d5465640 1 --2- 172.21.15.125:0/95624739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6a50 0x559a88ac0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42d5465640 1 -- 172.21.15.125:0/95624739 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a885be890 con 0x559a88ab5810 2023-10-30T12:29:07.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42d5c66640 1 --2- 172.21.15.125:0/95624739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6a50 0x559a88ac0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:07.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42d5465640 1 --2- 172.21.15.125:0/95624739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88ab5bf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f42cc097dd0 tx=0x7f42cc004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c372e5844aac3d20 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42b2ffd640 1 -- 172.21.15.125:0/95624739 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0b4400 con 0x559a88ab5810 2023-10-30T12:29:07.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42b2ffd640 1 -- 172.21.15.125:0/95624739 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42cc0bb040 con 0x559a88ab5810 2023-10-30T12:29:07.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42b2ffd640 1 -- 172.21.15.125:0/95624739 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0b4760 con 0x559a88ab5810 2023-10-30T12:29:07.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 -- 172.21.15.125:0/95624739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 msgr2=0x559a88ab5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 --2- 172.21.15.125:0/95624739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88ab5bf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f42cc097dd0 tx=0x7f42cc004740 comp rx=0 tx=0).stop 2023-10-30T12:29:07.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 -- 172.21.15.125:0/95624739 shutdown_connections 2023-10-30T12:29:07.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 --2- 172.21.15.125:0/95624739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6a50 0x559a88ac0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 --2- 172.21.15.125:0/95624739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6130 0x559a88ab6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 --2- 172.21.15.125:0/95624739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88ab5bf0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 -- 172.21.15.125:0/95624739 >> 172.21.15.125:0/95624739 conn(0x559a88a6bfc0 msgr2=0x559a88ab35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 -- 172.21.15.125:0/95624739 shutdown_connections 2023-10-30T12:29:07.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 -- 172.21.15.125:0/95624739 wait complete. 2023-10-30T12:29:07.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 -- start start 2023-10-30T12:29:07.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.247+0000 7f42daa49000 1 Processor -- start 2023-10-30T12:29:07.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88aaff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6130 0x559a88ab0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88abbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a888c5740 con 0x559a88ab5810 2023-10-30T12:29:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a88884d10 con 0x559a88ab6130 2023-10-30T12:29:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x559a888849b0 con 0x559a88ab6a50 2023-10-30T12:29:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 10 client.0 _ll_register_callbacks cb 0x559a88a60390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T12:29:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42d5465640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88aaff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42d5465640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88aaff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54690/0 (socket says 172.21.15.125:54690) 2023-10-30T12:29:07.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42d5465640 1 -- 172.21.15.125:0/4222609205 learned_addr learned my addr 172.21.15.125:0/4222609205 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42d4c64640 1 --2- 172.21.15.125:0/4222609205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6130 0x559a88ab0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42d5c66640 1 --2- 172.21.15.125:0/4222609205 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88abbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42d5465640 1 -- 172.21.15.125:0/4222609205 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 msgr2=0x559a88abbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42d5465640 1 --2- 172.21.15.125:0/4222609205 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88abbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42d5465640 1 -- 172.21.15.125:0/4222609205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6130 msgr2=0x559a88ab0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42d5465640 1 --2- 172.21.15.125:0/4222609205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6130 0x559a88ab0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42d5465640 1 -- 172.21.15.125:0/4222609205 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d015a2e0 con 0x559a88ab5810 2023-10-30T12:29:07.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42d5c66640 1 --2- 172.21.15.125:0/4222609205 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88abbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:07.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42d5465640 1 --2- 172.21.15.125:0/4222609205 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88aaff50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f42cc01c9e0 tx=0x7f42cc0046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913b96ad7956b72b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42abfff640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2023-10-30T12:29:07.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42abfff640 1 -- 172.21.15.125:0/4222609205 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0b4d90 con 0x559a88ab5810 2023-10-30T12:29:07.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42abfff640 1 -- 172.21.15.125:0/4222609205 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42cc0c8420 con 0x559a88ab5810 2023-10-30T12:29:07.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 1 -- 172.21.15.125:0/4222609205 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559a88879370 con 0x559a88ab5810 2023-10-30T12:29:07.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42abfff640 1 -- 172.21.15.125:0/4222609205 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42cc0c85e0 con 0x559a88ab5810 2023-10-30T12:29:07.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 10 client.4701 fetch_fsmap learned FSMap version 12 2023-10-30T12:29:07.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 1 -- 172.21.15.125:0/4222609205 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x559a88887460 con 0x559a88ab5810 2023-10-30T12:29:07.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42abfff640 1 -- 172.21.15.125:0/4222609205 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559a88879370 con 0x559a88ab5810 2023-10-30T12:29:07.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42abfff640 1 -- 172.21.15.125:0/4222609205 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f42cc0be070 con 0x559a88ab5810 2023-10-30T12:29:07.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 10 client.4701 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T12:29:07.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 10 client.4701 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-30T12:29:07.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 20 client.4701 populate_metadata read hostname 'smithi125' 2023-10-30T12:29:07.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 1 --2- 172.21.15.125:0/4222609205 >> [v2:172.21.15.136:6838/2250748765,v1:172.21.15.136:6839/2250748765] conn(0x559a88abc9a0 0x559a88abcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 4 client.4701 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:29:07.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42daa49000 1 -- 172.21.15.125:0/4222609205 --> [v2:172.21.15.136:6838/2250748765,v1:172.21.15.136:6839/2250748765] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559a8885f2d0 con 0x559a88abc9a0 2023-10-30T12:29:07.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42d4c64640 1 --2- 172.21.15.125:0/4222609205 >> [v2:172.21.15.136:6838/2250748765,v1:172.21.15.136:6839/2250748765] conn(0x559a88abc9a0 0x559a88abcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42abfff640 10 client.4701 ms_handle_connect on v2:172.21.15.136:6838/2250748765 2023-10-30T12:29:07.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.251+0000 7f42d4c64640 1 --2- 172.21.15.125:0/4222609205 >> [v2:172.21.15.136:6838/2250748765,v1:172.21.15.136:6839/2250748765] conn(0x559a88abc9a0 0x559a88abcd80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=640fef82142c7076 server_cookie=af766bb5515bc423 in_seq=0 out_seq=0 2023-10-30T12:29:07.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42abfff640 1 -- 172.21.15.125:0/4222609205 <== mds.1 v2:172.21.15.136:6838/2250748765 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x559a8885f2d0 con 0x559a88abc9a0 2023-10-30T12:29:07.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42abfff640 10 client.4701 handle_command_reply: tid=0 2023-10-30T12:29:07.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42daa49000 1 client.4701 shutdown 2023-10-30T12:29:07.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42daa49000 1 -- 172.21.15.125:0/4222609205 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 msgr2=0x559a88aaff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42daa49000 1 --2- 172.21.15.125:0/4222609205 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88aaff50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f42cc01c9e0 tx=0x7f42cc0046d0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42daa49000 1 -- 172.21.15.125:0/4222609205 shutdown_connections 2023-10-30T12:29:07.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42daa49000 1 --2- 172.21.15.125:0/4222609205 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88abbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42daa49000 1 --2- 172.21.15.125:0/4222609205 >> [v2:172.21.15.136:6838/2250748765,v1:172.21.15.136:6839/2250748765] conn(0x559a88abc9a0 0x559a88abcd80 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42daa49000 1 --2- 172.21.15.125:0/4222609205 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6130 0x559a88ab0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42daa49000 1 --2- 172.21.15.125:0/4222609205 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88aaff50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42abfff640 0 client.4701 ms_handle_reset on v2:172.21.15.136:6838/2250748765 2023-10-30T12:29:07.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42daa49000 1 -- 172.21.15.125:0/4222609205 >> 172.21.15.125:0/4222609205 conn(0x559a88a6bfc0 msgr2=0x559a88ab5090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42daa49000 1 -- 172.21.15.125:0/4222609205 shutdown_connections 2023-10-30T12:29:07.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42daa49000 1 -- 172.21.15.125:0/4222609205 wait complete. 2023-10-30T12:29:07.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.255+0000 7f42daa49000 20 client.4701 trim_cache size 0 max 16384 2023-10-30T12:29:07.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 Processor -- start 2023-10-30T12:29:07.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 -- start start 2023-10-30T12:29:07.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88a79c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6130 0x559a88a7a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88ab94e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a8857f940 con 0x559a88ab5810 2023-10-30T12:29:07.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a88ab5340 con 0x559a88ab6130 2023-10-30T12:29:07.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x559a888c2f50 con 0x559a88ab6a50 2023-10-30T12:29:07.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42d5465640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88a79c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42d5465640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88a79c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54692/0 (socket says 172.21.15.125:54692) 2023-10-30T12:29:07.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42d5465640 1 -- 172.21.15.125:0/3624903398 learned_addr learned my addr 172.21.15.125:0/3624903398 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42d4c64640 1 --2- 172.21.15.125:0/3624903398 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6130 0x559a88a7a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42d5c66640 1 --2- 172.21.15.125:0/3624903398 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88ab94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42d5465640 1 -- 172.21.15.125:0/3624903398 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 msgr2=0x559a88ab94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42d5465640 1 --2- 172.21.15.125:0/3624903398 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88ab94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42d5465640 1 -- 172.21.15.125:0/3624903398 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6130 msgr2=0x559a88a7a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42d5465640 1 --2- 172.21.15.125:0/3624903398 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6130 0x559a88a7a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42d5465640 1 -- 172.21.15.125:0/3624903398 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a8886d760 con 0x559a88ab5810 2023-10-30T12:29:07.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42d5c66640 1 --2- 172.21.15.125:0/3624903398 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88ab94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:07.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42d5465640 1 --2- 172.21.15.125:0/3624903398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88a79c00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0ab6f0 tx=0x7f42cc0bc000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38d344450901c888 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42b37fe640 1 -- 172.21.15.125:0/3624903398 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0b4d90 con 0x559a88ab5810 2023-10-30T12:29:07.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42b37fe640 1 -- 172.21.15.125:0/3624903398 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42cc0bb040 con 0x559a88ab5810 2023-10-30T12:29:07.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42b37fe640 1 -- 172.21.15.125:0/3624903398 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc0c8620 con 0x559a88ab5810 2023-10-30T12:29:07.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 -- 172.21.15.125:0/3624903398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 msgr2=0x559a88a79c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 --2- 172.21.15.125:0/3624903398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88a79c00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0ab6f0 tx=0x7f42cc0bc000 comp rx=0 tx=0).stop 2023-10-30T12:29:07.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 -- 172.21.15.125:0/3624903398 shutdown_connections 2023-10-30T12:29:07.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 --2- 172.21.15.125:0/3624903398 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88ab94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 --2- 172.21.15.125:0/3624903398 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab6130 0x559a88a7a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 --2- 172.21.15.125:0/3624903398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab5810 0x559a88a79c00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 -- 172.21.15.125:0/3624903398 >> 172.21.15.125:0/3624903398 conn(0x559a88a6bfc0 msgr2=0x559a88abdfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 -- 172.21.15.125:0/3624903398 shutdown_connections 2023-10-30T12:29:07.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 -- 172.21.15.125:0/3624903398 wait complete. 2023-10-30T12:29:07.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.271+0000 7f42daa49000 1 -- start start 2023-10-30T12:29:07.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 1 Processor -- start 2023-10-30T12:29:07.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab5810 0x559a88b15dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab6130 0x559a88b12670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88b12bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a888736c0 con 0x559a88ab6130 2023-10-30T12:29:07.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a888c4460 con 0x559a88ab5810 2023-10-30T12:29:07.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x559a88ab7060 con 0x559a88ab6a50 2023-10-30T12:29:07.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 10 client.0 _ll_register_callbacks cb 0x7f42cc0ae070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T12:29:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42d4c64640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab6130 0x559a88b12670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42d4c64640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab6130 0x559a88b12670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54706/0 (socket says 172.21.15.125:54706) 2023-10-30T12:29:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42d4c64640 1 -- 172.21.15.125:0/1465493332 learned_addr learned my addr 172.21.15.125:0/1465493332 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42d5465640 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab5810 0x559a88b15dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42d5c66640 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88b12bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42d4c64640 1 -- 172.21.15.125:0/1465493332 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 msgr2=0x559a88b12bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42d4c64640 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88b12bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42d4c64640 1 -- 172.21.15.125:0/1465493332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab5810 msgr2=0x559a88b15dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42d4c64640 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab5810 0x559a88b15dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42d4c64640 1 -- 172.21.15.125:0/1465493332 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559a88887460 con 0x559a88ab6130 2023-10-30T12:29:07.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42d5c66640 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88b12bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:07.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42d5465640 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab5810 0x559a88b15dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:07.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42d4c64640 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab6130 0x559a88b12670 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f42c000e950 tx=0x7f42c0016430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5faf8f33ff0869c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42abfff640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2023-10-30T12:29:07.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42abfff640 1 -- 172.21.15.125:0/1465493332 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c0023400 con 0x559a88ab6130 2023-10-30T12:29:07.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42abfff640 1 -- 172.21.15.125:0/1465493332 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42c0023560 con 0x559a88ab6130 2023-10-30T12:29:07.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 1 -- 172.21.15.125:0/1465493332 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559a8885d6f0 con 0x559a88ab6130 2023-10-30T12:29:07.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42abfff640 1 -- 172.21.15.125:0/1465493332 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f42c0023720 con 0x559a88ab6130 2023-10-30T12:29:07.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 10 client.4707 fetch_fsmap learned FSMap version 12 2023-10-30T12:29:07.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 1 -- 172.21.15.125:0/1465493332 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x559a885becc0 con 0x559a88ab6130 2023-10-30T12:29:07.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42abfff640 1 -- 172.21.15.125:0/1465493332 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x559a8885d6f0 con 0x559a88ab6130 2023-10-30T12:29:07.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42abfff640 1 -- 172.21.15.125:0/1465493332 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f42c0023880 con 0x559a88ab6130 2023-10-30T12:29:07.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 10 client.4707 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T12:29:07.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 10 client.4707 resolve_mds: resolved name 'f' to daemon mds.f 2023-10-30T12:29:07.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 20 client.4707 populate_metadata read hostname 'smithi125' 2023-10-30T12:29:07.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.136:6838/2250748765,v1:172.21.15.136:6839/2250748765] conn(0x559a88b1f7c0 0x559a88b13cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 4 client.4707 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T12:29:07.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42daa49000 1 -- 172.21.15.125:0/1465493332 --> [v2:172.21.15.136:6838/2250748765,v1:172.21.15.136:6839/2250748765] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559a888be720 con 0x559a88b1f7c0 2023-10-30T12:29:07.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.275+0000 7f42d5465640 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.136:6838/2250748765,v1:172.21.15.136:6839/2250748765] conn(0x559a88b1f7c0 0x559a88b13cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42abfff640 10 client.4707 ms_handle_connect on v2:172.21.15.136:6838/2250748765 2023-10-30T12:29:07.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42d5465640 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.136:6838/2250748765,v1:172.21.15.136:6839/2250748765] conn(0x559a88b1f7c0 0x559a88b13cd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=55ede550131d97dd server_cookie=4320deceb9855ea5 in_seq=0 out_seq=0 2023-10-30T12:29:07.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42abfff640 1 -- 172.21.15.125:0/1465493332 <== mds.1 v2:172.21.15.136:6838/2250748765 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x559a888be720 con 0x559a88b1f7c0 2023-10-30T12:29:07.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42abfff640 10 client.4707 handle_command_reply: tid=0 2023-10-30T12:29:07.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42daa49000 1 client.4707 shutdown 2023-10-30T12:29:07.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42daa49000 1 -- 172.21.15.125:0/1465493332 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab6130 msgr2=0x559a88b12670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42daa49000 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab6130 0x559a88b12670 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f42c000e950 tx=0x7f42c0016430 comp rx=0 tx=0).stop 2023-10-30T12:29:07.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42daa49000 1 -- 172.21.15.125:0/1465493332 shutdown_connections 2023-10-30T12:29:07.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42daa49000 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x559a88ab6a50 0x559a88b12bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42daa49000 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x559a88ab6130 0x559a88b12670 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42daa49000 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.136:6838/2250748765,v1:172.21.15.136:6839/2250748765] conn(0x559a88b1f7c0 0x559a88b13cd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42daa49000 1 --2- 172.21.15.125:0/1465493332 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x559a88ab5810 0x559a88b15dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42daa49000 1 -- 172.21.15.125:0/1465493332 >> 172.21.15.125:0/1465493332 conn(0x559a88a6bfc0 msgr2=0x559a88a7a6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42abfff640 0 client.4707 ms_handle_reset on v2:172.21.15.136:6838/2250748765 2023-10-30T12:29:07.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42daa49000 1 -- 172.21.15.125:0/1465493332 shutdown_connections 2023-10-30T12:29:07.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42daa49000 1 -- 172.21.15.125:0/1465493332 wait complete. 2023-10-30T12:29:07.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42daa49000 20 client.4707 trim_cache size 0 max 16384 2023-10-30T12:29:07.293 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2023-10-30T12:29:07.293 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_fsid": "714f08a3-69cf-4c1b-8fab-64b3ba5e42e9", 2023-10-30T12:29:07.294 INFO:teuthology.orchestra.run.smithi125.stdout: "whoami": 1, 2023-10-30T12:29:07.294 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4285, 2023-10-30T12:29:07.294 INFO:teuthology.orchestra.run.smithi125.stdout: "want_state": "up:active", 2023-10-30T12:29:07.294 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "up:active", 2023-10-30T12:29:07.294 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs", 2023-10-30T12:29:07.295 INFO:teuthology.orchestra.run.smithi125.stdout: "rank_uptime": 17.168516874000002, 2023-10-30T12:29:07.295 INFO:teuthology.orchestra.run.smithi125.stdout: "mdsmap_epoch": 12, 2023-10-30T12:29:07.295 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch": 22, 2023-10-30T12:29:07.295 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch_barrier": 22, 2023-10-30T12:29:07.295 INFO:teuthology.orchestra.run.smithi125.stdout: "uptime": 28.917361669999998 2023-10-30T12:29:07.295 INFO:teuthology.orchestra.run.smithi125.stdout:} 2023-10-30T12:29:07.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42d76c9640 1 -- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42b40664d0 msgr2=0x7f42b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42d76c9640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42b40664d0 0x7f42b4068990 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f42cc0ad000 tx=0x7f42cc008040 comp rx=0 tx=0).stop 2023-10-30T12:29:07.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42d76c9640 1 -- 172.21.15.125:0/735117121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d0159830 msgr2=0x7f42d01f22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42d76c9640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d0159830 0x7f42d01f22c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f42c800c750 tx=0x7f42c800cc10 comp rx=0 tx=0).stop 2023-10-30T12:29:07.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42d76c9640 1 -- 172.21.15.125:0/735117121 shutdown_connections 2023-10-30T12:29:07.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42d76c9640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42b40664d0 0x7f42b4068990 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42d76c9640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d0159830 0x7f42d01f22c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42d76c9640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d0153c60 0x7f42d00b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42d76c9640 1 --2- 172.21.15.125:0/735117121 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d00c4430 0x7f42d00ce580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42d76c9640 1 -- 172.21.15.125:0/735117121 >> 172.21.15.125:0/735117121 conn(0x7f42d006bfe0 msgr2=0x7f42d0158d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42d76c9640 1 -- 172.21.15.125:0/735117121 shutdown_connections 2023-10-30T12:29:07.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.279+0000 7f42d76c9640 1 -- 172.21.15.125:0/735117121 wait complete. 2023-10-30T12:29:07.300 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2023-10-30T12:29:07.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.435+0000 7f8f4919d640 1 Processor -- start 2023-10-30T12:29:07.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.435+0000 7f8f4919d640 1 -- start start 2023-10-30T12:29:07.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f4919d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f4415a000 0x7f8f4415a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f4919d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f440cb570 0x7f8f440cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f4919d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4415ce10 con 0x7f8f44153d10 2023-10-30T12:29:07.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f4919d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f44159820 con 0x7f8f4415a000 2023-10-30T12:29:07.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f4919d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f440c8430 con 0x7f8f440cb570 2023-10-30T12:29:07.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f42d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8f44153d10 0x7f8f440c5f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35174/0 (socket says 172.21.15.125:35174) 2023-10-30T12:29:07.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f42d9d640 1 -- 172.21.15.125:0/4021267557 learned_addr learned my addr 172.21.15.125:0/4021267557 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:29:07.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/4021267557 >> v2:172.21.15.125:3300/0 conn(0x7f8f4415a000 0x7f8f4415a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f4359e640 1 --2- 172.21.15.125:0/4021267557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f440cb570 0x7f8f440cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f3a59c640 1 -- 172.21.15.125:0/4021267557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f440cb570 msgr2=0x7f8f440cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/4021267557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f440cb570 0x7f8f440cb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f3a59c640 1 -- 172.21.15.125:0/4021267557 >> v1:172.21.15.125:6789/0 conn(0x7f8f44153d10 legacy=0x7f8f440c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f3a59c640 1 -- 172.21.15.125:0/4021267557 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f440c8710 con 0x7f8f4415a000 2023-10-30T12:29:07.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f4359e640 1 --2- 172.21.15.125:0/4021267557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f440cb570 0x7f8f440cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:07.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f4259c640 1 -- 172.21.15.125:0/4021267557 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756996910 0 0) 0x7f8f4415ce10 con 0x7f8f44153d10 2023-10-30T12:29:07.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.439+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/4021267557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f4415a000 0x7f8f4415a420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0097c0 tx=0x7f8f2c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68a0577a4b52798 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4259c640 1 -- 172.21.15.125:0/4021267557 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c0ab650 con 0x7f8f4415a000 2023-10-30T12:29:07.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4259c640 1 -- 172.21.15.125:0/4021267557 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c0ab7b0 con 0x7f8f4415a000 2023-10-30T12:29:07.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4259c640 1 -- 172.21.15.125:0/4021267557 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c0abb40 con 0x7f8f4415a000 2023-10-30T12:29:07.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 -- 172.21.15.125:0/4021267557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f4415a000 msgr2=0x7f8f4415a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 --2- 172.21.15.125:0/4021267557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f4415a000 0x7f8f4415a420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0097c0 tx=0x7f8f2c01c450 comp rx=0 tx=0).stop 2023-10-30T12:29:07.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 -- 172.21.15.125:0/4021267557 shutdown_connections 2023-10-30T12:29:07.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 --2- 172.21.15.125:0/4021267557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f4415a000 0x7f8f4415a420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 --2- 172.21.15.125:0/4021267557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f440cb570 0x7f8f440cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 -- 172.21.15.125:0/4021267557 >> 172.21.15.125:0/4021267557 conn(0x7f8f4406c6b0 msgr2=0x7f8f44069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 -- 172.21.15.125:0/4021267557 shutdown_connections 2023-10-30T12:29:07.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 -- 172.21.15.125:0/4021267557 wait complete. 2023-10-30T12:29:07.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 Processor -- start 2023-10-30T12:29:07.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 -- start start 2023-10-30T12:29:07.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440cb570 0x7f8f4415bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f44153d10 0x7f8f44148400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f4415a000 0x7f8f44148940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f44152880 con 0x7f8f440cb570 2023-10-30T12:29:07.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f44152460 con 0x7f8f4415a000 2023-10-30T12:29:07.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f4919d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f44150ec0 con 0x7f8f44153d10 2023-10-30T12:29:07.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f42d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440cb570 0x7f8f4415bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f42d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440cb570 0x7f8f4415bed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54732/0 (socket says 172.21.15.125:54732) 2023-10-30T12:29:07.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.443+0000 7f8f42d9d640 1 -- 172.21.15.125:0/2377272518 learned_addr learned my addr 172.21.15.125:0/2377272518 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f4359e640 1 --2- 172.21.15.125:0/2377272518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f4415a000 0x7f8f44148940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/2377272518 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f44153d10 0x7f8f44148400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f42d9d640 1 -- 172.21.15.125:0/2377272518 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f44153d10 msgr2=0x7f8f44148400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f42d9d640 1 --2- 172.21.15.125:0/2377272518 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f44153d10 0x7f8f44148400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f42d9d640 1 -- 172.21.15.125:0/2377272518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f4415a000 msgr2=0x7f8f44148940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f42d9d640 1 --2- 172.21.15.125:0/2377272518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f4415a000 0x7f8f44148940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f42d9d640 1 -- 172.21.15.125:0/2377272518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2c009420 con 0x7f8f440cb570 2023-10-30T12:29:07.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f42d9d640 1 --2- 172.21.15.125:0/2377272518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440cb570 0x7f8f4415bed0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400c480 tx=0x7f8f3400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f3bfff640 1 -- 172.21.15.125:0/2377272518 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f340074e0 con 0x7f8f440cb570 2023-10-30T12:29:07.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f3bfff640 1 -- 172.21.15.125:0/2377272518 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f34007660 con 0x7f8f440cb570 2023-10-30T12:29:07.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f3bfff640 1 -- 172.21.15.125:0/2377272518 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f34004a30 con 0x7f8f440cb570 2023-10-30T12:29:07.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f4919d640 1 -- 172.21.15.125:0/2377272518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f441511a0 con 0x7f8f440cb570 2023-10-30T12:29:07.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f4919d640 1 -- 172.21.15.125:0/2377272518 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f4414e940 con 0x7f8f440cb570 2023-10-30T12:29:07.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f3bfff640 1 -- 172.21.15.125:0/2377272518 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f34004b90 con 0x7f8f440cb570 2023-10-30T12:29:07.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f3bfff640 1 --2- 172.21.15.125:0/2377272518 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f240664d0 0x7f8f24068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.447+0000 7f8f3bfff640 1 -- 172.21.15.125:0/2377272518 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f34082ff0 con 0x7f8f440cb570 2023-10-30T12:29:07.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/2377272518 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f240664d0 0x7f8f24068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 Processor -- start 2023-10-30T12:29:07.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 -- start start 2023-10-30T12:29:07.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/2377272518 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f240664d0 0x7f8f24068990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0af000 tx=0x7f8f2c0a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f87810 0x55ae85f87bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f88130 0x55ae85f88510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88a50 0x55ae85f92aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae85d3bcc0 con 0x55ae85f88a50 2023-10-30T12:29:07.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae85d9e920 con 0x55ae85f88130 2023-10-30T12:29:07.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ae85d9be50 con 0x55ae85f87810 2023-10-30T12:29:07.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4359e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88a50 0x55ae85f92aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4359e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88a50 0x55ae85f92aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54736/0 (socket says 172.21.15.125:54736) 2023-10-30T12:29:07.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4359e640 1 -- 172.21.15.125:0/3301368910 learned_addr learned my addr 172.21.15.125:0/3301368910 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f42d9d640 1 --2- 172.21.15.125:0/3301368910 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f87810 0x55ae85f87bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/3301368910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f88130 0x55ae85f88510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4359e640 1 -- 172.21.15.125:0/3301368910 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f87810 msgr2=0x55ae85f87bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4359e640 1 --2- 172.21.15.125:0/3301368910 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f87810 0x55ae85f87bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4359e640 1 -- 172.21.15.125:0/3301368910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f88130 msgr2=0x55ae85f88510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4359e640 1 --2- 172.21.15.125:0/3301368910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f88130 0x55ae85f88510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4359e640 1 -- 172.21.15.125:0/3301368910 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae85a90890 con 0x55ae85f88a50 2023-10-30T12:29:07.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4359e640 1 --2- 172.21.15.125:0/3301368910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88a50 0x55ae85f92aa0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c007fc0 tx=0x7f8f3c00e810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fdad872920787b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f38d99640 1 -- 172.21.15.125:0/3301368910 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c00f6b0 con 0x55ae85f88a50 2023-10-30T12:29:07.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f38d99640 1 -- 172.21.15.125:0/3301368910 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f3c013070 con 0x55ae85f88a50 2023-10-30T12:29:07.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f38d99640 1 -- 172.21.15.125:0/3301368910 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3c00fa10 con 0x55ae85f88a50 2023-10-30T12:29:07.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 -- 172.21.15.125:0/3301368910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88a50 msgr2=0x55ae85f92aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/3301368910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88a50 0x55ae85f92aa0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c007fc0 tx=0x7f8f3c00e810 comp rx=0 tx=0).stop 2023-10-30T12:29:07.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 -- 172.21.15.125:0/3301368910 shutdown_connections 2023-10-30T12:29:07.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/3301368910 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88a50 0x55ae85f92aa0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/3301368910 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f88130 0x55ae85f88510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/3301368910 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f87810 0x55ae85f87bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 -- 172.21.15.125:0/3301368910 >> 172.21.15.125:0/3301368910 conn(0x55ae85f3dfc0 msgr2=0x55ae85f855d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 -- 172.21.15.125:0/3301368910 shutdown_connections 2023-10-30T12:29:07.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.451+0000 7f8f4c52b000 1 -- 172.21.15.125:0/3301368910 wait complete. 2023-10-30T12:29:07.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 1 -- start start 2023-10-30T12:29:07.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 1 Processor -- start 2023-10-30T12:29:07.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 0x55ae85f81b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85f820a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 0x55ae85f825e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae85d97740 con 0x55ae85f88130 2023-10-30T12:29:07.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae85d56d10 con 0x55ae85f87810 2023-10-30T12:29:07.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ae85d569b0 con 0x55ae85f88a50 2023-10-30T12:29:07.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 10 client.0 _ll_register_callbacks cb 0x55ae85f32390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T12:29:07.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f3a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85f820a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f3a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85f820a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54738/0 (socket says 172.21.15.125:54738) 2023-10-30T12:29:07.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f3a59c640 1 -- 172.21.15.125:0/889068994 learned_addr learned my addr 172.21.15.125:0/889068994 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f42d9d640 1 --2- 172.21.15.125:0/889068994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 0x55ae85f81b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4359e640 1 --2- 172.21.15.125:0/889068994 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 0x55ae85f825e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f3a59c640 1 -- 172.21.15.125:0/889068994 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 msgr2=0x55ae85f825e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/889068994 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 0x55ae85f825e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f3a59c640 1 -- 172.21.15.125:0/889068994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 msgr2=0x55ae85f81b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/889068994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 0x55ae85f81b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f3a59c640 1 -- 172.21.15.125:0/889068994 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f3c007bf0 con 0x55ae85f88130 2023-10-30T12:29:07.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4359e640 1 --2- 172.21.15.125:0/889068994 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 0x55ae85f825e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:07.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/889068994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85f820a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c01a080 tx=0x7f8f2c01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4aee37c38f93e04 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2023-10-30T12:29:07.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f21ffb640 1 -- 172.21.15.125:0/889068994 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c003d70 con 0x55ae85f88130 2023-10-30T12:29:07.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 1 -- 172.21.15.125:0/889068994 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ae85d4b370 con 0x55ae85f88130 2023-10-30T12:29:07.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f21ffb640 1 -- 172.21.15.125:0/889068994 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f2c0cd440 con 0x55ae85f88130 2023-10-30T12:29:07.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f21ffb640 1 -- 172.21.15.125:0/889068994 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f2c0cd600 con 0x55ae85f88130 2023-10-30T12:29:07.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 10 client.4722 fetch_fsmap learned FSMap version 12 2023-10-30T12:29:07.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 1 -- 172.21.15.125:0/889068994 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55ae85d59460 con 0x55ae85f88130 2023-10-30T12:29:07.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f21ffb640 1 -- 172.21.15.125:0/889068994 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ae85d4b370 con 0x55ae85f88130 2023-10-30T12:29:07.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f21ffb640 1 -- 172.21.15.125:0/889068994 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f8f2c0cd760 con 0x55ae85f88130 2023-10-30T12:29:07.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 10 client.4722 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T12:29:07.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 10 client.4722 resolve_mds: resolved name 'e' to daemon mds.e 2023-10-30T12:29:07.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 20 client.4722 populate_metadata read hostname 'smithi125' 2023-10-30T12:29:07.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/889068994 >> [v2:172.21.15.125:6833/2017382874,v1:172.21.15.125:6836/2017382874] conn(0x55ae85f9b260 0x55ae85f9d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 4 client.4722 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-30T12:29:07.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f4c52b000 1 -- 172.21.15.125:0/889068994 --> [v2:172.21.15.125:6833/2017382874,v1:172.21.15.125:6836/2017382874] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ae85d312d0 con 0x55ae85f9b260 2023-10-30T12:29:07.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f42d9d640 1 --2- 172.21.15.125:0/889068994 >> [v2:172.21.15.125:6833/2017382874,v1:172.21.15.125:6836/2017382874] conn(0x55ae85f9b260 0x55ae85f9d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.455+0000 7f8f42d9d640 1 --2- 172.21.15.125:0/889068994 >> [v2:172.21.15.125:6833/2017382874,v1:172.21.15.125:6836/2017382874] conn(0x55ae85f9b260 0x55ae85f9d650 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa1295714bdc19f5 server_cookie=d38dea1ec5d6f374 in_seq=0 out_seq=0 2023-10-30T12:29:07.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f21ffb640 10 client.4722 ms_handle_connect on v2:172.21.15.125:6833/2017382874 2023-10-30T12:29:07.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f21ffb640 1 -- 172.21.15.125:0/889068994 <== mds.0 v2:172.21.15.125:6833/2017382874 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55ae85d312d0 con 0x55ae85f9b260 2023-10-30T12:29:07.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f21ffb640 10 client.4722 handle_command_reply: tid=0 2023-10-30T12:29:07.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f4c52b000 1 client.4722 shutdown 2023-10-30T12:29:07.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f4c52b000 1 -- 172.21.15.125:0/889068994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 msgr2=0x55ae85f820a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/889068994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85f820a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c01a080 tx=0x7f8f2c01ac90 comp rx=0 tx=0).stop 2023-10-30T12:29:07.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f4c52b000 1 -- 172.21.15.125:0/889068994 shutdown_connections 2023-10-30T12:29:07.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/889068994 >> [v2:172.21.15.125:6833/2017382874,v1:172.21.15.125:6836/2017382874] conn(0x55ae85f9b260 0x55ae85f9d650 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/889068994 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 0x55ae85f825e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/889068994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85f820a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/889068994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 0x55ae85f81b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f21ffb640 0 client.4722 ms_handle_reset on v2:172.21.15.125:6833/2017382874 2023-10-30T12:29:07.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f4c52b000 1 -- 172.21.15.125:0/889068994 >> 172.21.15.125:0/889068994 conn(0x55ae85f3dfc0 msgr2=0x55ae85f87090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f4c52b000 1 -- 172.21.15.125:0/889068994 shutdown_connections 2023-10-30T12:29:07.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f4c52b000 1 -- 172.21.15.125:0/889068994 wait complete. 2023-10-30T12:29:07.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.459+0000 7f8f4c52b000 20 client.4722 trim_cache size 0 max 16384 2023-10-30T12:29:07.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 Processor -- start 2023-10-30T12:29:07.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 -- start start 2023-10-30T12:29:07.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 0x55ae85f8ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85f8d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 0x55ae85fe1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae85a51940 con 0x55ae85f88130 2023-10-30T12:29:07.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae85f99d90 con 0x55ae85f87810 2023-10-30T12:29:07.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ae85d9b740 con 0x55ae85f88a50 2023-10-30T12:29:07.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f3a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85f8d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f3a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85f8d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54746/0 (socket says 172.21.15.125:54746) 2023-10-30T12:29:07.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f3a59c640 1 -- 172.21.15.125:0/4004060923 learned_addr learned my addr 172.21.15.125:0/4004060923 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f42d9d640 1 --2- 172.21.15.125:0/4004060923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 0x55ae85f8ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4359e640 1 --2- 172.21.15.125:0/4004060923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 0x55ae85fe1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f3a59c640 1 -- 172.21.15.125:0/4004060923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 msgr2=0x55ae85fe1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/4004060923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 0x55ae85fe1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f3a59c640 1 -- 172.21.15.125:0/4004060923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 msgr2=0x55ae85f8ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/4004060923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 0x55ae85f8ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f3a59c640 1 -- 172.21.15.125:0/4004060923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae85d3f760 con 0x55ae85f88130 2023-10-30T12:29:07.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4359e640 1 --2- 172.21.15.125:0/4004060923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 0x55ae85fe1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:07.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f42d9d640 1 --2- 172.21.15.125:0/4004060923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 0x55ae85f8ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:07.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/4004060923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85f8d3b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c004710 tx=0x7f8f2c0c6040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f0182c61c72c65 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f3959a640 1 -- 172.21.15.125:0/4004060923 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c004510 con 0x55ae85f88130 2023-10-30T12:29:07.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f3959a640 1 -- 172.21.15.125:0/4004060923 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c0cdd10 con 0x55ae85f88130 2023-10-30T12:29:07.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f3959a640 1 -- 172.21.15.125:0/4004060923 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c0cd640 con 0x55ae85f88130 2023-10-30T12:29:07.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 -- 172.21.15.125:0/4004060923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 msgr2=0x55ae85f8d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/4004060923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85f8d3b0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c004710 tx=0x7f8f2c0c6040 comp rx=0 tx=0).stop 2023-10-30T12:29:07.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 -- 172.21.15.125:0/4004060923 shutdown_connections 2023-10-30T12:29:07.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/4004060923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 0x55ae85fe1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/4004060923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85f8d3b0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/4004060923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 0x55ae85f8ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 -- 172.21.15.125:0/4004060923 >> 172.21.15.125:0/4004060923 conn(0x55ae85f3dfc0 msgr2=0x55ae85fdf9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 -- 172.21.15.125:0/4004060923 shutdown_connections 2023-10-30T12:29:07.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 -- 172.21.15.125:0/4004060923 wait complete. 2023-10-30T12:29:07.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 -- start start 2023-10-30T12:29:07.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.475+0000 7f8f4c52b000 1 Processor -- start 2023-10-30T12:29:07.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 0x55ae85feae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85feb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 0x55ae85fa6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae85d564d0 con 0x55ae85f88130 2023-10-30T12:29:07.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae85d930b0 con 0x55ae85f87810 2023-10-30T12:29:07.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ae85d965d0 con 0x55ae85f88a50 2023-10-30T12:29:07.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 10 client.0 _ll_register_callbacks cb 0x7f8f3c007770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-30T12:29:07.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f3a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85feb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f3a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85feb3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54760/0 (socket says 172.21.15.125:54760) 2023-10-30T12:29:07.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f3a59c640 1 -- 172.21.15.125:0/1170681747 learned_addr learned my addr 172.21.15.125:0/1170681747 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f42d9d640 1 --2- 172.21.15.125:0/1170681747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 0x55ae85feae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4359e640 1 --2- 172.21.15.125:0/1170681747 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 0x55ae85fa6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f3a59c640 1 -- 172.21.15.125:0/1170681747 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 msgr2=0x55ae85fa6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/1170681747 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 0x55ae85fa6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f3a59c640 1 -- 172.21.15.125:0/1170681747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 msgr2=0x55ae85feae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/1170681747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 0x55ae85feae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f3a59c640 1 -- 172.21.15.125:0/1170681747 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ae85d59460 con 0x55ae85f88130 2023-10-30T12:29:07.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f42d9d640 1 --2- 172.21.15.125:0/1170681747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 0x55ae85feae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:07.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f3a59c640 1 --2- 172.21.15.125:0/1170681747 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85feb3d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c01c9c0 tx=0x7f8f2c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7b338e66ff839e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.125:3300/0 2023-10-30T12:29:07.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f21ffb640 1 -- 172.21.15.125:0/1170681747 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c0059a0 con 0x55ae85f88130 2023-10-30T12:29:07.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f21ffb640 1 -- 172.21.15.125:0/1170681747 <== mon.0 v2:172.21.15.125:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8f2c005b00 con 0x55ae85f88130 2023-10-30T12:29:07.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 -- 172.21.15.125:0/1170681747 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ae85d2f6f0 con 0x55ae85f88130 2023-10-30T12:29:07.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f21ffb640 1 -- 172.21.15.125:0/1170681747 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_get_version_reply(handle=1 version=12) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f2c005d00 con 0x55ae85f88130 2023-10-30T12:29:07.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 10 client.4728 fetch_fsmap learned FSMap version 12 2023-10-30T12:29:07.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 -- 172.21.15.125:0/1170681747 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,fsmap=12,monmap=2+}) v3 -- 0x55ae85a90cc0 con 0x55ae85f88130 2023-10-30T12:29:07.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f21ffb640 1 -- 172.21.15.125:0/1170681747 <== mon.0 v2:172.21.15.125:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55ae85d2f6f0 con 0x55ae85f88130 2023-10-30T12:29:07.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f21ffb640 1 -- 172.21.15.125:0/1170681747 <== mon.0 v2:172.21.15.125:3300/0 5 ==== fsmap(e 12) v1 ==== 4468+0+0 (secure 0 0 0) 0x7f8f2c0039f0 con 0x55ae85f88130 2023-10-30T12:29:07.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 10 client.4728 fetch_fsmap finished waiting for FSMap version 12 2023-10-30T12:29:07.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 10 client.4728 resolve_mds: resolved name 'e' to daemon mds.e 2023-10-30T12:29:07.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 20 client.4728 populate_metadata read hostname 'smithi125' 2023-10-30T12:29:07.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/1170681747 >> [v2:172.21.15.125:6833/2017382874,v1:172.21.15.125:6836/2017382874] conn(0x55ae85fa73a0 0x55ae85fa7780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 4 client.4728 mds_command: new command op to 4294 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-30T12:29:07.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 -- 172.21.15.125:0/1170681747 --> [v2:172.21.15.125:6833/2017382874,v1:172.21.15.125:6836/2017382874] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ae85d90720 con 0x55ae85fa73a0 2023-10-30T12:29:07.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f42d9d640 1 --2- 172.21.15.125:0/1170681747 >> [v2:172.21.15.125:6833/2017382874,v1:172.21.15.125:6836/2017382874] conn(0x55ae85fa73a0 0x55ae85fa7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f42d9d640 1 --2- 172.21.15.125:0/1170681747 >> [v2:172.21.15.125:6833/2017382874,v1:172.21.15.125:6836/2017382874] conn(0x55ae85fa73a0 0x55ae85fa7780 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab326e374afcbdb1 server_cookie=ac31aaad30b49ad3 in_seq=0 out_seq=0 2023-10-30T12:29:07.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f21ffb640 10 client.4728 ms_handle_connect on v2:172.21.15.125:6833/2017382874 2023-10-30T12:29:07.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f21ffb640 1 -- 172.21.15.125:0/1170681747 <== mds.0 v2:172.21.15.125:6833/2017382874 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55ae85d90720 con 0x55ae85fa73a0 2023-10-30T12:29:07.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f21ffb640 10 client.4728 handle_command_reply: tid=0 2023-10-30T12:29:07.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 client.4728 shutdown 2023-10-30T12:29:07.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 -- 172.21.15.125:0/1170681747 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 msgr2=0x55ae85feb3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/1170681747 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85feb3d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c01c9c0 tx=0x7f8f2c008040 comp rx=0 tx=0).stop 2023-10-30T12:29:07.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 -- 172.21.15.125:0/1170681747 shutdown_connections 2023-10-30T12:29:07.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/1170681747 >> [v2:172.21.15.125:6833/2017382874,v1:172.21.15.125:6836/2017382874] conn(0x55ae85fa73a0 0x55ae85fa7780 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/1170681747 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x55ae85f88a50 0x55ae85fa6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/1170681747 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x55ae85f88130 0x55ae85feb3d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 --2- 172.21.15.125:0/1170681747 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x55ae85f87810 0x55ae85feae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f21ffb640 0 client.4728 ms_handle_reset on v2:172.21.15.125:6833/2017382874 2023-10-30T12:29:07.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 -- 172.21.15.125:0/1170681747 >> 172.21.15.125:0/1170681747 conn(0x55ae85f3dfc0 msgr2=0x55ae85f88e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 -- 172.21.15.125:0/1170681747 shutdown_connections 2023-10-30T12:29:07.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 1 -- 172.21.15.125:0/1170681747 wait complete. 2023-10-30T12:29:07.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.479+0000 7f8f4c52b000 20 client.4728 trim_cache size 0 max 16384 2023-10-30T12:29:07.496 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2023-10-30T12:29:07.496 INFO:teuthology.orchestra.run.smithi125.stdout: "cluster_fsid": "714f08a3-69cf-4c1b-8fab-64b3ba5e42e9", 2023-10-30T12:29:07.496 INFO:teuthology.orchestra.run.smithi125.stdout: "whoami": 0, 2023-10-30T12:29:07.497 INFO:teuthology.orchestra.run.smithi125.stdout: "id": 4294, 2023-10-30T12:29:07.497 INFO:teuthology.orchestra.run.smithi125.stdout: "want_state": "up:active", 2023-10-30T12:29:07.497 INFO:teuthology.orchestra.run.smithi125.stdout: "state": "up:active", 2023-10-30T12:29:07.497 INFO:teuthology.orchestra.run.smithi125.stdout: "fs_name": "cephfs", 2023-10-30T12:29:07.497 INFO:teuthology.orchestra.run.smithi125.stdout: "rank_uptime": 22.437396154999998, 2023-10-30T12:29:07.498 INFO:teuthology.orchestra.run.smithi125.stdout: "mdsmap_epoch": 12, 2023-10-30T12:29:07.498 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch": 22, 2023-10-30T12:29:07.498 INFO:teuthology.orchestra.run.smithi125.stdout: "osdmap_epoch_barrier": 22, 2023-10-30T12:29:07.498 INFO:teuthology.orchestra.run.smithi125.stdout: "uptime": 29.118500523000002 2023-10-30T12:29:07.498 INFO:teuthology.orchestra.run.smithi125.stdout:} 2023-10-30T12:29:07.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.483+0000 7f8f4919d640 1 -- 172.21.15.125:0/2377272518 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f240664d0 msgr2=0x7f8f24068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.483+0000 7f8f4919d640 1 --2- 172.21.15.125:0/2377272518 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f240664d0 0x7f8f24068990 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8f2c0af000 tx=0x7f8f2c0a7000 comp rx=0 tx=0).stop 2023-10-30T12:29:07.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.483+0000 7f8f4919d640 1 -- 172.21.15.125:0/2377272518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440cb570 msgr2=0x7f8f4415bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.483+0000 7f8f4919d640 1 --2- 172.21.15.125:0/2377272518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440cb570 0x7f8f4415bed0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8f3400c480 tx=0x7f8f3400c940 comp rx=0 tx=0).stop 2023-10-30T12:29:07.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.483+0000 7f8f4919d640 1 -- 172.21.15.125:0/2377272518 shutdown_connections 2023-10-30T12:29:07.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.483+0000 7f8f4919d640 1 --2- 172.21.15.125:0/2377272518 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f240664d0 0x7f8f24068990 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.483+0000 7f8f4919d640 1 --2- 172.21.15.125:0/2377272518 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f4415a000 0x7f8f44148940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.483+0000 7f8f4919d640 1 --2- 172.21.15.125:0/2377272518 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f44153d10 0x7f8f44148400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.483+0000 7f8f4919d640 1 --2- 172.21.15.125:0/2377272518 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440cb570 0x7f8f4415bed0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.483+0000 7f8f4919d640 1 -- 172.21.15.125:0/2377272518 >> 172.21.15.125:0/2377272518 conn(0x7f8f4406c6b0 msgr2=0x7f8f440c9dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.483+0000 7f8f4919d640 1 -- 172.21.15.125:0/2377272518 shutdown_connections 2023-10-30T12:29:07.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.483+0000 7f8f4919d640 1 -- 172.21.15.125:0/2377272518 wait complete. 2023-10-30T12:29:07.503 INFO:teuthology.run_tasks:Running task kclient... 2023-10-30T12:29:07.524 INFO:tasks.kclient:Mounting kernel clients... 2023-10-30T12:29:07.525 INFO:tasks.kclient:config is {'client.0': {}} 2023-10-30T12:29:07.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:07.525 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2023-10-30T12:29:07.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:07.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link delete ceph-brx 2023-10-30T12:29:07.598 INFO:teuthology.orchestra.run.smithi125.stderr:Cannot find device "ceph-brx" 2023-10-30T12:29:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T12:29:07.601 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-10-30T12:29:07.601 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-30T12:29:07.601 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:07.602 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-30T12:29:07.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b67a0640 1 Processor -- start 2023-10-30T12:29:07.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b67a0640 1 -- start start 2023-10-30T12:29:07.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b67a0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f35b015cd50 0x7f35b00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b67a0640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f35b00cefa0 0x7f35b00cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b67a0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35b015a6c0 con 0x7f35b01538f0 2023-10-30T12:29:07.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b67a0640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b00cf8f0 con 0x7f35b015cd50 2023-10-30T12:29:07.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b67a0640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35b015d300 con 0x7f35b00cefa0 2023-10-30T12:29:07.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b4f9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f35b015cd50 0x7f35b00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b4f9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f35b015cd50 0x7f35b00c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54766/0 (socket says 172.21.15.125:54766) 2023-10-30T12:29:07.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b4f9d640 1 -- 172.21.15.125:0/340081884 learned_addr learned my addr 172.21.15.125:0/340081884 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b5f9f640 1 --2- 172.21.15.125:0/340081884 >> v2:172.21.15.136:3300/0 conn(0x7f35b00cefa0 0x7f35b00cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b4f9d640 1 -- 172.21.15.125:0/340081884 >> v2:172.21.15.136:3300/0 conn(0x7f35b00cefa0 msgr2=0x7f35b00cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:07.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b4f9d640 1 --2- 172.21.15.125:0/340081884 >> v2:172.21.15.136:3300/0 conn(0x7f35b00cefa0 0x7f35b00cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b4f9d640 1 -- 172.21.15.125:0/340081884 >> v1:172.21.15.136:6789/0 conn(0x7f35b01538f0 legacy=0x7f35b0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.779+0000 7f35b4f9d640 1 -- 172.21.15.125:0/340081884 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35b015d5e0 con 0x7f35b015cd50 2023-10-30T12:29:07.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.783+0000 7f35b4f9d640 1 --2- 172.21.15.125:0/340081884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b015cd50 0x7f35b00c9f90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f35b0066a40 tx=0x7f35a001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be88a036fd7d4587 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.783+0000 7f35a7fff640 1 -- 172.21.15.125:0/340081884 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a00a89f0 con 0x7f35b015cd50 2023-10-30T12:29:07.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.783+0000 7f35a7fff640 1 -- 172.21.15.125:0/340081884 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a00a8b50 con 0x7f35b015cd50 2023-10-30T12:29:07.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.783+0000 7f35a7fff640 1 -- 172.21.15.125:0/340081884 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a00b0440 con 0x7f35b015cd50 2023-10-30T12:29:07.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.783+0000 7f35b67a0640 1 -- 172.21.15.125:0/340081884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b015cd50 msgr2=0x7f35b00c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.783+0000 7f35b67a0640 1 --2- 172.21.15.125:0/340081884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b015cd50 0x7f35b00c9f90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f35b0066a40 tx=0x7f35a001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.783+0000 7f35b67a0640 1 -- 172.21.15.125:0/340081884 shutdown_connections 2023-10-30T12:29:07.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.783+0000 7f35b67a0640 1 --2- 172.21.15.125:0/340081884 >> v2:172.21.15.136:3300/0 conn(0x7f35b00cefa0 0x7f35b00cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.783+0000 7f35b67a0640 1 --2- 172.21.15.125:0/340081884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b015cd50 0x7f35b00c9f90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.783+0000 7f35b67a0640 1 -- 172.21.15.125:0/340081884 >> 172.21.15.125:0/340081884 conn(0x7f35b006a640 msgr2=0x7f35b0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.783+0000 7f35b67a0640 1 -- 172.21.15.125:0/340081884 shutdown_connections 2023-10-30T12:29:07.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.783+0000 7f35b67a0640 1 -- 172.21.15.125:0/340081884 wait complete. 2023-10-30T12:29:07.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b67a0640 1 Processor -- start 2023-10-30T12:29:07.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b67a0640 1 -- start start 2023-10-30T12:29:07.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b67a0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35b00cefa0 0x7f35b01453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b67a0640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b01538f0 0x7f35b015a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b67a0640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35b015cd50 0x7f35b0158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b67a0640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b01f02f0 con 0x7f35b01538f0 2023-10-30T12:29:07.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b67a0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b0157a20 con 0x7f35b00cefa0 2023-10-30T12:29:07.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b67a0640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35b01ef2e0 con 0x7f35b015cd50 2023-10-30T12:29:07.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b4f9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b01538f0 0x7f35b015a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b4f9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b01538f0 0x7f35b015a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54782/0 (socket says 172.21.15.125:54782) 2023-10-30T12:29:07.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b4f9d640 1 -- 172.21.15.125:0/2193232088 learned_addr learned my addr 172.21.15.125:0/2193232088 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:07.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b579e640 1 --2- 172.21.15.125:0/2193232088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35b00cefa0 0x7f35b01453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b5f9f640 1 --2- 172.21.15.125:0/2193232088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35b015cd50 0x7f35b0158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b4f9d640 1 -- 172.21.15.125:0/2193232088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35b015cd50 msgr2=0x7f35b0158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b4f9d640 1 --2- 172.21.15.125:0/2193232088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35b015cd50 0x7f35b0158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b4f9d640 1 -- 172.21.15.125:0/2193232088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35b00cefa0 msgr2=0x7f35b01453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b4f9d640 1 --2- 172.21.15.125:0/2193232088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35b00cefa0 0x7f35b01453c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b4f9d640 1 -- 172.21.15.125:0/2193232088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a00094d0 con 0x7f35b01538f0 2023-10-30T12:29:07.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b4f9d640 1 --2- 172.21.15.125:0/2193232088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b01538f0 0x7f35b015a920 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f35a001b900 tx=0x7f35a001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35a5ffb640 1 -- 172.21.15.125:0/2193232088 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a00ba600 con 0x7f35b01538f0 2023-10-30T12:29:07.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35a5ffb640 1 -- 172.21.15.125:0/2193232088 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35a00ba760 con 0x7f35b01538f0 2023-10-30T12:29:07.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35a5ffb640 1 -- 172.21.15.125:0/2193232088 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a00bab10 con 0x7f35b01538f0 2023-10-30T12:29:07.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b67a0640 1 -- 172.21.15.125:0/2193232088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35b01efe60 con 0x7f35b01538f0 2023-10-30T12:29:07.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.787+0000 7f35b67a0640 1 -- 172.21.15.125:0/2193232088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35b01ef530 con 0x7f35b01538f0 2023-10-30T12:29:07.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.791+0000 7f35b67a0640 1 -- 172.21.15.125:0/2193232088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35b01efff0 con 0x7f35b01538f0 2023-10-30T12:29:07.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.791+0000 7f35a5ffb640 1 -- 172.21.15.125:0/2193232088 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f35a00bac70 con 0x7f35b01538f0 2023-10-30T12:29:07.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.791+0000 7f35a5ffb640 1 --2- 172.21.15.125:0/2193232088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35940664d0 0x7f3594068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:07.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.791+0000 7f35a5ffb640 1 -- 172.21.15.125:0/2193232088 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f35a01223d0 con 0x7f35b01538f0 2023-10-30T12:29:07.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.791+0000 7f35b579e640 1 --2- 172.21.15.125:0/2193232088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35940664d0 0x7f3594068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:07.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.791+0000 7f35b579e640 1 --2- 172.21.15.125:0/2193232088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35940664d0 0x7f3594068990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f35ac000f70 tx=0x7f35ac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:07.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.795+0000 7f35a5ffb640 1 -- 172.21.15.125:0/2193232088 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f35a00ef8d0 con 0x7f35b01538f0 2023-10-30T12:29:07.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.979+0000 7f35b67a0640 1 -- 172.21.15.125:0/2193232088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f35b004e170 con 0x7f35b01538f0 2023-10-30T12:29:07.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.979+0000 7f35a5ffb640 1 -- 172.21.15.125:0/2193232088 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v12) v1 ==== 53+0+73 (secure 0 0 0) 0x7f35a00f2ef0 con 0x7f35b01538f0 2023-10-30T12:29:07.982 INFO:teuthology.orchestra.run.smithi125.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-30T12:29:07.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.979+0000 7f35b67a0640 1 -- 172.21.15.125:0/2193232088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35940664d0 msgr2=0x7f3594068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.979+0000 7f35b67a0640 1 --2- 172.21.15.125:0/2193232088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35940664d0 0x7f3594068990 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f35ac000f70 tx=0x7f35ac009040 comp rx=0 tx=0).stop 2023-10-30T12:29:07.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.979+0000 7f35b67a0640 1 -- 172.21.15.125:0/2193232088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b01538f0 msgr2=0x7f35b015a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:07.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.979+0000 7f35b67a0640 1 --2- 172.21.15.125:0/2193232088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b01538f0 0x7f35b015a920 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f35a001b900 tx=0x7f35a001ba90 comp rx=0 tx=0).stop 2023-10-30T12:29:07.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.979+0000 7f35b67a0640 1 -- 172.21.15.125:0/2193232088 shutdown_connections 2023-10-30T12:29:07.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.979+0000 7f35b67a0640 1 --2- 172.21.15.125:0/2193232088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35940664d0 0x7f3594068990 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.979+0000 7f35b67a0640 1 --2- 172.21.15.125:0/2193232088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35b015cd50 0x7f35b0158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.979+0000 7f35b67a0640 1 --2- 172.21.15.125:0/2193232088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35b01538f0 0x7f35b015a920 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.979+0000 7f35b67a0640 1 --2- 172.21.15.125:0/2193232088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35b00cefa0 0x7f35b01453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:07.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.979+0000 7f35b67a0640 1 -- 172.21.15.125:0/2193232088 >> 172.21.15.125:0/2193232088 conn(0x7f35b006a640 msgr2=0x7f35b00cd710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:07.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.979+0000 7f35b67a0640 1 -- 172.21.15.125:0/2193232088 shutdown_connections 2023-10-30T12:29:07.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:07.979+0000 7f35b67a0640 1 -- 172.21.15.125:0/2193232088 wait complete. 2023-10-30T12:29:08.003 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-30T12:29:08.003 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-30T12:29:08.004 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi125.front.sepia.ceph.com 2023-10-30T12:29:08.004 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-30T12:29:08.004 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:08.004 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-30T12:29:08.005 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-30T12:29:08.005 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-30T12:29:08.005 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-30T12:29:08.006 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:08.006 DEBUG:teuthology.orchestra.run.smithi125:> ip addr 2023-10-30T12:29:08.015 INFO:teuthology.orchestra.run.smithi125.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-30T12:29:08.016 INFO:teuthology.orchestra.run.smithi125.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-30T12:29:08.016 INFO:teuthology.orchestra.run.smithi125.stdout: inet 127.0.0.1/8 scope host lo 2023-10-30T12:29:08.016 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-10-30T12:29:08.017 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 ::1/128 scope host 2023-10-30T12:29:08.017 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-10-30T12:29:08.017 INFO:teuthology.orchestra.run.smithi125.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T12:29:08.017 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f8 brd ff:ff:ff:ff:ff:ff 2023-10-30T12:29:08.017 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f0 2023-10-30T12:29:08.018 INFO:teuthology.orchestra.run.smithi125.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T12:29:08.018 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:d8:da:f9 brd ff:ff:ff:ff:ff:ff 2023-10-30T12:29:08.018 INFO:teuthology.orchestra.run.smithi125.stdout: altname enp7s0f1 2023-10-30T12:29:08.018 INFO:teuthology.orchestra.run.smithi125.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2023-10-30T12:29:08.019 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:38 brd ff:ff:ff:ff:ff:ff 2023-10-30T12:29:08.019 INFO:teuthology.orchestra.run.smithi125.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-30T12:29:08.019 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 0c:c4:7a:8f:d7:39 brd ff:ff:ff:ff:ff:ff 2023-10-30T12:29:08.019 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.21.15.125/20 metric 100 brd 172.21.15.255 scope global dynamic enp3s0f1 2023-10-30T12:29:08.020 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft 42430sec preferred_lft 42430sec 2023-10-30T12:29:08.020 INFO:teuthology.orchestra.run.smithi125.stdout: inet6 fe80::ec4:7aff:fe8f:d739/64 scope link 2023-10-30T12:29:08.020 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-10-30T12:29:08.020 INFO:teuthology.orchestra.run.smithi125.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2023-10-30T12:29:08.020 INFO:teuthology.orchestra.run.smithi125.stdout: link/ether 02:42:b1:5b:2b:ac brd ff:ff:ff:ff:ff:ff 2023-10-30T12:29:08.021 INFO:teuthology.orchestra.run.smithi125.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2023-10-30T12:29:08.021 INFO:teuthology.orchestra.run.smithi125.stdout: valid_lft forever preferred_lft forever 2023-10-30T12:29:08.022 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-30T12:29:08.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:08.023 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2023-10-30T12:29:08.023 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2023-10-30T12:29:08.023 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link add name ceph-brx type bridge 2023-10-30T12:29:08.024 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip addr flush dev ceph-brx 2023-10-30T12:29:08.024 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set ceph-brx up 2023-10-30T12:29:08.024 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-30T12:29:08.025 DEBUG:teuthology.orchestra.run.smithi125:> ') 2023-10-30T12:29:08.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:08.137 DEBUG:teuthology.orchestra.run.smithi125:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-30T12:29:08.156 INFO:teuthology.orchestra.run.smithi125.stdout:1 2023-10-30T12:29:08.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:08.159 DEBUG:teuthology.orchestra.run.smithi125:> route 2023-10-30T12:29:08.221 INFO:teuthology.orchestra.run.smithi125.stdout:Kernel IP routing table 2023-10-30T12:29:08.222 INFO:teuthology.orchestra.run.smithi125.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-30T12:29:08.222 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2023-10-30T12:29:08.222 INFO:teuthology.orchestra.run.smithi125.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-30T12:29:08.222 INFO:teuthology.orchestra.run.smithi125.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2023-10-30T12:29:08.223 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-30T12:29:08.223 INFO:teuthology.orchestra.run.smithi125.stdout:ns1.front.sepia 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-30T12:29:08.223 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.0.2 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-30T12:29:08.223 INFO:teuthology.orchestra.run.smithi125.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2023-10-30T12:29:08.223 INFO:teuthology.orchestra.run.smithi125.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-30T12:29:08.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:08.224 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2023-10-30T12:29:08.224 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2023-10-30T12:29:08.225 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-30T12:29:08.225 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-30T12:29:08.225 DEBUG:teuthology.orchestra.run.smithi125:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-30T12:29:08.225 DEBUG:teuthology.orchestra.run.smithi125:> ') 2023-10-30T12:29:08.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:08.314 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2023-10-30T12:29:08.324 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:08.324 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list-id 2023-10-30T12:29:08.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:08.381 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2023-10-30T12:29:08.381 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2023-10-30T12:29:08.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-30T12:29:08.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-30T12:29:08.382 DEBUG:teuthology.orchestra.run.smithi125:> ') 2023-10-30T12:29:08.466 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-30T12:29:08.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:08.467 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2023-10-30T12:29:08.467 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2023-10-30T12:29:08.467 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-30T12:29:08.467 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-30T12:29:08.468 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-30T12:29:08.468 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-30T12:29:08.468 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-30T12:29:08.468 DEBUG:teuthology.orchestra.run.smithi125:> ') 2023-10-30T12:29:08.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:08.671 DEBUG:teuthology.orchestra.run.smithi125:> (cd / && exec bash -c ' 2023-10-30T12:29:08.671 DEBUG:teuthology.orchestra.run.smithi125:> set -e 2023-10-30T12:29:08.671 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set brx.0 up 2023-10-30T12:29:08.671 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link set dev brx.0 master ceph-brx 2023-10-30T12:29:08.672 DEBUG:teuthology.orchestra.run.smithi125:> ') 2023-10-30T12:29:08.713 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-30T12:29:08.713 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:08.764 INFO:teuthology.orchestra.run.smithi125.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-30T12:29:08.764 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-30T12:29:08.765 DEBUG:teuthology.orchestra.run.smithi125:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:08.815 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-30T12:29:08.816 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:08.816 DEBUG:teuthology.orchestra.run.smithi125:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T12:29:08.917 INFO:teuthology.orchestra.run.smithi125.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T12:29:08.918 INFO:teuthology.orchestra.run.smithi125.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T12:29:08.918 INFO:teuthology.orchestra.run.smithi125.stdout:invalid new device string format 2023-10-30T12:29:09.046 INFO:teuthology.orchestra.run.smithi125.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T12:29:09.046 INFO:teuthology.orchestra.run.smithi125.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T12:29:09.047 INFO:teuthology.orchestra.run.smithi125.stdout:invalid new device string format 2023-10-30T12:29:09.047 INFO:teuthology.orchestra.run.smithi125.stdout:mount.ceph: resolved to: "172.21.15.125:3300,172.21.15.136:3300,172.21.15.136:3301" 2023-10-30T12:29:09.047 INFO:teuthology.orchestra.run.smithi125.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-30T12:29:09.059 INFO:teuthology.orchestra.run.smithi125.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-30T12:29:09.059 INFO:teuthology.orchestra.run.smithi125.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-30T12:29:09.060 INFO:teuthology.orchestra.run.smithi125.stdout:invalid new device string format 2023-10-30T12:29:09.060 INFO:teuthology.orchestra.run.smithi125.stdout:mount.ceph: resolved to: "172.21.15.125:3300,172.21.15.136:3300,172.21.15.136:3301" 2023-10-30T12:29:09.060 INFO:teuthology.orchestra.run.smithi125.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-30T12:29:09.060 INFO:teuthology.orchestra.run.smithi125.stdout:mount.ceph: resolved to: "172.21.15.125:6789,172.21.15.136:6789,172.21.15.136:6790" 2023-10-30T12:29:09.061 INFO:teuthology.orchestra.run.smithi125.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.125:6789,172.21.15.136:6789,172.21.15.136:6790:/ 2023-10-30T12:29:09.061 INFO:teuthology.orchestra.run.smithi125.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=714f08a3-69cf-4c1b-8fab-64b3ba5e42e9" will pass to kernel 2023-10-30T12:29:09.063 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-30T12:29:09.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:09.064 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:09.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:09.442 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T12:29:09.486 INFO:teuthology.orchestra.run.smithi125.stdout:client4592 2023-10-30T12:29:09.486 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:29:09.486 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T12:29:09.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:09.536 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T12:29:09.597 INFO:teuthology.orchestra.run.smithi125.stdout:client4592 2023-10-30T12:29:09.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:09.598 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-30T12:29:10.426 INFO:teuthology.orchestra.run.smithi125.stdout:client4592 2023-10-30T12:29:10.426 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T12:29:10.427 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/714f08a3-69cf-4c1b-8fab-64b3ba5e42e9.client4592/status') 2023-10-30T12:29:10.450 INFO:teuthology.orchestra.run.smithi125.stdout:instance: client.4592 (3)172.21.15.125:0/3551331932 2023-10-30T12:29:10.450 INFO:teuthology.orchestra.run.smithi125.stdout:blocklisted: false 2023-10-30T12:29:10.451 INFO:teuthology.orchestra.run.smithi125.stderr:0+1 records in 2023-10-30T12:29:10.451 INFO:teuthology.orchestra.run.smithi125.stderr:0+1 records out 2023-10-30T12:29:10.451 INFO:teuthology.orchestra.run.smithi125.stderr:71 bytes copied, 5.8864e-05 s, 1.2 MB/s 2023-10-30T12:29:10.453 INFO:teuthology.run_tasks:Running task mds_thrash... 2023-10-30T12:29:10.467 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:29:10.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda46e46640 1 Processor -- start 2023-10-30T12:29:10.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda46e46640 1 -- start start 2023-10-30T12:29:10.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda46e46640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fda400c6b80 0x7fda400bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:10.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda46e46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda400c6f60 0x7fda400c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:10.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda46e46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fda400c8430 0x7fda4013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:10.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda46e46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda400c0a50 con 0x7fda400c8430 2023-10-30T12:29:10.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda46e46640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda4006f6e0 con 0x7fda400c6b80 2023-10-30T12:29:10.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda46e46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4006f9c0 con 0x7fda400c6f60 2023-10-30T12:29:10.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda453e3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fda400c8430 0x7fda4013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:10.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda453e3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fda400c8430 0x7fda4013ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54788/0 (socket says 172.21.15.125:54788) 2023-10-30T12:29:10.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda453e3640 1 -- 172.21.15.125:0/2028126322 learned_addr learned my addr 172.21.15.125:0/2028126322 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:10.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda37fff640 1 --2- 172.21.15.125:0/2028126322 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda400c6f60 0x7fda400c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:10.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda44be2640 1 --2- 172.21.15.125:0/2028126322 >> v2:172.21.15.136:3300/0 conn(0x7fda400c6b80 0x7fda400bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:10.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda453e3640 1 -- 172.21.15.125:0/2028126322 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda400c6f60 msgr2=0x7fda400c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:10.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda453e3640 1 --2- 172.21.15.125:0/2028126322 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda400c6f60 0x7fda400c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda453e3640 1 -- 172.21.15.125:0/2028126322 >> v2:172.21.15.136:3300/0 conn(0x7fda400c6b80 msgr2=0x7fda400bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:10.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda453e3640 1 --2- 172.21.15.125:0/2028126322 >> v2:172.21.15.136:3300/0 conn(0x7fda400c6b80 0x7fda400bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.639+0000 7fda453e3640 1 -- 172.21.15.125:0/2028126322 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4006fca0 con 0x7fda400c8430 2023-10-30T12:29:10.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.643+0000 7fda453e3640 1 --2- 172.21.15.125:0/2028126322 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda400c8430 0x7fda4013ed90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fda4006f1b0 tx=0x7fda3801cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee41c09d0495cae9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:10.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.643+0000 7fda377fe640 1 -- 172.21.15.125:0/2028126322 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda380a5970 con 0x7fda400c8430 2023-10-30T12:29:10.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.643+0000 7fda377fe640 1 -- 172.21.15.125:0/2028126322 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda380a5ad0 con 0x7fda400c8430 2023-10-30T12:29:10.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.643+0000 7fda377fe640 1 -- 172.21.15.125:0/2028126322 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda380a5e60 con 0x7fda400c8430 2023-10-30T12:29:10.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.643+0000 7fda46e46640 1 -- 172.21.15.125:0/2028126322 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda400c8430 msgr2=0x7fda4013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:10.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.643+0000 7fda46e46640 1 --2- 172.21.15.125:0/2028126322 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda400c8430 0x7fda4013ed90 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fda4006f1b0 tx=0x7fda3801cbd0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.643+0000 7fda46e46640 1 -- 172.21.15.125:0/2028126322 shutdown_connections 2023-10-30T12:29:10.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.643+0000 7fda46e46640 1 --2- 172.21.15.125:0/2028126322 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda400c8430 0x7fda4013ed90 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.643+0000 7fda46e46640 1 --2- 172.21.15.125:0/2028126322 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda400c6f60 0x7fda400c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.643+0000 7fda46e46640 1 --2- 172.21.15.125:0/2028126322 >> v2:172.21.15.136:3300/0 conn(0x7fda400c6b80 0x7fda400bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.643+0000 7fda46e46640 1 -- 172.21.15.125:0/2028126322 >> 172.21.15.125:0/2028126322 conn(0x7fda400c7910 msgr2=0x7fda400c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:10.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.643+0000 7fda46e46640 1 -- 172.21.15.125:0/2028126322 shutdown_connections 2023-10-30T12:29:10.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.643+0000 7fda46e46640 1 -- 172.21.15.125:0/2028126322 wait complete. 2023-10-30T12:29:10.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda46e46640 1 Processor -- start 2023-10-30T12:29:10.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda46e46640 1 -- start start 2023-10-30T12:29:10.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda46e46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda400c6b80 0x7fda4013d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:10.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda46e46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda400c6f60 0x7fda40155cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:10.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda46e46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda400c8430 0x7fda400c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:10.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda46e46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda4013cdc0 con 0x7fda400c6f60 2023-10-30T12:29:10.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda46e46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda40134560 con 0x7fda400c6b80 2023-10-30T12:29:10.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda46e46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda4013cb00 con 0x7fda400c8430 2023-10-30T12:29:10.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda37fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda400c6f60 0x7fda40155cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:10.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda37fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda400c6f60 0x7fda40155cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54800/0 (socket says 172.21.15.125:54800) 2023-10-30T12:29:10.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda37fff640 1 -- 172.21.15.125:0/3121165132 learned_addr learned my addr 172.21.15.125:0/3121165132 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:10.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda453e3640 1 --2- 172.21.15.125:0/3121165132 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda400c8430 0x7fda400c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:10.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda44be2640 1 --2- 172.21.15.125:0/3121165132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda400c6b80 0x7fda4013d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:10.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda37fff640 1 -- 172.21.15.125:0/3121165132 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda400c8430 msgr2=0x7fda400c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:10.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda37fff640 1 --2- 172.21.15.125:0/3121165132 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda400c8430 0x7fda400c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda37fff640 1 -- 172.21.15.125:0/3121165132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda400c6b80 msgr2=0x7fda4013d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:10.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda37fff640 1 --2- 172.21.15.125:0/3121165132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda400c6b80 0x7fda4013d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda37fff640 1 -- 172.21.15.125:0/3121165132 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda38009b30 con 0x7fda400c6f60 2023-10-30T12:29:10.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda37fff640 1 --2- 172.21.15.125:0/3121165132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda400c6f60 0x7fda40155cd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fda30011ac0 tx=0x7fda30011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda357fa640 1 -- 172.21.15.125:0/3121165132 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda3000fc20 con 0x7fda400c6f60 2023-10-30T12:29:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda357fa640 1 -- 172.21.15.125:0/3121165132 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda30013040 con 0x7fda400c6f60 2023-10-30T12:29:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda357fa640 1 -- 172.21.15.125:0/3121165132 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda30019400 con 0x7fda400c6f60 2023-10-30T12:29:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda46e46640 1 -- 172.21.15.125:0/3121165132 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda4013c820 con 0x7fda400c6f60 2023-10-30T12:29:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.647+0000 7fda46e46640 1 -- 172.21.15.125:0/3121165132 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda40134280 con 0x7fda400c6f60 2023-10-30T12:29:10.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.651+0000 7fda46e46640 1 -- 172.21.15.125:0/3121165132 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda4004e170 con 0x7fda400c6f60 2023-10-30T12:29:10.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.651+0000 7fda357fa640 1 -- 172.21.15.125:0/3121165132 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fda30019560 con 0x7fda400c6f60 2023-10-30T12:29:10.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.651+0000 7fda357fa640 1 --2- 172.21.15.125:0/3121165132 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda2c0664a0 0x7fda2c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:10.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.651+0000 7fda357fa640 1 -- 172.21.15.125:0/3121165132 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fda30087c20 con 0x7fda400c6f60 2023-10-30T12:29:10.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.651+0000 7fda44be2640 1 --2- 172.21.15.125:0/3121165132 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda2c0664a0 0x7fda2c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:10.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.651+0000 7fda44be2640 1 --2- 172.21.15.125:0/3121165132 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda2c0664a0 0x7fda2c068960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00ba30 tx=0x7fda3c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:10.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.655+0000 7fda357fa640 1 -- 172.21.15.125:0/3121165132 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fda30055140 con 0x7fda400c6f60 2023-10-30T12:29:10.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.807+0000 7fda46e46640 1 -- 172.21.15.125:0/3121165132 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda401391f0 con 0x7fda400c6f60 2023-10-30T12:29:10.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.807+0000 7fda357fa640 1 -- 172.21.15.125:0/3121165132 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16557 (secure 0 0 0) 0x7fda30058760 con 0x7fda400c6f60 2023-10-30T12:29:10.810 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:29:10.810 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:45.927862+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:29:10.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.807+0000 7fda46e46640 1 -- 172.21.15.125:0/3121165132 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda2c0664a0 msgr2=0x7fda2c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:10.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.807+0000 7fda46e46640 1 --2- 172.21.15.125:0/3121165132 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda2c0664a0 0x7fda2c068960 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00ba30 tx=0x7fda3c008040 comp rx=0 tx=0).stop 2023-10-30T12:29:10.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.807+0000 7fda46e46640 1 -- 172.21.15.125:0/3121165132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda400c6f60 msgr2=0x7fda40155cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:10.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.807+0000 7fda46e46640 1 --2- 172.21.15.125:0/3121165132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda400c6f60 0x7fda40155cd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fda30011ac0 tx=0x7fda30011f80 comp rx=0 tx=0).stop 2023-10-30T12:29:10.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.811+0000 7fda46e46640 1 -- 172.21.15.125:0/3121165132 shutdown_connections 2023-10-30T12:29:10.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.811+0000 7fda46e46640 1 --2- 172.21.15.125:0/3121165132 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda2c0664a0 0x7fda2c068960 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.811+0000 7fda46e46640 1 --2- 172.21.15.125:0/3121165132 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda400c8430 0x7fda400c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.811+0000 7fda46e46640 1 --2- 172.21.15.125:0/3121165132 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda400c6f60 0x7fda40155cd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.811+0000 7fda46e46640 1 --2- 172.21.15.125:0/3121165132 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda400c6b80 0x7fda4013d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.811+0000 7fda46e46640 1 -- 172.21.15.125:0/3121165132 >> 172.21.15.125:0/3121165132 conn(0x7fda400c7910 msgr2=0x7fda401314b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:10.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.811+0000 7fda46e46640 1 -- 172.21.15.125:0/3121165132 shutdown_connections 2023-10-30T12:29:10.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.811+0000 7fda46e46640 1 -- 172.21.15.125:0/3121165132 wait complete. 2023-10-30T12:29:10.832 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:31.063403+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T12:28:43.007849+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T12:28:43.401231+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:29:10.832 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:29:10.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0c048d6640 1 Processor -- start 2023-10-30T12:29:10.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0c048d6640 1 -- start start 2023-10-30T12:29:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0c048d6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0c00153980 0x7f0c000caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:10.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0c048d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c000cb1e0 0x7f0c0015d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0c048d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c0015a920 con 0x7f0c00159930 2023-10-30T12:29:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0c048d6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c000cb930 con 0x7f0c00153980 2023-10-30T12:29:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0c048d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c00159dc0 con 0x7f0c000cb1e0 2023-10-30T12:29:10.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0bf7fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0c00159930 0x7f0c000cf1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60054/0 (socket says 172.21.15.125:60054) 2023-10-30T12:29:10.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0bf7fff640 1 -- 172.21.15.125:0/1429491371 learned_addr learned my addr 172.21.15.125:0/1429491371 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:29:10.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0bfe59c640 1 --2- 172.21.15.125:0/1429491371 >> v2:172.21.15.136:3300/0 conn(0x7f0c00153980 0x7f0c000caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:10.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0bfed9d640 1 --2- 172.21.15.125:0/1429491371 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c000cb1e0 0x7f0c0015d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:10.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0bfdd9b640 1 -- 172.21.15.125:0/1429491371 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227709775 0 0) 0x7f0c0015a920 con 0x7f0c00159930 2023-10-30T12:29:10.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0bfdd9b640 1 -- 172.21.15.125:0/1429491371 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bec003530 con 0x7f0c00159930 2023-10-30T12:29:10.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0bfdd9b640 1 -- 172.21.15.125:0/1429491371 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3365346143 0 0) 0x7f0bec003530 con 0x7f0c00159930 2023-10-30T12:29:10.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0bfe59c640 1 -- 172.21.15.125:0/1429491371 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c000cb1e0 msgr2=0x7f0c0015d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:10.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0bfe59c640 1 --2- 172.21.15.125:0/1429491371 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c000cb1e0 0x7f0c0015d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0bfe59c640 1 -- 172.21.15.125:0/1429491371 >> v1:172.21.15.125:6789/0 conn(0x7f0c00159930 legacy=0x7f0c000cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:10.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.971+0000 7f0bfe59c640 1 -- 172.21.15.125:0/1429491371 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c0015a080 con 0x7f0c00153980 2023-10-30T12:29:10.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.975+0000 7f0bfe59c640 1 --2- 172.21.15.125:0/1429491371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c00153980 0x7f0c000caca0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00097c0 tx=0x7f0bf001c370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=646c02f84cc2452 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:10.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.975+0000 7f0bfdd9b640 1 -- 172.21.15.125:0/1429491371 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8001680 con 0x7f0c00153980 2023-10-30T12:29:10.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.975+0000 7f0bfdd9b640 1 -- 172.21.15.125:0/1429491371 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf00a7050 con 0x7f0c00153980 2023-10-30T12:29:10.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.975+0000 7f0bfdd9b640 1 -- 172.21.15.125:0/1429491371 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf00ab600 con 0x7f0c00153980 2023-10-30T12:29:10.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.975+0000 7f0c048d6640 1 -- 172.21.15.125:0/1429491371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c00153980 msgr2=0x7f0c000caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:10.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.975+0000 7f0c048d6640 1 --2- 172.21.15.125:0/1429491371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c00153980 0x7f0c000caca0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00097c0 tx=0x7f0bf001c370 comp rx=0 tx=0).stop 2023-10-30T12:29:10.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.975+0000 7f0c048d6640 1 -- 172.21.15.125:0/1429491371 shutdown_connections 2023-10-30T12:29:10.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.975+0000 7f0c048d6640 1 --2- 172.21.15.125:0/1429491371 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c000cb1e0 0x7f0c0015d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.975+0000 7f0c048d6640 1 --2- 172.21.15.125:0/1429491371 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c00153980 0x7f0c000caca0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.975+0000 7f0c048d6640 1 -- 172.21.15.125:0/1429491371 >> 172.21.15.125:0/1429491371 conn(0x7f0c0006c0f0 msgr2=0x7f0c0006c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:10.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.975+0000 7f0c048d6640 1 -- 172.21.15.125:0/1429491371 shutdown_connections 2023-10-30T12:29:10.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.975+0000 7f0c048d6640 1 -- 172.21.15.125:0/1429491371 wait complete. 2023-10-30T12:29:10.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0c048d6640 1 Processor -- start 2023-10-30T12:29:10.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0c048d6640 1 -- start start 2023-10-30T12:29:10.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0c048d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c000cb1e0 0x7f0c000b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:10.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0c048d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c00153980 0x7f0c00151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:10.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0c048d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c00152160 0x7f0c0015bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:10.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0c048d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000b51a0 con 0x7f0c000cb1e0 2023-10-30T12:29:10.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0c048d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000b4f50 con 0x7f0c00152160 2023-10-30T12:29:10.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0c048d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c000cefb0 con 0x7f0c00153980 2023-10-30T12:29:10.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bfe59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c000cb1e0 0x7f0c000b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:10.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bfe59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c000cb1e0 0x7f0c000b44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39866/0 (socket says 172.21.15.125:39866) 2023-10-30T12:29:10.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bfe59c640 1 -- 172.21.15.125:0/914490554 learned_addr learned my addr 172.21.15.125:0/914490554 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:10.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bfed9d640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c00152160 0x7f0c0015bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:10.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bf7fff640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c00153980 0x7f0c00151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:10.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bfe59c640 1 -- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c00153980 msgr2=0x7f0c00151c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:10.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bfe59c640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c00153980 0x7f0c00151c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bfe59c640 1 -- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c00152160 msgr2=0x7f0c0015bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:10.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bfe59c640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c00152160 0x7f0c0015bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:10.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bfe59c640 1 -- 172.21.15.125:0/914490554 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf0009420 con 0x7f0c000cb1e0 2023-10-30T12:29:10.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bfed9d640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c00152160 0x7f0c0015bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:10.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bfe59c640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c000cb1e0 0x7f0c000b44e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0000c10 tx=0x7f0bf0019110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:10.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bf6ffd640 1 -- 172.21.15.125:0/914490554 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf00bb070 con 0x7f0c000cb1e0 2023-10-30T12:29:10.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bf6ffd640 1 -- 172.21.15.125:0/914490554 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf00038b0 con 0x7f0c000cb1e0 2023-10-30T12:29:10.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0bf6ffd640 1 -- 172.21.15.125:0/914490554 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0003c40 con 0x7f0c000cb1e0 2023-10-30T12:29:10.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0c048d6640 1 -- 172.21.15.125:0/914490554 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c00157570 con 0x7f0c000cb1e0 2023-10-30T12:29:10.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0c048d6640 1 -- 172.21.15.125:0/914490554 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c000ce960 con 0x7f0c000cb1e0 2023-10-30T12:29:10.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.979+0000 7f0c048d6640 1 -- 172.21.15.125:0/914490554 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0004e170 con 0x7f0c000cb1e0 2023-10-30T12:29:10.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.983+0000 7f0bf6ffd640 1 -- 172.21.15.125:0/914490554 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0bf00b2050 con 0x7f0c000cb1e0 2023-10-30T12:29:10.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.983+0000 7f0bf6ffd640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0be00664a0 0x7f0be0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:10.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.983+0000 7f0bf6ffd640 1 -- 172.21.15.125:0/914490554 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0bf011d800 con 0x7f0c000cb1e0 2023-10-30T12:29:10.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.983+0000 7f0bf7fff640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0be00664a0 0x7f0be0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:10.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.983+0000 7f0bf7fff640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0be00664a0 0x7f0be0068960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0c001540e0 tx=0x7f0be8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:10.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:10.987+0000 7f0bf6ffd640 1 -- 172.21.15.125:0/914490554 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0bf00eabe0 con 0x7f0c000cb1e0 2023-10-30T12:29:11.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.139+0000 7f0c048d6640 1 -- 172.21.15.125:0/914490554 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0c0015d050 con 0x7f0c000cb1e0 2023-10-30T12:29:11.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.139+0000 7f0bf6ffd640 1 -- 172.21.15.125:0/914490554 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0bf00ee200 con 0x7f0c000cb1e0 2023-10-30T12:29:11.141 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2023-10-30T12:29:11.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.139+0000 7f0c048d6640 1 -- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0be00664a0 msgr2=0x7f0be0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.139+0000 7f0c048d6640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0be00664a0 0x7f0be0068960 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0c001540e0 tx=0x7f0be8009040 comp rx=0 tx=0).stop 2023-10-30T12:29:11.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.139+0000 7f0c048d6640 1 -- 172.21.15.125:0/914490554 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c000cb1e0 msgr2=0x7f0c000b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.139+0000 7f0c048d6640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c000cb1e0 0x7f0c000b44e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0bf0000c10 tx=0x7f0bf0019110 comp rx=0 tx=0).stop 2023-10-30T12:29:11.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.139+0000 7f0c048d6640 1 -- 172.21.15.125:0/914490554 shutdown_connections 2023-10-30T12:29:11.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.139+0000 7f0c048d6640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0be00664a0 0x7f0be0068960 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.139+0000 7f0c048d6640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c00152160 0x7f0c0015bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.139+0000 7f0c048d6640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c00153980 0x7f0c00151c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.139+0000 7f0c048d6640 1 --2- 172.21.15.125:0/914490554 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c000cb1e0 0x7f0c000b44e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.139+0000 7f0c048d6640 1 -- 172.21.15.125:0/914490554 >> 172.21.15.125:0/914490554 conn(0x7f0c0006c0f0 msgr2=0x7f0c000c4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:11.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.139+0000 7f0c048d6640 1 -- 172.21.15.125:0/914490554 shutdown_connections 2023-10-30T12:29:11.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.139+0000 7f0c048d6640 1 -- 172.21.15.125:0/914490554 wait complete. 2023-10-30T12:29:11.162 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T12:29:11.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.299+0000 7ff06e59d640 1 Processor -- start 2023-10-30T12:29:11.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.299+0000 7ff06e59d640 1 -- start start 2023-10-30T12:29:11.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.303+0000 7ff06e59d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff0680c5810 0x7ff0680c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.303+0000 7ff06e59d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff06806a980 0x7ff068161f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.303+0000 7ff06e59d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0680c6cb0 con 0x7ff0680c6130 2023-10-30T12:29:11.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.303+0000 7ff06e59d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff068071d90 con 0x7ff0680c5810 2023-10-30T12:29:11.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.303+0000 7ff06e59d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0681585d0 con 0x7ff06806a980 2023-10-30T12:29:11.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.303+0000 7ff067fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff0680c5810 0x7ff0680c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.303+0000 7ff067fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff0680c5810 0x7ff0680c5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39882/0 (socket says 172.21.15.125:39882) 2023-10-30T12:29:11.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.303+0000 7ff067fff640 1 -- 172.21.15.125:0/1356235808 learned_addr learned my addr 172.21.15.125:0/1356235808 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:11.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.303+0000 7ff06cb3a640 1 --2- 172.21.15.125:0/1356235808 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff06806a980 0x7ff068161f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.303+0000 7ff067fff640 1 -- 172.21.15.125:0/1356235808 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff06806a980 msgr2=0x7ff068161f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:11.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.303+0000 7ff067fff640 1 --2- 172.21.15.125:0/1356235808 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff06806a980 0x7ff068161f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.303+0000 7ff067fff640 1 -- 172.21.15.125:0/1356235808 >> v1:172.21.15.136:6789/0 conn(0x7ff0680c6130 legacy=0x7ff06806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:11.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.303+0000 7ff067fff640 1 -- 172.21.15.125:0/1356235808 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0680c6620 con 0x7ff0680c5810 2023-10-30T12:29:11.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.307+0000 7ff067fff640 1 --2- 172.21.15.125:0/1356235808 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0680c5810 0x7ff0680c5bf0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff058009870 tx=0x7ff05801c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34dd44b6821ff9c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:11.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.307+0000 7ff066ffd640 1 -- 172.21.15.125:0/1356235808 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0580ac640 con 0x7ff0680c5810 2023-10-30T12:29:11.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.307+0000 7ff066ffd640 1 -- 172.21.15.125:0/1356235808 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0580ac7a0 con 0x7ff0680c5810 2023-10-30T12:29:11.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.307+0000 7ff066ffd640 1 -- 172.21.15.125:0/1356235808 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0580acb30 con 0x7ff0680c5810 2023-10-30T12:29:11.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.307+0000 7ff06e59d640 1 -- 172.21.15.125:0/1356235808 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0680c5810 msgr2=0x7ff0680c5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.307+0000 7ff06e59d640 1 --2- 172.21.15.125:0/1356235808 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0680c5810 0x7ff0680c5bf0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff058009870 tx=0x7ff05801c400 comp rx=0 tx=0).stop 2023-10-30T12:29:11.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.307+0000 7ff06e59d640 1 -- 172.21.15.125:0/1356235808 shutdown_connections 2023-10-30T12:29:11.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.307+0000 7ff06e59d640 1 --2- 172.21.15.125:0/1356235808 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff06806a980 0x7ff068161f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.307+0000 7ff06e59d640 1 --2- 172.21.15.125:0/1356235808 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0680c5810 0x7ff0680c5bf0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.307+0000 7ff06e59d640 1 -- 172.21.15.125:0/1356235808 >> 172.21.15.125:0/1356235808 conn(0x7ff0680c7910 msgr2=0x7ff0680c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:11.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.307+0000 7ff06e59d640 1 -- 172.21.15.125:0/1356235808 shutdown_connections 2023-10-30T12:29:11.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.307+0000 7ff06e59d640 1 -- 172.21.15.125:0/1356235808 wait complete. 2023-10-30T12:29:11.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.307+0000 7ff06e59d640 1 Processor -- start 2023-10-30T12:29:11.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.307+0000 7ff06e59d640 1 -- start start 2023-10-30T12:29:11.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06e59d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff06806a980 0x7ff0680c25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06e59d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0680c5810 0x7ff0681f6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06e59d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0680c6130 0x7ff06815b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06e59d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0680c49a0 con 0x7ff0680c6130 2023-10-30T12:29:11.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06e59d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0680c4c40 con 0x7ff06806a980 2023-10-30T12:29:11.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06e59d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0680c4f20 con 0x7ff0680c5810 2023-10-30T12:29:11.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06cb3a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0680c6130 0x7ff06815b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06cb3a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0680c6130 0x7ff06815b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39884/0 (socket says 172.21.15.125:39884) 2023-10-30T12:29:11.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06cb3a640 1 -- 172.21.15.125:0/815933790 learned_addr learned my addr 172.21.15.125:0/815933790 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:11.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff067fff640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff06806a980 0x7ff0680c25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff0677fe640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0680c5810 0x7ff0681f6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06cb3a640 1 -- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0680c5810 msgr2=0x7ff0681f6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06cb3a640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0680c5810 0x7ff0681f6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06cb3a640 1 -- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff06806a980 msgr2=0x7ff0680c25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06cb3a640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff06806a980 0x7ff0680c25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06cb3a640 1 -- 172.21.15.125:0/815933790 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff058009480 con 0x7ff0680c6130 2023-10-30T12:29:11.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff067fff640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff06806a980 0x7ff0680c25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:11.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06cb3a640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0680c6130 0x7ff06815b410 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff060007e60 tx=0x7ff06000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:11.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff064ff9640 1 -- 172.21.15.125:0/815933790 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff060014420 con 0x7ff0680c6130 2023-10-30T12:29:11.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff064ff9640 1 -- 172.21.15.125:0/815933790 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0600145a0 con 0x7ff0680c6130 2023-10-30T12:29:11.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06e59d640 1 -- 172.21.15.125:0/815933790 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0681f5c90 con 0x7ff0680c6130 2023-10-30T12:29:11.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff064ff9640 1 -- 172.21.15.125:0/815933790 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff060014950 con 0x7ff0680c6130 2023-10-30T12:29:11.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06e59d640 1 -- 172.21.15.125:0/815933790 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0681f6250 con 0x7ff0680c6130 2023-10-30T12:29:11.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff06e59d640 1 -- 172.21.15.125:0/815933790 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0681f6470 con 0x7ff0680c6130 2023-10-30T12:29:11.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.311+0000 7ff064ff9640 1 -- 172.21.15.125:0/815933790 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff060014ab0 con 0x7ff0680c6130 2023-10-30T12:29:11.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.315+0000 7ff064ff9640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0540664a0 0x7ff054068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.315+0000 7ff064ff9640 1 -- 172.21.15.125:0/815933790 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff060084940 con 0x7ff0680c6130 2023-10-30T12:29:11.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.315+0000 7ff067fff640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0540664a0 0x7ff054068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.315+0000 7ff067fff640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0540664a0 0x7ff054068960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0580ae000 tx=0x7ff0580a6070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:11.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.319+0000 7ff064ff9640 1 -- 172.21.15.125:0/815933790 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff060051c90 con 0x7ff0680c6130 2023-10-30T12:29:11.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.467+0000 7ff06e59d640 1 -- 172.21.15.125:0/815933790 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff0680c75d0 con 0x7ff0680c6130 2023-10-30T12:29:11.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.467+0000 7ff064ff9640 1 -- 172.21.15.125:0/815933790 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff0600552b0 con 0x7ff0680c6130 2023-10-30T12:29:11.470 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2023-10-30T12:29:11.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.467+0000 7ff06e59d640 1 -- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0540664a0 msgr2=0x7ff054068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.467+0000 7ff06e59d640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0540664a0 0x7ff054068960 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff0580ae000 tx=0x7ff0580a6070 comp rx=0 tx=0).stop 2023-10-30T12:29:11.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.467+0000 7ff06e59d640 1 -- 172.21.15.125:0/815933790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0680c6130 msgr2=0x7ff06815b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.467+0000 7ff06e59d640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0680c6130 0x7ff06815b410 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff060007e60 tx=0x7ff06000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.467+0000 7ff06e59d640 1 -- 172.21.15.125:0/815933790 shutdown_connections 2023-10-30T12:29:11.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.467+0000 7ff06e59d640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0540664a0 0x7ff054068960 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.467+0000 7ff06e59d640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0680c6130 0x7ff06815b410 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.467+0000 7ff06e59d640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0680c5810 0x7ff0681f6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.467+0000 7ff06e59d640 1 --2- 172.21.15.125:0/815933790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff06806a980 0x7ff0680c25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.467+0000 7ff06e59d640 1 -- 172.21.15.125:0/815933790 >> 172.21.15.125:0/815933790 conn(0x7ff0680c7910 msgr2=0x7ff0680c92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:11.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.467+0000 7ff06e59d640 1 -- 172.21.15.125:0/815933790 shutdown_connections 2023-10-30T12:29:11.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.467+0000 7ff06e59d640 1 -- 172.21.15.125:0/815933790 wait complete. 2023-10-30T12:29:11.490 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T12:29:11.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.627+0000 7fb7804d6640 1 Processor -- start 2023-10-30T12:29:11.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.627+0000 7fb7804d6640 1 -- start start 2023-10-30T12:29:11.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.627+0000 7fb7804d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77806f880 0x7fb77806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.627+0000 7fb7804d6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb77806a7b0 0x7fb778161f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.627+0000 7fb7804d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7780c6cb0 con 0x7fb7780c5810 2023-10-30T12:29:11.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.627+0000 7fb7804d6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb778071d90 con 0x7fb77806a7b0 2023-10-30T12:29:11.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.627+0000 7fb7804d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7781585d0 con 0x7fb77806f880 2023-10-30T12:29:11.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.631+0000 7fb77ea73640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb77806a7b0 0x7fb778161f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.631+0000 7fb77ea73640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb77806a7b0 0x7fb778161f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39888/0 (socket says 172.21.15.125:39888) 2023-10-30T12:29:11.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.631+0000 7fb77ea73640 1 -- 172.21.15.125:0/570385727 learned_addr learned my addr 172.21.15.125:0/570385727 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:11.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.631+0000 7fb77da71640 1 --2- 172.21.15.125:0/570385727 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77806f880 0x7fb77806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.631+0000 7fb77ea73640 1 -- 172.21.15.125:0/570385727 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77806f880 msgr2=0x7fb77806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:11.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.631+0000 7fb77ea73640 1 --2- 172.21.15.125:0/570385727 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77806f880 0x7fb77806a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.631+0000 7fb77ea73640 1 -- 172.21.15.125:0/570385727 >> v1:172.21.15.136:6789/0 conn(0x7fb7780c5810 legacy=0x7fb7780c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:11.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.631+0000 7fb77ea73640 1 -- 172.21.15.125:0/570385727 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7780c6620 con 0x7fb77806a7b0 2023-10-30T12:29:11.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.631+0000 7fb77ea73640 1 --2- 172.21.15.125:0/570385727 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb77806a7b0 0x7fb778161f10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb770009e80 tx=0x7fb77001c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=663f94739759b503 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:11.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.631+0000 7fb77d270640 1 -- 172.21.15.125:0/570385727 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7700a5590 con 0x7fb77806a7b0 2023-10-30T12:29:11.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.631+0000 7fb77d270640 1 -- 172.21.15.125:0/570385727 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7700a56f0 con 0x7fb77806a7b0 2023-10-30T12:29:11.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.631+0000 7fb77d270640 1 -- 172.21.15.125:0/570385727 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7700a5a80 con 0x7fb77806a7b0 2023-10-30T12:29:11.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 -- 172.21.15.125:0/570385727 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb77806a7b0 msgr2=0x7fb778161f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 --2- 172.21.15.125:0/570385727 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb77806a7b0 0x7fb778161f10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb770009e80 tx=0x7fb77001c8d0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 -- 172.21.15.125:0/570385727 shutdown_connections 2023-10-30T12:29:11.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 --2- 172.21.15.125:0/570385727 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb77806a7b0 0x7fb778161f10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 --2- 172.21.15.125:0/570385727 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77806f880 0x7fb77806a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 -- 172.21.15.125:0/570385727 >> 172.21.15.125:0/570385727 conn(0x7fb7780c7910 msgr2=0x7fb7780c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:11.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 -- 172.21.15.125:0/570385727 shutdown_connections 2023-10-30T12:29:11.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 -- 172.21.15.125:0/570385727 wait complete. 2023-10-30T12:29:11.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 Processor -- start 2023-10-30T12:29:11.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 -- start start 2023-10-30T12:29:11.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77806a7b0 0x7fb778154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77806f880 0x7fb778070b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7780c5810 0x7fb7781558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7781f3520 con 0x7fb7780c5810 2023-10-30T12:29:11.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77814eb00 con 0x7fb77806f880 2023-10-30T12:29:11.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb7804d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77814f610 con 0x7fb77806a7b0 2023-10-30T12:29:11.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb77ea73640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7780c5810 0x7fb7781558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb77ea73640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7780c5810 0x7fb7781558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39894/0 (socket says 172.21.15.125:39894) 2023-10-30T12:29:11.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb77ea73640 1 -- 172.21.15.125:0/482830276 learned_addr learned my addr 172.21.15.125:0/482830276 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:11.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb77da71640 1 --2- 172.21.15.125:0/482830276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77806f880 0x7fb778070b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb77e272640 1 --2- 172.21.15.125:0/482830276 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77806a7b0 0x7fb778154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb77ea73640 1 -- 172.21.15.125:0/482830276 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77806a7b0 msgr2=0x7fb778154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb77ea73640 1 --2- 172.21.15.125:0/482830276 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77806a7b0 0x7fb778154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb77ea73640 1 -- 172.21.15.125:0/482830276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77806f880 msgr2=0x7fb778070b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb77ea73640 1 --2- 172.21.15.125:0/482830276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77806f880 0x7fb778070b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.635+0000 7fb77ea73640 1 -- 172.21.15.125:0/482830276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb770009b30 con 0x7fb7780c5810 2023-10-30T12:29:11.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.639+0000 7fb77ea73640 1 --2- 172.21.15.125:0/482830276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7780c5810 0x7fb7781558a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb7700af040 tx=0x7fb770019670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:11.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.639+0000 7fb76effd640 1 -- 172.21.15.125:0/482830276 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7700a5590 con 0x7fb7780c5810 2023-10-30T12:29:11.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.639+0000 7fb76effd640 1 -- 172.21.15.125:0/482830276 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7700b8670 con 0x7fb7780c5810 2023-10-30T12:29:11.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.639+0000 7fb76effd640 1 -- 172.21.15.125:0/482830276 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7700b8a20 con 0x7fb7780c5810 2023-10-30T12:29:11.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.639+0000 7fb7804d6640 1 -- 172.21.15.125:0/482830276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb778154da0 con 0x7fb7780c5810 2023-10-30T12:29:11.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.639+0000 7fb7804d6640 1 -- 172.21.15.125:0/482830276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77814f050 con 0x7fb7780c5810 2023-10-30T12:29:11.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.639+0000 7fb7804d6640 1 -- 172.21.15.125:0/482830276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7780c75d0 con 0x7fb7780c5810 2023-10-30T12:29:11.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.639+0000 7fb76effd640 1 -- 172.21.15.125:0/482830276 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb7700b8b80 con 0x7fb7780c5810 2023-10-30T12:29:11.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.639+0000 7fb76effd640 1 --2- 172.21.15.125:0/482830276 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb75c0664a0 0x7fb75c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.639+0000 7fb76effd640 1 -- 172.21.15.125:0/482830276 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb77011f280 con 0x7fb7780c5810 2023-10-30T12:29:11.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.639+0000 7fb77e272640 1 --2- 172.21.15.125:0/482830276 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb75c0664a0 0x7fb75c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.643+0000 7fb77e272640 1 --2- 172.21.15.125:0/482830276 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb75c0664a0 0x7fb75c068960 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb774000fc0 tx=0x7fb774009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:11.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.643+0000 7fb76effd640 1 -- 172.21.15.125:0/482830276 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb7700ec670 con 0x7fb7780c5810 2023-10-30T12:29:11.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.791+0000 7fb7804d6640 1 -- 172.21.15.125:0/482830276 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb77804e170 con 0x7fb7780c5810 2023-10-30T12:29:11.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.795+0000 7fb76effd640 1 -- 172.21.15.125:0/482830276 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb770009510 con 0x7fb7780c5810 2023-10-30T12:29:11.797 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2023-10-30T12:29:11.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.795+0000 7fb7804d6640 1 -- 172.21.15.125:0/482830276 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb75c0664a0 msgr2=0x7fb75c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.795+0000 7fb7804d6640 1 --2- 172.21.15.125:0/482830276 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb75c0664a0 0x7fb75c068960 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb774000fc0 tx=0x7fb774009040 comp rx=0 tx=0).stop 2023-10-30T12:29:11.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.795+0000 7fb7804d6640 1 -- 172.21.15.125:0/482830276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7780c5810 msgr2=0x7fb7781558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.795+0000 7fb7804d6640 1 --2- 172.21.15.125:0/482830276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7780c5810 0x7fb7781558a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fb7700af040 tx=0x7fb770019670 comp rx=0 tx=0).stop 2023-10-30T12:29:11.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.795+0000 7fb7804d6640 1 -- 172.21.15.125:0/482830276 shutdown_connections 2023-10-30T12:29:11.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.795+0000 7fb7804d6640 1 --2- 172.21.15.125:0/482830276 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb75c0664a0 0x7fb75c068960 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.795+0000 7fb7804d6640 1 --2- 172.21.15.125:0/482830276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7780c5810 0x7fb7781558a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.795+0000 7fb7804d6640 1 --2- 172.21.15.125:0/482830276 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77806f880 0x7fb778070b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.795+0000 7fb7804d6640 1 --2- 172.21.15.125:0/482830276 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77806a7b0 0x7fb778154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.795+0000 7fb7804d6640 1 -- 172.21.15.125:0/482830276 >> 172.21.15.125:0/482830276 conn(0x7fb7780c7910 msgr2=0x7fb77806d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:11.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.795+0000 7fb7804d6640 1 -- 172.21.15.125:0/482830276 shutdown_connections 2023-10-30T12:29:11.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.795+0000 7fb7804d6640 1 -- 172.21.15.125:0/482830276 wait complete. 2023-10-30T12:29:11.818 INFO:tasks.mds_thrash:mds thrasher using random seed: 1698668951 2023-10-30T12:29:11.818 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:29:11.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.951+0000 7f12e709a640 1 Processor -- start 2023-10-30T12:29:11.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.951+0000 7f12e709a640 1 -- start start 2023-10-30T12:29:11.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.955+0000 7f12e709a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f12e00c05d0 0x7f12e013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.955+0000 7f12e709a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f12e00c6f60 0x7f12e013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.955+0000 7f12e709a640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e00c0a50 con 0x7f12e00c6b80 2023-10-30T12:29:11.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.955+0000 7f12e709a640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e006f6e0 con 0x7f12e00c6f60 2023-10-30T12:29:11.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.955+0000 7f12e709a640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e006f9c0 con 0x7f12e00c05d0 2023-10-30T12:29:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.955+0000 7f12e4e36640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f12e00c6b80 0x7f12e00bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60060/0 (socket says 172.21.15.125:60060) 2023-10-30T12:29:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.955+0000 7f12e4e36640 1 -- 172.21.15.125:0/2186256723 learned_addr learned my addr 172.21.15.125:0/2186256723 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:29:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.955+0000 7f12e5637640 1 --2- 172.21.15.125:0/2186256723 >> v2:172.21.15.125:3300/0 conn(0x7f12e00c6f60 0x7f12e013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.955+0000 7f12dffff640 1 --2- 172.21.15.125:0/2186256723 >> v2:172.21.15.136:3300/0 conn(0x7f12e00c05d0 0x7f12e013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.955+0000 7f12e5637640 1 -- 172.21.15.125:0/2186256723 >> v2:172.21.15.136:3300/0 conn(0x7f12e00c05d0 msgr2=0x7f12e013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.955+0000 7f12e5637640 1 --2- 172.21.15.125:0/2186256723 >> v2:172.21.15.136:3300/0 conn(0x7f12e00c05d0 0x7f12e013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.955+0000 7f12e5637640 1 -- 172.21.15.125:0/2186256723 >> v1:172.21.15.125:6789/0 conn(0x7f12e00c6b80 legacy=0x7f12e00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.955+0000 7f12e5637640 1 -- 172.21.15.125:0/2186256723 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e006fca0 con 0x7f12e00c6f60 2023-10-30T12:29:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.955+0000 7f12df7fe640 1 -- 172.21.15.125:0/2186256723 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040089085 0 0) 0x7f12e00c0a50 con 0x7f12e00c6b80 2023-10-30T12:29:11.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.959+0000 7f12e5637640 1 --2- 172.21.15.125:0/2186256723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e00c6f60 0x7f12e013f2d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f12e006c7e0 tx=0x7f12d801eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b7086ede3b4b03 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:11.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.959+0000 7f12df7fe640 1 -- 172.21.15.125:0/2186256723 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d80a7b00 con 0x7f12e00c6f60 2023-10-30T12:29:11.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.959+0000 7f12df7fe640 1 -- 172.21.15.125:0/2186256723 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d80a7c60 con 0x7f12e00c6f60 2023-10-30T12:29:11.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.959+0000 7f12df7fe640 1 -- 172.21.15.125:0/2186256723 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d80af440 con 0x7f12e00c6f60 2023-10-30T12:29:11.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.959+0000 7f12e709a640 1 -- 172.21.15.125:0/2186256723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e00c6f60 msgr2=0x7f12e013f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.959+0000 7f12e709a640 1 --2- 172.21.15.125:0/2186256723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e00c6f60 0x7f12e013f2d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f12e006c7e0 tx=0x7f12d801eb20 comp rx=0 tx=0).stop 2023-10-30T12:29:11.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.959+0000 7f12e709a640 1 -- 172.21.15.125:0/2186256723 shutdown_connections 2023-10-30T12:29:11.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.959+0000 7f12e709a640 1 --2- 172.21.15.125:0/2186256723 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e00c6f60 0x7f12e013f2d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.959+0000 7f12e709a640 1 --2- 172.21.15.125:0/2186256723 >> v2:172.21.15.136:3300/0 conn(0x7f12e00c05d0 0x7f12e013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.959+0000 7f12e709a640 1 -- 172.21.15.125:0/2186256723 >> 172.21.15.125:0/2186256723 conn(0x7f12e00c7910 msgr2=0x7f12e00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:11.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.959+0000 7f12e709a640 1 -- 172.21.15.125:0/2186256723 shutdown_connections 2023-10-30T12:29:11.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.959+0000 7f12e709a640 1 -- 172.21.15.125:0/2186256723 wait complete. 2023-10-30T12:29:11.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.959+0000 7f12e709a640 1 Processor -- start 2023-10-30T12:29:11.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.959+0000 7f12e709a640 1 -- start start 2023-10-30T12:29:11.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e709a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e00c05d0 0x7f12e0151ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e709a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12e00c6b80 0x7f12e01546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e709a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12e00c6f60 0x7f12e0157cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e709a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e0155b10 con 0x7f12e00c05d0 2023-10-30T12:29:11.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e709a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e0157820 con 0x7f12e00c6f60 2023-10-30T12:29:11.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e709a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e0152530 con 0x7f12e00c6b80 2023-10-30T12:29:11.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e4e36640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e00c05d0 0x7f12e0151ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e4e36640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e00c05d0 0x7f12e0151ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39912/0 (socket says 172.21.15.125:39912) 2023-10-30T12:29:11.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e4e36640 1 -- 172.21.15.125:0/107031827 learned_addr learned my addr 172.21.15.125:0/107031827 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:11.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12dffff640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12e00c6b80 0x7f12e01546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e5637640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12e00c6f60 0x7f12e0157cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e4e36640 1 -- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12e00c6b80 msgr2=0x7f12e01546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e4e36640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12e00c6b80 0x7f12e01546e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e4e36640 1 -- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12e00c6f60 msgr2=0x7f12e0157cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:11.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e4e36640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12e00c6f60 0x7f12e0157cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:11.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e4e36640 1 -- 172.21.15.125:0/107031827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d8009a70 con 0x7f12e00c05d0 2023-10-30T12:29:11.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e5637640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12e00c6f60 0x7f12e0157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:11.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e4e36640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e00c05d0 0x7f12e0151ff0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f12d400c520 tx=0x7f12d400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:11.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12dd7fa640 1 -- 172.21.15.125:0/107031827 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d40074e0 con 0x7f12e00c05d0 2023-10-30T12:29:11.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12dd7fa640 1 -- 172.21.15.125:0/107031827 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12d4007640 con 0x7f12e00c05d0 2023-10-30T12:29:11.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12dd7fa640 1 -- 172.21.15.125:0/107031827 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12d40049f0 con 0x7f12e00c05d0 2023-10-30T12:29:11.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e709a640 1 -- 172.21.15.125:0/107031827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e0150090 con 0x7f12e00c05d0 2023-10-30T12:29:11.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e709a640 1 -- 172.21.15.125:0/107031827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e0150c10 con 0x7f12e00c05d0 2023-10-30T12:29:11.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12e709a640 1 -- 172.21.15.125:0/107031827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e004e170 con 0x7f12e00c05d0 2023-10-30T12:29:11.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.963+0000 7f12dd7fa640 1 -- 172.21.15.125:0/107031827 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f12d4004b50 con 0x7f12e00c05d0 2023-10-30T12:29:11.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.967+0000 7f12dd7fa640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12c40664d0 0x7f12c4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:11.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.967+0000 7f12dd7fa640 1 -- 172.21.15.125:0/107031827 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12d4083120 con 0x7f12e00c05d0 2023-10-30T12:29:11.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.967+0000 7f12dffff640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12c40664d0 0x7f12c4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:11.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.967+0000 7f12dffff640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12c40664d0 0x7f12c4068990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f12e0159380 tx=0x7f12cc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:11.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:11.971+0000 7f12dd7fa640 1 -- 172.21.15.125:0/107031827 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f12d4050590 con 0x7f12e00c05d0 2023-10-30T12:29:12.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.123+0000 7f12e709a640 1 -- 172.21.15.125:0/107031827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12e0066510 con 0x7f12e00c05d0 2023-10-30T12:29:12.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.123+0000 7f12dd7fa640 1 -- 172.21.15.125:0/107031827 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16557 (secure 0 0 0) 0x7f12d4018090 con 0x7f12e00c05d0 2023-10-30T12:29:12.127 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:29:12.127 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:45.927862+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:29:12.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.127+0000 7f12e709a640 1 -- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12c40664d0 msgr2=0x7f12c4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.127+0000 7f12e709a640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12c40664d0 0x7f12c4068990 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f12e0159380 tx=0x7f12cc008040 comp rx=0 tx=0).stop 2023-10-30T12:29:12.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.127+0000 7f12e709a640 1 -- 172.21.15.125:0/107031827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e00c05d0 msgr2=0x7f12e0151ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.127+0000 7f12e709a640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e00c05d0 0x7f12e0151ff0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f12d400c520 tx=0x7f12d400c9e0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.127+0000 7f12e709a640 1 -- 172.21.15.125:0/107031827 shutdown_connections 2023-10-30T12:29:12.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.127+0000 7f12e709a640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12c40664d0 0x7f12c4068990 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.127+0000 7f12e709a640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12e00c6f60 0x7f12e0157cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.127+0000 7f12e709a640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12e00c6b80 0x7f12e01546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.127+0000 7f12e709a640 1 --2- 172.21.15.125:0/107031827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e00c05d0 0x7f12e0151ff0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.127+0000 7f12e709a640 1 -- 172.21.15.125:0/107031827 >> 172.21.15.125:0/107031827 conn(0x7f12e00c7910 msgr2=0x7f12e00c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:12.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.127+0000 7f12e709a640 1 -- 172.21.15.125:0/107031827 shutdown_connections 2023-10-30T12:29:12.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.127+0000 7f12e709a640 1 -- 172.21.15.125:0/107031827 wait complete. 2023-10-30T12:29:12.147 INFO:tasks.mds_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:31.063403+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T12:28:43.007849+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T12:28:43.401231+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:29:12.148 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:29:12.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.287+0000 7fd014809640 1 Processor -- start 2023-10-30T12:29:12.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.287+0000 7fd014809640 1 -- start start 2023-10-30T12:29:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd014809640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd010153c20 0x7fd01015a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd014809640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0100cb4c0 0x7fd0100cef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd014809640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd01015a6b0 con 0x7fd010153840 2023-10-30T12:29:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd014809640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd01015dbc0 con 0x7fd010153c20 2023-10-30T12:29:12.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd014809640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd01015d730 con 0x7fd0100cb4c0 2023-10-30T12:29:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd00ffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0100cb4c0 0x7fd0100cef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd00effd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd010153c20 0x7fd01015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd00effd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd010153c20 0x7fd01015a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:60606/0 (socket says 172.21.15.125:60606) 2023-10-30T12:29:12.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd00f7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd010153840 0x7fd0100c3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:51004/0 (socket says 172.21.15.125:51004) 2023-10-30T12:29:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd00effd640 1 -- 172.21.15.125:0/4114169270 learned_addr learned my addr 172.21.15.125:0/4114169270 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd00effd640 1 -- 172.21.15.125:0/4114169270 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0100cb4c0 msgr2=0x7fd0100cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd00effd640 1 --2- 172.21.15.125:0/4114169270 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0100cb4c0 0x7fd0100cef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd00effd640 1 -- 172.21.15.125:0/4114169270 >> v1:172.21.15.136:6789/0 conn(0x7fd010153840 legacy=0x7fd0100c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd00effd640 1 -- 172.21.15.125:0/4114169270 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0100ca8f0 con 0x7fd010153c20 2023-10-30T12:29:12.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.291+0000 7fd00ffff640 1 --2- 172.21.15.125:0/4114169270 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0100cb4c0 0x7fd0100cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:29:12.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.295+0000 7fd00effd640 1 --2- 172.21.15.125:0/4114169270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd010153c20 0x7fd01015a170 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd004009870 tx=0x7fd00401ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4a0f8b715d156f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.295+0000 7fd00e7fc640 1 -- 172.21.15.125:0/4114169270 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0040a8a30 con 0x7fd010153c20 2023-10-30T12:29:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.295+0000 7fd00e7fc640 1 -- 172.21.15.125:0/4114169270 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0040a8b90 con 0x7fd010153c20 2023-10-30T12:29:12.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.295+0000 7fd00e7fc640 1 -- 172.21.15.125:0/4114169270 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0040b0440 con 0x7fd010153c20 2023-10-30T12:29:12.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.295+0000 7fd014809640 1 -- 172.21.15.125:0/4114169270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd010153c20 msgr2=0x7fd01015a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.295+0000 7fd014809640 1 --2- 172.21.15.125:0/4114169270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd010153c20 0x7fd01015a170 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fd004009870 tx=0x7fd00401ed30 comp rx=0 tx=0).stop 2023-10-30T12:29:12.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.295+0000 7fd014809640 1 -- 172.21.15.125:0/4114169270 shutdown_connections 2023-10-30T12:29:12.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.295+0000 7fd014809640 1 --2- 172.21.15.125:0/4114169270 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0100cb4c0 0x7fd0100cef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.295+0000 7fd014809640 1 --2- 172.21.15.125:0/4114169270 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd010153c20 0x7fd01015a170 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.295+0000 7fd014809640 1 -- 172.21.15.125:0/4114169270 >> 172.21.15.125:0/4114169270 conn(0x7fd01006be40 msgr2=0x7fd01006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:12.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.295+0000 7fd014809640 1 -- 172.21.15.125:0/4114169270 shutdown_connections 2023-10-30T12:29:12.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.295+0000 7fd014809640 1 -- 172.21.15.125:0/4114169270 wait complete. 2023-10-30T12:29:12.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.295+0000 7fd014809640 1 Processor -- start 2023-10-30T12:29:12.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.295+0000 7fd014809640 1 -- start start 2023-10-30T12:29:12.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd014809640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0100cb4c0 0x7fd0101f2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd014809640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd010153840 0x7fd0100c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd014809640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd010153c20 0x7fd0100c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd014809640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd010159640 con 0x7fd010153840 2023-10-30T12:29:12.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd014809640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd010152b50 con 0x7fd0100cb4c0 2023-10-30T12:29:12.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd014809640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0100c7ee0 con 0x7fd010153c20 2023-10-30T12:29:12.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd00effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd010153840 0x7fd0100c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd00effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd010153840 0x7fd0100c8370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39928/0 (socket says 172.21.15.125:39928) 2023-10-30T12:29:12.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd00effd640 1 -- 172.21.15.125:0/2065456294 learned_addr learned my addr 172.21.15.125:0/2065456294 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:12.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd00ffff640 1 --2- 172.21.15.125:0/2065456294 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd010153c20 0x7fd0100c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd00f7fe640 1 --2- 172.21.15.125:0/2065456294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0100cb4c0 0x7fd0101f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd00effd640 1 -- 172.21.15.125:0/2065456294 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd010153c20 msgr2=0x7fd0100c88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd00effd640 1 --2- 172.21.15.125:0/2065456294 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd010153c20 0x7fd0100c88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd00effd640 1 -- 172.21.15.125:0/2065456294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0100cb4c0 msgr2=0x7fd0101f2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd00effd640 1 --2- 172.21.15.125:0/2065456294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0100cb4c0 0x7fd0101f2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd00effd640 1 -- 172.21.15.125:0/2065456294 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd008009a70 con 0x7fd010153840 2023-10-30T12:29:12.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd00effd640 1 --2- 172.21.15.125:0/2065456294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd010153840 0x7fd0100c8370 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd0040b3040 tx=0x7fd00401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:12.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fcff7fff640 1 -- 172.21.15.125:0/2065456294 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0040b0dd0 con 0x7fd010153840 2023-10-30T12:29:12.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fcff7fff640 1 -- 172.21.15.125:0/2065456294 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0040bb460 con 0x7fd010153840 2023-10-30T12:29:12.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fcff7fff640 1 -- 172.21.15.125:0/2065456294 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0040bb810 con 0x7fd010153840 2023-10-30T12:29:12.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd014809640 1 -- 172.21.15.125:0/2065456294 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0040094d0 con 0x7fd010153840 2023-10-30T12:29:12.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd014809640 1 -- 172.21.15.125:0/2065456294 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0101f2fc0 con 0x7fd010153840 2023-10-30T12:29:12.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fd014809640 1 -- 172.21.15.125:0/2065456294 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd01004e9c0 con 0x7fd010153840 2023-10-30T12:29:12.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.299+0000 7fcff7fff640 1 -- 172.21.15.125:0/2065456294 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0040b83f0 con 0x7fd010153840 2023-10-30T12:29:12.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.303+0000 7fcff7fff640 1 --2- 172.21.15.125:0/2065456294 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcffc0664a0 0x7fcffc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.303+0000 7fcff7fff640 1 -- 172.21.15.125:0/2065456294 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd004121760 con 0x7fd010153840 2023-10-30T12:29:12.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.303+0000 7fd00f7fe640 1 --2- 172.21.15.125:0/2065456294 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcffc0664a0 0x7fcffc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.303+0000 7fd00f7fe640 1 --2- 172.21.15.125:0/2065456294 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcffc0664a0 0x7fcffc068960 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd000000f70 tx=0x7fd000007c90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:12.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.307+0000 7fcff7fff640 1 -- 172.21.15.125:0/2065456294 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd0040eeb40 con 0x7fd010153840 2023-10-30T12:29:12.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.455+0000 7fd014809640 1 -- 172.21.15.125:0/2065456294 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd0100cb000 con 0x7fd010153840 2023-10-30T12:29:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.455+0000 7fcff7fff640 1 -- 172.21.15.125:0/2065456294 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd0040f2160 con 0x7fd010153840 2023-10-30T12:29:12.458 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2023-10-30T12:29:12.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.455+0000 7fd014809640 1 -- 172.21.15.125:0/2065456294 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcffc0664a0 msgr2=0x7fcffc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.455+0000 7fd014809640 1 --2- 172.21.15.125:0/2065456294 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcffc0664a0 0x7fcffc068960 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd000000f70 tx=0x7fd000007c90 comp rx=0 tx=0).stop 2023-10-30T12:29:12.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.455+0000 7fd014809640 1 -- 172.21.15.125:0/2065456294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd010153840 msgr2=0x7fd0100c8370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.455+0000 7fd014809640 1 --2- 172.21.15.125:0/2065456294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd010153840 0x7fd0100c8370 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd0040b3040 tx=0x7fd00401bad0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.455+0000 7fd014809640 1 -- 172.21.15.125:0/2065456294 shutdown_connections 2023-10-30T12:29:12.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.455+0000 7fd014809640 1 --2- 172.21.15.125:0/2065456294 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcffc0664a0 0x7fcffc068960 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.455+0000 7fd014809640 1 --2- 172.21.15.125:0/2065456294 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd010153c20 0x7fd0100c88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.455+0000 7fd014809640 1 --2- 172.21.15.125:0/2065456294 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd010153840 0x7fd0100c8370 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.455+0000 7fd014809640 1 --2- 172.21.15.125:0/2065456294 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0100cb4c0 0x7fd0101f2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.455+0000 7fd014809640 1 -- 172.21.15.125:0/2065456294 >> 172.21.15.125:0/2065456294 conn(0x7fd01006be40 msgr2=0x7fd0101520d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:12.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.455+0000 7fd014809640 1 -- 172.21.15.125:0/2065456294 shutdown_connections 2023-10-30T12:29:12.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.455+0000 7fd014809640 1 -- 172.21.15.125:0/2065456294 wait complete. 2023-10-30T12:29:12.479 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T12:29:12.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf2833e640 1 Processor -- start 2023-10-30T12:29:12.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf2833e640 1 -- start start 2023-10-30T12:29:12.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf2833e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fcf200c5810 0x7fcf200c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf2833e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcf200c6130 0x7fcf2006a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf2833e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf200c6cb0 con 0x7fcf2006f880 2023-10-30T12:29:12.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf2833e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf20071d90 con 0x7fcf200c6130 2023-10-30T12:29:12.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf2833e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf201585d0 con 0x7fcf200c5810 2023-10-30T12:29:12.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf258d9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcf200c6130 0x7fcf2006a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf258d9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcf200c6130 0x7fcf2006a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39932/0 (socket says 172.21.15.125:39932) 2023-10-30T12:29:12.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf258d9640 1 -- 172.21.15.125:0/3557054096 learned_addr learned my addr 172.21.15.125:0/3557054096 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:12.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf260da640 1 --2- 172.21.15.125:0/3557054096 >> v2:172.21.15.136:3300/0 conn(0x7fcf200c5810 0x7fcf200c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf268db640 1 --1- 172.21.15.125:0/3557054096 >> v1:172.21.15.136:6789/0 conn(0x7fcf2006f880 0x7fcf2015ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:51020/0 (socket says 172.21.15.125:51020) 2023-10-30T12:29:12.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf258d9640 1 -- 172.21.15.125:0/3557054096 >> v2:172.21.15.136:3300/0 conn(0x7fcf200c5810 msgr2=0x7fcf200c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:12.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf258d9640 1 --2- 172.21.15.125:0/3557054096 >> v2:172.21.15.136:3300/0 conn(0x7fcf200c5810 0x7fcf200c5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf258d9640 1 -- 172.21.15.125:0/3557054096 >> v1:172.21.15.136:6789/0 conn(0x7fcf2006f880 legacy=0x7fcf2015ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.619+0000 7fcf258d9640 1 -- 172.21.15.125:0/3557054096 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf200c6620 con 0x7fcf200c6130 2023-10-30T12:29:12.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.623+0000 7fcf258d9640 1 --2- 172.21.15.125:0/3557054096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf200c6130 0x7fcf2006a270 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcf2006b380 tx=0x7fcf1001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270e0338f5b3c4d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:12.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.623+0000 7fcf250d8640 1 -- 172.21.15.125:0/3557054096 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf100ab600 con 0x7fcf200c6130 2023-10-30T12:29:12.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.623+0000 7fcf250d8640 1 -- 172.21.15.125:0/3557054096 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf100a7050 con 0x7fcf200c6130 2023-10-30T12:29:12.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.623+0000 7fcf250d8640 1 -- 172.21.15.125:0/3557054096 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf100ab980 con 0x7fcf200c6130 2023-10-30T12:29:12.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.623+0000 7fcf2833e640 1 -- 172.21.15.125:0/3557054096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf200c6130 msgr2=0x7fcf2006a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.623+0000 7fcf2833e640 1 --2- 172.21.15.125:0/3557054096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf200c6130 0x7fcf2006a270 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fcf2006b380 tx=0x7fcf1001c3e0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.623+0000 7fcf2833e640 1 -- 172.21.15.125:0/3557054096 shutdown_connections 2023-10-30T12:29:12.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.623+0000 7fcf2833e640 1 --2- 172.21.15.125:0/3557054096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf200c6130 0x7fcf2006a270 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.623+0000 7fcf2833e640 1 --2- 172.21.15.125:0/3557054096 >> v2:172.21.15.136:3300/0 conn(0x7fcf200c5810 0x7fcf200c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.623+0000 7fcf2833e640 1 -- 172.21.15.125:0/3557054096 >> 172.21.15.125:0/3557054096 conn(0x7fcf200c7910 msgr2=0x7fcf200c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:12.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.623+0000 7fcf2833e640 1 -- 172.21.15.125:0/3557054096 shutdown_connections 2023-10-30T12:29:12.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.623+0000 7fcf2833e640 1 -- 172.21.15.125:0/3557054096 wait complete. 2023-10-30T12:29:12.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf2833e640 1 Processor -- start 2023-10-30T12:29:12.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf2833e640 1 -- start start 2023-10-30T12:29:12.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf2833e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf2006f880 0x7fcf2014b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf2833e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf200c5810 0x7fcf201f0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf2833e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcf200c6130 0x7fcf2014ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf2833e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf201f4020 con 0x7fcf200c5810 2023-10-30T12:29:12.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf2833e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf20151170 con 0x7fcf2006f880 2023-10-30T12:29:12.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf2833e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2014bbb0 con 0x7fcf200c6130 2023-10-30T12:29:12.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf258d9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf200c5810 0x7fcf201f0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf258d9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf200c5810 0x7fcf201f0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39944/0 (socket says 172.21.15.125:39944) 2023-10-30T12:29:12.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf258d9640 1 -- 172.21.15.125:0/102589653 learned_addr learned my addr 172.21.15.125:0/102589653 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:12.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf260da640 1 --2- 172.21.15.125:0/102589653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf2006f880 0x7fcf2014b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf268db640 1 --2- 172.21.15.125:0/102589653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcf200c6130 0x7fcf2014ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf258d9640 1 -- 172.21.15.125:0/102589653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcf200c6130 msgr2=0x7fcf2014ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf258d9640 1 --2- 172.21.15.125:0/102589653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcf200c6130 0x7fcf2014ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf258d9640 1 -- 172.21.15.125:0/102589653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf2006f880 msgr2=0x7fcf2014b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf258d9640 1 --2- 172.21.15.125:0/102589653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf2006f880 0x7fcf2014b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf258d9640 1 -- 172.21.15.125:0/102589653 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf100094d0 con 0x7fcf200c5810 2023-10-30T12:29:12.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf258d9640 1 --2- 172.21.15.125:0/102589653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf200c5810 0x7fcf201f0270 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcf100ae040 tx=0x7fcf10019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:12.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf16ffd640 1 -- 172.21.15.125:0/102589653 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf100b4420 con 0x7fcf200c5810 2023-10-30T12:29:12.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf16ffd640 1 -- 172.21.15.125:0/102589653 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf100b4580 con 0x7fcf200c5810 2023-10-30T12:29:12.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf2833e640 1 -- 172.21.15.125:0/102589653 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2014be90 con 0x7fcf200c5810 2023-10-30T12:29:12.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf16ffd640 1 -- 172.21.15.125:0/102589653 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf100b4930 con 0x7fcf200c5810 2023-10-30T12:29:12.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.627+0000 7fcf2833e640 1 -- 172.21.15.125:0/102589653 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf201f3c40 con 0x7fcf200c5810 2023-10-30T12:29:12.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.631+0000 7fcf2833e640 1 -- 172.21.15.125:0/102589653 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf200c2e70 con 0x7fcf200c5810 2023-10-30T12:29:12.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.631+0000 7fcf16ffd640 1 -- 172.21.15.125:0/102589653 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcf100b2070 con 0x7fcf200c5810 2023-10-30T12:29:12.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.631+0000 7fcf16ffd640 1 --2- 172.21.15.125:0/102589653 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcf040664a0 0x7fcf04068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.631+0000 7fcf16ffd640 1 -- 172.21.15.125:0/102589653 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcf1011d960 con 0x7fcf200c5810 2023-10-30T12:29:12.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.631+0000 7fcf260da640 1 --2- 172.21.15.125:0/102589653 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcf040664a0 0x7fcf04068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.631+0000 7fcf260da640 1 --2- 172.21.15.125:0/102589653 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcf040664a0 0x7fcf04068960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009a30 tx=0x7fcf1c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:12.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.635+0000 7fcf16ffd640 1 -- 172.21.15.125:0/102589653 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcf100eae60 con 0x7fcf200c5810 2023-10-30T12:29:12.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.783+0000 7fcf2833e640 1 -- 172.21.15.125:0/102589653 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcf201f4160 con 0x7fcf200c5810 2023-10-30T12:29:12.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.783+0000 7fcf16ffd640 1 -- 172.21.15.125:0/102589653 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fcf100ee480 con 0x7fcf200c5810 2023-10-30T12:29:12.787 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2023-10-30T12:29:12.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.787+0000 7fcf2833e640 1 -- 172.21.15.125:0/102589653 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcf040664a0 msgr2=0x7fcf04068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.787+0000 7fcf2833e640 1 --2- 172.21.15.125:0/102589653 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcf040664a0 0x7fcf04068960 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009a30 tx=0x7fcf1c008040 comp rx=0 tx=0).stop 2023-10-30T12:29:12.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.787+0000 7fcf2833e640 1 -- 172.21.15.125:0/102589653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf200c5810 msgr2=0x7fcf201f0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.787+0000 7fcf2833e640 1 --2- 172.21.15.125:0/102589653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf200c5810 0x7fcf201f0270 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fcf100ae040 tx=0x7fcf10019180 comp rx=0 tx=0).stop 2023-10-30T12:29:12.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.787+0000 7fcf2833e640 1 -- 172.21.15.125:0/102589653 shutdown_connections 2023-10-30T12:29:12.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.787+0000 7fcf2833e640 1 --2- 172.21.15.125:0/102589653 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcf040664a0 0x7fcf04068960 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.787+0000 7fcf2833e640 1 --2- 172.21.15.125:0/102589653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcf200c6130 0x7fcf2014ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.787+0000 7fcf2833e640 1 --2- 172.21.15.125:0/102589653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcf200c5810 0x7fcf201f0270 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.787+0000 7fcf2833e640 1 --2- 172.21.15.125:0/102589653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcf2006f880 0x7fcf2014b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.787+0000 7fcf2833e640 1 -- 172.21.15.125:0/102589653 >> 172.21.15.125:0/102589653 conn(0x7fcf200c7910 msgr2=0x7fcf20156ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:12.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.787+0000 7fcf2833e640 1 -- 172.21.15.125:0/102589653 shutdown_connections 2023-10-30T12:29:12.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.787+0000 7fcf2833e640 1 -- 172.21.15.125:0/102589653 wait complete. 2023-10-30T12:29:12.807 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T12:29:12.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.943+0000 7f07da99d640 1 Processor -- start 2023-10-30T12:29:12.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.943+0000 7f07da99d640 1 -- start start 2023-10-30T12:29:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07da99d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07d41508c0 0x7f07d40c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07da99d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f07d40722a0 0x7f07d40c6e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07da99d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f07d40bfec0 0x7f07d40c0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07da99d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d40c85b0 con 0x7f07d40722a0 2023-10-30T12:29:12.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07da99d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d40c0a30 con 0x7f07d40bfec0 2023-10-30T12:29:12.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07da99d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d406f650 con 0x7f07d41508c0 2023-10-30T12:29:12.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07d37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f07d40722a0 0x7f07d40c6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07d37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f07d40722a0 0x7f07d40c6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39958/0 (socket says 172.21.15.125:39958) 2023-10-30T12:29:12.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07d37fe640 1 -- 172.21.15.125:0/1319482055 learned_addr learned my addr 172.21.15.125:0/1319482055 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:12.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07d8f3a640 1 --2- 172.21.15.125:0/1319482055 >> v2:172.21.15.136:3300/0 conn(0x7f07d40bfec0 0x7f07d40c0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07d3fff640 1 --2- 172.21.15.125:0/1319482055 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07d41508c0 0x7f07d40c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07d37fe640 1 -- 172.21.15.125:0/1319482055 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07d41508c0 msgr2=0x7f07d40c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:12.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07d37fe640 1 --2- 172.21.15.125:0/1319482055 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07d41508c0 0x7f07d40c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07d37fe640 1 -- 172.21.15.125:0/1319482055 >> v2:172.21.15.136:3300/0 conn(0x7f07d40bfec0 msgr2=0x7f07d40c0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:12.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07d37fe640 1 --2- 172.21.15.125:0/1319482055 >> v2:172.21.15.136:3300/0 conn(0x7f07d40bfec0 0x7f07d40c0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07d37fe640 1 -- 172.21.15.125:0/1319482055 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d406f930 con 0x7f07d40722a0 2023-10-30T12:29:12.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.947+0000 7f07d3fff640 1 --2- 172.21.15.125:0/1319482055 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07d41508c0 0x7f07d40c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:12.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07d37fe640 1 --2- 172.21.15.125:0/1319482055 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07d40722a0 0x7f07d40c6e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f07d406f1b0 tx=0x7f07c801e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4196164842ca92af server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:12.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07d2ffd640 1 -- 172.21.15.125:0/1319482055 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c80ab400 con 0x7f07d40722a0 2023-10-30T12:29:12.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07d2ffd640 1 -- 172.21.15.125:0/1319482055 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c80ab560 con 0x7f07d40722a0 2023-10-30T12:29:12.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07d2ffd640 1 -- 172.21.15.125:0/1319482055 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c80ab8f0 con 0x7f07d40722a0 2023-10-30T12:29:12.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07da99d640 1 -- 172.21.15.125:0/1319482055 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07d40722a0 msgr2=0x7f07d40c6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07da99d640 1 --2- 172.21.15.125:0/1319482055 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07d40722a0 0x7f07d40c6e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f07d406f1b0 tx=0x7f07c801e680 comp rx=0 tx=0).stop 2023-10-30T12:29:12.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07da99d640 1 -- 172.21.15.125:0/1319482055 shutdown_connections 2023-10-30T12:29:12.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07da99d640 1 --2- 172.21.15.125:0/1319482055 >> v2:172.21.15.136:3300/0 conn(0x7f07d40bfec0 0x7f07d40c0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07da99d640 1 --2- 172.21.15.125:0/1319482055 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07d40722a0 0x7f07d40c6e60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07da99d640 1 --2- 172.21.15.125:0/1319482055 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07d41508c0 0x7f07d40c6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07da99d640 1 -- 172.21.15.125:0/1319482055 >> 172.21.15.125:0/1319482055 conn(0x7f07d40c7910 msgr2=0x7f07d40c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:12.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07da99d640 1 -- 172.21.15.125:0/1319482055 shutdown_connections 2023-10-30T12:29:12.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07da99d640 1 -- 172.21.15.125:0/1319482055 wait complete. 2023-10-30T12:29:12.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07da99d640 1 Processor -- start 2023-10-30T12:29:12.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.951+0000 7f07da99d640 1 -- start start 2023-10-30T12:29:12.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07da99d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07d40722a0 0x7f07d41f2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07da99d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07d40bfec0 0x7f07d4145870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07da99d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07d41508c0 0x7f07d414ac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07da99d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d4156260 con 0x7f07d41508c0 2023-10-30T12:29:12.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07da99d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d4160da0 con 0x7f07d40bfec0 2023-10-30T12:29:12.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07da99d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d4155090 con 0x7f07d40722a0 2023-10-30T12:29:12.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07d8f3a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07d41508c0 0x7f07d414ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07d8f3a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07d41508c0 0x7f07d414ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39970/0 (socket says 172.21.15.125:39970) 2023-10-30T12:29:12.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07d8f3a640 1 -- 172.21.15.125:0/3742605154 learned_addr learned my addr 172.21.15.125:0/3742605154 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:12.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07d37fe640 1 --2- 172.21.15.125:0/3742605154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07d40bfec0 0x7f07d4145870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07d3fff640 1 --2- 172.21.15.125:0/3742605154 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07d40722a0 0x7f07d41f2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07d8f3a640 1 -- 172.21.15.125:0/3742605154 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07d40722a0 msgr2=0x7f07d41f2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07d8f3a640 1 --2- 172.21.15.125:0/3742605154 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07d40722a0 0x7f07d41f2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07d8f3a640 1 -- 172.21.15.125:0/3742605154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07d40bfec0 msgr2=0x7f07d4145870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:12.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07d8f3a640 1 --2- 172.21.15.125:0/3742605154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07d40bfec0 0x7f07d4145870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:12.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07d8f3a640 1 -- 172.21.15.125:0/3742605154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c80094d0 con 0x7f07d41508c0 2023-10-30T12:29:12.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07d8f3a640 1 --2- 172.21.15.125:0/3742605154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07d41508c0 0x7f07d414ac00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f07cc00ef00 tx=0x7f07cc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:12.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07d0ff9640 1 -- 172.21.15.125:0/3742605154 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc010040 con 0x7f07d41508c0 2023-10-30T12:29:12.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07d0ff9640 1 -- 172.21.15.125:0/3742605154 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07cc014400 con 0x7f07d41508c0 2023-10-30T12:29:12.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07da99d640 1 -- 172.21.15.125:0/3742605154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d4155300 con 0x7f07d41508c0 2023-10-30T12:29:12.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07d0ff9640 1 -- 172.21.15.125:0/3742605154 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc009ea0 con 0x7f07d41508c0 2023-10-30T12:29:12.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07da99d640 1 -- 172.21.15.125:0/3742605154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d41558c0 con 0x7f07d41508c0 2023-10-30T12:29:12.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.955+0000 7f07da99d640 1 -- 172.21.15.125:0/3742605154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d41563a0 con 0x7f07d41508c0 2023-10-30T12:29:12.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.959+0000 7f07d0ff9640 1 -- 172.21.15.125:0/3742605154 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f07cc0147b0 con 0x7f07d41508c0 2023-10-30T12:29:12.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.959+0000 7f07d0ff9640 1 --2- 172.21.15.125:0/3742605154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07c00664a0 0x7f07c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:12.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.959+0000 7f07d0ff9640 1 -- 172.21.15.125:0/3742605154 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07cc084870 con 0x7f07d41508c0 2023-10-30T12:29:12.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.959+0000 7f07d3fff640 1 --2- 172.21.15.125:0/3742605154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07c00664a0 0x7f07c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:12.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.959+0000 7f07d3fff640 1 --2- 172.21.15.125:0/3742605154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07c00664a0 0x7f07c0068960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f07c4006fd0 tx=0x7f07c4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:12.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:12.963+0000 7f07d0ff9640 1 -- 172.21.15.125:0/3742605154 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f07d41563a0 con 0x7f07d41508c0 2023-10-30T12:29:13.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.115+0000 7f07da99d640 1 -- 172.21.15.125:0/3742605154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f07d41563a0 con 0x7f07d41508c0 2023-10-30T12:29:13.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.119+0000 7f07d0ff9640 1 -- 172.21.15.125:0/3742605154 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f07d41563a0 con 0x7f07d41508c0 2023-10-30T12:29:13.121 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2023-10-30T12:29:13.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.119+0000 7f07da99d640 1 -- 172.21.15.125:0/3742605154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07c00664a0 msgr2=0x7f07c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.119+0000 7f07da99d640 1 --2- 172.21.15.125:0/3742605154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07c00664a0 0x7f07c0068960 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f07c4006fd0 tx=0x7f07c4008040 comp rx=0 tx=0).stop 2023-10-30T12:29:13.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.119+0000 7f07da99d640 1 -- 172.21.15.125:0/3742605154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07d41508c0 msgr2=0x7f07d414ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.119+0000 7f07da99d640 1 --2- 172.21.15.125:0/3742605154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07d41508c0 0x7f07d414ac00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f07cc00ef00 tx=0x7f07cc00b480 comp rx=0 tx=0).stop 2023-10-30T12:29:13.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.119+0000 7f07da99d640 1 -- 172.21.15.125:0/3742605154 shutdown_connections 2023-10-30T12:29:13.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.119+0000 7f07da99d640 1 --2- 172.21.15.125:0/3742605154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07c00664a0 0x7f07c0068960 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.119+0000 7f07da99d640 1 --2- 172.21.15.125:0/3742605154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07d41508c0 0x7f07d414ac00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.119+0000 7f07da99d640 1 --2- 172.21.15.125:0/3742605154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07d40bfec0 0x7f07d4145870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.119+0000 7f07da99d640 1 --2- 172.21.15.125:0/3742605154 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07d40722a0 0x7f07d41f2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.119+0000 7f07da99d640 1 -- 172.21.15.125:0/3742605154 >> 172.21.15.125:0/3742605154 conn(0x7f07d40c7910 msgr2=0x7f07d406d220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.119+0000 7f07da99d640 1 -- 172.21.15.125:0/3742605154 shutdown_connections 2023-10-30T12:29:13.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.119+0000 7f07da99d640 1 -- 172.21.15.125:0/3742605154 wait complete. 2023-10-30T12:29:13.141 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2023-10-30T12:29:13.142 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:29:13.145 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:29:13.190 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:29:13.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.279+0000 7fd416417640 1 Processor -- start 2023-10-30T12:29:13.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.279+0000 7fd416417640 1 -- start start 2023-10-30T12:29:13.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.279+0000 7fd416417640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd410153bf0 0x7fd410071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.279+0000 7fd416417640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4100c5f60 0x7fd4100cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.279+0000 7fd416417640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4100700b0 con 0x7fd4100c6fe0 2023-10-30T12:29:13.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.279+0000 7fd416417640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4100c6810 con 0x7fd410153bf0 2023-10-30T12:29:13.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.279+0000 7fd416417640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd410159900 con 0x7fd4100c5f60 2023-10-30T12:29:13.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.279+0000 7fd40ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd410153bf0 0x7fd410071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.279+0000 7fd40ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd410153bf0 0x7fd410071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39974/0 (socket says 172.21.15.125:39974) 2023-10-30T12:29:13.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.279+0000 7fd40ffff640 1 -- 172.21.15.125:0/2851058326 learned_addr learned my addr 172.21.15.125:0/2851058326 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:13.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.279+0000 7fd4149b4640 1 --2- 172.21.15.125:0/2851058326 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4100c5f60 0x7fd4100cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.283+0000 7fd40ffff640 1 -- 172.21.15.125:0/2851058326 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4100c5f60 msgr2=0x7fd4100cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.283+0000 7fd40ffff640 1 --2- 172.21.15.125:0/2851058326 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4100c5f60 0x7fd4100cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.283+0000 7fd40ffff640 1 -- 172.21.15.125:0/2851058326 >> v1:172.21.15.125:6789/0 conn(0x7fd4100c6fe0 legacy=0x7fd4100c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.283+0000 7fd40ffff640 1 -- 172.21.15.125:0/2851058326 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4100c6530 con 0x7fd410153bf0 2023-10-30T12:29:13.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.283+0000 7fd40effd640 1 -- 172.21.15.125:0/2851058326 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378014920 0 0) 0x7fd4100700b0 con 0x7fd4100c6fe0 2023-10-30T12:29:13.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.283+0000 7fd40ffff640 1 --2- 172.21.15.125:0/2851058326 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd410153bf0 0x7fd410071260 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd400009870 tx=0x7fd40001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf240367ae2a0551 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.283+0000 7fd40effd640 1 -- 172.21.15.125:0/2851058326 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4000a8a30 con 0x7fd410153bf0 2023-10-30T12:29:13.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.283+0000 7fd40effd640 1 -- 172.21.15.125:0/2851058326 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4000a8b90 con 0x7fd410153bf0 2023-10-30T12:29:13.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.283+0000 7fd40effd640 1 -- 172.21.15.125:0/2851058326 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4000b0440 con 0x7fd410153bf0 2023-10-30T12:29:13.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.283+0000 7fd416417640 1 -- 172.21.15.125:0/2851058326 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd410153bf0 msgr2=0x7fd410071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.283+0000 7fd416417640 1 --2- 172.21.15.125:0/2851058326 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd410153bf0 0x7fd410071260 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd400009870 tx=0x7fd40001ed30 comp rx=0 tx=0).stop 2023-10-30T12:29:13.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd416417640 1 -- 172.21.15.125:0/2851058326 shutdown_connections 2023-10-30T12:29:13.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd416417640 1 --2- 172.21.15.125:0/2851058326 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4100c5f60 0x7fd4100cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd416417640 1 --2- 172.21.15.125:0/2851058326 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd410153bf0 0x7fd410071260 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd416417640 1 -- 172.21.15.125:0/2851058326 >> 172.21.15.125:0/2851058326 conn(0x7fd41006c6e0 msgr2=0x7fd410069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd416417640 1 -- 172.21.15.125:0/2851058326 shutdown_connections 2023-10-30T12:29:13.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd416417640 1 -- 172.21.15.125:0/2851058326 wait complete. 2023-10-30T12:29:13.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd416417640 1 Processor -- start 2023-10-30T12:29:13.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd416417640 1 -- start start 2023-10-30T12:29:13.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd416417640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4100c5f60 0x7fd4100ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd416417640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4100c6fe0 0x7fd41014e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd416417640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd410153bf0 0x7fd41014e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd416417640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd41014d1f0 con 0x7fd4100c6fe0 2023-10-30T12:29:13.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd416417640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4101eefd0 con 0x7fd4100c5f60 2023-10-30T12:29:13.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd416417640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4101efdf0 con 0x7fd410153bf0 2023-10-30T12:29:13.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd40f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4100c6fe0 0x7fd41014e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd4149b4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd410153bf0 0x7fd41014e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd40ffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4100c5f60 0x7fd4100ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd40f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4100c6fe0 0x7fd41014e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39978/0 (socket says 172.21.15.125:39978) 2023-10-30T12:29:13.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd40f7fe640 1 -- 172.21.15.125:0/3179199534 learned_addr learned my addr 172.21.15.125:0/3179199534 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:13.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd4149b4640 1 --2- 172.21.15.125:0/3179199534 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd410153bf0 0x7fd41014e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:34946/0 (socket says 172.21.15.125:34946) 2023-10-30T12:29:13.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd40f7fe640 1 -- 172.21.15.125:0/3179199534 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd410153bf0 msgr2=0x7fd41014e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd40f7fe640 1 --2- 172.21.15.125:0/3179199534 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd410153bf0 0x7fd41014e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd40f7fe640 1 -- 172.21.15.125:0/3179199534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4100c5f60 msgr2=0x7fd4100ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd40f7fe640 1 --2- 172.21.15.125:0/3179199534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4100c5f60 0x7fd4100ca790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.287+0000 7fd40f7fe640 1 -- 172.21.15.125:0/3179199534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4000094d0 con 0x7fd4100c6fe0 2023-10-30T12:29:13.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.291+0000 7fd40f7fe640 1 --2- 172.21.15.125:0/3179199534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4100c6fe0 0x7fd41014e050 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd40400c4d0 tx=0x7fd40400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.291+0000 7fd40cff9640 1 -- 172.21.15.125:0/3179199534 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd404005480 con 0x7fd4100c6fe0 2023-10-30T12:29:13.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.291+0000 7fd40cff9640 1 -- 172.21.15.125:0/3179199534 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd404003600 con 0x7fd4100c6fe0 2023-10-30T12:29:13.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.291+0000 7fd40cff9640 1 -- 172.21.15.125:0/3179199534 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4040039b0 con 0x7fd4100c6fe0 2023-10-30T12:29:13.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.291+0000 7fd416417640 1 -- 172.21.15.125:0/3179199534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd41014f120 con 0x7fd4100c6fe0 2023-10-30T12:29:13.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.291+0000 7fd416417640 1 -- 172.21.15.125:0/3179199534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4101ef4c0 con 0x7fd4100c6fe0 2023-10-30T12:29:13.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.291+0000 7fd416417640 1 -- 172.21.15.125:0/3179199534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd41004e170 con 0x7fd4100c6fe0 2023-10-30T12:29:13.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.291+0000 7fd40cff9640 1 -- 172.21.15.125:0/3179199534 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd404003b10 con 0x7fd4100c6fe0 2023-10-30T12:29:13.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.291+0000 7fd40cff9640 1 --2- 172.21.15.125:0/3179199534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3fc0664a0 0x7fd3fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.291+0000 7fd40cff9640 1 -- 172.21.15.125:0/3179199534 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd404083440 con 0x7fd4100c6fe0 2023-10-30T12:29:13.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.295+0000 7fd40ffff640 1 --2- 172.21.15.125:0/3179199534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3fc0664a0 0x7fd3fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.295+0000 7fd40ffff640 1 --2- 172.21.15.125:0/3179199534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3fc0664a0 0x7fd3fc068960 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd4000b2000 tx=0x7fd400009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.295+0000 7fd40cff9640 1 -- 172.21.15.125:0/3179199534 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd4040509a0 con 0x7fd4100c6fe0 2023-10-30T12:29:13.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.479+0000 7fd416417640 1 -- 172.21.15.125:0/3179199534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4101590f0 con 0x7fd4100c6fe0 2023-10-30T12:29:13.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.479+0000 7fd40cff9640 1 -- 172.21.15.125:0/3179199534 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6537 (secure 0 0 0) 0x7fd404017020 con 0x7fd4100c6fe0 2023-10-30T12:29:13.484 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:29:13.484 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:29:01.095107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.136:6835/3346501886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3346501886},{"type":"v1","addr":"172.21.15.136:6835","nonce":3346501886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:29:13.485 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 12 2023-10-30T12:29:13.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.483+0000 7fd416417640 1 -- 172.21.15.125:0/3179199534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3fc0664a0 msgr2=0x7fd3fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.483+0000 7fd416417640 1 --2- 172.21.15.125:0/3179199534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3fc0664a0 0x7fd3fc068960 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd4000b2000 tx=0x7fd400009040 comp rx=0 tx=0).stop 2023-10-30T12:29:13.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.483+0000 7fd416417640 1 -- 172.21.15.125:0/3179199534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4100c6fe0 msgr2=0x7fd41014e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.483+0000 7fd416417640 1 --2- 172.21.15.125:0/3179199534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4100c6fe0 0x7fd41014e050 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd40400c4d0 tx=0x7fd40400c990 comp rx=0 tx=0).stop 2023-10-30T12:29:13.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.483+0000 7fd416417640 1 -- 172.21.15.125:0/3179199534 shutdown_connections 2023-10-30T12:29:13.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.483+0000 7fd416417640 1 --2- 172.21.15.125:0/3179199534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3fc0664a0 0x7fd3fc068960 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.483+0000 7fd416417640 1 --2- 172.21.15.125:0/3179199534 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd410153bf0 0x7fd41014e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.483+0000 7fd416417640 1 --2- 172.21.15.125:0/3179199534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4100c6fe0 0x7fd41014e050 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.483+0000 7fd416417640 1 --2- 172.21.15.125:0/3179199534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4100c5f60 0x7fd4100ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.483+0000 7fd416417640 1 -- 172.21.15.125:0/3179199534 >> 172.21.15.125:0/3179199534 conn(0x7fd41006c6e0 msgr2=0x7fd410151ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.483+0000 7fd416417640 1 -- 172.21.15.125:0/3179199534 shutdown_connections 2023-10-30T12:29:13.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.483+0000 7fd416417640 1 -- 172.21.15.125:0/3179199534 wait complete. 2023-10-30T12:29:13.505 INFO:tasks.mds_thrash:Ready to start thrashing 2023-10-30T12:29:13.505 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2023-10-30T12:29:13.505 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:29:13.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d7a4d6640 1 Processor -- start 2023-10-30T12:29:13.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d7a4d6640 1 -- start start 2023-10-30T12:29:13.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d7a4d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d7415d6f0 0x7f7d7415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d7a4d6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7d740cf160 0x7f7d740cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d7a4d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d740cb800 con 0x7f7d741538a0 2023-10-30T12:29:13.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d7a4d6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d740cb330 con 0x7f7d740cf160 2023-10-30T12:29:13.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d7a4d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d740cad70 con 0x7f7d7415d6f0 2023-10-30T12:29:13.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d78a73640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7d740cf160 0x7f7d740cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d737fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d7415d6f0 0x7f7d7415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d78a73640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7d740cf160 0x7f7d740cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:60652/0 (socket says 172.21.15.125:60652) 2023-10-30T12:29:13.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d73fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f7d741538a0 0x7f7d740ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:51034/0 (socket says 172.21.15.125:51034) 2023-10-30T12:29:13.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d78a73640 1 -- 172.21.15.125:0/151120001 learned_addr learned my addr 172.21.15.125:0/151120001 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:13.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d78a73640 1 -- 172.21.15.125:0/151120001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d7415d6f0 msgr2=0x7f7d7415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:13.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d78a73640 1 --2- 172.21.15.125:0/151120001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d7415d6f0 0x7f7d7415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d78a73640 1 -- 172.21.15.125:0/151120001 >> v1:172.21.15.136:6789/0 conn(0x7f7d741538a0 legacy=0x7f7d740ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d78a73640 1 -- 172.21.15.125:0/151120001 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d740cb050 con 0x7f7d740cf160 2023-10-30T12:29:13.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.683+0000 7f7d72ffd640 1 -- 172.21.15.125:0/151120001 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305967826 0 0) 0x7f7d740cb800 con 0x7f7d741538a0 2023-10-30T12:29:13.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.687+0000 7f7d78a73640 1 --2- 172.21.15.125:0/151120001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d740cf160 0x7f7d740cf590 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c009e50 tx=0x7f7d6c01f1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f68a82a1a4e63635 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.687+0000 7f7d72ffd640 1 -- 172.21.15.125:0/151120001 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c0af400 con 0x7f7d740cf160 2023-10-30T12:29:13.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.687+0000 7f7d72ffd640 1 -- 172.21.15.125:0/151120001 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6c0ab050 con 0x7f7d740cf160 2023-10-30T12:29:13.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.687+0000 7f7d72ffd640 1 -- 172.21.15.125:0/151120001 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6c0af780 con 0x7f7d740cf160 2023-10-30T12:29:13.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.687+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/151120001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d740cf160 msgr2=0x7f7d740cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.687+0000 7f7d7a4d6640 1 --2- 172.21.15.125:0/151120001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d740cf160 0x7f7d740cf590 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7d6c009e50 tx=0x7f7d6c01f1f0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.687+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/151120001 shutdown_connections 2023-10-30T12:29:13.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.687+0000 7f7d7a4d6640 1 --2- 172.21.15.125:0/151120001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d740cf160 0x7f7d740cf590 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.687+0000 7f7d7a4d6640 1 --2- 172.21.15.125:0/151120001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d7415d6f0 0x7f7d7415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.687+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/151120001 >> 172.21.15.125:0/151120001 conn(0x7f7d7406bfe0 msgr2=0x7f7d7406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.687+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/151120001 shutdown_connections 2023-10-30T12:29:13.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.687+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/151120001 wait complete. 2023-10-30T12:29:13.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d7a4d6640 1 Processor -- start 2023-10-30T12:29:13.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d7a4d6640 1 -- start start 2023-10-30T12:29:13.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d7a4d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d740cf160 0x7f7d741e5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d7a4d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d741538a0 0x7f7d741e3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d7a4d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d7415d6f0 0x7f7d740c4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d7a4d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d741e55c0 con 0x7f7d741538a0 2023-10-30T12:29:13.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d7a4d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d740b3ff0 con 0x7f7d740cf160 2023-10-30T12:29:13.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d7a4d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d740b3700 con 0x7f7d7415d6f0 2023-10-30T12:29:13.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d737fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d741538a0 0x7f7d741e3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d737fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d741538a0 0x7f7d741e3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39980/0 (socket says 172.21.15.125:39980) 2023-10-30T12:29:13.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d737fe640 1 -- 172.21.15.125:0/2607377323 learned_addr learned my addr 172.21.15.125:0/2607377323 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:13.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d73fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d740cf160 0x7f7d741e5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d78a73640 1 --2- 172.21.15.125:0/2607377323 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d7415d6f0 0x7f7d740c4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d737fe640 1 -- 172.21.15.125:0/2607377323 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d7415d6f0 msgr2=0x7f7d740c4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d737fe640 1 --2- 172.21.15.125:0/2607377323 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d7415d6f0 0x7f7d740c4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d737fe640 1 -- 172.21.15.125:0/2607377323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d740cf160 msgr2=0x7f7d741e5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d737fe640 1 --2- 172.21.15.125:0/2607377323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d740cf160 0x7f7d741e5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d737fe640 1 -- 172.21.15.125:0/2607377323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c009b30 con 0x7f7d741538a0 2023-10-30T12:29:13.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d737fe640 1 --2- 172.21.15.125:0/2607377323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d741538a0 0x7f7d741e3eb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800e9b0 tx=0x7f7d6800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d70ff9640 1 -- 172.21.15.125:0/2607377323 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d680096b0 con 0x7f7d741538a0 2023-10-30T12:29:13.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d70ff9640 1 -- 172.21.15.125:0/2607377323 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6800c9a0 con 0x7f7d741538a0 2023-10-30T12:29:13.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/2607377323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d740b5100 con 0x7f7d741538a0 2023-10-30T12:29:13.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d70ff9640 1 -- 172.21.15.125:0/2607377323 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6800cd30 con 0x7f7d741538a0 2023-10-30T12:29:13.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.691+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/2607377323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d740b39a0 con 0x7f7d741538a0 2023-10-30T12:29:13.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.695+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/2607377323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d740b2a50 con 0x7f7d741538a0 2023-10-30T12:29:13.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.695+0000 7f7d70ff9640 1 -- 172.21.15.125:0/2607377323 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7d68013470 con 0x7f7d741538a0 2023-10-30T12:29:13.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.695+0000 7f7d70ff9640 1 --2- 172.21.15.125:0/2607377323 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d600664a0 0x7f7d60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:13.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.695+0000 7f7d70ff9640 1 -- 172.21.15.125:0/2607377323 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d68084200 con 0x7f7d741538a0 2023-10-30T12:29:13.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.695+0000 7f7d73fff640 1 --2- 172.21.15.125:0/2607377323 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d600664a0 0x7f7d60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:13.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.695+0000 7f7d73fff640 1 --2- 172.21.15.125:0/2607377323 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d600664a0 0x7f7d60068960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7d64007fd0 tx=0x7f7d64007d60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:13.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.699+0000 7f7d70ff9640 1 -- 172.21.15.125:0/2607377323 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7d68051600 con 0x7f7d741538a0 2023-10-30T12:29:13.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.847+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/2607377323 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d74159b70 con 0x7f7d600664a0 2023-10-30T12:29:13.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.867+0000 7f7d70ff9640 1 -- 172.21.15.125:0/2607377323 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264578 (secure 0 0 0) 0x7f7d64004c80 con 0x7f7d600664a0 2023-10-30T12:29:13.872 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:29:13.873 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T12:29:13.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.871+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/2607377323 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d600664a0 msgr2=0x7f7d60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.871+0000 7f7d7a4d6640 1 --2- 172.21.15.125:0/2607377323 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d600664a0 0x7f7d60068960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7d64007fd0 tx=0x7f7d64007d60 comp rx=0 tx=0).stop 2023-10-30T12:29:13.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.871+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/2607377323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d741538a0 msgr2=0x7f7d741e3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:13.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.871+0000 7f7d7a4d6640 1 --2- 172.21.15.125:0/2607377323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d741538a0 0x7f7d741e3eb0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7d6800e9b0 tx=0x7f7d6800ee70 comp rx=0 tx=0).stop 2023-10-30T12:29:13.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.871+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/2607377323 shutdown_connections 2023-10-30T12:29:13.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.871+0000 7f7d7a4d6640 1 --2- 172.21.15.125:0/2607377323 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d600664a0 0x7f7d60068960 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.871+0000 7f7d7a4d6640 1 --2- 172.21.15.125:0/2607377323 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d7415d6f0 0x7f7d740c4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.871+0000 7f7d7a4d6640 1 --2- 172.21.15.125:0/2607377323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d741538a0 0x7f7d741e3eb0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.871+0000 7f7d7a4d6640 1 --2- 172.21.15.125:0/2607377323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d740cf160 0x7f7d741e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:13.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.871+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/2607377323 >> 172.21.15.125:0/2607377323 conn(0x7f7d7406bfe0 msgr2=0x7f7d740ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:13.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.871+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/2607377323 shutdown_connections 2023-10-30T12:29:13.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:13.871+0000 7f7d7a4d6640 1 -- 172.21.15.125:0/2607377323 wait complete. 2023-10-30T12:29:13.892 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-10-30T12:29:12.970063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595590,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":284,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227464,"kb_used_data":3016,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754747256,"statfs":{"total":773094113280,"available":772861190144,"internally_reserved":0,"allocated":3088384,"data_stored":1799369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229756473},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1406,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004517"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104854+0000","last_change":"2023-10-30T12:28:44.957488+0000","last_active":"2023-10-30T12:28:46.104854+0000","last_peered":"2023-10-30T12:28:46.104854+0000","last_clean":"2023-10-30T12:28:46.104854+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T12:28:46.104854+0000","last_undegraded":"2023-10-30T12:28:46.104854+0000","last_fullsized":"2023-10-30T12:28:46.104854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:30.240483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106772+0000","last_change":"2023-10-30T12:28:44.978898+0000","last_active":"2023-10-30T12:28:46.106772+0000","last_peered":"2023-10-30T12:28:46.106772+0000","last_clean":"2023-10-30T12:28:46.106772+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T12:28:46.106772+0000","last_undegraded":"2023-10-30T12:28:46.106772+0000","last_fullsized":"2023-10-30T12:28:46.106772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:57:21.463798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106769+0000","last_change":"2023-10-30T12:28:44.980722+0000","last_active":"2023-10-30T12:28:46.106769+0000","last_peered":"2023-10-30T12:28:46.106769+0000","last_clean":"2023-10-30T12:28:46.106769+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T12:28:46.106769+0000","last_undegraded":"2023-10-30T12:28:46.106769+0000","last_fullsized":"2023-10-30T12:28:46.106769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:25.009946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103812+0000","last_change":"2023-10-30T12:28:44.965061+0000","last_active":"2023-10-30T12:28:46.103812+0000","last_peered":"2023-10-30T12:28:46.103812+0000","last_clean":"2023-10-30T12:28:46.103812+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T12:28:46.103812+0000","last_undegraded":"2023-10-30T12:28:46.103812+0000","last_fullsized":"2023-10-30T12:28:46.103812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:23:43.749064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104285+0000","last_change":"2023-10-30T12:28:44.976985+0000","last_active":"2023-10-30T12:28:46.104285+0000","last_peered":"2023-10-30T12:28:46.104285+0000","last_clean":"2023-10-30T12:28:46.104285+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T12:28:46.104285+0000","last_undegraded":"2023-10-30T12:28:46.104285+0000","last_fullsized":"2023-10-30T12:28:46.104285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:08.758140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103030+0000","last_change":"2023-10-30T12:28:44.965453+0000","last_active":"2023-10-30T12:28:46.103030+0000","last_peered":"2023-10-30T12:28:46.103030+0000","last_clean":"2023-10-30T12:28:46.103030+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T12:28:46.103030+0000","last_undegraded":"2023-10-30T12:28:46.103030+0000","last_fullsized":"2023-10-30T12:28:46.103030+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:19.975226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103434+0000","last_change":"2023-10-30T12:28:44.968493+0000","last_active":"2023-10-30T12:28:46.103434+0000","last_peered":"2023-10-30T12:28:46.103434+0000","last_clean":"2023-10-30T12:28:46.103434+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T12:28:46.103434+0000","last_undegraded":"2023-10-30T12:28:46.103434+0000","last_fullsized":"2023-10-30T12:28:46.103434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:58.895635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106402+0000","last_change":"2023-10-30T12:28:44.972695+0000","last_active":"2023-10-30T12:28:46.106402+0000","last_peered":"2023-10-30T12:28:46.106402+0000","last_clean":"2023-10-30T12:28:46.106402+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T12:28:46.106402+0000","last_undegraded":"2023-10-30T12:28:46.106402+0000","last_fullsized":"2023-10-30T12:28:46.106402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:45.353683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939624+0000","last_change":"2023-10-30T12:28:44.946156+0000","last_active":"2023-10-30T12:28:45.939624+0000","last_peered":"2023-10-30T12:28:45.939624+0000","last_clean":"2023-10-30T12:28:45.939624+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T12:28:45.939624+0000","last_undegraded":"2023-10-30T12:28:45.939624+0000","last_fullsized":"2023-10-30T12:28:45.939624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:42:36.408944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104638+0000","last_change":"2023-10-30T12:28:44.957225+0000","last_active":"2023-10-30T12:28:46.104638+0000","last_peered":"2023-10-30T12:28:46.104638+0000","last_clean":"2023-10-30T12:28:46.104638+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T12:28:46.104638+0000","last_undegraded":"2023-10-30T12:28:46.104638+0000","last_fullsized":"2023-10-30T12:28:46.104638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:55:20.194553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104526+0000","last_change":"2023-10-30T12:28:44.956025+0000","last_active":"2023-10-30T12:28:46.104526+0000","last_peered":"2023-10-30T12:28:46.104526+0000","last_clean":"2023-10-30T12:28:46.104526+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T12:28:46.104526+0000","last_undegraded":"2023-10-30T12:28:46.104526+0000","last_fullsized":"2023-10-30T12:28:46.104526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:12.113264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104464+0000","last_change":"2023-10-30T12:28:44.956832+0000","last_active":"2023-10-30T12:28:46.104464+0000","last_peered":"2023-10-30T12:28:46.104464+0000","last_clean":"2023-10-30T12:28:46.104464+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T12:28:46.104464+0000","last_undegraded":"2023-10-30T12:28:46.104464+0000","last_fullsized":"2023-10-30T12:28:46.104464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:36.932024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945824+0000","last_change":"2023-10-30T12:28:44.962961+0000","last_active":"2023-10-30T12:28:45.945824+0000","last_peered":"2023-10-30T12:28:45.945824+0000","last_clean":"2023-10-30T12:28:45.945824+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T12:28:45.945824+0000","last_undegraded":"2023-10-30T12:28:45.945824+0000","last_fullsized":"2023-10-30T12:28:45.945824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:13.200996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945413+0000","last_change":"2023-10-30T12:28:44.968005+0000","last_active":"2023-10-30T12:28:45.945413+0000","last_peered":"2023-10-30T12:28:45.945413+0000","last_clean":"2023-10-30T12:28:45.945413+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T12:28:45.945413+0000","last_undegraded":"2023-10-30T12:28:45.945413+0000","last_fullsized":"2023-10-30T12:28:45.945413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:53:25.168553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106306+0000","last_change":"2023-10-30T12:28:44.977064+0000","last_active":"2023-10-30T12:28:46.106306+0000","last_peered":"2023-10-30T12:28:46.106306+0000","last_clean":"2023-10-30T12:28:46.106306+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T12:28:46.106306+0000","last_undegraded":"2023-10-30T12:28:46.106306+0000","last_fullsized":"2023-10-30T12:28:46.106306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:27:44.043189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102782+0000","last_change":"2023-10-30T12:28:44.950169+0000","last_active":"2023-10-30T12:28:46.102782+0000","last_peered":"2023-10-30T12:28:46.102782+0000","last_clean":"2023-10-30T12:28:46.102782+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T12:28:46.102782+0000","last_undegraded":"2023-10-30T12:28:46.102782+0000","last_fullsized":"2023-10-30T12:28:46.102782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:35.297525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103374+0000","last_change":"2023-10-30T12:28:44.960467+0000","last_active":"2023-10-30T12:28:46.103374+0000","last_peered":"2023-10-30T12:28:46.103374+0000","last_clean":"2023-10-30T12:28:46.103374+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T12:28:46.103374+0000","last_undegraded":"2023-10-30T12:28:46.103374+0000","last_fullsized":"2023-10-30T12:28:46.103374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:33.495256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.108582+0000","last_change":"2023-10-30T12:28:44.968948+0000","last_active":"2023-10-30T12:28:46.108582+0000","last_peered":"2023-10-30T12:28:46.108582+0000","last_clean":"2023-10-30T12:28:46.108582+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T12:28:46.108582+0000","last_undegraded":"2023-10-30T12:28:46.108582+0000","last_fullsized":"2023-10-30T12:28:46.108582+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:26:14.661648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103539+0000","last_change":"2023-10-30T12:28:44.966743+0000","last_active":"2023-10-30T12:28:46.103539+0000","last_peered":"2023-10-30T12:28:46.103539+0000","last_clean":"2023-10-30T12:28:46.103539+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T12:28:46.103539+0000","last_undegraded":"2023-10-30T12:28:46.103539+0000","last_fullsized":"2023-10-30T12:28:46.103539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:30.077107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104499+0000","last_change":"2023-10-30T12:28:44.955323+0000","last_active":"2023-10-30T12:28:46.104499+0000","last_peered":"2023-10-30T12:28:46.104499+0000","last_clean":"2023-10-30T12:28:46.104499+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T12:28:46.104499+0000","last_undegraded":"2023-10-30T12:28:46.104499+0000","last_fullsized":"2023-10-30T12:28:46.104499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:59:22.424571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945425+0000","last_change":"2023-10-30T12:28:44.946048+0000","last_active":"2023-10-30T12:28:45.945425+0000","last_peered":"2023-10-30T12:28:45.945425+0000","last_clean":"2023-10-30T12:28:45.945425+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T12:28:45.945425+0000","last_undegraded":"2023-10-30T12:28:45.945425+0000","last_fullsized":"2023-10-30T12:28:45.945425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:20:21.145943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939444+0000","last_change":"2023-10-30T12:28:44.954167+0000","last_active":"2023-10-30T12:28:45.939444+0000","last_peered":"2023-10-30T12:28:45.939444+0000","last_clean":"2023-10-30T12:28:45.939444+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T12:28:45.939444+0000","last_undegraded":"2023-10-30T12:28:45.939444+0000","last_fullsized":"2023-10-30T12:28:45.939444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:08.457457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106494+0000","last_change":"2023-10-30T12:28:44.982724+0000","last_active":"2023-10-30T12:28:46.106494+0000","last_peered":"2023-10-30T12:28:46.106494+0000","last_clean":"2023-10-30T12:28:46.106494+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T12:28:46.106494+0000","last_undegraded":"2023-10-30T12:28:46.106494+0000","last_fullsized":"2023-10-30T12:28:46.106494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:13.873943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103197+0000","last_change":"2023-10-30T12:28:44.981034+0000","last_active":"2023-10-30T12:28:46.103197+0000","last_peered":"2023-10-30T12:28:46.103197+0000","last_clean":"2023-10-30T12:28:46.103197+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T12:28:46.103197+0000","last_undegraded":"2023-10-30T12:28:46.103197+0000","last_fullsized":"2023-10-30T12:28:46.103197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:39:25.263228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107211+0000","last_change":"2023-10-30T12:28:44.975010+0000","last_active":"2023-10-30T12:28:46.107211+0000","last_peered":"2023-10-30T12:28:46.107211+0000","last_clean":"2023-10-30T12:28:46.107211+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T12:28:46.107211+0000","last_undegraded":"2023-10-30T12:28:46.107211+0000","last_fullsized":"2023-10-30T12:28:46.107211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:13.768876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104243+0000","last_change":"2023-10-30T12:28:44.951099+0000","last_active":"2023-10-30T12:28:46.104243+0000","last_peered":"2023-10-30T12:28:46.104243+0000","last_clean":"2023-10-30T12:28:46.104243+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T12:28:46.104243+0000","last_undegraded":"2023-10-30T12:28:46.104243+0000","last_fullsized":"2023-10-30T12:28:46.104243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:21:27.902671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946065+0000","last_change":"2023-10-30T12:28:44.959915+0000","last_active":"2023-10-30T12:28:45.946065+0000","last_peered":"2023-10-30T12:28:45.946065+0000","last_clean":"2023-10-30T12:28:45.946065+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T12:28:45.946065+0000","last_undegraded":"2023-10-30T12:28:45.946065+0000","last_fullsized":"2023-10-30T12:28:45.946065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:57:49.711309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946417+0000","last_change":"2023-10-30T12:28:44.972677+0000","last_active":"2023-10-30T12:28:45.946417+0000","last_peered":"2023-10-30T12:28:45.946417+0000","last_clean":"2023-10-30T12:28:45.946417+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T12:28:45.946417+0000","last_undegraded":"2023-10-30T12:28:45.946417+0000","last_fullsized":"2023-10-30T12:28:45.946417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:12.072355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106685+0000","last_change":"2023-10-30T12:28:44.975973+0000","last_active":"2023-10-30T12:28:46.106685+0000","last_peered":"2023-10-30T12:28:46.106685+0000","last_clean":"2023-10-30T12:28:46.106685+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T12:28:46.106685+0000","last_undegraded":"2023-10-30T12:28:46.106685+0000","last_fullsized":"2023-10-30T12:28:46.106685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:48.152823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103531+0000","last_change":"2023-10-30T12:28:44.980814+0000","last_active":"2023-10-30T12:28:46.103531+0000","last_peered":"2023-10-30T12:28:46.103531+0000","last_clean":"2023-10-30T12:28:46.103531+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T12:28:46.103531+0000","last_undegraded":"2023-10-30T12:28:46.103531+0000","last_fullsized":"2023-10-30T12:28:46.103531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:51:22.453883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107798+0000","last_change":"2023-10-30T12:28:44.965026+0000","last_active":"2023-10-30T12:28:46.107798+0000","last_peered":"2023-10-30T12:28:46.107798+0000","last_clean":"2023-10-30T12:28:46.107798+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T12:28:46.107798+0000","last_undegraded":"2023-10-30T12:28:46.107798+0000","last_fullsized":"2023-10-30T12:28:46.107798+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:04:08.680744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102782+0000","last_change":"2023-10-30T12:28:44.973401+0000","last_active":"2023-10-30T12:28:46.102782+0000","last_peered":"2023-10-30T12:28:46.102782+0000","last_clean":"2023-10-30T12:28:46.102782+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T12:28:46.102782+0000","last_undegraded":"2023-10-30T12:28:46.102782+0000","last_fullsized":"2023-10-30T12:28:46.102782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:34.198424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939578+0000","last_change":"2023-10-30T12:28:44.971817+0000","last_active":"2023-10-30T12:28:45.939578+0000","last_peered":"2023-10-30T12:28:45.939578+0000","last_clean":"2023-10-30T12:28:45.939578+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T12:28:45.939578+0000","last_undegraded":"2023-10-30T12:28:45.939578+0000","last_fullsized":"2023-10-30T12:28:45.939578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:45:35.205305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.130160+0000","last_change":"2023-10-30T12:28:44.074842+0000","last_active":"2023-10-30T12:28:55.130160+0000","last_peered":"2023-10-30T12:28:55.130160+0000","last_clean":"2023-10-30T12:28:55.130160+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T12:28:55.130160+0000","last_undegraded":"2023-10-30T12:28:55.130160+0000","last_fullsized":"2023-10-30T12:28:55.130160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:01.428785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945765+0000","last_change":"2023-10-30T12:28:44.958560+0000","last_active":"2023-10-30T12:28:45.945765+0000","last_peered":"2023-10-30T12:28:45.945765+0000","last_clean":"2023-10-30T12:28:45.945765+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T12:28:45.945765+0000","last_undegraded":"2023-10-30T12:28:45.945765+0000","last_fullsized":"2023-10-30T12:28:45.945765+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:45.564493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.963498+0000","last_change":"2023-10-30T12:28:44.060621+0000","last_active":"2023-10-30T12:29:00.963498+0000","last_peered":"2023-10-30T12:29:00.963498+0000","last_clean":"2023-10-30T12:29:00.963498+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T12:29:00.963498+0000","last_undegraded":"2023-10-30T12:29:00.963498+0000","last_fullsized":"2023-10-30T12:29:00.963498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:53.741238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.967207+0000","last_change":"2023-10-30T12:28:44.060171+0000","last_active":"2023-10-30T12:29:00.967207+0000","last_peered":"2023-10-30T12:29:00.967207+0000","last_clean":"2023-10-30T12:29:00.967207+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T12:29:00.967207+0000","last_undegraded":"2023-10-30T12:29:00.967207+0000","last_fullsized":"2023-10-30T12:29:00.967207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:18.181418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106893+0000","last_change":"2023-10-30T12:28:44.978631+0000","last_active":"2023-10-30T12:28:46.106893+0000","last_peered":"2023-10-30T12:28:46.106893+0000","last_clean":"2023-10-30T12:28:46.106893+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T12:28:46.106893+0000","last_undegraded":"2023-10-30T12:28:46.106893+0000","last_fullsized":"2023-10-30T12:28:46.106893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:37.006427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:47.984674+0000","last_change":"2023-10-30T12:28:44.058798+0000","last_active":"2023-10-30T12:28:47.984674+0000","last_peered":"2023-10-30T12:28:47.984674+0000","last_clean":"2023-10-30T12:28:47.984674+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T12:28:47.984674+0000","last_undegraded":"2023-10-30T12:28:47.984674+0000","last_fullsized":"2023-10-30T12:28:47.984674+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:08.643080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945025+0000","last_change":"2023-10-30T12:28:44.949118+0000","last_active":"2023-10-30T12:28:45.945025+0000","last_peered":"2023-10-30T12:28:45.945025+0000","last_clean":"2023-10-30T12:28:45.945025+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T12:28:45.945025+0000","last_undegraded":"2023-10-30T12:28:45.945025+0000","last_fullsized":"2023-10-30T12:28:45.945025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:11.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.130828+0000","last_change":"2023-10-30T12:28:44.063439+0000","last_active":"2023-10-30T12:28:55.130828+0000","last_peered":"2023-10-30T12:28:55.130828+0000","last_clean":"2023-10-30T12:28:55.130828+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T12:28:55.130828+0000","last_undegraded":"2023-10-30T12:28:55.130828+0000","last_fullsized":"2023-10-30T12:28:55.130828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:08.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104375+0000","last_change":"2023-10-30T12:28:44.951319+0000","last_active":"2023-10-30T12:28:46.104375+0000","last_peered":"2023-10-30T12:28:46.104375+0000","last_clean":"2023-10-30T12:28:46.104375+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T12:28:46.104375+0000","last_undegraded":"2023-10-30T12:28:46.104375+0000","last_fullsized":"2023-10-30T12:28:46.104375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:20.108360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.134075+0000","last_change":"2023-10-30T12:28:44.052641+0000","last_active":"2023-10-30T12:28:55.134075+0000","last_peered":"2023-10-30T12:28:55.134075+0000","last_clean":"2023-10-30T12:28:55.134075+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T12:28:55.134075+0000","last_undegraded":"2023-10-30T12:28:55.134075+0000","last_fullsized":"2023-10-30T12:28:55.134075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107640+0000","last_change":"2023-10-30T12:28:44.981614+0000","last_active":"2023-10-30T12:28:46.107640+0000","last_peered":"2023-10-30T12:28:46.107640+0000","last_clean":"2023-10-30T12:28:46.107640+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T12:28:46.107640+0000","last_undegraded":"2023-10-30T12:28:46.107640+0000","last_fullsized":"2023-10-30T12:28:46.107640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:15:11.750243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.068083+0000","last_change":"2023-10-30T12:28:44.075783+0000","last_active":"2023-10-30T12:28:50.068083+0000","last_peered":"2023-10-30T12:28:50.068083+0000","last_clean":"2023-10-30T12:28:50.068083+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T12:28:50.068083+0000","last_undegraded":"2023-10-30T12:28:50.068083+0000","last_fullsized":"2023-10-30T12:28:50.068083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:01.094929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103142+0000","last_change":"2023-10-30T12:28:44.950899+0000","last_active":"2023-10-30T12:28:46.103142+0000","last_peered":"2023-10-30T12:28:46.103142+0000","last_clean":"2023-10-30T12:28:46.103142+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T12:28:46.103142+0000","last_undegraded":"2023-10-30T12:28:46.103142+0000","last_fullsized":"2023-10-30T12:28:46.103142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:34.005863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.132373+0000","last_change":"2023-10-30T12:28:44.063367+0000","last_active":"2023-10-30T12:28:55.132373+0000","last_peered":"2023-10-30T12:28:55.132373+0000","last_clean":"2023-10-30T12:28:55.132373+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T12:28:55.132373+0000","last_undegraded":"2023-10-30T12:28:55.132373+0000","last_fullsized":"2023-10-30T12:28:55.132373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:45.585121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104587+0000","last_change":"2023-10-30T12:28:44.972056+0000","last_active":"2023-10-30T12:28:46.104587+0000","last_peered":"2023-10-30T12:28:46.104587+0000","last_clean":"2023-10-30T12:28:46.104587+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T12:28:46.104587+0000","last_undegraded":"2023-10-30T12:28:46.104587+0000","last_fullsized":"2023-10-30T12:28:46.104587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:06:08.345856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.966521+0000","last_change":"2023-10-30T12:28:44.065545+0000","last_active":"2023-10-30T12:29:00.966521+0000","last_peered":"2023-10-30T12:29:00.966521+0000","last_clean":"2023-10-30T12:29:00.966521+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T12:29:00.966521+0000","last_undegraded":"2023-10-30T12:29:00.966521+0000","last_fullsized":"2023-10-30T12:29:00.966521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:09.900189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945563+0000","last_change":"2023-10-30T12:28:44.961237+0000","last_active":"2023-10-30T12:28:45.945563+0000","last_peered":"2023-10-30T12:28:45.945563+0000","last_clean":"2023-10-30T12:28:45.945563+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T12:28:45.945563+0000","last_undegraded":"2023-10-30T12:28:45.945563+0000","last_fullsized":"2023-10-30T12:28:45.945563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:17.160724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.130351+0000","last_change":"2023-10-30T12:28:44.061922+0000","last_active":"2023-10-30T12:28:55.130351+0000","last_peered":"2023-10-30T12:28:55.130351+0000","last_clean":"2023-10-30T12:28:55.130351+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T12:28:55.130351+0000","last_undegraded":"2023-10-30T12:28:55.130351+0000","last_fullsized":"2023-10-30T12:28:55.130351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:34.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103258+0000","last_change":"2023-10-30T12:28:44.958245+0000","last_active":"2023-10-30T12:28:46.103258+0000","last_peered":"2023-10-30T12:28:46.103258+0000","last_clean":"2023-10-30T12:28:46.103258+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T12:28:46.103258+0000","last_undegraded":"2023-10-30T12:28:46.103258+0000","last_fullsized":"2023-10-30T12:28:46.103258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:42:10.840490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.964145+0000","last_change":"2023-10-30T12:28:44.059784+0000","last_active":"2023-10-30T12:29:00.964145+0000","last_peered":"2023-10-30T12:29:00.964145+0000","last_clean":"2023-10-30T12:29:00.964145+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T12:29:00.964145+0000","last_undegraded":"2023-10-30T12:29:00.964145+0000","last_fullsized":"2023-10-30T12:29:00.964145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:11.468221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939230+0000","last_change":"2023-10-30T12:28:44.980813+0000","last_active":"2023-10-30T12:28:45.939230+0000","last_peered":"2023-10-30T12:28:45.939230+0000","last_clean":"2023-10-30T12:28:45.939230+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T12:28:45.939230+0000","last_undegraded":"2023-10-30T12:28:45.939230+0000","last_fullsized":"2023-10-30T12:28:45.939230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:10.858970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'15","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.975410+0000","last_change":"2023-10-30T12:28:44.065885+0000","last_active":"2023-10-30T12:29:00.975410+0000","last_peered":"2023-10-30T12:29:00.975410+0000","last_clean":"2023-10-30T12:29:00.975410+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T12:29:00.975410+0000","last_undegraded":"2023-10-30T12:29:00.975410+0000","last_fullsized":"2023-10-30T12:29:00.975410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:26.313595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103080+0000","last_change":"2023-10-30T12:28:44.952458+0000","last_active":"2023-10-30T12:28:46.103080+0000","last_peered":"2023-10-30T12:28:46.103080+0000","last_clean":"2023-10-30T12:28:46.103080+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T12:28:46.103080+0000","last_undegraded":"2023-10-30T12:28:46.103080+0000","last_fullsized":"2023-10-30T12:28:46.103080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:48:19.752492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.062786+0000","last_change":"2023-10-30T12:28:44.080011+0000","last_active":"2023-10-30T12:28:50.062786+0000","last_peered":"2023-10-30T12:28:50.062786+0000","last_clean":"2023-10-30T12:28:50.062786+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T12:28:50.062786+0000","last_undegraded":"2023-10-30T12:28:50.062786+0000","last_fullsized":"2023-10-30T12:28:50.062786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104494+0000","last_change":"2023-10-30T12:28:44.961296+0000","last_active":"2023-10-30T12:28:46.104494+0000","last_peered":"2023-10-30T12:28:46.104494+0000","last_clean":"2023-10-30T12:28:46.104494+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T12:28:46.104494+0000","last_undegraded":"2023-10-30T12:28:46.104494+0000","last_fullsized":"2023-10-30T12:28:46.104494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:56.540392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.066004+0000","last_change":"2023-10-30T12:28:44.076352+0000","last_active":"2023-10-30T12:28:50.066004+0000","last_peered":"2023-10-30T12:28:50.066004+0000","last_clean":"2023-10-30T12:28:50.066004+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T12:28:50.066004+0000","last_undegraded":"2023-10-30T12:28:50.066004+0000","last_fullsized":"2023-10-30T12:28:50.066004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104918+0000","last_change":"2023-10-30T12:28:44.956918+0000","last_active":"2023-10-30T12:28:46.104918+0000","last_peered":"2023-10-30T12:28:46.104918+0000","last_clean":"2023-10-30T12:28:46.104918+0000","last_became_active":"2023-10-30T12:28:44.956397+0000","last_became_peered":"2023-10-30T12:28:44.956397+0000","last_unstale":"2023-10-30T12:28:46.104918+0000","last_undegraded":"2023-10-30T12:28:46.104918+0000","last_fullsized":"2023-10-30T12:28:46.104918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:26.252196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106813+0000","last_change":"2023-10-30T12:28:44.075684+0000","last_active":"2023-10-30T12:28:46.106813+0000","last_peered":"2023-10-30T12:28:46.106813+0000","last_clean":"2023-10-30T12:28:46.106813+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T12:28:46.106813+0000","last_undegraded":"2023-10-30T12:28:46.106813+0000","last_fullsized":"2023-10-30T12:28:46.106813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:07.375795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104588+0000","last_change":"2023-10-30T12:28:44.956047+0000","last_active":"2023-10-30T12:28:46.104588+0000","last_peered":"2023-10-30T12:28:46.104588+0000","last_clean":"2023-10-30T12:28:46.104588+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T12:28:46.104588+0000","last_undegraded":"2023-10-30T12:28:46.104588+0000","last_fullsized":"2023-10-30T12:28:46.104588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:49.924947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.970394+0000","last_change":"2023-10-30T12:28:44.060686+0000","last_active":"2023-10-30T12:29:00.970394+0000","last_peered":"2023-10-30T12:29:00.970394+0000","last_clean":"2023-10-30T12:29:00.970394+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T12:29:00.970394+0000","last_undegraded":"2023-10-30T12:29:00.970394+0000","last_fullsized":"2023-10-30T12:29:00.970394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:23.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103562+0000","last_change":"2023-10-30T12:28:44.946758+0000","last_active":"2023-10-30T12:28:46.103562+0000","last_peered":"2023-10-30T12:28:46.103562+0000","last_clean":"2023-10-30T12:28:46.103562+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T12:28:46.103562+0000","last_undegraded":"2023-10-30T12:28:46.103562+0000","last_fullsized":"2023-10-30T12:28:46.103562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:20.520148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.932509+0000","last_change":"2023-10-30T12:28:33.433824+0000","last_active":"2023-10-30T12:28:44.932509+0000","last_peered":"2023-10-30T12:28:44.932509+0000","last_clean":"2023-10-30T12:28:44.932509+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T12:28:44.932509+0000","last_undegraded":"2023-10-30T12:28:44.932509+0000","last_fullsized":"2023-10-30T12:28:44.932509+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_clean_scrub_stamp":"2023-10-30T12:28:31.408255+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:17.624179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.132911+0000","last_change":"2023-10-30T12:28:44.059051+0000","last_active":"2023-10-30T12:28:55.132911+0000","last_peered":"2023-10-30T12:28:55.132911+0000","last_clean":"2023-10-30T12:28:55.132911+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T12:28:55.132911+0000","last_undegraded":"2023-10-30T12:28:55.132911+0000","last_fullsized":"2023-10-30T12:28:55.132911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:54.810636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945062+0000","last_change":"2023-10-30T12:28:44.965183+0000","last_active":"2023-10-30T12:28:45.945062+0000","last_peered":"2023-10-30T12:28:45.945062+0000","last_clean":"2023-10-30T12:28:45.945062+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T12:28:45.945062+0000","last_undegraded":"2023-10-30T12:28:45.945062+0000","last_fullsized":"2023-10-30T12:28:45.945062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:57.655687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.962924+0000","last_change":"2023-10-30T12:28:44.080378+0000","last_active":"2023-10-30T12:29:00.962924+0000","last_peered":"2023-10-30T12:29:00.962924+0000","last_clean":"2023-10-30T12:29:00.962924+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T12:29:00.962924+0000","last_undegraded":"2023-10-30T12:29:00.962924+0000","last_fullsized":"2023-10-30T12:29:00.962924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:22.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944927+0000","last_change":"2023-10-30T12:28:44.961659+0000","last_active":"2023-10-30T12:28:45.944927+0000","last_peered":"2023-10-30T12:28:45.944927+0000","last_clean":"2023-10-30T12:28:45.944927+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T12:28:45.944927+0000","last_undegraded":"2023-10-30T12:28:45.944927+0000","last_fullsized":"2023-10-30T12:28:45.944927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:34:00.000084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.969477+0000","last_change":"2023-10-30T12:28:44.082480+0000","last_active":"2023-10-30T12:29:00.969477+0000","last_peered":"2023-10-30T12:29:00.969477+0000","last_clean":"2023-10-30T12:29:00.969477+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T12:29:00.969477+0000","last_undegraded":"2023-10-30T12:29:00.969477+0000","last_fullsized":"2023-10-30T12:29:00.969477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:20.126782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106385+0000","last_change":"2023-10-30T12:28:44.973906+0000","last_active":"2023-10-30T12:28:46.106385+0000","last_peered":"2023-10-30T12:28:46.106385+0000","last_clean":"2023-10-30T12:28:46.106385+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T12:28:46.106385+0000","last_undegraded":"2023-10-30T12:28:46.106385+0000","last_fullsized":"2023-10-30T12:28:46.106385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:31.845894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:47.984766+0000","last_change":"2023-10-30T12:28:44.059806+0000","last_active":"2023-10-30T12:28:47.984766+0000","last_peered":"2023-10-30T12:28:47.984766+0000","last_clean":"2023-10-30T12:28:47.984766+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T12:28:47.984766+0000","last_undegraded":"2023-10-30T12:28:47.984766+0000","last_fullsized":"2023-10-30T12:28:47.984766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:50.939946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104302+0000","last_change":"2023-10-30T12:28:44.955423+0000","last_active":"2023-10-30T12:28:46.104302+0000","last_peered":"2023-10-30T12:28:46.104302+0000","last_clean":"2023-10-30T12:28:46.104302+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T12:28:46.104302+0000","last_undegraded":"2023-10-30T12:28:46.104302+0000","last_fullsized":"2023-10-30T12:28:46.104302+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:30.054048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.982937+0000","last_change":"2023-10-30T12:28:44.074970+0000","last_active":"2023-10-30T12:28:45.982937+0000","last_peered":"2023-10-30T12:28:45.982937+0000","last_clean":"2023-10-30T12:28:45.982937+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T12:28:45.982937+0000","last_undegraded":"2023-10-30T12:28:45.982937+0000","last_fullsized":"2023-10-30T12:28:45.982937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:33.562703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103885+0000","last_change":"2023-10-30T12:28:44.973482+0000","last_active":"2023-10-30T12:28:46.103885+0000","last_peered":"2023-10-30T12:28:46.103885+0000","last_clean":"2023-10-30T12:28:46.103885+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T12:28:46.103885+0000","last_undegraded":"2023-10-30T12:28:46.103885+0000","last_fullsized":"2023-10-30T12:28:46.103885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:25.393161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:47.984860+0000","last_change":"2023-10-30T12:28:44.063256+0000","last_active":"2023-10-30T12:28:47.984860+0000","last_peered":"2023-10-30T12:28:47.984860+0000","last_clean":"2023-10-30T12:28:47.984860+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T12:28:47.984860+0000","last_undegraded":"2023-10-30T12:28:47.984860+0000","last_fullsized":"2023-10-30T12:28:47.984860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:01:36.144966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946945+0000","last_change":"2023-10-30T12:28:44.975443+0000","last_active":"2023-10-30T12:28:45.946945+0000","last_peered":"2023-10-30T12:28:45.946945+0000","last_clean":"2023-10-30T12:28:45.946945+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T12:28:45.946945+0000","last_undegraded":"2023-10-30T12:28:45.946945+0000","last_fullsized":"2023-10-30T12:28:45.946945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:23.765723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.968669+0000","last_change":"2023-10-30T12:28:44.055374+0000","last_active":"2023-10-30T12:29:00.968669+0000","last_peered":"2023-10-30T12:29:00.968669+0000","last_clean":"2023-10-30T12:29:00.968669+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T12:29:00.968669+0000","last_undegraded":"2023-10-30T12:29:00.968669+0000","last_fullsized":"2023-10-30T12:29:00.968669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946991+0000","last_change":"2023-10-30T12:28:44.974359+0000","last_active":"2023-10-30T12:28:45.946991+0000","last_peered":"2023-10-30T12:28:45.946991+0000","last_clean":"2023-10-30T12:28:45.946991+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T12:28:45.946991+0000","last_undegraded":"2023-10-30T12:28:45.946991+0000","last_fullsized":"2023-10-30T12:28:45.946991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:48.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.132893+0000","last_change":"2023-10-30T12:28:44.058663+0000","last_active":"2023-10-30T12:28:55.132893+0000","last_peered":"2023-10-30T12:28:55.132893+0000","last_clean":"2023-10-30T12:28:55.132893+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T12:28:55.132893+0000","last_undegraded":"2023-10-30T12:28:55.132893+0000","last_fullsized":"2023-10-30T12:28:55.132893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:42:22.279653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946815+0000","last_change":"2023-10-30T12:28:44.957471+0000","last_active":"2023-10-30T12:28:45.946815+0000","last_peered":"2023-10-30T12:28:45.946815+0000","last_clean":"2023-10-30T12:28:45.946815+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T12:28:45.946815+0000","last_undegraded":"2023-10-30T12:28:45.946815+0000","last_fullsized":"2023-10-30T12:28:45.946815+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:27:38.669452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.131577+0000","last_change":"2023-10-30T12:28:44.054338+0000","last_active":"2023-10-30T12:28:55.131577+0000","last_peered":"2023-10-30T12:28:55.131577+0000","last_clean":"2023-10-30T12:28:55.131577+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T12:28:55.131577+0000","last_undegraded":"2023-10-30T12:28:55.131577+0000","last_fullsized":"2023-10-30T12:28:55.131577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:55:35.370652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103200+0000","last_change":"2023-10-30T12:28:44.965759+0000","last_active":"2023-10-30T12:28:46.103200+0000","last_peered":"2023-10-30T12:28:46.103200+0000","last_clean":"2023-10-30T12:28:46.103200+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T12:28:46.103200+0000","last_undegraded":"2023-10-30T12:28:46.103200+0000","last_fullsized":"2023-10-30T12:28:46.103200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:13:52.965500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.134624+0000","last_change":"2023-10-30T12:28:44.064606+0000","last_active":"2023-10-30T12:28:55.134624+0000","last_peered":"2023-10-30T12:28:55.134624+0000","last_clean":"2023-10-30T12:28:55.134624+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T12:28:55.134624+0000","last_undegraded":"2023-10-30T12:28:55.134624+0000","last_fullsized":"2023-10-30T12:28:55.134624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944678+0000","last_change":"2023-10-30T12:28:44.958947+0000","last_active":"2023-10-30T12:28:45.944678+0000","last_peered":"2023-10-30T12:28:45.944678+0000","last_clean":"2023-10-30T12:28:45.944678+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T12:28:45.944678+0000","last_undegraded":"2023-10-30T12:28:45.944678+0000","last_fullsized":"2023-10-30T12:28:45.944678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:01:16.329994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.979140+0000","last_change":"2023-10-30T12:28:44.072754+0000","last_active":"2023-10-30T12:28:46.979140+0000","last_peered":"2023-10-30T12:28:46.979140+0000","last_clean":"2023-10-30T12:28:46.979140+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T12:28:46.979140+0000","last_undegraded":"2023-10-30T12:28:46.979140+0000","last_fullsized":"2023-10-30T12:28:46.979140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:51:55.865069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.108063+0000","last_change":"2023-10-30T12:28:44.978376+0000","last_active":"2023-10-30T12:28:46.108063+0000","last_peered":"2023-10-30T12:28:46.108063+0000","last_clean":"2023-10-30T12:28:46.108063+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T12:28:46.108063+0000","last_undegraded":"2023-10-30T12:28:46.108063+0000","last_fullsized":"2023-10-30T12:28:46.108063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:59.643447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.131363+0000","last_change":"2023-10-30T12:28:44.058801+0000","last_active":"2023-10-30T12:28:55.131363+0000","last_peered":"2023-10-30T12:28:55.131363+0000","last_clean":"2023-10-30T12:28:55.131363+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T12:28:55.131363+0000","last_undegraded":"2023-10-30T12:28:55.131363+0000","last_fullsized":"2023-10-30T12:28:55.131363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:11:24.706813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103647+0000","last_change":"2023-10-30T12:28:44.943292+0000","last_active":"2023-10-30T12:28:46.103647+0000","last_peered":"2023-10-30T12:28:46.103647+0000","last_clean":"2023-10-30T12:28:46.103647+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T12:28:46.103647+0000","last_undegraded":"2023-10-30T12:28:46.103647+0000","last_fullsized":"2023-10-30T12:28:46.103647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:37.538525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103472+0000","last_change":"2023-10-30T12:28:44.958607+0000","last_active":"2023-10-30T12:28:46.103472+0000","last_peered":"2023-10-30T12:28:46.103472+0000","last_clean":"2023-10-30T12:28:46.103472+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T12:28:46.103472+0000","last_undegraded":"2023-10-30T12:28:46.103472+0000","last_fullsized":"2023-10-30T12:28:46.103472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:28:55.169198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.059856+0000","last_change":"2023-10-30T12:28:44.054652+0000","last_active":"2023-10-30T12:28:50.059856+0000","last_peered":"2023-10-30T12:28:50.059856+0000","last_clean":"2023-10-30T12:28:50.059856+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T12:28:50.059856+0000","last_undegraded":"2023-10-30T12:28:50.059856+0000","last_fullsized":"2023-10-30T12:28:50.059856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:05.023374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939383+0000","last_change":"2023-10-30T12:28:44.958246+0000","last_active":"2023-10-30T12:28:45.939383+0000","last_peered":"2023-10-30T12:28:45.939383+0000","last_clean":"2023-10-30T12:28:45.939383+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T12:28:45.939383+0000","last_undegraded":"2023-10-30T12:28:45.939383+0000","last_fullsized":"2023-10-30T12:28:45.939383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:36.947398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.963915+0000","last_change":"2023-10-30T12:28:44.052633+0000","last_active":"2023-10-30T12:29:00.963915+0000","last_peered":"2023-10-30T12:29:00.963915+0000","last_clean":"2023-10-30T12:29:00.963915+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T12:29:00.963915+0000","last_undegraded":"2023-10-30T12:29:00.963915+0000","last_fullsized":"2023-10-30T12:29:00.963915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107324+0000","last_change":"2023-10-30T12:28:44.974457+0000","last_active":"2023-10-30T12:28:46.107324+0000","last_peered":"2023-10-30T12:28:46.107324+0000","last_clean":"2023-10-30T12:28:46.107324+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T12:28:46.107324+0000","last_undegraded":"2023-10-30T12:28:46.107324+0000","last_fullsized":"2023-10-30T12:28:46.107324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:25:56.791053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.064135+0000","last_change":"2023-10-30T12:28:44.075493+0000","last_active":"2023-10-30T12:28:50.064135+0000","last_peered":"2023-10-30T12:28:50.064135+0000","last_clean":"2023-10-30T12:28:50.064135+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T12:28:50.064135+0000","last_undegraded":"2023-10-30T12:28:50.064135+0000","last_fullsized":"2023-10-30T12:28:50.064135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:00:11.718669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107054+0000","last_change":"2023-10-30T12:28:44.974424+0000","last_active":"2023-10-30T12:28:46.107054+0000","last_peered":"2023-10-30T12:28:46.107054+0000","last_clean":"2023-10-30T12:28:46.107054+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T12:28:46.107054+0000","last_undegraded":"2023-10-30T12:28:46.107054+0000","last_fullsized":"2023-10-30T12:28:46.107054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:14.375671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'9","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.971551+0000","last_change":"2023-10-30T12:28:44.065843+0000","last_active":"2023-10-30T12:29:00.971551+0000","last_peered":"2023-10-30T12:29:00.971551+0000","last_clean":"2023-10-30T12:29:00.971551+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T12:29:00.971551+0000","last_undegraded":"2023-10-30T12:29:00.971551+0000","last_fullsized":"2023-10-30T12:29:00.971551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:32.405199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5222,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476748,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":77094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]}]},{"osd":2,"up_from":16,"seq":68719476748,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":77514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]}]},{"osd":1,"up_from":16,"seq":68719476746,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":819200,"data_stored":666396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]}]},{"osd":0,"up_from":16,"seq":68719476747,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":77184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509452,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":77072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0669999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509453,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31432,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340408,"statfs":{"total":96636764160,"available":96604577792,"internally_reserved":0,"allocated":262144,"data_stored":80293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]}]},{"osd":7,"up_from":14,"seq":60129542155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":76776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542154,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":667040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":4425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:29:13.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:29:14.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc42e994640 1 Processor -- start 2023-10-30T12:29:14.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc42e994640 1 -- start start 2023-10-30T12:29:14.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc42e994640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4281538a0 0x7fc42815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc42e994640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4280cf160 0x7fc4280cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc42e994640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4280cb800 con 0x7fc4280ca0e0 2023-10-30T12:29:14.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc42e994640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc42815dc30 con 0x7fc4281538a0 2023-10-30T12:29:14.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc42e994640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4280cb330 con 0x7fc4280cf160 2023-10-30T12:29:14.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc427fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4281538a0 0x7fc42815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc427fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4281538a0 0x7fc42815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39986/0 (socket says 172.21.15.125:39986) 2023-10-30T12:29:14.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc427fff640 1 -- 172.21.15.125:0/3060330188 learned_addr learned my addr 172.21.15.125:0/3060330188 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:14.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc42cf31640 1 --2- 172.21.15.125:0/3060330188 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4280cf160 0x7fc4280cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc427fff640 1 -- 172.21.15.125:0/3060330188 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4280cf160 msgr2=0x7fc4280cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:14.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc427fff640 1 --2- 172.21.15.125:0/3060330188 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4280cf160 0x7fc4280cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc427fff640 1 -- 172.21.15.125:0/3060330188 >> v1:172.21.15.136:6789/0 conn(0x7fc4280ca0e0 legacy=0x7fc42815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.075+0000 7fc427fff640 1 -- 172.21.15.125:0/3060330188 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4280cad70 con 0x7fc4281538a0 2023-10-30T12:29:14.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.079+0000 7fc427fff640 1 --2- 172.21.15.125:0/3060330188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4281538a0 0x7fc42815d6f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc418009870 tx=0x7fc41801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669f8840ccd815a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:14.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.079+0000 7fc426ffd640 1 -- 172.21.15.125:0/3060330188 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4180a8a30 con 0x7fc4281538a0 2023-10-30T12:29:14.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.079+0000 7fc426ffd640 1 -- 172.21.15.125:0/3060330188 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4180a8b90 con 0x7fc4281538a0 2023-10-30T12:29:14.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.079+0000 7fc426ffd640 1 -- 172.21.15.125:0/3060330188 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4180b0440 con 0x7fc4281538a0 2023-10-30T12:29:14.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.079+0000 7fc42e994640 1 -- 172.21.15.125:0/3060330188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4281538a0 msgr2=0x7fc42815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.079+0000 7fc42e994640 1 --2- 172.21.15.125:0/3060330188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4281538a0 0x7fc42815d6f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc418009870 tx=0x7fc41801ed30 comp rx=0 tx=0).stop 2023-10-30T12:29:14.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.079+0000 7fc42e994640 1 -- 172.21.15.125:0/3060330188 shutdown_connections 2023-10-30T12:29:14.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.079+0000 7fc42e994640 1 --2- 172.21.15.125:0/3060330188 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4280cf160 0x7fc4280cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.079+0000 7fc42e994640 1 --2- 172.21.15.125:0/3060330188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4281538a0 0x7fc42815d6f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.079+0000 7fc42e994640 1 -- 172.21.15.125:0/3060330188 >> 172.21.15.125:0/3060330188 conn(0x7fc42806bfe0 msgr2=0x7fc42806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:14.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.079+0000 7fc42e994640 1 -- 172.21.15.125:0/3060330188 shutdown_connections 2023-10-30T12:29:14.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.079+0000 7fc42e994640 1 -- 172.21.15.125:0/3060330188 wait complete. 2023-10-30T12:29:14.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc42e994640 1 Processor -- start 2023-10-30T12:29:14.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc42e994640 1 -- start start 2023-10-30T12:29:14.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc42e994640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4280ca0e0 0x7fc4281e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc42e994640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4280cf160 0x7fc42815b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc42e994640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4281538a0 0x7fc42815bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc42e994640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc42814d4f0 con 0x7fc4280ca0e0 2023-10-30T12:29:14.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc42e994640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4281e4d40 con 0x7fc4281538a0 2023-10-30T12:29:14.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc42e994640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc42815b2a0 con 0x7fc4280cf160 2023-10-30T12:29:14.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc427fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4280ca0e0 0x7fc4281e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc427fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4280ca0e0 0x7fc4281e4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39990/0 (socket says 172.21.15.125:39990) 2023-10-30T12:29:14.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc427fff640 1 -- 172.21.15.125:0/2803644413 learned_addr learned my addr 172.21.15.125:0/2803644413 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:14.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc4277fe640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4280cf160 0x7fc42815b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc42cf31640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4281538a0 0x7fc42815bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc427fff640 1 -- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4280cf160 msgr2=0x7fc42815b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc427fff640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4280cf160 0x7fc42815b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc427fff640 1 -- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4281538a0 msgr2=0x7fc42815bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc427fff640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4281538a0 0x7fc42815bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc427fff640 1 -- 172.21.15.125:0/2803644413 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4180094d0 con 0x7fc4280ca0e0 2023-10-30T12:29:14.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc4277fe640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4280cf160 0x7fc42815b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:14.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc42cf31640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4281538a0 0x7fc42815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:29:14.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc427fff640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4280ca0e0 0x7fc4281e4090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc4180b3040 tx=0x7fc41801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:14.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc424ff9640 1 -- 172.21.15.125:0/2803644413 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4180ba600 con 0x7fc4280ca0e0 2023-10-30T12:29:14.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc424ff9640 1 -- 172.21.15.125:0/2803644413 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4180ba780 con 0x7fc4280ca0e0 2023-10-30T12:29:14.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc42e994640 1 -- 172.21.15.125:0/2803644413 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4280cef10 con 0x7fc4280ca0e0 2023-10-30T12:29:14.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc424ff9640 1 -- 172.21.15.125:0/2803644413 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4180bab30 con 0x7fc4280ca0e0 2023-10-30T12:29:14.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc42e994640 1 -- 172.21.15.125:0/2803644413 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4280c50b0 con 0x7fc4280ca0e0 2023-10-30T12:29:14.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.083+0000 7fc42e994640 1 -- 172.21.15.125:0/2803644413 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4280b2a50 con 0x7fc4280ca0e0 2023-10-30T12:29:14.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.087+0000 7fc424ff9640 1 -- 172.21.15.125:0/2803644413 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc4180bac90 con 0x7fc4280ca0e0 2023-10-30T12:29:14.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.087+0000 7fc424ff9640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4140664a0 0x7fc414068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.087+0000 7fc424ff9640 1 -- 172.21.15.125:0/2803644413 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc418122400 con 0x7fc4280ca0e0 2023-10-30T12:29:14.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.087+0000 7fc4277fe640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4140664a0 0x7fc414068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.087+0000 7fc4277fe640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4140664a0 0x7fc414068960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc4280c4660 tx=0x7fc41c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:14.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.091+0000 7fc424ff9640 1 -- 172.21.15.125:0/2803644413 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc4180ef7e0 con 0x7fc4280ca0e0 2023-10-30T12:29:14.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.239+0000 7fc42e994640 1 -- 172.21.15.125:0/2803644413 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc4280cb050 con 0x7fc4140664a0 2023-10-30T12:29:14.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.259+0000 7fc424ff9640 1 -- 172.21.15.125:0/2803644413 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264578 (secure 0 0 0) 0x7fc41c001cc0 con 0x7fc4140664a0 2023-10-30T12:29:14.263 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:29:14.265 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T12:29:14.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.263+0000 7fc42e994640 1 -- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4140664a0 msgr2=0x7fc414068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.263+0000 7fc42e994640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4140664a0 0x7fc414068960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc4280c4660 tx=0x7fc41c009040 comp rx=0 tx=0).stop 2023-10-30T12:29:14.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.263+0000 7fc42e994640 1 -- 172.21.15.125:0/2803644413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4280ca0e0 msgr2=0x7fc4281e4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.263+0000 7fc42e994640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4280ca0e0 0x7fc4281e4090 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fc4180b3040 tx=0x7fc41801bad0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.263+0000 7fc42e994640 1 -- 172.21.15.125:0/2803644413 shutdown_connections 2023-10-30T12:29:14.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.263+0000 7fc42e994640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4140664a0 0x7fc414068960 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.263+0000 7fc42e994640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4281538a0 0x7fc42815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.263+0000 7fc42e994640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4280cf160 0x7fc42815b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.263+0000 7fc42e994640 1 --2- 172.21.15.125:0/2803644413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4280ca0e0 0x7fc4281e4090 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.263+0000 7fc42e994640 1 -- 172.21.15.125:0/2803644413 >> 172.21.15.125:0/2803644413 conn(0x7fc42806bfe0 msgr2=0x7fc428065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:14.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.263+0000 7fc42e994640 1 -- 172.21.15.125:0/2803644413 shutdown_connections 2023-10-30T12:29:14.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.263+0000 7fc42e994640 1 -- 172.21.15.125:0/2803644413 wait complete. 2023-10-30T12:29:14.284 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-10-30T12:29:12.970063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595590,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":284,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227464,"kb_used_data":3016,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754747256,"statfs":{"total":773094113280,"available":772861190144,"internally_reserved":0,"allocated":3088384,"data_stored":1799369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229756473},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1406,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004517"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104854+0000","last_change":"2023-10-30T12:28:44.957488+0000","last_active":"2023-10-30T12:28:46.104854+0000","last_peered":"2023-10-30T12:28:46.104854+0000","last_clean":"2023-10-30T12:28:46.104854+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T12:28:46.104854+0000","last_undegraded":"2023-10-30T12:28:46.104854+0000","last_fullsized":"2023-10-30T12:28:46.104854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:30.240483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106772+0000","last_change":"2023-10-30T12:28:44.978898+0000","last_active":"2023-10-30T12:28:46.106772+0000","last_peered":"2023-10-30T12:28:46.106772+0000","last_clean":"2023-10-30T12:28:46.106772+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T12:28:46.106772+0000","last_undegraded":"2023-10-30T12:28:46.106772+0000","last_fullsized":"2023-10-30T12:28:46.106772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:57:21.463798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106769+0000","last_change":"2023-10-30T12:28:44.980722+0000","last_active":"2023-10-30T12:28:46.106769+0000","last_peered":"2023-10-30T12:28:46.106769+0000","last_clean":"2023-10-30T12:28:46.106769+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T12:28:46.106769+0000","last_undegraded":"2023-10-30T12:28:46.106769+0000","last_fullsized":"2023-10-30T12:28:46.106769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:25.009946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103812+0000","last_change":"2023-10-30T12:28:44.965061+0000","last_active":"2023-10-30T12:28:46.103812+0000","last_peered":"2023-10-30T12:28:46.103812+0000","last_clean":"2023-10-30T12:28:46.103812+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T12:28:46.103812+0000","last_undegraded":"2023-10-30T12:28:46.103812+0000","last_fullsized":"2023-10-30T12:28:46.103812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:23:43.749064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104285+0000","last_change":"2023-10-30T12:28:44.976985+0000","last_active":"2023-10-30T12:28:46.104285+0000","last_peered":"2023-10-30T12:28:46.104285+0000","last_clean":"2023-10-30T12:28:46.104285+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T12:28:46.104285+0000","last_undegraded":"2023-10-30T12:28:46.104285+0000","last_fullsized":"2023-10-30T12:28:46.104285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:08.758140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103030+0000","last_change":"2023-10-30T12:28:44.965453+0000","last_active":"2023-10-30T12:28:46.103030+0000","last_peered":"2023-10-30T12:28:46.103030+0000","last_clean":"2023-10-30T12:28:46.103030+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T12:28:46.103030+0000","last_undegraded":"2023-10-30T12:28:46.103030+0000","last_fullsized":"2023-10-30T12:28:46.103030+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:19.975226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103434+0000","last_change":"2023-10-30T12:28:44.968493+0000","last_active":"2023-10-30T12:28:46.103434+0000","last_peered":"2023-10-30T12:28:46.103434+0000","last_clean":"2023-10-30T12:28:46.103434+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T12:28:46.103434+0000","last_undegraded":"2023-10-30T12:28:46.103434+0000","last_fullsized":"2023-10-30T12:28:46.103434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:58.895635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106402+0000","last_change":"2023-10-30T12:28:44.972695+0000","last_active":"2023-10-30T12:28:46.106402+0000","last_peered":"2023-10-30T12:28:46.106402+0000","last_clean":"2023-10-30T12:28:46.106402+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T12:28:46.106402+0000","last_undegraded":"2023-10-30T12:28:46.106402+0000","last_fullsized":"2023-10-30T12:28:46.106402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:45.353683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939624+0000","last_change":"2023-10-30T12:28:44.946156+0000","last_active":"2023-10-30T12:28:45.939624+0000","last_peered":"2023-10-30T12:28:45.939624+0000","last_clean":"2023-10-30T12:28:45.939624+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T12:28:45.939624+0000","last_undegraded":"2023-10-30T12:28:45.939624+0000","last_fullsized":"2023-10-30T12:28:45.939624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:42:36.408944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104638+0000","last_change":"2023-10-30T12:28:44.957225+0000","last_active":"2023-10-30T12:28:46.104638+0000","last_peered":"2023-10-30T12:28:46.104638+0000","last_clean":"2023-10-30T12:28:46.104638+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T12:28:46.104638+0000","last_undegraded":"2023-10-30T12:28:46.104638+0000","last_fullsized":"2023-10-30T12:28:46.104638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:55:20.194553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104526+0000","last_change":"2023-10-30T12:28:44.956025+0000","last_active":"2023-10-30T12:28:46.104526+0000","last_peered":"2023-10-30T12:28:46.104526+0000","last_clean":"2023-10-30T12:28:46.104526+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T12:28:46.104526+0000","last_undegraded":"2023-10-30T12:28:46.104526+0000","last_fullsized":"2023-10-30T12:28:46.104526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:12.113264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104464+0000","last_change":"2023-10-30T12:28:44.956832+0000","last_active":"2023-10-30T12:28:46.104464+0000","last_peered":"2023-10-30T12:28:46.104464+0000","last_clean":"2023-10-30T12:28:46.104464+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T12:28:46.104464+0000","last_undegraded":"2023-10-30T12:28:46.104464+0000","last_fullsized":"2023-10-30T12:28:46.104464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:36.932024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945824+0000","last_change":"2023-10-30T12:28:44.962961+0000","last_active":"2023-10-30T12:28:45.945824+0000","last_peered":"2023-10-30T12:28:45.945824+0000","last_clean":"2023-10-30T12:28:45.945824+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T12:28:45.945824+0000","last_undegraded":"2023-10-30T12:28:45.945824+0000","last_fullsized":"2023-10-30T12:28:45.945824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:13.200996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945413+0000","last_change":"2023-10-30T12:28:44.968005+0000","last_active":"2023-10-30T12:28:45.945413+0000","last_peered":"2023-10-30T12:28:45.945413+0000","last_clean":"2023-10-30T12:28:45.945413+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T12:28:45.945413+0000","last_undegraded":"2023-10-30T12:28:45.945413+0000","last_fullsized":"2023-10-30T12:28:45.945413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:53:25.168553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106306+0000","last_change":"2023-10-30T12:28:44.977064+0000","last_active":"2023-10-30T12:28:46.106306+0000","last_peered":"2023-10-30T12:28:46.106306+0000","last_clean":"2023-10-30T12:28:46.106306+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T12:28:46.106306+0000","last_undegraded":"2023-10-30T12:28:46.106306+0000","last_fullsized":"2023-10-30T12:28:46.106306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:27:44.043189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102782+0000","last_change":"2023-10-30T12:28:44.950169+0000","last_active":"2023-10-30T12:28:46.102782+0000","last_peered":"2023-10-30T12:28:46.102782+0000","last_clean":"2023-10-30T12:28:46.102782+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T12:28:46.102782+0000","last_undegraded":"2023-10-30T12:28:46.102782+0000","last_fullsized":"2023-10-30T12:28:46.102782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:35.297525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103374+0000","last_change":"2023-10-30T12:28:44.960467+0000","last_active":"2023-10-30T12:28:46.103374+0000","last_peered":"2023-10-30T12:28:46.103374+0000","last_clean":"2023-10-30T12:28:46.103374+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T12:28:46.103374+0000","last_undegraded":"2023-10-30T12:28:46.103374+0000","last_fullsized":"2023-10-30T12:28:46.103374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:33.495256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.108582+0000","last_change":"2023-10-30T12:28:44.968948+0000","last_active":"2023-10-30T12:28:46.108582+0000","last_peered":"2023-10-30T12:28:46.108582+0000","last_clean":"2023-10-30T12:28:46.108582+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T12:28:46.108582+0000","last_undegraded":"2023-10-30T12:28:46.108582+0000","last_fullsized":"2023-10-30T12:28:46.108582+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:26:14.661648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103539+0000","last_change":"2023-10-30T12:28:44.966743+0000","last_active":"2023-10-30T12:28:46.103539+0000","last_peered":"2023-10-30T12:28:46.103539+0000","last_clean":"2023-10-30T12:28:46.103539+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T12:28:46.103539+0000","last_undegraded":"2023-10-30T12:28:46.103539+0000","last_fullsized":"2023-10-30T12:28:46.103539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:30.077107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104499+0000","last_change":"2023-10-30T12:28:44.955323+0000","last_active":"2023-10-30T12:28:46.104499+0000","last_peered":"2023-10-30T12:28:46.104499+0000","last_clean":"2023-10-30T12:28:46.104499+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T12:28:46.104499+0000","last_undegraded":"2023-10-30T12:28:46.104499+0000","last_fullsized":"2023-10-30T12:28:46.104499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:59:22.424571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945425+0000","last_change":"2023-10-30T12:28:44.946048+0000","last_active":"2023-10-30T12:28:45.945425+0000","last_peered":"2023-10-30T12:28:45.945425+0000","last_clean":"2023-10-30T12:28:45.945425+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T12:28:45.945425+0000","last_undegraded":"2023-10-30T12:28:45.945425+0000","last_fullsized":"2023-10-30T12:28:45.945425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:20:21.145943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939444+0000","last_change":"2023-10-30T12:28:44.954167+0000","last_active":"2023-10-30T12:28:45.939444+0000","last_peered":"2023-10-30T12:28:45.939444+0000","last_clean":"2023-10-30T12:28:45.939444+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T12:28:45.939444+0000","last_undegraded":"2023-10-30T12:28:45.939444+0000","last_fullsized":"2023-10-30T12:28:45.939444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:08.457457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106494+0000","last_change":"2023-10-30T12:28:44.982724+0000","last_active":"2023-10-30T12:28:46.106494+0000","last_peered":"2023-10-30T12:28:46.106494+0000","last_clean":"2023-10-30T12:28:46.106494+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T12:28:46.106494+0000","last_undegraded":"2023-10-30T12:28:46.106494+0000","last_fullsized":"2023-10-30T12:28:46.106494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:13.873943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103197+0000","last_change":"2023-10-30T12:28:44.981034+0000","last_active":"2023-10-30T12:28:46.103197+0000","last_peered":"2023-10-30T12:28:46.103197+0000","last_clean":"2023-10-30T12:28:46.103197+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T12:28:46.103197+0000","last_undegraded":"2023-10-30T12:28:46.103197+0000","last_fullsized":"2023-10-30T12:28:46.103197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:39:25.263228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107211+0000","last_change":"2023-10-30T12:28:44.975010+0000","last_active":"2023-10-30T12:28:46.107211+0000","last_peered":"2023-10-30T12:28:46.107211+0000","last_clean":"2023-10-30T12:28:46.107211+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T12:28:46.107211+0000","last_undegraded":"2023-10-30T12:28:46.107211+0000","last_fullsized":"2023-10-30T12:28:46.107211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:13.768876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104243+0000","last_change":"2023-10-30T12:28:44.951099+0000","last_active":"2023-10-30T12:28:46.104243+0000","last_peered":"2023-10-30T12:28:46.104243+0000","last_clean":"2023-10-30T12:28:46.104243+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T12:28:46.104243+0000","last_undegraded":"2023-10-30T12:28:46.104243+0000","last_fullsized":"2023-10-30T12:28:46.104243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:21:27.902671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946065+0000","last_change":"2023-10-30T12:28:44.959915+0000","last_active":"2023-10-30T12:28:45.946065+0000","last_peered":"2023-10-30T12:28:45.946065+0000","last_clean":"2023-10-30T12:28:45.946065+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T12:28:45.946065+0000","last_undegraded":"2023-10-30T12:28:45.946065+0000","last_fullsized":"2023-10-30T12:28:45.946065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:57:49.711309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946417+0000","last_change":"2023-10-30T12:28:44.972677+0000","last_active":"2023-10-30T12:28:45.946417+0000","last_peered":"2023-10-30T12:28:45.946417+0000","last_clean":"2023-10-30T12:28:45.946417+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T12:28:45.946417+0000","last_undegraded":"2023-10-30T12:28:45.946417+0000","last_fullsized":"2023-10-30T12:28:45.946417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:12.072355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106685+0000","last_change":"2023-10-30T12:28:44.975973+0000","last_active":"2023-10-30T12:28:46.106685+0000","last_peered":"2023-10-30T12:28:46.106685+0000","last_clean":"2023-10-30T12:28:46.106685+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T12:28:46.106685+0000","last_undegraded":"2023-10-30T12:28:46.106685+0000","last_fullsized":"2023-10-30T12:28:46.106685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:48.152823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103531+0000","last_change":"2023-10-30T12:28:44.980814+0000","last_active":"2023-10-30T12:28:46.103531+0000","last_peered":"2023-10-30T12:28:46.103531+0000","last_clean":"2023-10-30T12:28:46.103531+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T12:28:46.103531+0000","last_undegraded":"2023-10-30T12:28:46.103531+0000","last_fullsized":"2023-10-30T12:28:46.103531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:51:22.453883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107798+0000","last_change":"2023-10-30T12:28:44.965026+0000","last_active":"2023-10-30T12:28:46.107798+0000","last_peered":"2023-10-30T12:28:46.107798+0000","last_clean":"2023-10-30T12:28:46.107798+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T12:28:46.107798+0000","last_undegraded":"2023-10-30T12:28:46.107798+0000","last_fullsized":"2023-10-30T12:28:46.107798+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:04:08.680744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102782+0000","last_change":"2023-10-30T12:28:44.973401+0000","last_active":"2023-10-30T12:28:46.102782+0000","last_peered":"2023-10-30T12:28:46.102782+0000","last_clean":"2023-10-30T12:28:46.102782+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T12:28:46.102782+0000","last_undegraded":"2023-10-30T12:28:46.102782+0000","last_fullsized":"2023-10-30T12:28:46.102782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:34.198424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939578+0000","last_change":"2023-10-30T12:28:44.971817+0000","last_active":"2023-10-30T12:28:45.939578+0000","last_peered":"2023-10-30T12:28:45.939578+0000","last_clean":"2023-10-30T12:28:45.939578+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T12:28:45.939578+0000","last_undegraded":"2023-10-30T12:28:45.939578+0000","last_fullsized":"2023-10-30T12:28:45.939578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:45:35.205305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.130160+0000","last_change":"2023-10-30T12:28:44.074842+0000","last_active":"2023-10-30T12:28:55.130160+0000","last_peered":"2023-10-30T12:28:55.130160+0000","last_clean":"2023-10-30T12:28:55.130160+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T12:28:55.130160+0000","last_undegraded":"2023-10-30T12:28:55.130160+0000","last_fullsized":"2023-10-30T12:28:55.130160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:01.428785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945765+0000","last_change":"2023-10-30T12:28:44.958560+0000","last_active":"2023-10-30T12:28:45.945765+0000","last_peered":"2023-10-30T12:28:45.945765+0000","last_clean":"2023-10-30T12:28:45.945765+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T12:28:45.945765+0000","last_undegraded":"2023-10-30T12:28:45.945765+0000","last_fullsized":"2023-10-30T12:28:45.945765+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:45.564493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.963498+0000","last_change":"2023-10-30T12:28:44.060621+0000","last_active":"2023-10-30T12:29:00.963498+0000","last_peered":"2023-10-30T12:29:00.963498+0000","last_clean":"2023-10-30T12:29:00.963498+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T12:29:00.963498+0000","last_undegraded":"2023-10-30T12:29:00.963498+0000","last_fullsized":"2023-10-30T12:29:00.963498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:53.741238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.967207+0000","last_change":"2023-10-30T12:28:44.060171+0000","last_active":"2023-10-30T12:29:00.967207+0000","last_peered":"2023-10-30T12:29:00.967207+0000","last_clean":"2023-10-30T12:29:00.967207+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T12:29:00.967207+0000","last_undegraded":"2023-10-30T12:29:00.967207+0000","last_fullsized":"2023-10-30T12:29:00.967207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:18.181418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106893+0000","last_change":"2023-10-30T12:28:44.978631+0000","last_active":"2023-10-30T12:28:46.106893+0000","last_peered":"2023-10-30T12:28:46.106893+0000","last_clean":"2023-10-30T12:28:46.106893+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T12:28:46.106893+0000","last_undegraded":"2023-10-30T12:28:46.106893+0000","last_fullsized":"2023-10-30T12:28:46.106893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:37.006427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:47.984674+0000","last_change":"2023-10-30T12:28:44.058798+0000","last_active":"2023-10-30T12:28:47.984674+0000","last_peered":"2023-10-30T12:28:47.984674+0000","last_clean":"2023-10-30T12:28:47.984674+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T12:28:47.984674+0000","last_undegraded":"2023-10-30T12:28:47.984674+0000","last_fullsized":"2023-10-30T12:28:47.984674+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:08.643080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945025+0000","last_change":"2023-10-30T12:28:44.949118+0000","last_active":"2023-10-30T12:28:45.945025+0000","last_peered":"2023-10-30T12:28:45.945025+0000","last_clean":"2023-10-30T12:28:45.945025+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T12:28:45.945025+0000","last_undegraded":"2023-10-30T12:28:45.945025+0000","last_fullsized":"2023-10-30T12:28:45.945025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:11.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.130828+0000","last_change":"2023-10-30T12:28:44.063439+0000","last_active":"2023-10-30T12:28:55.130828+0000","last_peered":"2023-10-30T12:28:55.130828+0000","last_clean":"2023-10-30T12:28:55.130828+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T12:28:55.130828+0000","last_undegraded":"2023-10-30T12:28:55.130828+0000","last_fullsized":"2023-10-30T12:28:55.130828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:08.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104375+0000","last_change":"2023-10-30T12:28:44.951319+0000","last_active":"2023-10-30T12:28:46.104375+0000","last_peered":"2023-10-30T12:28:46.104375+0000","last_clean":"2023-10-30T12:28:46.104375+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T12:28:46.104375+0000","last_undegraded":"2023-10-30T12:28:46.104375+0000","last_fullsized":"2023-10-30T12:28:46.104375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:20.108360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.134075+0000","last_change":"2023-10-30T12:28:44.052641+0000","last_active":"2023-10-30T12:28:55.134075+0000","last_peered":"2023-10-30T12:28:55.134075+0000","last_clean":"2023-10-30T12:28:55.134075+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T12:28:55.134075+0000","last_undegraded":"2023-10-30T12:28:55.134075+0000","last_fullsized":"2023-10-30T12:28:55.134075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107640+0000","last_change":"2023-10-30T12:28:44.981614+0000","last_active":"2023-10-30T12:28:46.107640+0000","last_peered":"2023-10-30T12:28:46.107640+0000","last_clean":"2023-10-30T12:28:46.107640+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T12:28:46.107640+0000","last_undegraded":"2023-10-30T12:28:46.107640+0000","last_fullsized":"2023-10-30T12:28:46.107640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:15:11.750243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.068083+0000","last_change":"2023-10-30T12:28:44.075783+0000","last_active":"2023-10-30T12:28:50.068083+0000","last_peered":"2023-10-30T12:28:50.068083+0000","last_clean":"2023-10-30T12:28:50.068083+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T12:28:50.068083+0000","last_undegraded":"2023-10-30T12:28:50.068083+0000","last_fullsized":"2023-10-30T12:28:50.068083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:01.094929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103142+0000","last_change":"2023-10-30T12:28:44.950899+0000","last_active":"2023-10-30T12:28:46.103142+0000","last_peered":"2023-10-30T12:28:46.103142+0000","last_clean":"2023-10-30T12:28:46.103142+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T12:28:46.103142+0000","last_undegraded":"2023-10-30T12:28:46.103142+0000","last_fullsized":"2023-10-30T12:28:46.103142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:34.005863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.132373+0000","last_change":"2023-10-30T12:28:44.063367+0000","last_active":"2023-10-30T12:28:55.132373+0000","last_peered":"2023-10-30T12:28:55.132373+0000","last_clean":"2023-10-30T12:28:55.132373+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T12:28:55.132373+0000","last_undegraded":"2023-10-30T12:28:55.132373+0000","last_fullsized":"2023-10-30T12:28:55.132373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:45.585121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104587+0000","last_change":"2023-10-30T12:28:44.972056+0000","last_active":"2023-10-30T12:28:46.104587+0000","last_peered":"2023-10-30T12:28:46.104587+0000","last_clean":"2023-10-30T12:28:46.104587+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T12:28:46.104587+0000","last_undegraded":"2023-10-30T12:28:46.104587+0000","last_fullsized":"2023-10-30T12:28:46.104587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:06:08.345856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.966521+0000","last_change":"2023-10-30T12:28:44.065545+0000","last_active":"2023-10-30T12:29:00.966521+0000","last_peered":"2023-10-30T12:29:00.966521+0000","last_clean":"2023-10-30T12:29:00.966521+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T12:29:00.966521+0000","last_undegraded":"2023-10-30T12:29:00.966521+0000","last_fullsized":"2023-10-30T12:29:00.966521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:09.900189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945563+0000","last_change":"2023-10-30T12:28:44.961237+0000","last_active":"2023-10-30T12:28:45.945563+0000","last_peered":"2023-10-30T12:28:45.945563+0000","last_clean":"2023-10-30T12:28:45.945563+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T12:28:45.945563+0000","last_undegraded":"2023-10-30T12:28:45.945563+0000","last_fullsized":"2023-10-30T12:28:45.945563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:17.160724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.130351+0000","last_change":"2023-10-30T12:28:44.061922+0000","last_active":"2023-10-30T12:28:55.130351+0000","last_peered":"2023-10-30T12:28:55.130351+0000","last_clean":"2023-10-30T12:28:55.130351+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T12:28:55.130351+0000","last_undegraded":"2023-10-30T12:28:55.130351+0000","last_fullsized":"2023-10-30T12:28:55.130351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:34.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103258+0000","last_change":"2023-10-30T12:28:44.958245+0000","last_active":"2023-10-30T12:28:46.103258+0000","last_peered":"2023-10-30T12:28:46.103258+0000","last_clean":"2023-10-30T12:28:46.103258+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T12:28:46.103258+0000","last_undegraded":"2023-10-30T12:28:46.103258+0000","last_fullsized":"2023-10-30T12:28:46.103258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:42:10.840490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.964145+0000","last_change":"2023-10-30T12:28:44.059784+0000","last_active":"2023-10-30T12:29:00.964145+0000","last_peered":"2023-10-30T12:29:00.964145+0000","last_clean":"2023-10-30T12:29:00.964145+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T12:29:00.964145+0000","last_undegraded":"2023-10-30T12:29:00.964145+0000","last_fullsized":"2023-10-30T12:29:00.964145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:11.468221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939230+0000","last_change":"2023-10-30T12:28:44.980813+0000","last_active":"2023-10-30T12:28:45.939230+0000","last_peered":"2023-10-30T12:28:45.939230+0000","last_clean":"2023-10-30T12:28:45.939230+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T12:28:45.939230+0000","last_undegraded":"2023-10-30T12:28:45.939230+0000","last_fullsized":"2023-10-30T12:28:45.939230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:10.858970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'15","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.975410+0000","last_change":"2023-10-30T12:28:44.065885+0000","last_active":"2023-10-30T12:29:00.975410+0000","last_peered":"2023-10-30T12:29:00.975410+0000","last_clean":"2023-10-30T12:29:00.975410+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T12:29:00.975410+0000","last_undegraded":"2023-10-30T12:29:00.975410+0000","last_fullsized":"2023-10-30T12:29:00.975410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:26.313595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103080+0000","last_change":"2023-10-30T12:28:44.952458+0000","last_active":"2023-10-30T12:28:46.103080+0000","last_peered":"2023-10-30T12:28:46.103080+0000","last_clean":"2023-10-30T12:28:46.103080+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T12:28:46.103080+0000","last_undegraded":"2023-10-30T12:28:46.103080+0000","last_fullsized":"2023-10-30T12:28:46.103080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:48:19.752492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.062786+0000","last_change":"2023-10-30T12:28:44.080011+0000","last_active":"2023-10-30T12:28:50.062786+0000","last_peered":"2023-10-30T12:28:50.062786+0000","last_clean":"2023-10-30T12:28:50.062786+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T12:28:50.062786+0000","last_undegraded":"2023-10-30T12:28:50.062786+0000","last_fullsized":"2023-10-30T12:28:50.062786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104494+0000","last_change":"2023-10-30T12:28:44.961296+0000","last_active":"2023-10-30T12:28:46.104494+0000","last_peered":"2023-10-30T12:28:46.104494+0000","last_clean":"2023-10-30T12:28:46.104494+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T12:28:46.104494+0000","last_undegraded":"2023-10-30T12:28:46.104494+0000","last_fullsized":"2023-10-30T12:28:46.104494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:56.540392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.066004+0000","last_change":"2023-10-30T12:28:44.076352+0000","last_active":"2023-10-30T12:28:50.066004+0000","last_peered":"2023-10-30T12:28:50.066004+0000","last_clean":"2023-10-30T12:28:50.066004+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T12:28:50.066004+0000","last_undegraded":"2023-10-30T12:28:50.066004+0000","last_fullsized":"2023-10-30T12:28:50.066004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104918+0000","last_change":"2023-10-30T12:28:44.956918+0000","last_active":"2023-10-30T12:28:46.104918+0000","last_peered":"2023-10-30T12:28:46.104918+0000","last_clean":"2023-10-30T12:28:46.104918+0000","last_became_active":"2023-10-30T12:28:44.956397+0000","last_became_peered":"2023-10-30T12:28:44.956397+0000","last_unstale":"2023-10-30T12:28:46.104918+0000","last_undegraded":"2023-10-30T12:28:46.104918+0000","last_fullsized":"2023-10-30T12:28:46.104918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:26.252196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106813+0000","last_change":"2023-10-30T12:28:44.075684+0000","last_active":"2023-10-30T12:28:46.106813+0000","last_peered":"2023-10-30T12:28:46.106813+0000","last_clean":"2023-10-30T12:28:46.106813+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T12:28:46.106813+0000","last_undegraded":"2023-10-30T12:28:46.106813+0000","last_fullsized":"2023-10-30T12:28:46.106813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:07.375795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104588+0000","last_change":"2023-10-30T12:28:44.956047+0000","last_active":"2023-10-30T12:28:46.104588+0000","last_peered":"2023-10-30T12:28:46.104588+0000","last_clean":"2023-10-30T12:28:46.104588+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T12:28:46.104588+0000","last_undegraded":"2023-10-30T12:28:46.104588+0000","last_fullsized":"2023-10-30T12:28:46.104588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:49.924947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.970394+0000","last_change":"2023-10-30T12:28:44.060686+0000","last_active":"2023-10-30T12:29:00.970394+0000","last_peered":"2023-10-30T12:29:00.970394+0000","last_clean":"2023-10-30T12:29:00.970394+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T12:29:00.970394+0000","last_undegraded":"2023-10-30T12:29:00.970394+0000","last_fullsized":"2023-10-30T12:29:00.970394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:23.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103562+0000","last_change":"2023-10-30T12:28:44.946758+0000","last_active":"2023-10-30T12:28:46.103562+0000","last_peered":"2023-10-30T12:28:46.103562+0000","last_clean":"2023-10-30T12:28:46.103562+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T12:28:46.103562+0000","last_undegraded":"2023-10-30T12:28:46.103562+0000","last_fullsized":"2023-10-30T12:28:46.103562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:20.520148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.932509+0000","last_change":"2023-10-30T12:28:33.433824+0000","last_active":"2023-10-30T12:28:44.932509+0000","last_peered":"2023-10-30T12:28:44.932509+0000","last_clean":"2023-10-30T12:28:44.932509+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T12:28:44.932509+0000","last_undegraded":"2023-10-30T12:28:44.932509+0000","last_fullsized":"2023-10-30T12:28:44.932509+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_clean_scrub_stamp":"2023-10-30T12:28:31.408255+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:17.624179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.132911+0000","last_change":"2023-10-30T12:28:44.059051+0000","last_active":"2023-10-30T12:28:55.132911+0000","last_peered":"2023-10-30T12:28:55.132911+0000","last_clean":"2023-10-30T12:28:55.132911+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T12:28:55.132911+0000","last_undegraded":"2023-10-30T12:28:55.132911+0000","last_fullsized":"2023-10-30T12:28:55.132911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:54.810636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945062+0000","last_change":"2023-10-30T12:28:44.965183+0000","last_active":"2023-10-30T12:28:45.945062+0000","last_peered":"2023-10-30T12:28:45.945062+0000","last_clean":"2023-10-30T12:28:45.945062+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T12:28:45.945062+0000","last_undegraded":"2023-10-30T12:28:45.945062+0000","last_fullsized":"2023-10-30T12:28:45.945062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:57.655687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.962924+0000","last_change":"2023-10-30T12:28:44.080378+0000","last_active":"2023-10-30T12:29:00.962924+0000","last_peered":"2023-10-30T12:29:00.962924+0000","last_clean":"2023-10-30T12:29:00.962924+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T12:29:00.962924+0000","last_undegraded":"2023-10-30T12:29:00.962924+0000","last_fullsized":"2023-10-30T12:29:00.962924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:22.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944927+0000","last_change":"2023-10-30T12:28:44.961659+0000","last_active":"2023-10-30T12:28:45.944927+0000","last_peered":"2023-10-30T12:28:45.944927+0000","last_clean":"2023-10-30T12:28:45.944927+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T12:28:45.944927+0000","last_undegraded":"2023-10-30T12:28:45.944927+0000","last_fullsized":"2023-10-30T12:28:45.944927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:34:00.000084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.969477+0000","last_change":"2023-10-30T12:28:44.082480+0000","last_active":"2023-10-30T12:29:00.969477+0000","last_peered":"2023-10-30T12:29:00.969477+0000","last_clean":"2023-10-30T12:29:00.969477+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T12:29:00.969477+0000","last_undegraded":"2023-10-30T12:29:00.969477+0000","last_fullsized":"2023-10-30T12:29:00.969477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:20.126782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106385+0000","last_change":"2023-10-30T12:28:44.973906+0000","last_active":"2023-10-30T12:28:46.106385+0000","last_peered":"2023-10-30T12:28:46.106385+0000","last_clean":"2023-10-30T12:28:46.106385+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T12:28:46.106385+0000","last_undegraded":"2023-10-30T12:28:46.106385+0000","last_fullsized":"2023-10-30T12:28:46.106385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:31.845894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:47.984766+0000","last_change":"2023-10-30T12:28:44.059806+0000","last_active":"2023-10-30T12:28:47.984766+0000","last_peered":"2023-10-30T12:28:47.984766+0000","last_clean":"2023-10-30T12:28:47.984766+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T12:28:47.984766+0000","last_undegraded":"2023-10-30T12:28:47.984766+0000","last_fullsized":"2023-10-30T12:28:47.984766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:50.939946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104302+0000","last_change":"2023-10-30T12:28:44.955423+0000","last_active":"2023-10-30T12:28:46.104302+0000","last_peered":"2023-10-30T12:28:46.104302+0000","last_clean":"2023-10-30T12:28:46.104302+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T12:28:46.104302+0000","last_undegraded":"2023-10-30T12:28:46.104302+0000","last_fullsized":"2023-10-30T12:28:46.104302+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:30.054048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.982937+0000","last_change":"2023-10-30T12:28:44.074970+0000","last_active":"2023-10-30T12:28:45.982937+0000","last_peered":"2023-10-30T12:28:45.982937+0000","last_clean":"2023-10-30T12:28:45.982937+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T12:28:45.982937+0000","last_undegraded":"2023-10-30T12:28:45.982937+0000","last_fullsized":"2023-10-30T12:28:45.982937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:33.562703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103885+0000","last_change":"2023-10-30T12:28:44.973482+0000","last_active":"2023-10-30T12:28:46.103885+0000","last_peered":"2023-10-30T12:28:46.103885+0000","last_clean":"2023-10-30T12:28:46.103885+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T12:28:46.103885+0000","last_undegraded":"2023-10-30T12:28:46.103885+0000","last_fullsized":"2023-10-30T12:28:46.103885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:25.393161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:47.984860+0000","last_change":"2023-10-30T12:28:44.063256+0000","last_active":"2023-10-30T12:28:47.984860+0000","last_peered":"2023-10-30T12:28:47.984860+0000","last_clean":"2023-10-30T12:28:47.984860+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T12:28:47.984860+0000","last_undegraded":"2023-10-30T12:28:47.984860+0000","last_fullsized":"2023-10-30T12:28:47.984860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:01:36.144966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946945+0000","last_change":"2023-10-30T12:28:44.975443+0000","last_active":"2023-10-30T12:28:45.946945+0000","last_peered":"2023-10-30T12:28:45.946945+0000","last_clean":"2023-10-30T12:28:45.946945+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T12:28:45.946945+0000","last_undegraded":"2023-10-30T12:28:45.946945+0000","last_fullsized":"2023-10-30T12:28:45.946945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:23.765723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.968669+0000","last_change":"2023-10-30T12:28:44.055374+0000","last_active":"2023-10-30T12:29:00.968669+0000","last_peered":"2023-10-30T12:29:00.968669+0000","last_clean":"2023-10-30T12:29:00.968669+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T12:29:00.968669+0000","last_undegraded":"2023-10-30T12:29:00.968669+0000","last_fullsized":"2023-10-30T12:29:00.968669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946991+0000","last_change":"2023-10-30T12:28:44.974359+0000","last_active":"2023-10-30T12:28:45.946991+0000","last_peered":"2023-10-30T12:28:45.946991+0000","last_clean":"2023-10-30T12:28:45.946991+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T12:28:45.946991+0000","last_undegraded":"2023-10-30T12:28:45.946991+0000","last_fullsized":"2023-10-30T12:28:45.946991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:48.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.132893+0000","last_change":"2023-10-30T12:28:44.058663+0000","last_active":"2023-10-30T12:28:55.132893+0000","last_peered":"2023-10-30T12:28:55.132893+0000","last_clean":"2023-10-30T12:28:55.132893+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T12:28:55.132893+0000","last_undegraded":"2023-10-30T12:28:55.132893+0000","last_fullsized":"2023-10-30T12:28:55.132893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:42:22.279653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946815+0000","last_change":"2023-10-30T12:28:44.957471+0000","last_active":"2023-10-30T12:28:45.946815+0000","last_peered":"2023-10-30T12:28:45.946815+0000","last_clean":"2023-10-30T12:28:45.946815+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T12:28:45.946815+0000","last_undegraded":"2023-10-30T12:28:45.946815+0000","last_fullsized":"2023-10-30T12:28:45.946815+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:27:38.669452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.131577+0000","last_change":"2023-10-30T12:28:44.054338+0000","last_active":"2023-10-30T12:28:55.131577+0000","last_peered":"2023-10-30T12:28:55.131577+0000","last_clean":"2023-10-30T12:28:55.131577+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T12:28:55.131577+0000","last_undegraded":"2023-10-30T12:28:55.131577+0000","last_fullsized":"2023-10-30T12:28:55.131577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:55:35.370652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103200+0000","last_change":"2023-10-30T12:28:44.965759+0000","last_active":"2023-10-30T12:28:46.103200+0000","last_peered":"2023-10-30T12:28:46.103200+0000","last_clean":"2023-10-30T12:28:46.103200+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T12:28:46.103200+0000","last_undegraded":"2023-10-30T12:28:46.103200+0000","last_fullsized":"2023-10-30T12:28:46.103200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:13:52.965500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.134624+0000","last_change":"2023-10-30T12:28:44.064606+0000","last_active":"2023-10-30T12:28:55.134624+0000","last_peered":"2023-10-30T12:28:55.134624+0000","last_clean":"2023-10-30T12:28:55.134624+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T12:28:55.134624+0000","last_undegraded":"2023-10-30T12:28:55.134624+0000","last_fullsized":"2023-10-30T12:28:55.134624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944678+0000","last_change":"2023-10-30T12:28:44.958947+0000","last_active":"2023-10-30T12:28:45.944678+0000","last_peered":"2023-10-30T12:28:45.944678+0000","last_clean":"2023-10-30T12:28:45.944678+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T12:28:45.944678+0000","last_undegraded":"2023-10-30T12:28:45.944678+0000","last_fullsized":"2023-10-30T12:28:45.944678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:01:16.329994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.979140+0000","last_change":"2023-10-30T12:28:44.072754+0000","last_active":"2023-10-30T12:28:46.979140+0000","last_peered":"2023-10-30T12:28:46.979140+0000","last_clean":"2023-10-30T12:28:46.979140+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T12:28:46.979140+0000","last_undegraded":"2023-10-30T12:28:46.979140+0000","last_fullsized":"2023-10-30T12:28:46.979140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:51:55.865069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.108063+0000","last_change":"2023-10-30T12:28:44.978376+0000","last_active":"2023-10-30T12:28:46.108063+0000","last_peered":"2023-10-30T12:28:46.108063+0000","last_clean":"2023-10-30T12:28:46.108063+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T12:28:46.108063+0000","last_undegraded":"2023-10-30T12:28:46.108063+0000","last_fullsized":"2023-10-30T12:28:46.108063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:59.643447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.131363+0000","last_change":"2023-10-30T12:28:44.058801+0000","last_active":"2023-10-30T12:28:55.131363+0000","last_peered":"2023-10-30T12:28:55.131363+0000","last_clean":"2023-10-30T12:28:55.131363+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T12:28:55.131363+0000","last_undegraded":"2023-10-30T12:28:55.131363+0000","last_fullsized":"2023-10-30T12:28:55.131363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:11:24.706813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103647+0000","last_change":"2023-10-30T12:28:44.943292+0000","last_active":"2023-10-30T12:28:46.103647+0000","last_peered":"2023-10-30T12:28:46.103647+0000","last_clean":"2023-10-30T12:28:46.103647+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T12:28:46.103647+0000","last_undegraded":"2023-10-30T12:28:46.103647+0000","last_fullsized":"2023-10-30T12:28:46.103647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:37.538525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103472+0000","last_change":"2023-10-30T12:28:44.958607+0000","last_active":"2023-10-30T12:28:46.103472+0000","last_peered":"2023-10-30T12:28:46.103472+0000","last_clean":"2023-10-30T12:28:46.103472+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T12:28:46.103472+0000","last_undegraded":"2023-10-30T12:28:46.103472+0000","last_fullsized":"2023-10-30T12:28:46.103472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:28:55.169198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.059856+0000","last_change":"2023-10-30T12:28:44.054652+0000","last_active":"2023-10-30T12:28:50.059856+0000","last_peered":"2023-10-30T12:28:50.059856+0000","last_clean":"2023-10-30T12:28:50.059856+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T12:28:50.059856+0000","last_undegraded":"2023-10-30T12:28:50.059856+0000","last_fullsized":"2023-10-30T12:28:50.059856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:05.023374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939383+0000","last_change":"2023-10-30T12:28:44.958246+0000","last_active":"2023-10-30T12:28:45.939383+0000","last_peered":"2023-10-30T12:28:45.939383+0000","last_clean":"2023-10-30T12:28:45.939383+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T12:28:45.939383+0000","last_undegraded":"2023-10-30T12:28:45.939383+0000","last_fullsized":"2023-10-30T12:28:45.939383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:36.947398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.963915+0000","last_change":"2023-10-30T12:28:44.052633+0000","last_active":"2023-10-30T12:29:00.963915+0000","last_peered":"2023-10-30T12:29:00.963915+0000","last_clean":"2023-10-30T12:29:00.963915+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T12:29:00.963915+0000","last_undegraded":"2023-10-30T12:29:00.963915+0000","last_fullsized":"2023-10-30T12:29:00.963915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107324+0000","last_change":"2023-10-30T12:28:44.974457+0000","last_active":"2023-10-30T12:28:46.107324+0000","last_peered":"2023-10-30T12:28:46.107324+0000","last_clean":"2023-10-30T12:28:46.107324+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T12:28:46.107324+0000","last_undegraded":"2023-10-30T12:28:46.107324+0000","last_fullsized":"2023-10-30T12:28:46.107324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:25:56.791053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.064135+0000","last_change":"2023-10-30T12:28:44.075493+0000","last_active":"2023-10-30T12:28:50.064135+0000","last_peered":"2023-10-30T12:28:50.064135+0000","last_clean":"2023-10-30T12:28:50.064135+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T12:28:50.064135+0000","last_undegraded":"2023-10-30T12:28:50.064135+0000","last_fullsized":"2023-10-30T12:28:50.064135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:00:11.718669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107054+0000","last_change":"2023-10-30T12:28:44.974424+0000","last_active":"2023-10-30T12:28:46.107054+0000","last_peered":"2023-10-30T12:28:46.107054+0000","last_clean":"2023-10-30T12:28:46.107054+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T12:28:46.107054+0000","last_undegraded":"2023-10-30T12:28:46.107054+0000","last_fullsized":"2023-10-30T12:28:46.107054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:14.375671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'9","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.971551+0000","last_change":"2023-10-30T12:28:44.065843+0000","last_active":"2023-10-30T12:29:00.971551+0000","last_peered":"2023-10-30T12:29:00.971551+0000","last_clean":"2023-10-30T12:29:00.971551+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T12:29:00.971551+0000","last_undegraded":"2023-10-30T12:29:00.971551+0000","last_fullsized":"2023-10-30T12:29:00.971551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:32.405199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5222,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476748,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":77094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]}]},{"osd":2,"up_from":16,"seq":68719476748,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":77514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]}]},{"osd":1,"up_from":16,"seq":68719476746,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":819200,"data_stored":666396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]}]},{"osd":0,"up_from":16,"seq":68719476747,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":77184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509452,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":77072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0669999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509453,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31432,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340408,"statfs":{"total":96636764160,"available":96604577792,"internally_reserved":0,"allocated":262144,"data_stored":80293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]}]},{"osd":7,"up_from":14,"seq":60129542155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":76776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542154,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":667040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":4425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:29:14.288 INFO:tasks.mds_thrash.ceph_manager:clean! 2023-10-30T12:29:14.289 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T12:29:14.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.463+0000 7f73e4b9d640 1 Processor -- start 2023-10-30T12:29:14.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.463+0000 7f73e4b9d640 1 -- start start 2023-10-30T12:29:14.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73e4b9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f73e006f7a0 0x7f73e006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73e4b9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f73e00c6bd0 0x7f73e0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73e4b9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f73e0071110 0x7f73e0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73e4b9d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e00700c0 con 0x7f73e0071110 2023-10-30T12:29:14.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73e4b9d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73e006a3d0 con 0x7f73e006f7a0 2023-10-30T12:29:14.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73e4b9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0066cf0 con 0x7f73e00c6bd0 2023-10-30T12:29:14.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73ded9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f73e0071110 0x7f73e0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73ded9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f73e0071110 0x7f73e0158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40006/0 (socket says 172.21.15.125:40006) 2023-10-30T12:29:14.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73ded9d640 1 -- 172.21.15.125:0/2815025286 learned_addr learned my addr 172.21.15.125:0/2815025286 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:14.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73de59c640 1 --2- 172.21.15.125:0/2815025286 >> v2:172.21.15.136:3300/0 conn(0x7f73e006f7a0 0x7f73e006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73d5d9b640 1 --2- 172.21.15.125:0/2815025286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f73e00c6bd0 0x7f73e0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73ded9d640 1 -- 172.21.15.125:0/2815025286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f73e00c6bd0 msgr2=0x7f73e0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:14.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73ded9d640 1 --2- 172.21.15.125:0/2815025286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f73e00c6bd0 0x7f73e0069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73ded9d640 1 -- 172.21.15.125:0/2815025286 >> v2:172.21.15.136:3300/0 conn(0x7f73e006f7a0 msgr2=0x7f73e006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:14.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73ded9d640 1 --2- 172.21.15.125:0/2815025286 >> v2:172.21.15.136:3300/0 conn(0x7f73e006f7a0 0x7f73e006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.467+0000 7f73ded9d640 1 -- 172.21.15.125:0/2815025286 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73e00c70d0 con 0x7f73e0071110 2023-10-30T12:29:14.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73ded9d640 1 --2- 172.21.15.125:0/2815025286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73e0071110 0x7f73e0158110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f73e006c830 tx=0x7f73d801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b2acd23618815 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:14.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73ddd9b640 1 -- 172.21.15.125:0/2815025286 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d80a89b0 con 0x7f73e0071110 2023-10-30T12:29:14.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73ddd9b640 1 -- 172.21.15.125:0/2815025286 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73d80a8b10 con 0x7f73e0071110 2023-10-30T12:29:14.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73ddd9b640 1 -- 172.21.15.125:0/2815025286 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d80a8ea0 con 0x7f73e0071110 2023-10-30T12:29:14.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2815025286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73e0071110 msgr2=0x7f73e0158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73e4b9d640 1 --2- 172.21.15.125:0/2815025286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73e0071110 0x7f73e0158110 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f73e006c830 tx=0x7f73d801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2815025286 shutdown_connections 2023-10-30T12:29:14.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73e4b9d640 1 --2- 172.21.15.125:0/2815025286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73e0071110 0x7f73e0158110 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73e4b9d640 1 --2- 172.21.15.125:0/2815025286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f73e00c6bd0 0x7f73e0069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73e4b9d640 1 --2- 172.21.15.125:0/2815025286 >> v2:172.21.15.136:3300/0 conn(0x7f73e006f7a0 0x7f73e006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2815025286 >> 172.21.15.125:0/2815025286 conn(0x7f73e00c7910 msgr2=0x7f73e00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:14.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2815025286 shutdown_connections 2023-10-30T12:29:14.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2815025286 wait complete. 2023-10-30T12:29:14.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73e4b9d640 1 Processor -- start 2023-10-30T12:29:14.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.471+0000 7f73e4b9d640 1 -- start start 2023-10-30T12:29:14.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73e4b9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f73e006f7a0 0x7f73e00c1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73e4b9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73e0071110 0x7f73e00aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73e4b9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73e00c6bd0 0x7f73e00c5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73e4b9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0149800 con 0x7f73e0071110 2023-10-30T12:29:14.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73e4b9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e00c1ec0 con 0x7f73e00c6bd0 2023-10-30T12:29:14.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73e4b9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73e01f25a0 con 0x7f73e006f7a0 2023-10-30T12:29:14.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73d5d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73e0071110 0x7f73e00aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73d5d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73e0071110 0x7f73e00aee30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40010/0 (socket says 172.21.15.125:40010) 2023-10-30T12:29:14.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73d5d9b640 1 -- 172.21.15.125:0/2036629666 learned_addr learned my addr 172.21.15.125:0/2036629666 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:14.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73ded9d640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73e00c6bd0 0x7f73e00c5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73de59c640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f73e006f7a0 0x7f73e00c1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73d5d9b640 1 -- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f73e006f7a0 msgr2=0x7f73e00c1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73d5d9b640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f73e006f7a0 0x7f73e00c1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73d5d9b640 1 -- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73e00c6bd0 msgr2=0x7f73e00c5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73d5d9b640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73e00c6bd0 0x7f73e00c5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73d5d9b640 1 -- 172.21.15.125:0/2036629666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d80094d0 con 0x7f73e0071110 2023-10-30T12:29:14.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73de59c640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f73e006f7a0 0x7f73e00c1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:14.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73d5d9b640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73e0071110 0x7f73e00aee30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f73c8007e40 tx=0x7f73c800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:14.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73d77fe640 1 -- 172.21.15.125:0/2036629666 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c8014620 con 0x7f73e0071110 2023-10-30T12:29:14.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73d77fe640 1 -- 172.21.15.125:0/2036629666 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c800ee90 con 0x7f73e0071110 2023-10-30T12:29:14.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2036629666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e01f2880 con 0x7f73e0071110 2023-10-30T12:29:14.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73d77fe640 1 -- 172.21.15.125:0/2036629666 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73c80149c0 con 0x7f73e0071110 2023-10-30T12:29:14.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2036629666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e01f2e40 con 0x7f73e0071110 2023-10-30T12:29:14.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2036629666 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e006ad80 con 0x7f73e0071110 2023-10-30T12:29:14.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.475+0000 7f73d77fe640 1 -- 172.21.15.125:0/2036629666 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f73c8014b20 con 0x7f73e0071110 2023-10-30T12:29:14.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.479+0000 7f73d77fe640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f73c00664a0 0x7f73c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.479+0000 7f73d77fe640 1 -- 172.21.15.125:0/2036629666 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73c80847f0 con 0x7f73e0071110 2023-10-30T12:29:14.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.479+0000 7f73de59c640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f73c00664a0 0x7f73c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.479+0000 7f73de59c640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f73c00664a0 0x7f73c0068960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f73d000ba50 tx=0x7f73d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:14.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.483+0000 7f73d77fe640 1 -- 172.21.15.125:0/2036629666 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f73c8051bd0 con 0x7f73e0071110 2023-10-30T12:29:14.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.627+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2036629666 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f73e0158650 con 0x7f73c00664a0 2023-10-30T12:29:14.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.647+0000 7f73d77fe640 1 -- 172.21.15.125:0/2036629666 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264578 (secure 0 0 0) 0x7f73d000b6f0 con 0x7f73c00664a0 2023-10-30T12:29:14.653 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:29:14.655 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T12:29:14.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.651+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f73c00664a0 msgr2=0x7f73c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.651+0000 7f73e4b9d640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f73c00664a0 0x7f73c0068960 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f73d000ba50 tx=0x7f73d0008040 comp rx=0 tx=0).stop 2023-10-30T12:29:14.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.651+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2036629666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73e0071110 msgr2=0x7f73e00aee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.651+0000 7f73e4b9d640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73e0071110 0x7f73e00aee30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f73c8007e40 tx=0x7f73c800e4b0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.651+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2036629666 shutdown_connections 2023-10-30T12:29:14.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.651+0000 7f73e4b9d640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f73c00664a0 0x7f73c0068960 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.651+0000 7f73e4b9d640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f73e00c6bd0 0x7f73e00c5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.651+0000 7f73e4b9d640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f73e0071110 0x7f73e00aee30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.651+0000 7f73e4b9d640 1 --2- 172.21.15.125:0/2036629666 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f73e006f7a0 0x7f73e00c1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.651+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2036629666 >> 172.21.15.125:0/2036629666 conn(0x7f73e00c7910 msgr2=0x7f73e00c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:14.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.651+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2036629666 shutdown_connections 2023-10-30T12:29:14.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.651+0000 7f73e4b9d640 1 -- 172.21.15.125:0/2036629666 wait complete. 2023-10-30T12:29:14.673 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2023-10-30T12:29:12.970063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595590,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":284,"num_write_kb":2454,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":227464,"kb_used_data":3016,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":754747256,"statfs":{"total":773094113280,"available":772861190144,"internally_reserved":0,"allocated":3088384,"data_stored":1799369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":229756473},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1406,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":41,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004517"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104854+0000","last_change":"2023-10-30T12:28:44.957488+0000","last_active":"2023-10-30T12:28:46.104854+0000","last_peered":"2023-10-30T12:28:46.104854+0000","last_clean":"2023-10-30T12:28:46.104854+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T12:28:46.104854+0000","last_undegraded":"2023-10-30T12:28:46.104854+0000","last_fullsized":"2023-10-30T12:28:46.104854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:24:30.240483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106772+0000","last_change":"2023-10-30T12:28:44.978898+0000","last_active":"2023-10-30T12:28:46.106772+0000","last_peered":"2023-10-30T12:28:46.106772+0000","last_clean":"2023-10-30T12:28:46.106772+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T12:28:46.106772+0000","last_undegraded":"2023-10-30T12:28:46.106772+0000","last_fullsized":"2023-10-30T12:28:46.106772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:57:21.463798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106769+0000","last_change":"2023-10-30T12:28:44.980722+0000","last_active":"2023-10-30T12:28:46.106769+0000","last_peered":"2023-10-30T12:28:46.106769+0000","last_clean":"2023-10-30T12:28:46.106769+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T12:28:46.106769+0000","last_undegraded":"2023-10-30T12:28:46.106769+0000","last_fullsized":"2023-10-30T12:28:46.106769+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:49:25.009946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103812+0000","last_change":"2023-10-30T12:28:44.965061+0000","last_active":"2023-10-30T12:28:46.103812+0000","last_peered":"2023-10-30T12:28:46.103812+0000","last_clean":"2023-10-30T12:28:46.103812+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T12:28:46.103812+0000","last_undegraded":"2023-10-30T12:28:46.103812+0000","last_fullsized":"2023-10-30T12:28:46.103812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:23:43.749064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104285+0000","last_change":"2023-10-30T12:28:44.976985+0000","last_active":"2023-10-30T12:28:46.104285+0000","last_peered":"2023-10-30T12:28:46.104285+0000","last_clean":"2023-10-30T12:28:46.104285+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T12:28:46.104285+0000","last_undegraded":"2023-10-30T12:28:46.104285+0000","last_fullsized":"2023-10-30T12:28:46.104285+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:33:08.758140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103030+0000","last_change":"2023-10-30T12:28:44.965453+0000","last_active":"2023-10-30T12:28:46.103030+0000","last_peered":"2023-10-30T12:28:46.103030+0000","last_clean":"2023-10-30T12:28:46.103030+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T12:28:46.103030+0000","last_undegraded":"2023-10-30T12:28:46.103030+0000","last_fullsized":"2023-10-30T12:28:46.103030+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:19.975226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103434+0000","last_change":"2023-10-30T12:28:44.968493+0000","last_active":"2023-10-30T12:28:46.103434+0000","last_peered":"2023-10-30T12:28:46.103434+0000","last_clean":"2023-10-30T12:28:46.103434+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T12:28:46.103434+0000","last_undegraded":"2023-10-30T12:28:46.103434+0000","last_fullsized":"2023-10-30T12:28:46.103434+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:58.895635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106402+0000","last_change":"2023-10-30T12:28:44.972695+0000","last_active":"2023-10-30T12:28:46.106402+0000","last_peered":"2023-10-30T12:28:46.106402+0000","last_clean":"2023-10-30T12:28:46.106402+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T12:28:46.106402+0000","last_undegraded":"2023-10-30T12:28:46.106402+0000","last_fullsized":"2023-10-30T12:28:46.106402+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:19:45.353683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939624+0000","last_change":"2023-10-30T12:28:44.946156+0000","last_active":"2023-10-30T12:28:45.939624+0000","last_peered":"2023-10-30T12:28:45.939624+0000","last_clean":"2023-10-30T12:28:45.939624+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T12:28:45.939624+0000","last_undegraded":"2023-10-30T12:28:45.939624+0000","last_fullsized":"2023-10-30T12:28:45.939624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:42:36.408944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104638+0000","last_change":"2023-10-30T12:28:44.957225+0000","last_active":"2023-10-30T12:28:46.104638+0000","last_peered":"2023-10-30T12:28:46.104638+0000","last_clean":"2023-10-30T12:28:46.104638+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T12:28:46.104638+0000","last_undegraded":"2023-10-30T12:28:46.104638+0000","last_fullsized":"2023-10-30T12:28:46.104638+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:55:20.194553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104526+0000","last_change":"2023-10-30T12:28:44.956025+0000","last_active":"2023-10-30T12:28:46.104526+0000","last_peered":"2023-10-30T12:28:46.104526+0000","last_clean":"2023-10-30T12:28:46.104526+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T12:28:46.104526+0000","last_undegraded":"2023-10-30T12:28:46.104526+0000","last_fullsized":"2023-10-30T12:28:46.104526+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:02:12.113264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104464+0000","last_change":"2023-10-30T12:28:44.956832+0000","last_active":"2023-10-30T12:28:46.104464+0000","last_peered":"2023-10-30T12:28:46.104464+0000","last_clean":"2023-10-30T12:28:46.104464+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T12:28:46.104464+0000","last_undegraded":"2023-10-30T12:28:46.104464+0000","last_fullsized":"2023-10-30T12:28:46.104464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:36.932024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945824+0000","last_change":"2023-10-30T12:28:44.962961+0000","last_active":"2023-10-30T12:28:45.945824+0000","last_peered":"2023-10-30T12:28:45.945824+0000","last_clean":"2023-10-30T12:28:45.945824+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T12:28:45.945824+0000","last_undegraded":"2023-10-30T12:28:45.945824+0000","last_fullsized":"2023-10-30T12:28:45.945824+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:02:13.200996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945413+0000","last_change":"2023-10-30T12:28:44.968005+0000","last_active":"2023-10-30T12:28:45.945413+0000","last_peered":"2023-10-30T12:28:45.945413+0000","last_clean":"2023-10-30T12:28:45.945413+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T12:28:45.945413+0000","last_undegraded":"2023-10-30T12:28:45.945413+0000","last_fullsized":"2023-10-30T12:28:45.945413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:53:25.168553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106306+0000","last_change":"2023-10-30T12:28:44.977064+0000","last_active":"2023-10-30T12:28:46.106306+0000","last_peered":"2023-10-30T12:28:46.106306+0000","last_clean":"2023-10-30T12:28:46.106306+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T12:28:46.106306+0000","last_undegraded":"2023-10-30T12:28:46.106306+0000","last_fullsized":"2023-10-30T12:28:46.106306+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:27:44.043189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102782+0000","last_change":"2023-10-30T12:28:44.950169+0000","last_active":"2023-10-30T12:28:46.102782+0000","last_peered":"2023-10-30T12:28:46.102782+0000","last_clean":"2023-10-30T12:28:46.102782+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T12:28:46.102782+0000","last_undegraded":"2023-10-30T12:28:46.102782+0000","last_fullsized":"2023-10-30T12:28:46.102782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:35.297525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103374+0000","last_change":"2023-10-30T12:28:44.960467+0000","last_active":"2023-10-30T12:28:46.103374+0000","last_peered":"2023-10-30T12:28:46.103374+0000","last_clean":"2023-10-30T12:28:46.103374+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T12:28:46.103374+0000","last_undegraded":"2023-10-30T12:28:46.103374+0000","last_fullsized":"2023-10-30T12:28:46.103374+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:35:33.495256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.108582+0000","last_change":"2023-10-30T12:28:44.968948+0000","last_active":"2023-10-30T12:28:46.108582+0000","last_peered":"2023-10-30T12:28:46.108582+0000","last_clean":"2023-10-30T12:28:46.108582+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T12:28:46.108582+0000","last_undegraded":"2023-10-30T12:28:46.108582+0000","last_fullsized":"2023-10-30T12:28:46.108582+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:26:14.661648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103539+0000","last_change":"2023-10-30T12:28:44.966743+0000","last_active":"2023-10-30T12:28:46.103539+0000","last_peered":"2023-10-30T12:28:46.103539+0000","last_clean":"2023-10-30T12:28:46.103539+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T12:28:46.103539+0000","last_undegraded":"2023-10-30T12:28:46.103539+0000","last_fullsized":"2023-10-30T12:28:46.103539+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:30.077107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104499+0000","last_change":"2023-10-30T12:28:44.955323+0000","last_active":"2023-10-30T12:28:46.104499+0000","last_peered":"2023-10-30T12:28:46.104499+0000","last_clean":"2023-10-30T12:28:46.104499+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T12:28:46.104499+0000","last_undegraded":"2023-10-30T12:28:46.104499+0000","last_fullsized":"2023-10-30T12:28:46.104499+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:59:22.424571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945425+0000","last_change":"2023-10-30T12:28:44.946048+0000","last_active":"2023-10-30T12:28:45.945425+0000","last_peered":"2023-10-30T12:28:45.945425+0000","last_clean":"2023-10-30T12:28:45.945425+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T12:28:45.945425+0000","last_undegraded":"2023-10-30T12:28:45.945425+0000","last_fullsized":"2023-10-30T12:28:45.945425+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:20:21.145943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939444+0000","last_change":"2023-10-30T12:28:44.954167+0000","last_active":"2023-10-30T12:28:45.939444+0000","last_peered":"2023-10-30T12:28:45.939444+0000","last_clean":"2023-10-30T12:28:45.939444+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T12:28:45.939444+0000","last_undegraded":"2023-10-30T12:28:45.939444+0000","last_fullsized":"2023-10-30T12:28:45.939444+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:51:08.457457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106494+0000","last_change":"2023-10-30T12:28:44.982724+0000","last_active":"2023-10-30T12:28:46.106494+0000","last_peered":"2023-10-30T12:28:46.106494+0000","last_clean":"2023-10-30T12:28:46.106494+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T12:28:46.106494+0000","last_undegraded":"2023-10-30T12:28:46.106494+0000","last_fullsized":"2023-10-30T12:28:46.106494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:13.873943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103197+0000","last_change":"2023-10-30T12:28:44.981034+0000","last_active":"2023-10-30T12:28:46.103197+0000","last_peered":"2023-10-30T12:28:46.103197+0000","last_clean":"2023-10-30T12:28:46.103197+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T12:28:46.103197+0000","last_undegraded":"2023-10-30T12:28:46.103197+0000","last_fullsized":"2023-10-30T12:28:46.103197+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:39:25.263228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107211+0000","last_change":"2023-10-30T12:28:44.975010+0000","last_active":"2023-10-30T12:28:46.107211+0000","last_peered":"2023-10-30T12:28:46.107211+0000","last_clean":"2023-10-30T12:28:46.107211+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T12:28:46.107211+0000","last_undegraded":"2023-10-30T12:28:46.107211+0000","last_fullsized":"2023-10-30T12:28:46.107211+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:21:13.768876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104243+0000","last_change":"2023-10-30T12:28:44.951099+0000","last_active":"2023-10-30T12:28:46.104243+0000","last_peered":"2023-10-30T12:28:46.104243+0000","last_clean":"2023-10-30T12:28:46.104243+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T12:28:46.104243+0000","last_undegraded":"2023-10-30T12:28:46.104243+0000","last_fullsized":"2023-10-30T12:28:46.104243+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:21:27.902671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946065+0000","last_change":"2023-10-30T12:28:44.959915+0000","last_active":"2023-10-30T12:28:45.946065+0000","last_peered":"2023-10-30T12:28:45.946065+0000","last_clean":"2023-10-30T12:28:45.946065+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T12:28:45.946065+0000","last_undegraded":"2023-10-30T12:28:45.946065+0000","last_fullsized":"2023-10-30T12:28:45.946065+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:57:49.711309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946417+0000","last_change":"2023-10-30T12:28:44.972677+0000","last_active":"2023-10-30T12:28:45.946417+0000","last_peered":"2023-10-30T12:28:45.946417+0000","last_clean":"2023-10-30T12:28:45.946417+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T12:28:45.946417+0000","last_undegraded":"2023-10-30T12:28:45.946417+0000","last_fullsized":"2023-10-30T12:28:45.946417+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:12.072355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106685+0000","last_change":"2023-10-30T12:28:44.975973+0000","last_active":"2023-10-30T12:28:46.106685+0000","last_peered":"2023-10-30T12:28:46.106685+0000","last_clean":"2023-10-30T12:28:46.106685+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T12:28:46.106685+0000","last_undegraded":"2023-10-30T12:28:46.106685+0000","last_fullsized":"2023-10-30T12:28:46.106685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:35:48.152823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103531+0000","last_change":"2023-10-30T12:28:44.980814+0000","last_active":"2023-10-30T12:28:46.103531+0000","last_peered":"2023-10-30T12:28:46.103531+0000","last_clean":"2023-10-30T12:28:46.103531+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T12:28:46.103531+0000","last_undegraded":"2023-10-30T12:28:46.103531+0000","last_fullsized":"2023-10-30T12:28:46.103531+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:51:22.453883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107798+0000","last_change":"2023-10-30T12:28:44.965026+0000","last_active":"2023-10-30T12:28:46.107798+0000","last_peered":"2023-10-30T12:28:46.107798+0000","last_clean":"2023-10-30T12:28:46.107798+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T12:28:46.107798+0000","last_undegraded":"2023-10-30T12:28:46.107798+0000","last_fullsized":"2023-10-30T12:28:46.107798+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:04:08.680744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.102782+0000","last_change":"2023-10-30T12:28:44.973401+0000","last_active":"2023-10-30T12:28:46.102782+0000","last_peered":"2023-10-30T12:28:46.102782+0000","last_clean":"2023-10-30T12:28:46.102782+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T12:28:46.102782+0000","last_undegraded":"2023-10-30T12:28:46.102782+0000","last_fullsized":"2023-10-30T12:28:46.102782+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:17:34.198424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939578+0000","last_change":"2023-10-30T12:28:44.971817+0000","last_active":"2023-10-30T12:28:45.939578+0000","last_peered":"2023-10-30T12:28:45.939578+0000","last_clean":"2023-10-30T12:28:45.939578+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T12:28:45.939578+0000","last_undegraded":"2023-10-30T12:28:45.939578+0000","last_fullsized":"2023-10-30T12:28:45.939578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:45:35.205305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"22'5","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.130160+0000","last_change":"2023-10-30T12:28:44.074842+0000","last_active":"2023-10-30T12:28:55.130160+0000","last_peered":"2023-10-30T12:28:55.130160+0000","last_clean":"2023-10-30T12:28:55.130160+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T12:28:55.130160+0000","last_undegraded":"2023-10-30T12:28:55.130160+0000","last_fullsized":"2023-10-30T12:28:55.130160+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:01.428785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945765+0000","last_change":"2023-10-30T12:28:44.958560+0000","last_active":"2023-10-30T12:28:45.945765+0000","last_peered":"2023-10-30T12:28:45.945765+0000","last_clean":"2023-10-30T12:28:45.945765+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T12:28:45.945765+0000","last_undegraded":"2023-10-30T12:28:45.945765+0000","last_fullsized":"2023-10-30T12:28:45.945765+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:42:45.564493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.963498+0000","last_change":"2023-10-30T12:28:44.060621+0000","last_active":"2023-10-30T12:29:00.963498+0000","last_peered":"2023-10-30T12:29:00.963498+0000","last_clean":"2023-10-30T12:29:00.963498+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T12:29:00.963498+0000","last_undegraded":"2023-10-30T12:29:00.963498+0000","last_fullsized":"2023-10-30T12:29:00.963498+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:53.741238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.967207+0000","last_change":"2023-10-30T12:28:44.060171+0000","last_active":"2023-10-30T12:29:00.967207+0000","last_peered":"2023-10-30T12:29:00.967207+0000","last_clean":"2023-10-30T12:29:00.967207+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T12:29:00.967207+0000","last_undegraded":"2023-10-30T12:29:00.967207+0000","last_fullsized":"2023-10-30T12:29:00.967207+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:18.181418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106893+0000","last_change":"2023-10-30T12:28:44.978631+0000","last_active":"2023-10-30T12:28:46.106893+0000","last_peered":"2023-10-30T12:28:46.106893+0000","last_clean":"2023-10-30T12:28:46.106893+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T12:28:46.106893+0000","last_undegraded":"2023-10-30T12:28:46.106893+0000","last_fullsized":"2023-10-30T12:28:46.106893+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:03:37.006427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:47.984674+0000","last_change":"2023-10-30T12:28:44.058798+0000","last_active":"2023-10-30T12:28:47.984674+0000","last_peered":"2023-10-30T12:28:47.984674+0000","last_clean":"2023-10-30T12:28:47.984674+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T12:28:47.984674+0000","last_undegraded":"2023-10-30T12:28:47.984674+0000","last_fullsized":"2023-10-30T12:28:47.984674+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:08.643080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945025+0000","last_change":"2023-10-30T12:28:44.949118+0000","last_active":"2023-10-30T12:28:45.945025+0000","last_peered":"2023-10-30T12:28:45.945025+0000","last_clean":"2023-10-30T12:28:45.945025+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T12:28:45.945025+0000","last_undegraded":"2023-10-30T12:28:45.945025+0000","last_fullsized":"2023-10-30T12:28:45.945025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:52:11.973878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.130828+0000","last_change":"2023-10-30T12:28:44.063439+0000","last_active":"2023-10-30T12:28:55.130828+0000","last_peered":"2023-10-30T12:28:55.130828+0000","last_clean":"2023-10-30T12:28:55.130828+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T12:28:55.130828+0000","last_undegraded":"2023-10-30T12:28:55.130828+0000","last_fullsized":"2023-10-30T12:28:55.130828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:08.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104375+0000","last_change":"2023-10-30T12:28:44.951319+0000","last_active":"2023-10-30T12:28:46.104375+0000","last_peered":"2023-10-30T12:28:46.104375+0000","last_clean":"2023-10-30T12:28:46.104375+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T12:28:46.104375+0000","last_undegraded":"2023-10-30T12:28:46.104375+0000","last_fullsized":"2023-10-30T12:28:46.104375+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:20.108360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.134075+0000","last_change":"2023-10-30T12:28:44.052641+0000","last_active":"2023-10-30T12:28:55.134075+0000","last_peered":"2023-10-30T12:28:55.134075+0000","last_clean":"2023-10-30T12:28:55.134075+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T12:28:55.134075+0000","last_undegraded":"2023-10-30T12:28:55.134075+0000","last_fullsized":"2023-10-30T12:28:55.134075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107640+0000","last_change":"2023-10-30T12:28:44.981614+0000","last_active":"2023-10-30T12:28:46.107640+0000","last_peered":"2023-10-30T12:28:46.107640+0000","last_clean":"2023-10-30T12:28:46.107640+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T12:28:46.107640+0000","last_undegraded":"2023-10-30T12:28:46.107640+0000","last_fullsized":"2023-10-30T12:28:46.107640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:15:11.750243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.068083+0000","last_change":"2023-10-30T12:28:44.075783+0000","last_active":"2023-10-30T12:28:50.068083+0000","last_peered":"2023-10-30T12:28:50.068083+0000","last_clean":"2023-10-30T12:28:50.068083+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T12:28:50.068083+0000","last_undegraded":"2023-10-30T12:28:50.068083+0000","last_fullsized":"2023-10-30T12:28:50.068083+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:01.094929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103142+0000","last_change":"2023-10-30T12:28:44.950899+0000","last_active":"2023-10-30T12:28:46.103142+0000","last_peered":"2023-10-30T12:28:46.103142+0000","last_clean":"2023-10-30T12:28:46.103142+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T12:28:46.103142+0000","last_undegraded":"2023-10-30T12:28:46.103142+0000","last_fullsized":"2023-10-30T12:28:46.103142+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:46:34.005863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"22'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.132373+0000","last_change":"2023-10-30T12:28:44.063367+0000","last_active":"2023-10-30T12:28:55.132373+0000","last_peered":"2023-10-30T12:28:55.132373+0000","last_clean":"2023-10-30T12:28:55.132373+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T12:28:55.132373+0000","last_undegraded":"2023-10-30T12:28:55.132373+0000","last_fullsized":"2023-10-30T12:28:55.132373+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:45.585121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104587+0000","last_change":"2023-10-30T12:28:44.972056+0000","last_active":"2023-10-30T12:28:46.104587+0000","last_peered":"2023-10-30T12:28:46.104587+0000","last_clean":"2023-10-30T12:28:46.104587+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T12:28:46.104587+0000","last_undegraded":"2023-10-30T12:28:46.104587+0000","last_fullsized":"2023-10-30T12:28:46.104587+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:06:08.345856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"22'6","reported_seq":18,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.966521+0000","last_change":"2023-10-30T12:28:44.065545+0000","last_active":"2023-10-30T12:29:00.966521+0000","last_peered":"2023-10-30T12:29:00.966521+0000","last_clean":"2023-10-30T12:29:00.966521+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T12:29:00.966521+0000","last_undegraded":"2023-10-30T12:29:00.966521+0000","last_fullsized":"2023-10-30T12:29:00.966521+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:09.900189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945563+0000","last_change":"2023-10-30T12:28:44.961237+0000","last_active":"2023-10-30T12:28:45.945563+0000","last_peered":"2023-10-30T12:28:45.945563+0000","last_clean":"2023-10-30T12:28:45.945563+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T12:28:45.945563+0000","last_undegraded":"2023-10-30T12:28:45.945563+0000","last_fullsized":"2023-10-30T12:28:45.945563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:17.160724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.130351+0000","last_change":"2023-10-30T12:28:44.061922+0000","last_active":"2023-10-30T12:28:55.130351+0000","last_peered":"2023-10-30T12:28:55.130351+0000","last_clean":"2023-10-30T12:28:55.130351+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T12:28:55.130351+0000","last_undegraded":"2023-10-30T12:28:55.130351+0000","last_fullsized":"2023-10-30T12:28:55.130351+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:34.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103258+0000","last_change":"2023-10-30T12:28:44.958245+0000","last_active":"2023-10-30T12:28:46.103258+0000","last_peered":"2023-10-30T12:28:46.103258+0000","last_clean":"2023-10-30T12:28:46.103258+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T12:28:46.103258+0000","last_undegraded":"2023-10-30T12:28:46.103258+0000","last_fullsized":"2023-10-30T12:28:46.103258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:42:10.840490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.964145+0000","last_change":"2023-10-30T12:28:44.059784+0000","last_active":"2023-10-30T12:29:00.964145+0000","last_peered":"2023-10-30T12:29:00.964145+0000","last_clean":"2023-10-30T12:29:00.964145+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T12:29:00.964145+0000","last_undegraded":"2023-10-30T12:29:00.964145+0000","last_fullsized":"2023-10-30T12:29:00.964145+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:11.468221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939230+0000","last_change":"2023-10-30T12:28:44.980813+0000","last_active":"2023-10-30T12:28:45.939230+0000","last_peered":"2023-10-30T12:28:45.939230+0000","last_clean":"2023-10-30T12:28:45.939230+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T12:28:45.939230+0000","last_undegraded":"2023-10-30T12:28:45.939230+0000","last_fullsized":"2023-10-30T12:28:45.939230+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:51:10.858970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"22'15","reported_seq":27,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.975410+0000","last_change":"2023-10-30T12:28:44.065885+0000","last_active":"2023-10-30T12:29:00.975410+0000","last_peered":"2023-10-30T12:29:00.975410+0000","last_clean":"2023-10-30T12:29:00.975410+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T12:29:00.975410+0000","last_undegraded":"2023-10-30T12:29:00.975410+0000","last_fullsized":"2023-10-30T12:29:00.975410+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:26.313595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103080+0000","last_change":"2023-10-30T12:28:44.952458+0000","last_active":"2023-10-30T12:28:46.103080+0000","last_peered":"2023-10-30T12:28:46.103080+0000","last_clean":"2023-10-30T12:28:46.103080+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T12:28:46.103080+0000","last_undegraded":"2023-10-30T12:28:46.103080+0000","last_fullsized":"2023-10-30T12:28:46.103080+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:48:19.752492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.062786+0000","last_change":"2023-10-30T12:28:44.080011+0000","last_active":"2023-10-30T12:28:50.062786+0000","last_peered":"2023-10-30T12:28:50.062786+0000","last_clean":"2023-10-30T12:28:50.062786+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T12:28:50.062786+0000","last_undegraded":"2023-10-30T12:28:50.062786+0000","last_fullsized":"2023-10-30T12:28:50.062786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104494+0000","last_change":"2023-10-30T12:28:44.961296+0000","last_active":"2023-10-30T12:28:46.104494+0000","last_peered":"2023-10-30T12:28:46.104494+0000","last_clean":"2023-10-30T12:28:46.104494+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T12:28:46.104494+0000","last_undegraded":"2023-10-30T12:28:46.104494+0000","last_fullsized":"2023-10-30T12:28:46.104494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:02:56.540392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.066004+0000","last_change":"2023-10-30T12:28:44.076352+0000","last_active":"2023-10-30T12:28:50.066004+0000","last_peered":"2023-10-30T12:28:50.066004+0000","last_clean":"2023-10-30T12:28:50.066004+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T12:28:50.066004+0000","last_undegraded":"2023-10-30T12:28:50.066004+0000","last_fullsized":"2023-10-30T12:28:50.066004+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104918+0000","last_change":"2023-10-30T12:28:44.956918+0000","last_active":"2023-10-30T12:28:46.104918+0000","last_peered":"2023-10-30T12:28:46.104918+0000","last_clean":"2023-10-30T12:28:46.104918+0000","last_became_active":"2023-10-30T12:28:44.956397+0000","last_became_peered":"2023-10-30T12:28:44.956397+0000","last_unstale":"2023-10-30T12:28:46.104918+0000","last_undegraded":"2023-10-30T12:28:46.104918+0000","last_fullsized":"2023-10-30T12:28:46.104918+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:27:26.252196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106813+0000","last_change":"2023-10-30T12:28:44.075684+0000","last_active":"2023-10-30T12:28:46.106813+0000","last_peered":"2023-10-30T12:28:46.106813+0000","last_clean":"2023-10-30T12:28:46.106813+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T12:28:46.106813+0000","last_undegraded":"2023-10-30T12:28:46.106813+0000","last_fullsized":"2023-10-30T12:28:46.106813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:07.375795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104588+0000","last_change":"2023-10-30T12:28:44.956047+0000","last_active":"2023-10-30T12:28:46.104588+0000","last_peered":"2023-10-30T12:28:46.104588+0000","last_clean":"2023-10-30T12:28:46.104588+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T12:28:46.104588+0000","last_undegraded":"2023-10-30T12:28:46.104588+0000","last_fullsized":"2023-10-30T12:28:46.104588+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:16:49.924947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.970394+0000","last_change":"2023-10-30T12:28:44.060686+0000","last_active":"2023-10-30T12:29:00.970394+0000","last_peered":"2023-10-30T12:29:00.970394+0000","last_clean":"2023-10-30T12:29:00.970394+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T12:29:00.970394+0000","last_undegraded":"2023-10-30T12:29:00.970394+0000","last_fullsized":"2023-10-30T12:29:00.970394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:23.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103562+0000","last_change":"2023-10-30T12:28:44.946758+0000","last_active":"2023-10-30T12:28:46.103562+0000","last_peered":"2023-10-30T12:28:46.103562+0000","last_clean":"2023-10-30T12:28:46.103562+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T12:28:46.103562+0000","last_undegraded":"2023-10-30T12:28:46.103562+0000","last_fullsized":"2023-10-30T12:28:46.103562+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:16:20.520148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:44.932509+0000","last_change":"2023-10-30T12:28:33.433824+0000","last_active":"2023-10-30T12:28:44.932509+0000","last_peered":"2023-10-30T12:28:44.932509+0000","last_clean":"2023-10-30T12:28:44.932509+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T12:28:44.932509+0000","last_undegraded":"2023-10-30T12:28:44.932509+0000","last_fullsized":"2023-10-30T12:28:44.932509+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_clean_scrub_stamp":"2023-10-30T12:28:31.408255+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:17.624179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.132911+0000","last_change":"2023-10-30T12:28:44.059051+0000","last_active":"2023-10-30T12:28:55.132911+0000","last_peered":"2023-10-30T12:28:55.132911+0000","last_clean":"2023-10-30T12:28:55.132911+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T12:28:55.132911+0000","last_undegraded":"2023-10-30T12:28:55.132911+0000","last_fullsized":"2023-10-30T12:28:55.132911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:54.810636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.945062+0000","last_change":"2023-10-30T12:28:44.965183+0000","last_active":"2023-10-30T12:28:45.945062+0000","last_peered":"2023-10-30T12:28:45.945062+0000","last_clean":"2023-10-30T12:28:45.945062+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T12:28:45.945062+0000","last_undegraded":"2023-10-30T12:28:45.945062+0000","last_fullsized":"2023-10-30T12:28:45.945062+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:48:57.655687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.962924+0000","last_change":"2023-10-30T12:28:44.080378+0000","last_active":"2023-10-30T12:29:00.962924+0000","last_peered":"2023-10-30T12:29:00.962924+0000","last_clean":"2023-10-30T12:29:00.962924+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T12:29:00.962924+0000","last_undegraded":"2023-10-30T12:29:00.962924+0000","last_fullsized":"2023-10-30T12:29:00.962924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:22.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944927+0000","last_change":"2023-10-30T12:28:44.961659+0000","last_active":"2023-10-30T12:28:45.944927+0000","last_peered":"2023-10-30T12:28:45.944927+0000","last_clean":"2023-10-30T12:28:45.944927+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T12:28:45.944927+0000","last_undegraded":"2023-10-30T12:28:45.944927+0000","last_fullsized":"2023-10-30T12:28:45.944927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:34:00.000084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.969477+0000","last_change":"2023-10-30T12:28:44.082480+0000","last_active":"2023-10-30T12:29:00.969477+0000","last_peered":"2023-10-30T12:29:00.969477+0000","last_clean":"2023-10-30T12:29:00.969477+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T12:29:00.969477+0000","last_undegraded":"2023-10-30T12:29:00.969477+0000","last_fullsized":"2023-10-30T12:29:00.969477+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:20.126782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.106385+0000","last_change":"2023-10-30T12:28:44.973906+0000","last_active":"2023-10-30T12:28:46.106385+0000","last_peered":"2023-10-30T12:28:46.106385+0000","last_clean":"2023-10-30T12:28:46.106385+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T12:28:46.106385+0000","last_undegraded":"2023-10-30T12:28:46.106385+0000","last_fullsized":"2023-10-30T12:28:46.106385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:28:31.845894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:47.984766+0000","last_change":"2023-10-30T12:28:44.059806+0000","last_active":"2023-10-30T12:28:47.984766+0000","last_peered":"2023-10-30T12:28:47.984766+0000","last_clean":"2023-10-30T12:28:47.984766+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T12:28:47.984766+0000","last_undegraded":"2023-10-30T12:28:47.984766+0000","last_fullsized":"2023-10-30T12:28:47.984766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:50.939946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.104302+0000","last_change":"2023-10-30T12:28:44.955423+0000","last_active":"2023-10-30T12:28:46.104302+0000","last_peered":"2023-10-30T12:28:46.104302+0000","last_clean":"2023-10-30T12:28:46.104302+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T12:28:46.104302+0000","last_undegraded":"2023-10-30T12:28:46.104302+0000","last_fullsized":"2023-10-30T12:28:46.104302+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:30.054048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.982937+0000","last_change":"2023-10-30T12:28:44.074970+0000","last_active":"2023-10-30T12:28:45.982937+0000","last_peered":"2023-10-30T12:28:45.982937+0000","last_clean":"2023-10-30T12:28:45.982937+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T12:28:45.982937+0000","last_undegraded":"2023-10-30T12:28:45.982937+0000","last_fullsized":"2023-10-30T12:28:45.982937+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:33.562703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103885+0000","last_change":"2023-10-30T12:28:44.973482+0000","last_active":"2023-10-30T12:28:46.103885+0000","last_peered":"2023-10-30T12:28:46.103885+0000","last_clean":"2023-10-30T12:28:46.103885+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T12:28:46.103885+0000","last_undegraded":"2023-10-30T12:28:46.103885+0000","last_fullsized":"2023-10-30T12:28:46.103885+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:18:25.393161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:47.984860+0000","last_change":"2023-10-30T12:28:44.063256+0000","last_active":"2023-10-30T12:28:47.984860+0000","last_peered":"2023-10-30T12:28:47.984860+0000","last_clean":"2023-10-30T12:28:47.984860+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T12:28:47.984860+0000","last_undegraded":"2023-10-30T12:28:47.984860+0000","last_fullsized":"2023-10-30T12:28:47.984860+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:01:36.144966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946945+0000","last_change":"2023-10-30T12:28:44.975443+0000","last_active":"2023-10-30T12:28:45.946945+0000","last_peered":"2023-10-30T12:28:45.946945+0000","last_clean":"2023-10-30T12:28:45.946945+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T12:28:45.946945+0000","last_undegraded":"2023-10-30T12:28:45.946945+0000","last_fullsized":"2023-10-30T12:28:45.946945+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:08:23.765723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.968669+0000","last_change":"2023-10-30T12:28:44.055374+0000","last_active":"2023-10-30T12:29:00.968669+0000","last_peered":"2023-10-30T12:29:00.968669+0000","last_clean":"2023-10-30T12:29:00.968669+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T12:29:00.968669+0000","last_undegraded":"2023-10-30T12:29:00.968669+0000","last_fullsized":"2023-10-30T12:29:00.968669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946991+0000","last_change":"2023-10-30T12:28:44.974359+0000","last_active":"2023-10-30T12:28:45.946991+0000","last_peered":"2023-10-30T12:28:45.946991+0000","last_clean":"2023-10-30T12:28:45.946991+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T12:28:45.946991+0000","last_undegraded":"2023-10-30T12:28:45.946991+0000","last_fullsized":"2023-10-30T12:28:45.946991+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:49:48.189126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.132893+0000","last_change":"2023-10-30T12:28:44.058663+0000","last_active":"2023-10-30T12:28:55.132893+0000","last_peered":"2023-10-30T12:28:55.132893+0000","last_clean":"2023-10-30T12:28:55.132893+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T12:28:55.132893+0000","last_undegraded":"2023-10-30T12:28:55.132893+0000","last_fullsized":"2023-10-30T12:28:55.132893+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:42:22.279653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.946815+0000","last_change":"2023-10-30T12:28:44.957471+0000","last_active":"2023-10-30T12:28:45.946815+0000","last_peered":"2023-10-30T12:28:45.946815+0000","last_clean":"2023-10-30T12:28:45.946815+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T12:28:45.946815+0000","last_undegraded":"2023-10-30T12:28:45.946815+0000","last_fullsized":"2023-10-30T12:28:45.946815+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:27:38.669452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.131577+0000","last_change":"2023-10-30T12:28:44.054338+0000","last_active":"2023-10-30T12:28:55.131577+0000","last_peered":"2023-10-30T12:28:55.131577+0000","last_clean":"2023-10-30T12:28:55.131577+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T12:28:55.131577+0000","last_undegraded":"2023-10-30T12:28:55.131577+0000","last_fullsized":"2023-10-30T12:28:55.131577+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:55:35.370652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103200+0000","last_change":"2023-10-30T12:28:44.965759+0000","last_active":"2023-10-30T12:28:46.103200+0000","last_peered":"2023-10-30T12:28:46.103200+0000","last_clean":"2023-10-30T12:28:46.103200+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T12:28:46.103200+0000","last_undegraded":"2023-10-30T12:28:46.103200+0000","last_fullsized":"2023-10-30T12:28:46.103200+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:13:52.965500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"22'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.134624+0000","last_change":"2023-10-30T12:28:44.064606+0000","last_active":"2023-10-30T12:28:55.134624+0000","last_peered":"2023-10-30T12:28:55.134624+0000","last_clean":"2023-10-30T12:28:55.134624+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T12:28:55.134624+0000","last_undegraded":"2023-10-30T12:28:55.134624+0000","last_fullsized":"2023-10-30T12:28:55.134624+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.944678+0000","last_change":"2023-10-30T12:28:44.958947+0000","last_active":"2023-10-30T12:28:45.944678+0000","last_peered":"2023-10-30T12:28:45.944678+0000","last_clean":"2023-10-30T12:28:45.944678+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T12:28:45.944678+0000","last_undegraded":"2023-10-30T12:28:45.944678+0000","last_fullsized":"2023-10-30T12:28:45.944678+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:01:16.329994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.979140+0000","last_change":"2023-10-30T12:28:44.072754+0000","last_active":"2023-10-30T12:28:46.979140+0000","last_peered":"2023-10-30T12:28:46.979140+0000","last_clean":"2023-10-30T12:28:46.979140+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T12:28:46.979140+0000","last_undegraded":"2023-10-30T12:28:46.979140+0000","last_fullsized":"2023-10-30T12:28:46.979140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:51:55.865069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.108063+0000","last_change":"2023-10-30T12:28:44.978376+0000","last_active":"2023-10-30T12:28:46.108063+0000","last_peered":"2023-10-30T12:28:46.108063+0000","last_clean":"2023-10-30T12:28:46.108063+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T12:28:46.108063+0000","last_undegraded":"2023-10-30T12:28:46.108063+0000","last_fullsized":"2023-10-30T12:28:46.108063+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:58:59.643447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:55.131363+0000","last_change":"2023-10-30T12:28:44.058801+0000","last_active":"2023-10-30T12:28:55.131363+0000","last_peered":"2023-10-30T12:28:55.131363+0000","last_clean":"2023-10-30T12:28:55.131363+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T12:28:55.131363+0000","last_undegraded":"2023-10-30T12:28:55.131363+0000","last_fullsized":"2023-10-30T12:28:55.131363+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:11:24.706813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103647+0000","last_change":"2023-10-30T12:28:44.943292+0000","last_active":"2023-10-30T12:28:46.103647+0000","last_peered":"2023-10-30T12:28:46.103647+0000","last_clean":"2023-10-30T12:28:46.103647+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T12:28:46.103647+0000","last_undegraded":"2023-10-30T12:28:46.103647+0000","last_fullsized":"2023-10-30T12:28:46.103647+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:37.538525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.103472+0000","last_change":"2023-10-30T12:28:44.958607+0000","last_active":"2023-10-30T12:28:46.103472+0000","last_peered":"2023-10-30T12:28:46.103472+0000","last_clean":"2023-10-30T12:28:46.103472+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T12:28:46.103472+0000","last_undegraded":"2023-10-30T12:28:46.103472+0000","last_fullsized":"2023-10-30T12:28:46.103472+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:28:55.169198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.059856+0000","last_change":"2023-10-30T12:28:44.054652+0000","last_active":"2023-10-30T12:28:50.059856+0000","last_peered":"2023-10-30T12:28:50.059856+0000","last_clean":"2023-10-30T12:28:50.059856+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T12:28:50.059856+0000","last_undegraded":"2023-10-30T12:28:50.059856+0000","last_fullsized":"2023-10-30T12:28:50.059856+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:05.023374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:45.939383+0000","last_change":"2023-10-30T12:28:44.958246+0000","last_active":"2023-10-30T12:28:45.939383+0000","last_peered":"2023-10-30T12:28:45.939383+0000","last_clean":"2023-10-30T12:28:45.939383+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T12:28:45.939383+0000","last_undegraded":"2023-10-30T12:28:45.939383+0000","last_fullsized":"2023-10-30T12:28:45.939383+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:13:36.947398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"22'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.963915+0000","last_change":"2023-10-30T12:28:44.052633+0000","last_active":"2023-10-30T12:29:00.963915+0000","last_peered":"2023-10-30T12:29:00.963915+0000","last_clean":"2023-10-30T12:29:00.963915+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T12:29:00.963915+0000","last_undegraded":"2023-10-30T12:29:00.963915+0000","last_fullsized":"2023-10-30T12:29:00.963915+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107324+0000","last_change":"2023-10-30T12:28:44.974457+0000","last_active":"2023-10-30T12:28:46.107324+0000","last_peered":"2023-10-30T12:28:46.107324+0000","last_clean":"2023-10-30T12:28:46.107324+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T12:28:46.107324+0000","last_undegraded":"2023-10-30T12:28:46.107324+0000","last_fullsized":"2023-10-30T12:28:46.107324+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:25:56.791053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"22'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:50.064135+0000","last_change":"2023-10-30T12:28:44.075493+0000","last_active":"2023-10-30T12:28:50.064135+0000","last_peered":"2023-10-30T12:28:50.064135+0000","last_clean":"2023-10-30T12:28:50.064135+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T12:28:50.064135+0000","last_undegraded":"2023-10-30T12:28:50.064135+0000","last_fullsized":"2023-10-30T12:28:50.064135+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:00:11.718669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:28:46.107054+0000","last_change":"2023-10-30T12:28:44.974424+0000","last_active":"2023-10-30T12:28:46.107054+0000","last_peered":"2023-10-30T12:28:46.107054+0000","last_clean":"2023-10-30T12:28:46.107054+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T12:28:46.107054+0000","last_undegraded":"2023-10-30T12:28:46.107054+0000","last_fullsized":"2023-10-30T12:28:46.107054+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:14.375671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"22'9","reported_seq":21,"reported_epoch":22,"state":"active+clean","last_fresh":"2023-10-30T12:29:00.971551+0000","last_change":"2023-10-30T12:28:44.065843+0000","last_active":"2023-10-30T12:29:00.971551+0000","last_peered":"2023-10-30T12:29:00.971551+0000","last_clean":"2023-10-30T12:29:00.971551+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T12:29:00.971551+0000","last_undegraded":"2023-10-30T12:29:00.971551+0000","last_fullsized":"2023-10-30T12:29:00.971551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:32.405199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5222,"num_objects":60,"num_object_clones":0,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":125,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":11689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":109,"ondisk_log_size":109,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719476748,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":77094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.441}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63800000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79100000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]}]},{"osd":2,"up_from":16,"seq":68719476748,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":77514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93000000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.262}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94899999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88400000000000001}]}]},{"osd":1,"up_from":16,"seq":68719476746,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344024,"statfs":{"total":96636764160,"available":96608280576,"internally_reserved":0,"allocated":819200,"data_stored":666396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91400000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.002}]}]},{"osd":0,"up_from":16,"seq":68719476747,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":77184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.501}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]}]},{"osd":6,"up_from":15,"seq":64424509452,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31400,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340440,"statfs":{"total":96636764160,"available":96604610560,"internally_reserved":0,"allocated":229376,"data_stored":77072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55300000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0669999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509453,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31432,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340408,"statfs":{"total":96636764160,"available":96604577792,"internally_reserved":0,"allocated":262144,"data_stored":80293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0189999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95999999999999996}]}]},{"osd":7,"up_from":14,"seq":60129542155,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27252,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344588,"statfs":{"total":96636764160,"available":96608858112,"internally_reserved":0,"allocated":241664,"data_stored":76776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88700000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.01}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.042}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1919999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542154,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":831488,"data_stored":667040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":4425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T12:29:14.679 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:29:14.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a8a91640 1 Processor -- start 2023-10-30T12:29:14.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a8a91640 1 -- start start 2023-10-30T12:29:14.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a8a91640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f72a40cf160 0x7f72a40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a8a91640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a40cb800 con 0x7f72a41538a0 2023-10-30T12:29:14.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a8a91640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72a40cb330 con 0x7f72a415d6f0 2023-10-30T12:29:14.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a8a91640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72a40cad70 con 0x7f72a40cf160 2023-10-30T12:29:14.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a259c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f72a41538a0 0x7f72a40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60090/0 (socket says 172.21.15.125:60090) 2023-10-30T12:29:14.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a259c640 1 -- 172.21.15.125:0/319830357 learned_addr learned my addr 172.21.15.125:0/319830357 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:29:14.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a2d9d640 1 --2- 172.21.15.125:0/319830357 >> v2:172.21.15.136:3300/0 conn(0x7f72a40cf160 0x7f72a40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 -- 172.21.15.125:0/319830357 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986299413 0 0) 0x7f72a40cb800 con 0x7f72a41538a0 2023-10-30T12:29:14.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 -- 172.21.15.125:0/319830357 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f728c003530 con 0x7f72a41538a0 2023-10-30T12:29:14.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 -- 172.21.15.125:0/319830357 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3206871263 0 0) 0x7f728c003530 con 0x7f72a41538a0 2023-10-30T12:29:14.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 -- 172.21.15.125:0/319830357 >> v2:172.21.15.136:3300/0 conn(0x7f72a40cf160 msgr2=0x7f72a40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:14.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 --2- 172.21.15.125:0/319830357 >> v2:172.21.15.136:3300/0 conn(0x7f72a40cf160 0x7f72a40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 -- 172.21.15.125:0/319830357 >> v1:172.21.15.136:6789/0 conn(0x7f72a415d6f0 legacy=0x7f72a415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 -- 172.21.15.125:0/319830357 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a40cb050 con 0x7f72a41538a0 2023-10-30T12:29:14.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 -- 172.21.15.125:0/319830357 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f7294001680 con 0x7f72a41538a0 2023-10-30T12:29:14.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 -- 172.21.15.125:0/319830357 >> v1:172.21.15.125:6789/0 conn(0x7f72a41538a0 legacy=0x7f72a40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 --2- 172.21.15.125:0/319830357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f728c003980 0x7f728c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 --2- 172.21.15.125:0/319830357 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f728c0047d0 0x7f728c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 --2- 172.21.15.125:0/319830357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f728c005190 0x7f728c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 -- 172.21.15.125:0/319830357 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a40cad70 con 0x7f728c003980 2023-10-30T12:29:14.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 -- 172.21.15.125:0/319830357 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f728c00c040 con 0x7f728c005190 2023-10-30T12:29:14.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a159a640 1 -- 172.21.15.125:0/319830357 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f728c00c200 con 0x7f728c0047d0 2023-10-30T12:29:14.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a259c640 1 --2- 172.21.15.125:0/319830357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f728c003980 0x7f728c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a1d9b640 1 --2- 172.21.15.125:0/319830357 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f728c0047d0 0x7f728c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a2d9d640 1 --2- 172.21.15.125:0/319830357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f728c005190 0x7f728c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a259c640 1 -- 172.21.15.125:0/319830357 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f728c0047d0 msgr2=0x7f728c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:14.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a259c640 1 --2- 172.21.15.125:0/319830357 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f728c0047d0 0x7f728c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a259c640 1 -- 172.21.15.125:0/319830357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f728c005190 msgr2=0x7f728c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:14.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a259c640 1 --2- 172.21.15.125:0/319830357 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f728c005190 0x7f728c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a259c640 1 -- 172.21.15.125:0/319830357 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f728c003530 con 0x7f728c003980 2023-10-30T12:29:14.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.855+0000 7f72a2d9d640 1 -- 172.21.15.125:0/319830357 reap_dead start 2023-10-30T12:29:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.859+0000 7f72a259c640 1 --2- 172.21.15.125:0/319830357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f728c003980 0x7f728c003da0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f729400c240 tx=0x7f729401f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=250fbf22c55c90c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.859+0000 7f72a159a640 1 -- 172.21.15.125:0/319830357 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7294005ae0 con 0x7f728c003980 2023-10-30T12:29:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.859+0000 7f72a159a640 1 -- 172.21.15.125:0/319830357 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72940ab070 con 0x7f728c003980 2023-10-30T12:29:14.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.859+0000 7f72a159a640 1 -- 172.21.15.125:0/319830357 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7294007070 con 0x7f728c003980 2023-10-30T12:29:14.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.859+0000 7f72a8a91640 1 -- 172.21.15.125:0/319830357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f728c003980 msgr2=0x7f728c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.859+0000 7f72a8a91640 1 --2- 172.21.15.125:0/319830357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f728c003980 0x7f728c003da0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f729400c240 tx=0x7f729401f370 comp rx=0 tx=0).stop 2023-10-30T12:29:14.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.859+0000 7f72a8a91640 1 -- 172.21.15.125:0/319830357 shutdown_connections 2023-10-30T12:29:14.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.859+0000 7f72a8a91640 1 --2- 172.21.15.125:0/319830357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f728c003980 0x7f728c003da0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.859+0000 7f72a8a91640 1 -- 172.21.15.125:0/319830357 >> 172.21.15.125:0/319830357 conn(0x7f72a406bfe0 msgr2=0x7f72a406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:14.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.859+0000 7f72a8a91640 1 -- 172.21.15.125:0/319830357 shutdown_connections 2023-10-30T12:29:14.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.859+0000 7f72a8a91640 1 -- 172.21.15.125:0/319830357 wait complete. 2023-10-30T12:29:14.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a8a91640 1 Processor -- start 2023-10-30T12:29:14.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a8a91640 1 -- start start 2023-10-30T12:29:14.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a8a91640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f728c003980 0x7f72a4146420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a8a91640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72a40b3a40 0x7f72a415c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a8a91640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72a41ef0f0 0x7f72a41ef580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a8a91640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a40b4070 con 0x7f72a40b3a40 2023-10-30T12:29:14.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a8a91640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a41f35c0 con 0x7f72a41ef0f0 2023-10-30T12:29:14.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a8a91640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72a41f3130 con 0x7f728c003980 2023-10-30T12:29:14.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a1d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72a40b3a40 0x7f72a415c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a1d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72a40b3a40 0x7f72a415c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40020/0 (socket says 172.21.15.125:40020) 2023-10-30T12:29:14.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a1d9b640 1 -- 172.21.15.125:0/3594680103 learned_addr learned my addr 172.21.15.125:0/3594680103 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:14.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a259c640 1 --2- 172.21.15.125:0/3594680103 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f728c003980 0x7f72a4146420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a2d9d640 1 --2- 172.21.15.125:0/3594680103 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72a41ef0f0 0x7f72a41ef580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a1d9b640 1 -- 172.21.15.125:0/3594680103 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f728c003980 msgr2=0x7f72a4146420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a1d9b640 1 --2- 172.21.15.125:0/3594680103 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f728c003980 0x7f72a4146420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a1d9b640 1 -- 172.21.15.125:0/3594680103 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72a41ef0f0 msgr2=0x7f72a41ef580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:14.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a1d9b640 1 --2- 172.21.15.125:0/3594680103 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72a41ef0f0 0x7f72a41ef580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:14.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a1d9b640 1 -- 172.21.15.125:0/3594680103 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4146ad0 con 0x7f72a40b3a40 2023-10-30T12:29:14.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a1d9b640 1 --2- 172.21.15.125:0/3594680103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72a40b3a40 0x7f72a415c050 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7298001ec0 tx=0x7f729800e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:14.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f7292ffd640 1 -- 172.21.15.125:0/3594680103 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72980052e0 con 0x7f72a40b3a40 2023-10-30T12:29:14.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f7292ffd640 1 -- 172.21.15.125:0/3594680103 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7298005440 con 0x7f72a40b3a40 2023-10-30T12:29:14.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f7292ffd640 1 -- 172.21.15.125:0/3594680103 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7298003820 con 0x7f72a40b3a40 2023-10-30T12:29:14.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a8a91640 1 -- 172.21.15.125:0/3594680103 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a41f2800 con 0x7f72a40b3a40 2023-10-30T12:29:14.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.863+0000 7f72a8a91640 1 -- 172.21.15.125:0/3594680103 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a41f2dc0 con 0x7f72a40b3a40 2023-10-30T12:29:14.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.867+0000 7f72a8a91640 1 -- 172.21.15.125:0/3594680103 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a41f2f80 con 0x7f72a40b3a40 2023-10-30T12:29:14.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.867+0000 7f7292ffd640 1 -- 172.21.15.125:0/3594680103 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7298003980 con 0x7f72a40b3a40 2023-10-30T12:29:14.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.867+0000 7f7292ffd640 1 --2- 172.21.15.125:0/3594680103 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72840664a0 0x7f7284068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:14.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.867+0000 7f7292ffd640 1 -- 172.21.15.125:0/3594680103 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7298083750 con 0x7f72a40b3a40 2023-10-30T12:29:14.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.867+0000 7f72a259c640 1 --2- 172.21.15.125:0/3594680103 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72840664a0 0x7f7284068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:14.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.867+0000 7f72a259c640 1 --2- 172.21.15.125:0/3594680103 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72840664a0 0x7f7284068960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f72940af000 tx=0x7f72940aa000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:14.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:14.871+0000 7f7292ffd640 1 -- 172.21.15.125:0/3594680103 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7298050b50 con 0x7f72a40b3a40 2023-10-30T12:29:15.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.023+0000 7f72a8a91640 1 -- 172.21.15.125:0/3594680103 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72a41f32a0 con 0x7f72a40b3a40 2023-10-30T12:29:15.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.023+0000 7f7292ffd640 1 -- 172.21.15.125:0/3594680103 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16557 (secure 0 0 0) 0x7f7298019090 con 0x7f72a40b3a40 2023-10-30T12:29:15.027 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:29:15.028 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:45.927862+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:29:15.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.027+0000 7f72a8a91640 1 -- 172.21.15.125:0/3594680103 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72840664a0 msgr2=0x7f7284068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.027+0000 7f72a8a91640 1 --2- 172.21.15.125:0/3594680103 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72840664a0 0x7f7284068960 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f72940af000 tx=0x7f72940aa000 comp rx=0 tx=0).stop 2023-10-30T12:29:15.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.027+0000 7f72a8a91640 1 -- 172.21.15.125:0/3594680103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72a40b3a40 msgr2=0x7f72a415c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.027+0000 7f72a8a91640 1 --2- 172.21.15.125:0/3594680103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72a40b3a40 0x7f72a415c050 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7298001ec0 tx=0x7f729800e8a0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.027+0000 7f72a8a91640 1 -- 172.21.15.125:0/3594680103 shutdown_connections 2023-10-30T12:29:15.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.027+0000 7f72a8a91640 1 --2- 172.21.15.125:0/3594680103 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72840664a0 0x7f7284068960 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.027+0000 7f72a8a91640 1 --2- 172.21.15.125:0/3594680103 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72a41ef0f0 0x7f72a41ef580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.027+0000 7f72a8a91640 1 --2- 172.21.15.125:0/3594680103 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72a40b3a40 0x7f72a415c050 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.027+0000 7f72a8a91640 1 --2- 172.21.15.125:0/3594680103 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f728c003980 0x7f72a4146420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.027+0000 7f72a8a91640 1 -- 172.21.15.125:0/3594680103 >> 172.21.15.125:0/3594680103 conn(0x7f72a406bfe0 msgr2=0x7f72a40c4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:15.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.027+0000 7f72a8a91640 1 -- 172.21.15.125:0/3594680103 shutdown_connections 2023-10-30T12:29:15.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.027+0000 7f72a8a91640 1 -- 172.21.15.125:0/3594680103 wait complete. 2023-10-30T12:29:15.049 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:31.063403+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T12:28:43.007849+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T12:28:43.401231+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T12:29:15.050 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T12:29:15.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.187+0000 7f2fa359e640 1 Processor -- start 2023-10-30T12:29:15.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.187+0000 7f2fa359e640 1 -- start start 2023-10-30T12:29:15.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa359e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2fa40cb4c0 0x7f2fa415a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa359e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa415a6b0 con 0x7f2fa4153840 2023-10-30T12:29:15.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa359e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa415d730 con 0x7f2fa4153c20 2023-10-30T12:29:15.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa359e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fa415dbc0 con 0x7f2fa40cb4c0 2023-10-30T12:29:15.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa259c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2fa4153840 0x7f2fa40c3eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60098/0 (socket says 172.21.15.125:60098) 2023-10-30T12:29:15.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa259c640 1 -- 172.21.15.125:0/886468627 learned_addr learned my addr 172.21.15.125:0/886468627 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:29:15.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa2d9d640 1 --2- 172.21.15.125:0/886468627 >> v2:172.21.15.136:3300/0 conn(0x7f2fa40cb4c0 0x7f2fa415a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 -- 172.21.15.125:0/886468627 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309916599 0 0) 0x7f2f94001120 con 0x7f2fa4153840 2023-10-30T12:29:15.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 -- 172.21.15.125:0/886468627 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f90003530 con 0x7f2fa4153840 2023-10-30T12:29:15.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 -- 172.21.15.125:0/886468627 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2596514715 0 0) 0x7f2f90003530 con 0x7f2fa4153840 2023-10-30T12:29:15.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 -- 172.21.15.125:0/886468627 >> v2:172.21.15.136:3300/0 conn(0x7f2fa40cb4c0 msgr2=0x7f2fa415a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:15.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 --2- 172.21.15.125:0/886468627 >> v2:172.21.15.136:3300/0 conn(0x7f2fa40cb4c0 0x7f2fa415a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 -- 172.21.15.125:0/886468627 >> v1:172.21.15.136:6789/0 conn(0x7f2fa4153c20 legacy=0x7f2fa40cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 -- 172.21.15.125:0/886468627 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa40ca8f0 con 0x7f2fa4153840 2023-10-30T12:29:15.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 -- 172.21.15.125:0/886468627 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f2f94001810 con 0x7f2fa4153840 2023-10-30T12:29:15.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 -- 172.21.15.125:0/886468627 >> v1:172.21.15.125:6789/0 conn(0x7f2fa4153840 legacy=0x7f2fa40c3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 --2- 172.21.15.125:0/886468627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f90003980 0x7f2f90003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 --2- 172.21.15.125:0/886468627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f900047d0 0x7f2f90004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 --2- 172.21.15.125:0/886468627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f90005190 0x7f2f9000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 -- 172.21.15.125:0/886468627 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa415dbc0 con 0x7f2f90005190 2023-10-30T12:29:15.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 -- 172.21.15.125:0/886468627 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9000c040 con 0x7f2f900047d0 2023-10-30T12:29:15.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa1d9b640 1 -- 172.21.15.125:0/886468627 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9000c200 con 0x7f2f90003980 2023-10-30T12:29:15.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa2d9d640 1 --2- 172.21.15.125:0/886468627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f90005190 0x7f2f9000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2f99d9b640 1 --2- 172.21.15.125:0/886468627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f900047d0 0x7f2f90004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa259c640 1 --2- 172.21.15.125:0/886468627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f90003980 0x7f2f90003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa2d9d640 1 -- 172.21.15.125:0/886468627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f90003980 msgr2=0x7f2f90003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:15.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa2d9d640 1 --2- 172.21.15.125:0/886468627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f90003980 0x7f2f90003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa2d9d640 1 -- 172.21.15.125:0/886468627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f900047d0 msgr2=0x7f2f90004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:15.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa2d9d640 1 --2- 172.21.15.125:0/886468627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f900047d0 0x7f2f90004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.191+0000 7f2fa2d9d640 1 -- 172.21.15.125:0/886468627 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f90003530 con 0x7f2f90005190 2023-10-30T12:29:15.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.195+0000 7f2fa2d9d640 1 -- 172.21.15.125:0/886468627 reap_dead start 2023-10-30T12:29:15.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.195+0000 7f2fa2d9d640 1 --2- 172.21.15.125:0/886468627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f90005190 0x7f2f9000b990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009fc0 tx=0x7f2f9c01cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ccfc17cc41154c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.195+0000 7f2fa1d9b640 1 -- 172.21.15.125:0/886468627 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c0a5cb0 con 0x7f2f90005190 2023-10-30T12:29:15.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.195+0000 7f2fa1d9b640 1 -- 172.21.15.125:0/886468627 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c00f040 con 0x7f2f90005190 2023-10-30T12:29:15.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.195+0000 7f2fa1d9b640 1 -- 172.21.15.125:0/886468627 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c003be0 con 0x7f2f90005190 2023-10-30T12:29:15.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.195+0000 7f2fa359e640 1 -- 172.21.15.125:0/886468627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f90005190 msgr2=0x7f2f9000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.195+0000 7f2fa359e640 1 --2- 172.21.15.125:0/886468627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f90005190 0x7f2f9000b990 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009fc0 tx=0x7f2f9c01cca0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.195+0000 7f2fa359e640 1 -- 172.21.15.125:0/886468627 shutdown_connections 2023-10-30T12:29:15.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.195+0000 7f2fa359e640 1 --2- 172.21.15.125:0/886468627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f90005190 0x7f2f9000b990 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.195+0000 7f2fa359e640 1 -- 172.21.15.125:0/886468627 >> 172.21.15.125:0/886468627 conn(0x7f2fa406be40 msgr2=0x7f2fa406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:15.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.195+0000 7f2fa359e640 1 -- 172.21.15.125:0/886468627 shutdown_connections 2023-10-30T12:29:15.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.195+0000 7f2fa359e640 1 -- 172.21.15.125:0/886468627 wait complete. 2023-10-30T12:29:15.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.195+0000 7f2fa359e640 1 Processor -- start 2023-10-30T12:29:15.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa359e640 1 -- start start 2023-10-30T12:29:15.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa359e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f90005190 0x7f2fa40c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa359e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fa4154060 0x7f2fa41f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa359e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2fa40c4e50 0x7f2fa41498f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa359e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa40c7500 con 0x7f2f90005190 2023-10-30T12:29:15.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa359e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa40c8680 con 0x7f2fa4154060 2023-10-30T12:29:15.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa359e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fa40c8960 con 0x7f2fa40c4e50 2023-10-30T12:29:15.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa259c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f90005190 0x7f2fa40c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa259c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f90005190 0x7f2fa40c7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40042/0 (socket says 172.21.15.125:40042) 2023-10-30T12:29:15.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa259c640 1 -- 172.21.15.125:0/1641790796 learned_addr learned my addr 172.21.15.125:0/1641790796 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:15.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa2d9d640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2fa40c4e50 0x7f2fa41498f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2f99d9b640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fa4154060 0x7f2fa41f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa259c640 1 -- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2fa40c4e50 msgr2=0x7f2fa41498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa259c640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2fa40c4e50 0x7f2fa41498f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa259c640 1 -- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fa4154060 msgr2=0x7f2fa41f23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa259c640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fa4154060 0x7f2fa41f23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa259c640 1 -- 172.21.15.125:0/1641790796 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa40c8c40 con 0x7f2f90005190 2023-10-30T12:29:15.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa2d9d640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2fa40c4e50 0x7f2fa41498f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:15.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2f99d9b640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fa4154060 0x7f2fa41f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:15.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa259c640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f90005190 0x7f2fa40c7990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400e880 tx=0x7f2f9400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2f9b7fe640 1 -- 172.21.15.125:0/1641790796 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9400c8d0 con 0x7f2f90005190 2023-10-30T12:29:15.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2f9b7fe640 1 -- 172.21.15.125:0/1641790796 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9400ca50 con 0x7f2f90005190 2023-10-30T12:29:15.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2f9b7fe640 1 -- 172.21.15.125:0/1641790796 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9400ce00 con 0x7f2f90005190 2023-10-30T12:29:15.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa359e640 1 -- 172.21.15.125:0/1641790796 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa40c7190 con 0x7f2f90005190 2023-10-30T12:29:15.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa359e640 1 -- 172.21.15.125:0/1641790796 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa4155560 con 0x7f2f90005190 2023-10-30T12:29:15.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.199+0000 7f2fa359e640 1 -- 172.21.15.125:0/1641790796 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa40c6ff0 con 0x7f2f90005190 2023-10-30T12:29:15.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.203+0000 7f2f9b7fe640 1 -- 172.21.15.125:0/1641790796 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2f9401d460 con 0x7f2f90005190 2023-10-30T12:29:15.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.203+0000 7f2f9b7fe640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f840664d0 0x7f2f84068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.203+0000 7f2f9b7fe640 1 -- 172.21.15.125:0/1641790796 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f94083ef0 con 0x7f2f90005190 2023-10-30T12:29:15.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.203+0000 7f2f99d9b640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f840664d0 0x7f2f84068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.203+0000 7f2f99d9b640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f840664d0 0x7f2f84068990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2fa40cfc50 tx=0x7f2f8c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.207+0000 7f2f9b7fe640 1 -- 172.21.15.125:0/1641790796 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2fa40c6ff0 con 0x7f2f90005190 2023-10-30T12:29:15.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.355+0000 7f2fa359e640 1 -- 172.21.15.125:0/1641790796 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2fa415a6b0 con 0x7f2f90005190 2023-10-30T12:29:15.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.355+0000 7f2f9b7fe640 1 -- 172.21.15.125:0/1641790796 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2fa415a6b0 con 0x7f2f90005190 2023-10-30T12:29:15.358 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2023-10-30T12:29:15.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.355+0000 7f2fa359e640 1 -- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f840664d0 msgr2=0x7f2f84068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.355+0000 7f2fa359e640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f840664d0 0x7f2f84068990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2fa40cfc50 tx=0x7f2f8c009040 comp rx=0 tx=0).stop 2023-10-30T12:29:15.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.355+0000 7f2fa359e640 1 -- 172.21.15.125:0/1641790796 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f90005190 msgr2=0x7f2fa40c7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.355+0000 7f2fa359e640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f90005190 0x7f2fa40c7990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400e880 tx=0x7f2f9400ed40 comp rx=0 tx=0).stop 2023-10-30T12:29:15.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.355+0000 7f2fa359e640 1 -- 172.21.15.125:0/1641790796 shutdown_connections 2023-10-30T12:29:15.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.355+0000 7f2fa359e640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f840664d0 0x7f2f84068990 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.355+0000 7f2fa359e640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2fa40c4e50 0x7f2fa41498f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.355+0000 7f2fa359e640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2fa4154060 0x7f2fa41f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.355+0000 7f2fa359e640 1 --2- 172.21.15.125:0/1641790796 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f90005190 0x7f2fa40c7990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.355+0000 7f2fa359e640 1 -- 172.21.15.125:0/1641790796 >> 172.21.15.125:0/1641790796 conn(0x7f2fa406be40 msgr2=0x7f2fa40c3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:15.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.355+0000 7f2fa359e640 1 -- 172.21.15.125:0/1641790796 shutdown_connections 2023-10-30T12:29:15.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.355+0000 7f2fa359e640 1 -- 172.21.15.125:0/1641790796 wait complete. 2023-10-30T12:29:15.378 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T12:29:15.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e91c46640 1 Processor -- start 2023-10-30T12:29:15.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e91c46640 1 -- start start 2023-10-30T12:29:15.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e91c46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e8c0c5810 0x7f2e8c0c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e91c46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2e8c06a980 0x7f2e8c161f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e91c46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8c0c6cb0 con 0x7f2e8c0c6130 2023-10-30T12:29:15.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e91c46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e8c071d90 con 0x7f2e8c06a980 2023-10-30T12:29:15.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e91c46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c1585d0 con 0x7f2e8c0c5810 2023-10-30T12:29:15.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e8bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2e8c06a980 0x7f2e8c161f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e8bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2e8c06a980 0x7f2e8c161f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40050/0 (socket says 172.21.15.125:40050) 2023-10-30T12:29:15.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e8bfff640 1 -- 172.21.15.125:0/531112993 learned_addr learned my addr 172.21.15.125:0/531112993 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:15.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e8b7fe640 1 --2- 172.21.15.125:0/531112993 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e8c0c5810 0x7f2e8c0c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e8bfff640 1 -- 172.21.15.125:0/531112993 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e8c0c5810 msgr2=0x7f2e8c0c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:15.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e8bfff640 1 --2- 172.21.15.125:0/531112993 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e8c0c5810 0x7f2e8c0c5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e8bfff640 1 -- 172.21.15.125:0/531112993 >> v1:172.21.15.125:6789/0 conn(0x7f2e8c0c6130 legacy=0x7f2e8c06a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e8bfff640 1 -- 172.21.15.125:0/531112993 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8c0c6620 con 0x7f2e8c06a980 2023-10-30T12:29:15.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.515+0000 7f2e8a7fc640 1 -- 172.21.15.125:0/531112993 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285178768 0 0) 0x7f2e8c0c6cb0 con 0x7f2e8c0c6130 2023-10-30T12:29:15.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.519+0000 7f2e8bfff640 1 --2- 172.21.15.125:0/531112993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8c06a980 0x7f2e8c161f10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2e84009e80 tx=0x7f2e8401c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b188c59c4ebb1ff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.519+0000 7f2e8a7fc640 1 -- 172.21.15.125:0/531112993 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e840a56a0 con 0x7f2e8c06a980 2023-10-30T12:29:15.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.519+0000 7f2e8a7fc640 1 -- 172.21.15.125:0/531112993 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e840a5800 con 0x7f2e8c06a980 2023-10-30T12:29:15.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.519+0000 7f2e8a7fc640 1 -- 172.21.15.125:0/531112993 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e840a5b90 con 0x7f2e8c06a980 2023-10-30T12:29:15.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.519+0000 7f2e91c46640 1 -- 172.21.15.125:0/531112993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8c06a980 msgr2=0x7f2e8c161f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.519+0000 7f2e91c46640 1 --2- 172.21.15.125:0/531112993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8c06a980 0x7f2e8c161f10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2e84009e80 tx=0x7f2e8401c8d0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.519+0000 7f2e91c46640 1 -- 172.21.15.125:0/531112993 shutdown_connections 2023-10-30T12:29:15.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.519+0000 7f2e91c46640 1 --2- 172.21.15.125:0/531112993 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8c06a980 0x7f2e8c161f10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.519+0000 7f2e91c46640 1 --2- 172.21.15.125:0/531112993 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e8c0c5810 0x7f2e8c0c5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.519+0000 7f2e91c46640 1 -- 172.21.15.125:0/531112993 >> 172.21.15.125:0/531112993 conn(0x7f2e8c0c7910 msgr2=0x7f2e8c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:15.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.519+0000 7f2e91c46640 1 -- 172.21.15.125:0/531112993 shutdown_connections 2023-10-30T12:29:15.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.519+0000 7f2e91c46640 1 -- 172.21.15.125:0/531112993 wait complete. 2023-10-30T12:29:15.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e91c46640 1 Processor -- start 2023-10-30T12:29:15.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e91c46640 1 -- start start 2023-10-30T12:29:15.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e91c46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e8c06a980 0x7f2e8c155400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e91c46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8c0c5810 0x7f2e8c145710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e91c46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e8c0c6130 0x7f2e8c0ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e91c46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c0c3ac0 con 0x7f2e8c0c5810 2023-10-30T12:29:15.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e91c46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c157680 con 0x7f2e8c0c6130 2023-10-30T12:29:15.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e91c46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8c1f32d0 con 0x7f2e8c06a980 2023-10-30T12:29:15.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e8affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8c0c5810 0x7f2e8c145710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e8affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8c0c5810 0x7f2e8c145710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40054/0 (socket says 172.21.15.125:40054) 2023-10-30T12:29:15.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e8affd640 1 -- 172.21.15.125:0/1854681972 learned_addr learned my addr 172.21.15.125:0/1854681972 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:15.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e8b7fe640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e8c06a980 0x7f2e8c155400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e8bfff640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e8c0c6130 0x7f2e8c0ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e8affd640 1 -- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e8c06a980 msgr2=0x7f2e8c155400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e8affd640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e8c06a980 0x7f2e8c155400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e8affd640 1 -- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e8c0c6130 msgr2=0x7f2e8c0ca370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e8affd640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e8c0c6130 0x7f2e8c0ca370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e8affd640 1 -- 172.21.15.125:0/1854681972 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e84009b30 con 0x7f2e8c0c5810 2023-10-30T12:29:15.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e8b7fe640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e8c06a980 0x7f2e8c155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:15.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e8bfff640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e8c0c6130 0x7f2e8c0ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:15.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e8affd640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8c0c5810 0x7f2e8c145710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000c3f0 tx=0x7f2e8000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e73fff640 1 -- 172.21.15.125:0/1854681972 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e800054e0 con 0x7f2e8c0c5810 2023-10-30T12:29:15.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e73fff640 1 -- 172.21.15.125:0/1854681972 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e80003600 con 0x7f2e8c0c5810 2023-10-30T12:29:15.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e91c46640 1 -- 172.21.15.125:0/1854681972 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e8c145c50 con 0x7f2e8c0c5810 2023-10-30T12:29:15.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e73fff640 1 -- 172.21.15.125:0/1854681972 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80003990 con 0x7f2e8c0c5810 2023-10-30T12:29:15.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.523+0000 7f2e91c46640 1 -- 172.21.15.125:0/1854681972 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e8c155a90 con 0x7f2e8c0c5810 2023-10-30T12:29:15.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.527+0000 7f2e91c46640 1 -- 172.21.15.125:0/1854681972 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e8c1f3440 con 0x7f2e8c0c5810 2023-10-30T12:29:15.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.527+0000 7f2e73fff640 1 -- 172.21.15.125:0/1854681972 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2e80003af0 con 0x7f2e8c0c5810 2023-10-30T12:29:15.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.527+0000 7f2e73fff640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e780664a0 0x7f2e78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.527+0000 7f2e73fff640 1 -- 172.21.15.125:0/1854681972 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e80083510 con 0x7f2e8c0c5810 2023-10-30T12:29:15.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.527+0000 7f2e8b7fe640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e780664a0 0x7f2e78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.527+0000 7f2e8b7fe640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e780664a0 0x7f2e78068960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009af0 tx=0x7f2e7c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.531+0000 7f2e73fff640 1 -- 172.21.15.125:0/1854681972 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2e80050900 con 0x7f2e8c0c5810 2023-10-30T12:29:15.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.679+0000 7f2e91c46640 1 -- 172.21.15.125:0/1854681972 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2e8c0c75d0 con 0x7f2e8c0c5810 2023-10-30T12:29:15.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.679+0000 7f2e73fff640 1 -- 172.21.15.125:0/1854681972 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2e80017020 con 0x7f2e8c0c5810 2023-10-30T12:29:15.684 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2023-10-30T12:29:15.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.683+0000 7f2e91c46640 1 -- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e780664a0 msgr2=0x7f2e78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.683+0000 7f2e91c46640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e780664a0 0x7f2e78068960 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009af0 tx=0x7f2e7c008040 comp rx=0 tx=0).stop 2023-10-30T12:29:15.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.683+0000 7f2e91c46640 1 -- 172.21.15.125:0/1854681972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8c0c5810 msgr2=0x7f2e8c145710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.683+0000 7f2e91c46640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8c0c5810 0x7f2e8c145710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2e8000c3f0 tx=0x7f2e8000c8b0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.683+0000 7f2e91c46640 1 -- 172.21.15.125:0/1854681972 shutdown_connections 2023-10-30T12:29:15.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.683+0000 7f2e91c46640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e780664a0 0x7f2e78068960 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.683+0000 7f2e91c46640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e8c0c6130 0x7f2e8c0ca370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.683+0000 7f2e91c46640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e8c0c5810 0x7f2e8c145710 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.683+0000 7f2e91c46640 1 --2- 172.21.15.125:0/1854681972 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e8c06a980 0x7f2e8c155400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.683+0000 7f2e91c46640 1 -- 172.21.15.125:0/1854681972 >> 172.21.15.125:0/1854681972 conn(0x7f2e8c0c7910 msgr2=0x7f2e8c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:15.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.683+0000 7f2e91c46640 1 -- 172.21.15.125:0/1854681972 shutdown_connections 2023-10-30T12:29:15.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.683+0000 7f2e91c46640 1 -- 172.21.15.125:0/1854681972 wait complete. 2023-10-30T12:29:15.705 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T12:29:15.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69e1246640 1 Processor -- start 2023-10-30T12:29:15.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69e1246640 1 -- start start 2023-10-30T12:29:15.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69e1246640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69dc0c1f50 0x7f69dc0c7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69e1246640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc0c9200 con 0x7f69dc151650 2023-10-30T12:29:15.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69e1246640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69dc157d80 con 0x7f69dc06fe40 2023-10-30T12:29:15.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69e1246640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc0c8ed0 con 0x7f69dc0c1f50 2023-10-30T12:29:15.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69dad9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f69dc151650 0x7f69dc157670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60116/0 (socket says 172.21.15.125:60116) 2023-10-30T12:29:15.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69dad9d640 1 -- 172.21.15.125:0/2674544881 learned_addr learned my addr 172.21.15.125:0/2674544881 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:29:15.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69db59e640 1 --2- 172.21.15.125:0/2674544881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69dc0c1f50 0x7f69dc0c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 -- 172.21.15.125:0/2674544881 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054529741 0 0) 0x7f69dc0c9200 con 0x7f69dc151650 2023-10-30T12:29:15.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 -- 172.21.15.125:0/2674544881 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69c8003530 con 0x7f69dc151650 2023-10-30T12:29:15.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 -- 172.21.15.125:0/2674544881 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3546590930 0 0) 0x7f69c8003530 con 0x7f69dc151650 2023-10-30T12:29:15.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 -- 172.21.15.125:0/2674544881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69dc0c1f50 msgr2=0x7f69dc0c7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:15.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 --2- 172.21.15.125:0/2674544881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69dc0c1f50 0x7f69dc0c7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 -- 172.21.15.125:0/2674544881 >> v1:172.21.15.136:6789/0 conn(0x7f69dc06fe40 legacy=0x7f69dc06f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 -- 172.21.15.125:0/2674544881 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc0c3b10 con 0x7f69dc151650 2023-10-30T12:29:15.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 -- 172.21.15.125:0/2674544881 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f69cc001680 con 0x7f69dc151650 2023-10-30T12:29:15.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 -- 172.21.15.125:0/2674544881 >> v1:172.21.15.125:6789/0 conn(0x7f69dc151650 legacy=0x7f69dc157670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 --2- 172.21.15.125:0/2674544881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69c80039b0 0x7f69c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 --2- 172.21.15.125:0/2674544881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69dc0c1f50 0x7f69c80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 --2- 172.21.15.125:0/2674544881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69c8004d10 0x7f69c8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 -- 172.21.15.125:0/2674544881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc0c8ed0 con 0x7f69c8004d10 2023-10-30T12:29:15.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 -- 172.21.15.125:0/2674544881 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c800bb00 con 0x7f69c80039b0 2023-10-30T12:29:15.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69da59c640 1 -- 172.21.15.125:0/2674544881 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69c800bcc0 con 0x7f69dc0c1f50 2023-10-30T12:29:15.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69db59e640 1 --2- 172.21.15.125:0/2674544881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69c8004d10 0x7f69c8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69d259c640 1 --2- 172.21.15.125:0/2674544881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69dc0c1f50 0x7f69c80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69dad9d640 1 --2- 172.21.15.125:0/2674544881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69c80039b0 0x7f69c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69db59e640 1 -- 172.21.15.125:0/2674544881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69dc0c1f50 msgr2=0x7f69c80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:15.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69db59e640 1 --2- 172.21.15.125:0/2674544881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69dc0c1f50 0x7f69c80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69db59e640 1 -- 172.21.15.125:0/2674544881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69c80039b0 msgr2=0x7f69c8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:15.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69db59e640 1 --2- 172.21.15.125:0/2674544881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69c80039b0 0x7f69c8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69db59e640 1 -- 172.21.15.125:0/2674544881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c8003530 con 0x7f69c8004d10 2023-10-30T12:29:15.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.843+0000 7f69d259c640 1 --2- 172.21.15.125:0/2674544881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69dc0c1f50 0x7f69c80047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:15.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.847+0000 7f69db59e640 1 --2- 172.21.15.125:0/2674544881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69c8004d10 0x7f69c8005190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009f90 tx=0x7f69d401cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14bdee61afe36542 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.847+0000 7f69da59c640 1 -- 172.21.15.125:0/2674544881 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d40a5c30 con 0x7f69c8004d10 2023-10-30T12:29:15.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.847+0000 7f69da59c640 1 -- 172.21.15.125:0/2674544881 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d400f040 con 0x7f69c8004d10 2023-10-30T12:29:15.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.847+0000 7f69da59c640 1 -- 172.21.15.125:0/2674544881 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d40ad440 con 0x7f69c8004d10 2023-10-30T12:29:15.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.847+0000 7f69e1246640 1 -- 172.21.15.125:0/2674544881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69c8004d10 msgr2=0x7f69c8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.847+0000 7f69e1246640 1 --2- 172.21.15.125:0/2674544881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69c8004d10 0x7f69c8005190 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f69d4009f90 tx=0x7f69d401cc20 comp rx=0 tx=0).stop 2023-10-30T12:29:15.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.847+0000 7f69db59e640 1 -- 172.21.15.125:0/2674544881 reap_dead start 2023-10-30T12:29:15.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.847+0000 7f69e1246640 1 -- 172.21.15.125:0/2674544881 shutdown_connections 2023-10-30T12:29:15.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.847+0000 7f69e1246640 1 -- 172.21.15.125:0/2674544881 >> 172.21.15.125:0/2674544881 conn(0x7f69dc069e90 msgr2=0x7f69dc06a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:15.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.847+0000 7f69e1246640 1 -- 172.21.15.125:0/2674544881 shutdown_connections 2023-10-30T12:29:15.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.847+0000 7f69e1246640 1 -- 172.21.15.125:0/2674544881 wait complete. 2023-10-30T12:29:15.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69e1246640 1 Processor -- start 2023-10-30T12:29:15.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69e1246640 1 -- start start 2023-10-30T12:29:15.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69e1246640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69c8004d10 0x7f69dc0c5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69e1246640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69dc0c5ae0 0x7f69dc157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69e1246640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69dc156980 0x7f69dc1eff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69e1246640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc156530 con 0x7f69c8004d10 2023-10-30T12:29:15.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69e1246640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc0c7190 con 0x7f69dc0c5ae0 2023-10-30T12:29:15.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69e1246640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69dc0c6860 con 0x7f69dc156980 2023-10-30T12:29:15.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69dad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69c8004d10 0x7f69dc0c5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69dad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69c8004d10 0x7f69dc0c5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40070/0 (socket says 172.21.15.125:40070) 2023-10-30T12:29:15.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69dad9d640 1 -- 172.21.15.125:0/571801536 learned_addr learned my addr 172.21.15.125:0/571801536 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:15.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69d259c640 1 --2- 172.21.15.125:0/571801536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69dc0c5ae0 0x7f69dc157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69db59e640 1 --2- 172.21.15.125:0/571801536 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69dc156980 0x7f69dc1eff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69dad9d640 1 -- 172.21.15.125:0/571801536 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69dc156980 msgr2=0x7f69dc1eff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69dad9d640 1 --2- 172.21.15.125:0/571801536 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69dc156980 0x7f69dc1eff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69dad9d640 1 -- 172.21.15.125:0/571801536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69dc0c5ae0 msgr2=0x7f69dc157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:15.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69dad9d640 1 --2- 172.21.15.125:0/571801536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69dc0c5ae0 0x7f69dc157500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:15.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69dad9d640 1 -- 172.21.15.125:0/571801536 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69dc0c6b40 con 0x7f69c8004d10 2023-10-30T12:29:15.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69dad9d640 1 --2- 172.21.15.125:0/571801536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69c8004d10 0x7f69dc0c5fb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f69cc00e7b0 tx=0x7f69cc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69d3fff640 1 -- 172.21.15.125:0/571801536 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69cc00eec0 con 0x7f69c8004d10 2023-10-30T12:29:15.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69d3fff640 1 -- 172.21.15.125:0/571801536 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69cc00c8d0 con 0x7f69c8004d10 2023-10-30T12:29:15.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69d3fff640 1 -- 172.21.15.125:0/571801536 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69cc00cc60 con 0x7f69c8004d10 2023-10-30T12:29:15.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69e1246640 1 -- 172.21.15.125:0/571801536 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69dc0c6e20 con 0x7f69c8004d10 2023-10-30T12:29:15.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.851+0000 7f69e1246640 1 -- 172.21.15.125:0/571801536 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69dc153a70 con 0x7f69c8004d10 2023-10-30T12:29:15.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.855+0000 7f69e1246640 1 -- 172.21.15.125:0/571801536 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69dc0c72d0 con 0x7f69c8004d10 2023-10-30T12:29:15.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.855+0000 7f69d3fff640 1 -- 172.21.15.125:0/571801536 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f69cc01d460 con 0x7f69c8004d10 2023-10-30T12:29:15.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.855+0000 7f69d3fff640 1 --2- 172.21.15.125:0/571801536 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69bc0664a0 0x7f69bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:15.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.855+0000 7f69d3fff640 1 -- 172.21.15.125:0/571801536 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69cc083f60 con 0x7f69c8004d10 2023-10-30T12:29:15.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.855+0000 7f69d259c640 1 --2- 172.21.15.125:0/571801536 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69bc0664a0 0x7f69bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:15.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.859+0000 7f69d259c640 1 --2- 172.21.15.125:0/571801536 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69bc0664a0 0x7f69bc068960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f69dc14bb70 tx=0x7f69c4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:15.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:15.859+0000 7f69d3fff640 1 -- 172.21.15.125:0/571801536 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f69cc0512d0 con 0x7f69c8004d10 2023-10-30T12:29:16.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.011+0000 7f69e1246640 1 -- 172.21.15.125:0/571801536 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f69dc04e170 con 0x7f69c8004d10 2023-10-30T12:29:16.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.011+0000 7f69d3fff640 1 -- 172.21.15.125:0/571801536 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f69cc0548f0 con 0x7f69c8004d10 2023-10-30T12:29:16.014 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2023-10-30T12:29:16.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.011+0000 7f69e1246640 1 -- 172.21.15.125:0/571801536 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69bc0664a0 msgr2=0x7f69bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.011+0000 7f69e1246640 1 --2- 172.21.15.125:0/571801536 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69bc0664a0 0x7f69bc068960 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f69dc14bb70 tx=0x7f69c4009040 comp rx=0 tx=0).stop 2023-10-30T12:29:16.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.011+0000 7f69e1246640 1 -- 172.21.15.125:0/571801536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69c8004d10 msgr2=0x7f69dc0c5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.011+0000 7f69e1246640 1 --2- 172.21.15.125:0/571801536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69c8004d10 0x7f69dc0c5fb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f69cc00e7b0 tx=0x7f69cc00ec70 comp rx=0 tx=0).stop 2023-10-30T12:29:16.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.011+0000 7f69e1246640 1 -- 172.21.15.125:0/571801536 shutdown_connections 2023-10-30T12:29:16.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.011+0000 7f69e1246640 1 --2- 172.21.15.125:0/571801536 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69bc0664a0 0x7f69bc068960 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.011+0000 7f69e1246640 1 --2- 172.21.15.125:0/571801536 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69dc156980 0x7f69dc1eff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.011+0000 7f69e1246640 1 --2- 172.21.15.125:0/571801536 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69dc0c5ae0 0x7f69dc157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.011+0000 7f69e1246640 1 --2- 172.21.15.125:0/571801536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69c8004d10 0x7f69dc0c5fb0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.011+0000 7f69e1246640 1 -- 172.21.15.125:0/571801536 >> 172.21.15.125:0/571801536 conn(0x7f69dc069e90 msgr2=0x7f69dc14bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:16.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.011+0000 7f69e1246640 1 -- 172.21.15.125:0/571801536 shutdown_connections 2023-10-30T12:29:16.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.011+0000 7f69e1246640 1 -- 172.21.15.125:0/571801536 wait complete. 2023-10-30T12:29:16.035 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:29:16.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.171+0000 7f7118594640 1 Processor -- start 2023-10-30T12:29:16.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.171+0000 7f7118594640 1 -- start start 2023-10-30T12:29:16.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7118594640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f71101538a0 0x7f711015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7118594640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71100ca0e0 0x7f711015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7118594640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f71100cf160 0x7f71100cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7118594640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71100cb800 con 0x7f71100cf160 2023-10-30T12:29:16.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7118594640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f711015dc30 con 0x7f71101538a0 2023-10-30T12:29:16.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7118594640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71100cb330 con 0x7f71100ca0e0 2023-10-30T12:29:16.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7116b31640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f71100cf160 0x7f71100cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7116b31640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f71100cf160 0x7f71100cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40072/0 (socket says 172.21.15.125:40072) 2023-10-30T12:29:16.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7116b31640 1 -- 172.21.15.125:0/555887923 learned_addr learned my addr 172.21.15.125:0/555887923 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:16.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7116330640 1 --2- 172.21.15.125:0/555887923 >> v2:172.21.15.136:3300/0 conn(0x7f71101538a0 0x7f711015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7115b2f640 1 --2- 172.21.15.125:0/555887923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71100ca0e0 0x7f711015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7116b31640 1 -- 172.21.15.125:0/555887923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71100ca0e0 msgr2=0x7f711015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:16.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7116b31640 1 --2- 172.21.15.125:0/555887923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71100ca0e0 0x7f711015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7116b31640 1 -- 172.21.15.125:0/555887923 >> v2:172.21.15.136:3300/0 conn(0x7f71101538a0 msgr2=0x7f711015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:16.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7116b31640 1 --2- 172.21.15.125:0/555887923 >> v2:172.21.15.136:3300/0 conn(0x7f71101538a0 0x7f711015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7116b31640 1 -- 172.21.15.125:0/555887923 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71100cad70 con 0x7f71100cf160 2023-10-30T12:29:16.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.175+0000 7f7116330640 1 --2- 172.21.15.125:0/555887923 >> v2:172.21.15.136:3300/0 conn(0x7f71101538a0 0x7f711015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:16.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f7116b31640 1 --2- 172.21.15.125:0/555887923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71100cf160 0x7f71100cf590 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f71100cfd10 tx=0x7f710801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a02895fb109ed36 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f711532e640 1 -- 172.21.15.125:0/555887923 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71080af400 con 0x7f71100cf160 2023-10-30T12:29:16.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f711532e640 1 -- 172.21.15.125:0/555887923 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71080ab050 con 0x7f71100cf160 2023-10-30T12:29:16.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f711532e640 1 -- 172.21.15.125:0/555887923 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71080af780 con 0x7f71100cf160 2023-10-30T12:29:16.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f7118594640 1 -- 172.21.15.125:0/555887923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71100cf160 msgr2=0x7f71100cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f7118594640 1 --2- 172.21.15.125:0/555887923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71100cf160 0x7f71100cf590 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f71100cfd10 tx=0x7f710801f1c0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f7118594640 1 -- 172.21.15.125:0/555887923 shutdown_connections 2023-10-30T12:29:16.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f7118594640 1 --2- 172.21.15.125:0/555887923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71100cf160 0x7f71100cf590 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f7118594640 1 --2- 172.21.15.125:0/555887923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71100ca0e0 0x7f711015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f7118594640 1 --2- 172.21.15.125:0/555887923 >> v2:172.21.15.136:3300/0 conn(0x7f71101538a0 0x7f711015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f7118594640 1 -- 172.21.15.125:0/555887923 >> 172.21.15.125:0/555887923 conn(0x7f711006bfe0 msgr2=0x7f711006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:16.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f7118594640 1 -- 172.21.15.125:0/555887923 shutdown_connections 2023-10-30T12:29:16.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f7118594640 1 -- 172.21.15.125:0/555887923 wait complete. 2023-10-30T12:29:16.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f7118594640 1 Processor -- start 2023-10-30T12:29:16.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.179+0000 7f7118594640 1 -- start start 2023-10-30T12:29:16.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7118594640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71100ca0e0 0x7f71101f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7118594640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71100cf160 0x7f71101f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7118594640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71101538a0 0x7f71100b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7118594640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f711015aac0 con 0x7f71101538a0 2023-10-30T12:29:16.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7118594640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71100cd9c0 con 0x7f71100ca0e0 2023-10-30T12:29:16.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7118594640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71101f35f0 con 0x7f71100cf160 2023-10-30T12:29:16.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7116b31640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71101538a0 0x7f71100b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7116b31640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71101538a0 0x7f71100b3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40074/0 (socket says 172.21.15.125:40074) 2023-10-30T12:29:16.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7116b31640 1 -- 172.21.15.125:0/778022435 learned_addr learned my addr 172.21.15.125:0/778022435 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:16.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7115b2f640 1 --2- 172.21.15.125:0/778022435 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71100cf160 0x7f71101f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7116330640 1 --2- 172.21.15.125:0/778022435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71100ca0e0 0x7f71101f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7116b31640 1 -- 172.21.15.125:0/778022435 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71100cf160 msgr2=0x7f71101f2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7116b31640 1 --2- 172.21.15.125:0/778022435 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71100cf160 0x7f71101f2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7116b31640 1 -- 172.21.15.125:0/778022435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71100ca0e0 msgr2=0x7f71101f2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7116b31640 1 --2- 172.21.15.125:0/778022435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71100ca0e0 0x7f71101f2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7116b31640 1 -- 172.21.15.125:0/778022435 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7108009b30 con 0x7f71101538a0 2023-10-30T12:29:16.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7116b31640 1 --2- 172.21.15.125:0/778022435 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71101538a0 0x7f71100b3440 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f71080b2040 tx=0x7f710801bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7106ffd640 1 -- 172.21.15.125:0/778022435 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71080afd10 con 0x7f71101538a0 2023-10-30T12:29:16.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7106ffd640 1 -- 172.21.15.125:0/778022435 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71080afe90 con 0x7f71101538a0 2023-10-30T12:29:16.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7106ffd640 1 -- 172.21.15.125:0/778022435 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71080ba660 con 0x7f71101538a0 2023-10-30T12:29:16.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7118594640 1 -- 172.21.15.125:0/778022435 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71100cdc80 con 0x7f71101538a0 2023-10-30T12:29:16.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7118594640 1 -- 172.21.15.125:0/778022435 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71100ce240 con 0x7f71101538a0 2023-10-30T12:29:16.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7118594640 1 -- 172.21.15.125:0/778022435 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71100b2a50 con 0x7f71101538a0 2023-10-30T12:29:16.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7106ffd640 1 -- 172.21.15.125:0/778022435 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f71080b6040 con 0x7f71101538a0 2023-10-30T12:29:16.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7106ffd640 1 --2- 172.21.15.125:0/778022435 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70f40664d0 0x7f70f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.183+0000 7f7106ffd640 1 -- 172.21.15.125:0/778022435 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7108121ef0 con 0x7f71101538a0 2023-10-30T12:29:16.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.187+0000 7f7116330640 1 --2- 172.21.15.125:0/778022435 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70f40664d0 0x7f70f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.187+0000 7f7116330640 1 --2- 172.21.15.125:0/778022435 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70f40664d0 0x7f70f4068990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f710c00ba10 tx=0x7f710c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.187+0000 7f7106ffd640 1 -- 172.21.15.125:0/778022435 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f71080ef2d0 con 0x7f71101538a0 2023-10-30T12:29:16.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.367+0000 7f7118594640 1 -- 172.21.15.125:0/778022435 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71100ce760 con 0x7f71101538a0 2023-10-30T12:29:16.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.367+0000 7f7106ffd640 1 -- 172.21.15.125:0/778022435 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6537 (secure 0 0 0) 0x7f71100ce760 con 0x7f71101538a0 2023-10-30T12:29:16.371 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:29:16.372 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:29:01.095107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.136:6835/3346501886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3346501886},{"type":"v1","addr":"172.21.15.136:6835","nonce":3346501886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:29:16.372 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 12 2023-10-30T12:29:16.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.371+0000 7f7118594640 1 -- 172.21.15.125:0/778022435 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70f40664d0 msgr2=0x7f70f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.371+0000 7f7118594640 1 --2- 172.21.15.125:0/778022435 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70f40664d0 0x7f70f4068990 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f710c00ba10 tx=0x7f710c008040 comp rx=0 tx=0).stop 2023-10-30T12:29:16.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.371+0000 7f7118594640 1 -- 172.21.15.125:0/778022435 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71101538a0 msgr2=0x7f71100b3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.371+0000 7f7118594640 1 --2- 172.21.15.125:0/778022435 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71101538a0 0x7f71100b3440 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f71080b2040 tx=0x7f710801bf60 comp rx=0 tx=0).stop 2023-10-30T12:29:16.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.371+0000 7f7118594640 1 -- 172.21.15.125:0/778022435 shutdown_connections 2023-10-30T12:29:16.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.371+0000 7f7118594640 1 --2- 172.21.15.125:0/778022435 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70f40664d0 0x7f70f4068990 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.371+0000 7f7118594640 1 --2- 172.21.15.125:0/778022435 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71101538a0 0x7f71100b3440 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.371+0000 7f7118594640 1 --2- 172.21.15.125:0/778022435 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71100cf160 0x7f71101f2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.371+0000 7f7118594640 1 --2- 172.21.15.125:0/778022435 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71100ca0e0 0x7f71101f2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.371+0000 7f7118594640 1 -- 172.21.15.125:0/778022435 >> 172.21.15.125:0/778022435 conn(0x7f711006bfe0 msgr2=0x7f7110065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:16.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.371+0000 7f7118594640 1 -- 172.21.15.125:0/778022435 shutdown_connections 2023-10-30T12:29:16.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.371+0000 7f7118594640 1 -- 172.21.15.125:0/778022435 wait complete. 2023-10-30T12:29:16.392 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T12:29:16.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc433e640 1 Processor -- start 2023-10-30T12:29:16.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc433e640 1 -- start start 2023-10-30T12:29:16.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc433e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdbbc069e90 0x7fdbbc15a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc433e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdbbc06a270 0x7fdbbc0c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc433e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbbc0c6c10 con 0x7fdbbc06cb80 2023-10-30T12:29:16.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc433e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbbc0c8570 con 0x7fdbbc069e90 2023-10-30T12:29:16.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc433e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbbc0c6ef0 con 0x7fdbbc06a270 2023-10-30T12:29:16.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc20da640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdbbc069e90 0x7fdbbc15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc20da640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdbbc069e90 0x7fdbbc15a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40082/0 (socket says 172.21.15.125:40082) 2023-10-30T12:29:16.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc20da640 1 -- 172.21.15.125:0/1086469908 learned_addr learned my addr 172.21.15.125:0/1086469908 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:16.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc18d9640 1 --2- 172.21.15.125:0/1086469908 >> v2:172.21.15.136:3300/0 conn(0x7fdbbc06a270 0x7fdbbc0c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc20da640 1 -- 172.21.15.125:0/1086469908 >> v2:172.21.15.136:3300/0 conn(0x7fdbbc06a270 msgr2=0x7fdbbc0c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:29:16.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc20da640 1 --2- 172.21.15.125:0/1086469908 >> v2:172.21.15.136:3300/0 conn(0x7fdbbc06a270 0x7fdbbc0c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc20da640 1 -- 172.21.15.125:0/1086469908 >> v1:172.21.15.136:6789/0 conn(0x7fdbbc06cb80 legacy=0x7fdbbc06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.527+0000 7fdbc20da640 1 -- 172.21.15.125:0/1086469908 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc0c71d0 con 0x7fdbbc069e90 2023-10-30T12:29:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.531+0000 7fdbc20da640 1 --2- 172.21.15.125:0/1086469908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbbc069e90 0x7fdbbc15a410 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc06feb0 tx=0x7fdbb801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8174e0723ae3549 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.531+0000 7fdbc10d8640 1 -- 172.21.15.125:0/1086469908 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb80ab5f0 con 0x7fdbbc069e90 2023-10-30T12:29:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.531+0000 7fdbc10d8640 1 -- 172.21.15.125:0/1086469908 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb80ab750 con 0x7fdbbc069e90 2023-10-30T12:29:16.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.531+0000 7fdbc10d8640 1 -- 172.21.15.125:0/1086469908 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb80abae0 con 0x7fdbbc069e90 2023-10-30T12:29:16.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.531+0000 7fdbc433e640 1 -- 172.21.15.125:0/1086469908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbbc069e90 msgr2=0x7fdbbc15a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.531+0000 7fdbc433e640 1 --2- 172.21.15.125:0/1086469908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbbc069e90 0x7fdbbc15a410 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc06feb0 tx=0x7fdbb801c500 comp rx=0 tx=0).stop 2023-10-30T12:29:16.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.531+0000 7fdbc433e640 1 -- 172.21.15.125:0/1086469908 shutdown_connections 2023-10-30T12:29:16.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.531+0000 7fdbc433e640 1 --2- 172.21.15.125:0/1086469908 >> v2:172.21.15.136:3300/0 conn(0x7fdbbc06a270 0x7fdbbc0c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.531+0000 7fdbc433e640 1 --2- 172.21.15.125:0/1086469908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbbc069e90 0x7fdbbc15a410 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.531+0000 7fdbc433e640 1 -- 172.21.15.125:0/1086469908 >> 172.21.15.125:0/1086469908 conn(0x7fdbbc06dfd0 msgr2=0x7fdbbc06e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:16.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.531+0000 7fdbc433e640 1 -- 172.21.15.125:0/1086469908 shutdown_connections 2023-10-30T12:29:16.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.531+0000 7fdbc433e640 1 -- 172.21.15.125:0/1086469908 wait complete. 2023-10-30T12:29:16.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc433e640 1 Processor -- start 2023-10-30T12:29:16.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc433e640 1 -- start start 2023-10-30T12:29:16.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc433e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbbc069e90 0x7fdbbc14d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc433e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbbc06a270 0x7fdbbc1f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc433e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbbc06cb80 0x7fdbbc148af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc433e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc1554a0 con 0x7fdbbc06a270 2023-10-30T12:29:16.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc433e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc1f1a90 con 0x7fdbbc069e90 2023-10-30T12:29:16.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc433e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbbc1546e0 con 0x7fdbbc06cb80 2023-10-30T12:29:16.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc18d9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbbc06a270 0x7fdbbc1f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc18d9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbbc06a270 0x7fdbbc1f2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40090/0 (socket says 172.21.15.125:40090) 2023-10-30T12:29:16.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc18d9640 1 -- 172.21.15.125:0/2338824903 learned_addr learned my addr 172.21.15.125:0/2338824903 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:29:16.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc20da640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbbc069e90 0x7fdbbc14d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc28db640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbbc06cb80 0x7fdbbc148af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc18d9640 1 -- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbbc06cb80 msgr2=0x7fdbbc148af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc18d9640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbbc06cb80 0x7fdbbc148af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc18d9640 1 -- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbbc069e90 msgr2=0x7fdbbc14d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc18d9640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbbc069e90 0x7fdbbc14d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc18d9640 1 -- 172.21.15.125:0/2338824903 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb80094d0 con 0x7fdbbc06a270 2023-10-30T12:29:16.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc20da640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbbc069e90 0x7fdbbc14d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:29:16.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc18d9640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbbc06a270 0x7fdbbc1f2550 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00e930 tx=0x7fdbac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbb2ffd640 1 -- 172.21.15.125:0/2338824903 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac009690 con 0x7fdbbc06a270 2023-10-30T12:29:16.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbb2ffd640 1 -- 172.21.15.125:0/2338824903 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbac00ca00 con 0x7fdbbc06a270 2023-10-30T12:29:16.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbb2ffd640 1 -- 172.21.15.125:0/2338824903 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac00cd70 con 0x7fdbbc06a270 2023-10-30T12:29:16.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc433e640 1 -- 172.21.15.125:0/2338824903 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbbc1556f0 con 0x7fdbbc06a270 2023-10-30T12:29:16.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.535+0000 7fdbc433e640 1 -- 172.21.15.125:0/2338824903 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbbc1f2030 con 0x7fdbbc06a270 2023-10-30T12:29:16.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.539+0000 7fdbc433e640 1 -- 172.21.15.125:0/2338824903 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbbc1f21f0 con 0x7fdbbc06a270 2023-10-30T12:29:16.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.539+0000 7fdbb2ffd640 1 -- 172.21.15.125:0/2338824903 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdbac013470 con 0x7fdbbc06a270 2023-10-30T12:29:16.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.539+0000 7fdbb2ffd640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdba0066470 0x7fdba0068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:29:16.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.539+0000 7fdbb2ffd640 1 -- 172.21.15.125:0/2338824903 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdbac0840c0 con 0x7fdbbc06a270 2023-10-30T12:29:16.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.539+0000 7fdbc20da640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdba0066470 0x7fdba0068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:29:16.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.539+0000 7fdbc20da640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdba0066470 0x7fdba0068930 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdbb80af000 tx=0x7fdbb80a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:29:16.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.543+0000 7fdbb2ffd640 1 -- 172.21.15.125:0/2338824903 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdbac0515f0 con 0x7fdbbc06a270 2023-10-30T12:29:16.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.691+0000 7fdbc433e640 1 -- 172.21.15.125:0/2338824903 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdbbc04e170 con 0x7fdbbc06a270 2023-10-30T12:29:16.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.695+0000 7fdbb2ffd640 1 -- 172.21.15.125:0/2338824903 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16557 (secure 0 0 0) 0x7fdbac054c10 con 0x7fdbbc06a270 2023-10-30T12:29:16.697 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:29:16.698 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T12:28:45.927862+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T12:29:16.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.695+0000 7fdbc433e640 1 -- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdba0066470 msgr2=0x7fdba0068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.695+0000 7fdbc433e640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdba0066470 0x7fdba0068930 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdbb80af000 tx=0x7fdbb80a7000 comp rx=0 tx=0).stop 2023-10-30T12:29:16.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.695+0000 7fdbc433e640 1 -- 172.21.15.125:0/2338824903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbbc06a270 msgr2=0x7fdbbc1f2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:29:16.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.695+0000 7fdbc433e640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbbc06a270 0x7fdbbc1f2550 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00e930 tx=0x7fdbac00edf0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.695+0000 7fdbc433e640 1 -- 172.21.15.125:0/2338824903 shutdown_connections 2023-10-30T12:29:16.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.695+0000 7fdbc433e640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdba0066470 0x7fdba0068930 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.695+0000 7fdbc433e640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbbc06cb80 0x7fdbbc148af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.695+0000 7fdbc433e640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbbc06a270 0x7fdbbc1f2550 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.695+0000 7fdbc433e640 1 --2- 172.21.15.125:0/2338824903 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbbc069e90 0x7fdbbc14d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:29:16.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.695+0000 7fdbc433e640 1 -- 172.21.15.125:0/2338824903 >> 172.21.15.125:0/2338824903 conn(0x7fdbbc06dfd0 msgr2=0x7fdbbc0ca610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:29:16.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.695+0000 7fdbc433e640 1 -- 172.21.15.125:0/2338824903 shutdown_connections 2023-10-30T12:29:16.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:29:16.695+0000 7fdbc433e640 1 -- 172.21.15.125:0/2338824903 wait complete. 2023-10-30T12:29:16.719 DEBUG:tasks.mds_thrash:Yielding 2023-10-30T12:29:16.719 INFO:teuthology.run_tasks:Running task workunit... 2023-10-30T12:29:16.734 INFO:tasks.workunit:Pulling workunits from ref 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:29:16.735 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-30T12:29:16.735 INFO:tasks.workunit:timeout=6h 2023-10-30T12:29:16.735 INFO:tasks.workunit:cleanup=True 2023-10-30T12:29:16.736 DEBUG:teuthology.orchestra.run.smithi125:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:16.737 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2023-10-30T12:29:16.737 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 55 secs before thrashing 2023-10-30T12:29:16.746 INFO:teuthology.orchestra.run.smithi125.stdout: File: /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:16.746 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-30T12:29:16.746 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 2bh/43d Inode: 1 Links: 2 2023-10-30T12:29:16.747 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-30T12:29:16.747 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2023-10-30T12:29:16.747 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2023-10-30 12:28:50.047819409 +0000 2023-10-30T12:29:16.747 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2023-10-30 12:29:09.079193239 +0000 2023-10-30T12:29:16.747 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: 2023-10-30 12:28:50.047819409 +0000 2023-10-30T12:29:16.748 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2023-10-30T12:29:16.748 DEBUG:teuthology.orchestra.run.smithi125:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2023-10-30T12:29:17.225 DEBUG:teuthology.orchestra.run.smithi125:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 00c1c13cea21a686b046584596fbeac5315a7d71 2023-10-30T12:29:17.237 INFO:tasks.workunit.client.0.smithi125.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-30T12:29:43.245 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:29:43.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:30:11.738 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:11.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.965+0000 7ff48b59e640 1 Processor -- start 2023-10-30T12:30:11.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.965+0000 7ff48b59e640 1 -- start start 2023-10-30T12:30:11.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.969+0000 7ff48b59e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff48c1538f0 0x7ff48c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:11.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.969+0000 7ff48b59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff48c0cefa0 0x7ff48c0cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:11.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.969+0000 7ff48b59e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff48c0ca6a0 con 0x7ff48c15cd50 2023-10-30T12:30:11.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.969+0000 7ff48b59e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff48c15d300 con 0x7ff48c1538f0 2023-10-30T12:30:11.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.969+0000 7ff48b59e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c15d5e0 con 0x7ff48c0cefa0 2023-10-30T12:30:11.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.969+0000 7ff48a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff48c1538f0 0x7ff48c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:11.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.969+0000 7ff48a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff48c1538f0 0x7ff48c159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48482/0 (socket says 172.21.15.125:48482) 2023-10-30T12:30:11.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.969+0000 7ff48a59c640 1 -- 172.21.15.125:0/2820143447 learned_addr learned my addr 172.21.15.125:0/2820143447 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:11.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.969+0000 7ff48ad9d640 1 --2- 172.21.15.125:0/2820143447 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff48c0cefa0 0x7ff48c0cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:11.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.969+0000 7ff48a59c640 1 -- 172.21.15.125:0/2820143447 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff48c0cefa0 msgr2=0x7ff48c0cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:11.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.969+0000 7ff48a59c640 1 --2- 172.21.15.125:0/2820143447 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff48c0cefa0 0x7ff48c0cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:11.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.969+0000 7ff48a59c640 1 -- 172.21.15.125:0/2820143447 >> v1:172.21.15.136:6789/0 conn(0x7ff48c15cd50 legacy=0x7ff48c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:11.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.969+0000 7ff48a59c640 1 -- 172.21.15.125:0/2820143447 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48c15d8c0 con 0x7ff48c1538f0 2023-10-30T12:30:11.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.977+0000 7ff48a59c640 1 --2- 172.21.15.125:0/2820143447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff48c1538f0 0x7ff48c159fb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff47c009870 tx=0x7ff47c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a170c4115f909b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:11.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.977+0000 7ff48959a640 1 -- 172.21.15.125:0/2820143447 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c0a8a30 con 0x7ff48c1538f0 2023-10-30T12:30:11.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.977+0000 7ff48959a640 1 -- 172.21.15.125:0/2820143447 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47c0a8b90 con 0x7ff48c1538f0 2023-10-30T12:30:11.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.977+0000 7ff48959a640 1 -- 172.21.15.125:0/2820143447 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c0b0440 con 0x7ff48c1538f0 2023-10-30T12:30:11.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.977+0000 7ff48b59e640 1 -- 172.21.15.125:0/2820143447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff48c1538f0 msgr2=0x7ff48c159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:11.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.977+0000 7ff48b59e640 1 --2- 172.21.15.125:0/2820143447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff48c1538f0 0x7ff48c159fb0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7ff47c009870 tx=0x7ff47c01ed30 comp rx=0 tx=0).stop 2023-10-30T12:30:11.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.977+0000 7ff48b59e640 1 -- 172.21.15.125:0/2820143447 shutdown_connections 2023-10-30T12:30:11.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.977+0000 7ff48b59e640 1 --2- 172.21.15.125:0/2820143447 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff48c0cefa0 0x7ff48c0cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:11.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.977+0000 7ff48b59e640 1 --2- 172.21.15.125:0/2820143447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff48c1538f0 0x7ff48c159fb0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:11.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.977+0000 7ff48b59e640 1 -- 172.21.15.125:0/2820143447 >> 172.21.15.125:0/2820143447 conn(0x7ff48c06a640 msgr2=0x7ff48c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:11.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.977+0000 7ff48b59e640 1 -- 172.21.15.125:0/2820143447 shutdown_connections 2023-10-30T12:30:11.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.977+0000 7ff48b59e640 1 -- 172.21.15.125:0/2820143447 wait complete. 2023-10-30T12:30:11.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff48b59e640 1 Processor -- start 2023-10-30T12:30:11.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff48b59e640 1 -- start start 2023-10-30T12:30:11.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff48b59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff48c0cefa0 0x7ff48c0ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:11.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff48b59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff48c1538f0 0x7ff48c1ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:11.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff48b59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff48c15cd50 0x7ff48c145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:11.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff48b59e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c15a9c0 con 0x7ff48c1538f0 2023-10-30T12:30:11.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff48b59e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c1590c0 con 0x7ff48c15cd50 2023-10-30T12:30:11.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff48b59e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff48c15ac60 con 0x7ff48c0cefa0 2023-10-30T12:30:11.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff489d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff48c1538f0 0x7ff48c1ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:11.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff489d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff48c1538f0 0x7ff48c1ef940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48496/0 (socket says 172.21.15.125:48496) 2023-10-30T12:30:11.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff489d9b640 1 -- 172.21.15.125:0/849192286 learned_addr learned my addr 172.21.15.125:0/849192286 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:11.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff48a59c640 1 --2- 172.21.15.125:0/849192286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff48c0cefa0 0x7ff48c0ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:11.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff48ad9d640 1 --2- 172.21.15.125:0/849192286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff48c15cd50 0x7ff48c145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:11.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff489d9b640 1 -- 172.21.15.125:0/849192286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff48c0cefa0 msgr2=0x7ff48c0ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:11.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff489d9b640 1 --2- 172.21.15.125:0/849192286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff48c0cefa0 0x7ff48c0ce1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:11.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff489d9b640 1 -- 172.21.15.125:0/849192286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff48c15cd50 msgr2=0x7ff48c145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:11.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff489d9b640 1 --2- 172.21.15.125:0/849192286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff48c15cd50 0x7ff48c145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:11.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.981+0000 7ff489d9b640 1 -- 172.21.15.125:0/849192286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47c0094d0 con 0x7ff48c1538f0 2023-10-30T12:30:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.985+0000 7ff489d9b640 1 --2- 172.21.15.125:0/849192286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff48c1538f0 0x7ff48c1ef940 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff48000c3c0 tx=0x7ff48000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.985+0000 7ff47affd640 1 -- 172.21.15.125:0/849192286 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4800052d0 con 0x7ff48c1538f0 2023-10-30T12:30:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.985+0000 7ff47affd640 1 -- 172.21.15.125:0/849192286 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff480005430 con 0x7ff48c1538f0 2023-10-30T12:30:11.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.985+0000 7ff48b59e640 1 -- 172.21.15.125:0/849192286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff48c15af40 con 0x7ff48c1538f0 2023-10-30T12:30:11.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.985+0000 7ff47affd640 1 -- 172.21.15.125:0/849192286 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff480003840 con 0x7ff48c1538f0 2023-10-30T12:30:11.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.985+0000 7ff48b59e640 1 -- 172.21.15.125:0/849192286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48c1ef420 con 0x7ff48c1538f0 2023-10-30T12:30:11.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.985+0000 7ff48b59e640 1 -- 172.21.15.125:0/849192286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff48c04e170 con 0x7ff48c1538f0 2023-10-30T12:30:11.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.989+0000 7ff47affd640 1 -- 172.21.15.125:0/849192286 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4800039a0 con 0x7ff48c1538f0 2023-10-30T12:30:11.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.989+0000 7ff47affd640 1 --2- 172.21.15.125:0/849192286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff46c066570 0x7ff46c068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:11.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.989+0000 7ff47affd640 1 -- 172.21.15.125:0/849192286 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff480083390 con 0x7ff48c1538f0 2023-10-30T12:30:11.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.989+0000 7ff48a59c640 1 --2- 172.21.15.125:0/849192286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff46c066570 0x7ff46c068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:11.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.989+0000 7ff48a59c640 1 --2- 172.21.15.125:0/849192286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff46c066570 0x7ff46c068a30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0b2000 tx=0x7ff47c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:12.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:11.997+0000 7ff47affd640 1 -- 172.21.15.125:0/849192286 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff480050880 con 0x7ff48c1538f0 2023-10-30T12:30:12.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.381+0000 7ff48b59e640 1 -- 172.21.15.125:0/849192286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff48c1ef610 con 0x7ff48c1538f0 2023-10-30T12:30:12.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.381+0000 7ff47affd640 1 -- 172.21.15.125:0/849192286 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6537 (secure 0 0 0) 0x7ff480053ea0 con 0x7ff48c1538f0 2023-10-30T12:30:12.386 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:12.387 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:29:01.095107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.136:6835/3346501886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3346501886},{"type":"v1","addr":"172.21.15.136:6835","nonce":3346501886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:12.387 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 12 2023-10-30T12:30:12.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.381+0000 7ff48b59e640 1 -- 172.21.15.125:0/849192286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff46c066570 msgr2=0x7ff46c068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:12.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.381+0000 7ff48b59e640 1 --2- 172.21.15.125:0/849192286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff46c066570 0x7ff46c068a30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0b2000 tx=0x7ff47c009040 comp rx=0 tx=0).stop 2023-10-30T12:30:12.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.381+0000 7ff48b59e640 1 -- 172.21.15.125:0/849192286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff48c1538f0 msgr2=0x7ff48c1ef940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:12.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.381+0000 7ff48b59e640 1 --2- 172.21.15.125:0/849192286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff48c1538f0 0x7ff48c1ef940 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ff48000c3c0 tx=0x7ff48000c880 comp rx=0 tx=0).stop 2023-10-30T12:30:12.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.385+0000 7ff48b59e640 1 -- 172.21.15.125:0/849192286 shutdown_connections 2023-10-30T12:30:12.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.385+0000 7ff48b59e640 1 --2- 172.21.15.125:0/849192286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff46c066570 0x7ff46c068a30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:12.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.385+0000 7ff48b59e640 1 --2- 172.21.15.125:0/849192286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff48c15cd50 0x7ff48c145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:12.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.385+0000 7ff48b59e640 1 --2- 172.21.15.125:0/849192286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff48c1538f0 0x7ff48c1ef940 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:12.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.385+0000 7ff48b59e640 1 --2- 172.21.15.125:0/849192286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff48c0cefa0 0x7ff48c0ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:12.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.385+0000 7ff48b59e640 1 -- 172.21.15.125:0/849192286 >> 172.21.15.125:0/849192286 conn(0x7ff48c06a640 msgr2=0x7ff48c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:12.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.385+0000 7ff48b59e640 1 -- 172.21.15.125:0/849192286 shutdown_connections 2023-10-30T12:30:12.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.385+0000 7ff48b59e640 1 -- 172.21.15.125:0/849192286 wait complete. 2023-10-30T12:30:12.420 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=2) 2023-10-30T12:30:12.420 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T12:30:12.420 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:30:12.433 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T12:30:12.433 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:12.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.661+0000 7f1a626d6640 1 Processor -- start 2023-10-30T12:30:12.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.661+0000 7f1a626d6640 1 -- start start 2023-10-30T12:30:12.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.661+0000 7f1a626d6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1a5c1538a0 0x7f1a5c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:12.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.661+0000 7f1a626d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a5c0ca0e0 0x7f1a5c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:12.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.661+0000 7f1a626d6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1a5c0cf160 0x7f1a5c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:12.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.661+0000 7f1a626d6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c0cb800 con 0x7f1a5c0cf160 2023-10-30T12:30:12.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.661+0000 7f1a626d6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c15dc30 con 0x7f1a5c1538a0 2023-10-30T12:30:12.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.661+0000 7f1a626d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0cb330 con 0x7f1a5c0ca0e0 2023-10-30T12:30:12.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.661+0000 7f1a60c73640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1a5c0cf160 0x7f1a5c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:12.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.661+0000 7f1a5b7fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a5c0ca0e0 0x7f1a5c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:12.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.661+0000 7f1a5b7fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a5c0ca0e0 0x7f1a5c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:54528/0 (socket says 172.21.15.125:54528) 2023-10-30T12:30:12.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.661+0000 7f1a5b7fe640 1 -- 172.21.15.125:0/1458445886 learned_addr learned my addr 172.21.15.125:0/1458445886 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:12.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.661+0000 7f1a5bfff640 1 --2- 172.21.15.125:0/1458445886 >> v2:172.21.15.136:3300/0 conn(0x7f1a5c1538a0 0x7f1a5c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:12.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.665+0000 7f1a60c73640 1 -- 172.21.15.125:0/1458445886 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a5c0ca0e0 msgr2=0x7f1a5c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:12.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.665+0000 7f1a60c73640 1 --2- 172.21.15.125:0/1458445886 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a5c0ca0e0 0x7f1a5c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:12.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.665+0000 7f1a60c73640 1 -- 172.21.15.125:0/1458445886 >> v2:172.21.15.136:3300/0 conn(0x7f1a5c1538a0 msgr2=0x7f1a5c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:12.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.665+0000 7f1a60c73640 1 --2- 172.21.15.125:0/1458445886 >> v2:172.21.15.136:3300/0 conn(0x7f1a5c1538a0 0x7f1a5c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:12.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.665+0000 7f1a60c73640 1 -- 172.21.15.125:0/1458445886 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c0cad70 con 0x7f1a5c0cf160 2023-10-30T12:30:12.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.669+0000 7f1a60c73640 1 --2- 172.21.15.125:0/1458445886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5c0cf160 0x7f1a5c0cf590 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c0cfd10 tx=0x7f1a5401f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6588f2766daa6a26 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:12.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.669+0000 7f1a5affd640 1 -- 172.21.15.125:0/1458445886 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a540b0440 con 0x7f1a5c0cf160 2023-10-30T12:30:12.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.669+0000 7f1a5affd640 1 -- 172.21.15.125:0/1458445886 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a540b05a0 con 0x7f1a5c0cf160 2023-10-30T12:30:12.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.669+0000 7f1a5affd640 1 -- 172.21.15.125:0/1458445886 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a540b0930 con 0x7f1a5c0cf160 2023-10-30T12:30:12.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.669+0000 7f1a626d6640 1 -- 172.21.15.125:0/1458445886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5c0cf160 msgr2=0x7f1a5c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:12.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.669+0000 7f1a626d6640 1 --2- 172.21.15.125:0/1458445886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5c0cf160 0x7f1a5c0cf590 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c0cfd10 tx=0x7f1a5401f0f0 comp rx=0 tx=0).stop 2023-10-30T12:30:12.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.669+0000 7f1a626d6640 1 -- 172.21.15.125:0/1458445886 shutdown_connections 2023-10-30T12:30:12.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.669+0000 7f1a626d6640 1 --2- 172.21.15.125:0/1458445886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5c0cf160 0x7f1a5c0cf590 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:12.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.669+0000 7f1a626d6640 1 --2- 172.21.15.125:0/1458445886 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a5c0ca0e0 0x7f1a5c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:12.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.669+0000 7f1a626d6640 1 --2- 172.21.15.125:0/1458445886 >> v2:172.21.15.136:3300/0 conn(0x7f1a5c1538a0 0x7f1a5c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:12.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.669+0000 7f1a626d6640 1 -- 172.21.15.125:0/1458445886 >> 172.21.15.125:0/1458445886 conn(0x7f1a5c06bfe0 msgr2=0x7f1a5c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:12.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.669+0000 7f1a626d6640 1 -- 172.21.15.125:0/1458445886 shutdown_connections 2023-10-30T12:30:12.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.669+0000 7f1a626d6640 1 -- 172.21.15.125:0/1458445886 wait complete. 2023-10-30T12:30:12.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.673+0000 7f1a626d6640 1 Processor -- start 2023-10-30T12:30:12.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.673+0000 7f1a626d6640 1 -- start start 2023-10-30T12:30:12.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.673+0000 7f1a626d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a5c0ca0e0 0x7f1a5c1f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:12.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.673+0000 7f1a626d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a5c0cf160 0x7f1a5c1f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:12.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.673+0000 7f1a626d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5c1538a0 0x7f1a5c0b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:12.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.673+0000 7f1a626d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c15aac0 con 0x7f1a5c1538a0 2023-10-30T12:30:12.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.673+0000 7f1a626d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0cd9c0 con 0x7f1a5c0ca0e0 2023-10-30T12:30:12.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.673+0000 7f1a626d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c1f35f0 con 0x7f1a5c0cf160 2023-10-30T12:30:12.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.673+0000 7f1a60c73640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5c1538a0 0x7f1a5c0b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:12.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.673+0000 7f1a60c73640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5c1538a0 0x7f1a5c0b3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48504/0 (socket says 172.21.15.125:48504) 2023-10-30T12:30:12.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.673+0000 7f1a60c73640 1 -- 172.21.15.125:0/35597101 learned_addr learned my addr 172.21.15.125:0/35597101 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:12.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.673+0000 7f1a5bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a5c0ca0e0 0x7f1a5c1f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:12.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.673+0000 7f1a5b7fe640 1 --2- 172.21.15.125:0/35597101 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a5c0cf160 0x7f1a5c1f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:12.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.677+0000 7f1a60c73640 1 -- 172.21.15.125:0/35597101 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a5c0cf160 msgr2=0x7f1a5c1f2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:12.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.677+0000 7f1a60c73640 1 --2- 172.21.15.125:0/35597101 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a5c0cf160 0x7f1a5c1f2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:12.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.677+0000 7f1a60c73640 1 -- 172.21.15.125:0/35597101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a5c0ca0e0 msgr2=0x7f1a5c1f2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:12.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.677+0000 7f1a60c73640 1 --2- 172.21.15.125:0/35597101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a5c0ca0e0 0x7f1a5c1f2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:12.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.677+0000 7f1a60c73640 1 -- 172.21.15.125:0/35597101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a54009a70 con 0x7f1a5c1538a0 2023-10-30T12:30:12.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.677+0000 7f1a60c73640 1 --2- 172.21.15.125:0/35597101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5c1538a0 0x7f1a5c0b3440 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1a5401d7c0 tx=0x7f1a5401beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:12.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.677+0000 7f1a58ff9640 1 -- 172.21.15.125:0/35597101 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a540b0440 con 0x7f1a5c1538a0 2023-10-30T12:30:12.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.677+0000 7f1a58ff9640 1 -- 172.21.15.125:0/35597101 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a540bc650 con 0x7f1a5c1538a0 2023-10-30T12:30:12.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.677+0000 7f1a626d6640 1 -- 172.21.15.125:0/35597101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c0cdc80 con 0x7f1a5c1538a0 2023-10-30T12:30:12.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.677+0000 7f1a58ff9640 1 -- 172.21.15.125:0/35597101 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a540bca00 con 0x7f1a5c1538a0 2023-10-30T12:30:12.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.677+0000 7f1a626d6640 1 -- 172.21.15.125:0/35597101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c0ce240 con 0x7f1a5c1538a0 2023-10-30T12:30:12.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.677+0000 7f1a626d6640 1 -- 172.21.15.125:0/35597101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c0b2a50 con 0x7f1a5c1538a0 2023-10-30T12:30:12.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.681+0000 7f1a58ff9640 1 -- 172.21.15.125:0/35597101 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1a540bcb60 con 0x7f1a5c1538a0 2023-10-30T12:30:12.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.681+0000 7f1a58ff9640 1 --2- 172.21.15.125:0/35597101 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a48066520 0x7f1a480689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:12.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.681+0000 7f1a58ff9640 1 -- 172.21.15.125:0/35597101 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f1a541229b0 con 0x7f1a5c1538a0 2023-10-30T12:30:12.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.681+0000 7f1a5bfff640 1 --2- 172.21.15.125:0/35597101 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a48066520 0x7f1a480689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:12.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.681+0000 7f1a5bfff640 1 --2- 172.21.15.125:0/35597101 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a48066520 0x7f1a480689e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009ac0 tx=0x7f1a4c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:12.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:12.689+0000 7f1a58ff9640 1 -- 172.21.15.125:0/35597101 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1a540efea0 con 0x7f1a5c1538a0 2023-10-30T12:30:13.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:13.065+0000 7f1a626d6640 1 -- 172.21.15.125:0/35597101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a5c0ce760 con 0x7f1a5c1538a0 2023-10-30T12:30:13.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:13.065+0000 7f1a58ff9640 1 -- 172.21.15.125:0/35597101 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6537 (secure 0 0 0) 0x7f1a5c0ce760 con 0x7f1a5c1538a0 2023-10-30T12:30:13.072 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:13.072 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:29:01.095107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.136:6835/3346501886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3346501886},{"type":"v1","addr":"172.21.15.136:6835","nonce":3346501886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:13.072 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 12 2023-10-30T12:30:13.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:13.069+0000 7f1a626d6640 1 -- 172.21.15.125:0/35597101 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a48066520 msgr2=0x7f1a480689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:13.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:13.069+0000 7f1a626d6640 1 --2- 172.21.15.125:0/35597101 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a48066520 0x7f1a480689e0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c009ac0 tx=0x7f1a4c009190 comp rx=0 tx=0).stop 2023-10-30T12:30:13.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:13.069+0000 7f1a626d6640 1 -- 172.21.15.125:0/35597101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5c1538a0 msgr2=0x7f1a5c0b3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:13.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:13.069+0000 7f1a626d6640 1 --2- 172.21.15.125:0/35597101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5c1538a0 0x7f1a5c0b3440 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1a5401d7c0 tx=0x7f1a5401beb0 comp rx=0 tx=0).stop 2023-10-30T12:30:13.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:13.069+0000 7f1a626d6640 1 -- 172.21.15.125:0/35597101 shutdown_connections 2023-10-30T12:30:13.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:13.069+0000 7f1a626d6640 1 --2- 172.21.15.125:0/35597101 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a48066520 0x7f1a480689e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:13.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:13.069+0000 7f1a626d6640 1 --2- 172.21.15.125:0/35597101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a5c1538a0 0x7f1a5c0b3440 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:13.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:13.069+0000 7f1a626d6640 1 --2- 172.21.15.125:0/35597101 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a5c0cf160 0x7f1a5c1f2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:13.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:13.069+0000 7f1a626d6640 1 --2- 172.21.15.125:0/35597101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a5c0ca0e0 0x7f1a5c1f2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:13.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:13.069+0000 7f1a626d6640 1 -- 172.21.15.125:0/35597101 >> 172.21.15.125:0/35597101 conn(0x7f1a5c06bfe0 msgr2=0x7f1a5c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:13.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:13.069+0000 7f1a626d6640 1 -- 172.21.15.125:0/35597101 shutdown_connections 2023-10-30T12:30:13.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:13.069+0000 7f1a626d6640 1 -- 172.21.15.125:0/35597101 wait complete. 2023-10-30T12:30:13.103 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:30:13.343 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:30:13.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:30:15.104 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:15.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.333+0000 7fb715b77640 1 Processor -- start 2023-10-30T12:30:15.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.333+0000 7fb715b77640 1 -- start start 2023-10-30T12:30:15.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb715b77640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7101538f0 0x7fb710159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:15.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb715b77640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb71015cd50 0x7fb7100c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:15.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb715b77640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb7100cefa0 0x7fb7100cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:15.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb715b77640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb71015d300 con 0x7fb71015cd50 2023-10-30T12:30:15.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb715b77640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb71015d5e0 con 0x7fb7100cefa0 2023-10-30T12:30:15.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb715b77640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb71015d8c0 con 0x7fb7101538f0 2023-10-30T12:30:15.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb70ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb71015cd50 0x7fb7100c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:15.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb70ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb71015cd50 0x7fb7100c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48506/0 (socket says 172.21.15.125:48506) 2023-10-30T12:30:15.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb70ffff640 1 -- 172.21.15.125:0/3855876231 learned_addr learned my addr 172.21.15.125:0/3855876231 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:15.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb714b75640 1 --2- 172.21.15.125:0/3855876231 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7101538f0 0x7fb710159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:15.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb715376640 1 --2- 172.21.15.125:0/3855876231 >> v2:172.21.15.136:3300/0 conn(0x7fb7100cefa0 0x7fb7100cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:15.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb70ffff640 1 -- 172.21.15.125:0/3855876231 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7101538f0 msgr2=0x7fb710159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:15.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb70ffff640 1 --2- 172.21.15.125:0/3855876231 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7101538f0 0x7fb710159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:15.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb70ffff640 1 -- 172.21.15.125:0/3855876231 >> v2:172.21.15.136:3300/0 conn(0x7fb7100cefa0 msgr2=0x7fb7100cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:15.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb70ffff640 1 --2- 172.21.15.125:0/3855876231 >> v2:172.21.15.136:3300/0 conn(0x7fb7100cefa0 0x7fb7100cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:15.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.337+0000 7fb70ffff640 1 -- 172.21.15.125:0/3855876231 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb71015dba0 con 0x7fb71015cd50 2023-10-30T12:30:15.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.341+0000 7fb70ffff640 1 --2- 172.21.15.125:0/3855876231 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb71015cd50 0x7fb7100c9f90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb7100b1ae0 tx=0x7fb6fc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b3a8835d2b631f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:15.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.341+0000 7fb70f7fe640 1 -- 172.21.15.125:0/3855876231 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc0a89b0 con 0x7fb71015cd50 2023-10-30T12:30:15.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.341+0000 7fb70f7fe640 1 -- 172.21.15.125:0/3855876231 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6fc0a8b10 con 0x7fb71015cd50 2023-10-30T12:30:15.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.341+0000 7fb70f7fe640 1 -- 172.21.15.125:0/3855876231 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc0a8ea0 con 0x7fb71015cd50 2023-10-30T12:30:15.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.345+0000 7fb715b77640 1 -- 172.21.15.125:0/3855876231 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb71015cd50 msgr2=0x7fb7100c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:15.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.345+0000 7fb715b77640 1 --2- 172.21.15.125:0/3855876231 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb71015cd50 0x7fb7100c9f90 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb7100b1ae0 tx=0x7fb6fc01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:30:15.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.345+0000 7fb715b77640 1 -- 172.21.15.125:0/3855876231 shutdown_connections 2023-10-30T12:30:15.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.345+0000 7fb715b77640 1 --2- 172.21.15.125:0/3855876231 >> v2:172.21.15.136:3300/0 conn(0x7fb7100cefa0 0x7fb7100cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:15.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.345+0000 7fb715b77640 1 --2- 172.21.15.125:0/3855876231 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb71015cd50 0x7fb7100c9f90 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:15.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.345+0000 7fb715b77640 1 --2- 172.21.15.125:0/3855876231 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7101538f0 0x7fb710159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:15.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.345+0000 7fb715b77640 1 -- 172.21.15.125:0/3855876231 >> 172.21.15.125:0/3855876231 conn(0x7fb71006a640 msgr2=0x7fb710065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:15.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.345+0000 7fb715b77640 1 -- 172.21.15.125:0/3855876231 shutdown_connections 2023-10-30T12:30:15.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.345+0000 7fb715b77640 1 -- 172.21.15.125:0/3855876231 wait complete. 2023-10-30T12:30:15.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.345+0000 7fb715b77640 1 Processor -- start 2023-10-30T12:30:15.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb715b77640 1 -- start start 2023-10-30T12:30:15.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb715b77640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7100cefa0 0x7fb710145040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb715b77640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7101538f0 0x7fb7101efcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb715b77640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb71015cd50 0x7fb71015bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb715b77640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7101f0200 con 0x7fb7100cefa0 2023-10-30T12:30:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb715b77640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7101f2cc0 con 0x7fb7101538f0 2023-10-30T12:30:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb715b77640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb710143fc0 con 0x7fb71015cd50 2023-10-30T12:30:15.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb714b75640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7100cefa0 0x7fb710145040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:15.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb714b75640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7100cefa0 0x7fb710145040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48520/0 (socket says 172.21.15.125:48520) 2023-10-30T12:30:15.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb714b75640 1 -- 172.21.15.125:0/281276015 learned_addr learned my addr 172.21.15.125:0/281276015 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:15.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb715376640 1 --2- 172.21.15.125:0/281276015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb71015cd50 0x7fb71015bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:15.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb70ffff640 1 --2- 172.21.15.125:0/281276015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7101538f0 0x7fb7101efcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:15.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb714b75640 1 -- 172.21.15.125:0/281276015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb71015cd50 msgr2=0x7fb71015bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:15.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb714b75640 1 --2- 172.21.15.125:0/281276015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb71015cd50 0x7fb71015bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:15.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb714b75640 1 -- 172.21.15.125:0/281276015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7101538f0 msgr2=0x7fb7101efcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:15.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb714b75640 1 --2- 172.21.15.125:0/281276015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7101538f0 0x7fb7101efcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:15.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb714b75640 1 -- 172.21.15.125:0/281276015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc0094d0 con 0x7fb7100cefa0 2023-10-30T12:30:15.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb714b75640 1 --2- 172.21.15.125:0/281276015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7100cefa0 0x7fb710145040 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb70400e930 tx=0x7fb70400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:15.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb70d7fa640 1 -- 172.21.15.125:0/281276015 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb704009690 con 0x7fb7100cefa0 2023-10-30T12:30:15.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb70d7fa640 1 -- 172.21.15.125:0/281276015 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb70400ca00 con 0x7fb7100cefa0 2023-10-30T12:30:15.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb715b77640 1 -- 172.21.15.125:0/281276015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7101442a0 con 0x7fb7100cefa0 2023-10-30T12:30:15.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb70d7fa640 1 -- 172.21.15.125:0/281276015 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb70400cd70 con 0x7fb7100cefa0 2023-10-30T12:30:15.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.349+0000 7fb715b77640 1 -- 172.21.15.125:0/281276015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb710144860 con 0x7fb7100cefa0 2023-10-30T12:30:15.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.353+0000 7fb715b77640 1 -- 172.21.15.125:0/281276015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb71004e170 con 0x7fb7100cefa0 2023-10-30T12:30:15.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.353+0000 7fb70d7fa640 1 -- 172.21.15.125:0/281276015 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb704013470 con 0x7fb7100cefa0 2023-10-30T12:30:15.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.357+0000 7fb70d7fa640 1 --2- 172.21.15.125:0/281276015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb6f40664d0 0x7fb6f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:15.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.357+0000 7fb70d7fa640 1 -- 172.21.15.125:0/281276015 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fb704084140 con 0x7fb7100cefa0 2023-10-30T12:30:15.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.357+0000 7fb70ffff640 1 --2- 172.21.15.125:0/281276015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb6f40664d0 0x7fb6f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:15.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.357+0000 7fb70ffff640 1 --2- 172.21.15.125:0/281276015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb6f40664d0 0x7fb6f4068990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc0b2000 tx=0x7fb6fc0aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:15.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.365+0000 7fb70d7fa640 1 -- 172.21.15.125:0/281276015 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb7040515e0 con 0x7fb7100cefa0 2023-10-30T12:30:15.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.741+0000 7fb715b77640 1 -- 172.21.15.125:0/281276015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb710159730 con 0x7fb7100cefa0 2023-10-30T12:30:15.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.745+0000 7fb70d7fa640 1 -- 172.21.15.125:0/281276015 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6537 (secure 0 0 0) 0x7fb704054c00 con 0x7fb7100cefa0 2023-10-30T12:30:15.749 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:15.750 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:29:01.095107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.136:6835/3346501886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3346501886},{"type":"v1","addr":"172.21.15.136:6835","nonce":3346501886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:15.750 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 12 2023-10-30T12:30:15.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.745+0000 7fb715b77640 1 -- 172.21.15.125:0/281276015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb6f40664d0 msgr2=0x7fb6f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:15.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.745+0000 7fb715b77640 1 --2- 172.21.15.125:0/281276015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb6f40664d0 0x7fb6f4068990 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc0b2000 tx=0x7fb6fc0aa070 comp rx=0 tx=0).stop 2023-10-30T12:30:15.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.745+0000 7fb715b77640 1 -- 172.21.15.125:0/281276015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7100cefa0 msgr2=0x7fb710145040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:15.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.745+0000 7fb715b77640 1 --2- 172.21.15.125:0/281276015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7100cefa0 0x7fb710145040 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fb70400e930 tx=0x7fb70400edf0 comp rx=0 tx=0).stop 2023-10-30T12:30:15.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.745+0000 7fb715b77640 1 -- 172.21.15.125:0/281276015 shutdown_connections 2023-10-30T12:30:15.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.745+0000 7fb715b77640 1 --2- 172.21.15.125:0/281276015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb6f40664d0 0x7fb6f4068990 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:15.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.745+0000 7fb715b77640 1 --2- 172.21.15.125:0/281276015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb71015cd50 0x7fb71015bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:15.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.745+0000 7fb715b77640 1 --2- 172.21.15.125:0/281276015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7101538f0 0x7fb7101efcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:15.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.745+0000 7fb715b77640 1 --2- 172.21.15.125:0/281276015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7100cefa0 0x7fb710145040 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:15.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.745+0000 7fb715b77640 1 -- 172.21.15.125:0/281276015 >> 172.21.15.125:0/281276015 conn(0x7fb71006a640 msgr2=0x7fb7100c4940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:15.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.745+0000 7fb715b77640 1 -- 172.21.15.125:0/281276015 shutdown_connections 2023-10-30T12:30:15.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:15.745+0000 7fb715b77640 1 -- 172.21.15.125:0/281276015 wait complete. 2023-10-30T12:30:15.782 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:30:17.784 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:18.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.009+0000 7f8c2fb65640 1 Processor -- start 2023-10-30T12:30:18.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.009+0000 7f8c2fb65640 1 -- start start 2023-10-30T12:30:18.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.013+0000 7f8c2fb65640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8c281538a0 0x7f8c2815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:18.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.013+0000 7f8c2fb65640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c280cb800 con 0x7f8c280ca0e0 2023-10-30T12:30:18.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.013+0000 7f8c2fb65640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2815dc30 con 0x7f8c280cf160 2023-10-30T12:30:18.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.013+0000 7f8c2fb65640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c280cb330 con 0x7f8c281538a0 2023-10-30T12:30:18.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.013+0000 7f8c2d901640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8c281538a0 0x7f8c2815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:18.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.013+0000 7f8c2d100640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8c280ca0e0 0x7f8c2815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48236/0 (socket says 172.21.15.125:48236) 2023-10-30T12:30:18.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.013+0000 7f8c2d100640 1 -- 172.21.15.125:0/2568103006 learned_addr learned my addr 172.21.15.125:0/2568103006 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:30:18.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.013+0000 7f8c2d901640 1 -- 172.21.15.125:0/2568103006 >> v1:172.21.15.136:6789/0 conn(0x7f8c280cf160 legacy=0x7f8c28164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:18.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.013+0000 7f8c2d901640 1 -- 172.21.15.125:0/2568103006 >> v1:172.21.15.125:6789/0 conn(0x7f8c280ca0e0 legacy=0x7f8c2815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:18.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.013+0000 7f8c2d901640 1 -- 172.21.15.125:0/2568103006 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c280cad70 con 0x7f8c281538a0 2023-10-30T12:30:18.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.017+0000 7f8c2d901640 1 --2- 172.21.15.125:0/2568103006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c281538a0 0x7f8c2815d6f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8c240097c0 tx=0x7f8c2401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad86567ee71abce5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:18.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.017+0000 7f8c2c8ff640 1 -- 172.21.15.125:0/2568103006 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c240a8970 con 0x7f8c281538a0 2023-10-30T12:30:18.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.017+0000 7f8c2c8ff640 1 -- 172.21.15.125:0/2568103006 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c240a8ad0 con 0x7f8c281538a0 2023-10-30T12:30:18.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.017+0000 7f8c2c8ff640 1 -- 172.21.15.125:0/2568103006 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c240a8e60 con 0x7f8c281538a0 2023-10-30T12:30:18.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.021+0000 7f8c2fb65640 1 -- 172.21.15.125:0/2568103006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c281538a0 msgr2=0x7f8c2815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:18.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.021+0000 7f8c2fb65640 1 --2- 172.21.15.125:0/2568103006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c281538a0 0x7f8c2815d6f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8c240097c0 tx=0x7f8c2401ec70 comp rx=0 tx=0).stop 2023-10-30T12:30:18.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.021+0000 7f8c2fb65640 1 -- 172.21.15.125:0/2568103006 shutdown_connections 2023-10-30T12:30:18.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.021+0000 7f8c2fb65640 1 --2- 172.21.15.125:0/2568103006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c281538a0 0x7f8c2815d6f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:18.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.021+0000 7f8c2fb65640 1 -- 172.21.15.125:0/2568103006 >> 172.21.15.125:0/2568103006 conn(0x7f8c2806bfe0 msgr2=0x7f8c2806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:18.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.021+0000 7f8c2fb65640 1 -- 172.21.15.125:0/2568103006 shutdown_connections 2023-10-30T12:30:18.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.021+0000 7f8c2fb65640 1 -- 172.21.15.125:0/2568103006 wait complete. 2023-10-30T12:30:18.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.021+0000 7f8c2fb65640 1 Processor -- start 2023-10-30T12:30:18.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2fb65640 1 -- start start 2023-10-30T12:30:18.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2fb65640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c280ca0e0 0x7f8c280b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:18.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2fb65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8c280cf160 0x7f8c2815b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:18.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2fb65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c28159400 0x7f8c2815b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:18.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2fb65640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c281e5010 con 0x7f8c28159400 2023-10-30T12:30:18.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2fb65640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c28158c30 con 0x7f8c280ca0e0 2023-10-30T12:30:18.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2fb65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c28158e60 con 0x7f8c280cf160 2023-10-30T12:30:18.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2e102640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c28159400 0x7f8c2815b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:18.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2d901640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c280ca0e0 0x7f8c280b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:18.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2d100640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8c280cf160 0x7f8c2815b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:18.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2e102640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c28159400 0x7f8c2815b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48526/0 (socket says 172.21.15.125:48526) 2023-10-30T12:30:18.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2e102640 1 -- 172.21.15.125:0/4272260006 learned_addr learned my addr 172.21.15.125:0/4272260006 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:18.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2e102640 1 -- 172.21.15.125:0/4272260006 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8c280cf160 msgr2=0x7f8c2815b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:18.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2e102640 1 --2- 172.21.15.125:0/4272260006 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8c280cf160 0x7f8c2815b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:18.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2e102640 1 -- 172.21.15.125:0/4272260006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c280ca0e0 msgr2=0x7f8c280b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:18.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2e102640 1 --2- 172.21.15.125:0/4272260006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c280ca0e0 0x7f8c280b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:18.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2e102640 1 -- 172.21.15.125:0/4272260006 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c24009420 con 0x7f8c28159400 2023-10-30T12:30:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2e102640 1 --2- 172.21.15.125:0/4272260006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c28159400 0x7f8c2815b940 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000c6c0 tx=0x7f8c2000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c1e7fc640 1 -- 172.21.15.125:0/4272260006 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c20003b80 con 0x7f8c28159400 2023-10-30T12:30:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c1e7fc640 1 -- 172.21.15.125:0/4272260006 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c20003ce0 con 0x7f8c28159400 2023-10-30T12:30:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2fb65640 1 -- 172.21.15.125:0/4272260006 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c281e3060 con 0x7f8c28159400 2023-10-30T12:30:18.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c1e7fc640 1 -- 172.21.15.125:0/4272260006 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c200052f0 con 0x7f8c28159400 2023-10-30T12:30:18.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.025+0000 7f8c2fb65640 1 -- 172.21.15.125:0/4272260006 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c28158530 con 0x7f8c28159400 2023-10-30T12:30:18.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.029+0000 7f8c2fb65640 1 -- 172.21.15.125:0/4272260006 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c2804e170 con 0x7f8c28159400 2023-10-30T12:30:18.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.029+0000 7f8c1e7fc640 1 -- 172.21.15.125:0/4272260006 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8c20005450 con 0x7f8c28159400 2023-10-30T12:30:18.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.033+0000 7f8c1e7fc640 1 --2- 172.21.15.125:0/4272260006 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8c0c0664a0 0x7f8c0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:18.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.033+0000 7f8c1e7fc640 1 -- 172.21.15.125:0/4272260006 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5738+0+0 (secure 0 0 0) 0x7f8c20083a10 con 0x7f8c28159400 2023-10-30T12:30:18.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.033+0000 7f8c2d901640 1 --2- 172.21.15.125:0/4272260006 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8c0c0664a0 0x7f8c0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:18.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.033+0000 7f8c2d901640 1 --2- 172.21.15.125:0/4272260006 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8c0c0664a0 0x7f8c0c068960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8c240b2000 tx=0x7f8c240aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:18.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.037+0000 7f8c1e7fc640 1 -- 172.21.15.125:0/4272260006 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8c20050e80 con 0x7f8c28159400 2023-10-30T12:30:18.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.261+0000 7f8c2fb65640 1 -- 172.21.15.125:0/4272260006 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c2815dc30 con 0x7f8c28159400 2023-10-30T12:30:18.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.261+0000 7f8c1e7fc640 1 -- 172.21.15.125:0/4272260006 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6537 (secure 0 0 0) 0x7f8c2815dc30 con 0x7f8c28159400 2023-10-30T12:30:18.267 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:18.268 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:29:01.095107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.136:6835/3346501886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3346501886},{"type":"v1","addr":"172.21.15.136:6835","nonce":3346501886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:18.268 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 12 2023-10-30T12:30:18.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.265+0000 7f8c2fb65640 1 -- 172.21.15.125:0/4272260006 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8c0c0664a0 msgr2=0x7f8c0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:18.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.265+0000 7f8c2fb65640 1 --2- 172.21.15.125:0/4272260006 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8c0c0664a0 0x7f8c0c068960 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8c240b2000 tx=0x7f8c240aa070 comp rx=0 tx=0).stop 2023-10-30T12:30:18.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.265+0000 7f8c2fb65640 1 -- 172.21.15.125:0/4272260006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c28159400 msgr2=0x7f8c2815b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:18.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.265+0000 7f8c2fb65640 1 --2- 172.21.15.125:0/4272260006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c28159400 0x7f8c2815b940 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8c2000c6c0 tx=0x7f8c2000cb80 comp rx=0 tx=0).stop 2023-10-30T12:30:18.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.265+0000 7f8c2fb65640 1 -- 172.21.15.125:0/4272260006 shutdown_connections 2023-10-30T12:30:18.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.265+0000 7f8c2fb65640 1 --2- 172.21.15.125:0/4272260006 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8c0c0664a0 0x7f8c0c068960 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:18.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.265+0000 7f8c2fb65640 1 --2- 172.21.15.125:0/4272260006 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8c28159400 0x7f8c2815b940 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:18.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.265+0000 7f8c2fb65640 1 --2- 172.21.15.125:0/4272260006 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8c280cf160 0x7f8c2815b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:18.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.265+0000 7f8c2fb65640 1 --2- 172.21.15.125:0/4272260006 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8c280ca0e0 0x7f8c280b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:18.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.265+0000 7f8c2fb65640 1 -- 172.21.15.125:0/4272260006 >> 172.21.15.125:0/4272260006 conn(0x7f8c2806bfe0 msgr2=0x7f8c280cc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:18.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.265+0000 7f8c2fb65640 1 -- 172.21.15.125:0/4272260006 shutdown_connections 2023-10-30T12:30:18.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:18.265+0000 7f8c2fb65640 1 -- 172.21.15.125:0/4272260006 wait complete. 2023-10-30T12:30:18.299 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:30:20.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:20.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.432+0000 7faa04254640 1 Processor -- start 2023-10-30T12:30:20.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.436+0000 7faa04254640 1 -- start start 2023-10-30T12:30:20.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.436+0000 7faa04254640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa9fc1538a0 0x7fa9fc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:20.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.436+0000 7faa04254640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa9fc0cf160 0x7fa9fc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:20.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.436+0000 7faa04254640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc0cb800 con 0x7fa9fc0ca0e0 2023-10-30T12:30:20.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.436+0000 7faa04254640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9fc15dc30 con 0x7fa9fc1538a0 2023-10-30T12:30:20.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.436+0000 7faa04254640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9fc0cb330 con 0x7fa9fc0cf160 2023-10-30T12:30:20.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.436+0000 7faa01ff0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa9fc1538a0 0x7fa9fc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:20.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.436+0000 7faa01ff0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa9fc1538a0 0x7fa9fc15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48542/0 (socket says 172.21.15.125:48542) 2023-10-30T12:30:20.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.436+0000 7faa01ff0640 1 -- 172.21.15.125:0/3003240514 learned_addr learned my addr 172.21.15.125:0/3003240514 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:20.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.436+0000 7faa027f1640 1 --2- 172.21.15.125:0/3003240514 >> v2:172.21.15.136:3300/0 conn(0x7fa9fc0cf160 0x7fa9fc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:20.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.436+0000 7faa01ff0640 1 -- 172.21.15.125:0/3003240514 >> v2:172.21.15.136:3300/0 conn(0x7fa9fc0cf160 msgr2=0x7fa9fc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:20.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.436+0000 7faa01ff0640 1 --2- 172.21.15.125:0/3003240514 >> v2:172.21.15.136:3300/0 conn(0x7fa9fc0cf160 0x7fa9fc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:20.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.436+0000 7faa01ff0640 1 -- 172.21.15.125:0/3003240514 >> v1:172.21.15.136:6789/0 conn(0x7fa9fc0ca0e0 legacy=0x7fa9fc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:20.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.436+0000 7faa01ff0640 1 -- 172.21.15.125:0/3003240514 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc0cad70 con 0x7fa9fc1538a0 2023-10-30T12:30:20.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.440+0000 7faa01ff0640 1 --2- 172.21.15.125:0/3003240514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9fc1538a0 0x7fa9fc15d6f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc0b5330 tx=0x7fa9f801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2adee7e2755357c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:20.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.440+0000 7faa00fee640 1 -- 172.21.15.125:0/3003240514 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f80a89f0 con 0x7fa9fc1538a0 2023-10-30T12:30:20.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.440+0000 7faa00fee640 1 -- 172.21.15.125:0/3003240514 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f80a8b50 con 0x7fa9fc1538a0 2023-10-30T12:30:20.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.440+0000 7faa00fee640 1 -- 172.21.15.125:0/3003240514 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f80b0440 con 0x7fa9fc1538a0 2023-10-30T12:30:20.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.440+0000 7faa04254640 1 -- 172.21.15.125:0/3003240514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9fc1538a0 msgr2=0x7fa9fc15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:20.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.440+0000 7faa04254640 1 --2- 172.21.15.125:0/3003240514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9fc1538a0 0x7fa9fc15d6f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc0b5330 tx=0x7fa9f801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:30:20.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.440+0000 7faa04254640 1 -- 172.21.15.125:0/3003240514 shutdown_connections 2023-10-30T12:30:20.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.440+0000 7faa04254640 1 --2- 172.21.15.125:0/3003240514 >> v2:172.21.15.136:3300/0 conn(0x7fa9fc0cf160 0x7fa9fc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:20.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.440+0000 7faa04254640 1 --2- 172.21.15.125:0/3003240514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9fc1538a0 0x7fa9fc15d6f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:20.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.440+0000 7faa04254640 1 -- 172.21.15.125:0/3003240514 >> 172.21.15.125:0/3003240514 conn(0x7fa9fc06bfe0 msgr2=0x7fa9fc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:20.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.440+0000 7faa04254640 1 -- 172.21.15.125:0/3003240514 shutdown_connections 2023-10-30T12:30:20.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.440+0000 7faa04254640 1 -- 172.21.15.125:0/3003240514 wait complete. 2023-10-30T12:30:20.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa04254640 1 Processor -- start 2023-10-30T12:30:20.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa04254640 1 -- start start 2023-10-30T12:30:20.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa04254640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9fc0ca0e0 0x7fa9fc158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:20.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa04254640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9fc0cf160 0x7fa9fc1e40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:20.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa04254640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9fc1538a0 0x7fa9fc1e4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:20.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa04254640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc15d9f0 con 0x7fa9fc1538a0 2023-10-30T12:30:20.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa04254640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc15d710 con 0x7fa9fc0ca0e0 2023-10-30T12:30:20.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa04254640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9fc1e3600 con 0x7fa9fc0cf160 2023-10-30T12:30:20.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa027f1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9fc1538a0 0x7fa9fc1e4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:20.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa027f1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9fc1538a0 0x7fa9fc1e4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48552/0 (socket says 172.21.15.125:48552) 2023-10-30T12:30:20.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa027f1640 1 -- 172.21.15.125:0/3503990419 learned_addr learned my addr 172.21.15.125:0/3503990419 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:20.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa017ef640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9fc0cf160 0x7fa9fc1e40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:20.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa01ff0640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9fc0ca0e0 0x7fa9fc158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:20.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa027f1640 1 -- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9fc0cf160 msgr2=0x7fa9fc1e40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:20.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa027f1640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9fc0cf160 0x7fa9fc1e40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:20.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa027f1640 1 -- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9fc0ca0e0 msgr2=0x7fa9fc158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:20.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa027f1640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9fc0ca0e0 0x7fa9fc158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:20.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa027f1640 1 -- 172.21.15.125:0/3503990419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f80094d0 con 0x7fa9fc1538a0 2023-10-30T12:30:20.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa01ff0640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9fc0ca0e0 0x7fa9fc158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:30:20.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa027f1640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9fc1538a0 0x7fa9fc1e4620 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4007d90 tx=0x7fa9f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:20.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7fa9f27fc640 1 -- 172.21.15.125:0/3503990419 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4014420 con 0x7fa9fc1538a0 2023-10-30T12:30:20.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7fa9f27fc640 1 -- 172.21.15.125:0/3503990419 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f40145a0 con 0x7fa9fc1538a0 2023-10-30T12:30:20.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7fa9f27fc640 1 -- 172.21.15.125:0/3503990419 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4014950 con 0x7fa9fc1538a0 2023-10-30T12:30:20.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa04254640 1 -- 172.21.15.125:0/3503990419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9fc1e38e0 con 0x7fa9fc1538a0 2023-10-30T12:30:20.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa04254640 1 -- 172.21.15.125:0/3503990419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9fc1595d0 con 0x7fa9fc1538a0 2023-10-30T12:30:20.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.444+0000 7faa04254640 1 -- 172.21.15.125:0/3503990419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9fc0b2a50 con 0x7fa9fc1538a0 2023-10-30T12:30:20.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.448+0000 7fa9f27fc640 1 -- 172.21.15.125:0/3503990419 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa9f4014ab0 con 0x7fa9fc1538a0 2023-10-30T12:30:20.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.448+0000 7fa9f27fc640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa9e00664a0 0x7fa9e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:20.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.448+0000 7fa9f27fc640 1 -- 172.21.15.125:0/3503990419 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5738+0+0 (secure 0 0 0) 0x7fa9f40847f0 con 0x7fa9fc1538a0 2023-10-30T12:30:20.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.448+0000 7faa01ff0640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa9e00664a0 0x7fa9e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:20.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.448+0000 7faa01ff0640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa9e00664a0 0x7fa9e0068960 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa9f80b2000 tx=0x7fa9f80aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:20.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.452+0000 7fa9f27fc640 1 -- 172.21.15.125:0/3503990419 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa9f4051bb0 con 0x7fa9fc1538a0 2023-10-30T12:30:20.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.652+0000 7faa04254640 1 -- 172.21.15.125:0/3503990419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9fc1e3e00 con 0x7fa9fc1538a0 2023-10-30T12:30:20.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.656+0000 7fa9f27fc640 1 -- 172.21.15.125:0/3503990419 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6537 (secure 0 0 0) 0x7fa9fc1e3e00 con 0x7fa9fc1538a0 2023-10-30T12:30:20.661 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:20.662 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:29:01.095107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.136:6835/3346501886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3346501886},{"type":"v1","addr":"172.21.15.136:6835","nonce":3346501886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:20.662 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 12 2023-10-30T12:30:20.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.656+0000 7faa04254640 1 -- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa9e00664a0 msgr2=0x7fa9e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:20.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.656+0000 7faa04254640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa9e00664a0 0x7fa9e0068960 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa9f80b2000 tx=0x7fa9f80aa070 comp rx=0 tx=0).stop 2023-10-30T12:30:20.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.656+0000 7faa04254640 1 -- 172.21.15.125:0/3503990419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9fc1538a0 msgr2=0x7fa9fc1e4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:20.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.656+0000 7faa04254640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9fc1538a0 0x7fa9fc1e4620 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4007d90 tx=0x7fa9f400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:30:20.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.656+0000 7faa04254640 1 -- 172.21.15.125:0/3503990419 shutdown_connections 2023-10-30T12:30:20.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.656+0000 7faa04254640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa9e00664a0 0x7fa9e0068960 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:20.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.656+0000 7faa04254640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9fc1538a0 0x7fa9fc1e4620 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:20.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.656+0000 7faa04254640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9fc0cf160 0x7fa9fc1e40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:20.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.656+0000 7faa04254640 1 --2- 172.21.15.125:0/3503990419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9fc0ca0e0 0x7fa9fc158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:20.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.656+0000 7faa04254640 1 -- 172.21.15.125:0/3503990419 >> 172.21.15.125:0/3503990419 conn(0x7fa9fc06bfe0 msgr2=0x7fa9fc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:20.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.656+0000 7faa04254640 1 -- 172.21.15.125:0/3503990419 shutdown_connections 2023-10-30T12:30:20.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:20.656+0000 7faa04254640 1 -- 172.21.15.125:0/3503990419 wait complete. 2023-10-30T12:30:20.682 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:30:22.684 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:22.721 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 69% (8339/12071) Updating files: 70% (8450/12071) Updating files: 71% (8571/12071) Updating files: 72% (8692/12071) Updating files: 73% (8812/12071) Updating files: 74% (8933/12071) Updating files: 75% (9054/12071) Updating files: 76% (9174/12071) Updating files: 77% (9295/12071) Updating files: 78% (9416/12071) Updating files: 79% (9537/12071) Updating files: 80% (9657/12071) Updating files: 81% (9778/12071) Updating files: 82% (9899/12071) Updating files: 83% (10019/12071) Updating files: 84% (10140/12071) Updating files: 85% (10261/12071) Updating files: 86% (10382/12071) Updating files: 87% (10502/12071) Updating files: 88% (10623/12071) Updating files: 89% (10744/12071) Updating files: 90% (10864/12071) Updating files: 91% (10985/12071) Updating files: 92% (11106/12071) Updating files: 93% (11227/12071) Updating files: 94% (11347/12071) Updating files: 95% (11468/12071) Updating files: 96% (11589/12071) Updating files: 97% (11709/12071) Updating files: 98% (11830/12071) Updating files: 99% (11951/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T12:30:22.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.820+0000 7ff136d3e640 1 Processor -- start 2023-10-30T12:30:22.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.820+0000 7ff136d3e640 1 -- start start 2023-10-30T12:30:22.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff136d3e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff13015d6f0 0x7ff13015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:22.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff136d3e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff1300cf160 0x7ff1300cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:22.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff136d3e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1300cb800 con 0x7ff1301538a0 2023-10-30T12:30:22.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff136d3e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1300cb330 con 0x7ff1300cf160 2023-10-30T12:30:22.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff136d3e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1300cad70 con 0x7ff13015d6f0 2023-10-30T12:30:22.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff134ada640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff1301538a0 0x7ff1300ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44736/0 (socket says 172.21.15.125:44736) 2023-10-30T12:30:22.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff134ada640 1 -- 172.21.15.125:0/3573163357 learned_addr learned my addr 172.21.15.125:0/3573163357 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:30:22.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff1352db640 1 --2- 172.21.15.125:0/3573163357 >> v2:172.21.15.125:3300/0 conn(0x7ff1300cf160 0x7ff1300cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:22.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff12ffff640 1 --2- 172.21.15.125:0/3573163357 >> v2:172.21.15.136:3300/0 conn(0x7ff13015d6f0 0x7ff13015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:22.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff12f7fe640 1 -- 172.21.15.125:0/3573163357 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755894507 0 0) 0x7ff1300cb800 con 0x7ff1301538a0 2023-10-30T12:30:22.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff12f7fe640 1 -- 172.21.15.125:0/3573163357 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff120003530 con 0x7ff1301538a0 2023-10-30T12:30:22.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff1352db640 1 -- 172.21.15.125:0/3573163357 >> v2:172.21.15.136:3300/0 conn(0x7ff13015d6f0 msgr2=0x7ff13015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:22.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff1352db640 1 --2- 172.21.15.125:0/3573163357 >> v2:172.21.15.136:3300/0 conn(0x7ff13015d6f0 0x7ff13015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:22.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff1352db640 1 -- 172.21.15.125:0/3573163357 >> v1:172.21.15.125:6789/0 conn(0x7ff1301538a0 legacy=0x7ff1300ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:22.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff1352db640 1 -- 172.21.15.125:0/3573163357 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1300cb050 con 0x7ff1300cf160 2023-10-30T12:30:22.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.824+0000 7ff12ffff640 1 --2- 172.21.15.125:0/3573163357 >> v2:172.21.15.136:3300/0 conn(0x7ff13015d6f0 0x7ff13015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:30:22.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.828+0000 7ff1352db640 1 --2- 172.21.15.125:0/3573163357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1300cf160 0x7ff1300cf590 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff1300b30a0 tx=0x7ff12801f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c871c52328c74336 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:22.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.828+0000 7ff12f7fe640 1 -- 172.21.15.125:0/3573163357 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1280b0440 con 0x7ff1300cf160 2023-10-30T12:30:22.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.828+0000 7ff12f7fe640 1 -- 172.21.15.125:0/3573163357 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1280b05a0 con 0x7ff1300cf160 2023-10-30T12:30:22.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.828+0000 7ff12f7fe640 1 -- 172.21.15.125:0/3573163357 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1280b0930 con 0x7ff1300cf160 2023-10-30T12:30:22.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.828+0000 7ff136d3e640 1 -- 172.21.15.125:0/3573163357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1300cf160 msgr2=0x7ff1300cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:22.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.828+0000 7ff136d3e640 1 --2- 172.21.15.125:0/3573163357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1300cf160 0x7ff1300cf590 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ff1300b30a0 tx=0x7ff12801f0f0 comp rx=0 tx=0).stop 2023-10-30T12:30:22.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.828+0000 7ff136d3e640 1 -- 172.21.15.125:0/3573163357 shutdown_connections 2023-10-30T12:30:22.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.828+0000 7ff136d3e640 1 --2- 172.21.15.125:0/3573163357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1300cf160 0x7ff1300cf590 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:22.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.828+0000 7ff136d3e640 1 --2- 172.21.15.125:0/3573163357 >> v2:172.21.15.136:3300/0 conn(0x7ff13015d6f0 0x7ff13015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:22.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.828+0000 7ff136d3e640 1 -- 172.21.15.125:0/3573163357 >> 172.21.15.125:0/3573163357 conn(0x7ff13006bfe0 msgr2=0x7ff13006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:22.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.828+0000 7ff136d3e640 1 -- 172.21.15.125:0/3573163357 shutdown_connections 2023-10-30T12:30:22.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.828+0000 7ff136d3e640 1 -- 172.21.15.125:0/3573163357 wait complete. 2023-10-30T12:30:22.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.828+0000 7ff136d3e640 1 Processor -- start 2023-10-30T12:30:22.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.828+0000 7ff136d3e640 1 -- start start 2023-10-30T12:30:22.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff136d3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1300cf160 0x7ff130159100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:22.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff136d3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1301538a0 0x7ff1300b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:22.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff136d3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff13015d6f0 0x7ff13014f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:22.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff136d3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13014eba0 con 0x7ff13015d6f0 2023-10-30T12:30:22.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff136d3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13014ee80 con 0x7ff1300cf160 2023-10-30T12:30:22.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff136d3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1301501a0 con 0x7ff1301538a0 2023-10-30T12:30:22.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff1352db640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff13015d6f0 0x7ff13014f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:22.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff1352db640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff13015d6f0 0x7ff13014f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36646/0 (socket says 172.21.15.125:36646) 2023-10-30T12:30:22.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff1352db640 1 -- 172.21.15.125:0/761617035 learned_addr learned my addr 172.21.15.125:0/761617035 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:22.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff134ada640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1300cf160 0x7ff130159100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:22.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff12ffff640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1301538a0 0x7ff1300b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:22.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff1352db640 1 -- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1301538a0 msgr2=0x7ff1300b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:22.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff1352db640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1301538a0 0x7ff1300b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:22.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff1352db640 1 -- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1300cf160 msgr2=0x7ff130159100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:22.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff1352db640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1300cf160 0x7ff130159100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:22.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff1352db640 1 -- 172.21.15.125:0/761617035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff128009a70 con 0x7ff13015d6f0 2023-10-30T12:30:22.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff12ffff640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1301538a0 0x7ff1300b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:30:22.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff1352db640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff13015d6f0 0x7ff13014f310 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff1280b3040 tx=0x7ff12801be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:22.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff12d7fa640 1 -- 172.21.15.125:0/761617035 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1280b0910 con 0x7ff13015d6f0 2023-10-30T12:30:22.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff12d7fa640 1 -- 172.21.15.125:0/761617035 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1280b0a70 con 0x7ff13015d6f0 2023-10-30T12:30:22.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff136d3e640 1 -- 172.21.15.125:0/761617035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff130150460 con 0x7ff13015d6f0 2023-10-30T12:30:22.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff12d7fa640 1 -- 172.21.15.125:0/761617035 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1280b0e20 con 0x7ff13015d6f0 2023-10-30T12:30:22.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff136d3e640 1 -- 172.21.15.125:0/761617035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff130150a20 con 0x7ff13015d6f0 2023-10-30T12:30:22.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.832+0000 7ff136d3e640 1 -- 172.21.15.125:0/761617035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1300b2a50 con 0x7ff13015d6f0 2023-10-30T12:30:22.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.836+0000 7ff12d7fa640 1 -- 172.21.15.125:0/761617035 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff1280c6070 con 0x7ff13015d6f0 2023-10-30T12:30:22.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.836+0000 7ff12d7fa640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1140664a0 0x7ff114068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:22.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.836+0000 7ff12d7fa640 1 -- 172.21.15.125:0/761617035 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5738+0+0 (secure 0 0 0) 0x7ff12812a880 con 0x7ff13015d6f0 2023-10-30T12:30:22.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.836+0000 7ff134ada640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1140664a0 0x7ff114068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:22.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.836+0000 7ff134ada640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1140664a0 0x7ff114068960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff1240011f0 tx=0x7ff124007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:22.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:22.840+0000 7ff12d7fa640 1 -- 172.21.15.125:0/761617035 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff1280f7e90 con 0x7ff13015d6f0 2023-10-30T12:30:23.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:23.020+0000 7ff136d3e640 1 -- 172.21.15.125:0/761617035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff13015a5a0 con 0x7ff13015d6f0 2023-10-30T12:30:23.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:23.020+0000 7ff12d7fa640 1 -- 172.21.15.125:0/761617035 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+6537 (secure 0 0 0) 0x7ff1280b5070 con 0x7ff13015d6f0 2023-10-30T12:30:23.029 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:23.029 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4265,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:29:01.095107+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4279},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4279":{"gid":4279,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.136:6835/3346501886","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3346501886},{"type":"v1","addr":"172.21.15.136:6835","nonce":3346501886}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:23.029 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 12 2023-10-30T12:30:23.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:23.024+0000 7ff136d3e640 1 -- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1140664a0 msgr2=0x7ff114068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:23.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:23.024+0000 7ff136d3e640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1140664a0 0x7ff114068960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff1240011f0 tx=0x7ff124007d40 comp rx=0 tx=0).stop 2023-10-30T12:30:23.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:23.024+0000 7ff136d3e640 1 -- 172.21.15.125:0/761617035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff13015d6f0 msgr2=0x7ff13014f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:23.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:23.024+0000 7ff136d3e640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff13015d6f0 0x7ff13014f310 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ff1280b3040 tx=0x7ff12801be90 comp rx=0 tx=0).stop 2023-10-30T12:30:23.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:23.024+0000 7ff136d3e640 1 -- 172.21.15.125:0/761617035 shutdown_connections 2023-10-30T12:30:23.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:23.024+0000 7ff136d3e640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1140664a0 0x7ff114068960 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:23.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:23.024+0000 7ff136d3e640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff13015d6f0 0x7ff13014f310 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:23.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:23.024+0000 7ff136d3e640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1301538a0 0x7ff1300b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:23.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:23.024+0000 7ff136d3e640 1 --2- 172.21.15.125:0/761617035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1300cf160 0x7ff130159100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:23.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:23.024+0000 7ff136d3e640 1 -- 172.21.15.125:0/761617035 >> 172.21.15.125:0/761617035 conn(0x7ff13006bfe0 msgr2=0x7ff130152110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:23.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:23.024+0000 7ff136d3e640 1 -- 172.21.15.125:0/761617035 shutdown_connections 2023-10-30T12:30:23.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:23.024+0000 7ff136d3e640 1 -- 172.21.15.125:0/761617035 wait complete. 2023-10-30T12:30:23.050 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:30:23.265 INFO:tasks.workunit.client.0.smithi125.stderr:Note: switching to '00c1c13cea21a686b046584596fbeac5315a7d71'. 2023-10-30T12:30:23.265 INFO:tasks.workunit.client.0.smithi125.stderr: 2023-10-30T12:30:23.265 INFO:tasks.workunit.client.0.smithi125.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-30T12:30:23.266 INFO:tasks.workunit.client.0.smithi125.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-30T12:30:23.266 INFO:tasks.workunit.client.0.smithi125.stderr:state without impacting any branches by switching back to a branch. 2023-10-30T12:30:23.266 INFO:tasks.workunit.client.0.smithi125.stderr: 2023-10-30T12:30:23.266 INFO:tasks.workunit.client.0.smithi125.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-30T12:30:23.266 INFO:tasks.workunit.client.0.smithi125.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-30T12:30:23.267 INFO:tasks.workunit.client.0.smithi125.stderr: 2023-10-30T12:30:23.267 INFO:tasks.workunit.client.0.smithi125.stderr: git switch -c 2023-10-30T12:30:23.267 INFO:tasks.workunit.client.0.smithi125.stderr: 2023-10-30T12:30:23.267 INFO:tasks.workunit.client.0.smithi125.stderr:Or undo this operation with: 2023-10-30T12:30:23.267 INFO:tasks.workunit.client.0.smithi125.stderr: 2023-10-30T12:30:23.267 INFO:tasks.workunit.client.0.smithi125.stderr: git switch - 2023-10-30T12:30:23.268 INFO:tasks.workunit.client.0.smithi125.stderr: 2023-10-30T12:30:23.268 INFO:tasks.workunit.client.0.smithi125.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-30T12:30:23.268 INFO:tasks.workunit.client.0.smithi125.stderr: 2023-10-30T12:30:23.268 INFO:tasks.workunit.client.0.smithi125.stderr:HEAD is now at 00c1c13cea2 mds/MDBalancer: ignore queued callbacks if MDS is not active 2023-10-30T12:30:23.270 DEBUG:teuthology.orchestra.run.smithi125:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-30T12:30:23.279 INFO:tasks.workunit.client.0.smithi125.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-30T12:30:23.283 INFO:tasks.workunit.client.0.smithi125.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T12:30:23.283 INFO:tasks.workunit.client.0.smithi125.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-30T12:30:24.088 INFO:tasks.workunit.client.0.smithi125.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-30T12:30:24.166 INFO:tasks.workunit.client.0.smithi125.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-30T12:30:24.229 INFO:tasks.workunit.client.0.smithi125.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-30T12:30:24.232 INFO:tasks.workunit.client.0.smithi125.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T12:30:24.232 INFO:tasks.workunit.client.0.smithi125.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-30T12:30:24.285 INFO:tasks.workunit.client.0.smithi125.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-30T12:30:24.290 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T12:30:24.291 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-30T12:30:24.342 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2023-10-30T12:30:24.343 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2023-10-30T12:30:24.344 DEBUG:teuthology.orchestra.run.smithi125:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2023-10-30T12:30:25.051 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:25.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.192+0000 7fabb7405640 1 Processor -- start 2023-10-30T12:30:25.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.192+0000 7fabb7405640 1 -- start start 2023-10-30T12:30:25.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7fabb7405640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fabb00c2570 0x7fabb00cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:25.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7fabb7405640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabb00c29d0 0x7fabb00cb610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:25.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7fabb7405640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabb00c68b0 con 0x7fabb0150920 2023-10-30T12:30:25.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7fabb7405640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabb0150d00 con 0x7fabb00c2570 2023-10-30T12:30:25.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7fabb7405640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb00c2e90 con 0x7fabb00c29d0 2023-10-30T12:30:25.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7fabb49a0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fabb00c2570 0x7fabb00cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:25.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7fabb51a1640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fabb0150920 0x7fabb006d420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44748/0 (socket says 172.21.15.125:44748) 2023-10-30T12:30:25.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7fabb51a1640 1 -- 172.21.15.125:0/3126238628 learned_addr learned my addr 172.21.15.125:0/3126238628 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:30:25.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7fabb49a0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fabb00c2570 0x7fabb00cb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36658/0 (socket says 172.21.15.125:36658) 2023-10-30T12:30:25.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7fabb59a2640 1 --2- 172.21.15.125:0/3126238628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabb00c29d0 0x7fabb00cb610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:25.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7fabb49a0640 1 -- 172.21.15.125:0/3126238628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabb00c29d0 msgr2=0x7fabb00cb610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:25.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7fabb49a0640 1 --2- 172.21.15.125:0/3126238628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabb00c29d0 0x7fabb00cb610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7fabb49a0640 1 -- 172.21.15.125:0/3126238628 >> v1:172.21.15.125:6789/0 conn(0x7fabb0150920 legacy=0x7fabb006d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:25.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7fabb49a0640 1 -- 172.21.15.125:0/3126238628 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb00c6b00 con 0x7fabb00c2570 2023-10-30T12:30:25.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.196+0000 7faba7fff640 1 -- 172.21.15.125:0/3126238628 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773450439 0 0) 0x7fabb00c68b0 con 0x7fabb0150920 2023-10-30T12:30:25.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.200+0000 7fabb49a0640 1 --2- 172.21.15.125:0/3126238628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabb00c2570 0x7fabb00cb0d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faba000ba60 tx=0x7faba001eac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=765ce88fc1deab48 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:25.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.200+0000 7faba7fff640 1 -- 172.21.15.125:0/3126238628 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba00a7a80 con 0x7fabb00c2570 2023-10-30T12:30:25.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.200+0000 7faba7fff640 1 -- 172.21.15.125:0/3126238628 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faba00a7be0 con 0x7fabb00c2570 2023-10-30T12:30:25.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.200+0000 7faba7fff640 1 -- 172.21.15.125:0/3126238628 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faba00af440 con 0x7fabb00c2570 2023-10-30T12:30:25.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.200+0000 7fabb7405640 1 -- 172.21.15.125:0/3126238628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabb00c2570 msgr2=0x7fabb00cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:25.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.200+0000 7fabb7405640 1 --2- 172.21.15.125:0/3126238628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabb00c2570 0x7fabb00cb0d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7faba000ba60 tx=0x7faba001eac0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.200+0000 7fabb7405640 1 -- 172.21.15.125:0/3126238628 shutdown_connections 2023-10-30T12:30:25.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.200+0000 7fabb7405640 1 --2- 172.21.15.125:0/3126238628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabb00c2570 0x7fabb00cb0d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.200+0000 7fabb7405640 1 --2- 172.21.15.125:0/3126238628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabb00c29d0 0x7fabb00cb610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.200+0000 7fabb7405640 1 -- 172.21.15.125:0/3126238628 >> 172.21.15.125:0/3126238628 conn(0x7fabb006dfd0 msgr2=0x7fabb006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:25.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.200+0000 7fabb7405640 1 -- 172.21.15.125:0/3126238628 shutdown_connections 2023-10-30T12:30:25.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.200+0000 7fabb7405640 1 -- 172.21.15.125:0/3126238628 wait complete. 2023-10-30T12:30:25.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.200+0000 7fabb7405640 1 Processor -- start 2023-10-30T12:30:25.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.200+0000 7fabb7405640 1 -- start start 2023-10-30T12:30:25.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb7405640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabb00c2570 0x7fabb01454a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:25.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb7405640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fabb00c29d0 0x7fabb00cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:25.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb7405640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabb0150920 0x7fabb00cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:25.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb7405640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb014c3e0 con 0x7fabb00c2570 2023-10-30T12:30:25.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb7405640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb01589f0 con 0x7fabb00c29d0 2023-10-30T12:30:25.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb7405640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb01f16a0 con 0x7fabb0150920 2023-10-30T12:30:25.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb51a1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabb00c2570 0x7fabb01454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:25.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb51a1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabb00c2570 0x7fabb01454a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36670/0 (socket says 172.21.15.125:36670) 2023-10-30T12:30:25.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb51a1640 1 -- 172.21.15.125:0/2555557335 learned_addr learned my addr 172.21.15.125:0/2555557335 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:25.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb49a0640 1 --2- 172.21.15.125:0/2555557335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fabb00c29d0 0x7fabb00cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:25.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb59a2640 1 --2- 172.21.15.125:0/2555557335 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabb0150920 0x7fabb00cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:25.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb51a1640 1 -- 172.21.15.125:0/2555557335 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabb0150920 msgr2=0x7fabb00cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:25.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb51a1640 1 --2- 172.21.15.125:0/2555557335 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabb0150920 0x7fabb00cb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb51a1640 1 -- 172.21.15.125:0/2555557335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fabb00c29d0 msgr2=0x7fabb00cac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:25.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb51a1640 1 --2- 172.21.15.125:0/2555557335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fabb00c29d0 0x7fabb00cac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb51a1640 1 -- 172.21.15.125:0/2555557335 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faba000b6c0 con 0x7fabb00c2570 2023-10-30T12:30:25.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb51a1640 1 --2- 172.21.15.125:0/2555557335 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabb00c2570 0x7fabb01454a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fabac00c480 tx=0x7fabac00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:25.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7faba5ffb640 1 -- 172.21.15.125:0/2555557335 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac0074e0 con 0x7fabb00c2570 2023-10-30T12:30:25.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7faba5ffb640 1 -- 172.21.15.125:0/2555557335 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabac007660 con 0x7fabb00c2570 2023-10-30T12:30:25.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7faba5ffb640 1 -- 172.21.15.125:0/2555557335 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabac004a30 con 0x7fabb00c2570 2023-10-30T12:30:25.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb7405640 1 -- 172.21.15.125:0/2555557335 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabb01f1960 con 0x7fabb00c2570 2023-10-30T12:30:25.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb7405640 1 -- 172.21.15.125:0/2555557335 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb01f1f20 con 0x7fabb00c2570 2023-10-30T12:30:25.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7fabb7405640 1 -- 172.21.15.125:0/2555557335 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabb014dd00 con 0x7fabb00c2570 2023-10-30T12:30:25.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.204+0000 7faba5ffb640 1 -- 172.21.15.125:0/2555557335 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fabac004b90 con 0x7fabb00c2570 2023-10-30T12:30:25.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.208+0000 7faba5ffb640 1 --2- 172.21.15.125:0/2555557335 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fab940664a0 0x7fab94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:25.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.208+0000 7faba5ffb640 1 -- 172.21.15.125:0/2555557335 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fabac082f40 con 0x7fabb00c2570 2023-10-30T12:30:25.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.208+0000 7fabb49a0640 1 --2- 172.21.15.125:0/2555557335 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fab940664a0 0x7fab94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:25.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.208+0000 7fabb49a0640 1 --2- 172.21.15.125:0/2555557335 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fab940664a0 0x7fab94068960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faba00b1000 tx=0x7faba0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:25.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.212+0000 7faba5ffb640 1 -- 172.21.15.125:0/2555557335 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fabac053f20 con 0x7fabb00c2570 2023-10-30T12:30:25.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.392+0000 7fabb7405640 1 -- 172.21.15.125:0/2555557335 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabb004e170 con 0x7fabb00c2570 2023-10-30T12:30:25.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.392+0000 7faba5ffb640 1 -- 172.21.15.125:0/2555557335 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5745 (secure 0 0 0) 0x7fabac053ab0 con 0x7fabb00c2570 2023-10-30T12:30:25.398 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:25.398 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:30:24.948098+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:25.399 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 13 2023-10-30T12:30:25.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.392+0000 7fabb7405640 1 -- 172.21.15.125:0/2555557335 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fab940664a0 msgr2=0x7fab94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.392+0000 7fabb7405640 1 --2- 172.21.15.125:0/2555557335 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fab940664a0 0x7fab94068960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7faba00b1000 tx=0x7faba0008040 comp rx=0 tx=0).stop 2023-10-30T12:30:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.392+0000 7fabb7405640 1 -- 172.21.15.125:0/2555557335 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabb00c2570 msgr2=0x7fabb01454a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.392+0000 7fabb7405640 1 --2- 172.21.15.125:0/2555557335 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabb00c2570 0x7fabb01454a0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fabac00c480 tx=0x7fabac00c940 comp rx=0 tx=0).stop 2023-10-30T12:30:25.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.392+0000 7fabb7405640 1 -- 172.21.15.125:0/2555557335 shutdown_connections 2023-10-30T12:30:25.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.392+0000 7fabb7405640 1 --2- 172.21.15.125:0/2555557335 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fab940664a0 0x7fab94068960 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.392+0000 7fabb7405640 1 --2- 172.21.15.125:0/2555557335 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabb0150920 0x7fabb00cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.392+0000 7fabb7405640 1 --2- 172.21.15.125:0/2555557335 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fabb00c29d0 0x7fabb00cac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.392+0000 7fabb7405640 1 --2- 172.21.15.125:0/2555557335 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabb00c2570 0x7fabb01454a0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.392+0000 7fabb7405640 1 -- 172.21.15.125:0/2555557335 >> 172.21.15.125:0/2555557335 conn(0x7fabb006dfd0 msgr2=0x7fabb0159b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:25.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.392+0000 7fabb7405640 1 -- 172.21.15.125:0/2555557335 shutdown_connections 2023-10-30T12:30:25.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.392+0000 7fabb7405640 1 -- 172.21.15.125:0/2555557335 wait complete. 2023-10-30T12:30:25.419 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T12:30:25.419 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:30:25.420 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:25.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f290d9f5640 1 Processor -- start 2023-10-30T12:30:25.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f290d9f5640 1 -- start start 2023-10-30T12:30:25.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f290d9f5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29080c8cf0 0x7f29080c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:25.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f290d9f5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f29080ca340 0x7f290815d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:25.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f290d9f5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f290815ce30 con 0x7f29080c90d0 2023-10-30T12:30:25.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f290d9f5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2908159880 con 0x7f29080ca340 2023-10-30T12:30:25.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f290d9f5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2908159ae0 con 0x7f29080c8cf0 2023-10-30T12:30:25.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f29077fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f29080ca340 0x7f290815d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:25.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f29067fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f29080c90d0 0x7f29080cef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44756/0 (socket says 172.21.15.125:44756) 2023-10-30T12:30:25.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f29067fc640 1 -- 172.21.15.125:0/2144833185 learned_addr learned my addr 172.21.15.125:0/2144833185 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:30:25.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f2906ffd640 1 --2- 172.21.15.125:0/2144833185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29080c8cf0 0x7f29080c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:25.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f29077fe640 1 -- 172.21.15.125:0/2144833185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29080c8cf0 msgr2=0x7f29080c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:25.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f29077fe640 1 --2- 172.21.15.125:0/2144833185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29080c8cf0 0x7f29080c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f29077fe640 1 -- 172.21.15.125:0/2144833185 >> v1:172.21.15.125:6789/0 conn(0x7f29080c90d0 legacy=0x7f29080cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:25.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f29077fe640 1 -- 172.21.15.125:0/2144833185 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2908159dc0 con 0x7f29080ca340 2023-10-30T12:30:25.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.552+0000 7f2906ffd640 1 --2- 172.21.15.125:0/2144833185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29080c8cf0 0x7f29080c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:30:25.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.556+0000 7f29077fe640 1 --2- 172.21.15.125:0/2144833185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29080ca340 0x7f290815d2c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2900009d90 tx=0x7f290001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4514146e3e437f40 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:25.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.556+0000 7f2905ffb640 1 -- 172.21.15.125:0/2144833185 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29000a59e0 con 0x7f29080ca340 2023-10-30T12:30:25.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.556+0000 7f2905ffb640 1 -- 172.21.15.125:0/2144833185 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29000a5b40 con 0x7f29080ca340 2023-10-30T12:30:25.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.556+0000 7f2905ffb640 1 -- 172.21.15.125:0/2144833185 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29000ad440 con 0x7f29080ca340 2023-10-30T12:30:25.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.556+0000 7f290d9f5640 1 -- 172.21.15.125:0/2144833185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29080ca340 msgr2=0x7f290815d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:25.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.556+0000 7f290d9f5640 1 --2- 172.21.15.125:0/2144833185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29080ca340 0x7f290815d2c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2900009d90 tx=0x7f290001c920 comp rx=0 tx=0).stop 2023-10-30T12:30:25.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.556+0000 7f290d9f5640 1 -- 172.21.15.125:0/2144833185 shutdown_connections 2023-10-30T12:30:25.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.556+0000 7f290d9f5640 1 --2- 172.21.15.125:0/2144833185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29080ca340 0x7f290815d2c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.556+0000 7f290d9f5640 1 --2- 172.21.15.125:0/2144833185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29080c8cf0 0x7f29080c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.556+0000 7f290d9f5640 1 -- 172.21.15.125:0/2144833185 >> 172.21.15.125:0/2144833185 conn(0x7f290806be40 msgr2=0x7f290806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:25.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.556+0000 7f290d9f5640 1 -- 172.21.15.125:0/2144833185 shutdown_connections 2023-10-30T12:30:25.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.556+0000 7f290d9f5640 1 -- 172.21.15.125:0/2144833185 wait complete. 2023-10-30T12:30:25.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.556+0000 7f290d9f5640 1 Processor -- start 2023-10-30T12:30:25.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f290d9f5640 1 -- start start 2023-10-30T12:30:25.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f290d9f5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29080c8cf0 0x7f2908155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:25.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f290d9f5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29080c90d0 0x7f29080cbcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:25.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f290d9f5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29080ca340 0x7f29081575e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:25.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f290d9f5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2908156c50 con 0x7f29080ca340 2023-10-30T12:30:25.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f290d9f5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29080c3b50 con 0x7f29080c8cf0 2023-10-30T12:30:25.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f290d9f5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29080c2fd0 con 0x7f29080c90d0 2023-10-30T12:30:25.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f29077fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29080ca340 0x7f29081575e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:25.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f29077fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29080ca340 0x7f29081575e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36700/0 (socket says 172.21.15.125:36700) 2023-10-30T12:30:25.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f29077fe640 1 -- 172.21.15.125:0/2637216872 learned_addr learned my addr 172.21.15.125:0/2637216872 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:25.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f2906ffd640 1 --2- 172.21.15.125:0/2637216872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29080c8cf0 0x7f2908155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:25.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f29067fc640 1 --2- 172.21.15.125:0/2637216872 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29080c90d0 0x7f29080cbcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:25.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f29077fe640 1 -- 172.21.15.125:0/2637216872 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29080c90d0 msgr2=0x7f29080cbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:25.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f29077fe640 1 --2- 172.21.15.125:0/2637216872 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29080c90d0 0x7f29080cbcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f29077fe640 1 -- 172.21.15.125:0/2637216872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29080c8cf0 msgr2=0x7f2908155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:25.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f29077fe640 1 --2- 172.21.15.125:0/2637216872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29080c8cf0 0x7f2908155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f29077fe640 1 -- 172.21.15.125:0/2637216872 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2900009a70 con 0x7f29080ca340 2023-10-30T12:30:25.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f29077fe640 1 --2- 172.21.15.125:0/2637216872 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29080ca340 0x7f29081575e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f29000b0040 tx=0x7f29000196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:25.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f28effff640 1 -- 172.21.15.125:0/2637216872 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29000ade40 con 0x7f29080ca340 2023-10-30T12:30:25.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f28effff640 1 -- 172.21.15.125:0/2637216872 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2900009450 con 0x7f29080ca340 2023-10-30T12:30:25.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f290d9f5640 1 -- 172.21.15.125:0/2637216872 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29080c32b0 con 0x7f29080ca340 2023-10-30T12:30:25.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f28effff640 1 -- 172.21.15.125:0/2637216872 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29000b7630 con 0x7f29080ca340 2023-10-30T12:30:25.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f290d9f5640 1 -- 172.21.15.125:0/2637216872 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29080c3870 con 0x7f29080ca340 2023-10-30T12:30:25.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.560+0000 7f290d9f5640 1 -- 172.21.15.125:0/2637216872 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f290804e170 con 0x7f29080ca340 2023-10-30T12:30:25.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.564+0000 7f28effff640 1 -- 172.21.15.125:0/2637216872 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29000b4050 con 0x7f29080ca340 2023-10-30T12:30:25.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.564+0000 7f28effff640 1 --2- 172.21.15.125:0/2637216872 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f28f40664a0 0x7f28f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:25.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.564+0000 7f28effff640 1 -- 172.21.15.125:0/2637216872 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f290011f160 con 0x7f29080ca340 2023-10-30T12:30:25.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.564+0000 7f2906ffd640 1 --2- 172.21.15.125:0/2637216872 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f28f40664a0 0x7f28f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:25.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.564+0000 7f2906ffd640 1 --2- 172.21.15.125:0/2637216872 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f28f40664a0 0x7f28f4068960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f28f80045f0 tx=0x7f28f8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:25.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.568+0000 7f28effff640 1 -- 172.21.15.125:0/2637216872 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29000ec610 con 0x7f29080ca340 2023-10-30T12:30:25.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.748+0000 7f290d9f5640 1 -- 172.21.15.125:0/2637216872 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29080b30c0 con 0x7f29080ca340 2023-10-30T12:30:25.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.748+0000 7f28effff640 1 -- 172.21.15.125:0/2637216872 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f29000efc30 con 0x7f29080ca340 2023-10-30T12:30:25.755 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:25.755 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:30:24.948098+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:25.756 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 13 2023-10-30T12:30:25.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.748+0000 7f290d9f5640 1 -- 172.21.15.125:0/2637216872 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f28f40664a0 msgr2=0x7f28f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:25.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.748+0000 7f290d9f5640 1 --2- 172.21.15.125:0/2637216872 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f28f40664a0 0x7f28f4068960 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f28f80045f0 tx=0x7f28f8008040 comp rx=0 tx=0).stop 2023-10-30T12:30:25.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.748+0000 7f290d9f5640 1 -- 172.21.15.125:0/2637216872 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29080ca340 msgr2=0x7f29081575e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:25.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.748+0000 7f290d9f5640 1 --2- 172.21.15.125:0/2637216872 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29080ca340 0x7f29081575e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f29000b0040 tx=0x7f29000196c0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.748+0000 7f290d9f5640 1 -- 172.21.15.125:0/2637216872 shutdown_connections 2023-10-30T12:30:25.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.748+0000 7f290d9f5640 1 --2- 172.21.15.125:0/2637216872 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f28f40664a0 0x7f28f4068960 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.748+0000 7f290d9f5640 1 --2- 172.21.15.125:0/2637216872 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29080ca340 0x7f29081575e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.748+0000 7f290d9f5640 1 --2- 172.21.15.125:0/2637216872 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29080c90d0 0x7f29080cbcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.748+0000 7f290d9f5640 1 --2- 172.21.15.125:0/2637216872 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29080c8cf0 0x7f2908155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:25.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.748+0000 7f290d9f5640 1 -- 172.21.15.125:0/2637216872 >> 172.21.15.125:0/2637216872 conn(0x7f290806be40 msgr2=0x7f29080cabf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:25.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.752+0000 7f290d9f5640 1 -- 172.21.15.125:0/2637216872 shutdown_connections 2023-10-30T12:30:25.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:25.752+0000 7f290d9f5640 1 -- 172.21.15.125:0/2637216872 wait complete. 2023-10-30T12:30:25.776 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 13, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 13, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4265": { "addr": "172.21.15.125:6837/2291877604", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2291877604, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2291877604, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4265, "incarnation": 13, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_4285": { "addr": "172.21.15.136:6839/2250748765", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 2250748765, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 2250748765, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4294": { "addr": "172.21.15.125:6836/2017382874", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2017382874, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2017382874, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4294, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:30:24.948098+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4294, "mds_1": 4285, "mds_2": 4265 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:30:27.777 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:27.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.916+0000 7f852393e640 1 Processor -- start 2023-10-30T12:30:27.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.916+0000 7f852393e640 1 -- start start 2023-10-30T12:30:27.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f852393e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f851c0c05d0 0x7f851c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:27.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f852393e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f851c0c6f60 0x7f851c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:27.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f852393e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f851c0c0a50 con 0x7f851c0c6b80 2023-10-30T12:30:27.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f852393e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f851c06f6e0 con 0x7f851c0c05d0 2023-10-30T12:30:27.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f852393e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f851c06f9c0 con 0x7f851c0c6f60 2023-10-30T12:30:27.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f8520ed9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f851c0c05d0 0x7f851c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:27.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f85216da640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f851c0c6b80 0x7f851c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44772/0 (socket says 172.21.15.125:44772) 2023-10-30T12:30:27.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f85216da640 1 -- 172.21.15.125:0/1697567345 learned_addr learned my addr 172.21.15.125:0/1697567345 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:30:27.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f8520ed9640 1 --2- 172.21.15.125:0/1697567345 >> v2:172.21.15.125:3300/0 conn(0x7f851c0c05d0 0x7f851c13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36710/0 (socket says 172.21.15.125:36710) 2023-10-30T12:30:27.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f8521edb640 1 --2- 172.21.15.125:0/1697567345 >> v2:172.21.15.136:3300/0 conn(0x7f851c0c6f60 0x7f851c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:27.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f8520ed9640 1 -- 172.21.15.125:0/1697567345 >> v2:172.21.15.136:3300/0 conn(0x7f851c0c6f60 msgr2=0x7f851c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:27.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f8520ed9640 1 --2- 172.21.15.125:0/1697567345 >> v2:172.21.15.136:3300/0 conn(0x7f851c0c6f60 0x7f851c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:27.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f8520ed9640 1 -- 172.21.15.125:0/1697567345 >> v1:172.21.15.125:6789/0 conn(0x7f851c0c6b80 legacy=0x7f851c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:27.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f8520ed9640 1 -- 172.21.15.125:0/1697567345 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f851c06fca0 con 0x7f851c0c05d0 2023-10-30T12:30:27.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f8513fff640 1 -- 172.21.15.125:0/1697567345 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181266306 0 0) 0x7f851c0c0a50 con 0x7f851c0c6b80 2023-10-30T12:30:27.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f8520ed9640 1 --2- 172.21.15.125:0/1697567345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f851c0c05d0 0x7f851c13ed90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f851c0c89b0 tx=0x7f850c01eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f45392dd64539948 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:27.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.920+0000 7f8513fff640 1 -- 172.21.15.125:0/1697567345 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c0a7b10 con 0x7f851c0c05d0 2023-10-30T12:30:27.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f8513fff640 1 -- 172.21.15.125:0/1697567345 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f850c0a7c70 con 0x7f851c0c05d0 2023-10-30T12:30:27.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f8513fff640 1 -- 172.21.15.125:0/1697567345 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850c0af440 con 0x7f851c0c05d0 2023-10-30T12:30:27.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 -- 172.21.15.125:0/1697567345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f851c0c05d0 msgr2=0x7f851c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:27.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 --2- 172.21.15.125:0/1697567345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f851c0c05d0 0x7f851c13ed90 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f851c0c89b0 tx=0x7f850c01eb30 comp rx=0 tx=0).stop 2023-10-30T12:30:27.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 -- 172.21.15.125:0/1697567345 shutdown_connections 2023-10-30T12:30:27.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 --2- 172.21.15.125:0/1697567345 >> v2:172.21.15.136:3300/0 conn(0x7f851c0c6f60 0x7f851c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:27.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 --2- 172.21.15.125:0/1697567345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f851c0c05d0 0x7f851c13ed90 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:27.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 -- 172.21.15.125:0/1697567345 >> 172.21.15.125:0/1697567345 conn(0x7f851c0c7910 msgr2=0x7f851c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:27.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 -- 172.21.15.125:0/1697567345 shutdown_connections 2023-10-30T12:30:27.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 -- 172.21.15.125:0/1697567345 wait complete. 2023-10-30T12:30:27.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 Processor -- start 2023-10-30T12:30:27.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 -- start start 2023-10-30T12:30:27.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f851c0c05d0 0x7f851c13e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:27.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f851c0c6b80 0x7f851c0ca300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:27.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f851c0c6f60 0x7f851c0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:27.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851c139b00 con 0x7f851c0c6f60 2023-10-30T12:30:27.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f851c13e6b0 con 0x7f851c0c6b80 2023-10-30T12:30:27.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f852393e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f851c13d830 con 0x7f851c0c05d0 2023-10-30T12:30:27.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f8521edb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f851c0c6f60 0x7f851c0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:27.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f8521edb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f851c0c6f60 0x7f851c0ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36720/0 (socket says 172.21.15.125:36720) 2023-10-30T12:30:27.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f8521edb640 1 -- 172.21.15.125:0/761999320 learned_addr learned my addr 172.21.15.125:0/761999320 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:27.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f85216da640 1 --2- 172.21.15.125:0/761999320 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f851c0c05d0 0x7f851c13e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:27.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.924+0000 7f8520ed9640 1 --2- 172.21.15.125:0/761999320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f851c0c6b80 0x7f851c0ca300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:27.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f8521edb640 1 -- 172.21.15.125:0/761999320 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f851c0c05d0 msgr2=0x7f851c13e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:27.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f8521edb640 1 --2- 172.21.15.125:0/761999320 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f851c0c05d0 0x7f851c13e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:27.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f8521edb640 1 -- 172.21.15.125:0/761999320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f851c0c6b80 msgr2=0x7f851c0ca300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:27.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f8521edb640 1 --2- 172.21.15.125:0/761999320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f851c0c6b80 0x7f851c0ca300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:27.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f8521edb640 1 -- 172.21.15.125:0/761999320 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c00b6c0 con 0x7f851c0c6f60 2023-10-30T12:30:27.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f8521edb640 1 --2- 172.21.15.125:0/761999320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f851c0c6f60 0x7f851c0ca840 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8514007e60 tx=0x7f851400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:27.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f8511ffb640 1 -- 172.21.15.125:0/761999320 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8514014420 con 0x7f851c0c6f60 2023-10-30T12:30:27.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f8511ffb640 1 -- 172.21.15.125:0/761999320 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85140145a0 con 0x7f851c0c6f60 2023-10-30T12:30:27.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f852393e640 1 -- 172.21.15.125:0/761999320 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f851c1346e0 con 0x7f851c0c6f60 2023-10-30T12:30:27.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f8511ffb640 1 -- 172.21.15.125:0/761999320 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8514014950 con 0x7f851c0c6f60 2023-10-30T12:30:27.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f852393e640 1 -- 172.21.15.125:0/761999320 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f851c1562c0 con 0x7f851c0c6f60 2023-10-30T12:30:27.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f852393e640 1 -- 172.21.15.125:0/761999320 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f851c1544a0 con 0x7f851c0c6f60 2023-10-30T12:30:27.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f8511ffb640 1 -- 172.21.15.125:0/761999320 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8514014ab0 con 0x7f851c0c6f60 2023-10-30T12:30:27.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f8511ffb640 1 --2- 172.21.15.125:0/761999320 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85000664a0 0x7f8500068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:27.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f8511ffb640 1 -- 172.21.15.125:0/761999320 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f8514084760 con 0x7f851c0c6f60 2023-10-30T12:30:27.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.928+0000 7f85216da640 1 --2- 172.21.15.125:0/761999320 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85000664a0 0x7f8500068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:27.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.932+0000 7f85216da640 1 --2- 172.21.15.125:0/761999320 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85000664a0 0x7f8500068960 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8518007fd0 tx=0x7f8518007d60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:27.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:27.932+0000 7f8511ffb640 1 -- 172.21.15.125:0/761999320 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f851c1544a0 con 0x7f851c0c6f60 2023-10-30T12:30:28.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:28.120+0000 7f852393e640 1 -- 172.21.15.125:0/761999320 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f851c1544a0 con 0x7f851c0c6f60 2023-10-30T12:30:28.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:28.124+0000 7f8511ffb640 1 -- 172.21.15.125:0/761999320 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f851c1544a0 con 0x7f851c0c6f60 2023-10-30T12:30:28.129 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:28.130 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:30:24.948098+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:28.130 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 13 2023-10-30T12:30:28.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:28.124+0000 7f852393e640 1 -- 172.21.15.125:0/761999320 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85000664a0 msgr2=0x7f8500068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:28.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:28.124+0000 7f852393e640 1 --2- 172.21.15.125:0/761999320 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85000664a0 0x7f8500068960 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f8518007fd0 tx=0x7f8518007d60 comp rx=0 tx=0).stop 2023-10-30T12:30:28.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:28.124+0000 7f852393e640 1 -- 172.21.15.125:0/761999320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f851c0c6f60 msgr2=0x7f851c0ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:28.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:28.124+0000 7f852393e640 1 --2- 172.21.15.125:0/761999320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f851c0c6f60 0x7f851c0ca840 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f8514007e60 tx=0x7f851400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:30:28.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:28.124+0000 7f852393e640 1 -- 172.21.15.125:0/761999320 shutdown_connections 2023-10-30T12:30:28.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:28.124+0000 7f852393e640 1 --2- 172.21.15.125:0/761999320 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85000664a0 0x7f8500068960 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:28.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:28.124+0000 7f852393e640 1 --2- 172.21.15.125:0/761999320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f851c0c6f60 0x7f851c0ca840 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:28.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:28.124+0000 7f852393e640 1 --2- 172.21.15.125:0/761999320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f851c0c6b80 0x7f851c0ca300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:28.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:28.124+0000 7f852393e640 1 --2- 172.21.15.125:0/761999320 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f851c0c05d0 0x7f851c13e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:28.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:28.124+0000 7f852393e640 1 -- 172.21.15.125:0/761999320 >> 172.21.15.125:0/761999320 conn(0x7f851c0c7910 msgr2=0x7f851c0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:28.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:28.124+0000 7f852393e640 1 -- 172.21.15.125:0/761999320 shutdown_connections 2023-10-30T12:30:28.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:28.124+0000 7f852393e640 1 -- 172.21.15.125:0/761999320 wait complete. 2023-10-30T12:30:28.150 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:30:30.151 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:30.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.288+0000 7f4060365640 1 Processor -- start 2023-10-30T12:30:30.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.288+0000 7f4060365640 1 -- start start 2023-10-30T12:30:30.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.292+0000 7f4060365640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f40580c6bd0 0x7f405806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:30.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.292+0000 7f4060365640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4058066cf0 con 0x7f405806f7a0 2023-10-30T12:30:30.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.292+0000 7f4060365640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40580c70d0 con 0x7f4058071110 2023-10-30T12:30:30.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.292+0000 7f4060365640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40580700c0 con 0x7f40580c6bd0 2023-10-30T12:30:30.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.292+0000 7f405d900640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f40580c6bd0 0x7f405806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:30.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.292+0000 7f405d900640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f40580c6bd0 0x7f405806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36730/0 (socket says 172.21.15.125:36730) 2023-10-30T12:30:30.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.292+0000 7f405d900640 1 -- 172.21.15.125:0/2687036835 learned_addr learned my addr 172.21.15.125:0/2687036835 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:30.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.292+0000 7f405e101640 1 --1- 172.21.15.125:0/2687036835 >> v1:172.21.15.125:6789/0 conn(0x7f405806f7a0 0x7f4058069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44782/0 (socket says 172.21.15.125:44782) 2023-10-30T12:30:30.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.292+0000 7f405d900640 1 -- 172.21.15.125:0/2687036835 >> v1:172.21.15.136:6789/0 conn(0x7f4058071110 legacy=0x7f4058158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:30.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.292+0000 7f405d900640 1 -- 172.21.15.125:0/2687036835 >> v1:172.21.15.125:6789/0 conn(0x7f405806f7a0 legacy=0x7f4058069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:30.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.292+0000 7f405d900640 1 -- 172.21.15.125:0/2687036835 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40580c7390 con 0x7f40580c6bd0 2023-10-30T12:30:30.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.296+0000 7f405d900640 1 --2- 172.21.15.125:0/2687036835 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40580c6bd0 0x7f405806fb80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4048009870 tx=0x7f404801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59da8a06ce04b80 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:30.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.296+0000 7f405d0ff640 1 -- 172.21.15.125:0/2687036835 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40480a89f0 con 0x7f40580c6bd0 2023-10-30T12:30:30.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.296+0000 7f405d0ff640 1 -- 172.21.15.125:0/2687036835 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40480a8b50 con 0x7f40580c6bd0 2023-10-30T12:30:30.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.296+0000 7f405d0ff640 1 -- 172.21.15.125:0/2687036835 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40480b0440 con 0x7f40580c6bd0 2023-10-30T12:30:30.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.296+0000 7f4060365640 1 -- 172.21.15.125:0/2687036835 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40580c6bd0 msgr2=0x7f405806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:30.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.296+0000 7f4060365640 1 --2- 172.21.15.125:0/2687036835 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40580c6bd0 0x7f405806fb80 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4048009870 tx=0x7f404801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:30:30.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.296+0000 7f4060365640 1 -- 172.21.15.125:0/2687036835 shutdown_connections 2023-10-30T12:30:30.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.296+0000 7f4060365640 1 --2- 172.21.15.125:0/2687036835 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40580c6bd0 0x7f405806fb80 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:30.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.296+0000 7f4060365640 1 -- 172.21.15.125:0/2687036835 >> 172.21.15.125:0/2687036835 conn(0x7f40580c7910 msgr2=0x7f40580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:30.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.296+0000 7f4060365640 1 -- 172.21.15.125:0/2687036835 shutdown_connections 2023-10-30T12:30:30.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.296+0000 7f4060365640 1 -- 172.21.15.125:0/2687036835 wait complete. 2023-10-30T12:30:30.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.296+0000 7f4060365640 1 Processor -- start 2023-10-30T12:30:30.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.296+0000 7f4060365640 1 -- start start 2023-10-30T12:30:30.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f4060365640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f405806f7a0 0x7f40581ef9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:30.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f4060365640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4058071110 0x7f40580ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:30.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f4060365640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40580c6bd0 0x7f405814d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:30.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f4060365640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40581eff30 con 0x7f4058071110 2023-10-30T12:30:30.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f4060365640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40581eef70 con 0x7f405806f7a0 2023-10-30T12:30:30.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f4060365640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40580af560 con 0x7f40580c6bd0 2023-10-30T12:30:30.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f405d900640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4058071110 0x7f40580ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:30.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f405d900640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4058071110 0x7f40580ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36732/0 (socket says 172.21.15.125:36732) 2023-10-30T12:30:30.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f405d900640 1 -- 172.21.15.125:0/1941419960 learned_addr learned my addr 172.21.15.125:0/1941419960 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:30.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f405e101640 1 --2- 172.21.15.125:0/1941419960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f405806f7a0 0x7f40581ef9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:30.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f405e902640 1 --2- 172.21.15.125:0/1941419960 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40580c6bd0 0x7f405814d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:30.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f405d900640 1 -- 172.21.15.125:0/1941419960 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40580c6bd0 msgr2=0x7f405814d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:30.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f405d900640 1 --2- 172.21.15.125:0/1941419960 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40580c6bd0 0x7f405814d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:30.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f405d900640 1 -- 172.21.15.125:0/1941419960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f405806f7a0 msgr2=0x7f40581ef9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:30.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f405d900640 1 --2- 172.21.15.125:0/1941419960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f405806f7a0 0x7f40581ef9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:30.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f405d900640 1 -- 172.21.15.125:0/1941419960 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40480094d0 con 0x7f4058071110 2023-10-30T12:30:30.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f405d900640 1 --2- 172.21.15.125:0/1941419960 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4058071110 0x7f40580ad8d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f40480b3040 tx=0x7f404801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:30.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f404effd640 1 -- 172.21.15.125:0/1941419960 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40480b9620 con 0x7f4058071110 2023-10-30T12:30:30.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f404effd640 1 -- 172.21.15.125:0/1941419960 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40480b97c0 con 0x7f4058071110 2023-10-30T12:30:30.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f404effd640 1 -- 172.21.15.125:0/1941419960 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40480b9b70 con 0x7f4058071110 2023-10-30T12:30:30.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f4060365640 1 -- 172.21.15.125:0/1941419960 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40580af840 con 0x7f4058071110 2023-10-30T12:30:30.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f4060365640 1 -- 172.21.15.125:0/1941419960 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40581ef1f0 con 0x7f4058071110 2023-10-30T12:30:30.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f4060365640 1 -- 172.21.15.125:0/1941419960 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405804e170 con 0x7f4058071110 2023-10-30T12:30:30.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.300+0000 7f404effd640 1 -- 172.21.15.125:0/1941419960 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f40480b7070 con 0x7f4058071110 2023-10-30T12:30:30.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.304+0000 7f404effd640 1 --2- 172.21.15.125:0/1941419960 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f403c0664d0 0x7f403c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:30.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.304+0000 7f404effd640 1 -- 172.21.15.125:0/1941419960 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f4048121ed0 con 0x7f4058071110 2023-10-30T12:30:30.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.304+0000 7f405e101640 1 --2- 172.21.15.125:0/1941419960 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f403c0664d0 0x7f403c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:30.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.304+0000 7f405e101640 1 --2- 172.21.15.125:0/1941419960 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f403c0664d0 0x7f403c068990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4054000f70 tx=0x7f4054007c90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:30.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.308+0000 7f404effd640 1 -- 172.21.15.125:0/1941419960 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f40480ef2f0 con 0x7f4058071110 2023-10-30T12:30:30.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.488+0000 7f4060365640 1 -- 172.21.15.125:0/1941419960 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f405804e170 con 0x7f4058071110 2023-10-30T12:30:30.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.488+0000 7f404effd640 1 -- 172.21.15.125:0/1941419960 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+5745 (secure 0 0 0) 0x7f40480f2910 con 0x7f4058071110 2023-10-30T12:30:30.496 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:30.497 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:30:24.948098+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:30.497 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 13 2023-10-30T12:30:30.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.492+0000 7f4060365640 1 -- 172.21.15.125:0/1941419960 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f403c0664d0 msgr2=0x7f403c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:30.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.492+0000 7f4060365640 1 --2- 172.21.15.125:0/1941419960 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f403c0664d0 0x7f403c068990 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4054000f70 tx=0x7f4054007c90 comp rx=0 tx=0).stop 2023-10-30T12:30:30.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.492+0000 7f4060365640 1 -- 172.21.15.125:0/1941419960 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4058071110 msgr2=0x7f40580ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:30.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.492+0000 7f4060365640 1 --2- 172.21.15.125:0/1941419960 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4058071110 0x7f40580ad8d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f40480b3040 tx=0x7f404801ba90 comp rx=0 tx=0).stop 2023-10-30T12:30:30.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.492+0000 7f4060365640 1 -- 172.21.15.125:0/1941419960 shutdown_connections 2023-10-30T12:30:30.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.492+0000 7f4060365640 1 --2- 172.21.15.125:0/1941419960 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f403c0664d0 0x7f403c068990 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:30.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.492+0000 7f4060365640 1 --2- 172.21.15.125:0/1941419960 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40580c6bd0 0x7f405814d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:30.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.492+0000 7f4060365640 1 --2- 172.21.15.125:0/1941419960 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4058071110 0x7f40580ad8d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:30.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.492+0000 7f4060365640 1 --2- 172.21.15.125:0/1941419960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f405806f7a0 0x7f40581ef9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:30.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.492+0000 7f4060365640 1 -- 172.21.15.125:0/1941419960 >> 172.21.15.125:0/1941419960 conn(0x7f40580c7910 msgr2=0x7f40580c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:30.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.492+0000 7f4060365640 1 -- 172.21.15.125:0/1941419960 shutdown_connections 2023-10-30T12:30:30.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:30.492+0000 7f4060365640 1 -- 172.21.15.125:0/1941419960 wait complete. 2023-10-30T12:30:30.517 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:30:32.518 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:32.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98c190f640 1 Processor -- start 2023-10-30T12:30:32.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98c190f640 1 -- start start 2023-10-30T12:30:32.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98c190f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f98bc153c30 0x7f98bc06a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:32.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98c190f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98bc0b1ea0 0x7f98bc0cb9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:32.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98c190f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98bc06ade0 con 0x7f98bc06b4a0 2023-10-30T12:30:32.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98c190f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98bc1596e0 con 0x7f98bc153c30 2023-10-30T12:30:32.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98c190f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc1598f0 con 0x7f98bc0b1ea0 2023-10-30T12:30:32.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98baffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f98bc153c30 0x7f98bc06a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:32.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98bb7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f98bc06b4a0 0x7f98bc0cabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33660/0 (socket says 172.21.15.125:33660) 2023-10-30T12:30:32.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98bb7fe640 1 -- 172.21.15.125:0/2638900832 learned_addr learned my addr 172.21.15.125:0/2638900832 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:30:32.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98baffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f98bc153c30 0x7f98bc06a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51690/0 (socket says 172.21.15.125:51690) 2023-10-30T12:30:32.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98b27fc640 1 --2- 172.21.15.125:0/2638900832 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98bc0b1ea0 0x7f98bc0cb9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:32.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98baffd640 1 -- 172.21.15.125:0/2638900832 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98bc0b1ea0 msgr2=0x7f98bc0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:32.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98baffd640 1 --2- 172.21.15.125:0/2638900832 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98bc0b1ea0 0x7f98bc0cb9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:32.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98baffd640 1 -- 172.21.15.125:0/2638900832 >> v1:172.21.15.125:6789/0 conn(0x7f98bc06b4a0 legacy=0x7f98bc0cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:32.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98baffd640 1 -- 172.21.15.125:0/2638900832 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc159b90 con 0x7f98bc153c30 2023-10-30T12:30:32.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.656+0000 7f98ba7fc640 1 -- 172.21.15.125:0/2638900832 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318789710 0 0) 0x7f98b4001120 con 0x7f98bc06b4a0 2023-10-30T12:30:32.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.660+0000 7f98baffd640 1 --2- 172.21.15.125:0/2638900832 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98bc153c30 0x7f98bc06a8a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f98ac009d90 tx=0x7f98ac01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3c57fcca7bbb00 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:32.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.660+0000 7f98ba7fc640 1 -- 172.21.15.125:0/2638900832 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac0af600 con 0x7f98bc153c30 2023-10-30T12:30:32.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.660+0000 7f98ba7fc640 1 -- 172.21.15.125:0/2638900832 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ac0ab050 con 0x7f98bc153c30 2023-10-30T12:30:32.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.660+0000 7f98ba7fc640 1 -- 172.21.15.125:0/2638900832 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac0af980 con 0x7f98bc153c30 2023-10-30T12:30:32.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.660+0000 7f98c190f640 1 -- 172.21.15.125:0/2638900832 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98bc153c30 msgr2=0x7f98bc06a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:32.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.660+0000 7f98c190f640 1 --2- 172.21.15.125:0/2638900832 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98bc153c30 0x7f98bc06a8a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f98ac009d90 tx=0x7f98ac01f130 comp rx=0 tx=0).stop 2023-10-30T12:30:32.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.660+0000 7f98c190f640 1 -- 172.21.15.125:0/2638900832 shutdown_connections 2023-10-30T12:30:32.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.660+0000 7f98c190f640 1 --2- 172.21.15.125:0/2638900832 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98bc0b1ea0 0x7f98bc0cb9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:32.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.660+0000 7f98c190f640 1 --2- 172.21.15.125:0/2638900832 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98bc153c30 0x7f98bc06a8a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:32.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.660+0000 7f98c190f640 1 -- 172.21.15.125:0/2638900832 >> 172.21.15.125:0/2638900832 conn(0x7f98bc06bf40 msgr2=0x7f98bc06c350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:32.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.660+0000 7f98c190f640 1 -- 172.21.15.125:0/2638900832 shutdown_connections 2023-10-30T12:30:32.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.660+0000 7f98c190f640 1 -- 172.21.15.125:0/2638900832 wait complete. 2023-10-30T12:30:32.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98c190f640 1 Processor -- start 2023-10-30T12:30:32.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98c190f640 1 -- start start 2023-10-30T12:30:32.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98c190f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98bc06b4a0 0x7f98bc0ce5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:32.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98c190f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98bc0b1ea0 0x7f98bc1f2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:32.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98c190f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98bc153c30 0x7f98bc06a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:32.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98c190f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc1f2d80 con 0x7f98bc0b1ea0 2023-10-30T12:30:32.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98c190f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc0c7210 con 0x7f98bc06b4a0 2023-10-30T12:30:32.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98c190f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc0cec00 con 0x7f98bc153c30 2023-10-30T12:30:32.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98b27fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98bc0b1ea0 0x7f98bc1f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:32.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98b27fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98bc0b1ea0 0x7f98bc1f2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51706/0 (socket says 172.21.15.125:51706) 2023-10-30T12:30:32.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98b27fc640 1 -- 172.21.15.125:0/1917467988 learned_addr learned my addr 172.21.15.125:0/1917467988 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:32.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98baffd640 1 --2- 172.21.15.125:0/1917467988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98bc06b4a0 0x7f98bc0ce5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:32.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98bb7fe640 1 --2- 172.21.15.125:0/1917467988 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98bc153c30 0x7f98bc06a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:32.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98b27fc640 1 -- 172.21.15.125:0/1917467988 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98bc153c30 msgr2=0x7f98bc06a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:32.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98b27fc640 1 --2- 172.21.15.125:0/1917467988 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98bc153c30 0x7f98bc06a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:32.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98b27fc640 1 -- 172.21.15.125:0/1917467988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98bc06b4a0 msgr2=0x7f98bc0ce5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:32.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98b27fc640 1 --2- 172.21.15.125:0/1917467988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98bc06b4a0 0x7f98bc0ce5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:32.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98b27fc640 1 -- 172.21.15.125:0/1917467988 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac009a70 con 0x7f98bc0b1ea0 2023-10-30T12:30:32.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98b27fc640 1 --2- 172.21.15.125:0/1917467988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98bc0b1ea0 0x7f98bc1f2840 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f98a400d800 tx=0x7f98a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:32.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98b8ff9640 1 -- 172.21.15.125:0/1917467988 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4018070 con 0x7f98bc0b1ea0 2023-10-30T12:30:32.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98b8ff9640 1 -- 172.21.15.125:0/1917467988 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a40059a0 con 0x7f98bc0b1ea0 2023-10-30T12:30:32.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98b8ff9640 1 -- 172.21.15.125:0/1917467988 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4005b50 con 0x7f98bc0b1ea0 2023-10-30T12:30:32.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98c190f640 1 -- 172.21.15.125:0/1917467988 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98bc0c7930 con 0x7f98bc0b1ea0 2023-10-30T12:30:32.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98c190f640 1 -- 172.21.15.125:0/1917467988 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98bc0c7ef0 con 0x7f98bc0b1ea0 2023-10-30T12:30:32.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.664+0000 7f98c190f640 1 -- 172.21.15.125:0/1917467988 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98bc04e170 con 0x7f98bc0b1ea0 2023-10-30T12:30:32.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.668+0000 7f98b8ff9640 1 -- 172.21.15.125:0/1917467988 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f98a4005cb0 con 0x7f98bc0b1ea0 2023-10-30T12:30:32.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.668+0000 7f98b8ff9640 1 --2- 172.21.15.125:0/1917467988 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f989c0664a0 0x7f989c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:32.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.668+0000 7f98b8ff9640 1 -- 172.21.15.125:0/1917467988 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f98a4083eb0 con 0x7f98bc0b1ea0 2023-10-30T12:30:32.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.668+0000 7f98baffd640 1 --2- 172.21.15.125:0/1917467988 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f989c0664a0 0x7f989c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:32.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.668+0000 7f98baffd640 1 --2- 172.21.15.125:0/1917467988 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f989c0664a0 0x7f989c068960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0b1000 tx=0x7f98ac0aa000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:32.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.672+0000 7f98b8ff9640 1 -- 172.21.15.125:0/1917467988 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f98a40511b0 con 0x7f98bc0b1ea0 2023-10-30T12:30:32.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.856+0000 7f98c190f640 1 -- 172.21.15.125:0/1917467988 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98bc0c9440 con 0x7f98bc0b1ea0 2023-10-30T12:30:32.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.856+0000 7f98b8ff9640 1 -- 172.21.15.125:0/1917467988 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5747 (secure 0 0 0) 0x7f98a40547d0 con 0x7f98bc0b1ea0 2023-10-30T12:30:32.864 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:32.864 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:30:29.968105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:resolve","state_seq":29,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:32.865 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 14 2023-10-30T12:30:32.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.860+0000 7f98c190f640 1 -- 172.21.15.125:0/1917467988 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f989c0664a0 msgr2=0x7f989c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:32.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.860+0000 7f98c190f640 1 --2- 172.21.15.125:0/1917467988 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f989c0664a0 0x7f989c068960 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0b1000 tx=0x7f98ac0aa000 comp rx=0 tx=0).stop 2023-10-30T12:30:32.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.860+0000 7f98c190f640 1 -- 172.21.15.125:0/1917467988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98bc0b1ea0 msgr2=0x7f98bc1f2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:32.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.860+0000 7f98c190f640 1 --2- 172.21.15.125:0/1917467988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98bc0b1ea0 0x7f98bc1f2840 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f98a400d800 tx=0x7f98a400dcc0 comp rx=0 tx=0).stop 2023-10-30T12:30:32.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.860+0000 7f98c190f640 1 -- 172.21.15.125:0/1917467988 shutdown_connections 2023-10-30T12:30:32.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.860+0000 7f98c190f640 1 --2- 172.21.15.125:0/1917467988 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f989c0664a0 0x7f989c068960 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:32.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.860+0000 7f98c190f640 1 --2- 172.21.15.125:0/1917467988 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98bc153c30 0x7f98bc06a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:32.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.860+0000 7f98c190f640 1 --2- 172.21.15.125:0/1917467988 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98bc0b1ea0 0x7f98bc1f2840 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:32.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.860+0000 7f98c190f640 1 --2- 172.21.15.125:0/1917467988 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98bc06b4a0 0x7f98bc0ce5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:32.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.860+0000 7f98c190f640 1 -- 172.21.15.125:0/1917467988 >> 172.21.15.125:0/1917467988 conn(0x7f98bc06bf40 msgr2=0x7f98bc0c4750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:32.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.860+0000 7f98c190f640 1 -- 172.21.15.125:0/1917467988 shutdown_connections 2023-10-30T12:30:32.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:32.860+0000 7f98c190f640 1 -- 172.21.15.125:0/1917467988 wait complete. 2023-10-30T12:30:32.885 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:30:34.886 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:35.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc7897d3640 1 Processor -- start 2023-10-30T12:30:35.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc7897d3640 1 -- start start 2023-10-30T12:30:35.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc7897d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc7841505b0 0x7fc7840c29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:35.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc7897d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc7841568e0 0x7fc784156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:35.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc7897d3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7840c2f30 con 0x7fc7840c3140 2023-10-30T12:30:35.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc7897d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7840c6a10 con 0x7fc7841568e0 2023-10-30T12:30:35.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc7897d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc784150990 con 0x7fc7841505b0 2023-10-30T12:30:35.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc7837fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc7841568e0 0x7fc784156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:35.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc7837fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc7841568e0 0x7fc784156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51712/0 (socket says 172.21.15.125:51712) 2023-10-30T12:30:35.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc7837fe640 1 -- 172.21.15.125:0/55041109 learned_addr learned my addr 172.21.15.125:0/55041109 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:35.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc782ffd640 1 --2- 172.21.15.125:0/55041109 >> v2:172.21.15.136:3300/0 conn(0x7fc7841505b0 0x7fc7840c29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:35.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc7837fe640 1 -- 172.21.15.125:0/55041109 >> v2:172.21.15.136:3300/0 conn(0x7fc7841505b0 msgr2=0x7fc7840c29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:35.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc7837fe640 1 --2- 172.21.15.125:0/55041109 >> v2:172.21.15.136:3300/0 conn(0x7fc7841505b0 0x7fc7840c29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:35.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc7837fe640 1 -- 172.21.15.125:0/55041109 >> v1:172.21.15.136:6789/0 conn(0x7fc7840c3140 legacy=0x7fc78406cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:35.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc7837fe640 1 -- 172.21.15.125:0/55041109 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7840c7220 con 0x7fc7841568e0 2023-10-30T12:30:35.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.028+0000 7fc782ffd640 1 --2- 172.21.15.125:0/55041109 >> v2:172.21.15.136:3300/0 conn(0x7fc7841505b0 0x7fc7840c29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:30:35.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.032+0000 7fc7837fe640 1 --2- 172.21.15.125:0/55041109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7841568e0 0x7fc784156ce0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc78406df40 tx=0x7fc77c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8595207c6cbd6e0e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:35.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.032+0000 7fc781ffb640 1 -- 172.21.15.125:0/55041109 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c0a56a0 con 0x7fc7841568e0 2023-10-30T12:30:35.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.032+0000 7fc781ffb640 1 -- 172.21.15.125:0/55041109 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc77c0a5800 con 0x7fc7841568e0 2023-10-30T12:30:35.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.032+0000 7fc781ffb640 1 -- 172.21.15.125:0/55041109 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc77c0a5b90 con 0x7fc7841568e0 2023-10-30T12:30:35.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.032+0000 7fc7897d3640 1 -- 172.21.15.125:0/55041109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7841568e0 msgr2=0x7fc784156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:35.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.032+0000 7fc7897d3640 1 --2- 172.21.15.125:0/55041109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7841568e0 0x7fc784156ce0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc78406df40 tx=0x7fc77c01c9d0 comp rx=0 tx=0).stop 2023-10-30T12:30:35.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.032+0000 7fc7897d3640 1 -- 172.21.15.125:0/55041109 shutdown_connections 2023-10-30T12:30:35.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.032+0000 7fc7897d3640 1 --2- 172.21.15.125:0/55041109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7841568e0 0x7fc784156ce0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:35.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.032+0000 7fc7897d3640 1 --2- 172.21.15.125:0/55041109 >> v2:172.21.15.136:3300/0 conn(0x7fc7841505b0 0x7fc7840c29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:35.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.032+0000 7fc7897d3640 1 -- 172.21.15.125:0/55041109 >> 172.21.15.125:0/55041109 conn(0x7fc7840cac50 msgr2=0x7fc7840cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:35.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.032+0000 7fc7897d3640 1 -- 172.21.15.125:0/55041109 shutdown_connections 2023-10-30T12:30:35.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.032+0000 7fc7897d3640 1 -- 172.21.15.125:0/55041109 wait complete. 2023-10-30T12:30:35.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.032+0000 7fc7897d3640 1 Processor -- start 2023-10-30T12:30:35.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7897d3640 1 -- start start 2023-10-30T12:30:35.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7897d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7840c3140 0x7fc7841468f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:35.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7897d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7841505b0 0x7fc7840c99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:35.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7897d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7841568e0 0x7fc7840c9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:35.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7897d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc78414ccb0 con 0x7fc7841505b0 2023-10-30T12:30:35.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7897d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7840ca500 con 0x7fc7840c3140 2023-10-30T12:30:35.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7897d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc78414b570 con 0x7fc7841568e0 2023-10-30T12:30:35.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7827fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7841505b0 0x7fc7840c99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:35.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7827fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7841505b0 0x7fc7840c99e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51714/0 (socket says 172.21.15.125:51714) 2023-10-30T12:30:35.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7827fc640 1 -- 172.21.15.125:0/2420838483 learned_addr learned my addr 172.21.15.125:0/2420838483 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:35.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc782ffd640 1 --2- 172.21.15.125:0/2420838483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7840c3140 0x7fc7841468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:35.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7837fe640 1 --2- 172.21.15.125:0/2420838483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7841568e0 0x7fc7840c9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:35.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7827fc640 1 -- 172.21.15.125:0/2420838483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7841568e0 msgr2=0x7fc7840c9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:35.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7827fc640 1 --2- 172.21.15.125:0/2420838483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7841568e0 0x7fc7840c9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:35.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7827fc640 1 -- 172.21.15.125:0/2420838483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7840c3140 msgr2=0x7fc7841468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:35.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7827fc640 1 --2- 172.21.15.125:0/2420838483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7840c3140 0x7fc7841468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:35.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7827fc640 1 -- 172.21.15.125:0/2420838483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77c009b30 con 0x7fc7841505b0 2023-10-30T12:30:35.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7827fc640 1 --2- 172.21.15.125:0/2420838483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7841505b0 0x7fc7840c99e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc77800c3f0 tx=0x7fc77800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:35.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc76b7fe640 1 -- 172.21.15.125:0/2420838483 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7780052d0 con 0x7fc7841505b0 2023-10-30T12:30:35.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc76b7fe640 1 -- 172.21.15.125:0/2420838483 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc778005430 con 0x7fc7841505b0 2023-10-30T12:30:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc76b7fe640 1 -- 172.21.15.125:0/2420838483 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc778003840 con 0x7fc7841505b0 2023-10-30T12:30:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7897d3640 1 -- 172.21.15.125:0/2420838483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc78414cf20 con 0x7fc7841505b0 2023-10-30T12:30:35.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7897d3640 1 -- 172.21.15.125:0/2420838483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc78414d4e0 con 0x7fc7841505b0 2023-10-30T12:30:35.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.036+0000 7fc7897d3640 1 -- 172.21.15.125:0/2420838483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc78404e170 con 0x7fc7841505b0 2023-10-30T12:30:35.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.040+0000 7fc76b7fe640 1 -- 172.21.15.125:0/2420838483 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7780039a0 con 0x7fc7841505b0 2023-10-30T12:30:35.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.040+0000 7fc76b7fe640 1 --2- 172.21.15.125:0/2420838483 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7700664a0 0x7fc770068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:35.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.040+0000 7fc76b7fe640 1 -- 172.21.15.125:0/2420838483 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fc7780831e0 con 0x7fc7841505b0 2023-10-30T12:30:35.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.040+0000 7fc782ffd640 1 --2- 172.21.15.125:0/2420838483 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7700664a0 0x7fc770068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:35.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.040+0000 7fc782ffd640 1 --2- 172.21.15.125:0/2420838483 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7700664a0 0x7fc770068960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc774009950 tx=0x7fc774008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:35.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.044+0000 7fc76b7fe640 1 -- 172.21.15.125:0/2420838483 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7780506b0 con 0x7fc7841505b0 2023-10-30T12:30:35.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.224+0000 7fc7897d3640 1 -- 172.21.15.125:0/2420838483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7840c2f30 con 0x7fc7841505b0 2023-10-30T12:30:35.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.228+0000 7fc76b7fe640 1 -- 172.21.15.125:0/2420838483 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+5747 (secure 0 0 0) 0x7fc7840c2f30 con 0x7fc7841505b0 2023-10-30T12:30:35.234 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:35.235 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":14,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:30:29.968105+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:resolve","state_seq":29,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:35.236 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 14 2023-10-30T12:30:35.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.228+0000 7fc7897d3640 1 -- 172.21.15.125:0/2420838483 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7700664a0 msgr2=0x7fc770068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:35.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.228+0000 7fc7897d3640 1 --2- 172.21.15.125:0/2420838483 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7700664a0 0x7fc770068960 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc774009950 tx=0x7fc774008040 comp rx=0 tx=0).stop 2023-10-30T12:30:35.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.228+0000 7fc7897d3640 1 -- 172.21.15.125:0/2420838483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7841505b0 msgr2=0x7fc7840c99e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:35.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.228+0000 7fc7897d3640 1 --2- 172.21.15.125:0/2420838483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7841505b0 0x7fc7840c99e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc77800c3f0 tx=0x7fc77800c8b0 comp rx=0 tx=0).stop 2023-10-30T12:30:35.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.228+0000 7fc7897d3640 1 -- 172.21.15.125:0/2420838483 shutdown_connections 2023-10-30T12:30:35.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.228+0000 7fc7897d3640 1 --2- 172.21.15.125:0/2420838483 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7700664a0 0x7fc770068960 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:35.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.228+0000 7fc7897d3640 1 --2- 172.21.15.125:0/2420838483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7841568e0 0x7fc7840c9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:35.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.228+0000 7fc7897d3640 1 --2- 172.21.15.125:0/2420838483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7841505b0 0x7fc7840c99e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:35.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.228+0000 7fc7897d3640 1 --2- 172.21.15.125:0/2420838483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7840c3140 0x7fc7841468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:35.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.228+0000 7fc7897d3640 1 -- 172.21.15.125:0/2420838483 >> 172.21.15.125:0/2420838483 conn(0x7fc7840cac50 msgr2=0x7fc7840c5bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:35.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.228+0000 7fc7897d3640 1 -- 172.21.15.125:0/2420838483 shutdown_connections 2023-10-30T12:30:35.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:35.228+0000 7fc7897d3640 1 -- 172.21.15.125:0/2420838483 wait complete. 2023-10-30T12:30:35.255 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:30:37.255 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:37.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.396+0000 7f41f4b01640 1 Processor -- start 2023-10-30T12:30:37.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.396+0000 7f41f4b01640 1 -- start start 2023-10-30T12:30:37.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.396+0000 7f41f4b01640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f41f006f7a0 0x7f41f006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:37.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.396+0000 7f41f4b01640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f0071110 0x7f41f0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:37.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.396+0000 7f41f4b01640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f0066cf0 con 0x7f41f00c6bd0 2023-10-30T12:30:37.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.396+0000 7f41f4b01640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41f00700c0 con 0x7f41f006f7a0 2023-10-30T12:30:37.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.396+0000 7f41f4b01640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00c70d0 con 0x7f41f0071110 2023-10-30T12:30:37.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.396+0000 7f41ee59c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f41f006f7a0 0x7f41f006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:37.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.396+0000 7f41e5d9b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f41f00c6bd0 0x7f41f0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33664/0 (socket says 172.21.15.125:33664) 2023-10-30T12:30:37.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.396+0000 7f41e5d9b640 1 -- 172.21.15.125:0/3116361512 learned_addr learned my addr 172.21.15.125:0/3116361512 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:30:37.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.396+0000 7f41eed9d640 1 --2- 172.21.15.125:0/3116361512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f0071110 0x7f41f0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:37.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.396+0000 7f41edd9b640 1 -- 172.21.15.125:0/3116361512 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372737627 0 0) 0x7f41f0066cf0 con 0x7f41f00c6bd0 2023-10-30T12:30:37.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.396+0000 7f41edd9b640 1 -- 172.21.15.125:0/3116361512 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41dc003530 con 0x7f41f00c6bd0 2023-10-30T12:30:37.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.400+0000 7f41ee59c640 1 -- 172.21.15.125:0/3116361512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f0071110 msgr2=0x7f41f0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:37.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.400+0000 7f41ee59c640 1 --2- 172.21.15.125:0/3116361512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f0071110 0x7f41f0158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:37.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.400+0000 7f41ee59c640 1 -- 172.21.15.125:0/3116361512 >> v1:172.21.15.125:6789/0 conn(0x7f41f00c6bd0 legacy=0x7f41f0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:37.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.400+0000 7f41ee59c640 1 -- 172.21.15.125:0/3116361512 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f00c7390 con 0x7f41f006f7a0 2023-10-30T12:30:37.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.400+0000 7f41eed9d640 1 --2- 172.21.15.125:0/3116361512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f0071110 0x7f41f0158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:30:37.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.400+0000 7f41ee59c640 1 --2- 172.21.15.125:0/3116361512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f006f7a0 0x7f41f006fb80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f41e00097c0 tx=0x7f41e001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b37761f577a5b6be server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:37.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.400+0000 7f41edd9b640 1 -- 172.21.15.125:0/3116361512 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e00a8970 con 0x7f41f006f7a0 2023-10-30T12:30:37.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.400+0000 7f41edd9b640 1 -- 172.21.15.125:0/3116361512 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e00a8ad0 con 0x7f41f006f7a0 2023-10-30T12:30:37.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.400+0000 7f41edd9b640 1 -- 172.21.15.125:0/3116361512 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e00a8e60 con 0x7f41f006f7a0 2023-10-30T12:30:37.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 -- 172.21.15.125:0/3116361512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f006f7a0 msgr2=0x7f41f006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:37.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 --2- 172.21.15.125:0/3116361512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f006f7a0 0x7f41f006fb80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f41e00097c0 tx=0x7f41e001ec70 comp rx=0 tx=0).stop 2023-10-30T12:30:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 -- 172.21.15.125:0/3116361512 shutdown_connections 2023-10-30T12:30:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 --2- 172.21.15.125:0/3116361512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f0071110 0x7f41f0158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 --2- 172.21.15.125:0/3116361512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f006f7a0 0x7f41f006fb80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 -- 172.21.15.125:0/3116361512 >> 172.21.15.125:0/3116361512 conn(0x7f41f00c7910 msgr2=0x7f41f00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:37.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 -- 172.21.15.125:0/3116361512 shutdown_connections 2023-10-30T12:30:37.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 -- 172.21.15.125:0/3116361512 wait complete. 2023-10-30T12:30:37.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 Processor -- start 2023-10-30T12:30:37.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 -- start start 2023-10-30T12:30:37.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f006f7a0 0x7f41f01ecda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:37.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f0071110 0x7f41f00c5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:37.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41f00c6bd0 0x7f41f01f0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:37.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f01f0cb0 con 0x7f41f00c6bd0 2023-10-30T12:30:37.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f01f0f90 con 0x7f41f0071110 2023-10-30T12:30:37.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41f4b01640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f01f1270 con 0x7f41f006f7a0 2023-10-30T12:30:37.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41eed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41f00c6bd0 0x7f41f01f0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41eed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41f00c6bd0 0x7f41f01f0390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51724/0 (socket says 172.21.15.125:51724) 2023-10-30T12:30:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41eed9d640 1 -- 172.21.15.125:0/84401184 learned_addr learned my addr 172.21.15.125:0/84401184 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41ee59c640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f006f7a0 0x7f41f01ecda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41e5d9b640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f0071110 0x7f41f00c5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:37.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.404+0000 7f41eed9d640 1 -- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f006f7a0 msgr2=0x7f41f01ecda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:37.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.408+0000 7f41eed9d640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f006f7a0 0x7f41f01ecda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:37.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.408+0000 7f41eed9d640 1 -- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f0071110 msgr2=0x7f41f00c5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:37.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.408+0000 7f41eed9d640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f0071110 0x7f41f00c5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:37.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.408+0000 7f41eed9d640 1 -- 172.21.15.125:0/84401184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e8009420 con 0x7f41f00c6bd0 2023-10-30T12:30:37.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.408+0000 7f41e5d9b640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f0071110 0x7f41f00c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:30:37.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.408+0000 7f41eed9d640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41f00c6bd0 0x7f41f01f0390 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f41e800e950 tx=0x7f41e800ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:37.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.408+0000 7f41e77fe640 1 -- 172.21.15.125:0/84401184 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e8009720 con 0x7f41f00c6bd0 2023-10-30T12:30:37.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.408+0000 7f41e77fe640 1 -- 172.21.15.125:0/84401184 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e800ca00 con 0x7f41f00c6bd0 2023-10-30T12:30:37.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.408+0000 7f41e77fe640 1 -- 172.21.15.125:0/84401184 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e800cd90 con 0x7f41f00c6bd0 2023-10-30T12:30:37.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.408+0000 7f41f4b01640 1 -- 172.21.15.125:0/84401184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e0009420 con 0x7f41f00c6bd0 2023-10-30T12:30:37.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.408+0000 7f41f4b01640 1 -- 172.21.15.125:0/84401184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f0151e60 con 0x7f41f00c6bd0 2023-10-30T12:30:37.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.408+0000 7f41f4b01640 1 -- 172.21.15.125:0/84401184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f004e170 con 0x7f41f00c6bd0 2023-10-30T12:30:37.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.408+0000 7f41e77fe640 1 -- 172.21.15.125:0/84401184 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41e8013470 con 0x7f41f00c6bd0 2023-10-30T12:30:37.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.408+0000 7f41e77fe640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41d00664a0 0x7f41d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:37.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.412+0000 7f41e77fe640 1 -- 172.21.15.125:0/84401184 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f41e8083e00 con 0x7f41f00c6bd0 2023-10-30T12:30:37.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.412+0000 7f41ee59c640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41d00664a0 0x7f41d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:37.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.412+0000 7f41ee59c640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41d00664a0 0x7f41d0068960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f41e00b2000 tx=0x7f41e0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:37.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.416+0000 7f41e77fe640 1 -- 172.21.15.125:0/84401184 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41e80512c0 con 0x7f41f00c6bd0 2023-10-30T12:30:37.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.596+0000 7f41f4b01640 1 -- 172.21.15.125:0/84401184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41f0072150 con 0x7f41f00c6bd0 2023-10-30T12:30:37.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.596+0000 7f41e77fe640 1 -- 172.21.15.125:0/84401184 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5749 (secure 0 0 0) 0x7f41e80548e0 con 0x7f41f00c6bd0 2023-10-30T12:30:37.603 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:37.603 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:30:35.962834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:37.604 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 15 2023-10-30T12:30:37.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.596+0000 7f41f4b01640 1 -- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41d00664a0 msgr2=0x7f41d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:37.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.596+0000 7f41f4b01640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41d00664a0 0x7f41d0068960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f41e00b2000 tx=0x7f41e0008040 comp rx=0 tx=0).stop 2023-10-30T12:30:37.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.596+0000 7f41f4b01640 1 -- 172.21.15.125:0/84401184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41f00c6bd0 msgr2=0x7f41f01f0390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:37.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.596+0000 7f41f4b01640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41f00c6bd0 0x7f41f01f0390 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f41e800e950 tx=0x7f41e800ee10 comp rx=0 tx=0).stop 2023-10-30T12:30:37.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.596+0000 7f41f4b01640 1 -- 172.21.15.125:0/84401184 shutdown_connections 2023-10-30T12:30:37.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.596+0000 7f41f4b01640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41d00664a0 0x7f41d0068960 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:37.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.596+0000 7f41f4b01640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41f00c6bd0 0x7f41f01f0390 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:37.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.596+0000 7f41f4b01640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f0071110 0x7f41f00c5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:37.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.596+0000 7f41f4b01640 1 --2- 172.21.15.125:0/84401184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f006f7a0 0x7f41f01ecda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:37.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.596+0000 7f41f4b01640 1 -- 172.21.15.125:0/84401184 >> 172.21.15.125:0/84401184 conn(0x7f41f00c7910 msgr2=0x7f41f00c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:37.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.596+0000 7f41f4b01640 1 -- 172.21.15.125:0/84401184 shutdown_connections 2023-10-30T12:30:37.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:37.596+0000 7f41f4b01640 1 -- 172.21.15.125:0/84401184 wait complete. 2023-10-30T12:30:37.625 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 15, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 15, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4265": { "addr": "172.21.15.125:6837/2291877604", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2291877604, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2291877604, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4265, "incarnation": 13, "join_fscid": -1, "name": "c", "rank": 2, "state": "up:reconnect", "state_seq": 31 }, "gid_4285": { "addr": "172.21.15.136:6839/2250748765", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 2250748765, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 2250748765, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4294": { "addr": "172.21.15.125:6836/2017382874", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2017382874, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2017382874, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4294, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 4 } }, "last_failure": 0, "last_failure_osd_epoch": 25, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:30:35.962834+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4294, "mds_1": 4285, "mds_2": 4265 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:30:39.626 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:39.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.764+0000 7fd2ffc32640 1 Processor -- start 2023-10-30T12:30:39.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2ffc32640 1 -- start start 2023-10-30T12:30:39.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2ffc32640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2f8069e90 0x7fd2f815a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:39.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2ffc32640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd2f806cb80 0x7fd2f806d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:39.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2ffc32640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f806d550 con 0x7fd2f806a270 2023-10-30T12:30:39.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2ffc32640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f80c8570 con 0x7fd2f806cb80 2023-10-30T12:30:39.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2ffc32640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f80c6c10 con 0x7fd2f8069e90 2023-10-30T12:30:39.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2fe1cf640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd2f806cb80 0x7fd2f806d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:39.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2fd1cd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd2f806a270 0x7fd2f80c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33680/0 (socket says 172.21.15.125:33680) 2023-10-30T12:30:39.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2fe1cf640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd2f806cb80 0x7fd2f806d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51736/0 (socket says 172.21.15.125:51736) 2023-10-30T12:30:39.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2fd1cd640 1 -- 172.21.15.125:0/3872589425 learned_addr learned my addr 172.21.15.125:0/3872589425 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:30:39.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2fd9ce640 1 --2- 172.21.15.125:0/3872589425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2f8069e90 0x7fd2f815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:39.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2fe1cf640 1 -- 172.21.15.125:0/3872589425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2f8069e90 msgr2=0x7fd2f815a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:39.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2fe1cf640 1 --2- 172.21.15.125:0/3872589425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2f8069e90 0x7fd2f815a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:39.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2fe1cf640 1 -- 172.21.15.125:0/3872589425 >> v1:172.21.15.125:6789/0 conn(0x7fd2f806a270 legacy=0x7fd2f80c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:39.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2fe1cf640 1 -- 172.21.15.125:0/3872589425 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f80c6ef0 con 0x7fd2f806cb80 2023-10-30T12:30:39.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.768+0000 7fd2fc9cc640 1 -- 172.21.15.125:0/3872589425 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698485082 0 0) 0x7fd2f806d550 con 0x7fd2f806a270 2023-10-30T12:30:39.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.772+0000 7fd2fe1cf640 1 --2- 172.21.15.125:0/3872589425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2f806cb80 0x7fd2f806d010 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009d90 tx=0x7fd2f001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41cfda4062e4ee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:39.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.772+0000 7fd2fc9cc640 1 -- 172.21.15.125:0/3872589425 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f00a5900 con 0x7fd2f806cb80 2023-10-30T12:30:39.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.772+0000 7fd2fc9cc640 1 -- 172.21.15.125:0/3872589425 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2f00a5a60 con 0x7fd2f806cb80 2023-10-30T12:30:39.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.772+0000 7fd2fc9cc640 1 -- 172.21.15.125:0/3872589425 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2f00a5df0 con 0x7fd2f806cb80 2023-10-30T12:30:39.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.772+0000 7fd2ffc32640 1 -- 172.21.15.125:0/3872589425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2f806cb80 msgr2=0x7fd2f806d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:39.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.772+0000 7fd2ffc32640 1 --2- 172.21.15.125:0/3872589425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2f806cb80 0x7fd2f806d010 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd2f0009d90 tx=0x7fd2f001c950 comp rx=0 tx=0).stop 2023-10-30T12:30:39.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.772+0000 7fd2ffc32640 1 -- 172.21.15.125:0/3872589425 shutdown_connections 2023-10-30T12:30:39.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.772+0000 7fd2ffc32640 1 --2- 172.21.15.125:0/3872589425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2f806cb80 0x7fd2f806d010 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:39.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.772+0000 7fd2ffc32640 1 --2- 172.21.15.125:0/3872589425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2f8069e90 0x7fd2f815a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:39.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.772+0000 7fd2ffc32640 1 -- 172.21.15.125:0/3872589425 >> 172.21.15.125:0/3872589425 conn(0x7fd2f806dfd0 msgr2=0x7fd2f806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:39.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.772+0000 7fd2ffc32640 1 -- 172.21.15.125:0/3872589425 shutdown_connections 2023-10-30T12:30:39.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.772+0000 7fd2ffc32640 1 -- 172.21.15.125:0/3872589425 wait complete. 2023-10-30T12:30:39.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.772+0000 7fd2ffc32640 1 Processor -- start 2023-10-30T12:30:39.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.772+0000 7fd2ffc32640 1 -- start start 2023-10-30T12:30:39.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2ffc32640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2f8069e90 0x7fd2f81f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:39.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2ffc32640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2f806a270 0x7fd2f81569b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:39.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2ffc32640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2f806cb80 0x7fd2f8154860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:39.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2ffc32640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f8071500 con 0x7fd2f806a270 2023-10-30T12:30:39.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2ffc32640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f8155b90 con 0x7fd2f806cb80 2023-10-30T12:30:39.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2ffc32640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f81540f0 con 0x7fd2f8069e90 2023-10-30T12:30:39.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2fd1cd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2f806a270 0x7fd2f81569b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:39.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2fd1cd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2f806a270 0x7fd2f81569b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51742/0 (socket says 172.21.15.125:51742) 2023-10-30T12:30:39.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2fd1cd640 1 -- 172.21.15.125:0/250221170 learned_addr learned my addr 172.21.15.125:0/250221170 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:39.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2fe1cf640 1 --2- 172.21.15.125:0/250221170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2f806cb80 0x7fd2f8154860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:39.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2fd9ce640 1 --2- 172.21.15.125:0/250221170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2f8069e90 0x7fd2f81f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:39.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2fd1cd640 1 -- 172.21.15.125:0/250221170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2f8069e90 msgr2=0x7fd2f81f2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:39.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2fd1cd640 1 --2- 172.21.15.125:0/250221170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2f8069e90 0x7fd2f81f2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:39.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2fd1cd640 1 -- 172.21.15.125:0/250221170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2f806cb80 msgr2=0x7fd2f8154860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:39.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2fd1cd640 1 --2- 172.21.15.125:0/250221170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2f806cb80 0x7fd2f8154860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:39.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2fd1cd640 1 -- 172.21.15.125:0/250221170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f0009a70 con 0x7fd2f806a270 2023-10-30T12:30:39.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2fd1cd640 1 --2- 172.21.15.125:0/250221170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2f806a270 0x7fd2f81569b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800c520 tx=0x7fd2e800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:39.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2ee7fc640 1 -- 172.21.15.125:0/250221170 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e80074e0 con 0x7fd2f806a270 2023-10-30T12:30:39.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2ee7fc640 1 -- 172.21.15.125:0/250221170 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e8007640 con 0x7fd2f806a270 2023-10-30T12:30:39.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2ee7fc640 1 -- 172.21.15.125:0/250221170 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e80049f0 con 0x7fd2f806a270 2023-10-30T12:30:39.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2ffc32640 1 -- 172.21.15.125:0/250221170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f81543d0 con 0x7fd2f806a270 2023-10-30T12:30:39.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2ffc32640 1 -- 172.21.15.125:0/250221170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f8071220 con 0x7fd2f806a270 2023-10-30T12:30:39.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.776+0000 7fd2ffc32640 1 -- 172.21.15.125:0/250221170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f8154560 con 0x7fd2f806a270 2023-10-30T12:30:39.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.780+0000 7fd2ee7fc640 1 -- 172.21.15.125:0/250221170 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd2e8004b50 con 0x7fd2f806a270 2023-10-30T12:30:39.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.780+0000 7fd2ee7fc640 1 --2- 172.21.15.125:0/250221170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2dc0664a0 0x7fd2dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:39.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.780+0000 7fd2ee7fc640 1 -- 172.21.15.125:0/250221170 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fd2e80831a0 con 0x7fd2f806a270 2023-10-30T12:30:39.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.780+0000 7fd2fd9ce640 1 --2- 172.21.15.125:0/250221170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2dc0664a0 0x7fd2dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:39.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.780+0000 7fd2fd9ce640 1 --2- 172.21.15.125:0/250221170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2dc0664a0 0x7fd2dc068960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400b9d0 tx=0x7fd2f4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:39.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.784+0000 7fd2ee7fc640 1 -- 172.21.15.125:0/250221170 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd2e80505c0 con 0x7fd2f806a270 2023-10-30T12:30:39.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.968+0000 7fd2ffc32640 1 -- 172.21.15.125:0/250221170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2f804e170 con 0x7fd2f806a270 2023-10-30T12:30:39.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.968+0000 7fd2ee7fc640 1 -- 172.21.15.125:0/250221170 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+5749 (secure 0 0 0) 0x7fd2e8018090 con 0x7fd2f806a270 2023-10-30T12:30:39.976 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:39.977 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:30:35.962834+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:reconnect","state_seq":31,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:39.977 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 15 2023-10-30T12:30:39.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.972+0000 7fd2ffc32640 1 -- 172.21.15.125:0/250221170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2dc0664a0 msgr2=0x7fd2dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:39.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.972+0000 7fd2ffc32640 1 --2- 172.21.15.125:0/250221170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2dc0664a0 0x7fd2dc068960 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd2f400b9d0 tx=0x7fd2f4008040 comp rx=0 tx=0).stop 2023-10-30T12:30:39.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.972+0000 7fd2ffc32640 1 -- 172.21.15.125:0/250221170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2f806a270 msgr2=0x7fd2f81569b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:39.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.972+0000 7fd2ffc32640 1 --2- 172.21.15.125:0/250221170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2f806a270 0x7fd2f81569b0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd2e800c520 tx=0x7fd2e800c9e0 comp rx=0 tx=0).stop 2023-10-30T12:30:39.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.972+0000 7fd2ffc32640 1 -- 172.21.15.125:0/250221170 shutdown_connections 2023-10-30T12:30:39.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.972+0000 7fd2ffc32640 1 --2- 172.21.15.125:0/250221170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2dc0664a0 0x7fd2dc068960 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:39.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.972+0000 7fd2ffc32640 1 --2- 172.21.15.125:0/250221170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2f806cb80 0x7fd2f8154860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:39.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.972+0000 7fd2ffc32640 1 --2- 172.21.15.125:0/250221170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2f806a270 0x7fd2f81569b0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:39.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.972+0000 7fd2ffc32640 1 --2- 172.21.15.125:0/250221170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2f8069e90 0x7fd2f81f2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:39.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.972+0000 7fd2ffc32640 1 -- 172.21.15.125:0/250221170 >> 172.21.15.125:0/250221170 conn(0x7fd2f806dfd0 msgr2=0x7fd2f814f5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:39.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.972+0000 7fd2ffc32640 1 -- 172.21.15.125:0/250221170 shutdown_connections 2023-10-30T12:30:39.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:39.972+0000 7fd2ffc32640 1 -- 172.21.15.125:0/250221170 wait complete. 2023-10-30T12:30:39.997 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:30:41.998 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:42.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c82009640 1 Processor -- start 2023-10-30T12:30:42.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c82009640 1 -- start start 2023-10-30T12:30:42.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c82009640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9c7c15d6f0 0x7f9c7c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:42.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c82009640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9c7c0cf160 0x7f9c7c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:42.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c82009640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c7c0cb800 con 0x7f9c7c1538a0 2023-10-30T12:30:42.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c82009640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c7c0cb330 con 0x7f9c7c15d6f0 2023-10-30T12:30:42.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c82009640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c7c0cad70 con 0x7f9c7c0cf160 2023-10-30T12:30:42.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c73fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9c7c15d6f0 0x7f9c7c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:42.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c7b7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9c7c1538a0 0x7f9c7c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53476/0 (socket says 172.21.15.125:53476) 2023-10-30T12:30:42.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c7b7fe640 1 -- 172.21.15.125:0/1133571942 learned_addr learned my addr 172.21.15.125:0/1133571942 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:30:42.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c7bfff640 1 --2- 172.21.15.125:0/1133571942 >> v2:172.21.15.136:3300/0 conn(0x7f9c7c0cf160 0x7f9c7c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:42.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c73fff640 1 -- 172.21.15.125:0/1133571942 >> v2:172.21.15.136:3300/0 conn(0x7f9c7c0cf160 msgr2=0x7f9c7c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:42.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c73fff640 1 --2- 172.21.15.125:0/1133571942 >> v2:172.21.15.136:3300/0 conn(0x7f9c7c0cf160 0x7f9c7c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:42.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c73fff640 1 -- 172.21.15.125:0/1133571942 >> v1:172.21.15.125:6789/0 conn(0x7f9c7c1538a0 legacy=0x7f9c7c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:42.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c73fff640 1 -- 172.21.15.125:0/1133571942 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7c0cb050 con 0x7f9c7c15d6f0 2023-10-30T12:30:42.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.140+0000 7f9c7affd640 1 -- 172.21.15.125:0/1133571942 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136703003 0 0) 0x7f9c7c0cb800 con 0x7f9c7c1538a0 2023-10-30T12:30:42.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.144+0000 7f9c73fff640 1 --2- 172.21.15.125:0/1133571942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c7c15d6f0 0x7f9c7c15a090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c0b5330 tx=0x7f9c6401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d877d543e4d8c67 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:42.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.144+0000 7f9c7affd640 1 -- 172.21.15.125:0/1133571942 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c640a8950 con 0x7f9c7c15d6f0 2023-10-30T12:30:42.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.144+0000 7f9c7affd640 1 -- 172.21.15.125:0/1133571942 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c640a8ab0 con 0x7f9c7c15d6f0 2023-10-30T12:30:42.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.144+0000 7f9c7affd640 1 -- 172.21.15.125:0/1133571942 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c640a8e40 con 0x7f9c7c15d6f0 2023-10-30T12:30:42.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.144+0000 7f9c82009640 1 -- 172.21.15.125:0/1133571942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c7c15d6f0 msgr2=0x7f9c7c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:42.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.144+0000 7f9c82009640 1 --2- 172.21.15.125:0/1133571942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c7c15d6f0 0x7f9c7c15a090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9c7c0b5330 tx=0x7f9c6401ec50 comp rx=0 tx=0).stop 2023-10-30T12:30:42.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.144+0000 7f9c82009640 1 -- 172.21.15.125:0/1133571942 shutdown_connections 2023-10-30T12:30:42.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.144+0000 7f9c82009640 1 --2- 172.21.15.125:0/1133571942 >> v2:172.21.15.136:3300/0 conn(0x7f9c7c0cf160 0x7f9c7c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:42.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.144+0000 7f9c82009640 1 --2- 172.21.15.125:0/1133571942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c7c15d6f0 0x7f9c7c15a090 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:42.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.144+0000 7f9c82009640 1 -- 172.21.15.125:0/1133571942 >> 172.21.15.125:0/1133571942 conn(0x7f9c7c06bfe0 msgr2=0x7f9c7c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:42.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.144+0000 7f9c82009640 1 -- 172.21.15.125:0/1133571942 shutdown_connections 2023-10-30T12:30:42.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.144+0000 7f9c82009640 1 -- 172.21.15.125:0/1133571942 wait complete. 2023-10-30T12:30:42.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c82009640 1 Processor -- start 2023-10-30T12:30:42.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c82009640 1 -- start start 2023-10-30T12:30:42.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c82009640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c7c0cf160 0x7f9c7c1f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:42.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c82009640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c7c1538a0 0x7f9c7c0b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:42.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c82009640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c7c15d6f0 0x7f9c7c14c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:42.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c82009640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c157be0 con 0x7f9c7c0cf160 2023-10-30T12:30:42.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c82009640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c1f3250 con 0x7f9c7c15d6f0 2023-10-30T12:30:42.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c82009640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c14fd40 con 0x7f9c7c1538a0 2023-10-30T12:30:42.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c7b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c7c0cf160 0x7f9c7c1f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:42.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c7b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c7c0cf160 0x7f9c7c1f25a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43574/0 (socket says 172.21.15.125:43574) 2023-10-30T12:30:42.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c7b7fe640 1 -- 172.21.15.125:0/2050737100 learned_addr learned my addr 172.21.15.125:0/2050737100 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:42.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c7bfff640 1 --2- 172.21.15.125:0/2050737100 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c7c15d6f0 0x7f9c7c14c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:42.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c73fff640 1 --2- 172.21.15.125:0/2050737100 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c7c1538a0 0x7f9c7c0b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:42.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c7b7fe640 1 -- 172.21.15.125:0/2050737100 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c7c1538a0 msgr2=0x7f9c7c0b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:42.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c7b7fe640 1 --2- 172.21.15.125:0/2050737100 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c7c1538a0 0x7f9c7c0b2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:42.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c7b7fe640 1 -- 172.21.15.125:0/2050737100 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c7c15d6f0 msgr2=0x7f9c7c14c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:42.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c7b7fe640 1 --2- 172.21.15.125:0/2050737100 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c7c15d6f0 0x7f9c7c14c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:42.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c7b7fe640 1 -- 172.21.15.125:0/2050737100 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c64009420 con 0x7f9c7c0cf160 2023-10-30T12:30:42.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c7b7fe640 1 --2- 172.21.15.125:0/2050737100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c7c0cf160 0x7f9c7c1f25a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c0011f0 tx=0x7f9c6c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:42.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c78ff9640 1 -- 172.21.15.125:0/2050737100 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c0074e0 con 0x7f9c7c0cf160 2023-10-30T12:30:42.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c78ff9640 1 -- 172.21.15.125:0/2050737100 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6c007640 con 0x7f9c7c0cf160 2023-10-30T12:30:42.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c82009640 1 -- 172.21.15.125:0/2050737100 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7c159680 con 0x7f9c7c0cf160 2023-10-30T12:30:42.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c78ff9640 1 -- 172.21.15.125:0/2050737100 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c004a10 con 0x7f9c7c0cf160 2023-10-30T12:30:42.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.148+0000 7f9c82009640 1 -- 172.21.15.125:0/2050737100 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7c1f2f70 con 0x7f9c7c0cf160 2023-10-30T12:30:42.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.152+0000 7f9c82009640 1 -- 172.21.15.125:0/2050737100 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c7c0b2a50 con 0x7f9c7c0cf160 2023-10-30T12:30:42.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.152+0000 7f9c78ff9640 1 -- 172.21.15.125:0/2050737100 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9c6c004b70 con 0x7f9c7c0cf160 2023-10-30T12:30:42.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.152+0000 7f9c78ff9640 1 --2- 172.21.15.125:0/2050737100 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c5c0664a0 0x7f9c5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:42.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.152+0000 7f9c78ff9640 1 -- 172.21.15.125:0/2050737100 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f9c6c083470 con 0x7f9c7c0cf160 2023-10-30T12:30:42.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.152+0000 7f9c73fff640 1 --2- 172.21.15.125:0/2050737100 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c5c0664a0 0x7f9c5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:42.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.152+0000 7f9c73fff640 1 --2- 172.21.15.125:0/2050737100 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c5c0664a0 0x7f9c5c068960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9c640b2000 tx=0x7f9c64008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:42.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.156+0000 7f9c78ff9640 1 -- 172.21.15.125:0/2050737100 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9c6c050890 con 0x7f9c7c0cf160 2023-10-30T12:30:42.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.340+0000 7f9c82009640 1 -- 172.21.15.125:0/2050737100 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c7c159870 con 0x7f9c7c0cf160 2023-10-30T12:30:42.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.340+0000 7f9c78ff9640 1 -- 172.21.15.125:0/2050737100 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5746 (secure 0 0 0) 0x7f9c7c159870 con 0x7f9c7c0cf160 2023-10-30T12:30:42.347 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:42.347 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:30:41.961152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:42.348 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 16 2023-10-30T12:30:42.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.340+0000 7f9c82009640 1 -- 172.21.15.125:0/2050737100 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c5c0664a0 msgr2=0x7f9c5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:42.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.340+0000 7f9c82009640 1 --2- 172.21.15.125:0/2050737100 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c5c0664a0 0x7f9c5c068960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9c640b2000 tx=0x7f9c64008040 comp rx=0 tx=0).stop 2023-10-30T12:30:42.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.340+0000 7f9c82009640 1 -- 172.21.15.125:0/2050737100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c7c0cf160 msgr2=0x7f9c7c1f25a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:42.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.340+0000 7f9c82009640 1 --2- 172.21.15.125:0/2050737100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c7c0cf160 0x7f9c7c1f25a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c0011f0 tx=0x7f9c6c00c9b0 comp rx=0 tx=0).stop 2023-10-30T12:30:42.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.340+0000 7f9c82009640 1 -- 172.21.15.125:0/2050737100 shutdown_connections 2023-10-30T12:30:42.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.340+0000 7f9c82009640 1 --2- 172.21.15.125:0/2050737100 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c5c0664a0 0x7f9c5c068960 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:42.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.340+0000 7f9c82009640 1 --2- 172.21.15.125:0/2050737100 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c7c15d6f0 0x7f9c7c14c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:42.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.340+0000 7f9c82009640 1 --2- 172.21.15.125:0/2050737100 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c7c1538a0 0x7f9c7c0b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:42.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.340+0000 7f9c82009640 1 --2- 172.21.15.125:0/2050737100 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c7c0cf160 0x7f9c7c1f25a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:42.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.340+0000 7f9c82009640 1 -- 172.21.15.125:0/2050737100 >> 172.21.15.125:0/2050737100 conn(0x7f9c7c06bfe0 msgr2=0x7f9c7c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:42.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.340+0000 7f9c82009640 1 -- 172.21.15.125:0/2050737100 shutdown_connections 2023-10-30T12:30:42.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:42.340+0000 7f9c82009640 1 -- 172.21.15.125:0/2050737100 wait complete. 2023-10-30T12:30:42.368 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:30:44.125 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:30:44.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:30:44.369 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:44.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.512+0000 7ff4a4501640 1 Processor -- start 2023-10-30T12:30:44.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.512+0000 7ff4a4501640 1 -- start start 2023-10-30T12:30:44.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.512+0000 7ff4a4501640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff49c0c6b80 0x7ff49c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:44.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.512+0000 7ff4a4501640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff49c13f4a0 0x7ff49c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:44.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.512+0000 7ff4a4501640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49c0c0a50 con 0x7ff49c0c6f60 2023-10-30T12:30:44.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.512+0000 7ff4a4501640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff49c06f6e0 con 0x7ff49c0c6b80 2023-10-30T12:30:44.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.512+0000 7ff4a4501640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c06f9c0 con 0x7ff49c13f4a0 2023-10-30T12:30:44.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.516+0000 7ff4a229d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff49c0c6b80 0x7ff49c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:44.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.516+0000 7ff4a229d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff49c0c6b80 0x7ff49c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43580/0 (socket says 172.21.15.125:43580) 2023-10-30T12:30:44.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.516+0000 7ff4a229d640 1 -- 172.21.15.125:0/129610403 learned_addr learned my addr 172.21.15.125:0/129610403 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:44.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.516+0000 7ff4a2a9e640 1 --2- 172.21.15.125:0/129610403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff49c13f4a0 0x7ff49c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:44.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.516+0000 7ff4a229d640 1 -- 172.21.15.125:0/129610403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff49c13f4a0 msgr2=0x7ff49c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:44.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.516+0000 7ff4a229d640 1 --2- 172.21.15.125:0/129610403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff49c13f4a0 0x7ff49c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:44.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.516+0000 7ff4a229d640 1 -- 172.21.15.125:0/129610403 >> v1:172.21.15.136:6789/0 conn(0x7ff49c0c6f60 legacy=0x7ff49c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:44.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.516+0000 7ff4a229d640 1 -- 172.21.15.125:0/129610403 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49c06fca0 con 0x7ff49c0c6b80 2023-10-30T12:30:44.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.516+0000 7ff4a229d640 1 --2- 172.21.15.125:0/129610403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c0c6b80 0x7ff49c0bfec0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff49800bb10 tx=0x7ff49801ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46f1678209c0c0f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:44.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.516+0000 7ff4a129b640 1 -- 172.21.15.125:0/129610403 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4980a7bf0 con 0x7ff49c0c6b80 2023-10-30T12:30:44.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.516+0000 7ff4a129b640 1 -- 172.21.15.125:0/129610403 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4980a7d50 con 0x7ff49c0c6b80 2023-10-30T12:30:44.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.516+0000 7ff4a129b640 1 -- 172.21.15.125:0/129610403 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4980af660 con 0x7ff49c0c6b80 2023-10-30T12:30:44.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 -- 172.21.15.125:0/129610403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c0c6b80 msgr2=0x7ff49c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:44.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 --2- 172.21.15.125:0/129610403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c0c6b80 0x7ff49c0bfec0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ff49800bb10 tx=0x7ff49801ec10 comp rx=0 tx=0).stop 2023-10-30T12:30:44.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 -- 172.21.15.125:0/129610403 shutdown_connections 2023-10-30T12:30:44.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 --2- 172.21.15.125:0/129610403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff49c13f4a0 0x7ff49c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:44.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 --2- 172.21.15.125:0/129610403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c0c6b80 0x7ff49c0bfec0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:44.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 -- 172.21.15.125:0/129610403 >> 172.21.15.125:0/129610403 conn(0x7ff49c0c7910 msgr2=0x7ff49c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:44.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 -- 172.21.15.125:0/129610403 shutdown_connections 2023-10-30T12:30:44.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 -- 172.21.15.125:0/129610403 wait complete. 2023-10-30T12:30:44.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 Processor -- start 2023-10-30T12:30:44.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 -- start start 2023-10-30T12:30:44.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff49c0c6b80 0x7ff49c0c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:44.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff49c0c6f60 0x7ff49c157750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:44.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c13f4a0 0x7ff49c157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:44.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c136100 con 0x7ff49c13f4a0 2023-10-30T12:30:44.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c155120 con 0x7ff49c0c6b80 2023-10-30T12:30:44.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a4501640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c134eb0 con 0x7ff49c0c6f60 2023-10-30T12:30:44.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a2a9e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c13f4a0 0x7ff49c157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:44.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a2a9e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c13f4a0 0x7ff49c157c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43582/0 (socket says 172.21.15.125:43582) 2023-10-30T12:30:44.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a2a9e640 1 -- 172.21.15.125:0/1209114426 learned_addr learned my addr 172.21.15.125:0/1209114426 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:44.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a229d640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff49c0c6b80 0x7ff49c0c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:44.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a1a9c640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff49c0c6f60 0x7ff49c157750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:44.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a2a9e640 1 -- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff49c0c6f60 msgr2=0x7ff49c157750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:44.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.520+0000 7ff4a2a9e640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff49c0c6f60 0x7ff49c157750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:44.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff4a2a9e640 1 -- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff49c0c6b80 msgr2=0x7ff49c0c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:44.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff4a2a9e640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff49c0c6b80 0x7ff49c0c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:44.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff4a2a9e640 1 -- 172.21.15.125:0/1209114426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49800b770 con 0x7ff49c13f4a0 2023-10-30T12:30:44.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff4a229d640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff49c0c6b80 0x7ff49c0c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:30:44.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff4a2a9e640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c13f4a0 0x7ff49c157c90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff494007e90 tx=0x7ff49400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:44.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff492ffd640 1 -- 172.21.15.125:0/1209114426 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff494014420 con 0x7ff49c13f4a0 2023-10-30T12:30:44.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff492ffd640 1 -- 172.21.15.125:0/1209114426 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff494014580 con 0x7ff49c13f4a0 2023-10-30T12:30:44.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff4a4501640 1 -- 172.21.15.125:0/1209114426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49c0c1530 con 0x7ff49c13f4a0 2023-10-30T12:30:44.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff492ffd640 1 -- 172.21.15.125:0/1209114426 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff494014930 con 0x7ff49c13f4a0 2023-10-30T12:30:44.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff4a4501640 1 -- 172.21.15.125:0/1209114426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49c134bd0 con 0x7ff49c13f4a0 2023-10-30T12:30:44.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff4a4501640 1 -- 172.21.15.125:0/1209114426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49c1552b0 con 0x7ff49c13f4a0 2023-10-30T12:30:44.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff492ffd640 1 -- 172.21.15.125:0/1209114426 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff494014a90 con 0x7ff49c13f4a0 2023-10-30T12:30:44.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff492ffd640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4800664a0 0x7ff480068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:44.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff492ffd640 1 -- 172.21.15.125:0/1209114426 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5824+0+0 (secure 0 0 0) 0x7ff494084800 con 0x7ff49c13f4a0 2023-10-30T12:30:44.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.524+0000 7ff4a229d640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4800664a0 0x7ff480068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:44.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.528+0000 7ff4a229d640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4800664a0 0x7ff480068960 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff4980b1000 tx=0x7ff498008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:44.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.528+0000 7ff492ffd640 1 -- 172.21.15.125:0/1209114426 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff494051c10 con 0x7ff49c13f4a0 2023-10-30T12:30:44.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.708+0000 7ff4a4501640 1 -- 172.21.15.125:0/1209114426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff49c04e170 con 0x7ff49c13f4a0 2023-10-30T12:30:44.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.708+0000 7ff492ffd640 1 -- 172.21.15.125:0/1209114426 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5746 (secure 0 0 0) 0x7ff494055230 con 0x7ff49c13f4a0 2023-10-30T12:30:44.717 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:44.717 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:30:41.961152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:44.719 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 16 2023-10-30T12:30:44.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.712+0000 7ff4a4501640 1 -- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4800664a0 msgr2=0x7ff480068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:44.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.712+0000 7ff4a4501640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4800664a0 0x7ff480068960 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff4980b1000 tx=0x7ff498008040 comp rx=0 tx=0).stop 2023-10-30T12:30:44.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.712+0000 7ff4a4501640 1 -- 172.21.15.125:0/1209114426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c13f4a0 msgr2=0x7ff49c157c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:44.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.712+0000 7ff4a4501640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c13f4a0 0x7ff49c157c90 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff494007e90 tx=0x7ff49400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:30:44.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.712+0000 7ff4a4501640 1 -- 172.21.15.125:0/1209114426 shutdown_connections 2023-10-30T12:30:44.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.712+0000 7ff4a4501640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4800664a0 0x7ff480068960 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:44.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.712+0000 7ff4a4501640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff49c13f4a0 0x7ff49c157c90 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:44.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.712+0000 7ff4a4501640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff49c0c6f60 0x7ff49c157750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:44.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.712+0000 7ff4a4501640 1 --2- 172.21.15.125:0/1209114426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff49c0c6b80 0x7ff49c0c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:44.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.712+0000 7ff4a4501640 1 -- 172.21.15.125:0/1209114426 >> 172.21.15.125:0/1209114426 conn(0x7ff49c0c7910 msgr2=0x7ff49c15dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:44.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.712+0000 7ff4a4501640 1 -- 172.21.15.125:0/1209114426 shutdown_connections 2023-10-30T12:30:44.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:44.712+0000 7ff4a4501640 1 -- 172.21.15.125:0/1209114426 wait complete. 2023-10-30T12:30:44.738 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:30:46.739 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:46.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f9693e640 1 Processor -- start 2023-10-30T12:30:46.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f9693e640 1 -- start start 2023-10-30T12:30:46.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f9693e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6f901538a0 0x7f6f9015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:46.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f9693e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6f900cf160 0x7f6f900cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:46.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f9693e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f900cb800 con 0x7f6f900ca0e0 2023-10-30T12:30:46.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f9693e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f9015dc30 con 0x7f6f900cf160 2023-10-30T12:30:46.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f9693e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f900cb330 con 0x7f6f901538a0 2023-10-30T12:30:46.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f8f7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6f900ca0e0 0x7f6f9015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53490/0 (socket says 172.21.15.125:53490) 2023-10-30T12:30:46.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f8f7fe640 1 -- 172.21.15.125:0/2635923705 learned_addr learned my addr 172.21.15.125:0/2635923705 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:30:46.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f94edb640 1 --2- 172.21.15.125:0/2635923705 >> v2:172.21.15.125:3300/0 conn(0x7f6f900cf160 0x7f6f900cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:46.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f8ffff640 1 --2- 172.21.15.125:0/2635923705 >> v2:172.21.15.136:3300/0 conn(0x7f6f901538a0 0x7f6f9015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:46.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f8effd640 1 -- 172.21.15.125:0/2635923705 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323895749 0 0) 0x7f6f900cb800 con 0x7f6f900ca0e0 2023-10-30T12:30:46.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f8effd640 1 -- 172.21.15.125:0/2635923705 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f78003530 con 0x7f6f900ca0e0 2023-10-30T12:30:46.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f94edb640 1 -- 172.21.15.125:0/2635923705 >> v2:172.21.15.136:3300/0 conn(0x7f6f901538a0 msgr2=0x7f6f9015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:46.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f94edb640 1 --2- 172.21.15.125:0/2635923705 >> v2:172.21.15.136:3300/0 conn(0x7f6f901538a0 0x7f6f9015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:46.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f94edb640 1 -- 172.21.15.125:0/2635923705 >> v1:172.21.15.125:6789/0 conn(0x7f6f900ca0e0 legacy=0x7f6f9015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:46.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f94edb640 1 -- 172.21.15.125:0/2635923705 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f900cad70 con 0x7f6f900cf160 2023-10-30T12:30:46.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.880+0000 7f6f8ffff640 1 --2- 172.21.15.125:0/2635923705 >> v2:172.21.15.136:3300/0 conn(0x7f6f901538a0 0x7f6f9015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:30:46.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.884+0000 7f6f94edb640 1 --2- 172.21.15.125:0/2635923705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f900cf160 0x7f6f900cf590 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6f900cfd10 tx=0x7f6f8801f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e78a7f124d12b0d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:46.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.884+0000 7f6f8effd640 1 -- 172.21.15.125:0/2635923705 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f880b0440 con 0x7f6f900cf160 2023-10-30T12:30:46.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.884+0000 7f6f8effd640 1 -- 172.21.15.125:0/2635923705 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f880b05a0 con 0x7f6f900cf160 2023-10-30T12:30:46.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.884+0000 7f6f8effd640 1 -- 172.21.15.125:0/2635923705 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f880b0930 con 0x7f6f900cf160 2023-10-30T12:30:46.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.884+0000 7f6f9693e640 1 -- 172.21.15.125:0/2635923705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f900cf160 msgr2=0x7f6f900cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:46.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.884+0000 7f6f9693e640 1 --2- 172.21.15.125:0/2635923705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f900cf160 0x7f6f900cf590 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6f900cfd10 tx=0x7f6f8801f0f0 comp rx=0 tx=0).stop 2023-10-30T12:30:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.884+0000 7f6f9693e640 1 -- 172.21.15.125:0/2635923705 shutdown_connections 2023-10-30T12:30:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.884+0000 7f6f9693e640 1 --2- 172.21.15.125:0/2635923705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f900cf160 0x7f6f900cf590 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.884+0000 7f6f9693e640 1 --2- 172.21.15.125:0/2635923705 >> v2:172.21.15.136:3300/0 conn(0x7f6f901538a0 0x7f6f9015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:46.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.884+0000 7f6f9693e640 1 -- 172.21.15.125:0/2635923705 >> 172.21.15.125:0/2635923705 conn(0x7f6f9006bfe0 msgr2=0x7f6f9006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:46.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.884+0000 7f6f9693e640 1 -- 172.21.15.125:0/2635923705 shutdown_connections 2023-10-30T12:30:46.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.884+0000 7f6f9693e640 1 -- 172.21.15.125:0/2635923705 wait complete. 2023-10-30T12:30:46.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f9693e640 1 Processor -- start 2023-10-30T12:30:46.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f9693e640 1 -- start start 2023-10-30T12:30:46.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f9693e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f900ca0e0 0x7f6f900b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:46.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f9693e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f900cf160 0x7f6f900ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:46.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f9693e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f901538a0 0x7f6f90144150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:46.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f9693e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9015aac0 con 0x7f6f900ca0e0 2023-10-30T12:30:46.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f9693e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f901f6870 con 0x7f6f900cf160 2023-10-30T12:30:46.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f9693e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f90143560 con 0x7f6f901538a0 2023-10-30T12:30:46.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f8ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f900ca0e0 0x7f6f900b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:46.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f8ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f900ca0e0 0x7f6f900b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43600/0 (socket says 172.21.15.125:43600) 2023-10-30T12:30:46.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f8ffff640 1 -- 172.21.15.125:0/3907510154 learned_addr learned my addr 172.21.15.125:0/3907510154 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:46.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f94edb640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f901538a0 0x7f6f90144150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:46.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f8f7fe640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f900cf160 0x7f6f900ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:46.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f8ffff640 1 -- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f901538a0 msgr2=0x7f6f90144150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:46.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f8ffff640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f901538a0 0x7f6f90144150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:46.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f8ffff640 1 -- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f900cf160 msgr2=0x7f6f900ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:46.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f8ffff640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f900cf160 0x7f6f900ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:46.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f8ffff640 1 -- 172.21.15.125:0/3907510154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f88009a70 con 0x7f6f900ca0e0 2023-10-30T12:30:46.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f8f7fe640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f900cf160 0x7f6f900ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:30:46.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f8ffff640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f900ca0e0 0x7f6f900b1870 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6f8000e810 tx=0x7f6f8000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:46.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f8cff9640 1 -- 172.21.15.125:0/3907510154 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8000c8d0 con 0x7f6f900ca0e0 2023-10-30T12:30:46.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f8cff9640 1 -- 172.21.15.125:0/3907510154 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8000ca30 con 0x7f6f900ca0e0 2023-10-30T12:30:46.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f9693e640 1 -- 172.21.15.125:0/3907510154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f90143840 con 0x7f6f900ca0e0 2023-10-30T12:30:46.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f8cff9640 1 -- 172.21.15.125:0/3907510154 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8000cdc0 con 0x7f6f900ca0e0 2023-10-30T12:30:46.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f9693e640 1 -- 172.21.15.125:0/3907510154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f901f6da0 con 0x7f6f900ca0e0 2023-10-30T12:30:46.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.888+0000 7f6f9693e640 1 -- 172.21.15.125:0/3907510154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f900b2a50 con 0x7f6f900ca0e0 2023-10-30T12:30:46.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.892+0000 7f6f8cff9640 1 -- 172.21.15.125:0/3907510154 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6f80013410 con 0x7f6f900ca0e0 2023-10-30T12:30:46.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.892+0000 7f6f8cff9640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f7c0664d0 0x7f6f7c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:46.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.892+0000 7f6f8cff9640 1 -- 172.21.15.125:0/3907510154 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5824+0+0 (secure 0 0 0) 0x7f6f80083d20 con 0x7f6f900ca0e0 2023-10-30T12:30:46.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.892+0000 7f6f8f7fe640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f7c0664d0 0x7f6f7c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:46.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.892+0000 7f6f8f7fe640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f7c0664d0 0x7f6f7c068990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6f901466f0 tx=0x7f6f84009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:46.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:46.896+0000 7f6f8cff9640 1 -- 172.21.15.125:0/3907510154 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6f80051140 con 0x7f6f900ca0e0 2023-10-30T12:30:47.074 INFO:tasks.workunit.client.0.smithi125.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2023-10-30T12:30:47.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:47.076+0000 7f6f9693e640 1 -- 172.21.15.125:0/3907510154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f9004e170 con 0x7f6f900ca0e0 2023-10-30T12:30:47.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:47.076+0000 7f6f8cff9640 1 -- 172.21.15.125:0/3907510154 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+5746 (secure 0 0 0) 0x7f6f80054760 con 0x7f6f900ca0e0 2023-10-30T12:30:47.084 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:47.085 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:30:41.961152+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:47.085 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 16 2023-10-30T12:30:47.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:47.080+0000 7f6f9693e640 1 -- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f7c0664d0 msgr2=0x7f6f7c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:47.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:47.080+0000 7f6f9693e640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f7c0664d0 0x7f6f7c068990 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6f901466f0 tx=0x7f6f84009040 comp rx=0 tx=0).stop 2023-10-30T12:30:47.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:47.080+0000 7f6f9693e640 1 -- 172.21.15.125:0/3907510154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f900ca0e0 msgr2=0x7f6f900b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:47.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:47.080+0000 7f6f9693e640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f900ca0e0 0x7f6f900b1870 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6f8000e810 tx=0x7f6f8000ecd0 comp rx=0 tx=0).stop 2023-10-30T12:30:47.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:47.080+0000 7f6f9693e640 1 -- 172.21.15.125:0/3907510154 shutdown_connections 2023-10-30T12:30:47.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:47.080+0000 7f6f9693e640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f7c0664d0 0x7f6f7c068990 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:47.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:47.080+0000 7f6f9693e640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f901538a0 0x7f6f90144150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:47.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:47.080+0000 7f6f9693e640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f900cf160 0x7f6f900ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:47.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:47.080+0000 7f6f9693e640 1 --2- 172.21.15.125:0/3907510154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f900ca0e0 0x7f6f900b1870 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:47.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:47.080+0000 7f6f9693e640 1 -- 172.21.15.125:0/3907510154 >> 172.21.15.125:0/3907510154 conn(0x7f6f9006bfe0 msgr2=0x7f6f90148200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:47.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:47.080+0000 7f6f9693e640 1 -- 172.21.15.125:0/3907510154 shutdown_connections 2023-10-30T12:30:47.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:47.080+0000 7f6f9693e640 1 -- 172.21.15.125:0/3907510154 wait complete. 2023-10-30T12:30:47.109 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:30:49.110 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:49.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.248+0000 7fbba3bab640 1 Processor -- start 2023-10-30T12:30:49.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.248+0000 7fbba3bab640 1 -- start start 2023-10-30T12:30:49.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.248+0000 7fbba3bab640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbb9c06f7a0 0x7fbb9c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:49.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.248+0000 7fbba3bab640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb9c071110 0x7fbb9c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:49.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.248+0000 7fbba3bab640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9c066cf0 con 0x7fbb9c0c6bd0 2023-10-30T12:30:49.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.248+0000 7fbba3bab640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb9c0700c0 con 0x7fbb9c071110 2023-10-30T12:30:49.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.248+0000 7fbba3bab640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb9c0c70d0 con 0x7fbb9c06f7a0 2023-10-30T12:30:49.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.248+0000 7fbba1146640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fbb9c0c6bd0 0x7fbb9c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53506/0 (socket says 172.21.15.125:53506) 2023-10-30T12:30:49.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.248+0000 7fbba1146640 1 -- 172.21.15.125:0/2530253996 learned_addr learned my addr 172.21.15.125:0/2530253996 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:30:49.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.248+0000 7fbba2148640 1 --2- 172.21.15.125:0/2530253996 >> v2:172.21.15.125:3300/0 conn(0x7fbb9c071110 0x7fbb9c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:49.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.252+0000 7fbba1947640 1 --2- 172.21.15.125:0/2530253996 >> v2:172.21.15.136:3300/0 conn(0x7fbb9c06f7a0 0x7fbb9c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:49.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.252+0000 7fbba2148640 1 -- 172.21.15.125:0/2530253996 >> v2:172.21.15.136:3300/0 conn(0x7fbb9c06f7a0 msgr2=0x7fbb9c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:49.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.252+0000 7fbba2148640 1 --2- 172.21.15.125:0/2530253996 >> v2:172.21.15.136:3300/0 conn(0x7fbb9c06f7a0 0x7fbb9c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:49.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.252+0000 7fbba2148640 1 -- 172.21.15.125:0/2530253996 >> v1:172.21.15.125:6789/0 conn(0x7fbb9c0c6bd0 legacy=0x7fbb9c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:49.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.252+0000 7fbba2148640 1 -- 172.21.15.125:0/2530253996 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9c0c7390 con 0x7fbb9c071110 2023-10-30T12:30:49.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.252+0000 7fbba0945640 1 -- 172.21.15.125:0/2530253996 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669247250 0 0) 0x7fbb9c066cf0 con 0x7fbb9c0c6bd0 2023-10-30T12:30:49.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.252+0000 7fbba2148640 1 --2- 172.21.15.125:0/2530253996 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9c071110 0x7fbb9c158110 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c06c830 tx=0x7fbb9401f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a73d34c0a9ba6d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:49.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.252+0000 7fbba0945640 1 -- 172.21.15.125:0/2530253996 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb940b0440 con 0x7fbb9c071110 2023-10-30T12:30:49.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.252+0000 7fbba0945640 1 -- 172.21.15.125:0/2530253996 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb940b05a0 con 0x7fbb9c071110 2023-10-30T12:30:49.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.252+0000 7fbba0945640 1 -- 172.21.15.125:0/2530253996 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb940b0930 con 0x7fbb9c071110 2023-10-30T12:30:49.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 -- 172.21.15.125:0/2530253996 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9c071110 msgr2=0x7fbb9c158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:49.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 --2- 172.21.15.125:0/2530253996 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9c071110 0x7fbb9c158110 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c06c830 tx=0x7fbb9401f0f0 comp rx=0 tx=0).stop 2023-10-30T12:30:49.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 -- 172.21.15.125:0/2530253996 shutdown_connections 2023-10-30T12:30:49.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 --2- 172.21.15.125:0/2530253996 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9c071110 0x7fbb9c158110 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:49.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 --2- 172.21.15.125:0/2530253996 >> v2:172.21.15.136:3300/0 conn(0x7fbb9c06f7a0 0x7fbb9c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:49.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 -- 172.21.15.125:0/2530253996 >> 172.21.15.125:0/2530253996 conn(0x7fbb9c0c7910 msgr2=0x7fbb9c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:49.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 -- 172.21.15.125:0/2530253996 shutdown_connections 2023-10-30T12:30:49.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 -- 172.21.15.125:0/2530253996 wait complete. 2023-10-30T12:30:49.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 Processor -- start 2023-10-30T12:30:49.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 -- start start 2023-10-30T12:30:49.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9c06f7a0 0x7fbb9c0ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:49.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb9c071110 0x7fbb9c0c11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:49.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb9c0c6bd0 0x7fbb9c0c1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:49.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c0af330 con 0x7fbb9c06f7a0 2023-10-30T12:30:49.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c1ef720 con 0x7fbb9c071110 2023-10-30T12:30:49.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba3bab640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c0c01a0 con 0x7fbb9c0c6bd0 2023-10-30T12:30:49.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba1947640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9c06f7a0 0x7fbb9c0ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:49.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba1947640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9c06f7a0 0x7fbb9c0ae3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43620/0 (socket says 172.21.15.125:43620) 2023-10-30T12:30:49.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba1947640 1 -- 172.21.15.125:0/1226624999 learned_addr learned my addr 172.21.15.125:0/1226624999 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:49.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba1146640 1 --2- 172.21.15.125:0/1226624999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb9c071110 0x7fbb9c0c11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:49.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba2148640 1 --2- 172.21.15.125:0/1226624999 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb9c0c6bd0 0x7fbb9c0c1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:49.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba1947640 1 -- 172.21.15.125:0/1226624999 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb9c0c6bd0 msgr2=0x7fbb9c0c1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:49.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba1947640 1 --2- 172.21.15.125:0/1226624999 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb9c0c6bd0 0x7fbb9c0c1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:49.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba1947640 1 -- 172.21.15.125:0/1226624999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb9c071110 msgr2=0x7fbb9c0c11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:49.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.256+0000 7fbba1947640 1 --2- 172.21.15.125:0/1226624999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb9c071110 0x7fbb9c0c11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:49.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.260+0000 7fbba1947640 1 -- 172.21.15.125:0/1226624999 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb94009a70 con 0x7fbb9c06f7a0 2023-10-30T12:30:49.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.260+0000 7fbba1947640 1 --2- 172.21.15.125:0/1226624999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9c06f7a0 0x7fbb9c0ae3b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800d5f0 tx=0x7fbb9800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:49.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.260+0000 7fbb927fc640 1 -- 172.21.15.125:0/1226624999 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb980125f0 con 0x7fbb9c06f7a0 2023-10-30T12:30:49.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.260+0000 7fbb927fc640 1 -- 172.21.15.125:0/1226624999 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb98012750 con 0x7fbb9c06f7a0 2023-10-30T12:30:49.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.260+0000 7fbb927fc640 1 -- 172.21.15.125:0/1226624999 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb98012ae0 con 0x7fbb9c06f7a0 2023-10-30T12:30:49.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.260+0000 7fbba3bab640 1 -- 172.21.15.125:0/1226624999 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb9c0c0480 con 0x7fbb9c06f7a0 2023-10-30T12:30:49.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.260+0000 7fbba3bab640 1 -- 172.21.15.125:0/1226624999 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb9c0c0cd0 con 0x7fbb9c06f7a0 2023-10-30T12:30:49.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.260+0000 7fbba3bab640 1 -- 172.21.15.125:0/1226624999 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb9c0c0e90 con 0x7fbb9c06f7a0 2023-10-30T12:30:49.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.260+0000 7fbb927fc640 1 -- 172.21.15.125:0/1226624999 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb98012c40 con 0x7fbb9c06f7a0 2023-10-30T12:30:49.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.260+0000 7fbb927fc640 1 --2- 172.21.15.125:0/1226624999 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb800664a0 0x7fbb80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:49.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.260+0000 7fbb927fc640 1 -- 172.21.15.125:0/1226624999 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fbb980844d0 con 0x7fbb9c06f7a0 2023-10-30T12:30:49.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.260+0000 7fbba1146640 1 --2- 172.21.15.125:0/1226624999 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb800664a0 0x7fbb80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:49.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.264+0000 7fbba1146640 1 --2- 172.21.15.125:0/1226624999 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb800664a0 0x7fbb80068960 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c071f70 tx=0x7fbb8c007b60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:49.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.264+0000 7fbb927fc640 1 -- 172.21.15.125:0/1226624999 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbb980517d0 con 0x7fbb9c06f7a0 2023-10-30T12:30:49.393 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:30:49.384+0000 7f6a6fdf8640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T12:30:49.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.448+0000 7fbba3bab640 1 -- 172.21.15.125:0/1226624999 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb9c04e170 con 0x7fbb9c06f7a0 2023-10-30T12:30:49.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.448+0000 7fbb927fc640 1 -- 172.21.15.125:0/1226624999 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5746 (secure 0 0 0) 0x7fbb98007380 con 0x7fbb9c06f7a0 2023-10-30T12:30:49.457 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:49.457 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:30:47.067933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:49.457 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 17 2023-10-30T12:30:49.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.452+0000 7fbba3bab640 1 -- 172.21.15.125:0/1226624999 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb800664a0 msgr2=0x7fbb80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:49.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.452+0000 7fbba3bab640 1 --2- 172.21.15.125:0/1226624999 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb800664a0 0x7fbb80068960 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c071f70 tx=0x7fbb8c007b60 comp rx=0 tx=0).stop 2023-10-30T12:30:49.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.452+0000 7fbba3bab640 1 -- 172.21.15.125:0/1226624999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9c06f7a0 msgr2=0x7fbb9c0ae3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:49.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.452+0000 7fbba3bab640 1 --2- 172.21.15.125:0/1226624999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9c06f7a0 0x7fbb9c0ae3b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbb9800d5f0 tx=0x7fbb9800dab0 comp rx=0 tx=0).stop 2023-10-30T12:30:49.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.452+0000 7fbba3bab640 1 -- 172.21.15.125:0/1226624999 shutdown_connections 2023-10-30T12:30:49.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.452+0000 7fbba3bab640 1 --2- 172.21.15.125:0/1226624999 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb800664a0 0x7fbb80068960 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:49.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.452+0000 7fbba3bab640 1 --2- 172.21.15.125:0/1226624999 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb9c0c6bd0 0x7fbb9c0c1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:49.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.452+0000 7fbba3bab640 1 --2- 172.21.15.125:0/1226624999 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb9c071110 0x7fbb9c0c11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:49.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.452+0000 7fbba3bab640 1 --2- 172.21.15.125:0/1226624999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9c06f7a0 0x7fbb9c0ae3b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:49.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.452+0000 7fbba3bab640 1 -- 172.21.15.125:0/1226624999 >> 172.21.15.125:0/1226624999 conn(0x7fbb9c0c7910 msgr2=0x7fbb9c0ca6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:49.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.452+0000 7fbba3bab640 1 -- 172.21.15.125:0/1226624999 shutdown_connections 2023-10-30T12:30:49.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:49.452+0000 7fbba3bab640 1 -- 172.21.15.125:0/1226624999 wait complete. 2023-10-30T12:30:49.478 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=2, replacing gid=4279 2023-10-30T12:30:49.478 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.d 2023-10-30T12:30:49.582 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir foo 2023-10-30T12:30:49.590 INFO:tasks.workunit.client.0.smithi125.stderr:+ cp linux-2.6.33.tar.bz2 foo 2023-10-30T12:30:49.677 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir foo/.snap/barsnap 2023-10-30T12:30:52.110 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm foo/linux-2.6.33.tar.bz2 2023-10-30T12:30:52.124 INFO:tasks.workunit.client.0.smithi125.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2023-10-30T12:30:52.503 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo passed: files are identical 2023-10-30T12:30:52.503 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir foo/.snap/barsnap 2023-10-30T12:30:52.504 INFO:tasks.workunit.client.0.smithi125.stdout:passed: files are identical 2023-10-30T12:30:52.533 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo OK 2023-10-30T12:30:52.535 INFO:tasks.workunit.client.0.smithi125.stdout:OK 2023-10-30T12:30:52.535 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:30:52.536 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:30:53.484 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2023-10-30T12:30:53.485 DEBUG:teuthology.orchestra.run.smithi125:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2023-10-30T12:30:54.479 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T12:30:54.480 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T12:30:54.480 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T12:30:54.483 INFO:tasks.ceph.mds.d:Started 2023-10-30T12:30:54.484 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:54.559 INFO:tasks.ceph.mds.d.smithi136.stdout:starting mds.d at 2023-10-30T12:30:54.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.624+0000 7fc591d1d640 1 Processor -- start 2023-10-30T12:30:54.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.624+0000 7fc591d1d640 1 -- start start 2023-10-30T12:30:54.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.628+0000 7fc591d1d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc58c150130 0x7fc58c0c5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:54.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.628+0000 7fc591d1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c18c0 0x7fc58c0c20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:54.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.628+0000 7fc591d1d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c155ce0 con 0x7fc58c069e90 2023-10-30T12:30:54.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.628+0000 7fc591d1d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58c150510 con 0x7fc58c150130 2023-10-30T12:30:54.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.628+0000 7fc591d1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c0c2620 con 0x7fc58c0c18c0 2023-10-30T12:30:54.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.628+0000 7fc58b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc58c150130 0x7fc58c0c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:54.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.628+0000 7fc58b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc58c150130 0x7fc58c0c5660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43638/0 (socket says 172.21.15.125:43638) 2023-10-30T12:30:54.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.628+0000 7fc58b7fe640 1 -- 172.21.15.125:0/4078773764 learned_addr learned my addr 172.21.15.125:0/4078773764 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:54.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.628+0000 7fc58affd640 1 --2- 172.21.15.125:0/4078773764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c18c0 0x7fc58c0c20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:54.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.628+0000 7fc58b7fe640 1 -- 172.21.15.125:0/4078773764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c18c0 msgr2=0x7fc58c0c20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:30:54.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.628+0000 7fc58b7fe640 1 --2- 172.21.15.125:0/4078773764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c18c0 0x7fc58c0c20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:54.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.628+0000 7fc58b7fe640 1 -- 172.21.15.125:0/4078773764 >> v1:172.21.15.136:6789/0 conn(0x7fc58c069e90 legacy=0x7fc58c1561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:54.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.628+0000 7fc58b7fe640 1 -- 172.21.15.125:0/4078773764 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c0c3f60 con 0x7fc58c150130 2023-10-30T12:30:54.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc58b7fe640 1 --2- 172.21.15.125:0/4078773764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c150130 0x7fc58c0c5660 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc57c009870 tx=0x7fc57c01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f60fafdd2793b52 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:54.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc58a7fc640 1 -- 172.21.15.125:0/4078773764 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c0aa430 con 0x7fc58c150130 2023-10-30T12:30:54.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc58a7fc640 1 -- 172.21.15.125:0/4078773764 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57c0aa590 con 0x7fc58c150130 2023-10-30T12:30:54.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc58a7fc640 1 -- 172.21.15.125:0/4078773764 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c0aa920 con 0x7fc58c150130 2023-10-30T12:30:54.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc591d1d640 1 -- 172.21.15.125:0/4078773764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c150130 msgr2=0x7fc58c0c5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:54.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc591d1d640 1 --2- 172.21.15.125:0/4078773764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c150130 0x7fc58c0c5660 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc57c009870 tx=0x7fc57c01c4b0 comp rx=0 tx=0).stop 2023-10-30T12:30:54.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc591d1d640 1 -- 172.21.15.125:0/4078773764 shutdown_connections 2023-10-30T12:30:54.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc591d1d640 1 --2- 172.21.15.125:0/4078773764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c18c0 0x7fc58c0c20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:54.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc591d1d640 1 --2- 172.21.15.125:0/4078773764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c150130 0x7fc58c0c5660 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:54.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc591d1d640 1 -- 172.21.15.125:0/4078773764 >> 172.21.15.125:0/4078773764 conn(0x7fc58c0cace0 msgr2=0x7fc58c0cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:54.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc591d1d640 1 -- 172.21.15.125:0/4078773764 shutdown_connections 2023-10-30T12:30:54.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc591d1d640 1 -- 172.21.15.125:0/4078773764 wait complete. 2023-10-30T12:30:54.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc591d1d640 1 Processor -- start 2023-10-30T12:30:54.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc591d1d640 1 -- start start 2023-10-30T12:30:54.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc591d1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c069e90 0x7fc58c14e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:54.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc591d1d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c18c0 0x7fc58c14aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:54.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.632+0000 7fc591d1d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58c150130 0x7fc58c1f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:54.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc591d1d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c14b010 con 0x7fc58c0c18c0 2023-10-30T12:30:54.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc591d1d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c1f27f0 con 0x7fc58c150130 2023-10-30T12:30:54.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc591d1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c149da0 con 0x7fc58c069e90 2023-10-30T12:30:54.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc58affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c18c0 0x7fc58c14aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:54.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc58affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c18c0 0x7fc58c14aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43646/0 (socket says 172.21.15.125:43646) 2023-10-30T12:30:54.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc58affd640 1 -- 172.21.15.125:0/3953630048 learned_addr learned my addr 172.21.15.125:0/3953630048 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:54.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc58b7fe640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c069e90 0x7fc58c14e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:54.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc58bfff640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58c150130 0x7fc58c1f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:54.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc58affd640 1 -- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c069e90 msgr2=0x7fc58c14e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:54.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc58affd640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c069e90 0x7fc58c14e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:54.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc58affd640 1 -- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58c150130 msgr2=0x7fc58c1f22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:54.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc58affd640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58c150130 0x7fc58c1f22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:54.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc58affd640 1 -- 172.21.15.125:0/3953630048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc57c0094d0 con 0x7fc58c0c18c0 2023-10-30T12:30:54.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc58bfff640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58c150130 0x7fc58c1f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:30:54.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc58affd640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c18c0 0x7fc58c14aad0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc58000d800 tx=0x7fc58000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:54.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc573fff640 1 -- 172.21.15.125:0/3953630048 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58000b660 con 0x7fc58c0c18c0 2023-10-30T12:30:54.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc573fff640 1 -- 172.21.15.125:0/3953630048 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58000b7c0 con 0x7fc58c0c18c0 2023-10-30T12:30:54.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc573fff640 1 -- 172.21.15.125:0/3953630048 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58000bb70 con 0x7fc58c0c18c0 2023-10-30T12:30:54.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc591d1d640 1 -- 172.21.15.125:0/3953630048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58c14d8f0 con 0x7fc58c0c18c0 2023-10-30T12:30:54.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc591d1d640 1 -- 172.21.15.125:0/3953630048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58c14de90 con 0x7fc58c0c18c0 2023-10-30T12:30:54.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc591d1d640 1 -- 172.21.15.125:0/3953630048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58c04e170 con 0x7fc58c0c18c0 2023-10-30T12:30:54.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.636+0000 7fc573fff640 1 -- 172.21.15.125:0/3953630048 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc58000bcd0 con 0x7fc58c0c18c0 2023-10-30T12:30:54.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.640+0000 7fc573fff640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5780664d0 0x7fc578068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:54.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.640+0000 7fc573fff640 1 -- 172.21.15.125:0/3953630048 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5824+0+0 (secure 0 0 0) 0x7fc580084360 con 0x7fc58c0c18c0 2023-10-30T12:30:54.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.640+0000 7fc58b7fe640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5780664d0 0x7fc578068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:54.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.640+0000 7fc58b7fe640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5780664d0 0x7fc578068990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc57c0ae000 tx=0x7fc57c0b5040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:54.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.644+0000 7fc573fff640 1 -- 172.21.15.125:0/3953630048 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc5800516f0 con 0x7fc58c0c18c0 2023-10-30T12:30:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.824+0000 7fc591d1d640 1 -- 172.21.15.125:0/3953630048 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc58c155ac0 con 0x7fc58c0c18c0 2023-10-30T12:30:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.824+0000 7fc573fff640 1 -- 172.21.15.125:0/3953630048 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+5746 (secure 0 0 0) 0x7fc580054d10 con 0x7fc58c0c18c0 2023-10-30T12:30:54.831 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:54.832 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:30:47.067933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 17 2023-10-30T12:30:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.828+0000 7fc591d1d640 1 -- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5780664d0 msgr2=0x7fc578068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.828+0000 7fc591d1d640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5780664d0 0x7fc578068990 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc57c0ae000 tx=0x7fc57c0b5040 comp rx=0 tx=0).stop 2023-10-30T12:30:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.828+0000 7fc591d1d640 1 -- 172.21.15.125:0/3953630048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c18c0 msgr2=0x7fc58c14aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.828+0000 7fc591d1d640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c18c0 0x7fc58c14aad0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc58000d800 tx=0x7fc58000dcc0 comp rx=0 tx=0).stop 2023-10-30T12:30:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.828+0000 7fc591d1d640 1 -- 172.21.15.125:0/3953630048 shutdown_connections 2023-10-30T12:30:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.828+0000 7fc591d1d640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5780664d0 0x7fc578068990 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.828+0000 7fc591d1d640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58c150130 0x7fc58c1f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:54.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.828+0000 7fc591d1d640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c18c0 0x7fc58c14aad0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:54.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.828+0000 7fc591d1d640 1 --2- 172.21.15.125:0/3953630048 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c069e90 0x7fc58c14e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:54.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.828+0000 7fc591d1d640 1 -- 172.21.15.125:0/3953630048 >> 172.21.15.125:0/3953630048 conn(0x7fc58c0cace0 msgr2=0x7fc58c1617d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:54.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.828+0000 7fc591d1d640 1 -- 172.21.15.125:0/3953630048 shutdown_connections 2023-10-30T12:30:54.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:54.828+0000 7fc591d1d640 1 -- 172.21.15.125:0/3953630048 wait complete. 2023-10-30T12:30:54.852 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T12:30:55.125 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo 1 2023-10-30T12:30:55.126 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo 2 2023-10-30T12:30:55.127 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo 3 2023-10-30T12:30:55.127 INFO:tasks.workunit.client.0.smithi125.stderr:+ '[' -e file4 ']' 2023-10-30T12:30:55.127 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/snap1 2023-10-30T12:30:55.140 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo 4 2023-10-30T12:30:55.141 INFO:tasks.workunit.client.0.smithi125.stderr:++ ls 2023-10-30T12:30:55.146 INFO:tasks.workunit.client.0.smithi125.stderr:+ now='file1 2023-10-30T12:30:55.146 INFO:tasks.workunit.client.0.smithi125.stderr:file2 2023-10-30T12:30:55.147 INFO:tasks.workunit.client.0.smithi125.stderr:file3 2023-10-30T12:30:55.147 INFO:tasks.workunit.client.0.smithi125.stderr:file4' 2023-10-30T12:30:55.147 INFO:tasks.workunit.client.0.smithi125.stderr:++ ls .snap/snap1 2023-10-30T12:30:56.853 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:30:56.906 INFO:tasks.workunit.client.0.smithi125.stderr:+ then='file1 2023-10-30T12:30:56.907 INFO:tasks.workunit.client.0.smithi125.stderr:file2 2023-10-30T12:30:56.907 INFO:tasks.workunit.client.0.smithi125.stderr:file3' 2023-10-30T12:30:56.907 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir .snap/snap1 2023-10-30T12:30:56.917 INFO:tasks.workunit.client.0.smithi125.stderr:+ '[' 'file1 2023-10-30T12:30:56.917 INFO:tasks.workunit.client.0.smithi125.stderr:file2 2023-10-30T12:30:56.917 INFO:tasks.workunit.client.0.smithi125.stderr:file3 2023-10-30T12:30:56.918 INFO:tasks.workunit.client.0.smithi125.stderr:file4' = 'file1 2023-10-30T12:30:56.918 INFO:tasks.workunit.client.0.smithi125.stderr:file2 2023-10-30T12:30:56.918 INFO:tasks.workunit.client.0.smithi125.stderr:file3' ']' 2023-10-30T12:30:56.918 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo 1 2023-10-30T12:30:56.925 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo 2 2023-10-30T12:30:56.945 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo 3 2023-10-30T12:30:56.953 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/snap1 2023-10-30T12:30:56.964 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo 4 2023-10-30T12:30:56.972 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir .snap/snap1 2023-10-30T12:30:56.983 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm file1 file2 file3 file4 2023-10-30T12:30:56.990 INFO:tasks.workunit.client.0.smithi125.stdout:OK 2023-10-30T12:30:56.991 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo OK 2023-10-30T12:30:56.991 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:30:56.992 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:30:57.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.012+0000 7fb810dd3640 1 Processor -- start 2023-10-30T12:30:57.020 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2023-10-30T12:30:57.021 DEBUG:teuthology.orchestra.run.smithi125:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2023-10-30T12:30:57.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.012+0000 7fb810dd3640 1 -- start start 2023-10-30T12:30:57.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.016+0000 7fb810dd3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb80c1538a0 0x7fb80c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:57.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.016+0000 7fb810dd3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb80c0cb800 con 0x7fb80c0cf160 2023-10-30T12:30:57.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.016+0000 7fb810dd3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb80c15dc30 con 0x7fb80c0ca0e0 2023-10-30T12:30:57.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.016+0000 7fb810dd3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb80c0cb330 con 0x7fb80c1538a0 2023-10-30T12:30:57.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.016+0000 7fb80a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb80c1538a0 0x7fb80c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:57.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.016+0000 7fb80a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb80c1538a0 0x7fb80c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43650/0 (socket says 172.21.15.125:43650) 2023-10-30T12:30:57.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.016+0000 7fb80ad9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb80c0cf160 0x7fb80c164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:32832/0 (socket says 172.21.15.125:32832) 2023-10-30T12:30:57.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.016+0000 7fb80a59c640 1 -- 172.21.15.125:0/1327145681 learned_addr learned my addr 172.21.15.125:0/1327145681 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:57.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.016+0000 7fb80a59c640 1 -- 172.21.15.125:0/1327145681 >> v1:172.21.15.136:6789/0 conn(0x7fb80c0ca0e0 legacy=0x7fb80c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:57.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.016+0000 7fb80a59c640 1 -- 172.21.15.125:0/1327145681 >> v1:172.21.15.125:6789/0 conn(0x7fb80c0cf160 legacy=0x7fb80c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:57.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.016+0000 7fb80a59c640 1 -- 172.21.15.125:0/1327145681 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80c0cad70 con 0x7fb80c1538a0 2023-10-30T12:30:57.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.016+0000 7fb80959a640 1 -- 172.21.15.125:0/1327145681 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971355947 0 0) 0x7fb80c0cb800 con 0x7fb80c0cf160 2023-10-30T12:30:57.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.020+0000 7fb80a59c640 1 --2- 172.21.15.125:0/1327145681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb80c1538a0 0x7fb80c15d6f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009870 tx=0x7fb7fc01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a7772a3063251fb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:57.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.020+0000 7fb80959a640 1 -- 172.21.15.125:0/1327145681 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc0a8a30 con 0x7fb80c1538a0 2023-10-30T12:30:57.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.020+0000 7fb80959a640 1 -- 172.21.15.125:0/1327145681 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7fc0a8b90 con 0x7fb80c1538a0 2023-10-30T12:30:57.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.020+0000 7fb80959a640 1 -- 172.21.15.125:0/1327145681 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc0b0440 con 0x7fb80c1538a0 2023-10-30T12:30:57.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.020+0000 7fb810dd3640 1 -- 172.21.15.125:0/1327145681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb80c1538a0 msgr2=0x7fb80c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:57.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.020+0000 7fb810dd3640 1 --2- 172.21.15.125:0/1327145681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb80c1538a0 0x7fb80c15d6f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc009870 tx=0x7fb7fc01ed30 comp rx=0 tx=0).stop 2023-10-30T12:30:57.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.020+0000 7fb810dd3640 1 -- 172.21.15.125:0/1327145681 shutdown_connections 2023-10-30T12:30:57.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.020+0000 7fb810dd3640 1 --2- 172.21.15.125:0/1327145681 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb80c1538a0 0x7fb80c15d6f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:57.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.020+0000 7fb810dd3640 1 -- 172.21.15.125:0/1327145681 >> 172.21.15.125:0/1327145681 conn(0x7fb80c06bfe0 msgr2=0x7fb80c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:57.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.020+0000 7fb810dd3640 1 -- 172.21.15.125:0/1327145681 shutdown_connections 2023-10-30T12:30:57.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.020+0000 7fb810dd3640 1 -- 172.21.15.125:0/1327145681 wait complete. 2023-10-30T12:30:57.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.020+0000 7fb810dd3640 1 Processor -- start 2023-10-30T12:30:57.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.020+0000 7fb810dd3640 1 -- start start 2023-10-30T12:30:57.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb810dd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb80c0ca0e0 0x7fb80c0c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:57.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb810dd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb80c0cf160 0x7fb80c1507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:57.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb810dd3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb80c1538a0 0x7fb80c0b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:57.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb810dd3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c157830 con 0x7fb80c1538a0 2023-10-30T12:30:57.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb810dd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c1575c0 con 0x7fb80c0ca0e0 2023-10-30T12:30:57.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb810dd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb80c0b3d90 con 0x7fb80c0cf160 2023-10-30T12:30:57.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb80ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb80c1538a0 0x7fb80c0b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:57.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb80ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb80c1538a0 0x7fb80c0b42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43666/0 (socket says 172.21.15.125:43666) 2023-10-30T12:30:57.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb80ad9d640 1 -- 172.21.15.125:0/2343557267 learned_addr learned my addr 172.21.15.125:0/2343557267 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:30:57.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb809d9b640 1 --2- 172.21.15.125:0/2343557267 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb80c0cf160 0x7fb80c1507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:57.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb80a59c640 1 --2- 172.21.15.125:0/2343557267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb80c0ca0e0 0x7fb80c0c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:57.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb80ad9d640 1 -- 172.21.15.125:0/2343557267 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb80c0cf160 msgr2=0x7fb80c1507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:57.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb80ad9d640 1 --2- 172.21.15.125:0/2343557267 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb80c0cf160 0x7fb80c1507d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:57.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb80ad9d640 1 -- 172.21.15.125:0/2343557267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb80c0ca0e0 msgr2=0x7fb80c0c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:57.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb80ad9d640 1 --2- 172.21.15.125:0/2343557267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb80c0ca0e0 0x7fb80c0c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:57.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb80ad9d640 1 -- 172.21.15.125:0/2343557267 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7fc0094d0 con 0x7fb80c1538a0 2023-10-30T12:30:57.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb80ad9d640 1 --2- 172.21.15.125:0/2343557267 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb80c1538a0 0x7fb80c0b42e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb804001570 tx=0x7fb80400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:57.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb7faffd640 1 -- 172.21.15.125:0/2343557267 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80400cea0 con 0x7fb80c1538a0 2023-10-30T12:30:57.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb7faffd640 1 -- 172.21.15.125:0/2343557267 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80400a5e0 con 0x7fb80c1538a0 2023-10-30T12:30:57.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb810dd3640 1 -- 172.21.15.125:0/2343557267 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb80c159040 con 0x7fb80c1538a0 2023-10-30T12:30:57.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb7faffd640 1 -- 172.21.15.125:0/2343557267 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80400a990 con 0x7fb80c1538a0 2023-10-30T12:30:57.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb810dd3640 1 -- 172.21.15.125:0/2343557267 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80c0b4010 con 0x7fb80c1538a0 2023-10-30T12:30:57.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb810dd3640 1 -- 172.21.15.125:0/2343557267 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb80c04e170 con 0x7fb80c1538a0 2023-10-30T12:30:57.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.024+0000 7fb7faffd640 1 -- 172.21.15.125:0/2343557267 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb80400aaf0 con 0x7fb80c1538a0 2023-10-30T12:30:57.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.028+0000 7fb7faffd640 1 --2- 172.21.15.125:0/2343557267 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7ec0664a0 0x7fb7ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:30:57.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.028+0000 7fb7faffd640 1 -- 172.21.15.125:0/2343557267 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5880+0+0 (secure 0 0 0) 0x7fb8040835d0 con 0x7fb80c1538a0 2023-10-30T12:30:57.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.028+0000 7fb80a59c640 1 --2- 172.21.15.125:0/2343557267 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7ec0664a0 0x7fb7ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:30:57.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.028+0000 7fb80a59c640 1 --2- 172.21.15.125:0/2343557267 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7ec0664a0 0x7fb7ec068960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0b2000 tx=0x7fb7fc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:30:57.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.032+0000 7fb7faffd640 1 -- 172.21.15.125:0/2343557267 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb804050a50 con 0x7fb80c1538a0 2023-10-30T12:30:57.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.216+0000 7fb810dd3640 1 -- 172.21.15.125:0/2343557267 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb80c159b40 con 0x7fb80c1538a0 2023-10-30T12:30:57.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.220+0000 7fb7faffd640 1 -- 172.21.15.125:0/2343557267 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+6540 (secure 0 0 0) 0x7fb804054070 con 0x7fb80c1538a0 2023-10-30T12:30:57.225 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:30:57.226 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":18,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:30:47.067933+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:30:57.226 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 18 2023-10-30T12:30:57.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.220+0000 7fb810dd3640 1 -- 172.21.15.125:0/2343557267 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7ec0664a0 msgr2=0x7fb7ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:57.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.220+0000 7fb810dd3640 1 --2- 172.21.15.125:0/2343557267 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7ec0664a0 0x7fb7ec068960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0b2000 tx=0x7fb7fc009040 comp rx=0 tx=0).stop 2023-10-30T12:30:57.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.220+0000 7fb810dd3640 1 -- 172.21.15.125:0/2343557267 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb80c1538a0 msgr2=0x7fb80c0b42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:30:57.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.220+0000 7fb810dd3640 1 --2- 172.21.15.125:0/2343557267 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb80c1538a0 0x7fb80c0b42e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fb804001570 tx=0x7fb80400cc50 comp rx=0 tx=0).stop 2023-10-30T12:30:57.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.220+0000 7fb810dd3640 1 -- 172.21.15.125:0/2343557267 shutdown_connections 2023-10-30T12:30:57.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.220+0000 7fb810dd3640 1 --2- 172.21.15.125:0/2343557267 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7ec0664a0 0x7fb7ec068960 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:57.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.220+0000 7fb810dd3640 1 --2- 172.21.15.125:0/2343557267 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb80c1538a0 0x7fb80c0b42e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:57.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.220+0000 7fb810dd3640 1 --2- 172.21.15.125:0/2343557267 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb80c0cf160 0x7fb80c1507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:57.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.220+0000 7fb810dd3640 1 --2- 172.21.15.125:0/2343557267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb80c0ca0e0 0x7fb80c0c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:30:57.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.220+0000 7fb810dd3640 1 -- 172.21.15.125:0/2343557267 >> 172.21.15.125:0/2343557267 conn(0x7fb80c06bfe0 msgr2=0x7fb80c152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:30:57.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.220+0000 7fb810dd3640 1 -- 172.21.15.125:0/2343557267 shutdown_connections 2023-10-30T12:30:57.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:30:57.220+0000 7fb810dd3640 1 -- 172.21.15.125:0/2343557267 wait complete. 2023-10-30T12:30:57.246 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T12:30:57.247 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 85 secs before thrashing 2023-10-30T12:31:00.123 INFO:tasks.workunit.client.0.smithi125.stdout:Create dir 100 to 199 ... 2023-10-30T12:31:00.124 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 100 2023-10-30T12:31:00.131 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 101 2023-10-30T12:31:00.141 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 102 2023-10-30T12:31:00.151 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 103 2023-10-30T12:31:00.160 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 104 2023-10-30T12:31:00.170 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 105 2023-10-30T12:31:00.179 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 106 2023-10-30T12:31:00.188 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 107 2023-10-30T12:31:00.197 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 108 2023-10-30T12:31:00.206 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 109 2023-10-30T12:31:00.220 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 110 2023-10-30T12:31:00.236 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 111 2023-10-30T12:31:00.251 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 112 2023-10-30T12:31:00.266 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 113 2023-10-30T12:31:00.281 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 114 2023-10-30T12:31:00.296 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 115 2023-10-30T12:31:00.311 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 116 2023-10-30T12:31:00.326 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 117 2023-10-30T12:31:00.341 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 118 2023-10-30T12:31:00.357 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 119 2023-10-30T12:31:00.619 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 120 2023-10-30T12:31:00.633 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 121 2023-10-30T12:31:00.644 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 122 2023-10-30T12:31:00.654 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 123 2023-10-30T12:31:00.665 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 124 2023-10-30T12:31:00.675 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 125 2023-10-30T12:31:00.734 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 126 2023-10-30T12:31:01.599 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 127 2023-10-30T12:31:01.613 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 128 2023-10-30T12:31:01.626 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 129 2023-10-30T12:31:01.640 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 130 2023-10-30T12:31:01.650 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 131 2023-10-30T12:31:01.658 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 132 2023-10-30T12:31:01.667 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 133 2023-10-30T12:31:01.676 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 134 2023-10-30T12:31:01.685 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 135 2023-10-30T12:31:01.698 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 136 2023-10-30T12:31:01.707 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 137 2023-10-30T12:31:01.730 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 138 2023-10-30T12:31:02.522 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 139 2023-10-30T12:31:02.719 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 140 2023-10-30T12:31:02.831 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 141 2023-10-30T12:31:02.840 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 142 2023-10-30T12:31:02.849 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 143 2023-10-30T12:31:02.859 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 144 2023-10-30T12:31:02.868 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 145 2023-10-30T12:31:02.877 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 146 2023-10-30T12:31:03.672 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 147 2023-10-30T12:31:03.687 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 148 2023-10-30T12:31:03.702 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 149 2023-10-30T12:31:03.712 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 150 2023-10-30T12:31:03.722 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 151 2023-10-30T12:31:03.732 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 152 2023-10-30T12:31:03.742 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 153 2023-10-30T12:31:03.752 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 154 2023-10-30T12:31:04.419 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 155 2023-10-30T12:31:04.430 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 156 2023-10-30T12:31:04.440 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 157 2023-10-30T12:31:04.451 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 158 2023-10-30T12:31:04.462 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 159 2023-10-30T12:31:04.472 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 160 2023-10-30T12:31:04.483 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 161 2023-10-30T12:31:04.735 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 162 2023-10-30T12:31:04.824 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 163 2023-10-30T12:31:04.834 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 164 2023-10-30T12:31:04.844 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 165 2023-10-30T12:31:04.854 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 166 2023-10-30T12:31:05.123 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 167 2023-10-30T12:31:06.722 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 168 2023-10-30T12:31:07.015 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 169 2023-10-30T12:31:07.026 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 170 2023-10-30T12:31:07.038 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 171 2023-10-30T12:31:07.049 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 172 2023-10-30T12:31:07.061 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 173 2023-10-30T12:31:07.073 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 174 2023-10-30T12:31:07.084 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 175 2023-10-30T12:31:07.095 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 176 2023-10-30T12:31:07.106 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 177 2023-10-30T12:31:07.116 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 178 2023-10-30T12:31:07.127 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 179 2023-10-30T12:31:07.743 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 180 2023-10-30T12:31:08.038 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 181 2023-10-30T12:31:08.050 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 182 2023-10-30T12:31:08.063 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 183 2023-10-30T12:31:08.076 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 184 2023-10-30T12:31:08.087 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 185 2023-10-30T12:31:08.171 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 186 2023-10-30T12:31:08.186 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 187 2023-10-30T12:31:08.735 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 188 2023-10-30T12:31:09.025 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 189 2023-10-30T12:31:09.035 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 190 2023-10-30T12:31:09.295 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 191 2023-10-30T12:31:09.308 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 192 2023-10-30T12:31:09.321 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 193 2023-10-30T12:31:09.337 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 194 2023-10-30T12:31:09.989 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 195 2023-10-30T12:31:10.003 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 196 2023-10-30T12:31:10.017 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 197 2023-10-30T12:31:10.491 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 198 2023-10-30T12:31:10.501 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 199 2023-10-30T12:31:10.511 INFO:tasks.workunit.client.0.smithi125.stdout:Take first snapshot .snap/test1 2023-10-30T12:31:10.680 INFO:tasks.workunit.client.0.smithi125.stdout:Create dir 200 to 299 ... 2023-10-30T12:31:10.682 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 200 2023-10-30T12:31:10.688 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 201 2023-10-30T12:31:10.701 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 202 2023-10-30T12:31:10.713 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 203 2023-10-30T12:31:10.724 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 204 2023-10-30T12:31:10.733 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 205 2023-10-30T12:31:10.741 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 206 2023-10-30T12:31:10.750 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 207 2023-10-30T12:31:10.759 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 208 2023-10-30T12:31:10.767 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 209 2023-10-30T12:31:10.776 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 210 2023-10-30T12:31:10.785 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 211 2023-10-30T12:31:10.794 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 212 2023-10-30T12:31:10.802 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 213 2023-10-30T12:31:10.810 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 214 2023-10-30T12:31:10.819 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 215 2023-10-30T12:31:11.078 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 216 2023-10-30T12:31:11.088 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 217 2023-10-30T12:31:11.346 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 218 2023-10-30T12:31:11.354 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 219 2023-10-30T12:31:11.363 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 220 2023-10-30T12:31:11.371 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 221 2023-10-30T12:31:11.380 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 222 2023-10-30T12:31:11.388 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 223 2023-10-30T12:31:11.397 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 224 2023-10-30T12:31:11.406 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 225 2023-10-30T12:31:11.414 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 226 2023-10-30T12:31:11.422 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 227 2023-10-30T12:31:11.742 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 228 2023-10-30T12:31:12.365 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 229 2023-10-30T12:31:12.507 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 230 2023-10-30T12:31:12.517 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 231 2023-10-30T12:31:12.735 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 232 2023-10-30T12:31:13.314 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 233 2023-10-30T12:31:13.325 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 234 2023-10-30T12:31:13.335 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 235 2023-10-30T12:31:13.346 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 236 2023-10-30T12:31:13.356 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 237 2023-10-30T12:31:13.366 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 238 2023-10-30T12:31:13.376 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 239 2023-10-30T12:31:13.386 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 240 2023-10-30T12:31:13.395 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 241 2023-10-30T12:31:13.405 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 242 2023-10-30T12:31:13.415 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 243 2023-10-30T12:31:13.424 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 244 2023-10-30T12:31:13.434 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 245 2023-10-30T12:31:13.444 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 246 2023-10-30T12:31:13.454 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 247 2023-10-30T12:31:13.464 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 248 2023-10-30T12:31:13.473 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 249 2023-10-30T12:31:13.481 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 250 2023-10-30T12:31:13.504 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 251 2023-10-30T12:31:14.195 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:31:14.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:31:15.142 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 252 2023-10-30T12:31:15.151 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 253 2023-10-30T12:31:15.161 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 254 2023-10-30T12:31:15.170 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 255 2023-10-30T12:31:15.179 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 256 2023-10-30T12:31:15.189 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 257 2023-10-30T12:31:15.198 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 258 2023-10-30T12:31:15.207 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 259 2023-10-30T12:31:15.216 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 260 2023-10-30T12:31:15.225 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 261 2023-10-30T12:31:15.583 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 262 2023-10-30T12:31:15.843 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 263 2023-10-30T12:31:16.103 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 264 2023-10-30T12:31:16.156 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 265 2023-10-30T12:31:16.166 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 266 2023-10-30T12:31:16.176 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 267 2023-10-30T12:31:16.186 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 268 2023-10-30T12:31:16.196 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 269 2023-10-30T12:31:16.206 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 270 2023-10-30T12:31:16.217 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 271 2023-10-30T12:31:16.227 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 272 2023-10-30T12:31:16.238 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 273 2023-10-30T12:31:16.248 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 274 2023-10-30T12:31:16.258 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 275 2023-10-30T12:31:16.268 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 276 2023-10-30T12:31:16.277 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 277 2023-10-30T12:31:16.287 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 278 2023-10-30T12:31:16.297 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 279 2023-10-30T12:31:16.306 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 280 2023-10-30T12:31:16.315 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 281 2023-10-30T12:31:16.325 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 282 2023-10-30T12:31:16.334 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 283 2023-10-30T12:31:16.343 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 284 2023-10-30T12:31:16.352 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 285 2023-10-30T12:31:16.361 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 286 2023-10-30T12:31:16.370 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 287 2023-10-30T12:31:16.379 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 288 2023-10-30T12:31:16.735 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 289 2023-10-30T12:31:16.761 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 290 2023-10-30T12:31:16.773 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 291 2023-10-30T12:31:16.782 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 292 2023-10-30T12:31:16.790 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 293 2023-10-30T12:31:16.799 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 294 2023-10-30T12:31:16.809 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 295 2023-10-30T12:31:16.818 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 296 2023-10-30T12:31:16.886 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 297 2023-10-30T12:31:16.899 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 298 2023-10-30T12:31:16.913 INFO:tasks.workunit.client.0.smithi125.stdout: create dir 299 2023-10-30T12:31:16.925 INFO:tasks.workunit.client.0.smithi125.stdout:Create a snapshot in every first level dir ... 2023-10-30T12:31:17.089 INFO:tasks.workunit.client.0.smithi125.stdout: create 100/.snap/snap-subdir-test 2023-10-30T12:31:17.109 INFO:tasks.workunit.client.0.smithi125.stdout: create 100/30 file after the snapshot 2023-10-30T12:31:17.111 INFO:tasks.workunit.client.0.smithi125.stdout: create 100/31 file after the snapshot 2023-10-30T12:31:17.111 INFO:tasks.workunit.client.0.smithi125.stdout: create 100/32 file after the snapshot 2023-10-30T12:31:17.112 INFO:tasks.workunit.client.0.smithi125.stdout: create 100/33 file after the snapshot 2023-10-30T12:31:17.112 INFO:tasks.workunit.client.0.smithi125.stdout: create 100/34 file after the snapshot 2023-10-30T12:31:17.112 INFO:tasks.workunit.client.0.smithi125.stdout: create 100/35 file after the snapshot 2023-10-30T12:31:17.112 INFO:tasks.workunit.client.0.smithi125.stdout: create 100/36 file after the snapshot 2023-10-30T12:31:17.112 INFO:tasks.workunit.client.0.smithi125.stdout: create 100/37 file after the snapshot 2023-10-30T12:31:17.113 INFO:tasks.workunit.client.0.smithi125.stdout: create 100/38 file after the snapshot 2023-10-30T12:31:17.113 INFO:tasks.workunit.client.0.smithi125.stdout: create 100/39 file after the snapshot 2023-10-30T12:31:17.113 INFO:tasks.workunit.client.0.smithi125.stdout: create 101/.snap/snap-subdir-test 2023-10-30T12:31:17.139 INFO:tasks.workunit.client.0.smithi125.stdout: create 101/30 file after the snapshot 2023-10-30T12:31:17.141 INFO:tasks.workunit.client.0.smithi125.stdout: create 101/31 file after the snapshot 2023-10-30T12:31:17.142 INFO:tasks.workunit.client.0.smithi125.stdout: create 101/32 file after the snapshot 2023-10-30T12:31:17.142 INFO:tasks.workunit.client.0.smithi125.stdout: create 101/33 file after the snapshot 2023-10-30T12:31:17.142 INFO:tasks.workunit.client.0.smithi125.stdout: create 101/34 file after the snapshot 2023-10-30T12:31:17.142 INFO:tasks.workunit.client.0.smithi125.stdout: create 101/35 file after the snapshot 2023-10-30T12:31:17.142 INFO:tasks.workunit.client.0.smithi125.stdout: create 101/36 file after the snapshot 2023-10-30T12:31:17.143 INFO:tasks.workunit.client.0.smithi125.stdout: create 101/37 file after the snapshot 2023-10-30T12:31:17.143 INFO:tasks.workunit.client.0.smithi125.stdout: create 101/38 file after the snapshot 2023-10-30T12:31:17.143 INFO:tasks.workunit.client.0.smithi125.stdout: create 101/39 file after the snapshot 2023-10-30T12:31:17.143 INFO:tasks.workunit.client.0.smithi125.stdout: create 102/.snap/snap-subdir-test 2023-10-30T12:31:17.169 INFO:tasks.workunit.client.0.smithi125.stdout: create 102/30 file after the snapshot 2023-10-30T12:31:17.171 INFO:tasks.workunit.client.0.smithi125.stdout: create 102/31 file after the snapshot 2023-10-30T12:31:17.171 INFO:tasks.workunit.client.0.smithi125.stdout: create 102/32 file after the snapshot 2023-10-30T12:31:17.172 INFO:tasks.workunit.client.0.smithi125.stdout: create 102/33 file after the snapshot 2023-10-30T12:31:17.172 INFO:tasks.workunit.client.0.smithi125.stdout: create 102/34 file after the snapshot 2023-10-30T12:31:17.172 INFO:tasks.workunit.client.0.smithi125.stdout: create 102/35 file after the snapshot 2023-10-30T12:31:17.172 INFO:tasks.workunit.client.0.smithi125.stdout: create 102/36 file after the snapshot 2023-10-30T12:31:17.173 INFO:tasks.workunit.client.0.smithi125.stdout: create 102/37 file after the snapshot 2023-10-30T12:31:17.173 INFO:tasks.workunit.client.0.smithi125.stdout: create 102/38 file after the snapshot 2023-10-30T12:31:17.173 INFO:tasks.workunit.client.0.smithi125.stdout: create 102/39 file after the snapshot 2023-10-30T12:31:17.173 INFO:tasks.workunit.client.0.smithi125.stdout: create 103/.snap/snap-subdir-test 2023-10-30T12:31:17.609 INFO:tasks.workunit.client.0.smithi125.stdout: create 103/30 file after the snapshot 2023-10-30T12:31:17.611 INFO:tasks.workunit.client.0.smithi125.stdout: create 103/31 file after the snapshot 2023-10-30T12:31:17.611 INFO:tasks.workunit.client.0.smithi125.stdout: create 103/32 file after the snapshot 2023-10-30T12:31:17.612 INFO:tasks.workunit.client.0.smithi125.stdout: create 103/33 file after the snapshot 2023-10-30T12:31:17.612 INFO:tasks.workunit.client.0.smithi125.stdout: create 103/34 file after the snapshot 2023-10-30T12:31:17.612 INFO:tasks.workunit.client.0.smithi125.stdout: create 103/35 file after the snapshot 2023-10-30T12:31:17.612 INFO:tasks.workunit.client.0.smithi125.stdout: create 103/36 file after the snapshot 2023-10-30T12:31:17.612 INFO:tasks.workunit.client.0.smithi125.stdout: create 103/37 file after the snapshot 2023-10-30T12:31:17.613 INFO:tasks.workunit.client.0.smithi125.stdout: create 103/38 file after the snapshot 2023-10-30T12:31:17.613 INFO:tasks.workunit.client.0.smithi125.stdout: create 103/39 file after the snapshot 2023-10-30T12:31:17.613 INFO:tasks.workunit.client.0.smithi125.stdout: create 104/.snap/snap-subdir-test 2023-10-30T12:31:17.644 INFO:tasks.workunit.client.0.smithi125.stdout: create 104/30 file after the snapshot 2023-10-30T12:31:17.646 INFO:tasks.workunit.client.0.smithi125.stdout: create 104/31 file after the snapshot 2023-10-30T12:31:17.646 INFO:tasks.workunit.client.0.smithi125.stdout: create 104/32 file after the snapshot 2023-10-30T12:31:17.646 INFO:tasks.workunit.client.0.smithi125.stdout: create 104/33 file after the snapshot 2023-10-30T12:31:17.646 INFO:tasks.workunit.client.0.smithi125.stdout: create 104/34 file after the snapshot 2023-10-30T12:31:17.646 INFO:tasks.workunit.client.0.smithi125.stdout: create 104/35 file after the snapshot 2023-10-30T12:31:17.647 INFO:tasks.workunit.client.0.smithi125.stdout: create 104/36 file after the snapshot 2023-10-30T12:31:17.647 INFO:tasks.workunit.client.0.smithi125.stdout: create 104/37 file after the snapshot 2023-10-30T12:31:17.647 INFO:tasks.workunit.client.0.smithi125.stdout: create 104/38 file after the snapshot 2023-10-30T12:31:17.647 INFO:tasks.workunit.client.0.smithi125.stdout: create 104/39 file after the snapshot 2023-10-30T12:31:17.647 INFO:tasks.workunit.client.0.smithi125.stdout: create 105/.snap/snap-subdir-test 2023-10-30T12:31:17.674 INFO:tasks.workunit.client.0.smithi125.stdout: create 105/30 file after the snapshot 2023-10-30T12:31:17.675 INFO:tasks.workunit.client.0.smithi125.stdout: create 105/31 file after the snapshot 2023-10-30T12:31:17.676 INFO:tasks.workunit.client.0.smithi125.stdout: create 105/32 file after the snapshot 2023-10-30T12:31:17.676 INFO:tasks.workunit.client.0.smithi125.stdout: create 105/33 file after the snapshot 2023-10-30T12:31:17.676 INFO:tasks.workunit.client.0.smithi125.stdout: create 105/34 file after the snapshot 2023-10-30T12:31:17.676 INFO:tasks.workunit.client.0.smithi125.stdout: create 105/35 file after the snapshot 2023-10-30T12:31:17.677 INFO:tasks.workunit.client.0.smithi125.stdout: create 105/36 file after the snapshot 2023-10-30T12:31:17.677 INFO:tasks.workunit.client.0.smithi125.stdout: create 105/37 file after the snapshot 2023-10-30T12:31:17.677 INFO:tasks.workunit.client.0.smithi125.stdout: create 105/38 file after the snapshot 2023-10-30T12:31:17.677 INFO:tasks.workunit.client.0.smithi125.stdout: create 105/39 file after the snapshot 2023-10-30T12:31:17.677 INFO:tasks.workunit.client.0.smithi125.stdout: create 106/.snap/snap-subdir-test 2023-10-30T12:31:17.704 INFO:tasks.workunit.client.0.smithi125.stdout: create 106/30 file after the snapshot 2023-10-30T12:31:17.706 INFO:tasks.workunit.client.0.smithi125.stdout: create 106/31 file after the snapshot 2023-10-30T12:31:17.706 INFO:tasks.workunit.client.0.smithi125.stdout: create 106/32 file after the snapshot 2023-10-30T12:31:17.706 INFO:tasks.workunit.client.0.smithi125.stdout: create 106/33 file after the snapshot 2023-10-30T12:31:17.706 INFO:tasks.workunit.client.0.smithi125.stdout: create 106/34 file after the snapshot 2023-10-30T12:31:17.706 INFO:tasks.workunit.client.0.smithi125.stdout: create 106/35 file after the snapshot 2023-10-30T12:31:17.707 INFO:tasks.workunit.client.0.smithi125.stdout: create 106/36 file after the snapshot 2023-10-30T12:31:17.707 INFO:tasks.workunit.client.0.smithi125.stdout: create 106/37 file after the snapshot 2023-10-30T12:31:17.707 INFO:tasks.workunit.client.0.smithi125.stdout: create 106/38 file after the snapshot 2023-10-30T12:31:17.707 INFO:tasks.workunit.client.0.smithi125.stdout: create 106/39 file after the snapshot 2023-10-30T12:31:17.707 INFO:tasks.workunit.client.0.smithi125.stdout: create 107/.snap/snap-subdir-test 2023-10-30T12:31:18.013 INFO:tasks.workunit.client.0.smithi125.stdout: create 107/30 file after the snapshot 2023-10-30T12:31:18.018 INFO:tasks.workunit.client.0.smithi125.stdout: create 107/31 file after the snapshot 2023-10-30T12:31:18.019 INFO:tasks.workunit.client.0.smithi125.stdout: create 107/32 file after the snapshot 2023-10-30T12:31:18.019 INFO:tasks.workunit.client.0.smithi125.stdout: create 107/33 file after the snapshot 2023-10-30T12:31:18.020 INFO:tasks.workunit.client.0.smithi125.stdout: create 107/34 file after the snapshot 2023-10-30T12:31:18.020 INFO:tasks.workunit.client.0.smithi125.stdout: create 107/35 file after the snapshot 2023-10-30T12:31:18.021 INFO:tasks.workunit.client.0.smithi125.stdout: create 107/36 file after the snapshot 2023-10-30T12:31:18.021 INFO:tasks.workunit.client.0.smithi125.stdout: create 107/37 file after the snapshot 2023-10-30T12:31:18.021 INFO:tasks.workunit.client.0.smithi125.stdout: create 107/38 file after the snapshot 2023-10-30T12:31:18.022 INFO:tasks.workunit.client.0.smithi125.stdout: create 107/39 file after the snapshot 2023-10-30T12:31:18.022 INFO:tasks.workunit.client.0.smithi125.stdout: create 108/.snap/snap-subdir-test 2023-10-30T12:31:18.094 INFO:tasks.workunit.client.0.smithi125.stdout: create 108/30 file after the snapshot 2023-10-30T12:31:18.096 INFO:tasks.workunit.client.0.smithi125.stdout: create 108/31 file after the snapshot 2023-10-30T12:31:18.097 INFO:tasks.workunit.client.0.smithi125.stdout: create 108/32 file after the snapshot 2023-10-30T12:31:18.097 INFO:tasks.workunit.client.0.smithi125.stdout: create 108/33 file after the snapshot 2023-10-30T12:31:18.097 INFO:tasks.workunit.client.0.smithi125.stdout: create 108/34 file after the snapshot 2023-10-30T12:31:18.097 INFO:tasks.workunit.client.0.smithi125.stdout: create 108/35 file after the snapshot 2023-10-30T12:31:18.098 INFO:tasks.workunit.client.0.smithi125.stdout: create 108/36 file after the snapshot 2023-10-30T12:31:18.098 INFO:tasks.workunit.client.0.smithi125.stdout: create 108/37 file after the snapshot 2023-10-30T12:31:18.098 INFO:tasks.workunit.client.0.smithi125.stdout: create 108/38 file after the snapshot 2023-10-30T12:31:18.098 INFO:tasks.workunit.client.0.smithi125.stdout: create 108/39 file after the snapshot 2023-10-30T12:31:18.098 INFO:tasks.workunit.client.0.smithi125.stdout: create 109/.snap/snap-subdir-test 2023-10-30T12:31:18.133 INFO:tasks.workunit.client.0.smithi125.stdout: create 109/30 file after the snapshot 2023-10-30T12:31:18.135 INFO:tasks.workunit.client.0.smithi125.stdout: create 109/31 file after the snapshot 2023-10-30T12:31:18.135 INFO:tasks.workunit.client.0.smithi125.stdout: create 109/32 file after the snapshot 2023-10-30T12:31:18.135 INFO:tasks.workunit.client.0.smithi125.stdout: create 109/33 file after the snapshot 2023-10-30T12:31:18.136 INFO:tasks.workunit.client.0.smithi125.stdout: create 109/34 file after the snapshot 2023-10-30T12:31:18.136 INFO:tasks.workunit.client.0.smithi125.stdout: create 109/35 file after the snapshot 2023-10-30T12:31:18.136 INFO:tasks.workunit.client.0.smithi125.stdout: create 109/36 file after the snapshot 2023-10-30T12:31:18.136 INFO:tasks.workunit.client.0.smithi125.stdout: create 109/37 file after the snapshot 2023-10-30T12:31:18.136 INFO:tasks.workunit.client.0.smithi125.stdout: create 109/38 file after the snapshot 2023-10-30T12:31:18.137 INFO:tasks.workunit.client.0.smithi125.stdout: create 109/39 file after the snapshot 2023-10-30T12:31:18.137 INFO:tasks.workunit.client.0.smithi125.stdout: create 110/.snap/snap-subdir-test 2023-10-30T12:31:18.178 INFO:tasks.workunit.client.0.smithi125.stdout: create 110/30 file after the snapshot 2023-10-30T12:31:18.180 INFO:tasks.workunit.client.0.smithi125.stdout: create 110/31 file after the snapshot 2023-10-30T12:31:18.180 INFO:tasks.workunit.client.0.smithi125.stdout: create 110/32 file after the snapshot 2023-10-30T12:31:18.181 INFO:tasks.workunit.client.0.smithi125.stdout: create 110/33 file after the snapshot 2023-10-30T12:31:18.181 INFO:tasks.workunit.client.0.smithi125.stdout: create 110/34 file after the snapshot 2023-10-30T12:31:18.181 INFO:tasks.workunit.client.0.smithi125.stdout: create 110/35 file after the snapshot 2023-10-30T12:31:18.181 INFO:tasks.workunit.client.0.smithi125.stdout: create 110/36 file after the snapshot 2023-10-30T12:31:18.181 INFO:tasks.workunit.client.0.smithi125.stdout: create 110/37 file after the snapshot 2023-10-30T12:31:18.182 INFO:tasks.workunit.client.0.smithi125.stdout: create 110/38 file after the snapshot 2023-10-30T12:31:18.182 INFO:tasks.workunit.client.0.smithi125.stdout: create 110/39 file after the snapshot 2023-10-30T12:31:18.182 INFO:tasks.workunit.client.0.smithi125.stdout: create 111/.snap/snap-subdir-test 2023-10-30T12:31:18.220 INFO:tasks.workunit.client.0.smithi125.stdout: create 111/30 file after the snapshot 2023-10-30T12:31:18.223 INFO:tasks.workunit.client.0.smithi125.stdout: create 111/31 file after the snapshot 2023-10-30T12:31:18.223 INFO:tasks.workunit.client.0.smithi125.stdout: create 111/32 file after the snapshot 2023-10-30T12:31:18.224 INFO:tasks.workunit.client.0.smithi125.stdout: create 111/33 file after the snapshot 2023-10-30T12:31:18.224 INFO:tasks.workunit.client.0.smithi125.stdout: create 111/34 file after the snapshot 2023-10-30T12:31:18.224 INFO:tasks.workunit.client.0.smithi125.stdout: create 111/35 file after the snapshot 2023-10-30T12:31:18.224 INFO:tasks.workunit.client.0.smithi125.stdout: create 111/36 file after the snapshot 2023-10-30T12:31:18.224 INFO:tasks.workunit.client.0.smithi125.stdout: create 111/37 file after the snapshot 2023-10-30T12:31:18.225 INFO:tasks.workunit.client.0.smithi125.stdout: create 111/38 file after the snapshot 2023-10-30T12:31:18.225 INFO:tasks.workunit.client.0.smithi125.stdout: create 111/39 file after the snapshot 2023-10-30T12:31:18.225 INFO:tasks.workunit.client.0.smithi125.stdout: create 112/.snap/snap-subdir-test 2023-10-30T12:31:18.263 INFO:tasks.workunit.client.0.smithi125.stdout: create 112/30 file after the snapshot 2023-10-30T12:31:18.265 INFO:tasks.workunit.client.0.smithi125.stdout: create 112/31 file after the snapshot 2023-10-30T12:31:18.265 INFO:tasks.workunit.client.0.smithi125.stdout: create 112/32 file after the snapshot 2023-10-30T12:31:18.266 INFO:tasks.workunit.client.0.smithi125.stdout: create 112/33 file after the snapshot 2023-10-30T12:31:18.266 INFO:tasks.workunit.client.0.smithi125.stdout: create 112/34 file after the snapshot 2023-10-30T12:31:18.266 INFO:tasks.workunit.client.0.smithi125.stdout: create 112/35 file after the snapshot 2023-10-30T12:31:18.266 INFO:tasks.workunit.client.0.smithi125.stdout: create 112/36 file after the snapshot 2023-10-30T12:31:18.266 INFO:tasks.workunit.client.0.smithi125.stdout: create 112/37 file after the snapshot 2023-10-30T12:31:18.267 INFO:tasks.workunit.client.0.smithi125.stdout: create 112/38 file after the snapshot 2023-10-30T12:31:18.267 INFO:tasks.workunit.client.0.smithi125.stdout: create 112/39 file after the snapshot 2023-10-30T12:31:18.267 INFO:tasks.workunit.client.0.smithi125.stdout: create 113/.snap/snap-subdir-test 2023-10-30T12:31:18.293 INFO:tasks.workunit.client.0.smithi125.stdout: create 113/30 file after the snapshot 2023-10-30T12:31:18.296 INFO:tasks.workunit.client.0.smithi125.stdout: create 113/31 file after the snapshot 2023-10-30T12:31:18.296 INFO:tasks.workunit.client.0.smithi125.stdout: create 113/32 file after the snapshot 2023-10-30T12:31:18.296 INFO:tasks.workunit.client.0.smithi125.stdout: create 113/33 file after the snapshot 2023-10-30T12:31:18.297 INFO:tasks.workunit.client.0.smithi125.stdout: create 113/34 file after the snapshot 2023-10-30T12:31:18.297 INFO:tasks.workunit.client.0.smithi125.stdout: create 113/35 file after the snapshot 2023-10-30T12:31:18.297 INFO:tasks.workunit.client.0.smithi125.stdout: create 113/36 file after the snapshot 2023-10-30T12:31:18.297 INFO:tasks.workunit.client.0.smithi125.stdout: create 113/37 file after the snapshot 2023-10-30T12:31:18.297 INFO:tasks.workunit.client.0.smithi125.stdout: create 113/38 file after the snapshot 2023-10-30T12:31:18.298 INFO:tasks.workunit.client.0.smithi125.stdout: create 113/39 file after the snapshot 2023-10-30T12:31:18.298 INFO:tasks.workunit.client.0.smithi125.stdout: create 114/.snap/snap-subdir-test 2023-10-30T12:31:18.323 INFO:tasks.workunit.client.0.smithi125.stdout: create 114/30 file after the snapshot 2023-10-30T12:31:18.324 INFO:tasks.workunit.client.0.smithi125.stdout: create 114/31 file after the snapshot 2023-10-30T12:31:18.324 INFO:tasks.workunit.client.0.smithi125.stdout: create 114/32 file after the snapshot 2023-10-30T12:31:18.325 INFO:tasks.workunit.client.0.smithi125.stdout: create 114/33 file after the snapshot 2023-10-30T12:31:18.325 INFO:tasks.workunit.client.0.smithi125.stdout: create 114/34 file after the snapshot 2023-10-30T12:31:18.325 INFO:tasks.workunit.client.0.smithi125.stdout: create 114/35 file after the snapshot 2023-10-30T12:31:18.326 INFO:tasks.workunit.client.0.smithi125.stdout: create 114/36 file after the snapshot 2023-10-30T12:31:18.326 INFO:tasks.workunit.client.0.smithi125.stdout: create 114/37 file after the snapshot 2023-10-30T12:31:18.326 INFO:tasks.workunit.client.0.smithi125.stdout: create 114/38 file after the snapshot 2023-10-30T12:31:18.326 INFO:tasks.workunit.client.0.smithi125.stdout: create 114/39 file after the snapshot 2023-10-30T12:31:18.327 INFO:tasks.workunit.client.0.smithi125.stdout: create 115/.snap/snap-subdir-test 2023-10-30T12:31:18.354 INFO:tasks.workunit.client.0.smithi125.stdout: create 115/30 file after the snapshot 2023-10-30T12:31:18.607 INFO:tasks.workunit.client.0.smithi125.stdout: create 115/31 file after the snapshot 2023-10-30T12:31:18.607 INFO:tasks.workunit.client.0.smithi125.stdout: create 115/32 file after the snapshot 2023-10-30T12:31:18.607 INFO:tasks.workunit.client.0.smithi125.stdout: create 115/33 file after the snapshot 2023-10-30T12:31:18.608 INFO:tasks.workunit.client.0.smithi125.stdout: create 115/34 file after the snapshot 2023-10-30T12:31:18.608 INFO:tasks.workunit.client.0.smithi125.stdout: create 115/35 file after the snapshot 2023-10-30T12:31:18.608 INFO:tasks.workunit.client.0.smithi125.stdout: create 115/36 file after the snapshot 2023-10-30T12:31:18.608 INFO:tasks.workunit.client.0.smithi125.stdout: create 115/37 file after the snapshot 2023-10-30T12:31:18.608 INFO:tasks.workunit.client.0.smithi125.stdout: create 115/38 file after the snapshot 2023-10-30T12:31:18.609 INFO:tasks.workunit.client.0.smithi125.stdout: create 115/39 file after the snapshot 2023-10-30T12:31:18.609 INFO:tasks.workunit.client.0.smithi125.stdout: create 116/.snap/snap-subdir-test 2023-10-30T12:31:18.885 INFO:tasks.workunit.client.0.smithi125.stdout: create 116/30 file after the snapshot 2023-10-30T12:31:18.887 INFO:tasks.workunit.client.0.smithi125.stdout: create 116/31 file after the snapshot 2023-10-30T12:31:18.887 INFO:tasks.workunit.client.0.smithi125.stdout: create 116/32 file after the snapshot 2023-10-30T12:31:18.887 INFO:tasks.workunit.client.0.smithi125.stdout: create 116/33 file after the snapshot 2023-10-30T12:31:18.888 INFO:tasks.workunit.client.0.smithi125.stdout: create 116/34 file after the snapshot 2023-10-30T12:31:18.888 INFO:tasks.workunit.client.0.smithi125.stdout: create 116/35 file after the snapshot 2023-10-30T12:31:18.889 INFO:tasks.workunit.client.0.smithi125.stdout: create 116/36 file after the snapshot 2023-10-30T12:31:18.889 INFO:tasks.workunit.client.0.smithi125.stdout: create 116/37 file after the snapshot 2023-10-30T12:31:18.889 INFO:tasks.workunit.client.0.smithi125.stdout: create 116/38 file after the snapshot 2023-10-30T12:31:18.889 INFO:tasks.workunit.client.0.smithi125.stdout: create 116/39 file after the snapshot 2023-10-30T12:31:18.890 INFO:tasks.workunit.client.0.smithi125.stdout: create 117/.snap/snap-subdir-test 2023-10-30T12:31:18.917 INFO:tasks.workunit.client.0.smithi125.stdout: create 117/30 file after the snapshot 2023-10-30T12:31:18.919 INFO:tasks.workunit.client.0.smithi125.stdout: create 117/31 file after the snapshot 2023-10-30T12:31:18.919 INFO:tasks.workunit.client.0.smithi125.stdout: create 117/32 file after the snapshot 2023-10-30T12:31:18.919 INFO:tasks.workunit.client.0.smithi125.stdout: create 117/33 file after the snapshot 2023-10-30T12:31:18.920 INFO:tasks.workunit.client.0.smithi125.stdout: create 117/34 file after the snapshot 2023-10-30T12:31:18.920 INFO:tasks.workunit.client.0.smithi125.stdout: create 117/35 file after the snapshot 2023-10-30T12:31:18.920 INFO:tasks.workunit.client.0.smithi125.stdout: create 117/36 file after the snapshot 2023-10-30T12:31:18.920 INFO:tasks.workunit.client.0.smithi125.stdout: create 117/37 file after the snapshot 2023-10-30T12:31:18.920 INFO:tasks.workunit.client.0.smithi125.stdout: create 117/38 file after the snapshot 2023-10-30T12:31:18.920 INFO:tasks.workunit.client.0.smithi125.stdout: create 117/39 file after the snapshot 2023-10-30T12:31:18.921 INFO:tasks.workunit.client.0.smithi125.stdout: create 118/.snap/snap-subdir-test 2023-10-30T12:31:19.328 INFO:tasks.workunit.client.0.smithi125.stdout: create 118/30 file after the snapshot 2023-10-30T12:31:19.330 INFO:tasks.workunit.client.0.smithi125.stdout: create 118/31 file after the snapshot 2023-10-30T12:31:19.330 INFO:tasks.workunit.client.0.smithi125.stdout: create 118/32 file after the snapshot 2023-10-30T12:31:19.331 INFO:tasks.workunit.client.0.smithi125.stdout: create 118/33 file after the snapshot 2023-10-30T12:31:19.331 INFO:tasks.workunit.client.0.smithi125.stdout: create 118/34 file after the snapshot 2023-10-30T12:31:19.331 INFO:tasks.workunit.client.0.smithi125.stdout: create 118/35 file after the snapshot 2023-10-30T12:31:19.332 INFO:tasks.workunit.client.0.smithi125.stdout: create 118/36 file after the snapshot 2023-10-30T12:31:19.332 INFO:tasks.workunit.client.0.smithi125.stdout: create 118/37 file after the snapshot 2023-10-30T12:31:19.332 INFO:tasks.workunit.client.0.smithi125.stdout: create 118/38 file after the snapshot 2023-10-30T12:31:19.332 INFO:tasks.workunit.client.0.smithi125.stdout: create 118/39 file after the snapshot 2023-10-30T12:31:19.332 INFO:tasks.workunit.client.0.smithi125.stdout: create 119/.snap/snap-subdir-test 2023-10-30T12:31:19.361 INFO:tasks.workunit.client.0.smithi125.stdout: create 119/30 file after the snapshot 2023-10-30T12:31:19.363 INFO:tasks.workunit.client.0.smithi125.stdout: create 119/31 file after the snapshot 2023-10-30T12:31:19.364 INFO:tasks.workunit.client.0.smithi125.stdout: create 119/32 file after the snapshot 2023-10-30T12:31:19.364 INFO:tasks.workunit.client.0.smithi125.stdout: create 119/33 file after the snapshot 2023-10-30T12:31:19.364 INFO:tasks.workunit.client.0.smithi125.stdout: create 119/34 file after the snapshot 2023-10-30T12:31:19.364 INFO:tasks.workunit.client.0.smithi125.stdout: create 119/35 file after the snapshot 2023-10-30T12:31:19.365 INFO:tasks.workunit.client.0.smithi125.stdout: create 119/36 file after the snapshot 2023-10-30T12:31:19.365 INFO:tasks.workunit.client.0.smithi125.stdout: create 119/37 file after the snapshot 2023-10-30T12:31:19.365 INFO:tasks.workunit.client.0.smithi125.stdout: create 119/38 file after the snapshot 2023-10-30T12:31:19.365 INFO:tasks.workunit.client.0.smithi125.stdout: create 119/39 file after the snapshot 2023-10-30T12:31:19.366 INFO:tasks.workunit.client.0.smithi125.stdout: create 120/.snap/snap-subdir-test 2023-10-30T12:31:20.129 INFO:tasks.workunit.client.0.smithi125.stdout: create 120/30 file after the snapshot 2023-10-30T12:31:20.131 INFO:tasks.workunit.client.0.smithi125.stdout: create 120/31 file after the snapshot 2023-10-30T12:31:20.131 INFO:tasks.workunit.client.0.smithi125.stdout: create 120/32 file after the snapshot 2023-10-30T12:31:20.131 INFO:tasks.workunit.client.0.smithi125.stdout: create 120/33 file after the snapshot 2023-10-30T12:31:20.131 INFO:tasks.workunit.client.0.smithi125.stdout: create 120/34 file after the snapshot 2023-10-30T12:31:20.132 INFO:tasks.workunit.client.0.smithi125.stdout: create 120/35 file after the snapshot 2023-10-30T12:31:20.132 INFO:tasks.workunit.client.0.smithi125.stdout: create 120/36 file after the snapshot 2023-10-30T12:31:20.133 INFO:tasks.workunit.client.0.smithi125.stdout: create 120/37 file after the snapshot 2023-10-30T12:31:20.133 INFO:tasks.workunit.client.0.smithi125.stdout: create 120/38 file after the snapshot 2023-10-30T12:31:20.133 INFO:tasks.workunit.client.0.smithi125.stdout: create 120/39 file after the snapshot 2023-10-30T12:31:20.133 INFO:tasks.workunit.client.0.smithi125.stdout: create 121/.snap/snap-subdir-test 2023-10-30T12:31:20.163 INFO:tasks.workunit.client.0.smithi125.stdout: create 121/30 file after the snapshot 2023-10-30T12:31:20.164 INFO:tasks.workunit.client.0.smithi125.stdout: create 121/31 file after the snapshot 2023-10-30T12:31:20.164 INFO:tasks.workunit.client.0.smithi125.stdout: create 121/32 file after the snapshot 2023-10-30T12:31:20.165 INFO:tasks.workunit.client.0.smithi125.stdout: create 121/33 file after the snapshot 2023-10-30T12:31:20.166 INFO:tasks.workunit.client.0.smithi125.stdout: create 121/34 file after the snapshot 2023-10-30T12:31:20.166 INFO:tasks.workunit.client.0.smithi125.stdout: create 121/35 file after the snapshot 2023-10-30T12:31:20.166 INFO:tasks.workunit.client.0.smithi125.stdout: create 121/36 file after the snapshot 2023-10-30T12:31:20.166 INFO:tasks.workunit.client.0.smithi125.stdout: create 121/37 file after the snapshot 2023-10-30T12:31:20.167 INFO:tasks.workunit.client.0.smithi125.stdout: create 121/38 file after the snapshot 2023-10-30T12:31:20.167 INFO:tasks.workunit.client.0.smithi125.stdout: create 121/39 file after the snapshot 2023-10-30T12:31:20.167 INFO:tasks.workunit.client.0.smithi125.stdout: create 122/.snap/snap-subdir-test 2023-10-30T12:31:20.204 INFO:tasks.workunit.client.0.smithi125.stdout: create 122/30 file after the snapshot 2023-10-30T12:31:20.206 INFO:tasks.workunit.client.0.smithi125.stdout: create 122/31 file after the snapshot 2023-10-30T12:31:20.206 INFO:tasks.workunit.client.0.smithi125.stdout: create 122/32 file after the snapshot 2023-10-30T12:31:20.206 INFO:tasks.workunit.client.0.smithi125.stdout: create 122/33 file after the snapshot 2023-10-30T12:31:20.207 INFO:tasks.workunit.client.0.smithi125.stdout: create 122/34 file after the snapshot 2023-10-30T12:31:20.207 INFO:tasks.workunit.client.0.smithi125.stdout: create 122/35 file after the snapshot 2023-10-30T12:31:20.207 INFO:tasks.workunit.client.0.smithi125.stdout: create 122/36 file after the snapshot 2023-10-30T12:31:20.207 INFO:tasks.workunit.client.0.smithi125.stdout: create 122/37 file after the snapshot 2023-10-30T12:31:20.207 INFO:tasks.workunit.client.0.smithi125.stdout: create 122/38 file after the snapshot 2023-10-30T12:31:20.208 INFO:tasks.workunit.client.0.smithi125.stdout: create 122/39 file after the snapshot 2023-10-30T12:31:20.208 INFO:tasks.workunit.client.0.smithi125.stdout: create 123/.snap/snap-subdir-test 2023-10-30T12:31:20.243 INFO:tasks.workunit.client.0.smithi125.stdout: create 123/30 file after the snapshot 2023-10-30T12:31:20.246 INFO:tasks.workunit.client.0.smithi125.stdout: create 123/31 file after the snapshot 2023-10-30T12:31:20.246 INFO:tasks.workunit.client.0.smithi125.stdout: create 123/32 file after the snapshot 2023-10-30T12:31:20.246 INFO:tasks.workunit.client.0.smithi125.stdout: create 123/33 file after the snapshot 2023-10-30T12:31:20.246 INFO:tasks.workunit.client.0.smithi125.stdout: create 123/34 file after the snapshot 2023-10-30T12:31:20.247 INFO:tasks.workunit.client.0.smithi125.stdout: create 123/35 file after the snapshot 2023-10-30T12:31:20.247 INFO:tasks.workunit.client.0.smithi125.stdout: create 123/36 file after the snapshot 2023-10-30T12:31:20.247 INFO:tasks.workunit.client.0.smithi125.stdout: create 123/37 file after the snapshot 2023-10-30T12:31:20.247 INFO:tasks.workunit.client.0.smithi125.stdout: create 123/38 file after the snapshot 2023-10-30T12:31:20.247 INFO:tasks.workunit.client.0.smithi125.stdout: create 123/39 file after the snapshot 2023-10-30T12:31:20.248 INFO:tasks.workunit.client.0.smithi125.stdout: create 124/.snap/snap-subdir-test 2023-10-30T12:31:20.282 INFO:tasks.workunit.client.0.smithi125.stdout: create 124/30 file after the snapshot 2023-10-30T12:31:20.284 INFO:tasks.workunit.client.0.smithi125.stdout: create 124/31 file after the snapshot 2023-10-30T12:31:20.284 INFO:tasks.workunit.client.0.smithi125.stdout: create 124/32 file after the snapshot 2023-10-30T12:31:20.284 INFO:tasks.workunit.client.0.smithi125.stdout: create 124/33 file after the snapshot 2023-10-30T12:31:20.284 INFO:tasks.workunit.client.0.smithi125.stdout: create 124/34 file after the snapshot 2023-10-30T12:31:20.284 INFO:tasks.workunit.client.0.smithi125.stdout: create 124/35 file after the snapshot 2023-10-30T12:31:20.285 INFO:tasks.workunit.client.0.smithi125.stdout: create 124/36 file after the snapshot 2023-10-30T12:31:20.285 INFO:tasks.workunit.client.0.smithi125.stdout: create 124/37 file after the snapshot 2023-10-30T12:31:20.285 INFO:tasks.workunit.client.0.smithi125.stdout: create 124/38 file after the snapshot 2023-10-30T12:31:20.285 INFO:tasks.workunit.client.0.smithi125.stdout: create 124/39 file after the snapshot 2023-10-30T12:31:20.285 INFO:tasks.workunit.client.0.smithi125.stdout: create 125/.snap/snap-subdir-test 2023-10-30T12:31:20.311 INFO:tasks.workunit.client.0.smithi125.stdout: create 125/30 file after the snapshot 2023-10-30T12:31:20.313 INFO:tasks.workunit.client.0.smithi125.stdout: create 125/31 file after the snapshot 2023-10-30T12:31:20.313 INFO:tasks.workunit.client.0.smithi125.stdout: create 125/32 file after the snapshot 2023-10-30T12:31:20.313 INFO:tasks.workunit.client.0.smithi125.stdout: create 125/33 file after the snapshot 2023-10-30T12:31:20.313 INFO:tasks.workunit.client.0.smithi125.stdout: create 125/34 file after the snapshot 2023-10-30T12:31:20.314 INFO:tasks.workunit.client.0.smithi125.stdout: create 125/35 file after the snapshot 2023-10-30T12:31:20.314 INFO:tasks.workunit.client.0.smithi125.stdout: create 125/36 file after the snapshot 2023-10-30T12:31:20.314 INFO:tasks.workunit.client.0.smithi125.stdout: create 125/37 file after the snapshot 2023-10-30T12:31:20.314 INFO:tasks.workunit.client.0.smithi125.stdout: create 125/38 file after the snapshot 2023-10-30T12:31:20.314 INFO:tasks.workunit.client.0.smithi125.stdout: create 125/39 file after the snapshot 2023-10-30T12:31:20.315 INFO:tasks.workunit.client.0.smithi125.stdout: create 126/.snap/snap-subdir-test 2023-10-30T12:31:20.340 INFO:tasks.workunit.client.0.smithi125.stdout: create 126/30 file after the snapshot 2023-10-30T12:31:20.342 INFO:tasks.workunit.client.0.smithi125.stdout: create 126/31 file after the snapshot 2023-10-30T12:31:20.342 INFO:tasks.workunit.client.0.smithi125.stdout: create 126/32 file after the snapshot 2023-10-30T12:31:20.342 INFO:tasks.workunit.client.0.smithi125.stdout: create 126/33 file after the snapshot 2023-10-30T12:31:20.342 INFO:tasks.workunit.client.0.smithi125.stdout: create 126/34 file after the snapshot 2023-10-30T12:31:20.343 INFO:tasks.workunit.client.0.smithi125.stdout: create 126/35 file after the snapshot 2023-10-30T12:31:20.343 INFO:tasks.workunit.client.0.smithi125.stdout: create 126/36 file after the snapshot 2023-10-30T12:31:20.343 INFO:tasks.workunit.client.0.smithi125.stdout: create 126/37 file after the snapshot 2023-10-30T12:31:20.343 INFO:tasks.workunit.client.0.smithi125.stdout: create 126/38 file after the snapshot 2023-10-30T12:31:20.343 INFO:tasks.workunit.client.0.smithi125.stdout: create 126/39 file after the snapshot 2023-10-30T12:31:20.344 INFO:tasks.workunit.client.0.smithi125.stdout: create 127/.snap/snap-subdir-test 2023-10-30T12:31:20.369 INFO:tasks.workunit.client.0.smithi125.stdout: create 127/30 file after the snapshot 2023-10-30T12:31:20.734 INFO:tasks.workunit.client.0.smithi125.stdout: create 127/31 file after the snapshot 2023-10-30T12:31:20.735 INFO:tasks.workunit.client.0.smithi125.stdout: create 127/32 file after the snapshot 2023-10-30T12:31:20.735 INFO:tasks.workunit.client.0.smithi125.stdout: create 127/33 file after the snapshot 2023-10-30T12:31:20.735 INFO:tasks.workunit.client.0.smithi125.stdout: create 127/34 file after the snapshot 2023-10-30T12:31:20.735 INFO:tasks.workunit.client.0.smithi125.stdout: create 127/35 file after the snapshot 2023-10-30T12:31:20.736 INFO:tasks.workunit.client.0.smithi125.stdout: create 127/36 file after the snapshot 2023-10-30T12:31:20.736 INFO:tasks.workunit.client.0.smithi125.stdout: create 127/37 file after the snapshot 2023-10-30T12:31:20.736 INFO:tasks.workunit.client.0.smithi125.stdout: create 127/38 file after the snapshot 2023-10-30T12:31:20.736 INFO:tasks.workunit.client.0.smithi125.stdout: create 127/39 file after the snapshot 2023-10-30T12:31:20.736 INFO:tasks.workunit.client.0.smithi125.stdout: create 128/.snap/snap-subdir-test 2023-10-30T12:31:20.774 INFO:tasks.workunit.client.0.smithi125.stdout: create 128/30 file after the snapshot 2023-10-30T12:31:20.776 INFO:tasks.workunit.client.0.smithi125.stdout: create 128/31 file after the snapshot 2023-10-30T12:31:20.776 INFO:tasks.workunit.client.0.smithi125.stdout: create 128/32 file after the snapshot 2023-10-30T12:31:20.776 INFO:tasks.workunit.client.0.smithi125.stdout: create 128/33 file after the snapshot 2023-10-30T12:31:20.777 INFO:tasks.workunit.client.0.smithi125.stdout: create 128/34 file after the snapshot 2023-10-30T12:31:20.777 INFO:tasks.workunit.client.0.smithi125.stdout: create 128/35 file after the snapshot 2023-10-30T12:31:20.777 INFO:tasks.workunit.client.0.smithi125.stdout: create 128/36 file after the snapshot 2023-10-30T12:31:20.777 INFO:tasks.workunit.client.0.smithi125.stdout: create 128/37 file after the snapshot 2023-10-30T12:31:20.777 INFO:tasks.workunit.client.0.smithi125.stdout: create 128/38 file after the snapshot 2023-10-30T12:31:20.778 INFO:tasks.workunit.client.0.smithi125.stdout: create 128/39 file after the snapshot 2023-10-30T12:31:20.778 INFO:tasks.workunit.client.0.smithi125.stdout: create 129/.snap/snap-subdir-test 2023-10-30T12:31:20.804 INFO:tasks.workunit.client.0.smithi125.stdout: create 129/30 file after the snapshot 2023-10-30T12:31:20.805 INFO:tasks.workunit.client.0.smithi125.stdout: create 129/31 file after the snapshot 2023-10-30T12:31:20.806 INFO:tasks.workunit.client.0.smithi125.stdout: create 129/32 file after the snapshot 2023-10-30T12:31:20.806 INFO:tasks.workunit.client.0.smithi125.stdout: create 129/33 file after the snapshot 2023-10-30T12:31:20.806 INFO:tasks.workunit.client.0.smithi125.stdout: create 129/34 file after the snapshot 2023-10-30T12:31:20.806 INFO:tasks.workunit.client.0.smithi125.stdout: create 129/35 file after the snapshot 2023-10-30T12:31:20.806 INFO:tasks.workunit.client.0.smithi125.stdout: create 129/36 file after the snapshot 2023-10-30T12:31:20.807 INFO:tasks.workunit.client.0.smithi125.stdout: create 129/37 file after the snapshot 2023-10-30T12:31:20.807 INFO:tasks.workunit.client.0.smithi125.stdout: create 129/38 file after the snapshot 2023-10-30T12:31:20.807 INFO:tasks.workunit.client.0.smithi125.stdout: create 129/39 file after the snapshot 2023-10-30T12:31:20.807 INFO:tasks.workunit.client.0.smithi125.stdout: create 130/.snap/snap-subdir-test 2023-10-30T12:31:20.835 INFO:tasks.workunit.client.0.smithi125.stdout: create 130/30 file after the snapshot 2023-10-30T12:31:20.836 INFO:tasks.workunit.client.0.smithi125.stdout: create 130/31 file after the snapshot 2023-10-30T12:31:20.837 INFO:tasks.workunit.client.0.smithi125.stdout: create 130/32 file after the snapshot 2023-10-30T12:31:20.837 INFO:tasks.workunit.client.0.smithi125.stdout: create 130/33 file after the snapshot 2023-10-30T12:31:20.837 INFO:tasks.workunit.client.0.smithi125.stdout: create 130/34 file after the snapshot 2023-10-30T12:31:20.837 INFO:tasks.workunit.client.0.smithi125.stdout: create 130/35 file after the snapshot 2023-10-30T12:31:20.837 INFO:tasks.workunit.client.0.smithi125.stdout: create 130/36 file after the snapshot 2023-10-30T12:31:20.837 INFO:tasks.workunit.client.0.smithi125.stdout: create 130/37 file after the snapshot 2023-10-30T12:31:20.838 INFO:tasks.workunit.client.0.smithi125.stdout: create 130/38 file after the snapshot 2023-10-30T12:31:20.838 INFO:tasks.workunit.client.0.smithi125.stdout: create 130/39 file after the snapshot 2023-10-30T12:31:20.838 INFO:tasks.workunit.client.0.smithi125.stdout: create 131/.snap/snap-subdir-test 2023-10-30T12:31:21.733 INFO:tasks.workunit.client.0.smithi125.stdout: create 131/30 file after the snapshot 2023-10-30T12:31:21.735 INFO:tasks.workunit.client.0.smithi125.stdout: create 131/31 file after the snapshot 2023-10-30T12:31:21.735 INFO:tasks.workunit.client.0.smithi125.stdout: create 131/32 file after the snapshot 2023-10-30T12:31:21.735 INFO:tasks.workunit.client.0.smithi125.stdout: create 131/33 file after the snapshot 2023-10-30T12:31:21.735 INFO:tasks.workunit.client.0.smithi125.stdout: create 131/34 file after the snapshot 2023-10-30T12:31:21.736 INFO:tasks.workunit.client.0.smithi125.stdout: create 131/35 file after the snapshot 2023-10-30T12:31:21.736 INFO:tasks.workunit.client.0.smithi125.stdout: create 131/36 file after the snapshot 2023-10-30T12:31:21.737 INFO:tasks.workunit.client.0.smithi125.stdout: create 131/37 file after the snapshot 2023-10-30T12:31:21.737 INFO:tasks.workunit.client.0.smithi125.stdout: create 131/38 file after the snapshot 2023-10-30T12:31:21.737 INFO:tasks.workunit.client.0.smithi125.stdout: create 131/39 file after the snapshot 2023-10-30T12:31:21.737 INFO:tasks.workunit.client.0.smithi125.stdout: create 132/.snap/snap-subdir-test 2023-10-30T12:31:21.951 INFO:tasks.workunit.client.0.smithi125.stdout: create 132/30 file after the snapshot 2023-10-30T12:31:21.952 INFO:tasks.workunit.client.0.smithi125.stdout: create 132/31 file after the snapshot 2023-10-30T12:31:21.953 INFO:tasks.workunit.client.0.smithi125.stdout: create 132/32 file after the snapshot 2023-10-30T12:31:21.953 INFO:tasks.workunit.client.0.smithi125.stdout: create 132/33 file after the snapshot 2023-10-30T12:31:21.953 INFO:tasks.workunit.client.0.smithi125.stdout: create 132/34 file after the snapshot 2023-10-30T12:31:21.954 INFO:tasks.workunit.client.0.smithi125.stdout: create 132/35 file after the snapshot 2023-10-30T12:31:21.954 INFO:tasks.workunit.client.0.smithi125.stdout: create 132/36 file after the snapshot 2023-10-30T12:31:21.954 INFO:tasks.workunit.client.0.smithi125.stdout: create 132/37 file after the snapshot 2023-10-30T12:31:21.955 INFO:tasks.workunit.client.0.smithi125.stdout: create 132/38 file after the snapshot 2023-10-30T12:31:21.955 INFO:tasks.workunit.client.0.smithi125.stdout: create 132/39 file after the snapshot 2023-10-30T12:31:21.955 INFO:tasks.workunit.client.0.smithi125.stdout: create 133/.snap/snap-subdir-test 2023-10-30T12:31:22.810 INFO:tasks.workunit.client.0.smithi125.stdout: create 133/30 file after the snapshot 2023-10-30T12:31:22.812 INFO:tasks.workunit.client.0.smithi125.stdout: create 133/31 file after the snapshot 2023-10-30T12:31:22.812 INFO:tasks.workunit.client.0.smithi125.stdout: create 133/32 file after the snapshot 2023-10-30T12:31:22.812 INFO:tasks.workunit.client.0.smithi125.stdout: create 133/33 file after the snapshot 2023-10-30T12:31:22.813 INFO:tasks.workunit.client.0.smithi125.stdout: create 133/34 file after the snapshot 2023-10-30T12:31:22.813 INFO:tasks.workunit.client.0.smithi125.stdout: create 133/35 file after the snapshot 2023-10-30T12:31:22.814 INFO:tasks.workunit.client.0.smithi125.stdout: create 133/36 file after the snapshot 2023-10-30T12:31:22.814 INFO:tasks.workunit.client.0.smithi125.stdout: create 133/37 file after the snapshot 2023-10-30T12:31:22.814 INFO:tasks.workunit.client.0.smithi125.stdout: create 133/38 file after the snapshot 2023-10-30T12:31:22.814 INFO:tasks.workunit.client.0.smithi125.stdout: create 133/39 file after the snapshot 2023-10-30T12:31:22.815 INFO:tasks.workunit.client.0.smithi125.stdout: create 134/.snap/snap-subdir-test 2023-10-30T12:31:22.845 INFO:tasks.workunit.client.0.smithi125.stdout: create 134/30 file after the snapshot 2023-10-30T12:31:22.846 INFO:tasks.workunit.client.0.smithi125.stdout: create 134/31 file after the snapshot 2023-10-30T12:31:22.847 INFO:tasks.workunit.client.0.smithi125.stdout: create 134/32 file after the snapshot 2023-10-30T12:31:22.847 INFO:tasks.workunit.client.0.smithi125.stdout: create 134/33 file after the snapshot 2023-10-30T12:31:22.848 INFO:tasks.workunit.client.0.smithi125.stdout: create 134/34 file after the snapshot 2023-10-30T12:31:22.848 INFO:tasks.workunit.client.0.smithi125.stdout: create 134/35 file after the snapshot 2023-10-30T12:31:22.848 INFO:tasks.workunit.client.0.smithi125.stdout: create 134/36 file after the snapshot 2023-10-30T12:31:22.849 INFO:tasks.workunit.client.0.smithi125.stdout: create 134/37 file after the snapshot 2023-10-30T12:31:22.849 INFO:tasks.workunit.client.0.smithi125.stdout: create 134/38 file after the snapshot 2023-10-30T12:31:22.849 INFO:tasks.workunit.client.0.smithi125.stdout: create 134/39 file after the snapshot 2023-10-30T12:31:22.849 INFO:tasks.workunit.client.0.smithi125.stdout: create 135/.snap/snap-subdir-test 2023-10-30T12:31:22.878 INFO:tasks.workunit.client.0.smithi125.stdout: create 135/30 file after the snapshot 2023-10-30T12:31:22.880 INFO:tasks.workunit.client.0.smithi125.stdout: create 135/31 file after the snapshot 2023-10-30T12:31:22.880 INFO:tasks.workunit.client.0.smithi125.stdout: create 135/32 file after the snapshot 2023-10-30T12:31:22.880 INFO:tasks.workunit.client.0.smithi125.stdout: create 135/33 file after the snapshot 2023-10-30T12:31:22.880 INFO:tasks.workunit.client.0.smithi125.stdout: create 135/34 file after the snapshot 2023-10-30T12:31:22.880 INFO:tasks.workunit.client.0.smithi125.stdout: create 135/35 file after the snapshot 2023-10-30T12:31:22.881 INFO:tasks.workunit.client.0.smithi125.stdout: create 135/36 file after the snapshot 2023-10-30T12:31:22.881 INFO:tasks.workunit.client.0.smithi125.stdout: create 135/37 file after the snapshot 2023-10-30T12:31:22.881 INFO:tasks.workunit.client.0.smithi125.stdout: create 135/38 file after the snapshot 2023-10-30T12:31:22.881 INFO:tasks.workunit.client.0.smithi125.stdout: create 135/39 file after the snapshot 2023-10-30T12:31:22.881 INFO:tasks.workunit.client.0.smithi125.stdout: create 136/.snap/snap-subdir-test 2023-10-30T12:31:22.908 INFO:tasks.workunit.client.0.smithi125.stdout: create 136/30 file after the snapshot 2023-10-30T12:31:22.911 INFO:tasks.workunit.client.0.smithi125.stdout: create 136/31 file after the snapshot 2023-10-30T12:31:22.912 INFO:tasks.workunit.client.0.smithi125.stdout: create 136/32 file after the snapshot 2023-10-30T12:31:22.912 INFO:tasks.workunit.client.0.smithi125.stdout: create 136/33 file after the snapshot 2023-10-30T12:31:22.912 INFO:tasks.workunit.client.0.smithi125.stdout: create 136/34 file after the snapshot 2023-10-30T12:31:22.913 INFO:tasks.workunit.client.0.smithi125.stdout: create 136/35 file after the snapshot 2023-10-30T12:31:22.913 INFO:tasks.workunit.client.0.smithi125.stdout: create 136/36 file after the snapshot 2023-10-30T12:31:22.914 INFO:tasks.workunit.client.0.smithi125.stdout: create 136/37 file after the snapshot 2023-10-30T12:31:22.914 INFO:tasks.workunit.client.0.smithi125.stdout: create 136/38 file after the snapshot 2023-10-30T12:31:22.914 INFO:tasks.workunit.client.0.smithi125.stdout: create 136/39 file after the snapshot 2023-10-30T12:31:22.915 INFO:tasks.workunit.client.0.smithi125.stdout: create 137/.snap/snap-subdir-test 2023-10-30T12:31:22.941 INFO:tasks.workunit.client.0.smithi125.stdout: create 137/30 file after the snapshot 2023-10-30T12:31:22.944 INFO:tasks.workunit.client.0.smithi125.stdout: create 137/31 file after the snapshot 2023-10-30T12:31:22.944 INFO:tasks.workunit.client.0.smithi125.stdout: create 137/32 file after the snapshot 2023-10-30T12:31:22.944 INFO:tasks.workunit.client.0.smithi125.stdout: create 137/33 file after the snapshot 2023-10-30T12:31:22.944 INFO:tasks.workunit.client.0.smithi125.stdout: create 137/34 file after the snapshot 2023-10-30T12:31:22.945 INFO:tasks.workunit.client.0.smithi125.stdout: create 137/35 file after the snapshot 2023-10-30T12:31:22.945 INFO:tasks.workunit.client.0.smithi125.stdout: create 137/36 file after the snapshot 2023-10-30T12:31:22.945 INFO:tasks.workunit.client.0.smithi125.stdout: create 137/37 file after the snapshot 2023-10-30T12:31:22.945 INFO:tasks.workunit.client.0.smithi125.stdout: create 137/38 file after the snapshot 2023-10-30T12:31:22.945 INFO:tasks.workunit.client.0.smithi125.stdout: create 137/39 file after the snapshot 2023-10-30T12:31:22.946 INFO:tasks.workunit.client.0.smithi125.stdout: create 138/.snap/snap-subdir-test 2023-10-30T12:31:23.697 INFO:tasks.workunit.client.0.smithi125.stdout: create 138/30 file after the snapshot 2023-10-30T12:31:23.951 INFO:tasks.workunit.client.0.smithi125.stdout: create 138/31 file after the snapshot 2023-10-30T12:31:23.951 INFO:tasks.workunit.client.0.smithi125.stdout: create 138/32 file after the snapshot 2023-10-30T12:31:23.951 INFO:tasks.workunit.client.0.smithi125.stdout: create 138/33 file after the snapshot 2023-10-30T12:31:23.952 INFO:tasks.workunit.client.0.smithi125.stdout: create 138/34 file after the snapshot 2023-10-30T12:31:23.952 INFO:tasks.workunit.client.0.smithi125.stdout: create 138/35 file after the snapshot 2023-10-30T12:31:23.952 INFO:tasks.workunit.client.0.smithi125.stdout: create 138/36 file after the snapshot 2023-10-30T12:31:23.952 INFO:tasks.workunit.client.0.smithi125.stdout: create 138/37 file after the snapshot 2023-10-30T12:31:23.952 INFO:tasks.workunit.client.0.smithi125.stdout: create 138/38 file after the snapshot 2023-10-30T12:31:23.953 INFO:tasks.workunit.client.0.smithi125.stdout: create 138/39 file after the snapshot 2023-10-30T12:31:23.953 INFO:tasks.workunit.client.0.smithi125.stdout: create 139/.snap/snap-subdir-test 2023-10-30T12:31:23.982 INFO:tasks.workunit.client.0.smithi125.stdout: create 139/30 file after the snapshot 2023-10-30T12:31:24.585 INFO:tasks.workunit.client.0.smithi125.stdout: create 139/31 file after the snapshot 2023-10-30T12:31:24.585 INFO:tasks.workunit.client.0.smithi125.stdout: create 139/32 file after the snapshot 2023-10-30T12:31:24.586 INFO:tasks.workunit.client.0.smithi125.stdout: create 139/33 file after the snapshot 2023-10-30T12:31:24.586 INFO:tasks.workunit.client.0.smithi125.stdout: create 139/34 file after the snapshot 2023-10-30T12:31:24.586 INFO:tasks.workunit.client.0.smithi125.stdout: create 139/35 file after the snapshot 2023-10-30T12:31:24.586 INFO:tasks.workunit.client.0.smithi125.stdout: create 139/36 file after the snapshot 2023-10-30T12:31:24.586 INFO:tasks.workunit.client.0.smithi125.stdout: create 139/37 file after the snapshot 2023-10-30T12:31:24.587 INFO:tasks.workunit.client.0.smithi125.stdout: create 139/38 file after the snapshot 2023-10-30T12:31:24.587 INFO:tasks.workunit.client.0.smithi125.stdout: create 139/39 file after the snapshot 2023-10-30T12:31:24.587 INFO:tasks.workunit.client.0.smithi125.stdout: create 140/.snap/snap-subdir-test 2023-10-30T12:31:24.617 INFO:tasks.workunit.client.0.smithi125.stdout: create 140/30 file after the snapshot 2023-10-30T12:31:24.618 INFO:tasks.workunit.client.0.smithi125.stdout: create 140/31 file after the snapshot 2023-10-30T12:31:24.619 INFO:tasks.workunit.client.0.smithi125.stdout: create 140/32 file after the snapshot 2023-10-30T12:31:24.619 INFO:tasks.workunit.client.0.smithi125.stdout: create 140/33 file after the snapshot 2023-10-30T12:31:24.619 INFO:tasks.workunit.client.0.smithi125.stdout: create 140/34 file after the snapshot 2023-10-30T12:31:24.619 INFO:tasks.workunit.client.0.smithi125.stdout: create 140/35 file after the snapshot 2023-10-30T12:31:24.620 INFO:tasks.workunit.client.0.smithi125.stdout: create 140/36 file after the snapshot 2023-10-30T12:31:24.620 INFO:tasks.workunit.client.0.smithi125.stdout: create 140/37 file after the snapshot 2023-10-30T12:31:24.620 INFO:tasks.workunit.client.0.smithi125.stdout: create 140/38 file after the snapshot 2023-10-30T12:31:24.620 INFO:tasks.workunit.client.0.smithi125.stdout: create 140/39 file after the snapshot 2023-10-30T12:31:24.620 INFO:tasks.workunit.client.0.smithi125.stdout: create 141/.snap/snap-subdir-test 2023-10-30T12:31:24.646 INFO:tasks.workunit.client.0.smithi125.stdout: create 141/30 file after the snapshot 2023-10-30T12:31:24.648 INFO:tasks.workunit.client.0.smithi125.stdout: create 141/31 file after the snapshot 2023-10-30T12:31:24.648 INFO:tasks.workunit.client.0.smithi125.stdout: create 141/32 file after the snapshot 2023-10-30T12:31:24.649 INFO:tasks.workunit.client.0.smithi125.stdout: create 141/33 file after the snapshot 2023-10-30T12:31:24.649 INFO:tasks.workunit.client.0.smithi125.stdout: create 141/34 file after the snapshot 2023-10-30T12:31:24.649 INFO:tasks.workunit.client.0.smithi125.stdout: create 141/35 file after the snapshot 2023-10-30T12:31:24.649 INFO:tasks.workunit.client.0.smithi125.stdout: create 141/36 file after the snapshot 2023-10-30T12:31:24.649 INFO:tasks.workunit.client.0.smithi125.stdout: create 141/37 file after the snapshot 2023-10-30T12:31:24.650 INFO:tasks.workunit.client.0.smithi125.stdout: create 141/38 file after the snapshot 2023-10-30T12:31:24.650 INFO:tasks.workunit.client.0.smithi125.stdout: create 141/39 file after the snapshot 2023-10-30T12:31:24.650 INFO:tasks.workunit.client.0.smithi125.stdout: create 142/.snap/snap-subdir-test 2023-10-30T12:31:24.675 INFO:tasks.workunit.client.0.smithi125.stdout: create 142/30 file after the snapshot 2023-10-30T12:31:24.676 INFO:tasks.workunit.client.0.smithi125.stdout: create 142/31 file after the snapshot 2023-10-30T12:31:24.677 INFO:tasks.workunit.client.0.smithi125.stdout: create 142/32 file after the snapshot 2023-10-30T12:31:24.677 INFO:tasks.workunit.client.0.smithi125.stdout: create 142/33 file after the snapshot 2023-10-30T12:31:24.677 INFO:tasks.workunit.client.0.smithi125.stdout: create 142/34 file after the snapshot 2023-10-30T12:31:24.677 INFO:tasks.workunit.client.0.smithi125.stdout: create 142/35 file after the snapshot 2023-10-30T12:31:24.677 INFO:tasks.workunit.client.0.smithi125.stdout: create 142/36 file after the snapshot 2023-10-30T12:31:24.678 INFO:tasks.workunit.client.0.smithi125.stdout: create 142/37 file after the snapshot 2023-10-30T12:31:24.678 INFO:tasks.workunit.client.0.smithi125.stdout: create 142/38 file after the snapshot 2023-10-30T12:31:24.678 INFO:tasks.workunit.client.0.smithi125.stdout: create 142/39 file after the snapshot 2023-10-30T12:31:24.678 INFO:tasks.workunit.client.0.smithi125.stdout: create 143/.snap/snap-subdir-test 2023-10-30T12:31:24.704 INFO:tasks.workunit.client.0.smithi125.stdout: create 143/30 file after the snapshot 2023-10-30T12:31:24.705 INFO:tasks.workunit.client.0.smithi125.stdout: create 143/31 file after the snapshot 2023-10-30T12:31:24.706 INFO:tasks.workunit.client.0.smithi125.stdout: create 143/32 file after the snapshot 2023-10-30T12:31:24.706 INFO:tasks.workunit.client.0.smithi125.stdout: create 143/33 file after the snapshot 2023-10-30T12:31:24.706 INFO:tasks.workunit.client.0.smithi125.stdout: create 143/34 file after the snapshot 2023-10-30T12:31:24.706 INFO:tasks.workunit.client.0.smithi125.stdout: create 143/35 file after the snapshot 2023-10-30T12:31:24.706 INFO:tasks.workunit.client.0.smithi125.stdout: create 143/36 file after the snapshot 2023-10-30T12:31:24.707 INFO:tasks.workunit.client.0.smithi125.stdout: create 143/37 file after the snapshot 2023-10-30T12:31:24.707 INFO:tasks.workunit.client.0.smithi125.stdout: create 143/38 file after the snapshot 2023-10-30T12:31:24.707 INFO:tasks.workunit.client.0.smithi125.stdout: create 143/39 file after the snapshot 2023-10-30T12:31:24.707 INFO:tasks.workunit.client.0.smithi125.stdout: create 144/.snap/snap-subdir-test 2023-10-30T12:31:25.118 INFO:tasks.workunit.client.0.smithi125.stdout: create 144/30 file after the snapshot 2023-10-30T12:31:25.120 INFO:tasks.workunit.client.0.smithi125.stdout: create 144/31 file after the snapshot 2023-10-30T12:31:25.120 INFO:tasks.workunit.client.0.smithi125.stdout: create 144/32 file after the snapshot 2023-10-30T12:31:25.120 INFO:tasks.workunit.client.0.smithi125.stdout: create 144/33 file after the snapshot 2023-10-30T12:31:25.121 INFO:tasks.workunit.client.0.smithi125.stdout: create 144/34 file after the snapshot 2023-10-30T12:31:25.121 INFO:tasks.workunit.client.0.smithi125.stdout: create 144/35 file after the snapshot 2023-10-30T12:31:25.122 INFO:tasks.workunit.client.0.smithi125.stdout: create 144/36 file after the snapshot 2023-10-30T12:31:25.122 INFO:tasks.workunit.client.0.smithi125.stdout: create 144/37 file after the snapshot 2023-10-30T12:31:25.122 INFO:tasks.workunit.client.0.smithi125.stdout: create 144/38 file after the snapshot 2023-10-30T12:31:25.122 INFO:tasks.workunit.client.0.smithi125.stdout: create 144/39 file after the snapshot 2023-10-30T12:31:25.122 INFO:tasks.workunit.client.0.smithi125.stdout: create 145/.snap/snap-subdir-test 2023-10-30T12:31:25.149 INFO:tasks.workunit.client.0.smithi125.stdout: create 145/30 file after the snapshot 2023-10-30T12:31:25.151 INFO:tasks.workunit.client.0.smithi125.stdout: create 145/31 file after the snapshot 2023-10-30T12:31:25.151 INFO:tasks.workunit.client.0.smithi125.stdout: create 145/32 file after the snapshot 2023-10-30T12:31:25.151 INFO:tasks.workunit.client.0.smithi125.stdout: create 145/33 file after the snapshot 2023-10-30T12:31:25.152 INFO:tasks.workunit.client.0.smithi125.stdout: create 145/34 file after the snapshot 2023-10-30T12:31:25.152 INFO:tasks.workunit.client.0.smithi125.stdout: create 145/35 file after the snapshot 2023-10-30T12:31:25.152 INFO:tasks.workunit.client.0.smithi125.stdout: create 145/36 file after the snapshot 2023-10-30T12:31:25.152 INFO:tasks.workunit.client.0.smithi125.stdout: create 145/37 file after the snapshot 2023-10-30T12:31:25.152 INFO:tasks.workunit.client.0.smithi125.stdout: create 145/38 file after the snapshot 2023-10-30T12:31:25.153 INFO:tasks.workunit.client.0.smithi125.stdout: create 145/39 file after the snapshot 2023-10-30T12:31:25.153 INFO:tasks.workunit.client.0.smithi125.stdout: create 146/.snap/snap-subdir-test 2023-10-30T12:31:25.181 INFO:tasks.workunit.client.0.smithi125.stdout: create 146/30 file after the snapshot 2023-10-30T12:31:25.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 146/31 file after the snapshot 2023-10-30T12:31:25.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 146/32 file after the snapshot 2023-10-30T12:31:25.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 146/33 file after the snapshot 2023-10-30T12:31:25.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 146/34 file after the snapshot 2023-10-30T12:31:25.184 INFO:tasks.workunit.client.0.smithi125.stdout: create 146/35 file after the snapshot 2023-10-30T12:31:25.184 INFO:tasks.workunit.client.0.smithi125.stdout: create 146/36 file after the snapshot 2023-10-30T12:31:25.184 INFO:tasks.workunit.client.0.smithi125.stdout: create 146/37 file after the snapshot 2023-10-30T12:31:25.184 INFO:tasks.workunit.client.0.smithi125.stdout: create 146/38 file after the snapshot 2023-10-30T12:31:25.184 INFO:tasks.workunit.client.0.smithi125.stdout: create 146/39 file after the snapshot 2023-10-30T12:31:25.185 INFO:tasks.workunit.client.0.smithi125.stdout: create 147/.snap/snap-subdir-test 2023-10-30T12:31:25.216 INFO:tasks.workunit.client.0.smithi125.stdout: create 147/30 file after the snapshot 2023-10-30T12:31:25.218 INFO:tasks.workunit.client.0.smithi125.stdout: create 147/31 file after the snapshot 2023-10-30T12:31:25.218 INFO:tasks.workunit.client.0.smithi125.stdout: create 147/32 file after the snapshot 2023-10-30T12:31:25.219 INFO:tasks.workunit.client.0.smithi125.stdout: create 147/33 file after the snapshot 2023-10-30T12:31:25.219 INFO:tasks.workunit.client.0.smithi125.stdout: create 147/34 file after the snapshot 2023-10-30T12:31:25.219 INFO:tasks.workunit.client.0.smithi125.stdout: create 147/35 file after the snapshot 2023-10-30T12:31:25.219 INFO:tasks.workunit.client.0.smithi125.stdout: create 147/36 file after the snapshot 2023-10-30T12:31:25.219 INFO:tasks.workunit.client.0.smithi125.stdout: create 147/37 file after the snapshot 2023-10-30T12:31:25.219 INFO:tasks.workunit.client.0.smithi125.stdout: create 147/38 file after the snapshot 2023-10-30T12:31:25.220 INFO:tasks.workunit.client.0.smithi125.stdout: create 147/39 file after the snapshot 2023-10-30T12:31:25.220 INFO:tasks.workunit.client.0.smithi125.stdout: create 148/.snap/snap-subdir-test 2023-10-30T12:31:25.252 INFO:tasks.workunit.client.0.smithi125.stdout: create 148/30 file after the snapshot 2023-10-30T12:31:25.254 INFO:tasks.workunit.client.0.smithi125.stdout: create 148/31 file after the snapshot 2023-10-30T12:31:25.254 INFO:tasks.workunit.client.0.smithi125.stdout: create 148/32 file after the snapshot 2023-10-30T12:31:25.254 INFO:tasks.workunit.client.0.smithi125.stdout: create 148/33 file after the snapshot 2023-10-30T12:31:25.254 INFO:tasks.workunit.client.0.smithi125.stdout: create 148/34 file after the snapshot 2023-10-30T12:31:25.255 INFO:tasks.workunit.client.0.smithi125.stdout: create 148/35 file after the snapshot 2023-10-30T12:31:25.255 INFO:tasks.workunit.client.0.smithi125.stdout: create 148/36 file after the snapshot 2023-10-30T12:31:25.255 INFO:tasks.workunit.client.0.smithi125.stdout: create 148/37 file after the snapshot 2023-10-30T12:31:25.255 INFO:tasks.workunit.client.0.smithi125.stdout: create 148/38 file after the snapshot 2023-10-30T12:31:25.255 INFO:tasks.workunit.client.0.smithi125.stdout: create 148/39 file after the snapshot 2023-10-30T12:31:25.256 INFO:tasks.workunit.client.0.smithi125.stdout: create 149/.snap/snap-subdir-test 2023-10-30T12:31:25.296 INFO:tasks.workunit.client.0.smithi125.stdout: create 149/30 file after the snapshot 2023-10-30T12:31:25.299 INFO:tasks.workunit.client.0.smithi125.stdout: create 149/31 file after the snapshot 2023-10-30T12:31:25.299 INFO:tasks.workunit.client.0.smithi125.stdout: create 149/32 file after the snapshot 2023-10-30T12:31:25.299 INFO:tasks.workunit.client.0.smithi125.stdout: create 149/33 file after the snapshot 2023-10-30T12:31:25.299 INFO:tasks.workunit.client.0.smithi125.stdout: create 149/34 file after the snapshot 2023-10-30T12:31:25.299 INFO:tasks.workunit.client.0.smithi125.stdout: create 149/35 file after the snapshot 2023-10-30T12:31:25.300 INFO:tasks.workunit.client.0.smithi125.stdout: create 149/36 file after the snapshot 2023-10-30T12:31:25.300 INFO:tasks.workunit.client.0.smithi125.stdout: create 149/37 file after the snapshot 2023-10-30T12:31:25.300 INFO:tasks.workunit.client.0.smithi125.stdout: create 149/38 file after the snapshot 2023-10-30T12:31:25.300 INFO:tasks.workunit.client.0.smithi125.stdout: create 149/39 file after the snapshot 2023-10-30T12:31:25.300 INFO:tasks.workunit.client.0.smithi125.stdout: create 150/.snap/snap-subdir-test 2023-10-30T12:31:25.993 INFO:tasks.workunit.client.0.smithi125.stdout: create 150/30 file after the snapshot 2023-10-30T12:31:26.779 INFO:tasks.workunit.client.0.smithi125.stdout: create 150/31 file after the snapshot 2023-10-30T12:31:26.779 INFO:tasks.workunit.client.0.smithi125.stdout: create 150/32 file after the snapshot 2023-10-30T12:31:26.779 INFO:tasks.workunit.client.0.smithi125.stdout: create 150/33 file after the snapshot 2023-10-30T12:31:26.779 INFO:tasks.workunit.client.0.smithi125.stdout: create 150/34 file after the snapshot 2023-10-30T12:31:26.779 INFO:tasks.workunit.client.0.smithi125.stdout: create 150/35 file after the snapshot 2023-10-30T12:31:26.780 INFO:tasks.workunit.client.0.smithi125.stdout: create 150/36 file after the snapshot 2023-10-30T12:31:26.780 INFO:tasks.workunit.client.0.smithi125.stdout: create 150/37 file after the snapshot 2023-10-30T12:31:26.780 INFO:tasks.workunit.client.0.smithi125.stdout: create 150/38 file after the snapshot 2023-10-30T12:31:26.780 INFO:tasks.workunit.client.0.smithi125.stdout: create 150/39 file after the snapshot 2023-10-30T12:31:26.780 INFO:tasks.workunit.client.0.smithi125.stdout: create 151/.snap/snap-subdir-test 2023-10-30T12:31:26.817 INFO:tasks.workunit.client.0.smithi125.stdout: create 151/30 file after the snapshot 2023-10-30T12:31:26.818 INFO:tasks.workunit.client.0.smithi125.stdout: create 151/31 file after the snapshot 2023-10-30T12:31:26.819 INFO:tasks.workunit.client.0.smithi125.stdout: create 151/32 file after the snapshot 2023-10-30T12:31:26.819 INFO:tasks.workunit.client.0.smithi125.stdout: create 151/33 file after the snapshot 2023-10-30T12:31:26.819 INFO:tasks.workunit.client.0.smithi125.stdout: create 151/34 file after the snapshot 2023-10-30T12:31:26.819 INFO:tasks.workunit.client.0.smithi125.stdout: create 151/35 file after the snapshot 2023-10-30T12:31:26.820 INFO:tasks.workunit.client.0.smithi125.stdout: create 151/36 file after the snapshot 2023-10-30T12:31:26.820 INFO:tasks.workunit.client.0.smithi125.stdout: create 151/37 file after the snapshot 2023-10-30T12:31:26.820 INFO:tasks.workunit.client.0.smithi125.stdout: create 151/38 file after the snapshot 2023-10-30T12:31:26.821 INFO:tasks.workunit.client.0.smithi125.stdout: create 151/39 file after the snapshot 2023-10-30T12:31:26.821 INFO:tasks.workunit.client.0.smithi125.stdout: create 152/.snap/snap-subdir-test 2023-10-30T12:31:26.854 INFO:tasks.workunit.client.0.smithi125.stdout: create 152/30 file after the snapshot 2023-10-30T12:31:26.856 INFO:tasks.workunit.client.0.smithi125.stdout: create 152/31 file after the snapshot 2023-10-30T12:31:26.856 INFO:tasks.workunit.client.0.smithi125.stdout: create 152/32 file after the snapshot 2023-10-30T12:31:26.856 INFO:tasks.workunit.client.0.smithi125.stdout: create 152/33 file after the snapshot 2023-10-30T12:31:26.857 INFO:tasks.workunit.client.0.smithi125.stdout: create 152/34 file after the snapshot 2023-10-30T12:31:26.857 INFO:tasks.workunit.client.0.smithi125.stdout: create 152/35 file after the snapshot 2023-10-30T12:31:26.857 INFO:tasks.workunit.client.0.smithi125.stdout: create 152/36 file after the snapshot 2023-10-30T12:31:26.857 INFO:tasks.workunit.client.0.smithi125.stdout: create 152/37 file after the snapshot 2023-10-30T12:31:26.857 INFO:tasks.workunit.client.0.smithi125.stdout: create 152/38 file after the snapshot 2023-10-30T12:31:26.857 INFO:tasks.workunit.client.0.smithi125.stdout: create 152/39 file after the snapshot 2023-10-30T12:31:26.858 INFO:tasks.workunit.client.0.smithi125.stdout: create 153/.snap/snap-subdir-test 2023-10-30T12:31:26.890 INFO:tasks.workunit.client.0.smithi125.stdout: create 153/30 file after the snapshot 2023-10-30T12:31:26.892 INFO:tasks.workunit.client.0.smithi125.stdout: create 153/31 file after the snapshot 2023-10-30T12:31:26.892 INFO:tasks.workunit.client.0.smithi125.stdout: create 153/32 file after the snapshot 2023-10-30T12:31:26.892 INFO:tasks.workunit.client.0.smithi125.stdout: create 153/33 file after the snapshot 2023-10-30T12:31:26.892 INFO:tasks.workunit.client.0.smithi125.stdout: create 153/34 file after the snapshot 2023-10-30T12:31:26.892 INFO:tasks.workunit.client.0.smithi125.stdout: create 153/35 file after the snapshot 2023-10-30T12:31:26.893 INFO:tasks.workunit.client.0.smithi125.stdout: create 153/36 file after the snapshot 2023-10-30T12:31:26.893 INFO:tasks.workunit.client.0.smithi125.stdout: create 153/37 file after the snapshot 2023-10-30T12:31:26.893 INFO:tasks.workunit.client.0.smithi125.stdout: create 153/38 file after the snapshot 2023-10-30T12:31:26.893 INFO:tasks.workunit.client.0.smithi125.stdout: create 153/39 file after the snapshot 2023-10-30T12:31:26.893 INFO:tasks.workunit.client.0.smithi125.stdout: create 154/.snap/snap-subdir-test 2023-10-30T12:31:26.923 INFO:tasks.workunit.client.0.smithi125.stdout: create 154/30 file after the snapshot 2023-10-30T12:31:26.924 INFO:tasks.workunit.client.0.smithi125.stdout: create 154/31 file after the snapshot 2023-10-30T12:31:26.924 INFO:tasks.workunit.client.0.smithi125.stdout: create 154/32 file after the snapshot 2023-10-30T12:31:26.925 INFO:tasks.workunit.client.0.smithi125.stdout: create 154/33 file after the snapshot 2023-10-30T12:31:26.925 INFO:tasks.workunit.client.0.smithi125.stdout: create 154/34 file after the snapshot 2023-10-30T12:31:26.925 INFO:tasks.workunit.client.0.smithi125.stdout: create 154/35 file after the snapshot 2023-10-30T12:31:26.925 INFO:tasks.workunit.client.0.smithi125.stdout: create 154/36 file after the snapshot 2023-10-30T12:31:26.925 INFO:tasks.workunit.client.0.smithi125.stdout: create 154/37 file after the snapshot 2023-10-30T12:31:26.926 INFO:tasks.workunit.client.0.smithi125.stdout: create 154/38 file after the snapshot 2023-10-30T12:31:26.926 INFO:tasks.workunit.client.0.smithi125.stdout: create 154/39 file after the snapshot 2023-10-30T12:31:26.926 INFO:tasks.workunit.client.0.smithi125.stdout: create 155/.snap/snap-subdir-test 2023-10-30T12:31:26.955 INFO:tasks.workunit.client.0.smithi125.stdout: create 155/30 file after the snapshot 2023-10-30T12:31:26.956 INFO:tasks.workunit.client.0.smithi125.stdout: create 155/31 file after the snapshot 2023-10-30T12:31:26.957 INFO:tasks.workunit.client.0.smithi125.stdout: create 155/32 file after the snapshot 2023-10-30T12:31:26.957 INFO:tasks.workunit.client.0.smithi125.stdout: create 155/33 file after the snapshot 2023-10-30T12:31:26.957 INFO:tasks.workunit.client.0.smithi125.stdout: create 155/34 file after the snapshot 2023-10-30T12:31:26.957 INFO:tasks.workunit.client.0.smithi125.stdout: create 155/35 file after the snapshot 2023-10-30T12:31:26.957 INFO:tasks.workunit.client.0.smithi125.stdout: create 155/36 file after the snapshot 2023-10-30T12:31:26.958 INFO:tasks.workunit.client.0.smithi125.stdout: create 155/37 file after the snapshot 2023-10-30T12:31:26.958 INFO:tasks.workunit.client.0.smithi125.stdout: create 155/38 file after the snapshot 2023-10-30T12:31:26.958 INFO:tasks.workunit.client.0.smithi125.stdout: create 155/39 file after the snapshot 2023-10-30T12:31:26.958 INFO:tasks.workunit.client.0.smithi125.stdout: create 156/.snap/snap-subdir-test 2023-10-30T12:31:26.986 INFO:tasks.workunit.client.0.smithi125.stdout: create 156/30 file after the snapshot 2023-10-30T12:31:26.987 INFO:tasks.workunit.client.0.smithi125.stdout: create 156/31 file after the snapshot 2023-10-30T12:31:26.987 INFO:tasks.workunit.client.0.smithi125.stdout: create 156/32 file after the snapshot 2023-10-30T12:31:26.988 INFO:tasks.workunit.client.0.smithi125.stdout: create 156/33 file after the snapshot 2023-10-30T12:31:26.988 INFO:tasks.workunit.client.0.smithi125.stdout: create 156/34 file after the snapshot 2023-10-30T12:31:26.988 INFO:tasks.workunit.client.0.smithi125.stdout: create 156/35 file after the snapshot 2023-10-30T12:31:26.988 INFO:tasks.workunit.client.0.smithi125.stdout: create 156/36 file after the snapshot 2023-10-30T12:31:26.989 INFO:tasks.workunit.client.0.smithi125.stdout: create 156/37 file after the snapshot 2023-10-30T12:31:26.989 INFO:tasks.workunit.client.0.smithi125.stdout: create 156/38 file after the snapshot 2023-10-30T12:31:26.989 INFO:tasks.workunit.client.0.smithi125.stdout: create 156/39 file after the snapshot 2023-10-30T12:31:26.989 INFO:tasks.workunit.client.0.smithi125.stdout: create 157/.snap/snap-subdir-test 2023-10-30T12:31:27.017 INFO:tasks.workunit.client.0.smithi125.stdout: create 157/30 file after the snapshot 2023-10-30T12:31:27.019 INFO:tasks.workunit.client.0.smithi125.stdout: create 157/31 file after the snapshot 2023-10-30T12:31:27.020 INFO:tasks.workunit.client.0.smithi125.stdout: create 157/32 file after the snapshot 2023-10-30T12:31:27.020 INFO:tasks.workunit.client.0.smithi125.stdout: create 157/33 file after the snapshot 2023-10-30T12:31:27.020 INFO:tasks.workunit.client.0.smithi125.stdout: create 157/34 file after the snapshot 2023-10-30T12:31:27.020 INFO:tasks.workunit.client.0.smithi125.stdout: create 157/35 file after the snapshot 2023-10-30T12:31:27.021 INFO:tasks.workunit.client.0.smithi125.stdout: create 157/36 file after the snapshot 2023-10-30T12:31:27.021 INFO:tasks.workunit.client.0.smithi125.stdout: create 157/37 file after the snapshot 2023-10-30T12:31:27.021 INFO:tasks.workunit.client.0.smithi125.stdout: create 157/38 file after the snapshot 2023-10-30T12:31:27.021 INFO:tasks.workunit.client.0.smithi125.stdout: create 157/39 file after the snapshot 2023-10-30T12:31:27.021 INFO:tasks.workunit.client.0.smithi125.stdout: create 158/.snap/snap-subdir-test 2023-10-30T12:31:27.049 INFO:tasks.workunit.client.0.smithi125.stdout: create 158/30 file after the snapshot 2023-10-30T12:31:27.303 INFO:tasks.workunit.client.0.smithi125.stdout: create 158/31 file after the snapshot 2023-10-30T12:31:27.303 INFO:tasks.workunit.client.0.smithi125.stdout: create 158/32 file after the snapshot 2023-10-30T12:31:27.304 INFO:tasks.workunit.client.0.smithi125.stdout: create 158/33 file after the snapshot 2023-10-30T12:31:27.304 INFO:tasks.workunit.client.0.smithi125.stdout: create 158/34 file after the snapshot 2023-10-30T12:31:27.304 INFO:tasks.workunit.client.0.smithi125.stdout: create 158/35 file after the snapshot 2023-10-30T12:31:27.304 INFO:tasks.workunit.client.0.smithi125.stdout: create 158/36 file after the snapshot 2023-10-30T12:31:27.304 INFO:tasks.workunit.client.0.smithi125.stdout: create 158/37 file after the snapshot 2023-10-30T12:31:27.305 INFO:tasks.workunit.client.0.smithi125.stdout: create 158/38 file after the snapshot 2023-10-30T12:31:27.305 INFO:tasks.workunit.client.0.smithi125.stdout: create 158/39 file after the snapshot 2023-10-30T12:31:27.305 INFO:tasks.workunit.client.0.smithi125.stdout: create 159/.snap/snap-subdir-test 2023-10-30T12:31:27.585 INFO:tasks.workunit.client.0.smithi125.stdout: create 159/30 file after the snapshot 2023-10-30T12:31:27.586 INFO:tasks.workunit.client.0.smithi125.stdout: create 159/31 file after the snapshot 2023-10-30T12:31:27.587 INFO:tasks.workunit.client.0.smithi125.stdout: create 159/32 file after the snapshot 2023-10-30T12:31:27.587 INFO:tasks.workunit.client.0.smithi125.stdout: create 159/33 file after the snapshot 2023-10-30T12:31:27.587 INFO:tasks.workunit.client.0.smithi125.stdout: create 159/34 file after the snapshot 2023-10-30T12:31:27.587 INFO:tasks.workunit.client.0.smithi125.stdout: create 159/35 file after the snapshot 2023-10-30T12:31:27.587 INFO:tasks.workunit.client.0.smithi125.stdout: create 159/36 file after the snapshot 2023-10-30T12:31:27.588 INFO:tasks.workunit.client.0.smithi125.stdout: create 159/37 file after the snapshot 2023-10-30T12:31:27.588 INFO:tasks.workunit.client.0.smithi125.stdout: create 159/38 file after the snapshot 2023-10-30T12:31:27.588 INFO:tasks.workunit.client.0.smithi125.stdout: create 159/39 file after the snapshot 2023-10-30T12:31:27.588 INFO:tasks.workunit.client.0.smithi125.stdout: create 160/.snap/snap-subdir-test 2023-10-30T12:31:27.617 INFO:tasks.workunit.client.0.smithi125.stdout: create 160/30 file after the snapshot 2023-10-30T12:31:27.618 INFO:tasks.workunit.client.0.smithi125.stdout: create 160/31 file after the snapshot 2023-10-30T12:31:27.619 INFO:tasks.workunit.client.0.smithi125.stdout: create 160/32 file after the snapshot 2023-10-30T12:31:27.619 INFO:tasks.workunit.client.0.smithi125.stdout: create 160/33 file after the snapshot 2023-10-30T12:31:27.619 INFO:tasks.workunit.client.0.smithi125.stdout: create 160/34 file after the snapshot 2023-10-30T12:31:27.619 INFO:tasks.workunit.client.0.smithi125.stdout: create 160/35 file after the snapshot 2023-10-30T12:31:27.620 INFO:tasks.workunit.client.0.smithi125.stdout: create 160/36 file after the snapshot 2023-10-30T12:31:27.620 INFO:tasks.workunit.client.0.smithi125.stdout: create 160/37 file after the snapshot 2023-10-30T12:31:27.620 INFO:tasks.workunit.client.0.smithi125.stdout: create 160/38 file after the snapshot 2023-10-30T12:31:27.620 INFO:tasks.workunit.client.0.smithi125.stdout: create 160/39 file after the snapshot 2023-10-30T12:31:27.620 INFO:tasks.workunit.client.0.smithi125.stdout: create 161/.snap/snap-subdir-test 2023-10-30T12:31:28.187 INFO:tasks.workunit.client.0.smithi125.stdout: create 161/30 file after the snapshot 2023-10-30T12:31:28.189 INFO:tasks.workunit.client.0.smithi125.stdout: create 161/31 file after the snapshot 2023-10-30T12:31:28.189 INFO:tasks.workunit.client.0.smithi125.stdout: create 161/32 file after the snapshot 2023-10-30T12:31:28.190 INFO:tasks.workunit.client.0.smithi125.stdout: create 161/33 file after the snapshot 2023-10-30T12:31:28.190 INFO:tasks.workunit.client.0.smithi125.stdout: create 161/34 file after the snapshot 2023-10-30T12:31:28.190 INFO:tasks.workunit.client.0.smithi125.stdout: create 161/35 file after the snapshot 2023-10-30T12:31:28.190 INFO:tasks.workunit.client.0.smithi125.stdout: create 161/36 file after the snapshot 2023-10-30T12:31:28.191 INFO:tasks.workunit.client.0.smithi125.stdout: create 161/37 file after the snapshot 2023-10-30T12:31:28.191 INFO:tasks.workunit.client.0.smithi125.stdout: create 161/38 file after the snapshot 2023-10-30T12:31:28.191 INFO:tasks.workunit.client.0.smithi125.stdout: create 161/39 file after the snapshot 2023-10-30T12:31:28.191 INFO:tasks.workunit.client.0.smithi125.stdout: create 162/.snap/snap-subdir-test 2023-10-30T12:31:29.107 INFO:tasks.workunit.client.0.smithi125.stdout: create 162/30 file after the snapshot 2023-10-30T12:31:29.110 INFO:tasks.workunit.client.0.smithi125.stdout: create 162/31 file after the snapshot 2023-10-30T12:31:29.110 INFO:tasks.workunit.client.0.smithi125.stdout: create 162/32 file after the snapshot 2023-10-30T12:31:29.110 INFO:tasks.workunit.client.0.smithi125.stdout: create 162/33 file after the snapshot 2023-10-30T12:31:29.110 INFO:tasks.workunit.client.0.smithi125.stdout: create 162/34 file after the snapshot 2023-10-30T12:31:29.111 INFO:tasks.workunit.client.0.smithi125.stdout: create 162/35 file after the snapshot 2023-10-30T12:31:29.111 INFO:tasks.workunit.client.0.smithi125.stdout: create 162/36 file after the snapshot 2023-10-30T12:31:29.111 INFO:tasks.workunit.client.0.smithi125.stdout: create 162/37 file after the snapshot 2023-10-30T12:31:29.112 INFO:tasks.workunit.client.0.smithi125.stdout: create 162/38 file after the snapshot 2023-10-30T12:31:29.112 INFO:tasks.workunit.client.0.smithi125.stdout: create 162/39 file after the snapshot 2023-10-30T12:31:29.112 INFO:tasks.workunit.client.0.smithi125.stdout: create 163/.snap/snap-subdir-test 2023-10-30T12:31:29.394 INFO:tasks.workunit.client.0.smithi125.stdout: create 163/30 file after the snapshot 2023-10-30T12:31:29.395 INFO:tasks.workunit.client.0.smithi125.stdout: create 163/31 file after the snapshot 2023-10-30T12:31:29.395 INFO:tasks.workunit.client.0.smithi125.stdout: create 163/32 file after the snapshot 2023-10-30T12:31:29.396 INFO:tasks.workunit.client.0.smithi125.stdout: create 163/33 file after the snapshot 2023-10-30T12:31:29.396 INFO:tasks.workunit.client.0.smithi125.stdout: create 163/34 file after the snapshot 2023-10-30T12:31:29.396 INFO:tasks.workunit.client.0.smithi125.stdout: create 163/35 file after the snapshot 2023-10-30T12:31:29.396 INFO:tasks.workunit.client.0.smithi125.stdout: create 163/36 file after the snapshot 2023-10-30T12:31:29.396 INFO:tasks.workunit.client.0.smithi125.stdout: create 163/37 file after the snapshot 2023-10-30T12:31:29.397 INFO:tasks.workunit.client.0.smithi125.stdout: create 163/38 file after the snapshot 2023-10-30T12:31:29.397 INFO:tasks.workunit.client.0.smithi125.stdout: create 163/39 file after the snapshot 2023-10-30T12:31:29.397 INFO:tasks.workunit.client.0.smithi125.stdout: create 164/.snap/snap-subdir-test 2023-10-30T12:31:29.686 INFO:tasks.workunit.client.0.smithi125.stdout: create 164/30 file after the snapshot 2023-10-30T12:31:29.688 INFO:tasks.workunit.client.0.smithi125.stdout: create 164/31 file after the snapshot 2023-10-30T12:31:29.688 INFO:tasks.workunit.client.0.smithi125.stdout: create 164/32 file after the snapshot 2023-10-30T12:31:29.688 INFO:tasks.workunit.client.0.smithi125.stdout: create 164/33 file after the snapshot 2023-10-30T12:31:29.688 INFO:tasks.workunit.client.0.smithi125.stdout: create 164/34 file after the snapshot 2023-10-30T12:31:29.689 INFO:tasks.workunit.client.0.smithi125.stdout: create 164/35 file after the snapshot 2023-10-30T12:31:29.689 INFO:tasks.workunit.client.0.smithi125.stdout: create 164/36 file after the snapshot 2023-10-30T12:31:29.689 INFO:tasks.workunit.client.0.smithi125.stdout: create 164/37 file after the snapshot 2023-10-30T12:31:29.689 INFO:tasks.workunit.client.0.smithi125.stdout: create 164/38 file after the snapshot 2023-10-30T12:31:29.689 INFO:tasks.workunit.client.0.smithi125.stdout: create 164/39 file after the snapshot 2023-10-30T12:31:29.690 INFO:tasks.workunit.client.0.smithi125.stdout: create 165/.snap/snap-subdir-test 2023-10-30T12:31:29.720 INFO:tasks.workunit.client.0.smithi125.stdout: create 165/30 file after the snapshot 2023-10-30T12:31:29.722 INFO:tasks.workunit.client.0.smithi125.stdout: create 165/31 file after the snapshot 2023-10-30T12:31:29.722 INFO:tasks.workunit.client.0.smithi125.stdout: create 165/32 file after the snapshot 2023-10-30T12:31:29.722 INFO:tasks.workunit.client.0.smithi125.stdout: create 165/33 file after the snapshot 2023-10-30T12:31:29.722 INFO:tasks.workunit.client.0.smithi125.stdout: create 165/34 file after the snapshot 2023-10-30T12:31:29.722 INFO:tasks.workunit.client.0.smithi125.stdout: create 165/35 file after the snapshot 2023-10-30T12:31:29.723 INFO:tasks.workunit.client.0.smithi125.stdout: create 165/36 file after the snapshot 2023-10-30T12:31:29.723 INFO:tasks.workunit.client.0.smithi125.stdout: create 165/37 file after the snapshot 2023-10-30T12:31:29.723 INFO:tasks.workunit.client.0.smithi125.stdout: create 165/38 file after the snapshot 2023-10-30T12:31:29.723 INFO:tasks.workunit.client.0.smithi125.stdout: create 165/39 file after the snapshot 2023-10-30T12:31:29.724 INFO:tasks.workunit.client.0.smithi125.stdout: create 166/.snap/snap-subdir-test 2023-10-30T12:31:29.752 INFO:tasks.workunit.client.0.smithi125.stdout: create 166/30 file after the snapshot 2023-10-30T12:31:29.753 INFO:tasks.workunit.client.0.smithi125.stdout: create 166/31 file after the snapshot 2023-10-30T12:31:29.753 INFO:tasks.workunit.client.0.smithi125.stdout: create 166/32 file after the snapshot 2023-10-30T12:31:29.754 INFO:tasks.workunit.client.0.smithi125.stdout: create 166/33 file after the snapshot 2023-10-30T12:31:29.754 INFO:tasks.workunit.client.0.smithi125.stdout: create 166/34 file after the snapshot 2023-10-30T12:31:29.754 INFO:tasks.workunit.client.0.smithi125.stdout: create 166/35 file after the snapshot 2023-10-30T12:31:29.754 INFO:tasks.workunit.client.0.smithi125.stdout: create 166/36 file after the snapshot 2023-10-30T12:31:29.754 INFO:tasks.workunit.client.0.smithi125.stdout: create 166/37 file after the snapshot 2023-10-30T12:31:29.755 INFO:tasks.workunit.client.0.smithi125.stdout: create 166/38 file after the snapshot 2023-10-30T12:31:29.755 INFO:tasks.workunit.client.0.smithi125.stdout: create 166/39 file after the snapshot 2023-10-30T12:31:29.755 INFO:tasks.workunit.client.0.smithi125.stdout: create 167/.snap/snap-subdir-test 2023-10-30T12:31:29.783 INFO:tasks.workunit.client.0.smithi125.stdout: create 167/30 file after the snapshot 2023-10-30T12:31:29.785 INFO:tasks.workunit.client.0.smithi125.stdout: create 167/31 file after the snapshot 2023-10-30T12:31:29.785 INFO:tasks.workunit.client.0.smithi125.stdout: create 167/32 file after the snapshot 2023-10-30T12:31:29.785 INFO:tasks.workunit.client.0.smithi125.stdout: create 167/33 file after the snapshot 2023-10-30T12:31:29.785 INFO:tasks.workunit.client.0.smithi125.stdout: create 167/34 file after the snapshot 2023-10-30T12:31:29.785 INFO:tasks.workunit.client.0.smithi125.stdout: create 167/35 file after the snapshot 2023-10-30T12:31:29.786 INFO:tasks.workunit.client.0.smithi125.stdout: create 167/36 file after the snapshot 2023-10-30T12:31:29.786 INFO:tasks.workunit.client.0.smithi125.stdout: create 167/37 file after the snapshot 2023-10-30T12:31:29.786 INFO:tasks.workunit.client.0.smithi125.stdout: create 167/38 file after the snapshot 2023-10-30T12:31:29.786 INFO:tasks.workunit.client.0.smithi125.stdout: create 167/39 file after the snapshot 2023-10-30T12:31:29.786 INFO:tasks.workunit.client.0.smithi125.stdout: create 168/.snap/snap-subdir-test 2023-10-30T12:31:29.815 INFO:tasks.workunit.client.0.smithi125.stdout: create 168/30 file after the snapshot 2023-10-30T12:31:29.816 INFO:tasks.workunit.client.0.smithi125.stdout: create 168/31 file after the snapshot 2023-10-30T12:31:29.816 INFO:tasks.workunit.client.0.smithi125.stdout: create 168/32 file after the snapshot 2023-10-30T12:31:29.816 INFO:tasks.workunit.client.0.smithi125.stdout: create 168/33 file after the snapshot 2023-10-30T12:31:29.817 INFO:tasks.workunit.client.0.smithi125.stdout: create 168/34 file after the snapshot 2023-10-30T12:31:29.817 INFO:tasks.workunit.client.0.smithi125.stdout: create 168/35 file after the snapshot 2023-10-30T12:31:29.818 INFO:tasks.workunit.client.0.smithi125.stdout: create 168/36 file after the snapshot 2023-10-30T12:31:29.818 INFO:tasks.workunit.client.0.smithi125.stdout: create 168/37 file after the snapshot 2023-10-30T12:31:29.818 INFO:tasks.workunit.client.0.smithi125.stdout: create 168/38 file after the snapshot 2023-10-30T12:31:29.818 INFO:tasks.workunit.client.0.smithi125.stdout: create 168/39 file after the snapshot 2023-10-30T12:31:29.819 INFO:tasks.workunit.client.0.smithi125.stdout: create 169/.snap/snap-subdir-test 2023-10-30T12:31:29.846 INFO:tasks.workunit.client.0.smithi125.stdout: create 169/30 file after the snapshot 2023-10-30T12:31:29.847 INFO:tasks.workunit.client.0.smithi125.stdout: create 169/31 file after the snapshot 2023-10-30T12:31:29.847 INFO:tasks.workunit.client.0.smithi125.stdout: create 169/32 file after the snapshot 2023-10-30T12:31:29.848 INFO:tasks.workunit.client.0.smithi125.stdout: create 169/33 file after the snapshot 2023-10-30T12:31:29.848 INFO:tasks.workunit.client.0.smithi125.stdout: create 169/34 file after the snapshot 2023-10-30T12:31:29.848 INFO:tasks.workunit.client.0.smithi125.stdout: create 169/35 file after the snapshot 2023-10-30T12:31:29.849 INFO:tasks.workunit.client.0.smithi125.stdout: create 169/36 file after the snapshot 2023-10-30T12:31:29.849 INFO:tasks.workunit.client.0.smithi125.stdout: create 169/37 file after the snapshot 2023-10-30T12:31:29.849 INFO:tasks.workunit.client.0.smithi125.stdout: create 169/38 file after the snapshot 2023-10-30T12:31:29.850 INFO:tasks.workunit.client.0.smithi125.stdout: create 169/39 file after the snapshot 2023-10-30T12:31:29.850 INFO:tasks.workunit.client.0.smithi125.stdout: create 170/.snap/snap-subdir-test 2023-10-30T12:31:29.878 INFO:tasks.workunit.client.0.smithi125.stdout: create 170/30 file after the snapshot 2023-10-30T12:31:29.880 INFO:tasks.workunit.client.0.smithi125.stdout: create 170/31 file after the snapshot 2023-10-30T12:31:29.880 INFO:tasks.workunit.client.0.smithi125.stdout: create 170/32 file after the snapshot 2023-10-30T12:31:29.880 INFO:tasks.workunit.client.0.smithi125.stdout: create 170/33 file after the snapshot 2023-10-30T12:31:29.880 INFO:tasks.workunit.client.0.smithi125.stdout: create 170/34 file after the snapshot 2023-10-30T12:31:29.880 INFO:tasks.workunit.client.0.smithi125.stdout: create 170/35 file after the snapshot 2023-10-30T12:31:29.881 INFO:tasks.workunit.client.0.smithi125.stdout: create 170/36 file after the snapshot 2023-10-30T12:31:29.882 INFO:tasks.workunit.client.0.smithi125.stdout: create 170/37 file after the snapshot 2023-10-30T12:31:29.882 INFO:tasks.workunit.client.0.smithi125.stdout: create 170/38 file after the snapshot 2023-10-30T12:31:29.882 INFO:tasks.workunit.client.0.smithi125.stdout: create 170/39 file after the snapshot 2023-10-30T12:31:29.882 INFO:tasks.workunit.client.0.smithi125.stdout: create 171/.snap/snap-subdir-test 2023-10-30T12:31:29.911 INFO:tasks.workunit.client.0.smithi125.stdout: create 171/30 file after the snapshot 2023-10-30T12:31:29.913 INFO:tasks.workunit.client.0.smithi125.stdout: create 171/31 file after the snapshot 2023-10-30T12:31:29.913 INFO:tasks.workunit.client.0.smithi125.stdout: create 171/32 file after the snapshot 2023-10-30T12:31:29.914 INFO:tasks.workunit.client.0.smithi125.stdout: create 171/33 file after the snapshot 2023-10-30T12:31:29.914 INFO:tasks.workunit.client.0.smithi125.stdout: create 171/34 file after the snapshot 2023-10-30T12:31:29.914 INFO:tasks.workunit.client.0.smithi125.stdout: create 171/35 file after the snapshot 2023-10-30T12:31:29.914 INFO:tasks.workunit.client.0.smithi125.stdout: create 171/36 file after the snapshot 2023-10-30T12:31:29.914 INFO:tasks.workunit.client.0.smithi125.stdout: create 171/37 file after the snapshot 2023-10-30T12:31:29.915 INFO:tasks.workunit.client.0.smithi125.stdout: create 171/38 file after the snapshot 2023-10-30T12:31:29.915 INFO:tasks.workunit.client.0.smithi125.stdout: create 171/39 file after the snapshot 2023-10-30T12:31:29.915 INFO:tasks.workunit.client.0.smithi125.stdout: create 172/.snap/snap-subdir-test 2023-10-30T12:31:29.950 INFO:tasks.workunit.client.0.smithi125.stdout: create 172/30 file after the snapshot 2023-10-30T12:31:29.951 INFO:tasks.workunit.client.0.smithi125.stdout: create 172/31 file after the snapshot 2023-10-30T12:31:29.951 INFO:tasks.workunit.client.0.smithi125.stdout: create 172/32 file after the snapshot 2023-10-30T12:31:29.951 INFO:tasks.workunit.client.0.smithi125.stdout: create 172/33 file after the snapshot 2023-10-30T12:31:29.952 INFO:tasks.workunit.client.0.smithi125.stdout: create 172/34 file after the snapshot 2023-10-30T12:31:29.953 INFO:tasks.workunit.client.0.smithi125.stdout: create 172/35 file after the snapshot 2023-10-30T12:31:29.953 INFO:tasks.workunit.client.0.smithi125.stdout: create 172/36 file after the snapshot 2023-10-30T12:31:29.953 INFO:tasks.workunit.client.0.smithi125.stdout: create 172/37 file after the snapshot 2023-10-30T12:31:29.953 INFO:tasks.workunit.client.0.smithi125.stdout: create 172/38 file after the snapshot 2023-10-30T12:31:29.953 INFO:tasks.workunit.client.0.smithi125.stdout: create 172/39 file after the snapshot 2023-10-30T12:31:29.954 INFO:tasks.workunit.client.0.smithi125.stdout: create 173/.snap/snap-subdir-test 2023-10-30T12:31:30.897 INFO:tasks.workunit.client.0.smithi125.stdout: create 173/30 file after the snapshot 2023-10-30T12:31:30.900 INFO:tasks.workunit.client.0.smithi125.stdout: create 173/31 file after the snapshot 2023-10-30T12:31:30.900 INFO:tasks.workunit.client.0.smithi125.stdout: create 173/32 file after the snapshot 2023-10-30T12:31:30.900 INFO:tasks.workunit.client.0.smithi125.stdout: create 173/33 file after the snapshot 2023-10-30T12:31:30.900 INFO:tasks.workunit.client.0.smithi125.stdout: create 173/34 file after the snapshot 2023-10-30T12:31:30.900 INFO:tasks.workunit.client.0.smithi125.stdout: create 173/35 file after the snapshot 2023-10-30T12:31:30.901 INFO:tasks.workunit.client.0.smithi125.stdout: create 173/36 file after the snapshot 2023-10-30T12:31:30.901 INFO:tasks.workunit.client.0.smithi125.stdout: create 173/37 file after the snapshot 2023-10-30T12:31:30.901 INFO:tasks.workunit.client.0.smithi125.stdout: create 173/38 file after the snapshot 2023-10-30T12:31:30.901 INFO:tasks.workunit.client.0.smithi125.stdout: create 173/39 file after the snapshot 2023-10-30T12:31:30.902 INFO:tasks.workunit.client.0.smithi125.stdout: create 174/.snap/snap-subdir-test 2023-10-30T12:31:30.935 INFO:tasks.workunit.client.0.smithi125.stdout: create 174/30 file after the snapshot 2023-10-30T12:31:30.937 INFO:tasks.workunit.client.0.smithi125.stdout: create 174/31 file after the snapshot 2023-10-30T12:31:30.937 INFO:tasks.workunit.client.0.smithi125.stdout: create 174/32 file after the snapshot 2023-10-30T12:31:30.937 INFO:tasks.workunit.client.0.smithi125.stdout: create 174/33 file after the snapshot 2023-10-30T12:31:30.937 INFO:tasks.workunit.client.0.smithi125.stdout: create 174/34 file after the snapshot 2023-10-30T12:31:30.937 INFO:tasks.workunit.client.0.smithi125.stdout: create 174/35 file after the snapshot 2023-10-30T12:31:30.938 INFO:tasks.workunit.client.0.smithi125.stdout: create 174/36 file after the snapshot 2023-10-30T12:31:30.938 INFO:tasks.workunit.client.0.smithi125.stdout: create 174/37 file after the snapshot 2023-10-30T12:31:30.938 INFO:tasks.workunit.client.0.smithi125.stdout: create 174/38 file after the snapshot 2023-10-30T12:31:30.938 INFO:tasks.workunit.client.0.smithi125.stdout: create 174/39 file after the snapshot 2023-10-30T12:31:30.938 INFO:tasks.workunit.client.0.smithi125.stdout: create 175/.snap/snap-subdir-test 2023-10-30T12:31:30.969 INFO:tasks.workunit.client.0.smithi125.stdout: create 175/30 file after the snapshot 2023-10-30T12:31:30.971 INFO:tasks.workunit.client.0.smithi125.stdout: create 175/31 file after the snapshot 2023-10-30T12:31:30.971 INFO:tasks.workunit.client.0.smithi125.stdout: create 175/32 file after the snapshot 2023-10-30T12:31:30.971 INFO:tasks.workunit.client.0.smithi125.stdout: create 175/33 file after the snapshot 2023-10-30T12:31:30.971 INFO:tasks.workunit.client.0.smithi125.stdout: create 175/34 file after the snapshot 2023-10-30T12:31:30.972 INFO:tasks.workunit.client.0.smithi125.stdout: create 175/35 file after the snapshot 2023-10-30T12:31:30.972 INFO:tasks.workunit.client.0.smithi125.stdout: create 175/36 file after the snapshot 2023-10-30T12:31:30.972 INFO:tasks.workunit.client.0.smithi125.stdout: create 175/37 file after the snapshot 2023-10-30T12:31:30.972 INFO:tasks.workunit.client.0.smithi125.stdout: create 175/38 file after the snapshot 2023-10-30T12:31:30.972 INFO:tasks.workunit.client.0.smithi125.stdout: create 175/39 file after the snapshot 2023-10-30T12:31:30.973 INFO:tasks.workunit.client.0.smithi125.stdout: create 176/.snap/snap-subdir-test 2023-10-30T12:31:31.007 INFO:tasks.workunit.client.0.smithi125.stdout: create 176/30 file after the snapshot 2023-10-30T12:31:31.009 INFO:tasks.workunit.client.0.smithi125.stdout: create 176/31 file after the snapshot 2023-10-30T12:31:31.009 INFO:tasks.workunit.client.0.smithi125.stdout: create 176/32 file after the snapshot 2023-10-30T12:31:31.009 INFO:tasks.workunit.client.0.smithi125.stdout: create 176/33 file after the snapshot 2023-10-30T12:31:31.010 INFO:tasks.workunit.client.0.smithi125.stdout: create 176/34 file after the snapshot 2023-10-30T12:31:31.010 INFO:tasks.workunit.client.0.smithi125.stdout: create 176/35 file after the snapshot 2023-10-30T12:31:31.010 INFO:tasks.workunit.client.0.smithi125.stdout: create 176/36 file after the snapshot 2023-10-30T12:31:31.010 INFO:tasks.workunit.client.0.smithi125.stdout: create 176/37 file after the snapshot 2023-10-30T12:31:31.010 INFO:tasks.workunit.client.0.smithi125.stdout: create 176/38 file after the snapshot 2023-10-30T12:31:31.011 INFO:tasks.workunit.client.0.smithi125.stdout: create 176/39 file after the snapshot 2023-10-30T12:31:31.011 INFO:tasks.workunit.client.0.smithi125.stdout: create 177/.snap/snap-subdir-test 2023-10-30T12:31:31.749 INFO:tasks.workunit.client.0.smithi125.stdout: create 177/30 file after the snapshot 2023-10-30T12:31:31.848 INFO:tasks.workunit.client.0.smithi125.stdout: create 177/31 file after the snapshot 2023-10-30T12:31:31.849 INFO:tasks.workunit.client.0.smithi125.stdout: create 177/32 file after the snapshot 2023-10-30T12:31:31.850 INFO:tasks.workunit.client.0.smithi125.stdout: create 177/33 file after the snapshot 2023-10-30T12:31:31.851 INFO:tasks.workunit.client.0.smithi125.stdout: create 177/34 file after the snapshot 2023-10-30T12:31:31.853 INFO:tasks.workunit.client.0.smithi125.stdout: create 177/35 file after the snapshot 2023-10-30T12:31:31.853 INFO:tasks.workunit.client.0.smithi125.stdout: create 177/36 file after the snapshot 2023-10-30T12:31:31.854 INFO:tasks.workunit.client.0.smithi125.stdout: create 177/37 file after the snapshot 2023-10-30T12:31:31.855 INFO:tasks.workunit.client.0.smithi125.stdout: create 177/38 file after the snapshot 2023-10-30T12:31:31.856 INFO:tasks.workunit.client.0.smithi125.stdout: create 177/39 file after the snapshot 2023-10-30T12:31:31.857 INFO:tasks.workunit.client.0.smithi125.stdout: create 178/.snap/snap-subdir-test 2023-10-30T12:31:32.708 INFO:tasks.workunit.client.0.smithi125.stdout: create 178/30 file after the snapshot 2023-10-30T12:31:32.717 INFO:tasks.workunit.client.0.smithi125.stdout: create 178/31 file after the snapshot 2023-10-30T12:31:32.720 INFO:tasks.workunit.client.0.smithi125.stdout: create 178/32 file after the snapshot 2023-10-30T12:31:32.720 INFO:tasks.workunit.client.0.smithi125.stdout: create 178/33 file after the snapshot 2023-10-30T12:31:32.720 INFO:tasks.workunit.client.0.smithi125.stdout: create 178/34 file after the snapshot 2023-10-30T12:31:32.721 INFO:tasks.workunit.client.0.smithi125.stdout: create 178/35 file after the snapshot 2023-10-30T12:31:32.721 INFO:tasks.workunit.client.0.smithi125.stdout: create 178/36 file after the snapshot 2023-10-30T12:31:32.721 INFO:tasks.workunit.client.0.smithi125.stdout: create 178/37 file after the snapshot 2023-10-30T12:31:32.721 INFO:tasks.workunit.client.0.smithi125.stdout: create 178/38 file after the snapshot 2023-10-30T12:31:32.722 INFO:tasks.workunit.client.0.smithi125.stdout: create 178/39 file after the snapshot 2023-10-30T12:31:32.722 INFO:tasks.workunit.client.0.smithi125.stdout: create 179/.snap/snap-subdir-test 2023-10-30T12:31:32.763 INFO:tasks.workunit.client.0.smithi125.stdout: create 179/30 file after the snapshot 2023-10-30T12:31:32.768 INFO:tasks.workunit.client.0.smithi125.stdout: create 179/31 file after the snapshot 2023-10-30T12:31:32.769 INFO:tasks.workunit.client.0.smithi125.stdout: create 179/32 file after the snapshot 2023-10-30T12:31:32.769 INFO:tasks.workunit.client.0.smithi125.stdout: create 179/33 file after the snapshot 2023-10-30T12:31:32.769 INFO:tasks.workunit.client.0.smithi125.stdout: create 179/34 file after the snapshot 2023-10-30T12:31:32.769 INFO:tasks.workunit.client.0.smithi125.stdout: create 179/35 file after the snapshot 2023-10-30T12:31:32.770 INFO:tasks.workunit.client.0.smithi125.stdout: create 179/36 file after the snapshot 2023-10-30T12:31:32.770 INFO:tasks.workunit.client.0.smithi125.stdout: create 179/37 file after the snapshot 2023-10-30T12:31:32.770 INFO:tasks.workunit.client.0.smithi125.stdout: create 179/38 file after the snapshot 2023-10-30T12:31:32.770 INFO:tasks.workunit.client.0.smithi125.stdout: create 179/39 file after the snapshot 2023-10-30T12:31:32.770 INFO:tasks.workunit.client.0.smithi125.stdout: create 180/.snap/snap-subdir-test 2023-10-30T12:31:32.789 INFO:tasks.workunit.client.0.smithi125.stdout: create 180/30 file after the snapshot 2023-10-30T12:31:33.043 INFO:tasks.workunit.client.0.smithi125.stdout: create 180/31 file after the snapshot 2023-10-30T12:31:33.043 INFO:tasks.workunit.client.0.smithi125.stdout: create 180/32 file after the snapshot 2023-10-30T12:31:33.044 INFO:tasks.workunit.client.0.smithi125.stdout: create 180/33 file after the snapshot 2023-10-30T12:31:33.044 INFO:tasks.workunit.client.0.smithi125.stdout: create 180/34 file after the snapshot 2023-10-30T12:31:33.044 INFO:tasks.workunit.client.0.smithi125.stdout: create 180/35 file after the snapshot 2023-10-30T12:31:33.044 INFO:tasks.workunit.client.0.smithi125.stdout: create 180/36 file after the snapshot 2023-10-30T12:31:33.045 INFO:tasks.workunit.client.0.smithi125.stdout: create 180/37 file after the snapshot 2023-10-30T12:31:33.045 INFO:tasks.workunit.client.0.smithi125.stdout: create 180/38 file after the snapshot 2023-10-30T12:31:33.045 INFO:tasks.workunit.client.0.smithi125.stdout: create 180/39 file after the snapshot 2023-10-30T12:31:33.045 INFO:tasks.workunit.client.0.smithi125.stdout: create 181/.snap/snap-subdir-test 2023-10-30T12:31:33.453 INFO:tasks.workunit.client.0.smithi125.stdout: create 181/30 file after the snapshot 2023-10-30T12:31:33.482 INFO:tasks.workunit.client.0.smithi125.stdout: create 181/31 file after the snapshot 2023-10-30T12:31:33.482 INFO:tasks.workunit.client.0.smithi125.stdout: create 181/32 file after the snapshot 2023-10-30T12:31:33.482 INFO:tasks.workunit.client.0.smithi125.stdout: create 181/33 file after the snapshot 2023-10-30T12:31:33.483 INFO:tasks.workunit.client.0.smithi125.stdout: create 181/34 file after the snapshot 2023-10-30T12:31:33.483 INFO:tasks.workunit.client.0.smithi125.stdout: create 181/35 file after the snapshot 2023-10-30T12:31:33.483 INFO:tasks.workunit.client.0.smithi125.stdout: create 181/36 file after the snapshot 2023-10-30T12:31:33.483 INFO:tasks.workunit.client.0.smithi125.stdout: create 181/37 file after the snapshot 2023-10-30T12:31:33.483 INFO:tasks.workunit.client.0.smithi125.stdout: create 181/38 file after the snapshot 2023-10-30T12:31:33.483 INFO:tasks.workunit.client.0.smithi125.stdout: create 181/39 file after the snapshot 2023-10-30T12:31:33.484 INFO:tasks.workunit.client.0.smithi125.stdout: create 182/.snap/snap-subdir-test 2023-10-30T12:31:33.632 INFO:tasks.workunit.client.0.smithi125.stdout: create 182/30 file after the snapshot 2023-10-30T12:31:33.633 INFO:tasks.workunit.client.0.smithi125.stdout: create 182/31 file after the snapshot 2023-10-30T12:31:33.633 INFO:tasks.workunit.client.0.smithi125.stdout: create 182/32 file after the snapshot 2023-10-30T12:31:33.634 INFO:tasks.workunit.client.0.smithi125.stdout: create 182/33 file after the snapshot 2023-10-30T12:31:33.634 INFO:tasks.workunit.client.0.smithi125.stdout: create 182/34 file after the snapshot 2023-10-30T12:31:33.634 INFO:tasks.workunit.client.0.smithi125.stdout: create 182/35 file after the snapshot 2023-10-30T12:31:33.634 INFO:tasks.workunit.client.0.smithi125.stdout: create 182/36 file after the snapshot 2023-10-30T12:31:33.635 INFO:tasks.workunit.client.0.smithi125.stdout: create 182/37 file after the snapshot 2023-10-30T12:31:33.635 INFO:tasks.workunit.client.0.smithi125.stdout: create 182/38 file after the snapshot 2023-10-30T12:31:33.635 INFO:tasks.workunit.client.0.smithi125.stdout: create 182/39 file after the snapshot 2023-10-30T12:31:33.635 INFO:tasks.workunit.client.0.smithi125.stdout: create 183/.snap/snap-subdir-test 2023-10-30T12:31:33.657 INFO:tasks.workunit.client.0.smithi125.stdout: create 183/30 file after the snapshot 2023-10-30T12:31:33.660 INFO:tasks.workunit.client.0.smithi125.stdout: create 183/31 file after the snapshot 2023-10-30T12:31:33.660 INFO:tasks.workunit.client.0.smithi125.stdout: create 183/32 file after the snapshot 2023-10-30T12:31:33.660 INFO:tasks.workunit.client.0.smithi125.stdout: create 183/33 file after the snapshot 2023-10-30T12:31:33.660 INFO:tasks.workunit.client.0.smithi125.stdout: create 183/34 file after the snapshot 2023-10-30T12:31:33.661 INFO:tasks.workunit.client.0.smithi125.stdout: create 183/35 file after the snapshot 2023-10-30T12:31:33.661 INFO:tasks.workunit.client.0.smithi125.stdout: create 183/36 file after the snapshot 2023-10-30T12:31:33.661 INFO:tasks.workunit.client.0.smithi125.stdout: create 183/37 file after the snapshot 2023-10-30T12:31:33.661 INFO:tasks.workunit.client.0.smithi125.stdout: create 183/38 file after the snapshot 2023-10-30T12:31:33.661 INFO:tasks.workunit.client.0.smithi125.stdout: create 183/39 file after the snapshot 2023-10-30T12:31:33.662 INFO:tasks.workunit.client.0.smithi125.stdout: create 184/.snap/snap-subdir-test 2023-10-30T12:31:33.683 INFO:tasks.workunit.client.0.smithi125.stdout: create 184/30 file after the snapshot 2023-10-30T12:31:33.685 INFO:tasks.workunit.client.0.smithi125.stdout: create 184/31 file after the snapshot 2023-10-30T12:31:33.685 INFO:tasks.workunit.client.0.smithi125.stdout: create 184/32 file after the snapshot 2023-10-30T12:31:33.685 INFO:tasks.workunit.client.0.smithi125.stdout: create 184/33 file after the snapshot 2023-10-30T12:31:33.686 INFO:tasks.workunit.client.0.smithi125.stdout: create 184/34 file after the snapshot 2023-10-30T12:31:33.686 INFO:tasks.workunit.client.0.smithi125.stdout: create 184/35 file after the snapshot 2023-10-30T12:31:33.686 INFO:tasks.workunit.client.0.smithi125.stdout: create 184/36 file after the snapshot 2023-10-30T12:31:33.686 INFO:tasks.workunit.client.0.smithi125.stdout: create 184/37 file after the snapshot 2023-10-30T12:31:33.687 INFO:tasks.workunit.client.0.smithi125.stdout: create 184/38 file after the snapshot 2023-10-30T12:31:33.687 INFO:tasks.workunit.client.0.smithi125.stdout: create 184/39 file after the snapshot 2023-10-30T12:31:33.687 INFO:tasks.workunit.client.0.smithi125.stdout: create 185/.snap/snap-subdir-test 2023-10-30T12:31:33.701 INFO:tasks.workunit.client.0.smithi125.stdout: create 185/30 file after the snapshot 2023-10-30T12:31:33.703 INFO:tasks.workunit.client.0.smithi125.stdout: create 185/31 file after the snapshot 2023-10-30T12:31:33.703 INFO:tasks.workunit.client.0.smithi125.stdout: create 185/32 file after the snapshot 2023-10-30T12:31:33.703 INFO:tasks.workunit.client.0.smithi125.stdout: create 185/33 file after the snapshot 2023-10-30T12:31:33.703 INFO:tasks.workunit.client.0.smithi125.stdout: create 185/34 file after the snapshot 2023-10-30T12:31:33.704 INFO:tasks.workunit.client.0.smithi125.stdout: create 185/35 file after the snapshot 2023-10-30T12:31:33.704 INFO:tasks.workunit.client.0.smithi125.stdout: create 185/36 file after the snapshot 2023-10-30T12:31:33.704 INFO:tasks.workunit.client.0.smithi125.stdout: create 185/37 file after the snapshot 2023-10-30T12:31:33.704 INFO:tasks.workunit.client.0.smithi125.stdout: create 185/38 file after the snapshot 2023-10-30T12:31:33.704 INFO:tasks.workunit.client.0.smithi125.stdout: create 185/39 file after the snapshot 2023-10-30T12:31:33.705 INFO:tasks.workunit.client.0.smithi125.stdout: create 186/.snap/snap-subdir-test 2023-10-30T12:31:33.723 INFO:tasks.workunit.client.0.smithi125.stdout: create 186/30 file after the snapshot 2023-10-30T12:31:33.725 INFO:tasks.workunit.client.0.smithi125.stdout: create 186/31 file after the snapshot 2023-10-30T12:31:33.725 INFO:tasks.workunit.client.0.smithi125.stdout: create 186/32 file after the snapshot 2023-10-30T12:31:33.726 INFO:tasks.workunit.client.0.smithi125.stdout: create 186/33 file after the snapshot 2023-10-30T12:31:33.726 INFO:tasks.workunit.client.0.smithi125.stdout: create 186/34 file after the snapshot 2023-10-30T12:31:33.726 INFO:tasks.workunit.client.0.smithi125.stdout: create 186/35 file after the snapshot 2023-10-30T12:31:33.726 INFO:tasks.workunit.client.0.smithi125.stdout: create 186/36 file after the snapshot 2023-10-30T12:31:33.726 INFO:tasks.workunit.client.0.smithi125.stdout: create 186/37 file after the snapshot 2023-10-30T12:31:33.727 INFO:tasks.workunit.client.0.smithi125.stdout: create 186/38 file after the snapshot 2023-10-30T12:31:33.727 INFO:tasks.workunit.client.0.smithi125.stdout: create 186/39 file after the snapshot 2023-10-30T12:31:33.727 INFO:tasks.workunit.client.0.smithi125.stdout: create 187/.snap/snap-subdir-test 2023-10-30T12:31:33.998 INFO:tasks.workunit.client.0.smithi125.stdout: create 187/30 file after the snapshot 2023-10-30T12:31:34.000 INFO:tasks.workunit.client.0.smithi125.stdout: create 187/31 file after the snapshot 2023-10-30T12:31:34.000 INFO:tasks.workunit.client.0.smithi125.stdout: create 187/32 file after the snapshot 2023-10-30T12:31:34.000 INFO:tasks.workunit.client.0.smithi125.stdout: create 187/33 file after the snapshot 2023-10-30T12:31:34.000 INFO:tasks.workunit.client.0.smithi125.stdout: create 187/34 file after the snapshot 2023-10-30T12:31:34.000 INFO:tasks.workunit.client.0.smithi125.stdout: create 187/35 file after the snapshot 2023-10-30T12:31:34.001 INFO:tasks.workunit.client.0.smithi125.stdout: create 187/36 file after the snapshot 2023-10-30T12:31:34.001 INFO:tasks.workunit.client.0.smithi125.stdout: create 187/37 file after the snapshot 2023-10-30T12:31:34.001 INFO:tasks.workunit.client.0.smithi125.stdout: create 187/38 file after the snapshot 2023-10-30T12:31:34.001 INFO:tasks.workunit.client.0.smithi125.stdout: create 187/39 file after the snapshot 2023-10-30T12:31:34.002 INFO:tasks.workunit.client.0.smithi125.stdout: create 188/.snap/snap-subdir-test 2023-10-30T12:31:34.023 INFO:tasks.workunit.client.0.smithi125.stdout: create 188/30 file after the snapshot 2023-10-30T12:31:34.276 INFO:tasks.workunit.client.0.smithi125.stdout: create 188/31 file after the snapshot 2023-10-30T12:31:34.277 INFO:tasks.workunit.client.0.smithi125.stdout: create 188/32 file after the snapshot 2023-10-30T12:31:34.277 INFO:tasks.workunit.client.0.smithi125.stdout: create 188/33 file after the snapshot 2023-10-30T12:31:34.277 INFO:tasks.workunit.client.0.smithi125.stdout: create 188/34 file after the snapshot 2023-10-30T12:31:34.277 INFO:tasks.workunit.client.0.smithi125.stdout: create 188/35 file after the snapshot 2023-10-30T12:31:34.277 INFO:tasks.workunit.client.0.smithi125.stdout: create 188/36 file after the snapshot 2023-10-30T12:31:34.278 INFO:tasks.workunit.client.0.smithi125.stdout: create 188/37 file after the snapshot 2023-10-30T12:31:34.278 INFO:tasks.workunit.client.0.smithi125.stdout: create 188/38 file after the snapshot 2023-10-30T12:31:34.278 INFO:tasks.workunit.client.0.smithi125.stdout: create 188/39 file after the snapshot 2023-10-30T12:31:34.278 INFO:tasks.workunit.client.0.smithi125.stdout: create 189/.snap/snap-subdir-test 2023-10-30T12:31:34.307 INFO:tasks.workunit.client.0.smithi125.stdout: create 189/30 file after the snapshot 2023-10-30T12:31:34.308 INFO:tasks.workunit.client.0.smithi125.stdout: create 189/31 file after the snapshot 2023-10-30T12:31:34.309 INFO:tasks.workunit.client.0.smithi125.stdout: create 189/32 file after the snapshot 2023-10-30T12:31:34.309 INFO:tasks.workunit.client.0.smithi125.stdout: create 189/33 file after the snapshot 2023-10-30T12:31:34.309 INFO:tasks.workunit.client.0.smithi125.stdout: create 189/34 file after the snapshot 2023-10-30T12:31:34.309 INFO:tasks.workunit.client.0.smithi125.stdout: create 189/35 file after the snapshot 2023-10-30T12:31:34.310 INFO:tasks.workunit.client.0.smithi125.stdout: create 189/36 file after the snapshot 2023-10-30T12:31:34.310 INFO:tasks.workunit.client.0.smithi125.stdout: create 189/37 file after the snapshot 2023-10-30T12:31:34.310 INFO:tasks.workunit.client.0.smithi125.stdout: create 189/38 file after the snapshot 2023-10-30T12:31:34.310 INFO:tasks.workunit.client.0.smithi125.stdout: create 189/39 file after the snapshot 2023-10-30T12:31:34.310 INFO:tasks.workunit.client.0.smithi125.stdout: create 190/.snap/snap-subdir-test 2023-10-30T12:31:34.323 INFO:tasks.workunit.client.0.smithi125.stdout: create 190/30 file after the snapshot 2023-10-30T12:31:34.324 INFO:tasks.workunit.client.0.smithi125.stdout: create 190/31 file after the snapshot 2023-10-30T12:31:34.324 INFO:tasks.workunit.client.0.smithi125.stdout: create 190/32 file after the snapshot 2023-10-30T12:31:34.325 INFO:tasks.workunit.client.0.smithi125.stdout: create 190/33 file after the snapshot 2023-10-30T12:31:34.325 INFO:tasks.workunit.client.0.smithi125.stdout: create 190/34 file after the snapshot 2023-10-30T12:31:34.326 INFO:tasks.workunit.client.0.smithi125.stdout: create 190/35 file after the snapshot 2023-10-30T12:31:34.326 INFO:tasks.workunit.client.0.smithi125.stdout: create 190/36 file after the snapshot 2023-10-30T12:31:34.326 INFO:tasks.workunit.client.0.smithi125.stdout: create 190/37 file after the snapshot 2023-10-30T12:31:34.326 INFO:tasks.workunit.client.0.smithi125.stdout: create 190/38 file after the snapshot 2023-10-30T12:31:34.327 INFO:tasks.workunit.client.0.smithi125.stdout: create 190/39 file after the snapshot 2023-10-30T12:31:34.327 INFO:tasks.workunit.client.0.smithi125.stdout: create 191/.snap/snap-subdir-test 2023-10-30T12:31:34.348 INFO:tasks.workunit.client.0.smithi125.stdout: create 191/30 file after the snapshot 2023-10-30T12:31:34.351 INFO:tasks.workunit.client.0.smithi125.stdout: create 191/31 file after the snapshot 2023-10-30T12:31:34.351 INFO:tasks.workunit.client.0.smithi125.stdout: create 191/32 file after the snapshot 2023-10-30T12:31:34.351 INFO:tasks.workunit.client.0.smithi125.stdout: create 191/33 file after the snapshot 2023-10-30T12:31:34.351 INFO:tasks.workunit.client.0.smithi125.stdout: create 191/34 file after the snapshot 2023-10-30T12:31:34.352 INFO:tasks.workunit.client.0.smithi125.stdout: create 191/35 file after the snapshot 2023-10-30T12:31:34.352 INFO:tasks.workunit.client.0.smithi125.stdout: create 191/36 file after the snapshot 2023-10-30T12:31:34.352 INFO:tasks.workunit.client.0.smithi125.stdout: create 191/37 file after the snapshot 2023-10-30T12:31:34.352 INFO:tasks.workunit.client.0.smithi125.stdout: create 191/38 file after the snapshot 2023-10-30T12:31:34.352 INFO:tasks.workunit.client.0.smithi125.stdout: create 191/39 file after the snapshot 2023-10-30T12:31:34.353 INFO:tasks.workunit.client.0.smithi125.stdout: create 192/.snap/snap-subdir-test 2023-10-30T12:31:34.380 INFO:tasks.workunit.client.0.smithi125.stdout: create 192/30 file after the snapshot 2023-10-30T12:31:34.382 INFO:tasks.workunit.client.0.smithi125.stdout: create 192/31 file after the snapshot 2023-10-30T12:31:34.382 INFO:tasks.workunit.client.0.smithi125.stdout: create 192/32 file after the snapshot 2023-10-30T12:31:34.382 INFO:tasks.workunit.client.0.smithi125.stdout: create 192/33 file after the snapshot 2023-10-30T12:31:34.382 INFO:tasks.workunit.client.0.smithi125.stdout: create 192/34 file after the snapshot 2023-10-30T12:31:34.383 INFO:tasks.workunit.client.0.smithi125.stdout: create 192/35 file after the snapshot 2023-10-30T12:31:34.383 INFO:tasks.workunit.client.0.smithi125.stdout: create 192/36 file after the snapshot 2023-10-30T12:31:34.383 INFO:tasks.workunit.client.0.smithi125.stdout: create 192/37 file after the snapshot 2023-10-30T12:31:34.383 INFO:tasks.workunit.client.0.smithi125.stdout: create 192/38 file after the snapshot 2023-10-30T12:31:34.383 INFO:tasks.workunit.client.0.smithi125.stdout: create 192/39 file after the snapshot 2023-10-30T12:31:34.384 INFO:tasks.workunit.client.0.smithi125.stdout: create 193/.snap/snap-subdir-test 2023-10-30T12:31:34.407 INFO:tasks.workunit.client.0.smithi125.stdout: create 193/30 file after the snapshot 2023-10-30T12:31:34.410 INFO:tasks.workunit.client.0.smithi125.stdout: create 193/31 file after the snapshot 2023-10-30T12:31:34.410 INFO:tasks.workunit.client.0.smithi125.stdout: create 193/32 file after the snapshot 2023-10-30T12:31:34.410 INFO:tasks.workunit.client.0.smithi125.stdout: create 193/33 file after the snapshot 2023-10-30T12:31:34.410 INFO:tasks.workunit.client.0.smithi125.stdout: create 193/34 file after the snapshot 2023-10-30T12:31:34.410 INFO:tasks.workunit.client.0.smithi125.stdout: create 193/35 file after the snapshot 2023-10-30T12:31:34.413 INFO:tasks.workunit.client.0.smithi125.stdout: create 193/36 file after the snapshot 2023-10-30T12:31:34.413 INFO:tasks.workunit.client.0.smithi125.stdout: create 193/37 file after the snapshot 2023-10-30T12:31:34.413 INFO:tasks.workunit.client.0.smithi125.stdout: create 193/38 file after the snapshot 2023-10-30T12:31:34.414 INFO:tasks.workunit.client.0.smithi125.stdout: create 193/39 file after the snapshot 2023-10-30T12:31:34.414 INFO:tasks.workunit.client.0.smithi125.stdout: create 194/.snap/snap-subdir-test 2023-10-30T12:31:34.424 INFO:tasks.workunit.client.0.smithi125.stdout: create 194/30 file after the snapshot 2023-10-30T12:31:34.425 INFO:tasks.workunit.client.0.smithi125.stdout: create 194/31 file after the snapshot 2023-10-30T12:31:34.425 INFO:tasks.workunit.client.0.smithi125.stdout: create 194/32 file after the snapshot 2023-10-30T12:31:34.425 INFO:tasks.workunit.client.0.smithi125.stdout: create 194/33 file after the snapshot 2023-10-30T12:31:34.426 INFO:tasks.workunit.client.0.smithi125.stdout: create 194/34 file after the snapshot 2023-10-30T12:31:34.427 INFO:tasks.workunit.client.0.smithi125.stdout: create 194/35 file after the snapshot 2023-10-30T12:31:34.427 INFO:tasks.workunit.client.0.smithi125.stdout: create 194/36 file after the snapshot 2023-10-30T12:31:34.427 INFO:tasks.workunit.client.0.smithi125.stdout: create 194/37 file after the snapshot 2023-10-30T12:31:34.427 INFO:tasks.workunit.client.0.smithi125.stdout: create 194/38 file after the snapshot 2023-10-30T12:31:34.427 INFO:tasks.workunit.client.0.smithi125.stdout: create 194/39 file after the snapshot 2023-10-30T12:31:34.428 INFO:tasks.workunit.client.0.smithi125.stdout: create 195/.snap/snap-subdir-test 2023-10-30T12:31:34.446 INFO:tasks.workunit.client.0.smithi125.stdout: create 195/30 file after the snapshot 2023-10-30T12:31:34.448 INFO:tasks.workunit.client.0.smithi125.stdout: create 195/31 file after the snapshot 2023-10-30T12:31:34.448 INFO:tasks.workunit.client.0.smithi125.stdout: create 195/32 file after the snapshot 2023-10-30T12:31:34.448 INFO:tasks.workunit.client.0.smithi125.stdout: create 195/33 file after the snapshot 2023-10-30T12:31:34.448 INFO:tasks.workunit.client.0.smithi125.stdout: create 195/34 file after the snapshot 2023-10-30T12:31:34.448 INFO:tasks.workunit.client.0.smithi125.stdout: create 195/35 file after the snapshot 2023-10-30T12:31:34.449 INFO:tasks.workunit.client.0.smithi125.stdout: create 195/36 file after the snapshot 2023-10-30T12:31:34.449 INFO:tasks.workunit.client.0.smithi125.stdout: create 195/37 file after the snapshot 2023-10-30T12:31:34.449 INFO:tasks.workunit.client.0.smithi125.stdout: create 195/38 file after the snapshot 2023-10-30T12:31:34.449 INFO:tasks.workunit.client.0.smithi125.stdout: create 195/39 file after the snapshot 2023-10-30T12:31:34.449 INFO:tasks.workunit.client.0.smithi125.stdout: create 196/.snap/snap-subdir-test 2023-10-30T12:31:34.471 INFO:tasks.workunit.client.0.smithi125.stdout: create 196/30 file after the snapshot 2023-10-30T12:31:34.473 INFO:tasks.workunit.client.0.smithi125.stdout: create 196/31 file after the snapshot 2023-10-30T12:31:34.473 INFO:tasks.workunit.client.0.smithi125.stdout: create 196/32 file after the snapshot 2023-10-30T12:31:34.473 INFO:tasks.workunit.client.0.smithi125.stdout: create 196/33 file after the snapshot 2023-10-30T12:31:34.473 INFO:tasks.workunit.client.0.smithi125.stdout: create 196/34 file after the snapshot 2023-10-30T12:31:34.474 INFO:tasks.workunit.client.0.smithi125.stdout: create 196/35 file after the snapshot 2023-10-30T12:31:34.474 INFO:tasks.workunit.client.0.smithi125.stdout: create 196/36 file after the snapshot 2023-10-30T12:31:34.474 INFO:tasks.workunit.client.0.smithi125.stdout: create 196/37 file after the snapshot 2023-10-30T12:31:34.474 INFO:tasks.workunit.client.0.smithi125.stdout: create 196/38 file after the snapshot 2023-10-30T12:31:34.474 INFO:tasks.workunit.client.0.smithi125.stdout: create 196/39 file after the snapshot 2023-10-30T12:31:34.475 INFO:tasks.workunit.client.0.smithi125.stdout: create 197/.snap/snap-subdir-test 2023-10-30T12:31:34.502 INFO:tasks.workunit.client.0.smithi125.stdout: create 197/30 file after the snapshot 2023-10-30T12:31:34.504 INFO:tasks.workunit.client.0.smithi125.stdout: create 197/31 file after the snapshot 2023-10-30T12:31:34.504 INFO:tasks.workunit.client.0.smithi125.stdout: create 197/32 file after the snapshot 2023-10-30T12:31:34.505 INFO:tasks.workunit.client.0.smithi125.stdout: create 197/33 file after the snapshot 2023-10-30T12:31:34.505 INFO:tasks.workunit.client.0.smithi125.stdout: create 197/34 file after the snapshot 2023-10-30T12:31:34.505 INFO:tasks.workunit.client.0.smithi125.stdout: create 197/35 file after the snapshot 2023-10-30T12:31:34.505 INFO:tasks.workunit.client.0.smithi125.stdout: create 197/36 file after the snapshot 2023-10-30T12:31:34.505 INFO:tasks.workunit.client.0.smithi125.stdout: create 197/37 file after the snapshot 2023-10-30T12:31:34.506 INFO:tasks.workunit.client.0.smithi125.stdout: create 197/38 file after the snapshot 2023-10-30T12:31:34.506 INFO:tasks.workunit.client.0.smithi125.stdout: create 197/39 file after the snapshot 2023-10-30T12:31:34.506 INFO:tasks.workunit.client.0.smithi125.stdout: create 198/.snap/snap-subdir-test 2023-10-30T12:31:34.520 INFO:tasks.workunit.client.0.smithi125.stdout: create 198/30 file after the snapshot 2023-10-30T12:31:34.522 INFO:tasks.workunit.client.0.smithi125.stdout: create 198/31 file after the snapshot 2023-10-30T12:31:34.522 INFO:tasks.workunit.client.0.smithi125.stdout: create 198/32 file after the snapshot 2023-10-30T12:31:34.523 INFO:tasks.workunit.client.0.smithi125.stdout: create 198/33 file after the snapshot 2023-10-30T12:31:34.523 INFO:tasks.workunit.client.0.smithi125.stdout: create 198/34 file after the snapshot 2023-10-30T12:31:34.523 INFO:tasks.workunit.client.0.smithi125.stdout: create 198/35 file after the snapshot 2023-10-30T12:31:34.523 INFO:tasks.workunit.client.0.smithi125.stdout: create 198/36 file after the snapshot 2023-10-30T12:31:34.523 INFO:tasks.workunit.client.0.smithi125.stdout: create 198/37 file after the snapshot 2023-10-30T12:31:34.524 INFO:tasks.workunit.client.0.smithi125.stdout: create 198/38 file after the snapshot 2023-10-30T12:31:34.524 INFO:tasks.workunit.client.0.smithi125.stdout: create 198/39 file after the snapshot 2023-10-30T12:31:34.524 INFO:tasks.workunit.client.0.smithi125.stdout: create 199/.snap/snap-subdir-test 2023-10-30T12:31:34.817 INFO:tasks.workunit.client.0.smithi125.stdout: create 199/30 file after the snapshot 2023-10-30T12:31:34.822 INFO:tasks.workunit.client.0.smithi125.stdout: create 199/31 file after the snapshot 2023-10-30T12:31:34.822 INFO:tasks.workunit.client.0.smithi125.stdout: create 199/32 file after the snapshot 2023-10-30T12:31:34.822 INFO:tasks.workunit.client.0.smithi125.stdout: create 199/33 file after the snapshot 2023-10-30T12:31:34.822 INFO:tasks.workunit.client.0.smithi125.stdout: create 199/34 file after the snapshot 2023-10-30T12:31:34.823 INFO:tasks.workunit.client.0.smithi125.stdout: create 199/35 file after the snapshot 2023-10-30T12:31:34.823 INFO:tasks.workunit.client.0.smithi125.stdout: create 199/36 file after the snapshot 2023-10-30T12:31:34.823 INFO:tasks.workunit.client.0.smithi125.stdout: create 199/37 file after the snapshot 2023-10-30T12:31:34.823 INFO:tasks.workunit.client.0.smithi125.stdout: create 199/38 file after the snapshot 2023-10-30T12:31:34.823 INFO:tasks.workunit.client.0.smithi125.stdout: create 199/39 file after the snapshot 2023-10-30T12:31:34.824 INFO:tasks.workunit.client.0.smithi125.stdout: create 200/.snap/snap-subdir-test 2023-10-30T12:31:34.838 INFO:tasks.workunit.client.0.smithi125.stdout: create 200/30 file after the snapshot 2023-10-30T12:31:34.839 INFO:tasks.workunit.client.0.smithi125.stdout: create 200/31 file after the snapshot 2023-10-30T12:31:34.840 INFO:tasks.workunit.client.0.smithi125.stdout: create 200/32 file after the snapshot 2023-10-30T12:31:34.840 INFO:tasks.workunit.client.0.smithi125.stdout: create 200/33 file after the snapshot 2023-10-30T12:31:34.840 INFO:tasks.workunit.client.0.smithi125.stdout: create 200/34 file after the snapshot 2023-10-30T12:31:34.840 INFO:tasks.workunit.client.0.smithi125.stdout: create 200/35 file after the snapshot 2023-10-30T12:31:34.840 INFO:tasks.workunit.client.0.smithi125.stdout: create 200/36 file after the snapshot 2023-10-30T12:31:34.841 INFO:tasks.workunit.client.0.smithi125.stdout: create 200/37 file after the snapshot 2023-10-30T12:31:34.841 INFO:tasks.workunit.client.0.smithi125.stdout: create 200/38 file after the snapshot 2023-10-30T12:31:34.841 INFO:tasks.workunit.client.0.smithi125.stdout: create 200/39 file after the snapshot 2023-10-30T12:31:34.841 INFO:tasks.workunit.client.0.smithi125.stdout: create 201/.snap/snap-subdir-test 2023-10-30T12:31:34.870 INFO:tasks.workunit.client.0.smithi125.stdout: create 201/30 file after the snapshot 2023-10-30T12:31:34.873 INFO:tasks.workunit.client.0.smithi125.stdout: create 201/31 file after the snapshot 2023-10-30T12:31:34.873 INFO:tasks.workunit.client.0.smithi125.stdout: create 201/32 file after the snapshot 2023-10-30T12:31:34.873 INFO:tasks.workunit.client.0.smithi125.stdout: create 201/33 file after the snapshot 2023-10-30T12:31:34.873 INFO:tasks.workunit.client.0.smithi125.stdout: create 201/34 file after the snapshot 2023-10-30T12:31:34.873 INFO:tasks.workunit.client.0.smithi125.stdout: create 201/35 file after the snapshot 2023-10-30T12:31:34.874 INFO:tasks.workunit.client.0.smithi125.stdout: create 201/36 file after the snapshot 2023-10-30T12:31:34.874 INFO:tasks.workunit.client.0.smithi125.stdout: create 201/37 file after the snapshot 2023-10-30T12:31:34.874 INFO:tasks.workunit.client.0.smithi125.stdout: create 201/38 file after the snapshot 2023-10-30T12:31:34.874 INFO:tasks.workunit.client.0.smithi125.stdout: create 201/39 file after the snapshot 2023-10-30T12:31:34.874 INFO:tasks.workunit.client.0.smithi125.stdout: create 202/.snap/snap-subdir-test 2023-10-30T12:31:34.889 INFO:tasks.workunit.client.0.smithi125.stdout: create 202/30 file after the snapshot 2023-10-30T12:31:34.891 INFO:tasks.workunit.client.0.smithi125.stdout: create 202/31 file after the snapshot 2023-10-30T12:31:34.891 INFO:tasks.workunit.client.0.smithi125.stdout: create 202/32 file after the snapshot 2023-10-30T12:31:34.891 INFO:tasks.workunit.client.0.smithi125.stdout: create 202/33 file after the snapshot 2023-10-30T12:31:34.891 INFO:tasks.workunit.client.0.smithi125.stdout: create 202/34 file after the snapshot 2023-10-30T12:31:34.891 INFO:tasks.workunit.client.0.smithi125.stdout: create 202/35 file after the snapshot 2023-10-30T12:31:34.892 INFO:tasks.workunit.client.0.smithi125.stdout: create 202/36 file after the snapshot 2023-10-30T12:31:34.892 INFO:tasks.workunit.client.0.smithi125.stdout: create 202/37 file after the snapshot 2023-10-30T12:31:34.892 INFO:tasks.workunit.client.0.smithi125.stdout: create 202/38 file after the snapshot 2023-10-30T12:31:34.892 INFO:tasks.workunit.client.0.smithi125.stdout: create 202/39 file after the snapshot 2023-10-30T12:31:34.892 INFO:tasks.workunit.client.0.smithi125.stdout: create 203/.snap/snap-subdir-test 2023-10-30T12:31:34.912 INFO:tasks.workunit.client.0.smithi125.stdout: create 203/30 file after the snapshot 2023-10-30T12:31:35.769 INFO:tasks.workunit.client.0.smithi125.stdout: create 203/31 file after the snapshot 2023-10-30T12:31:35.770 INFO:tasks.workunit.client.0.smithi125.stdout: create 203/32 file after the snapshot 2023-10-30T12:31:35.770 INFO:tasks.workunit.client.0.smithi125.stdout: create 203/33 file after the snapshot 2023-10-30T12:31:35.770 INFO:tasks.workunit.client.0.smithi125.stdout: create 203/34 file after the snapshot 2023-10-30T12:31:35.770 INFO:tasks.workunit.client.0.smithi125.stdout: create 203/35 file after the snapshot 2023-10-30T12:31:35.770 INFO:tasks.workunit.client.0.smithi125.stdout: create 203/36 file after the snapshot 2023-10-30T12:31:35.771 INFO:tasks.workunit.client.0.smithi125.stdout: create 203/37 file after the snapshot 2023-10-30T12:31:35.771 INFO:tasks.workunit.client.0.smithi125.stdout: create 203/38 file after the snapshot 2023-10-30T12:31:35.771 INFO:tasks.workunit.client.0.smithi125.stdout: create 203/39 file after the snapshot 2023-10-30T12:31:35.771 INFO:tasks.workunit.client.0.smithi125.stdout: create 204/.snap/snap-subdir-test 2023-10-30T12:31:35.792 INFO:tasks.workunit.client.0.smithi125.stdout: create 204/30 file after the snapshot 2023-10-30T12:31:35.794 INFO:tasks.workunit.client.0.smithi125.stdout: create 204/31 file after the snapshot 2023-10-30T12:31:35.794 INFO:tasks.workunit.client.0.smithi125.stdout: create 204/32 file after the snapshot 2023-10-30T12:31:35.794 INFO:tasks.workunit.client.0.smithi125.stdout: create 204/33 file after the snapshot 2023-10-30T12:31:35.795 INFO:tasks.workunit.client.0.smithi125.stdout: create 204/34 file after the snapshot 2023-10-30T12:31:35.795 INFO:tasks.workunit.client.0.smithi125.stdout: create 204/35 file after the snapshot 2023-10-30T12:31:35.795 INFO:tasks.workunit.client.0.smithi125.stdout: create 204/36 file after the snapshot 2023-10-30T12:31:35.795 INFO:tasks.workunit.client.0.smithi125.stdout: create 204/37 file after the snapshot 2023-10-30T12:31:35.795 INFO:tasks.workunit.client.0.smithi125.stdout: create 204/38 file after the snapshot 2023-10-30T12:31:35.796 INFO:tasks.workunit.client.0.smithi125.stdout: create 204/39 file after the snapshot 2023-10-30T12:31:35.796 INFO:tasks.workunit.client.0.smithi125.stdout: create 205/.snap/snap-subdir-test 2023-10-30T12:31:35.822 INFO:tasks.workunit.client.0.smithi125.stdout: create 205/30 file after the snapshot 2023-10-30T12:31:35.826 INFO:tasks.workunit.client.0.smithi125.stdout: create 205/31 file after the snapshot 2023-10-30T12:31:35.827 INFO:tasks.workunit.client.0.smithi125.stdout: create 205/32 file after the snapshot 2023-10-30T12:31:35.827 INFO:tasks.workunit.client.0.smithi125.stdout: create 205/33 file after the snapshot 2023-10-30T12:31:35.827 INFO:tasks.workunit.client.0.smithi125.stdout: create 205/34 file after the snapshot 2023-10-30T12:31:35.827 INFO:tasks.workunit.client.0.smithi125.stdout: create 205/35 file after the snapshot 2023-10-30T12:31:35.827 INFO:tasks.workunit.client.0.smithi125.stdout: create 205/36 file after the snapshot 2023-10-30T12:31:35.828 INFO:tasks.workunit.client.0.smithi125.stdout: create 205/37 file after the snapshot 2023-10-30T12:31:35.828 INFO:tasks.workunit.client.0.smithi125.stdout: create 205/38 file after the snapshot 2023-10-30T12:31:35.828 INFO:tasks.workunit.client.0.smithi125.stdout: create 205/39 file after the snapshot 2023-10-30T12:31:35.828 INFO:tasks.workunit.client.0.smithi125.stdout: create 206/.snap/snap-subdir-test 2023-10-30T12:31:36.046 INFO:tasks.workunit.client.0.smithi125.stdout: create 206/30 file after the snapshot 2023-10-30T12:31:36.047 INFO:tasks.workunit.client.0.smithi125.stdout: create 206/31 file after the snapshot 2023-10-30T12:31:36.047 INFO:tasks.workunit.client.0.smithi125.stdout: create 206/32 file after the snapshot 2023-10-30T12:31:36.047 INFO:tasks.workunit.client.0.smithi125.stdout: create 206/33 file after the snapshot 2023-10-30T12:31:36.047 INFO:tasks.workunit.client.0.smithi125.stdout: create 206/34 file after the snapshot 2023-10-30T12:31:36.048 INFO:tasks.workunit.client.0.smithi125.stdout: create 206/35 file after the snapshot 2023-10-30T12:31:36.048 INFO:tasks.workunit.client.0.smithi125.stdout: create 206/36 file after the snapshot 2023-10-30T12:31:36.048 INFO:tasks.workunit.client.0.smithi125.stdout: create 206/37 file after the snapshot 2023-10-30T12:31:36.048 INFO:tasks.workunit.client.0.smithi125.stdout: create 206/38 file after the snapshot 2023-10-30T12:31:36.048 INFO:tasks.workunit.client.0.smithi125.stdout: create 206/39 file after the snapshot 2023-10-30T12:31:36.049 INFO:tasks.workunit.client.0.smithi125.stdout: create 207/.snap/snap-subdir-test 2023-10-30T12:31:36.370 INFO:tasks.workunit.client.0.smithi125.stdout: create 207/30 file after the snapshot 2023-10-30T12:31:36.371 INFO:tasks.workunit.client.0.smithi125.stdout: create 207/31 file after the snapshot 2023-10-30T12:31:36.372 INFO:tasks.workunit.client.0.smithi125.stdout: create 207/32 file after the snapshot 2023-10-30T12:31:36.372 INFO:tasks.workunit.client.0.smithi125.stdout: create 207/33 file after the snapshot 2023-10-30T12:31:36.372 INFO:tasks.workunit.client.0.smithi125.stdout: create 207/34 file after the snapshot 2023-10-30T12:31:36.372 INFO:tasks.workunit.client.0.smithi125.stdout: create 207/35 file after the snapshot 2023-10-30T12:31:36.373 INFO:tasks.workunit.client.0.smithi125.stdout: create 207/36 file after the snapshot 2023-10-30T12:31:36.373 INFO:tasks.workunit.client.0.smithi125.stdout: create 207/37 file after the snapshot 2023-10-30T12:31:36.373 INFO:tasks.workunit.client.0.smithi125.stdout: create 207/38 file after the snapshot 2023-10-30T12:31:36.373 INFO:tasks.workunit.client.0.smithi125.stdout: create 207/39 file after the snapshot 2023-10-30T12:31:36.373 INFO:tasks.workunit.client.0.smithi125.stdout: create 208/.snap/snap-subdir-test 2023-10-30T12:31:36.394 INFO:tasks.workunit.client.0.smithi125.stdout: create 208/30 file after the snapshot 2023-10-30T12:31:36.396 INFO:tasks.workunit.client.0.smithi125.stdout: create 208/31 file after the snapshot 2023-10-30T12:31:36.396 INFO:tasks.workunit.client.0.smithi125.stdout: create 208/32 file after the snapshot 2023-10-30T12:31:36.396 INFO:tasks.workunit.client.0.smithi125.stdout: create 208/33 file after the snapshot 2023-10-30T12:31:36.396 INFO:tasks.workunit.client.0.smithi125.stdout: create 208/34 file after the snapshot 2023-10-30T12:31:36.396 INFO:tasks.workunit.client.0.smithi125.stdout: create 208/35 file after the snapshot 2023-10-30T12:31:36.397 INFO:tasks.workunit.client.0.smithi125.stdout: create 208/36 file after the snapshot 2023-10-30T12:31:36.397 INFO:tasks.workunit.client.0.smithi125.stdout: create 208/37 file after the snapshot 2023-10-30T12:31:36.397 INFO:tasks.workunit.client.0.smithi125.stdout: create 208/38 file after the snapshot 2023-10-30T12:31:36.397 INFO:tasks.workunit.client.0.smithi125.stdout: create 208/39 file after the snapshot 2023-10-30T12:31:36.397 INFO:tasks.workunit.client.0.smithi125.stdout: create 209/.snap/snap-subdir-test 2023-10-30T12:31:36.422 INFO:tasks.workunit.client.0.smithi125.stdout: create 209/30 file after the snapshot 2023-10-30T12:31:36.424 INFO:tasks.workunit.client.0.smithi125.stdout: create 209/31 file after the snapshot 2023-10-30T12:31:36.424 INFO:tasks.workunit.client.0.smithi125.stdout: create 209/32 file after the snapshot 2023-10-30T12:31:36.424 INFO:tasks.workunit.client.0.smithi125.stdout: create 209/33 file after the snapshot 2023-10-30T12:31:36.424 INFO:tasks.workunit.client.0.smithi125.stdout: create 209/34 file after the snapshot 2023-10-30T12:31:36.424 INFO:tasks.workunit.client.0.smithi125.stdout: create 209/35 file after the snapshot 2023-10-30T12:31:36.425 INFO:tasks.workunit.client.0.smithi125.stdout: create 209/36 file after the snapshot 2023-10-30T12:31:36.425 INFO:tasks.workunit.client.0.smithi125.stdout: create 209/37 file after the snapshot 2023-10-30T12:31:36.425 INFO:tasks.workunit.client.0.smithi125.stdout: create 209/38 file after the snapshot 2023-10-30T12:31:36.425 INFO:tasks.workunit.client.0.smithi125.stdout: create 209/39 file after the snapshot 2023-10-30T12:31:36.425 INFO:tasks.workunit.client.0.smithi125.stdout: create 210/.snap/snap-subdir-test 2023-10-30T12:31:36.445 INFO:tasks.workunit.client.0.smithi125.stdout: create 210/30 file after the snapshot 2023-10-30T12:31:36.447 INFO:tasks.workunit.client.0.smithi125.stdout: create 210/31 file after the snapshot 2023-10-30T12:31:36.449 INFO:tasks.workunit.client.0.smithi125.stdout: create 210/32 file after the snapshot 2023-10-30T12:31:36.449 INFO:tasks.workunit.client.0.smithi125.stdout: create 210/33 file after the snapshot 2023-10-30T12:31:36.450 INFO:tasks.workunit.client.0.smithi125.stdout: create 210/34 file after the snapshot 2023-10-30T12:31:36.450 INFO:tasks.workunit.client.0.smithi125.stdout: create 210/35 file after the snapshot 2023-10-30T12:31:36.450 INFO:tasks.workunit.client.0.smithi125.stdout: create 210/36 file after the snapshot 2023-10-30T12:31:36.450 INFO:tasks.workunit.client.0.smithi125.stdout: create 210/37 file after the snapshot 2023-10-30T12:31:36.450 INFO:tasks.workunit.client.0.smithi125.stdout: create 210/38 file after the snapshot 2023-10-30T12:31:36.451 INFO:tasks.workunit.client.0.smithi125.stdout: create 210/39 file after the snapshot 2023-10-30T12:31:36.451 INFO:tasks.workunit.client.0.smithi125.stdout: create 211/.snap/snap-subdir-test 2023-10-30T12:31:36.463 INFO:tasks.workunit.client.0.smithi125.stdout: create 211/30 file after the snapshot 2023-10-30T12:31:36.467 INFO:tasks.workunit.client.0.smithi125.stdout: create 211/31 file after the snapshot 2023-10-30T12:31:36.467 INFO:tasks.workunit.client.0.smithi125.stdout: create 211/32 file after the snapshot 2023-10-30T12:31:36.467 INFO:tasks.workunit.client.0.smithi125.stdout: create 211/33 file after the snapshot 2023-10-30T12:31:36.467 INFO:tasks.workunit.client.0.smithi125.stdout: create 211/34 file after the snapshot 2023-10-30T12:31:36.467 INFO:tasks.workunit.client.0.smithi125.stdout: create 211/35 file after the snapshot 2023-10-30T12:31:36.468 INFO:tasks.workunit.client.0.smithi125.stdout: create 211/36 file after the snapshot 2023-10-30T12:31:36.468 INFO:tasks.workunit.client.0.smithi125.stdout: create 211/37 file after the snapshot 2023-10-30T12:31:36.468 INFO:tasks.workunit.client.0.smithi125.stdout: create 211/38 file after the snapshot 2023-10-30T12:31:36.468 INFO:tasks.workunit.client.0.smithi125.stdout: create 211/39 file after the snapshot 2023-10-30T12:31:36.468 INFO:tasks.workunit.client.0.smithi125.stdout: create 212/.snap/snap-subdir-test 2023-10-30T12:31:38.082 INFO:tasks.workunit.client.0.smithi125.stdout: create 212/30 file after the snapshot 2023-10-30T12:31:38.084 INFO:tasks.workunit.client.0.smithi125.stdout: create 212/31 file after the snapshot 2023-10-30T12:31:38.085 INFO:tasks.workunit.client.0.smithi125.stdout: create 212/32 file after the snapshot 2023-10-30T12:31:38.085 INFO:tasks.workunit.client.0.smithi125.stdout: create 212/33 file after the snapshot 2023-10-30T12:31:38.085 INFO:tasks.workunit.client.0.smithi125.stdout: create 212/34 file after the snapshot 2023-10-30T12:31:38.085 INFO:tasks.workunit.client.0.smithi125.stdout: create 212/35 file after the snapshot 2023-10-30T12:31:38.085 INFO:tasks.workunit.client.0.smithi125.stdout: create 212/36 file after the snapshot 2023-10-30T12:31:38.086 INFO:tasks.workunit.client.0.smithi125.stdout: create 212/37 file after the snapshot 2023-10-30T12:31:38.086 INFO:tasks.workunit.client.0.smithi125.stdout: create 212/38 file after the snapshot 2023-10-30T12:31:38.086 INFO:tasks.workunit.client.0.smithi125.stdout: create 212/39 file after the snapshot 2023-10-30T12:31:38.086 INFO:tasks.workunit.client.0.smithi125.stdout: create 213/.snap/snap-subdir-test 2023-10-30T12:31:38.100 INFO:tasks.workunit.client.0.smithi125.stdout: create 213/30 file after the snapshot 2023-10-30T12:31:38.102 INFO:tasks.workunit.client.0.smithi125.stdout: create 213/31 file after the snapshot 2023-10-30T12:31:38.102 INFO:tasks.workunit.client.0.smithi125.stdout: create 213/32 file after the snapshot 2023-10-30T12:31:38.102 INFO:tasks.workunit.client.0.smithi125.stdout: create 213/33 file after the snapshot 2023-10-30T12:31:38.103 INFO:tasks.workunit.client.0.smithi125.stdout: create 213/34 file after the snapshot 2023-10-30T12:31:38.103 INFO:tasks.workunit.client.0.smithi125.stdout: create 213/35 file after the snapshot 2023-10-30T12:31:38.103 INFO:tasks.workunit.client.0.smithi125.stdout: create 213/36 file after the snapshot 2023-10-30T12:31:38.103 INFO:tasks.workunit.client.0.smithi125.stdout: create 213/37 file after the snapshot 2023-10-30T12:31:38.103 INFO:tasks.workunit.client.0.smithi125.stdout: create 213/38 file after the snapshot 2023-10-30T12:31:38.104 INFO:tasks.workunit.client.0.smithi125.stdout: create 213/39 file after the snapshot 2023-10-30T12:31:38.104 INFO:tasks.workunit.client.0.smithi125.stdout: create 214/.snap/snap-subdir-test 2023-10-30T12:31:38.127 INFO:tasks.workunit.client.0.smithi125.stdout: create 214/30 file after the snapshot 2023-10-30T12:31:38.129 INFO:tasks.workunit.client.0.smithi125.stdout: create 214/31 file after the snapshot 2023-10-30T12:31:38.129 INFO:tasks.workunit.client.0.smithi125.stdout: create 214/32 file after the snapshot 2023-10-30T12:31:38.129 INFO:tasks.workunit.client.0.smithi125.stdout: create 214/33 file after the snapshot 2023-10-30T12:31:38.129 INFO:tasks.workunit.client.0.smithi125.stdout: create 214/34 file after the snapshot 2023-10-30T12:31:38.130 INFO:tasks.workunit.client.0.smithi125.stdout: create 214/35 file after the snapshot 2023-10-30T12:31:38.130 INFO:tasks.workunit.client.0.smithi125.stdout: create 214/36 file after the snapshot 2023-10-30T12:31:38.130 INFO:tasks.workunit.client.0.smithi125.stdout: create 214/37 file after the snapshot 2023-10-30T12:31:38.130 INFO:tasks.workunit.client.0.smithi125.stdout: create 214/38 file after the snapshot 2023-10-30T12:31:38.130 INFO:tasks.workunit.client.0.smithi125.stdout: create 214/39 file after the snapshot 2023-10-30T12:31:38.131 INFO:tasks.workunit.client.0.smithi125.stdout: create 215/.snap/snap-subdir-test 2023-10-30T12:31:38.161 INFO:tasks.workunit.client.0.smithi125.stdout: create 215/30 file after the snapshot 2023-10-30T12:31:38.162 INFO:tasks.workunit.client.0.smithi125.stdout: create 215/31 file after the snapshot 2023-10-30T12:31:38.163 INFO:tasks.workunit.client.0.smithi125.stdout: create 215/32 file after the snapshot 2023-10-30T12:31:38.163 INFO:tasks.workunit.client.0.smithi125.stdout: create 215/33 file after the snapshot 2023-10-30T12:31:38.163 INFO:tasks.workunit.client.0.smithi125.stdout: create 215/34 file after the snapshot 2023-10-30T12:31:38.163 INFO:tasks.workunit.client.0.smithi125.stdout: create 215/35 file after the snapshot 2023-10-30T12:31:38.163 INFO:tasks.workunit.client.0.smithi125.stdout: create 215/36 file after the snapshot 2023-10-30T12:31:38.164 INFO:tasks.workunit.client.0.smithi125.stdout: create 215/37 file after the snapshot 2023-10-30T12:31:38.164 INFO:tasks.workunit.client.0.smithi125.stdout: create 215/38 file after the snapshot 2023-10-30T12:31:38.164 INFO:tasks.workunit.client.0.smithi125.stdout: create 215/39 file after the snapshot 2023-10-30T12:31:38.164 INFO:tasks.workunit.client.0.smithi125.stdout: create 216/.snap/snap-subdir-test 2023-10-30T12:31:38.186 INFO:tasks.workunit.client.0.smithi125.stdout: create 216/30 file after the snapshot 2023-10-30T12:31:38.187 INFO:tasks.workunit.client.0.smithi125.stdout: create 216/31 file after the snapshot 2023-10-30T12:31:38.187 INFO:tasks.workunit.client.0.smithi125.stdout: create 216/32 file after the snapshot 2023-10-30T12:31:38.188 INFO:tasks.workunit.client.0.smithi125.stdout: create 216/33 file after the snapshot 2023-10-30T12:31:38.188 INFO:tasks.workunit.client.0.smithi125.stdout: create 216/34 file after the snapshot 2023-10-30T12:31:38.188 INFO:tasks.workunit.client.0.smithi125.stdout: create 216/35 file after the snapshot 2023-10-30T12:31:38.188 INFO:tasks.workunit.client.0.smithi125.stdout: create 216/36 file after the snapshot 2023-10-30T12:31:38.188 INFO:tasks.workunit.client.0.smithi125.stdout: create 216/37 file after the snapshot 2023-10-30T12:31:38.189 INFO:tasks.workunit.client.0.smithi125.stdout: create 216/38 file after the snapshot 2023-10-30T12:31:38.189 INFO:tasks.workunit.client.0.smithi125.stdout: create 216/39 file after the snapshot 2023-10-30T12:31:38.189 INFO:tasks.workunit.client.0.smithi125.stdout: create 217/.snap/snap-subdir-test 2023-10-30T12:31:38.208 INFO:tasks.workunit.client.0.smithi125.stdout: create 217/30 file after the snapshot 2023-10-30T12:31:38.210 INFO:tasks.workunit.client.0.smithi125.stdout: create 217/31 file after the snapshot 2023-10-30T12:31:38.210 INFO:tasks.workunit.client.0.smithi125.stdout: create 217/32 file after the snapshot 2023-10-30T12:31:38.210 INFO:tasks.workunit.client.0.smithi125.stdout: create 217/33 file after the snapshot 2023-10-30T12:31:38.211 INFO:tasks.workunit.client.0.smithi125.stdout: create 217/34 file after the snapshot 2023-10-30T12:31:38.211 INFO:tasks.workunit.client.0.smithi125.stdout: create 217/35 file after the snapshot 2023-10-30T12:31:38.211 INFO:tasks.workunit.client.0.smithi125.stdout: create 217/36 file after the snapshot 2023-10-30T12:31:38.211 INFO:tasks.workunit.client.0.smithi125.stdout: create 217/37 file after the snapshot 2023-10-30T12:31:38.212 INFO:tasks.workunit.client.0.smithi125.stdout: create 217/38 file after the snapshot 2023-10-30T12:31:38.212 INFO:tasks.workunit.client.0.smithi125.stdout: create 217/39 file after the snapshot 2023-10-30T12:31:38.212 INFO:tasks.workunit.client.0.smithi125.stdout: create 218/.snap/snap-subdir-test 2023-10-30T12:31:38.231 INFO:tasks.workunit.client.0.smithi125.stdout: create 218/30 file after the snapshot 2023-10-30T12:31:38.233 INFO:tasks.workunit.client.0.smithi125.stdout: create 218/31 file after the snapshot 2023-10-30T12:31:38.233 INFO:tasks.workunit.client.0.smithi125.stdout: create 218/32 file after the snapshot 2023-10-30T12:31:38.233 INFO:tasks.workunit.client.0.smithi125.stdout: create 218/33 file after the snapshot 2023-10-30T12:31:38.233 INFO:tasks.workunit.client.0.smithi125.stdout: create 218/34 file after the snapshot 2023-10-30T12:31:38.234 INFO:tasks.workunit.client.0.smithi125.stdout: create 218/35 file after the snapshot 2023-10-30T12:31:38.234 INFO:tasks.workunit.client.0.smithi125.stdout: create 218/36 file after the snapshot 2023-10-30T12:31:38.234 INFO:tasks.workunit.client.0.smithi125.stdout: create 218/37 file after the snapshot 2023-10-30T12:31:38.234 INFO:tasks.workunit.client.0.smithi125.stdout: create 218/38 file after the snapshot 2023-10-30T12:31:38.234 INFO:tasks.workunit.client.0.smithi125.stdout: create 218/39 file after the snapshot 2023-10-30T12:31:38.235 INFO:tasks.workunit.client.0.smithi125.stdout: create 219/.snap/snap-subdir-test 2023-10-30T12:31:38.247 INFO:tasks.workunit.client.0.smithi125.stdout: create 219/30 file after the snapshot 2023-10-30T12:31:38.249 INFO:tasks.workunit.client.0.smithi125.stdout: create 219/31 file after the snapshot 2023-10-30T12:31:38.250 INFO:tasks.workunit.client.0.smithi125.stdout: create 219/32 file after the snapshot 2023-10-30T12:31:38.250 INFO:tasks.workunit.client.0.smithi125.stdout: create 219/33 file after the snapshot 2023-10-30T12:31:38.250 INFO:tasks.workunit.client.0.smithi125.stdout: create 219/34 file after the snapshot 2023-10-30T12:31:38.250 INFO:tasks.workunit.client.0.smithi125.stdout: create 219/35 file after the snapshot 2023-10-30T12:31:38.251 INFO:tasks.workunit.client.0.smithi125.stdout: create 219/36 file after the snapshot 2023-10-30T12:31:38.251 INFO:tasks.workunit.client.0.smithi125.stdout: create 219/37 file after the snapshot 2023-10-30T12:31:38.251 INFO:tasks.workunit.client.0.smithi125.stdout: create 219/38 file after the snapshot 2023-10-30T12:31:38.251 INFO:tasks.workunit.client.0.smithi125.stdout: create 219/39 file after the snapshot 2023-10-30T12:31:38.251 INFO:tasks.workunit.client.0.smithi125.stdout: create 220/.snap/snap-subdir-test 2023-10-30T12:31:38.273 INFO:tasks.workunit.client.0.smithi125.stdout: create 220/30 file after the snapshot 2023-10-30T12:31:38.275 INFO:tasks.workunit.client.0.smithi125.stdout: create 220/31 file after the snapshot 2023-10-30T12:31:38.275 INFO:tasks.workunit.client.0.smithi125.stdout: create 220/32 file after the snapshot 2023-10-30T12:31:38.276 INFO:tasks.workunit.client.0.smithi125.stdout: create 220/33 file after the snapshot 2023-10-30T12:31:38.276 INFO:tasks.workunit.client.0.smithi125.stdout: create 220/34 file after the snapshot 2023-10-30T12:31:38.276 INFO:tasks.workunit.client.0.smithi125.stdout: create 220/35 file after the snapshot 2023-10-30T12:31:38.276 INFO:tasks.workunit.client.0.smithi125.stdout: create 220/36 file after the snapshot 2023-10-30T12:31:38.276 INFO:tasks.workunit.client.0.smithi125.stdout: create 220/37 file after the snapshot 2023-10-30T12:31:38.277 INFO:tasks.workunit.client.0.smithi125.stdout: create 220/38 file after the snapshot 2023-10-30T12:31:38.277 INFO:tasks.workunit.client.0.smithi125.stdout: create 220/39 file after the snapshot 2023-10-30T12:31:38.277 INFO:tasks.workunit.client.0.smithi125.stdout: create 221/.snap/snap-subdir-test 2023-10-30T12:31:38.528 INFO:tasks.workunit.client.0.smithi125.stdout: create 221/30 file after the snapshot 2023-10-30T12:31:38.666 INFO:tasks.workunit.client.0.smithi125.stdout: create 221/31 file after the snapshot 2023-10-30T12:31:38.666 INFO:tasks.workunit.client.0.smithi125.stdout: create 221/32 file after the snapshot 2023-10-30T12:31:38.667 INFO:tasks.workunit.client.0.smithi125.stdout: create 221/33 file after the snapshot 2023-10-30T12:31:38.667 INFO:tasks.workunit.client.0.smithi125.stdout: create 221/34 file after the snapshot 2023-10-30T12:31:38.667 INFO:tasks.workunit.client.0.smithi125.stdout: create 221/35 file after the snapshot 2023-10-30T12:31:38.667 INFO:tasks.workunit.client.0.smithi125.stdout: create 221/36 file after the snapshot 2023-10-30T12:31:38.667 INFO:tasks.workunit.client.0.smithi125.stdout: create 221/37 file after the snapshot 2023-10-30T12:31:38.668 INFO:tasks.workunit.client.0.smithi125.stdout: create 221/38 file after the snapshot 2023-10-30T12:31:38.668 INFO:tasks.workunit.client.0.smithi125.stdout: create 221/39 file after the snapshot 2023-10-30T12:31:38.668 INFO:tasks.workunit.client.0.smithi125.stdout: create 222/.snap/snap-subdir-test 2023-10-30T12:31:39.680 INFO:tasks.workunit.client.0.smithi125.stdout: create 222/30 file after the snapshot 2023-10-30T12:31:39.682 INFO:tasks.workunit.client.0.smithi125.stdout: create 222/31 file after the snapshot 2023-10-30T12:31:39.682 INFO:tasks.workunit.client.0.smithi125.stdout: create 222/32 file after the snapshot 2023-10-30T12:31:39.682 INFO:tasks.workunit.client.0.smithi125.stdout: create 222/33 file after the snapshot 2023-10-30T12:31:39.683 INFO:tasks.workunit.client.0.smithi125.stdout: create 222/34 file after the snapshot 2023-10-30T12:31:39.683 INFO:tasks.workunit.client.0.smithi125.stdout: create 222/35 file after the snapshot 2023-10-30T12:31:39.683 INFO:tasks.workunit.client.0.smithi125.stdout: create 222/36 file after the snapshot 2023-10-30T12:31:39.683 INFO:tasks.workunit.client.0.smithi125.stdout: create 222/37 file after the snapshot 2023-10-30T12:31:39.683 INFO:tasks.workunit.client.0.smithi125.stdout: create 222/38 file after the snapshot 2023-10-30T12:31:39.683 INFO:tasks.workunit.client.0.smithi125.stdout: create 222/39 file after the snapshot 2023-10-30T12:31:39.684 INFO:tasks.workunit.client.0.smithi125.stdout: create 223/.snap/snap-subdir-test 2023-10-30T12:31:39.961 INFO:tasks.workunit.client.0.smithi125.stdout: create 223/30 file after the snapshot 2023-10-30T12:31:39.963 INFO:tasks.workunit.client.0.smithi125.stdout: create 223/31 file after the snapshot 2023-10-30T12:31:39.963 INFO:tasks.workunit.client.0.smithi125.stdout: create 223/32 file after the snapshot 2023-10-30T12:31:39.963 INFO:tasks.workunit.client.0.smithi125.stdout: create 223/33 file after the snapshot 2023-10-30T12:31:39.964 INFO:tasks.workunit.client.0.smithi125.stdout: create 223/34 file after the snapshot 2023-10-30T12:31:39.964 INFO:tasks.workunit.client.0.smithi125.stdout: create 223/35 file after the snapshot 2023-10-30T12:31:39.964 INFO:tasks.workunit.client.0.smithi125.stdout: create 223/36 file after the snapshot 2023-10-30T12:31:39.964 INFO:tasks.workunit.client.0.smithi125.stdout: create 223/37 file after the snapshot 2023-10-30T12:31:39.964 INFO:tasks.workunit.client.0.smithi125.stdout: create 223/38 file after the snapshot 2023-10-30T12:31:39.965 INFO:tasks.workunit.client.0.smithi125.stdout: create 223/39 file after the snapshot 2023-10-30T12:31:39.965 INFO:tasks.workunit.client.0.smithi125.stdout: create 224/.snap/snap-subdir-test 2023-10-30T12:31:39.993 INFO:tasks.workunit.client.0.smithi125.stdout: create 224/30 file after the snapshot 2023-10-30T12:31:39.995 INFO:tasks.workunit.client.0.smithi125.stdout: create 224/31 file after the snapshot 2023-10-30T12:31:39.995 INFO:tasks.workunit.client.0.smithi125.stdout: create 224/32 file after the snapshot 2023-10-30T12:31:39.995 INFO:tasks.workunit.client.0.smithi125.stdout: create 224/33 file after the snapshot 2023-10-30T12:31:39.995 INFO:tasks.workunit.client.0.smithi125.stdout: create 224/34 file after the snapshot 2023-10-30T12:31:39.996 INFO:tasks.workunit.client.0.smithi125.stdout: create 224/35 file after the snapshot 2023-10-30T12:31:39.996 INFO:tasks.workunit.client.0.smithi125.stdout: create 224/36 file after the snapshot 2023-10-30T12:31:39.996 INFO:tasks.workunit.client.0.smithi125.stdout: create 224/37 file after the snapshot 2023-10-30T12:31:39.996 INFO:tasks.workunit.client.0.smithi125.stdout: create 224/38 file after the snapshot 2023-10-30T12:31:39.996 INFO:tasks.workunit.client.0.smithi125.stdout: create 224/39 file after the snapshot 2023-10-30T12:31:39.997 INFO:tasks.workunit.client.0.smithi125.stdout: create 225/.snap/snap-subdir-test 2023-10-30T12:31:40.023 INFO:tasks.workunit.client.0.smithi125.stdout: create 225/30 file after the snapshot 2023-10-30T12:31:40.024 INFO:tasks.workunit.client.0.smithi125.stdout: create 225/31 file after the snapshot 2023-10-30T12:31:40.024 INFO:tasks.workunit.client.0.smithi125.stdout: create 225/32 file after the snapshot 2023-10-30T12:31:40.025 INFO:tasks.workunit.client.0.smithi125.stdout: create 225/33 file after the snapshot 2023-10-30T12:31:40.025 INFO:tasks.workunit.client.0.smithi125.stdout: create 225/34 file after the snapshot 2023-10-30T12:31:40.025 INFO:tasks.workunit.client.0.smithi125.stdout: create 225/35 file after the snapshot 2023-10-30T12:31:40.025 INFO:tasks.workunit.client.0.smithi125.stdout: create 225/36 file after the snapshot 2023-10-30T12:31:40.025 INFO:tasks.workunit.client.0.smithi125.stdout: create 225/37 file after the snapshot 2023-10-30T12:31:40.026 INFO:tasks.workunit.client.0.smithi125.stdout: create 225/38 file after the snapshot 2023-10-30T12:31:40.026 INFO:tasks.workunit.client.0.smithi125.stdout: create 225/39 file after the snapshot 2023-10-30T12:31:40.026 INFO:tasks.workunit.client.0.smithi125.stdout: create 226/.snap/snap-subdir-test 2023-10-30T12:31:40.048 INFO:tasks.workunit.client.0.smithi125.stdout: create 226/30 file after the snapshot 2023-10-30T12:31:40.050 INFO:tasks.workunit.client.0.smithi125.stdout: create 226/31 file after the snapshot 2023-10-30T12:31:40.050 INFO:tasks.workunit.client.0.smithi125.stdout: create 226/32 file after the snapshot 2023-10-30T12:31:40.050 INFO:tasks.workunit.client.0.smithi125.stdout: create 226/33 file after the snapshot 2023-10-30T12:31:40.051 INFO:tasks.workunit.client.0.smithi125.stdout: create 226/34 file after the snapshot 2023-10-30T12:31:40.051 INFO:tasks.workunit.client.0.smithi125.stdout: create 226/35 file after the snapshot 2023-10-30T12:31:40.051 INFO:tasks.workunit.client.0.smithi125.stdout: create 226/36 file after the snapshot 2023-10-30T12:31:40.051 INFO:tasks.workunit.client.0.smithi125.stdout: create 226/37 file after the snapshot 2023-10-30T12:31:40.051 INFO:tasks.workunit.client.0.smithi125.stdout: create 226/38 file after the snapshot 2023-10-30T12:31:40.051 INFO:tasks.workunit.client.0.smithi125.stdout: create 226/39 file after the snapshot 2023-10-30T12:31:40.052 INFO:tasks.workunit.client.0.smithi125.stdout: create 227/.snap/snap-subdir-test 2023-10-30T12:31:40.076 INFO:tasks.workunit.client.0.smithi125.stdout: create 227/30 file after the snapshot 2023-10-30T12:31:40.077 INFO:tasks.workunit.client.0.smithi125.stdout: create 227/31 file after the snapshot 2023-10-30T12:31:40.078 INFO:tasks.workunit.client.0.smithi125.stdout: create 227/32 file after the snapshot 2023-10-30T12:31:40.078 INFO:tasks.workunit.client.0.smithi125.stdout: create 227/33 file after the snapshot 2023-10-30T12:31:40.078 INFO:tasks.workunit.client.0.smithi125.stdout: create 227/34 file after the snapshot 2023-10-30T12:31:40.079 INFO:tasks.workunit.client.0.smithi125.stdout: create 227/35 file after the snapshot 2023-10-30T12:31:40.079 INFO:tasks.workunit.client.0.smithi125.stdout: create 227/36 file after the snapshot 2023-10-30T12:31:40.079 INFO:tasks.workunit.client.0.smithi125.stdout: create 227/37 file after the snapshot 2023-10-30T12:31:40.079 INFO:tasks.workunit.client.0.smithi125.stdout: create 227/38 file after the snapshot 2023-10-30T12:31:40.080 INFO:tasks.workunit.client.0.smithi125.stdout: create 227/39 file after the snapshot 2023-10-30T12:31:40.080 INFO:tasks.workunit.client.0.smithi125.stdout: create 228/.snap/snap-subdir-test 2023-10-30T12:31:40.176 INFO:tasks.workunit.client.0.smithi125.stdout: create 228/30 file after the snapshot 2023-10-30T12:31:40.178 INFO:tasks.workunit.client.0.smithi125.stdout: create 228/31 file after the snapshot 2023-10-30T12:31:40.178 INFO:tasks.workunit.client.0.smithi125.stdout: create 228/32 file after the snapshot 2023-10-30T12:31:40.178 INFO:tasks.workunit.client.0.smithi125.stdout: create 228/33 file after the snapshot 2023-10-30T12:31:40.179 INFO:tasks.workunit.client.0.smithi125.stdout: create 228/34 file after the snapshot 2023-10-30T12:31:40.179 INFO:tasks.workunit.client.0.smithi125.stdout: create 228/35 file after the snapshot 2023-10-30T12:31:40.179 INFO:tasks.workunit.client.0.smithi125.stdout: create 228/36 file after the snapshot 2023-10-30T12:31:40.179 INFO:tasks.workunit.client.0.smithi125.stdout: create 228/37 file after the snapshot 2023-10-30T12:31:40.180 INFO:tasks.workunit.client.0.smithi125.stdout: create 228/38 file after the snapshot 2023-10-30T12:31:40.180 INFO:tasks.workunit.client.0.smithi125.stdout: create 228/39 file after the snapshot 2023-10-30T12:31:40.180 INFO:tasks.workunit.client.0.smithi125.stdout: create 229/.snap/snap-subdir-test 2023-10-30T12:31:40.964 INFO:tasks.workunit.client.0.smithi125.stdout: create 229/30 file after the snapshot 2023-10-30T12:31:40.966 INFO:tasks.workunit.client.0.smithi125.stdout: create 229/31 file after the snapshot 2023-10-30T12:31:40.966 INFO:tasks.workunit.client.0.smithi125.stdout: create 229/32 file after the snapshot 2023-10-30T12:31:40.966 INFO:tasks.workunit.client.0.smithi125.stdout: create 229/33 file after the snapshot 2023-10-30T12:31:40.967 INFO:tasks.workunit.client.0.smithi125.stdout: create 229/34 file after the snapshot 2023-10-30T12:31:40.967 INFO:tasks.workunit.client.0.smithi125.stdout: create 229/35 file after the snapshot 2023-10-30T12:31:40.967 INFO:tasks.workunit.client.0.smithi125.stdout: create 229/36 file after the snapshot 2023-10-30T12:31:40.967 INFO:tasks.workunit.client.0.smithi125.stdout: create 229/37 file after the snapshot 2023-10-30T12:31:40.967 INFO:tasks.workunit.client.0.smithi125.stdout: create 229/38 file after the snapshot 2023-10-30T12:31:40.968 INFO:tasks.workunit.client.0.smithi125.stdout: create 229/39 file after the snapshot 2023-10-30T12:31:40.968 INFO:tasks.workunit.client.0.smithi125.stdout: create 230/.snap/snap-subdir-test 2023-10-30T12:31:40.983 INFO:tasks.workunit.client.0.smithi125.stdout: create 230/30 file after the snapshot 2023-10-30T12:31:40.985 INFO:tasks.workunit.client.0.smithi125.stdout: create 230/31 file after the snapshot 2023-10-30T12:31:40.986 INFO:tasks.workunit.client.0.smithi125.stdout: create 230/32 file after the snapshot 2023-10-30T12:31:40.986 INFO:tasks.workunit.client.0.smithi125.stdout: create 230/33 file after the snapshot 2023-10-30T12:31:40.986 INFO:tasks.workunit.client.0.smithi125.stdout: create 230/34 file after the snapshot 2023-10-30T12:31:40.986 INFO:tasks.workunit.client.0.smithi125.stdout: create 230/35 file after the snapshot 2023-10-30T12:31:40.986 INFO:tasks.workunit.client.0.smithi125.stdout: create 230/36 file after the snapshot 2023-10-30T12:31:40.987 INFO:tasks.workunit.client.0.smithi125.stdout: create 230/37 file after the snapshot 2023-10-30T12:31:40.987 INFO:tasks.workunit.client.0.smithi125.stdout: create 230/38 file after the snapshot 2023-10-30T12:31:40.987 INFO:tasks.workunit.client.0.smithi125.stdout: create 230/39 file after the snapshot 2023-10-30T12:31:40.987 INFO:tasks.workunit.client.0.smithi125.stdout: create 231/.snap/snap-subdir-test 2023-10-30T12:31:41.246 INFO:tasks.workunit.client.0.smithi125.stdout: create 231/30 file after the snapshot 2023-10-30T12:31:41.247 INFO:tasks.workunit.client.0.smithi125.stdout: create 231/31 file after the snapshot 2023-10-30T12:31:41.248 INFO:tasks.workunit.client.0.smithi125.stdout: create 231/32 file after the snapshot 2023-10-30T12:31:41.248 INFO:tasks.workunit.client.0.smithi125.stdout: create 231/33 file after the snapshot 2023-10-30T12:31:41.248 INFO:tasks.workunit.client.0.smithi125.stdout: create 231/34 file after the snapshot 2023-10-30T12:31:41.248 INFO:tasks.workunit.client.0.smithi125.stdout: create 231/35 file after the snapshot 2023-10-30T12:31:41.248 INFO:tasks.workunit.client.0.smithi125.stdout: create 231/36 file after the snapshot 2023-10-30T12:31:41.249 INFO:tasks.workunit.client.0.smithi125.stdout: create 231/37 file after the snapshot 2023-10-30T12:31:41.249 INFO:tasks.workunit.client.0.smithi125.stdout: create 231/38 file after the snapshot 2023-10-30T12:31:41.249 INFO:tasks.workunit.client.0.smithi125.stdout: create 231/39 file after the snapshot 2023-10-30T12:31:41.249 INFO:tasks.workunit.client.0.smithi125.stdout: create 232/.snap/snap-subdir-test 2023-10-30T12:31:41.264 INFO:tasks.workunit.client.0.smithi125.stdout: create 232/30 file after the snapshot 2023-10-30T12:31:41.265 INFO:tasks.workunit.client.0.smithi125.stdout: create 232/31 file after the snapshot 2023-10-30T12:31:41.265 INFO:tasks.workunit.client.0.smithi125.stdout: create 232/32 file after the snapshot 2023-10-30T12:31:41.266 INFO:tasks.workunit.client.0.smithi125.stdout: create 232/33 file after the snapshot 2023-10-30T12:31:41.266 INFO:tasks.workunit.client.0.smithi125.stdout: create 232/34 file after the snapshot 2023-10-30T12:31:41.266 INFO:tasks.workunit.client.0.smithi125.stdout: create 232/35 file after the snapshot 2023-10-30T12:31:41.266 INFO:tasks.workunit.client.0.smithi125.stdout: create 232/36 file after the snapshot 2023-10-30T12:31:41.266 INFO:tasks.workunit.client.0.smithi125.stdout: create 232/37 file after the snapshot 2023-10-30T12:31:41.267 INFO:tasks.workunit.client.0.smithi125.stdout: create 232/38 file after the snapshot 2023-10-30T12:31:41.267 INFO:tasks.workunit.client.0.smithi125.stdout: create 232/39 file after the snapshot 2023-10-30T12:31:41.267 INFO:tasks.workunit.client.0.smithi125.stdout: create 233/.snap/snap-subdir-test 2023-10-30T12:31:41.289 INFO:tasks.workunit.client.0.smithi125.stdout: create 233/30 file after the snapshot 2023-10-30T12:31:41.291 INFO:tasks.workunit.client.0.smithi125.stdout: create 233/31 file after the snapshot 2023-10-30T12:31:41.291 INFO:tasks.workunit.client.0.smithi125.stdout: create 233/32 file after the snapshot 2023-10-30T12:31:41.291 INFO:tasks.workunit.client.0.smithi125.stdout: create 233/33 file after the snapshot 2023-10-30T12:31:41.292 INFO:tasks.workunit.client.0.smithi125.stdout: create 233/34 file after the snapshot 2023-10-30T12:31:41.292 INFO:tasks.workunit.client.0.smithi125.stdout: create 233/35 file after the snapshot 2023-10-30T12:31:41.292 INFO:tasks.workunit.client.0.smithi125.stdout: create 233/36 file after the snapshot 2023-10-30T12:31:41.292 INFO:tasks.workunit.client.0.smithi125.stdout: create 233/37 file after the snapshot 2023-10-30T12:31:41.293 INFO:tasks.workunit.client.0.smithi125.stdout: create 233/38 file after the snapshot 2023-10-30T12:31:41.293 INFO:tasks.workunit.client.0.smithi125.stdout: create 233/39 file after the snapshot 2023-10-30T12:31:41.293 INFO:tasks.workunit.client.0.smithi125.stdout: create 234/.snap/snap-subdir-test 2023-10-30T12:31:41.315 INFO:tasks.workunit.client.0.smithi125.stdout: create 234/30 file after the snapshot 2023-10-30T12:31:41.316 INFO:tasks.workunit.client.0.smithi125.stdout: create 234/31 file after the snapshot 2023-10-30T12:31:41.317 INFO:tasks.workunit.client.0.smithi125.stdout: create 234/32 file after the snapshot 2023-10-30T12:31:41.317 INFO:tasks.workunit.client.0.smithi125.stdout: create 234/33 file after the snapshot 2023-10-30T12:31:41.317 INFO:tasks.workunit.client.0.smithi125.stdout: create 234/34 file after the snapshot 2023-10-30T12:31:41.317 INFO:tasks.workunit.client.0.smithi125.stdout: create 234/35 file after the snapshot 2023-10-30T12:31:41.317 INFO:tasks.workunit.client.0.smithi125.stdout: create 234/36 file after the snapshot 2023-10-30T12:31:41.318 INFO:tasks.workunit.client.0.smithi125.stdout: create 234/37 file after the snapshot 2023-10-30T12:31:41.318 INFO:tasks.workunit.client.0.smithi125.stdout: create 234/38 file after the snapshot 2023-10-30T12:31:41.318 INFO:tasks.workunit.client.0.smithi125.stdout: create 234/39 file after the snapshot 2023-10-30T12:31:41.318 INFO:tasks.workunit.client.0.smithi125.stdout: create 235/.snap/snap-subdir-test 2023-10-30T12:31:41.344 INFO:tasks.workunit.client.0.smithi125.stdout: create 235/30 file after the snapshot 2023-10-30T12:31:41.346 INFO:tasks.workunit.client.0.smithi125.stdout: create 235/31 file after the snapshot 2023-10-30T12:31:41.346 INFO:tasks.workunit.client.0.smithi125.stdout: create 235/32 file after the snapshot 2023-10-30T12:31:41.346 INFO:tasks.workunit.client.0.smithi125.stdout: create 235/33 file after the snapshot 2023-10-30T12:31:41.346 INFO:tasks.workunit.client.0.smithi125.stdout: create 235/34 file after the snapshot 2023-10-30T12:31:41.347 INFO:tasks.workunit.client.0.smithi125.stdout: create 235/35 file after the snapshot 2023-10-30T12:31:41.347 INFO:tasks.workunit.client.0.smithi125.stdout: create 235/36 file after the snapshot 2023-10-30T12:31:41.347 INFO:tasks.workunit.client.0.smithi125.stdout: create 235/37 file after the snapshot 2023-10-30T12:31:41.347 INFO:tasks.workunit.client.0.smithi125.stdout: create 235/38 file after the snapshot 2023-10-30T12:31:41.347 INFO:tasks.workunit.client.0.smithi125.stdout: create 235/39 file after the snapshot 2023-10-30T12:31:41.348 INFO:tasks.workunit.client.0.smithi125.stdout: create 236/.snap/snap-subdir-test 2023-10-30T12:31:41.362 INFO:tasks.workunit.client.0.smithi125.stdout: create 236/30 file after the snapshot 2023-10-30T12:31:41.365 INFO:tasks.workunit.client.0.smithi125.stdout: create 236/31 file after the snapshot 2023-10-30T12:31:41.365 INFO:tasks.workunit.client.0.smithi125.stdout: create 236/32 file after the snapshot 2023-10-30T12:31:41.365 INFO:tasks.workunit.client.0.smithi125.stdout: create 236/33 file after the snapshot 2023-10-30T12:31:41.365 INFO:tasks.workunit.client.0.smithi125.stdout: create 236/34 file after the snapshot 2023-10-30T12:31:41.366 INFO:tasks.workunit.client.0.smithi125.stdout: create 236/35 file after the snapshot 2023-10-30T12:31:41.366 INFO:tasks.workunit.client.0.smithi125.stdout: create 236/36 file after the snapshot 2023-10-30T12:31:41.366 INFO:tasks.workunit.client.0.smithi125.stdout: create 236/37 file after the snapshot 2023-10-30T12:31:41.366 INFO:tasks.workunit.client.0.smithi125.stdout: create 236/38 file after the snapshot 2023-10-30T12:31:41.366 INFO:tasks.workunit.client.0.smithi125.stdout: create 236/39 file after the snapshot 2023-10-30T12:31:41.367 INFO:tasks.workunit.client.0.smithi125.stdout: create 237/.snap/snap-subdir-test 2023-10-30T12:31:41.628 INFO:tasks.workunit.client.0.smithi125.stdout: create 237/30 file after the snapshot 2023-10-30T12:31:41.629 INFO:tasks.workunit.client.0.smithi125.stdout: create 237/31 file after the snapshot 2023-10-30T12:31:41.629 INFO:tasks.workunit.client.0.smithi125.stdout: create 237/32 file after the snapshot 2023-10-30T12:31:41.630 INFO:tasks.workunit.client.0.smithi125.stdout: create 237/33 file after the snapshot 2023-10-30T12:31:41.630 INFO:tasks.workunit.client.0.smithi125.stdout: create 237/34 file after the snapshot 2023-10-30T12:31:41.630 INFO:tasks.workunit.client.0.smithi125.stdout: create 237/35 file after the snapshot 2023-10-30T12:31:41.630 INFO:tasks.workunit.client.0.smithi125.stdout: create 237/36 file after the snapshot 2023-10-30T12:31:41.630 INFO:tasks.workunit.client.0.smithi125.stdout: create 237/37 file after the snapshot 2023-10-30T12:31:41.631 INFO:tasks.workunit.client.0.smithi125.stdout: create 237/38 file after the snapshot 2023-10-30T12:31:41.631 INFO:tasks.workunit.client.0.smithi125.stdout: create 237/39 file after the snapshot 2023-10-30T12:31:41.631 INFO:tasks.workunit.client.0.smithi125.stdout: create 238/.snap/snap-subdir-test 2023-10-30T12:31:41.655 INFO:tasks.workunit.client.0.smithi125.stdout: create 238/30 file after the snapshot 2023-10-30T12:31:41.656 INFO:tasks.workunit.client.0.smithi125.stdout: create 238/31 file after the snapshot 2023-10-30T12:31:41.657 INFO:tasks.workunit.client.0.smithi125.stdout: create 238/32 file after the snapshot 2023-10-30T12:31:41.657 INFO:tasks.workunit.client.0.smithi125.stdout: create 238/33 file after the snapshot 2023-10-30T12:31:41.657 INFO:tasks.workunit.client.0.smithi125.stdout: create 238/34 file after the snapshot 2023-10-30T12:31:41.657 INFO:tasks.workunit.client.0.smithi125.stdout: create 238/35 file after the snapshot 2023-10-30T12:31:41.657 INFO:tasks.workunit.client.0.smithi125.stdout: create 238/36 file after the snapshot 2023-10-30T12:31:41.657 INFO:tasks.workunit.client.0.smithi125.stdout: create 238/37 file after the snapshot 2023-10-30T12:31:41.658 INFO:tasks.workunit.client.0.smithi125.stdout: create 238/38 file after the snapshot 2023-10-30T12:31:41.658 INFO:tasks.workunit.client.0.smithi125.stdout: create 238/39 file after the snapshot 2023-10-30T12:31:41.658 INFO:tasks.workunit.client.0.smithi125.stdout: create 239/.snap/snap-subdir-test 2023-10-30T12:31:41.683 INFO:tasks.workunit.client.0.smithi125.stdout: create 239/30 file after the snapshot 2023-10-30T12:31:41.684 INFO:tasks.workunit.client.0.smithi125.stdout: create 239/31 file after the snapshot 2023-10-30T12:31:41.685 INFO:tasks.workunit.client.0.smithi125.stdout: create 239/32 file after the snapshot 2023-10-30T12:31:41.685 INFO:tasks.workunit.client.0.smithi125.stdout: create 239/33 file after the snapshot 2023-10-30T12:31:41.685 INFO:tasks.workunit.client.0.smithi125.stdout: create 239/34 file after the snapshot 2023-10-30T12:31:41.685 INFO:tasks.workunit.client.0.smithi125.stdout: create 239/35 file after the snapshot 2023-10-30T12:31:41.685 INFO:tasks.workunit.client.0.smithi125.stdout: create 239/36 file after the snapshot 2023-10-30T12:31:41.686 INFO:tasks.workunit.client.0.smithi125.stdout: create 239/37 file after the snapshot 2023-10-30T12:31:41.686 INFO:tasks.workunit.client.0.smithi125.stdout: create 239/38 file after the snapshot 2023-10-30T12:31:41.686 INFO:tasks.workunit.client.0.smithi125.stdout: create 239/39 file after the snapshot 2023-10-30T12:31:41.686 INFO:tasks.workunit.client.0.smithi125.stdout: create 240/.snap/snap-subdir-test 2023-10-30T12:31:41.711 INFO:tasks.workunit.client.0.smithi125.stdout: create 240/30 file after the snapshot 2023-10-30T12:31:41.714 INFO:tasks.workunit.client.0.smithi125.stdout: create 240/31 file after the snapshot 2023-10-30T12:31:41.714 INFO:tasks.workunit.client.0.smithi125.stdout: create 240/32 file after the snapshot 2023-10-30T12:31:41.715 INFO:tasks.workunit.client.0.smithi125.stdout: create 240/33 file after the snapshot 2023-10-30T12:31:41.715 INFO:tasks.workunit.client.0.smithi125.stdout: create 240/34 file after the snapshot 2023-10-30T12:31:41.715 INFO:tasks.workunit.client.0.smithi125.stdout: create 240/35 file after the snapshot 2023-10-30T12:31:41.715 INFO:tasks.workunit.client.0.smithi125.stdout: create 240/36 file after the snapshot 2023-10-30T12:31:41.715 INFO:tasks.workunit.client.0.smithi125.stdout: create 240/37 file after the snapshot 2023-10-30T12:31:41.716 INFO:tasks.workunit.client.0.smithi125.stdout: create 240/38 file after the snapshot 2023-10-30T12:31:41.716 INFO:tasks.workunit.client.0.smithi125.stdout: create 240/39 file after the snapshot 2023-10-30T12:31:41.716 INFO:tasks.workunit.client.0.smithi125.stdout: create 241/.snap/snap-subdir-test 2023-10-30T12:31:41.737 INFO:tasks.workunit.client.0.smithi125.stdout: create 241/30 file after the snapshot 2023-10-30T12:31:41.739 INFO:tasks.workunit.client.0.smithi125.stdout: create 241/31 file after the snapshot 2023-10-30T12:31:41.739 INFO:tasks.workunit.client.0.smithi125.stdout: create 241/32 file after the snapshot 2023-10-30T12:31:41.740 INFO:tasks.workunit.client.0.smithi125.stdout: create 241/33 file after the snapshot 2023-10-30T12:31:41.740 INFO:tasks.workunit.client.0.smithi125.stdout: create 241/34 file after the snapshot 2023-10-30T12:31:41.740 INFO:tasks.workunit.client.0.smithi125.stdout: create 241/35 file after the snapshot 2023-10-30T12:31:41.740 INFO:tasks.workunit.client.0.smithi125.stdout: create 241/36 file after the snapshot 2023-10-30T12:31:41.740 INFO:tasks.workunit.client.0.smithi125.stdout: create 241/37 file after the snapshot 2023-10-30T12:31:41.740 INFO:tasks.workunit.client.0.smithi125.stdout: create 241/38 file after the snapshot 2023-10-30T12:31:41.741 INFO:tasks.workunit.client.0.smithi125.stdout: create 241/39 file after the snapshot 2023-10-30T12:31:41.741 INFO:tasks.workunit.client.0.smithi125.stdout: create 242/.snap/snap-subdir-test 2023-10-30T12:31:42.126 INFO:tasks.workunit.client.0.smithi125.stdout: create 242/30 file after the snapshot 2023-10-30T12:31:42.127 INFO:tasks.workunit.client.0.smithi125.stdout: create 242/31 file after the snapshot 2023-10-30T12:31:42.127 INFO:tasks.workunit.client.0.smithi125.stdout: create 242/32 file after the snapshot 2023-10-30T12:31:42.128 INFO:tasks.workunit.client.0.smithi125.stdout: create 242/33 file after the snapshot 2023-10-30T12:31:42.128 INFO:tasks.workunit.client.0.smithi125.stdout: create 242/34 file after the snapshot 2023-10-30T12:31:42.128 INFO:tasks.workunit.client.0.smithi125.stdout: create 242/35 file after the snapshot 2023-10-30T12:31:42.128 INFO:tasks.workunit.client.0.smithi125.stdout: create 242/36 file after the snapshot 2023-10-30T12:31:42.128 INFO:tasks.workunit.client.0.smithi125.stdout: create 242/37 file after the snapshot 2023-10-30T12:31:42.129 INFO:tasks.workunit.client.0.smithi125.stdout: create 242/38 file after the snapshot 2023-10-30T12:31:42.129 INFO:tasks.workunit.client.0.smithi125.stdout: create 242/39 file after the snapshot 2023-10-30T12:31:42.129 INFO:tasks.workunit.client.0.smithi125.stdout: create 243/.snap/snap-subdir-test 2023-10-30T12:31:42.151 INFO:tasks.workunit.client.0.smithi125.stdout: create 243/30 file after the snapshot 2023-10-30T12:31:42.152 INFO:tasks.workunit.client.0.smithi125.stdout: create 243/31 file after the snapshot 2023-10-30T12:31:42.153 INFO:tasks.workunit.client.0.smithi125.stdout: create 243/32 file after the snapshot 2023-10-30T12:31:42.153 INFO:tasks.workunit.client.0.smithi125.stdout: create 243/33 file after the snapshot 2023-10-30T12:31:42.153 INFO:tasks.workunit.client.0.smithi125.stdout: create 243/34 file after the snapshot 2023-10-30T12:31:42.153 INFO:tasks.workunit.client.0.smithi125.stdout: create 243/35 file after the snapshot 2023-10-30T12:31:42.153 INFO:tasks.workunit.client.0.smithi125.stdout: create 243/36 file after the snapshot 2023-10-30T12:31:42.154 INFO:tasks.workunit.client.0.smithi125.stdout: create 243/37 file after the snapshot 2023-10-30T12:31:42.154 INFO:tasks.workunit.client.0.smithi125.stdout: create 243/38 file after the snapshot 2023-10-30T12:31:42.154 INFO:tasks.workunit.client.0.smithi125.stdout: create 243/39 file after the snapshot 2023-10-30T12:31:42.154 INFO:tasks.workunit.client.0.smithi125.stdout: create 244/.snap/snap-subdir-test 2023-10-30T12:31:42.181 INFO:tasks.workunit.client.0.smithi125.stdout: create 244/30 file after the snapshot 2023-10-30T12:31:42.182 INFO:tasks.workunit.client.0.smithi125.stdout: create 244/31 file after the snapshot 2023-10-30T12:31:42.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 244/32 file after the snapshot 2023-10-30T12:31:42.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 244/33 file after the snapshot 2023-10-30T12:31:42.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 244/34 file after the snapshot 2023-10-30T12:31:42.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 244/35 file after the snapshot 2023-10-30T12:31:42.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 244/36 file after the snapshot 2023-10-30T12:31:42.184 INFO:tasks.workunit.client.0.smithi125.stdout: create 244/37 file after the snapshot 2023-10-30T12:31:42.184 INFO:tasks.workunit.client.0.smithi125.stdout: create 244/38 file after the snapshot 2023-10-30T12:31:42.184 INFO:tasks.workunit.client.0.smithi125.stdout: create 244/39 file after the snapshot 2023-10-30T12:31:42.184 INFO:tasks.workunit.client.0.smithi125.stdout: create 245/.snap/snap-subdir-test 2023-10-30T12:31:42.195 INFO:tasks.workunit.client.0.smithi125.stdout: create 245/30 file after the snapshot 2023-10-30T12:31:42.196 INFO:tasks.workunit.client.0.smithi125.stdout: create 245/31 file after the snapshot 2023-10-30T12:31:42.197 INFO:tasks.workunit.client.0.smithi125.stdout: create 245/32 file after the snapshot 2023-10-30T12:31:42.197 INFO:tasks.workunit.client.0.smithi125.stdout: create 245/33 file after the snapshot 2023-10-30T12:31:42.197 INFO:tasks.workunit.client.0.smithi125.stdout: create 245/34 file after the snapshot 2023-10-30T12:31:42.197 INFO:tasks.workunit.client.0.smithi125.stdout: create 245/35 file after the snapshot 2023-10-30T12:31:42.197 INFO:tasks.workunit.client.0.smithi125.stdout: create 245/36 file after the snapshot 2023-10-30T12:31:42.198 INFO:tasks.workunit.client.0.smithi125.stdout: create 245/37 file after the snapshot 2023-10-30T12:31:42.198 INFO:tasks.workunit.client.0.smithi125.stdout: create 245/38 file after the snapshot 2023-10-30T12:31:42.198 INFO:tasks.workunit.client.0.smithi125.stdout: create 245/39 file after the snapshot 2023-10-30T12:31:42.198 INFO:tasks.workunit.client.0.smithi125.stdout: create 246/.snap/snap-subdir-test 2023-10-30T12:31:42.221 INFO:tasks.workunit.client.0.smithi125.stdout: create 246/30 file after the snapshot 2023-10-30T12:31:42.223 INFO:tasks.workunit.client.0.smithi125.stdout: create 246/31 file after the snapshot 2023-10-30T12:31:42.223 INFO:tasks.workunit.client.0.smithi125.stdout: create 246/32 file after the snapshot 2023-10-30T12:31:42.223 INFO:tasks.workunit.client.0.smithi125.stdout: create 246/33 file after the snapshot 2023-10-30T12:31:42.223 INFO:tasks.workunit.client.0.smithi125.stdout: create 246/34 file after the snapshot 2023-10-30T12:31:42.224 INFO:tasks.workunit.client.0.smithi125.stdout: create 246/35 file after the snapshot 2023-10-30T12:31:42.224 INFO:tasks.workunit.client.0.smithi125.stdout: create 246/36 file after the snapshot 2023-10-30T12:31:42.224 INFO:tasks.workunit.client.0.smithi125.stdout: create 246/37 file after the snapshot 2023-10-30T12:31:42.224 INFO:tasks.workunit.client.0.smithi125.stdout: create 246/38 file after the snapshot 2023-10-30T12:31:42.224 INFO:tasks.workunit.client.0.smithi125.stdout: create 246/39 file after the snapshot 2023-10-30T12:31:42.224 INFO:tasks.workunit.client.0.smithi125.stdout: create 247/.snap/snap-subdir-test 2023-10-30T12:31:42.292 INFO:tasks.workunit.client.0.smithi125.stdout: create 247/30 file after the snapshot 2023-10-30T12:31:42.294 INFO:tasks.workunit.client.0.smithi125.stdout: create 247/31 file after the snapshot 2023-10-30T12:31:42.294 INFO:tasks.workunit.client.0.smithi125.stdout: create 247/32 file after the snapshot 2023-10-30T12:31:42.294 INFO:tasks.workunit.client.0.smithi125.stdout: create 247/33 file after the snapshot 2023-10-30T12:31:42.294 INFO:tasks.workunit.client.0.smithi125.stdout: create 247/34 file after the snapshot 2023-10-30T12:31:42.295 INFO:tasks.workunit.client.0.smithi125.stdout: create 247/35 file after the snapshot 2023-10-30T12:31:42.295 INFO:tasks.workunit.client.0.smithi125.stdout: create 247/36 file after the snapshot 2023-10-30T12:31:42.295 INFO:tasks.workunit.client.0.smithi125.stdout: create 247/37 file after the snapshot 2023-10-30T12:31:42.295 INFO:tasks.workunit.client.0.smithi125.stdout: create 247/38 file after the snapshot 2023-10-30T12:31:42.295 INFO:tasks.workunit.client.0.smithi125.stdout: create 247/39 file after the snapshot 2023-10-30T12:31:42.296 INFO:tasks.workunit.client.0.smithi125.stdout: create 248/.snap/snap-subdir-test 2023-10-30T12:31:42.321 INFO:tasks.workunit.client.0.smithi125.stdout: create 248/30 file after the snapshot 2023-10-30T12:31:42.322 INFO:tasks.workunit.client.0.smithi125.stdout: create 248/31 file after the snapshot 2023-10-30T12:31:42.323 INFO:tasks.workunit.client.0.smithi125.stdout: create 248/32 file after the snapshot 2023-10-30T12:31:42.323 INFO:tasks.workunit.client.0.smithi125.stdout: create 248/33 file after the snapshot 2023-10-30T12:31:42.323 INFO:tasks.workunit.client.0.smithi125.stdout: create 248/34 file after the snapshot 2023-10-30T12:31:42.323 INFO:tasks.workunit.client.0.smithi125.stdout: create 248/35 file after the snapshot 2023-10-30T12:31:42.323 INFO:tasks.workunit.client.0.smithi125.stdout: create 248/36 file after the snapshot 2023-10-30T12:31:42.324 INFO:tasks.workunit.client.0.smithi125.stdout: create 248/37 file after the snapshot 2023-10-30T12:31:42.324 INFO:tasks.workunit.client.0.smithi125.stdout: create 248/38 file after the snapshot 2023-10-30T12:31:42.324 INFO:tasks.workunit.client.0.smithi125.stdout: create 248/39 file after the snapshot 2023-10-30T12:31:42.324 INFO:tasks.workunit.client.0.smithi125.stdout: create 249/.snap/snap-subdir-test 2023-10-30T12:31:43.191 INFO:tasks.workunit.client.0.smithi125.stdout: create 249/30 file after the snapshot 2023-10-30T12:31:43.443 INFO:tasks.workunit.client.0.smithi125.stdout: create 249/31 file after the snapshot 2023-10-30T12:31:43.443 INFO:tasks.workunit.client.0.smithi125.stdout: create 249/32 file after the snapshot 2023-10-30T12:31:43.443 INFO:tasks.workunit.client.0.smithi125.stdout: create 249/33 file after the snapshot 2023-10-30T12:31:43.444 INFO:tasks.workunit.client.0.smithi125.stdout: create 249/34 file after the snapshot 2023-10-30T12:31:43.444 INFO:tasks.workunit.client.0.smithi125.stdout: create 249/35 file after the snapshot 2023-10-30T12:31:43.444 INFO:tasks.workunit.client.0.smithi125.stdout: create 249/36 file after the snapshot 2023-10-30T12:31:43.444 INFO:tasks.workunit.client.0.smithi125.stdout: create 249/37 file after the snapshot 2023-10-30T12:31:43.444 INFO:tasks.workunit.client.0.smithi125.stdout: create 249/38 file after the snapshot 2023-10-30T12:31:43.445 INFO:tasks.workunit.client.0.smithi125.stdout: create 249/39 file after the snapshot 2023-10-30T12:31:43.445 INFO:tasks.workunit.client.0.smithi125.stdout: create 250/.snap/snap-subdir-test 2023-10-30T12:31:43.472 INFO:tasks.workunit.client.0.smithi125.stdout: create 250/30 file after the snapshot 2023-10-30T12:31:43.474 INFO:tasks.workunit.client.0.smithi125.stdout: create 250/31 file after the snapshot 2023-10-30T12:31:43.474 INFO:tasks.workunit.client.0.smithi125.stdout: create 250/32 file after the snapshot 2023-10-30T12:31:43.474 INFO:tasks.workunit.client.0.smithi125.stdout: create 250/33 file after the snapshot 2023-10-30T12:31:43.475 INFO:tasks.workunit.client.0.smithi125.stdout: create 250/34 file after the snapshot 2023-10-30T12:31:43.475 INFO:tasks.workunit.client.0.smithi125.stdout: create 250/35 file after the snapshot 2023-10-30T12:31:43.475 INFO:tasks.workunit.client.0.smithi125.stdout: create 250/36 file after the snapshot 2023-10-30T12:31:43.475 INFO:tasks.workunit.client.0.smithi125.stdout: create 250/37 file after the snapshot 2023-10-30T12:31:43.476 INFO:tasks.workunit.client.0.smithi125.stdout: create 250/38 file after the snapshot 2023-10-30T12:31:43.476 INFO:tasks.workunit.client.0.smithi125.stdout: create 250/39 file after the snapshot 2023-10-30T12:31:43.476 INFO:tasks.workunit.client.0.smithi125.stdout: create 251/.snap/snap-subdir-test 2023-10-30T12:31:43.501 INFO:tasks.workunit.client.0.smithi125.stdout: create 251/30 file after the snapshot 2023-10-30T12:31:43.503 INFO:tasks.workunit.client.0.smithi125.stdout: create 251/31 file after the snapshot 2023-10-30T12:31:43.503 INFO:tasks.workunit.client.0.smithi125.stdout: create 251/32 file after the snapshot 2023-10-30T12:31:43.503 INFO:tasks.workunit.client.0.smithi125.stdout: create 251/33 file after the snapshot 2023-10-30T12:31:43.504 INFO:tasks.workunit.client.0.smithi125.stdout: create 251/34 file after the snapshot 2023-10-30T12:31:43.504 INFO:tasks.workunit.client.0.smithi125.stdout: create 251/35 file after the snapshot 2023-10-30T12:31:43.504 INFO:tasks.workunit.client.0.smithi125.stdout: create 251/36 file after the snapshot 2023-10-30T12:31:43.505 INFO:tasks.workunit.client.0.smithi125.stdout: create 251/37 file after the snapshot 2023-10-30T12:31:43.505 INFO:tasks.workunit.client.0.smithi125.stdout: create 251/38 file after the snapshot 2023-10-30T12:31:43.505 INFO:tasks.workunit.client.0.smithi125.stdout: create 251/39 file after the snapshot 2023-10-30T12:31:43.505 INFO:tasks.workunit.client.0.smithi125.stdout: create 252/.snap/snap-subdir-test 2023-10-30T12:31:43.549 INFO:tasks.workunit.client.0.smithi125.stdout: create 252/30 file after the snapshot 2023-10-30T12:31:43.562 INFO:tasks.workunit.client.0.smithi125.stdout: create 252/31 file after the snapshot 2023-10-30T12:31:43.564 INFO:tasks.workunit.client.0.smithi125.stdout: create 252/32 file after the snapshot 2023-10-30T12:31:43.566 INFO:tasks.workunit.client.0.smithi125.stdout: create 252/33 file after the snapshot 2023-10-30T12:31:43.572 INFO:tasks.workunit.client.0.smithi125.stdout: create 252/34 file after the snapshot 2023-10-30T12:31:43.573 INFO:tasks.workunit.client.0.smithi125.stdout: create 252/35 file after the snapshot 2023-10-30T12:31:43.573 INFO:tasks.workunit.client.0.smithi125.stdout: create 252/36 file after the snapshot 2023-10-30T12:31:43.574 INFO:tasks.workunit.client.0.smithi125.stdout: create 252/37 file after the snapshot 2023-10-30T12:31:43.574 INFO:tasks.workunit.client.0.smithi125.stdout: create 252/38 file after the snapshot 2023-10-30T12:31:43.574 INFO:tasks.workunit.client.0.smithi125.stdout: create 252/39 file after the snapshot 2023-10-30T12:31:43.574 INFO:tasks.workunit.client.0.smithi125.stdout: create 253/.snap/snap-subdir-test 2023-10-30T12:31:44.244 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:31:44.248 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:31:44.290 INFO:tasks.workunit.client.0.smithi125.stdout: create 253/30 file after the snapshot 2023-10-30T12:31:44.291 INFO:tasks.workunit.client.0.smithi125.stdout: create 253/31 file after the snapshot 2023-10-30T12:31:44.291 INFO:tasks.workunit.client.0.smithi125.stdout: create 253/32 file after the snapshot 2023-10-30T12:31:44.292 INFO:tasks.workunit.client.0.smithi125.stdout: create 253/33 file after the snapshot 2023-10-30T12:31:44.292 INFO:tasks.workunit.client.0.smithi125.stdout: create 253/34 file after the snapshot 2023-10-30T12:31:44.292 INFO:tasks.workunit.client.0.smithi125.stdout: create 253/35 file after the snapshot 2023-10-30T12:31:44.292 INFO:tasks.workunit.client.0.smithi125.stdout: create 253/36 file after the snapshot 2023-10-30T12:31:44.292 INFO:tasks.workunit.client.0.smithi125.stdout: create 253/37 file after the snapshot 2023-10-30T12:31:44.292 INFO:tasks.workunit.client.0.smithi125.stdout: create 253/38 file after the snapshot 2023-10-30T12:31:44.293 INFO:tasks.workunit.client.0.smithi125.stdout: create 253/39 file after the snapshot 2023-10-30T12:31:44.293 INFO:tasks.workunit.client.0.smithi125.stdout: create 254/.snap/snap-subdir-test 2023-10-30T12:31:44.313 INFO:tasks.workunit.client.0.smithi125.stdout: create 254/30 file after the snapshot 2023-10-30T12:31:44.314 INFO:tasks.workunit.client.0.smithi125.stdout: create 254/31 file after the snapshot 2023-10-30T12:31:44.314 INFO:tasks.workunit.client.0.smithi125.stdout: create 254/32 file after the snapshot 2023-10-30T12:31:44.315 INFO:tasks.workunit.client.0.smithi125.stdout: create 254/33 file after the snapshot 2023-10-30T12:31:44.315 INFO:tasks.workunit.client.0.smithi125.stdout: create 254/34 file after the snapshot 2023-10-30T12:31:44.315 INFO:tasks.workunit.client.0.smithi125.stdout: create 254/35 file after the snapshot 2023-10-30T12:31:44.315 INFO:tasks.workunit.client.0.smithi125.stdout: create 254/36 file after the snapshot 2023-10-30T12:31:44.316 INFO:tasks.workunit.client.0.smithi125.stdout: create 254/37 file after the snapshot 2023-10-30T12:31:44.316 INFO:tasks.workunit.client.0.smithi125.stdout: create 254/38 file after the snapshot 2023-10-30T12:31:44.316 INFO:tasks.workunit.client.0.smithi125.stdout: create 254/39 file after the snapshot 2023-10-30T12:31:44.316 INFO:tasks.workunit.client.0.smithi125.stdout: create 255/.snap/snap-subdir-test 2023-10-30T12:31:44.861 INFO:tasks.workunit.client.0.smithi125.stdout: create 255/30 file after the snapshot 2023-10-30T12:31:44.862 INFO:tasks.workunit.client.0.smithi125.stdout: create 255/31 file after the snapshot 2023-10-30T12:31:44.863 INFO:tasks.workunit.client.0.smithi125.stdout: create 255/32 file after the snapshot 2023-10-30T12:31:44.863 INFO:tasks.workunit.client.0.smithi125.stdout: create 255/33 file after the snapshot 2023-10-30T12:31:44.863 INFO:tasks.workunit.client.0.smithi125.stdout: create 255/34 file after the snapshot 2023-10-30T12:31:44.863 INFO:tasks.workunit.client.0.smithi125.stdout: create 255/35 file after the snapshot 2023-10-30T12:31:44.864 INFO:tasks.workunit.client.0.smithi125.stdout: create 255/36 file after the snapshot 2023-10-30T12:31:44.864 INFO:tasks.workunit.client.0.smithi125.stdout: create 255/37 file after the snapshot 2023-10-30T12:31:44.864 INFO:tasks.workunit.client.0.smithi125.stdout: create 255/38 file after the snapshot 2023-10-30T12:31:44.864 INFO:tasks.workunit.client.0.smithi125.stdout: create 255/39 file after the snapshot 2023-10-30T12:31:44.865 INFO:tasks.workunit.client.0.smithi125.stdout: create 256/.snap/snap-subdir-test 2023-10-30T12:31:45.134 INFO:tasks.workunit.client.0.smithi125.stdout: create 256/30 file after the snapshot 2023-10-30T12:31:45.135 INFO:tasks.workunit.client.0.smithi125.stdout: create 256/31 file after the snapshot 2023-10-30T12:31:45.135 INFO:tasks.workunit.client.0.smithi125.stdout: create 256/32 file after the snapshot 2023-10-30T12:31:45.135 INFO:tasks.workunit.client.0.smithi125.stdout: create 256/33 file after the snapshot 2023-10-30T12:31:45.135 INFO:tasks.workunit.client.0.smithi125.stdout: create 256/34 file after the snapshot 2023-10-30T12:31:45.135 INFO:tasks.workunit.client.0.smithi125.stdout: create 256/35 file after the snapshot 2023-10-30T12:31:45.136 INFO:tasks.workunit.client.0.smithi125.stdout: create 256/36 file after the snapshot 2023-10-30T12:31:45.137 INFO:tasks.workunit.client.0.smithi125.stdout: create 256/37 file after the snapshot 2023-10-30T12:31:45.137 INFO:tasks.workunit.client.0.smithi125.stdout: create 256/38 file after the snapshot 2023-10-30T12:31:45.137 INFO:tasks.workunit.client.0.smithi125.stdout: create 256/39 file after the snapshot 2023-10-30T12:31:45.137 INFO:tasks.workunit.client.0.smithi125.stdout: create 257/.snap/snap-subdir-test 2023-10-30T12:31:45.708 INFO:tasks.workunit.client.0.smithi125.stdout: create 257/30 file after the snapshot 2023-10-30T12:31:45.711 INFO:tasks.workunit.client.0.smithi125.stdout: create 257/31 file after the snapshot 2023-10-30T12:31:45.711 INFO:tasks.workunit.client.0.smithi125.stdout: create 257/32 file after the snapshot 2023-10-30T12:31:45.711 INFO:tasks.workunit.client.0.smithi125.stdout: create 257/33 file after the snapshot 2023-10-30T12:31:45.712 INFO:tasks.workunit.client.0.smithi125.stdout: create 257/34 file after the snapshot 2023-10-30T12:31:45.712 INFO:tasks.workunit.client.0.smithi125.stdout: create 257/35 file after the snapshot 2023-10-30T12:31:45.712 INFO:tasks.workunit.client.0.smithi125.stdout: create 257/36 file after the snapshot 2023-10-30T12:31:45.712 INFO:tasks.workunit.client.0.smithi125.stdout: create 257/37 file after the snapshot 2023-10-30T12:31:45.713 INFO:tasks.workunit.client.0.smithi125.stdout: create 257/38 file after the snapshot 2023-10-30T12:31:45.713 INFO:tasks.workunit.client.0.smithi125.stdout: create 257/39 file after the snapshot 2023-10-30T12:31:45.713 INFO:tasks.workunit.client.0.smithi125.stdout: create 258/.snap/snap-subdir-test 2023-10-30T12:31:45.740 INFO:tasks.workunit.client.0.smithi125.stdout: create 258/30 file after the snapshot 2023-10-30T12:31:45.742 INFO:tasks.workunit.client.0.smithi125.stdout: create 258/31 file after the snapshot 2023-10-30T12:31:45.743 INFO:tasks.workunit.client.0.smithi125.stdout: create 258/32 file after the snapshot 2023-10-30T12:31:45.743 INFO:tasks.workunit.client.0.smithi125.stdout: create 258/33 file after the snapshot 2023-10-30T12:31:45.743 INFO:tasks.workunit.client.0.smithi125.stdout: create 258/34 file after the snapshot 2023-10-30T12:31:45.743 INFO:tasks.workunit.client.0.smithi125.stdout: create 258/35 file after the snapshot 2023-10-30T12:31:45.743 INFO:tasks.workunit.client.0.smithi125.stdout: create 258/36 file after the snapshot 2023-10-30T12:31:45.744 INFO:tasks.workunit.client.0.smithi125.stdout: create 258/37 file after the snapshot 2023-10-30T12:31:45.744 INFO:tasks.workunit.client.0.smithi125.stdout: create 258/38 file after the snapshot 2023-10-30T12:31:45.744 INFO:tasks.workunit.client.0.smithi125.stdout: create 258/39 file after the snapshot 2023-10-30T12:31:45.744 INFO:tasks.workunit.client.0.smithi125.stdout: create 259/.snap/snap-subdir-test 2023-10-30T12:31:46.272 INFO:tasks.workunit.client.0.smithi125.stdout: create 259/30 file after the snapshot 2023-10-30T12:31:46.273 INFO:tasks.workunit.client.0.smithi125.stdout: create 259/31 file after the snapshot 2023-10-30T12:31:46.274 INFO:tasks.workunit.client.0.smithi125.stdout: create 259/32 file after the snapshot 2023-10-30T12:31:46.274 INFO:tasks.workunit.client.0.smithi125.stdout: create 259/33 file after the snapshot 2023-10-30T12:31:46.274 INFO:tasks.workunit.client.0.smithi125.stdout: create 259/34 file after the snapshot 2023-10-30T12:31:46.274 INFO:tasks.workunit.client.0.smithi125.stdout: create 259/35 file after the snapshot 2023-10-30T12:31:46.274 INFO:tasks.workunit.client.0.smithi125.stdout: create 259/36 file after the snapshot 2023-10-30T12:31:46.275 INFO:tasks.workunit.client.0.smithi125.stdout: create 259/37 file after the snapshot 2023-10-30T12:31:46.275 INFO:tasks.workunit.client.0.smithi125.stdout: create 259/38 file after the snapshot 2023-10-30T12:31:46.277 INFO:tasks.workunit.client.0.smithi125.stdout: create 259/39 file after the snapshot 2023-10-30T12:31:46.277 INFO:tasks.workunit.client.0.smithi125.stdout: create 260/.snap/snap-subdir-test 2023-10-30T12:31:47.153 INFO:tasks.workunit.client.0.smithi125.stdout: create 260/30 file after the snapshot 2023-10-30T12:31:47.154 INFO:tasks.workunit.client.0.smithi125.stdout: create 260/31 file after the snapshot 2023-10-30T12:31:47.155 INFO:tasks.workunit.client.0.smithi125.stdout: create 260/32 file after the snapshot 2023-10-30T12:31:47.155 INFO:tasks.workunit.client.0.smithi125.stdout: create 260/33 file after the snapshot 2023-10-30T12:31:47.156 INFO:tasks.workunit.client.0.smithi125.stdout: create 260/34 file after the snapshot 2023-10-30T12:31:47.156 INFO:tasks.workunit.client.0.smithi125.stdout: create 260/35 file after the snapshot 2023-10-30T12:31:47.156 INFO:tasks.workunit.client.0.smithi125.stdout: create 260/36 file after the snapshot 2023-10-30T12:31:47.156 INFO:tasks.workunit.client.0.smithi125.stdout: create 260/37 file after the snapshot 2023-10-30T12:31:47.156 INFO:tasks.workunit.client.0.smithi125.stdout: create 260/38 file after the snapshot 2023-10-30T12:31:47.157 INFO:tasks.workunit.client.0.smithi125.stdout: create 260/39 file after the snapshot 2023-10-30T12:31:47.157 INFO:tasks.workunit.client.0.smithi125.stdout: create 261/.snap/snap-subdir-test 2023-10-30T12:31:47.178 INFO:tasks.workunit.client.0.smithi125.stdout: create 261/30 file after the snapshot 2023-10-30T12:31:47.182 INFO:tasks.workunit.client.0.smithi125.stdout: create 261/31 file after the snapshot 2023-10-30T12:31:47.182 INFO:tasks.workunit.client.0.smithi125.stdout: create 261/32 file after the snapshot 2023-10-30T12:31:47.182 INFO:tasks.workunit.client.0.smithi125.stdout: create 261/33 file after the snapshot 2023-10-30T12:31:47.182 INFO:tasks.workunit.client.0.smithi125.stdout: create 261/34 file after the snapshot 2023-10-30T12:31:47.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 261/35 file after the snapshot 2023-10-30T12:31:47.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 261/36 file after the snapshot 2023-10-30T12:31:47.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 261/37 file after the snapshot 2023-10-30T12:31:47.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 261/38 file after the snapshot 2023-10-30T12:31:47.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 261/39 file after the snapshot 2023-10-30T12:31:47.183 INFO:tasks.workunit.client.0.smithi125.stdout: create 262/.snap/snap-subdir-test 2023-10-30T12:31:47.457 INFO:tasks.workunit.client.0.smithi125.stdout: create 262/30 file after the snapshot 2023-10-30T12:31:47.459 INFO:tasks.workunit.client.0.smithi125.stdout: create 262/31 file after the snapshot 2023-10-30T12:31:47.459 INFO:tasks.workunit.client.0.smithi125.stdout: create 262/32 file after the snapshot 2023-10-30T12:31:47.459 INFO:tasks.workunit.client.0.smithi125.stdout: create 262/33 file after the snapshot 2023-10-30T12:31:47.459 INFO:tasks.workunit.client.0.smithi125.stdout: create 262/34 file after the snapshot 2023-10-30T12:31:47.460 INFO:tasks.workunit.client.0.smithi125.stdout: create 262/35 file after the snapshot 2023-10-30T12:31:47.460 INFO:tasks.workunit.client.0.smithi125.stdout: create 262/36 file after the snapshot 2023-10-30T12:31:47.460 INFO:tasks.workunit.client.0.smithi125.stdout: create 262/37 file after the snapshot 2023-10-30T12:31:47.461 INFO:tasks.workunit.client.0.smithi125.stdout: create 262/38 file after the snapshot 2023-10-30T12:31:47.461 INFO:tasks.workunit.client.0.smithi125.stdout: create 262/39 file after the snapshot 2023-10-30T12:31:47.461 INFO:tasks.workunit.client.0.smithi125.stdout: create 263/.snap/snap-subdir-test 2023-10-30T12:31:47.485 INFO:tasks.workunit.client.0.smithi125.stdout: create 263/30 file after the snapshot 2023-10-30T12:31:47.487 INFO:tasks.workunit.client.0.smithi125.stdout: create 263/31 file after the snapshot 2023-10-30T12:31:47.487 INFO:tasks.workunit.client.0.smithi125.stdout: create 263/32 file after the snapshot 2023-10-30T12:31:47.488 INFO:tasks.workunit.client.0.smithi125.stdout: create 263/33 file after the snapshot 2023-10-30T12:31:47.488 INFO:tasks.workunit.client.0.smithi125.stdout: create 263/34 file after the snapshot 2023-10-30T12:31:47.488 INFO:tasks.workunit.client.0.smithi125.stdout: create 263/35 file after the snapshot 2023-10-30T12:31:47.488 INFO:tasks.workunit.client.0.smithi125.stdout: create 263/36 file after the snapshot 2023-10-30T12:31:47.488 INFO:tasks.workunit.client.0.smithi125.stdout: create 263/37 file after the snapshot 2023-10-30T12:31:47.489 INFO:tasks.workunit.client.0.smithi125.stdout: create 263/38 file after the snapshot 2023-10-30T12:31:47.489 INFO:tasks.workunit.client.0.smithi125.stdout: create 263/39 file after the snapshot 2023-10-30T12:31:47.489 INFO:tasks.workunit.client.0.smithi125.stdout: create 264/.snap/snap-subdir-test 2023-10-30T12:31:47.721 INFO:tasks.workunit.client.0.smithi125.stdout: create 264/30 file after the snapshot 2023-10-30T12:31:47.722 INFO:tasks.workunit.client.0.smithi125.stdout: create 264/31 file after the snapshot 2023-10-30T12:31:47.723 INFO:tasks.workunit.client.0.smithi125.stdout: create 264/32 file after the snapshot 2023-10-30T12:31:47.723 INFO:tasks.workunit.client.0.smithi125.stdout: create 264/33 file after the snapshot 2023-10-30T12:31:47.723 INFO:tasks.workunit.client.0.smithi125.stdout: create 264/34 file after the snapshot 2023-10-30T12:31:47.723 INFO:tasks.workunit.client.0.smithi125.stdout: create 264/35 file after the snapshot 2023-10-30T12:31:47.724 INFO:tasks.workunit.client.0.smithi125.stdout: create 264/36 file after the snapshot 2023-10-30T12:31:47.724 INFO:tasks.workunit.client.0.smithi125.stdout: create 264/37 file after the snapshot 2023-10-30T12:31:47.725 INFO:tasks.workunit.client.0.smithi125.stdout: create 264/38 file after the snapshot 2023-10-30T12:31:47.725 INFO:tasks.workunit.client.0.smithi125.stdout: create 264/39 file after the snapshot 2023-10-30T12:31:47.725 INFO:tasks.workunit.client.0.smithi125.stdout: create 265/.snap/snap-subdir-test 2023-10-30T12:31:47.751 INFO:tasks.workunit.client.0.smithi125.stdout: create 265/30 file after the snapshot 2023-10-30T12:31:47.753 INFO:tasks.workunit.client.0.smithi125.stdout: create 265/31 file after the snapshot 2023-10-30T12:31:47.754 INFO:tasks.workunit.client.0.smithi125.stdout: create 265/32 file after the snapshot 2023-10-30T12:31:47.754 INFO:tasks.workunit.client.0.smithi125.stdout: create 265/33 file after the snapshot 2023-10-30T12:31:47.754 INFO:tasks.workunit.client.0.smithi125.stdout: create 265/34 file after the snapshot 2023-10-30T12:31:47.754 INFO:tasks.workunit.client.0.smithi125.stdout: create 265/35 file after the snapshot 2023-10-30T12:31:47.754 INFO:tasks.workunit.client.0.smithi125.stdout: create 265/36 file after the snapshot 2023-10-30T12:31:47.755 INFO:tasks.workunit.client.0.smithi125.stdout: create 265/37 file after the snapshot 2023-10-30T12:31:47.755 INFO:tasks.workunit.client.0.smithi125.stdout: create 265/38 file after the snapshot 2023-10-30T12:31:47.755 INFO:tasks.workunit.client.0.smithi125.stdout: create 265/39 file after the snapshot 2023-10-30T12:31:47.755 INFO:tasks.workunit.client.0.smithi125.stdout: create 266/.snap/snap-subdir-test 2023-10-30T12:31:47.782 INFO:tasks.workunit.client.0.smithi125.stdout: create 266/30 file after the snapshot 2023-10-30T12:31:47.786 INFO:tasks.workunit.client.0.smithi125.stdout: create 266/31 file after the snapshot 2023-10-30T12:31:47.787 INFO:tasks.workunit.client.0.smithi125.stdout: create 266/32 file after the snapshot 2023-10-30T12:31:47.787 INFO:tasks.workunit.client.0.smithi125.stdout: create 266/33 file after the snapshot 2023-10-30T12:31:47.787 INFO:tasks.workunit.client.0.smithi125.stdout: create 266/34 file after the snapshot 2023-10-30T12:31:47.787 INFO:tasks.workunit.client.0.smithi125.stdout: create 266/35 file after the snapshot 2023-10-30T12:31:47.787 INFO:tasks.workunit.client.0.smithi125.stdout: create 266/36 file after the snapshot 2023-10-30T12:31:47.788 INFO:tasks.workunit.client.0.smithi125.stdout: create 266/37 file after the snapshot 2023-10-30T12:31:47.788 INFO:tasks.workunit.client.0.smithi125.stdout: create 266/38 file after the snapshot 2023-10-30T12:31:47.788 INFO:tasks.workunit.client.0.smithi125.stdout: create 266/39 file after the snapshot 2023-10-30T12:31:47.788 INFO:tasks.workunit.client.0.smithi125.stdout: create 267/.snap/snap-subdir-test 2023-10-30T12:31:47.807 INFO:tasks.workunit.client.0.smithi125.stdout: create 267/30 file after the snapshot 2023-10-30T12:31:47.808 INFO:tasks.workunit.client.0.smithi125.stdout: create 267/31 file after the snapshot 2023-10-30T12:31:47.808 INFO:tasks.workunit.client.0.smithi125.stdout: create 267/32 file after the snapshot 2023-10-30T12:31:47.808 INFO:tasks.workunit.client.0.smithi125.stdout: create 267/33 file after the snapshot 2023-10-30T12:31:47.808 INFO:tasks.workunit.client.0.smithi125.stdout: create 267/34 file after the snapshot 2023-10-30T12:31:47.809 INFO:tasks.workunit.client.0.smithi125.stdout: create 267/35 file after the snapshot 2023-10-30T12:31:47.809 INFO:tasks.workunit.client.0.smithi125.stdout: create 267/36 file after the snapshot 2023-10-30T12:31:47.809 INFO:tasks.workunit.client.0.smithi125.stdout: create 267/37 file after the snapshot 2023-10-30T12:31:47.809 INFO:tasks.workunit.client.0.smithi125.stdout: create 267/38 file after the snapshot 2023-10-30T12:31:47.809 INFO:tasks.workunit.client.0.smithi125.stdout: create 267/39 file after the snapshot 2023-10-30T12:31:47.810 INFO:tasks.workunit.client.0.smithi125.stdout: create 268/.snap/snap-subdir-test 2023-10-30T12:31:47.832 INFO:tasks.workunit.client.0.smithi125.stdout: create 268/30 file after the snapshot 2023-10-30T12:31:47.835 INFO:tasks.workunit.client.0.smithi125.stdout: create 268/31 file after the snapshot 2023-10-30T12:31:47.835 INFO:tasks.workunit.client.0.smithi125.stdout: create 268/32 file after the snapshot 2023-10-30T12:31:47.835 INFO:tasks.workunit.client.0.smithi125.stdout: create 268/33 file after the snapshot 2023-10-30T12:31:47.836 INFO:tasks.workunit.client.0.smithi125.stdout: create 268/34 file after the snapshot 2023-10-30T12:31:47.836 INFO:tasks.workunit.client.0.smithi125.stdout: create 268/35 file after the snapshot 2023-10-30T12:31:47.836 INFO:tasks.workunit.client.0.smithi125.stdout: create 268/36 file after the snapshot 2023-10-30T12:31:47.837 INFO:tasks.workunit.client.0.smithi125.stdout: create 268/37 file after the snapshot 2023-10-30T12:31:47.837 INFO:tasks.workunit.client.0.smithi125.stdout: create 268/38 file after the snapshot 2023-10-30T12:31:47.837 INFO:tasks.workunit.client.0.smithi125.stdout: create 268/39 file after the snapshot 2023-10-30T12:31:47.838 INFO:tasks.workunit.client.0.smithi125.stdout: create 269/.snap/snap-subdir-test 2023-10-30T12:31:47.859 INFO:tasks.workunit.client.0.smithi125.stdout: create 269/30 file after the snapshot 2023-10-30T12:31:47.861 INFO:tasks.workunit.client.0.smithi125.stdout: create 269/31 file after the snapshot 2023-10-30T12:31:47.861 INFO:tasks.workunit.client.0.smithi125.stdout: create 269/32 file after the snapshot 2023-10-30T12:31:47.862 INFO:tasks.workunit.client.0.smithi125.stdout: create 269/33 file after the snapshot 2023-10-30T12:31:47.862 INFO:tasks.workunit.client.0.smithi125.stdout: create 269/34 file after the snapshot 2023-10-30T12:31:47.862 INFO:tasks.workunit.client.0.smithi125.stdout: create 269/35 file after the snapshot 2023-10-30T12:31:47.862 INFO:tasks.workunit.client.0.smithi125.stdout: create 269/36 file after the snapshot 2023-10-30T12:31:47.862 INFO:tasks.workunit.client.0.smithi125.stdout: create 269/37 file after the snapshot 2023-10-30T12:31:47.863 INFO:tasks.workunit.client.0.smithi125.stdout: create 269/38 file after the snapshot 2023-10-30T12:31:47.863 INFO:tasks.workunit.client.0.smithi125.stdout: create 269/39 file after the snapshot 2023-10-30T12:31:47.863 INFO:tasks.workunit.client.0.smithi125.stdout: create 270/.snap/snap-subdir-test 2023-10-30T12:31:47.873 INFO:tasks.workunit.client.0.smithi125.stdout: create 270/30 file after the snapshot 2023-10-30T12:31:47.875 INFO:tasks.workunit.client.0.smithi125.stdout: create 270/31 file after the snapshot 2023-10-30T12:31:47.876 INFO:tasks.workunit.client.0.smithi125.stdout: create 270/32 file after the snapshot 2023-10-30T12:31:47.876 INFO:tasks.workunit.client.0.smithi125.stdout: create 270/33 file after the snapshot 2023-10-30T12:31:47.876 INFO:tasks.workunit.client.0.smithi125.stdout: create 270/34 file after the snapshot 2023-10-30T12:31:47.877 INFO:tasks.workunit.client.0.smithi125.stdout: create 270/35 file after the snapshot 2023-10-30T12:31:47.877 INFO:tasks.workunit.client.0.smithi125.stdout: create 270/36 file after the snapshot 2023-10-30T12:31:47.877 INFO:tasks.workunit.client.0.smithi125.stdout: create 270/37 file after the snapshot 2023-10-30T12:31:47.877 INFO:tasks.workunit.client.0.smithi125.stdout: create 270/38 file after the snapshot 2023-10-30T12:31:47.878 INFO:tasks.workunit.client.0.smithi125.stdout: create 270/39 file after the snapshot 2023-10-30T12:31:47.878 INFO:tasks.workunit.client.0.smithi125.stdout: create 271/.snap/snap-subdir-test 2023-10-30T12:31:48.140 INFO:tasks.workunit.client.0.smithi125.stdout: create 271/30 file after the snapshot 2023-10-30T12:31:48.142 INFO:tasks.workunit.client.0.smithi125.stdout: create 271/31 file after the snapshot 2023-10-30T12:31:48.142 INFO:tasks.workunit.client.0.smithi125.stdout: create 271/32 file after the snapshot 2023-10-30T12:31:48.142 INFO:tasks.workunit.client.0.smithi125.stdout: create 271/33 file after the snapshot 2023-10-30T12:31:48.143 INFO:tasks.workunit.client.0.smithi125.stdout: create 271/34 file after the snapshot 2023-10-30T12:31:48.143 INFO:tasks.workunit.client.0.smithi125.stdout: create 271/35 file after the snapshot 2023-10-30T12:31:48.143 INFO:tasks.workunit.client.0.smithi125.stdout: create 271/36 file after the snapshot 2023-10-30T12:31:48.143 INFO:tasks.workunit.client.0.smithi125.stdout: create 271/37 file after the snapshot 2023-10-30T12:31:48.144 INFO:tasks.workunit.client.0.smithi125.stdout: create 271/38 file after the snapshot 2023-10-30T12:31:48.144 INFO:tasks.workunit.client.0.smithi125.stdout: create 271/39 file after the snapshot 2023-10-30T12:31:48.144 INFO:tasks.workunit.client.0.smithi125.stdout: create 272/.snap/snap-subdir-test 2023-10-30T12:31:48.155 INFO:tasks.workunit.client.0.smithi125.stdout: create 272/30 file after the snapshot 2023-10-30T12:31:48.157 INFO:tasks.workunit.client.0.smithi125.stdout: create 272/31 file after the snapshot 2023-10-30T12:31:48.158 INFO:tasks.workunit.client.0.smithi125.stdout: create 272/32 file after the snapshot 2023-10-30T12:31:48.158 INFO:tasks.workunit.client.0.smithi125.stdout: create 272/33 file after the snapshot 2023-10-30T12:31:48.158 INFO:tasks.workunit.client.0.smithi125.stdout: create 272/34 file after the snapshot 2023-10-30T12:31:48.158 INFO:tasks.workunit.client.0.smithi125.stdout: create 272/35 file after the snapshot 2023-10-30T12:31:48.159 INFO:tasks.workunit.client.0.smithi125.stdout: create 272/36 file after the snapshot 2023-10-30T12:31:48.159 INFO:tasks.workunit.client.0.smithi125.stdout: create 272/37 file after the snapshot 2023-10-30T12:31:48.159 INFO:tasks.workunit.client.0.smithi125.stdout: create 272/38 file after the snapshot 2023-10-30T12:31:48.160 INFO:tasks.workunit.client.0.smithi125.stdout: create 272/39 file after the snapshot 2023-10-30T12:31:48.160 INFO:tasks.workunit.client.0.smithi125.stdout: create 273/.snap/snap-subdir-test 2023-10-30T12:31:48.185 INFO:tasks.workunit.client.0.smithi125.stdout: create 273/30 file after the snapshot 2023-10-30T12:31:48.186 INFO:tasks.workunit.client.0.smithi125.stdout: create 273/31 file after the snapshot 2023-10-30T12:31:48.186 INFO:tasks.workunit.client.0.smithi125.stdout: create 273/32 file after the snapshot 2023-10-30T12:31:48.187 INFO:tasks.workunit.client.0.smithi125.stdout: create 273/33 file after the snapshot 2023-10-30T12:31:48.187 INFO:tasks.workunit.client.0.smithi125.stdout: create 273/34 file after the snapshot 2023-10-30T12:31:48.187 INFO:tasks.workunit.client.0.smithi125.stdout: create 273/35 file after the snapshot 2023-10-30T12:31:48.187 INFO:tasks.workunit.client.0.smithi125.stdout: create 273/36 file after the snapshot 2023-10-30T12:31:48.187 INFO:tasks.workunit.client.0.smithi125.stdout: create 273/37 file after the snapshot 2023-10-30T12:31:48.188 INFO:tasks.workunit.client.0.smithi125.stdout: create 273/38 file after the snapshot 2023-10-30T12:31:48.188 INFO:tasks.workunit.client.0.smithi125.stdout: create 273/39 file after the snapshot 2023-10-30T12:31:48.188 INFO:tasks.workunit.client.0.smithi125.stdout: create 274/.snap/snap-subdir-test 2023-10-30T12:31:48.213 INFO:tasks.workunit.client.0.smithi125.stdout: create 274/30 file after the snapshot 2023-10-30T12:31:48.215 INFO:tasks.workunit.client.0.smithi125.stdout: create 274/31 file after the snapshot 2023-10-30T12:31:48.215 INFO:tasks.workunit.client.0.smithi125.stdout: create 274/32 file after the snapshot 2023-10-30T12:31:48.215 INFO:tasks.workunit.client.0.smithi125.stdout: create 274/33 file after the snapshot 2023-10-30T12:31:48.215 INFO:tasks.workunit.client.0.smithi125.stdout: create 274/34 file after the snapshot 2023-10-30T12:31:48.215 INFO:tasks.workunit.client.0.smithi125.stdout: create 274/35 file after the snapshot 2023-10-30T12:31:48.216 INFO:tasks.workunit.client.0.smithi125.stdout: create 274/36 file after the snapshot 2023-10-30T12:31:48.216 INFO:tasks.workunit.client.0.smithi125.stdout: create 274/37 file after the snapshot 2023-10-30T12:31:48.216 INFO:tasks.workunit.client.0.smithi125.stdout: create 274/38 file after the snapshot 2023-10-30T12:31:48.216 INFO:tasks.workunit.client.0.smithi125.stdout: create 274/39 file after the snapshot 2023-10-30T12:31:48.216 INFO:tasks.workunit.client.0.smithi125.stdout: create 275/.snap/snap-subdir-test 2023-10-30T12:31:48.237 INFO:tasks.workunit.client.0.smithi125.stdout: create 275/30 file after the snapshot 2023-10-30T12:31:48.239 INFO:tasks.workunit.client.0.smithi125.stdout: create 275/31 file after the snapshot 2023-10-30T12:31:48.239 INFO:tasks.workunit.client.0.smithi125.stdout: create 275/32 file after the snapshot 2023-10-30T12:31:48.239 INFO:tasks.workunit.client.0.smithi125.stdout: create 275/33 file after the snapshot 2023-10-30T12:31:48.239 INFO:tasks.workunit.client.0.smithi125.stdout: create 275/34 file after the snapshot 2023-10-30T12:31:48.239 INFO:tasks.workunit.client.0.smithi125.stdout: create 275/35 file after the snapshot 2023-10-30T12:31:48.240 INFO:tasks.workunit.client.0.smithi125.stdout: create 275/36 file after the snapshot 2023-10-30T12:31:48.240 INFO:tasks.workunit.client.0.smithi125.stdout: create 275/37 file after the snapshot 2023-10-30T12:31:48.240 INFO:tasks.workunit.client.0.smithi125.stdout: create 275/38 file after the snapshot 2023-10-30T12:31:48.240 INFO:tasks.workunit.client.0.smithi125.stdout: create 275/39 file after the snapshot 2023-10-30T12:31:48.240 INFO:tasks.workunit.client.0.smithi125.stdout: create 276/.snap/snap-subdir-test 2023-10-30T12:31:48.260 INFO:tasks.workunit.client.0.smithi125.stdout: create 276/30 file after the snapshot 2023-10-30T12:31:48.262 INFO:tasks.workunit.client.0.smithi125.stdout: create 276/31 file after the snapshot 2023-10-30T12:31:48.262 INFO:tasks.workunit.client.0.smithi125.stdout: create 276/32 file after the snapshot 2023-10-30T12:31:48.263 INFO:tasks.workunit.client.0.smithi125.stdout: create 276/33 file after the snapshot 2023-10-30T12:31:48.263 INFO:tasks.workunit.client.0.smithi125.stdout: create 276/34 file after the snapshot 2023-10-30T12:31:48.263 INFO:tasks.workunit.client.0.smithi125.stdout: create 276/35 file after the snapshot 2023-10-30T12:31:48.263 INFO:tasks.workunit.client.0.smithi125.stdout: create 276/36 file after the snapshot 2023-10-30T12:31:48.263 INFO:tasks.workunit.client.0.smithi125.stdout: create 276/37 file after the snapshot 2023-10-30T12:31:48.263 INFO:tasks.workunit.client.0.smithi125.stdout: create 276/38 file after the snapshot 2023-10-30T12:31:48.264 INFO:tasks.workunit.client.0.smithi125.stdout: create 276/39 file after the snapshot 2023-10-30T12:31:48.264 INFO:tasks.workunit.client.0.smithi125.stdout: create 277/.snap/snap-subdir-test 2023-10-30T12:31:48.874 INFO:tasks.workunit.client.0.smithi125.stdout: create 277/30 file after the snapshot 2023-10-30T12:31:48.953 INFO:tasks.workunit.client.0.smithi125.stdout: create 277/31 file after the snapshot 2023-10-30T12:31:48.953 INFO:tasks.workunit.client.0.smithi125.stdout: create 277/32 file after the snapshot 2023-10-30T12:31:48.954 INFO:tasks.workunit.client.0.smithi125.stdout: create 277/33 file after the snapshot 2023-10-30T12:31:48.954 INFO:tasks.workunit.client.0.smithi125.stdout: create 277/34 file after the snapshot 2023-10-30T12:31:48.955 INFO:tasks.workunit.client.0.smithi125.stdout: create 277/35 file after the snapshot 2023-10-30T12:31:48.955 INFO:tasks.workunit.client.0.smithi125.stdout: create 277/36 file after the snapshot 2023-10-30T12:31:48.955 INFO:tasks.workunit.client.0.smithi125.stdout: create 277/37 file after the snapshot 2023-10-30T12:31:48.956 INFO:tasks.workunit.client.0.smithi125.stdout: create 277/38 file after the snapshot 2023-10-30T12:31:48.956 INFO:tasks.workunit.client.0.smithi125.stdout: create 277/39 file after the snapshot 2023-10-30T12:31:48.956 INFO:tasks.workunit.client.0.smithi125.stdout: create 278/.snap/snap-subdir-test 2023-10-30T12:31:50.010 INFO:tasks.workunit.client.0.smithi125.stdout: create 278/30 file after the snapshot 2023-10-30T12:31:50.013 INFO:tasks.workunit.client.0.smithi125.stdout: create 278/31 file after the snapshot 2023-10-30T12:31:50.013 INFO:tasks.workunit.client.0.smithi125.stdout: create 278/32 file after the snapshot 2023-10-30T12:31:50.013 INFO:tasks.workunit.client.0.smithi125.stdout: create 278/33 file after the snapshot 2023-10-30T12:31:50.013 INFO:tasks.workunit.client.0.smithi125.stdout: create 278/34 file after the snapshot 2023-10-30T12:31:50.014 INFO:tasks.workunit.client.0.smithi125.stdout: create 278/35 file after the snapshot 2023-10-30T12:31:50.014 INFO:tasks.workunit.client.0.smithi125.stdout: create 278/36 file after the snapshot 2023-10-30T12:31:50.014 INFO:tasks.workunit.client.0.smithi125.stdout: create 278/37 file after the snapshot 2023-10-30T12:31:50.014 INFO:tasks.workunit.client.0.smithi125.stdout: create 278/38 file after the snapshot 2023-10-30T12:31:50.014 INFO:tasks.workunit.client.0.smithi125.stdout: create 278/39 file after the snapshot 2023-10-30T12:31:50.015 INFO:tasks.workunit.client.0.smithi125.stdout: create 279/.snap/snap-subdir-test 2023-10-30T12:31:50.559 INFO:tasks.workunit.client.0.smithi125.stdout: create 279/30 file after the snapshot 2023-10-30T12:31:50.560 INFO:tasks.workunit.client.0.smithi125.stdout: create 279/31 file after the snapshot 2023-10-30T12:31:50.561 INFO:tasks.workunit.client.0.smithi125.stdout: create 279/32 file after the snapshot 2023-10-30T12:31:50.561 INFO:tasks.workunit.client.0.smithi125.stdout: create 279/33 file after the snapshot 2023-10-30T12:31:50.561 INFO:tasks.workunit.client.0.smithi125.stdout: create 279/34 file after the snapshot 2023-10-30T12:31:50.562 INFO:tasks.workunit.client.0.smithi125.stdout: create 279/35 file after the snapshot 2023-10-30T12:31:50.562 INFO:tasks.workunit.client.0.smithi125.stdout: create 279/36 file after the snapshot 2023-10-30T12:31:50.562 INFO:tasks.workunit.client.0.smithi125.stdout: create 279/37 file after the snapshot 2023-10-30T12:31:50.563 INFO:tasks.workunit.client.0.smithi125.stdout: create 279/38 file after the snapshot 2023-10-30T12:31:50.563 INFO:tasks.workunit.client.0.smithi125.stdout: create 279/39 file after the snapshot 2023-10-30T12:31:50.563 INFO:tasks.workunit.client.0.smithi125.stdout: create 280/.snap/snap-subdir-test 2023-10-30T12:31:50.579 INFO:tasks.workunit.client.0.smithi125.stdout: create 280/30 file after the snapshot 2023-10-30T12:31:50.580 INFO:tasks.workunit.client.0.smithi125.stdout: create 280/31 file after the snapshot 2023-10-30T12:31:50.581 INFO:tasks.workunit.client.0.smithi125.stdout: create 280/32 file after the snapshot 2023-10-30T12:31:50.581 INFO:tasks.workunit.client.0.smithi125.stdout: create 280/33 file after the snapshot 2023-10-30T12:31:50.582 INFO:tasks.workunit.client.0.smithi125.stdout: create 280/34 file after the snapshot 2023-10-30T12:31:50.582 INFO:tasks.workunit.client.0.smithi125.stdout: create 280/35 file after the snapshot 2023-10-30T12:31:50.582 INFO:tasks.workunit.client.0.smithi125.stdout: create 280/36 file after the snapshot 2023-10-30T12:31:50.583 INFO:tasks.workunit.client.0.smithi125.stdout: create 280/37 file after the snapshot 2023-10-30T12:31:50.583 INFO:tasks.workunit.client.0.smithi125.stdout: create 280/38 file after the snapshot 2023-10-30T12:31:50.583 INFO:tasks.workunit.client.0.smithi125.stdout: create 280/39 file after the snapshot 2023-10-30T12:31:50.583 INFO:tasks.workunit.client.0.smithi125.stdout: create 281/.snap/snap-subdir-test 2023-10-30T12:31:50.608 INFO:tasks.workunit.client.0.smithi125.stdout: create 281/30 file after the snapshot 2023-10-30T12:31:50.609 INFO:tasks.workunit.client.0.smithi125.stdout: create 281/31 file after the snapshot 2023-10-30T12:31:50.610 INFO:tasks.workunit.client.0.smithi125.stdout: create 281/32 file after the snapshot 2023-10-30T12:31:50.610 INFO:tasks.workunit.client.0.smithi125.stdout: create 281/33 file after the snapshot 2023-10-30T12:31:50.610 INFO:tasks.workunit.client.0.smithi125.stdout: create 281/34 file after the snapshot 2023-10-30T12:31:50.610 INFO:tasks.workunit.client.0.smithi125.stdout: create 281/35 file after the snapshot 2023-10-30T12:31:50.611 INFO:tasks.workunit.client.0.smithi125.stdout: create 281/36 file after the snapshot 2023-10-30T12:31:50.611 INFO:tasks.workunit.client.0.smithi125.stdout: create 281/37 file after the snapshot 2023-10-30T12:31:50.612 INFO:tasks.workunit.client.0.smithi125.stdout: create 281/38 file after the snapshot 2023-10-30T12:31:50.612 INFO:tasks.workunit.client.0.smithi125.stdout: create 281/39 file after the snapshot 2023-10-30T12:31:50.612 INFO:tasks.workunit.client.0.smithi125.stdout: create 282/.snap/snap-subdir-test 2023-10-30T12:31:50.889 INFO:tasks.workunit.client.0.smithi125.stdout: create 282/30 file after the snapshot 2023-10-30T12:31:50.892 INFO:tasks.workunit.client.0.smithi125.stdout: create 282/31 file after the snapshot 2023-10-30T12:31:50.892 INFO:tasks.workunit.client.0.smithi125.stdout: create 282/32 file after the snapshot 2023-10-30T12:31:50.892 INFO:tasks.workunit.client.0.smithi125.stdout: create 282/33 file after the snapshot 2023-10-30T12:31:50.893 INFO:tasks.workunit.client.0.smithi125.stdout: create 282/34 file after the snapshot 2023-10-30T12:31:50.894 INFO:tasks.workunit.client.0.smithi125.stdout: create 282/35 file after the snapshot 2023-10-30T12:31:50.894 INFO:tasks.workunit.client.0.smithi125.stdout: create 282/36 file after the snapshot 2023-10-30T12:31:50.894 INFO:tasks.workunit.client.0.smithi125.stdout: create 282/37 file after the snapshot 2023-10-30T12:31:50.894 INFO:tasks.workunit.client.0.smithi125.stdout: create 282/38 file after the snapshot 2023-10-30T12:31:50.894 INFO:tasks.workunit.client.0.smithi125.stdout: create 282/39 file after the snapshot 2023-10-30T12:31:50.895 INFO:tasks.workunit.client.0.smithi125.stdout: create 283/.snap/snap-subdir-test 2023-10-30T12:31:50.914 INFO:tasks.workunit.client.0.smithi125.stdout: create 283/30 file after the snapshot 2023-10-30T12:31:50.915 INFO:tasks.workunit.client.0.smithi125.stdout: create 283/31 file after the snapshot 2023-10-30T12:31:50.916 INFO:tasks.workunit.client.0.smithi125.stdout: create 283/32 file after the snapshot 2023-10-30T12:31:50.916 INFO:tasks.workunit.client.0.smithi125.stdout: create 283/33 file after the snapshot 2023-10-30T12:31:50.916 INFO:tasks.workunit.client.0.smithi125.stdout: create 283/34 file after the snapshot 2023-10-30T12:31:50.917 INFO:tasks.workunit.client.0.smithi125.stdout: create 283/35 file after the snapshot 2023-10-30T12:31:50.918 INFO:tasks.workunit.client.0.smithi125.stdout: create 283/36 file after the snapshot 2023-10-30T12:31:50.918 INFO:tasks.workunit.client.0.smithi125.stdout: create 283/37 file after the snapshot 2023-10-30T12:31:50.918 INFO:tasks.workunit.client.0.smithi125.stdout: create 283/38 file after the snapshot 2023-10-30T12:31:50.918 INFO:tasks.workunit.client.0.smithi125.stdout: create 283/39 file after the snapshot 2023-10-30T12:31:50.918 INFO:tasks.workunit.client.0.smithi125.stdout: create 284/.snap/snap-subdir-test 2023-10-30T12:31:50.948 INFO:tasks.workunit.client.0.smithi125.stdout: create 284/30 file after the snapshot 2023-10-30T12:31:50.950 INFO:tasks.workunit.client.0.smithi125.stdout: create 284/31 file after the snapshot 2023-10-30T12:31:50.950 INFO:tasks.workunit.client.0.smithi125.stdout: create 284/32 file after the snapshot 2023-10-30T12:31:50.950 INFO:tasks.workunit.client.0.smithi125.stdout: create 284/33 file after the snapshot 2023-10-30T12:31:50.950 INFO:tasks.workunit.client.0.smithi125.stdout: create 284/34 file after the snapshot 2023-10-30T12:31:50.951 INFO:tasks.workunit.client.0.smithi125.stdout: create 284/35 file after the snapshot 2023-10-30T12:31:50.951 INFO:tasks.workunit.client.0.smithi125.stdout: create 284/36 file after the snapshot 2023-10-30T12:31:50.951 INFO:tasks.workunit.client.0.smithi125.stdout: create 284/37 file after the snapshot 2023-10-30T12:31:50.951 INFO:tasks.workunit.client.0.smithi125.stdout: create 284/38 file after the snapshot 2023-10-30T12:31:50.951 INFO:tasks.workunit.client.0.smithi125.stdout: create 284/39 file after the snapshot 2023-10-30T12:31:50.951 INFO:tasks.workunit.client.0.smithi125.stdout: create 285/.snap/snap-subdir-test 2023-10-30T12:31:51.493 INFO:tasks.workunit.client.0.smithi125.stdout: create 285/30 file after the snapshot 2023-10-30T12:31:51.494 INFO:tasks.workunit.client.0.smithi125.stdout: create 285/31 file after the snapshot 2023-10-30T12:31:51.495 INFO:tasks.workunit.client.0.smithi125.stdout: create 285/32 file after the snapshot 2023-10-30T12:31:51.495 INFO:tasks.workunit.client.0.smithi125.stdout: create 285/33 file after the snapshot 2023-10-30T12:31:51.495 INFO:tasks.workunit.client.0.smithi125.stdout: create 285/34 file after the snapshot 2023-10-30T12:31:51.495 INFO:tasks.workunit.client.0.smithi125.stdout: create 285/35 file after the snapshot 2023-10-30T12:31:51.495 INFO:tasks.workunit.client.0.smithi125.stdout: create 285/36 file after the snapshot 2023-10-30T12:31:51.496 INFO:tasks.workunit.client.0.smithi125.stdout: create 285/37 file after the snapshot 2023-10-30T12:31:51.496 INFO:tasks.workunit.client.0.smithi125.stdout: create 285/38 file after the snapshot 2023-10-30T12:31:51.496 INFO:tasks.workunit.client.0.smithi125.stdout: create 285/39 file after the snapshot 2023-10-30T12:31:51.496 INFO:tasks.workunit.client.0.smithi125.stdout: create 286/.snap/snap-subdir-test 2023-10-30T12:31:51.522 INFO:tasks.workunit.client.0.smithi125.stdout: create 286/30 file after the snapshot 2023-10-30T12:31:51.524 INFO:tasks.workunit.client.0.smithi125.stdout: create 286/31 file after the snapshot 2023-10-30T12:31:51.524 INFO:tasks.workunit.client.0.smithi125.stdout: create 286/32 file after the snapshot 2023-10-30T12:31:51.524 INFO:tasks.workunit.client.0.smithi125.stdout: create 286/33 file after the snapshot 2023-10-30T12:31:51.525 INFO:tasks.workunit.client.0.smithi125.stdout: create 286/34 file after the snapshot 2023-10-30T12:31:51.525 INFO:tasks.workunit.client.0.smithi125.stdout: create 286/35 file after the snapshot 2023-10-30T12:31:51.525 INFO:tasks.workunit.client.0.smithi125.stdout: create 286/36 file after the snapshot 2023-10-30T12:31:51.525 INFO:tasks.workunit.client.0.smithi125.stdout: create 286/37 file after the snapshot 2023-10-30T12:31:51.526 INFO:tasks.workunit.client.0.smithi125.stdout: create 286/38 file after the snapshot 2023-10-30T12:31:51.526 INFO:tasks.workunit.client.0.smithi125.stdout: create 286/39 file after the snapshot 2023-10-30T12:31:51.526 INFO:tasks.workunit.client.0.smithi125.stdout: create 287/.snap/snap-subdir-test 2023-10-30T12:31:51.549 INFO:tasks.workunit.client.0.smithi125.stdout: create 287/30 file after the snapshot 2023-10-30T12:31:51.551 INFO:tasks.workunit.client.0.smithi125.stdout: create 287/31 file after the snapshot 2023-10-30T12:31:51.551 INFO:tasks.workunit.client.0.smithi125.stdout: create 287/32 file after the snapshot 2023-10-30T12:31:51.551 INFO:tasks.workunit.client.0.smithi125.stdout: create 287/33 file after the snapshot 2023-10-30T12:31:51.552 INFO:tasks.workunit.client.0.smithi125.stdout: create 287/34 file after the snapshot 2023-10-30T12:31:51.552 INFO:tasks.workunit.client.0.smithi125.stdout: create 287/35 file after the snapshot 2023-10-30T12:31:51.552 INFO:tasks.workunit.client.0.smithi125.stdout: create 287/36 file after the snapshot 2023-10-30T12:31:51.552 INFO:tasks.workunit.client.0.smithi125.stdout: create 287/37 file after the snapshot 2023-10-30T12:31:51.552 INFO:tasks.workunit.client.0.smithi125.stdout: create 287/38 file after the snapshot 2023-10-30T12:31:51.553 INFO:tasks.workunit.client.0.smithi125.stdout: create 287/39 file after the snapshot 2023-10-30T12:31:51.553 INFO:tasks.workunit.client.0.smithi125.stdout: create 288/.snap/snap-subdir-test 2023-10-30T12:31:51.576 INFO:tasks.workunit.client.0.smithi125.stdout: create 288/30 file after the snapshot 2023-10-30T12:31:51.578 INFO:tasks.workunit.client.0.smithi125.stdout: create 288/31 file after the snapshot 2023-10-30T12:31:51.578 INFO:tasks.workunit.client.0.smithi125.stdout: create 288/32 file after the snapshot 2023-10-30T12:31:51.578 INFO:tasks.workunit.client.0.smithi125.stdout: create 288/33 file after the snapshot 2023-10-30T12:31:51.579 INFO:tasks.workunit.client.0.smithi125.stdout: create 288/34 file after the snapshot 2023-10-30T12:31:51.579 INFO:tasks.workunit.client.0.smithi125.stdout: create 288/35 file after the snapshot 2023-10-30T12:31:51.579 INFO:tasks.workunit.client.0.smithi125.stdout: create 288/36 file after the snapshot 2023-10-30T12:31:51.579 INFO:tasks.workunit.client.0.smithi125.stdout: create 288/37 file after the snapshot 2023-10-30T12:31:51.579 INFO:tasks.workunit.client.0.smithi125.stdout: create 288/38 file after the snapshot 2023-10-30T12:31:51.579 INFO:tasks.workunit.client.0.smithi125.stdout: create 288/39 file after the snapshot 2023-10-30T12:31:51.580 INFO:tasks.workunit.client.0.smithi125.stdout: create 289/.snap/snap-subdir-test 2023-10-30T12:31:51.603 INFO:tasks.workunit.client.0.smithi125.stdout: create 289/30 file after the snapshot 2023-10-30T12:31:51.605 INFO:tasks.workunit.client.0.smithi125.stdout: create 289/31 file after the snapshot 2023-10-30T12:31:51.605 INFO:tasks.workunit.client.0.smithi125.stdout: create 289/32 file after the snapshot 2023-10-30T12:31:51.605 INFO:tasks.workunit.client.0.smithi125.stdout: create 289/33 file after the snapshot 2023-10-30T12:31:51.605 INFO:tasks.workunit.client.0.smithi125.stdout: create 289/34 file after the snapshot 2023-10-30T12:31:51.605 INFO:tasks.workunit.client.0.smithi125.stdout: create 289/35 file after the snapshot 2023-10-30T12:31:51.606 INFO:tasks.workunit.client.0.smithi125.stdout: create 289/36 file after the snapshot 2023-10-30T12:31:51.606 INFO:tasks.workunit.client.0.smithi125.stdout: create 289/37 file after the snapshot 2023-10-30T12:31:51.606 INFO:tasks.workunit.client.0.smithi125.stdout: create 289/38 file after the snapshot 2023-10-30T12:31:51.606 INFO:tasks.workunit.client.0.smithi125.stdout: create 289/39 file after the snapshot 2023-10-30T12:31:51.606 INFO:tasks.workunit.client.0.smithi125.stdout: create 290/.snap/snap-subdir-test 2023-10-30T12:31:51.628 INFO:tasks.workunit.client.0.smithi125.stdout: create 290/30 file after the snapshot 2023-10-30T12:31:51.630 INFO:tasks.workunit.client.0.smithi125.stdout: create 290/31 file after the snapshot 2023-10-30T12:31:51.630 INFO:tasks.workunit.client.0.smithi125.stdout: create 290/32 file after the snapshot 2023-10-30T12:31:51.630 INFO:tasks.workunit.client.0.smithi125.stdout: create 290/33 file after the snapshot 2023-10-30T12:31:51.631 INFO:tasks.workunit.client.0.smithi125.stdout: create 290/34 file after the snapshot 2023-10-30T12:31:51.631 INFO:tasks.workunit.client.0.smithi125.stdout: create 290/35 file after the snapshot 2023-10-30T12:31:51.631 INFO:tasks.workunit.client.0.smithi125.stdout: create 290/36 file after the snapshot 2023-10-30T12:31:51.631 INFO:tasks.workunit.client.0.smithi125.stdout: create 290/37 file after the snapshot 2023-10-30T12:31:51.631 INFO:tasks.workunit.client.0.smithi125.stdout: create 290/38 file after the snapshot 2023-10-30T12:31:51.632 INFO:tasks.workunit.client.0.smithi125.stdout: create 290/39 file after the snapshot 2023-10-30T12:31:51.632 INFO:tasks.workunit.client.0.smithi125.stdout: create 291/.snap/snap-subdir-test 2023-10-30T12:31:51.653 INFO:tasks.workunit.client.0.smithi125.stdout: create 291/30 file after the snapshot 2023-10-30T12:31:51.655 INFO:tasks.workunit.client.0.smithi125.stdout: create 291/31 file after the snapshot 2023-10-30T12:31:51.655 INFO:tasks.workunit.client.0.smithi125.stdout: create 291/32 file after the snapshot 2023-10-30T12:31:51.655 INFO:tasks.workunit.client.0.smithi125.stdout: create 291/33 file after the snapshot 2023-10-30T12:31:51.655 INFO:tasks.workunit.client.0.smithi125.stdout: create 291/34 file after the snapshot 2023-10-30T12:31:51.655 INFO:tasks.workunit.client.0.smithi125.stdout: create 291/35 file after the snapshot 2023-10-30T12:31:51.656 INFO:tasks.workunit.client.0.smithi125.stdout: create 291/36 file after the snapshot 2023-10-30T12:31:51.656 INFO:tasks.workunit.client.0.smithi125.stdout: create 291/37 file after the snapshot 2023-10-30T12:31:51.656 INFO:tasks.workunit.client.0.smithi125.stdout: create 291/38 file after the snapshot 2023-10-30T12:31:51.656 INFO:tasks.workunit.client.0.smithi125.stdout: create 291/39 file after the snapshot 2023-10-30T12:31:51.656 INFO:tasks.workunit.client.0.smithi125.stdout: create 292/.snap/snap-subdir-test 2023-10-30T12:31:52.326 INFO:tasks.workunit.client.0.smithi125.stdout: create 292/30 file after the snapshot 2023-10-30T12:31:52.328 INFO:tasks.workunit.client.0.smithi125.stdout: create 292/31 file after the snapshot 2023-10-30T12:31:52.328 INFO:tasks.workunit.client.0.smithi125.stdout: create 292/32 file after the snapshot 2023-10-30T12:31:52.328 INFO:tasks.workunit.client.0.smithi125.stdout: create 292/33 file after the snapshot 2023-10-30T12:31:52.328 INFO:tasks.workunit.client.0.smithi125.stdout: create 292/34 file after the snapshot 2023-10-30T12:31:52.329 INFO:tasks.workunit.client.0.smithi125.stdout: create 292/35 file after the snapshot 2023-10-30T12:31:52.329 INFO:tasks.workunit.client.0.smithi125.stdout: create 292/36 file after the snapshot 2023-10-30T12:31:52.329 INFO:tasks.workunit.client.0.smithi125.stdout: create 292/37 file after the snapshot 2023-10-30T12:31:52.329 INFO:tasks.workunit.client.0.smithi125.stdout: create 292/38 file after the snapshot 2023-10-30T12:31:52.329 INFO:tasks.workunit.client.0.smithi125.stdout: create 292/39 file after the snapshot 2023-10-30T12:31:52.330 INFO:tasks.workunit.client.0.smithi125.stdout: create 293/.snap/snap-subdir-test 2023-10-30T12:31:52.345 INFO:tasks.workunit.client.0.smithi125.stdout: create 293/30 file after the snapshot 2023-10-30T12:31:52.347 INFO:tasks.workunit.client.0.smithi125.stdout: create 293/31 file after the snapshot 2023-10-30T12:31:52.347 INFO:tasks.workunit.client.0.smithi125.stdout: create 293/32 file after the snapshot 2023-10-30T12:31:52.347 INFO:tasks.workunit.client.0.smithi125.stdout: create 293/33 file after the snapshot 2023-10-30T12:31:52.347 INFO:tasks.workunit.client.0.smithi125.stdout: create 293/34 file after the snapshot 2023-10-30T12:31:52.347 INFO:tasks.workunit.client.0.smithi125.stdout: create 293/35 file after the snapshot 2023-10-30T12:31:52.348 INFO:tasks.workunit.client.0.smithi125.stdout: create 293/36 file after the snapshot 2023-10-30T12:31:52.348 INFO:tasks.workunit.client.0.smithi125.stdout: create 293/37 file after the snapshot 2023-10-30T12:31:52.348 INFO:tasks.workunit.client.0.smithi125.stdout: create 293/38 file after the snapshot 2023-10-30T12:31:52.348 INFO:tasks.workunit.client.0.smithi125.stdout: create 293/39 file after the snapshot 2023-10-30T12:31:52.348 INFO:tasks.workunit.client.0.smithi125.stdout: create 294/.snap/snap-subdir-test 2023-10-30T12:31:52.371 INFO:tasks.workunit.client.0.smithi125.stdout: create 294/30 file after the snapshot 2023-10-30T12:31:52.373 INFO:tasks.workunit.client.0.smithi125.stdout: create 294/31 file after the snapshot 2023-10-30T12:31:52.373 INFO:tasks.workunit.client.0.smithi125.stdout: create 294/32 file after the snapshot 2023-10-30T12:31:52.373 INFO:tasks.workunit.client.0.smithi125.stdout: create 294/33 file after the snapshot 2023-10-30T12:31:52.373 INFO:tasks.workunit.client.0.smithi125.stdout: create 294/34 file after the snapshot 2023-10-30T12:31:52.374 INFO:tasks.workunit.client.0.smithi125.stdout: create 294/35 file after the snapshot 2023-10-30T12:31:52.374 INFO:tasks.workunit.client.0.smithi125.stdout: create 294/36 file after the snapshot 2023-10-30T12:31:52.374 INFO:tasks.workunit.client.0.smithi125.stdout: create 294/37 file after the snapshot 2023-10-30T12:31:52.374 INFO:tasks.workunit.client.0.smithi125.stdout: create 294/38 file after the snapshot 2023-10-30T12:31:52.374 INFO:tasks.workunit.client.0.smithi125.stdout: create 294/39 file after the snapshot 2023-10-30T12:31:52.375 INFO:tasks.workunit.client.0.smithi125.stdout: create 295/.snap/snap-subdir-test 2023-10-30T12:31:52.392 INFO:tasks.workunit.client.0.smithi125.stdout: create 295/30 file after the snapshot 2023-10-30T12:31:52.394 INFO:tasks.workunit.client.0.smithi125.stdout: create 295/31 file after the snapshot 2023-10-30T12:31:52.394 INFO:tasks.workunit.client.0.smithi125.stdout: create 295/32 file after the snapshot 2023-10-30T12:31:52.395 INFO:tasks.workunit.client.0.smithi125.stdout: create 295/33 file after the snapshot 2023-10-30T12:31:52.395 INFO:tasks.workunit.client.0.smithi125.stdout: create 295/34 file after the snapshot 2023-10-30T12:31:52.395 INFO:tasks.workunit.client.0.smithi125.stdout: create 295/35 file after the snapshot 2023-10-30T12:31:52.395 INFO:tasks.workunit.client.0.smithi125.stdout: create 295/36 file after the snapshot 2023-10-30T12:31:52.395 INFO:tasks.workunit.client.0.smithi125.stdout: create 295/37 file after the snapshot 2023-10-30T12:31:52.396 INFO:tasks.workunit.client.0.smithi125.stdout: create 295/38 file after the snapshot 2023-10-30T12:31:52.396 INFO:tasks.workunit.client.0.smithi125.stdout: create 295/39 file after the snapshot 2023-10-30T12:31:52.396 INFO:tasks.workunit.client.0.smithi125.stdout: create 296/.snap/snap-subdir-test 2023-10-30T12:31:52.418 INFO:tasks.workunit.client.0.smithi125.stdout: create 296/30 file after the snapshot 2023-10-30T12:31:52.419 INFO:tasks.workunit.client.0.smithi125.stdout: create 296/31 file after the snapshot 2023-10-30T12:31:52.420 INFO:tasks.workunit.client.0.smithi125.stdout: create 296/32 file after the snapshot 2023-10-30T12:31:52.420 INFO:tasks.workunit.client.0.smithi125.stdout: create 296/33 file after the snapshot 2023-10-30T12:31:52.420 INFO:tasks.workunit.client.0.smithi125.stdout: create 296/34 file after the snapshot 2023-10-30T12:31:52.420 INFO:tasks.workunit.client.0.smithi125.stdout: create 296/35 file after the snapshot 2023-10-30T12:31:52.420 INFO:tasks.workunit.client.0.smithi125.stdout: create 296/36 file after the snapshot 2023-10-30T12:31:52.421 INFO:tasks.workunit.client.0.smithi125.stdout: create 296/37 file after the snapshot 2023-10-30T12:31:52.421 INFO:tasks.workunit.client.0.smithi125.stdout: create 296/38 file after the snapshot 2023-10-30T12:31:52.421 INFO:tasks.workunit.client.0.smithi125.stdout: create 296/39 file after the snapshot 2023-10-30T12:31:52.421 INFO:tasks.workunit.client.0.smithi125.stdout: create 297/.snap/snap-subdir-test 2023-10-30T12:31:52.439 INFO:tasks.workunit.client.0.smithi125.stdout: create 297/30 file after the snapshot 2023-10-30T12:31:52.440 INFO:tasks.workunit.client.0.smithi125.stdout: create 297/31 file after the snapshot 2023-10-30T12:31:52.440 INFO:tasks.workunit.client.0.smithi125.stdout: create 297/32 file after the snapshot 2023-10-30T12:31:52.441 INFO:tasks.workunit.client.0.smithi125.stdout: create 297/33 file after the snapshot 2023-10-30T12:31:52.441 INFO:tasks.workunit.client.0.smithi125.stdout: create 297/34 file after the snapshot 2023-10-30T12:31:52.441 INFO:tasks.workunit.client.0.smithi125.stdout: create 297/35 file after the snapshot 2023-10-30T12:31:52.441 INFO:tasks.workunit.client.0.smithi125.stdout: create 297/36 file after the snapshot 2023-10-30T12:31:52.441 INFO:tasks.workunit.client.0.smithi125.stdout: create 297/37 file after the snapshot 2023-10-30T12:31:52.442 INFO:tasks.workunit.client.0.smithi125.stdout: create 297/38 file after the snapshot 2023-10-30T12:31:52.442 INFO:tasks.workunit.client.0.smithi125.stdout: create 297/39 file after the snapshot 2023-10-30T12:31:52.442 INFO:tasks.workunit.client.0.smithi125.stdout: create 298/.snap/snap-subdir-test 2023-10-30T12:31:52.717 INFO:tasks.workunit.client.0.smithi125.stdout: create 298/30 file after the snapshot 2023-10-30T12:31:52.719 INFO:tasks.workunit.client.0.smithi125.stdout: create 298/31 file after the snapshot 2023-10-30T12:31:52.719 INFO:tasks.workunit.client.0.smithi125.stdout: create 298/32 file after the snapshot 2023-10-30T12:31:52.719 INFO:tasks.workunit.client.0.smithi125.stdout: create 298/33 file after the snapshot 2023-10-30T12:31:52.719 INFO:tasks.workunit.client.0.smithi125.stdout: create 298/34 file after the snapshot 2023-10-30T12:31:52.720 INFO:tasks.workunit.client.0.smithi125.stdout: create 298/35 file after the snapshot 2023-10-30T12:31:52.720 INFO:tasks.workunit.client.0.smithi125.stdout: create 298/36 file after the snapshot 2023-10-30T12:31:52.720 INFO:tasks.workunit.client.0.smithi125.stdout: create 298/37 file after the snapshot 2023-10-30T12:31:52.720 INFO:tasks.workunit.client.0.smithi125.stdout: create 298/38 file after the snapshot 2023-10-30T12:31:52.720 INFO:tasks.workunit.client.0.smithi125.stdout: create 298/39 file after the snapshot 2023-10-30T12:31:52.721 INFO:tasks.workunit.client.0.smithi125.stdout: create 299/.snap/snap-subdir-test 2023-10-30T12:31:52.741 INFO:tasks.workunit.client.0.smithi125.stdout: create 299/30 file after the snapshot 2023-10-30T12:31:52.743 INFO:tasks.workunit.client.0.smithi125.stdout: create 299/31 file after the snapshot 2023-10-30T12:31:52.743 INFO:tasks.workunit.client.0.smithi125.stdout: create 299/32 file after the snapshot 2023-10-30T12:31:52.743 INFO:tasks.workunit.client.0.smithi125.stdout: create 299/33 file after the snapshot 2023-10-30T12:31:52.743 INFO:tasks.workunit.client.0.smithi125.stdout: create 299/34 file after the snapshot 2023-10-30T12:31:52.744 INFO:tasks.workunit.client.0.smithi125.stdout: create 299/35 file after the snapshot 2023-10-30T12:31:52.744 INFO:tasks.workunit.client.0.smithi125.stdout: create 299/36 file after the snapshot 2023-10-30T12:31:52.744 INFO:tasks.workunit.client.0.smithi125.stdout: create 299/37 file after the snapshot 2023-10-30T12:31:52.744 INFO:tasks.workunit.client.0.smithi125.stdout: create 299/38 file after the snapshot 2023-10-30T12:31:52.744 INFO:tasks.workunit.client.0.smithi125.stdout: create 299/39 file after the snapshot 2023-10-30T12:31:52.745 INFO:tasks.workunit.client.0.smithi125.stdout:Take second snapshot .snap/test2 2023-10-30T12:31:57.135 INFO:tasks.workunit.client.0.smithi125.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2023-10-30T12:31:58.685 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1' -> 'copyofsnap1/test1' 2023-10-30T12:31:58.686 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/100' -> 'copyofsnap1/test1/100' 2023-10-30T12:31:58.686 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/100/11' -> 'copyofsnap1/test1/100/11' 2023-10-30T12:31:58.687 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/100/12' -> 'copyofsnap1/test1/100/12' 2023-10-30T12:31:58.687 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/100/13' -> 'copyofsnap1/test1/100/13' 2023-10-30T12:31:58.687 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/100/14' -> 'copyofsnap1/test1/100/14' 2023-10-30T12:31:58.688 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/100/15' -> 'copyofsnap1/test1/100/15' 2023-10-30T12:31:58.688 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/100/16' -> 'copyofsnap1/test1/100/16' 2023-10-30T12:31:58.688 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/100/17' -> 'copyofsnap1/test1/100/17' 2023-10-30T12:31:58.688 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/100/18' -> 'copyofsnap1/test1/100/18' 2023-10-30T12:31:58.688 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/100/19' -> 'copyofsnap1/test1/100/19' 2023-10-30T12:31:58.689 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/100/20' -> 'copyofsnap1/test1/100/20' 2023-10-30T12:31:58.689 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/100/10' -> 'copyofsnap1/test1/100/10' 2023-10-30T12:31:58.689 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/101' -> 'copyofsnap1/test1/101' 2023-10-30T12:31:58.689 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/101/11' -> 'copyofsnap1/test1/101/11' 2023-10-30T12:31:58.690 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/101/12' -> 'copyofsnap1/test1/101/12' 2023-10-30T12:31:58.690 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/101/13' -> 'copyofsnap1/test1/101/13' 2023-10-30T12:31:58.690 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/101/14' -> 'copyofsnap1/test1/101/14' 2023-10-30T12:31:58.690 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/101/15' -> 'copyofsnap1/test1/101/15' 2023-10-30T12:31:58.690 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/101/16' -> 'copyofsnap1/test1/101/16' 2023-10-30T12:31:58.691 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/101/17' -> 'copyofsnap1/test1/101/17' 2023-10-30T12:31:58.691 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/101/18' -> 'copyofsnap1/test1/101/18' 2023-10-30T12:31:58.691 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/101/19' -> 'copyofsnap1/test1/101/19' 2023-10-30T12:31:58.691 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/101/20' -> 'copyofsnap1/test1/101/20' 2023-10-30T12:31:58.691 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/101/10' -> 'copyofsnap1/test1/101/10' 2023-10-30T12:31:58.692 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/102' -> 'copyofsnap1/test1/102' 2023-10-30T12:31:58.692 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/102/11' -> 'copyofsnap1/test1/102/11' 2023-10-30T12:31:58.692 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/102/12' -> 'copyofsnap1/test1/102/12' 2023-10-30T12:31:58.692 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/102/13' -> 'copyofsnap1/test1/102/13' 2023-10-30T12:31:58.692 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/102/14' -> 'copyofsnap1/test1/102/14' 2023-10-30T12:31:58.692 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/102/15' -> 'copyofsnap1/test1/102/15' 2023-10-30T12:31:58.693 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/102/16' -> 'copyofsnap1/test1/102/16' 2023-10-30T12:31:58.693 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/102/17' -> 'copyofsnap1/test1/102/17' 2023-10-30T12:31:58.693 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/102/18' -> 'copyofsnap1/test1/102/18' 2023-10-30T12:31:58.693 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/102/19' -> 'copyofsnap1/test1/102/19' 2023-10-30T12:31:58.693 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/102/20' -> 'copyofsnap1/test1/102/20' 2023-10-30T12:31:58.694 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/102/10' -> 'copyofsnap1/test1/102/10' 2023-10-30T12:31:58.694 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/103' -> 'copyofsnap1/test1/103' 2023-10-30T12:31:58.694 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/103/11' -> 'copyofsnap1/test1/103/11' 2023-10-30T12:31:58.694 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/103/12' -> 'copyofsnap1/test1/103/12' 2023-10-30T12:31:58.695 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/103/13' -> 'copyofsnap1/test1/103/13' 2023-10-30T12:31:58.695 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/103/14' -> 'copyofsnap1/test1/103/14' 2023-10-30T12:31:58.695 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/103/15' -> 'copyofsnap1/test1/103/15' 2023-10-30T12:31:58.696 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/103/16' -> 'copyofsnap1/test1/103/16' 2023-10-30T12:31:58.696 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/103/17' -> 'copyofsnap1/test1/103/17' 2023-10-30T12:31:58.696 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/103/18' -> 'copyofsnap1/test1/103/18' 2023-10-30T12:31:58.696 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/103/19' -> 'copyofsnap1/test1/103/19' 2023-10-30T12:31:58.697 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/103/20' -> 'copyofsnap1/test1/103/20' 2023-10-30T12:31:58.697 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/103/10' -> 'copyofsnap1/test1/103/10' 2023-10-30T12:31:58.697 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/104' -> 'copyofsnap1/test1/104' 2023-10-30T12:31:58.697 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/104/11' -> 'copyofsnap1/test1/104/11' 2023-10-30T12:31:58.698 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/104/12' -> 'copyofsnap1/test1/104/12' 2023-10-30T12:31:58.698 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/104/13' -> 'copyofsnap1/test1/104/13' 2023-10-30T12:31:58.698 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/104/14' -> 'copyofsnap1/test1/104/14' 2023-10-30T12:31:58.698 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/104/15' -> 'copyofsnap1/test1/104/15' 2023-10-30T12:31:58.698 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/104/16' -> 'copyofsnap1/test1/104/16' 2023-10-30T12:31:58.699 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/104/17' -> 'copyofsnap1/test1/104/17' 2023-10-30T12:31:58.699 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/104/18' -> 'copyofsnap1/test1/104/18' 2023-10-30T12:31:58.699 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/104/19' -> 'copyofsnap1/test1/104/19' 2023-10-30T12:31:58.699 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/104/20' -> 'copyofsnap1/test1/104/20' 2023-10-30T12:31:58.699 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/104/10' -> 'copyofsnap1/test1/104/10' 2023-10-30T12:31:58.700 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/105' -> 'copyofsnap1/test1/105' 2023-10-30T12:31:58.700 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/105/11' -> 'copyofsnap1/test1/105/11' 2023-10-30T12:31:58.700 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/105/12' -> 'copyofsnap1/test1/105/12' 2023-10-30T12:31:58.700 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/105/13' -> 'copyofsnap1/test1/105/13' 2023-10-30T12:31:58.700 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/105/14' -> 'copyofsnap1/test1/105/14' 2023-10-30T12:31:58.701 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/105/15' -> 'copyofsnap1/test1/105/15' 2023-10-30T12:31:58.701 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/105/16' -> 'copyofsnap1/test1/105/16' 2023-10-30T12:31:58.701 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/105/17' -> 'copyofsnap1/test1/105/17' 2023-10-30T12:31:58.701 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/105/18' -> 'copyofsnap1/test1/105/18' 2023-10-30T12:31:58.701 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/105/19' -> 'copyofsnap1/test1/105/19' 2023-10-30T12:31:58.702 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/105/20' -> 'copyofsnap1/test1/105/20' 2023-10-30T12:31:58.702 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/105/10' -> 'copyofsnap1/test1/105/10' 2023-10-30T12:31:58.702 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/106' -> 'copyofsnap1/test1/106' 2023-10-30T12:31:58.702 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/106/11' -> 'copyofsnap1/test1/106/11' 2023-10-30T12:31:58.702 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/106/12' -> 'copyofsnap1/test1/106/12' 2023-10-30T12:31:58.703 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/106/13' -> 'copyofsnap1/test1/106/13' 2023-10-30T12:31:58.703 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/106/14' -> 'copyofsnap1/test1/106/14' 2023-10-30T12:31:58.703 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/106/15' -> 'copyofsnap1/test1/106/15' 2023-10-30T12:31:58.703 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/106/16' -> 'copyofsnap1/test1/106/16' 2023-10-30T12:31:58.703 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/106/17' -> 'copyofsnap1/test1/106/17' 2023-10-30T12:32:00.540 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/106/18' -> 'copyofsnap1/test1/106/18' 2023-10-30T12:32:00.541 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/106/19' -> 'copyofsnap1/test1/106/19' 2023-10-30T12:32:00.541 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/106/20' -> 'copyofsnap1/test1/106/20' 2023-10-30T12:32:00.541 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/106/10' -> 'copyofsnap1/test1/106/10' 2023-10-30T12:32:00.541 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/107' -> 'copyofsnap1/test1/107' 2023-10-30T12:32:00.541 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/107/11' -> 'copyofsnap1/test1/107/11' 2023-10-30T12:32:00.542 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/107/12' -> 'copyofsnap1/test1/107/12' 2023-10-30T12:32:00.542 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/107/13' -> 'copyofsnap1/test1/107/13' 2023-10-30T12:32:00.542 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/107/14' -> 'copyofsnap1/test1/107/14' 2023-10-30T12:32:00.542 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/107/15' -> 'copyofsnap1/test1/107/15' 2023-10-30T12:32:00.542 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/107/16' -> 'copyofsnap1/test1/107/16' 2023-10-30T12:32:00.543 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/107/17' -> 'copyofsnap1/test1/107/17' 2023-10-30T12:32:00.543 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/107/18' -> 'copyofsnap1/test1/107/18' 2023-10-30T12:32:00.543 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/107/19' -> 'copyofsnap1/test1/107/19' 2023-10-30T12:32:00.543 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/107/20' -> 'copyofsnap1/test1/107/20' 2023-10-30T12:32:00.543 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/107/10' -> 'copyofsnap1/test1/107/10' 2023-10-30T12:32:00.544 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/108' -> 'copyofsnap1/test1/108' 2023-10-30T12:32:00.544 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/108/11' -> 'copyofsnap1/test1/108/11' 2023-10-30T12:32:00.544 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/108/12' -> 'copyofsnap1/test1/108/12' 2023-10-30T12:32:00.544 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/108/13' -> 'copyofsnap1/test1/108/13' 2023-10-30T12:32:00.544 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/108/14' -> 'copyofsnap1/test1/108/14' 2023-10-30T12:32:00.544 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/108/15' -> 'copyofsnap1/test1/108/15' 2023-10-30T12:32:00.545 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/108/16' -> 'copyofsnap1/test1/108/16' 2023-10-30T12:32:00.545 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/108/17' -> 'copyofsnap1/test1/108/17' 2023-10-30T12:32:00.545 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/108/18' -> 'copyofsnap1/test1/108/18' 2023-10-30T12:32:00.545 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/108/19' -> 'copyofsnap1/test1/108/19' 2023-10-30T12:32:00.545 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/108/20' -> 'copyofsnap1/test1/108/20' 2023-10-30T12:32:00.546 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/108/10' -> 'copyofsnap1/test1/108/10' 2023-10-30T12:32:00.546 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/109' -> 'copyofsnap1/test1/109' 2023-10-30T12:32:00.546 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/109/11' -> 'copyofsnap1/test1/109/11' 2023-10-30T12:32:00.546 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/109/12' -> 'copyofsnap1/test1/109/12' 2023-10-30T12:32:00.546 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/109/13' -> 'copyofsnap1/test1/109/13' 2023-10-30T12:32:00.547 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/109/14' -> 'copyofsnap1/test1/109/14' 2023-10-30T12:32:00.547 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/109/15' -> 'copyofsnap1/test1/109/15' 2023-10-30T12:32:00.547 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/109/16' -> 'copyofsnap1/test1/109/16' 2023-10-30T12:32:00.547 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/109/17' -> 'copyofsnap1/test1/109/17' 2023-10-30T12:32:00.547 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/109/18' -> 'copyofsnap1/test1/109/18' 2023-10-30T12:32:00.548 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/109/19' -> 'copyofsnap1/test1/109/19' 2023-10-30T12:32:00.548 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/109/20' -> 'copyofsnap1/test1/109/20' 2023-10-30T12:32:00.548 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/109/10' -> 'copyofsnap1/test1/109/10' 2023-10-30T12:32:00.548 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/110' -> 'copyofsnap1/test1/110' 2023-10-30T12:32:00.548 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/110/11' -> 'copyofsnap1/test1/110/11' 2023-10-30T12:32:00.548 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/110/12' -> 'copyofsnap1/test1/110/12' 2023-10-30T12:32:00.549 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/110/13' -> 'copyofsnap1/test1/110/13' 2023-10-30T12:32:00.549 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/110/14' -> 'copyofsnap1/test1/110/14' 2023-10-30T12:32:00.549 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/110/15' -> 'copyofsnap1/test1/110/15' 2023-10-30T12:32:00.549 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/110/16' -> 'copyofsnap1/test1/110/16' 2023-10-30T12:32:00.550 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/110/17' -> 'copyofsnap1/test1/110/17' 2023-10-30T12:32:00.550 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/110/18' -> 'copyofsnap1/test1/110/18' 2023-10-30T12:32:00.550 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/110/19' -> 'copyofsnap1/test1/110/19' 2023-10-30T12:32:00.550 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/110/20' -> 'copyofsnap1/test1/110/20' 2023-10-30T12:32:00.551 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/110/10' -> 'copyofsnap1/test1/110/10' 2023-10-30T12:32:00.551 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/111' -> 'copyofsnap1/test1/111' 2023-10-30T12:32:00.551 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/111/11' -> 'copyofsnap1/test1/111/11' 2023-10-30T12:32:00.551 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/111/12' -> 'copyofsnap1/test1/111/12' 2023-10-30T12:32:00.551 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/111/13' -> 'copyofsnap1/test1/111/13' 2023-10-30T12:32:00.552 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/111/14' -> 'copyofsnap1/test1/111/14' 2023-10-30T12:32:00.552 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/111/15' -> 'copyofsnap1/test1/111/15' 2023-10-30T12:32:00.552 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/111/16' -> 'copyofsnap1/test1/111/16' 2023-10-30T12:32:00.552 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/111/17' -> 'copyofsnap1/test1/111/17' 2023-10-30T12:32:00.552 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/111/18' -> 'copyofsnap1/test1/111/18' 2023-10-30T12:32:00.552 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/111/19' -> 'copyofsnap1/test1/111/19' 2023-10-30T12:32:00.553 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/111/20' -> 'copyofsnap1/test1/111/20' 2023-10-30T12:32:00.553 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/111/10' -> 'copyofsnap1/test1/111/10' 2023-10-30T12:32:00.553 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/112' -> 'copyofsnap1/test1/112' 2023-10-30T12:32:00.553 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/112/11' -> 'copyofsnap1/test1/112/11' 2023-10-30T12:32:00.553 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/112/12' -> 'copyofsnap1/test1/112/12' 2023-10-30T12:32:00.554 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/112/13' -> 'copyofsnap1/test1/112/13' 2023-10-30T12:32:00.554 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/112/14' -> 'copyofsnap1/test1/112/14' 2023-10-30T12:32:00.554 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/112/15' -> 'copyofsnap1/test1/112/15' 2023-10-30T12:32:00.554 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/112/16' -> 'copyofsnap1/test1/112/16' 2023-10-30T12:32:00.554 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/112/17' -> 'copyofsnap1/test1/112/17' 2023-10-30T12:32:00.555 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/112/18' -> 'copyofsnap1/test1/112/18' 2023-10-30T12:32:00.555 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/112/19' -> 'copyofsnap1/test1/112/19' 2023-10-30T12:32:00.555 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/112/20' -> 'copyofsnap1/test1/112/20' 2023-10-30T12:32:00.555 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/112/10' -> 'copyofsnap1/test1/112/10' 2023-10-30T12:32:00.555 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/113' -> 'copyofsnap1/test1/113' 2023-10-30T12:32:00.555 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/113/11' -> 'copyofsnap1/test1/113/11' 2023-10-30T12:32:00.556 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/113/12' -> 'copyofsnap1/test1/113/12' 2023-10-30T12:32:00.556 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/113/13' -> 'copyofsnap1/test1/113/13' 2023-10-30T12:32:00.556 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/113/14' -> 'copyofsnap1/test1/113/14' 2023-10-30T12:32:02.344 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/113/15' -> 'copyofsnap1/test1/113/15' 2023-10-30T12:32:02.345 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/113/16' -> 'copyofsnap1/test1/113/16' 2023-10-30T12:32:02.345 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/113/17' -> 'copyofsnap1/test1/113/17' 2023-10-30T12:32:02.345 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/113/18' -> 'copyofsnap1/test1/113/18' 2023-10-30T12:32:02.346 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/113/19' -> 'copyofsnap1/test1/113/19' 2023-10-30T12:32:02.346 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/113/20' -> 'copyofsnap1/test1/113/20' 2023-10-30T12:32:02.346 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/113/10' -> 'copyofsnap1/test1/113/10' 2023-10-30T12:32:02.347 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/114' -> 'copyofsnap1/test1/114' 2023-10-30T12:32:02.347 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/114/11' -> 'copyofsnap1/test1/114/11' 2023-10-30T12:32:02.347 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/114/12' -> 'copyofsnap1/test1/114/12' 2023-10-30T12:32:02.347 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/114/13' -> 'copyofsnap1/test1/114/13' 2023-10-30T12:32:02.347 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/114/14' -> 'copyofsnap1/test1/114/14' 2023-10-30T12:32:02.348 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/114/15' -> 'copyofsnap1/test1/114/15' 2023-10-30T12:32:02.348 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/114/16' -> 'copyofsnap1/test1/114/16' 2023-10-30T12:32:02.348 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/114/17' -> 'copyofsnap1/test1/114/17' 2023-10-30T12:32:02.349 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/114/18' -> 'copyofsnap1/test1/114/18' 2023-10-30T12:32:02.349 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/114/19' -> 'copyofsnap1/test1/114/19' 2023-10-30T12:32:02.349 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/114/20' -> 'copyofsnap1/test1/114/20' 2023-10-30T12:32:02.349 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/114/10' -> 'copyofsnap1/test1/114/10' 2023-10-30T12:32:02.349 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/115' -> 'copyofsnap1/test1/115' 2023-10-30T12:32:02.350 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/115/11' -> 'copyofsnap1/test1/115/11' 2023-10-30T12:32:02.350 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/115/12' -> 'copyofsnap1/test1/115/12' 2023-10-30T12:32:02.350 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/115/13' -> 'copyofsnap1/test1/115/13' 2023-10-30T12:32:02.351 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/115/14' -> 'copyofsnap1/test1/115/14' 2023-10-30T12:32:02.351 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/115/15' -> 'copyofsnap1/test1/115/15' 2023-10-30T12:32:02.351 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/115/16' -> 'copyofsnap1/test1/115/16' 2023-10-30T12:32:02.351 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/115/17' -> 'copyofsnap1/test1/115/17' 2023-10-30T12:32:02.351 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/115/18' -> 'copyofsnap1/test1/115/18' 2023-10-30T12:32:02.352 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/115/19' -> 'copyofsnap1/test1/115/19' 2023-10-30T12:32:02.352 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/115/20' -> 'copyofsnap1/test1/115/20' 2023-10-30T12:32:02.352 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/115/10' -> 'copyofsnap1/test1/115/10' 2023-10-30T12:32:02.352 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/116' -> 'copyofsnap1/test1/116' 2023-10-30T12:32:02.352 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/116/11' -> 'copyofsnap1/test1/116/11' 2023-10-30T12:32:02.353 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/116/12' -> 'copyofsnap1/test1/116/12' 2023-10-30T12:32:02.353 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/116/13' -> 'copyofsnap1/test1/116/13' 2023-10-30T12:32:02.353 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/116/14' -> 'copyofsnap1/test1/116/14' 2023-10-30T12:32:02.353 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/116/15' -> 'copyofsnap1/test1/116/15' 2023-10-30T12:32:02.353 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/116/16' -> 'copyofsnap1/test1/116/16' 2023-10-30T12:32:02.353 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/116/17' -> 'copyofsnap1/test1/116/17' 2023-10-30T12:32:02.354 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/116/18' -> 'copyofsnap1/test1/116/18' 2023-10-30T12:32:02.354 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/116/19' -> 'copyofsnap1/test1/116/19' 2023-10-30T12:32:02.354 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/116/20' -> 'copyofsnap1/test1/116/20' 2023-10-30T12:32:02.354 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/116/10' -> 'copyofsnap1/test1/116/10' 2023-10-30T12:32:02.354 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/117' -> 'copyofsnap1/test1/117' 2023-10-30T12:32:02.355 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/117/11' -> 'copyofsnap1/test1/117/11' 2023-10-30T12:32:02.355 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/117/12' -> 'copyofsnap1/test1/117/12' 2023-10-30T12:32:02.355 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/117/13' -> 'copyofsnap1/test1/117/13' 2023-10-30T12:32:02.355 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/117/14' -> 'copyofsnap1/test1/117/14' 2023-10-30T12:32:02.355 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/117/15' -> 'copyofsnap1/test1/117/15' 2023-10-30T12:32:02.356 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/117/16' -> 'copyofsnap1/test1/117/16' 2023-10-30T12:32:02.356 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/117/17' -> 'copyofsnap1/test1/117/17' 2023-10-30T12:32:02.356 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/117/18' -> 'copyofsnap1/test1/117/18' 2023-10-30T12:32:02.356 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/117/19' -> 'copyofsnap1/test1/117/19' 2023-10-30T12:32:02.356 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/117/20' -> 'copyofsnap1/test1/117/20' 2023-10-30T12:32:02.356 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/117/10' -> 'copyofsnap1/test1/117/10' 2023-10-30T12:32:02.357 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/118' -> 'copyofsnap1/test1/118' 2023-10-30T12:32:02.357 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/118/11' -> 'copyofsnap1/test1/118/11' 2023-10-30T12:32:02.357 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/118/12' -> 'copyofsnap1/test1/118/12' 2023-10-30T12:32:02.357 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/118/13' -> 'copyofsnap1/test1/118/13' 2023-10-30T12:32:02.358 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/118/14' -> 'copyofsnap1/test1/118/14' 2023-10-30T12:32:02.358 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/118/15' -> 'copyofsnap1/test1/118/15' 2023-10-30T12:32:02.358 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/118/16' -> 'copyofsnap1/test1/118/16' 2023-10-30T12:32:02.358 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/118/17' -> 'copyofsnap1/test1/118/17' 2023-10-30T12:32:02.359 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/118/18' -> 'copyofsnap1/test1/118/18' 2023-10-30T12:32:02.359 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/118/19' -> 'copyofsnap1/test1/118/19' 2023-10-30T12:32:02.359 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/118/20' -> 'copyofsnap1/test1/118/20' 2023-10-30T12:32:02.359 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/118/10' -> 'copyofsnap1/test1/118/10' 2023-10-30T12:32:02.359 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/119' -> 'copyofsnap1/test1/119' 2023-10-30T12:32:02.360 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/119/11' -> 'copyofsnap1/test1/119/11' 2023-10-30T12:32:02.360 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/119/12' -> 'copyofsnap1/test1/119/12' 2023-10-30T12:32:02.360 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/119/13' -> 'copyofsnap1/test1/119/13' 2023-10-30T12:32:02.360 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/119/14' -> 'copyofsnap1/test1/119/14' 2023-10-30T12:32:02.360 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/119/15' -> 'copyofsnap1/test1/119/15' 2023-10-30T12:32:02.360 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/119/16' -> 'copyofsnap1/test1/119/16' 2023-10-30T12:32:02.361 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/119/17' -> 'copyofsnap1/test1/119/17' 2023-10-30T12:32:02.361 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/119/18' -> 'copyofsnap1/test1/119/18' 2023-10-30T12:32:02.361 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/119/19' -> 'copyofsnap1/test1/119/19' 2023-10-30T12:32:02.361 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/119/20' -> 'copyofsnap1/test1/119/20' 2023-10-30T12:32:02.361 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/119/10' -> 'copyofsnap1/test1/119/10' 2023-10-30T12:32:02.362 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/120' -> 'copyofsnap1/test1/120' 2023-10-30T12:32:02.362 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/120/11' -> 'copyofsnap1/test1/120/11' 2023-10-30T12:32:03.121 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/120/12' -> 'copyofsnap1/test1/120/12' 2023-10-30T12:32:03.121 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/120/13' -> 'copyofsnap1/test1/120/13' 2023-10-30T12:32:03.121 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/120/14' -> 'copyofsnap1/test1/120/14' 2023-10-30T12:32:03.122 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/120/15' -> 'copyofsnap1/test1/120/15' 2023-10-30T12:32:03.122 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/120/16' -> 'copyofsnap1/test1/120/16' 2023-10-30T12:32:03.122 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/120/17' -> 'copyofsnap1/test1/120/17' 2023-10-30T12:32:03.122 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/120/18' -> 'copyofsnap1/test1/120/18' 2023-10-30T12:32:03.122 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/120/19' -> 'copyofsnap1/test1/120/19' 2023-10-30T12:32:03.123 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/120/20' -> 'copyofsnap1/test1/120/20' 2023-10-30T12:32:03.123 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/120/10' -> 'copyofsnap1/test1/120/10' 2023-10-30T12:32:03.123 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/121' -> 'copyofsnap1/test1/121' 2023-10-30T12:32:03.123 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/121/11' -> 'copyofsnap1/test1/121/11' 2023-10-30T12:32:03.123 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/121/12' -> 'copyofsnap1/test1/121/12' 2023-10-30T12:32:03.124 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/121/13' -> 'copyofsnap1/test1/121/13' 2023-10-30T12:32:03.124 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/121/14' -> 'copyofsnap1/test1/121/14' 2023-10-30T12:32:03.124 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/121/15' -> 'copyofsnap1/test1/121/15' 2023-10-30T12:32:03.124 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/121/16' -> 'copyofsnap1/test1/121/16' 2023-10-30T12:32:03.124 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/121/17' -> 'copyofsnap1/test1/121/17' 2023-10-30T12:32:03.124 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/121/18' -> 'copyofsnap1/test1/121/18' 2023-10-30T12:32:03.125 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/121/19' -> 'copyofsnap1/test1/121/19' 2023-10-30T12:32:03.125 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/121/20' -> 'copyofsnap1/test1/121/20' 2023-10-30T12:32:03.125 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/121/10' -> 'copyofsnap1/test1/121/10' 2023-10-30T12:32:03.125 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/122' -> 'copyofsnap1/test1/122' 2023-10-30T12:32:03.125 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/122/11' -> 'copyofsnap1/test1/122/11' 2023-10-30T12:32:03.126 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/122/12' -> 'copyofsnap1/test1/122/12' 2023-10-30T12:32:03.126 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/122/13' -> 'copyofsnap1/test1/122/13' 2023-10-30T12:32:03.126 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/122/14' -> 'copyofsnap1/test1/122/14' 2023-10-30T12:32:03.126 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/122/15' -> 'copyofsnap1/test1/122/15' 2023-10-30T12:32:03.127 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/122/16' -> 'copyofsnap1/test1/122/16' 2023-10-30T12:32:03.127 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/122/17' -> 'copyofsnap1/test1/122/17' 2023-10-30T12:32:03.127 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/122/18' -> 'copyofsnap1/test1/122/18' 2023-10-30T12:32:03.127 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/122/19' -> 'copyofsnap1/test1/122/19' 2023-10-30T12:32:03.127 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/122/20' -> 'copyofsnap1/test1/122/20' 2023-10-30T12:32:03.127 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/122/10' -> 'copyofsnap1/test1/122/10' 2023-10-30T12:32:03.128 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/123' -> 'copyofsnap1/test1/123' 2023-10-30T12:32:03.128 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/123/11' -> 'copyofsnap1/test1/123/11' 2023-10-30T12:32:03.128 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/123/12' -> 'copyofsnap1/test1/123/12' 2023-10-30T12:32:03.128 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/123/13' -> 'copyofsnap1/test1/123/13' 2023-10-30T12:32:03.128 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/123/14' -> 'copyofsnap1/test1/123/14' 2023-10-30T12:32:03.128 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/123/15' -> 'copyofsnap1/test1/123/15' 2023-10-30T12:32:03.129 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/123/16' -> 'copyofsnap1/test1/123/16' 2023-10-30T12:32:03.129 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/123/17' -> 'copyofsnap1/test1/123/17' 2023-10-30T12:32:03.129 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/123/18' -> 'copyofsnap1/test1/123/18' 2023-10-30T12:32:03.129 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/123/19' -> 'copyofsnap1/test1/123/19' 2023-10-30T12:32:03.129 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/123/20' -> 'copyofsnap1/test1/123/20' 2023-10-30T12:32:03.130 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/123/10' -> 'copyofsnap1/test1/123/10' 2023-10-30T12:32:03.130 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/124' -> 'copyofsnap1/test1/124' 2023-10-30T12:32:03.130 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/124/11' -> 'copyofsnap1/test1/124/11' 2023-10-30T12:32:03.130 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/124/12' -> 'copyofsnap1/test1/124/12' 2023-10-30T12:32:03.130 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/124/13' -> 'copyofsnap1/test1/124/13' 2023-10-30T12:32:03.131 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/124/14' -> 'copyofsnap1/test1/124/14' 2023-10-30T12:32:03.131 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/124/15' -> 'copyofsnap1/test1/124/15' 2023-10-30T12:32:03.131 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/124/16' -> 'copyofsnap1/test1/124/16' 2023-10-30T12:32:03.131 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/124/17' -> 'copyofsnap1/test1/124/17' 2023-10-30T12:32:03.131 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/124/18' -> 'copyofsnap1/test1/124/18' 2023-10-30T12:32:03.131 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/124/19' -> 'copyofsnap1/test1/124/19' 2023-10-30T12:32:03.132 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/124/20' -> 'copyofsnap1/test1/124/20' 2023-10-30T12:32:03.132 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/124/10' -> 'copyofsnap1/test1/124/10' 2023-10-30T12:32:03.132 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/125' -> 'copyofsnap1/test1/125' 2023-10-30T12:32:03.132 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/125/11' -> 'copyofsnap1/test1/125/11' 2023-10-30T12:32:03.132 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/125/12' -> 'copyofsnap1/test1/125/12' 2023-10-30T12:32:03.133 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/125/13' -> 'copyofsnap1/test1/125/13' 2023-10-30T12:32:03.133 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/125/14' -> 'copyofsnap1/test1/125/14' 2023-10-30T12:32:03.133 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/125/15' -> 'copyofsnap1/test1/125/15' 2023-10-30T12:32:03.133 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/125/16' -> 'copyofsnap1/test1/125/16' 2023-10-30T12:32:03.133 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/125/17' -> 'copyofsnap1/test1/125/17' 2023-10-30T12:32:03.133 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/125/18' -> 'copyofsnap1/test1/125/18' 2023-10-30T12:32:03.134 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/125/19' -> 'copyofsnap1/test1/125/19' 2023-10-30T12:32:03.134 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/125/20' -> 'copyofsnap1/test1/125/20' 2023-10-30T12:32:03.134 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/125/10' -> 'copyofsnap1/test1/125/10' 2023-10-30T12:32:03.134 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/126' -> 'copyofsnap1/test1/126' 2023-10-30T12:32:03.134 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/126/11' -> 'copyofsnap1/test1/126/11' 2023-10-30T12:32:03.135 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/126/12' -> 'copyofsnap1/test1/126/12' 2023-10-30T12:32:03.135 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/126/13' -> 'copyofsnap1/test1/126/13' 2023-10-30T12:32:03.135 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/126/14' -> 'copyofsnap1/test1/126/14' 2023-10-30T12:32:03.135 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/126/15' -> 'copyofsnap1/test1/126/15' 2023-10-30T12:32:03.135 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/126/16' -> 'copyofsnap1/test1/126/16' 2023-10-30T12:32:03.136 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/126/17' -> 'copyofsnap1/test1/126/17' 2023-10-30T12:32:03.136 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/126/18' -> 'copyofsnap1/test1/126/18' 2023-10-30T12:32:03.136 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/126/19' -> 'copyofsnap1/test1/126/19' 2023-10-30T12:32:03.136 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/126/20' -> 'copyofsnap1/test1/126/20' 2023-10-30T12:32:03.422 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/126/10' -> 'copyofsnap1/test1/126/10' 2023-10-30T12:32:03.422 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/127' -> 'copyofsnap1/test1/127' 2023-10-30T12:32:03.422 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/127/11' -> 'copyofsnap1/test1/127/11' 2023-10-30T12:32:03.422 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/127/12' -> 'copyofsnap1/test1/127/12' 2023-10-30T12:32:03.423 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/127/13' -> 'copyofsnap1/test1/127/13' 2023-10-30T12:32:03.423 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/127/14' -> 'copyofsnap1/test1/127/14' 2023-10-30T12:32:03.423 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/127/15' -> 'copyofsnap1/test1/127/15' 2023-10-30T12:32:03.423 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/127/16' -> 'copyofsnap1/test1/127/16' 2023-10-30T12:32:03.423 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/127/17' -> 'copyofsnap1/test1/127/17' 2023-10-30T12:32:03.423 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/127/18' -> 'copyofsnap1/test1/127/18' 2023-10-30T12:32:03.424 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/127/19' -> 'copyofsnap1/test1/127/19' 2023-10-30T12:32:03.424 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/127/20' -> 'copyofsnap1/test1/127/20' 2023-10-30T12:32:03.424 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/127/10' -> 'copyofsnap1/test1/127/10' 2023-10-30T12:32:03.424 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/128' -> 'copyofsnap1/test1/128' 2023-10-30T12:32:03.424 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/128/11' -> 'copyofsnap1/test1/128/11' 2023-10-30T12:32:03.425 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/128/12' -> 'copyofsnap1/test1/128/12' 2023-10-30T12:32:03.425 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/128/13' -> 'copyofsnap1/test1/128/13' 2023-10-30T12:32:03.425 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/128/14' -> 'copyofsnap1/test1/128/14' 2023-10-30T12:32:03.425 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/128/15' -> 'copyofsnap1/test1/128/15' 2023-10-30T12:32:03.425 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/128/16' -> 'copyofsnap1/test1/128/16' 2023-10-30T12:32:03.425 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/128/17' -> 'copyofsnap1/test1/128/17' 2023-10-30T12:32:03.426 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/128/18' -> 'copyofsnap1/test1/128/18' 2023-10-30T12:32:03.426 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/128/19' -> 'copyofsnap1/test1/128/19' 2023-10-30T12:32:03.426 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/128/20' -> 'copyofsnap1/test1/128/20' 2023-10-30T12:32:03.426 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/128/10' -> 'copyofsnap1/test1/128/10' 2023-10-30T12:32:03.426 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/129' -> 'copyofsnap1/test1/129' 2023-10-30T12:32:03.427 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/129/11' -> 'copyofsnap1/test1/129/11' 2023-10-30T12:32:03.427 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/129/12' -> 'copyofsnap1/test1/129/12' 2023-10-30T12:32:03.427 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/129/13' -> 'copyofsnap1/test1/129/13' 2023-10-30T12:32:03.427 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/129/14' -> 'copyofsnap1/test1/129/14' 2023-10-30T12:32:03.427 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/129/15' -> 'copyofsnap1/test1/129/15' 2023-10-30T12:32:03.428 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/129/16' -> 'copyofsnap1/test1/129/16' 2023-10-30T12:32:03.428 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/129/17' -> 'copyofsnap1/test1/129/17' 2023-10-30T12:32:03.428 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/129/18' -> 'copyofsnap1/test1/129/18' 2023-10-30T12:32:03.428 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/129/19' -> 'copyofsnap1/test1/129/19' 2023-10-30T12:32:03.428 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/129/20' -> 'copyofsnap1/test1/129/20' 2023-10-30T12:32:03.429 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/129/10' -> 'copyofsnap1/test1/129/10' 2023-10-30T12:32:03.429 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/130' -> 'copyofsnap1/test1/130' 2023-10-30T12:32:03.429 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/130/11' -> 'copyofsnap1/test1/130/11' 2023-10-30T12:32:03.429 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/130/12' -> 'copyofsnap1/test1/130/12' 2023-10-30T12:32:03.429 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/130/13' -> 'copyofsnap1/test1/130/13' 2023-10-30T12:32:03.429 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/130/14' -> 'copyofsnap1/test1/130/14' 2023-10-30T12:32:03.430 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/130/15' -> 'copyofsnap1/test1/130/15' 2023-10-30T12:32:03.430 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/130/16' -> 'copyofsnap1/test1/130/16' 2023-10-30T12:32:03.430 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/130/17' -> 'copyofsnap1/test1/130/17' 2023-10-30T12:32:03.430 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/130/18' -> 'copyofsnap1/test1/130/18' 2023-10-30T12:32:03.430 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/130/19' -> 'copyofsnap1/test1/130/19' 2023-10-30T12:32:03.431 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/130/20' -> 'copyofsnap1/test1/130/20' 2023-10-30T12:32:03.431 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/130/10' -> 'copyofsnap1/test1/130/10' 2023-10-30T12:32:03.431 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/131' -> 'copyofsnap1/test1/131' 2023-10-30T12:32:03.431 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/131/11' -> 'copyofsnap1/test1/131/11' 2023-10-30T12:32:03.431 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/131/12' -> 'copyofsnap1/test1/131/12' 2023-10-30T12:32:03.432 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/131/13' -> 'copyofsnap1/test1/131/13' 2023-10-30T12:32:03.432 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/131/14' -> 'copyofsnap1/test1/131/14' 2023-10-30T12:32:03.432 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/131/15' -> 'copyofsnap1/test1/131/15' 2023-10-30T12:32:03.432 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/131/16' -> 'copyofsnap1/test1/131/16' 2023-10-30T12:32:03.432 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/131/17' -> 'copyofsnap1/test1/131/17' 2023-10-30T12:32:03.432 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/131/18' -> 'copyofsnap1/test1/131/18' 2023-10-30T12:32:03.433 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/131/19' -> 'copyofsnap1/test1/131/19' 2023-10-30T12:32:03.433 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/131/20' -> 'copyofsnap1/test1/131/20' 2023-10-30T12:32:03.433 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/131/10' -> 'copyofsnap1/test1/131/10' 2023-10-30T12:32:03.433 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/132' -> 'copyofsnap1/test1/132' 2023-10-30T12:32:03.433 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/132/11' -> 'copyofsnap1/test1/132/11' 2023-10-30T12:32:03.433 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/132/12' -> 'copyofsnap1/test1/132/12' 2023-10-30T12:32:03.434 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/132/13' -> 'copyofsnap1/test1/132/13' 2023-10-30T12:32:03.434 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/132/14' -> 'copyofsnap1/test1/132/14' 2023-10-30T12:32:03.434 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/132/15' -> 'copyofsnap1/test1/132/15' 2023-10-30T12:32:03.434 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/132/16' -> 'copyofsnap1/test1/132/16' 2023-10-30T12:32:03.434 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/132/17' -> 'copyofsnap1/test1/132/17' 2023-10-30T12:32:03.435 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/132/18' -> 'copyofsnap1/test1/132/18' 2023-10-30T12:32:03.435 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/132/19' -> 'copyofsnap1/test1/132/19' 2023-10-30T12:32:03.435 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/132/20' -> 'copyofsnap1/test1/132/20' 2023-10-30T12:32:03.435 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/132/10' -> 'copyofsnap1/test1/132/10' 2023-10-30T12:32:03.435 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/133' -> 'copyofsnap1/test1/133' 2023-10-30T12:32:03.436 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/133/11' -> 'copyofsnap1/test1/133/11' 2023-10-30T12:32:03.436 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/133/12' -> 'copyofsnap1/test1/133/12' 2023-10-30T12:32:03.436 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/133/13' -> 'copyofsnap1/test1/133/13' 2023-10-30T12:32:03.436 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/133/14' -> 'copyofsnap1/test1/133/14' 2023-10-30T12:32:03.436 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/133/15' -> 'copyofsnap1/test1/133/15' 2023-10-30T12:32:03.436 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/133/16' -> 'copyofsnap1/test1/133/16' 2023-10-30T12:32:03.437 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/133/17' -> 'copyofsnap1/test1/133/17' 2023-10-30T12:32:04.812 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/133/18' -> 'copyofsnap1/test1/133/18' 2023-10-30T12:32:04.812 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/133/19' -> 'copyofsnap1/test1/133/19' 2023-10-30T12:32:04.812 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/133/20' -> 'copyofsnap1/test1/133/20' 2023-10-30T12:32:04.813 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/133/10' -> 'copyofsnap1/test1/133/10' 2023-10-30T12:32:04.813 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/134' -> 'copyofsnap1/test1/134' 2023-10-30T12:32:04.813 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/134/11' -> 'copyofsnap1/test1/134/11' 2023-10-30T12:32:04.813 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/134/12' -> 'copyofsnap1/test1/134/12' 2023-10-30T12:32:04.813 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/134/13' -> 'copyofsnap1/test1/134/13' 2023-10-30T12:32:04.814 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/134/14' -> 'copyofsnap1/test1/134/14' 2023-10-30T12:32:04.814 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/134/15' -> 'copyofsnap1/test1/134/15' 2023-10-30T12:32:04.814 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/134/16' -> 'copyofsnap1/test1/134/16' 2023-10-30T12:32:04.814 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/134/17' -> 'copyofsnap1/test1/134/17' 2023-10-30T12:32:04.814 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/134/18' -> 'copyofsnap1/test1/134/18' 2023-10-30T12:32:04.815 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/134/19' -> 'copyofsnap1/test1/134/19' 2023-10-30T12:32:04.815 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/134/20' -> 'copyofsnap1/test1/134/20' 2023-10-30T12:32:04.815 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/134/10' -> 'copyofsnap1/test1/134/10' 2023-10-30T12:32:04.815 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/135' -> 'copyofsnap1/test1/135' 2023-10-30T12:32:04.815 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/135/11' -> 'copyofsnap1/test1/135/11' 2023-10-30T12:32:04.815 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/135/12' -> 'copyofsnap1/test1/135/12' 2023-10-30T12:32:04.816 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/135/13' -> 'copyofsnap1/test1/135/13' 2023-10-30T12:32:04.816 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/135/14' -> 'copyofsnap1/test1/135/14' 2023-10-30T12:32:04.816 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/135/15' -> 'copyofsnap1/test1/135/15' 2023-10-30T12:32:04.816 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/135/16' -> 'copyofsnap1/test1/135/16' 2023-10-30T12:32:04.816 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/135/17' -> 'copyofsnap1/test1/135/17' 2023-10-30T12:32:04.817 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/135/18' -> 'copyofsnap1/test1/135/18' 2023-10-30T12:32:04.817 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/135/19' -> 'copyofsnap1/test1/135/19' 2023-10-30T12:32:04.817 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/135/20' -> 'copyofsnap1/test1/135/20' 2023-10-30T12:32:04.817 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/135/10' -> 'copyofsnap1/test1/135/10' 2023-10-30T12:32:04.817 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/136' -> 'copyofsnap1/test1/136' 2023-10-30T12:32:04.818 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/136/11' -> 'copyofsnap1/test1/136/11' 2023-10-30T12:32:04.818 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/136/12' -> 'copyofsnap1/test1/136/12' 2023-10-30T12:32:04.818 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/136/13' -> 'copyofsnap1/test1/136/13' 2023-10-30T12:32:04.818 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/136/14' -> 'copyofsnap1/test1/136/14' 2023-10-30T12:32:04.818 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/136/15' -> 'copyofsnap1/test1/136/15' 2023-10-30T12:32:04.819 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/136/16' -> 'copyofsnap1/test1/136/16' 2023-10-30T12:32:04.819 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/136/17' -> 'copyofsnap1/test1/136/17' 2023-10-30T12:32:04.819 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/136/18' -> 'copyofsnap1/test1/136/18' 2023-10-30T12:32:04.819 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/136/19' -> 'copyofsnap1/test1/136/19' 2023-10-30T12:32:04.819 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/136/20' -> 'copyofsnap1/test1/136/20' 2023-10-30T12:32:04.819 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/136/10' -> 'copyofsnap1/test1/136/10' 2023-10-30T12:32:04.820 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/137' -> 'copyofsnap1/test1/137' 2023-10-30T12:32:04.820 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/137/11' -> 'copyofsnap1/test1/137/11' 2023-10-30T12:32:04.820 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/137/12' -> 'copyofsnap1/test1/137/12' 2023-10-30T12:32:04.820 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/137/13' -> 'copyofsnap1/test1/137/13' 2023-10-30T12:32:04.820 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/137/14' -> 'copyofsnap1/test1/137/14' 2023-10-30T12:32:04.821 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/137/15' -> 'copyofsnap1/test1/137/15' 2023-10-30T12:32:04.821 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/137/16' -> 'copyofsnap1/test1/137/16' 2023-10-30T12:32:04.821 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/137/17' -> 'copyofsnap1/test1/137/17' 2023-10-30T12:32:04.821 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/137/18' -> 'copyofsnap1/test1/137/18' 2023-10-30T12:32:04.821 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/137/19' -> 'copyofsnap1/test1/137/19' 2023-10-30T12:32:04.822 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/137/20' -> 'copyofsnap1/test1/137/20' 2023-10-30T12:32:04.822 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/137/10' -> 'copyofsnap1/test1/137/10' 2023-10-30T12:32:04.822 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/138' -> 'copyofsnap1/test1/138' 2023-10-30T12:32:04.822 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/138/11' -> 'copyofsnap1/test1/138/11' 2023-10-30T12:32:04.822 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/138/12' -> 'copyofsnap1/test1/138/12' 2023-10-30T12:32:04.823 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/138/13' -> 'copyofsnap1/test1/138/13' 2023-10-30T12:32:04.823 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/138/14' -> 'copyofsnap1/test1/138/14' 2023-10-30T12:32:04.823 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/138/15' -> 'copyofsnap1/test1/138/15' 2023-10-30T12:32:04.823 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/138/16' -> 'copyofsnap1/test1/138/16' 2023-10-30T12:32:04.823 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/138/17' -> 'copyofsnap1/test1/138/17' 2023-10-30T12:32:04.823 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/138/18' -> 'copyofsnap1/test1/138/18' 2023-10-30T12:32:04.824 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/138/19' -> 'copyofsnap1/test1/138/19' 2023-10-30T12:32:04.824 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/138/20' -> 'copyofsnap1/test1/138/20' 2023-10-30T12:32:04.824 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/138/10' -> 'copyofsnap1/test1/138/10' 2023-10-30T12:32:04.824 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/139' -> 'copyofsnap1/test1/139' 2023-10-30T12:32:04.824 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/139/11' -> 'copyofsnap1/test1/139/11' 2023-10-30T12:32:04.825 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/139/12' -> 'copyofsnap1/test1/139/12' 2023-10-30T12:32:04.825 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/139/13' -> 'copyofsnap1/test1/139/13' 2023-10-30T12:32:04.825 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/139/14' -> 'copyofsnap1/test1/139/14' 2023-10-30T12:32:04.825 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/139/15' -> 'copyofsnap1/test1/139/15' 2023-10-30T12:32:04.825 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/139/16' -> 'copyofsnap1/test1/139/16' 2023-10-30T12:32:04.826 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/139/17' -> 'copyofsnap1/test1/139/17' 2023-10-30T12:32:04.826 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/139/18' -> 'copyofsnap1/test1/139/18' 2023-10-30T12:32:04.826 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/139/19' -> 'copyofsnap1/test1/139/19' 2023-10-30T12:32:04.826 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/139/20' -> 'copyofsnap1/test1/139/20' 2023-10-30T12:32:04.826 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/139/10' -> 'copyofsnap1/test1/139/10' 2023-10-30T12:32:04.827 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/140' -> 'copyofsnap1/test1/140' 2023-10-30T12:32:04.827 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/140/11' -> 'copyofsnap1/test1/140/11' 2023-10-30T12:32:04.827 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/140/12' -> 'copyofsnap1/test1/140/12' 2023-10-30T12:32:04.827 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/140/13' -> 'copyofsnap1/test1/140/13' 2023-10-30T12:32:04.827 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/140/14' -> 'copyofsnap1/test1/140/14' 2023-10-30T12:32:05.800 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/140/15' -> 'copyofsnap1/test1/140/15' 2023-10-30T12:32:05.800 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/140/16' -> 'copyofsnap1/test1/140/16' 2023-10-30T12:32:05.800 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/140/17' -> 'copyofsnap1/test1/140/17' 2023-10-30T12:32:05.801 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/140/18' -> 'copyofsnap1/test1/140/18' 2023-10-30T12:32:05.801 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/140/19' -> 'copyofsnap1/test1/140/19' 2023-10-30T12:32:05.801 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/140/20' -> 'copyofsnap1/test1/140/20' 2023-10-30T12:32:05.802 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/140/10' -> 'copyofsnap1/test1/140/10' 2023-10-30T12:32:05.802 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/141' -> 'copyofsnap1/test1/141' 2023-10-30T12:32:05.803 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/141/11' -> 'copyofsnap1/test1/141/11' 2023-10-30T12:32:05.803 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/141/12' -> 'copyofsnap1/test1/141/12' 2023-10-30T12:32:05.803 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/141/13' -> 'copyofsnap1/test1/141/13' 2023-10-30T12:32:05.804 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/141/14' -> 'copyofsnap1/test1/141/14' 2023-10-30T12:32:05.804 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/141/15' -> 'copyofsnap1/test1/141/15' 2023-10-30T12:32:05.804 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/141/16' -> 'copyofsnap1/test1/141/16' 2023-10-30T12:32:05.805 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/141/17' -> 'copyofsnap1/test1/141/17' 2023-10-30T12:32:05.805 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/141/18' -> 'copyofsnap1/test1/141/18' 2023-10-30T12:32:05.805 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/141/19' -> 'copyofsnap1/test1/141/19' 2023-10-30T12:32:05.806 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/141/20' -> 'copyofsnap1/test1/141/20' 2023-10-30T12:32:05.806 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/141/10' -> 'copyofsnap1/test1/141/10' 2023-10-30T12:32:05.806 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/142' -> 'copyofsnap1/test1/142' 2023-10-30T12:32:05.807 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/142/11' -> 'copyofsnap1/test1/142/11' 2023-10-30T12:32:05.807 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/142/12' -> 'copyofsnap1/test1/142/12' 2023-10-30T12:32:05.807 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/142/13' -> 'copyofsnap1/test1/142/13' 2023-10-30T12:32:05.807 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/142/14' -> 'copyofsnap1/test1/142/14' 2023-10-30T12:32:05.808 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/142/15' -> 'copyofsnap1/test1/142/15' 2023-10-30T12:32:05.808 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/142/16' -> 'copyofsnap1/test1/142/16' 2023-10-30T12:32:05.808 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/142/17' -> 'copyofsnap1/test1/142/17' 2023-10-30T12:32:05.809 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/142/18' -> 'copyofsnap1/test1/142/18' 2023-10-30T12:32:05.809 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/142/19' -> 'copyofsnap1/test1/142/19' 2023-10-30T12:32:05.809 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/142/20' -> 'copyofsnap1/test1/142/20' 2023-10-30T12:32:05.809 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/142/10' -> 'copyofsnap1/test1/142/10' 2023-10-30T12:32:05.810 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/143' -> 'copyofsnap1/test1/143' 2023-10-30T12:32:05.810 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/143/11' -> 'copyofsnap1/test1/143/11' 2023-10-30T12:32:05.810 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/143/12' -> 'copyofsnap1/test1/143/12' 2023-10-30T12:32:05.811 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/143/13' -> 'copyofsnap1/test1/143/13' 2023-10-30T12:32:05.811 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/143/14' -> 'copyofsnap1/test1/143/14' 2023-10-30T12:32:05.811 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/143/15' -> 'copyofsnap1/test1/143/15' 2023-10-30T12:32:05.811 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/143/16' -> 'copyofsnap1/test1/143/16' 2023-10-30T12:32:05.811 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/143/17' -> 'copyofsnap1/test1/143/17' 2023-10-30T12:32:05.812 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/143/18' -> 'copyofsnap1/test1/143/18' 2023-10-30T12:32:05.812 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/143/19' -> 'copyofsnap1/test1/143/19' 2023-10-30T12:32:05.812 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/143/20' -> 'copyofsnap1/test1/143/20' 2023-10-30T12:32:05.812 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/143/10' -> 'copyofsnap1/test1/143/10' 2023-10-30T12:32:05.812 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/144' -> 'copyofsnap1/test1/144' 2023-10-30T12:32:05.813 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/144/11' -> 'copyofsnap1/test1/144/11' 2023-10-30T12:32:05.813 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/144/12' -> 'copyofsnap1/test1/144/12' 2023-10-30T12:32:05.813 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/144/13' -> 'copyofsnap1/test1/144/13' 2023-10-30T12:32:05.813 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/144/14' -> 'copyofsnap1/test1/144/14' 2023-10-30T12:32:05.813 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/144/15' -> 'copyofsnap1/test1/144/15' 2023-10-30T12:32:05.813 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/144/16' -> 'copyofsnap1/test1/144/16' 2023-10-30T12:32:05.814 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/144/17' -> 'copyofsnap1/test1/144/17' 2023-10-30T12:32:05.814 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/144/18' -> 'copyofsnap1/test1/144/18' 2023-10-30T12:32:05.814 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/144/19' -> 'copyofsnap1/test1/144/19' 2023-10-30T12:32:05.814 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/144/20' -> 'copyofsnap1/test1/144/20' 2023-10-30T12:32:05.814 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/144/10' -> 'copyofsnap1/test1/144/10' 2023-10-30T12:32:05.815 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/145' -> 'copyofsnap1/test1/145' 2023-10-30T12:32:05.815 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/145/11' -> 'copyofsnap1/test1/145/11' 2023-10-30T12:32:05.815 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/145/12' -> 'copyofsnap1/test1/145/12' 2023-10-30T12:32:05.815 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/145/13' -> 'copyofsnap1/test1/145/13' 2023-10-30T12:32:05.816 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/145/14' -> 'copyofsnap1/test1/145/14' 2023-10-30T12:32:05.816 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/145/15' -> 'copyofsnap1/test1/145/15' 2023-10-30T12:32:05.816 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/145/16' -> 'copyofsnap1/test1/145/16' 2023-10-30T12:32:05.816 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/145/17' -> 'copyofsnap1/test1/145/17' 2023-10-30T12:32:05.816 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/145/18' -> 'copyofsnap1/test1/145/18' 2023-10-30T12:32:05.817 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/145/19' -> 'copyofsnap1/test1/145/19' 2023-10-30T12:32:05.817 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/145/20' -> 'copyofsnap1/test1/145/20' 2023-10-30T12:32:05.817 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/145/10' -> 'copyofsnap1/test1/145/10' 2023-10-30T12:32:05.817 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/146' -> 'copyofsnap1/test1/146' 2023-10-30T12:32:05.817 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/146/11' -> 'copyofsnap1/test1/146/11' 2023-10-30T12:32:05.817 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/146/12' -> 'copyofsnap1/test1/146/12' 2023-10-30T12:32:05.818 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/146/13' -> 'copyofsnap1/test1/146/13' 2023-10-30T12:32:05.818 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/146/14' -> 'copyofsnap1/test1/146/14' 2023-10-30T12:32:05.818 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/146/15' -> 'copyofsnap1/test1/146/15' 2023-10-30T12:32:05.818 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/146/16' -> 'copyofsnap1/test1/146/16' 2023-10-30T12:32:05.818 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/146/17' -> 'copyofsnap1/test1/146/17' 2023-10-30T12:32:05.819 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/146/18' -> 'copyofsnap1/test1/146/18' 2023-10-30T12:32:05.819 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/146/19' -> 'copyofsnap1/test1/146/19' 2023-10-30T12:32:05.819 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/146/20' -> 'copyofsnap1/test1/146/20' 2023-10-30T12:32:05.819 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/146/10' -> 'copyofsnap1/test1/146/10' 2023-10-30T12:32:05.819 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/147' -> 'copyofsnap1/test1/147' 2023-10-30T12:32:05.819 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/147/11' -> 'copyofsnap1/test1/147/11' 2023-10-30T12:32:05.820 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/147/12' -> 'copyofsnap1/test1/147/12' 2023-10-30T12:32:06.372 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/147/13' -> 'copyofsnap1/test1/147/13' 2023-10-30T12:32:06.372 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/147/14' -> 'copyofsnap1/test1/147/14' 2023-10-30T12:32:06.373 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/147/15' -> 'copyofsnap1/test1/147/15' 2023-10-30T12:32:06.373 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/147/16' -> 'copyofsnap1/test1/147/16' 2023-10-30T12:32:06.373 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/147/17' -> 'copyofsnap1/test1/147/17' 2023-10-30T12:32:06.373 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/147/18' -> 'copyofsnap1/test1/147/18' 2023-10-30T12:32:06.373 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/147/19' -> 'copyofsnap1/test1/147/19' 2023-10-30T12:32:06.374 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/147/20' -> 'copyofsnap1/test1/147/20' 2023-10-30T12:32:06.374 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/147/10' -> 'copyofsnap1/test1/147/10' 2023-10-30T12:32:06.374 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/148' -> 'copyofsnap1/test1/148' 2023-10-30T12:32:06.374 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/148/11' -> 'copyofsnap1/test1/148/11' 2023-10-30T12:32:06.374 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/148/12' -> 'copyofsnap1/test1/148/12' 2023-10-30T12:32:06.375 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/148/13' -> 'copyofsnap1/test1/148/13' 2023-10-30T12:32:06.375 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/148/14' -> 'copyofsnap1/test1/148/14' 2023-10-30T12:32:06.375 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/148/15' -> 'copyofsnap1/test1/148/15' 2023-10-30T12:32:06.375 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/148/16' -> 'copyofsnap1/test1/148/16' 2023-10-30T12:32:06.375 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/148/17' -> 'copyofsnap1/test1/148/17' 2023-10-30T12:32:06.375 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/148/18' -> 'copyofsnap1/test1/148/18' 2023-10-30T12:32:06.376 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/148/19' -> 'copyofsnap1/test1/148/19' 2023-10-30T12:32:06.376 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/148/20' -> 'copyofsnap1/test1/148/20' 2023-10-30T12:32:06.376 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/148/10' -> 'copyofsnap1/test1/148/10' 2023-10-30T12:32:06.376 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/149' -> 'copyofsnap1/test1/149' 2023-10-30T12:32:06.376 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/149/11' -> 'copyofsnap1/test1/149/11' 2023-10-30T12:32:06.376 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/149/12' -> 'copyofsnap1/test1/149/12' 2023-10-30T12:32:06.377 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/149/13' -> 'copyofsnap1/test1/149/13' 2023-10-30T12:32:06.377 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/149/14' -> 'copyofsnap1/test1/149/14' 2023-10-30T12:32:06.377 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/149/15' -> 'copyofsnap1/test1/149/15' 2023-10-30T12:32:06.377 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/149/16' -> 'copyofsnap1/test1/149/16' 2023-10-30T12:32:06.377 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/149/17' -> 'copyofsnap1/test1/149/17' 2023-10-30T12:32:06.377 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/149/18' -> 'copyofsnap1/test1/149/18' 2023-10-30T12:32:06.378 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/149/19' -> 'copyofsnap1/test1/149/19' 2023-10-30T12:32:06.378 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/149/20' -> 'copyofsnap1/test1/149/20' 2023-10-30T12:32:06.378 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/149/10' -> 'copyofsnap1/test1/149/10' 2023-10-30T12:32:06.378 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/150' -> 'copyofsnap1/test1/150' 2023-10-30T12:32:06.378 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/150/11' -> 'copyofsnap1/test1/150/11' 2023-10-30T12:32:06.379 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/150/12' -> 'copyofsnap1/test1/150/12' 2023-10-30T12:32:06.379 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/150/13' -> 'copyofsnap1/test1/150/13' 2023-10-30T12:32:06.379 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/150/14' -> 'copyofsnap1/test1/150/14' 2023-10-30T12:32:06.379 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/150/15' -> 'copyofsnap1/test1/150/15' 2023-10-30T12:32:06.379 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/150/16' -> 'copyofsnap1/test1/150/16' 2023-10-30T12:32:06.379 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/150/17' -> 'copyofsnap1/test1/150/17' 2023-10-30T12:32:06.380 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/150/18' -> 'copyofsnap1/test1/150/18' 2023-10-30T12:32:06.380 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/150/19' -> 'copyofsnap1/test1/150/19' 2023-10-30T12:32:06.380 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/150/20' -> 'copyofsnap1/test1/150/20' 2023-10-30T12:32:06.380 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/150/10' -> 'copyofsnap1/test1/150/10' 2023-10-30T12:32:06.380 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/151' -> 'copyofsnap1/test1/151' 2023-10-30T12:32:06.380 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/151/11' -> 'copyofsnap1/test1/151/11' 2023-10-30T12:32:06.381 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/151/12' -> 'copyofsnap1/test1/151/12' 2023-10-30T12:32:06.381 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/151/13' -> 'copyofsnap1/test1/151/13' 2023-10-30T12:32:06.381 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/151/14' -> 'copyofsnap1/test1/151/14' 2023-10-30T12:32:06.381 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/151/15' -> 'copyofsnap1/test1/151/15' 2023-10-30T12:32:06.381 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/151/16' -> 'copyofsnap1/test1/151/16' 2023-10-30T12:32:06.381 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/151/17' -> 'copyofsnap1/test1/151/17' 2023-10-30T12:32:06.382 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/151/18' -> 'copyofsnap1/test1/151/18' 2023-10-30T12:32:06.382 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/151/19' -> 'copyofsnap1/test1/151/19' 2023-10-30T12:32:06.382 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/151/20' -> 'copyofsnap1/test1/151/20' 2023-10-30T12:32:06.382 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/151/10' -> 'copyofsnap1/test1/151/10' 2023-10-30T12:32:06.382 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/152' -> 'copyofsnap1/test1/152' 2023-10-30T12:32:06.382 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/152/11' -> 'copyofsnap1/test1/152/11' 2023-10-30T12:32:06.383 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/152/12' -> 'copyofsnap1/test1/152/12' 2023-10-30T12:32:06.383 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/152/13' -> 'copyofsnap1/test1/152/13' 2023-10-30T12:32:06.383 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/152/14' -> 'copyofsnap1/test1/152/14' 2023-10-30T12:32:06.383 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/152/15' -> 'copyofsnap1/test1/152/15' 2023-10-30T12:32:06.383 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/152/16' -> 'copyofsnap1/test1/152/16' 2023-10-30T12:32:06.383 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/152/17' -> 'copyofsnap1/test1/152/17' 2023-10-30T12:32:06.384 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/152/18' -> 'copyofsnap1/test1/152/18' 2023-10-30T12:32:06.384 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/152/19' -> 'copyofsnap1/test1/152/19' 2023-10-30T12:32:06.384 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/152/20' -> 'copyofsnap1/test1/152/20' 2023-10-30T12:32:06.384 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/152/10' -> 'copyofsnap1/test1/152/10' 2023-10-30T12:32:06.384 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/153' -> 'copyofsnap1/test1/153' 2023-10-30T12:32:06.384 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/153/11' -> 'copyofsnap1/test1/153/11' 2023-10-30T12:32:06.385 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/153/12' -> 'copyofsnap1/test1/153/12' 2023-10-30T12:32:06.385 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/153/13' -> 'copyofsnap1/test1/153/13' 2023-10-30T12:32:06.385 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/153/14' -> 'copyofsnap1/test1/153/14' 2023-10-30T12:32:06.385 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/153/15' -> 'copyofsnap1/test1/153/15' 2023-10-30T12:32:06.385 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/153/16' -> 'copyofsnap1/test1/153/16' 2023-10-30T12:32:06.385 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/153/17' -> 'copyofsnap1/test1/153/17' 2023-10-30T12:32:06.386 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/153/18' -> 'copyofsnap1/test1/153/18' 2023-10-30T12:32:06.386 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/153/19' -> 'copyofsnap1/test1/153/19' 2023-10-30T12:32:06.386 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/153/20' -> 'copyofsnap1/test1/153/20' 2023-10-30T12:32:06.386 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/153/10' -> 'copyofsnap1/test1/153/10' 2023-10-30T12:32:06.639 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/154' -> 'copyofsnap1/test1/154' 2023-10-30T12:32:06.640 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/154/11' -> 'copyofsnap1/test1/154/11' 2023-10-30T12:32:06.640 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/154/12' -> 'copyofsnap1/test1/154/12' 2023-10-30T12:32:06.640 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/154/13' -> 'copyofsnap1/test1/154/13' 2023-10-30T12:32:06.640 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/154/14' -> 'copyofsnap1/test1/154/14' 2023-10-30T12:32:06.641 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/154/15' -> 'copyofsnap1/test1/154/15' 2023-10-30T12:32:06.641 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/154/16' -> 'copyofsnap1/test1/154/16' 2023-10-30T12:32:06.641 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/154/17' -> 'copyofsnap1/test1/154/17' 2023-10-30T12:32:06.641 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/154/18' -> 'copyofsnap1/test1/154/18' 2023-10-30T12:32:06.641 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/154/19' -> 'copyofsnap1/test1/154/19' 2023-10-30T12:32:06.642 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/154/20' -> 'copyofsnap1/test1/154/20' 2023-10-30T12:32:06.642 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/154/10' -> 'copyofsnap1/test1/154/10' 2023-10-30T12:32:06.642 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/155' -> 'copyofsnap1/test1/155' 2023-10-30T12:32:06.642 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/155/11' -> 'copyofsnap1/test1/155/11' 2023-10-30T12:32:06.643 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/155/12' -> 'copyofsnap1/test1/155/12' 2023-10-30T12:32:06.643 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/155/13' -> 'copyofsnap1/test1/155/13' 2023-10-30T12:32:06.643 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/155/14' -> 'copyofsnap1/test1/155/14' 2023-10-30T12:32:06.644 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/155/15' -> 'copyofsnap1/test1/155/15' 2023-10-30T12:32:06.644 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/155/16' -> 'copyofsnap1/test1/155/16' 2023-10-30T12:32:06.644 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/155/17' -> 'copyofsnap1/test1/155/17' 2023-10-30T12:32:06.644 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/155/18' -> 'copyofsnap1/test1/155/18' 2023-10-30T12:32:06.644 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/155/19' -> 'copyofsnap1/test1/155/19' 2023-10-30T12:32:06.645 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/155/20' -> 'copyofsnap1/test1/155/20' 2023-10-30T12:32:06.645 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/155/10' -> 'copyofsnap1/test1/155/10' 2023-10-30T12:32:06.645 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/156' -> 'copyofsnap1/test1/156' 2023-10-30T12:32:06.645 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/156/11' -> 'copyofsnap1/test1/156/11' 2023-10-30T12:32:06.645 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/156/12' -> 'copyofsnap1/test1/156/12' 2023-10-30T12:32:06.645 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/156/13' -> 'copyofsnap1/test1/156/13' 2023-10-30T12:32:06.646 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/156/14' -> 'copyofsnap1/test1/156/14' 2023-10-30T12:32:06.646 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/156/15' -> 'copyofsnap1/test1/156/15' 2023-10-30T12:32:06.646 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/156/16' -> 'copyofsnap1/test1/156/16' 2023-10-30T12:32:06.646 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/156/17' -> 'copyofsnap1/test1/156/17' 2023-10-30T12:32:06.646 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/156/18' -> 'copyofsnap1/test1/156/18' 2023-10-30T12:32:06.647 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/156/19' -> 'copyofsnap1/test1/156/19' 2023-10-30T12:32:06.647 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/156/20' -> 'copyofsnap1/test1/156/20' 2023-10-30T12:32:06.647 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/156/10' -> 'copyofsnap1/test1/156/10' 2023-10-30T12:32:06.647 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/157' -> 'copyofsnap1/test1/157' 2023-10-30T12:32:06.647 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/157/11' -> 'copyofsnap1/test1/157/11' 2023-10-30T12:32:06.648 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/157/12' -> 'copyofsnap1/test1/157/12' 2023-10-30T12:32:06.648 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/157/13' -> 'copyofsnap1/test1/157/13' 2023-10-30T12:32:06.648 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/157/14' -> 'copyofsnap1/test1/157/14' 2023-10-30T12:32:06.648 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/157/15' -> 'copyofsnap1/test1/157/15' 2023-10-30T12:32:06.648 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/157/16' -> 'copyofsnap1/test1/157/16' 2023-10-30T12:32:06.649 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/157/17' -> 'copyofsnap1/test1/157/17' 2023-10-30T12:32:06.649 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/157/18' -> 'copyofsnap1/test1/157/18' 2023-10-30T12:32:06.649 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/157/19' -> 'copyofsnap1/test1/157/19' 2023-10-30T12:32:06.649 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/157/20' -> 'copyofsnap1/test1/157/20' 2023-10-30T12:32:06.649 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/157/10' -> 'copyofsnap1/test1/157/10' 2023-10-30T12:32:06.650 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/158' -> 'copyofsnap1/test1/158' 2023-10-30T12:32:06.650 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/158/11' -> 'copyofsnap1/test1/158/11' 2023-10-30T12:32:06.650 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/158/12' -> 'copyofsnap1/test1/158/12' 2023-10-30T12:32:06.650 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/158/13' -> 'copyofsnap1/test1/158/13' 2023-10-30T12:32:06.650 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/158/14' -> 'copyofsnap1/test1/158/14' 2023-10-30T12:32:06.651 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/158/15' -> 'copyofsnap1/test1/158/15' 2023-10-30T12:32:06.651 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/158/16' -> 'copyofsnap1/test1/158/16' 2023-10-30T12:32:06.651 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/158/17' -> 'copyofsnap1/test1/158/17' 2023-10-30T12:32:06.651 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/158/18' -> 'copyofsnap1/test1/158/18' 2023-10-30T12:32:06.651 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/158/19' -> 'copyofsnap1/test1/158/19' 2023-10-30T12:32:06.651 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/158/20' -> 'copyofsnap1/test1/158/20' 2023-10-30T12:32:06.652 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/158/10' -> 'copyofsnap1/test1/158/10' 2023-10-30T12:32:06.652 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/159' -> 'copyofsnap1/test1/159' 2023-10-30T12:32:06.652 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/159/11' -> 'copyofsnap1/test1/159/11' 2023-10-30T12:32:06.652 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/159/12' -> 'copyofsnap1/test1/159/12' 2023-10-30T12:32:06.652 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/159/13' -> 'copyofsnap1/test1/159/13' 2023-10-30T12:32:06.653 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/159/14' -> 'copyofsnap1/test1/159/14' 2023-10-30T12:32:06.653 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/159/15' -> 'copyofsnap1/test1/159/15' 2023-10-30T12:32:06.653 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/159/16' -> 'copyofsnap1/test1/159/16' 2023-10-30T12:32:06.653 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/159/17' -> 'copyofsnap1/test1/159/17' 2023-10-30T12:32:06.653 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/159/18' -> 'copyofsnap1/test1/159/18' 2023-10-30T12:32:06.654 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/159/19' -> 'copyofsnap1/test1/159/19' 2023-10-30T12:32:06.654 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/159/20' -> 'copyofsnap1/test1/159/20' 2023-10-30T12:32:06.654 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/159/10' -> 'copyofsnap1/test1/159/10' 2023-10-30T12:32:06.654 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/160' -> 'copyofsnap1/test1/160' 2023-10-30T12:32:06.654 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/160/11' -> 'copyofsnap1/test1/160/11' 2023-10-30T12:32:06.655 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/160/12' -> 'copyofsnap1/test1/160/12' 2023-10-30T12:32:06.655 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/160/13' -> 'copyofsnap1/test1/160/13' 2023-10-30T12:32:06.655 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/160/14' -> 'copyofsnap1/test1/160/14' 2023-10-30T12:32:06.655 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/160/15' -> 'copyofsnap1/test1/160/15' 2023-10-30T12:32:06.655 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/160/16' -> 'copyofsnap1/test1/160/16' 2023-10-30T12:32:06.656 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/160/17' -> 'copyofsnap1/test1/160/17' 2023-10-30T12:32:06.656 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/160/18' -> 'copyofsnap1/test1/160/18' 2023-10-30T12:32:10.054 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/160/19' -> 'copyofsnap1/test1/160/19' 2023-10-30T12:32:10.055 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/160/20' -> 'copyofsnap1/test1/160/20' 2023-10-30T12:32:10.055 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/160/10' -> 'copyofsnap1/test1/160/10' 2023-10-30T12:32:10.055 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/161' -> 'copyofsnap1/test1/161' 2023-10-30T12:32:10.056 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/161/11' -> 'copyofsnap1/test1/161/11' 2023-10-30T12:32:10.056 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/161/12' -> 'copyofsnap1/test1/161/12' 2023-10-30T12:32:10.056 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/161/13' -> 'copyofsnap1/test1/161/13' 2023-10-30T12:32:10.056 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/161/14' -> 'copyofsnap1/test1/161/14' 2023-10-30T12:32:10.056 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/161/15' -> 'copyofsnap1/test1/161/15' 2023-10-30T12:32:10.057 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/161/16' -> 'copyofsnap1/test1/161/16' 2023-10-30T12:32:10.057 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/161/17' -> 'copyofsnap1/test1/161/17' 2023-10-30T12:32:10.057 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/161/18' -> 'copyofsnap1/test1/161/18' 2023-10-30T12:32:10.057 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/161/19' -> 'copyofsnap1/test1/161/19' 2023-10-30T12:32:10.058 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/161/20' -> 'copyofsnap1/test1/161/20' 2023-10-30T12:32:10.058 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/161/10' -> 'copyofsnap1/test1/161/10' 2023-10-30T12:32:10.058 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/162' -> 'copyofsnap1/test1/162' 2023-10-30T12:32:10.058 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/162/11' -> 'copyofsnap1/test1/162/11' 2023-10-30T12:32:10.058 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/162/12' -> 'copyofsnap1/test1/162/12' 2023-10-30T12:32:10.059 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/162/13' -> 'copyofsnap1/test1/162/13' 2023-10-30T12:32:10.059 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/162/14' -> 'copyofsnap1/test1/162/14' 2023-10-30T12:32:10.059 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/162/15' -> 'copyofsnap1/test1/162/15' 2023-10-30T12:32:10.059 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/162/16' -> 'copyofsnap1/test1/162/16' 2023-10-30T12:32:10.059 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/162/17' -> 'copyofsnap1/test1/162/17' 2023-10-30T12:32:10.060 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/162/18' -> 'copyofsnap1/test1/162/18' 2023-10-30T12:32:10.060 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/162/19' -> 'copyofsnap1/test1/162/19' 2023-10-30T12:32:10.060 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/162/20' -> 'copyofsnap1/test1/162/20' 2023-10-30T12:32:10.060 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/162/10' -> 'copyofsnap1/test1/162/10' 2023-10-30T12:32:10.060 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/163' -> 'copyofsnap1/test1/163' 2023-10-30T12:32:10.061 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/163/11' -> 'copyofsnap1/test1/163/11' 2023-10-30T12:32:10.061 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/163/12' -> 'copyofsnap1/test1/163/12' 2023-10-30T12:32:10.061 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/163/13' -> 'copyofsnap1/test1/163/13' 2023-10-30T12:32:10.061 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/163/14' -> 'copyofsnap1/test1/163/14' 2023-10-30T12:32:10.061 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/163/15' -> 'copyofsnap1/test1/163/15' 2023-10-30T12:32:10.061 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/163/16' -> 'copyofsnap1/test1/163/16' 2023-10-30T12:32:10.062 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/163/17' -> 'copyofsnap1/test1/163/17' 2023-10-30T12:32:10.062 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/163/18' -> 'copyofsnap1/test1/163/18' 2023-10-30T12:32:10.062 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/163/19' -> 'copyofsnap1/test1/163/19' 2023-10-30T12:32:10.062 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/163/20' -> 'copyofsnap1/test1/163/20' 2023-10-30T12:32:10.063 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/163/10' -> 'copyofsnap1/test1/163/10' 2023-10-30T12:32:10.063 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/164' -> 'copyofsnap1/test1/164' 2023-10-30T12:32:10.063 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/164/11' -> 'copyofsnap1/test1/164/11' 2023-10-30T12:32:10.063 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/164/12' -> 'copyofsnap1/test1/164/12' 2023-10-30T12:32:10.063 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/164/13' -> 'copyofsnap1/test1/164/13' 2023-10-30T12:32:10.063 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/164/14' -> 'copyofsnap1/test1/164/14' 2023-10-30T12:32:10.064 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/164/15' -> 'copyofsnap1/test1/164/15' 2023-10-30T12:32:10.064 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/164/16' -> 'copyofsnap1/test1/164/16' 2023-10-30T12:32:10.064 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/164/17' -> 'copyofsnap1/test1/164/17' 2023-10-30T12:32:10.064 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/164/18' -> 'copyofsnap1/test1/164/18' 2023-10-30T12:32:10.064 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/164/19' -> 'copyofsnap1/test1/164/19' 2023-10-30T12:32:10.065 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/164/20' -> 'copyofsnap1/test1/164/20' 2023-10-30T12:32:10.065 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/164/10' -> 'copyofsnap1/test1/164/10' 2023-10-30T12:32:10.065 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/165' -> 'copyofsnap1/test1/165' 2023-10-30T12:32:10.065 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/165/11' -> 'copyofsnap1/test1/165/11' 2023-10-30T12:32:10.065 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/165/12' -> 'copyofsnap1/test1/165/12' 2023-10-30T12:32:10.065 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/165/13' -> 'copyofsnap1/test1/165/13' 2023-10-30T12:32:10.066 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/165/14' -> 'copyofsnap1/test1/165/14' 2023-10-30T12:32:10.066 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/165/15' -> 'copyofsnap1/test1/165/15' 2023-10-30T12:32:10.066 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/165/16' -> 'copyofsnap1/test1/165/16' 2023-10-30T12:32:10.066 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/165/17' -> 'copyofsnap1/test1/165/17' 2023-10-30T12:32:10.067 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/165/18' -> 'copyofsnap1/test1/165/18' 2023-10-30T12:32:10.067 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/165/19' -> 'copyofsnap1/test1/165/19' 2023-10-30T12:32:10.067 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/165/20' -> 'copyofsnap1/test1/165/20' 2023-10-30T12:32:10.067 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/165/10' -> 'copyofsnap1/test1/165/10' 2023-10-30T12:32:10.067 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/166' -> 'copyofsnap1/test1/166' 2023-10-30T12:32:10.068 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/166/11' -> 'copyofsnap1/test1/166/11' 2023-10-30T12:32:10.068 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/166/12' -> 'copyofsnap1/test1/166/12' 2023-10-30T12:32:10.068 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/166/13' -> 'copyofsnap1/test1/166/13' 2023-10-30T12:32:10.068 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/166/14' -> 'copyofsnap1/test1/166/14' 2023-10-30T12:32:10.068 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/166/15' -> 'copyofsnap1/test1/166/15' 2023-10-30T12:32:10.069 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/166/16' -> 'copyofsnap1/test1/166/16' 2023-10-30T12:32:10.069 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/166/17' -> 'copyofsnap1/test1/166/17' 2023-10-30T12:32:10.069 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/166/18' -> 'copyofsnap1/test1/166/18' 2023-10-30T12:32:10.069 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/166/19' -> 'copyofsnap1/test1/166/19' 2023-10-30T12:32:10.069 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/166/20' -> 'copyofsnap1/test1/166/20' 2023-10-30T12:32:10.069 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/166/10' -> 'copyofsnap1/test1/166/10' 2023-10-30T12:32:10.070 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/167' -> 'copyofsnap1/test1/167' 2023-10-30T12:32:10.070 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/167/11' -> 'copyofsnap1/test1/167/11' 2023-10-30T12:32:10.070 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/167/12' -> 'copyofsnap1/test1/167/12' 2023-10-30T12:32:10.070 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/167/13' -> 'copyofsnap1/test1/167/13' 2023-10-30T12:32:10.070 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/167/14' -> 'copyofsnap1/test1/167/14' 2023-10-30T12:32:10.071 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/167/15' -> 'copyofsnap1/test1/167/15' 2023-10-30T12:32:10.938 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/167/16' -> 'copyofsnap1/test1/167/16' 2023-10-30T12:32:10.938 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/167/17' -> 'copyofsnap1/test1/167/17' 2023-10-30T12:32:10.938 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/167/18' -> 'copyofsnap1/test1/167/18' 2023-10-30T12:32:10.939 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/167/19' -> 'copyofsnap1/test1/167/19' 2023-10-30T12:32:10.939 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/167/20' -> 'copyofsnap1/test1/167/20' 2023-10-30T12:32:10.939 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/167/10' -> 'copyofsnap1/test1/167/10' 2023-10-30T12:32:10.939 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/168' -> 'copyofsnap1/test1/168' 2023-10-30T12:32:10.939 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/168/11' -> 'copyofsnap1/test1/168/11' 2023-10-30T12:32:10.939 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/168/12' -> 'copyofsnap1/test1/168/12' 2023-10-30T12:32:10.940 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/168/13' -> 'copyofsnap1/test1/168/13' 2023-10-30T12:32:10.940 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/168/14' -> 'copyofsnap1/test1/168/14' 2023-10-30T12:32:10.940 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/168/15' -> 'copyofsnap1/test1/168/15' 2023-10-30T12:32:10.940 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/168/16' -> 'copyofsnap1/test1/168/16' 2023-10-30T12:32:10.940 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/168/17' -> 'copyofsnap1/test1/168/17' 2023-10-30T12:32:10.941 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/168/18' -> 'copyofsnap1/test1/168/18' 2023-10-30T12:32:10.941 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/168/19' -> 'copyofsnap1/test1/168/19' 2023-10-30T12:32:10.941 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/168/20' -> 'copyofsnap1/test1/168/20' 2023-10-30T12:32:10.941 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/168/10' -> 'copyofsnap1/test1/168/10' 2023-10-30T12:32:10.941 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/169' -> 'copyofsnap1/test1/169' 2023-10-30T12:32:10.942 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/169/11' -> 'copyofsnap1/test1/169/11' 2023-10-30T12:32:10.942 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/169/12' -> 'copyofsnap1/test1/169/12' 2023-10-30T12:32:10.942 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/169/13' -> 'copyofsnap1/test1/169/13' 2023-10-30T12:32:10.942 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/169/14' -> 'copyofsnap1/test1/169/14' 2023-10-30T12:32:10.942 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/169/15' -> 'copyofsnap1/test1/169/15' 2023-10-30T12:32:10.943 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/169/16' -> 'copyofsnap1/test1/169/16' 2023-10-30T12:32:10.943 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/169/17' -> 'copyofsnap1/test1/169/17' 2023-10-30T12:32:10.943 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/169/18' -> 'copyofsnap1/test1/169/18' 2023-10-30T12:32:10.943 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/169/19' -> 'copyofsnap1/test1/169/19' 2023-10-30T12:32:10.943 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/169/20' -> 'copyofsnap1/test1/169/20' 2023-10-30T12:32:10.943 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/169/10' -> 'copyofsnap1/test1/169/10' 2023-10-30T12:32:10.944 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/170' -> 'copyofsnap1/test1/170' 2023-10-30T12:32:10.944 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/170/11' -> 'copyofsnap1/test1/170/11' 2023-10-30T12:32:10.944 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/170/12' -> 'copyofsnap1/test1/170/12' 2023-10-30T12:32:10.944 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/170/13' -> 'copyofsnap1/test1/170/13' 2023-10-30T12:32:10.944 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/170/14' -> 'copyofsnap1/test1/170/14' 2023-10-30T12:32:10.945 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/170/15' -> 'copyofsnap1/test1/170/15' 2023-10-30T12:32:10.945 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/170/16' -> 'copyofsnap1/test1/170/16' 2023-10-30T12:32:10.945 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/170/17' -> 'copyofsnap1/test1/170/17' 2023-10-30T12:32:10.945 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/170/18' -> 'copyofsnap1/test1/170/18' 2023-10-30T12:32:10.946 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/170/19' -> 'copyofsnap1/test1/170/19' 2023-10-30T12:32:10.946 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/170/20' -> 'copyofsnap1/test1/170/20' 2023-10-30T12:32:10.946 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/170/10' -> 'copyofsnap1/test1/170/10' 2023-10-30T12:32:10.946 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/171' -> 'copyofsnap1/test1/171' 2023-10-30T12:32:10.947 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/171/11' -> 'copyofsnap1/test1/171/11' 2023-10-30T12:32:10.947 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/171/12' -> 'copyofsnap1/test1/171/12' 2023-10-30T12:32:10.947 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/171/13' -> 'copyofsnap1/test1/171/13' 2023-10-30T12:32:10.947 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/171/14' -> 'copyofsnap1/test1/171/14' 2023-10-30T12:32:10.947 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/171/15' -> 'copyofsnap1/test1/171/15' 2023-10-30T12:32:10.948 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/171/16' -> 'copyofsnap1/test1/171/16' 2023-10-30T12:32:10.948 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/171/17' -> 'copyofsnap1/test1/171/17' 2023-10-30T12:32:10.948 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/171/18' -> 'copyofsnap1/test1/171/18' 2023-10-30T12:32:10.948 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/171/19' -> 'copyofsnap1/test1/171/19' 2023-10-30T12:32:10.948 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/171/20' -> 'copyofsnap1/test1/171/20' 2023-10-30T12:32:10.948 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/171/10' -> 'copyofsnap1/test1/171/10' 2023-10-30T12:32:10.949 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/172' -> 'copyofsnap1/test1/172' 2023-10-30T12:32:10.949 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/172/11' -> 'copyofsnap1/test1/172/11' 2023-10-30T12:32:10.949 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/172/12' -> 'copyofsnap1/test1/172/12' 2023-10-30T12:32:10.949 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/172/13' -> 'copyofsnap1/test1/172/13' 2023-10-30T12:32:10.949 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/172/14' -> 'copyofsnap1/test1/172/14' 2023-10-30T12:32:10.950 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/172/15' -> 'copyofsnap1/test1/172/15' 2023-10-30T12:32:10.950 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/172/16' -> 'copyofsnap1/test1/172/16' 2023-10-30T12:32:10.950 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/172/17' -> 'copyofsnap1/test1/172/17' 2023-10-30T12:32:10.950 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/172/18' -> 'copyofsnap1/test1/172/18' 2023-10-30T12:32:10.950 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/172/19' -> 'copyofsnap1/test1/172/19' 2023-10-30T12:32:10.950 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/172/20' -> 'copyofsnap1/test1/172/20' 2023-10-30T12:32:10.951 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/172/10' -> 'copyofsnap1/test1/172/10' 2023-10-30T12:32:10.951 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/173' -> 'copyofsnap1/test1/173' 2023-10-30T12:32:10.951 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/173/11' -> 'copyofsnap1/test1/173/11' 2023-10-30T12:32:10.951 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/173/12' -> 'copyofsnap1/test1/173/12' 2023-10-30T12:32:10.951 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/173/13' -> 'copyofsnap1/test1/173/13' 2023-10-30T12:32:10.952 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/173/14' -> 'copyofsnap1/test1/173/14' 2023-10-30T12:32:10.952 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/173/15' -> 'copyofsnap1/test1/173/15' 2023-10-30T12:32:10.952 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/173/16' -> 'copyofsnap1/test1/173/16' 2023-10-30T12:32:10.952 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/173/17' -> 'copyofsnap1/test1/173/17' 2023-10-30T12:32:10.952 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/173/18' -> 'copyofsnap1/test1/173/18' 2023-10-30T12:32:10.952 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/173/19' -> 'copyofsnap1/test1/173/19' 2023-10-30T12:32:10.953 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/173/20' -> 'copyofsnap1/test1/173/20' 2023-10-30T12:32:10.953 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/173/10' -> 'copyofsnap1/test1/173/10' 2023-10-30T12:32:10.953 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/174' -> 'copyofsnap1/test1/174' 2023-10-30T12:32:10.953 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/174/11' -> 'copyofsnap1/test1/174/11' 2023-10-30T12:32:10.953 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/174/12' -> 'copyofsnap1/test1/174/12' 2023-10-30T12:32:11.233 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/174/13' -> 'copyofsnap1/test1/174/13' 2023-10-30T12:32:11.233 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/174/14' -> 'copyofsnap1/test1/174/14' 2023-10-30T12:32:11.234 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/174/15' -> 'copyofsnap1/test1/174/15' 2023-10-30T12:32:11.234 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/174/16' -> 'copyofsnap1/test1/174/16' 2023-10-30T12:32:11.234 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/174/17' -> 'copyofsnap1/test1/174/17' 2023-10-30T12:32:11.234 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/174/18' -> 'copyofsnap1/test1/174/18' 2023-10-30T12:32:11.234 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/174/19' -> 'copyofsnap1/test1/174/19' 2023-10-30T12:32:11.235 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/174/20' -> 'copyofsnap1/test1/174/20' 2023-10-30T12:32:11.235 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/174/10' -> 'copyofsnap1/test1/174/10' 2023-10-30T12:32:11.235 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/175' -> 'copyofsnap1/test1/175' 2023-10-30T12:32:11.235 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/175/11' -> 'copyofsnap1/test1/175/11' 2023-10-30T12:32:11.235 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/175/12' -> 'copyofsnap1/test1/175/12' 2023-10-30T12:32:11.236 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/175/13' -> 'copyofsnap1/test1/175/13' 2023-10-30T12:32:11.236 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/175/14' -> 'copyofsnap1/test1/175/14' 2023-10-30T12:32:11.236 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/175/15' -> 'copyofsnap1/test1/175/15' 2023-10-30T12:32:11.236 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/175/16' -> 'copyofsnap1/test1/175/16' 2023-10-30T12:32:11.236 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/175/17' -> 'copyofsnap1/test1/175/17' 2023-10-30T12:32:11.237 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/175/18' -> 'copyofsnap1/test1/175/18' 2023-10-30T12:32:11.237 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/175/19' -> 'copyofsnap1/test1/175/19' 2023-10-30T12:32:11.237 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/175/20' -> 'copyofsnap1/test1/175/20' 2023-10-30T12:32:11.237 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/175/10' -> 'copyofsnap1/test1/175/10' 2023-10-30T12:32:11.237 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/176' -> 'copyofsnap1/test1/176' 2023-10-30T12:32:11.238 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/176/11' -> 'copyofsnap1/test1/176/11' 2023-10-30T12:32:11.238 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/176/12' -> 'copyofsnap1/test1/176/12' 2023-10-30T12:32:11.238 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/176/13' -> 'copyofsnap1/test1/176/13' 2023-10-30T12:32:11.238 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/176/14' -> 'copyofsnap1/test1/176/14' 2023-10-30T12:32:11.238 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/176/15' -> 'copyofsnap1/test1/176/15' 2023-10-30T12:32:11.238 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/176/16' -> 'copyofsnap1/test1/176/16' 2023-10-30T12:32:11.239 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/176/17' -> 'copyofsnap1/test1/176/17' 2023-10-30T12:32:11.239 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/176/18' -> 'copyofsnap1/test1/176/18' 2023-10-30T12:32:11.239 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/176/19' -> 'copyofsnap1/test1/176/19' 2023-10-30T12:32:11.239 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/176/20' -> 'copyofsnap1/test1/176/20' 2023-10-30T12:32:11.239 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/176/10' -> 'copyofsnap1/test1/176/10' 2023-10-30T12:32:11.240 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/177' -> 'copyofsnap1/test1/177' 2023-10-30T12:32:11.240 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/177/11' -> 'copyofsnap1/test1/177/11' 2023-10-30T12:32:11.240 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/177/12' -> 'copyofsnap1/test1/177/12' 2023-10-30T12:32:11.240 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/177/13' -> 'copyofsnap1/test1/177/13' 2023-10-30T12:32:11.240 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/177/14' -> 'copyofsnap1/test1/177/14' 2023-10-30T12:32:11.241 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/177/15' -> 'copyofsnap1/test1/177/15' 2023-10-30T12:32:11.241 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/177/16' -> 'copyofsnap1/test1/177/16' 2023-10-30T12:32:11.241 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/177/17' -> 'copyofsnap1/test1/177/17' 2023-10-30T12:32:11.241 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/177/18' -> 'copyofsnap1/test1/177/18' 2023-10-30T12:32:11.241 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/177/19' -> 'copyofsnap1/test1/177/19' 2023-10-30T12:32:11.242 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/177/20' -> 'copyofsnap1/test1/177/20' 2023-10-30T12:32:11.242 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/177/10' -> 'copyofsnap1/test1/177/10' 2023-10-30T12:32:11.242 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/178' -> 'copyofsnap1/test1/178' 2023-10-30T12:32:11.242 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/178/11' -> 'copyofsnap1/test1/178/11' 2023-10-30T12:32:11.242 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/178/12' -> 'copyofsnap1/test1/178/12' 2023-10-30T12:32:11.243 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/178/13' -> 'copyofsnap1/test1/178/13' 2023-10-30T12:32:11.243 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/178/14' -> 'copyofsnap1/test1/178/14' 2023-10-30T12:32:11.243 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/178/15' -> 'copyofsnap1/test1/178/15' 2023-10-30T12:32:11.243 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/178/16' -> 'copyofsnap1/test1/178/16' 2023-10-30T12:32:11.243 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/178/17' -> 'copyofsnap1/test1/178/17' 2023-10-30T12:32:11.244 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/178/18' -> 'copyofsnap1/test1/178/18' 2023-10-30T12:32:11.244 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/178/19' -> 'copyofsnap1/test1/178/19' 2023-10-30T12:32:11.244 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/178/20' -> 'copyofsnap1/test1/178/20' 2023-10-30T12:32:11.244 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/178/10' -> 'copyofsnap1/test1/178/10' 2023-10-30T12:32:11.244 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/179' -> 'copyofsnap1/test1/179' 2023-10-30T12:32:11.244 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/179/11' -> 'copyofsnap1/test1/179/11' 2023-10-30T12:32:11.245 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/179/12' -> 'copyofsnap1/test1/179/12' 2023-10-30T12:32:11.245 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/179/13' -> 'copyofsnap1/test1/179/13' 2023-10-30T12:32:11.245 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/179/14' -> 'copyofsnap1/test1/179/14' 2023-10-30T12:32:11.245 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/179/15' -> 'copyofsnap1/test1/179/15' 2023-10-30T12:32:11.245 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/179/16' -> 'copyofsnap1/test1/179/16' 2023-10-30T12:32:11.245 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/179/17' -> 'copyofsnap1/test1/179/17' 2023-10-30T12:32:11.246 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/179/18' -> 'copyofsnap1/test1/179/18' 2023-10-30T12:32:11.246 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/179/19' -> 'copyofsnap1/test1/179/19' 2023-10-30T12:32:11.246 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/179/20' -> 'copyofsnap1/test1/179/20' 2023-10-30T12:32:11.246 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/179/10' -> 'copyofsnap1/test1/179/10' 2023-10-30T12:32:11.246 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/180' -> 'copyofsnap1/test1/180' 2023-10-30T12:32:11.247 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/180/11' -> 'copyofsnap1/test1/180/11' 2023-10-30T12:32:11.247 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/180/12' -> 'copyofsnap1/test1/180/12' 2023-10-30T12:32:11.247 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/180/13' -> 'copyofsnap1/test1/180/13' 2023-10-30T12:32:11.247 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/180/14' -> 'copyofsnap1/test1/180/14' 2023-10-30T12:32:11.247 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/180/15' -> 'copyofsnap1/test1/180/15' 2023-10-30T12:32:11.247 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/180/16' -> 'copyofsnap1/test1/180/16' 2023-10-30T12:32:11.248 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/180/17' -> 'copyofsnap1/test1/180/17' 2023-10-30T12:32:11.248 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/180/18' -> 'copyofsnap1/test1/180/18' 2023-10-30T12:32:11.248 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/180/19' -> 'copyofsnap1/test1/180/19' 2023-10-30T12:32:11.248 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/180/20' -> 'copyofsnap1/test1/180/20' 2023-10-30T12:32:11.248 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/180/10' -> 'copyofsnap1/test1/180/10' 2023-10-30T12:32:11.752 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/181' -> 'copyofsnap1/test1/181' 2023-10-30T12:32:11.752 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/181/11' -> 'copyofsnap1/test1/181/11' 2023-10-30T12:32:11.752 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/181/12' -> 'copyofsnap1/test1/181/12' 2023-10-30T12:32:11.752 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/181/13' -> 'copyofsnap1/test1/181/13' 2023-10-30T12:32:11.753 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/181/14' -> 'copyofsnap1/test1/181/14' 2023-10-30T12:32:11.753 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/181/15' -> 'copyofsnap1/test1/181/15' 2023-10-30T12:32:11.753 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/181/16' -> 'copyofsnap1/test1/181/16' 2023-10-30T12:32:11.753 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/181/17' -> 'copyofsnap1/test1/181/17' 2023-10-30T12:32:11.753 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/181/18' -> 'copyofsnap1/test1/181/18' 2023-10-30T12:32:11.753 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/181/19' -> 'copyofsnap1/test1/181/19' 2023-10-30T12:32:11.754 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/181/20' -> 'copyofsnap1/test1/181/20' 2023-10-30T12:32:11.754 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/181/10' -> 'copyofsnap1/test1/181/10' 2023-10-30T12:32:11.754 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/182' -> 'copyofsnap1/test1/182' 2023-10-30T12:32:11.754 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/182/11' -> 'copyofsnap1/test1/182/11' 2023-10-30T12:32:11.754 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/182/12' -> 'copyofsnap1/test1/182/12' 2023-10-30T12:32:11.755 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/182/13' -> 'copyofsnap1/test1/182/13' 2023-10-30T12:32:11.755 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/182/14' -> 'copyofsnap1/test1/182/14' 2023-10-30T12:32:11.755 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/182/15' -> 'copyofsnap1/test1/182/15' 2023-10-30T12:32:11.755 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/182/16' -> 'copyofsnap1/test1/182/16' 2023-10-30T12:32:11.755 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/182/17' -> 'copyofsnap1/test1/182/17' 2023-10-30T12:32:11.756 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/182/18' -> 'copyofsnap1/test1/182/18' 2023-10-30T12:32:11.756 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/182/19' -> 'copyofsnap1/test1/182/19' 2023-10-30T12:32:11.756 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/182/20' -> 'copyofsnap1/test1/182/20' 2023-10-30T12:32:11.756 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/182/10' -> 'copyofsnap1/test1/182/10' 2023-10-30T12:32:11.757 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/183' -> 'copyofsnap1/test1/183' 2023-10-30T12:32:11.757 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/183/11' -> 'copyofsnap1/test1/183/11' 2023-10-30T12:32:11.757 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/183/12' -> 'copyofsnap1/test1/183/12' 2023-10-30T12:32:11.757 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/183/13' -> 'copyofsnap1/test1/183/13' 2023-10-30T12:32:11.757 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/183/14' -> 'copyofsnap1/test1/183/14' 2023-10-30T12:32:11.758 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/183/15' -> 'copyofsnap1/test1/183/15' 2023-10-30T12:32:11.758 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/183/16' -> 'copyofsnap1/test1/183/16' 2023-10-30T12:32:11.758 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/183/17' -> 'copyofsnap1/test1/183/17' 2023-10-30T12:32:11.758 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/183/18' -> 'copyofsnap1/test1/183/18' 2023-10-30T12:32:11.758 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/183/19' -> 'copyofsnap1/test1/183/19' 2023-10-30T12:32:11.759 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/183/20' -> 'copyofsnap1/test1/183/20' 2023-10-30T12:32:11.759 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/183/10' -> 'copyofsnap1/test1/183/10' 2023-10-30T12:32:11.759 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/184' -> 'copyofsnap1/test1/184' 2023-10-30T12:32:11.759 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/184/11' -> 'copyofsnap1/test1/184/11' 2023-10-30T12:32:11.760 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/184/12' -> 'copyofsnap1/test1/184/12' 2023-10-30T12:32:11.760 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/184/13' -> 'copyofsnap1/test1/184/13' 2023-10-30T12:32:11.760 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/184/14' -> 'copyofsnap1/test1/184/14' 2023-10-30T12:32:11.760 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/184/15' -> 'copyofsnap1/test1/184/15' 2023-10-30T12:32:11.760 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/184/16' -> 'copyofsnap1/test1/184/16' 2023-10-30T12:32:11.761 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/184/17' -> 'copyofsnap1/test1/184/17' 2023-10-30T12:32:11.761 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/184/18' -> 'copyofsnap1/test1/184/18' 2023-10-30T12:32:11.761 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/184/19' -> 'copyofsnap1/test1/184/19' 2023-10-30T12:32:11.761 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/184/20' -> 'copyofsnap1/test1/184/20' 2023-10-30T12:32:11.761 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/184/10' -> 'copyofsnap1/test1/184/10' 2023-10-30T12:32:11.762 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/185' -> 'copyofsnap1/test1/185' 2023-10-30T12:32:11.762 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/185/11' -> 'copyofsnap1/test1/185/11' 2023-10-30T12:32:11.762 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/185/12' -> 'copyofsnap1/test1/185/12' 2023-10-30T12:32:11.762 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/185/13' -> 'copyofsnap1/test1/185/13' 2023-10-30T12:32:11.762 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/185/14' -> 'copyofsnap1/test1/185/14' 2023-10-30T12:32:11.763 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/185/15' -> 'copyofsnap1/test1/185/15' 2023-10-30T12:32:11.763 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/185/16' -> 'copyofsnap1/test1/185/16' 2023-10-30T12:32:11.763 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/185/17' -> 'copyofsnap1/test1/185/17' 2023-10-30T12:32:11.763 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/185/18' -> 'copyofsnap1/test1/185/18' 2023-10-30T12:32:11.763 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/185/19' -> 'copyofsnap1/test1/185/19' 2023-10-30T12:32:11.763 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/185/20' -> 'copyofsnap1/test1/185/20' 2023-10-30T12:32:11.764 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/185/10' -> 'copyofsnap1/test1/185/10' 2023-10-30T12:32:11.764 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/186' -> 'copyofsnap1/test1/186' 2023-10-30T12:32:11.764 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/186/11' -> 'copyofsnap1/test1/186/11' 2023-10-30T12:32:11.764 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/186/12' -> 'copyofsnap1/test1/186/12' 2023-10-30T12:32:11.764 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/186/13' -> 'copyofsnap1/test1/186/13' 2023-10-30T12:32:11.765 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/186/14' -> 'copyofsnap1/test1/186/14' 2023-10-30T12:32:11.765 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/186/15' -> 'copyofsnap1/test1/186/15' 2023-10-30T12:32:11.765 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/186/16' -> 'copyofsnap1/test1/186/16' 2023-10-30T12:32:11.765 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/186/17' -> 'copyofsnap1/test1/186/17' 2023-10-30T12:32:11.765 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/186/18' -> 'copyofsnap1/test1/186/18' 2023-10-30T12:32:11.766 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/186/19' -> 'copyofsnap1/test1/186/19' 2023-10-30T12:32:11.766 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/186/20' -> 'copyofsnap1/test1/186/20' 2023-10-30T12:32:11.766 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/186/10' -> 'copyofsnap1/test1/186/10' 2023-10-30T12:32:11.766 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/187' -> 'copyofsnap1/test1/187' 2023-10-30T12:32:11.766 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/187/11' -> 'copyofsnap1/test1/187/11' 2023-10-30T12:32:11.767 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/187/12' -> 'copyofsnap1/test1/187/12' 2023-10-30T12:32:11.767 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/187/13' -> 'copyofsnap1/test1/187/13' 2023-10-30T12:32:11.767 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/187/14' -> 'copyofsnap1/test1/187/14' 2023-10-30T12:32:11.767 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/187/15' -> 'copyofsnap1/test1/187/15' 2023-10-30T12:32:11.767 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/187/16' -> 'copyofsnap1/test1/187/16' 2023-10-30T12:32:11.768 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/187/17' -> 'copyofsnap1/test1/187/17' 2023-10-30T12:32:11.768 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/187/18' -> 'copyofsnap1/test1/187/18' 2023-10-30T12:32:12.473 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/187/19' -> 'copyofsnap1/test1/187/19' 2023-10-30T12:32:12.473 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/187/20' -> 'copyofsnap1/test1/187/20' 2023-10-30T12:32:12.473 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/187/10' -> 'copyofsnap1/test1/187/10' 2023-10-30T12:32:12.473 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/188' -> 'copyofsnap1/test1/188' 2023-10-30T12:32:12.473 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/188/11' -> 'copyofsnap1/test1/188/11' 2023-10-30T12:32:12.474 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/188/12' -> 'copyofsnap1/test1/188/12' 2023-10-30T12:32:12.474 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/188/13' -> 'copyofsnap1/test1/188/13' 2023-10-30T12:32:12.474 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/188/14' -> 'copyofsnap1/test1/188/14' 2023-10-30T12:32:12.474 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/188/15' -> 'copyofsnap1/test1/188/15' 2023-10-30T12:32:12.475 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/188/16' -> 'copyofsnap1/test1/188/16' 2023-10-30T12:32:12.475 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/188/17' -> 'copyofsnap1/test1/188/17' 2023-10-30T12:32:12.475 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/188/18' -> 'copyofsnap1/test1/188/18' 2023-10-30T12:32:12.475 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/188/19' -> 'copyofsnap1/test1/188/19' 2023-10-30T12:32:12.475 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/188/20' -> 'copyofsnap1/test1/188/20' 2023-10-30T12:32:12.476 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/188/10' -> 'copyofsnap1/test1/188/10' 2023-10-30T12:32:12.476 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/189' -> 'copyofsnap1/test1/189' 2023-10-30T12:32:12.476 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/189/11' -> 'copyofsnap1/test1/189/11' 2023-10-30T12:32:12.476 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/189/12' -> 'copyofsnap1/test1/189/12' 2023-10-30T12:32:12.476 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/189/13' -> 'copyofsnap1/test1/189/13' 2023-10-30T12:32:12.476 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/189/14' -> 'copyofsnap1/test1/189/14' 2023-10-30T12:32:12.477 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/189/15' -> 'copyofsnap1/test1/189/15' 2023-10-30T12:32:12.477 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/189/16' -> 'copyofsnap1/test1/189/16' 2023-10-30T12:32:12.477 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/189/17' -> 'copyofsnap1/test1/189/17' 2023-10-30T12:32:12.477 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/189/18' -> 'copyofsnap1/test1/189/18' 2023-10-30T12:32:12.477 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/189/19' -> 'copyofsnap1/test1/189/19' 2023-10-30T12:32:12.478 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/189/20' -> 'copyofsnap1/test1/189/20' 2023-10-30T12:32:12.478 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/189/10' -> 'copyofsnap1/test1/189/10' 2023-10-30T12:32:12.478 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/190' -> 'copyofsnap1/test1/190' 2023-10-30T12:32:12.478 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/190/11' -> 'copyofsnap1/test1/190/11' 2023-10-30T12:32:12.478 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/190/12' -> 'copyofsnap1/test1/190/12' 2023-10-30T12:32:12.479 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/190/13' -> 'copyofsnap1/test1/190/13' 2023-10-30T12:32:12.479 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/190/14' -> 'copyofsnap1/test1/190/14' 2023-10-30T12:32:12.479 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/190/15' -> 'copyofsnap1/test1/190/15' 2023-10-30T12:32:12.479 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/190/16' -> 'copyofsnap1/test1/190/16' 2023-10-30T12:32:12.479 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/190/17' -> 'copyofsnap1/test1/190/17' 2023-10-30T12:32:12.479 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/190/18' -> 'copyofsnap1/test1/190/18' 2023-10-30T12:32:12.480 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/190/19' -> 'copyofsnap1/test1/190/19' 2023-10-30T12:32:12.480 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/190/20' -> 'copyofsnap1/test1/190/20' 2023-10-30T12:32:12.480 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/190/10' -> 'copyofsnap1/test1/190/10' 2023-10-30T12:32:12.480 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/191' -> 'copyofsnap1/test1/191' 2023-10-30T12:32:12.480 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/191/11' -> 'copyofsnap1/test1/191/11' 2023-10-30T12:32:12.481 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/191/12' -> 'copyofsnap1/test1/191/12' 2023-10-30T12:32:12.481 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/191/13' -> 'copyofsnap1/test1/191/13' 2023-10-30T12:32:12.481 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/191/14' -> 'copyofsnap1/test1/191/14' 2023-10-30T12:32:12.481 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/191/15' -> 'copyofsnap1/test1/191/15' 2023-10-30T12:32:12.481 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/191/16' -> 'copyofsnap1/test1/191/16' 2023-10-30T12:32:12.481 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/191/17' -> 'copyofsnap1/test1/191/17' 2023-10-30T12:32:12.482 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/191/18' -> 'copyofsnap1/test1/191/18' 2023-10-30T12:32:12.482 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/191/19' -> 'copyofsnap1/test1/191/19' 2023-10-30T12:32:12.482 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/191/20' -> 'copyofsnap1/test1/191/20' 2023-10-30T12:32:12.482 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/191/10' -> 'copyofsnap1/test1/191/10' 2023-10-30T12:32:12.482 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/192' -> 'copyofsnap1/test1/192' 2023-10-30T12:32:12.483 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/192/11' -> 'copyofsnap1/test1/192/11' 2023-10-30T12:32:12.483 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/192/12' -> 'copyofsnap1/test1/192/12' 2023-10-30T12:32:12.483 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/192/13' -> 'copyofsnap1/test1/192/13' 2023-10-30T12:32:12.483 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/192/14' -> 'copyofsnap1/test1/192/14' 2023-10-30T12:32:12.483 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/192/15' -> 'copyofsnap1/test1/192/15' 2023-10-30T12:32:12.484 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/192/16' -> 'copyofsnap1/test1/192/16' 2023-10-30T12:32:12.484 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/192/17' -> 'copyofsnap1/test1/192/17' 2023-10-30T12:32:12.484 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/192/18' -> 'copyofsnap1/test1/192/18' 2023-10-30T12:32:12.484 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/192/19' -> 'copyofsnap1/test1/192/19' 2023-10-30T12:32:12.484 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/192/20' -> 'copyofsnap1/test1/192/20' 2023-10-30T12:32:12.485 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/192/10' -> 'copyofsnap1/test1/192/10' 2023-10-30T12:32:12.485 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/193' -> 'copyofsnap1/test1/193' 2023-10-30T12:32:12.485 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/193/11' -> 'copyofsnap1/test1/193/11' 2023-10-30T12:32:12.485 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/193/12' -> 'copyofsnap1/test1/193/12' 2023-10-30T12:32:12.485 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/193/13' -> 'copyofsnap1/test1/193/13' 2023-10-30T12:32:12.486 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/193/14' -> 'copyofsnap1/test1/193/14' 2023-10-30T12:32:12.486 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/193/15' -> 'copyofsnap1/test1/193/15' 2023-10-30T12:32:12.486 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/193/16' -> 'copyofsnap1/test1/193/16' 2023-10-30T12:32:12.486 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/193/17' -> 'copyofsnap1/test1/193/17' 2023-10-30T12:32:12.486 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/193/18' -> 'copyofsnap1/test1/193/18' 2023-10-30T12:32:12.487 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/193/19' -> 'copyofsnap1/test1/193/19' 2023-10-30T12:32:12.487 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/193/20' -> 'copyofsnap1/test1/193/20' 2023-10-30T12:32:12.487 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/193/10' -> 'copyofsnap1/test1/193/10' 2023-10-30T12:32:12.487 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/194' -> 'copyofsnap1/test1/194' 2023-10-30T12:32:12.487 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/194/11' -> 'copyofsnap1/test1/194/11' 2023-10-30T12:32:12.488 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/194/12' -> 'copyofsnap1/test1/194/12' 2023-10-30T12:32:12.488 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/194/13' -> 'copyofsnap1/test1/194/13' 2023-10-30T12:32:12.488 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/194/14' -> 'copyofsnap1/test1/194/14' 2023-10-30T12:32:12.488 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/194/15' -> 'copyofsnap1/test1/194/15' 2023-10-30T12:32:13.334 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/194/16' -> 'copyofsnap1/test1/194/16' 2023-10-30T12:32:13.335 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/194/17' -> 'copyofsnap1/test1/194/17' 2023-10-30T12:32:13.335 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/194/18' -> 'copyofsnap1/test1/194/18' 2023-10-30T12:32:13.335 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/194/19' -> 'copyofsnap1/test1/194/19' 2023-10-30T12:32:13.336 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/194/20' -> 'copyofsnap1/test1/194/20' 2023-10-30T12:32:13.336 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/194/10' -> 'copyofsnap1/test1/194/10' 2023-10-30T12:32:13.336 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/195' -> 'copyofsnap1/test1/195' 2023-10-30T12:32:13.336 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/195/11' -> 'copyofsnap1/test1/195/11' 2023-10-30T12:32:13.337 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/195/12' -> 'copyofsnap1/test1/195/12' 2023-10-30T12:32:13.337 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/195/13' -> 'copyofsnap1/test1/195/13' 2023-10-30T12:32:13.337 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/195/14' -> 'copyofsnap1/test1/195/14' 2023-10-30T12:32:13.337 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/195/15' -> 'copyofsnap1/test1/195/15' 2023-10-30T12:32:13.337 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/195/16' -> 'copyofsnap1/test1/195/16' 2023-10-30T12:32:13.337 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/195/17' -> 'copyofsnap1/test1/195/17' 2023-10-30T12:32:13.338 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/195/18' -> 'copyofsnap1/test1/195/18' 2023-10-30T12:32:13.338 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/195/19' -> 'copyofsnap1/test1/195/19' 2023-10-30T12:32:13.338 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/195/20' -> 'copyofsnap1/test1/195/20' 2023-10-30T12:32:13.338 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/195/10' -> 'copyofsnap1/test1/195/10' 2023-10-30T12:32:13.338 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/196' -> 'copyofsnap1/test1/196' 2023-10-30T12:32:13.339 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/196/11' -> 'copyofsnap1/test1/196/11' 2023-10-30T12:32:13.339 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/196/12' -> 'copyofsnap1/test1/196/12' 2023-10-30T12:32:13.339 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/196/13' -> 'copyofsnap1/test1/196/13' 2023-10-30T12:32:13.339 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/196/14' -> 'copyofsnap1/test1/196/14' 2023-10-30T12:32:13.339 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/196/15' -> 'copyofsnap1/test1/196/15' 2023-10-30T12:32:13.340 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/196/16' -> 'copyofsnap1/test1/196/16' 2023-10-30T12:32:13.340 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/196/17' -> 'copyofsnap1/test1/196/17' 2023-10-30T12:32:13.340 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/196/18' -> 'copyofsnap1/test1/196/18' 2023-10-30T12:32:13.340 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/196/19' -> 'copyofsnap1/test1/196/19' 2023-10-30T12:32:13.340 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/196/20' -> 'copyofsnap1/test1/196/20' 2023-10-30T12:32:13.340 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/196/10' -> 'copyofsnap1/test1/196/10' 2023-10-30T12:32:13.341 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/197' -> 'copyofsnap1/test1/197' 2023-10-30T12:32:13.341 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/197/11' -> 'copyofsnap1/test1/197/11' 2023-10-30T12:32:13.341 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/197/12' -> 'copyofsnap1/test1/197/12' 2023-10-30T12:32:13.341 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/197/13' -> 'copyofsnap1/test1/197/13' 2023-10-30T12:32:13.341 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/197/14' -> 'copyofsnap1/test1/197/14' 2023-10-30T12:32:13.341 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/197/15' -> 'copyofsnap1/test1/197/15' 2023-10-30T12:32:13.342 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/197/16' -> 'copyofsnap1/test1/197/16' 2023-10-30T12:32:13.342 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/197/17' -> 'copyofsnap1/test1/197/17' 2023-10-30T12:32:13.342 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/197/18' -> 'copyofsnap1/test1/197/18' 2023-10-30T12:32:13.342 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/197/19' -> 'copyofsnap1/test1/197/19' 2023-10-30T12:32:13.342 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/197/20' -> 'copyofsnap1/test1/197/20' 2023-10-30T12:32:13.343 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/197/10' -> 'copyofsnap1/test1/197/10' 2023-10-30T12:32:13.343 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/198' -> 'copyofsnap1/test1/198' 2023-10-30T12:32:13.343 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/198/11' -> 'copyofsnap1/test1/198/11' 2023-10-30T12:32:13.343 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/198/12' -> 'copyofsnap1/test1/198/12' 2023-10-30T12:32:13.343 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/198/13' -> 'copyofsnap1/test1/198/13' 2023-10-30T12:32:13.344 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/198/14' -> 'copyofsnap1/test1/198/14' 2023-10-30T12:32:13.344 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/198/15' -> 'copyofsnap1/test1/198/15' 2023-10-30T12:32:13.344 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/198/16' -> 'copyofsnap1/test1/198/16' 2023-10-30T12:32:13.344 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/198/17' -> 'copyofsnap1/test1/198/17' 2023-10-30T12:32:13.344 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/198/18' -> 'copyofsnap1/test1/198/18' 2023-10-30T12:32:13.345 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/198/19' -> 'copyofsnap1/test1/198/19' 2023-10-30T12:32:13.345 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/198/20' -> 'copyofsnap1/test1/198/20' 2023-10-30T12:32:13.345 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/198/10' -> 'copyofsnap1/test1/198/10' 2023-10-30T12:32:13.345 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/199' -> 'copyofsnap1/test1/199' 2023-10-30T12:32:13.345 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/199/11' -> 'copyofsnap1/test1/199/11' 2023-10-30T12:32:13.345 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/199/12' -> 'copyofsnap1/test1/199/12' 2023-10-30T12:32:13.346 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/199/13' -> 'copyofsnap1/test1/199/13' 2023-10-30T12:32:13.346 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/199/14' -> 'copyofsnap1/test1/199/14' 2023-10-30T12:32:13.346 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/199/15' -> 'copyofsnap1/test1/199/15' 2023-10-30T12:32:13.346 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/199/16' -> 'copyofsnap1/test1/199/16' 2023-10-30T12:32:13.346 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/199/17' -> 'copyofsnap1/test1/199/17' 2023-10-30T12:32:13.347 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/199/18' -> 'copyofsnap1/test1/199/18' 2023-10-30T12:32:13.347 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/199/19' -> 'copyofsnap1/test1/199/19' 2023-10-30T12:32:13.347 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/199/20' -> 'copyofsnap1/test1/199/20' 2023-10-30T12:32:13.347 INFO:tasks.workunit.client.0.smithi125.stdout:'.snap/test1/199/10' -> 'copyofsnap1/test1/199/10' 2023-10-30T12:32:13.348 INFO:tasks.workunit.client.0.smithi125.stdout:Take third snapshot .snap/test3 2023-10-30T12:32:13.365 INFO:tasks.workunit.client.0.smithi125.stdout:Delete the snapshots... 2023-10-30T12:32:14.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:32:14.343 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:32:15.710 INFO:tasks.workunit.client.0.smithi125.stdout:.//.snap/test1 2023-10-30T12:32:15.710 INFO:tasks.workunit.client.0.smithi125.stdout:.//.snap/test2 2023-10-30T12:32:15.710 INFO:tasks.workunit.client.0.smithi125.stdout:.//.snap/test3 2023-10-30T12:32:16.468 INFO:tasks.workunit.client.0.smithi125.stdout:./274/.snap/snap-subdir-test 2023-10-30T12:32:16.478 INFO:tasks.workunit.client.0.smithi125.stdout:./166/.snap/snap-subdir-test 2023-10-30T12:32:16.489 INFO:tasks.workunit.client.0.smithi125.stdout:./256/.snap/snap-subdir-test 2023-10-30T12:32:16.499 INFO:tasks.workunit.client.0.smithi125.stdout:./247/.snap/snap-subdir-test 2023-10-30T12:32:16.507 INFO:tasks.workunit.client.0.smithi125.stdout:./248/.snap/snap-subdir-test 2023-10-30T12:32:16.766 INFO:tasks.workunit.client.0.smithi125.stdout:./147/.snap/snap-subdir-test 2023-10-30T12:32:16.807 INFO:tasks.workunit.client.0.smithi125.stdout:./151/.snap/snap-subdir-test 2023-10-30T12:32:16.839 INFO:tasks.workunit.client.0.smithi125.stdout:./170/.snap/snap-subdir-test 2023-10-30T12:32:16.847 INFO:tasks.workunit.client.0.smithi125.stdout:./167/.snap/snap-subdir-test 2023-10-30T12:32:17.135 INFO:tasks.workunit.client.0.smithi125.stdout:./140/.snap/snap-subdir-test 2023-10-30T12:32:17.146 INFO:tasks.workunit.client.0.smithi125.stdout:./142/.snap/snap-subdir-test 2023-10-30T12:32:17.160 INFO:tasks.workunit.client.0.smithi125.stdout:./150/.snap/snap-subdir-test 2023-10-30T12:32:17.244 INFO:tasks.workunit.client.0.smithi125.stdout:./273/.snap/snap-subdir-test 2023-10-30T12:32:17.254 INFO:tasks.workunit.client.0.smithi125.stdout:./269/.snap/snap-subdir-test 2023-10-30T12:32:17.263 INFO:tasks.workunit.client.0.smithi125.stdout:./105/.snap/snap-subdir-test 2023-10-30T12:32:17.657 INFO:tasks.workunit.client.0.smithi125.stdout:./210/.snap/snap-subdir-test 2023-10-30T12:32:17.667 INFO:tasks.workunit.client.0.smithi125.stdout:./265/.snap/snap-subdir-test 2023-10-30T12:32:17.678 INFO:tasks.workunit.client.0.smithi125.stdout:./218/.snap/snap-subdir-test 2023-10-30T12:32:17.687 INFO:tasks.workunit.client.0.smithi125.stdout:./290/.snap/snap-subdir-test 2023-10-30T12:32:17.696 INFO:tasks.workunit.client.0.smithi125.stdout:./149/.snap/snap-subdir-test 2023-10-30T12:32:17.708 INFO:tasks.workunit.client.0.smithi125.stdout:./109/.snap/snap-subdir-test 2023-10-30T12:32:17.719 INFO:tasks.workunit.client.0.smithi125.stdout:./292/.snap/snap-subdir-test 2023-10-30T12:32:17.893 INFO:tasks.workunit.client.0.smithi125.stdout:./242/.snap/snap-subdir-test 2023-10-30T12:32:17.905 INFO:tasks.workunit.client.0.smithi125.stdout:./158/.snap/snap-subdir-test 2023-10-30T12:32:17.914 INFO:tasks.workunit.client.0.smithi125.stdout:./243/.snap/snap-subdir-test 2023-10-30T12:32:17.927 INFO:tasks.workunit.client.0.smithi125.stdout:./106/.snap/snap-subdir-test 2023-10-30T12:32:17.937 INFO:tasks.workunit.client.0.smithi125.stdout:./250/.snap/snap-subdir-test 2023-10-30T12:32:17.947 INFO:tasks.workunit.client.0.smithi125.stdout:./221/.snap/snap-subdir-test 2023-10-30T12:32:17.957 INFO:tasks.workunit.client.0.smithi125.stdout:./244/.snap/snap-subdir-test 2023-10-30T12:32:17.970 INFO:tasks.workunit.client.0.smithi125.stdout:./258/.snap/snap-subdir-test 2023-10-30T12:32:17.978 INFO:tasks.workunit.client.0.smithi125.stdout:./268/.snap/snap-subdir-test 2023-10-30T12:32:17.987 INFO:tasks.workunit.client.0.smithi125.stdout:./189/.snap/snap-subdir-test 2023-10-30T12:32:17.990 INFO:tasks.workunit.client.0.smithi125.stdout:./285/.snap/snap-subdir-test 2023-10-30T12:32:17.998 INFO:tasks.workunit.client.0.smithi125.stdout:./208/.snap/snap-subdir-test 2023-10-30T12:32:18.002 INFO:tasks.workunit.client.0.smithi125.stdout:./288/.snap/snap-subdir-test 2023-10-30T12:32:18.012 INFO:tasks.workunit.client.0.smithi125.stdout:./209/.snap/snap-subdir-test 2023-10-30T12:32:18.020 INFO:tasks.workunit.client.0.smithi125.stdout:./253/.snap/snap-subdir-test 2023-10-30T12:32:18.465 INFO:tasks.workunit.client.0.smithi125.stdout:./249/.snap/snap-subdir-test 2023-10-30T12:32:18.477 INFO:tasks.workunit.client.0.smithi125.stdout:./133/.snap/snap-subdir-test 2023-10-30T12:32:18.480 INFO:tasks.workunit.client.0.smithi125.stdout:./291/.snap/snap-subdir-test 2023-10-30T12:32:18.491 INFO:tasks.workunit.client.0.smithi125.stdout:./226/.snap/snap-subdir-test 2023-10-30T12:32:18.501 INFO:tasks.workunit.client.0.smithi125.stdout:./173/.snap/snap-subdir-test 2023-10-30T12:32:18.509 INFO:tasks.workunit.client.0.smithi125.stdout:./174/.snap/snap-subdir-test 2023-10-30T12:32:18.518 INFO:tasks.workunit.client.0.smithi125.stdout:./299/.snap/snap-subdir-test 2023-10-30T12:32:18.528 INFO:tasks.workunit.client.0.smithi125.stdout:./262/.snap/snap-subdir-test 2023-10-30T12:32:18.536 INFO:tasks.workunit.client.0.smithi125.stdout:./231/.snap/snap-subdir-test 2023-10-30T12:32:18.544 INFO:tasks.workunit.client.0.smithi125.stdout:./214/.snap/snap-subdir-test 2023-10-30T12:32:18.547 INFO:tasks.workunit.client.0.smithi125.stdout:./281/.snap/snap-subdir-test 2023-10-30T12:32:18.551 INFO:tasks.workunit.client.0.smithi125.stdout:./289/.snap/snap-subdir-test 2023-10-30T12:32:18.560 INFO:tasks.workunit.client.0.smithi125.stdout:./283/.snap/snap-subdir-test 2023-10-30T12:32:18.569 INFO:tasks.workunit.client.0.smithi125.stdout:./232/.snap/snap-subdir-test 2023-10-30T12:32:18.577 INFO:tasks.workunit.client.0.smithi125.stdout:./298/.snap/snap-subdir-test 2023-10-30T12:32:18.585 INFO:tasks.workunit.client.0.smithi125.stdout:./233/.snap/snap-subdir-test 2023-10-30T12:32:18.591 INFO:tasks.workunit.client.0.smithi125.stdout:./155/.snap/snap-subdir-test 2023-10-30T12:32:19.741 INFO:tasks.workunit.client.0.smithi125.stdout:./114/.snap/snap-subdir-test 2023-10-30T12:32:20.947 INFO:tasks.workunit.client.0.smithi125.stdout:./159/.snap/snap-subdir-test 2023-10-30T12:32:20.956 INFO:tasks.workunit.client.0.smithi125.stdout:./187/.snap/snap-subdir-test 2023-10-30T12:32:20.967 INFO:tasks.workunit.client.0.smithi125.stdout:./185/.snap/snap-subdir-test 2023-10-30T12:32:20.974 INFO:tasks.workunit.client.0.smithi125.stdout:./293/.snap/snap-subdir-test 2023-10-30T12:32:20.982 INFO:tasks.workunit.client.0.smithi125.stdout:./228/.snap/snap-subdir-test 2023-10-30T12:32:20.989 INFO:tasks.workunit.client.0.smithi125.stdout:./120/.snap/snap-subdir-test 2023-10-30T12:32:20.995 INFO:tasks.workunit.client.0.smithi125.stdout:./110/.snap/snap-subdir-test 2023-10-30T12:32:21.038 INFO:tasks.workunit.client.0.smithi125.stdout:./267/.snap/snap-subdir-test 2023-10-30T12:32:21.982 INFO:tasks.workunit.client.0.smithi125.stdout:./202/.snap/snap-subdir-test 2023-10-30T12:32:21.993 INFO:tasks.workunit.client.0.smithi125.stdout:./137/.snap/snap-subdir-test 2023-10-30T12:32:22.002 INFO:tasks.workunit.client.0.smithi125.stdout:./246/.snap/snap-subdir-test 2023-10-30T12:32:22.248 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:22.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.391+0000 7f5927c17640 1 Processor -- start 2023-10-30T12:32:22.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.391+0000 7f5927c17640 1 -- start start 2023-10-30T12:32:22.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.391+0000 7f5927c17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59200c6f60 0x7f59201566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:22.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.391+0000 7f5927c17640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59200c0a50 con 0x7f59200c6b80 2023-10-30T12:32:22.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.395+0000 7f5927c17640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f592006f6e0 con 0x7f59200c05d0 2023-10-30T12:32:22.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.395+0000 7f5927c17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f592006f9c0 con 0x7f59200c6f60 2023-10-30T12:32:22.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.395+0000 7f59261b4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59200c6f60 0x7f59201566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:22.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.395+0000 7f59251b2640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f59200c05d0 0x7f592013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:55564/0 (socket says 172.21.15.125:55564) 2023-10-30T12:32:22.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.395+0000 7f59261b4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59200c6f60 0x7f59201566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:35778/0 (socket says 172.21.15.125:35778) 2023-10-30T12:32:22.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.395+0000 7f59251b2640 1 -- 172.21.15.125:0/3299824027 learned_addr learned my addr 172.21.15.125:0/3299824027 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:32:22.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.395+0000 7f59249b1640 1 -- 172.21.15.125:0/3299824027 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168941738 0 0) 0x7f59200c0a50 con 0x7f59200c6b80 2023-10-30T12:32:22.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.395+0000 7f59249b1640 1 -- 172.21.15.125:0/3299824027 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f590c005740 con 0x7f59200c6b80 2023-10-30T12:32:22.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.395+0000 7f59249b1640 1 -- 172.21.15.125:0/3299824027 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603799388 0 0) 0x7f592006f6e0 con 0x7f59200c05d0 2023-10-30T12:32:22.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.395+0000 7f59261b4640 1 -- 172.21.15.125:0/3299824027 >> v1:172.21.15.136:6789/0 conn(0x7f59200c05d0 legacy=0x7f592013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:22.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.395+0000 7f59261b4640 1 -- 172.21.15.125:0/3299824027 >> v1:172.21.15.125:6789/0 conn(0x7f59200c6b80 legacy=0x7f59200bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:22.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.395+0000 7f59261b4640 1 -- 172.21.15.125:0/3299824027 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f592006fca0 con 0x7f59200c6f60 2023-10-30T12:32:22.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.399+0000 7f59261b4640 1 --2- 172.21.15.125:0/3299824027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59200c6f60 0x7f59201566a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5918009dc0 tx=0x7f591801c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=569edd1a5511d89c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:22.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.399+0000 7f59249b1640 1 -- 172.21.15.125:0/3299824027 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59180a5840 con 0x7f59200c6f60 2023-10-30T12:32:22.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.399+0000 7f59249b1640 1 -- 172.21.15.125:0/3299824027 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59180a59a0 con 0x7f59200c6f60 2023-10-30T12:32:22.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.399+0000 7f59249b1640 1 -- 172.21.15.125:0/3299824027 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59180a5d10 con 0x7f59200c6f60 2023-10-30T12:32:22.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.399+0000 7f5927c17640 1 -- 172.21.15.125:0/3299824027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59200c6f60 msgr2=0x7f59201566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:22.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.399+0000 7f5927c17640 1 --2- 172.21.15.125:0/3299824027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59200c6f60 0x7f59201566a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5918009dc0 tx=0x7f591801c970 comp rx=0 tx=0).stop 2023-10-30T12:32:22.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.399+0000 7f5927c17640 1 -- 172.21.15.125:0/3299824027 shutdown_connections 2023-10-30T12:32:22.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.399+0000 7f5927c17640 1 --2- 172.21.15.125:0/3299824027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59200c6f60 0x7f59201566a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:22.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.399+0000 7f5927c17640 1 -- 172.21.15.125:0/3299824027 >> 172.21.15.125:0/3299824027 conn(0x7f59200c7910 msgr2=0x7f59200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:22.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.399+0000 7f5927c17640 1 -- 172.21.15.125:0/3299824027 shutdown_connections 2023-10-30T12:32:22.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.399+0000 7f5927c17640 1 -- 172.21.15.125:0/3299824027 wait complete. 2023-10-30T12:32:22.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f5927c17640 1 Processor -- start 2023-10-30T12:32:22.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f5927c17640 1 -- start start 2023-10-30T12:32:22.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f5927c17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59200c05d0 0x7f59200c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:22.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f5927c17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59200c6b80 0x7f5920153360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:22.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f5927c17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59200c6f60 0x7f59201538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:22.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f5927c17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59200adf00 con 0x7f59200c6b80 2023-10-30T12:32:22.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f5927c17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f592013e210 con 0x7f59200c05d0 2023-10-30T12:32:22.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f5927c17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59200ad8d0 con 0x7f59200c6f60 2023-10-30T12:32:22.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f59251b2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59200c6b80 0x7f5920153360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:22.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f59251b2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59200c6b80 0x7f5920153360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57460/0 (socket says 172.21.15.125:57460) 2023-10-30T12:32:22.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f59251b2640 1 -- 172.21.15.125:0/3906293737 learned_addr learned my addr 172.21.15.125:0/3906293737 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:22.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f59259b3640 1 --2- 172.21.15.125:0/3906293737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59200c05d0 0x7f59200c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:22.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f59261b4640 1 --2- 172.21.15.125:0/3906293737 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59200c6f60 0x7f59201538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:22.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f59251b2640 1 -- 172.21.15.125:0/3906293737 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59200c6f60 msgr2=0x7f59201538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:22.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f59251b2640 1 --2- 172.21.15.125:0/3906293737 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59200c6f60 0x7f59201538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:22.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f59251b2640 1 -- 172.21.15.125:0/3906293737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59200c05d0 msgr2=0x7f59200c9e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:22.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f59251b2640 1 --2- 172.21.15.125:0/3906293737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59200c05d0 0x7f59200c9e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:22.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f59251b2640 1 -- 172.21.15.125:0/3906293737 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5918009a70 con 0x7f59200c6b80 2023-10-30T12:32:22.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f59251b2640 1 --2- 172.21.15.125:0/3906293737 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59200c6b80 0x7f5920153360 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f591000c4b0 tx=0x7f591000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:22.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f59167fc640 1 -- 172.21.15.125:0/3906293737 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59100074e0 con 0x7f59200c6b80 2023-10-30T12:32:22.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f59167fc640 1 -- 172.21.15.125:0/3906293737 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5910007660 con 0x7f59200c6b80 2023-10-30T12:32:22.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f59167fc640 1 -- 172.21.15.125:0/3906293737 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5910004a30 con 0x7f59200c6b80 2023-10-30T12:32:22.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f5927c17640 1 -- 172.21.15.125:0/3906293737 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f592013e460 con 0x7f59200c6b80 2023-10-30T12:32:22.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f5927c17640 1 -- 172.21.15.125:0/3906293737 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59200adb90 con 0x7f59200c6b80 2023-10-30T12:32:22.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.403+0000 7f5927c17640 1 -- 172.21.15.125:0/3906293737 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5920154100 con 0x7f59200c6b80 2023-10-30T12:32:22.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.407+0000 7f59167fc640 1 -- 172.21.15.125:0/3906293737 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5910004b90 con 0x7f59200c6b80 2023-10-30T12:32:22.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.407+0000 7f59167fc640 1 --2- 172.21.15.125:0/3906293737 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59040664d0 0x7f5904068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:22.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.407+0000 7f59167fc640 1 -- 172.21.15.125:0/3906293737 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5852+0+0 (secure 0 0 0) 0x7f5910083040 con 0x7f59200c6b80 2023-10-30T12:32:22.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.407+0000 7f59259b3640 1 --2- 172.21.15.125:0/3906293737 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59040664d0 0x7f5904068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:22.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.407+0000 7f59259b3640 1 --2- 172.21.15.125:0/3906293737 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59040664d0 0x7f5904068990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f591c007fd0 tx=0x7f591c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:22.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.411+0000 7f59167fc640 1 -- 172.21.15.125:0/3906293737 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f59100b4090 con 0x7f59200c6b80 2023-10-30T12:32:22.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.591+0000 7f5927c17640 1 -- 172.21.15.125:0/3906293737 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59200add50 con 0x7f59200c6b80 2023-10-30T12:32:22.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.591+0000 7f59167fc640 1 -- 172.21.15.125:0/3906293737 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6546 (secure 0 0 0) 0x7f5910018020 con 0x7f59200c6b80 2023-10-30T12:32:22.594 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:22.594 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:02.105923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:22.595 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 22 2023-10-30T12:32:22.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.591+0000 7f5927c17640 1 -- 172.21.15.125:0/3906293737 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59040664d0 msgr2=0x7f5904068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:22.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.591+0000 7f5927c17640 1 --2- 172.21.15.125:0/3906293737 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59040664d0 0x7f5904068990 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f591c007fd0 tx=0x7f591c009040 comp rx=0 tx=0).stop 2023-10-30T12:32:22.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.591+0000 7f5927c17640 1 -- 172.21.15.125:0/3906293737 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59200c6b80 msgr2=0x7f5920153360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:22.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.591+0000 7f5927c17640 1 --2- 172.21.15.125:0/3906293737 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59200c6b80 0x7f5920153360 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f591000c4b0 tx=0x7f591000c970 comp rx=0 tx=0).stop 2023-10-30T12:32:22.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.591+0000 7f5927c17640 1 -- 172.21.15.125:0/3906293737 shutdown_connections 2023-10-30T12:32:22.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.591+0000 7f5927c17640 1 --2- 172.21.15.125:0/3906293737 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59040664d0 0x7f5904068990 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:22.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.591+0000 7f5927c17640 1 --2- 172.21.15.125:0/3906293737 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59200c6f60 0x7f59201538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:22.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.591+0000 7f5927c17640 1 --2- 172.21.15.125:0/3906293737 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59200c6b80 0x7f5920153360 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:22.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.591+0000 7f5927c17640 1 --2- 172.21.15.125:0/3906293737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59200c05d0 0x7f59200c9e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:22.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.591+0000 7f5927c17640 1 -- 172.21.15.125:0/3906293737 >> 172.21.15.125:0/3906293737 conn(0x7f59200c7910 msgr2=0x7f59200c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:22.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.591+0000 7f5927c17640 1 -- 172.21.15.125:0/3906293737 shutdown_connections 2023-10-30T12:32:22.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.591+0000 7f5927c17640 1 -- 172.21.15.125:0/3906293737 wait complete. 2023-10-30T12:32:22.615 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=2) 2023-10-30T12:32:22.615 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T12:32:22.615 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:32:22.631 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T12:32:22.632 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:22.721 INFO:tasks.workunit.client.0.smithi125.stdout:./219/.snap/snap-subdir-test 2023-10-30T12:32:22.728 INFO:tasks.workunit.client.0.smithi125.stdout:./198/.snap/snap-subdir-test 2023-10-30T12:32:22.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.807+0000 7f5148091640 1 Processor -- start 2023-10-30T12:32:22.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.807+0000 7f5148091640 1 -- start start 2023-10-30T12:32:22.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.807+0000 7f5148091640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f51401580e0 0x7f5140070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:22.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.807+0000 7f5148091640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f514015e700 0x7f514006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:22.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.807+0000 7f5148091640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51400c5810 con 0x7f51400c6540 2023-10-30T12:32:22.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.807+0000 7f5148091640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51400c6990 con 0x7f51401580e0 2023-10-30T12:32:22.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.807+0000 7f5148091640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514015eb90 con 0x7f514015e700 2023-10-30T12:32:22.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.807+0000 7f5145e2d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f51401580e0 0x7f5140070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:22.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.807+0000 7f514562c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f51400c6540 0x7f5140069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37546/0 (socket says 172.21.15.125:37546) 2023-10-30T12:32:22.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.807+0000 7f514562c640 1 -- 172.21.15.125:0/3641089577 learned_addr learned my addr 172.21.15.125:0/3641089577 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:32:22.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.807+0000 7f514662e640 1 --2- 172.21.15.125:0/3641089577 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f514015e700 0x7f514006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:22.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5145e2d640 1 -- 172.21.15.125:0/3641089577 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f514015e700 msgr2=0x7f514006a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:22.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5145e2d640 1 --2- 172.21.15.125:0/3641089577 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f514015e700 0x7f514006a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:22.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5145e2d640 1 -- 172.21.15.125:0/3641089577 >> v1:172.21.15.125:6789/0 conn(0x7f51400c6540 legacy=0x7f5140069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:22.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5145e2d640 1 -- 172.21.15.125:0/3641089577 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51400c6bb0 con 0x7f51401580e0 2023-10-30T12:32:22.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f514662e640 1 --2- 172.21.15.125:0/3641089577 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f514015e700 0x7f514006a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:32:22.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5144e2b640 1 -- 172.21.15.125:0/3641089577 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055744836 0 0) 0x7f51400c5810 con 0x7f51400c6540 2023-10-30T12:32:22.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5145e2d640 1 --2- 172.21.15.125:0/3641089577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51401580e0 0x7f5140070740 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f513c0097c0 tx=0x7f513c01c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96fb12521f15b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:22.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5144e2b640 1 -- 172.21.15.125:0/3641089577 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c0ab650 con 0x7f51401580e0 2023-10-30T12:32:22.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5144e2b640 1 -- 172.21.15.125:0/3641089577 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513c0ab7b0 con 0x7f51401580e0 2023-10-30T12:32:22.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5144e2b640 1 -- 172.21.15.125:0/3641089577 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c0abb40 con 0x7f51401580e0 2023-10-30T12:32:22.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5148091640 1 -- 172.21.15.125:0/3641089577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51401580e0 msgr2=0x7f5140070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:22.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5148091640 1 --2- 172.21.15.125:0/3641089577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51401580e0 0x7f5140070740 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f513c0097c0 tx=0x7f513c01c450 comp rx=0 tx=0).stop 2023-10-30T12:32:22.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5148091640 1 -- 172.21.15.125:0/3641089577 shutdown_connections 2023-10-30T12:32:22.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5148091640 1 --2- 172.21.15.125:0/3641089577 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f514015e700 0x7f514006a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:22.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5148091640 1 --2- 172.21.15.125:0/3641089577 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51401580e0 0x7f5140070740 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:22.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.811+0000 7f5148091640 1 -- 172.21.15.125:0/3641089577 >> 172.21.15.125:0/3641089577 conn(0x7f51400c7910 msgr2=0x7f51400c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:22.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5148091640 1 -- 172.21.15.125:0/3641089577 shutdown_connections 2023-10-30T12:32:22.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5148091640 1 -- 172.21.15.125:0/3641089577 wait complete. 2023-10-30T12:32:22.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5148091640 1 Processor -- start 2023-10-30T12:32:22.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5148091640 1 -- start start 2023-10-30T12:32:22.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5148091640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51400c6540 0x7f514014e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:22.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5148091640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51401580e0 0x7f51400c14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:22.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5148091640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f514015e700 0x7f51401f98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:22.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5148091640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51400c99e0 con 0x7f51400c6540 2023-10-30T12:32:22.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5148091640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51400c9e70 con 0x7f514015e700 2023-10-30T12:32:22.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5148091640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51400ca610 con 0x7f51401580e0 2023-10-30T12:32:22.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5145e2d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51400c6540 0x7f514014e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:22.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5145e2d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51400c6540 0x7f514014e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57488/0 (socket says 172.21.15.125:57488) 2023-10-30T12:32:22.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5145e2d640 1 -- 172.21.15.125:0/92340361 learned_addr learned my addr 172.21.15.125:0/92340361 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:22.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f514562c640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51401580e0 0x7f51400c14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:22.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f514662e640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f514015e700 0x7f51401f98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:22.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5145e2d640 1 -- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51401580e0 msgr2=0x7f51400c14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:22.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5145e2d640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51401580e0 0x7f51400c14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:22.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5145e2d640 1 -- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f514015e700 msgr2=0x7f51401f98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:22.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5145e2d640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f514015e700 0x7f51401f98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:22.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5145e2d640 1 -- 172.21.15.125:0/92340361 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c009420 con 0x7f51400c6540 2023-10-30T12:32:22.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f514662e640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f514015e700 0x7f51401f98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:32:22.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.815+0000 7f5145e2d640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51400c6540 0x7f514014e180 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f513c0ad040 tx=0x7f513c01ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:22.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.819+0000 7f51367fc640 1 -- 172.21.15.125:0/92340361 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c0b5810 con 0x7f51400c6540 2023-10-30T12:32:22.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.819+0000 7f51367fc640 1 -- 172.21.15.125:0/92340361 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f513c0b5990 con 0x7f51400c6540 2023-10-30T12:32:22.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.819+0000 7f5148091640 1 -- 172.21.15.125:0/92340361 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51400ca8f0 con 0x7f51400c6540 2023-10-30T12:32:22.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.819+0000 7f51367fc640 1 -- 172.21.15.125:0/92340361 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513c0b5d40 con 0x7f51400c6540 2023-10-30T12:32:22.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.819+0000 7f5148091640 1 -- 172.21.15.125:0/92340361 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51400c9390 con 0x7f51400c6540 2023-10-30T12:32:22.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.819+0000 7f5148091640 1 -- 172.21.15.125:0/92340361 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514004e170 con 0x7f51400c6540 2023-10-30T12:32:22.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.819+0000 7f51367fc640 1 -- 172.21.15.125:0/92340361 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f513c0bf410 con 0x7f51400c6540 2023-10-30T12:32:22.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.819+0000 7f51367fc640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51240664d0 0x7f5124068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:22.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.819+0000 7f51367fc640 1 -- 172.21.15.125:0/92340361 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5852+0+0 (secure 0 0 0) 0x7f513c11ee30 con 0x7f51400c6540 2023-10-30T12:32:22.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.819+0000 7f514562c640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51240664d0 0x7f5124068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:22.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.823+0000 7f514562c640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51240664d0 0x7f5124068990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5140072450 tx=0x7f5130007b80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:22.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:22.823+0000 7f51367fc640 1 -- 172.21.15.125:0/92340361 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f513c0ec330 con 0x7f51400c6540 2023-10-30T12:32:23.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:23.003+0000 7f5148091640 1 -- 172.21.15.125:0/92340361 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f514015b5c0 con 0x7f51400c6540 2023-10-30T12:32:23.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:23.007+0000 7f51367fc640 1 -- 172.21.15.125:0/92340361 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6546 (secure 0 0 0) 0x7f513c0ef950 con 0x7f51400c6540 2023-10-30T12:32:23.009 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:23.009 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:02.105923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:23.010 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 22 2023-10-30T12:32:23.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:23.007+0000 7f5148091640 1 -- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51240664d0 msgr2=0x7f5124068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:23.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:23.007+0000 7f5148091640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51240664d0 0x7f5124068990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5140072450 tx=0x7f5130007b80 comp rx=0 tx=0).stop 2023-10-30T12:32:23.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:23.007+0000 7f5148091640 1 -- 172.21.15.125:0/92340361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51400c6540 msgr2=0x7f514014e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:23.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:23.007+0000 7f5148091640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51400c6540 0x7f514014e180 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f513c0ad040 tx=0x7f513c01ac90 comp rx=0 tx=0).stop 2023-10-30T12:32:23.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:23.007+0000 7f5148091640 1 -- 172.21.15.125:0/92340361 shutdown_connections 2023-10-30T12:32:23.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:23.007+0000 7f5148091640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51240664d0 0x7f5124068990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:23.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:23.007+0000 7f5148091640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f514015e700 0x7f51401f98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:23.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:23.007+0000 7f5148091640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51401580e0 0x7f51400c14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:23.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:23.007+0000 7f5148091640 1 --2- 172.21.15.125:0/92340361 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51400c6540 0x7f514014e180 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:23.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:23.007+0000 7f5148091640 1 -- 172.21.15.125:0/92340361 >> 172.21.15.125:0/92340361 conn(0x7f51400c7910 msgr2=0x7f5140153300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:23.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:23.007+0000 7f5148091640 1 -- 172.21.15.125:0/92340361 shutdown_connections 2023-10-30T12:32:23.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:23.007+0000 7f5148091640 1 -- 172.21.15.125:0/92340361 wait complete. 2023-10-30T12:32:23.030 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:32:25.031 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:25.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.175+0000 7f9c347f5640 1 Processor -- start 2023-10-30T12:32:25.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.175+0000 7f9c347f5640 1 -- start start 2023-10-30T12:32:25.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.175+0000 7f9c347f5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c2c0c6b80 0x7f9c2c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:25.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.175+0000 7f9c347f5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9c2c0c6f60 0x7f9c2c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:25.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.175+0000 7f9c347f5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c2c0c0a50 con 0x7f9c2c0c8430 2023-10-30T12:32:25.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.175+0000 7f9c347f5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2c06f6e0 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.175+0000 7f9c347f5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c06f9c0 con 0x7f9c2c0c6b80 2023-10-30T12:32:25.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.179+0000 7f9c31d90640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9c2c0c6f60 0x7f9c2c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:25.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.179+0000 7f9c31d90640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9c2c0c6f60 0x7f9c2c0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57492/0 (socket says 172.21.15.125:57492) 2023-10-30T12:32:25.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.179+0000 7f9c31d90640 1 -- 172.21.15.125:0/4233809215 learned_addr learned my addr 172.21.15.125:0/4233809215 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:25.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.179+0000 7f9c32591640 1 --2- 172.21.15.125:0/4233809215 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c2c0c6b80 0x7f9c2c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:25.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.179+0000 7f9c31d90640 1 -- 172.21.15.125:0/4233809215 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c2c0c6b80 msgr2=0x7f9c2c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:25.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.179+0000 7f9c31d90640 1 --2- 172.21.15.125:0/4233809215 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c2c0c6b80 0x7f9c2c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:25.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.179+0000 7f9c31d90640 1 -- 172.21.15.125:0/4233809215 >> v1:172.21.15.136:6789/0 conn(0x7f9c2c0c8430 legacy=0x7f9c2c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:25.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.179+0000 7f9c31d90640 1 -- 172.21.15.125:0/4233809215 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c06fca0 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.179+0000 7f9c31d90640 1 --2- 172.21.15.125:0/4233809215 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c2c0c6f60 0x7f9c2c0c0400 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c009870 tx=0x7f9c1c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b52b34b27a775a1f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:25.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c3158f640 1 -- 172.21.15.125:0/4233809215 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c0ab650 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c3158f640 1 -- 172.21.15.125:0/4233809215 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c0ab7b0 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c3158f640 1 -- 172.21.15.125:0/4233809215 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c0abb40 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 -- 172.21.15.125:0/4233809215 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c2c0c6f60 msgr2=0x7f9c2c0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:25.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 --2- 172.21.15.125:0/4233809215 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c2c0c6f60 0x7f9c2c0c0400 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c009870 tx=0x7f9c1c01c540 comp rx=0 tx=0).stop 2023-10-30T12:32:25.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 -- 172.21.15.125:0/4233809215 shutdown_connections 2023-10-30T12:32:25.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 --2- 172.21.15.125:0/4233809215 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c2c0c6f60 0x7f9c2c0c0400 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:25.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 --2- 172.21.15.125:0/4233809215 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c2c0c6b80 0x7f9c2c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:25.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 -- 172.21.15.125:0/4233809215 >> 172.21.15.125:0/4233809215 conn(0x7f9c2c0c7910 msgr2=0x7f9c2c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:25.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 -- 172.21.15.125:0/4233809215 shutdown_connections 2023-10-30T12:32:25.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 -- 172.21.15.125:0/4233809215 wait complete. 2023-10-30T12:32:25.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 Processor -- start 2023-10-30T12:32:25.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 -- start start 2023-10-30T12:32:25.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c2c0c6b80 0x7f9c2c154e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:25.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c2c0c6f60 0x7f9c2c150080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:25.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c2c0c8430 0x7f9c2c1505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:25.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c0ca350 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c13e850 con 0x7f9c2c0c6b80 2023-10-30T12:32:25.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c347f5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c158590 con 0x7f9c2c0c8430 2023-10-30T12:32:25.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c31d90640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c2c0c6f60 0x7f9c2c150080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:25.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c31d90640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c2c0c6f60 0x7f9c2c150080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57498/0 (socket says 172.21.15.125:57498) 2023-10-30T12:32:25.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c31d90640 1 -- 172.21.15.125:0/1456629302 learned_addr learned my addr 172.21.15.125:0/1456629302 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:25.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c32591640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c2c0c6b80 0x7f9c2c154e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:25.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.183+0000 7f9c32d92640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c2c0c8430 0x7f9c2c1505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:25.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c31d90640 1 -- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c2c0c8430 msgr2=0x7f9c2c1505c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:25.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c31d90640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c2c0c8430 0x7f9c2c1505c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:25.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c31d90640 1 -- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c2c0c6b80 msgr2=0x7f9c2c154e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:25.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c31d90640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c2c0c6b80 0x7f9c2c154e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:25.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c31d90640 1 -- 172.21.15.125:0/1456629302 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c1c0094d0 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c32d92640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c2c0c8430 0x7f9c2c1505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:32:25.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c31d90640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c2c0c6f60 0x7f9c2c150080 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c0ad040 tx=0x7f9c1c01ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:25.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c22ffd640 1 -- 172.21.15.125:0/1456629302 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c0b4800 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c22ffd640 1 -- 172.21.15.125:0/1456629302 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1c0b4960 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c347f5640 1 -- 172.21.15.125:0/1456629302 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2c159080 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c22ffd640 1 -- 172.21.15.125:0/1456629302 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1c0b4cf0 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c347f5640 1 -- 172.21.15.125:0/1456629302 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2c158ac0 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c347f5640 1 -- 172.21.15.125:0/1456629302 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2c04e170 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c22ffd640 1 -- 172.21.15.125:0/1456629302 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9c1c0aa430 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.187+0000 7f9c22ffd640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c100664a0 0x7f9c10068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:25.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.191+0000 7f9c22ffd640 1 -- 172.21.15.125:0/1456629302 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5852+0+0 (secure 0 0 0) 0x7f9c1c11ecb0 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.191+0000 7f9c32591640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c100664a0 0x7f9c10068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:25.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.191+0000 7f9c32591640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c100664a0 0x7f9c10068960 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9c28009b60 tx=0x7f9c28009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:25.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.195+0000 7f9c22ffd640 1 -- 172.21.15.125:0/1456629302 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9c1c0ec140 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.375+0000 7f9c347f5640 1 -- 172.21.15.125:0/1456629302 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c2c06a130 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.375+0000 7f9c22ffd640 1 -- 172.21.15.125:0/1456629302 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6546 (secure 0 0 0) 0x7f9c1c0c4090 con 0x7f9c2c0c6f60 2023-10-30T12:32:25.379 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:25.379 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:02.105923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:25.380 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 22 2023-10-30T12:32:25.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.375+0000 7f9c347f5640 1 -- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c100664a0 msgr2=0x7f9c10068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:25.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.375+0000 7f9c347f5640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c100664a0 0x7f9c10068960 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9c28009b60 tx=0x7f9c28009190 comp rx=0 tx=0).stop 2023-10-30T12:32:25.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.375+0000 7f9c347f5640 1 -- 172.21.15.125:0/1456629302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c2c0c6f60 msgr2=0x7f9c2c150080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:25.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.375+0000 7f9c347f5640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c2c0c6f60 0x7f9c2c150080 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c0ad040 tx=0x7f9c1c01ad80 comp rx=0 tx=0).stop 2023-10-30T12:32:25.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.379+0000 7f9c347f5640 1 -- 172.21.15.125:0/1456629302 shutdown_connections 2023-10-30T12:32:25.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.379+0000 7f9c347f5640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c100664a0 0x7f9c10068960 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:25.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.379+0000 7f9c347f5640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c2c0c8430 0x7f9c2c1505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:25.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.379+0000 7f9c347f5640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c2c0c6f60 0x7f9c2c150080 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:25.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.379+0000 7f9c347f5640 1 --2- 172.21.15.125:0/1456629302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c2c0c6b80 0x7f9c2c154e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:25.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.379+0000 7f9c347f5640 1 -- 172.21.15.125:0/1456629302 >> 172.21.15.125:0/1456629302 conn(0x7f9c2c0c7910 msgr2=0x7f9c2c13eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:25.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.379+0000 7f9c347f5640 1 -- 172.21.15.125:0/1456629302 shutdown_connections 2023-10-30T12:32:25.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:25.379+0000 7f9c347f5640 1 -- 172.21.15.125:0/1456629302 wait complete. 2023-10-30T12:32:25.399 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:32:27.400 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:27.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.543+0000 7fa86226b640 1 Processor -- start 2023-10-30T12:32:27.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.543+0000 7fa86226b640 1 -- start start 2023-10-30T12:32:27.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.543+0000 7fa86226b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa85c070a00 0x7fa85c070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:27.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.543+0000 7fa86226b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa85c069e90 0x7fa85c06a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:27.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.543+0000 7fa86226b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85c0713b0 con 0x7fa85c1539a0 2023-10-30T12:32:27.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.543+0000 7fa86226b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa85c06fbb0 con 0x7fa85c070a00 2023-10-30T12:32:27.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.543+0000 7fa86226b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa85c153d80 con 0x7fa85c069e90 2023-10-30T12:32:27.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.547+0000 7fa85b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa85c070a00 0x7fa85c070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:27.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.547+0000 7fa85b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa85c070a00 0x7fa85c070e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57504/0 (socket says 172.21.15.125:57504) 2023-10-30T12:32:27.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.547+0000 7fa85b7fe640 1 -- 172.21.15.125:0/2363678725 learned_addr learned my addr 172.21.15.125:0/2363678725 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:27.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.547+0000 7fa85bfff640 1 --1- 172.21.15.125:0/2363678725 >> v1:172.21.15.125:6789/0 conn(0x7fa85c1539a0 0x7fa85c15a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37550/0 (socket says 172.21.15.125:37550) 2023-10-30T12:32:27.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.547+0000 7fa860808640 1 --2- 172.21.15.125:0/2363678725 >> v2:172.21.15.136:3300/0 conn(0x7fa85c069e90 0x7fa85c06a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:27.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.547+0000 7fa85b7fe640 1 -- 172.21.15.125:0/2363678725 >> v2:172.21.15.136:3300/0 conn(0x7fa85c069e90 msgr2=0x7fa85c06a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:27.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.547+0000 7fa85b7fe640 1 --2- 172.21.15.125:0/2363678725 >> v2:172.21.15.136:3300/0 conn(0x7fa85c069e90 0x7fa85c06a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:27.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.547+0000 7fa85b7fe640 1 -- 172.21.15.125:0/2363678725 >> v1:172.21.15.125:6789/0 conn(0x7fa85c1539a0 legacy=0x7fa85c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:27.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.547+0000 7fa85b7fe640 1 -- 172.21.15.125:0/2363678725 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c0c5c50 con 0x7fa85c070a00 2023-10-30T12:32:27.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.547+0000 7fa85b7fe640 1 --2- 172.21.15.125:0/2363678725 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa85c070a00 0x7fa85c070e70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0b1080 tx=0x7fa85001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0435b5c0b71ac8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:27.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.547+0000 7fa85affd640 1 -- 172.21.15.125:0/2363678725 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa850004230 con 0x7fa85c070a00 2023-10-30T12:32:27.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.547+0000 7fa85affd640 1 -- 172.21.15.125:0/2363678725 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8500a7050 con 0x7fa85c070a00 2023-10-30T12:32:27.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.547+0000 7fa85affd640 1 -- 172.21.15.125:0/2363678725 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa850003620 con 0x7fa85c070a00 2023-10-30T12:32:27.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 -- 172.21.15.125:0/2363678725 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa85c070a00 msgr2=0x7fa85c070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:27.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 --2- 172.21.15.125:0/2363678725 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa85c070a00 0x7fa85c070e70 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa85c0b1080 tx=0x7fa85001c440 comp rx=0 tx=0).stop 2023-10-30T12:32:27.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 -- 172.21.15.125:0/2363678725 shutdown_connections 2023-10-30T12:32:27.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 --2- 172.21.15.125:0/2363678725 >> v2:172.21.15.136:3300/0 conn(0x7fa85c069e90 0x7fa85c06a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:27.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 --2- 172.21.15.125:0/2363678725 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa85c070a00 0x7fa85c070e70 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:27.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 -- 172.21.15.125:0/2363678725 >> 172.21.15.125:0/2363678725 conn(0x7fa85c0c7910 msgr2=0x7fa85c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:27.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 -- 172.21.15.125:0/2363678725 shutdown_connections 2023-10-30T12:32:27.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 -- 172.21.15.125:0/2363678725 wait complete. 2023-10-30T12:32:27.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 Processor -- start 2023-10-30T12:32:27.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 -- start start 2023-10-30T12:32:27.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa85c069e90 0x7fa85c143de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:27.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa85c070a00 0x7fa85c144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:27.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa85c1539a0 0x7fa85c1f6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:27.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c0c9550 con 0x7fa85c070a00 2023-10-30T12:32:27.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c0c8cb0 con 0x7fa85c1539a0 2023-10-30T12:32:27.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa86226b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c0c8f90 con 0x7fa85c069e90 2023-10-30T12:32:27.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa85b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa85c070a00 0x7fa85c144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:27.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa85b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa85c070a00 0x7fa85c144320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57508/0 (socket says 172.21.15.125:57508) 2023-10-30T12:32:27.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa85b7fe640 1 -- 172.21.15.125:0/76318161 learned_addr learned my addr 172.21.15.125:0/76318161 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:27.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa860808640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa85c1539a0 0x7fa85c1f6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:27.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa85bfff640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa85c069e90 0x7fa85c143de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:27.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa85b7fe640 1 -- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa85c069e90 msgr2=0x7fa85c143de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:27.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa85b7fe640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa85c069e90 0x7fa85c143de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:27.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa85b7fe640 1 -- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa85c1539a0 msgr2=0x7fa85c1f6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:27.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa85b7fe640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa85c1539a0 0x7fa85c1f6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:27.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa85b7fe640 1 -- 172.21.15.125:0/76318161 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8500094d0 con 0x7fa85c070a00 2023-10-30T12:32:27.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa85bfff640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa85c069e90 0x7fa85c143de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:32:27.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.551+0000 7fa860808640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa85c1539a0 0x7fa85c1f6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:32:27.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.555+0000 7fa85b7fe640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa85c070a00 0x7fa85c144320 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa850019050 tx=0x7fa8500191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:27.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.555+0000 7fa858ff9640 1 -- 172.21.15.125:0/76318161 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa850003c70 con 0x7fa85c070a00 2023-10-30T12:32:27.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.555+0000 7fa858ff9640 1 -- 172.21.15.125:0/76318161 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa850003df0 con 0x7fa85c070a00 2023-10-30T12:32:27.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.555+0000 7fa86226b640 1 -- 172.21.15.125:0/76318161 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85c0c9270 con 0x7fa85c070a00 2023-10-30T12:32:27.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.555+0000 7fa858ff9640 1 -- 172.21.15.125:0/76318161 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8500ab680 con 0x7fa85c070a00 2023-10-30T12:32:27.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.555+0000 7fa86226b640 1 -- 172.21.15.125:0/76318161 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85c143430 con 0x7fa85c070a00 2023-10-30T12:32:27.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.555+0000 7fa86226b640 1 -- 172.21.15.125:0/76318161 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85c0c8780 con 0x7fa85c070a00 2023-10-30T12:32:27.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.555+0000 7fa858ff9640 1 -- 172.21.15.125:0/76318161 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa8500b2040 con 0x7fa85c070a00 2023-10-30T12:32:27.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.555+0000 7fa858ff9640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa8480664a0 0x7fa848068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:27.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.555+0000 7fa858ff9640 1 -- 172.21.15.125:0/76318161 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5852+0+0 (secure 0 0 0) 0x7fa85011e3e0 con 0x7fa85c070a00 2023-10-30T12:32:27.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.559+0000 7fa85bfff640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa8480664a0 0x7fa848068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:27.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.559+0000 7fa85bfff640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa8480664a0 0x7fa848068960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa84c000f70 tx=0x7fa84c007d50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:27.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.563+0000 7fa858ff9640 1 -- 172.21.15.125:0/76318161 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa8500eb730 con 0x7fa85c070a00 2023-10-30T12:32:27.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.743+0000 7fa86226b640 1 -- 172.21.15.125:0/76318161 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa85c04e170 con 0x7fa85c070a00 2023-10-30T12:32:27.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.743+0000 7fa858ff9640 1 -- 172.21.15.125:0/76318161 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6546 (secure 0 0 0) 0x7fa8500eed50 con 0x7fa85c070a00 2023-10-30T12:32:27.746 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:27.746 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:02.105923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:27.747 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 22 2023-10-30T12:32:27.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.743+0000 7fa86226b640 1 -- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa8480664a0 msgr2=0x7fa848068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:27.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.743+0000 7fa86226b640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa8480664a0 0x7fa848068960 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa84c000f70 tx=0x7fa84c007d50 comp rx=0 tx=0).stop 2023-10-30T12:32:27.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.743+0000 7fa86226b640 1 -- 172.21.15.125:0/76318161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa85c070a00 msgr2=0x7fa85c144320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:27.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.743+0000 7fa86226b640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa85c070a00 0x7fa85c144320 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa850019050 tx=0x7fa8500191e0 comp rx=0 tx=0).stop 2023-10-30T12:32:27.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.743+0000 7fa86226b640 1 -- 172.21.15.125:0/76318161 shutdown_connections 2023-10-30T12:32:27.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.743+0000 7fa86226b640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa8480664a0 0x7fa848068960 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:27.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.743+0000 7fa86226b640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa85c1539a0 0x7fa85c1f6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:27.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.743+0000 7fa86226b640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa85c070a00 0x7fa85c144320 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:27.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.743+0000 7fa86226b640 1 --2- 172.21.15.125:0/76318161 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa85c069e90 0x7fa85c143de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:27.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.743+0000 7fa86226b640 1 -- 172.21.15.125:0/76318161 >> 172.21.15.125:0/76318161 conn(0x7fa85c0c7910 msgr2=0x7fa85c151c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:27.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.743+0000 7fa86226b640 1 -- 172.21.15.125:0/76318161 shutdown_connections 2023-10-30T12:32:27.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:27.743+0000 7fa86226b640 1 -- 172.21.15.125:0/76318161 wait complete. 2023-10-30T12:32:27.767 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:32:29.768 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:29.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04d0068640 1 Processor -- start 2023-10-30T12:32:29.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04d0068640 1 -- start start 2023-10-30T12:32:29.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04d0068640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c81538a0 0x7f04c815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04d0068640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f04c80ca0e0 0x7f04c815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04d0068640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f04c80cf160 0x7f04c80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04d0068640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c80cb800 con 0x7f04c80ca0e0 2023-10-30T12:32:29.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04d0068640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04c815dc30 con 0x7f04c80cf160 2023-10-30T12:32:29.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04d0068640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c80cb330 con 0x7f04c81538a0 2023-10-30T12:32:29.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04cd603640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f04c80ca0e0 0x7f04c815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:29.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04ce605640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f04c80cf160 0x7f04c80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:29.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04ce605640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f04c80cf160 0x7f04c80cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:48850/0 (socket says 172.21.15.125:48850) 2023-10-30T12:32:29.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04cd603640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f04c80ca0e0 0x7f04c815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57510/0 (socket says 172.21.15.125:57510) 2023-10-30T12:32:29.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04ce605640 1 -- 172.21.15.125:0/3895984341 learned_addr learned my addr 172.21.15.125:0/3895984341 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:29.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04cde04640 1 --2- 172.21.15.125:0/3895984341 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c81538a0 0x7f04c815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:29.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04cd603640 1 -- 172.21.15.125:0/3895984341 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c81538a0 msgr2=0x7f04c815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:29.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04cd603640 1 --2- 172.21.15.125:0/3895984341 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c81538a0 0x7f04c815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04cd603640 1 -- 172.21.15.125:0/3895984341 >> v2:172.21.15.136:3300/0 conn(0x7f04c80cf160 msgr2=0x7f04c80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:29.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04cd603640 1 --2- 172.21.15.125:0/3895984341 >> v2:172.21.15.136:3300/0 conn(0x7f04c80cf160 0x7f04c80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.911+0000 7f04cd603640 1 -- 172.21.15.125:0/3895984341 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c80cad70 con 0x7f04c80ca0e0 2023-10-30T12:32:29.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.915+0000 7f04cd603640 1 --2- 172.21.15.125:0/3895984341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c80ca0e0 0x7f04c815a090 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f04c806cb00 tx=0x7f04b801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6336eeb64a09d09f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:29.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.915+0000 7f04cce02640 1 -- 172.21.15.125:0/3895984341 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b80a8910 con 0x7f04c80ca0e0 2023-10-30T12:32:29.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.915+0000 7f04cce02640 1 -- 172.21.15.125:0/3895984341 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b80a8a70 con 0x7f04c80ca0e0 2023-10-30T12:32:29.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.915+0000 7f04cce02640 1 -- 172.21.15.125:0/3895984341 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b80a8e00 con 0x7f04c80ca0e0 2023-10-30T12:32:29.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.915+0000 7f04d0068640 1 -- 172.21.15.125:0/3895984341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c80ca0e0 msgr2=0x7f04c815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:29.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.915+0000 7f04d0068640 1 --2- 172.21.15.125:0/3895984341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c80ca0e0 0x7f04c815a090 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f04c806cb00 tx=0x7f04b801ec50 comp rx=0 tx=0).stop 2023-10-30T12:32:29.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.915+0000 7f04d0068640 1 -- 172.21.15.125:0/3895984341 shutdown_connections 2023-10-30T12:32:29.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.915+0000 7f04d0068640 1 --2- 172.21.15.125:0/3895984341 >> v2:172.21.15.136:3300/0 conn(0x7f04c80cf160 0x7f04c80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.915+0000 7f04d0068640 1 --2- 172.21.15.125:0/3895984341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c80ca0e0 0x7f04c815a090 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.915+0000 7f04d0068640 1 --2- 172.21.15.125:0/3895984341 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c81538a0 0x7f04c815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.915+0000 7f04d0068640 1 -- 172.21.15.125:0/3895984341 >> 172.21.15.125:0/3895984341 conn(0x7f04c806bfe0 msgr2=0x7f04c806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:29.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.915+0000 7f04d0068640 1 -- 172.21.15.125:0/3895984341 shutdown_connections 2023-10-30T12:32:29.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.915+0000 7f04d0068640 1 -- 172.21.15.125:0/3895984341 wait complete. 2023-10-30T12:32:29.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04d0068640 1 Processor -- start 2023-10-30T12:32:29.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04d0068640 1 -- start start 2023-10-30T12:32:29.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04d0068640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c80ca0e0 0x7f04c81efac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04d0068640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c80cf160 0x7f04c80ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04d0068640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04c81538a0 0x7f04c81e5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04d0068640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c8142670 con 0x7f04c80cf160 2023-10-30T12:32:29.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04d0068640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c815bf20 con 0x7f04c81538a0 2023-10-30T12:32:29.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04d0068640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c81f0000 con 0x7f04c80ca0e0 2023-10-30T12:32:29.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04cd603640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c80cf160 0x7f04c80ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:29.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04cd603640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c80cf160 0x7f04c80ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57516/0 (socket says 172.21.15.125:57516) 2023-10-30T12:32:29.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04cd603640 1 -- 172.21.15.125:0/436511752 learned_addr learned my addr 172.21.15.125:0/436511752 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:29.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04ce605640 1 --2- 172.21.15.125:0/436511752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04c81538a0 0x7f04c81e5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:29.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04cde04640 1 --2- 172.21.15.125:0/436511752 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c80ca0e0 0x7f04c81efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:29.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04cd603640 1 -- 172.21.15.125:0/436511752 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c80ca0e0 msgr2=0x7f04c81efac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:29.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04cd603640 1 --2- 172.21.15.125:0/436511752 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c80ca0e0 0x7f04c81efac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04cd603640 1 -- 172.21.15.125:0/436511752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04c81538a0 msgr2=0x7f04c81e5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:29.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04cd603640 1 --2- 172.21.15.125:0/436511752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04c81538a0 0x7f04c81e5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:29.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04cd603640 1 -- 172.21.15.125:0/436511752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b8009420 con 0x7f04c80cf160 2023-10-30T12:32:29.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04cd603640 1 --2- 172.21.15.125:0/436511752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c80cf160 0x7f04c80ce260 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f04b80b3040 tx=0x7f04b801b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:29.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04be7fc640 1 -- 172.21.15.125:0/436511752 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b80b0850 con 0x7f04c80cf160 2023-10-30T12:32:29.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04be7fc640 1 -- 172.21.15.125:0/436511752 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b80b09b0 con 0x7f04c80cf160 2023-10-30T12:32:29.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04d0068640 1 -- 172.21.15.125:0/436511752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c81e6450 con 0x7f04c80cf160 2023-10-30T12:32:29.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04be7fc640 1 -- 172.21.15.125:0/436511752 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b80b0d60 con 0x7f04c80cf160 2023-10-30T12:32:29.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.919+0000 7f04d0068640 1 -- 172.21.15.125:0/436511752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c81ef070 con 0x7f04c80cf160 2023-10-30T12:32:29.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.923+0000 7f04d0068640 1 -- 172.21.15.125:0/436511752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c80b2a50 con 0x7f04c80cf160 2023-10-30T12:32:29.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.923+0000 7f04be7fc640 1 -- 172.21.15.125:0/436511752 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f04b80b7040 con 0x7f04c80cf160 2023-10-30T12:32:29.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.923+0000 7f04be7fc640 1 --2- 172.21.15.125:0/436511752 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04ac0664a0 0x7f04ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:29.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.923+0000 7f04be7fc640 1 -- 172.21.15.125:0/436511752 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5852+0+0 (secure 0 0 0) 0x7f04b81217f0 con 0x7f04c80cf160 2023-10-30T12:32:29.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.923+0000 7f04cde04640 1 --2- 172.21.15.125:0/436511752 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04ac0664a0 0x7f04ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:29.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.923+0000 7f04cde04640 1 --2- 172.21.15.125:0/436511752 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04ac0664a0 0x7f04ac068960 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f04c400b930 tx=0x7f04c4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:29.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:29.927+0000 7f04be7fc640 1 -- 172.21.15.125:0/436511752 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f04b80eeab0 con 0x7f04c80cf160 2023-10-30T12:32:30.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:30.115+0000 7f04d0068640 1 -- 172.21.15.125:0/436511752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04c81e3c30 con 0x7f04c80cf160 2023-10-30T12:32:30.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:30.119+0000 7f04be7fc640 1 -- 172.21.15.125:0/436511752 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6546 (secure 0 0 0) 0x7f04b80b5090 con 0x7f04c80cf160 2023-10-30T12:32:30.121 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:30.121 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:02.105923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:30.122 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 22 2023-10-30T12:32:30.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:30.119+0000 7f04d0068640 1 -- 172.21.15.125:0/436511752 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04ac0664a0 msgr2=0x7f04ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:30.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:30.119+0000 7f04d0068640 1 --2- 172.21.15.125:0/436511752 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04ac0664a0 0x7f04ac068960 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f04c400b930 tx=0x7f04c4008040 comp rx=0 tx=0).stop 2023-10-30T12:32:30.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:30.119+0000 7f04d0068640 1 -- 172.21.15.125:0/436511752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c80cf160 msgr2=0x7f04c80ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:30.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:30.119+0000 7f04d0068640 1 --2- 172.21.15.125:0/436511752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c80cf160 0x7f04c80ce260 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f04b80b3040 tx=0x7f04b801b9f0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:30.119+0000 7f04d0068640 1 -- 172.21.15.125:0/436511752 shutdown_connections 2023-10-30T12:32:30.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:30.119+0000 7f04d0068640 1 --2- 172.21.15.125:0/436511752 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04ac0664a0 0x7f04ac068960 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:30.119+0000 7f04d0068640 1 --2- 172.21.15.125:0/436511752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04c81538a0 0x7f04c81e5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:30.119+0000 7f04d0068640 1 --2- 172.21.15.125:0/436511752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c80cf160 0x7f04c80ce260 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:30.119+0000 7f04d0068640 1 --2- 172.21.15.125:0/436511752 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c80ca0e0 0x7f04c81efac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:30.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:30.119+0000 7f04d0068640 1 -- 172.21.15.125:0/436511752 >> 172.21.15.125:0/436511752 conn(0x7f04c806bfe0 msgr2=0x7f04c81525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:30.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:30.119+0000 7f04d0068640 1 -- 172.21.15.125:0/436511752 shutdown_connections 2023-10-30T12:32:30.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:30.119+0000 7f04d0068640 1 -- 172.21.15.125:0/436511752 wait complete. 2023-10-30T12:32:30.142 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:32:32.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:32.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff068046640 1 Processor -- start 2023-10-30T12:32:32.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff068046640 1 -- start start 2023-10-30T12:32:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff068046640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0601538a0 0x7ff06015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff068046640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff0600ca0e0 0x7ff06015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff068046640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff0600cf160 0x7ff0600cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff068046640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0600cb800 con 0x7ff0600cf160 2023-10-30T12:32:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff068046640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06015dc30 con 0x7ff0600ca0e0 2023-10-30T12:32:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff068046640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0600cb330 con 0x7ff0601538a0 2023-10-30T12:32:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff0665e3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff0600cf160 0x7ff0600cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff0665e3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff0600cf160 0x7ff0600cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50136/0 (socket says 172.21.15.125:50136) 2023-10-30T12:32:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff0665e3640 1 -- 172.21.15.125:0/4257133514 learned_addr learned my addr 172.21.15.125:0/4257133514 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff065de2640 1 --2- 172.21.15.125:0/4257133514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0601538a0 0x7ff06015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff0655e1640 1 --2- 172.21.15.125:0/4257133514 >> v2:172.21.15.136:3300/0 conn(0x7ff0600ca0e0 0x7ff06015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff0665e3640 1 -- 172.21.15.125:0/4257133514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0601538a0 msgr2=0x7ff06015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff0665e3640 1 --2- 172.21.15.125:0/4257133514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0601538a0 0x7ff06015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff0665e3640 1 -- 172.21.15.125:0/4257133514 >> v2:172.21.15.136:3300/0 conn(0x7ff0600ca0e0 msgr2=0x7ff06015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:32.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff0665e3640 1 --2- 172.21.15.125:0/4257133514 >> v2:172.21.15.136:3300/0 conn(0x7ff0600ca0e0 0x7ff06015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff0665e3640 1 -- 172.21.15.125:0/4257133514 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0600cad70 con 0x7ff0600cf160 2023-10-30T12:32:32.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.287+0000 7ff065de2640 1 --2- 172.21.15.125:0/4257133514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0601538a0 0x7ff06015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:32:32.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.291+0000 7ff0665e3640 1 --2- 172.21.15.125:0/4257133514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0600cf160 0x7ff0600cf590 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff0600ccd10 tx=0x7ff05801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63986486457f4f23 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:32.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.291+0000 7ff064de0640 1 -- 172.21.15.125:0/4257133514 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0580af400 con 0x7ff0600cf160 2023-10-30T12:32:32.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.291+0000 7ff064de0640 1 -- 172.21.15.125:0/4257133514 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0580ab050 con 0x7ff0600cf160 2023-10-30T12:32:32.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.291+0000 7ff064de0640 1 -- 172.21.15.125:0/4257133514 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0580af780 con 0x7ff0600cf160 2023-10-30T12:32:32.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.291+0000 7ff068046640 1 -- 172.21.15.125:0/4257133514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0600cf160 msgr2=0x7ff0600cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:32.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.291+0000 7ff068046640 1 --2- 172.21.15.125:0/4257133514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0600cf160 0x7ff0600cf590 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7ff0600ccd10 tx=0x7ff05801f1c0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.291+0000 7ff068046640 1 -- 172.21.15.125:0/4257133514 shutdown_connections 2023-10-30T12:32:32.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.291+0000 7ff068046640 1 --2- 172.21.15.125:0/4257133514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0600cf160 0x7ff0600cf590 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.291+0000 7ff068046640 1 --2- 172.21.15.125:0/4257133514 >> v2:172.21.15.136:3300/0 conn(0x7ff0600ca0e0 0x7ff06015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.291+0000 7ff068046640 1 --2- 172.21.15.125:0/4257133514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0601538a0 0x7ff06015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.291+0000 7ff068046640 1 -- 172.21.15.125:0/4257133514 >> 172.21.15.125:0/4257133514 conn(0x7ff06006bfe0 msgr2=0x7ff06006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:32.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.291+0000 7ff068046640 1 -- 172.21.15.125:0/4257133514 shutdown_connections 2023-10-30T12:32:32.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.291+0000 7ff068046640 1 -- 172.21.15.125:0/4257133514 wait complete. 2023-10-30T12:32:32.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff068046640 1 Processor -- start 2023-10-30T12:32:32.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff068046640 1 -- start start 2023-10-30T12:32:32.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff068046640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0600ca0e0 0x7ff0601efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:32.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff068046640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0600cf160 0x7ff0601e5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:32.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff068046640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0601538a0 0x7ff0601e2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:32.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff068046640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06015afa0 con 0x7ff0600cf160 2023-10-30T12:32:32.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff068046640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06015a9e0 con 0x7ff0601538a0 2023-10-30T12:32:32.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff068046640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff06015acc0 con 0x7ff0600ca0e0 2023-10-30T12:32:32.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff0655e1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0600cf160 0x7ff0601e5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:32.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff0655e1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0600cf160 0x7ff0601e5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50146/0 (socket says 172.21.15.125:50146) 2023-10-30T12:32:32.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff0655e1640 1 -- 172.21.15.125:0/2446336414 learned_addr learned my addr 172.21.15.125:0/2446336414 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:32.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff0665e3640 1 --2- 172.21.15.125:0/2446336414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0601538a0 0x7ff0601e2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:32.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff065de2640 1 --2- 172.21.15.125:0/2446336414 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0600ca0e0 0x7ff0601efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:32.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff0655e1640 1 -- 172.21.15.125:0/2446336414 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0600ca0e0 msgr2=0x7ff0601efb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:32.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff0655e1640 1 --2- 172.21.15.125:0/2446336414 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0600ca0e0 0x7ff0601efb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff0655e1640 1 -- 172.21.15.125:0/2446336414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0601538a0 msgr2=0x7ff0601e2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:32.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff0655e1640 1 --2- 172.21.15.125:0/2446336414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0601538a0 0x7ff0601e2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff0655e1640 1 -- 172.21.15.125:0/2446336414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff058009b30 con 0x7ff0600cf160 2023-10-30T12:32:32.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff0655e1640 1 --2- 172.21.15.125:0/2446336414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0600cf160 0x7ff0601e5620 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff05000d800 tx=0x7ff05000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:32.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff0567fc640 1 -- 172.21.15.125:0/2446336414 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0500096a0 con 0x7ff0600cf160 2023-10-30T12:32:32.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff0567fc640 1 -- 172.21.15.125:0/2446336414 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff050009800 con 0x7ff0600cf160 2023-10-30T12:32:32.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff068046640 1 -- 172.21.15.125:0/2446336414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0601ef080 con 0x7ff0600cf160 2023-10-30T12:32:32.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff0567fc640 1 -- 172.21.15.125:0/2446336414 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff050009b70 con 0x7ff0600cf160 2023-10-30T12:32:32.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff068046640 1 -- 172.21.15.125:0/2446336414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0601ef640 con 0x7ff0600cf160 2023-10-30T12:32:32.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.295+0000 7ff068046640 1 -- 172.21.15.125:0/2446336414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0600b2a50 con 0x7ff0600cf160 2023-10-30T12:32:32.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.299+0000 7ff0567fc640 1 -- 172.21.15.125:0/2446336414 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff050009cd0 con 0x7ff0600cf160 2023-10-30T12:32:32.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.299+0000 7ff0567fc640 1 --2- 172.21.15.125:0/2446336414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0440664a0 0x7ff044068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:32.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.299+0000 7ff0567fc640 1 -- 172.21.15.125:0/2446336414 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5852+0+0 (secure 0 0 0) 0x7ff0500845e0 con 0x7ff0600cf160 2023-10-30T12:32:32.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.299+0000 7ff065de2640 1 --2- 172.21.15.125:0/2446336414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0440664a0 0x7ff044068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:32.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.299+0000 7ff065de2640 1 --2- 172.21.15.125:0/2446336414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0440664a0 0x7ff044068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff05c00ba10 tx=0x7ff05c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:32.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.303+0000 7ff0567fc640 1 -- 172.21.15.125:0/2446336414 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff050051a60 con 0x7ff0600cf160 2023-10-30T12:32:32.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.483+0000 7ff068046640 1 -- 172.21.15.125:0/2446336414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff060159830 con 0x7ff0600cf160 2023-10-30T12:32:32.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.487+0000 7ff0567fc640 1 -- 172.21.15.125:0/2446336414 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6546 (secure 0 0 0) 0x7ff050055080 con 0x7ff0600cf160 2023-10-30T12:32:32.489 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:32.489 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:02.105923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:32.489 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 22 2023-10-30T12:32:32.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.487+0000 7ff068046640 1 -- 172.21.15.125:0/2446336414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0440664a0 msgr2=0x7ff044068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:32.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.487+0000 7ff068046640 1 --2- 172.21.15.125:0/2446336414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0440664a0 0x7ff044068960 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff05c00ba10 tx=0x7ff05c008040 comp rx=0 tx=0).stop 2023-10-30T12:32:32.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.487+0000 7ff068046640 1 -- 172.21.15.125:0/2446336414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0600cf160 msgr2=0x7ff0601e5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:32.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.487+0000 7ff068046640 1 --2- 172.21.15.125:0/2446336414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0600cf160 0x7ff0601e5620 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff05000d800 tx=0x7ff05000dcc0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.487+0000 7ff068046640 1 -- 172.21.15.125:0/2446336414 shutdown_connections 2023-10-30T12:32:32.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.487+0000 7ff068046640 1 --2- 172.21.15.125:0/2446336414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0440664a0 0x7ff044068960 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.487+0000 7ff068046640 1 --2- 172.21.15.125:0/2446336414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0601538a0 0x7ff0601e2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.487+0000 7ff068046640 1 --2- 172.21.15.125:0/2446336414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0600cf160 0x7ff0601e5620 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.487+0000 7ff068046640 1 --2- 172.21.15.125:0/2446336414 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0600ca0e0 0x7ff0601efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:32.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.487+0000 7ff068046640 1 -- 172.21.15.125:0/2446336414 >> 172.21.15.125:0/2446336414 conn(0x7ff06006bfe0 msgr2=0x7ff0601526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:32.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.487+0000 7ff068046640 1 -- 172.21.15.125:0/2446336414 shutdown_connections 2023-10-30T12:32:32.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:32.487+0000 7ff068046640 1 -- 172.21.15.125:0/2446336414 wait complete. 2023-10-30T12:32:32.509 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:32:34.510 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:34.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.651+0000 7f1b065b5640 1 Processor -- start 2023-10-30T12:32:34.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b065b5640 1 -- start start 2023-10-30T12:32:34.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b065b5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1b0006ab20 0x7f1b000c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b065b5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b00069e90 0x7f1b0006a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b065b5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b000c54d0 con 0x7f1b00153890 2023-10-30T12:32:34.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b065b5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b000c1cd0 con 0x7f1b0006ab20 2023-10-30T12:32:34.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b065b5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b000c5a90 con 0x7f1b00069e90 2023-10-30T12:32:34.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b05db4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b00069e90 0x7f1b0006a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b04db2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1b0006ab20 0x7f1b000c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b05db4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b00069e90 0x7f1b0006a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:41996/0 (socket says 172.21.15.125:41996) 2023-10-30T12:32:34.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b05db4640 1 -- 172.21.15.125:0/2433131403 learned_addr learned my addr 172.21.15.125:0/2433131403 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:34.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b055b3640 1 --1- 172.21.15.125:0/2433131403 >> v1:172.21.15.136:6789/0 conn(0x7f1b00153890 0x7f1b00159630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:50686/0 (socket says 172.21.15.125:50686) 2023-10-30T12:32:34.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b04db2640 1 -- 172.21.15.125:0/2433131403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b00069e90 msgr2=0x7f1b0006a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:34.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b04db2640 1 --2- 172.21.15.125:0/2433131403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b00069e90 0x7f1b0006a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b04db2640 1 -- 172.21.15.125:0/2433131403 >> v1:172.21.15.136:6789/0 conn(0x7f1b00153890 legacy=0x7f1b00159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b04db2640 1 -- 172.21.15.125:0/2433131403 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b00159de0 con 0x7f1b0006ab20 2023-10-30T12:32:34.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.655+0000 7f1b05db4640 1 --2- 172.21.15.125:0/2433131403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b00069e90 0x7f1b0006a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:32:34.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.659+0000 7f1b04db2640 1 --2- 172.21.15.125:0/2433131403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b0006ab20 0x7f1b000c5f20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1af00097c0 tx=0x7f1af001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b85e5f4b960df8a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:34.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.659+0000 7f1af7fff640 1 -- 172.21.15.125:0/2433131403 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af00ab650 con 0x7f1b0006ab20 2023-10-30T12:32:34.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.659+0000 7f1af7fff640 1 -- 172.21.15.125:0/2433131403 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af00ab7b0 con 0x7f1b0006ab20 2023-10-30T12:32:34.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.659+0000 7f1af7fff640 1 -- 172.21.15.125:0/2433131403 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af00abb40 con 0x7f1b0006ab20 2023-10-30T12:32:34.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.659+0000 7f1b065b5640 1 -- 172.21.15.125:0/2433131403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b0006ab20 msgr2=0x7f1b000c5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.659+0000 7f1b065b5640 1 --2- 172.21.15.125:0/2433131403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b0006ab20 0x7f1b000c5f20 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1af00097c0 tx=0x7f1af001c450 comp rx=0 tx=0).stop 2023-10-30T12:32:34.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.659+0000 7f1b065b5640 1 -- 172.21.15.125:0/2433131403 shutdown_connections 2023-10-30T12:32:34.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.659+0000 7f1b065b5640 1 --2- 172.21.15.125:0/2433131403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b00069e90 0x7f1b0006a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.659+0000 7f1b065b5640 1 --2- 172.21.15.125:0/2433131403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b0006ab20 0x7f1b000c5f20 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.659+0000 7f1b065b5640 1 -- 172.21.15.125:0/2433131403 >> 172.21.15.125:0/2433131403 conn(0x7f1b000c75d0 msgr2=0x7f1b000c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:34.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.659+0000 7f1b065b5640 1 -- 172.21.15.125:0/2433131403 shutdown_connections 2023-10-30T12:32:34.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.659+0000 7f1b065b5640 1 -- 172.21.15.125:0/2433131403 wait complete. 2023-10-30T12:32:34.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b065b5640 1 Processor -- start 2023-10-30T12:32:34.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b065b5640 1 -- start start 2023-10-30T12:32:34.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b065b5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b00069e90 0x7f1b00150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b065b5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0006ab20 0x7f1b001f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b065b5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b00153890 0x7f1b001f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b065b5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b001f1d40 con 0x7f1b0006ab20 2023-10-30T12:32:34.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b065b5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b001f5fb0 con 0x7f1b00153890 2023-10-30T12:32:34.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b065b5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b001f5430 con 0x7f1b00069e90 2023-10-30T12:32:34.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b04db2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0006ab20 0x7f1b001f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b04db2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0006ab20 0x7f1b001f21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50150/0 (socket says 172.21.15.125:50150) 2023-10-30T12:32:34.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b04db2640 1 -- 172.21.15.125:0/231358566 learned_addr learned my addr 172.21.15.125:0/231358566 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:34.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b055b3640 1 --2- 172.21.15.125:0/231358566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b00069e90 0x7f1b00150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b05db4640 1 --2- 172.21.15.125:0/231358566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b00153890 0x7f1b001f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b04db2640 1 -- 172.21.15.125:0/231358566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b00069e90 msgr2=0x7f1b00150480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b04db2640 1 --2- 172.21.15.125:0/231358566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b00069e90 0x7f1b00150480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b04db2640 1 -- 172.21.15.125:0/231358566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b00153890 msgr2=0x7f1b001f2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b04db2640 1 --2- 172.21.15.125:0/231358566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b00153890 0x7f1b001f2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b04db2640 1 -- 172.21.15.125:0/231358566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af8009b30 con 0x7f1b0006ab20 2023-10-30T12:32:34.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b04db2640 1 --2- 172.21.15.125:0/231358566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0006ab20 0x7f1b001f21d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1af00ad040 tx=0x7f1af001ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:34.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1af5ffb640 1 -- 172.21.15.125:0/231358566 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af00bf610 con 0x7f1b0006ab20 2023-10-30T12:32:34.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1af5ffb640 1 -- 172.21.15.125:0/231358566 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af00bf770 con 0x7f1b0006ab20 2023-10-30T12:32:34.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1af5ffb640 1 -- 172.21.15.125:0/231358566 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af00bfb20 con 0x7f1b0006ab20 2023-10-30T12:32:34.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b065b5640 1 -- 172.21.15.125:0/231358566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af0009420 con 0x7f1b0006ab20 2023-10-30T12:32:34.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b065b5640 1 -- 172.21.15.125:0/231358566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b001f59f0 con 0x7f1b0006ab20 2023-10-30T12:32:34.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.663+0000 7f1b065b5640 1 -- 172.21.15.125:0/231358566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b001f1eb0 con 0x7f1b0006ab20 2023-10-30T12:32:34.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.667+0000 7f1af5ffb640 1 -- 172.21.15.125:0/231358566 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1af00b5400 con 0x7f1b0006ab20 2023-10-30T12:32:34.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.667+0000 7f1af5ffb640 1 --2- 172.21.15.125:0/231358566 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ae40664d0 0x7f1ae4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:34.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.667+0000 7f1af5ffb640 1 -- 172.21.15.125:0/231358566 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5852+0+0 (secure 0 0 0) 0x7f1af011eb00 con 0x7f1b0006ab20 2023-10-30T12:32:34.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.667+0000 7f1b055b3640 1 --2- 172.21.15.125:0/231358566 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ae40664d0 0x7f1ae4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:34.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.667+0000 7f1b055b3640 1 --2- 172.21.15.125:0/231358566 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ae40664d0 0x7f1ae4068990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1afc000fc0 tx=0x7f1afc000f30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:34.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.671+0000 7f1af5ffb640 1 -- 172.21.15.125:0/231358566 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1af0120e60 con 0x7f1b0006ab20 2023-10-30T12:32:34.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.975+0000 7f1b065b5640 1 -- 172.21.15.125:0/231358566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b0004e170 con 0x7f1b0006ab20 2023-10-30T12:32:34.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.975+0000 7f1af5ffb640 1 -- 172.21.15.125:0/231358566 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+6546 (secure 0 0 0) 0x7f1af00ec020 con 0x7f1b0006ab20 2023-10-30T12:32:34.981 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:34.981 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4957,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18}],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:02.105923+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":25,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4265},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4265":{"gid":4265,"name":"c","rank":2,"incarnation":13,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2291877604","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2291877604},{"type":"v1","addr":"172.21.15.125:6837","nonce":2291877604}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:34.981 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 22 2023-10-30T12:32:34.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.979+0000 7f1b065b5640 1 -- 172.21.15.125:0/231358566 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ae40664d0 msgr2=0x7f1ae4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.979+0000 7f1b065b5640 1 --2- 172.21.15.125:0/231358566 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ae40664d0 0x7f1ae4068990 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1afc000fc0 tx=0x7f1afc000f30 comp rx=0 tx=0).stop 2023-10-30T12:32:34.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.979+0000 7f1b065b5640 1 -- 172.21.15.125:0/231358566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0006ab20 msgr2=0x7f1b001f21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:34.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.979+0000 7f1b065b5640 1 --2- 172.21.15.125:0/231358566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0006ab20 0x7f1b001f21d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1af00ad040 tx=0x7f1af001ac90 comp rx=0 tx=0).stop 2023-10-30T12:32:34.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.979+0000 7f1b065b5640 1 -- 172.21.15.125:0/231358566 shutdown_connections 2023-10-30T12:32:34.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.979+0000 7f1b065b5640 1 --2- 172.21.15.125:0/231358566 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ae40664d0 0x7f1ae4068990 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.979+0000 7f1b065b5640 1 --2- 172.21.15.125:0/231358566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b00153890 0x7f1b001f2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.979+0000 7f1b065b5640 1 --2- 172.21.15.125:0/231358566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b0006ab20 0x7f1b001f21d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.979+0000 7f1b065b5640 1 --2- 172.21.15.125:0/231358566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b00069e90 0x7f1b00150480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:34.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.979+0000 7f1b065b5640 1 -- 172.21.15.125:0/231358566 >> 172.21.15.125:0/231358566 conn(0x7f1b000c75d0 msgr2=0x7f1b0006f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:34.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.979+0000 7f1b065b5640 1 -- 172.21.15.125:0/231358566 shutdown_connections 2023-10-30T12:32:34.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:34.979+0000 7f1b065b5640 1 -- 172.21.15.125:0/231358566 wait complete. 2023-10-30T12:32:35.004 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:32:37.004 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:37.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.151+0000 7f8320bd3640 1 Processor -- start 2023-10-30T12:32:37.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.151+0000 7f8320bd3640 1 -- start start 2023-10-30T12:32:37.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f8320bd3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f831c0c05d0 0x7f831c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f8320bd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f831c0c6f60 0x7f831c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f8320bd3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f831c0c0a50 con 0x7f831c0c6b80 2023-10-30T12:32:37.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f8320bd3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f831c06f6e0 con 0x7f831c0c05d0 2023-10-30T12:32:37.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f8320bd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831c06f9c0 con 0x7f831c0c6f60 2023-10-30T12:32:37.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f831a59c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f831c0c6b80 0x7f831c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41054/0 (socket says 172.21.15.125:41054) 2023-10-30T12:32:37.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f831a59c640 1 -- 172.21.15.125:0/3246327082 learned_addr learned my addr 172.21.15.125:0/3246327082 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:32:37.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f8311d9b640 1 --2- 172.21.15.125:0/3246327082 >> v2:172.21.15.125:3300/0 conn(0x7f831c0c05d0 0x7f831c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f831ad9d640 1 --2- 172.21.15.125:0/3246327082 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f831c0c6f60 0x7f831c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f8311d9b640 1 -- 172.21.15.125:0/3246327082 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f831c0c6f60 msgr2=0x7f831c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:37.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f8311d9b640 1 --2- 172.21.15.125:0/3246327082 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f831c0c6f60 0x7f831c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f8311d9b640 1 -- 172.21.15.125:0/3246327082 >> v1:172.21.15.125:6789/0 conn(0x7f831c0c6b80 legacy=0x7f831c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f8311d9b640 1 -- 172.21.15.125:0/3246327082 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831c06fca0 con 0x7f831c0c05d0 2023-10-30T12:32:37.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f8319d9b640 1 -- 172.21.15.125:0/3246327082 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780956760 0 0) 0x7f831c0c0a50 con 0x7f831c0c6b80 2023-10-30T12:32:37.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.155+0000 7f8311d9b640 1 --2- 172.21.15.125:0/3246327082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f831c0c05d0 0x7f831c13ed90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f83040097c0 tx=0x7f830401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9503d78b8a6b1538 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8319d9b640 1 -- 172.21.15.125:0/3246327082 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83040ad640 con 0x7f831c0c05d0 2023-10-30T12:32:37.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8319d9b640 1 -- 172.21.15.125:0/3246327082 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83040ad7a0 con 0x7f831c0c05d0 2023-10-30T12:32:37.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8319d9b640 1 -- 172.21.15.125:0/3246327082 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83040adb30 con 0x7f831c0c05d0 2023-10-30T12:32:37.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 -- 172.21.15.125:0/3246327082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f831c0c05d0 msgr2=0x7f831c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 --2- 172.21.15.125:0/3246327082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f831c0c05d0 0x7f831c13ed90 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f83040097c0 tx=0x7f830401c480 comp rx=0 tx=0).stop 2023-10-30T12:32:37.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 -- 172.21.15.125:0/3246327082 shutdown_connections 2023-10-30T12:32:37.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 --2- 172.21.15.125:0/3246327082 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f831c0c05d0 0x7f831c13ed90 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 --2- 172.21.15.125:0/3246327082 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f831c0c6f60 0x7f831c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 -- 172.21.15.125:0/3246327082 >> 172.21.15.125:0/3246327082 conn(0x7f831c0c7910 msgr2=0x7f831c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:37.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 -- 172.21.15.125:0/3246327082 shutdown_connections 2023-10-30T12:32:37.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 -- 172.21.15.125:0/3246327082 wait complete. 2023-10-30T12:32:37.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 Processor -- start 2023-10-30T12:32:37.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 -- start start 2023-10-30T12:32:37.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f831c0c05d0 0x7f831c151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f831c0c6b80 0x7f831c152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f831c0c6f60 0x7f831c154aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c0c9620 con 0x7f831c0c6f60 2023-10-30T12:32:37.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c154350 con 0x7f831c0c6b80 2023-10-30T12:32:37.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.159+0000 7f8320bd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831c0c9850 con 0x7f831c0c05d0 2023-10-30T12:32:37.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f831ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f831c0c6f60 0x7f831c154aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f831ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f831c0c6f60 0x7f831c154aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50178/0 (socket says 172.21.15.125:50178) 2023-10-30T12:32:37.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f831ad9d640 1 -- 172.21.15.125:0/2465126683 learned_addr learned my addr 172.21.15.125:0/2465126683 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:37.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f8311d9b640 1 --2- 172.21.15.125:0/2465126683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f831c0c6b80 0x7f831c152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f831a59c640 1 --2- 172.21.15.125:0/2465126683 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f831c0c05d0 0x7f831c151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f831ad9d640 1 -- 172.21.15.125:0/2465126683 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f831c0c05d0 msgr2=0x7f831c151300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f831ad9d640 1 --2- 172.21.15.125:0/2465126683 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f831c0c05d0 0x7f831c151300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f831ad9d640 1 -- 172.21.15.125:0/2465126683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f831c0c6b80 msgr2=0x7f831c152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f831ad9d640 1 --2- 172.21.15.125:0/2465126683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f831c0c6b80 0x7f831c152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f831ad9d640 1 -- 172.21.15.125:0/2465126683 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8304009420 con 0x7f831c0c6f60 2023-10-30T12:32:37.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f831ad9d640 1 --2- 172.21.15.125:0/2465126683 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f831c0c6f60 0x7f831c154aa0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8314007e60 tx=0x7f831400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f83137fe640 1 -- 172.21.15.125:0/2465126683 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314014420 con 0x7f831c0c6f60 2023-10-30T12:32:37.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f83137fe640 1 -- 172.21.15.125:0/2465126683 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83140145a0 con 0x7f831c0c6f60 2023-10-30T12:32:37.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f83137fe640 1 -- 172.21.15.125:0/2465126683 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8314014950 con 0x7f831c0c6f60 2023-10-30T12:32:37.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f8320bd3640 1 -- 172.21.15.125:0/2465126683 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f831c1371f0 con 0x7f831c0c6f60 2023-10-30T12:32:37.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f8320bd3640 1 -- 172.21.15.125:0/2465126683 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f831c154580 con 0x7f831c0c6f60 2023-10-30T12:32:37.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f8320bd3640 1 -- 172.21.15.125:0/2465126683 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f831c154770 con 0x7f831c0c6f60 2023-10-30T12:32:37.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f83137fe640 1 -- 172.21.15.125:0/2465126683 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8314014ab0 con 0x7f831c0c6f60 2023-10-30T12:32:37.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f83137fe640 1 --2- 172.21.15.125:0/2465126683 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82fc0664d0 0x7f82fc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.163+0000 7f83137fe640 1 -- 172.21.15.125:0/2465126683 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f83140848a0 con 0x7f831c0c6f60 2023-10-30T12:32:37.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.167+0000 7f831a59c640 1 --2- 172.21.15.125:0/2465126683 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82fc0664d0 0x7f82fc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.167+0000 7f831a59c640 1 --2- 172.21.15.125:0/2465126683 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82fc0664d0 0x7f82fc068990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f830c000f70 tx=0x7f830c007b60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.167+0000 7f83137fe640 1 -- 172.21.15.125:0/2465126683 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8314051c40 con 0x7f831c0c6f60 2023-10-30T12:32:37.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.351+0000 7f8320bd3640 1 -- 172.21.15.125:0/2465126683 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f831c04e170 con 0x7f831c0c6f60 2023-10-30T12:32:37.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.351+0000 7f83137fe640 1 -- 172.21.15.125:0/2465126683 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5748 (secure 0 0 0) 0x7f8314055260 con 0x7f831c0c6f60 2023-10-30T12:32:37.355 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:37.355 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:34.980698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:37.356 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 23 2023-10-30T12:32:37.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.351+0000 7f8320bd3640 1 -- 172.21.15.125:0/2465126683 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82fc0664d0 msgr2=0x7f82fc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.355+0000 7f8320bd3640 1 --2- 172.21.15.125:0/2465126683 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82fc0664d0 0x7f82fc068990 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f830c000f70 tx=0x7f830c007b60 comp rx=0 tx=0).stop 2023-10-30T12:32:37.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.355+0000 7f8320bd3640 1 -- 172.21.15.125:0/2465126683 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f831c0c6f60 msgr2=0x7f831c154aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.355+0000 7f8320bd3640 1 --2- 172.21.15.125:0/2465126683 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f831c0c6f60 0x7f831c154aa0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8314007e60 tx=0x7f831400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.355+0000 7f8320bd3640 1 -- 172.21.15.125:0/2465126683 shutdown_connections 2023-10-30T12:32:37.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.355+0000 7f8320bd3640 1 --2- 172.21.15.125:0/2465126683 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82fc0664d0 0x7f82fc068990 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.355+0000 7f8320bd3640 1 --2- 172.21.15.125:0/2465126683 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f831c0c6f60 0x7f831c154aa0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.355+0000 7f8320bd3640 1 --2- 172.21.15.125:0/2465126683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f831c0c6b80 0x7f831c152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.355+0000 7f8320bd3640 1 --2- 172.21.15.125:0/2465126683 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f831c0c05d0 0x7f831c151300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.355+0000 7f8320bd3640 1 -- 172.21.15.125:0/2465126683 >> 172.21.15.125:0/2465126683 conn(0x7f831c0c7910 msgr2=0x7f831c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:37.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.355+0000 7f8320bd3640 1 -- 172.21.15.125:0/2465126683 shutdown_connections 2023-10-30T12:32:37.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.355+0000 7f8320bd3640 1 -- 172.21.15.125:0/2465126683 wait complete. 2023-10-30T12:32:37.375 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T12:32:37.376 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:32:37.376 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:37.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f86822ea640 1 Processor -- start 2023-10-30T12:32:37.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f86822ea640 1 -- start start 2023-10-30T12:32:37.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f86822ea640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f867c0c6bd0 0x7f867c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f86822ea640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867c066cf0 con 0x7f867c06f7a0 2023-10-30T12:32:37.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f86822ea640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f867c0c70d0 con 0x7f867c071110 2023-10-30T12:32:37.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f86822ea640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f867c0700c0 con 0x7f867c0c6bd0 2023-10-30T12:32:37.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f867bfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f867c06f7a0 0x7f867c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41068/0 (socket says 172.21.15.125:41068) 2023-10-30T12:32:37.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f867bfff640 1 -- 172.21.15.125:0/495039800 learned_addr learned my addr 172.21.15.125:0/495039800 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:32:37.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f867b7fe640 1 --2- 172.21.15.125:0/495039800 >> v2:172.21.15.136:3300/0 conn(0x7f867c0c6bd0 0x7f867c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f867affd640 1 -- 172.21.15.125:0/495039800 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431222203 0 0) 0x7f867c066cf0 con 0x7f867c06f7a0 2023-10-30T12:32:37.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f867affd640 1 -- 172.21.15.125:0/495039800 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8664003530 con 0x7f867c06f7a0 2023-10-30T12:32:37.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f867affd640 1 -- 172.21.15.125:0/495039800 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3877030594 0 0) 0x7f8664003530 con 0x7f867c06f7a0 2023-10-30T12:32:37.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f867affd640 1 -- 172.21.15.125:0/495039800 >> v2:172.21.15.136:3300/0 conn(0x7f867c0c6bd0 msgr2=0x7f867c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:37.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f867affd640 1 --2- 172.21.15.125:0/495039800 >> v2:172.21.15.136:3300/0 conn(0x7f867c0c6bd0 0x7f867c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f867affd640 1 -- 172.21.15.125:0/495039800 >> v1:172.21.15.136:6789/0 conn(0x7f867c071110 legacy=0x7f867c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f867affd640 1 -- 172.21.15.125:0/495039800 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c0c7390 con 0x7f867c06f7a0 2023-10-30T12:32:37.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f867affd640 1 -- 172.21.15.125:0/495039800 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f866c001680 con 0x7f867c06f7a0 2023-10-30T12:32:37.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.511+0000 7f867affd640 1 -- 172.21.15.125:0/495039800 >> v1:172.21.15.125:6789/0 conn(0x7f867c06f7a0 legacy=0x7f867c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f867affd640 1 --2- 172.21.15.125:0/495039800 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8664003980 0x7f8664003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f867affd640 1 --2- 172.21.15.125:0/495039800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86640047d0 0x7f8664004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f867affd640 1 --2- 172.21.15.125:0/495039800 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8664005190 0x7f866400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f867affd640 1 -- 172.21.15.125:0/495039800 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c0700c0 con 0x7f8664005190 2023-10-30T12:32:37.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f867affd640 1 -- 172.21.15.125:0/495039800 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866400c040 con 0x7f86640047d0 2023-10-30T12:32:37.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f867affd640 1 -- 172.21.15.125:0/495039800 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866400c200 con 0x7f8664003980 2023-10-30T12:32:37.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f8680887640 1 --2- 172.21.15.125:0/495039800 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8664005190 0x7f866400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f867bfff640 1 --2- 172.21.15.125:0/495039800 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8664003980 0x7f8664003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f867b7fe640 1 --2- 172.21.15.125:0/495039800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86640047d0 0x7f8664004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f8680887640 1 -- 172.21.15.125:0/495039800 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8664003980 msgr2=0x7f8664003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:37.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f8680887640 1 --2- 172.21.15.125:0/495039800 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8664003980 0x7f8664003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f8680887640 1 -- 172.21.15.125:0/495039800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86640047d0 msgr2=0x7f8664004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:37.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f8680887640 1 --2- 172.21.15.125:0/495039800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86640047d0 0x7f8664004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f8680887640 1 -- 172.21.15.125:0/495039800 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8664003530 con 0x7f8664005190 2023-10-30T12:32:37.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f8680887640 1 -- 172.21.15.125:0/495039800 reap_dead start 2023-10-30T12:32:37.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f8680887640 1 --2- 172.21.15.125:0/495039800 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8664005190 0x7f866400b990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8674001e70 tx=0x7f867401c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7221c5b0462200e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f867affd640 1 -- 172.21.15.125:0/495039800 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674003b80 con 0x7f8664005190 2023-10-30T12:32:37.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f867affd640 1 -- 172.21.15.125:0/495039800 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8674003d20 con 0x7f8664005190 2023-10-30T12:32:37.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.515+0000 7f867affd640 1 -- 172.21.15.125:0/495039800 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674005260 con 0x7f8664005190 2023-10-30T12:32:37.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 -- 172.21.15.125:0/495039800 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8664005190 msgr2=0x7f866400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 --2- 172.21.15.125:0/495039800 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8664005190 0x7f866400b990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f8674001e70 tx=0x7f867401c760 comp rx=0 tx=0).stop 2023-10-30T12:32:37.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 -- 172.21.15.125:0/495039800 shutdown_connections 2023-10-30T12:32:37.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 --2- 172.21.15.125:0/495039800 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8664005190 0x7f866400b990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 -- 172.21.15.125:0/495039800 >> 172.21.15.125:0/495039800 conn(0x7f867c0c7910 msgr2=0x7f867c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:37.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 -- 172.21.15.125:0/495039800 shutdown_connections 2023-10-30T12:32:37.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 -- 172.21.15.125:0/495039800 wait complete. 2023-10-30T12:32:37.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 Processor -- start 2023-10-30T12:32:37.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 -- start start 2023-10-30T12:32:37.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8664005190 0x7f867c1ec6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f867c14bcc0 0x7f867c148d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f867c144df0 0x7f867c145270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c14a8b0 con 0x7f867c14bcc0 2023-10-30T12:32:37.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f867c14c100 con 0x7f867c144df0 2023-10-30T12:32:37.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f86822ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f867c155110 con 0x7f8664005190 2023-10-30T12:32:37.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f867b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f867c14bcc0 0x7f867c148d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f867b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f867c14bcc0 0x7f867c148d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50202/0 (socket says 172.21.15.125:50202) 2023-10-30T12:32:37.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f867b7fe640 1 -- 172.21.15.125:0/3914346208 learned_addr learned my addr 172.21.15.125:0/3914346208 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:37.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f867bfff640 1 --2- 172.21.15.125:0/3914346208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8664005190 0x7f867c1ec6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f8680887640 1 --2- 172.21.15.125:0/3914346208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f867c144df0 0x7f867c145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f867b7fe640 1 -- 172.21.15.125:0/3914346208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8664005190 msgr2=0x7f867c1ec6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f867b7fe640 1 --2- 172.21.15.125:0/3914346208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8664005190 0x7f867c1ec6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f867b7fe640 1 -- 172.21.15.125:0/3914346208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f867c144df0 msgr2=0x7f867c145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f867b7fe640 1 --2- 172.21.15.125:0/3914346208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f867c144df0 0x7f867c145270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.519+0000 7f867b7fe640 1 -- 172.21.15.125:0/3914346208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f867c1ecd10 con 0x7f867c14bcc0 2023-10-30T12:32:37.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.523+0000 7f867b7fe640 1 --2- 172.21.15.125:0/3914346208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f867c14bcc0 0x7f867c148d80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f86700118b0 tx=0x7f8670011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.523+0000 7f8678ff9640 1 -- 172.21.15.125:0/3914346208 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867000fa30 con 0x7f867c14bcc0 2023-10-30T12:32:37.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.523+0000 7f8678ff9640 1 -- 172.21.15.125:0/3914346208 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867000fb90 con 0x7f867c14bcc0 2023-10-30T12:32:37.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.523+0000 7f8678ff9640 1 -- 172.21.15.125:0/3914346208 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8670014440 con 0x7f867c14bcc0 2023-10-30T12:32:37.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.523+0000 7f86822ea640 1 -- 172.21.15.125:0/3914346208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867c1ecff0 con 0x7f867c14bcc0 2023-10-30T12:32:37.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.523+0000 7f86822ea640 1 -- 172.21.15.125:0/3914346208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f867c1445f0 con 0x7f867c14bcc0 2023-10-30T12:32:37.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.523+0000 7f86822ea640 1 -- 172.21.15.125:0/3914346208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f867c04e170 con 0x7f867c14bcc0 2023-10-30T12:32:37.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.523+0000 7f8678ff9640 1 -- 172.21.15.125:0/3914346208 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f86700145a0 con 0x7f867c14bcc0 2023-10-30T12:32:37.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.523+0000 7f8678ff9640 1 --2- 172.21.15.125:0/3914346208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f86680664d0 0x7f8668068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:37.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.523+0000 7f8678ff9640 1 -- 172.21.15.125:0/3914346208 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f86700848b0 con 0x7f867c14bcc0 2023-10-30T12:32:37.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.523+0000 7f867bfff640 1 --2- 172.21.15.125:0/3914346208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f86680664d0 0x7f8668068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:37.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.527+0000 7f867bfff640 1 --2- 172.21.15.125:0/3914346208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f86680664d0 0x7f8668068990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f866c001060 tx=0x7f866c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:37.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.527+0000 7f8678ff9640 1 -- 172.21.15.125:0/3914346208 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8670051b30 con 0x7f867c14bcc0 2023-10-30T12:32:37.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.707+0000 7f86822ea640 1 -- 172.21.15.125:0/3914346208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f867c0721c0 con 0x7f867c14bcc0 2023-10-30T12:32:37.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.707+0000 7f8678ff9640 1 -- 172.21.15.125:0/3914346208 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5748 (secure 0 0 0) 0x7f8670022070 con 0x7f867c14bcc0 2023-10-30T12:32:37.710 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:37.710 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:34.980698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:37.711 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 23 2023-10-30T12:32:37.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.707+0000 7f86822ea640 1 -- 172.21.15.125:0/3914346208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f86680664d0 msgr2=0x7f8668068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.707+0000 7f86822ea640 1 --2- 172.21.15.125:0/3914346208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f86680664d0 0x7f8668068990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f866c001060 tx=0x7f866c009040 comp rx=0 tx=0).stop 2023-10-30T12:32:37.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.707+0000 7f86822ea640 1 -- 172.21.15.125:0/3914346208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f867c14bcc0 msgr2=0x7f867c148d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:37.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.707+0000 7f86822ea640 1 --2- 172.21.15.125:0/3914346208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f867c14bcc0 0x7f867c148d80 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f86700118b0 tx=0x7f8670011d70 comp rx=0 tx=0).stop 2023-10-30T12:32:37.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.707+0000 7f86822ea640 1 -- 172.21.15.125:0/3914346208 shutdown_connections 2023-10-30T12:32:37.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.707+0000 7f86822ea640 1 --2- 172.21.15.125:0/3914346208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f86680664d0 0x7f8668068990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.707+0000 7f86822ea640 1 --2- 172.21.15.125:0/3914346208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f867c144df0 0x7f867c145270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.707+0000 7f86822ea640 1 --2- 172.21.15.125:0/3914346208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f867c14bcc0 0x7f867c148d80 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.707+0000 7f86822ea640 1 --2- 172.21.15.125:0/3914346208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8664005190 0x7f867c1ec6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:37.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.707+0000 7f86822ea640 1 -- 172.21.15.125:0/3914346208 >> 172.21.15.125:0/3914346208 conn(0x7f867c0c7910 msgr2=0x7f867c0ca5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:37.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.707+0000 7f86822ea640 1 -- 172.21.15.125:0/3914346208 shutdown_connections 2023-10-30T12:32:37.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:37.707+0000 7f86822ea640 1 -- 172.21.15.125:0/3914346208 wait complete. 2023-10-30T12:32:37.732 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 23, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 23, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4285": { "addr": "172.21.15.136:6839/2250748765", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 2250748765, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 2250748765, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4294": { "addr": "172.21.15.125:6836/2017382874", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2017382874, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2017382874, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4294, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4957": { "addr": "172.21.15.136:6835/2548530941", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2548530941, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2548530941, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4957, "incarnation": 23, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:32:34.980698+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4294, "mds_1": 4285, "mds_2": 4957 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:32:39.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:39.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf15765640 1 Processor -- start 2023-10-30T12:32:39.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf15765640 1 -- start start 2023-10-30T12:32:39.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf15765640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf100c8cf0 0x7fbf100c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf15765640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbf100ca340 0x7fbf1015d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf15765640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf1015ce30 con 0x7fbf100c90d0 2023-10-30T12:32:39.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf15765640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf10159880 con 0x7fbf100ca340 2023-10-30T12:32:39.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf15765640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf10159ae0 con 0x7fbf100c8cf0 2023-10-30T12:32:39.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf0f7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbf100ca340 0x7fbf1015d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf0f7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbf100ca340 0x7fbf1015d2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50222/0 (socket says 172.21.15.125:50222) 2023-10-30T12:32:39.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf0f7fe640 1 -- 172.21.15.125:0/318894151 learned_addr learned my addr 172.21.15.125:0/318894151 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:39.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf0effd640 1 --2- 172.21.15.125:0/318894151 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf100c8cf0 0x7fbf100c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf0f7fe640 1 -- 172.21.15.125:0/318894151 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf100c8cf0 msgr2=0x7fbf100c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:39.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf0f7fe640 1 --2- 172.21.15.125:0/318894151 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf100c8cf0 0x7fbf100c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf0f7fe640 1 -- 172.21.15.125:0/318894151 >> v1:172.21.15.136:6789/0 conn(0x7fbf100c90d0 legacy=0x7fbf100cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.879+0000 7fbf0f7fe640 1 -- 172.21.15.125:0/318894151 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf10159dc0 con 0x7fbf100ca340 2023-10-30T12:32:39.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.883+0000 7fbf0f7fe640 1 --2- 172.21.15.125:0/318894151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100ca340 0x7fbf1015d2c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009e50 tx=0x7fbf0801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be861bf5b339ae9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:39.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.883+0000 7fbf0dffb640 1 -- 172.21.15.125:0/318894151 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf080a58a0 con 0x7fbf100ca340 2023-10-30T12:32:39.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.883+0000 7fbf0dffb640 1 -- 172.21.15.125:0/318894151 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf080a5a00 con 0x7fbf100ca340 2023-10-30T12:32:39.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.883+0000 7fbf0dffb640 1 -- 172.21.15.125:0/318894151 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf080a5d90 con 0x7fbf100ca340 2023-10-30T12:32:39.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.883+0000 7fbf15765640 1 -- 172.21.15.125:0/318894151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100ca340 msgr2=0x7fbf1015d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.883+0000 7fbf15765640 1 --2- 172.21.15.125:0/318894151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100ca340 0x7fbf1015d2c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbf08009e50 tx=0x7fbf0801c9e0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.883+0000 7fbf15765640 1 -- 172.21.15.125:0/318894151 shutdown_connections 2023-10-30T12:32:39.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.883+0000 7fbf15765640 1 --2- 172.21.15.125:0/318894151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100ca340 0x7fbf1015d2c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.883+0000 7fbf15765640 1 --2- 172.21.15.125:0/318894151 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf100c8cf0 0x7fbf100c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.883+0000 7fbf15765640 1 -- 172.21.15.125:0/318894151 >> 172.21.15.125:0/318894151 conn(0x7fbf1006be40 msgr2=0x7fbf1006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:39.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.883+0000 7fbf15765640 1 -- 172.21.15.125:0/318894151 shutdown_connections 2023-10-30T12:32:39.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.883+0000 7fbf15765640 1 -- 172.21.15.125:0/318894151 wait complete. 2023-10-30T12:32:39.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf15765640 1 Processor -- start 2023-10-30T12:32:39.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf15765640 1 -- start start 2023-10-30T12:32:39.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf15765640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100c8cf0 0x7fbf101f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf15765640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf100c90d0 0x7fbf101f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf15765640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf100ca340 0x7fbf10153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf15765640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf10156c30 con 0x7fbf100c8cf0 2023-10-30T12:32:39.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf15765640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf101f5150 con 0x7fbf100ca340 2023-10-30T12:32:39.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf15765640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf100c7d60 con 0x7fbf100c90d0 2023-10-30T12:32:39.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf0effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100c8cf0 0x7fbf101f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf0effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100c8cf0 0x7fbf101f5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50238/0 (socket says 172.21.15.125:50238) 2023-10-30T12:32:39.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf0effd640 1 -- 172.21.15.125:0/4106028855 learned_addr learned my addr 172.21.15.125:0/4106028855 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:39.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf0f7fe640 1 --2- 172.21.15.125:0/4106028855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf100ca340 0x7fbf10153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf0e7fc640 1 --2- 172.21.15.125:0/4106028855 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf100c90d0 0x7fbf101f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf0effd640 1 -- 172.21.15.125:0/4106028855 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf100c90d0 msgr2=0x7fbf101f1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf0effd640 1 --2- 172.21.15.125:0/4106028855 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf100c90d0 0x7fbf101f1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf0effd640 1 -- 172.21.15.125:0/4106028855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf100ca340 msgr2=0x7fbf10153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:39.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf0effd640 1 --2- 172.21.15.125:0/4106028855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf100ca340 0x7fbf10153680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:39.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf0effd640 1 -- 172.21.15.125:0/4106028855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf08009b30 con 0x7fbf100c8cf0 2023-10-30T12:32:39.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf0effd640 1 --2- 172.21.15.125:0/4106028855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100c8cf0 0x7fbf101f5bf0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000e870 tx=0x7fbf0000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:39.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbef77fe640 1 -- 172.21.15.125:0/4106028855 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf00009510 con 0x7fbf100c8cf0 2023-10-30T12:32:39.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbef77fe640 1 -- 172.21.15.125:0/4106028855 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0000eec0 con 0x7fbf100c8cf0 2023-10-30T12:32:39.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbef77fe640 1 -- 172.21.15.125:0/4106028855 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0000cb40 con 0x7fbf100c8cf0 2023-10-30T12:32:39.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf15765640 1 -- 172.21.15.125:0/4106028855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf100c8040 con 0x7fbf100c8cf0 2023-10-30T12:32:39.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf15765640 1 -- 172.21.15.125:0/4106028855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf101f56d0 con 0x7fbf100c8cf0 2023-10-30T12:32:39.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.887+0000 7fbf15765640 1 -- 172.21.15.125:0/4106028855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf101f5890 con 0x7fbf100c8cf0 2023-10-30T12:32:39.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.891+0000 7fbef77fe640 1 -- 172.21.15.125:0/4106028855 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbf0000cca0 con 0x7fbf100c8cf0 2023-10-30T12:32:39.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.891+0000 7fbef77fe640 1 --2- 172.21.15.125:0/4106028855 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbefc0664a0 0x7fbefc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:39.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.891+0000 7fbef77fe640 1 -- 172.21.15.125:0/4106028855 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fbf000840a0 con 0x7fbf100c8cf0 2023-10-30T12:32:39.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.891+0000 7fbf0e7fc640 1 --2- 172.21.15.125:0/4106028855 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbefc0664a0 0x7fbefc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:39.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.891+0000 7fbf0e7fc640 1 --2- 172.21.15.125:0/4106028855 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbefc0664a0 0x7fbefc068960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbf10158e20 tx=0x7fbf04009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:39.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:39.895+0000 7fbef77fe640 1 -- 172.21.15.125:0/4106028855 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbf000513b0 con 0x7fbf100c8cf0 2023-10-30T12:32:40.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:40.083+0000 7fbf15765640 1 -- 172.21.15.125:0/4106028855 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf1004e170 con 0x7fbf100c8cf0 2023-10-30T12:32:40.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:40.083+0000 7fbef77fe640 1 -- 172.21.15.125:0/4106028855 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+5748 (secure 0 0 0) 0x7fbf000549d0 con 0x7fbf100c8cf0 2023-10-30T12:32:40.088 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:40.088 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:34.980698+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:40.088 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 23 2023-10-30T12:32:40.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:40.087+0000 7fbf15765640 1 -- 172.21.15.125:0/4106028855 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbefc0664a0 msgr2=0x7fbefc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:40.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:40.087+0000 7fbf15765640 1 --2- 172.21.15.125:0/4106028855 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbefc0664a0 0x7fbefc068960 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbf10158e20 tx=0x7fbf04009040 comp rx=0 tx=0).stop 2023-10-30T12:32:40.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:40.087+0000 7fbf15765640 1 -- 172.21.15.125:0/4106028855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100c8cf0 msgr2=0x7fbf101f5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:40.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:40.087+0000 7fbf15765640 1 --2- 172.21.15.125:0/4106028855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100c8cf0 0x7fbf101f5bf0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbf0000e870 tx=0x7fbf0000ec70 comp rx=0 tx=0).stop 2023-10-30T12:32:40.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:40.087+0000 7fbf15765640 1 -- 172.21.15.125:0/4106028855 shutdown_connections 2023-10-30T12:32:40.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:40.087+0000 7fbf15765640 1 --2- 172.21.15.125:0/4106028855 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbefc0664a0 0x7fbefc068960 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:40.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:40.087+0000 7fbf15765640 1 --2- 172.21.15.125:0/4106028855 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf100ca340 0x7fbf10153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:40.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:40.087+0000 7fbf15765640 1 --2- 172.21.15.125:0/4106028855 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf100c90d0 0x7fbf101f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:40.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:40.087+0000 7fbf15765640 1 --2- 172.21.15.125:0/4106028855 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf100c8cf0 0x7fbf101f5bf0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:40.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:40.087+0000 7fbf15765640 1 -- 172.21.15.125:0/4106028855 >> 172.21.15.125:0/4106028855 conn(0x7fbf1006be40 msgr2=0x7fbf100b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:40.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:40.087+0000 7fbf15765640 1 -- 172.21.15.125:0/4106028855 shutdown_connections 2023-10-30T12:32:40.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:40.087+0000 7fbf15765640 1 -- 172.21.15.125:0/4106028855 wait complete. 2023-10-30T12:32:40.108 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:32:42.109 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:42.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361f59e640 1 Processor -- start 2023-10-30T12:32:42.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361f59e640 1 -- start start 2023-10-30T12:32:42.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361f59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f362015cd50 0x7f36200c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:42.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361f59e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f362015a6c0 con 0x7f36200cefa0 2023-10-30T12:32:42.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361f59e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f362015d300 con 0x7f36201538f0 2023-10-30T12:32:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361f59e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362015d5e0 con 0x7f362015cd50 2023-10-30T12:32:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361ed9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f36200cefa0 0x7f36201640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44572/0 (socket says 172.21.15.125:44572) 2023-10-30T12:32:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361ed9d640 1 -- 172.21.15.125:0/1329693709 learned_addr learned my addr 172.21.15.125:0/1329693709 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:32:42.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361dd9b640 1 --2- 172.21.15.125:0/1329693709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f362015cd50 0x7f36200c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:42.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361d59a640 1 -- 172.21.15.125:0/1329693709 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263754796 0 0) 0x7f362015a6c0 con 0x7f36200cefa0 2023-10-30T12:32:42.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361d59a640 1 -- 172.21.15.125:0/1329693709 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3608003530 con 0x7f36200cefa0 2023-10-30T12:32:42.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361d59a640 1 -- 172.21.15.125:0/1329693709 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636137744 0 0) 0x7f362015d300 con 0x7f36201538f0 2023-10-30T12:32:42.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361d59a640 1 -- 172.21.15.125:0/1329693709 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f362015a6c0 con 0x7f36201538f0 2023-10-30T12:32:42.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361d59a640 1 -- 172.21.15.125:0/1329693709 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3174459184 0 0) 0x7f3608003530 con 0x7f36200cefa0 2023-10-30T12:32:42.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361dd9b640 1 -- 172.21.15.125:0/1329693709 >> v1:172.21.15.136:6789/0 conn(0x7f36201538f0 legacy=0x7f3620159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:42.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361dd9b640 1 -- 172.21.15.125:0/1329693709 >> v1:172.21.15.125:6789/0 conn(0x7f36200cefa0 legacy=0x7f36201640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:42.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.255+0000 7f361dd9b640 1 -- 172.21.15.125:0/1329693709 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362015d8c0 con 0x7f362015cd50 2023-10-30T12:32:42.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.259+0000 7f361dd9b640 1 --2- 172.21.15.125:0/1329693709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f362015cd50 0x7f36200c9f90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f36140097f0 tx=0x7f361401ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e819b8ae40f42c82 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:42.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.259+0000 7f361d59a640 1 -- 172.21.15.125:0/1329693709 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3618001a00 con 0x7f362015cd50 2023-10-30T12:32:42.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.259+0000 7f361d59a640 1 -- 172.21.15.125:0/1329693709 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36140a8650 con 0x7f362015cd50 2023-10-30T12:32:42.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.259+0000 7f361d59a640 1 -- 172.21.15.125:0/1329693709 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36140a89c0 con 0x7f362015cd50 2023-10-30T12:32:42.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.259+0000 7f361f59e640 1 -- 172.21.15.125:0/1329693709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f362015cd50 msgr2=0x7f36200c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:42.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.259+0000 7f361f59e640 1 --2- 172.21.15.125:0/1329693709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f362015cd50 0x7f36200c9f90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f36140097f0 tx=0x7f361401ebd0 comp rx=0 tx=0).stop 2023-10-30T12:32:42.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.259+0000 7f361f59e640 1 -- 172.21.15.125:0/1329693709 shutdown_connections 2023-10-30T12:32:42.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.259+0000 7f361f59e640 1 --2- 172.21.15.125:0/1329693709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f362015cd50 0x7f36200c9f90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:42.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.259+0000 7f361f59e640 1 -- 172.21.15.125:0/1329693709 >> 172.21.15.125:0/1329693709 conn(0x7f362006a640 msgr2=0x7f3620065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:42.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.259+0000 7f361f59e640 1 -- 172.21.15.125:0/1329693709 shutdown_connections 2023-10-30T12:32:42.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.259+0000 7f361f59e640 1 -- 172.21.15.125:0/1329693709 wait complete. 2023-10-30T12:32:42.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361f59e640 1 Processor -- start 2023-10-30T12:32:42.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361f59e640 1 -- start start 2023-10-30T12:32:42.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361f59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f36200cefa0 0x7f3620141ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:42.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361f59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f36201538f0 0x7f3620151340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:42.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361f59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f362015cd50 0x7f362013f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:42.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361f59e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3620150ef0 con 0x7f362015cd50 2023-10-30T12:32:42.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361f59e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3620159370 con 0x7f36201538f0 2023-10-30T12:32:42.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361f59e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3620159580 con 0x7f36200cefa0 2023-10-30T12:32:42.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361ed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f362015cd50 0x7f362013f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:42.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361ed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f362015cd50 0x7f362013f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41222/0 (socket says 172.21.15.125:41222) 2023-10-30T12:32:42.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361ed9d640 1 -- 172.21.15.125:0/1308259617 learned_addr learned my addr 172.21.15.125:0/1308259617 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:42.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361e59c640 1 --2- 172.21.15.125:0/1308259617 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f36200cefa0 0x7f3620141ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:42.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361dd9b640 1 --2- 172.21.15.125:0/1308259617 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f36201538f0 0x7f3620151340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:42.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361ed9d640 1 -- 172.21.15.125:0/1308259617 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f36200cefa0 msgr2=0x7f3620141ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:42.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361ed9d640 1 --2- 172.21.15.125:0/1308259617 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f36200cefa0 0x7f3620141ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:42.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361ed9d640 1 -- 172.21.15.125:0/1308259617 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f36201538f0 msgr2=0x7f3620151340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:42.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361ed9d640 1 --2- 172.21.15.125:0/1308259617 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f36201538f0 0x7f3620151340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:42.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361ed9d640 1 -- 172.21.15.125:0/1308259617 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3614009420 con 0x7f362015cd50 2023-10-30T12:32:42.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361ed9d640 1 --2- 172.21.15.125:0/1308259617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f362015cd50 0x7f362013f8b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3618001f70 tx=0x7f361800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:42.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f360effd640 1 -- 172.21.15.125:0/1308259617 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3618012400 con 0x7f362015cd50 2023-10-30T12:32:42.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f360effd640 1 -- 172.21.15.125:0/1308259617 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3618012560 con 0x7f362015cd50 2023-10-30T12:32:42.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361f59e640 1 -- 172.21.15.125:0/1308259617 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3620157e70 con 0x7f362015cd50 2023-10-30T12:32:42.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f360effd640 1 -- 172.21.15.125:0/1308259617 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3618012910 con 0x7f362015cd50 2023-10-30T12:32:42.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.263+0000 7f361f59e640 1 -- 172.21.15.125:0/1308259617 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3620142020 con 0x7f362015cd50 2023-10-30T12:32:42.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.267+0000 7f361f59e640 1 -- 172.21.15.125:0/1308259617 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3620157cd0 con 0x7f362015cd50 2023-10-30T12:32:42.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.267+0000 7f360effd640 1 -- 172.21.15.125:0/1308259617 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3618012a70 con 0x7f362015cd50 2023-10-30T12:32:42.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.267+0000 7f360effd640 1 --2- 172.21.15.125:0/1308259617 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f36000664d0 0x7f3600068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:42.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.267+0000 7f360effd640 1 -- 172.21.15.125:0/1308259617 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f3618084600 con 0x7f362015cd50 2023-10-30T12:32:42.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.267+0000 7f361e59c640 1 --2- 172.21.15.125:0/1308259617 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f36000664d0 0x7f3600068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:42.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.267+0000 7f361e59c640 1 --2- 172.21.15.125:0/1308259617 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f36000664d0 0x7f3600068990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f36100017c0 tx=0x7f3610009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:42.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.271+0000 7f360effd640 1 -- 172.21.15.125:0/1308259617 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3620157cd0 con 0x7f362015cd50 2023-10-30T12:32:42.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.451+0000 7f361f59e640 1 -- 172.21.15.125:0/1308259617 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3620159a70 con 0x7f362015cd50 2023-10-30T12:32:42.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.455+0000 7f360effd640 1 -- 172.21.15.125:0/1308259617 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5750 (secure 0 0 0) 0x7f3620159a70 con 0x7f362015cd50 2023-10-30T12:32:42.457 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:42.457 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:40.356978+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:resolve","state_seq":28,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:42.458 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 24 2023-10-30T12:32:42.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.455+0000 7f361f59e640 1 -- 172.21.15.125:0/1308259617 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f36000664d0 msgr2=0x7f3600068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.455+0000 7f361f59e640 1 --2- 172.21.15.125:0/1308259617 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f36000664d0 0x7f3600068990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f36100017c0 tx=0x7f3610009040 comp rx=0 tx=0).stop 2023-10-30T12:32:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.455+0000 7f361f59e640 1 -- 172.21.15.125:0/1308259617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f362015cd50 msgr2=0x7f362013f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.455+0000 7f361f59e640 1 --2- 172.21.15.125:0/1308259617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f362015cd50 0x7f362013f8b0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f3618001f70 tx=0x7f361800a430 comp rx=0 tx=0).stop 2023-10-30T12:32:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.455+0000 7f361f59e640 1 -- 172.21.15.125:0/1308259617 shutdown_connections 2023-10-30T12:32:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.455+0000 7f361f59e640 1 --2- 172.21.15.125:0/1308259617 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f36000664d0 0x7f3600068990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:42.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.455+0000 7f361f59e640 1 --2- 172.21.15.125:0/1308259617 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f362015cd50 0x7f362013f8b0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.455+0000 7f361f59e640 1 --2- 172.21.15.125:0/1308259617 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f36201538f0 0x7f3620151340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.455+0000 7f361f59e640 1 --2- 172.21.15.125:0/1308259617 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f36200cefa0 0x7f3620141ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.455+0000 7f361f59e640 1 -- 172.21.15.125:0/1308259617 >> 172.21.15.125:0/1308259617 conn(0x7f362006a640 msgr2=0x7f36200c9a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.455+0000 7f361f59e640 1 -- 172.21.15.125:0/1308259617 shutdown_connections 2023-10-30T12:32:42.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:42.455+0000 7f361f59e640 1 -- 172.21.15.125:0/1308259617 wait complete. 2023-10-30T12:32:42.477 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:32:44.410 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:32:44.415 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:32:44.478 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:44.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f29367d3640 1 Processor -- start 2023-10-30T12:32:44.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f29367d3640 1 -- start start 2023-10-30T12:32:44.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f29367d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f293015d6f0 0x7f293015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:44.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f29367d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f29300cf160 0x7f29300cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:44.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f29367d3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29300cb800 con 0x7f29301538a0 2023-10-30T12:32:44.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f29367d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29300cb330 con 0x7f29300cf160 2023-10-30T12:32:44.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f29367d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29300cad70 con 0x7f293015d6f0 2023-10-30T12:32:44.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f2934d70640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f29300cf160 0x7f29300cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:44.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f2934d70640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f29300cf160 0x7f29300cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41224/0 (socket says 172.21.15.125:41224) 2023-10-30T12:32:44.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f2934d70640 1 -- 172.21.15.125:0/4200711758 learned_addr learned my addr 172.21.15.125:0/4200711758 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:44.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f292f7fe640 1 --2- 172.21.15.125:0/4200711758 >> v2:172.21.15.136:3300/0 conn(0x7f293015d6f0 0x7f293015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:44.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f2934d70640 1 -- 172.21.15.125:0/4200711758 >> v2:172.21.15.136:3300/0 conn(0x7f293015d6f0 msgr2=0x7f293015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:44.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f2934d70640 1 --2- 172.21.15.125:0/4200711758 >> v2:172.21.15.136:3300/0 conn(0x7f293015d6f0 0x7f293015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:44.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f2934d70640 1 -- 172.21.15.125:0/4200711758 >> v1:172.21.15.136:6789/0 conn(0x7f29301538a0 legacy=0x7f29300ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:44.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.619+0000 7f2934d70640 1 -- 172.21.15.125:0/4200711758 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29300cb050 con 0x7f29300cf160 2023-10-30T12:32:44.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.623+0000 7f2934d70640 1 --2- 172.21.15.125:0/4200711758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29300cf160 0x7f29300cf590 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f29300b30a0 tx=0x7f292801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=173f691cab1e92e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:44.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.623+0000 7f292effd640 1 -- 172.21.15.125:0/4200711758 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29280af400 con 0x7f29300cf160 2023-10-30T12:32:44.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.623+0000 7f292effd640 1 -- 172.21.15.125:0/4200711758 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29280ab050 con 0x7f29300cf160 2023-10-30T12:32:44.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.623+0000 7f292effd640 1 -- 172.21.15.125:0/4200711758 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29280af780 con 0x7f29300cf160 2023-10-30T12:32:44.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.623+0000 7f29367d3640 1 -- 172.21.15.125:0/4200711758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29300cf160 msgr2=0x7f29300cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:44.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.623+0000 7f29367d3640 1 --2- 172.21.15.125:0/4200711758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29300cf160 0x7f29300cf590 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f29300b30a0 tx=0x7f292801f1c0 comp rx=0 tx=0).stop 2023-10-30T12:32:44.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.623+0000 7f29367d3640 1 -- 172.21.15.125:0/4200711758 shutdown_connections 2023-10-30T12:32:44.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.623+0000 7f29367d3640 1 --2- 172.21.15.125:0/4200711758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29300cf160 0x7f29300cf590 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:44.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.623+0000 7f29367d3640 1 --2- 172.21.15.125:0/4200711758 >> v2:172.21.15.136:3300/0 conn(0x7f293015d6f0 0x7f293015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:44.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.623+0000 7f29367d3640 1 -- 172.21.15.125:0/4200711758 >> 172.21.15.125:0/4200711758 conn(0x7f293006bfe0 msgr2=0x7f293006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:44.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.623+0000 7f29367d3640 1 -- 172.21.15.125:0/4200711758 shutdown_connections 2023-10-30T12:32:44.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.623+0000 7f29367d3640 1 -- 172.21.15.125:0/4200711758 wait complete. 2023-10-30T12:32:44.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f29367d3640 1 Processor -- start 2023-10-30T12:32:44.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f29367d3640 1 -- start start 2023-10-30T12:32:44.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f29367d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29300cf160 0x7f29301f0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:44.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f29367d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29301538a0 0x7f29300b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:44.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f29367d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f293015d6f0 0x7f29301f2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:44.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f29367d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29301f0070 con 0x7f293015d6f0 2023-10-30T12:32:44.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f29367d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29301e37b0 con 0x7f29301538a0 2023-10-30T12:32:44.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f29367d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29301e2be0 con 0x7f29300cf160 2023-10-30T12:32:44.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f2934d70640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f293015d6f0 0x7f29301f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:44.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f2934d70640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f293015d6f0 0x7f29301f2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41240/0 (socket says 172.21.15.125:41240) 2023-10-30T12:32:44.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f2934d70640 1 -- 172.21.15.125:0/563433777 learned_addr learned my addr 172.21.15.125:0/563433777 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:44.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f292f7fe640 1 --2- 172.21.15.125:0/563433777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29301538a0 0x7f29300b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:44.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f292ffff640 1 --2- 172.21.15.125:0/563433777 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29300cf160 0x7f29301f0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:44.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f2934d70640 1 -- 172.21.15.125:0/563433777 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29300cf160 msgr2=0x7f29301f0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:44.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f2934d70640 1 --2- 172.21.15.125:0/563433777 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29300cf160 0x7f29301f0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:44.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f2934d70640 1 -- 172.21.15.125:0/563433777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29301538a0 msgr2=0x7f29300b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:44.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f2934d70640 1 --2- 172.21.15.125:0/563433777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29301538a0 0x7f29300b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:44.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f2934d70640 1 -- 172.21.15.125:0/563433777 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2928009b30 con 0x7f293015d6f0 2023-10-30T12:32:44.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f2934d70640 1 --2- 172.21.15.125:0/563433777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f293015d6f0 0x7f29301f2600 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f292801cdf0 tx=0x7f292801bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:44.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f292cff9640 1 -- 172.21.15.125:0/563433777 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29280afd10 con 0x7f293015d6f0 2023-10-30T12:32:44.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f292cff9640 1 -- 172.21.15.125:0/563433777 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29280afe90 con 0x7f293015d6f0 2023-10-30T12:32:44.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f292cff9640 1 -- 172.21.15.125:0/563433777 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29280ba680 con 0x7f293015d6f0 2023-10-30T12:32:44.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f29367d3640 1 -- 172.21.15.125:0/563433777 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29301e2e80 con 0x7f293015d6f0 2023-10-30T12:32:44.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.627+0000 7f29367d3640 1 -- 172.21.15.125:0/563433777 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29301e3440 con 0x7f293015d6f0 2023-10-30T12:32:44.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.631+0000 7f29367d3640 1 -- 172.21.15.125:0/563433777 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29300b2a50 con 0x7f293015d6f0 2023-10-30T12:32:44.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.631+0000 7f292cff9640 1 -- 172.21.15.125:0/563433777 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29280b6040 con 0x7f293015d6f0 2023-10-30T12:32:44.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.631+0000 7f292cff9640 1 --2- 172.21.15.125:0/563433777 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f291c0664d0 0x7f291c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:44.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.631+0000 7f292cff9640 1 -- 172.21.15.125:0/563433777 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f2928122070 con 0x7f293015d6f0 2023-10-30T12:32:44.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.631+0000 7f292ffff640 1 --2- 172.21.15.125:0/563433777 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f291c0664d0 0x7f291c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:44.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.631+0000 7f292ffff640 1 --2- 172.21.15.125:0/563433777 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f291c0664d0 0x7f291c068990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2920007fd0 tx=0x7f2920009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:44.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.635+0000 7f292cff9640 1 -- 172.21.15.125:0/563433777 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29280ef3a0 con 0x7f293015d6f0 2023-10-30T12:32:44.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.815+0000 7f29367d3640 1 -- 172.21.15.125:0/563433777 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29301e38f0 con 0x7f293015d6f0 2023-10-30T12:32:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.815+0000 7f292cff9640 1 -- 172.21.15.125:0/563433777 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+5750 (secure 0 0 0) 0x7f2928009510 con 0x7f293015d6f0 2023-10-30T12:32:44.820 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:44.821 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:40.356978+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:resolve","state_seq":28,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:44.823 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 24 2023-10-30T12:32:44.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.819+0000 7f29367d3640 1 -- 172.21.15.125:0/563433777 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f291c0664d0 msgr2=0x7f291c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:44.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.819+0000 7f29367d3640 1 --2- 172.21.15.125:0/563433777 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f291c0664d0 0x7f291c068990 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2920007fd0 tx=0x7f2920009040 comp rx=0 tx=0).stop 2023-10-30T12:32:44.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.819+0000 7f29367d3640 1 -- 172.21.15.125:0/563433777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f293015d6f0 msgr2=0x7f29301f2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:44.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.819+0000 7f29367d3640 1 --2- 172.21.15.125:0/563433777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f293015d6f0 0x7f29301f2600 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f292801cdf0 tx=0x7f292801bf60 comp rx=0 tx=0).stop 2023-10-30T12:32:44.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.819+0000 7f29367d3640 1 -- 172.21.15.125:0/563433777 shutdown_connections 2023-10-30T12:32:44.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.819+0000 7f29367d3640 1 --2- 172.21.15.125:0/563433777 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f291c0664d0 0x7f291c068990 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:44.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.819+0000 7f29367d3640 1 --2- 172.21.15.125:0/563433777 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f293015d6f0 0x7f29301f2600 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:44.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.819+0000 7f29367d3640 1 --2- 172.21.15.125:0/563433777 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29301538a0 0x7f29300b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:44.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.819+0000 7f29367d3640 1 --2- 172.21.15.125:0/563433777 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29300cf160 0x7f29301f0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:44.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.819+0000 7f29367d3640 1 -- 172.21.15.125:0/563433777 >> 172.21.15.125:0/563433777 conn(0x7f293006bfe0 msgr2=0x7f293015b340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:44.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.819+0000 7f29367d3640 1 -- 172.21.15.125:0/563433777 shutdown_connections 2023-10-30T12:32:44.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:44.819+0000 7f29367d3640 1 -- 172.21.15.125:0/563433777 wait complete. 2023-10-30T12:32:44.841 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:32:46.841 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:46.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.983+0000 7fcdd8a91640 1 Processor -- start 2023-10-30T12:32:46.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.983+0000 7fcdd8a91640 1 -- start start 2023-10-30T12:32:46.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.987+0000 7fcdd8a91640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcdd40c6bd0 0x7fcdd406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:46.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.987+0000 7fcdd8a91640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcdd4071110 0x7fcdd4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:46.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.987+0000 7fcdd8a91640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd4066cf0 con 0x7fcdd406f7a0 2023-10-30T12:32:46.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.987+0000 7fcdd8a91640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd40700c0 con 0x7fcdd40c6bd0 2023-10-30T12:32:46.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.987+0000 7fcdd8a91640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd40c70d0 con 0x7fcdd4071110 2023-10-30T12:32:46.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.987+0000 7fcdc9d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcdd40c6bd0 0x7fcdd406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:46.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.987+0000 7fcdc9d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcdd40c6bd0 0x7fcdd406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41244/0 (socket says 172.21.15.125:41244) 2023-10-30T12:32:46.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.987+0000 7fcdc9d9b640 1 -- 172.21.15.125:0/695749354 learned_addr learned my addr 172.21.15.125:0/695749354 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:46.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.987+0000 7fcdd2d9d640 1 --2- 172.21.15.125:0/695749354 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcdd4071110 0x7fcdd4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:46.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.987+0000 7fcdc9d9b640 1 -- 172.21.15.125:0/695749354 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcdd4071110 msgr2=0x7fcdd4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:46.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.987+0000 7fcdc9d9b640 1 --2- 172.21.15.125:0/695749354 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcdd4071110 0x7fcdd4158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:46.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.987+0000 7fcdc9d9b640 1 -- 172.21.15.125:0/695749354 >> v1:172.21.15.136:6789/0 conn(0x7fcdd406f7a0 legacy=0x7fcdd4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:46.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.987+0000 7fcdc9d9b640 1 -- 172.21.15.125:0/695749354 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd40c7390 con 0x7fcdd40c6bd0 2023-10-30T12:32:46.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.987+0000 7fcdd2d9d640 1 --2- 172.21.15.125:0/695749354 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcdd4071110 0x7fcdd4158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:32:46.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.991+0000 7fcdc9d9b640 1 --2- 172.21.15.125:0/695749354 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcdd40c6bd0 0x7fcdd406fb80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009870 tx=0x7fcdbc01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d34c4aa31e5e176 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:46.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.991+0000 7fcdd1d9b640 1 -- 172.21.15.125:0/695749354 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc0a8a30 con 0x7fcdd40c6bd0 2023-10-30T12:32:46.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.991+0000 7fcdd1d9b640 1 -- 172.21.15.125:0/695749354 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdbc0a8b90 con 0x7fcdd40c6bd0 2023-10-30T12:32:46.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.991+0000 7fcdd1d9b640 1 -- 172.21.15.125:0/695749354 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc0b0440 con 0x7fcdd40c6bd0 2023-10-30T12:32:46.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.991+0000 7fcdd8a91640 1 -- 172.21.15.125:0/695749354 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcdd40c6bd0 msgr2=0x7fcdd406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:46.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.991+0000 7fcdd8a91640 1 --2- 172.21.15.125:0/695749354 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcdd40c6bd0 0x7fcdd406fb80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009870 tx=0x7fcdbc01ed30 comp rx=0 tx=0).stop 2023-10-30T12:32:46.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.991+0000 7fcdd8a91640 1 -- 172.21.15.125:0/695749354 shutdown_connections 2023-10-30T12:32:46.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.991+0000 7fcdd8a91640 1 --2- 172.21.15.125:0/695749354 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcdd4071110 0x7fcdd4158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:46.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.991+0000 7fcdd8a91640 1 --2- 172.21.15.125:0/695749354 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcdd40c6bd0 0x7fcdd406fb80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:46.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.991+0000 7fcdd8a91640 1 -- 172.21.15.125:0/695749354 >> 172.21.15.125:0/695749354 conn(0x7fcdd40c7910 msgr2=0x7fcdd40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:46.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.991+0000 7fcdd8a91640 1 -- 172.21.15.125:0/695749354 shutdown_connections 2023-10-30T12:32:46.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.991+0000 7fcdd8a91640 1 -- 172.21.15.125:0/695749354 wait complete. 2023-10-30T12:32:46.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.991+0000 7fcdd8a91640 1 Processor -- start 2023-10-30T12:32:46.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.991+0000 7fcdd8a91640 1 -- start start 2023-10-30T12:32:47.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd8a91640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcdd406f7a0 0x7fcdd4156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:47.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd8a91640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdd4071110 0x7fcdd4153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:47.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd8a91640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcdd40c6bd0 0x7fcdd4153bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:47.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd8a91640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd41ef5d0 con 0x7fcdd40c6bd0 2023-10-30T12:32:47.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd8a91640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd41531f0 con 0x7fcdd4071110 2023-10-30T12:32:47.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd8a91640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd41efac0 con 0x7fcdd406f7a0 2023-10-30T12:32:47.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd2d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcdd40c6bd0 0x7fcdd4153bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:47.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd2d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcdd40c6bd0 0x7fcdd4153bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41254/0 (socket says 172.21.15.125:41254) 2023-10-30T12:32:47.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd2d9d640 1 -- 172.21.15.125:0/2711250292 learned_addr learned my addr 172.21.15.125:0/2711250292 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:47.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd259c640 1 --2- 172.21.15.125:0/2711250292 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcdd406f7a0 0x7fcdd4156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:47.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdc9d9b640 1 --2- 172.21.15.125:0/2711250292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdd4071110 0x7fcdd4153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:47.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd2d9d640 1 -- 172.21.15.125:0/2711250292 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcdd406f7a0 msgr2=0x7fcdd4156da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:47.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd2d9d640 1 --2- 172.21.15.125:0/2711250292 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcdd406f7a0 0x7fcdd4156da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:47.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd2d9d640 1 -- 172.21.15.125:0/2711250292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdd4071110 msgr2=0x7fcdd4153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:47.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd2d9d640 1 --2- 172.21.15.125:0/2711250292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdd4071110 0x7fcdd4153680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:47.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd2d9d640 1 -- 172.21.15.125:0/2711250292 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc0094d0 con 0x7fcdd40c6bd0 2023-10-30T12:32:47.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd2d9d640 1 --2- 172.21.15.125:0/2711250292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcdd40c6bd0 0x7fcdd4153bc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc007ef0 tx=0x7fcdcc00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:47.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdcb7fe640 1 -- 172.21.15.125:0/2711250292 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc003ba0 con 0x7fcdd40c6bd0 2023-10-30T12:32:47.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdcb7fe640 1 -- 172.21.15.125:0/2711250292 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdcc00f040 con 0x7fcdd40c6bd0 2023-10-30T12:32:47.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd8a91640 1 -- 172.21.15.125:0/2711250292 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd4152ec0 con 0x7fcdd40c6bd0 2023-10-30T12:32:47.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdcb7fe640 1 -- 172.21.15.125:0/2711250292 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc00de80 con 0x7fcdd40c6bd0 2023-10-30T12:32:47.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd8a91640 1 -- 172.21.15.125:0/2711250292 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd41ef7e0 con 0x7fcdd40c6bd0 2023-10-30T12:32:47.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdd8a91640 1 -- 172.21.15.125:0/2711250292 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd4153360 con 0x7fcdd40c6bd0 2023-10-30T12:32:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.995+0000 7fcdcb7fe640 1 -- 172.21.15.125:0/2711250292 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcdcc004030 con 0x7fcdd40c6bd0 2023-10-30T12:32:47.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.999+0000 7fcdcb7fe640 1 --2- 172.21.15.125:0/2711250292 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcdb40664a0 0x7fcdb4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:47.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.999+0000 7fcdcb7fe640 1 -- 172.21.15.125:0/2711250292 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fcdcc084130 con 0x7fcdd40c6bd0 2023-10-30T12:32:47.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.999+0000 7fcdd259c640 1 --2- 172.21.15.125:0/2711250292 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcdb40664a0 0x7fcdb4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:47.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:46.999+0000 7fcdd259c640 1 --2- 172.21.15.125:0/2711250292 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcdb40664a0 0x7fcdb4068960 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4007fd0 tx=0x7fcdc4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:47.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.003+0000 7fcdcb7fe640 1 -- 172.21.15.125:0/2711250292 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcdcc051620 con 0x7fcdd40c6bd0 2023-10-30T12:32:47.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.183+0000 7fcdd8a91640 1 -- 172.21.15.125:0/2711250292 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcdd404e170 con 0x7fcdd40c6bd0 2023-10-30T12:32:47.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.183+0000 7fcdcb7fe640 1 -- 172.21.15.125:0/2711250292 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5752 (secure 0 0 0) 0x7fcdcc054c40 con 0x7fcdd40c6bd0 2023-10-30T12:32:47.191 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:47.191 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:46.178886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:47.192 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 25 2023-10-30T12:32:47.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.187+0000 7fcdd8a91640 1 -- 172.21.15.125:0/2711250292 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcdb40664a0 msgr2=0x7fcdb4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:47.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.187+0000 7fcdd8a91640 1 --2- 172.21.15.125:0/2711250292 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcdb40664a0 0x7fcdb4068960 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4007fd0 tx=0x7fcdc4009040 comp rx=0 tx=0).stop 2023-10-30T12:32:47.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.187+0000 7fcdd8a91640 1 -- 172.21.15.125:0/2711250292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcdd40c6bd0 msgr2=0x7fcdd4153bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:47.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.187+0000 7fcdd8a91640 1 --2- 172.21.15.125:0/2711250292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcdd40c6bd0 0x7fcdd4153bc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc007ef0 tx=0x7fcdcc00d4a0 comp rx=0 tx=0).stop 2023-10-30T12:32:47.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.187+0000 7fcdd8a91640 1 -- 172.21.15.125:0/2711250292 shutdown_connections 2023-10-30T12:32:47.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.187+0000 7fcdd8a91640 1 --2- 172.21.15.125:0/2711250292 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcdb40664a0 0x7fcdb4068960 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:47.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.187+0000 7fcdd8a91640 1 --2- 172.21.15.125:0/2711250292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcdd40c6bd0 0x7fcdd4153bc0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:47.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.187+0000 7fcdd8a91640 1 --2- 172.21.15.125:0/2711250292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcdd4071110 0x7fcdd4153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:47.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.187+0000 7fcdd8a91640 1 --2- 172.21.15.125:0/2711250292 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcdd406f7a0 0x7fcdd4156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:47.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.187+0000 7fcdd8a91640 1 -- 172.21.15.125:0/2711250292 >> 172.21.15.125:0/2711250292 conn(0x7fcdd40c7910 msgr2=0x7fcdd40c5f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:47.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.187+0000 7fcdd8a91640 1 -- 172.21.15.125:0/2711250292 shutdown_connections 2023-10-30T12:32:47.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:47.187+0000 7fcdd8a91640 1 -- 172.21.15.125:0/2711250292 wait complete. 2023-10-30T12:32:47.211 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:32:49.212 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:49.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.355+0000 7ff50a3b8640 1 Processor -- start 2023-10-30T12:32:49.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.355+0000 7ff50a3b8640 1 -- start start 2023-10-30T12:32:49.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.359+0000 7ff50a3b8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff504070a00 0x7ff504070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:49.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.359+0000 7ff50a3b8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff504069e90 0x7ff50406a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:49.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.359+0000 7ff50a3b8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff504071380 con 0x7ff5041539a0 2023-10-30T12:32:49.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.359+0000 7ff50a3b8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50406fbb0 con 0x7ff504069e90 2023-10-30T12:32:49.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.359+0000 7ff50a3b8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff504153d80 con 0x7ff504070a00 2023-10-30T12:32:49.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.359+0000 7ff5037fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff504070a00 0x7ff504070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:49.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.359+0000 7ff508955640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff504069e90 0x7ff50406a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:49.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.359+0000 7ff503fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff5041539a0 0x7ff50415a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:45430/0 (socket says 172.21.15.125:45430) 2023-10-30T12:32:49.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.359+0000 7ff503fff640 1 -- 172.21.15.125:0/665010091 learned_addr learned my addr 172.21.15.125:0/665010091 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:32:49.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.359+0000 7ff508955640 1 -- 172.21.15.125:0/665010091 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff504070a00 msgr2=0x7ff504070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:49.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.359+0000 7ff508955640 1 --2- 172.21.15.125:0/665010091 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff504070a00 0x7ff504070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:49.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.359+0000 7ff508955640 1 -- 172.21.15.125:0/665010091 >> v1:172.21.15.136:6789/0 conn(0x7ff5041539a0 legacy=0x7ff50415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:49.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.359+0000 7ff508955640 1 -- 172.21.15.125:0/665010091 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5040c5c50 con 0x7ff504069e90 2023-10-30T12:32:49.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.363+0000 7ff508955640 1 --2- 172.21.15.125:0/665010091 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff504069e90 0x7ff50406a300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009d90 tx=0x7ff4fc01c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4b95996966fb62a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:49.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.363+0000 7ff502ffd640 1 -- 172.21.15.125:0/665010091 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0a5860 con 0x7ff504069e90 2023-10-30T12:32:49.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.363+0000 7ff502ffd640 1 -- 172.21.15.125:0/665010091 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc0a59c0 con 0x7ff504069e90 2023-10-30T12:32:49.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.363+0000 7ff502ffd640 1 -- 172.21.15.125:0/665010091 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc0a5d50 con 0x7ff504069e90 2023-10-30T12:32:49.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.363+0000 7ff50a3b8640 1 -- 172.21.15.125:0/665010091 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff504069e90 msgr2=0x7ff50406a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:49.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.363+0000 7ff50a3b8640 1 --2- 172.21.15.125:0/665010091 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff504069e90 0x7ff50406a300 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009d90 tx=0x7ff4fc01c890 comp rx=0 tx=0).stop 2023-10-30T12:32:49.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.363+0000 7ff50a3b8640 1 -- 172.21.15.125:0/665010091 shutdown_connections 2023-10-30T12:32:49.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.363+0000 7ff50a3b8640 1 --2- 172.21.15.125:0/665010091 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff504069e90 0x7ff50406a300 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:49.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.363+0000 7ff50a3b8640 1 --2- 172.21.15.125:0/665010091 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff504070a00 0x7ff504070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:49.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.363+0000 7ff50a3b8640 1 -- 172.21.15.125:0/665010091 >> 172.21.15.125:0/665010091 conn(0x7ff5040c7910 msgr2=0x7ff5040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:49.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.363+0000 7ff50a3b8640 1 -- 172.21.15.125:0/665010091 shutdown_connections 2023-10-30T12:32:49.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.363+0000 7ff50a3b8640 1 -- 172.21.15.125:0/665010091 wait complete. 2023-10-30T12:32:49.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.363+0000 7ff50a3b8640 1 Processor -- start 2023-10-30T12:32:49.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.363+0000 7ff50a3b8640 1 -- start start 2023-10-30T12:32:49.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff50a3b8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff504069e90 0x7ff504143e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:49.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff50a3b8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff504070a00 0x7ff504143240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff50a3b8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5041539a0 0x7ff5040c0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff50a3b8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5040cabd0 con 0x7ff504069e90 2023-10-30T12:32:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff50a3b8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5040b0f50 con 0x7ff5041539a0 2023-10-30T12:32:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff50a3b8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff504142db0 con 0x7ff504070a00 2023-10-30T12:32:49.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff503fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff504069e90 0x7ff504143e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:49.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff503fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff504069e90 0x7ff504143e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41264/0 (socket says 172.21.15.125:41264) 2023-10-30T12:32:49.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff503fff640 1 -- 172.21.15.125:0/633457767 learned_addr learned my addr 172.21.15.125:0/633457767 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:49.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff5037fe640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff504070a00 0x7ff504143240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:49.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff508955640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5041539a0 0x7ff5040c0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:49.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff503fff640 1 -- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff504070a00 msgr2=0x7ff504143240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:49.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff503fff640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff504070a00 0x7ff504143240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:49.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff503fff640 1 -- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5041539a0 msgr2=0x7ff5040c0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:49.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff503fff640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5041539a0 0x7ff5040c0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:49.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff503fff640 1 -- 172.21.15.125:0/633457767 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc009a70 con 0x7ff504069e90 2023-10-30T12:32:49.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff5037fe640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff504070a00 0x7ff504143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:32:49.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff503fff640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff504069e90 0x7ff504143e70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400c500 tx=0x7ff4f400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:49.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff500ff9640 1 -- 172.21.15.125:0/633457767 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4005cf0 con 0x7ff504069e90 2023-10-30T12:32:49.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff500ff9640 1 -- 172.21.15.125:0/633457767 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f4005e50 con 0x7ff504069e90 2023-10-30T12:32:49.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff500ff9640 1 -- 172.21.15.125:0/633457767 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f40072f0 con 0x7ff504069e90 2023-10-30T12:32:49.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff50a3b8640 1 -- 172.21.15.125:0/633457767 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5040ca930 con 0x7ff504069e90 2023-10-30T12:32:49.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff50a3b8640 1 -- 172.21.15.125:0/633457767 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff504142a40 con 0x7ff504069e90 2023-10-30T12:32:49.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff50a3b8640 1 -- 172.21.15.125:0/633457767 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5041425c0 con 0x7ff504069e90 2023-10-30T12:32:49.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.367+0000 7ff500ff9640 1 -- 172.21.15.125:0/633457767 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4f4007450 con 0x7ff504069e90 2023-10-30T12:32:49.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.371+0000 7ff500ff9640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4f00664a0 0x7ff4f0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:49.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.371+0000 7ff500ff9640 1 -- 172.21.15.125:0/633457767 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5910+0+0 (secure 0 0 0) 0x7ff4f40834d0 con 0x7ff504069e90 2023-10-30T12:32:49.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.371+0000 7ff5037fe640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4f00664a0 0x7ff4f0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:49.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.371+0000 7ff5037fe640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4f00664a0 0x7ff4f0068960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff504067c10 tx=0x7ff4f8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:49.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.375+0000 7ff500ff9640 1 -- 172.21.15.125:0/633457767 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5041425c0 con 0x7ff504069e90 2023-10-30T12:32:49.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.555+0000 7ff50a3b8640 1 -- 172.21.15.125:0/633457767 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5041425c0 con 0x7ff504069e90 2023-10-30T12:32:49.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.555+0000 7ff500ff9640 1 -- 172.21.15.125:0/633457767 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5752 (secure 0 0 0) 0x7ff5041425c0 con 0x7ff504069e90 2023-10-30T12:32:49.559 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:49.559 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:46.178886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:49.560 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 25 2023-10-30T12:32:49.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.555+0000 7ff50a3b8640 1 -- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4f00664a0 msgr2=0x7ff4f0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:49.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.555+0000 7ff50a3b8640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4f00664a0 0x7ff4f0068960 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff504067c10 tx=0x7ff4f8008040 comp rx=0 tx=0).stop 2023-10-30T12:32:49.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.555+0000 7ff50a3b8640 1 -- 172.21.15.125:0/633457767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff504069e90 msgr2=0x7ff504143e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:49.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.555+0000 7ff50a3b8640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff504069e90 0x7ff504143e70 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400c500 tx=0x7ff4f400c9c0 comp rx=0 tx=0).stop 2023-10-30T12:32:49.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.555+0000 7ff50a3b8640 1 -- 172.21.15.125:0/633457767 shutdown_connections 2023-10-30T12:32:49.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.555+0000 7ff50a3b8640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4f00664a0 0x7ff4f0068960 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:49.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.555+0000 7ff50a3b8640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5041539a0 0x7ff5040c0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:49.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.555+0000 7ff50a3b8640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff504070a00 0x7ff504143240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:49.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.555+0000 7ff50a3b8640 1 --2- 172.21.15.125:0/633457767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff504069e90 0x7ff504143e70 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:49.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.555+0000 7ff50a3b8640 1 -- 172.21.15.125:0/633457767 >> 172.21.15.125:0/633457767 conn(0x7ff5040c7910 msgr2=0x7ff50414f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:49.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.555+0000 7ff50a3b8640 1 -- 172.21.15.125:0/633457767 shutdown_connections 2023-10-30T12:32:49.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:49.555+0000 7ff50a3b8640 1 -- 172.21.15.125:0/633457767 wait complete. 2023-10-30T12:32:49.581 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 25, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 25, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4285": { "addr": "172.21.15.136:6839/2250748765", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 2250748765, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 2250748765, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4285, "incarnation": 9, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_4294": { "addr": "172.21.15.125:6836/2017382874", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2017382874, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2017382874, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4294, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4957": { "addr": "172.21.15.136:6835/2548530941", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2548530941, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2548530941, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4957, "incarnation": 23, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:reconnect", "state_seq": 30 } }, "last_failure": 0, "last_failure_osd_epoch": 30, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:32:46.178886+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4294, "mds_1": 4285, "mds_2": 4957 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:32:51.582 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:51.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.727+0000 7fad92965640 1 Processor -- start 2023-10-30T12:32:51.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.727+0000 7fad92965640 1 -- start start 2023-10-30T12:32:51.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.731+0000 7fad92965640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad8c06f7a0 0x7fad8c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:51.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.731+0000 7fad92965640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8c066cf0 con 0x7fad8c0c6bd0 2023-10-30T12:32:51.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.731+0000 7fad92965640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad8c0c70d0 con 0x7fad8c071110 2023-10-30T12:32:51.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.731+0000 7fad92965640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad8c0700c0 con 0x7fad8c06f7a0 2023-10-30T12:32:51.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.731+0000 7fad8bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad8c06f7a0 0x7fad8c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:51.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.731+0000 7fad8bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fad8c06f7a0 0x7fad8c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41118/0 (socket says 172.21.15.125:41118) 2023-10-30T12:32:51.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.731+0000 7fad8bfff640 1 -- 172.21.15.125:0/1257454748 learned_addr learned my addr 172.21.15.125:0/1257454748 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:51.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.731+0000 7fad8bfff640 1 -- 172.21.15.125:0/1257454748 >> v1:172.21.15.136:6789/0 conn(0x7fad8c071110 legacy=0x7fad8c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:51.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.731+0000 7fad8bfff640 1 -- 172.21.15.125:0/1257454748 >> v1:172.21.15.125:6789/0 conn(0x7fad8c0c6bd0 legacy=0x7fad8c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:51.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.731+0000 7fad8bfff640 1 -- 172.21.15.125:0/1257454748 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8c0c7390 con 0x7fad8c06f7a0 2023-10-30T12:32:51.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad8bfff640 1 --2- 172.21.15.125:0/1257454748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8c06f7a0 0x7fad8c06fb80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fad7c009870 tx=0x7fad7c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acaad9db56e10a41 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:51.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad8affd640 1 -- 172.21.15.125:0/1257454748 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c0a89f0 con 0x7fad8c06f7a0 2023-10-30T12:32:51.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad8affd640 1 -- 172.21.15.125:0/1257454748 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c0a8b50 con 0x7fad8c06f7a0 2023-10-30T12:32:51.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad8affd640 1 -- 172.21.15.125:0/1257454748 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c0b0440 con 0x7fad8c06f7a0 2023-10-30T12:32:51.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad92965640 1 -- 172.21.15.125:0/1257454748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8c06f7a0 msgr2=0x7fad8c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:51.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad92965640 1 --2- 172.21.15.125:0/1257454748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8c06f7a0 0x7fad8c06fb80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fad7c009870 tx=0x7fad7c01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:32:51.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad92965640 1 -- 172.21.15.125:0/1257454748 shutdown_connections 2023-10-30T12:32:51.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad92965640 1 --2- 172.21.15.125:0/1257454748 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8c06f7a0 0x7fad8c06fb80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:51.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad92965640 1 -- 172.21.15.125:0/1257454748 >> 172.21.15.125:0/1257454748 conn(0x7fad8c0c7910 msgr2=0x7fad8c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:51.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad92965640 1 -- 172.21.15.125:0/1257454748 shutdown_connections 2023-10-30T12:32:51.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad92965640 1 -- 172.21.15.125:0/1257454748 wait complete. 2023-10-30T12:32:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad92965640 1 Processor -- start 2023-10-30T12:32:51.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad92965640 1 -- start start 2023-10-30T12:32:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad92965640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad8c06f7a0 0x7fad8c0c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.735+0000 7fad92965640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fad8c071110 0x7fad8c1524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad92965640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8c0c6bd0 0x7fad8c152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad92965640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c157070 con 0x7fad8c0c6bd0 2023-10-30T12:32:51.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad92965640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c152040 con 0x7fad8c06f7a0 2023-10-30T12:32:51.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad92965640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8c150250 con 0x7fad8c071110 2023-10-30T12:32:51.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad90f02640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8c0c6bd0 0x7fad8c152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:51.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad90f02640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8c0c6bd0 0x7fad8c152a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41124/0 (socket says 172.21.15.125:41124) 2023-10-30T12:32:51.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad90f02640 1 -- 172.21.15.125:0/3166641551 learned_addr learned my addr 172.21.15.125:0/3166641551 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:51.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad8b7fe640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fad8c071110 0x7fad8c1524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:51.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad8bfff640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad8c06f7a0 0x7fad8c0c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:51.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad90f02640 1 -- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fad8c071110 msgr2=0x7fad8c1524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:51.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad90f02640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fad8c071110 0x7fad8c1524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:51.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad90f02640 1 -- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad8c06f7a0 msgr2=0x7fad8c0c41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:51.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad90f02640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad8c06f7a0 0x7fad8c0c41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:51.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad90f02640 1 -- 172.21.15.125:0/3166641551 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7c0094d0 con 0x7fad8c0c6bd0 2023-10-30T12:32:51.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad8bfff640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad8c06f7a0 0x7fad8c0c41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:32:51.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad90f02640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8c0c6bd0 0x7fad8c152a10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fad8400b970 tx=0x7fad8400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:51.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad88ff9640 1 -- 172.21.15.125:0/3166641551 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad8400cb40 con 0x7fad8c0c6bd0 2023-10-30T12:32:51.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad88ff9640 1 -- 172.21.15.125:0/3166641551 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad8400cca0 con 0x7fad8c0c6bd0 2023-10-30T12:32:51.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad88ff9640 1 -- 172.21.15.125:0/3166641551 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad84014680 con 0x7fad8c0c6bd0 2023-10-30T12:32:51.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad92965640 1 -- 172.21.15.125:0/3166641551 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad8c1edbd0 con 0x7fad8c0c6bd0 2023-10-30T12:32:51.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad92965640 1 -- 172.21.15.125:0/3166641551 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad8c1ed330 con 0x7fad8c0c6bd0 2023-10-30T12:32:51.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad92965640 1 -- 172.21.15.125:0/3166641551 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad8c1521f0 con 0x7fad8c0c6bd0 2023-10-30T12:32:51.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.739+0000 7fad88ff9640 1 -- 172.21.15.125:0/3166641551 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fad8401d3b0 con 0x7fad8c0c6bd0 2023-10-30T12:32:51.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.743+0000 7fad88ff9640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fad780664a0 0x7fad78068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:51.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.743+0000 7fad88ff9640 1 -- 172.21.15.125:0/3166641551 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fad84083d80 con 0x7fad8c0c6bd0 2023-10-30T12:32:51.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.743+0000 7fad8bfff640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fad780664a0 0x7fad78068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:51.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.743+0000 7fad8bfff640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fad780664a0 0x7fad78068960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0b2000 tx=0x7fad7c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:51.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.747+0000 7fad88ff9640 1 -- 172.21.15.125:0/3166641551 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fad8c1521f0 con 0x7fad8c0c6bd0 2023-10-30T12:32:51.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.923+0000 7fad92965640 1 -- 172.21.15.125:0/3166641551 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad8c1521b0 con 0x7fad8c0c6bd0 2023-10-30T12:32:51.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.927+0000 7fad88ff9640 1 -- 172.21.15.125:0/3166641551 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+5752 (secure 0 0 0) 0x7fad840510b0 con 0x7fad8c0c6bd0 2023-10-30T12:32:51.929 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:51.929 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:46.178886+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:reconnect","state_seq":30,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:51.930 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 25 2023-10-30T12:32:51.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.927+0000 7fad92965640 1 -- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fad780664a0 msgr2=0x7fad78068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:51.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.927+0000 7fad92965640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fad780664a0 0x7fad78068960 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0b2000 tx=0x7fad7c008040 comp rx=0 tx=0).stop 2023-10-30T12:32:51.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.927+0000 7fad92965640 1 -- 172.21.15.125:0/3166641551 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8c0c6bd0 msgr2=0x7fad8c152a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:51.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.927+0000 7fad92965640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8c0c6bd0 0x7fad8c152a10 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fad8400b970 tx=0x7fad8400be30 comp rx=0 tx=0).stop 2023-10-30T12:32:51.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.927+0000 7fad92965640 1 -- 172.21.15.125:0/3166641551 shutdown_connections 2023-10-30T12:32:51.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.927+0000 7fad92965640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fad780664a0 0x7fad78068960 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:51.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.927+0000 7fad92965640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fad8c0c6bd0 0x7fad8c152a10 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:51.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.927+0000 7fad92965640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fad8c071110 0x7fad8c1524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:51.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.927+0000 7fad92965640 1 --2- 172.21.15.125:0/3166641551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fad8c06f7a0 0x7fad8c0c41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:51.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.927+0000 7fad92965640 1 -- 172.21.15.125:0/3166641551 >> 172.21.15.125:0/3166641551 conn(0x7fad8c0c7910 msgr2=0x7fad8c0ca0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:51.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.927+0000 7fad92965640 1 -- 172.21.15.125:0/3166641551 shutdown_connections 2023-10-30T12:32:51.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:51.927+0000 7fad92965640 1 -- 172.21.15.125:0/3166641551 wait complete. 2023-10-30T12:32:51.951 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:32:53.952 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:54.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.095+0000 7f2d3ab65640 1 Processor -- start 2023-10-30T12:32:54.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.095+0000 7f2d3ab65640 1 -- start start 2023-10-30T12:32:54.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d3ab65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d34153bf0 0x7f2d34071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:54.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d3ab65640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d340700b0 con 0x7f2d340c5f60 2023-10-30T12:32:54.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d3ab65640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d340c6530 con 0x7f2d340c6fe0 2023-10-30T12:32:54.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d3ab65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d340c6810 con 0x7f2d34153bf0 2023-10-30T12:32:54.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d39102640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2d340c5f60 0x7f2d340cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54932/0 (socket says 172.21.15.125:54932) 2023-10-30T12:32:54.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d39102640 1 -- 172.21.15.125:0/2137774592 learned_addr learned my addr 172.21.15.125:0/2137774592 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:32:54.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d38901640 1 --2- 172.21.15.125:0/2137774592 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d34153bf0 0x7f2d34071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:54.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d2b7fe640 1 -- 172.21.15.125:0/2137774592 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279899476 0 0) 0x7f2d340700b0 con 0x7f2d340c5f60 2023-10-30T12:32:54.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d2b7fe640 1 -- 172.21.15.125:0/2137774592 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d1c003530 con 0x7f2d340c5f60 2023-10-30T12:32:54.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d2b7fe640 1 -- 172.21.15.125:0/2137774592 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922554700 0 0) 0x7f2d340c6530 con 0x7f2d340c6fe0 2023-10-30T12:32:54.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d2b7fe640 1 -- 172.21.15.125:0/2137774592 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d340700b0 con 0x7f2d340c6fe0 2023-10-30T12:32:54.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d2b7fe640 1 -- 172.21.15.125:0/2137774592 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4068666251 0 0) 0x7f2d1c003530 con 0x7f2d340c5f60 2023-10-30T12:32:54.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d38901640 1 -- 172.21.15.125:0/2137774592 >> v1:172.21.15.136:6789/0 conn(0x7f2d340c6fe0 legacy=0x7f2d340c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:54.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d38901640 1 -- 172.21.15.125:0/2137774592 >> v1:172.21.15.125:6789/0 conn(0x7f2d340c5f60 legacy=0x7f2d340cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:54.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.099+0000 7f2d38901640 1 -- 172.21.15.125:0/2137774592 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d34159b10 con 0x7f2d34153bf0 2023-10-30T12:32:54.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.103+0000 7f2d38901640 1 --2- 172.21.15.125:0/2137774592 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d34153bf0 0x7f2d34071260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d300097f0 tx=0x7f2d3001ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bffd6bac2d78a7e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:54.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.103+0000 7f2d2b7fe640 1 -- 172.21.15.125:0/2137774592 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d2c001a00 con 0x7f2d34153bf0 2023-10-30T12:32:54.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.103+0000 7f2d2b7fe640 1 -- 172.21.15.125:0/2137774592 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d300a8650 con 0x7f2d34153bf0 2023-10-30T12:32:54.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.103+0000 7f2d2b7fe640 1 -- 172.21.15.125:0/2137774592 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d300a89e0 con 0x7f2d34153bf0 2023-10-30T12:32:54.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.103+0000 7f2d3ab65640 1 -- 172.21.15.125:0/2137774592 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d34153bf0 msgr2=0x7f2d34071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:54.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.103+0000 7f2d3ab65640 1 --2- 172.21.15.125:0/2137774592 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d34153bf0 0x7f2d34071260 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2d300097f0 tx=0x7f2d3001ebd0 comp rx=0 tx=0).stop 2023-10-30T12:32:54.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.103+0000 7f2d3ab65640 1 -- 172.21.15.125:0/2137774592 shutdown_connections 2023-10-30T12:32:54.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.103+0000 7f2d3ab65640 1 --2- 172.21.15.125:0/2137774592 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d34153bf0 0x7f2d34071260 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:54.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.103+0000 7f2d3ab65640 1 -- 172.21.15.125:0/2137774592 >> 172.21.15.125:0/2137774592 conn(0x7f2d3406c6e0 msgr2=0x7f2d34069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:54.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.103+0000 7f2d3ab65640 1 -- 172.21.15.125:0/2137774592 shutdown_connections 2023-10-30T12:32:54.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.103+0000 7f2d3ab65640 1 -- 172.21.15.125:0/2137774592 wait complete. 2023-10-30T12:32:54.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.103+0000 7f2d3ab65640 1 Processor -- start 2023-10-30T12:32:54.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.103+0000 7f2d3ab65640 1 -- start start 2023-10-30T12:32:54.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d3ab65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d340c5f60 0x7f2d34156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:54.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d3ab65640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d340c6fe0 0x7f2d34158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:54.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d3ab65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d34153bf0 0x7f2d341559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:54.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d3ab65640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d34155560 con 0x7f2d340c5f60 2023-10-30T12:32:54.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d3ab65640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d34154ca0 con 0x7f2d340c6fe0 2023-10-30T12:32:54.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d3ab65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d341efad0 con 0x7f2d34153bf0 2023-10-30T12:32:54.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d38901640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d340c5f60 0x7f2d34156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:54.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d38901640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d340c5f60 0x7f2d34156f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41134/0 (socket says 172.21.15.125:41134) 2023-10-30T12:32:54.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d38901640 1 -- 172.21.15.125:0/3940382894 learned_addr learned my addr 172.21.15.125:0/3940382894 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:54.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d39102640 1 --2- 172.21.15.125:0/3940382894 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d34153bf0 0x7f2d341559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:54.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d2bfff640 1 --2- 172.21.15.125:0/3940382894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d340c6fe0 0x7f2d34158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:54.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d38901640 1 -- 172.21.15.125:0/3940382894 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d34153bf0 msgr2=0x7f2d341559f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:54.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d38901640 1 --2- 172.21.15.125:0/3940382894 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d34153bf0 0x7f2d341559f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:54.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d38901640 1 -- 172.21.15.125:0/3940382894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d340c6fe0 msgr2=0x7f2d34158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:54.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d38901640 1 --2- 172.21.15.125:0/3940382894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d340c6fe0 0x7f2d34158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:54.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d38901640 1 -- 172.21.15.125:0/3940382894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d30009420 con 0x7f2d340c5f60 2023-10-30T12:32:54.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d38901640 1 --2- 172.21.15.125:0/3940382894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d340c5f60 0x7f2d34156f40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2d30009740 tx=0x7f2d3001b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:54.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d297fa640 1 -- 172.21.15.125:0/3940382894 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d300bf070 con 0x7f2d340c5f60 2023-10-30T12:32:54.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d297fa640 1 -- 172.21.15.125:0/3940382894 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d30006e20 con 0x7f2d340c5f60 2023-10-30T12:32:54.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d297fa640 1 -- 172.21.15.125:0/3940382894 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d300b0870 con 0x7f2d340c5f60 2023-10-30T12:32:54.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d3ab65640 1 -- 172.21.15.125:0/3940382894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d341efdb0 con 0x7f2d340c5f60 2023-10-30T12:32:54.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d3ab65640 1 -- 172.21.15.125:0/3940382894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d341551f0 con 0x7f2d340c5f60 2023-10-30T12:32:54.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.107+0000 7f2d3ab65640 1 -- 172.21.15.125:0/3940382894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d3404e170 con 0x7f2d340c5f60 2023-10-30T12:32:54.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.111+0000 7f2d297fa640 1 -- 172.21.15.125:0/3940382894 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2d300b7040 con 0x7f2d340c5f60 2023-10-30T12:32:54.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.111+0000 7f2d297fa640 1 --2- 172.21.15.125:0/3940382894 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2d200664a0 0x7f2d20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:54.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.111+0000 7f2d297fa640 1 -- 172.21.15.125:0/3940382894 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f2d30120d50 con 0x7f2d340c5f60 2023-10-30T12:32:54.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.111+0000 7f2d2bfff640 1 --2- 172.21.15.125:0/3940382894 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2d200664a0 0x7f2d20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:54.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.111+0000 7f2d2bfff640 1 --2- 172.21.15.125:0/3940382894 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2d200664a0 0x7f2d20068960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2d34156cc0 tx=0x7f2d24009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:54.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.115+0000 7f2d297fa640 1 -- 172.21.15.125:0/3940382894 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2d300edfe0 con 0x7f2d340c5f60 2023-10-30T12:32:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.295+0000 7f2d3ab65640 1 -- 172.21.15.125:0/3940382894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d3406fe90 con 0x7f2d340c5f60 2023-10-30T12:32:54.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.295+0000 7f2d297fa640 1 -- 172.21.15.125:0/3940382894 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5749 (secure 0 0 0) 0x7f2d300b5090 con 0x7f2d340c5f60 2023-10-30T12:32:54.300 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:54.300 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:52.185012+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:54.300 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 26 2023-10-30T12:32:54.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.299+0000 7f2d3ab65640 1 -- 172.21.15.125:0/3940382894 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2d200664a0 msgr2=0x7f2d20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:54.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.299+0000 7f2d3ab65640 1 --2- 172.21.15.125:0/3940382894 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2d200664a0 0x7f2d20068960 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2d34156cc0 tx=0x7f2d24009040 comp rx=0 tx=0).stop 2023-10-30T12:32:54.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.299+0000 7f2d3ab65640 1 -- 172.21.15.125:0/3940382894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d340c5f60 msgr2=0x7f2d34156f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:54.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.299+0000 7f2d3ab65640 1 --2- 172.21.15.125:0/3940382894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d340c5f60 0x7f2d34156f40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f2d30009740 tx=0x7f2d3001b970 comp rx=0 tx=0).stop 2023-10-30T12:32:54.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.299+0000 7f2d3ab65640 1 -- 172.21.15.125:0/3940382894 shutdown_connections 2023-10-30T12:32:54.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.299+0000 7f2d3ab65640 1 --2- 172.21.15.125:0/3940382894 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2d200664a0 0x7f2d20068960 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:54.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.299+0000 7f2d3ab65640 1 --2- 172.21.15.125:0/3940382894 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2d34153bf0 0x7f2d341559f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:54.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.299+0000 7f2d3ab65640 1 --2- 172.21.15.125:0/3940382894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2d340c6fe0 0x7f2d34158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:54.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.299+0000 7f2d3ab65640 1 --2- 172.21.15.125:0/3940382894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2d340c5f60 0x7f2d34156f40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:54.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.299+0000 7f2d3ab65640 1 -- 172.21.15.125:0/3940382894 >> 172.21.15.125:0/3940382894 conn(0x7f2d3406c6e0 msgr2=0x7f2d34152ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:54.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.299+0000 7f2d3ab65640 1 -- 172.21.15.125:0/3940382894 shutdown_connections 2023-10-30T12:32:54.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:54.299+0000 7f2d3ab65640 1 -- 172.21.15.125:0/3940382894 wait complete. 2023-10-30T12:32:54.320 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:32:56.321 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:56.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.463+0000 7fed75b3e640 1 Processor -- start 2023-10-30T12:32:56.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.463+0000 7fed75b3e640 1 -- start start 2023-10-30T12:32:56.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed75b3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed700c6b80 0x7fed700bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:56.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed75b3e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed700c0a50 con 0x7fed7013f4a0 2023-10-30T12:32:56.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed75b3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7006f6e0 con 0x7fed700c6f60 2023-10-30T12:32:56.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed75b3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7006f9c0 con 0x7fed700c6b80 2023-10-30T12:32:56.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6ffff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fed7013f4a0 0x7fed701566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54942/0 (socket says 172.21.15.125:54942) 2023-10-30T12:32:56.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6ffff640 1 -- 172.21.15.125:0/1845864633 learned_addr learned my addr 172.21.15.125:0/1845864633 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:32:56.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6f7fe640 1 --2- 172.21.15.125:0/1845864633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed700c6b80 0x7fed700bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:56.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 -- 172.21.15.125:0/1845864633 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3140781830 0 0) 0x7fed700c0a50 con 0x7fed7013f4a0 2023-10-30T12:32:56.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 -- 172.21.15.125:0/1845864633 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed5c005740 con 0x7fed7013f4a0 2023-10-30T12:32:56.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 -- 172.21.15.125:0/1845864633 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3429584390 0 0) 0x7fed5c005740 con 0x7fed7013f4a0 2023-10-30T12:32:56.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 -- 172.21.15.125:0/1845864633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed700c6b80 msgr2=0x7fed700bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:56.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 --2- 172.21.15.125:0/1845864633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed700c6b80 0x7fed700bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:56.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 -- 172.21.15.125:0/1845864633 >> v1:172.21.15.136:6789/0 conn(0x7fed700c6f60 legacy=0x7fed7013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:56.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 -- 172.21.15.125:0/1845864633 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7006fca0 con 0x7fed7013f4a0 2023-10-30T12:32:56.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 -- 172.21.15.125:0/1845864633 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fed68001a00 con 0x7fed7013f4a0 2023-10-30T12:32:56.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 -- 172.21.15.125:0/1845864633 >> v1:172.21.15.125:6789/0 conn(0x7fed7013f4a0 legacy=0x7fed701566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:56.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 --2- 172.21.15.125:0/1845864633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed5c005bc0 0x7fed5c005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:56.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 --2- 172.21.15.125:0/1845864633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed700c6b80 0x7fed5c0069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:56.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 --2- 172.21.15.125:0/1845864633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed5c006f20 0x7fed5c0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:56.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 -- 172.21.15.125:0/1845864633 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7006f9c0 con 0x7fed5c006f20 2023-10-30T12:32:56.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 -- 172.21.15.125:0/1845864633 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c00dc50 con 0x7fed5c005bc0 2023-10-30T12:32:56.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6effd640 1 -- 172.21.15.125:0/1845864633 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c00de10 con 0x7fed700c6b80 2023-10-30T12:32:56.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6ffff640 1 --2- 172.21.15.125:0/1845864633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed5c006f20 0x7fed5c0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:56.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed67fff640 1 --2- 172.21.15.125:0/1845864633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed700c6b80 0x7fed5c0069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:56.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6f7fe640 1 --2- 172.21.15.125:0/1845864633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed5c005bc0 0x7fed5c005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:56.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6ffff640 1 -- 172.21.15.125:0/1845864633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed700c6b80 msgr2=0x7fed5c0069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:56.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6ffff640 1 --2- 172.21.15.125:0/1845864633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed700c6b80 0x7fed5c0069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:56.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6ffff640 1 -- 172.21.15.125:0/1845864633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed5c005bc0 msgr2=0x7fed5c005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:56.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6ffff640 1 --2- 172.21.15.125:0/1845864633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed5c005bc0 0x7fed5c005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:56.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.467+0000 7fed6ffff640 1 -- 172.21.15.125:0/1845864633 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c005740 con 0x7fed5c006f20 2023-10-30T12:32:56.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.471+0000 7fed6ffff640 1 --2- 172.21.15.125:0/1845864633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed5c006f20 0x7fed5c0073a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fed7013f8d0 tx=0x7fed6801d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48481772c8d7af56 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:56.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.471+0000 7fed6effd640 1 -- 172.21.15.125:0/1845864633 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed680072d0 con 0x7fed5c006f20 2023-10-30T12:32:56.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.471+0000 7fed6effd640 1 -- 172.21.15.125:0/1845864633 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed680a8070 con 0x7fed5c006f20 2023-10-30T12:32:56.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.471+0000 7fed6effd640 1 -- 172.21.15.125:0/1845864633 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed680076b0 con 0x7fed5c006f20 2023-10-30T12:32:56.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.471+0000 7fed75b3e640 1 -- 172.21.15.125:0/1845864633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed5c006f20 msgr2=0x7fed5c0073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:56.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.471+0000 7fed75b3e640 1 --2- 172.21.15.125:0/1845864633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed5c006f20 0x7fed5c0073a0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fed7013f8d0 tx=0x7fed6801d2a0 comp rx=0 tx=0).stop 2023-10-30T12:32:56.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.471+0000 7fed6ffff640 1 -- 172.21.15.125:0/1845864633 reap_dead start 2023-10-30T12:32:56.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.471+0000 7fed75b3e640 1 -- 172.21.15.125:0/1845864633 shutdown_connections 2023-10-30T12:32:56.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.471+0000 7fed75b3e640 1 -- 172.21.15.125:0/1845864633 >> 172.21.15.125:0/1845864633 conn(0x7fed700c7910 msgr2=0x7fed700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:56.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.471+0000 7fed75b3e640 1 -- 172.21.15.125:0/1845864633 shutdown_connections 2023-10-30T12:32:56.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.471+0000 7fed75b3e640 1 -- 172.21.15.125:0/1845864633 wait complete. 2023-10-30T12:32:56.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed75b3e640 1 Processor -- start 2023-10-30T12:32:56.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed75b3e640 1 -- start start 2023-10-30T12:32:56.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed75b3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed5c006f20 0x7fed70159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:56.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed75b3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed70069e90 0x7fed701544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:56.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed75b3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed701edfe0 0x7fed701ee450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:56.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed75b3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed701eb630 con 0x7fed70069e90 2023-10-30T12:32:56.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed75b3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed701ee990 con 0x7fed701edfe0 2023-10-30T12:32:56.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed75b3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed70154dc0 con 0x7fed5c006f20 2023-10-30T12:32:56.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed67fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed70069e90 0x7fed701544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:56.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed67fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed70069e90 0x7fed701544a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41152/0 (socket says 172.21.15.125:41152) 2023-10-30T12:32:56.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed67fff640 1 -- 172.21.15.125:0/3129194011 learned_addr learned my addr 172.21.15.125:0/3129194011 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:56.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed6f7fe640 1 --2- 172.21.15.125:0/3129194011 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed5c006f20 0x7fed70159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:56.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed6ffff640 1 --2- 172.21.15.125:0/3129194011 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed701edfe0 0x7fed701ee450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:56.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed67fff640 1 -- 172.21.15.125:0/3129194011 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed5c006f20 msgr2=0x7fed70159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:56.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed67fff640 1 --2- 172.21.15.125:0/3129194011 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed5c006f20 0x7fed70159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:56.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed67fff640 1 -- 172.21.15.125:0/3129194011 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed701edfe0 msgr2=0x7fed701ee450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:56.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed67fff640 1 --2- 172.21.15.125:0/3129194011 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed701edfe0 0x7fed701ee450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:56.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed67fff640 1 -- 172.21.15.125:0/3129194011 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed701eaa20 con 0x7fed70069e90 2023-10-30T12:32:56.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed67fff640 1 --2- 172.21.15.125:0/3129194011 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed70069e90 0x7fed701544a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fed5800f850 tx=0x7fed5800fd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:56.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed6cff9640 1 -- 172.21.15.125:0/3129194011 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed58001350 con 0x7fed70069e90 2023-10-30T12:32:56.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed6cff9640 1 -- 172.21.15.125:0/3129194011 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed58001d10 con 0x7fed70069e90 2023-10-30T12:32:56.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed75b3e640 1 -- 172.21.15.125:0/3129194011 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed701ead00 con 0x7fed70069e90 2023-10-30T12:32:56.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed6cff9640 1 -- 172.21.15.125:0/3129194011 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5800d730 con 0x7fed70069e90 2023-10-30T12:32:56.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed75b3e640 1 -- 172.21.15.125:0/3129194011 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed701eb2c0 con 0x7fed70069e90 2023-10-30T12:32:56.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.475+0000 7fed75b3e640 1 -- 172.21.15.125:0/3129194011 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed700c7710 con 0x7fed70069e90 2023-10-30T12:32:56.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.479+0000 7fed6cff9640 1 -- 172.21.15.125:0/3129194011 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fed5800d890 con 0x7fed70069e90 2023-10-30T12:32:56.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.479+0000 7fed6cff9640 1 --2- 172.21.15.125:0/3129194011 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed50066470 0x7fed50068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:56.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.479+0000 7fed6cff9640 1 -- 172.21.15.125:0/3129194011 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fed58083ef0 con 0x7fed70069e90 2023-10-30T12:32:56.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.479+0000 7fed6f7fe640 1 --2- 172.21.15.125:0/3129194011 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed50066470 0x7fed50068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.479+0000 7fed6f7fe640 1 --2- 172.21.15.125:0/3129194011 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed50066470 0x7fed50068930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed60009b20 tx=0x7fed60008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:56.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.483+0000 7fed6cff9640 1 -- 172.21.15.125:0/3129194011 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fed580512a0 con 0x7fed70069e90 2023-10-30T12:32:56.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.663+0000 7fed75b3e640 1 -- 172.21.15.125:0/3129194011 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed701eb450 con 0x7fed70069e90 2023-10-30T12:32:56.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.667+0000 7fed6cff9640 1 -- 172.21.15.125:0/3129194011 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+5749 (secure 0 0 0) 0x7fed580548c0 con 0x7fed70069e90 2023-10-30T12:32:56.669 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:56.670 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:52.185012+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:56.671 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 26 2023-10-30T12:32:56.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.667+0000 7fed75b3e640 1 -- 172.21.15.125:0/3129194011 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed50066470 msgr2=0x7fed50068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:56.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.667+0000 7fed75b3e640 1 --2- 172.21.15.125:0/3129194011 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed50066470 0x7fed50068930 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed60009b20 tx=0x7fed60008040 comp rx=0 tx=0).stop 2023-10-30T12:32:56.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.667+0000 7fed75b3e640 1 -- 172.21.15.125:0/3129194011 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed70069e90 msgr2=0x7fed701544a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:56.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.667+0000 7fed75b3e640 1 --2- 172.21.15.125:0/3129194011 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed70069e90 0x7fed701544a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fed5800f850 tx=0x7fed5800fd10 comp rx=0 tx=0).stop 2023-10-30T12:32:56.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.667+0000 7fed75b3e640 1 -- 172.21.15.125:0/3129194011 shutdown_connections 2023-10-30T12:32:56.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.667+0000 7fed75b3e640 1 --2- 172.21.15.125:0/3129194011 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed50066470 0x7fed50068930 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:56.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.667+0000 7fed75b3e640 1 --2- 172.21.15.125:0/3129194011 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed701edfe0 0x7fed701ee450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:56.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.667+0000 7fed75b3e640 1 --2- 172.21.15.125:0/3129194011 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed70069e90 0x7fed701544a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:56.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.667+0000 7fed75b3e640 1 --2- 172.21.15.125:0/3129194011 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed5c006f20 0x7fed70159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:56.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.667+0000 7fed75b3e640 1 -- 172.21.15.125:0/3129194011 >> 172.21.15.125:0/3129194011 conn(0x7fed700c7910 msgr2=0x7fed7013b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:56.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.667+0000 7fed75b3e640 1 -- 172.21.15.125:0/3129194011 shutdown_connections 2023-10-30T12:32:56.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:56.667+0000 7fed75b3e640 1 -- 172.21.15.125:0/3129194011 wait complete. 2023-10-30T12:32:56.690 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:32:58.691 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:32:58.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.835+0000 7fb11cc68640 1 Processor -- start 2023-10-30T12:32:58.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.835+0000 7fb11cc68640 1 -- start start 2023-10-30T12:32:58.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.835+0000 7fb11cc68640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb118153ca0 0x7fb1180c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:58.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.835+0000 7fb11cc68640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb1180c0a40 0x7fb1180c0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:58.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.835+0000 7fb11cc68640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1181598f0 con 0x7fb1180c7510 2023-10-30T12:32:58.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.835+0000 7fb11cc68640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1180c1410 con 0x7fb1180c0a40 2023-10-30T12:32:58.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.835+0000 7fb11cc68640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb118159b60 con 0x7fb118153ca0 2023-10-30T12:32:58.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.835+0000 7fb116d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb1180c0a40 0x7fb1180c0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:58.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.835+0000 7fb116d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb1180c0a40 0x7fb1180c0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41164/0 (socket says 172.21.15.125:41164) 2023-10-30T12:32:58.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.835+0000 7fb116d9d640 1 -- 172.21.15.125:0/2011686817 learned_addr learned my addr 172.21.15.125:0/2011686817 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:58.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.835+0000 7fb11659c640 1 --2- 172.21.15.125:0/2011686817 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb118153ca0 0x7fb1180c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:58.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.839+0000 7fb116d9d640 1 -- 172.21.15.125:0/2011686817 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb118153ca0 msgr2=0x7fb1180c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:32:58.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.839+0000 7fb116d9d640 1 --2- 172.21.15.125:0/2011686817 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb118153ca0 0x7fb1180c5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:58.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.839+0000 7fb116d9d640 1 -- 172.21.15.125:0/2011686817 >> v1:172.21.15.136:6789/0 conn(0x7fb1180c7510 legacy=0x7fb1180c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:58.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.839+0000 7fb116d9d640 1 -- 172.21.15.125:0/2011686817 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb118159e40 con 0x7fb1180c0a40 2023-10-30T12:32:58.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.839+0000 7fb116d9d640 1 --2- 172.21.15.125:0/2011686817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1180c0a40 0x7fb1180c0ed0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb110009e50 tx=0x7fb11001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64fd2360ba83466 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:58.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.839+0000 7fb115d9b640 1 -- 172.21.15.125:0/2011686817 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1100af400 con 0x7fb1180c0a40 2023-10-30T12:32:58.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.839+0000 7fb115d9b640 1 -- 172.21.15.125:0/2011686817 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1100ab050 con 0x7fb1180c0a40 2023-10-30T12:32:58.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.839+0000 7fb115d9b640 1 -- 172.21.15.125:0/2011686817 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1100af780 con 0x7fb1180c0a40 2023-10-30T12:32:58.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.839+0000 7fb11cc68640 1 -- 172.21.15.125:0/2011686817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1180c0a40 msgr2=0x7fb1180c0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:58.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.839+0000 7fb11cc68640 1 --2- 172.21.15.125:0/2011686817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1180c0a40 0x7fb1180c0ed0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fb110009e50 tx=0x7fb11001f1f0 comp rx=0 tx=0).stop 2023-10-30T12:32:58.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11cc68640 1 -- 172.21.15.125:0/2011686817 shutdown_connections 2023-10-30T12:32:58.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11cc68640 1 --2- 172.21.15.125:0/2011686817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1180c0a40 0x7fb1180c0ed0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:58.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11cc68640 1 --2- 172.21.15.125:0/2011686817 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb118153ca0 0x7fb1180c5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:58.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11cc68640 1 -- 172.21.15.125:0/2011686817 >> 172.21.15.125:0/2011686817 conn(0x7fb11806d770 msgr2=0x7fb11806db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:58.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11cc68640 1 -- 172.21.15.125:0/2011686817 shutdown_connections 2023-10-30T12:32:58.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11cc68640 1 -- 172.21.15.125:0/2011686817 wait complete. 2023-10-30T12:32:58.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11cc68640 1 Processor -- start 2023-10-30T12:32:58.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11cc68640 1 -- start start 2023-10-30T12:32:58.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11cc68640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1180c0a40 0x7fb1180c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:58.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11cc68640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1180c7510 0x7fb1181efb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:58.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11cc68640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb118153ca0 0x7fb1181f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:58.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11cc68640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1180c48f0 con 0x7fb1180c7510 2023-10-30T12:32:58.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11cc68640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1181f0070 con 0x7fb118153ca0 2023-10-30T12:32:58.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11cc68640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1180c4400 con 0x7fb1180c0a40 2023-10-30T12:32:58.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb10dd9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1180c7510 0x7fb1181efb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:58.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb10dd9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1180c7510 0x7fb1181efb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41166/0 (socket says 172.21.15.125:41166) 2023-10-30T12:32:58.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb10dd9b640 1 -- 172.21.15.125:0/2155161344 learned_addr learned my addr 172.21.15.125:0/2155161344 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:32:58.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb11659c640 1 --2- 172.21.15.125:0/2155161344 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1180c0a40 0x7fb1180c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:58.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb116d9d640 1 --2- 172.21.15.125:0/2155161344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb118153ca0 0x7fb1181f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:58.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb10dd9b640 1 -- 172.21.15.125:0/2155161344 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1180c0a40 msgr2=0x7fb1180c4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:58.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb10dd9b640 1 --2- 172.21.15.125:0/2155161344 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1180c0a40 0x7fb1180c4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:58.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb10dd9b640 1 -- 172.21.15.125:0/2155161344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb118153ca0 msgr2=0x7fb1181f2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:58.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb10dd9b640 1 --2- 172.21.15.125:0/2155161344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb118153ca0 0x7fb1181f2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:58.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.843+0000 7fb10dd9b640 1 -- 172.21.15.125:0/2155161344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb110009b30 con 0x7fb1180c7510 2023-10-30T12:32:58.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.847+0000 7fb10dd9b640 1 --2- 172.21.15.125:0/2155161344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1180c7510 0x7fb1181efb30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb10000c450 tx=0x7fb10000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:58.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.847+0000 7fb10f7fe640 1 -- 172.21.15.125:0/2155161344 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1000052d0 con 0x7fb1180c7510 2023-10-30T12:32:58.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.847+0000 7fb10f7fe640 1 -- 172.21.15.125:0/2155161344 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb100005430 con 0x7fb1180c7510 2023-10-30T12:32:58.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.847+0000 7fb10f7fe640 1 -- 172.21.15.125:0/2155161344 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb100007260 con 0x7fb1180c7510 2023-10-30T12:32:58.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.847+0000 7fb11cc68640 1 -- 172.21.15.125:0/2155161344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1181458f0 con 0x7fb1180c7510 2023-10-30T12:32:58.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.847+0000 7fb11cc68640 1 -- 172.21.15.125:0/2155161344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1180ae2e0 con 0x7fb1180c7510 2023-10-30T12:32:58.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.847+0000 7fb11cc68640 1 -- 172.21.15.125:0/2155161344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb118074250 con 0x7fb1180c7510 2023-10-30T12:32:58.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.847+0000 7fb10f7fe640 1 -- 172.21.15.125:0/2155161344 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb1000073c0 con 0x7fb1180c7510 2023-10-30T12:32:58.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.847+0000 7fb10f7fe640 1 --2- 172.21.15.125:0/2155161344 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0f80664d0 0x7fb0f8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:32:58.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.847+0000 7fb10f7fe640 1 -- 172.21.15.125:0/2155161344 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5910+0+0 (secure 0 0 0) 0x7fb1000836c0 con 0x7fb1180c7510 2023-10-30T12:32:58.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.851+0000 7fb11659c640 1 --2- 172.21.15.125:0/2155161344 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0f80664d0 0x7fb0f8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:32:58.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.851+0000 7fb11659c640 1 --2- 172.21.15.125:0/2155161344 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0f80664d0 0x7fb0f8068990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb108009b60 tx=0x7fb108009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:32:58.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:58.851+0000 7fb10f7fe640 1 -- 172.21.15.125:0/2155161344 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb100050b00 con 0x7fb1180c7510 2023-10-30T12:32:59.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:59.031+0000 7fb11cc68640 1 -- 172.21.15.125:0/2155161344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1181f34a0 con 0x7fb1180c7510 2023-10-30T12:32:59.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:59.035+0000 7fb10f7fe640 1 -- 172.21.15.125:0/2155161344 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5749 (secure 0 0 0) 0x7fb100054120 con 0x7fb1180c7510 2023-10-30T12:32:59.037 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:32:59.037 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:57.239180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:32:59.039 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 27 2023-10-30T12:32:59.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:59.035+0000 7fb11cc68640 1 -- 172.21.15.125:0/2155161344 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0f80664d0 msgr2=0x7fb0f8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:59.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:59.035+0000 7fb11cc68640 1 --2- 172.21.15.125:0/2155161344 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0f80664d0 0x7fb0f8068990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb108009b60 tx=0x7fb108009190 comp rx=0 tx=0).stop 2023-10-30T12:32:59.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:59.035+0000 7fb11cc68640 1 -- 172.21.15.125:0/2155161344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1180c7510 msgr2=0x7fb1181efb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:32:59.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:59.035+0000 7fb11cc68640 1 --2- 172.21.15.125:0/2155161344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1180c7510 0x7fb1181efb30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb10000c450 tx=0x7fb10000c910 comp rx=0 tx=0).stop 2023-10-30T12:32:59.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:59.035+0000 7fb11cc68640 1 -- 172.21.15.125:0/2155161344 shutdown_connections 2023-10-30T12:32:59.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:59.035+0000 7fb11cc68640 1 --2- 172.21.15.125:0/2155161344 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0f80664d0 0x7fb0f8068990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:59.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:59.035+0000 7fb11cc68640 1 --2- 172.21.15.125:0/2155161344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb118153ca0 0x7fb1181f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:59.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:59.035+0000 7fb11cc68640 1 --2- 172.21.15.125:0/2155161344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1180c7510 0x7fb1181efb30 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:59.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:59.035+0000 7fb11cc68640 1 --2- 172.21.15.125:0/2155161344 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1180c0a40 0x7fb1180c4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:32:59.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:59.035+0000 7fb11cc68640 1 -- 172.21.15.125:0/2155161344 >> 172.21.15.125:0/2155161344 conn(0x7fb11806d770 msgr2=0x7fb118158390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:32:59.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:59.035+0000 7fb11cc68640 1 -- 172.21.15.125:0/2155161344 shutdown_connections 2023-10-30T12:32:59.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:32:59.035+0000 7fb11cc68640 1 -- 172.21.15.125:0/2155161344 wait complete. 2023-10-30T12:32:59.058 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=2, replacing gid=4265 2023-10-30T12:32:59.059 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.c 2023-10-30T12:32:59.400 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:32:59.399+0000 7f6a6fdf8640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T12:33:03.320 INFO:tasks.workunit.client.0.smithi125.stdout:./136/.snap/snap-subdir-test 2023-10-30T12:33:03.325 INFO:tasks.workunit.client.0.smithi125.stdout:./270/.snap/snap-subdir-test 2023-10-30T12:33:03.330 INFO:tasks.workunit.client.0.smithi125.stdout:./125/.snap/snap-subdir-test 2023-10-30T12:33:03.335 INFO:tasks.workunit.client.0.smithi125.stdout:./194/.snap/snap-subdir-test 2023-10-30T12:33:03.340 INFO:tasks.workunit.client.0.smithi125.stdout:./119/.snap/snap-subdir-test 2023-10-30T12:33:03.346 INFO:tasks.workunit.client.0.smithi125.stdout:./297/.snap/snap-subdir-test 2023-10-30T12:33:03.351 INFO:tasks.workunit.client.0.smithi125.stdout:./230/.snap/snap-subdir-test 2023-10-30T12:33:03.357 INFO:tasks.workunit.client.0.smithi125.stdout:./260/.snap/snap-subdir-test 2023-10-30T12:33:03.362 INFO:tasks.workunit.client.0.smithi125.stdout:./157/.snap/snap-subdir-test 2023-10-30T12:33:03.368 INFO:tasks.workunit.client.0.smithi125.stdout:./172/.snap/snap-subdir-test 2023-10-30T12:33:03.372 INFO:tasks.workunit.client.0.smithi125.stdout:./277/.snap/snap-subdir-test 2023-10-30T12:33:03.377 INFO:tasks.workunit.client.0.smithi125.stdout:./186/.snap/snap-subdir-test 2023-10-30T12:33:03.382 INFO:tasks.workunit.client.0.smithi125.stdout:./237/.snap/snap-subdir-test 2023-10-30T12:33:03.386 INFO:tasks.workunit.client.0.smithi125.stdout:./190/.snap/snap-subdir-test 2023-10-30T12:33:03.392 INFO:tasks.workunit.client.0.smithi125.stdout:./213/.snap/snap-subdir-test 2023-10-30T12:33:03.398 INFO:tasks.workunit.client.0.smithi125.stdout:./104/.snap/snap-subdir-test 2023-10-30T12:33:03.403 INFO:tasks.workunit.client.0.smithi125.stdout:./164/.snap/snap-subdir-test 2023-10-30T12:33:03.409 INFO:tasks.workunit.client.0.smithi125.stdout:./112/.snap/snap-subdir-test 2023-10-30T12:33:03.415 INFO:tasks.workunit.client.0.smithi125.stdout:./295/.snap/snap-subdir-test 2023-10-30T12:33:03.420 INFO:tasks.workunit.client.0.smithi125.stdout:./272/.snap/snap-subdir-test 2023-10-30T12:33:03.425 INFO:tasks.workunit.client.0.smithi125.stdout:./100/.snap/snap-subdir-test 2023-10-30T12:33:03.430 INFO:tasks.workunit.client.0.smithi125.stdout:./134/.snap/snap-subdir-test 2023-10-30T12:33:03.435 INFO:tasks.workunit.client.0.smithi125.stdout:./200/.snap/snap-subdir-test 2023-10-30T12:33:03.440 INFO:tasks.workunit.client.0.smithi125.stdout:./115/.snap/snap-subdir-test 2023-10-30T12:33:03.445 INFO:tasks.workunit.client.0.smithi125.stdout:./284/.snap/snap-subdir-test 2023-10-30T12:33:03.450 INFO:tasks.workunit.client.0.smithi125.stdout:./280/.snap/snap-subdir-test 2023-10-30T12:33:03.454 INFO:tasks.workunit.client.0.smithi125.stdout:./116/.snap/snap-subdir-test 2023-10-30T12:33:03.460 INFO:tasks.workunit.client.0.smithi125.stdout:./245/.snap/snap-subdir-test 2023-10-30T12:33:03.465 INFO:tasks.workunit.client.0.smithi125.stdout:./195/.snap/snap-subdir-test 2023-10-30T12:33:03.470 INFO:tasks.workunit.client.0.smithi125.stdout:./138/.snap/snap-subdir-test 2023-10-30T12:33:03.478 INFO:tasks.workunit.client.0.smithi125.stdout:./135/.snap/snap-subdir-test 2023-10-30T12:33:03.485 INFO:tasks.workunit.client.0.smithi125.stdout:./117/.snap/snap-subdir-test 2023-10-30T12:33:03.491 INFO:tasks.workunit.client.0.smithi125.stdout:./241/.snap/snap-subdir-test 2023-10-30T12:33:03.496 INFO:tasks.workunit.client.0.smithi125.stdout:./143/.snap/snap-subdir-test 2023-10-30T12:33:03.501 INFO:tasks.workunit.client.0.smithi125.stdout:./180/.snap/snap-subdir-test 2023-10-30T12:33:03.507 INFO:tasks.workunit.client.0.smithi125.stdout:./126/.snap/snap-subdir-test 2023-10-30T12:33:03.514 INFO:tasks.workunit.client.0.smithi125.stdout:./161/.snap/snap-subdir-test 2023-10-30T12:33:03.520 INFO:tasks.workunit.client.0.smithi125.stdout:./204/.snap/snap-subdir-test 2023-10-30T12:33:03.526 INFO:tasks.workunit.client.0.smithi125.stdout:./176/.snap/snap-subdir-test 2023-10-30T12:33:03.533 INFO:tasks.workunit.client.0.smithi125.stdout:./255/.snap/snap-subdir-test 2023-10-30T12:33:03.539 INFO:tasks.workunit.client.0.smithi125.stdout:./215/.snap/snap-subdir-test 2023-10-30T12:33:03.545 INFO:tasks.workunit.client.0.smithi125.stdout:./111/.snap/snap-subdir-test 2023-10-30T12:33:03.550 INFO:tasks.workunit.client.0.smithi125.stdout:./238/.snap/snap-subdir-test 2023-10-30T12:33:03.556 INFO:tasks.workunit.client.0.smithi125.stdout:./191/.snap/snap-subdir-test 2023-10-30T12:33:03.562 INFO:tasks.workunit.client.0.smithi125.stdout:./184/.snap/snap-subdir-test 2023-10-30T12:33:03.569 INFO:tasks.workunit.client.0.smithi125.stdout:./192/.snap/snap-subdir-test 2023-10-30T12:33:03.575 INFO:tasks.workunit.client.0.smithi125.stdout:./103/.snap/snap-subdir-test 2023-10-30T12:33:03.581 INFO:tasks.workunit.client.0.smithi125.stdout:./128/.snap/snap-subdir-test 2023-10-30T12:33:03.587 INFO:tasks.workunit.client.0.smithi125.stdout:./229/.snap/snap-subdir-test 2023-10-30T12:33:03.593 INFO:tasks.workunit.client.0.smithi125.stdout:./144/.snap/snap-subdir-test 2023-10-30T12:33:04.214 INFO:tasks.workunit.client.0.smithi125.stdout:./146/.snap/snap-subdir-test 2023-10-30T12:33:04.221 INFO:tasks.workunit.client.0.smithi125.stdout:./227/.snap/snap-subdir-test 2023-10-30T12:33:04.228 INFO:tasks.workunit.client.0.smithi125.stdout:./278/.snap/snap-subdir-test 2023-10-30T12:33:04.235 INFO:tasks.workunit.client.0.smithi125.stdout:./141/.snap/snap-subdir-test 2023-10-30T12:33:04.241 INFO:tasks.workunit.client.0.smithi125.stdout:./234/.snap/snap-subdir-test 2023-10-30T12:33:04.248 INFO:tasks.workunit.client.0.smithi125.stdout:./225/.snap/snap-subdir-test 2023-10-30T12:33:04.254 INFO:tasks.workunit.client.0.smithi125.stdout:./182/.snap/snap-subdir-test 2023-10-30T12:33:04.261 INFO:tasks.workunit.client.0.smithi125.stdout:./107/.snap/snap-subdir-test 2023-10-30T12:33:04.268 INFO:tasks.workunit.client.0.smithi125.stdout:./178/.snap/snap-subdir-test 2023-10-30T12:33:04.274 INFO:tasks.workunit.client.0.smithi125.stdout:./266/.snap/snap-subdir-test 2023-10-30T12:33:04.280 INFO:tasks.workunit.client.0.smithi125.stdout:./240/.snap/snap-subdir-test 2023-10-30T12:33:04.287 INFO:tasks.workunit.client.0.smithi125.stdout:./156/.snap/snap-subdir-test 2023-10-30T12:33:04.293 INFO:tasks.workunit.client.0.smithi125.stdout:./131/.snap/snap-subdir-test 2023-10-30T12:33:04.300 INFO:tasks.workunit.client.0.smithi125.stdout:./123/.snap/snap-subdir-test 2023-10-30T12:33:04.307 INFO:tasks.workunit.client.0.smithi125.stdout:./160/.snap/snap-subdir-test 2023-10-30T12:33:04.313 INFO:tasks.workunit.client.0.smithi125.stdout:./165/.snap/snap-subdir-test 2023-10-30T12:33:04.320 INFO:tasks.workunit.client.0.smithi125.stdout:./101/.snap/snap-subdir-test 2023-10-30T12:33:04.327 INFO:tasks.workunit.client.0.smithi125.stdout:./148/.snap/snap-subdir-test 2023-10-30T12:33:04.334 INFO:tasks.workunit.client.0.smithi125.stdout:./168/.snap/snap-subdir-test 2023-10-30T12:33:04.341 INFO:tasks.workunit.client.0.smithi125.stdout:./179/.snap/snap-subdir-test 2023-10-30T12:33:04.347 INFO:tasks.workunit.client.0.smithi125.stdout:./257/.snap/snap-subdir-test 2023-10-30T12:33:04.354 INFO:tasks.workunit.client.0.smithi125.stdout:./252/.snap/snap-subdir-test 2023-10-30T12:33:04.361 INFO:tasks.workunit.client.0.smithi125.stdout:./132/.snap/snap-subdir-test 2023-10-30T12:33:04.368 INFO:tasks.workunit.client.0.smithi125.stdout:./236/.snap/snap-subdir-test 2023-10-30T12:33:04.374 INFO:tasks.workunit.client.0.smithi125.stdout:./211/.snap/snap-subdir-test 2023-10-30T12:33:04.381 INFO:tasks.workunit.client.0.smithi125.stdout:./199/.snap/snap-subdir-test 2023-10-30T12:33:04.388 INFO:tasks.workunit.client.0.smithi125.stdout:./205/.snap/snap-subdir-test 2023-10-30T12:33:04.394 INFO:tasks.workunit.client.0.smithi125.stdout:./163/.snap/snap-subdir-test 2023-10-30T12:33:04.400 INFO:tasks.workunit.client.0.smithi125.stdout:./261/.snap/snap-subdir-test 2023-10-30T12:33:04.406 INFO:tasks.workunit.client.0.smithi125.stdout:./201/.snap/snap-subdir-test 2023-10-30T12:33:04.413 INFO:tasks.workunit.client.0.smithi125.stdout:./282/.snap/snap-subdir-test 2023-10-30T12:33:04.419 INFO:tasks.workunit.client.0.smithi125.stdout:./296/.snap/snap-subdir-test 2023-10-30T12:33:04.426 INFO:tasks.workunit.client.0.smithi125.stdout:./217/.snap/snap-subdir-test 2023-10-30T12:33:04.433 INFO:tasks.workunit.client.0.smithi125.stdout:./153/.snap/snap-subdir-test 2023-10-30T12:33:04.439 INFO:tasks.workunit.client.0.smithi125.stdout:./206/.snap/snap-subdir-test 2023-10-30T12:33:04.446 INFO:tasks.workunit.client.0.smithi125.stdout:./122/.snap/snap-subdir-test 2023-10-30T12:33:04.453 INFO:tasks.workunit.client.0.smithi125.stdout:./162/.snap/snap-subdir-test 2023-10-30T12:33:04.460 INFO:tasks.workunit.client.0.smithi125.stdout:./197/.snap/snap-subdir-test 2023-10-30T12:33:04.467 INFO:tasks.workunit.client.0.smithi125.stdout:./216/.snap/snap-subdir-test 2023-10-30T12:33:05.059 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T12:33:05.059 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T12:33:05.060 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T12:33:05.064 INFO:tasks.ceph.mds.c:Started 2023-10-30T12:33:05.064 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:05.162 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2023-10-30T12:33:05.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.219+0000 7f5ebcaea640 1 Processor -- start 2023-10-30T12:33:05.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.219+0000 7f5ebcaea640 1 -- start start 2023-10-30T12:33:05.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.219+0000 7f5ebcaea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5eb80c6f60 0x7f5eb81566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:05.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.219+0000 7f5ebcaea640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb80c0a50 con 0x7f5eb80c6b80 2023-10-30T12:33:05.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.219+0000 7f5ebcaea640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5eb806f6e0 con 0x7f5eb80c05d0 2023-10-30T12:33:05.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.219+0000 7f5ebcaea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb806f9c0 con 0x7f5eb80c6f60 2023-10-30T12:33:05.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.219+0000 7f5eb659c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5eb80c6b80 0x7f5eb80bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38528/0 (socket says 172.21.15.125:38528) 2023-10-30T12:33:05.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.219+0000 7f5eb659c640 1 -- 172.21.15.125:0/2135886941 learned_addr learned my addr 172.21.15.125:0/2135886941 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:33:05.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.219+0000 7f5eb6d9d640 1 --2- 172.21.15.125:0/2135886941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5eb80c6f60 0x7f5eb81566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:05.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.219+0000 7f5eb5d9b640 1 -- 172.21.15.125:0/2135886941 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724554453 0 0) 0x7f5eb80c0a50 con 0x7f5eb80c6b80 2023-10-30T12:33:05.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.219+0000 7f5eb5d9b640 1 -- 172.21.15.125:0/2135886941 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ea4005740 con 0x7f5eb80c6b80 2023-10-30T12:33:05.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb5d9b640 1 -- 172.21.15.125:0/2135886941 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 435970252 0 0) 0x7f5ea4005740 con 0x7f5eb80c6b80 2023-10-30T12:33:05.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb5d9b640 1 -- 172.21.15.125:0/2135886941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5eb80c6f60 msgr2=0x7f5eb81566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:05.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb5d9b640 1 --2- 172.21.15.125:0/2135886941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5eb80c6f60 0x7f5eb81566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:05.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb5d9b640 1 -- 172.21.15.125:0/2135886941 >> v1:172.21.15.136:6789/0 conn(0x7f5eb80c05d0 legacy=0x7f5eb813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:05.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb5d9b640 1 -- 172.21.15.125:0/2135886941 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb806fca0 con 0x7f5eb80c6b80 2023-10-30T12:33:05.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb5d9b640 1 -- 172.21.15.125:0/2135886941 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f5ea0001680 con 0x7f5eb80c6b80 2023-10-30T12:33:05.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb6d9d640 1 --2- 172.21.15.125:0/2135886941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5eb80c6f60 0x7f5eb81566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:33:05.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb5d9b640 1 -- 172.21.15.125:0/2135886941 >> v1:172.21.15.125:6789/0 conn(0x7f5eb80c6b80 legacy=0x7f5eb80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:05.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb5d9b640 1 --2- 172.21.15.125:0/2135886941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea4005bc0 0x7f5ea4005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:05.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb5d9b640 1 --2- 172.21.15.125:0/2135886941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5eb80c6f60 0x7f5ea40069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:05.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb5d9b640 1 --2- 172.21.15.125:0/2135886941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ea4006f20 0x7f5ea40073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:05.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb5d9b640 1 -- 172.21.15.125:0/2135886941 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb806f9c0 con 0x7f5ea4006f20 2023-10-30T12:33:05.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb5d9b640 1 -- 172.21.15.125:0/2135886941 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea400dba0 con 0x7f5ea4005bc0 2023-10-30T12:33:05.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb5d9b640 1 -- 172.21.15.125:0/2135886941 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea400dd10 con 0x7f5eb80c6f60 2023-10-30T12:33:05.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb6d9d640 1 --2- 172.21.15.125:0/2135886941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ea4006f20 0x7f5ea40073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:05.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb659c640 1 --2- 172.21.15.125:0/2135886941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea4005bc0 0x7f5ea4005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:05.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eadd9b640 1 --2- 172.21.15.125:0/2135886941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5eb80c6f60 0x7f5ea40069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:05.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb6d9d640 1 -- 172.21.15.125:0/2135886941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5eb80c6f60 msgr2=0x7f5ea40069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:05.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb6d9d640 1 --2- 172.21.15.125:0/2135886941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5eb80c6f60 0x7f5ea40069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:05.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb6d9d640 1 -- 172.21.15.125:0/2135886941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea4005bc0 msgr2=0x7f5ea4005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:05.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb6d9d640 1 --2- 172.21.15.125:0/2135886941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea4005bc0 0x7f5ea4005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:05.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb6d9d640 1 -- 172.21.15.125:0/2135886941 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb0009a70 con 0x7f5ea4006f20 2023-10-30T12:33:05.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.223+0000 7f5eb6d9d640 1 --2- 172.21.15.125:0/2135886941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ea4006f20 0x7f5ea40073a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000e350 tx=0x7f5eb001cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=638a011f6478de4e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:05.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5eb5d9b640 1 -- 172.21.15.125:0/2135886941 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea0007ea0 con 0x7f5ea4006f20 2023-10-30T12:33:05.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5eb5d9b640 1 -- 172.21.15.125:0/2135886941 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb000f040 con 0x7f5ea4006f20 2023-10-30T12:33:05.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5eb5d9b640 1 -- 172.21.15.125:0/2135886941 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb00ad440 con 0x7f5ea4006f20 2023-10-30T12:33:05.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5ebcaea640 1 -- 172.21.15.125:0/2135886941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ea4006f20 msgr2=0x7f5ea40073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:05.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5ebcaea640 1 --2- 172.21.15.125:0/2135886941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ea4006f20 0x7f5ea40073a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000e350 tx=0x7f5eb001cde0 comp rx=0 tx=0).stop 2023-10-30T12:33:05.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5eb6d9d640 1 -- 172.21.15.125:0/2135886941 reap_dead start 2023-10-30T12:33:05.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5ebcaea640 1 -- 172.21.15.125:0/2135886941 shutdown_connections 2023-10-30T12:33:05.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5ebcaea640 1 -- 172.21.15.125:0/2135886941 >> 172.21.15.125:0/2135886941 conn(0x7f5eb80c7910 msgr2=0x7f5eb80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:05.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5ebcaea640 1 -- 172.21.15.125:0/2135886941 shutdown_connections 2023-10-30T12:33:05.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5ebcaea640 1 -- 172.21.15.125:0/2135886941 wait complete. 2023-10-30T12:33:05.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5ebcaea640 1 Processor -- start 2023-10-30T12:33:05.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5ebcaea640 1 -- start start 2023-10-30T12:33:05.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5ebcaea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea4006f20 0x7f5eb80c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:05.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5ebcaea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5eb813e630 0x7f5eb80ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:05.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5ebcaea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5eb813d720 0x7f5eb806a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:05.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5ebcaea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb8155110 con 0x7f5eb813d720 2023-10-30T12:33:05.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5ebcaea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb8155cd0 con 0x7f5ea4006f20 2023-10-30T12:33:05.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.227+0000 7f5ebcaea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5eb813d0a0 con 0x7f5eb813e630 2023-10-30T12:33:05.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eb6d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5eb813d720 0x7f5eb806a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:05.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eb6d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5eb813d720 0x7f5eb806a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55254/0 (socket says 172.21.15.125:55254) 2023-10-30T12:33:05.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eb6d9d640 1 -- 172.21.15.125:0/3683489119 learned_addr learned my addr 172.21.15.125:0/3683489119 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:33:05.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eadd9b640 1 --2- 172.21.15.125:0/3683489119 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5eb813e630 0x7f5eb80ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:05.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eb659c640 1 --2- 172.21.15.125:0/3683489119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea4006f20 0x7f5eb80c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:05.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eb6d9d640 1 -- 172.21.15.125:0/3683489119 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5eb813e630 msgr2=0x7f5eb80ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:05.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eb6d9d640 1 --2- 172.21.15.125:0/3683489119 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5eb813e630 0x7f5eb80ca7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:05.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eb6d9d640 1 -- 172.21.15.125:0/3683489119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea4006f20 msgr2=0x7f5eb80c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:05.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eb6d9d640 1 --2- 172.21.15.125:0/3683489119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea4006f20 0x7f5eb80c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:05.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eb6d9d640 1 -- 172.21.15.125:0/3683489119 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb813d310 con 0x7f5eb813d720 2023-10-30T12:33:05.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eb6d9d640 1 --2- 172.21.15.125:0/3683489119 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5eb813d720 0x7f5eb806a0f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00b0040 tx=0x7f5eb0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:05.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eaf7fe640 1 -- 172.21.15.125:0/3683489119 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb00bc070 con 0x7f5eb813d720 2023-10-30T12:33:05.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eaf7fe640 1 -- 172.21.15.125:0/3683489119 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb00aeb60 con 0x7f5eb813d720 2023-10-30T12:33:05.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5ebcaea640 1 -- 172.21.15.125:0/3683489119 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5eb80c9fc0 con 0x7f5eb813d720 2023-10-30T12:33:05.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eaf7fe640 1 -- 172.21.15.125:0/3683489119 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb00aecd0 con 0x7f5eb813d720 2023-10-30T12:33:05.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5ebcaea640 1 -- 172.21.15.125:0/3683489119 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5eb80b0910 con 0x7f5eb813d720 2023-10-30T12:33:05.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5ebcaea640 1 -- 172.21.15.125:0/3683489119 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5eb80c9e20 con 0x7f5eb813d720 2023-10-30T12:33:05.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.231+0000 7f5eaf7fe640 1 -- 172.21.15.125:0/3683489119 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5eb00b4040 con 0x7f5eb813d720 2023-10-30T12:33:05.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.235+0000 7f5eaf7fe640 1 --2- 172.21.15.125:0/3683489119 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e980664d0 0x7f5e98068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:05.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.235+0000 7f5eaf7fe640 1 -- 172.21.15.125:0/3683489119 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f5eb011e2f0 con 0x7f5eb813d720 2023-10-30T12:33:05.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.235+0000 7f5eb659c640 1 --2- 172.21.15.125:0/3683489119 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e980664d0 0x7f5e98068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.235+0000 7f5eb659c640 1 --2- 172.21.15.125:0/3683489119 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e980664d0 0x7f5e98068990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0001740 tx=0x7f5ea0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:05.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.239+0000 7f5eaf7fe640 1 -- 172.21.15.125:0/3683489119 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5eb80c9e20 con 0x7f5eb813d720 2023-10-30T12:33:05.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.419+0000 7f5ebcaea640 1 -- 172.21.15.125:0/3683489119 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5eb80c9e20 con 0x7f5eb813d720 2023-10-30T12:33:05.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.419+0000 7f5eaf7fe640 1 -- 172.21.15.125:0/3683489119 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+5749 (secure 0 0 0) 0x7f5eb80c9e20 con 0x7f5eb813d720 2023-10-30T12:33:05.422 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:33:05.422 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:57.239180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:05.422 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 27 2023-10-30T12:33:05.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.419+0000 7f5ebcaea640 1 -- 172.21.15.125:0/3683489119 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e980664d0 msgr2=0x7f5e98068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:05.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.419+0000 7f5ebcaea640 1 --2- 172.21.15.125:0/3683489119 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e980664d0 0x7f5e98068990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0001740 tx=0x7f5ea0009040 comp rx=0 tx=0).stop 2023-10-30T12:33:05.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.419+0000 7f5ebcaea640 1 -- 172.21.15.125:0/3683489119 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5eb813d720 msgr2=0x7f5eb806a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:05.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.419+0000 7f5ebcaea640 1 --2- 172.21.15.125:0/3683489119 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5eb813d720 0x7f5eb806a0f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f5eb00b0040 tx=0x7f5eb0009750 comp rx=0 tx=0).stop 2023-10-30T12:33:05.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.419+0000 7f5ebcaea640 1 -- 172.21.15.125:0/3683489119 shutdown_connections 2023-10-30T12:33:05.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.419+0000 7f5ebcaea640 1 --2- 172.21.15.125:0/3683489119 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e980664d0 0x7f5e98068990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:05.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.419+0000 7f5ebcaea640 1 --2- 172.21.15.125:0/3683489119 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5eb813d720 0x7f5eb806a0f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:05.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.419+0000 7f5ebcaea640 1 --2- 172.21.15.125:0/3683489119 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5eb813e630 0x7f5eb80ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:05.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.419+0000 7f5ebcaea640 1 --2- 172.21.15.125:0/3683489119 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ea4006f20 0x7f5eb80c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:05.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.419+0000 7f5ebcaea640 1 -- 172.21.15.125:0/3683489119 >> 172.21.15.125:0/3683489119 conn(0x7f5eb80c7910 msgr2=0x7f5eb8138600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:05.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.419+0000 7f5ebcaea640 1 -- 172.21.15.125:0/3683489119 shutdown_connections 2023-10-30T12:33:05.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:05.419+0000 7f5ebcaea640 1 -- 172.21.15.125:0/3683489119 wait complete. 2023-10-30T12:33:05.443 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T12:33:06.471 INFO:tasks.workunit.client.0.smithi125.stdout:./254/.snap/snap-subdir-test 2023-10-30T12:33:06.477 INFO:tasks.workunit.client.0.smithi125.stdout:./113/.snap/snap-subdir-test 2023-10-30T12:33:06.485 INFO:tasks.workunit.client.0.smithi125.stdout:./152/.snap/snap-subdir-test 2023-10-30T12:33:06.492 INFO:tasks.workunit.client.0.smithi125.stdout:./121/.snap/snap-subdir-test 2023-10-30T12:33:06.498 INFO:tasks.workunit.client.0.smithi125.stdout:./118/.snap/snap-subdir-test 2023-10-30T12:33:06.505 INFO:tasks.workunit.client.0.smithi125.stdout:./196/.snap/snap-subdir-test 2023-10-30T12:33:06.512 INFO:tasks.workunit.client.0.smithi125.stdout:./124/.snap/snap-subdir-test 2023-10-30T12:33:06.519 INFO:tasks.workunit.client.0.smithi125.stdout:./239/.snap/snap-subdir-test 2023-10-30T12:33:06.525 INFO:tasks.workunit.client.0.smithi125.stdout:./188/.snap/snap-subdir-test 2023-10-30T12:33:06.532 INFO:tasks.workunit.client.0.smithi125.stdout:./287/.snap/snap-subdir-test 2023-10-30T12:33:06.539 INFO:tasks.workunit.client.0.smithi125.stdout:./275/.snap/snap-subdir-test 2023-10-30T12:33:06.546 INFO:tasks.workunit.client.0.smithi125.stdout:./259/.snap/snap-subdir-test 2023-10-30T12:33:06.552 INFO:tasks.workunit.client.0.smithi125.stdout:./224/.snap/snap-subdir-test 2023-10-30T12:33:06.560 INFO:tasks.workunit.client.0.smithi125.stdout:./264/.snap/snap-subdir-test 2023-10-30T12:33:06.567 INFO:tasks.workunit.client.0.smithi125.stdout:./145/.snap/snap-subdir-test 2023-10-30T12:33:06.573 INFO:tasks.workunit.client.0.smithi125.stdout:./279/.snap/snap-subdir-test 2023-10-30T12:33:06.579 INFO:tasks.workunit.client.0.smithi125.stdout:./263/.snap/snap-subdir-test 2023-10-30T12:33:06.586 INFO:tasks.workunit.client.0.smithi125.stdout:./130/.snap/snap-subdir-test 2023-10-30T12:33:06.592 INFO:tasks.workunit.client.0.smithi125.stdout:./193/.snap/snap-subdir-test 2023-10-30T12:33:06.598 INFO:tasks.workunit.client.0.smithi125.stdout:./223/.snap/snap-subdir-test 2023-10-30T12:33:06.603 INFO:tasks.workunit.client.0.smithi125.stdout:./207/.snap/snap-subdir-test 2023-10-30T12:33:06.609 INFO:tasks.workunit.client.0.smithi125.stdout:./276/.snap/snap-subdir-test 2023-10-30T12:33:06.614 INFO:tasks.workunit.client.0.smithi125.stdout:./271/.snap/snap-subdir-test 2023-10-30T12:33:06.620 INFO:tasks.workunit.client.0.smithi125.stdout:./251/.snap/snap-subdir-test 2023-10-30T12:33:06.626 INFO:tasks.workunit.client.0.smithi125.stdout:./175/.snap/snap-subdir-test 2023-10-30T12:33:06.632 INFO:tasks.workunit.client.0.smithi125.stdout:./222/.snap/snap-subdir-test 2023-10-30T12:33:06.637 INFO:tasks.workunit.client.0.smithi125.stdout:./220/.snap/snap-subdir-test 2023-10-30T12:33:06.642 INFO:tasks.workunit.client.0.smithi125.stdout:./203/.snap/snap-subdir-test 2023-10-30T12:33:06.648 INFO:tasks.workunit.client.0.smithi125.stdout:./181/.snap/snap-subdir-test 2023-10-30T12:33:06.654 INFO:tasks.workunit.client.0.smithi125.stdout:./286/.snap/snap-subdir-test 2023-10-30T12:33:06.660 INFO:tasks.workunit.client.0.smithi125.stdout:./127/.snap/snap-subdir-test 2023-10-30T12:33:06.918 INFO:tasks.workunit.client.0.smithi125.stdout:./108/.snap/snap-subdir-test 2023-10-30T12:33:06.923 INFO:tasks.workunit.client.0.smithi125.stdout:./183/.snap/snap-subdir-test 2023-10-30T12:33:06.929 INFO:tasks.workunit.client.0.smithi125.stdout:./294/.snap/snap-subdir-test 2023-10-30T12:33:06.936 INFO:tasks.workunit.client.0.smithi125.stdout:./235/.snap/snap-subdir-test 2023-10-30T12:33:06.943 INFO:tasks.workunit.client.0.smithi125.stdout:./177/.snap/snap-subdir-test 2023-10-30T12:33:06.948 INFO:tasks.workunit.client.0.smithi125.stdout:./169/.snap/snap-subdir-test 2023-10-30T12:33:06.954 INFO:tasks.workunit.client.0.smithi125.stdout:./212/.snap/snap-subdir-test 2023-10-30T12:33:06.960 INFO:tasks.workunit.client.0.smithi125.stdout:./154/.snap/snap-subdir-test 2023-10-30T12:33:06.966 INFO:tasks.workunit.client.0.smithi125.stdout:./171/.snap/snap-subdir-test 2023-10-30T12:33:06.972 INFO:tasks.workunit.client.0.smithi125.stdout:./139/.snap/snap-subdir-test 2023-10-30T12:33:06.979 INFO:tasks.workunit.client.0.smithi125.stdout:./102/.snap/snap-subdir-test 2023-10-30T12:33:06.985 INFO:tasks.workunit.client.0.smithi125.stdout:./129/.snap/snap-subdir-test 2023-10-30T12:33:07.444 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:33:07.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404f59e640 1 Processor -- start 2023-10-30T12:33:07.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404f59e640 1 -- start start 2023-10-30T12:33:07.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404f59e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f40501538f0 0x7f4050159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:07.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404f59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40500cefa0 0x7f40500cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:07.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404f59e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40500ca6a0 con 0x7f405015cd50 2023-10-30T12:33:07.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404f59e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f405015d300 con 0x7f40501538f0 2023-10-30T12:33:07.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404f59e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f405015d5e0 con 0x7f40500cefa0 2023-10-30T12:33:07.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404e59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f40501538f0 0x7f4050159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:07.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404e59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f40501538f0 0x7f4050159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55256/0 (socket says 172.21.15.125:55256) 2023-10-30T12:33:07.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404e59c640 1 -- 172.21.15.125:0/2142115741 learned_addr learned my addr 172.21.15.125:0/2142115741 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:33:07.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404ed9d640 1 --2- 172.21.15.125:0/2142115741 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40500cefa0 0x7f40500cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:07.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404e59c640 1 -- 172.21.15.125:0/2142115741 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40500cefa0 msgr2=0x7f40500cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:07.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404e59c640 1 --2- 172.21.15.125:0/2142115741 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40500cefa0 0x7f40500cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:07.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404e59c640 1 -- 172.21.15.125:0/2142115741 >> v1:172.21.15.136:6789/0 conn(0x7f405015cd50 legacy=0x7f40500c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:33:07.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.591+0000 7f404e59c640 1 -- 172.21.15.125:0/2142115741 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405015d8c0 con 0x7f40501538f0 2023-10-30T12:33:07.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.595+0000 7f404e59c640 1 --2- 172.21.15.125:0/2142115741 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40501538f0 0x7f4050159fb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4040009870 tx=0x7f404001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17295d92dcd79ee7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:07.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.595+0000 7f404d59a640 1 -- 172.21.15.125:0/2142115741 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40400a89b0 con 0x7f40501538f0 2023-10-30T12:33:07.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.595+0000 7f404d59a640 1 -- 172.21.15.125:0/2142115741 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40400a8b10 con 0x7f40501538f0 2023-10-30T12:33:07.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.595+0000 7f404d59a640 1 -- 172.21.15.125:0/2142115741 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40400a8ea0 con 0x7f40501538f0 2023-10-30T12:33:07.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.595+0000 7f404f59e640 1 -- 172.21.15.125:0/2142115741 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40501538f0 msgr2=0x7f4050159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:07.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.595+0000 7f404f59e640 1 --2- 172.21.15.125:0/2142115741 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40501538f0 0x7f4050159fb0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4040009870 tx=0x7f404001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:33:07.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.595+0000 7f404f59e640 1 -- 172.21.15.125:0/2142115741 shutdown_connections 2023-10-30T12:33:07.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.595+0000 7f404f59e640 1 --2- 172.21.15.125:0/2142115741 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40500cefa0 0x7f40500cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:07.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.595+0000 7f404f59e640 1 --2- 172.21.15.125:0/2142115741 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40501538f0 0x7f4050159fb0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:07.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.595+0000 7f404f59e640 1 -- 172.21.15.125:0/2142115741 >> 172.21.15.125:0/2142115741 conn(0x7f405006a640 msgr2=0x7f4050065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:07.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.595+0000 7f404f59e640 1 -- 172.21.15.125:0/2142115741 shutdown_connections 2023-10-30T12:33:07.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.595+0000 7f404f59e640 1 -- 172.21.15.125:0/2142115741 wait complete. 2023-10-30T12:33:07.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404f59e640 1 Processor -- start 2023-10-30T12:33:07.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404f59e640 1 -- start start 2023-10-30T12:33:07.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404f59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40500cefa0 0x7f40500c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:07.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404f59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40501538f0 0x7f405013ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:07.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404f59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f405015cd50 0x7f40500c4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:07.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404f59e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405015b880 con 0x7f40500cefa0 2023-10-30T12:33:07.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404f59e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40500c8e90 con 0x7f405015cd50 2023-10-30T12:33:07.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404f59e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f405013f200 con 0x7f40501538f0 2023-10-30T12:33:07.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40500cefa0 0x7f40500c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:07.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40500cefa0 0x7f40500c8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55266/0 (socket says 172.21.15.125:55266) 2023-10-30T12:33:07.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404e59c640 1 -- 172.21.15.125:0/169399613 learned_addr learned my addr 172.21.15.125:0/169399613 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:33:07.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404dd9b640 1 --2- 172.21.15.125:0/169399613 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40501538f0 0x7f405013ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:07.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404ed9d640 1 --2- 172.21.15.125:0/169399613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f405015cd50 0x7f40500c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:07.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404e59c640 1 -- 172.21.15.125:0/169399613 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40501538f0 msgr2=0x7f405013ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404e59c640 1 --2- 172.21.15.125:0/169399613 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40501538f0 0x7f405013ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404e59c640 1 -- 172.21.15.125:0/169399613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f405015cd50 msgr2=0x7f40500c4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404e59c640 1 --2- 172.21.15.125:0/169399613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f405015cd50 0x7f40500c4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404e59c640 1 -- 172.21.15.125:0/169399613 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40400094d0 con 0x7f40500cefa0 2023-10-30T12:33:07.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404e59c640 1 --2- 172.21.15.125:0/169399613 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40500cefa0 0x7f40500c8950 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f40400b3040 tx=0x7f404001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:07.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f403effd640 1 -- 172.21.15.125:0/169399613 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40400a89b0 con 0x7f40500cefa0 2023-10-30T12:33:07.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f403effd640 1 -- 172.21.15.125:0/169399613 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40400bb480 con 0x7f40500cefa0 2023-10-30T12:33:07.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404f59e640 1 -- 172.21.15.125:0/169399613 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40500c90e0 con 0x7f40500cefa0 2023-10-30T12:33:07.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f403effd640 1 -- 172.21.15.125:0/169399613 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40400bb830 con 0x7f40500cefa0 2023-10-30T12:33:07.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404f59e640 1 -- 172.21.15.125:0/169399613 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405015b230 con 0x7f40500cefa0 2023-10-30T12:33:07.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.599+0000 7f404f59e640 1 -- 172.21.15.125:0/169399613 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405004e170 con 0x7f40500cefa0 2023-10-30T12:33:07.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.603+0000 7f403effd640 1 -- 172.21.15.125:0/169399613 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f40400bb990 con 0x7f40500cefa0 2023-10-30T12:33:07.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.603+0000 7f403effd640 1 --2- 172.21.15.125:0/169399613 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f40300664a0 0x7f4030068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:33:07.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.603+0000 7f403effd640 1 -- 172.21.15.125:0/169399613 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5910+0+0 (secure 0 0 0) 0x7f40401225f0 con 0x7f40500cefa0 2023-10-30T12:33:07.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.603+0000 7f404dd9b640 1 --2- 172.21.15.125:0/169399613 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f40300664a0 0x7f4030068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:33:07.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.603+0000 7f404dd9b640 1 --2- 172.21.15.125:0/169399613 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f40300664a0 0x7f4030068960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f40501578f0 tx=0x7f4044009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:33:07.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.607+0000 7f403effd640 1 -- 172.21.15.125:0/169399613 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f40400ef9c0 con 0x7f40500cefa0 2023-10-30T12:33:07.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.799+0000 7f404f59e640 1 -- 172.21.15.125:0/169399613 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f405004e170 con 0x7f40500cefa0 2023-10-30T12:33:07.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.799+0000 7f403effd640 1 -- 172.21.15.125:0/169399613 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+6543 (secure 0 0 0) 0x7f40400b5020 con 0x7f40500cefa0 2023-10-30T12:33:07.804 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:33:07.804 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:32:57.239180+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:33:07.805 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 28 2023-10-30T12:33:07.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.803+0000 7f404f59e640 1 -- 172.21.15.125:0/169399613 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f40300664a0 msgr2=0x7f4030068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:07.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.803+0000 7f404f59e640 1 --2- 172.21.15.125:0/169399613 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f40300664a0 0x7f4030068960 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f40501578f0 tx=0x7f4044009060 comp rx=0 tx=0).stop 2023-10-30T12:33:07.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.803+0000 7f404f59e640 1 -- 172.21.15.125:0/169399613 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40500cefa0 msgr2=0x7f40500c8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:33:07.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.803+0000 7f404f59e640 1 --2- 172.21.15.125:0/169399613 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40500cefa0 0x7f40500c8950 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f40400b3040 tx=0x7f404001ba90 comp rx=0 tx=0).stop 2023-10-30T12:33:07.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.803+0000 7f404f59e640 1 -- 172.21.15.125:0/169399613 shutdown_connections 2023-10-30T12:33:07.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.803+0000 7f404f59e640 1 --2- 172.21.15.125:0/169399613 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f40300664a0 0x7f4030068960 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:07.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.803+0000 7f404f59e640 1 --2- 172.21.15.125:0/169399613 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f405015cd50 0x7f40500c4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:07.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.803+0000 7f404f59e640 1 --2- 172.21.15.125:0/169399613 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f40501538f0 0x7f405013ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:07.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.803+0000 7f404f59e640 1 --2- 172.21.15.125:0/169399613 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f40500cefa0 0x7f40500c8950 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:33:07.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.803+0000 7f404f59e640 1 -- 172.21.15.125:0/169399613 >> 172.21.15.125:0/169399613 conn(0x7f405006a640 msgr2=0x7f40500caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:33:07.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.803+0000 7f404f59e640 1 -- 172.21.15.125:0/169399613 shutdown_connections 2023-10-30T12:33:07.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:33:07.803+0000 7f404f59e640 1 -- 172.21.15.125:0/169399613 wait complete. 2023-10-30T12:33:07.824 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T12:33:07.824 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 112 secs before thrashing 2023-10-30T12:33:14.465 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:33:14.469 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:33:23.252 INFO:tasks.workunit.client.0.smithi125.stdout:Delete all the files and directories ... 2023-10-30T12:33:27.309 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/19' 2023-10-30T12:33:27.310 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/15' 2023-10-30T12:33:27.310 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/31' 2023-10-30T12:33:27.310 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/36' 2023-10-30T12:33:27.310 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/20' 2023-10-30T12:33:27.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/12' 2023-10-30T12:33:27.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/39' 2023-10-30T12:33:27.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/10' 2023-10-30T12:33:27.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/34' 2023-10-30T12:33:27.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/38' 2023-10-30T12:33:27.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/11' 2023-10-30T12:33:27.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/16' 2023-10-30T12:33:27.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/32' 2023-10-30T12:33:27.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/30' 2023-10-30T12:33:27.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/14' 2023-10-30T12:33:27.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/18' 2023-10-30T12:33:27.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/13' 2023-10-30T12:33:27.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/33' 2023-10-30T12:33:27.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/17' 2023-10-30T12:33:27.314 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/35' 2023-10-30T12:33:27.314 INFO:tasks.workunit.client.0.smithi125.stdout:removed './100/37' 2023-10-30T12:33:27.314 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './100' 2023-10-30T12:33:27.315 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/19' 2023-10-30T12:33:27.315 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/15' 2023-10-30T12:33:27.315 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/31' 2023-10-30T12:33:27.315 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/36' 2023-10-30T12:33:27.316 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/20' 2023-10-30T12:33:27.316 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/12' 2023-10-30T12:33:27.316 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/39' 2023-10-30T12:33:27.316 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/10' 2023-10-30T12:33:27.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/34' 2023-10-30T12:33:27.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/38' 2023-10-30T12:33:27.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/11' 2023-10-30T12:33:27.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/16' 2023-10-30T12:33:27.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/32' 2023-10-30T12:33:27.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/30' 2023-10-30T12:33:27.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/14' 2023-10-30T12:33:27.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/18' 2023-10-30T12:33:27.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/13' 2023-10-30T12:33:27.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/33' 2023-10-30T12:33:27.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/17' 2023-10-30T12:33:27.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/35' 2023-10-30T12:33:27.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed './101/37' 2023-10-30T12:33:27.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './101' 2023-10-30T12:33:27.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/19' 2023-10-30T12:33:27.320 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/15' 2023-10-30T12:33:27.320 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/31' 2023-10-30T12:33:27.320 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/36' 2023-10-30T12:33:27.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/20' 2023-10-30T12:33:27.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/12' 2023-10-30T12:33:27.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/39' 2023-10-30T12:33:27.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/10' 2023-10-30T12:33:27.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/34' 2023-10-30T12:33:27.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/38' 2023-10-30T12:33:27.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/11' 2023-10-30T12:33:27.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/16' 2023-10-30T12:33:27.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/32' 2023-10-30T12:33:27.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/30' 2023-10-30T12:33:27.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/14' 2023-10-30T12:33:27.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/18' 2023-10-30T12:33:27.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/13' 2023-10-30T12:33:27.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/33' 2023-10-30T12:33:27.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/17' 2023-10-30T12:33:27.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/35' 2023-10-30T12:33:27.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed './102/37' 2023-10-30T12:33:27.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './102' 2023-10-30T12:33:27.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/19' 2023-10-30T12:33:27.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/15' 2023-10-30T12:33:27.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/31' 2023-10-30T12:33:27.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/36' 2023-10-30T12:33:27.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/20' 2023-10-30T12:33:27.326 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/12' 2023-10-30T12:33:27.326 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/39' 2023-10-30T12:33:27.326 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/10' 2023-10-30T12:33:27.326 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/34' 2023-10-30T12:33:27.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/38' 2023-10-30T12:33:27.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/11' 2023-10-30T12:33:27.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/16' 2023-10-30T12:33:27.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/32' 2023-10-30T12:33:27.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/30' 2023-10-30T12:33:27.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/14' 2023-10-30T12:33:27.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/18' 2023-10-30T12:33:27.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/13' 2023-10-30T12:33:27.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/33' 2023-10-30T12:33:27.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/17' 2023-10-30T12:33:27.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/35' 2023-10-30T12:33:27.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './103/37' 2023-10-30T12:33:27.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './103' 2023-10-30T12:33:27.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/19' 2023-10-30T12:33:27.330 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/15' 2023-10-30T12:33:27.330 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/31' 2023-10-30T12:33:27.330 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/36' 2023-10-30T12:33:27.330 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/20' 2023-10-30T12:33:27.331 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/12' 2023-10-30T12:33:27.331 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/39' 2023-10-30T12:33:27.331 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/10' 2023-10-30T12:33:27.331 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/34' 2023-10-30T12:33:27.331 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/38' 2023-10-30T12:33:27.332 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/11' 2023-10-30T12:33:27.332 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/16' 2023-10-30T12:33:27.332 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/32' 2023-10-30T12:33:27.332 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/30' 2023-10-30T12:33:27.332 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/14' 2023-10-30T12:33:27.333 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/18' 2023-10-30T12:33:27.333 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/13' 2023-10-30T12:33:27.333 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/33' 2023-10-30T12:33:27.333 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/17' 2023-10-30T12:33:27.334 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/35' 2023-10-30T12:33:27.334 INFO:tasks.workunit.client.0.smithi125.stdout:removed './104/37' 2023-10-30T12:33:27.334 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './104' 2023-10-30T12:33:27.334 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/19' 2023-10-30T12:33:27.334 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/15' 2023-10-30T12:33:27.335 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/31' 2023-10-30T12:33:27.335 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/36' 2023-10-30T12:33:27.335 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/20' 2023-10-30T12:33:27.335 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/12' 2023-10-30T12:33:27.335 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/39' 2023-10-30T12:33:27.336 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/10' 2023-10-30T12:33:27.336 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/34' 2023-10-30T12:33:27.336 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/38' 2023-10-30T12:33:27.337 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/11' 2023-10-30T12:33:27.337 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/16' 2023-10-30T12:33:27.337 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/32' 2023-10-30T12:33:27.337 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/30' 2023-10-30T12:33:27.337 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/14' 2023-10-30T12:33:27.338 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/18' 2023-10-30T12:33:27.338 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/13' 2023-10-30T12:33:27.338 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/33' 2023-10-30T12:33:27.338 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/17' 2023-10-30T12:33:27.339 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/35' 2023-10-30T12:33:27.339 INFO:tasks.workunit.client.0.smithi125.stdout:removed './105/37' 2023-10-30T12:33:27.339 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './105' 2023-10-30T12:33:27.339 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/19' 2023-10-30T12:33:27.340 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/15' 2023-10-30T12:33:27.340 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/31' 2023-10-30T12:33:27.340 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/36' 2023-10-30T12:33:27.340 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/20' 2023-10-30T12:33:27.340 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/12' 2023-10-30T12:33:27.341 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/39' 2023-10-30T12:33:27.341 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/10' 2023-10-30T12:33:27.341 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/34' 2023-10-30T12:33:27.341 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/38' 2023-10-30T12:33:27.341 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/11' 2023-10-30T12:33:27.342 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/16' 2023-10-30T12:33:27.342 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/32' 2023-10-30T12:33:27.342 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/30' 2023-10-30T12:33:27.342 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/14' 2023-10-30T12:33:27.342 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/18' 2023-10-30T12:33:27.343 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/13' 2023-10-30T12:33:27.343 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/33' 2023-10-30T12:33:27.343 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/17' 2023-10-30T12:33:27.343 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/35' 2023-10-30T12:33:27.343 INFO:tasks.workunit.client.0.smithi125.stdout:removed './106/37' 2023-10-30T12:33:27.344 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './106' 2023-10-30T12:33:27.344 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/19' 2023-10-30T12:33:27.344 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/15' 2023-10-30T12:33:27.344 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/31' 2023-10-30T12:33:27.345 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/36' 2023-10-30T12:33:27.345 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/20' 2023-10-30T12:33:27.345 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/12' 2023-10-30T12:33:27.345 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/39' 2023-10-30T12:33:27.345 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/10' 2023-10-30T12:33:27.346 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/34' 2023-10-30T12:33:27.346 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/38' 2023-10-30T12:33:27.346 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/11' 2023-10-30T12:33:27.346 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/16' 2023-10-30T12:33:27.347 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/32' 2023-10-30T12:33:27.347 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/30' 2023-10-30T12:33:27.347 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/14' 2023-10-30T12:33:27.347 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/18' 2023-10-30T12:33:27.347 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/13' 2023-10-30T12:33:27.348 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/33' 2023-10-30T12:33:27.348 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/17' 2023-10-30T12:33:27.348 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/35' 2023-10-30T12:33:27.348 INFO:tasks.workunit.client.0.smithi125.stdout:removed './107/37' 2023-10-30T12:33:27.348 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './107' 2023-10-30T12:33:27.349 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/19' 2023-10-30T12:33:27.349 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/15' 2023-10-30T12:33:27.349 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/31' 2023-10-30T12:33:27.349 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/36' 2023-10-30T12:33:27.350 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/20' 2023-10-30T12:33:27.350 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/12' 2023-10-30T12:33:27.350 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/39' 2023-10-30T12:33:27.350 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/10' 2023-10-30T12:33:27.351 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/34' 2023-10-30T12:33:27.351 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/38' 2023-10-30T12:33:27.351 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/11' 2023-10-30T12:33:27.351 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/16' 2023-10-30T12:33:27.351 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/32' 2023-10-30T12:33:27.352 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/30' 2023-10-30T12:33:27.352 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/14' 2023-10-30T12:33:27.352 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/18' 2023-10-30T12:33:27.352 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/13' 2023-10-30T12:33:27.352 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/33' 2023-10-30T12:33:27.353 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/17' 2023-10-30T12:33:27.353 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/35' 2023-10-30T12:33:27.353 INFO:tasks.workunit.client.0.smithi125.stdout:removed './108/37' 2023-10-30T12:33:27.353 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './108' 2023-10-30T12:33:27.353 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/19' 2023-10-30T12:33:27.353 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/15' 2023-10-30T12:33:27.354 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/31' 2023-10-30T12:33:27.354 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/36' 2023-10-30T12:33:27.354 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/20' 2023-10-30T12:33:27.354 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/12' 2023-10-30T12:33:27.355 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/39' 2023-10-30T12:33:27.355 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/10' 2023-10-30T12:33:27.355 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/34' 2023-10-30T12:33:27.355 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/38' 2023-10-30T12:33:27.355 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/11' 2023-10-30T12:33:27.356 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/16' 2023-10-30T12:33:27.356 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/32' 2023-10-30T12:33:27.356 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/30' 2023-10-30T12:33:30.601 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/14' 2023-10-30T12:33:30.602 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/18' 2023-10-30T12:33:30.602 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/13' 2023-10-30T12:33:30.602 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/33' 2023-10-30T12:33:30.602 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/17' 2023-10-30T12:33:30.603 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/35' 2023-10-30T12:33:30.603 INFO:tasks.workunit.client.0.smithi125.stdout:removed './109/37' 2023-10-30T12:33:30.603 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './109' 2023-10-30T12:33:30.603 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/19' 2023-10-30T12:33:30.604 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/15' 2023-10-30T12:33:30.604 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/31' 2023-10-30T12:33:30.604 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/36' 2023-10-30T12:33:30.604 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/20' 2023-10-30T12:33:30.604 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/12' 2023-10-30T12:33:30.605 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/39' 2023-10-30T12:33:30.605 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/10' 2023-10-30T12:33:30.605 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/34' 2023-10-30T12:33:30.605 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/38' 2023-10-30T12:33:30.605 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/11' 2023-10-30T12:33:30.606 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/16' 2023-10-30T12:33:30.606 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/32' 2023-10-30T12:33:30.606 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/30' 2023-10-30T12:33:30.606 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/14' 2023-10-30T12:33:30.606 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/18' 2023-10-30T12:33:30.607 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/13' 2023-10-30T12:33:30.607 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/33' 2023-10-30T12:33:30.607 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/17' 2023-10-30T12:33:30.607 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/35' 2023-10-30T12:33:30.607 INFO:tasks.workunit.client.0.smithi125.stdout:removed './110/37' 2023-10-30T12:33:30.608 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './110' 2023-10-30T12:33:30.608 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/19' 2023-10-30T12:33:30.608 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/15' 2023-10-30T12:33:30.608 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/31' 2023-10-30T12:33:30.608 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/36' 2023-10-30T12:33:30.609 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/20' 2023-10-30T12:33:30.609 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/12' 2023-10-30T12:33:30.609 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/39' 2023-10-30T12:33:30.609 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/10' 2023-10-30T12:33:30.609 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/34' 2023-10-30T12:33:30.609 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/38' 2023-10-30T12:33:30.610 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/11' 2023-10-30T12:33:30.610 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/16' 2023-10-30T12:33:30.610 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/32' 2023-10-30T12:33:30.610 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/30' 2023-10-30T12:33:30.610 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/14' 2023-10-30T12:33:30.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/18' 2023-10-30T12:33:30.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/13' 2023-10-30T12:33:30.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/33' 2023-10-30T12:33:30.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/17' 2023-10-30T12:33:30.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/35' 2023-10-30T12:33:30.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed './111/37' 2023-10-30T12:33:30.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './111' 2023-10-30T12:33:30.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/19' 2023-10-30T12:33:30.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/15' 2023-10-30T12:33:30.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/31' 2023-10-30T12:33:30.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/36' 2023-10-30T12:33:30.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/20' 2023-10-30T12:33:30.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/12' 2023-10-30T12:33:30.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/39' 2023-10-30T12:33:30.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/10' 2023-10-30T12:33:30.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/34' 2023-10-30T12:33:30.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/38' 2023-10-30T12:33:30.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/11' 2023-10-30T12:33:30.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/16' 2023-10-30T12:33:30.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/32' 2023-10-30T12:33:30.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/30' 2023-10-30T12:33:30.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/14' 2023-10-30T12:33:30.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/18' 2023-10-30T12:33:30.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/13' 2023-10-30T12:33:30.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/33' 2023-10-30T12:33:30.616 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/17' 2023-10-30T12:33:30.616 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/35' 2023-10-30T12:33:30.616 INFO:tasks.workunit.client.0.smithi125.stdout:removed './112/37' 2023-10-30T12:33:30.616 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './112' 2023-10-30T12:33:30.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/19' 2023-10-30T12:33:30.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/15' 2023-10-30T12:33:30.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/31' 2023-10-30T12:33:30.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/36' 2023-10-30T12:33:30.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/20' 2023-10-30T12:33:30.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/12' 2023-10-30T12:33:30.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/39' 2023-10-30T12:33:30.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/10' 2023-10-30T12:33:30.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/34' 2023-10-30T12:33:30.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/38' 2023-10-30T12:33:30.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/11' 2023-10-30T12:33:30.621 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/16' 2023-10-30T12:33:30.622 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/32' 2023-10-30T12:33:30.622 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/30' 2023-10-30T12:33:30.622 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/14' 2023-10-30T12:33:30.622 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/18' 2023-10-30T12:33:30.623 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/13' 2023-10-30T12:33:30.623 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/33' 2023-10-30T12:33:30.623 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/17' 2023-10-30T12:33:30.623 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/35' 2023-10-30T12:33:30.623 INFO:tasks.workunit.client.0.smithi125.stdout:removed './113/37' 2023-10-30T12:33:30.624 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './113' 2023-10-30T12:33:30.624 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/19' 2023-10-30T12:33:30.624 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/15' 2023-10-30T12:33:30.624 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/31' 2023-10-30T12:33:30.624 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/36' 2023-10-30T12:33:30.625 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/20' 2023-10-30T12:33:30.625 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/12' 2023-10-30T12:33:30.625 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/39' 2023-10-30T12:33:30.625 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/10' 2023-10-30T12:33:30.625 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/34' 2023-10-30T12:33:30.626 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/38' 2023-10-30T12:33:30.626 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/11' 2023-10-30T12:33:30.626 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/16' 2023-10-30T12:33:30.626 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/32' 2023-10-30T12:33:30.626 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/30' 2023-10-30T12:33:30.627 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/14' 2023-10-30T12:33:30.627 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/18' 2023-10-30T12:33:30.627 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/13' 2023-10-30T12:33:30.627 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/33' 2023-10-30T12:33:30.627 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/17' 2023-10-30T12:33:30.628 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/35' 2023-10-30T12:33:30.628 INFO:tasks.workunit.client.0.smithi125.stdout:removed './114/37' 2023-10-30T12:33:30.628 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './114' 2023-10-30T12:33:30.628 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/19' 2023-10-30T12:33:30.628 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/15' 2023-10-30T12:33:30.629 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/31' 2023-10-30T12:33:30.629 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/36' 2023-10-30T12:33:30.629 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/20' 2023-10-30T12:33:30.629 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/12' 2023-10-30T12:33:30.629 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/39' 2023-10-30T12:33:30.630 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/10' 2023-10-30T12:33:30.630 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/34' 2023-10-30T12:33:30.630 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/38' 2023-10-30T12:33:30.630 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/11' 2023-10-30T12:33:30.630 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/16' 2023-10-30T12:33:30.631 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/32' 2023-10-30T12:33:30.631 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/30' 2023-10-30T12:33:30.631 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/14' 2023-10-30T12:33:30.631 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/18' 2023-10-30T12:33:30.631 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/13' 2023-10-30T12:33:30.632 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/33' 2023-10-30T12:33:30.632 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/17' 2023-10-30T12:33:30.632 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/35' 2023-10-30T12:33:30.632 INFO:tasks.workunit.client.0.smithi125.stdout:removed './115/37' 2023-10-30T12:33:30.632 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './115' 2023-10-30T12:33:30.632 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/19' 2023-10-30T12:33:30.633 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/15' 2023-10-30T12:33:30.633 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/31' 2023-10-30T12:33:30.633 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/36' 2023-10-30T12:33:30.633 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/20' 2023-10-30T12:33:30.633 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/12' 2023-10-30T12:33:30.634 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/39' 2023-10-30T12:33:30.634 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/10' 2023-10-30T12:33:30.634 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/34' 2023-10-30T12:33:30.634 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/38' 2023-10-30T12:33:30.634 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/11' 2023-10-30T12:33:30.635 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/16' 2023-10-30T12:33:30.635 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/32' 2023-10-30T12:33:30.635 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/30' 2023-10-30T12:33:30.635 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/14' 2023-10-30T12:33:30.635 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/18' 2023-10-30T12:33:30.636 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/13' 2023-10-30T12:33:30.636 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/33' 2023-10-30T12:33:30.636 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/17' 2023-10-30T12:33:30.636 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/35' 2023-10-30T12:33:30.636 INFO:tasks.workunit.client.0.smithi125.stdout:removed './116/37' 2023-10-30T12:33:30.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './116' 2023-10-30T12:33:30.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/19' 2023-10-30T12:33:30.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/15' 2023-10-30T12:33:30.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/31' 2023-10-30T12:33:30.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/36' 2023-10-30T12:33:30.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/20' 2023-10-30T12:33:30.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/12' 2023-10-30T12:33:30.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/39' 2023-10-30T12:33:30.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/10' 2023-10-30T12:33:30.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/34' 2023-10-30T12:33:30.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/38' 2023-10-30T12:33:30.639 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/11' 2023-10-30T12:33:30.639 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/16' 2023-10-30T12:33:30.639 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/32' 2023-10-30T12:33:30.639 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/30' 2023-10-30T12:33:30.639 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/14' 2023-10-30T12:33:30.640 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/18' 2023-10-30T12:33:30.640 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/13' 2023-10-30T12:33:30.640 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/33' 2023-10-30T12:33:30.640 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/17' 2023-10-30T12:33:30.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/35' 2023-10-30T12:33:30.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed './117/37' 2023-10-30T12:33:30.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './117' 2023-10-30T12:33:30.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/19' 2023-10-30T12:33:30.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/15' 2023-10-30T12:33:30.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/31' 2023-10-30T12:33:30.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/36' 2023-10-30T12:33:30.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/20' 2023-10-30T12:33:30.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/12' 2023-10-30T12:33:30.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/39' 2023-10-30T12:33:30.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/10' 2023-10-30T12:33:30.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/34' 2023-10-30T12:33:30.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/38' 2023-10-30T12:33:30.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/11' 2023-10-30T12:33:30.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/16' 2023-10-30T12:33:30.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/32' 2023-10-30T12:33:30.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/30' 2023-10-30T12:33:30.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/14' 2023-10-30T12:33:30.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/18' 2023-10-30T12:33:30.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/13' 2023-10-30T12:33:30.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/33' 2023-10-30T12:33:30.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/17' 2023-10-30T12:33:30.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/35' 2023-10-30T12:33:30.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed './118/37' 2023-10-30T12:33:30.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './118' 2023-10-30T12:33:30.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/19' 2023-10-30T12:33:30.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/15' 2023-10-30T12:33:30.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/31' 2023-10-30T12:33:30.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/36' 2023-10-30T12:33:30.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/20' 2023-10-30T12:33:30.647 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/12' 2023-10-30T12:33:32.981 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/39' 2023-10-30T12:33:32.981 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/10' 2023-10-30T12:33:32.982 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/34' 2023-10-30T12:33:32.982 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/38' 2023-10-30T12:33:32.982 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/11' 2023-10-30T12:33:32.982 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/16' 2023-10-30T12:33:32.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/32' 2023-10-30T12:33:32.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/30' 2023-10-30T12:33:32.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/14' 2023-10-30T12:33:32.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/18' 2023-10-30T12:33:32.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/13' 2023-10-30T12:33:32.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/33' 2023-10-30T12:33:32.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/17' 2023-10-30T12:33:32.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/35' 2023-10-30T12:33:32.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed './119/37' 2023-10-30T12:33:32.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './119' 2023-10-30T12:33:32.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/19' 2023-10-30T12:33:32.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/15' 2023-10-30T12:33:32.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/31' 2023-10-30T12:33:32.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/36' 2023-10-30T12:33:32.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/20' 2023-10-30T12:33:32.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/12' 2023-10-30T12:33:32.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/39' 2023-10-30T12:33:32.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/10' 2023-10-30T12:33:32.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/34' 2023-10-30T12:33:32.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/38' 2023-10-30T12:33:32.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/11' 2023-10-30T12:33:32.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/16' 2023-10-30T12:33:32.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/32' 2023-10-30T12:33:32.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/30' 2023-10-30T12:33:32.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/14' 2023-10-30T12:33:32.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/18' 2023-10-30T12:33:32.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/13' 2023-10-30T12:33:32.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/33' 2023-10-30T12:33:32.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/17' 2023-10-30T12:33:32.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/35' 2023-10-30T12:33:32.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed './120/37' 2023-10-30T12:33:32.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './120' 2023-10-30T12:33:32.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/19' 2023-10-30T12:33:32.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/15' 2023-10-30T12:33:32.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/31' 2023-10-30T12:33:32.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/36' 2023-10-30T12:33:32.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/20' 2023-10-30T12:33:32.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/12' 2023-10-30T12:33:32.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/39' 2023-10-30T12:33:32.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/10' 2023-10-30T12:33:32.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/34' 2023-10-30T12:33:32.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/38' 2023-10-30T12:33:32.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/11' 2023-10-30T12:33:32.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/16' 2023-10-30T12:33:32.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/32' 2023-10-30T12:33:32.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/30' 2023-10-30T12:33:32.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/14' 2023-10-30T12:33:32.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/18' 2023-10-30T12:33:32.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/13' 2023-10-30T12:33:32.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/33' 2023-10-30T12:33:32.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/17' 2023-10-30T12:33:32.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/35' 2023-10-30T12:33:32.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed './121/37' 2023-10-30T12:33:32.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './121' 2023-10-30T12:33:32.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/19' 2023-10-30T12:33:32.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/15' 2023-10-30T12:33:32.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/31' 2023-10-30T12:33:32.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/36' 2023-10-30T12:33:32.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/20' 2023-10-30T12:33:32.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/12' 2023-10-30T12:33:32.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/39' 2023-10-30T12:33:32.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/10' 2023-10-30T12:33:32.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/34' 2023-10-30T12:33:32.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/38' 2023-10-30T12:33:32.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/11' 2023-10-30T12:33:32.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/16' 2023-10-30T12:33:32.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/32' 2023-10-30T12:33:32.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/30' 2023-10-30T12:33:32.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/14' 2023-10-30T12:33:32.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/18' 2023-10-30T12:33:32.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/13' 2023-10-30T12:33:32.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/33' 2023-10-30T12:33:32.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/17' 2023-10-30T12:33:32.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/35' 2023-10-30T12:33:32.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './122/37' 2023-10-30T12:33:32.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './122' 2023-10-30T12:33:32.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/19' 2023-10-30T12:33:32.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/15' 2023-10-30T12:33:32.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/31' 2023-10-30T12:33:32.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/36' 2023-10-30T12:33:32.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/20' 2023-10-30T12:33:32.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/12' 2023-10-30T12:33:32.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/39' 2023-10-30T12:33:32.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/10' 2023-10-30T12:33:32.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/34' 2023-10-30T12:33:32.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/38' 2023-10-30T12:33:32.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/11' 2023-10-30T12:33:32.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/16' 2023-10-30T12:33:32.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/32' 2023-10-30T12:33:32.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/30' 2023-10-30T12:33:33.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/14' 2023-10-30T12:33:33.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/18' 2023-10-30T12:33:33.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/13' 2023-10-30T12:33:33.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/33' 2023-10-30T12:33:33.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/17' 2023-10-30T12:33:33.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/35' 2023-10-30T12:33:33.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed './123/37' 2023-10-30T12:33:33.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './123' 2023-10-30T12:33:33.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/19' 2023-10-30T12:33:33.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/15' 2023-10-30T12:33:33.002 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/31' 2023-10-30T12:33:33.002 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/36' 2023-10-30T12:33:33.002 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/20' 2023-10-30T12:33:33.002 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/12' 2023-10-30T12:33:33.002 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/39' 2023-10-30T12:33:33.003 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/10' 2023-10-30T12:33:33.003 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/34' 2023-10-30T12:33:33.003 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/38' 2023-10-30T12:33:33.003 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/11' 2023-10-30T12:33:33.003 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/16' 2023-10-30T12:33:33.004 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/32' 2023-10-30T12:33:33.004 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/30' 2023-10-30T12:33:33.004 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/14' 2023-10-30T12:33:33.004 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/18' 2023-10-30T12:33:33.004 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/13' 2023-10-30T12:33:33.005 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/33' 2023-10-30T12:33:33.005 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/17' 2023-10-30T12:33:33.005 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/35' 2023-10-30T12:33:33.005 INFO:tasks.workunit.client.0.smithi125.stdout:removed './124/37' 2023-10-30T12:33:33.005 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './124' 2023-10-30T12:33:33.006 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/19' 2023-10-30T12:33:33.006 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/15' 2023-10-30T12:33:33.006 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/31' 2023-10-30T12:33:33.006 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/36' 2023-10-30T12:33:33.006 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/20' 2023-10-30T12:33:33.007 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/12' 2023-10-30T12:33:33.007 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/39' 2023-10-30T12:33:33.007 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/10' 2023-10-30T12:33:33.007 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/34' 2023-10-30T12:33:33.007 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/38' 2023-10-30T12:33:33.008 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/11' 2023-10-30T12:33:33.008 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/16' 2023-10-30T12:33:33.008 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/32' 2023-10-30T12:33:33.008 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/30' 2023-10-30T12:33:33.008 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/14' 2023-10-30T12:33:33.009 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/18' 2023-10-30T12:33:33.009 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/13' 2023-10-30T12:33:33.009 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/33' 2023-10-30T12:33:33.009 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/17' 2023-10-30T12:33:33.009 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/35' 2023-10-30T12:33:33.010 INFO:tasks.workunit.client.0.smithi125.stdout:removed './125/37' 2023-10-30T12:33:33.010 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './125' 2023-10-30T12:33:33.010 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/19' 2023-10-30T12:33:33.010 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/15' 2023-10-30T12:33:33.010 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/31' 2023-10-30T12:33:33.011 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/36' 2023-10-30T12:33:33.011 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/20' 2023-10-30T12:33:33.011 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/12' 2023-10-30T12:33:33.011 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/39' 2023-10-30T12:33:33.011 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/10' 2023-10-30T12:33:33.012 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/34' 2023-10-30T12:33:33.012 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/38' 2023-10-30T12:33:33.012 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/11' 2023-10-30T12:33:33.012 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/16' 2023-10-30T12:33:33.012 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/32' 2023-10-30T12:33:33.012 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/30' 2023-10-30T12:33:33.013 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/14' 2023-10-30T12:33:33.013 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/18' 2023-10-30T12:33:33.013 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/13' 2023-10-30T12:33:33.013 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/33' 2023-10-30T12:33:33.013 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/17' 2023-10-30T12:33:33.014 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/35' 2023-10-30T12:33:33.014 INFO:tasks.workunit.client.0.smithi125.stdout:removed './126/37' 2023-10-30T12:33:33.014 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './126' 2023-10-30T12:33:33.014 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/19' 2023-10-30T12:33:33.014 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/15' 2023-10-30T12:33:33.015 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/31' 2023-10-30T12:33:33.015 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/36' 2023-10-30T12:33:33.015 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/20' 2023-10-30T12:33:33.015 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/12' 2023-10-30T12:33:33.015 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/39' 2023-10-30T12:33:33.015 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/10' 2023-10-30T12:33:33.016 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/34' 2023-10-30T12:33:33.016 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/38' 2023-10-30T12:33:33.016 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/11' 2023-10-30T12:33:33.016 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/16' 2023-10-30T12:33:33.016 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/32' 2023-10-30T12:33:33.017 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/30' 2023-10-30T12:33:33.017 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/14' 2023-10-30T12:33:33.017 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/18' 2023-10-30T12:33:33.017 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/13' 2023-10-30T12:33:33.017 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/33' 2023-10-30T12:33:33.018 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/17' 2023-10-30T12:33:33.018 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/35' 2023-10-30T12:33:33.018 INFO:tasks.workunit.client.0.smithi125.stdout:removed './127/37' 2023-10-30T12:33:33.018 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './127' 2023-10-30T12:33:33.018 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/19' 2023-10-30T12:33:33.019 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/15' 2023-10-30T12:33:33.019 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/31' 2023-10-30T12:33:33.019 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/36' 2023-10-30T12:33:33.019 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/20' 2023-10-30T12:33:33.020 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/12' 2023-10-30T12:33:33.020 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/39' 2023-10-30T12:33:33.020 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/10' 2023-10-30T12:33:33.020 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/34' 2023-10-30T12:33:33.020 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/38' 2023-10-30T12:33:33.020 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/11' 2023-10-30T12:33:33.021 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/16' 2023-10-30T12:33:33.021 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/32' 2023-10-30T12:33:33.021 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/30' 2023-10-30T12:33:33.021 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/14' 2023-10-30T12:33:33.021 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/18' 2023-10-30T12:33:33.022 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/13' 2023-10-30T12:33:33.022 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/33' 2023-10-30T12:33:33.022 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/17' 2023-10-30T12:33:33.022 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/35' 2023-10-30T12:33:44.536 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:33:44.541 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:33:45.493 INFO:tasks.workunit.client.0.smithi125.stdout:removed './128/37' 2023-10-30T12:33:45.494 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './128' 2023-10-30T12:33:45.494 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/19' 2023-10-30T12:33:45.494 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/15' 2023-10-30T12:33:45.495 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/31' 2023-10-30T12:33:45.495 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/36' 2023-10-30T12:33:45.495 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/20' 2023-10-30T12:33:45.495 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/12' 2023-10-30T12:33:45.495 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/39' 2023-10-30T12:33:45.496 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/10' 2023-10-30T12:33:45.496 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/34' 2023-10-30T12:33:45.496 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/38' 2023-10-30T12:33:45.496 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/11' 2023-10-30T12:33:45.496 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/16' 2023-10-30T12:33:45.497 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/32' 2023-10-30T12:33:45.497 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/30' 2023-10-30T12:33:45.497 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/14' 2023-10-30T12:33:45.497 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/18' 2023-10-30T12:33:45.497 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/13' 2023-10-30T12:33:45.498 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/33' 2023-10-30T12:33:45.498 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/17' 2023-10-30T12:33:45.498 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/35' 2023-10-30T12:33:45.498 INFO:tasks.workunit.client.0.smithi125.stdout:removed './129/37' 2023-10-30T12:33:45.498 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './129' 2023-10-30T12:33:45.499 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/19' 2023-10-30T12:33:45.499 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/15' 2023-10-30T12:33:45.499 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/31' 2023-10-30T12:33:45.499 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/36' 2023-10-30T12:33:45.499 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/20' 2023-10-30T12:33:45.500 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/12' 2023-10-30T12:33:45.500 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/39' 2023-10-30T12:33:45.500 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/10' 2023-10-30T12:33:45.500 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/34' 2023-10-30T12:33:45.500 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/38' 2023-10-30T12:33:45.501 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/11' 2023-10-30T12:33:45.501 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/16' 2023-10-30T12:33:45.501 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/32' 2023-10-30T12:33:45.501 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/30' 2023-10-30T12:33:45.501 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/14' 2023-10-30T12:33:45.502 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/18' 2023-10-30T12:33:45.502 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/13' 2023-10-30T12:33:45.502 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/33' 2023-10-30T12:33:45.502 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/17' 2023-10-30T12:33:45.503 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/35' 2023-10-30T12:33:45.503 INFO:tasks.workunit.client.0.smithi125.stdout:removed './130/37' 2023-10-30T12:33:45.503 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './130' 2023-10-30T12:33:45.503 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/19' 2023-10-30T12:33:45.503 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/15' 2023-10-30T12:33:45.503 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/31' 2023-10-30T12:33:45.504 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/36' 2023-10-30T12:33:45.504 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/20' 2023-10-30T12:33:45.504 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/12' 2023-10-30T12:33:45.504 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/39' 2023-10-30T12:33:45.505 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/10' 2023-10-30T12:33:45.505 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/34' 2023-10-30T12:33:45.505 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/38' 2023-10-30T12:33:45.505 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/11' 2023-10-30T12:33:45.505 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/16' 2023-10-30T12:33:45.506 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/32' 2023-10-30T12:33:45.506 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/30' 2023-10-30T12:33:45.506 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/14' 2023-10-30T12:33:45.506 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/18' 2023-10-30T12:33:45.506 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/13' 2023-10-30T12:33:45.507 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/33' 2023-10-30T12:33:45.507 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/17' 2023-10-30T12:33:45.507 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/35' 2023-10-30T12:33:45.507 INFO:tasks.workunit.client.0.smithi125.stdout:removed './131/37' 2023-10-30T12:33:45.507 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './131' 2023-10-30T12:33:45.508 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/19' 2023-10-30T12:33:45.508 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/15' 2023-10-30T12:33:45.508 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/31' 2023-10-30T12:33:45.508 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/36' 2023-10-30T12:33:45.508 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/20' 2023-10-30T12:33:45.509 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/12' 2023-10-30T12:33:45.509 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/39' 2023-10-30T12:33:45.509 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/10' 2023-10-30T12:33:45.509 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/34' 2023-10-30T12:33:45.509 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/38' 2023-10-30T12:33:45.509 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/11' 2023-10-30T12:33:45.510 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/16' 2023-10-30T12:33:45.510 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/32' 2023-10-30T12:33:45.510 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/30' 2023-10-30T12:33:45.510 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/14' 2023-10-30T12:33:45.510 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/18' 2023-10-30T12:33:45.511 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/13' 2023-10-30T12:33:45.511 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/33' 2023-10-30T12:33:45.511 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/17' 2023-10-30T12:33:45.511 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/35' 2023-10-30T12:33:45.511 INFO:tasks.workunit.client.0.smithi125.stdout:removed './132/37' 2023-10-30T12:33:45.512 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './132' 2023-10-30T12:33:45.512 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/19' 2023-10-30T12:33:45.512 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/15' 2023-10-30T12:33:45.512 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/31' 2023-10-30T12:33:45.512 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/36' 2023-10-30T12:33:45.513 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/20' 2023-10-30T12:33:45.513 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/12' 2023-10-30T12:33:45.513 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/39' 2023-10-30T12:33:45.513 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/10' 2023-10-30T12:33:45.513 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/34' 2023-10-30T12:33:45.514 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/38' 2023-10-30T12:33:45.514 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/11' 2023-10-30T12:33:45.514 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/16' 2023-10-30T12:33:45.514 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/32' 2023-10-30T12:33:45.514 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/30' 2023-10-30T12:33:45.515 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/14' 2023-10-30T12:33:45.515 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/18' 2023-10-30T12:33:45.515 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/13' 2023-10-30T12:33:45.515 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/33' 2023-10-30T12:33:45.515 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/17' 2023-10-30T12:33:45.516 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/35' 2023-10-30T12:33:45.516 INFO:tasks.workunit.client.0.smithi125.stdout:removed './133/37' 2023-10-30T12:33:45.516 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './133' 2023-10-30T12:33:45.516 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/19' 2023-10-30T12:33:45.516 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/15' 2023-10-30T12:33:45.517 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/31' 2023-10-30T12:33:45.517 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/36' 2023-10-30T12:33:45.517 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/20' 2023-10-30T12:33:45.517 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/12' 2023-10-30T12:33:45.517 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/39' 2023-10-30T12:33:45.518 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/10' 2023-10-30T12:33:45.518 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/34' 2023-10-30T12:33:45.518 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/38' 2023-10-30T12:33:45.518 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/11' 2023-10-30T12:33:45.518 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/16' 2023-10-30T12:33:45.519 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/32' 2023-10-30T12:33:45.519 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/30' 2023-10-30T12:33:45.519 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/14' 2023-10-30T12:33:45.519 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/18' 2023-10-30T12:33:45.519 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/13' 2023-10-30T12:33:45.520 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/33' 2023-10-30T12:33:45.520 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/17' 2023-10-30T12:33:45.520 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/35' 2023-10-30T12:33:45.520 INFO:tasks.workunit.client.0.smithi125.stdout:removed './134/37' 2023-10-30T12:33:45.521 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './134' 2023-10-30T12:33:45.521 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/19' 2023-10-30T12:33:45.521 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/15' 2023-10-30T12:33:45.521 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/31' 2023-10-30T12:33:45.521 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/36' 2023-10-30T12:33:45.522 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/20' 2023-10-30T12:33:45.522 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/12' 2023-10-30T12:33:45.522 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/39' 2023-10-30T12:33:45.522 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/10' 2023-10-30T12:33:45.522 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/34' 2023-10-30T12:33:45.523 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/38' 2023-10-30T12:33:45.523 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/11' 2023-10-30T12:33:45.523 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/16' 2023-10-30T12:33:45.523 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/32' 2023-10-30T12:33:45.523 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/30' 2023-10-30T12:33:45.523 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/14' 2023-10-30T12:33:45.524 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/18' 2023-10-30T12:33:45.524 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/13' 2023-10-30T12:33:45.524 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/33' 2023-10-30T12:33:45.524 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/17' 2023-10-30T12:33:45.524 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/35' 2023-10-30T12:33:45.525 INFO:tasks.workunit.client.0.smithi125.stdout:removed './135/37' 2023-10-30T12:33:45.525 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './135' 2023-10-30T12:33:45.525 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/19' 2023-10-30T12:33:45.525 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/15' 2023-10-30T12:33:45.525 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/31' 2023-10-30T12:33:45.526 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/36' 2023-10-30T12:33:45.526 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/20' 2023-10-30T12:33:45.526 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/12' 2023-10-30T12:33:45.526 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/39' 2023-10-30T12:33:45.526 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/10' 2023-10-30T12:33:45.526 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/34' 2023-10-30T12:33:45.527 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/38' 2023-10-30T12:33:45.527 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/11' 2023-10-30T12:33:45.527 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/16' 2023-10-30T12:33:45.527 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/32' 2023-10-30T12:33:45.527 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/30' 2023-10-30T12:33:45.528 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/14' 2023-10-30T12:33:45.528 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/18' 2023-10-30T12:33:45.528 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/13' 2023-10-30T12:33:45.528 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/33' 2023-10-30T12:33:45.528 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/17' 2023-10-30T12:33:45.529 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/35' 2023-10-30T12:33:45.529 INFO:tasks.workunit.client.0.smithi125.stdout:removed './136/37' 2023-10-30T12:33:45.529 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './136' 2023-10-30T12:33:45.529 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/19' 2023-10-30T12:33:45.529 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/15' 2023-10-30T12:33:45.529 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/31' 2023-10-30T12:33:45.530 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/36' 2023-10-30T12:33:45.530 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/20' 2023-10-30T12:33:45.530 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/12' 2023-10-30T12:33:45.530 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/39' 2023-10-30T12:33:45.530 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/10' 2023-10-30T12:33:45.531 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/34' 2023-10-30T12:33:45.531 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/38' 2023-10-30T12:33:45.531 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/11' 2023-10-30T12:33:45.531 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/16' 2023-10-30T12:33:45.531 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/32' 2023-10-30T12:33:45.532 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/30' 2023-10-30T12:33:45.532 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/14' 2023-10-30T12:33:45.532 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/18' 2023-10-30T12:33:45.532 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/13' 2023-10-30T12:33:45.532 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/33' 2023-10-30T12:33:45.533 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/17' 2023-10-30T12:33:45.533 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/35' 2023-10-30T12:33:45.533 INFO:tasks.workunit.client.0.smithi125.stdout:removed './137/37' 2023-10-30T12:33:45.533 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './137' 2023-10-30T12:33:45.533 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/19' 2023-10-30T12:33:45.533 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/15' 2023-10-30T12:33:45.534 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/31' 2023-10-30T12:33:45.534 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/36' 2023-10-30T12:33:45.534 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/20' 2023-10-30T12:33:45.534 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/12' 2023-10-30T12:33:45.534 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/39' 2023-10-30T12:33:45.535 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/10' 2023-10-30T12:33:45.535 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/34' 2023-10-30T12:33:45.535 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/38' 2023-10-30T12:33:45.535 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/11' 2023-10-30T12:33:45.535 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/16' 2023-10-30T12:33:48.309 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/32' 2023-10-30T12:33:48.309 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/30' 2023-10-30T12:33:48.309 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/14' 2023-10-30T12:33:48.310 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/18' 2023-10-30T12:33:48.310 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/13' 2023-10-30T12:33:48.310 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/33' 2023-10-30T12:33:48.310 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/17' 2023-10-30T12:33:48.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/35' 2023-10-30T12:33:48.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed './138/37' 2023-10-30T12:33:48.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './138' 2023-10-30T12:33:48.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/19' 2023-10-30T12:33:48.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/15' 2023-10-30T12:33:48.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/31' 2023-10-30T12:33:48.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/36' 2023-10-30T12:33:48.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/20' 2023-10-30T12:33:48.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/12' 2023-10-30T12:33:48.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/39' 2023-10-30T12:33:48.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/10' 2023-10-30T12:33:48.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/34' 2023-10-30T12:33:48.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/38' 2023-10-30T12:33:48.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/11' 2023-10-30T12:33:48.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/16' 2023-10-30T12:33:48.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/32' 2023-10-30T12:33:48.314 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/30' 2023-10-30T12:33:48.314 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/14' 2023-10-30T12:33:48.314 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/18' 2023-10-30T12:33:48.314 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/13' 2023-10-30T12:33:48.314 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/33' 2023-10-30T12:33:48.315 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/17' 2023-10-30T12:33:48.315 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/35' 2023-10-30T12:33:48.315 INFO:tasks.workunit.client.0.smithi125.stdout:removed './139/37' 2023-10-30T12:33:48.315 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './139' 2023-10-30T12:33:48.315 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/19' 2023-10-30T12:33:48.316 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/15' 2023-10-30T12:33:48.316 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/31' 2023-10-30T12:33:48.316 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/36' 2023-10-30T12:33:48.316 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/20' 2023-10-30T12:33:48.316 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/12' 2023-10-30T12:33:48.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/39' 2023-10-30T12:33:48.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/10' 2023-10-30T12:33:48.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/34' 2023-10-30T12:33:48.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/38' 2023-10-30T12:33:48.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/11' 2023-10-30T12:33:48.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/16' 2023-10-30T12:33:48.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/32' 2023-10-30T12:33:48.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/30' 2023-10-30T12:33:48.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/14' 2023-10-30T12:33:48.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/18' 2023-10-30T12:33:48.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/13' 2023-10-30T12:33:48.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/33' 2023-10-30T12:33:48.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/17' 2023-10-30T12:33:48.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/35' 2023-10-30T12:33:48.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed './140/37' 2023-10-30T12:33:48.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './140' 2023-10-30T12:33:48.320 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/19' 2023-10-30T12:33:48.320 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/15' 2023-10-30T12:33:48.320 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/31' 2023-10-30T12:33:48.320 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/36' 2023-10-30T12:33:48.320 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/20' 2023-10-30T12:33:48.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/12' 2023-10-30T12:33:48.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/39' 2023-10-30T12:33:48.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/10' 2023-10-30T12:33:48.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/34' 2023-10-30T12:33:48.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/38' 2023-10-30T12:33:48.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/11' 2023-10-30T12:33:48.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/16' 2023-10-30T12:33:48.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/32' 2023-10-30T12:33:48.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/30' 2023-10-30T12:33:48.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/14' 2023-10-30T12:33:48.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/18' 2023-10-30T12:33:48.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/13' 2023-10-30T12:33:48.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/33' 2023-10-30T12:33:48.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/17' 2023-10-30T12:33:48.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/35' 2023-10-30T12:33:48.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './141/37' 2023-10-30T12:33:48.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './141' 2023-10-30T12:33:48.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/19' 2023-10-30T12:33:48.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/15' 2023-10-30T12:33:48.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/31' 2023-10-30T12:33:48.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/36' 2023-10-30T12:33:48.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/20' 2023-10-30T12:33:48.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/12' 2023-10-30T12:33:48.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/39' 2023-10-30T12:33:48.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/10' 2023-10-30T12:33:48.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/34' 2023-10-30T12:33:48.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/38' 2023-10-30T12:33:48.326 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/11' 2023-10-30T12:33:48.326 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/16' 2023-10-30T12:33:48.326 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/32' 2023-10-30T12:33:48.326 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/30' 2023-10-30T12:33:48.326 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/14' 2023-10-30T12:33:48.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/18' 2023-10-30T12:33:48.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/13' 2023-10-30T12:33:48.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/33' 2023-10-30T12:33:48.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/17' 2023-10-30T12:33:48.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/35' 2023-10-30T12:33:48.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed './142/37' 2023-10-30T12:33:48.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './142' 2023-10-30T12:33:48.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/19' 2023-10-30T12:33:48.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/15' 2023-10-30T12:33:48.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/31' 2023-10-30T12:33:48.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/36' 2023-10-30T12:33:48.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/20' 2023-10-30T12:33:48.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/12' 2023-10-30T12:33:48.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/39' 2023-10-30T12:33:48.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/10' 2023-10-30T12:33:48.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/34' 2023-10-30T12:33:48.330 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/38' 2023-10-30T12:33:48.330 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/11' 2023-10-30T12:33:48.330 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/16' 2023-10-30T12:33:48.330 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/32' 2023-10-30T12:33:48.330 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/30' 2023-10-30T12:33:48.331 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/14' 2023-10-30T12:33:48.331 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/18' 2023-10-30T12:33:48.331 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/13' 2023-10-30T12:33:48.331 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/33' 2023-10-30T12:33:48.331 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/17' 2023-10-30T12:33:48.332 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/35' 2023-10-30T12:33:48.332 INFO:tasks.workunit.client.0.smithi125.stdout:removed './143/37' 2023-10-30T12:33:48.332 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './143' 2023-10-30T12:33:48.332 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/19' 2023-10-30T12:33:48.332 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/15' 2023-10-30T12:33:48.332 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/31' 2023-10-30T12:33:48.333 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/36' 2023-10-30T12:33:48.333 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/20' 2023-10-30T12:33:48.333 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/12' 2023-10-30T12:33:48.333 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/39' 2023-10-30T12:33:48.333 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/10' 2023-10-30T12:33:48.334 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/34' 2023-10-30T12:33:48.334 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/38' 2023-10-30T12:33:48.334 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/11' 2023-10-30T12:33:48.334 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/16' 2023-10-30T12:33:48.334 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/32' 2023-10-30T12:33:48.335 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/30' 2023-10-30T12:33:48.335 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/14' 2023-10-30T12:33:48.335 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/18' 2023-10-30T12:33:48.335 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/13' 2023-10-30T12:33:48.335 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/33' 2023-10-30T12:33:48.336 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/17' 2023-10-30T12:33:48.336 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/35' 2023-10-30T12:33:48.336 INFO:tasks.workunit.client.0.smithi125.stdout:removed './144/37' 2023-10-30T12:33:48.336 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './144' 2023-10-30T12:33:48.336 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/19' 2023-10-30T12:33:48.336 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/15' 2023-10-30T12:33:48.337 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/31' 2023-10-30T12:33:48.337 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/36' 2023-10-30T12:33:48.337 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/20' 2023-10-30T12:33:48.337 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/12' 2023-10-30T12:33:48.337 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/39' 2023-10-30T12:33:48.338 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/10' 2023-10-30T12:33:48.338 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/34' 2023-10-30T12:33:48.338 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/38' 2023-10-30T12:33:48.338 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/11' 2023-10-30T12:33:48.338 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/16' 2023-10-30T12:33:48.339 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/32' 2023-10-30T12:33:48.339 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/30' 2023-10-30T12:33:48.339 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/14' 2023-10-30T12:33:48.339 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/18' 2023-10-30T12:33:48.339 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/13' 2023-10-30T12:33:48.339 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/33' 2023-10-30T12:33:48.340 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/17' 2023-10-30T12:33:48.340 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/35' 2023-10-30T12:33:48.340 INFO:tasks.workunit.client.0.smithi125.stdout:removed './145/37' 2023-10-30T12:33:48.340 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './145' 2023-10-30T12:33:48.340 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/19' 2023-10-30T12:33:48.341 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/15' 2023-10-30T12:33:48.341 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/31' 2023-10-30T12:33:48.341 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/36' 2023-10-30T12:33:48.341 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/20' 2023-10-30T12:33:48.341 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/12' 2023-10-30T12:33:48.342 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/39' 2023-10-30T12:33:48.342 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/10' 2023-10-30T12:33:48.342 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/34' 2023-10-30T12:33:48.342 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/38' 2023-10-30T12:33:48.342 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/11' 2023-10-30T12:33:48.342 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/16' 2023-10-30T12:33:48.343 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/32' 2023-10-30T12:33:48.343 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/30' 2023-10-30T12:33:48.343 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/14' 2023-10-30T12:33:48.343 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/18' 2023-10-30T12:33:48.343 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/13' 2023-10-30T12:33:48.344 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/33' 2023-10-30T12:33:48.344 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/17' 2023-10-30T12:33:48.344 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/35' 2023-10-30T12:33:48.344 INFO:tasks.workunit.client.0.smithi125.stdout:removed './146/37' 2023-10-30T12:33:48.344 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './146' 2023-10-30T12:33:48.345 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/19' 2023-10-30T12:33:48.345 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/15' 2023-10-30T12:33:48.345 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/31' 2023-10-30T12:33:48.345 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/36' 2023-10-30T12:33:48.345 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/20' 2023-10-30T12:33:48.345 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/12' 2023-10-30T12:33:48.346 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/39' 2023-10-30T12:33:48.346 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/10' 2023-10-30T12:33:48.346 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/34' 2023-10-30T12:33:48.346 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/38' 2023-10-30T12:33:48.346 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/11' 2023-10-30T12:33:48.347 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/16' 2023-10-30T12:33:48.347 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/32' 2023-10-30T12:33:48.347 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/30' 2023-10-30T12:33:48.347 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/14' 2023-10-30T12:33:48.347 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/18' 2023-10-30T12:33:48.348 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/13' 2023-10-30T12:33:48.348 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/33' 2023-10-30T12:33:48.348 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/17' 2023-10-30T12:33:48.348 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/35' 2023-10-30T12:33:48.348 INFO:tasks.workunit.client.0.smithi125.stdout:removed './147/37' 2023-10-30T12:33:48.348 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './147' 2023-10-30T12:33:48.349 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/19' 2023-10-30T12:33:48.349 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/15' 2023-10-30T12:33:48.349 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/31' 2023-10-30T12:33:48.349 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/36' 2023-10-30T12:33:52.910 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/20' 2023-10-30T12:33:52.911 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/12' 2023-10-30T12:33:52.911 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/39' 2023-10-30T12:33:52.911 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/10' 2023-10-30T12:33:52.911 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/34' 2023-10-30T12:33:52.911 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/38' 2023-10-30T12:33:52.912 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/11' 2023-10-30T12:33:52.912 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/16' 2023-10-30T12:33:52.912 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/32' 2023-10-30T12:33:52.912 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/30' 2023-10-30T12:33:52.912 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/14' 2023-10-30T12:33:52.913 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/18' 2023-10-30T12:33:52.913 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/13' 2023-10-30T12:33:52.913 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/33' 2023-10-30T12:33:52.913 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/17' 2023-10-30T12:33:52.913 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/35' 2023-10-30T12:33:52.914 INFO:tasks.workunit.client.0.smithi125.stdout:removed './148/37' 2023-10-30T12:33:52.914 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './148' 2023-10-30T12:33:52.914 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/19' 2023-10-30T12:33:52.915 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/15' 2023-10-30T12:33:52.915 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/31' 2023-10-30T12:33:52.915 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/36' 2023-10-30T12:33:52.915 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/20' 2023-10-30T12:33:52.915 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/12' 2023-10-30T12:33:52.916 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/39' 2023-10-30T12:33:52.916 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/10' 2023-10-30T12:33:52.916 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/34' 2023-10-30T12:33:52.916 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/38' 2023-10-30T12:33:52.916 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/11' 2023-10-30T12:33:52.917 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/16' 2023-10-30T12:33:52.917 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/32' 2023-10-30T12:33:52.917 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/30' 2023-10-30T12:33:52.917 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/14' 2023-10-30T12:33:52.917 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/18' 2023-10-30T12:33:52.917 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/13' 2023-10-30T12:33:52.918 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/33' 2023-10-30T12:33:52.918 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/17' 2023-10-30T12:33:52.918 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/35' 2023-10-30T12:33:52.918 INFO:tasks.workunit.client.0.smithi125.stdout:removed './149/37' 2023-10-30T12:33:52.919 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './149' 2023-10-30T12:33:52.919 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/19' 2023-10-30T12:33:52.919 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/15' 2023-10-30T12:33:52.919 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/31' 2023-10-30T12:33:52.919 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/36' 2023-10-30T12:33:52.919 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/20' 2023-10-30T12:33:52.920 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/12' 2023-10-30T12:33:52.920 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/39' 2023-10-30T12:33:52.920 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/10' 2023-10-30T12:33:52.920 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/34' 2023-10-30T12:33:52.920 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/38' 2023-10-30T12:33:52.921 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/11' 2023-10-30T12:33:52.921 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/16' 2023-10-30T12:33:52.921 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/32' 2023-10-30T12:33:52.921 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/30' 2023-10-30T12:33:52.921 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/14' 2023-10-30T12:33:52.922 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/18' 2023-10-30T12:33:52.922 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/13' 2023-10-30T12:33:52.922 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/33' 2023-10-30T12:33:52.922 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/17' 2023-10-30T12:33:52.922 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/35' 2023-10-30T12:33:52.923 INFO:tasks.workunit.client.0.smithi125.stdout:removed './150/37' 2023-10-30T12:33:52.923 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './150' 2023-10-30T12:33:52.923 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/19' 2023-10-30T12:33:52.923 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/15' 2023-10-30T12:33:52.923 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/31' 2023-10-30T12:33:52.923 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/36' 2023-10-30T12:33:52.924 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/20' 2023-10-30T12:33:52.924 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/12' 2023-10-30T12:33:52.924 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/39' 2023-10-30T12:33:52.924 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/10' 2023-10-30T12:33:52.924 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/34' 2023-10-30T12:33:52.925 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/38' 2023-10-30T12:33:52.925 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/11' 2023-10-30T12:33:52.925 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/16' 2023-10-30T12:33:52.925 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/32' 2023-10-30T12:33:52.925 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/30' 2023-10-30T12:33:52.925 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/14' 2023-10-30T12:33:52.926 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/18' 2023-10-30T12:33:52.926 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/13' 2023-10-30T12:33:52.926 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/33' 2023-10-30T12:33:52.926 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/17' 2023-10-30T12:33:52.926 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/35' 2023-10-30T12:33:52.927 INFO:tasks.workunit.client.0.smithi125.stdout:removed './151/37' 2023-10-30T12:33:52.927 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './151' 2023-10-30T12:33:52.927 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/19' 2023-10-30T12:33:52.927 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/15' 2023-10-30T12:33:52.927 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/31' 2023-10-30T12:33:52.928 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/36' 2023-10-30T12:33:52.928 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/20' 2023-10-30T12:33:52.928 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/12' 2023-10-30T12:33:52.928 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/39' 2023-10-30T12:33:52.928 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/10' 2023-10-30T12:33:52.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/34' 2023-10-30T12:33:52.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/38' 2023-10-30T12:33:52.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/11' 2023-10-30T12:33:52.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/16' 2023-10-30T12:33:52.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/32' 2023-10-30T12:33:52.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/30' 2023-10-30T12:33:52.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/14' 2023-10-30T12:33:52.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/18' 2023-10-30T12:33:52.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/13' 2023-10-30T12:33:52.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/33' 2023-10-30T12:33:52.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/17' 2023-10-30T12:33:52.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/35' 2023-10-30T12:33:52.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed './152/37' 2023-10-30T12:33:52.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './152' 2023-10-30T12:33:52.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/19' 2023-10-30T12:33:52.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/15' 2023-10-30T12:33:52.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/31' 2023-10-30T12:33:52.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/36' 2023-10-30T12:33:52.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/20' 2023-10-30T12:33:52.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/12' 2023-10-30T12:33:52.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/39' 2023-10-30T12:33:52.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/10' 2023-10-30T12:33:52.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/34' 2023-10-30T12:33:52.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/38' 2023-10-30T12:33:52.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/11' 2023-10-30T12:33:52.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/16' 2023-10-30T12:33:52.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/32' 2023-10-30T12:33:52.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/30' 2023-10-30T12:33:52.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/14' 2023-10-30T12:33:52.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/18' 2023-10-30T12:33:52.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/13' 2023-10-30T12:33:52.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/33' 2023-10-30T12:33:52.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/17' 2023-10-30T12:33:52.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/35' 2023-10-30T12:33:52.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed './153/37' 2023-10-30T12:33:52.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './153' 2023-10-30T12:33:52.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/19' 2023-10-30T12:33:52.936 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/15' 2023-10-30T12:33:52.936 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/31' 2023-10-30T12:33:52.936 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/36' 2023-10-30T12:33:52.936 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/20' 2023-10-30T12:33:52.936 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/12' 2023-10-30T12:33:52.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/39' 2023-10-30T12:33:52.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/10' 2023-10-30T12:33:52.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/34' 2023-10-30T12:33:52.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/38' 2023-10-30T12:33:52.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/11' 2023-10-30T12:33:52.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/16' 2023-10-30T12:33:52.938 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/32' 2023-10-30T12:33:52.938 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/30' 2023-10-30T12:33:52.938 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/14' 2023-10-30T12:33:52.938 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/18' 2023-10-30T12:33:52.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/13' 2023-10-30T12:33:52.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/33' 2023-10-30T12:33:52.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/17' 2023-10-30T12:33:52.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/35' 2023-10-30T12:33:52.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './154/37' 2023-10-30T12:33:52.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './154' 2023-10-30T12:33:52.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/19' 2023-10-30T12:33:52.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/15' 2023-10-30T12:33:52.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/31' 2023-10-30T12:33:52.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/36' 2023-10-30T12:33:52.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/20' 2023-10-30T12:33:52.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/12' 2023-10-30T12:33:52.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/39' 2023-10-30T12:33:52.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/10' 2023-10-30T12:33:52.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/34' 2023-10-30T12:33:52.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/38' 2023-10-30T12:33:52.942 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/11' 2023-10-30T12:33:52.942 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/16' 2023-10-30T12:33:52.942 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/32' 2023-10-30T12:33:52.942 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/30' 2023-10-30T12:33:52.942 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/14' 2023-10-30T12:33:52.943 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/18' 2023-10-30T12:33:52.943 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/13' 2023-10-30T12:33:52.943 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/33' 2023-10-30T12:33:52.943 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/17' 2023-10-30T12:33:52.944 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/35' 2023-10-30T12:33:52.944 INFO:tasks.workunit.client.0.smithi125.stdout:removed './155/37' 2023-10-30T12:33:52.944 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './155' 2023-10-30T12:33:52.944 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/19' 2023-10-30T12:33:52.944 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/15' 2023-10-30T12:33:52.945 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/31' 2023-10-30T12:33:52.945 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/36' 2023-10-30T12:33:52.945 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/20' 2023-10-30T12:33:52.945 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/12' 2023-10-30T12:33:52.945 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/39' 2023-10-30T12:33:52.945 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/10' 2023-10-30T12:33:52.946 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/34' 2023-10-30T12:33:52.946 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/38' 2023-10-30T12:33:52.946 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/11' 2023-10-30T12:33:52.946 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/16' 2023-10-30T12:33:52.946 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/32' 2023-10-30T12:33:52.947 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/30' 2023-10-30T12:33:52.947 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/14' 2023-10-30T12:33:52.947 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/18' 2023-10-30T12:33:52.947 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/13' 2023-10-30T12:33:52.947 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/33' 2023-10-30T12:33:52.948 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/17' 2023-10-30T12:33:52.948 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/35' 2023-10-30T12:33:52.948 INFO:tasks.workunit.client.0.smithi125.stdout:removed './156/37' 2023-10-30T12:33:52.948 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './156' 2023-10-30T12:33:52.948 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/19' 2023-10-30T12:33:52.949 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/15' 2023-10-30T12:33:52.949 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/31' 2023-10-30T12:33:52.949 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/36' 2023-10-30T12:33:52.949 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/20' 2023-10-30T12:33:52.949 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/12' 2023-10-30T12:33:52.949 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/39' 2023-10-30T12:33:52.950 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/10' 2023-10-30T12:33:52.950 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/34' 2023-10-30T12:33:52.950 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/38' 2023-10-30T12:33:52.950 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/11' 2023-10-30T12:33:52.951 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/16' 2023-10-30T12:33:52.951 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/32' 2023-10-30T12:33:52.951 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/30' 2023-10-30T12:33:52.951 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/14' 2023-10-30T12:33:52.951 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/18' 2023-10-30T12:33:52.951 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/13' 2023-10-30T12:33:52.952 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/33' 2023-10-30T12:33:55.580 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/17' 2023-10-30T12:33:55.580 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/35' 2023-10-30T12:33:55.581 INFO:tasks.workunit.client.0.smithi125.stdout:removed './157/37' 2023-10-30T12:33:55.581 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './157' 2023-10-30T12:33:55.581 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/19' 2023-10-30T12:33:55.581 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/15' 2023-10-30T12:33:55.581 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/31' 2023-10-30T12:33:55.582 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/36' 2023-10-30T12:33:55.582 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/20' 2023-10-30T12:33:55.582 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/12' 2023-10-30T12:33:55.582 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/39' 2023-10-30T12:33:55.583 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/10' 2023-10-30T12:33:55.583 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/34' 2023-10-30T12:33:55.583 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/38' 2023-10-30T12:33:55.583 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/11' 2023-10-30T12:33:55.583 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/16' 2023-10-30T12:33:55.584 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/32' 2023-10-30T12:33:55.584 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/30' 2023-10-30T12:33:55.584 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/14' 2023-10-30T12:33:55.584 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/18' 2023-10-30T12:33:55.584 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/13' 2023-10-30T12:33:55.585 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/33' 2023-10-30T12:33:55.585 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/17' 2023-10-30T12:33:55.585 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/35' 2023-10-30T12:33:55.585 INFO:tasks.workunit.client.0.smithi125.stdout:removed './158/37' 2023-10-30T12:33:55.585 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './158' 2023-10-30T12:33:55.586 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/19' 2023-10-30T12:33:55.586 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/15' 2023-10-30T12:33:55.586 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/31' 2023-10-30T12:33:55.586 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/36' 2023-10-30T12:33:55.586 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/20' 2023-10-30T12:33:55.587 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/12' 2023-10-30T12:33:55.587 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/39' 2023-10-30T12:33:55.587 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/10' 2023-10-30T12:33:55.587 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/34' 2023-10-30T12:33:55.587 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/38' 2023-10-30T12:33:55.588 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/11' 2023-10-30T12:33:55.588 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/16' 2023-10-30T12:33:55.588 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/32' 2023-10-30T12:33:55.588 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/30' 2023-10-30T12:33:55.588 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/14' 2023-10-30T12:33:55.589 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/18' 2023-10-30T12:33:55.589 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/13' 2023-10-30T12:33:55.589 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/33' 2023-10-30T12:33:55.589 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/17' 2023-10-30T12:33:55.589 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/35' 2023-10-30T12:33:55.589 INFO:tasks.workunit.client.0.smithi125.stdout:removed './159/37' 2023-10-30T12:33:55.590 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './159' 2023-10-30T12:33:55.590 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/19' 2023-10-30T12:33:55.590 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/15' 2023-10-30T12:33:55.590 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/31' 2023-10-30T12:33:55.590 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/36' 2023-10-30T12:33:55.591 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/20' 2023-10-30T12:33:55.591 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/12' 2023-10-30T12:33:55.591 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/39' 2023-10-30T12:33:55.591 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/10' 2023-10-30T12:33:55.591 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/34' 2023-10-30T12:33:55.592 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/38' 2023-10-30T12:33:55.592 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/11' 2023-10-30T12:33:55.592 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/16' 2023-10-30T12:33:55.592 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/32' 2023-10-30T12:33:55.592 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/30' 2023-10-30T12:33:55.593 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/14' 2023-10-30T12:33:55.593 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/18' 2023-10-30T12:33:55.593 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/13' 2023-10-30T12:33:55.593 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/33' 2023-10-30T12:33:55.593 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/17' 2023-10-30T12:33:55.594 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/35' 2023-10-30T12:33:55.594 INFO:tasks.workunit.client.0.smithi125.stdout:removed './160/37' 2023-10-30T12:33:55.594 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './160' 2023-10-30T12:33:55.594 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/19' 2023-10-30T12:33:55.595 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/15' 2023-10-30T12:33:55.595 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/31' 2023-10-30T12:33:55.595 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/36' 2023-10-30T12:33:55.595 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/20' 2023-10-30T12:33:55.595 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/12' 2023-10-30T12:33:55.595 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/39' 2023-10-30T12:33:55.596 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/10' 2023-10-30T12:33:55.596 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/34' 2023-10-30T12:33:55.596 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/38' 2023-10-30T12:33:55.596 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/11' 2023-10-30T12:33:55.596 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/16' 2023-10-30T12:33:55.597 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/32' 2023-10-30T12:33:55.597 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/30' 2023-10-30T12:33:55.597 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/14' 2023-10-30T12:33:55.597 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/18' 2023-10-30T12:33:55.597 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/13' 2023-10-30T12:33:55.598 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/33' 2023-10-30T12:33:55.598 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/17' 2023-10-30T12:33:55.598 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/35' 2023-10-30T12:33:55.598 INFO:tasks.workunit.client.0.smithi125.stdout:removed './161/37' 2023-10-30T12:33:55.598 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './161' 2023-10-30T12:33:55.599 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/19' 2023-10-30T12:33:55.599 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/15' 2023-10-30T12:33:55.599 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/31' 2023-10-30T12:33:55.599 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/36' 2023-10-30T12:33:55.599 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/20' 2023-10-30T12:33:55.600 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/12' 2023-10-30T12:33:55.600 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/39' 2023-10-30T12:33:55.600 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/10' 2023-10-30T12:33:55.600 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/34' 2023-10-30T12:33:55.600 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/38' 2023-10-30T12:33:55.601 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/11' 2023-10-30T12:33:55.601 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/16' 2023-10-30T12:33:55.601 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/32' 2023-10-30T12:33:55.601 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/30' 2023-10-30T12:33:55.601 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/14' 2023-10-30T12:33:55.602 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/18' 2023-10-30T12:33:55.602 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/13' 2023-10-30T12:33:55.602 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/33' 2023-10-30T12:33:55.602 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/17' 2023-10-30T12:33:55.602 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/35' 2023-10-30T12:33:55.603 INFO:tasks.workunit.client.0.smithi125.stdout:removed './162/37' 2023-10-30T12:33:55.603 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './162' 2023-10-30T12:33:55.603 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/19' 2023-10-30T12:33:55.603 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/15' 2023-10-30T12:33:55.603 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/31' 2023-10-30T12:33:55.604 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/36' 2023-10-30T12:33:55.604 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/20' 2023-10-30T12:33:55.604 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/12' 2023-10-30T12:33:55.604 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/39' 2023-10-30T12:33:55.604 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/10' 2023-10-30T12:33:55.605 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/34' 2023-10-30T12:33:55.605 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/38' 2023-10-30T12:33:55.605 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/11' 2023-10-30T12:33:55.605 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/16' 2023-10-30T12:33:55.605 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/32' 2023-10-30T12:33:55.605 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/30' 2023-10-30T12:33:55.606 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/14' 2023-10-30T12:33:55.606 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/18' 2023-10-30T12:33:55.606 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/13' 2023-10-30T12:33:55.606 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/33' 2023-10-30T12:33:55.606 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/17' 2023-10-30T12:33:55.607 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/35' 2023-10-30T12:33:55.607 INFO:tasks.workunit.client.0.smithi125.stdout:removed './163/37' 2023-10-30T12:33:55.607 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './163' 2023-10-30T12:33:55.607 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/19' 2023-10-30T12:33:55.607 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/15' 2023-10-30T12:33:55.608 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/31' 2023-10-30T12:33:55.608 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/36' 2023-10-30T12:33:55.608 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/20' 2023-10-30T12:33:55.608 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/12' 2023-10-30T12:33:55.608 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/39' 2023-10-30T12:33:55.609 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/10' 2023-10-30T12:33:55.609 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/34' 2023-10-30T12:33:55.609 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/38' 2023-10-30T12:33:55.609 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/11' 2023-10-30T12:33:55.609 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/16' 2023-10-30T12:33:55.610 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/32' 2023-10-30T12:33:55.610 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/30' 2023-10-30T12:33:55.610 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/14' 2023-10-30T12:33:55.610 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/18' 2023-10-30T12:33:55.610 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/13' 2023-10-30T12:33:55.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/33' 2023-10-30T12:33:55.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/17' 2023-10-30T12:33:55.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/35' 2023-10-30T12:33:55.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed './164/37' 2023-10-30T12:33:55.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './164' 2023-10-30T12:33:55.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/19' 2023-10-30T12:33:55.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/15' 2023-10-30T12:33:55.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/31' 2023-10-30T12:33:55.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/36' 2023-10-30T12:33:55.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/20' 2023-10-30T12:33:55.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/12' 2023-10-30T12:33:55.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/39' 2023-10-30T12:33:55.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/10' 2023-10-30T12:33:55.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/34' 2023-10-30T12:33:55.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/38' 2023-10-30T12:33:55.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/11' 2023-10-30T12:33:55.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/16' 2023-10-30T12:33:55.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/32' 2023-10-30T12:33:55.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/30' 2023-10-30T12:33:55.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/14' 2023-10-30T12:33:55.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/18' 2023-10-30T12:33:55.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/13' 2023-10-30T12:33:55.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/33' 2023-10-30T12:33:55.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/17' 2023-10-30T12:33:55.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/35' 2023-10-30T12:33:55.616 INFO:tasks.workunit.client.0.smithi125.stdout:removed './165/37' 2023-10-30T12:33:55.616 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './165' 2023-10-30T12:33:55.616 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/19' 2023-10-30T12:33:55.616 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/15' 2023-10-30T12:33:55.616 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/31' 2023-10-30T12:33:55.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/36' 2023-10-30T12:33:55.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/20' 2023-10-30T12:33:55.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/12' 2023-10-30T12:33:55.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/39' 2023-10-30T12:33:55.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/10' 2023-10-30T12:33:55.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/34' 2023-10-30T12:33:55.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/38' 2023-10-30T12:33:55.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/11' 2023-10-30T12:33:55.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/16' 2023-10-30T12:33:55.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/32' 2023-10-30T12:33:55.619 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/30' 2023-10-30T12:33:55.619 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/14' 2023-10-30T12:33:55.619 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/18' 2023-10-30T12:33:55.619 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/13' 2023-10-30T12:33:55.619 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/33' 2023-10-30T12:33:55.620 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/17' 2023-10-30T12:33:55.620 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/35' 2023-10-30T12:33:55.620 INFO:tasks.workunit.client.0.smithi125.stdout:removed './166/37' 2023-10-30T12:33:55.620 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './166' 2023-10-30T12:33:55.620 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/19' 2023-10-30T12:33:55.621 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/15' 2023-10-30T12:33:55.621 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/31' 2023-10-30T12:33:55.621 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/36' 2023-10-30T12:33:55.621 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/20' 2023-10-30T12:33:55.621 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/12' 2023-10-30T12:33:55.622 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/39' 2023-10-30T12:33:55.622 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/10' 2023-10-30T12:33:55.622 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/34' 2023-10-30T12:33:55.622 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/38' 2023-10-30T12:34:01.187 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/11' 2023-10-30T12:34:01.187 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/16' 2023-10-30T12:34:01.187 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/32' 2023-10-30T12:34:01.187 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/30' 2023-10-30T12:34:01.188 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/14' 2023-10-30T12:34:01.188 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/18' 2023-10-30T12:34:01.188 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/13' 2023-10-30T12:34:01.188 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/33' 2023-10-30T12:34:01.188 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/17' 2023-10-30T12:34:01.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/35' 2023-10-30T12:34:01.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './167/37' 2023-10-30T12:34:01.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './167' 2023-10-30T12:34:01.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/19' 2023-10-30T12:34:01.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/15' 2023-10-30T12:34:01.190 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/31' 2023-10-30T12:34:01.190 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/36' 2023-10-30T12:34:01.190 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/20' 2023-10-30T12:34:01.190 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/12' 2023-10-30T12:34:01.190 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/39' 2023-10-30T12:34:01.191 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/10' 2023-10-30T12:34:01.191 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/34' 2023-10-30T12:34:01.191 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/38' 2023-10-30T12:34:01.191 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/11' 2023-10-30T12:34:01.191 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/16' 2023-10-30T12:34:01.192 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/32' 2023-10-30T12:34:01.192 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/30' 2023-10-30T12:34:01.192 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/14' 2023-10-30T12:34:01.192 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/18' 2023-10-30T12:34:01.192 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/13' 2023-10-30T12:34:01.193 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/33' 2023-10-30T12:34:01.193 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/17' 2023-10-30T12:34:01.193 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/35' 2023-10-30T12:34:01.193 INFO:tasks.workunit.client.0.smithi125.stdout:removed './168/37' 2023-10-30T12:34:01.193 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './168' 2023-10-30T12:34:01.194 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/19' 2023-10-30T12:34:01.194 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/15' 2023-10-30T12:34:01.194 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/31' 2023-10-30T12:34:01.194 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/36' 2023-10-30T12:34:01.194 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/20' 2023-10-30T12:34:01.195 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/12' 2023-10-30T12:34:01.195 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/39' 2023-10-30T12:34:01.195 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/10' 2023-10-30T12:34:01.195 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/34' 2023-10-30T12:34:01.195 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/38' 2023-10-30T12:34:01.196 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/11' 2023-10-30T12:34:01.196 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/16' 2023-10-30T12:34:01.196 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/32' 2023-10-30T12:34:01.196 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/30' 2023-10-30T12:34:01.196 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/14' 2023-10-30T12:34:01.197 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/18' 2023-10-30T12:34:01.197 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/13' 2023-10-30T12:34:01.197 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/33' 2023-10-30T12:34:01.197 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/17' 2023-10-30T12:34:01.198 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/35' 2023-10-30T12:34:01.198 INFO:tasks.workunit.client.0.smithi125.stdout:removed './169/37' 2023-10-30T12:34:01.198 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './169' 2023-10-30T12:34:01.198 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/19' 2023-10-30T12:34:01.198 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/15' 2023-10-30T12:34:01.199 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/31' 2023-10-30T12:34:01.199 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/36' 2023-10-30T12:34:01.199 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/20' 2023-10-30T12:34:01.199 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/12' 2023-10-30T12:34:01.199 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/39' 2023-10-30T12:34:01.200 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/10' 2023-10-30T12:34:01.200 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/34' 2023-10-30T12:34:01.200 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/38' 2023-10-30T12:34:01.200 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/11' 2023-10-30T12:34:01.200 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/16' 2023-10-30T12:34:01.201 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/32' 2023-10-30T12:34:01.201 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/30' 2023-10-30T12:34:01.201 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/14' 2023-10-30T12:34:01.201 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/18' 2023-10-30T12:34:01.202 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/13' 2023-10-30T12:34:01.202 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/33' 2023-10-30T12:34:01.202 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/17' 2023-10-30T12:34:01.202 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/35' 2023-10-30T12:34:01.202 INFO:tasks.workunit.client.0.smithi125.stdout:removed './170/37' 2023-10-30T12:34:01.203 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './170' 2023-10-30T12:34:01.203 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/19' 2023-10-30T12:34:01.203 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/15' 2023-10-30T12:34:01.203 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/31' 2023-10-30T12:34:01.203 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/36' 2023-10-30T12:34:01.204 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/20' 2023-10-30T12:34:01.204 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/12' 2023-10-30T12:34:01.204 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/39' 2023-10-30T12:34:01.204 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/10' 2023-10-30T12:34:01.204 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/34' 2023-10-30T12:34:01.204 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/38' 2023-10-30T12:34:01.205 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/11' 2023-10-30T12:34:01.205 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/16' 2023-10-30T12:34:01.205 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/32' 2023-10-30T12:34:01.205 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/30' 2023-10-30T12:34:01.205 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/14' 2023-10-30T12:34:01.206 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/18' 2023-10-30T12:34:01.206 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/13' 2023-10-30T12:34:01.206 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/33' 2023-10-30T12:34:01.206 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/17' 2023-10-30T12:34:01.206 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/35' 2023-10-30T12:34:01.207 INFO:tasks.workunit.client.0.smithi125.stdout:removed './171/37' 2023-10-30T12:34:01.207 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './171' 2023-10-30T12:34:01.207 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/19' 2023-10-30T12:34:01.207 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/15' 2023-10-30T12:34:01.207 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/31' 2023-10-30T12:34:01.208 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/36' 2023-10-30T12:34:01.208 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/20' 2023-10-30T12:34:01.208 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/12' 2023-10-30T12:34:01.208 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/39' 2023-10-30T12:34:01.208 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/10' 2023-10-30T12:34:01.209 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/34' 2023-10-30T12:34:01.209 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/38' 2023-10-30T12:34:01.209 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/11' 2023-10-30T12:34:01.209 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/16' 2023-10-30T12:34:01.210 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/32' 2023-10-30T12:34:01.210 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/30' 2023-10-30T12:34:01.210 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/14' 2023-10-30T12:34:01.210 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/18' 2023-10-30T12:34:01.210 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/13' 2023-10-30T12:34:01.211 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/33' 2023-10-30T12:34:01.211 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/17' 2023-10-30T12:34:01.211 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/35' 2023-10-30T12:34:01.211 INFO:tasks.workunit.client.0.smithi125.stdout:removed './172/37' 2023-10-30T12:34:01.211 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './172' 2023-10-30T12:34:01.212 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/19' 2023-10-30T12:34:01.212 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/15' 2023-10-30T12:34:01.212 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/31' 2023-10-30T12:34:01.212 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/36' 2023-10-30T12:34:01.212 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/20' 2023-10-30T12:34:01.213 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/12' 2023-10-30T12:34:01.213 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/39' 2023-10-30T12:34:01.213 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/10' 2023-10-30T12:34:01.213 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/34' 2023-10-30T12:34:01.213 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/38' 2023-10-30T12:34:01.214 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/11' 2023-10-30T12:34:01.214 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/16' 2023-10-30T12:34:01.214 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/32' 2023-10-30T12:34:01.214 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/30' 2023-10-30T12:34:01.215 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/14' 2023-10-30T12:34:01.215 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/18' 2023-10-30T12:34:01.215 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/13' 2023-10-30T12:34:01.215 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/33' 2023-10-30T12:34:01.215 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/17' 2023-10-30T12:34:01.216 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/35' 2023-10-30T12:34:01.216 INFO:tasks.workunit.client.0.smithi125.stdout:removed './173/37' 2023-10-30T12:34:01.216 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './173' 2023-10-30T12:34:01.216 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/19' 2023-10-30T12:34:01.216 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/15' 2023-10-30T12:34:01.217 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/31' 2023-10-30T12:34:01.217 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/36' 2023-10-30T12:34:01.217 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/20' 2023-10-30T12:34:01.217 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/12' 2023-10-30T12:34:01.217 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/39' 2023-10-30T12:34:01.217 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/10' 2023-10-30T12:34:01.218 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/34' 2023-10-30T12:34:01.218 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/38' 2023-10-30T12:34:01.218 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/11' 2023-10-30T12:34:01.218 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/16' 2023-10-30T12:34:01.218 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/32' 2023-10-30T12:34:01.219 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/30' 2023-10-30T12:34:01.219 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/14' 2023-10-30T12:34:01.219 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/18' 2023-10-30T12:34:01.219 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/13' 2023-10-30T12:34:01.219 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/33' 2023-10-30T12:34:01.220 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/17' 2023-10-30T12:34:01.220 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/35' 2023-10-30T12:34:01.220 INFO:tasks.workunit.client.0.smithi125.stdout:removed './174/37' 2023-10-30T12:34:01.220 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './174' 2023-10-30T12:34:01.220 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/19' 2023-10-30T12:34:01.221 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/15' 2023-10-30T12:34:01.221 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/31' 2023-10-30T12:34:01.221 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/36' 2023-10-30T12:34:01.221 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/20' 2023-10-30T12:34:01.221 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/12' 2023-10-30T12:34:01.222 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/39' 2023-10-30T12:34:01.222 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/10' 2023-10-30T12:34:01.222 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/34' 2023-10-30T12:34:01.222 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/38' 2023-10-30T12:34:01.222 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/11' 2023-10-30T12:34:01.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/16' 2023-10-30T12:34:01.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/32' 2023-10-30T12:34:01.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/30' 2023-10-30T12:34:01.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/14' 2023-10-30T12:34:01.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/18' 2023-10-30T12:34:01.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/13' 2023-10-30T12:34:01.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/33' 2023-10-30T12:34:01.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/17' 2023-10-30T12:34:01.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/35' 2023-10-30T12:34:01.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './175/37' 2023-10-30T12:34:01.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './175' 2023-10-30T12:34:01.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/19' 2023-10-30T12:34:01.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/15' 2023-10-30T12:34:01.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/31' 2023-10-30T12:34:01.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/36' 2023-10-30T12:34:01.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/20' 2023-10-30T12:34:01.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/12' 2023-10-30T12:34:01.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/39' 2023-10-30T12:34:01.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/10' 2023-10-30T12:34:01.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/34' 2023-10-30T12:34:01.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/38' 2023-10-30T12:34:01.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/11' 2023-10-30T12:34:01.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/16' 2023-10-30T12:34:01.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/32' 2023-10-30T12:34:01.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/30' 2023-10-30T12:34:01.228 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/14' 2023-10-30T12:34:01.228 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/18' 2023-10-30T12:34:01.228 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/13' 2023-10-30T12:34:01.228 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/33' 2023-10-30T12:34:01.229 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/17' 2023-10-30T12:34:01.229 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/35' 2023-10-30T12:34:01.229 INFO:tasks.workunit.client.0.smithi125.stdout:removed './176/37' 2023-10-30T12:34:01.229 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './176' 2023-10-30T12:34:01.229 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/19' 2023-10-30T12:34:01.230 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/15' 2023-10-30T12:34:02.897 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/31' 2023-10-30T12:34:02.897 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/36' 2023-10-30T12:34:02.897 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/20' 2023-10-30T12:34:02.898 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/12' 2023-10-30T12:34:02.898 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/39' 2023-10-30T12:34:02.898 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/10' 2023-10-30T12:34:02.898 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/34' 2023-10-30T12:34:02.898 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/38' 2023-10-30T12:34:02.899 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/11' 2023-10-30T12:34:02.899 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/16' 2023-10-30T12:34:02.899 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/32' 2023-10-30T12:34:02.899 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/30' 2023-10-30T12:34:02.899 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/14' 2023-10-30T12:34:02.900 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/18' 2023-10-30T12:34:02.900 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/13' 2023-10-30T12:34:02.900 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/33' 2023-10-30T12:34:02.900 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/17' 2023-10-30T12:34:02.900 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/35' 2023-10-30T12:34:02.901 INFO:tasks.workunit.client.0.smithi125.stdout:removed './177/37' 2023-10-30T12:34:02.901 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './177' 2023-10-30T12:34:02.901 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/19' 2023-10-30T12:34:02.901 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/15' 2023-10-30T12:34:02.901 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/31' 2023-10-30T12:34:02.902 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/36' 2023-10-30T12:34:02.902 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/20' 2023-10-30T12:34:02.902 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/12' 2023-10-30T12:34:02.902 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/39' 2023-10-30T12:34:02.903 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/10' 2023-10-30T12:34:02.903 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/34' 2023-10-30T12:34:02.903 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/38' 2023-10-30T12:34:02.903 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/11' 2023-10-30T12:34:02.904 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/16' 2023-10-30T12:34:02.904 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/32' 2023-10-30T12:34:02.904 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/30' 2023-10-30T12:34:02.904 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/14' 2023-10-30T12:34:02.905 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/18' 2023-10-30T12:34:02.905 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/13' 2023-10-30T12:34:02.906 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/33' 2023-10-30T12:34:02.906 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/17' 2023-10-30T12:34:02.906 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/35' 2023-10-30T12:34:02.906 INFO:tasks.workunit.client.0.smithi125.stdout:removed './178/37' 2023-10-30T12:34:02.906 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './178' 2023-10-30T12:34:02.907 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/19' 2023-10-30T12:34:02.907 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/15' 2023-10-30T12:34:02.907 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/31' 2023-10-30T12:34:02.907 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/36' 2023-10-30T12:34:02.908 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/20' 2023-10-30T12:34:02.908 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/12' 2023-10-30T12:34:02.908 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/39' 2023-10-30T12:34:02.908 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/10' 2023-10-30T12:34:02.908 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/34' 2023-10-30T12:34:02.909 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/38' 2023-10-30T12:34:02.910 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/11' 2023-10-30T12:34:02.910 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/16' 2023-10-30T12:34:02.910 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/32' 2023-10-30T12:34:02.910 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/30' 2023-10-30T12:34:02.911 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/14' 2023-10-30T12:34:02.911 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/18' 2023-10-30T12:34:02.911 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/13' 2023-10-30T12:34:02.911 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/33' 2023-10-30T12:34:02.911 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/17' 2023-10-30T12:34:02.911 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/35' 2023-10-30T12:34:02.912 INFO:tasks.workunit.client.0.smithi125.stdout:removed './179/37' 2023-10-30T12:34:02.912 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './179' 2023-10-30T12:34:02.912 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/19' 2023-10-30T12:34:02.912 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/15' 2023-10-30T12:34:02.912 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/31' 2023-10-30T12:34:02.913 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/36' 2023-10-30T12:34:02.913 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/20' 2023-10-30T12:34:02.913 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/12' 2023-10-30T12:34:02.913 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/39' 2023-10-30T12:34:02.913 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/10' 2023-10-30T12:34:02.914 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/34' 2023-10-30T12:34:02.914 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/38' 2023-10-30T12:34:02.914 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/11' 2023-10-30T12:34:02.914 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/16' 2023-10-30T12:34:02.914 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/32' 2023-10-30T12:34:02.915 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/30' 2023-10-30T12:34:02.915 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/14' 2023-10-30T12:34:02.915 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/18' 2023-10-30T12:34:02.916 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/13' 2023-10-30T12:34:02.916 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/33' 2023-10-30T12:34:02.916 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/17' 2023-10-30T12:34:02.916 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/35' 2023-10-30T12:34:02.917 INFO:tasks.workunit.client.0.smithi125.stdout:removed './180/37' 2023-10-30T12:34:02.917 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './180' 2023-10-30T12:34:02.917 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/19' 2023-10-30T12:34:02.917 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/15' 2023-10-30T12:34:02.917 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/31' 2023-10-30T12:34:02.918 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/36' 2023-10-30T12:34:02.918 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/20' 2023-10-30T12:34:02.918 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/12' 2023-10-30T12:34:02.918 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/39' 2023-10-30T12:34:02.918 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/10' 2023-10-30T12:34:02.919 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/34' 2023-10-30T12:34:02.919 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/38' 2023-10-30T12:34:02.919 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/11' 2023-10-30T12:34:02.919 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/16' 2023-10-30T12:34:02.919 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/32' 2023-10-30T12:34:02.920 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/30' 2023-10-30T12:34:02.920 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/14' 2023-10-30T12:34:02.920 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/18' 2023-10-30T12:34:02.920 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/13' 2023-10-30T12:34:02.920 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/33' 2023-10-30T12:34:02.921 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/17' 2023-10-30T12:34:02.921 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/35' 2023-10-30T12:34:02.921 INFO:tasks.workunit.client.0.smithi125.stdout:removed './181/37' 2023-10-30T12:34:02.921 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './181' 2023-10-30T12:34:02.922 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/19' 2023-10-30T12:34:02.922 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/15' 2023-10-30T12:34:02.923 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/31' 2023-10-30T12:34:02.923 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/36' 2023-10-30T12:34:02.923 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/20' 2023-10-30T12:34:02.923 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/12' 2023-10-30T12:34:02.924 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/39' 2023-10-30T12:34:02.924 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/10' 2023-10-30T12:34:02.924 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/34' 2023-10-30T12:34:02.924 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/38' 2023-10-30T12:34:02.924 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/11' 2023-10-30T12:34:02.925 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/16' 2023-10-30T12:34:02.925 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/32' 2023-10-30T12:34:02.925 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/30' 2023-10-30T12:34:02.925 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/14' 2023-10-30T12:34:02.926 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/18' 2023-10-30T12:34:02.926 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/13' 2023-10-30T12:34:02.926 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/33' 2023-10-30T12:34:02.926 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/17' 2023-10-30T12:34:02.926 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/35' 2023-10-30T12:34:02.927 INFO:tasks.workunit.client.0.smithi125.stdout:removed './182/37' 2023-10-30T12:34:02.927 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './182' 2023-10-30T12:34:02.927 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/19' 2023-10-30T12:34:02.927 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/15' 2023-10-30T12:34:02.927 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/31' 2023-10-30T12:34:02.928 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/36' 2023-10-30T12:34:02.928 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/20' 2023-10-30T12:34:02.928 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/12' 2023-10-30T12:34:02.928 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/39' 2023-10-30T12:34:02.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/10' 2023-10-30T12:34:02.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/34' 2023-10-30T12:34:02.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/38' 2023-10-30T12:34:02.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/11' 2023-10-30T12:34:02.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/16' 2023-10-30T12:34:02.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/32' 2023-10-30T12:34:02.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/30' 2023-10-30T12:34:02.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/14' 2023-10-30T12:34:02.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/18' 2023-10-30T12:34:02.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/13' 2023-10-30T12:34:02.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/33' 2023-10-30T12:34:02.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/17' 2023-10-30T12:34:02.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/35' 2023-10-30T12:34:02.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed './183/37' 2023-10-30T12:34:02.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './183' 2023-10-30T12:34:02.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/19' 2023-10-30T12:34:02.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/15' 2023-10-30T12:34:02.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/31' 2023-10-30T12:34:02.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/36' 2023-10-30T12:34:02.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/20' 2023-10-30T12:34:02.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/12' 2023-10-30T12:34:02.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/39' 2023-10-30T12:34:02.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/10' 2023-10-30T12:34:02.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/34' 2023-10-30T12:34:02.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/38' 2023-10-30T12:34:02.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/11' 2023-10-30T12:34:02.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/16' 2023-10-30T12:34:02.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/32' 2023-10-30T12:34:02.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/30' 2023-10-30T12:34:02.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/14' 2023-10-30T12:34:02.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/18' 2023-10-30T12:34:02.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/13' 2023-10-30T12:34:02.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/33' 2023-10-30T12:34:02.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/17' 2023-10-30T12:34:02.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/35' 2023-10-30T12:34:02.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed './184/37' 2023-10-30T12:34:02.936 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './184' 2023-10-30T12:34:02.936 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/19' 2023-10-30T12:34:02.936 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/15' 2023-10-30T12:34:02.936 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/31' 2023-10-30T12:34:02.936 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/36' 2023-10-30T12:34:02.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/20' 2023-10-30T12:34:02.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/12' 2023-10-30T12:34:02.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/39' 2023-10-30T12:34:02.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/10' 2023-10-30T12:34:02.938 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/34' 2023-10-30T12:34:02.938 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/38' 2023-10-30T12:34:02.938 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/11' 2023-10-30T12:34:02.938 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/16' 2023-10-30T12:34:02.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/32' 2023-10-30T12:34:02.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/30' 2023-10-30T12:34:02.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/14' 2023-10-30T12:34:02.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/18' 2023-10-30T12:34:02.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/13' 2023-10-30T12:34:02.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/33' 2023-10-30T12:34:02.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/17' 2023-10-30T12:34:02.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/35' 2023-10-30T12:34:02.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed './185/37' 2023-10-30T12:34:02.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './185' 2023-10-30T12:34:02.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/19' 2023-10-30T12:34:02.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/15' 2023-10-30T12:34:02.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/31' 2023-10-30T12:34:02.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/36' 2023-10-30T12:34:02.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/20' 2023-10-30T12:34:02.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/12' 2023-10-30T12:34:02.942 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/39' 2023-10-30T12:34:02.942 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/10' 2023-10-30T12:34:02.942 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/34' 2023-10-30T12:34:02.942 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/38' 2023-10-30T12:34:02.942 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/11' 2023-10-30T12:34:02.943 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/16' 2023-10-30T12:34:02.943 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/32' 2023-10-30T12:34:02.943 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/30' 2023-10-30T12:34:02.943 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/14' 2023-10-30T12:34:02.943 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/18' 2023-10-30T12:34:06.961 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/13' 2023-10-30T12:34:06.961 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/33' 2023-10-30T12:34:06.961 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/17' 2023-10-30T12:34:06.962 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/35' 2023-10-30T12:34:06.962 INFO:tasks.workunit.client.0.smithi125.stdout:removed './186/37' 2023-10-30T12:34:06.962 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './186' 2023-10-30T12:34:06.962 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/19' 2023-10-30T12:34:06.963 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/15' 2023-10-30T12:34:06.963 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/31' 2023-10-30T12:34:06.963 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/36' 2023-10-30T12:34:06.963 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/20' 2023-10-30T12:34:06.963 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/12' 2023-10-30T12:34:06.963 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/39' 2023-10-30T12:34:06.964 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/10' 2023-10-30T12:34:06.964 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/34' 2023-10-30T12:34:06.964 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/38' 2023-10-30T12:34:06.964 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/11' 2023-10-30T12:34:06.964 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/16' 2023-10-30T12:34:06.965 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/32' 2023-10-30T12:34:06.965 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/30' 2023-10-30T12:34:06.965 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/14' 2023-10-30T12:34:06.965 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/18' 2023-10-30T12:34:06.965 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/13' 2023-10-30T12:34:06.966 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/33' 2023-10-30T12:34:06.966 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/17' 2023-10-30T12:34:06.966 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/35' 2023-10-30T12:34:06.966 INFO:tasks.workunit.client.0.smithi125.stdout:removed './187/37' 2023-10-30T12:34:06.966 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './187' 2023-10-30T12:34:06.967 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/19' 2023-10-30T12:34:06.967 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/15' 2023-10-30T12:34:06.967 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/31' 2023-10-30T12:34:06.967 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/36' 2023-10-30T12:34:06.967 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/20' 2023-10-30T12:34:06.968 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/12' 2023-10-30T12:34:06.968 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/39' 2023-10-30T12:34:06.968 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/10' 2023-10-30T12:34:06.968 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/34' 2023-10-30T12:34:06.968 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/38' 2023-10-30T12:34:06.969 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/11' 2023-10-30T12:34:06.969 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/16' 2023-10-30T12:34:06.969 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/32' 2023-10-30T12:34:06.969 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/30' 2023-10-30T12:34:06.969 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/14' 2023-10-30T12:34:06.969 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/18' 2023-10-30T12:34:06.970 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/13' 2023-10-30T12:34:06.970 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/33' 2023-10-30T12:34:06.970 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/17' 2023-10-30T12:34:06.970 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/35' 2023-10-30T12:34:06.970 INFO:tasks.workunit.client.0.smithi125.stdout:removed './188/37' 2023-10-30T12:34:06.971 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './188' 2023-10-30T12:34:06.971 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/19' 2023-10-30T12:34:06.971 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/15' 2023-10-30T12:34:06.971 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/31' 2023-10-30T12:34:06.972 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/36' 2023-10-30T12:34:06.972 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/20' 2023-10-30T12:34:06.972 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/12' 2023-10-30T12:34:06.972 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/39' 2023-10-30T12:34:06.972 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/10' 2023-10-30T12:34:06.973 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/34' 2023-10-30T12:34:06.973 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/38' 2023-10-30T12:34:06.973 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/11' 2023-10-30T12:34:06.973 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/16' 2023-10-30T12:34:06.973 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/32' 2023-10-30T12:34:06.973 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/30' 2023-10-30T12:34:06.974 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/14' 2023-10-30T12:34:06.974 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/18' 2023-10-30T12:34:06.974 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/13' 2023-10-30T12:34:06.974 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/33' 2023-10-30T12:34:06.975 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/17' 2023-10-30T12:34:06.975 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/35' 2023-10-30T12:34:06.975 INFO:tasks.workunit.client.0.smithi125.stdout:removed './189/37' 2023-10-30T12:34:06.975 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './189' 2023-10-30T12:34:06.975 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/19' 2023-10-30T12:34:06.976 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/15' 2023-10-30T12:34:06.976 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/31' 2023-10-30T12:34:06.976 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/36' 2023-10-30T12:34:06.976 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/20' 2023-10-30T12:34:06.976 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/12' 2023-10-30T12:34:06.977 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/39' 2023-10-30T12:34:06.977 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/10' 2023-10-30T12:34:06.977 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/34' 2023-10-30T12:34:06.977 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/38' 2023-10-30T12:34:06.977 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/11' 2023-10-30T12:34:06.978 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/16' 2023-10-30T12:34:06.978 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/32' 2023-10-30T12:34:06.978 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/30' 2023-10-30T12:34:06.978 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/14' 2023-10-30T12:34:06.978 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/18' 2023-10-30T12:34:06.979 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/13' 2023-10-30T12:34:06.979 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/33' 2023-10-30T12:34:06.979 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/17' 2023-10-30T12:34:06.979 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/35' 2023-10-30T12:34:06.979 INFO:tasks.workunit.client.0.smithi125.stdout:removed './190/37' 2023-10-30T12:34:06.979 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './190' 2023-10-30T12:34:06.980 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/19' 2023-10-30T12:34:06.980 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/15' 2023-10-30T12:34:06.980 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/31' 2023-10-30T12:34:06.980 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/36' 2023-10-30T12:34:06.980 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/20' 2023-10-30T12:34:06.981 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/12' 2023-10-30T12:34:06.981 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/39' 2023-10-30T12:34:06.981 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/10' 2023-10-30T12:34:06.981 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/34' 2023-10-30T12:34:06.981 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/38' 2023-10-30T12:34:06.981 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/11' 2023-10-30T12:34:06.982 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/16' 2023-10-30T12:34:06.982 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/32' 2023-10-30T12:34:06.982 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/30' 2023-10-30T12:34:06.982 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/14' 2023-10-30T12:34:06.982 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/18' 2023-10-30T12:34:06.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/13' 2023-10-30T12:34:06.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/33' 2023-10-30T12:34:06.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/17' 2023-10-30T12:34:06.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/35' 2023-10-30T12:34:06.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './191/37' 2023-10-30T12:34:06.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './191' 2023-10-30T12:34:06.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/19' 2023-10-30T12:34:06.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/15' 2023-10-30T12:34:06.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/31' 2023-10-30T12:34:06.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/36' 2023-10-30T12:34:06.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/20' 2023-10-30T12:34:06.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/12' 2023-10-30T12:34:06.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/39' 2023-10-30T12:34:06.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/10' 2023-10-30T12:34:06.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/34' 2023-10-30T12:34:06.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/38' 2023-10-30T12:34:06.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/11' 2023-10-30T12:34:06.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/16' 2023-10-30T12:34:06.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/32' 2023-10-30T12:34:06.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/30' 2023-10-30T12:34:06.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/14' 2023-10-30T12:34:06.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/18' 2023-10-30T12:34:06.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/13' 2023-10-30T12:34:06.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/33' 2023-10-30T12:34:06.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/17' 2023-10-30T12:34:06.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/35' 2023-10-30T12:34:06.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed './192/37' 2023-10-30T12:34:06.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './192' 2023-10-30T12:34:06.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/19' 2023-10-30T12:34:06.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/15' 2023-10-30T12:34:06.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/31' 2023-10-30T12:34:06.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/36' 2023-10-30T12:34:06.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/20' 2023-10-30T12:34:06.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/12' 2023-10-30T12:34:06.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/39' 2023-10-30T12:34:06.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/10' 2023-10-30T12:34:06.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/34' 2023-10-30T12:34:06.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/38' 2023-10-30T12:34:06.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/11' 2023-10-30T12:34:06.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/16' 2023-10-30T12:34:06.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/32' 2023-10-30T12:34:06.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/30' 2023-10-30T12:34:06.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/14' 2023-10-30T12:34:06.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/18' 2023-10-30T12:34:06.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/13' 2023-10-30T12:34:06.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/33' 2023-10-30T12:34:06.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/17' 2023-10-30T12:34:06.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/35' 2023-10-30T12:34:06.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed './193/37' 2023-10-30T12:34:06.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './193' 2023-10-30T12:34:06.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/19' 2023-10-30T12:34:06.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/15' 2023-10-30T12:34:06.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/31' 2023-10-30T12:34:06.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/36' 2023-10-30T12:34:06.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/20' 2023-10-30T12:34:06.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/12' 2023-10-30T12:34:06.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/39' 2023-10-30T12:34:06.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/10' 2023-10-30T12:34:06.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/34' 2023-10-30T12:34:06.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/38' 2023-10-30T12:34:06.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/11' 2023-10-30T12:34:06.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/16' 2023-10-30T12:34:06.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/32' 2023-10-30T12:34:06.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/30' 2023-10-30T12:34:06.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/14' 2023-10-30T12:34:06.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/18' 2023-10-30T12:34:06.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/13' 2023-10-30T12:34:06.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/33' 2023-10-30T12:34:06.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/17' 2023-10-30T12:34:06.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/35' 2023-10-30T12:34:06.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed './194/37' 2023-10-30T12:34:06.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './194' 2023-10-30T12:34:06.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/19' 2023-10-30T12:34:06.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/15' 2023-10-30T12:34:06.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/31' 2023-10-30T12:34:06.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/36' 2023-10-30T12:34:06.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/20' 2023-10-30T12:34:06.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/12' 2023-10-30T12:34:06.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/39' 2023-10-30T12:34:06.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/10' 2023-10-30T12:34:06.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/34' 2023-10-30T12:34:06.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/38' 2023-10-30T12:34:06.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/11' 2023-10-30T12:34:06.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/16' 2023-10-30T12:34:06.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/32' 2023-10-30T12:34:06.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/30' 2023-10-30T12:34:06.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/14' 2023-10-30T12:34:06.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/18' 2023-10-30T12:34:06.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/13' 2023-10-30T12:34:07.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/33' 2023-10-30T12:34:07.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/17' 2023-10-30T12:34:07.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/35' 2023-10-30T12:34:07.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './195/37' 2023-10-30T12:34:07.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './195' 2023-10-30T12:34:07.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/19' 2023-10-30T12:34:07.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/15' 2023-10-30T12:34:07.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/31' 2023-10-30T12:34:07.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/36' 2023-10-30T12:34:07.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/20' 2023-10-30T12:34:07.002 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/12' 2023-10-30T12:34:07.002 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/39' 2023-10-30T12:34:07.002 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/10' 2023-10-30T12:34:08.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/34' 2023-10-30T12:34:08.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/38' 2023-10-30T12:34:08.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/11' 2023-10-30T12:34:08.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/16' 2023-10-30T12:34:08.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/32' 2023-10-30T12:34:08.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/30' 2023-10-30T12:34:08.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/14' 2023-10-30T12:34:08.988 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/18' 2023-10-30T12:34:08.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/13' 2023-10-30T12:34:08.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/33' 2023-10-30T12:34:08.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/17' 2023-10-30T12:34:08.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/35' 2023-10-30T12:34:08.989 INFO:tasks.workunit.client.0.smithi125.stdout:removed './196/37' 2023-10-30T12:34:08.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './196' 2023-10-30T12:34:08.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/19' 2023-10-30T12:34:08.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/15' 2023-10-30T12:34:08.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/31' 2023-10-30T12:34:08.990 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/36' 2023-10-30T12:34:08.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/20' 2023-10-30T12:34:08.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/12' 2023-10-30T12:34:08.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/39' 2023-10-30T12:34:08.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/10' 2023-10-30T12:34:08.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/34' 2023-10-30T12:34:08.991 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/38' 2023-10-30T12:34:08.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/11' 2023-10-30T12:34:08.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/16' 2023-10-30T12:34:08.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/32' 2023-10-30T12:34:08.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/30' 2023-10-30T12:34:08.992 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/14' 2023-10-30T12:34:08.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/18' 2023-10-30T12:34:08.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/13' 2023-10-30T12:34:08.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/33' 2023-10-30T12:34:08.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/17' 2023-10-30T12:34:08.993 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/35' 2023-10-30T12:34:08.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed './197/37' 2023-10-30T12:34:08.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './197' 2023-10-30T12:34:08.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/19' 2023-10-30T12:34:08.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/15' 2023-10-30T12:34:08.994 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/31' 2023-10-30T12:34:08.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/36' 2023-10-30T12:34:08.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/20' 2023-10-30T12:34:08.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/12' 2023-10-30T12:34:08.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/39' 2023-10-30T12:34:08.995 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/10' 2023-10-30T12:34:08.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/34' 2023-10-30T12:34:08.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/38' 2023-10-30T12:34:08.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/11' 2023-10-30T12:34:08.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/16' 2023-10-30T12:34:08.996 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/32' 2023-10-30T12:34:08.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/30' 2023-10-30T12:34:08.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/14' 2023-10-30T12:34:08.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/18' 2023-10-30T12:34:08.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/13' 2023-10-30T12:34:08.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/33' 2023-10-30T12:34:08.997 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/17' 2023-10-30T12:34:08.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/35' 2023-10-30T12:34:08.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './198/37' 2023-10-30T12:34:08.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './198' 2023-10-30T12:34:08.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/19' 2023-10-30T12:34:08.998 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/15' 2023-10-30T12:34:08.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/31' 2023-10-30T12:34:08.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/36' 2023-10-30T12:34:08.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/20' 2023-10-30T12:34:08.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/12' 2023-10-30T12:34:08.999 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/39' 2023-10-30T12:34:09.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/10' 2023-10-30T12:34:09.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/34' 2023-10-30T12:34:09.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/38' 2023-10-30T12:34:09.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/11' 2023-10-30T12:34:09.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/16' 2023-10-30T12:34:09.000 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/32' 2023-10-30T12:34:09.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/30' 2023-10-30T12:34:09.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/14' 2023-10-30T12:34:09.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/18' 2023-10-30T12:34:09.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/13' 2023-10-30T12:34:09.001 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/33' 2023-10-30T12:34:09.002 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/17' 2023-10-30T12:34:09.002 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/35' 2023-10-30T12:34:09.002 INFO:tasks.workunit.client.0.smithi125.stdout:removed './199/37' 2023-10-30T12:34:09.002 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './199' 2023-10-30T12:34:09.002 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/21' 2023-10-30T12:34:09.003 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/27' 2023-10-30T12:34:09.003 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/31' 2023-10-30T12:34:09.003 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/36' 2023-10-30T12:34:09.003 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/20' 2023-10-30T12:34:09.003 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/39' 2023-10-30T12:34:09.003 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/34' 2023-10-30T12:34:09.004 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/38' 2023-10-30T12:34:09.004 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/28' 2023-10-30T12:34:09.004 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/24' 2023-10-30T12:34:09.004 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/29' 2023-10-30T12:34:09.004 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/32' 2023-10-30T12:34:09.005 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/22' 2023-10-30T12:34:09.005 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/25' 2023-10-30T12:34:09.005 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/30' 2023-10-30T12:34:09.005 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/33' 2023-10-30T12:34:09.005 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/23' 2023-10-30T12:34:09.006 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/35' 2023-10-30T12:34:09.006 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/26' 2023-10-30T12:34:09.006 INFO:tasks.workunit.client.0.smithi125.stdout:removed './200/37' 2023-10-30T12:34:09.006 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './200' 2023-10-30T12:34:09.007 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/21' 2023-10-30T12:34:09.007 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/27' 2023-10-30T12:34:09.007 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/31' 2023-10-30T12:34:09.007 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/36' 2023-10-30T12:34:09.007 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/20' 2023-10-30T12:34:09.008 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/39' 2023-10-30T12:34:09.008 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/34' 2023-10-30T12:34:09.008 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/38' 2023-10-30T12:34:09.008 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/28' 2023-10-30T12:34:09.008 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/24' 2023-10-30T12:34:09.008 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/29' 2023-10-30T12:34:09.009 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/32' 2023-10-30T12:34:09.009 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/22' 2023-10-30T12:34:09.009 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/25' 2023-10-30T12:34:09.009 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/30' 2023-10-30T12:34:09.009 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/33' 2023-10-30T12:34:09.010 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/23' 2023-10-30T12:34:09.010 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/35' 2023-10-30T12:34:09.010 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/26' 2023-10-30T12:34:09.010 INFO:tasks.workunit.client.0.smithi125.stdout:removed './201/37' 2023-10-30T12:34:09.010 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './201' 2023-10-30T12:34:09.011 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/21' 2023-10-30T12:34:09.011 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/27' 2023-10-30T12:34:09.011 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/31' 2023-10-30T12:34:09.011 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/36' 2023-10-30T12:34:09.011 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/20' 2023-10-30T12:34:09.011 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/39' 2023-10-30T12:34:09.012 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/34' 2023-10-30T12:34:09.012 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/38' 2023-10-30T12:34:09.012 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/28' 2023-10-30T12:34:09.012 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/24' 2023-10-30T12:34:09.012 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/29' 2023-10-30T12:34:09.013 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/32' 2023-10-30T12:34:09.013 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/22' 2023-10-30T12:34:09.013 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/25' 2023-10-30T12:34:09.013 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/30' 2023-10-30T12:34:09.013 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/33' 2023-10-30T12:34:09.013 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/23' 2023-10-30T12:34:09.014 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/35' 2023-10-30T12:34:09.014 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/26' 2023-10-30T12:34:09.014 INFO:tasks.workunit.client.0.smithi125.stdout:removed './202/37' 2023-10-30T12:34:09.014 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './202' 2023-10-30T12:34:09.014 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/21' 2023-10-30T12:34:09.015 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/27' 2023-10-30T12:34:09.015 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/31' 2023-10-30T12:34:09.015 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/36' 2023-10-30T12:34:09.015 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/20' 2023-10-30T12:34:09.015 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/39' 2023-10-30T12:34:09.016 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/34' 2023-10-30T12:34:09.016 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/38' 2023-10-30T12:34:09.016 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/28' 2023-10-30T12:34:09.016 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/24' 2023-10-30T12:34:09.016 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/29' 2023-10-30T12:34:09.016 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/32' 2023-10-30T12:34:09.017 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/22' 2023-10-30T12:34:09.017 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/25' 2023-10-30T12:34:09.017 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/30' 2023-10-30T12:34:09.017 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/33' 2023-10-30T12:34:09.017 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/23' 2023-10-30T12:34:09.018 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/35' 2023-10-30T12:34:09.018 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/26' 2023-10-30T12:34:09.018 INFO:tasks.workunit.client.0.smithi125.stdout:removed './203/37' 2023-10-30T12:34:09.018 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './203' 2023-10-30T12:34:09.018 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/21' 2023-10-30T12:34:09.019 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/27' 2023-10-30T12:34:09.019 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/31' 2023-10-30T12:34:09.019 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/36' 2023-10-30T12:34:09.019 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/20' 2023-10-30T12:34:09.019 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/39' 2023-10-30T12:34:09.019 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/34' 2023-10-30T12:34:09.020 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/38' 2023-10-30T12:34:09.020 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/28' 2023-10-30T12:34:09.020 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/24' 2023-10-30T12:34:09.020 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/29' 2023-10-30T12:34:09.020 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/32' 2023-10-30T12:34:09.021 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/22' 2023-10-30T12:34:09.021 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/25' 2023-10-30T12:34:09.021 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/30' 2023-10-30T12:34:09.021 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/33' 2023-10-30T12:34:09.021 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/23' 2023-10-30T12:34:09.021 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/35' 2023-10-30T12:34:09.022 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/26' 2023-10-30T12:34:09.022 INFO:tasks.workunit.client.0.smithi125.stdout:removed './204/37' 2023-10-30T12:34:09.022 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './204' 2023-10-30T12:34:09.022 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/21' 2023-10-30T12:34:09.022 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/27' 2023-10-30T12:34:09.023 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/31' 2023-10-30T12:34:09.023 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/36' 2023-10-30T12:34:09.023 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/20' 2023-10-30T12:34:09.023 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/39' 2023-10-30T12:34:09.023 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/34' 2023-10-30T12:34:09.024 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/38' 2023-10-30T12:34:09.024 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/28' 2023-10-30T12:34:09.025 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/24' 2023-10-30T12:34:09.025 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/29' 2023-10-30T12:34:09.025 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/32' 2023-10-30T12:34:09.025 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/22' 2023-10-30T12:34:09.025 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/25' 2023-10-30T12:34:09.026 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/30' 2023-10-30T12:34:09.026 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/33' 2023-10-30T12:34:09.026 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/23' 2023-10-30T12:34:09.026 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/35' 2023-10-30T12:34:09.026 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/26' 2023-10-30T12:34:09.027 INFO:tasks.workunit.client.0.smithi125.stdout:removed './205/37' 2023-10-30T12:34:09.027 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './205' 2023-10-30T12:34:09.027 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/21' 2023-10-30T12:34:09.027 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/27' 2023-10-30T12:34:09.027 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/31' 2023-10-30T12:34:09.028 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/36' 2023-10-30T12:34:09.028 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/20' 2023-10-30T12:34:09.028 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/39' 2023-10-30T12:34:12.844 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/34' 2023-10-30T12:34:12.845 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/38' 2023-10-30T12:34:12.845 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/28' 2023-10-30T12:34:12.845 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/24' 2023-10-30T12:34:12.846 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/29' 2023-10-30T12:34:12.846 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/32' 2023-10-30T12:34:12.846 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/22' 2023-10-30T12:34:12.846 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/25' 2023-10-30T12:34:12.847 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/30' 2023-10-30T12:34:12.847 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/33' 2023-10-30T12:34:12.847 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/23' 2023-10-30T12:34:12.847 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/35' 2023-10-30T12:34:12.847 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/26' 2023-10-30T12:34:12.848 INFO:tasks.workunit.client.0.smithi125.stdout:removed './206/37' 2023-10-30T12:34:12.848 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './206' 2023-10-30T12:34:12.848 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/21' 2023-10-30T12:34:12.848 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/27' 2023-10-30T12:34:12.848 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/31' 2023-10-30T12:34:12.849 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/36' 2023-10-30T12:34:12.849 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/20' 2023-10-30T12:34:12.849 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/39' 2023-10-30T12:34:12.849 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/34' 2023-10-30T12:34:12.849 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/38' 2023-10-30T12:34:12.850 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/28' 2023-10-30T12:34:12.850 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/24' 2023-10-30T12:34:12.850 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/29' 2023-10-30T12:34:12.850 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/32' 2023-10-30T12:34:12.851 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/22' 2023-10-30T12:34:12.851 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/25' 2023-10-30T12:34:12.851 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/30' 2023-10-30T12:34:12.851 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/33' 2023-10-30T12:34:12.851 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/23' 2023-10-30T12:34:12.851 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/35' 2023-10-30T12:34:12.852 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/26' 2023-10-30T12:34:12.852 INFO:tasks.workunit.client.0.smithi125.stdout:removed './207/37' 2023-10-30T12:34:12.852 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './207' 2023-10-30T12:34:12.852 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/21' 2023-10-30T12:34:12.852 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/27' 2023-10-30T12:34:12.853 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/31' 2023-10-30T12:34:12.853 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/36' 2023-10-30T12:34:12.853 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/20' 2023-10-30T12:34:12.854 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/39' 2023-10-30T12:34:12.854 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/34' 2023-10-30T12:34:12.854 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/38' 2023-10-30T12:34:12.854 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/28' 2023-10-30T12:34:12.854 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/24' 2023-10-30T12:34:12.855 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/29' 2023-10-30T12:34:12.855 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/32' 2023-10-30T12:34:12.855 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/22' 2023-10-30T12:34:12.855 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/25' 2023-10-30T12:34:12.856 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/30' 2023-10-30T12:34:12.856 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/33' 2023-10-30T12:34:12.856 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/23' 2023-10-30T12:34:12.856 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/35' 2023-10-30T12:34:12.857 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/26' 2023-10-30T12:34:12.857 INFO:tasks.workunit.client.0.smithi125.stdout:removed './208/37' 2023-10-30T12:34:12.857 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './208' 2023-10-30T12:34:12.857 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/21' 2023-10-30T12:34:12.857 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/27' 2023-10-30T12:34:12.858 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/31' 2023-10-30T12:34:12.858 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/36' 2023-10-30T12:34:12.858 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/20' 2023-10-30T12:34:12.858 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/39' 2023-10-30T12:34:12.858 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/34' 2023-10-30T12:34:12.859 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/38' 2023-10-30T12:34:12.859 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/28' 2023-10-30T12:34:12.859 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/24' 2023-10-30T12:34:12.859 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/29' 2023-10-30T12:34:12.859 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/32' 2023-10-30T12:34:12.860 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/22' 2023-10-30T12:34:12.860 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/25' 2023-10-30T12:34:12.860 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/30' 2023-10-30T12:34:12.860 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/33' 2023-10-30T12:34:12.860 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/23' 2023-10-30T12:34:12.861 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/35' 2023-10-30T12:34:12.861 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/26' 2023-10-30T12:34:12.861 INFO:tasks.workunit.client.0.smithi125.stdout:removed './209/37' 2023-10-30T12:34:12.861 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './209' 2023-10-30T12:34:12.861 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/21' 2023-10-30T12:34:12.862 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/27' 2023-10-30T12:34:12.862 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/31' 2023-10-30T12:34:12.862 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/36' 2023-10-30T12:34:12.862 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/20' 2023-10-30T12:34:12.862 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/39' 2023-10-30T12:34:12.863 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/34' 2023-10-30T12:34:12.863 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/38' 2023-10-30T12:34:12.863 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/28' 2023-10-30T12:34:12.863 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/24' 2023-10-30T12:34:12.863 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/29' 2023-10-30T12:34:12.864 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/32' 2023-10-30T12:34:12.864 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/22' 2023-10-30T12:34:12.864 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/25' 2023-10-30T12:34:12.864 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/30' 2023-10-30T12:34:12.864 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/33' 2023-10-30T12:34:12.864 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/23' 2023-10-30T12:34:12.865 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/35' 2023-10-30T12:34:12.865 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/26' 2023-10-30T12:34:12.865 INFO:tasks.workunit.client.0.smithi125.stdout:removed './210/37' 2023-10-30T12:34:12.865 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './210' 2023-10-30T12:34:12.866 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/21' 2023-10-30T12:34:12.866 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/27' 2023-10-30T12:34:12.866 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/31' 2023-10-30T12:34:12.866 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/36' 2023-10-30T12:34:12.866 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/20' 2023-10-30T12:34:12.867 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/39' 2023-10-30T12:34:12.867 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/34' 2023-10-30T12:34:12.867 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/38' 2023-10-30T12:34:12.867 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/28' 2023-10-30T12:34:12.867 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/24' 2023-10-30T12:34:12.868 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/29' 2023-10-30T12:34:12.868 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/32' 2023-10-30T12:34:12.868 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/22' 2023-10-30T12:34:12.868 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/25' 2023-10-30T12:34:12.868 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/30' 2023-10-30T12:34:12.868 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/33' 2023-10-30T12:34:12.869 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/23' 2023-10-30T12:34:12.869 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/35' 2023-10-30T12:34:12.869 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/26' 2023-10-30T12:34:12.869 INFO:tasks.workunit.client.0.smithi125.stdout:removed './211/37' 2023-10-30T12:34:12.869 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './211' 2023-10-30T12:34:12.870 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/21' 2023-10-30T12:34:12.870 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/27' 2023-10-30T12:34:12.870 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/31' 2023-10-30T12:34:12.870 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/36' 2023-10-30T12:34:12.870 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/20' 2023-10-30T12:34:12.871 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/39' 2023-10-30T12:34:12.871 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/34' 2023-10-30T12:34:12.871 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/38' 2023-10-30T12:34:12.871 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/28' 2023-10-30T12:34:12.871 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/24' 2023-10-30T12:34:12.872 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/29' 2023-10-30T12:34:12.872 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/32' 2023-10-30T12:34:12.872 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/22' 2023-10-30T12:34:12.872 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/25' 2023-10-30T12:34:12.872 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/30' 2023-10-30T12:34:12.873 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/33' 2023-10-30T12:34:12.873 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/23' 2023-10-30T12:34:12.873 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/35' 2023-10-30T12:34:12.873 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/26' 2023-10-30T12:34:12.873 INFO:tasks.workunit.client.0.smithi125.stdout:removed './212/37' 2023-10-30T12:34:12.873 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './212' 2023-10-30T12:34:12.874 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/21' 2023-10-30T12:34:12.874 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/27' 2023-10-30T12:34:12.874 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/31' 2023-10-30T12:34:12.874 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/36' 2023-10-30T12:34:12.874 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/20' 2023-10-30T12:34:12.875 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/39' 2023-10-30T12:34:12.875 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/34' 2023-10-30T12:34:12.875 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/38' 2023-10-30T12:34:12.875 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/28' 2023-10-30T12:34:12.875 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/24' 2023-10-30T12:34:12.876 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/29' 2023-10-30T12:34:12.876 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/32' 2023-10-30T12:34:12.876 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/22' 2023-10-30T12:34:12.876 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/25' 2023-10-30T12:34:12.876 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/30' 2023-10-30T12:34:12.877 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/33' 2023-10-30T12:34:12.877 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/23' 2023-10-30T12:34:12.877 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/35' 2023-10-30T12:34:12.877 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/26' 2023-10-30T12:34:12.877 INFO:tasks.workunit.client.0.smithi125.stdout:removed './213/37' 2023-10-30T12:34:12.877 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './213' 2023-10-30T12:34:12.878 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/21' 2023-10-30T12:34:12.878 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/27' 2023-10-30T12:34:12.878 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/31' 2023-10-30T12:34:12.878 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/36' 2023-10-30T12:34:12.878 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/20' 2023-10-30T12:34:12.879 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/39' 2023-10-30T12:34:12.879 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/34' 2023-10-30T12:34:12.879 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/38' 2023-10-30T12:34:12.879 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/28' 2023-10-30T12:34:12.879 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/24' 2023-10-30T12:34:12.880 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/29' 2023-10-30T12:34:12.880 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/32' 2023-10-30T12:34:12.880 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/22' 2023-10-30T12:34:12.880 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/25' 2023-10-30T12:34:12.880 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/30' 2023-10-30T12:34:12.881 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/33' 2023-10-30T12:34:12.881 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/23' 2023-10-30T12:34:12.881 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/35' 2023-10-30T12:34:12.881 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/26' 2023-10-30T12:34:12.881 INFO:tasks.workunit.client.0.smithi125.stdout:removed './214/37' 2023-10-30T12:34:12.882 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './214' 2023-10-30T12:34:12.882 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/21' 2023-10-30T12:34:12.882 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/27' 2023-10-30T12:34:12.882 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/31' 2023-10-30T12:34:12.882 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/36' 2023-10-30T12:34:12.883 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/20' 2023-10-30T12:34:12.883 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/39' 2023-10-30T12:34:12.883 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/34' 2023-10-30T12:34:12.883 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/38' 2023-10-30T12:34:12.883 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/28' 2023-10-30T12:34:12.883 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/24' 2023-10-30T12:34:12.884 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/29' 2023-10-30T12:34:12.884 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/32' 2023-10-30T12:34:12.884 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/22' 2023-10-30T12:34:12.884 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/25' 2023-10-30T12:34:12.884 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/30' 2023-10-30T12:34:12.885 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/33' 2023-10-30T12:34:12.885 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/23' 2023-10-30T12:34:12.885 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/35' 2023-10-30T12:34:12.885 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/26' 2023-10-30T12:34:12.885 INFO:tasks.workunit.client.0.smithi125.stdout:removed './215/37' 2023-10-30T12:34:12.886 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './215' 2023-10-30T12:34:12.886 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/21' 2023-10-30T12:34:12.886 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/27' 2023-10-30T12:34:12.886 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/31' 2023-10-30T12:34:12.886 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/36' 2023-10-30T12:34:12.887 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/20' 2023-10-30T12:34:12.887 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/39' 2023-10-30T12:34:12.887 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/34' 2023-10-30T12:34:12.887 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/38' 2023-10-30T12:34:14.288 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/28' 2023-10-30T12:34:14.288 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/24' 2023-10-30T12:34:14.288 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/29' 2023-10-30T12:34:14.288 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/32' 2023-10-30T12:34:14.289 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/22' 2023-10-30T12:34:14.289 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/25' 2023-10-30T12:34:14.289 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/30' 2023-10-30T12:34:14.289 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/33' 2023-10-30T12:34:14.289 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/23' 2023-10-30T12:34:14.289 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/35' 2023-10-30T12:34:14.290 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/26' 2023-10-30T12:34:14.290 INFO:tasks.workunit.client.0.smithi125.stdout:removed './216/37' 2023-10-30T12:34:14.290 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './216' 2023-10-30T12:34:14.290 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/21' 2023-10-30T12:34:14.290 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/27' 2023-10-30T12:34:14.291 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/31' 2023-10-30T12:34:14.291 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/36' 2023-10-30T12:34:14.291 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/20' 2023-10-30T12:34:14.291 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/39' 2023-10-30T12:34:14.291 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/34' 2023-10-30T12:34:14.292 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/38' 2023-10-30T12:34:14.292 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/28' 2023-10-30T12:34:14.292 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/24' 2023-10-30T12:34:14.292 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/29' 2023-10-30T12:34:14.293 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/32' 2023-10-30T12:34:14.293 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/22' 2023-10-30T12:34:14.293 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/25' 2023-10-30T12:34:14.293 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/30' 2023-10-30T12:34:14.293 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/33' 2023-10-30T12:34:14.294 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/23' 2023-10-30T12:34:14.294 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/35' 2023-10-30T12:34:14.294 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/26' 2023-10-30T12:34:14.294 INFO:tasks.workunit.client.0.smithi125.stdout:removed './217/37' 2023-10-30T12:34:14.294 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './217' 2023-10-30T12:34:14.295 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/21' 2023-10-30T12:34:14.295 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/27' 2023-10-30T12:34:14.295 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/31' 2023-10-30T12:34:14.295 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/36' 2023-10-30T12:34:14.295 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/20' 2023-10-30T12:34:14.295 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/39' 2023-10-30T12:34:14.296 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/34' 2023-10-30T12:34:14.296 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/38' 2023-10-30T12:34:14.296 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/28' 2023-10-30T12:34:14.296 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/24' 2023-10-30T12:34:14.296 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/29' 2023-10-30T12:34:14.297 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/32' 2023-10-30T12:34:14.297 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/22' 2023-10-30T12:34:14.297 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/25' 2023-10-30T12:34:14.297 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/30' 2023-10-30T12:34:14.297 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/33' 2023-10-30T12:34:14.298 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/23' 2023-10-30T12:34:14.298 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/35' 2023-10-30T12:34:14.298 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/26' 2023-10-30T12:34:14.298 INFO:tasks.workunit.client.0.smithi125.stdout:removed './218/37' 2023-10-30T12:34:14.298 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './218' 2023-10-30T12:34:14.299 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/21' 2023-10-30T12:34:14.299 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/27' 2023-10-30T12:34:14.299 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/31' 2023-10-30T12:34:14.299 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/36' 2023-10-30T12:34:14.299 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/20' 2023-10-30T12:34:14.300 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/39' 2023-10-30T12:34:14.300 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/34' 2023-10-30T12:34:14.300 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/38' 2023-10-30T12:34:14.300 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/28' 2023-10-30T12:34:14.300 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/24' 2023-10-30T12:34:14.301 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/29' 2023-10-30T12:34:14.301 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/32' 2023-10-30T12:34:14.301 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/22' 2023-10-30T12:34:14.301 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/25' 2023-10-30T12:34:14.301 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/30' 2023-10-30T12:34:14.302 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/33' 2023-10-30T12:34:14.302 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/23' 2023-10-30T12:34:14.302 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/35' 2023-10-30T12:34:14.303 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/26' 2023-10-30T12:34:14.303 INFO:tasks.workunit.client.0.smithi125.stdout:removed './219/37' 2023-10-30T12:34:14.303 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './219' 2023-10-30T12:34:14.303 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/21' 2023-10-30T12:34:14.304 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/27' 2023-10-30T12:34:14.304 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/31' 2023-10-30T12:34:14.304 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/36' 2023-10-30T12:34:14.304 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/20' 2023-10-30T12:34:14.304 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/39' 2023-10-30T12:34:14.305 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/34' 2023-10-30T12:34:14.305 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/38' 2023-10-30T12:34:14.305 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/28' 2023-10-30T12:34:14.305 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/24' 2023-10-30T12:34:14.305 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/29' 2023-10-30T12:34:14.306 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/32' 2023-10-30T12:34:14.306 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/22' 2023-10-30T12:34:14.306 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/25' 2023-10-30T12:34:14.306 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/30' 2023-10-30T12:34:14.306 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/33' 2023-10-30T12:34:14.307 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/23' 2023-10-30T12:34:14.307 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/35' 2023-10-30T12:34:14.307 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/26' 2023-10-30T12:34:14.307 INFO:tasks.workunit.client.0.smithi125.stdout:removed './220/37' 2023-10-30T12:34:14.307 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './220' 2023-10-30T12:34:14.308 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/21' 2023-10-30T12:34:14.308 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/27' 2023-10-30T12:34:14.308 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/31' 2023-10-30T12:34:14.308 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/36' 2023-10-30T12:34:14.308 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/20' 2023-10-30T12:34:14.308 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/39' 2023-10-30T12:34:14.309 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/34' 2023-10-30T12:34:14.309 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/38' 2023-10-30T12:34:14.309 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/28' 2023-10-30T12:34:14.309 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/24' 2023-10-30T12:34:14.309 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/29' 2023-10-30T12:34:14.310 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/32' 2023-10-30T12:34:14.310 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/22' 2023-10-30T12:34:14.310 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/25' 2023-10-30T12:34:14.310 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/30' 2023-10-30T12:34:14.310 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/33' 2023-10-30T12:34:14.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/23' 2023-10-30T12:34:14.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/35' 2023-10-30T12:34:14.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/26' 2023-10-30T12:34:14.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed './221/37' 2023-10-30T12:34:14.311 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './221' 2023-10-30T12:34:14.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/21' 2023-10-30T12:34:14.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/27' 2023-10-30T12:34:14.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/31' 2023-10-30T12:34:14.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/36' 2023-10-30T12:34:14.312 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/20' 2023-10-30T12:34:14.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/39' 2023-10-30T12:34:14.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/34' 2023-10-30T12:34:14.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/38' 2023-10-30T12:34:14.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/28' 2023-10-30T12:34:14.313 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/24' 2023-10-30T12:34:14.314 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/29' 2023-10-30T12:34:14.314 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/32' 2023-10-30T12:34:14.314 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/22' 2023-10-30T12:34:14.314 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/25' 2023-10-30T12:34:14.314 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/30' 2023-10-30T12:34:14.315 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/33' 2023-10-30T12:34:14.315 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/23' 2023-10-30T12:34:14.315 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/35' 2023-10-30T12:34:14.315 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/26' 2023-10-30T12:34:14.315 INFO:tasks.workunit.client.0.smithi125.stdout:removed './222/37' 2023-10-30T12:34:14.316 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './222' 2023-10-30T12:34:14.316 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/21' 2023-10-30T12:34:14.316 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/27' 2023-10-30T12:34:14.316 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/31' 2023-10-30T12:34:14.316 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/36' 2023-10-30T12:34:14.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/20' 2023-10-30T12:34:14.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/39' 2023-10-30T12:34:14.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/34' 2023-10-30T12:34:14.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/38' 2023-10-30T12:34:14.317 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/28' 2023-10-30T12:34:14.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/24' 2023-10-30T12:34:14.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/29' 2023-10-30T12:34:14.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/32' 2023-10-30T12:34:14.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/22' 2023-10-30T12:34:14.318 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/25' 2023-10-30T12:34:14.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/30' 2023-10-30T12:34:14.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/33' 2023-10-30T12:34:14.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/23' 2023-10-30T12:34:14.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/35' 2023-10-30T12:34:14.319 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/26' 2023-10-30T12:34:14.320 INFO:tasks.workunit.client.0.smithi125.stdout:removed './223/37' 2023-10-30T12:34:14.320 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './223' 2023-10-30T12:34:14.320 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/21' 2023-10-30T12:34:14.320 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/27' 2023-10-30T12:34:14.320 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/31' 2023-10-30T12:34:14.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/36' 2023-10-30T12:34:14.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/20' 2023-10-30T12:34:14.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/39' 2023-10-30T12:34:14.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/34' 2023-10-30T12:34:14.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/38' 2023-10-30T12:34:14.321 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/28' 2023-10-30T12:34:14.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/24' 2023-10-30T12:34:14.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/29' 2023-10-30T12:34:14.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/32' 2023-10-30T12:34:14.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/22' 2023-10-30T12:34:14.322 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/25' 2023-10-30T12:34:14.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/30' 2023-10-30T12:34:14.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/33' 2023-10-30T12:34:14.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/23' 2023-10-30T12:34:14.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/35' 2023-10-30T12:34:14.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/26' 2023-10-30T12:34:14.323 INFO:tasks.workunit.client.0.smithi125.stdout:removed './224/37' 2023-10-30T12:34:14.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './224' 2023-10-30T12:34:14.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/21' 2023-10-30T12:34:14.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/27' 2023-10-30T12:34:14.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/31' 2023-10-30T12:34:14.324 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/36' 2023-10-30T12:34:14.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/20' 2023-10-30T12:34:14.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/39' 2023-10-30T12:34:14.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/34' 2023-10-30T12:34:14.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/38' 2023-10-30T12:34:14.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/28' 2023-10-30T12:34:14.325 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/24' 2023-10-30T12:34:14.326 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/29' 2023-10-30T12:34:14.326 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/32' 2023-10-30T12:34:14.326 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/22' 2023-10-30T12:34:14.326 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/25' 2023-10-30T12:34:14.326 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/30' 2023-10-30T12:34:14.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/33' 2023-10-30T12:34:14.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/23' 2023-10-30T12:34:14.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/35' 2023-10-30T12:34:14.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/26' 2023-10-30T12:34:14.327 INFO:tasks.workunit.client.0.smithi125.stdout:removed './225/37' 2023-10-30T12:34:14.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './225' 2023-10-30T12:34:14.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/21' 2023-10-30T12:34:14.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/27' 2023-10-30T12:34:14.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/31' 2023-10-30T12:34:14.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/36' 2023-10-30T12:34:14.328 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/20' 2023-10-30T12:34:14.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/39' 2023-10-30T12:34:14.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/34' 2023-10-30T12:34:14.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/38' 2023-10-30T12:34:14.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/28' 2023-10-30T12:34:14.329 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/24' 2023-10-30T12:34:14.591 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:34:14.596 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:34:15.735 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/29' 2023-10-30T12:34:15.735 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/32' 2023-10-30T12:34:15.735 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/22' 2023-10-30T12:34:15.736 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/25' 2023-10-30T12:34:15.736 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/30' 2023-10-30T12:34:15.736 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/33' 2023-10-30T12:34:15.736 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/23' 2023-10-30T12:34:15.736 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/35' 2023-10-30T12:34:15.737 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/26' 2023-10-30T12:34:15.737 INFO:tasks.workunit.client.0.smithi125.stdout:removed './226/37' 2023-10-30T12:34:15.737 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './226' 2023-10-30T12:34:15.738 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/21' 2023-10-30T12:34:15.738 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/27' 2023-10-30T12:34:15.738 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/31' 2023-10-30T12:34:15.738 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/36' 2023-10-30T12:34:15.738 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/20' 2023-10-30T12:34:15.739 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/39' 2023-10-30T12:34:15.739 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/34' 2023-10-30T12:34:15.739 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/38' 2023-10-30T12:34:15.739 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/28' 2023-10-30T12:34:15.739 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/24' 2023-10-30T12:34:15.740 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/29' 2023-10-30T12:34:15.740 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/32' 2023-10-30T12:34:15.740 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/22' 2023-10-30T12:34:15.741 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/25' 2023-10-30T12:34:15.741 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/30' 2023-10-30T12:34:15.741 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/33' 2023-10-30T12:34:15.741 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/23' 2023-10-30T12:34:15.741 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/35' 2023-10-30T12:34:15.742 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/26' 2023-10-30T12:34:15.742 INFO:tasks.workunit.client.0.smithi125.stdout:removed './227/37' 2023-10-30T12:34:15.742 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './227' 2023-10-30T12:34:15.742 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/21' 2023-10-30T12:34:15.742 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/27' 2023-10-30T12:34:15.743 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/31' 2023-10-30T12:34:15.743 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/36' 2023-10-30T12:34:15.743 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/20' 2023-10-30T12:34:15.743 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/39' 2023-10-30T12:34:15.743 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/34' 2023-10-30T12:34:15.744 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/38' 2023-10-30T12:34:15.744 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/28' 2023-10-30T12:34:15.744 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/24' 2023-10-30T12:34:15.745 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/29' 2023-10-30T12:34:15.745 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/32' 2023-10-30T12:34:15.745 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/22' 2023-10-30T12:34:15.745 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/25' 2023-10-30T12:34:15.745 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/30' 2023-10-30T12:34:15.746 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/33' 2023-10-30T12:34:15.746 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/23' 2023-10-30T12:34:15.746 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/35' 2023-10-30T12:34:15.746 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/26' 2023-10-30T12:34:15.746 INFO:tasks.workunit.client.0.smithi125.stdout:removed './228/37' 2023-10-30T12:34:15.747 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './228' 2023-10-30T12:34:15.747 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/21' 2023-10-30T12:34:15.747 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/27' 2023-10-30T12:34:15.747 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/31' 2023-10-30T12:34:15.747 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/36' 2023-10-30T12:34:15.748 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/20' 2023-10-30T12:34:15.748 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/39' 2023-10-30T12:34:15.748 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/34' 2023-10-30T12:34:15.748 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/38' 2023-10-30T12:34:15.749 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/28' 2023-10-30T12:34:15.749 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/24' 2023-10-30T12:34:15.749 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/29' 2023-10-30T12:34:15.749 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/32' 2023-10-30T12:34:15.749 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/22' 2023-10-30T12:34:15.750 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/25' 2023-10-30T12:34:15.750 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/30' 2023-10-30T12:34:15.750 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/33' 2023-10-30T12:34:15.751 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/23' 2023-10-30T12:34:15.751 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/35' 2023-10-30T12:34:15.751 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/26' 2023-10-30T12:34:15.751 INFO:tasks.workunit.client.0.smithi125.stdout:removed './229/37' 2023-10-30T12:34:15.751 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './229' 2023-10-30T12:34:15.752 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/21' 2023-10-30T12:34:15.752 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/27' 2023-10-30T12:34:15.752 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/31' 2023-10-30T12:34:15.752 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/36' 2023-10-30T12:34:15.752 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/20' 2023-10-30T12:34:15.753 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/39' 2023-10-30T12:34:15.753 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/34' 2023-10-30T12:34:15.753 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/38' 2023-10-30T12:34:15.753 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/28' 2023-10-30T12:34:15.753 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/24' 2023-10-30T12:34:15.754 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/29' 2023-10-30T12:34:15.754 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/32' 2023-10-30T12:34:15.754 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/22' 2023-10-30T12:34:15.754 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/25' 2023-10-30T12:34:15.754 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/30' 2023-10-30T12:34:15.755 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/33' 2023-10-30T12:34:15.755 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/23' 2023-10-30T12:34:15.755 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/35' 2023-10-30T12:34:15.755 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/26' 2023-10-30T12:34:15.755 INFO:tasks.workunit.client.0.smithi125.stdout:removed './230/37' 2023-10-30T12:34:15.756 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './230' 2023-10-30T12:34:15.756 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/21' 2023-10-30T12:34:15.756 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/27' 2023-10-30T12:34:15.756 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/31' 2023-10-30T12:34:15.756 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/36' 2023-10-30T12:34:15.757 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/20' 2023-10-30T12:34:15.757 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/39' 2023-10-30T12:34:15.757 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/34' 2023-10-30T12:34:15.757 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/38' 2023-10-30T12:34:15.757 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/28' 2023-10-30T12:34:15.757 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/24' 2023-10-30T12:34:15.758 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/29' 2023-10-30T12:34:15.758 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/32' 2023-10-30T12:34:15.758 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/22' 2023-10-30T12:34:15.758 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/25' 2023-10-30T12:34:15.758 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/30' 2023-10-30T12:34:15.759 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/33' 2023-10-30T12:34:15.759 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/23' 2023-10-30T12:34:15.759 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/35' 2023-10-30T12:34:15.759 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/26' 2023-10-30T12:34:15.759 INFO:tasks.workunit.client.0.smithi125.stdout:removed './231/37' 2023-10-30T12:34:15.760 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './231' 2023-10-30T12:34:15.760 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/21' 2023-10-30T12:34:15.760 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/27' 2023-10-30T12:34:15.760 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/31' 2023-10-30T12:34:15.760 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/36' 2023-10-30T12:34:15.761 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/20' 2023-10-30T12:34:15.761 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/39' 2023-10-30T12:34:15.761 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/34' 2023-10-30T12:34:15.762 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/38' 2023-10-30T12:34:15.762 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/28' 2023-10-30T12:34:15.762 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/24' 2023-10-30T12:34:15.762 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/29' 2023-10-30T12:34:15.763 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/32' 2023-10-30T12:34:15.763 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/22' 2023-10-30T12:34:15.763 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/25' 2023-10-30T12:34:15.763 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/30' 2023-10-30T12:34:15.763 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/33' 2023-10-30T12:34:15.764 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/23' 2023-10-30T12:34:15.764 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/35' 2023-10-30T12:34:15.764 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/26' 2023-10-30T12:34:15.764 INFO:tasks.workunit.client.0.smithi125.stdout:removed './232/37' 2023-10-30T12:34:15.764 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './232' 2023-10-30T12:34:15.764 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/21' 2023-10-30T12:34:15.765 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/27' 2023-10-30T12:34:15.765 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/31' 2023-10-30T12:34:15.765 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/36' 2023-10-30T12:34:15.765 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/20' 2023-10-30T12:34:15.765 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/39' 2023-10-30T12:34:15.766 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/34' 2023-10-30T12:34:15.766 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/38' 2023-10-30T12:34:15.766 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/28' 2023-10-30T12:34:15.766 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/24' 2023-10-30T12:34:15.767 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/29' 2023-10-30T12:34:15.767 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/32' 2023-10-30T12:34:15.767 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/22' 2023-10-30T12:34:15.767 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/25' 2023-10-30T12:34:15.767 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/30' 2023-10-30T12:34:15.767 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/33' 2023-10-30T12:34:15.768 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/23' 2023-10-30T12:34:15.768 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/35' 2023-10-30T12:34:15.768 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/26' 2023-10-30T12:34:15.768 INFO:tasks.workunit.client.0.smithi125.stdout:removed './233/37' 2023-10-30T12:34:15.768 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './233' 2023-10-30T12:34:15.769 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/21' 2023-10-30T12:34:15.769 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/27' 2023-10-30T12:34:15.769 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/31' 2023-10-30T12:34:15.769 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/36' 2023-10-30T12:34:15.769 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/20' 2023-10-30T12:34:15.770 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/39' 2023-10-30T12:34:15.770 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/34' 2023-10-30T12:34:15.770 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/38' 2023-10-30T12:34:15.770 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/28' 2023-10-30T12:34:15.770 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/24' 2023-10-30T12:34:15.771 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/29' 2023-10-30T12:34:15.771 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/32' 2023-10-30T12:34:15.771 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/22' 2023-10-30T12:34:15.771 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/25' 2023-10-30T12:34:15.771 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/30' 2023-10-30T12:34:15.772 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/33' 2023-10-30T12:34:15.772 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/23' 2023-10-30T12:34:15.772 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/35' 2023-10-30T12:34:15.772 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/26' 2023-10-30T12:34:15.772 INFO:tasks.workunit.client.0.smithi125.stdout:removed './234/37' 2023-10-30T12:34:15.773 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './234' 2023-10-30T12:34:15.773 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/21' 2023-10-30T12:34:15.773 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/27' 2023-10-30T12:34:15.773 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/31' 2023-10-30T12:34:15.773 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/36' 2023-10-30T12:34:15.774 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/20' 2023-10-30T12:34:15.774 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/39' 2023-10-30T12:34:15.774 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/34' 2023-10-30T12:34:15.774 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/38' 2023-10-30T12:34:15.774 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/28' 2023-10-30T12:34:15.775 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/24' 2023-10-30T12:34:15.775 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/29' 2023-10-30T12:34:15.775 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/32' 2023-10-30T12:34:15.775 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/22' 2023-10-30T12:34:15.775 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/25' 2023-10-30T12:34:15.776 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/30' 2023-10-30T12:34:15.776 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/33' 2023-10-30T12:34:15.776 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/23' 2023-10-30T12:34:15.776 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/35' 2023-10-30T12:34:15.776 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/26' 2023-10-30T12:34:15.777 INFO:tasks.workunit.client.0.smithi125.stdout:removed './235/37' 2023-10-30T12:34:15.777 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './235' 2023-10-30T12:34:15.777 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/21' 2023-10-30T12:34:15.777 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/27' 2023-10-30T12:34:15.777 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/31' 2023-10-30T12:34:15.778 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/36' 2023-10-30T12:34:15.778 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/20' 2023-10-30T12:34:15.778 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/39' 2023-10-30T12:34:15.778 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/34' 2023-10-30T12:34:15.778 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/38' 2023-10-30T12:34:15.779 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/28' 2023-10-30T12:34:15.779 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/24' 2023-10-30T12:34:15.779 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/29' 2023-10-30T12:34:15.779 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/32' 2023-10-30T12:34:18.186 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/22' 2023-10-30T12:34:18.187 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/25' 2023-10-30T12:34:18.187 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/30' 2023-10-30T12:34:18.187 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/33' 2023-10-30T12:34:18.188 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/23' 2023-10-30T12:34:18.188 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/35' 2023-10-30T12:34:18.188 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/26' 2023-10-30T12:34:18.188 INFO:tasks.workunit.client.0.smithi125.stdout:removed './236/37' 2023-10-30T12:34:18.188 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './236' 2023-10-30T12:34:18.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/21' 2023-10-30T12:34:18.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/27' 2023-10-30T12:34:18.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/31' 2023-10-30T12:34:18.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/36' 2023-10-30T12:34:18.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/20' 2023-10-30T12:34:18.190 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/39' 2023-10-30T12:34:18.190 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/34' 2023-10-30T12:34:18.190 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/38' 2023-10-30T12:34:18.190 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/28' 2023-10-30T12:34:18.190 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/24' 2023-10-30T12:34:18.191 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/29' 2023-10-30T12:34:18.191 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/32' 2023-10-30T12:34:18.191 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/22' 2023-10-30T12:34:18.191 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/25' 2023-10-30T12:34:18.191 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/30' 2023-10-30T12:34:18.192 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/33' 2023-10-30T12:34:18.192 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/23' 2023-10-30T12:34:18.192 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/35' 2023-10-30T12:34:18.192 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/26' 2023-10-30T12:34:18.192 INFO:tasks.workunit.client.0.smithi125.stdout:removed './237/37' 2023-10-30T12:34:18.192 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './237' 2023-10-30T12:34:18.193 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/21' 2023-10-30T12:34:18.193 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/27' 2023-10-30T12:34:18.193 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/31' 2023-10-30T12:34:18.193 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/36' 2023-10-30T12:34:18.193 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/20' 2023-10-30T12:34:18.194 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/39' 2023-10-30T12:34:18.194 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/34' 2023-10-30T12:34:18.194 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/38' 2023-10-30T12:34:18.194 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/28' 2023-10-30T12:34:18.194 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/24' 2023-10-30T12:34:18.195 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/29' 2023-10-30T12:34:18.195 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/32' 2023-10-30T12:34:18.195 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/22' 2023-10-30T12:34:18.195 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/25' 2023-10-30T12:34:18.195 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/30' 2023-10-30T12:34:18.196 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/33' 2023-10-30T12:34:18.196 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/23' 2023-10-30T12:34:18.196 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/35' 2023-10-30T12:34:18.196 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/26' 2023-10-30T12:34:18.196 INFO:tasks.workunit.client.0.smithi125.stdout:removed './238/37' 2023-10-30T12:34:18.197 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './238' 2023-10-30T12:34:18.197 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/21' 2023-10-30T12:34:18.197 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/27' 2023-10-30T12:34:18.197 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/31' 2023-10-30T12:34:18.197 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/36' 2023-10-30T12:34:18.198 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/20' 2023-10-30T12:34:18.198 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/39' 2023-10-30T12:34:18.198 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/34' 2023-10-30T12:34:18.198 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/38' 2023-10-30T12:34:18.198 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/28' 2023-10-30T12:34:18.199 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/24' 2023-10-30T12:34:18.199 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/29' 2023-10-30T12:34:18.199 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/32' 2023-10-30T12:34:18.199 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/22' 2023-10-30T12:34:18.199 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/25' 2023-10-30T12:34:18.200 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/30' 2023-10-30T12:34:18.200 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/33' 2023-10-30T12:34:18.200 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/23' 2023-10-30T12:34:18.200 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/35' 2023-10-30T12:34:18.200 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/26' 2023-10-30T12:34:18.201 INFO:tasks.workunit.client.0.smithi125.stdout:removed './239/37' 2023-10-30T12:34:18.201 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './239' 2023-10-30T12:34:18.201 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/21' 2023-10-30T12:34:18.201 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/27' 2023-10-30T12:34:18.201 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/31' 2023-10-30T12:34:18.202 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/36' 2023-10-30T12:34:18.202 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/20' 2023-10-30T12:34:18.202 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/39' 2023-10-30T12:34:18.202 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/34' 2023-10-30T12:34:18.202 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/38' 2023-10-30T12:34:18.203 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/28' 2023-10-30T12:34:18.203 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/24' 2023-10-30T12:34:18.203 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/29' 2023-10-30T12:34:18.203 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/32' 2023-10-30T12:34:18.203 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/22' 2023-10-30T12:34:18.204 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/25' 2023-10-30T12:34:18.204 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/30' 2023-10-30T12:34:18.204 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/33' 2023-10-30T12:34:18.204 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/23' 2023-10-30T12:34:18.204 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/35' 2023-10-30T12:34:18.204 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/26' 2023-10-30T12:34:18.205 INFO:tasks.workunit.client.0.smithi125.stdout:removed './240/37' 2023-10-30T12:34:18.205 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './240' 2023-10-30T12:34:18.205 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/21' 2023-10-30T12:34:18.205 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/27' 2023-10-30T12:34:18.206 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/31' 2023-10-30T12:34:18.206 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/36' 2023-10-30T12:34:18.206 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/20' 2023-10-30T12:34:18.206 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/39' 2023-10-30T12:34:18.206 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/34' 2023-10-30T12:34:18.206 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/38' 2023-10-30T12:34:18.207 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/28' 2023-10-30T12:34:18.207 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/24' 2023-10-30T12:34:18.207 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/29' 2023-10-30T12:34:18.207 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/32' 2023-10-30T12:34:18.207 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/22' 2023-10-30T12:34:18.208 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/25' 2023-10-30T12:34:18.208 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/30' 2023-10-30T12:34:18.208 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/33' 2023-10-30T12:34:18.208 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/23' 2023-10-30T12:34:18.208 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/35' 2023-10-30T12:34:18.209 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/26' 2023-10-30T12:34:18.209 INFO:tasks.workunit.client.0.smithi125.stdout:removed './241/37' 2023-10-30T12:34:18.209 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './241' 2023-10-30T12:34:18.209 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/21' 2023-10-30T12:34:18.209 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/27' 2023-10-30T12:34:18.210 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/31' 2023-10-30T12:34:18.210 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/36' 2023-10-30T12:34:18.210 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/20' 2023-10-30T12:34:18.210 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/39' 2023-10-30T12:34:18.210 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/34' 2023-10-30T12:34:18.211 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/38' 2023-10-30T12:34:18.211 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/28' 2023-10-30T12:34:18.211 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/24' 2023-10-30T12:34:18.211 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/29' 2023-10-30T12:34:18.211 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/32' 2023-10-30T12:34:18.212 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/22' 2023-10-30T12:34:18.212 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/25' 2023-10-30T12:34:18.212 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/30' 2023-10-30T12:34:18.212 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/33' 2023-10-30T12:34:18.212 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/23' 2023-10-30T12:34:18.213 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/35' 2023-10-30T12:34:18.213 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/26' 2023-10-30T12:34:18.213 INFO:tasks.workunit.client.0.smithi125.stdout:removed './242/37' 2023-10-30T12:34:18.213 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './242' 2023-10-30T12:34:18.213 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/21' 2023-10-30T12:34:18.213 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/27' 2023-10-30T12:34:18.214 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/31' 2023-10-30T12:34:18.214 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/36' 2023-10-30T12:34:18.214 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/20' 2023-10-30T12:34:18.214 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/39' 2023-10-30T12:34:18.214 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/34' 2023-10-30T12:34:18.215 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/38' 2023-10-30T12:34:18.215 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/28' 2023-10-30T12:34:18.215 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/24' 2023-10-30T12:34:18.215 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/29' 2023-10-30T12:34:18.215 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/32' 2023-10-30T12:34:18.216 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/22' 2023-10-30T12:34:18.216 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/25' 2023-10-30T12:34:18.216 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/30' 2023-10-30T12:34:18.216 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/33' 2023-10-30T12:34:18.216 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/23' 2023-10-30T12:34:18.217 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/35' 2023-10-30T12:34:18.217 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/26' 2023-10-30T12:34:18.217 INFO:tasks.workunit.client.0.smithi125.stdout:removed './243/37' 2023-10-30T12:34:18.217 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './243' 2023-10-30T12:34:18.217 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/21' 2023-10-30T12:34:18.218 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/27' 2023-10-30T12:34:18.218 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/31' 2023-10-30T12:34:18.218 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/36' 2023-10-30T12:34:18.218 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/20' 2023-10-30T12:34:18.218 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/39' 2023-10-30T12:34:18.219 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/34' 2023-10-30T12:34:18.219 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/38' 2023-10-30T12:34:18.219 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/28' 2023-10-30T12:34:18.219 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/24' 2023-10-30T12:34:18.219 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/29' 2023-10-30T12:34:18.220 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/32' 2023-10-30T12:34:18.220 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/22' 2023-10-30T12:34:18.220 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/25' 2023-10-30T12:34:18.220 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/30' 2023-10-30T12:34:18.220 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/33' 2023-10-30T12:34:18.221 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/23' 2023-10-30T12:34:18.221 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/35' 2023-10-30T12:34:18.221 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/26' 2023-10-30T12:34:18.221 INFO:tasks.workunit.client.0.smithi125.stdout:removed './244/37' 2023-10-30T12:34:18.221 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './244' 2023-10-30T12:34:18.222 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/21' 2023-10-30T12:34:18.222 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/27' 2023-10-30T12:34:18.222 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/31' 2023-10-30T12:34:18.222 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/36' 2023-10-30T12:34:18.222 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/20' 2023-10-30T12:34:18.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/39' 2023-10-30T12:34:18.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/34' 2023-10-30T12:34:18.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/38' 2023-10-30T12:34:18.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/28' 2023-10-30T12:34:18.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/24' 2023-10-30T12:34:18.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/29' 2023-10-30T12:34:18.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/32' 2023-10-30T12:34:18.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/22' 2023-10-30T12:34:18.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/25' 2023-10-30T12:34:18.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/30' 2023-10-30T12:34:18.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/33' 2023-10-30T12:34:18.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/23' 2023-10-30T12:34:18.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/35' 2023-10-30T12:34:18.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/26' 2023-10-30T12:34:18.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed './245/37' 2023-10-30T12:34:18.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './245' 2023-10-30T12:34:18.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/21' 2023-10-30T12:34:18.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/27' 2023-10-30T12:34:18.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/31' 2023-10-30T12:34:18.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/36' 2023-10-30T12:34:18.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/20' 2023-10-30T12:34:18.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/39' 2023-10-30T12:34:18.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/34' 2023-10-30T12:34:18.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/38' 2023-10-30T12:34:18.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/28' 2023-10-30T12:34:18.228 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/24' 2023-10-30T12:34:18.228 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/29' 2023-10-30T12:34:18.228 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/32' 2023-10-30T12:34:18.228 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/22' 2023-10-30T12:34:20.610 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/25' 2023-10-30T12:34:20.610 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/30' 2023-10-30T12:34:20.610 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/33' 2023-10-30T12:34:20.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/23' 2023-10-30T12:34:20.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/35' 2023-10-30T12:34:20.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/26' 2023-10-30T12:34:20.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed './246/37' 2023-10-30T12:34:20.611 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './246' 2023-10-30T12:34:20.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/21' 2023-10-30T12:34:20.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/27' 2023-10-30T12:34:20.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/31' 2023-10-30T12:34:20.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/36' 2023-10-30T12:34:20.612 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/20' 2023-10-30T12:34:20.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/39' 2023-10-30T12:34:20.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/34' 2023-10-30T12:34:20.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/38' 2023-10-30T12:34:20.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/28' 2023-10-30T12:34:20.613 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/24' 2023-10-30T12:34:20.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/29' 2023-10-30T12:34:20.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/32' 2023-10-30T12:34:20.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/22' 2023-10-30T12:34:20.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/25' 2023-10-30T12:34:20.614 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/30' 2023-10-30T12:34:20.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/33' 2023-10-30T12:34:20.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/23' 2023-10-30T12:34:20.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/35' 2023-10-30T12:34:20.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/26' 2023-10-30T12:34:20.615 INFO:tasks.workunit.client.0.smithi125.stdout:removed './247/37' 2023-10-30T12:34:20.616 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './247' 2023-10-30T12:34:20.616 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/21' 2023-10-30T12:34:20.616 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/27' 2023-10-30T12:34:20.616 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/31' 2023-10-30T12:34:20.616 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/36' 2023-10-30T12:34:20.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/20' 2023-10-30T12:34:20.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/39' 2023-10-30T12:34:20.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/34' 2023-10-30T12:34:20.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/38' 2023-10-30T12:34:20.617 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/28' 2023-10-30T12:34:20.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/24' 2023-10-30T12:34:20.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/29' 2023-10-30T12:34:20.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/32' 2023-10-30T12:34:20.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/22' 2023-10-30T12:34:20.618 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/25' 2023-10-30T12:34:20.619 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/30' 2023-10-30T12:34:20.619 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/33' 2023-10-30T12:34:20.619 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/23' 2023-10-30T12:34:20.619 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/35' 2023-10-30T12:34:20.619 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/26' 2023-10-30T12:34:20.620 INFO:tasks.workunit.client.0.smithi125.stdout:removed './248/37' 2023-10-30T12:34:20.620 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './248' 2023-10-30T12:34:20.620 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/21' 2023-10-30T12:34:20.620 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/27' 2023-10-30T12:34:20.620 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/31' 2023-10-30T12:34:20.621 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/36' 2023-10-30T12:34:20.621 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/20' 2023-10-30T12:34:20.621 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/39' 2023-10-30T12:34:20.621 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/34' 2023-10-30T12:34:20.621 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/38' 2023-10-30T12:34:20.622 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/28' 2023-10-30T12:34:20.622 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/24' 2023-10-30T12:34:20.622 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/29' 2023-10-30T12:34:20.622 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/32' 2023-10-30T12:34:20.622 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/22' 2023-10-30T12:34:20.623 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/25' 2023-10-30T12:34:20.623 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/30' 2023-10-30T12:34:20.623 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/33' 2023-10-30T12:34:20.623 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/23' 2023-10-30T12:34:20.623 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/35' 2023-10-30T12:34:20.623 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/26' 2023-10-30T12:34:20.624 INFO:tasks.workunit.client.0.smithi125.stdout:removed './249/37' 2023-10-30T12:34:20.624 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './249' 2023-10-30T12:34:20.624 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/21' 2023-10-30T12:34:20.624 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/27' 2023-10-30T12:34:20.624 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/31' 2023-10-30T12:34:20.625 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/36' 2023-10-30T12:34:20.625 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/20' 2023-10-30T12:34:20.625 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/39' 2023-10-30T12:34:20.625 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/34' 2023-10-30T12:34:20.625 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/38' 2023-10-30T12:34:20.626 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/28' 2023-10-30T12:34:20.626 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/24' 2023-10-30T12:34:20.626 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/29' 2023-10-30T12:34:20.626 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/32' 2023-10-30T12:34:20.626 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/22' 2023-10-30T12:34:20.627 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/25' 2023-10-30T12:34:20.627 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/30' 2023-10-30T12:34:20.627 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/33' 2023-10-30T12:34:20.627 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/23' 2023-10-30T12:34:20.627 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/35' 2023-10-30T12:34:20.628 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/26' 2023-10-30T12:34:20.628 INFO:tasks.workunit.client.0.smithi125.stdout:removed './250/37' 2023-10-30T12:34:20.628 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './250' 2023-10-30T12:34:20.628 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/21' 2023-10-30T12:34:20.628 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/27' 2023-10-30T12:34:20.629 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/31' 2023-10-30T12:34:20.629 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/36' 2023-10-30T12:34:20.629 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/20' 2023-10-30T12:34:20.629 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/39' 2023-10-30T12:34:20.629 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/34' 2023-10-30T12:34:20.629 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/38' 2023-10-30T12:34:20.630 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/28' 2023-10-30T12:34:20.630 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/24' 2023-10-30T12:34:20.630 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/29' 2023-10-30T12:34:20.630 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/32' 2023-10-30T12:34:20.630 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/22' 2023-10-30T12:34:20.631 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/25' 2023-10-30T12:34:20.631 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/30' 2023-10-30T12:34:20.631 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/33' 2023-10-30T12:34:20.631 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/23' 2023-10-30T12:34:20.631 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/35' 2023-10-30T12:34:20.632 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/26' 2023-10-30T12:34:20.632 INFO:tasks.workunit.client.0.smithi125.stdout:removed './251/37' 2023-10-30T12:34:20.632 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './251' 2023-10-30T12:34:20.632 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/21' 2023-10-30T12:34:20.632 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/27' 2023-10-30T12:34:20.633 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/31' 2023-10-30T12:34:20.633 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/36' 2023-10-30T12:34:20.633 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/20' 2023-10-30T12:34:20.633 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/39' 2023-10-30T12:34:20.633 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/34' 2023-10-30T12:34:20.634 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/38' 2023-10-30T12:34:20.634 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/28' 2023-10-30T12:34:20.634 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/24' 2023-10-30T12:34:20.634 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/29' 2023-10-30T12:34:20.635 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/32' 2023-10-30T12:34:20.635 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/22' 2023-10-30T12:34:20.635 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/25' 2023-10-30T12:34:20.635 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/30' 2023-10-30T12:34:20.635 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/33' 2023-10-30T12:34:20.636 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/23' 2023-10-30T12:34:20.636 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/35' 2023-10-30T12:34:20.636 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/26' 2023-10-30T12:34:20.636 INFO:tasks.workunit.client.0.smithi125.stdout:removed './252/37' 2023-10-30T12:34:20.636 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './252' 2023-10-30T12:34:20.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/21' 2023-10-30T12:34:20.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/27' 2023-10-30T12:34:20.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/31' 2023-10-30T12:34:20.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/36' 2023-10-30T12:34:20.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/20' 2023-10-30T12:34:20.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/39' 2023-10-30T12:34:20.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/34' 2023-10-30T12:34:20.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/38' 2023-10-30T12:34:20.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/28' 2023-10-30T12:34:20.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/24' 2023-10-30T12:34:20.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/29' 2023-10-30T12:34:20.639 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/32' 2023-10-30T12:34:20.639 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/22' 2023-10-30T12:34:20.639 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/25' 2023-10-30T12:34:20.639 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/30' 2023-10-30T12:34:20.639 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/33' 2023-10-30T12:34:20.640 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/23' 2023-10-30T12:34:20.640 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/35' 2023-10-30T12:34:20.640 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/26' 2023-10-30T12:34:20.640 INFO:tasks.workunit.client.0.smithi125.stdout:removed './253/37' 2023-10-30T12:34:20.640 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './253' 2023-10-30T12:34:20.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/21' 2023-10-30T12:34:20.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/27' 2023-10-30T12:34:20.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/31' 2023-10-30T12:34:20.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/36' 2023-10-30T12:34:20.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/20' 2023-10-30T12:34:20.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/39' 2023-10-30T12:34:20.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/34' 2023-10-30T12:34:20.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/38' 2023-10-30T12:34:20.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/28' 2023-10-30T12:34:20.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/24' 2023-10-30T12:34:20.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/29' 2023-10-30T12:34:20.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/32' 2023-10-30T12:34:20.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/22' 2023-10-30T12:34:20.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/25' 2023-10-30T12:34:20.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/30' 2023-10-30T12:34:20.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/33' 2023-10-30T12:34:20.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/23' 2023-10-30T12:34:20.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/35' 2023-10-30T12:34:20.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/26' 2023-10-30T12:34:20.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed './254/37' 2023-10-30T12:34:20.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './254' 2023-10-30T12:34:20.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/21' 2023-10-30T12:34:20.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/27' 2023-10-30T12:34:20.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/31' 2023-10-30T12:34:20.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/36' 2023-10-30T12:34:20.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/20' 2023-10-30T12:34:20.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/39' 2023-10-30T12:34:20.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/34' 2023-10-30T12:34:20.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/38' 2023-10-30T12:34:20.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/28' 2023-10-30T12:34:20.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/24' 2023-10-30T12:34:20.647 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/29' 2023-10-30T12:34:20.647 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/32' 2023-10-30T12:34:20.647 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/22' 2023-10-30T12:34:20.647 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/25' 2023-10-30T12:34:20.647 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/30' 2023-10-30T12:34:20.647 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/33' 2023-10-30T12:34:20.648 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/23' 2023-10-30T12:34:20.648 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/35' 2023-10-30T12:34:20.648 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/26' 2023-10-30T12:34:20.648 INFO:tasks.workunit.client.0.smithi125.stdout:removed './255/37' 2023-10-30T12:34:20.648 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './255' 2023-10-30T12:34:20.649 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/21' 2023-10-30T12:34:20.649 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/27' 2023-10-30T12:34:20.649 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/31' 2023-10-30T12:34:20.649 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/36' 2023-10-30T12:34:20.649 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/20' 2023-10-30T12:34:20.650 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/39' 2023-10-30T12:34:20.650 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/34' 2023-10-30T12:34:20.650 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/38' 2023-10-30T12:34:20.650 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/28' 2023-10-30T12:34:20.650 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/24' 2023-10-30T12:34:20.651 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/29' 2023-10-30T12:34:20.651 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/32' 2023-10-30T12:34:20.651 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/22' 2023-10-30T12:34:20.651 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/25' 2023-10-30T12:34:20.651 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/30' 2023-10-30T12:34:27.761 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/33' 2023-10-30T12:34:27.761 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/23' 2023-10-30T12:34:27.762 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/35' 2023-10-30T12:34:27.762 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/26' 2023-10-30T12:34:27.762 INFO:tasks.workunit.client.0.smithi125.stdout:removed './256/37' 2023-10-30T12:34:27.762 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './256' 2023-10-30T12:34:27.762 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/21' 2023-10-30T12:34:27.763 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/27' 2023-10-30T12:34:27.763 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/31' 2023-10-30T12:34:27.763 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/36' 2023-10-30T12:34:27.763 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/20' 2023-10-30T12:34:27.763 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/39' 2023-10-30T12:34:27.764 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/34' 2023-10-30T12:34:27.764 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/38' 2023-10-30T12:34:27.764 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/28' 2023-10-30T12:34:27.764 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/24' 2023-10-30T12:34:27.765 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/29' 2023-10-30T12:34:27.765 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/32' 2023-10-30T12:34:27.765 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/22' 2023-10-30T12:34:27.765 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/25' 2023-10-30T12:34:27.765 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/30' 2023-10-30T12:34:27.766 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/33' 2023-10-30T12:34:27.766 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/23' 2023-10-30T12:34:27.766 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/35' 2023-10-30T12:34:27.766 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/26' 2023-10-30T12:34:27.766 INFO:tasks.workunit.client.0.smithi125.stdout:removed './257/37' 2023-10-30T12:34:27.767 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './257' 2023-10-30T12:34:27.767 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/21' 2023-10-30T12:34:27.767 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/27' 2023-10-30T12:34:27.767 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/31' 2023-10-30T12:34:27.767 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/36' 2023-10-30T12:34:27.767 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/20' 2023-10-30T12:34:27.768 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/39' 2023-10-30T12:34:27.768 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/34' 2023-10-30T12:34:27.768 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/38' 2023-10-30T12:34:27.768 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/28' 2023-10-30T12:34:27.768 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/24' 2023-10-30T12:34:27.769 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/29' 2023-10-30T12:34:27.769 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/32' 2023-10-30T12:34:27.769 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/22' 2023-10-30T12:34:27.769 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/25' 2023-10-30T12:34:27.769 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/30' 2023-10-30T12:34:27.770 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/33' 2023-10-30T12:34:27.770 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/23' 2023-10-30T12:34:27.770 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/35' 2023-10-30T12:34:27.770 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/26' 2023-10-30T12:34:27.770 INFO:tasks.workunit.client.0.smithi125.stdout:removed './258/37' 2023-10-30T12:34:27.771 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './258' 2023-10-30T12:34:27.771 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/21' 2023-10-30T12:34:27.771 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/27' 2023-10-30T12:34:27.771 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/31' 2023-10-30T12:34:27.771 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/36' 2023-10-30T12:34:27.772 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/20' 2023-10-30T12:34:27.772 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/39' 2023-10-30T12:34:27.772 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/34' 2023-10-30T12:34:27.772 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/38' 2023-10-30T12:34:27.772 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/28' 2023-10-30T12:34:27.772 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/24' 2023-10-30T12:34:27.773 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/29' 2023-10-30T12:34:27.773 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/32' 2023-10-30T12:34:27.773 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/22' 2023-10-30T12:34:27.773 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/25' 2023-10-30T12:34:27.773 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/30' 2023-10-30T12:34:27.774 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/33' 2023-10-30T12:34:27.774 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/23' 2023-10-30T12:34:27.774 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/35' 2023-10-30T12:34:27.774 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/26' 2023-10-30T12:34:27.774 INFO:tasks.workunit.client.0.smithi125.stdout:removed './259/37' 2023-10-30T12:34:27.775 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './259' 2023-10-30T12:34:27.775 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/21' 2023-10-30T12:34:27.775 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/27' 2023-10-30T12:34:27.775 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/31' 2023-10-30T12:34:27.775 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/36' 2023-10-30T12:34:27.776 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/20' 2023-10-30T12:34:27.776 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/39' 2023-10-30T12:34:27.776 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/34' 2023-10-30T12:34:27.776 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/38' 2023-10-30T12:34:27.776 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/28' 2023-10-30T12:34:27.776 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/24' 2023-10-30T12:34:27.777 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/29' 2023-10-30T12:34:27.777 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/32' 2023-10-30T12:34:27.777 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/22' 2023-10-30T12:34:27.777 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/25' 2023-10-30T12:34:27.777 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/30' 2023-10-30T12:34:27.778 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/33' 2023-10-30T12:34:27.778 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/23' 2023-10-30T12:34:27.778 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/35' 2023-10-30T12:34:27.778 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/26' 2023-10-30T12:34:27.778 INFO:tasks.workunit.client.0.smithi125.stdout:removed './260/37' 2023-10-30T12:34:27.779 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './260' 2023-10-30T12:34:27.779 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/21' 2023-10-30T12:34:27.779 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/27' 2023-10-30T12:34:27.779 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/31' 2023-10-30T12:34:27.779 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/36' 2023-10-30T12:34:27.780 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/20' 2023-10-30T12:34:27.780 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/39' 2023-10-30T12:34:27.780 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/34' 2023-10-30T12:34:27.780 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/38' 2023-10-30T12:34:27.780 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/28' 2023-10-30T12:34:27.781 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/24' 2023-10-30T12:34:27.781 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/29' 2023-10-30T12:34:27.781 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/32' 2023-10-30T12:34:27.781 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/22' 2023-10-30T12:34:27.781 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/25' 2023-10-30T12:34:27.781 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/30' 2023-10-30T12:34:27.782 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/33' 2023-10-30T12:34:27.782 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/23' 2023-10-30T12:34:27.782 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/35' 2023-10-30T12:34:27.782 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/26' 2023-10-30T12:34:27.782 INFO:tasks.workunit.client.0.smithi125.stdout:removed './261/37' 2023-10-30T12:34:27.783 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './261' 2023-10-30T12:34:27.783 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/21' 2023-10-30T12:34:27.783 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/27' 2023-10-30T12:34:27.783 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/31' 2023-10-30T12:34:27.783 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/36' 2023-10-30T12:34:27.784 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/20' 2023-10-30T12:34:27.784 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/39' 2023-10-30T12:34:27.784 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/34' 2023-10-30T12:34:27.784 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/38' 2023-10-30T12:34:27.784 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/28' 2023-10-30T12:34:27.785 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/24' 2023-10-30T12:34:27.785 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/29' 2023-10-30T12:34:27.785 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/32' 2023-10-30T12:34:27.785 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/22' 2023-10-30T12:34:27.785 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/25' 2023-10-30T12:34:27.786 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/30' 2023-10-30T12:34:27.786 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/33' 2023-10-30T12:34:27.786 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/23' 2023-10-30T12:34:27.786 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/35' 2023-10-30T12:34:27.787 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/26' 2023-10-30T12:34:27.787 INFO:tasks.workunit.client.0.smithi125.stdout:removed './262/37' 2023-10-30T12:34:27.787 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './262' 2023-10-30T12:34:27.787 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/21' 2023-10-30T12:34:27.787 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/27' 2023-10-30T12:34:27.788 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/31' 2023-10-30T12:34:27.788 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/36' 2023-10-30T12:34:27.788 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/20' 2023-10-30T12:34:27.788 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/39' 2023-10-30T12:34:27.788 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/34' 2023-10-30T12:34:27.789 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/38' 2023-10-30T12:34:27.789 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/28' 2023-10-30T12:34:27.789 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/24' 2023-10-30T12:34:27.789 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/29' 2023-10-30T12:34:27.789 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/32' 2023-10-30T12:34:27.790 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/22' 2023-10-30T12:34:27.790 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/25' 2023-10-30T12:34:27.790 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/30' 2023-10-30T12:34:27.790 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/33' 2023-10-30T12:34:27.790 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/23' 2023-10-30T12:34:27.791 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/35' 2023-10-30T12:34:27.791 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/26' 2023-10-30T12:34:27.791 INFO:tasks.workunit.client.0.smithi125.stdout:removed './263/37' 2023-10-30T12:34:27.791 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './263' 2023-10-30T12:34:27.791 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/21' 2023-10-30T12:34:27.792 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/27' 2023-10-30T12:34:27.792 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/31' 2023-10-30T12:34:27.792 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/36' 2023-10-30T12:34:27.792 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/20' 2023-10-30T12:34:27.792 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/39' 2023-10-30T12:34:27.793 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/34' 2023-10-30T12:34:27.793 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/38' 2023-10-30T12:34:27.793 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/28' 2023-10-30T12:34:27.793 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/24' 2023-10-30T12:34:27.793 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/29' 2023-10-30T12:34:27.794 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/32' 2023-10-30T12:34:27.794 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/22' 2023-10-30T12:34:27.794 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/25' 2023-10-30T12:34:27.794 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/30' 2023-10-30T12:34:27.794 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/33' 2023-10-30T12:34:27.795 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/23' 2023-10-30T12:34:27.795 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/35' 2023-10-30T12:34:27.795 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/26' 2023-10-30T12:34:27.795 INFO:tasks.workunit.client.0.smithi125.stdout:removed './264/37' 2023-10-30T12:34:27.795 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './264' 2023-10-30T12:34:27.796 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/21' 2023-10-30T12:34:27.796 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/27' 2023-10-30T12:34:27.796 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/31' 2023-10-30T12:34:27.796 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/36' 2023-10-30T12:34:27.796 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/20' 2023-10-30T12:34:27.797 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/39' 2023-10-30T12:34:27.797 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/34' 2023-10-30T12:34:27.797 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/38' 2023-10-30T12:34:27.797 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/28' 2023-10-30T12:34:27.797 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/24' 2023-10-30T12:34:27.798 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/29' 2023-10-30T12:34:27.798 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/32' 2023-10-30T12:34:27.798 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/22' 2023-10-30T12:34:27.798 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/25' 2023-10-30T12:34:27.798 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/30' 2023-10-30T12:34:27.799 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/33' 2023-10-30T12:34:27.799 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/23' 2023-10-30T12:34:27.799 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/35' 2023-10-30T12:34:27.799 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/26' 2023-10-30T12:34:27.799 INFO:tasks.workunit.client.0.smithi125.stdout:removed './265/37' 2023-10-30T12:34:27.800 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './265' 2023-10-30T12:34:27.800 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/21' 2023-10-30T12:34:27.800 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/27' 2023-10-30T12:34:27.800 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/31' 2023-10-30T12:34:27.800 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/36' 2023-10-30T12:34:27.800 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/20' 2023-10-30T12:34:27.801 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/39' 2023-10-30T12:34:27.801 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/34' 2023-10-30T12:34:27.801 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/38' 2023-10-30T12:34:27.801 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/28' 2023-10-30T12:34:27.801 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/24' 2023-10-30T12:34:27.802 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/29' 2023-10-30T12:34:27.802 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/32' 2023-10-30T12:34:27.802 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/22' 2023-10-30T12:34:27.802 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/25' 2023-10-30T12:34:27.802 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/30' 2023-10-30T12:34:27.803 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/33' 2023-10-30T12:34:27.803 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/23' 2023-10-30T12:34:32.823 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/35' 2023-10-30T12:34:32.823 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/26' 2023-10-30T12:34:32.823 INFO:tasks.workunit.client.0.smithi125.stdout:removed './266/37' 2023-10-30T12:34:32.824 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './266' 2023-10-30T12:34:32.824 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/21' 2023-10-30T12:34:32.824 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/27' 2023-10-30T12:34:32.824 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/31' 2023-10-30T12:34:32.824 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/36' 2023-10-30T12:34:32.824 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/20' 2023-10-30T12:34:32.825 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/39' 2023-10-30T12:34:32.825 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/34' 2023-10-30T12:34:32.825 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/38' 2023-10-30T12:34:32.825 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/28' 2023-10-30T12:34:32.825 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/24' 2023-10-30T12:34:32.825 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/29' 2023-10-30T12:34:32.826 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/32' 2023-10-30T12:34:32.826 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/22' 2023-10-30T12:34:32.826 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/25' 2023-10-30T12:34:32.826 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/30' 2023-10-30T12:34:32.826 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/33' 2023-10-30T12:34:32.827 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/23' 2023-10-30T12:34:32.827 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/35' 2023-10-30T12:34:32.827 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/26' 2023-10-30T12:34:32.827 INFO:tasks.workunit.client.0.smithi125.stdout:removed './267/37' 2023-10-30T12:34:32.827 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './267' 2023-10-30T12:34:32.828 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/21' 2023-10-30T12:34:32.828 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/27' 2023-10-30T12:34:32.828 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/31' 2023-10-30T12:34:32.828 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/36' 2023-10-30T12:34:32.828 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/20' 2023-10-30T12:34:32.828 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/39' 2023-10-30T12:34:32.829 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/34' 2023-10-30T12:34:32.829 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/38' 2023-10-30T12:34:32.829 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/28' 2023-10-30T12:34:32.829 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/24' 2023-10-30T12:34:32.829 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/29' 2023-10-30T12:34:32.830 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/32' 2023-10-30T12:34:32.830 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/22' 2023-10-30T12:34:32.831 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/25' 2023-10-30T12:34:32.831 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/30' 2023-10-30T12:34:32.831 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/33' 2023-10-30T12:34:32.831 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/23' 2023-10-30T12:34:32.831 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/35' 2023-10-30T12:34:32.832 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/26' 2023-10-30T12:34:32.832 INFO:tasks.workunit.client.0.smithi125.stdout:removed './268/37' 2023-10-30T12:34:32.832 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './268' 2023-10-30T12:34:32.832 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/21' 2023-10-30T12:34:32.832 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/27' 2023-10-30T12:34:32.833 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/31' 2023-10-30T12:34:32.833 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/36' 2023-10-30T12:34:32.833 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/20' 2023-10-30T12:34:32.833 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/39' 2023-10-30T12:34:32.834 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/34' 2023-10-30T12:34:32.834 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/38' 2023-10-30T12:34:32.834 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/28' 2023-10-30T12:34:32.834 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/24' 2023-10-30T12:34:32.834 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/29' 2023-10-30T12:34:32.835 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/32' 2023-10-30T12:34:32.835 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/22' 2023-10-30T12:34:32.835 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/25' 2023-10-30T12:34:32.835 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/30' 2023-10-30T12:34:32.835 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/33' 2023-10-30T12:34:32.836 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/23' 2023-10-30T12:34:32.836 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/35' 2023-10-30T12:34:32.836 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/26' 2023-10-30T12:34:32.836 INFO:tasks.workunit.client.0.smithi125.stdout:removed './269/37' 2023-10-30T12:34:32.837 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './269' 2023-10-30T12:34:32.837 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/21' 2023-10-30T12:34:32.837 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/27' 2023-10-30T12:34:32.837 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/31' 2023-10-30T12:34:32.837 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/36' 2023-10-30T12:34:32.837 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/20' 2023-10-30T12:34:32.838 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/39' 2023-10-30T12:34:32.838 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/34' 2023-10-30T12:34:32.838 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/38' 2023-10-30T12:34:32.838 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/28' 2023-10-30T12:34:32.838 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/24' 2023-10-30T12:34:32.839 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/29' 2023-10-30T12:34:32.839 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/32' 2023-10-30T12:34:32.839 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/22' 2023-10-30T12:34:32.839 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/25' 2023-10-30T12:34:32.839 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/30' 2023-10-30T12:34:32.839 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/33' 2023-10-30T12:34:32.840 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/23' 2023-10-30T12:34:32.840 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/35' 2023-10-30T12:34:32.840 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/26' 2023-10-30T12:34:32.841 INFO:tasks.workunit.client.0.smithi125.stdout:removed './270/37' 2023-10-30T12:34:32.841 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './270' 2023-10-30T12:34:32.841 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/21' 2023-10-30T12:34:32.841 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/27' 2023-10-30T12:34:32.841 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/31' 2023-10-30T12:34:32.842 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/36' 2023-10-30T12:34:32.842 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/20' 2023-10-30T12:34:32.842 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/39' 2023-10-30T12:34:32.842 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/34' 2023-10-30T12:34:32.842 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/38' 2023-10-30T12:34:32.843 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/28' 2023-10-30T12:34:32.843 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/24' 2023-10-30T12:34:32.843 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/29' 2023-10-30T12:34:32.843 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/32' 2023-10-30T12:34:32.843 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/22' 2023-10-30T12:34:32.844 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/25' 2023-10-30T12:34:32.844 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/30' 2023-10-30T12:34:32.844 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/33' 2023-10-30T12:34:32.844 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/23' 2023-10-30T12:34:32.844 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/35' 2023-10-30T12:34:32.845 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/26' 2023-10-30T12:34:32.845 INFO:tasks.workunit.client.0.smithi125.stdout:removed './271/37' 2023-10-30T12:34:32.845 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './271' 2023-10-30T12:34:32.845 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/21' 2023-10-30T12:34:32.845 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/27' 2023-10-30T12:34:32.845 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/31' 2023-10-30T12:34:32.846 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/36' 2023-10-30T12:34:32.846 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/20' 2023-10-30T12:34:32.846 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/39' 2023-10-30T12:34:32.846 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/34' 2023-10-30T12:34:32.847 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/38' 2023-10-30T12:34:32.847 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/28' 2023-10-30T12:34:32.847 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/24' 2023-10-30T12:34:32.847 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/29' 2023-10-30T12:34:32.847 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/32' 2023-10-30T12:34:32.848 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/22' 2023-10-30T12:34:32.848 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/25' 2023-10-30T12:34:32.848 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/30' 2023-10-30T12:34:32.848 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/33' 2023-10-30T12:34:32.848 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/23' 2023-10-30T12:34:32.848 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/35' 2023-10-30T12:34:32.849 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/26' 2023-10-30T12:34:32.849 INFO:tasks.workunit.client.0.smithi125.stdout:removed './272/37' 2023-10-30T12:34:32.849 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './272' 2023-10-30T12:34:32.849 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/21' 2023-10-30T12:34:32.849 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/27' 2023-10-30T12:34:32.850 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/31' 2023-10-30T12:34:32.850 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/36' 2023-10-30T12:34:32.850 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/20' 2023-10-30T12:34:32.851 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/39' 2023-10-30T12:34:32.851 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/34' 2023-10-30T12:34:32.851 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/38' 2023-10-30T12:34:32.851 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/28' 2023-10-30T12:34:32.852 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/24' 2023-10-30T12:34:32.852 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/29' 2023-10-30T12:34:32.852 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/32' 2023-10-30T12:34:32.852 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/22' 2023-10-30T12:34:32.852 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/25' 2023-10-30T12:34:32.853 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/30' 2023-10-30T12:34:32.853 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/33' 2023-10-30T12:34:32.853 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/23' 2023-10-30T12:34:32.854 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/35' 2023-10-30T12:34:32.854 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/26' 2023-10-30T12:34:32.854 INFO:tasks.workunit.client.0.smithi125.stdout:removed './273/37' 2023-10-30T12:34:32.854 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './273' 2023-10-30T12:34:32.854 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/21' 2023-10-30T12:34:32.855 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/27' 2023-10-30T12:34:32.855 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/31' 2023-10-30T12:34:32.855 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/36' 2023-10-30T12:34:32.855 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/20' 2023-10-30T12:34:32.855 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/39' 2023-10-30T12:34:32.855 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/34' 2023-10-30T12:34:32.856 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/38' 2023-10-30T12:34:32.856 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/28' 2023-10-30T12:34:32.856 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/24' 2023-10-30T12:34:32.856 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/29' 2023-10-30T12:34:32.856 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/32' 2023-10-30T12:34:32.857 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/22' 2023-10-30T12:34:32.857 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/25' 2023-10-30T12:34:32.857 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/30' 2023-10-30T12:34:32.857 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/33' 2023-10-30T12:34:32.857 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/23' 2023-10-30T12:34:32.858 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/35' 2023-10-30T12:34:32.858 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/26' 2023-10-30T12:34:32.858 INFO:tasks.workunit.client.0.smithi125.stdout:removed './274/37' 2023-10-30T12:34:32.858 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './274' 2023-10-30T12:34:32.858 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/21' 2023-10-30T12:34:32.859 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/27' 2023-10-30T12:34:32.859 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/31' 2023-10-30T12:34:32.859 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/36' 2023-10-30T12:34:32.859 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/20' 2023-10-30T12:34:32.859 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/39' 2023-10-30T12:34:32.860 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/34' 2023-10-30T12:34:32.862 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/38' 2023-10-30T12:34:32.862 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/28' 2023-10-30T12:34:32.862 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/24' 2023-10-30T12:34:32.862 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/29' 2023-10-30T12:34:32.862 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/32' 2023-10-30T12:34:32.863 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/22' 2023-10-30T12:34:32.863 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/25' 2023-10-30T12:34:32.863 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/30' 2023-10-30T12:34:32.863 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/33' 2023-10-30T12:34:32.863 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/23' 2023-10-30T12:34:32.864 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/35' 2023-10-30T12:34:32.864 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/26' 2023-10-30T12:34:32.864 INFO:tasks.workunit.client.0.smithi125.stdout:removed './275/37' 2023-10-30T12:34:32.864 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './275' 2023-10-30T12:34:32.864 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/21' 2023-10-30T12:34:32.865 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/27' 2023-10-30T12:34:32.865 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/31' 2023-10-30T12:34:32.865 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/36' 2023-10-30T12:34:32.865 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/20' 2023-10-30T12:34:32.865 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/39' 2023-10-30T12:34:32.865 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/34' 2023-10-30T12:34:32.866 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/38' 2023-10-30T12:34:32.866 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/28' 2023-10-30T12:34:32.866 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/24' 2023-10-30T12:34:32.866 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/29' 2023-10-30T12:34:32.866 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/32' 2023-10-30T12:34:32.867 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/22' 2023-10-30T12:34:32.867 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/25' 2023-10-30T12:34:32.867 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/30' 2023-10-30T12:34:32.867 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/33' 2023-10-30T12:34:32.867 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/23' 2023-10-30T12:34:32.868 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/35' 2023-10-30T12:34:32.868 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/26' 2023-10-30T12:34:35.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed './276/37' 2023-10-30T12:34:35.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './276' 2023-10-30T12:34:35.637 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/21' 2023-10-30T12:34:35.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/27' 2023-10-30T12:34:35.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/31' 2023-10-30T12:34:35.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/36' 2023-10-30T12:34:35.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/20' 2023-10-30T12:34:35.638 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/39' 2023-10-30T12:34:35.639 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/34' 2023-10-30T12:34:35.639 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/38' 2023-10-30T12:34:35.639 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/28' 2023-10-30T12:34:35.639 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/24' 2023-10-30T12:34:35.640 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/29' 2023-10-30T12:34:35.640 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/32' 2023-10-30T12:34:35.640 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/22' 2023-10-30T12:34:35.640 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/25' 2023-10-30T12:34:35.640 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/30' 2023-10-30T12:34:35.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/33' 2023-10-30T12:34:35.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/23' 2023-10-30T12:34:35.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/35' 2023-10-30T12:34:35.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/26' 2023-10-30T12:34:35.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed './277/37' 2023-10-30T12:34:35.641 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './277' 2023-10-30T12:34:35.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/21' 2023-10-30T12:34:35.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/27' 2023-10-30T12:34:35.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/31' 2023-10-30T12:34:35.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/36' 2023-10-30T12:34:35.642 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/20' 2023-10-30T12:34:35.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/39' 2023-10-30T12:34:35.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/34' 2023-10-30T12:34:35.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/38' 2023-10-30T12:34:35.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/28' 2023-10-30T12:34:35.643 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/24' 2023-10-30T12:34:35.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/29' 2023-10-30T12:34:35.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/32' 2023-10-30T12:34:35.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/22' 2023-10-30T12:34:35.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/25' 2023-10-30T12:34:35.644 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/30' 2023-10-30T12:34:35.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/33' 2023-10-30T12:34:35.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/23' 2023-10-30T12:34:35.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/35' 2023-10-30T12:34:35.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/26' 2023-10-30T12:34:35.645 INFO:tasks.workunit.client.0.smithi125.stdout:removed './278/37' 2023-10-30T12:34:35.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './278' 2023-10-30T12:34:35.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/21' 2023-10-30T12:34:35.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/27' 2023-10-30T12:34:35.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/31' 2023-10-30T12:34:35.646 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/36' 2023-10-30T12:34:35.647 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/20' 2023-10-30T12:34:35.647 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/39' 2023-10-30T12:34:35.647 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/34' 2023-10-30T12:34:35.647 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/38' 2023-10-30T12:34:35.647 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/28' 2023-10-30T12:34:35.647 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/24' 2023-10-30T12:34:35.648 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/29' 2023-10-30T12:34:35.648 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/32' 2023-10-30T12:34:35.648 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/22' 2023-10-30T12:34:35.648 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/25' 2023-10-30T12:34:35.648 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/30' 2023-10-30T12:34:35.649 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/33' 2023-10-30T12:34:35.649 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/23' 2023-10-30T12:34:35.649 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/35' 2023-10-30T12:34:35.649 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/26' 2023-10-30T12:34:35.649 INFO:tasks.workunit.client.0.smithi125.stdout:removed './279/37' 2023-10-30T12:34:35.650 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './279' 2023-10-30T12:34:35.650 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/21' 2023-10-30T12:34:35.650 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/27' 2023-10-30T12:34:35.650 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/31' 2023-10-30T12:34:35.650 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/36' 2023-10-30T12:34:35.651 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/20' 2023-10-30T12:34:35.651 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/39' 2023-10-30T12:34:35.651 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/34' 2023-10-30T12:34:35.651 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/38' 2023-10-30T12:34:35.651 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/28' 2023-10-30T12:34:35.652 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/24' 2023-10-30T12:34:35.652 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/29' 2023-10-30T12:34:35.652 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/32' 2023-10-30T12:34:35.652 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/22' 2023-10-30T12:34:35.652 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/25' 2023-10-30T12:34:35.652 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/30' 2023-10-30T12:34:35.653 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/33' 2023-10-30T12:34:35.653 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/23' 2023-10-30T12:34:35.653 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/35' 2023-10-30T12:34:35.653 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/26' 2023-10-30T12:34:35.653 INFO:tasks.workunit.client.0.smithi125.stdout:removed './280/37' 2023-10-30T12:34:35.654 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './280' 2023-10-30T12:34:35.654 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/21' 2023-10-30T12:34:35.654 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/27' 2023-10-30T12:34:35.654 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/31' 2023-10-30T12:34:35.654 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/36' 2023-10-30T12:34:35.654 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/20' 2023-10-30T12:34:35.655 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/39' 2023-10-30T12:34:35.655 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/34' 2023-10-30T12:34:35.655 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/38' 2023-10-30T12:34:35.655 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/28' 2023-10-30T12:34:35.655 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/24' 2023-10-30T12:34:35.656 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/29' 2023-10-30T12:34:35.656 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/32' 2023-10-30T12:34:35.656 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/22' 2023-10-30T12:34:35.656 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/25' 2023-10-30T12:34:35.656 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/30' 2023-10-30T12:34:35.657 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/33' 2023-10-30T12:34:35.657 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/23' 2023-10-30T12:34:35.657 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/35' 2023-10-30T12:34:35.657 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/26' 2023-10-30T12:34:35.657 INFO:tasks.workunit.client.0.smithi125.stdout:removed './281/37' 2023-10-30T12:34:35.657 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './281' 2023-10-30T12:34:35.658 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/21' 2023-10-30T12:34:35.658 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/27' 2023-10-30T12:34:35.658 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/31' 2023-10-30T12:34:35.658 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/36' 2023-10-30T12:34:35.658 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/20' 2023-10-30T12:34:35.659 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/39' 2023-10-30T12:34:35.659 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/34' 2023-10-30T12:34:35.659 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/38' 2023-10-30T12:34:35.659 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/28' 2023-10-30T12:34:35.659 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/24' 2023-10-30T12:34:35.660 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/29' 2023-10-30T12:34:35.660 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/32' 2023-10-30T12:34:35.660 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/22' 2023-10-30T12:34:35.660 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/25' 2023-10-30T12:34:35.660 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/30' 2023-10-30T12:34:35.660 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/33' 2023-10-30T12:34:35.661 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/23' 2023-10-30T12:34:35.661 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/35' 2023-10-30T12:34:35.661 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/26' 2023-10-30T12:34:35.661 INFO:tasks.workunit.client.0.smithi125.stdout:removed './282/37' 2023-10-30T12:34:35.661 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './282' 2023-10-30T12:34:35.662 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/21' 2023-10-30T12:34:35.662 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/27' 2023-10-30T12:34:35.662 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/31' 2023-10-30T12:34:35.662 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/36' 2023-10-30T12:34:35.662 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/20' 2023-10-30T12:34:35.663 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/39' 2023-10-30T12:34:35.663 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/34' 2023-10-30T12:34:35.663 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/38' 2023-10-30T12:34:35.663 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/28' 2023-10-30T12:34:35.663 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/24' 2023-10-30T12:34:35.664 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/29' 2023-10-30T12:34:35.664 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/32' 2023-10-30T12:34:35.664 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/22' 2023-10-30T12:34:35.664 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/25' 2023-10-30T12:34:35.664 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/30' 2023-10-30T12:34:35.664 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/33' 2023-10-30T12:34:35.665 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/23' 2023-10-30T12:34:35.665 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/35' 2023-10-30T12:34:35.665 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/26' 2023-10-30T12:34:35.665 INFO:tasks.workunit.client.0.smithi125.stdout:removed './283/37' 2023-10-30T12:34:35.665 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './283' 2023-10-30T12:34:35.666 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/21' 2023-10-30T12:34:35.666 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/27' 2023-10-30T12:34:35.666 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/31' 2023-10-30T12:34:35.666 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/36' 2023-10-30T12:34:35.666 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/20' 2023-10-30T12:34:35.667 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/39' 2023-10-30T12:34:35.667 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/34' 2023-10-30T12:34:35.667 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/38' 2023-10-30T12:34:35.667 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/28' 2023-10-30T12:34:35.667 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/24' 2023-10-30T12:34:35.667 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/29' 2023-10-30T12:34:35.668 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/32' 2023-10-30T12:34:35.668 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/22' 2023-10-30T12:34:35.668 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/25' 2023-10-30T12:34:35.668 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/30' 2023-10-30T12:34:35.668 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/33' 2023-10-30T12:34:35.669 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/23' 2023-10-30T12:34:35.669 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/35' 2023-10-30T12:34:35.669 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/26' 2023-10-30T12:34:35.669 INFO:tasks.workunit.client.0.smithi125.stdout:removed './284/37' 2023-10-30T12:34:35.669 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './284' 2023-10-30T12:34:35.669 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/21' 2023-10-30T12:34:35.670 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/27' 2023-10-30T12:34:35.670 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/31' 2023-10-30T12:34:35.670 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/36' 2023-10-30T12:34:35.670 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/20' 2023-10-30T12:34:35.670 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/39' 2023-10-30T12:34:35.671 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/34' 2023-10-30T12:34:35.671 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/38' 2023-10-30T12:34:35.671 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/28' 2023-10-30T12:34:35.671 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/24' 2023-10-30T12:34:35.671 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/29' 2023-10-30T12:34:35.672 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/32' 2023-10-30T12:34:35.672 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/22' 2023-10-30T12:34:35.672 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/25' 2023-10-30T12:34:35.672 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/30' 2023-10-30T12:34:35.672 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/33' 2023-10-30T12:34:35.672 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/23' 2023-10-30T12:34:35.673 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/35' 2023-10-30T12:34:35.673 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/26' 2023-10-30T12:34:35.673 INFO:tasks.workunit.client.0.smithi125.stdout:removed './285/37' 2023-10-30T12:34:35.673 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './285' 2023-10-30T12:34:35.673 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/21' 2023-10-30T12:34:35.674 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/27' 2023-10-30T12:34:35.674 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/31' 2023-10-30T12:34:35.674 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/36' 2023-10-30T12:34:35.674 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/20' 2023-10-30T12:34:35.674 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/39' 2023-10-30T12:34:35.675 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/34' 2023-10-30T12:34:35.675 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/38' 2023-10-30T12:34:35.675 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/28' 2023-10-30T12:34:35.675 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/24' 2023-10-30T12:34:35.675 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/29' 2023-10-30T12:34:35.676 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/32' 2023-10-30T12:34:35.676 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/22' 2023-10-30T12:34:35.676 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/25' 2023-10-30T12:34:35.676 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/30' 2023-10-30T12:34:35.676 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/33' 2023-10-30T12:34:35.677 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/23' 2023-10-30T12:34:35.677 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/35' 2023-10-30T12:34:35.677 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/26' 2023-10-30T12:34:35.677 INFO:tasks.workunit.client.0.smithi125.stdout:removed './286/37' 2023-10-30T12:34:35.677 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './286' 2023-10-30T12:34:37.783 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/21' 2023-10-30T12:34:37.784 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/27' 2023-10-30T12:34:37.784 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/31' 2023-10-30T12:34:37.784 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/36' 2023-10-30T12:34:37.784 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/20' 2023-10-30T12:34:37.785 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/39' 2023-10-30T12:34:37.785 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/34' 2023-10-30T12:34:37.785 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/38' 2023-10-30T12:34:37.785 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/28' 2023-10-30T12:34:37.785 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/24' 2023-10-30T12:34:37.785 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/29' 2023-10-30T12:34:37.786 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/32' 2023-10-30T12:34:37.786 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/22' 2023-10-30T12:34:37.786 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/25' 2023-10-30T12:34:37.786 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/30' 2023-10-30T12:34:37.787 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/33' 2023-10-30T12:34:37.787 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/23' 2023-10-30T12:34:37.787 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/35' 2023-10-30T12:34:37.787 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/26' 2023-10-30T12:34:37.787 INFO:tasks.workunit.client.0.smithi125.stdout:removed './287/37' 2023-10-30T12:34:37.788 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './287' 2023-10-30T12:34:37.788 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/21' 2023-10-30T12:34:37.788 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/27' 2023-10-30T12:34:37.788 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/31' 2023-10-30T12:34:37.788 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/36' 2023-10-30T12:34:37.788 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/20' 2023-10-30T12:34:37.789 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/39' 2023-10-30T12:34:37.789 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/34' 2023-10-30T12:34:37.789 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/38' 2023-10-30T12:34:37.789 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/28' 2023-10-30T12:34:37.789 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/24' 2023-10-30T12:34:37.790 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/29' 2023-10-30T12:34:37.790 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/32' 2023-10-30T12:34:37.790 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/22' 2023-10-30T12:34:37.790 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/25' 2023-10-30T12:34:37.790 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/30' 2023-10-30T12:34:37.794 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/33' 2023-10-30T12:34:37.794 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/23' 2023-10-30T12:34:37.794 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/35' 2023-10-30T12:34:37.795 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/26' 2023-10-30T12:34:37.795 INFO:tasks.workunit.client.0.smithi125.stdout:removed './288/37' 2023-10-30T12:34:37.795 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './288' 2023-10-30T12:34:37.795 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/21' 2023-10-30T12:34:37.795 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/27' 2023-10-30T12:34:37.796 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/31' 2023-10-30T12:34:37.796 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/36' 2023-10-30T12:34:37.796 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/20' 2023-10-30T12:34:37.796 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/39' 2023-10-30T12:34:37.796 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/34' 2023-10-30T12:34:37.797 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/38' 2023-10-30T12:34:37.797 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/28' 2023-10-30T12:34:37.797 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/24' 2023-10-30T12:34:37.797 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/29' 2023-10-30T12:34:37.797 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/32' 2023-10-30T12:34:37.798 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/22' 2023-10-30T12:34:37.798 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/25' 2023-10-30T12:34:37.798 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/30' 2023-10-30T12:34:37.798 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/33' 2023-10-30T12:34:37.798 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/23' 2023-10-30T12:34:37.799 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/35' 2023-10-30T12:34:37.799 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/26' 2023-10-30T12:34:37.799 INFO:tasks.workunit.client.0.smithi125.stdout:removed './289/37' 2023-10-30T12:34:37.799 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './289' 2023-10-30T12:34:37.799 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/21' 2023-10-30T12:34:37.800 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/27' 2023-10-30T12:34:37.800 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/31' 2023-10-30T12:34:37.800 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/36' 2023-10-30T12:34:37.800 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/20' 2023-10-30T12:34:37.800 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/39' 2023-10-30T12:34:37.800 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/34' 2023-10-30T12:34:37.801 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/38' 2023-10-30T12:34:37.801 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/28' 2023-10-30T12:34:37.801 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/24' 2023-10-30T12:34:37.801 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/29' 2023-10-30T12:34:37.801 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/32' 2023-10-30T12:34:37.802 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/22' 2023-10-30T12:34:37.802 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/25' 2023-10-30T12:34:37.802 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/30' 2023-10-30T12:34:37.802 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/33' 2023-10-30T12:34:37.802 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/23' 2023-10-30T12:34:37.803 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/35' 2023-10-30T12:34:37.803 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/26' 2023-10-30T12:34:37.803 INFO:tasks.workunit.client.0.smithi125.stdout:removed './290/37' 2023-10-30T12:34:37.803 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './290' 2023-10-30T12:34:37.803 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/21' 2023-10-30T12:34:37.804 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/27' 2023-10-30T12:34:37.804 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/31' 2023-10-30T12:34:37.804 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/36' 2023-10-30T12:34:37.804 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/20' 2023-10-30T12:34:37.804 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/39' 2023-10-30T12:34:37.805 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/34' 2023-10-30T12:34:37.805 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/38' 2023-10-30T12:34:37.805 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/28' 2023-10-30T12:34:37.805 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/24' 2023-10-30T12:34:37.805 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/29' 2023-10-30T12:34:37.806 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/32' 2023-10-30T12:34:37.806 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/22' 2023-10-30T12:34:37.806 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/25' 2023-10-30T12:34:37.806 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/30' 2023-10-30T12:34:37.806 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/33' 2023-10-30T12:34:37.807 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/23' 2023-10-30T12:34:37.807 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/35' 2023-10-30T12:34:37.807 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/26' 2023-10-30T12:34:37.807 INFO:tasks.workunit.client.0.smithi125.stdout:removed './291/37' 2023-10-30T12:34:37.807 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './291' 2023-10-30T12:34:37.808 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/21' 2023-10-30T12:34:37.808 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/27' 2023-10-30T12:34:37.808 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/31' 2023-10-30T12:34:37.808 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/36' 2023-10-30T12:34:37.808 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/20' 2023-10-30T12:34:37.808 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/39' 2023-10-30T12:34:37.809 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/34' 2023-10-30T12:34:37.809 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/38' 2023-10-30T12:34:37.809 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/28' 2023-10-30T12:34:37.809 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/24' 2023-10-30T12:34:37.809 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/29' 2023-10-30T12:34:37.810 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/32' 2023-10-30T12:34:37.810 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/22' 2023-10-30T12:34:37.810 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/25' 2023-10-30T12:34:37.810 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/30' 2023-10-30T12:34:37.810 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/33' 2023-10-30T12:34:37.811 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/23' 2023-10-30T12:34:37.811 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/35' 2023-10-30T12:34:37.811 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/26' 2023-10-30T12:34:37.811 INFO:tasks.workunit.client.0.smithi125.stdout:removed './292/37' 2023-10-30T12:34:37.811 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './292' 2023-10-30T12:34:37.812 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/21' 2023-10-30T12:34:37.812 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/27' 2023-10-30T12:34:37.812 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/31' 2023-10-30T12:34:37.812 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/36' 2023-10-30T12:34:37.812 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/20' 2023-10-30T12:34:37.813 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/39' 2023-10-30T12:34:37.813 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/34' 2023-10-30T12:34:37.813 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/38' 2023-10-30T12:34:37.813 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/28' 2023-10-30T12:34:37.813 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/24' 2023-10-30T12:34:37.814 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/29' 2023-10-30T12:34:37.814 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/32' 2023-10-30T12:34:37.814 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/22' 2023-10-30T12:34:37.814 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/25' 2023-10-30T12:34:37.814 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/30' 2023-10-30T12:34:37.814 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/33' 2023-10-30T12:34:37.815 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/23' 2023-10-30T12:34:37.815 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/35' 2023-10-30T12:34:37.815 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/26' 2023-10-30T12:34:37.815 INFO:tasks.workunit.client.0.smithi125.stdout:removed './293/37' 2023-10-30T12:34:37.815 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './293' 2023-10-30T12:34:37.816 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/21' 2023-10-30T12:34:37.816 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/27' 2023-10-30T12:34:37.816 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/31' 2023-10-30T12:34:37.816 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/36' 2023-10-30T12:34:37.816 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/20' 2023-10-30T12:34:37.817 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/39' 2023-10-30T12:34:37.817 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/34' 2023-10-30T12:34:37.817 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/38' 2023-10-30T12:34:37.817 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/28' 2023-10-30T12:34:37.817 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/24' 2023-10-30T12:34:37.817 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/29' 2023-10-30T12:34:37.818 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/32' 2023-10-30T12:34:37.818 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/22' 2023-10-30T12:34:37.818 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/25' 2023-10-30T12:34:37.818 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/30' 2023-10-30T12:34:37.819 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/33' 2023-10-30T12:34:37.819 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/23' 2023-10-30T12:34:37.819 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/35' 2023-10-30T12:34:37.819 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/26' 2023-10-30T12:34:37.819 INFO:tasks.workunit.client.0.smithi125.stdout:removed './294/37' 2023-10-30T12:34:37.819 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './294' 2023-10-30T12:34:37.820 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/21' 2023-10-30T12:34:37.820 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/27' 2023-10-30T12:34:37.820 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/31' 2023-10-30T12:34:37.820 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/36' 2023-10-30T12:34:37.820 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/20' 2023-10-30T12:34:37.821 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/39' 2023-10-30T12:34:37.821 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/34' 2023-10-30T12:34:37.821 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/38' 2023-10-30T12:34:37.821 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/28' 2023-10-30T12:34:37.821 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/24' 2023-10-30T12:34:37.822 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/29' 2023-10-30T12:34:37.822 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/32' 2023-10-30T12:34:37.822 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/22' 2023-10-30T12:34:37.822 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/25' 2023-10-30T12:34:37.822 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/30' 2023-10-30T12:34:37.822 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/33' 2023-10-30T12:34:37.823 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/23' 2023-10-30T12:34:37.823 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/35' 2023-10-30T12:34:37.823 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/26' 2023-10-30T12:34:37.823 INFO:tasks.workunit.client.0.smithi125.stdout:removed './295/37' 2023-10-30T12:34:37.824 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './295' 2023-10-30T12:34:37.824 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/21' 2023-10-30T12:34:37.824 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/27' 2023-10-30T12:34:37.824 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/31' 2023-10-30T12:34:37.824 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/36' 2023-10-30T12:34:37.825 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/20' 2023-10-30T12:34:37.825 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/39' 2023-10-30T12:34:37.825 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/34' 2023-10-30T12:34:37.825 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/38' 2023-10-30T12:34:37.825 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/28' 2023-10-30T12:34:37.826 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/24' 2023-10-30T12:34:37.826 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/29' 2023-10-30T12:34:37.826 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/32' 2023-10-30T12:34:37.826 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/22' 2023-10-30T12:34:37.826 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/25' 2023-10-30T12:34:37.827 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/30' 2023-10-30T12:34:37.827 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/33' 2023-10-30T12:34:37.827 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/23' 2023-10-30T12:34:37.827 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/35' 2023-10-30T12:34:37.828 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/26' 2023-10-30T12:34:37.828 INFO:tasks.workunit.client.0.smithi125.stdout:removed './296/37' 2023-10-30T12:34:37.828 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './296' 2023-10-30T12:34:37.828 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/21' 2023-10-30T12:34:37.828 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/27' 2023-10-30T12:34:44.380 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/31' 2023-10-30T12:34:44.381 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/36' 2023-10-30T12:34:44.381 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/20' 2023-10-30T12:34:44.381 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/39' 2023-10-30T12:34:44.382 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/34' 2023-10-30T12:34:44.382 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/38' 2023-10-30T12:34:44.382 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/28' 2023-10-30T12:34:44.382 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/24' 2023-10-30T12:34:44.382 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/29' 2023-10-30T12:34:44.383 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/32' 2023-10-30T12:34:44.383 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/22' 2023-10-30T12:34:44.383 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/25' 2023-10-30T12:34:44.383 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/30' 2023-10-30T12:34:44.383 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/33' 2023-10-30T12:34:44.384 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/23' 2023-10-30T12:34:44.384 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/35' 2023-10-30T12:34:44.384 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/26' 2023-10-30T12:34:44.384 INFO:tasks.workunit.client.0.smithi125.stdout:removed './297/37' 2023-10-30T12:34:44.384 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './297' 2023-10-30T12:34:44.385 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/21' 2023-10-30T12:34:44.385 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/27' 2023-10-30T12:34:44.385 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/31' 2023-10-30T12:34:44.385 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/36' 2023-10-30T12:34:44.385 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/20' 2023-10-30T12:34:44.386 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/39' 2023-10-30T12:34:44.386 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/34' 2023-10-30T12:34:44.386 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/38' 2023-10-30T12:34:44.386 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/28' 2023-10-30T12:34:44.386 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/24' 2023-10-30T12:34:44.387 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/29' 2023-10-30T12:34:44.387 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/32' 2023-10-30T12:34:44.387 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/22' 2023-10-30T12:34:44.388 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/25' 2023-10-30T12:34:44.388 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/30' 2023-10-30T12:34:44.388 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/33' 2023-10-30T12:34:44.388 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/23' 2023-10-30T12:34:44.388 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/35' 2023-10-30T12:34:44.388 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/26' 2023-10-30T12:34:44.389 INFO:tasks.workunit.client.0.smithi125.stdout:removed './298/37' 2023-10-30T12:34:44.389 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './298' 2023-10-30T12:34:44.389 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/21' 2023-10-30T12:34:44.389 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/27' 2023-10-30T12:34:44.389 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/31' 2023-10-30T12:34:44.390 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/36' 2023-10-30T12:34:44.390 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/20' 2023-10-30T12:34:44.390 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/39' 2023-10-30T12:34:44.390 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/34' 2023-10-30T12:34:44.390 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/38' 2023-10-30T12:34:44.391 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/28' 2023-10-30T12:34:44.391 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/24' 2023-10-30T12:34:44.391 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/29' 2023-10-30T12:34:44.391 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/32' 2023-10-30T12:34:44.391 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/22' 2023-10-30T12:34:44.392 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/25' 2023-10-30T12:34:44.392 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/30' 2023-10-30T12:34:44.392 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/33' 2023-10-30T12:34:44.392 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/23' 2023-10-30T12:34:44.392 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/35' 2023-10-30T12:34:44.392 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/26' 2023-10-30T12:34:44.393 INFO:tasks.workunit.client.0.smithi125.stdout:removed './299/37' 2023-10-30T12:34:44.393 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './299' 2023-10-30T12:34:44.393 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/166/19' 2023-10-30T12:34:44.393 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/166/15' 2023-10-30T12:34:44.393 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/166/20' 2023-10-30T12:34:44.394 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/166/12' 2023-10-30T12:34:44.394 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/166/10' 2023-10-30T12:34:44.394 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/166/11' 2023-10-30T12:34:44.394 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/166/16' 2023-10-30T12:34:44.394 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/166/14' 2023-10-30T12:34:44.395 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/166/18' 2023-10-30T12:34:44.395 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/166/13' 2023-10-30T12:34:44.395 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/166/17' 2023-10-30T12:34:44.395 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/166' 2023-10-30T12:34:44.395 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/147/19' 2023-10-30T12:34:44.396 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/147/15' 2023-10-30T12:34:44.396 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/147/20' 2023-10-30T12:34:44.396 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/147/12' 2023-10-30T12:34:44.396 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/147/10' 2023-10-30T12:34:44.396 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/147/11' 2023-10-30T12:34:44.397 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/147/16' 2023-10-30T12:34:44.397 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/147/14' 2023-10-30T12:34:44.397 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/147/18' 2023-10-30T12:34:44.398 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/147/13' 2023-10-30T12:34:44.398 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/147/17' 2023-10-30T12:34:44.398 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/147' 2023-10-30T12:34:44.398 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/151/19' 2023-10-30T12:34:44.399 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/151/15' 2023-10-30T12:34:44.399 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/151/20' 2023-10-30T12:34:44.399 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/151/12' 2023-10-30T12:34:44.399 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/151/10' 2023-10-30T12:34:44.399 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/151/11' 2023-10-30T12:34:44.400 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/151/16' 2023-10-30T12:34:44.400 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/151/14' 2023-10-30T12:34:44.400 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/151/18' 2023-10-30T12:34:44.400 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/151/13' 2023-10-30T12:34:44.400 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/151/17' 2023-10-30T12:34:44.401 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/151' 2023-10-30T12:34:44.401 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/170/19' 2023-10-30T12:34:44.401 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/170/15' 2023-10-30T12:34:44.401 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/170/20' 2023-10-30T12:34:44.401 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/170/12' 2023-10-30T12:34:44.402 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/170/10' 2023-10-30T12:34:44.402 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/170/11' 2023-10-30T12:34:44.402 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/170/16' 2023-10-30T12:34:44.402 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/170/14' 2023-10-30T12:34:44.402 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/170/18' 2023-10-30T12:34:44.403 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/170/13' 2023-10-30T12:34:44.403 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/170/17' 2023-10-30T12:34:44.403 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/170' 2023-10-30T12:34:44.403 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/167/19' 2023-10-30T12:34:44.403 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/167/15' 2023-10-30T12:34:44.404 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/167/20' 2023-10-30T12:34:44.404 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/167/12' 2023-10-30T12:34:44.404 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/167/10' 2023-10-30T12:34:44.404 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/167/11' 2023-10-30T12:34:44.404 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/167/16' 2023-10-30T12:34:44.405 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/167/14' 2023-10-30T12:34:44.405 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/167/18' 2023-10-30T12:34:44.405 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/167/13' 2023-10-30T12:34:44.405 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/167/17' 2023-10-30T12:34:44.405 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/167' 2023-10-30T12:34:44.406 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/140/19' 2023-10-30T12:34:44.406 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/140/15' 2023-10-30T12:34:44.406 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/140/20' 2023-10-30T12:34:44.406 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/140/12' 2023-10-30T12:34:44.406 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/140/10' 2023-10-30T12:34:44.407 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/140/11' 2023-10-30T12:34:44.407 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/140/16' 2023-10-30T12:34:44.407 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/140/14' 2023-10-30T12:34:44.407 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/140/18' 2023-10-30T12:34:44.408 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/140/13' 2023-10-30T12:34:44.408 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/140/17' 2023-10-30T12:34:44.408 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/140' 2023-10-30T12:34:44.408 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/142/19' 2023-10-30T12:34:44.408 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/142/15' 2023-10-30T12:34:44.409 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/142/20' 2023-10-30T12:34:44.409 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/142/12' 2023-10-30T12:34:44.409 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/142/10' 2023-10-30T12:34:44.664 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:34:44.668 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:34:48.246 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/142/11' 2023-10-30T12:34:48.246 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/142/16' 2023-10-30T12:34:48.247 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/142/14' 2023-10-30T12:34:48.247 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/142/18' 2023-10-30T12:34:48.247 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/142/13' 2023-10-30T12:34:48.247 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/142/17' 2023-10-30T12:34:48.247 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/142' 2023-10-30T12:34:48.248 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/150/19' 2023-10-30T12:34:48.248 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/150/15' 2023-10-30T12:34:48.248 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/150/20' 2023-10-30T12:34:48.248 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/150/12' 2023-10-30T12:34:48.248 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/150/10' 2023-10-30T12:34:48.249 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/150/11' 2023-10-30T12:34:48.249 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/150/16' 2023-10-30T12:34:48.249 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/150/14' 2023-10-30T12:34:48.249 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/150/18' 2023-10-30T12:34:48.249 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/150/13' 2023-10-30T12:34:48.250 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/150/17' 2023-10-30T12:34:48.250 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/150' 2023-10-30T12:34:48.250 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/105/19' 2023-10-30T12:34:48.250 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/105/15' 2023-10-30T12:34:48.250 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/105/20' 2023-10-30T12:34:48.251 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/105/12' 2023-10-30T12:34:48.251 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/105/10' 2023-10-30T12:34:48.251 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/105/11' 2023-10-30T12:34:48.251 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/105/16' 2023-10-30T12:34:48.251 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/105/14' 2023-10-30T12:34:48.251 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/105/18' 2023-10-30T12:34:48.252 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/105/13' 2023-10-30T12:34:48.252 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/105/17' 2023-10-30T12:34:48.252 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/105' 2023-10-30T12:34:48.252 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/149/19' 2023-10-30T12:34:48.252 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/149/15' 2023-10-30T12:34:48.253 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/149/20' 2023-10-30T12:34:48.253 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/149/12' 2023-10-30T12:34:48.253 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/149/10' 2023-10-30T12:34:48.253 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/149/11' 2023-10-30T12:34:48.253 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/149/16' 2023-10-30T12:34:48.253 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/149/14' 2023-10-30T12:34:48.254 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/149/18' 2023-10-30T12:34:48.254 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/149/13' 2023-10-30T12:34:48.254 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/149/17' 2023-10-30T12:34:48.254 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/149' 2023-10-30T12:34:48.254 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/109/19' 2023-10-30T12:34:48.255 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/109/15' 2023-10-30T12:34:48.255 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/109/20' 2023-10-30T12:34:48.255 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/109/12' 2023-10-30T12:34:48.255 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/109/10' 2023-10-30T12:34:48.255 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/109/11' 2023-10-30T12:34:48.256 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/109/16' 2023-10-30T12:34:48.256 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/109/14' 2023-10-30T12:34:48.256 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/109/18' 2023-10-30T12:34:48.256 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/109/13' 2023-10-30T12:34:48.256 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/109/17' 2023-10-30T12:34:48.257 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/109' 2023-10-30T12:34:48.257 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/158/19' 2023-10-30T12:34:48.257 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/158/15' 2023-10-30T12:34:48.257 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/158/20' 2023-10-30T12:34:48.257 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/158/12' 2023-10-30T12:34:48.258 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/158/10' 2023-10-30T12:34:48.258 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/158/11' 2023-10-30T12:34:48.258 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/158/16' 2023-10-30T12:34:48.258 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/158/14' 2023-10-30T12:34:48.258 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/158/18' 2023-10-30T12:34:48.259 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/158/13' 2023-10-30T12:34:48.259 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/158/17' 2023-10-30T12:34:48.259 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/158' 2023-10-30T12:34:48.259 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/106/19' 2023-10-30T12:34:48.259 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/106/15' 2023-10-30T12:34:48.260 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/106/20' 2023-10-30T12:34:48.260 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/106/12' 2023-10-30T12:34:48.260 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/106/10' 2023-10-30T12:34:48.260 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/106/11' 2023-10-30T12:34:48.260 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/106/16' 2023-10-30T12:34:48.260 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/106/14' 2023-10-30T12:34:48.261 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/106/18' 2023-10-30T12:34:48.261 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/106/13' 2023-10-30T12:34:48.261 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/106/17' 2023-10-30T12:34:48.261 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/106' 2023-10-30T12:34:48.261 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/189/19' 2023-10-30T12:34:48.262 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/189/15' 2023-10-30T12:34:48.262 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/189/20' 2023-10-30T12:34:48.262 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/189/12' 2023-10-30T12:34:48.262 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/189/10' 2023-10-30T12:34:48.262 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/189/11' 2023-10-30T12:34:48.263 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/189/16' 2023-10-30T12:34:48.263 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/189/14' 2023-10-30T12:34:48.263 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/189/18' 2023-10-30T12:34:48.263 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/189/13' 2023-10-30T12:34:48.263 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/189/17' 2023-10-30T12:34:48.264 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/189' 2023-10-30T12:34:48.264 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/133/19' 2023-10-30T12:34:48.264 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/133/15' 2023-10-30T12:34:48.264 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/133/20' 2023-10-30T12:34:48.264 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/133/12' 2023-10-30T12:34:48.265 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/133/10' 2023-10-30T12:34:48.265 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/133/11' 2023-10-30T12:34:48.265 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/133/16' 2023-10-30T12:34:48.265 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/133/14' 2023-10-30T12:34:48.265 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/133/18' 2023-10-30T12:34:48.265 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/133/13' 2023-10-30T12:34:48.266 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/133/17' 2023-10-30T12:34:48.266 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/133' 2023-10-30T12:34:48.266 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/173/19' 2023-10-30T12:34:48.266 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/173/15' 2023-10-30T12:34:48.266 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/173/20' 2023-10-30T12:34:48.267 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/173/12' 2023-10-30T12:34:48.267 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/173/10' 2023-10-30T12:34:48.267 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/173/11' 2023-10-30T12:34:51.706 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/173/16' 2023-10-30T12:34:51.706 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/173/14' 2023-10-30T12:34:51.706 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/173/18' 2023-10-30T12:34:51.706 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/173/13' 2023-10-30T12:34:51.707 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/173/17' 2023-10-30T12:34:51.707 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/173' 2023-10-30T12:34:51.707 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/174/19' 2023-10-30T12:34:51.707 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/174/15' 2023-10-30T12:34:51.708 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/174/20' 2023-10-30T12:34:51.708 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/174/12' 2023-10-30T12:34:51.708 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/174/10' 2023-10-30T12:34:51.708 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/174/11' 2023-10-30T12:34:51.709 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/174/16' 2023-10-30T12:34:51.709 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/174/14' 2023-10-30T12:34:51.709 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/174/18' 2023-10-30T12:34:51.710 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/174/13' 2023-10-30T12:34:51.710 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/174/17' 2023-10-30T12:34:51.710 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/174' 2023-10-30T12:34:51.710 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/155/19' 2023-10-30T12:34:51.711 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/155/15' 2023-10-30T12:34:51.711 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/155/20' 2023-10-30T12:34:51.711 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/155/12' 2023-10-30T12:34:51.711 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/155/10' 2023-10-30T12:34:51.711 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/155/11' 2023-10-30T12:34:51.712 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/155/16' 2023-10-30T12:34:51.712 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/155/14' 2023-10-30T12:34:51.712 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/155/18' 2023-10-30T12:34:51.712 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/155/13' 2023-10-30T12:34:51.712 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/155/17' 2023-10-30T12:34:51.713 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/155' 2023-10-30T12:34:51.713 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/114/19' 2023-10-30T12:34:51.713 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/114/15' 2023-10-30T12:34:51.713 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/114/20' 2023-10-30T12:34:51.713 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/114/12' 2023-10-30T12:34:51.714 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/114/10' 2023-10-30T12:34:51.714 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/114/11' 2023-10-30T12:34:51.714 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/114/16' 2023-10-30T12:34:51.714 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/114/14' 2023-10-30T12:34:51.715 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/114/18' 2023-10-30T12:34:51.715 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/114/13' 2023-10-30T12:34:51.715 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/114/17' 2023-10-30T12:34:51.716 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/114' 2023-10-30T12:34:51.716 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/159/19' 2023-10-30T12:34:51.716 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/159/15' 2023-10-30T12:34:51.717 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/159/20' 2023-10-30T12:34:51.717 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/159/12' 2023-10-30T12:34:51.717 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/159/10' 2023-10-30T12:34:51.718 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/159/11' 2023-10-30T12:34:51.718 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/159/16' 2023-10-30T12:34:51.719 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/159/14' 2023-10-30T12:34:51.719 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/159/18' 2023-10-30T12:34:51.719 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/159/13' 2023-10-30T12:34:51.719 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/159/17' 2023-10-30T12:34:51.719 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/159' 2023-10-30T12:34:51.720 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/187/19' 2023-10-30T12:34:51.720 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/187/15' 2023-10-30T12:34:51.720 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/187/20' 2023-10-30T12:34:51.720 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/187/12' 2023-10-30T12:34:51.721 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/187/10' 2023-10-30T12:34:51.721 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/187/11' 2023-10-30T12:34:51.721 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/187/16' 2023-10-30T12:34:51.721 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/187/14' 2023-10-30T12:34:51.721 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/187/18' 2023-10-30T12:34:51.721 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/187/13' 2023-10-30T12:34:51.722 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/187/17' 2023-10-30T12:34:51.722 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/187' 2023-10-30T12:34:51.722 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/185/19' 2023-10-30T12:34:51.722 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/185/15' 2023-10-30T12:34:51.723 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/185/20' 2023-10-30T12:34:51.723 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/185/12' 2023-10-30T12:34:51.723 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/185/10' 2023-10-30T12:34:51.723 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/185/11' 2023-10-30T12:34:51.725 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/185/16' 2023-10-30T12:34:51.725 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/185/14' 2023-10-30T12:34:51.726 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/185/18' 2023-10-30T12:34:51.726 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/185/13' 2023-10-30T12:34:51.726 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/185/17' 2023-10-30T12:34:51.726 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/185' 2023-10-30T12:34:51.726 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/120/19' 2023-10-30T12:34:51.727 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/120/15' 2023-10-30T12:34:51.727 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/120/20' 2023-10-30T12:34:51.727 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/120/12' 2023-10-30T12:34:51.727 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/120/10' 2023-10-30T12:34:51.727 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/120/11' 2023-10-30T12:34:51.728 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/120/16' 2023-10-30T12:34:51.728 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/120/14' 2023-10-30T12:34:51.728 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/120/18' 2023-10-30T12:34:51.728 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/120/13' 2023-10-30T12:34:51.728 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/120/17' 2023-10-30T12:34:51.729 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/120' 2023-10-30T12:34:51.729 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/110/19' 2023-10-30T12:34:51.729 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/110/15' 2023-10-30T12:34:51.729 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/110/20' 2023-10-30T12:34:51.729 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/110/12' 2023-10-30T12:34:51.730 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/110/10' 2023-10-30T12:34:51.730 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/110/11' 2023-10-30T12:34:51.730 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/110/16' 2023-10-30T12:34:51.730 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/110/14' 2023-10-30T12:34:51.730 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/110/18' 2023-10-30T12:34:51.731 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/110/13' 2023-10-30T12:34:51.731 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/110/17' 2023-10-30T12:34:51.731 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/110' 2023-10-30T12:34:51.731 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/137/19' 2023-10-30T12:34:51.731 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/137/15' 2023-10-30T12:34:51.732 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/137/20' 2023-10-30T12:34:51.732 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/137/12' 2023-10-30T12:34:51.732 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/137/10' 2023-10-30T12:34:51.733 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/137/11' 2023-10-30T12:34:51.733 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/137/16' 2023-10-30T12:34:54.966 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/137/14' 2023-10-30T12:34:54.966 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/137/18' 2023-10-30T12:34:54.966 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/137/13' 2023-10-30T12:34:54.967 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/137/17' 2023-10-30T12:34:54.967 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/137' 2023-10-30T12:34:54.967 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/198/19' 2023-10-30T12:34:54.967 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/198/15' 2023-10-30T12:34:54.967 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/198/20' 2023-10-30T12:34:54.968 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/198/12' 2023-10-30T12:34:54.968 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/198/10' 2023-10-30T12:34:54.968 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/198/11' 2023-10-30T12:34:54.968 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/198/16' 2023-10-30T12:34:54.969 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/198/14' 2023-10-30T12:34:54.969 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/198/18' 2023-10-30T12:34:54.969 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/198/13' 2023-10-30T12:34:54.969 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/198/17' 2023-10-30T12:34:54.969 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/198' 2023-10-30T12:34:54.970 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/136/19' 2023-10-30T12:34:54.970 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/136/15' 2023-10-30T12:34:54.970 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/136/20' 2023-10-30T12:34:54.971 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/136/12' 2023-10-30T12:34:54.971 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/136/10' 2023-10-30T12:34:54.971 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/136/11' 2023-10-30T12:34:54.971 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/136/16' 2023-10-30T12:34:54.972 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/136/14' 2023-10-30T12:34:54.972 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/136/18' 2023-10-30T12:34:54.972 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/136/13' 2023-10-30T12:34:54.972 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/136/17' 2023-10-30T12:34:54.972 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/136' 2023-10-30T12:34:54.973 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/125/19' 2023-10-30T12:34:54.973 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/125/15' 2023-10-30T12:34:54.973 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/125/20' 2023-10-30T12:34:54.973 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/125/12' 2023-10-30T12:34:54.973 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/125/10' 2023-10-30T12:34:54.973 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/125/11' 2023-10-30T12:34:54.974 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/125/16' 2023-10-30T12:34:54.974 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/125/14' 2023-10-30T12:34:54.974 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/125/18' 2023-10-30T12:34:54.974 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/125/13' 2023-10-30T12:34:54.974 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/125/17' 2023-10-30T12:34:54.975 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/125' 2023-10-30T12:34:54.975 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/194/19' 2023-10-30T12:34:54.975 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/194/15' 2023-10-30T12:34:54.975 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/194/20' 2023-10-30T12:34:54.975 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/194/12' 2023-10-30T12:34:54.976 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/194/10' 2023-10-30T12:34:54.976 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/194/11' 2023-10-30T12:34:54.976 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/194/16' 2023-10-30T12:34:54.976 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/194/14' 2023-10-30T12:34:54.976 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/194/18' 2023-10-30T12:34:54.976 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/194/13' 2023-10-30T12:34:54.977 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/194/17' 2023-10-30T12:34:54.977 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/194' 2023-10-30T12:34:54.977 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/119/19' 2023-10-30T12:34:54.977 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/119/15' 2023-10-30T12:34:54.977 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/119/20' 2023-10-30T12:34:54.978 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/119/12' 2023-10-30T12:34:54.978 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/119/10' 2023-10-30T12:34:54.978 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/119/11' 2023-10-30T12:34:54.978 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/119/16' 2023-10-30T12:34:54.978 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/119/14' 2023-10-30T12:34:54.979 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/119/18' 2023-10-30T12:34:54.979 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/119/13' 2023-10-30T12:34:54.979 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/119/17' 2023-10-30T12:34:54.979 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/119' 2023-10-30T12:34:54.980 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/157/19' 2023-10-30T12:34:54.980 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/157/15' 2023-10-30T12:34:54.980 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/157/20' 2023-10-30T12:34:54.980 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/157/12' 2023-10-30T12:34:54.980 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/157/10' 2023-10-30T12:34:54.980 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/157/11' 2023-10-30T12:34:54.981 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/157/16' 2023-10-30T12:34:54.981 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/157/14' 2023-10-30T12:34:54.981 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/157/18' 2023-10-30T12:34:54.981 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/157/13' 2023-10-30T12:34:54.981 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/157/17' 2023-10-30T12:34:54.981 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/157' 2023-10-30T12:34:54.982 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/172/19' 2023-10-30T12:34:54.982 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/172/15' 2023-10-30T12:34:54.982 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/172/20' 2023-10-30T12:34:54.982 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/172/12' 2023-10-30T12:34:54.982 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/172/10' 2023-10-30T12:34:54.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/172/11' 2023-10-30T12:34:54.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/172/16' 2023-10-30T12:34:54.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/172/14' 2023-10-30T12:34:54.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/172/18' 2023-10-30T12:34:54.983 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/172/13' 2023-10-30T12:34:54.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/172/17' 2023-10-30T12:34:54.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/172' 2023-10-30T12:34:54.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/186/19' 2023-10-30T12:34:54.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/186/15' 2023-10-30T12:34:54.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/186/20' 2023-10-30T12:34:54.984 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/186/12' 2023-10-30T12:34:54.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/186/10' 2023-10-30T12:34:54.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/186/11' 2023-10-30T12:34:54.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/186/16' 2023-10-30T12:34:54.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/186/14' 2023-10-30T12:34:54.985 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/186/18' 2023-10-30T12:34:54.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/186/13' 2023-10-30T12:34:54.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/186/17' 2023-10-30T12:34:54.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/186' 2023-10-30T12:34:54.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/190/19' 2023-10-30T12:34:54.986 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/190/15' 2023-10-30T12:34:54.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/190/20' 2023-10-30T12:34:54.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/190/12' 2023-10-30T12:34:54.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/190/10' 2023-10-30T12:34:54.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/190/11' 2023-10-30T12:34:54.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/190/16' 2023-10-30T12:34:54.987 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/190/14' 2023-10-30T12:34:57.564 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/190/18' 2023-10-30T12:34:57.565 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/190/13' 2023-10-30T12:34:57.565 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/190/17' 2023-10-30T12:34:57.565 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/190' 2023-10-30T12:34:57.565 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/104/19' 2023-10-30T12:34:57.565 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/104/15' 2023-10-30T12:34:57.566 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/104/20' 2023-10-30T12:34:57.566 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/104/12' 2023-10-30T12:34:57.566 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/104/10' 2023-10-30T12:34:57.566 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/104/11' 2023-10-30T12:34:57.567 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/104/16' 2023-10-30T12:34:57.567 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/104/14' 2023-10-30T12:34:57.567 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/104/18' 2023-10-30T12:34:57.567 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/104/13' 2023-10-30T12:34:57.567 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/104/17' 2023-10-30T12:34:57.567 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/104' 2023-10-30T12:34:57.568 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/164/19' 2023-10-30T12:34:57.568 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/164/15' 2023-10-30T12:34:57.568 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/164/20' 2023-10-30T12:34:57.568 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/164/12' 2023-10-30T12:34:57.568 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/164/10' 2023-10-30T12:34:57.569 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/164/11' 2023-10-30T12:34:57.569 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/164/16' 2023-10-30T12:34:57.569 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/164/14' 2023-10-30T12:34:57.569 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/164/18' 2023-10-30T12:34:57.569 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/164/13' 2023-10-30T12:34:57.570 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/164/17' 2023-10-30T12:34:57.570 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/164' 2023-10-30T12:34:57.570 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/112/19' 2023-10-30T12:34:57.570 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/112/15' 2023-10-30T12:34:57.570 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/112/20' 2023-10-30T12:34:57.571 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/112/12' 2023-10-30T12:34:57.571 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/112/10' 2023-10-30T12:34:57.571 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/112/11' 2023-10-30T12:34:57.571 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/112/16' 2023-10-30T12:34:57.571 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/112/14' 2023-10-30T12:34:57.571 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/112/18' 2023-10-30T12:34:57.572 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/112/13' 2023-10-30T12:34:57.572 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/112/17' 2023-10-30T12:34:57.572 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/112' 2023-10-30T12:34:57.572 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/100/19' 2023-10-30T12:34:57.572 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/100/15' 2023-10-30T12:34:57.573 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/100/20' 2023-10-30T12:34:57.573 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/100/12' 2023-10-30T12:34:57.573 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/100/10' 2023-10-30T12:34:57.573 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/100/11' 2023-10-30T12:34:57.573 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/100/16' 2023-10-30T12:34:57.573 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/100/14' 2023-10-30T12:34:57.574 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/100/18' 2023-10-30T12:34:57.574 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/100/13' 2023-10-30T12:34:57.574 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/100/17' 2023-10-30T12:34:57.574 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/100' 2023-10-30T12:34:57.574 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/134/19' 2023-10-30T12:34:57.575 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/134/15' 2023-10-30T12:34:57.575 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/134/20' 2023-10-30T12:34:57.575 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/134/12' 2023-10-30T12:34:57.575 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/134/10' 2023-10-30T12:34:57.575 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/134/11' 2023-10-30T12:34:57.575 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/134/16' 2023-10-30T12:34:57.576 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/134/14' 2023-10-30T12:34:57.576 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/134/18' 2023-10-30T12:34:57.576 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/134/13' 2023-10-30T12:34:57.576 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/134/17' 2023-10-30T12:34:57.576 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/134' 2023-10-30T12:34:57.577 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/115/19' 2023-10-30T12:34:57.577 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/115/15' 2023-10-30T12:34:57.577 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/115/20' 2023-10-30T12:34:57.577 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/115/12' 2023-10-30T12:34:57.577 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/115/10' 2023-10-30T12:34:57.578 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/115/11' 2023-10-30T12:34:57.578 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/115/16' 2023-10-30T12:34:57.578 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/115/14' 2023-10-30T12:34:57.578 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/115/18' 2023-10-30T12:34:57.578 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/115/13' 2023-10-30T12:34:57.578 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/115/17' 2023-10-30T12:34:57.579 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/115' 2023-10-30T12:34:57.579 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/116/19' 2023-10-30T12:34:57.579 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/116/15' 2023-10-30T12:34:57.579 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/116/20' 2023-10-30T12:34:57.579 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/116/12' 2023-10-30T12:34:57.580 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/116/10' 2023-10-30T12:34:57.580 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/116/11' 2023-10-30T12:34:57.580 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/116/16' 2023-10-30T12:34:57.580 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/116/14' 2023-10-30T12:34:57.580 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/116/18' 2023-10-30T12:34:57.580 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/116/13' 2023-10-30T12:34:57.581 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/116/17' 2023-10-30T12:34:57.581 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/116' 2023-10-30T12:34:57.581 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/195/19' 2023-10-30T12:34:57.581 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/195/15' 2023-10-30T12:34:57.581 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/195/20' 2023-10-30T12:34:57.582 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/195/12' 2023-10-30T12:34:57.582 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/195/10' 2023-10-30T12:34:57.582 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/195/11' 2023-10-30T12:34:57.582 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/195/16' 2023-10-30T12:34:57.583 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/195/14' 2023-10-30T12:34:57.583 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/195/18' 2023-10-30T12:34:57.583 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/195/13' 2023-10-30T12:34:57.583 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/195/17' 2023-10-30T12:34:57.583 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/195' 2023-10-30T12:34:57.584 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/138/19' 2023-10-30T12:34:57.584 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/138/15' 2023-10-30T12:34:57.584 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/138/20' 2023-10-30T12:34:57.584 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/138/12' 2023-10-30T12:34:57.584 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/138/10' 2023-10-30T12:34:57.584 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/138/11' 2023-10-30T12:34:57.585 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/138/16' 2023-10-30T12:34:57.585 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/138/14' 2023-10-30T12:34:57.585 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/138/18' 2023-10-30T12:34:59.826 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:34:59.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b54fa0640 1 Processor -- start 2023-10-30T12:34:59.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b54fa0640 1 -- start start 2023-10-30T12:34:59.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b54fa0640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1b5014ff10 0x7f1b500c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:59.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b54fa0640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b5006fc90 0x7f1b501566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:59.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b54fa0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1b500c1460 0x7f1b500c18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:59.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b54fa0640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b50156220 con 0x7f1b500c1460 2023-10-30T12:34:59.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b54fa0640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b50155bf0 con 0x7f1b5014ff10 2023-10-30T12:34:59.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b54fa0640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b50070110 con 0x7f1b5006fc90 2023-10-30T12:34:59.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b4ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1b500c1460 0x7f1b500c18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:59.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b4ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1b500c1460 0x7f1b500c18b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42842/0 (socket says 172.21.15.125:42842) 2023-10-30T12:34:59.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b4ffff640 1 -- 172.21.15.125:0/334234229 learned_addr learned my addr 172.21.15.125:0/334234229 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:34:59.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b4f7fe640 1 --2- 172.21.15.125:0/334234229 >> v2:172.21.15.136:3300/0 conn(0x7f1b5014ff10 0x7f1b500c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:59.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b47fff640 1 --2- 172.21.15.125:0/334234229 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b5006fc90 0x7f1b501566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:59.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b4ffff640 1 -- 172.21.15.125:0/334234229 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b5006fc90 msgr2=0x7f1b501566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:59.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b4ffff640 1 --2- 172.21.15.125:0/334234229 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b5006fc90 0x7f1b501566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:59.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b4ffff640 1 -- 172.21.15.125:0/334234229 >> v2:172.21.15.136:3300/0 conn(0x7f1b5014ff10 msgr2=0x7f1b500c54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:34:59.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b4ffff640 1 --2- 172.21.15.125:0/334234229 >> v2:172.21.15.136:3300/0 conn(0x7f1b5014ff10 0x7f1b500c54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:59.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b4ffff640 1 -- 172.21.15.125:0/334234229 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b500703f0 con 0x7f1b500c1460 2023-10-30T12:34:59.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.970+0000 7f1b4f7fe640 1 --2- 172.21.15.125:0/334234229 >> v2:172.21.15.136:3300/0 conn(0x7f1b5014ff10 0x7f1b500c54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:34:59.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.974+0000 7f1b4ffff640 1 --2- 172.21.15.125:0/334234229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b500c1460 0x7f1b500c18b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1b500668b0 tx=0x7f1b4801ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8602932c0e1063b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:59.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.974+0000 7f1b4effd640 1 -- 172.21.15.125:0/334234229 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b4800a2d0 con 0x7f1b500c1460 2023-10-30T12:34:59.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.974+0000 7f1b4effd640 1 -- 172.21.15.125:0/334234229 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b480a7960 con 0x7f1b500c1460 2023-10-30T12:34:59.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.974+0000 7f1b4effd640 1 -- 172.21.15.125:0/334234229 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b480a7cf0 con 0x7f1b500c1460 2023-10-30T12:34:59.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.974+0000 7f1b54fa0640 1 -- 172.21.15.125:0/334234229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b500c1460 msgr2=0x7f1b500c18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:59.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.974+0000 7f1b54fa0640 1 --2- 172.21.15.125:0/334234229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b500c1460 0x7f1b500c18b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1b500668b0 tx=0x7f1b4801ebe0 comp rx=0 tx=0).stop 2023-10-30T12:34:59.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.974+0000 7f1b54fa0640 1 -- 172.21.15.125:0/334234229 shutdown_connections 2023-10-30T12:34:59.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.974+0000 7f1b54fa0640 1 --2- 172.21.15.125:0/334234229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b500c1460 0x7f1b500c18b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:59.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.974+0000 7f1b54fa0640 1 --2- 172.21.15.125:0/334234229 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b5006fc90 0x7f1b501566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:59.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.974+0000 7f1b54fa0640 1 --2- 172.21.15.125:0/334234229 >> v2:172.21.15.136:3300/0 conn(0x7f1b5014ff10 0x7f1b500c54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:59.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.974+0000 7f1b54fa0640 1 -- 172.21.15.125:0/334234229 >> 172.21.15.125:0/334234229 conn(0x7f1b500c75d0 msgr2=0x7f1b500c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:34:59.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.974+0000 7f1b54fa0640 1 -- 172.21.15.125:0/334234229 shutdown_connections 2023-10-30T12:34:59.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.974+0000 7f1b54fa0640 1 -- 172.21.15.125:0/334234229 wait complete. 2023-10-30T12:34:59.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b54fa0640 1 Processor -- start 2023-10-30T12:34:59.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b54fa0640 1 -- start start 2023-10-30T12:34:59.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b54fa0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b5006fc90 0x7f1b501463f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:59.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b54fa0640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b500c1460 0x7f1b501421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:59.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b54fa0640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b5014ff10 0x7f1b50142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:59.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b54fa0640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5013e070 con 0x7f1b500c1460 2023-10-30T12:34:59.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b54fa0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b50143340 con 0x7f1b5006fc90 2023-10-30T12:34:59.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b54fa0640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b50145930 con 0x7f1b5014ff10 2023-10-30T12:34:59.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b47fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b500c1460 0x7f1b501421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:59.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b47fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b500c1460 0x7f1b501421d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42856/0 (socket says 172.21.15.125:42856) 2023-10-30T12:34:59.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b47fff640 1 -- 172.21.15.125:0/1677062057 learned_addr learned my addr 172.21.15.125:0/1677062057 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:34:59.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b4f7fe640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b5006fc90 0x7f1b501463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:59.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b4ffff640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b5014ff10 0x7f1b50142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:59.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b47fff640 1 -- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b5014ff10 msgr2=0x7f1b50142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:59.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b47fff640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b5014ff10 0x7f1b50142710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:59.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b47fff640 1 -- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b5006fc90 msgr2=0x7f1b501463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:34:59.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b47fff640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b5006fc90 0x7f1b501463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:34:59.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b47fff640 1 -- 172.21.15.125:0/1677062057 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4800ac40 con 0x7f1b500c1460 2023-10-30T12:34:59.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b4f7fe640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b5006fc90 0x7f1b501463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:34:59.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b47fff640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b500c1460 0x7f1b501421d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1b3800d800 tx=0x7f1b3800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:59.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b4cff9640 1 -- 172.21.15.125:0/1677062057 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b380096a0 con 0x7f1b500c1460 2023-10-30T12:34:59.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b4cff9640 1 -- 172.21.15.125:0/1677062057 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b38009800 con 0x7f1b500c1460 2023-10-30T12:34:59.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b4cff9640 1 -- 172.21.15.125:0/1677062057 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b38009990 con 0x7f1b500c1460 2023-10-30T12:34:59.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b54fa0640 1 -- 172.21.15.125:0/1677062057 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5014c130 con 0x7f1b500c1460 2023-10-30T12:34:59.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b54fa0640 1 -- 172.21.15.125:0/1677062057 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b501438c0 con 0x7f1b500c1460 2023-10-30T12:34:59.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.978+0000 7f1b54fa0640 1 -- 172.21.15.125:0/1677062057 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b50146930 con 0x7f1b500c1460 2023-10-30T12:34:59.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.982+0000 7f1b4cff9640 1 -- 172.21.15.125:0/1677062057 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1b38009af0 con 0x7f1b500c1460 2023-10-30T12:34:59.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.982+0000 7f1b4cff9640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1b300664a0 0x7f1b30068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:34:59.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.982+0000 7f1b4cff9640 1 -- 172.21.15.125:0/1677062057 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6626+0+0 (secure 0 0 0) 0x7f1b38084680 con 0x7f1b500c1460 2023-10-30T12:34:59.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.982+0000 7f1b4f7fe640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1b300664a0 0x7f1b30068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:34:59.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.982+0000 7f1b4f7fe640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1b300664a0 0x7f1b30068960 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1b40006fd0 tx=0x7f1b40008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:34:59.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:34:59.986+0000 7f1b4cff9640 1 -- 172.21.15.125:0/1677062057 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1b38051760 con 0x7f1b500c1460 2023-10-30T12:35:00.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.174+0000 7f1b54fa0640 1 -- 172.21.15.125:0/1677062057 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b5006c6a0 con 0x7f1b500c1460 2023-10-30T12:35:00.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.174+0000 7f1b4cff9640 1 -- 172.21.15.125:0/1677062057 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6547 (secure 0 0 0) 0x7f1b3800b3a0 con 0x7f1b500c1460 2023-10-30T12:35:00.179 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:00.179 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:34:43.270738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:00.181 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 35 2023-10-30T12:35:00.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.178+0000 7f1b54fa0640 1 -- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1b300664a0 msgr2=0x7f1b30068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.178+0000 7f1b54fa0640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1b300664a0 0x7f1b30068960 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1b40006fd0 tx=0x7f1b40008040 comp rx=0 tx=0).stop 2023-10-30T12:35:00.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.178+0000 7f1b54fa0640 1 -- 172.21.15.125:0/1677062057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b500c1460 msgr2=0x7f1b501421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.178+0000 7f1b54fa0640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b500c1460 0x7f1b501421d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1b3800d800 tx=0x7f1b3800dcc0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.178+0000 7f1b54fa0640 1 -- 172.21.15.125:0/1677062057 shutdown_connections 2023-10-30T12:35:00.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.178+0000 7f1b54fa0640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1b300664a0 0x7f1b30068960 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.178+0000 7f1b54fa0640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1b5014ff10 0x7f1b50142710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.178+0000 7f1b54fa0640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1b500c1460 0x7f1b501421d0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.178+0000 7f1b54fa0640 1 --2- 172.21.15.125:0/1677062057 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1b5006fc90 0x7f1b501463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.178+0000 7f1b54fa0640 1 -- 172.21.15.125:0/1677062057 >> 172.21.15.125:0/1677062057 conn(0x7f1b500c75d0 msgr2=0x7f1b500c8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:00.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.178+0000 7f1b54fa0640 1 -- 172.21.15.125:0/1677062057 shutdown_connections 2023-10-30T12:35:00.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.178+0000 7f1b54fa0640 1 -- 172.21.15.125:0/1677062057 wait complete. 2023-10-30T12:35:00.200 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=1) 2023-10-30T12:35:00.200 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T12:35:00.201 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:35:00.225 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T12:35:00.225 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:00.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.358+0000 7efce2f72640 1 Processor -- start 2023-10-30T12:35:00.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.358+0000 7efce2f72640 1 -- start start 2023-10-30T12:35:00.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.362+0000 7efce2f72640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efcdc06fd50 0x7efcdc15ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:00.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.362+0000 7efce2f72640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcdc157340 con 0x7efcdc156f60 2023-10-30T12:35:00.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.362+0000 7efce2f72640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcdc06ca30 con 0x7efcdc06afa0 2023-10-30T12:35:00.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.362+0000 7efce2f72640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcdc06bea0 con 0x7efcdc06fd50 2023-10-30T12:35:00.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.362+0000 7efcd3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efcdc06fd50 0x7efcdc15ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:00.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.362+0000 7efcd3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efcdc06fd50 0x7efcdc15ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42862/0 (socket says 172.21.15.125:42862) 2023-10-30T12:35:00.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.362+0000 7efce0d0e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7efcdc156f60 0x7efcdc071b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38020/0 (socket says 172.21.15.125:38020) 2023-10-30T12:35:00.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.362+0000 7efcd3fff640 1 -- 172.21.15.125:0/525050501 learned_addr learned my addr 172.21.15.125:0/525050501 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:00.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.362+0000 7efcd3fff640 1 -- 172.21.15.125:0/525050501 >> v1:172.21.15.136:6789/0 conn(0x7efcdc06afa0 legacy=0x7efcdc164160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.362+0000 7efcd3fff640 1 -- 172.21.15.125:0/525050501 >> v1:172.21.15.125:6789/0 conn(0x7efcdc156f60 legacy=0x7efcdc071b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.362+0000 7efcd3fff640 1 -- 172.21.15.125:0/525050501 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc06cf80 con 0x7efcdc06fd50 2023-10-30T12:35:00.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efcd3fff640 1 --2- 172.21.15.125:0/525050501 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcdc06fd50 0x7efcdc15ce10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7efccc009870 tx=0x7efccc01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bfb1ac3b5497e04 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:00.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efcd37fe640 1 -- 172.21.15.125:0/525050501 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc0ab600 con 0x7efcdc06fd50 2023-10-30T12:35:00.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efcd37fe640 1 -- 172.21.15.125:0/525050501 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efccc0a7050 con 0x7efcdc06fd50 2023-10-30T12:35:00.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efcd37fe640 1 -- 172.21.15.125:0/525050501 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efccc0ab980 con 0x7efcdc06fd50 2023-10-30T12:35:00.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 -- 172.21.15.125:0/525050501 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcdc06fd50 msgr2=0x7efcdc15ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 --2- 172.21.15.125:0/525050501 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcdc06fd50 0x7efcdc15ce10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7efccc009870 tx=0x7efccc01c540 comp rx=0 tx=0).stop 2023-10-30T12:35:00.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 -- 172.21.15.125:0/525050501 shutdown_connections 2023-10-30T12:35:00.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 --2- 172.21.15.125:0/525050501 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcdc06fd50 0x7efcdc15ce10 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 -- 172.21.15.125:0/525050501 >> 172.21.15.125:0/525050501 conn(0x7efcdc069e90 msgr2=0x7efcdc06a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:00.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 -- 172.21.15.125:0/525050501 shutdown_connections 2023-10-30T12:35:00.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 -- 172.21.15.125:0/525050501 wait complete. 2023-10-30T12:35:00.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 Processor -- start 2023-10-30T12:35:00.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 -- start start 2023-10-30T12:35:00.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcdc06afa0 0x7efcdc1f3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:00.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efcdc06fd50 0x7efcdc15b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:00.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcdc156f60 0x7efcdc153190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:00.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc1f31c0 con 0x7efcdc06afa0 2023-10-30T12:35:00.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc1f6b60 con 0x7efcdc156f60 2023-10-30T12:35:00.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.366+0000 7efce2f72640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc1f2b70 con 0x7efcdc06fd50 2023-10-30T12:35:00.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efce0d0e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcdc06afa0 0x7efcdc1f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:00.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efce0d0e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcdc06afa0 0x7efcdc1f3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42872/0 (socket says 172.21.15.125:42872) 2023-10-30T12:35:00.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efce0d0e640 1 -- 172.21.15.125:0/3002813738 learned_addr learned my addr 172.21.15.125:0/3002813738 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:00.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efcd3fff640 1 --2- 172.21.15.125:0/3002813738 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efcdc06fd50 0x7efcdc15b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:00.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efce150f640 1 --2- 172.21.15.125:0/3002813738 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcdc156f60 0x7efcdc153190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:00.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efce0d0e640 1 -- 172.21.15.125:0/3002813738 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efcdc06fd50 msgr2=0x7efcdc15b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efce0d0e640 1 --2- 172.21.15.125:0/3002813738 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efcdc06fd50 0x7efcdc15b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efce0d0e640 1 -- 172.21.15.125:0/3002813738 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcdc156f60 msgr2=0x7efcdc153190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efce0d0e640 1 --2- 172.21.15.125:0/3002813738 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcdc156f60 0x7efcdc153190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efce0d0e640 1 -- 172.21.15.125:0/3002813738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efccc0094d0 con 0x7efcdc06afa0 2023-10-30T12:35:00.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efce0d0e640 1 --2- 172.21.15.125:0/3002813738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcdc06afa0 0x7efcdc1f3650 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7efcd800c490 tx=0x7efcd800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:00.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efcd17fa640 1 -- 172.21.15.125:0/3002813738 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd8007480 con 0x7efcdc06afa0 2023-10-30T12:35:00.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efcd17fa640 1 -- 172.21.15.125:0/3002813738 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd80075e0 con 0x7efcdc06afa0 2023-10-30T12:35:00.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efcd17fa640 1 -- 172.21.15.125:0/3002813738 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcd8004a10 con 0x7efcdc06afa0 2023-10-30T12:35:00.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efce2f72640 1 -- 172.21.15.125:0/3002813738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcdc1f2e50 con 0x7efcdc06afa0 2023-10-30T12:35:00.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efce2f72640 1 -- 172.21.15.125:0/3002813738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcdc1f6e20 con 0x7efcdc06afa0 2023-10-30T12:35:00.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efce2f72640 1 -- 172.21.15.125:0/3002813738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcdc04e170 con 0x7efcdc06afa0 2023-10-30T12:35:00.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efcd17fa640 1 -- 172.21.15.125:0/3002813738 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efcd8004b70 con 0x7efcdc06afa0 2023-10-30T12:35:00.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.370+0000 7efcd17fa640 1 --2- 172.21.15.125:0/3002813738 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efcc80664a0 0x7efcc8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:00.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.374+0000 7efcd17fa640 1 -- 172.21.15.125:0/3002813738 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6626+0+0 (secure 0 0 0) 0x7efcd8083550 con 0x7efcdc06afa0 2023-10-30T12:35:00.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.374+0000 7efcd3fff640 1 --2- 172.21.15.125:0/3002813738 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efcc80664a0 0x7efcc8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:00.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.374+0000 7efcd3fff640 1 --2- 172.21.15.125:0/3002813738 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efcc80664a0 0x7efcc8068960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0ce070 tx=0x7efccc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:00.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.378+0000 7efcd17fa640 1 -- 172.21.15.125:0/3002813738 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efcd8050500 con 0x7efcdc06afa0 2023-10-30T12:35:00.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.554+0000 7efce2f72640 1 -- 172.21.15.125:0/3002813738 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcdc0c8590 con 0x7efcdc06afa0 2023-10-30T12:35:00.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.554+0000 7efcd17fa640 1 -- 172.21.15.125:0/3002813738 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+6547 (secure 0 0 0) 0x7efcd8018090 con 0x7efcdc06afa0 2023-10-30T12:35:00.559 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:00.559 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:34:43.270738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:00.560 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 35 2023-10-30T12:35:00.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.558+0000 7efce2f72640 1 -- 172.21.15.125:0/3002813738 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efcc80664a0 msgr2=0x7efcc8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.558+0000 7efce2f72640 1 --2- 172.21.15.125:0/3002813738 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efcc80664a0 0x7efcc8068960 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0ce070 tx=0x7efccc008040 comp rx=0 tx=0).stop 2023-10-30T12:35:00.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.558+0000 7efce2f72640 1 -- 172.21.15.125:0/3002813738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcdc06afa0 msgr2=0x7efcdc1f3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:00.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.558+0000 7efce2f72640 1 --2- 172.21.15.125:0/3002813738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcdc06afa0 0x7efcdc1f3650 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7efcd800c490 tx=0x7efcd800c950 comp rx=0 tx=0).stop 2023-10-30T12:35:00.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.558+0000 7efce2f72640 1 -- 172.21.15.125:0/3002813738 shutdown_connections 2023-10-30T12:35:00.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.558+0000 7efce2f72640 1 --2- 172.21.15.125:0/3002813738 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efcc80664a0 0x7efcc8068960 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.558+0000 7efce2f72640 1 --2- 172.21.15.125:0/3002813738 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efcdc156f60 0x7efcdc153190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.558+0000 7efce2f72640 1 --2- 172.21.15.125:0/3002813738 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efcdc06fd50 0x7efcdc15b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.558+0000 7efce2f72640 1 --2- 172.21.15.125:0/3002813738 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efcdc06afa0 0x7efcdc1f3650 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.558+0000 7efce2f72640 1 -- 172.21.15.125:0/3002813738 >> 172.21.15.125:0/3002813738 conn(0x7efcdc069e90 msgr2=0x7efcdc0c7890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.558+0000 7efce2f72640 1 -- 172.21.15.125:0/3002813738 shutdown_connections 2023-10-30T12:35:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:00.558+0000 7efce2f72640 1 -- 172.21.15.125:0/3002813738 wait complete. 2023-10-30T12:35:00.579 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:35:02.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:02.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f205d3640 1 Processor -- start 2023-10-30T12:35:02.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f205d3640 1 -- start start 2023-10-30T12:35:02.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f205d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4f18153d70 0x7f4f1806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f205d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f180c8490 0x7f4f180c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f205d3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f1806afa0 con 0x7f4f1806e470 2023-10-30T12:35:02.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f205d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f1806d780 con 0x7f4f18153d70 2023-10-30T12:35:02.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f205d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f18154150 con 0x7f4f180c8490 2023-10-30T12:35:02.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1db6e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4f1806e470 0x7f4f180cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:56652/0 (socket says 172.21.15.125:56652) 2023-10-30T12:35:02.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1db6e640 1 -- 172.21.15.125:0/3705860054 learned_addr learned my addr 172.21.15.125:0/3705860054 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:35:02.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1eb70640 1 --2- 172.21.15.125:0/3705860054 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f180c8490 0x7f4f180c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1e36f640 1 --2- 172.21.15.125:0/3705860054 >> v2:172.21.15.136:3300/0 conn(0x7f4f18153d70 0x7f4f1806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 -- 172.21.15.125:0/3705860054 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422745255 0 0) 0x7f4f1806afa0 con 0x7f4f1806e470 2023-10-30T12:35:02.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 -- 172.21.15.125:0/3705860054 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f04003530 con 0x7f4f1806e470 2023-10-30T12:35:02.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 -- 172.21.15.125:0/3705860054 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2178643526 0 0) 0x7f4f04003530 con 0x7f4f1806e470 2023-10-30T12:35:02.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 -- 172.21.15.125:0/3705860054 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f180c8490 msgr2=0x7f4f180c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:02.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 --2- 172.21.15.125:0/3705860054 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f180c8490 0x7f4f180c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 -- 172.21.15.125:0/3705860054 >> v2:172.21.15.136:3300/0 conn(0x7f4f18153d70 msgr2=0x7f4f1806ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:02.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 --2- 172.21.15.125:0/3705860054 >> v2:172.21.15.136:3300/0 conn(0x7f4f18153d70 0x7f4f1806ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 -- 172.21.15.125:0/3705860054 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f1806e030 con 0x7f4f1806e470 2023-10-30T12:35:02.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 -- 172.21.15.125:0/3705860054 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f4f08001680 con 0x7f4f1806e470 2023-10-30T12:35:02.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 -- 172.21.15.125:0/3705860054 >> v1:172.21.15.125:6789/0 conn(0x7f4f1806e470 legacy=0x7f4f180cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:02.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 --2- 172.21.15.125:0/3705860054 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f04003980 0x7f4f04003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 --2- 172.21.15.125:0/3705860054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f04004370 0x7f4f04004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 --2- 172.21.15.125:0/3705860054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f04004ec0 0x7f4f0400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 -- 172.21.15.125:0/3705860054 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1806d780 con 0x7f4f04004ec0 2023-10-30T12:35:02.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 -- 172.21.15.125:0/3705860054 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f18154150 con 0x7f4f04004370 2023-10-30T12:35:02.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1d36d640 1 -- 172.21.15.125:0/3705860054 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0400bed0 con 0x7f4f04003980 2023-10-30T12:35:02.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1eb70640 1 --2- 172.21.15.125:0/3705860054 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f180c8490 0x7f4f180c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:35:02.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1eb70640 1 --2- 172.21.15.125:0/3705860054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f04004ec0 0x7f4f0400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1e36f640 1 --2- 172.21.15.125:0/3705860054 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f04003980 0x7f4f04003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.726+0000 7f4f1db6e640 1 --2- 172.21.15.125:0/3705860054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f04004370 0x7f4f04004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f1eb70640 1 -- 172.21.15.125:0/3705860054 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f04003980 msgr2=0x7f4f04003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:02.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f1eb70640 1 --2- 172.21.15.125:0/3705860054 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f04003980 0x7f4f04003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f1eb70640 1 -- 172.21.15.125:0/3705860054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f04004370 msgr2=0x7f4f04004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:02.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f1eb70640 1 --2- 172.21.15.125:0/3705860054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f04004370 0x7f4f04004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f1eb70640 1 -- 172.21.15.125:0/3705860054 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f04003530 con 0x7f4f04004ec0 2023-10-30T12:35:02.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f1db6e640 1 --2- 172.21.15.125:0/3705860054 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f04004370 0x7f4f04004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:02.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f1eb70640 1 --2- 172.21.15.125:0/3705860054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f04004ec0 0x7f4f0400b990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4f180c8870 tx=0x7f4f1001cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d3e1da1bd3253d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:02.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f1d36d640 1 -- 172.21.15.125:0/3705860054 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f100a5d50 con 0x7f4f04004ec0 2023-10-30T12:35:02.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f1d36d640 1 -- 172.21.15.125:0/3705860054 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1000f040 con 0x7f4f04004ec0 2023-10-30T12:35:02.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f1d36d640 1 -- 172.21.15.125:0/3705860054 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f10003bc0 con 0x7f4f04004ec0 2023-10-30T12:35:02.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f205d3640 1 -- 172.21.15.125:0/3705860054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f04004ec0 msgr2=0x7f4f0400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:02.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f205d3640 1 --2- 172.21.15.125:0/3705860054 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f04004ec0 0x7f4f0400b990 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4f180c8870 tx=0x7f4f1001cd00 comp rx=0 tx=0).stop 2023-10-30T12:35:02.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f1eb70640 1 -- 172.21.15.125:0/3705860054 reap_dead start 2023-10-30T12:35:02.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f205d3640 1 -- 172.21.15.125:0/3705860054 shutdown_connections 2023-10-30T12:35:02.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f205d3640 1 -- 172.21.15.125:0/3705860054 >> 172.21.15.125:0/3705860054 conn(0x7f4f18071ea0 msgr2=0x7f4f18065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:02.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f205d3640 1 -- 172.21.15.125:0/3705860054 shutdown_connections 2023-10-30T12:35:02.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.730+0000 7f4f205d3640 1 -- 172.21.15.125:0/3705860054 wait complete. 2023-10-30T12:35:02.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f205d3640 1 Processor -- start 2023-10-30T12:35:02.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f205d3640 1 -- start start 2023-10-30T12:35:02.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f205d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f04004ec0 0x7f4f181e4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f205d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f181e5f90 0x7f4f1814ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f205d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f181f2840 0x7f4f181f2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f205d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f1814da40 con 0x7f4f181f2840 2023-10-30T12:35:02.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f205d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f181e40a0 con 0x7f4f181e5f90 2023-10-30T12:35:02.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f205d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f181f31f0 con 0x7f4f04004ec0 2023-10-30T12:35:02.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f1eb70640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f181f2840 0x7f4f181f2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f1eb70640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f181f2840 0x7f4f181f2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46496/0 (socket says 172.21.15.125:46496) 2023-10-30T12:35:02.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f1eb70640 1 -- 172.21.15.125:0/3451844906 learned_addr learned my addr 172.21.15.125:0/3451844906 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:02.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f1e36f640 1 --2- 172.21.15.125:0/3451844906 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f04004ec0 0x7f4f181e4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f1db6e640 1 --2- 172.21.15.125:0/3451844906 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f181e5f90 0x7f4f1814ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f1eb70640 1 -- 172.21.15.125:0/3451844906 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f04004ec0 msgr2=0x7f4f181e4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:02.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f1eb70640 1 --2- 172.21.15.125:0/3451844906 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f04004ec0 0x7f4f181e4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f1eb70640 1 -- 172.21.15.125:0/3451844906 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f181e5f90 msgr2=0x7f4f1814ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:02.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f1eb70640 1 --2- 172.21.15.125:0/3451844906 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f181e5f90 0x7f4f1814ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f1eb70640 1 -- 172.21.15.125:0/3451844906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f181e5130 con 0x7f4f181f2840 2023-10-30T12:35:02.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f1eb70640 1 --2- 172.21.15.125:0/3451844906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f181f2840 0x7f4f181f2cb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4f1001c770 tx=0x7f4f10009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:02.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f0effd640 1 -- 172.21.15.125:0/3451844906 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f100a5d50 con 0x7f4f181f2840 2023-10-30T12:35:02.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f0effd640 1 -- 172.21.15.125:0/3451844906 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f10003e60 con 0x7f4f181f2840 2023-10-30T12:35:02.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f205d3640 1 -- 172.21.15.125:0/3451844906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f181e3420 con 0x7f4f181f2840 2023-10-30T12:35:02.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f0effd640 1 -- 172.21.15.125:0/3451844906 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f10004230 con 0x7f4f181f2840 2023-10-30T12:35:02.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.734+0000 7f4f205d3640 1 -- 172.21.15.125:0/3451844906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f181e39e0 con 0x7f4f181f2840 2023-10-30T12:35:02.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.738+0000 7f4f205d3640 1 -- 172.21.15.125:0/3451844906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f181f2560 con 0x7f4f181f2840 2023-10-30T12:35:02.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.738+0000 7f4f0effd640 1 -- 172.21.15.125:0/3451844906 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4f10004390 con 0x7f4f181f2840 2023-10-30T12:35:02.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.738+0000 7f4f0effd640 1 --2- 172.21.15.125:0/3451844906 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4efc0664d0 0x7f4efc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:02.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.738+0000 7f4f0effd640 1 -- 172.21.15.125:0/3451844906 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6626+0+0 (secure 0 0 0) 0x7f4f1011f5d0 con 0x7f4f181f2840 2023-10-30T12:35:02.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.738+0000 7f4f1e36f640 1 --2- 172.21.15.125:0/3451844906 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4efc0664d0 0x7f4efc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:02.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.738+0000 7f4f1e36f640 1 --2- 172.21.15.125:0/3451844906 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4efc0664d0 0x7f4efc068990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4f14009be0 tx=0x7f4f14009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:02.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.742+0000 7f4f0effd640 1 -- 172.21.15.125:0/3451844906 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4f181f2560 con 0x7f4f181f2840 2023-10-30T12:35:02.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.922+0000 7f4f205d3640 1 -- 172.21.15.125:0/3451844906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f181f2560 con 0x7f4f181f2840 2023-10-30T12:35:02.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.922+0000 7f4f0effd640 1 -- 172.21.15.125:0/3451844906 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f4f181f2560 con 0x7f4f181f2840 2023-10-30T12:35:02.926 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:02.927 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:00.120705+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:02.927 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 36 2023-10-30T12:35:02.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.926+0000 7f4f205d3640 1 -- 172.21.15.125:0/3451844906 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4efc0664d0 msgr2=0x7f4efc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:02.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.926+0000 7f4f205d3640 1 --2- 172.21.15.125:0/3451844906 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4efc0664d0 0x7f4efc068990 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4f14009be0 tx=0x7f4f14009190 comp rx=0 tx=0).stop 2023-10-30T12:35:02.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.926+0000 7f4f205d3640 1 -- 172.21.15.125:0/3451844906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f181f2840 msgr2=0x7f4f181f2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:02.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.926+0000 7f4f205d3640 1 --2- 172.21.15.125:0/3451844906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f181f2840 0x7f4f181f2cb0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4f1001c770 tx=0x7f4f10009750 comp rx=0 tx=0).stop 2023-10-30T12:35:02.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.926+0000 7f4f205d3640 1 -- 172.21.15.125:0/3451844906 shutdown_connections 2023-10-30T12:35:02.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.926+0000 7f4f205d3640 1 --2- 172.21.15.125:0/3451844906 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4efc0664d0 0x7f4efc068990 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.926+0000 7f4f205d3640 1 --2- 172.21.15.125:0/3451844906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f181f2840 0x7f4f181f2cb0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.926+0000 7f4f205d3640 1 --2- 172.21.15.125:0/3451844906 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f181e5f90 0x7f4f1814ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.926+0000 7f4f205d3640 1 --2- 172.21.15.125:0/3451844906 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f04004ec0 0x7f4f181e4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:02.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.926+0000 7f4f205d3640 1 -- 172.21.15.125:0/3451844906 >> 172.21.15.125:0/3451844906 conn(0x7f4f18071ea0 msgr2=0x7f4f181529e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:02.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.926+0000 7f4f205d3640 1 -- 172.21.15.125:0/3451844906 shutdown_connections 2023-10-30T12:35:02.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:02.926+0000 7f4f205d3640 1 -- 172.21.15.125:0/3451844906 wait complete. 2023-10-30T12:35:02.947 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:35:04.948 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:05.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.090+0000 7f4fd8b94640 1 Processor -- start 2023-10-30T12:35:05.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.090+0000 7f4fd8b94640 1 -- start start 2023-10-30T12:35:05.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.094+0000 7f4fd8b94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4fd40c6b80 0x7f4fd40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:05.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.094+0000 7f4fd8b94640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fd40c0a50 con 0x7f4fd40c6f60 2023-10-30T12:35:05.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.094+0000 7f4fd8b94640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fd406f6e0 con 0x7f4fd413f4a0 2023-10-30T12:35:05.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.094+0000 7f4fd8b94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd406f9c0 con 0x7f4fd40c6b80 2023-10-30T12:35:05.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.094+0000 7f4fc9d9b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4fd40c6f60 0x7f4fd413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:56666/0 (socket says 172.21.15.125:56666) 2023-10-30T12:35:05.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.094+0000 7f4fc9d9b640 1 -- 172.21.15.125:0/411638316 learned_addr learned my addr 172.21.15.125:0/411638316 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:35:05.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.094+0000 7f4fd259c640 1 --2- 172.21.15.125:0/411638316 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4fd40c6b80 0x7f4fd40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:05.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.094+0000 7f4fd1d9b640 1 -- 172.21.15.125:0/411638316 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043212865 0 0) 0x7f4fd40c0a50 con 0x7f4fd40c6f60 2023-10-30T12:35:05.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.094+0000 7f4fd1d9b640 1 -- 172.21.15.125:0/411638316 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fc0003530 con 0x7f4fd40c6f60 2023-10-30T12:35:05.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.094+0000 7f4fd1d9b640 1 -- 172.21.15.125:0/411638316 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683326855 0 0) 0x7f4fd406f6e0 con 0x7f4fd413f4a0 2023-10-30T12:35:05.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.094+0000 7f4fd1d9b640 1 -- 172.21.15.125:0/411638316 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd40c0a50 con 0x7f4fd413f4a0 2023-10-30T12:35:05.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.094+0000 7f4fd259c640 1 -- 172.21.15.125:0/411638316 >> v1:172.21.15.136:6789/0 conn(0x7f4fd413f4a0 legacy=0x7f4fd41566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:05.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.094+0000 7f4fd259c640 1 -- 172.21.15.125:0/411638316 >> v1:172.21.15.125:6789/0 conn(0x7f4fd40c6f60 legacy=0x7f4fd413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:05.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.094+0000 7f4fd259c640 1 -- 172.21.15.125:0/411638316 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fd406fca0 con 0x7f4fd40c6b80 2023-10-30T12:35:05.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.098+0000 7f4fd259c640 1 --2- 172.21.15.125:0/411638316 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4fd40c6b80 0x7f4fd40bfec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4009dc0 tx=0x7f4fc401c970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ceb365e98d061b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:05.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.098+0000 7f4fd1d9b640 1 -- 172.21.15.125:0/411638316 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc40a5840 con 0x7f4fd40c6b80 2023-10-30T12:35:05.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.098+0000 7f4fd1d9b640 1 -- 172.21.15.125:0/411638316 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fc40a59a0 con 0x7f4fd40c6b80 2023-10-30T12:35:05.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.098+0000 7f4fd1d9b640 1 -- 172.21.15.125:0/411638316 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fc40a5d30 con 0x7f4fd40c6b80 2023-10-30T12:35:05.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.098+0000 7f4fd8b94640 1 -- 172.21.15.125:0/411638316 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4fd40c6b80 msgr2=0x7f4fd40bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:05.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.098+0000 7f4fd8b94640 1 --2- 172.21.15.125:0/411638316 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4fd40c6b80 0x7f4fd40bfec0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4fc4009dc0 tx=0x7f4fc401c970 comp rx=0 tx=0).stop 2023-10-30T12:35:05.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.098+0000 7f4fd8b94640 1 -- 172.21.15.125:0/411638316 shutdown_connections 2023-10-30T12:35:05.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.098+0000 7f4fd8b94640 1 --2- 172.21.15.125:0/411638316 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4fd40c6b80 0x7f4fd40bfec0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:05.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.098+0000 7f4fd8b94640 1 -- 172.21.15.125:0/411638316 >> 172.21.15.125:0/411638316 conn(0x7f4fd40c7910 msgr2=0x7f4fd40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:05.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.098+0000 7f4fd8b94640 1 -- 172.21.15.125:0/411638316 shutdown_connections 2023-10-30T12:35:05.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.098+0000 7f4fd8b94640 1 -- 172.21.15.125:0/411638316 wait complete. 2023-10-30T12:35:05.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.098+0000 7f4fd8b94640 1 Processor -- start 2023-10-30T12:35:05.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.098+0000 7f4fd8b94640 1 -- start start 2023-10-30T12:35:05.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fd8b94640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fd40c6b80 0x7f4fd4157d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:05.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fd8b94640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fd40c6f60 0x7f4fd40c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:05.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fd8b94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4fd413f4a0 0x7f4fd413bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:05.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fd8b94640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd41f5750 con 0x7f4fd40c6f60 2023-10-30T12:35:05.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fd8b94640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd41f5a30 con 0x7f4fd40c6b80 2023-10-30T12:35:05.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fd8b94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fd40ca8b0 con 0x7f4fd413f4a0 2023-10-30T12:35:05.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fc9d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fd40c6f60 0x7f4fd40c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:05.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fc9d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fd40c6f60 0x7f4fd40c9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46506/0 (socket says 172.21.15.125:46506) 2023-10-30T12:35:05.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fc9d9b640 1 -- 172.21.15.125:0/1836890885 learned_addr learned my addr 172.21.15.125:0/1836890885 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:05.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fd259c640 1 --2- 172.21.15.125:0/1836890885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fd40c6b80 0x7f4fd4157d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:05.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fd2d9d640 1 --2- 172.21.15.125:0/1836890885 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4fd413f4a0 0x7f4fd413bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:05.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fc9d9b640 1 -- 172.21.15.125:0/1836890885 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4fd413f4a0 msgr2=0x7f4fd413bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:05.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fc9d9b640 1 --2- 172.21.15.125:0/1836890885 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4fd413f4a0 0x7f4fd413bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:05.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fc9d9b640 1 -- 172.21.15.125:0/1836890885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fd40c6b80 msgr2=0x7f4fd4157d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:05.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fc9d9b640 1 --2- 172.21.15.125:0/1836890885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fd40c6b80 0x7f4fd4157d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:05.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fc9d9b640 1 -- 172.21.15.125:0/1836890885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fc4009a70 con 0x7f4fd40c6f60 2023-10-30T12:35:05.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fc9d9b640 1 --2- 172.21.15.125:0/1836890885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fd40c6f60 0x7f4fd40c9d90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00c4d0 tx=0x7f4fbc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:05.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fcb7fe640 1 -- 172.21.15.125:0/1836890885 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc005cf0 con 0x7f4fd40c6f60 2023-10-30T12:35:05.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fcb7fe640 1 -- 172.21.15.125:0/1836890885 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fbc005e50 con 0x7f4fd40c6f60 2023-10-30T12:35:05.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fcb7fe640 1 -- 172.21.15.125:0/1836890885 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fbc0072b0 con 0x7f4fd40c6f60 2023-10-30T12:35:05.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fd8b94640 1 -- 172.21.15.125:0/1836890885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fd4136c60 con 0x7f4fd40c6f60 2023-10-30T12:35:05.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fd8b94640 1 -- 172.21.15.125:0/1836890885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fd40cab90 con 0x7f4fd40c6f60 2023-10-30T12:35:05.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fd8b94640 1 -- 172.21.15.125:0/1836890885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fd404e170 con 0x7f4fd40c6f60 2023-10-30T12:35:05.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.102+0000 7f4fcb7fe640 1 -- 172.21.15.125:0/1836890885 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4fbc007410 con 0x7f4fd40c6f60 2023-10-30T12:35:05.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.106+0000 7f4fcb7fe640 1 --2- 172.21.15.125:0/1836890885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4fb40664d0 0x7f4fb4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:05.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.106+0000 7f4fcb7fe640 1 -- 172.21.15.125:0/1836890885 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6626+0+0 (secure 0 0 0) 0x7f4fbc0837d0 con 0x7f4fd40c6f60 2023-10-30T12:35:05.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.106+0000 7f4fd259c640 1 --2- 172.21.15.125:0/1836890885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4fb40664d0 0x7f4fb4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:05.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.106+0000 7f4fd259c640 1 --2- 172.21.15.125:0/1836890885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4fb40664d0 0x7f4fb4068990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40af000 tx=0x7f4fc40a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:05.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.110+0000 7f4fcb7fe640 1 -- 172.21.15.125:0/1836890885 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4fbc050830 con 0x7f4fd40c6f60 2023-10-30T12:35:05.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.294+0000 7f4fd8b94640 1 -- 172.21.15.125:0/1836890885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fd40c6600 con 0x7f4fd40c6f60 2023-10-30T12:35:05.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.294+0000 7f4fcb7fe640 1 -- 172.21.15.125:0/1836890885 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f4fbc017090 con 0x7f4fd40c6f60 2023-10-30T12:35:05.299 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:05.299 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:00.120705+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:05.299 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 36 2023-10-30T12:35:05.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.298+0000 7f4fd8b94640 1 -- 172.21.15.125:0/1836890885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4fb40664d0 msgr2=0x7f4fb4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:05.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.298+0000 7f4fd8b94640 1 --2- 172.21.15.125:0/1836890885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4fb40664d0 0x7f4fb4068990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4fc40af000 tx=0x7f4fc40a7070 comp rx=0 tx=0).stop 2023-10-30T12:35:05.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.298+0000 7f4fd8b94640 1 -- 172.21.15.125:0/1836890885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fd40c6f60 msgr2=0x7f4fd40c9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:05.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.298+0000 7f4fd8b94640 1 --2- 172.21.15.125:0/1836890885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fd40c6f60 0x7f4fd40c9d90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4fbc00c4d0 tx=0x7f4fbc00c990 comp rx=0 tx=0).stop 2023-10-30T12:35:05.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.298+0000 7f4fd8b94640 1 -- 172.21.15.125:0/1836890885 shutdown_connections 2023-10-30T12:35:05.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.298+0000 7f4fd8b94640 1 --2- 172.21.15.125:0/1836890885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4fb40664d0 0x7f4fb4068990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:05.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.298+0000 7f4fd8b94640 1 --2- 172.21.15.125:0/1836890885 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4fd413f4a0 0x7f4fd413bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:05.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.298+0000 7f4fd8b94640 1 --2- 172.21.15.125:0/1836890885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4fd40c6f60 0x7f4fd40c9d90 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:05.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.298+0000 7f4fd8b94640 1 --2- 172.21.15.125:0/1836890885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4fd40c6b80 0x7f4fd4157d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:05.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.298+0000 7f4fd8b94640 1 -- 172.21.15.125:0/1836890885 >> 172.21.15.125:0/1836890885 conn(0x7f4fd40c7910 msgr2=0x7f4fd4139970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:05.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.298+0000 7f4fd8b94640 1 -- 172.21.15.125:0/1836890885 shutdown_connections 2023-10-30T12:35:05.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:05.298+0000 7f4fd8b94640 1 -- 172.21.15.125:0/1836890885 wait complete. 2023-10-30T12:35:05.319 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:35:07.320 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:07.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.462+0000 7f74d950f640 1 Processor -- start 2023-10-30T12:35:07.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d950f640 1 -- start start 2023-10-30T12:35:07.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d950f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f74d40cc0b0 0x7f74d406a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d950f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d40b1ea0 0x7f74d40cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d950f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d406ade0 con 0x7f74d4153c30 2023-10-30T12:35:07.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d950f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d41596e0 con 0x7f74d40cc0b0 2023-10-30T12:35:07.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d950f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d41598f0 con 0x7f74d40b1ea0 2023-10-30T12:35:07.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d2ffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f74d4153c30 0x7f74d40cb9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:56672/0 (socket says 172.21.15.125:56672) 2023-10-30T12:35:07.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d2ffd640 1 -- 172.21.15.125:0/2361224101 learned_addr learned my addr 172.21.15.125:0/2361224101 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:35:07.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74ca7fc640 1 --2- 172.21.15.125:0/2361224101 >> v2:172.21.15.136:3300/0 conn(0x7f74d40cc0b0 0x7f74d406a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d37fe640 1 --2- 172.21.15.125:0/2361224101 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d40b1ea0 0x7f74d40cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 -- 172.21.15.125:0/2361224101 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3978943037 0 0) 0x7f74c4001120 con 0x7f74d4153c30 2023-10-30T12:35:07.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 -- 172.21.15.125:0/2361224101 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c0003530 con 0x7f74d4153c30 2023-10-30T12:35:07.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 -- 172.21.15.125:0/2361224101 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 63445457 0 0) 0x7f74c0003530 con 0x7f74d4153c30 2023-10-30T12:35:07.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 -- 172.21.15.125:0/2361224101 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d40b1ea0 msgr2=0x7f74d40cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:07.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 --2- 172.21.15.125:0/2361224101 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d40b1ea0 0x7f74d40cabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 -- 172.21.15.125:0/2361224101 >> v2:172.21.15.136:3300/0 conn(0x7f74d40cc0b0 msgr2=0x7f74d406a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:07.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 --2- 172.21.15.125:0/2361224101 >> v2:172.21.15.136:3300/0 conn(0x7f74d40cc0b0 0x7f74d406a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 -- 172.21.15.125:0/2361224101 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d4159b90 con 0x7f74d4153c30 2023-10-30T12:35:07.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 -- 172.21.15.125:0/2361224101 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f74c4001810 con 0x7f74d4153c30 2023-10-30T12:35:07.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 -- 172.21.15.125:0/2361224101 >> v1:172.21.15.125:6789/0 conn(0x7f74d4153c30 legacy=0x7f74d40cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:07.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 --2- 172.21.15.125:0/2361224101 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d40b1ea0 0x7f74c0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 --2- 172.21.15.125:0/2361224101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74c0003f00 0x7f74c00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 --2- 172.21.15.125:0/2361224101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74c0004d10 0x7f74c0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 -- 172.21.15.125:0/2361224101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d41596e0 con 0x7f74c0003f00 2023-10-30T12:35:07.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 -- 172.21.15.125:0/2361224101 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d41598f0 con 0x7f74c0004d10 2023-10-30T12:35:07.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d27fc640 1 -- 172.21.15.125:0/2361224101 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74c000b9c0 con 0x7f74d40b1ea0 2023-10-30T12:35:07.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74ca7fc640 1 --2- 172.21.15.125:0/2361224101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74c0003f00 0x7f74c00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d2ffd640 1 --2- 172.21.15.125:0/2361224101 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d40b1ea0 0x7f74c0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d37fe640 1 --2- 172.21.15.125:0/2361224101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74c0004d10 0x7f74c0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74ca7fc640 1 -- 172.21.15.125:0/2361224101 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d40b1ea0 msgr2=0x7f74c0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:07.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74ca7fc640 1 --2- 172.21.15.125:0/2361224101 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d40b1ea0 0x7f74c0003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74ca7fc640 1 -- 172.21.15.125:0/2361224101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74c0004d10 msgr2=0x7f74c0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:07.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74ca7fc640 1 --2- 172.21.15.125:0/2361224101 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74c0004d10 0x7f74c0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74ca7fc640 1 -- 172.21.15.125:0/2361224101 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c0003530 con 0x7f74c0003f00 2023-10-30T12:35:07.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.466+0000 7f74d2ffd640 1 --2- 172.21.15.125:0/2361224101 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d40b1ea0 0x7f74c0003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:07.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.470+0000 7f74ca7fc640 1 --2- 172.21.15.125:0/2361224101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74c0003f00 0x7f74c00047d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00e170 tx=0x7f74bc01cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38758aac77f11893 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:07.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.470+0000 7f74d27fc640 1 -- 172.21.15.125:0/2361224101 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc0a5cf0 con 0x7f74c0003f00 2023-10-30T12:35:07.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.470+0000 7f74d27fc640 1 -- 172.21.15.125:0/2361224101 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74bc00f040 con 0x7f74c0003f00 2023-10-30T12:35:07.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.470+0000 7f74d27fc640 1 -- 172.21.15.125:0/2361224101 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc0ad680 con 0x7f74c0003f00 2023-10-30T12:35:07.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.470+0000 7f74d950f640 1 -- 172.21.15.125:0/2361224101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74c0003f00 msgr2=0x7f74c00047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:07.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.470+0000 7f74d950f640 1 --2- 172.21.15.125:0/2361224101 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74c0003f00 0x7f74c00047d0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f74bc00e170 tx=0x7f74bc01cce0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.470+0000 7f74d37fe640 1 -- 172.21.15.125:0/2361224101 reap_dead start 2023-10-30T12:35:07.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.470+0000 7f74d950f640 1 -- 172.21.15.125:0/2361224101 shutdown_connections 2023-10-30T12:35:07.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.470+0000 7f74d950f640 1 -- 172.21.15.125:0/2361224101 >> 172.21.15.125:0/2361224101 conn(0x7f74d406bf40 msgr2=0x7f74d406c350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:07.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.470+0000 7f74d950f640 1 -- 172.21.15.125:0/2361224101 shutdown_connections 2023-10-30T12:35:07.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.470+0000 7f74d950f640 1 -- 172.21.15.125:0/2361224101 wait complete. 2023-10-30T12:35:07.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74d950f640 1 Processor -- start 2023-10-30T12:35:07.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74d950f640 1 -- start start 2023-10-30T12:35:07.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74d950f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74c0003f00 0x7f74d40c7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74d950f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d40cf300 0x7f74d40c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74d950f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d40c6230 0x7f74d415baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74d950f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40ce350 con 0x7f74d40cf300 2023-10-30T12:35:07.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74d950f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40cfe00 con 0x7f74c0003f00 2023-10-30T12:35:07.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74d950f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d40cf840 con 0x7f74d40c6230 2023-10-30T12:35:07.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74ca7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d40cf300 0x7f74d40c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74ca7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d40cf300 0x7f74d40c4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46536/0 (socket says 172.21.15.125:46536) 2023-10-30T12:35:07.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74ca7fc640 1 -- 172.21.15.125:0/2837997196 learned_addr learned my addr 172.21.15.125:0/2837997196 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:07.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74d37fe640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d40c6230 0x7f74d415baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74d2ffd640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74c0003f00 0x7f74d40c7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74ca7fc640 1 -- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d40c6230 msgr2=0x7f74d415baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:07.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74ca7fc640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d40c6230 0x7f74d415baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74ca7fc640 1 -- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74c0003f00 msgr2=0x7f74d40c7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:07.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74ca7fc640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74c0003f00 0x7f74d40c7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74ca7fc640 1 -- 172.21.15.125:0/2837997196 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d40cfb20 con 0x7f74d40cf300 2023-10-30T12:35:07.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74d2ffd640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74c0003f00 0x7f74d40c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:07.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74ca7fc640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d40cf300 0x7f74d40c4d40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f74bc0b0040 tx=0x7f74bc009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:07.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74cbfff640 1 -- 172.21.15.125:0/2837997196 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc0adc00 con 0x7f74d40cf300 2023-10-30T12:35:07.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74cbfff640 1 -- 172.21.15.125:0/2837997196 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74bc0add60 con 0x7f74d40cf300 2023-10-30T12:35:07.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74cbfff640 1 -- 172.21.15.125:0/2837997196 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74bc0b6650 con 0x7f74d40cf300 2023-10-30T12:35:07.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74d950f640 1 -- 172.21.15.125:0/2837997196 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d40cdc90 con 0x7f74d40cf300 2023-10-30T12:35:07.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.474+0000 7f74d950f640 1 -- 172.21.15.125:0/2837997196 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d415d7c0 con 0x7f74d40cf300 2023-10-30T12:35:07.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.478+0000 7f74d950f640 1 -- 172.21.15.125:0/2837997196 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d404e170 con 0x7f74d40cf300 2023-10-30T12:35:07.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.478+0000 7f74cbfff640 1 -- 172.21.15.125:0/2837997196 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f74bc0b4040 con 0x7f74d40cf300 2023-10-30T12:35:07.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.478+0000 7f74cbfff640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f74b40664a0 0x7f74b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:07.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.478+0000 7f74cbfff640 1 -- 172.21.15.125:0/2837997196 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6626+0+0 (secure 0 0 0) 0x7f74bc11ed80 con 0x7f74d40cf300 2023-10-30T12:35:07.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.478+0000 7f74d2ffd640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f74b40664a0 0x7f74b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:07.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.478+0000 7f74d2ffd640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f74b40664a0 0x7f74b4068960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f74c4001020 tx=0x7f74c4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:07.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.482+0000 7f74cbfff640 1 -- 172.21.15.125:0/2837997196 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f74bc0ebef0 con 0x7f74d40cf300 2023-10-30T12:35:07.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.662+0000 7f74d950f640 1 -- 172.21.15.125:0/2837997196 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74d415a120 con 0x7f74d40cf300 2023-10-30T12:35:07.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.666+0000 7f74cbfff640 1 -- 172.21.15.125:0/2837997196 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f74bc0b2020 con 0x7f74d40cf300 2023-10-30T12:35:07.668 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:07.668 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:00.120705+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:07.668 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 36 2023-10-30T12:35:07.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.666+0000 7f74d950f640 1 -- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f74b40664a0 msgr2=0x7f74b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:07.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.666+0000 7f74d950f640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f74b40664a0 0x7f74b4068960 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f74c4001020 tx=0x7f74c4009040 comp rx=0 tx=0).stop 2023-10-30T12:35:07.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.666+0000 7f74d950f640 1 -- 172.21.15.125:0/2837997196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d40cf300 msgr2=0x7f74d40c4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:07.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.666+0000 7f74d950f640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d40cf300 0x7f74d40c4d40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f74bc0b0040 tx=0x7f74bc009750 comp rx=0 tx=0).stop 2023-10-30T12:35:07.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.666+0000 7f74d950f640 1 -- 172.21.15.125:0/2837997196 shutdown_connections 2023-10-30T12:35:07.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.666+0000 7f74d950f640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f74b40664a0 0x7f74b4068960 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.666+0000 7f74d950f640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d40c6230 0x7f74d415baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.666+0000 7f74d950f640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d40cf300 0x7f74d40c4d40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.666+0000 7f74d950f640 1 --2- 172.21.15.125:0/2837997196 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74c0003f00 0x7f74d40c7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.666+0000 7f74d950f640 1 -- 172.21.15.125:0/2837997196 >> 172.21.15.125:0/2837997196 conn(0x7f74d406bf40 msgr2=0x7f74d406c350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.666+0000 7f74d950f640 1 -- 172.21.15.125:0/2837997196 shutdown_connections 2023-10-30T12:35:07.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:07.666+0000 7f74d950f640 1 -- 172.21.15.125:0/2837997196 wait complete. 2023-10-30T12:35:07.688 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:35:09.689 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:09.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.834+0000 7fd731479640 1 Processor -- start 2023-10-30T12:35:09.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.834+0000 7fd731479640 1 -- start start 2023-10-30T12:35:09.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.834+0000 7fd731479640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd72c1538a0 0x7fd72c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:09.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.834+0000 7fd731479640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd72c0cb800 con 0x7fd72c0ca0e0 2023-10-30T12:35:09.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.834+0000 7fd731479640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd72c15dc30 con 0x7fd72c0cf160 2023-10-30T12:35:09.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.834+0000 7fd731479640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd72c0cb330 con 0x7fd72c1538a0 2023-10-30T12:35:09.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.834+0000 7fd72affd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd72c1538a0 0x7fd72c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:09.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.834+0000 7fd723fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd72c0ca0e0 0x7fd72c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:56674/0 (socket says 172.21.15.125:56674) 2023-10-30T12:35:09.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.834+0000 7fd723fff640 1 -- 172.21.15.125:0/3814695684 learned_addr learned my addr 172.21.15.125:0/3814695684 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:35:09.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.834+0000 7fd72affd640 1 -- 172.21.15.125:0/3814695684 >> v1:172.21.15.136:6789/0 conn(0x7fd72c0cf160 legacy=0x7fd72c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:09.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.834+0000 7fd72affd640 1 -- 172.21.15.125:0/3814695684 >> v1:172.21.15.125:6789/0 conn(0x7fd72c0ca0e0 legacy=0x7fd72c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:09.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.834+0000 7fd72a7fc640 1 -- 172.21.15.125:0/3814695684 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812713939 0 0) 0x7fd72c0cb800 con 0x7fd72c0ca0e0 2023-10-30T12:35:09.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.834+0000 7fd72affd640 1 -- 172.21.15.125:0/3814695684 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd72c0cad70 con 0x7fd72c1538a0 2023-10-30T12:35:09.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.838+0000 7fd72affd640 1 --2- 172.21.15.125:0/3814695684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd72c1538a0 0x7fd72c15d6f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd71c0097c0 tx=0x7fd71c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=132e8adcdc839036 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:09.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.838+0000 7fd72a7fc640 1 -- 172.21.15.125:0/3814695684 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71c0a8970 con 0x7fd72c1538a0 2023-10-30T12:35:09.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.838+0000 7fd72a7fc640 1 -- 172.21.15.125:0/3814695684 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71c0a8ad0 con 0x7fd72c1538a0 2023-10-30T12:35:09.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.838+0000 7fd72a7fc640 1 -- 172.21.15.125:0/3814695684 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd71c0a8e60 con 0x7fd72c1538a0 2023-10-30T12:35:09.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.838+0000 7fd731479640 1 -- 172.21.15.125:0/3814695684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd72c1538a0 msgr2=0x7fd72c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:09.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.838+0000 7fd731479640 1 --2- 172.21.15.125:0/3814695684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd72c1538a0 0x7fd72c15d6f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd71c0097c0 tx=0x7fd71c01ec70 comp rx=0 tx=0).stop 2023-10-30T12:35:09.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.838+0000 7fd731479640 1 -- 172.21.15.125:0/3814695684 shutdown_connections 2023-10-30T12:35:09.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.838+0000 7fd731479640 1 --2- 172.21.15.125:0/3814695684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd72c1538a0 0x7fd72c15d6f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:09.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.838+0000 7fd731479640 1 -- 172.21.15.125:0/3814695684 >> 172.21.15.125:0/3814695684 conn(0x7fd72c06bfe0 msgr2=0x7fd72c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:09.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.838+0000 7fd731479640 1 -- 172.21.15.125:0/3814695684 shutdown_connections 2023-10-30T12:35:09.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.838+0000 7fd731479640 1 -- 172.21.15.125:0/3814695684 wait complete. 2023-10-30T12:35:09.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd731479640 1 Processor -- start 2023-10-30T12:35:09.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd731479640 1 -- start start 2023-10-30T12:35:09.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd731479640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd72c0ca0e0 0x7fd72c0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:09.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd731479640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd72c0cf160 0x7fd72c148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:09.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd731479640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd72c1538a0 0x7fd72c1448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:09.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd731479640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c1e5480 con 0x7fd72c0cf160 2023-10-30T12:35:09.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd731479640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c0b4d10 con 0x7fd72c1538a0 2023-10-30T12:35:09.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd731479640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c144010 con 0x7fd72c0ca0e0 2023-10-30T12:35:09.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd723fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd72c0cf160 0x7fd72c148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:09.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd723fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd72c0cf160 0x7fd72c148080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46546/0 (socket says 172.21.15.125:46546) 2023-10-30T12:35:09.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd723fff640 1 -- 172.21.15.125:0/253417450 learned_addr learned my addr 172.21.15.125:0/253417450 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:09.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd72affd640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd72c0ca0e0 0x7fd72c0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:09.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd72b7fe640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd72c1538a0 0x7fd72c1448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:09.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd723fff640 1 -- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd72c0ca0e0 msgr2=0x7fd72c0b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd723fff640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd72c0ca0e0 0x7fd72c0b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd723fff640 1 -- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd72c1538a0 msgr2=0x7fd72c1448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd723fff640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd72c1538a0 0x7fd72c1448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:09.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd723fff640 1 -- 172.21.15.125:0/253417450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd71c009420 con 0x7fd72c0cf160 2023-10-30T12:35:09.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd72b7fe640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd72c1538a0 0x7fd72c1448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:09.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd723fff640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd72c0cf160 0x7fd72c148080 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd71400c410 tx=0x7fd71400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:09.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd7237fe640 1 -- 172.21.15.125:0/253417450 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7140074e0 con 0x7fd72c0cf160 2023-10-30T12:35:09.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd7237fe640 1 -- 172.21.15.125:0/253417450 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd714007640 con 0x7fd72c0cf160 2023-10-30T12:35:09.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd7237fe640 1 -- 172.21.15.125:0/253417450 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd714004a30 con 0x7fd72c0cf160 2023-10-30T12:35:09.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd731479640 1 -- 172.21.15.125:0/253417450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd72c144420 con 0x7fd72c0cf160 2023-10-30T12:35:09.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd731479640 1 -- 172.21.15.125:0/253417450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd72c1e49e0 con 0x7fd72c0cf160 2023-10-30T12:35:09.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.842+0000 7fd731479640 1 -- 172.21.15.125:0/253417450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd72c0b4240 con 0x7fd72c0cf160 2023-10-30T12:35:09.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.846+0000 7fd7237fe640 1 -- 172.21.15.125:0/253417450 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd714004b90 con 0x7fd72c0cf160 2023-10-30T12:35:09.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.846+0000 7fd7237fe640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd70c0664a0 0x7fd70c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:09.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.846+0000 7fd7237fe640 1 -- 172.21.15.125:0/253417450 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6626+0+0 (secure 0 0 0) 0x7fd714083f00 con 0x7fd72c0cf160 2023-10-30T12:35:09.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.846+0000 7fd72affd640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd70c0664a0 0x7fd70c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:09.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.846+0000 7fd72affd640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd70c0664a0 0x7fd70c068960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd71c0b2000 tx=0x7fd71c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:09.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:09.850+0000 7fd7237fe640 1 -- 172.21.15.125:0/253417450 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd714050fd0 con 0x7fd72c0cf160 2023-10-30T12:35:10.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:10.034+0000 7fd731479640 1 -- 172.21.15.125:0/253417450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd72c04e170 con 0x7fd72c0cf160 2023-10-30T12:35:10.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:10.034+0000 7fd7237fe640 1 -- 172.21.15.125:0/253417450 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6549 (secure 0 0 0) 0x7fd7140545f0 con 0x7fd72c0cf160 2023-10-30T12:35:10.037 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:10.037 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:00.120705+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:10.037 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 36 2023-10-30T12:35:10.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:10.034+0000 7fd731479640 1 -- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd70c0664a0 msgr2=0x7fd70c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:10.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:10.034+0000 7fd731479640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd70c0664a0 0x7fd70c068960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd71c0b2000 tx=0x7fd71c008040 comp rx=0 tx=0).stop 2023-10-30T12:35:10.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:10.034+0000 7fd731479640 1 -- 172.21.15.125:0/253417450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd72c0cf160 msgr2=0x7fd72c148080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:10.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:10.034+0000 7fd731479640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd72c0cf160 0x7fd72c148080 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd71400c410 tx=0x7fd71400c8d0 comp rx=0 tx=0).stop 2023-10-30T12:35:10.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:10.034+0000 7fd731479640 1 -- 172.21.15.125:0/253417450 shutdown_connections 2023-10-30T12:35:10.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:10.034+0000 7fd731479640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd70c0664a0 0x7fd70c068960 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:10.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:10.034+0000 7fd731479640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd72c1538a0 0x7fd72c1448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:10.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:10.034+0000 7fd731479640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd72c0cf160 0x7fd72c148080 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:10.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:10.034+0000 7fd731479640 1 --2- 172.21.15.125:0/253417450 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd72c0ca0e0 0x7fd72c0b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:10.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:10.034+0000 7fd731479640 1 -- 172.21.15.125:0/253417450 >> 172.21.15.125:0/253417450 conn(0x7fd72c06bfe0 msgr2=0x7fd72c142980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:10.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:10.034+0000 7fd731479640 1 -- 172.21.15.125:0/253417450 shutdown_connections 2023-10-30T12:35:10.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:10.034+0000 7fd731479640 1 -- 172.21.15.125:0/253417450 wait complete. 2023-10-30T12:35:10.057 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:35:12.058 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:12.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.201+0000 7fd8b591d640 1 Processor -- start 2023-10-30T12:35:12.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.201+0000 7fd8b591d640 1 -- start start 2023-10-30T12:35:12.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8b591d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd8b0069e90 0x7fd8b006a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:12.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8b591d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b0153d80 con 0x7fd8b01539a0 2023-10-30T12:35:12.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8b591d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b00c5c50 con 0x7fd8b0070a00 2023-10-30T12:35:12.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8b591d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b006fbb0 con 0x7fd8b0069e90 2023-10-30T12:35:12.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8af7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd8b0069e90 0x7fd8b006a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:12.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8af7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd8b0069e90 0x7fd8b006a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60852/0 (socket says 172.21.15.125:60852) 2023-10-30T12:35:12.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8af7fe640 1 -- 172.21.15.125:0/782616861 learned_addr learned my addr 172.21.15.125:0/782616861 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:12.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8aeffd640 1 --1- 172.21.15.125:0/782616861 >> v1:172.21.15.125:6789/0 conn(0x7fd8b01539a0 0x7fd8b015a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33840/0 (socket says 172.21.15.125:33840) 2023-10-30T12:35:12.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8af7fe640 1 -- 172.21.15.125:0/782616861 >> v1:172.21.15.136:6789/0 conn(0x7fd8b0070a00 legacy=0x7fd8b0070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:12.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8af7fe640 1 -- 172.21.15.125:0/782616861 >> v1:172.21.15.125:6789/0 conn(0x7fd8b01539a0 legacy=0x7fd8b015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:12.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8af7fe640 1 -- 172.21.15.125:0/782616861 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00c5f30 con 0x7fd8b0069e90 2023-10-30T12:35:12.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8af7fe640 1 --2- 172.21.15.125:0/782616861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b0069e90 0x7fd8b006a300 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009e50 tx=0x7fd8a801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9168bf2aa24ea5a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:12.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8ae7fc640 1 -- 172.21.15.125:0/782616861 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a80a5720 con 0x7fd8b0069e90 2023-10-30T12:35:12.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8ae7fc640 1 -- 172.21.15.125:0/782616861 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a80a5880 con 0x7fd8b0069e90 2023-10-30T12:35:12.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.205+0000 7fd8ae7fc640 1 -- 172.21.15.125:0/782616861 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a80a5c10 con 0x7fd8b0069e90 2023-10-30T12:35:12.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 -- 172.21.15.125:0/782616861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b0069e90 msgr2=0x7fd8b006a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:12.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 --2- 172.21.15.125:0/782616861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b0069e90 0x7fd8b006a300 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd8a8009e50 tx=0x7fd8a801c950 comp rx=0 tx=0).stop 2023-10-30T12:35:12.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 -- 172.21.15.125:0/782616861 shutdown_connections 2023-10-30T12:35:12.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 --2- 172.21.15.125:0/782616861 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b0069e90 0x7fd8b006a300 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 -- 172.21.15.125:0/782616861 >> 172.21.15.125:0/782616861 conn(0x7fd8b00c7910 msgr2=0x7fd8b00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:12.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 -- 172.21.15.125:0/782616861 shutdown_connections 2023-10-30T12:35:12.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 -- 172.21.15.125:0/782616861 wait complete. 2023-10-30T12:35:12.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 Processor -- start 2023-10-30T12:35:12.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 -- start start 2023-10-30T12:35:12.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b0069e90 0x7fd8b01450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:12.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8b0070a00 0x7fd8b00ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:12.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8b01539a0 0x7fd8b00c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:12.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b0146cf0 con 0x7fd8b0069e90 2023-10-30T12:35:12.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b01479a0 con 0x7fd8b01539a0 2023-10-30T12:35:12.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8b591d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b0144190 con 0x7fd8b0070a00 2023-10-30T12:35:12.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8aeffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b0069e90 0x7fd8b01450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:12.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8aeffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b0069e90 0x7fd8b01450f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60856/0 (socket says 172.21.15.125:60856) 2023-10-30T12:35:12.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8aeffd640 1 -- 172.21.15.125:0/261849735 learned_addr learned my addr 172.21.15.125:0/261849735 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:12.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8a7fff640 1 --2- 172.21.15.125:0/261849735 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8b0070a00 0x7fd8b00ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:12.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8af7fe640 1 --2- 172.21.15.125:0/261849735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8b01539a0 0x7fd8b00c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:12.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8aeffd640 1 -- 172.21.15.125:0/261849735 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8b0070a00 msgr2=0x7fd8b00ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:12.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8aeffd640 1 --2- 172.21.15.125:0/261849735 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8b0070a00 0x7fd8b00ca820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8aeffd640 1 -- 172.21.15.125:0/261849735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8b01539a0 msgr2=0x7fd8b00c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:12.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8aeffd640 1 --2- 172.21.15.125:0/261849735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8b01539a0 0x7fd8b00c12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.209+0000 7fd8aeffd640 1 -- 172.21.15.125:0/261849735 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a8009b30 con 0x7fd8b0069e90 2023-10-30T12:35:12.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.213+0000 7fd8aeffd640 1 --2- 172.21.15.125:0/261849735 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b0069e90 0x7fd8b01450f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000c490 tx=0x7fd8a000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:12.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.213+0000 7fd8acff9640 1 -- 172.21.15.125:0/261849735 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0007480 con 0x7fd8b0069e90 2023-10-30T12:35:12.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.213+0000 7fd8acff9640 1 -- 172.21.15.125:0/261849735 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a00075e0 con 0x7fd8b0069e90 2023-10-30T12:35:12.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.213+0000 7fd8acff9640 1 -- 172.21.15.125:0/261849735 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0004a10 con 0x7fd8b0069e90 2023-10-30T12:35:12.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.213+0000 7fd8b591d640 1 -- 172.21.15.125:0/261849735 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b0143ec0 con 0x7fd8b0069e90 2023-10-30T12:35:12.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.213+0000 7fd8b591d640 1 -- 172.21.15.125:0/261849735 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b0147070 con 0x7fd8b0069e90 2023-10-30T12:35:12.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.213+0000 7fd8b591d640 1 -- 172.21.15.125:0/261849735 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b004e170 con 0x7fd8b0069e90 2023-10-30T12:35:12.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.213+0000 7fd8acff9640 1 -- 172.21.15.125:0/261849735 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd8a0004b70 con 0x7fd8b0069e90 2023-10-30T12:35:12.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.213+0000 7fd8acff9640 1 --2- 172.21.15.125:0/261849735 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8900664d0 0x7fd890068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:12.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.213+0000 7fd8acff9640 1 -- 172.21.15.125:0/261849735 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6626+0+0 (secure 0 0 0) 0x7fd8a00832a0 con 0x7fd8b0069e90 2023-10-30T12:35:12.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.213+0000 7fd8a7fff640 1 --2- 172.21.15.125:0/261849735 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8900664d0 0x7fd890068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:12.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.217+0000 7fd8a7fff640 1 --2- 172.21.15.125:0/261849735 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8900664d0 0x7fd890068990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0146c60 tx=0x7fd898009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:12.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.217+0000 7fd8acff9640 1 -- 172.21.15.125:0/261849735 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd8a00503b0 con 0x7fd8b0069e90 2023-10-30T12:35:12.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.397+0000 7fd8b591d640 1 -- 172.21.15.125:0/261849735 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8b00c72f0 con 0x7fd8b0069e90 2023-10-30T12:35:12.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.397+0000 7fd8acff9640 1 -- 172.21.15.125:0/261849735 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6549 (secure 0 0 0) 0x7fd8a00539d0 con 0x7fd8b0069e90 2023-10-30T12:35:12.403 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:12.403 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:00.120705+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:12.403 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 36 2023-10-30T12:35:12.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.401+0000 7fd8b591d640 1 -- 172.21.15.125:0/261849735 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8900664d0 msgr2=0x7fd890068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:12.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.401+0000 7fd8b591d640 1 --2- 172.21.15.125:0/261849735 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8900664d0 0x7fd890068990 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd8b0146c60 tx=0x7fd898009040 comp rx=0 tx=0).stop 2023-10-30T12:35:12.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.401+0000 7fd8b591d640 1 -- 172.21.15.125:0/261849735 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b0069e90 msgr2=0x7fd8b01450f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:12.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.401+0000 7fd8b591d640 1 --2- 172.21.15.125:0/261849735 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b0069e90 0x7fd8b01450f0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000c490 tx=0x7fd8a000c950 comp rx=0 tx=0).stop 2023-10-30T12:35:12.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.401+0000 7fd8b591d640 1 -- 172.21.15.125:0/261849735 shutdown_connections 2023-10-30T12:35:12.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.401+0000 7fd8b591d640 1 --2- 172.21.15.125:0/261849735 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8900664d0 0x7fd890068990 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.401+0000 7fd8b591d640 1 --2- 172.21.15.125:0/261849735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8b01539a0 0x7fd8b00c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.401+0000 7fd8b591d640 1 --2- 172.21.15.125:0/261849735 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8b0070a00 0x7fd8b00ca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.401+0000 7fd8b591d640 1 --2- 172.21.15.125:0/261849735 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b0069e90 0x7fd8b01450f0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:12.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.401+0000 7fd8b591d640 1 -- 172.21.15.125:0/261849735 >> 172.21.15.125:0/261849735 conn(0x7fd8b00c7910 msgr2=0x7fd8b0149c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:12.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.401+0000 7fd8b591d640 1 -- 172.21.15.125:0/261849735 shutdown_connections 2023-10-30T12:35:12.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:12.401+0000 7fd8b591d640 1 -- 172.21.15.125:0/261849735 wait complete. 2023-10-30T12:35:12.423 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:35:14.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:14.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea3e09640 1 Processor -- start 2023-10-30T12:35:14.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea3e09640 1 -- start start 2023-10-30T12:35:14.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea3e09640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4e9c06f7a0 0x7f4e9c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:14.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea3e09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e9c0c6bd0 0x7f4e9c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:14.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea3e09640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9c06a3d0 con 0x7f4e9c071110 2023-10-30T12:35:14.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea3e09640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e9c0700c0 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea3e09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c066cf0 con 0x7f4e9c0c6bd0 2023-10-30T12:35:14.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea1ba5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4e9c06f7a0 0x7f4e9c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:14.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea1ba5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4e9c06f7a0 0x7f4e9c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60866/0 (socket says 172.21.15.125:60866) 2023-10-30T12:35:14.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea1ba5640 1 -- 172.21.15.125:0/1881137097 learned_addr learned my addr 172.21.15.125:0/1881137097 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:14.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea13a4640 1 --2- 172.21.15.125:0/1881137097 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e9c0c6bd0 0x7f4e9c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:14.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea1ba5640 1 -- 172.21.15.125:0/1881137097 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e9c0c6bd0 msgr2=0x7f4e9c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:14.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea1ba5640 1 --2- 172.21.15.125:0/1881137097 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e9c0c6bd0 0x7f4e9c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea1ba5640 1 -- 172.21.15.125:0/1881137097 >> v1:172.21.15.136:6789/0 conn(0x7f4e9c071110 legacy=0x7f4e9c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:14.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.569+0000 7f4ea1ba5640 1 -- 172.21.15.125:0/1881137097 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c0c70d0 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.573+0000 7f4ea1ba5640 1 --2- 172.21.15.125:0/1881137097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e9c06f7a0 0x7f4e9c06fb80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4e98009870 tx=0x7f4e9801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=791f80a4a1f1632f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:14.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.573+0000 7f4ea0ba3640 1 -- 172.21.15.125:0/1881137097 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e980a8a30 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.573+0000 7f4ea0ba3640 1 -- 172.21.15.125:0/1881137097 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e980a8b90 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.573+0000 7f4ea0ba3640 1 -- 172.21.15.125:0/1881137097 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e980b0440 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.573+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1881137097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e9c06f7a0 msgr2=0x7f4e9c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:14.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.573+0000 7f4ea3e09640 1 --2- 172.21.15.125:0/1881137097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e9c06f7a0 0x7f4e9c06fb80 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4e98009870 tx=0x7f4e9801ed30 comp rx=0 tx=0).stop 2023-10-30T12:35:14.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.573+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1881137097 shutdown_connections 2023-10-30T12:35:14.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.573+0000 7f4ea3e09640 1 --2- 172.21.15.125:0/1881137097 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e9c0c6bd0 0x7f4e9c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.573+0000 7f4ea3e09640 1 --2- 172.21.15.125:0/1881137097 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e9c06f7a0 0x7f4e9c06fb80 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.573+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1881137097 >> 172.21.15.125:0/1881137097 conn(0x7f4e9c0c7910 msgr2=0x7f4e9c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:14.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.573+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1881137097 shutdown_connections 2023-10-30T12:35:14.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.573+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1881137097 wait complete. 2023-10-30T12:35:14.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea3e09640 1 Processor -- start 2023-10-30T12:35:14.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea3e09640 1 -- start start 2023-10-30T12:35:14.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea3e09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e9c06f7a0 0x7f4e9c14d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:14.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea3e09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e9c071110 0x7f4e9c1ed7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:14.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea3e09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e9c0c6bd0 0x7f4e9c1556b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:14.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea3e09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c1edd00 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea3e09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c14ad00 con 0x7f4e9c071110 2023-10-30T12:35:14.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea3e09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c15b7c0 con 0x7f4e9c0c6bd0 2023-10-30T12:35:14.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea1ba5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e9c06f7a0 0x7f4e9c14d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:14.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea1ba5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e9c06f7a0 0x7f4e9c14d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60878/0 (socket says 172.21.15.125:60878) 2023-10-30T12:35:14.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea1ba5640 1 -- 172.21.15.125:0/1488721497 learned_addr learned my addr 172.21.15.125:0/1488721497 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:14.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea13a4640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e9c071110 0x7f4e9c1ed7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:14.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea23a6640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e9c0c6bd0 0x7f4e9c1556b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:14.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea1ba5640 1 -- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e9c0c6bd0 msgr2=0x7f4e9c1556b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:14.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea1ba5640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e9c0c6bd0 0x7f4e9c1556b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea1ba5640 1 -- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e9c071110 msgr2=0x7f4e9c1ed7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:14.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea1ba5640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e9c071110 0x7f4e9c1ed7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea1ba5640 1 -- 172.21.15.125:0/1488721497 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e980094d0 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea13a4640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e9c071110 0x7f4e9c1ed7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:14.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea23a6640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e9c0c6bd0 0x7f4e9c1556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:35:14.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea1ba5640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e9c06f7a0 0x7f4e9c14d640 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4e980b3040 tx=0x7f4e9801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:14.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4e927fc640 1 -- 172.21.15.125:0/1488721497 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e980be070 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4e927fc640 1 -- 172.21.15.125:0/1488721497 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e980ba400 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4e927fc640 1 -- 172.21.15.125:0/1488721497 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e980ba7d0 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1488721497 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e9c15baa0 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1488721497 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9c1f0590 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.577+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1488721497 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e9c04e170 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.581+0000 7f4e927fc640 1 -- 172.21.15.125:0/1488721497 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4e980c3410 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.581+0000 7f4e927fc640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4e800664a0 0x7f4e80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:14.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.581+0000 7f4e927fc640 1 -- 172.21.15.125:0/1488721497 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6626+0+0 (secure 0 0 0) 0x7f4e98121e60 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.581+0000 7f4ea13a4640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4e800664a0 0x7f4e80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:14.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.581+0000 7f4ea13a4640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4e800664a0 0x7f4e80068960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c15bdb0 tx=0x7f4e8c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:14.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.585+0000 7f4e927fc640 1 -- 172.21.15.125:0/1488721497 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4e980eeea0 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.719 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:35:14.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:35:14.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.765+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1488721497 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e9c151120 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.765+0000 7f4e927fc640 1 -- 172.21.15.125:0/1488721497 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f4e980b5090 con 0x7f4e9c06f7a0 2023-10-30T12:35:14.770 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:14.770 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5130,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:00.120705+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":30,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":4285,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"f","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.136:6839/2250748765","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2250748765},{"type":"v1","addr":"172.21.15.136:6839","nonce":2250748765}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:14.770 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 36 2023-10-30T12:35:14.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.769+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4e800664a0 msgr2=0x7f4e80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:14.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.769+0000 7f4ea3e09640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4e800664a0 0x7f4e80068960 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c15bdb0 tx=0x7f4e8c008040 comp rx=0 tx=0).stop 2023-10-30T12:35:14.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.769+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1488721497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e9c06f7a0 msgr2=0x7f4e9c14d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:14.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.769+0000 7f4ea3e09640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e9c06f7a0 0x7f4e9c14d640 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4e980b3040 tx=0x7f4e9801bad0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.769+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1488721497 shutdown_connections 2023-10-30T12:35:14.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.769+0000 7f4ea3e09640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4e800664a0 0x7f4e80068960 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.769+0000 7f4ea3e09640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e9c0c6bd0 0x7f4e9c1556b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.769+0000 7f4ea3e09640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e9c071110 0x7f4e9c1ed7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.769+0000 7f4ea3e09640 1 --2- 172.21.15.125:0/1488721497 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e9c06f7a0 0x7f4e9c14d640 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:14.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.769+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1488721497 >> 172.21.15.125:0/1488721497 conn(0x7f4e9c0c7910 msgr2=0x7f4e9c0c61d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:14.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.769+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1488721497 shutdown_connections 2023-10-30T12:35:14.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:14.769+0000 7f4ea3e09640 1 -- 172.21.15.125:0/1488721497 wait complete. 2023-10-30T12:35:14.790 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:35:16.791 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:16.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.933+0000 7fb4703f5640 1 Processor -- start 2023-10-30T12:35:16.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.933+0000 7fb4703f5640 1 -- start start 2023-10-30T12:35:16.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.933+0000 7fb4703f5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb4680c6fe0 0x7fb468071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:16.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.933+0000 7fb4703f5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb4680c5f60 0x7fb4680cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:16.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.933+0000 7fb4703f5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4680700b0 con 0x7fb468153bf0 2023-10-30T12:35:16.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.933+0000 7fb4703f5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4680c6810 con 0x7fb4680c6fe0 2023-10-30T12:35:16.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.933+0000 7fb4703f5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb468159900 con 0x7fb4680c5f60 2023-10-30T12:35:16.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.933+0000 7fb46d990640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb4680c6fe0 0x7fb468071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:16.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.933+0000 7fb46e191640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb468153bf0 0x7fb4680c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33850/0 (socket says 172.21.15.125:33850) 2023-10-30T12:35:16.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.933+0000 7fb46e191640 1 -- 172.21.15.125:0/3543485500 learned_addr learned my addr 172.21.15.125:0/3543485500 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:35:16.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.933+0000 7fb46e992640 1 --2- 172.21.15.125:0/3543485500 >> v2:172.21.15.136:3300/0 conn(0x7fb4680c5f60 0x7fb4680cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:16.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.937+0000 7fb46d990640 1 -- 172.21.15.125:0/3543485500 >> v2:172.21.15.136:3300/0 conn(0x7fb4680c5f60 msgr2=0x7fb4680cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:16.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.937+0000 7fb46d990640 1 --2- 172.21.15.125:0/3543485500 >> v2:172.21.15.136:3300/0 conn(0x7fb4680c5f60 0x7fb4680cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:16.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.937+0000 7fb46d990640 1 -- 172.21.15.125:0/3543485500 >> v1:172.21.15.125:6789/0 conn(0x7fb468153bf0 legacy=0x7fb4680c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:16.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.937+0000 7fb46d990640 1 -- 172.21.15.125:0/3543485500 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4680c6530 con 0x7fb4680c6fe0 2023-10-30T12:35:16.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.937+0000 7fb46d18f640 1 -- 172.21.15.125:0/3543485500 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279847538 0 0) 0x7fb4680700b0 con 0x7fb468153bf0 2023-10-30T12:35:16.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.937+0000 7fb46d990640 1 --2- 172.21.15.125:0/3543485500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4680c6fe0 0x7fb468071260 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb46806ce90 tx=0x7fb45801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e141b2cbbb8a4fc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:16.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.937+0000 7fb46d18f640 1 -- 172.21.15.125:0/3543485500 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4580a8950 con 0x7fb4680c6fe0 2023-10-30T12:35:16.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.937+0000 7fb46d18f640 1 -- 172.21.15.125:0/3543485500 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4580a8ab0 con 0x7fb4680c6fe0 2023-10-30T12:35:16.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.937+0000 7fb46d18f640 1 -- 172.21.15.125:0/3543485500 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4580a8e40 con 0x7fb4680c6fe0 2023-10-30T12:35:16.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.937+0000 7fb4703f5640 1 -- 172.21.15.125:0/3543485500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4680c6fe0 msgr2=0x7fb468071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:16.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.937+0000 7fb4703f5640 1 --2- 172.21.15.125:0/3543485500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4680c6fe0 0x7fb468071260 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb46806ce90 tx=0x7fb45801ec50 comp rx=0 tx=0).stop 2023-10-30T12:35:16.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 -- 172.21.15.125:0/3543485500 shutdown_connections 2023-10-30T12:35:16.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 --2- 172.21.15.125:0/3543485500 >> v2:172.21.15.136:3300/0 conn(0x7fb4680c5f60 0x7fb4680cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:16.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 --2- 172.21.15.125:0/3543485500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4680c6fe0 0x7fb468071260 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:16.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 -- 172.21.15.125:0/3543485500 >> 172.21.15.125:0/3543485500 conn(0x7fb46806c6e0 msgr2=0x7fb468069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:16.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 -- 172.21.15.125:0/3543485500 shutdown_connections 2023-10-30T12:35:16.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 -- 172.21.15.125:0/3543485500 wait complete. 2023-10-30T12:35:16.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 Processor -- start 2023-10-30T12:35:16.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 -- start start 2023-10-30T12:35:16.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4680c5f60 0x7fb468070eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:16.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4680c6fe0 0x7fb4680c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:16.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb468153bf0 0x7fb4680c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:16.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4681f20f0 con 0x7fb4680c5f60 2023-10-30T12:35:16.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4681f23d0 con 0x7fb468153bf0 2023-10-30T12:35:16.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4681f26b0 con 0x7fb4680c6fe0 2023-10-30T12:35:16.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb46e191640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4680c5f60 0x7fb468070eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:16.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb46e191640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4680c5f60 0x7fb468070eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60902/0 (socket says 172.21.15.125:60902) 2023-10-30T12:35:16.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb46e191640 1 -- 172.21.15.125:0/3407731387 learned_addr learned my addr 172.21.15.125:0/3407731387 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:16.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb46e992640 1 --2- 172.21.15.125:0/3407731387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb468153bf0 0x7fb4680c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:16.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb46d990640 1 --2- 172.21.15.125:0/3407731387 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4680c6fe0 0x7fb4680c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:16.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb46e191640 1 -- 172.21.15.125:0/3407731387 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4680c6fe0 msgr2=0x7fb4680c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:16.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb46e191640 1 --2- 172.21.15.125:0/3407731387 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4680c6fe0 0x7fb4680c36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:16.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb46e191640 1 -- 172.21.15.125:0/3407731387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb468153bf0 msgr2=0x7fb4680c3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:16.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb46e191640 1 --2- 172.21.15.125:0/3407731387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb468153bf0 0x7fb4680c3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:16.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb46e191640 1 -- 172.21.15.125:0/3407731387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb458009420 con 0x7fb4680c5f60 2023-10-30T12:35:16.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb46e191640 1 --2- 172.21.15.125:0/3407731387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4680c5f60 0x7fb468070eb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb46400c480 tx=0x7fb46400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:16.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb45effd640 1 -- 172.21.15.125:0/3407731387 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4640074e0 con 0x7fb4680c5f60 2023-10-30T12:35:16.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb45effd640 1 -- 172.21.15.125:0/3407731387 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb464007660 con 0x7fb4680c5f60 2023-10-30T12:35:16.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb4703f5640 1 -- 172.21.15.125:0/3407731387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4681f2990 con 0x7fb4680c5f60 2023-10-30T12:35:16.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.941+0000 7fb45effd640 1 -- 172.21.15.125:0/3407731387 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb464004a30 con 0x7fb4680c5f60 2023-10-30T12:35:16.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.945+0000 7fb4703f5640 1 -- 172.21.15.125:0/3407731387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46814dc70 con 0x7fb4680c5f60 2023-10-30T12:35:16.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.945+0000 7fb4703f5640 1 -- 172.21.15.125:0/3407731387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4681f2eb0 con 0x7fb4680c5f60 2023-10-30T12:35:16.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.945+0000 7fb45effd640 1 -- 172.21.15.125:0/3407731387 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb464004b90 con 0x7fb4680c5f60 2023-10-30T12:35:16.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.945+0000 7fb45effd640 1 --2- 172.21.15.125:0/3407731387 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb44c0664a0 0x7fb44c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:16.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.945+0000 7fb45effd640 1 -- 172.21.15.125:0/3407731387 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5996+0+0 (secure 0 0 0) 0x7fb464083570 con 0x7fb4680c5f60 2023-10-30T12:35:16.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.945+0000 7fb46d990640 1 --2- 172.21.15.125:0/3407731387 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb44c0664a0 0x7fb44c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:16.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.945+0000 7fb46d990640 1 --2- 172.21.15.125:0/3407731387 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb44c0664a0 0x7fb44c068960 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb4580b2000 tx=0x7fb4580aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:16.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:16.949+0000 7fb45effd640 1 -- 172.21.15.125:0/3407731387 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb4640542c0 con 0x7fb4680c5f60 2023-10-30T12:35:17.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.133+0000 7fb4703f5640 1 -- 172.21.15.125:0/3407731387 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb46804e170 con 0x7fb4680c5f60 2023-10-30T12:35:17.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.137+0000 7fb45effd640 1 -- 172.21.15.125:0/3407731387 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5753 (secure 0 0 0) 0x7fb464053e50 con 0x7fb4680c5f60 2023-10-30T12:35:17.139 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:17.140 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:15.030211+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:17.141 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 37 2023-10-30T12:35:17.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.137+0000 7fb4703f5640 1 -- 172.21.15.125:0/3407731387 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb44c0664a0 msgr2=0x7fb44c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:17.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.137+0000 7fb4703f5640 1 --2- 172.21.15.125:0/3407731387 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb44c0664a0 0x7fb44c068960 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb4580b2000 tx=0x7fb4580aa070 comp rx=0 tx=0).stop 2023-10-30T12:35:17.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.137+0000 7fb4703f5640 1 -- 172.21.15.125:0/3407731387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4680c5f60 msgr2=0x7fb468070eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:17.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.137+0000 7fb4703f5640 1 --2- 172.21.15.125:0/3407731387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4680c5f60 0x7fb468070eb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb46400c480 tx=0x7fb46400c940 comp rx=0 tx=0).stop 2023-10-30T12:35:17.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.137+0000 7fb4703f5640 1 -- 172.21.15.125:0/3407731387 shutdown_connections 2023-10-30T12:35:17.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.137+0000 7fb4703f5640 1 --2- 172.21.15.125:0/3407731387 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb44c0664a0 0x7fb44c068960 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.137+0000 7fb4703f5640 1 --2- 172.21.15.125:0/3407731387 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb468153bf0 0x7fb4680c3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.137+0000 7fb4703f5640 1 --2- 172.21.15.125:0/3407731387 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4680c6fe0 0x7fb4680c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.137+0000 7fb4703f5640 1 --2- 172.21.15.125:0/3407731387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4680c5f60 0x7fb468070eb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.137+0000 7fb4703f5640 1 -- 172.21.15.125:0/3407731387 >> 172.21.15.125:0/3407731387 conn(0x7fb46806c6e0 msgr2=0x7fb468158750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:17.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.137+0000 7fb4703f5640 1 -- 172.21.15.125:0/3407731387 shutdown_connections 2023-10-30T12:35:17.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.137+0000 7fb4703f5640 1 -- 172.21.15.125:0/3407731387 wait complete. 2023-10-30T12:35:17.160 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T12:35:17.160 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:35:17.160 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:17.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6b493e640 1 Processor -- start 2023-10-30T12:35:17.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6b493e640 1 -- start start 2023-10-30T12:35:17.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6b493e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff6b015d6f0 0x7ff6b015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:17.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6b493e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff6b00cf160 0x7ff6b00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:17.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6b493e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6b00cb800 con 0x7ff6b01538a0 2023-10-30T12:35:17.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6b493e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b00cb330 con 0x7ff6b00cf160 2023-10-30T12:35:17.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6b493e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6b00cad70 con 0x7ff6b015d6f0 2023-10-30T12:35:17.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6aed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff6b00cf160 0x7ff6b00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:17.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6aed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff6b00cf160 0x7ff6b00cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60914/0 (socket says 172.21.15.125:60914) 2023-10-30T12:35:17.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6aed9d640 1 -- 172.21.15.125:0/1266361216 learned_addr learned my addr 172.21.15.125:0/1266361216 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:17.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6a5d9b640 1 --2- 172.21.15.125:0/1266361216 >> v2:172.21.15.136:3300/0 conn(0x7ff6b015d6f0 0x7ff6b015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:17.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6aed9d640 1 -- 172.21.15.125:0/1266361216 >> v2:172.21.15.136:3300/0 conn(0x7ff6b015d6f0 msgr2=0x7ff6b015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:17.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6aed9d640 1 --2- 172.21.15.125:0/1266361216 >> v2:172.21.15.136:3300/0 conn(0x7ff6b015d6f0 0x7ff6b015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6aed9d640 1 -- 172.21.15.125:0/1266361216 >> v1:172.21.15.136:6789/0 conn(0x7ff6b01538a0 legacy=0x7ff6b00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:17.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.297+0000 7ff6aed9d640 1 -- 172.21.15.125:0/1266361216 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b00cb050 con 0x7ff6b00cf160 2023-10-30T12:35:17.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.301+0000 7ff6aed9d640 1 --2- 172.21.15.125:0/1266361216 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6b00cf160 0x7ff6b00cf590 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00b30a0 tx=0x7ff6a801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30832c4fa598328a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:17.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.301+0000 7ff6add9b640 1 -- 172.21.15.125:0/1266361216 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a80a89b0 con 0x7ff6b00cf160 2023-10-30T12:35:17.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.301+0000 7ff6add9b640 1 -- 172.21.15.125:0/1266361216 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a80a8b10 con 0x7ff6b00cf160 2023-10-30T12:35:17.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.301+0000 7ff6add9b640 1 -- 172.21.15.125:0/1266361216 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a80a8ea0 con 0x7ff6b00cf160 2023-10-30T12:35:17.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.301+0000 7ff6b493e640 1 -- 172.21.15.125:0/1266361216 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6b00cf160 msgr2=0x7ff6b00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:17.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.301+0000 7ff6b493e640 1 --2- 172.21.15.125:0/1266361216 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6b00cf160 0x7ff6b00cf590 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff6b00b30a0 tx=0x7ff6a801ecb0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.301+0000 7ff6b493e640 1 -- 172.21.15.125:0/1266361216 shutdown_connections 2023-10-30T12:35:17.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.301+0000 7ff6b493e640 1 --2- 172.21.15.125:0/1266361216 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6b00cf160 0x7ff6b00cf590 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.301+0000 7ff6b493e640 1 --2- 172.21.15.125:0/1266361216 >> v2:172.21.15.136:3300/0 conn(0x7ff6b015d6f0 0x7ff6b015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.301+0000 7ff6b493e640 1 -- 172.21.15.125:0/1266361216 >> 172.21.15.125:0/1266361216 conn(0x7ff6b006bfe0 msgr2=0x7ff6b006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:17.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.301+0000 7ff6b493e640 1 -- 172.21.15.125:0/1266361216 shutdown_connections 2023-10-30T12:35:17.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.301+0000 7ff6b493e640 1 -- 172.21.15.125:0/1266361216 wait complete. 2023-10-30T12:35:17.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6b493e640 1 Processor -- start 2023-10-30T12:35:17.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6b493e640 1 -- start start 2023-10-30T12:35:17.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6b493e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6b00cf160 0x7ff6b01ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:17.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6b493e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6b01538a0 0x7ff6b00b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:17.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6b493e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6b015d6f0 0x7ff6b01e3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:17.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6b493e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01e4e10 con 0x7ff6b015d6f0 2023-10-30T12:35:17.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6b493e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b014d680 con 0x7ff6b01538a0 2023-10-30T12:35:17.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6b493e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6b01e2a40 con 0x7ff6b00cf160 2023-10-30T12:35:17.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6aed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6b015d6f0 0x7ff6b01e3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:17.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6aed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6b015d6f0 0x7ff6b01e3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60926/0 (socket says 172.21.15.125:60926) 2023-10-30T12:35:17.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6aed9d640 1 -- 172.21.15.125:0/2881357864 learned_addr learned my addr 172.21.15.125:0/2881357864 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:17.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6a5d9b640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6b01538a0 0x7ff6b00b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:17.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6ae59c640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6b00cf160 0x7ff6b01ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:17.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6aed9d640 1 -- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6b00cf160 msgr2=0x7ff6b01ef890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:17.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6aed9d640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6b00cf160 0x7ff6b01ef890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6aed9d640 1 -- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6b01538a0 msgr2=0x7ff6b00b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:17.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6aed9d640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6b01538a0 0x7ff6b00b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6aed9d640 1 -- 172.21.15.125:0/2881357864 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6a80094d0 con 0x7ff6b015d6f0 2023-10-30T12:35:17.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6a5d9b640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6b01538a0 0x7ff6b00b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:17.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6aed9d640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6b015d6f0 0x7ff6b01e3a10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff6a801c8e0 tx=0x7ff6a801ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:17.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6a77fe640 1 -- 172.21.15.125:0/2881357864 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a80b0d70 con 0x7ff6b015d6f0 2023-10-30T12:35:17.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6a77fe640 1 -- 172.21.15.125:0/2881357864 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6a80bb460 con 0x7ff6b015d6f0 2023-10-30T12:35:17.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6b493e640 1 -- 172.21.15.125:0/2881357864 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6b0158f70 con 0x7ff6b015d6f0 2023-10-30T12:35:17.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6a77fe640 1 -- 172.21.15.125:0/2881357864 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6a80bb7d0 con 0x7ff6b015d6f0 2023-10-30T12:35:17.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6b493e640 1 -- 172.21.15.125:0/2881357864 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6b01594f0 con 0x7ff6b015d6f0 2023-10-30T12:35:17.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.305+0000 7ff6b493e640 1 -- 172.21.15.125:0/2881357864 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6b00b2a50 con 0x7ff6b015d6f0 2023-10-30T12:35:17.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.309+0000 7ff6a77fe640 1 -- 172.21.15.125:0/2881357864 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff6a80bb930 con 0x7ff6b015d6f0 2023-10-30T12:35:17.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.309+0000 7ff6a77fe640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6900664a0 0x7ff690068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:17.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.309+0000 7ff6a77fe640 1 -- 172.21.15.125:0/2881357864 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5996+0+0 (secure 0 0 0) 0x7ff6a8121ab0 con 0x7ff6b015d6f0 2023-10-30T12:35:17.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.309+0000 7ff6ae59c640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6900664a0 0x7ff690068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:17.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.309+0000 7ff6ae59c640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6900664a0 0x7ff690068960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0005530 tx=0x7ff6a0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:17.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.313+0000 7ff6a77fe640 1 -- 172.21.15.125:0/2881357864 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff6a80eee20 con 0x7ff6b015d6f0 2023-10-30T12:35:17.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.493+0000 7ff6b493e640 1 -- 172.21.15.125:0/2881357864 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6b006c3c0 con 0x7ff6b015d6f0 2023-10-30T12:35:17.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.493+0000 7ff6a77fe640 1 -- 172.21.15.125:0/2881357864 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5753 (secure 0 0 0) 0x7ff6a80b5020 con 0x7ff6b015d6f0 2023-10-30T12:35:17.498 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:17.498 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:15.030211+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:17.499 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 37 2023-10-30T12:35:17.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.497+0000 7ff6b493e640 1 -- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6900664a0 msgr2=0x7ff690068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:17.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.497+0000 7ff6b493e640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6900664a0 0x7ff690068960 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0005530 tx=0x7ff6a0009040 comp rx=0 tx=0).stop 2023-10-30T12:35:17.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.497+0000 7ff6b493e640 1 -- 172.21.15.125:0/2881357864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6b015d6f0 msgr2=0x7ff6b01e3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:17.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.497+0000 7ff6b493e640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6b015d6f0 0x7ff6b01e3a10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ff6a801c8e0 tx=0x7ff6a801ba50 comp rx=0 tx=0).stop 2023-10-30T12:35:17.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.497+0000 7ff6b493e640 1 -- 172.21.15.125:0/2881357864 shutdown_connections 2023-10-30T12:35:17.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.497+0000 7ff6b493e640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6900664a0 0x7ff690068960 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.497+0000 7ff6b493e640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6b015d6f0 0x7ff6b01e3a10 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.497+0000 7ff6b493e640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6b01538a0 0x7ff6b00b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.497+0000 7ff6b493e640 1 --2- 172.21.15.125:0/2881357864 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6b00cf160 0x7ff6b01ef890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:17.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.497+0000 7ff6b493e640 1 -- 172.21.15.125:0/2881357864 >> 172.21.15.125:0/2881357864 conn(0x7ff6b006bfe0 msgr2=0x7ff6b00c4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:17.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.497+0000 7ff6b493e640 1 -- 172.21.15.125:0/2881357864 shutdown_connections 2023-10-30T12:35:17.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:17.497+0000 7ff6b493e640 1 -- 172.21.15.125:0/2881357864 wait complete. 2023-10-30T12:35:17.519 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 37, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 37, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4294": { "addr": "172.21.15.125:6836/2017382874", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2017382874, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2017382874, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4294, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4957": { "addr": "172.21.15.136:6835/2548530941", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2548530941, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2548530941, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4957, "incarnation": 23, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 34 }, "gid_5130": { "addr": "172.21.15.125:6837/1624923396", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 1624923396, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 1624923396, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5130, "incarnation": 37, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 42, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:35:15.030211+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4294, "mds_1": 5130, "mds_2": 4957 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:35:19.520 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:19.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.661+0000 7ff806617640 1 Processor -- start 2023-10-30T12:35:19.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.661+0000 7ff806617640 1 -- start start 2023-10-30T12:35:19.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff806617640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff800159e60 0x7ff80006f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:19.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff806617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8000bfec0 0x7ff8000c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:19.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff806617640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff80006a720 0x7ff80006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:19.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff806617640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff80006fa80 con 0x7ff80006a720 2023-10-30T12:35:19.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff806617640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff800071290 con 0x7ff800159e60 2023-10-30T12:35:19.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff806617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff800071570 con 0x7ff8000bfec0 2023-10-30T12:35:19.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff804bb4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff80006a720 0x7ff80006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:19.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff804bb4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff80006a720 0x7ff80006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60930/0 (socket says 172.21.15.125:60930) 2023-10-30T12:35:19.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff804bb4640 1 -- 172.21.15.125:0/588854323 learned_addr learned my addr 172.21.15.125:0/588854323 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:19.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff7fffff640 1 --2- 172.21.15.125:0/588854323 >> v2:172.21.15.136:3300/0 conn(0x7ff800159e60 0x7ff80006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:19.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff7ff7fe640 1 --2- 172.21.15.125:0/588854323 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8000bfec0 0x7ff8000c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:19.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff804bb4640 1 -- 172.21.15.125:0/588854323 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8000bfec0 msgr2=0x7ff8000c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:19.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff804bb4640 1 --2- 172.21.15.125:0/588854323 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8000bfec0 0x7ff8000c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff804bb4640 1 -- 172.21.15.125:0/588854323 >> v2:172.21.15.136:3300/0 conn(0x7ff800159e60 msgr2=0x7ff80006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:19.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff804bb4640 1 --2- 172.21.15.125:0/588854323 >> v2:172.21.15.136:3300/0 conn(0x7ff800159e60 0x7ff80006f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.665+0000 7ff804bb4640 1 -- 172.21.15.125:0/588854323 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff800071850 con 0x7ff80006a720 2023-10-30T12:35:19.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff804bb4640 1 --2- 172.21.15.125:0/588854323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80006a720 0x7ff80006abb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff8000c8950 tx=0x7ff7f80213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb9919843257eb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:19.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff7feffd640 1 -- 172.21.15.125:0/588854323 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f800b2d0 con 0x7ff80006a720 2023-10-30T12:35:19.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff7feffd640 1 -- 172.21.15.125:0/588854323 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f80ad050 con 0x7ff80006a720 2023-10-30T12:35:19.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff7feffd640 1 -- 172.21.15.125:0/588854323 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8009820 con 0x7ff80006a720 2023-10-30T12:35:19.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff806617640 1 -- 172.21.15.125:0/588854323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80006a720 msgr2=0x7ff80006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:19.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff806617640 1 --2- 172.21.15.125:0/588854323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80006a720 0x7ff80006abb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff8000c8950 tx=0x7ff7f80213c0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff806617640 1 -- 172.21.15.125:0/588854323 shutdown_connections 2023-10-30T12:35:19.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff806617640 1 --2- 172.21.15.125:0/588854323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80006a720 0x7ff80006abb0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff806617640 1 --2- 172.21.15.125:0/588854323 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8000bfec0 0x7ff8000c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff806617640 1 --2- 172.21.15.125:0/588854323 >> v2:172.21.15.136:3300/0 conn(0x7ff800159e60 0x7ff80006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff806617640 1 -- 172.21.15.125:0/588854323 >> 172.21.15.125:0/588854323 conn(0x7ff8000c7910 msgr2=0x7ff8000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:19.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff806617640 1 -- 172.21.15.125:0/588854323 shutdown_connections 2023-10-30T12:35:19.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff806617640 1 -- 172.21.15.125:0/588854323 wait complete. 2023-10-30T12:35:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff806617640 1 Processor -- start 2023-10-30T12:35:19.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.669+0000 7ff806617640 1 -- start start 2023-10-30T12:35:19.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff806617640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80006a720 0x7ff8000c5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:19.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff806617640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8000bfec0 0x7ff80014abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:19.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff806617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff800159e60 0x7ff8000c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:19.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff806617640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8001e0c20 con 0x7ff80006a720 2023-10-30T12:35:19.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff806617640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8001e0940 con 0x7ff8000bfec0 2023-10-30T12:35:19.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff806617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8000c2af0 con 0x7ff800159e60 2023-10-30T12:35:19.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff7fffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80006a720 0x7ff8000c5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:19.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff7fffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80006a720 0x7ff8000c5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60940/0 (socket says 172.21.15.125:60940) 2023-10-30T12:35:19.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff7fffff640 1 -- 172.21.15.125:0/4129012419 learned_addr learned my addr 172.21.15.125:0/4129012419 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff7ff7fe640 1 --2- 172.21.15.125:0/4129012419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8000bfec0 0x7ff80014abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff804bb4640 1 --2- 172.21.15.125:0/4129012419 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff800159e60 0x7ff8000c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff7fffff640 1 -- 172.21.15.125:0/4129012419 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff800159e60 msgr2=0x7ff8000c43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff7fffff640 1 --2- 172.21.15.125:0/4129012419 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff800159e60 0x7ff8000c43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff7fffff640 1 -- 172.21.15.125:0/4129012419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8000bfec0 msgr2=0x7ff80014abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff7fffff640 1 --2- 172.21.15.125:0/4129012419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8000bfec0 0x7ff80014abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff7fffff640 1 -- 172.21.15.125:0/4129012419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7f800bd40 con 0x7ff80006a720 2023-10-30T12:35:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff7fffff640 1 --2- 172.21.15.125:0/4129012419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80006a720 0x7ff8000c5500 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff7f000d620 tx=0x7ff7f000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff7fcff9640 1 -- 172.21.15.125:0/4129012419 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f00125f0 con 0x7ff80006a720 2023-10-30T12:35:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff7fcff9640 1 -- 172.21.15.125:0/4129012419 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f0012750 con 0x7ff80006a720 2023-10-30T12:35:19.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff806617640 1 -- 172.21.15.125:0/4129012419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8000c22e0 con 0x7ff80006a720 2023-10-30T12:35:19.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff7fcff9640 1 -- 172.21.15.125:0/4129012419 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f0012ae0 con 0x7ff80006a720 2023-10-30T12:35:19.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff806617640 1 -- 172.21.15.125:0/4129012419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8000c3f60 con 0x7ff80006a720 2023-10-30T12:35:19.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff806617640 1 -- 172.21.15.125:0/4129012419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8000b1090 con 0x7ff80006a720 2023-10-30T12:35:19.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.673+0000 7ff7fcff9640 1 -- 172.21.15.125:0/4129012419 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff7f0012c40 con 0x7ff80006a720 2023-10-30T12:35:19.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.677+0000 7ff7fcff9640 1 --2- 172.21.15.125:0/4129012419 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7ec0664a0 0x7ff7ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:19.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.677+0000 7ff7fcff9640 1 -- 172.21.15.125:0/4129012419 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5996+0+0 (secure 0 0 0) 0x7ff7f0084500 con 0x7ff80006a720 2023-10-30T12:35:19.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.677+0000 7ff7ff7fe640 1 --2- 172.21.15.125:0/4129012419 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7ec0664a0 0x7ff7ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:19.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.677+0000 7ff7ff7fe640 1 --2- 172.21.15.125:0/4129012419 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7ec0664a0 0x7ff7ec068960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff8000c7020 tx=0x7ff7f4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:19.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.681+0000 7ff7fcff9640 1 -- 172.21.15.125:0/4129012419 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff7f0051730 con 0x7ff80006a720 2023-10-30T12:35:19.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.861+0000 7ff806617640 1 -- 172.21.15.125:0/4129012419 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff80004e170 con 0x7ff80006a720 2023-10-30T12:35:19.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.861+0000 7ff7fcff9640 1 -- 172.21.15.125:0/4129012419 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+5753 (secure 0 0 0) 0x7ff7f0007380 con 0x7ff80006a720 2023-10-30T12:35:19.865 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:19.865 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":37,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":37,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:15.030211+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:19.866 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 37 2023-10-30T12:35:19.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.861+0000 7ff806617640 1 -- 172.21.15.125:0/4129012419 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7ec0664a0 msgr2=0x7ff7ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:19.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.861+0000 7ff806617640 1 --2- 172.21.15.125:0/4129012419 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7ec0664a0 0x7ff7ec068960 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff8000c7020 tx=0x7ff7f4008040 comp rx=0 tx=0).stop 2023-10-30T12:35:19.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.861+0000 7ff806617640 1 -- 172.21.15.125:0/4129012419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80006a720 msgr2=0x7ff8000c5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:19.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.861+0000 7ff806617640 1 --2- 172.21.15.125:0/4129012419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80006a720 0x7ff8000c5500 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff7f000d620 tx=0x7ff7f000dae0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.861+0000 7ff806617640 1 -- 172.21.15.125:0/4129012419 shutdown_connections 2023-10-30T12:35:19.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.861+0000 7ff806617640 1 --2- 172.21.15.125:0/4129012419 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7ec0664a0 0x7ff7ec068960 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.861+0000 7ff806617640 1 --2- 172.21.15.125:0/4129012419 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff800159e60 0x7ff8000c43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.861+0000 7ff806617640 1 --2- 172.21.15.125:0/4129012419 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8000bfec0 0x7ff80014abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.861+0000 7ff806617640 1 --2- 172.21.15.125:0/4129012419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff80006a720 0x7ff8000c5500 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:19.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.861+0000 7ff806617640 1 -- 172.21.15.125:0/4129012419 >> 172.21.15.125:0/4129012419 conn(0x7ff8000c7910 msgr2=0x7ff8000c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:19.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.861+0000 7ff806617640 1 -- 172.21.15.125:0/4129012419 shutdown_connections 2023-10-30T12:35:19.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:19.865+0000 7ff806617640 1 -- 172.21.15.125:0/4129012419 wait complete. 2023-10-30T12:35:19.886 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:21.888 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:22.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.029+0000 7fd103901640 1 Processor -- start 2023-10-30T12:35:22.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.029+0000 7fd103901640 1 -- start start 2023-10-30T12:35:22.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd103901640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd0fc1539a0 0x7fd0fc15a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:22.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd103901640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd0fc069e90 0x7fd0fc06a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:22.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd103901640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0fc15a8c0 con 0x7fd0fc070a00 2023-10-30T12:35:22.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd103901640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0fc06fbb0 con 0x7fd0fc069e90 2023-10-30T12:35:22.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd103901640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0fc153d80 con 0x7fd0fc1539a0 2023-10-30T12:35:22.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd101e9e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd0fc069e90 0x7fd0fc06a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:22.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd101e9e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd0fc069e90 0x7fd0fc06a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53830/0 (socket says 172.21.15.125:53830) 2023-10-30T12:35:22.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd101e9e640 1 -- 172.21.15.125:0/504640096 learned_addr learned my addr 172.21.15.125:0/504640096 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:22.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd10169d640 1 --2- 172.21.15.125:0/504640096 >> v2:172.21.15.136:3300/0 conn(0x7fd0fc1539a0 0x7fd0fc15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:22.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd101e9e640 1 -- 172.21.15.125:0/504640096 >> v2:172.21.15.136:3300/0 conn(0x7fd0fc1539a0 msgr2=0x7fd0fc15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:22.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd101e9e640 1 --2- 172.21.15.125:0/504640096 >> v2:172.21.15.136:3300/0 conn(0x7fd0fc1539a0 0x7fd0fc15a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:22.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd101e9e640 1 -- 172.21.15.125:0/504640096 >> v1:172.21.15.136:6789/0 conn(0x7fd0fc070a00 legacy=0x7fd0fc070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:22.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd101e9e640 1 -- 172.21.15.125:0/504640096 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc0c5c50 con 0x7fd0fc069e90 2023-10-30T12:35:22.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd101e9e640 1 --2- 172.21.15.125:0/504640096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0fc069e90 0x7fd0fc06a300 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0c8950 tx=0x7fd0f401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e858a4bb85ef512a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:22.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.033+0000 7fd0f3fff640 1 -- 172.21.15.125:0/504640096 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40a56e0 con 0x7fd0fc069e90 2023-10-30T12:35:22.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd0f3fff640 1 -- 172.21.15.125:0/504640096 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f40a5840 con 0x7fd0fc069e90 2023-10-30T12:35:22.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd0f3fff640 1 -- 172.21.15.125:0/504640096 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40a5bd0 con 0x7fd0fc069e90 2023-10-30T12:35:22.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 -- 172.21.15.125:0/504640096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0fc069e90 msgr2=0x7fd0fc06a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:22.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 --2- 172.21.15.125:0/504640096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0fc069e90 0x7fd0fc06a300 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0c8950 tx=0x7fd0f401c910 comp rx=0 tx=0).stop 2023-10-30T12:35:22.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 -- 172.21.15.125:0/504640096 shutdown_connections 2023-10-30T12:35:22.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 --2- 172.21.15.125:0/504640096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0fc069e90 0x7fd0fc06a300 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:22.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 --2- 172.21.15.125:0/504640096 >> v2:172.21.15.136:3300/0 conn(0x7fd0fc1539a0 0x7fd0fc15a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:22.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 -- 172.21.15.125:0/504640096 >> 172.21.15.125:0/504640096 conn(0x7fd0fc0c7910 msgr2=0x7fd0fc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:22.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 -- 172.21.15.125:0/504640096 shutdown_connections 2023-10-30T12:35:22.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 -- 172.21.15.125:0/504640096 wait complete. 2023-10-30T12:35:22.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 Processor -- start 2023-10-30T12:35:22.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 -- start start 2023-10-30T12:35:22.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0fc069e90 0x7fd0fc143df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:22.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0fc070a00 0x7fd0fc148560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:22.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0fc1539a0 0x7fd0fc1eb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:22.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc1f2ef0 con 0x7fd0fc1539a0 2023-10-30T12:35:22.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc1f2060 con 0x7fd0fc070a00 2023-10-30T12:35:22.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd103901640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc144330 con 0x7fd0fc069e90 2023-10-30T12:35:22.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd101e9e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0fc1539a0 0x7fd0fc1eb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:22.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd101e9e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0fc1539a0 0x7fd0fc1eb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53832/0 (socket says 172.21.15.125:53832) 2023-10-30T12:35:22.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd101e9e640 1 -- 172.21.15.125:0/3810837034 learned_addr learned my addr 172.21.15.125:0/3810837034 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:22.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd100e9c640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0fc070a00 0x7fd0fc148560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:22.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd10169d640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0fc069e90 0x7fd0fc143df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:22.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd101e9e640 1 -- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0fc069e90 msgr2=0x7fd0fc143df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:22.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd101e9e640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0fc069e90 0x7fd0fc143df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:22.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd101e9e640 1 -- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0fc070a00 msgr2=0x7fd0fc148560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:22.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd101e9e640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0fc070a00 0x7fd0fc148560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:22.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd101e9e640 1 -- 172.21.15.125:0/3810837034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0f4009b30 con 0x7fd0fc1539a0 2023-10-30T12:35:22.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.037+0000 7fd10169d640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0fc069e90 0x7fd0fc143df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:22.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.041+0000 7fd101e9e640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0fc1539a0 0x7fd0fc1eb960 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd0f40af040 tx=0x7fd0f40196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:22.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.041+0000 7fd0f1ffb640 1 -- 172.21.15.125:0/3810837034 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40b8670 con 0x7fd0fc1539a0 2023-10-30T12:35:22.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.041+0000 7fd0f1ffb640 1 -- 172.21.15.125:0/3810837034 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f40b87f0 con 0x7fd0fc1539a0 2023-10-30T12:35:22.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.041+0000 7fd0f1ffb640 1 -- 172.21.15.125:0/3810837034 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40b8ba0 con 0x7fd0fc1539a0 2023-10-30T12:35:22.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.041+0000 7fd103901640 1 -- 172.21.15.125:0/3810837034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0fc1f2270 con 0x7fd0fc1539a0 2023-10-30T12:35:22.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.041+0000 7fd103901640 1 -- 172.21.15.125:0/3810837034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0fc1f2830 con 0x7fd0fc1539a0 2023-10-30T12:35:22.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.041+0000 7fd103901640 1 -- 172.21.15.125:0/3810837034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0fc04e170 con 0x7fd0fc1539a0 2023-10-30T12:35:22.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.041+0000 7fd0f1ffb640 1 -- 172.21.15.125:0/3810837034 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0f40b8d00 con 0x7fd0fc1539a0 2023-10-30T12:35:22.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.041+0000 7fd0f1ffb640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0e00664d0 0x7fd0e0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:22.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.041+0000 7fd0f1ffb640 1 -- 172.21.15.125:0/3810837034 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5996+0+0 (secure 0 0 0) 0x7fd0f411f3b0 con 0x7fd0fc1539a0 2023-10-30T12:35:22.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.045+0000 7fd10169d640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0e00664d0 0x7fd0e0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:22.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.045+0000 7fd10169d640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0e00664d0 0x7fd0e0068990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8007a70 tx=0x7fd0f8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:22.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.045+0000 7fd0f1ffb640 1 -- 172.21.15.125:0/3810837034 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd0f40ec850 con 0x7fd0fc1539a0 2023-10-30T12:35:22.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.229+0000 7fd103901640 1 -- 172.21.15.125:0/3810837034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0fc0c72f0 con 0x7fd0fc1539a0 2023-10-30T12:35:22.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.229+0000 7fd0f1ffb640 1 -- 172.21.15.125:0/3810837034 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5755 (secure 0 0 0) 0x7fd0f4009510 con 0x7fd0fc1539a0 2023-10-30T12:35:22.234 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:22.234 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:20.644450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:resolve","state_seq":35,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:22.235 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 38 2023-10-30T12:35:22.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.233+0000 7fd103901640 1 -- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0e00664d0 msgr2=0x7fd0e0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:22.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.233+0000 7fd103901640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0e00664d0 0x7fd0e0068990 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fd0f8007a70 tx=0x7fd0f8008040 comp rx=0 tx=0).stop 2023-10-30T12:35:22.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.233+0000 7fd103901640 1 -- 172.21.15.125:0/3810837034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0fc1539a0 msgr2=0x7fd0fc1eb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:22.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.233+0000 7fd103901640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0fc1539a0 0x7fd0fc1eb960 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fd0f40af040 tx=0x7fd0f40196b0 comp rx=0 tx=0).stop 2023-10-30T12:35:22.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.233+0000 7fd103901640 1 -- 172.21.15.125:0/3810837034 shutdown_connections 2023-10-30T12:35:22.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.233+0000 7fd103901640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0e00664d0 0x7fd0e0068990 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:22.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.233+0000 7fd103901640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0fc1539a0 0x7fd0fc1eb960 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:22.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.233+0000 7fd103901640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0fc070a00 0x7fd0fc148560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:22.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.233+0000 7fd103901640 1 --2- 172.21.15.125:0/3810837034 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0fc069e90 0x7fd0fc143df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:22.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.233+0000 7fd103901640 1 -- 172.21.15.125:0/3810837034 >> 172.21.15.125:0/3810837034 conn(0x7fd0fc0c7910 msgr2=0x7fd0fc14e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:22.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.233+0000 7fd103901640 1 -- 172.21.15.125:0/3810837034 shutdown_connections 2023-10-30T12:35:22.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:22.233+0000 7fd103901640 1 -- 172.21.15.125:0/3810837034 wait complete. 2023-10-30T12:35:22.254 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:24.256 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:24.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.397+0000 7fb137868640 1 Processor -- start 2023-10-30T12:35:24.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.397+0000 7fb137868640 1 -- start start 2023-10-30T12:35:24.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.397+0000 7fb137868640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb13006f7a0 0x7fb13006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:24.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.397+0000 7fb137868640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb1300c6bd0 0x7fb130069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:24.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.397+0000 7fb137868640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb130071110 0x7fb130158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:24.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.397+0000 7fb137868640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1300700c0 con 0x7fb1300c6bd0 2023-10-30T12:35:24.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.397+0000 7fb137868640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13006a3d0 con 0x7fb13006f7a0 2023-10-30T12:35:24.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.397+0000 7fb137868640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb130066cf0 con 0x7fb130071110 2023-10-30T12:35:24.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb134e03640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb1300c6bd0 0x7fb130069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:24.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb135e05640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb130071110 0x7fb130158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:24.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb134e03640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb1300c6bd0 0x7fb130069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53834/0 (socket says 172.21.15.125:53834) 2023-10-30T12:35:24.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb134e03640 1 -- 172.21.15.125:0/836736506 learned_addr learned my addr 172.21.15.125:0/836736506 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:24.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb135604640 1 --2- 172.21.15.125:0/836736506 >> v2:172.21.15.136:3300/0 conn(0x7fb13006f7a0 0x7fb13006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:24.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb134e03640 1 -- 172.21.15.125:0/836736506 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb130071110 msgr2=0x7fb130158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:24.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb134e03640 1 --2- 172.21.15.125:0/836736506 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb130071110 0x7fb130158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb134e03640 1 -- 172.21.15.125:0/836736506 >> v2:172.21.15.136:3300/0 conn(0x7fb13006f7a0 msgr2=0x7fb13006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:24.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb134e03640 1 --2- 172.21.15.125:0/836736506 >> v2:172.21.15.136:3300/0 conn(0x7fb13006f7a0 0x7fb13006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb134e03640 1 -- 172.21.15.125:0/836736506 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1300c70d0 con 0x7fb1300c6bd0 2023-10-30T12:35:24.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb135604640 1 --2- 172.21.15.125:0/836736506 >> v2:172.21.15.136:3300/0 conn(0x7fb13006f7a0 0x7fb13006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:24.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb134e03640 1 --2- 172.21.15.125:0/836736506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1300c6bd0 0x7fb130069e90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb13006c830 tx=0x7fb12001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd171dd03cf3c30b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:24.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb127fff640 1 -- 172.21.15.125:0/836736506 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1200a89b0 con 0x7fb1300c6bd0 2023-10-30T12:35:24.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb127fff640 1 -- 172.21.15.125:0/836736506 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1200a8b10 con 0x7fb1300c6bd0 2023-10-30T12:35:24.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.401+0000 7fb127fff640 1 -- 172.21.15.125:0/836736506 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1200a8ea0 con 0x7fb1300c6bd0 2023-10-30T12:35:24.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 -- 172.21.15.125:0/836736506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1300c6bd0 msgr2=0x7fb130069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:24.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 --2- 172.21.15.125:0/836736506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1300c6bd0 0x7fb130069e90 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb13006c830 tx=0x7fb12001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 -- 172.21.15.125:0/836736506 shutdown_connections 2023-10-30T12:35:24.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 --2- 172.21.15.125:0/836736506 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb130071110 0x7fb130158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 --2- 172.21.15.125:0/836736506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1300c6bd0 0x7fb130069e90 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 --2- 172.21.15.125:0/836736506 >> v2:172.21.15.136:3300/0 conn(0x7fb13006f7a0 0x7fb13006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 -- 172.21.15.125:0/836736506 >> 172.21.15.125:0/836736506 conn(0x7fb1300c7910 msgr2=0x7fb1300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:24.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 -- 172.21.15.125:0/836736506 shutdown_connections 2023-10-30T12:35:24.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 -- 172.21.15.125:0/836736506 wait complete. 2023-10-30T12:35:24.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 Processor -- start 2023-10-30T12:35:24.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 -- start start 2023-10-30T12:35:24.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb13006f7a0 0x7fb1301f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:24.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb130071110 0x7fb130154530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:24.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1300c6bd0 0x7fb1301e5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:24.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1301574b0 con 0x7fb130071110 2023-10-30T12:35:24.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb130155810 con 0x7fb13006f7a0 2023-10-30T12:35:24.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb137868640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1301eeea0 con 0x7fb1300c6bd0 2023-10-30T12:35:24.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb134e03640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb130071110 0x7fb130154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:24.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb134e03640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb130071110 0x7fb130154530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53842/0 (socket says 172.21.15.125:53842) 2023-10-30T12:35:24.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.405+0000 7fb134e03640 1 -- 172.21.15.125:0/828085095 learned_addr learned my addr 172.21.15.125:0/828085095 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:24.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb135604640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb13006f7a0 0x7fb1301f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:24.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb135e05640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1300c6bd0 0x7fb1301e5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:24.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb134e03640 1 -- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1300c6bd0 msgr2=0x7fb1301e5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:24.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb134e03640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1300c6bd0 0x7fb1301e5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb134e03640 1 -- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb13006f7a0 msgr2=0x7fb1301f2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:24.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb134e03640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb13006f7a0 0x7fb1301f2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb134e03640 1 -- 172.21.15.125:0/828085095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1200094d0 con 0x7fb130071110 2023-10-30T12:35:24.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb135e05640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1300c6bd0 0x7fb1301e5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:24.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb134e03640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb130071110 0x7fb130154530 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb120009930 tx=0x7fb12001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:24.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb125ffb640 1 -- 172.21.15.125:0/828085095 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1200c0070 con 0x7fb130071110 2023-10-30T12:35:24.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb125ffb640 1 -- 172.21.15.125:0/828085095 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1200b1be0 con 0x7fb130071110 2023-10-30T12:35:24.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb137868640 1 -- 172.21.15.125:0/828085095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1301ef140 con 0x7fb130071110 2023-10-30T12:35:24.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb125ffb640 1 -- 172.21.15.125:0/828085095 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1200bb440 con 0x7fb130071110 2023-10-30T12:35:24.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb137868640 1 -- 172.21.15.125:0/828085095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1301ef700 con 0x7fb130071110 2023-10-30T12:35:24.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb137868640 1 -- 172.21.15.125:0/828085095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb130070330 con 0x7fb130071110 2023-10-30T12:35:24.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.409+0000 7fb125ffb640 1 -- 172.21.15.125:0/828085095 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb1200b7040 con 0x7fb130071110 2023-10-30T12:35:24.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.413+0000 7fb125ffb640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1140664a0 0x7fb114068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:24.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.413+0000 7fb125ffb640 1 -- 172.21.15.125:0/828085095 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5996+0+0 (secure 0 0 0) 0x7fb120121840 con 0x7fb130071110 2023-10-30T12:35:24.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.413+0000 7fb135604640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1140664a0 0x7fb114068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:24.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.413+0000 7fb135604640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1140664a0 0x7fb114068960 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0045f0 tx=0x7fb12c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:24.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.417+0000 7fb125ffb640 1 -- 172.21.15.125:0/828085095 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb130070330 con 0x7fb130071110 2023-10-30T12:35:24.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.593+0000 7fb137868640 1 -- 172.21.15.125:0/828085095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb130070330 con 0x7fb130071110 2023-10-30T12:35:24.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.597+0000 7fb125ffb640 1 -- 172.21.15.125:0/828085095 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5755 (secure 0 0 0) 0x7fb130070330 con 0x7fb130071110 2023-10-30T12:35:24.600 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:24.600 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:20.644450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:resolve","state_seq":35,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:24.600 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 38 2023-10-30T12:35:24.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.597+0000 7fb137868640 1 -- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1140664a0 msgr2=0x7fb114068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:24.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.597+0000 7fb137868640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1140664a0 0x7fb114068960 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0045f0 tx=0x7fb12c008040 comp rx=0 tx=0).stop 2023-10-30T12:35:24.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.597+0000 7fb137868640 1 -- 172.21.15.125:0/828085095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb130071110 msgr2=0x7fb130154530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:24.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.597+0000 7fb137868640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb130071110 0x7fb130154530 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb120009930 tx=0x7fb12001ba90 comp rx=0 tx=0).stop 2023-10-30T12:35:24.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.597+0000 7fb137868640 1 -- 172.21.15.125:0/828085095 shutdown_connections 2023-10-30T12:35:24.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.597+0000 7fb137868640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1140664a0 0x7fb114068960 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.597+0000 7fb137868640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1300c6bd0 0x7fb1301e5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.597+0000 7fb137868640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb130071110 0x7fb130154530 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.597+0000 7fb137868640 1 --2- 172.21.15.125:0/828085095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb13006f7a0 0x7fb1301f2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:24.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.597+0000 7fb137868640 1 -- 172.21.15.125:0/828085095 >> 172.21.15.125:0/828085095 conn(0x7fb1300c7910 msgr2=0x7fb1300c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:24.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.597+0000 7fb137868640 1 -- 172.21.15.125:0/828085095 shutdown_connections 2023-10-30T12:35:24.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:24.597+0000 7fb137868640 1 -- 172.21.15.125:0/828085095 wait complete. 2023-10-30T12:35:24.620 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:26.622 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:26.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.765+0000 7fb13c29a640 1 Processor -- start 2023-10-30T12:35:26.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.765+0000 7fb13c29a640 1 -- start start 2023-10-30T12:35:26.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.765+0000 7fb13c29a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb1340c6540 0x7fb13415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:26.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.765+0000 7fb13c29a640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1340c5810 con 0x7fb1341580e0 2023-10-30T12:35:26.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.765+0000 7fb13c29a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1340c69b0 con 0x7fb134070740 2023-10-30T12:35:26.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.765+0000 7fb13c29a640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13406acb0 con 0x7fb1340c6540 2023-10-30T12:35:26.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.765+0000 7fb13a036640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb1341580e0 0x7fb134069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35320/0 (socket says 172.21.15.125:35320) 2023-10-30T12:35:26.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.765+0000 7fb13a036640 1 -- 172.21.15.125:0/2268854519 learned_addr learned my addr 172.21.15.125:0/2268854519 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:35:26.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.765+0000 7fb13a837640 1 --2- 172.21.15.125:0/2268854519 >> v2:172.21.15.125:3300/0 conn(0x7fb1340c6540 0x7fb13415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:26.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.765+0000 7fb13a837640 1 -- 172.21.15.125:0/2268854519 >> v1:172.21.15.136:6789/0 conn(0x7fb134070740 legacy=0x7fb13406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:26.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.765+0000 7fb13a837640 1 -- 172.21.15.125:0/2268854519 >> v1:172.21.15.125:6789/0 conn(0x7fb1341580e0 legacy=0x7fb134069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:26.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.765+0000 7fb13a837640 1 -- 172.21.15.125:0/2268854519 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1340c6bb0 con 0x7fb1340c6540 2023-10-30T12:35:26.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.769+0000 7fb13a837640 1 --2- 172.21.15.125:0/2268854519 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340c6540 0x7fb13415e700 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009d90 tx=0x7fb12c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1d4bd1b244c13f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:26.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.769+0000 7fb139034640 1 -- 172.21.15.125:0/2268854519 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0a5920 con 0x7fb1340c6540 2023-10-30T12:35:26.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.769+0000 7fb139034640 1 -- 172.21.15.125:0/2268854519 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12c0a5a80 con 0x7fb1340c6540 2023-10-30T12:35:26.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.769+0000 7fb139034640 1 -- 172.21.15.125:0/2268854519 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c0a5e10 con 0x7fb1340c6540 2023-10-30T12:35:26.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.769+0000 7fb13c29a640 1 -- 172.21.15.125:0/2268854519 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340c6540 msgr2=0x7fb13415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:26.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.769+0000 7fb13c29a640 1 --2- 172.21.15.125:0/2268854519 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340c6540 0x7fb13415e700 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009d90 tx=0x7fb12c01c950 comp rx=0 tx=0).stop 2023-10-30T12:35:26.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.769+0000 7fb13c29a640 1 -- 172.21.15.125:0/2268854519 shutdown_connections 2023-10-30T12:35:26.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.769+0000 7fb13c29a640 1 --2- 172.21.15.125:0/2268854519 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340c6540 0x7fb13415e700 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.769+0000 7fb13c29a640 1 -- 172.21.15.125:0/2268854519 >> 172.21.15.125:0/2268854519 conn(0x7fb1340c7910 msgr2=0x7fb1340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:26.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.769+0000 7fb13c29a640 1 -- 172.21.15.125:0/2268854519 shutdown_connections 2023-10-30T12:35:26.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.769+0000 7fb13c29a640 1 -- 172.21.15.125:0/2268854519 wait complete. 2023-10-30T12:35:26.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb13c29a640 1 Processor -- start 2023-10-30T12:35:26.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb13c29a640 1 -- start start 2023-10-30T12:35:26.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb13c29a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb134070740 0x7fb134153850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:26.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb13c29a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340c6540 0x7fb13414d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:26.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb13c29a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1341580e0 0x7fb1341603b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:26.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb13c29a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13414d4e0 con 0x7fb1340c6540 2023-10-30T12:35:26.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb13c29a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1341608f0 con 0x7fb1341580e0 2023-10-30T12:35:26.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb13c29a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb134153e40 con 0x7fb134070740 2023-10-30T12:35:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb139835640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340c6540 0x7fb13414d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb139835640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340c6540 0x7fb13414d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53852/0 (socket says 172.21.15.125:53852) 2023-10-30T12:35:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb139835640 1 -- 172.21.15.125:0/663615841 learned_addr learned my addr 172.21.15.125:0/663615841 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb13a036640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb134070740 0x7fb134153850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb13a837640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1341580e0 0x7fb1341603b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:26.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb139835640 1 -- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb134070740 msgr2=0x7fb134153850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:26.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb139835640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb134070740 0x7fb134153850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb139835640 1 -- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1341580e0 msgr2=0x7fb1341603b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:26.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb139835640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1341580e0 0x7fb1341603b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb139835640 1 -- 172.21.15.125:0/663615841 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c009a70 con 0x7fb1340c6540 2023-10-30T12:35:26.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb13a036640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb134070740 0x7fb134153850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:26.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb139835640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340c6540 0x7fb13414d970 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb12400c420 tx=0x7fb12400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:26.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb12affd640 1 -- 172.21.15.125:0/663615841 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1240052d0 con 0x7fb1340c6540 2023-10-30T12:35:26.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb12affd640 1 -- 172.21.15.125:0/663615841 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb124005430 con 0x7fb1340c6540 2023-10-30T12:35:26.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb12affd640 1 -- 172.21.15.125:0/663615841 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb124007260 con 0x7fb1340c6540 2023-10-30T12:35:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb13c29a640 1 -- 172.21.15.125:0/663615841 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb134154100 con 0x7fb1340c6540 2023-10-30T12:35:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb13c29a640 1 -- 172.21.15.125:0/663615841 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13414cbb0 con 0x7fb1340c6540 2023-10-30T12:35:26.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.773+0000 7fb13c29a640 1 -- 172.21.15.125:0/663615841 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13414d690 con 0x7fb1340c6540 2023-10-30T12:35:26.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.777+0000 7fb12affd640 1 -- 172.21.15.125:0/663615841 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb1240073c0 con 0x7fb1340c6540 2023-10-30T12:35:26.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.777+0000 7fb12affd640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1180664a0 0x7fb118068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:26.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.777+0000 7fb12affd640 1 -- 172.21.15.125:0/663615841 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5996+0+0 (secure 0 0 0) 0x7fb1240833b0 con 0x7fb1340c6540 2023-10-30T12:35:26.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.777+0000 7fb13a036640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1180664a0 0x7fb118068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:26.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.777+0000 7fb13a036640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1180664a0 0x7fb118068960 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb130005890 tx=0x7fb130009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:26.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.781+0000 7fb12affd640 1 -- 172.21.15.125:0/663615841 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb124050710 con 0x7fb1340c6540 2023-10-30T12:35:26.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.961+0000 7fb13c29a640 1 -- 172.21.15.125:0/663615841 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb13404e170 con 0x7fb1340c6540 2023-10-30T12:35:26.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.961+0000 7fb12affd640 1 -- 172.21.15.125:0/663615841 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+5755 (secure 0 0 0) 0x7fb124053d30 con 0x7fb1340c6540 2023-10-30T12:35:26.965 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:26.965 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:20.644450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:resolve","state_seq":35,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:26.967 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 38 2023-10-30T12:35:26.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.961+0000 7fb13c29a640 1 -- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1180664a0 msgr2=0x7fb118068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:26.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.961+0000 7fb13c29a640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1180664a0 0x7fb118068960 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb130005890 tx=0x7fb130009040 comp rx=0 tx=0).stop 2023-10-30T12:35:26.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.961+0000 7fb13c29a640 1 -- 172.21.15.125:0/663615841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340c6540 msgr2=0x7fb13414d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:26.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.961+0000 7fb13c29a640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340c6540 0x7fb13414d970 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb12400c420 tx=0x7fb12400c8e0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.961+0000 7fb13c29a640 1 -- 172.21.15.125:0/663615841 shutdown_connections 2023-10-30T12:35:26.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.961+0000 7fb13c29a640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1180664a0 0x7fb118068960 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.961+0000 7fb13c29a640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1341580e0 0x7fb1341603b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.961+0000 7fb13c29a640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1340c6540 0x7fb13414d970 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.961+0000 7fb13c29a640 1 --2- 172.21.15.125:0/663615841 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb134070740 0x7fb134153850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:26.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.961+0000 7fb13c29a640 1 -- 172.21.15.125:0/663615841 >> 172.21.15.125:0/663615841 conn(0x7fb1340c7910 msgr2=0x7fb134156820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:26.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.961+0000 7fb13c29a640 1 -- 172.21.15.125:0/663615841 shutdown_connections 2023-10-30T12:35:26.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:26.961+0000 7fb13c29a640 1 -- 172.21.15.125:0/663615841 wait complete. 2023-10-30T12:35:26.984 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:28.985 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:29.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46d54ad640 1 Processor -- start 2023-10-30T12:35:29.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46d54ad640 1 -- start start 2023-10-30T12:35:29.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46d54ad640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f46d015cd50 0x7f46d00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:29.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46d54ad640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f46d00cefa0 0x7f46d00cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:29.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46d54ad640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d015a6c0 con 0x7f46d01538f0 2023-10-30T12:35:29.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46d54ad640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d00cf8f0 con 0x7f46d015cd50 2023-10-30T12:35:29.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46d54ad640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d015d300 con 0x7f46d00cefa0 2023-10-30T12:35:29.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46cf7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f46d015cd50 0x7f46d00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:29.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46cf7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f46d015cd50 0x7f46d00c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53856/0 (socket says 172.21.15.125:53856) 2023-10-30T12:35:29.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46cf7fe640 1 -- 172.21.15.125:0/3669949825 learned_addr learned my addr 172.21.15.125:0/3669949825 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:29.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46d4cac640 1 --2- 172.21.15.125:0/3669949825 >> v2:172.21.15.136:3300/0 conn(0x7f46d00cefa0 0x7f46d00cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:29.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46cf7fe640 1 -- 172.21.15.125:0/3669949825 >> v2:172.21.15.136:3300/0 conn(0x7f46d00cefa0 msgr2=0x7f46d00cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:29.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46cf7fe640 1 --2- 172.21.15.125:0/3669949825 >> v2:172.21.15.136:3300/0 conn(0x7f46d00cefa0 0x7f46d00cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:29.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46cf7fe640 1 -- 172.21.15.125:0/3669949825 >> v1:172.21.15.136:6789/0 conn(0x7f46d01538f0 legacy=0x7f46d0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:29.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.129+0000 7f46cf7fe640 1 -- 172.21.15.125:0/3669949825 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d015d5e0 con 0x7f46d015cd50 2023-10-30T12:35:29.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.133+0000 7f46cf7fe640 1 --2- 172.21.15.125:0/3669949825 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d015cd50 0x7f46d00c9f90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f46d0066a40 tx=0x7f46c401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d36c27b17596b72a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:29.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.133+0000 7f46ceffd640 1 -- 172.21.15.125:0/3669949825 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c40a89b0 con 0x7f46d015cd50 2023-10-30T12:35:29.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.133+0000 7f46ceffd640 1 -- 172.21.15.125:0/3669949825 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c40a8b10 con 0x7f46d015cd50 2023-10-30T12:35:29.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.133+0000 7f46ceffd640 1 -- 172.21.15.125:0/3669949825 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c40a8ea0 con 0x7f46d015cd50 2023-10-30T12:35:29.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.133+0000 7f46d54ad640 1 -- 172.21.15.125:0/3669949825 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d015cd50 msgr2=0x7f46d00c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:29.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.133+0000 7f46d54ad640 1 --2- 172.21.15.125:0/3669949825 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d015cd50 0x7f46d00c9f90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f46d0066a40 tx=0x7f46c401ecb0 comp rx=0 tx=0).stop 2023-10-30T12:35:29.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.133+0000 7f46d54ad640 1 -- 172.21.15.125:0/3669949825 shutdown_connections 2023-10-30T12:35:29.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.133+0000 7f46d54ad640 1 --2- 172.21.15.125:0/3669949825 >> v2:172.21.15.136:3300/0 conn(0x7f46d00cefa0 0x7f46d00cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:29.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.133+0000 7f46d54ad640 1 --2- 172.21.15.125:0/3669949825 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d015cd50 0x7f46d00c9f90 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:29.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.133+0000 7f46d54ad640 1 -- 172.21.15.125:0/3669949825 >> 172.21.15.125:0/3669949825 conn(0x7f46d006a640 msgr2=0x7f46d0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:29.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.133+0000 7f46d54ad640 1 -- 172.21.15.125:0/3669949825 shutdown_connections 2023-10-30T12:35:29.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.133+0000 7f46d54ad640 1 -- 172.21.15.125:0/3669949825 wait complete. 2023-10-30T12:35:29.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.133+0000 7f46d54ad640 1 Processor -- start 2023-10-30T12:35:29.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46d54ad640 1 -- start start 2023-10-30T12:35:29.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46d54ad640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d00cefa0 0x7f46d00b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:29.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46d54ad640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46d01538f0 0x7f46d0149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:29.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46d54ad640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46d015cd50 0x7f46d00b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:29.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46d54ad640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d00b3960 con 0x7f46d00cefa0 2023-10-30T12:35:29.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46d54ad640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d0149980 con 0x7f46d01538f0 2023-10-30T12:35:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46d54ad640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d00b48f0 con 0x7f46d015cd50 2023-10-30T12:35:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46cffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d00cefa0 0x7f46d00b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46cffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d00cefa0 0x7f46d00b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53866/0 (socket says 172.21.15.125:53866) 2023-10-30T12:35:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46cffff640 1 -- 172.21.15.125:0/1485875809 learned_addr learned my addr 172.21.15.125:0/1485875809 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:29.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46cf7fe640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46d01538f0 0x7f46d0149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:29.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46d4cac640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46d015cd50 0x7f46d00b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:29.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46cffff640 1 -- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46d015cd50 msgr2=0x7f46d00b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:29.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46cffff640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46d015cd50 0x7f46d00b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:29.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46cffff640 1 -- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46d01538f0 msgr2=0x7f46d0149440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:29.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46cffff640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46d01538f0 0x7f46d0149440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46cffff640 1 -- 172.21.15.125:0/1485875809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c40094d0 con 0x7f46d00cefa0 2023-10-30T12:35:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46cf7fe640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46d01538f0 0x7f46d0149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46cffff640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d00cefa0 0x7f46d00b4ed0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f46c000c450 tx=0x7f46c000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46ccff9640 1 -- 172.21.15.125:0/1485875809 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0012830 con 0x7f46d00cefa0 2023-10-30T12:35:29.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46ccff9640 1 -- 172.21.15.125:0/1485875809 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c00129b0 con 0x7f46d00cefa0 2023-10-30T12:35:29.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46d54ad640 1 -- 172.21.15.125:0/1485875809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d01f32a0 con 0x7f46d00cefa0 2023-10-30T12:35:29.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46ccff9640 1 -- 172.21.15.125:0/1485875809 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0012d60 con 0x7f46d00cefa0 2023-10-30T12:35:29.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46d54ad640 1 -- 172.21.15.125:0/1485875809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d00c65f0 con 0x7f46d00cefa0 2023-10-30T12:35:29.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.137+0000 7f46d54ad640 1 -- 172.21.15.125:0/1485875809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d00c67e0 con 0x7f46d00cefa0 2023-10-30T12:35:29.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.141+0000 7f46ccff9640 1 -- 172.21.15.125:0/1485875809 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f46c001c460 con 0x7f46d00cefa0 2023-10-30T12:35:29.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.141+0000 7f46ccff9640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46bc0664a0 0x7f46bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:29.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.141+0000 7f46ccff9640 1 -- 172.21.15.125:0/1485875809 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5996+0+0 (secure 0 0 0) 0x7f46c0083e50 con 0x7f46d00cefa0 2023-10-30T12:35:29.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.141+0000 7f46cf7fe640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46bc0664a0 0x7f46bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:29.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.141+0000 7f46cf7fe640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46bc0664a0 0x7f46bc068960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f46c40b2000 tx=0x7f46c4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:29.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.145+0000 7f46ccff9640 1 -- 172.21.15.125:0/1485875809 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f46c0051110 con 0x7f46d00cefa0 2023-10-30T12:35:29.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.329+0000 7f46d54ad640 1 -- 172.21.15.125:0/1485875809 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46d004e170 con 0x7f46d00cefa0 2023-10-30T12:35:29.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.329+0000 7f46ccff9640 1 -- 172.21.15.125:0/1485875809 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f46c0054730 con 0x7f46d00cefa0 2023-10-30T12:35:29.333 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:29.333 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:26.229795+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:29.334 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 39 2023-10-30T12:35:29.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.329+0000 7f46d54ad640 1 -- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46bc0664a0 msgr2=0x7f46bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:29.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.329+0000 7f46d54ad640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46bc0664a0 0x7f46bc068960 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f46c40b2000 tx=0x7f46c4008040 comp rx=0 tx=0).stop 2023-10-30T12:35:29.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.329+0000 7f46d54ad640 1 -- 172.21.15.125:0/1485875809 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d00cefa0 msgr2=0x7f46d00b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:29.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.329+0000 7f46d54ad640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d00cefa0 0x7f46d00b4ed0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f46c000c450 tx=0x7f46c000c910 comp rx=0 tx=0).stop 2023-10-30T12:35:29.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.329+0000 7f46d54ad640 1 -- 172.21.15.125:0/1485875809 shutdown_connections 2023-10-30T12:35:29.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.329+0000 7f46d54ad640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46bc0664a0 0x7f46bc068960 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:29.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.329+0000 7f46d54ad640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46d015cd50 0x7f46d00b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:29.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.329+0000 7f46d54ad640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46d01538f0 0x7f46d0149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:29.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.329+0000 7f46d54ad640 1 --2- 172.21.15.125:0/1485875809 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46d00cefa0 0x7f46d00b4ed0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:29.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.329+0000 7f46d54ad640 1 -- 172.21.15.125:0/1485875809 >> 172.21.15.125:0/1485875809 conn(0x7f46d006a640 msgr2=0x7f46d00c57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:29.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.329+0000 7f46d54ad640 1 -- 172.21.15.125:0/1485875809 shutdown_connections 2023-10-30T12:35:29.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:29.329+0000 7f46d54ad640 1 -- 172.21.15.125:0/1485875809 wait complete. 2023-10-30T12:35:29.354 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 39, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 39, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4294": { "addr": "172.21.15.125:6836/2017382874", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2017382874, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2017382874, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 4294, "incarnation": 4, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 4 }, "gid_4957": { "addr": "172.21.15.136:6835/2548530941", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2548530941, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2548530941, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4957, "incarnation": 23, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 34 }, "gid_5130": { "addr": "172.21.15.125:6837/1624923396", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 1624923396, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 1624923396, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5130, "incarnation": 37, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:reconnect", "state_seq": 37 } }, "last_failure": 0, "last_failure_osd_epoch": 42, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:35:26.229795+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4294, "mds_1": 5130, "mds_2": 4957 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:35:31.355 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:31.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.497+0000 7f5928a54640 1 Processor -- start 2023-10-30T12:35:31.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.497+0000 7f5928a54640 1 -- start start 2023-10-30T12:35:31.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.497+0000 7f5928a54640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59240c6fe0 0x7f5924071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:31.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.497+0000 7f5928a54640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59240700b0 con 0x7f59240c5f60 2023-10-30T12:35:31.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.497+0000 7f5928a54640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59240c6530 con 0x7f5924153bf0 2023-10-30T12:35:31.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.497+0000 7f5928a54640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59240c6810 con 0x7f59240c6fe0 2023-10-30T12:35:31.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.497+0000 7f5922d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f59240c5f60 0x7f59240cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45856/0 (socket says 172.21.15.125:45856) 2023-10-30T12:35:31.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.497+0000 7f5922d9d640 1 -- 172.21.15.125:0/1676268922 learned_addr learned my addr 172.21.15.125:0/1676268922 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:35:31.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.497+0000 7f5919d9b640 1 --2- 172.21.15.125:0/1676268922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59240c6fe0 0x7f5924071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:31.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.497+0000 7f5921d9b640 1 -- 172.21.15.125:0/1676268922 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879279702 0 0) 0x7f59240700b0 con 0x7f59240c5f60 2023-10-30T12:35:31.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5921d9b640 1 -- 172.21.15.125:0/1676268922 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5910003530 con 0x7f59240c5f60 2023-10-30T12:35:31.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5921d9b640 1 -- 172.21.15.125:0/1676268922 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2109893293 0 0) 0x7f5910003530 con 0x7f59240c5f60 2023-10-30T12:35:31.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5921d9b640 1 -- 172.21.15.125:0/1676268922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59240c6fe0 msgr2=0x7f5924071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:31.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5921d9b640 1 --2- 172.21.15.125:0/1676268922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59240c6fe0 0x7f5924071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5921d9b640 1 -- 172.21.15.125:0/1676268922 >> v1:172.21.15.136:6789/0 conn(0x7f5924153bf0 legacy=0x7f59240c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:31.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5921d9b640 1 -- 172.21.15.125:0/1676268922 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5924159b10 con 0x7f59240c5f60 2023-10-30T12:35:31.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5921d9b640 1 -- 172.21.15.125:0/1676268922 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f591c001a00 con 0x7f59240c5f60 2023-10-30T12:35:31.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5921d9b640 1 -- 172.21.15.125:0/1676268922 >> v1:172.21.15.125:6789/0 conn(0x7f59240c5f60 legacy=0x7f59240cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:31.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5921d9b640 1 --2- 172.21.15.125:0/1676268922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59100039b0 0x7f5910003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:31.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5921d9b640 1 --2- 172.21.15.125:0/1676268922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59240c6fe0 0x7f59100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:31.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5921d9b640 1 --2- 172.21.15.125:0/1676268922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5910004d10 0x7f5910005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:31.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5921d9b640 1 -- 172.21.15.125:0/1676268922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59240c6810 con 0x7f59100039b0 2023-10-30T12:35:31.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5921d9b640 1 -- 172.21.15.125:0/1676268922 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591000b990 con 0x7f5910004d10 2023-10-30T12:35:31.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5921d9b640 1 -- 172.21.15.125:0/1676268922 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591000bb50 con 0x7f59240c6fe0 2023-10-30T12:35:31.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f592259c640 1 --2- 172.21.15.125:0/1676268922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59100039b0 0x7f5910003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:31.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5922d9d640 1 --2- 172.21.15.125:0/1676268922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5910004d10 0x7f5910005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:31.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f5919d9b640 1 --2- 172.21.15.125:0/1676268922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59240c6fe0 0x7f59100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:31.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f592259c640 1 -- 172.21.15.125:0/1676268922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59240c6fe0 msgr2=0x7f59100047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:31.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f592259c640 1 --2- 172.21.15.125:0/1676268922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f59240c6fe0 0x7f59100047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f592259c640 1 -- 172.21.15.125:0/1676268922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5910004d10 msgr2=0x7f5910005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:31.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f592259c640 1 --2- 172.21.15.125:0/1676268922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5910004d10 0x7f5910005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f592259c640 1 -- 172.21.15.125:0/1676268922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5910003530 con 0x7f59100039b0 2023-10-30T12:35:31.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.501+0000 7f592259c640 1 --2- 172.21.15.125:0/1676268922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59100039b0 0x7f5910003d90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f590c001860 tx=0x7f590c01ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b0b8b4a79d1eff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:31.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5921d9b640 1 -- 172.21.15.125:0/1676268922 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c0a87d0 con 0x7f59100039b0 2023-10-30T12:35:31.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5921d9b640 1 -- 172.21.15.125:0/1676268922 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590c0a8930 con 0x7f59100039b0 2023-10-30T12:35:31.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5921d9b640 1 -- 172.21.15.125:0/1676268922 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590c0a8ce0 con 0x7f59100039b0 2023-10-30T12:35:31.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5928a54640 1 -- 172.21.15.125:0/1676268922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59100039b0 msgr2=0x7f5910003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:31.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5928a54640 1 --2- 172.21.15.125:0/1676268922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59100039b0 0x7f5910003d90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f590c001860 tx=0x7f590c01ebb0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5922d9d640 1 -- 172.21.15.125:0/1676268922 reap_dead start 2023-10-30T12:35:31.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5928a54640 1 -- 172.21.15.125:0/1676268922 shutdown_connections 2023-10-30T12:35:31.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5928a54640 1 -- 172.21.15.125:0/1676268922 >> 172.21.15.125:0/1676268922 conn(0x7f592406c6e0 msgr2=0x7f5924069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:31.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5928a54640 1 -- 172.21.15.125:0/1676268922 shutdown_connections 2023-10-30T12:35:31.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5928a54640 1 -- 172.21.15.125:0/1676268922 wait complete. 2023-10-30T12:35:31.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5928a54640 1 Processor -- start 2023-10-30T12:35:31.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5928a54640 1 -- start start 2023-10-30T12:35:31.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5928a54640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59100039b0 0x7f59241513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:31.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5928a54640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f592414fce0 0x7f592414c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:31.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5928a54640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f592414e660 0x7f592414a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:31.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5928a54640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5924150790 con 0x7f592414fce0 2023-10-30T12:35:31.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5928a54640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59241ef4e0 con 0x7f59100039b0 2023-10-30T12:35:31.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.505+0000 7f5928a54640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59241504b0 con 0x7f592414e660 2023-10-30T12:35:31.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f5919d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f592414fce0 0x7f592414c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:31.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f5919d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f592414fce0 0x7f592414c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41978/0 (socket says 172.21.15.125:41978) 2023-10-30T12:35:31.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f5919d9b640 1 -- 172.21.15.125:0/295763701 learned_addr learned my addr 172.21.15.125:0/295763701 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:31.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f592259c640 1 --2- 172.21.15.125:0/295763701 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59100039b0 0x7f59241513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:31.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f5922d9d640 1 --2- 172.21.15.125:0/295763701 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f592414e660 0x7f592414a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:31.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f5919d9b640 1 -- 172.21.15.125:0/295763701 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f592414e660 msgr2=0x7f592414a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:31.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f5919d9b640 1 --2- 172.21.15.125:0/295763701 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f592414e660 0x7f592414a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f5919d9b640 1 -- 172.21.15.125:0/295763701 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59100039b0 msgr2=0x7f59241513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:31.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f5919d9b640 1 --2- 172.21.15.125:0/295763701 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59100039b0 0x7f59241513b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f5919d9b640 1 -- 172.21.15.125:0/295763701 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59241ee720 con 0x7f592414fce0 2023-10-30T12:35:31.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f5919d9b640 1 --2- 172.21.15.125:0/295763701 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f592414fce0 0x7f592414c600 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f59140115e0 tx=0x7f5914011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:31.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f591b7fe640 1 -- 172.21.15.125:0/295763701 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591400f8f0 con 0x7f592414fce0 2023-10-30T12:35:31.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f591b7fe640 1 -- 172.21.15.125:0/295763701 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591400fa50 con 0x7f592414fce0 2023-10-30T12:35:31.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f591b7fe640 1 -- 172.21.15.125:0/295763701 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f591400fdc0 con 0x7f592414fce0 2023-10-30T12:35:31.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f5928a54640 1 -- 172.21.15.125:0/295763701 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59241eea00 con 0x7f592414fce0 2023-10-30T12:35:31.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f5928a54640 1 -- 172.21.15.125:0/295763701 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59241eefc0 con 0x7f592414fce0 2023-10-30T12:35:31.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f5928a54640 1 -- 172.21.15.125:0/295763701 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59241ef180 con 0x7f592414fce0 2023-10-30T12:35:31.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f591b7fe640 1 -- 172.21.15.125:0/295763701 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5914015460 con 0x7f592414fce0 2023-10-30T12:35:31.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f591b7fe640 1 --2- 172.21.15.125:0/295763701 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59040664d0 0x7f5904068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:31.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.509+0000 7f591b7fe640 1 -- 172.21.15.125:0/295763701 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5996+0+0 (secure 0 0 0) 0x7f5914084510 con 0x7f592414fce0 2023-10-30T12:35:31.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.513+0000 7f592259c640 1 --2- 172.21.15.125:0/295763701 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59040664d0 0x7f5904068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:31.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.513+0000 7f592259c640 1 --2- 172.21.15.125:0/295763701 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59040664d0 0x7f5904068990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f590c0b2000 tx=0x7f590c0aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:31.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.517+0000 7f591b7fe640 1 -- 172.21.15.125:0/295763701 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5914051740 con 0x7f592414fce0 2023-10-30T12:35:31.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.693+0000 7f5928a54640 1 -- 172.21.15.125:0/295763701 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f592404e170 con 0x7f592414fce0 2023-10-30T12:35:31.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.693+0000 7f591b7fe640 1 -- 172.21.15.125:0/295763701 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f5914054d60 con 0x7f592414fce0 2023-10-30T12:35:31.698 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:31.699 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":39,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:26.229795+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:31.699 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 39 2023-10-30T12:35:31.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.697+0000 7f5928a54640 1 -- 172.21.15.125:0/295763701 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59040664d0 msgr2=0x7f5904068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:31.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.697+0000 7f5928a54640 1 --2- 172.21.15.125:0/295763701 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59040664d0 0x7f5904068990 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f590c0b2000 tx=0x7f590c0aa070 comp rx=0 tx=0).stop 2023-10-30T12:35:31.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.697+0000 7f5928a54640 1 -- 172.21.15.125:0/295763701 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f592414fce0 msgr2=0x7f592414c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:31.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.697+0000 7f5928a54640 1 --2- 172.21.15.125:0/295763701 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f592414fce0 0x7f592414c600 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f59140115e0 tx=0x7f5914011aa0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.697+0000 7f5928a54640 1 -- 172.21.15.125:0/295763701 shutdown_connections 2023-10-30T12:35:31.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.697+0000 7f5928a54640 1 --2- 172.21.15.125:0/295763701 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59040664d0 0x7f5904068990 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.697+0000 7f5928a54640 1 --2- 172.21.15.125:0/295763701 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f592414e660 0x7f592414a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.697+0000 7f5928a54640 1 --2- 172.21.15.125:0/295763701 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f592414fce0 0x7f592414c600 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.697+0000 7f5928a54640 1 --2- 172.21.15.125:0/295763701 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59100039b0 0x7f59241513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:31.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.697+0000 7f5928a54640 1 -- 172.21.15.125:0/295763701 >> 172.21.15.125:0/295763701 conn(0x7f592406c6e0 msgr2=0x7f59240c94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:31.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.697+0000 7f5928a54640 1 -- 172.21.15.125:0/295763701 shutdown_connections 2023-10-30T12:35:31.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:31.697+0000 7f5928a54640 1 -- 172.21.15.125:0/295763701 wait complete. 2023-10-30T12:35:31.723 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:33.725 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:33.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f2265f65640 1 Processor -- start 2023-10-30T12:35:33.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f2265f65640 1 -- start start 2023-10-30T12:35:33.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f2265f65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226015d6f0 0x7f226015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:33.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f2265f65640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f22600cf160 0x7f22600cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:33.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f2265f65640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22600cb800 con 0x7f22601538a0 2023-10-30T12:35:33.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f2265f65640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22600cb330 con 0x7f22600cf160 2023-10-30T12:35:33.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f2265f65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22600cad70 con 0x7f226015d6f0 2023-10-30T12:35:33.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f225f7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f22601538a0 0x7f22600ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45864/0 (socket says 172.21.15.125:45864) 2023-10-30T12:35:33.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f225f7fe640 1 -- 172.21.15.125:0/2251108542 learned_addr learned my addr 172.21.15.125:0/2251108542 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:35:33.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f225ffff640 1 --2- 172.21.15.125:0/2251108542 >> v2:172.21.15.125:3300/0 conn(0x7f22600cf160 0x7f22600cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:33.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f225effd640 1 --2- 172.21.15.125:0/2251108542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226015d6f0 0x7f226015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:33.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f225ffff640 1 -- 172.21.15.125:0/2251108542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226015d6f0 msgr2=0x7f226015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:33.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f225ffff640 1 --2- 172.21.15.125:0/2251108542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226015d6f0 0x7f226015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:33.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f225ffff640 1 -- 172.21.15.125:0/2251108542 >> v1:172.21.15.125:6789/0 conn(0x7f22601538a0 legacy=0x7f22600ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:33.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f225e7fc640 1 -- 172.21.15.125:0/2251108542 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355149504 0 0) 0x7f22600cb800 con 0x7f22601538a0 2023-10-30T12:35:33.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.869+0000 7f225ffff640 1 -- 172.21.15.125:0/2251108542 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22600cb050 con 0x7f22600cf160 2023-10-30T12:35:33.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.873+0000 7f225ffff640 1 --2- 172.21.15.125:0/2251108542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22600cf160 0x7f22600cf590 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2258009d90 tx=0x7f225801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=917946d81c516333 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:33.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.873+0000 7f225e7fc640 1 -- 172.21.15.125:0/2251108542 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22580af600 con 0x7f22600cf160 2023-10-30T12:35:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.873+0000 7f225e7fc640 1 -- 172.21.15.125:0/2251108542 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22580ab050 con 0x7f22600cf160 2023-10-30T12:35:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.873+0000 7f225e7fc640 1 -- 172.21.15.125:0/2251108542 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22580af980 con 0x7f22600cf160 2023-10-30T12:35:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.873+0000 7f2265f65640 1 -- 172.21.15.125:0/2251108542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22600cf160 msgr2=0x7f22600cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.873+0000 7f2265f65640 1 --2- 172.21.15.125:0/2251108542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22600cf160 0x7f22600cf590 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2258009d90 tx=0x7f225801f130 comp rx=0 tx=0).stop 2023-10-30T12:35:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.873+0000 7f2265f65640 1 -- 172.21.15.125:0/2251108542 shutdown_connections 2023-10-30T12:35:33.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.873+0000 7f2265f65640 1 --2- 172.21.15.125:0/2251108542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22600cf160 0x7f22600cf590 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:33.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.873+0000 7f2265f65640 1 --2- 172.21.15.125:0/2251108542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226015d6f0 0x7f226015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:33.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.873+0000 7f2265f65640 1 -- 172.21.15.125:0/2251108542 >> 172.21.15.125:0/2251108542 conn(0x7f226006bfe0 msgr2=0x7f226006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:33.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.873+0000 7f2265f65640 1 -- 172.21.15.125:0/2251108542 shutdown_connections 2023-10-30T12:35:33.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.873+0000 7f2265f65640 1 -- 172.21.15.125:0/2251108542 wait complete. 2023-10-30T12:35:33.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f2265f65640 1 Processor -- start 2023-10-30T12:35:33.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f2265f65640 1 -- start start 2023-10-30T12:35:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f2265f65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22600cf160 0x7f22601e2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f2265f65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22601538a0 0x7f22600b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f2265f65640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226015d6f0 0x7f22601e4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f2265f65640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22601e5860 con 0x7f22601538a0 2023-10-30T12:35:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f2265f65640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2260157d60 con 0x7f226015d6f0 2023-10-30T12:35:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f2265f65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2260158690 con 0x7f22600cf160 2023-10-30T12:35:33.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f225effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22601538a0 0x7f22600b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:33.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f225effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22601538a0 0x7f22600b4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42006/0 (socket says 172.21.15.125:42006) 2023-10-30T12:35:33.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f225effd640 1 -- 172.21.15.125:0/1947951330 learned_addr learned my addr 172.21.15.125:0/1947951330 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:33.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f225f7fe640 1 --2- 172.21.15.125:0/1947951330 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22600cf160 0x7f22601e2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:33.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f225ffff640 1 --2- 172.21.15.125:0/1947951330 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226015d6f0 0x7f22601e4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:33.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f225effd640 1 -- 172.21.15.125:0/1947951330 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22600cf160 msgr2=0x7f22601e2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:33.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f225effd640 1 --2- 172.21.15.125:0/1947951330 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22600cf160 0x7f22601e2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:33.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f225effd640 1 -- 172.21.15.125:0/1947951330 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226015d6f0 msgr2=0x7f22601e4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:33.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f225effd640 1 --2- 172.21.15.125:0/1947951330 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226015d6f0 0x7f22601e4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:33.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f225effd640 1 -- 172.21.15.125:0/1947951330 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2258009a70 con 0x7f22601538a0 2023-10-30T12:35:33.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f225effd640 1 --2- 172.21.15.125:0/1947951330 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22601538a0 0x7f22600b4580 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f225400d5f0 tx=0x7f225400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:33.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f2247fff640 1 -- 172.21.15.125:0/1947951330 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22540125f0 con 0x7f22601538a0 2023-10-30T12:35:33.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f2247fff640 1 -- 172.21.15.125:0/1947951330 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2254012750 con 0x7f22601538a0 2023-10-30T12:35:33.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f2247fff640 1 -- 172.21.15.125:0/1947951330 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254012ae0 con 0x7f22601538a0 2023-10-30T12:35:33.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f2265f65640 1 -- 172.21.15.125:0/1947951330 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2260158950 con 0x7f22601538a0 2023-10-30T12:35:33.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f2265f65640 1 -- 172.21.15.125:0/1947951330 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22601574e0 con 0x7f22601538a0 2023-10-30T12:35:33.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.877+0000 7f2265f65640 1 -- 172.21.15.125:0/1947951330 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22600b2a50 con 0x7f22601538a0 2023-10-30T12:35:33.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.881+0000 7f2247fff640 1 -- 172.21.15.125:0/1947951330 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2254012c40 con 0x7f22601538a0 2023-10-30T12:35:33.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.881+0000 7f2247fff640 1 --2- 172.21.15.125:0/1947951330 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f224c0664a0 0x7f224c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:33.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.881+0000 7f2247fff640 1 -- 172.21.15.125:0/1947951330 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5996+0+0 (secure 0 0 0) 0x7f22540844d0 con 0x7f22601538a0 2023-10-30T12:35:33.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.881+0000 7f225f7fe640 1 --2- 172.21.15.125:0/1947951330 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f224c0664a0 0x7f224c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:33.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.881+0000 7f225f7fe640 1 --2- 172.21.15.125:0/1947951330 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f224c0664a0 0x7f224c068960 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2250000f70 tx=0x7f2250007b60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:33.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:33.885+0000 7f2247fff640 1 -- 172.21.15.125:0/1947951330 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2254051790 con 0x7f22601538a0 2023-10-30T12:35:34.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:34.061+0000 7f2265f65640 1 -- 172.21.15.125:0/1947951330 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f226004e170 con 0x7f22601538a0 2023-10-30T12:35:34.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:34.065+0000 7f2247fff640 1 -- 172.21.15.125:0/1947951330 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5754 (secure 0 0 0) 0x7f2254007380 con 0x7f22601538a0 2023-10-30T12:35:34.068 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:34.068 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:32.042041+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:34.068 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 40 2023-10-30T12:35:34.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:34.065+0000 7f2265f65640 1 -- 172.21.15.125:0/1947951330 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f224c0664a0 msgr2=0x7f224c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:34.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:34.065+0000 7f2265f65640 1 --2- 172.21.15.125:0/1947951330 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f224c0664a0 0x7f224c068960 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2250000f70 tx=0x7f2250007b60 comp rx=0 tx=0).stop 2023-10-30T12:35:34.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:34.065+0000 7f2265f65640 1 -- 172.21.15.125:0/1947951330 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22601538a0 msgr2=0x7f22600b4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:34.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:34.065+0000 7f2265f65640 1 --2- 172.21.15.125:0/1947951330 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22601538a0 0x7f22600b4580 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f225400d5f0 tx=0x7f225400dab0 comp rx=0 tx=0).stop 2023-10-30T12:35:34.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:34.065+0000 7f2265f65640 1 -- 172.21.15.125:0/1947951330 shutdown_connections 2023-10-30T12:35:34.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:34.065+0000 7f2265f65640 1 --2- 172.21.15.125:0/1947951330 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f224c0664a0 0x7f224c068960 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:34.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:34.065+0000 7f2265f65640 1 --2- 172.21.15.125:0/1947951330 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226015d6f0 0x7f22601e4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:34.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:34.065+0000 7f2265f65640 1 --2- 172.21.15.125:0/1947951330 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22601538a0 0x7f22600b4580 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:34.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:34.065+0000 7f2265f65640 1 --2- 172.21.15.125:0/1947951330 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22600cf160 0x7f22601e2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:34.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:34.065+0000 7f2265f65640 1 -- 172.21.15.125:0/1947951330 >> 172.21.15.125:0/1947951330 conn(0x7f226006bfe0 msgr2=0x7f2260151a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:34.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:34.065+0000 7f2265f65640 1 -- 172.21.15.125:0/1947951330 shutdown_connections 2023-10-30T12:35:34.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:34.065+0000 7f2265f65640 1 -- 172.21.15.125:0/1947951330 wait complete. 2023-10-30T12:35:34.088 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:36.090 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:36.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.233+0000 7f145ccac640 1 Processor -- start 2023-10-30T12:35:36.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.233+0000 7f145ccac640 1 -- start start 2023-10-30T12:35:36.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.233+0000 7f145ccac640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f145806fc90 0x7f14580c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:36.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.233+0000 7f145ccac640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14581566b0 0x7f145806a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:36.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.233+0000 7f145ccac640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1458156220 con 0x7f145814ff10 2023-10-30T12:35:36.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.233+0000 7f145ccac640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1458156ac0 con 0x7f145806fc90 2023-10-30T12:35:36.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.233+0000 7f145ccac640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1458155bf0 con 0x7f14581566b0 2023-10-30T12:35:36.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.233+0000 7f1456ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f145806fc90 0x7f14580c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:36.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.233+0000 7f1456ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f145806fc90 0x7f14580c54d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42010/0 (socket says 172.21.15.125:42010) 2023-10-30T12:35:36.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.233+0000 7f1456ffd640 1 -- 172.21.15.125:0/3962081075 learned_addr learned my addr 172.21.15.125:0/3962081075 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:36.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.233+0000 7f1457fff640 1 --2- 172.21.15.125:0/3962081075 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14581566b0 0x7f145806a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:36.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f1456ffd640 1 -- 172.21.15.125:0/3962081075 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14581566b0 msgr2=0x7f145806a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:36.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f1456ffd640 1 --2- 172.21.15.125:0/3962081075 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14581566b0 0x7f145806a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:36.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f1456ffd640 1 -- 172.21.15.125:0/3962081075 >> v1:172.21.15.136:6789/0 conn(0x7f145814ff10 legacy=0x7f14580c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:36.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f1456ffd640 1 -- 172.21.15.125:0/3962081075 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1458070110 con 0x7f145806fc90 2023-10-30T12:35:36.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f1456ffd640 1 --2- 172.21.15.125:0/3962081075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f145806fc90 0x7f14580c54d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f144c009870 tx=0x7f144c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6735585ca1a49259 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:36.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f14567fc640 1 -- 172.21.15.125:0/3962081075 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c0ab600 con 0x7f145806fc90 2023-10-30T12:35:36.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f14567fc640 1 -- 172.21.15.125:0/3962081075 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f144c0a7050 con 0x7f145806fc90 2023-10-30T12:35:36.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f14567fc640 1 -- 172.21.15.125:0/3962081075 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f144c0ab980 con 0x7f145806fc90 2023-10-30T12:35:36.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f145ccac640 1 -- 172.21.15.125:0/3962081075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f145806fc90 msgr2=0x7f14580c54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:36.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f145ccac640 1 --2- 172.21.15.125:0/3962081075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f145806fc90 0x7f14580c54d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f144c009870 tx=0x7f144c01c540 comp rx=0 tx=0).stop 2023-10-30T12:35:36.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f145ccac640 1 -- 172.21.15.125:0/3962081075 shutdown_connections 2023-10-30T12:35:36.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f145ccac640 1 --2- 172.21.15.125:0/3962081075 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14581566b0 0x7f145806a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:36.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f145ccac640 1 --2- 172.21.15.125:0/3962081075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f145806fc90 0x7f14580c54d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:36.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f145ccac640 1 -- 172.21.15.125:0/3962081075 >> 172.21.15.125:0/3962081075 conn(0x7f14580c75d0 msgr2=0x7f14580c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:36.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.237+0000 7f145ccac640 1 -- 172.21.15.125:0/3962081075 shutdown_connections 2023-10-30T12:35:36.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f145ccac640 1 -- 172.21.15.125:0/3962081075 wait complete. 2023-10-30T12:35:36.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f145ccac640 1 Processor -- start 2023-10-30T12:35:36.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f145ccac640 1 -- start start 2023-10-30T12:35:36.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f145ccac640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f145806fc90 0x7f1458147b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:36.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f145ccac640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f145814ff10 0x7f14580c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:36.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f145ccac640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14581566b0 0x7f14580c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:36.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f145ccac640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14581484a0 con 0x7f145806fc90 2023-10-30T12:35:36.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f145ccac640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14580c2bc0 con 0x7f145814ff10 2023-10-30T12:35:36.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f145ccac640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1458149600 con 0x7f14581566b0 2023-10-30T12:35:36.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f14577fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f145806fc90 0x7f1458147b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:36.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f14577fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f145806fc90 0x7f1458147b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42018/0 (socket says 172.21.15.125:42018) 2023-10-30T12:35:36.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f14577fe640 1 -- 172.21.15.125:0/4185405595 learned_addr learned my addr 172.21.15.125:0/4185405595 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:36.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f1456ffd640 1 --2- 172.21.15.125:0/4185405595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f145814ff10 0x7f14580c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:36.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f1457fff640 1 --2- 172.21.15.125:0/4185405595 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14581566b0 0x7f14580c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:36.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f14577fe640 1 -- 172.21.15.125:0/4185405595 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14581566b0 msgr2=0x7f14580c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:36.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f14577fe640 1 --2- 172.21.15.125:0/4185405595 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14581566b0 0x7f14580c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:36.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f14577fe640 1 -- 172.21.15.125:0/4185405595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f145814ff10 msgr2=0x7f14580c9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:36.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f14577fe640 1 --2- 172.21.15.125:0/4185405595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f145814ff10 0x7f14580c9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:36.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f14577fe640 1 -- 172.21.15.125:0/4185405595 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f144c0094d0 con 0x7f145806fc90 2023-10-30T12:35:36.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f14577fe640 1 --2- 172.21.15.125:0/4185405595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f145806fc90 0x7f1458147b30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f144800c450 tx=0x7f144800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:36.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.241+0000 7f143ffff640 1 -- 172.21.15.125:0/4185405595 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14480052d0 con 0x7f145806fc90 2023-10-30T12:35:36.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.245+0000 7f143ffff640 1 -- 172.21.15.125:0/4185405595 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1448005430 con 0x7f145806fc90 2023-10-30T12:35:36.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.245+0000 7f143ffff640 1 -- 172.21.15.125:0/4185405595 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1448003890 con 0x7f145806fc90 2023-10-30T12:35:36.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.245+0000 7f145ccac640 1 -- 172.21.15.125:0/4185405595 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1458148070 con 0x7f145806fc90 2023-10-30T12:35:36.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.245+0000 7f145ccac640 1 -- 172.21.15.125:0/4185405595 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14580c4780 con 0x7f145806fc90 2023-10-30T12:35:36.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.245+0000 7f145ccac640 1 -- 172.21.15.125:0/4185405595 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f145804e170 con 0x7f145806fc90 2023-10-30T12:35:36.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.245+0000 7f143ffff640 1 -- 172.21.15.125:0/4185405595 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f14480039f0 con 0x7f145806fc90 2023-10-30T12:35:36.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.245+0000 7f143ffff640 1 --2- 172.21.15.125:0/4185405595 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14440664a0 0x7f1444068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:36.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.245+0000 7f143ffff640 1 -- 172.21.15.125:0/4185405595 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5996+0+0 (secure 0 0 0) 0x7f1448083610 con 0x7f145806fc90 2023-10-30T12:35:36.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.245+0000 7f1456ffd640 1 --2- 172.21.15.125:0/4185405595 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14440664a0 0x7f1444068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:36.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.249+0000 7f1456ffd640 1 --2- 172.21.15.125:0/4185405595 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14440664a0 0x7f1444068960 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f144c0ad000 tx=0x7f144c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:36.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.249+0000 7f143ffff640 1 -- 172.21.15.125:0/4185405595 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f14480508d0 con 0x7f145806fc90 2023-10-30T12:35:36.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.429+0000 7f145ccac640 1 -- 172.21.15.125:0/4185405595 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14580c71a0 con 0x7f145806fc90 2023-10-30T12:35:36.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.429+0000 7f143ffff640 1 -- 172.21.15.125:0/4185405595 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+5754 (secure 0 0 0) 0x7f1448053ef0 con 0x7f145806fc90 2023-10-30T12:35:36.436 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:36.436 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:32.042041+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:36.437 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 40 2023-10-30T12:35:36.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.433+0000 7f145ccac640 1 -- 172.21.15.125:0/4185405595 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14440664a0 msgr2=0x7f1444068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:36.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.433+0000 7f145ccac640 1 --2- 172.21.15.125:0/4185405595 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14440664a0 0x7f1444068960 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f144c0ad000 tx=0x7f144c008040 comp rx=0 tx=0).stop 2023-10-30T12:35:36.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.433+0000 7f145ccac640 1 -- 172.21.15.125:0/4185405595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f145806fc90 msgr2=0x7f1458147b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:36.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.433+0000 7f145ccac640 1 --2- 172.21.15.125:0/4185405595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f145806fc90 0x7f1458147b30 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f144800c450 tx=0x7f144800c910 comp rx=0 tx=0).stop 2023-10-30T12:35:36.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.433+0000 7f145ccac640 1 -- 172.21.15.125:0/4185405595 shutdown_connections 2023-10-30T12:35:36.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.433+0000 7f145ccac640 1 --2- 172.21.15.125:0/4185405595 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14440664a0 0x7f1444068960 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:36.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.433+0000 7f145ccac640 1 --2- 172.21.15.125:0/4185405595 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14581566b0 0x7f14580c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:36.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.433+0000 7f145ccac640 1 --2- 172.21.15.125:0/4185405595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f145814ff10 0x7f14580c9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:36.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.433+0000 7f145ccac640 1 --2- 172.21.15.125:0/4185405595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f145806fc90 0x7f1458147b30 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:36.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.433+0000 7f145ccac640 1 -- 172.21.15.125:0/4185405595 >> 172.21.15.125:0/4185405595 conn(0x7f14580c75d0 msgr2=0x7f1458154b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:36.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.433+0000 7f145ccac640 1 -- 172.21.15.125:0/4185405595 shutdown_connections 2023-10-30T12:35:36.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:36.433+0000 7f145ccac640 1 -- 172.21.15.125:0/4185405595 wait complete. 2023-10-30T12:35:36.455 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:35:38.478 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:38.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.621+0000 7fd231aea640 1 Processor -- start 2023-10-30T12:35:38.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.621+0000 7fd231aea640 1 -- start start 2023-10-30T12:35:38.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.621+0000 7fd231aea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd22c070740 0x7fd22c0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:38.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.621+0000 7fd231aea640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd22c15e700 0x7fd22c06a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:38.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.621+0000 7fd231aea640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22c0c5810 con 0x7fd22c1580e0 2023-10-30T12:35:38.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.621+0000 7fd231aea640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd22c070b60 con 0x7fd22c15e700 2023-10-30T12:35:38.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.621+0000 7fd231aea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c15eb90 con 0x7fd22c070740 2023-10-30T12:35:38.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.621+0000 7fd22b7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd22c1580e0 0x7fd22c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45874/0 (socket says 172.21.15.125:45874) 2023-10-30T12:35:38.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.621+0000 7fd22b7fe640 1 -- 172.21.15.125:0/4174230421 learned_addr learned my addr 172.21.15.125:0/4174230421 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:35:38.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22affd640 1 --2- 172.21.15.125:0/4174230421 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd22c070740 0x7fd22c0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:38.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22bfff640 1 --2- 172.21.15.125:0/4174230421 >> v2:172.21.15.136:3300/0 conn(0x7fd22c15e700 0x7fd22c06a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:38.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 -- 172.21.15.125:0/4174230421 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521959338 0 0) 0x7fd22c0c5810 con 0x7fd22c1580e0 2023-10-30T12:35:38.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 -- 172.21.15.125:0/4174230421 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd214003530 con 0x7fd22c1580e0 2023-10-30T12:35:38.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 -- 172.21.15.125:0/4174230421 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1048951146 0 0) 0x7fd214003530 con 0x7fd22c1580e0 2023-10-30T12:35:38.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 -- 172.21.15.125:0/4174230421 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd22c070740 msgr2=0x7fd22c0c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:38.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 --2- 172.21.15.125:0/4174230421 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd22c070740 0x7fd22c0c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:38.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 -- 172.21.15.125:0/4174230421 >> v2:172.21.15.136:3300/0 conn(0x7fd22c15e700 msgr2=0x7fd22c06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:38.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 --2- 172.21.15.125:0/4174230421 >> v2:172.21.15.136:3300/0 conn(0x7fd22c15e700 0x7fd22c06a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:38.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 -- 172.21.15.125:0/4174230421 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c0c6bb0 con 0x7fd22c1580e0 2023-10-30T12:35:38.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 -- 172.21.15.125:0/4174230421 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fd21c001680 con 0x7fd22c1580e0 2023-10-30T12:35:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 -- 172.21.15.125:0/4174230421 >> v1:172.21.15.125:6789/0 conn(0x7fd22c1580e0 legacy=0x7fd22c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 --2- 172.21.15.125:0/4174230421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd214003980 0x7fd214003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 --2- 172.21.15.125:0/4174230421 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd22c070740 0x7fd2140047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 --2- 172.21.15.125:0/4174230421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd214004d10 0x7fd214005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 -- 172.21.15.125:0/4174230421 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c070b60 con 0x7fd214003980 2023-10-30T12:35:38.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 -- 172.21.15.125:0/4174230421 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c15eb90 con 0x7fd214004d10 2023-10-30T12:35:38.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22a7fc640 1 -- 172.21.15.125:0/4174230421 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd21400b990 con 0x7fd22c070740 2023-10-30T12:35:38.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22b7fe640 1 --2- 172.21.15.125:0/4174230421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd214003980 0x7fd214003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:38.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22affd640 1 --2- 172.21.15.125:0/4174230421 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd22c070740 0x7fd2140047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:38.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22bfff640 1 --2- 172.21.15.125:0/4174230421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd214004d10 0x7fd214005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:38.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22b7fe640 1 -- 172.21.15.125:0/4174230421 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd22c070740 msgr2=0x7fd2140047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:38.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22b7fe640 1 --2- 172.21.15.125:0/4174230421 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd22c070740 0x7fd2140047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:38.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22b7fe640 1 -- 172.21.15.125:0/4174230421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd214004d10 msgr2=0x7fd214005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:38.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22b7fe640 1 --2- 172.21.15.125:0/4174230421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd214004d10 0x7fd214005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:38.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.625+0000 7fd22b7fe640 1 -- 172.21.15.125:0/4174230421 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd214003530 con 0x7fd214003980 2023-10-30T12:35:38.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.629+0000 7fd22b7fe640 1 --2- 172.21.15.125:0/4174230421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd214003980 0x7fd214003da0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd21c00c270 tx=0x7fd21c01cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3579f70cdee00e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:38.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.629+0000 7fd22a7fc640 1 -- 172.21.15.125:0/4174230421 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c0a5930 con 0x7fd214003980 2023-10-30T12:35:38.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.629+0000 7fd22a7fc640 1 -- 172.21.15.125:0/4174230421 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21c0a5a90 con 0x7fd214003980 2023-10-30T12:35:38.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.629+0000 7fd22a7fc640 1 -- 172.21.15.125:0/4174230421 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c0a5e60 con 0x7fd214003980 2023-10-30T12:35:38.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.629+0000 7fd231aea640 1 -- 172.21.15.125:0/4174230421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd214003980 msgr2=0x7fd214003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:38.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.629+0000 7fd231aea640 1 --2- 172.21.15.125:0/4174230421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd214003980 0x7fd214003da0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fd21c00c270 tx=0x7fd21c01cc30 comp rx=0 tx=0).stop 2023-10-30T12:35:38.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.629+0000 7fd22bfff640 1 -- 172.21.15.125:0/4174230421 reap_dead start 2023-10-30T12:35:38.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.629+0000 7fd231aea640 1 -- 172.21.15.125:0/4174230421 shutdown_connections 2023-10-30T12:35:38.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.629+0000 7fd231aea640 1 -- 172.21.15.125:0/4174230421 >> 172.21.15.125:0/4174230421 conn(0x7fd22c0c7910 msgr2=0x7fd22c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:38.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.629+0000 7fd231aea640 1 -- 172.21.15.125:0/4174230421 shutdown_connections 2023-10-30T12:35:38.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.629+0000 7fd231aea640 1 -- 172.21.15.125:0/4174230421 wait complete. 2023-10-30T12:35:38.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.629+0000 7fd231aea640 1 Processor -- start 2023-10-30T12:35:38.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.629+0000 7fd231aea640 1 -- start start 2023-10-30T12:35:38.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd231aea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd214003980 0x7fd22c1f9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:38.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd231aea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd22c14d850 0x7fd22c06b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:38.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd231aea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd22c1f63e0 0x7fd22c1f6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:38.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd231aea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c15f580 con 0x7fd22c1f63e0 2023-10-30T12:35:38.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd231aea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c1f5f90 con 0x7fd214003980 2023-10-30T12:35:38.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd231aea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd22c1f6da0 con 0x7fd22c14d850 2023-10-30T12:35:38.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd22bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd22c1f63e0 0x7fd22c1f6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:38.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd22bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd22c1f63e0 0x7fd22c1f6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42032/0 (socket says 172.21.15.125:42032) 2023-10-30T12:35:38.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd22bfff640 1 -- 172.21.15.125:0/2404546817 learned_addr learned my addr 172.21.15.125:0/2404546817 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:38.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd22affd640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd22c14d850 0x7fd22c06b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:38.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd22b7fe640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd214003980 0x7fd22c1f9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:38.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd22bfff640 1 -- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd22c14d850 msgr2=0x7fd22c06b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:38.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd22bfff640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd22c14d850 0x7fd22c06b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:38.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd22bfff640 1 -- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd214003980 msgr2=0x7fd22c1f9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:38.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd22bfff640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd214003980 0x7fd22c1f9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:38.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd22bfff640 1 -- 172.21.15.125:0/2404546817 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c1fa400 con 0x7fd22c1f63e0 2023-10-30T12:35:38.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd22affd640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd22c14d850 0x7fd22c06b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:35:38.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd22bfff640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd22c1f63e0 0x7fd22c1f6860 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd22c06f4f0 tx=0x7fd22400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:38.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd213fff640 1 -- 172.21.15.125:0/2404546817 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22400e040 con 0x7fd22c1f63e0 2023-10-30T12:35:38.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd213fff640 1 -- 172.21.15.125:0/2404546817 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22400f040 con 0x7fd22c1f63e0 2023-10-30T12:35:38.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd213fff640 1 -- 172.21.15.125:0/2404546817 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd224015430 con 0x7fd22c1f63e0 2023-10-30T12:35:38.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd231aea640 1 -- 172.21.15.125:0/2404546817 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd22c0c4e10 con 0x7fd22c1f63e0 2023-10-30T12:35:38.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd231aea640 1 -- 172.21.15.125:0/2404546817 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22c15efc0 con 0x7fd22c1f63e0 2023-10-30T12:35:38.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd231aea640 1 -- 172.21.15.125:0/2404546817 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22c04e170 con 0x7fd22c1f63e0 2023-10-30T12:35:38.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.633+0000 7fd213fff640 1 -- 172.21.15.125:0/2404546817 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd224015590 con 0x7fd22c1f63e0 2023-10-30T12:35:38.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.637+0000 7fd213fff640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2180664a0 0x7fd218068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:38.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.637+0000 7fd213fff640 1 -- 172.21.15.125:0/2404546817 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5996+0+0 (secure 0 0 0) 0x7fd224084790 con 0x7fd22c1f63e0 2023-10-30T12:35:38.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.637+0000 7fd22b7fe640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2180664a0 0x7fd218068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:38.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.637+0000 7fd22b7fe640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2180664a0 0x7fd218068960 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd21c0aa000 tx=0x7fd21c001a80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:38.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.641+0000 7fd213fff640 1 -- 172.21.15.125:0/2404546817 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd224051a50 con 0x7fd22c1f63e0 2023-10-30T12:35:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.821+0000 7fd231aea640 1 -- 172.21.15.125:0/2404546817 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd22c1578a0 con 0x7fd22c1f63e0 2023-10-30T12:35:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.825+0000 7fd213fff640 1 -- 172.21.15.125:0/2404546817 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+5754 (secure 0 0 0) 0x7fd224055070 con 0x7fd22c1f63e0 2023-10-30T12:35:38.828 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:38.828 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":41,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:37.348231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 41 2023-10-30T12:35:38.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.825+0000 7fd231aea640 1 -- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2180664a0 msgr2=0x7fd218068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:38.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.825+0000 7fd231aea640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2180664a0 0x7fd218068960 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd21c0aa000 tx=0x7fd21c001a80 comp rx=0 tx=0).stop 2023-10-30T12:35:38.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.825+0000 7fd231aea640 1 -- 172.21.15.125:0/2404546817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd22c1f63e0 msgr2=0x7fd22c1f6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:38.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.825+0000 7fd231aea640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd22c1f63e0 0x7fd22c1f6860 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd22c06f4f0 tx=0x7fd22400f580 comp rx=0 tx=0).stop 2023-10-30T12:35:38.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.825+0000 7fd231aea640 1 -- 172.21.15.125:0/2404546817 shutdown_connections 2023-10-30T12:35:38.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.825+0000 7fd231aea640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2180664a0 0x7fd218068960 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:38.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.825+0000 7fd231aea640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd22c1f63e0 0x7fd22c1f6860 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:38.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.825+0000 7fd231aea640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd22c14d850 0x7fd22c06b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:38.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.825+0000 7fd231aea640 1 --2- 172.21.15.125:0/2404546817 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd214003980 0x7fd22c1f9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:38.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.825+0000 7fd231aea640 1 -- 172.21.15.125:0/2404546817 >> 172.21.15.125:0/2404546817 conn(0x7fd22c0c7910 msgr2=0x7fd22c155c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:38.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.825+0000 7fd231aea640 1 -- 172.21.15.125:0/2404546817 shutdown_connections 2023-10-30T12:35:38.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:38.825+0000 7fd231aea640 1 -- 172.21.15.125:0/2404546817 wait complete. 2023-10-30T12:35:38.848 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=1, replacing gid=4285 2023-10-30T12:35:38.849 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.f 2023-10-30T12:35:39.409 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:35:39.405+0000 7f6a6fdf8640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:35:40.850 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T12:35:40.850 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T12:35:40.851 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T12:35:40.854 INFO:tasks.ceph.mds.f:Started 2023-10-30T12:35:40.854 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:35:40.930 INFO:tasks.ceph.mds.f.smithi136.stdout:starting mds.f at 2023-10-30T12:35:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.001+0000 7ff59a0ea640 1 Processor -- start 2023-10-30T12:35:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.001+0000 7ff59a0ea640 1 -- start start 2023-10-30T12:35:41.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.001+0000 7ff59a0ea640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff594070740 0x7ff5940c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:41.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.001+0000 7ff59a0ea640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff59415e700 0x7ff59406a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:41.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.001+0000 7ff59a0ea640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5940c5810 con 0x7ff5941580e0 2023-10-30T12:35:41.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.001+0000 7ff59a0ea640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff59415eb70 con 0x7ff59415e700 2023-10-30T12:35:41.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.001+0000 7ff59a0ea640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5940c6bb0 con 0x7ff594070740 2023-10-30T12:35:41.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.001+0000 7ff593fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff59415e700 0x7ff59406a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:41.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.001+0000 7ff593fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff59415e700 0x7ff59406a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55994/0 (socket says 172.21.15.125:55994) 2023-10-30T12:35:41.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.001+0000 7ff593fff640 1 -- 172.21.15.125:0/442331078 learned_addr learned my addr 172.21.15.125:0/442331078 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:41.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.001+0000 7ff5937fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff5941580e0 0x7ff594069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34126/0 (socket says 172.21.15.125:34126) 2023-10-30T12:35:41.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.001+0000 7ff592ffd640 1 --2- 172.21.15.125:0/442331078 >> v2:172.21.15.136:3300/0 conn(0x7ff594070740 0x7ff5940c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:41.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff593fff640 1 -- 172.21.15.125:0/442331078 >> v2:172.21.15.136:3300/0 conn(0x7ff594070740 msgr2=0x7ff5940c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:35:41.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff593fff640 1 --2- 172.21.15.125:0/442331078 >> v2:172.21.15.136:3300/0 conn(0x7ff594070740 0x7ff5940c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:41.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff593fff640 1 -- 172.21.15.125:0/442331078 >> v1:172.21.15.125:6789/0 conn(0x7ff5941580e0 legacy=0x7ff594069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:41.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff5927fc640 1 -- 172.21.15.125:0/442331078 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933353910 0 0) 0x7ff5940c5810 con 0x7ff5941580e0 2023-10-30T12:35:41.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff593fff640 1 -- 172.21.15.125:0/442331078 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5940c6e70 con 0x7ff59415e700 2023-10-30T12:35:41.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff593fff640 1 --2- 172.21.15.125:0/442331078 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff59415e700 0x7ff59406a5a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff59406b1a0 tx=0x7ff58c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab2ecdae6efc7f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:41.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff5927fc640 1 -- 172.21.15.125:0/442331078 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c0a5780 con 0x7ff59415e700 2023-10-30T12:35:41.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff5927fc640 1 -- 172.21.15.125:0/442331078 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58c0a58e0 con 0x7ff59415e700 2023-10-30T12:35:41.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff5927fc640 1 -- 172.21.15.125:0/442331078 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58c0a5c70 con 0x7ff59415e700 2023-10-30T12:35:41.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff59a0ea640 1 -- 172.21.15.125:0/442331078 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff59415e700 msgr2=0x7ff59406a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:41.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff59a0ea640 1 --2- 172.21.15.125:0/442331078 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff59415e700 0x7ff59406a5a0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7ff59406b1a0 tx=0x7ff58c01c9d0 comp rx=0 tx=0).stop 2023-10-30T12:35:41.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff59a0ea640 1 -- 172.21.15.125:0/442331078 shutdown_connections 2023-10-30T12:35:41.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff59a0ea640 1 --2- 172.21.15.125:0/442331078 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff59415e700 0x7ff59406a5a0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:41.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff59a0ea640 1 --2- 172.21.15.125:0/442331078 >> v2:172.21.15.136:3300/0 conn(0x7ff594070740 0x7ff5940c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:41.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff59a0ea640 1 -- 172.21.15.125:0/442331078 >> 172.21.15.125:0/442331078 conn(0x7ff5940c7910 msgr2=0x7ff5940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:41.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff59a0ea640 1 -- 172.21.15.125:0/442331078 shutdown_connections 2023-10-30T12:35:41.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.005+0000 7ff59a0ea640 1 -- 172.21.15.125:0/442331078 wait complete. 2023-10-30T12:35:41.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff59a0ea640 1 Processor -- start 2023-10-30T12:35:41.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff59a0ea640 1 -- start start 2023-10-30T12:35:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff59a0ea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff594070740 0x7ff5940c9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff59a0ea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5941580e0 0x7ff5940ca530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff59a0ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff59415e700 0x7ff594143d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff59a0ea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff594142530 con 0x7ff5941580e0 2023-10-30T12:35:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff59a0ea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5940c9b40 con 0x7ff594070740 2023-10-30T12:35:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff59a0ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59414a1a0 con 0x7ff59415e700 2023-10-30T12:35:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff592ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5941580e0 0x7ff5940ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff592ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5941580e0 0x7ff5940ca530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56002/0 (socket says 172.21.15.125:56002) 2023-10-30T12:35:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff592ffd640 1 -- 172.21.15.125:0/328593243 learned_addr learned my addr 172.21.15.125:0/328593243 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:35:41.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff5937fe640 1 --2- 172.21.15.125:0/328593243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff594070740 0x7ff5940c9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:41.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff593fff640 1 --2- 172.21.15.125:0/328593243 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff59415e700 0x7ff594143d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:41.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff592ffd640 1 -- 172.21.15.125:0/328593243 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff59415e700 msgr2=0x7ff594143d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:41.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff592ffd640 1 --2- 172.21.15.125:0/328593243 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff59415e700 0x7ff594143d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:41.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff592ffd640 1 -- 172.21.15.125:0/328593243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff594070740 msgr2=0x7ff5940c9ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:41.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff592ffd640 1 --2- 172.21.15.125:0/328593243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff594070740 0x7ff5940c9ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:41.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff592ffd640 1 -- 172.21.15.125:0/328593243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c009b30 con 0x7ff5941580e0 2023-10-30T12:35:41.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff592ffd640 1 --2- 172.21.15.125:0/328593243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5941580e0 0x7ff5940ca530 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff58800e930 tx=0x7ff58800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:41.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff57bfff640 1 -- 172.21.15.125:0/328593243 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff588009690 con 0x7ff5941580e0 2023-10-30T12:35:41.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff57bfff640 1 -- 172.21.15.125:0/328593243 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58800ca00 con 0x7ff5941580e0 2023-10-30T12:35:41.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff57bfff640 1 -- 172.21.15.125:0/328593243 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58800cd70 con 0x7ff5941580e0 2023-10-30T12:35:41.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.009+0000 7ff59a0ea640 1 -- 172.21.15.125:0/328593243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff594148570 con 0x7ff5941580e0 2023-10-30T12:35:41.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.013+0000 7ff59a0ea640 1 -- 172.21.15.125:0/328593243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff594148b30 con 0x7ff5941580e0 2023-10-30T12:35:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.013+0000 7ff59a0ea640 1 -- 172.21.15.125:0/328593243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59404e170 con 0x7ff5941580e0 2023-10-30T12:35:41.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.013+0000 7ff57bfff640 1 -- 172.21.15.125:0/328593243 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff588013470 con 0x7ff5941580e0 2023-10-30T12:35:41.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.013+0000 7ff57bfff640 1 --2- 172.21.15.125:0/328593243 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5800664a0 0x7ff580068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:35:41.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.013+0000 7ff57bfff640 1 -- 172.21.15.125:0/328593243 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 5996+0+0 (secure 0 0 0) 0x7ff588084320 con 0x7ff5941580e0 2023-10-30T12:35:41.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.013+0000 7ff5937fe640 1 --2- 172.21.15.125:0/328593243 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5800664a0 0x7ff580068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:35:41.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.013+0000 7ff5937fe640 1 --2- 172.21.15.125:0/328593243 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5800664a0 0x7ff580068960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff584000f70 tx=0x7ff584007b60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:35:41.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.017+0000 7ff57bfff640 1 -- 172.21.15.125:0/328593243 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff588051610 con 0x7ff5941580e0 2023-10-30T12:35:41.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.205+0000 7ff59a0ea640 1 -- 172.21.15.125:0/328593243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5941683e0 con 0x7ff5941580e0 2023-10-30T12:35:41.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.205+0000 7ff57bfff640 1 -- 172.21.15.125:0/328593243 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+6548 (secure 0 0 0) 0x7ff588054c30 con 0x7ff5941580e0 2023-10-30T12:35:41.209 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:35:41.209 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4973,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":41,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:35:37.348231+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:35:41.209 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 42 2023-10-30T12:35:41.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.205+0000 7ff59a0ea640 1 -- 172.21.15.125:0/328593243 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5800664a0 msgr2=0x7ff580068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:41.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.205+0000 7ff59a0ea640 1 --2- 172.21.15.125:0/328593243 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5800664a0 0x7ff580068960 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff584000f70 tx=0x7ff584007b60 comp rx=0 tx=0).stop 2023-10-30T12:35:41.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.205+0000 7ff59a0ea640 1 -- 172.21.15.125:0/328593243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5941580e0 msgr2=0x7ff5940ca530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:35:41.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.205+0000 7ff59a0ea640 1 --2- 172.21.15.125:0/328593243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5941580e0 0x7ff5940ca530 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7ff58800e930 tx=0x7ff58800edf0 comp rx=0 tx=0).stop 2023-10-30T12:35:41.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.205+0000 7ff59a0ea640 1 -- 172.21.15.125:0/328593243 shutdown_connections 2023-10-30T12:35:41.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.205+0000 7ff59a0ea640 1 --2- 172.21.15.125:0/328593243 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5800664a0 0x7ff580068960 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:41.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.205+0000 7ff59a0ea640 1 --2- 172.21.15.125:0/328593243 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff59415e700 0x7ff594143d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:41.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.205+0000 7ff59a0ea640 1 --2- 172.21.15.125:0/328593243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5941580e0 0x7ff5940ca530 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:41.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.205+0000 7ff59a0ea640 1 --2- 172.21.15.125:0/328593243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff594070740 0x7ff5940c9ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:35:41.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.205+0000 7ff59a0ea640 1 -- 172.21.15.125:0/328593243 >> 172.21.15.125:0/328593243 conn(0x7ff5940c7910 msgr2=0x7ff5941561a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:35:41.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.205+0000 7ff59a0ea640 1 -- 172.21.15.125:0/328593243 shutdown_connections 2023-10-30T12:35:41.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:35:41.205+0000 7ff59a0ea640 1 -- 172.21.15.125:0/328593243 wait complete. 2023-10-30T12:35:41.230 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T12:35:41.231 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 31 secs before thrashing 2023-10-30T12:35:44.775 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:35:44.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:35:45.736 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/138/13' 2023-10-30T12:35:45.737 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/138/17' 2023-10-30T12:35:45.737 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/138' 2023-10-30T12:35:45.737 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/135/19' 2023-10-30T12:35:45.738 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/135/15' 2023-10-30T12:35:45.738 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/135/20' 2023-10-30T12:35:45.738 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/135/12' 2023-10-30T12:35:45.738 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/135/10' 2023-10-30T12:35:45.738 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/135/11' 2023-10-30T12:35:45.739 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/135/16' 2023-10-30T12:35:45.739 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/135/14' 2023-10-30T12:35:45.739 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/135/18' 2023-10-30T12:35:45.739 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/135/13' 2023-10-30T12:35:45.739 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/135/17' 2023-10-30T12:35:45.739 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/135' 2023-10-30T12:35:45.740 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/117/19' 2023-10-30T12:35:45.740 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/117/15' 2023-10-30T12:35:45.740 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/117/20' 2023-10-30T12:35:45.740 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/117/12' 2023-10-30T12:35:45.740 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/117/10' 2023-10-30T12:35:45.741 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/117/11' 2023-10-30T12:35:45.741 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/117/16' 2023-10-30T12:35:45.741 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/117/14' 2023-10-30T12:35:45.741 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/117/18' 2023-10-30T12:35:45.741 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/117/13' 2023-10-30T12:35:45.742 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/117/17' 2023-10-30T12:35:45.742 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/117' 2023-10-30T12:35:45.742 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/143/19' 2023-10-30T12:35:45.742 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/143/15' 2023-10-30T12:35:45.743 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/143/20' 2023-10-30T12:35:45.743 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/143/12' 2023-10-30T12:35:45.743 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/143/10' 2023-10-30T12:35:45.743 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/143/11' 2023-10-30T12:35:45.743 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/143/16' 2023-10-30T12:35:45.743 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/143/14' 2023-10-30T12:35:45.744 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/143/18' 2023-10-30T12:35:45.744 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/143/13' 2023-10-30T12:35:45.744 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/143/17' 2023-10-30T12:35:45.744 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/143' 2023-10-30T12:35:45.744 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/180/19' 2023-10-30T12:35:45.745 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/180/15' 2023-10-30T12:35:45.745 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/180/20' 2023-10-30T12:35:45.745 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/180/12' 2023-10-30T12:35:45.745 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/180/10' 2023-10-30T12:35:45.745 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/180/11' 2023-10-30T12:35:45.746 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/180/16' 2023-10-30T12:35:45.746 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/180/14' 2023-10-30T12:35:45.746 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/180/18' 2023-10-30T12:35:45.746 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/180/13' 2023-10-30T12:35:45.746 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/180/17' 2023-10-30T12:35:45.747 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/180' 2023-10-30T12:35:45.747 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/126/19' 2023-10-30T12:35:45.747 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/126/15' 2023-10-30T12:35:45.747 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/126/20' 2023-10-30T12:35:45.747 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/126/12' 2023-10-30T12:35:45.747 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/126/10' 2023-10-30T12:35:45.748 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/126/11' 2023-10-30T12:35:45.748 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/126/16' 2023-10-30T12:35:45.748 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/126/14' 2023-10-30T12:35:45.748 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/126/18' 2023-10-30T12:35:45.748 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/126/13' 2023-10-30T12:35:45.749 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/126/17' 2023-10-30T12:35:45.749 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/126' 2023-10-30T12:35:45.749 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/161/19' 2023-10-30T12:35:45.749 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/161/15' 2023-10-30T12:35:45.749 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/161/20' 2023-10-30T12:35:45.750 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/161/12' 2023-10-30T12:35:45.750 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/161/10' 2023-10-30T12:35:45.750 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/161/11' 2023-10-30T12:35:45.750 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/161/16' 2023-10-30T12:35:45.750 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/161/14' 2023-10-30T12:35:45.751 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/161/18' 2023-10-30T12:35:45.751 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/161/13' 2023-10-30T12:35:45.751 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/161/17' 2023-10-30T12:35:45.751 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/161' 2023-10-30T12:35:45.751 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/176/19' 2023-10-30T12:35:45.752 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/176/15' 2023-10-30T12:35:45.752 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/176/20' 2023-10-30T12:35:45.752 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/176/12' 2023-10-30T12:35:45.752 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/176/10' 2023-10-30T12:35:45.752 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/176/11' 2023-10-30T12:35:45.752 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/176/16' 2023-10-30T12:35:45.753 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/176/14' 2023-10-30T12:35:45.753 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/176/18' 2023-10-30T12:35:45.753 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/176/13' 2023-10-30T12:35:45.753 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/176/17' 2023-10-30T12:35:45.753 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/176' 2023-10-30T12:35:45.754 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/111/19' 2023-10-30T12:35:45.754 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/111/15' 2023-10-30T12:35:45.754 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/111/20' 2023-10-30T12:35:45.754 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/111/12' 2023-10-30T12:35:45.754 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/111/10' 2023-10-30T12:35:45.755 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/111/11' 2023-10-30T12:35:45.755 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/111/16' 2023-10-30T12:35:45.755 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/111/14' 2023-10-30T12:35:45.755 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/111/18' 2023-10-30T12:35:45.755 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/111/13' 2023-10-30T12:35:45.755 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/111/17' 2023-10-30T12:35:45.756 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/111' 2023-10-30T12:35:45.756 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/191/19' 2023-10-30T12:35:45.756 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/191/15' 2023-10-30T12:35:45.756 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/191/20' 2023-10-30T12:35:45.756 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/191/12' 2023-10-30T12:35:45.757 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/191/10' 2023-10-30T12:35:45.757 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/191/11' 2023-10-30T12:35:45.757 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/191/16' 2023-10-30T12:35:45.757 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/191/14' 2023-10-30T12:35:45.757 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/191/18' 2023-10-30T12:35:45.758 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/191/13' 2023-10-30T12:35:49.222 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/191/17' 2023-10-30T12:35:49.222 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/191' 2023-10-30T12:35:49.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/184/19' 2023-10-30T12:35:49.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/184/15' 2023-10-30T12:35:49.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/184/20' 2023-10-30T12:35:49.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/184/12' 2023-10-30T12:35:49.223 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/184/10' 2023-10-30T12:35:49.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/184/11' 2023-10-30T12:35:49.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/184/16' 2023-10-30T12:35:49.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/184/14' 2023-10-30T12:35:49.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/184/18' 2023-10-30T12:35:49.224 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/184/13' 2023-10-30T12:35:49.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/184/17' 2023-10-30T12:35:49.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/184' 2023-10-30T12:35:49.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/192/19' 2023-10-30T12:35:49.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/192/15' 2023-10-30T12:35:49.225 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/192/20' 2023-10-30T12:35:49.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/192/12' 2023-10-30T12:35:49.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/192/10' 2023-10-30T12:35:49.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/192/11' 2023-10-30T12:35:49.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/192/16' 2023-10-30T12:35:49.226 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/192/14' 2023-10-30T12:35:49.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/192/18' 2023-10-30T12:35:49.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/192/13' 2023-10-30T12:35:49.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/192/17' 2023-10-30T12:35:49.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/192' 2023-10-30T12:35:49.227 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/103/19' 2023-10-30T12:35:49.228 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/103/15' 2023-10-30T12:35:49.228 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/103/20' 2023-10-30T12:35:49.228 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/103/12' 2023-10-30T12:35:49.228 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/103/10' 2023-10-30T12:35:49.228 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/103/11' 2023-10-30T12:35:49.229 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/103/16' 2023-10-30T12:35:49.229 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/103/14' 2023-10-30T12:35:49.229 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/103/18' 2023-10-30T12:35:49.229 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/103/13' 2023-10-30T12:35:49.229 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/103/17' 2023-10-30T12:35:49.230 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/103' 2023-10-30T12:35:49.230 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/128/19' 2023-10-30T12:35:49.230 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/128/15' 2023-10-30T12:35:49.230 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/128/20' 2023-10-30T12:35:49.231 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/128/12' 2023-10-30T12:35:49.231 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/128/10' 2023-10-30T12:35:49.231 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/128/11' 2023-10-30T12:35:49.231 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/128/16' 2023-10-30T12:35:49.231 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/128/14' 2023-10-30T12:35:49.232 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/128/18' 2023-10-30T12:35:49.232 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/128/13' 2023-10-30T12:35:49.232 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/128/17' 2023-10-30T12:35:49.232 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/128' 2023-10-30T12:35:49.232 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/144/19' 2023-10-30T12:35:49.232 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/144/15' 2023-10-30T12:35:49.233 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/144/20' 2023-10-30T12:35:49.233 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/144/12' 2023-10-30T12:35:49.233 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/144/10' 2023-10-30T12:35:49.233 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/144/11' 2023-10-30T12:35:49.233 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/144/16' 2023-10-30T12:35:49.234 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/144/14' 2023-10-30T12:35:49.234 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/144/18' 2023-10-30T12:35:49.234 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/144/13' 2023-10-30T12:35:49.234 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/144/17' 2023-10-30T12:35:49.234 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/144' 2023-10-30T12:35:49.235 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/146/19' 2023-10-30T12:35:49.235 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/146/15' 2023-10-30T12:35:49.235 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/146/20' 2023-10-30T12:35:49.235 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/146/12' 2023-10-30T12:35:49.235 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/146/10' 2023-10-30T12:35:49.236 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/146/11' 2023-10-30T12:35:49.236 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/146/16' 2023-10-30T12:35:49.236 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/146/14' 2023-10-30T12:35:49.236 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/146/18' 2023-10-30T12:35:49.236 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/146/13' 2023-10-30T12:35:49.237 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/146/17' 2023-10-30T12:35:49.237 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/146' 2023-10-30T12:35:49.237 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/141/19' 2023-10-30T12:35:49.237 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/141/15' 2023-10-30T12:35:49.237 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/141/20' 2023-10-30T12:35:49.238 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/141/12' 2023-10-30T12:35:49.238 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/141/10' 2023-10-30T12:35:49.238 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/141/11' 2023-10-30T12:35:49.238 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/141/16' 2023-10-30T12:35:49.238 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/141/14' 2023-10-30T12:35:49.239 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/141/18' 2023-10-30T12:35:49.239 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/141/13' 2023-10-30T12:35:49.239 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/141/17' 2023-10-30T12:35:49.239 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/141' 2023-10-30T12:35:49.239 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/182/19' 2023-10-30T12:35:49.240 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/182/15' 2023-10-30T12:35:49.240 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/182/20' 2023-10-30T12:35:49.240 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/182/12' 2023-10-30T12:35:49.240 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/182/10' 2023-10-30T12:35:49.240 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/182/11' 2023-10-30T12:35:49.240 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/182/16' 2023-10-30T12:35:49.241 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/182/14' 2023-10-30T12:35:49.241 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/182/18' 2023-10-30T12:35:49.241 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/182/13' 2023-10-30T12:35:49.241 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/182/17' 2023-10-30T12:35:49.241 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/182' 2023-10-30T12:35:49.242 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/107/19' 2023-10-30T12:35:49.242 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/107/15' 2023-10-30T12:35:49.242 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/107/20' 2023-10-30T12:35:49.242 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/107/12' 2023-10-30T12:35:49.243 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/107/10' 2023-10-30T12:35:49.243 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/107/11' 2023-10-30T12:35:49.243 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/107/16' 2023-10-30T12:35:49.243 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/107/14' 2023-10-30T12:35:49.243 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/107/18' 2023-10-30T12:35:49.243 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/107/13' 2023-10-30T12:35:49.244 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/107/17' 2023-10-30T12:35:51.422 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/107' 2023-10-30T12:35:51.423 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/178/19' 2023-10-30T12:35:51.423 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/178/15' 2023-10-30T12:35:51.423 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/178/20' 2023-10-30T12:35:51.423 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/178/12' 2023-10-30T12:35:51.424 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/178/10' 2023-10-30T12:35:51.424 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/178/11' 2023-10-30T12:35:51.424 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/178/16' 2023-10-30T12:35:51.424 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/178/14' 2023-10-30T12:35:51.424 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/178/18' 2023-10-30T12:35:51.425 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/178/13' 2023-10-30T12:35:51.425 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/178/17' 2023-10-30T12:35:51.425 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/178' 2023-10-30T12:35:51.425 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/156/19' 2023-10-30T12:35:51.425 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/156/15' 2023-10-30T12:35:51.426 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/156/20' 2023-10-30T12:35:51.426 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/156/12' 2023-10-30T12:35:51.426 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/156/10' 2023-10-30T12:35:51.426 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/156/11' 2023-10-30T12:35:51.426 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/156/16' 2023-10-30T12:35:51.427 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/156/14' 2023-10-30T12:35:51.427 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/156/18' 2023-10-30T12:35:51.427 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/156/13' 2023-10-30T12:35:51.427 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/156/17' 2023-10-30T12:35:51.427 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/156' 2023-10-30T12:35:51.427 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/131/19' 2023-10-30T12:35:51.428 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/131/15' 2023-10-30T12:35:51.428 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/131/20' 2023-10-30T12:35:51.428 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/131/12' 2023-10-30T12:35:51.428 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/131/10' 2023-10-30T12:35:51.429 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/131/11' 2023-10-30T12:35:51.429 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/131/16' 2023-10-30T12:35:51.429 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/131/14' 2023-10-30T12:35:51.429 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/131/18' 2023-10-30T12:35:51.429 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/131/13' 2023-10-30T12:35:51.430 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/131/17' 2023-10-30T12:35:51.430 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/131' 2023-10-30T12:35:51.430 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/123/19' 2023-10-30T12:35:51.430 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/123/15' 2023-10-30T12:35:51.430 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/123/20' 2023-10-30T12:35:51.431 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/123/12' 2023-10-30T12:35:51.431 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/123/10' 2023-10-30T12:35:51.431 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/123/11' 2023-10-30T12:35:51.431 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/123/16' 2023-10-30T12:35:51.431 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/123/14' 2023-10-30T12:35:51.432 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/123/18' 2023-10-30T12:35:51.432 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/123/13' 2023-10-30T12:35:51.432 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/123/17' 2023-10-30T12:35:51.432 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/123' 2023-10-30T12:35:51.433 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/160/19' 2023-10-30T12:35:51.433 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/160/15' 2023-10-30T12:35:51.433 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/160/20' 2023-10-30T12:35:51.433 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/160/12' 2023-10-30T12:35:51.433 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/160/10' 2023-10-30T12:35:51.434 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/160/11' 2023-10-30T12:35:51.434 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/160/16' 2023-10-30T12:35:51.434 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/160/14' 2023-10-30T12:35:51.434 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/160/18' 2023-10-30T12:35:51.434 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/160/13' 2023-10-30T12:35:51.435 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/160/17' 2023-10-30T12:35:51.435 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/160' 2023-10-30T12:35:51.435 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/165/19' 2023-10-30T12:35:51.435 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/165/15' 2023-10-30T12:35:51.435 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/165/20' 2023-10-30T12:35:51.436 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/165/12' 2023-10-30T12:35:51.436 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/165/10' 2023-10-30T12:35:51.436 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/165/11' 2023-10-30T12:35:51.436 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/165/16' 2023-10-30T12:35:51.436 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/165/14' 2023-10-30T12:35:51.437 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/165/18' 2023-10-30T12:35:51.437 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/165/13' 2023-10-30T12:35:51.437 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/165/17' 2023-10-30T12:35:51.437 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/165' 2023-10-30T12:35:51.437 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/101/19' 2023-10-30T12:35:51.438 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/101/15' 2023-10-30T12:35:51.438 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/101/20' 2023-10-30T12:35:51.438 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/101/12' 2023-10-30T12:35:51.438 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/101/10' 2023-10-30T12:35:51.438 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/101/11' 2023-10-30T12:35:51.439 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/101/16' 2023-10-30T12:35:51.439 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/101/14' 2023-10-30T12:35:51.439 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/101/18' 2023-10-30T12:35:51.439 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/101/13' 2023-10-30T12:35:51.440 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/101/17' 2023-10-30T12:35:51.440 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/101' 2023-10-30T12:35:51.440 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/148/19' 2023-10-30T12:35:51.440 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/148/15' 2023-10-30T12:35:51.440 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/148/20' 2023-10-30T12:35:51.441 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/148/12' 2023-10-30T12:35:51.441 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/148/10' 2023-10-30T12:35:51.441 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/148/11' 2023-10-30T12:35:51.441 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/148/16' 2023-10-30T12:35:51.442 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/148/14' 2023-10-30T12:35:51.442 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/148/18' 2023-10-30T12:35:51.442 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/148/13' 2023-10-30T12:35:51.442 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/148/17' 2023-10-30T12:35:51.443 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/148' 2023-10-30T12:35:51.443 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/168/19' 2023-10-30T12:35:51.443 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/168/15' 2023-10-30T12:35:51.443 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/168/20' 2023-10-30T12:35:51.443 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/168/12' 2023-10-30T12:35:51.444 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/168/10' 2023-10-30T12:35:51.444 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/168/11' 2023-10-30T12:35:51.444 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/168/16' 2023-10-30T12:35:51.444 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/168/14' 2023-10-30T12:35:51.444 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/168/18' 2023-10-30T12:35:51.445 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/168/13' 2023-10-30T12:35:51.445 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/168/17' 2023-10-30T12:35:51.445 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/168' 2023-10-30T12:35:54.523 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/179/19' 2023-10-30T12:35:54.523 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/179/15' 2023-10-30T12:35:54.523 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/179/20' 2023-10-30T12:35:54.523 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/179/12' 2023-10-30T12:35:54.524 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/179/10' 2023-10-30T12:35:54.524 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/179/11' 2023-10-30T12:35:54.524 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/179/16' 2023-10-30T12:35:54.524 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/179/14' 2023-10-30T12:35:54.524 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/179/18' 2023-10-30T12:35:54.524 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/179/13' 2023-10-30T12:35:54.525 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/179/17' 2023-10-30T12:35:54.525 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/179' 2023-10-30T12:35:54.525 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/132/19' 2023-10-30T12:35:54.525 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/132/15' 2023-10-30T12:35:54.525 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/132/20' 2023-10-30T12:35:54.526 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/132/12' 2023-10-30T12:35:54.526 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/132/10' 2023-10-30T12:35:54.526 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/132/11' 2023-10-30T12:35:54.526 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/132/16' 2023-10-30T12:35:54.526 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/132/14' 2023-10-30T12:35:54.527 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/132/18' 2023-10-30T12:35:54.527 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/132/13' 2023-10-30T12:35:54.527 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/132/17' 2023-10-30T12:35:54.527 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/132' 2023-10-30T12:35:54.527 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/199/19' 2023-10-30T12:35:54.528 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/199/15' 2023-10-30T12:35:54.528 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/199/20' 2023-10-30T12:35:54.528 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/199/12' 2023-10-30T12:35:54.528 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/199/10' 2023-10-30T12:35:54.528 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/199/11' 2023-10-30T12:35:54.529 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/199/16' 2023-10-30T12:35:54.529 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/199/14' 2023-10-30T12:35:54.529 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/199/18' 2023-10-30T12:35:54.529 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/199/13' 2023-10-30T12:35:54.529 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/199/17' 2023-10-30T12:35:54.529 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/199' 2023-10-30T12:35:54.530 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/163/19' 2023-10-30T12:35:54.530 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/163/15' 2023-10-30T12:35:54.530 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/163/20' 2023-10-30T12:35:54.530 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/163/12' 2023-10-30T12:35:54.530 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/163/10' 2023-10-30T12:35:54.531 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/163/11' 2023-10-30T12:35:54.531 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/163/16' 2023-10-30T12:35:54.531 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/163/14' 2023-10-30T12:35:54.531 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/163/18' 2023-10-30T12:35:54.531 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/163/13' 2023-10-30T12:35:54.532 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/163/17' 2023-10-30T12:35:54.532 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/163' 2023-10-30T12:35:54.532 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/153/19' 2023-10-30T12:35:54.532 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/153/15' 2023-10-30T12:35:54.532 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/153/20' 2023-10-30T12:35:54.532 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/153/12' 2023-10-30T12:35:54.533 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/153/10' 2023-10-30T12:35:54.533 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/153/11' 2023-10-30T12:35:54.533 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/153/16' 2023-10-30T12:35:54.533 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/153/14' 2023-10-30T12:35:54.533 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/153/18' 2023-10-30T12:35:54.534 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/153/13' 2023-10-30T12:35:54.534 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/153/17' 2023-10-30T12:35:54.534 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/153' 2023-10-30T12:35:54.534 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/122/19' 2023-10-30T12:35:54.534 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/122/15' 2023-10-30T12:35:54.535 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/122/20' 2023-10-30T12:35:54.535 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/122/12' 2023-10-30T12:35:54.535 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/122/10' 2023-10-30T12:35:54.535 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/122/11' 2023-10-30T12:35:54.535 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/122/16' 2023-10-30T12:35:54.536 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/122/14' 2023-10-30T12:35:54.536 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/122/18' 2023-10-30T12:35:54.536 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/122/13' 2023-10-30T12:35:54.536 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/122/17' 2023-10-30T12:35:54.536 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/122' 2023-10-30T12:35:54.536 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/162/19' 2023-10-30T12:35:54.537 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/162/15' 2023-10-30T12:35:54.537 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/162/20' 2023-10-30T12:35:54.537 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/162/12' 2023-10-30T12:35:54.537 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/162/10' 2023-10-30T12:35:54.537 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/162/11' 2023-10-30T12:35:54.538 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/162/16' 2023-10-30T12:35:54.538 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/162/14' 2023-10-30T12:35:54.538 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/162/18' 2023-10-30T12:35:54.538 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/162/13' 2023-10-30T12:35:54.538 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/162/17' 2023-10-30T12:35:54.539 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/162' 2023-10-30T12:35:54.539 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/197/19' 2023-10-30T12:35:54.539 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/197/15' 2023-10-30T12:35:54.539 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/197/20' 2023-10-30T12:35:54.539 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/197/12' 2023-10-30T12:35:54.539 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/197/10' 2023-10-30T12:35:54.540 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/197/11' 2023-10-30T12:35:54.540 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/197/16' 2023-10-30T12:35:54.540 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/197/14' 2023-10-30T12:35:54.540 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/197/18' 2023-10-30T12:35:54.540 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/197/13' 2023-10-30T12:35:54.541 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/197/17' 2023-10-30T12:35:54.541 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/197' 2023-10-30T12:35:54.541 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/113/19' 2023-10-30T12:35:54.541 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/113/15' 2023-10-30T12:35:54.541 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/113/20' 2023-10-30T12:35:54.542 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/113/12' 2023-10-30T12:35:54.542 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/113/10' 2023-10-30T12:35:54.542 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/113/11' 2023-10-30T12:35:54.542 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/113/16' 2023-10-30T12:35:54.542 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/113/14' 2023-10-30T12:35:54.542 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/113/18' 2023-10-30T12:35:54.543 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/113/13' 2023-10-30T12:35:54.543 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/113/17' 2023-10-30T12:35:54.543 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/113' 2023-10-30T12:35:54.543 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/152/19' 2023-10-30T12:35:57.167 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/152/15' 2023-10-30T12:35:57.168 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/152/20' 2023-10-30T12:35:57.168 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/152/12' 2023-10-30T12:35:57.168 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/152/10' 2023-10-30T12:35:57.168 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/152/11' 2023-10-30T12:35:57.169 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/152/16' 2023-10-30T12:35:57.169 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/152/14' 2023-10-30T12:35:57.169 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/152/18' 2023-10-30T12:35:57.169 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/152/13' 2023-10-30T12:35:57.169 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/152/17' 2023-10-30T12:35:57.170 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/152' 2023-10-30T12:35:57.170 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/121/19' 2023-10-30T12:35:57.170 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/121/15' 2023-10-30T12:35:57.171 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/121/20' 2023-10-30T12:35:57.171 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/121/12' 2023-10-30T12:35:57.171 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/121/10' 2023-10-30T12:35:57.171 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/121/11' 2023-10-30T12:35:57.171 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/121/16' 2023-10-30T12:35:57.171 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/121/14' 2023-10-30T12:35:57.172 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/121/18' 2023-10-30T12:35:57.172 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/121/13' 2023-10-30T12:35:57.172 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/121/17' 2023-10-30T12:35:57.172 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/121' 2023-10-30T12:35:57.172 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/118/19' 2023-10-30T12:35:57.173 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/118/15' 2023-10-30T12:35:57.173 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/118/20' 2023-10-30T12:35:57.173 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/118/12' 2023-10-30T12:35:57.173 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/118/10' 2023-10-30T12:35:57.173 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/118/11' 2023-10-30T12:35:57.174 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/118/16' 2023-10-30T12:35:57.174 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/118/14' 2023-10-30T12:35:57.174 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/118/18' 2023-10-30T12:35:57.174 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/118/13' 2023-10-30T12:35:57.174 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/118/17' 2023-10-30T12:35:57.175 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/118' 2023-10-30T12:35:57.175 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/196/19' 2023-10-30T12:35:57.175 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/196/15' 2023-10-30T12:35:57.175 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/196/20' 2023-10-30T12:35:57.176 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/196/12' 2023-10-30T12:35:57.176 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/196/10' 2023-10-30T12:35:57.176 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/196/11' 2023-10-30T12:35:57.176 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/196/16' 2023-10-30T12:35:57.177 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/196/14' 2023-10-30T12:35:57.177 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/196/18' 2023-10-30T12:35:57.177 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/196/13' 2023-10-30T12:35:57.177 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/196/17' 2023-10-30T12:35:57.178 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/196' 2023-10-30T12:35:57.178 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/124/19' 2023-10-30T12:35:57.178 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/124/15' 2023-10-30T12:35:57.178 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/124/20' 2023-10-30T12:35:57.178 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/124/12' 2023-10-30T12:35:57.179 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/124/10' 2023-10-30T12:35:57.179 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/124/11' 2023-10-30T12:35:57.179 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/124/16' 2023-10-30T12:35:57.179 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/124/14' 2023-10-30T12:35:57.180 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/124/18' 2023-10-30T12:35:57.180 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/124/13' 2023-10-30T12:35:57.180 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/124/17' 2023-10-30T12:35:57.180 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/124' 2023-10-30T12:35:57.180 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/188/19' 2023-10-30T12:35:57.181 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/188/15' 2023-10-30T12:35:57.181 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/188/20' 2023-10-30T12:35:57.181 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/188/12' 2023-10-30T12:35:57.181 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/188/10' 2023-10-30T12:35:57.181 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/188/11' 2023-10-30T12:35:57.182 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/188/16' 2023-10-30T12:35:57.182 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/188/14' 2023-10-30T12:35:57.182 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/188/18' 2023-10-30T12:35:57.182 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/188/13' 2023-10-30T12:35:57.182 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/188/17' 2023-10-30T12:35:57.183 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/188' 2023-10-30T12:35:57.183 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/145/19' 2023-10-30T12:35:57.183 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/145/15' 2023-10-30T12:35:57.183 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/145/20' 2023-10-30T12:35:57.183 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/145/12' 2023-10-30T12:35:57.184 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/145/10' 2023-10-30T12:35:57.184 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/145/11' 2023-10-30T12:35:57.184 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/145/16' 2023-10-30T12:35:57.184 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/145/14' 2023-10-30T12:35:57.184 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/145/18' 2023-10-30T12:35:57.185 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/145/13' 2023-10-30T12:35:57.185 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/145/17' 2023-10-30T12:35:57.185 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/145' 2023-10-30T12:35:57.185 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/130/19' 2023-10-30T12:35:57.185 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/130/15' 2023-10-30T12:35:57.186 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/130/20' 2023-10-30T12:35:57.186 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/130/12' 2023-10-30T12:35:57.186 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/130/10' 2023-10-30T12:35:57.186 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/130/11' 2023-10-30T12:35:57.186 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/130/16' 2023-10-30T12:35:57.187 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/130/14' 2023-10-30T12:35:57.187 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/130/18' 2023-10-30T12:35:57.187 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/130/13' 2023-10-30T12:35:57.187 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/130/17' 2023-10-30T12:35:57.187 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/130' 2023-10-30T12:35:57.188 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/193/19' 2023-10-30T12:35:57.188 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/193/15' 2023-10-30T12:35:57.188 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/193/20' 2023-10-30T12:35:57.188 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/193/12' 2023-10-30T12:35:57.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/193/10' 2023-10-30T12:35:57.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/193/11' 2023-10-30T12:35:57.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/193/16' 2023-10-30T12:35:57.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/193/14' 2023-10-30T12:35:57.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/193/18' 2023-10-30T12:35:57.189 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/193/13' 2023-10-30T12:35:57.190 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/193/17' 2023-10-30T12:35:57.190 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/193' 2023-10-30T12:35:57.190 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/175/19' 2023-10-30T12:35:57.190 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/175/15' 2023-10-30T12:35:59.926 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/175/20' 2023-10-30T12:35:59.927 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/175/12' 2023-10-30T12:35:59.927 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/175/10' 2023-10-30T12:35:59.927 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/175/11' 2023-10-30T12:35:59.927 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/175/16' 2023-10-30T12:35:59.928 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/175/14' 2023-10-30T12:35:59.928 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/175/18' 2023-10-30T12:35:59.928 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/175/13' 2023-10-30T12:35:59.928 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/175/17' 2023-10-30T12:35:59.928 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/175' 2023-10-30T12:35:59.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/181/19' 2023-10-30T12:35:59.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/181/15' 2023-10-30T12:35:59.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/181/20' 2023-10-30T12:35:59.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/181/12' 2023-10-30T12:35:59.929 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/181/10' 2023-10-30T12:35:59.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/181/11' 2023-10-30T12:35:59.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/181/16' 2023-10-30T12:35:59.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/181/14' 2023-10-30T12:35:59.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/181/18' 2023-10-30T12:35:59.930 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/181/13' 2023-10-30T12:35:59.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/181/17' 2023-10-30T12:35:59.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/181' 2023-10-30T12:35:59.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/127/19' 2023-10-30T12:35:59.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/127/15' 2023-10-30T12:35:59.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/127/20' 2023-10-30T12:35:59.931 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/127/12' 2023-10-30T12:35:59.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/127/10' 2023-10-30T12:35:59.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/127/11' 2023-10-30T12:35:59.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/127/16' 2023-10-30T12:35:59.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/127/14' 2023-10-30T12:35:59.932 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/127/18' 2023-10-30T12:35:59.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/127/13' 2023-10-30T12:35:59.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/127/17' 2023-10-30T12:35:59.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/127' 2023-10-30T12:35:59.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/108/19' 2023-10-30T12:35:59.933 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/108/15' 2023-10-30T12:35:59.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/108/20' 2023-10-30T12:35:59.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/108/12' 2023-10-30T12:35:59.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/108/10' 2023-10-30T12:35:59.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/108/11' 2023-10-30T12:35:59.934 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/108/16' 2023-10-30T12:35:59.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/108/14' 2023-10-30T12:35:59.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/108/18' 2023-10-30T12:35:59.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/108/13' 2023-10-30T12:35:59.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/108/17' 2023-10-30T12:35:59.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/108' 2023-10-30T12:35:59.935 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/183/19' 2023-10-30T12:35:59.936 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/183/15' 2023-10-30T12:35:59.936 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/183/20' 2023-10-30T12:35:59.936 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/183/12' 2023-10-30T12:35:59.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/183/10' 2023-10-30T12:35:59.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/183/11' 2023-10-30T12:35:59.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/183/16' 2023-10-30T12:35:59.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/183/14' 2023-10-30T12:35:59.937 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/183/18' 2023-10-30T12:35:59.938 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/183/13' 2023-10-30T12:35:59.938 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/183/17' 2023-10-30T12:35:59.938 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/183' 2023-10-30T12:35:59.938 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/177/19' 2023-10-30T12:35:59.938 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/177/15' 2023-10-30T12:35:59.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/177/20' 2023-10-30T12:35:59.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/177/12' 2023-10-30T12:35:59.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/177/10' 2023-10-30T12:35:59.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/177/11' 2023-10-30T12:35:59.939 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/177/16' 2023-10-30T12:35:59.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/177/14' 2023-10-30T12:35:59.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/177/18' 2023-10-30T12:35:59.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/177/13' 2023-10-30T12:35:59.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/177/17' 2023-10-30T12:35:59.940 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/177' 2023-10-30T12:35:59.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/169/19' 2023-10-30T12:35:59.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/169/15' 2023-10-30T12:35:59.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/169/20' 2023-10-30T12:35:59.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/169/12' 2023-10-30T12:35:59.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/169/10' 2023-10-30T12:35:59.941 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/169/11' 2023-10-30T12:35:59.942 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/169/16' 2023-10-30T12:35:59.942 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/169/14' 2023-10-30T12:35:59.942 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/169/18' 2023-10-30T12:35:59.942 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/169/13' 2023-10-30T12:35:59.943 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/169/17' 2023-10-30T12:35:59.943 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/169' 2023-10-30T12:35:59.943 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/154/19' 2023-10-30T12:35:59.943 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/154/15' 2023-10-30T12:35:59.944 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/154/20' 2023-10-30T12:35:59.945 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/154/12' 2023-10-30T12:35:59.945 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/154/10' 2023-10-30T12:35:59.945 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/154/11' 2023-10-30T12:35:59.945 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/154/16' 2023-10-30T12:35:59.945 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/154/14' 2023-10-30T12:35:59.946 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/154/18' 2023-10-30T12:35:59.946 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/154/13' 2023-10-30T12:35:59.946 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/154/17' 2023-10-30T12:35:59.946 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/154' 2023-10-30T12:35:59.947 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/171/19' 2023-10-30T12:35:59.947 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/171/15' 2023-10-30T12:35:59.947 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/171/20' 2023-10-30T12:35:59.947 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/171/12' 2023-10-30T12:35:59.948 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/171/10' 2023-10-30T12:35:59.948 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/171/11' 2023-10-30T12:35:59.948 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/171/16' 2023-10-30T12:35:59.948 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/171/14' 2023-10-30T12:35:59.949 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/171/18' 2023-10-30T12:35:59.949 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/171/13' 2023-10-30T12:35:59.949 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/171/17' 2023-10-30T12:35:59.949 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/171' 2023-10-30T12:35:59.949 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/139/19' 2023-10-30T12:35:59.950 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/139/15' 2023-10-30T12:35:59.950 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/139/20' 2023-10-30T12:36:00.543 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/139/12' 2023-10-30T12:36:00.543 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/139/10' 2023-10-30T12:36:00.543 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/139/11' 2023-10-30T12:36:00.543 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/139/16' 2023-10-30T12:36:00.544 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/139/14' 2023-10-30T12:36:00.544 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/139/18' 2023-10-30T12:36:00.544 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/139/13' 2023-10-30T12:36:00.544 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/139/17' 2023-10-30T12:36:00.544 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/139' 2023-10-30T12:36:00.545 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/102/19' 2023-10-30T12:36:00.545 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/102/15' 2023-10-30T12:36:00.545 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/102/20' 2023-10-30T12:36:00.545 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/102/12' 2023-10-30T12:36:00.545 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/102/10' 2023-10-30T12:36:00.546 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/102/11' 2023-10-30T12:36:00.546 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/102/16' 2023-10-30T12:36:00.546 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/102/14' 2023-10-30T12:36:00.546 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/102/18' 2023-10-30T12:36:00.547 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/102/13' 2023-10-30T12:36:00.547 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/102/17' 2023-10-30T12:36:00.547 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/102' 2023-10-30T12:36:00.547 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/129/19' 2023-10-30T12:36:00.547 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/129/15' 2023-10-30T12:36:00.547 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/129/20' 2023-10-30T12:36:00.548 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/129/12' 2023-10-30T12:36:00.548 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/129/10' 2023-10-30T12:36:00.548 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/129/11' 2023-10-30T12:36:00.548 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/129/16' 2023-10-30T12:36:00.549 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/129/14' 2023-10-30T12:36:00.549 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/129/18' 2023-10-30T12:36:00.549 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/129/13' 2023-10-30T12:36:00.549 INFO:tasks.workunit.client.0.smithi125.stdout:removed './copyofsnap1/test1/129/17' 2023-10-30T12:36:00.549 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1/129' 2023-10-30T12:36:00.550 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1/test1' 2023-10-30T12:36:00.550 INFO:tasks.workunit.client.0.smithi125.stdout:removed directory './copyofsnap1' 2023-10-30T12:36:00.550 INFO:tasks.workunit.client.0.smithi125.stdout:OK 2023-10-30T12:36:00.551 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:36:00.552 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:36:00.937 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2023-10-30T12:36:00.938 DEBUG:teuthology.orchestra.run.smithi125:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2023-10-30T12:36:00.955 INFO:tasks.workunit.client.0.smithi125.stderr:+ set -e 2023-10-30T12:36:00.955 INFO:tasks.workunit.client.0.smithi125.stderr:+ touch foo 2023-10-30T12:36:00.957 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod +x foo 2023-10-30T12:36:00.958 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/s 2023-10-30T12:36:00.972 INFO:tasks.workunit.client.0.smithi125.stderr:+ find .snap/s/foo -executable 2023-10-30T12:36:00.973 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep foo 2023-10-30T12:36:00.977 INFO:tasks.workunit.client.0.smithi125.stdout:.snap/s/foo 2023-10-30T12:36:00.977 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir .snap/s 2023-10-30T12:36:00.987 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm foo 2023-10-30T12:36:00.991 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo OK 2023-10-30T12:36:00.992 INFO:tasks.workunit.client.0.smithi125.stdout:OK 2023-10-30T12:36:00.992 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:36:00.992 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:36:01.064 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2023-10-30T12:36:01.065 DEBUG:teuthology.orchestra.run.smithi125:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2023-10-30T12:36:01.127 INFO:tasks.workunit.client.0.smithi125.stderr:+ set -e 2023-10-30T12:36:01.127 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir foo 2023-10-30T12:36:01.130 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo x 2023-10-30T12:36:01.130 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:36:01.783 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir foo/.snap/ss 2023-10-30T12:36:01.794 INFO:tasks.workunit.client.0.smithi125.stderr:+ ln foo/x foo/xx 2023-10-30T12:36:01.796 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat foo/.snap/ss/x 2023-10-30T12:36:01.804 INFO:tasks.workunit.client.0.smithi125.stdout:x 2023-10-30T12:36:01.804 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir foo/.snap/ss 2023-10-30T12:36:01.815 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo a 2023-10-30T12:36:01.815 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo b 2023-10-30T12:36:01.815 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir foo/.snap/s 2023-10-30T12:36:02.069 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat foo/.snap/s/a 2023-10-30T12:36:02.081 INFO:tasks.workunit.client.0.smithi125.stderr:+ r=a 2023-10-30T12:36:02.081 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ -z a ] 2023-10-30T12:36:02.082 INFO:tasks.workunit.client.0.smithi125.stderr:+ ln foo/b foo/b2 2023-10-30T12:36:02.083 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat foo/.snap/s/b 2023-10-30T12:36:02.093 INFO:tasks.workunit.client.0.smithi125.stdout:b 2023-10-30T12:36:02.094 INFO:tasks.workunit.client.0.smithi125.stdout:this used to hang: 2023-10-30T12:36:02.094 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo this used to hang: 2023-10-30T12:36:02.094 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo more 2023-10-30T12:36:02.095 INFO:tasks.workunit.client.0.smithi125.stdout:oh, it didn't hang! good job. 2023-10-30T12:36:02.095 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo oh, it didn't hang! good job. 2023-10-30T12:36:02.095 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat foo/b 2023-10-30T12:36:02.096 INFO:tasks.workunit.client.0.smithi125.stdout:b 2023-10-30T12:36:02.096 INFO:tasks.workunit.client.0.smithi125.stdout:more 2023-10-30T12:36:02.097 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir foo/.snap/s 2023-10-30T12:36:02.104 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm -r foo 2023-10-30T12:36:02.124 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo OK 2023-10-30T12:36:02.125 INFO:tasks.workunit.client.0.smithi125.stdout:OK 2023-10-30T12:36:02.125 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:36:02.126 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:36:02.204 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2023-10-30T12:36:02.204 DEBUG:teuthology.orchestra.run.smithi125:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2023-10-30T12:36:02.267 INFO:tasks.workunit.client.0.smithi125.stderr:+ set -e 2023-10-30T12:36:02.268 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:36:02.269 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/bar 2023-10-30T12:36:02.286 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/foo 2023-10-30T12:36:02.297 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/foo 2023-10-30T12:36:02.306 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:36:02.308 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/bar/a 2023-10-30T12:36:05.238 INFO:tasks.workunit.client.0.smithi125.stdout: File: .snap/bar/a 2023-10-30T12:36:05.238 INFO:tasks.workunit.client.0.smithi125.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2023-10-30T12:36:05.239 INFO:tasks.workunit.client.0.smithi125.stdout:Device: fah/250d Inode: 1099511631569 Links: 2 2023-10-30T12:36:05.239 INFO:tasks.workunit.client.0.smithi125.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1271/ ubuntu) 2023-10-30T12:36:05.239 INFO:tasks.workunit.client.0.smithi125.stdout:Access: 2023-10-30 12:36:02.265517187 +0000 2023-10-30T12:36:05.239 INFO:tasks.workunit.client.0.smithi125.stdout:Modify: 2023-10-30 12:36:02.265517187 +0000 2023-10-30T12:36:05.239 INFO:tasks.workunit.client.0.smithi125.stdout:Change: 2023-10-30 12:36:02.265517187 +0000 2023-10-30T12:36:05.239 INFO:tasks.workunit.client.0.smithi125.stdout: Birth: 2023-10-30 12:36:02.265517187 +0000 2023-10-30T12:36:05.240 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir .snap/bar 2023-10-30T12:36:05.250 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo OK 2023-10-30T12:36:05.251 INFO:tasks.workunit.client.0.smithi125.stdout:OK 2023-10-30T12:36:05.251 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:36:05.252 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:36:05.326 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2023-10-30T12:36:05.327 DEBUG:teuthology.orchestra.run.smithi125:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2023-10-30T12:36:10.141 INFO:tasks.workunit.client.0.smithi125.stderr:+ set -e 2023-10-30T12:36:10.142 INFO:tasks.workunit.client.0.smithi125.stderr:+ seq 1 20 2023-10-30T12:36:10.143 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:36:10.146 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:36:10.147 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:36:10.429 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:36:10.439 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:36:10.440 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:36:10.989 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:36:11.004 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:36:11.005 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:36:11.005 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:36:11.005 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:36:11.005 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:36:11.006 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:36:11.006 INFO:tasks.workunit.client.0.smithi125.stdout: 49962 ? 00:00:00 ps 2023-10-30T12:36:11.006 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:36:11.022 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:36:11.026 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:36:11.066 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:36:11.069 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:36:11.070 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:36:11.759 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:36:11.771 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:36:11.772 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:36:11.856 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:36:11.871 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:36:11.872 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:36:11.872 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:36:11.872 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:36:11.872 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:36:11.873 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:36:11.873 INFO:tasks.workunit.client.0.smithi125.stdout: 49972 ? 00:00:00 ps 2023-10-30T12:36:11.873 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:36:11.889 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:36:11.892 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:36:11.902 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:36:11.903 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:36:11.904 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:36:11.918 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:36:11.927 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:36:11.929 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:36:11.978 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:36:11.993 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:36:11.993 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:36:11.994 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:36:11.994 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:36:11.994 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:36:11.994 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:36:11.995 INFO:tasks.workunit.client.0.smithi125.stdout: 49982 ? 00:00:00 ps 2023-10-30T12:36:11.995 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:36:12.010 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:36:12.013 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:36:12.021 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:36:12.023 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:36:12.024 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:36:12.231 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:12.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.373+0000 7fc269a6b640 1 Processor -- start 2023-10-30T12:36:12.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.373+0000 7fc269a6b640 1 -- start start 2023-10-30T12:36:12.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.373+0000 7fc269a6b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc26415d6f0 0x7fc26415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:12.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.373+0000 7fc269a6b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2640cb800 con 0x7fc2640cf160 2023-10-30T12:36:12.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.373+0000 7fc269a6b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2640cb330 con 0x7fc2641538a0 2023-10-30T12:36:12.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.373+0000 7fc269a6b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2640cad70 con 0x7fc26415d6f0 2023-10-30T12:36:12.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.373+0000 7fc262ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc26415d6f0 0x7fc26415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:12.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.373+0000 7fc263fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc2640cf160 0x7fc264164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55478/0 (socket says 172.21.15.125:55478) 2023-10-30T12:36:12.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.373+0000 7fc263fff640 1 -- 172.21.15.125:0/1155049468 learned_addr learned my addr 172.21.15.125:0/1155049468 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:12.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.373+0000 7fc262ffd640 1 -- 172.21.15.125:0/1155049468 >> v1:172.21.15.136:6789/0 conn(0x7fc2641538a0 legacy=0x7fc2640ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:12.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.373+0000 7fc262ffd640 1 -- 172.21.15.125:0/1155049468 >> v1:172.21.15.125:6789/0 conn(0x7fc2640cf160 legacy=0x7fc264164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:12.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.373+0000 7fc2627fc640 1 -- 172.21.15.125:0/1155049468 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494221412 0 0) 0x7fc2640cb800 con 0x7fc2640cf160 2023-10-30T12:36:12.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.373+0000 7fc262ffd640 1 -- 172.21.15.125:0/1155049468 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2640cb050 con 0x7fc26415d6f0 2023-10-30T12:36:12.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.377+0000 7fc262ffd640 1 --2- 172.21.15.125:0/1155049468 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc26415d6f0 0x7fc26415a090 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc2580097c0 tx=0x7fc25801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5098e22707ecad8b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:12.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.377+0000 7fc2627fc640 1 -- 172.21.15.125:0/1155049468 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2580a8970 con 0x7fc26415d6f0 2023-10-30T12:36:12.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.377+0000 7fc2627fc640 1 -- 172.21.15.125:0/1155049468 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2580a8ad0 con 0x7fc26415d6f0 2023-10-30T12:36:12.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.377+0000 7fc2627fc640 1 -- 172.21.15.125:0/1155049468 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2580a8e60 con 0x7fc26415d6f0 2023-10-30T12:36:12.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.377+0000 7fc269a6b640 1 -- 172.21.15.125:0/1155049468 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc26415d6f0 msgr2=0x7fc26415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:12.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.377+0000 7fc269a6b640 1 --2- 172.21.15.125:0/1155049468 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc26415d6f0 0x7fc26415a090 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc2580097c0 tx=0x7fc25801ec70 comp rx=0 tx=0).stop 2023-10-30T12:36:12.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.377+0000 7fc269a6b640 1 -- 172.21.15.125:0/1155049468 shutdown_connections 2023-10-30T12:36:12.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.377+0000 7fc269a6b640 1 --2- 172.21.15.125:0/1155049468 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc26415d6f0 0x7fc26415a090 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.377+0000 7fc269a6b640 1 -- 172.21.15.125:0/1155049468 >> 172.21.15.125:0/1155049468 conn(0x7fc26406bfe0 msgr2=0x7fc26406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:12.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.377+0000 7fc269a6b640 1 -- 172.21.15.125:0/1155049468 shutdown_connections 2023-10-30T12:36:12.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.377+0000 7fc269a6b640 1 -- 172.21.15.125:0/1155049468 wait complete. 2023-10-30T12:36:12.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc269a6b640 1 Processor -- start 2023-10-30T12:36:12.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc269a6b640 1 -- start start 2023-10-30T12:36:12.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc269a6b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2640cf160 0x7fc26414ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:12.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc269a6b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2641538a0 0x7fc2640ce710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:12.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc269a6b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc26415d6f0 0x7fc2640c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:12.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc269a6b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2641ee880 con 0x7fc26415d6f0 2023-10-30T12:36:12.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc269a6b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26414d380 con 0x7fc2641538a0 2023-10-30T12:36:12.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc269a6b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26415cfa0 con 0x7fc2640cf160 2023-10-30T12:36:12.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc263fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc26415d6f0 0x7fc2640c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:12.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc263fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc26415d6f0 0x7fc2640c4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34706/0 (socket says 172.21.15.125:34706) 2023-10-30T12:36:12.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc263fff640 1 -- 172.21.15.125:0/2630961968 learned_addr learned my addr 172.21.15.125:0/2630961968 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:12.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc262ffd640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2641538a0 0x7fc2640ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:12.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc2637fe640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2640cf160 0x7fc26414ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:12.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc263fff640 1 -- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2640cf160 msgr2=0x7fc26414ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:12.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc263fff640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2640cf160 0x7fc26414ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc263fff640 1 -- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2641538a0 msgr2=0x7fc2640ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:12.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc263fff640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2641538a0 0x7fc2640ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc263fff640 1 -- 172.21.15.125:0/2630961968 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc258009420 con 0x7fc26415d6f0 2023-10-30T12:36:12.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc2637fe640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2640cf160 0x7fc26414ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:36:12.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc263fff640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc26415d6f0 0x7fc2640c4070 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc25c00c7b0 tx=0x7fc25c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:12.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc24bfff640 1 -- 172.21.15.125:0/2630961968 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c00ce60 con 0x7fc26415d6f0 2023-10-30T12:36:12.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc24bfff640 1 -- 172.21.15.125:0/2630961968 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25c00a640 con 0x7fc26415d6f0 2023-10-30T12:36:12.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc269a6b640 1 -- 172.21.15.125:0/2630961968 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2641eea70 con 0x7fc26415d6f0 2023-10-30T12:36:12.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc24bfff640 1 -- 172.21.15.125:0/2630961968 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25c00a9d0 con 0x7fc26415d6f0 2023-10-30T12:36:12.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc269a6b640 1 -- 172.21.15.125:0/2630961968 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2641ef030 con 0x7fc26415d6f0 2023-10-30T12:36:12.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.381+0000 7fc269a6b640 1 -- 172.21.15.125:0/2630961968 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26404e170 con 0x7fc26415d6f0 2023-10-30T12:36:12.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.385+0000 7fc24bfff640 1 -- 172.21.15.125:0/2630961968 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc25c00ab30 con 0x7fc26415d6f0 2023-10-30T12:36:12.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.385+0000 7fc24bfff640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2500664a0 0x7fc250068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.385+0000 7fc24bfff640 1 -- 172.21.15.125:0/2630961968 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6068+0+0 (secure 0 0 0) 0x7fc25c083700 con 0x7fc26415d6f0 2023-10-30T12:36:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.385+0000 7fc2637fe640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2500664a0 0x7fc250068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.385+0000 7fc2637fe640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2500664a0 0x7fc250068960 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc254000f70 tx=0x7fc254009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:12.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.389+0000 7fc24bfff640 1 -- 172.21.15.125:0/2630961968 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc25c050a20 con 0x7fc26415d6f0 2023-10-30T12:36:12.445 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:36:12.454 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:36:12.455 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:36:12.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.573+0000 7fc269a6b640 1 -- 172.21.15.125:0/2630961968 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2640ca9b0 con 0x7fc26415d6f0 2023-10-30T12:36:12.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.577+0000 7fc24bfff640 1 -- 172.21.15.125:0/2630961968 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+6551 (secure 0 0 0) 0x7fc25c054040 con 0x7fc26415d6f0 2023-10-30T12:36:12.580 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:12.580 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4973,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:03.067774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:12.580 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 44 2023-10-30T12:36:12.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.577+0000 7fc269a6b640 1 -- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2500664a0 msgr2=0x7fc250068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:12.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.577+0000 7fc269a6b640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2500664a0 0x7fc250068960 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fc254000f70 tx=0x7fc254009040 comp rx=0 tx=0).stop 2023-10-30T12:36:12.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.577+0000 7fc269a6b640 1 -- 172.21.15.125:0/2630961968 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc26415d6f0 msgr2=0x7fc2640c4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:12.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.577+0000 7fc269a6b640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc26415d6f0 0x7fc2640c4070 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc25c00c7b0 tx=0x7fc25c00cc70 comp rx=0 tx=0).stop 2023-10-30T12:36:12.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.577+0000 7fc269a6b640 1 -- 172.21.15.125:0/2630961968 shutdown_connections 2023-10-30T12:36:12.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.577+0000 7fc269a6b640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2500664a0 0x7fc250068960 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.577+0000 7fc269a6b640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc26415d6f0 0x7fc2640c4070 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.577+0000 7fc269a6b640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2641538a0 0x7fc2640ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.577+0000 7fc269a6b640 1 --2- 172.21.15.125:0/2630961968 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2640cf160 0x7fc26414ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.577+0000 7fc269a6b640 1 -- 172.21.15.125:0/2630961968 >> 172.21.15.125:0/2630961968 conn(0x7fc26406bfe0 msgr2=0x7fc264152a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:12.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.577+0000 7fc269a6b640 1 -- 172.21.15.125:0/2630961968 shutdown_connections 2023-10-30T12:36:12.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.577+0000 7fc269a6b640 1 -- 172.21.15.125:0/2630961968 wait complete. 2023-10-30T12:36:12.600 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T12:36:12.600 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T12:36:12.600 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:36:12.613 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:36:12.617 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T12:36:12.617 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:12.628 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:36:12.628 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:36:12.628 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:36:12.629 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:36:12.629 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:36:12.629 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:36:12.629 INFO:tasks.workunit.client.0.smithi125.stdout: 50018 ? 00:00:00 ps 2023-10-30T12:36:12.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.761+0000 7f4213765640 1 Processor -- start 2023-10-30T12:36:12.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.761+0000 7f4213765640 1 -- start start 2023-10-30T12:36:12.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4213765640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f420c1538a0 0x7f420c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:12.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4213765640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f420c0cb800 con 0x7f420c0ca0e0 2023-10-30T12:36:12.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4213765640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f420c15dc30 con 0x7f420c0cf160 2023-10-30T12:36:12.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4213765640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f420c0cb330 con 0x7f420c1538a0 2023-10-30T12:36:12.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4210d00640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f420c0ca0e0 0x7f420c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55488/0 (socket says 172.21.15.125:55488) 2023-10-30T12:36:12.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4210d00640 1 -- 172.21.15.125:0/3906217573 learned_addr learned my addr 172.21.15.125:0/3906217573 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:12.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4211501640 1 --2- 172.21.15.125:0/3906217573 >> v2:172.21.15.136:3300/0 conn(0x7f420c1538a0 0x7f420c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:12.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 -- 172.21.15.125:0/3906217573 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607197985 0 0) 0x7f420c0cb800 con 0x7f420c0ca0e0 2023-10-30T12:36:12.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 -- 172.21.15.125:0/3906217573 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f8003530 con 0x7f420c0ca0e0 2023-10-30T12:36:12.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 -- 172.21.15.125:0/3906217573 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2275226844 0 0) 0x7f41f8003530 con 0x7f420c0ca0e0 2023-10-30T12:36:12.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 -- 172.21.15.125:0/3906217573 >> v2:172.21.15.136:3300/0 conn(0x7f420c1538a0 msgr2=0x7f420c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:12.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 --2- 172.21.15.125:0/3906217573 >> v2:172.21.15.136:3300/0 conn(0x7f420c1538a0 0x7f420c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 -- 172.21.15.125:0/3906217573 >> v1:172.21.15.136:6789/0 conn(0x7f420c0cf160 legacy=0x7f420c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:12.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 -- 172.21.15.125:0/3906217573 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c0cad70 con 0x7f420c0ca0e0 2023-10-30T12:36:12.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 -- 172.21.15.125:0/3906217573 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f41fc001680 con 0x7f420c0ca0e0 2023-10-30T12:36:12.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 -- 172.21.15.125:0/3906217573 >> v1:172.21.15.125:6789/0 conn(0x7f420c0ca0e0 legacy=0x7f420c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:12.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 --2- 172.21.15.125:0/3906217573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f8003980 0x7f41f8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:12.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 --2- 172.21.15.125:0/3906217573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41f80047d0 0x7f41f8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:12.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 --2- 172.21.15.125:0/3906217573 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f8005190 0x7f41f800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:12.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 -- 172.21.15.125:0/3906217573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c0cb330 con 0x7f41f80047d0 2023-10-30T12:36:12.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 -- 172.21.15.125:0/3906217573 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f800c040 con 0x7f41f8003980 2023-10-30T12:36:12.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4203fff640 1 -- 172.21.15.125:0/3906217573 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f800c200 con 0x7f41f8005190 2023-10-30T12:36:12.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4210d00640 1 --2- 172.21.15.125:0/3906217573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41f80047d0 0x7f41f8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:12.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4211501640 1 --2- 172.21.15.125:0/3906217573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f8003980 0x7f41f8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:12.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4211d02640 1 --2- 172.21.15.125:0/3906217573 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f8005190 0x7f41f800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:12.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4210d00640 1 -- 172.21.15.125:0/3906217573 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f8005190 msgr2=0x7f41f800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:12.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4210d00640 1 --2- 172.21.15.125:0/3906217573 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41f8005190 0x7f41f800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4210d00640 1 -- 172.21.15.125:0/3906217573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f8003980 msgr2=0x7f41f8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:12.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4210d00640 1 --2- 172.21.15.125:0/3906217573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f8003980 0x7f41f8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4210d00640 1 -- 172.21.15.125:0/3906217573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f8003530 con 0x7f41f80047d0 2023-10-30T12:36:12.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.765+0000 7f4211d02640 1 -- 172.21.15.125:0/3906217573 reap_dead start 2023-10-30T12:36:12.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.769+0000 7f4210d00640 1 --2- 172.21.15.125:0/3906217573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41f80047d0 0x7f41f8004c50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00c1c0 tx=0x7f41fc01cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420d448c61eabed1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:12.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.769+0000 7f4203fff640 1 -- 172.21.15.125:0/3906217573 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0a5b20 con 0x7f41f80047d0 2023-10-30T12:36:12.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.769+0000 7f4203fff640 1 -- 172.21.15.125:0/3906217573 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41fc0a5c80 con 0x7f41f80047d0 2023-10-30T12:36:12.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.769+0000 7f4203fff640 1 -- 172.21.15.125:0/3906217573 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc005b20 con 0x7f41f80047d0 2023-10-30T12:36:12.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.769+0000 7f4213765640 1 -- 172.21.15.125:0/3906217573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41f80047d0 msgr2=0x7f41f8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:12.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.769+0000 7f4213765640 1 --2- 172.21.15.125:0/3906217573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41f80047d0 0x7f41f8004c50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00c1c0 tx=0x7f41fc01cc20 comp rx=0 tx=0).stop 2023-10-30T12:36:12.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.769+0000 7f4213765640 1 -- 172.21.15.125:0/3906217573 shutdown_connections 2023-10-30T12:36:12.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.769+0000 7f4213765640 1 --2- 172.21.15.125:0/3906217573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41f80047d0 0x7f41f8004c50 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.769+0000 7f4213765640 1 -- 172.21.15.125:0/3906217573 >> 172.21.15.125:0/3906217573 conn(0x7f420c06bfe0 msgr2=0x7f420c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:12.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.769+0000 7f4213765640 1 -- 172.21.15.125:0/3906217573 shutdown_connections 2023-10-30T12:36:12.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.769+0000 7f4213765640 1 -- 172.21.15.125:0/3906217573 wait complete. 2023-10-30T12:36:12.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4213765640 1 Processor -- start 2023-10-30T12:36:12.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4213765640 1 -- start start 2023-10-30T12:36:12.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4213765640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f80047d0 0x7f420c0b4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:12.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4213765640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f420c157560 0x7f420c158c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:12.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4213765640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f420c0b1960 0x7f420c159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:12.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4213765640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c0b3bb0 con 0x7f420c157560 2023-10-30T12:36:12.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4213765640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f420c1e2e20 con 0x7f41f80047d0 2023-10-30T12:36:12.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4213765640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f420c159680 con 0x7f420c0b1960 2023-10-30T12:36:12.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4210d00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f420c157560 0x7f420c158c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:12.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4210d00640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f420c157560 0x7f420c158c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34716/0 (socket says 172.21.15.125:34716) 2023-10-30T12:36:12.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4210d00640 1 -- 172.21.15.125:0/1023245668 learned_addr learned my addr 172.21.15.125:0/1023245668 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:12.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4211501640 1 --2- 172.21.15.125:0/1023245668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f80047d0 0x7f420c0b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:12.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4211d02640 1 --2- 172.21.15.125:0/1023245668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f420c0b1960 0x7f420c159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:12.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4210d00640 1 -- 172.21.15.125:0/1023245668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f420c0b1960 msgr2=0x7f420c159140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:12.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4210d00640 1 --2- 172.21.15.125:0/1023245668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f420c0b1960 0x7f420c159140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4210d00640 1 -- 172.21.15.125:0/1023245668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f80047d0 msgr2=0x7f420c0b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:12.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4210d00640 1 --2- 172.21.15.125:0/1023245668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f80047d0 0x7f420c0b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4210d00640 1 -- 172.21.15.125:0/1023245668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c1e2b80 con 0x7f420c157560 2023-10-30T12:36:12.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4210d00640 1 --2- 172.21.15.125:0/1023245668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f420c157560 0x7f420c158c00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f41fc01c690 tx=0x7f41f8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:12.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4201ffb640 1 -- 172.21.15.125:0/1023245668 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc0b3070 con 0x7f420c157560 2023-10-30T12:36:12.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4201ffb640 1 -- 172.21.15.125:0/1023245668 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41fc000ee0 con 0x7f420c157560 2023-10-30T12:36:12.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4201ffb640 1 -- 172.21.15.125:0/1023245668 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41fc004e50 con 0x7f420c157560 2023-10-30T12:36:12.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4213765640 1 -- 172.21.15.125:0/1023245668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f420c158260 con 0x7f420c157560 2023-10-30T12:36:12.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4213765640 1 -- 172.21.15.125:0/1023245668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f420c0b3e00 con 0x7f420c157560 2023-10-30T12:36:12.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.773+0000 7f4213765640 1 -- 172.21.15.125:0/1023245668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f420c158710 con 0x7f420c157560 2023-10-30T12:36:12.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.777+0000 7f4201ffb640 1 -- 172.21.15.125:0/1023245668 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41fc001440 con 0x7f420c157560 2023-10-30T12:36:12.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.777+0000 7f4201ffb640 1 --2- 172.21.15.125:0/1023245668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41f00664d0 0x7f41f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:12.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.777+0000 7f4201ffb640 1 -- 172.21.15.125:0/1023245668 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 6068+0+0 (secure 0 0 0) 0x7f41fc11d6a0 con 0x7f420c157560 2023-10-30T12:36:12.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.777+0000 7f4211501640 1 --2- 172.21.15.125:0/1023245668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41f00664d0 0x7f41f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:12.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.777+0000 7f4211501640 1 --2- 172.21.15.125:0/1023245668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41f00664d0 0x7f41f0068990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4208009c60 tx=0x7f4208009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:12.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.781+0000 7f4201ffb640 1 -- 172.21.15.125:0/1023245668 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41fc0ea910 con 0x7f420c157560 2023-10-30T12:36:12.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.961+0000 7f4213765640 1 -- 172.21.15.125:0/1023245668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f420c04e170 con 0x7f420c157560 2023-10-30T12:36:12.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.961+0000 7f4201ffb640 1 -- 172.21.15.125:0/1023245668 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f41fc0edf30 con 0x7f420c157560 2023-10-30T12:36:12.966 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:12.966 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4973,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:03.067774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:12.967 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 44 2023-10-30T12:36:12.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.965+0000 7f4213765640 1 -- 172.21.15.125:0/1023245668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41f00664d0 msgr2=0x7f41f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:12.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.965+0000 7f4213765640 1 --2- 172.21.15.125:0/1023245668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41f00664d0 0x7f41f0068990 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4208009c60 tx=0x7f4208009190 comp rx=0 tx=0).stop 2023-10-30T12:36:12.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.965+0000 7f4213765640 1 -- 172.21.15.125:0/1023245668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f420c157560 msgr2=0x7f420c158c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:12.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.965+0000 7f4213765640 1 --2- 172.21.15.125:0/1023245668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f420c157560 0x7f420c158c00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f41fc01c690 tx=0x7f41f8005800 comp rx=0 tx=0).stop 2023-10-30T12:36:12.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.965+0000 7f4213765640 1 -- 172.21.15.125:0/1023245668 shutdown_connections 2023-10-30T12:36:12.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.965+0000 7f4213765640 1 --2- 172.21.15.125:0/1023245668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41f00664d0 0x7f41f0068990 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.965+0000 7f4213765640 1 --2- 172.21.15.125:0/1023245668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f420c0b1960 0x7f420c159140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.965+0000 7f4213765640 1 --2- 172.21.15.125:0/1023245668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f420c157560 0x7f420c158c00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.965+0000 7f4213765640 1 --2- 172.21.15.125:0/1023245668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41f80047d0 0x7f420c0b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:12.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.965+0000 7f4213765640 1 -- 172.21.15.125:0/1023245668 >> 172.21.15.125:0/1023245668 conn(0x7f420c06bfe0 msgr2=0x7f420c0c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:12.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.965+0000 7f4213765640 1 -- 172.21.15.125:0/1023245668 shutdown_connections 2023-10-30T12:36:12.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:12.965+0000 7f4213765640 1 -- 172.21.15.125:0/1023245668 wait complete. 2023-10-30T12:36:12.989 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:36:14.830 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:36:14.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:36:14.990 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:15.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.133+0000 7f6ad26c9640 1 Processor -- start 2023-10-30T12:36:15.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.133+0000 7f6ad26c9640 1 -- start start 2023-10-30T12:36:15.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.137+0000 7f6ad26c9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6acc0c6fe0 0x7f6acc071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:15.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.137+0000 7f6ad26c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6acc0c5f60 0x7f6acc0cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:15.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.137+0000 7f6ad26c9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6acc0700b0 con 0x7f6acc153bf0 2023-10-30T12:36:15.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.137+0000 7f6ad26c9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6acc0c6810 con 0x7f6acc0c6fe0 2023-10-30T12:36:15.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.137+0000 7f6ad26c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc159900 con 0x7f6acc0c5f60 2023-10-30T12:36:15.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.137+0000 7f6acb7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6acc0c6fe0 0x7f6acc071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:15.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.137+0000 7f6acb7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6acc0c6fe0 0x7f6acc071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34718/0 (socket says 172.21.15.125:34718) 2023-10-30T12:36:15.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.137+0000 7f6acb7fe640 1 -- 172.21.15.125:0/775492026 learned_addr learned my addr 172.21.15.125:0/775492026 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:15.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.137+0000 7f6ad0c66640 1 --2- 172.21.15.125:0/775492026 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6acc0c5f60 0x7f6acc0cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:15.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.137+0000 7f6acb7fe640 1 -- 172.21.15.125:0/775492026 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6acc0c5f60 msgr2=0x7f6acc0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:15.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.137+0000 7f6acb7fe640 1 --2- 172.21.15.125:0/775492026 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6acc0c5f60 0x7f6acc0cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:15.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.137+0000 7f6acb7fe640 1 -- 172.21.15.125:0/775492026 >> v1:172.21.15.136:6789/0 conn(0x7f6acc153bf0 legacy=0x7f6acc0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:15.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.137+0000 7f6acb7fe640 1 -- 172.21.15.125:0/775492026 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc0c6530 con 0x7f6acc0c6fe0 2023-10-30T12:36:15.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.141+0000 7f6acb7fe640 1 --2- 172.21.15.125:0/775492026 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc0c6fe0 0x7f6acc071260 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009870 tx=0x7f6ac001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7374249ace4fa47a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:15.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.141+0000 7f6acaffd640 1 -- 172.21.15.125:0/775492026 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac00a8a30 con 0x7f6acc0c6fe0 2023-10-30T12:36:15.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.141+0000 7f6acaffd640 1 -- 172.21.15.125:0/775492026 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac00a8b90 con 0x7f6acc0c6fe0 2023-10-30T12:36:15.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.141+0000 7f6acaffd640 1 -- 172.21.15.125:0/775492026 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac00b0440 con 0x7f6acc0c6fe0 2023-10-30T12:36:15.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.141+0000 7f6ad26c9640 1 -- 172.21.15.125:0/775492026 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc0c6fe0 msgr2=0x7f6acc071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:15.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.141+0000 7f6ad26c9640 1 --2- 172.21.15.125:0/775492026 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc0c6fe0 0x7f6acc071260 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6ac0009870 tx=0x7f6ac001ed30 comp rx=0 tx=0).stop 2023-10-30T12:36:15.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.141+0000 7f6ad26c9640 1 -- 172.21.15.125:0/775492026 shutdown_connections 2023-10-30T12:36:15.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.141+0000 7f6ad26c9640 1 --2- 172.21.15.125:0/775492026 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6acc0c5f60 0x7f6acc0cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:15.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.141+0000 7f6ad26c9640 1 --2- 172.21.15.125:0/775492026 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc0c6fe0 0x7f6acc071260 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:15.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.141+0000 7f6ad26c9640 1 -- 172.21.15.125:0/775492026 >> 172.21.15.125:0/775492026 conn(0x7f6acc06c6e0 msgr2=0x7f6acc069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:15.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.141+0000 7f6ad26c9640 1 -- 172.21.15.125:0/775492026 shutdown_connections 2023-10-30T12:36:15.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.141+0000 7f6ad26c9640 1 -- 172.21.15.125:0/775492026 wait complete. 2023-10-30T12:36:15.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.141+0000 7f6ad26c9640 1 Processor -- start 2023-10-30T12:36:15.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.141+0000 7f6ad26c9640 1 -- start start 2023-10-30T12:36:15.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6ad26c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc0c5f60 0x7f6acc149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:15.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6ad26c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6acc0c6fe0 0x7f6acc149f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:15.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6ad26c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6acc153bf0 0x7f6acc14dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:15.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6ad26c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc14f570 con 0x7f6acc0c5f60 2023-10-30T12:36:15.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6ad26c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc14c930 con 0x7f6acc153bf0 2023-10-30T12:36:15.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6ad26c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc14bc00 con 0x7f6acc0c6fe0 2023-10-30T12:36:15.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6ad0c66640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6acc153bf0 0x7f6acc14dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:15.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6ad0c66640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6acc153bf0 0x7f6acc14dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:53736/0 (socket says 172.21.15.125:53736) 2023-10-30T12:36:15.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6ad0c66640 1 -- 172.21.15.125:0/2979381185 learned_addr learned my addr 172.21.15.125:0/2979381185 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:15.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6acbfff640 1 --2- 172.21.15.125:0/2979381185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc0c5f60 0x7f6acc149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:15.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6acb7fe640 1 --2- 172.21.15.125:0/2979381185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6acc0c6fe0 0x7f6acc149f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:15.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6acbfff640 1 -- 172.21.15.125:0/2979381185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6acc0c6fe0 msgr2=0x7f6acc149f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:15.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6acbfff640 1 --2- 172.21.15.125:0/2979381185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6acc0c6fe0 0x7f6acc149f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:15.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6acbfff640 1 -- 172.21.15.125:0/2979381185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6acc153bf0 msgr2=0x7f6acc14dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:15.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6acbfff640 1 --2- 172.21.15.125:0/2979381185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6acc153bf0 0x7f6acc14dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:15.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6acbfff640 1 -- 172.21.15.125:0/2979381185 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac00094d0 con 0x7f6acc0c5f60 2023-10-30T12:36:15.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6acbfff640 1 --2- 172.21.15.125:0/2979381185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc0c5f60 0x7f6acc149a10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6abc001ec0 tx=0x7f6abc00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:15.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6ac8ff9640 1 -- 172.21.15.125:0/2979381185 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc0052d0 con 0x7f6acc0c5f60 2023-10-30T12:36:15.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6ac8ff9640 1 -- 172.21.15.125:0/2979381185 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6abc005430 con 0x7f6acc0c5f60 2023-10-30T12:36:15.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6ac8ff9640 1 -- 172.21.15.125:0/2979381185 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6abc003840 con 0x7f6acc0c5f60 2023-10-30T12:36:15.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6ad26c9640 1 -- 172.21.15.125:0/2979381185 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6acc1e6600 con 0x7f6acc0c5f60 2023-10-30T12:36:15.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.145+0000 7f6ad26c9640 1 -- 172.21.15.125:0/2979381185 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6acc14be50 con 0x7f6acc0c5f60 2023-10-30T12:36:15.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.149+0000 7f6ad26c9640 1 -- 172.21.15.125:0/2979381185 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6acc04e170 con 0x7f6acc0c5f60 2023-10-30T12:36:15.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.149+0000 7f6ac8ff9640 1 -- 172.21.15.125:0/2979381185 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6abc0039a0 con 0x7f6acc0c5f60 2023-10-30T12:36:15.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.149+0000 7f6ac8ff9640 1 --2- 172.21.15.125:0/2979381185 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ab80664a0 0x7f6ab8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:15.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.149+0000 7f6ac8ff9640 1 -- 172.21.15.125:0/2979381185 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6040+0+0 (secure 0 0 0) 0x7f6abc0835e0 con 0x7f6acc0c5f60 2023-10-30T12:36:15.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.149+0000 7f6acb7fe640 1 --2- 172.21.15.125:0/2979381185 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ab80664a0 0x7f6ab8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:15.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.153+0000 7f6acb7fe640 1 --2- 172.21.15.125:0/2979381185 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ab80664a0 0x7f6ab8068960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00b2000 tx=0x7f6ac0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:15.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.153+0000 7f6ac8ff9640 1 -- 172.21.15.125:0/2979381185 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6abc050890 con 0x7f6acc0c5f60 2023-10-30T12:36:15.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.341+0000 7f6ad26c9640 1 -- 172.21.15.125:0/2979381185 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6acc14f6b0 con 0x7f6acc0c5f60 2023-10-30T12:36:15.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.341+0000 7f6ac8ff9640 1 -- 172.21.15.125:0/2979381185 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f6abc053eb0 con 0x7f6acc0c5f60 2023-10-30T12:36:15.345 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:15.345 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4973,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:03.067774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:15.345 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 44 2023-10-30T12:36:15.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.341+0000 7f6ad26c9640 1 -- 172.21.15.125:0/2979381185 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ab80664a0 msgr2=0x7f6ab8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:15.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.341+0000 7f6ad26c9640 1 --2- 172.21.15.125:0/2979381185 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ab80664a0 0x7f6ab8068960 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00b2000 tx=0x7f6ac0009040 comp rx=0 tx=0).stop 2023-10-30T12:36:15.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.341+0000 7f6ad26c9640 1 -- 172.21.15.125:0/2979381185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc0c5f60 msgr2=0x7f6acc149a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:15.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.341+0000 7f6ad26c9640 1 --2- 172.21.15.125:0/2979381185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc0c5f60 0x7f6acc149a10 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6abc001ec0 tx=0x7f6abc00c890 comp rx=0 tx=0).stop 2023-10-30T12:36:15.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.341+0000 7f6ad26c9640 1 -- 172.21.15.125:0/2979381185 shutdown_connections 2023-10-30T12:36:15.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.341+0000 7f6ad26c9640 1 --2- 172.21.15.125:0/2979381185 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ab80664a0 0x7f6ab8068960 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:15.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.341+0000 7f6ad26c9640 1 --2- 172.21.15.125:0/2979381185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6acc153bf0 0x7f6acc14dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:15.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.341+0000 7f6ad26c9640 1 --2- 172.21.15.125:0/2979381185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6acc0c6fe0 0x7f6acc149f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:15.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.341+0000 7f6ad26c9640 1 --2- 172.21.15.125:0/2979381185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6acc0c5f60 0x7f6acc149a10 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:15.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.341+0000 7f6ad26c9640 1 -- 172.21.15.125:0/2979381185 >> 172.21.15.125:0/2979381185 conn(0x7f6acc06c6e0 msgr2=0x7f6acc0c7430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:15.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.341+0000 7f6ad26c9640 1 -- 172.21.15.125:0/2979381185 shutdown_connections 2023-10-30T12:36:15.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:15.341+0000 7f6ad26c9640 1 -- 172.21.15.125:0/2979381185 wait complete. 2023-10-30T12:36:15.365 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:36:17.367 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:17.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.513+0000 7f3d06dd3640 1 Processor -- start 2023-10-30T12:36:17.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.513+0000 7f3d06dd3640 1 -- start start 2023-10-30T12:36:17.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.513+0000 7f3d06dd3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3d000c6b80 0x7f3d000bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:17.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.513+0000 7f3d06dd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d000c6f60 0x7f3d000c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:17.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.513+0000 7f3d06dd3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d000c0a50 con 0x7f3d000c8430 2023-10-30T12:36:17.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.513+0000 7f3d06dd3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d0006f6e0 con 0x7f3d000c6b80 2023-10-30T12:36:17.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.513+0000 7f3d06dd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0006f9c0 con 0x7f3d000c6f60 2023-10-30T12:36:17.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.517+0000 7f3cf7fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d000c6f60 0x7f3d000c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:17.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.517+0000 7f3d05370640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f3d000c8430 0x7f3d0013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:39966/0 (socket says 172.21.15.125:39966) 2023-10-30T12:36:17.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.517+0000 7f3d05370640 1 -- 172.21.15.125:0/3393780210 learned_addr learned my addr 172.21.15.125:0/3393780210 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:17.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.517+0000 7f3d04b6f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3d000c6b80 0x7f3d000bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:17.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.517+0000 7f3cf7fff640 1 -- 172.21.15.125:0/3393780210 >> v2:172.21.15.136:3300/0 conn(0x7f3d000c6b80 msgr2=0x7f3d000bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:17.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.517+0000 7f3cf7fff640 1 --2- 172.21.15.125:0/3393780210 >> v2:172.21.15.136:3300/0 conn(0x7f3d000c6b80 0x7f3d000bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:17.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.517+0000 7f3cf7fff640 1 -- 172.21.15.125:0/3393780210 >> v1:172.21.15.136:6789/0 conn(0x7f3d000c8430 legacy=0x7f3d0013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:17.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.517+0000 7f3cf7fff640 1 -- 172.21.15.125:0/3393780210 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d0006fca0 con 0x7f3d000c6f60 2023-10-30T12:36:17.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3cf7fff640 1 --2- 172.21.15.125:0/3393780210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d000c6f60 0x7f3d000c0400 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3cf00097c0 tx=0x7f3cf001c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=510d3f0c3518b734 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:17.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3cf77fe640 1 -- 172.21.15.125:0/3393780210 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf00ab600 con 0x7f3d000c6f60 2023-10-30T12:36:17.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3cf77fe640 1 -- 172.21.15.125:0/3393780210 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf00a7050 con 0x7f3d000c6f60 2023-10-30T12:36:17.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3cf77fe640 1 -- 172.21.15.125:0/3393780210 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf00ab980 con 0x7f3d000c6f60 2023-10-30T12:36:17.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3d06dd3640 1 -- 172.21.15.125:0/3393780210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d000c6f60 msgr2=0x7f3d000c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:17.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3d06dd3640 1 --2- 172.21.15.125:0/3393780210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d000c6f60 0x7f3d000c0400 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f3cf00097c0 tx=0x7f3cf001c3f0 comp rx=0 tx=0).stop 2023-10-30T12:36:17.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3d06dd3640 1 -- 172.21.15.125:0/3393780210 shutdown_connections 2023-10-30T12:36:17.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3d06dd3640 1 --2- 172.21.15.125:0/3393780210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d000c6f60 0x7f3d000c0400 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:17.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3d06dd3640 1 --2- 172.21.15.125:0/3393780210 >> v2:172.21.15.136:3300/0 conn(0x7f3d000c6b80 0x7f3d000bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:17.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3d06dd3640 1 -- 172.21.15.125:0/3393780210 >> 172.21.15.125:0/3393780210 conn(0x7f3d000c7910 msgr2=0x7f3d000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:17.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3d06dd3640 1 -- 172.21.15.125:0/3393780210 shutdown_connections 2023-10-30T12:36:17.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3d06dd3640 1 -- 172.21.15.125:0/3393780210 wait complete. 2023-10-30T12:36:17.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3d06dd3640 1 Processor -- start 2023-10-30T12:36:17.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3d06dd3640 1 -- start start 2023-10-30T12:36:17.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3d06dd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d000c6b80 0x7f3d000c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:17.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3d06dd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d000c6f60 0x7f3d0013d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:17.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.521+0000 7f3d06dd3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d000c8430 0x7f3d000c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:17.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d06dd3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d00158a20 con 0x7f3d000c8430 2023-10-30T12:36:17.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d06dd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d00154000 con 0x7f3d000c6f60 2023-10-30T12:36:17.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d06dd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d001592d0 con 0x7f3d000c6b80 2023-10-30T12:36:17.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d05370640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d000c8430 0x7f3d000c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:17.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d05370640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d000c8430 0x7f3d000c6010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34748/0 (socket says 172.21.15.125:34748) 2023-10-30T12:36:17.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d05370640 1 -- 172.21.15.125:0/1507743323 learned_addr learned my addr 172.21.15.125:0/1507743323 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:17.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d04b6f640 1 --2- 172.21.15.125:0/1507743323 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d000c6b80 0x7f3d000c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:17.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3cf7fff640 1 --2- 172.21.15.125:0/1507743323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d000c6f60 0x7f3d0013d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:17.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d05370640 1 -- 172.21.15.125:0/1507743323 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d000c6b80 msgr2=0x7f3d000c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:17.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d05370640 1 --2- 172.21.15.125:0/1507743323 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d000c6b80 0x7f3d000c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:17.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d05370640 1 -- 172.21.15.125:0/1507743323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d000c6f60 msgr2=0x7f3d0013d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:17.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d05370640 1 --2- 172.21.15.125:0/1507743323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d000c6f60 0x7f3d0013d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:17.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d05370640 1 -- 172.21.15.125:0/1507743323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf0009420 con 0x7f3d000c8430 2023-10-30T12:36:17.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d05370640 1 --2- 172.21.15.125:0/1507743323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d000c8430 0x7f3d000c6010 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3cf800cab0 tx=0x7f3cf800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:17.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3cf57fa640 1 -- 172.21.15.125:0/1507743323 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf800ac30 con 0x7f3d000c8430 2023-10-30T12:36:17.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3cf57fa640 1 -- 172.21.15.125:0/1507743323 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cf800ad90 con 0x7f3d000c8430 2023-10-30T12:36:17.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3cf57fa640 1 -- 172.21.15.125:0/1507743323 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8012600 con 0x7f3d000c8430 2023-10-30T12:36:17.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d06dd3640 1 -- 172.21.15.125:0/1507743323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d0013cbe0 con 0x7f3d000c8430 2023-10-30T12:36:17.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d06dd3640 1 -- 172.21.15.125:0/1507743323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d001597e0 con 0x7f3d000c8430 2023-10-30T12:36:17.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3d06dd3640 1 -- 172.21.15.125:0/1507743323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d0004e170 con 0x7f3d000c8430 2023-10-30T12:36:17.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.525+0000 7f3cf57fa640 1 -- 172.21.15.125:0/1507743323 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3cf8012760 con 0x7f3d000c8430 2023-10-30T12:36:17.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.529+0000 7f3cf57fa640 1 --2- 172.21.15.125:0/1507743323 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cec0664d0 0x7f3cec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:17.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.529+0000 7f3cf57fa640 1 -- 172.21.15.125:0/1507743323 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6040+0+0 (secure 0 0 0) 0x7f3cf8083d70 con 0x7f3d000c8430 2023-10-30T12:36:17.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.529+0000 7f3d04b6f640 1 --2- 172.21.15.125:0/1507743323 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cec0664d0 0x7f3cec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:17.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.529+0000 7f3d04b6f640 1 --2- 172.21.15.125:0/1507743323 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cec0664d0 0x7f3cec068990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc009b60 tx=0x7f3cfc009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:17.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.533+0000 7f3cf57fa640 1 -- 172.21.15.125:0/1507743323 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3cf8051140 con 0x7f3d000c8430 2023-10-30T12:36:17.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.713+0000 7f3d06dd3640 1 -- 172.21.15.125:0/1507743323 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d00160a30 con 0x7f3d000c8430 2023-10-30T12:36:17.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.713+0000 7f3cf57fa640 1 -- 172.21.15.125:0/1507743323 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f3cf8054760 con 0x7f3d000c8430 2023-10-30T12:36:17.718 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:17.719 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4973,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:03.067774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:17.720 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 44 2023-10-30T12:36:17.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.717+0000 7f3d06dd3640 1 -- 172.21.15.125:0/1507743323 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cec0664d0 msgr2=0x7f3cec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:17.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.717+0000 7f3d06dd3640 1 --2- 172.21.15.125:0/1507743323 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cec0664d0 0x7f3cec068990 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f3cfc009b60 tx=0x7f3cfc009190 comp rx=0 tx=0).stop 2023-10-30T12:36:17.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.717+0000 7f3d06dd3640 1 -- 172.21.15.125:0/1507743323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d000c8430 msgr2=0x7f3d000c6010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:17.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.717+0000 7f3d06dd3640 1 --2- 172.21.15.125:0/1507743323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d000c8430 0x7f3d000c6010 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3cf800cab0 tx=0x7f3cf800cf70 comp rx=0 tx=0).stop 2023-10-30T12:36:17.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.717+0000 7f3d06dd3640 1 -- 172.21.15.125:0/1507743323 shutdown_connections 2023-10-30T12:36:17.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.717+0000 7f3d06dd3640 1 --2- 172.21.15.125:0/1507743323 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cec0664d0 0x7f3cec068990 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:17.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.717+0000 7f3d06dd3640 1 --2- 172.21.15.125:0/1507743323 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d000c8430 0x7f3d000c6010 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:17.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.717+0000 7f3d06dd3640 1 --2- 172.21.15.125:0/1507743323 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d000c6f60 0x7f3d0013d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:17.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.717+0000 7f3d06dd3640 1 --2- 172.21.15.125:0/1507743323 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d000c6b80 0x7f3d000c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:17.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.717+0000 7f3d06dd3640 1 -- 172.21.15.125:0/1507743323 >> 172.21.15.125:0/1507743323 conn(0x7f3d000c7910 msgr2=0x7f3d000c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:17.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.717+0000 7f3d06dd3640 1 -- 172.21.15.125:0/1507743323 shutdown_connections 2023-10-30T12:36:17.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:17.717+0000 7f3d06dd3640 1 -- 172.21.15.125:0/1507743323 wait complete. 2023-10-30T12:36:17.739 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:36:19.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:19.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.881+0000 7f06b59f8640 1 Processor -- start 2023-10-30T12:36:19.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.881+0000 7f06b59f8640 1 -- start start 2023-10-30T12:36:19.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06b59f8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f06b01538a0 0x7f06b015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:19.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06b59f8640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b00cb800 con 0x7f06b00cf160 2023-10-30T12:36:19.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06b59f8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b015dc30 con 0x7f06b00ca0e0 2023-10-30T12:36:19.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06b59f8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b00cb330 con 0x7f06b01538a0 2023-10-30T12:36:19.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06af7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f06b00cf160 0x7f06b0164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55490/0 (socket says 172.21.15.125:55490) 2023-10-30T12:36:19.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06af7fe640 1 -- 172.21.15.125:0/58249745 learned_addr learned my addr 172.21.15.125:0/58249745 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:19.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06aeffd640 1 --2- 172.21.15.125:0/58249745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f06b01538a0 0x7f06b015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:19.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 -- 172.21.15.125:0/58249745 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936045078 0 0) 0x7f06b00cb800 con 0x7f06b00cf160 2023-10-30T12:36:19.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 -- 172.21.15.125:0/58249745 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069c003530 con 0x7f06b00cf160 2023-10-30T12:36:19.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 -- 172.21.15.125:0/58249745 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2136304838 0 0) 0x7f069c003530 con 0x7f06b00cf160 2023-10-30T12:36:19.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 -- 172.21.15.125:0/58249745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f06b01538a0 msgr2=0x7f06b015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:19.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 --2- 172.21.15.125:0/58249745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f06b01538a0 0x7f06b015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:19.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 -- 172.21.15.125:0/58249745 >> v1:172.21.15.136:6789/0 conn(0x7f06b00ca0e0 legacy=0x7f06b015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:19.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 -- 172.21.15.125:0/58249745 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b00cad70 con 0x7f06b00cf160 2023-10-30T12:36:19.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 -- 172.21.15.125:0/58249745 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f06a8001680 con 0x7f06b00cf160 2023-10-30T12:36:19.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 -- 172.21.15.125:0/58249745 >> v1:172.21.15.125:6789/0 conn(0x7f06b00cf160 legacy=0x7f06b0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:19.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 --2- 172.21.15.125:0/58249745 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f069c0039b0 0x7f069c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:19.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 --2- 172.21.15.125:0/58249745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f06b01538a0 0x7f069c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:19.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 --2- 172.21.15.125:0/58249745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f069c004d10 0x7f069c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:19.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 -- 172.21.15.125:0/58249745 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b00cb330 con 0x7f069c0039b0 2023-10-30T12:36:19.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 -- 172.21.15.125:0/58249745 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f069c00b990 con 0x7f069c004d10 2023-10-30T12:36:19.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06ae7fc640 1 -- 172.21.15.125:0/58249745 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f069c00bb50 con 0x7f06b01538a0 2023-10-30T12:36:19.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06aeffd640 1 --2- 172.21.15.125:0/58249745 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f069c0039b0 0x7f069c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:19.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06a67fc640 1 --2- 172.21.15.125:0/58249745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f06b01538a0 0x7f069c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:19.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06af7fe640 1 --2- 172.21.15.125:0/58249745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f069c004d10 0x7f069c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:19.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06aeffd640 1 -- 172.21.15.125:0/58249745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f06b01538a0 msgr2=0x7f069c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:19.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06aeffd640 1 --2- 172.21.15.125:0/58249745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f06b01538a0 0x7f069c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:19.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06aeffd640 1 -- 172.21.15.125:0/58249745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f069c004d10 msgr2=0x7f069c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:19.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06aeffd640 1 --2- 172.21.15.125:0/58249745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f069c004d10 0x7f069c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:19.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.885+0000 7f06aeffd640 1 -- 172.21.15.125:0/58249745 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c003530 con 0x7f069c0039b0 2023-10-30T12:36:19.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.889+0000 7f06aeffd640 1 --2- 172.21.15.125:0/58249745 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f069c0039b0 0x7f069c003d90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009fc0 tx=0x7f06a001cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bba795317fcc427c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:19.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.889+0000 7f06ae7fc640 1 -- 172.21.15.125:0/58249745 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a00a5bb0 con 0x7f069c0039b0 2023-10-30T12:36:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.889+0000 7f06ae7fc640 1 -- 172.21.15.125:0/58249745 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a000f040 con 0x7f069c0039b0 2023-10-30T12:36:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.889+0000 7f06ae7fc640 1 -- 172.21.15.125:0/58249745 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a00ad440 con 0x7f069c0039b0 2023-10-30T12:36:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.889+0000 7f06b59f8640 1 -- 172.21.15.125:0/58249745 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f069c0039b0 msgr2=0x7f069c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.889+0000 7f06b59f8640 1 --2- 172.21.15.125:0/58249745 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f069c0039b0 0x7f069c003d90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009fc0 tx=0x7f06a001cba0 comp rx=0 tx=0).stop 2023-10-30T12:36:19.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.889+0000 7f06af7fe640 1 -- 172.21.15.125:0/58249745 reap_dead start 2023-10-30T12:36:19.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.889+0000 7f06b59f8640 1 -- 172.21.15.125:0/58249745 shutdown_connections 2023-10-30T12:36:19.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.889+0000 7f06b59f8640 1 -- 172.21.15.125:0/58249745 >> 172.21.15.125:0/58249745 conn(0x7f06b006bfe0 msgr2=0x7f06b006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:19.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.889+0000 7f06b59f8640 1 -- 172.21.15.125:0/58249745 shutdown_connections 2023-10-30T12:36:19.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.889+0000 7f06b59f8640 1 -- 172.21.15.125:0/58249745 wait complete. 2023-10-30T12:36:19.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06b59f8640 1 Processor -- start 2023-10-30T12:36:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06b59f8640 1 -- start start 2023-10-30T12:36:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06b59f8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f069c0039b0 0x7f06b01f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06b59f8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06b00c7220 0x7f06b0145be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06b59f8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f06b00c7680 0x7f06b0146120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:19.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06b59f8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b01e3f10 con 0x7f06b00c7220 2023-10-30T12:36:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06b59f8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06b00c7e10 con 0x7f069c0039b0 2023-10-30T12:36:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06b59f8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06b0146660 con 0x7f06b00c7680 2023-10-30T12:36:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06a67fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06b00c7220 0x7f06b0145be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06a67fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06b00c7220 0x7f06b0145be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34780/0 (socket says 172.21.15.125:34780) 2023-10-30T12:36:19.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06a67fc640 1 -- 172.21.15.125:0/1203615015 learned_addr learned my addr 172.21.15.125:0/1203615015 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:19.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06aeffd640 1 --2- 172.21.15.125:0/1203615015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f069c0039b0 0x7f06b01f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:19.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06af7fe640 1 --2- 172.21.15.125:0/1203615015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f06b00c7680 0x7f06b0146120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:19.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06a67fc640 1 -- 172.21.15.125:0/1203615015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f06b00c7680 msgr2=0x7f06b0146120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:19.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06a67fc640 1 --2- 172.21.15.125:0/1203615015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f06b00c7680 0x7f06b0146120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:19.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06a67fc640 1 -- 172.21.15.125:0/1203615015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f069c0039b0 msgr2=0x7f06b01f2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:19.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06a67fc640 1 --2- 172.21.15.125:0/1203615015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f069c0039b0 0x7f06b01f2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:19.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06a67fc640 1 -- 172.21.15.125:0/1203615015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b00c7b30 con 0x7f06b00c7220 2023-10-30T12:36:19.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06a67fc640 1 --2- 172.21.15.125:0/1203615015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06b00c7220 0x7f06b0145be0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0698010f40 tx=0x7f069800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:19.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06acff9640 1 -- 172.21.15.125:0/1203615015 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f069800ae70 con 0x7f06b00c7220 2023-10-30T12:36:19.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06acff9640 1 -- 172.21.15.125:0/1203615015 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069800d030 con 0x7f06b00c7220 2023-10-30T12:36:19.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06acff9640 1 -- 172.21.15.125:0/1203615015 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06980052d0 con 0x7f06b00c7220 2023-10-30T12:36:19.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06b59f8640 1 -- 172.21.15.125:0/1203615015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b01f3320 con 0x7f06b00c7220 2023-10-30T12:36:19.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06b59f8640 1 -- 172.21.15.125:0/1203615015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b00c8fe0 con 0x7f06b00c7220 2023-10-30T12:36:19.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.893+0000 7f06b59f8640 1 -- 172.21.15.125:0/1203615015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b004e170 con 0x7f06b00c7220 2023-10-30T12:36:19.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.897+0000 7f06acff9640 1 -- 172.21.15.125:0/1203615015 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0698003600 con 0x7f06b00c7220 2023-10-30T12:36:19.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.897+0000 7f06acff9640 1 --2- 172.21.15.125:0/1203615015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f06900664a0 0x7f0690068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.897+0000 7f06acff9640 1 -- 172.21.15.125:0/1203615015 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6040+0+0 (secure 0 0 0) 0x7f0698087200 con 0x7f06b00c7220 2023-10-30T12:36:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.897+0000 7f06aeffd640 1 --2- 172.21.15.125:0/1203615015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f06900664a0 0x7f0690068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.897+0000 7f06aeffd640 1 --2- 172.21.15.125:0/1203615015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f06900664a0 0x7f0690068960 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f06a00af000 tx=0x7f06a0004ac0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:19.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:19.901+0000 7f06acff9640 1 -- 172.21.15.125:0/1203615015 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0698054540 con 0x7f06b00c7220 2023-10-30T12:36:20.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:20.085+0000 7f06b59f8640 1 -- 172.21.15.125:0/1203615015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06b015dc30 con 0x7f06b00c7220 2023-10-30T12:36:20.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:20.085+0000 7f06acff9640 1 -- 172.21.15.125:0/1203615015 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f06b015dc30 con 0x7f06b00c7220 2023-10-30T12:36:20.091 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:20.092 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4973,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:03.067774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:20.094 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 44 2023-10-30T12:36:20.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:20.089+0000 7f06b59f8640 1 -- 172.21.15.125:0/1203615015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f06900664a0 msgr2=0x7f0690068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:20.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:20.089+0000 7f06b59f8640 1 --2- 172.21.15.125:0/1203615015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f06900664a0 0x7f0690068960 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f06a00af000 tx=0x7f06a0004ac0 comp rx=0 tx=0).stop 2023-10-30T12:36:20.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:20.089+0000 7f06b59f8640 1 -- 172.21.15.125:0/1203615015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06b00c7220 msgr2=0x7f06b0145be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:20.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:20.089+0000 7f06b59f8640 1 --2- 172.21.15.125:0/1203615015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06b00c7220 0x7f06b0145be0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0698010f40 tx=0x7f069800e530 comp rx=0 tx=0).stop 2023-10-30T12:36:20.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:20.089+0000 7f06b59f8640 1 -- 172.21.15.125:0/1203615015 shutdown_connections 2023-10-30T12:36:20.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:20.089+0000 7f06b59f8640 1 --2- 172.21.15.125:0/1203615015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f06900664a0 0x7f0690068960 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:20.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:20.089+0000 7f06b59f8640 1 --2- 172.21.15.125:0/1203615015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f06b00c7680 0x7f06b0146120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:20.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:20.089+0000 7f06b59f8640 1 --2- 172.21.15.125:0/1203615015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f06b00c7220 0x7f06b0145be0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:20.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:20.089+0000 7f06b59f8640 1 --2- 172.21.15.125:0/1203615015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f069c0039b0 0x7f06b01f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:20.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:20.089+0000 7f06b59f8640 1 -- 172.21.15.125:0/1203615015 >> 172.21.15.125:0/1203615015 conn(0x7f06b006bfe0 msgr2=0x7f06b01494e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:20.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:20.089+0000 7f06b59f8640 1 -- 172.21.15.125:0/1203615015 shutdown_connections 2023-10-30T12:36:20.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:20.089+0000 7f06b59f8640 1 -- 172.21.15.125:0/1203615015 wait complete. 2023-10-30T12:36:20.112 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:36:22.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:22.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.261+0000 7f0981b72640 1 Processor -- start 2023-10-30T12:36:22.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.261+0000 7f0981b72640 1 -- start start 2023-10-30T12:36:22.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.261+0000 7f0981b72640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f097c1538a0 0x7f097c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:22.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.261+0000 7f0981b72640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097c0cb800 con 0x7f097c0cf160 2023-10-30T12:36:22.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.261+0000 7f0981b72640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f097c15dc30 con 0x7f097c0ca0e0 2023-10-30T12:36:22.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.261+0000 7f0981b72640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f097c0cb330 con 0x7f097c1538a0 2023-10-30T12:36:22.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.261+0000 7f097b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f097c1538a0 0x7f097c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:22.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.261+0000 7f097bfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f097c0cf160 0x7f097c164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50090/0 (socket says 172.21.15.125:50090) 2023-10-30T12:36:22.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.261+0000 7f097bfff640 1 -- 172.21.15.125:0/3148609482 learned_addr learned my addr 172.21.15.125:0/3148609482 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:22.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.261+0000 7f097b7fe640 1 -- 172.21.15.125:0/3148609482 >> v1:172.21.15.136:6789/0 conn(0x7f097c0ca0e0 legacy=0x7f097c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:22.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.261+0000 7f097b7fe640 1 -- 172.21.15.125:0/3148609482 >> v1:172.21.15.125:6789/0 conn(0x7f097c0cf160 legacy=0x7f097c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:22.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.261+0000 7f097a7fc640 1 -- 172.21.15.125:0/3148609482 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208757152 0 0) 0x7f097c0cb800 con 0x7f097c0cf160 2023-10-30T12:36:22.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.261+0000 7f097b7fe640 1 -- 172.21.15.125:0/3148609482 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f097c0cad70 con 0x7f097c1538a0 2023-10-30T12:36:22.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.265+0000 7f097b7fe640 1 --2- 172.21.15.125:0/3148609482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f097c1538a0 0x7f097c15d6f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f096c0097c0 tx=0x7f096c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed9e6f78c302a96 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:22.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.265+0000 7f097a7fc640 1 -- 172.21.15.125:0/3148609482 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c0a8970 con 0x7f097c1538a0 2023-10-30T12:36:22.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.265+0000 7f097a7fc640 1 -- 172.21.15.125:0/3148609482 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f096c0a8ad0 con 0x7f097c1538a0 2023-10-30T12:36:22.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.265+0000 7f097a7fc640 1 -- 172.21.15.125:0/3148609482 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f096c0a8e60 con 0x7f097c1538a0 2023-10-30T12:36:22.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.265+0000 7f0981b72640 1 -- 172.21.15.125:0/3148609482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f097c1538a0 msgr2=0x7f097c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:22.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.265+0000 7f0981b72640 1 --2- 172.21.15.125:0/3148609482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f097c1538a0 0x7f097c15d6f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f096c0097c0 tx=0x7f096c01ec70 comp rx=0 tx=0).stop 2023-10-30T12:36:22.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.265+0000 7f0981b72640 1 -- 172.21.15.125:0/3148609482 shutdown_connections 2023-10-30T12:36:22.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.265+0000 7f0981b72640 1 --2- 172.21.15.125:0/3148609482 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f097c1538a0 0x7f097c15d6f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:22.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.265+0000 7f0981b72640 1 -- 172.21.15.125:0/3148609482 >> 172.21.15.125:0/3148609482 conn(0x7f097c06bfe0 msgr2=0x7f097c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:22.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.265+0000 7f0981b72640 1 -- 172.21.15.125:0/3148609482 shutdown_connections 2023-10-30T12:36:22.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.265+0000 7f0981b72640 1 -- 172.21.15.125:0/3148609482 wait complete. 2023-10-30T12:36:22.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f0981b72640 1 Processor -- start 2023-10-30T12:36:22.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f0981b72640 1 -- start start 2023-10-30T12:36:22.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f0981b72640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f097c0ca0e0 0x7f097c0c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:22.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f0981b72640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f097c0cf160 0x7f097c1507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:22.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f0981b72640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f097c1538a0 0x7f097c0b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:22.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f0981b72640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c157830 con 0x7f097c0cf160 2023-10-30T12:36:22.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f0981b72640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097c1575c0 con 0x7f097c1538a0 2023-10-30T12:36:22.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f0981b72640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f097c0b3d90 con 0x7f097c0ca0e0 2023-10-30T12:36:22.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f097affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f097c0cf160 0x7f097c1507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:22.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f097affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f097c0cf160 0x7f097c1507d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49596/0 (socket says 172.21.15.125:49596) 2023-10-30T12:36:22.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f097affd640 1 -- 172.21.15.125:0/3820087456 learned_addr learned my addr 172.21.15.125:0/3820087456 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:22.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f097b7fe640 1 --2- 172.21.15.125:0/3820087456 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f097c0ca0e0 0x7f097c0c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:22.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f097bfff640 1 --2- 172.21.15.125:0/3820087456 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f097c1538a0 0x7f097c0b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:22.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f097affd640 1 -- 172.21.15.125:0/3820087456 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f097c0ca0e0 msgr2=0x7f097c0c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:22.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f097affd640 1 --2- 172.21.15.125:0/3820087456 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f097c0ca0e0 0x7f097c0c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:22.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f097affd640 1 -- 172.21.15.125:0/3820087456 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f097c1538a0 msgr2=0x7f097c0b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:22.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f097affd640 1 --2- 172.21.15.125:0/3820087456 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f097c1538a0 0x7f097c0b42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:22.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f097affd640 1 -- 172.21.15.125:0/3820087456 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096c009420 con 0x7f097c0cf160 2023-10-30T12:36:22.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f097affd640 1 --2- 172.21.15.125:0/3820087456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f097c0cf160 0x7f097c1507d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f097000c3c0 tx=0x7f097000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:22.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f0963fff640 1 -- 172.21.15.125:0/3820087456 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09700052d0 con 0x7f097c0cf160 2023-10-30T12:36:22.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f0963fff640 1 -- 172.21.15.125:0/3820087456 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0970005430 con 0x7f097c0cf160 2023-10-30T12:36:22.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f0981b72640 1 -- 172.21.15.125:0/3820087456 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097c159040 con 0x7f097c0cf160 2023-10-30T12:36:22.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f0963fff640 1 -- 172.21.15.125:0/3820087456 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0970003840 con 0x7f097c0cf160 2023-10-30T12:36:22.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.269+0000 7f0981b72640 1 -- 172.21.15.125:0/3820087456 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f097c0b4010 con 0x7f097c0cf160 2023-10-30T12:36:22.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.273+0000 7f0981b72640 1 -- 172.21.15.125:0/3820087456 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f097c04e170 con 0x7f097c0cf160 2023-10-30T12:36:22.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.273+0000 7f0963fff640 1 -- 172.21.15.125:0/3820087456 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f09700039a0 con 0x7f097c0cf160 2023-10-30T12:36:22.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.273+0000 7f0963fff640 1 --2- 172.21.15.125:0/3820087456 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f09680664a0 0x7f0968068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:22.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.273+0000 7f0963fff640 1 -- 172.21.15.125:0/3820087456 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6040+0+0 (secure 0 0 0) 0x7f0970083430 con 0x7f097c0cf160 2023-10-30T12:36:22.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.273+0000 7f097b7fe640 1 --2- 172.21.15.125:0/3820087456 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f09680664a0 0x7f0968068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:22.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.273+0000 7f097b7fe640 1 --2- 172.21.15.125:0/3820087456 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f09680664a0 0x7f0968068960 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f096c0b2000 tx=0x7f096c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:22.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.277+0000 7f0963fff640 1 -- 172.21.15.125:0/3820087456 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f09700506c0 con 0x7f097c0cf160 2023-10-30T12:36:22.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.457+0000 7f0981b72640 1 -- 172.21.15.125:0/3820087456 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f097c159b40 con 0x7f097c0cf160 2023-10-30T12:36:22.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.457+0000 7f0963fff640 1 -- 172.21.15.125:0/3820087456 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f0970053ce0 con 0x7f097c0cf160 2023-10-30T12:36:22.462 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:22.462 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4973,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:03.067774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:22.463 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 44 2023-10-30T12:36:22.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.461+0000 7f0981b72640 1 -- 172.21.15.125:0/3820087456 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f09680664a0 msgr2=0x7f0968068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:22.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.461+0000 7f0981b72640 1 --2- 172.21.15.125:0/3820087456 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f09680664a0 0x7f0968068960 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f096c0b2000 tx=0x7f096c008040 comp rx=0 tx=0).stop 2023-10-30T12:36:22.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.461+0000 7f0981b72640 1 -- 172.21.15.125:0/3820087456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f097c0cf160 msgr2=0x7f097c1507d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:22.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.461+0000 7f0981b72640 1 --2- 172.21.15.125:0/3820087456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f097c0cf160 0x7f097c1507d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f097000c3c0 tx=0x7f097000c880 comp rx=0 tx=0).stop 2023-10-30T12:36:22.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.461+0000 7f0981b72640 1 -- 172.21.15.125:0/3820087456 shutdown_connections 2023-10-30T12:36:22.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.461+0000 7f0981b72640 1 --2- 172.21.15.125:0/3820087456 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f09680664a0 0x7f0968068960 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:22.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.461+0000 7f0981b72640 1 --2- 172.21.15.125:0/3820087456 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f097c1538a0 0x7f097c0b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:22.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.461+0000 7f0981b72640 1 --2- 172.21.15.125:0/3820087456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f097c0cf160 0x7f097c1507d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:22.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.461+0000 7f0981b72640 1 --2- 172.21.15.125:0/3820087456 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f097c0ca0e0 0x7f097c0c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:22.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.461+0000 7f0981b72640 1 -- 172.21.15.125:0/3820087456 >> 172.21.15.125:0/3820087456 conn(0x7f097c06bfe0 msgr2=0x7f097c152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:22.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.461+0000 7f0981b72640 1 -- 172.21.15.125:0/3820087456 shutdown_connections 2023-10-30T12:36:22.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:22.461+0000 7f0981b72640 1 -- 172.21.15.125:0/3820087456 wait complete. 2023-10-30T12:36:22.482 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:36:24.483 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:24.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.625+0000 7f2953f3e640 1 Processor -- start 2023-10-30T12:36:24.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.625+0000 7f2953f3e640 1 -- start start 2023-10-30T12:36:24.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.629+0000 7f2953f3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f294c06f7a0 0x7f294c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:24.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.629+0000 7f2953f3e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f294c071110 0x7f294c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:24.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.629+0000 7f2953f3e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294c066cf0 con 0x7f294c0c6bd0 2023-10-30T12:36:24.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.629+0000 7f2953f3e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f294c0700c0 con 0x7f294c071110 2023-10-30T12:36:24.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.629+0000 7f2953f3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294c0c70d0 con 0x7f294c06f7a0 2023-10-30T12:36:24.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.629+0000 7f29524db640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f294c071110 0x7f294c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:24.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.629+0000 7f29514d9640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f294c0c6bd0 0x7f294c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50096/0 (socket says 172.21.15.125:50096) 2023-10-30T12:36:24.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.629+0000 7f29514d9640 1 -- 172.21.15.125:0/603814525 learned_addr learned my addr 172.21.15.125:0/603814525 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:24.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.629+0000 7f2951cda640 1 --2- 172.21.15.125:0/603814525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f294c06f7a0 0x7f294c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:24.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.629+0000 7f29524db640 1 -- 172.21.15.125:0/603814525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f294c06f7a0 msgr2=0x7f294c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:24.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.629+0000 7f29524db640 1 --2- 172.21.15.125:0/603814525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f294c06f7a0 0x7f294c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:24.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.629+0000 7f29524db640 1 -- 172.21.15.125:0/603814525 >> v1:172.21.15.125:6789/0 conn(0x7f294c0c6bd0 legacy=0x7f294c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:24.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.629+0000 7f29524db640 1 -- 172.21.15.125:0/603814525 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c0c7390 con 0x7f294c071110 2023-10-30T12:36:24.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.629+0000 7f2950cd8640 1 -- 172.21.15.125:0/603814525 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210165927 0 0) 0x7f294c066cf0 con 0x7f294c0c6bd0 2023-10-30T12:36:24.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.633+0000 7f29524db640 1 --2- 172.21.15.125:0/603814525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f294c071110 0x7f294c158110 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2944009d90 tx=0x7f294401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0e2a12839fd63e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:24.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.633+0000 7f2950cd8640 1 -- 172.21.15.125:0/603814525 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29440af600 con 0x7f294c071110 2023-10-30T12:36:24.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.633+0000 7f2950cd8640 1 -- 172.21.15.125:0/603814525 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29440ab050 con 0x7f294c071110 2023-10-30T12:36:24.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.633+0000 7f2950cd8640 1 -- 172.21.15.125:0/603814525 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29440af980 con 0x7f294c071110 2023-10-30T12:36:24.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.633+0000 7f2953f3e640 1 -- 172.21.15.125:0/603814525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f294c071110 msgr2=0x7f294c158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:24.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.633+0000 7f2953f3e640 1 --2- 172.21.15.125:0/603814525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f294c071110 0x7f294c158110 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f2944009d90 tx=0x7f294401f130 comp rx=0 tx=0).stop 2023-10-30T12:36:24.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.633+0000 7f2953f3e640 1 -- 172.21.15.125:0/603814525 shutdown_connections 2023-10-30T12:36:24.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.633+0000 7f2953f3e640 1 --2- 172.21.15.125:0/603814525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f294c071110 0x7f294c158110 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:24.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.633+0000 7f2953f3e640 1 --2- 172.21.15.125:0/603814525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f294c06f7a0 0x7f294c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:24.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.633+0000 7f2953f3e640 1 -- 172.21.15.125:0/603814525 >> 172.21.15.125:0/603814525 conn(0x7f294c0c7910 msgr2=0x7f294c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:24.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.633+0000 7f2953f3e640 1 -- 172.21.15.125:0/603814525 shutdown_connections 2023-10-30T12:36:24.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.633+0000 7f2953f3e640 1 -- 172.21.15.125:0/603814525 wait complete. 2023-10-30T12:36:24.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.633+0000 7f2953f3e640 1 Processor -- start 2023-10-30T12:36:24.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.633+0000 7f2953f3e640 1 -- start start 2023-10-30T12:36:24.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f2953f3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f294c06f7a0 0x7f294c0b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:24.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f2953f3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f294c071110 0x7f294c0af190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:24.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f2953f3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f294c0c6bd0 0x7f294c0c0a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:24.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f2953f3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c0c05b0 con 0x7f294c071110 2023-10-30T12:36:24.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f2953f3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f294c1ef6c0 con 0x7f294c06f7a0 2023-10-30T12:36:24.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f2953f3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294c1ef100 con 0x7f294c0c6bd0 2023-10-30T12:36:24.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29514d9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f294c071110 0x7f294c0af190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:24.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29514d9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f294c071110 0x7f294c0af190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49622/0 (socket says 172.21.15.125:49622) 2023-10-30T12:36:24.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29514d9640 1 -- 172.21.15.125:0/2427067639 learned_addr learned my addr 172.21.15.125:0/2427067639 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:24.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f2951cda640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f294c06f7a0 0x7f294c0b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:24.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29524db640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f294c0c6bd0 0x7f294c0c0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:24.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29514d9640 1 -- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f294c0c6bd0 msgr2=0x7f294c0c0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:24.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29514d9640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f294c0c6bd0 0x7f294c0c0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:24.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29514d9640 1 -- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f294c06f7a0 msgr2=0x7f294c0b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:24.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29514d9640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f294c06f7a0 0x7f294c0b06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:24.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29514d9640 1 -- 172.21.15.125:0/2427067639 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2944009a70 con 0x7f294c071110 2023-10-30T12:36:24.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29524db640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f294c0c6bd0 0x7f294c0c0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:36:24.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29514d9640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f294c071110 0x7f294c0af190 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f293c0011f0 tx=0x7f293c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:24.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29427fc640 1 -- 172.21.15.125:0/2427067639 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293c0074e0 con 0x7f294c071110 2023-10-30T12:36:24.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29427fc640 1 -- 172.21.15.125:0/2427067639 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293c007640 con 0x7f294c071110 2023-10-30T12:36:24.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f2953f3e640 1 -- 172.21.15.125:0/2427067639 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294c1ef3e0 con 0x7f294c071110 2023-10-30T12:36:24.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29427fc640 1 -- 172.21.15.125:0/2427067639 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293c004a10 con 0x7f294c071110 2023-10-30T12:36:24.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f2953f3e640 1 -- 172.21.15.125:0/2427067639 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294c0c0240 con 0x7f294c071110 2023-10-30T12:36:24.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f2953f3e640 1 -- 172.21.15.125:0/2427067639 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294c04e170 con 0x7f294c071110 2023-10-30T12:36:24.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.637+0000 7f29427fc640 1 -- 172.21.15.125:0/2427067639 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f293c004b70 con 0x7f294c071110 2023-10-30T12:36:24.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.641+0000 7f29427fc640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29300664d0 0x7f2930068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:24.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.641+0000 7f29427fc640 1 -- 172.21.15.125:0/2427067639 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6040+0+0 (secure 0 0 0) 0x7f293c083670 con 0x7f294c071110 2023-10-30T12:36:24.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.641+0000 7f2951cda640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29300664d0 0x7f2930068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:24.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.641+0000 7f2951cda640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29300664d0 0x7f2930068990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f294800b930 tx=0x7f2948008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:24.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.641+0000 7f29427fc640 1 -- 172.21.15.125:0/2427067639 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f293c050870 con 0x7f294c071110 2023-10-30T12:36:24.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.825+0000 7f2953f3e640 1 -- 172.21.15.125:0/2427067639 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f294c158620 con 0x7f294c071110 2023-10-30T12:36:24.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.825+0000 7f29427fc640 1 -- 172.21.15.125:0/2427067639 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f293c018090 con 0x7f294c071110 2023-10-30T12:36:24.829 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:24.830 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4973,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:03.067774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:24.831 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 44 2023-10-30T12:36:24.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.829+0000 7f2953f3e640 1 -- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29300664d0 msgr2=0x7f2930068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:24.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.829+0000 7f2953f3e640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29300664d0 0x7f2930068990 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f294800b930 tx=0x7f2948008040 comp rx=0 tx=0).stop 2023-10-30T12:36:24.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.829+0000 7f2953f3e640 1 -- 172.21.15.125:0/2427067639 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f294c071110 msgr2=0x7f294c0af190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:24.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.829+0000 7f2953f3e640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f294c071110 0x7f294c0af190 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f293c0011f0 tx=0x7f293c00c9b0 comp rx=0 tx=0).stop 2023-10-30T12:36:24.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.829+0000 7f2953f3e640 1 -- 172.21.15.125:0/2427067639 shutdown_connections 2023-10-30T12:36:24.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.829+0000 7f2953f3e640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29300664d0 0x7f2930068990 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:24.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.829+0000 7f2953f3e640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f294c0c6bd0 0x7f294c0c0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:24.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.829+0000 7f2953f3e640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f294c071110 0x7f294c0af190 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:24.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.829+0000 7f2953f3e640 1 --2- 172.21.15.125:0/2427067639 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f294c06f7a0 0x7f294c0b06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:24.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.829+0000 7f2953f3e640 1 -- 172.21.15.125:0/2427067639 >> 172.21.15.125:0/2427067639 conn(0x7f294c0c7910 msgr2=0x7f294c06c180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:24.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.829+0000 7f2953f3e640 1 -- 172.21.15.125:0/2427067639 shutdown_connections 2023-10-30T12:36:24.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:24.829+0000 7f2953f3e640 1 -- 172.21.15.125:0/2427067639 wait complete. 2023-10-30T12:36:24.850 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:36:26.851 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:26.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2c169c640 1 Processor -- start 2023-10-30T12:36:26.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2c169c640 1 -- start start 2023-10-30T12:36:27.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2c169c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe2bc0c6b80 0x7fe2bc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:27.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2c169c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe2bc0c6f60 0x7fe2bc0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:27.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2c169c640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2bc0c0a50 con 0x7fe2bc0c8430 2023-10-30T12:36:27.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2c169c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2bc06f6e0 con 0x7fe2bc0c6b80 2023-10-30T12:36:27.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2c169c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2bc06f9c0 con 0x7fe2bc0c6f60 2023-10-30T12:36:27.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2baffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe2bc0c6b80 0x7fe2bc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:27.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2bb7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe2bc0c8430 0x7fe2bc13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50098/0 (socket says 172.21.15.125:50098) 2023-10-30T12:36:27.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2bb7fe640 1 -- 172.21.15.125:0/3675904528 learned_addr learned my addr 172.21.15.125:0/3675904528 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:27.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2ba7fc640 1 --2- 172.21.15.125:0/3675904528 >> v2:172.21.15.136:3300/0 conn(0x7fe2bc0c6f60 0x7fe2bc0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:27.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2b9ffb640 1 -- 172.21.15.125:0/3675904528 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526058920 0 0) 0x7fe2bc0c0a50 con 0x7fe2bc0c8430 2023-10-30T12:36:27.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2baffd640 1 -- 172.21.15.125:0/3675904528 >> v2:172.21.15.136:3300/0 conn(0x7fe2bc0c6f60 msgr2=0x7fe2bc0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:27.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2baffd640 1 --2- 172.21.15.125:0/3675904528 >> v2:172.21.15.136:3300/0 conn(0x7fe2bc0c6f60 0x7fe2bc0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:27.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2baffd640 1 -- 172.21.15.125:0/3675904528 >> v1:172.21.15.125:6789/0 conn(0x7fe2bc0c8430 legacy=0x7fe2bc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:27.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:26.997+0000 7fe2baffd640 1 -- 172.21.15.125:0/3675904528 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc06fca0 con 0x7fe2bc0c6b80 2023-10-30T12:36:27.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.001+0000 7fe2baffd640 1 --2- 172.21.15.125:0/3675904528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2bc0c6b80 0x7fe2bc0bfec0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc159b90 tx=0x7fe2ac01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41cfeaf16b5486c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:27.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.001+0000 7fe2b9ffb640 1 -- 172.21.15.125:0/3675904528 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac0ab600 con 0x7fe2bc0c6b80 2023-10-30T12:36:27.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.001+0000 7fe2b9ffb640 1 -- 172.21.15.125:0/3675904528 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2ac0a7050 con 0x7fe2bc0c6b80 2023-10-30T12:36:27.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.001+0000 7fe2b9ffb640 1 -- 172.21.15.125:0/3675904528 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2ac0ab980 con 0x7fe2bc0c6b80 2023-10-30T12:36:27.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.001+0000 7fe2c169c640 1 -- 172.21.15.125:0/3675904528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2bc0c6b80 msgr2=0x7fe2bc0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:27.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.001+0000 7fe2c169c640 1 --2- 172.21.15.125:0/3675904528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2bc0c6b80 0x7fe2bc0bfec0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc159b90 tx=0x7fe2ac01c460 comp rx=0 tx=0).stop 2023-10-30T12:36:27.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.001+0000 7fe2c169c640 1 -- 172.21.15.125:0/3675904528 shutdown_connections 2023-10-30T12:36:27.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.001+0000 7fe2c169c640 1 --2- 172.21.15.125:0/3675904528 >> v2:172.21.15.136:3300/0 conn(0x7fe2bc0c6f60 0x7fe2bc0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:27.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.001+0000 7fe2c169c640 1 --2- 172.21.15.125:0/3675904528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2bc0c6b80 0x7fe2bc0bfec0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:27.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.001+0000 7fe2c169c640 1 -- 172.21.15.125:0/3675904528 >> 172.21.15.125:0/3675904528 conn(0x7fe2bc0c7910 msgr2=0x7fe2bc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:27.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.001+0000 7fe2c169c640 1 -- 172.21.15.125:0/3675904528 shutdown_connections 2023-10-30T12:36:27.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.001+0000 7fe2c169c640 1 -- 172.21.15.125:0/3675904528 wait complete. 2023-10-30T12:36:27.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2c169c640 1 Processor -- start 2023-10-30T12:36:27.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2c169c640 1 -- start start 2023-10-30T12:36:27.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2c169c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2bc0c6b80 0x7fe2bc1303f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:27.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2c169c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2bc0c6f60 0x7fe2bc132a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:27.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2c169c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2bc0c8430 0x7fe2bc132f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:27.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2c169c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc1345c0 con 0x7fe2bc0c6f60 2023-10-30T12:36:27.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2c169c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc158fc0 con 0x7fe2bc0c6b80 2023-10-30T12:36:27.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2c169c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2bc1592a0 con 0x7fe2bc0c8430 2023-10-30T12:36:27.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2ba7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2bc0c6f60 0x7fe2bc132a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:27.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2ba7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2bc0c6f60 0x7fe2bc132a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49628/0 (socket says 172.21.15.125:49628) 2023-10-30T12:36:27.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2ba7fc640 1 -- 172.21.15.125:0/4184123403 learned_addr learned my addr 172.21.15.125:0/4184123403 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:27.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2bb7fe640 1 --2- 172.21.15.125:0/4184123403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2bc0c8430 0x7fe2bc132f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:27.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2baffd640 1 --2- 172.21.15.125:0/4184123403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2bc0c6b80 0x7fe2bc1303f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:27.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2ba7fc640 1 -- 172.21.15.125:0/4184123403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2bc0c8430 msgr2=0x7fe2bc132f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:27.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2ba7fc640 1 --2- 172.21.15.125:0/4184123403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2bc0c8430 0x7fe2bc132f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:27.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2ba7fc640 1 -- 172.21.15.125:0/4184123403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2bc0c6b80 msgr2=0x7fe2bc1303f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:27.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2ba7fc640 1 --2- 172.21.15.125:0/4184123403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2bc0c6b80 0x7fe2bc1303f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:27.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2ba7fc640 1 -- 172.21.15.125:0/4184123403 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2ac009420 con 0x7fe2bc0c6f60 2023-10-30T12:36:27.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2ba7fc640 1 --2- 172.21.15.125:0/4184123403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2bc0c6f60 0x7fe2bc132a50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000e930 tx=0x7fe2b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:27.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2a37fe640 1 -- 172.21.15.125:0/4184123403 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b0009690 con 0x7fe2bc0c6f60 2023-10-30T12:36:27.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2a37fe640 1 -- 172.21.15.125:0/4184123403 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2b000ca00 con 0x7fe2bc0c6f60 2023-10-30T12:36:27.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2a37fe640 1 -- 172.21.15.125:0/4184123403 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2b000cd70 con 0x7fe2bc0c6f60 2023-10-30T12:36:27.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2c169c640 1 -- 172.21.15.125:0/4184123403 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2bc159580 con 0x7fe2bc0c6f60 2023-10-30T12:36:27.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2c169c640 1 -- 172.21.15.125:0/4184123403 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2bc130930 con 0x7fe2bc0c6f60 2023-10-30T12:36:27.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.005+0000 7fe2c169c640 1 -- 172.21.15.125:0/4184123403 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2bc132770 con 0x7fe2bc0c6f60 2023-10-30T12:36:27.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.009+0000 7fe2a37fe640 1 -- 172.21.15.125:0/4184123403 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe2b0013470 con 0x7fe2bc0c6f60 2023-10-30T12:36:27.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.009+0000 7fe2a37fe640 1 --2- 172.21.15.125:0/4184123403 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2a80664d0 0x7fe2a8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:27.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.009+0000 7fe2a37fe640 1 -- 172.21.15.125:0/4184123403 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6040+0+0 (secure 0 0 0) 0x7fe2b00842c0 con 0x7fe2bc0c6f60 2023-10-30T12:36:27.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.009+0000 7fe2baffd640 1 --2- 172.21.15.125:0/4184123403 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2a80664d0 0x7fe2a8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:27.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.009+0000 7fe2baffd640 1 --2- 172.21.15.125:0/4184123403 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2a80664d0 0x7fe2a8068990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac0ad000 tx=0x7fe2ac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:27.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.013+0000 7fe2a37fe640 1 -- 172.21.15.125:0/4184123403 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe2bc132770 con 0x7fe2bc0c6f60 2023-10-30T12:36:27.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.193+0000 7fe2c169c640 1 -- 172.21.15.125:0/4184123403 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe2bc132770 con 0x7fe2bc0c6f60 2023-10-30T12:36:27.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.193+0000 7fe2a37fe640 1 -- 172.21.15.125:0/4184123403 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+6551 (secure 0 0 0) 0x7fe2bc132770 con 0x7fe2bc0c6f60 2023-10-30T12:36:27.198 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:27.199 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4973,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:03.067774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:27.200 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 44 2023-10-30T12:36:27.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.197+0000 7fe2c169c640 1 -- 172.21.15.125:0/4184123403 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2a80664d0 msgr2=0x7fe2a8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:27.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.197+0000 7fe2c169c640 1 --2- 172.21.15.125:0/4184123403 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2a80664d0 0x7fe2a8068990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe2ac0ad000 tx=0x7fe2ac008040 comp rx=0 tx=0).stop 2023-10-30T12:36:27.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.197+0000 7fe2c169c640 1 -- 172.21.15.125:0/4184123403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2bc0c6f60 msgr2=0x7fe2bc132a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:27.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.197+0000 7fe2c169c640 1 --2- 172.21.15.125:0/4184123403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2bc0c6f60 0x7fe2bc132a50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe2b000e930 tx=0x7fe2b000edf0 comp rx=0 tx=0).stop 2023-10-30T12:36:27.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.197+0000 7fe2c169c640 1 -- 172.21.15.125:0/4184123403 shutdown_connections 2023-10-30T12:36:27.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.197+0000 7fe2c169c640 1 --2- 172.21.15.125:0/4184123403 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2a80664d0 0x7fe2a8068990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:27.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.197+0000 7fe2c169c640 1 --2- 172.21.15.125:0/4184123403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2bc0c8430 0x7fe2bc132f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:27.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.197+0000 7fe2c169c640 1 --2- 172.21.15.125:0/4184123403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2bc0c6f60 0x7fe2bc132a50 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:27.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.197+0000 7fe2c169c640 1 --2- 172.21.15.125:0/4184123403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2bc0c6b80 0x7fe2bc1303f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:27.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.197+0000 7fe2c169c640 1 -- 172.21.15.125:0/4184123403 >> 172.21.15.125:0/4184123403 conn(0x7fe2bc0c7910 msgr2=0x7fe2bc0b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:27.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.197+0000 7fe2c169c640 1 -- 172.21.15.125:0/4184123403 shutdown_connections 2023-10-30T12:36:27.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:27.197+0000 7fe2c169c640 1 -- 172.21.15.125:0/4184123403 wait complete. 2023-10-30T12:36:27.220 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:36:29.221 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:29.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f0818ec9640 1 Processor -- start 2023-10-30T12:36:29.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f0818ec9640 1 -- start start 2023-10-30T12:36:29.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f0818ec9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f08140c6fe0 0x7f0814071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:29.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f0818ec9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140700b0 con 0x7f0814153bf0 2023-10-30T12:36:29.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f0818ec9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08140c6530 con 0x7f08140c5f60 2023-10-30T12:36:29.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f0818ec9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08140c6810 con 0x7f08140c6fe0 2023-10-30T12:36:29.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081259c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0814153bf0 0x7f08140c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:50112/0 (socket says 172.21.15.125:50112) 2023-10-30T12:36:29.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081259c640 1 -- 172.21.15.125:0/1180780469 learned_addr learned my addr 172.21.15.125:0/1180780469 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:29.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f0811d9b640 1 --2- 172.21.15.125:0/1180780469 >> v2:172.21.15.136:3300/0 conn(0x7f08140c6fe0 0x7f0814071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:29.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 -- 172.21.15.125:0/1180780469 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876341233 0 0) 0x7f08140700b0 con 0x7f0814153bf0 2023-10-30T12:36:29.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 -- 172.21.15.125:0/1180780469 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07fc003530 con 0x7f0814153bf0 2023-10-30T12:36:29.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 -- 172.21.15.125:0/1180780469 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3464158731 0 0) 0x7f07fc003530 con 0x7f0814153bf0 2023-10-30T12:36:29.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 -- 172.21.15.125:0/1180780469 >> v2:172.21.15.136:3300/0 conn(0x7f08140c6fe0 msgr2=0x7f0814071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:29.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 --2- 172.21.15.125:0/1180780469 >> v2:172.21.15.136:3300/0 conn(0x7f08140c6fe0 0x7f0814071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:29.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 -- 172.21.15.125:0/1180780469 >> v1:172.21.15.136:6789/0 conn(0x7f08140c5f60 legacy=0x7f08140cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:29.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 -- 172.21.15.125:0/1180780469 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0814159b10 con 0x7f0814153bf0 2023-10-30T12:36:29.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 -- 172.21.15.125:0/1180780469 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f0804001680 con 0x7f0814153bf0 2023-10-30T12:36:29.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f0811d9b640 1 --2- 172.21.15.125:0/1180780469 >> v2:172.21.15.136:3300/0 conn(0x7f08140c6fe0 0x7f0814071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:36:29.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 -- 172.21.15.125:0/1180780469 >> v1:172.21.15.125:6789/0 conn(0x7f0814153bf0 legacy=0x7f08140c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:29.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 --2- 172.21.15.125:0/1180780469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07fc003980 0x7f07fc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:29.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 --2- 172.21.15.125:0/1180780469 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07fc0047d0 0x7f07fc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:29.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 --2- 172.21.15.125:0/1180780469 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07fc005190 0x7f07fc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:29.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 -- 172.21.15.125:0/1180780469 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08140c6810 con 0x7f07fc0047d0 2023-10-30T12:36:29.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 -- 172.21.15.125:0/1180780469 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc00c040 con 0x7f07fc003980 2023-10-30T12:36:29.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f081159a640 1 -- 172.21.15.125:0/1180780469 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07fc00c200 con 0x7f07fc005190 2023-10-30T12:36:29.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.369+0000 7f0811d9b640 1 --2- 172.21.15.125:0/1180780469 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07fc0047d0 0x7f07fc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:29.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.373+0000 7f081259c640 1 --2- 172.21.15.125:0/1180780469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07fc003980 0x7f07fc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:29.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.373+0000 7f0812d9d640 1 --2- 172.21.15.125:0/1180780469 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07fc005190 0x7f07fc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:29.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.373+0000 7f0811d9b640 1 -- 172.21.15.125:0/1180780469 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07fc005190 msgr2=0x7f07fc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:29.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.373+0000 7f0811d9b640 1 --2- 172.21.15.125:0/1180780469 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07fc005190 0x7f07fc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:29.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.373+0000 7f0811d9b640 1 -- 172.21.15.125:0/1180780469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07fc003980 msgr2=0x7f07fc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:29.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.373+0000 7f0811d9b640 1 --2- 172.21.15.125:0/1180780469 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07fc003980 0x7f07fc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:29.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.373+0000 7f0811d9b640 1 -- 172.21.15.125:0/1180780469 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07fc003530 con 0x7f07fc0047d0 2023-10-30T12:36:29.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.373+0000 7f0812d9d640 1 -- 172.21.15.125:0/1180780469 reap_dead start 2023-10-30T12:36:29.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.373+0000 7f0811d9b640 1 --2- 172.21.15.125:0/1180780469 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07fc0047d0 0x7f07fc004c50 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0808009cb0 tx=0x7f080801cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f5bdac411875a5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:29.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.373+0000 7f081159a640 1 -- 172.21.15.125:0/1180780469 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08080a58b0 con 0x7f07fc0047d0 2023-10-30T12:36:29.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.373+0000 7f081159a640 1 -- 172.21.15.125:0/1180780469 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08080a5a10 con 0x7f07fc0047d0 2023-10-30T12:36:29.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.373+0000 7f081159a640 1 -- 172.21.15.125:0/1180780469 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08080a5e00 con 0x7f07fc0047d0 2023-10-30T12:36:29.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 -- 172.21.15.125:0/1180780469 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07fc0047d0 msgr2=0x7f07fc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:29.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 --2- 172.21.15.125:0/1180780469 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07fc0047d0 0x7f07fc004c50 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0808009cb0 tx=0x7f080801cbf0 comp rx=0 tx=0).stop 2023-10-30T12:36:29.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 -- 172.21.15.125:0/1180780469 shutdown_connections 2023-10-30T12:36:29.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 --2- 172.21.15.125:0/1180780469 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07fc0047d0 0x7f07fc004c50 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:29.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 -- 172.21.15.125:0/1180780469 >> 172.21.15.125:0/1180780469 conn(0x7f081406c6e0 msgr2=0x7f0814069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:29.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 -- 172.21.15.125:0/1180780469 shutdown_connections 2023-10-30T12:36:29.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 -- 172.21.15.125:0/1180780469 wait complete. 2023-10-30T12:36:29.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 Processor -- start 2023-10-30T12:36:29.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 -- start start 2023-10-30T12:36:29.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07fc0047d0 0x7f08141543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:29.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0814158f40 0x7f0814154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:29.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08140cb010 0x7f081414ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:29.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0814154920 con 0x7f0814158f40 2023-10-30T12:36:29.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0814156f50 con 0x7f08140cb010 2023-10-30T12:36:29.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0818ec9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08141571f0 con 0x7f07fc0047d0 2023-10-30T12:36:29.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0811d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0814158f40 0x7f0814154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:29.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0811d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0814158f40 0x7f0814154ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49642/0 (socket says 172.21.15.125:49642) 2023-10-30T12:36:29.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0811d9b640 1 -- 172.21.15.125:0/2071767939 learned_addr learned my addr 172.21.15.125:0/2071767939 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:29.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0812d9d640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08140cb010 0x7f081414ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:29.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f081259c640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07fc0047d0 0x7f08141543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:29.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0811d9b640 1 -- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07fc0047d0 msgr2=0x7f08141543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:29.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0811d9b640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07fc0047d0 0x7f08141543e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:29.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0811d9b640 1 -- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08140cb010 msgr2=0x7f081414ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:29.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0811d9b640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08140cb010 0x7f081414ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:29.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0811d9b640 1 -- 172.21.15.125:0/2071767939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0814158180 con 0x7f0814158f40 2023-10-30T12:36:29.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f081259c640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07fc0047d0 0x7f08141543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:36:29.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0812d9d640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08140cb010 0x7f081414ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:36:29.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.377+0000 7f0811d9b640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0814158f40 0x7f0814154ca0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0808009cb0 tx=0x7f0808019990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:29.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.381+0000 7f0802ffd640 1 -- 172.21.15.125:0/2071767939 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08080b8860 con 0x7f0814158f40 2023-10-30T12:36:29.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.381+0000 7f0802ffd640 1 -- 172.21.15.125:0/2071767939 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08080b89e0 con 0x7f0814158f40 2023-10-30T12:36:29.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.381+0000 7f0802ffd640 1 -- 172.21.15.125:0/2071767939 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08080b8d90 con 0x7f0814158f40 2023-10-30T12:36:29.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.381+0000 7f0818ec9640 1 -- 172.21.15.125:0/2071767939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0814158460 con 0x7f0814158f40 2023-10-30T12:36:29.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.381+0000 7f0818ec9640 1 -- 172.21.15.125:0/2071767939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0814158a20 con 0x7f0814158f40 2023-10-30T12:36:29.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.381+0000 7f0818ec9640 1 -- 172.21.15.125:0/2071767939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0814158bb0 con 0x7f0814158f40 2023-10-30T12:36:29.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.381+0000 7f0802ffd640 1 -- 172.21.15.125:0/2071767939 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f08080b4070 con 0x7f0814158f40 2023-10-30T12:36:29.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.381+0000 7f0802ffd640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07f40664a0 0x7f07f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:29.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.381+0000 7f0802ffd640 1 -- 172.21.15.125:0/2071767939 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6040+0+0 (secure 0 0 0) 0x7f080811e0e0 con 0x7f0814158f40 2023-10-30T12:36:29.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.381+0000 7f081259c640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07f40664a0 0x7f07f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:29.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.385+0000 7f081259c640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07f40664a0 0x7f07f4068960 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0804001060 tx=0x7f0804009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:29.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.385+0000 7f0802ffd640 1 -- 172.21.15.125:0/2071767939 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f08080eb370 con 0x7f0814158f40 2023-10-30T12:36:29.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.565+0000 7f0818ec9640 1 -- 172.21.15.125:0/2071767939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f081404e170 con 0x7f0814158f40 2023-10-30T12:36:29.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.565+0000 7f0802ffd640 1 -- 172.21.15.125:0/2071767939 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+6551 (secure 0 0 0) 0x7f08080ee990 con 0x7f0814158f40 2023-10-30T12:36:29.571 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:29.571 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4973,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:03.067774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4294,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4294":{"gid":4294,"name":"e","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.125:6836/2017382874","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2017382874},{"type":"v1","addr":"172.21.15.125:6836","nonce":2017382874}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:29.572 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 44 2023-10-30T12:36:29.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.569+0000 7f0818ec9640 1 -- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07f40664a0 msgr2=0x7f07f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:29.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.569+0000 7f0818ec9640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07f40664a0 0x7f07f4068960 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0804001060 tx=0x7f0804009040 comp rx=0 tx=0).stop 2023-10-30T12:36:29.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.569+0000 7f0818ec9640 1 -- 172.21.15.125:0/2071767939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0814158f40 msgr2=0x7f0814154ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:29.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.569+0000 7f0818ec9640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0814158f40 0x7f0814154ca0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0808009cb0 tx=0x7f0808019990 comp rx=0 tx=0).stop 2023-10-30T12:36:29.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.569+0000 7f0818ec9640 1 -- 172.21.15.125:0/2071767939 shutdown_connections 2023-10-30T12:36:29.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.569+0000 7f0818ec9640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07f40664a0 0x7f07f4068960 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:29.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.569+0000 7f0818ec9640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08140cb010 0x7f081414ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:29.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.569+0000 7f0818ec9640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0814158f40 0x7f0814154ca0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:29.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.569+0000 7f0818ec9640 1 --2- 172.21.15.125:0/2071767939 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07fc0047d0 0x7f08141543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:29.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.569+0000 7f0818ec9640 1 -- 172.21.15.125:0/2071767939 >> 172.21.15.125:0/2071767939 conn(0x7f081406c6e0 msgr2=0x7f08140c7450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:29.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.569+0000 7f0818ec9640 1 -- 172.21.15.125:0/2071767939 shutdown_connections 2023-10-30T12:36:29.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:29.569+0000 7f0818ec9640 1 -- 172.21.15.125:0/2071767939 wait complete. 2023-10-30T12:36:29.592 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:36:31.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:31.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.737+0000 7f2300fa1640 1 Processor -- start 2023-10-30T12:36:31.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.737+0000 7f2300fa1640 1 -- start start 2023-10-30T12:36:31.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.737+0000 7f2300fa1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f22fc15d6f0 0x7f22fc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:31.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.737+0000 7f2300fa1640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc0cb800 con 0x7f22fc0cf160 2023-10-30T12:36:31.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.737+0000 7f2300fa1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22fc0cb330 con 0x7f22fc1538a0 2023-10-30T12:36:31.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.737+0000 7f2300fa1640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22fc0cad70 con 0x7f22fc15d6f0 2023-10-30T12:36:31.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.737+0000 7f22f1d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f22fc15d6f0 0x7f22fc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:31.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.737+0000 7f22fad9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f22fc0cf160 0x7f22fc164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38230/0 (socket says 172.21.15.125:38230) 2023-10-30T12:36:31.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.737+0000 7f22fad9d640 1 -- 172.21.15.125:0/3688144983 learned_addr learned my addr 172.21.15.125:0/3688144983 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:31.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.737+0000 7f22f1d9b640 1 -- 172.21.15.125:0/3688144983 >> v1:172.21.15.136:6789/0 conn(0x7f22fc1538a0 legacy=0x7f22fc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:31.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.737+0000 7f22f1d9b640 1 -- 172.21.15.125:0/3688144983 >> v1:172.21.15.125:6789/0 conn(0x7f22fc0cf160 legacy=0x7f22fc164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:31.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.737+0000 7f22f1d9b640 1 -- 172.21.15.125:0/3688144983 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22fc0cb050 con 0x7f22fc15d6f0 2023-10-30T12:36:31.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.741+0000 7f22f1d9b640 1 --2- 172.21.15.125:0/3688144983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22fc15d6f0 0x7f22fc15a090 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0097c0 tx=0x7f22ec01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa3d6bab43ddb4e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:31.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.741+0000 7f22f9d9b640 1 -- 172.21.15.125:0/3688144983 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec0a8970 con 0x7f22fc15d6f0 2023-10-30T12:36:31.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.741+0000 7f22f9d9b640 1 -- 172.21.15.125:0/3688144983 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ec0a8ad0 con 0x7f22fc15d6f0 2023-10-30T12:36:31.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.741+0000 7f22f9d9b640 1 -- 172.21.15.125:0/3688144983 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec0a8e60 con 0x7f22fc15d6f0 2023-10-30T12:36:31.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.741+0000 7f2300fa1640 1 -- 172.21.15.125:0/3688144983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22fc15d6f0 msgr2=0x7f22fc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:31.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.741+0000 7f2300fa1640 1 --2- 172.21.15.125:0/3688144983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22fc15d6f0 0x7f22fc15a090 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0097c0 tx=0x7f22ec01ec70 comp rx=0 tx=0).stop 2023-10-30T12:36:31.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.741+0000 7f2300fa1640 1 -- 172.21.15.125:0/3688144983 shutdown_connections 2023-10-30T12:36:31.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.741+0000 7f2300fa1640 1 --2- 172.21.15.125:0/3688144983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22fc15d6f0 0x7f22fc15a090 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:31.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.741+0000 7f2300fa1640 1 -- 172.21.15.125:0/3688144983 >> 172.21.15.125:0/3688144983 conn(0x7f22fc06bfe0 msgr2=0x7f22fc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:31.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.741+0000 7f2300fa1640 1 -- 172.21.15.125:0/3688144983 shutdown_connections 2023-10-30T12:36:31.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.741+0000 7f2300fa1640 1 -- 172.21.15.125:0/3688144983 wait complete. 2023-10-30T12:36:31.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f2300fa1640 1 Processor -- start 2023-10-30T12:36:31.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f2300fa1640 1 -- start start 2023-10-30T12:36:31.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f2300fa1640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22fc0cf160 0x7f22fc1e3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:31.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f2300fa1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22fc1538a0 0x7f22fc1592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:31.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f2300fa1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22fc15d6f0 0x7f22fc0c87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:31.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f2300fa1640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc0c8ce0 con 0x7f22fc1538a0 2023-10-30T12:36:31.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f2300fa1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc0c8310 con 0x7f22fc15d6f0 2023-10-30T12:36:31.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f2300fa1640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22fc0c7060 con 0x7f22fc0cf160 2023-10-30T12:36:31.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f22f1d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22fc1538a0 0x7f22fc1592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:31.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f22f1d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22fc1538a0 0x7f22fc1592a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33560/0 (socket says 172.21.15.125:33560) 2023-10-30T12:36:31.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f22f1d9b640 1 -- 172.21.15.125:0/2345106251 learned_addr learned my addr 172.21.15.125:0/2345106251 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:31.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f22fad9d640 1 --2- 172.21.15.125:0/2345106251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22fc15d6f0 0x7f22fc0c87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:31.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f22fa59c640 1 --2- 172.21.15.125:0/2345106251 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22fc0cf160 0x7f22fc1e3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:31.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f22f1d9b640 1 -- 172.21.15.125:0/2345106251 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22fc0cf160 msgr2=0x7f22fc1e3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:31.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f22f1d9b640 1 --2- 172.21.15.125:0/2345106251 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22fc0cf160 0x7f22fc1e3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:31.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f22f1d9b640 1 -- 172.21.15.125:0/2345106251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22fc15d6f0 msgr2=0x7f22fc0c87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:31.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f22f1d9b640 1 --2- 172.21.15.125:0/2345106251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22fc15d6f0 0x7f22fc0c87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:31.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f22f1d9b640 1 -- 172.21.15.125:0/2345106251 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec009420 con 0x7f22fc1538a0 2023-10-30T12:36:31.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f22f1d9b640 1 --2- 172.21.15.125:0/2345106251 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22fc1538a0 0x7f22fc1592a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0b3040 tx=0x7f22ec01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:31.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f22f37fe640 1 -- 172.21.15.125:0/2345106251 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec0b9630 con 0x7f22fc1538a0 2023-10-30T12:36:31.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f22f37fe640 1 -- 172.21.15.125:0/2345106251 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22ec0b97b0 con 0x7f22fc1538a0 2023-10-30T12:36:31.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f2300fa1640 1 -- 172.21.15.125:0/2345106251 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22fc0c7d00 con 0x7f22fc1538a0 2023-10-30T12:36:31.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f22f37fe640 1 -- 172.21.15.125:0/2345106251 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ec0b9b60 con 0x7f22fc1538a0 2023-10-30T12:36:31.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.745+0000 7f2300fa1640 1 -- 172.21.15.125:0/2345106251 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22fc0c6aa0 con 0x7f22fc1538a0 2023-10-30T12:36:31.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.749+0000 7f2300fa1640 1 -- 172.21.15.125:0/2345106251 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22fc0c84a0 con 0x7f22fc1538a0 2023-10-30T12:36:31.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.749+0000 7f22f37fe640 1 -- 172.21.15.125:0/2345106251 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f22ec0b7040 con 0x7f22fc1538a0 2023-10-30T12:36:31.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.749+0000 7f22f37fe640 1 --2- 172.21.15.125:0/2345106251 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22dc0664a0 0x7f22dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:31.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.749+0000 7f22f37fe640 1 -- 172.21.15.125:0/2345106251 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f22ec1220f0 con 0x7f22fc1538a0 2023-10-30T12:36:31.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.749+0000 7f22fa59c640 1 --2- 172.21.15.125:0/2345106251 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22dc0664a0 0x7f22dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:31.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.753+0000 7f22fa59c640 1 --2- 172.21.15.125:0/2345106251 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22dc0664a0 0x7f22dc068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f22e4000f70 tx=0x7f22e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:31.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.753+0000 7f22f37fe640 1 -- 172.21.15.125:0/2345106251 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f22ec0ef470 con 0x7f22fc1538a0 2023-10-30T12:36:31.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.933+0000 7f2300fa1640 1 -- 172.21.15.125:0/2345106251 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22fc04e170 con 0x7f22fc1538a0 2023-10-30T12:36:31.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.937+0000 7f22f37fe640 1 -- 172.21.15.125:0/2345106251 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5755 (secure 0 0 0) 0x7f22ec0b5090 con 0x7f22fc1538a0 2023-10-30T12:36:31.939 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:31.940 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:30.049299+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:31.940 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 45 2023-10-30T12:36:31.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.937+0000 7f2300fa1640 1 -- 172.21.15.125:0/2345106251 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22dc0664a0 msgr2=0x7f22dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:31.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.937+0000 7f2300fa1640 1 --2- 172.21.15.125:0/2345106251 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22dc0664a0 0x7f22dc068960 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f22e4000f70 tx=0x7f22e4009040 comp rx=0 tx=0).stop 2023-10-30T12:36:31.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.937+0000 7f2300fa1640 1 -- 172.21.15.125:0/2345106251 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22fc1538a0 msgr2=0x7f22fc1592a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:31.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.937+0000 7f2300fa1640 1 --2- 172.21.15.125:0/2345106251 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22fc1538a0 0x7f22fc1592a0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0b3040 tx=0x7f22ec01ba10 comp rx=0 tx=0).stop 2023-10-30T12:36:31.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.937+0000 7f2300fa1640 1 -- 172.21.15.125:0/2345106251 shutdown_connections 2023-10-30T12:36:31.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.937+0000 7f2300fa1640 1 --2- 172.21.15.125:0/2345106251 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22dc0664a0 0x7f22dc068960 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:31.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.937+0000 7f2300fa1640 1 --2- 172.21.15.125:0/2345106251 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22fc15d6f0 0x7f22fc0c87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:31.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.937+0000 7f2300fa1640 1 --2- 172.21.15.125:0/2345106251 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22fc1538a0 0x7f22fc1592a0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:31.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.937+0000 7f2300fa1640 1 --2- 172.21.15.125:0/2345106251 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22fc0cf160 0x7f22fc1e3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:31.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.937+0000 7f2300fa1640 1 -- 172.21.15.125:0/2345106251 >> 172.21.15.125:0/2345106251 conn(0x7f22fc06bfe0 msgr2=0x7f22fc0ce1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:31.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.937+0000 7f2300fa1640 1 -- 172.21.15.125:0/2345106251 shutdown_connections 2023-10-30T12:36:31.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:31.937+0000 7f2300fa1640 1 -- 172.21.15.125:0/2345106251 wait complete. 2023-10-30T12:36:31.960 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T12:36:31.960 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:36:31.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:32.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7b2fa1640 1 Processor -- start 2023-10-30T12:36:32.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7b2fa1640 1 -- start start 2023-10-30T12:36:32.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7b2fa1640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7ac1538a0 0x7fc7ac15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:32.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7b2fa1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc7ac0cf160 0x7fc7ac0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:32.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7b2fa1640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ac0cb800 con 0x7fc7ac0ca0e0 2023-10-30T12:36:32.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7b2fa1640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ac15dc30 con 0x7fc7ac0cf160 2023-10-30T12:36:32.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7b2fa1640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0cb330 con 0x7fc7ac1538a0 2023-10-30T12:36:32.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a3fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc7ac0ca0e0 0x7fc7ac15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38240/0 (socket says 172.21.15.125:38240) 2023-10-30T12:36:32.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a3fff640 1 -- 172.21.15.125:0/507406854 learned_addr learned my addr 172.21.15.125:0/507406854 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:32.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7b0d3d640 1 --2- 172.21.15.125:0/507406854 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7ac1538a0 0x7fc7ac15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:32.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7b153e640 1 --2- 172.21.15.125:0/507406854 >> v2:172.21.15.136:3300/0 conn(0x7fc7ac0cf160 0x7fc7ac0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:32.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 -- 172.21.15.125:0/507406854 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067071480 0 0) 0x7fc7ac0cb800 con 0x7fc7ac0ca0e0 2023-10-30T12:36:32.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 -- 172.21.15.125:0/507406854 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc794003530 con 0x7fc7ac0ca0e0 2023-10-30T12:36:32.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 -- 172.21.15.125:0/507406854 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 614859161 0 0) 0x7fc794003530 con 0x7fc7ac0ca0e0 2023-10-30T12:36:32.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 -- 172.21.15.125:0/507406854 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7ac1538a0 msgr2=0x7fc7ac15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:32.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 --2- 172.21.15.125:0/507406854 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7ac1538a0 0x7fc7ac15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:32.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 -- 172.21.15.125:0/507406854 >> v2:172.21.15.136:3300/0 conn(0x7fc7ac0cf160 msgr2=0x7fc7ac0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:32.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 --2- 172.21.15.125:0/507406854 >> v2:172.21.15.136:3300/0 conn(0x7fc7ac0cf160 0x7fc7ac0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:32.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 -- 172.21.15.125:0/507406854 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac0cad70 con 0x7fc7ac0ca0e0 2023-10-30T12:36:32.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 -- 172.21.15.125:0/507406854 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fc79c001680 con 0x7fc7ac0ca0e0 2023-10-30T12:36:32.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 -- 172.21.15.125:0/507406854 >> v1:172.21.15.125:6789/0 conn(0x7fc7ac0ca0e0 legacy=0x7fc7ac15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:32.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 --2- 172.21.15.125:0/507406854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc794003980 0x7fc794003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:32.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 --2- 172.21.15.125:0/507406854 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7940047d0 0x7fc794004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:32.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 --2- 172.21.15.125:0/507406854 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7ac1538a0 0x7fc794005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:32.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 -- 172.21.15.125:0/507406854 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac15dc30 con 0x7fc7940047d0 2023-10-30T12:36:32.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 -- 172.21.15.125:0/507406854 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0cb330 con 0x7fc794003980 2023-10-30T12:36:32.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a37fe640 1 -- 172.21.15.125:0/507406854 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc79400b990 con 0x7fc7ac1538a0 2023-10-30T12:36:32.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7a3fff640 1 --2- 172.21.15.125:0/507406854 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7940047d0 0x7fc794004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:32.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7b0d3d640 1 --2- 172.21.15.125:0/507406854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc794003980 0x7fc794003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:32.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.097+0000 7fc7b153e640 1 --2- 172.21.15.125:0/507406854 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7ac1538a0 0x7fc794005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:32.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7a3fff640 1 -- 172.21.15.125:0/507406854 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7ac1538a0 msgr2=0x7fc794005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:32.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7a3fff640 1 --2- 172.21.15.125:0/507406854 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7ac1538a0 0x7fc794005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:32.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7a3fff640 1 -- 172.21.15.125:0/507406854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc794003980 msgr2=0x7fc794003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:32.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7a3fff640 1 --2- 172.21.15.125:0/507406854 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc794003980 0x7fc794003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:32.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7a3fff640 1 -- 172.21.15.125:0/507406854 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc794003530 con 0x7fc7940047d0 2023-10-30T12:36:32.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7a3fff640 1 --2- 172.21.15.125:0/507406854 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7940047d0 0x7fc794004c50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc79c00c240 tx=0x7fc79c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1368862d98e06736 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:32.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7a37fe640 1 -- 172.21.15.125:0/507406854 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c0058e0 con 0x7fc7940047d0 2023-10-30T12:36:32.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7a37fe640 1 -- 172.21.15.125:0/507406854 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c005a40 con 0x7fc7940047d0 2023-10-30T12:36:32.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7a37fe640 1 -- 172.21.15.125:0/507406854 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c005e10 con 0x7fc7940047d0 2023-10-30T12:36:32.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/507406854 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7940047d0 msgr2=0x7fc794004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:32.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7b2fa1640 1 --2- 172.21.15.125:0/507406854 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7940047d0 0x7fc794004c50 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc79c00c240 tx=0x7fc79c01cc60 comp rx=0 tx=0).stop 2023-10-30T12:36:32.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7b153e640 1 -- 172.21.15.125:0/507406854 reap_dead start 2023-10-30T12:36:32.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/507406854 shutdown_connections 2023-10-30T12:36:32.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/507406854 >> 172.21.15.125:0/507406854 conn(0x7fc7ac06bfe0 msgr2=0x7fc7ac06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:32.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/507406854 shutdown_connections 2023-10-30T12:36:32.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.101+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/507406854 wait complete. 2023-10-30T12:36:32.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b2fa1640 1 Processor -- start 2023-10-30T12:36:32.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b2fa1640 1 -- start start 2023-10-30T12:36:32.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b2fa1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7940047d0 0x7fc7ac0cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:32.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b2fa1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7ac0ce4b0 0x7fc7ac147dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:32.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b2fa1640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7ac1593d0 0x7fc7ac0b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:32.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b2fa1640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0cd840 con 0x7fc7940047d0 2023-10-30T12:36:32.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b2fa1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac15cc50 con 0x7fc7ac0ce4b0 2023-10-30T12:36:32.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b2fa1640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ac0b1960 con 0x7fc7ac1593d0 2023-10-30T12:36:32.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b0d3d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7940047d0 0x7fc7ac0cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:32.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b0d3d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7940047d0 0x7fc7ac0cdca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33570/0 (socket says 172.21.15.125:33570) 2023-10-30T12:36:32.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b0d3d640 1 -- 172.21.15.125:0/2386715327 learned_addr learned my addr 172.21.15.125:0/2386715327 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:32.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b153e640 1 --2- 172.21.15.125:0/2386715327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7ac1593d0 0x7fc7ac0b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:32.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7a3fff640 1 --2- 172.21.15.125:0/2386715327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7ac0ce4b0 0x7fc7ac147dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:32.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b0d3d640 1 -- 172.21.15.125:0/2386715327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7ac1593d0 msgr2=0x7fc7ac0b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:32.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b0d3d640 1 --2- 172.21.15.125:0/2386715327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7ac1593d0 0x7fc7ac0b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:32.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b0d3d640 1 -- 172.21.15.125:0/2386715327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7ac0ce4b0 msgr2=0x7fc7ac147dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:32.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b0d3d640 1 --2- 172.21.15.125:0/2386715327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7ac0ce4b0 0x7fc7ac147dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:32.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b0d3d640 1 -- 172.21.15.125:0/2386715327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ac0ce1e0 con 0x7fc7940047d0 2023-10-30T12:36:32.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b0d3d640 1 --2- 172.21.15.125:0/2386715327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7940047d0 0x7fc7ac0cdca0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8011960 tx=0x7fc7a8011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:32.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7a17fa640 1 -- 172.21.15.125:0/2386715327 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a800faf0 con 0x7fc7940047d0 2023-10-30T12:36:32.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7a17fa640 1 -- 172.21.15.125:0/2386715327 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a800fc50 con 0x7fc7940047d0 2023-10-30T12:36:32.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/2386715327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ac0b1bb0 con 0x7fc7940047d0 2023-10-30T12:36:32.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7a17fa640 1 -- 172.21.15.125:0/2386715327 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a8014440 con 0x7fc7940047d0 2023-10-30T12:36:32.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.105+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/2386715327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ac0ceb70 con 0x7fc7940047d0 2023-10-30T12:36:32.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.109+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/2386715327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ac0ce9d0 con 0x7fc7940047d0 2023-10-30T12:36:32.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.109+0000 7fc7a17fa640 1 -- 172.21.15.125:0/2386715327 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7a80145a0 con 0x7fc7940047d0 2023-10-30T12:36:32.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.109+0000 7fc7a17fa640 1 --2- 172.21.15.125:0/2386715327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7980664d0 0x7fc798068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:32.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.109+0000 7fc7a17fa640 1 -- 172.21.15.125:0/2386715327 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fc7a8084810 con 0x7fc7940047d0 2023-10-30T12:36:32.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.109+0000 7fc7a3fff640 1 --2- 172.21.15.125:0/2386715327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7980664d0 0x7fc798068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:32.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.109+0000 7fc7a3fff640 1 --2- 172.21.15.125:0/2386715327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7980664d0 0x7fc798068990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0aa000 tx=0x7fc79c001a80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:32.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.113+0000 7fc7a17fa640 1 -- 172.21.15.125:0/2386715327 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7ac0ce9d0 con 0x7fc7940047d0 2023-10-30T12:36:32.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.297+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/2386715327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7ac0ce9d0 con 0x7fc7940047d0 2023-10-30T12:36:32.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.297+0000 7fc7a17fa640 1 -- 172.21.15.125:0/2386715327 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5755 (secure 0 0 0) 0x7fc7ac0ce9d0 con 0x7fc7940047d0 2023-10-30T12:36:32.300 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:32.300 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:30.049299+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:32.301 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 45 2023-10-30T12:36:32.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.297+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/2386715327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7980664d0 msgr2=0x7fc798068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:32.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.297+0000 7fc7b2fa1640 1 --2- 172.21.15.125:0/2386715327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7980664d0 0x7fc798068990 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc79c0aa000 tx=0x7fc79c001a80 comp rx=0 tx=0).stop 2023-10-30T12:36:32.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.297+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/2386715327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7940047d0 msgr2=0x7fc7ac0cdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:32.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.297+0000 7fc7b2fa1640 1 --2- 172.21.15.125:0/2386715327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7940047d0 0x7fc7ac0cdca0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fc7a8011960 tx=0x7fc7a8011e20 comp rx=0 tx=0).stop 2023-10-30T12:36:32.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.297+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/2386715327 shutdown_connections 2023-10-30T12:36:32.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.297+0000 7fc7b2fa1640 1 --2- 172.21.15.125:0/2386715327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7980664d0 0x7fc798068990 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:32.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.297+0000 7fc7b2fa1640 1 --2- 172.21.15.125:0/2386715327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7ac1593d0 0x7fc7ac0b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:32.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.297+0000 7fc7b2fa1640 1 --2- 172.21.15.125:0/2386715327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7ac0ce4b0 0x7fc7ac147dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:32.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.297+0000 7fc7b2fa1640 1 --2- 172.21.15.125:0/2386715327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7940047d0 0x7fc7ac0cdca0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:32.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.297+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/2386715327 >> 172.21.15.125:0/2386715327 conn(0x7fc7ac06bfe0 msgr2=0x7fc7ac152550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:32.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.297+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/2386715327 shutdown_connections 2023-10-30T12:36:32.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:32.297+0000 7fc7b2fa1640 1 -- 172.21.15.125:0/2386715327 wait complete. 2023-10-30T12:36:32.322 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 45, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 45, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4957": { "addr": "172.21.15.136:6835/2548530941", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2548530941, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2548530941, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4957, "incarnation": 23, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 34 }, "gid_4973": { "addr": "172.21.15.136:6839/1454326577", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 1454326577, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 1454326577, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4973, "incarnation": 45, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_5130": { "addr": "172.21.15.125:6837/1624923396", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 1624923396, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 1624923396, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5130, "incarnation": 37, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:36:30.049299+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4973, "mds_1": 5130, "mds_2": 4957 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:36:34.324 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:34.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdecb09c640 1 Processor -- start 2023-10-30T12:36:34.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdecb09c640 1 -- start start 2023-10-30T12:36:34.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdecb09c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdec41538a0 0x7fdec415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:34.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdecb09c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdec40cf160 0x7fdec40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:34.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdecb09c640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec40cb800 con 0x7fdec40ca0e0 2023-10-30T12:36:34.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdecb09c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec415dc30 con 0x7fdec41538a0 2023-10-30T12:36:34.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdecb09c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec40cb330 con 0x7fdec40cf160 2023-10-30T12:36:34.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebbfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdec40ca0e0 0x7fdec415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38244/0 (socket says 172.21.15.125:38244) 2023-10-30T12:36:34.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebbfff640 1 -- 172.21.15.125:0/3086393126 learned_addr learned my addr 172.21.15.125:0/3086393126 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:34.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdec8e38640 1 --2- 172.21.15.125:0/3086393126 >> v2:172.21.15.136:3300/0 conn(0x7fdec41538a0 0x7fdec415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:34.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdec9639640 1 --2- 172.21.15.125:0/3086393126 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdec40cf160 0x7fdec40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:34.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 -- 172.21.15.125:0/3086393126 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249428511 0 0) 0x7fdec40cb800 con 0x7fdec40ca0e0 2023-10-30T12:36:34.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 -- 172.21.15.125:0/3086393126 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdeac003530 con 0x7fdec40ca0e0 2023-10-30T12:36:34.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 -- 172.21.15.125:0/3086393126 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 48561332 0 0) 0x7fdeac003530 con 0x7fdec40ca0e0 2023-10-30T12:36:34.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 -- 172.21.15.125:0/3086393126 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdec40cf160 msgr2=0x7fdec40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:34.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 --2- 172.21.15.125:0/3086393126 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdec40cf160 0x7fdec40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:34.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 -- 172.21.15.125:0/3086393126 >> v2:172.21.15.136:3300/0 conn(0x7fdec41538a0 msgr2=0x7fdec415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:34.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 --2- 172.21.15.125:0/3086393126 >> v2:172.21.15.136:3300/0 conn(0x7fdec41538a0 0x7fdec415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:34.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 -- 172.21.15.125:0/3086393126 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec40cad70 con 0x7fdec40ca0e0 2023-10-30T12:36:34.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 -- 172.21.15.125:0/3086393126 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fdeb4001680 con 0x7fdec40ca0e0 2023-10-30T12:36:34.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 -- 172.21.15.125:0/3086393126 >> v1:172.21.15.125:6789/0 conn(0x7fdec40ca0e0 legacy=0x7fdec415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:34.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 --2- 172.21.15.125:0/3086393126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeac003980 0x7fdeac003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:34.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 --2- 172.21.15.125:0/3086393126 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdec40cf160 0x7fdeac0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:34.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 --2- 172.21.15.125:0/3086393126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdeac004d10 0x7fdeac005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:34.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 -- 172.21.15.125:0/3086393126 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec415dc30 con 0x7fdeac003980 2023-10-30T12:36:34.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 -- 172.21.15.125:0/3086393126 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec40cb330 con 0x7fdeac004d10 2023-10-30T12:36:34.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebb7fe640 1 -- 172.21.15.125:0/3086393126 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeac00b990 con 0x7fdec40cf160 2023-10-30T12:36:34.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdec8e38640 1 --2- 172.21.15.125:0/3086393126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeac003980 0x7fdeac003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:34.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebbfff640 1 --2- 172.21.15.125:0/3086393126 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdec40cf160 0x7fdeac0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:34.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdec9639640 1 --2- 172.21.15.125:0/3086393126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdeac004d10 0x7fdeac005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:34.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdec8e38640 1 -- 172.21.15.125:0/3086393126 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdec40cf160 msgr2=0x7fdeac0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:34.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdec8e38640 1 --2- 172.21.15.125:0/3086393126 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdec40cf160 0x7fdeac0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:34.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdec8e38640 1 -- 172.21.15.125:0/3086393126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdeac004d10 msgr2=0x7fdeac005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:34.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdec8e38640 1 --2- 172.21.15.125:0/3086393126 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdeac004d10 0x7fdeac005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:34.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdec8e38640 1 -- 172.21.15.125:0/3086393126 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac003530 con 0x7fdeac003980 2023-10-30T12:36:34.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.469+0000 7fdebbfff640 1 --2- 172.21.15.125:0/3086393126 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdec40cf160 0x7fdeac0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:36:34.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.473+0000 7fdec8e38640 1 --2- 172.21.15.125:0/3086393126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeac003980 0x7fdeac003da0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009ae0 tx=0x7fdec001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28e6cb9bece85c59 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:34.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.473+0000 7fdebb7fe640 1 -- 172.21.15.125:0/3086393126 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec00a5680 con 0x7fdeac003980 2023-10-30T12:36:34.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.473+0000 7fdebb7fe640 1 -- 172.21.15.125:0/3086393126 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec00a57e0 con 0x7fdeac003980 2023-10-30T12:36:34.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.473+0000 7fdebb7fe640 1 -- 172.21.15.125:0/3086393126 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec00a5b90 con 0x7fdeac003980 2023-10-30T12:36:34.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.473+0000 7fdecb09c640 1 -- 172.21.15.125:0/3086393126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeac003980 msgr2=0x7fdeac003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:34.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.473+0000 7fdecb09c640 1 --2- 172.21.15.125:0/3086393126 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdeac003980 0x7fdeac003da0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fdec0009ae0 tx=0x7fdec001ca90 comp rx=0 tx=0).stop 2023-10-30T12:36:34.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.473+0000 7fdec9639640 1 -- 172.21.15.125:0/3086393126 reap_dead start 2023-10-30T12:36:34.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.473+0000 7fdecb09c640 1 -- 172.21.15.125:0/3086393126 shutdown_connections 2023-10-30T12:36:34.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.473+0000 7fdecb09c640 1 -- 172.21.15.125:0/3086393126 >> 172.21.15.125:0/3086393126 conn(0x7fdec406bfe0 msgr2=0x7fdec406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:34.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.473+0000 7fdecb09c640 1 -- 172.21.15.125:0/3086393126 shutdown_connections 2023-10-30T12:36:34.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.473+0000 7fdecb09c640 1 -- 172.21.15.125:0/3086393126 wait complete. 2023-10-30T12:36:34.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdecb09c640 1 Processor -- start 2023-10-30T12:36:34.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdecb09c640 1 -- start start 2023-10-30T12:36:34.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdecb09c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdeac003980 0x7fdec4158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:34.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdecb09c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec415bcc0 0x7fdec41507c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:34.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdecb09c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec40ce260 0x7fdec41efac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:34.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdecb09c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec406c990 con 0x7fdec40ce260 2023-10-30T12:36:34.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdecb09c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec415b4e0 con 0x7fdec415bcc0 2023-10-30T12:36:34.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdecb09c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec41ef630 con 0x7fdeac003980 2023-10-30T12:36:34.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdec9639640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec40ce260 0x7fdec41efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:34.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdec9639640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec40ce260 0x7fdec41efac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33588/0 (socket says 172.21.15.125:33588) 2023-10-30T12:36:34.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdec9639640 1 -- 172.21.15.125:0/3942162709 learned_addr learned my addr 172.21.15.125:0/3942162709 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:34.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdebbfff640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec415bcc0 0x7fdec41507c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:34.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdec8e38640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdeac003980 0x7fdec4158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:34.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdec9639640 1 -- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdeac003980 msgr2=0x7fdec4158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:34.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdec9639640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdeac003980 0x7fdec4158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:34.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdec9639640 1 -- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec415bcc0 msgr2=0x7fdec41507c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:34.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdec9639640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec415bcc0 0x7fdec41507c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:34.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdec9639640 1 -- 172.21.15.125:0/3942162709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec4159100 con 0x7fdec40ce260 2023-10-30T12:36:34.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdebbfff640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec415bcc0 0x7fdec41507c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:36:34.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdec9639640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec40ce260 0x7fdec41efac0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fdec406cb00 tx=0x7fdebc0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:34.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdeb97fa640 1 -- 172.21.15.125:0/3942162709 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc00e070 con 0x7fdec40ce260 2023-10-30T12:36:34.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdeb97fa640 1 -- 172.21.15.125:0/3942162709 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdebc011e40 con 0x7fdec40ce260 2023-10-30T12:36:34.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdecb09c640 1 -- 172.21.15.125:0/3942162709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec41593a0 con 0x7fdec40ce260 2023-10-30T12:36:34.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.477+0000 7fdeb97fa640 1 -- 172.21.15.125:0/3942162709 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc0039a0 con 0x7fdec40ce260 2023-10-30T12:36:34.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.481+0000 7fdecb09c640 1 -- 172.21.15.125:0/3942162709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec415b9d0 con 0x7fdec40ce260 2023-10-30T12:36:34.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.481+0000 7fdecb09c640 1 -- 172.21.15.125:0/3942162709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec404e170 con 0x7fdec40ce260 2023-10-30T12:36:34.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.481+0000 7fdeb97fa640 1 -- 172.21.15.125:0/3942162709 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdebc003b00 con 0x7fdec40ce260 2023-10-30T12:36:34.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.481+0000 7fdeb97fa640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeb00664a0 0x7fdeb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:34.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.481+0000 7fdeb97fa640 1 -- 172.21.15.125:0/3942162709 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fdebc0848b0 con 0x7fdec40ce260 2023-10-30T12:36:34.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.481+0000 7fdec8e38640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeb00664a0 0x7fdeb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:34.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.485+0000 7fdec8e38640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeb00664a0 0x7fdeb0068960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdec00af000 tx=0x7fdec00046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:34.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.485+0000 7fdeb97fa640 1 -- 172.21.15.125:0/3942162709 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdebc051a80 con 0x7fdec40ce260 2023-10-30T12:36:34.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.665+0000 7fdecb09c640 1 -- 172.21.15.125:0/3942162709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdec40b1dd0 con 0x7fdec40ce260 2023-10-30T12:36:34.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.669+0000 7fdeb97fa640 1 -- 172.21.15.125:0/3942162709 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+5755 (secure 0 0 0) 0x7fdebc003db0 con 0x7fdec40ce260 2023-10-30T12:36:34.671 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:34.671 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":45,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:30.049299+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:34.672 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 45 2023-10-30T12:36:34.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.669+0000 7fdecb09c640 1 -- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeb00664a0 msgr2=0x7fdeb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:34.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.669+0000 7fdecb09c640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeb00664a0 0x7fdeb0068960 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdec00af000 tx=0x7fdec00046d0 comp rx=0 tx=0).stop 2023-10-30T12:36:34.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.669+0000 7fdecb09c640 1 -- 172.21.15.125:0/3942162709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec40ce260 msgr2=0x7fdec41efac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:34.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.669+0000 7fdecb09c640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec40ce260 0x7fdec41efac0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fdec406cb00 tx=0x7fdebc0115a0 comp rx=0 tx=0).stop 2023-10-30T12:36:34.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.669+0000 7fdecb09c640 1 -- 172.21.15.125:0/3942162709 shutdown_connections 2023-10-30T12:36:34.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.669+0000 7fdecb09c640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeb00664a0 0x7fdeb0068960 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:34.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.669+0000 7fdecb09c640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec40ce260 0x7fdec41efac0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:34.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.669+0000 7fdecb09c640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec415bcc0 0x7fdec41507c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:34.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.669+0000 7fdecb09c640 1 --2- 172.21.15.125:0/3942162709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdeac003980 0x7fdec4158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:34.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.669+0000 7fdecb09c640 1 -- 172.21.15.125:0/3942162709 >> 172.21.15.125:0/3942162709 conn(0x7fdec406bfe0 msgr2=0x7fdec4152660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.669+0000 7fdecb09c640 1 -- 172.21.15.125:0/3942162709 shutdown_connections 2023-10-30T12:36:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:34.669+0000 7fdecb09c640 1 -- 172.21.15.125:0/3942162709 wait complete. 2023-10-30T12:36:34.692 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:36:36.693 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:36.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e3ef0f640 1 Processor -- start 2023-10-30T12:36:36.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e3ef0f640 1 -- start start 2023-10-30T12:36:36.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e3ef0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e380c6b80 0x7f9e380bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:36.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e3ef0f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9e380c6f60 0x7f9e380c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:36.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e3ef0f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e380c0a50 con 0x7f9e380c8430 2023-10-30T12:36:36.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e3ef0f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e3806f6e0 con 0x7f9e380c6f60 2023-10-30T12:36:36.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e3ef0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e3806f9c0 con 0x7f9e380c6b80 2023-10-30T12:36:36.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e2ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9e380c6f60 0x7f9e380c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:36.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e3d4ac640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9e380c8430 0x7f9e3813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38256/0 (socket says 172.21.15.125:38256) 2023-10-30T12:36:36.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e3d4ac640 1 -- 172.21.15.125:0/3922816634 learned_addr learned my addr 172.21.15.125:0/3922816634 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:36.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e2ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9e380c6f60 0x7f9e380c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33600/0 (socket says 172.21.15.125:33600) 2023-10-30T12:36:36.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e3ccab640 1 --2- 172.21.15.125:0/3922816634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e380c6b80 0x7f9e380bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:36.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e2ffff640 1 -- 172.21.15.125:0/3922816634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e380c6b80 msgr2=0x7f9e380bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:36.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e2ffff640 1 --2- 172.21.15.125:0/3922816634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e380c6b80 0x7f9e380bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:36.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e2ffff640 1 -- 172.21.15.125:0/3922816634 >> v1:172.21.15.125:6789/0 conn(0x7f9e380c8430 legacy=0x7f9e3813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:36.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e2ffff640 1 -- 172.21.15.125:0/3922816634 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e3806fca0 con 0x7f9e380c6f60 2023-10-30T12:36:36.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e2f7fe640 1 -- 172.21.15.125:0/3922816634 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171169365 0 0) 0x7f9e380c0a50 con 0x7f9e380c8430 2023-10-30T12:36:36.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.837+0000 7f9e3ccab640 1 --2- 172.21.15.125:0/3922816634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e380c6b80 0x7f9e380bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:36:36.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.841+0000 7f9e2ffff640 1 --2- 172.21.15.125:0/3922816634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e380c6f60 0x7f9e380c0400 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9e280097c0 tx=0x7f9e2801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c287dd13c16a3a2a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:36.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.841+0000 7f9e2f7fe640 1 -- 172.21.15.125:0/3922816634 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e280ad640 con 0x7f9e380c6f60 2023-10-30T12:36:36.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.841+0000 7f9e2f7fe640 1 -- 172.21.15.125:0/3922816634 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e280ad7a0 con 0x7f9e380c6f60 2023-10-30T12:36:36.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.841+0000 7f9e2f7fe640 1 -- 172.21.15.125:0/3922816634 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e280adb30 con 0x7f9e380c6f60 2023-10-30T12:36:36.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.841+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/3922816634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e380c6f60 msgr2=0x7f9e380c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:36.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.841+0000 7f9e3ef0f640 1 --2- 172.21.15.125:0/3922816634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e380c6f60 0x7f9e380c0400 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f9e280097c0 tx=0x7f9e2801c480 comp rx=0 tx=0).stop 2023-10-30T12:36:36.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.841+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/3922816634 shutdown_connections 2023-10-30T12:36:36.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.841+0000 7f9e3ef0f640 1 --2- 172.21.15.125:0/3922816634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e380c6f60 0x7f9e380c0400 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:36.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.841+0000 7f9e3ef0f640 1 --2- 172.21.15.125:0/3922816634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e380c6b80 0x7f9e380bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:36.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.841+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/3922816634 >> 172.21.15.125:0/3922816634 conn(0x7f9e380c7910 msgr2=0x7f9e380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:36.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.841+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/3922816634 shutdown_connections 2023-10-30T12:36:36.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.841+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/3922816634 wait complete. 2023-10-30T12:36:36.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e3ef0f640 1 Processor -- start 2023-10-30T12:36:36.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e3ef0f640 1 -- start start 2023-10-30T12:36:36.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e3ef0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e380c6b80 0x7f9e3813df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:36.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e3ef0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e380c6f60 0x7f9e381f1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:36.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e3ef0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e380c8430 0x7f9e38155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:36.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e3ef0f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e380c7cf0 con 0x7f9e380c6f60 2023-10-30T12:36:36.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e3ef0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e381f5980 con 0x7f9e380c6b80 2023-10-30T12:36:36.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e3ef0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e381f5e10 con 0x7f9e380c8430 2023-10-30T12:36:36.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e2ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e380c6f60 0x7f9e381f1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:36.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e2ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e380c6f60 0x7f9e381f1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33616/0 (socket says 172.21.15.125:33616) 2023-10-30T12:36:36.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e2ffff640 1 -- 172.21.15.125:0/2384232818 learned_addr learned my addr 172.21.15.125:0/2384232818 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:36.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e3d4ac640 1 --2- 172.21.15.125:0/2384232818 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e380c8430 0x7f9e38155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:36.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e3ccab640 1 --2- 172.21.15.125:0/2384232818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e380c6b80 0x7f9e3813df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:36.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e2ffff640 1 -- 172.21.15.125:0/2384232818 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e380c8430 msgr2=0x7f9e38155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:36.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e2ffff640 1 --2- 172.21.15.125:0/2384232818 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e380c8430 0x7f9e38155860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:36.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e2ffff640 1 -- 172.21.15.125:0/2384232818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e380c6b80 msgr2=0x7f9e3813df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:36.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e2ffff640 1 --2- 172.21.15.125:0/2384232818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e380c6b80 0x7f9e3813df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:36.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e2ffff640 1 -- 172.21.15.125:0/2384232818 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e28009420 con 0x7f9e380c6f60 2023-10-30T12:36:36.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e2ffff640 1 --2- 172.21.15.125:0/2384232818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e380c6f60 0x7f9e381f1a40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9e2801a0b0 tx=0x7f9e28019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:36.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e2d7fa640 1 -- 172.21.15.125:0/2384232818 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e280b68e0 con 0x7f9e380c6f60 2023-10-30T12:36:36.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e2d7fa640 1 -- 172.21.15.125:0/2384232818 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e280a6040 con 0x7f9e380c6f60 2023-10-30T12:36:36.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e2d7fa640 1 -- 172.21.15.125:0/2384232818 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e280b6ca0 con 0x7f9e380c6f60 2023-10-30T12:36:36.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/2384232818 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e381f5050 con 0x7f9e380c6f60 2023-10-30T12:36:36.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.845+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/2384232818 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e381f5610 con 0x7f9e380c6f60 2023-10-30T12:36:36.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.849+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/2384232818 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e3804e170 con 0x7f9e380c6f60 2023-10-30T12:36:36.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.849+0000 7f9e2d7fa640 1 -- 172.21.15.125:0/2384232818 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9e280b4040 con 0x7f9e380c6f60 2023-10-30T12:36:36.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.849+0000 7f9e2d7fa640 1 --2- 172.21.15.125:0/2384232818 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9e240664d0 0x7f9e24068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:36.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.849+0000 7f9e2d7fa640 1 -- 172.21.15.125:0/2384232818 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f9e2811e5a0 con 0x7f9e380c6f60 2023-10-30T12:36:36.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.849+0000 7f9e3ccab640 1 --2- 172.21.15.125:0/2384232818 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9e240664d0 0x7f9e24068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:36.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.849+0000 7f9e3ccab640 1 --2- 172.21.15.125:0/2384232818 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9e240664d0 0x7f9e24068990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400c900 tx=0x7f9e34008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:36.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:36.853+0000 7f9e2d7fa640 1 -- 172.21.15.125:0/2384232818 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9e280eb8b0 con 0x7f9e380c6f60 2023-10-30T12:36:37.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:37.037+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/2384232818 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e38071c50 con 0x7f9e380c6f60 2023-10-30T12:36:37.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:37.041+0000 7f9e2d7fa640 1 -- 172.21.15.125:0/2384232818 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5757 (secure 0 0 0) 0x7f9e280eeed0 con 0x7f9e380c6f60 2023-10-30T12:36:37.043 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:37.043 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:36.055149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:resolve","state_seq":15,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:37.044 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 46 2023-10-30T12:36:37.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:37.041+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/2384232818 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9e240664d0 msgr2=0x7f9e24068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:37.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:37.041+0000 7f9e3ef0f640 1 --2- 172.21.15.125:0/2384232818 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9e240664d0 0x7f9e24068990 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9e3400c900 tx=0x7f9e34008040 comp rx=0 tx=0).stop 2023-10-30T12:36:37.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:37.041+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/2384232818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e380c6f60 msgr2=0x7f9e381f1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:37.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:37.041+0000 7f9e3ef0f640 1 --2- 172.21.15.125:0/2384232818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e380c6f60 0x7f9e381f1a40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f9e2801a0b0 tx=0x7f9e28019220 comp rx=0 tx=0).stop 2023-10-30T12:36:37.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:37.041+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/2384232818 shutdown_connections 2023-10-30T12:36:37.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:37.041+0000 7f9e3ef0f640 1 --2- 172.21.15.125:0/2384232818 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9e240664d0 0x7f9e24068990 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:37.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:37.041+0000 7f9e3ef0f640 1 --2- 172.21.15.125:0/2384232818 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e380c8430 0x7f9e38155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:37.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:37.041+0000 7f9e3ef0f640 1 --2- 172.21.15.125:0/2384232818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e380c6f60 0x7f9e381f1a40 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:37.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:37.041+0000 7f9e3ef0f640 1 --2- 172.21.15.125:0/2384232818 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e380c6b80 0x7f9e3813df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:37.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:37.041+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/2384232818 >> 172.21.15.125:0/2384232818 conn(0x7f9e380c7910 msgr2=0x7f9e3813ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:37.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:37.041+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/2384232818 shutdown_connections 2023-10-30T12:36:37.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:37.041+0000 7f9e3ef0f640 1 -- 172.21.15.125:0/2384232818 wait complete. 2023-10-30T12:36:37.064 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:36:39.065 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:39.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.209+0000 7fa49c9c4640 1 Processor -- start 2023-10-30T12:36:39.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.209+0000 7fa49c9c4640 1 -- start start 2023-10-30T12:36:39.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.213+0000 7fa49c9c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4980c6bd0 0x7fa49806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:39.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.213+0000 7fa49c9c4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa498071110 0x7fa498158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:39.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.213+0000 7fa49c9c4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa498066cf0 con 0x7fa49806f7a0 2023-10-30T12:36:39.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.213+0000 7fa49c9c4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4980700c0 con 0x7fa498071110 2023-10-30T12:36:39.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.213+0000 7fa49c9c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4980c70d0 con 0x7fa4980c6bd0 2023-10-30T12:36:39.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.213+0000 7fa496d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa498071110 0x7fa498158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:39.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.213+0000 7fa48ffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4980c6bd0 0x7fa49806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:39.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.213+0000 7fa49659c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa49806f7a0 0x7fa498069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:41094/0 (socket says 172.21.15.125:41094) 2023-10-30T12:36:39.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.213+0000 7fa49659c640 1 -- 172.21.15.125:0/3222701669 learned_addr learned my addr 172.21.15.125:0/3222701669 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:39.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.213+0000 7fa48ffff640 1 -- 172.21.15.125:0/3222701669 >> v2:172.21.15.136:3300/0 conn(0x7fa498071110 msgr2=0x7fa498158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:39.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.213+0000 7fa48ffff640 1 --2- 172.21.15.125:0/3222701669 >> v2:172.21.15.136:3300/0 conn(0x7fa498071110 0x7fa498158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:39.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.213+0000 7fa48ffff640 1 -- 172.21.15.125:0/3222701669 >> v1:172.21.15.136:6789/0 conn(0x7fa49806f7a0 legacy=0x7fa498069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:39.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.213+0000 7fa48ffff640 1 -- 172.21.15.125:0/3222701669 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4980c7390 con 0x7fa4980c6bd0 2023-10-30T12:36:39.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.213+0000 7fa496d9d640 1 --2- 172.21.15.125:0/3222701669 >> v2:172.21.15.136:3300/0 conn(0x7fa498071110 0x7fa498158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:36:39.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.217+0000 7fa48ffff640 1 --2- 172.21.15.125:0/3222701669 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4980c6bd0 0x7fa49806fb80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa4800097c0 tx=0x7fa48001ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c9404a6612ce75f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:39.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.217+0000 7fa495d9b640 1 -- 172.21.15.125:0/3222701669 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4800a8970 con 0x7fa4980c6bd0 2023-10-30T12:36:39.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.217+0000 7fa495d9b640 1 -- 172.21.15.125:0/3222701669 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4800a8ad0 con 0x7fa4980c6bd0 2023-10-30T12:36:39.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.217+0000 7fa495d9b640 1 -- 172.21.15.125:0/3222701669 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4800a8e60 con 0x7fa4980c6bd0 2023-10-30T12:36:39.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.217+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3222701669 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4980c6bd0 msgr2=0x7fa49806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:39.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.217+0000 7fa49c9c4640 1 --2- 172.21.15.125:0/3222701669 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4980c6bd0 0x7fa49806fb80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa4800097c0 tx=0x7fa48001ec70 comp rx=0 tx=0).stop 2023-10-30T12:36:39.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.217+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3222701669 shutdown_connections 2023-10-30T12:36:39.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.217+0000 7fa49c9c4640 1 --2- 172.21.15.125:0/3222701669 >> v2:172.21.15.136:3300/0 conn(0x7fa498071110 0x7fa498158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:39.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.217+0000 7fa49c9c4640 1 --2- 172.21.15.125:0/3222701669 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4980c6bd0 0x7fa49806fb80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:39.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.217+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3222701669 >> 172.21.15.125:0/3222701669 conn(0x7fa4980c7910 msgr2=0x7fa4980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:39.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.217+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3222701669 shutdown_connections 2023-10-30T12:36:39.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.217+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3222701669 wait complete. 2023-10-30T12:36:39.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.217+0000 7fa49c9c4640 1 Processor -- start 2023-10-30T12:36:39.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.217+0000 7fa49c9c4640 1 -- start start 2023-10-30T12:36:39.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49c9c4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa49806f7a0 0x7fa498155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:39.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49c9c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa498071110 0x7fa4980c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:39.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49c9c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4980c6bd0 0x7fa4981f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:39.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49c9c4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4981565a0 con 0x7fa49806f7a0 2023-10-30T12:36:39.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49c9c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa49814e1d0 con 0x7fa4980c6bd0 2023-10-30T12:36:39.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49c9c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa49814f7b0 con 0x7fa498071110 2023-10-30T12:36:39.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa49806f7a0 0x7fa498155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:39.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa49806f7a0 0x7fa498155af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33632/0 (socket says 172.21.15.125:33632) 2023-10-30T12:36:39.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49659c640 1 -- 172.21.15.125:0/3600042358 learned_addr learned my addr 172.21.15.125:0/3600042358 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:39.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa48ffff640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa498071110 0x7fa4980c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:39.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa496d9d640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4980c6bd0 0x7fa4981f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:39.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49659c640 1 -- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa498071110 msgr2=0x7fa4980c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:39.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49659c640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa498071110 0x7fa4980c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:39.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49659c640 1 -- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4980c6bd0 msgr2=0x7fa4981f2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:39.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49659c640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4980c6bd0 0x7fa4981f2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:39.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49659c640 1 -- 172.21.15.125:0/3600042358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa490009a70 con 0x7fa49806f7a0 2023-10-30T12:36:39.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa48ffff640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa498071110 0x7fa4980c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:36:39.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49659c640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa49806f7a0 0x7fa498155af0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa48800c400 tx=0x7fa48800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:39.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa48effd640 1 -- 172.21.15.125:0/3600042358 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488005cf0 con 0x7fa49806f7a0 2023-10-30T12:36:39.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa48effd640 1 -- 172.21.15.125:0/3600042358 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa488005e50 con 0x7fa49806f7a0 2023-10-30T12:36:39.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3600042358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa480009420 con 0x7fa49806f7a0 2023-10-30T12:36:39.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa48effd640 1 -- 172.21.15.125:0/3600042358 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4880072d0 con 0x7fa49806f7a0 2023-10-30T12:36:39.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3600042358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4981562c0 con 0x7fa49806f7a0 2023-10-30T12:36:39.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3600042358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa49804e170 con 0x7fa49806f7a0 2023-10-30T12:36:39.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.221+0000 7fa48effd640 1 -- 172.21.15.125:0/3600042358 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa488007430 con 0x7fa49806f7a0 2023-10-30T12:36:39.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.225+0000 7fa48effd640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4780664d0 0x7fa478068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:39.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.225+0000 7fa48effd640 1 -- 172.21.15.125:0/3600042358 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6082+0+0 (secure 0 0 0) 0x7fa488083550 con 0x7fa49806f7a0 2023-10-30T12:36:39.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.225+0000 7fa48ffff640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4780664d0 0x7fa478068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:39.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.225+0000 7fa48ffff640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4780664d0 0x7fa478068990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa4800b2000 tx=0x7fa480008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:39.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.229+0000 7fa48effd640 1 -- 172.21.15.125:0/3600042358 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa4880507d0 con 0x7fa49806f7a0 2023-10-30T12:36:39.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.409+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3600042358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4980c8400 con 0x7fa49806f7a0 2023-10-30T12:36:39.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.409+0000 7fa48effd640 1 -- 172.21.15.125:0/3600042358 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5757 (secure 0 0 0) 0x7fa488053df0 con 0x7fa49806f7a0 2023-10-30T12:36:39.412 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:39.412 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:36.055149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:resolve","state_seq":15,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:39.413 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 46 2023-10-30T12:36:39.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.409+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4780664d0 msgr2=0x7fa478068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:39.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.409+0000 7fa49c9c4640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4780664d0 0x7fa478068990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa4800b2000 tx=0x7fa480008040 comp rx=0 tx=0).stop 2023-10-30T12:36:39.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.409+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3600042358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa49806f7a0 msgr2=0x7fa498155af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:39.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.409+0000 7fa49c9c4640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa49806f7a0 0x7fa498155af0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa48800c400 tx=0x7fa48800c8c0 comp rx=0 tx=0).stop 2023-10-30T12:36:39.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.409+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3600042358 shutdown_connections 2023-10-30T12:36:39.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.409+0000 7fa49c9c4640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4780664d0 0x7fa478068990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:39.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.409+0000 7fa49c9c4640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4980c6bd0 0x7fa4981f2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:39.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.409+0000 7fa49c9c4640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa498071110 0x7fa4980c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:39.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.409+0000 7fa49c9c4640 1 --2- 172.21.15.125:0/3600042358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa49806f7a0 0x7fa498155af0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:39.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.409+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3600042358 >> 172.21.15.125:0/3600042358 conn(0x7fa4980c7910 msgr2=0x7fa4980c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:39.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.409+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3600042358 shutdown_connections 2023-10-30T12:36:39.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:39.409+0000 7fa49c9c4640 1 -- 172.21.15.125:0/3600042358 wait complete. 2023-10-30T12:36:39.433 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:36:41.434 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:41.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.577+0000 7fec09b3e640 1 Processor -- start 2023-10-30T12:36:41.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.577+0000 7fec09b3e640 1 -- start start 2023-10-30T12:36:41.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec09b3e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fec04153b10 0x7fec0415ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:41.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec09b3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec040c5e70 0x7fec041641f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:41.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec09b3e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec040c74f0 con 0x7fec040b0dc0 2023-10-30T12:36:41.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec09b3e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec040c1f20 con 0x7fec04153b10 2023-10-30T12:36:41.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec09b3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec040c77b0 con 0x7fec040c5e70 2023-10-30T12:36:41.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec02ffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fec040b0dc0 0x7fec040c7e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45232/0 (socket says 172.21.15.125:45232) 2023-10-30T12:36:41.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec02ffd640 1 -- 172.21.15.125:0/706180459 learned_addr learned my addr 172.21.15.125:0/706180459 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:41.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec037fe640 1 --2- 172.21.15.125:0/706180459 >> v2:172.21.15.136:3300/0 conn(0x7fec04153b10 0x7fec0415ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:41.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec03fff640 1 --2- 172.21.15.125:0/706180459 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec040c5e70 0x7fec041641f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:41.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec027fc640 1 -- 172.21.15.125:0/706180459 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811598589 0 0) 0x7fec040c74f0 con 0x7fec040b0dc0 2023-10-30T12:36:41.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec027fc640 1 -- 172.21.15.125:0/706180459 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febec003530 con 0x7fec040b0dc0 2023-10-30T12:36:41.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec027fc640 1 -- 172.21.15.125:0/706180459 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 215718996 0 0) 0x7febec003530 con 0x7fec040b0dc0 2023-10-30T12:36:41.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec037fe640 1 -- 172.21.15.125:0/706180459 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec040c5e70 msgr2=0x7fec041641f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:41.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec037fe640 1 --2- 172.21.15.125:0/706180459 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec040c5e70 0x7fec041641f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:41.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec037fe640 1 -- 172.21.15.125:0/706180459 >> v1:172.21.15.125:6789/0 conn(0x7fec040b0dc0 legacy=0x7fec040c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:41.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.581+0000 7fec037fe640 1 -- 172.21.15.125:0/706180459 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec04159b80 con 0x7fec04153b10 2023-10-30T12:36:41.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.585+0000 7fec037fe640 1 --2- 172.21.15.125:0/706180459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec04153b10 0x7fec0415ce60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7febf40097c0 tx=0x7febf401eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cd7043b7d536396 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:41.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.585+0000 7fec027fc640 1 -- 172.21.15.125:0/706180459 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8001680 con 0x7fec04153b10 2023-10-30T12:36:41.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.585+0000 7fec027fc640 1 -- 172.21.15.125:0/706180459 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf40a85d0 con 0x7fec04153b10 2023-10-30T12:36:41.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.585+0000 7fec027fc640 1 -- 172.21.15.125:0/706180459 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf40a8960 con 0x7fec04153b10 2023-10-30T12:36:41.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.585+0000 7fec09b3e640 1 -- 172.21.15.125:0/706180459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec04153b10 msgr2=0x7fec0415ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:41.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.585+0000 7fec09b3e640 1 --2- 172.21.15.125:0/706180459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec04153b10 0x7fec0415ce60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7febf40097c0 tx=0x7febf401eb90 comp rx=0 tx=0).stop 2023-10-30T12:36:41.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.585+0000 7fec09b3e640 1 -- 172.21.15.125:0/706180459 shutdown_connections 2023-10-30T12:36:41.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.585+0000 7fec09b3e640 1 --2- 172.21.15.125:0/706180459 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec040c5e70 0x7fec041641f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:41.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.585+0000 7fec09b3e640 1 --2- 172.21.15.125:0/706180459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec04153b10 0x7fec0415ce60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:41.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.585+0000 7fec09b3e640 1 -- 172.21.15.125:0/706180459 >> 172.21.15.125:0/706180459 conn(0x7fec0406d770 msgr2=0x7fec0406db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:41.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.585+0000 7fec09b3e640 1 -- 172.21.15.125:0/706180459 shutdown_connections 2023-10-30T12:36:41.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.585+0000 7fec09b3e640 1 -- 172.21.15.125:0/706180459 wait complete. 2023-10-30T12:36:41.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.585+0000 7fec09b3e640 1 Processor -- start 2023-10-30T12:36:41.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.585+0000 7fec09b3e640 1 -- start start 2023-10-30T12:36:41.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec09b3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec040b0dc0 0x7fec0413fed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:41.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec09b3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec040c5e70 0x7fec041478c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:41.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec09b3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec04153b10 0x7fec04147e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:41.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec09b3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec041f2c30 con 0x7fec04153b10 2023-10-30T12:36:41.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec09b3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec04140410 con 0x7fec040b0dc0 2023-10-30T12:36:41.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec09b3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec040c9590 con 0x7fec040c5e70 2023-10-30T12:36:41.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec03fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec04153b10 0x7fec04147e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:41.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec03fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec04153b10 0x7fec04147e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40364/0 (socket says 172.21.15.125:40364) 2023-10-30T12:36:41.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec03fff640 1 -- 172.21.15.125:0/4163697874 learned_addr learned my addr 172.21.15.125:0/4163697874 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:41.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec037fe640 1 --2- 172.21.15.125:0/4163697874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec040b0dc0 0x7fec0413fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:41.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec02ffd640 1 --2- 172.21.15.125:0/4163697874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec040c5e70 0x7fec041478c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:41.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec03fff640 1 -- 172.21.15.125:0/4163697874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec040c5e70 msgr2=0x7fec041478c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:41.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec03fff640 1 --2- 172.21.15.125:0/4163697874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec040c5e70 0x7fec041478c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:41.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec03fff640 1 -- 172.21.15.125:0/4163697874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec040b0dc0 msgr2=0x7fec0413fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:41.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec03fff640 1 --2- 172.21.15.125:0/4163697874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec040b0dc0 0x7fec0413fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:41.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec03fff640 1 -- 172.21.15.125:0/4163697874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf4009420 con 0x7fec04153b10 2023-10-30T12:36:41.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec03fff640 1 --2- 172.21.15.125:0/4163697874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec04153b10 0x7fec04147e00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7febfc009dd0 tx=0x7febfc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:41.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7febebfff640 1 -- 172.21.15.125:0/4163697874 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc00ced0 con 0x7fec04153b10 2023-10-30T12:36:41.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7febebfff640 1 -- 172.21.15.125:0/4163697874 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febfc00f040 con 0x7fec04153b10 2023-10-30T12:36:41.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7febebfff640 1 -- 172.21.15.125:0/4163697874 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc0146a0 con 0x7fec04153b10 2023-10-30T12:36:41.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec09b3e640 1 -- 172.21.15.125:0/4163697874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec040cbe50 con 0x7fec04153b10 2023-10-30T12:36:41.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec09b3e640 1 -- 172.21.15.125:0/4163697874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec0415ae40 con 0x7fec04153b10 2023-10-30T12:36:41.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.589+0000 7fec09b3e640 1 -- 172.21.15.125:0/4163697874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec04074250 con 0x7fec04153b10 2023-10-30T12:36:41.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.593+0000 7febebfff640 1 -- 172.21.15.125:0/4163697874 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7febfc014800 con 0x7fec04153b10 2023-10-30T12:36:41.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.593+0000 7febebfff640 1 --2- 172.21.15.125:0/4163697874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febf00664d0 0x7febf0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:41.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.593+0000 7febebfff640 1 -- 172.21.15.125:0/4163697874 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6082+0+0 (secure 0 0 0) 0x7febfc084c80 con 0x7fec04153b10 2023-10-30T12:36:41.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.593+0000 7fec037fe640 1 --2- 172.21.15.125:0/4163697874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febf00664d0 0x7febf0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:41.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.593+0000 7fec037fe640 1 --2- 172.21.15.125:0/4163697874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febf00664d0 0x7febf0068990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7febf40b2000 tx=0x7febf4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:41.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.597+0000 7febebfff640 1 -- 172.21.15.125:0/4163697874 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7febfc0520c0 con 0x7fec04153b10 2023-10-30T12:36:41.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.777+0000 7fec09b3e640 1 -- 172.21.15.125:0/4163697874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec041f2d70 con 0x7fec04153b10 2023-10-30T12:36:41.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.777+0000 7febebfff640 1 -- 172.21.15.125:0/4163697874 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+5757 (secure 0 0 0) 0x7fec041f2d70 con 0x7fec04153b10 2023-10-30T12:36:41.781 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:41.781 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:36.055149+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:resolve","state_seq":15,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:41.782 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 46 2023-10-30T12:36:41.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.781+0000 7fec09b3e640 1 -- 172.21.15.125:0/4163697874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febf00664d0 msgr2=0x7febf0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:41.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.781+0000 7fec09b3e640 1 --2- 172.21.15.125:0/4163697874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febf00664d0 0x7febf0068990 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7febf40b2000 tx=0x7febf4008040 comp rx=0 tx=0).stop 2023-10-30T12:36:41.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.781+0000 7fec09b3e640 1 -- 172.21.15.125:0/4163697874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec04153b10 msgr2=0x7fec04147e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:41.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.781+0000 7fec09b3e640 1 --2- 172.21.15.125:0/4163697874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec04153b10 0x7fec04147e00 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7febfc009dd0 tx=0x7febfc00c490 comp rx=0 tx=0).stop 2023-10-30T12:36:41.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.781+0000 7fec09b3e640 1 -- 172.21.15.125:0/4163697874 shutdown_connections 2023-10-30T12:36:41.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.781+0000 7fec09b3e640 1 --2- 172.21.15.125:0/4163697874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febf00664d0 0x7febf0068990 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:41.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.781+0000 7fec09b3e640 1 --2- 172.21.15.125:0/4163697874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec04153b10 0x7fec04147e00 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:41.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.781+0000 7fec09b3e640 1 --2- 172.21.15.125:0/4163697874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec040c5e70 0x7fec041478c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.781+0000 7fec09b3e640 1 --2- 172.21.15.125:0/4163697874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec040b0dc0 0x7fec0413fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.781+0000 7fec09b3e640 1 -- 172.21.15.125:0/4163697874 >> 172.21.15.125:0/4163697874 conn(0x7fec0406d770 msgr2=0x7fec04148990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.781+0000 7fec09b3e640 1 -- 172.21.15.125:0/4163697874 shutdown_connections 2023-10-30T12:36:41.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:41.781+0000 7fec09b3e640 1 -- 172.21.15.125:0/4163697874 wait complete. 2023-10-30T12:36:41.802 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:36:43.803 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:43.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.945+0000 7ffae049c640 1 Processor -- start 2023-10-30T12:36:43.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.945+0000 7ffae049c640 1 -- start start 2023-10-30T12:36:43.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffae049c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffad80c6bd0 0x7ffad806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:43.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffae049c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ffad8071110 0x7ffad8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:43.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffae049c640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad8066cf0 con 0x7ffad806f7a0 2023-10-30T12:36:43.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffae049c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad80700c0 con 0x7ffad80c6bd0 2023-10-30T12:36:43.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffae049c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffad80c70d0 con 0x7ffad8071110 2023-10-30T12:36:43.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffade238640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ffad806f7a0 0x7ffad8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45248/0 (socket says 172.21.15.125:45248) 2023-10-30T12:36:43.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffade238640 1 -- 172.21.15.125:0/3970460193 learned_addr learned my addr 172.21.15.125:0/3970460193 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:43.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffadda37640 1 --2- 172.21.15.125:0/3970460193 >> v2:172.21.15.125:3300/0 conn(0x7ffad80c6bd0 0x7ffad806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:43.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffadea39640 1 --2- 172.21.15.125:0/3970460193 >> v2:172.21.15.136:3300/0 conn(0x7ffad8071110 0x7ffad8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:43.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffadd236640 1 -- 172.21.15.125:0/3970460193 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292461569 0 0) 0x7ffad8066cf0 con 0x7ffad806f7a0 2023-10-30T12:36:43.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffadd236640 1 -- 172.21.15.125:0/3970460193 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac4003530 con 0x7ffad806f7a0 2023-10-30T12:36:43.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffadda37640 1 -- 172.21.15.125:0/3970460193 >> v2:172.21.15.136:3300/0 conn(0x7ffad8071110 msgr2=0x7ffad8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:43.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffadda37640 1 --2- 172.21.15.125:0/3970460193 >> v2:172.21.15.136:3300/0 conn(0x7ffad8071110 0x7ffad8158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:43.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffadda37640 1 -- 172.21.15.125:0/3970460193 >> v1:172.21.15.125:6789/0 conn(0x7ffad806f7a0 legacy=0x7ffad8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:43.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffadda37640 1 -- 172.21.15.125:0/3970460193 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad80c7390 con 0x7ffad80c6bd0 2023-10-30T12:36:43.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.949+0000 7ffadea39640 1 --2- 172.21.15.125:0/3970460193 >> v2:172.21.15.136:3300/0 conn(0x7ffad8071110 0x7ffad8158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:36:43.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.953+0000 7ffadda37640 1 --2- 172.21.15.125:0/3970460193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffad80c6bd0 0x7ffad806fb80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ffad806cdd0 tx=0x7ffac801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183ee19fafa272c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:43.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.953+0000 7ffadd236640 1 -- 172.21.15.125:0/3970460193 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac80a8950 con 0x7ffad80c6bd0 2023-10-30T12:36:43.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.953+0000 7ffadd236640 1 -- 172.21.15.125:0/3970460193 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac80a8ab0 con 0x7ffad80c6bd0 2023-10-30T12:36:43.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.953+0000 7ffadd236640 1 -- 172.21.15.125:0/3970460193 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac80a8e40 con 0x7ffad80c6bd0 2023-10-30T12:36:43.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.953+0000 7ffae049c640 1 -- 172.21.15.125:0/3970460193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffad80c6bd0 msgr2=0x7ffad806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:43.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.953+0000 7ffae049c640 1 --2- 172.21.15.125:0/3970460193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffad80c6bd0 0x7ffad806fb80 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7ffad806cdd0 tx=0x7ffac801ec50 comp rx=0 tx=0).stop 2023-10-30T12:36:43.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.953+0000 7ffae049c640 1 -- 172.21.15.125:0/3970460193 shutdown_connections 2023-10-30T12:36:43.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.953+0000 7ffae049c640 1 --2- 172.21.15.125:0/3970460193 >> v2:172.21.15.136:3300/0 conn(0x7ffad8071110 0x7ffad8158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:43.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.953+0000 7ffae049c640 1 --2- 172.21.15.125:0/3970460193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffad80c6bd0 0x7ffad806fb80 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:43.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.953+0000 7ffae049c640 1 -- 172.21.15.125:0/3970460193 >> 172.21.15.125:0/3970460193 conn(0x7ffad80c7910 msgr2=0x7ffad80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:43.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.953+0000 7ffae049c640 1 -- 172.21.15.125:0/3970460193 shutdown_connections 2023-10-30T12:36:43.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.953+0000 7ffae049c640 1 -- 172.21.15.125:0/3970460193 wait complete. 2023-10-30T12:36:43.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.953+0000 7ffae049c640 1 Processor -- start 2023-10-30T12:36:43.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.953+0000 7ffae049c640 1 -- start start 2023-10-30T12:36:43.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffae049c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffad806f7a0 0x7ffad81f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:43.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffae049c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad8071110 0x7ffad814ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:43.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffae049c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffad80c6bd0 0x7ffad814f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:43.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffae049c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad81541a0 con 0x7ffad806f7a0 2023-10-30T12:36:43.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffae049c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffad814fd80 con 0x7ffad8071110 2023-10-30T12:36:43.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffae049c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffad81f3170 con 0x7ffad80c6bd0 2023-10-30T12:36:43.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffade238640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffad806f7a0 0x7ffad81f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:43.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffade238640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffad806f7a0 0x7ffad81f24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40390/0 (socket says 172.21.15.125:40390) 2023-10-30T12:36:43.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffade238640 1 -- 172.21.15.125:0/2421293923 learned_addr learned my addr 172.21.15.125:0/2421293923 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:43.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffadda37640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad8071110 0x7ffad814ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:43.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffadea39640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffad80c6bd0 0x7ffad814f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:43.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffade238640 1 -- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffad80c6bd0 msgr2=0x7ffad814f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:43.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffade238640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffad80c6bd0 0x7ffad814f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:43.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffade238640 1 -- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad8071110 msgr2=0x7ffad814ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:43.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffade238640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad8071110 0x7ffad814ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:43.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffade238640 1 -- 172.21.15.125:0/2421293923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac8009420 con 0x7ffad806f7a0 2023-10-30T12:36:43.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffadea39640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffad80c6bd0 0x7ffad814f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:36:43.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffade238640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffad806f7a0 0x7ffad81f24c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ffad400c420 tx=0x7ffad400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:43.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffaceffd640 1 -- 172.21.15.125:0/2421293923 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad4005cf0 con 0x7ffad806f7a0 2023-10-30T12:36:43.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffaceffd640 1 -- 172.21.15.125:0/2421293923 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffad4005e50 con 0x7ffad806f7a0 2023-10-30T12:36:43.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffaceffd640 1 -- 172.21.15.125:0/2421293923 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffad40072d0 con 0x7ffad806f7a0 2023-10-30T12:36:43.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffae049c640 1 -- 172.21.15.125:0/2421293923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad80c5d80 con 0x7ffad806f7a0 2023-10-30T12:36:43.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffae049c640 1 -- 172.21.15.125:0/2421293923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad80c5ac0 con 0x7ffad806f7a0 2023-10-30T12:36:43.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffae049c640 1 -- 172.21.15.125:0/2421293923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad804e170 con 0x7ffad806f7a0 2023-10-30T12:36:43.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.957+0000 7ffaceffd640 1 -- 172.21.15.125:0/2421293923 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffad4007430 con 0x7ffad806f7a0 2023-10-30T12:36:43.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.961+0000 7ffaceffd640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffabc0664a0 0x7ffabc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:43.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.961+0000 7ffaceffd640 1 -- 172.21.15.125:0/2421293923 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6082+0+0 (secure 0 0 0) 0x7ffad4083510 con 0x7ffad806f7a0 2023-10-30T12:36:43.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.961+0000 7ffadda37640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffabc0664a0 0x7ffabc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:43.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.961+0000 7ffadda37640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffabc0664a0 0x7ffabc068960 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffac80b2000 tx=0x7ffac80aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:43.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:43.965+0000 7ffaceffd640 1 -- 172.21.15.125:0/2421293923 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffad40508c0 con 0x7ffad806f7a0 2023-10-30T12:36:44.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:44.149+0000 7ffae049c640 1 -- 172.21.15.125:0/2421293923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffad80b1230 con 0x7ffad806f7a0 2023-10-30T12:36:44.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:44.149+0000 7ffaceffd640 1 -- 172.21.15.125:0/2421293923 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+5759 (secure 0 0 0) 0x7ffad4053ee0 con 0x7ffad806f7a0 2023-10-30T12:36:44.152 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:44.152 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:41.487721+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:44.153 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 47 2023-10-30T12:36:44.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:44.149+0000 7ffae049c640 1 -- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffabc0664a0 msgr2=0x7ffabc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:44.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:44.149+0000 7ffae049c640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffabc0664a0 0x7ffabc068960 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffac80b2000 tx=0x7ffac80aa070 comp rx=0 tx=0).stop 2023-10-30T12:36:44.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:44.149+0000 7ffae049c640 1 -- 172.21.15.125:0/2421293923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffad806f7a0 msgr2=0x7ffad81f24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:44.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:44.149+0000 7ffae049c640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffad806f7a0 0x7ffad81f24c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ffad400c420 tx=0x7ffad400c8e0 comp rx=0 tx=0).stop 2023-10-30T12:36:44.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:44.149+0000 7ffae049c640 1 -- 172.21.15.125:0/2421293923 shutdown_connections 2023-10-30T12:36:44.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:44.149+0000 7ffae049c640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffabc0664a0 0x7ffabc068960 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:44.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:44.149+0000 7ffae049c640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffad80c6bd0 0x7ffad814f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:44.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:44.149+0000 7ffae049c640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffad8071110 0x7ffad814ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:44.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:44.149+0000 7ffae049c640 1 --2- 172.21.15.125:0/2421293923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffad806f7a0 0x7ffad81f24c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:44.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:44.149+0000 7ffae049c640 1 -- 172.21.15.125:0/2421293923 >> 172.21.15.125:0/2421293923 conn(0x7ffad80c7910 msgr2=0x7ffad80c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:44.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:44.149+0000 7ffae049c640 1 -- 172.21.15.125:0/2421293923 shutdown_connections 2023-10-30T12:36:44.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:44.149+0000 7ffae049c640 1 -- 172.21.15.125:0/2421293923 wait complete. 2023-10-30T12:36:44.173 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 47, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 47, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4957": { "addr": "172.21.15.136:6835/2548530941", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2548530941, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2548530941, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4957, "incarnation": 23, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 34 }, "gid_4973": { "addr": "172.21.15.136:6839/1454326577", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 1454326577, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 1454326577, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4973, "incarnation": 45, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:reconnect", "state_seq": 17 }, "gid_5130": { "addr": "172.21.15.125:6837/1624923396", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 1624923396, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 1624923396, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5130, "incarnation": 37, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:36:41.487721+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4973, "mds_1": 5130, "mds_2": 4957 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:36:44.888 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:36:44.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:36:46.174 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:46.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.317+0000 7f2654217640 1 Processor -- start 2023-10-30T12:36:46.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.317+0000 7f2654217640 1 -- start start 2023-10-30T12:36:46.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.321+0000 7f2654217640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f264c0c05d0 0x7f264c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:46.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.321+0000 7f2654217640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f264c0c6f60 0x7f264c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:46.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.321+0000 7f2654217640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264c0c0a50 con 0x7f264c0c6b80 2023-10-30T12:36:46.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.321+0000 7f2654217640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f264c06f6e0 con 0x7f264c0c6f60 2023-10-30T12:36:46.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.321+0000 7f2654217640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f264c06f9c0 con 0x7f264c0c05d0 2023-10-30T12:36:46.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.321+0000 7f26527b4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f264c0c6f60 0x7f264c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:46.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.321+0000 7f26527b4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f264c0c6f60 0x7f264c13f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40396/0 (socket says 172.21.15.125:40396) 2023-10-30T12:36:46.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.321+0000 7f26527b4640 1 -- 172.21.15.125:0/490991320 learned_addr learned my addr 172.21.15.125:0/490991320 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:46.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.321+0000 7f26517b2640 1 --2- 172.21.15.125:0/490991320 >> v2:172.21.15.136:3300/0 conn(0x7f264c0c05d0 0x7f264c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:46.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.321+0000 7f26527b4640 1 -- 172.21.15.125:0/490991320 >> v2:172.21.15.136:3300/0 conn(0x7f264c0c05d0 msgr2=0x7f264c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:46.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.321+0000 7f26527b4640 1 --2- 172.21.15.125:0/490991320 >> v2:172.21.15.136:3300/0 conn(0x7f264c0c05d0 0x7f264c13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:46.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.321+0000 7f26527b4640 1 -- 172.21.15.125:0/490991320 >> v1:172.21.15.136:6789/0 conn(0x7f264c0c6b80 legacy=0x7f264c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:46.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.321+0000 7f26527b4640 1 -- 172.21.15.125:0/490991320 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264c06fca0 con 0x7f264c0c6f60 2023-10-30T12:36:46.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.325+0000 7f26527b4640 1 --2- 172.21.15.125:0/490991320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f264c0c6f60 0x7f264c13f2d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f264c06c7e0 tx=0x7f264401ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5645783e7cbeaf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:46.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.325+0000 7f2650fb1640 1 -- 172.21.15.125:0/490991320 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264400b2d0 con 0x7f264c0c6f60 2023-10-30T12:36:46.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.325+0000 7f2650fb1640 1 -- 172.21.15.125:0/490991320 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264400b430 con 0x7f264c0c6f60 2023-10-30T12:36:46.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.325+0000 7f2650fb1640 1 -- 172.21.15.125:0/490991320 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26440a7bd0 con 0x7f264c0c6f60 2023-10-30T12:36:46.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.325+0000 7f2654217640 1 -- 172.21.15.125:0/490991320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f264c0c6f60 msgr2=0x7f264c13f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:46.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.325+0000 7f2654217640 1 --2- 172.21.15.125:0/490991320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f264c0c6f60 0x7f264c13f2d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f264c06c7e0 tx=0x7f264401ebd0 comp rx=0 tx=0).stop 2023-10-30T12:36:46.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.325+0000 7f2654217640 1 -- 172.21.15.125:0/490991320 shutdown_connections 2023-10-30T12:36:46.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.325+0000 7f2654217640 1 --2- 172.21.15.125:0/490991320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f264c0c6f60 0x7f264c13f2d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:46.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.325+0000 7f2654217640 1 --2- 172.21.15.125:0/490991320 >> v2:172.21.15.136:3300/0 conn(0x7f264c0c05d0 0x7f264c13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:46.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.325+0000 7f2654217640 1 -- 172.21.15.125:0/490991320 >> 172.21.15.125:0/490991320 conn(0x7f264c0c7910 msgr2=0x7f264c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:46.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.325+0000 7f2654217640 1 -- 172.21.15.125:0/490991320 shutdown_connections 2023-10-30T12:36:46.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.325+0000 7f2654217640 1 -- 172.21.15.125:0/490991320 wait complete. 2023-10-30T12:36:46.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.325+0000 7f2654217640 1 Processor -- start 2023-10-30T12:36:46.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.325+0000 7f2654217640 1 -- start start 2023-10-30T12:36:46.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f2654217640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f264c0c05d0 0x7f264c1607e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:46.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f2654217640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f264c0c6b80 0x7f264c14ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:46.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f2654217640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f264c0c6f60 0x7f264c12fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:46.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f2654217640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264c14f910 con 0x7f264c0c6b80 2023-10-30T12:36:46.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f2654217640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264c14c540 con 0x7f264c0c05d0 2023-10-30T12:36:46.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f2654217640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264c160d20 con 0x7f264c0c6f60 2023-10-30T12:36:46.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f26517b2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f264c0c6b80 0x7f264c14ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:46.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f26517b2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f264c0c6b80 0x7f264c14ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40410/0 (socket says 172.21.15.125:40410) 2023-10-30T12:36:46.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f26517b2640 1 -- 172.21.15.125:0/762032892 learned_addr learned my addr 172.21.15.125:0/762032892 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:46.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f26527b4640 1 --2- 172.21.15.125:0/762032892 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f264c0c6f60 0x7f264c12fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:46.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f2651fb3640 1 --2- 172.21.15.125:0/762032892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f264c0c05d0 0x7f264c1607e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:46.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f26517b2640 1 -- 172.21.15.125:0/762032892 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f264c0c6f60 msgr2=0x7f264c12fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:46.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f26517b2640 1 --2- 172.21.15.125:0/762032892 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f264c0c6f60 0x7f264c12fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:46.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f26517b2640 1 -- 172.21.15.125:0/762032892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f264c0c05d0 msgr2=0x7f264c1607e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:46.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f26517b2640 1 --2- 172.21.15.125:0/762032892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f264c0c05d0 0x7f264c1607e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:46.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f26517b2640 1 -- 172.21.15.125:0/762032892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264400bd40 con 0x7f264c0c6b80 2023-10-30T12:36:46.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f26517b2640 1 --2- 172.21.15.125:0/762032892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f264c0c6b80 0x7f264c14ca80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f263c00d830 tx=0x7f263c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:46.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f26427fc640 1 -- 172.21.15.125:0/762032892 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c00b660 con 0x7f264c0c6b80 2023-10-30T12:36:46.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f26427fc640 1 -- 172.21.15.125:0/762032892 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263c00b7e0 con 0x7f264c0c6b80 2023-10-30T12:36:46.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f26427fc640 1 -- 172.21.15.125:0/762032892 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c00bb90 con 0x7f264c0c6b80 2023-10-30T12:36:46.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f2654217640 1 -- 172.21.15.125:0/762032892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f264c1307d0 con 0x7f264c0c6b80 2023-10-30T12:36:46.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f2654217640 1 -- 172.21.15.125:0/762032892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264c130d90 con 0x7f264c0c6b80 2023-10-30T12:36:46.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f2654217640 1 -- 172.21.15.125:0/762032892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f264c04e170 con 0x7f264c0c6b80 2023-10-30T12:36:46.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.329+0000 7f26427fc640 1 -- 172.21.15.125:0/762032892 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f263c01e460 con 0x7f264c0c6b80 2023-10-30T12:36:46.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.333+0000 7f26427fc640 1 --2- 172.21.15.125:0/762032892 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26300664d0 0x7f2630068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:46.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.333+0000 7f26427fc640 1 -- 172.21.15.125:0/762032892 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f263c0847a0 con 0x7f264c0c6b80 2023-10-30T12:36:46.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.333+0000 7f2651fb3640 1 --2- 172.21.15.125:0/762032892 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26300664d0 0x7f2630068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:46.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.333+0000 7f2651fb3640 1 --2- 172.21.15.125:0/762032892 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26300664d0 0x7f2630068990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2648000f70 tx=0x7f2648009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:46.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.337+0000 7f26427fc640 1 -- 172.21.15.125:0/762032892 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f263c051a10 con 0x7f264c0c6b80 2023-10-30T12:36:46.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.517+0000 7f2654217640 1 -- 172.21.15.125:0/762032892 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f264c14afe0 con 0x7f264c0c6b80 2023-10-30T12:36:46.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.517+0000 7f26427fc640 1 -- 172.21.15.125:0/762032892 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+5759 (secure 0 0 0) 0x7f263c055030 con 0x7f264c0c6b80 2023-10-30T12:36:46.522 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:46.522 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:41.487721+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:reconnect","state_seq":17,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:46.522 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 47 2023-10-30T12:36:46.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.521+0000 7f2654217640 1 -- 172.21.15.125:0/762032892 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26300664d0 msgr2=0x7f2630068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:46.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.521+0000 7f2654217640 1 --2- 172.21.15.125:0/762032892 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26300664d0 0x7f2630068990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f2648000f70 tx=0x7f2648009040 comp rx=0 tx=0).stop 2023-10-30T12:36:46.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.521+0000 7f2654217640 1 -- 172.21.15.125:0/762032892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f264c0c6b80 msgr2=0x7f264c14ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:46.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.521+0000 7f2654217640 1 --2- 172.21.15.125:0/762032892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f264c0c6b80 0x7f264c14ca80 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f263c00d830 tx=0x7f263c00dcf0 comp rx=0 tx=0).stop 2023-10-30T12:36:46.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.521+0000 7f2654217640 1 -- 172.21.15.125:0/762032892 shutdown_connections 2023-10-30T12:36:46.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.521+0000 7f2654217640 1 --2- 172.21.15.125:0/762032892 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26300664d0 0x7f2630068990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:46.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.521+0000 7f2654217640 1 --2- 172.21.15.125:0/762032892 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f264c0c6f60 0x7f264c12fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:46.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.521+0000 7f2654217640 1 --2- 172.21.15.125:0/762032892 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f264c0c6b80 0x7f264c14ca80 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:46.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.521+0000 7f2654217640 1 --2- 172.21.15.125:0/762032892 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f264c0c05d0 0x7f264c1607e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:46.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.521+0000 7f2654217640 1 -- 172.21.15.125:0/762032892 >> 172.21.15.125:0/762032892 conn(0x7f264c0c7910 msgr2=0x7f264c0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:46.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.521+0000 7f2654217640 1 -- 172.21.15.125:0/762032892 shutdown_connections 2023-10-30T12:36:46.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:46.521+0000 7f2654217640 1 -- 172.21.15.125:0/762032892 wait complete. 2023-10-30T12:36:46.543 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:36:48.544 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:48.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.681+0000 7f3d6f301640 1 Processor -- start 2023-10-30T12:36:48.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.681+0000 7f3d6f301640 1 -- start start 2023-10-30T12:36:48.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d6f301640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3d680c05d0 0x7f3d6813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:48.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d6f301640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3d680c6f60 0x7f3d6813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:48.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d6f301640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d680c0a50 con 0x7f3d680c6b80 2023-10-30T12:36:48.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d6f301640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d6806f6e0 con 0x7f3d680c05d0 2023-10-30T12:36:48.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d6f301640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d6806f9c0 con 0x7f3d680c6f60 2023-10-30T12:36:48.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d6c89c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3d680c05d0 0x7f3d6813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:48.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d6c89c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3d680c05d0 0x7f3d6813ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40422/0 (socket says 172.21.15.125:40422) 2023-10-30T12:36:48.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d6c89c640 1 -- 172.21.15.125:0/3403211003 learned_addr learned my addr 172.21.15.125:0/3403211003 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:48.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d6d89e640 1 --2- 172.21.15.125:0/3403211003 >> v2:172.21.15.136:3300/0 conn(0x7f3d680c6f60 0x7f3d6813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:48.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d6c89c640 1 -- 172.21.15.125:0/3403211003 >> v2:172.21.15.136:3300/0 conn(0x7f3d680c6f60 msgr2=0x7f3d6813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:48.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d6c89c640 1 --2- 172.21.15.125:0/3403211003 >> v2:172.21.15.136:3300/0 conn(0x7f3d680c6f60 0x7f3d6813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:48.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d6c89c640 1 -- 172.21.15.125:0/3403211003 >> v1:172.21.15.125:6789/0 conn(0x7f3d680c6b80 legacy=0x7f3d680bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:48.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d6c89c640 1 -- 172.21.15.125:0/3403211003 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6806fca0 con 0x7f3d680c05d0 2023-10-30T12:36:48.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d6d89e640 1 --2- 172.21.15.125:0/3403211003 >> v2:172.21.15.136:3300/0 conn(0x7f3d680c6f60 0x7f3d6813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:36:48.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.685+0000 7f3d5ffff640 1 -- 172.21.15.125:0/3403211003 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166928873 0 0) 0x7f3d680c0a50 con 0x7f3d680c6b80 2023-10-30T12:36:48.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.689+0000 7f3d6c89c640 1 --2- 172.21.15.125:0/3403211003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d680c05d0 0x7f3d6813ed90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3d680c89b0 tx=0x7f3d5801e710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191f40d24137605 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:48.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.689+0000 7f3d5ffff640 1 -- 172.21.15.125:0/3403211003 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d580aa650 con 0x7f3d680c05d0 2023-10-30T12:36:48.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.689+0000 7f3d5ffff640 1 -- 172.21.15.125:0/3403211003 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d580aa7b0 con 0x7f3d680c05d0 2023-10-30T12:36:48.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.689+0000 7f3d5ffff640 1 -- 172.21.15.125:0/3403211003 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d580aab40 con 0x7f3d680c05d0 2023-10-30T12:36:48.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.689+0000 7f3d6f301640 1 -- 172.21.15.125:0/3403211003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d680c05d0 msgr2=0x7f3d6813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:48.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.689+0000 7f3d6f301640 1 --2- 172.21.15.125:0/3403211003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d680c05d0 0x7f3d6813ed90 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f3d680c89b0 tx=0x7f3d5801e710 comp rx=0 tx=0).stop 2023-10-30T12:36:48.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.689+0000 7f3d6f301640 1 -- 172.21.15.125:0/3403211003 shutdown_connections 2023-10-30T12:36:48.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.689+0000 7f3d6f301640 1 --2- 172.21.15.125:0/3403211003 >> v2:172.21.15.136:3300/0 conn(0x7f3d680c6f60 0x7f3d6813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:48.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.689+0000 7f3d6f301640 1 --2- 172.21.15.125:0/3403211003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d680c05d0 0x7f3d6813ed90 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:48.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.689+0000 7f3d6f301640 1 -- 172.21.15.125:0/3403211003 >> 172.21.15.125:0/3403211003 conn(0x7f3d680c7910 msgr2=0x7f3d680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:48.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.689+0000 7f3d6f301640 1 -- 172.21.15.125:0/3403211003 shutdown_connections 2023-10-30T12:36:48.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.689+0000 7f3d6f301640 1 -- 172.21.15.125:0/3403211003 wait complete. 2023-10-30T12:36:48.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.689+0000 7f3d6f301640 1 Processor -- start 2023-10-30T12:36:48.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.689+0000 7f3d6f301640 1 -- start start 2023-10-30T12:36:48.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6f301640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d680c05d0 0x7f3d6806a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:48.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6f301640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d680c6b80 0x7f3d68154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:48.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6f301640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d680c6f60 0x7f3d681550a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:48.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6f301640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d680c9d10 con 0x7f3d680c6f60 2023-10-30T12:36:48.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6f301640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d6813d730 con 0x7f3d680c6b80 2023-10-30T12:36:48.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6f301640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d681546d0 con 0x7f3d680c05d0 2023-10-30T12:36:48.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6d89e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d680c6f60 0x7f3d681550a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:48.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6d89e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d680c6f60 0x7f3d681550a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40430/0 (socket says 172.21.15.125:40430) 2023-10-30T12:36:48.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6d89e640 1 -- 172.21.15.125:0/852714008 learned_addr learned my addr 172.21.15.125:0/852714008 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:48.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6d09d640 1 --2- 172.21.15.125:0/852714008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d680c05d0 0x7f3d6806a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:48.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6c89c640 1 --2- 172.21.15.125:0/852714008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d680c6b80 0x7f3d68154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:48.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6d89e640 1 -- 172.21.15.125:0/852714008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d680c05d0 msgr2=0x7f3d6806a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:48.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6d89e640 1 --2- 172.21.15.125:0/852714008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d680c05d0 0x7f3d6806a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:48.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6d89e640 1 -- 172.21.15.125:0/852714008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d680c6b80 msgr2=0x7f3d68154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:48.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6d89e640 1 --2- 172.21.15.125:0/852714008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d680c6b80 0x7f3d68154b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:48.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6d89e640 1 -- 172.21.15.125:0/852714008 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d580094d0 con 0x7f3d680c6f60 2023-10-30T12:36:48.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6d89e640 1 --2- 172.21.15.125:0/852714008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d680c6f60 0x7f3d681550a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3d60007df0 tx=0x7f3d6000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:48.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d5dffb640 1 -- 172.21.15.125:0/852714008 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d600039a0 con 0x7f3d680c6f60 2023-10-30T12:36:48.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d5dffb640 1 -- 172.21.15.125:0/852714008 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6000f040 con 0x7f3d680c6f60 2023-10-30T12:36:48.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d5dffb640 1 -- 172.21.15.125:0/852714008 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d60003d40 con 0x7f3d680c6f60 2023-10-30T12:36:48.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6f301640 1 -- 172.21.15.125:0/852714008 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d680c9f40 con 0x7f3d680c6f60 2023-10-30T12:36:48.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6f301640 1 -- 172.21.15.125:0/852714008 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d680ca720 con 0x7f3d680c6f60 2023-10-30T12:36:48.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d6f301640 1 -- 172.21.15.125:0/852714008 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d6804e170 con 0x7f3d680c6f60 2023-10-30T12:36:48.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.693+0000 7f3d5dffb640 1 -- 172.21.15.125:0/852714008 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3d60004030 con 0x7f3d680c6f60 2023-10-30T12:36:48.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.697+0000 7f3d5dffb640 1 --2- 172.21.15.125:0/852714008 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3d4c0664d0 0x7f3d4c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:48.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.697+0000 7f3d5dffb640 1 -- 172.21.15.125:0/852714008 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f3d60084350 con 0x7f3d680c6f60 2023-10-30T12:36:48.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.697+0000 7f3d6d09d640 1 --2- 172.21.15.125:0/852714008 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3d4c0664d0 0x7f3d4c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:48.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.697+0000 7f3d6d09d640 1 --2- 172.21.15.125:0/852714008 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3d4c0664d0 0x7f3d4c068990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3d64000f70 tx=0x7f3d64009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:48.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.701+0000 7f3d5dffb640 1 -- 172.21.15.125:0/852714008 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3d600515b0 con 0x7f3d680c6f60 2023-10-30T12:36:48.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.877+0000 7f3d6f301640 1 -- 172.21.15.125:0/852714008 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d68154810 con 0x7f3d680c6f60 2023-10-30T12:36:48.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.881+0000 7f3d5dffb640 1 -- 172.21.15.125:0/852714008 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5756 (secure 0 0 0) 0x7f3d60054bd0 con 0x7f3d680c6f60 2023-10-30T12:36:48.883 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:48.883 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:47.206128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:48.883 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 48 2023-10-30T12:36:48.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.881+0000 7f3d6f301640 1 -- 172.21.15.125:0/852714008 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3d4c0664d0 msgr2=0x7f3d4c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:48.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.881+0000 7f3d6f301640 1 --2- 172.21.15.125:0/852714008 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3d4c0664d0 0x7f3d4c068990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3d64000f70 tx=0x7f3d64009040 comp rx=0 tx=0).stop 2023-10-30T12:36:48.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.881+0000 7f3d6f301640 1 -- 172.21.15.125:0/852714008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d680c6f60 msgr2=0x7f3d681550a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:48.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.881+0000 7f3d6f301640 1 --2- 172.21.15.125:0/852714008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d680c6f60 0x7f3d681550a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3d60007df0 tx=0x7f3d6000d4a0 comp rx=0 tx=0).stop 2023-10-30T12:36:48.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.881+0000 7f3d6f301640 1 -- 172.21.15.125:0/852714008 shutdown_connections 2023-10-30T12:36:48.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.881+0000 7f3d6f301640 1 --2- 172.21.15.125:0/852714008 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3d4c0664d0 0x7f3d4c068990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:48.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.881+0000 7f3d6f301640 1 --2- 172.21.15.125:0/852714008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d680c6f60 0x7f3d681550a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.881+0000 7f3d6f301640 1 --2- 172.21.15.125:0/852714008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d680c6b80 0x7f3d68154b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.881+0000 7f3d6f301640 1 --2- 172.21.15.125:0/852714008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d680c05d0 0x7f3d6806a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.881+0000 7f3d6f301640 1 -- 172.21.15.125:0/852714008 >> 172.21.15.125:0/852714008 conn(0x7f3d680c7910 msgr2=0x7f3d680b0880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.881+0000 7f3d6f301640 1 -- 172.21.15.125:0/852714008 shutdown_connections 2023-10-30T12:36:48.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:48.881+0000 7f3d6f301640 1 -- 172.21.15.125:0/852714008 wait complete. 2023-10-30T12:36:48.904 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:36:50.905 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:51.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.045+0000 7feeeeb0f640 1 Processor -- start 2023-10-30T12:36:51.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeeeb0f640 1 -- start start 2023-10-30T12:36:51.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeeeb0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feee81538a0 0x7feee815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:51.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeeeb0f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee80cb800 con 0x7feee80cf160 2023-10-30T12:36:51.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeeeb0f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee815dc30 con 0x7feee80ca0e0 2023-10-30T12:36:51.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeeeb0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee80cb330 con 0x7feee81538a0 2023-10-30T12:36:51.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeed0ac640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7feee80cf160 0x7feee8164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40454/0 (socket says 172.21.15.125:40454) 2023-10-30T12:36:51.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeed0ac640 1 -- 172.21.15.125:0/839539060 learned_addr learned my addr 172.21.15.125:0/839539060 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:36:51.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeec8ab640 1 --2- 172.21.15.125:0/839539060 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feee81538a0 0x7feee815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:51.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 -- 172.21.15.125:0/839539060 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328932513 0 0) 0x7feee80cb800 con 0x7feee80cf160 2023-10-30T12:36:51.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 -- 172.21.15.125:0/839539060 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed8003530 con 0x7feee80cf160 2023-10-30T12:36:51.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 -- 172.21.15.125:0/839539060 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2313926902 0 0) 0x7feed8003530 con 0x7feee80cf160 2023-10-30T12:36:51.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 -- 172.21.15.125:0/839539060 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feee81538a0 msgr2=0x7feee815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:51.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 --2- 172.21.15.125:0/839539060 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feee81538a0 0x7feee815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:51.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 -- 172.21.15.125:0/839539060 >> v1:172.21.15.136:6789/0 conn(0x7feee80ca0e0 legacy=0x7feee815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:51.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 -- 172.21.15.125:0/839539060 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee80cad70 con 0x7feee80cf160 2023-10-30T12:36:51.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 -- 172.21.15.125:0/839539060 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7feee0001a00 con 0x7feee80cf160 2023-10-30T12:36:51.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 -- 172.21.15.125:0/839539060 >> v1:172.21.15.125:6789/0 conn(0x7feee80cf160 legacy=0x7feee8164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:51.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 --2- 172.21.15.125:0/839539060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed80039b0 0x7feed8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:51.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 --2- 172.21.15.125:0/839539060 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feee81538a0 0x7feed80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:51.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 --2- 172.21.15.125:0/839539060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feed8004d10 0x7feed8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:51.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 -- 172.21.15.125:0/839539060 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee80cb330 con 0x7feed8004d10 2023-10-30T12:36:51.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 -- 172.21.15.125:0/839539060 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed800b990 con 0x7feed80039b0 2023-10-30T12:36:51.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee77fe640 1 -- 172.21.15.125:0/839539060 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed800bb50 con 0x7feee81538a0 2023-10-30T12:36:51.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeed0ac640 1 --2- 172.21.15.125:0/839539060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feed8004d10 0x7feed8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:51.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeec8ab640 1 --2- 172.21.15.125:0/839539060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed80039b0 0x7feed8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:51.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feee7fff640 1 --2- 172.21.15.125:0/839539060 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feee81538a0 0x7feed80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:51.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeed0ac640 1 -- 172.21.15.125:0/839539060 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feee81538a0 msgr2=0x7feed80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:51.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeed0ac640 1 --2- 172.21.15.125:0/839539060 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feee81538a0 0x7feed80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:51.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeed0ac640 1 -- 172.21.15.125:0/839539060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed80039b0 msgr2=0x7feed8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:51.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeed0ac640 1 --2- 172.21.15.125:0/839539060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed80039b0 0x7feed8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:51.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.049+0000 7feeed0ac640 1 -- 172.21.15.125:0/839539060 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed8003530 con 0x7feed8004d10 2023-10-30T12:36:51.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.053+0000 7feeed0ac640 1 --2- 172.21.15.125:0/839539060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feed8004d10 0x7feed8005190 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7feee000a7a0 tx=0x7feee001d2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ffc0b1e788d9dd7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:51.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.053+0000 7feee77fe640 1 -- 172.21.15.125:0/839539060 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee00ad630 con 0x7feed8004d10 2023-10-30T12:36:51.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.053+0000 7feee77fe640 1 -- 172.21.15.125:0/839539060 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee00a8070 con 0x7feed8004d10 2023-10-30T12:36:51.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.053+0000 7feee77fe640 1 -- 172.21.15.125:0/839539060 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee00ad9f0 con 0x7feed8004d10 2023-10-30T12:36:51.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.053+0000 7feeeeb0f640 1 -- 172.21.15.125:0/839539060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feed8004d10 msgr2=0x7feed8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:51.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.053+0000 7feeeeb0f640 1 --2- 172.21.15.125:0/839539060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feed8004d10 0x7feed8005190 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7feee000a7a0 tx=0x7feee001d2a0 comp rx=0 tx=0).stop 2023-10-30T12:36:51.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.053+0000 7feeed0ac640 1 -- 172.21.15.125:0/839539060 reap_dead start 2023-10-30T12:36:51.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.053+0000 7feeeeb0f640 1 -- 172.21.15.125:0/839539060 shutdown_connections 2023-10-30T12:36:51.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.053+0000 7feeeeb0f640 1 -- 172.21.15.125:0/839539060 >> 172.21.15.125:0/839539060 conn(0x7feee806bfe0 msgr2=0x7feee806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:51.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.053+0000 7feeeeb0f640 1 -- 172.21.15.125:0/839539060 shutdown_connections 2023-10-30T12:36:51.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.053+0000 7feeeeb0f640 1 -- 172.21.15.125:0/839539060 wait complete. 2023-10-30T12:36:51.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeeeb0f640 1 Processor -- start 2023-10-30T12:36:51.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeeeb0f640 1 -- start start 2023-10-30T12:36:51.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeeeb0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed8004d10 0x7feee815b150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:51.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeeeb0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feee80b4d30 0x7feee815b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:51.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeeeb0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feee81f2800 0x7feee81f2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:51.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeeeb0f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee815acc0 con 0x7feee81f2800 2023-10-30T12:36:51.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeeeb0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feee815c9a0 con 0x7feed8004d10 2023-10-30T12:36:51.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeeeb0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feee815d220 con 0x7feee80b4d30 2023-10-30T12:36:51.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeed0ac640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feee81f2800 0x7feee81f2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:51.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeed0ac640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feee81f2800 0x7feee81f2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34824/0 (socket says 172.21.15.125:34824) 2023-10-30T12:36:51.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeed0ac640 1 -- 172.21.15.125:0/790568206 learned_addr learned my addr 172.21.15.125:0/790568206 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:51.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeec8ab640 1 --2- 172.21.15.125:0/790568206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed8004d10 0x7feee815b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:51.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feee7fff640 1 --2- 172.21.15.125:0/790568206 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feee80b4d30 0x7feee815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:51.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeed0ac640 1 -- 172.21.15.125:0/790568206 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feee80b4d30 msgr2=0x7feee815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:51.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeed0ac640 1 --2- 172.21.15.125:0/790568206 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feee80b4d30 0x7feee815b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:51.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeed0ac640 1 -- 172.21.15.125:0/790568206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed8004d10 msgr2=0x7feee815b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:51.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeed0ac640 1 --2- 172.21.15.125:0/790568206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed8004d10 0x7feee815b150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:51.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeed0ac640 1 -- 172.21.15.125:0/790568206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee815cc60 con 0x7feee81f2800 2023-10-30T12:36:51.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeed0ac640 1 --2- 172.21.15.125:0/790568206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feee81f2800 0x7feee81f2c80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7feee001d0d0 tx=0x7feed8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:51.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feee57fa640 1 -- 172.21.15.125:0/790568206 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee0001940 con 0x7feee81f2800 2023-10-30T12:36:51.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feee57fa640 1 -- 172.21.15.125:0/790568206 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feee00011d0 con 0x7feee81f2800 2023-10-30T12:36:51.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeeeb0f640 1 -- 172.21.15.125:0/790568206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee815cf40 con 0x7feee81f2800 2023-10-30T12:36:51.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feee57fa640 1 -- 172.21.15.125:0/790568206 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feee0004d70 con 0x7feee81f2800 2023-10-30T12:36:51.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.057+0000 7feeeeb0f640 1 -- 172.21.15.125:0/790568206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feee814e040 con 0x7feee81f2800 2023-10-30T12:36:51.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.061+0000 7feeeeb0f640 1 -- 172.21.15.125:0/790568206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee804e170 con 0x7feee81f2800 2023-10-30T12:36:51.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.061+0000 7feee57fa640 1 -- 172.21.15.125:0/790568206 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feee00aa030 con 0x7feee81f2800 2023-10-30T12:36:51.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.061+0000 7feee57fa640 1 --2- 172.21.15.125:0/790568206 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feecc0664a0 0x7feecc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:51.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.061+0000 7feee57fa640 1 -- 172.21.15.125:0/790568206 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6082+0+0 (secure 0 0 0) 0x7feee011dcf0 con 0x7feee81f2800 2023-10-30T12:36:51.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.061+0000 7feeec8ab640 1 --2- 172.21.15.125:0/790568206 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feecc0664a0 0x7feecc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:51.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.061+0000 7feeec8ab640 1 --2- 172.21.15.125:0/790568206 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feecc0664a0 0x7feecc068960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7feedc009be0 tx=0x7feedc009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:51.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.065+0000 7feee57fa640 1 -- 172.21.15.125:0/790568206 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feee00eaf70 con 0x7feee81f2800 2023-10-30T12:36:51.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.249+0000 7feeeeb0f640 1 -- 172.21.15.125:0/790568206 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feee80b1dd0 con 0x7feee81f2800 2023-10-30T12:36:51.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.249+0000 7feee57fa640 1 -- 172.21.15.125:0/790568206 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+5756 (secure 0 0 0) 0x7feee00ee590 con 0x7feee81f2800 2023-10-30T12:36:51.253 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:51.253 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:47.206128+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:51.254 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 48 2023-10-30T12:36:51.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.253+0000 7feeeeb0f640 1 -- 172.21.15.125:0/790568206 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feecc0664a0 msgr2=0x7feecc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:51.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.253+0000 7feeeeb0f640 1 --2- 172.21.15.125:0/790568206 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feecc0664a0 0x7feecc068960 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7feedc009be0 tx=0x7feedc009190 comp rx=0 tx=0).stop 2023-10-30T12:36:51.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.253+0000 7feeeeb0f640 1 -- 172.21.15.125:0/790568206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feee81f2800 msgr2=0x7feee81f2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:51.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.253+0000 7feeeeb0f640 1 --2- 172.21.15.125:0/790568206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feee81f2800 0x7feee81f2c80 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7feee001d0d0 tx=0x7feed8005800 comp rx=0 tx=0).stop 2023-10-30T12:36:51.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.253+0000 7feeeeb0f640 1 -- 172.21.15.125:0/790568206 shutdown_connections 2023-10-30T12:36:51.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.253+0000 7feeeeb0f640 1 --2- 172.21.15.125:0/790568206 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feecc0664a0 0x7feecc068960 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:51.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.253+0000 7feeeeb0f640 1 --2- 172.21.15.125:0/790568206 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feee81f2800 0x7feee81f2c80 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:51.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.253+0000 7feeeeb0f640 1 --2- 172.21.15.125:0/790568206 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feee80b4d30 0x7feee815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:51.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.253+0000 7feeeeb0f640 1 --2- 172.21.15.125:0/790568206 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feed8004d10 0x7feee815b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:51.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.253+0000 7feeeeb0f640 1 -- 172.21.15.125:0/790568206 >> 172.21.15.125:0/790568206 conn(0x7feee806bfe0 msgr2=0x7feee80cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:51.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.253+0000 7feeeeb0f640 1 -- 172.21.15.125:0/790568206 shutdown_connections 2023-10-30T12:36:51.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:51.253+0000 7feeeeb0f640 1 -- 172.21.15.125:0/790568206 wait complete. 2023-10-30T12:36:51.274 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:36:53.276 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:53.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.417+0000 7f1ce1605640 1 Processor -- start 2023-10-30T12:36:53.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.417+0000 7f1ce1605640 1 -- start start 2023-10-30T12:36:53.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.421+0000 7f1ce1605640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1cdc1538a0 0x7f1cdc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:53.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.421+0000 7f1ce1605640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cdc0ca0e0 0x7f1cdc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:53.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.421+0000 7f1ce1605640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cdc0cb800 con 0x7f1cdc0cf160 2023-10-30T12:36:53.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.421+0000 7f1ce1605640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cdc15dc30 con 0x7f1cdc1538a0 2023-10-30T12:36:53.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.421+0000 7f1ce1605640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc0cb330 con 0x7f1cdc0ca0e0 2023-10-30T12:36:53.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.421+0000 7f1cdaffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1cdc1538a0 0x7f1cdc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:53.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.421+0000 7f1cdaffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1cdc1538a0 0x7f1cdc15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34826/0 (socket says 172.21.15.125:34826) 2023-10-30T12:36:53.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.421+0000 7f1cdaffd640 1 -- 172.21.15.125:0/2516542008 learned_addr learned my addr 172.21.15.125:0/2516542008 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:53.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.421+0000 7f1cda7fc640 1 --2- 172.21.15.125:0/2516542008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cdc0ca0e0 0x7f1cdc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:53.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.421+0000 7f1cdaffd640 1 -- 172.21.15.125:0/2516542008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cdc0ca0e0 msgr2=0x7f1cdc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:53.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.421+0000 7f1cdaffd640 1 --2- 172.21.15.125:0/2516542008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cdc0ca0e0 0x7f1cdc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:53.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.421+0000 7f1cdaffd640 1 -- 172.21.15.125:0/2516542008 >> v1:172.21.15.136:6789/0 conn(0x7f1cdc0cf160 legacy=0x7f1cdc0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:53.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.421+0000 7f1cdaffd640 1 -- 172.21.15.125:0/2516542008 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cdc0cad70 con 0x7f1cdc1538a0 2023-10-30T12:36:53.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.421+0000 7f1cdaffd640 1 --2- 172.21.15.125:0/2516542008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cdc1538a0 0x7f1cdc15d6f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009870 tx=0x7f1ccc01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1400e29bbc37ab44 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:53.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1cd9ffb640 1 -- 172.21.15.125:0/2516542008 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc0a8a30 con 0x7f1cdc1538a0 2023-10-30T12:36:53.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1cd9ffb640 1 -- 172.21.15.125:0/2516542008 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc0a8b90 con 0x7f1cdc1538a0 2023-10-30T12:36:53.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1cd9ffb640 1 -- 172.21.15.125:0/2516542008 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ccc0b0440 con 0x7f1cdc1538a0 2023-10-30T12:36:53.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 -- 172.21.15.125:0/2516542008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cdc1538a0 msgr2=0x7f1cdc15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:53.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 --2- 172.21.15.125:0/2516542008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cdc1538a0 0x7f1cdc15d6f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc009870 tx=0x7f1ccc01ed30 comp rx=0 tx=0).stop 2023-10-30T12:36:53.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 -- 172.21.15.125:0/2516542008 shutdown_connections 2023-10-30T12:36:53.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 --2- 172.21.15.125:0/2516542008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cdc0ca0e0 0x7f1cdc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:53.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 --2- 172.21.15.125:0/2516542008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cdc1538a0 0x7f1cdc15d6f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:53.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 -- 172.21.15.125:0/2516542008 >> 172.21.15.125:0/2516542008 conn(0x7f1cdc06bfe0 msgr2=0x7f1cdc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:53.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 -- 172.21.15.125:0/2516542008 shutdown_connections 2023-10-30T12:36:53.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 -- 172.21.15.125:0/2516542008 wait complete. 2023-10-30T12:36:53.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 Processor -- start 2023-10-30T12:36:53.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 -- start start 2023-10-30T12:36:53.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cdc0ca0e0 0x7f1cdc15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:53.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cdc0cf160 0x7f1cdc15adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:53.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cdc1538a0 0x7f1cdc1e4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:53.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc1e3b30 con 0x7f1cdc0cf160 2023-10-30T12:36:53.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc1589e0 con 0x7f1cdc1538a0 2023-10-30T12:36:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1ce1605640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cdc157170 con 0x7f1cdc0ca0e0 2023-10-30T12:36:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.425+0000 7f1cda7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cdc0cf160 0x7f1cdc15adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cda7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cdc0cf160 0x7f1cdc15adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34830/0 (socket says 172.21.15.125:34830) 2023-10-30T12:36:53.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cda7fc640 1 -- 172.21.15.125:0/3097368272 learned_addr learned my addr 172.21.15.125:0/3097368272 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:53.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cdaffd640 1 --2- 172.21.15.125:0/3097368272 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cdc0ca0e0 0x7f1cdc15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:53.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cdb7fe640 1 --2- 172.21.15.125:0/3097368272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cdc1538a0 0x7f1cdc1e4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:53.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cda7fc640 1 -- 172.21.15.125:0/3097368272 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cdc0ca0e0 msgr2=0x7f1cdc15a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:53.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cda7fc640 1 --2- 172.21.15.125:0/3097368272 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cdc0ca0e0 0x7f1cdc15a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:53.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cda7fc640 1 -- 172.21.15.125:0/3097368272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cdc1538a0 msgr2=0x7f1cdc1e4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:53.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cda7fc640 1 --2- 172.21.15.125:0/3097368272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cdc1538a0 0x7f1cdc1e4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:53.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cda7fc640 1 -- 172.21.15.125:0/3097368272 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ccc0094d0 con 0x7f1cdc0cf160 2023-10-30T12:36:53.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cda7fc640 1 --2- 172.21.15.125:0/3097368272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cdc0cf160 0x7f1cdc15adf0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000d800 tx=0x7f1cd000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:53.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cc37fe640 1 -- 172.21.15.125:0/3097368272 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd000b6c0 con 0x7f1cdc0cf160 2023-10-30T12:36:53.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cc37fe640 1 -- 172.21.15.125:0/3097368272 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cd000b820 con 0x7f1cdc0cf160 2023-10-30T12:36:53.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1ce1605640 1 -- 172.21.15.125:0/3097368272 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cdc158ca0 con 0x7f1cdc0cf160 2023-10-30T12:36:53.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cc37fe640 1 -- 172.21.15.125:0/3097368272 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cd000bb90 con 0x7f1cdc0cf160 2023-10-30T12:36:53.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1ce1605640 1 -- 172.21.15.125:0/3097368272 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cdc1e4060 con 0x7f1cdc0cf160 2023-10-30T12:36:53.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1ce1605640 1 -- 172.21.15.125:0/3097368272 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cdc0b2a50 con 0x7f1cdc0cf160 2023-10-30T12:36:53.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cc37fe640 1 -- 172.21.15.125:0/3097368272 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1cd000bcf0 con 0x7f1cdc0cf160 2023-10-30T12:36:53.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cc37fe640 1 --2- 172.21.15.125:0/3097368272 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1cc80664a0 0x7f1cc8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.429+0000 7f1cc37fe640 1 -- 172.21.15.125:0/3097368272 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 6082+0+0 (secure 0 0 0) 0x7f1cd0084510 con 0x7f1cdc0cf160 2023-10-30T12:36:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.433+0000 7f1cdaffd640 1 --2- 172.21.15.125:0/3097368272 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1cc80664a0 0x7f1cc8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.433+0000 7f1cdaffd640 1 --2- 172.21.15.125:0/3097368272 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1cc80664a0 0x7f1cc8068960 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc0b2000 tx=0x7f1ccc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:53.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.433+0000 7f1cc37fe640 1 -- 172.21.15.125:0/3097368272 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1cd0051780 con 0x7f1cdc0cf160 2023-10-30T12:36:53.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.613+0000 7f1ce1605640 1 -- 172.21.15.125:0/3097368272 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cdc04e170 con 0x7f1cdc0cf160 2023-10-30T12:36:53.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.617+0000 7f1cc37fe640 1 -- 172.21.15.125:0/3097368272 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5756 (secure 0 0 0) 0x7f1cd0054da0 con 0x7f1cdc0cf160 2023-10-30T12:36:53.619 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:53.619 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:53.109978+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:53.620 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 49 2023-10-30T12:36:53.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.617+0000 7f1ce1605640 1 -- 172.21.15.125:0/3097368272 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1cc80664a0 msgr2=0x7f1cc8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:53.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.617+0000 7f1ce1605640 1 --2- 172.21.15.125:0/3097368272 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1cc80664a0 0x7f1cc8068960 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1ccc0b2000 tx=0x7f1ccc009040 comp rx=0 tx=0).stop 2023-10-30T12:36:53.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.617+0000 7f1ce1605640 1 -- 172.21.15.125:0/3097368272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cdc0cf160 msgr2=0x7f1cdc15adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:53.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.617+0000 7f1ce1605640 1 --2- 172.21.15.125:0/3097368272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cdc0cf160 0x7f1cdc15adf0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1cd000d800 tx=0x7f1cd000dcc0 comp rx=0 tx=0).stop 2023-10-30T12:36:53.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.617+0000 7f1ce1605640 1 -- 172.21.15.125:0/3097368272 shutdown_connections 2023-10-30T12:36:53.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.617+0000 7f1ce1605640 1 --2- 172.21.15.125:0/3097368272 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1cc80664a0 0x7f1cc8068960 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:53.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.617+0000 7f1ce1605640 1 --2- 172.21.15.125:0/3097368272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cdc1538a0 0x7f1cdc1e4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:53.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.617+0000 7f1ce1605640 1 --2- 172.21.15.125:0/3097368272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cdc0cf160 0x7f1cdc15adf0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:53.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.617+0000 7f1ce1605640 1 --2- 172.21.15.125:0/3097368272 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cdc0ca0e0 0x7f1cdc15a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:53.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.617+0000 7f1ce1605640 1 -- 172.21.15.125:0/3097368272 >> 172.21.15.125:0/3097368272 conn(0x7f1cdc06bfe0 msgr2=0x7f1cdc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:53.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.617+0000 7f1ce1605640 1 -- 172.21.15.125:0/3097368272 shutdown_connections 2023-10-30T12:36:53.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:53.617+0000 7f1ce1605640 1 -- 172.21.15.125:0/3097368272 wait complete. 2023-10-30T12:36:53.640 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=4294 2023-10-30T12:36:53.640 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.e 2023-10-30T12:36:58.245 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:36:58.371 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:36:58.421 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:36:59.029 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:36:59.033 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:36:59.036 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:36:59.054 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:36:59.065 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:36:59.067 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:36:59.379 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:36:59.393 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:36:59.393 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:36:59.393 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:36:59.393 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:36:59.393 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:36:59.394 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:36:59.394 INFO:tasks.workunit.client.0.smithi125.stdout: 50530 ? 00:00:00 ps 2023-10-30T12:36:59.394 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:36:59.417 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:36:59.423 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:36:59.437 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:36:59.442 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:36:59.445 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:36:59.465 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:36:59.476 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:36:59.477 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:36:59.595 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:36:59.608 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:36:59.608 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:36:59.609 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:36:59.609 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:36:59.609 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:36:59.609 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:36:59.610 INFO:tasks.workunit.client.0.smithi125.stdout: 50540 ? 00:00:00 ps 2023-10-30T12:36:59.610 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:36:59.629 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:36:59.635 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:36:59.641 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T12:36:59.641 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T12:36:59.642 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T12:36:59.645 INFO:tasks.ceph.mds.e:Started 2023-10-30T12:36:59.646 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:36:59.647 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:36:59.651 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:36:59.654 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:36:59.671 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:36:59.683 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:36:59.684 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:36:59.733 INFO:tasks.ceph.mds.e.smithi125.stdout:starting mds.e at 2023-10-30T12:36:59.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.784+0000 7fbde8ed6640 1 Processor -- start 2023-10-30T12:36:59.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.784+0000 7fbde8ed6640 1 -- start start 2023-10-30T12:36:59.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.784+0000 7fbde8ed6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbde40c6b80 0x7fbde40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:59.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.784+0000 7fbde8ed6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbde40c6f60 0x7fbde40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:59.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.784+0000 7fbde8ed6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde40c0a50 con 0x7fbde40c8430 2023-10-30T12:36:59.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.784+0000 7fbde8ed6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde406f6e0 con 0x7fbde40c6f60 2023-10-30T12:36:59.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.784+0000 7fbde8ed6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde406f9c0 con 0x7fbde40c6b80 2023-10-30T12:36:59.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.784+0000 7fbdd9d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbde40c6f60 0x7fbde40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:59.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.784+0000 7fbdd9d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbde40c6f60 0x7fbde40c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34850/0 (socket says 172.21.15.125:34850) 2023-10-30T12:36:59.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.784+0000 7fbdd9d9b640 1 -- 172.21.15.125:0/583972685 learned_addr learned my addr 172.21.15.125:0/583972685 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:59.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.784+0000 7fbde259c640 1 --2- 172.21.15.125:0/583972685 >> v2:172.21.15.136:3300/0 conn(0x7fbde40c6b80 0x7fbde40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:59.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.788+0000 7fbdd9d9b640 1 -- 172.21.15.125:0/583972685 >> v2:172.21.15.136:3300/0 conn(0x7fbde40c6b80 msgr2=0x7fbde40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:36:59.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.788+0000 7fbdd9d9b640 1 --2- 172.21.15.125:0/583972685 >> v2:172.21.15.136:3300/0 conn(0x7fbde40c6b80 0x7fbde40bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:59.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.788+0000 7fbdd9d9b640 1 -- 172.21.15.125:0/583972685 >> v1:172.21.15.136:6789/0 conn(0x7fbde40c8430 legacy=0x7fbde413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:59.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.788+0000 7fbdd9d9b640 1 -- 172.21.15.125:0/583972685 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde406fca0 con 0x7fbde40c6f60 2023-10-30T12:36:59.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.788+0000 7fbdd9d9b640 1 --2- 172.21.15.125:0/583972685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbde40c6f60 0x7fbde40c0400 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbde406f1b0 tx=0x7fbdcc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e54b65755cb9f027 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:59.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.788+0000 7fbde1d9b640 1 -- 172.21.15.125:0/583972685 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc0ab610 con 0x7fbde40c6f60 2023-10-30T12:36:59.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.788+0000 7fbde1d9b640 1 -- 172.21.15.125:0/583972685 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdcc0ab770 con 0x7fbde40c6f60 2023-10-30T12:36:59.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.788+0000 7fbde1d9b640 1 -- 172.21.15.125:0/583972685 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdcc0abb00 con 0x7fbde40c6f60 2023-10-30T12:36:59.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.788+0000 7fbde8ed6640 1 -- 172.21.15.125:0/583972685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbde40c6f60 msgr2=0x7fbde40c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:59.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.788+0000 7fbde8ed6640 1 --2- 172.21.15.125:0/583972685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbde40c6f60 0x7fbde40c0400 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbde406f1b0 tx=0x7fbdcc01c500 comp rx=0 tx=0).stop 2023-10-30T12:36:59.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde8ed6640 1 -- 172.21.15.125:0/583972685 shutdown_connections 2023-10-30T12:36:59.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde8ed6640 1 --2- 172.21.15.125:0/583972685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbde40c6f60 0x7fbde40c0400 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:59.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde8ed6640 1 --2- 172.21.15.125:0/583972685 >> v2:172.21.15.136:3300/0 conn(0x7fbde40c6b80 0x7fbde40bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:59.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde8ed6640 1 -- 172.21.15.125:0/583972685 >> 172.21.15.125:0/583972685 conn(0x7fbde40c7910 msgr2=0x7fbde40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:59.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde8ed6640 1 -- 172.21.15.125:0/583972685 shutdown_connections 2023-10-30T12:36:59.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde8ed6640 1 -- 172.21.15.125:0/583972685 wait complete. 2023-10-30T12:36:59.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde8ed6640 1 Processor -- start 2023-10-30T12:36:59.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde8ed6640 1 -- start start 2023-10-30T12:36:59.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde8ed6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbde40c6b80 0x7fbde4154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:59.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde8ed6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbde40c6f60 0x7fbde4139e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:59.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde8ed6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbde40c8430 0x7fbde40c2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:59.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde8ed6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde40c3980 con 0x7fbde40c8430 2023-10-30T12:36:59.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde8ed6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde41f19d0 con 0x7fbde40c6b80 2023-10-30T12:36:59.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde8ed6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde40c4590 con 0x7fbde40c6f60 2023-10-30T12:36:59.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde2d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbde40c8430 0x7fbde40c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:59.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde2d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbde40c8430 0x7fbde40c2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34862/0 (socket says 172.21.15.125:34862) 2023-10-30T12:36:59.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde2d9d640 1 -- 172.21.15.125:0/2103791933 learned_addr learned my addr 172.21.15.125:0/2103791933 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:36:59.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbdd9d9b640 1 --2- 172.21.15.125:0/2103791933 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbde40c6f60 0x7fbde4139e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:59.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde259c640 1 --2- 172.21.15.125:0/2103791933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbde40c6b80 0x7fbde4154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:59.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde2d9d640 1 -- 172.21.15.125:0/2103791933 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbde40c6f60 msgr2=0x7fbde4139e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:59.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde2d9d640 1 --2- 172.21.15.125:0/2103791933 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbde40c6f60 0x7fbde4139e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:59.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde2d9d640 1 -- 172.21.15.125:0/2103791933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbde40c6b80 msgr2=0x7fbde4154480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:59.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde2d9d640 1 --2- 172.21.15.125:0/2103791933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbde40c6b80 0x7fbde4154480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:59.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.792+0000 7fbde2d9d640 1 -- 172.21.15.125:0/2103791933 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdcc0094d0 con 0x7fbde40c8430 2023-10-30T12:36:59.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.796+0000 7fbde2d9d640 1 --2- 172.21.15.125:0/2103791933 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbde40c8430 0x7fbde40c2210 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbddc00c340 tx=0x7fbddc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:59.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.796+0000 7fbddb7fe640 1 -- 172.21.15.125:0/2103791933 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbddc005220 con 0x7fbde40c8430 2023-10-30T12:36:59.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.796+0000 7fbddb7fe640 1 -- 172.21.15.125:0/2103791933 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbddc005380 con 0x7fbde40c8430 2023-10-30T12:36:59.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.796+0000 7fbddb7fe640 1 -- 172.21.15.125:0/2103791933 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbddc007270 con 0x7fbde40c8430 2023-10-30T12:36:59.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.796+0000 7fbde8ed6640 1 -- 172.21.15.125:0/2103791933 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde40c4870 con 0x7fbde40c8430 2023-10-30T12:36:59.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.796+0000 7fbde8ed6640 1 -- 172.21.15.125:0/2103791933 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde41f1f70 con 0x7fbde40c8430 2023-10-30T12:36:59.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.796+0000 7fbde8ed6640 1 -- 172.21.15.125:0/2103791933 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde404e170 con 0x7fbde40c8430 2023-10-30T12:36:59.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.796+0000 7fbddb7fe640 1 -- 172.21.15.125:0/2103791933 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbddc0073d0 con 0x7fbde40c8430 2023-10-30T12:36:59.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.796+0000 7fbddb7fe640 1 --2- 172.21.15.125:0/2103791933 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbdc40664d0 0x7fbdc4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:36:59.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.796+0000 7fbddb7fe640 1 -- 172.21.15.125:0/2103791933 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 6170+0+0 (secure 0 0 0) 0x7fbddc083320 con 0x7fbde40c8430 2023-10-30T12:36:59.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.796+0000 7fbde259c640 1 --2- 172.21.15.125:0/2103791933 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbdc40664d0 0x7fbdc4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:36:59.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.800+0000 7fbde259c640 1 --2- 172.21.15.125:0/2103791933 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbdc40664d0 0x7fbdc4068990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbdd40113e0 tx=0x7fbdd400b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:36:59.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.800+0000 7fbddb7fe640 1 -- 172.21.15.125:0/2103791933 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbddc0505e0 con 0x7fbde40c8430 2023-10-30T12:36:59.894 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:36:59.908 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:36:59.909 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:36:59.909 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:36:59.909 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:36:59.909 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:36:59.909 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:36:59.910 INFO:tasks.workunit.client.0.smithi125.stdout: 50596 ? 00:00:00 ps 2023-10-30T12:36:59.910 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:36:59.933 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:36:59.938 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:36:59.952 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:36:59.955 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:36:59.957 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:36:59.977 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:36:59.988 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:36:59.989 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:36:59.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.988+0000 7fbde8ed6640 1 -- 172.21.15.125:0/2103791933 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbde40c60f0 con 0x7fbde40c8430 2023-10-30T12:36:59.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.988+0000 7fbddb7fe640 1 -- 172.21.15.125:0/2103791933 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+5756 (secure 0 0 0) 0x7fbddc053c00 con 0x7fbde40c8430 2023-10-30T12:36:59.993 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:36:59.993 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:53.109978+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:36:59.994 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 49 2023-10-30T12:36:59.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.992+0000 7fbde8ed6640 1 -- 172.21.15.125:0/2103791933 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbdc40664d0 msgr2=0x7fbdc4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:59.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.992+0000 7fbde8ed6640 1 --2- 172.21.15.125:0/2103791933 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbdc40664d0 0x7fbdc4068990 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbdd40113e0 tx=0x7fbdd400b040 comp rx=0 tx=0).stop 2023-10-30T12:36:59.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.992+0000 7fbde8ed6640 1 -- 172.21.15.125:0/2103791933 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbde40c8430 msgr2=0x7fbde40c2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:36:59.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.992+0000 7fbde8ed6640 1 --2- 172.21.15.125:0/2103791933 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbde40c8430 0x7fbde40c2210 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fbddc00c340 tx=0x7fbddc00c800 comp rx=0 tx=0).stop 2023-10-30T12:36:59.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.992+0000 7fbde8ed6640 1 -- 172.21.15.125:0/2103791933 shutdown_connections 2023-10-30T12:36:59.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.992+0000 7fbde8ed6640 1 --2- 172.21.15.125:0/2103791933 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbdc40664d0 0x7fbdc4068990 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:59.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.992+0000 7fbde8ed6640 1 --2- 172.21.15.125:0/2103791933 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbde40c8430 0x7fbde40c2210 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:59.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.992+0000 7fbde8ed6640 1 --2- 172.21.15.125:0/2103791933 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbde40c6f60 0x7fbde4139e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:59.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.992+0000 7fbde8ed6640 1 --2- 172.21.15.125:0/2103791933 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbde40c6b80 0x7fbde4154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:36:59.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.992+0000 7fbde8ed6640 1 -- 172.21.15.125:0/2103791933 >> 172.21.15.125:0/2103791933 conn(0x7fbde40c7910 msgr2=0x7fbde40bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:36:59.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.992+0000 7fbde8ed6640 1 -- 172.21.15.125:0/2103791933 shutdown_connections 2023-10-30T12:36:59.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:36:59.992+0000 7fbde8ed6640 1 -- 172.21.15.125:0/2103791933 wait complete. 2023-10-30T12:37:00.016 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T12:37:00.061 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:37:00.078 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:37:00.079 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:37:00.079 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:37:00.079 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:37:00.079 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:37:00.080 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:37:00.080 INFO:tasks.workunit.client.0.smithi125.stdout: 50608 ? 00:00:00 ps 2023-10-30T12:37:00.080 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:00.106 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:37:00.110 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:37:00.120 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:37:00.124 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:00.127 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:37:00.143 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:00.151 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:37:00.152 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:37:00.271 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:37:00.286 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:37:00.286 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:37:00.286 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:37:00.286 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:37:00.287 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:37:00.287 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:37:00.287 INFO:tasks.workunit.client.0.smithi125.stdout: 50619 ? 00:00:00 ps 2023-10-30T12:37:00.288 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:00.311 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:37:00.316 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:37:00.329 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:37:00.333 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:00.335 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:37:00.356 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:00.373 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:37:00.374 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:37:00.470 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:37:00.485 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:37:00.485 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:37:00.485 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:37:00.485 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:37:00.486 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:37:00.486 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:37:00.486 INFO:tasks.workunit.client.0.smithi125.stdout: 50629 ? 00:00:00 ps 2023-10-30T12:37:00.487 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:00.981 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:37:00.986 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:37:01.000 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:37:01.004 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:01.007 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:37:01.027 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:01.038 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:37:01.039 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:37:01.094 INFO:tasks.ceph.mds.f.smithi136.stderr:2023-10-30T12:37:01.093+0000 7f3543399640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:37:01.095 INFO:tasks.ceph.mds.f.smithi136.stderr:2023-10-30T12:37:01.093+0000 7f3543399640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:37:01.095 INFO:tasks.ceph.mds.f.smithi136.stderr:2023-10-30T12:37:01.093+0000 7f3543399640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T12:37:01.095 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:37:01.109 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:37:01.109 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:37:01.110 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:37:01.110 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:37:01.110 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:37:01.110 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:37:01.110 INFO:tasks.workunit.client.0.smithi125.stdout: 50639 ? 00:00:00 ps 2023-10-30T12:37:01.111 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:01.132 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:37:01.138 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:37:01.153 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:37:01.157 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:01.160 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:37:01.181 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:01.194 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:37:01.195 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:37:01.236 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:37:01.258 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:37:01.258 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:37:01.258 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:37:01.258 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:37:01.259 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:37:01.259 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:37:01.259 INFO:tasks.workunit.client.0.smithi125.stdout: 50649 ? 00:00:00 ps 2023-10-30T12:37:01.261 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:01.312 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:37:01.315 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:37:01.323 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:37:01.326 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:01.329 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:37:01.342 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:01.351 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:37:01.352 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:37:01.463 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:37:01.477 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:37:01.478 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:37:01.478 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:37:01.478 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:37:01.478 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:37:01.479 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:37:01.479 INFO:tasks.workunit.client.0.smithi125.stdout: 50659 ? 00:00:00 ps 2023-10-30T12:37:01.479 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:01.510 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:37:01.515 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:37:01.535 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:37:01.545 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:01.549 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:37:01.584 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:01.605 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:37:01.607 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:37:01.729 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:37:01.743 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:37:01.744 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:37:01.744 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:37:01.744 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:37:01.744 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:37:01.745 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:37:01.745 INFO:tasks.workunit.client.0.smithi125.stdout: 50669 ? 00:00:00 ps 2023-10-30T12:37:01.746 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:01.815 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:37:01.819 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:37:01.831 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:37:01.835 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:01.838 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:37:01.857 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:01.869 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:37:01.870 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:37:01.921 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:37:01.935 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:37:01.935 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:37:01.935 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:37:01.936 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:37:01.936 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:37:01.936 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:37:01.936 INFO:tasks.workunit.client.0.smithi125.stdout: 50679 ? 00:00:00 ps 2023-10-30T12:37:01.937 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:01.958 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:37:01.962 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:37:01.974 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:37:01.978 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:01.981 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:37:01.999 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:02.011 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:37:02.012 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:37:02.016 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:02.054 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:37:02.069 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:37:02.069 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:37:02.069 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:37:02.069 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:37:02.069 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:37:02.070 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:37:02.070 INFO:tasks.workunit.client.0.smithi125.stdout: 50693 ? 00:00:00 ps 2023-10-30T12:37:02.070 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:02.092 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:37:02.096 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:37:02.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff674ff5640 1 Processor -- start 2023-10-30T12:37:02.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff674ff5640 1 -- start start 2023-10-30T12:37:02.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff674ff5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff670153bf0 0x7ff670071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:02.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff674ff5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff6700c5f60 0x7ff6700cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:02.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff674ff5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6700700b0 con 0x7ff6700c6fe0 2023-10-30T12:37:02.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff674ff5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6700c6810 con 0x7ff6700c5f60 2023-10-30T12:37:02.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff674ff5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff670159900 con 0x7ff670153bf0 2023-10-30T12:37:02.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff66ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff6700c5f60 0x7ff6700cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:02.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff66dd9b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff6700c6fe0 0x7ff6700c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36236/0 (socket says 172.21.15.125:36236) 2023-10-30T12:37:02.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff66ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff6700c5f60 0x7ff6700cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33346/0 (socket says 172.21.15.125:33346) 2023-10-30T12:37:02.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff66dd9b640 1 -- 172.21.15.125:0/2418915525 learned_addr learned my addr 172.21.15.125:0/2418915525 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:37:02.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff66e59c640 1 --2- 172.21.15.125:0/2418915525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff670153bf0 0x7ff670071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:02.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff66ed9d640 1 -- 172.21.15.125:0/2418915525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff670153bf0 msgr2=0x7ff670071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:02.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff66ed9d640 1 --2- 172.21.15.125:0/2418915525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff670153bf0 0x7ff670071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:02.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff66ed9d640 1 -- 172.21.15.125:0/2418915525 >> v1:172.21.15.125:6789/0 conn(0x7ff6700c6fe0 legacy=0x7ff6700c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:02.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.156+0000 7ff66ed9d640 1 -- 172.21.15.125:0/2418915525 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6700c6530 con 0x7ff6700c5f60 2023-10-30T12:37:02.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.160+0000 7ff66ed9d640 1 --2- 172.21.15.125:0/2418915525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6700c5f60 0x7ff6700cb570 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff668009d90 tx=0x7ff66801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb1798c4a801c50c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:02.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.160+0000 7ff66d59a640 1 -- 172.21.15.125:0/2418915525 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6680af600 con 0x7ff6700c5f60 2023-10-30T12:37:02.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.160+0000 7ff66d59a640 1 -- 172.21.15.125:0/2418915525 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6680ab050 con 0x7ff6700c5f60 2023-10-30T12:37:02.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.160+0000 7ff66d59a640 1 -- 172.21.15.125:0/2418915525 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6680af980 con 0x7ff6700c5f60 2023-10-30T12:37:02.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.160+0000 7ff674ff5640 1 -- 172.21.15.125:0/2418915525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6700c5f60 msgr2=0x7ff6700cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:02.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.160+0000 7ff674ff5640 1 --2- 172.21.15.125:0/2418915525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6700c5f60 0x7ff6700cb570 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ff668009d90 tx=0x7ff66801f130 comp rx=0 tx=0).stop 2023-10-30T12:37:02.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.160+0000 7ff674ff5640 1 -- 172.21.15.125:0/2418915525 shutdown_connections 2023-10-30T12:37:02.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.160+0000 7ff674ff5640 1 --2- 172.21.15.125:0/2418915525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6700c5f60 0x7ff6700cb570 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:02.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.160+0000 7ff674ff5640 1 --2- 172.21.15.125:0/2418915525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff670153bf0 0x7ff670071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:02.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.160+0000 7ff674ff5640 1 -- 172.21.15.125:0/2418915525 >> 172.21.15.125:0/2418915525 conn(0x7ff67006c6e0 msgr2=0x7ff670069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:02.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.160+0000 7ff674ff5640 1 -- 172.21.15.125:0/2418915525 shutdown_connections 2023-10-30T12:37:02.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.160+0000 7ff674ff5640 1 -- 172.21.15.125:0/2418915525 wait complete. 2023-10-30T12:37:02.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff674ff5640 1 Processor -- start 2023-10-30T12:37:02.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff674ff5640 1 -- start start 2023-10-30T12:37:02.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff674ff5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6700c5f60 0x7ff670148390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:02.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff674ff5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6700c6fe0 0x7ff6701f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:02.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff674ff5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff670153bf0 0x7ff6700c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:02.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff674ff5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6701f2850 con 0x7ff6700c6fe0 2023-10-30T12:37:02.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff674ff5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6701570a0 con 0x7ff670153bf0 2023-10-30T12:37:02.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff674ff5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6700ca040 con 0x7ff6700c5f60 2023-10-30T12:37:02.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff66dd9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6700c6fe0 0x7ff6701f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:02.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff66dd9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6700c6fe0 0x7ff6701f2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33348/0 (socket says 172.21.15.125:33348) 2023-10-30T12:37:02.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff66dd9b640 1 -- 172.21.15.125:0/2879680442 learned_addr learned my addr 172.21.15.125:0/2879680442 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:37:02.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff66e59c640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6700c5f60 0x7ff670148390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:02.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff66ed9d640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff670153bf0 0x7ff6700c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:02.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff66dd9b640 1 -- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6700c5f60 msgr2=0x7ff670148390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:02.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff66dd9b640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6700c5f60 0x7ff670148390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:02.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff66dd9b640 1 -- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff670153bf0 msgr2=0x7ff6700c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:02.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff66dd9b640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff670153bf0 0x7ff6700c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:02.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff66dd9b640 1 -- 172.21.15.125:0/2879680442 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff668009a70 con 0x7ff6700c6fe0 2023-10-30T12:37:02.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff66e59c640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6700c5f60 0x7ff670148390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:02.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff66dd9b640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6700c6fe0 0x7ff6701f2ce0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff66400c500 tx=0x7ff66400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:02.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff65effd640 1 -- 172.21.15.125:0/2879680442 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664005cf0 con 0x7ff6700c6fe0 2023-10-30T12:37:02.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff65effd640 1 -- 172.21.15.125:0/2879680442 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff664005e50 con 0x7ff6700c6fe0 2023-10-30T12:37:02.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff65effd640 1 -- 172.21.15.125:0/2879680442 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6640072f0 con 0x7ff6700c6fe0 2023-10-30T12:37:02.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff674ff5640 1 -- 172.21.15.125:0/2879680442 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6701f2220 con 0x7ff6700c6fe0 2023-10-30T12:37:02.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff674ff5640 1 -- 172.21.15.125:0/2879680442 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6700c4130 con 0x7ff6700c6fe0 2023-10-30T12:37:02.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.164+0000 7ff674ff5640 1 -- 172.21.15.125:0/2879680442 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff67004e170 con 0x7ff6700c6fe0 2023-10-30T12:37:02.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.168+0000 7ff65effd640 1 -- 172.21.15.125:0/2879680442 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff664007450 con 0x7ff6700c6fe0 2023-10-30T12:37:02.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.168+0000 7ff65effd640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6500664a0 0x7ff650068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:02.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.168+0000 7ff65effd640 1 -- 172.21.15.125:0/2879680442 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6154+0+0 (secure 0 0 0) 0x7ff664083550 con 0x7ff6700c6fe0 2023-10-30T12:37:02.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.168+0000 7ff66e59c640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6500664a0 0x7ff650068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:02.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.168+0000 7ff66e59c640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6500664a0 0x7ff650068960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff66000b9f0 tx=0x7ff660008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:02.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.172+0000 7ff65effd640 1 -- 172.21.15.125:0/2879680442 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff6640508b0 con 0x7ff6700c6fe0 2023-10-30T12:37:02.363 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:37:02.372 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:02.395 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:37:02.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.400+0000 7ff674ff5640 1 -- 172.21.15.125:0/2879680442 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff67006fe90 con 0x7ff6700c6fe0 2023-10-30T12:37:02.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.404+0000 7ff65effd640 1 -- 172.21.15.125:0/2879680442 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 50 v50) v1 ==== 94+0+6550 (secure 0 0 0) 0x7ff664053ed0 con 0x7ff6700c6fe0 2023-10-30T12:37:02.407 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:37:02.407 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":50,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:36:53.109978+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:02.408 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 50 2023-10-30T12:37:02.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.404+0000 7ff674ff5640 1 -- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6500664a0 msgr2=0x7ff650068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:02.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.404+0000 7ff674ff5640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6500664a0 0x7ff650068960 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff66000b9f0 tx=0x7ff660008040 comp rx=0 tx=0).stop 2023-10-30T12:37:02.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.404+0000 7ff674ff5640 1 -- 172.21.15.125:0/2879680442 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6700c6fe0 msgr2=0x7ff6701f2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:02.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.408+0000 7ff674ff5640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6700c6fe0 0x7ff6701f2ce0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7ff66400c500 tx=0x7ff66400c9c0 comp rx=0 tx=0).stop 2023-10-30T12:37:02.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.408+0000 7ff674ff5640 1 -- 172.21.15.125:0/2879680442 shutdown_connections 2023-10-30T12:37:02.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.408+0000 7ff674ff5640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6500664a0 0x7ff650068960 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:02.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.408+0000 7ff674ff5640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff670153bf0 0x7ff6700c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:02.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.408+0000 7ff674ff5640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6700c6fe0 0x7ff6701f2ce0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:02.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.408+0000 7ff674ff5640 1 --2- 172.21.15.125:0/2879680442 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6700c5f60 0x7ff670148390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:02.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.408+0000 7ff674ff5640 1 -- 172.21.15.125:0/2879680442 >> 172.21.15.125:0/2879680442 conn(0x7ff67006c6e0 msgr2=0x7ff670157d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:02.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.408+0000 7ff674ff5640 1 -- 172.21.15.125:0/2879680442 shutdown_connections 2023-10-30T12:37:02.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:02.408+0000 7ff674ff5640 1 -- 172.21.15.125:0/2879680442 wait complete. 2023-10-30T12:37:02.437 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T12:37:02.438 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 47 secs before thrashing 2023-10-30T12:37:02.652 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:02.667 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:37:02.668 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:37:03.057 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:37:03.072 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:37:03.072 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:37:03.073 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:37:03.073 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:37:03.073 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:37:03.073 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:37:03.074 INFO:tasks.workunit.client.0.smithi125.stdout: 50725 ? 00:00:00 ps 2023-10-30T12:37:03.074 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:03.097 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:37:03.101 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:37:03.115 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:37:03.120 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:03.122 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:37:03.144 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:03.158 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:37:03.159 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:37:03.222 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:37:03.238 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:37:03.238 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:37:03.239 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:37:03.239 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:37:03.239 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:37:03.240 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:37:03.240 INFO:tasks.workunit.client.0.smithi125.stdout: 50735 ? 00:00:00 ps 2023-10-30T12:37:03.241 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:03.262 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:37:03.279 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:37:03.317 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:37:03.323 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:03.325 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:37:03.371 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:03.578 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:37:03.579 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:37:03.731 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:37:03.746 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:37:03.746 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:37:03.746 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:37:03.746 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:37:03.746 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:37:03.747 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:37:03.747 INFO:tasks.workunit.client.0.smithi125.stdout: 50745 ? 00:00:00 ps 2023-10-30T12:37:03.747 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:03.762 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:37:03.765 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:37:03.790 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a 2023-10-30T12:37:03.795 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/one 2023-10-30T12:37:03.797 INFO:tasks.workunit.client.0.smithi125.stderr:+ cat 2023-10-30T12:37:04.346 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir a/.snap/two 2023-10-30T12:37:04.359 INFO:tasks.workunit.client.0.smithi125.stderr:+ chmod 777 a/foo 2023-10-30T12:37:04.360 INFO:tasks.workunit.client.0.smithi125.stderr:+ sync 2023-10-30T12:37:04.456 INFO:tasks.workunit.client.0.smithi125.stderr:+ ps 2023-10-30T12:37:04.470 INFO:tasks.workunit.client.0.smithi125.stdout: PID TTY TIME CMD 2023-10-30T12:37:04.470 INFO:tasks.workunit.client.0.smithi125.stdout: 1121 ? 00:00:00 systemd 2023-10-30T12:37:04.470 INFO:tasks.workunit.client.0.smithi125.stdout: 1130 ? 00:00:00 (sd-pam) 2023-10-30T12:37:04.470 INFO:tasks.workunit.client.0.smithi125.stdout: 27984 ? 00:00:01 sshd 2023-10-30T12:37:04.471 INFO:tasks.workunit.client.0.smithi125.stdout: 49950 ? 00:00:00 timeout 2023-10-30T12:37:04.471 INFO:tasks.workunit.client.0.smithi125.stdout: 49953 ? 00:00:00 snaptest-double 2023-10-30T12:37:04.471 INFO:tasks.workunit.client.0.smithi125.stdout: 50755 ? 00:00:00 ps 2023-10-30T12:37:04.472 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a/.snap/one a/.snap/two 2023-10-30T12:37:04.494 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm a/foo 2023-10-30T12:37:04.498 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir a 2023-10-30T12:37:04.511 INFO:tasks.workunit.client.0.smithi125.stdout:OK 2023-10-30T12:37:04.512 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo OK 2023-10-30T12:37:04.512 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:37:04.513 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:37:04.549 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2023-10-30T12:37:04.550 DEBUG:teuthology.orchestra.run.smithi125:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2023-10-30T12:37:04.612 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/foo 2023-10-30T12:37:04.629 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo We want ENOENT, not ESTALE, here. 2023-10-30T12:37:04.629 INFO:tasks.workunit.client.0.smithi125.stdout:We want ENOENT, not ESTALE, here. 2023-10-30T12:37:04.630 INFO:tasks.workunit.client.0.smithi125.stderr:+ seq 1 100 2023-10-30T12:37:04.631 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/1 2023-10-30T12:37:04.631 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.635 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/1': No such file or directory 2023-10-30T12:37:04.636 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/2 2023-10-30T12:37:04.636 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.637 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/2': No such file or directory 2023-10-30T12:37:04.638 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/3 2023-10-30T12:37:04.639 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.640 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/3': No such file or directory 2023-10-30T12:37:04.641 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/4 2023-10-30T12:37:04.641 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.643 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/4': No such file or directory 2023-10-30T12:37:04.644 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/5 2023-10-30T12:37:04.644 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.645 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/5': No such file or directory 2023-10-30T12:37:04.646 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/6 2023-10-30T12:37:04.647 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.648 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/6': No such file or directory 2023-10-30T12:37:04.649 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/7 2023-10-30T12:37:04.649 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.650 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/7': No such file or directory 2023-10-30T12:37:04.651 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/8 2023-10-30T12:37:04.652 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.653 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/8': No such file or directory 2023-10-30T12:37:04.654 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/9 2023-10-30T12:37:04.654 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.656 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/9': No such file or directory 2023-10-30T12:37:04.657 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/10 2023-10-30T12:37:04.657 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.659 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/10': No such file or directory 2023-10-30T12:37:04.659 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/11 2023-10-30T12:37:04.660 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.661 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/11': No such file or directory 2023-10-30T12:37:04.662 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/12 2023-10-30T12:37:04.663 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.664 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/12': No such file or directory 2023-10-30T12:37:04.665 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/13 2023-10-30T12:37:04.665 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.666 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/13': No such file or directory 2023-10-30T12:37:04.667 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/14 2023-10-30T12:37:04.668 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.669 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/14': No such file or directory 2023-10-30T12:37:04.670 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/15 2023-10-30T12:37:04.670 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.673 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/15': No such file or directory 2023-10-30T12:37:04.674 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/16 2023-10-30T12:37:04.674 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.675 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/16': No such file or directory 2023-10-30T12:37:04.675 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/17 2023-10-30T12:37:04.676 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.677 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/17': No such file or directory 2023-10-30T12:37:04.678 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/18 2023-10-30T12:37:04.678 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.679 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/18': No such file or directory 2023-10-30T12:37:04.680 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/19 2023-10-30T12:37:04.680 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.682 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/19': No such file or directory 2023-10-30T12:37:04.683 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/20 2023-10-30T12:37:04.683 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.685 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/20': No such file or directory 2023-10-30T12:37:04.685 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/21 2023-10-30T12:37:04.686 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.687 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/21': No such file or directory 2023-10-30T12:37:04.688 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/22 2023-10-30T12:37:04.689 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.690 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/22': No such file or directory 2023-10-30T12:37:04.691 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/23 2023-10-30T12:37:04.691 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.692 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/23': No such file or directory 2023-10-30T12:37:04.693 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/24 2023-10-30T12:37:04.693 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.695 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/24': No such file or directory 2023-10-30T12:37:04.696 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/25 2023-10-30T12:37:04.696 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.698 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/25': No such file or directory 2023-10-30T12:37:04.699 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/26 2023-10-30T12:37:04.699 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.700 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/26': No such file or directory 2023-10-30T12:37:04.701 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/27 2023-10-30T12:37:04.701 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.703 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/27': No such file or directory 2023-10-30T12:37:04.704 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/28 2023-10-30T12:37:04.704 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.706 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/28': No such file or directory 2023-10-30T12:37:04.707 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/29 2023-10-30T12:37:04.707 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.708 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/29': No such file or directory 2023-10-30T12:37:04.709 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/30 2023-10-30T12:37:04.709 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.711 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/30': No such file or directory 2023-10-30T12:37:04.712 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/31 2023-10-30T12:37:04.713 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.714 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/31': No such file or directory 2023-10-30T12:37:04.715 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/32 2023-10-30T12:37:04.715 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.716 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/32': No such file or directory 2023-10-30T12:37:04.717 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/33 2023-10-30T12:37:04.718 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.719 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/33': No such file or directory 2023-10-30T12:37:04.720 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/34 2023-10-30T12:37:04.720 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.722 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/34': No such file or directory 2023-10-30T12:37:04.723 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/35 2023-10-30T12:37:04.723 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.724 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/35': No such file or directory 2023-10-30T12:37:04.725 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/36 2023-10-30T12:37:04.726 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.727 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/36': No such file or directory 2023-10-30T12:37:04.728 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/37 2023-10-30T12:37:04.728 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.730 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/37': No such file or directory 2023-10-30T12:37:04.731 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/38 2023-10-30T12:37:04.731 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.733 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/38': No such file or directory 2023-10-30T12:37:04.734 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/39 2023-10-30T12:37:04.734 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.735 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/39': No such file or directory 2023-10-30T12:37:04.736 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/40 2023-10-30T12:37:04.736 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.738 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/40': No such file or directory 2023-10-30T12:37:04.739 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/41 2023-10-30T12:37:04.739 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.741 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/41': No such file or directory 2023-10-30T12:37:04.741 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/42 2023-10-30T12:37:04.742 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.743 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/42': No such file or directory 2023-10-30T12:37:04.744 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/43 2023-10-30T12:37:04.744 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.746 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/43': No such file or directory 2023-10-30T12:37:04.747 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/44 2023-10-30T12:37:04.747 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.748 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/44': No such file or directory 2023-10-30T12:37:04.749 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/45 2023-10-30T12:37:04.749 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.751 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/45': No such file or directory 2023-10-30T12:37:04.752 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/46 2023-10-30T12:37:04.752 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.753 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/46': No such file or directory 2023-10-30T12:37:04.754 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/47 2023-10-30T12:37:04.755 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.756 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/47': No such file or directory 2023-10-30T12:37:04.757 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/48 2023-10-30T12:37:04.757 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.759 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/48': No such file or directory 2023-10-30T12:37:04.759 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/49 2023-10-30T12:37:04.760 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.761 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/49': No such file or directory 2023-10-30T12:37:04.762 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/50 2023-10-30T12:37:04.762 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.764 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/50': No such file or directory 2023-10-30T12:37:04.765 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/51 2023-10-30T12:37:04.765 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.766 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/51': No such file or directory 2023-10-30T12:37:04.767 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/52 2023-10-30T12:37:04.768 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.769 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/52': No such file or directory 2023-10-30T12:37:04.770 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/53 2023-10-30T12:37:04.771 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.772 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/53': No such file or directory 2023-10-30T12:37:04.773 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/54 2023-10-30T12:37:04.773 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.775 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/54': No such file or directory 2023-10-30T12:37:04.776 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/55 2023-10-30T12:37:04.776 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.779 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/55': No such file or directory 2023-10-30T12:37:04.780 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/56 2023-10-30T12:37:04.780 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.781 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/56': No such file or directory 2023-10-30T12:37:04.782 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/57 2023-10-30T12:37:04.782 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.783 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/57': No such file or directory 2023-10-30T12:37:04.784 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/58 2023-10-30T12:37:04.784 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.786 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/58': No such file or directory 2023-10-30T12:37:04.787 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/59 2023-10-30T12:37:04.787 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.788 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/59': No such file or directory 2023-10-30T12:37:04.789 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/60 2023-10-30T12:37:04.789 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.791 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/60': No such file or directory 2023-10-30T12:37:04.792 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/61 2023-10-30T12:37:04.792 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.793 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/61': No such file or directory 2023-10-30T12:37:04.795 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/62 2023-10-30T12:37:04.795 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.796 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/62': No such file or directory 2023-10-30T12:37:04.797 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/63 2023-10-30T12:37:04.797 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.799 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/63': No such file or directory 2023-10-30T12:37:04.800 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/64 2023-10-30T12:37:04.800 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.801 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/64': No such file or directory 2023-10-30T12:37:04.802 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/65 2023-10-30T12:37:04.802 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.804 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/65': No such file or directory 2023-10-30T12:37:04.805 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/66 2023-10-30T12:37:04.805 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.806 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/66': No such file or directory 2023-10-30T12:37:04.807 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/67 2023-10-30T12:37:04.808 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.809 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/67': No such file or directory 2023-10-30T12:37:04.810 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/68 2023-10-30T12:37:04.810 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.812 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/68': No such file or directory 2023-10-30T12:37:04.813 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/69 2023-10-30T12:37:04.813 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.814 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/69': No such file or directory 2023-10-30T12:37:04.815 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/70 2023-10-30T12:37:04.816 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.817 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/70': No such file or directory 2023-10-30T12:37:04.818 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/71 2023-10-30T12:37:04.818 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.820 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/71': No such file or directory 2023-10-30T12:37:04.822 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/72 2023-10-30T12:37:04.822 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.822 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/72': No such file or directory 2023-10-30T12:37:04.823 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/73 2023-10-30T12:37:04.823 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.825 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/73': No such file or directory 2023-10-30T12:37:04.825 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/74 2023-10-30T12:37:04.826 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.827 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/74': No such file or directory 2023-10-30T12:37:04.828 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/75 2023-10-30T12:37:04.829 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.830 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/75': No such file or directory 2023-10-30T12:37:04.831 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/76 2023-10-30T12:37:04.831 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.833 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/76': No such file or directory 2023-10-30T12:37:04.834 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/77 2023-10-30T12:37:04.834 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.835 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/77': No such file or directory 2023-10-30T12:37:04.836 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/78 2023-10-30T12:37:04.836 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.838 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/78': No such file or directory 2023-10-30T12:37:04.839 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/79 2023-10-30T12:37:04.839 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.840 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/79': No such file or directory 2023-10-30T12:37:04.841 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/80 2023-10-30T12:37:04.842 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.843 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/80': No such file or directory 2023-10-30T12:37:04.844 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/81 2023-10-30T12:37:04.845 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.846 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/81': No such file or directory 2023-10-30T12:37:04.847 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/82 2023-10-30T12:37:04.847 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.849 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/82': No such file or directory 2023-10-30T12:37:04.849 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/83 2023-10-30T12:37:04.850 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.851 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/83': No such file or directory 2023-10-30T12:37:04.852 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/84 2023-10-30T12:37:04.852 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.854 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/84': No such file or directory 2023-10-30T12:37:04.855 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/85 2023-10-30T12:37:04.855 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.856 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/85': No such file or directory 2023-10-30T12:37:04.857 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/86 2023-10-30T12:37:04.857 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.859 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/86': No such file or directory 2023-10-30T12:37:04.860 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/87 2023-10-30T12:37:04.860 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.861 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/87': No such file or directory 2023-10-30T12:37:04.862 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/88 2023-10-30T12:37:04.863 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.864 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/88': No such file or directory 2023-10-30T12:37:04.865 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/89 2023-10-30T12:37:04.865 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.867 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/89': No such file or directory 2023-10-30T12:37:04.868 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/90 2023-10-30T12:37:04.868 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.869 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/90': No such file or directory 2023-10-30T12:37:04.870 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/91 2023-10-30T12:37:04.871 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.872 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/91': No such file or directory 2023-10-30T12:37:04.873 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/92 2023-10-30T12:37:04.873 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.875 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/92': No such file or directory 2023-10-30T12:37:04.875 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/93 2023-10-30T12:37:04.876 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.877 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/93': No such file or directory 2023-10-30T12:37:04.878 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/94 2023-10-30T12:37:04.878 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.881 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/94': No such file or directory 2023-10-30T12:37:04.882 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/95 2023-10-30T12:37:04.882 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.883 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/95': No such file or directory 2023-10-30T12:37:04.883 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/96 2023-10-30T12:37:04.884 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.885 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/96': No such file or directory 2023-10-30T12:37:04.886 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/97 2023-10-30T12:37:04.887 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.887 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/97': No such file or directory 2023-10-30T12:37:04.888 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/98 2023-10-30T12:37:04.888 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.890 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/98': No such file or directory 2023-10-30T12:37:04.891 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/99 2023-10-30T12:37:04.891 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.892 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/99': No such file or directory 2023-10-30T12:37:04.893 INFO:tasks.workunit.client.0.smithi125.stderr:+ stat .snap/foo/100 2023-10-30T12:37:04.894 INFO:tasks.workunit.client.0.smithi125.stderr:+ grep No such file 2023-10-30T12:37:04.895 INFO:tasks.workunit.client.0.smithi125.stdout:stat: cannot statx '.snap/foo/100': No such file or directory 2023-10-30T12:37:04.896 INFO:tasks.workunit.client.0.smithi125.stderr:+ rmdir .snap/foo 2023-10-30T12:37:04.909 INFO:tasks.workunit.client.0.smithi125.stdout:OK 2023-10-30T12:37:04.910 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo OK 2023-10-30T12:37:04.911 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-10-30T12:37:04.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-30T12:37:04.988 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2023-10-30T12:37:04.988 DEBUG:teuthology.orchestra.run.smithi125:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2023-10-30T12:37:05.052 INFO:tasks.workunit.client.0.smithi125.stderr:+ set -e 2023-10-30T12:37:05.053 INFO:tasks.workunit.client.0.smithi125.stderr:+ sudo git config --global http.sslVerify false 2023-10-30T12:37:05.070 INFO:tasks.workunit.client.0.smithi125.stderr:+ sudo git config --global http.postBuffer 1048576000 2023-10-30T12:37:05.089 INFO:tasks.workunit.client.0.smithi125.stderr:+ retried=false 2023-10-30T12:37:05.089 INFO:tasks.workunit.client.0.smithi125.stderr:+ trap -- retry EXIT 2023-10-30T12:37:05.090 INFO:tasks.workunit.client.0.smithi125.stderr:+ rm -rf ceph 2023-10-30T12:37:05.090 INFO:tasks.workunit.client.0.smithi125.stderr:+ timeout 1800 git clone https://git.ceph.com/ceph.git 2023-10-30T12:37:05.103 INFO:tasks.workunit.client.0.smithi125.stderr:Cloning into 'ceph'... 2023-10-30T12:37:14.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:37:14.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:37:45.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:37:45.021 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:37:49.439 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:49.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.668+0000 7f950b2d6640 1 Processor -- start 2023-10-30T12:37:49.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.668+0000 7f950b2d6640 1 -- start start 2023-10-30T12:37:49.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f950b2d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95040c05d0 0x7f950413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:49.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f950b2d6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f95040c6f60 0x7f950413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:49.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f950b2d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95040c0a50 con 0x7f95040c6b80 2023-10-30T12:37:49.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f950b2d6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f950406f6e0 con 0x7f95040c6f60 2023-10-30T12:37:49.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f950b2d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f950406f9c0 con 0x7f95040c05d0 2023-10-30T12:37:49.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f9509873640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f95040c6f60 0x7f950413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:49.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f9509873640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f95040c6f60 0x7f950413f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39556/0 (socket says 172.21.15.125:39556) 2023-10-30T12:37:49.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f9509873640 1 -- 172.21.15.125:0/457482129 learned_addr learned my addr 172.21.15.125:0/457482129 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:37:49.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f9508871640 1 --2- 172.21.15.125:0/457482129 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95040c05d0 0x7f950413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:49.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f94fbfff640 1 -- 172.21.15.125:0/457482129 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3325663889 0 0) 0x7f95040c0a50 con 0x7f95040c6b80 2023-10-30T12:37:49.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f94fbfff640 1 -- 172.21.15.125:0/457482129 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94f0003530 con 0x7f95040c6b80 2023-10-30T12:37:49.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f9509873640 1 -- 172.21.15.125:0/457482129 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95040c05d0 msgr2=0x7f950413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:49.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f9509873640 1 --2- 172.21.15.125:0/457482129 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95040c05d0 0x7f950413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:49.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f9509873640 1 -- 172.21.15.125:0/457482129 >> v1:172.21.15.125:6789/0 conn(0x7f95040c6b80 legacy=0x7f95040bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:49.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.672+0000 7f9509873640 1 -- 172.21.15.125:0/457482129 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950406fca0 con 0x7f95040c6f60 2023-10-30T12:37:49.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.676+0000 7f9509873640 1 --2- 172.21.15.125:0/457482129 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040c6f60 0x7f950413f2d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f94fc009e50 tx=0x7f94fc01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d20fce98cf82c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:49.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.680+0000 7f94fbfff640 1 -- 172.21.15.125:0/457482129 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc0a57f0 con 0x7f95040c6f60 2023-10-30T12:37:49.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.680+0000 7f94fbfff640 1 -- 172.21.15.125:0/457482129 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94fc0a5950 con 0x7f95040c6f60 2023-10-30T12:37:49.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.680+0000 7f94fbfff640 1 -- 172.21.15.125:0/457482129 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94fc0a5cc0 con 0x7f95040c6f60 2023-10-30T12:37:49.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.680+0000 7f950b2d6640 1 -- 172.21.15.125:0/457482129 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040c6f60 msgr2=0x7f950413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:49.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.680+0000 7f950b2d6640 1 --2- 172.21.15.125:0/457482129 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040c6f60 0x7f950413f2d0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f94fc009e50 tx=0x7f94fc01ca10 comp rx=0 tx=0).stop 2023-10-30T12:37:49.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.680+0000 7f950b2d6640 1 -- 172.21.15.125:0/457482129 shutdown_connections 2023-10-30T12:37:49.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.680+0000 7f950b2d6640 1 --2- 172.21.15.125:0/457482129 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040c6f60 0x7f950413f2d0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:49.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.680+0000 7f950b2d6640 1 --2- 172.21.15.125:0/457482129 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95040c05d0 0x7f950413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:49.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.680+0000 7f950b2d6640 1 -- 172.21.15.125:0/457482129 >> 172.21.15.125:0/457482129 conn(0x7f95040c7910 msgr2=0x7f95040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:49.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.680+0000 7f950b2d6640 1 -- 172.21.15.125:0/457482129 shutdown_connections 2023-10-30T12:37:49.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.680+0000 7f950b2d6640 1 -- 172.21.15.125:0/457482129 wait complete. 2023-10-30T12:37:49.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f950b2d6640 1 Processor -- start 2023-10-30T12:37:49.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f950b2d6640 1 -- start start 2023-10-30T12:37:49.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f950b2d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95040c05d0 0x7f9504133000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:49.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f950b2d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040c6b80 0x7f9504132790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:49.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f950b2d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95040c6f60 0x7f950413cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:49.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f950b2d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95041f5ff0 con 0x7f95040c6b80 2023-10-30T12:37:49.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f950b2d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95040c0400 con 0x7f95040c6f60 2023-10-30T12:37:49.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f950b2d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9504162090 con 0x7f95040c05d0 2023-10-30T12:37:49.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f9508871640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040c6b80 0x7f9504132790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:49.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f9508871640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040c6b80 0x7f9504132790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39572/0 (socket says 172.21.15.125:39572) 2023-10-30T12:37:49.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f9508871640 1 -- 172.21.15.125:0/1987413513 learned_addr learned my addr 172.21.15.125:0/1987413513 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:37:49.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f9509072640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95040c05d0 0x7f9504133000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:49.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f9509873640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95040c6f60 0x7f950413cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:49.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f9508871640 1 -- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95040c05d0 msgr2=0x7f9504133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:49.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f9508871640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95040c05d0 0x7f9504133000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:49.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f9508871640 1 -- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95040c6f60 msgr2=0x7f950413cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:49.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f9508871640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95040c6f60 0x7f950413cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:49.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f9508871640 1 -- 172.21.15.125:0/1987413513 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc009b30 con 0x7f95040c6b80 2023-10-30T12:37:49.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f9509072640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95040c05d0 0x7f9504133000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:37:49.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f9509873640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95040c6f60 0x7f950413cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:37:49.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.684+0000 7f9508871640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040c6b80 0x7f9504132790 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f94f400e930 tx=0x7f94f400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:49.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.688+0000 7f94f9ffb640 1 -- 172.21.15.125:0/1987413513 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4009690 con 0x7f95040c6b80 2023-10-30T12:37:49.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.688+0000 7f94f9ffb640 1 -- 172.21.15.125:0/1987413513 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f400ca00 con 0x7f95040c6b80 2023-10-30T12:37:49.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.688+0000 7f950b2d6640 1 -- 172.21.15.125:0/1987413513 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9504162370 con 0x7f95040c6b80 2023-10-30T12:37:49.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.688+0000 7f94f9ffb640 1 -- 172.21.15.125:0/1987413513 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f400cd70 con 0x7f95040c6b80 2023-10-30T12:37:49.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.688+0000 7f950b2d6640 1 -- 172.21.15.125:0/1987413513 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950413ca50 con 0x7f95040c6b80 2023-10-30T12:37:49.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.688+0000 7f950b2d6640 1 -- 172.21.15.125:0/1987413513 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95041625b0 con 0x7f95040c6b80 2023-10-30T12:37:49.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.692+0000 7f94f9ffb640 1 -- 172.21.15.125:0/1987413513 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94f4013470 con 0x7f95040c6b80 2023-10-30T12:37:49.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.692+0000 7f94f9ffb640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94e80664a0 0x7f94e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:49.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.692+0000 7f94f9ffb640 1 -- 172.21.15.125:0/1987413513 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f94f4084280 con 0x7f95040c6b80 2023-10-30T12:37:49.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.692+0000 7f9509072640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94e80664a0 0x7f94e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:49.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.692+0000 7f9509072640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94e80664a0 0x7f94e8068960 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9500001740 tx=0x7f9500009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:49.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:49.700+0000 7f94f9ffb640 1 -- 172.21.15.125:0/1987413513 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f95041625b0 con 0x7f95040c6b80 2023-10-30T12:37:50.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.076+0000 7f950b2d6640 1 -- 172.21.15.125:0/1987413513 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9504162560 con 0x7f95040c6b80 2023-10-30T12:37:50.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.076+0000 7f94f9ffb640 1 -- 172.21.15.125:0/1987413513 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f94f4051610 con 0x7f95040c6b80 2023-10-30T12:37:50.081 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:37:50.081 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:37:18.218892+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:active","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:50.082 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 52 2023-10-30T12:37:50.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.080+0000 7f950b2d6640 1 -- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94e80664a0 msgr2=0x7f94e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:50.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.080+0000 7f950b2d6640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94e80664a0 0x7f94e8068960 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9500001740 tx=0x7f9500009040 comp rx=0 tx=0).stop 2023-10-30T12:37:50.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.080+0000 7f950b2d6640 1 -- 172.21.15.125:0/1987413513 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040c6b80 msgr2=0x7f9504132790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:50.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.080+0000 7f950b2d6640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040c6b80 0x7f9504132790 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f94f400e930 tx=0x7f94f400edf0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.080+0000 7f950b2d6640 1 -- 172.21.15.125:0/1987413513 shutdown_connections 2023-10-30T12:37:50.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.080+0000 7f950b2d6640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94e80664a0 0x7f94e8068960 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.080+0000 7f950b2d6640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95040c6f60 0x7f950413cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.080+0000 7f950b2d6640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95040c6b80 0x7f9504132790 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.080+0000 7f950b2d6640 1 --2- 172.21.15.125:0/1987413513 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95040c05d0 0x7f9504133000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.080+0000 7f950b2d6640 1 -- 172.21.15.125:0/1987413513 >> 172.21.15.125:0/1987413513 conn(0x7f95040c7910 msgr2=0x7f9504133e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:50.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.084+0000 7f950b2d6640 1 -- 172.21.15.125:0/1987413513 shutdown_connections 2023-10-30T12:37:50.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.084+0000 7f950b2d6640 1 -- 172.21.15.125:0/1987413513 wait complete. 2023-10-30T12:37:50.114 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 3 -> 2 2023-10-30T12:37:50.115 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 2 2023-10-30T12:37:50.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.348+0000 7f0fed4f7640 1 Processor -- start 2023-10-30T12:37:50.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.348+0000 7f0fed4f7640 1 -- start start 2023-10-30T12:37:50.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fed4f7640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0fe80ca940 0x7f0fe815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fed4f7640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0fe815d700 0x7f0fe8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fed4f7640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fe815a620 con 0x7f0fe815cbd0 2023-10-30T12:37:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fed4f7640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fe8153e90 con 0x7f0fe80ca940 2023-10-30T12:37:50.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fed4f7640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe806e0a0 con 0x7f0fe815d700 2023-10-30T12:37:50.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fe6ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0fe80ca940 0x7f0fe815a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:50.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fe67fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0fe815cbd0 0x7f0fe815cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45900/0 (socket says 172.21.15.125:45900) 2023-10-30T12:37:50.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fe67fc640 1 -- 172.21.15.125:0/3224298660 learned_addr learned my addr 172.21.15.125:0/3224298660 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:37:50.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fe7fff640 1 --2- 172.21.15.125:0/3224298660 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0fe815d700 0x7f0fe8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:50.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fe5ffb640 1 -- 172.21.15.125:0/3224298660 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205649824 0 0) 0x7f0fdc001120 con 0x7f0fe815cbd0 2023-10-30T12:37:50.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fe5ffb640 1 -- 172.21.15.125:0/3224298660 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd0003530 con 0x7f0fe815cbd0 2023-10-30T12:37:50.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fe7fff640 1 -- 172.21.15.125:0/3224298660 >> v2:172.21.15.136:3300/0 conn(0x7f0fe80ca940 msgr2=0x7f0fe815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:50.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fe7fff640 1 --2- 172.21.15.125:0/3224298660 >> v2:172.21.15.136:3300/0 conn(0x7f0fe80ca940 0x7f0fe815a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fe7fff640 1 -- 172.21.15.125:0/3224298660 >> v1:172.21.15.125:6789/0 conn(0x7f0fe815cbd0 legacy=0x7f0fe815cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:50.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.352+0000 7f0fe7fff640 1 -- 172.21.15.125:0/3224298660 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe806e340 con 0x7f0fe815d700 2023-10-30T12:37:50.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.356+0000 7f0fe7fff640 1 --2- 172.21.15.125:0/3224298660 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0fe815d700 0x7f0fe8069e90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0009d90 tx=0x7f0fe001c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b30efe336e6c9a3f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:50.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.356+0000 7f0fe5ffb640 1 -- 172.21.15.125:0/3224298660 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe00a5960 con 0x7f0fe815d700 2023-10-30T12:37:50.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.356+0000 7f0fe5ffb640 1 -- 172.21.15.125:0/3224298660 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe00a5ac0 con 0x7f0fe815d700 2023-10-30T12:37:50.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.356+0000 7f0fe5ffb640 1 -- 172.21.15.125:0/3224298660 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe00a5e30 con 0x7f0fe815d700 2023-10-30T12:37:50.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.356+0000 7f0fed4f7640 1 -- 172.21.15.125:0/3224298660 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0fe815d700 msgr2=0x7f0fe8069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:50.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.356+0000 7f0fed4f7640 1 --2- 172.21.15.125:0/3224298660 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0fe815d700 0x7f0fe8069e90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0fe0009d90 tx=0x7f0fe001c8e0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.360+0000 7f0fed4f7640 1 -- 172.21.15.125:0/3224298660 shutdown_connections 2023-10-30T12:37:50.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.360+0000 7f0fed4f7640 1 --2- 172.21.15.125:0/3224298660 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0fe815d700 0x7f0fe8069e90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.360+0000 7f0fed4f7640 1 --2- 172.21.15.125:0/3224298660 >> v2:172.21.15.136:3300/0 conn(0x7f0fe80ca940 0x7f0fe815a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.360+0000 7f0fed4f7640 1 -- 172.21.15.125:0/3224298660 >> 172.21.15.125:0/3224298660 conn(0x7f0fe80cd270 msgr2=0x7f0fe80cd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:50.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.360+0000 7f0fed4f7640 1 -- 172.21.15.125:0/3224298660 shutdown_connections 2023-10-30T12:37:50.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.360+0000 7f0fed4f7640 1 -- 172.21.15.125:0/3224298660 wait complete. 2023-10-30T12:37:50.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fed4f7640 1 Processor -- start 2023-10-30T12:37:50.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fed4f7640 1 -- start start 2023-10-30T12:37:50.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fed4f7640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0fe80ca940 0x7f0fe8153790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:50.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fed4f7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0fe815cbd0 0x7f0fe80c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:50.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fed4f7640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0fe815d700 0x7f0fe80c92c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:50.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fed4f7640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe815c910 con 0x7f0fe815d700 2023-10-30T12:37:50.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fed4f7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe80c72c0 con 0x7f0fe815cbd0 2023-10-30T12:37:50.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fed4f7640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fe815c670 con 0x7f0fe80ca940 2023-10-30T12:37:50.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fe7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0fe815d700 0x7f0fe80c92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:50.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fe6ffd640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0fe80ca940 0x7f0fe8153790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:50.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fe7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0fe815d700 0x7f0fe80c92c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39576/0 (socket says 172.21.15.125:39576) 2023-10-30T12:37:50.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fe7fff640 1 -- 172.21.15.125:0/1664407263 learned_addr learned my addr 172.21.15.125:0/1664407263 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:37:50.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fe67fc640 1 --2- 172.21.15.125:0/1664407263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0fe815cbd0 0x7f0fe80c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:50.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fe7fff640 1 -- 172.21.15.125:0/1664407263 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0fe80ca940 msgr2=0x7f0fe8153790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:50.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fe7fff640 1 --2- 172.21.15.125:0/1664407263 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0fe80ca940 0x7f0fe8153790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fe7fff640 1 -- 172.21.15.125:0/1664407263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0fe815cbd0 msgr2=0x7f0fe80c8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:50.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fe7fff640 1 --2- 172.21.15.125:0/1664407263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0fe815cbd0 0x7f0fe80c8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:50.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.364+0000 7f0fe7fff640 1 -- 172.21.15.125:0/1664407263 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fe0009a70 con 0x7f0fe815d700 2023-10-30T12:37:50.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.368+0000 7f0fe7fff640 1 --2- 172.21.15.125:0/1664407263 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0fe815d700 0x7f0fe80c92c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0fe001af90 tx=0x7f0fe0019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:50.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.368+0000 7f0fcf7fe640 1 -- 172.21.15.125:0/1664407263 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0007000 con 0x7f0fe815d700 2023-10-30T12:37:50.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.368+0000 7f0fcf7fe640 1 -- 172.21.15.125:0/1664407263 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fe00aed30 con 0x7f0fe815d700 2023-10-30T12:37:50.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.368+0000 7f0fcf7fe640 1 -- 172.21.15.125:0/1664407263 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fe00aeea0 con 0x7f0fe815d700 2023-10-30T12:37:50.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.368+0000 7f0fed4f7640 1 -- 172.21.15.125:0/1664407263 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fe815b350 con 0x7f0fe815d700 2023-10-30T12:37:50.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.368+0000 7f0fed4f7640 1 -- 172.21.15.125:0/1664407263 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fe8151820 con 0x7f0fe815d700 2023-10-30T12:37:50.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.368+0000 7f0fed4f7640 1 -- 172.21.15.125:0/1664407263 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fe8151a10 con 0x7f0fe815d700 2023-10-30T12:37:50.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.368+0000 7f0fcf7fe640 1 -- 172.21.15.125:0/1664407263 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0fe00b4050 con 0x7f0fe815d700 2023-10-30T12:37:50.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.372+0000 7f0fcf7fe640 1 --2- 172.21.15.125:0/1664407263 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0fd40664a0 0x7f0fd4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:50.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.372+0000 7f0fcf7fe640 1 -- 172.21.15.125:0/1664407263 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f0fe011f100 con 0x7f0fe815d700 2023-10-30T12:37:50.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.372+0000 7f0fe6ffd640 1 --2- 172.21.15.125:0/1664407263 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0fd40664a0 0x7f0fd4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:50.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.372+0000 7f0fe6ffd640 1 --2- 172.21.15.125:0/1664407263 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0fd40664a0 0x7f0fd4068960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8009ac0 tx=0x7f0fd8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:50.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.380+0000 7f0fcf7fe640 1 -- 172.21.15.125:0/1664407263 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0fe00ec360 con 0x7f0fe815d700 2023-10-30T12:37:50.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:50.760+0000 7f0fed4f7640 1 -- 172.21.15.125:0/1664407263 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"} v 0) v1 -- 0x7f0fe8151a10 con 0x7f0fe815d700 2023-10-30T12:37:51.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.232+0000 7f0fcf7fe640 1 -- 172.21.15.125:0/1664407263 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "2"}]=0 v53) v1 ==== 105+0+0 (secure 0 0 0) 0x7f0fe00b20b0 con 0x7f0fe815d700 2023-10-30T12:37:51.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.236+0000 7f0fed4f7640 1 -- 172.21.15.125:0/1664407263 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0fd40664a0 msgr2=0x7f0fd4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:51.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.236+0000 7f0fed4f7640 1 --2- 172.21.15.125:0/1664407263 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0fd40664a0 0x7f0fd4068960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0fd8009ac0 tx=0x7f0fd8009190 comp rx=0 tx=0).stop 2023-10-30T12:37:51.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.236+0000 7f0fed4f7640 1 -- 172.21.15.125:0/1664407263 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0fe815d700 msgr2=0x7f0fe80c92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:51.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.236+0000 7f0fed4f7640 1 --2- 172.21.15.125:0/1664407263 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0fe815d700 0x7f0fe80c92c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f0fe001af90 tx=0x7f0fe0019680 comp rx=0 tx=0).stop 2023-10-30T12:37:51.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.236+0000 7f0fed4f7640 1 -- 172.21.15.125:0/1664407263 shutdown_connections 2023-10-30T12:37:51.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.236+0000 7f0fed4f7640 1 --2- 172.21.15.125:0/1664407263 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0fd40664a0 0x7f0fd4068960 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:51.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.236+0000 7f0fed4f7640 1 --2- 172.21.15.125:0/1664407263 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0fe815d700 0x7f0fe80c92c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:51.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.236+0000 7f0fed4f7640 1 --2- 172.21.15.125:0/1664407263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0fe815cbd0 0x7f0fe80c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:51.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.236+0000 7f0fed4f7640 1 --2- 172.21.15.125:0/1664407263 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0fe80ca940 0x7f0fe8153790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:51.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.236+0000 7f0fed4f7640 1 -- 172.21.15.125:0/1664407263 >> 172.21.15.125:0/1664407263 conn(0x7f0fe80cd270 msgr2=0x7f0fe80cb6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:51.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.236+0000 7f0fed4f7640 1 -- 172.21.15.125:0/1664407263 shutdown_connections 2023-10-30T12:37:51.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.236+0000 7f0fed4f7640 1 -- 172.21.15.125:0/1664407263 wait complete. 2023-10-30T12:37:51.270 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:37:51.270 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:51.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.496+0000 7efd82d62640 1 Processor -- start 2023-10-30T12:37:51.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd82d62640 1 -- start start 2023-10-30T12:37:51.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd82d62640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efd7c1538f0 0x7efd7c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:51.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd82d62640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7efd7c15cd50 0x7efd7c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:51.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd82d62640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd7c15d300 con 0x7efd7c0cefa0 2023-10-30T12:37:51.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd82d62640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd7c15d5e0 con 0x7efd7c1538f0 2023-10-30T12:37:51.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd82d62640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd7c15d8c0 con 0x7efd7c15cd50 2023-10-30T12:37:51.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd81d60640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efd7c1538f0 0x7efd7c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:51.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd81d60640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efd7c1538f0 0x7efd7c159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56142/0 (socket says 172.21.15.125:56142) 2023-10-30T12:37:51.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd81d60640 1 -- 172.21.15.125:0/2602265595 learned_addr learned my addr 172.21.15.125:0/2602265595 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:37:51.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd8155f640 1 --2- 172.21.15.125:0/2602265595 >> v2:172.21.15.136:3300/0 conn(0x7efd7c15cd50 0x7efd7c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:51.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd81d60640 1 -- 172.21.15.125:0/2602265595 >> v2:172.21.15.136:3300/0 conn(0x7efd7c15cd50 msgr2=0x7efd7c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:51.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd81d60640 1 --2- 172.21.15.125:0/2602265595 >> v2:172.21.15.136:3300/0 conn(0x7efd7c15cd50 0x7efd7c0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:51.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd81d60640 1 -- 172.21.15.125:0/2602265595 >> v1:172.21.15.125:6789/0 conn(0x7efd7c0cefa0 legacy=0x7efd7c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:51.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd81d60640 1 -- 172.21.15.125:0/2602265595 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd7c15dba0 con 0x7efd7c1538f0 2023-10-30T12:37:51.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.500+0000 7efd80d5e640 1 -- 172.21.15.125:0/2602265595 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270988924 0 0) 0x7efd7c15d300 con 0x7efd7c0cefa0 2023-10-30T12:37:51.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.508+0000 7efd81d60640 1 --2- 172.21.15.125:0/2602265595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd7c1538f0 0x7efd7c159fb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7efd7c072e90 tx=0x7efd7801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c09b0f9b05d229dc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:51.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.508+0000 7efd80d5e640 1 -- 172.21.15.125:0/2602265595 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd780a89f0 con 0x7efd7c1538f0 2023-10-30T12:37:51.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.508+0000 7efd80d5e640 1 -- 172.21.15.125:0/2602265595 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd780a8b50 con 0x7efd7c1538f0 2023-10-30T12:37:51.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.508+0000 7efd80d5e640 1 -- 172.21.15.125:0/2602265595 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd780b0440 con 0x7efd7c1538f0 2023-10-30T12:37:51.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.508+0000 7efd82d62640 1 -- 172.21.15.125:0/2602265595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd7c1538f0 msgr2=0x7efd7c159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:51.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.508+0000 7efd82d62640 1 --2- 172.21.15.125:0/2602265595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd7c1538f0 0x7efd7c159fb0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7efd7c072e90 tx=0x7efd7801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:37:51.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.508+0000 7efd82d62640 1 -- 172.21.15.125:0/2602265595 shutdown_connections 2023-10-30T12:37:51.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.508+0000 7efd82d62640 1 --2- 172.21.15.125:0/2602265595 >> v2:172.21.15.136:3300/0 conn(0x7efd7c15cd50 0x7efd7c0c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:51.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.508+0000 7efd82d62640 1 --2- 172.21.15.125:0/2602265595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd7c1538f0 0x7efd7c159fb0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:51.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.508+0000 7efd82d62640 1 -- 172.21.15.125:0/2602265595 >> 172.21.15.125:0/2602265595 conn(0x7efd7c06a640 msgr2=0x7efd7c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:51.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.508+0000 7efd82d62640 1 -- 172.21.15.125:0/2602265595 shutdown_connections 2023-10-30T12:37:51.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.508+0000 7efd82d62640 1 -- 172.21.15.125:0/2602265595 wait complete. 2023-10-30T12:37:51.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd82d62640 1 Processor -- start 2023-10-30T12:37:51.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd82d62640 1 -- start start 2023-10-30T12:37:51.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd82d62640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd7c0cefa0 0x7efd7c146960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:51.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd82d62640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd7c1538f0 0x7efd7c1505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:51.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd82d62640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efd7c15cd50 0x7efd7c150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:51.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd82d62640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c1e4af0 con 0x7efd7c0cefa0 2023-10-30T12:37:51.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd82d62640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c1f0180 con 0x7efd7c1538f0 2023-10-30T12:37:51.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd82d62640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd7c1e3e70 con 0x7efd7c15cd50 2023-10-30T12:37:51.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd81d60640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd7c0cefa0 0x7efd7c146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:51.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd81d60640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd7c0cefa0 0x7efd7c146960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56158/0 (socket says 172.21.15.125:56158) 2023-10-30T12:37:51.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd81d60640 1 -- 172.21.15.125:0/1293093271 learned_addr learned my addr 172.21.15.125:0/1293093271 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:37:51.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd8155f640 1 --2- 172.21.15.125:0/1293093271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd7c1538f0 0x7efd7c1505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:51.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd82561640 1 --2- 172.21.15.125:0/1293093271 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efd7c15cd50 0x7efd7c150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:51.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd81d60640 1 -- 172.21.15.125:0/1293093271 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efd7c15cd50 msgr2=0x7efd7c150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:51.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd81d60640 1 --2- 172.21.15.125:0/1293093271 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efd7c15cd50 0x7efd7c150af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:51.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd81d60640 1 -- 172.21.15.125:0/1293093271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd7c1538f0 msgr2=0x7efd7c1505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:51.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd81d60640 1 --2- 172.21.15.125:0/1293093271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd7c1538f0 0x7efd7c1505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:51.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd81d60640 1 -- 172.21.15.125:0/1293093271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd780094d0 con 0x7efd7c0cefa0 2023-10-30T12:37:51.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd81d60640 1 --2- 172.21.15.125:0/1293093271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd7c0cefa0 0x7efd7c146960 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7efd7801c920 tx=0x7efd7801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:51.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.512+0000 7efd727fc640 1 -- 172.21.15.125:0/1293093271 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd780b1c60 con 0x7efd7c0cefa0 2023-10-30T12:37:51.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.516+0000 7efd727fc640 1 -- 172.21.15.125:0/1293093271 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd780b1dc0 con 0x7efd7c0cefa0 2023-10-30T12:37:51.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.516+0000 7efd727fc640 1 -- 172.21.15.125:0/1293093271 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd780b9640 con 0x7efd7c0cefa0 2023-10-30T12:37:51.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.516+0000 7efd82d62640 1 -- 172.21.15.125:0/1293093271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd7c1e4150 con 0x7efd7c0cefa0 2023-10-30T12:37:51.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.516+0000 7efd82d62640 1 -- 172.21.15.125:0/1293093271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd7c1e4710 con 0x7efd7c0cefa0 2023-10-30T12:37:51.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.516+0000 7efd82d62640 1 -- 172.21.15.125:0/1293093271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd7c1e4d60 con 0x7efd7c0cefa0 2023-10-30T12:37:51.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.516+0000 7efd727fc640 1 -- 172.21.15.125:0/1293093271 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efd780b7070 con 0x7efd7c0cefa0 2023-10-30T12:37:51.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.520+0000 7efd727fc640 1 --2- 172.21.15.125:0/1293093271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efd600664a0 0x7efd60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:51.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.520+0000 7efd727fc640 1 -- 172.21.15.125:0/1293093271 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7efd78121930 con 0x7efd7c0cefa0 2023-10-30T12:37:51.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.520+0000 7efd8155f640 1 --2- 172.21.15.125:0/1293093271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efd600664a0 0x7efd60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:51.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.520+0000 7efd8155f640 1 --2- 172.21.15.125:0/1293093271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efd600664a0 0x7efd60068960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0cebb0 tx=0x7efd6c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:51.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.528+0000 7efd727fc640 1 -- 172.21.15.125:0/1293093271 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efd780eeb70 con 0x7efd7c0cefa0 2023-10-30T12:37:51.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.904+0000 7efd82d62640 1 -- 172.21.15.125:0/1293093271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd7c0cb750 con 0x7efd7c0cefa0 2023-10-30T12:37:51.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.904+0000 7efd727fc640 1 -- 172.21.15.125:0/1293093271 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+6551 (secure 0 0 0) 0x7efd780b5090 con 0x7efd7c0cefa0 2023-10-30T12:37:51.910 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:37:51.910 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:37:51.241243+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:51.910 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 54 2023-10-30T12:37:51.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.908+0000 7efd82d62640 1 -- 172.21.15.125:0/1293093271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efd600664a0 msgr2=0x7efd60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:51.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.908+0000 7efd82d62640 1 --2- 172.21.15.125:0/1293093271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efd600664a0 0x7efd60068960 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7efd7c0cebb0 tx=0x7efd6c009190 comp rx=0 tx=0).stop 2023-10-30T12:37:51.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.908+0000 7efd82d62640 1 -- 172.21.15.125:0/1293093271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd7c0cefa0 msgr2=0x7efd7c146960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:51.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.908+0000 7efd82d62640 1 --2- 172.21.15.125:0/1293093271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd7c0cefa0 0x7efd7c146960 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7efd7801c920 tx=0x7efd7801ba90 comp rx=0 tx=0).stop 2023-10-30T12:37:51.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.908+0000 7efd82d62640 1 -- 172.21.15.125:0/1293093271 shutdown_connections 2023-10-30T12:37:51.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.908+0000 7efd82d62640 1 --2- 172.21.15.125:0/1293093271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efd600664a0 0x7efd60068960 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:51.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.908+0000 7efd82d62640 1 --2- 172.21.15.125:0/1293093271 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efd7c15cd50 0x7efd7c150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:51.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.908+0000 7efd82d62640 1 --2- 172.21.15.125:0/1293093271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efd7c1538f0 0x7efd7c1505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:51.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.908+0000 7efd82d62640 1 --2- 172.21.15.125:0/1293093271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efd7c0cefa0 0x7efd7c146960 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:51.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.908+0000 7efd82d62640 1 -- 172.21.15.125:0/1293093271 >> 172.21.15.125:0/1293093271 conn(0x7efd7c06a640 msgr2=0x7efd7c0c4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:51.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.908+0000 7efd82d62640 1 -- 172.21.15.125:0/1293093271 shutdown_connections 2023-10-30T12:37:51.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:51.908+0000 7efd82d62640 1 -- 172.21.15.125:0/1293093271 wait complete. 2023-10-30T12:37:51.941 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T12:37:51.942 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 54, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 54, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4957": { "addr": "172.21.15.136:6835/2548530941", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2548530941, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2548530941, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4957, "incarnation": 23, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:stopping", "state_seq": 34 }, "gid_4973": { "addr": "172.21.15.136:6839/1454326577", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 1454326577, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 1454326577, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4973, "incarnation": 45, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 21 }, "gid_5130": { "addr": "172.21.15.125:6837/1624923396", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 1624923396, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 1624923396, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 5130, "incarnation": 37, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:37:51.241243+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4973, "mds_1": 5130, "mds_2": 4957 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6836/2595438444", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2595438444, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2595438444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 50, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5433, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:37:53.944 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:54.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.268+0000 7f57c5adc640 1 Processor -- start 2023-10-30T12:37:54.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.268+0000 7f57c5adc640 1 -- start start 2023-10-30T12:37:54.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.272+0000 7f57c5adc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f57c00c4030 0x7f57c006ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:54.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.272+0000 7f57c4ada640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f57c00c4030 0x7f57c006ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:54.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.272+0000 7f57c4ada640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f57c00c4030 0x7f57c006ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56160/0 (socket says 172.21.15.125:56160) 2023-10-30T12:37:54.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.272+0000 7f57c4ada640 1 -- 172.21.15.125:0/2722796687 learned_addr learned my addr 172.21.15.125:0/2722796687 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:37:54.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.272+0000 7f57c5adc640 1 --2- 172.21.15.125:0/2722796687 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57c0070200 0x7f57c00c6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:54.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.276+0000 7f57c52db640 1 --2- 172.21.15.125:0/2722796687 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57c0070200 0x7f57c00c6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:54.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.276+0000 7f57c5adc640 1 -- 172.21.15.125:0/2722796687 --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c0156310 con 0x7f57c00c4410 2023-10-30T12:37:54.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.276+0000 7f57c5adc640 1 -- 172.21.15.125:0/2722796687 --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c006fb10 con 0x7f57c00c4030 2023-10-30T12:37:54.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.276+0000 7f57bf7fe640 1 -- 172.21.15.125:0/2722796687 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4251409265 0 0) 0x7f57c0156310 con 0x7f57c00c4410 2023-10-30T12:37:54.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.280+0000 7f57c5adc640 1 -- 172.21.15.125:0/2722796687 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c006fdb0 con 0x7f57c0070200 2023-10-30T12:37:54.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.284+0000 7f57bf7fe640 1 -- 172.21.15.125:0/2722796687 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a8003530 con 0x7f57c00c4410 2023-10-30T12:37:54.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.284+0000 7f57c4ada640 1 -- 172.21.15.125:0/2722796687 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57c0070200 msgr2=0x7f57c00c6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:54.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.284+0000 7f57c4ada640 1 --2- 172.21.15.125:0/2722796687 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57c0070200 0x7f57c00c6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.284+0000 7f57c4ada640 1 -- 172.21.15.125:0/2722796687 >> v1:172.21.15.125:6789/0 conn(0x7f57c00c4410 legacy=0x7f57c006f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:54.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.288+0000 7f57c4ada640 1 -- 172.21.15.125:0/2722796687 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c0155720 con 0x7f57c00c4030 2023-10-30T12:37:54.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.296+0000 7f57c4ada640 1 --2- 172.21.15.125:0/2722796687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57c00c4030 0x7f57c006ab20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f57b4006fd0 tx=0x7f57b401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=930ee41521070f75 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:54.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.300+0000 7f57bf7fe640 1 -- 172.21.15.125:0/2722796687 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b4008be0 con 0x7f57c00c4030 2023-10-30T12:37:54.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.300+0000 7f57bf7fe640 1 -- 172.21.15.125:0/2722796687 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b4008d40 con 0x7f57c00c4030 2023-10-30T12:37:54.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.300+0000 7f57bf7fe640 1 -- 172.21.15.125:0/2722796687 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b400a2f0 con 0x7f57c00c4030 2023-10-30T12:37:54.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.300+0000 7f57c5adc640 1 -- 172.21.15.125:0/2722796687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57c00c4030 msgr2=0x7f57c006ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:54.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.300+0000 7f57c5adc640 1 --2- 172.21.15.125:0/2722796687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57c00c4030 0x7f57c006ab20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f57b4006fd0 tx=0x7f57b401c4b0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.300+0000 7f57c5adc640 1 -- 172.21.15.125:0/2722796687 shutdown_connections 2023-10-30T12:37:54.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.300+0000 7f57c5adc640 1 --2- 172.21.15.125:0/2722796687 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57c0070200 0x7f57c00c6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.300+0000 7f57c5adc640 1 --2- 172.21.15.125:0/2722796687 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57c00c4030 0x7f57c006ab20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.300+0000 7f57c5adc640 1 -- 172.21.15.125:0/2722796687 >> 172.21.15.125:0/2722796687 conn(0x7f57c00c75d0 msgr2=0x7f57c00c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:54.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.304+0000 7f57c5adc640 1 -- 172.21.15.125:0/2722796687 shutdown_connections 2023-10-30T12:37:54.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.304+0000 7f57c5adc640 1 -- 172.21.15.125:0/2722796687 wait complete. 2023-10-30T12:37:54.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.304+0000 7f57c5adc640 1 Processor -- start 2023-10-30T12:37:54.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.304+0000 7f57c5adc640 1 -- start start 2023-10-30T12:37:54.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57c5adc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57c0070200 0x7f57c01f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:54.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57c5adc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57c00c4030 0x7f57c006a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:54.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57c5adc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57c00c4410 0x7f57c014bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:54.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57c5adc640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c0152600 con 0x7f57c0070200 2023-10-30T12:37:54.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57c5adc640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57c014a660 con 0x7f57c00c4410 2023-10-30T12:37:54.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57c5adc640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c014ac60 con 0x7f57c00c4030 2023-10-30T12:37:54.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57bffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57c00c4030 0x7f57c006a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:54.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57bffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57c00c4030 0x7f57c006a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:41316/0 (socket says 172.21.15.125:41316) 2023-10-30T12:37:54.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57bffff640 1 -- 172.21.15.125:0/898876765 learned_addr learned my addr 172.21.15.125:0/898876765 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:37:54.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57c52db640 1 --2- 172.21.15.125:0/898876765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57c00c4410 0x7f57c014bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:54.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57bffff640 1 -- 172.21.15.125:0/898876765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57c00c4410 msgr2=0x7f57c014bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:54.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57bffff640 1 --2- 172.21.15.125:0/898876765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57c00c4410 0x7f57c014bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57bffff640 1 -- 172.21.15.125:0/898876765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57c0070200 msgr2=0x7f57c01f17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:54.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57bffff640 1 --2- 172.21.15.125:0/898876765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57c0070200 0x7f57c01f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57bffff640 1 -- 172.21.15.125:0/898876765 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b4008050 con 0x7f57c00c4030 2023-10-30T12:37:54.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57bffff640 1 --2- 172.21.15.125:0/898876765 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57c00c4030 0x7f57c006a0f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57b000c500 tx=0x7f57b000c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:54.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57bd7fa640 1 -- 172.21.15.125:0/898876765 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b00052d0 con 0x7f57c00c4030 2023-10-30T12:37:54.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57bd7fa640 1 -- 172.21.15.125:0/898876765 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b0005430 con 0x7f57c00c4030 2023-10-30T12:37:54.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57c5adc640 1 -- 172.21.15.125:0/898876765 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c014b730 con 0x7f57c00c4030 2023-10-30T12:37:54.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.308+0000 7f57c5adc640 1 -- 172.21.15.125:0/898876765 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c0152040 con 0x7f57c00c4030 2023-10-30T12:37:54.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.312+0000 7f57bd7fa640 1 -- 172.21.15.125:0/898876765 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57b00052d0 con 0x7f57c00c4030 2023-10-30T12:37:54.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.312+0000 7f57a6ffd640 1 -- 172.21.15.125:0/898876765 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c004e170 con 0x7f57c00c4030 2023-10-30T12:37:54.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.312+0000 7f57bd7fa640 1 -- 172.21.15.125:0/898876765 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f57b0012630 con 0x7f57c00c4030 2023-10-30T12:37:54.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.312+0000 7f57bd7fa640 1 --2- 172.21.15.125:0/898876765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57ac066500 0x7f57ac0689c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:54.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.316+0000 7f57bd7fa640 1 -- 172.21.15.125:0/898876765 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f57b0083860 con 0x7f57c00c4030 2023-10-30T12:37:54.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.316+0000 7f57c4ada640 1 --2- 172.21.15.125:0/898876765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57ac066500 0x7f57ac0689c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:54.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.316+0000 7f57c4ada640 1 --2- 172.21.15.125:0/898876765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57ac066500 0x7f57ac0689c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f57b40ae000 tx=0x7f57b40046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:54.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.328+0000 7f57bd7fa640 1 -- 172.21.15.125:0/898876765 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f57b0050ab0 con 0x7f57c00c4030 2023-10-30T12:37:54.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.564+0000 7f57a6ffd640 1 -- 172.21.15.125:0/898876765 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57c014b8c0 con 0x7f57c00c4030 2023-10-30T12:37:54.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.564+0000 7f57bd7fa640 1 -- 172.21.15.125:0/898876765 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f57b00540d0 con 0x7f57c00c4030 2023-10-30T12:37:54.569 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:37:54.569 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:37:53.067888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:54.570 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 55 2023-10-30T12:37:54.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.568+0000 7f57a6ffd640 1 -- 172.21.15.125:0/898876765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57ac066500 msgr2=0x7f57ac0689c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:54.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.568+0000 7f57a6ffd640 1 --2- 172.21.15.125:0/898876765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57ac066500 0x7f57ac0689c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f57b40ae000 tx=0x7f57b40046d0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.568+0000 7f57a6ffd640 1 -- 172.21.15.125:0/898876765 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57c00c4030 msgr2=0x7f57c006a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:54.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.568+0000 7f57a6ffd640 1 --2- 172.21.15.125:0/898876765 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57c00c4030 0x7f57c006a0f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f57b000c500 tx=0x7f57b000c9c0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.568+0000 7f57a6ffd640 1 -- 172.21.15.125:0/898876765 shutdown_connections 2023-10-30T12:37:54.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.568+0000 7f57a6ffd640 1 --2- 172.21.15.125:0/898876765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57ac066500 0x7f57ac0689c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.568+0000 7f57a6ffd640 1 --2- 172.21.15.125:0/898876765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57c00c4410 0x7f57c014bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.568+0000 7f57a6ffd640 1 --2- 172.21.15.125:0/898876765 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57c00c4030 0x7f57c006a0f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.568+0000 7f57a6ffd640 1 --2- 172.21.15.125:0/898876765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57c0070200 0x7f57c01f17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:54.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.568+0000 7f57a6ffd640 1 -- 172.21.15.125:0/898876765 >> 172.21.15.125:0/898876765 conn(0x7f57c00c75d0 msgr2=0x7f57c00ca2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:54.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.568+0000 7f57a6ffd640 1 -- 172.21.15.125:0/898876765 shutdown_connections 2023-10-30T12:37:54.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:54.568+0000 7f57a6ffd640 1 -- 172.21.15.125:0/898876765 wait complete. 2023-10-30T12:37:54.592 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:37:56.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:56.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.828+0000 7f9dfba17640 1 Processor -- start 2023-10-30T12:37:56.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9dfba17640 1 -- start start 2023-10-30T12:37:56.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9dfba17640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9df40665c0 0x7f9df415a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:56.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9dfba17640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9df40660c0 0x7f9df4153530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:56.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9dfba17640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df415a6e0 con 0x7f9df4065ce0 2023-10-30T12:37:56.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9dfba17640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df40ca820 con 0x7f9df40660c0 2023-10-30T12:37:56.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9dfba17640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df40ca390 con 0x7f9df40665c0 2023-10-30T12:37:56.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9df97b3640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9df4065ce0 0x7f9df40c3c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:40876/0 (socket says 172.21.15.125:40876) 2023-10-30T12:37:56.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9df97b3640 1 -- 172.21.15.125:0/2795851458 learned_addr learned my addr 172.21.15.125:0/2795851458 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:37:56.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9df8fb2640 1 --2- 172.21.15.125:0/2795851458 >> v2:172.21.15.136:3300/0 conn(0x7f9df40665c0 0x7f9df415a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:56.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9debfff640 1 -- 172.21.15.125:0/2795851458 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4264479058 0 0) 0x7f9df0001120 con 0x7f9df4065ce0 2023-10-30T12:37:56.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9debfff640 1 -- 172.21.15.125:0/2795851458 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de0003530 con 0x7f9df4065ce0 2023-10-30T12:37:56.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9df8fb2640 1 -- 172.21.15.125:0/2795851458 >> v2:172.21.15.125:3300/0 conn(0x7f9df40660c0 msgr2=0x7f9df4153530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:56.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9df8fb2640 1 --2- 172.21.15.125:0/2795851458 >> v2:172.21.15.125:3300/0 conn(0x7f9df40660c0 0x7f9df4153530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:56.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9df8fb2640 1 -- 172.21.15.125:0/2795851458 >> v1:172.21.15.136:6789/0 conn(0x7f9df4065ce0 legacy=0x7f9df40c3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:56.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.832+0000 7f9df8fb2640 1 -- 172.21.15.125:0/2795851458 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df4066a70 con 0x7f9df40665c0 2023-10-30T12:37:56.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.844+0000 7f9df8fb2640 1 --2- 172.21.15.125:0/2795851458 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9df40665c0 0x7f9df415a1a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9df408bff0 tx=0x7f9de401ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=991c587543781b10 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:56.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.844+0000 7f9debfff640 1 -- 172.21.15.125:0/2795851458 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de40a8950 con 0x7f9df40665c0 2023-10-30T12:37:56.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.844+0000 7f9debfff640 1 -- 172.21.15.125:0/2795851458 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de40a8ab0 con 0x7f9df40665c0 2023-10-30T12:37:56.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.844+0000 7f9debfff640 1 -- 172.21.15.125:0/2795851458 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de40a8e40 con 0x7f9df40665c0 2023-10-30T12:37:56.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.844+0000 7f9dfba17640 1 -- 172.21.15.125:0/2795851458 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9df40665c0 msgr2=0x7f9df415a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:56.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.844+0000 7f9dfba17640 1 --2- 172.21.15.125:0/2795851458 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9df40665c0 0x7f9df415a1a0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9df408bff0 tx=0x7f9de401ec50 comp rx=0 tx=0).stop 2023-10-30T12:37:56.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.848+0000 7f9dfba17640 1 -- 172.21.15.125:0/2795851458 shutdown_connections 2023-10-30T12:37:56.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.848+0000 7f9dfba17640 1 --2- 172.21.15.125:0/2795851458 >> v2:172.21.15.125:3300/0 conn(0x7f9df40660c0 0x7f9df4153530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:56.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.848+0000 7f9dfba17640 1 --2- 172.21.15.125:0/2795851458 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9df40665c0 0x7f9df415a1a0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:56.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.848+0000 7f9dfba17640 1 -- 172.21.15.125:0/2795851458 >> 172.21.15.125:0/2795851458 conn(0x7f9df40cee00 msgr2=0x7f9df40cf210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:56.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.848+0000 7f9dfba17640 1 -- 172.21.15.125:0/2795851458 shutdown_connections 2023-10-30T12:37:56.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.848+0000 7f9dfba17640 1 -- 172.21.15.125:0/2795851458 wait complete. 2023-10-30T12:37:56.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.852+0000 7f9dfba17640 1 Processor -- start 2023-10-30T12:37:56.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.856+0000 7f9dfba17640 1 -- start start 2023-10-30T12:37:56.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.860+0000 7f9dfba17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9df4065ce0 0x7f9df40ce550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:56.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.860+0000 7f9dfba17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9df40660c0 0x7f9df4154070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:56.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.860+0000 7f9df97b3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9df4065ce0 0x7f9df40ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:56.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.860+0000 7f9df97b3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9df4065ce0 0x7f9df40ce550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:46308/0 (socket says 172.21.15.125:46308) 2023-10-30T12:37:56.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9df8fb2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9df40660c0 0x7f9df4154070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:56.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9df8fb2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9df40660c0 0x7f9df4154070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56184/0 (socket says 172.21.15.125:56184) 2023-10-30T12:37:56.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9df97b3640 1 -- 172.21.15.125:0/2476196609 learned_addr learned my addr 172.21.15.125:0/2476196609 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:37:56.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9dfba17640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9df40665c0 0x7f9df41545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:56.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9dfba17640 1 -- 172.21.15.125:0/2476196609 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df41fc0c0 con 0x7f9df40660c0 2023-10-30T12:37:56.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9dfba17640 1 -- 172.21.15.125:0/2476196609 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df41fb4d0 con 0x7f9df4065ce0 2023-10-30T12:37:56.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9dfba17640 1 -- 172.21.15.125:0/2476196609 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df41ff790 con 0x7f9df40665c0 2023-10-30T12:37:56.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9df9fb4640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9df40665c0 0x7f9df41545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:56.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9df9fb4640 1 -- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9df4065ce0 msgr2=0x7f9df40ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:56.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9df9fb4640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9df4065ce0 0x7f9df40ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:56.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9df9fb4640 1 -- 172.21.15.125:0/2476196609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9df40660c0 msgr2=0x7f9df4154070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:56.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9df9fb4640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9df40660c0 0x7f9df4154070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:56.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9df9fb4640 1 -- 172.21.15.125:0/2476196609 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de4009420 con 0x7f9df40665c0 2023-10-30T12:37:56.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9df97b3640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9df4065ce0 0x7f9df40ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:37:56.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9df8fb2640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9df40660c0 0x7f9df4154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:37:56.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9df9fb4640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9df40665c0 0x7f9df41545b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00bdb0 tx=0x7f9dec00bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:56.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9de9ffb640 1 -- 172.21.15.125:0/2476196609 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec011710 con 0x7f9df40665c0 2023-10-30T12:37:56.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9de9ffb640 1 -- 172.21.15.125:0/2476196609 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dec00f050 con 0x7f9df40665c0 2023-10-30T12:37:56.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9de9ffb640 1 -- 172.21.15.125:0/2476196609 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9dec011ad0 con 0x7f9df40665c0 2023-10-30T12:37:56.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9dfba17640 1 -- 172.21.15.125:0/2476196609 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df41fb790 con 0x7f9df40665c0 2023-10-30T12:37:56.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.864+0000 7f9dfba17640 1 -- 172.21.15.125:0/2476196609 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df41fbd50 con 0x7f9df40665c0 2023-10-30T12:37:56.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.868+0000 7f9dfba17640 1 -- 172.21.15.125:0/2476196609 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df41fc270 con 0x7f9df40665c0 2023-10-30T12:37:56.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.868+0000 7f9de9ffb640 1 -- 172.21.15.125:0/2476196609 <== mon.2 v2:172.21.15.136:3301/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9dec011c30 con 0x7f9df40665c0 2023-10-30T12:37:56.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.872+0000 7f9de9ffb640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9dd8066570 0x7f9dd8068a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:56.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.872+0000 7f9de9ffb640 1 -- 172.21.15.125:0/2476196609 <== mon.2 v2:172.21.15.136:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f9dec0845a0 con 0x7f9df40665c0 2023-10-30T12:37:56.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.876+0000 7f9df97b3640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9dd8066570 0x7f9dd8068a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:56.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.876+0000 7f9df97b3640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9dd8066570 0x7f9dd8068a30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9df0001380 tx=0x7f9df0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:56.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:56.876+0000 7f9de9ffb640 1 -- 172.21.15.125:0/2476196609 <== mon.2 v2:172.21.15.136:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9df41fc270 con 0x7f9df40665c0 2023-10-30T12:37:57.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:57.256+0000 7f9dfba17640 1 -- 172.21.15.125:0/2476196609 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9df41fc270 con 0x7f9df40665c0 2023-10-30T12:37:57.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:57.256+0000 7f9de9ffb640 1 -- 172.21.15.125:0/2476196609 <== mon.2 v2:172.21.15.136:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f9df41fc270 con 0x7f9df40665c0 2023-10-30T12:37:57.261 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:37:57.261 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:37:53.067888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:57.261 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 55 2023-10-30T12:37:57.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:57.260+0000 7f9dfba17640 1 -- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9dd8066570 msgr2=0x7f9dd8068a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:57.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:57.260+0000 7f9dfba17640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9dd8066570 0x7f9dd8068a30 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9df0001380 tx=0x7f9df0009040 comp rx=0 tx=0).stop 2023-10-30T12:37:57.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:57.260+0000 7f9dfba17640 1 -- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9df40665c0 msgr2=0x7f9df41545b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:57.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:57.260+0000 7f9dfba17640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9df40665c0 0x7f9df41545b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9dec00bdb0 tx=0x7f9dec00bea0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:57.260+0000 7f9dfba17640 1 -- 172.21.15.125:0/2476196609 shutdown_connections 2023-10-30T12:37:57.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:57.260+0000 7f9dfba17640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9dd8066570 0x7f9dd8068a30 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:57.260+0000 7f9dfba17640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9df40665c0 0x7f9df41545b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:57.260+0000 7f9dfba17640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9df40660c0 0x7f9df4154070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:57.260+0000 7f9dfba17640 1 --2- 172.21.15.125:0/2476196609 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9df4065ce0 0x7f9df40ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:57.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:57.260+0000 7f9dfba17640 1 -- 172.21.15.125:0/2476196609 >> 172.21.15.125:0/2476196609 conn(0x7f9df40cee00 msgr2=0x7f9df40c3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:57.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:57.260+0000 7f9dfba17640 1 -- 172.21.15.125:0/2476196609 shutdown_connections 2023-10-30T12:37:57.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:57.260+0000 7f9dfba17640 1 -- 172.21.15.125:0/2476196609 wait complete. 2023-10-30T12:37:57.284 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:37:59.284 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:37:59.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.512+0000 7f31e12c9640 1 Processor -- start 2023-10-30T12:37:59.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.512+0000 7f31e12c9640 1 -- start start 2023-10-30T12:37:59.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31e12c9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f31dc153d60 0x7f31dc0cc5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31e12c9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31dc0ccd30 0x7f31dc0cd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31e12c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31dc0cd7c0 0x7f31dc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31e12c9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31dc0ccb10 con 0x7f31dc0ccd30 2023-10-30T12:37:59.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31e12c9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31dc06ccf0 con 0x7f31dc153d60 2023-10-30T12:37:59.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31e12c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc15a540 con 0x7f31dc0cd7c0 2023-10-30T12:37:59.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31da7fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31dc0ccd30 0x7f31dc0cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31da7fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31dc0ccd30 0x7f31dc0cd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56194/0 (socket says 172.21.15.125:56194) 2023-10-30T12:37:59.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31da7fc640 1 -- 172.21.15.125:0/860554076 learned_addr learned my addr 172.21.15.125:0/860554076 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:37:59.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31dbfff640 1 --2- 172.21.15.125:0/860554076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31dc0cd7c0 0x7f31dc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31daffd640 1 --2- 172.21.15.125:0/860554076 >> v2:172.21.15.136:3300/0 conn(0x7f31dc153d60 0x7f31dc0cc5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31da7fc640 1 -- 172.21.15.125:0/860554076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31dc0cd7c0 msgr2=0x7f31dc06d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31da7fc640 1 --2- 172.21.15.125:0/860554076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31dc0cd7c0 0x7f31dc06d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31da7fc640 1 -- 172.21.15.125:0/860554076 >> v2:172.21.15.136:3300/0 conn(0x7f31dc153d60 msgr2=0x7f31dc0cc5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:37:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31da7fc640 1 --2- 172.21.15.125:0/860554076 >> v2:172.21.15.136:3300/0 conn(0x7f31dc153d60 0x7f31dc0cc5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.516+0000 7f31da7fc640 1 -- 172.21.15.125:0/860554076 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc0cb4c0 con 0x7f31dc0ccd30 2023-10-30T12:37:59.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.520+0000 7f31da7fc640 1 --2- 172.21.15.125:0/860554076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc0ccd30 0x7f31dc0cd280 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009870 tx=0x7f31d001ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3915fd88c1192275 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:59.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.520+0000 7f31d9ffb640 1 -- 172.21.15.125:0/860554076 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d00a89b0 con 0x7f31dc0ccd30 2023-10-30T12:37:59.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.520+0000 7f31d9ffb640 1 -- 172.21.15.125:0/860554076 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d00a8b10 con 0x7f31dc0ccd30 2023-10-30T12:37:59.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.520+0000 7f31d9ffb640 1 -- 172.21.15.125:0/860554076 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d00a8ea0 con 0x7f31dc0ccd30 2023-10-30T12:37:59.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.524+0000 7f31e12c9640 1 -- 172.21.15.125:0/860554076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc0ccd30 msgr2=0x7f31dc0cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:59.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.524+0000 7f31e12c9640 1 --2- 172.21.15.125:0/860554076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc0ccd30 0x7f31dc0cd280 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f31d0009870 tx=0x7f31d001ed10 comp rx=0 tx=0).stop 2023-10-30T12:37:59.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.524+0000 7f31e12c9640 1 -- 172.21.15.125:0/860554076 shutdown_connections 2023-10-30T12:37:59.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.524+0000 7f31e12c9640 1 --2- 172.21.15.125:0/860554076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31dc0cd7c0 0x7f31dc06d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.524+0000 7f31e12c9640 1 --2- 172.21.15.125:0/860554076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc0ccd30 0x7f31dc0cd280 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.524+0000 7f31e12c9640 1 --2- 172.21.15.125:0/860554076 >> v2:172.21.15.136:3300/0 conn(0x7f31dc153d60 0x7f31dc0cc5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.524+0000 7f31e12c9640 1 -- 172.21.15.125:0/860554076 >> 172.21.15.125:0/860554076 conn(0x7f31dc06e840 msgr2=0x7f31dc06ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:59.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.524+0000 7f31e12c9640 1 -- 172.21.15.125:0/860554076 shutdown_connections 2023-10-30T12:37:59.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.524+0000 7f31e12c9640 1 -- 172.21.15.125:0/860554076 wait complete. 2023-10-30T12:37:59.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.524+0000 7f31e12c9640 1 Processor -- start 2023-10-30T12:37:59.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.524+0000 7f31e12c9640 1 -- start start 2023-10-30T12:37:59.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31e12c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31dc0ccd30 0x7f31dc0c97b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31e12c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31dc0cd7c0 0x7f31dc1594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31e12c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc153d60 0x7f31dc1471a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31e12c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc1485e0 con 0x7f31dc153d60 2023-10-30T12:37:59.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31e12c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc0c9080 con 0x7f31dc0ccd30 2023-10-30T12:37:59.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31e12c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc1534c0 con 0x7f31dc0cd7c0 2023-10-30T12:37:59.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31daffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31dc0ccd30 0x7f31dc0c97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31daffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31dc0ccd30 0x7f31dc0c97b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:46318/0 (socket says 172.21.15.125:46318) 2023-10-30T12:37:59.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31dbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc153d60 0x7f31dc1471a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31daffd640 1 -- 172.21.15.125:0/228972847 learned_addr learned my addr 172.21.15.125:0/228972847 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:37:59.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31da7fc640 1 --2- 172.21.15.125:0/228972847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31dc0cd7c0 0x7f31dc1594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31dbfff640 1 -- 172.21.15.125:0/228972847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31dc0cd7c0 msgr2=0x7f31dc1594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:59.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31dbfff640 1 --2- 172.21.15.125:0/228972847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31dc0cd7c0 0x7f31dc1594f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31dbfff640 1 -- 172.21.15.125:0/228972847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31dc0ccd30 msgr2=0x7f31dc0c97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:59.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31dbfff640 1 --2- 172.21.15.125:0/228972847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31dc0ccd30 0x7f31dc0c97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31dbfff640 1 -- 172.21.15.125:0/228972847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d00094d0 con 0x7f31dc153d60 2023-10-30T12:37:59.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31daffd640 1 --2- 172.21.15.125:0/228972847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31dc0ccd30 0x7f31dc0c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:37:59.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31dbfff640 1 --2- 172.21.15.125:0/228972847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc153d60 0x7f31dc1471a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f31d400eed0 tx=0x7f31d400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:59.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31c37fe640 1 -- 172.21.15.125:0/228972847 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4010040 con 0x7f31dc153d60 2023-10-30T12:37:59.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31c37fe640 1 -- 172.21.15.125:0/228972847 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d4009e60 con 0x7f31dc153d60 2023-10-30T12:37:59.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31c37fe640 1 -- 172.21.15.125:0/228972847 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d4014640 con 0x7f31dc153d60 2023-10-30T12:37:59.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31e12c9640 1 -- 172.21.15.125:0/228972847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31dc0c81d0 con 0x7f31dc153d60 2023-10-30T12:37:59.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.528+0000 7f31e12c9640 1 -- 172.21.15.125:0/228972847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31dc0c8750 con 0x7f31dc153d60 2023-10-30T12:37:59.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.532+0000 7f31e12c9640 1 -- 172.21.15.125:0/228972847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31dc0cf750 con 0x7f31dc153d60 2023-10-30T12:37:59.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.532+0000 7f31c37fe640 1 -- 172.21.15.125:0/228972847 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f31d40147a0 con 0x7f31dc153d60 2023-10-30T12:37:59.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.536+0000 7f31c37fe640 1 --2- 172.21.15.125:0/228972847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31c80664f0 0x7f31c80689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:37:59.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.536+0000 7f31c37fe640 1 -- 172.21.15.125:0/228972847 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f31d4084a70 con 0x7f31dc153d60 2023-10-30T12:37:59.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.536+0000 7f31daffd640 1 --2- 172.21.15.125:0/228972847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31c80664f0 0x7f31c80689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:37:59.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.536+0000 7f31daffd640 1 --2- 172.21.15.125:0/228972847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31c80664f0 0x7f31c80689b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f31dc157410 tx=0x7f31cc009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:37:59.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.544+0000 7f31c37fe640 1 -- 172.21.15.125:0/228972847 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f31dc0cf750 con 0x7f31dc153d60 2023-10-30T12:37:59.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.916+0000 7f31e12c9640 1 -- 172.21.15.125:0/228972847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31dc0cf750 con 0x7f31dc153d60 2023-10-30T12:37:59.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.920+0000 7f31c37fe640 1 -- 172.21.15.125:0/228972847 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f31dc0cf750 con 0x7f31dc153d60 2023-10-30T12:37:59.923 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:37:59.924 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:37:53.067888+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:37:59.924 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 55 2023-10-30T12:37:59.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.920+0000 7f31e12c9640 1 -- 172.21.15.125:0/228972847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31c80664f0 msgr2=0x7f31c80689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:59.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.920+0000 7f31e12c9640 1 --2- 172.21.15.125:0/228972847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31c80664f0 0x7f31c80689b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f31dc157410 tx=0x7f31cc009190 comp rx=0 tx=0).stop 2023-10-30T12:37:59.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.920+0000 7f31e12c9640 1 -- 172.21.15.125:0/228972847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc153d60 msgr2=0x7f31dc1471a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:37:59.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.920+0000 7f31e12c9640 1 --2- 172.21.15.125:0/228972847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc153d60 0x7f31dc1471a0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f31d400eed0 tx=0x7f31d400b480 comp rx=0 tx=0).stop 2023-10-30T12:37:59.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.920+0000 7f31e12c9640 1 -- 172.21.15.125:0/228972847 shutdown_connections 2023-10-30T12:37:59.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.920+0000 7f31e12c9640 1 --2- 172.21.15.125:0/228972847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31c80664f0 0x7f31c80689b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.920+0000 7f31e12c9640 1 --2- 172.21.15.125:0/228972847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31dc153d60 0x7f31dc1471a0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.920+0000 7f31e12c9640 1 --2- 172.21.15.125:0/228972847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31dc0cd7c0 0x7f31dc1594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.920+0000 7f31e12c9640 1 --2- 172.21.15.125:0/228972847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31dc0ccd30 0x7f31dc0c97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:37:59.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.920+0000 7f31e12c9640 1 -- 172.21.15.125:0/228972847 >> 172.21.15.125:0/228972847 conn(0x7f31dc06e840 msgr2=0x7f31dc14e560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:37:59.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.920+0000 7f31e12c9640 1 -- 172.21.15.125:0/228972847 shutdown_connections 2023-10-30T12:37:59.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:37:59.920+0000 7f31e12c9640 1 -- 172.21.15.125:0/228972847 wait complete. 2023-10-30T12:37:59.955 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:38:01.956 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:02.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.092+0000 7fb77aeea640 1 Processor -- start 2023-10-30T12:38:02.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.096+0000 7fb77aeea640 1 -- start start 2023-10-30T12:38:02.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.096+0000 7fb77aeea640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb7740c05d0 0x7fb77413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:02.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.096+0000 7fb77aeea640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb7740c6f60 0x7fb77413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:02.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.096+0000 7fb77aeea640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7740c0a50 con 0x7fb7740c6b80 2023-10-30T12:38:02.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.096+0000 7fb77aeea640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77406f6e0 con 0x7fb7740c6f60 2023-10-30T12:38:02.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.096+0000 7fb77aeea640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77406f9c0 con 0x7fb7740c05d0 2023-10-30T12:38:02.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.096+0000 7fb779487640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb7740c6f60 0x7fb77413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:02.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.096+0000 7fb779487640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb7740c6f60 0x7fb77413f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38210/0 (socket says 172.21.15.125:38210) 2023-10-30T12:38:02.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.096+0000 7fb779487640 1 -- 172.21.15.125:0/325392368 learned_addr learned my addr 172.21.15.125:0/325392368 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:02.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.096+0000 7fb773fff640 1 --2- 172.21.15.125:0/325392368 >> v2:172.21.15.136:3300/0 conn(0x7fb7740c05d0 0x7fb77413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:02.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.096+0000 7fb779487640 1 -- 172.21.15.125:0/325392368 >> v2:172.21.15.136:3300/0 conn(0x7fb7740c05d0 msgr2=0x7fb77413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:02.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.096+0000 7fb779487640 1 --2- 172.21.15.125:0/325392368 >> v2:172.21.15.136:3300/0 conn(0x7fb7740c05d0 0x7fb77413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.096+0000 7fb779487640 1 -- 172.21.15.125:0/325392368 >> v1:172.21.15.136:6789/0 conn(0x7fb7740c6b80 legacy=0x7fb7740bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:02.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.096+0000 7fb779487640 1 -- 172.21.15.125:0/325392368 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77406fca0 con 0x7fb7740c6f60 2023-10-30T12:38:02.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.100+0000 7fb779487640 1 --2- 172.21.15.125:0/325392368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7740c6f60 0x7fb77413f2d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb77406c7e0 tx=0x7fb76c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=490d91010f20bed2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:02.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.100+0000 7fb7737fe640 1 -- 172.21.15.125:0/325392368 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0a57a0 con 0x7fb7740c6f60 2023-10-30T12:38:02.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.100+0000 7fb7737fe640 1 -- 172.21.15.125:0/325392368 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76c0a5900 con 0x7fb7740c6f60 2023-10-30T12:38:02.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.100+0000 7fb7737fe640 1 -- 172.21.15.125:0/325392368 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0a5c90 con 0x7fb7740c6f60 2023-10-30T12:38:02.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.100+0000 7fb77aeea640 1 -- 172.21.15.125:0/325392368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7740c6f60 msgr2=0x7fb77413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:02.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.100+0000 7fb77aeea640 1 --2- 172.21.15.125:0/325392368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7740c6f60 0x7fb77413f2d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb77406c7e0 tx=0x7fb76c01c9d0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.100+0000 7fb77aeea640 1 -- 172.21.15.125:0/325392368 shutdown_connections 2023-10-30T12:38:02.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.100+0000 7fb77aeea640 1 --2- 172.21.15.125:0/325392368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7740c6f60 0x7fb77413f2d0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.100+0000 7fb77aeea640 1 --2- 172.21.15.125:0/325392368 >> v2:172.21.15.136:3300/0 conn(0x7fb7740c05d0 0x7fb77413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.100+0000 7fb77aeea640 1 -- 172.21.15.125:0/325392368 >> 172.21.15.125:0/325392368 conn(0x7fb7740c7910 msgr2=0x7fb7740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:02.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.100+0000 7fb77aeea640 1 -- 172.21.15.125:0/325392368 shutdown_connections 2023-10-30T12:38:02.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.100+0000 7fb77aeea640 1 -- 172.21.15.125:0/325392368 wait complete. 2023-10-30T12:38:02.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.100+0000 7fb77aeea640 1 Processor -- start 2023-10-30T12:38:02.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.100+0000 7fb77aeea640 1 -- start start 2023-10-30T12:38:02.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb77aeea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7740c05d0 0x7fb77413d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:02.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb77aeea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7740c6b80 0x7fb7740c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:02.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb77aeea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7740c6f60 0x7fb7740ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:02.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb77aeea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7740c6130 con 0x7fb7740c6b80 2023-10-30T12:38:02.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb77aeea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7740c5e50 con 0x7fb7740c6f60 2023-10-30T12:38:02.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb77aeea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7740c8db0 con 0x7fb7740c05d0 2023-10-30T12:38:02.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb773fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7740c6b80 0x7fb7740c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:02.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb773fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7740c6b80 0x7fb7740c9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38218/0 (socket says 172.21.15.125:38218) 2023-10-30T12:38:02.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb773fff640 1 -- 172.21.15.125:0/2806416460 learned_addr learned my addr 172.21.15.125:0/2806416460 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:02.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb779487640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7740c6f60 0x7fb7740ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:02.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb778c86640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7740c05d0 0x7fb77413d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:02.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb773fff640 1 -- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7740c05d0 msgr2=0x7fb77413d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:02.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb773fff640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7740c05d0 0x7fb77413d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb773fff640 1 -- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7740c6f60 msgr2=0x7fb7740ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:02.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb773fff640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7740c6f60 0x7fb7740ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb773fff640 1 -- 172.21.15.125:0/2806416460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c009b30 con 0x7fb7740c6b80 2023-10-30T12:38:02.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb779487640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7740c6f60 0x7fb7740ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:02.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb778c86640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7740c05d0 0x7fb77413d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:02.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb773fff640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7740c6b80 0x7fb7740c9b70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb76000d7d0 tx=0x7fb76000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:02.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb7717fa640 1 -- 172.21.15.125:0/2806416460 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76000b6c0 con 0x7fb7740c6b80 2023-10-30T12:38:02.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb7717fa640 1 -- 172.21.15.125:0/2806416460 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76000b820 con 0x7fb7740c6b80 2023-10-30T12:38:02.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb7717fa640 1 -- 172.21.15.125:0/2806416460 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76000bbd0 con 0x7fb7740c6b80 2023-10-30T12:38:02.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb77aeea640 1 -- 172.21.15.125:0/2806416460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7740c9090 con 0x7fb7740c6b80 2023-10-30T12:38:02.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb77aeea640 1 -- 172.21.15.125:0/2806416460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7740c9650 con 0x7fb7740c6b80 2023-10-30T12:38:02.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb77aeea640 1 -- 172.21.15.125:0/2806416460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77404e170 con 0x7fb7740c6b80 2023-10-30T12:38:02.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.104+0000 7fb7717fa640 1 -- 172.21.15.125:0/2806416460 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb76000bd30 con 0x7fb7740c6b80 2023-10-30T12:38:02.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.108+0000 7fb7717fa640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7580664a0 0x7fb758068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:02.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.108+0000 7fb7717fa640 1 -- 172.21.15.125:0/2806416460 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fb7600845b0 con 0x7fb7740c6b80 2023-10-30T12:38:02.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.108+0000 7fb778c86640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7580664a0 0x7fb758068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:02.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.108+0000 7fb778c86640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7580664a0 0x7fb758068960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb768000f70 tx=0x7fb768009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:02.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.112+0000 7fb7717fa640 1 -- 172.21.15.125:0/2806416460 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb760051880 con 0x7fb7740c6b80 2023-10-30T12:38:02.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.292+0000 7fb77aeea640 1 -- 172.21.15.125:0/2806416460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb774157450 con 0x7fb7740c6b80 2023-10-30T12:38:02.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.296+0000 7fb7717fa640 1 -- 172.21.15.125:0/2806416460 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6549 (secure 0 0 0) 0x7fb760054ea0 con 0x7fb7740c6b80 2023-10-30T12:38:02.299 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:02.299 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:01.249416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:02.299 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 56 2023-10-30T12:38:02.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.296+0000 7fb77aeea640 1 -- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7580664a0 msgr2=0x7fb758068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:02.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.296+0000 7fb77aeea640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7580664a0 0x7fb758068960 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb768000f70 tx=0x7fb768009040 comp rx=0 tx=0).stop 2023-10-30T12:38:02.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.296+0000 7fb77aeea640 1 -- 172.21.15.125:0/2806416460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7740c6b80 msgr2=0x7fb7740c9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:02.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.296+0000 7fb77aeea640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7740c6b80 0x7fb7740c9b70 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb76000d7d0 tx=0x7fb76000dc90 comp rx=0 tx=0).stop 2023-10-30T12:38:02.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.296+0000 7fb77aeea640 1 -- 172.21.15.125:0/2806416460 shutdown_connections 2023-10-30T12:38:02.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.296+0000 7fb77aeea640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7580664a0 0x7fb758068960 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.296+0000 7fb77aeea640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7740c6f60 0x7fb7740ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.296+0000 7fb77aeea640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7740c6b80 0x7fb7740c9b70 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.296+0000 7fb77aeea640 1 --2- 172.21.15.125:0/2806416460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7740c05d0 0x7fb77413d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:02.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.296+0000 7fb77aeea640 1 -- 172.21.15.125:0/2806416460 >> 172.21.15.125:0/2806416460 conn(0x7fb7740c7910 msgr2=0x7fb7740c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:02.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.296+0000 7fb77aeea640 1 -- 172.21.15.125:0/2806416460 shutdown_connections 2023-10-30T12:38:02.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:02.296+0000 7fb77aeea640 1 -- 172.21.15.125:0/2806416460 wait complete. 2023-10-30T12:38:02.319 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:38:04.320 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:04.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f0216e09640 1 Processor -- start 2023-10-30T12:38:04.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f0216e09640 1 -- start start 2023-10-30T12:38:04.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f0216e09640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0210153bf0 0x7f0210071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:04.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f0216e09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02100c6fe0 0x7f02100c7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:04.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f0216e09640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02100c60e0 con 0x7f02100cb550 2023-10-30T12:38:04.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f0216e09640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f021015d870 con 0x7f0210153bf0 2023-10-30T12:38:04.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f0216e09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02100c63a0 con 0x7f02100c6fe0 2023-10-30T12:38:04.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f02153a6640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f02100cb550 0x7f0210159900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34556/0 (socket says 172.21.15.125:34556) 2023-10-30T12:38:04.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f02153a6640 1 -- 172.21.15.125:0/2063660400 learned_addr learned my addr 172.21.15.125:0/2063660400 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:38:04.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020ffff640 1 --2- 172.21.15.125:0/2063660400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02100c6fe0 0x7f02100c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:04.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f0214ba5640 1 --2- 172.21.15.125:0/2063660400 >> v2:172.21.15.136:3300/0 conn(0x7f0210153bf0 0x7f0210071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:04.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 -- 172.21.15.125:0/2063660400 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139869209 0 0) 0x7f02100c60e0 con 0x7f02100cb550 2023-10-30T12:38:04.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 -- 172.21.15.125:0/2063660400 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0200003530 con 0x7f02100cb550 2023-10-30T12:38:04.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 -- 172.21.15.125:0/2063660400 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2542130756 0 0) 0x7f0200003530 con 0x7f02100cb550 2023-10-30T12:38:04.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 -- 172.21.15.125:0/2063660400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02100c6fe0 msgr2=0x7f02100c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:04.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 --2- 172.21.15.125:0/2063660400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02100c6fe0 0x7f02100c7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:04.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 -- 172.21.15.125:0/2063660400 >> v2:172.21.15.136:3300/0 conn(0x7f0210153bf0 msgr2=0x7f0210071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:04.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 --2- 172.21.15.125:0/2063660400 >> v2:172.21.15.136:3300/0 conn(0x7f0210153bf0 0x7f0210071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:04.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 -- 172.21.15.125:0/2063660400 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02100c6680 con 0x7f02100cb550 2023-10-30T12:38:04.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 -- 172.21.15.125:0/2063660400 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f0208001a00 con 0x7f02100cb550 2023-10-30T12:38:04.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 -- 172.21.15.125:0/2063660400 >> v1:172.21.15.125:6789/0 conn(0x7f02100cb550 legacy=0x7f0210159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:04.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 --2- 172.21.15.125:0/2063660400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200003980 0x7f0200003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:04.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 --2- 172.21.15.125:0/2063660400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02000047d0 0x7f0200004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:04.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 --2- 172.21.15.125:0/2063660400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02100c6fe0 0x7f0200005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:04.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 -- 172.21.15.125:0/2063660400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021015d870 con 0x7f0200003980 2023-10-30T12:38:04.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 -- 172.21.15.125:0/2063660400 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02100c63a0 con 0x7f02000047d0 2023-10-30T12:38:04.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020f7fe640 1 -- 172.21.15.125:0/2063660400 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f020000b990 con 0x7f02100c6fe0 2023-10-30T12:38:04.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f0214ba5640 1 --2- 172.21.15.125:0/2063660400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200003980 0x7f0200003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:04.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f020ffff640 1 --2- 172.21.15.125:0/2063660400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02000047d0 0x7f0200004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:04.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f02153a6640 1 --2- 172.21.15.125:0/2063660400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02100c6fe0 0x7f0200005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:04.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f0214ba5640 1 -- 172.21.15.125:0/2063660400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02100c6fe0 msgr2=0x7f0200005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:04.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f0214ba5640 1 --2- 172.21.15.125:0/2063660400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02100c6fe0 0x7f0200005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:04.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f0214ba5640 1 -- 172.21.15.125:0/2063660400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02000047d0 msgr2=0x7f0200004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:04.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f0214ba5640 1 --2- 172.21.15.125:0/2063660400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02000047d0 0x7f0200004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:04.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f0214ba5640 1 -- 172.21.15.125:0/2063660400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0200003530 con 0x7f0200003980 2023-10-30T12:38:04.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.464+0000 7f02153a6640 1 --2- 172.21.15.125:0/2063660400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02100c6fe0 0x7f0200005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:04.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.468+0000 7f0214ba5640 1 --2- 172.21.15.125:0/2063660400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200003980 0x7f0200003da0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0204009ae0 tx=0x7f020401ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76a53106d414431 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:04.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.468+0000 7f020f7fe640 1 -- 172.21.15.125:0/2063660400 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040a5550 con 0x7f0200003980 2023-10-30T12:38:04.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.468+0000 7f020f7fe640 1 -- 172.21.15.125:0/2063660400 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02040a56b0 con 0x7f0200003980 2023-10-30T12:38:04.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.468+0000 7f020f7fe640 1 -- 172.21.15.125:0/2063660400 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040a5a60 con 0x7f0200003980 2023-10-30T12:38:04.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.468+0000 7f0216e09640 1 -- 172.21.15.125:0/2063660400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200003980 msgr2=0x7f0200003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:04.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.468+0000 7f0216e09640 1 --2- 172.21.15.125:0/2063660400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200003980 0x7f0200003da0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0204009ae0 tx=0x7f020401ca60 comp rx=0 tx=0).stop 2023-10-30T12:38:04.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.468+0000 7f02153a6640 1 -- 172.21.15.125:0/2063660400 reap_dead start 2023-10-30T12:38:04.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.468+0000 7f0216e09640 1 -- 172.21.15.125:0/2063660400 shutdown_connections 2023-10-30T12:38:04.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.468+0000 7f0216e09640 1 -- 172.21.15.125:0/2063660400 >> 172.21.15.125:0/2063660400 conn(0x7f021006c6e0 msgr2=0x7f0210069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:04.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.468+0000 7f0216e09640 1 -- 172.21.15.125:0/2063660400 shutdown_connections 2023-10-30T12:38:04.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.468+0000 7f0216e09640 1 -- 172.21.15.125:0/2063660400 wait complete. 2023-10-30T12:38:04.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0216e09640 1 Processor -- start 2023-10-30T12:38:04.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0216e09640 1 -- start start 2023-10-30T12:38:04.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0216e09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200003980 0x7f021015ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:04.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0216e09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02100c9570 0x7f021015f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:04.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0216e09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f021015fdd0 0x7f0210149080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:04.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0216e09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0210159c20 con 0x7f0200003980 2023-10-30T12:38:04.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0216e09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02101587d0 con 0x7f02100c9570 2023-10-30T12:38:04.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0216e09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02101578e0 con 0x7f021015fdd0 2023-10-30T12:38:04.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0214ba5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200003980 0x7f021015ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:04.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0214ba5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200003980 0x7f021015ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38236/0 (socket says 172.21.15.125:38236) 2023-10-30T12:38:04.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0214ba5640 1 -- 172.21.15.125:0/2734570338 learned_addr learned my addr 172.21.15.125:0/2734570338 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:04.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f02153a6640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f021015fdd0 0x7f0210149080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:04.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f020ffff640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02100c9570 0x7f021015f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:04.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0214ba5640 1 -- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f021015fdd0 msgr2=0x7f0210149080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:04.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0214ba5640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f021015fdd0 0x7f0210149080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:04.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0214ba5640 1 -- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02100c9570 msgr2=0x7f021015f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:04.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0214ba5640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02100c9570 0x7f021015f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:04.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0214ba5640 1 -- 172.21.15.125:0/2734570338 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0210157bc0 con 0x7f0200003980 2023-10-30T12:38:04.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f020ffff640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02100c9570 0x7f021015f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:04.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0214ba5640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200003980 0x7f021015ded0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0204009ae0 tx=0x7f0200005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:04.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f020d7fa640 1 -- 172.21.15.125:0/2734570338 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040be070 con 0x7f0200003980 2023-10-30T12:38:04.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f020d7fa640 1 -- 172.21.15.125:0/2734570338 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020400f000 con 0x7f0200003980 2023-10-30T12:38:04.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f020d7fa640 1 -- 172.21.15.125:0/2734570338 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040a5550 con 0x7f0200003980 2023-10-30T12:38:04.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.472+0000 7f0216e09640 1 -- 172.21.15.125:0/2734570338 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0210157ea0 con 0x7f0200003980 2023-10-30T12:38:04.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.476+0000 7f0216e09640 1 -- 172.21.15.125:0/2734570338 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0210158460 con 0x7f0200003980 2023-10-30T12:38:04.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.476+0000 7f0216e09640 1 -- 172.21.15.125:0/2734570338 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f021004e170 con 0x7f0200003980 2023-10-30T12:38:04.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.476+0000 7f020d7fa640 1 -- 172.21.15.125:0/2734570338 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f02040b4040 con 0x7f0200003980 2023-10-30T12:38:04.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.476+0000 7f020d7fa640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01f40664a0 0x7f01f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:04.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.476+0000 7f020d7fa640 1 -- 172.21.15.125:0/2734570338 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f020411da10 con 0x7f0200003980 2023-10-30T12:38:04.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.476+0000 7f020ffff640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01f40664a0 0x7f01f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:04.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.480+0000 7f020ffff640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01f40664a0 0x7f01f4068960 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f021006ff50 tx=0x7f01fc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:04.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.480+0000 7f020d7fa640 1 -- 172.21.15.125:0/2734570338 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f02040ead70 con 0x7f0200003980 2023-10-30T12:38:04.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.660+0000 7f0216e09640 1 -- 172.21.15.125:0/2734570338 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02100cb150 con 0x7f0200003980 2023-10-30T12:38:04.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.660+0000 7f020d7fa640 1 -- 172.21.15.125:0/2734570338 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f02040ee390 con 0x7f0200003980 2023-10-30T12:38:04.662 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:04.662 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:01.249416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:04.663 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 56 2023-10-30T12:38:04.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.664+0000 7f0216e09640 1 -- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01f40664a0 msgr2=0x7f01f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:04.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.664+0000 7f0216e09640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01f40664a0 0x7f01f4068960 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f021006ff50 tx=0x7f01fc008040 comp rx=0 tx=0).stop 2023-10-30T12:38:04.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.664+0000 7f0216e09640 1 -- 172.21.15.125:0/2734570338 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200003980 msgr2=0x7f021015ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:04.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.664+0000 7f0216e09640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200003980 0x7f021015ded0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0204009ae0 tx=0x7f0200005800 comp rx=0 tx=0).stop 2023-10-30T12:38:04.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.664+0000 7f0216e09640 1 -- 172.21.15.125:0/2734570338 shutdown_connections 2023-10-30T12:38:04.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.664+0000 7f0216e09640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01f40664a0 0x7f01f4068960 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:04.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.664+0000 7f0216e09640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f021015fdd0 0x7f0210149080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:04.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.664+0000 7f0216e09640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02100c9570 0x7f021015f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:04.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.664+0000 7f0216e09640 1 --2- 172.21.15.125:0/2734570338 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0200003980 0x7f021015ded0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:04.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.664+0000 7f0216e09640 1 -- 172.21.15.125:0/2734570338 >> 172.21.15.125:0/2734570338 conn(0x7f021006c6e0 msgr2=0x7f02100ca2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:04.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.664+0000 7f0216e09640 1 -- 172.21.15.125:0/2734570338 shutdown_connections 2023-10-30T12:38:04.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:04.664+0000 7f0216e09640 1 -- 172.21.15.125:0/2734570338 wait complete. 2023-10-30T12:38:04.683 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T12:38:04.683 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 56, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 56, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4957": { "addr": "172.21.15.136:6835/2548530941", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2548530941, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2548530941, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4957, "incarnation": 23, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:stopping", "state_seq": 34 }, "gid_4973": { "addr": "172.21.15.136:6839/1454326577", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 1454326577, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 1454326577, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4973, "incarnation": 45, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 21 }, "gid_5130": { "addr": "172.21.15.125:6837/1624923396", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 1624923396, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 1624923396, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5130, "incarnation": 37, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:38:01.249416+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4973, "mds_1": 5130, "mds_2": 4957 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6836/2595438444", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2595438444, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2595438444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 50, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5433, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:38:06.685 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:06.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.832+0000 7fe0dba17640 1 Processor -- start 2023-10-30T12:38:06.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.832+0000 7fe0dba17640 1 -- start start 2023-10-30T12:38:06.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.832+0000 7fe0dba17640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe0d41538a0 0x7fe0d415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:06.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.832+0000 7fe0dba17640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d40cb800 con 0x7fe0d40ca0e0 2023-10-30T12:38:06.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.832+0000 7fe0dba17640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d415dc30 con 0x7fe0d40cf160 2023-10-30T12:38:06.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.832+0000 7fe0dba17640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0d40cb330 con 0x7fe0d41538a0 2023-10-30T12:38:06.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.832+0000 7fe0d97b3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe0d41538a0 0x7fe0d415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:06.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.832+0000 7fe0d8fb2640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe0d40ca0e0 0x7fe0d415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34572/0 (socket says 172.21.15.125:34572) 2023-10-30T12:38:06.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.832+0000 7fe0d8fb2640 1 -- 172.21.15.125:0/2557084976 learned_addr learned my addr 172.21.15.125:0/2557084976 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:38:06.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.832+0000 7fe0d97b3640 1 -- 172.21.15.125:0/2557084976 >> v1:172.21.15.136:6789/0 conn(0x7fe0d40cf160 legacy=0x7fe0d4164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:06.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.832+0000 7fe0cbfff640 1 -- 172.21.15.125:0/2557084976 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766679374 0 0) 0x7fe0d40cb800 con 0x7fe0d40ca0e0 2023-10-30T12:38:06.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.832+0000 7fe0d97b3640 1 -- 172.21.15.125:0/2557084976 >> v1:172.21.15.125:6789/0 conn(0x7fe0d40ca0e0 legacy=0x7fe0d415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:06.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.832+0000 7fe0d97b3640 1 -- 172.21.15.125:0/2557084976 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d40cad70 con 0x7fe0d41538a0 2023-10-30T12:38:06.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.836+0000 7fe0d97b3640 1 --2- 172.21.15.125:0/2557084976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0d41538a0 0x7fe0d415d6f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe0d00097c0 tx=0x7fe0d001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=911d702c02eb92d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:06.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.836+0000 7fe0cbfff640 1 -- 172.21.15.125:0/2557084976 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00a8970 con 0x7fe0d41538a0 2023-10-30T12:38:06.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.836+0000 7fe0cbfff640 1 -- 172.21.15.125:0/2557084976 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d00a8ad0 con 0x7fe0d41538a0 2023-10-30T12:38:06.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.836+0000 7fe0cbfff640 1 -- 172.21.15.125:0/2557084976 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00a8e60 con 0x7fe0d41538a0 2023-10-30T12:38:06.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.836+0000 7fe0dba17640 1 -- 172.21.15.125:0/2557084976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0d41538a0 msgr2=0x7fe0d415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:06.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.836+0000 7fe0dba17640 1 --2- 172.21.15.125:0/2557084976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0d41538a0 0x7fe0d415d6f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe0d00097c0 tx=0x7fe0d001ec70 comp rx=0 tx=0).stop 2023-10-30T12:38:06.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.836+0000 7fe0dba17640 1 -- 172.21.15.125:0/2557084976 shutdown_connections 2023-10-30T12:38:06.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.836+0000 7fe0dba17640 1 --2- 172.21.15.125:0/2557084976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0d41538a0 0x7fe0d415d6f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:06.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.836+0000 7fe0dba17640 1 -- 172.21.15.125:0/2557084976 >> 172.21.15.125:0/2557084976 conn(0x7fe0d406bfe0 msgr2=0x7fe0d406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:06.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.836+0000 7fe0dba17640 1 -- 172.21.15.125:0/2557084976 shutdown_connections 2023-10-30T12:38:06.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.836+0000 7fe0dba17640 1 -- 172.21.15.125:0/2557084976 wait complete. 2023-10-30T12:38:06.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.836+0000 7fe0dba17640 1 Processor -- start 2023-10-30T12:38:06.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0dba17640 1 -- start start 2023-10-30T12:38:06.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0dba17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0d40ca0e0 0x7fe0d40b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:06.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0dba17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0d40cf160 0x7fe0d4148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:06.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0dba17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0d41538a0 0x7fe0d41448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:06.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0dba17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d41e5480 con 0x7fe0d40ca0e0 2023-10-30T12:38:06.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0dba17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d40b4d10 con 0x7fe0d40cf160 2023-10-30T12:38:06.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0dba17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d4144010 con 0x7fe0d41538a0 2023-10-30T12:38:06.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0d97b3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0d40ca0e0 0x7fe0d40b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:06.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0d97b3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0d40ca0e0 0x7fe0d40b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38262/0 (socket says 172.21.15.125:38262) 2023-10-30T12:38:06.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0d97b3640 1 -- 172.21.15.125:0/1415939403 learned_addr learned my addr 172.21.15.125:0/1415939403 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:06.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0d8fb2640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0d40cf160 0x7fe0d4148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:06.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0d9fb4640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0d41538a0 0x7fe0d41448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:06.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0d97b3640 1 -- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0d41538a0 msgr2=0x7fe0d41448b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:06.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0d97b3640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0d41538a0 0x7fe0d41448b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:06.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0d97b3640 1 -- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0d40cf160 msgr2=0x7fe0d4148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:06.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0d97b3640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0d40cf160 0x7fe0d4148080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:06.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0d97b3640 1 -- 172.21.15.125:0/1415939403 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d0009420 con 0x7fe0d40ca0e0 2023-10-30T12:38:06.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0d8fb2640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0d40cf160 0x7fe0d4148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:06.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0d97b3640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0d40ca0e0 0x7fe0d40b47d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe0d001b880 tx=0x7fe0d001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:06.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0c9ffb640 1 -- 172.21.15.125:0/1415939403 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00a8970 con 0x7fe0d40ca0e0 2023-10-30T12:38:06.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0c9ffb640 1 -- 172.21.15.125:0/1415939403 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d00b9410 con 0x7fe0d40ca0e0 2023-10-30T12:38:06.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0c9ffb640 1 -- 172.21.15.125:0/1415939403 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d00b97a0 con 0x7fe0d40ca0e0 2023-10-30T12:38:06.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0dba17640 1 -- 172.21.15.125:0/1415939403 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d4144420 con 0x7fe0d40ca0e0 2023-10-30T12:38:06.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0dba17640 1 -- 172.21.15.125:0/1415939403 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d40b4490 con 0x7fe0d40ca0e0 2023-10-30T12:38:06.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.840+0000 7fe0dba17640 1 -- 172.21.15.125:0/1415939403 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d404e170 con 0x7fe0d40ca0e0 2023-10-30T12:38:06.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.844+0000 7fe0c9ffb640 1 -- 172.21.15.125:0/1415939403 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe0d00b7040 con 0x7fe0d40ca0e0 2023-10-30T12:38:06.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.844+0000 7fe0c9ffb640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0b80664a0 0x7fe0b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:06.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.844+0000 7fe0c9ffb640 1 -- 172.21.15.125:0/1415939403 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fe0d0121cf0 con 0x7fe0d40ca0e0 2023-10-30T12:38:06.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.844+0000 7fe0d8fb2640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0b80664a0 0x7fe0b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:06.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.844+0000 7fe0d8fb2640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0b80664a0 0x7fe0b8068960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4151060 tx=0x7fe0c4007dd0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:06.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:06.848+0000 7fe0c9ffb640 1 -- 172.21.15.125:0/1415939403 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe0d00eefc0 con 0x7fe0d40ca0e0 2023-10-30T12:38:07.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:07.028+0000 7fe0dba17640 1 -- 172.21.15.125:0/1415939403 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0d40666a0 con 0x7fe0d40ca0e0 2023-10-30T12:38:07.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:07.028+0000 7fe0c9ffb640 1 -- 172.21.15.125:0/1415939403 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6549 (secure 0 0 0) 0x7fe0d00b5020 con 0x7fe0d40ca0e0 2023-10-30T12:38:07.029 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:07.029 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:01.249416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:07.031 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 56 2023-10-30T12:38:07.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:07.028+0000 7fe0dba17640 1 -- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0b80664a0 msgr2=0x7fe0b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:07.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:07.028+0000 7fe0dba17640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0b80664a0 0x7fe0b8068960 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4151060 tx=0x7fe0c4007dd0 comp rx=0 tx=0).stop 2023-10-30T12:38:07.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:07.028+0000 7fe0dba17640 1 -- 172.21.15.125:0/1415939403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0d40ca0e0 msgr2=0x7fe0d40b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:07.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:07.028+0000 7fe0dba17640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0d40ca0e0 0x7fe0d40b47d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe0d001b880 tx=0x7fe0d001ba10 comp rx=0 tx=0).stop 2023-10-30T12:38:07.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:07.028+0000 7fe0dba17640 1 -- 172.21.15.125:0/1415939403 shutdown_connections 2023-10-30T12:38:07.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:07.028+0000 7fe0dba17640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0b80664a0 0x7fe0b8068960 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:07.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:07.028+0000 7fe0dba17640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0d41538a0 0x7fe0d41448b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:07.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:07.028+0000 7fe0dba17640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0d40cf160 0x7fe0d4148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:07.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:07.028+0000 7fe0dba17640 1 --2- 172.21.15.125:0/1415939403 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0d40ca0e0 0x7fe0d40b47d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:07.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:07.028+0000 7fe0dba17640 1 -- 172.21.15.125:0/1415939403 >> 172.21.15.125:0/1415939403 conn(0x7fe0d406bfe0 msgr2=0x7fe0d4142980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:07.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:07.028+0000 7fe0dba17640 1 -- 172.21.15.125:0/1415939403 shutdown_connections 2023-10-30T12:38:07.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:07.028+0000 7fe0dba17640 1 -- 172.21.15.125:0/1415939403 wait complete. 2023-10-30T12:38:07.049 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:38:09.050 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:09.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.196+0000 7f1616a17640 1 Processor -- start 2023-10-30T12:38:09.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.196+0000 7f1616a17640 1 -- start start 2023-10-30T12:38:09.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.196+0000 7f1616a17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1610153bf0 0x7f1610071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:09.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.196+0000 7f1616a17640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f16100c5f60 0x7f16100cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:09.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.196+0000 7f1616a17640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16100700b0 con 0x7f16100c6fe0 2023-10-30T12:38:09.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.196+0000 7f1616a17640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16100c6810 con 0x7f16100c5f60 2023-10-30T12:38:09.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.196+0000 7f1616a17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1610159900 con 0x7f1610153bf0 2023-10-30T12:38:09.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.196+0000 7f1614fb4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f16100c5f60 0x7f16100cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:09.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.196+0000 7f1614fb4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f16100c5f60 0x7f16100cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38276/0 (socket says 172.21.15.125:38276) 2023-10-30T12:38:09.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.196+0000 7f1614fb4640 1 -- 172.21.15.125:0/3063624210 learned_addr learned my addr 172.21.15.125:0/3063624210 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:09.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.196+0000 7f160ffff640 1 --2- 172.21.15.125:0/3063624210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1610153bf0 0x7f1610071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:09.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.196+0000 7f1614fb4640 1 -- 172.21.15.125:0/3063624210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1610153bf0 msgr2=0x7f1610071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:09.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.196+0000 7f1614fb4640 1 --2- 172.21.15.125:0/3063624210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1610153bf0 0x7f1610071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:09.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.200+0000 7f1614fb4640 1 -- 172.21.15.125:0/3063624210 >> v1:172.21.15.136:6789/0 conn(0x7f16100c6fe0 legacy=0x7f16100c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:09.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.200+0000 7f1614fb4640 1 -- 172.21.15.125:0/3063624210 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16100c6530 con 0x7f16100c5f60 2023-10-30T12:38:09.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.200+0000 7f1614fb4640 1 --2- 172.21.15.125:0/3063624210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16100c5f60 0x7f16100cb570 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f1608009e50 tx=0x7f160801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc893ecd5ae5ede server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:09.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.200+0000 7f160effd640 1 -- 172.21.15.125:0/3063624210 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16080af400 con 0x7f16100c5f60 2023-10-30T12:38:09.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.200+0000 7f160effd640 1 -- 172.21.15.125:0/3063624210 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16080ab050 con 0x7f16100c5f60 2023-10-30T12:38:09.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.200+0000 7f160effd640 1 -- 172.21.15.125:0/3063624210 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16080af780 con 0x7f16100c5f60 2023-10-30T12:38:09.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.200+0000 7f1616a17640 1 -- 172.21.15.125:0/3063624210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16100c5f60 msgr2=0x7f16100cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:09.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.200+0000 7f1616a17640 1 --2- 172.21.15.125:0/3063624210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16100c5f60 0x7f16100cb570 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f1608009e50 tx=0x7f160801f1f0 comp rx=0 tx=0).stop 2023-10-30T12:38:09.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.200+0000 7f1616a17640 1 -- 172.21.15.125:0/3063624210 shutdown_connections 2023-10-30T12:38:09.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.200+0000 7f1616a17640 1 --2- 172.21.15.125:0/3063624210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16100c5f60 0x7f16100cb570 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:09.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.200+0000 7f1616a17640 1 --2- 172.21.15.125:0/3063624210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1610153bf0 0x7f1610071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:09.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.200+0000 7f1616a17640 1 -- 172.21.15.125:0/3063624210 >> 172.21.15.125:0/3063624210 conn(0x7f161006c6e0 msgr2=0x7f1610069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:09.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.200+0000 7f1616a17640 1 -- 172.21.15.125:0/3063624210 shutdown_connections 2023-10-30T12:38:09.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.200+0000 7f1616a17640 1 -- 172.21.15.125:0/3063624210 wait complete. 2023-10-30T12:38:09.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f1616a17640 1 Processor -- start 2023-10-30T12:38:09.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f1616a17640 1 -- start start 2023-10-30T12:38:09.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f1616a17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16100c5f60 0x7f1610148390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:09.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f1616a17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16100c6fe0 0x7f16101f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:09.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f1616a17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1610153bf0 0x7f16100c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:09.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f1616a17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16101f2850 con 0x7f16100c6fe0 2023-10-30T12:38:09.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f1616a17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16101570a0 con 0x7f16100c5f60 2023-10-30T12:38:09.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f1616a17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16100ca040 con 0x7f1610153bf0 2023-10-30T12:38:09.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f160f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16100c6fe0 0x7f16101f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:09.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f160f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16100c6fe0 0x7f16101f2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38288/0 (socket says 172.21.15.125:38288) 2023-10-30T12:38:09.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f160f7fe640 1 -- 172.21.15.125:0/2607668512 learned_addr learned my addr 172.21.15.125:0/2607668512 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:09.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f160ffff640 1 --2- 172.21.15.125:0/2607668512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16100c5f60 0x7f1610148390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:09.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f1614fb4640 1 --2- 172.21.15.125:0/2607668512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1610153bf0 0x7f16100c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:09.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f160f7fe640 1 -- 172.21.15.125:0/2607668512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1610153bf0 msgr2=0x7f16100c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:09.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f160f7fe640 1 --2- 172.21.15.125:0/2607668512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1610153bf0 0x7f16100c9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:09.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f160f7fe640 1 -- 172.21.15.125:0/2607668512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16100c5f60 msgr2=0x7f1610148390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:09.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f160f7fe640 1 --2- 172.21.15.125:0/2607668512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16100c5f60 0x7f1610148390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:09.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f160f7fe640 1 -- 172.21.15.125:0/2607668512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1608009b30 con 0x7f16100c6fe0 2023-10-30T12:38:09.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f160f7fe640 1 --2- 172.21.15.125:0/2607668512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16100c6fe0 0x7f16101f2ce0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f160400c500 tx=0x7f160400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:09.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.204+0000 7f160cff9640 1 -- 172.21.15.125:0/2607668512 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1604005220 con 0x7f16100c6fe0 2023-10-30T12:38:09.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.208+0000 7f160cff9640 1 -- 172.21.15.125:0/2607668512 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1604005380 con 0x7f16100c6fe0 2023-10-30T12:38:09.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.208+0000 7f160cff9640 1 -- 172.21.15.125:0/2607668512 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1604007270 con 0x7f16100c6fe0 2023-10-30T12:38:09.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.208+0000 7f1616a17640 1 -- 172.21.15.125:0/2607668512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16101f2220 con 0x7f16100c6fe0 2023-10-30T12:38:09.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.208+0000 7f1616a17640 1 -- 172.21.15.125:0/2607668512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16100c4130 con 0x7f16100c6fe0 2023-10-30T12:38:09.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.208+0000 7f1616a17640 1 -- 172.21.15.125:0/2607668512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f161004e170 con 0x7f16100c6fe0 2023-10-30T12:38:09.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.208+0000 7f160cff9640 1 -- 172.21.15.125:0/2607668512 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f16040073d0 con 0x7f16100c6fe0 2023-10-30T12:38:09.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.208+0000 7f160cff9640 1 --2- 172.21.15.125:0/2607668512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15fc0664a0 0x7f15fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:09.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.208+0000 7f160cff9640 1 -- 172.21.15.125:0/2607668512 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f1604083870 con 0x7f16100c6fe0 2023-10-30T12:38:09.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.208+0000 7f160ffff640 1 --2- 172.21.15.125:0/2607668512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15fc0664a0 0x7f15fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:09.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.212+0000 7f160ffff640 1 --2- 172.21.15.125:0/2607668512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15fc0664a0 0x7f15fc068960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f16000079c0 tx=0x7f1600008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:09.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.212+0000 7f160cff9640 1 -- 172.21.15.125:0/2607668512 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1604050b70 con 0x7f16100c6fe0 2023-10-30T12:38:09.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.396+0000 7f1616a17640 1 -- 172.21.15.125:0/2607668512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f161006fe90 con 0x7f16100c6fe0 2023-10-30T12:38:09.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.396+0000 7f160cff9640 1 -- 172.21.15.125:0/2607668512 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f1604054190 con 0x7f16100c6fe0 2023-10-30T12:38:09.396 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:09.397 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:01.249416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:09.398 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 56 2023-10-30T12:38:09.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.396+0000 7f1616a17640 1 -- 172.21.15.125:0/2607668512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15fc0664a0 msgr2=0x7f15fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:09.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.396+0000 7f1616a17640 1 --2- 172.21.15.125:0/2607668512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15fc0664a0 0x7f15fc068960 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f16000079c0 tx=0x7f1600008040 comp rx=0 tx=0).stop 2023-10-30T12:38:09.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.396+0000 7f1616a17640 1 -- 172.21.15.125:0/2607668512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16100c6fe0 msgr2=0x7f16101f2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:09.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.396+0000 7f1616a17640 1 --2- 172.21.15.125:0/2607668512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16100c6fe0 0x7f16101f2ce0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f160400c500 tx=0x7f160400c9c0 comp rx=0 tx=0).stop 2023-10-30T12:38:09.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.396+0000 7f1616a17640 1 -- 172.21.15.125:0/2607668512 shutdown_connections 2023-10-30T12:38:09.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.396+0000 7f1616a17640 1 --2- 172.21.15.125:0/2607668512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15fc0664a0 0x7f15fc068960 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:09.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.396+0000 7f1616a17640 1 --2- 172.21.15.125:0/2607668512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1610153bf0 0x7f16100c9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:09.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.396+0000 7f1616a17640 1 --2- 172.21.15.125:0/2607668512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f16100c6fe0 0x7f16101f2ce0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:09.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.396+0000 7f1616a17640 1 --2- 172.21.15.125:0/2607668512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16100c5f60 0x7f1610148390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:09.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.396+0000 7f1616a17640 1 -- 172.21.15.125:0/2607668512 >> 172.21.15.125:0/2607668512 conn(0x7f161006c6e0 msgr2=0x7f1610157d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:09.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.396+0000 7f1616a17640 1 -- 172.21.15.125:0/2607668512 shutdown_connections 2023-10-30T12:38:09.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:09.396+0000 7f1616a17640 1 -- 172.21.15.125:0/2607668512 wait complete. 2023-10-30T12:38:09.417 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:38:11.418 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:11.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b4a6c9640 1 Processor -- start 2023-10-30T12:38:11.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b4a6c9640 1 -- start start 2023-10-30T12:38:11.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b4a6c9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4b441538a0 0x7f4b4415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:11.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b4a6c9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4b440cf160 0x7f4b440cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:11.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b4a6c9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b440cb800 con 0x7f4b440ca0e0 2023-10-30T12:38:11.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b4a6c9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b4415dc30 con 0x7f4b441538a0 2023-10-30T12:38:11.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b4a6c9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b440cb330 con 0x7f4b440cf160 2023-10-30T12:38:11.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b43fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4b441538a0 0x7f4b4415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:11.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b437fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4b440ca0e0 0x7f4b4415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48362/0 (socket says 172.21.15.125:48362) 2023-10-30T12:38:11.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b437fe640 1 -- 172.21.15.125:0/547053705 learned_addr learned my addr 172.21.15.125:0/547053705 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:38:11.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b48c66640 1 --2- 172.21.15.125:0/547053705 >> v2:172.21.15.136:3300/0 conn(0x7f4b440cf160 0x7f4b440cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:11.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b43fff640 1 -- 172.21.15.125:0/547053705 >> v2:172.21.15.136:3300/0 conn(0x7f4b440cf160 msgr2=0x7f4b440cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:11.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b43fff640 1 --2- 172.21.15.125:0/547053705 >> v2:172.21.15.136:3300/0 conn(0x7f4b440cf160 0x7f4b440cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:11.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b43fff640 1 -- 172.21.15.125:0/547053705 >> v1:172.21.15.125:6789/0 conn(0x7f4b440ca0e0 legacy=0x7f4b4415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:11.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b43fff640 1 -- 172.21.15.125:0/547053705 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b440cad70 con 0x7f4b441538a0 2023-10-30T12:38:11.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.564+0000 7f4b42ffd640 1 -- 172.21.15.125:0/547053705 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776176253 0 0) 0x7f4b440cb800 con 0x7f4b440ca0e0 2023-10-30T12:38:11.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.568+0000 7f4b43fff640 1 --2- 172.21.15.125:0/547053705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b441538a0 0x7f4b4415d6f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4b440b5330 tx=0x7f4b3401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73be4ef9576e3f97 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:11.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.568+0000 7f4b42ffd640 1 -- 172.21.15.125:0/547053705 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b340a8950 con 0x7f4b441538a0 2023-10-30T12:38:11.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.568+0000 7f4b42ffd640 1 -- 172.21.15.125:0/547053705 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b340a8ab0 con 0x7f4b441538a0 2023-10-30T12:38:11.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.568+0000 7f4b42ffd640 1 -- 172.21.15.125:0/547053705 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b340a8e40 con 0x7f4b441538a0 2023-10-30T12:38:11.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.568+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/547053705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b441538a0 msgr2=0x7f4b4415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:11.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.568+0000 7f4b4a6c9640 1 --2- 172.21.15.125:0/547053705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b441538a0 0x7f4b4415d6f0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4b440b5330 tx=0x7f4b3401ec50 comp rx=0 tx=0).stop 2023-10-30T12:38:11.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.568+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/547053705 shutdown_connections 2023-10-30T12:38:11.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.568+0000 7f4b4a6c9640 1 --2- 172.21.15.125:0/547053705 >> v2:172.21.15.136:3300/0 conn(0x7f4b440cf160 0x7f4b440cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:11.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.568+0000 7f4b4a6c9640 1 --2- 172.21.15.125:0/547053705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b441538a0 0x7f4b4415d6f0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:11.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.568+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/547053705 >> 172.21.15.125:0/547053705 conn(0x7f4b4406bfe0 msgr2=0x7f4b4406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:11.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.568+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/547053705 shutdown_connections 2023-10-30T12:38:11.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.568+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/547053705 wait complete. 2023-10-30T12:38:11.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b4a6c9640 1 Processor -- start 2023-10-30T12:38:11.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b4a6c9640 1 -- start start 2023-10-30T12:38:11.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b4a6c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b440ca0e0 0x7f4b441f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:11.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b4a6c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b440cf160 0x7f4b4414b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:11.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b4a6c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b441538a0 0x7f4b44148e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:11.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b4a6c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b441f34c0 con 0x7f4b440cf160 2023-10-30T12:38:11.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b4a6c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4414bb50 con 0x7f4b441538a0 2023-10-30T12:38:11.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b4a6c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4414a570 con 0x7f4b440ca0e0 2023-10-30T12:38:11.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b437fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b440cf160 0x7f4b4414b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:11.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b437fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b440cf160 0x7f4b4414b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59834/0 (socket says 172.21.15.125:59834) 2023-10-30T12:38:11.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b437fe640 1 -- 172.21.15.125:0/3005556105 learned_addr learned my addr 172.21.15.125:0/3005556105 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:11.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b43fff640 1 --2- 172.21.15.125:0/3005556105 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b440ca0e0 0x7f4b441f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:11.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b48c66640 1 --2- 172.21.15.125:0/3005556105 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b441538a0 0x7f4b44148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:11.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b437fe640 1 -- 172.21.15.125:0/3005556105 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b440ca0e0 msgr2=0x7f4b441f2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:11.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b437fe640 1 --2- 172.21.15.125:0/3005556105 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b440ca0e0 0x7f4b441f2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:11.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b437fe640 1 -- 172.21.15.125:0/3005556105 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b441538a0 msgr2=0x7f4b44148e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:11.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b437fe640 1 --2- 172.21.15.125:0/3005556105 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b441538a0 0x7f4b44148e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:11.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b437fe640 1 -- 172.21.15.125:0/3005556105 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b34009420 con 0x7f4b440cf160 2023-10-30T12:38:11.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b437fe640 1 --2- 172.21.15.125:0/3005556105 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b440cf160 0x7f4b4414b610 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4b3800c480 tx=0x7f4b3800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:11.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b40ff9640 1 -- 172.21.15.125:0/3005556105 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b380074e0 con 0x7f4b440cf160 2023-10-30T12:38:11.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b40ff9640 1 -- 172.21.15.125:0/3005556105 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b38007660 con 0x7f4b440cf160 2023-10-30T12:38:11.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/3005556105 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b4414a850 con 0x7f4b440cf160 2023-10-30T12:38:11.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b40ff9640 1 -- 172.21.15.125:0/3005556105 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b38004a30 con 0x7f4b440cf160 2023-10-30T12:38:11.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/3005556105 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4414ae10 con 0x7f4b440cf160 2023-10-30T12:38:11.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.572+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/3005556105 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b440b2a50 con 0x7f4b440cf160 2023-10-30T12:38:11.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.576+0000 7f4b40ff9640 1 -- 172.21.15.125:0/3005556105 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4b38004b90 con 0x7f4b440cf160 2023-10-30T12:38:11.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.576+0000 7f4b40ff9640 1 --2- 172.21.15.125:0/3005556105 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b300664f0 0x7f4b300689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:11.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.576+0000 7f4b40ff9640 1 -- 172.21.15.125:0/3005556105 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f4b38083470 con 0x7f4b440cf160 2023-10-30T12:38:11.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.576+0000 7f4b43fff640 1 --2- 172.21.15.125:0/3005556105 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b300664f0 0x7f4b300689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:11.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.576+0000 7f4b43fff640 1 --2- 172.21.15.125:0/3005556105 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b300664f0 0x7f4b300689b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4b340b2000 tx=0x7f4b340aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:11.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.580+0000 7f4b40ff9640 1 -- 172.21.15.125:0/3005556105 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4b38054400 con 0x7f4b440cf160 2023-10-30T12:38:11.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.760+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/3005556105 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b4404e170 con 0x7f4b440cf160 2023-10-30T12:38:11.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.760+0000 7f4b40ff9640 1 -- 172.21.15.125:0/3005556105 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f4b38018090 con 0x7f4b440cf160 2023-10-30T12:38:11.761 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:11.762 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:01.249416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:11.762 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 56 2023-10-30T12:38:11.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.764+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/3005556105 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b300664f0 msgr2=0x7f4b300689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:11.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.764+0000 7f4b4a6c9640 1 --2- 172.21.15.125:0/3005556105 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b300664f0 0x7f4b300689b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4b340b2000 tx=0x7f4b340aa070 comp rx=0 tx=0).stop 2023-10-30T12:38:11.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.764+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/3005556105 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b440cf160 msgr2=0x7f4b4414b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:11.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.764+0000 7f4b4a6c9640 1 --2- 172.21.15.125:0/3005556105 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b440cf160 0x7f4b4414b610 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4b3800c480 tx=0x7f4b3800c940 comp rx=0 tx=0).stop 2023-10-30T12:38:11.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.764+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/3005556105 shutdown_connections 2023-10-30T12:38:11.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.764+0000 7f4b4a6c9640 1 --2- 172.21.15.125:0/3005556105 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b300664f0 0x7f4b300689b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:11.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.764+0000 7f4b4a6c9640 1 --2- 172.21.15.125:0/3005556105 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b441538a0 0x7f4b44148e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:11.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.764+0000 7f4b4a6c9640 1 --2- 172.21.15.125:0/3005556105 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b440cf160 0x7f4b4414b610 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:11.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.764+0000 7f4b4a6c9640 1 --2- 172.21.15.125:0/3005556105 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b440ca0e0 0x7f4b441f2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:11.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.764+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/3005556105 >> 172.21.15.125:0/3005556105 conn(0x7f4b4406bfe0 msgr2=0x7f4b441513e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:11.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.764+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/3005556105 shutdown_connections 2023-10-30T12:38:11.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:11.764+0000 7f4b4a6c9640 1 -- 172.21.15.125:0/3005556105 wait complete. 2023-10-30T12:38:11.782 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:38:13.783 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:13.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.928+0000 7f9a7099d640 1 Processor -- start 2023-10-30T12:38:13.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.928+0000 7f9a7099d640 1 -- start start 2023-10-30T12:38:13.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.932+0000 7f9a7099d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9a6c1538a0 0x7f9a6c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:13.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.932+0000 7f9a7099d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9a6c0ca0e0 0x7f9a6c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:13.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.932+0000 7f9a7099d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a6c0cb800 con 0x7f9a6c0cf160 2023-10-30T12:38:13.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.932+0000 7f9a7099d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6c15dc30 con 0x7f9a6c1538a0 2023-10-30T12:38:13.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.932+0000 7f9a7099d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c0cb330 con 0x7f9a6c0ca0e0 2023-10-30T12:38:13.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.932+0000 7f9a6a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9a6c1538a0 0x7f9a6c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:13.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.932+0000 7f9a6a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9a6c1538a0 0x7f9a6c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59844/0 (socket says 172.21.15.125:59844) 2023-10-30T12:38:13.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.932+0000 7f9a6a59c640 1 -- 172.21.15.125:0/2301266003 learned_addr learned my addr 172.21.15.125:0/2301266003 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:13.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.932+0000 7f9a6ad9d640 1 --1- 172.21.15.125:0/2301266003 >> v1:172.21.15.125:6789/0 conn(0x7f9a6c0cf160 0x7f9a6c0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48370/0 (socket says 172.21.15.125:48370) 2023-10-30T12:38:13.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.932+0000 7f9a69d9b640 1 --2- 172.21.15.125:0/2301266003 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9a6c0ca0e0 0x7f9a6c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:13.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.932+0000 7f9a6a59c640 1 -- 172.21.15.125:0/2301266003 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9a6c0ca0e0 msgr2=0x7f9a6c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:13.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.932+0000 7f9a6a59c640 1 --2- 172.21.15.125:0/2301266003 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9a6c0ca0e0 0x7f9a6c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:13.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.932+0000 7f9a6a59c640 1 -- 172.21.15.125:0/2301266003 >> v1:172.21.15.125:6789/0 conn(0x7f9a6c0cf160 legacy=0x7f9a6c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:13.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.932+0000 7f9a6a59c640 1 -- 172.21.15.125:0/2301266003 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6c0cad70 con 0x7f9a6c1538a0 2023-10-30T12:38:13.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.936+0000 7f9a6a59c640 1 --2- 172.21.15.125:0/2301266003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a6c1538a0 0x7f9a6c15d6f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009870 tx=0x7f9a5c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5b5be8c674a365 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:13.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.936+0000 7f9a6959a640 1 -- 172.21.15.125:0/2301266003 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c0a8a30 con 0x7f9a6c1538a0 2023-10-30T12:38:13.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.936+0000 7f9a6959a640 1 -- 172.21.15.125:0/2301266003 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5c0a8b90 con 0x7f9a6c1538a0 2023-10-30T12:38:13.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.936+0000 7f9a6959a640 1 -- 172.21.15.125:0/2301266003 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c0b0440 con 0x7f9a6c1538a0 2023-10-30T12:38:13.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.936+0000 7f9a7099d640 1 -- 172.21.15.125:0/2301266003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a6c1538a0 msgr2=0x7f9a6c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:13.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.936+0000 7f9a7099d640 1 --2- 172.21.15.125:0/2301266003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a6c1538a0 0x7f9a6c15d6f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009870 tx=0x7f9a5c01ed30 comp rx=0 tx=0).stop 2023-10-30T12:38:13.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.936+0000 7f9a7099d640 1 -- 172.21.15.125:0/2301266003 shutdown_connections 2023-10-30T12:38:13.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.936+0000 7f9a7099d640 1 --2- 172.21.15.125:0/2301266003 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9a6c0ca0e0 0x7f9a6c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:13.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.936+0000 7f9a7099d640 1 --2- 172.21.15.125:0/2301266003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a6c1538a0 0x7f9a6c15d6f0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:13.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.936+0000 7f9a7099d640 1 -- 172.21.15.125:0/2301266003 >> 172.21.15.125:0/2301266003 conn(0x7f9a6c06bfe0 msgr2=0x7f9a6c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:13.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.936+0000 7f9a7099d640 1 -- 172.21.15.125:0/2301266003 shutdown_connections 2023-10-30T12:38:13.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.936+0000 7f9a7099d640 1 -- 172.21.15.125:0/2301266003 wait complete. 2023-10-30T12:38:13.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.936+0000 7f9a7099d640 1 Processor -- start 2023-10-30T12:38:13.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.936+0000 7f9a7099d640 1 -- start start 2023-10-30T12:38:13.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a7099d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a6c0ca0e0 0x7f9a6c15b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:13.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a7099d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a6c0cf160 0x7f9a6c1e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:13.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a7099d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9a6c1538a0 0x7f9a6c1e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:13.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a7099d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c1e5580 con 0x7f9a6c0cf160 2023-10-30T12:38:13.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a7099d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c15bc60 con 0x7f9a6c0ca0e0 2023-10-30T12:38:13.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a7099d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6c1e4670 con 0x7f9a6c1538a0 2023-10-30T12:38:13.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a69d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a6c0cf160 0x7f9a6c1e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:13.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a69d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a6c0cf160 0x7f9a6c1e4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59852/0 (socket says 172.21.15.125:59852) 2023-10-30T12:38:13.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a69d9b640 1 -- 172.21.15.125:0/829949471 learned_addr learned my addr 172.21.15.125:0/829949471 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:13.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a6ad9d640 1 --2- 172.21.15.125:0/829949471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9a6c1538a0 0x7f9a6c1e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:13.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a6a59c640 1 --2- 172.21.15.125:0/829949471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a6c0ca0e0 0x7f9a6c15b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:13.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a69d9b640 1 -- 172.21.15.125:0/829949471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9a6c1538a0 msgr2=0x7f9a6c1e5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:13.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a69d9b640 1 --2- 172.21.15.125:0/829949471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9a6c1538a0 0x7f9a6c1e5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:13.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a69d9b640 1 -- 172.21.15.125:0/829949471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a6c0ca0e0 msgr2=0x7f9a6c15b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:13.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a69d9b640 1 --2- 172.21.15.125:0/829949471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a6c0ca0e0 0x7f9a6c15b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:13.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a69d9b640 1 -- 172.21.15.125:0/829949471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c0094d0 con 0x7f9a6c0cf160 2023-10-30T12:38:13.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a69d9b640 1 --2- 172.21.15.125:0/829949471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a6c0cf160 0x7f9a6c1e4b00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9a6000d860 tx=0x7f9a6000dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:13.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a5affd640 1 -- 172.21.15.125:0/829949471 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6000de50 con 0x7f9a6c0cf160 2023-10-30T12:38:13.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a5affd640 1 -- 172.21.15.125:0/829949471 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6000b8d0 con 0x7f9a6c0cf160 2023-10-30T12:38:13.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a5affd640 1 -- 172.21.15.125:0/829949471 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6000bc40 con 0x7f9a6c0cf160 2023-10-30T12:38:13.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a7099d640 1 -- 172.21.15.125:0/829949471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6c158840 con 0x7f9a6c0cf160 2023-10-30T12:38:13.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a7099d640 1 -- 172.21.15.125:0/829949471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6c158ae0 con 0x7f9a6c0cf160 2023-10-30T12:38:13.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.940+0000 7f9a7099d640 1 -- 172.21.15.125:0/829949471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a6c0b2a50 con 0x7f9a6c0cf160 2023-10-30T12:38:13.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.944+0000 7f9a5affd640 1 -- 172.21.15.125:0/829949471 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9a6001e460 con 0x7f9a6c0cf160 2023-10-30T12:38:13.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.944+0000 7f9a5affd640 1 --2- 172.21.15.125:0/829949471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9a4c0664a0 0x7f9a4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:13.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.944+0000 7f9a5affd640 1 -- 172.21.15.125:0/829949471 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f9a600846c0 con 0x7f9a6c0cf160 2023-10-30T12:38:13.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.944+0000 7f9a6a59c640 1 --2- 172.21.15.125:0/829949471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9a4c0664a0 0x7f9a4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:13.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.944+0000 7f9a6a59c640 1 --2- 172.21.15.125:0/829949471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9a4c0664a0 0x7f9a4c068960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c0b2000 tx=0x7f9a5c0aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:13.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:13.948+0000 7f9a5affd640 1 -- 172.21.15.125:0/829949471 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9a60051870 con 0x7f9a6c0cf160 2023-10-30T12:38:14.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:14.128+0000 7f9a7099d640 1 -- 172.21.15.125:0/829949471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a6c04e170 con 0x7f9a6c0cf160 2023-10-30T12:38:14.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:14.128+0000 7f9a5affd640 1 -- 172.21.15.125:0/829949471 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f9a60054e90 con 0x7f9a6c0cf160 2023-10-30T12:38:14.127 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:14.128 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:01.249416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:14.128 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 56 2023-10-30T12:38:14.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:14.128+0000 7f9a7099d640 1 -- 172.21.15.125:0/829949471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9a4c0664a0 msgr2=0x7f9a4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:14.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:14.128+0000 7f9a7099d640 1 --2- 172.21.15.125:0/829949471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9a4c0664a0 0x7f9a4c068960 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c0b2000 tx=0x7f9a5c0aa070 comp rx=0 tx=0).stop 2023-10-30T12:38:14.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:14.128+0000 7f9a7099d640 1 -- 172.21.15.125:0/829949471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a6c0cf160 msgr2=0x7f9a6c1e4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:14.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:14.128+0000 7f9a7099d640 1 --2- 172.21.15.125:0/829949471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a6c0cf160 0x7f9a6c1e4b00 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9a6000d860 tx=0x7f9a6000dd20 comp rx=0 tx=0).stop 2023-10-30T12:38:14.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:14.128+0000 7f9a7099d640 1 -- 172.21.15.125:0/829949471 shutdown_connections 2023-10-30T12:38:14.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:14.128+0000 7f9a7099d640 1 --2- 172.21.15.125:0/829949471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9a4c0664a0 0x7f9a4c068960 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:14.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:14.128+0000 7f9a7099d640 1 --2- 172.21.15.125:0/829949471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9a6c1538a0 0x7f9a6c1e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:14.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:14.128+0000 7f9a7099d640 1 --2- 172.21.15.125:0/829949471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9a6c0cf160 0x7f9a6c1e4b00 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:14.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:14.128+0000 7f9a7099d640 1 --2- 172.21.15.125:0/829949471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9a6c0ca0e0 0x7f9a6c15b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:14.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:14.128+0000 7f9a7099d640 1 -- 172.21.15.125:0/829949471 >> 172.21.15.125:0/829949471 conn(0x7f9a6c06bfe0 msgr2=0x7f9a6c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:14.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:14.128+0000 7f9a7099d640 1 -- 172.21.15.125:0/829949471 shutdown_connections 2023-10-30T12:38:14.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:14.128+0000 7f9a7099d640 1 -- 172.21.15.125:0/829949471 wait complete. 2023-10-30T12:38:14.148 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:38:15.104 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:38:15.109 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:38:16.149 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:16.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.300+0000 7f1e262c9640 1 Processor -- start 2023-10-30T12:38:16.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.300+0000 7f1e262c9640 1 -- start start 2023-10-30T12:38:16.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.300+0000 7f1e262c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e20070740 0x7f1e200c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:16.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.300+0000 7f1e262c9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1e2015e700 0x7f1e2006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:16.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.300+0000 7f1e262c9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e200c5810 con 0x7f1e201580e0 2023-10-30T12:38:16.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.300+0000 7f1e262c9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e20070b60 con 0x7f1e2015e700 2023-10-30T12:38:16.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.300+0000 7f1e262c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2015eb90 con 0x7f1e20070740 2023-10-30T12:38:16.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.300+0000 7f1e1f7fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e20070740 0x7f1e200c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:16.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.300+0000 7f1e1f7fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e20070740 0x7f1e200c6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:36106/0 (socket says 172.21.15.125:36106) 2023-10-30T12:38:16.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.300+0000 7f1e1f7fe640 1 -- 172.21.15.125:0/2377987266 learned_addr learned my addr 172.21.15.125:0/2377987266 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:16.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.300+0000 7f1e24866640 1 --2- 172.21.15.125:0/2377987266 >> v2:172.21.15.136:3300/0 conn(0x7f1e2015e700 0x7f1e2006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:16.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.304+0000 7f1e1f7fe640 1 -- 172.21.15.125:0/2377987266 >> v2:172.21.15.136:3300/0 conn(0x7f1e2015e700 msgr2=0x7f1e2006a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:16.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.304+0000 7f1e1f7fe640 1 --2- 172.21.15.125:0/2377987266 >> v2:172.21.15.136:3300/0 conn(0x7f1e2015e700 0x7f1e2006a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:16.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.304+0000 7f1e1f7fe640 1 -- 172.21.15.125:0/2377987266 >> v1:172.21.15.136:6789/0 conn(0x7f1e201580e0 legacy=0x7f1e20069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:16.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.304+0000 7f1e1f7fe640 1 -- 172.21.15.125:0/2377987266 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e200c6bb0 con 0x7f1e20070740 2023-10-30T12:38:16.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.304+0000 7f1e1f7fe640 1 --2- 172.21.15.125:0/2377987266 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e20070740 0x7f1e200c6540 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1e14009870 tx=0x7f1e1401c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c2626eb48f3381b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:16.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.304+0000 7f1e1effd640 1 -- 172.21.15.125:0/2377987266 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e140aa670 con 0x7f1e20070740 2023-10-30T12:38:16.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.304+0000 7f1e1effd640 1 -- 172.21.15.125:0/2377987266 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e140aa7d0 con 0x7f1e20070740 2023-10-30T12:38:16.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.304+0000 7f1e1effd640 1 -- 172.21.15.125:0/2377987266 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e140aab60 con 0x7f1e20070740 2023-10-30T12:38:16.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 -- 172.21.15.125:0/2377987266 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e20070740 msgr2=0x7f1e200c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:16.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 --2- 172.21.15.125:0/2377987266 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e20070740 0x7f1e200c6540 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1e14009870 tx=0x7f1e1401c510 comp rx=0 tx=0).stop 2023-10-30T12:38:16.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 -- 172.21.15.125:0/2377987266 shutdown_connections 2023-10-30T12:38:16.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 --2- 172.21.15.125:0/2377987266 >> v2:172.21.15.136:3300/0 conn(0x7f1e2015e700 0x7f1e2006a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:16.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 --2- 172.21.15.125:0/2377987266 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e20070740 0x7f1e200c6540 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:16.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 -- 172.21.15.125:0/2377987266 >> 172.21.15.125:0/2377987266 conn(0x7f1e200c7910 msgr2=0x7f1e200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:16.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 -- 172.21.15.125:0/2377987266 shutdown_connections 2023-10-30T12:38:16.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 -- 172.21.15.125:0/2377987266 wait complete. 2023-10-30T12:38:16.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 Processor -- start 2023-10-30T12:38:16.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 -- start start 2023-10-30T12:38:16.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e20070740 0x7f1e2014d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:16.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e201580e0 0x7f1e2015baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:16.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e2015e700 0x7f1e2014b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:16.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e200cacb0 con 0x7f1e2015e700 2023-10-30T12:38:16.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2015df10 con 0x7f1e20070740 2023-10-30T12:38:16.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e262c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2015d990 con 0x7f1e201580e0 2023-10-30T12:38:16.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e24866640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e2015e700 0x7f1e2014b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:16.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e24866640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e2015e700 0x7f1e2014b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59864/0 (socket says 172.21.15.125:59864) 2023-10-30T12:38:16.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e24866640 1 -- 172.21.15.125:0/1242891682 learned_addr learned my addr 172.21.15.125:0/1242891682 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:16.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e1f7fe640 1 --2- 172.21.15.125:0/1242891682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e201580e0 0x7f1e2015baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:16.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.308+0000 7f1e1ffff640 1 --2- 172.21.15.125:0/1242891682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e20070740 0x7f1e2014d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:16.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e24866640 1 -- 172.21.15.125:0/1242891682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e201580e0 msgr2=0x7f1e2015baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:16.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e24866640 1 --2- 172.21.15.125:0/1242891682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e201580e0 0x7f1e2015baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:16.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e24866640 1 -- 172.21.15.125:0/1242891682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e20070740 msgr2=0x7f1e2014d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:16.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e24866640 1 --2- 172.21.15.125:0/1242891682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e20070740 0x7f1e2014d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:16.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e24866640 1 -- 172.21.15.125:0/1242891682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e140094d0 con 0x7f1e2015e700 2023-10-30T12:38:16.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e24866640 1 --2- 172.21.15.125:0/1242891682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e2015e700 0x7f1e2014b300 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800efc0 tx=0x7f1e1800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:16.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e1cff9640 1 -- 172.21.15.125:0/1242891682 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e18009ba0 con 0x7f1e2015e700 2023-10-30T12:38:16.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e1cff9640 1 -- 172.21.15.125:0/1242891682 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1800f040 con 0x7f1e2015e700 2023-10-30T12:38:16.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e1cff9640 1 -- 172.21.15.125:0/1242891682 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e180146c0 con 0x7f1e2015e700 2023-10-30T12:38:16.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e262c9640 1 -- 172.21.15.125:0/1242891682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2015dc30 con 0x7f1e2015e700 2023-10-30T12:38:16.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e262c9640 1 -- 172.21.15.125:0/1242891682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e200ca8d0 con 0x7f1e2015e700 2023-10-30T12:38:16.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e262c9640 1 -- 172.21.15.125:0/1242891682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2004e170 con 0x7f1e2015e700 2023-10-30T12:38:16.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e1cff9640 1 -- 172.21.15.125:0/1242891682 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1e18014820 con 0x7f1e2015e700 2023-10-30T12:38:16.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e1cff9640 1 --2- 172.21.15.125:0/1242891682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1e0c0664a0 0x7f1e0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:16.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e1cff9640 1 -- 172.21.15.125:0/1242891682 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f1e180849f0 con 0x7f1e2015e700 2023-10-30T12:38:16.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.312+0000 7f1e1ffff640 1 --2- 172.21.15.125:0/1242891682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1e0c0664a0 0x7f1e0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:16.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.316+0000 7f1e1ffff640 1 --2- 172.21.15.125:0/1242891682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1e0c0664a0 0x7f1e0c068960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1e10007fd0 tx=0x7f1e10009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:16.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.316+0000 7f1e1cff9640 1 -- 172.21.15.125:0/1242891682 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1e18051d70 con 0x7f1e2015e700 2023-10-30T12:38:16.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.496+0000 7f1e262c9640 1 -- 172.21.15.125:0/1242891682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e201572f0 con 0x7f1e2015e700 2023-10-30T12:38:16.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.496+0000 7f1e1cff9640 1 -- 172.21.15.125:0/1242891682 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f1e18018090 con 0x7f1e2015e700 2023-10-30T12:38:16.497 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:16.497 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:01.249416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:16.499 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 56 2023-10-30T12:38:16.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.500+0000 7f1e262c9640 1 -- 172.21.15.125:0/1242891682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1e0c0664a0 msgr2=0x7f1e0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:16.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.500+0000 7f1e262c9640 1 --2- 172.21.15.125:0/1242891682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1e0c0664a0 0x7f1e0c068960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1e10007fd0 tx=0x7f1e10009040 comp rx=0 tx=0).stop 2023-10-30T12:38:16.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.500+0000 7f1e262c9640 1 -- 172.21.15.125:0/1242891682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e2015e700 msgr2=0x7f1e2014b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:16.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.500+0000 7f1e262c9640 1 --2- 172.21.15.125:0/1242891682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e2015e700 0x7f1e2014b300 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1e1800efc0 tx=0x7f1e1800c490 comp rx=0 tx=0).stop 2023-10-30T12:38:16.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.500+0000 7f1e262c9640 1 -- 172.21.15.125:0/1242891682 shutdown_connections 2023-10-30T12:38:16.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.500+0000 7f1e262c9640 1 --2- 172.21.15.125:0/1242891682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1e0c0664a0 0x7f1e0c068960 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:16.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.500+0000 7f1e262c9640 1 --2- 172.21.15.125:0/1242891682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e2015e700 0x7f1e2014b300 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:16.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.500+0000 7f1e262c9640 1 --2- 172.21.15.125:0/1242891682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e201580e0 0x7f1e2015baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:16.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.500+0000 7f1e262c9640 1 --2- 172.21.15.125:0/1242891682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e20070740 0x7f1e2014d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:16.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.500+0000 7f1e262c9640 1 -- 172.21.15.125:0/1242891682 >> 172.21.15.125:0/1242891682 conn(0x7f1e200c7910 msgr2=0x7f1e20071bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:16.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.500+0000 7f1e262c9640 1 -- 172.21.15.125:0/1242891682 shutdown_connections 2023-10-30T12:38:16.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:16.500+0000 7f1e262c9640 1 -- 172.21.15.125:0/1242891682 wait complete. 2023-10-30T12:38:16.518 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T12:38:16.519 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 56, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 56, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4957": { "addr": "172.21.15.136:6835/2548530941", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2548530941, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2548530941, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4957, "incarnation": 23, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:stopping", "state_seq": 34 }, "gid_4973": { "addr": "172.21.15.136:6839/1454326577", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 1454326577, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 1454326577, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4973, "incarnation": 45, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 21 }, "gid_5130": { "addr": "172.21.15.125:6837/1624923396", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 1624923396, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 1624923396, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5130, "incarnation": 37, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:38:01.249416+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4973, "mds_1": 5130, "mds_2": 4957 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6836/2595438444", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2595438444, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2595438444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 50, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5433, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:38:18.519 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:18.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.663+0000 7f8b3bf9d640 1 Processor -- start 2023-10-30T12:38:18.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.663+0000 7f8b3bf9d640 1 -- start start 2023-10-30T12:38:18.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b3bf9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b340c05d0 0x7f8b3413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:18.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b3bf9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8b340c6f60 0x7f8b3413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:18.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b3bf9d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b340c0a50 con 0x7f8b340c6b80 2023-10-30T12:38:18.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b3bf9d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b3406f6e0 con 0x7f8b340c6f60 2023-10-30T12:38:18.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b3bf9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3406f9c0 con 0x7f8b340c05d0 2023-10-30T12:38:18.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b39d39640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8b340c6b80 0x7f8b340bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48386/0 (socket says 172.21.15.125:48386) 2023-10-30T12:38:18.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b39d39640 1 -- 172.21.15.125:0/1101663966 learned_addr learned my addr 172.21.15.125:0/1101663966 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:38:18.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b3a53a640 1 --2- 172.21.15.125:0/1101663966 >> v2:172.21.15.136:3300/0 conn(0x7f8b340c6f60 0x7f8b3413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:18.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b39538640 1 --2- 172.21.15.125:0/1101663966 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b340c05d0 0x7f8b3413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:18.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 -- 172.21.15.125:0/1101663966 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401113643 0 0) 0x7f8b340c0a50 con 0x7f8b340c6b80 2023-10-30T12:38:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 -- 172.21.15.125:0/1101663966 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b20003530 con 0x7f8b340c6b80 2023-10-30T12:38:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 -- 172.21.15.125:0/1101663966 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2272970266 0 0) 0x7f8b20003530 con 0x7f8b340c6b80 2023-10-30T12:38:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 -- 172.21.15.125:0/1101663966 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b340c05d0 msgr2=0x7f8b3413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 --2- 172.21.15.125:0/1101663966 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b340c05d0 0x7f8b3413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:18.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 -- 172.21.15.125:0/1101663966 >> v2:172.21.15.136:3300/0 conn(0x7f8b340c6f60 msgr2=0x7f8b3413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 --2- 172.21.15.125:0/1101663966 >> v2:172.21.15.136:3300/0 conn(0x7f8b340c6f60 0x7f8b3413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 -- 172.21.15.125:0/1101663966 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3406fca0 con 0x7f8b340c6b80 2023-10-30T12:38:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 -- 172.21.15.125:0/1101663966 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f8b30001680 con 0x7f8b340c6b80 2023-10-30T12:38:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 -- 172.21.15.125:0/1101663966 >> v1:172.21.15.125:6789/0 conn(0x7f8b340c6b80 legacy=0x7f8b340bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:18.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 --2- 172.21.15.125:0/1101663966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b20003980 0x7f8b20003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:18.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 --2- 172.21.15.125:0/1101663966 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b200047d0 0x7f8b20004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:18.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 --2- 172.21.15.125:0/1101663966 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b340c05d0 0x7f8b20005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:18.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 -- 172.21.15.125:0/1101663966 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3406f6e0 con 0x7f8b200047d0 2023-10-30T12:38:18.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 -- 172.21.15.125:0/1101663966 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3406f9c0 con 0x7f8b20003980 2023-10-30T12:38:18.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b38d37640 1 -- 172.21.15.125:0/1101663966 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2000b990 con 0x7f8b340c05d0 2023-10-30T12:38:18.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b39538640 1 --2- 172.21.15.125:0/1101663966 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b200047d0 0x7f8b20004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:18.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b39d39640 1 --2- 172.21.15.125:0/1101663966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b20003980 0x7f8b20003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:18.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b3a53a640 1 --2- 172.21.15.125:0/1101663966 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b340c05d0 0x7f8b20005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:18.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b39538640 1 -- 172.21.15.125:0/1101663966 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b340c05d0 msgr2=0x7f8b20005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:18.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b39538640 1 --2- 172.21.15.125:0/1101663966 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b340c05d0 0x7f8b20005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:18.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b39538640 1 -- 172.21.15.125:0/1101663966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b20003980 msgr2=0x7f8b20003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:18.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b39538640 1 --2- 172.21.15.125:0/1101663966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b20003980 0x7f8b20003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:18.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.667+0000 7f8b39538640 1 -- 172.21.15.125:0/1101663966 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b20003530 con 0x7f8b200047d0 2023-10-30T12:38:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.671+0000 7f8b39538640 1 --2- 172.21.15.125:0/1101663966 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b200047d0 0x7f8b20004c50 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009ae0 tx=0x7f8b2401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d147904b1d897dd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.671+0000 7f8b38d37640 1 -- 172.21.15.125:0/1101663966 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b240a5560 con 0x7f8b200047d0 2023-10-30T12:38:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.671+0000 7f8b38d37640 1 -- 172.21.15.125:0/1101663966 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b240a56c0 con 0x7f8b200047d0 2023-10-30T12:38:18.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.671+0000 7f8b38d37640 1 -- 172.21.15.125:0/1101663966 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b240a5a70 con 0x7f8b200047d0 2023-10-30T12:38:18.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.671+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1101663966 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b200047d0 msgr2=0x7f8b20004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:18.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.671+0000 7f8b3bf9d640 1 --2- 172.21.15.125:0/1101663966 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b200047d0 0x7f8b20004c50 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f8b24009ae0 tx=0x7f8b2401ca50 comp rx=0 tx=0).stop 2023-10-30T12:38:18.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.671+0000 7f8b3a53a640 1 -- 172.21.15.125:0/1101663966 reap_dead start 2023-10-30T12:38:18.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.671+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1101663966 shutdown_connections 2023-10-30T12:38:18.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.671+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1101663966 >> 172.21.15.125:0/1101663966 conn(0x7f8b340c7910 msgr2=0x7f8b340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:18.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.671+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1101663966 shutdown_connections 2023-10-30T12:38:18.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.671+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1101663966 wait complete. 2023-10-30T12:38:18.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b3bf9d640 1 Processor -- start 2023-10-30T12:38:18.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b3bf9d640 1 -- start start 2023-10-30T12:38:18.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b3bf9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b200047d0 0x7f8b3413c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:18.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b3bf9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b34137f20 0x7f8b3413d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:18.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b3bf9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b3413dbd0 0x7f8b341510d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:18.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b3bf9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b34156740 con 0x7f8b200047d0 2023-10-30T12:38:18.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b3bf9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b34155880 con 0x7f8b34137f20 2023-10-30T12:38:18.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b3bf9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3413e060 con 0x7f8b3413dbd0 2023-10-30T12:38:18.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b39d39640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b200047d0 0x7f8b3413c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:18.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b39d39640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b200047d0 0x7f8b3413c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59880/0 (socket says 172.21.15.125:59880) 2023-10-30T12:38:18.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b39d39640 1 -- 172.21.15.125:0/1006906745 learned_addr learned my addr 172.21.15.125:0/1006906745 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:18.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b39538640 1 --2- 172.21.15.125:0/1006906745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b34137f20 0x7f8b3413d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:18.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b3a53a640 1 --2- 172.21.15.125:0/1006906745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b3413dbd0 0x7f8b341510d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:18.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b39d39640 1 -- 172.21.15.125:0/1006906745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b3413dbd0 msgr2=0x7f8b341510d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:18.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b39d39640 1 --2- 172.21.15.125:0/1006906745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b3413dbd0 0x7f8b341510d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:18.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b39d39640 1 -- 172.21.15.125:0/1006906745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b34137f20 msgr2=0x7f8b3413d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:18.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b39d39640 1 --2- 172.21.15.125:0/1006906745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b34137f20 0x7f8b3413d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:18.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b39d39640 1 -- 172.21.15.125:0/1006906745 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b34157160 con 0x7f8b200047d0 2023-10-30T12:38:18.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b39d39640 1 --2- 172.21.15.125:0/1006906745 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b200047d0 0x7f8b3413c870 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8b300125a0 tx=0x7f8b30012a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:18.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b2a7fc640 1 -- 172.21.15.125:0/1006906745 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30014810 con 0x7f8b200047d0 2023-10-30T12:38:18.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b2a7fc640 1 -- 172.21.15.125:0/1006906745 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b30014970 con 0x7f8b200047d0 2023-10-30T12:38:18.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1006906745 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b34154f50 con 0x7f8b200047d0 2023-10-30T12:38:18.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b2a7fc640 1 -- 172.21.15.125:0/1006906745 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b30014d20 con 0x7f8b200047d0 2023-10-30T12:38:18.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1006906745 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b34155510 con 0x7f8b200047d0 2023-10-30T12:38:18.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.675+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1006906745 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b3404e170 con 0x7f8b200047d0 2023-10-30T12:38:18.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.679+0000 7f8b2a7fc640 1 -- 172.21.15.125:0/1006906745 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8b30016480 con 0x7f8b200047d0 2023-10-30T12:38:18.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.679+0000 7f8b2a7fc640 1 --2- 172.21.15.125:0/1006906745 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b180664a0 0x7f8b18068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:18.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.679+0000 7f8b2a7fc640 1 -- 172.21.15.125:0/1006906745 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f8b300873c0 con 0x7f8b200047d0 2023-10-30T12:38:18.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.679+0000 7f8b39538640 1 --2- 172.21.15.125:0/1006906745 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b180664a0 0x7f8b18068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:18.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.679+0000 7f8b39538640 1 --2- 172.21.15.125:0/1006906745 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b180664a0 0x7f8b18068960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8b240af000 tx=0x7f8b240046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:18.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.683+0000 7f8b2a7fc640 1 -- 172.21.15.125:0/1006906745 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8b300546e0 con 0x7f8b200047d0 2023-10-30T12:38:18.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.863+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1006906745 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b3413d2b0 con 0x7f8b200047d0 2023-10-30T12:38:18.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.863+0000 7f8b2a7fc640 1 -- 172.21.15.125:0/1006906745 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f8b30057d00 con 0x7f8b200047d0 2023-10-30T12:38:18.867 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:18.867 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:01.249416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:18.868 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 56 2023-10-30T12:38:18.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.867+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1006906745 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b180664a0 msgr2=0x7f8b18068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:18.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.867+0000 7f8b3bf9d640 1 --2- 172.21.15.125:0/1006906745 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b180664a0 0x7f8b18068960 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8b240af000 tx=0x7f8b240046d0 comp rx=0 tx=0).stop 2023-10-30T12:38:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.867+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1006906745 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b200047d0 msgr2=0x7f8b3413c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.867+0000 7f8b3bf9d640 1 --2- 172.21.15.125:0/1006906745 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b200047d0 0x7f8b3413c870 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f8b300125a0 tx=0x7f8b30012a60 comp rx=0 tx=0).stop 2023-10-30T12:38:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.867+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1006906745 shutdown_connections 2023-10-30T12:38:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.867+0000 7f8b3bf9d640 1 --2- 172.21.15.125:0/1006906745 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b180664a0 0x7f8b18068960 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.867+0000 7f8b3bf9d640 1 --2- 172.21.15.125:0/1006906745 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b3413dbd0 0x7f8b341510d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.867+0000 7f8b3bf9d640 1 --2- 172.21.15.125:0/1006906745 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b34137f20 0x7f8b3413d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.867+0000 7f8b3bf9d640 1 --2- 172.21.15.125:0/1006906745 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b200047d0 0x7f8b3413c870 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.867+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1006906745 >> 172.21.15.125:0/1006906745 conn(0x7f8b340c7910 msgr2=0x7f8b340c5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.867+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1006906745 shutdown_connections 2023-10-30T12:38:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:18.867+0000 7f8b3bf9d640 1 -- 172.21.15.125:0/1006906745 wait complete. 2023-10-30T12:38:18.885 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:38:20.886 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:21.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.039+0000 7f0c92f9d640 1 Processor -- start 2023-10-30T12:38:21.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.039+0000 7f0c92f9d640 1 -- start start 2023-10-30T12:38:21.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.039+0000 7f0c92f9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c8c070740 0x7f0c8c0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:21.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.039+0000 7f0c92f9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0c8c15e700 0x7f0c8c06a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:21.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.039+0000 7f0c92f9d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c8c0c5810 con 0x7f0c8c1580e0 2023-10-30T12:38:21.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.039+0000 7f0c92f9d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c8c070b60 con 0x7f0c8c15e700 2023-10-30T12:38:21.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.039+0000 7f0c92f9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c15eb90 con 0x7f0c8c070740 2023-10-30T12:38:21.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.043+0000 7f0c83fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c8c070740 0x7f0c8c0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:21.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.043+0000 7f0c9153a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0c8c15e700 0x7f0c8c06a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:21.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.043+0000 7f0c83fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c8c070740 0x7f0c8c0c6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:35418/0 (socket says 172.21.15.125:35418) 2023-10-30T12:38:21.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.043+0000 7f0c83fff640 1 -- 172.21.15.125:0/1923285137 learned_addr learned my addr 172.21.15.125:0/1923285137 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:21.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.043+0000 7f0c83fff640 1 -- 172.21.15.125:0/1923285137 >> v2:172.21.15.136:3300/0 conn(0x7f0c8c15e700 msgr2=0x7f0c8c06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:21.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.043+0000 7f0c83fff640 1 --2- 172.21.15.125:0/1923285137 >> v2:172.21.15.136:3300/0 conn(0x7f0c8c15e700 0x7f0c8c06a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:21.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.043+0000 7f0c83fff640 1 -- 172.21.15.125:0/1923285137 >> v1:172.21.15.136:6789/0 conn(0x7f0c8c1580e0 legacy=0x7f0c8c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:21.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.043+0000 7f0c83fff640 1 -- 172.21.15.125:0/1923285137 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8c0c6bb0 con 0x7f0c8c070740 2023-10-30T12:38:21.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.043+0000 7f0c83fff640 1 --2- 172.21.15.125:0/1923285137 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c8c070740 0x7f0c8c0c6540 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c009870 tx=0x7f0c7c01c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e0ee3b0ced2cb3e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:21.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c837fe640 1 -- 172.21.15.125:0/1923285137 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c0aa670 con 0x7f0c8c070740 2023-10-30T12:38:21.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c837fe640 1 -- 172.21.15.125:0/1923285137 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7c0aa7d0 con 0x7f0c8c070740 2023-10-30T12:38:21.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c837fe640 1 -- 172.21.15.125:0/1923285137 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c0aab60 con 0x7f0c8c070740 2023-10-30T12:38:21.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c92f9d640 1 -- 172.21.15.125:0/1923285137 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c8c070740 msgr2=0x7f0c8c0c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:21.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c92f9d640 1 --2- 172.21.15.125:0/1923285137 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c8c070740 0x7f0c8c0c6540 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c009870 tx=0x7f0c7c01c510 comp rx=0 tx=0).stop 2023-10-30T12:38:21.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c92f9d640 1 -- 172.21.15.125:0/1923285137 shutdown_connections 2023-10-30T12:38:21.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c92f9d640 1 --2- 172.21.15.125:0/1923285137 >> v2:172.21.15.136:3300/0 conn(0x7f0c8c15e700 0x7f0c8c06a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:21.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c92f9d640 1 --2- 172.21.15.125:0/1923285137 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c8c070740 0x7f0c8c0c6540 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:21.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c92f9d640 1 -- 172.21.15.125:0/1923285137 >> 172.21.15.125:0/1923285137 conn(0x7f0c8c0c7910 msgr2=0x7f0c8c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:21.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c92f9d640 1 -- 172.21.15.125:0/1923285137 shutdown_connections 2023-10-30T12:38:21.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c92f9d640 1 -- 172.21.15.125:0/1923285137 wait complete. 2023-10-30T12:38:21.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c92f9d640 1 Processor -- start 2023-10-30T12:38:21.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c92f9d640 1 -- start start 2023-10-30T12:38:21.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c92f9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c8c070740 0x7f0c8c1f6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:21.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c92f9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c8c1580e0 0x7f0c8c0c4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:21.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c92f9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c8c15e700 0x7f0c8c149220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:21.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.047+0000 7f0c92f9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c0c3850 con 0x7f0c8c15e700 2023-10-30T12:38:21.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c92f9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c0c4350 con 0x7f0c8c1580e0 2023-10-30T12:38:21.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c92f9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8c1fa1f0 con 0x7f0c8c070740 2023-10-30T12:38:21.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c9153a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c8c15e700 0x7f0c8c149220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:21.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c9153a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c8c15e700 0x7f0c8c149220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60168/0 (socket says 172.21.15.125:60168) 2023-10-30T12:38:21.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c9153a640 1 -- 172.21.15.125:0/2896928943 learned_addr learned my addr 172.21.15.125:0/2896928943 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:21.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c83fff640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c8c1580e0 0x7f0c8c0c4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:21.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c90d39640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c8c070740 0x7f0c8c1f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:21.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c9153a640 1 -- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c8c070740 msgr2=0x7f0c8c1f6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:21.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c9153a640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c8c070740 0x7f0c8c1f6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:21.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c9153a640 1 -- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c8c1580e0 msgr2=0x7f0c8c0c4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:21.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c9153a640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c8c1580e0 0x7f0c8c0c4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:21.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c9153a640 1 -- 172.21.15.125:0/2896928943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7c0094d0 con 0x7f0c8c15e700 2023-10-30T12:38:21.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c83fff640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c8c1580e0 0x7f0c8c0c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:21.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c90d39640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c8c070740 0x7f0c8c1f6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:21.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c9153a640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c8c15e700 0x7f0c8c149220 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009d30 tx=0x7f0c8400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:21.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c817fa640 1 -- 172.21.15.125:0/2896928943 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c84009ba0 con 0x7f0c8c15e700 2023-10-30T12:38:21.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c817fa640 1 -- 172.21.15.125:0/2896928943 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8400f040 con 0x7f0c8c15e700 2023-10-30T12:38:21.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c817fa640 1 -- 172.21.15.125:0/2896928943 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c840146c0 con 0x7f0c8c15e700 2023-10-30T12:38:21.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c92f9d640 1 -- 172.21.15.125:0/2896928943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8c1fa720 con 0x7f0c8c15e700 2023-10-30T12:38:21.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c92f9d640 1 -- 172.21.15.125:0/2896928943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8c0c48a0 con 0x7f0c8c15e700 2023-10-30T12:38:21.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c92f9d640 1 -- 172.21.15.125:0/2896928943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c8c0c4a30 con 0x7f0c8c15e700 2023-10-30T12:38:21.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.051+0000 7f0c817fa640 1 -- 172.21.15.125:0/2896928943 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0c84014820 con 0x7f0c8c15e700 2023-10-30T12:38:21.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.055+0000 7f0c817fa640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c780664d0 0x7f0c78068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:21.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.055+0000 7f0c817fa640 1 -- 172.21.15.125:0/2896928943 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f0c84084d00 con 0x7f0c8c15e700 2023-10-30T12:38:21.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.055+0000 7f0c90d39640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c780664d0 0x7f0c78068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:21.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.055+0000 7f0c90d39640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c780664d0 0x7f0c78068990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0c88000f70 tx=0x7f0c88009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:21.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.059+0000 7f0c817fa640 1 -- 172.21.15.125:0/2896928943 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0c84052070 con 0x7f0c8c15e700 2023-10-30T12:38:21.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.239+0000 7f0c92f9d640 1 -- 172.21.15.125:0/2896928943 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c8c04e170 con 0x7f0c8c15e700 2023-10-30T12:38:21.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.239+0000 7f0c817fa640 1 -- 172.21.15.125:0/2896928943 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f0c84055690 con 0x7f0c8c15e700 2023-10-30T12:38:21.242 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:21.242 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:01.249416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:21.242 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 56 2023-10-30T12:38:21.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.243+0000 7f0c92f9d640 1 -- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c780664d0 msgr2=0x7f0c78068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:21.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.243+0000 7f0c92f9d640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c780664d0 0x7f0c78068990 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0c88000f70 tx=0x7f0c88009040 comp rx=0 tx=0).stop 2023-10-30T12:38:21.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.243+0000 7f0c92f9d640 1 -- 172.21.15.125:0/2896928943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c8c15e700 msgr2=0x7f0c8c149220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:21.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.243+0000 7f0c92f9d640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c8c15e700 0x7f0c8c149220 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f0c84009d30 tx=0x7f0c8400c490 comp rx=0 tx=0).stop 2023-10-30T12:38:21.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.243+0000 7f0c92f9d640 1 -- 172.21.15.125:0/2896928943 shutdown_connections 2023-10-30T12:38:21.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.243+0000 7f0c92f9d640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c780664d0 0x7f0c78068990 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:21.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.243+0000 7f0c92f9d640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c8c15e700 0x7f0c8c149220 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:21.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.243+0000 7f0c92f9d640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c8c1580e0 0x7f0c8c0c4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:21.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.243+0000 7f0c92f9d640 1 --2- 172.21.15.125:0/2896928943 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c8c070740 0x7f0c8c1f6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:21.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.243+0000 7f0c92f9d640 1 -- 172.21.15.125:0/2896928943 >> 172.21.15.125:0/2896928943 conn(0x7f0c8c0c7910 msgr2=0x7f0c8c157220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:21.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.243+0000 7f0c92f9d640 1 -- 172.21.15.125:0/2896928943 shutdown_connections 2023-10-30T12:38:21.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:21.243+0000 7f0c92f9d640 1 -- 172.21.15.125:0/2896928943 wait complete. 2023-10-30T12:38:21.266 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:38:23.267 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:23.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb4657c3640 1 Processor -- start 2023-10-30T12:38:23.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb4657c3640 1 -- start start 2023-10-30T12:38:23.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb4657c3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb4601538f0 0x7fb460159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:23.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb4657c3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4600cefa0 0x7fb4600cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:23.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb4657c3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4600ca6a0 con 0x7fb46015cd50 2023-10-30T12:38:23.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb4657c3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb46015d300 con 0x7fb4601538f0 2023-10-30T12:38:23.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb4657c3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46015d5e0 con 0x7fb4600cefa0 2023-10-30T12:38:23.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb45ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb4601538f0 0x7fb460159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:23.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb45ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb4601538f0 0x7fb460159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60184/0 (socket says 172.21.15.125:60184) 2023-10-30T12:38:23.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb45ffff640 1 -- 172.21.15.125:0/3613856044 learned_addr learned my addr 172.21.15.125:0/3613856044 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:23.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb45f7fe640 1 --1- 172.21.15.125:0/3613856044 >> v1:172.21.15.125:6789/0 conn(0x7fb46015cd50 0x7fb4600c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38396/0 (socket says 172.21.15.125:38396) 2023-10-30T12:38:23.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb464fc2640 1 --2- 172.21.15.125:0/3613856044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4600cefa0 0x7fb4600cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:23.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb45effd640 1 -- 172.21.15.125:0/3613856044 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748907723 0 0) 0x7fb4600ca6a0 con 0x7fb46015cd50 2023-10-30T12:38:23.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb45effd640 1 -- 172.21.15.125:0/3613856044 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb448003530 con 0x7fb46015cd50 2023-10-30T12:38:23.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb45ffff640 1 -- 172.21.15.125:0/3613856044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4600cefa0 msgr2=0x7fb4600cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:23.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb45ffff640 1 --2- 172.21.15.125:0/3613856044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4600cefa0 0x7fb4600cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:23.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb45ffff640 1 -- 172.21.15.125:0/3613856044 >> v1:172.21.15.125:6789/0 conn(0x7fb46015cd50 legacy=0x7fb4600c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:23.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.415+0000 7fb45ffff640 1 -- 172.21.15.125:0/3613856044 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46015d8c0 con 0x7fb4601538f0 2023-10-30T12:38:23.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.419+0000 7fb45ffff640 1 --2- 172.21.15.125:0/3613856044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4601538f0 0x7fb460159fb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb450009870 tx=0x7fb45001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4eda9d3ca9e0e98 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:23.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.419+0000 7fb45effd640 1 -- 172.21.15.125:0/3613856044 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4500a8a30 con 0x7fb4601538f0 2023-10-30T12:38:23.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.419+0000 7fb45effd640 1 -- 172.21.15.125:0/3613856044 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4500a8b90 con 0x7fb4601538f0 2023-10-30T12:38:23.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.419+0000 7fb45effd640 1 -- 172.21.15.125:0/3613856044 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4500b0440 con 0x7fb4601538f0 2023-10-30T12:38:23.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.419+0000 7fb4657c3640 1 -- 172.21.15.125:0/3613856044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4601538f0 msgr2=0x7fb460159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:23.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.419+0000 7fb4657c3640 1 --2- 172.21.15.125:0/3613856044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4601538f0 0x7fb460159fb0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb450009870 tx=0x7fb45001ed30 comp rx=0 tx=0).stop 2023-10-30T12:38:23.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.419+0000 7fb4657c3640 1 -- 172.21.15.125:0/3613856044 shutdown_connections 2023-10-30T12:38:23.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.419+0000 7fb4657c3640 1 --2- 172.21.15.125:0/3613856044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4600cefa0 0x7fb4600cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:23.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.419+0000 7fb4657c3640 1 --2- 172.21.15.125:0/3613856044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4601538f0 0x7fb460159fb0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:23.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.419+0000 7fb4657c3640 1 -- 172.21.15.125:0/3613856044 >> 172.21.15.125:0/3613856044 conn(0x7fb46006a640 msgr2=0x7fb460065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:23.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.419+0000 7fb4657c3640 1 -- 172.21.15.125:0/3613856044 shutdown_connections 2023-10-30T12:38:23.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.419+0000 7fb4657c3640 1 -- 172.21.15.125:0/3613856044 wait complete. 2023-10-30T12:38:23.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb4657c3640 1 Processor -- start 2023-10-30T12:38:23.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb4657c3640 1 -- start start 2023-10-30T12:38:23.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb4657c3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4600cefa0 0x7fb4601f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:23.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb4657c3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4601538f0 0x7fb4601513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:23.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb4657c3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb46015cd50 0x7fb4601eee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:23.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb4657c3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb460151bc0 con 0x7fb4601538f0 2023-10-30T12:38:23.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb4657c3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb460153100 con 0x7fb46015cd50 2023-10-30T12:38:23.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb4657c3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4601efe50 con 0x7fb4600cefa0 2023-10-30T12:38:23.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb45f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4601538f0 0x7fb4601513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:23.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb45f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4601538f0 0x7fb4601513f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60200/0 (socket says 172.21.15.125:60200) 2023-10-30T12:38:23.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb45f7fe640 1 -- 172.21.15.125:0/914450740 learned_addr learned my addr 172.21.15.125:0/914450740 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:23.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb45ffff640 1 --2- 172.21.15.125:0/914450740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4600cefa0 0x7fb4601f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:23.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb464fc2640 1 --2- 172.21.15.125:0/914450740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb46015cd50 0x7fb4601eee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:23.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb45f7fe640 1 -- 172.21.15.125:0/914450740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4600cefa0 msgr2=0x7fb4601f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:23.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb45f7fe640 1 --2- 172.21.15.125:0/914450740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4600cefa0 0x7fb4601f2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:23.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb45f7fe640 1 -- 172.21.15.125:0/914450740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb46015cd50 msgr2=0x7fb4601eee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:23.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb45f7fe640 1 --2- 172.21.15.125:0/914450740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb46015cd50 0x7fb4601eee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:23.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb45f7fe640 1 -- 172.21.15.125:0/914450740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4500094d0 con 0x7fb4601538f0 2023-10-30T12:38:23.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb45f7fe640 1 --2- 172.21.15.125:0/914450740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4601538f0 0x7fb4601513f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb45400c450 tx=0x7fb45400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:23.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb45cff9640 1 -- 172.21.15.125:0/914450740 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454007480 con 0x7fb4601538f0 2023-10-30T12:38:23.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb45cff9640 1 -- 172.21.15.125:0/914450740 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb454007600 con 0x7fb4601538f0 2023-10-30T12:38:23.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb4657c3640 1 -- 172.21.15.125:0/914450740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4601ef390 con 0x7fb4601538f0 2023-10-30T12:38:23.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb45cff9640 1 -- 172.21.15.125:0/914450740 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454004a30 con 0x7fb4601538f0 2023-10-30T12:38:23.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb4657c3640 1 -- 172.21.15.125:0/914450740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46015ada0 con 0x7fb4601538f0 2023-10-30T12:38:23.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.423+0000 7fb4657c3640 1 -- 172.21.15.125:0/914450740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4601597a0 con 0x7fb4601538f0 2023-10-30T12:38:23.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.427+0000 7fb45cff9640 1 -- 172.21.15.125:0/914450740 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb454004b90 con 0x7fb4601538f0 2023-10-30T12:38:23.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.427+0000 7fb45cff9640 1 --2- 172.21.15.125:0/914450740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb44c0664d0 0x7fb44c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:23.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.427+0000 7fb45cff9640 1 -- 172.21.15.125:0/914450740 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fb4540830f0 con 0x7fb4601538f0 2023-10-30T12:38:23.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.427+0000 7fb45ffff640 1 --2- 172.21.15.125:0/914450740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb44c0664d0 0x7fb44c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:23.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.427+0000 7fb45ffff640 1 --2- 172.21.15.125:0/914450740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb44c0664d0 0x7fb44c068990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb4500b2000 tx=0x7fb450009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:23.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.431+0000 7fb45cff9640 1 -- 172.21.15.125:0/914450740 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb4540503e0 con 0x7fb4601538f0 2023-10-30T12:38:23.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.615+0000 7fb4657c3640 1 -- 172.21.15.125:0/914450740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb46004e170 con 0x7fb4601538f0 2023-10-30T12:38:23.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.615+0000 7fb45cff9640 1 -- 172.21.15.125:0/914450740 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6549 (secure 0 0 0) 0x7fb454053a00 con 0x7fb4601538f0 2023-10-30T12:38:23.615 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:23.616 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:01.249416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 56 2023-10-30T12:38:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.615+0000 7fb4657c3640 1 -- 172.21.15.125:0/914450740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb44c0664d0 msgr2=0x7fb44c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.615+0000 7fb4657c3640 1 --2- 172.21.15.125:0/914450740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb44c0664d0 0x7fb44c068990 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb4500b2000 tx=0x7fb450009040 comp rx=0 tx=0).stop 2023-10-30T12:38:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.615+0000 7fb4657c3640 1 -- 172.21.15.125:0/914450740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4601538f0 msgr2=0x7fb4601513f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.615+0000 7fb4657c3640 1 --2- 172.21.15.125:0/914450740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4601538f0 0x7fb4601513f0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fb45400c450 tx=0x7fb45400c910 comp rx=0 tx=0).stop 2023-10-30T12:38:23.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.615+0000 7fb4657c3640 1 -- 172.21.15.125:0/914450740 shutdown_connections 2023-10-30T12:38:23.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.615+0000 7fb4657c3640 1 --2- 172.21.15.125:0/914450740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb44c0664d0 0x7fb44c068990 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:23.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.615+0000 7fb4657c3640 1 --2- 172.21.15.125:0/914450740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb46015cd50 0x7fb4601eee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:23.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.615+0000 7fb4657c3640 1 --2- 172.21.15.125:0/914450740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4601538f0 0x7fb4601513f0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:23.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.615+0000 7fb4657c3640 1 --2- 172.21.15.125:0/914450740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4600cefa0 0x7fb4601f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:23.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.615+0000 7fb4657c3640 1 -- 172.21.15.125:0/914450740 >> 172.21.15.125:0/914450740 conn(0x7fb46006a640 msgr2=0x7fb4600c4740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:23.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.615+0000 7fb4657c3640 1 -- 172.21.15.125:0/914450740 shutdown_connections 2023-10-30T12:38:23.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:23.615+0000 7fb4657c3640 1 -- 172.21.15.125:0/914450740 wait complete. 2023-10-30T12:38:23.636 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:38:25.637 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:25.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.783+0000 7fc2a11c4640 1 Processor -- start 2023-10-30T12:38:25.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.783+0000 7fc2a11c4640 1 -- start start 2023-10-30T12:38:25.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.783+0000 7fc2a11c4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc29c0c05d0 0x7fc29c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:25.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.783+0000 7fc2a11c4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc29c0c6f60 0x7fc29c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:25.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.783+0000 7fc2a11c4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29c0c0a50 con 0x7fc29c0c6b80 2023-10-30T12:38:25.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.783+0000 7fc2a11c4640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc29c06f6e0 con 0x7fc29c0c05d0 2023-10-30T12:38:25.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.783+0000 7fc2a11c4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc29c06f9c0 con 0x7fc29c0c6f60 2023-10-30T12:38:25.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.783+0000 7fc29a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc29c0c05d0 0x7fc29c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:25.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.783+0000 7fc29a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc29c0c05d0 0x7fc29c13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60216/0 (socket says 172.21.15.125:60216) 2023-10-30T12:38:25.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.783+0000 7fc29a59c640 1 -- 172.21.15.125:0/2084215445 learned_addr learned my addr 172.21.15.125:0/2084215445 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:25.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.783+0000 7fc29b59e640 1 --2- 172.21.15.125:0/2084215445 >> v2:172.21.15.136:3300/0 conn(0x7fc29c0c6f60 0x7fc29c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:25.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.783+0000 7fc29a59c640 1 -- 172.21.15.125:0/2084215445 >> v2:172.21.15.136:3300/0 conn(0x7fc29c0c6f60 msgr2=0x7fc29c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:25.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc29a59c640 1 --2- 172.21.15.125:0/2084215445 >> v2:172.21.15.136:3300/0 conn(0x7fc29c0c6f60 0x7fc29c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:25.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc29a59c640 1 -- 172.21.15.125:0/2084215445 >> v1:172.21.15.136:6789/0 conn(0x7fc29c0c6b80 legacy=0x7fc29c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:25.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc29a59c640 1 -- 172.21.15.125:0/2084215445 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c06fca0 con 0x7fc29c0c05d0 2023-10-30T12:38:25.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc29a59c640 1 --2- 172.21.15.125:0/2084215445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0c05d0 0x7fc29c13ed90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0c89b0 tx=0x7fc29001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=647c39c29d78e90a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:25.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc299d9b640 1 -- 172.21.15.125:0/2084215445 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2900ab610 con 0x7fc29c0c05d0 2023-10-30T12:38:25.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc299d9b640 1 -- 172.21.15.125:0/2084215445 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2900ab770 con 0x7fc29c0c05d0 2023-10-30T12:38:25.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc299d9b640 1 -- 172.21.15.125:0/2084215445 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2900abb00 con 0x7fc29c0c05d0 2023-10-30T12:38:25.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc2a11c4640 1 -- 172.21.15.125:0/2084215445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0c05d0 msgr2=0x7fc29c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:25.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc2a11c4640 1 --2- 172.21.15.125:0/2084215445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0c05d0 0x7fc29c13ed90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0c89b0 tx=0x7fc29001c500 comp rx=0 tx=0).stop 2023-10-30T12:38:25.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc2a11c4640 1 -- 172.21.15.125:0/2084215445 shutdown_connections 2023-10-30T12:38:25.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc2a11c4640 1 --2- 172.21.15.125:0/2084215445 >> v2:172.21.15.136:3300/0 conn(0x7fc29c0c6f60 0x7fc29c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:25.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc2a11c4640 1 --2- 172.21.15.125:0/2084215445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0c05d0 0x7fc29c13ed90 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:25.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc2a11c4640 1 -- 172.21.15.125:0/2084215445 >> 172.21.15.125:0/2084215445 conn(0x7fc29c0c7910 msgr2=0x7fc29c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:25.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc2a11c4640 1 -- 172.21.15.125:0/2084215445 shutdown_connections 2023-10-30T12:38:25.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.787+0000 7fc2a11c4640 1 -- 172.21.15.125:0/2084215445 wait complete. 2023-10-30T12:38:25.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc2a11c4640 1 Processor -- start 2023-10-30T12:38:25.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc2a11c4640 1 -- start start 2023-10-30T12:38:25.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc2a11c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc29c0c05d0 0x7fc29c1317c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:25.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc2a11c4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0c6b80 0x7fc29c133b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:25.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc2a11c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc29c0c6f60 0x7fc29c12ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:25.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc2a11c4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c1589f0 con 0x7fc29c0c6b80 2023-10-30T12:38:25.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc2a11c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c12fcd0 con 0x7fc29c0c05d0 2023-10-30T12:38:25.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc2a11c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc29c12e6c0 con 0x7fc29c0c6f60 2023-10-30T12:38:25.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc29a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0c6b80 0x7fc29c133b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:25.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc29a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0c6b80 0x7fc29c133b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60226/0 (socket says 172.21.15.125:60226) 2023-10-30T12:38:25.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc29a59c640 1 -- 172.21.15.125:0/1093090066 learned_addr learned my addr 172.21.15.125:0/1093090066 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:25.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc29ad9d640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc29c0c05d0 0x7fc29c1317c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:25.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc29b59e640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc29c0c6f60 0x7fc29c12ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:25.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc29a59c640 1 -- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc29c0c6f60 msgr2=0x7fc29c12ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:25.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc29a59c640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc29c0c6f60 0x7fc29c12ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:25.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc29a59c640 1 -- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc29c0c05d0 msgr2=0x7fc29c1317c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:25.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc29a59c640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc29c0c05d0 0x7fc29c1317c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:25.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc29a59c640 1 -- 172.21.15.125:0/1093090066 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2900094d0 con 0x7fc29c0c6b80 2023-10-30T12:38:25.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc29ad9d640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc29c0c05d0 0x7fc29c1317c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:25.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.791+0000 7fc29a59c640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0c6b80 0x7fc29c133b60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc29001c9e0 tx=0x7fc29001ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:25.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.795+0000 7fc2837fe640 1 -- 172.21.15.125:0/1093090066 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2900b4800 con 0x7fc29c0c6b80 2023-10-30T12:38:25.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.795+0000 7fc2837fe640 1 -- 172.21.15.125:0/1093090066 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2900b4980 con 0x7fc29c0c6b80 2023-10-30T12:38:25.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.795+0000 7fc2837fe640 1 -- 172.21.15.125:0/1093090066 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2900b4d30 con 0x7fc29c0c6b80 2023-10-30T12:38:25.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.795+0000 7fc2a11c4640 1 -- 172.21.15.125:0/1093090066 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29c12e9a0 con 0x7fc29c0c6b80 2023-10-30T12:38:25.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.795+0000 7fc2a11c4640 1 -- 172.21.15.125:0/1093090066 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29c13b420 con 0x7fc29c0c6b80 2023-10-30T12:38:25.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.795+0000 7fc2a11c4640 1 -- 172.21.15.125:0/1093090066 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29c04e170 con 0x7fc29c0c6b80 2023-10-30T12:38:25.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.795+0000 7fc2837fe640 1 -- 172.21.15.125:0/1093090066 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc2900c7440 con 0x7fc29c0c6b80 2023-10-30T12:38:25.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.795+0000 7fc2837fe640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2880664a0 0x7fc288068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:25.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.795+0000 7fc2837fe640 1 -- 172.21.15.125:0/1093090066 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fc29011ee00 con 0x7fc29c0c6b80 2023-10-30T12:38:25.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.795+0000 7fc29ad9d640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2880664a0 0x7fc288068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:25.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.799+0000 7fc29ad9d640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2880664a0 0x7fc288068960 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc28c000f70 tx=0x7fc28c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:25.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.799+0000 7fc2837fe640 1 -- 172.21.15.125:0/1093090066 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc2900ec0d0 con 0x7fc29c0c6b80 2023-10-30T12:38:25.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.991+0000 7fc2a11c4640 1 -- 172.21.15.125:0/1093090066 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc29c0b0a00 con 0x7fc29c0c6b80 2023-10-30T12:38:25.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.991+0000 7fc2837fe640 1 -- 172.21.15.125:0/1093090066 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6549 (secure 0 0 0) 0x7fc2900ef6f0 con 0x7fc29c0c6b80 2023-10-30T12:38:25.992 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:25.992 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:01.249416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:25.993 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 56 2023-10-30T12:38:25.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.991+0000 7fc2a11c4640 1 -- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2880664a0 msgr2=0x7fc288068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:25.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.991+0000 7fc2a11c4640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2880664a0 0x7fc288068960 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc28c000f70 tx=0x7fc28c009040 comp rx=0 tx=0).stop 2023-10-30T12:38:25.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.991+0000 7fc2a11c4640 1 -- 172.21.15.125:0/1093090066 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0c6b80 msgr2=0x7fc29c133b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:25.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.991+0000 7fc2a11c4640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0c6b80 0x7fc29c133b60 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fc29001c9e0 tx=0x7fc29001ad40 comp rx=0 tx=0).stop 2023-10-30T12:38:25.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.991+0000 7fc2a11c4640 1 -- 172.21.15.125:0/1093090066 shutdown_connections 2023-10-30T12:38:25.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.991+0000 7fc2a11c4640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2880664a0 0x7fc288068960 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:25.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.991+0000 7fc2a11c4640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc29c0c6f60 0x7fc29c12ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:25.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.991+0000 7fc2a11c4640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc29c0c6b80 0x7fc29c133b60 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:25.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.991+0000 7fc2a11c4640 1 --2- 172.21.15.125:0/1093090066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc29c0c05d0 0x7fc29c1317c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:25.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.991+0000 7fc2a11c4640 1 -- 172.21.15.125:0/1093090066 >> 172.21.15.125:0/1093090066 conn(0x7fc29c0c7910 msgr2=0x7fc29c0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:25.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.991+0000 7fc2a11c4640 1 -- 172.21.15.125:0/1093090066 shutdown_connections 2023-10-30T12:38:25.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:25.995+0000 7fc2a11c4640 1 -- 172.21.15.125:0/1093090066 wait complete. 2023-10-30T12:38:26.013 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:38:28.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:28.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cfdf1d640 1 Processor -- start 2023-10-30T12:38:28.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cfdf1d640 1 -- start start 2023-10-30T12:38:28.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cfdf1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cf81538a0 0x7f0cf815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:28.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cfdf1d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0cf80cf160 0x7f0cf80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:28.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cfdf1d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf80cb800 con 0x7f0cf80ca0e0 2023-10-30T12:38:28.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cfdf1d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cf815dc30 con 0x7f0cf80cf160 2023-10-30T12:38:28.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cfdf1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf80cb330 con 0x7f0cf81538a0 2023-10-30T12:38:28.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cf77fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cf81538a0 0x7f0cf815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:28.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cf77fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cf81538a0 0x7f0cf815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:35460/0 (socket says 172.21.15.125:35460) 2023-10-30T12:38:28.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cf77fe640 1 -- 172.21.15.125:0/30377711 learned_addr learned my addr 172.21.15.125:0/30377711 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:28.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cf7fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0cf80cf160 0x7f0cf80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:28.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cf7fff640 1 -- 172.21.15.125:0/30377711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cf81538a0 msgr2=0x7f0cf815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:28.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cf7fff640 1 --2- 172.21.15.125:0/30377711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cf81538a0 0x7f0cf815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:28.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cf7fff640 1 -- 172.21.15.125:0/30377711 >> v1:172.21.15.136:6789/0 conn(0x7f0cf80ca0e0 legacy=0x7f0cf815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:28.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cf7fff640 1 -- 172.21.15.125:0/30377711 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf80cad70 con 0x7f0cf80cf160 2023-10-30T12:38:28.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.167+0000 7f0cf77fe640 1 --2- 172.21.15.125:0/30377711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cf81538a0 0x7f0cf815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:38:28.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.171+0000 7f0cf7fff640 1 --2- 172.21.15.125:0/30377711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cf80cf160 0x7f0cf80cf590 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0009d90 tx=0x7f0cf001f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a90048b27f4b15a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:28.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.171+0000 7f0cf67fc640 1 -- 172.21.15.125:0/30377711 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf00af600 con 0x7f0cf80cf160 2023-10-30T12:38:28.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.171+0000 7f0cf67fc640 1 -- 172.21.15.125:0/30377711 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf00ab050 con 0x7f0cf80cf160 2023-10-30T12:38:28.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.171+0000 7f0cf67fc640 1 -- 172.21.15.125:0/30377711 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf00af980 con 0x7f0cf80cf160 2023-10-30T12:38:28.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.171+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/30377711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cf80cf160 msgr2=0x7f0cf80cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:28.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.171+0000 7f0cfdf1d640 1 --2- 172.21.15.125:0/30377711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cf80cf160 0x7f0cf80cf590 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0009d90 tx=0x7f0cf001f130 comp rx=0 tx=0).stop 2023-10-30T12:38:28.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.171+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/30377711 shutdown_connections 2023-10-30T12:38:28.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.171+0000 7f0cfdf1d640 1 --2- 172.21.15.125:0/30377711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cf80cf160 0x7f0cf80cf590 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:28.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.171+0000 7f0cfdf1d640 1 --2- 172.21.15.125:0/30377711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cf81538a0 0x7f0cf815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:28.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.171+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/30377711 >> 172.21.15.125:0/30377711 conn(0x7f0cf806bfe0 msgr2=0x7f0cf806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:28.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.171+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/30377711 shutdown_connections 2023-10-30T12:38:28.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.171+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/30377711 wait complete. 2023-10-30T12:38:28.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cfdf1d640 1 Processor -- start 2023-10-30T12:38:28.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cfdf1d640 1 -- start start 2023-10-30T12:38:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cfdf1d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cf80ca0e0 0x7f0cf80ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cfdf1d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cf80cf160 0x7f0cf815c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cfdf1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cf81538a0 0x7f0cf8144770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cfdf1d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf81442e0 con 0x7f0cf80ca0e0 2023-10-30T12:38:28.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cfdf1d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf8143fe0 con 0x7f0cf80cf160 2023-10-30T12:38:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cfdf1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cf814c570 con 0x7f0cf81538a0 2023-10-30T12:38:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cf77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cf80ca0e0 0x7f0cf80ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cf77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cf80ca0e0 0x7f0cf80ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60238/0 (socket says 172.21.15.125:60238) 2023-10-30T12:38:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cf77fe640 1 -- 172.21.15.125:0/3380291034 learned_addr learned my addr 172.21.15.125:0/3380291034 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:28.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cf7fff640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cf81538a0 0x7f0cf8144770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cf6ffd640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cf80cf160 0x7f0cf815c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cf77fe640 1 -- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cf81538a0 msgr2=0x7f0cf8144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cf77fe640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cf81538a0 0x7f0cf8144770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cf77fe640 1 -- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cf80cf160 msgr2=0x7f0cf815c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cf77fe640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cf80cf160 0x7f0cf815c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:28.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cf77fe640 1 -- 172.21.15.125:0/3380291034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce80094d0 con 0x7f0cf80ca0e0 2023-10-30T12:38:28.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cf6ffd640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cf80cf160 0x7f0cf815c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:28.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cf77fe640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cf80ca0e0 0x7f0cf80ce260 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0ce800e960 tx=0x7f0ce800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:28.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cdffff640 1 -- 172.21.15.125:0/3380291034 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce800ca00 con 0x7f0cf80ca0e0 2023-10-30T12:38:28.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cdffff640 1 -- 172.21.15.125:0/3380291034 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce800cb60 con 0x7f0cf80ca0e0 2023-10-30T12:38:28.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cdffff640 1 -- 172.21.15.125:0/3380291034 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce800ced0 con 0x7f0cf80ca0e0 2023-10-30T12:38:28.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/3380291034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf0009a70 con 0x7f0cf80ca0e0 2023-10-30T12:38:28.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.175+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/3380291034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf814cb30 con 0x7f0cf80ca0e0 2023-10-30T12:38:28.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.179+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/3380291034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf80b2a50 con 0x7f0cf80ca0e0 2023-10-30T12:38:28.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.179+0000 7f0cdffff640 1 -- 172.21.15.125:0/3380291034 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0ce8013470 con 0x7f0cf80ca0e0 2023-10-30T12:38:28.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.179+0000 7f0cdffff640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ce40664a0 0x7f0ce4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:28.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.179+0000 7f0cdffff640 1 -- 172.21.15.125:0/3380291034 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f0ce8084200 con 0x7f0cf80ca0e0 2023-10-30T12:38:28.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.179+0000 7f0cf6ffd640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ce40664a0 0x7f0ce4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:28.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.179+0000 7f0cf6ffd640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ce40664a0 0x7f0ce4068960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80b1da0 tx=0x7f0cec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:28.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.183+0000 7f0cdffff640 1 -- 172.21.15.125:0/3380291034 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0ce8051580 con 0x7f0cf80ca0e0 2023-10-30T12:38:28.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.387+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/3380291034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cf804e170 con 0x7f0cf80ca0e0 2023-10-30T12:38:28.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.387+0000 7f0cdffff640 1 -- 172.21.15.125:0/3380291034 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f0ce8054ba0 con 0x7f0cf80ca0e0 2023-10-30T12:38:28.388 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:28.388 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:01.249416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:28.389 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 56 2023-10-30T12:38:28.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.391+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ce40664a0 msgr2=0x7f0ce4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:28.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.391+0000 7f0cfdf1d640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ce40664a0 0x7f0ce4068960 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0cf80b1da0 tx=0x7f0cec009040 comp rx=0 tx=0).stop 2023-10-30T12:38:28.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.391+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/3380291034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cf80ca0e0 msgr2=0x7f0cf80ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:28.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.391+0000 7f0cfdf1d640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cf80ca0e0 0x7f0cf80ce260 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0ce800e960 tx=0x7f0ce800ee20 comp rx=0 tx=0).stop 2023-10-30T12:38:28.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.391+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/3380291034 shutdown_connections 2023-10-30T12:38:28.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.391+0000 7f0cfdf1d640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ce40664a0 0x7f0ce4068960 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:28.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.391+0000 7f0cfdf1d640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cf81538a0 0x7f0cf8144770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:28.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.391+0000 7f0cfdf1d640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cf80cf160 0x7f0cf815c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:28.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.391+0000 7f0cfdf1d640 1 --2- 172.21.15.125:0/3380291034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cf80ca0e0 0x7f0cf80ce260 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:28.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.391+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/3380291034 >> 172.21.15.125:0/3380291034 conn(0x7f0cf806bfe0 msgr2=0x7f0cf8151320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:28.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.391+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/3380291034 shutdown_connections 2023-10-30T12:38:28.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:28.391+0000 7f0cfdf1d640 1 -- 172.21.15.125:0/3380291034 wait complete. 2023-10-30T12:38:28.411 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T12:38:28.412 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 56, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 56, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_4957": { "addr": "172.21.15.136:6835/2548530941", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2548530941, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2548530941, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 4957, "incarnation": 23, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:stopping", "state_seq": 34 }, "gid_4973": { "addr": "172.21.15.136:6839/1454326577", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 1454326577, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 1454326577, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 4973, "incarnation": 45, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 21 }, "gid_5130": { "addr": "172.21.15.125:6837/1624923396", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 1624923396, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 1624923396, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5130, "incarnation": 37, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 41 } }, "last_failure": 0, "last_failure_osd_epoch": 46, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:38:01.249416+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4973, "mds_1": 5130, "mds_2": 4957 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6836/2595438444", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2595438444, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2595438444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 50, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5433, "incarnation": 0, "join_fscid": -1, "name": "e", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:38:30.413 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:30.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.559+0000 7f79714d6640 1 Processor -- start 2023-10-30T12:38:30.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.559+0000 7f79714d6640 1 -- start start 2023-10-30T12:38:30.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.563+0000 7f79714d6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f796c153bf0 0x7f796c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:30.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.563+0000 7f79714d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f796c0c6fe0 0x7f796c0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:30.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.563+0000 7f79714d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f796c0700b0 con 0x7f796c0c79e0 2023-10-30T12:38:30.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.563+0000 7f79714d6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f796c0c6810 con 0x7f796c153bf0 2023-10-30T12:38:30.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.563+0000 7f79714d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c159900 con 0x7f796c0c6fe0 2023-10-30T12:38:30.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.563+0000 7f796affd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f796c153bf0 0x7f796c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:30.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.563+0000 7f796affd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f796c153bf0 0x7f796c071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60248/0 (socket says 172.21.15.125:60248) 2023-10-30T12:38:30.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.563+0000 7f796affd640 1 -- 172.21.15.125:0/319541137 learned_addr learned my addr 172.21.15.125:0/319541137 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:30.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.563+0000 7f79627fc640 1 --2- 172.21.15.125:0/319541137 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f796c0c6fe0 0x7f796c0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:30.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.563+0000 7f796affd640 1 -- 172.21.15.125:0/319541137 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f796c0c6fe0 msgr2=0x7f796c0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:30.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.563+0000 7f796affd640 1 --2- 172.21.15.125:0/319541137 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f796c0c6fe0 0x7f796c0c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:30.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.563+0000 7f796affd640 1 -- 172.21.15.125:0/319541137 >> v1:172.21.15.136:6789/0 conn(0x7f796c0c79e0 legacy=0x7f796c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:30.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.563+0000 7f796affd640 1 -- 172.21.15.125:0/319541137 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c0c6530 con 0x7f796c153bf0 2023-10-30T12:38:30.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.567+0000 7f796affd640 1 --2- 172.21.15.125:0/319541137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f796c153bf0 0x7f796c071260 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f795c009e50 tx=0x7f795c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b734f69f8f85b1b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:30.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.567+0000 7f796a7fc640 1 -- 172.21.15.125:0/319541137 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c0af400 con 0x7f796c153bf0 2023-10-30T12:38:30.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.567+0000 7f796a7fc640 1 -- 172.21.15.125:0/319541137 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f795c0ab050 con 0x7f796c153bf0 2023-10-30T12:38:30.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.567+0000 7f796a7fc640 1 -- 172.21.15.125:0/319541137 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f795c0af780 con 0x7f796c153bf0 2023-10-30T12:38:30.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.567+0000 7f79714d6640 1 -- 172.21.15.125:0/319541137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f796c153bf0 msgr2=0x7f796c071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:30.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.567+0000 7f79714d6640 1 --2- 172.21.15.125:0/319541137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f796c153bf0 0x7f796c071260 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f795c009e50 tx=0x7f795c01f1f0 comp rx=0 tx=0).stop 2023-10-30T12:38:30.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.567+0000 7f79714d6640 1 -- 172.21.15.125:0/319541137 shutdown_connections 2023-10-30T12:38:30.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.567+0000 7f79714d6640 1 --2- 172.21.15.125:0/319541137 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f796c0c6fe0 0x7f796c0c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:30.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.567+0000 7f79714d6640 1 --2- 172.21.15.125:0/319541137 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f796c153bf0 0x7f796c071260 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:30.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.567+0000 7f79714d6640 1 -- 172.21.15.125:0/319541137 >> 172.21.15.125:0/319541137 conn(0x7f796c06c6e0 msgr2=0x7f796c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:30.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.567+0000 7f79714d6640 1 -- 172.21.15.125:0/319541137 shutdown_connections 2023-10-30T12:38:30.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.567+0000 7f79714d6640 1 -- 172.21.15.125:0/319541137 wait complete. 2023-10-30T12:38:30.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.567+0000 7f79714d6640 1 Processor -- start 2023-10-30T12:38:30.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.567+0000 7f79714d6640 1 -- start start 2023-10-30T12:38:30.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f79714d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f796c0c6fe0 0x7f796c157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:30.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f79714d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f796c0c79e0 0x7f796c154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:30.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f79714d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f796c153bf0 0x7f796c1550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:30.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f79714d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c1f4450 con 0x7f796c153bf0 2023-10-30T12:38:30.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f79714d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c1f3ed0 con 0x7f796c0c6fe0 2023-10-30T12:38:30.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f79714d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c14c8b0 con 0x7f796c0c79e0 2023-10-30T12:38:30.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f796b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f796c153bf0 0x7f796c1550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:30.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f796b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f796c153bf0 0x7f796c1550e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60254/0 (socket says 172.21.15.125:60254) 2023-10-30T12:38:30.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f796b7fe640 1 -- 172.21.15.125:0/290334000 learned_addr learned my addr 172.21.15.125:0/290334000 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:30.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f796affd640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f796c0c6fe0 0x7f796c157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:30.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f79627fc640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f796c0c79e0 0x7f796c154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:30.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f796b7fe640 1 -- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f796c0c79e0 msgr2=0x7f796c154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:30.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f796b7fe640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f796c0c79e0 0x7f796c154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:30.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f796b7fe640 1 -- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f796c0c6fe0 msgr2=0x7f796c157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:30.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f796b7fe640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f796c0c6fe0 0x7f796c157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:30.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f796b7fe640 1 -- 172.21.15.125:0/290334000 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c009b30 con 0x7f796c153bf0 2023-10-30T12:38:30.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f796affd640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f796c0c6fe0 0x7f796c157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:30.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f796b7fe640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f796c153bf0 0x7f796c1550e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f796400c430 tx=0x7f796400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:30.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f7963fff640 1 -- 172.21.15.125:0/290334000 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964005220 con 0x7f796c153bf0 2023-10-30T12:38:30.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f7963fff640 1 -- 172.21.15.125:0/290334000 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7964005380 con 0x7f796c153bf0 2023-10-30T12:38:30.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f79714d6640 1 -- 172.21.15.125:0/290334000 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f796c1f4170 con 0x7f796c153bf0 2023-10-30T12:38:30.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f7963fff640 1 -- 172.21.15.125:0/290334000 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7964007270 con 0x7f796c153bf0 2023-10-30T12:38:30.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f79714d6640 1 -- 172.21.15.125:0/290334000 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f796c14dea0 con 0x7f796c153bf0 2023-10-30T12:38:30.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f79714d6640 1 -- 172.21.15.125:0/290334000 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f796c04e170 con 0x7f796c153bf0 2023-10-30T12:38:30.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.571+0000 7f7963fff640 1 -- 172.21.15.125:0/290334000 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f79640073d0 con 0x7f796c153bf0 2023-10-30T12:38:30.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.575+0000 7f7963fff640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f794c0664a0 0x7f794c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:30.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.575+0000 7f7963fff640 1 -- 172.21.15.125:0/290334000 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f7964083530 con 0x7f796c153bf0 2023-10-30T12:38:30.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.575+0000 7f796affd640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f794c0664a0 0x7f794c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:30.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.575+0000 7f796affd640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f794c0664a0 0x7f794c068960 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f795c0b1000 tx=0x7f795c0aa000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:30.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.579+0000 7f7963fff640 1 -- 172.21.15.125:0/290334000 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7964050800 con 0x7f796c153bf0 2023-10-30T12:38:30.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.771+0000 7f79714d6640 1 -- 172.21.15.125:0/290334000 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f796c155b80 con 0x7f796c153bf0 2023-10-30T12:38:30.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.771+0000 7f7963fff640 1 -- 172.21.15.125:0/290334000 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+6549 (secure 0 0 0) 0x7f7964053e20 con 0x7f796c153bf0 2023-10-30T12:38:30.771 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:30.772 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:01.249416+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1,2],"up":{"mds_0":4973,"mds_1":5130,"mds_2":4957},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4957":{"gid":4957,"name":"d","rank":2,"incarnation":23,"state":"up:stopping","state_seq":34,"addr":"172.21.15.136:6835/2548530941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2548530941},{"type":"v1","addr":"172.21.15.136:6835","nonce":2548530941}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:30.773 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 56 2023-10-30T12:38:30.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.771+0000 7f79714d6640 1 -- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f794c0664a0 msgr2=0x7f794c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:30.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.771+0000 7f79714d6640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f794c0664a0 0x7f794c068960 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f795c0b1000 tx=0x7f795c0aa000 comp rx=0 tx=0).stop 2023-10-30T12:38:30.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.771+0000 7f79714d6640 1 -- 172.21.15.125:0/290334000 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f796c153bf0 msgr2=0x7f796c1550e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:30.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.771+0000 7f79714d6640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f796c153bf0 0x7f796c1550e0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f796400c430 tx=0x7f796400c8f0 comp rx=0 tx=0).stop 2023-10-30T12:38:30.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.771+0000 7f79714d6640 1 -- 172.21.15.125:0/290334000 shutdown_connections 2023-10-30T12:38:30.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.771+0000 7f79714d6640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f794c0664a0 0x7f794c068960 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:30.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.771+0000 7f79714d6640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f796c153bf0 0x7f796c1550e0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:30.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.771+0000 7f79714d6640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f796c0c79e0 0x7f796c154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:30.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.771+0000 7f79714d6640 1 --2- 172.21.15.125:0/290334000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f796c0c6fe0 0x7f796c157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:30.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.771+0000 7f79714d6640 1 -- 172.21.15.125:0/290334000 >> 172.21.15.125:0/290334000 conn(0x7f796c06c6e0 msgr2=0x7f796c070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:30.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.771+0000 7f79714d6640 1 -- 172.21.15.125:0/290334000 shutdown_connections 2023-10-30T12:38:30.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:30.771+0000 7f79714d6640 1 -- 172.21.15.125:0/290334000 wait complete. 2023-10-30T12:38:30.792 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:38:31.330 INFO:tasks.ceph.mds.d.smithi136.stdout:starting mds.d at 2023-10-30T12:38:32.793 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:32.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.939+0000 7eff291ab640 1 Processor -- start 2023-10-30T12:38:32.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.939+0000 7eff291ab640 1 -- start start 2023-10-30T12:38:32.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.939+0000 7eff291ab640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7eff240c6f60 0x7eff241566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:32.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.939+0000 7eff291ab640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff240c0a50 con 0x7eff240c6b80 2023-10-30T12:38:32.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.939+0000 7eff291ab640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff2406f6e0 con 0x7eff240c05d0 2023-10-30T12:38:32.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.939+0000 7eff291ab640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff2406f9c0 con 0x7eff240c6f60 2023-10-30T12:38:32.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.939+0000 7eff22d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7eff240c6b80 0x7eff240bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44076/0 (socket says 172.21.15.125:44076) 2023-10-30T12:38:32.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.939+0000 7eff22d9d640 1 -- 172.21.15.125:0/1262063646 learned_addr learned my addr 172.21.15.125:0/1262063646 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:38:32.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.939+0000 7eff2359e640 1 --2- 172.21.15.125:0/1262063646 >> v2:172.21.15.136:3300/0 conn(0x7eff240c6f60 0x7eff241566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:32.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.939+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545228706 0 0) 0x7eff240c0a50 con 0x7eff240c6b80 2023-10-30T12:38:32.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff10003530 con 0x7eff240c6b80 2023-10-30T12:38:32.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310979795 0 0) 0x7eff2406f6e0 con 0x7eff240c05d0 2023-10-30T12:38:32.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff240c0a50 con 0x7eff240c05d0 2023-10-30T12:38:32.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2846058616 0 0) 0x7eff10003530 con 0x7eff240c6b80 2023-10-30T12:38:32.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 >> v2:172.21.15.136:3300/0 conn(0x7eff240c6f60 msgr2=0x7eff241566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:32.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 --2- 172.21.15.125:0/1262063646 >> v2:172.21.15.136:3300/0 conn(0x7eff240c6f60 0x7eff241566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:32.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 >> v1:172.21.15.136:6789/0 conn(0x7eff240c05d0 legacy=0x7eff2413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:32.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2406fca0 con 0x7eff240c6b80 2023-10-30T12:38:32.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7eff14001a00 con 0x7eff240c6b80 2023-10-30T12:38:32.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 >> v1:172.21.15.125:6789/0 conn(0x7eff240c6b80 legacy=0x7eff240bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:32.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 --2- 172.21.15.125:0/1262063646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff10003b10 0x7eff10003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:32.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 --2- 172.21.15.125:0/1262063646 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff100047d0 0x7eff10004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:32.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 --2- 172.21.15.125:0/1262063646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff10005190 0x7eff1000dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:32.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2406f9c0 con 0x7eff10003b10 2023-10-30T12:38:32.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1000e220 con 0x7eff10005190 2023-10-30T12:38:32.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff1000e3e0 con 0x7eff100047d0 2023-10-30T12:38:32.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff22d9d640 1 --2- 172.21.15.125:0/1262063646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff10003b10 0x7eff10003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:32.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2359e640 1 --2- 172.21.15.125:0/1262063646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff10005190 0x7eff1000dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:32.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff1a59c640 1 --2- 172.21.15.125:0/1262063646 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff100047d0 0x7eff10004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:32.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff22d9d640 1 -- 172.21.15.125:0/1262063646 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff100047d0 msgr2=0x7eff10004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:32.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff22d9d640 1 --2- 172.21.15.125:0/1262063646 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff100047d0 0x7eff10004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:32.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff22d9d640 1 -- 172.21.15.125:0/1262063646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff10005190 msgr2=0x7eff1000dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:32.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff22d9d640 1 --2- 172.21.15.125:0/1262063646 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff10005190 0x7eff1000dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:32.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff22d9d640 1 -- 172.21.15.125:0/1262063646 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff10003530 con 0x7eff10003b10 2023-10-30T12:38:32.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.943+0000 7eff2359e640 1 -- 172.21.15.125:0/1262063646 reap_dead start 2023-10-30T12:38:32.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.947+0000 7eff22d9d640 1 --2- 172.21.15.125:0/1262063646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff10003b10 0x7eff10003f10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7eff1400a7a0 tx=0x7eff1401d2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ed64179ad70ccf6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:32.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.947+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff140ad630 con 0x7eff10003b10 2023-10-30T12:38:32.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.947+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff140a8070 con 0x7eff10003b10 2023-10-30T12:38:32.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.947+0000 7eff2259c640 1 -- 172.21.15.125:0/1262063646 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff140ada30 con 0x7eff10003b10 2023-10-30T12:38:32.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.947+0000 7eff291ab640 1 -- 172.21.15.125:0/1262063646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff10003b10 msgr2=0x7eff10003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:32.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.947+0000 7eff291ab640 1 --2- 172.21.15.125:0/1262063646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff10003b10 0x7eff10003f10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7eff1400a7a0 tx=0x7eff1401d2c0 comp rx=0 tx=0).stop 2023-10-30T12:38:32.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.947+0000 7eff291ab640 1 -- 172.21.15.125:0/1262063646 shutdown_connections 2023-10-30T12:38:32.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.947+0000 7eff291ab640 1 --2- 172.21.15.125:0/1262063646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff10003b10 0x7eff10003f10 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:32.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.947+0000 7eff291ab640 1 -- 172.21.15.125:0/1262063646 >> 172.21.15.125:0/1262063646 conn(0x7eff240c7910 msgr2=0x7eff240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:32.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.947+0000 7eff291ab640 1 -- 172.21.15.125:0/1262063646 shutdown_connections 2023-10-30T12:38:32.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.947+0000 7eff291ab640 1 -- 172.21.15.125:0/1262063646 wait complete. 2023-10-30T12:38:32.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.947+0000 7eff291ab640 1 Processor -- start 2023-10-30T12:38:32.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.947+0000 7eff291ab640 1 -- start start 2023-10-30T12:38:32.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff291ab640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff10003b10 0x7eff24159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:32.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff291ab640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff241575a0 0x7eff241f0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:32.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff291ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff241f1240 0x7eff2413a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:32.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff291ab640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff240ca420 con 0x7eff241f1240 2023-10-30T12:38:32.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff291ab640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff240ca700 con 0x7eff10003b10 2023-10-30T12:38:32.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff291ab640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff240cab90 con 0x7eff241575a0 2023-10-30T12:38:32.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff2359e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff241f1240 0x7eff2413a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:32.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff1a59c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff241575a0 0x7eff241f0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:32.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff22d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff10003b10 0x7eff24159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:32.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff2359e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff241f1240 0x7eff2413a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43276/0 (socket says 172.21.15.125:43276) 2023-10-30T12:38:32.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff2359e640 1 -- 172.21.15.125:0/3435227358 learned_addr learned my addr 172.21.15.125:0/3435227358 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:32.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff22d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff10003b10 0x7eff24159160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:40806/0 (socket says 172.21.15.125:40806) 2023-10-30T12:38:32.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff2359e640 1 -- 172.21.15.125:0/3435227358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff241575a0 msgr2=0x7eff241f0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:32.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff2359e640 1 --2- 172.21.15.125:0/3435227358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff241575a0 0x7eff241f0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:32.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff2359e640 1 -- 172.21.15.125:0/3435227358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff10003b10 msgr2=0x7eff24159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:32.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff2359e640 1 --2- 172.21.15.125:0/3435227358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff10003b10 0x7eff24159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:32.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff2359e640 1 -- 172.21.15.125:0/3435227358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff24154a70 con 0x7eff241f1240 2023-10-30T12:38:32.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff2359e640 1 --2- 172.21.15.125:0/3435227358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff241f1240 0x7eff2413a210 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7eff24159b90 tx=0x7eff1c011c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:32.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff1bfff640 1 -- 172.21.15.125:0/3435227358 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c011e80 con 0x7eff241f1240 2023-10-30T12:38:32.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff1bfff640 1 -- 172.21.15.125:0/3435227358 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff1c004240 con 0x7eff241f1240 2023-10-30T12:38:32.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff1bfff640 1 -- 172.21.15.125:0/3435227358 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c00f670 con 0x7eff241f1240 2023-10-30T12:38:32.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff291ab640 1 -- 172.21.15.125:0/3435227358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff2413af50 con 0x7eff241f1240 2023-10-30T12:38:32.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff291ab640 1 -- 172.21.15.125:0/3435227358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff2413b510 con 0x7eff241f1240 2023-10-30T12:38:32.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff291ab640 1 -- 172.21.15.125:0/3435227358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff2413b6d0 con 0x7eff241f1240 2023-10-30T12:38:32.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.951+0000 7eff1bfff640 1 -- 172.21.15.125:0/3435227358 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7eff1c00f7d0 con 0x7eff241f1240 2023-10-30T12:38:32.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.955+0000 7eff1bfff640 1 --2- 172.21.15.125:0/3435227358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff040664a0 0x7eff04068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:32.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.955+0000 7eff1bfff640 1 -- 172.21.15.125:0/3435227358 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7eff1c084190 con 0x7eff241f1240 2023-10-30T12:38:32.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.955+0000 7eff22d9d640 1 --2- 172.21.15.125:0/3435227358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff040664a0 0x7eff04068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:32.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.955+0000 7eff22d9d640 1 --2- 172.21.15.125:0/3435227358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff040664a0 0x7eff04068960 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7eff140ad000 tx=0x7eff140af040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:32.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:32.959+0000 7eff1bfff640 1 -- 172.21.15.125:0/3435227358 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7eff1c051470 con 0x7eff241f1240 2023-10-30T12:38:33.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.143+0000 7eff291ab640 1 -- 172.21.15.125:0/3435227358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff240ca870 con 0x7eff241f1240 2023-10-30T12:38:33.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.143+0000 7eff1bfff640 1 -- 172.21.15.125:0/3435227358 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6532 (secure 0 0 0) 0x7eff1c054a90 con 0x7eff241f1240 2023-10-30T12:38:33.144 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:33.144 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5174,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2993299045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2993299045},{"type":"v1","addr":"172.21.15.136:6835","nonce":2993299045}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:31.250027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:33.145 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 58 2023-10-30T12:38:33.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.143+0000 7eff291ab640 1 -- 172.21.15.125:0/3435227358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff040664a0 msgr2=0x7eff04068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:33.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.143+0000 7eff291ab640 1 --2- 172.21.15.125:0/3435227358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff040664a0 0x7eff04068960 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7eff140ad000 tx=0x7eff140af040 comp rx=0 tx=0).stop 2023-10-30T12:38:33.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.143+0000 7eff291ab640 1 -- 172.21.15.125:0/3435227358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff241f1240 msgr2=0x7eff2413a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:33.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.143+0000 7eff291ab640 1 --2- 172.21.15.125:0/3435227358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff241f1240 0x7eff2413a210 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7eff24159b90 tx=0x7eff1c011c30 comp rx=0 tx=0).stop 2023-10-30T12:38:33.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.143+0000 7eff291ab640 1 -- 172.21.15.125:0/3435227358 shutdown_connections 2023-10-30T12:38:33.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.143+0000 7eff291ab640 1 --2- 172.21.15.125:0/3435227358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff040664a0 0x7eff04068960 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.143+0000 7eff291ab640 1 --2- 172.21.15.125:0/3435227358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff241f1240 0x7eff2413a210 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.143+0000 7eff291ab640 1 --2- 172.21.15.125:0/3435227358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff241575a0 0x7eff241f0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.143+0000 7eff291ab640 1 --2- 172.21.15.125:0/3435227358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff10003b10 0x7eff24159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.143+0000 7eff291ab640 1 -- 172.21.15.125:0/3435227358 >> 172.21.15.125:0/3435227358 conn(0x7eff240c7910 msgr2=0x7eff2406b570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:33.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.143+0000 7eff291ab640 1 -- 172.21.15.125:0/3435227358 shutdown_connections 2023-10-30T12:38:33.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.143+0000 7eff291ab640 1 -- 172.21.15.125:0/3435227358 wait complete. 2023-10-30T12:38:33.164 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 2 alive and active, now stable! 2023-10-30T12:38:33.164 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=2) 2023-10-30T12:38:33.165 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T12:38:33.165 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:38:33.177 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T12:38:33.178 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:33.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.347+0000 7f4be9c79640 1 Processor -- start 2023-10-30T12:38:33.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be9c79640 1 -- start start 2023-10-30T12:38:33.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be9c79640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be4156f80 0x7f4be4159880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:33.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be9c79640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4be4070650 0x7f4be4071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:33.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be9c79640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4be4070c90 0x7f4be40720f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:33.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be9c79640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be4157360 con 0x7f4be4070650 2023-10-30T12:38:33.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be9c79640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be40c2f20 con 0x7f4be4070c90 2023-10-30T12:38:33.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be9c79640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be40c85c0 con 0x7f4be4156f80 2023-10-30T12:38:33.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be2ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4be4070650 0x7f4be4071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:33.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be2ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4be4070650 0x7f4be4071bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43292/0 (socket says 172.21.15.125:43292) 2023-10-30T12:38:33.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be2ffd640 1 -- 172.21.15.125:0/2904799432 learned_addr learned my addr 172.21.15.125:0/2904799432 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:33.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be37fe640 1 --2- 172.21.15.125:0/2904799432 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be4156f80 0x7f4be4159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:33.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be3fff640 1 --2- 172.21.15.125:0/2904799432 >> v2:172.21.15.136:3300/0 conn(0x7f4be4070c90 0x7f4be40720f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:33.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be2ffd640 1 -- 172.21.15.125:0/2904799432 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be4156f80 msgr2=0x7f4be4159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:33.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be2ffd640 1 --2- 172.21.15.125:0/2904799432 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be4156f80 0x7f4be4159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be2ffd640 1 -- 172.21.15.125:0/2904799432 >> v2:172.21.15.136:3300/0 conn(0x7f4be4070c90 msgr2=0x7f4be40720f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:33.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be2ffd640 1 --2- 172.21.15.125:0/2904799432 >> v2:172.21.15.136:3300/0 conn(0x7f4be4070c90 0x7f4be40720f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.351+0000 7f4be2ffd640 1 -- 172.21.15.125:0/2904799432 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be4071230 con 0x7f4be4070650 2023-10-30T12:38:33.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.355+0000 7f4be2ffd640 1 --2- 172.21.15.125:0/2904799432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be4070650 0x7f4be4071bb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4be40cce40 tx=0x7f4bd801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2def7984d268e1dd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:33.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.355+0000 7f4be27fc640 1 -- 172.21.15.125:0/2904799432 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd80ab680 con 0x7f4be4070650 2023-10-30T12:38:33.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.355+0000 7f4be27fc640 1 -- 172.21.15.125:0/2904799432 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd80ab7e0 con 0x7f4be4070650 2023-10-30T12:38:33.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.355+0000 7f4be27fc640 1 -- 172.21.15.125:0/2904799432 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd80abb70 con 0x7f4be4070650 2023-10-30T12:38:33.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.355+0000 7f4be9c79640 1 -- 172.21.15.125:0/2904799432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be4070650 msgr2=0x7f4be4071bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:33.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.355+0000 7f4be9c79640 1 --2- 172.21.15.125:0/2904799432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be4070650 0x7f4be4071bb0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f4be40cce40 tx=0x7f4bd801c500 comp rx=0 tx=0).stop 2023-10-30T12:38:33.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.355+0000 7f4be9c79640 1 -- 172.21.15.125:0/2904799432 shutdown_connections 2023-10-30T12:38:33.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.355+0000 7f4be9c79640 1 --2- 172.21.15.125:0/2904799432 >> v2:172.21.15.136:3300/0 conn(0x7f4be4070c90 0x7f4be40720f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.355+0000 7f4be9c79640 1 --2- 172.21.15.125:0/2904799432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be4070650 0x7f4be4071bb0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.355+0000 7f4be9c79640 1 --2- 172.21.15.125:0/2904799432 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be4156f80 0x7f4be4159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.355+0000 7f4be9c79640 1 -- 172.21.15.125:0/2904799432 >> 172.21.15.125:0/2904799432 conn(0x7f4be40cbeb0 msgr2=0x7f4be40cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:33.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.355+0000 7f4be9c79640 1 -- 172.21.15.125:0/2904799432 shutdown_connections 2023-10-30T12:38:33.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.355+0000 7f4be9c79640 1 -- 172.21.15.125:0/2904799432 wait complete. 2023-10-30T12:38:33.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.355+0000 7f4be9c79640 1 Processor -- start 2023-10-30T12:38:33.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be9c79640 1 -- start start 2023-10-30T12:38:33.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be9c79640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be4070650 0x7f4be415a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:33.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be9c79640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be4070c90 0x7f4be414e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:33.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be9c79640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4be4156f80 0x7f4be4147c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:33.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be9c79640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be41f1fb0 con 0x7f4be4070650 2023-10-30T12:38:33.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be9c79640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be41f2f60 con 0x7f4be4156f80 2023-10-30T12:38:33.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be9c79640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be41f6270 con 0x7f4be4070c90 2023-10-30T12:38:33.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be4070650 0x7f4be415a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:33.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be4070650 0x7f4be415a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43300/0 (socket says 172.21.15.125:43300) 2023-10-30T12:38:33.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be37fe640 1 -- 172.21.15.125:0/1515883421 learned_addr learned my addr 172.21.15.125:0/1515883421 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:33.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be2ffd640 1 --2- 172.21.15.125:0/1515883421 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be4070c90 0x7f4be414e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:33.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be3fff640 1 --2- 172.21.15.125:0/1515883421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4be4156f80 0x7f4be4147c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:33.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be37fe640 1 -- 172.21.15.125:0/1515883421 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be4070c90 msgr2=0x7f4be414e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:33.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be37fe640 1 --2- 172.21.15.125:0/1515883421 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be4070c90 0x7f4be414e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be37fe640 1 -- 172.21.15.125:0/1515883421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4be4156f80 msgr2=0x7f4be4147c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:33.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be37fe640 1 --2- 172.21.15.125:0/1515883421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4be4156f80 0x7f4be4147c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be37fe640 1 -- 172.21.15.125:0/1515883421 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd80094d0 con 0x7f4be4070650 2023-10-30T12:38:33.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be37fe640 1 --2- 172.21.15.125:0/1515883421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be4070650 0x7f4be415a9e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400e8a0 tx=0x7f4bd400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:33.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4bcbfff640 1 -- 172.21.15.125:0/1515883421 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4009510 con 0x7f4be4070650 2023-10-30T12:38:33.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4bcbfff640 1 -- 172.21.15.125:0/1515883421 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd400c940 con 0x7f4be4070650 2023-10-30T12:38:33.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be9c79640 1 -- 172.21.15.125:0/1515883421 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be41f21a0 con 0x7f4be4070650 2023-10-30T12:38:33.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4bcbfff640 1 -- 172.21.15.125:0/1515883421 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd400ccb0 con 0x7f4be4070650 2023-10-30T12:38:33.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be9c79640 1 -- 172.21.15.125:0/1515883421 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be41f2760 con 0x7f4be4070650 2023-10-30T12:38:33.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.359+0000 7f4be9c79640 1 -- 172.21.15.125:0/1515883421 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be404e170 con 0x7f4be4070650 2023-10-30T12:38:33.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.363+0000 7f4bcbfff640 1 -- 172.21.15.125:0/1515883421 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4bd4013470 con 0x7f4be4070650 2023-10-30T12:38:33.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.363+0000 7f4bcbfff640 1 --2- 172.21.15.125:0/1515883421 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bd00664a0 0x7f4bd0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:33.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.363+0000 7f4bcbfff640 1 -- 172.21.15.125:0/1515883421 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f4bd4084920 con 0x7f4be4070650 2023-10-30T12:38:33.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.363+0000 7f4be2ffd640 1 --2- 172.21.15.125:0/1515883421 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bd00664a0 0x7f4bd0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:33.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.363+0000 7f4be2ffd640 1 --2- 172.21.15.125:0/1515883421 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bd00664a0 0x7f4bd0068960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4bd80af000 tx=0x7f4bd80a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:33.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.367+0000 7f4bcbfff640 1 -- 172.21.15.125:0/1515883421 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4bd4051be0 con 0x7f4be4070650 2023-10-30T12:38:33.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.547+0000 7f4be9c79640 1 -- 172.21.15.125:0/1515883421 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4be40c8220 con 0x7f4be4070650 2023-10-30T12:38:33.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.547+0000 7f4bcbfff640 1 -- 172.21.15.125:0/1515883421 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f4bd4055200 con 0x7f4be4070650 2023-10-30T12:38:33.547 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:33.548 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5174,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2993299045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2993299045},{"type":"v1","addr":"172.21.15.136:6835","nonce":2993299045}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:31.250027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 58 2023-10-30T12:38:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.547+0000 7f4be9c79640 1 -- 172.21.15.125:0/1515883421 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bd00664a0 msgr2=0x7f4bd0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.547+0000 7f4be9c79640 1 --2- 172.21.15.125:0/1515883421 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bd00664a0 0x7f4bd0068960 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4bd80af000 tx=0x7f4bd80a7000 comp rx=0 tx=0).stop 2023-10-30T12:38:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.547+0000 7f4be9c79640 1 -- 172.21.15.125:0/1515883421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be4070650 msgr2=0x7f4be415a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.547+0000 7f4be9c79640 1 --2- 172.21.15.125:0/1515883421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be4070650 0x7f4be415a9e0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400e8a0 tx=0x7f4bd400eca0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.547+0000 7f4be9c79640 1 -- 172.21.15.125:0/1515883421 shutdown_connections 2023-10-30T12:38:33.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.547+0000 7f4be9c79640 1 --2- 172.21.15.125:0/1515883421 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bd00664a0 0x7f4bd0068960 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.547+0000 7f4be9c79640 1 --2- 172.21.15.125:0/1515883421 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4be4156f80 0x7f4be4147c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.547+0000 7f4be9c79640 1 --2- 172.21.15.125:0/1515883421 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be4070c90 0x7f4be414e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.547+0000 7f4be9c79640 1 --2- 172.21.15.125:0/1515883421 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be4070650 0x7f4be415a9e0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:33.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.547+0000 7f4be9c79640 1 -- 172.21.15.125:0/1515883421 >> 172.21.15.125:0/1515883421 conn(0x7f4be40cbeb0 msgr2=0x7f4be40c1740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:33.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.547+0000 7f4be9c79640 1 -- 172.21.15.125:0/1515883421 shutdown_connections 2023-10-30T12:38:33.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:33.547+0000 7f4be9c79640 1 -- 172.21.15.125:0/1515883421 wait complete. 2023-10-30T12:38:33.568 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:38:35.569 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:35.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.715+0000 7fd6bc11d640 1 Processor -- start 2023-10-30T12:38:35.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.715+0000 7fd6bc11d640 1 -- start start 2023-10-30T12:38:35.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.719+0000 7fd6bc11d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd6b40c05d0 0x7fd6b413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:35.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.719+0000 7fd6bc11d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6b40c6f60 0x7fd6b413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:35.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.719+0000 7fd6bc11d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b40c0a50 con 0x7fd6b40c6b80 2023-10-30T12:38:35.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.719+0000 7fd6bc11d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6b406f6e0 con 0x7fd6b40c05d0 2023-10-30T12:38:35.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.719+0000 7fd6bc11d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b406f9c0 con 0x7fd6b40c6f60 2023-10-30T12:38:35.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.719+0000 7fd6b96b8640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd6b40c05d0 0x7fd6b413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:35.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.719+0000 7fd6b96b8640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd6b40c05d0 0x7fd6b413ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43302/0 (socket says 172.21.15.125:43302) 2023-10-30T12:38:35.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.719+0000 7fd6b96b8640 1 -- 172.21.15.125:0/3680831504 learned_addr learned my addr 172.21.15.125:0/3680831504 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:35.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.719+0000 7fd6ba6ba640 1 --2- 172.21.15.125:0/3680831504 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6b40c6f60 0x7fd6b413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:35.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.719+0000 7fd6b96b8640 1 -- 172.21.15.125:0/3680831504 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6b40c6f60 msgr2=0x7fd6b413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:35.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.719+0000 7fd6b96b8640 1 --2- 172.21.15.125:0/3680831504 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6b40c6f60 0x7fd6b413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:35.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.719+0000 7fd6b96b8640 1 -- 172.21.15.125:0/3680831504 >> v1:172.21.15.136:6789/0 conn(0x7fd6b40c6b80 legacy=0x7fd6b40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:35.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.719+0000 7fd6b96b8640 1 -- 172.21.15.125:0/3680831504 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b406fca0 con 0x7fd6b40c05d0 2023-10-30T12:38:35.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.723+0000 7fd6b96b8640 1 --2- 172.21.15.125:0/3680831504 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6b40c05d0 0x7fd6b413ed90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009870 tx=0x7fd6a401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32c84f79ae92dd02 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:35.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.723+0000 7fd6b8eb7640 1 -- 172.21.15.125:0/3680831504 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a40ab650 con 0x7fd6b40c05d0 2023-10-30T12:38:35.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.723+0000 7fd6b8eb7640 1 -- 172.21.15.125:0/3680831504 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a40ab7b0 con 0x7fd6b40c05d0 2023-10-30T12:38:35.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.723+0000 7fd6b8eb7640 1 -- 172.21.15.125:0/3680831504 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a40abb40 con 0x7fd6b40c05d0 2023-10-30T12:38:35.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.723+0000 7fd6bc11d640 1 -- 172.21.15.125:0/3680831504 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6b40c05d0 msgr2=0x7fd6b413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:35.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.723+0000 7fd6bc11d640 1 --2- 172.21.15.125:0/3680831504 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6b40c05d0 0x7fd6b413ed90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4009870 tx=0x7fd6a401c540 comp rx=0 tx=0).stop 2023-10-30T12:38:35.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.723+0000 7fd6bc11d640 1 -- 172.21.15.125:0/3680831504 shutdown_connections 2023-10-30T12:38:35.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.723+0000 7fd6bc11d640 1 --2- 172.21.15.125:0/3680831504 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6b40c6f60 0x7fd6b413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:35.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.723+0000 7fd6bc11d640 1 --2- 172.21.15.125:0/3680831504 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6b40c05d0 0x7fd6b413ed90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.723+0000 7fd6bc11d640 1 -- 172.21.15.125:0/3680831504 >> 172.21.15.125:0/3680831504 conn(0x7fd6b40c7910 msgr2=0x7fd6b40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.723+0000 7fd6bc11d640 1 -- 172.21.15.125:0/3680831504 shutdown_connections 2023-10-30T12:38:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.723+0000 7fd6bc11d640 1 -- 172.21.15.125:0/3680831504 wait complete. 2023-10-30T12:38:35.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.723+0000 7fd6bc11d640 1 Processor -- start 2023-10-30T12:38:35.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.723+0000 7fd6bc11d640 1 -- start start 2023-10-30T12:38:35.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6bc11d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6b40c05d0 0x7fd6b412edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:35.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6bc11d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b40c6b80 0x7fd6b412d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:35.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6bc11d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6b40c6f60 0x7fd6b414b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:35.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6bc11d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b4132310 con 0x7fd6b40c05d0 2023-10-30T12:38:35.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6bc11d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b412f580 con 0x7fd6b40c6b80 2023-10-30T12:38:35.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6bc11d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b412e840 con 0x7fd6b40c6f60 2023-10-30T12:38:35.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6b9eb9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6b40c05d0 0x7fd6b412edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:35.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6b9eb9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6b40c05d0 0x7fd6b412edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43314/0 (socket says 172.21.15.125:43314) 2023-10-30T12:38:35.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6b9eb9640 1 -- 172.21.15.125:0/78145304 learned_addr learned my addr 172.21.15.125:0/78145304 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:35.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6ba6ba640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6b40c6f60 0x7fd6b414b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:35.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6b96b8640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b40c6b80 0x7fd6b412d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:35.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6b9eb9640 1 -- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6b40c6f60 msgr2=0x7fd6b414b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:35.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6b9eb9640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6b40c6f60 0x7fd6b414b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:35.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6b9eb9640 1 -- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b40c6b80 msgr2=0x7fd6b412d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:35.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6b9eb9640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b40c6b80 0x7fd6b412d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:35.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6b9eb9640 1 -- 172.21.15.125:0/78145304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a40094d0 con 0x7fd6b40c05d0 2023-10-30T12:38:35.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6b96b8640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b40c6b80 0x7fd6b412d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:35.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6b9eb9640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6b40c05d0 0x7fd6b412edc0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000c500 tx=0x7fd6b000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:35.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6aa7fc640 1 -- 172.21.15.125:0/78145304 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0005220 con 0x7fd6b40c05d0 2023-10-30T12:38:35.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6aa7fc640 1 -- 172.21.15.125:0/78145304 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6b0005380 con 0x7fd6b40c05d0 2023-10-30T12:38:35.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6bc11d640 1 -- 172.21.15.125:0/78145304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b412ff90 con 0x7fd6b40c05d0 2023-10-30T12:38:35.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6aa7fc640 1 -- 172.21.15.125:0/78145304 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0007270 con 0x7fd6b40c05d0 2023-10-30T12:38:35.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6bc11d640 1 -- 172.21.15.125:0/78145304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b412eae0 con 0x7fd6b40c05d0 2023-10-30T12:38:35.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6bc11d640 1 -- 172.21.15.125:0/78145304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b4132450 con 0x7fd6b40c05d0 2023-10-30T12:38:35.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6aa7fc640 1 -- 172.21.15.125:0/78145304 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6b00073d0 con 0x7fd6b40c05d0 2023-10-30T12:38:35.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.727+0000 7fd6aa7fc640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6980664a0 0x7fd698068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:35.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.731+0000 7fd6aa7fc640 1 -- 172.21.15.125:0/78145304 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fd6b0083560 con 0x7fd6b40c05d0 2023-10-30T12:38:35.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.731+0000 7fd6b96b8640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6980664a0 0x7fd698068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:35.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.731+0000 7fd6b96b8640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6980664a0 0x7fd698068960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd6a40af000 tx=0x7fd6a40a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:35.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.735+0000 7fd6aa7fc640 1 -- 172.21.15.125:0/78145304 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6b0050830 con 0x7fd6b40c05d0 2023-10-30T12:38:35.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.911+0000 7fd6bc11d640 1 -- 172.21.15.125:0/78145304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6b404e170 con 0x7fd6b40c05d0 2023-10-30T12:38:35.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.915+0000 7fd6aa7fc640 1 -- 172.21.15.125:0/78145304 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fd6b0053e50 con 0x7fd6b40c05d0 2023-10-30T12:38:35.915 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:35.915 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5174,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2993299045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2993299045},{"type":"v1","addr":"172.21.15.136:6835","nonce":2993299045}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:31.250027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:35.915 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 58 2023-10-30T12:38:35.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.915+0000 7fd6bc11d640 1 -- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6980664a0 msgr2=0x7fd698068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:35.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.915+0000 7fd6bc11d640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6980664a0 0x7fd698068960 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fd6a40af000 tx=0x7fd6a40a7000 comp rx=0 tx=0).stop 2023-10-30T12:38:35.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.915+0000 7fd6bc11d640 1 -- 172.21.15.125:0/78145304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6b40c05d0 msgr2=0x7fd6b412edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:35.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.915+0000 7fd6bc11d640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6b40c05d0 0x7fd6b412edc0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000c500 tx=0x7fd6b000c9c0 comp rx=0 tx=0).stop 2023-10-30T12:38:35.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.915+0000 7fd6bc11d640 1 -- 172.21.15.125:0/78145304 shutdown_connections 2023-10-30T12:38:35.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.915+0000 7fd6bc11d640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6980664a0 0x7fd698068960 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:35.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.915+0000 7fd6bc11d640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6b40c6f60 0x7fd6b414b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:35.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.915+0000 7fd6bc11d640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6b40c6b80 0x7fd6b412d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:35.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.915+0000 7fd6bc11d640 1 --2- 172.21.15.125:0/78145304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6b40c05d0 0x7fd6b412edc0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:35.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.915+0000 7fd6bc11d640 1 -- 172.21.15.125:0/78145304 >> 172.21.15.125:0/78145304 conn(0x7fd6b40c7910 msgr2=0x7fd6b406bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:35.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.915+0000 7fd6bc11d640 1 -- 172.21.15.125:0/78145304 shutdown_connections 2023-10-30T12:38:35.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:35.915+0000 7fd6bc11d640 1 -- 172.21.15.125:0/78145304 wait complete. 2023-10-30T12:38:35.936 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:38:37.937 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:38.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc124d65640 1 Processor -- start 2023-10-30T12:38:38.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc124d65640 1 -- start start 2023-10-30T12:38:38.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc124d65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc12015d6f0 0x7fc12015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:38.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc124d65640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc1200cf160 0x7fc1200cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:38.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc124d65640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1200cb800 con 0x7fc1201538a0 2023-10-30T12:38:38.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc124d65640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1200cb330 con 0x7fc1200cf160 2023-10-30T12:38:38.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc124d65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1200cad70 con 0x7fc12015d6f0 2023-10-30T12:38:38.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc11ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc1200cf160 0x7fc1200cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:38.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc11ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc1200cf160 0x7fc1200cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43322/0 (socket says 172.21.15.125:43322) 2023-10-30T12:38:38.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc11ed9d640 1 -- 172.21.15.125:0/3638368644 learned_addr learned my addr 172.21.15.125:0/3638368644 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:38.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc115d9b640 1 --2- 172.21.15.125:0/3638368644 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc12015d6f0 0x7fc12015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:38.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc11ed9d640 1 -- 172.21.15.125:0/3638368644 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc12015d6f0 msgr2=0x7fc12015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:38.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc11ed9d640 1 --2- 172.21.15.125:0/3638368644 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc12015d6f0 0x7fc12015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:38.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc11ed9d640 1 -- 172.21.15.125:0/3638368644 >> v1:172.21.15.136:6789/0 conn(0x7fc1201538a0 legacy=0x7fc1200ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:38.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.087+0000 7fc11ed9d640 1 -- 172.21.15.125:0/3638368644 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1200cb050 con 0x7fc1200cf160 2023-10-30T12:38:38.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.091+0000 7fc11ed9d640 1 --2- 172.21.15.125:0/3638368644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1200cf160 0x7fc1200cf590 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc118009870 tx=0x7fc11801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47b666b6914a6469 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:38.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.091+0000 7fc11dd9b640 1 -- 172.21.15.125:0/3638368644 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1180a8a30 con 0x7fc1200cf160 2023-10-30T12:38:38.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.091+0000 7fc11dd9b640 1 -- 172.21.15.125:0/3638368644 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1180a8b90 con 0x7fc1200cf160 2023-10-30T12:38:38.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.091+0000 7fc11dd9b640 1 -- 172.21.15.125:0/3638368644 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1180b0440 con 0x7fc1200cf160 2023-10-30T12:38:38.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.091+0000 7fc124d65640 1 -- 172.21.15.125:0/3638368644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1200cf160 msgr2=0x7fc1200cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:38.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.091+0000 7fc124d65640 1 --2- 172.21.15.125:0/3638368644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1200cf160 0x7fc1200cf590 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc118009870 tx=0x7fc11801ed30 comp rx=0 tx=0).stop 2023-10-30T12:38:38.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.091+0000 7fc124d65640 1 -- 172.21.15.125:0/3638368644 shutdown_connections 2023-10-30T12:38:38.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.091+0000 7fc124d65640 1 --2- 172.21.15.125:0/3638368644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1200cf160 0x7fc1200cf590 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:38.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.091+0000 7fc124d65640 1 --2- 172.21.15.125:0/3638368644 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc12015d6f0 0x7fc12015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:38.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.091+0000 7fc124d65640 1 -- 172.21.15.125:0/3638368644 >> 172.21.15.125:0/3638368644 conn(0x7fc12006bfe0 msgr2=0x7fc12006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:38.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.091+0000 7fc124d65640 1 -- 172.21.15.125:0/3638368644 shutdown_connections 2023-10-30T12:38:38.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.091+0000 7fc124d65640 1 -- 172.21.15.125:0/3638368644 wait complete. 2023-10-30T12:38:38.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc124d65640 1 Processor -- start 2023-10-30T12:38:38.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc124d65640 1 -- start start 2023-10-30T12:38:38.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc124d65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1200cf160 0x7fc1201e49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:38.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc124d65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1201538a0 0x7fc1200b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:38.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc124d65640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc12015d6f0 0x7fc1201f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:38.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc124d65640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1201e3180 con 0x7fc1200cf160 2023-10-30T12:38:38.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc124d65640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1201e2a10 con 0x7fc12015d6f0 2023-10-30T12:38:38.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc124d65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1201e2cf0 con 0x7fc1201538a0 2023-10-30T12:38:38.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc11e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1200cf160 0x7fc1201e49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:38.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc11e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1200cf160 0x7fc1201e49e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43338/0 (socket says 172.21.15.125:43338) 2023-10-30T12:38:38.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc11e59c640 1 -- 172.21.15.125:0/2533700678 learned_addr learned my addr 172.21.15.125:0/2533700678 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:38.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc11ed9d640 1 --2- 172.21.15.125:0/2533700678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc12015d6f0 0x7fc1201f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:38.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc115d9b640 1 --2- 172.21.15.125:0/2533700678 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1201538a0 0x7fc1200b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:38.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc11e59c640 1 -- 172.21.15.125:0/2533700678 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1201538a0 msgr2=0x7fc1200b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:38.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc11e59c640 1 --2- 172.21.15.125:0/2533700678 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1201538a0 0x7fc1200b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:38.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc11e59c640 1 -- 172.21.15.125:0/2533700678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc12015d6f0 msgr2=0x7fc1201f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:38.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc11e59c640 1 --2- 172.21.15.125:0/2533700678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc12015d6f0 0x7fc1201f2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:38.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc11e59c640 1 -- 172.21.15.125:0/2533700678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1180094d0 con 0x7fc1200cf160 2023-10-30T12:38:38.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.095+0000 7fc11e59c640 1 --2- 172.21.15.125:0/2533700678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1200cf160 0x7fc1201e49e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc11000c7f0 tx=0x7fc11000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:38.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.099+0000 7fc1177fe640 1 -- 172.21.15.125:0/2533700678 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc110003b80 con 0x7fc1200cf160 2023-10-30T12:38:38.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.099+0000 7fc1177fe640 1 -- 172.21.15.125:0/2533700678 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc110003ce0 con 0x7fc1200cf160 2023-10-30T12:38:38.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.099+0000 7fc124d65640 1 -- 172.21.15.125:0/2533700678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1200b4a90 con 0x7fc1200cf160 2023-10-30T12:38:38.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.099+0000 7fc1177fe640 1 -- 172.21.15.125:0/2533700678 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11000aa90 con 0x7fc1200cf160 2023-10-30T12:38:38.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.099+0000 7fc124d65640 1 -- 172.21.15.125:0/2533700678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1200b4d30 con 0x7fc1200cf160 2023-10-30T12:38:38.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.099+0000 7fc124d65640 1 -- 172.21.15.125:0/2533700678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1200b2a50 con 0x7fc1200cf160 2023-10-30T12:38:38.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.099+0000 7fc1177fe640 1 -- 172.21.15.125:0/2533700678 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc11000abf0 con 0x7fc1200cf160 2023-10-30T12:38:38.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.099+0000 7fc1177fe640 1 --2- 172.21.15.125:0/2533700678 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1000664a0 0x7fc100068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:38.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.099+0000 7fc1177fe640 1 -- 172.21.15.125:0/2533700678 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fc110083ea0 con 0x7fc1200cf160 2023-10-30T12:38:38.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.099+0000 7fc115d9b640 1 --2- 172.21.15.125:0/2533700678 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1000664a0 0x7fc100068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:38.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.103+0000 7fc115d9b640 1 --2- 172.21.15.125:0/2533700678 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1000664a0 0x7fc100068960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc1200c5da0 tx=0x7fc108008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:38.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.103+0000 7fc1177fe640 1 -- 172.21.15.125:0/2533700678 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc110051100 con 0x7fc1200cf160 2023-10-30T12:38:38.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.307+0000 7fc124d65640 1 -- 172.21.15.125:0/2533700678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc12004e170 con 0x7fc1200cf160 2023-10-30T12:38:38.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.311+0000 7fc1177fe640 1 -- 172.21.15.125:0/2533700678 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fc110054720 con 0x7fc1200cf160 2023-10-30T12:38:38.310 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:38.311 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5174,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2993299045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2993299045},{"type":"v1","addr":"172.21.15.136:6835","nonce":2993299045}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:31.250027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:38.311 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 58 2023-10-30T12:38:38.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.311+0000 7fc124d65640 1 -- 172.21.15.125:0/2533700678 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1000664a0 msgr2=0x7fc100068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:38.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.311+0000 7fc124d65640 1 --2- 172.21.15.125:0/2533700678 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1000664a0 0x7fc100068960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc1200c5da0 tx=0x7fc108008040 comp rx=0 tx=0).stop 2023-10-30T12:38:38.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.311+0000 7fc124d65640 1 -- 172.21.15.125:0/2533700678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1200cf160 msgr2=0x7fc1201e49e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:38.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.311+0000 7fc124d65640 1 --2- 172.21.15.125:0/2533700678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1200cf160 0x7fc1201e49e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fc11000c7f0 tx=0x7fc11000ccb0 comp rx=0 tx=0).stop 2023-10-30T12:38:38.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.311+0000 7fc124d65640 1 -- 172.21.15.125:0/2533700678 shutdown_connections 2023-10-30T12:38:38.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.311+0000 7fc124d65640 1 --2- 172.21.15.125:0/2533700678 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1000664a0 0x7fc100068960 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:38.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.311+0000 7fc124d65640 1 --2- 172.21.15.125:0/2533700678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc12015d6f0 0x7fc1201f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:38.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.311+0000 7fc124d65640 1 --2- 172.21.15.125:0/2533700678 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1201538a0 0x7fc1200b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:38.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.311+0000 7fc124d65640 1 --2- 172.21.15.125:0/2533700678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1200cf160 0x7fc1201e49e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:38.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.311+0000 7fc124d65640 1 -- 172.21.15.125:0/2533700678 >> 172.21.15.125:0/2533700678 conn(0x7fc12006bfe0 msgr2=0x7fc120151cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:38.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.311+0000 7fc124d65640 1 -- 172.21.15.125:0/2533700678 shutdown_connections 2023-10-30T12:38:38.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:38.311+0000 7fc124d65640 1 -- 172.21.15.125:0/2533700678 wait complete. 2023-10-30T12:38:38.331 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:38:40.332 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:40.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07d0605640 1 Processor -- start 2023-10-30T12:38:40.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07d0605640 1 -- start start 2023-10-30T12:38:40.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07d0605640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07c80c05d0 0x7f07c813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:40.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07d0605640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c80c0a50 con 0x7f07c80c6f60 2023-10-30T12:38:40.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07d0605640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c806f6e0 con 0x7f07c80c6b80 2023-10-30T12:38:40.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07d0605640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c806f9c0 con 0x7f07c80c05d0 2023-10-30T12:38:40.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07ceba2640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f07c80c6f60 0x7f07c81566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44088/0 (socket says 172.21.15.125:44088) 2023-10-30T12:38:40.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07ceba2640 1 -- 172.21.15.125:0/22555074 learned_addr learned my addr 172.21.15.125:0/22555074 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:38:40.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07cdba0640 1 --2- 172.21.15.125:0/22555074 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07c80c05d0 0x7f07c813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:40.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07cd39f640 1 -- 172.21.15.125:0/22555074 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753238500 0 0) 0x7f07c80c0a50 con 0x7f07c80c6f60 2023-10-30T12:38:40.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07cd39f640 1 -- 172.21.15.125:0/22555074 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07b4003530 con 0x7f07c80c6f60 2023-10-30T12:38:40.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07cd39f640 1 -- 172.21.15.125:0/22555074 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 418903478 0 0) 0x7f07b4003530 con 0x7f07c80c6f60 2023-10-30T12:38:40.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07cd39f640 1 -- 172.21.15.125:0/22555074 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07c80c05d0 msgr2=0x7f07c813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:40.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07cd39f640 1 --2- 172.21.15.125:0/22555074 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07c80c05d0 0x7f07c813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:40.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07cd39f640 1 -- 172.21.15.125:0/22555074 >> v1:172.21.15.136:6789/0 conn(0x7f07c80c6b80 legacy=0x7f07c80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:40.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07cd39f640 1 -- 172.21.15.125:0/22555074 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c806fca0 con 0x7f07c80c6f60 2023-10-30T12:38:40.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07cd39f640 1 -- 172.21.15.125:0/22555074 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f07c0001a00 con 0x7f07c80c6f60 2023-10-30T12:38:40.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.479+0000 7f07cd39f640 1 -- 172.21.15.125:0/22555074 >> v1:172.21.15.125:6789/0 conn(0x7f07c80c6f60 legacy=0x7f07c81566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:40.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07cd39f640 1 --2- 172.21.15.125:0/22555074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07b40039b0 0x7f07b4003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:40.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07cd39f640 1 --2- 172.21.15.125:0/22555074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07b40047d0 0x7f07b4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:40.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07cd39f640 1 --2- 172.21.15.125:0/22555074 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07c80c05d0 0x7f07b4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:40.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07cd39f640 1 -- 172.21.15.125:0/22555074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c806f9c0 con 0x7f07b40039b0 2023-10-30T12:38:40.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07cd39f640 1 -- 172.21.15.125:0/22555074 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b400ba70 con 0x7f07b40047d0 2023-10-30T12:38:40.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07cd39f640 1 -- 172.21.15.125:0/22555074 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b400de40 con 0x7f07c80c05d0 2023-10-30T12:38:40.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07ce3a1640 1 --2- 172.21.15.125:0/22555074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07b40039b0 0x7f07b4003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:40.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07cdba0640 1 --2- 172.21.15.125:0/22555074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07b40047d0 0x7f07b4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:40.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07ceba2640 1 --2- 172.21.15.125:0/22555074 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07c80c05d0 0x7f07b4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:40.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07ce3a1640 1 -- 172.21.15.125:0/22555074 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07c80c05d0 msgr2=0x7f07b4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:40.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07ce3a1640 1 --2- 172.21.15.125:0/22555074 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07c80c05d0 0x7f07b4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:40.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07ce3a1640 1 -- 172.21.15.125:0/22555074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07b40047d0 msgr2=0x7f07b4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:40.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07ce3a1640 1 --2- 172.21.15.125:0/22555074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07b40047d0 0x7f07b4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:40.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07ce3a1640 1 -- 172.21.15.125:0/22555074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b4003530 con 0x7f07b40039b0 2023-10-30T12:38:40.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07ceba2640 1 --2- 172.21.15.125:0/22555074 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07c80c05d0 0x7f07b4005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:40.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.483+0000 7f07ce3a1640 1 --2- 172.21.15.125:0/22555074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07b40039b0 0x7f07b4003dd0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f07c4001750 tx=0x7f07c401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a97c2e5f89bbe11 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:40.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07cd39f640 1 -- 172.21.15.125:0/22555074 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c40052d0 con 0x7f07b40039b0 2023-10-30T12:38:40.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07cd39f640 1 -- 172.21.15.125:0/22555074 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c40a7050 con 0x7f07b40039b0 2023-10-30T12:38:40.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07cd39f640 1 -- 172.21.15.125:0/22555074 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c4003600 con 0x7f07b40039b0 2023-10-30T12:38:40.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07d0605640 1 -- 172.21.15.125:0/22555074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07b40039b0 msgr2=0x7f07b4003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:40.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07d0605640 1 --2- 172.21.15.125:0/22555074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07b40039b0 0x7f07b4003dd0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f07c4001750 tx=0x7f07c401c3a0 comp rx=0 tx=0).stop 2023-10-30T12:38:40.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07ceba2640 1 -- 172.21.15.125:0/22555074 reap_dead start 2023-10-30T12:38:40.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07d0605640 1 -- 172.21.15.125:0/22555074 shutdown_connections 2023-10-30T12:38:40.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07d0605640 1 -- 172.21.15.125:0/22555074 >> 172.21.15.125:0/22555074 conn(0x7f07c80c7910 msgr2=0x7f07c80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:40.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07d0605640 1 -- 172.21.15.125:0/22555074 shutdown_connections 2023-10-30T12:38:40.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07d0605640 1 -- 172.21.15.125:0/22555074 wait complete. 2023-10-30T12:38:40.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07d0605640 1 Processor -- start 2023-10-30T12:38:40.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07d0605640 1 -- start start 2023-10-30T12:38:40.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07d0605640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07b40039b0 0x7f07c81544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:40.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07d0605640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07c8157370 0x7f07c81f5df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:40.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07d0605640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07c81549e0 0x7f07c8138400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:40.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07d0605640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80afb50 con 0x7f07c81549e0 2023-10-30T12:38:40.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07d0605640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80b0190 con 0x7f07c8157370 2023-10-30T12:38:40.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07d0605640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80af800 con 0x7f07b40039b0 2023-10-30T12:38:40.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07ceba2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07c81549e0 0x7f07c8138400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:40.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07ceba2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07c81549e0 0x7f07c8138400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43352/0 (socket says 172.21.15.125:43352) 2023-10-30T12:38:40.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07ceba2640 1 -- 172.21.15.125:0/287835031 learned_addr learned my addr 172.21.15.125:0/287835031 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:40.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07ce3a1640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07b40039b0 0x7f07c81544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:40.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.487+0000 7f07cdba0640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07c8157370 0x7f07c81f5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:40.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07ceba2640 1 -- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07b40039b0 msgr2=0x7f07c81544a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:40.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07ceba2640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07b40039b0 0x7f07c81544a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:40.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07ceba2640 1 -- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07c8157370 msgr2=0x7f07c81f5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:40.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07ceba2640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07c8157370 0x7f07c81f5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:40.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07ceba2640 1 -- 172.21.15.125:0/287835031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c80aff70 con 0x7f07c81549e0 2023-10-30T12:38:40.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07cdba0640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07c8157370 0x7f07c81f5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:40.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07ceba2640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07c81549e0 0x7f07c8138400 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f07c80c7340 tx=0x7f07c000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:40.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07beffd640 1 -- 172.21.15.125:0/287835031 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c000eb90 con 0x7f07c81549e0 2023-10-30T12:38:40.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07beffd640 1 -- 172.21.15.125:0/287835031 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c000ecf0 con 0x7f07c81549e0 2023-10-30T12:38:40.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07d0605640 1 -- 172.21.15.125:0/287835031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c80b03e0 con 0x7f07c81549e0 2023-10-30T12:38:40.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07beffd640 1 -- 172.21.15.125:0/287835031 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c0013660 con 0x7f07c81549e0 2023-10-30T12:38:40.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07d0605640 1 -- 172.21.15.125:0/287835031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c80c1b70 con 0x7f07c81549e0 2023-10-30T12:38:40.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07d0605640 1 -- 172.21.15.125:0/287835031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c815cda0 con 0x7f07c81549e0 2023-10-30T12:38:40.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07beffd640 1 -- 172.21.15.125:0/287835031 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f07c00137c0 con 0x7f07c81549e0 2023-10-30T12:38:40.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07beffd640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07ac0664d0 0x7f07ac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:40.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07beffd640 1 -- 172.21.15.125:0/287835031 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f07c00842f0 con 0x7f07c81549e0 2023-10-30T12:38:40.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.491+0000 7f07ce3a1640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07ac0664d0 0x7f07ac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:40.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.495+0000 7f07ce3a1640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07ac0664d0 0x7f07ac068990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f07b40057b0 tx=0x7f07c40011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:40.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.495+0000 7f07beffd640 1 -- 172.21.15.125:0/287835031 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f07c0051660 con 0x7f07c81549e0 2023-10-30T12:38:40.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.675+0000 7f07d0605640 1 -- 172.21.15.125:0/287835031 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07c80afc90 con 0x7f07c81549e0 2023-10-30T12:38:40.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.675+0000 7f07beffd640 1 -- 172.21.15.125:0/287835031 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f07c80afc90 con 0x7f07c81549e0 2023-10-30T12:38:40.677 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:40.677 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5174,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2993299045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2993299045},{"type":"v1","addr":"172.21.15.136:6835","nonce":2993299045}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:31.250027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:40.678 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 58 2023-10-30T12:38:40.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.679+0000 7f07d0605640 1 -- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07ac0664d0 msgr2=0x7f07ac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:40.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.679+0000 7f07d0605640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07ac0664d0 0x7f07ac068990 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f07b40057b0 tx=0x7f07c40011a0 comp rx=0 tx=0).stop 2023-10-30T12:38:40.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.679+0000 7f07d0605640 1 -- 172.21.15.125:0/287835031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07c81549e0 msgr2=0x7f07c8138400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:40.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.679+0000 7f07d0605640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07c81549e0 0x7f07c8138400 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f07c80c7340 tx=0x7f07c000e530 comp rx=0 tx=0).stop 2023-10-30T12:38:40.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.679+0000 7f07d0605640 1 -- 172.21.15.125:0/287835031 shutdown_connections 2023-10-30T12:38:40.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.679+0000 7f07d0605640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07ac0664d0 0x7f07ac068990 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:40.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.679+0000 7f07d0605640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f07c81549e0 0x7f07c8138400 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:40.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.679+0000 7f07d0605640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f07c8157370 0x7f07c81f5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:40.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.679+0000 7f07d0605640 1 --2- 172.21.15.125:0/287835031 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f07b40039b0 0x7f07c81544a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:40.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.679+0000 7f07d0605640 1 -- 172.21.15.125:0/287835031 >> 172.21.15.125:0/287835031 conn(0x7f07c80c7910 msgr2=0x7f07c80b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:40.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.679+0000 7f07d0605640 1 -- 172.21.15.125:0/287835031 shutdown_connections 2023-10-30T12:38:40.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:40.679+0000 7f07d0605640 1 -- 172.21.15.125:0/287835031 wait complete. 2023-10-30T12:38:40.698 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:38:42.699 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:42.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1408079640 1 Processor -- start 2023-10-30T12:38:42.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1408079640 1 -- start start 2023-10-30T12:38:42.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1408079640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14001538a0 0x7f140015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:42.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1408079640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14000cb800 con 0x7f14000cf160 2023-10-30T12:38:42.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1408079640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f140015dc30 con 0x7f14000ca0e0 2023-10-30T12:38:42.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1408079640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14000cb330 con 0x7f14001538a0 2023-10-30T12:38:42.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1406616640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f14000cf160 0x7f1400164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40098/0 (socket says 172.21.15.125:40098) 2023-10-30T12:38:42.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1406616640 1 -- 172.21.15.125:0/1064126627 learned_addr learned my addr 172.21.15.125:0/1064126627 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:38:42.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1405e15640 1 --2- 172.21.15.125:0/1064126627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14001538a0 0x7f140015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:42.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 -- 172.21.15.125:0/1064126627 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3886888935 0 0) 0x7f14000cb800 con 0x7f14000cf160 2023-10-30T12:38:42.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 -- 172.21.15.125:0/1064126627 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13ec003530 con 0x7f14000cf160 2023-10-30T12:38:42.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 -- 172.21.15.125:0/1064126627 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2312706956 0 0) 0x7f13ec003530 con 0x7f14000cf160 2023-10-30T12:38:42.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 -- 172.21.15.125:0/1064126627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14001538a0 msgr2=0x7f140015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:42.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 --2- 172.21.15.125:0/1064126627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14001538a0 0x7f140015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:42.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 -- 172.21.15.125:0/1064126627 >> v1:172.21.15.136:6789/0 conn(0x7f14000ca0e0 legacy=0x7f140015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:42.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 -- 172.21.15.125:0/1064126627 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14000cad70 con 0x7f14000cf160 2023-10-30T12:38:42.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 -- 172.21.15.125:0/1064126627 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f13f8001a00 con 0x7f14000cf160 2023-10-30T12:38:42.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 -- 172.21.15.125:0/1064126627 >> v1:172.21.15.125:6789/0 conn(0x7f14000cf160 legacy=0x7f1400164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:42.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 --2- 172.21.15.125:0/1064126627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13ec0039b0 0x7f13ec003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:42.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 --2- 172.21.15.125:0/1064126627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14001538a0 0x7f13ec0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:42.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 --2- 172.21.15.125:0/1064126627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13ec004d10 0x7f13ec005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:42.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 -- 172.21.15.125:0/1064126627 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14000cb330 con 0x7f13ec0039b0 2023-10-30T12:38:42.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 -- 172.21.15.125:0/1064126627 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec00bad0 con 0x7f13ec004d10 2023-10-30T12:38:42.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1404e13640 1 -- 172.21.15.125:0/1064126627 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec00bc90 con 0x7f14001538a0 2023-10-30T12:38:42.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1405e15640 1 --2- 172.21.15.125:0/1064126627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13ec0039b0 0x7f13ec003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:42.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1406616640 1 --2- 172.21.15.125:0/1064126627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13ec004d10 0x7f13ec005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:42.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1405614640 1 --2- 172.21.15.125:0/1064126627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14001538a0 0x7f13ec0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:42.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1405e15640 1 -- 172.21.15.125:0/1064126627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14001538a0 msgr2=0x7f13ec0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:42.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1405e15640 1 --2- 172.21.15.125:0/1064126627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14001538a0 0x7f13ec0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:42.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1405e15640 1 -- 172.21.15.125:0/1064126627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13ec004d10 msgr2=0x7f13ec005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:42.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1405e15640 1 --2- 172.21.15.125:0/1064126627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13ec004d10 0x7f13ec005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:42.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1405e15640 1 -- 172.21.15.125:0/1064126627 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec003530 con 0x7f13ec0039b0 2023-10-30T12:38:42.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.847+0000 7f1406616640 1 --2- 172.21.15.125:0/1064126627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13ec004d10 0x7f13ec005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:42.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.851+0000 7f1405e15640 1 --2- 172.21.15.125:0/1064126627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13ec0039b0 0x7f13ec003dd0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f13fc009ab0 tx=0x7f13fc01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4db9f4a8a19790 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:42.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.851+0000 7f1404e13640 1 -- 172.21.15.125:0/1064126627 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc0a5510 con 0x7f13ec0039b0 2023-10-30T12:38:42.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.851+0000 7f1404e13640 1 -- 172.21.15.125:0/1064126627 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13fc0a5670 con 0x7f13ec0039b0 2023-10-30T12:38:42.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.851+0000 7f1404e13640 1 -- 172.21.15.125:0/1064126627 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13fc0a5a20 con 0x7f13ec0039b0 2023-10-30T12:38:42.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.851+0000 7f1408079640 1 -- 172.21.15.125:0/1064126627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13ec0039b0 msgr2=0x7f13ec003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:42.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.851+0000 7f1408079640 1 --2- 172.21.15.125:0/1064126627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13ec0039b0 0x7f13ec003dd0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f13fc009ab0 tx=0x7f13fc01ca30 comp rx=0 tx=0).stop 2023-10-30T12:38:42.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.851+0000 7f1406616640 1 -- 172.21.15.125:0/1064126627 reap_dead start 2023-10-30T12:38:42.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.851+0000 7f1408079640 1 -- 172.21.15.125:0/1064126627 shutdown_connections 2023-10-30T12:38:42.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.851+0000 7f1408079640 1 -- 172.21.15.125:0/1064126627 >> 172.21.15.125:0/1064126627 conn(0x7f140006bfe0 msgr2=0x7f140006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:42.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.851+0000 7f1408079640 1 -- 172.21.15.125:0/1064126627 shutdown_connections 2023-10-30T12:38:42.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.851+0000 7f1408079640 1 -- 172.21.15.125:0/1064126627 wait complete. 2023-10-30T12:38:42.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1408079640 1 Processor -- start 2023-10-30T12:38:42.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1408079640 1 -- start start 2023-10-30T12:38:42.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1408079640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13ec0039b0 0x7f14001f3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:42.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1408079640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14000b4b90 0x7f14001e5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:42.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1408079640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1400157b70 0x7f1400157ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:42.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1408079640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14001e6c30 con 0x7f14000b4b90 2023-10-30T12:38:42.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1408079640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1400158f00 con 0x7f13ec0039b0 2023-10-30T12:38:42.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1408079640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14001576e0 con 0x7f1400157b70 2023-10-30T12:38:42.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1405614640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14000b4b90 0x7f14001e5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:42.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1405614640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14000b4b90 0x7f14001e5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41286/0 (socket says 172.21.15.125:41286) 2023-10-30T12:38:42.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1405614640 1 -- 172.21.15.125:0/1205622427 learned_addr learned my addr 172.21.15.125:0/1205622427 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:42.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1406616640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1400157b70 0x7f1400157ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:42.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1405e15640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13ec0039b0 0x7f14001f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:42.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1405614640 1 -- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1400157b70 msgr2=0x7f1400157ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:42.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1405614640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1400157b70 0x7f1400157ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:42.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1405614640 1 -- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13ec0039b0 msgr2=0x7f14001f3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:42.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1405614640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13ec0039b0 0x7f14001f3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:42.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1405614640 1 -- 172.21.15.125:0/1205622427 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14001f0470 con 0x7f14000b4b90 2023-10-30T12:38:42.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1405e15640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13ec0039b0 0x7f14001f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:42.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1405614640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14000b4b90 0x7f14001e5e70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f13f000e580 tx=0x7f13f000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:42.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f13f67fc640 1 -- 172.21.15.125:0/1205622427 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f00052e0 con 0x7f14000b4b90 2023-10-30T12:38:42.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f13f67fc640 1 -- 172.21.15.125:0/1205622427 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13f0005440 con 0x7f14000b4b90 2023-10-30T12:38:42.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1408079640 1 -- 172.21.15.125:0/1205622427 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14001f3940 con 0x7f14000b4b90 2023-10-30T12:38:42.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f13f67fc640 1 -- 172.21.15.125:0/1205622427 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13f0003820 con 0x7f14000b4b90 2023-10-30T12:38:42.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.855+0000 7f1408079640 1 -- 172.21.15.125:0/1205622427 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14001efbd0 con 0x7f14000b4b90 2023-10-30T12:38:42.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.859+0000 7f1408079640 1 -- 172.21.15.125:0/1205622427 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1400157890 con 0x7f14000b4b90 2023-10-30T12:38:42.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.859+0000 7f13f67fc640 1 -- 172.21.15.125:0/1205622427 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f13f0003980 con 0x7f14000b4b90 2023-10-30T12:38:42.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.859+0000 7f13f67fc640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13e40664a0 0x7f13e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:42.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.859+0000 7f13f67fc640 1 -- 172.21.15.125:0/1205622427 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f13f0083c60 con 0x7f14000b4b90 2023-10-30T12:38:42.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.859+0000 7f1405e15640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13e40664a0 0x7f13e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:42.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.859+0000 7f1405e15640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13e40664a0 0x7f13e4068960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f13fc0af000 tx=0x7f13fc0046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:42.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:42.863+0000 7f13f67fc640 1 -- 172.21.15.125:0/1205622427 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1400157890 con 0x7f14000b4b90 2023-10-30T12:38:43.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:43.043+0000 7f1408079640 1 -- 172.21.15.125:0/1205622427 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f140015dc30 con 0x7f14000b4b90 2023-10-30T12:38:43.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:43.047+0000 7f13f67fc640 1 -- 172.21.15.125:0/1205622427 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f140015dc30 con 0x7f14000b4b90 2023-10-30T12:38:43.047 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:43.047 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5174,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2993299045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2993299045},{"type":"v1","addr":"172.21.15.136:6835","nonce":2993299045}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:31.250027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:43.047 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 58 2023-10-30T12:38:43.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:43.047+0000 7f1408079640 1 -- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13e40664a0 msgr2=0x7f13e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:43.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:43.047+0000 7f1408079640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13e40664a0 0x7f13e4068960 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f13fc0af000 tx=0x7f13fc0046d0 comp rx=0 tx=0).stop 2023-10-30T12:38:43.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:43.047+0000 7f1408079640 1 -- 172.21.15.125:0/1205622427 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14000b4b90 msgr2=0x7f14001e5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:43.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:43.047+0000 7f1408079640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14000b4b90 0x7f14001e5e70 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f13f000e580 tx=0x7f13f000ea40 comp rx=0 tx=0).stop 2023-10-30T12:38:43.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:43.047+0000 7f1408079640 1 -- 172.21.15.125:0/1205622427 shutdown_connections 2023-10-30T12:38:43.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:43.047+0000 7f1408079640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13e40664a0 0x7f13e4068960 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:43.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:43.047+0000 7f1408079640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1400157b70 0x7f1400157ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:43.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:43.047+0000 7f1408079640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14000b4b90 0x7f14001e5e70 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:43.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:43.047+0000 7f1408079640 1 --2- 172.21.15.125:0/1205622427 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13ec0039b0 0x7f14001f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:43.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:43.047+0000 7f1408079640 1 -- 172.21.15.125:0/1205622427 >> 172.21.15.125:0/1205622427 conn(0x7f140006bfe0 msgr2=0x7f1400151800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:43.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:43.047+0000 7f1408079640 1 -- 172.21.15.125:0/1205622427 shutdown_connections 2023-10-30T12:38:43.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:43.047+0000 7f1408079640 1 -- 172.21.15.125:0/1205622427 wait complete. 2023-10-30T12:38:43.067 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:38:45.068 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:45.166 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:38:45.170 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:38:45.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.219+0000 7f4ecdb65640 1 Processor -- start 2023-10-30T12:38:45.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.219+0000 7f4ecdb65640 1 -- start start 2023-10-30T12:38:45.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ecdb65640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4ec80c6bd0 0x7f4ec806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:45.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ecdb65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ec8071110 0x7f4ec8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:45.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ecdb65640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ec8066cf0 con 0x7f4ec806f7a0 2023-10-30T12:38:45.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ecdb65640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ec80700c0 con 0x7f4ec80c6bd0 2023-10-30T12:38:45.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ecdb65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec80c70d0 con 0x7f4ec8071110 2023-10-30T12:38:45.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec77fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4ec806f7a0 0x7f4ec8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40104/0 (socket says 172.21.15.125:40104) 2023-10-30T12:38:45.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec77fe640 1 -- 172.21.15.125:0/417477063 learned_addr learned my addr 172.21.15.125:0/417477063 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:38:45.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec6ffd640 1 --2- 172.21.15.125:0/417477063 >> v2:172.21.15.136:3300/0 conn(0x7f4ec80c6bd0 0x7f4ec806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:45.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec7fff640 1 --2- 172.21.15.125:0/417477063 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ec8071110 0x7f4ec8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:45.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 -- 172.21.15.125:0/417477063 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969143121 0 0) 0x7f4ec8066cf0 con 0x7f4ec806f7a0 2023-10-30T12:38:45.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 -- 172.21.15.125:0/417477063 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb0003530 con 0x7f4ec806f7a0 2023-10-30T12:38:45.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 -- 172.21.15.125:0/417477063 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4271234590 0 0) 0x7f4eb0003530 con 0x7f4ec806f7a0 2023-10-30T12:38:45.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 -- 172.21.15.125:0/417477063 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ec8071110 msgr2=0x7f4ec8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:45.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 --2- 172.21.15.125:0/417477063 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ec8071110 0x7f4ec8158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:45.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 -- 172.21.15.125:0/417477063 >> v2:172.21.15.136:3300/0 conn(0x7f4ec80c6bd0 msgr2=0x7f4ec806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:45.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 --2- 172.21.15.125:0/417477063 >> v2:172.21.15.136:3300/0 conn(0x7f4ec80c6bd0 0x7f4ec806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:45.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 -- 172.21.15.125:0/417477063 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec80c7390 con 0x7f4ec806f7a0 2023-10-30T12:38:45.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 -- 172.21.15.125:0/417477063 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f4eb8001680 con 0x7f4ec806f7a0 2023-10-30T12:38:45.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 -- 172.21.15.125:0/417477063 >> v1:172.21.15.125:6789/0 conn(0x7f4ec806f7a0 legacy=0x7f4ec8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:45.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 --2- 172.21.15.125:0/417477063 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ec8071110 0x7f4eb0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:45.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 --2- 172.21.15.125:0/417477063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4eb0003f00 0x7f4eb00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:45.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 --2- 172.21.15.125:0/417477063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4eb0004d10 0x7f4eb0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:45.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 -- 172.21.15.125:0/417477063 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec80700c0 con 0x7f4eb0003f00 2023-10-30T12:38:45.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 -- 172.21.15.125:0/417477063 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec80c70d0 con 0x7f4eb0004d10 2023-10-30T12:38:45.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec67fc640 1 -- 172.21.15.125:0/417477063 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb000b9c0 con 0x7f4ec8071110 2023-10-30T12:38:45.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec6ffd640 1 --2- 172.21.15.125:0/417477063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4eb0003f00 0x7f4eb00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:45.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec77fe640 1 --2- 172.21.15.125:0/417477063 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ec8071110 0x7f4eb0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:45.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec7fff640 1 --2- 172.21.15.125:0/417477063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4eb0004d10 0x7f4eb0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:45.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec6ffd640 1 -- 172.21.15.125:0/417477063 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ec8071110 msgr2=0x7f4eb0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:45.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec6ffd640 1 --2- 172.21.15.125:0/417477063 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ec8071110 0x7f4eb0003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:45.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec6ffd640 1 -- 172.21.15.125:0/417477063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4eb0004d10 msgr2=0x7f4eb0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:45.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec6ffd640 1 --2- 172.21.15.125:0/417477063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4eb0004d10 0x7f4eb0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec6ffd640 1 -- 172.21.15.125:0/417477063 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb0003530 con 0x7f4eb0003f00 2023-10-30T12:38:45.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.223+0000 7f4ec7fff640 1 --2- 172.21.15.125:0/417477063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4eb0004d10 0x7f4eb0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:45.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.227+0000 7f4ec6ffd640 1 --2- 172.21.15.125:0/417477063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4eb0003f00 0x7f4eb00047d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc009ab0 tx=0x7f4ebc01ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652ea6d71eddef2e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:45.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.227+0000 7f4ec67fc640 1 -- 172.21.15.125:0/417477063 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc0a5660 con 0x7f4eb0003f00 2023-10-30T12:38:45.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.227+0000 7f4ec67fc640 1 -- 172.21.15.125:0/417477063 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc0a57c0 con 0x7f4eb0003f00 2023-10-30T12:38:45.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.227+0000 7f4ec67fc640 1 -- 172.21.15.125:0/417477063 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc0a5b70 con 0x7f4eb0003f00 2023-10-30T12:38:45.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.227+0000 7f4ecdb65640 1 -- 172.21.15.125:0/417477063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4eb0003f00 msgr2=0x7f4eb00047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:45.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.227+0000 7f4ecdb65640 1 --2- 172.21.15.125:0/417477063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4eb0003f00 0x7f4eb00047d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc009ab0 tx=0x7f4ebc01ca70 comp rx=0 tx=0).stop 2023-10-30T12:38:45.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.227+0000 7f4ec7fff640 1 -- 172.21.15.125:0/417477063 reap_dead start 2023-10-30T12:38:45.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.227+0000 7f4ecdb65640 1 -- 172.21.15.125:0/417477063 shutdown_connections 2023-10-30T12:38:45.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.227+0000 7f4ecdb65640 1 -- 172.21.15.125:0/417477063 >> 172.21.15.125:0/417477063 conn(0x7f4ec80c7910 msgr2=0x7f4ec80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:45.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.227+0000 7f4ecdb65640 1 -- 172.21.15.125:0/417477063 shutdown_connections 2023-10-30T12:38:45.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.227+0000 7f4ecdb65640 1 -- 172.21.15.125:0/417477063 wait complete. 2023-10-30T12:38:45.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.227+0000 7f4ecdb65640 1 Processor -- start 2023-10-30T12:38:45.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ecdb65640 1 -- start start 2023-10-30T12:38:45.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ecdb65640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4eb0003f00 0x7f4ec80aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:45.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ecdb65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ec8154ea0 0x7f4ec81556c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:45.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ecdb65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ec8154810 0x7f4ec8155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:45.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ecdb65640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec8156fb0 con 0x7f4ec8154ea0 2023-10-30T12:38:45.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ecdb65640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec8157290 con 0x7f4eb0003f00 2023-10-30T12:38:45.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ecdb65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec8157570 con 0x7f4ec8154810 2023-10-30T12:38:45.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ec6ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ec8154ea0 0x7f4ec81556c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:45.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ec6ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ec8154ea0 0x7f4ec81556c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41302/0 (socket says 172.21.15.125:41302) 2023-10-30T12:38:45.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ec6ffd640 1 -- 172.21.15.125:0/1877506296 learned_addr learned my addr 172.21.15.125:0/1877506296 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:45.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ec7fff640 1 --2- 172.21.15.125:0/1877506296 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ec8154810 0x7f4ec8155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:45.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ec77fe640 1 --2- 172.21.15.125:0/1877506296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4eb0003f00 0x7f4ec80aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:45.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ec6ffd640 1 -- 172.21.15.125:0/1877506296 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ec8154810 msgr2=0x7f4ec8155c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:45.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ec6ffd640 1 --2- 172.21.15.125:0/1877506296 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ec8154810 0x7f4ec8155c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:45.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ec6ffd640 1 -- 172.21.15.125:0/1877506296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4eb0003f00 msgr2=0x7f4ec80aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:45.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ec6ffd640 1 --2- 172.21.15.125:0/1877506296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4eb0003f00 0x7f4ec80aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:45.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ec6ffd640 1 -- 172.21.15.125:0/1877506296 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec8157850 con 0x7f4ec8154ea0 2023-10-30T12:38:45.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ec6ffd640 1 --2- 172.21.15.125:0/1877506296 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ec8154ea0 0x7f4ec81556c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc0b0040 tx=0x7f4eb0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:45.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4eaffff640 1 -- 172.21.15.125:0/1877506296 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc0ad850 con 0x7f4ec8154ea0 2023-10-30T12:38:45.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4eaffff640 1 -- 172.21.15.125:0/1877506296 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ebc0a5e90 con 0x7f4ec8154ea0 2023-10-30T12:38:45.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4eaffff640 1 -- 172.21.15.125:0/1877506296 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ebc0adbf0 con 0x7f4ec8154ea0 2023-10-30T12:38:45.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ecdb65640 1 -- 172.21.15.125:0/1877506296 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec8157b30 con 0x7f4ec8154ea0 2023-10-30T12:38:45.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ecdb65640 1 -- 172.21.15.125:0/1877506296 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec80af0c0 con 0x7f4ec8154ea0 2023-10-30T12:38:45.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.231+0000 7f4ecdb65640 1 -- 172.21.15.125:0/1877506296 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec804e170 con 0x7f4ec8154ea0 2023-10-30T12:38:45.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.235+0000 7f4eaffff640 1 -- 172.21.15.125:0/1877506296 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4ebc0add50 con 0x7f4ec8154ea0 2023-10-30T12:38:45.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.235+0000 7f4eaffff640 1 --2- 172.21.15.125:0/1877506296 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4eb40664a0 0x7f4eb4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:45.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.235+0000 7f4eaffff640 1 -- 172.21.15.125:0/1877506296 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f4ebc11e2a0 con 0x7f4ec8154ea0 2023-10-30T12:38:45.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.235+0000 7f4ec77fe640 1 --2- 172.21.15.125:0/1877506296 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4eb40664a0 0x7f4eb4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:45.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.235+0000 7f4ec77fe640 1 --2- 172.21.15.125:0/1877506296 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4eb40664a0 0x7f4eb4068960 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8001600 tx=0x7f4eb8007900 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:45.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.239+0000 7f4eaffff640 1 -- 172.21.15.125:0/1877506296 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4ebc0eb620 con 0x7f4ec8154ea0 2023-10-30T12:38:45.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.423+0000 7f4ecdb65640 1 -- 172.21.15.125:0/1877506296 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ec81510d0 con 0x7f4ec8154ea0 2023-10-30T12:38:45.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.423+0000 7f4eaffff640 1 -- 172.21.15.125:0/1877506296 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f4ebc0eec40 con 0x7f4ec8154ea0 2023-10-30T12:38:45.424 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:45.424 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5174,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2993299045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2993299045},{"type":"v1","addr":"172.21.15.136:6835","nonce":2993299045}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:31.250027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:45.424 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 58 2023-10-30T12:38:45.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.423+0000 7f4ecdb65640 1 -- 172.21.15.125:0/1877506296 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4eb40664a0 msgr2=0x7f4eb4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:45.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.423+0000 7f4ecdb65640 1 --2- 172.21.15.125:0/1877506296 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4eb40664a0 0x7f4eb4068960 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f4eb8001600 tx=0x7f4eb8007900 comp rx=0 tx=0).stop 2023-10-30T12:38:45.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.423+0000 7f4ecdb65640 1 -- 172.21.15.125:0/1877506296 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ec8154ea0 msgr2=0x7f4ec81556c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:45.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.423+0000 7f4ecdb65640 1 --2- 172.21.15.125:0/1877506296 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ec8154ea0 0x7f4ec81556c0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f4ebc0b0040 tx=0x7f4eb0005800 comp rx=0 tx=0).stop 2023-10-30T12:38:45.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.423+0000 7f4ecdb65640 1 -- 172.21.15.125:0/1877506296 shutdown_connections 2023-10-30T12:38:45.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.423+0000 7f4ecdb65640 1 --2- 172.21.15.125:0/1877506296 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4eb40664a0 0x7f4eb4068960 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:45.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.423+0000 7f4ecdb65640 1 --2- 172.21.15.125:0/1877506296 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ec8154810 0x7f4ec8155c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:45.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.423+0000 7f4ecdb65640 1 --2- 172.21.15.125:0/1877506296 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ec8154ea0 0x7f4ec81556c0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:45.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.423+0000 7f4ecdb65640 1 --2- 172.21.15.125:0/1877506296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4eb0003f00 0x7f4ec80aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:45.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.423+0000 7f4ecdb65640 1 -- 172.21.15.125:0/1877506296 >> 172.21.15.125:0/1877506296 conn(0x7f4ec80c7910 msgr2=0x7f4ec80c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:45.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.427+0000 7f4ecdb65640 1 -- 172.21.15.125:0/1877506296 shutdown_connections 2023-10-30T12:38:45.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:45.427+0000 7f4ecdb65640 1 -- 172.21.15.125:0/1877506296 wait complete. 2023-10-30T12:38:45.446 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:38:47.447 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:47.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15dadd3640 1 Processor -- start 2023-10-30T12:38:47.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15dadd3640 1 -- start start 2023-10-30T12:38:47.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15dadd3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f15d41538a0 0x7f15d415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:47.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15dadd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15d40ca0e0 0x7f15d415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:47.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15dadd3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15d40cb800 con 0x7f15d40cf160 2023-10-30T12:38:47.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15dadd3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15d415dc30 con 0x7f15d41538a0 2023-10-30T12:38:47.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15dadd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d40cb330 con 0x7f15d40ca0e0 2023-10-30T12:38:47.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15d9370640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f15d40cf160 0x7f15d40cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40112/0 (socket says 172.21.15.125:40112) 2023-10-30T12:38:47.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15d9370640 1 -- 172.21.15.125:0/3816760791 learned_addr learned my addr 172.21.15.125:0/3816760791 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:38:47.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15cbfff640 1 --2- 172.21.15.125:0/3816760791 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15d40ca0e0 0x7f15d415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:47.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15d8b6f640 1 --2- 172.21.15.125:0/3816760791 >> v2:172.21.15.136:3300/0 conn(0x7f15d41538a0 0x7f15d415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:47.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15cb7fe640 1 -- 172.21.15.125:0/3816760791 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858168566 0 0) 0x7f15d40cb800 con 0x7f15d40cf160 2023-10-30T12:38:47.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15cb7fe640 1 -- 172.21.15.125:0/3816760791 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15bc003530 con 0x7f15d40cf160 2023-10-30T12:38:47.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15cbfff640 1 -- 172.21.15.125:0/3816760791 >> v2:172.21.15.136:3300/0 conn(0x7f15d41538a0 msgr2=0x7f15d415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:47.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15cbfff640 1 --2- 172.21.15.125:0/3816760791 >> v2:172.21.15.136:3300/0 conn(0x7f15d41538a0 0x7f15d415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:47.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15cbfff640 1 -- 172.21.15.125:0/3816760791 >> v1:172.21.15.125:6789/0 conn(0x7f15d40cf160 legacy=0x7f15d40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:47.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15cb7fe640 1 -- 172.21.15.125:0/3816760791 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2346762150 0 0) 0x7f15bc003530 con 0x7f15d40cf160 2023-10-30T12:38:47.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.591+0000 7f15cbfff640 1 -- 172.21.15.125:0/3816760791 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d40cad70 con 0x7f15d40ca0e0 2023-10-30T12:38:47.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.595+0000 7f15cbfff640 1 --2- 172.21.15.125:0/3816760791 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15d40ca0e0 0x7f15d415a090 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f15c40097c0 tx=0x7f15c401eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b21d48427889fe1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:47.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.595+0000 7f15cb7fe640 1 -- 172.21.15.125:0/3816760791 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15cc001a00 con 0x7f15d40ca0e0 2023-10-30T12:38:47.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.595+0000 7f15cb7fe640 1 -- 172.21.15.125:0/3816760791 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c40a85d0 con 0x7f15d40ca0e0 2023-10-30T12:38:47.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.595+0000 7f15cb7fe640 1 -- 172.21.15.125:0/3816760791 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c40a8960 con 0x7f15d40ca0e0 2023-10-30T12:38:47.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.595+0000 7f15dadd3640 1 -- 172.21.15.125:0/3816760791 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15d40ca0e0 msgr2=0x7f15d415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:47.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.595+0000 7f15dadd3640 1 --2- 172.21.15.125:0/3816760791 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15d40ca0e0 0x7f15d415a090 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f15c40097c0 tx=0x7f15c401eb90 comp rx=0 tx=0).stop 2023-10-30T12:38:47.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.595+0000 7f15dadd3640 1 -- 172.21.15.125:0/3816760791 shutdown_connections 2023-10-30T12:38:47.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.595+0000 7f15dadd3640 1 --2- 172.21.15.125:0/3816760791 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15d40ca0e0 0x7f15d415a090 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:47.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.595+0000 7f15dadd3640 1 --2- 172.21.15.125:0/3816760791 >> v2:172.21.15.136:3300/0 conn(0x7f15d41538a0 0x7f15d415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:47.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.595+0000 7f15dadd3640 1 -- 172.21.15.125:0/3816760791 >> 172.21.15.125:0/3816760791 conn(0x7f15d406bfe0 msgr2=0x7f15d406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:47.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.595+0000 7f15dadd3640 1 -- 172.21.15.125:0/3816760791 shutdown_connections 2023-10-30T12:38:47.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.595+0000 7f15dadd3640 1 -- 172.21.15.125:0/3816760791 wait complete. 2023-10-30T12:38:47.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15dadd3640 1 Processor -- start 2023-10-30T12:38:47.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15dadd3640 1 -- start start 2023-10-30T12:38:47.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15dadd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15d40ca0e0 0x7f15d4143870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:47.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15dadd3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15d40cf160 0x7f15d41442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:47.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15dadd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15d41538a0 0x7f15d41ef550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:47.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15dadd3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d41450c0 con 0x7f15d40cf160 2023-10-30T12:38:47.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15dadd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d41ef130 con 0x7f15d41538a0 2023-10-30T12:38:47.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15dadd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15d415d420 con 0x7f15d40ca0e0 2023-10-30T12:38:47.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15cbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15d40cf160 0x7f15d41442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:47.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15d8b6f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15d40ca0e0 0x7f15d4143870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:47.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15cbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15d40cf160 0x7f15d41442b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41312/0 (socket says 172.21.15.125:41312) 2023-10-30T12:38:47.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15cbfff640 1 -- 172.21.15.125:0/2437908744 learned_addr learned my addr 172.21.15.125:0/2437908744 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:47.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15d9370640 1 --2- 172.21.15.125:0/2437908744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15d41538a0 0x7f15d41ef550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:47.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15cbfff640 1 -- 172.21.15.125:0/2437908744 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15d40ca0e0 msgr2=0x7f15d4143870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:47.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15cbfff640 1 --2- 172.21.15.125:0/2437908744 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15d40ca0e0 0x7f15d4143870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15cbfff640 1 -- 172.21.15.125:0/2437908744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15d41538a0 msgr2=0x7f15d41ef550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15cbfff640 1 --2- 172.21.15.125:0/2437908744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15d41538a0 0x7f15d41ef550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15cbfff640 1 -- 172.21.15.125:0/2437908744 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c4009420 con 0x7f15d40cf160 2023-10-30T12:38:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15cbfff640 1 --2- 172.21.15.125:0/2437908744 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15d40cf160 0x7f15d41442b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f15c401d240 tx=0x7f15c401b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:47.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15c97fa640 1 -- 172.21.15.125:0/2437908744 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c40b0850 con 0x7f15d40cf160 2023-10-30T12:38:47.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15c97fa640 1 -- 172.21.15.125:0/2437908744 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15c40b09d0 con 0x7f15d40cf160 2023-10-30T12:38:47.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15dadd3640 1 -- 172.21.15.125:0/2437908744 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d4145b50 con 0x7f15d40cf160 2023-10-30T12:38:47.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15c97fa640 1 -- 172.21.15.125:0/2437908744 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15c40b0d80 con 0x7f15d40cf160 2023-10-30T12:38:47.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.599+0000 7f15dadd3640 1 -- 172.21.15.125:0/2437908744 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d415d8f0 con 0x7f15d40cf160 2023-10-30T12:38:47.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.603+0000 7f15dadd3640 1 -- 172.21.15.125:0/2437908744 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d40b2a50 con 0x7f15d40cf160 2023-10-30T12:38:47.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.603+0000 7f15c97fa640 1 -- 172.21.15.125:0/2437908744 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f15c40b7040 con 0x7f15d40cf160 2023-10-30T12:38:47.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.603+0000 7f15c97fa640 1 --2- 172.21.15.125:0/2437908744 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15c00664a0 0x7f15c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:47.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.603+0000 7f15c97fa640 1 -- 172.21.15.125:0/2437908744 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f15c41216d0 con 0x7f15d40cf160 2023-10-30T12:38:47.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.603+0000 7f15d8b6f640 1 --2- 172.21.15.125:0/2437908744 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15c00664a0 0x7f15c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:47.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.603+0000 7f15d8b6f640 1 --2- 172.21.15.125:0/2437908744 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15c00664a0 0x7f15c0068960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f15d0009ac0 tx=0x7f15d0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:47.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.607+0000 7f15c97fa640 1 -- 172.21.15.125:0/2437908744 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f15c40ee880 con 0x7f15d40cf160 2023-10-30T12:38:47.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.787+0000 7f15dadd3640 1 -- 172.21.15.125:0/2437908744 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15d41ef270 con 0x7f15d40cf160 2023-10-30T12:38:47.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.787+0000 7f15c97fa640 1 -- 172.21.15.125:0/2437908744 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f15d41ef270 con 0x7f15d40cf160 2023-10-30T12:38:47.789 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:47.789 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5174,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2993299045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2993299045},{"type":"v1","addr":"172.21.15.136:6835","nonce":2993299045}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:31.250027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:47.790 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 58 2023-10-30T12:38:47.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.791+0000 7f15dadd3640 1 -- 172.21.15.125:0/2437908744 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15c00664a0 msgr2=0x7f15c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:47.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.791+0000 7f15dadd3640 1 --2- 172.21.15.125:0/2437908744 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15c00664a0 0x7f15c0068960 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f15d0009ac0 tx=0x7f15d0009190 comp rx=0 tx=0).stop 2023-10-30T12:38:47.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.791+0000 7f15dadd3640 1 -- 172.21.15.125:0/2437908744 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15d40cf160 msgr2=0x7f15d41442b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:47.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.791+0000 7f15dadd3640 1 --2- 172.21.15.125:0/2437908744 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15d40cf160 0x7f15d41442b0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f15c401d240 tx=0x7f15c401b930 comp rx=0 tx=0).stop 2023-10-30T12:38:47.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.791+0000 7f15dadd3640 1 -- 172.21.15.125:0/2437908744 shutdown_connections 2023-10-30T12:38:47.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.791+0000 7f15dadd3640 1 --2- 172.21.15.125:0/2437908744 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15c00664a0 0x7f15c0068960 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:47.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.791+0000 7f15dadd3640 1 --2- 172.21.15.125:0/2437908744 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15d41538a0 0x7f15d41ef550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:47.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.791+0000 7f15dadd3640 1 --2- 172.21.15.125:0/2437908744 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15d40cf160 0x7f15d41442b0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:47.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.791+0000 7f15dadd3640 1 --2- 172.21.15.125:0/2437908744 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15d40ca0e0 0x7f15d4143870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:47.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.791+0000 7f15dadd3640 1 -- 172.21.15.125:0/2437908744 >> 172.21.15.125:0/2437908744 conn(0x7f15d406bfe0 msgr2=0x7f15d406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:47.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.791+0000 7f15dadd3640 1 -- 172.21.15.125:0/2437908744 shutdown_connections 2023-10-30T12:38:47.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:47.791+0000 7f15dadd3640 1 -- 172.21.15.125:0/2437908744 wait complete. 2023-10-30T12:38:47.809 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T12:38:49.810 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:49.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd75b2c9640 1 Processor -- start 2023-10-30T12:38:49.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd75b2c9640 1 -- start start 2023-10-30T12:38:49.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd75b2c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd75406e470 0x7fd75406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:49.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd75b2c9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd7540c8490 0x7fd7540c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:49.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd75b2c9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd75406afa0 con 0x7fd754153d70 2023-10-30T12:38:49.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd75b2c9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd75406d780 con 0x7fd7540c8490 2023-10-30T12:38:49.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd75b2c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd754154150 con 0x7fd75406e470 2023-10-30T12:38:49.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd759866640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd7540c8490 0x7fd7540c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:49.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd759866640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd7540c8490 0x7fd7540c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41320/0 (socket says 172.21.15.125:41320) 2023-10-30T12:38:49.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd759866640 1 -- 172.21.15.125:0/1339559785 learned_addr learned my addr 172.21.15.125:0/1339559785 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:49.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd759065640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd754153d70 0x7fd7540cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40114/0 (socket says 172.21.15.125:40114) 2023-10-30T12:38:49.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd758864640 1 --2- 172.21.15.125:0/1339559785 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd75406e470 0x7fd75406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:49.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd759866640 1 -- 172.21.15.125:0/1339559785 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd75406e470 msgr2=0x7fd75406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:49.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd759866640 1 --2- 172.21.15.125:0/1339559785 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd75406e470 0x7fd75406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:49.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd759866640 1 -- 172.21.15.125:0/1339559785 >> v1:172.21.15.125:6789/0 conn(0x7fd754153d70 legacy=0x7fd7540cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:49.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.971+0000 7fd759866640 1 -- 172.21.15.125:0/1339559785 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75406e030 con 0x7fd7540c8490 2023-10-30T12:38:49.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.975+0000 7fd759866640 1 --2- 172.21.15.125:0/1339559785 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7540c8490 0x7fd7540c8cc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009e50 tx=0x7fd74c01c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f54a4cc1ecd0ce0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:49.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.975+0000 7fd74bfff640 1 -- 172.21.15.125:0/1339559785 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0a58a0 con 0x7fd7540c8490 2023-10-30T12:38:49.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.975+0000 7fd74bfff640 1 -- 172.21.15.125:0/1339559785 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd74c0a5a00 con 0x7fd7540c8490 2023-10-30T12:38:49.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.975+0000 7fd74bfff640 1 -- 172.21.15.125:0/1339559785 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74c0a5d90 con 0x7fd7540c8490 2023-10-30T12:38:49.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.975+0000 7fd75b2c9640 1 -- 172.21.15.125:0/1339559785 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7540c8490 msgr2=0x7fd7540c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:49.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.975+0000 7fd75b2c9640 1 --2- 172.21.15.125:0/1339559785 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7540c8490 0x7fd7540c8cc0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fd74c009e50 tx=0x7fd74c01c9e0 comp rx=0 tx=0).stop 2023-10-30T12:38:49.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.975+0000 7fd75b2c9640 1 -- 172.21.15.125:0/1339559785 shutdown_connections 2023-10-30T12:38:49.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.975+0000 7fd75b2c9640 1 --2- 172.21.15.125:0/1339559785 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7540c8490 0x7fd7540c8cc0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:49.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.975+0000 7fd75b2c9640 1 --2- 172.21.15.125:0/1339559785 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd75406e470 0x7fd75406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:49.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.975+0000 7fd75b2c9640 1 -- 172.21.15.125:0/1339559785 >> 172.21.15.125:0/1339559785 conn(0x7fd754071ea0 msgr2=0x7fd754065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:49.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.975+0000 7fd75b2c9640 1 -- 172.21.15.125:0/1339559785 shutdown_connections 2023-10-30T12:38:49.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.975+0000 7fd75b2c9640 1 -- 172.21.15.125:0/1339559785 wait complete. 2023-10-30T12:38:49.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd75b2c9640 1 Processor -- start 2023-10-30T12:38:49.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd75b2c9640 1 -- start start 2023-10-30T12:38:49.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd75b2c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd75406e470 0x7fd7540caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:49.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd75b2c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7540c8490 0x7fd7540b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:49.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd75b2c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd754153d70 0x7fd7540b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:49.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd75b2c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7540ae9a0 con 0x7fd75406e470 2023-10-30T12:38:49.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd75b2c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7540ae5f0 con 0x7fd754153d70 2023-10-30T12:38:49.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd75b2c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7540c8020 con 0x7fd7540c8490 2023-10-30T12:38:49.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd759065640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd75406e470 0x7fd7540caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:49.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd759065640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd75406e470 0x7fd7540caed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41322/0 (socket says 172.21.15.125:41322) 2023-10-30T12:38:49.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd759065640 1 -- 172.21.15.125:0/882191668 learned_addr learned my addr 172.21.15.125:0/882191668 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:49.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd758864640 1 --2- 172.21.15.125:0/882191668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7540c8490 0x7fd7540b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:49.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd759866640 1 --2- 172.21.15.125:0/882191668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd754153d70 0x7fd7540b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:49.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd759065640 1 -- 172.21.15.125:0/882191668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7540c8490 msgr2=0x7fd7540b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:49.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd759065640 1 --2- 172.21.15.125:0/882191668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7540c8490 0x7fd7540b0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:49.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd759065640 1 -- 172.21.15.125:0/882191668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd754153d70 msgr2=0x7fd7540b14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:49.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd759065640 1 --2- 172.21.15.125:0/882191668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd754153d70 0x7fd7540b14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:49.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd759065640 1 -- 172.21.15.125:0/882191668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c009b30 con 0x7fd75406e470 2023-10-30T12:38:49.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd759065640 1 --2- 172.21.15.125:0/882191668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd75406e470 0x7fd7540caed0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd75000c490 tx=0x7fd75000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:49.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd749ffb640 1 -- 172.21.15.125:0/882191668 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd750007480 con 0x7fd75406e470 2023-10-30T12:38:49.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd749ffb640 1 -- 172.21.15.125:0/882191668 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7500075e0 con 0x7fd75406e470 2023-10-30T12:38:49.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd749ffb640 1 -- 172.21.15.125:0/882191668 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd750004a10 con 0x7fd75406e470 2023-10-30T12:38:49.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd75b2c9640 1 -- 172.21.15.125:0/882191668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7540ada70 con 0x7fd75406e470 2023-10-30T12:38:49.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.979+0000 7fd75b2c9640 1 -- 172.21.15.125:0/882191668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7540ae030 con 0x7fd75406e470 2023-10-30T12:38:49.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.983+0000 7fd75b2c9640 1 -- 172.21.15.125:0/882191668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd75404e170 con 0x7fd75406e470 2023-10-30T12:38:49.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.983+0000 7fd749ffb640 1 -- 172.21.15.125:0/882191668 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd750004b70 con 0x7fd75406e470 2023-10-30T12:38:49.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.983+0000 7fd749ffb640 1 --2- 172.21.15.125:0/882191668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7380664f0 0x7fd7380689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:49.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.983+0000 7fd749ffb640 1 -- 172.21.15.125:0/882191668 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fd750083540 con 0x7fd75406e470 2023-10-30T12:38:49.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.983+0000 7fd758864640 1 --2- 172.21.15.125:0/882191668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7380664f0 0x7fd7380689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:49.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.983+0000 7fd758864640 1 --2- 172.21.15.125:0/882191668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7380664f0 0x7fd7380689b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd754153510 tx=0x7fd744008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:49.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:49.987+0000 7fd749ffb640 1 -- 172.21.15.125:0/882191668 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd750050890 con 0x7fd75406e470 2023-10-30T12:38:50.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.171+0000 7fd75b2c9640 1 -- 172.21.15.125:0/882191668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd75406c2a0 con 0x7fd75406e470 2023-10-30T12:38:50.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.171+0000 7fd749ffb640 1 -- 172.21.15.125:0/882191668 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fd750053eb0 con 0x7fd75406e470 2023-10-30T12:38:50.173 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:50.173 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:31.250027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:50.175 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 59 2023-10-30T12:38:50.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.171+0000 7fd75b2c9640 1 -- 172.21.15.125:0/882191668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7380664f0 msgr2=0x7fd7380689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:50.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.171+0000 7fd75b2c9640 1 --2- 172.21.15.125:0/882191668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7380664f0 0x7fd7380689b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd754153510 tx=0x7fd744008040 comp rx=0 tx=0).stop 2023-10-30T12:38:50.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.171+0000 7fd75b2c9640 1 -- 172.21.15.125:0/882191668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd75406e470 msgr2=0x7fd7540caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:50.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.171+0000 7fd75b2c9640 1 --2- 172.21.15.125:0/882191668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd75406e470 0x7fd7540caed0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd75000c490 tx=0x7fd75000c950 comp rx=0 tx=0).stop 2023-10-30T12:38:50.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.175+0000 7fd75b2c9640 1 -- 172.21.15.125:0/882191668 shutdown_connections 2023-10-30T12:38:50.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.175+0000 7fd75b2c9640 1 --2- 172.21.15.125:0/882191668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7380664f0 0x7fd7380689b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:50.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.175+0000 7fd75b2c9640 1 --2- 172.21.15.125:0/882191668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd754153d70 0x7fd7540b14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:50.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.175+0000 7fd75b2c9640 1 --2- 172.21.15.125:0/882191668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7540c8490 0x7fd7540b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:50.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.175+0000 7fd75b2c9640 1 --2- 172.21.15.125:0/882191668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd75406e470 0x7fd7540caed0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:50.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.175+0000 7fd75b2c9640 1 -- 172.21.15.125:0/882191668 >> 172.21.15.125:0/882191668 conn(0x7fd754071ea0 msgr2=0x7fd75414fd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:50.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.175+0000 7fd75b2c9640 1 -- 172.21.15.125:0/882191668 shutdown_connections 2023-10-30T12:38:50.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.175+0000 7fd75b2c9640 1 -- 172.21.15.125:0/882191668 wait complete. 2023-10-30T12:38:50.192 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T12:38:50.192 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:38:50.193 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:50.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.375+0000 7fd1c6c91640 1 Processor -- start 2023-10-30T12:38:50.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.375+0000 7fd1c6c91640 1 -- start start 2023-10-30T12:38:50.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.375+0000 7fd1c6c91640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd1c00c6b80 0x7fd1c00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:50.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.375+0000 7fd1c6c91640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd1c013f4a0 0x7fd1c00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:50.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.375+0000 7fd1c6c91640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1c00c0a50 con 0x7fd1c00c6f60 2023-10-30T12:38:50.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.375+0000 7fd1c6c91640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1c006f6e0 con 0x7fd1c013f4a0 2023-10-30T12:38:50.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.375+0000 7fd1c6c91640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c006f9c0 con 0x7fd1c00c6b80 2023-10-30T12:38:50.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.379+0000 7fd1bffff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd1c00c6f60 0x7fd1c013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40120/0 (socket says 172.21.15.125:40120) 2023-10-30T12:38:50.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.379+0000 7fd1bffff640 1 -- 172.21.15.125:0/3852023207 learned_addr learned my addr 172.21.15.125:0/3852023207 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:38:50.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.379+0000 7fd1c522e640 1 --2- 172.21.15.125:0/3852023207 >> v2:172.21.15.125:3300/0 conn(0x7fd1c013f4a0 0x7fd1c00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:50.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.379+0000 7fd1c4a2d640 1 --2- 172.21.15.125:0/3852023207 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd1c00c6b80 0x7fd1c00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:50.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.379+0000 7fd1bf7fe640 1 -- 172.21.15.125:0/3852023207 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042724481 0 0) 0x7fd1c00c0a50 con 0x7fd1c00c6f60 2023-10-30T12:38:50.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.379+0000 7fd1bf7fe640 1 -- 172.21.15.125:0/3852023207 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b0003530 con 0x7fd1c00c6f60 2023-10-30T12:38:50.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.379+0000 7fd1c522e640 1 -- 172.21.15.125:0/3852023207 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd1c00c6b80 msgr2=0x7fd1c00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:50.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.379+0000 7fd1c522e640 1 --2- 172.21.15.125:0/3852023207 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd1c00c6b80 0x7fd1c00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:50.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.379+0000 7fd1c522e640 1 -- 172.21.15.125:0/3852023207 >> v1:172.21.15.125:6789/0 conn(0x7fd1c00c6f60 legacy=0x7fd1c013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:50.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.379+0000 7fd1c522e640 1 -- 172.21.15.125:0/3852023207 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1c006fca0 con 0x7fd1c013f4a0 2023-10-30T12:38:50.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.379+0000 7fd1c522e640 1 --2- 172.21.15.125:0/3852023207 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1c013f4a0 0x7fd1c00c0400 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009d90 tx=0x7fd1b801c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=971762fda46cfdf3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:50.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.379+0000 7fd1bf7fe640 1 -- 172.21.15.125:0/3852023207 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b80a5920 con 0x7fd1c013f4a0 2023-10-30T12:38:50.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.379+0000 7fd1bf7fe640 1 -- 172.21.15.125:0/3852023207 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b80a5a80 con 0x7fd1c013f4a0 2023-10-30T12:38:50.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.379+0000 7fd1bf7fe640 1 -- 172.21.15.125:0/3852023207 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b80a5e10 con 0x7fd1c013f4a0 2023-10-30T12:38:50.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 -- 172.21.15.125:0/3852023207 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1c013f4a0 msgr2=0x7fd1c00c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:50.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 --2- 172.21.15.125:0/3852023207 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1c013f4a0 0x7fd1c00c0400 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd1b8009d90 tx=0x7fd1b801c950 comp rx=0 tx=0).stop 2023-10-30T12:38:50.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 -- 172.21.15.125:0/3852023207 shutdown_connections 2023-10-30T12:38:50.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 --2- 172.21.15.125:0/3852023207 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1c013f4a0 0x7fd1c00c0400 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:50.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 --2- 172.21.15.125:0/3852023207 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd1c00c6b80 0x7fd1c00bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:50.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 -- 172.21.15.125:0/3852023207 >> 172.21.15.125:0/3852023207 conn(0x7fd1c00c7910 msgr2=0x7fd1c00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:50.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 -- 172.21.15.125:0/3852023207 shutdown_connections 2023-10-30T12:38:50.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 -- 172.21.15.125:0/3852023207 wait complete. 2023-10-30T12:38:50.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 Processor -- start 2023-10-30T12:38:50.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 -- start start 2023-10-30T12:38:50.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1c00c6b80 0x7fd1c0159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:50.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1c00c6f60 0x7fd1c01595d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:50.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd1c013f4a0 0x7fd1c0153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:50.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c0157fc0 con 0x7fd1c00c6b80 2023-10-30T12:38:50.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c01395d0 con 0x7fd1c00c6f60 2023-10-30T12:38:50.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c6c91640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1c01f2a40 con 0x7fd1c013f4a0 2023-10-30T12:38:50.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c4a2d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1c00c6b80 0x7fd1c0159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:50.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c4a2d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1c00c6b80 0x7fd1c0159090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41342/0 (socket says 172.21.15.125:41342) 2023-10-30T12:38:50.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c4a2d640 1 -- 172.21.15.125:0/249798793 learned_addr learned my addr 172.21.15.125:0/249798793 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:50.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c522e640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd1c013f4a0 0x7fd1c0153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:50.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1bffff640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1c00c6f60 0x7fd1c01595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:50.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c4a2d640 1 -- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd1c013f4a0 msgr2=0x7fd1c0153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:50.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.383+0000 7fd1c4a2d640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd1c013f4a0 0x7fd1c0153310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:50.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1c4a2d640 1 -- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1c00c6f60 msgr2=0x7fd1c01595d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:50.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1c4a2d640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1c00c6f60 0x7fd1c01595d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:50.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1c4a2d640 1 -- 172.21.15.125:0/249798793 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b8009a70 con 0x7fd1c00c6b80 2023-10-30T12:38:50.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1bffff640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1c00c6f60 0x7fd1c01595d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:38:50.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1c4a2d640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1c00c6b80 0x7fd1c0159090 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400e960 tx=0x7fd1b400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:50.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1bd7fa640 1 -- 172.21.15.125:0/249798793 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b4009690 con 0x7fd1c00c6b80 2023-10-30T12:38:50.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1bd7fa640 1 -- 172.21.15.125:0/249798793 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1b400ca00 con 0x7fd1c00c6b80 2023-10-30T12:38:50.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1c6c91640 1 -- 172.21.15.125:0/249798793 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1c01f1e30 con 0x7fd1c00c6b80 2023-10-30T12:38:50.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1bd7fa640 1 -- 172.21.15.125:0/249798793 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1b400cd70 con 0x7fd1c00c6b80 2023-10-30T12:38:50.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1c6c91640 1 -- 172.21.15.125:0/249798793 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1c01f23f0 con 0x7fd1c00c6b80 2023-10-30T12:38:50.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1c6c91640 1 -- 172.21.15.125:0/249798793 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c004e170 con 0x7fd1c00c6b80 2023-10-30T12:38:50.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1bd7fa640 1 -- 172.21.15.125:0/249798793 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd1b4013470 con 0x7fd1c00c6b80 2023-10-30T12:38:50.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1bd7fa640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd1a40664a0 0x7fd1a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:50.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1bd7fa640 1 -- 172.21.15.125:0/249798793 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fd1b4084350 con 0x7fd1c00c6b80 2023-10-30T12:38:50.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.387+0000 7fd1bffff640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd1a40664a0 0x7fd1a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:50.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.391+0000 7fd1bffff640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd1a40664a0 0x7fd1a4068960 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0137530 tx=0x7fd1ac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:50.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.391+0000 7fd1bd7fa640 1 -- 172.21.15.125:0/249798793 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd1b4051590 con 0x7fd1c00c6b80 2023-10-30T12:38:50.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.575+0000 7fd1c6c91640 1 -- 172.21.15.125:0/249798793 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1c006a980 con 0x7fd1c00c6b80 2023-10-30T12:38:50.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.575+0000 7fd1bd7fa640 1 -- 172.21.15.125:0/249798793 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fd1b4054bb0 con 0x7fd1c00c6b80 2023-10-30T12:38:50.576 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:50.576 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:31.250027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:50.577 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 59 2023-10-30T12:38:50.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.575+0000 7fd1c6c91640 1 -- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd1a40664a0 msgr2=0x7fd1a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:50.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.575+0000 7fd1c6c91640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd1a40664a0 0x7fd1a4068960 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd1c0137530 tx=0x7fd1ac009040 comp rx=0 tx=0).stop 2023-10-30T12:38:50.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.575+0000 7fd1c6c91640 1 -- 172.21.15.125:0/249798793 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1c00c6b80 msgr2=0x7fd1c0159090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:50.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.575+0000 7fd1c6c91640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1c00c6b80 0x7fd1c0159090 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd1b400e960 tx=0x7fd1b400ee20 comp rx=0 tx=0).stop 2023-10-30T12:38:50.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.575+0000 7fd1c6c91640 1 -- 172.21.15.125:0/249798793 shutdown_connections 2023-10-30T12:38:50.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.575+0000 7fd1c6c91640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd1a40664a0 0x7fd1a4068960 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:50.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.575+0000 7fd1c6c91640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd1c013f4a0 0x7fd1c0153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:50.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.575+0000 7fd1c6c91640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd1c00c6f60 0x7fd1c01595d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:50.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.575+0000 7fd1c6c91640 1 --2- 172.21.15.125:0/249798793 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd1c00c6b80 0x7fd1c0159090 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:50.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.575+0000 7fd1c6c91640 1 -- 172.21.15.125:0/249798793 >> 172.21.15.125:0/249798793 conn(0x7fd1c00c7910 msgr2=0x7fd1c006bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:50.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.575+0000 7fd1c6c91640 1 -- 172.21.15.125:0/249798793 shutdown_connections 2023-10-30T12:38:50.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:50.575+0000 7fd1c6c91640 1 -- 172.21.15.125:0/249798793 wait complete. 2023-10-30T12:38:50.596 INFO:tasks.mds_thrash.fs.[cephfs]:cluster has 2 actives (max_mds is 2), no MDS can replace rank 2 2023-10-30T12:38:50.596 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.d 2023-10-30T12:38:54.597 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T12:38:54.597 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T12:38:54.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T12:38:54.600 INFO:tasks.ceph.mds.d:Started 2023-10-30T12:38:54.601 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:54.677 INFO:tasks.ceph.mds.d.smithi136.stdout:starting mds.d at 2023-10-30T12:38:54.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f920cebb640 1 Processor -- start 2023-10-30T12:38:54.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f920cebb640 1 -- start start 2023-10-30T12:38:54.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f920cebb640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f92081538f0 0x7f9208159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:54.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f920cebb640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f92080cefa0 0x7f92080cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:54.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f920cebb640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92080ca6a0 con 0x7f920815cd50 2023-10-30T12:38:54.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f920cebb640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92080cf8f0 con 0x7f92080cefa0 2023-10-30T12:38:54.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f920cebb640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f920815d300 con 0x7f92081538f0 2023-10-30T12:38:54.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f9207fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f92080cefa0 0x7f92080cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:54.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f9206ffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f920815cd50 0x7f92080c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41580/0 (socket says 172.21.15.125:41580) 2023-10-30T12:38:54.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f9206ffd640 1 -- 172.21.15.125:0/1120452927 learned_addr learned my addr 172.21.15.125:0/1120452927 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:38:54.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f9207fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f92080cefa0 0x7f92080cf3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39674/0 (socket says 172.21.15.125:39674) 2023-10-30T12:38:54.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f92077fe640 1 --2- 172.21.15.125:0/1120452927 >> v2:172.21.15.136:3300/0 conn(0x7f92081538f0 0x7f9208159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:54.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f9207fff640 1 -- 172.21.15.125:0/1120452927 >> v2:172.21.15.136:3300/0 conn(0x7f92081538f0 msgr2=0x7f9208159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:54.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f9207fff640 1 --2- 172.21.15.125:0/1120452927 >> v2:172.21.15.136:3300/0 conn(0x7f92081538f0 0x7f9208159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:54.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f9207fff640 1 -- 172.21.15.125:0/1120452927 >> v1:172.21.15.125:6789/0 conn(0x7f920815cd50 legacy=0x7f92080c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:54.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f9207fff640 1 -- 172.21.15.125:0/1120452927 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920815d5e0 con 0x7f92080cefa0 2023-10-30T12:38:54.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.751+0000 7f92067fc640 1 -- 172.21.15.125:0/1120452927 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086995657 0 0) 0x7f92080ca6a0 con 0x7f920815cd50 2023-10-30T12:38:54.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.755+0000 7f9207fff640 1 --2- 172.21.15.125:0/1120452927 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92080cefa0 0x7f92080cf3b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f92080c9cb0 tx=0x7f920001f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77bb4247ef0dc3df server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:54.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.755+0000 7f92067fc640 1 -- 172.21.15.125:0/1120452927 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92000b0440 con 0x7f92080cefa0 2023-10-30T12:38:54.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.755+0000 7f92067fc640 1 -- 172.21.15.125:0/1120452927 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92000b05a0 con 0x7f92080cefa0 2023-10-30T12:38:54.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.755+0000 7f92067fc640 1 -- 172.21.15.125:0/1120452927 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92000b0930 con 0x7f92080cefa0 2023-10-30T12:38:54.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.755+0000 7f920cebb640 1 -- 172.21.15.125:0/1120452927 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92080cefa0 msgr2=0x7f92080cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:54.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.755+0000 7f920cebb640 1 --2- 172.21.15.125:0/1120452927 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92080cefa0 0x7f92080cf3b0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f92080c9cb0 tx=0x7f920001f0f0 comp rx=0 tx=0).stop 2023-10-30T12:38:54.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.755+0000 7f920cebb640 1 -- 172.21.15.125:0/1120452927 shutdown_connections 2023-10-30T12:38:54.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.755+0000 7f920cebb640 1 --2- 172.21.15.125:0/1120452927 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92080cefa0 0x7f92080cf3b0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:54.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.755+0000 7f920cebb640 1 --2- 172.21.15.125:0/1120452927 >> v2:172.21.15.136:3300/0 conn(0x7f92081538f0 0x7f9208159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:54.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.755+0000 7f920cebb640 1 -- 172.21.15.125:0/1120452927 >> 172.21.15.125:0/1120452927 conn(0x7f920806a640 msgr2=0x7f9208065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:54.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.755+0000 7f920cebb640 1 -- 172.21.15.125:0/1120452927 shutdown_connections 2023-10-30T12:38:54.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.755+0000 7f920cebb640 1 -- 172.21.15.125:0/1120452927 wait complete. 2023-10-30T12:38:54.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.755+0000 7f920cebb640 1 Processor -- start 2023-10-30T12:38:54.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f920cebb640 1 -- start start 2023-10-30T12:38:54.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f920cebb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92080cefa0 0x7f9208147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:54.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f920cebb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92081538f0 0x7f92081efba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:54.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f920cebb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f920815cd50 0x7f920813fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:54.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f920cebb640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92081456a0 con 0x7f920815cd50 2023-10-30T12:38:54.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f920cebb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9208145370 con 0x7f92081538f0 2023-10-30T12:38:54.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f920cebb640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9208144b20 con 0x7f92080cefa0 2023-10-30T12:38:54.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f9207fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f920815cd50 0x7f920813fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:54.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f9207fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f920815cd50 0x7f920813fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39688/0 (socket says 172.21.15.125:39688) 2023-10-30T12:38:54.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f9207fff640 1 -- 172.21.15.125:0/2859865255 learned_addr learned my addr 172.21.15.125:0/2859865255 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:54.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f92077fe640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92080cefa0 0x7f9208147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:54.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f9206ffd640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92081538f0 0x7f92081efba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:54.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f9207fff640 1 -- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92080cefa0 msgr2=0x7f9208147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:54.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f9207fff640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92080cefa0 0x7f9208147a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:54.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f9207fff640 1 -- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92081538f0 msgr2=0x7f92081efba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:54.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f9207fff640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92081538f0 0x7f92081efba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:54.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f9207fff640 1 -- 172.21.15.125:0/2859865255 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9200009a70 con 0x7f920815cd50 2023-10-30T12:38:54.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f9206ffd640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92081538f0 0x7f92081efba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:54.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f92077fe640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92080cefa0 0x7f9208147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:38:54.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f9207fff640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f920815cd50 0x7f920813fae0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f920001bd00 tx=0x7f920001be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:54.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f91effff640 1 -- 172.21.15.125:0/2859865255 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92000b0440 con 0x7f920815cd50 2023-10-30T12:38:54.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f91effff640 1 -- 172.21.15.125:0/2859865255 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92000bc860 con 0x7f920815cd50 2023-10-30T12:38:54.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f920cebb640 1 -- 172.21.15.125:0/2859865255 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92080b4d90 con 0x7f920815cd50 2023-10-30T12:38:54.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f91effff640 1 -- 172.21.15.125:0/2859865255 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92000bcc10 con 0x7f920815cd50 2023-10-30T12:38:54.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f920cebb640 1 -- 172.21.15.125:0/2859865255 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92080b4a90 con 0x7f920815cd50 2023-10-30T12:38:54.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.759+0000 7f920cebb640 1 -- 172.21.15.125:0/2859865255 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92081454e0 con 0x7f920815cd50 2023-10-30T12:38:54.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.763+0000 7f91effff640 1 -- 172.21.15.125:0/2859865255 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f92000c7460 con 0x7f920815cd50 2023-10-30T12:38:54.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.763+0000 7f91effff640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91f40664a0 0x7f91f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:54.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.763+0000 7f91effff640 1 -- 172.21.15.125:0/2859865255 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f9200122ca0 con 0x7f920815cd50 2023-10-30T12:38:54.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.763+0000 7f92077fe640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91f40664a0 0x7f91f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:54.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.763+0000 7f92077fe640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91f40664a0 0x7f91f4068960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009bb0 tx=0x7f91f8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:54.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.767+0000 7f91effff640 1 -- 172.21.15.125:0/2859865255 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f92081454e0 con 0x7f920815cd50 2023-10-30T12:38:54.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.959+0000 7f920cebb640 1 -- 172.21.15.125:0/2859865255 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92081454e0 con 0x7f920815cd50 2023-10-30T12:38:54.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.959+0000 7f91effff640 1 -- 172.21.15.125:0/2859865255 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f92081454e0 con 0x7f920815cd50 2023-10-30T12:38:54.961 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:54.961 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":59,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:31.250027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:54.962 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 59 2023-10-30T12:38:54.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.963+0000 7f920cebb640 1 -- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91f40664a0 msgr2=0x7f91f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:54.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.963+0000 7f920cebb640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91f40664a0 0x7f91f4068960 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f91f8009bb0 tx=0x7f91f8008040 comp rx=0 tx=0).stop 2023-10-30T12:38:54.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.963+0000 7f920cebb640 1 -- 172.21.15.125:0/2859865255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f920815cd50 msgr2=0x7f920813fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:54.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.963+0000 7f920cebb640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f920815cd50 0x7f920813fae0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f920001bd00 tx=0x7f920001be90 comp rx=0 tx=0).stop 2023-10-30T12:38:54.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.963+0000 7f920cebb640 1 -- 172.21.15.125:0/2859865255 shutdown_connections 2023-10-30T12:38:54.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.963+0000 7f920cebb640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91f40664a0 0x7f91f4068960 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:54.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.963+0000 7f920cebb640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f920815cd50 0x7f920813fae0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:54.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.963+0000 7f920cebb640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92081538f0 0x7f92081efba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:54.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.963+0000 7f920cebb640 1 --2- 172.21.15.125:0/2859865255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92080cefa0 0x7f9208147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:54.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.963+0000 7f920cebb640 1 -- 172.21.15.125:0/2859865255 >> 172.21.15.125:0/2859865255 conn(0x7f920806a640 msgr2=0x7f92080c3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:54.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.963+0000 7f920cebb640 1 -- 172.21.15.125:0/2859865255 shutdown_connections 2023-10-30T12:38:54.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:54.963+0000 7f920cebb640 1 -- 172.21.15.125:0/2859865255 wait complete. 2023-10-30T12:38:54.982 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T12:38:56.983 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:38:57.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.127+0000 7f9fdf646640 1 Processor -- start 2023-10-30T12:38:57.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.131+0000 7f9fdf646640 1 -- start start 2023-10-30T12:38:57.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.131+0000 7f9fdf646640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9fd806f7a0 0x7f9fd806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:57.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.131+0000 7f9fdf646640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fd80c6bd0 0x7f9fd8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:57.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.131+0000 7f9fdf646640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd806a3d0 con 0x7f9fd8071110 2023-10-30T12:38:57.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.131+0000 7f9fdf646640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fd80700c0 con 0x7f9fd806f7a0 2023-10-30T12:38:57.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.131+0000 7f9fdf646640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd8066cf0 con 0x7f9fd80c6bd0 2023-10-30T12:38:57.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.131+0000 7f9fdd3e2640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9fd806f7a0 0x7f9fd806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:57.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.131+0000 7f9fdd3e2640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9fd806f7a0 0x7f9fd806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39700/0 (socket says 172.21.15.125:39700) 2023-10-30T12:38:57.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.131+0000 7f9fdd3e2640 1 -- 172.21.15.125:0/2381547512 learned_addr learned my addr 172.21.15.125:0/2381547512 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:57.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.131+0000 7f9fdcbe1640 1 --2- 172.21.15.125:0/2381547512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fd80c6bd0 0x7f9fd8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:57.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.131+0000 7f9fdd3e2640 1 -- 172.21.15.125:0/2381547512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fd80c6bd0 msgr2=0x7f9fd8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:57.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.131+0000 7f9fdd3e2640 1 --2- 172.21.15.125:0/2381547512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fd80c6bd0 0x7f9fd8069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:57.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.131+0000 7f9fdd3e2640 1 -- 172.21.15.125:0/2381547512 >> v1:172.21.15.136:6789/0 conn(0x7f9fd8071110 legacy=0x7f9fd8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:38:57.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.131+0000 7f9fdd3e2640 1 -- 172.21.15.125:0/2381547512 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd80c70d0 con 0x7f9fd806f7a0 2023-10-30T12:38:57.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.135+0000 7f9fdd3e2640 1 --2- 172.21.15.125:0/2381547512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fd806f7a0 0x7f9fd806fb80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009870 tx=0x7f9fd401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192157a7537e6859 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:57.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.135+0000 7f9fcffff640 1 -- 172.21.15.125:0/2381547512 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40a89b0 con 0x7f9fd806f7a0 2023-10-30T12:38:57.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.135+0000 7f9fcffff640 1 -- 172.21.15.125:0/2381547512 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd40a8b10 con 0x7f9fd806f7a0 2023-10-30T12:38:57.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.135+0000 7f9fcffff640 1 -- 172.21.15.125:0/2381547512 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40a8ea0 con 0x7f9fd806f7a0 2023-10-30T12:38:57.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.135+0000 7f9fdf646640 1 -- 172.21.15.125:0/2381547512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fd806f7a0 msgr2=0x7f9fd806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:57.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.135+0000 7f9fdf646640 1 --2- 172.21.15.125:0/2381547512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fd806f7a0 0x7f9fd806fb80 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009870 tx=0x7f9fd401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:38:57.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.135+0000 7f9fdf646640 1 -- 172.21.15.125:0/2381547512 shutdown_connections 2023-10-30T12:38:57.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.135+0000 7f9fdf646640 1 --2- 172.21.15.125:0/2381547512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fd80c6bd0 0x7f9fd8069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:57.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.135+0000 7f9fdf646640 1 --2- 172.21.15.125:0/2381547512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fd806f7a0 0x7f9fd806fb80 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:57.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.135+0000 7f9fdf646640 1 -- 172.21.15.125:0/2381547512 >> 172.21.15.125:0/2381547512 conn(0x7f9fd80c7910 msgr2=0x7f9fd80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:57.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.135+0000 7f9fdf646640 1 -- 172.21.15.125:0/2381547512 shutdown_connections 2023-10-30T12:38:57.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.135+0000 7f9fdf646640 1 -- 172.21.15.125:0/2381547512 wait complete. 2023-10-30T12:38:57.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.135+0000 7f9fdf646640 1 Processor -- start 2023-10-30T12:38:57.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.135+0000 7f9fdf646640 1 -- start start 2023-10-30T12:38:57.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdf646640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fd806f7a0 0x7f9fd80c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:57.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdf646640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fd8071110 0x7f9fd814d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:57.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdf646640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fd80c6bd0 0x7f9fd814d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:57.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdf646640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd814aa00 con 0x7f9fd806f7a0 2023-10-30T12:38:57.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdf646640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd814b860 con 0x7f9fd80c6bd0 2023-10-30T12:38:57.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdf646640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd81588a0 con 0x7f9fd8071110 2023-10-30T12:38:57.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdd3e2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fd806f7a0 0x7f9fd80c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:57.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdd3e2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fd806f7a0 0x7f9fd80c5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39716/0 (socket says 172.21.15.125:39716) 2023-10-30T12:38:57.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdd3e2640 1 -- 172.21.15.125:0/3422891001 learned_addr learned my addr 172.21.15.125:0/3422891001 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:38:57.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdcbe1640 1 --2- 172.21.15.125:0/3422891001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fd8071110 0x7f9fd814d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:57.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fddbe3640 1 --2- 172.21.15.125:0/3422891001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fd80c6bd0 0x7f9fd814d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:57.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdd3e2640 1 -- 172.21.15.125:0/3422891001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fd8071110 msgr2=0x7f9fd814d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:57.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdd3e2640 1 --2- 172.21.15.125:0/3422891001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fd8071110 0x7f9fd814d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:57.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdd3e2640 1 -- 172.21.15.125:0/3422891001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fd80c6bd0 msgr2=0x7f9fd814d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:57.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdd3e2640 1 --2- 172.21.15.125:0/3422891001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fd80c6bd0 0x7f9fd814d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:57.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdd3e2640 1 -- 172.21.15.125:0/3422891001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd40094d0 con 0x7f9fd806f7a0 2023-10-30T12:38:57.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdd3e2640 1 --2- 172.21.15.125:0/3422891001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fd806f7a0 0x7f9fd80c5c70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9fd401d3a0 tx=0x7f9fd401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:57.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fcdffb640 1 -- 172.21.15.125:0/3422891001 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40ba600 con 0x7f9fd806f7a0 2023-10-30T12:38:57.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fcdffb640 1 -- 172.21.15.125:0/3422891001 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd40ba760 con 0x7f9fd806f7a0 2023-10-30T12:38:57.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fcdffb640 1 -- 172.21.15.125:0/3422891001 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd40bab10 con 0x7f9fd806f7a0 2023-10-30T12:38:57.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdf646640 1 -- 172.21.15.125:0/3422891001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd814ac50 con 0x7f9fd806f7a0 2023-10-30T12:38:57.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdf646640 1 -- 172.21.15.125:0/3422891001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd814b210 con 0x7f9fd806f7a0 2023-10-30T12:38:57.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.139+0000 7f9fdf646640 1 -- 172.21.15.125:0/3422891001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fd804e170 con 0x7f9fd806f7a0 2023-10-30T12:38:57.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.143+0000 7f9fcdffb640 1 -- 172.21.15.125:0/3422891001 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9fd40bac70 con 0x7f9fd806f7a0 2023-10-30T12:38:57.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.143+0000 7f9fcdffb640 1 --2- 172.21.15.125:0/3422891001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fbc0664a0 0x7f9fbc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:38:57.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.143+0000 7f9fcdffb640 1 -- 172.21.15.125:0/3422891001 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f9fd41225b0 con 0x7f9fd806f7a0 2023-10-30T12:38:57.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.143+0000 7f9fdcbe1640 1 --2- 172.21.15.125:0/3422891001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fbc0664a0 0x7f9fbc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:38:57.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.143+0000 7f9fdcbe1640 1 --2- 172.21.15.125:0/3422891001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fbc0664a0 0x7f9fbc068960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8158fd0 tx=0x7f9fc8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:38:57.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.147+0000 7f9fcdffb640 1 -- 172.21.15.125:0/3422891001 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9fd40ef800 con 0x7f9fd806f7a0 2023-10-30T12:38:57.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.563+0000 7f9fdf646640 1 -- 172.21.15.125:0/3422891001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fd804e170 con 0x7f9fd806f7a0 2023-10-30T12:38:57.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.567+0000 7f9fcdffb640 1 -- 172.21.15.125:0/3422891001 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f9fd40f2e20 con 0x7f9fd806f7a0 2023-10-30T12:38:57.567 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:38:57.567 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:38:31.250027+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:38:57.568 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 60 2023-10-30T12:38:57.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.567+0000 7f9fdf646640 1 -- 172.21.15.125:0/3422891001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fbc0664a0 msgr2=0x7f9fbc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:57.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.567+0000 7f9fdf646640 1 --2- 172.21.15.125:0/3422891001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fbc0664a0 0x7f9fbc068960 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8158fd0 tx=0x7f9fc8008040 comp rx=0 tx=0).stop 2023-10-30T12:38:57.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.567+0000 7f9fdf646640 1 -- 172.21.15.125:0/3422891001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fd806f7a0 msgr2=0x7f9fd80c5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:38:57.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.567+0000 7f9fdf646640 1 --2- 172.21.15.125:0/3422891001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fd806f7a0 0x7f9fd80c5c70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9fd401d3a0 tx=0x7f9fd401ba90 comp rx=0 tx=0).stop 2023-10-30T12:38:57.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.567+0000 7f9fdf646640 1 -- 172.21.15.125:0/3422891001 shutdown_connections 2023-10-30T12:38:57.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.567+0000 7f9fdf646640 1 --2- 172.21.15.125:0/3422891001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fbc0664a0 0x7f9fbc068960 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:57.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.567+0000 7f9fdf646640 1 --2- 172.21.15.125:0/3422891001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fd80c6bd0 0x7f9fd814d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:57.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.567+0000 7f9fdf646640 1 --2- 172.21.15.125:0/3422891001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fd8071110 0x7f9fd814d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:57.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.567+0000 7f9fdf646640 1 --2- 172.21.15.125:0/3422891001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fd806f7a0 0x7f9fd80c5c70 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:38:57.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.567+0000 7f9fdf646640 1 -- 172.21.15.125:0/3422891001 >> 172.21.15.125:0/3422891001 conn(0x7f9fd80c7910 msgr2=0x7f9fd80b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:38:57.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.567+0000 7f9fdf646640 1 -- 172.21.15.125:0/3422891001 shutdown_connections 2023-10-30T12:38:57.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:38:57.567+0000 7f9fdf646640 1 -- 172.21.15.125:0/3422891001 wait complete. 2023-10-30T12:38:57.590 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T12:38:57.591 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 96 secs before thrashing 2023-10-30T12:39:15.239 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:39:15.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:39:45.291 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:39:45.296 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:40:15.351 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:40:15.355 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:40:33.592 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:33.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.738+0000 7f57847d3640 1 Processor -- start 2023-10-30T12:40:33.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.738+0000 7f57847d3640 1 -- start start 2023-10-30T12:40:33.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.738+0000 7f57847d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f577c0c6f60 0x7f577c1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:33.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.738+0000 7f57847d3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f577c0c0a50 con 0x7f577c0c05d0 2023-10-30T12:40:33.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.738+0000 7f57847d3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f577c06f6e0 con 0x7f577c0c6b80 2023-10-30T12:40:33.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.738+0000 7f57847d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f577c06f9c0 con 0x7f577c0c6f60 2023-10-30T12:40:33.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.738+0000 7f5781d6e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f577c0c05d0 0x7f577c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55182/0 (socket says 172.21.15.125:55182) 2023-10-30T12:40:33.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.738+0000 7f5781d6e640 1 -- 172.21.15.125:0/1071040786 learned_addr learned my addr 172.21.15.125:0/1071040786 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:40:33.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.738+0000 7f5782d70640 1 --2- 172.21.15.125:0/1071040786 >> v2:172.21.15.136:3300/0 conn(0x7f577c0c6f60 0x7f577c1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:33.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.738+0000 7f578156d640 1 -- 172.21.15.125:0/1071040786 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737717376 0 0) 0x7f577c0c0a50 con 0x7f577c0c05d0 2023-10-30T12:40:33.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.738+0000 7f578156d640 1 -- 172.21.15.125:0/1071040786 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5768005740 con 0x7f577c0c05d0 2023-10-30T12:40:33.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f578156d640 1 -- 172.21.15.125:0/1071040786 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 903349452 0 0) 0x7f5768005740 con 0x7f577c0c05d0 2023-10-30T12:40:33.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f578156d640 1 -- 172.21.15.125:0/1071040786 >> v2:172.21.15.136:3300/0 conn(0x7f577c0c6f60 msgr2=0x7f577c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:33.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f578156d640 1 --2- 172.21.15.125:0/1071040786 >> v2:172.21.15.136:3300/0 conn(0x7f577c0c6f60 0x7f577c1566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:33.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f578156d640 1 -- 172.21.15.125:0/1071040786 >> v1:172.21.15.136:6789/0 conn(0x7f577c0c6b80 legacy=0x7f577c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:33.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f578156d640 1 -- 172.21.15.125:0/1071040786 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577c06fca0 con 0x7f577c0c05d0 2023-10-30T12:40:33.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f578156d640 1 -- 172.21.15.125:0/1071040786 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f576c001680 con 0x7f577c0c05d0 2023-10-30T12:40:33.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f578156d640 1 -- 172.21.15.125:0/1071040786 >> v1:172.21.15.125:6789/0 conn(0x7f577c0c05d0 legacy=0x7f577c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:33.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f578156d640 1 --2- 172.21.15.125:0/1071040786 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5768005b90 0x7f5768005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:33.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f578156d640 1 --2- 172.21.15.125:0/1071040786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57680069e0 0x7f5768006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:33.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f578156d640 1 --2- 172.21.15.125:0/1071040786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57680073a0 0x7f576800dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:33.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f578156d640 1 -- 172.21.15.125:0/1071040786 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c06f9c0 con 0x7f57680069e0 2023-10-30T12:40:33.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f578156d640 1 -- 172.21.15.125:0/1071040786 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f576800e240 con 0x7f57680073a0 2023-10-30T12:40:33.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f578156d640 1 -- 172.21.15.125:0/1071040786 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f576800e400 con 0x7f5768005b90 2023-10-30T12:40:33.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f5781d6e640 1 --2- 172.21.15.125:0/1071040786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57680069e0 0x7f5768006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:33.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f578256f640 1 --2- 172.21.15.125:0/1071040786 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5768005b90 0x7f5768005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:33.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f5782d70640 1 --2- 172.21.15.125:0/1071040786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57680073a0 0x7f576800dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:33.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f5781d6e640 1 -- 172.21.15.125:0/1071040786 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5768005b90 msgr2=0x7f5768005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:33.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f5781d6e640 1 --2- 172.21.15.125:0/1071040786 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5768005b90 0x7f5768005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:33.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f5781d6e640 1 -- 172.21.15.125:0/1071040786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57680073a0 msgr2=0x7f576800dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:33.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f5781d6e640 1 --2- 172.21.15.125:0/1071040786 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57680073a0 0x7f576800dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:33.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f5781d6e640 1 -- 172.21.15.125:0/1071040786 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5768005740 con 0x7f57680069e0 2023-10-30T12:40:33.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.742+0000 7f5782d70640 1 -- 172.21.15.125:0/1071040786 reap_dead start 2023-10-30T12:40:33.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f5781d6e640 1 --2- 172.21.15.125:0/1071040786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57680069e0 0x7f5768006e60 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c210 tx=0x7f576c01f2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd13a03232a8f53c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:33.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f578156d640 1 -- 172.21.15.125:0/1071040786 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c005ae0 con 0x7f57680069e0 2023-10-30T12:40:33.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f578156d640 1 -- 172.21.15.125:0/1071040786 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576c0ab070 con 0x7f57680069e0 2023-10-30T12:40:33.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f578156d640 1 -- 172.21.15.125:0/1071040786 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c007070 con 0x7f57680069e0 2023-10-30T12:40:33.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 -- 172.21.15.125:0/1071040786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57680069e0 msgr2=0x7f5768006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:33.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 --2- 172.21.15.125:0/1071040786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57680069e0 0x7f5768006e60 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f576c00c210 tx=0x7f576c01f2f0 comp rx=0 tx=0).stop 2023-10-30T12:40:33.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 -- 172.21.15.125:0/1071040786 shutdown_connections 2023-10-30T12:40:33.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 --2- 172.21.15.125:0/1071040786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57680069e0 0x7f5768006e60 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:33.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 -- 172.21.15.125:0/1071040786 >> 172.21.15.125:0/1071040786 conn(0x7f577c0c7910 msgr2=0x7f577c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:33.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 -- 172.21.15.125:0/1071040786 shutdown_connections 2023-10-30T12:40:33.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 -- 172.21.15.125:0/1071040786 wait complete. 2023-10-30T12:40:33.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 Processor -- start 2023-10-30T12:40:33.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 -- start start 2023-10-30T12:40:33.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57680069e0 0x7f577c0ca670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:33.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f577c158ce0 0x7f577c158180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:33.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f577c13ca80 0x7f577c13cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:33.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c0c2760 con 0x7f577c158ce0 2023-10-30T12:40:33.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577c157cf0 con 0x7f57680069e0 2023-10-30T12:40:33.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f57847d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577c157700 con 0x7f577c13ca80 2023-10-30T12:40:33.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.746+0000 7f5781d6e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f577c158ce0 0x7f577c158180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:33.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5781d6e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f577c158ce0 0x7f577c158180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44276/0 (socket says 172.21.15.125:44276) 2023-10-30T12:40:33.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5781d6e640 1 -- 172.21.15.125:0/3984595358 learned_addr learned my addr 172.21.15.125:0/3984595358 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:33.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5782d70640 1 --2- 172.21.15.125:0/3984595358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f577c13ca80 0x7f577c13cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:33.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f578256f640 1 --2- 172.21.15.125:0/3984595358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57680069e0 0x7f577c0ca670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:33.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5781d6e640 1 -- 172.21.15.125:0/3984595358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f577c13ca80 msgr2=0x7f577c13cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:33.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5781d6e640 1 --2- 172.21.15.125:0/3984595358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f577c13ca80 0x7f577c13cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:33.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5781d6e640 1 -- 172.21.15.125:0/3984595358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57680069e0 msgr2=0x7f577c0ca670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:33.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5781d6e640 1 --2- 172.21.15.125:0/3984595358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57680069e0 0x7f577c0ca670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:33.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5781d6e640 1 -- 172.21.15.125:0/3984595358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f577c13d440 con 0x7f577c158ce0 2023-10-30T12:40:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5781d6e640 1 --2- 172.21.15.125:0/3984595358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f577c158ce0 0x7f577c158180 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f576c01f120 tx=0x7f5768007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5772ffd640 1 -- 172.21.15.125:0/3984595358 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c004b30 con 0x7f577c158ce0 2023-10-30T12:40:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5772ffd640 1 -- 172.21.15.125:0/3984595358 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f576c004c90 con 0x7f577c158ce0 2023-10-30T12:40:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5772ffd640 1 -- 172.21.15.125:0/3984595358 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f576c007070 con 0x7f577c158ce0 2023-10-30T12:40:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f57847d3640 1 -- 172.21.15.125:0/3984595358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f577c159520 con 0x7f577c158ce0 2023-10-30T12:40:33.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f57847d3640 1 -- 172.21.15.125:0/3984595358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f577c157980 con 0x7f577c158ce0 2023-10-30T12:40:33.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f57847d3640 1 -- 172.21.15.125:0/3984595358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f577c04e170 con 0x7f577c158ce0 2023-10-30T12:40:33.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5772ffd640 1 -- 172.21.15.125:0/3984595358 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f576c0071f0 con 0x7f577c158ce0 2023-10-30T12:40:33.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5772ffd640 1 --2- 172.21.15.125:0/3984595358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57600664a0 0x7f5760068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:33.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.750+0000 7f5772ffd640 1 -- 172.21.15.125:0/3984595358 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f576c0b7070 con 0x7f577c158ce0 2023-10-30T12:40:33.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.754+0000 7f578256f640 1 --2- 172.21.15.125:0/3984595358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57600664a0 0x7f5760068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:33.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.754+0000 7f578256f640 1 --2- 172.21.15.125:0/3984595358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57600664a0 0x7f5760068960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f577c069010 tx=0x7f5778009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:33.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.758+0000 7f5772ffd640 1 -- 172.21.15.125:0/3984595358 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f576c0f1a90 con 0x7f577c158ce0 2023-10-30T12:40:33.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.938+0000 7f57847d3640 1 -- 172.21.15.125:0/3984595358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f577c138cd0 con 0x7f577c158ce0 2023-10-30T12:40:33.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.938+0000 7f5772ffd640 1 -- 172.21.15.125:0/3984595358 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f576c0f1620 con 0x7f577c158ce0 2023-10-30T12:40:33.941 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:40:33.941 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:39:53.073020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:33.942 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 61 2023-10-30T12:40:33.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.942+0000 7f57847d3640 1 -- 172.21.15.125:0/3984595358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57600664a0 msgr2=0x7f5760068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:33.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.942+0000 7f57847d3640 1 --2- 172.21.15.125:0/3984595358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57600664a0 0x7f5760068960 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f577c069010 tx=0x7f5778009040 comp rx=0 tx=0).stop 2023-10-30T12:40:33.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.942+0000 7f57847d3640 1 -- 172.21.15.125:0/3984595358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f577c158ce0 msgr2=0x7f577c158180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:33.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.942+0000 7f57847d3640 1 --2- 172.21.15.125:0/3984595358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f577c158ce0 0x7f577c158180 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f576c01f120 tx=0x7f5768007a10 comp rx=0 tx=0).stop 2023-10-30T12:40:33.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.942+0000 7f57847d3640 1 -- 172.21.15.125:0/3984595358 shutdown_connections 2023-10-30T12:40:33.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.942+0000 7f57847d3640 1 --2- 172.21.15.125:0/3984595358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57600664a0 0x7f5760068960 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:33.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.942+0000 7f57847d3640 1 --2- 172.21.15.125:0/3984595358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f577c13ca80 0x7f577c13cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:33.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.942+0000 7f57847d3640 1 --2- 172.21.15.125:0/3984595358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f577c158ce0 0x7f577c158180 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:33.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.942+0000 7f57847d3640 1 --2- 172.21.15.125:0/3984595358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57680069e0 0x7f577c0ca670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:33.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.942+0000 7f57847d3640 1 -- 172.21.15.125:0/3984595358 >> 172.21.15.125:0/3984595358 conn(0x7f577c0c7910 msgr2=0x7f577c0c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:33.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.942+0000 7f57847d3640 1 -- 172.21.15.125:0/3984595358 shutdown_connections 2023-10-30T12:40:33.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:33.942+0000 7f57847d3640 1 -- 172.21.15.125:0/3984595358 wait complete. 2023-10-30T12:40:33.960 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2023-10-30T12:40:33.961 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T12:40:33.961 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:40:33.980 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T12:40:33.980 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:34.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.114+0000 7f570e26b640 1 Processor -- start 2023-10-30T12:40:34.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.114+0000 7f570e26b640 1 -- start start 2023-10-30T12:40:34.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f570e26b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f57080c05d0 0x7f570813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:34.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f570e26b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f57080c6f60 0x7f570813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:34.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f570e26b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57080c0a50 con 0x7f57080c6b80 2023-10-30T12:40:34.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f570e26b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570806f6e0 con 0x7f57080c05d0 2023-10-30T12:40:34.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f570e26b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f570806f9c0 con 0x7f57080c6f60 2023-10-30T12:40:34.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f57077fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f57080c05d0 0x7f570813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:34.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f5707fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f57080c6b80 0x7f57080bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55192/0 (socket says 172.21.15.125:55192) 2023-10-30T12:40:34.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f57077fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f57080c05d0 0x7f570813ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44288/0 (socket says 172.21.15.125:44288) 2023-10-30T12:40:34.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f5707fff640 1 -- 172.21.15.125:0/1053546733 learned_addr learned my addr 172.21.15.125:0/1053546733 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:40:34.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f570c808640 1 --2- 172.21.15.125:0/1053546733 >> v2:172.21.15.136:3300/0 conn(0x7f57080c6f60 0x7f570813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:34.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f57077fe640 1 -- 172.21.15.125:0/1053546733 >> v2:172.21.15.136:3300/0 conn(0x7f57080c6f60 msgr2=0x7f570813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:34.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f57077fe640 1 --2- 172.21.15.125:0/1053546733 >> v2:172.21.15.136:3300/0 conn(0x7f57080c6f60 0x7f570813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f57077fe640 1 -- 172.21.15.125:0/1053546733 >> v1:172.21.15.125:6789/0 conn(0x7f57080c6b80 legacy=0x7f57080bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:34.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f57077fe640 1 -- 172.21.15.125:0/1053546733 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570806fca0 con 0x7f57080c05d0 2023-10-30T12:40:34.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.118+0000 7f5706ffd640 1 -- 172.21.15.125:0/1053546733 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279943210 0 0) 0x7f57080c0a50 con 0x7f57080c6b80 2023-10-30T12:40:34.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f57077fe640 1 --2- 172.21.15.125:0/1053546733 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57080c05d0 0x7f570813ed90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f57080c89b0 tx=0x7f56fc01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18e770efa1b1d093 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:34.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f5706ffd640 1 -- 172.21.15.125:0/1053546733 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc0ab600 con 0x7f57080c05d0 2023-10-30T12:40:34.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f5706ffd640 1 -- 172.21.15.125:0/1053546733 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56fc0a7050 con 0x7f57080c05d0 2023-10-30T12:40:34.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f5706ffd640 1 -- 172.21.15.125:0/1053546733 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56fc0ab980 con 0x7f57080c05d0 2023-10-30T12:40:34.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 -- 172.21.15.125:0/1053546733 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57080c05d0 msgr2=0x7f570813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:34.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 --2- 172.21.15.125:0/1053546733 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57080c05d0 0x7f570813ed90 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f57080c89b0 tx=0x7f56fc01c460 comp rx=0 tx=0).stop 2023-10-30T12:40:34.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 -- 172.21.15.125:0/1053546733 shutdown_connections 2023-10-30T12:40:34.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 --2- 172.21.15.125:0/1053546733 >> v2:172.21.15.136:3300/0 conn(0x7f57080c6f60 0x7f570813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 --2- 172.21.15.125:0/1053546733 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57080c05d0 0x7f570813ed90 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 -- 172.21.15.125:0/1053546733 >> 172.21.15.125:0/1053546733 conn(0x7f57080c7910 msgr2=0x7f57080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:34.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 -- 172.21.15.125:0/1053546733 shutdown_connections 2023-10-30T12:40:34.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 -- 172.21.15.125:0/1053546733 wait complete. 2023-10-30T12:40:34.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 Processor -- start 2023-10-30T12:40:34.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 -- start start 2023-10-30T12:40:34.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57080c05d0 0x7f570813dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:34.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57080c6b80 0x7f5708139c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:34.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57080c6f60 0x7f570813a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:34.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5708159a50 con 0x7f57080c05d0 2023-10-30T12:40:34.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57081f5f40 con 0x7f57080c6f60 2023-10-30T12:40:34.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f570e26b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57081511f0 con 0x7f57080c6b80 2023-10-30T12:40:34.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f5707fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57080c05d0 0x7f570813dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:34.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f5707fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57080c05d0 0x7f570813dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44298/0 (socket says 172.21.15.125:44298) 2023-10-30T12:40:34.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.122+0000 7f5707fff640 1 -- 172.21.15.125:0/2270996460 learned_addr learned my addr 172.21.15.125:0/2270996460 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:34.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f57077fe640 1 --2- 172.21.15.125:0/2270996460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57080c6b80 0x7f5708139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:34.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f570c808640 1 --2- 172.21.15.125:0/2270996460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57080c6f60 0x7f570813a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:34.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f5707fff640 1 -- 172.21.15.125:0/2270996460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57080c6b80 msgr2=0x7f5708139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:34.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f5707fff640 1 --2- 172.21.15.125:0/2270996460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57080c6b80 0x7f5708139c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f5707fff640 1 -- 172.21.15.125:0/2270996460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57080c6f60 msgr2=0x7f570813a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:34.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f5707fff640 1 --2- 172.21.15.125:0/2270996460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57080c6f60 0x7f570813a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f5707fff640 1 -- 172.21.15.125:0/2270996460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56fc009420 con 0x7f57080c05d0 2023-10-30T12:40:34.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f5707fff640 1 --2- 172.21.15.125:0/2270996460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57080c05d0 0x7f570813dcc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f56f800c520 tx=0x7f56f800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:34.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f5704ff9640 1 -- 172.21.15.125:0/2270996460 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f80074e0 con 0x7f57080c05d0 2023-10-30T12:40:34.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f5704ff9640 1 -- 172.21.15.125:0/2270996460 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f8007640 con 0x7f57080c05d0 2023-10-30T12:40:34.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f5704ff9640 1 -- 172.21.15.125:0/2270996460 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f80049f0 con 0x7f57080c05d0 2023-10-30T12:40:34.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f570e26b640 1 -- 172.21.15.125:0/2270996460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570813e800 con 0x7f57080c05d0 2023-10-30T12:40:34.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f570e26b640 1 -- 172.21.15.125:0/2270996460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57081f55a0 con 0x7f57080c05d0 2023-10-30T12:40:34.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f570e26b640 1 -- 172.21.15.125:0/2270996460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570804e170 con 0x7f57080c05d0 2023-10-30T12:40:34.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f5704ff9640 1 -- 172.21.15.125:0/2270996460 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f56f8004b50 con 0x7f57080c05d0 2023-10-30T12:40:34.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f5704ff9640 1 --2- 172.21.15.125:0/2270996460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f56f40664a0 0x7f56f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:34.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.126+0000 7f5704ff9640 1 -- 172.21.15.125:0/2270996460 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f56f8083190 con 0x7f57080c05d0 2023-10-30T12:40:34.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.130+0000 7f57077fe640 1 --2- 172.21.15.125:0/2270996460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f56f40664a0 0x7f56f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:34.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.130+0000 7f57077fe640 1 --2- 172.21.15.125:0/2270996460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f56f40664a0 0x7f56f4068960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f56fc0ad000 tx=0x7f56fc0a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:34.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.130+0000 7f5704ff9640 1 -- 172.21.15.125:0/2270996460 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f56f8050510 con 0x7f57080c05d0 2023-10-30T12:40:34.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.314+0000 7f570e26b640 1 -- 172.21.15.125:0/2270996460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57080c03b0 con 0x7f57080c05d0 2023-10-30T12:40:34.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.314+0000 7f5704ff9640 1 -- 172.21.15.125:0/2270996460 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f56f8053b30 con 0x7f57080c05d0 2023-10-30T12:40:34.314 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:40:34.315 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:39:53.073020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:34.315 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 61 2023-10-30T12:40:34.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.314+0000 7f570e26b640 1 -- 172.21.15.125:0/2270996460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f56f40664a0 msgr2=0x7f56f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:34.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.314+0000 7f570e26b640 1 --2- 172.21.15.125:0/2270996460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f56f40664a0 0x7f56f4068960 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f56fc0ad000 tx=0x7f56fc0a6000 comp rx=0 tx=0).stop 2023-10-30T12:40:34.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.314+0000 7f570e26b640 1 -- 172.21.15.125:0/2270996460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57080c05d0 msgr2=0x7f570813dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:34.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.314+0000 7f570e26b640 1 --2- 172.21.15.125:0/2270996460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57080c05d0 0x7f570813dcc0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f56f800c520 tx=0x7f56f800c9e0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.314+0000 7f570e26b640 1 -- 172.21.15.125:0/2270996460 shutdown_connections 2023-10-30T12:40:34.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.314+0000 7f570e26b640 1 --2- 172.21.15.125:0/2270996460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f56f40664a0 0x7f56f4068960 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.314+0000 7f570e26b640 1 --2- 172.21.15.125:0/2270996460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f57080c6f60 0x7f570813a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.314+0000 7f570e26b640 1 --2- 172.21.15.125:0/2270996460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f57080c6b80 0x7f5708139c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.314+0000 7f570e26b640 1 --2- 172.21.15.125:0/2270996460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f57080c05d0 0x7f570813dcc0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:34.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.314+0000 7f570e26b640 1 -- 172.21.15.125:0/2270996460 >> 172.21.15.125:0/2270996460 conn(0x7f57080c7910 msgr2=0x7f57080c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:34.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.314+0000 7f570e26b640 1 -- 172.21.15.125:0/2270996460 shutdown_connections 2023-10-30T12:40:34.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:34.314+0000 7f570e26b640 1 -- 172.21.15.125:0/2270996460 wait complete. 2023-10-30T12:40:34.337 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:40:36.337 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:36.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.482+0000 7fbb48e79640 1 Processor -- start 2023-10-30T12:40:36.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.482+0000 7fbb48e79640 1 -- start start 2023-10-30T12:40:36.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.482+0000 7fbb48e79640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbb4415d6f0 0x7fbb4415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.482+0000 7fbb48e79640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb440cf160 0x7fbb440cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.482+0000 7fbb48e79640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb440cb800 con 0x7fbb441538a0 2023-10-30T12:40:36.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.482+0000 7fbb48e79640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb440cb330 con 0x7fbb440cf160 2023-10-30T12:40:36.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.482+0000 7fbb48e79640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb440cad70 con 0x7fbb4415d6f0 2023-10-30T12:40:36.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.486+0000 7fbb42d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb440cf160 0x7fbb440cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.486+0000 7fbb42d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb440cf160 0x7fbb440cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44312/0 (socket says 172.21.15.125:44312) 2023-10-30T12:40:36.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.486+0000 7fbb42d9d640 1 -- 172.21.15.125:0/3016777168 learned_addr learned my addr 172.21.15.125:0/3016777168 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:36.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.486+0000 7fbb41d9b640 1 --2- 172.21.15.125:0/3016777168 >> v2:172.21.15.136:3300/0 conn(0x7fbb4415d6f0 0x7fbb4415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.486+0000 7fbb42d9d640 1 -- 172.21.15.125:0/3016777168 >> v2:172.21.15.136:3300/0 conn(0x7fbb4415d6f0 msgr2=0x7fbb4415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:36.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.486+0000 7fbb42d9d640 1 --2- 172.21.15.125:0/3016777168 >> v2:172.21.15.136:3300/0 conn(0x7fbb4415d6f0 0x7fbb4415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.486+0000 7fbb42d9d640 1 -- 172.21.15.125:0/3016777168 >> v1:172.21.15.136:6789/0 conn(0x7fbb441538a0 legacy=0x7fbb440ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:36.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.486+0000 7fbb42d9d640 1 -- 172.21.15.125:0/3016777168 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb440cb050 con 0x7fbb440cf160 2023-10-30T12:40:36.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.486+0000 7fbb42d9d640 1 --2- 172.21.15.125:0/3016777168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb440cf160 0x7fbb440cf590 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbb440b30a0 tx=0x7fbb3c01f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8f9f2409f721cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:36.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.486+0000 7fbb4159a640 1 -- 172.21.15.125:0/3016777168 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c0af400 con 0x7fbb440cf160 2023-10-30T12:40:36.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.486+0000 7fbb4159a640 1 -- 172.21.15.125:0/3016777168 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3c0ab050 con 0x7fbb440cf160 2023-10-30T12:40:36.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.486+0000 7fbb4159a640 1 -- 172.21.15.125:0/3016777168 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c0af780 con 0x7fbb440cf160 2023-10-30T12:40:36.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 -- 172.21.15.125:0/3016777168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb440cf160 msgr2=0x7fbb440cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 --2- 172.21.15.125:0/3016777168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb440cf160 0x7fbb440cf590 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbb440b30a0 tx=0x7fbb3c01f180 comp rx=0 tx=0).stop 2023-10-30T12:40:36.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 -- 172.21.15.125:0/3016777168 shutdown_connections 2023-10-30T12:40:36.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 --2- 172.21.15.125:0/3016777168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb440cf160 0x7fbb440cf590 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 --2- 172.21.15.125:0/3016777168 >> v2:172.21.15.136:3300/0 conn(0x7fbb4415d6f0 0x7fbb4415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 -- 172.21.15.125:0/3016777168 >> 172.21.15.125:0/3016777168 conn(0x7fbb4406bfe0 msgr2=0x7fbb4406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:36.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 -- 172.21.15.125:0/3016777168 shutdown_connections 2023-10-30T12:40:36.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 -- 172.21.15.125:0/3016777168 wait complete. 2023-10-30T12:40:36.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 Processor -- start 2023-10-30T12:40:36.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 -- start start 2023-10-30T12:40:36.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb440cf160 0x7fbb441ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb441538a0 0x7fbb440b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb4415d6f0 0x7fbb441e3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb441e4e10 con 0x7fbb440cf160 2023-10-30T12:40:36.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4414d680 con 0x7fbb441538a0 2023-10-30T12:40:36.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb48e79640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb441e2a40 con 0x7fbb4415d6f0 2023-10-30T12:40:36.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb4259c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb440cf160 0x7fbb441ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb4259c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb440cf160 0x7fbb441ef890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44318/0 (socket says 172.21.15.125:44318) 2023-10-30T12:40:36.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb4259c640 1 -- 172.21.15.125:0/1563416693 learned_addr learned my addr 172.21.15.125:0/1563416693 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:36.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb41d9b640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb441538a0 0x7fbb440b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.490+0000 7fbb42d9d640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb4415d6f0 0x7fbb441e3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb4259c640 1 -- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb4415d6f0 msgr2=0x7fbb441e3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb4259c640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb4415d6f0 0x7fbb441e3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb4259c640 1 -- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb441538a0 msgr2=0x7fbb440b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb4259c640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb441538a0 0x7fbb440b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb4259c640 1 -- 172.21.15.125:0/1563416693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb3c009b30 con 0x7fbb440cf160 2023-10-30T12:40:36.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb42d9d640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb4415d6f0 0x7fbb441e3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:36.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb41d9b640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb441538a0 0x7fbb440b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:36.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb4259c640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb440cf160 0x7fbb441ef890 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbb3400c480 tx=0x7fbb3400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:36.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb32ffd640 1 -- 172.21.15.125:0/1563416693 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb34012830 con 0x7fbb440cf160 2023-10-30T12:40:36.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb32ffd640 1 -- 172.21.15.125:0/1563416693 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb340129b0 con 0x7fbb440cf160 2023-10-30T12:40:36.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb48e79640 1 -- 172.21.15.125:0/1563416693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb44158f70 con 0x7fbb440cf160 2023-10-30T12:40:36.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb32ffd640 1 -- 172.21.15.125:0/1563416693 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb34012d60 con 0x7fbb440cf160 2023-10-30T12:40:36.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb48e79640 1 -- 172.21.15.125:0/1563416693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb441594f0 con 0x7fbb440cf160 2023-10-30T12:40:36.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb48e79640 1 -- 172.21.15.125:0/1563416693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb440b2a50 con 0x7fbb440cf160 2023-10-30T12:40:36.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb32ffd640 1 -- 172.21.15.125:0/1563416693 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb3401c460 con 0x7fbb440cf160 2023-10-30T12:40:36.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb32ffd640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb240664a0 0x7fbb24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:36.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.494+0000 7fbb32ffd640 1 -- 172.21.15.125:0/1563416693 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fbb34083ec0 con 0x7fbb440cf160 2023-10-30T12:40:36.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.498+0000 7fbb41d9b640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb240664a0 0x7fbb24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:36.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.498+0000 7fbb41d9b640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb240664a0 0x7fbb24068960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbb440c48a0 tx=0x7fbb38008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:36.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.502+0000 7fbb32ffd640 1 -- 172.21.15.125:0/1563416693 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbb34051190 con 0x7fbb440cf160 2023-10-30T12:40:36.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.682+0000 7fbb48e79640 1 -- 172.21.15.125:0/1563416693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb4406c3c0 con 0x7fbb440cf160 2023-10-30T12:40:36.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.682+0000 7fbb32ffd640 1 -- 172.21.15.125:0/1563416693 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fbb340547b0 con 0x7fbb440cf160 2023-10-30T12:40:36.684 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:40:36.684 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:39:53.073020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:36.685 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 61 2023-10-30T12:40:36.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.686+0000 7fbb48e79640 1 -- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb240664a0 msgr2=0x7fbb24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.686+0000 7fbb48e79640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb240664a0 0x7fbb24068960 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fbb440c48a0 tx=0x7fbb38008040 comp rx=0 tx=0).stop 2023-10-30T12:40:36.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.686+0000 7fbb48e79640 1 -- 172.21.15.125:0/1563416693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb440cf160 msgr2=0x7fbb441ef890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:36.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.686+0000 7fbb48e79640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb440cf160 0x7fbb441ef890 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbb3400c480 tx=0x7fbb3400c940 comp rx=0 tx=0).stop 2023-10-30T12:40:36.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.686+0000 7fbb48e79640 1 -- 172.21.15.125:0/1563416693 shutdown_connections 2023-10-30T12:40:36.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.686+0000 7fbb48e79640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb240664a0 0x7fbb24068960 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.686+0000 7fbb48e79640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb4415d6f0 0x7fbb441e3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.686+0000 7fbb48e79640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb441538a0 0x7fbb440b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.686+0000 7fbb48e79640 1 --2- 172.21.15.125:0/1563416693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb440cf160 0x7fbb441ef890 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:36.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.686+0000 7fbb48e79640 1 -- 172.21.15.125:0/1563416693 >> 172.21.15.125:0/1563416693 conn(0x7fbb4406bfe0 msgr2=0x7fbb440c4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:36.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.686+0000 7fbb48e79640 1 -- 172.21.15.125:0/1563416693 shutdown_connections 2023-10-30T12:40:36.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:36.686+0000 7fbb48e79640 1 -- 172.21.15.125:0/1563416693 wait complete. 2023-10-30T12:40:36.704 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:40:38.705 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:38.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc934b80640 1 Processor -- start 2023-10-30T12:40:38.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc934b80640 1 -- start start 2023-10-30T12:40:38.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc934b80640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc930159e60 0x7fc93006f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:38.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc934b80640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300bfec0 0x7fc9300c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:38.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc934b80640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc93006a720 0x7fc93006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:38.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc934b80640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc93006fa80 con 0x7fc93006a720 2023-10-30T12:40:38.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc934b80640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc930071290 con 0x7fc930159e60 2023-10-30T12:40:38.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc934b80640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc930071570 con 0x7fc9300bfec0 2023-10-30T12:40:38.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc92ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc93006a720 0x7fc93006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:38.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc92ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc93006a720 0x7fc93006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44326/0 (socket says 172.21.15.125:44326) 2023-10-30T12:40:38.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc92ed9d640 1 -- 172.21.15.125:0/4053570743 learned_addr learned my addr 172.21.15.125:0/4053570743 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:38.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc92e59c640 1 --2- 172.21.15.125:0/4053570743 >> v2:172.21.15.136:3300/0 conn(0x7fc930159e60 0x7fc93006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:38.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc92dd9b640 1 --2- 172.21.15.125:0/4053570743 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300bfec0 0x7fc9300c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:38.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc92ed9d640 1 -- 172.21.15.125:0/4053570743 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300bfec0 msgr2=0x7fc9300c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:38.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc92ed9d640 1 --2- 172.21.15.125:0/4053570743 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300bfec0 0x7fc9300c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:38.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc92ed9d640 1 -- 172.21.15.125:0/4053570743 >> v2:172.21.15.136:3300/0 conn(0x7fc930159e60 msgr2=0x7fc93006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:38.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc92ed9d640 1 --2- 172.21.15.125:0/4053570743 >> v2:172.21.15.136:3300/0 conn(0x7fc930159e60 0x7fc93006f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:38.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.850+0000 7fc92ed9d640 1 -- 172.21.15.125:0/4053570743 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc930071850 con 0x7fc93006a720 2023-10-30T12:40:38.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.854+0000 7fc92ed9d640 1 --2- 172.21.15.125:0/4053570743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006a720 0x7fc93006abb0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc9300c8950 tx=0x7fc92801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c66ec05f0877947e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:38.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.854+0000 7fc92d59a640 1 -- 172.21.15.125:0/4053570743 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9280af400 con 0x7fc93006a720 2023-10-30T12:40:38.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.854+0000 7fc92d59a640 1 -- 172.21.15.125:0/4053570743 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9280ab050 con 0x7fc93006a720 2023-10-30T12:40:38.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.854+0000 7fc92d59a640 1 -- 172.21.15.125:0/4053570743 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9280af780 con 0x7fc93006a720 2023-10-30T12:40:38.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.854+0000 7fc934b80640 1 -- 172.21.15.125:0/4053570743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006a720 msgr2=0x7fc93006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:38.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.854+0000 7fc934b80640 1 --2- 172.21.15.125:0/4053570743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006a720 0x7fc93006abb0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fc9300c8950 tx=0x7fc92801f1c0 comp rx=0 tx=0).stop 2023-10-30T12:40:38.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.854+0000 7fc934b80640 1 -- 172.21.15.125:0/4053570743 shutdown_connections 2023-10-30T12:40:38.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.854+0000 7fc934b80640 1 --2- 172.21.15.125:0/4053570743 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006a720 0x7fc93006abb0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:38.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.854+0000 7fc934b80640 1 --2- 172.21.15.125:0/4053570743 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300bfec0 0x7fc9300c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:38.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.854+0000 7fc934b80640 1 --2- 172.21.15.125:0/4053570743 >> v2:172.21.15.136:3300/0 conn(0x7fc930159e60 0x7fc93006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:38.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.854+0000 7fc934b80640 1 -- 172.21.15.125:0/4053570743 >> 172.21.15.125:0/4053570743 conn(0x7fc9300c7910 msgr2=0x7fc9300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:38.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.854+0000 7fc934b80640 1 -- 172.21.15.125:0/4053570743 shutdown_connections 2023-10-30T12:40:38.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.854+0000 7fc934b80640 1 -- 172.21.15.125:0/4053570743 wait complete. 2023-10-30T12:40:38.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc934b80640 1 Processor -- start 2023-10-30T12:40:38.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc934b80640 1 -- start start 2023-10-30T12:40:38.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc934b80640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc93006a720 0x7fc9300afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:38.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc934b80640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300bfec0 0x7fc9300b0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:38.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc934b80640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc930159e60 0x7fc9301ef6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:38.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc934b80640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9301ef2a0 con 0x7fc930159e60 2023-10-30T12:40:38.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc934b80640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9300c1eb0 con 0x7fc93006a720 2023-10-30T12:40:38.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc934b80640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9300c2c80 con 0x7fc9300bfec0 2023-10-30T12:40:38.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc92ed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc930159e60 0x7fc9301ef6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:38.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc92ed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc930159e60 0x7fc9301ef6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44336/0 (socket says 172.21.15.125:44336) 2023-10-30T12:40:38.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc92ed9d640 1 -- 172.21.15.125:0/2705899521 learned_addr learned my addr 172.21.15.125:0/2705899521 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:38.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc92e59c640 1 --2- 172.21.15.125:0/2705899521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc93006a720 0x7fc9300afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:38.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc92dd9b640 1 --2- 172.21.15.125:0/2705899521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300bfec0 0x7fc9300b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:38.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc92ed9d640 1 -- 172.21.15.125:0/2705899521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300bfec0 msgr2=0x7fc9300b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:38.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc92ed9d640 1 --2- 172.21.15.125:0/2705899521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300bfec0 0x7fc9300b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:38.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc92ed9d640 1 -- 172.21.15.125:0/2705899521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc93006a720 msgr2=0x7fc9300afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:38.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc92ed9d640 1 --2- 172.21.15.125:0/2705899521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc93006a720 0x7fc9300afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:38.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc92ed9d640 1 -- 172.21.15.125:0/2705899521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc928009b30 con 0x7fc930159e60 2023-10-30T12:40:38.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc92ed9d640 1 --2- 172.21.15.125:0/2705899521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc930159e60 0x7fc9301ef6c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc9280b2040 tx=0x7fc92801bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:38.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc91effd640 1 -- 172.21.15.125:0/2705899521 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9280afc60 con 0x7fc930159e60 2023-10-30T12:40:38.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc91effd640 1 -- 172.21.15.125:0/2705899521 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9280afde0 con 0x7fc930159e60 2023-10-30T12:40:38.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc91effd640 1 -- 172.21.15.125:0/2705899521 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9280b8640 con 0x7fc930159e60 2023-10-30T12:40:38.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc934b80640 1 -- 172.21.15.125:0/2705899521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9300c2100 con 0x7fc930159e60 2023-10-30T12:40:38.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.858+0000 7fc934b80640 1 -- 172.21.15.125:0/2705899521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9300c26c0 con 0x7fc930159e60 2023-10-30T12:40:38.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.862+0000 7fc934b80640 1 -- 172.21.15.125:0/2705899521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9301ef3e0 con 0x7fc930159e60 2023-10-30T12:40:38.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.862+0000 7fc91effd640 1 -- 172.21.15.125:0/2705899521 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc9280b6040 con 0x7fc930159e60 2023-10-30T12:40:38.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.862+0000 7fc91effd640 1 --2- 172.21.15.125:0/2705899521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9100664d0 0x7fc910068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:38.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.862+0000 7fc91effd640 1 -- 172.21.15.125:0/2705899521 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fc928121d30 con 0x7fc930159e60 2023-10-30T12:40:38.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.862+0000 7fc92e59c640 1 --2- 172.21.15.125:0/2705899521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9100664d0 0x7fc910068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:38.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.862+0000 7fc92e59c640 1 --2- 172.21.15.125:0/2705899521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9100664d0 0x7fc910068990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc92000b240 tx=0x7fc92000b3a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:38.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:38.866+0000 7fc91effd640 1 -- 172.21.15.125:0/2705899521 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc9301ef3e0 con 0x7fc930159e60 2023-10-30T12:40:39.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:39.042+0000 7fc934b80640 1 -- 172.21.15.125:0/2705899521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9301ef3e0 con 0x7fc930159e60 2023-10-30T12:40:39.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:39.046+0000 7fc91effd640 1 -- 172.21.15.125:0/2705899521 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fc9301ef3e0 con 0x7fc930159e60 2023-10-30T12:40:39.046 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:40:39.047 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:39:53.073020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:39.048 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 61 2023-10-30T12:40:39.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:39.046+0000 7fc934b80640 1 -- 172.21.15.125:0/2705899521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9100664d0 msgr2=0x7fc910068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:39.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:39.046+0000 7fc934b80640 1 --2- 172.21.15.125:0/2705899521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9100664d0 0x7fc910068990 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc92000b240 tx=0x7fc92000b3a0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:39.046+0000 7fc934b80640 1 -- 172.21.15.125:0/2705899521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc930159e60 msgr2=0x7fc9301ef6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:39.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:39.046+0000 7fc934b80640 1 --2- 172.21.15.125:0/2705899521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc930159e60 0x7fc9301ef6c0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fc9280b2040 tx=0x7fc92801bf60 comp rx=0 tx=0).stop 2023-10-30T12:40:39.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:39.046+0000 7fc934b80640 1 -- 172.21.15.125:0/2705899521 shutdown_connections 2023-10-30T12:40:39.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:39.046+0000 7fc934b80640 1 --2- 172.21.15.125:0/2705899521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9100664d0 0x7fc910068990 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:39.046+0000 7fc934b80640 1 --2- 172.21.15.125:0/2705899521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc930159e60 0x7fc9301ef6c0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:39.046+0000 7fc934b80640 1 --2- 172.21.15.125:0/2705899521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300bfec0 0x7fc9300b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:39.046+0000 7fc934b80640 1 --2- 172.21.15.125:0/2705899521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc93006a720 0x7fc9300afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:39.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:39.046+0000 7fc934b80640 1 -- 172.21.15.125:0/2705899521 >> 172.21.15.125:0/2705899521 conn(0x7fc9300c7910 msgr2=0x7fc9300c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:39.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:39.046+0000 7fc934b80640 1 -- 172.21.15.125:0/2705899521 shutdown_connections 2023-10-30T12:40:39.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:39.046+0000 7fc934b80640 1 -- 172.21.15.125:0/2705899521 wait complete. 2023-10-30T12:40:39.067 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:40:41.068 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:41.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28bcf3e640 1 Processor -- start 2023-10-30T12:40:41.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28bcf3e640 1 -- start start 2023-10-30T12:40:41.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28bcf3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f28b80c6b80 0x7f28b80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:41.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28bcf3e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f28b80c6f60 0x7f28b80c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:41.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28bcf3e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b80c0a50 con 0x7f28b80c8430 2023-10-30T12:40:41.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28bcf3e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b806f6e0 con 0x7f28b80c6f60 2023-10-30T12:40:41.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28bcf3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b806f9c0 con 0x7f28b80c6b80 2023-10-30T12:40:41.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28b6d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f28b80c8430 0x7f28b813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36728/0 (socket says 172.21.15.125:36728) 2023-10-30T12:40:41.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28b6d9d640 1 -- 172.21.15.125:0/374194776 learned_addr learned my addr 172.21.15.125:0/374194776 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:40:41.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28b659c640 1 --2- 172.21.15.125:0/374194776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f28b80c6b80 0x7f28b80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:41.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28affff640 1 --2- 172.21.15.125:0/374194776 >> v2:172.21.15.136:3300/0 conn(0x7f28b80c6f60 0x7f28b80c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:41.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28b5d9b640 1 -- 172.21.15.125:0/374194776 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621373375 0 0) 0x7f28b80c0a50 con 0x7f28b80c8430 2023-10-30T12:40:41.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28b5d9b640 1 -- 172.21.15.125:0/374194776 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28a4003530 con 0x7f28b80c8430 2023-10-30T12:40:41.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28b659c640 1 -- 172.21.15.125:0/374194776 >> v2:172.21.15.136:3300/0 conn(0x7f28b80c6f60 msgr2=0x7f28b80c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:41.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28b659c640 1 --2- 172.21.15.125:0/374194776 >> v2:172.21.15.136:3300/0 conn(0x7f28b80c6f60 0x7f28b80c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:41.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28b659c640 1 -- 172.21.15.125:0/374194776 >> v1:172.21.15.125:6789/0 conn(0x7f28b80c8430 legacy=0x7f28b813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:41.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28b5d9b640 1 -- 172.21.15.125:0/374194776 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2541796861 0 0) 0x7f28a4003530 con 0x7f28b80c8430 2023-10-30T12:40:41.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28b659c640 1 -- 172.21.15.125:0/374194776 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b806fca0 con 0x7f28b80c6b80 2023-10-30T12:40:41.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.214+0000 7f28affff640 1 --2- 172.21.15.125:0/374194776 >> v2:172.21.15.136:3300/0 conn(0x7f28b80c6f60 0x7f28b80c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:40:41.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.218+0000 7f28b659c640 1 --2- 172.21.15.125:0/374194776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f28b80c6b80 0x7f28b80bfec0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f28a80097c0 tx=0x7f28a801c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=616b8df132bb06d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:41.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.218+0000 7f28b5d9b640 1 -- 172.21.15.125:0/374194776 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28b0001a00 con 0x7f28b80c6b80 2023-10-30T12:40:41.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.218+0000 7f28b5d9b640 1 -- 172.21.15.125:0/374194776 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a80a7050 con 0x7f28b80c6b80 2023-10-30T12:40:41.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.218+0000 7f28b5d9b640 1 -- 172.21.15.125:0/374194776 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a80ab600 con 0x7f28b80c6b80 2023-10-30T12:40:41.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.218+0000 7f28bcf3e640 1 -- 172.21.15.125:0/374194776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f28b80c6b80 msgr2=0x7f28b80bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:41.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.218+0000 7f28bcf3e640 1 --2- 172.21.15.125:0/374194776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f28b80c6b80 0x7f28b80bfec0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f28a80097c0 tx=0x7f28a801c3a0 comp rx=0 tx=0).stop 2023-10-30T12:40:41.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.218+0000 7f28bcf3e640 1 -- 172.21.15.125:0/374194776 shutdown_connections 2023-10-30T12:40:41.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.218+0000 7f28bcf3e640 1 --2- 172.21.15.125:0/374194776 >> v2:172.21.15.136:3300/0 conn(0x7f28b80c6f60 0x7f28b80c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:41.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.218+0000 7f28bcf3e640 1 --2- 172.21.15.125:0/374194776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f28b80c6b80 0x7f28b80bfec0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:41.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.218+0000 7f28bcf3e640 1 -- 172.21.15.125:0/374194776 >> 172.21.15.125:0/374194776 conn(0x7f28b80c7910 msgr2=0x7f28b80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:41.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.218+0000 7f28bcf3e640 1 -- 172.21.15.125:0/374194776 shutdown_connections 2023-10-30T12:40:41.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.218+0000 7f28bcf3e640 1 -- 172.21.15.125:0/374194776 wait complete. 2023-10-30T12:40:41.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28bcf3e640 1 Processor -- start 2023-10-30T12:40:41.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28bcf3e640 1 -- start start 2023-10-30T12:40:41.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28bcf3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80c6b80 0x7f28b8158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:41.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28bcf3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28b80c6f60 0x7f28b813dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:41.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28bcf3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f28b80c8430 0x7f28b8134580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:41.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28bcf3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b80c9830 con 0x7f28b80c6f60 2023-10-30T12:40:41.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28bcf3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28b8157720 con 0x7f28b80c6b80 2023-10-30T12:40:41.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28bcf3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b80c8cb0 con 0x7f28b80c8430 2023-10-30T12:40:41.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28affff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28b80c6f60 0x7f28b813dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:41.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28affff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28b80c6f60 0x7f28b813dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51220/0 (socket says 172.21.15.125:51220) 2023-10-30T12:40:41.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28affff640 1 -- 172.21.15.125:0/4009313447 learned_addr learned my addr 172.21.15.125:0/4009313447 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:41.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28b6d9d640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f28b80c8430 0x7f28b8134580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:41.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28b659c640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80c6b80 0x7f28b8158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:41.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28affff640 1 -- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f28b80c8430 msgr2=0x7f28b8134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:41.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28affff640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f28b80c8430 0x7f28b8134580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:41.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28affff640 1 -- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80c6b80 msgr2=0x7f28b8158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:41.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28affff640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80c6b80 0x7f28b8158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:41.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28affff640 1 -- 172.21.15.125:0/4009313447 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a0009420 con 0x7f28b80c6f60 2023-10-30T12:40:41.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28b659c640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80c6b80 0x7f28b8158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:41.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28b6d9d640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f28b80c8430 0x7f28b8134580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:41.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28affff640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28b80c6f60 0x7f28b813dcd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f28a000e9d0 tx=0x7f28a000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:41.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28aeffd640 1 -- 172.21.15.125:0/4009313447 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a000ca60 con 0x7f28b80c6f60 2023-10-30T12:40:41.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28aeffd640 1 -- 172.21.15.125:0/4009313447 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28a000cbc0 con 0x7f28b80c6f60 2023-10-30T12:40:41.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28aeffd640 1 -- 172.21.15.125:0/4009313447 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28a0013470 con 0x7f28b80c6f60 2023-10-30T12:40:41.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28bcf3e640 1 -- 172.21.15.125:0/4009313447 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a8009420 con 0x7f28b80c6f60 2023-10-30T12:40:41.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.222+0000 7f28bcf3e640 1 -- 172.21.15.125:0/4009313447 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b80c9270 con 0x7f28b80c6f60 2023-10-30T12:40:41.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.226+0000 7f28bcf3e640 1 -- 172.21.15.125:0/4009313447 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b804e170 con 0x7f28b80c6f60 2023-10-30T12:40:41.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.226+0000 7f28aeffd640 1 -- 172.21.15.125:0/4009313447 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f28a00135d0 con 0x7f28b80c6f60 2023-10-30T12:40:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.226+0000 7f28aeffd640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f28980664d0 0x7f2898068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.226+0000 7f28aeffd640 1 -- 172.21.15.125:0/4009313447 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f28a0083fb0 con 0x7f28b80c6f60 2023-10-30T12:40:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.226+0000 7f28b659c640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f28980664d0 0x7f2898068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:41.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.226+0000 7f28b659c640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f28980664d0 0x7f2898068990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f28a80ad000 tx=0x7f28a8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:41.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.230+0000 7f28aeffd640 1 -- 172.21.15.125:0/4009313447 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f28a00b5090 con 0x7f28b80c6f60 2023-10-30T12:40:41.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.414+0000 7f28bcf3e640 1 -- 172.21.15.125:0/4009313447 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28b80c7f90 con 0x7f28b80c6f60 2023-10-30T12:40:41.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.414+0000 7f28aeffd640 1 -- 172.21.15.125:0/4009313447 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f28a0051270 con 0x7f28b80c6f60 2023-10-30T12:40:41.415 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:40:41.415 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:39:53.073020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:41.416 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 61 2023-10-30T12:40:41.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.414+0000 7f28bcf3e640 1 -- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f28980664d0 msgr2=0x7f2898068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:41.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.414+0000 7f28bcf3e640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f28980664d0 0x7f2898068990 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f28a80ad000 tx=0x7f28a8008040 comp rx=0 tx=0).stop 2023-10-30T12:40:41.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.414+0000 7f28bcf3e640 1 -- 172.21.15.125:0/4009313447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28b80c6f60 msgr2=0x7f28b813dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:41.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.414+0000 7f28bcf3e640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28b80c6f60 0x7f28b813dcd0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f28a000e9d0 tx=0x7f28a000ee90 comp rx=0 tx=0).stop 2023-10-30T12:40:41.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.414+0000 7f28bcf3e640 1 -- 172.21.15.125:0/4009313447 shutdown_connections 2023-10-30T12:40:41.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.414+0000 7f28bcf3e640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f28980664d0 0x7f2898068990 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:41.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.414+0000 7f28bcf3e640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f28b80c8430 0x7f28b8134580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:41.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.414+0000 7f28bcf3e640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f28b80c6f60 0x7f28b813dcd0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:41.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.414+0000 7f28bcf3e640 1 --2- 172.21.15.125:0/4009313447 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f28b80c6b80 0x7f28b8158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:41.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.414+0000 7f28bcf3e640 1 -- 172.21.15.125:0/4009313447 >> 172.21.15.125:0/4009313447 conn(0x7f28b80c7910 msgr2=0x7f28b813bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:41.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.414+0000 7f28bcf3e640 1 -- 172.21.15.125:0/4009313447 shutdown_connections 2023-10-30T12:40:41.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:41.414+0000 7f28bcf3e640 1 -- 172.21.15.125:0/4009313447 wait complete. 2023-10-30T12:40:41.435 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:40:43.436 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:43.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40f994640 1 Processor -- start 2023-10-30T12:40:43.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40f994640 1 -- start start 2023-10-30T12:40:43.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40f994640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe408070740 0x7fe4080c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:43.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40f994640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe40815e700 0x7fe40806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:43.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40f994640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4080c5810 con 0x7fe4081580e0 2023-10-30T12:40:43.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40f994640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe408070b60 con 0x7fe40815e700 2023-10-30T12:40:43.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40f994640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40815eb90 con 0x7fe408070740 2023-10-30T12:40:43.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40df31640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe40815e700 0x7fe40806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:43.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40d730640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe4081580e0 0x7fe408069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36736/0 (socket says 172.21.15.125:36736) 2023-10-30T12:40:43.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40d730640 1 -- 172.21.15.125:0/1074971958 learned_addr learned my addr 172.21.15.125:0/1074971958 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:40:43.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40cf2f640 1 --2- 172.21.15.125:0/1074971958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe408070740 0x7fe4080c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:43.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe3fffff640 1 -- 172.21.15.125:0/1074971958 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053016817 0 0) 0x7fe4080c5810 con 0x7fe4081580e0 2023-10-30T12:40:43.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40df31640 1 -- 172.21.15.125:0/1074971958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe408070740 msgr2=0x7fe4080c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:43.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40df31640 1 --2- 172.21.15.125:0/1074971958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe408070740 0x7fe4080c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:43.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40df31640 1 -- 172.21.15.125:0/1074971958 >> v1:172.21.15.125:6789/0 conn(0x7fe4081580e0 legacy=0x7fe408069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:43.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.581+0000 7fe40df31640 1 -- 172.21.15.125:0/1074971958 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4080c6bb0 con 0x7fe40815e700 2023-10-30T12:40:43.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.585+0000 7fe40df31640 1 --2- 172.21.15.125:0/1074971958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40815e700 0x7fe40806a5a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe400009d90 tx=0x7fe40001c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87bd7f0135102a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:43.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.585+0000 7fe3fffff640 1 -- 172.21.15.125:0/1074971958 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4000a59e0 con 0x7fe40815e700 2023-10-30T12:40:43.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.585+0000 7fe3fffff640 1 -- 172.21.15.125:0/1074971958 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4000a5b40 con 0x7fe40815e700 2023-10-30T12:40:43.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.585+0000 7fe3fffff640 1 -- 172.21.15.125:0/1074971958 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4000ad440 con 0x7fe40815e700 2023-10-30T12:40:43.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.585+0000 7fe40f994640 1 -- 172.21.15.125:0/1074971958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40815e700 msgr2=0x7fe40806a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:43.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.585+0000 7fe40f994640 1 --2- 172.21.15.125:0/1074971958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40815e700 0x7fe40806a5a0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fe400009d90 tx=0x7fe40001c920 comp rx=0 tx=0).stop 2023-10-30T12:40:43.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.585+0000 7fe40f994640 1 -- 172.21.15.125:0/1074971958 shutdown_connections 2023-10-30T12:40:43.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.585+0000 7fe40f994640 1 --2- 172.21.15.125:0/1074971958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40815e700 0x7fe40806a5a0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:43.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.585+0000 7fe40f994640 1 --2- 172.21.15.125:0/1074971958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe408070740 0x7fe4080c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:43.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.585+0000 7fe40f994640 1 -- 172.21.15.125:0/1074971958 >> 172.21.15.125:0/1074971958 conn(0x7fe4080c7910 msgr2=0x7fe4080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:43.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.585+0000 7fe40f994640 1 -- 172.21.15.125:0/1074971958 shutdown_connections 2023-10-30T12:40:43.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.585+0000 7fe40f994640 1 -- 172.21.15.125:0/1074971958 wait complete. 2023-10-30T12:40:43.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40f994640 1 Processor -- start 2023-10-30T12:40:43.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40f994640 1 -- start start 2023-10-30T12:40:43.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40f994640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe408070740 0x7fe4081f9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:43.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40f994640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4081580e0 0x7fe408154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:43.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40f994640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40815e700 0x7fe4080c2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:43.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40f994640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4081fa150 con 0x7fe40815e700 2023-10-30T12:40:43.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40f994640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40815efc0 con 0x7fe4081580e0 2023-10-30T12:40:43.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40f994640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4080c36d0 con 0x7fe408070740 2023-10-30T12:40:43.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40df31640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40815e700 0x7fe4080c2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:43.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40df31640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40815e700 0x7fe4080c2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51228/0 (socket says 172.21.15.125:51228) 2023-10-30T12:40:43.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40df31640 1 -- 172.21.15.125:0/207750184 learned_addr learned my addr 172.21.15.125:0/207750184 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:43.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40d730640 1 --2- 172.21.15.125:0/207750184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe408070740 0x7fe4081f9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:43.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40cf2f640 1 --2- 172.21.15.125:0/207750184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4081580e0 0x7fe408154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:43.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40df31640 1 -- 172.21.15.125:0/207750184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe408070740 msgr2=0x7fe4081f9460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:43.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40df31640 1 --2- 172.21.15.125:0/207750184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe408070740 0x7fe4081f9460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:43.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40df31640 1 -- 172.21.15.125:0/207750184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4081580e0 msgr2=0x7fe408154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:43.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40df31640 1 --2- 172.21.15.125:0/207750184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4081580e0 0x7fe408154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:43.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40df31640 1 -- 172.21.15.125:0/207750184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe400009a70 con 0x7fe40815e700 2023-10-30T12:40:43.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40df31640 1 --2- 172.21.15.125:0/207750184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40815e700 0x7fe4080c2af0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe40001afd0 tx=0x7fe4000196c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:43.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe3fdffb640 1 -- 172.21.15.125:0/207750184 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4000addc0 con 0x7fe40815e700 2023-10-30T12:40:43.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe3fdffb640 1 -- 172.21.15.125:0/207750184 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe400009450 con 0x7fe40815e700 2023-10-30T12:40:43.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe3fdffb640 1 -- 172.21.15.125:0/207750184 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4000b7610 con 0x7fe40815e700 2023-10-30T12:40:43.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40f994640 1 -- 172.21.15.125:0/207750184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4081f9e70 con 0x7fe40815e700 2023-10-30T12:40:43.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.589+0000 7fe40f994640 1 -- 172.21.15.125:0/207750184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40806be70 con 0x7fe40815e700 2023-10-30T12:40:43.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.593+0000 7fe40f994640 1 -- 172.21.15.125:0/207750184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe408154360 con 0x7fe40815e700 2023-10-30T12:40:43.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.593+0000 7fe3fdffb640 1 -- 172.21.15.125:0/207750184 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe4000b4050 con 0x7fe40815e700 2023-10-30T12:40:43.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.593+0000 7fe3fdffb640 1 --2- 172.21.15.125:0/207750184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3ec0664d0 0x7fe3ec068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:43.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.593+0000 7fe3fdffb640 1 -- 172.21.15.125:0/207750184 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fe40011f5a0 con 0x7fe40815e700 2023-10-30T12:40:43.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.593+0000 7fe40d730640 1 --2- 172.21.15.125:0/207750184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3ec0664d0 0x7fe3ec068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:43.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.593+0000 7fe40d730640 1 --2- 172.21.15.125:0/207750184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3ec0664d0 0x7fe3ec068990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe404007fd0 tx=0x7fe404007d60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:43.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.597+0000 7fe3fdffb640 1 -- 172.21.15.125:0/207750184 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe4000eca30 con 0x7fe40815e700 2023-10-30T12:40:43.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.777+0000 7fe40f994640 1 -- 172.21.15.125:0/207750184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe40804e170 con 0x7fe40815e700 2023-10-30T12:40:43.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.781+0000 7fe3fdffb640 1 -- 172.21.15.125:0/207750184 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fe4000f0050 con 0x7fe40815e700 2023-10-30T12:40:43.782 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:40:43.783 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:39:53.073020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:43.783 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 61 2023-10-30T12:40:43.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.781+0000 7fe40f994640 1 -- 172.21.15.125:0/207750184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3ec0664d0 msgr2=0x7fe3ec068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:43.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.781+0000 7fe40f994640 1 --2- 172.21.15.125:0/207750184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3ec0664d0 0x7fe3ec068990 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe404007fd0 tx=0x7fe404007d60 comp rx=0 tx=0).stop 2023-10-30T12:40:43.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.781+0000 7fe40f994640 1 -- 172.21.15.125:0/207750184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40815e700 msgr2=0x7fe4080c2af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:43.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.781+0000 7fe40f994640 1 --2- 172.21.15.125:0/207750184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40815e700 0x7fe4080c2af0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fe40001afd0 tx=0x7fe4000196c0 comp rx=0 tx=0).stop 2023-10-30T12:40:43.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.781+0000 7fe40f994640 1 -- 172.21.15.125:0/207750184 shutdown_connections 2023-10-30T12:40:43.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.781+0000 7fe40f994640 1 --2- 172.21.15.125:0/207750184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3ec0664d0 0x7fe3ec068990 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:43.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.781+0000 7fe40f994640 1 --2- 172.21.15.125:0/207750184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe40815e700 0x7fe4080c2af0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:43.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.781+0000 7fe40f994640 1 --2- 172.21.15.125:0/207750184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4081580e0 0x7fe408154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:43.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.781+0000 7fe40f994640 1 --2- 172.21.15.125:0/207750184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe408070740 0x7fe4081f9460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:43.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.781+0000 7fe40f994640 1 -- 172.21.15.125:0/207750184 >> 172.21.15.125:0/207750184 conn(0x7fe4080c7910 msgr2=0x7fe4081558c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:43.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.781+0000 7fe40f994640 1 -- 172.21.15.125:0/207750184 shutdown_connections 2023-10-30T12:40:43.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:43.781+0000 7fe40f994640 1 -- 172.21.15.125:0/207750184 wait complete. 2023-10-30T12:40:43.803 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:40:45.650 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:40:45.654 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:40:45.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:45.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.945+0000 7f8831d0f640 1 Processor -- start 2023-10-30T12:40:45.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.945+0000 7f8831d0f640 1 -- start start 2023-10-30T12:40:45.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.949+0000 7f8831d0f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f882c0c6f60 0x7f882c1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:45.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.949+0000 7f8831d0f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882c0c0a50 con 0x7f882c0c6b80 2023-10-30T12:40:45.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.949+0000 7f8831d0f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882c06f6e0 con 0x7f882c0c05d0 2023-10-30T12:40:45.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.949+0000 7f8831d0f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f882c06f9c0 con 0x7f882c0c6f60 2023-10-30T12:40:45.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.949+0000 7f882b7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f882c0c6b80 0x7f882c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36752/0 (socket says 172.21.15.125:36752) 2023-10-30T12:40:45.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.949+0000 7f882b7fe640 1 -- 172.21.15.125:0/2433651351 learned_addr learned my addr 172.21.15.125:0/2433651351 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:40:45.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.949+0000 7f882bfff640 1 --2- 172.21.15.125:0/2433651351 >> v2:172.21.15.125:3300/0 conn(0x7f882c0c6f60 0x7f882c1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.949+0000 7f882affd640 1 -- 172.21.15.125:0/2433651351 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080838175 0 0) 0x7f882c0c0a50 con 0x7f882c0c6b80 2023-10-30T12:40:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.949+0000 7f882affd640 1 -- 172.21.15.125:0/2433651351 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8818003530 con 0x7f882c0c6b80 2023-10-30T12:40:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.949+0000 7f882bfff640 1 -- 172.21.15.125:0/2433651351 >> v1:172.21.15.136:6789/0 conn(0x7f882c0c05d0 legacy=0x7f882c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:45.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.949+0000 7f882bfff640 1 -- 172.21.15.125:0/2433651351 >> v1:172.21.15.125:6789/0 conn(0x7f882c0c6b80 legacy=0x7f882c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:45.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.949+0000 7f882bfff640 1 -- 172.21.15.125:0/2433651351 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882c06fca0 con 0x7f882c0c6f60 2023-10-30T12:40:45.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.949+0000 7f882bfff640 1 --2- 172.21.15.125:0/2433651351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882c0c6f60 0x7f882c1566a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8824009d90 tx=0x7f882401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65bb686cce9d6575 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:45.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f882affd640 1 -- 172.21.15.125:0/2433651351 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88240a5920 con 0x7f882c0c6f60 2023-10-30T12:40:45.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f882affd640 1 -- 172.21.15.125:0/2433651351 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88240a5a80 con 0x7f882c0c6f60 2023-10-30T12:40:45.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f882affd640 1 -- 172.21.15.125:0/2433651351 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88240a5e10 con 0x7f882c0c6f60 2023-10-30T12:40:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 -- 172.21.15.125:0/2433651351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882c0c6f60 msgr2=0x7f882c1566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 --2- 172.21.15.125:0/2433651351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882c0c6f60 0x7f882c1566a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8824009d90 tx=0x7f882401c950 comp rx=0 tx=0).stop 2023-10-30T12:40:45.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 -- 172.21.15.125:0/2433651351 shutdown_connections 2023-10-30T12:40:45.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 --2- 172.21.15.125:0/2433651351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882c0c6f60 0x7f882c1566a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:45.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 -- 172.21.15.125:0/2433651351 >> 172.21.15.125:0/2433651351 conn(0x7f882c0c7910 msgr2=0x7f882c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:45.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 -- 172.21.15.125:0/2433651351 shutdown_connections 2023-10-30T12:40:45.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 -- 172.21.15.125:0/2433651351 wait complete. 2023-10-30T12:40:45.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 Processor -- start 2023-10-30T12:40:45.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 -- start start 2023-10-30T12:40:45.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f882c0c05d0 0x7f882c134c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:45.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882c0c6b80 0x7f882c1efbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:45.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882c0c6f60 0x7f882c1f3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:45.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c1341a0 con 0x7f882c0c6f60 2023-10-30T12:40:45.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f882c0ca7a0 con 0x7f882c0c05d0 2023-10-30T12:40:45.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f8831d0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882c1f3150 con 0x7f882c0c6b80 2023-10-30T12:40:45.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.953+0000 7f882bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882c0c6f60 0x7f882c1f3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:45.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f882bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882c0c6f60 0x7f882c1f3ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51252/0 (socket says 172.21.15.125:51252) 2023-10-30T12:40:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f882bfff640 1 -- 172.21.15.125:0/1966916644 learned_addr learned my addr 172.21.15.125:0/1966916644 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f8823fff640 1 --2- 172.21.15.125:0/1966916644 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882c0c6b80 0x7f882c1efbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f882b7fe640 1 --2- 172.21.15.125:0/1966916644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f882c0c05d0 0x7f882c134c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:45.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f882bfff640 1 -- 172.21.15.125:0/1966916644 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882c0c6b80 msgr2=0x7f882c1efbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f882bfff640 1 --2- 172.21.15.125:0/1966916644 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882c0c6b80 0x7f882c1efbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f882bfff640 1 -- 172.21.15.125:0/1966916644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f882c0c05d0 msgr2=0x7f882c134c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f882bfff640 1 --2- 172.21.15.125:0/1966916644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f882c0c05d0 0x7f882c134c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f882bfff640 1 -- 172.21.15.125:0/1966916644 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8824009a70 con 0x7f882c0c6f60 2023-10-30T12:40:45.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f882bfff640 1 --2- 172.21.15.125:0/1966916644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882c0c6f60 0x7f882c1f3ba0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f882401a580 tx=0x7f88240196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:45.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f8828ff9640 1 -- 172.21.15.125:0/1966916644 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88240ad850 con 0x7f882c0c6f60 2023-10-30T12:40:45.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f8828ff9640 1 -- 172.21.15.125:0/1966916644 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88240a5910 con 0x7f882c0c6f60 2023-10-30T12:40:45.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f8828ff9640 1 -- 172.21.15.125:0/1966916644 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88240adbb0 con 0x7f882c0c6f60 2023-10-30T12:40:45.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f8831d0f640 1 -- 172.21.15.125:0/1966916644 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f882c133ec0 con 0x7f882c0c6f60 2023-10-30T12:40:45.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f8831d0f640 1 -- 172.21.15.125:0/1966916644 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882c1f33a0 con 0x7f882c0c6f60 2023-10-30T12:40:45.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f8831d0f640 1 -- 172.21.15.125:0/1966916644 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882c0cabc0 con 0x7f882c0c6f60 2023-10-30T12:40:45.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f8828ff9640 1 -- 172.21.15.125:0/1966916644 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f88240b4050 con 0x7f882c0c6f60 2023-10-30T12:40:45.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f8828ff9640 1 --2- 172.21.15.125:0/1966916644 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f880c0664a0 0x7f880c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:45.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.957+0000 7f8828ff9640 1 -- 172.21.15.125:0/1966916644 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f882411ee70 con 0x7f882c0c6f60 2023-10-30T12:40:45.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.961+0000 7f882b7fe640 1 --2- 172.21.15.125:0/1966916644 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f880c0664a0 0x7f880c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:45.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.961+0000 7f882b7fe640 1 --2- 172.21.15.125:0/1966916644 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f880c0664a0 0x7f880c068960 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f881c0011f0 tx=0x7f881c007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:45.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:45.961+0000 7f8828ff9640 1 -- 172.21.15.125:0/1966916644 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f88240ec1f0 con 0x7f882c0c6f60 2023-10-30T12:40:46.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:46.145+0000 7f8831d0f640 1 -- 172.21.15.125:0/1966916644 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f882c04e170 con 0x7f882c0c6f60 2023-10-30T12:40:46.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:46.145+0000 7f8828ff9640 1 -- 172.21.15.125:0/1966916644 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f88240ef810 con 0x7f882c0c6f60 2023-10-30T12:40:46.147 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:40:46.147 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:39:53.073020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:46.147 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 61 2023-10-30T12:40:46.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:46.145+0000 7f8831d0f640 1 -- 172.21.15.125:0/1966916644 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f880c0664a0 msgr2=0x7f880c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:46.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:46.145+0000 7f8831d0f640 1 --2- 172.21.15.125:0/1966916644 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f880c0664a0 0x7f880c068960 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f881c0011f0 tx=0x7f881c007d40 comp rx=0 tx=0).stop 2023-10-30T12:40:46.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:46.145+0000 7f8831d0f640 1 -- 172.21.15.125:0/1966916644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882c0c6f60 msgr2=0x7f882c1f3ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:46.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:46.145+0000 7f8831d0f640 1 --2- 172.21.15.125:0/1966916644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882c0c6f60 0x7f882c1f3ba0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f882401a580 tx=0x7f88240196f0 comp rx=0 tx=0).stop 2023-10-30T12:40:46.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:46.145+0000 7f8831d0f640 1 -- 172.21.15.125:0/1966916644 shutdown_connections 2023-10-30T12:40:46.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:46.145+0000 7f8831d0f640 1 --2- 172.21.15.125:0/1966916644 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f880c0664a0 0x7f880c068960 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:46.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:46.145+0000 7f8831d0f640 1 --2- 172.21.15.125:0/1966916644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882c0c6f60 0x7f882c1f3ba0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:46.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:46.145+0000 7f8831d0f640 1 --2- 172.21.15.125:0/1966916644 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882c0c6b80 0x7f882c1efbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:46.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:46.145+0000 7f8831d0f640 1 --2- 172.21.15.125:0/1966916644 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f882c0c05d0 0x7f882c134c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:46.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:46.145+0000 7f8831d0f640 1 -- 172.21.15.125:0/1966916644 >> 172.21.15.125:0/1966916644 conn(0x7f882c0c7910 msgr2=0x7f882c0c0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:46.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:46.145+0000 7f8831d0f640 1 -- 172.21.15.125:0/1966916644 shutdown_connections 2023-10-30T12:40:46.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:46.145+0000 7f8831d0f640 1 -- 172.21.15.125:0/1966916644 wait complete. 2023-10-30T12:40:46.167 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:40:48.168 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:48.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.305+0000 7fc710f94640 1 Processor -- start 2023-10-30T12:40:48.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.305+0000 7fc710f94640 1 -- start start 2023-10-30T12:40:48.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.305+0000 7fc710f94640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc70c06f7a0 0x7fc70c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:48.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.305+0000 7fc710f94640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc70c0c6bd0 0x7fc70c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.305+0000 7fc710f94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70c071110 0x7fc70c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.305+0000 7fc710f94640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70c0700c0 con 0x7fc70c06f7a0 2023-10-30T12:40:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.305+0000 7fc710f94640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc70c06a3d0 con 0x7fc70c0c6bd0 2023-10-30T12:40:48.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.305+0000 7fc710f94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c066cf0 con 0x7fc70c071110 2023-10-30T12:40:48.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.305+0000 7fc70a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc70c06f7a0 0x7fc70c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:48.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.305+0000 7fc70a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc70c06f7a0 0x7fc70c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51268/0 (socket says 172.21.15.125:51268) 2023-10-30T12:40:48.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.305+0000 7fc70a59c640 1 -- 172.21.15.125:0/2603255390 learned_addr learned my addr 172.21.15.125:0/2603255390 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:48.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.305+0000 7fc701d9b640 1 --2- 172.21.15.125:0/2603255390 >> v2:172.21.15.136:3300/0 conn(0x7fc70c0c6bd0 0x7fc70c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:48.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.305+0000 7fc70ad9d640 1 --2- 172.21.15.125:0/2603255390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70c071110 0x7fc70c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:48.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc70a59c640 1 -- 172.21.15.125:0/2603255390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70c071110 msgr2=0x7fc70c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:48.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc70a59c640 1 --2- 172.21.15.125:0/2603255390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70c071110 0x7fc70c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:48.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc70a59c640 1 -- 172.21.15.125:0/2603255390 >> v2:172.21.15.136:3300/0 conn(0x7fc70c0c6bd0 msgr2=0x7fc70c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:48.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc70a59c640 1 --2- 172.21.15.125:0/2603255390 >> v2:172.21.15.136:3300/0 conn(0x7fc70c0c6bd0 0x7fc70c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:48.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc70a59c640 1 -- 172.21.15.125:0/2603255390 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc70c0c70d0 con 0x7fc70c06f7a0 2023-10-30T12:40:48.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc70a59c640 1 --2- 172.21.15.125:0/2603255390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70c06f7a0 0x7fc70c06fb80 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc70c06f1c0 tx=0x7fc6fc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e149ec4cf34d2cd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:48.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc709d9b640 1 -- 172.21.15.125:0/2603255390 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc0a89b0 con 0x7fc70c06f7a0 2023-10-30T12:40:48.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc709d9b640 1 -- 172.21.15.125:0/2603255390 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6fc0a8b10 con 0x7fc70c06f7a0 2023-10-30T12:40:48.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc709d9b640 1 -- 172.21.15.125:0/2603255390 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6fc0a8ea0 con 0x7fc70c06f7a0 2023-10-30T12:40:48.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc710f94640 1 -- 172.21.15.125:0/2603255390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70c06f7a0 msgr2=0x7fc70c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:48.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc710f94640 1 --2- 172.21.15.125:0/2603255390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70c06f7a0 0x7fc70c06fb80 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc70c06f1c0 tx=0x7fc6fc01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:40:48.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc710f94640 1 -- 172.21.15.125:0/2603255390 shutdown_connections 2023-10-30T12:40:48.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc710f94640 1 --2- 172.21.15.125:0/2603255390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70c071110 0x7fc70c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:48.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc710f94640 1 --2- 172.21.15.125:0/2603255390 >> v2:172.21.15.136:3300/0 conn(0x7fc70c0c6bd0 0x7fc70c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:48.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc710f94640 1 --2- 172.21.15.125:0/2603255390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70c06f7a0 0x7fc70c06fb80 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:48.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc710f94640 1 -- 172.21.15.125:0/2603255390 >> 172.21.15.125:0/2603255390 conn(0x7fc70c0c7910 msgr2=0x7fc70c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:48.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc710f94640 1 -- 172.21.15.125:0/2603255390 shutdown_connections 2023-10-30T12:40:48.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.309+0000 7fc710f94640 1 -- 172.21.15.125:0/2603255390 wait complete. 2023-10-30T12:40:48.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc710f94640 1 Processor -- start 2023-10-30T12:40:48.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc710f94640 1 -- start start 2023-10-30T12:40:48.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc710f94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70c06f7a0 0x7fc70c0adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:48.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc710f94640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70c071110 0x7fc70c146930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:48.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc710f94640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc70c0c6bd0 0x7fc70c146e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:48.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc710f94640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c148920 con 0x7fc70c071110 2023-10-30T12:40:48.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc710f94640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c1491c0 con 0x7fc70c0c6bd0 2023-10-30T12:40:48.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc710f94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70c1497b0 con 0x7fc70c06f7a0 2023-10-30T12:40:48.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc701d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70c071110 0x7fc70c146930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:48.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc701d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70c071110 0x7fc70c146930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51272/0 (socket says 172.21.15.125:51272) 2023-10-30T12:40:48.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc701d9b640 1 -- 172.21.15.125:0/3065897187 learned_addr learned my addr 172.21.15.125:0/3065897187 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:48.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc70ad9d640 1 --2- 172.21.15.125:0/3065897187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc70c0c6bd0 0x7fc70c146e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:48.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc70a59c640 1 --2- 172.21.15.125:0/3065897187 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70c06f7a0 0x7fc70c0adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:48.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc701d9b640 1 -- 172.21.15.125:0/3065897187 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70c06f7a0 msgr2=0x7fc70c0adaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:48.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc701d9b640 1 --2- 172.21.15.125:0/3065897187 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70c06f7a0 0x7fc70c0adaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:48.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc701d9b640 1 -- 172.21.15.125:0/3065897187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc70c0c6bd0 msgr2=0x7fc70c146e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:48.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc701d9b640 1 --2- 172.21.15.125:0/3065897187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc70c0c6bd0 0x7fc70c146e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:48.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.313+0000 7fc701d9b640 1 -- 172.21.15.125:0/3065897187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6fc0094d0 con 0x7fc70c071110 2023-10-30T12:40:48.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.317+0000 7fc701d9b640 1 --2- 172.21.15.125:0/3065897187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70c071110 0x7fc70c146930 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400d5d0 tx=0x7fc6f400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:48.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.317+0000 7fc7037fe640 1 -- 172.21.15.125:0/3065897187 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4004240 con 0x7fc70c071110 2023-10-30T12:40:48.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.317+0000 7fc7037fe640 1 -- 172.21.15.125:0/3065897187 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f40043a0 con 0x7fc70c071110 2023-10-30T12:40:48.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.317+0000 7fc710f94640 1 -- 172.21.15.125:0/3065897187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc70c1ef880 con 0x7fc70c071110 2023-10-30T12:40:48.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.317+0000 7fc7037fe640 1 -- 172.21.15.125:0/3065897187 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f4003840 con 0x7fc70c071110 2023-10-30T12:40:48.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.317+0000 7fc710f94640 1 -- 172.21.15.125:0/3065897187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70c148b70 con 0x7fc70c071110 2023-10-30T12:40:48.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.317+0000 7fc710f94640 1 -- 172.21.15.125:0/3065897187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70c04e170 con 0x7fc70c071110 2023-10-30T12:40:48.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.317+0000 7fc7037fe640 1 -- 172.21.15.125:0/3065897187 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6f40039a0 con 0x7fc70c071110 2023-10-30T12:40:48.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.317+0000 7fc7037fe640 1 --2- 172.21.15.125:0/3065897187 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6ec0664a0 0x7fc6ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:48.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.317+0000 7fc7037fe640 1 -- 172.21.15.125:0/3065897187 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fc6f4084180 con 0x7fc70c071110 2023-10-30T12:40:48.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.317+0000 7fc70a59c640 1 --2- 172.21.15.125:0/3065897187 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6ec0664a0 0x7fc6ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:48.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.321+0000 7fc70a59c640 1 --2- 172.21.15.125:0/3065897187 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6ec0664a0 0x7fc6ec068960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0b2000 tx=0x7fc6fc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:48.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.321+0000 7fc7037fe640 1 -- 172.21.15.125:0/3065897187 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6f4051420 con 0x7fc70c071110 2023-10-30T12:40:48.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.501+0000 7fc710f94640 1 -- 172.21.15.125:0/3065897187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc70c149330 con 0x7fc70c071110 2023-10-30T12:40:48.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.501+0000 7fc7037fe640 1 -- 172.21.15.125:0/3065897187 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fc6f4054a40 con 0x7fc70c071110 2023-10-30T12:40:48.505 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:40:48.505 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5433,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:39:53.073020+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":46,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":4973,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_4973":{"gid":4973,"name":"f","rank":0,"incarnation":45,"state":"up:active","state_seq":21,"addr":"172.21.15.136:6839/1454326577","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1454326577},{"type":"v1","addr":"172.21.15.136:6839","nonce":1454326577}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:48.505 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 61 2023-10-30T12:40:48.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.505+0000 7fc710f94640 1 -- 172.21.15.125:0/3065897187 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6ec0664a0 msgr2=0x7fc6ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:48.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.505+0000 7fc710f94640 1 --2- 172.21.15.125:0/3065897187 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6ec0664a0 0x7fc6ec068960 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fc6fc0b2000 tx=0x7fc6fc009040 comp rx=0 tx=0).stop 2023-10-30T12:40:48.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.505+0000 7fc710f94640 1 -- 172.21.15.125:0/3065897187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70c071110 msgr2=0x7fc70c146930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:48.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.505+0000 7fc710f94640 1 --2- 172.21.15.125:0/3065897187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70c071110 0x7fc70c146930 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc6f400d5d0 tx=0x7fc6f400da90 comp rx=0 tx=0).stop 2023-10-30T12:40:48.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.505+0000 7fc710f94640 1 -- 172.21.15.125:0/3065897187 shutdown_connections 2023-10-30T12:40:48.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.505+0000 7fc710f94640 1 --2- 172.21.15.125:0/3065897187 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6ec0664a0 0x7fc6ec068960 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:48.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.505+0000 7fc710f94640 1 --2- 172.21.15.125:0/3065897187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc70c0c6bd0 0x7fc70c146e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:48.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.505+0000 7fc710f94640 1 --2- 172.21.15.125:0/3065897187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70c071110 0x7fc70c146930 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:48.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.505+0000 7fc710f94640 1 --2- 172.21.15.125:0/3065897187 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70c06f7a0 0x7fc70c0adaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:48.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.505+0000 7fc710f94640 1 -- 172.21.15.125:0/3065897187 >> 172.21.15.125:0/3065897187 conn(0x7fc70c0c7910 msgr2=0x7fc70c06be40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:48.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.505+0000 7fc710f94640 1 -- 172.21.15.125:0/3065897187 shutdown_connections 2023-10-30T12:40:48.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:48.505+0000 7fc710f94640 1 -- 172.21.15.125:0/3065897187 wait complete. 2023-10-30T12:40:48.525 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:40:50.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:50.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.669+0000 7fc05333e640 1 Processor -- start 2023-10-30T12:40:50.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.669+0000 7fc05333e640 1 -- start start 2023-10-30T12:40:50.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.673+0000 7fc05333e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc04c0715c0 0x7fc04c070650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:50.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.673+0000 7fc05333e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c0c1530 con 0x7fc04c157010 2023-10-30T12:40:50.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.673+0000 7fc05333e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04c0c0f70 con 0x7fc04c0c8460 2023-10-30T12:40:50.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.673+0000 7fc05333e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc04c0c21b0 con 0x7fc04c0715c0 2023-10-30T12:40:50.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.673+0000 7fc0510da640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc04c157010 0x7fc04c071bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36756/0 (socket says 172.21.15.125:36756) 2023-10-30T12:40:50.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.673+0000 7fc0510da640 1 -- 172.21.15.125:0/914619289 learned_addr learned my addr 172.21.15.125:0/914619289 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:40:50.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.673+0000 7fc0518db640 1 --2- 172.21.15.125:0/914619289 >> v2:172.21.15.125:3300/0 conn(0x7fc04c0715c0 0x7fc04c070650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:50.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.673+0000 7fc0518db640 1 -- 172.21.15.125:0/914619289 >> v1:172.21.15.136:6789/0 conn(0x7fc04c0c8460 legacy=0x7fc04c1620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:50.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.673+0000 7fc0518db640 1 -- 172.21.15.125:0/914619289 >> v1:172.21.15.125:6789/0 conn(0x7fc04c157010 legacy=0x7fc04c071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:50.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.673+0000 7fc0518db640 1 -- 172.21.15.125:0/914619289 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c0c1250 con 0x7fc04c0715c0 2023-10-30T12:40:50.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.673+0000 7fc043fff640 1 -- 172.21.15.125:0/914619289 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705339967 0 0) 0x7fc04c0c1530 con 0x7fc04c157010 2023-10-30T12:40:50.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.677+0000 7fc0518db640 1 --2- 172.21.15.125:0/914619289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc04c0715c0 0x7fc04c070650 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc044009d90 tx=0x7fc04401c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b0385cc9c93c46 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:50.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.677+0000 7fc043fff640 1 -- 172.21.15.125:0/914619289 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0440a59e0 con 0x7fc04c0715c0 2023-10-30T12:40:50.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.677+0000 7fc043fff640 1 -- 172.21.15.125:0/914619289 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0440a5b40 con 0x7fc04c0715c0 2023-10-30T12:40:50.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.677+0000 7fc043fff640 1 -- 172.21.15.125:0/914619289 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0440ad440 con 0x7fc04c0715c0 2023-10-30T12:40:50.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.677+0000 7fc05333e640 1 -- 172.21.15.125:0/914619289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc04c0715c0 msgr2=0x7fc04c070650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:50.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.677+0000 7fc05333e640 1 --2- 172.21.15.125:0/914619289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc04c0715c0 0x7fc04c070650 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fc044009d90 tx=0x7fc04401c920 comp rx=0 tx=0).stop 2023-10-30T12:40:50.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.677+0000 7fc05333e640 1 -- 172.21.15.125:0/914619289 shutdown_connections 2023-10-30T12:40:50.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.677+0000 7fc05333e640 1 --2- 172.21.15.125:0/914619289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc04c0715c0 0x7fc04c070650 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:50.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.677+0000 7fc05333e640 1 -- 172.21.15.125:0/914619289 >> 172.21.15.125:0/914619289 conn(0x7fc04c0cbeb0 msgr2=0x7fc04c0cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:50.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.677+0000 7fc05333e640 1 -- 172.21.15.125:0/914619289 shutdown_connections 2023-10-30T12:40:50.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.677+0000 7fc05333e640 1 -- 172.21.15.125:0/914619289 wait complete. 2023-10-30T12:40:50.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.677+0000 7fc05333e640 1 Processor -- start 2023-10-30T12:40:50.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.677+0000 7fc05333e640 1 -- start start 2023-10-30T12:40:50.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.677+0000 7fc05333e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc04c0715c0 0x7fc04c14a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:50.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc05333e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc04c0c8460 0x7fc04c1465d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:50.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc05333e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc04c157010 0x7fc04c1f54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:50.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc05333e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c0caa50 con 0x7fc04c0715c0 2023-10-30T12:40:50.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc05333e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c151e00 con 0x7fc04c157010 2023-10-30T12:40:50.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc05333e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c14b6f0 con 0x7fc04c0c8460 2023-10-30T12:40:50.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc0510da640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc04c0715c0 0x7fc04c14a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:50.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc0510da640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc04c0715c0 0x7fc04c14a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51300/0 (socket says 172.21.15.125:51300) 2023-10-30T12:40:50.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc0510da640 1 -- 172.21.15.125:0/2184002211 learned_addr learned my addr 172.21.15.125:0/2184002211 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:50.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc0508d9640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc04c0c8460 0x7fc04c1465d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:50.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc0518db640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc04c157010 0x7fc04c1f54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:50.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc0510da640 1 -- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc04c0c8460 msgr2=0x7fc04c1465d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:50.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc0510da640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc04c0c8460 0x7fc04c1465d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:50.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc0510da640 1 -- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc04c157010 msgr2=0x7fc04c1f54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:50.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc0510da640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc04c157010 0x7fc04c1f54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:50.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc0510da640 1 -- 172.21.15.125:0/2184002211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc044009a70 con 0x7fc04c0715c0 2023-10-30T12:40:50.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc0508d9640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc04c0c8460 0x7fc04c1465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:50.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc0510da640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc04c0715c0 0x7fc04c14a520 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc04800c410 tx=0x7fc04800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:50.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc041ffb640 1 -- 172.21.15.125:0/2184002211 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0480074e0 con 0x7fc04c0715c0 2023-10-30T12:40:50.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc041ffb640 1 -- 172.21.15.125:0/2184002211 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc048007640 con 0x7fc04c0715c0 2023-10-30T12:40:50.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc041ffb640 1 -- 172.21.15.125:0/2184002211 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc048004a30 con 0x7fc04c0715c0 2023-10-30T12:40:50.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc05333e640 1 -- 172.21.15.125:0/2184002211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc04c14b9d0 con 0x7fc04c0715c0 2023-10-30T12:40:50.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc05333e640 1 -- 172.21.15.125:0/2184002211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc04c0ca400 con 0x7fc04c0715c0 2023-10-30T12:40:50.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc05333e640 1 -- 172.21.15.125:0/2184002211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc04c04e170 con 0x7fc04c0715c0 2023-10-30T12:40:50.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.681+0000 7fc041ffb640 1 -- 172.21.15.125:0/2184002211 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc048004b90 con 0x7fc04c0715c0 2023-10-30T12:40:50.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.685+0000 7fc041ffb640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc030066470 0x7fc030068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:50.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.685+0000 7fc041ffb640 1 -- 172.21.15.125:0/2184002211 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7fc048083f30 con 0x7fc04c0715c0 2023-10-30T12:40:50.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.685+0000 7fc0508d9640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc030066470 0x7fc030068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:50.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.685+0000 7fc0508d9640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc030066470 0x7fc030068930 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0c64b0 tx=0x7fc03c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:50.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.689+0000 7fc041ffb640 1 -- 172.21.15.125:0/2184002211 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc048051140 con 0x7fc04c0715c0 2023-10-30T12:40:50.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.869+0000 7fc05333e640 1 -- 172.21.15.125:0/2184002211 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc04c0c8240 con 0x7fc04c0715c0 2023-10-30T12:40:50.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.869+0000 7fc041ffb640 1 -- 172.21.15.125:0/2184002211 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5735 (secure 0 0 0) 0x7fc048054760 con 0x7fc04c0715c0 2023-10-30T12:40:50.873 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:40:50.873 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:40:50.129351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:50.873 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 62 2023-10-30T12:40:50.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.873+0000 7fc05333e640 1 -- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc030066470 msgr2=0x7fc030068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:50.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.873+0000 7fc05333e640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc030066470 0x7fc030068930 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fc04c0c64b0 tx=0x7fc03c009040 comp rx=0 tx=0).stop 2023-10-30T12:40:50.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.873+0000 7fc05333e640 1 -- 172.21.15.125:0/2184002211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc04c0715c0 msgr2=0x7fc04c14a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:50.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.873+0000 7fc05333e640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc04c0715c0 0x7fc04c14a520 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fc04800c410 tx=0x7fc04800c8d0 comp rx=0 tx=0).stop 2023-10-30T12:40:50.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.873+0000 7fc05333e640 1 -- 172.21.15.125:0/2184002211 shutdown_connections 2023-10-30T12:40:50.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.873+0000 7fc05333e640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc030066470 0x7fc030068930 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:50.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.873+0000 7fc05333e640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc04c157010 0x7fc04c1f54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:50.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.873+0000 7fc05333e640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc04c0c8460 0x7fc04c1465d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:50.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.873+0000 7fc05333e640 1 --2- 172.21.15.125:0/2184002211 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc04c0715c0 0x7fc04c14a520 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:50.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.873+0000 7fc05333e640 1 -- 172.21.15.125:0/2184002211 >> 172.21.15.125:0/2184002211 conn(0x7fc04c0cbeb0 msgr2=0x7fc04c0c79d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:50.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.873+0000 7fc05333e640 1 -- 172.21.15.125:0/2184002211 shutdown_connections 2023-10-30T12:40:50.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:50.873+0000 7fc05333e640 1 -- 172.21.15.125:0/2184002211 wait complete. 2023-10-30T12:40:50.894 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T12:40:50.894 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:40:50.894 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:51.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.029+0000 7fd6f249a640 1 Processor -- start 2023-10-30T12:40:51.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.029+0000 7fd6f249a640 1 -- start start 2023-10-30T12:40:51.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.029+0000 7fd6f249a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd6ec1539a0 0x7fd6ec15a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:51.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.029+0000 7fd6f249a640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ec15a8c0 con 0x7fd6ec069e90 2023-10-30T12:40:51.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.029+0000 7fd6f249a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6ec153d80 con 0x7fd6ec070a00 2023-10-30T12:40:51.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.029+0000 7fd6f249a640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6ec06fbb0 con 0x7fd6ec1539a0 2023-10-30T12:40:51.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.029+0000 7fd6ebfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd6ec1539a0 0x7fd6ec15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:51.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.029+0000 7fd6f0a37640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd6ec069e90 0x7fd6ec06a300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57936/0 (socket says 172.21.15.125:57936) 2023-10-30T12:40:51.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.029+0000 7fd6f0a37640 1 -- 172.21.15.125:0/1506275953 learned_addr learned my addr 172.21.15.125:0/1506275953 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:40:51.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.029+0000 7fd6ebfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd6ec1539a0 0x7fd6ec15a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49708/0 (socket says 172.21.15.125:49708) 2023-10-30T12:40:51.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.033+0000 7fd6ebfff640 1 -- 172.21.15.125:0/1506275953 >> v1:172.21.15.136:6789/0 conn(0x7fd6ec070a00 legacy=0x7fd6ec070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:51.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.033+0000 7fd6ebfff640 1 -- 172.21.15.125:0/1506275953 >> v1:172.21.15.125:6789/0 conn(0x7fd6ec069e90 legacy=0x7fd6ec06a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:51.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.033+0000 7fd6ebfff640 1 -- 172.21.15.125:0/1506275953 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6ec0c5c50 con 0x7fd6ec1539a0 2023-10-30T12:40:51.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.033+0000 7fd6eaffd640 1 -- 172.21.15.125:0/1506275953 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638847881 0 0) 0x7fd6ec15a8c0 con 0x7fd6ec069e90 2023-10-30T12:40:51.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.033+0000 7fd6ebfff640 1 --2- 172.21.15.125:0/1506275953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6ec1539a0 0x7fd6ec15a380 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc0097c0 tx=0x7fd6dc01c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1919482334348a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:51.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.033+0000 7fd6eaffd640 1 -- 172.21.15.125:0/1506275953 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc0ac620 con 0x7fd6ec1539a0 2023-10-30T12:40:51.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.033+0000 7fd6eaffd640 1 -- 172.21.15.125:0/1506275953 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6dc0ac780 con 0x7fd6ec1539a0 2023-10-30T12:40:51.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.033+0000 7fd6eaffd640 1 -- 172.21.15.125:0/1506275953 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc0acb10 con 0x7fd6ec1539a0 2023-10-30T12:40:51.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 -- 172.21.15.125:0/1506275953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6ec1539a0 msgr2=0x7fd6ec15a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:51.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 --2- 172.21.15.125:0/1506275953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6ec1539a0 0x7fd6ec15a380 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc0097c0 tx=0x7fd6dc01c3c0 comp rx=0 tx=0).stop 2023-10-30T12:40:51.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 -- 172.21.15.125:0/1506275953 shutdown_connections 2023-10-30T12:40:51.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 --2- 172.21.15.125:0/1506275953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6ec1539a0 0x7fd6ec15a380 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:51.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 -- 172.21.15.125:0/1506275953 >> 172.21.15.125:0/1506275953 conn(0x7fd6ec0c7910 msgr2=0x7fd6ec0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:51.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 -- 172.21.15.125:0/1506275953 shutdown_connections 2023-10-30T12:40:51.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 -- 172.21.15.125:0/1506275953 wait complete. 2023-10-30T12:40:51.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 Processor -- start 2023-10-30T12:40:51.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 -- start start 2023-10-30T12:40:51.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6ec069e90 0x7fd6ec1579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:51.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6ec070a00 0x7fd6ec144be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:51.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6ec1539a0 0x7fd6ec1fa1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:51.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ec155a40 con 0x7fd6ec1539a0 2023-10-30T12:40:51.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ec1541f0 con 0x7fd6ec069e90 2023-10-30T12:40:51.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f249a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6ec1544d0 con 0x7fd6ec070a00 2023-10-30T12:40:51.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f0a37640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6ec1539a0 0x7fd6ec1fa1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:51.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f0a37640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6ec1539a0 0x7fd6ec1fa1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49720/0 (socket says 172.21.15.125:49720) 2023-10-30T12:40:51.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f0a37640 1 -- 172.21.15.125:0/6805632 learned_addr learned my addr 172.21.15.125:0/6805632 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:51.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6ebfff640 1 --2- 172.21.15.125:0/6805632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6ec069e90 0x7fd6ec1579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:51.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6eb7fe640 1 --2- 172.21.15.125:0/6805632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6ec070a00 0x7fd6ec144be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:51.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f0a37640 1 -- 172.21.15.125:0/6805632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6ec070a00 msgr2=0x7fd6ec144be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:51.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f0a37640 1 --2- 172.21.15.125:0/6805632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6ec070a00 0x7fd6ec144be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:51.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f0a37640 1 -- 172.21.15.125:0/6805632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6ec069e90 msgr2=0x7fd6ec1579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:51.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f0a37640 1 --2- 172.21.15.125:0/6805632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6ec069e90 0x7fd6ec1579c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:51.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.037+0000 7fd6f0a37640 1 -- 172.21.15.125:0/6805632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc009420 con 0x7fd6ec1539a0 2023-10-30T12:40:51.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.041+0000 7fd6f0a37640 1 --2- 172.21.15.125:0/6805632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6ec1539a0 0x7fd6ec1fa1d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4001570 tx=0x7fd6e400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:51.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.041+0000 7fd6e8ff9640 1 -- 172.21.15.125:0/6805632 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e400a640 con 0x7fd6ec1539a0 2023-10-30T12:40:51.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.041+0000 7fd6e8ff9640 1 -- 172.21.15.125:0/6805632 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e400a7c0 con 0x7fd6ec1539a0 2023-10-30T12:40:51.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.041+0000 7fd6e8ff9640 1 -- 172.21.15.125:0/6805632 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e400ab70 con 0x7fd6ec1539a0 2023-10-30T12:40:51.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.041+0000 7fd6f249a640 1 -- 172.21.15.125:0/6805632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6ec1547b0 con 0x7fd6ec1539a0 2023-10-30T12:40:51.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.041+0000 7fd6f249a640 1 -- 172.21.15.125:0/6805632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6ec1f9cb0 con 0x7fd6ec1539a0 2023-10-30T12:40:51.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.041+0000 7fd6f249a640 1 -- 172.21.15.125:0/6805632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6ec1f9b10 con 0x7fd6ec1539a0 2023-10-30T12:40:51.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.041+0000 7fd6e8ff9640 1 -- 172.21.15.125:0/6805632 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6e400acd0 con 0x7fd6ec1539a0 2023-10-30T12:40:51.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.041+0000 7fd6e8ff9640 1 --2- 172.21.15.125:0/6805632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6d80664a0 0x7fd6d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:51.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.041+0000 7fd6e8ff9640 1 -- 172.21.15.125:0/6805632 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7fd6e4083610 con 0x7fd6ec1539a0 2023-10-30T12:40:51.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.041+0000 7fd6ebfff640 1 --2- 172.21.15.125:0/6805632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6d80664a0 0x7fd6d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:51.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.045+0000 7fd6ebfff640 1 --2- 172.21.15.125:0/6805632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6d80664a0 0x7fd6d8068960 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc0ae000 tx=0x7fd6dc0a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:51.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.045+0000 7fd6e8ff9640 1 -- 172.21.15.125:0/6805632 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6ec1f9b10 con 0x7fd6ec1539a0 2023-10-30T12:40:51.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.225+0000 7fd6f249a640 1 -- 172.21.15.125:0/6805632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6ec1f9b10 con 0x7fd6ec1539a0 2023-10-30T12:40:51.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.229+0000 7fd6e8ff9640 1 -- 172.21.15.125:0/6805632 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5735 (secure 0 0 0) 0x7fd6ec1f9b10 con 0x7fd6ec1539a0 2023-10-30T12:40:51.230 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:40:51.230 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:40:50.129351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:51.230 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 62 2023-10-30T12:40:51.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.229+0000 7fd6f249a640 1 -- 172.21.15.125:0/6805632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6d80664a0 msgr2=0x7fd6d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:51.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.229+0000 7fd6f249a640 1 --2- 172.21.15.125:0/6805632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6d80664a0 0x7fd6d8068960 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc0ae000 tx=0x7fd6dc0a7000 comp rx=0 tx=0).stop 2023-10-30T12:40:51.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.229+0000 7fd6f249a640 1 -- 172.21.15.125:0/6805632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6ec1539a0 msgr2=0x7fd6ec1fa1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:51.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.229+0000 7fd6f249a640 1 --2- 172.21.15.125:0/6805632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6ec1539a0 0x7fd6ec1fa1d0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4001570 tx=0x7fd6e400cca0 comp rx=0 tx=0).stop 2023-10-30T12:40:51.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.229+0000 7fd6f249a640 1 -- 172.21.15.125:0/6805632 shutdown_connections 2023-10-30T12:40:51.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.229+0000 7fd6f249a640 1 --2- 172.21.15.125:0/6805632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6d80664a0 0x7fd6d8068960 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:51.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.229+0000 7fd6f249a640 1 --2- 172.21.15.125:0/6805632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6ec1539a0 0x7fd6ec1fa1d0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:51.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.229+0000 7fd6f249a640 1 --2- 172.21.15.125:0/6805632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6ec070a00 0x7fd6ec144be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:51.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.229+0000 7fd6f249a640 1 --2- 172.21.15.125:0/6805632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6ec069e90 0x7fd6ec1579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:51.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.229+0000 7fd6f249a640 1 -- 172.21.15.125:0/6805632 >> 172.21.15.125:0/6805632 conn(0x7fd6ec0c7910 msgr2=0x7fd6ec14e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:51.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.229+0000 7fd6f249a640 1 -- 172.21.15.125:0/6805632 shutdown_connections 2023-10-30T12:40:51.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:51.229+0000 7fd6f249a640 1 -- 172.21.15.125:0/6805632 wait complete. 2023-10-30T12:40:51.251 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 62, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 62, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5130": { "addr": "172.21.15.125:6837/1624923396", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 1624923396, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 1624923396, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5130, "incarnation": 37, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_5433": { "addr": "172.21.15.125:6836/2595438444", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2595438444, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2595438444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5433, "incarnation": 62, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:40:50.129351+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5433, "mds_1": 5130 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:40:53.252 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:53.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.397+0000 7f32d1d0f640 1 Processor -- start 2023-10-30T12:40:53.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.397+0000 7f32d1d0f640 1 -- start start 2023-10-30T12:40:53.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.401+0000 7f32d1d0f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f32cc0c5f60 0x7f32cc071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:53.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.401+0000 7f32d1d0f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32cc0700b0 con 0x7f32cc153bf0 2023-10-30T12:40:53.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.401+0000 7f32d1d0f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32cc0c6530 con 0x7f32cc0c6fe0 2023-10-30T12:40:53.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.401+0000 7f32d1d0f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32cc0c6810 con 0x7f32cc0c5f60 2023-10-30T12:40:53.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.401+0000 7f32cb7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f32cc153bf0 0x7f32cc0c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57944/0 (socket says 172.21.15.125:57944) 2023-10-30T12:40:53.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.401+0000 7f32cb7fe640 1 -- 172.21.15.125:0/517961991 learned_addr learned my addr 172.21.15.125:0/517961991 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:40:53.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.401+0000 7f32cbfff640 1 --2- 172.21.15.125:0/517961991 >> v2:172.21.15.125:3300/0 conn(0x7f32cc0c5f60 0x7f32cc071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:53.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.401+0000 7f32cbfff640 1 -- 172.21.15.125:0/517961991 >> v1:172.21.15.136:6789/0 conn(0x7f32cc0c6fe0 legacy=0x7f32cc0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:53.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.401+0000 7f32cbfff640 1 -- 172.21.15.125:0/517961991 >> v1:172.21.15.125:6789/0 conn(0x7f32cc153bf0 legacy=0x7f32cc0c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:53.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.401+0000 7f32cbfff640 1 -- 172.21.15.125:0/517961991 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32cc159b10 con 0x7f32cc0c5f60 2023-10-30T12:40:53.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.401+0000 7f32ca7fc640 1 -- 172.21.15.125:0/517961991 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376420282 0 0) 0x7f32cc0700b0 con 0x7f32cc153bf0 2023-10-30T12:40:53.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.405+0000 7f32cbfff640 1 --2- 172.21.15.125:0/517961991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32cc0c5f60 0x7f32cc071260 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f32c4009d90 tx=0x7f32c401f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30935ccdf316f2ac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:53.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.405+0000 7f32ca7fc640 1 -- 172.21.15.125:0/517961991 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c40af600 con 0x7f32cc0c5f60 2023-10-30T12:40:53.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.405+0000 7f32ca7fc640 1 -- 172.21.15.125:0/517961991 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32c40ab050 con 0x7f32cc0c5f60 2023-10-30T12:40:53.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.405+0000 7f32ca7fc640 1 -- 172.21.15.125:0/517961991 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32c40af980 con 0x7f32cc0c5f60 2023-10-30T12:40:53.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.405+0000 7f32d1d0f640 1 -- 172.21.15.125:0/517961991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32cc0c5f60 msgr2=0x7f32cc071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:53.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.405+0000 7f32d1d0f640 1 --2- 172.21.15.125:0/517961991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32cc0c5f60 0x7f32cc071260 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f32c4009d90 tx=0x7f32c401f130 comp rx=0 tx=0).stop 2023-10-30T12:40:53.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.405+0000 7f32d1d0f640 1 -- 172.21.15.125:0/517961991 shutdown_connections 2023-10-30T12:40:53.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.405+0000 7f32d1d0f640 1 --2- 172.21.15.125:0/517961991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32cc0c5f60 0x7f32cc071260 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:53.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.405+0000 7f32d1d0f640 1 -- 172.21.15.125:0/517961991 >> 172.21.15.125:0/517961991 conn(0x7f32cc06c6e0 msgr2=0x7f32cc069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:53.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.405+0000 7f32d1d0f640 1 -- 172.21.15.125:0/517961991 shutdown_connections 2023-10-30T12:40:53.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.405+0000 7f32d1d0f640 1 -- 172.21.15.125:0/517961991 wait complete. 2023-10-30T12:40:53.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.405+0000 7f32d1d0f640 1 Processor -- start 2023-10-30T12:40:53.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.405+0000 7f32d1d0f640 1 -- start start 2023-10-30T12:40:53.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32d1d0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32cc0c5f60 0x7f32cc1583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:53.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32d1d0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32cc0c6fe0 0x7f32cc158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:53.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32d1d0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32cc153bf0 0x7f32cc0c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:53.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32d1d0f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc14a7c0 con 0x7f32cc0c5f60 2023-10-30T12:40:53.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32d1d0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc14aa20 con 0x7f32cc153bf0 2023-10-30T12:40:53.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32d1d0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc14ace0 con 0x7f32cc0c6fe0 2023-10-30T12:40:53.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32cb7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32cc0c5f60 0x7f32cc1583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:53.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32cb7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32cc0c5f60 0x7f32cc1583e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49744/0 (socket says 172.21.15.125:49744) 2023-10-30T12:40:53.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32cb7fe640 1 -- 172.21.15.125:0/1252251962 learned_addr learned my addr 172.21.15.125:0/1252251962 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:53.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32caffd640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32cc0c6fe0 0x7f32cc158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:53.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32cbfff640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32cc153bf0 0x7f32cc0c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:53.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32cb7fe640 1 -- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32cc0c6fe0 msgr2=0x7f32cc158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:53.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32cb7fe640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32cc0c6fe0 0x7f32cc158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:53.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32cb7fe640 1 -- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32cc153bf0 msgr2=0x7f32cc0c42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:53.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32cb7fe640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32cc153bf0 0x7f32cc0c42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:53.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32cb7fe640 1 -- 172.21.15.125:0/1252251962 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c4009a70 con 0x7f32cc0c5f60 2023-10-30T12:40:53.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32cbfff640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32cc153bf0 0x7f32cc0c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:40:53.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32cb7fe640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32cc0c5f60 0x7f32cc1583e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00c4a0 tx=0x7f32bc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:53.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32b3fff640 1 -- 172.21.15.125:0/1252251962 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc0074e0 con 0x7f32cc0c5f60 2023-10-30T12:40:53.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32b3fff640 1 -- 172.21.15.125:0/1252251962 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32bc007640 con 0x7f32cc0c5f60 2023-10-30T12:40:53.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32b3fff640 1 -- 172.21.15.125:0/1252251962 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32bc004a30 con 0x7f32cc0c5f60 2023-10-30T12:40:53.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32d1d0f640 1 -- 172.21.15.125:0/1252251962 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32cc0c3800 con 0x7f32cc0c5f60 2023-10-30T12:40:53.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32d1d0f640 1 -- 172.21.15.125:0/1252251962 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32cc0c3dc0 con 0x7f32cc0c5f60 2023-10-30T12:40:53.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32d1d0f640 1 -- 172.21.15.125:0/1252251962 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32cc04e170 con 0x7f32cc0c5f60 2023-10-30T12:40:53.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.409+0000 7f32b3fff640 1 -- 172.21.15.125:0/1252251962 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f32bc004b90 con 0x7f32cc0c5f60 2023-10-30T12:40:53.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.413+0000 7f32b3fff640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32b80664a0 0x7f32b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:53.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.413+0000 7f32b3fff640 1 -- 172.21.15.125:0/1252251962 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f32bc0832d0 con 0x7f32cc0c5f60 2023-10-30T12:40:53.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.413+0000 7f32caffd640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32b80664a0 0x7f32b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:53.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.413+0000 7f32caffd640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32b80664a0 0x7f32b8068960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f32cc069eb0 tx=0x7f32c0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:53.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.417+0000 7f32b3fff640 1 -- 172.21.15.125:0/1252251962 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f32bc050590 con 0x7f32cc0c5f60 2023-10-30T12:40:53.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.597+0000 7f32d1d0f640 1 -- 172.21.15.125:0/1252251962 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32cc06fe90 con 0x7f32cc0c5f60 2023-10-30T12:40:53.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.597+0000 7f32b3fff640 1 -- 172.21.15.125:0/1252251962 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f32bc018090 con 0x7f32cc0c5f60 2023-10-30T12:40:53.601 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:40:53.601 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:40:50.129351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:53.601 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 62 2023-10-30T12:40:53.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.601+0000 7f32d1d0f640 1 -- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32b80664a0 msgr2=0x7f32b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:53.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.601+0000 7f32d1d0f640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32b80664a0 0x7f32b8068960 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f32cc069eb0 tx=0x7f32c0009040 comp rx=0 tx=0).stop 2023-10-30T12:40:53.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.601+0000 7f32d1d0f640 1 -- 172.21.15.125:0/1252251962 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32cc0c5f60 msgr2=0x7f32cc1583e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:53.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.601+0000 7f32d1d0f640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32cc0c5f60 0x7f32cc1583e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00c4a0 tx=0x7f32bc00c960 comp rx=0 tx=0).stop 2023-10-30T12:40:53.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.601+0000 7f32d1d0f640 1 -- 172.21.15.125:0/1252251962 shutdown_connections 2023-10-30T12:40:53.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.601+0000 7f32d1d0f640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32b80664a0 0x7f32b8068960 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:53.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.601+0000 7f32d1d0f640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32cc153bf0 0x7f32cc0c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:53.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.601+0000 7f32d1d0f640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32cc0c6fe0 0x7f32cc158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:53.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.601+0000 7f32d1d0f640 1 --2- 172.21.15.125:0/1252251962 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32cc0c5f60 0x7f32cc1583e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:53.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.601+0000 7f32d1d0f640 1 -- 172.21.15.125:0/1252251962 >> 172.21.15.125:0/1252251962 conn(0x7f32cc06c6e0 msgr2=0x7f32cc150810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:53.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.601+0000 7f32d1d0f640 1 -- 172.21.15.125:0/1252251962 shutdown_connections 2023-10-30T12:40:53.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:53.601+0000 7f32d1d0f640 1 -- 172.21.15.125:0/1252251962 wait complete. 2023-10-30T12:40:53.622 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:40:55.623 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:55.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.765+0000 7f1d0c6d6640 1 Processor -- start 2023-10-30T12:40:55.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.765+0000 7f1d0c6d6640 1 -- start start 2023-10-30T12:40:55.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.769+0000 7f1d0c6d6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1d0406eb60 0x7f1d04069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:55.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.769+0000 7f1d0c6d6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1d0406a3d0 0x7f1d04157710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:55.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.769+0000 7f1d0c6d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d040c72d0 con 0x7f1d04151c80 2023-10-30T12:40:55.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.769+0000 7f1d0c6d6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d040c7760 con 0x7f1d0406eb60 2023-10-30T12:40:55.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.769+0000 7f1d0c6d6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d04158680 con 0x7f1d0406a3d0 2023-10-30T12:40:55.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.769+0000 7f1d09c71640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1d0406eb60 0x7f1d04069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:55.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.769+0000 7f1d09c71640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1d0406eb60 0x7f1d04069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49748/0 (socket says 172.21.15.125:49748) 2023-10-30T12:40:55.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.769+0000 7f1d09c71640 1 -- 172.21.15.125:0/242422802 learned_addr learned my addr 172.21.15.125:0/242422802 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:55.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.769+0000 7f1d0a472640 1 --1- 172.21.15.125:0/242422802 >> v1:172.21.15.125:6789/0 conn(0x7f1d04151c80 0x7f1d0406c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57954/0 (socket says 172.21.15.125:57954) 2023-10-30T12:40:55.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.769+0000 7f1d0ac73640 1 --2- 172.21.15.125:0/242422802 >> v2:172.21.15.136:3300/0 conn(0x7f1d0406a3d0 0x7f1d04157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:55.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.769+0000 7f1d09c71640 1 -- 172.21.15.125:0/242422802 >> v2:172.21.15.136:3300/0 conn(0x7f1d0406a3d0 msgr2=0x7f1d04157710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:55.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.769+0000 7f1d09c71640 1 --2- 172.21.15.125:0/242422802 >> v2:172.21.15.136:3300/0 conn(0x7f1d0406a3d0 0x7f1d04157710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:55.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.769+0000 7f1d09c71640 1 -- 172.21.15.125:0/242422802 >> v1:172.21.15.125:6789/0 conn(0x7f1d04151c80 legacy=0x7f1d0406c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:55.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.769+0000 7f1d09c71640 1 -- 172.21.15.125:0/242422802 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d040c69e0 con 0x7f1d0406eb60 2023-10-30T12:40:55.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.773+0000 7f1d09c71640 1 --2- 172.21.15.125:0/242422802 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d0406eb60 0x7f1d04069e90 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1d04072e50 tx=0x7f1cf401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81b5598f4b4260ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:55.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.773+0000 7f1d09470640 1 -- 172.21.15.125:0/242422802 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf40ac640 con 0x7f1d0406eb60 2023-10-30T12:40:55.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.773+0000 7f1d09470640 1 -- 172.21.15.125:0/242422802 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf40ac7a0 con 0x7f1d0406eb60 2023-10-30T12:40:55.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.773+0000 7f1d09470640 1 -- 172.21.15.125:0/242422802 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf40acb30 con 0x7f1d0406eb60 2023-10-30T12:40:55.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.773+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/242422802 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d0406eb60 msgr2=0x7f1d04069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:55.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.773+0000 7f1d0c6d6640 1 --2- 172.21.15.125:0/242422802 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d0406eb60 0x7f1d04069e90 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f1d04072e50 tx=0x7f1cf401c4d0 comp rx=0 tx=0).stop 2023-10-30T12:40:55.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.773+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/242422802 shutdown_connections 2023-10-30T12:40:55.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.773+0000 7f1d0c6d6640 1 --2- 172.21.15.125:0/242422802 >> v2:172.21.15.136:3300/0 conn(0x7f1d0406a3d0 0x7f1d04157710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:55.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.773+0000 7f1d0c6d6640 1 --2- 172.21.15.125:0/242422802 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d0406eb60 0x7f1d04069e90 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:55.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.773+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/242422802 >> 172.21.15.125:0/242422802 conn(0x7f1d0406dfd0 msgr2=0x7f1d0406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:55.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.773+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/242422802 shutdown_connections 2023-10-30T12:40:55.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.773+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/242422802 wait complete. 2023-10-30T12:40:55.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.773+0000 7f1d0c6d6640 1 Processor -- start 2023-10-30T12:40:55.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.773+0000 7f1d0c6d6640 1 -- start start 2023-10-30T12:40:55.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d0c6d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d0406a3d0 0x7f1d0414bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:55.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d0c6d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d0406eb60 0x7f1d04149620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:55.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d0c6d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d04151c80 0x7f1d04149b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:55.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d0c6d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d041f5820 con 0x7f1d0406eb60 2023-10-30T12:40:55.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d0c6d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0414b2c0 con 0x7f1d04151c80 2023-10-30T12:40:55.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d0c6d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d041f5a70 con 0x7f1d0406a3d0 2023-10-30T12:40:55.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d09c71640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d0406eb60 0x7f1d04149620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:55.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d09c71640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d0406eb60 0x7f1d04149620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49752/0 (socket says 172.21.15.125:49752) 2023-10-30T12:40:55.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d09c71640 1 -- 172.21.15.125:0/168691104 learned_addr learned my addr 172.21.15.125:0/168691104 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:55.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d0a472640 1 --2- 172.21.15.125:0/168691104 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d0406a3d0 0x7f1d0414bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:55.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d0ac73640 1 --2- 172.21.15.125:0/168691104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d04151c80 0x7f1d04149b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:55.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d09c71640 1 -- 172.21.15.125:0/168691104 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d0406a3d0 msgr2=0x7f1d0414bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:55.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d09c71640 1 --2- 172.21.15.125:0/168691104 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d0406a3d0 0x7f1d0414bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:55.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d09c71640 1 -- 172.21.15.125:0/168691104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d04151c80 msgr2=0x7f1d04149b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:55.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d09c71640 1 --2- 172.21.15.125:0/168691104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d04151c80 0x7f1d04149b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:55.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d09c71640 1 -- 172.21.15.125:0/168691104 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf40094d0 con 0x7f1d0406eb60 2023-10-30T12:40:55.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d09c71640 1 --2- 172.21.15.125:0/168691104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d0406eb60 0x7f1d04149620 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40190e0 tx=0x7f1cf4019270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:55.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1cfaffd640 1 -- 172.21.15.125:0/168691104 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf40b5640 con 0x7f1d0406eb60 2023-10-30T12:40:55.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1cfaffd640 1 -- 172.21.15.125:0/168691104 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf40b57a0 con 0x7f1d0406eb60 2023-10-30T12:40:55.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1cfaffd640 1 -- 172.21.15.125:0/168691104 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf40b5b10 con 0x7f1d0406eb60 2023-10-30T12:40:55.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/168691104 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d041f5d50 con 0x7f1d0406eb60 2023-10-30T12:40:55.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/168691104 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d041f6310 con 0x7f1d0406eb60 2023-10-30T12:40:55.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.777+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/168691104 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d041f6500 con 0x7f1d0406eb60 2023-10-30T12:40:55.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.781+0000 7f1cfaffd640 1 -- 172.21.15.125:0/168691104 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1cf40b3070 con 0x7f1d0406eb60 2023-10-30T12:40:55.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.781+0000 7f1cfaffd640 1 --2- 172.21.15.125:0/168691104 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ce80664a0 0x7f1ce8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:55.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.781+0000 7f1cfaffd640 1 -- 172.21.15.125:0/168691104 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f1cf411e290 con 0x7f1d0406eb60 2023-10-30T12:40:55.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.781+0000 7f1d0a472640 1 --2- 172.21.15.125:0/168691104 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ce80664a0 0x7f1ce8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:55.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.781+0000 7f1d0a472640 1 --2- 172.21.15.125:0/168691104 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ce80664a0 0x7f1ce8068960 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1d00000f70 tx=0x7f1d00007e50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:55.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:55.785+0000 7f1cfaffd640 1 -- 172.21.15.125:0/168691104 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1cf40eb600 con 0x7f1d0406eb60 2023-10-30T12:40:56.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:56.073+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/168691104 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d0404e170 con 0x7f1d0406eb60 2023-10-30T12:40:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:56.073+0000 7f1cfaffd640 1 -- 172.21.15.125:0/168691104 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+5735 (secure 0 0 0) 0x7f1cf40eec20 con 0x7f1d0406eb60 2023-10-30T12:40:56.077 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:40:56.077 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:40:50.129351+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 62 2023-10-30T12:40:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:56.077+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/168691104 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ce80664a0 msgr2=0x7f1ce8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:56.077+0000 7f1d0c6d6640 1 --2- 172.21.15.125:0/168691104 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ce80664a0 0x7f1ce8068960 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1d00000f70 tx=0x7f1d00007e50 comp rx=0 tx=0).stop 2023-10-30T12:40:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:56.077+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/168691104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d0406eb60 msgr2=0x7f1d04149620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:56.077+0000 7f1d0c6d6640 1 --2- 172.21.15.125:0/168691104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d0406eb60 0x7f1d04149620 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f1cf40190e0 tx=0x7f1cf4019270 comp rx=0 tx=0).stop 2023-10-30T12:40:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:56.077+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/168691104 shutdown_connections 2023-10-30T12:40:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:56.077+0000 7f1d0c6d6640 1 --2- 172.21.15.125:0/168691104 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ce80664a0 0x7f1ce8068960 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:56.077+0000 7f1d0c6d6640 1 --2- 172.21.15.125:0/168691104 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d04151c80 0x7f1d04149b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:56.077+0000 7f1d0c6d6640 1 --2- 172.21.15.125:0/168691104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d0406eb60 0x7f1d04149620 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:56.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:56.077+0000 7f1d0c6d6640 1 --2- 172.21.15.125:0/168691104 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d0406a3d0 0x7f1d0414bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:56.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:56.077+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/168691104 >> 172.21.15.125:0/168691104 conn(0x7f1d0406dfd0 msgr2=0x7f1d040c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:56.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:56.077+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/168691104 shutdown_connections 2023-10-30T12:40:56.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:56.077+0000 7f1d0c6d6640 1 -- 172.21.15.125:0/168691104 wait complete. 2023-10-30T12:40:56.098 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:40:58.099 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:40:58.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.233+0000 7f9b38ad6640 1 Processor -- start 2023-10-30T12:40:58.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.233+0000 7f9b38ad6640 1 -- start start 2023-10-30T12:40:58.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.233+0000 7f9b38ad6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9b340c6b80 0x7f9b340bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:58.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.233+0000 7f9b38ad6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b340c6f60 0x7f9b340c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:58.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.233+0000 7f9b38ad6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9b340c8430 0x7f9b3413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:58.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.233+0000 7f9b38ad6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b340c0a50 con 0x7f9b340c6b80 2023-10-30T12:40:58.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.233+0000 7f9b38ad6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b3406f6e0 con 0x7f9b340c8430 2023-10-30T12:40:58.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.233+0000 7f9b38ad6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3406f9c0 con 0x7f9b340c6f60 2023-10-30T12:40:58.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.237+0000 7f9b3259c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9b340c6b80 0x7f9b340bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:58.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.237+0000 7f9b3259c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9b340c6b80 0x7f9b340bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49764/0 (socket says 172.21.15.125:49764) 2023-10-30T12:40:58.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.237+0000 7f9b3259c640 1 -- 172.21.15.125:0/858852364 learned_addr learned my addr 172.21.15.125:0/858852364 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:58.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.237+0000 7f9b32d9d640 1 --2- 172.21.15.125:0/858852364 >> v2:172.21.15.136:3300/0 conn(0x7f9b340c8430 0x7f9b3413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:58.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.237+0000 7f9b29d9b640 1 --2- 172.21.15.125:0/858852364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b340c6f60 0x7f9b340c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:58.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.237+0000 7f9b3259c640 1 -- 172.21.15.125:0/858852364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b340c6f60 msgr2=0x7f9b340c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:58.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.237+0000 7f9b3259c640 1 --2- 172.21.15.125:0/858852364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b340c6f60 0x7f9b340c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:58.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.237+0000 7f9b3259c640 1 -- 172.21.15.125:0/858852364 >> v2:172.21.15.136:3300/0 conn(0x7f9b340c8430 msgr2=0x7f9b3413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:40:58.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.237+0000 7f9b3259c640 1 --2- 172.21.15.125:0/858852364 >> v2:172.21.15.136:3300/0 conn(0x7f9b340c8430 0x7f9b3413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:58.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.237+0000 7f9b3259c640 1 -- 172.21.15.125:0/858852364 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3406fca0 con 0x7f9b340c6b80 2023-10-30T12:40:58.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.237+0000 7f9b32d9d640 1 --2- 172.21.15.125:0/858852364 >> v2:172.21.15.136:3300/0 conn(0x7f9b340c8430 0x7f9b3413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:40:58.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.237+0000 7f9b3259c640 1 --2- 172.21.15.125:0/858852364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b340c6b80 0x7f9b340bfec0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9b340b0860 tx=0x7f9b2401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c041780d1378555 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:58.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b31d9b640 1 -- 172.21.15.125:0/858852364 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b240ab600 con 0x7f9b340c6b80 2023-10-30T12:40:58.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b31d9b640 1 -- 172.21.15.125:0/858852364 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b240a7050 con 0x7f9b340c6b80 2023-10-30T12:40:58.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b31d9b640 1 -- 172.21.15.125:0/858852364 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b240ab980 con 0x7f9b340c6b80 2023-10-30T12:40:58.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 -- 172.21.15.125:0/858852364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b340c6b80 msgr2=0x7f9b340bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:58.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 --2- 172.21.15.125:0/858852364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b340c6b80 0x7f9b340bfec0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9b340b0860 tx=0x7f9b2401c470 comp rx=0 tx=0).stop 2023-10-30T12:40:58.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 -- 172.21.15.125:0/858852364 shutdown_connections 2023-10-30T12:40:58.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 --2- 172.21.15.125:0/858852364 >> v2:172.21.15.136:3300/0 conn(0x7f9b340c8430 0x7f9b3413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:58.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 --2- 172.21.15.125:0/858852364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b340c6f60 0x7f9b340c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:58.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 --2- 172.21.15.125:0/858852364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b340c6b80 0x7f9b340bfec0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:58.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 -- 172.21.15.125:0/858852364 >> 172.21.15.125:0/858852364 conn(0x7f9b340c7910 msgr2=0x7f9b340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:58.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 -- 172.21.15.125:0/858852364 shutdown_connections 2023-10-30T12:40:58.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 -- 172.21.15.125:0/858852364 wait complete. 2023-10-30T12:40:58.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 Processor -- start 2023-10-30T12:40:58.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 -- start start 2023-10-30T12:40:58.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b340c6b80 0x7f9b340c9060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:58.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b340c6f60 0x7f9b341f01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:58.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b340c8430 0x7f9b34134040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:58.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b341f0700 con 0x7f9b340c8430 2023-10-30T12:40:58.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b341594a0 con 0x7f9b340c6b80 2023-10-30T12:40:58.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.241+0000 7f9b38ad6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b34159780 con 0x7f9b340c6f60 2023-10-30T12:40:58.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b32d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b340c8430 0x7f9b34134040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:58.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b32d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b340c8430 0x7f9b34134040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49780/0 (socket says 172.21.15.125:49780) 2023-10-30T12:40:58.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b32d9d640 1 -- 172.21.15.125:0/2996419947 learned_addr learned my addr 172.21.15.125:0/2996419947 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:40:58.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b3259c640 1 --2- 172.21.15.125:0/2996419947 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b340c6b80 0x7f9b340c9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:58.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b29d9b640 1 --2- 172.21.15.125:0/2996419947 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b340c6f60 0x7f9b341f01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:58.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b32d9d640 1 -- 172.21.15.125:0/2996419947 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b340c6f60 msgr2=0x7f9b341f01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:58.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b32d9d640 1 --2- 172.21.15.125:0/2996419947 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b340c6f60 0x7f9b341f01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:58.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b32d9d640 1 -- 172.21.15.125:0/2996419947 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b340c6b80 msgr2=0x7f9b340c9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:58.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b32d9d640 1 --2- 172.21.15.125:0/2996419947 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b340c6b80 0x7f9b340c9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:58.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b32d9d640 1 -- 172.21.15.125:0/2996419947 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b240094d0 con 0x7f9b340c8430 2023-10-30T12:40:58.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b32d9d640 1 --2- 172.21.15.125:0/2996419947 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b340c8430 0x7f9b34134040 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c00c7f0 tx=0x7f9b2c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:58.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b2b7fe640 1 -- 172.21.15.125:0/2996419947 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c004230 con 0x7f9b340c8430 2023-10-30T12:40:58.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b2b7fe640 1 -- 172.21.15.125:0/2996419947 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c0043b0 con 0x7f9b340c8430 2023-10-30T12:40:58.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b2b7fe640 1 -- 172.21.15.125:0/2996419947 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c003860 con 0x7f9b340c8430 2023-10-30T12:40:58.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b38ad6640 1 -- 172.21.15.125:0/2996419947 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b34135210 con 0x7f9b340c8430 2023-10-30T12:40:58.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b38ad6640 1 -- 172.21.15.125:0/2996419947 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b341357d0 con 0x7f9b340c8430 2023-10-30T12:40:58.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b38ad6640 1 -- 172.21.15.125:0/2996419947 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b340c8d80 con 0x7f9b340c8430 2023-10-30T12:40:58.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.245+0000 7f9b2b7fe640 1 -- 172.21.15.125:0/2996419947 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9b2c003bd0 con 0x7f9b340c8430 2023-10-30T12:40:58.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.249+0000 7f9b2b7fe640 1 --2- 172.21.15.125:0/2996419947 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b140664a0 0x7f9b14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:40:58.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.249+0000 7f9b2b7fe640 1 -- 172.21.15.125:0/2996419947 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f9b2c084410 con 0x7f9b340c8430 2023-10-30T12:40:58.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.249+0000 7f9b3259c640 1 --2- 172.21.15.125:0/2996419947 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b140664a0 0x7f9b14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:40:58.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.249+0000 7f9b3259c640 1 --2- 172.21.15.125:0/2996419947 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b140664a0 0x7f9b14068960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9b3406f1b0 tx=0x7f9b24008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:40:58.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.253+0000 7f9b2b7fe640 1 -- 172.21.15.125:0/2996419947 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9b340c8d80 con 0x7f9b340c8430 2023-10-30T12:40:58.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.437+0000 7f9b38ad6640 1 -- 172.21.15.125:0/2996419947 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b340c8d80 con 0x7f9b340c8430 2023-10-30T12:40:58.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.437+0000 7f9b2b7fe640 1 -- 172.21.15.125:0/2996419947 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f9b340c8d80 con 0x7f9b340c8430 2023-10-30T12:40:58.439 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:40:58.439 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:40:57.704216+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:resolve","state_seq":61,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:40:58.440 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 63 2023-10-30T12:40:58.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.437+0000 7f9b38ad6640 1 -- 172.21.15.125:0/2996419947 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b140664a0 msgr2=0x7f9b14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:58.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.437+0000 7f9b38ad6640 1 --2- 172.21.15.125:0/2996419947 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b140664a0 0x7f9b14068960 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f9b3406f1b0 tx=0x7f9b24008040 comp rx=0 tx=0).stop 2023-10-30T12:40:58.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.437+0000 7f9b38ad6640 1 -- 172.21.15.125:0/2996419947 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b340c8430 msgr2=0x7f9b34134040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:40:58.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.437+0000 7f9b38ad6640 1 --2- 172.21.15.125:0/2996419947 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b340c8430 0x7f9b34134040 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c00c7f0 tx=0x7f9b2c00ccb0 comp rx=0 tx=0).stop 2023-10-30T12:40:58.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.437+0000 7f9b38ad6640 1 -- 172.21.15.125:0/2996419947 shutdown_connections 2023-10-30T12:40:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.437+0000 7f9b38ad6640 1 --2- 172.21.15.125:0/2996419947 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b140664a0 0x7f9b14068960 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.437+0000 7f9b38ad6640 1 --2- 172.21.15.125:0/2996419947 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b340c8430 0x7f9b34134040 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.437+0000 7f9b38ad6640 1 --2- 172.21.15.125:0/2996419947 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b340c6f60 0x7f9b341f01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.437+0000 7f9b38ad6640 1 --2- 172.21.15.125:0/2996419947 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b340c6b80 0x7f9b340c9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:40:58.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.437+0000 7f9b38ad6640 1 -- 172.21.15.125:0/2996419947 >> 172.21.15.125:0/2996419947 conn(0x7f9b340c7910 msgr2=0x7f9b3406a620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:40:58.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.437+0000 7f9b38ad6640 1 -- 172.21.15.125:0/2996419947 shutdown_connections 2023-10-30T12:40:58.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:40:58.437+0000 7f9b38ad6640 1 -- 172.21.15.125:0/2996419947 wait complete. 2023-10-30T12:40:58.460 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:41:00.461 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:41:00.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcdfd3e640 1 Processor -- start 2023-10-30T12:41:00.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcdfd3e640 1 -- start start 2023-10-30T12:41:00.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcdfd3e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbcd81538a0 0x7fbcd815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:00.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcdfd3e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbcd80ca0e0 0x7fbcd815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcdfd3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbcd80cf160 0x7fbcd80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcdfd3e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd80cb800 con 0x7fbcd81538a0 2023-10-30T12:41:00.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcdfd3e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcd815dc30 con 0x7fbcd80ca0e0 2023-10-30T12:41:00.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcdfd3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd80cb330 con 0x7fbcd80cf160 2023-10-30T12:41:00.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcddada640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbcd81538a0 0x7fbcd815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:00.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcddada640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbcd81538a0 0x7fbcd815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49788/0 (socket says 172.21.15.125:49788) 2023-10-30T12:41:00.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcddada640 1 -- 172.21.15.125:0/2603785263 learned_addr learned my addr 172.21.15.125:0/2603785263 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:00.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcde2db640 1 --2- 172.21.15.125:0/2603785263 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbcd80cf160 0x7fbcd80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcdd2d9640 1 --2- 172.21.15.125:0/2603785263 >> v2:172.21.15.136:3300/0 conn(0x7fbcd80ca0e0 0x7fbcd815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcddada640 1 -- 172.21.15.125:0/2603785263 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbcd80cf160 msgr2=0x7fbcd80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcddada640 1 --2- 172.21.15.125:0/2603785263 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbcd80cf160 0x7fbcd80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcddada640 1 -- 172.21.15.125:0/2603785263 >> v2:172.21.15.136:3300/0 conn(0x7fbcd80ca0e0 msgr2=0x7fbcd815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:00.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcddada640 1 --2- 172.21.15.125:0/2603785263 >> v2:172.21.15.136:3300/0 conn(0x7fbcd80ca0e0 0x7fbcd815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:00.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.601+0000 7fbcddada640 1 -- 172.21.15.125:0/2603785263 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd80cad70 con 0x7fbcd81538a0 2023-10-30T12:41:00.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.605+0000 7fbcddada640 1 --2- 172.21.15.125:0/2603785263 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbcd81538a0 0x7fbcd815d6f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80b30a0 tx=0x7fbcd401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaaf43728630d976 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:00.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.605+0000 7fbcdcad8640 1 -- 172.21.15.125:0/2603785263 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd40a89b0 con 0x7fbcd81538a0 2023-10-30T12:41:00.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.605+0000 7fbcdcad8640 1 -- 172.21.15.125:0/2603785263 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd40a8b10 con 0x7fbcd81538a0 2023-10-30T12:41:00.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.605+0000 7fbcdcad8640 1 -- 172.21.15.125:0/2603785263 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd40a8ea0 con 0x7fbcd81538a0 2023-10-30T12:41:00.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.605+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/2603785263 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbcd81538a0 msgr2=0x7fbcd815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:00.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.605+0000 7fbcdfd3e640 1 --2- 172.21.15.125:0/2603785263 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbcd81538a0 0x7fbcd815d6f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fbcd80b30a0 tx=0x7fbcd401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:41:00.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.605+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/2603785263 shutdown_connections 2023-10-30T12:41:00.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.605+0000 7fbcdfd3e640 1 --2- 172.21.15.125:0/2603785263 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbcd80cf160 0x7fbcd80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.605+0000 7fbcdfd3e640 1 --2- 172.21.15.125:0/2603785263 >> v2:172.21.15.136:3300/0 conn(0x7fbcd80ca0e0 0x7fbcd815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.605+0000 7fbcdfd3e640 1 --2- 172.21.15.125:0/2603785263 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbcd81538a0 0x7fbcd815d6f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.605+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/2603785263 >> 172.21.15.125:0/2603785263 conn(0x7fbcd806bfe0 msgr2=0x7fbcd806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.605+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/2603785263 shutdown_connections 2023-10-30T12:41:00.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.605+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/2603785263 wait complete. 2023-10-30T12:41:00.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdfd3e640 1 Processor -- start 2023-10-30T12:41:00.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdfd3e640 1 -- start start 2023-10-30T12:41:00.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdfd3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcd80ca0e0 0x7fbcd8142d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:00.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdfd3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbcd80cf160 0x7fbcd81432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:00.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdfd3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbcd81538a0 0x7fbcd80b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:00.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdfd3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8158bb0 con 0x7fbcd80cf160 2023-10-30T12:41:00.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdfd3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd8159520 con 0x7fbcd80ca0e0 2023-10-30T12:41:00.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdfd3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcd81508e0 con 0x7fbcd81538a0 2023-10-30T12:41:00.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdd2d9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbcd80cf160 0x7fbcd81432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:00.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdd2d9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbcd80cf160 0x7fbcd81432a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49798/0 (socket says 172.21.15.125:49798) 2023-10-30T12:41:00.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdd2d9640 1 -- 172.21.15.125:0/3335843208 learned_addr learned my addr 172.21.15.125:0/3335843208 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:00.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcddada640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcd80ca0e0 0x7fbcd8142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:00.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcde2db640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbcd81538a0 0x7fbcd80b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:00.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdd2d9640 1 -- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbcd81538a0 msgr2=0x7fbcd80b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:00.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdd2d9640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbcd81538a0 0x7fbcd80b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:00.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdd2d9640 1 -- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcd80ca0e0 msgr2=0x7fbcd8142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:00.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdd2d9640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcd80ca0e0 0x7fbcd8142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:00.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdd2d9640 1 -- 172.21.15.125:0/3335843208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcd40094d0 con 0x7fbcd80cf160 2023-10-30T12:41:00.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcde2db640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbcd81538a0 0x7fbcd80b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:41:00.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdd2d9640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbcd80cf160 0x7fbcd81432a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800d800 tx=0x7fbcc800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:00.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcce7fc640 1 -- 172.21.15.125:0/3335843208 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc80096a0 con 0x7fbcd80cf160 2023-10-30T12:41:00.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcce7fc640 1 -- 172.21.15.125:0/3335843208 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc8009800 con 0x7fbcd80cf160 2023-10-30T12:41:00.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcce7fc640 1 -- 172.21.15.125:0/3335843208 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc8009b70 con 0x7fbcd80cf160 2023-10-30T12:41:00.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/3335843208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd8150bc0 con 0x7fbcd80cf160 2023-10-30T12:41:00.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.609+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/3335843208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcd8158e00 con 0x7fbcd80cf160 2023-10-30T12:41:00.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.613+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/3335843208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcd80b2a50 con 0x7fbcd80cf160 2023-10-30T12:41:00.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.613+0000 7fbcce7fc640 1 -- 172.21.15.125:0/3335843208 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbcc8009cd0 con 0x7fbcd80cf160 2023-10-30T12:41:00.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.613+0000 7fbcce7fc640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbcbc0664a0 0x7fbcbc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:00.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.613+0000 7fbcce7fc640 1 -- 172.21.15.125:0/3335843208 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7fbcc8084820 con 0x7fbcd80cf160 2023-10-30T12:41:00.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.613+0000 7fbcddada640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbcbc0664a0 0x7fbcbc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:00.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.613+0000 7fbcddada640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbcbc0664a0 0x7fbcbc068960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbcd40b2000 tx=0x7fbcd4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:00.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.617+0000 7fbcce7fc640 1 -- 172.21.15.125:0/3335843208 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbcc8051ac0 con 0x7fbcd80cf160 2023-10-30T12:41:00.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.797+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/3335843208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcd8159290 con 0x7fbcd80cf160 2023-10-30T12:41:00.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.797+0000 7fbcce7fc640 1 -- 172.21.15.125:0/3335843208 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5737 (secure 0 0 0) 0x7fbcc80550e0 con 0x7fbcd80cf160 2023-10-30T12:41:00.799 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:41:00.799 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:40:57.704216+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:resolve","state_seq":61,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:41:00.800 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 63 2023-10-30T12:41:00.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.797+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbcbc0664a0 msgr2=0x7fbcbc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:00.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.797+0000 7fbcdfd3e640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbcbc0664a0 0x7fbcbc068960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fbcd40b2000 tx=0x7fbcd4008040 comp rx=0 tx=0).stop 2023-10-30T12:41:00.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.797+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/3335843208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbcd80cf160 msgr2=0x7fbcd81432a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:00.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.797+0000 7fbcdfd3e640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbcd80cf160 0x7fbcd81432a0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fbcc800d800 tx=0x7fbcc800dcc0 comp rx=0 tx=0).stop 2023-10-30T12:41:00.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.797+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/3335843208 shutdown_connections 2023-10-30T12:41:00.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.797+0000 7fbcdfd3e640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbcbc0664a0 0x7fbcbc068960 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:00.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.797+0000 7fbcdfd3e640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbcd81538a0 0x7fbcd80b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:00.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.797+0000 7fbcdfd3e640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbcd80cf160 0x7fbcd81432a0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:00.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.797+0000 7fbcdfd3e640 1 --2- 172.21.15.125:0/3335843208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbcd80ca0e0 0x7fbcd8142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:00.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.797+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/3335843208 >> 172.21.15.125:0/3335843208 conn(0x7fbcd806bfe0 msgr2=0x7fbcd8066410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:00.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.797+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/3335843208 shutdown_connections 2023-10-30T12:41:00.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:00.797+0000 7fbcdfd3e640 1 -- 172.21.15.125:0/3335843208 wait complete. 2023-10-30T12:41:00.819 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:41:02.820 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:41:02.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb2c67640 1 Processor -- start 2023-10-30T12:41:02.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb2c67640 1 -- start start 2023-10-30T12:41:02.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb2c67640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1cac15cd50 0x7f1cac0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:02.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb2c67640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cac15a6c0 con 0x7f1cac1538f0 2023-10-30T12:41:02.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb2c67640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cac15d300 con 0x7f1cac0cefa0 2023-10-30T12:41:02.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb2c67640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cac15d5e0 con 0x7f1cac15cd50 2023-10-30T12:41:02.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb1c65640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1cac1538f0 0x7f1cac159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48480/0 (socket says 172.21.15.125:48480) 2023-10-30T12:41:02.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb1c65640 1 -- 172.21.15.125:0/2494785228 learned_addr learned my addr 172.21.15.125:0/2494785228 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:41:02.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb1464640 1 --2- 172.21.15.125:0/2494785228 >> v2:172.21.15.136:3300/0 conn(0x7f1cac15cd50 0x7f1cac0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:02.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 -- 172.21.15.125:0/2494785228 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837203478 0 0) 0x7f1cac15a6c0 con 0x7f1cac1538f0 2023-10-30T12:41:02.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 -- 172.21.15.125:0/2494785228 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c98003530 con 0x7f1cac1538f0 2023-10-30T12:41:02.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 -- 172.21.15.125:0/2494785228 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3482392831 0 0) 0x7f1c98003530 con 0x7f1cac1538f0 2023-10-30T12:41:02.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 -- 172.21.15.125:0/2494785228 >> v2:172.21.15.136:3300/0 conn(0x7f1cac15cd50 msgr2=0x7f1cac0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:02.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 --2- 172.21.15.125:0/2494785228 >> v2:172.21.15.136:3300/0 conn(0x7f1cac15cd50 0x7f1cac0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:02.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 -- 172.21.15.125:0/2494785228 >> v1:172.21.15.136:6789/0 conn(0x7f1cac0cefa0 legacy=0x7f1cac1640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:02.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 -- 172.21.15.125:0/2494785228 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac15d8c0 con 0x7f1cac1538f0 2023-10-30T12:41:02.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 -- 172.21.15.125:0/2494785228 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f1ca8001680 con 0x7f1cac1538f0 2023-10-30T12:41:02.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 -- 172.21.15.125:0/2494785228 >> v1:172.21.15.125:6789/0 conn(0x7f1cac1538f0 legacy=0x7f1cac159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:02.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 --2- 172.21.15.125:0/2494785228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c98003980 0x7f1c98003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:02.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 --2- 172.21.15.125:0/2494785228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c980047d0 0x7f1c98004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:02.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 --2- 172.21.15.125:0/2494785228 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c98005190 0x7f1c9800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:02.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 -- 172.21.15.125:0/2494785228 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac15d5e0 con 0x7f1c98003980 2023-10-30T12:41:02.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 -- 172.21.15.125:0/2494785228 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9800c040 con 0x7f1c980047d0 2023-10-30T12:41:02.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb0c63640 1 -- 172.21.15.125:0/2494785228 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9800c200 con 0x7f1c98005190 2023-10-30T12:41:02.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb1c65640 1 --2- 172.21.15.125:0/2494785228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c98003980 0x7f1c98003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:02.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb1464640 1 --2- 172.21.15.125:0/2494785228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c980047d0 0x7f1c98004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:02.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb2466640 1 --2- 172.21.15.125:0/2494785228 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c98005190 0x7f1c9800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:02.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb1c65640 1 -- 172.21.15.125:0/2494785228 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c98005190 msgr2=0x7f1c9800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:02.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb1c65640 1 --2- 172.21.15.125:0/2494785228 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c98005190 0x7f1c9800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:02.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb1c65640 1 -- 172.21.15.125:0/2494785228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c980047d0 msgr2=0x7f1c98004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:02.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb1c65640 1 --2- 172.21.15.125:0/2494785228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c980047d0 0x7f1c98004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:02.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb1c65640 1 -- 172.21.15.125:0/2494785228 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c98003530 con 0x7f1c98003980 2023-10-30T12:41:02.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb2466640 1 --2- 172.21.15.125:0/2494785228 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c98005190 0x7f1c9800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:41:02.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb1464640 1 --2- 172.21.15.125:0/2494785228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c980047d0 0x7f1c98004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:41:02.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.965+0000 7f1cb2466640 1 -- 172.21.15.125:0/2494785228 reap_dead start 2023-10-30T12:41:02.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.969+0000 7f1cb1c65640 1 --2- 172.21.15.125:0/2494785228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c98003980 0x7f1c98003da0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1ca800c240 tx=0x7f1ca801f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71b8b04be19b96dd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:02.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.969+0000 7f1cb0c63640 1 -- 172.21.15.125:0/2494785228 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8005ae0 con 0x7f1c98003980 2023-10-30T12:41:02.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.969+0000 7f1cb0c63640 1 -- 172.21.15.125:0/2494785228 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca80ab070 con 0x7f1c98003980 2023-10-30T12:41:02.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.969+0000 7f1cb0c63640 1 -- 172.21.15.125:0/2494785228 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8007070 con 0x7f1c98003980 2023-10-30T12:41:02.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.969+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2494785228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c98003980 msgr2=0x7f1c98003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:02.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.969+0000 7f1cb2c67640 1 --2- 172.21.15.125:0/2494785228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c98003980 0x7f1c98003da0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f1ca800c240 tx=0x7f1ca801f3b0 comp rx=0 tx=0).stop 2023-10-30T12:41:02.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.969+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2494785228 shutdown_connections 2023-10-30T12:41:02.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.969+0000 7f1cb2c67640 1 --2- 172.21.15.125:0/2494785228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c98003980 0x7f1c98003da0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:02.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.969+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2494785228 >> 172.21.15.125:0/2494785228 conn(0x7f1cac06a640 msgr2=0x7f1cac065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:02.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.969+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2494785228 shutdown_connections 2023-10-30T12:41:02.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.969+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2494785228 wait complete. 2023-10-30T12:41:02.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb2c67640 1 Processor -- start 2023-10-30T12:41:02.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb2c67640 1 -- start start 2023-10-30T12:41:02.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb2c67640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c98003980 0x7f1cac1f2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:02.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb2c67640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cac151e70 0x7f1cac15bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:02.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb2c67640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cac15c8a0 0x7f1cac1524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:02.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb2c67640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac151460 con 0x7f1cac151e70 2023-10-30T12:41:02.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb2c67640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac151180 con 0x7f1cac15c8a0 2023-10-30T12:41:02.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb2c67640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac148330 con 0x7f1c98003980 2023-10-30T12:41:02.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb1464640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cac151e70 0x7f1cac15bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:02.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb1464640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cac151e70 0x7f1cac15bd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55828/0 (socket says 172.21.15.125:55828) 2023-10-30T12:41:02.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb1464640 1 -- 172.21.15.125:0/2025260829 learned_addr learned my addr 172.21.15.125:0/2025260829 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:02.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb2466640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cac15c8a0 0x7f1cac1524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:02.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb1c65640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c98003980 0x7f1cac1f2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:02.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb1464640 1 -- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c98003980 msgr2=0x7f1cac1f2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:02.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb1464640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c98003980 0x7f1cac1f2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:02.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb1464640 1 -- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cac15c8a0 msgr2=0x7f1cac1524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:02.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb1464640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cac15c8a0 0x7f1cac1524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:02.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb1464640 1 -- 172.21.15.125:0/2025260829 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac1f2700 con 0x7f1cac151e70 2023-10-30T12:41:02.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb2466640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cac15c8a0 0x7f1cac1524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:41:02.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb1c65640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c98003980 0x7f1cac1f2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:41:02.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb1464640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cac151e70 0x7f1cac15bd50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c011510 tx=0x7f1c9c0119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:02.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1ca27fc640 1 -- 172.21.15.125:0/2025260829 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c004220 con 0x7f1cac151e70 2023-10-30T12:41:02.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1ca27fc640 1 -- 172.21.15.125:0/2025260829 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c9c004380 con 0x7f1cac151e70 2023-10-30T12:41:02.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2025260829 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cac1f29e0 con 0x7f1cac151e70 2023-10-30T12:41:02.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1ca27fc640 1 -- 172.21.15.125:0/2025260829 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c9c015890 con 0x7f1cac151e70 2023-10-30T12:41:02.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.973+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2025260829 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cac0b50b0 con 0x7f1cac151e70 2023-10-30T12:41:02.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.977+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2025260829 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cac1f2560 con 0x7f1cac151e70 2023-10-30T12:41:02.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.977+0000 7f1ca27fc640 1 -- 172.21.15.125:0/2025260829 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1c9c0159f0 con 0x7f1cac151e70 2023-10-30T12:41:02.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.977+0000 7f1ca27fc640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1c900664a0 0x7f1c90068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:02.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.977+0000 7f1ca27fc640 1 -- 172.21.15.125:0/2025260829 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f1c9c084570 con 0x7f1cac151e70 2023-10-30T12:41:02.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.977+0000 7f1cb1c65640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1c900664a0 0x7f1c90068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:02.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.977+0000 7f1cb1c65640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1c900664a0 0x7f1c90068960 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80af000 tx=0x7f1ca80aa000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:02.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:02.981+0000 7f1ca27fc640 1 -- 172.21.15.125:0/2025260829 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1cac1f2560 con 0x7f1cac151e70 2023-10-30T12:41:03.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:03.165+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2025260829 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cac159a70 con 0x7f1cac151e70 2023-10-30T12:41:03.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:03.165+0000 7f1ca27fc640 1 -- 172.21.15.125:0/2025260829 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f1cac159a70 con 0x7f1cac151e70 2023-10-30T12:41:03.167 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:41:03.167 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:40:57.704216+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:resolve","state_seq":61,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:41:03.168 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 63 2023-10-30T12:41:03.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:03.165+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1c900664a0 msgr2=0x7f1c90068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:03.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:03.165+0000 7f1cb2c67640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1c900664a0 0x7f1c90068960 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1ca80af000 tx=0x7f1ca80aa000 comp rx=0 tx=0).stop 2023-10-30T12:41:03.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:03.165+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2025260829 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cac151e70 msgr2=0x7f1cac15bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:03.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:03.165+0000 7f1cb2c67640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cac151e70 0x7f1cac15bd50 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c011510 tx=0x7f1c9c0119d0 comp rx=0 tx=0).stop 2023-10-30T12:41:03.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:03.165+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2025260829 shutdown_connections 2023-10-30T12:41:03.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:03.165+0000 7f1cb2c67640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1c900664a0 0x7f1c90068960 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:03.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:03.165+0000 7f1cb2c67640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cac15c8a0 0x7f1cac1524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:03.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:03.165+0000 7f1cb2c67640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cac151e70 0x7f1cac15bd50 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:03.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:03.165+0000 7f1cb2c67640 1 --2- 172.21.15.125:0/2025260829 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c98003980 0x7f1cac1f2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:03.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:03.165+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2025260829 >> 172.21.15.125:0/2025260829 conn(0x7f1cac06a640 msgr2=0x7f1cac0c4b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:03.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:03.165+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2025260829 shutdown_connections 2023-10-30T12:41:03.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:03.165+0000 7f1cb2c67640 1 -- 172.21.15.125:0/2025260829 wait complete. 2023-10-30T12:41:03.188 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 63, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 63, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5130": { "addr": "172.21.15.125:6837/1624923396", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 1624923396, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 1624923396, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5130, "incarnation": 37, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_5433": { "addr": "172.21.15.125:6836/2595438444", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2595438444, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2595438444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5433, "incarnation": 62, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:resolve", "state_seq": 61 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:40:57.704216+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5433, "mds_1": 5130 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:41:05.189 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:41:05.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.329+0000 7f6732bc5640 1 Processor -- start 2023-10-30T12:41:05.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.329+0000 7f6732bc5640 1 -- start start 2023-10-30T12:41:05.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.329+0000 7f6732bc5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f672c15d6f0 0x7f672c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:05.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.329+0000 7f6732bc5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f672c0cb800 con 0x7f672c1538a0 2023-10-30T12:41:05.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.329+0000 7f6732bc5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f672c0cb330 con 0x7f672c0cf160 2023-10-30T12:41:05.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.329+0000 7f6732bc5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f672c0cad70 con 0x7f672c15d6f0 2023-10-30T12:41:05.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f6730961640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f672c1538a0 0x7f672c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48494/0 (socket says 172.21.15.125:48494) 2023-10-30T12:41:05.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f6730961640 1 -- 172.21.15.125:0/743203210 learned_addr learned my addr 172.21.15.125:0/743203210 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:41:05.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f6723fff640 1 --2- 172.21.15.125:0/743203210 >> v2:172.21.15.136:3300/0 conn(0x7f672c15d6f0 0x7f672c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:05.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 -- 172.21.15.125:0/743203210 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860304669 0 0) 0x7f672c0cb800 con 0x7f672c1538a0 2023-10-30T12:41:05.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 -- 172.21.15.125:0/743203210 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6714003530 con 0x7f672c1538a0 2023-10-30T12:41:05.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 -- 172.21.15.125:0/743203210 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2183479982 0 0) 0x7f6714003530 con 0x7f672c1538a0 2023-10-30T12:41:05.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 -- 172.21.15.125:0/743203210 >> v2:172.21.15.136:3300/0 conn(0x7f672c15d6f0 msgr2=0x7f672c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:05.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 --2- 172.21.15.125:0/743203210 >> v2:172.21.15.136:3300/0 conn(0x7f672c15d6f0 0x7f672c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:05.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 -- 172.21.15.125:0/743203210 >> v1:172.21.15.136:6789/0 conn(0x7f672c0cf160 legacy=0x7f672c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:05.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 -- 172.21.15.125:0/743203210 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c0cb050 con 0x7f672c1538a0 2023-10-30T12:41:05.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 -- 172.21.15.125:0/743203210 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f6728001680 con 0x7f672c1538a0 2023-10-30T12:41:05.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 -- 172.21.15.125:0/743203210 >> v1:172.21.15.125:6789/0 conn(0x7f672c1538a0 legacy=0x7f672c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:05.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 --2- 172.21.15.125:0/743203210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6714003980 0x7f6714003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:05.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 --2- 172.21.15.125:0/743203210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67140047d0 0x7f6714004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:05.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 --2- 172.21.15.125:0/743203210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6714005190 0x7f671400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:05.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 -- 172.21.15.125:0/743203210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c0cad70 con 0x7f6714003980 2023-10-30T12:41:05.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 -- 172.21.15.125:0/743203210 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671400c040 con 0x7f6714005190 2023-10-30T12:41:05.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f67237fe640 1 -- 172.21.15.125:0/743203210 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f671400c200 con 0x7f67140047d0 2023-10-30T12:41:05.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f6730961640 1 --2- 172.21.15.125:0/743203210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6714003980 0x7f6714003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:05.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f6731162640 1 --2- 172.21.15.125:0/743203210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6714005190 0x7f671400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:05.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f6723fff640 1 --2- 172.21.15.125:0/743203210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67140047d0 0x7f6714004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:05.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f6730961640 1 -- 172.21.15.125:0/743203210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67140047d0 msgr2=0x7f6714004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:05.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f6730961640 1 --2- 172.21.15.125:0/743203210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67140047d0 0x7f6714004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:05.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f6730961640 1 -- 172.21.15.125:0/743203210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6714005190 msgr2=0x7f671400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:05.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f6730961640 1 --2- 172.21.15.125:0/743203210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6714005190 0x7f671400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:05.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f6730961640 1 -- 172.21.15.125:0/743203210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6714003530 con 0x7f6714003980 2023-10-30T12:41:05.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.333+0000 7f6731162640 1 -- 172.21.15.125:0/743203210 reap_dead start 2023-10-30T12:41:05.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.337+0000 7f6730961640 1 --2- 172.21.15.125:0/743203210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6714003980 0x7f6714003da0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f672800c240 tx=0x7f672801f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d1b6b1d31d568b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:05.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.337+0000 7f67237fe640 1 -- 172.21.15.125:0/743203210 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728005ae0 con 0x7f6714003980 2023-10-30T12:41:05.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.337+0000 7f67237fe640 1 -- 172.21.15.125:0/743203210 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67280ab070 con 0x7f6714003980 2023-10-30T12:41:05.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.337+0000 7f67237fe640 1 -- 172.21.15.125:0/743203210 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6728007070 con 0x7f6714003980 2023-10-30T12:41:05.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.337+0000 7f6732bc5640 1 -- 172.21.15.125:0/743203210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6714003980 msgr2=0x7f6714003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:05.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.337+0000 7f6732bc5640 1 --2- 172.21.15.125:0/743203210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6714003980 0x7f6714003da0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f672800c240 tx=0x7f672801f3b0 comp rx=0 tx=0).stop 2023-10-30T12:41:05.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.337+0000 7f6732bc5640 1 -- 172.21.15.125:0/743203210 shutdown_connections 2023-10-30T12:41:05.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.337+0000 7f6732bc5640 1 --2- 172.21.15.125:0/743203210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6714003980 0x7f6714003da0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:05.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.337+0000 7f6732bc5640 1 -- 172.21.15.125:0/743203210 >> 172.21.15.125:0/743203210 conn(0x7f672c06bfe0 msgr2=0x7f672c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:05.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.337+0000 7f6732bc5640 1 -- 172.21.15.125:0/743203210 shutdown_connections 2023-10-30T12:41:05.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.337+0000 7f6732bc5640 1 -- 172.21.15.125:0/743203210 wait complete. 2023-10-30T12:41:05.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.337+0000 7f6732bc5640 1 Processor -- start 2023-10-30T12:41:05.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.337+0000 7f6732bc5640 1 -- start start 2023-10-30T12:41:05.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6732bc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6714003980 0x7f672c1f0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:05.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6732bc5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f672c1f01b0 0x7f672c0b4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:05.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6732bc5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f672c158790 0x7f672c1f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:05.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6732bc5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c1e55b0 con 0x7f672c158790 2023-10-30T12:41:05.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6732bc5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f672c1ef4c0 con 0x7f672c1f01b0 2023-10-30T12:41:05.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6732bc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f672c1f3400 con 0x7f6714003980 2023-10-30T12:41:05.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6731162640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f672c158790 0x7f672c1f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:05.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6731162640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f672c158790 0x7f672c1f2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55838/0 (socket says 172.21.15.125:55838) 2023-10-30T12:41:05.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6731162640 1 -- 172.21.15.125:0/794479683 learned_addr learned my addr 172.21.15.125:0/794479683 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:05.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6730961640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6714003980 0x7f672c1f0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:05.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6723fff640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f672c1f01b0 0x7f672c0b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:05.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6731162640 1 -- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6714003980 msgr2=0x7f672c1f0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:05.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6731162640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6714003980 0x7f672c1f0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:05.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6731162640 1 -- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f672c1f01b0 msgr2=0x7f672c0b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:05.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6731162640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f672c1f01b0 0x7f672c0b4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:05.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6731162640 1 -- 172.21.15.125:0/794479683 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672c1f36e0 con 0x7f672c158790 2023-10-30T12:41:05.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6730961640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6714003980 0x7f672c1f0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:41:05.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6723fff640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f672c1f01b0 0x7f672c0b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:41:05.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6731162640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f672c158790 0x7f672c1f2a30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f672c06cb00 tx=0x7f672400e930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:05.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f67217fa640 1 -- 172.21.15.125:0/794479683 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724006e30 con 0x7f672c158790 2023-10-30T12:41:05.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f67217fa640 1 -- 172.21.15.125:0/794479683 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6724003980 con 0x7f672c158790 2023-10-30T12:41:05.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6732bc5640 1 -- 172.21.15.125:0/794479683 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f672c1ef6d0 con 0x7f672c158790 2023-10-30T12:41:05.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f67217fa640 1 -- 172.21.15.125:0/794479683 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724003d30 con 0x7f672c158790 2023-10-30T12:41:05.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6732bc5640 1 -- 172.21.15.125:0/794479683 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f672c1efc90 con 0x7f672c158790 2023-10-30T12:41:05.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f6732bc5640 1 -- 172.21.15.125:0/794479683 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f672c1efeb0 con 0x7f672c158790 2023-10-30T12:41:05.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.341+0000 7f67217fa640 1 -- 172.21.15.125:0/794479683 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6724005070 con 0x7f672c158790 2023-10-30T12:41:05.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.345+0000 7f67217fa640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67180664d0 0x7f6718068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:05.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.345+0000 7f67217fa640 1 -- 172.21.15.125:0/794479683 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f6724083880 con 0x7f672c158790 2023-10-30T12:41:05.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.345+0000 7f6730961640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67180664d0 0x7f6718068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:05.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.345+0000 7f6730961640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67180664d0 0x7f6718068990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f67280af000 tx=0x7f67280aa000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:05.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.349+0000 7f67217fa640 1 -- 172.21.15.125:0/794479683 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6724050a00 con 0x7f672c158790 2023-10-30T12:41:05.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.529+0000 7f6732bc5640 1 -- 172.21.15.125:0/794479683 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f672c04e170 con 0x7f672c158790 2023-10-30T12:41:05.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.529+0000 7f67217fa640 1 -- 172.21.15.125:0/794479683 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f6724054020 con 0x7f672c158790 2023-10-30T12:41:05.532 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:41:05.532 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:03.551763+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:reconnect","state_seq":63,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:41:05.532 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 64 2023-10-30T12:41:05.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.529+0000 7f6732bc5640 1 -- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67180664d0 msgr2=0x7f6718068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:05.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.529+0000 7f6732bc5640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67180664d0 0x7f6718068990 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f67280af000 tx=0x7f67280aa000 comp rx=0 tx=0).stop 2023-10-30T12:41:05.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.529+0000 7f6732bc5640 1 -- 172.21.15.125:0/794479683 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f672c158790 msgr2=0x7f672c1f2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:05.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.529+0000 7f6732bc5640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f672c158790 0x7f672c1f2a30 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f672c06cb00 tx=0x7f672400e930 comp rx=0 tx=0).stop 2023-10-30T12:41:05.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.529+0000 7f6732bc5640 1 -- 172.21.15.125:0/794479683 shutdown_connections 2023-10-30T12:41:05.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.533+0000 7f6732bc5640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67180664d0 0x7f6718068990 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:05.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.533+0000 7f6732bc5640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f672c158790 0x7f672c1f2a30 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:05.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.533+0000 7f6732bc5640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f672c1f01b0 0x7f672c0b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:05.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.533+0000 7f6732bc5640 1 --2- 172.21.15.125:0/794479683 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6714003980 0x7f672c1f0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:05.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.533+0000 7f6732bc5640 1 -- 172.21.15.125:0/794479683 >> 172.21.15.125:0/794479683 conn(0x7f672c06bfe0 msgr2=0x7f672c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:05.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.533+0000 7f6732bc5640 1 -- 172.21.15.125:0/794479683 shutdown_connections 2023-10-30T12:41:05.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:05.533+0000 7f6732bc5640 1 -- 172.21.15.125:0/794479683 wait complete. 2023-10-30T12:41:05.553 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:41:07.554 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:41:07.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a9ac9640 1 Processor -- start 2023-10-30T12:41:07.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a9ac9640 1 -- start start 2023-10-30T12:41:07.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a9ac9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f46a415d6f0 0x7f46a415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:07.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a9ac9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46a40cf160 0x7f46a40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:07.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a9ac9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a40cb800 con 0x7f46a41538a0 2023-10-30T12:41:07.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a9ac9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a40cb330 con 0x7f46a415d6f0 2023-10-30T12:41:07.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a9ac9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a40cad70 con 0x7f46a40cf160 2023-10-30T12:41:07.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a37fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f46a41538a0 0x7f46a40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48496/0 (socket says 172.21.15.125:48496) 2023-10-30T12:41:07.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a37fe640 1 -- 172.21.15.125:0/359500355 learned_addr learned my addr 172.21.15.125:0/359500355 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:41:07.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a3fff640 1 --2- 172.21.15.125:0/359500355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46a40cf160 0x7f46a40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:07.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f469bfff640 1 --2- 172.21.15.125:0/359500355 >> v2:172.21.15.136:3300/0 conn(0x7f46a415d6f0 0x7f46a415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:07.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 -- 172.21.15.125:0/359500355 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1411285956 0 0) 0x7f46a40cb800 con 0x7f46a41538a0 2023-10-30T12:41:07.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 -- 172.21.15.125:0/359500355 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4690003530 con 0x7f46a41538a0 2023-10-30T12:41:07.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 -- 172.21.15.125:0/359500355 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2371804078 0 0) 0x7f4690003530 con 0x7f46a41538a0 2023-10-30T12:41:07.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 -- 172.21.15.125:0/359500355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46a40cf160 msgr2=0x7f46a40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:07.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 --2- 172.21.15.125:0/359500355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46a40cf160 0x7f46a40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:07.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 -- 172.21.15.125:0/359500355 >> v2:172.21.15.136:3300/0 conn(0x7f46a415d6f0 msgr2=0x7f46a415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:07.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 --2- 172.21.15.125:0/359500355 >> v2:172.21.15.136:3300/0 conn(0x7f46a415d6f0 0x7f46a415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:07.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 -- 172.21.15.125:0/359500355 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a40cb050 con 0x7f46a41538a0 2023-10-30T12:41:07.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 -- 172.21.15.125:0/359500355 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f4694001680 con 0x7f46a41538a0 2023-10-30T12:41:07.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 -- 172.21.15.125:0/359500355 >> v1:172.21.15.125:6789/0 conn(0x7f46a41538a0 legacy=0x7f46a40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:07.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 --2- 172.21.15.125:0/359500355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4690003980 0x7f4690003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:07.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 --2- 172.21.15.125:0/359500355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46a40cf160 0x7f46900047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:07.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 --2- 172.21.15.125:0/359500355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4690004d10 0x7f4690005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:07.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 -- 172.21.15.125:0/359500355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a40cb330 con 0x7f4690004d10 2023-10-30T12:41:07.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 -- 172.21.15.125:0/359500355 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a40cad70 con 0x7f4690003980 2023-10-30T12:41:07.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a2ffd640 1 -- 172.21.15.125:0/359500355 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469000b990 con 0x7f46a40cf160 2023-10-30T12:41:07.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a3fff640 1 --2- 172.21.15.125:0/359500355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4690004d10 0x7f4690005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:07.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f469bfff640 1 --2- 172.21.15.125:0/359500355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46a40cf160 0x7f46900047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:07.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.697+0000 7f46a37fe640 1 --2- 172.21.15.125:0/359500355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4690003980 0x7f4690003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:07.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f46a3fff640 1 -- 172.21.15.125:0/359500355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46a40cf160 msgr2=0x7f46900047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:07.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f46a3fff640 1 --2- 172.21.15.125:0/359500355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46a40cf160 0x7f46900047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:07.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f46a3fff640 1 -- 172.21.15.125:0/359500355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4690003980 msgr2=0x7f4690003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:07.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f46a3fff640 1 --2- 172.21.15.125:0/359500355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4690003980 0x7f4690003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:07.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f46a3fff640 1 -- 172.21.15.125:0/359500355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4690003530 con 0x7f4690004d10 2023-10-30T12:41:07.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f469bfff640 1 --2- 172.21.15.125:0/359500355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46a40cf160 0x7f46900047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:41:07.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f46a3fff640 1 --2- 172.21.15.125:0/359500355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4690004d10 0x7f4690005190 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f469c009fc0 tx=0x7f469c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8209929c7dc0e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:07.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f46a2ffd640 1 -- 172.21.15.125:0/359500355 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469c0a5c70 con 0x7f4690004d10 2023-10-30T12:41:07.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f46a2ffd640 1 -- 172.21.15.125:0/359500355 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469c00f040 con 0x7f4690004d10 2023-10-30T12:41:07.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f46a2ffd640 1 -- 172.21.15.125:0/359500355 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469c0ad440 con 0x7f4690004d10 2023-10-30T12:41:07.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f46a9ac9640 1 -- 172.21.15.125:0/359500355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4690004d10 msgr2=0x7f4690005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:07.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f46a9ac9640 1 --2- 172.21.15.125:0/359500355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4690004d10 0x7f4690005190 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f469c009fc0 tx=0x7f469c01cc60 comp rx=0 tx=0).stop 2023-10-30T12:41:07.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f46a3fff640 1 -- 172.21.15.125:0/359500355 reap_dead start 2023-10-30T12:41:07.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f46a9ac9640 1 -- 172.21.15.125:0/359500355 shutdown_connections 2023-10-30T12:41:07.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.701+0000 7f46a9ac9640 1 -- 172.21.15.125:0/359500355 >> 172.21.15.125:0/359500355 conn(0x7f46a406bfe0 msgr2=0x7f46a406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:07.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a9ac9640 1 -- 172.21.15.125:0/359500355 shutdown_connections 2023-10-30T12:41:07.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a9ac9640 1 -- 172.21.15.125:0/359500355 wait complete. 2023-10-30T12:41:07.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a9ac9640 1 Processor -- start 2023-10-30T12:41:07.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a9ac9640 1 -- start start 2023-10-30T12:41:07.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a9ac9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4690004d10 0x7f46a414e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:07.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a9ac9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46a414dff0 0x7f46a40cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:07.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a9ac9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46a40cdd90 0x7f46a41f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:07.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a9ac9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a414f350 con 0x7f46a414dff0 2023-10-30T12:41:07.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a9ac9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a414d7b0 con 0x7f46a40cdd90 2023-10-30T12:41:07.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a9ac9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a40ce210 con 0x7f4690004d10 2023-10-30T12:41:07.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f469bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46a414dff0 0x7f46a40cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:07.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f469bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46a414dff0 0x7f46a40cd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55848/0 (socket says 172.21.15.125:55848) 2023-10-30T12:41:07.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f469bfff640 1 -- 172.21.15.125:0/4274629157 learned_addr learned my addr 172.21.15.125:0/4274629157 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:07.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a37fe640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4690004d10 0x7f46a414e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:07.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a3fff640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46a40cdd90 0x7f46a41f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:07.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f469bfff640 1 -- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4690004d10 msgr2=0x7f46a414e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:07.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f469bfff640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4690004d10 0x7f46a414e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:07.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f469bfff640 1 -- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46a40cdd90 msgr2=0x7f46a41f27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:07.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f469bfff640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46a40cdd90 0x7f46a41f27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:07.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f469bfff640 1 -- 172.21.15.125:0/4274629157 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a4150b30 con 0x7f46a414dff0 2023-10-30T12:41:07.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a3fff640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46a40cdd90 0x7f46a41f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:41:07.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f469bfff640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46a414dff0 0x7f46a40cd850 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f468c011930 tx=0x7f468c011df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:07.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a0ff9640 1 -- 172.21.15.125:0/4274629157 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c004220 con 0x7f46a414dff0 2023-10-30T12:41:07.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a0ff9640 1 -- 172.21.15.125:0/4274629157 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468c004380 con 0x7f46a414dff0 2023-10-30T12:41:07.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a9ac9640 1 -- 172.21.15.125:0/4274629157 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a41f00f0 con 0x7f46a414dff0 2023-10-30T12:41:07.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.705+0000 7f46a0ff9640 1 -- 172.21.15.125:0/4274629157 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c00fbd0 con 0x7f46a414dff0 2023-10-30T12:41:07.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.709+0000 7f46a9ac9640 1 -- 172.21.15.125:0/4274629157 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a4150850 con 0x7f46a414dff0 2023-10-30T12:41:07.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.709+0000 7f46a9ac9640 1 -- 172.21.15.125:0/4274629157 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a404e170 con 0x7f46a414dff0 2023-10-30T12:41:07.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.709+0000 7f46a0ff9640 1 -- 172.21.15.125:0/4274629157 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f468c00fd30 con 0x7f46a414dff0 2023-10-30T12:41:07.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.709+0000 7f46a0ff9640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46840664d0 0x7f4684068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:07.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.709+0000 7f46a0ff9640 1 -- 172.21.15.125:0/4274629157 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f468c084730 con 0x7f46a414dff0 2023-10-30T12:41:07.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.709+0000 7f46a37fe640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46840664d0 0x7f4684068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:07.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.709+0000 7f46a37fe640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46840664d0 0x7f4684068990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4694000c10 tx=0x7f4694007900 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:07.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.713+0000 7f46a0ff9640 1 -- 172.21.15.125:0/4274629157 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f468c0519f0 con 0x7f46a414dff0 2023-10-30T12:41:07.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.893+0000 7f46a9ac9640 1 -- 172.21.15.125:0/4274629157 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46a414d930 con 0x7f46a414dff0 2023-10-30T12:41:07.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.893+0000 7f46a0ff9640 1 -- 172.21.15.125:0/4274629157 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+5739 (secure 0 0 0) 0x7f468c055010 con 0x7f46a414dff0 2023-10-30T12:41:07.898 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:41:07.898 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":64,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:03.551763+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:reconnect","state_seq":63,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:41:07.899 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 64 2023-10-30T12:41:07.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.897+0000 7f46a9ac9640 1 -- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46840664d0 msgr2=0x7f4684068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:07.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.897+0000 7f46a9ac9640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46840664d0 0x7f4684068990 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f4694000c10 tx=0x7f4694007900 comp rx=0 tx=0).stop 2023-10-30T12:41:07.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.897+0000 7f46a9ac9640 1 -- 172.21.15.125:0/4274629157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46a414dff0 msgr2=0x7f46a40cd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:07.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.897+0000 7f46a9ac9640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46a414dff0 0x7f46a40cd850 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f468c011930 tx=0x7f468c011df0 comp rx=0 tx=0).stop 2023-10-30T12:41:07.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.897+0000 7f46a9ac9640 1 -- 172.21.15.125:0/4274629157 shutdown_connections 2023-10-30T12:41:07.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.897+0000 7f46a9ac9640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46840664d0 0x7f4684068990 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:07.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.897+0000 7f46a9ac9640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f46a40cdd90 0x7f46a41f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:07.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.897+0000 7f46a9ac9640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46a414dff0 0x7f46a40cd850 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:07.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.897+0000 7f46a9ac9640 1 --2- 172.21.15.125:0/4274629157 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4690004d10 0x7f46a414e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:07.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.897+0000 7f46a9ac9640 1 -- 172.21.15.125:0/4274629157 >> 172.21.15.125:0/4274629157 conn(0x7f46a406bfe0 msgr2=0x7f46a4152b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:07.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.897+0000 7f46a9ac9640 1 -- 172.21.15.125:0/4274629157 shutdown_connections 2023-10-30T12:41:07.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:07.897+0000 7f46a9ac9640 1 -- 172.21.15.125:0/4274629157 wait complete. 2023-10-30T12:41:07.919 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:41:09.920 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:41:10.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.065+0000 7f077346b640 1 Processor -- start 2023-10-30T12:41:10.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.065+0000 7f077346b640 1 -- start start 2023-10-30T12:41:10.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.069+0000 7f077346b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f076c0c05d0 0x7f076c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:10.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.069+0000 7f077346b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f076c0c6f60 0x7f076c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:10.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.069+0000 7f077346b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076c0c0a50 con 0x7f076c0c6b80 2023-10-30T12:41:10.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.069+0000 7f077346b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076c06f6e0 con 0x7f076c0c6f60 2023-10-30T12:41:10.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.069+0000 7f077346b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076c06f9c0 con 0x7f076c0c05d0 2023-10-30T12:41:10.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.069+0000 7f0771a08640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f076c0c6f60 0x7f076c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:10.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.069+0000 7f0771a08640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f076c0c6f60 0x7f076c13f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55858/0 (socket says 172.21.15.125:55858) 2023-10-30T12:41:10.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.069+0000 7f0771a08640 1 -- 172.21.15.125:0/1115051364 learned_addr learned my addr 172.21.15.125:0/1115051364 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.069+0000 7f0770a06640 1 --2- 172.21.15.125:0/1115051364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f076c0c05d0 0x7f076c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.069+0000 7f0771a08640 1 -- 172.21.15.125:0/1115051364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f076c0c05d0 msgr2=0x7f076c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.069+0000 7f0771a08640 1 --2- 172.21.15.125:0/1115051364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f076c0c05d0 0x7f076c13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.069+0000 7f0771a08640 1 -- 172.21.15.125:0/1115051364 >> v1:172.21.15.136:6789/0 conn(0x7f076c0c6b80 legacy=0x7f076c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:10.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.069+0000 7f0771a08640 1 -- 172.21.15.125:0/1115051364 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076c06fca0 con 0x7f076c0c6f60 2023-10-30T12:41:10.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.073+0000 7f0771a08640 1 --2- 172.21.15.125:0/1115051364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f076c0c6f60 0x7f076c13f2d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0764009e50 tx=0x7f076401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b1e001ad0e496b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:10.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.073+0000 7f0763fff640 1 -- 172.21.15.125:0/1115051364 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07640a57f0 con 0x7f076c0c6f60 2023-10-30T12:41:10.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.073+0000 7f0763fff640 1 -- 172.21.15.125:0/1115051364 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07640a5950 con 0x7f076c0c6f60 2023-10-30T12:41:10.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.073+0000 7f0763fff640 1 -- 172.21.15.125:0/1115051364 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07640a5ce0 con 0x7f076c0c6f60 2023-10-30T12:41:10.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.073+0000 7f077346b640 1 -- 172.21.15.125:0/1115051364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f076c0c6f60 msgr2=0x7f076c13f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:10.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.073+0000 7f077346b640 1 --2- 172.21.15.125:0/1115051364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f076c0c6f60 0x7f076c13f2d0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0764009e50 tx=0x7f076401ca10 comp rx=0 tx=0).stop 2023-10-30T12:41:10.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.073+0000 7f077346b640 1 -- 172.21.15.125:0/1115051364 shutdown_connections 2023-10-30T12:41:10.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.073+0000 7f077346b640 1 --2- 172.21.15.125:0/1115051364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f076c0c6f60 0x7f076c13f2d0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:10.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.073+0000 7f077346b640 1 --2- 172.21.15.125:0/1115051364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f076c0c05d0 0x7f076c13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:10.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.073+0000 7f077346b640 1 -- 172.21.15.125:0/1115051364 >> 172.21.15.125:0/1115051364 conn(0x7f076c0c7910 msgr2=0x7f076c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:10.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.073+0000 7f077346b640 1 -- 172.21.15.125:0/1115051364 shutdown_connections 2023-10-30T12:41:10.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.073+0000 7f077346b640 1 -- 172.21.15.125:0/1115051364 wait complete. 2023-10-30T12:41:10.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.073+0000 7f077346b640 1 Processor -- start 2023-10-30T12:41:10.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.073+0000 7f077346b640 1 -- start start 2023-10-30T12:41:10.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f077346b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f076c0c05d0 0x7f076c1501c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:10.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f077346b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f076c0c6b80 0x7f076c150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:10.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f077346b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f076c0c6f60 0x7f076c0c96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:10.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f077346b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c0c92d0 con 0x7f076c0c6f60 2023-10-30T12:41:10.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f077346b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c160df0 con 0x7f076c0c05d0 2023-10-30T12:41:10.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f077346b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f076c1eecb0 con 0x7f076c0c6b80 2023-10-30T12:41:10.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0771a08640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f076c0c6f60 0x7f076c0c96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:10.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0771a08640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f076c0c6f60 0x7f076c0c96f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55864/0 (socket says 172.21.15.125:55864) 2023-10-30T12:41:10.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0771a08640 1 -- 172.21.15.125:0/1343464364 learned_addr learned my addr 172.21.15.125:0/1343464364 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:10.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0771207640 1 --2- 172.21.15.125:0/1343464364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f076c0c05d0 0x7f076c1501c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:10.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0770a06640 1 --2- 172.21.15.125:0/1343464364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f076c0c6b80 0x7f076c150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:10.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0771a08640 1 -- 172.21.15.125:0/1343464364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f076c0c6b80 msgr2=0x7f076c150700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:10.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0771a08640 1 --2- 172.21.15.125:0/1343464364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f076c0c6b80 0x7f076c150700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:10.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0771a08640 1 -- 172.21.15.125:0/1343464364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f076c0c05d0 msgr2=0x7f076c1501c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:10.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0771a08640 1 --2- 172.21.15.125:0/1343464364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f076c0c05d0 0x7f076c1501c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:10.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0771a08640 1 -- 172.21.15.125:0/1343464364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0764009b30 con 0x7f076c0c6f60 2023-10-30T12:41:10.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0771a08640 1 --2- 172.21.15.125:0/1343464364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f076c0c6f60 0x7f076c0c96f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f07640af040 tx=0x7f07640197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:10.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0761ffb640 1 -- 172.21.15.125:0/1343464364 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07640b8880 con 0x7f076c0c6f60 2023-10-30T12:41:10.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0761ffb640 1 -- 172.21.15.125:0/1343464364 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07640b89e0 con 0x7f076c0c6f60 2023-10-30T12:41:10.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0761ffb640 1 -- 172.21.15.125:0/1343464364 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07640b8d70 con 0x7f076c0c6f60 2023-10-30T12:41:10.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f077346b640 1 -- 172.21.15.125:0/1343464364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076c134130 con 0x7f076c0c6f60 2023-10-30T12:41:10.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f077346b640 1 -- 172.21.15.125:0/1343464364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076c1343a0 con 0x7f076c0c6f60 2023-10-30T12:41:10.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f077346b640 1 -- 172.21.15.125:0/1343464364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f076c0c9410 con 0x7f076c0c6f60 2023-10-30T12:41:10.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.077+0000 7f0761ffb640 1 -- 172.21.15.125:0/1343464364 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f07640b6440 con 0x7f076c0c6f60 2023-10-30T12:41:10.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.081+0000 7f0761ffb640 1 --2- 172.21.15.125:0/1343464364 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07500664d0 0x7f0750068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:10.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.081+0000 7f0761ffb640 1 -- 172.21.15.125:0/1343464364 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f076411f8c0 con 0x7f076c0c6f60 2023-10-30T12:41:10.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.081+0000 7f0771207640 1 --2- 172.21.15.125:0/1343464364 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07500664d0 0x7f0750068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:10.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.081+0000 7f0771207640 1 --2- 172.21.15.125:0/1343464364 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07500664d0 0x7f0750068990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0768000f70 tx=0x7f0768009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:10.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.085+0000 7f0761ffb640 1 -- 172.21.15.125:0/1343464364 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f076c0c9410 con 0x7f076c0c6f60 2023-10-30T12:41:10.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.269+0000 7f077346b640 1 -- 172.21.15.125:0/1343464364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f076c0c9410 con 0x7f076c0c6f60 2023-10-30T12:41:10.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.269+0000 7f0761ffb640 1 -- 172.21.15.125:0/1343464364 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f076c0c9410 con 0x7f076c0c6f60 2023-10-30T12:41:10.273 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:41:10.273 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:09.247043+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":65,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:41:10.273 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 65 2023-10-30T12:41:10.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.273+0000 7f077346b640 1 -- 172.21.15.125:0/1343464364 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07500664d0 msgr2=0x7f0750068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:10.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.273+0000 7f077346b640 1 --2- 172.21.15.125:0/1343464364 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07500664d0 0x7f0750068990 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0768000f70 tx=0x7f0768009040 comp rx=0 tx=0).stop 2023-10-30T12:41:10.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.273+0000 7f077346b640 1 -- 172.21.15.125:0/1343464364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f076c0c6f60 msgr2=0x7f076c0c96f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:10.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.273+0000 7f077346b640 1 --2- 172.21.15.125:0/1343464364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f076c0c6f60 0x7f076c0c96f0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f07640af040 tx=0x7f07640197b0 comp rx=0 tx=0).stop 2023-10-30T12:41:10.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.273+0000 7f077346b640 1 -- 172.21.15.125:0/1343464364 shutdown_connections 2023-10-30T12:41:10.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.273+0000 7f077346b640 1 --2- 172.21.15.125:0/1343464364 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f07500664d0 0x7f0750068990 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:10.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.273+0000 7f077346b640 1 --2- 172.21.15.125:0/1343464364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f076c0c6f60 0x7f076c0c96f0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:10.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.273+0000 7f077346b640 1 --2- 172.21.15.125:0/1343464364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f076c0c6b80 0x7f076c150700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:10.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.273+0000 7f077346b640 1 --2- 172.21.15.125:0/1343464364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f076c0c05d0 0x7f076c1501c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:10.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.273+0000 7f077346b640 1 -- 172.21.15.125:0/1343464364 >> 172.21.15.125:0/1343464364 conn(0x7f076c0c7910 msgr2=0x7f076c1352c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:10.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.273+0000 7f077346b640 1 -- 172.21.15.125:0/1343464364 shutdown_connections 2023-10-30T12:41:10.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:10.273+0000 7f077346b640 1 -- 172.21.15.125:0/1343464364 wait complete. 2023-10-30T12:41:10.293 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:41:12.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:41:12.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f2272491640 1 Processor -- start 2023-10-30T12:41:12.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f2272491640 1 -- start start 2023-10-30T12:41:12.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f2272491640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226c0c6bd0 0x7f226c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:12.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f2272491640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f226c071110 0x7f226c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:12.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f2272491640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f226c066cf0 con 0x7f226c06f7a0 2023-10-30T12:41:12.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f2272491640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226c0700c0 con 0x7f226c071110 2023-10-30T12:41:12.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f2272491640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226c0c70d0 con 0x7f226c0c6bd0 2023-10-30T12:41:12.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226bfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f226c06f7a0 0x7f226c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60934/0 (socket says 172.21.15.125:60934) 2023-10-30T12:41:12.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226bfff640 1 -- 172.21.15.125:0/2488238983 learned_addr learned my addr 172.21.15.125:0/2488238983 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:41:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f2270a2e640 1 --2- 172.21.15.125:0/2488238983 >> v2:172.21.15.136:3300/0 conn(0x7f226c071110 0x7f226c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226b7fe640 1 --2- 172.21.15.125:0/2488238983 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226c0c6bd0 0x7f226c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 -- 172.21.15.125:0/2488238983 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722803522 0 0) 0x7f226c066cf0 con 0x7f226c06f7a0 2023-10-30T12:41:12.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 -- 172.21.15.125:0/2488238983 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2254003530 con 0x7f226c06f7a0 2023-10-30T12:41:12.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 -- 172.21.15.125:0/2488238983 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1186524253 0 0) 0x7f2254003530 con 0x7f226c06f7a0 2023-10-30T12:41:12.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 -- 172.21.15.125:0/2488238983 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226c0c6bd0 msgr2=0x7f226c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:12.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 --2- 172.21.15.125:0/2488238983 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226c0c6bd0 0x7f226c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:12.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 -- 172.21.15.125:0/2488238983 >> v2:172.21.15.136:3300/0 conn(0x7f226c071110 msgr2=0x7f226c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:12.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 --2- 172.21.15.125:0/2488238983 >> v2:172.21.15.136:3300/0 conn(0x7f226c071110 0x7f226c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:12.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 -- 172.21.15.125:0/2488238983 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c0c7390 con 0x7f226c06f7a0 2023-10-30T12:41:12.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 -- 172.21.15.125:0/2488238983 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f225c001680 con 0x7f226c06f7a0 2023-10-30T12:41:12.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 -- 172.21.15.125:0/2488238983 >> v1:172.21.15.125:6789/0 conn(0x7f226c06f7a0 legacy=0x7f226c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:12.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 --2- 172.21.15.125:0/2488238983 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226c0c6bd0 0x7f2254003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:12.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 --2- 172.21.15.125:0/2488238983 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2254003f00 0x7f22540047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:12.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 --2- 172.21.15.125:0/2488238983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2254004d10 0x7f2254005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:12.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 -- 172.21.15.125:0/2488238983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c0700c0 con 0x7f2254004d10 2023-10-30T12:41:12.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 -- 172.21.15.125:0/2488238983 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c0c70d0 con 0x7f2254003f00 2023-10-30T12:41:12.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226affd640 1 -- 172.21.15.125:0/2488238983 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225400b9c0 con 0x7f226c0c6bd0 2023-10-30T12:41:12.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f2270a2e640 1 --2- 172.21.15.125:0/2488238983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2254004d10 0x7f2254005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:12.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226bfff640 1 --2- 172.21.15.125:0/2488238983 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226c0c6bd0 0x7f2254003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:12.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.441+0000 7f226b7fe640 1 --2- 172.21.15.125:0/2488238983 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2254003f00 0x7f22540047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:12.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f2270a2e640 1 -- 172.21.15.125:0/2488238983 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226c0c6bd0 msgr2=0x7f2254003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:12.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f2270a2e640 1 --2- 172.21.15.125:0/2488238983 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226c0c6bd0 0x7f2254003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:12.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f2270a2e640 1 -- 172.21.15.125:0/2488238983 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2254003f00 msgr2=0x7f22540047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:12.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f2270a2e640 1 --2- 172.21.15.125:0/2488238983 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2254003f00 0x7f22540047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:12.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f2270a2e640 1 -- 172.21.15.125:0/2488238983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2254003530 con 0x7f2254004d10 2023-10-30T12:41:12.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f2270a2e640 1 --2- 172.21.15.125:0/2488238983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2254004d10 0x7f2254005190 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2264009fc0 tx=0x7f226401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fec872ea38e3e1d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:12.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f226affd640 1 -- 172.21.15.125:0/2488238983 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22640a5cb0 con 0x7f2254004d10 2023-10-30T12:41:12.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f226affd640 1 -- 172.21.15.125:0/2488238983 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226400f040 con 0x7f2254004d10 2023-10-30T12:41:12.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f226affd640 1 -- 172.21.15.125:0/2488238983 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22640ad680 con 0x7f2254004d10 2023-10-30T12:41:12.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f2272491640 1 -- 172.21.15.125:0/2488238983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2254004d10 msgr2=0x7f2254005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:12.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f2272491640 1 --2- 172.21.15.125:0/2488238983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2254004d10 0x7f2254005190 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2264009fc0 tx=0x7f226401cca0 comp rx=0 tx=0).stop 2023-10-30T12:41:12.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f2270a2e640 1 -- 172.21.15.125:0/2488238983 reap_dead start 2023-10-30T12:41:12.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f2272491640 1 -- 172.21.15.125:0/2488238983 shutdown_connections 2023-10-30T12:41:12.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f2272491640 1 -- 172.21.15.125:0/2488238983 >> 172.21.15.125:0/2488238983 conn(0x7f226c0c7910 msgr2=0x7f226c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:12.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f2272491640 1 -- 172.21.15.125:0/2488238983 shutdown_connections 2023-10-30T12:41:12.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.445+0000 7f2272491640 1 -- 172.21.15.125:0/2488238983 wait complete. 2023-10-30T12:41:12.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f2272491640 1 Processor -- start 2023-10-30T12:41:12.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f2272491640 1 -- start start 2023-10-30T12:41:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f2272491640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2254004d10 0x7f226c156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f2272491640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226c1ecab0 0x7f226c1ecef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f2272491640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226c159cc0 0x7f226c15a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f2272491640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c1588a0 con 0x7f2254004d10 2023-10-30T12:41:12.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f2272491640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c1ec3a0 con 0x7f226c1ecab0 2023-10-30T12:41:12.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f2272491640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226c152d30 con 0x7f226c159cc0 2023-10-30T12:41:12.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f226bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2254004d10 0x7f226c156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:12.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f226bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2254004d10 0x7f226c156700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41252/0 (socket says 172.21.15.125:41252) 2023-10-30T12:41:12.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f226bfff640 1 -- 172.21.15.125:0/1009101912 learned_addr learned my addr 172.21.15.125:0/1009101912 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:12.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f226b7fe640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226c1ecab0 0x7f226c1ecef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:12.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f2270a2e640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226c159cc0 0x7f226c15a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:12.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f226bfff640 1 -- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226c159cc0 msgr2=0x7f226c15a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:12.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f226bfff640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226c159cc0 0x7f226c15a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:12.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f226bfff640 1 -- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226c1ecab0 msgr2=0x7f226c1ecef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:12.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f226bfff640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226c1ecab0 0x7f226c1ecef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:12.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f226bfff640 1 -- 172.21.15.125:0/1009101912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c1521b0 con 0x7f2254004d10 2023-10-30T12:41:12.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f2270a2e640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226c159cc0 0x7f226c15a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:41:12.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f226b7fe640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226c1ecab0 0x7f226c1ecef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:41:12.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f226bfff640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2254004d10 0x7f226c156700 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f225c00e8c0 tx=0x7f225c00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:12.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f2268ff9640 1 -- 172.21.15.125:0/1009101912 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c00c8d0 con 0x7f2254004d10 2023-10-30T12:41:12.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f2268ff9640 1 -- 172.21.15.125:0/1009101912 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225c00ca50 con 0x7f2254004d10 2023-10-30T12:41:12.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f2272491640 1 -- 172.21.15.125:0/1009101912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226c152490 con 0x7f2254004d10 2023-10-30T12:41:12.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.449+0000 7f2268ff9640 1 -- 172.21.15.125:0/1009101912 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c00ce00 con 0x7f2254004d10 2023-10-30T12:41:12.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.453+0000 7f2272491640 1 -- 172.21.15.125:0/1009101912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226c152a50 con 0x7f2254004d10 2023-10-30T12:41:12.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.453+0000 7f2272491640 1 -- 172.21.15.125:0/1009101912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226c04e170 con 0x7f2254004d10 2023-10-30T12:41:12.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.453+0000 7f2268ff9640 1 -- 172.21.15.125:0/1009101912 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f225c01d460 con 0x7f2254004d10 2023-10-30T12:41:12.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.453+0000 7f2268ff9640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22580664a0 0x7f2258068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:12.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.453+0000 7f2268ff9640 1 -- 172.21.15.125:0/1009101912 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f225c084260 con 0x7f2254004d10 2023-10-30T12:41:12.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.453+0000 7f226b7fe640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22580664a0 0x7f2258068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:12.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.453+0000 7f226b7fe640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22580664a0 0x7f2258068960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f226c0c5f70 tx=0x7f2260008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:12.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.457+0000 7f2268ff9640 1 -- 172.21.15.125:0/1009101912 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f225c054e70 con 0x7f2254004d10 2023-10-30T12:41:12.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.637+0000 7f2272491640 1 -- 172.21.15.125:0/1009101912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f226c15a680 con 0x7f2254004d10 2023-10-30T12:41:12.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.637+0000 7f2268ff9640 1 -- 172.21.15.125:0/1009101912 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f225c054a00 con 0x7f2254004d10 2023-10-30T12:41:12.641 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:41:12.641 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:09.247043+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":65,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:41:12.641 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 65 2023-10-30T12:41:12.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.641+0000 7f2272491640 1 -- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22580664a0 msgr2=0x7f2258068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:12.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.641+0000 7f2272491640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22580664a0 0x7f2258068960 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f226c0c5f70 tx=0x7f2260008040 comp rx=0 tx=0).stop 2023-10-30T12:41:12.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.641+0000 7f2272491640 1 -- 172.21.15.125:0/1009101912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2254004d10 msgr2=0x7f226c156700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:12.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.641+0000 7f2272491640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2254004d10 0x7f226c156700 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f225c00e8c0 tx=0x7f225c00ed80 comp rx=0 tx=0).stop 2023-10-30T12:41:12.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.641+0000 7f2272491640 1 -- 172.21.15.125:0/1009101912 shutdown_connections 2023-10-30T12:41:12.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.641+0000 7f2272491640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22580664a0 0x7f2258068960 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:12.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.641+0000 7f2272491640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f226c159cc0 0x7f226c15a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:12.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.641+0000 7f2272491640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f226c1ecab0 0x7f226c1ecef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:12.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.641+0000 7f2272491640 1 --2- 172.21.15.125:0/1009101912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2254004d10 0x7f226c156700 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:12.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.641+0000 7f2272491640 1 -- 172.21.15.125:0/1009101912 >> 172.21.15.125:0/1009101912 conn(0x7f226c0c7910 msgr2=0x7f226c06a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:12.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.641+0000 7f2272491640 1 -- 172.21.15.125:0/1009101912 shutdown_connections 2023-10-30T12:41:12.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:12.641+0000 7f2272491640 1 -- 172.21.15.125:0/1009101912 wait complete. 2023-10-30T12:41:12.662 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:41:14.663 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:41:14.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.801+0000 7f713e054640 1 Processor -- start 2023-10-30T12:41:14.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.801+0000 7f713e054640 1 -- start start 2023-10-30T12:41:14.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.801+0000 7f713e054640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7138153bf0 0x7f7138071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:14.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.801+0000 7f713e054640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71380c5f60 0x7f71380cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:14.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.801+0000 7f713e054640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71380700b0 con 0x7f71380c6fe0 2023-10-30T12:41:14.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.801+0000 7f713e054640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71380c6810 con 0x7f7138153bf0 2023-10-30T12:41:14.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.801+0000 7f713e054640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7138159900 con 0x7f71380c5f60 2023-10-30T12:41:14.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.801+0000 7f7136ffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f71380c6fe0 0x7f71380c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60950/0 (socket says 172.21.15.125:60950) 2023-10-30T12:41:14.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.801+0000 7f7136ffd640 1 -- 172.21.15.125:0/232606756 learned_addr learned my addr 172.21.15.125:0/232606756 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:41:14.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.801+0000 7f7137fff640 1 --2- 172.21.15.125:0/232606756 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71380c5f60 0x7f71380cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:14.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.801+0000 7f71377fe640 1 --2- 172.21.15.125:0/232606756 >> v2:172.21.15.136:3300/0 conn(0x7f7138153bf0 0x7f7138071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:14.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 -- 172.21.15.125:0/232606756 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3259178170 0 0) 0x7f71380700b0 con 0x7f71380c6fe0 2023-10-30T12:41:14.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 -- 172.21.15.125:0/232606756 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7120003530 con 0x7f71380c6fe0 2023-10-30T12:41:14.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 -- 172.21.15.125:0/232606756 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2239039793 0 0) 0x7f7120003530 con 0x7f71380c6fe0 2023-10-30T12:41:14.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 -- 172.21.15.125:0/232606756 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71380c5f60 msgr2=0x7f71380cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:14.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 --2- 172.21.15.125:0/232606756 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71380c5f60 0x7f71380cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:14.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 -- 172.21.15.125:0/232606756 >> v2:172.21.15.136:3300/0 conn(0x7f7138153bf0 msgr2=0x7f7138071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:14.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 --2- 172.21.15.125:0/232606756 >> v2:172.21.15.136:3300/0 conn(0x7f7138153bf0 0x7f7138071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:14.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 -- 172.21.15.125:0/232606756 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71380c6530 con 0x7f71380c6fe0 2023-10-30T12:41:14.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 -- 172.21.15.125:0/232606756 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f712c001680 con 0x7f71380c6fe0 2023-10-30T12:41:14.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 -- 172.21.15.125:0/232606756 >> v1:172.21.15.125:6789/0 conn(0x7f71380c6fe0 legacy=0x7f71380c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:14.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 --2- 172.21.15.125:0/232606756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7120003980 0x7f7120003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:14.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 --2- 172.21.15.125:0/232606756 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71200047d0 0x7f7120004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:14.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 --2- 172.21.15.125:0/232606756 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71380c5f60 0x7f7120005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:14.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 -- 172.21.15.125:0/232606756 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71380c6810 con 0x7f71200047d0 2023-10-30T12:41:14.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 -- 172.21.15.125:0/232606756 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7138159900 con 0x7f7120003980 2023-10-30T12:41:14.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71367fc640 1 -- 172.21.15.125:0/232606756 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f712000b990 con 0x7f71380c5f60 2023-10-30T12:41:14.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f7136ffd640 1 --2- 172.21.15.125:0/232606756 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71200047d0 0x7f7120004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:14.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71377fe640 1 --2- 172.21.15.125:0/232606756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7120003980 0x7f7120003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:14.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f7137fff640 1 --2- 172.21.15.125:0/232606756 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71380c5f60 0x7f7120005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:14.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f7136ffd640 1 -- 172.21.15.125:0/232606756 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71380c5f60 msgr2=0x7f7120005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:14.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f7136ffd640 1 --2- 172.21.15.125:0/232606756 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71380c5f60 0x7f7120005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:14.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f7136ffd640 1 -- 172.21.15.125:0/232606756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7120003980 msgr2=0x7f7120003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:14.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f7136ffd640 1 --2- 172.21.15.125:0/232606756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7120003980 0x7f7120003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:14.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f7136ffd640 1 -- 172.21.15.125:0/232606756 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7120003530 con 0x7f71200047d0 2023-10-30T12:41:14.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.805+0000 7f71377fe640 1 --2- 172.21.15.125:0/232606756 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7120003980 0x7f7120003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:41:14.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f7136ffd640 1 --2- 172.21.15.125:0/232606756 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71200047d0 0x7f7120004c50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f712c00c240 tx=0x7f712c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16eb43c603b04265 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:14.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f71367fc640 1 -- 172.21.15.125:0/232606756 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c0058e0 con 0x7f71200047d0 2023-10-30T12:41:14.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f71367fc640 1 -- 172.21.15.125:0/232606756 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712c005a40 con 0x7f71200047d0 2023-10-30T12:41:14.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f71367fc640 1 -- 172.21.15.125:0/232606756 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712c005e10 con 0x7f71200047d0 2023-10-30T12:41:14.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f713e054640 1 -- 172.21.15.125:0/232606756 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71200047d0 msgr2=0x7f7120004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:14.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f713e054640 1 --2- 172.21.15.125:0/232606756 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71200047d0 0x7f7120004c50 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f712c00c240 tx=0x7f712c01cc60 comp rx=0 tx=0).stop 2023-10-30T12:41:14.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f7137fff640 1 -- 172.21.15.125:0/232606756 reap_dead start 2023-10-30T12:41:14.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f713e054640 1 -- 172.21.15.125:0/232606756 shutdown_connections 2023-10-30T12:41:14.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f713e054640 1 -- 172.21.15.125:0/232606756 >> 172.21.15.125:0/232606756 conn(0x7f713806c6e0 msgr2=0x7f7138069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:14.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f713e054640 1 -- 172.21.15.125:0/232606756 shutdown_connections 2023-10-30T12:41:14.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f713e054640 1 -- 172.21.15.125:0/232606756 wait complete. 2023-10-30T12:41:14.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f713e054640 1 Processor -- start 2023-10-30T12:41:14.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f713e054640 1 -- start start 2023-10-30T12:41:14.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f713e054640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71200047d0 0x7f7138156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:14.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f713e054640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71381526d0 0x7f713814c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:14.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f713e054640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71380ca600 0x7f71380caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:14.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f713e054640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713814faa0 con 0x7f71200047d0 2023-10-30T12:41:14.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f713e054640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713814e860 con 0x7f71380ca600 2023-10-30T12:41:14.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.809+0000 7f713e054640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7138157280 con 0x7f71381526d0 2023-10-30T12:41:14.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f71377fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71200047d0 0x7f7138156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:14.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f71377fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71200047d0 0x7f7138156d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41268/0 (socket says 172.21.15.125:41268) 2023-10-30T12:41:14.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f71377fe640 1 -- 172.21.15.125:0/2268851179 learned_addr learned my addr 172.21.15.125:0/2268851179 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:14.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f7136ffd640 1 --2- 172.21.15.125:0/2268851179 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71381526d0 0x7f713814c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:14.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f7137fff640 1 --2- 172.21.15.125:0/2268851179 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71380ca600 0x7f71380caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:14.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f71377fe640 1 -- 172.21.15.125:0/2268851179 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71381526d0 msgr2=0x7f713814c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:14.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f71377fe640 1 --2- 172.21.15.125:0/2268851179 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71381526d0 0x7f713814c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:14.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f71377fe640 1 -- 172.21.15.125:0/2268851179 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71380ca600 msgr2=0x7f71380caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:14.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f71377fe640 1 --2- 172.21.15.125:0/2268851179 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71380ca600 0x7f71380caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:14.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f71377fe640 1 -- 172.21.15.125:0/2268851179 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7138158c60 con 0x7f71200047d0 2023-10-30T12:41:14.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f71377fe640 1 --2- 172.21.15.125:0/2268851179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71200047d0 0x7f7138156d40 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f71280115f0 tx=0x7f7128011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:14.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f711ffff640 1 -- 172.21.15.125:0/2268851179 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7128004220 con 0x7f71200047d0 2023-10-30T12:41:14.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f711ffff640 1 -- 172.21.15.125:0/2268851179 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7128004380 con 0x7f71200047d0 2023-10-30T12:41:14.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f711ffff640 1 -- 172.21.15.125:0/2268851179 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f712800f910 con 0x7f71200047d0 2023-10-30T12:41:14.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f713e054640 1 -- 172.21.15.125:0/2268851179 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f713814f520 con 0x7f71200047d0 2023-10-30T12:41:14.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f713e054640 1 -- 172.21.15.125:0/2268851179 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f713814ea50 con 0x7f71200047d0 2023-10-30T12:41:14.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f713e054640 1 -- 172.21.15.125:0/2268851179 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f713804e170 con 0x7f71200047d0 2023-10-30T12:41:14.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f711ffff640 1 -- 172.21.15.125:0/2268851179 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f712800fa70 con 0x7f71200047d0 2023-10-30T12:41:14.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f711ffff640 1 --2- 172.21.15.125:0/2268851179 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71240664a0 0x7f7124068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:14.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.813+0000 7f711ffff640 1 -- 172.21.15.125:0/2268851179 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f71280845a0 con 0x7f71200047d0 2023-10-30T12:41:14.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.817+0000 7f7136ffd640 1 --2- 172.21.15.125:0/2268851179 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71240664a0 0x7f7124068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:14.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.817+0000 7f7136ffd640 1 --2- 172.21.15.125:0/2268851179 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71240664a0 0x7f7124068960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f712c0aa000 tx=0x7f712c001a80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:14.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.821+0000 7f711ffff640 1 -- 172.21.15.125:0/2268851179 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7128051850 con 0x7f71200047d0 2023-10-30T12:41:15.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:14.997+0000 7f713e054640 1 -- 172.21.15.125:0/2268851179 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7138150c50 con 0x7f71200047d0 2023-10-30T12:41:15.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:15.001+0000 7f711ffff640 1 -- 172.21.15.125:0/2268851179 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f7128054e70 con 0x7f71200047d0 2023-10-30T12:41:15.002 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:41:15.003 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:09.247043+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:rejoin","state_seq":65,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:41:15.004 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 65 2023-10-30T12:41:15.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:15.001+0000 7f713e054640 1 -- 172.21.15.125:0/2268851179 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71240664a0 msgr2=0x7f7124068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:15.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:15.001+0000 7f713e054640 1 --2- 172.21.15.125:0/2268851179 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71240664a0 0x7f7124068960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f712c0aa000 tx=0x7f712c001a80 comp rx=0 tx=0).stop 2023-10-30T12:41:15.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:15.001+0000 7f713e054640 1 -- 172.21.15.125:0/2268851179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71200047d0 msgr2=0x7f7138156d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:15.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:15.001+0000 7f713e054640 1 --2- 172.21.15.125:0/2268851179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71200047d0 0x7f7138156d40 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f71280115f0 tx=0x7f7128011ab0 comp rx=0 tx=0).stop 2023-10-30T12:41:15.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:15.001+0000 7f713e054640 1 -- 172.21.15.125:0/2268851179 shutdown_connections 2023-10-30T12:41:15.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:15.001+0000 7f713e054640 1 --2- 172.21.15.125:0/2268851179 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71240664a0 0x7f7124068960 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:15.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:15.001+0000 7f713e054640 1 --2- 172.21.15.125:0/2268851179 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f71380ca600 0x7f71380caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:15.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:15.001+0000 7f713e054640 1 --2- 172.21.15.125:0/2268851179 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71381526d0 0x7f713814c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:15.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:15.001+0000 7f713e054640 1 --2- 172.21.15.125:0/2268851179 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71200047d0 0x7f7138156d40 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:15.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:15.001+0000 7f713e054640 1 -- 172.21.15.125:0/2268851179 >> 172.21.15.125:0/2268851179 conn(0x7f713806c6e0 msgr2=0x7f7138158e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:15.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:15.001+0000 7f713e054640 1 -- 172.21.15.125:0/2268851179 shutdown_connections 2023-10-30T12:41:15.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:15.001+0000 7f713e054640 1 -- 172.21.15.125:0/2268851179 wait complete. 2023-10-30T12:41:15.023 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 65, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 65, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5130": { "addr": "172.21.15.125:6837/1624923396", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 1624923396, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 1624923396, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5130, "incarnation": 37, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 41 }, "gid_5433": { "addr": "172.21.15.125:6836/2595438444", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2595438444, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2595438444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5433, "incarnation": 62, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:rejoin", "state_seq": 65 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:41:09.247043+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5433, "mds_1": 5130 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:41:15.704 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:41:15.708 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:41:17.025 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:41:17.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.169+0000 7f4661cd6640 1 Processor -- start 2023-10-30T12:41:17.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.169+0000 7f4661cd6640 1 -- start start 2023-10-30T12:41:17.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.169+0000 7f4661cd6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f465c070740 0x7f465c0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:17.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.169+0000 7f4661cd6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f465c15e700 0x7f465c06a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:17.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.169+0000 7f4661cd6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f465c0c5810 con 0x7f465c1580e0 2023-10-30T12:41:17.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.169+0000 7f4661cd6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f465c070b60 con 0x7f465c15e700 2023-10-30T12:41:17.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.169+0000 7f4661cd6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f465c15eb90 con 0x7f465c070740 2023-10-30T12:41:17.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.169+0000 7f465b7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f465c1580e0 0x7f465c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60966/0 (socket says 172.21.15.125:60966) 2023-10-30T12:41:17.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.169+0000 7f465b7fe640 1 -- 172.21.15.125:0/715053375 learned_addr learned my addr 172.21.15.125:0/715053375 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:41:17.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465bfff640 1 --2- 172.21.15.125:0/715053375 >> v2:172.21.15.136:3300/0 conn(0x7f465c15e700 0x7f465c06a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:17.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f4653fff640 1 --2- 172.21.15.125:0/715053375 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f465c070740 0x7f465c0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:17.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 -- 172.21.15.125:0/715053375 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2572620692 0 0) 0x7f465c0c5810 con 0x7f465c1580e0 2023-10-30T12:41:17.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 -- 172.21.15.125:0/715053375 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4648003530 con 0x7f465c1580e0 2023-10-30T12:41:17.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 -- 172.21.15.125:0/715053375 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3782626075 0 0) 0x7f4648003530 con 0x7f465c1580e0 2023-10-30T12:41:17.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 -- 172.21.15.125:0/715053375 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f465c070740 msgr2=0x7f465c0c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:17.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 --2- 172.21.15.125:0/715053375 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f465c070740 0x7f465c0c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:17.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 -- 172.21.15.125:0/715053375 >> v2:172.21.15.136:3300/0 conn(0x7f465c15e700 msgr2=0x7f465c06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:17.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 --2- 172.21.15.125:0/715053375 >> v2:172.21.15.136:3300/0 conn(0x7f465c15e700 0x7f465c06a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:17.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 -- 172.21.15.125:0/715053375 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c0c6bb0 con 0x7f465c1580e0 2023-10-30T12:41:17.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 -- 172.21.15.125:0/715053375 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f464c001680 con 0x7f465c1580e0 2023-10-30T12:41:17.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 -- 172.21.15.125:0/715053375 >> v1:172.21.15.125:6789/0 conn(0x7f465c1580e0 legacy=0x7f465c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:17.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 --2- 172.21.15.125:0/715053375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4648003980 0x7f4648003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:17.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 --2- 172.21.15.125:0/715053375 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f465c070740 0x7f46480047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:17.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 --2- 172.21.15.125:0/715053375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4648004d10 0x7f4648005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:17.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 -- 172.21.15.125:0/715053375 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c070b60 con 0x7f4648004d10 2023-10-30T12:41:17.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 -- 172.21.15.125:0/715053375 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c15eb90 con 0x7f4648003980 2023-10-30T12:41:17.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465affd640 1 -- 172.21.15.125:0/715053375 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464800b990 con 0x7f465c070740 2023-10-30T12:41:17.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465bfff640 1 --2- 172.21.15.125:0/715053375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4648004d10 0x7f4648005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:17.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f4653fff640 1 --2- 172.21.15.125:0/715053375 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f465c070740 0x7f46480047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:17.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465b7fe640 1 --2- 172.21.15.125:0/715053375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4648003980 0x7f4648003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:17.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465bfff640 1 -- 172.21.15.125:0/715053375 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f465c070740 msgr2=0x7f46480047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:17.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465bfff640 1 --2- 172.21.15.125:0/715053375 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f465c070740 0x7f46480047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:17.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465bfff640 1 -- 172.21.15.125:0/715053375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4648003980 msgr2=0x7f4648003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465bfff640 1 --2- 172.21.15.125:0/715053375 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4648003980 0x7f4648003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.173+0000 7f465bfff640 1 -- 172.21.15.125:0/715053375 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4648003530 con 0x7f4648004d10 2023-10-30T12:41:17.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.177+0000 7f465bfff640 1 --2- 172.21.15.125:0/715053375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4648004d10 0x7f4648005190 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f465400e170 tx=0x7f465401cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dbfb946f53cacc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:17.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.177+0000 7f465affd640 1 -- 172.21.15.125:0/715053375 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46540a5c90 con 0x7f4648004d10 2023-10-30T12:41:17.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.177+0000 7f465affd640 1 -- 172.21.15.125:0/715053375 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465400f040 con 0x7f4648004d10 2023-10-30T12:41:17.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.177+0000 7f465affd640 1 -- 172.21.15.125:0/715053375 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46540ad440 con 0x7f4648004d10 2023-10-30T12:41:17.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.177+0000 7f4661cd6640 1 -- 172.21.15.125:0/715053375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4648004d10 msgr2=0x7f4648005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:17.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.177+0000 7f4661cd6640 1 --2- 172.21.15.125:0/715053375 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4648004d10 0x7f4648005190 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f465400e170 tx=0x7f465401cc80 comp rx=0 tx=0).stop 2023-10-30T12:41:17.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.177+0000 7f465bfff640 1 -- 172.21.15.125:0/715053375 reap_dead start 2023-10-30T12:41:17.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.177+0000 7f4661cd6640 1 -- 172.21.15.125:0/715053375 shutdown_connections 2023-10-30T12:41:17.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.177+0000 7f4661cd6640 1 -- 172.21.15.125:0/715053375 >> 172.21.15.125:0/715053375 conn(0x7f465c0c7910 msgr2=0x7f465c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:17.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.177+0000 7f4661cd6640 1 -- 172.21.15.125:0/715053375 shutdown_connections 2023-10-30T12:41:17.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.177+0000 7f4661cd6640 1 -- 172.21.15.125:0/715053375 wait complete. 2023-10-30T12:41:17.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.177+0000 7f4661cd6640 1 Processor -- start 2023-10-30T12:41:17.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.177+0000 7f4661cd6640 1 -- start start 2023-10-30T12:41:17.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f4661cd6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4648004d10 0x7f465c1f3360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:17.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f4661cd6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f465c1f74e0 0x7f465c0c1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:17.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f4661cd6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f465c149580 0x7f465c147d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:17.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f4661cd6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c14b330 con 0x7f4648004d10 2023-10-30T12:41:17.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f4661cd6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f465c15f410 con 0x7f465c149580 2023-10-30T12:41:17.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f4661cd6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f465c0c4440 con 0x7f465c1f74e0 2023-10-30T12:41:17.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f465b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4648004d10 0x7f465c1f3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:17.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f465b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4648004d10 0x7f465c1f3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41288/0 (socket says 172.21.15.125:41288) 2023-10-30T12:41:17.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f465b7fe640 1 -- 172.21.15.125:0/4168530769 learned_addr learned my addr 172.21.15.125:0/4168530769 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:17.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f465bfff640 1 --2- 172.21.15.125:0/4168530769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f465c149580 0x7f465c147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:17.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f4653fff640 1 --2- 172.21.15.125:0/4168530769 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f465c1f74e0 0x7f465c0c1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:17.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f465b7fe640 1 -- 172.21.15.125:0/4168530769 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f465c1f74e0 msgr2=0x7f465c0c1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:17.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f465b7fe640 1 --2- 172.21.15.125:0/4168530769 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f465c1f74e0 0x7f465c0c1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:17.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f465b7fe640 1 -- 172.21.15.125:0/4168530769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f465c149580 msgr2=0x7f465c147d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:17.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f465b7fe640 1 --2- 172.21.15.125:0/4168530769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f465c149580 0x7f465c147d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:17.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f465b7fe640 1 -- 172.21.15.125:0/4168530769 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f465c0c4670 con 0x7f4648004d10 2023-10-30T12:41:17.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f465b7fe640 1 --2- 172.21.15.125:0/4168530769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4648004d10 0x7f465c1f3360 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f464c007340 tx=0x7f464c007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:17.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f4658ff9640 1 -- 172.21.15.125:0/4168530769 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c00f9c0 con 0x7f4648004d10 2023-10-30T12:41:17.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f4658ff9640 1 -- 172.21.15.125:0/4168530769 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464c00fb20 con 0x7f4648004d10 2023-10-30T12:41:17.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f4661cd6640 1 -- 172.21.15.125:0/4168530769 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f465c0c4c10 con 0x7f4648004d10 2023-10-30T12:41:17.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f4658ff9640 1 -- 172.21.15.125:0/4168530769 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464c00fed0 con 0x7f4648004d10 2023-10-30T12:41:17.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f4661cd6640 1 -- 172.21.15.125:0/4168530769 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f465c1f6a00 con 0x7f4648004d10 2023-10-30T12:41:17.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f4661cd6640 1 -- 172.21.15.125:0/4168530769 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f465c04e170 con 0x7f4648004d10 2023-10-30T12:41:17.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.181+0000 7f4658ff9640 1 -- 172.21.15.125:0/4168530769 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f464c01c410 con 0x7f4648004d10 2023-10-30T12:41:17.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.185+0000 7f4658ff9640 1 --2- 172.21.15.125:0/4168530769 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f463c0664d0 0x7f463c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:17.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.185+0000 7f4658ff9640 1 -- 172.21.15.125:0/4168530769 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f464c083dc0 con 0x7f4648004d10 2023-10-30T12:41:17.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.185+0000 7f4653fff640 1 --2- 172.21.15.125:0/4168530769 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f463c0664d0 0x7f463c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:17.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.185+0000 7f4653fff640 1 --2- 172.21.15.125:0/4168530769 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f463c0664d0 0x7f463c068990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f465c0c4010 tx=0x7f4644008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:17.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.189+0000 7f4658ff9640 1 -- 172.21.15.125:0/4168530769 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f464c051060 con 0x7f4648004d10 2023-10-30T12:41:17.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.369+0000 7f4661cd6640 1 -- 172.21.15.125:0/4168530769 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f465c14e830 con 0x7f4648004d10 2023-10-30T12:41:17.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.369+0000 7f4658ff9640 1 -- 172.21.15.125:0/4168530769 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5742 (secure 0 0 0) 0x7f464c054680 con 0x7f4648004d10 2023-10-30T12:41:17.373 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:41:17.374 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:14.366855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:clientreplay","state_seq":67,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:41:17.374 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 66 2023-10-30T12:41:17.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.373+0000 7f4661cd6640 1 -- 172.21.15.125:0/4168530769 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f463c0664d0 msgr2=0x7f463c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:17.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.373+0000 7f4661cd6640 1 --2- 172.21.15.125:0/4168530769 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f463c0664d0 0x7f463c068990 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f465c0c4010 tx=0x7f4644008040 comp rx=0 tx=0).stop 2023-10-30T12:41:17.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.373+0000 7f4661cd6640 1 -- 172.21.15.125:0/4168530769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4648004d10 msgr2=0x7f465c1f3360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:17.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.373+0000 7f4661cd6640 1 --2- 172.21.15.125:0/4168530769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4648004d10 0x7f465c1f3360 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f464c007340 tx=0x7f464c007800 comp rx=0 tx=0).stop 2023-10-30T12:41:17.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.373+0000 7f4661cd6640 1 -- 172.21.15.125:0/4168530769 shutdown_connections 2023-10-30T12:41:17.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.373+0000 7f4661cd6640 1 --2- 172.21.15.125:0/4168530769 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f463c0664d0 0x7f463c068990 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:17.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.373+0000 7f4661cd6640 1 --2- 172.21.15.125:0/4168530769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f465c149580 0x7f465c147d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:17.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.373+0000 7f4661cd6640 1 --2- 172.21.15.125:0/4168530769 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f465c1f74e0 0x7f465c0c1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:17.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.373+0000 7f4661cd6640 1 --2- 172.21.15.125:0/4168530769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4648004d10 0x7f465c1f3360 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:17.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.373+0000 7f4661cd6640 1 -- 172.21.15.125:0/4168530769 >> 172.21.15.125:0/4168530769 conn(0x7f465c0c7910 msgr2=0x7f465c155840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:17.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.373+0000 7f4661cd6640 1 -- 172.21.15.125:0/4168530769 shutdown_connections 2023-10-30T12:41:17.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:17.373+0000 7f4661cd6640 1 -- 172.21.15.125:0/4168530769 wait complete. 2023-10-30T12:41:17.394 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:41:19.395 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:41:19.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe7cf65640 1 Processor -- start 2023-10-30T12:41:19.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe7cf65640 1 -- start start 2023-10-30T12:41:19.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe7cf65640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbe780c93d0 0x7fbe7806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:19.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe7cf65640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe7806afa0 con 0x7fbe7806e470 2023-10-30T12:41:19.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe7cf65640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe78154150 con 0x7fbe78153d70 2023-10-30T12:41:19.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe7cf65640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7806d780 con 0x7fbe780c93d0 2023-10-30T12:41:19.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe6dd9b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fbe7806e470 0x7fbe780c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60976/0 (socket says 172.21.15.125:60976) 2023-10-30T12:41:19.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe6dd9b640 1 -- 172.21.15.125:0/788571608 learned_addr learned my addr 172.21.15.125:0/788571608 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:41:19.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe76d9d640 1 --2- 172.21.15.125:0/788571608 >> v2:172.21.15.136:3300/0 conn(0x7fbe780c93d0 0x7fbe7806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:19.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 -- 172.21.15.125:0/788571608 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205262599 0 0) 0x7fbe7806afa0 con 0x7fbe7806e470 2023-10-30T12:41:19.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 -- 172.21.15.125:0/788571608 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe64003530 con 0x7fbe7806e470 2023-10-30T12:41:19.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 -- 172.21.15.125:0/788571608 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4173210740 0 0) 0x7fbe64003530 con 0x7fbe7806e470 2023-10-30T12:41:19.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 -- 172.21.15.125:0/788571608 >> v2:172.21.15.136:3300/0 conn(0x7fbe780c93d0 msgr2=0x7fbe7806ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:19.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 --2- 172.21.15.125:0/788571608 >> v2:172.21.15.136:3300/0 conn(0x7fbe780c93d0 0x7fbe7806ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:19.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 -- 172.21.15.125:0/788571608 >> v1:172.21.15.136:6789/0 conn(0x7fbe78153d70 legacy=0x7fbe780cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:19.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 -- 172.21.15.125:0/788571608 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7806e030 con 0x7fbe7806e470 2023-10-30T12:41:19.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 -- 172.21.15.125:0/788571608 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fbe68001680 con 0x7fbe7806e470 2023-10-30T12:41:19.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 -- 172.21.15.125:0/788571608 >> v1:172.21.15.125:6789/0 conn(0x7fbe7806e470 legacy=0x7fbe780c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:19.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 --2- 172.21.15.125:0/788571608 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe64003980 0x7fbe64003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:19.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 --2- 172.21.15.125:0/788571608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe640047d0 0x7fbe64004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:19.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 --2- 172.21.15.125:0/788571608 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe64005190 0x7fbe6400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:19.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 -- 172.21.15.125:0/788571608 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7806d780 con 0x7fbe64005190 2023-10-30T12:41:19.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 -- 172.21.15.125:0/788571608 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6400c030 con 0x7fbe640047d0 2023-10-30T12:41:19.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe75d9b640 1 -- 172.21.15.125:0/788571608 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe6400c1f0 con 0x7fbe64003980 2023-10-30T12:41:19.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe76d9d640 1 --2- 172.21.15.125:0/788571608 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe64005190 0x7fbe6400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:19.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe7659c640 1 --2- 172.21.15.125:0/788571608 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe64003980 0x7fbe64003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:19.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.541+0000 7fbe6dd9b640 1 --2- 172.21.15.125:0/788571608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe640047d0 0x7fbe64004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:19.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe76d9d640 1 -- 172.21.15.125:0/788571608 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe64003980 msgr2=0x7fbe64003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:19.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe76d9d640 1 --2- 172.21.15.125:0/788571608 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe64003980 0x7fbe64003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:19.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe76d9d640 1 -- 172.21.15.125:0/788571608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe640047d0 msgr2=0x7fbe64004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:19.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe76d9d640 1 --2- 172.21.15.125:0/788571608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe640047d0 0x7fbe64004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:19.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe76d9d640 1 -- 172.21.15.125:0/788571608 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe64003530 con 0x7fbe64005190 2023-10-30T12:41:19.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe6dd9b640 1 --2- 172.21.15.125:0/788571608 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe640047d0 0x7fbe64004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:41:19.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe7659c640 1 --2- 172.21.15.125:0/788571608 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe64003980 0x7fbe64003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:41:19.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe76d9d640 1 -- 172.21.15.125:0/788571608 reap_dead start 2023-10-30T12:41:19.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe76d9d640 1 --2- 172.21.15.125:0/788571608 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe64005190 0x7fbe6400b990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009fc0 tx=0x7fbe7001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bdf7b272a9f90a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:19.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe75d9b640 1 -- 172.21.15.125:0/788571608 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe700a5cb0 con 0x7fbe64005190 2023-10-30T12:41:19.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe75d9b640 1 -- 172.21.15.125:0/788571608 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7000f040 con 0x7fbe64005190 2023-10-30T12:41:19.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe75d9b640 1 -- 172.21.15.125:0/788571608 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70003be0 con 0x7fbe64005190 2023-10-30T12:41:19.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe7cf65640 1 -- 172.21.15.125:0/788571608 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe64005190 msgr2=0x7fbe6400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:19.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe7cf65640 1 --2- 172.21.15.125:0/788571608 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe64005190 0x7fbe6400b990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009fc0 tx=0x7fbe7001cca0 comp rx=0 tx=0).stop 2023-10-30T12:41:19.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe7cf65640 1 -- 172.21.15.125:0/788571608 shutdown_connections 2023-10-30T12:41:19.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe7cf65640 1 --2- 172.21.15.125:0/788571608 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe64005190 0x7fbe6400b990 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:19.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe7cf65640 1 -- 172.21.15.125:0/788571608 >> 172.21.15.125:0/788571608 conn(0x7fbe78071ea0 msgr2=0x7fbe78065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:19.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe7cf65640 1 -- 172.21.15.125:0/788571608 shutdown_connections 2023-10-30T12:41:19.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.545+0000 7fbe7cf65640 1 -- 172.21.15.125:0/788571608 wait complete. 2023-10-30T12:41:19.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe7cf65640 1 Processor -- start 2023-10-30T12:41:19.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe7cf65640 1 -- start start 2023-10-30T12:41:19.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe7cf65640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe64005190 0x7fbe7814f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:19.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe7cf65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe7806ca10 0x7fbe78154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:19.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe7cf65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe781f25b0 0x7fbe78148370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:19.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe7cf65640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe78157ad0 con 0x7fbe781f25b0 2023-10-30T12:41:19.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe7cf65640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7814f560 con 0x7fbe64005190 2023-10-30T12:41:19.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe7cf65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe78150680 con 0x7fbe7806ca10 2023-10-30T12:41:19.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe76d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe781f25b0 0x7fbe78148370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:19.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe76d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe781f25b0 0x7fbe78148370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41310/0 (socket says 172.21.15.125:41310) 2023-10-30T12:41:19.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe76d9d640 1 -- 172.21.15.125:0/2916757245 learned_addr learned my addr 172.21.15.125:0/2916757245 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:19.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe6dd9b640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe7806ca10 0x7fbe78154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:19.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe7659c640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe64005190 0x7fbe7814f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:19.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe76d9d640 1 -- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe7806ca10 msgr2=0x7fbe78154330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:19.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe76d9d640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe7806ca10 0x7fbe78154330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:19.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe76d9d640 1 -- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe64005190 msgr2=0x7fbe7814f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:19.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe76d9d640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe64005190 0x7fbe7814f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:19.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe76d9d640 1 -- 172.21.15.125:0/2916757245 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe781f2e80 con 0x7fbe781f25b0 2023-10-30T12:41:19.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe6dd9b640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe7806ca10 0x7fbe78154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:41:19.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.549+0000 7fbe76d9d640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe781f25b0 0x7fbe78148370 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fbe7001d180 tx=0x7fbe70009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:19.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.553+0000 7fbe6f7fe640 1 -- 172.21.15.125:0/2916757245 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe700aed40 con 0x7fbe781f25b0 2023-10-30T12:41:19.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.553+0000 7fbe6f7fe640 1 -- 172.21.15.125:0/2916757245 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe700aeec0 con 0x7fbe781f25b0 2023-10-30T12:41:19.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.553+0000 7fbe7cf65640 1 -- 172.21.15.125:0/2916757245 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe781f3160 con 0x7fbe781f25b0 2023-10-30T12:41:19.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.553+0000 7fbe6f7fe640 1 -- 172.21.15.125:0/2916757245 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70003be0 con 0x7fbe781f25b0 2023-10-30T12:41:19.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.553+0000 7fbe7cf65640 1 -- 172.21.15.125:0/2916757245 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7814f180 con 0x7fbe781f25b0 2023-10-30T12:41:19.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.553+0000 7fbe7cf65640 1 -- 172.21.15.125:0/2916757245 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7814f6a0 con 0x7fbe781f25b0 2023-10-30T12:41:19.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.553+0000 7fbe6f7fe640 1 -- 172.21.15.125:0/2916757245 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbe70003d40 con 0x7fbe781f25b0 2023-10-30T12:41:19.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.553+0000 7fbe6f7fe640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbe580664d0 0x7fbe58068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:19.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.553+0000 7fbe6f7fe640 1 -- 172.21.15.125:0/2916757245 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7fbe700a6e40 con 0x7fbe781f25b0 2023-10-30T12:41:19.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.553+0000 7fbe7659c640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbe580664d0 0x7fbe58068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:19.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.557+0000 7fbe7659c640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbe580664d0 0x7fbe58068990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbe60000c10 tx=0x7fbe60009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:19.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.557+0000 7fbe6f7fe640 1 -- 172.21.15.125:0/2916757245 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbe7814f6a0 con 0x7fbe781f25b0 2023-10-30T12:41:19.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.741+0000 7fbe7cf65640 1 -- 172.21.15.125:0/2916757245 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe7814f6a0 con 0x7fbe781f25b0 2023-10-30T12:41:19.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.741+0000 7fbe6f7fe640 1 -- 172.21.15.125:0/2916757245 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+5742 (secure 0 0 0) 0x7fbe7814f6a0 con 0x7fbe781f25b0 2023-10-30T12:41:19.742 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:41:19.742 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:14.366855+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:clientreplay","state_seq":67,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:41:19.743 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 66 2023-10-30T12:41:19.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.741+0000 7fbe7cf65640 1 -- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbe580664d0 msgr2=0x7fbe58068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:19.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.741+0000 7fbe7cf65640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbe580664d0 0x7fbe58068990 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fbe60000c10 tx=0x7fbe60009040 comp rx=0 tx=0).stop 2023-10-30T12:41:19.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.741+0000 7fbe7cf65640 1 -- 172.21.15.125:0/2916757245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe781f25b0 msgr2=0x7fbe78148370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:19.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.741+0000 7fbe7cf65640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe781f25b0 0x7fbe78148370 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fbe7001d180 tx=0x7fbe70009750 comp rx=0 tx=0).stop 2023-10-30T12:41:19.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.741+0000 7fbe7cf65640 1 -- 172.21.15.125:0/2916757245 shutdown_connections 2023-10-30T12:41:19.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.741+0000 7fbe7cf65640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbe580664d0 0x7fbe58068990 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:19.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.741+0000 7fbe7cf65640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe781f25b0 0x7fbe78148370 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:19.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.741+0000 7fbe7cf65640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe7806ca10 0x7fbe78154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:19.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.741+0000 7fbe7cf65640 1 --2- 172.21.15.125:0/2916757245 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe64005190 0x7fbe7814f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:19.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.741+0000 7fbe7cf65640 1 -- 172.21.15.125:0/2916757245 >> 172.21.15.125:0/2916757245 conn(0x7fbe78071ea0 msgr2=0x7fbe780cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:19.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.741+0000 7fbe7cf65640 1 -- 172.21.15.125:0/2916757245 shutdown_connections 2023-10-30T12:41:19.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:19.741+0000 7fbe7cf65640 1 -- 172.21.15.125:0/2916757245 wait complete. 2023-10-30T12:41:19.763 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:41:21.763 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:41:21.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.901+0000 7f85f2cd6640 1 Processor -- start 2023-10-30T12:41:21.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.901+0000 7f85f2cd6640 1 -- start start 2023-10-30T12:41:21.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.905+0000 7f85f2cd6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f85ec0c05d0 0x7f85ec13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:21.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.905+0000 7f85f2cd6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ec0c6f60 0x7f85ec13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:21.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.905+0000 7f85f2cd6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85ec0c0a50 con 0x7f85ec0c6b80 2023-10-30T12:41:21.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.905+0000 7f85f2cd6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85ec06f6e0 con 0x7f85ec0c05d0 2023-10-30T12:41:21.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.905+0000 7f85f2cd6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec06f9c0 con 0x7f85ec0c6f60 2023-10-30T12:41:21.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.905+0000 7f85e3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f85ec0c05d0 0x7f85ec13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:21.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.905+0000 7f85e3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f85ec0c05d0 0x7f85ec13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60352/0 (socket says 172.21.15.125:60352) 2023-10-30T12:41:21.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.905+0000 7f85e3fff640 1 -- 172.21.15.125:0/2943313854 learned_addr learned my addr 172.21.15.125:0/2943313854 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:21.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.905+0000 7f85f1273640 1 --2- 172.21.15.125:0/2943313854 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ec0c6f60 0x7f85ec13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:21.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.905+0000 7f85e3fff640 1 -- 172.21.15.125:0/2943313854 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ec0c6f60 msgr2=0x7f85ec13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:21.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.905+0000 7f85e3fff640 1 --2- 172.21.15.125:0/2943313854 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ec0c6f60 0x7f85ec13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:21.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.905+0000 7f85e3fff640 1 -- 172.21.15.125:0/2943313854 >> v1:172.21.15.136:6789/0 conn(0x7f85ec0c6b80 legacy=0x7f85ec0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:21.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.905+0000 7f85e3fff640 1 -- 172.21.15.125:0/2943313854 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ec06fca0 con 0x7f85ec0c05d0 2023-10-30T12:41:21.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.905+0000 7f85f1273640 1 --2- 172.21.15.125:0/2943313854 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ec0c6f60 0x7f85ec13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:41:21.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.909+0000 7f85e3fff640 1 --2- 172.21.15.125:0/2943313854 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ec0c05d0 0x7f85ec13ed90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f85dc009870 tx=0x7f85dc01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc4c9956e0c8526 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:21.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.909+0000 7f85e37fe640 1 -- 172.21.15.125:0/2943313854 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc0ab650 con 0x7f85ec0c05d0 2023-10-30T12:41:21.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.909+0000 7f85e37fe640 1 -- 172.21.15.125:0/2943313854 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85dc0ab7b0 con 0x7f85ec0c05d0 2023-10-30T12:41:21.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.909+0000 7f85e37fe640 1 -- 172.21.15.125:0/2943313854 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85dc0abb40 con 0x7f85ec0c05d0 2023-10-30T12:41:21.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.909+0000 7f85f2cd6640 1 -- 172.21.15.125:0/2943313854 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ec0c05d0 msgr2=0x7f85ec13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:21.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.909+0000 7f85f2cd6640 1 --2- 172.21.15.125:0/2943313854 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ec0c05d0 0x7f85ec13ed90 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f85dc009870 tx=0x7f85dc01c540 comp rx=0 tx=0).stop 2023-10-30T12:41:21.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.909+0000 7f85f2cd6640 1 -- 172.21.15.125:0/2943313854 shutdown_connections 2023-10-30T12:41:21.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.909+0000 7f85f2cd6640 1 --2- 172.21.15.125:0/2943313854 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ec0c6f60 0x7f85ec13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:21.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.909+0000 7f85f2cd6640 1 --2- 172.21.15.125:0/2943313854 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ec0c05d0 0x7f85ec13ed90 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:21.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.909+0000 7f85f2cd6640 1 -- 172.21.15.125:0/2943313854 >> 172.21.15.125:0/2943313854 conn(0x7f85ec0c7910 msgr2=0x7f85ec0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:21.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.909+0000 7f85f2cd6640 1 -- 172.21.15.125:0/2943313854 shutdown_connections 2023-10-30T12:41:21.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.909+0000 7f85f2cd6640 1 -- 172.21.15.125:0/2943313854 wait complete. 2023-10-30T12:41:21.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.909+0000 7f85f2cd6640 1 Processor -- start 2023-10-30T12:41:21.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.909+0000 7f85f2cd6640 1 -- start start 2023-10-30T12:41:21.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f2cd6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ec0c05d0 0x7f85ec13c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:21.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f2cd6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85ec0c6b80 0x7f85ec159320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:21.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f2cd6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ec0c6f60 0x7f85ec135680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:21.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f2cd6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec152ef0 con 0x7f85ec0c6f60 2023-10-30T12:41:21.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f2cd6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec13ce90 con 0x7f85ec0c6b80 2023-10-30T12:41:21.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f2cd6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ec13e240 con 0x7f85ec0c05d0 2023-10-30T12:41:21.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f1273640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ec0c6f60 0x7f85ec135680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:21.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f1273640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ec0c6f60 0x7f85ec135680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60354/0 (socket says 172.21.15.125:60354) 2023-10-30T12:41:21.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f1273640 1 -- 172.21.15.125:0/3663519114 learned_addr learned my addr 172.21.15.125:0/3663519114 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:21.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85e3fff640 1 --2- 172.21.15.125:0/3663519114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85ec0c6b80 0x7f85ec159320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:21.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f0a72640 1 --2- 172.21.15.125:0/3663519114 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ec0c05d0 0x7f85ec13c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:21.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f1273640 1 -- 172.21.15.125:0/3663519114 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ec0c05d0 msgr2=0x7f85ec13c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:21.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f1273640 1 --2- 172.21.15.125:0/3663519114 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ec0c05d0 0x7f85ec13c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:21.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f1273640 1 -- 172.21.15.125:0/3663519114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85ec0c6b80 msgr2=0x7f85ec159320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:21.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f1273640 1 --2- 172.21.15.125:0/3663519114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85ec0c6b80 0x7f85ec159320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:21.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f1273640 1 -- 172.21.15.125:0/3663519114 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85dc0094d0 con 0x7f85ec0c6f60 2023-10-30T12:41:21.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f1273640 1 --2- 172.21.15.125:0/3663519114 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ec0c6f60 0x7f85ec135680 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f85e4007e20 tx=0x7f85e400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:21.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85e17fa640 1 -- 172.21.15.125:0/3663519114 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e40039a0 con 0x7f85ec0c6f60 2023-10-30T12:41:21.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85e17fa640 1 -- 172.21.15.125:0/3663519114 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e400f040 con 0x7f85ec0c6f60 2023-10-30T12:41:21.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85e17fa640 1 -- 172.21.15.125:0/3663519114 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e4003d20 con 0x7f85ec0c6f60 2023-10-30T12:41:21.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f2cd6640 1 -- 172.21.15.125:0/3663519114 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85ec0c1f30 con 0x7f85ec0c6f60 2023-10-30T12:41:21.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f2cd6640 1 -- 172.21.15.125:0/3663519114 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85ec0ca680 con 0x7f85ec0c6f60 2023-10-30T12:41:21.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85f2cd6640 1 -- 172.21.15.125:0/3663519114 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85ec1353a0 con 0x7f85ec0c6f60 2023-10-30T12:41:21.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.913+0000 7f85e17fa640 1 -- 172.21.15.125:0/3663519114 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f85e4004030 con 0x7f85ec0c6f60 2023-10-30T12:41:21.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.917+0000 7f85e17fa640 1 --2- 172.21.15.125:0/3663519114 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85d80664a0 0x7f85d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:21.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.917+0000 7f85e17fa640 1 -- 172.21.15.125:0/3663519114 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f85e4084320 con 0x7f85ec0c6f60 2023-10-30T12:41:21.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.917+0000 7f85f0a72640 1 --2- 172.21.15.125:0/3663519114 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85d80664a0 0x7f85d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:21.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.917+0000 7f85f0a72640 1 --2- 172.21.15.125:0/3663519114 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85d80664a0 0x7f85d8068960 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f85e8007fd0 tx=0x7f85e8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:21.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:21.921+0000 7f85e17fa640 1 -- 172.21.15.125:0/3663519114 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f85ec1353a0 con 0x7f85ec0c6f60 2023-10-30T12:41:22.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:22.101+0000 7f85f2cd6640 1 -- 172.21.15.125:0/3663519114 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85ec1353a0 con 0x7f85ec0c6f60 2023-10-30T12:41:22.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:22.101+0000 7f85e17fa640 1 -- 172.21.15.125:0/3663519114 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f85ec1353a0 con 0x7f85ec0c6f60 2023-10-30T12:41:22.104 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:41:22.104 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:20.693301+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:41:22.105 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 67 2023-10-30T12:41:22.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:22.105+0000 7f85f2cd6640 1 -- 172.21.15.125:0/3663519114 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85d80664a0 msgr2=0x7f85d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:22.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:22.105+0000 7f85f2cd6640 1 --2- 172.21.15.125:0/3663519114 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85d80664a0 0x7f85d8068960 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f85e8007fd0 tx=0x7f85e8009040 comp rx=0 tx=0).stop 2023-10-30T12:41:22.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:22.105+0000 7f85f2cd6640 1 -- 172.21.15.125:0/3663519114 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ec0c6f60 msgr2=0x7f85ec135680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:22.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:22.105+0000 7f85f2cd6640 1 --2- 172.21.15.125:0/3663519114 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ec0c6f60 0x7f85ec135680 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f85e4007e20 tx=0x7f85e400d4a0 comp rx=0 tx=0).stop 2023-10-30T12:41:22.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:22.105+0000 7f85f2cd6640 1 -- 172.21.15.125:0/3663519114 shutdown_connections 2023-10-30T12:41:22.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:22.105+0000 7f85f2cd6640 1 --2- 172.21.15.125:0/3663519114 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85d80664a0 0x7f85d8068960 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:22.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:22.105+0000 7f85f2cd6640 1 --2- 172.21.15.125:0/3663519114 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ec0c6f60 0x7f85ec135680 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:22.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:22.105+0000 7f85f2cd6640 1 --2- 172.21.15.125:0/3663519114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85ec0c6b80 0x7f85ec159320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:22.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:22.105+0000 7f85f2cd6640 1 --2- 172.21.15.125:0/3663519114 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ec0c05d0 0x7f85ec13c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:22.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:22.105+0000 7f85f2cd6640 1 -- 172.21.15.125:0/3663519114 >> 172.21.15.125:0/3663519114 conn(0x7f85ec0c7910 msgr2=0x7f85ec0b0880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:22.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:22.105+0000 7f85f2cd6640 1 -- 172.21.15.125:0/3663519114 shutdown_connections 2023-10-30T12:41:22.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:22.105+0000 7f85f2cd6640 1 -- 172.21.15.125:0/3663519114 wait complete. 2023-10-30T12:41:22.124 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=4973 2023-10-30T12:41:22.124 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.f 2023-10-30T12:41:24.125 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T12:41:24.126 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T12:41:24.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T12:41:24.129 INFO:tasks.ceph.mds.f:Started 2023-10-30T12:41:24.129 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:41:24.205 INFO:tasks.ceph.mds.f.smithi136.stdout:starting mds.f at 2023-10-30T12:41:24.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.277+0000 7f63bab94640 1 Processor -- start 2023-10-30T12:41:24.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.277+0000 7f63bab94640 1 -- start start 2023-10-30T12:41:24.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.281+0000 7f63bab94640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f63b41539a0 0x7f63b415a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:24.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.281+0000 7f63bab94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f63b4069e90 0x7f63b406a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:24.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.281+0000 7f63bab94640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b415a8c0 con 0x7f63b4070a00 2023-10-30T12:41:24.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.281+0000 7f63bab94640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b406fbb0 con 0x7f63b41539a0 2023-10-30T12:41:24.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.281+0000 7f63bab94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b4153d80 con 0x7f63b4069e90 2023-10-30T12:41:24.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.281+0000 7f63b8930640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f63b41539a0 0x7f63b415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:24.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.281+0000 7f63b8930640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f63b41539a0 0x7f63b415a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60360/0 (socket says 172.21.15.125:60360) 2023-10-30T12:41:24.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.281+0000 7f63b8930640 1 -- 172.21.15.125:0/345424637 learned_addr learned my addr 172.21.15.125:0/345424637 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:24.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.281+0000 7f63b9131640 1 --2- 172.21.15.125:0/345424637 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f63b4069e90 0x7f63b406a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:24.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.281+0000 7f63b8930640 1 -- 172.21.15.125:0/345424637 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f63b4069e90 msgr2=0x7f63b406a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:24.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.281+0000 7f63b8930640 1 --2- 172.21.15.125:0/345424637 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f63b4069e90 0x7f63b406a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:24.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.281+0000 7f63b8930640 1 -- 172.21.15.125:0/345424637 >> v1:172.21.15.136:6789/0 conn(0x7f63b4070a00 legacy=0x7f63b4070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:24.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.281+0000 7f63b8930640 1 -- 172.21.15.125:0/345424637 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b40c5c50 con 0x7f63b41539a0 2023-10-30T12:41:24.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.281+0000 7f63b9131640 1 --2- 172.21.15.125:0/345424637 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f63b4069e90 0x7f63b406a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:41:24.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.285+0000 7f63b8930640 1 --2- 172.21.15.125:0/345424637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63b41539a0 0x7f63b415a380 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009870 tx=0x7f63b001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde224091bd31dce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:24.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.285+0000 7f63ab7fe640 1 -- 172.21.15.125:0/345424637 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b00ad640 con 0x7f63b41539a0 2023-10-30T12:41:24.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.285+0000 7f63ab7fe640 1 -- 172.21.15.125:0/345424637 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b00ad7a0 con 0x7f63b41539a0 2023-10-30T12:41:24.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.285+0000 7f63ab7fe640 1 -- 172.21.15.125:0/345424637 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b00adb30 con 0x7f63b41539a0 2023-10-30T12:41:24.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.285+0000 7f63bab94640 1 -- 172.21.15.125:0/345424637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63b41539a0 msgr2=0x7f63b415a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:24.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.285+0000 7f63bab94640 1 --2- 172.21.15.125:0/345424637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63b41539a0 0x7f63b415a380 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009870 tx=0x7f63b001c480 comp rx=0 tx=0).stop 2023-10-30T12:41:24.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.285+0000 7f63bab94640 1 -- 172.21.15.125:0/345424637 shutdown_connections 2023-10-30T12:41:24.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.285+0000 7f63bab94640 1 --2- 172.21.15.125:0/345424637 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f63b4069e90 0x7f63b406a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:24.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.285+0000 7f63bab94640 1 --2- 172.21.15.125:0/345424637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63b41539a0 0x7f63b415a380 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:24.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.285+0000 7f63bab94640 1 -- 172.21.15.125:0/345424637 >> 172.21.15.125:0/345424637 conn(0x7f63b40c7910 msgr2=0x7f63b40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:24.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.285+0000 7f63bab94640 1 -- 172.21.15.125:0/345424637 shutdown_connections 2023-10-30T12:41:24.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.285+0000 7f63bab94640 1 -- 172.21.15.125:0/345424637 wait complete. 2023-10-30T12:41:24.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.285+0000 7f63bab94640 1 Processor -- start 2023-10-30T12:41:24.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.285+0000 7f63bab94640 1 -- start start 2023-10-30T12:41:24.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63bab94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f63b4069e90 0x7f63b41f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:24.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63bab94640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63b4070a00 0x7f63b41ec200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:24.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63bab94640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63b41539a0 0x7f63b40ca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:24.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63bab94640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b41eb7b0 con 0x7f63b4070a00 2023-10-30T12:41:24.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63bab94640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b41f2ff0 con 0x7f63b41539a0 2023-10-30T12:41:24.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63bab94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b40c4af0 con 0x7f63b4069e90 2023-10-30T12:41:24.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63abfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63b4070a00 0x7f63b41ec200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:24.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63abfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63b4070a00 0x7f63b41ec200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60370/0 (socket says 172.21.15.125:60370) 2023-10-30T12:41:24.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63abfff640 1 -- 172.21.15.125:0/1863310220 learned_addr learned my addr 172.21.15.125:0/1863310220 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:24.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63b8930640 1 --2- 172.21.15.125:0/1863310220 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f63b4069e90 0x7f63b41f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:24.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63b9131640 1 --2- 172.21.15.125:0/1863310220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63b41539a0 0x7f63b40ca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:24.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63abfff640 1 -- 172.21.15.125:0/1863310220 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f63b4069e90 msgr2=0x7f63b41f2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:24.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63abfff640 1 --2- 172.21.15.125:0/1863310220 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f63b4069e90 0x7f63b41f2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:24.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63abfff640 1 -- 172.21.15.125:0/1863310220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63b41539a0 msgr2=0x7f63b40ca3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:24.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63abfff640 1 --2- 172.21.15.125:0/1863310220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63b41539a0 0x7f63b40ca3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:24.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63abfff640 1 -- 172.21.15.125:0/1863310220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b00094d0 con 0x7f63b4070a00 2023-10-30T12:41:24.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63abfff640 1 --2- 172.21.15.125:0/1863310220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63b4070a00 0x7f63b41ec200 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f63a400c500 tx=0x7f63a400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:24.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63a97fa640 1 -- 172.21.15.125:0/1863310220 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a4005220 con 0x7f63b4070a00 2023-10-30T12:41:24.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63a97fa640 1 -- 172.21.15.125:0/1863310220 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a4005380 con 0x7f63b4070a00 2023-10-30T12:41:24.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63a97fa640 1 -- 172.21.15.125:0/1863310220 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a4007270 con 0x7f63b4070a00 2023-10-30T12:41:24.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63bab94640 1 -- 172.21.15.125:0/1863310220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b40c4dd0 con 0x7f63b4070a00 2023-10-30T12:41:24.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63bab94640 1 -- 172.21.15.125:0/1863310220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b41eba00 con 0x7f63b4070a00 2023-10-30T12:41:24.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63bab94640 1 -- 172.21.15.125:0/1863310220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b404e170 con 0x7f63b4070a00 2023-10-30T12:41:24.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.289+0000 7f63a97fa640 1 -- 172.21.15.125:0/1863310220 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f63a40073d0 con 0x7f63b4070a00 2023-10-30T12:41:24.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.293+0000 7f63a97fa640 1 --2- 172.21.15.125:0/1863310220 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f63a00664a0 0x7f63a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:24.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.293+0000 7f63a97fa640 1 -- 172.21.15.125:0/1863310220 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f63a4083910 con 0x7f63b4070a00 2023-10-30T12:41:24.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.293+0000 7f63b8930640 1 --2- 172.21.15.125:0/1863310220 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f63a00664a0 0x7f63a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:24.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.293+0000 7f63b8930640 1 --2- 172.21.15.125:0/1863310220 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f63a00664a0 0x7f63a0068960 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f63b00af000 tx=0x7f63b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:24.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.297+0000 7f63a97fa640 1 -- 172.21.15.125:0/1863310220 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f63a4050b30 con 0x7f63b4070a00 2023-10-30T12:41:24.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.477+0000 7f63bab94640 1 -- 172.21.15.125:0/1863310220 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63b40c72f0 con 0x7f63b4070a00 2023-10-30T12:41:24.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.481+0000 7f63a97fa640 1 -- 172.21.15.125:0/1863310220 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+6530 (secure 0 0 0) 0x7f63a4054150 con 0x7f63b4070a00 2023-10-30T12:41:24.482 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:41:24.483 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5345,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:20.693301+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:41:24.483 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 68 2023-10-30T12:41:24.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.481+0000 7f63bab94640 1 -- 172.21.15.125:0/1863310220 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f63a00664a0 msgr2=0x7f63a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:24.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.481+0000 7f63bab94640 1 --2- 172.21.15.125:0/1863310220 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f63a00664a0 0x7f63a0068960 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f63b00af000 tx=0x7f63b0008040 comp rx=0 tx=0).stop 2023-10-30T12:41:24.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.481+0000 7f63bab94640 1 -- 172.21.15.125:0/1863310220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63b4070a00 msgr2=0x7f63b41ec200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:24.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.481+0000 7f63bab94640 1 --2- 172.21.15.125:0/1863310220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63b4070a00 0x7f63b41ec200 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f63a400c500 tx=0x7f63a400c9c0 comp rx=0 tx=0).stop 2023-10-30T12:41:24.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.481+0000 7f63bab94640 1 -- 172.21.15.125:0/1863310220 shutdown_connections 2023-10-30T12:41:24.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.481+0000 7f63bab94640 1 --2- 172.21.15.125:0/1863310220 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f63a00664a0 0x7f63a0068960 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:24.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.481+0000 7f63bab94640 1 --2- 172.21.15.125:0/1863310220 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63b41539a0 0x7f63b40ca3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:24.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.481+0000 7f63bab94640 1 --2- 172.21.15.125:0/1863310220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f63b4070a00 0x7f63b41ec200 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:24.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.481+0000 7f63bab94640 1 --2- 172.21.15.125:0/1863310220 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f63b4069e90 0x7f63b41f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:24.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.481+0000 7f63bab94640 1 -- 172.21.15.125:0/1863310220 >> 172.21.15.125:0/1863310220 conn(0x7f63b40c7910 msgr2=0x7f63b4158ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:24.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.481+0000 7f63bab94640 1 -- 172.21.15.125:0/1863310220 shutdown_connections 2023-10-30T12:41:24.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:24.481+0000 7f63bab94640 1 -- 172.21.15.125:0/1863310220 wait complete. 2023-10-30T12:41:24.503 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T12:41:24.503 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 35 secs before thrashing 2023-10-30T12:41:30.248 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:41:30.245+0000 7feb93272640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:41:30.249 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:41:30.245+0000 7feb93272640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:41:30.774 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 1% (238/12071) Updating files: 2% (242/12071) Updating files: 2% (354/12071) Updating files: 3% (363/12071) Updating files: 3% (474/12071) Updating files: 4% (483/12071) Updating files: 5% (604/12071) Updating files: 5% (630/12071) Updating files: 5% (686/12071) Updating files: 6% (725/12071) Updating files: 7% (845/12071) Updating files: 7% (898/12071) Updating files: 7% (964/12071) Updating files: 8% (966/12071) Updating files: 8% (968/12071) Updating files: 8% (1024/12071) Updating files: 9% (1087/12071) Updating files: 9% (1145/12071) Updating files: 10% (1208/12071) Updating files: 11% (1328/12071) Updating files: 11% (1329/12071) Updating files: 11% (1354/12071) Updating files: 11% (1411/12071) Updating files: 12% (1449/12071) Updating files: 12% (1486/12071) Updating files: 12% (1511/12071) Updating files: 13% (1570/12071) Updating files: 13% (1603/12071) Updating files: 13% (1639/12071) Updating files: 14% (1690/12071) Updating files: 14% (1710/12071) Updating files: 15% (1811/12071) Updating files: 15% (1903/12071) Updating files: 16% (1932/12071) Updating files: 17% (2053/12071) Updating files: 17% (2083/12071) Updating files: 17% (2149/12071) Updating files: 18% (2173/12071) Updating files: 18% (2183/12071) Updating files: 19% (2294/12071) Updating files: 19% (2376/12071) Updating files: 19% (2388/12071) Updating files: 20% (2415/12071) Updating files: 20% (2488/12071) Updating files: 21% (2535/12071) Updating files: 22% (2656/12071) Updating files: 22% (2705/12071) Updating files: 22% (2770/12071) Updating files: 23% (2777/12071) Updating files: 23% (2861/12071) Updating files: 23% (2863/12071) Updating files: 24% (2898/12071) Updating files: 24% (2965/12071) Updating files: 25% (3018/12071) Updating files: 26% (3139/12071) Updating files: 26% (3162/12071) Updating files: 27% (3260/12071) Updating files: 27% (3276/12071) Updating files: 27% (3338/12071) Updating files: 28% (3380/12071) Updating files: 28% (3417/12071) Updating files: 28% (3420/12071) Updating files: 29% (3501/12071) Updating files: 29% (3577/12071) Updating files: 30% (3622/12071) Updating files: 30% (3740/12071) Updating files: 31% (3743/12071) Updating files: 32% (3863/12071) Updating files: 33% (3984/12071) Updating files: 33% (4097/12071) Updating files: 34% (4105/12071) Updating files: 34% (4134/12071) Updating files: 34% (4192/12071) Updating files: 35% (4225/12071) Updating files: 35% (4242/12071) Updating files: 36% (4346/12071) Updating files: 37% (4467/12071) Updating files: 38% (4587/12071) Updating files: 38% (4664/12071) Updating files: 39% (4708/12071) Updating files: 39% (4735/12071) Updating files: 40% (4829/12071) Updating files: 41% (4950/12071) Updating files: 41% (5022/12071) Updating files: 42% (5070/12071) Updating files: 42% (5134/12071) Updating files: 43% (5191/12071) Updating files: 43% (5199/12071) Updating files: 44% (5312/12071) Updating files: 44% (5319/12071) Updating files: 44% (5376/12071) Updating files: 45% (5432/12071) Updating files: 45% (5455/12071) Updating files: 46% (5553/12071) Updating files: 47% (5674/12071) Updating files: 47% (5678/12071) Updating files: 48% (5795/12071) Updating files: 48% (5878/12071) Updating files: 49% (5915/12071) Updating files: 49% (5927/12071) Updating files: 50% (6036/12071) Updating files: 50% (6145/12071) Updating files: 51% (6157/12071) Updating files: 51% (6255/12071) Updating files: 52% (6277/12071) Updating files: 53% (6398/12071) Updating files: 53% (6441/12071) Updating files: 54% (6519/12071) Updating files: 54% (6544/12071) Updating files: 55% (6640/12071) Updating files: 56% (6760/12071) Updating files: 56% (6762/12071) Updating files: 57% (6881/12071) Updating files: 57% (6933/12071) Updating files: 57% (6963/12071) Updating files: 57% (6982/12071) Updating files: 58% (7002/12071) Updating files: 58% (7006/12071) Updating files: 58% (7075/12071) Updating files: 59% (7122/12071) Updating files: 60% (7243/12071) Updating files: 61% (7364/12071) Updating files: 61% (7403/12071) Updating files: 61% (7470/12071) Updating files: 62% (7485/12071) Updating files: 62% (7532/12071) Updating files: 63% (7605/12071) Updating files: 63% (7650/12071) Updating files: 64% (7726/12071) Updating files: 65% (7847/12071) Updating files: 65% (7928/12071) Updating files: 65% (7942/12071) Updating files: 66% (7967/12071) Updating files: 66% (8071/12071) Updating files: 67% (8088/12071) Updating files: 68% (8209/12071) Updating files: 68% (8272/12071) Updating files: 69% (8329/12071) Updating files: 70% (8450/12071) Updating files: 70% (8525/12071) Updating files: 71% (8571/12071) Updating files: 71% (8680/12071) Updating files: 72% (8692/12071) Updating files: 72% (8800/12071) Updating files: 73% (8812/12071) Updating files: 73% (8840/12071) Updating files: 73% (8931/12071) Updating files: 74% (8933/12071) Updating files: 74% (8941/12071) Updating files: 74% (9014/12071) Updating files: 74% (9023/12071) Updating files: 74% (9047/12071) Updating files: 75% (9054/12071) Updating files: 76% (9174/12071) Updating files: 76% (9176/12071) Updating files: 77% (9295/12071) Updating files: 77% (9393/12071) Updating files: 78% (9416/12071) Updating files: 78% (9532/12071) Updating files: 79% (9537/12071) Updating files: 79% (9614/12071) Updating files: 79% (9647/12071) Updating files: 80% (9657/12071) Updating files: 80% (9720/12071) Updating files: 80% (9741/12071) Updating files: 81% (9778/12071) Updating files: 81% (9892/12071) Updating files: 82% (9899/12071) Updating files: 83% (10019/12071) Updating files: 84% (10140/12071) Updating files: 85% (10261/12071) Updating files: 85% (10347/12071) Updating files: 86% (10382/12071) Updating files: 86% (10436/12071) Updating files: 86% (10499/12071) Updating files: 87% (10502/12071) Updating files: 87% (10535/12071) Updating files: 88% (10623/12071) Updating files: 88% (10632/12071) Updating files: 88% (10680/12071) Updating files: 88% (10682/12071) Updating files: 88% (10698/12071) Updating files: 88% (10731/12071) Updating files: 89% (10744/12071) Updating files: 90% (10864/12071) Updating files: 90% (10870/12071) Updating files: 91% (10985/12071) Updating files: 91% (11027/12071) Updating files: 92% (11106/12071) Updating files: 92% (11179/12071) Updating files: 93% (11227/12071) Updating files: 94% (11347/12071) Updating files: 94% (11370/12071) Updating files: 95% (11468/12071) Updating files: 95% (11476/12071) Updating files: 95% (11554/12071) Updating files: 96% (11589/12071) Updating files: 96% (11603/12071) Updating files: 96% (11604/12071) Updating files: 97% (11709/12071) Updating files: 97% (11738/12071) Updating files: 98% (11830/12071) Updating files: 98% (11891/12071) Updating files: 99% (11951/12071) Updating files: 99% (12007/12071) Updating files: 100% (12071/12071) Updating files: 100% (12071/12071), done. 2023-10-30T12:41:30.980 INFO:tasks.workunit.client.0.smithi125.stderr:+ trap - EXIT 2023-10-30T12:41:30.980 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ceph 2023-10-30T12:41:30.981 INFO:tasks.workunit.client.0.smithi125.stderr:+ seq 1 90 2023-10-30T12:41:30.982 INFO:tasks.workunit.client.0.smithi125.stderr:+ versions=1 2023-10-30T12:41:30.982 INFO:tasks.workunit.client.0.smithi125.stderr:2 2023-10-30T12:41:30.982 INFO:tasks.workunit.client.0.smithi125.stderr:3 2023-10-30T12:41:30.982 INFO:tasks.workunit.client.0.smithi125.stderr:4 2023-10-30T12:41:30.983 INFO:tasks.workunit.client.0.smithi125.stderr:5 2023-10-30T12:41:30.983 INFO:tasks.workunit.client.0.smithi125.stderr:6 2023-10-30T12:41:30.983 INFO:tasks.workunit.client.0.smithi125.stderr:7 2023-10-30T12:41:30.983 INFO:tasks.workunit.client.0.smithi125.stderr:8 2023-10-30T12:41:30.983 INFO:tasks.workunit.client.0.smithi125.stderr:9 2023-10-30T12:41:30.984 INFO:tasks.workunit.client.0.smithi125.stderr:10 2023-10-30T12:41:30.984 INFO:tasks.workunit.client.0.smithi125.stderr:11 2023-10-30T12:41:30.984 INFO:tasks.workunit.client.0.smithi125.stderr:12 2023-10-30T12:41:30.984 INFO:tasks.workunit.client.0.smithi125.stderr:13 2023-10-30T12:41:30.984 INFO:tasks.workunit.client.0.smithi125.stderr:14 2023-10-30T12:41:30.984 INFO:tasks.workunit.client.0.smithi125.stderr:15 2023-10-30T12:41:30.985 INFO:tasks.workunit.client.0.smithi125.stderr:16 2023-10-30T12:41:30.985 INFO:tasks.workunit.client.0.smithi125.stderr:17 2023-10-30T12:41:30.985 INFO:tasks.workunit.client.0.smithi125.stderr:18 2023-10-30T12:41:30.985 INFO:tasks.workunit.client.0.smithi125.stderr:19 2023-10-30T12:41:30.985 INFO:tasks.workunit.client.0.smithi125.stderr:20 2023-10-30T12:41:30.985 INFO:tasks.workunit.client.0.smithi125.stderr:21 2023-10-30T12:41:30.986 INFO:tasks.workunit.client.0.smithi125.stderr:22 2023-10-30T12:41:30.986 INFO:tasks.workunit.client.0.smithi125.stderr:23 2023-10-30T12:41:30.986 INFO:tasks.workunit.client.0.smithi125.stderr:24 2023-10-30T12:41:30.986 INFO:tasks.workunit.client.0.smithi125.stderr:25 2023-10-30T12:41:30.986 INFO:tasks.workunit.client.0.smithi125.stderr:26 2023-10-30T12:41:30.987 INFO:tasks.workunit.client.0.smithi125.stderr:27 2023-10-30T12:41:30.987 INFO:tasks.workunit.client.0.smithi125.stderr:28 2023-10-30T12:41:30.987 INFO:tasks.workunit.client.0.smithi125.stderr:29 2023-10-30T12:41:30.987 INFO:tasks.workunit.client.0.smithi125.stderr:30 2023-10-30T12:41:30.987 INFO:tasks.workunit.client.0.smithi125.stderr:31 2023-10-30T12:41:30.987 INFO:tasks.workunit.client.0.smithi125.stderr:32 2023-10-30T12:41:30.988 INFO:tasks.workunit.client.0.smithi125.stderr:33 2023-10-30T12:41:30.988 INFO:tasks.workunit.client.0.smithi125.stderr:34 2023-10-30T12:41:30.988 INFO:tasks.workunit.client.0.smithi125.stderr:35 2023-10-30T12:41:30.988 INFO:tasks.workunit.client.0.smithi125.stderr:36 2023-10-30T12:41:30.988 INFO:tasks.workunit.client.0.smithi125.stderr:37 2023-10-30T12:41:30.989 INFO:tasks.workunit.client.0.smithi125.stderr:38 2023-10-30T12:41:30.989 INFO:tasks.workunit.client.0.smithi125.stderr:39 2023-10-30T12:41:30.989 INFO:tasks.workunit.client.0.smithi125.stderr:40 2023-10-30T12:41:30.989 INFO:tasks.workunit.client.0.smithi125.stderr:41 2023-10-30T12:41:30.989 INFO:tasks.workunit.client.0.smithi125.stderr:42 2023-10-30T12:41:30.990 INFO:tasks.workunit.client.0.smithi125.stderr:43 2023-10-30T12:41:30.990 INFO:tasks.workunit.client.0.smithi125.stderr:44 2023-10-30T12:41:30.990 INFO:tasks.workunit.client.0.smithi125.stderr:45 2023-10-30T12:41:30.990 INFO:tasks.workunit.client.0.smithi125.stderr:46 2023-10-30T12:41:30.990 INFO:tasks.workunit.client.0.smithi125.stderr:47 2023-10-30T12:41:30.990 INFO:tasks.workunit.client.0.smithi125.stderr:48 2023-10-30T12:41:30.991 INFO:tasks.workunit.client.0.smithi125.stderr:49 2023-10-30T12:41:30.991 INFO:tasks.workunit.client.0.smithi125.stderr:50 2023-10-30T12:41:30.991 INFO:tasks.workunit.client.0.smithi125.stderr:51 2023-10-30T12:41:30.991 INFO:tasks.workunit.client.0.smithi125.stderr:52 2023-10-30T12:41:30.991 INFO:tasks.workunit.client.0.smithi125.stderr:53 2023-10-30T12:41:30.991 INFO:tasks.workunit.client.0.smithi125.stderr:54 2023-10-30T12:41:30.992 INFO:tasks.workunit.client.0.smithi125.stderr:55 2023-10-30T12:41:30.992 INFO:tasks.workunit.client.0.smithi125.stderr:56 2023-10-30T12:41:30.992 INFO:tasks.workunit.client.0.smithi125.stderr:57 2023-10-30T12:41:30.992 INFO:tasks.workunit.client.0.smithi125.stderr:58 2023-10-30T12:41:30.992 INFO:tasks.workunit.client.0.smithi125.stderr:59 2023-10-30T12:41:30.993 INFO:tasks.workunit.client.0.smithi125.stderr:60 2023-10-30T12:41:30.993 INFO:tasks.workunit.client.0.smithi125.stderr:61 2023-10-30T12:41:30.993 INFO:tasks.workunit.client.0.smithi125.stderr:62 2023-10-30T12:41:30.993 INFO:tasks.workunit.client.0.smithi125.stderr:63 2023-10-30T12:41:30.993 INFO:tasks.workunit.client.0.smithi125.stderr:64 2023-10-30T12:41:30.993 INFO:tasks.workunit.client.0.smithi125.stderr:65 2023-10-30T12:41:30.994 INFO:tasks.workunit.client.0.smithi125.stderr:66 2023-10-30T12:41:30.994 INFO:tasks.workunit.client.0.smithi125.stderr:67 2023-10-30T12:41:30.994 INFO:tasks.workunit.client.0.smithi125.stderr:68 2023-10-30T12:41:30.994 INFO:tasks.workunit.client.0.smithi125.stderr:69 2023-10-30T12:41:30.994 INFO:tasks.workunit.client.0.smithi125.stderr:70 2023-10-30T12:41:30.994 INFO:tasks.workunit.client.0.smithi125.stderr:71 2023-10-30T12:41:30.995 INFO:tasks.workunit.client.0.smithi125.stderr:72 2023-10-30T12:41:30.995 INFO:tasks.workunit.client.0.smithi125.stderr:73 2023-10-30T12:41:30.995 INFO:tasks.workunit.client.0.smithi125.stderr:74 2023-10-30T12:41:30.995 INFO:tasks.workunit.client.0.smithi125.stderr:75 2023-10-30T12:41:30.995 INFO:tasks.workunit.client.0.smithi125.stderr:76 2023-10-30T12:41:30.996 INFO:tasks.workunit.client.0.smithi125.stderr:77 2023-10-30T12:41:30.996 INFO:tasks.workunit.client.0.smithi125.stderr:78 2023-10-30T12:41:30.996 INFO:tasks.workunit.client.0.smithi125.stderr:79 2023-10-30T12:41:30.996 INFO:tasks.workunit.client.0.smithi125.stderr:80 2023-10-30T12:41:30.996 INFO:tasks.workunit.client.0.smithi125.stderr:81 2023-10-30T12:41:30.996 INFO:tasks.workunit.client.0.smithi125.stderr:82 2023-10-30T12:41:30.997 INFO:tasks.workunit.client.0.smithi125.stderr:83 2023-10-30T12:41:30.997 INFO:tasks.workunit.client.0.smithi125.stderr:84 2023-10-30T12:41:30.997 INFO:tasks.workunit.client.0.smithi125.stderr:85 2023-10-30T12:41:30.997 INFO:tasks.workunit.client.0.smithi125.stderr:86 2023-10-30T12:41:30.997 INFO:tasks.workunit.client.0.smithi125.stderr:87 2023-10-30T12:41:30.998 INFO:tasks.workunit.client.0.smithi125.stderr:88 2023-10-30T12:41:30.998 INFO:tasks.workunit.client.0.smithi125.stderr:89 2023-10-30T12:41:30.998 INFO:tasks.workunit.client.0.smithi125.stderr:90 2023-10-30T12:41:30.998 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 1 -eq 48 ] 2023-10-30T12:41:30.998 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.1 2023-10-30T12:41:30.999 INFO:tasks.workunit.client.0.smithi125.stdout:v0.1 2023-10-30T12:41:30.999 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.1 2023-10-30T12:41:30.999 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.1 2023-10-30T12:41:45.835 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:41:45.839 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:41:59.505 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:41:59.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.644+0000 7f31b9daa640 1 Processor -- start 2023-10-30T12:41:59.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.644+0000 7f31b9daa640 1 -- start start 2023-10-30T12:41:59.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.644+0000 7f31b9daa640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f31b40bfec0 0x7f31b40c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:59.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.644+0000 7f31b9daa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31b406a720 0x7f31b406abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:59.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.644+0000 7f31b9daa640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b4071290 con 0x7f31b4159e60 2023-10-30T12:41:59.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.644+0000 7f31b9daa640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b4071570 con 0x7f31b40bfec0 2023-10-30T12:41:59.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.644+0000 7f31b9daa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b4071850 con 0x7f31b406a720 2023-10-30T12:41:59.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.644+0000 7f31b37fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f31b4159e60 0x7f31b406f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42538/0 (socket says 172.21.15.125:42538) 2023-10-30T12:41:59.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b37fe640 1 -- 172.21.15.125:0/2220061673 learned_addr learned my addr 172.21.15.125:0/2220061673 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:41:59.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b3fff640 1 --2- 172.21.15.125:0/2220061673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31b406a720 0x7f31b406abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:59.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b2ffd640 1 --2- 172.21.15.125:0/2220061673 >> v2:172.21.15.136:3300/0 conn(0x7f31b40bfec0 0x7f31b40c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:59.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 -- 172.21.15.125:0/2220061673 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332891544 0 0) 0x7f31b4071290 con 0x7f31b4159e60 2023-10-30T12:41:59.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 -- 172.21.15.125:0/2220061673 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f319c003530 con 0x7f31b4159e60 2023-10-30T12:41:59.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 -- 172.21.15.125:0/2220061673 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1867879652 0 0) 0x7f319c003530 con 0x7f31b4159e60 2023-10-30T12:41:59.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 -- 172.21.15.125:0/2220061673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31b406a720 msgr2=0x7f31b406abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:59.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 --2- 172.21.15.125:0/2220061673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31b406a720 0x7f31b406abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:59.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 -- 172.21.15.125:0/2220061673 >> v2:172.21.15.136:3300/0 conn(0x7f31b40bfec0 msgr2=0x7f31b40c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:59.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 --2- 172.21.15.125:0/2220061673 >> v2:172.21.15.136:3300/0 conn(0x7f31b40bfec0 0x7f31b40c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:59.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 -- 172.21.15.125:0/2220061673 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b4071b30 con 0x7f31b4159e60 2023-10-30T12:41:59.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 -- 172.21.15.125:0/2220061673 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f31a4001680 con 0x7f31b4159e60 2023-10-30T12:41:59.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 -- 172.21.15.125:0/2220061673 >> v1:172.21.15.125:6789/0 conn(0x7f31b4159e60 legacy=0x7f31b406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:59.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 --2- 172.21.15.125:0/2220061673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31b406a720 0x7f319c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:59.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 --2- 172.21.15.125:0/2220061673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f319c003ec0 0x7f319c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:59.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 --2- 172.21.15.125:0/2220061673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f319c004d10 0x7f319c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:59.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 -- 172.21.15.125:0/2220061673 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b4071570 con 0x7f319c003ec0 2023-10-30T12:41:59.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 -- 172.21.15.125:0/2220061673 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b4071850 con 0x7f319c004d10 2023-10-30T12:41:59.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b27fc640 1 -- 172.21.15.125:0/2220061673 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f319c00dba0 con 0x7f31b406a720 2023-10-30T12:41:59.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b2ffd640 1 --2- 172.21.15.125:0/2220061673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f319c003ec0 0x7f319c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:59.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b37fe640 1 --2- 172.21.15.125:0/2220061673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31b406a720 0x7f319c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:59.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b3fff640 1 --2- 172.21.15.125:0/2220061673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f319c004d10 0x7f319c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:59.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b2ffd640 1 -- 172.21.15.125:0/2220061673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31b406a720 msgr2=0x7f319c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:59.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b2ffd640 1 --2- 172.21.15.125:0/2220061673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31b406a720 0x7f319c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:59.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b2ffd640 1 -- 172.21.15.125:0/2220061673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f319c004d10 msgr2=0x7f319c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:41:59.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b2ffd640 1 --2- 172.21.15.125:0/2220061673 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f319c004d10 0x7f319c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:59.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.648+0000 7f31b2ffd640 1 -- 172.21.15.125:0/2220061673 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c003530 con 0x7f319c003ec0 2023-10-30T12:41:59.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.652+0000 7f31b2ffd640 1 --2- 172.21.15.125:0/2220061673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f319c003ec0 0x7f319c0047d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009ae0 tx=0x7f31a801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e348d2e9f6fe70ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:59.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.652+0000 7f31b27fc640 1 -- 172.21.15.125:0/2220061673 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a80a5680 con 0x7f319c003ec0 2023-10-30T12:41:59.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.652+0000 7f31b27fc640 1 -- 172.21.15.125:0/2220061673 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a80a57e0 con 0x7f319c003ec0 2023-10-30T12:41:59.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.652+0000 7f31b27fc640 1 -- 172.21.15.125:0/2220061673 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a80a5b90 con 0x7f319c003ec0 2023-10-30T12:41:59.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.652+0000 7f31b9daa640 1 -- 172.21.15.125:0/2220061673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f319c003ec0 msgr2=0x7f319c0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:59.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.652+0000 7f31b9daa640 1 --2- 172.21.15.125:0/2220061673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f319c003ec0 0x7f319c0047d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009ae0 tx=0x7f31a801ca30 comp rx=0 tx=0).stop 2023-10-30T12:41:59.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.652+0000 7f31b3fff640 1 -- 172.21.15.125:0/2220061673 reap_dead start 2023-10-30T12:41:59.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.652+0000 7f31b9daa640 1 -- 172.21.15.125:0/2220061673 shutdown_connections 2023-10-30T12:41:59.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.652+0000 7f31b9daa640 1 -- 172.21.15.125:0/2220061673 >> 172.21.15.125:0/2220061673 conn(0x7f31b40c7910 msgr2=0x7f31b40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:59.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.652+0000 7f31b9daa640 1 -- 172.21.15.125:0/2220061673 shutdown_connections 2023-10-30T12:41:59.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.652+0000 7f31b9daa640 1 -- 172.21.15.125:0/2220061673 wait complete. 2023-10-30T12:41:59.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.652+0000 7f31b9daa640 1 Processor -- start 2023-10-30T12:41:59.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.652+0000 7f31b9daa640 1 -- start start 2023-10-30T12:41:59.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b9daa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f319c003ec0 0x7f31b41579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:59.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b9daa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31b414b3e0 0x7f31b4151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:59.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b9daa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31b4151bc0 0x7f31b4146440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:59.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b9daa640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b40af6d0 con 0x7f31b414b3e0 2023-10-30T12:41:59.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b9daa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b4150c10 con 0x7f31b4151bc0 2023-10-30T12:41:59.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b9daa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b414c4c0 con 0x7f319c003ec0 2023-10-30T12:41:59.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b2ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31b414b3e0 0x7f31b4151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:59.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b2ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31b414b3e0 0x7f31b4151680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56678/0 (socket says 172.21.15.125:56678) 2023-10-30T12:41:59.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b2ffd640 1 -- 172.21.15.125:0/3072559998 learned_addr learned my addr 172.21.15.125:0/3072559998 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:41:59.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b37fe640 1 --2- 172.21.15.125:0/3072559998 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f319c003ec0 0x7f31b41579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:59.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b3fff640 1 --2- 172.21.15.125:0/3072559998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31b4151bc0 0x7f31b4146440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:59.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b2ffd640 1 -- 172.21.15.125:0/3072559998 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f319c003ec0 msgr2=0x7f31b41579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:59.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b2ffd640 1 --2- 172.21.15.125:0/3072559998 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f319c003ec0 0x7f31b41579c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:59.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b2ffd640 1 -- 172.21.15.125:0/3072559998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31b4151bc0 msgr2=0x7f31b4146440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:59.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b2ffd640 1 --2- 172.21.15.125:0/3072559998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31b4151bc0 0x7f31b4146440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:59.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b2ffd640 1 -- 172.21.15.125:0/3072559998 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b414c760 con 0x7f31b414b3e0 2023-10-30T12:41:59.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b2ffd640 1 --2- 172.21.15.125:0/3072559998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31b414b3e0 0x7f31b4151680 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f31a801d2d0 tx=0x7f319c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:59.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f319bfff640 1 -- 172.21.15.125:0/3072559998 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a80be070 con 0x7f31b414b3e0 2023-10-30T12:41:59.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f319bfff640 1 -- 172.21.15.125:0/3072559998 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a8004020 con 0x7f31b414b3e0 2023-10-30T12:41:59.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f319bfff640 1 -- 172.21.15.125:0/3072559998 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8004390 con 0x7f31b414b3e0 2023-10-30T12:41:59.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b9daa640 1 -- 172.21.15.125:0/3072559998 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b4152020 con 0x7f31b414b3e0 2023-10-30T12:41:59.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b9daa640 1 -- 172.21.15.125:0/3072559998 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b4150e80 con 0x7f31b414b3e0 2023-10-30T12:41:59.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f31b9daa640 1 -- 172.21.15.125:0/3072559998 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b40c7530 con 0x7f31b414b3e0 2023-10-30T12:41:59.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.656+0000 7f319bfff640 1 -- 172.21.15.125:0/3072559998 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f31a80b4040 con 0x7f31b414b3e0 2023-10-30T12:41:59.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.660+0000 7f319bfff640 1 --2- 172.21.15.125:0/3072559998 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31a00664d0 0x7f31a0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:41:59.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.660+0000 7f319bfff640 1 -- 172.21.15.125:0/3072559998 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f31a811df00 con 0x7f31b414b3e0 2023-10-30T12:41:59.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.660+0000 7f31b37fe640 1 --2- 172.21.15.125:0/3072559998 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31a00664d0 0x7f31a0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:41:59.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.660+0000 7f31b37fe640 1 --2- 172.21.15.125:0/3072559998 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31a00664d0 0x7f31a0068990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f31a4000fc0 tx=0x7f31a4001020 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:41:59.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.664+0000 7f319bfff640 1 -- 172.21.15.125:0/3072559998 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f31b40c7530 con 0x7f31b414b3e0 2023-10-30T12:41:59.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.844+0000 7f31b9daa640 1 -- 172.21.15.125:0/3072559998 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31b40c7530 con 0x7f31b414b3e0 2023-10-30T12:41:59.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.844+0000 7f319bfff640 1 -- 172.21.15.125:0/3072559998 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f31b40c7530 con 0x7f31b414b3e0 2023-10-30T12:41:59.848 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:41:59.849 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5345,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:31.350689+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:41:59.849 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 69 2023-10-30T12:41:59.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.848+0000 7f31b9daa640 1 -- 172.21.15.125:0/3072559998 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31a00664d0 msgr2=0x7f31a0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:59.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.848+0000 7f31b9daa640 1 --2- 172.21.15.125:0/3072559998 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31a00664d0 0x7f31a0068990 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f31a4000fc0 tx=0x7f31a4001020 comp rx=0 tx=0).stop 2023-10-30T12:41:59.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.848+0000 7f31b9daa640 1 -- 172.21.15.125:0/3072559998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31b414b3e0 msgr2=0x7f31b4151680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:41:59.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.848+0000 7f31b9daa640 1 --2- 172.21.15.125:0/3072559998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31b414b3e0 0x7f31b4151680 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f31a801d2d0 tx=0x7f319c005800 comp rx=0 tx=0).stop 2023-10-30T12:41:59.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.848+0000 7f31b9daa640 1 -- 172.21.15.125:0/3072559998 shutdown_connections 2023-10-30T12:41:59.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.848+0000 7f31b9daa640 1 --2- 172.21.15.125:0/3072559998 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31a00664d0 0x7f31a0068990 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:59.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.848+0000 7f31b9daa640 1 --2- 172.21.15.125:0/3072559998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31b4151bc0 0x7f31b4146440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:59.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.848+0000 7f31b9daa640 1 --2- 172.21.15.125:0/3072559998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31b414b3e0 0x7f31b4151680 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:59.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.848+0000 7f31b9daa640 1 --2- 172.21.15.125:0/3072559998 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f319c003ec0 0x7f31b41579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:41:59.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.848+0000 7f31b9daa640 1 -- 172.21.15.125:0/3072559998 >> 172.21.15.125:0/3072559998 conn(0x7f31b40c7910 msgr2=0x7f31b414f880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:41:59.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.848+0000 7f31b9daa640 1 -- 172.21.15.125:0/3072559998 shutdown_connections 2023-10-30T12:41:59.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:41:59.848+0000 7f31b9daa640 1 -- 172.21.15.125:0/3072559998 wait complete. 2023-10-30T12:41:59.873 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=1) 2023-10-30T12:41:59.873 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T12:41:59.873 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:41:59.892 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T12:41:59.893 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:00.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.072+0000 7fcae7165640 1 Processor -- start 2023-10-30T12:42:00.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.072+0000 7fcae7165640 1 -- start start 2023-10-30T12:42:00.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.072+0000 7fcae7165640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcae00c6bd0 0x7fcae006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:00.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.072+0000 7fcae7165640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae0066cf0 con 0x7fcae006f7a0 2023-10-30T12:42:00.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.072+0000 7fcae7165640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcae00c70d0 con 0x7fcae0071110 2023-10-30T12:42:00.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.072+0000 7fcae7165640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae00700c0 con 0x7fcae00c6bd0 2023-10-30T12:42:00.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.072+0000 7fcadffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcae00c6bd0 0x7fcae006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:00.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.072+0000 7fcadffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcae00c6bd0 0x7fcae006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56682/0 (socket says 172.21.15.125:56682) 2023-10-30T12:42:00.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.072+0000 7fcadffff640 1 -- 172.21.15.125:0/1428352044 learned_addr learned my addr 172.21.15.125:0/1428352044 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:00.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.072+0000 7fcadffff640 1 -- 172.21.15.125:0/1428352044 >> v1:172.21.15.136:6789/0 conn(0x7fcae0071110 legacy=0x7fcae0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:00.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.072+0000 7fcadffff640 1 -- 172.21.15.125:0/1428352044 >> v1:172.21.15.125:6789/0 conn(0x7fcae006f7a0 legacy=0x7fcae0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:00.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.072+0000 7fcadffff640 1 -- 172.21.15.125:0/1428352044 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae00c7390 con 0x7fcae00c6bd0 2023-10-30T12:42:00.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.076+0000 7fcadffff640 1 --2- 172.21.15.125:0/1428352044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae00c6bd0 0x7fcae006fb80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fcacc009870 tx=0x7fcacc01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf852cba826b150 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:00.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.076+0000 7fcadf7fe640 1 -- 172.21.15.125:0/1428352044 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc0a8a30 con 0x7fcae00c6bd0 2023-10-30T12:42:00.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.076+0000 7fcadf7fe640 1 -- 172.21.15.125:0/1428352044 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcacc0a8b90 con 0x7fcae00c6bd0 2023-10-30T12:42:00.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.076+0000 7fcadf7fe640 1 -- 172.21.15.125:0/1428352044 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcacc0b0440 con 0x7fcae00c6bd0 2023-10-30T12:42:00.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.076+0000 7fcae7165640 1 -- 172.21.15.125:0/1428352044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae00c6bd0 msgr2=0x7fcae006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:00.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.076+0000 7fcae7165640 1 --2- 172.21.15.125:0/1428352044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae00c6bd0 0x7fcae006fb80 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fcacc009870 tx=0x7fcacc01ed30 comp rx=0 tx=0).stop 2023-10-30T12:42:00.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.076+0000 7fcae7165640 1 -- 172.21.15.125:0/1428352044 shutdown_connections 2023-10-30T12:42:00.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.076+0000 7fcae7165640 1 --2- 172.21.15.125:0/1428352044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae00c6bd0 0x7fcae006fb80 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:00.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.076+0000 7fcae7165640 1 -- 172.21.15.125:0/1428352044 >> 172.21.15.125:0/1428352044 conn(0x7fcae00c7910 msgr2=0x7fcae00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:00.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.076+0000 7fcae7165640 1 -- 172.21.15.125:0/1428352044 shutdown_connections 2023-10-30T12:42:00.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.076+0000 7fcae7165640 1 -- 172.21.15.125:0/1428352044 wait complete. 2023-10-30T12:42:00.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.080+0000 7fcae7165640 1 Processor -- start 2023-10-30T12:42:00.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.080+0000 7fcae7165640 1 -- start start 2023-10-30T12:42:00.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.080+0000 7fcae7165640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae006f7a0 0x7fcae0149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:00.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.080+0000 7fcae7165640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcae0071110 0x7fcae00c16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:00.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.080+0000 7fcae7165640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcae00c6bd0 0x7fcae01e3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:00.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.080+0000 7fcae7165640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae00b01f0 con 0x7fcae006f7a0 2023-10-30T12:42:00.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.080+0000 7fcae7165640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae0151f80 con 0x7fcae00c6bd0 2023-10-30T12:42:00.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.080+0000 7fcae7165640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae01f3360 con 0x7fcae0071110 2023-10-30T12:42:00.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.080+0000 7fcae4f01640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae006f7a0 0x7fcae0149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:00.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.080+0000 7fcae4f01640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae006f7a0 0x7fcae0149440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56696/0 (socket says 172.21.15.125:56696) 2023-10-30T12:42:00.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcae4f01640 1 -- 172.21.15.125:0/172817565 learned_addr learned my addr 172.21.15.125:0/172817565 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:00.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcae5702640 1 --2- 172.21.15.125:0/172817565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcae00c6bd0 0x7fcae01e3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:00.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcadffff640 1 --2- 172.21.15.125:0/172817565 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcae0071110 0x7fcae00c16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:00.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcae4f01640 1 -- 172.21.15.125:0/172817565 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcae0071110 msgr2=0x7fcae00c16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:00.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcae4f01640 1 --2- 172.21.15.125:0/172817565 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcae0071110 0x7fcae00c16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:00.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcae4f01640 1 -- 172.21.15.125:0/172817565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcae00c6bd0 msgr2=0x7fcae01e3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:00.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcae4f01640 1 --2- 172.21.15.125:0/172817565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcae00c6bd0 0x7fcae01e3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:00.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcae4f01640 1 -- 172.21.15.125:0/172817565 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcacc0094d0 con 0x7fcae006f7a0 2023-10-30T12:42:00.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcae4f01640 1 --2- 172.21.15.125:0/172817565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae006f7a0 0x7fcae0149440 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcad400c3c0 tx=0x7fcad400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:00.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcadd7fa640 1 -- 172.21.15.125:0/172817565 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad40052d0 con 0x7fcae006f7a0 2023-10-30T12:42:00.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcadd7fa640 1 -- 172.21.15.125:0/172817565 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad4005430 con 0x7fcae006f7a0 2023-10-30T12:42:00.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcae7165640 1 -- 172.21.15.125:0/172817565 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae00adb00 con 0x7fcae006f7a0 2023-10-30T12:42:00.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcadd7fa640 1 -- 172.21.15.125:0/172817565 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad4003840 con 0x7fcae006f7a0 2023-10-30T12:42:00.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcae7165640 1 -- 172.21.15.125:0/172817565 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae01f2ac0 con 0x7fcae006f7a0 2023-10-30T12:42:00.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcae7165640 1 -- 172.21.15.125:0/172817565 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae00ad8d0 con 0x7fcae006f7a0 2023-10-30T12:42:00.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.084+0000 7fcadd7fa640 1 -- 172.21.15.125:0/172817565 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcad40039a0 con 0x7fcae006f7a0 2023-10-30T12:42:00.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.088+0000 7fcadd7fa640 1 --2- 172.21.15.125:0/172817565 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcac40664f0 0x7fcac40689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:00.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.088+0000 7fcadd7fa640 1 -- 172.21.15.125:0/172817565 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7fcad40835a0 con 0x7fcae006f7a0 2023-10-30T12:42:00.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.088+0000 7fcadffff640 1 --2- 172.21.15.125:0/172817565 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcac40664f0 0x7fcac40689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:00.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.088+0000 7fcadffff640 1 --2- 172.21.15.125:0/172817565 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcac40664f0 0x7fcac40689b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcacc0b2000 tx=0x7fcacc0aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:00.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.088+0000 7fcadd7fa640 1 -- 172.21.15.125:0/172817565 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcae00ad8d0 con 0x7fcae006f7a0 2023-10-30T12:42:00.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.272+0000 7fcae7165640 1 -- 172.21.15.125:0/172817565 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcae004e170 con 0x7fcae006f7a0 2023-10-30T12:42:00.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.272+0000 7fcadd7fa640 1 -- 172.21.15.125:0/172817565 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fcad4050830 con 0x7fcae006f7a0 2023-10-30T12:42:00.276 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:00.276 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5345,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:31.350689+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:00.276 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 69 2023-10-30T12:42:00.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.276+0000 7fcae7165640 1 -- 172.21.15.125:0/172817565 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcac40664f0 msgr2=0x7fcac40689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:00.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.276+0000 7fcae7165640 1 --2- 172.21.15.125:0/172817565 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcac40664f0 0x7fcac40689b0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcacc0b2000 tx=0x7fcacc0aa070 comp rx=0 tx=0).stop 2023-10-30T12:42:00.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.276+0000 7fcae7165640 1 -- 172.21.15.125:0/172817565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae006f7a0 msgr2=0x7fcae0149440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:00.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.276+0000 7fcae7165640 1 --2- 172.21.15.125:0/172817565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae006f7a0 0x7fcae0149440 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fcad400c3c0 tx=0x7fcad400c880 comp rx=0 tx=0).stop 2023-10-30T12:42:00.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.276+0000 7fcae7165640 1 -- 172.21.15.125:0/172817565 shutdown_connections 2023-10-30T12:42:00.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.276+0000 7fcae7165640 1 --2- 172.21.15.125:0/172817565 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcac40664f0 0x7fcac40689b0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:00.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.276+0000 7fcae7165640 1 --2- 172.21.15.125:0/172817565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcae00c6bd0 0x7fcae01e3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:00.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.276+0000 7fcae7165640 1 --2- 172.21.15.125:0/172817565 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcae0071110 0x7fcae00c16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:00.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.276+0000 7fcae7165640 1 --2- 172.21.15.125:0/172817565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcae006f7a0 0x7fcae0149440 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:00.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.276+0000 7fcae7165640 1 -- 172.21.15.125:0/172817565 >> 172.21.15.125:0/172817565 conn(0x7fcae00c7910 msgr2=0x7fcae00c3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:00.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.276+0000 7fcae7165640 1 -- 172.21.15.125:0/172817565 shutdown_connections 2023-10-30T12:42:00.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:00.276+0000 7fcae7165640 1 -- 172.21.15.125:0/172817565 wait complete. 2023-10-30T12:42:00.297 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:42:02.298 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:02.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.464+0000 7fd64aa05640 1 Processor -- start 2023-10-30T12:42:02.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.464+0000 7fd64aa05640 1 -- start start 2023-10-30T12:42:02.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd64aa05640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd6441538a0 0x7fd64415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:02.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd64aa05640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd6440ca0e0 0x7fd64415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:02.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd64aa05640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6440cf160 0x7fd6440cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:02.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd64aa05640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6440cb800 con 0x7fd6441538a0 2023-10-30T12:42:02.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd64aa05640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd64415dc30 con 0x7fd6440ca0e0 2023-10-30T12:42:02.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd64aa05640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6440cb330 con 0x7fd6440cf160 2023-10-30T12:42:02.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd643fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd6441538a0 0x7fd64415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:02.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd643fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd6441538a0 0x7fd64415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56064/0 (socket says 172.21.15.125:56064) 2023-10-30T12:42:02.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd643fff640 1 -- 172.21.15.125:0/2756614672 learned_addr learned my addr 172.21.15.125:0/2756614672 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:02.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd6437fe640 1 --2- 172.21.15.125:0/2756614672 >> v2:172.21.15.136:3300/0 conn(0x7fd6440ca0e0 0x7fd64415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:02.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd648fa2640 1 --2- 172.21.15.125:0/2756614672 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6440cf160 0x7fd6440cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:02.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd643fff640 1 -- 172.21.15.125:0/2756614672 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6440cf160 msgr2=0x7fd6440cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:02.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd643fff640 1 --2- 172.21.15.125:0/2756614672 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6440cf160 0x7fd6440cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:02.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd643fff640 1 -- 172.21.15.125:0/2756614672 >> v2:172.21.15.136:3300/0 conn(0x7fd6440ca0e0 msgr2=0x7fd64415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:02.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd643fff640 1 --2- 172.21.15.125:0/2756614672 >> v2:172.21.15.136:3300/0 conn(0x7fd6440ca0e0 0x7fd64415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:02.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd643fff640 1 -- 172.21.15.125:0/2756614672 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6440cad70 con 0x7fd6441538a0 2023-10-30T12:42:02.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.468+0000 7fd643fff640 1 --2- 172.21.15.125:0/2756614672 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6441538a0 0x7fd64415d6f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd6440b30a0 tx=0x7fd63401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b11a97edcfd0b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:02.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd642ffd640 1 -- 172.21.15.125:0/2756614672 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6340a89b0 con 0x7fd6441538a0 2023-10-30T12:42:02.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd642ffd640 1 -- 172.21.15.125:0/2756614672 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6340a8b10 con 0x7fd6441538a0 2023-10-30T12:42:02.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd642ffd640 1 -- 172.21.15.125:0/2756614672 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6340a8ea0 con 0x7fd6441538a0 2023-10-30T12:42:02.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 -- 172.21.15.125:0/2756614672 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6441538a0 msgr2=0x7fd64415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:02.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 --2- 172.21.15.125:0/2756614672 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6441538a0 0x7fd64415d6f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd6440b30a0 tx=0x7fd63401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:42:02.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 -- 172.21.15.125:0/2756614672 shutdown_connections 2023-10-30T12:42:02.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 --2- 172.21.15.125:0/2756614672 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6440cf160 0x7fd6440cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:02.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 --2- 172.21.15.125:0/2756614672 >> v2:172.21.15.136:3300/0 conn(0x7fd6440ca0e0 0x7fd64415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:02.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 --2- 172.21.15.125:0/2756614672 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6441538a0 0x7fd64415d6f0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:02.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 -- 172.21.15.125:0/2756614672 >> 172.21.15.125:0/2756614672 conn(0x7fd64406bfe0 msgr2=0x7fd64406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:02.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 -- 172.21.15.125:0/2756614672 shutdown_connections 2023-10-30T12:42:02.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 -- 172.21.15.125:0/2756614672 wait complete. 2023-10-30T12:42:02.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 Processor -- start 2023-10-30T12:42:02.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 -- start start 2023-10-30T12:42:02.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6440ca0e0 0x7fd644142d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:02.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6440cf160 0x7fd6441432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:02.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6441538a0 0x7fd6440b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:02.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd644158bb0 con 0x7fd6441538a0 2023-10-30T12:42:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd644159520 con 0x7fd6440ca0e0 2023-10-30T12:42:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.472+0000 7fd64aa05640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6441508e0 con 0x7fd6440cf160 2023-10-30T12:42:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd648fa2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6441538a0 0x7fd6440b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd648fa2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6441538a0 0x7fd6440b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56070/0 (socket says 172.21.15.125:56070) 2023-10-30T12:42:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd648fa2640 1 -- 172.21.15.125:0/4016833616 learned_addr learned my addr 172.21.15.125:0/4016833616 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:02.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd6437fe640 1 --2- 172.21.15.125:0/4016833616 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6440cf160 0x7fd6441432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:02.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd643fff640 1 --2- 172.21.15.125:0/4016833616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6440ca0e0 0x7fd644142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:02.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd648fa2640 1 -- 172.21.15.125:0/4016833616 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6440cf160 msgr2=0x7fd6441432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:02.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd648fa2640 1 --2- 172.21.15.125:0/4016833616 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6440cf160 0x7fd6441432a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:02.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd648fa2640 1 -- 172.21.15.125:0/4016833616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6440ca0e0 msgr2=0x7fd644142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:02.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd648fa2640 1 --2- 172.21.15.125:0/4016833616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6440ca0e0 0x7fd644142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:02.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd648fa2640 1 -- 172.21.15.125:0/4016833616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6340094d0 con 0x7fd6441538a0 2023-10-30T12:42:02.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd648fa2640 1 --2- 172.21.15.125:0/4016833616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6441538a0 0x7fd6440b4450 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd63c00efc0 tx=0x7fd63c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:02.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd640ff9640 1 -- 172.21.15.125:0/4016833616 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c009b10 con 0x7fd6441538a0 2023-10-30T12:42:02.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd640ff9640 1 -- 172.21.15.125:0/4016833616 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63c00f040 con 0x7fd6441538a0 2023-10-30T12:42:02.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd64aa05640 1 -- 172.21.15.125:0/4016833616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd644150bc0 con 0x7fd6441538a0 2023-10-30T12:42:02.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd640ff9640 1 -- 172.21.15.125:0/4016833616 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c0146c0 con 0x7fd6441538a0 2023-10-30T12:42:02.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd64aa05640 1 -- 172.21.15.125:0/4016833616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd644158e00 con 0x7fd6441538a0 2023-10-30T12:42:02.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd64aa05640 1 -- 172.21.15.125:0/4016833616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6440b2a50 con 0x7fd6441538a0 2023-10-30T12:42:02.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd640ff9640 1 -- 172.21.15.125:0/4016833616 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd63c014820 con 0x7fd6441538a0 2023-10-30T12:42:02.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.476+0000 7fd640ff9640 1 --2- 172.21.15.125:0/4016833616 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6300664a0 0x7fd630068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:02.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.480+0000 7fd640ff9640 1 -- 172.21.15.125:0/4016833616 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7fd63c084980 con 0x7fd6441538a0 2023-10-30T12:42:02.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.480+0000 7fd643fff640 1 --2- 172.21.15.125:0/4016833616 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6300664a0 0x7fd630068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:02.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.480+0000 7fd643fff640 1 --2- 172.21.15.125:0/4016833616 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6300664a0 0x7fd630068960 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd6340b2000 tx=0x7fd634009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:02.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.484+0000 7fd640ff9640 1 -- 172.21.15.125:0/4016833616 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd63c051d70 con 0x7fd6441538a0 2023-10-30T12:42:02.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.664+0000 7fd64aa05640 1 -- 172.21.15.125:0/4016833616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd644159290 con 0x7fd6441538a0 2023-10-30T12:42:02.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.664+0000 7fd640ff9640 1 -- 172.21.15.125:0/4016833616 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fd63c018090 con 0x7fd6441538a0 2023-10-30T12:42:02.669 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:02.669 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5345,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:31.350689+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:02.669 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 69 2023-10-30T12:42:02.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.668+0000 7fd64aa05640 1 -- 172.21.15.125:0/4016833616 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6300664a0 msgr2=0x7fd630068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:02.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.668+0000 7fd64aa05640 1 --2- 172.21.15.125:0/4016833616 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6300664a0 0x7fd630068960 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd6340b2000 tx=0x7fd634009040 comp rx=0 tx=0).stop 2023-10-30T12:42:02.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.668+0000 7fd64aa05640 1 -- 172.21.15.125:0/4016833616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6441538a0 msgr2=0x7fd6440b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:02.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.668+0000 7fd64aa05640 1 --2- 172.21.15.125:0/4016833616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6441538a0 0x7fd6440b4450 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd63c00efc0 tx=0x7fd63c00c490 comp rx=0 tx=0).stop 2023-10-30T12:42:02.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.668+0000 7fd64aa05640 1 -- 172.21.15.125:0/4016833616 shutdown_connections 2023-10-30T12:42:02.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.668+0000 7fd64aa05640 1 --2- 172.21.15.125:0/4016833616 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6300664a0 0x7fd630068960 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:02.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.668+0000 7fd64aa05640 1 --2- 172.21.15.125:0/4016833616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6441538a0 0x7fd6440b4450 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:02.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.668+0000 7fd64aa05640 1 --2- 172.21.15.125:0/4016833616 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6440cf160 0x7fd6441432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:02.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.668+0000 7fd64aa05640 1 --2- 172.21.15.125:0/4016833616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6440ca0e0 0x7fd644142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:02.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.668+0000 7fd64aa05640 1 -- 172.21.15.125:0/4016833616 >> 172.21.15.125:0/4016833616 conn(0x7fd64406bfe0 msgr2=0x7fd644066410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:02.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.668+0000 7fd64aa05640 1 -- 172.21.15.125:0/4016833616 shutdown_connections 2023-10-30T12:42:02.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:02.668+0000 7fd64aa05640 1 -- 172.21.15.125:0/4016833616 wait complete. 2023-10-30T12:42:02.689 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:42:04.691 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:04.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff515067640 1 Processor -- start 2023-10-30T12:42:04.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff515067640 1 -- start start 2023-10-30T12:42:04.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff515067640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff5101538f0 0x7ff510159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:04.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff515067640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff51015cd50 0x7ff5100c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:04.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff515067640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff51015d300 con 0x7ff5100cefa0 2023-10-30T12:42:04.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff515067640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51015d5e0 con 0x7ff5101538f0 2023-10-30T12:42:04.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff515067640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff51015d8c0 con 0x7ff51015cd50 2023-10-30T12:42:04.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff50ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff5101538f0 0x7ff510159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:04.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff50ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff5101538f0 0x7ff510159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56072/0 (socket says 172.21.15.125:56072) 2023-10-30T12:42:04.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff50ffff640 1 -- 172.21.15.125:0/1676227437 learned_addr learned my addr 172.21.15.125:0/1676227437 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:04.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff50f7fe640 1 --2- 172.21.15.125:0/1676227437 >> v2:172.21.15.136:3300/0 conn(0x7ff51015cd50 0x7ff5100c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:04.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff50ffff640 1 -- 172.21.15.125:0/1676227437 >> v2:172.21.15.136:3300/0 conn(0x7ff51015cd50 msgr2=0x7ff5100c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:04.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff50ffff640 1 --2- 172.21.15.125:0/1676227437 >> v2:172.21.15.136:3300/0 conn(0x7ff51015cd50 0x7ff5100c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:04.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff50ffff640 1 -- 172.21.15.125:0/1676227437 >> v1:172.21.15.136:6789/0 conn(0x7ff5100cefa0 legacy=0x7ff5100cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:04.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff50ffff640 1 -- 172.21.15.125:0/1676227437 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51015dba0 con 0x7ff5101538f0 2023-10-30T12:42:04.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.836+0000 7ff50f7fe640 1 --2- 172.21.15.125:0/1676227437 >> v2:172.21.15.136:3300/0 conn(0x7ff51015cd50 0x7ff5100c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:42:04.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.840+0000 7ff50ffff640 1 --2- 172.21.15.125:0/1676227437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5101538f0 0x7ff510159fb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff510072e90 tx=0x7ff50001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1286c44b5167b4fb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:04.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.840+0000 7ff50effd640 1 -- 172.21.15.125:0/1676227437 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5000a89b0 con 0x7ff5101538f0 2023-10-30T12:42:04.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.840+0000 7ff50effd640 1 -- 172.21.15.125:0/1676227437 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5000a8b10 con 0x7ff5101538f0 2023-10-30T12:42:04.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.840+0000 7ff50effd640 1 -- 172.21.15.125:0/1676227437 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5000a8ea0 con 0x7ff5101538f0 2023-10-30T12:42:04.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.840+0000 7ff515067640 1 -- 172.21.15.125:0/1676227437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5101538f0 msgr2=0x7ff510159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:04.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.840+0000 7ff515067640 1 --2- 172.21.15.125:0/1676227437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5101538f0 0x7ff510159fb0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7ff510072e90 tx=0x7ff50001ecb0 comp rx=0 tx=0).stop 2023-10-30T12:42:04.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.840+0000 7ff515067640 1 -- 172.21.15.125:0/1676227437 shutdown_connections 2023-10-30T12:42:04.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.840+0000 7ff515067640 1 --2- 172.21.15.125:0/1676227437 >> v2:172.21.15.136:3300/0 conn(0x7ff51015cd50 0x7ff5100c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:04.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.840+0000 7ff515067640 1 --2- 172.21.15.125:0/1676227437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5101538f0 0x7ff510159fb0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:04.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.840+0000 7ff515067640 1 -- 172.21.15.125:0/1676227437 >> 172.21.15.125:0/1676227437 conn(0x7ff51006a640 msgr2=0x7ff510065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:04.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.840+0000 7ff515067640 1 -- 172.21.15.125:0/1676227437 shutdown_connections 2023-10-30T12:42:04.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.840+0000 7ff515067640 1 -- 172.21.15.125:0/1676227437 wait complete. 2023-10-30T12:42:04.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff515067640 1 Processor -- start 2023-10-30T12:42:04.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff515067640 1 -- start start 2023-10-30T12:42:04.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff515067640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5100cefa0 0x7ff5100b3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:04.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff515067640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5101538f0 0x7ff5101efa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:04.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff515067640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff51015cd50 0x7ff5101eff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:04.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff515067640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff510158ea0 con 0x7ff5101538f0 2023-10-30T12:42:04.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff515067640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5100b3530 con 0x7ff5100cefa0 2023-10-30T12:42:04.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff515067640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5101ef580 con 0x7ff51015cd50 2023-10-30T12:42:04.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff50f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5101538f0 0x7ff5101efa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:04.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff50f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5101538f0 0x7ff5101efa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56078/0 (socket says 172.21.15.125:56078) 2023-10-30T12:42:04.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff50f7fe640 1 -- 172.21.15.125:0/915321110 learned_addr learned my addr 172.21.15.125:0/915321110 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:04.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff50ffff640 1 --2- 172.21.15.125:0/915321110 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5100cefa0 0x7ff5100b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:04.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff514866640 1 --2- 172.21.15.125:0/915321110 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff51015cd50 0x7ff5101eff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:04.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff50f7fe640 1 -- 172.21.15.125:0/915321110 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff51015cd50 msgr2=0x7ff5101eff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff50f7fe640 1 --2- 172.21.15.125:0/915321110 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff51015cd50 0x7ff5101eff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff50f7fe640 1 -- 172.21.15.125:0/915321110 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5100cefa0 msgr2=0x7ff5100b3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff50f7fe640 1 --2- 172.21.15.125:0/915321110 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5100cefa0 0x7ff5100b3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff50f7fe640 1 -- 172.21.15.125:0/915321110 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5000094d0 con 0x7ff5101538f0 2023-10-30T12:42:04.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff50f7fe640 1 --2- 172.21.15.125:0/915321110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5101538f0 0x7ff5101efa10 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff50400c7f0 tx=0x7ff50400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:04.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff50cff9640 1 -- 172.21.15.125:0/915321110 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504007820 con 0x7ff5101538f0 2023-10-30T12:42:04.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff50cff9640 1 -- 172.21.15.125:0/915321110 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff504007980 con 0x7ff5101538f0 2023-10-30T12:42:04.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff50cff9640 1 -- 172.21.15.125:0/915321110 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff504007d30 con 0x7ff5101538f0 2023-10-30T12:42:04.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff515067640 1 -- 172.21.15.125:0/915321110 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5100b4fb0 con 0x7ff5101538f0 2023-10-30T12:42:04.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff515067640 1 -- 172.21.15.125:0/915321110 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5100b4710 con 0x7ff5101538f0 2023-10-30T12:42:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.844+0000 7ff515067640 1 -- 172.21.15.125:0/915321110 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5101ef730 con 0x7ff5101538f0 2023-10-30T12:42:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.848+0000 7ff50cff9640 1 -- 172.21.15.125:0/915321110 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5040113f0 con 0x7ff5101538f0 2023-10-30T12:42:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.848+0000 7ff50cff9640 1 --2- 172.21.15.125:0/915321110 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4fc0664a0 0x7ff4fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.848+0000 7ff50cff9640 1 -- 172.21.15.125:0/915321110 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7ff5040844e0 con 0x7ff5101538f0 2023-10-30T12:42:04.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.848+0000 7ff50ffff640 1 --2- 172.21.15.125:0/915321110 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4fc0664a0 0x7ff4fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.848+0000 7ff50ffff640 1 --2- 172.21.15.125:0/915321110 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4fc0664a0 0x7ff4fc068960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff5000b2000 tx=0x7ff500008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:04.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:04.852+0000 7ff50cff9640 1 -- 172.21.15.125:0/915321110 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5101ef730 con 0x7ff5101538f0 2023-10-30T12:42:05.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:05.036+0000 7ff515067640 1 -- 172.21.15.125:0/915321110 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5101ef730 con 0x7ff5101538f0 2023-10-30T12:42:05.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:05.036+0000 7ff50cff9640 1 -- 172.21.15.125:0/915321110 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6531 (secure 0 0 0) 0x7ff5101ef730 con 0x7ff5101538f0 2023-10-30T12:42:05.040 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:05.040 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5345,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:31.350689+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:05.041 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 69 2023-10-30T12:42:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:05.040+0000 7ff515067640 1 -- 172.21.15.125:0/915321110 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4fc0664a0 msgr2=0x7ff4fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:05.040+0000 7ff515067640 1 --2- 172.21.15.125:0/915321110 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4fc0664a0 0x7ff4fc068960 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff5000b2000 tx=0x7ff500008040 comp rx=0 tx=0).stop 2023-10-30T12:42:05.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:05.040+0000 7ff515067640 1 -- 172.21.15.125:0/915321110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5101538f0 msgr2=0x7ff5101efa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:05.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:05.040+0000 7ff515067640 1 --2- 172.21.15.125:0/915321110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5101538f0 0x7ff5101efa10 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7ff50400c7f0 tx=0x7ff50400ccb0 comp rx=0 tx=0).stop 2023-10-30T12:42:05.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:05.040+0000 7ff515067640 1 -- 172.21.15.125:0/915321110 shutdown_connections 2023-10-30T12:42:05.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:05.040+0000 7ff515067640 1 --2- 172.21.15.125:0/915321110 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4fc0664a0 0x7ff4fc068960 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:05.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:05.040+0000 7ff515067640 1 --2- 172.21.15.125:0/915321110 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff51015cd50 0x7ff5101eff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:05.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:05.040+0000 7ff515067640 1 --2- 172.21.15.125:0/915321110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5101538f0 0x7ff5101efa10 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:05.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:05.040+0000 7ff515067640 1 --2- 172.21.15.125:0/915321110 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5100cefa0 0x7ff5100b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:05.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:05.040+0000 7ff515067640 1 -- 172.21.15.125:0/915321110 >> 172.21.15.125:0/915321110 conn(0x7ff51006a640 msgr2=0x7ff5100c3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:05.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:05.040+0000 7ff515067640 1 -- 172.21.15.125:0/915321110 shutdown_connections 2023-10-30T12:42:05.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:05.040+0000 7ff515067640 1 -- 172.21.15.125:0/915321110 wait complete. 2023-10-30T12:42:05.061 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:42:07.061 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:07.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.204+0000 7f14ed29a640 1 Processor -- start 2023-10-30T12:42:07.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.204+0000 7f14ed29a640 1 -- start start 2023-10-30T12:42:07.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.208+0000 7f14ed29a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f14e81538a0 0x7f14e815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:07.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.208+0000 7f14ed29a640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e80cb800 con 0x7f14e80ca0e0 2023-10-30T12:42:07.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.208+0000 7f14ed29a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14e815dc30 con 0x7f14e80cf160 2023-10-30T12:42:07.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.208+0000 7f14ed29a640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14e80cb330 con 0x7f14e81538a0 2023-10-30T12:42:07.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.208+0000 7f14e6ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f14e81538a0 0x7f14e815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:07.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.208+0000 7f14e6ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f14e81538a0 0x7f14e815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56088/0 (socket says 172.21.15.125:56088) 2023-10-30T12:42:07.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.208+0000 7f14e6ffd640 1 -- 172.21.15.125:0/177417835 learned_addr learned my addr 172.21.15.125:0/177417835 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:07.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.208+0000 7f14e6ffd640 1 -- 172.21.15.125:0/177417835 >> v1:172.21.15.136:6789/0 conn(0x7f14e80cf160 legacy=0x7f14e8164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:07.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.208+0000 7f14e6ffd640 1 -- 172.21.15.125:0/177417835 >> v1:172.21.15.125:6789/0 conn(0x7f14e80ca0e0 legacy=0x7f14e815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:07.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.208+0000 7f14e6ffd640 1 -- 172.21.15.125:0/177417835 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14e80cad70 con 0x7f14e81538a0 2023-10-30T12:42:07.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.212+0000 7f14e6ffd640 1 --2- 172.21.15.125:0/177417835 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14e81538a0 0x7f14e815d6f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009870 tx=0x7f14d801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2f6c96d846d0d82 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:07.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.212+0000 7f14e67fc640 1 -- 172.21.15.125:0/177417835 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d80a8a30 con 0x7f14e81538a0 2023-10-30T12:42:07.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.212+0000 7f14e67fc640 1 -- 172.21.15.125:0/177417835 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d80a8b90 con 0x7f14e81538a0 2023-10-30T12:42:07.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.212+0000 7f14e67fc640 1 -- 172.21.15.125:0/177417835 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d80b0440 con 0x7f14e81538a0 2023-10-30T12:42:07.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.212+0000 7f14ed29a640 1 -- 172.21.15.125:0/177417835 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14e81538a0 msgr2=0x7f14e815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:07.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.212+0000 7f14ed29a640 1 --2- 172.21.15.125:0/177417835 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14e81538a0 0x7f14e815d6f0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f14d8009870 tx=0x7f14d801ed30 comp rx=0 tx=0).stop 2023-10-30T12:42:07.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.212+0000 7f14ed29a640 1 -- 172.21.15.125:0/177417835 shutdown_connections 2023-10-30T12:42:07.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.212+0000 7f14ed29a640 1 --2- 172.21.15.125:0/177417835 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14e81538a0 0x7f14e815d6f0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:07.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.212+0000 7f14ed29a640 1 -- 172.21.15.125:0/177417835 >> 172.21.15.125:0/177417835 conn(0x7f14e806bfe0 msgr2=0x7f14e806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:07.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.212+0000 7f14ed29a640 1 -- 172.21.15.125:0/177417835 shutdown_connections 2023-10-30T12:42:07.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.212+0000 7f14ed29a640 1 -- 172.21.15.125:0/177417835 wait complete. 2023-10-30T12:42:07.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.212+0000 7f14ed29a640 1 Processor -- start 2023-10-30T12:42:07.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.212+0000 7f14ed29a640 1 -- start start 2023-10-30T12:42:07.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.212+0000 7f14ed29a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14e80ca0e0 0x7f14e81e2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:07.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14ed29a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14e80cf160 0x7f14e8157f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:07.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14ed29a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14e81538a0 0x7f14e80b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:07.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14ed29a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e80c9670 con 0x7f14e80cf160 2023-10-30T12:42:07.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14ed29a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14e8158470 con 0x7f14e81538a0 2023-10-30T12:42:07.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14ed29a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14e81e5340 con 0x7f14e80ca0e0 2023-10-30T12:42:07.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14dffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14e80cf160 0x7f14e8157f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:07.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14dffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14e80cf160 0x7f14e8157f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56092/0 (socket says 172.21.15.125:56092) 2023-10-30T12:42:07.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14dffff640 1 -- 172.21.15.125:0/3206033516 learned_addr learned my addr 172.21.15.125:0/3206033516 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:07.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14e6ffd640 1 --2- 172.21.15.125:0/3206033516 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14e80ca0e0 0x7f14e81e2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:07.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14e77fe640 1 --2- 172.21.15.125:0/3206033516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14e81538a0 0x7f14e80b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:07.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14dffff640 1 -- 172.21.15.125:0/3206033516 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14e80ca0e0 msgr2=0x7f14e81e2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:07.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14dffff640 1 --2- 172.21.15.125:0/3206033516 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14e80ca0e0 0x7f14e81e2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:07.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14dffff640 1 -- 172.21.15.125:0/3206033516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14e81538a0 msgr2=0x7f14e80b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:07.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14dffff640 1 --2- 172.21.15.125:0/3206033516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14e81538a0 0x7f14e80b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:07.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14dffff640 1 -- 172.21.15.125:0/3206033516 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14d80094d0 con 0x7f14e80cf160 2023-10-30T12:42:07.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14dffff640 1 --2- 172.21.15.125:0/3206033516 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14e80cf160 0x7f14e8157f30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f14d000c3f0 tx=0x7f14d000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:07.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14df7fe640 1 -- 172.21.15.125:0/3206033516 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d00052d0 con 0x7f14e80cf160 2023-10-30T12:42:07.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14df7fe640 1 -- 172.21.15.125:0/3206033516 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14d0005430 con 0x7f14e80cf160 2023-10-30T12:42:07.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14df7fe640 1 -- 172.21.15.125:0/3206033516 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d0003840 con 0x7f14e80cf160 2023-10-30T12:42:07.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14ed29a640 1 -- 172.21.15.125:0/3206033516 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14e81e4eb0 con 0x7f14e80cf160 2023-10-30T12:42:07.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14ed29a640 1 -- 172.21.15.125:0/3206033516 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14e81e42a0 con 0x7f14e80cf160 2023-10-30T12:42:07.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14ed29a640 1 -- 172.21.15.125:0/3206033516 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14e81e5040 con 0x7f14e80cf160 2023-10-30T12:42:07.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14df7fe640 1 -- 172.21.15.125:0/3206033516 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f14d00039a0 con 0x7f14e80cf160 2023-10-30T12:42:07.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.216+0000 7f14df7fe640 1 --2- 172.21.15.125:0/3206033516 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14c80664a0 0x7f14c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:07.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.220+0000 7f14df7fe640 1 -- 172.21.15.125:0/3206033516 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f14d00836f0 con 0x7f14e80cf160 2023-10-30T12:42:07.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.220+0000 7f14e6ffd640 1 --2- 172.21.15.125:0/3206033516 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14c80664a0 0x7f14c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:07.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.220+0000 7f14e6ffd640 1 --2- 172.21.15.125:0/3206033516 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14c80664a0 0x7f14c8068960 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f14d80b2000 tx=0x7f14d8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:07.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.224+0000 7f14df7fe640 1 -- 172.21.15.125:0/3206033516 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f14d0050870 con 0x7f14e80cf160 2023-10-30T12:42:07.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.404+0000 7f14ed29a640 1 -- 172.21.15.125:0/3206033516 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14e8067a30 con 0x7f14e80cf160 2023-10-30T12:42:07.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.404+0000 7f14df7fe640 1 -- 172.21.15.125:0/3206033516 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f14d0053e90 con 0x7f14e80cf160 2023-10-30T12:42:07.408 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:07.409 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5345,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:31.350689+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:07.410 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 69 2023-10-30T12:42:07.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.408+0000 7f14ed29a640 1 -- 172.21.15.125:0/3206033516 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14c80664a0 msgr2=0x7f14c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:07.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.408+0000 7f14ed29a640 1 --2- 172.21.15.125:0/3206033516 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14c80664a0 0x7f14c8068960 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f14d80b2000 tx=0x7f14d8008040 comp rx=0 tx=0).stop 2023-10-30T12:42:07.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.408+0000 7f14ed29a640 1 -- 172.21.15.125:0/3206033516 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14e80cf160 msgr2=0x7f14e8157f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:07.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.408+0000 7f14ed29a640 1 --2- 172.21.15.125:0/3206033516 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14e80cf160 0x7f14e8157f30 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f14d000c3f0 tx=0x7f14d000c8b0 comp rx=0 tx=0).stop 2023-10-30T12:42:07.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.408+0000 7f14ed29a640 1 -- 172.21.15.125:0/3206033516 shutdown_connections 2023-10-30T12:42:07.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.408+0000 7f14ed29a640 1 --2- 172.21.15.125:0/3206033516 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14c80664a0 0x7f14c8068960 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:07.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.408+0000 7f14ed29a640 1 --2- 172.21.15.125:0/3206033516 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14e81538a0 0x7f14e80b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:07.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.408+0000 7f14ed29a640 1 --2- 172.21.15.125:0/3206033516 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14e80cf160 0x7f14e8157f30 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:07.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.408+0000 7f14ed29a640 1 --2- 172.21.15.125:0/3206033516 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14e80ca0e0 0x7f14e81e2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:07.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.408+0000 7f14ed29a640 1 -- 172.21.15.125:0/3206033516 >> 172.21.15.125:0/3206033516 conn(0x7f14e806bfe0 msgr2=0x7f14e80c54a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:07.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.408+0000 7f14ed29a640 1 -- 172.21.15.125:0/3206033516 shutdown_connections 2023-10-30T12:42:07.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:07.408+0000 7f14ed29a640 1 -- 172.21.15.125:0/3206033516 wait complete. 2023-10-30T12:42:07.429 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:42:09.430 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:09.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.572+0000 7f658acc9640 1 Processor -- start 2023-10-30T12:42:09.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.572+0000 7f658acc9640 1 -- start start 2023-10-30T12:42:09.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.576+0000 7f658acc9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6584153bf0 0x7f6584071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:09.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.576+0000 7f658acc9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65840c6fe0 0x7f65840c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:09.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.576+0000 7f658acc9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65840700b0 con 0x7f65840c79e0 2023-10-30T12:42:09.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.576+0000 7f658acc9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65840c6810 con 0x7f6584153bf0 2023-10-30T12:42:09.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.576+0000 7f658acc9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6584159900 con 0x7f65840c6fe0 2023-10-30T12:42:09.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.576+0000 7f6588a65640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6584153bf0 0x7f6584071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.576+0000 7f6588a65640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6584153bf0 0x7f6584071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56094/0 (socket says 172.21.15.125:56094) 2023-10-30T12:42:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.576+0000 7f6588a65640 1 -- 172.21.15.125:0/1022172772 learned_addr learned my addr 172.21.15.125:0/1022172772 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.576+0000 7f657bfff640 1 --2- 172.21.15.125:0/1022172772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65840c6fe0 0x7f65840c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.576+0000 7f6588a65640 1 -- 172.21.15.125:0/1022172772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65840c6fe0 msgr2=0x7f65840c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:09.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.576+0000 7f6588a65640 1 --2- 172.21.15.125:0/1022172772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65840c6fe0 0x7f65840c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.576+0000 7f6588a65640 1 -- 172.21.15.125:0/1022172772 >> v1:172.21.15.136:6789/0 conn(0x7f65840c79e0 legacy=0x7f65840cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:09.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.576+0000 7f6588a65640 1 -- 172.21.15.125:0/1022172772 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65840c6530 con 0x7f6584153bf0 2023-10-30T12:42:09.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.576+0000 7f657bfff640 1 --2- 172.21.15.125:0/1022172772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65840c6fe0 0x7f65840c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:42:09.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f6588a65640 1 --2- 172.21.15.125:0/1022172772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6584153bf0 0x7f6584071260 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f6580009870 tx=0x7f658001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e734cba5597c0bfd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f657b7fe640 1 -- 172.21.15.125:0/1022172772 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65800a89b0 con 0x7f6584153bf0 2023-10-30T12:42:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f657b7fe640 1 -- 172.21.15.125:0/1022172772 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65800a8b10 con 0x7f6584153bf0 2023-10-30T12:42:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f657b7fe640 1 -- 172.21.15.125:0/1022172772 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65800a8ea0 con 0x7f6584153bf0 2023-10-30T12:42:09.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f658acc9640 1 -- 172.21.15.125:0/1022172772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6584153bf0 msgr2=0x7f6584071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:09.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f658acc9640 1 --2- 172.21.15.125:0/1022172772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6584153bf0 0x7f6584071260 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f6580009870 tx=0x7f658001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f658acc9640 1 -- 172.21.15.125:0/1022172772 shutdown_connections 2023-10-30T12:42:09.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f658acc9640 1 --2- 172.21.15.125:0/1022172772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65840c6fe0 0x7f65840c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f658acc9640 1 --2- 172.21.15.125:0/1022172772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6584153bf0 0x7f6584071260 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f658acc9640 1 -- 172.21.15.125:0/1022172772 >> 172.21.15.125:0/1022172772 conn(0x7f658406c6e0 msgr2=0x7f6584069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:09.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f658acc9640 1 -- 172.21.15.125:0/1022172772 shutdown_connections 2023-10-30T12:42:09.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f658acc9640 1 -- 172.21.15.125:0/1022172772 wait complete. 2023-10-30T12:42:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f658acc9640 1 Processor -- start 2023-10-30T12:42:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f658acc9640 1 -- start start 2023-10-30T12:42:09.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f658acc9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65840c6fe0 0x7f65840c4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:09.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f658acc9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65840c79e0 0x7f6584154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:09.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.580+0000 7f658acc9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6584153bf0 0x7f65841f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:09.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f658acc9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658414e370 con 0x7f65840c79e0 2023-10-30T12:42:09.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f658acc9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65840c4070 con 0x7f6584153bf0 2023-10-30T12:42:09.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f658acc9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f658414dea0 con 0x7f65840c6fe0 2023-10-30T12:42:09.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f657bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65840c79e0 0x7f6584154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:09.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f657bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65840c79e0 0x7f6584154a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56108/0 (socket says 172.21.15.125:56108) 2023-10-30T12:42:09.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f657bfff640 1 -- 172.21.15.125:0/413215166 learned_addr learned my addr 172.21.15.125:0/413215166 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:09.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f6588a65640 1 --2- 172.21.15.125:0/413215166 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65840c6fe0 0x7f65840c4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:09.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f6589266640 1 --2- 172.21.15.125:0/413215166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6584153bf0 0x7f65841f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:09.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f657bfff640 1 -- 172.21.15.125:0/413215166 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65840c6fe0 msgr2=0x7f65840c4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:09.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f657bfff640 1 --2- 172.21.15.125:0/413215166 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65840c6fe0 0x7f65840c4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f657bfff640 1 -- 172.21.15.125:0/413215166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6584153bf0 msgr2=0x7f65841f28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:09.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f657bfff640 1 --2- 172.21.15.125:0/413215166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6584153bf0 0x7f65841f28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f657bfff640 1 -- 172.21.15.125:0/413215166 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65800094d0 con 0x7f65840c79e0 2023-10-30T12:42:09.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f657bfff640 1 --2- 172.21.15.125:0/413215166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65840c79e0 0x7f6584154a60 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f657400c7c0 tx=0x7f657400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:09.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f65797fa640 1 -- 172.21.15.125:0/413215166 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6574007a30 con 0x7f65840c79e0 2023-10-30T12:42:09.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f65797fa640 1 -- 172.21.15.125:0/413215166 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6574007b90 con 0x7f65840c79e0 2023-10-30T12:42:09.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f65797fa640 1 -- 172.21.15.125:0/413215166 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657400ce70 con 0x7f65840c79e0 2023-10-30T12:42:09.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f658acc9640 1 -- 172.21.15.125:0/413215166 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65841f3560 con 0x7f65840c79e0 2023-10-30T12:42:09.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f658acc9640 1 -- 172.21.15.125:0/413215166 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65841f3280 con 0x7f65840c79e0 2023-10-30T12:42:09.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f658acc9640 1 -- 172.21.15.125:0/413215166 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65840c41b0 con 0x7f65840c79e0 2023-10-30T12:42:09.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.584+0000 7f65797fa640 1 -- 172.21.15.125:0/413215166 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f65740113f0 con 0x7f65840c79e0 2023-10-30T12:42:09.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.588+0000 7f65797fa640 1 --2- 172.21.15.125:0/413215166 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65700664d0 0x7f6570068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:09.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.588+0000 7f65797fa640 1 -- 172.21.15.125:0/413215166 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f65740843d0 con 0x7f65840c79e0 2023-10-30T12:42:09.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.588+0000 7f6588a65640 1 --2- 172.21.15.125:0/413215166 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65700664d0 0x7f6570068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:09.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.588+0000 7f6588a65640 1 --2- 172.21.15.125:0/413215166 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65700664d0 0x7f6570068990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f65800b2000 tx=0x7f6580009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:09.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.592+0000 7f65797fa640 1 -- 172.21.15.125:0/413215166 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f65840c41b0 con 0x7f65840c79e0 2023-10-30T12:42:09.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.772+0000 7f658acc9640 1 -- 172.21.15.125:0/413215166 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65840c41b0 con 0x7f65840c79e0 2023-10-30T12:42:09.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.772+0000 7f65797fa640 1 -- 172.21.15.125:0/413215166 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f65840c41b0 con 0x7f65840c79e0 2023-10-30T12:42:09.776 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:09.777 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5345,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:31.350689+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:09.777 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 69 2023-10-30T12:42:09.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.776+0000 7f658acc9640 1 -- 172.21.15.125:0/413215166 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65700664d0 msgr2=0x7f6570068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:09.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.776+0000 7f658acc9640 1 --2- 172.21.15.125:0/413215166 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65700664d0 0x7f6570068990 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f65800b2000 tx=0x7f6580009040 comp rx=0 tx=0).stop 2023-10-30T12:42:09.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.776+0000 7f658acc9640 1 -- 172.21.15.125:0/413215166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65840c79e0 msgr2=0x7f6584154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:09.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.776+0000 7f658acc9640 1 --2- 172.21.15.125:0/413215166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65840c79e0 0x7f6584154a60 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f657400c7c0 tx=0x7f657400cc80 comp rx=0 tx=0).stop 2023-10-30T12:42:09.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.776+0000 7f658acc9640 1 -- 172.21.15.125:0/413215166 shutdown_connections 2023-10-30T12:42:09.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.776+0000 7f658acc9640 1 --2- 172.21.15.125:0/413215166 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65700664d0 0x7f6570068990 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.776+0000 7f658acc9640 1 --2- 172.21.15.125:0/413215166 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6584153bf0 0x7f65841f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.776+0000 7f658acc9640 1 --2- 172.21.15.125:0/413215166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65840c79e0 0x7f6584154a60 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.776+0000 7f658acc9640 1 --2- 172.21.15.125:0/413215166 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65840c6fe0 0x7f65840c4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:09.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.776+0000 7f658acc9640 1 -- 172.21.15.125:0/413215166 >> 172.21.15.125:0/413215166 conn(0x7f658406c6e0 msgr2=0x7f6584158e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:09.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.776+0000 7f658acc9640 1 -- 172.21.15.125:0/413215166 shutdown_connections 2023-10-30T12:42:09.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:09.776+0000 7f658acc9640 1 -- 172.21.15.125:0/413215166 wait complete. 2023-10-30T12:42:09.800 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:42:11.801 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:11.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.944+0000 7fc1c9d65640 1 Processor -- start 2023-10-30T12:42:11.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.944+0000 7fc1c9d65640 1 -- start start 2023-10-30T12:42:11.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.944+0000 7fc1c9d65640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc1c4070a00 0x7fc1c4070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.944+0000 7fc1c9d65640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c40713b0 con 0x7fc1c4069e90 2023-10-30T12:42:11.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.944+0000 7fc1c9d65640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c4153d80 con 0x7fc1c41539a0 2023-10-30T12:42:11.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.944+0000 7fc1c9d65640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c406fbb0 con 0x7fc1c4070a00 2023-10-30T12:42:11.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c3fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc1c4069e90 0x7fc1c406a2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38736/0 (socket says 172.21.15.125:38736) 2023-10-30T12:42:11.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c3fff640 1 -- 172.21.15.125:0/2290855864 learned_addr learned my addr 172.21.15.125:0/2290855864 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:42:11.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1bbfff640 1 --2- 172.21.15.125:0/2290855864 >> v2:172.21.15.136:3300/0 conn(0x7fc1c4070a00 0x7fc1c4070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 -- 172.21.15.125:0/2290855864 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492343232 0 0) 0x7fc1c40713b0 con 0x7fc1c4069e90 2023-10-30T12:42:11.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 -- 172.21.15.125:0/2290855864 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b0003530 con 0x7fc1c4069e90 2023-10-30T12:42:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 -- 172.21.15.125:0/2290855864 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 695499072 0 0) 0x7fc1b0003530 con 0x7fc1c4069e90 2023-10-30T12:42:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 -- 172.21.15.125:0/2290855864 >> v2:172.21.15.136:3300/0 conn(0x7fc1c4070a00 msgr2=0x7fc1c4070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 --2- 172.21.15.125:0/2290855864 >> v2:172.21.15.136:3300/0 conn(0x7fc1c4070a00 0x7fc1c4070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 -- 172.21.15.125:0/2290855864 >> v1:172.21.15.136:6789/0 conn(0x7fc1c41539a0 legacy=0x7fc1c415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 -- 172.21.15.125:0/2290855864 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c40c5c50 con 0x7fc1c4069e90 2023-10-30T12:42:11.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 -- 172.21.15.125:0/2290855864 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fc1bc001a00 con 0x7fc1c4069e90 2023-10-30T12:42:11.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1bbfff640 1 --2- 172.21.15.125:0/2290855864 >> v2:172.21.15.136:3300/0 conn(0x7fc1c4070a00 0x7fc1c4070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:42:11.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 -- 172.21.15.125:0/2290855864 >> v1:172.21.15.125:6789/0 conn(0x7fc1c4069e90 legacy=0x7fc1c406a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:11.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 --2- 172.21.15.125:0/2290855864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b0003980 0x7fc1b0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 --2- 172.21.15.125:0/2290855864 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1b00047d0 0x7fc1b0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 --2- 172.21.15.125:0/2290855864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1b0005190 0x7fc1b000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 -- 172.21.15.125:0/2290855864 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c406fbb0 con 0x7fc1b0003980 2023-10-30T12:42:11.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 -- 172.21.15.125:0/2290855864 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b000c130 con 0x7fc1b0005190 2023-10-30T12:42:11.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c2ffd640 1 -- 172.21.15.125:0/2290855864 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1b000c2f0 con 0x7fc1b00047d0 2023-10-30T12:42:11.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c37fe640 1 --2- 172.21.15.125:0/2290855864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b0003980 0x7fc1b0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1bbfff640 1 --2- 172.21.15.125:0/2290855864 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1b00047d0 0x7fc1b0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c3fff640 1 --2- 172.21.15.125:0/2290855864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1b0005190 0x7fc1b000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c37fe640 1 -- 172.21.15.125:0/2290855864 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1b00047d0 msgr2=0x7fc1b0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c37fe640 1 --2- 172.21.15.125:0/2290855864 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1b00047d0 0x7fc1b0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c37fe640 1 -- 172.21.15.125:0/2290855864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1b0005190 msgr2=0x7fc1b000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c37fe640 1 --2- 172.21.15.125:0/2290855864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1b0005190 0x7fc1b000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c37fe640 1 -- 172.21.15.125:0/2290855864 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b0003530 con 0x7fc1b0003980 2023-10-30T12:42:11.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.948+0000 7fc1c3fff640 1 -- 172.21.15.125:0/2290855864 reap_dead start 2023-10-30T12:42:11.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.952+0000 7fc1c37fe640 1 --2- 172.21.15.125:0/2290855864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b0003980 0x7fc1b0003da0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4001780 tx=0x7fc1b401c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ca470606d0dd39 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:11.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.952+0000 7fc1c2ffd640 1 -- 172.21.15.125:0/2290855864 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b40052d0 con 0x7fc1b0003980 2023-10-30T12:42:11.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.952+0000 7fc1c2ffd640 1 -- 172.21.15.125:0/2290855864 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b40a7050 con 0x7fc1b0003980 2023-10-30T12:42:11.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.952+0000 7fc1c2ffd640 1 -- 172.21.15.125:0/2290855864 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4003600 con 0x7fc1b0003980 2023-10-30T12:42:11.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.952+0000 7fc1c9d65640 1 -- 172.21.15.125:0/2290855864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b0003980 msgr2=0x7fc1b0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:11.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.952+0000 7fc1c9d65640 1 --2- 172.21.15.125:0/2290855864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b0003980 0x7fc1b0003da0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4001780 tx=0x7fc1b401c3b0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.952+0000 7fc1c9d65640 1 -- 172.21.15.125:0/2290855864 shutdown_connections 2023-10-30T12:42:11.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.952+0000 7fc1c9d65640 1 --2- 172.21.15.125:0/2290855864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1b0003980 0x7fc1b0003da0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.952+0000 7fc1c9d65640 1 -- 172.21.15.125:0/2290855864 >> 172.21.15.125:0/2290855864 conn(0x7fc1c40c7910 msgr2=0x7fc1c40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:11.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.952+0000 7fc1c9d65640 1 -- 172.21.15.125:0/2290855864 shutdown_connections 2023-10-30T12:42:11.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.952+0000 7fc1c9d65640 1 -- 172.21.15.125:0/2290855864 wait complete. 2023-10-30T12:42:11.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.952+0000 7fc1c9d65640 1 Processor -- start 2023-10-30T12:42:11.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.952+0000 7fc1c9d65640 1 -- start start 2023-10-30T12:42:11.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c9d65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1b0003980 0x7fc1c41f6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c9d65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1c4158d30 0x7fc1c4159190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c9d65640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1c4150210 0x7fc1c41506c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c9d65640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c4159930 con 0x7fc1c4158d30 2023-10-30T12:42:11.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c9d65640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40c5250 con 0x7fc1c4150210 2023-10-30T12:42:11.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c9d65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c40c49c0 con 0x7fc1b0003980 2023-10-30T12:42:11.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1bbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1c4158d30 0x7fc1c4159190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1bbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1c4158d30 0x7fc1c4159190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50502/0 (socket says 172.21.15.125:50502) 2023-10-30T12:42:11.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1bbfff640 1 -- 172.21.15.125:0/4050200095 learned_addr learned my addr 172.21.15.125:0/4050200095 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:11.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c37fe640 1 --2- 172.21.15.125:0/4050200095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1b0003980 0x7fc1c41f6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c3fff640 1 --2- 172.21.15.125:0/4050200095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1c4150210 0x7fc1c41506c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1bbfff640 1 -- 172.21.15.125:0/4050200095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1b0003980 msgr2=0x7fc1c41f6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:11.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1bbfff640 1 --2- 172.21.15.125:0/4050200095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1b0003980 0x7fc1c41f6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1bbfff640 1 -- 172.21.15.125:0/4050200095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1c4150210 msgr2=0x7fc1c41506c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:11.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1bbfff640 1 --2- 172.21.15.125:0/4050200095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1c4150210 0x7fc1c41506c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:11.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1bbfff640 1 -- 172.21.15.125:0/4050200095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c40c4c90 con 0x7fc1c4158d30 2023-10-30T12:42:11.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1bbfff640 1 --2- 172.21.15.125:0/4050200095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1c4158d30 0x7fc1c4159190 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac011900 tx=0x7fc1ac011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:11.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c0ff9640 1 -- 172.21.15.125:0/4050200095 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac00fbd0 con 0x7fc1c4158d30 2023-10-30T12:42:11.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c0ff9640 1 -- 172.21.15.125:0/4050200095 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1ac00fd30 con 0x7fc1c4158d30 2023-10-30T12:42:11.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c0ff9640 1 -- 172.21.15.125:0/4050200095 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1ac015660 con 0x7fc1c4158d30 2023-10-30T12:42:11.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c9d65640 1 -- 172.21.15.125:0/4050200095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c40c4f70 con 0x7fc1c4158d30 2023-10-30T12:42:11.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c9d65640 1 -- 172.21.15.125:0/4050200095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c4158810 con 0x7fc1c4158d30 2023-10-30T12:42:11.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c9d65640 1 -- 172.21.15.125:0/4050200095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c404e170 con 0x7fc1c4158d30 2023-10-30T12:42:11.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.956+0000 7fc1c0ff9640 1 -- 172.21.15.125:0/4050200095 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc1ac0157c0 con 0x7fc1c4158d30 2023-10-30T12:42:11.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.960+0000 7fc1c0ff9640 1 --2- 172.21.15.125:0/4050200095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1a40664d0 0x7fc1a4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:11.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.960+0000 7fc1c0ff9640 1 -- 172.21.15.125:0/4050200095 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7fc1ac084610 con 0x7fc1c4158d30 2023-10-30T12:42:11.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.960+0000 7fc1c37fe640 1 --2- 172.21.15.125:0/4050200095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1a40664d0 0x7fc1a4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:11.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.960+0000 7fc1c37fe640 1 --2- 172.21.15.125:0/4050200095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1a40664d0 0x7fc1a4068990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4001750 tx=0x7fc1b40011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:11.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:11.964+0000 7fc1c0ff9640 1 -- 172.21.15.125:0/4050200095 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc1ac0518b0 con 0x7fc1c4158d30 2023-10-30T12:42:12.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:12.144+0000 7fc1c9d65640 1 -- 172.21.15.125:0/4050200095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1c40ca9e0 con 0x7fc1c4158d30 2023-10-30T12:42:12.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:12.144+0000 7fc1c0ff9640 1 -- 172.21.15.125:0/4050200095 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fc1ac00fea0 con 0x7fc1c4158d30 2023-10-30T12:42:12.147 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:12.147 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5345,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:31.350689+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:12.147 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 69 2023-10-30T12:42:12.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:12.144+0000 7fc1c9d65640 1 -- 172.21.15.125:0/4050200095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1a40664d0 msgr2=0x7fc1a4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:12.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:12.144+0000 7fc1c9d65640 1 --2- 172.21.15.125:0/4050200095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1a40664d0 0x7fc1a4068990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc1b4001750 tx=0x7fc1b40011a0 comp rx=0 tx=0).stop 2023-10-30T12:42:12.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:12.144+0000 7fc1c9d65640 1 -- 172.21.15.125:0/4050200095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1c4158d30 msgr2=0x7fc1c4159190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:12.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:12.144+0000 7fc1c9d65640 1 --2- 172.21.15.125:0/4050200095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1c4158d30 0x7fc1c4159190 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fc1ac011900 tx=0x7fc1ac011dc0 comp rx=0 tx=0).stop 2023-10-30T12:42:12.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:12.144+0000 7fc1c9d65640 1 -- 172.21.15.125:0/4050200095 shutdown_connections 2023-10-30T12:42:12.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:12.144+0000 7fc1c9d65640 1 --2- 172.21.15.125:0/4050200095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1a40664d0 0x7fc1a4068990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:12.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:12.144+0000 7fc1c9d65640 1 --2- 172.21.15.125:0/4050200095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1c4150210 0x7fc1c41506c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:12.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:12.144+0000 7fc1c9d65640 1 --2- 172.21.15.125:0/4050200095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1c4158d30 0x7fc1c4159190 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:12.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:12.144+0000 7fc1c9d65640 1 --2- 172.21.15.125:0/4050200095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1b0003980 0x7fc1c41f6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:12.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:12.144+0000 7fc1c9d65640 1 -- 172.21.15.125:0/4050200095 >> 172.21.15.125:0/4050200095 conn(0x7fc1c40c7910 msgr2=0x7fc1c414dc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:12.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:12.144+0000 7fc1c9d65640 1 -- 172.21.15.125:0/4050200095 shutdown_connections 2023-10-30T12:42:12.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:12.144+0000 7fc1c9d65640 1 -- 172.21.15.125:0/4050200095 wait complete. 2023-10-30T12:42:12.168 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:42:14.168 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:14.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.312+0000 7f893ebf5640 1 Processor -- start 2023-10-30T12:42:14.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.312+0000 7f893ebf5640 1 -- start start 2023-10-30T12:42:14.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.312+0000 7f893ebf5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f89380cf160 0x7f89380cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:14.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.312+0000 7f893ebf5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89380cb800 con 0x7f89381538a0 2023-10-30T12:42:14.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.312+0000 7f893ebf5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89380cb330 con 0x7f893815d6f0 2023-10-30T12:42:14.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.312+0000 7f893ebf5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89380cad70 con 0x7f89380cf160 2023-10-30T12:42:14.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f893c991640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f89381538a0 0x7f89380ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38742/0 (socket says 172.21.15.125:38742) 2023-10-30T12:42:14.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f893c991640 1 -- 172.21.15.125:0/3015921370 learned_addr learned my addr 172.21.15.125:0/3015921370 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:42:14.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f893d192640 1 --2- 172.21.15.125:0/3015921370 >> v2:172.21.15.136:3300/0 conn(0x7f89380cf160 0x7f89380cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:14.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 -- 172.21.15.125:0/3015921370 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645517875 0 0) 0x7f89380cb800 con 0x7f89381538a0 2023-10-30T12:42:14.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 -- 172.21.15.125:0/3015921370 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8920003530 con 0x7f89381538a0 2023-10-30T12:42:14.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 -- 172.21.15.125:0/3015921370 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1801174276 0 0) 0x7f8920003530 con 0x7f89381538a0 2023-10-30T12:42:14.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 -- 172.21.15.125:0/3015921370 >> v2:172.21.15.136:3300/0 conn(0x7f89380cf160 msgr2=0x7f89380cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:14.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 --2- 172.21.15.125:0/3015921370 >> v2:172.21.15.136:3300/0 conn(0x7f89380cf160 0x7f89380cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:14.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 -- 172.21.15.125:0/3015921370 >> v1:172.21.15.136:6789/0 conn(0x7f893815d6f0 legacy=0x7f893815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:14.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 -- 172.21.15.125:0/3015921370 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89380cb050 con 0x7f89381538a0 2023-10-30T12:42:14.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 -- 172.21.15.125:0/3015921370 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f8934001680 con 0x7f89381538a0 2023-10-30T12:42:14.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 -- 172.21.15.125:0/3015921370 >> v1:172.21.15.125:6789/0 conn(0x7f89381538a0 legacy=0x7f89380ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:14.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 --2- 172.21.15.125:0/3015921370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8920003980 0x7f8920003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:14.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 --2- 172.21.15.125:0/3015921370 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89200047d0 0x7f8920004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:14.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 --2- 172.21.15.125:0/3015921370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8920005190 0x7f892000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:14.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 -- 172.21.15.125:0/3015921370 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89380cad70 con 0x7f8920003980 2023-10-30T12:42:14.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 -- 172.21.15.125:0/3015921370 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f892000c040 con 0x7f8920005190 2023-10-30T12:42:14.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892f7fe640 1 -- 172.21.15.125:0/3015921370 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892000c200 con 0x7f89200047d0 2023-10-30T12:42:14.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f893c991640 1 --2- 172.21.15.125:0/3015921370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8920003980 0x7f8920003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:14.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f892ffff640 1 --2- 172.21.15.125:0/3015921370 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89200047d0 0x7f8920004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:14.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f893d192640 1 --2- 172.21.15.125:0/3015921370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8920005190 0x7f892000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:14.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f893c991640 1 -- 172.21.15.125:0/3015921370 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89200047d0 msgr2=0x7f8920004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:14.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f893c991640 1 --2- 172.21.15.125:0/3015921370 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f89200047d0 0x7f8920004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:14.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f893c991640 1 -- 172.21.15.125:0/3015921370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8920005190 msgr2=0x7f892000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:14.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f893c991640 1 --2- 172.21.15.125:0/3015921370 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8920005190 0x7f892000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:14.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f893c991640 1 -- 172.21.15.125:0/3015921370 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8920003530 con 0x7f8920003980 2023-10-30T12:42:14.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.316+0000 7f893d192640 1 -- 172.21.15.125:0/3015921370 reap_dead start 2023-10-30T12:42:14.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.320+0000 7f893c991640 1 --2- 172.21.15.125:0/3015921370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8920003980 0x7f8920003da0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f893400c240 tx=0x7f893401f330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db5586982aa3d2e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:14.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.320+0000 7f892f7fe640 1 -- 172.21.15.125:0/3015921370 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934005ae0 con 0x7f8920003980 2023-10-30T12:42:14.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.320+0000 7f892f7fe640 1 -- 172.21.15.125:0/3015921370 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89340ab070 con 0x7f8920003980 2023-10-30T12:42:14.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.320+0000 7f892f7fe640 1 -- 172.21.15.125:0/3015921370 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934007070 con 0x7f8920003980 2023-10-30T12:42:14.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.320+0000 7f893ebf5640 1 -- 172.21.15.125:0/3015921370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8920003980 msgr2=0x7f8920003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:14.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.320+0000 7f893ebf5640 1 --2- 172.21.15.125:0/3015921370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8920003980 0x7f8920003da0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f893400c240 tx=0x7f893401f330 comp rx=0 tx=0).stop 2023-10-30T12:42:14.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.320+0000 7f893ebf5640 1 -- 172.21.15.125:0/3015921370 shutdown_connections 2023-10-30T12:42:14.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.320+0000 7f893ebf5640 1 --2- 172.21.15.125:0/3015921370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8920003980 0x7f8920003da0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:14.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.320+0000 7f893ebf5640 1 -- 172.21.15.125:0/3015921370 >> 172.21.15.125:0/3015921370 conn(0x7f893806bfe0 msgr2=0x7f893806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:14.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.320+0000 7f893ebf5640 1 -- 172.21.15.125:0/3015921370 shutdown_connections 2023-10-30T12:42:14.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.320+0000 7f893ebf5640 1 -- 172.21.15.125:0/3015921370 wait complete. 2023-10-30T12:42:14.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.320+0000 7f893ebf5640 1 Processor -- start 2023-10-30T12:42:14.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.320+0000 7f893ebf5640 1 -- start start 2023-10-30T12:42:14.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893ebf5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8920003980 0x7f8938157990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:14.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893ebf5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89381eff30 0x7f89381e63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:14.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893ebf5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89380b4d90 0x7f89380ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:14.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893ebf5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89381f3760 con 0x7f89380b4d90 2023-10-30T12:42:14.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893ebf5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89381ef240 con 0x7f89381eff30 2023-10-30T12:42:14.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893ebf5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89381efaa0 con 0x7f8920003980 2023-10-30T12:42:14.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893d192640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89380b4d90 0x7f89380ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:14.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893d192640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89380b4d90 0x7f89380ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50512/0 (socket says 172.21.15.125:50512) 2023-10-30T12:42:14.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893d192640 1 -- 172.21.15.125:0/2490342696 learned_addr learned my addr 172.21.15.125:0/2490342696 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:14.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f892ffff640 1 --2- 172.21.15.125:0/2490342696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89381eff30 0x7f89381e63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:14.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893c991640 1 --2- 172.21.15.125:0/2490342696 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8920003980 0x7f8938157990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:14.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893d192640 1 -- 172.21.15.125:0/2490342696 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8920003980 msgr2=0x7f8938157990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:14.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893d192640 1 --2- 172.21.15.125:0/2490342696 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8920003980 0x7f8938157990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:14.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893d192640 1 -- 172.21.15.125:0/2490342696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89381eff30 msgr2=0x7f89381e63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:14.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893d192640 1 --2- 172.21.15.125:0/2490342696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89381eff30 0x7f89381e63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:14.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893d192640 1 -- 172.21.15.125:0/2490342696 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89381f31c0 con 0x7f89380b4d90 2023-10-30T12:42:14.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893d192640 1 --2- 172.21.15.125:0/2490342696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89380b4d90 0x7f89380ccfb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f8938072ce0 tx=0x7f893000f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:14.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f892d7fa640 1 -- 172.21.15.125:0/2490342696 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893000e040 con 0x7f89380b4d90 2023-10-30T12:42:14.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f892d7fa640 1 -- 172.21.15.125:0/2490342696 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893000f040 con 0x7f89380b4d90 2023-10-30T12:42:14.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893ebf5640 1 -- 172.21.15.125:0/2490342696 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89381f3480 con 0x7f89380b4d90 2023-10-30T12:42:14.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f892d7fa640 1 -- 172.21.15.125:0/2490342696 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f893000fec0 con 0x7f89380b4d90 2023-10-30T12:42:14.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893ebf5640 1 -- 172.21.15.125:0/2490342696 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89381ef730 con 0x7f89380b4d90 2023-10-30T12:42:14.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f893ebf5640 1 -- 172.21.15.125:0/2490342696 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893804e170 con 0x7f89380b4d90 2023-10-30T12:42:14.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.324+0000 7f892d7fa640 1 -- 172.21.15.125:0/2490342696 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8930022460 con 0x7f89380b4d90 2023-10-30T12:42:14.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.328+0000 7f892d7fa640 1 --2- 172.21.15.125:0/2490342696 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f89240664a0 0x7f8924068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:14.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.328+0000 7f892d7fa640 1 -- 172.21.15.125:0/2490342696 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6168+0+0 (secure 0 0 0) 0x7f8930084890 con 0x7f89380b4d90 2023-10-30T12:42:14.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.328+0000 7f893c991640 1 --2- 172.21.15.125:0/2490342696 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f89240664a0 0x7f8924068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:14.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.328+0000 7f893c991640 1 --2- 172.21.15.125:0/2490342696 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f89240664a0 0x7f8924068960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f89340af000 tx=0x7f89340aa000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:14.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.332+0000 7f892d7fa640 1 -- 172.21.15.125:0/2490342696 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8930051ac0 con 0x7f89380b4d90 2023-10-30T12:42:14.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.512+0000 7f893ebf5640 1 -- 172.21.15.125:0/2490342696 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89381efbe0 con 0x7f89380b4d90 2023-10-30T12:42:14.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.516+0000 7f892d7fa640 1 -- 172.21.15.125:0/2490342696 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f89300550e0 con 0x7f89380b4d90 2023-10-30T12:42:14.518 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:14.518 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5345,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:41:31.350689+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5130},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5130":{"gid":5130,"name":"c","rank":1,"incarnation":37,"state":"up:active","state_seq":41,"addr":"172.21.15.125:6837/1624923396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1624923396},{"type":"v1","addr":"172.21.15.125:6837","nonce":1624923396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:14.519 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 69 2023-10-30T12:42:14.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.516+0000 7f893ebf5640 1 -- 172.21.15.125:0/2490342696 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f89240664a0 msgr2=0x7f8924068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:14.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.516+0000 7f893ebf5640 1 --2- 172.21.15.125:0/2490342696 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f89240664a0 0x7f8924068960 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f89340af000 tx=0x7f89340aa000 comp rx=0 tx=0).stop 2023-10-30T12:42:14.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.516+0000 7f893ebf5640 1 -- 172.21.15.125:0/2490342696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89380b4d90 msgr2=0x7f89380ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:14.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.516+0000 7f893ebf5640 1 --2- 172.21.15.125:0/2490342696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89380b4d90 0x7f89380ccfb0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f8938072ce0 tx=0x7f893000f1b0 comp rx=0 tx=0).stop 2023-10-30T12:42:14.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.516+0000 7f893ebf5640 1 -- 172.21.15.125:0/2490342696 shutdown_connections 2023-10-30T12:42:14.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.516+0000 7f893ebf5640 1 --2- 172.21.15.125:0/2490342696 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f89240664a0 0x7f8924068960 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:14.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.516+0000 7f893ebf5640 1 --2- 172.21.15.125:0/2490342696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f89380b4d90 0x7f89380ccfb0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:14.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.516+0000 7f893ebf5640 1 --2- 172.21.15.125:0/2490342696 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f89381eff30 0x7f89381e63d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:14.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.516+0000 7f893ebf5640 1 --2- 172.21.15.125:0/2490342696 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8920003980 0x7f8938157990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:14.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.516+0000 7f893ebf5640 1 -- 172.21.15.125:0/2490342696 >> 172.21.15.125:0/2490342696 conn(0x7f893806bfe0 msgr2=0x7f89380c4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:14.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.516+0000 7f893ebf5640 1 -- 172.21.15.125:0/2490342696 shutdown_connections 2023-10-30T12:42:14.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:14.516+0000 7f893ebf5640 1 -- 172.21.15.125:0/2490342696 wait complete. 2023-10-30T12:42:14.539 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:42:15.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:42:15.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:42:16.540 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:16.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc7ad291640 1 Processor -- start 2023-10-30T12:42:16.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc7ad291640 1 -- start start 2023-10-30T12:42:16.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc7ad291640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc7a81538a0 0x7fc7a815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc7ad291640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc7a80ca0e0 0x7fc7a815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc7ad291640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7a80cb800 con 0x7fc7a80cf160 2023-10-30T12:42:16.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc7ad291640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a815dc30 con 0x7fc7a80ca0e0 2023-10-30T12:42:16.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc7ad291640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a80cb330 con 0x7fc7a81538a0 2023-10-30T12:42:16.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc79ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc7a80ca0e0 0x7fc7a815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc79ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc7a80ca0e0 0x7fc7a815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50514/0 (socket says 172.21.15.125:50514) 2023-10-30T12:42:16.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc79ffff640 1 -- 172.21.15.125:0/3206486398 learned_addr learned my addr 172.21.15.125:0/3206486398 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:16.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc7a6ffd640 1 --2- 172.21.15.125:0/3206486398 >> v2:172.21.15.136:3300/0 conn(0x7fc7a81538a0 0x7fc7a815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc79ffff640 1 -- 172.21.15.125:0/3206486398 >> v2:172.21.15.136:3300/0 conn(0x7fc7a81538a0 msgr2=0x7fc7a815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:16.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc79ffff640 1 --2- 172.21.15.125:0/3206486398 >> v2:172.21.15.136:3300/0 conn(0x7fc7a81538a0 0x7fc7a815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc79ffff640 1 -- 172.21.15.125:0/3206486398 >> v1:172.21.15.125:6789/0 conn(0x7fc7a80cf160 legacy=0x7fc7a80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:16.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc79ffff640 1 -- 172.21.15.125:0/3206486398 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a80cad70 con 0x7fc7a80ca0e0 2023-10-30T12:42:16.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.684+0000 7fc7a6ffd640 1 --2- 172.21.15.125:0/3206486398 >> v2:172.21.15.136:3300/0 conn(0x7fc7a81538a0 0x7fc7a815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:16.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.688+0000 7fc79ffff640 1 --2- 172.21.15.125:0/3206486398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7a80ca0e0 0x7fc7a815a090 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc7a80cfd10 tx=0x7fc79001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7da02521c34ceaef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:16.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.688+0000 7fc7a67fc640 1 -- 172.21.15.125:0/3206486398 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7900a89f0 con 0x7fc7a80ca0e0 2023-10-30T12:42:16.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.688+0000 7fc7a67fc640 1 -- 172.21.15.125:0/3206486398 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7900a8b50 con 0x7fc7a80ca0e0 2023-10-30T12:42:16.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.688+0000 7fc7a67fc640 1 -- 172.21.15.125:0/3206486398 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7900b0440 con 0x7fc7a80ca0e0 2023-10-30T12:42:16.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.688+0000 7fc7ad291640 1 -- 172.21.15.125:0/3206486398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7a80ca0e0 msgr2=0x7fc7a815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:16.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.688+0000 7fc7ad291640 1 --2- 172.21.15.125:0/3206486398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7a80ca0e0 0x7fc7a815a090 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc7a80cfd10 tx=0x7fc79001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.688+0000 7fc7ad291640 1 -- 172.21.15.125:0/3206486398 shutdown_connections 2023-10-30T12:42:16.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.688+0000 7fc7ad291640 1 --2- 172.21.15.125:0/3206486398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7a80ca0e0 0x7fc7a815a090 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.688+0000 7fc7ad291640 1 --2- 172.21.15.125:0/3206486398 >> v2:172.21.15.136:3300/0 conn(0x7fc7a81538a0 0x7fc7a815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.688+0000 7fc7ad291640 1 -- 172.21.15.125:0/3206486398 >> 172.21.15.125:0/3206486398 conn(0x7fc7a806bfe0 msgr2=0x7fc7a806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:16.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.688+0000 7fc7ad291640 1 -- 172.21.15.125:0/3206486398 shutdown_connections 2023-10-30T12:42:16.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.688+0000 7fc7ad291640 1 -- 172.21.15.125:0/3206486398 wait complete. 2023-10-30T12:42:16.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7ad291640 1 Processor -- start 2023-10-30T12:42:16.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7ad291640 1 -- start start 2023-10-30T12:42:16.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7ad291640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7a80ca0e0 0x7fc7a8140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7ad291640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7a80cf160 0x7fc7a80ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7ad291640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7a81538a0 0x7fc7a80cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7ad291640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a81e4380 con 0x7fc7a81538a0 2023-10-30T12:42:16.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7ad291640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a8158620 con 0x7fc7a80ca0e0 2023-10-30T12:42:16.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7ad291640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a8159690 con 0x7fc7a80cf160 2023-10-30T12:42:16.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7a77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7a81538a0 0x7fc7a80cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7a77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7a81538a0 0x7fc7a80cd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50524/0 (socket says 172.21.15.125:50524) 2023-10-30T12:42:16.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7a77fe640 1 -- 172.21.15.125:0/3308631740 learned_addr learned my addr 172.21.15.125:0/3308631740 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:16.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc79ffff640 1 --2- 172.21.15.125:0/3308631740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7a80cf160 0x7fc7a80ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7a6ffd640 1 --2- 172.21.15.125:0/3308631740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7a80ca0e0 0x7fc7a8140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7a77fe640 1 -- 172.21.15.125:0/3308631740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7a80cf160 msgr2=0x7fc7a80ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:16.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7a77fe640 1 --2- 172.21.15.125:0/3308631740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7a80cf160 0x7fc7a80ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7a77fe640 1 -- 172.21.15.125:0/3308631740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7a80ca0e0 msgr2=0x7fc7a8140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:16.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7a77fe640 1 --2- 172.21.15.125:0/3308631740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7a80ca0e0 0x7fc7a8140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7a77fe640 1 -- 172.21.15.125:0/3308631740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7900094d0 con 0x7fc7a81538a0 2023-10-30T12:42:16.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7a77fe640 1 --2- 172.21.15.125:0/3308631740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7a81538a0 0x7fc7a80cd4f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000c6f0 tx=0x7fc7a000cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:16.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc79f7fe640 1 -- 172.21.15.125:0/3308631740 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0003980 con 0x7fc7a81538a0 2023-10-30T12:42:16.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc79f7fe640 1 -- 172.21.15.125:0/3308631740 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7a0003ae0 con 0x7fc7a81538a0 2023-10-30T12:42:16.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc79f7fe640 1 -- 172.21.15.125:0/3308631740 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0003e90 con 0x7fc7a81538a0 2023-10-30T12:42:16.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7ad291640 1 -- 172.21.15.125:0/3308631740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a81e51e0 con 0x7fc7a81538a0 2023-10-30T12:42:16.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.692+0000 7fc7ad291640 1 -- 172.21.15.125:0/3308631740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a81e5610 con 0x7fc7a81538a0 2023-10-30T12:42:16.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.696+0000 7fc7ad291640 1 -- 172.21.15.125:0/3308631740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a80b2a50 con 0x7fc7a81538a0 2023-10-30T12:42:16.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.696+0000 7fc79f7fe640 1 -- 172.21.15.125:0/3308631740 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc7a000a850 con 0x7fc7a81538a0 2023-10-30T12:42:16.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.696+0000 7fc79f7fe640 1 --2- 172.21.15.125:0/3308631740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7880664d0 0x7fc788068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:16.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.696+0000 7fc79f7fe640 1 -- 172.21.15.125:0/3308631740 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7fc7a0083dc0 con 0x7fc7a81538a0 2023-10-30T12:42:16.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.696+0000 7fc7a6ffd640 1 --2- 172.21.15.125:0/3308631740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7880664d0 0x7fc788068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:16.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.696+0000 7fc7a6ffd640 1 --2- 172.21.15.125:0/3308631740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7880664d0 0x7fc788068990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc7a80cf070 tx=0x7fc798008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:16.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.700+0000 7fc79f7fe640 1 -- 172.21.15.125:0/3308631740 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7a00510b0 con 0x7fc7a81538a0 2023-10-30T12:42:16.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.880+0000 7fc7ad291640 1 -- 172.21.15.125:0/3308631740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7a8158790 con 0x7fc7a81538a0 2023-10-30T12:42:16.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.880+0000 7fc79f7fe640 1 -- 172.21.15.125:0/3308631740 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5736 (secure 0 0 0) 0x7fc7a00546d0 con 0x7fc7a81538a0 2023-10-30T12:42:16.884 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:16.884 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:15.151625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:16.884 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 70 2023-10-30T12:42:16.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.880+0000 7fc7ad291640 1 -- 172.21.15.125:0/3308631740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7880664d0 msgr2=0x7fc788068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:16.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.880+0000 7fc7ad291640 1 --2- 172.21.15.125:0/3308631740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7880664d0 0x7fc788068990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc7a80cf070 tx=0x7fc798008040 comp rx=0 tx=0).stop 2023-10-30T12:42:16.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.880+0000 7fc7ad291640 1 -- 172.21.15.125:0/3308631740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7a81538a0 msgr2=0x7fc7a80cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:16.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.884+0000 7fc7ad291640 1 --2- 172.21.15.125:0/3308631740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7a81538a0 0x7fc7a80cd4f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc7a000c6f0 tx=0x7fc7a000cbb0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.884+0000 7fc7ad291640 1 -- 172.21.15.125:0/3308631740 shutdown_connections 2023-10-30T12:42:16.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.884+0000 7fc7ad291640 1 --2- 172.21.15.125:0/3308631740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7880664d0 0x7fc788068990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.884+0000 7fc7ad291640 1 --2- 172.21.15.125:0/3308631740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc7a81538a0 0x7fc7a80cd4f0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.884+0000 7fc7ad291640 1 --2- 172.21.15.125:0/3308631740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7a80cf160 0x7fc7a80ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.884+0000 7fc7ad291640 1 --2- 172.21.15.125:0/3308631740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7a80ca0e0 0x7fc7a8140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:16.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.884+0000 7fc7ad291640 1 -- 172.21.15.125:0/3308631740 >> 172.21.15.125:0/3308631740 conn(0x7fc7a806bfe0 msgr2=0x7fc7a80c5210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:16.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.884+0000 7fc7ad291640 1 -- 172.21.15.125:0/3308631740 shutdown_connections 2023-10-30T12:42:16.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:16.884+0000 7fc7ad291640 1 -- 172.21.15.125:0/3308631740 wait complete. 2023-10-30T12:42:16.905 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T12:42:16.905 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:42:16.905 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:17.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0f666b640 1 Processor -- start 2023-10-30T12:42:17.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0f666b640 1 -- start start 2023-10-30T12:42:17.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0f666b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f00c6f60 0x7ff0f01566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:17.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0f666b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f00c0a50 con 0x7ff0f00c05d0 2023-10-30T12:42:17.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0f666b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f006f6e0 con 0x7ff0f00c6b80 2023-10-30T12:42:17.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0f666b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f006f9c0 con 0x7ff0f00c6f60 2023-10-30T12:42:17.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0ef7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff0f00c05d0 0x7ff0f013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38762/0 (socket says 172.21.15.125:38762) 2023-10-30T12:42:17.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0ef7fe640 1 -- 172.21.15.125:0/2041270275 learned_addr learned my addr 172.21.15.125:0/2041270275 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:42:17.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0f4c08640 1 --2- 172.21.15.125:0/2041270275 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f00c6f60 0x7ff0f01566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:17.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 -- 172.21.15.125:0/2041270275 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310408966 0 0) 0x7ff0f00c0a50 con 0x7ff0f00c05d0 2023-10-30T12:42:17.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 -- 172.21.15.125:0/2041270275 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0d8003530 con 0x7ff0f00c05d0 2023-10-30T12:42:17.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 -- 172.21.15.125:0/2041270275 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 330733641 0 0) 0x7ff0d8003530 con 0x7ff0f00c05d0 2023-10-30T12:42:17.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 -- 172.21.15.125:0/2041270275 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f00c6f60 msgr2=0x7ff0f01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:17.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 --2- 172.21.15.125:0/2041270275 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f00c6f60 0x7ff0f01566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:17.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 -- 172.21.15.125:0/2041270275 >> v1:172.21.15.136:6789/0 conn(0x7ff0f00c6b80 legacy=0x7ff0f00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:17.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 -- 172.21.15.125:0/2041270275 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f006fca0 con 0x7ff0f00c05d0 2023-10-30T12:42:17.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 -- 172.21.15.125:0/2041270275 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7ff0e4001680 con 0x7ff0f00c05d0 2023-10-30T12:42:17.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 -- 172.21.15.125:0/2041270275 >> v1:172.21.15.125:6789/0 conn(0x7ff0f00c05d0 legacy=0x7ff0f013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:17.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 --2- 172.21.15.125:0/2041270275 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0d80039b0 0x7ff0d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:17.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 --2- 172.21.15.125:0/2041270275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d8004310 0x7ff0d80047f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:17.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 --2- 172.21.15.125:0/2041270275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0d8004ff0 0x7ff0d800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:17.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 -- 172.21.15.125:0/2041270275 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f006f9c0 con 0x7ff0d8004310 2023-10-30T12:42:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 -- 172.21.15.125:0/2041270275 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d800bf00 con 0x7ff0d8004ff0 2023-10-30T12:42:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0eeffd640 1 -- 172.21.15.125:0/2041270275 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d800e2d0 con 0x7ff0d80039b0 2023-10-30T12:42:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0f4c08640 1 --2- 172.21.15.125:0/2041270275 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f00c6f60 0x7ff0f01566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:42:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.040+0000 7ff0ef7fe640 1 --2- 172.21.15.125:0/2041270275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d8004310 0x7ff0d80047f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0effff640 1 --2- 172.21.15.125:0/2041270275 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0d80039b0 0x7ff0d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:17.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0f4c08640 1 --2- 172.21.15.125:0/2041270275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0d8004ff0 0x7ff0d800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:17.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0ef7fe640 1 -- 172.21.15.125:0/2041270275 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0d80039b0 msgr2=0x7ff0d8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:17.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0ef7fe640 1 --2- 172.21.15.125:0/2041270275 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0d80039b0 0x7ff0d8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:17.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0ef7fe640 1 -- 172.21.15.125:0/2041270275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0d8004ff0 msgr2=0x7ff0d800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:17.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0ef7fe640 1 --2- 172.21.15.125:0/2041270275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0d8004ff0 0x7ff0d800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0ef7fe640 1 -- 172.21.15.125:0/2041270275 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d8003530 con 0x7ff0d8004310 2023-10-30T12:42:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0f4c08640 1 --2- 172.21.15.125:0/2041270275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0d8004ff0 0x7ff0d800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0ef7fe640 1 --2- 172.21.15.125:0/2041270275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d8004310 0x7ff0d80047f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400c160 tx=0x7ff0e401cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9eca09fd49c20bf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0eeffd640 1 -- 172.21.15.125:0/2041270275 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e4005ae0 con 0x7ff0d8004310 2023-10-30T12:42:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0eeffd640 1 -- 172.21.15.125:0/2041270275 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e4005c60 con 0x7ff0d8004310 2023-10-30T12:42:17.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0eeffd640 1 -- 172.21.15.125:0/2041270275 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e40a5ac0 con 0x7ff0d8004310 2023-10-30T12:42:17.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0f666b640 1 -- 172.21.15.125:0/2041270275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d8004310 msgr2=0x7ff0d80047f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:17.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0f666b640 1 --2- 172.21.15.125:0/2041270275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0d8004310 0x7ff0d80047f0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7ff0e400c160 tx=0x7ff0e401cb80 comp rx=0 tx=0).stop 2023-10-30T12:42:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.044+0000 7ff0f4c08640 1 -- 172.21.15.125:0/2041270275 reap_dead start 2023-10-30T12:42:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f666b640 1 -- 172.21.15.125:0/2041270275 shutdown_connections 2023-10-30T12:42:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f666b640 1 -- 172.21.15.125:0/2041270275 >> 172.21.15.125:0/2041270275 conn(0x7ff0f00c7910 msgr2=0x7ff0f00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f666b640 1 -- 172.21.15.125:0/2041270275 shutdown_connections 2023-10-30T12:42:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f666b640 1 -- 172.21.15.125:0/2041270275 wait complete. 2023-10-30T12:42:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f666b640 1 Processor -- start 2023-10-30T12:42:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f666b640 1 -- start start 2023-10-30T12:42:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f666b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0d8004310 0x7ff0f0136980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f666b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f013c820 0x7ff0f01f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f666b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0f01f25d0 0x7ff0f01544a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f666b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f01f2a50 con 0x7ff0f01f25d0 2023-10-30T12:42:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f666b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f013aef0 con 0x7ff0f013c820 2023-10-30T12:42:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f666b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f01568b0 con 0x7ff0d8004310 2023-10-30T12:42:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f4c08640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0f01f25d0 0x7ff0f01544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:17.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f4c08640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0f01f25d0 0x7ff0f01544a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50536/0 (socket says 172.21.15.125:50536) 2023-10-30T12:42:17.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f4c08640 1 -- 172.21.15.125:0/1109949557 learned_addr learned my addr 172.21.15.125:0/1109949557 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:17.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0effff640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0d8004310 0x7ff0f0136980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:17.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0ef7fe640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f013c820 0x7ff0f01f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:17.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f4c08640 1 -- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0d8004310 msgr2=0x7ff0f0136980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:17.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f4c08640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0d8004310 0x7ff0f0136980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:17.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f4c08640 1 -- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f013c820 msgr2=0x7ff0f01f2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:17.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f4c08640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f013c820 0x7ff0f01f2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:17.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0f4c08640 1 -- 172.21.15.125:0/1109949557 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f0156b90 con 0x7ff0f01f25d0 2023-10-30T12:42:17.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.048+0000 7ff0ef7fe640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f013c820 0x7ff0f01f2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:17.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.052+0000 7ff0f4c08640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0f01f25d0 0x7ff0f01544a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0159b90 tx=0x7ff0e80117d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:17.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.052+0000 7ff0ecff9640 1 -- 172.21.15.125:0/1109949557 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e800e070 con 0x7ff0f01f25d0 2023-10-30T12:42:17.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.052+0000 7ff0ecff9640 1 -- 172.21.15.125:0/1109949557 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e80039a0 con 0x7ff0f01f25d0 2023-10-30T12:42:17.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.052+0000 7ff0ecff9640 1 -- 172.21.15.125:0/1109949557 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8003b30 con 0x7ff0f01f25d0 2023-10-30T12:42:17.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.052+0000 7ff0f666b640 1 -- 172.21.15.125:0/1109949557 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f0156e70 con 0x7ff0f01f25d0 2023-10-30T12:42:17.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.052+0000 7ff0f666b640 1 -- 172.21.15.125:0/1109949557 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f0069e90 con 0x7ff0f01f25d0 2023-10-30T12:42:17.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.052+0000 7ff0f666b640 1 -- 172.21.15.125:0/1109949557 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f004e170 con 0x7ff0f01f25d0 2023-10-30T12:42:17.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.052+0000 7ff0ecff9640 1 -- 172.21.15.125:0/1109949557 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff0e8003c90 con 0x7ff0f01f25d0 2023-10-30T12:42:17.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.052+0000 7ff0ecff9640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0dc0664a0 0x7ff0dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:17.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.052+0000 7ff0ecff9640 1 -- 172.21.15.125:0/1109949557 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7ff0e8084940 con 0x7ff0f01f25d0 2023-10-30T12:42:17.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.052+0000 7ff0effff640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0dc0664a0 0x7ff0dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:17.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.056+0000 7ff0effff640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0dc0664a0 0x7ff0dc068960 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0007fd0 tx=0x7ff0e0007cd0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:17.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.056+0000 7ff0ecff9640 1 -- 172.21.15.125:0/1109949557 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff0e8051ba0 con 0x7ff0f01f25d0 2023-10-30T12:42:17.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.244+0000 7ff0f666b640 1 -- 172.21.15.125:0/1109949557 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0f0157320 con 0x7ff0f01f25d0 2023-10-30T12:42:17.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.244+0000 7ff0ecff9640 1 -- 172.21.15.125:0/1109949557 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5736 (secure 0 0 0) 0x7ff0e80551c0 con 0x7ff0f01f25d0 2023-10-30T12:42:17.247 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:17.247 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:15.151625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:17.248 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 70 2023-10-30T12:42:17.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.244+0000 7ff0f666b640 1 -- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0dc0664a0 msgr2=0x7ff0dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:17.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.244+0000 7ff0f666b640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0dc0664a0 0x7ff0dc068960 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0007fd0 tx=0x7ff0e0007cd0 comp rx=0 tx=0).stop 2023-10-30T12:42:17.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.244+0000 7ff0f666b640 1 -- 172.21.15.125:0/1109949557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0f01f25d0 msgr2=0x7ff0f01544a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:17.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.244+0000 7ff0f666b640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0f01f25d0 0x7ff0f01544a0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0159b90 tx=0x7ff0e80117d0 comp rx=0 tx=0).stop 2023-10-30T12:42:17.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.244+0000 7ff0f666b640 1 -- 172.21.15.125:0/1109949557 shutdown_connections 2023-10-30T12:42:17.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.244+0000 7ff0f666b640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0dc0664a0 0x7ff0dc068960 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:17.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.244+0000 7ff0f666b640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0f01f25d0 0x7ff0f01544a0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:17.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.244+0000 7ff0f666b640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0f013c820 0x7ff0f01f2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:17.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.244+0000 7ff0f666b640 1 --2- 172.21.15.125:0/1109949557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0d8004310 0x7ff0f0136980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:17.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.244+0000 7ff0f666b640 1 -- 172.21.15.125:0/1109949557 >> 172.21.15.125:0/1109949557 conn(0x7ff0f00c7910 msgr2=0x7ff0f0137130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:17.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.244+0000 7ff0f666b640 1 -- 172.21.15.125:0/1109949557 shutdown_connections 2023-10-30T12:42:17.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:17.248+0000 7ff0f666b640 1 -- 172.21.15.125:0/1109949557 wait complete. 2023-10-30T12:42:17.269 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 70, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 70, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5345": { "addr": "172.21.15.136:6839/3922559697", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3922559697, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3922559697, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5345, "incarnation": 70, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:replay", "state_seq": 1 }, "gid_5433": { "addr": "172.21.15.125:6836/2595438444", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2595438444, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2595438444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5433, "incarnation": 62, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 69 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:42:15.151625+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5433, "mds_1": 5345 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:42:19.270 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:19.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.412+0000 7f08449c5640 1 Processor -- start 2023-10-30T12:42:19.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.412+0000 7f08449c5640 1 -- start start 2023-10-30T12:42:19.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.416+0000 7f08449c5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f08400c6fe0 0x7f0840071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:19.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.416+0000 7f08449c5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f08400c5f60 0x7f08400cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:19.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.416+0000 7f08449c5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08400700b0 con 0x7f0840153bf0 2023-10-30T12:42:19.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.416+0000 7f08449c5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08400c6810 con 0x7f08400c6fe0 2023-10-30T12:42:19.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.416+0000 7f08449c5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0840159900 con 0x7f08400c5f60 2023-10-30T12:42:19.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.416+0000 7f0837fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f08400c6fe0 0x7f0840071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:19.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.416+0000 7f0837fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f08400c6fe0 0x7f0840071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50542/0 (socket says 172.21.15.125:50542) 2023-10-30T12:42:19.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.416+0000 7f0837fff640 1 -- 172.21.15.125:0/1824494199 learned_addr learned my addr 172.21.15.125:0/1824494199 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:19.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.416+0000 7f083ed9d640 1 --2- 172.21.15.125:0/1824494199 >> v2:172.21.15.136:3300/0 conn(0x7f08400c5f60 0x7f08400cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:19.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.416+0000 7f0837fff640 1 -- 172.21.15.125:0/1824494199 >> v2:172.21.15.136:3300/0 conn(0x7f08400c5f60 msgr2=0x7f08400cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:19.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.416+0000 7f0837fff640 1 --2- 172.21.15.125:0/1824494199 >> v2:172.21.15.136:3300/0 conn(0x7f08400c5f60 0x7f08400cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:19.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.416+0000 7f0837fff640 1 -- 172.21.15.125:0/1824494199 >> v1:172.21.15.136:6789/0 conn(0x7f0840153bf0 legacy=0x7f08400c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:19.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.416+0000 7f0837fff640 1 -- 172.21.15.125:0/1824494199 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08400c6530 con 0x7f08400c6fe0 2023-10-30T12:42:19.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.420+0000 7f0837fff640 1 --2- 172.21.15.125:0/1824494199 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08400c6fe0 0x7f0840071260 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f084006ce90 tx=0x7f082801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dacadc9f0d8fa7b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:19.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.420+0000 7f083dd9b640 1 -- 172.21.15.125:0/1824494199 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08280a89b0 con 0x7f08400c6fe0 2023-10-30T12:42:19.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.420+0000 7f083dd9b640 1 -- 172.21.15.125:0/1824494199 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08280a8b10 con 0x7f08400c6fe0 2023-10-30T12:42:19.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.420+0000 7f083dd9b640 1 -- 172.21.15.125:0/1824494199 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08280a8ea0 con 0x7f08400c6fe0 2023-10-30T12:42:19.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.420+0000 7f08449c5640 1 -- 172.21.15.125:0/1824494199 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08400c6fe0 msgr2=0x7f0840071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:19.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.420+0000 7f08449c5640 1 --2- 172.21.15.125:0/1824494199 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08400c6fe0 0x7f0840071260 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f084006ce90 tx=0x7f082801ecb0 comp rx=0 tx=0).stop 2023-10-30T12:42:19.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.420+0000 7f08449c5640 1 -- 172.21.15.125:0/1824494199 shutdown_connections 2023-10-30T12:42:19.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.420+0000 7f08449c5640 1 --2- 172.21.15.125:0/1824494199 >> v2:172.21.15.136:3300/0 conn(0x7f08400c5f60 0x7f08400cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:19.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.420+0000 7f08449c5640 1 --2- 172.21.15.125:0/1824494199 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08400c6fe0 0x7f0840071260 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:19.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.420+0000 7f08449c5640 1 -- 172.21.15.125:0/1824494199 >> 172.21.15.125:0/1824494199 conn(0x7f084006c6e0 msgr2=0x7f0840069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:19.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.420+0000 7f08449c5640 1 -- 172.21.15.125:0/1824494199 shutdown_connections 2023-10-30T12:42:19.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.420+0000 7f08449c5640 1 -- 172.21.15.125:0/1824494199 wait complete. 2023-10-30T12:42:19.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.420+0000 7f08449c5640 1 Processor -- start 2023-10-30T12:42:19.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.420+0000 7f08449c5640 1 -- start start 2023-10-30T12:42:19.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f08449c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08400c5f60 0x7f0840152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:19.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f08449c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08400c6fe0 0x7f08401f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:19.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f08449c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0840153bf0 0x7f0840148300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:19.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f08449c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08400caa20 con 0x7f08400c5f60 2023-10-30T12:42:19.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f08449c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f084014e9e0 con 0x7f0840153bf0 2023-10-30T12:42:19.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f08449c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f084014e4e0 con 0x7f08400c6fe0 2023-10-30T12:42:19.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f083e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08400c5f60 0x7f0840152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:19.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f083e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08400c5f60 0x7f0840152bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50556/0 (socket says 172.21.15.125:50556) 2023-10-30T12:42:19.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f083e59c640 1 -- 172.21.15.125:0/1217952708 learned_addr learned my addr 172.21.15.125:0/1217952708 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:19.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f0837fff640 1 --2- 172.21.15.125:0/1217952708 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08400c6fe0 0x7f08401f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:19.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f083ed9d640 1 --2- 172.21.15.125:0/1217952708 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0840153bf0 0x7f0840148300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:19.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f083e59c640 1 -- 172.21.15.125:0/1217952708 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08400c6fe0 msgr2=0x7f08401f2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:19.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f083e59c640 1 --2- 172.21.15.125:0/1217952708 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08400c6fe0 0x7f08401f2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:19.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f083e59c640 1 -- 172.21.15.125:0/1217952708 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0840153bf0 msgr2=0x7f0840148300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:19.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f083e59c640 1 --2- 172.21.15.125:0/1217952708 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0840153bf0 0x7f0840148300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:19.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f083e59c640 1 -- 172.21.15.125:0/1217952708 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08280094d0 con 0x7f08400c5f60 2023-10-30T12:42:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f083e59c640 1 --2- 172.21.15.125:0/1217952708 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08400c5f60 0x7f0840152bb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f083000b440 tx=0x7f083000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f0836ffd640 1 -- 172.21.15.125:0/1217952708 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830011890 con 0x7f08400c5f60 2023-10-30T12:42:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f0836ffd640 1 -- 172.21.15.125:0/1217952708 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08300119f0 con 0x7f08400c5f60 2023-10-30T12:42:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f0836ffd640 1 -- 172.21.15.125:0/1217952708 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0830011da0 con 0x7f08400c5f60 2023-10-30T12:42:19.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f08449c5640 1 -- 172.21.15.125:0/1217952708 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f084014e7a0 con 0x7f08400c5f60 2023-10-30T12:42:19.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f08449c5640 1 -- 172.21.15.125:0/1217952708 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08400c9860 con 0x7f08400c5f60 2023-10-30T12:42:19.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f08449c5640 1 -- 172.21.15.125:0/1217952708 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f084004e170 con 0x7f08400c5f60 2023-10-30T12:42:19.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.424+0000 7f0836ffd640 1 -- 172.21.15.125:0/1217952708 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f083001a400 con 0x7f08400c5f60 2023-10-30T12:42:19.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.428+0000 7f0836ffd640 1 --2- 172.21.15.125:0/1217952708 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f08200664d0 0x7f0820068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:19.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.428+0000 7f0836ffd640 1 -- 172.21.15.125:0/1217952708 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f0830084110 con 0x7f08400c5f60 2023-10-30T12:42:19.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.428+0000 7f0837fff640 1 --2- 172.21.15.125:0/1217952708 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f08200664d0 0x7f0820068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:19.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.428+0000 7f0837fff640 1 --2- 172.21.15.125:0/1217952708 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f08200664d0 0x7f0820068990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f08280b2000 tx=0x7f0828009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:19.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.432+0000 7f0836ffd640 1 -- 172.21.15.125:0/1217952708 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0830051240 con 0x7f08400c5f60 2023-10-30T12:42:19.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.608+0000 7f08449c5640 1 -- 172.21.15.125:0/1217952708 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f084006fe90 con 0x7f08400c5f60 2023-10-30T12:42:19.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.612+0000 7f0836ffd640 1 -- 172.21.15.125:0/1217952708 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f0830015090 con 0x7f08400c5f60 2023-10-30T12:42:19.614 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:19.614 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:15.151625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:19.615 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 70 2023-10-30T12:42:19.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.612+0000 7f08449c5640 1 -- 172.21.15.125:0/1217952708 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f08200664d0 msgr2=0x7f0820068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:19.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.612+0000 7f08449c5640 1 --2- 172.21.15.125:0/1217952708 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f08200664d0 0x7f0820068990 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f08280b2000 tx=0x7f0828009040 comp rx=0 tx=0).stop 2023-10-30T12:42:19.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.612+0000 7f08449c5640 1 -- 172.21.15.125:0/1217952708 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08400c5f60 msgr2=0x7f0840152bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:19.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.612+0000 7f08449c5640 1 --2- 172.21.15.125:0/1217952708 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08400c5f60 0x7f0840152bb0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f083000b440 tx=0x7f083000b900 comp rx=0 tx=0).stop 2023-10-30T12:42:19.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.612+0000 7f08449c5640 1 -- 172.21.15.125:0/1217952708 shutdown_connections 2023-10-30T12:42:19.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.612+0000 7f08449c5640 1 --2- 172.21.15.125:0/1217952708 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f08200664d0 0x7f0820068990 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:19.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.612+0000 7f08449c5640 1 --2- 172.21.15.125:0/1217952708 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0840153bf0 0x7f0840148300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:19.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.612+0000 7f08449c5640 1 --2- 172.21.15.125:0/1217952708 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08400c6fe0 0x7f08401f2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:19.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.612+0000 7f08449c5640 1 --2- 172.21.15.125:0/1217952708 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08400c5f60 0x7f0840152bb0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:19.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.612+0000 7f08449c5640 1 -- 172.21.15.125:0/1217952708 >> 172.21.15.125:0/1217952708 conn(0x7f084006c6e0 msgr2=0x7f08400c73c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:19.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.612+0000 7f08449c5640 1 -- 172.21.15.125:0/1217952708 shutdown_connections 2023-10-30T12:42:19.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:19.612+0000 7f08449c5640 1 -- 172.21.15.125:0/1217952708 wait complete. 2023-10-30T12:42:19.635 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:21.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:21.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.780+0000 7f8d408ea640 1 Processor -- start 2023-10-30T12:42:21.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.780+0000 7f8d408ea640 1 -- start start 2023-10-30T12:42:21.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.780+0000 7f8d408ea640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8d3c0c05d0 0x7f8d3c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:21.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.780+0000 7f8d408ea640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c0c0a50 con 0x7f8d3c0c6b80 2023-10-30T12:42:21.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.780+0000 7f8d408ea640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d3c06f6e0 con 0x7f8d3c0c6f60 2023-10-30T12:42:21.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.780+0000 7f8d408ea640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d3c06f9c0 con 0x7f8d3c0c05d0 2023-10-30T12:42:21.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.784+0000 7f8d33fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8d3c0c05d0 0x7f8d3c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:21.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.784+0000 7f8d33fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8d3c0c05d0 0x7f8d3c13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48290/0 (socket says 172.21.15.125:48290) 2023-10-30T12:42:21.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.784+0000 7f8d33fff640 1 -- 172.21.15.125:0/1483202437 learned_addr learned my addr 172.21.15.125:0/1483202437 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:21.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.784+0000 7f8d33fff640 1 -- 172.21.15.125:0/1483202437 >> v1:172.21.15.136:6789/0 conn(0x7f8d3c0c6f60 legacy=0x7f8d3c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:21.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.784+0000 7f8d33fff640 1 -- 172.21.15.125:0/1483202437 >> v1:172.21.15.125:6789/0 conn(0x7f8d3c0c6b80 legacy=0x7f8d3c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:21.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.784+0000 7f8d33fff640 1 -- 172.21.15.125:0/1483202437 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c06fca0 con 0x7f8d3c0c05d0 2023-10-30T12:42:21.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.784+0000 7f8d33fff640 1 --2- 172.21.15.125:0/1483202437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d3c0c05d0 0x7f8d3c13ed90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8d24009870 tx=0x7f8d2401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d7a3e0f7323770 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:21.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.784+0000 7f8d39d9b640 1 -- 172.21.15.125:0/1483202437 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d240ab610 con 0x7f8d3c0c05d0 2023-10-30T12:42:21.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.784+0000 7f8d39d9b640 1 -- 172.21.15.125:0/1483202437 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d240ab770 con 0x7f8d3c0c05d0 2023-10-30T12:42:21.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.784+0000 7f8d39d9b640 1 -- 172.21.15.125:0/1483202437 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d240abb00 con 0x7f8d3c0c05d0 2023-10-30T12:42:21.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 -- 172.21.15.125:0/1483202437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d3c0c05d0 msgr2=0x7f8d3c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:21.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 --2- 172.21.15.125:0/1483202437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d3c0c05d0 0x7f8d3c13ed90 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f8d24009870 tx=0x7f8d2401c500 comp rx=0 tx=0).stop 2023-10-30T12:42:21.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 -- 172.21.15.125:0/1483202437 shutdown_connections 2023-10-30T12:42:21.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 --2- 172.21.15.125:0/1483202437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d3c0c05d0 0x7f8d3c13ed90 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:21.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 -- 172.21.15.125:0/1483202437 >> 172.21.15.125:0/1483202437 conn(0x7f8d3c0c7910 msgr2=0x7f8d3c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:21.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 -- 172.21.15.125:0/1483202437 shutdown_connections 2023-10-30T12:42:21.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 -- 172.21.15.125:0/1483202437 wait complete. 2023-10-30T12:42:21.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 Processor -- start 2023-10-30T12:42:21.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 -- start start 2023-10-30T12:42:21.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8d3c0c05d0 0x7f8d3c1f6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:21.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d3c0c6b80 0x7f8d3c131a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:21.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d3c0c6f60 0x7f8d3c132920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:21.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c1f6770 con 0x7f8d3c0c6f60 2023-10-30T12:42:21.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c130750 con 0x7f8d3c0c6b80 2023-10-30T12:42:21.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d408ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c131550 con 0x7f8d3c0c05d0 2023-10-30T12:42:21.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d3ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d3c0c6f60 0x7f8d3c132920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:21.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d3ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d3c0c6f60 0x7f8d3c132920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48306/0 (socket says 172.21.15.125:48306) 2023-10-30T12:42:21.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d3ad9d640 1 -- 172.21.15.125:0/4248119791 learned_addr learned my addr 172.21.15.125:0/4248119791 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:21.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d3a59c640 1 --2- 172.21.15.125:0/4248119791 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8d3c0c05d0 0x7f8d3c1f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:21.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d33fff640 1 --2- 172.21.15.125:0/4248119791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d3c0c6b80 0x7f8d3c131a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:21.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d3ad9d640 1 -- 172.21.15.125:0/4248119791 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8d3c0c05d0 msgr2=0x7f8d3c1f6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:21.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d3ad9d640 1 --2- 172.21.15.125:0/4248119791 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8d3c0c05d0 0x7f8d3c1f6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:21.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d3ad9d640 1 -- 172.21.15.125:0/4248119791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d3c0c6b80 msgr2=0x7f8d3c131a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:21.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.788+0000 7f8d3ad9d640 1 --2- 172.21.15.125:0/4248119791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d3c0c6b80 0x7f8d3c131a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:21.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.792+0000 7f8d3ad9d640 1 -- 172.21.15.125:0/4248119791 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d240094d0 con 0x7f8d3c0c6f60 2023-10-30T12:42:21.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.792+0000 7f8d3ad9d640 1 --2- 172.21.15.125:0/4248119791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d3c0c6f60 0x7f8d3c132920 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8d3400c7b0 tx=0x7f8d3400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:21.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.792+0000 7f8d32ffd640 1 -- 172.21.15.125:0/4248119791 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d34004b70 con 0x7f8d3c0c6f60 2023-10-30T12:42:21.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.792+0000 7f8d32ffd640 1 -- 172.21.15.125:0/4248119791 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d34004cd0 con 0x7f8d3c0c6f60 2023-10-30T12:42:21.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.792+0000 7f8d408ea640 1 -- 172.21.15.125:0/4248119791 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3c130980 con 0x7f8d3c0c6f60 2023-10-30T12:42:21.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.792+0000 7f8d32ffd640 1 -- 172.21.15.125:0/4248119791 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3400aa90 con 0x7f8d3c0c6f60 2023-10-30T12:42:21.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.792+0000 7f8d408ea640 1 -- 172.21.15.125:0/4248119791 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3c130f00 con 0x7f8d3c0c6f60 2023-10-30T12:42:21.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.792+0000 7f8d408ea640 1 -- 172.21.15.125:0/4248119791 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d3c04e170 con 0x7f8d3c0c6f60 2023-10-30T12:42:21.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.792+0000 7f8d32ffd640 1 -- 172.21.15.125:0/4248119791 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8d3400abf0 con 0x7f8d3c0c6f60 2023-10-30T12:42:21.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.792+0000 7f8d32ffd640 1 --2- 172.21.15.125:0/4248119791 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8d1c0664d0 0x7f8d1c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:21.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.792+0000 7f8d32ffd640 1 -- 172.21.15.125:0/4248119791 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f8d340842c0 con 0x7f8d3c0c6f60 2023-10-30T12:42:21.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.792+0000 7f8d3a59c640 1 --2- 172.21.15.125:0/4248119791 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8d1c0664d0 0x7f8d1c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:21.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.796+0000 7f8d3a59c640 1 --2- 172.21.15.125:0/4248119791 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8d1c0664d0 0x7f8d1c068990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c007fd0 tx=0x7f8d2c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:21.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.796+0000 7f8d32ffd640 1 -- 172.21.15.125:0/4248119791 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8d340513f0 con 0x7f8d3c0c6f60 2023-10-30T12:42:21.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.976+0000 7f8d408ea640 1 -- 172.21.15.125:0/4248119791 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d3c14bce0 con 0x7f8d3c0c6f60 2023-10-30T12:42:21.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.980+0000 7f8d32ffd640 1 -- 172.21.15.125:0/4248119791 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 70 v70) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f8d34054a10 con 0x7f8d3c0c6f60 2023-10-30T12:42:21.982 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:21.982 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":70,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":70,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:15.151625+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:21.983 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 70 2023-10-30T12:42:21.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.980+0000 7f8d408ea640 1 -- 172.21.15.125:0/4248119791 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8d1c0664d0 msgr2=0x7f8d1c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:21.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.980+0000 7f8d408ea640 1 --2- 172.21.15.125:0/4248119791 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8d1c0664d0 0x7f8d1c068990 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c007fd0 tx=0x7f8d2c009040 comp rx=0 tx=0).stop 2023-10-30T12:42:21.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.980+0000 7f8d408ea640 1 -- 172.21.15.125:0/4248119791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d3c0c6f60 msgr2=0x7f8d3c132920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:21.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.980+0000 7f8d408ea640 1 --2- 172.21.15.125:0/4248119791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d3c0c6f60 0x7f8d3c132920 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8d3400c7b0 tx=0x7f8d3400cc70 comp rx=0 tx=0).stop 2023-10-30T12:42:21.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.980+0000 7f8d408ea640 1 -- 172.21.15.125:0/4248119791 shutdown_connections 2023-10-30T12:42:21.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.980+0000 7f8d408ea640 1 --2- 172.21.15.125:0/4248119791 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8d1c0664d0 0x7f8d1c068990 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:21.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.980+0000 7f8d408ea640 1 --2- 172.21.15.125:0/4248119791 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8d3c0c6f60 0x7f8d3c132920 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:21.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.980+0000 7f8d408ea640 1 --2- 172.21.15.125:0/4248119791 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8d3c0c6b80 0x7f8d3c131a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:21.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.980+0000 7f8d408ea640 1 --2- 172.21.15.125:0/4248119791 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8d3c0c05d0 0x7f8d3c1f6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:21.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.980+0000 7f8d408ea640 1 -- 172.21.15.125:0/4248119791 >> 172.21.15.125:0/4248119791 conn(0x7f8d3c0c7910 msgr2=0x7f8d3c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:21.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.980+0000 7f8d408ea640 1 -- 172.21.15.125:0/4248119791 shutdown_connections 2023-10-30T12:42:21.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:21.980+0000 7f8d408ea640 1 -- 172.21.15.125:0/4248119791 wait complete. 2023-10-30T12:42:22.003 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:24.004 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:24.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f7668daa640 1 Processor -- start 2023-10-30T12:42:24.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f7668daa640 1 -- start start 2023-10-30T12:42:24.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f7668daa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76640cf160 0x7f76640cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:24.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f7668daa640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76640cb800 con 0x7f76641538a0 2023-10-30T12:42:24.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f7668daa640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76640cb330 con 0x7f766415d6f0 2023-10-30T12:42:24.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f7668daa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76640cad70 con 0x7f76640cf160 2023-10-30T12:42:24.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766259c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f76641538a0 0x7f76640ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41648/0 (socket says 172.21.15.125:41648) 2023-10-30T12:42:24.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766259c640 1 -- 172.21.15.125:0/2968923405 learned_addr learned my addr 172.21.15.125:0/2968923405 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:42:24.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f7662d9d640 1 --2- 172.21.15.125:0/2968923405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76640cf160 0x7f76640cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:24.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 -- 172.21.15.125:0/2968923405 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567055816 0 0) 0x7f76640cb800 con 0x7f76641538a0 2023-10-30T12:42:24.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 -- 172.21.15.125:0/2968923405 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f764c003530 con 0x7f76641538a0 2023-10-30T12:42:24.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 -- 172.21.15.125:0/2968923405 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 98773472 0 0) 0x7f764c003530 con 0x7f76641538a0 2023-10-30T12:42:24.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 -- 172.21.15.125:0/2968923405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76640cf160 msgr2=0x7f76640cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:24.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 --2- 172.21.15.125:0/2968923405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76640cf160 0x7f76640cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:24.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 -- 172.21.15.125:0/2968923405 >> v1:172.21.15.136:6789/0 conn(0x7f766415d6f0 legacy=0x7f766415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:24.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 -- 172.21.15.125:0/2968923405 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76640cb050 con 0x7f76641538a0 2023-10-30T12:42:24.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 -- 172.21.15.125:0/2968923405 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f7654001680 con 0x7f76641538a0 2023-10-30T12:42:24.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 -- 172.21.15.125:0/2968923405 >> v1:172.21.15.125:6789/0 conn(0x7f76641538a0 legacy=0x7f76640ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:24.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 --2- 172.21.15.125:0/2968923405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f764c0039b0 0x7f764c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:24.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 --2- 172.21.15.125:0/2968923405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f764c0047d0 0x7f764c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:24.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 --2- 172.21.15.125:0/2968923405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76640cf160 0x7f764c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:24.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 -- 172.21.15.125:0/2968923405 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76640cad70 con 0x7f764c0047d0 2023-10-30T12:42:24.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 -- 172.21.15.125:0/2968923405 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764c00bad0 con 0x7f764c0039b0 2023-10-30T12:42:24.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766159a640 1 -- 172.21.15.125:0/2968923405 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f764c00bc90 con 0x7f76640cf160 2023-10-30T12:42:24.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f7661d9b640 1 --2- 172.21.15.125:0/2968923405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f764c0047d0 0x7f764c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:24.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f7662d9d640 1 --2- 172.21.15.125:0/2968923405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76640cf160 0x7f764c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:24.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766259c640 1 --2- 172.21.15.125:0/2968923405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f764c0039b0 0x7f764c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:24.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f7661d9b640 1 -- 172.21.15.125:0/2968923405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76640cf160 msgr2=0x7f764c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:24.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f7661d9b640 1 --2- 172.21.15.125:0/2968923405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76640cf160 0x7f764c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:24.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f7661d9b640 1 -- 172.21.15.125:0/2968923405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f764c0039b0 msgr2=0x7f764c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:24.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f7661d9b640 1 --2- 172.21.15.125:0/2968923405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f764c0039b0 0x7f764c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:24.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f7661d9b640 1 -- 172.21.15.125:0/2968923405 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764c003530 con 0x7f764c0047d0 2023-10-30T12:42:24.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.148+0000 7f766259c640 1 --2- 172.21.15.125:0/2968923405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f764c0039b0 0x7f764c003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:24.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.152+0000 7f7661d9b640 1 --2- 172.21.15.125:0/2968923405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f764c0047d0 0x7f764c004c50 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7658001750 tx=0x7f765801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59059dcf9b6583e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:24.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.152+0000 7f766159a640 1 -- 172.21.15.125:0/2968923405 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76580052d0 con 0x7f764c0047d0 2023-10-30T12:42:24.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.152+0000 7f766159a640 1 -- 172.21.15.125:0/2968923405 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76580a7050 con 0x7f764c0047d0 2023-10-30T12:42:24.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.152+0000 7f766159a640 1 -- 172.21.15.125:0/2968923405 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658003600 con 0x7f764c0047d0 2023-10-30T12:42:24.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.152+0000 7f7668daa640 1 -- 172.21.15.125:0/2968923405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f764c0047d0 msgr2=0x7f764c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:24.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.152+0000 7f7668daa640 1 --2- 172.21.15.125:0/2968923405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f764c0047d0 0x7f764c004c50 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f7658001750 tx=0x7f765801c3a0 comp rx=0 tx=0).stop 2023-10-30T12:42:24.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.152+0000 7f7662d9d640 1 -- 172.21.15.125:0/2968923405 reap_dead start 2023-10-30T12:42:24.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.152+0000 7f7668daa640 1 -- 172.21.15.125:0/2968923405 shutdown_connections 2023-10-30T12:42:24.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.152+0000 7f7668daa640 1 -- 172.21.15.125:0/2968923405 >> 172.21.15.125:0/2968923405 conn(0x7f766406bfe0 msgr2=0x7f766406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:24.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.152+0000 7f7668daa640 1 -- 172.21.15.125:0/2968923405 shutdown_connections 2023-10-30T12:42:24.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.152+0000 7f7668daa640 1 -- 172.21.15.125:0/2968923405 wait complete. 2023-10-30T12:42:24.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7668daa640 1 Processor -- start 2023-10-30T12:42:24.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7668daa640 1 -- start start 2023-10-30T12:42:24.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7668daa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f764c0047d0 0x7f766414ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:24.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7668daa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7664150210 0x7f76640b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:24.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7668daa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7664150860 0x7f76641f3120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:24.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7668daa640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7664152e20 con 0x7f7664150210 2023-10-30T12:42:24.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7668daa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76641f2990 con 0x7f7664150860 2023-10-30T12:42:24.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7668daa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f766414dcd0 con 0x7f764c0047d0 2023-10-30T12:42:24.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7661d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7664150210 0x7f76640b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:24.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7661d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7664150210 0x7f76640b44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48318/0 (socket says 172.21.15.125:48318) 2023-10-30T12:42:24.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7661d9b640 1 -- 172.21.15.125:0/3737876028 learned_addr learned my addr 172.21.15.125:0/3737876028 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:24.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7662d9d640 1 --2- 172.21.15.125:0/3737876028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7664150860 0x7f76641f3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:24.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f766259c640 1 --2- 172.21.15.125:0/3737876028 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f764c0047d0 0x7f766414ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:24.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7661d9b640 1 -- 172.21.15.125:0/3737876028 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f764c0047d0 msgr2=0x7f766414ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:24.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7661d9b640 1 --2- 172.21.15.125:0/3737876028 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f764c0047d0 0x7f766414ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:24.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7661d9b640 1 -- 172.21.15.125:0/3737876028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7664150860 msgr2=0x7f76641f3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:24.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7661d9b640 1 --2- 172.21.15.125:0/3737876028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7664150860 0x7f76641f3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:24.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7661d9b640 1 -- 172.21.15.125:0/3737876028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766414df90 con 0x7f7664150210 2023-10-30T12:42:24.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7661d9b640 1 --2- 172.21.15.125:0/3737876028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7664150210 0x7f76640b44f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7658005040 tx=0x7f764c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:24.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7652ffd640 1 -- 172.21.15.125:0/3737876028 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658007a40 con 0x7f7664150210 2023-10-30T12:42:24.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7652ffd640 1 -- 172.21.15.125:0/3737876028 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7658007680 con 0x7f7664150210 2023-10-30T12:42:24.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7668daa640 1 -- 172.21.15.125:0/3737876028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76641f2c00 con 0x7f7664150210 2023-10-30T12:42:24.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.156+0000 7f7652ffd640 1 -- 172.21.15.125:0/3737876028 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7658006a00 con 0x7f7664150210 2023-10-30T12:42:24.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.160+0000 7f7668daa640 1 -- 172.21.15.125:0/3737876028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76640b4a30 con 0x7f7664150210 2023-10-30T12:42:24.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.160+0000 7f7668daa640 1 -- 172.21.15.125:0/3737876028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f766404e170 con 0x7f7664150210 2023-10-30T12:42:24.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.160+0000 7f7652ffd640 1 -- 172.21.15.125:0/3737876028 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7658006b60 con 0x7f7664150210 2023-10-30T12:42:24.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.160+0000 7f7652ffd640 1 --2- 172.21.15.125:0/3737876028 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f76440664d0 0x7f7644068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:24.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.160+0000 7f7652ffd640 1 -- 172.21.15.125:0/3737876028 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f765811e870 con 0x7f7664150210 2023-10-30T12:42:24.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.160+0000 7f766259c640 1 --2- 172.21.15.125:0/3737876028 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f76440664d0 0x7f7644068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:24.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.160+0000 7f766259c640 1 --2- 172.21.15.125:0/3737876028 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f76440664d0 0x7f7644068990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7654001210 tx=0x7f7654009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:24.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.164+0000 7f7652ffd640 1 -- 172.21.15.125:0/3737876028 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f76580eb9a0 con 0x7f7664150210 2023-10-30T12:42:24.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.344+0000 7f7668daa640 1 -- 172.21.15.125:0/3737876028 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f766415d4d0 con 0x7f7664150210 2023-10-30T12:42:24.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.344+0000 7f7652ffd640 1 -- 172.21.15.125:0/3737876028 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f76580eefc0 con 0x7f7664150210 2023-10-30T12:42:24.348 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:24.349 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:21.337703+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:resolve","state_seq":16,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:24.349 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 71 2023-10-30T12:42:24.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.348+0000 7f7668daa640 1 -- 172.21.15.125:0/3737876028 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f76440664d0 msgr2=0x7f7644068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.348+0000 7f7668daa640 1 --2- 172.21.15.125:0/3737876028 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f76440664d0 0x7f7644068990 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7654001210 tx=0x7f7654009040 comp rx=0 tx=0).stop 2023-10-30T12:42:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.348+0000 7f7668daa640 1 -- 172.21.15.125:0/3737876028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7664150210 msgr2=0x7f76640b44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.348+0000 7f7668daa640 1 --2- 172.21.15.125:0/3737876028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7664150210 0x7f76640b44f0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7658005040 tx=0x7f764c005800 comp rx=0 tx=0).stop 2023-10-30T12:42:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.348+0000 7f7668daa640 1 -- 172.21.15.125:0/3737876028 shutdown_connections 2023-10-30T12:42:24.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.348+0000 7f7668daa640 1 --2- 172.21.15.125:0/3737876028 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f76440664d0 0x7f7644068990 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:24.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.348+0000 7f7668daa640 1 --2- 172.21.15.125:0/3737876028 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7664150860 0x7f76641f3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:24.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.348+0000 7f7668daa640 1 --2- 172.21.15.125:0/3737876028 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7664150210 0x7f76640b44f0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:24.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.348+0000 7f7668daa640 1 --2- 172.21.15.125:0/3737876028 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f764c0047d0 0x7f766414ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:24.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.348+0000 7f7668daa640 1 -- 172.21.15.125:0/3737876028 >> 172.21.15.125:0/3737876028 conn(0x7f766406bfe0 msgr2=0x7f76640c49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:24.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.348+0000 7f7668daa640 1 -- 172.21.15.125:0/3737876028 shutdown_connections 2023-10-30T12:42:24.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:24.348+0000 7f7668daa640 1 -- 172.21.15.125:0/3737876028 wait complete. 2023-10-30T12:42:24.370 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:26.371 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:26.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41d029e640 1 Processor -- start 2023-10-30T12:42:26.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41d029e640 1 -- start start 2023-10-30T12:42:26.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41d029e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f41c8153bf0 0x7f41c8071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:26.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41d029e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f41c80c6fe0 0x7f41c80c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:26.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41d029e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41c80700b0 con 0x7f41c80c79e0 2023-10-30T12:42:26.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41d029e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c80c6810 con 0x7f41c8153bf0 2023-10-30T12:42:26.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41d029e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41c8159900 con 0x7f41c80c6fe0 2023-10-30T12:42:26.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41ce03a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f41c8153bf0 0x7f41c8071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:26.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41ce03a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f41c8153bf0 0x7f41c8071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48332/0 (socket says 172.21.15.125:48332) 2023-10-30T12:42:26.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41ce03a640 1 -- 172.21.15.125:0/92060599 learned_addr learned my addr 172.21.15.125:0/92060599 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:26.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41cd839640 1 --2- 172.21.15.125:0/92060599 >> v2:172.21.15.136:3300/0 conn(0x7f41c80c6fe0 0x7f41c80c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:26.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41ce03a640 1 -- 172.21.15.125:0/92060599 >> v2:172.21.15.136:3300/0 conn(0x7f41c80c6fe0 msgr2=0x7f41c80c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:26.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41ce03a640 1 --2- 172.21.15.125:0/92060599 >> v2:172.21.15.136:3300/0 conn(0x7f41c80c6fe0 0x7f41c80c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:26.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41ce03a640 1 -- 172.21.15.125:0/92060599 >> v1:172.21.15.125:6789/0 conn(0x7f41c80c79e0 legacy=0x7f41c80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:26.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.516+0000 7f41ce03a640 1 -- 172.21.15.125:0/92060599 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c80c6530 con 0x7f41c8153bf0 2023-10-30T12:42:26.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.520+0000 7f41ce03a640 1 --2- 172.21.15.125:0/92060599 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41c8153bf0 0x7f41c8071260 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f41c80723d0 tx=0x7f41c401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6615aab898a173ca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:26.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.520+0000 7f41cd038640 1 -- 172.21.15.125:0/92060599 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c40a89f0 con 0x7f41c8153bf0 2023-10-30T12:42:26.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.520+0000 7f41cd038640 1 -- 172.21.15.125:0/92060599 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c40a8b50 con 0x7f41c8153bf0 2023-10-30T12:42:26.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.520+0000 7f41cd038640 1 -- 172.21.15.125:0/92060599 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c40b0440 con 0x7f41c8153bf0 2023-10-30T12:42:26.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.520+0000 7f41d029e640 1 -- 172.21.15.125:0/92060599 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41c8153bf0 msgr2=0x7f41c8071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:26.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.520+0000 7f41d029e640 1 --2- 172.21.15.125:0/92060599 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41c8153bf0 0x7f41c8071260 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f41c80723d0 tx=0x7f41c401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:42:26.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.520+0000 7f41d029e640 1 -- 172.21.15.125:0/92060599 shutdown_connections 2023-10-30T12:42:26.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.520+0000 7f41d029e640 1 --2- 172.21.15.125:0/92060599 >> v2:172.21.15.136:3300/0 conn(0x7f41c80c6fe0 0x7f41c80c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:26.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.520+0000 7f41d029e640 1 --2- 172.21.15.125:0/92060599 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41c8153bf0 0x7f41c8071260 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:26.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.520+0000 7f41d029e640 1 -- 172.21.15.125:0/92060599 >> 172.21.15.125:0/92060599 conn(0x7f41c806c6e0 msgr2=0x7f41c8069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:26.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.520+0000 7f41d029e640 1 -- 172.21.15.125:0/92060599 shutdown_connections 2023-10-30T12:42:26.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.520+0000 7f41d029e640 1 -- 172.21.15.125:0/92060599 wait complete. 2023-10-30T12:42:26.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41d029e640 1 Processor -- start 2023-10-30T12:42:26.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41d029e640 1 -- start start 2023-10-30T12:42:26.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41d029e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41c80c6fe0 0x7f41c814cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:26.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41d029e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41c80c79e0 0x7f41c8154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:26.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41d029e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41c8153bf0 0x7f41c80ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:26.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41d029e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c8152e80 con 0x7f41c80c79e0 2023-10-30T12:42:26.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41d029e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c8153420 con 0x7f41c80c6fe0 2023-10-30T12:42:26.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41d029e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41c814d080 con 0x7f41c8153bf0 2023-10-30T12:42:26.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41cd839640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41c80c79e0 0x7f41c8154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:26.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41cd839640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41c80c79e0 0x7f41c8154930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48340/0 (socket says 172.21.15.125:48340) 2023-10-30T12:42:26.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41cd839640 1 -- 172.21.15.125:0/1746457041 learned_addr learned my addr 172.21.15.125:0/1746457041 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:26.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41ce03a640 1 --2- 172.21.15.125:0/1746457041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41c80c6fe0 0x7f41c814cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:26.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41ce83b640 1 --2- 172.21.15.125:0/1746457041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41c8153bf0 0x7f41c80ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:26.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41cd839640 1 -- 172.21.15.125:0/1746457041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41c8153bf0 msgr2=0x7f41c80ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:26.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41cd839640 1 --2- 172.21.15.125:0/1746457041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41c8153bf0 0x7f41c80ca6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:26.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41cd839640 1 -- 172.21.15.125:0/1746457041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41c80c6fe0 msgr2=0x7f41c814cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:26.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41cd839640 1 --2- 172.21.15.125:0/1746457041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41c80c6fe0 0x7f41c814cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:26.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41cd839640 1 -- 172.21.15.125:0/1746457041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c40094d0 con 0x7f41c80c79e0 2023-10-30T12:42:26.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41cd839640 1 --2- 172.21.15.125:0/1746457041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41c80c79e0 0x7f41c8154930 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f41b800d800 tx=0x7f41b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:26.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41beffd640 1 -- 172.21.15.125:0/1746457041 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b800b6c0 con 0x7f41c80c79e0 2023-10-30T12:42:26.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41beffd640 1 -- 172.21.15.125:0/1746457041 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41b800b820 con 0x7f41c80c79e0 2023-10-30T12:42:26.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41beffd640 1 -- 172.21.15.125:0/1746457041 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b800bb90 con 0x7f41c80c79e0 2023-10-30T12:42:26.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.524+0000 7f41d029e640 1 -- 172.21.15.125:0/1746457041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c8153140 con 0x7f41c80c79e0 2023-10-30T12:42:26.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.528+0000 7f41d029e640 1 -- 172.21.15.125:0/1746457041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c814be00 con 0x7f41c80c79e0 2023-10-30T12:42:26.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.528+0000 7f41d029e640 1 -- 172.21.15.125:0/1746457041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c804e170 con 0x7f41c80c79e0 2023-10-30T12:42:26.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.528+0000 7f41beffd640 1 -- 172.21.15.125:0/1746457041 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f41b800bcf0 con 0x7f41c80c79e0 2023-10-30T12:42:26.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.528+0000 7f41beffd640 1 --2- 172.21.15.125:0/1746457041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41ac0664a0 0x7f41ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:26.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.528+0000 7f41beffd640 1 -- 172.21.15.125:0/1746457041 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f41b80845d0 con 0x7f41c80c79e0 2023-10-30T12:42:26.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.528+0000 7f41ce03a640 1 --2- 172.21.15.125:0/1746457041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41ac0664a0 0x7f41ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:26.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.528+0000 7f41ce03a640 1 --2- 172.21.15.125:0/1746457041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41ac0664a0 0x7f41ac068960 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f41c40b2000 tx=0x7f41c4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:26.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.532+0000 7f41beffd640 1 -- 172.21.15.125:0/1746457041 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f41b8051820 con 0x7f41c80c79e0 2023-10-30T12:42:26.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.716+0000 7f41d029e640 1 -- 172.21.15.125:0/1746457041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41c806fe90 con 0x7f41c80c79e0 2023-10-30T12:42:26.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.716+0000 7f41beffd640 1 -- 172.21.15.125:0/1746457041 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f41b8054e40 con 0x7f41c80c79e0 2023-10-30T12:42:26.719 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:26.719 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:21.337703+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:resolve","state_seq":16,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:26.720 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 71 2023-10-30T12:42:26.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.716+0000 7f41d029e640 1 -- 172.21.15.125:0/1746457041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41ac0664a0 msgr2=0x7f41ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:26.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.716+0000 7f41d029e640 1 --2- 172.21.15.125:0/1746457041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41ac0664a0 0x7f41ac068960 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f41c40b2000 tx=0x7f41c4008040 comp rx=0 tx=0).stop 2023-10-30T12:42:26.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.716+0000 7f41d029e640 1 -- 172.21.15.125:0/1746457041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41c80c79e0 msgr2=0x7f41c8154930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:26.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.716+0000 7f41d029e640 1 --2- 172.21.15.125:0/1746457041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41c80c79e0 0x7f41c8154930 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f41b800d800 tx=0x7f41b800dcc0 comp rx=0 tx=0).stop 2023-10-30T12:42:26.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.716+0000 7f41d029e640 1 -- 172.21.15.125:0/1746457041 shutdown_connections 2023-10-30T12:42:26.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.716+0000 7f41d029e640 1 --2- 172.21.15.125:0/1746457041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41ac0664a0 0x7f41ac068960 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:26.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.716+0000 7f41d029e640 1 --2- 172.21.15.125:0/1746457041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41c8153bf0 0x7f41c80ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:26.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.716+0000 7f41d029e640 1 --2- 172.21.15.125:0/1746457041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41c80c79e0 0x7f41c8154930 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:26.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.716+0000 7f41d029e640 1 --2- 172.21.15.125:0/1746457041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41c80c6fe0 0x7f41c814cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:26.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.716+0000 7f41d029e640 1 -- 172.21.15.125:0/1746457041 >> 172.21.15.125:0/1746457041 conn(0x7f41c806c6e0 msgr2=0x7f41c8158a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:26.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.716+0000 7f41d029e640 1 -- 172.21.15.125:0/1746457041 shutdown_connections 2023-10-30T12:42:26.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:26.716+0000 7f41d029e640 1 -- 172.21.15.125:0/1746457041 wait complete. 2023-10-30T12:42:26.739 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:28.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:28.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.880+0000 7fa0b3f0f640 1 Processor -- start 2023-10-30T12:42:28.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.884+0000 7fa0b3f0f640 1 -- start start 2023-10-30T12:42:28.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.884+0000 7fa0b3f0f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa0ac1538a0 0x7fa0ac15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:28.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.884+0000 7fa0b3f0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ac0cf160 0x7fa0ac0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:28.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.884+0000 7fa0b3f0f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ac0cb800 con 0x7fa0ac0ca0e0 2023-10-30T12:42:28.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.884+0000 7fa0b3f0f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0ac15dc30 con 0x7fa0ac1538a0 2023-10-30T12:42:28.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.884+0000 7fa0b3f0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ac0cb330 con 0x7fa0ac0cf160 2023-10-30T12:42:28.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.884+0000 7fa0b24ac640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ac0cf160 0x7fa0ac0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:28.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.884+0000 7fa0b1cab640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa0ac1538a0 0x7fa0ac15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:28.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.884+0000 7fa0b24ac640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ac0cf160 0x7fa0ac0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:34246/0 (socket says 172.21.15.125:34246) 2023-10-30T12:42:28.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.884+0000 7fa0b24ac640 1 -- 172.21.15.125:0/1756125267 learned_addr learned my addr 172.21.15.125:0/1756125267 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:28.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.884+0000 7fa0b24ac640 1 -- 172.21.15.125:0/1756125267 >> v2:172.21.15.136:3300/0 conn(0x7fa0ac1538a0 msgr2=0x7fa0ac15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:28.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.884+0000 7fa0b24ac640 1 --2- 172.21.15.125:0/1756125267 >> v2:172.21.15.136:3300/0 conn(0x7fa0ac1538a0 0x7fa0ac15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.884+0000 7fa0b24ac640 1 -- 172.21.15.125:0/1756125267 >> v1:172.21.15.136:6789/0 conn(0x7fa0ac0ca0e0 legacy=0x7fa0ac15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:28.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.884+0000 7fa0b24ac640 1 -- 172.21.15.125:0/1756125267 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ac0cad70 con 0x7fa0ac0cf160 2023-10-30T12:42:28.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.888+0000 7fa0b24ac640 1 --2- 172.21.15.125:0/1756125267 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ac0cf160 0x7fa0ac0cf590 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009e50 tx=0x7fa0a401f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad34e21920fee515 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:28.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.888+0000 7fa0b0ca9640 1 -- 172.21.15.125:0/1756125267 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a40af400 con 0x7fa0ac0cf160 2023-10-30T12:42:28.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.888+0000 7fa0b0ca9640 1 -- 172.21.15.125:0/1756125267 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a40ab050 con 0x7fa0ac0cf160 2023-10-30T12:42:28.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.888+0000 7fa0b0ca9640 1 -- 172.21.15.125:0/1756125267 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a40af780 con 0x7fa0ac0cf160 2023-10-30T12:42:28.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.888+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1756125267 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ac0cf160 msgr2=0x7fa0ac0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:28.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.888+0000 7fa0b3f0f640 1 --2- 172.21.15.125:0/1756125267 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ac0cf160 0x7fa0ac0cf590 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009e50 tx=0x7fa0a401f1f0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.888+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1756125267 shutdown_connections 2023-10-30T12:42:28.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.888+0000 7fa0b3f0f640 1 --2- 172.21.15.125:0/1756125267 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ac0cf160 0x7fa0ac0cf590 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.888+0000 7fa0b3f0f640 1 --2- 172.21.15.125:0/1756125267 >> v2:172.21.15.136:3300/0 conn(0x7fa0ac1538a0 0x7fa0ac15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.888+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1756125267 >> 172.21.15.125:0/1756125267 conn(0x7fa0ac06bfe0 msgr2=0x7fa0ac06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:28.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.888+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1756125267 shutdown_connections 2023-10-30T12:42:28.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.888+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1756125267 wait complete. 2023-10-30T12:42:28.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b3f0f640 1 Processor -- start 2023-10-30T12:42:28.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b3f0f640 1 -- start start 2023-10-30T12:42:28.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b3f0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ac0ca0e0 0x7fa0ac0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:28.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b3f0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0ac0cf160 0x7fa0ac142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:28.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b3f0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0ac1538a0 0x7fa0ac157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:28.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b3f0f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ac0ce820 con 0x7fa0ac0cf160 2023-10-30T12:42:28.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b3f0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ac0ceae0 con 0x7fa0ac1538a0 2023-10-30T12:42:28.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b3f0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ac0cedc0 con 0x7fa0ac0ca0e0 2023-10-30T12:42:28.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b14aa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0ac0cf160 0x7fa0ac142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:28.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b14aa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0ac0cf160 0x7fa0ac142550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48348/0 (socket says 172.21.15.125:48348) 2023-10-30T12:42:28.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b14aa640 1 -- 172.21.15.125:0/1305453682 learned_addr learned my addr 172.21.15.125:0/1305453682 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:28.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b1cab640 1 --2- 172.21.15.125:0/1305453682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ac0ca0e0 0x7fa0ac0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:28.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b24ac640 1 --2- 172.21.15.125:0/1305453682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0ac1538a0 0x7fa0ac157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:28.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b14aa640 1 -- 172.21.15.125:0/1305453682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ac0ca0e0 msgr2=0x7fa0ac0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:28.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b14aa640 1 --2- 172.21.15.125:0/1305453682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ac0ca0e0 0x7fa0ac0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b14aa640 1 -- 172.21.15.125:0/1305453682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0ac1538a0 msgr2=0x7fa0ac157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:28.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b14aa640 1 --2- 172.21.15.125:0/1305453682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0ac1538a0 0x7fa0ac157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:28.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b14aa640 1 -- 172.21.15.125:0/1305453682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a4009b30 con 0x7fa0ac0cf160 2023-10-30T12:42:28.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b14aa640 1 --2- 172.21.15.125:0/1305453682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0ac0cf160 0x7fa0ac142550 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa09c00c500 tx=0x7fa09c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:28.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0a27fc640 1 -- 172.21.15.125:0/1305453682 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09c005220 con 0x7fa0ac0cf160 2023-10-30T12:42:28.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0a27fc640 1 -- 172.21.15.125:0/1305453682 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09c005380 con 0x7fa0ac0cf160 2023-10-30T12:42:28.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0a27fc640 1 -- 172.21.15.125:0/1305453682 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09c007270 con 0x7fa0ac0cf160 2023-10-30T12:42:28.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1305453682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0ac1efb80 con 0x7fa0ac0cf160 2023-10-30T12:42:28.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1305453682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0ac1f0140 con 0x7fa0ac0cf160 2023-10-30T12:42:28.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.892+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1305453682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0ac0b2a50 con 0x7fa0ac0cf160 2023-10-30T12:42:28.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.896+0000 7fa0a27fc640 1 -- 172.21.15.125:0/1305453682 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa09c0073d0 con 0x7fa0ac0cf160 2023-10-30T12:42:28.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.896+0000 7fa0a27fc640 1 --2- 172.21.15.125:0/1305453682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0900664a0 0x7fa090068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:28.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.896+0000 7fa0a27fc640 1 -- 172.21.15.125:0/1305453682 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7fa09c083930 con 0x7fa0ac0cf160 2023-10-30T12:42:28.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.896+0000 7fa0b1cab640 1 --2- 172.21.15.125:0/1305453682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0900664a0 0x7fa090068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:28.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.896+0000 7fa0b1cab640 1 --2- 172.21.15.125:0/1305453682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0900664a0 0x7fa090068960 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8009ac0 tx=0x7fa0a8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:28.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:28.900+0000 7fa0a27fc640 1 -- 172.21.15.125:0/1305453682 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa09c050bb0 con 0x7fa0ac0cf160 2023-10-30T12:42:29.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:29.084+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1305453682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0ac159850 con 0x7fa0ac0cf160 2023-10-30T12:42:29.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:29.084+0000 7fa0a27fc640 1 -- 172.21.15.125:0/1305453682 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5740 (secure 0 0 0) 0x7fa09c0541d0 con 0x7fa0ac0cf160 2023-10-30T12:42:29.088 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:29.089 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:27.441795+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:29.089 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 72 2023-10-30T12:42:29.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:29.088+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1305453682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0900664a0 msgr2=0x7fa090068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:29.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:29.088+0000 7fa0b3f0f640 1 --2- 172.21.15.125:0/1305453682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0900664a0 0x7fa090068960 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8009ac0 tx=0x7fa0a8009190 comp rx=0 tx=0).stop 2023-10-30T12:42:29.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:29.088+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1305453682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0ac0cf160 msgr2=0x7fa0ac142550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:29.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:29.088+0000 7fa0b3f0f640 1 --2- 172.21.15.125:0/1305453682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0ac0cf160 0x7fa0ac142550 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fa09c00c500 tx=0x7fa09c00c9c0 comp rx=0 tx=0).stop 2023-10-30T12:42:29.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:29.088+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1305453682 shutdown_connections 2023-10-30T12:42:29.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:29.088+0000 7fa0b3f0f640 1 --2- 172.21.15.125:0/1305453682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0900664a0 0x7fa090068960 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:29.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:29.088+0000 7fa0b3f0f640 1 --2- 172.21.15.125:0/1305453682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0ac1538a0 0x7fa0ac157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:29.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:29.088+0000 7fa0b3f0f640 1 --2- 172.21.15.125:0/1305453682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0ac0cf160 0x7fa0ac142550 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:29.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:29.088+0000 7fa0b3f0f640 1 --2- 172.21.15.125:0/1305453682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0ac0ca0e0 0x7fa0ac0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:29.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:29.088+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1305453682 >> 172.21.15.125:0/1305453682 conn(0x7fa0ac06bfe0 msgr2=0x7fa0ac0c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:29.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:29.088+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1305453682 shutdown_connections 2023-10-30T12:42:29.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:29.088+0000 7fa0b3f0f640 1 -- 172.21.15.125:0/1305453682 wait complete. 2023-10-30T12:42:29.110 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 72, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 72, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5345": { "addr": "172.21.15.136:6839/3922559697", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3922559697, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3922559697, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5345, "incarnation": 70, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:reconnect", "state_seq": 18 }, "gid_5433": { "addr": "172.21.15.125:6836/2595438444", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2595438444, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2595438444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5433, "incarnation": 62, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 69 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:42:27.441795+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5433, "mds_1": 5345 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:42:31.111 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:31.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.252+0000 7f93b2617640 1 Processor -- start 2023-10-30T12:42:31.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.252+0000 7f93b2617640 1 -- start start 2023-10-30T12:42:31.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.256+0000 7f93b2617640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f93ac0c6b80 0x7f93ac0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:31.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.256+0000 7f93b2617640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f93ac0c6f60 0x7f93ac0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:31.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.256+0000 7f93b2617640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac0c0a50 con 0x7f93ac0c8430 2023-10-30T12:42:31.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.256+0000 7f93b2617640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ac06f6e0 con 0x7f93ac0c6b80 2023-10-30T12:42:31.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.256+0000 7f93b2617640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ac06f9c0 con 0x7f93ac0c6f60 2023-10-30T12:42:31.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.256+0000 7f93abfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f93ac0c6b80 0x7f93ac0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:31.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.256+0000 7f93abfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f93ac0c6b80 0x7f93ac0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56626/0 (socket says 172.21.15.125:56626) 2023-10-30T12:42:31.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.256+0000 7f93abfff640 1 -- 172.21.15.125:0/2761786609 learned_addr learned my addr 172.21.15.125:0/2761786609 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:31.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.256+0000 7f93ab7fe640 1 --2- 172.21.15.125:0/2761786609 >> v2:172.21.15.136:3300/0 conn(0x7f93ac0c6f60 0x7f93ac0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:31.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.256+0000 7f93abfff640 1 -- 172.21.15.125:0/2761786609 >> v2:172.21.15.136:3300/0 conn(0x7f93ac0c6f60 msgr2=0x7f93ac0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:31.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.256+0000 7f93abfff640 1 --2- 172.21.15.125:0/2761786609 >> v2:172.21.15.136:3300/0 conn(0x7f93ac0c6f60 0x7f93ac0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:31.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.256+0000 7f93abfff640 1 -- 172.21.15.125:0/2761786609 >> v1:172.21.15.125:6789/0 conn(0x7f93ac0c8430 legacy=0x7f93ac13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:31.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.256+0000 7f93abfff640 1 -- 172.21.15.125:0/2761786609 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac06fca0 con 0x7f93ac0c6b80 2023-10-30T12:42:31.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.260+0000 7f93abfff640 1 --2- 172.21.15.125:0/2761786609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ac0c6b80 0x7f93ac0bfec0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f93ac159b90 tx=0x7f939c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb097a716a68465a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:31.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.260+0000 7f93aaffd640 1 -- 172.21.15.125:0/2761786609 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c0ab610 con 0x7f93ac0c6b80 2023-10-30T12:42:31.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.260+0000 7f93aaffd640 1 -- 172.21.15.125:0/2761786609 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939c0ab770 con 0x7f93ac0c6b80 2023-10-30T12:42:31.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.260+0000 7f93aaffd640 1 -- 172.21.15.125:0/2761786609 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c0abb00 con 0x7f93ac0c6b80 2023-10-30T12:42:31.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.260+0000 7f93b2617640 1 -- 172.21.15.125:0/2761786609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ac0c6b80 msgr2=0x7f93ac0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:31.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.260+0000 7f93b2617640 1 --2- 172.21.15.125:0/2761786609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ac0c6b80 0x7f93ac0bfec0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f93ac159b90 tx=0x7f939c01c500 comp rx=0 tx=0).stop 2023-10-30T12:42:31.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.260+0000 7f93b2617640 1 -- 172.21.15.125:0/2761786609 shutdown_connections 2023-10-30T12:42:31.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.260+0000 7f93b2617640 1 --2- 172.21.15.125:0/2761786609 >> v2:172.21.15.136:3300/0 conn(0x7f93ac0c6f60 0x7f93ac0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:31.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.260+0000 7f93b2617640 1 --2- 172.21.15.125:0/2761786609 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ac0c6b80 0x7f93ac0bfec0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:31.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.260+0000 7f93b2617640 1 -- 172.21.15.125:0/2761786609 >> 172.21.15.125:0/2761786609 conn(0x7f93ac0c7910 msgr2=0x7f93ac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:31.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.260+0000 7f93b2617640 1 -- 172.21.15.125:0/2761786609 shutdown_connections 2023-10-30T12:42:31.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.260+0000 7f93b2617640 1 -- 172.21.15.125:0/2761786609 wait complete. 2023-10-30T12:42:31.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.260+0000 7f93b2617640 1 Processor -- start 2023-10-30T12:42:31.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.260+0000 7f93b2617640 1 -- start start 2023-10-30T12:42:31.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93b2617640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ac0c6b80 0x7f93ac153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:31.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93b2617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93ac0c6f60 0x7f93ac13a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:31.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93b2617640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93ac0c8430 0x7f93ac13ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:31.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93b2617640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ac0cabb0 con 0x7f93ac0c6b80 2023-10-30T12:42:31.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93b2617640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ac0ca0a0 con 0x7f93ac0c8430 2023-10-30T12:42:31.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93b2617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ac153fa0 con 0x7f93ac0c6f60 2023-10-30T12:42:31.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93abfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ac0c6b80 0x7f93ac153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:31.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93abfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ac0c6b80 0x7f93ac153a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56630/0 (socket says 172.21.15.125:56630) 2023-10-30T12:42:31.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93abfff640 1 -- 172.21.15.125:0/4185228182 learned_addr learned my addr 172.21.15.125:0/4185228182 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:31.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93ab7fe640 1 --2- 172.21.15.125:0/4185228182 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93ac0c6f60 0x7f93ac13a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:31.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93b0bb4640 1 --2- 172.21.15.125:0/4185228182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93ac0c8430 0x7f93ac13ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:31.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93abfff640 1 -- 172.21.15.125:0/4185228182 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93ac0c6f60 msgr2=0x7f93ac13a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:31.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93abfff640 1 --2- 172.21.15.125:0/4185228182 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93ac0c6f60 0x7f93ac13a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:31.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93abfff640 1 -- 172.21.15.125:0/4185228182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93ac0c8430 msgr2=0x7f93ac13ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:31.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93abfff640 1 --2- 172.21.15.125:0/4185228182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93ac0c8430 0x7f93ac13ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:31.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93abfff640 1 -- 172.21.15.125:0/4185228182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939c0094d0 con 0x7f93ac0c6b80 2023-10-30T12:42:31.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93abfff640 1 --2- 172.21.15.125:0/4185228182 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ac0c6b80 0x7f93ac153a60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f939c019110 tx=0x7f939c01ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:31.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93a8ff9640 1 -- 172.21.15.125:0/4185228182 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c0b4800 con 0x7f93ac0c6b80 2023-10-30T12:42:31.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93a8ff9640 1 -- 172.21.15.125:0/4185228182 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939c0b4960 con 0x7f93ac0c6b80 2023-10-30T12:42:31.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93a8ff9640 1 -- 172.21.15.125:0/4185228182 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c0b4d10 con 0x7f93ac0c6b80 2023-10-30T12:42:31.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93b2617640 1 -- 172.21.15.125:0/4185228182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ac1f4060 con 0x7f93ac0c6b80 2023-10-30T12:42:31.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93b2617640 1 -- 172.21.15.125:0/4185228182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ac0ca5f0 con 0x7f93ac0c6b80 2023-10-30T12:42:31.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.264+0000 7f93b2617640 1 -- 172.21.15.125:0/4185228182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ac1f3ec0 con 0x7f93ac0c6b80 2023-10-30T12:42:31.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.268+0000 7f93a8ff9640 1 -- 172.21.15.125:0/4185228182 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f939c0b2410 con 0x7f93ac0c6b80 2023-10-30T12:42:31.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.268+0000 7f93a8ff9640 1 --2- 172.21.15.125:0/4185228182 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93980664a0 0x7f9398068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:31.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.268+0000 7f93a8ff9640 1 -- 172.21.15.125:0/4185228182 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f939c11e8e0 con 0x7f93ac0c6b80 2023-10-30T12:42:31.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.268+0000 7f93ab7fe640 1 --2- 172.21.15.125:0/4185228182 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93980664a0 0x7f9398068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:31.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.268+0000 7f93ab7fe640 1 --2- 172.21.15.125:0/4185228182 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93980664a0 0x7f9398068960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f93ac0c0fe0 tx=0x7f93a0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:31.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.272+0000 7f93a8ff9640 1 -- 172.21.15.125:0/4185228182 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f93ac1f3ec0 con 0x7f93ac0c6b80 2023-10-30T12:42:31.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.456+0000 7f93b2617640 1 -- 172.21.15.125:0/4185228182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93ac1f4260 con 0x7f93ac0c6b80 2023-10-30T12:42:31.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.456+0000 7f93a8ff9640 1 -- 172.21.15.125:0/4185228182 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f93ac1f4260 con 0x7f93ac0c6b80 2023-10-30T12:42:31.458 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:31.459 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:27.441795+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:31.459 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 72 2023-10-30T12:42:31.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.456+0000 7f93b2617640 1 -- 172.21.15.125:0/4185228182 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93980664a0 msgr2=0x7f9398068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:31.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.456+0000 7f93b2617640 1 --2- 172.21.15.125:0/4185228182 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93980664a0 0x7f9398068960 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f93ac0c0fe0 tx=0x7f93a0008040 comp rx=0 tx=0).stop 2023-10-30T12:42:31.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.456+0000 7f93b2617640 1 -- 172.21.15.125:0/4185228182 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ac0c6b80 msgr2=0x7f93ac153a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:31.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.456+0000 7f93b2617640 1 --2- 172.21.15.125:0/4185228182 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ac0c6b80 0x7f93ac153a60 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f939c019110 tx=0x7f939c01ad40 comp rx=0 tx=0).stop 2023-10-30T12:42:31.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.456+0000 7f93b2617640 1 -- 172.21.15.125:0/4185228182 shutdown_connections 2023-10-30T12:42:31.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.456+0000 7f93b2617640 1 --2- 172.21.15.125:0/4185228182 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93980664a0 0x7f9398068960 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:31.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.456+0000 7f93b2617640 1 --2- 172.21.15.125:0/4185228182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93ac0c8430 0x7f93ac13ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:31.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.456+0000 7f93b2617640 1 --2- 172.21.15.125:0/4185228182 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93ac0c6f60 0x7f93ac13a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:31.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.456+0000 7f93b2617640 1 --2- 172.21.15.125:0/4185228182 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ac0c6b80 0x7f93ac153a60 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:31.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.456+0000 7f93b2617640 1 -- 172.21.15.125:0/4185228182 >> 172.21.15.125:0/4185228182 conn(0x7f93ac0c7910 msgr2=0x7f93ac13ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:31.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.456+0000 7f93b2617640 1 -- 172.21.15.125:0/4185228182 shutdown_connections 2023-10-30T12:42:31.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:31.456+0000 7f93b2617640 1 -- 172.21.15.125:0/4185228182 wait complete. 2023-10-30T12:42:31.479 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:33.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:33.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70e4cd6640 1 Processor -- start 2023-10-30T12:42:33.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70e4cd6640 1 -- start start 2023-10-30T12:42:33.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70e4cd6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e01538a0 0x7f70e015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:33.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70e4cd6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e00cb800 con 0x7f70e00cf160 2023-10-30T12:42:33.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70e4cd6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70e015dc30 con 0x7f70e00ca0e0 2023-10-30T12:42:33.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70e4cd6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e00cb330 con 0x7f70e01538a0 2023-10-30T12:42:33.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70ded9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f70e00cf160 0x7f70e0164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59204/0 (socket says 172.21.15.125:59204) 2023-10-30T12:42:33.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70ded9d640 1 -- 172.21.15.125:0/1264515668 learned_addr learned my addr 172.21.15.125:0/1264515668 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:42:33.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70de59c640 1 --2- 172.21.15.125:0/1264515668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e01538a0 0x7f70e015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:33.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70ddd9b640 1 -- 172.21.15.125:0/1264515668 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194463577 0 0) 0x7f70e00cb800 con 0x7f70e00cf160 2023-10-30T12:42:33.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70ddd9b640 1 -- 172.21.15.125:0/1264515668 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70cc003530 con 0x7f70e00cf160 2023-10-30T12:42:33.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70ddd9b640 1 -- 172.21.15.125:0/1264515668 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940396166 0 0) 0x7f70c8001120 con 0x7f70e00ca0e0 2023-10-30T12:42:33.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70de59c640 1 -- 172.21.15.125:0/1264515668 >> v1:172.21.15.136:6789/0 conn(0x7f70e00ca0e0 legacy=0x7f70e015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:33.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70de59c640 1 -- 172.21.15.125:0/1264515668 >> v1:172.21.15.125:6789/0 conn(0x7f70e00cf160 legacy=0x7f70e0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:33.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.624+0000 7f70de59c640 1 -- 172.21.15.125:0/1264515668 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e00cad70 con 0x7f70e01538a0 2023-10-30T12:42:33.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.628+0000 7f70de59c640 1 --2- 172.21.15.125:0/1264515668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e01538a0 0x7f70e015d6f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f70d00097f0 tx=0x7f70d001ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1db2290c4081a33a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:33.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.628+0000 7f70ddd9b640 1 -- 172.21.15.125:0/1264515668 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d00a89b0 con 0x7f70e01538a0 2023-10-30T12:42:33.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.628+0000 7f70ddd9b640 1 -- 172.21.15.125:0/1264515668 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d00a8b10 con 0x7f70e01538a0 2023-10-30T12:42:33.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.628+0000 7f70ddd9b640 1 -- 172.21.15.125:0/1264515668 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d00a8e80 con 0x7f70e01538a0 2023-10-30T12:42:33.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.628+0000 7f70e4cd6640 1 -- 172.21.15.125:0/1264515668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e01538a0 msgr2=0x7f70e015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:33.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.628+0000 7f70e4cd6640 1 --2- 172.21.15.125:0/1264515668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e01538a0 0x7f70e015d6f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f70d00097f0 tx=0x7f70d001ecb0 comp rx=0 tx=0).stop 2023-10-30T12:42:33.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.628+0000 7f70e4cd6640 1 -- 172.21.15.125:0/1264515668 shutdown_connections 2023-10-30T12:42:33.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.628+0000 7f70e4cd6640 1 --2- 172.21.15.125:0/1264515668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e01538a0 0x7f70e015d6f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:33.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.628+0000 7f70e4cd6640 1 -- 172.21.15.125:0/1264515668 >> 172.21.15.125:0/1264515668 conn(0x7f70e006bfe0 msgr2=0x7f70e006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:33.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70e4cd6640 1 -- 172.21.15.125:0/1264515668 shutdown_connections 2023-10-30T12:42:33.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70e4cd6640 1 -- 172.21.15.125:0/1264515668 wait complete. 2023-10-30T12:42:33.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70e4cd6640 1 Processor -- start 2023-10-30T12:42:33.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70e4cd6640 1 -- start start 2023-10-30T12:42:33.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70e4cd6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e00ca0e0 0x7f70e015a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:33.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70e4cd6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e00cf160 0x7f70e01efc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:33.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70e4cd6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70e01538a0 0x7f70e00b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:33.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70e4cd6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e01ef7f0 con 0x7f70e00cf160 2023-10-30T12:42:33.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70e4cd6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e015c9f0 con 0x7f70e01538a0 2023-10-30T12:42:33.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70e4cd6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e015ccd0 con 0x7f70e00ca0e0 2023-10-30T12:42:33.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70d5d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e00cf160 0x7f70e01efc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:33.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70d5d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e00cf160 0x7f70e01efc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56644/0 (socket says 172.21.15.125:56644) 2023-10-30T12:42:33.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70d5d9b640 1 -- 172.21.15.125:0/4097743118 learned_addr learned my addr 172.21.15.125:0/4097743118 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:33.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70ded9d640 1 --2- 172.21.15.125:0/4097743118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70e01538a0 0x7f70e00b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:33.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70de59c640 1 --2- 172.21.15.125:0/4097743118 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e00ca0e0 0x7f70e015a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:33.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70d5d9b640 1 -- 172.21.15.125:0/4097743118 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e00ca0e0 msgr2=0x7f70e015a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:33.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70d5d9b640 1 --2- 172.21.15.125:0/4097743118 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e00ca0e0 0x7f70e015a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:33.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70d5d9b640 1 -- 172.21.15.125:0/4097743118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70e01538a0 msgr2=0x7f70e00b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:33.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70d5d9b640 1 --2- 172.21.15.125:0/4097743118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70e01538a0 0x7f70e00b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:33.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70d5d9b640 1 -- 172.21.15.125:0/4097743118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d0009420 con 0x7f70e00cf160 2023-10-30T12:42:33.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.632+0000 7f70d5d9b640 1 --2- 172.21.15.125:0/4097743118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e00cf160 0x7f70e01efc80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f70c800c750 tx=0x7f70c800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:33.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.636+0000 7f70d77fe640 1 -- 172.21.15.125:0/4097743118 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c8005270 con 0x7f70e00cf160 2023-10-30T12:42:33.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.636+0000 7f70d77fe640 1 -- 172.21.15.125:0/4097743118 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70c80053f0 con 0x7f70e00cf160 2023-10-30T12:42:33.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.636+0000 7f70e4cd6640 1 -- 172.21.15.125:0/4097743118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e015cfb0 con 0x7f70e00cf160 2023-10-30T12:42:33.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.636+0000 7f70d77fe640 1 -- 172.21.15.125:0/4097743118 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70c800a880 con 0x7f70e00cf160 2023-10-30T12:42:33.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.636+0000 7f70e4cd6640 1 -- 172.21.15.125:0/4097743118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e01ef130 con 0x7f70e00cf160 2023-10-30T12:42:33.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.636+0000 7f70e4cd6640 1 -- 172.21.15.125:0/4097743118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e004e170 con 0x7f70e00cf160 2023-10-30T12:42:33.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.636+0000 7f70d77fe640 1 -- 172.21.15.125:0/4097743118 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f70c800a9e0 con 0x7f70e00cf160 2023-10-30T12:42:33.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.636+0000 7f70d77fe640 1 --2- 172.21.15.125:0/4097743118 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70c00664a0 0x7f70c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:33.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.636+0000 7f70d77fe640 1 -- 172.21.15.125:0/4097743118 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f70c8083500 con 0x7f70e00cf160 2023-10-30T12:42:33.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.636+0000 7f70de59c640 1 --2- 172.21.15.125:0/4097743118 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70c00664a0 0x7f70c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:33.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.640+0000 7f70de59c640 1 --2- 172.21.15.125:0/4097743118 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70c00664a0 0x7f70c0068960 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f70d00b2000 tx=0x7f70d00aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:33.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.640+0000 7f70d77fe640 1 -- 172.21.15.125:0/4097743118 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f70c8050770 con 0x7f70e00cf160 2023-10-30T12:42:33.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.820+0000 7f70e4cd6640 1 -- 172.21.15.125:0/4097743118 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70e015dc30 con 0x7f70e00cf160 2023-10-30T12:42:33.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.820+0000 7f70d77fe640 1 -- 172.21.15.125:0/4097743118 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f70e015dc30 con 0x7f70e00cf160 2023-10-30T12:42:33.825 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:33.825 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:33.188811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:33.825 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 73 2023-10-30T12:42:33.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.824+0000 7f70e4cd6640 1 -- 172.21.15.125:0/4097743118 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70c00664a0 msgr2=0x7f70c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:33.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.824+0000 7f70e4cd6640 1 --2- 172.21.15.125:0/4097743118 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70c00664a0 0x7f70c0068960 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f70d00b2000 tx=0x7f70d00aa070 comp rx=0 tx=0).stop 2023-10-30T12:42:33.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.824+0000 7f70e4cd6640 1 -- 172.21.15.125:0/4097743118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e00cf160 msgr2=0x7f70e01efc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:33.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.824+0000 7f70e4cd6640 1 --2- 172.21.15.125:0/4097743118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e00cf160 0x7f70e01efc80 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f70c800c750 tx=0x7f70c800cc10 comp rx=0 tx=0).stop 2023-10-30T12:42:33.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.824+0000 7f70e4cd6640 1 -- 172.21.15.125:0/4097743118 shutdown_connections 2023-10-30T12:42:33.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.824+0000 7f70e4cd6640 1 --2- 172.21.15.125:0/4097743118 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70c00664a0 0x7f70c0068960 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:33.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.824+0000 7f70e4cd6640 1 --2- 172.21.15.125:0/4097743118 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70e01538a0 0x7f70e00b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:33.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.824+0000 7f70e4cd6640 1 --2- 172.21.15.125:0/4097743118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e00cf160 0x7f70e01efc80 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:33.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.824+0000 7f70e4cd6640 1 --2- 172.21.15.125:0/4097743118 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e00ca0e0 0x7f70e015a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:33.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.824+0000 7f70e4cd6640 1 -- 172.21.15.125:0/4097743118 >> 172.21.15.125:0/4097743118 conn(0x7f70e006bfe0 msgr2=0x7f70e0149380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:33.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.824+0000 7f70e4cd6640 1 -- 172.21.15.125:0/4097743118 shutdown_connections 2023-10-30T12:42:33.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:33.824+0000 7f70e4cd6640 1 -- 172.21.15.125:0/4097743118 wait complete. 2023-10-30T12:42:33.846 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:35.847 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:35.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.992+0000 7f5ad5901640 1 Processor -- start 2023-10-30T12:42:35.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.992+0000 7f5ad5901640 1 -- start start 2023-10-30T12:42:35.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.992+0000 7f5ad5901640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ad01538a0 0x7f5ad015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:35.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.992+0000 7f5ad5901640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad00cb800 con 0x7f5ad00cf160 2023-10-30T12:42:35.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.992+0000 7f5ad5901640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad015dc30 con 0x7f5ad00ca0e0 2023-10-30T12:42:35.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.992+0000 7f5ad5901640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad00cb330 con 0x7f5ad01538a0 2023-10-30T12:42:35.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.992+0000 7f5aceffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ad01538a0 0x7f5ad015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:35.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.992+0000 7f5aceffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ad01538a0 0x7f5ad015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56660/0 (socket says 172.21.15.125:56660) 2023-10-30T12:42:35.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.992+0000 7f5aceffd640 1 -- 172.21.15.125:0/3256002458 learned_addr learned my addr 172.21.15.125:0/3256002458 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:35.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.992+0000 7f5aceffd640 1 -- 172.21.15.125:0/3256002458 >> v1:172.21.15.136:6789/0 conn(0x7f5ad00ca0e0 legacy=0x7f5ad015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:35.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.992+0000 7f5aceffd640 1 -- 172.21.15.125:0/3256002458 >> v1:172.21.15.125:6789/0 conn(0x7f5ad00cf160 legacy=0x7f5ad0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:35.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.992+0000 7f5aceffd640 1 -- 172.21.15.125:0/3256002458 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad00cad70 con 0x7f5ad01538a0 2023-10-30T12:42:35.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.996+0000 7f5aceffd640 1 --2- 172.21.15.125:0/3256002458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ad01538a0 0x7f5ad015d6f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009870 tx=0x7f5ac001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425fe01a3f656032 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:35.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.996+0000 7f5acdffb640 1 -- 172.21.15.125:0/3256002458 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00a89f0 con 0x7f5ad01538a0 2023-10-30T12:42:36.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.996+0000 7f5acdffb640 1 -- 172.21.15.125:0/3256002458 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac00a8b50 con 0x7f5ad01538a0 2023-10-30T12:42:36.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.996+0000 7f5acdffb640 1 -- 172.21.15.125:0/3256002458 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00b0440 con 0x7f5ad01538a0 2023-10-30T12:42:36.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.996+0000 7f5ad5901640 1 -- 172.21.15.125:0/3256002458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ad01538a0 msgr2=0x7f5ad015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:36.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.996+0000 7f5ad5901640 1 --2- 172.21.15.125:0/3256002458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ad01538a0 0x7f5ad015d6f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0009870 tx=0x7f5ac001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:42:36.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.996+0000 7f5ad5901640 1 -- 172.21.15.125:0/3256002458 shutdown_connections 2023-10-30T12:42:36.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.996+0000 7f5ad5901640 1 --2- 172.21.15.125:0/3256002458 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ad01538a0 0x7f5ad015d6f0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:36.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.996+0000 7f5ad5901640 1 -- 172.21.15.125:0/3256002458 >> 172.21.15.125:0/3256002458 conn(0x7f5ad006bfe0 msgr2=0x7f5ad006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:36.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.996+0000 7f5ad5901640 1 -- 172.21.15.125:0/3256002458 shutdown_connections 2023-10-30T12:42:36.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:35.996+0000 7f5ad5901640 1 -- 172.21.15.125:0/3256002458 wait complete. 2023-10-30T12:42:36.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ad5901640 1 Processor -- start 2023-10-30T12:42:36.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ad5901640 1 -- start start 2023-10-30T12:42:36.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ad5901640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ad00ca0e0 0x7f5ad015b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:36.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ad5901640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ad00cf160 0x7f5ad00b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:36.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ad5901640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ad01538a0 0x7f5ad01453f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:36.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ad5901640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad01e4a70 con 0x7f5ad01538a0 2023-10-30T12:42:36.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ad5901640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad01e5920 con 0x7f5ad00ca0e0 2023-10-30T12:42:36.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ad5901640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad01e3310 con 0x7f5ad00cf160 2023-10-30T12:42:36.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5acf7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ad01538a0 0x7f5ad01453f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:36.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5acf7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ad01538a0 0x7f5ad01453f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56672/0 (socket says 172.21.15.125:56672) 2023-10-30T12:42:36.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5acf7fe640 1 -- 172.21.15.125:0/4056381080 learned_addr learned my addr 172.21.15.125:0/4056381080 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:36.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ace7fc640 1 --2- 172.21.15.125:0/4056381080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ad00cf160 0x7f5ad00b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:36.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5aceffd640 1 --2- 172.21.15.125:0/4056381080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ad00ca0e0 0x7f5ad015b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:36.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5acf7fe640 1 -- 172.21.15.125:0/4056381080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ad00cf160 msgr2=0x7f5ad00b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:36.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5acf7fe640 1 --2- 172.21.15.125:0/4056381080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ad00cf160 0x7f5ad00b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:36.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5acf7fe640 1 -- 172.21.15.125:0/4056381080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ad00ca0e0 msgr2=0x7f5ad015b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:36.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5acf7fe640 1 --2- 172.21.15.125:0/4056381080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ad00ca0e0 0x7f5ad015b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:36.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5acf7fe640 1 -- 172.21.15.125:0/4056381080 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac00094d0 con 0x7f5ad01538a0 2023-10-30T12:42:36.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5acf7fe640 1 --2- 172.21.15.125:0/4056381080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ad01538a0 0x7f5ad01453f0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800c6c0 tx=0x7f5ac800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:36.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ab7fff640 1 -- 172.21.15.125:0/4056381080 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac8003b80 con 0x7f5ad01538a0 2023-10-30T12:42:36.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ab7fff640 1 -- 172.21.15.125:0/4056381080 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac8003ce0 con 0x7f5ad01538a0 2023-10-30T12:42:36.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ad5901640 1 -- 172.21.15.125:0/4056381080 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad01e3050 con 0x7f5ad01538a0 2023-10-30T12:42:36.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ab7fff640 1 -- 172.21.15.125:0/4056381080 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac80052f0 con 0x7f5ad01538a0 2023-10-30T12:42:36.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ad5901640 1 -- 172.21.15.125:0/4056381080 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad01e43b0 con 0x7f5ad01538a0 2023-10-30T12:42:36.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.000+0000 7f5ad5901640 1 -- 172.21.15.125:0/4056381080 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad004e170 con 0x7f5ad01538a0 2023-10-30T12:42:36.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.004+0000 7f5ab7fff640 1 -- 172.21.15.125:0/4056381080 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5ac8005450 con 0x7f5ad01538a0 2023-10-30T12:42:36.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.004+0000 7f5ab7fff640 1 --2- 172.21.15.125:0/4056381080 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5abc0664a0 0x7f5abc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:36.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.004+0000 7f5ab7fff640 1 -- 172.21.15.125:0/4056381080 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f5ac8083fb0 con 0x7f5ad01538a0 2023-10-30T12:42:36.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.004+0000 7f5aceffd640 1 --2- 172.21.15.125:0/4056381080 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5abc0664a0 0x7f5abc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:36.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.004+0000 7f5aceffd640 1 --2- 172.21.15.125:0/4056381080 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5abc0664a0 0x7f5abc068960 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00b2000 tx=0x7f5ac0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:36.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.008+0000 7f5ab7fff640 1 -- 172.21.15.125:0/4056381080 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5ac8051170 con 0x7f5ad01538a0 2023-10-30T12:42:36.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.192+0000 7f5ad5901640 1 -- 172.21.15.125:0/4056381080 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ad00b1dd0 con 0x7f5ad01538a0 2023-10-30T12:42:36.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.192+0000 7f5ab7fff640 1 -- 172.21.15.125:0/4056381080 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f5ac8054790 con 0x7f5ad01538a0 2023-10-30T12:42:36.197 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:36.197 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:33.188811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:36.197 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 73 2023-10-30T12:42:36.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.196+0000 7f5ad5901640 1 -- 172.21.15.125:0/4056381080 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5abc0664a0 msgr2=0x7f5abc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:36.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.196+0000 7f5ad5901640 1 --2- 172.21.15.125:0/4056381080 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5abc0664a0 0x7f5abc068960 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00b2000 tx=0x7f5ac0008040 comp rx=0 tx=0).stop 2023-10-30T12:42:36.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.196+0000 7f5ad5901640 1 -- 172.21.15.125:0/4056381080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ad01538a0 msgr2=0x7f5ad01453f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:36.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.196+0000 7f5ad5901640 1 --2- 172.21.15.125:0/4056381080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ad01538a0 0x7f5ad01453f0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800c6c0 tx=0x7f5ac800cb80 comp rx=0 tx=0).stop 2023-10-30T12:42:36.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.196+0000 7f5ad5901640 1 -- 172.21.15.125:0/4056381080 shutdown_connections 2023-10-30T12:42:36.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.196+0000 7f5ad5901640 1 --2- 172.21.15.125:0/4056381080 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5abc0664a0 0x7f5abc068960 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:36.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.196+0000 7f5ad5901640 1 --2- 172.21.15.125:0/4056381080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ad01538a0 0x7f5ad01453f0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:36.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.196+0000 7f5ad5901640 1 --2- 172.21.15.125:0/4056381080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ad00cf160 0x7f5ad00b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:36.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.196+0000 7f5ad5901640 1 --2- 172.21.15.125:0/4056381080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ad00ca0e0 0x7f5ad015b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:36.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.196+0000 7f5ad5901640 1 -- 172.21.15.125:0/4056381080 >> 172.21.15.125:0/4056381080 conn(0x7f5ad006bfe0 msgr2=0x7f5ad01461b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:36.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.196+0000 7f5ad5901640 1 -- 172.21.15.125:0/4056381080 shutdown_connections 2023-10-30T12:42:36.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:36.196+0000 7f5ad5901640 1 -- 172.21.15.125:0/4056381080 wait complete. 2023-10-30T12:42:36.218 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:38.219 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:38.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.368+0000 7fd4767f5640 1 Processor -- start 2023-10-30T12:42:38.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.368+0000 7fd4767f5640 1 -- start start 2023-10-30T12:42:38.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.368+0000 7fd4767f5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4700c6b80 0x7fd4700bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:38.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd4767f5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd47013f4a0 0x7fd4700c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:38.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd4767f5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4700c0a50 con 0x7fd4700c6f60 2023-10-30T12:42:38.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd4767f5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd47006f6e0 con 0x7fd4700c6b80 2023-10-30T12:42:38.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd4767f5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd47006f9c0 con 0x7fd47013f4a0 2023-10-30T12:42:38.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd46ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4700c6b80 0x7fd4700bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:38.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd46ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4700c6b80 0x7fd4700bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56686/0 (socket says 172.21.15.125:56686) 2023-10-30T12:42:38.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd46ffff640 1 -- 172.21.15.125:0/714067408 learned_addr learned my addr 172.21.15.125:0/714067408 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:38.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd474d92640 1 --2- 172.21.15.125:0/714067408 >> v2:172.21.15.136:3300/0 conn(0x7fd47013f4a0 0x7fd4700c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:38.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd46ffff640 1 -- 172.21.15.125:0/714067408 >> v2:172.21.15.136:3300/0 conn(0x7fd47013f4a0 msgr2=0x7fd4700c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:38.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd46ffff640 1 --2- 172.21.15.125:0/714067408 >> v2:172.21.15.136:3300/0 conn(0x7fd47013f4a0 0x7fd4700c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:38.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd46ffff640 1 -- 172.21.15.125:0/714067408 >> v1:172.21.15.136:6789/0 conn(0x7fd4700c6f60 legacy=0x7fd47013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:38.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd46ffff640 1 -- 172.21.15.125:0/714067408 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47006fca0 con 0x7fd4700c6b80 2023-10-30T12:42:38.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd474d92640 1 --2- 172.21.15.125:0/714067408 >> v2:172.21.15.136:3300/0 conn(0x7fd47013f4a0 0x7fd4700c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:42:38.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd46ffff640 1 --2- 172.21.15.125:0/714067408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4700c6b80 0x7fd4700bfec0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd4700c8980 tx=0x7fd46001eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e530b2662bc43c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:38.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd46effd640 1 -- 172.21.15.125:0/714067408 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4600a7b70 con 0x7fd4700c6b80 2023-10-30T12:42:38.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd46effd640 1 -- 172.21.15.125:0/714067408 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4600a7cd0 con 0x7fd4700c6b80 2023-10-30T12:42:38.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.372+0000 7fd46effd640 1 -- 172.21.15.125:0/714067408 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4600af660 con 0x7fd4700c6b80 2023-10-30T12:42:38.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 -- 172.21.15.125:0/714067408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4700c6b80 msgr2=0x7fd4700bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:38.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 --2- 172.21.15.125:0/714067408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4700c6b80 0x7fd4700bfec0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7fd4700c8980 tx=0x7fd46001eb90 comp rx=0 tx=0).stop 2023-10-30T12:42:38.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 -- 172.21.15.125:0/714067408 shutdown_connections 2023-10-30T12:42:38.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 --2- 172.21.15.125:0/714067408 >> v2:172.21.15.136:3300/0 conn(0x7fd47013f4a0 0x7fd4700c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:38.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 --2- 172.21.15.125:0/714067408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4700c6b80 0x7fd4700bfec0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:38.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 -- 172.21.15.125:0/714067408 >> 172.21.15.125:0/714067408 conn(0x7fd4700c7910 msgr2=0x7fd4700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:38.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 -- 172.21.15.125:0/714067408 shutdown_connections 2023-10-30T12:42:38.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 -- 172.21.15.125:0/714067408 wait complete. 2023-10-30T12:42:38.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 Processor -- start 2023-10-30T12:42:38.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 -- start start 2023-10-30T12:42:38.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4700c6b80 0x7fd4701396f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:38.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4700c6f60 0x7fd4701f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:38.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd47013f4a0 0x7fd4700ca700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:38.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd47013acc0 con 0x7fd4700c6b80 2023-10-30T12:42:38.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4700ca270 con 0x7fd47013f4a0 2023-10-30T12:42:38.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd4767f5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4701544a0 con 0x7fd4700c6f60 2023-10-30T12:42:38.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd46ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4700c6b80 0x7fd4701396f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:38.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd46ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4700c6b80 0x7fd4701396f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56692/0 (socket says 172.21.15.125:56692) 2023-10-30T12:42:38.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd46ffff640 1 -- 172.21.15.125:0/2293390265 learned_addr learned my addr 172.21.15.125:0/2293390265 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:38.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd46f7fe640 1 --2- 172.21.15.125:0/2293390265 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4700c6f60 0x7fd4701f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:38.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.376+0000 7fd474d92640 1 --2- 172.21.15.125:0/2293390265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd47013f4a0 0x7fd4700ca700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:38.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd46ffff640 1 -- 172.21.15.125:0/2293390265 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4700c6f60 msgr2=0x7fd4701f2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:38.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd46ffff640 1 --2- 172.21.15.125:0/2293390265 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4700c6f60 0x7fd4701f2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:38.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd46ffff640 1 -- 172.21.15.125:0/2293390265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd47013f4a0 msgr2=0x7fd4700ca700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:38.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd46ffff640 1 --2- 172.21.15.125:0/2293390265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd47013f4a0 0x7fd4700ca700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:38.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd46ffff640 1 -- 172.21.15.125:0/2293390265 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46000b770 con 0x7fd4700c6b80 2023-10-30T12:42:38.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd46ffff640 1 --2- 172.21.15.125:0/2293390265 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4700c6b80 0x7fd4701396f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd46001c7c0 tx=0x7fd46001b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:38.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd46cff9640 1 -- 172.21.15.125:0/2293390265 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4600ba660 con 0x7fd4700c6b80 2023-10-30T12:42:38.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd46cff9640 1 -- 172.21.15.125:0/2293390265 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4600ba7e0 con 0x7fd4700c6b80 2023-10-30T12:42:38.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd46cff9640 1 -- 172.21.15.125:0/2293390265 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4600bab90 con 0x7fd4700c6b80 2023-10-30T12:42:38.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd4767f5640 1 -- 172.21.15.125:0/2293390265 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd470154780 con 0x7fd4700c6b80 2023-10-30T12:42:38.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd4767f5640 1 -- 172.21.15.125:0/2293390265 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4700c9c20 con 0x7fd4700c6b80 2023-10-30T12:42:38.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd4767f5640 1 -- 172.21.15.125:0/2293390265 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd47004e170 con 0x7fd4700c6b80 2023-10-30T12:42:38.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd46cff9640 1 -- 172.21.15.125:0/2293390265 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd4600c2360 con 0x7fd4700c6b80 2023-10-30T12:42:38.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd46cff9640 1 --2- 172.21.15.125:0/2293390265 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd45c0664d0 0x7fd45c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:38.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.380+0000 7fd46cff9640 1 -- 172.21.15.125:0/2293390265 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7fd4600a7e40 con 0x7fd4700c6b80 2023-10-30T12:42:38.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.384+0000 7fd46f7fe640 1 --2- 172.21.15.125:0/2293390265 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd45c0664d0 0x7fd45c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:38.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.384+0000 7fd46f7fe640 1 --2- 172.21.15.125:0/2293390265 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd45c0664d0 0x7fd45c068990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd470138b40 tx=0x7fd464009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:38.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.384+0000 7fd46cff9640 1 -- 172.21.15.125:0/2293390265 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd4600ec290 con 0x7fd4700c6b80 2023-10-30T12:42:38.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.572+0000 7fd4767f5640 1 -- 172.21.15.125:0/2293390265 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd470139270 con 0x7fd4700c6b80 2023-10-30T12:42:38.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.572+0000 7fd46cff9640 1 -- 172.21.15.125:0/2293390265 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+5737 (secure 0 0 0) 0x7fd4600bf090 con 0x7fd4700c6b80 2023-10-30T12:42:38.575 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:38.575 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:33.188811+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:38.576 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 73 2023-10-30T12:42:38.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.572+0000 7fd4767f5640 1 -- 172.21.15.125:0/2293390265 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd45c0664d0 msgr2=0x7fd45c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:38.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.572+0000 7fd4767f5640 1 --2- 172.21.15.125:0/2293390265 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd45c0664d0 0x7fd45c068990 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fd470138b40 tx=0x7fd464009040 comp rx=0 tx=0).stop 2023-10-30T12:42:38.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.572+0000 7fd4767f5640 1 -- 172.21.15.125:0/2293390265 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4700c6b80 msgr2=0x7fd4701396f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:38.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.572+0000 7fd4767f5640 1 --2- 172.21.15.125:0/2293390265 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4700c6b80 0x7fd4701396f0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fd46001c7c0 tx=0x7fd46001b930 comp rx=0 tx=0).stop 2023-10-30T12:42:38.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.572+0000 7fd4767f5640 1 -- 172.21.15.125:0/2293390265 shutdown_connections 2023-10-30T12:42:38.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.572+0000 7fd4767f5640 1 --2- 172.21.15.125:0/2293390265 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd45c0664d0 0x7fd45c068990 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:38.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.572+0000 7fd4767f5640 1 --2- 172.21.15.125:0/2293390265 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd47013f4a0 0x7fd4700ca700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:38.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.572+0000 7fd4767f5640 1 --2- 172.21.15.125:0/2293390265 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4700c6f60 0x7fd4701f2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:38.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.572+0000 7fd4767f5640 1 --2- 172.21.15.125:0/2293390265 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4700c6b80 0x7fd4701396f0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:38.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.572+0000 7fd4767f5640 1 -- 172.21.15.125:0/2293390265 >> 172.21.15.125:0/2293390265 conn(0x7fd4700c7910 msgr2=0x7fd4700c1560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:38.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.572+0000 7fd4767f5640 1 -- 172.21.15.125:0/2293390265 shutdown_connections 2023-10-30T12:42:38.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:38.572+0000 7fd4767f5640 1 -- 172.21.15.125:0/2293390265 wait complete. 2023-10-30T12:42:38.596 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:40.597 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:40.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.740+0000 7f963ee9a640 1 Processor -- start 2023-10-30T12:42:40.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.740+0000 7f963ee9a640 1 -- start start 2023-10-30T12:42:40.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.740+0000 7f963ee9a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f96380c6b80 0x7f96380bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:40.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.740+0000 7f963ee9a640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96380c0a50 con 0x7f963813f4a0 2023-10-30T12:42:40.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.740+0000 7f963ee9a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963806f6e0 con 0x7f96380c6f60 2023-10-30T12:42:40.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.740+0000 7f963ee9a640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f963806f9c0 con 0x7f96380c6b80 2023-10-30T12:42:40.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.740+0000 7f963cc36640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f96380c6b80 0x7f96380bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:40.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.744+0000 7f963cc36640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f96380c6b80 0x7f96380bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56706/0 (socket says 172.21.15.125:56706) 2023-10-30T12:42:40.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.744+0000 7f963cc36640 1 -- 172.21.15.125:0/2369857319 learned_addr learned my addr 172.21.15.125:0/2369857319 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:40.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.744+0000 7f963cc36640 1 -- 172.21.15.125:0/2369857319 >> v1:172.21.15.136:6789/0 conn(0x7f96380c6f60 legacy=0x7f963813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:40.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.744+0000 7f963cc36640 1 -- 172.21.15.125:0/2369857319 >> v1:172.21.15.125:6789/0 conn(0x7f963813f4a0 legacy=0x7f96381566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:40.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.744+0000 7f963cc36640 1 -- 172.21.15.125:0/2369857319 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963806fca0 con 0x7f96380c6b80 2023-10-30T12:42:40.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.744+0000 7f963cc36640 1 --2- 172.21.15.125:0/2369857319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96380c6b80 0x7f96380bfec0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f962c009870 tx=0x7f962c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bfdeb65c6dd04ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:40.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.744+0000 7f96377fe640 1 -- 172.21.15.125:0/2369857319 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c0ab650 con 0x7f96380c6b80 2023-10-30T12:42:40.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.744+0000 7f96377fe640 1 -- 172.21.15.125:0/2369857319 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962c0ab7b0 con 0x7f96380c6b80 2023-10-30T12:42:40.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.744+0000 7f96377fe640 1 -- 172.21.15.125:0/2369857319 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c0abb40 con 0x7f96380c6b80 2023-10-30T12:42:40.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 -- 172.21.15.125:0/2369857319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96380c6b80 msgr2=0x7f96380bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:40.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 --2- 172.21.15.125:0/2369857319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96380c6b80 0x7f96380bfec0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f962c009870 tx=0x7f962c01c540 comp rx=0 tx=0).stop 2023-10-30T12:42:40.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 -- 172.21.15.125:0/2369857319 shutdown_connections 2023-10-30T12:42:40.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 --2- 172.21.15.125:0/2369857319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96380c6b80 0x7f96380bfec0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 -- 172.21.15.125:0/2369857319 >> 172.21.15.125:0/2369857319 conn(0x7f96380c7910 msgr2=0x7f96380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:40.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 -- 172.21.15.125:0/2369857319 shutdown_connections 2023-10-30T12:42:40.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 -- 172.21.15.125:0/2369857319 wait complete. 2023-10-30T12:42:40.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 Processor -- start 2023-10-30T12:42:40.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 -- start start 2023-10-30T12:42:40.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f96380c6b80 0x7f96380c2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:40.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96380c6f60 0x7f96380c38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:40.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f963813f4a0 0x7f96380aa2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:40.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96380aa7f0 con 0x7f96380c6f60 2023-10-30T12:42:40.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96380a9e20 con 0x7f963813f4a0 2023-10-30T12:42:40.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963ee9a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96380a9560 con 0x7f96380c6b80 2023-10-30T12:42:40.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f9637fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96380c6f60 0x7f96380c38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:40.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963cc36640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f96380c6b80 0x7f96380c2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:40.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f9637fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96380c6f60 0x7f96380c38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54758/0 (socket says 172.21.15.125:54758) 2023-10-30T12:42:40.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f9637fff640 1 -- 172.21.15.125:0/3222896495 learned_addr learned my addr 172.21.15.125:0/3222896495 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:40.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.748+0000 7f963d437640 1 --2- 172.21.15.125:0/3222896495 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f963813f4a0 0x7f96380aa2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:40.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f9637fff640 1 -- 172.21.15.125:0/3222896495 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f96380c6b80 msgr2=0x7f96380c2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:40.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f9637fff640 1 --2- 172.21.15.125:0/3222896495 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f96380c6b80 0x7f96380c2210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f9637fff640 1 -- 172.21.15.125:0/3222896495 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f963813f4a0 msgr2=0x7f96380aa2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:40.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f9637fff640 1 --2- 172.21.15.125:0/3222896495 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f963813f4a0 0x7f96380aa2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f9637fff640 1 -- 172.21.15.125:0/3222896495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962c0094d0 con 0x7f96380c6f60 2023-10-30T12:42:40.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f9637fff640 1 --2- 172.21.15.125:0/3222896495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96380c6f60 0x7f96380c38a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f962400c560 tx=0x7f962400ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:40.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f96357fa640 1 -- 172.21.15.125:0/3222896495 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624005220 con 0x7f96380c6f60 2023-10-30T12:42:40.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f96357fa640 1 -- 172.21.15.125:0/3222896495 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9624005380 con 0x7f96380c6f60 2023-10-30T12:42:40.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f96357fa640 1 -- 172.21.15.125:0/3222896495 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624007270 con 0x7f96380c6f60 2023-10-30T12:42:40.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f963ee9a640 1 -- 172.21.15.125:0/3222896495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96380c51c0 con 0x7f96380c6f60 2023-10-30T12:42:40.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f963ee9a640 1 -- 172.21.15.125:0/3222896495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96380a9ab0 con 0x7f96380c6f60 2023-10-30T12:42:40.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f963ee9a640 1 -- 172.21.15.125:0/3222896495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f963804e170 con 0x7f96380c6f60 2023-10-30T12:42:40.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f96357fa640 1 -- 172.21.15.125:0/3222896495 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f96240073d0 con 0x7f96380c6f60 2023-10-30T12:42:40.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f96357fa640 1 --2- 172.21.15.125:0/3222896495 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f961c0664a0 0x7f961c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:40.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.752+0000 7f96357fa640 1 -- 172.21.15.125:0/3222896495 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f96240838f0 con 0x7f96380c6f60 2023-10-30T12:42:40.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.756+0000 7f963cc36640 1 --2- 172.21.15.125:0/3222896495 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f961c0664a0 0x7f961c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:40.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.756+0000 7f963cc36640 1 --2- 172.21.15.125:0/3222896495 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f961c0664a0 0x7f961c068960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f962c0af000 tx=0x7f962c0b8040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:40.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.756+0000 7f96357fa640 1 -- 172.21.15.125:0/3222896495 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9624050b60 con 0x7f96380c6f60 2023-10-30T12:42:40.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.936+0000 7f963ee9a640 1 -- 172.21.15.125:0/3222896495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f963806a450 con 0x7f96380c6f60 2023-10-30T12:42:40.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.936+0000 7f96357fa640 1 -- 172.21.15.125:0/3222896495 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f9624054180 con 0x7f96380c6f60 2023-10-30T12:42:40.941 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:40.941 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:38.511939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:40.941 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 74 2023-10-30T12:42:40.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.940+0000 7f963ee9a640 1 -- 172.21.15.125:0/3222896495 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f961c0664a0 msgr2=0x7f961c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:40.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.940+0000 7f963ee9a640 1 --2- 172.21.15.125:0/3222896495 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f961c0664a0 0x7f961c068960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f962c0af000 tx=0x7f962c0b8040 comp rx=0 tx=0).stop 2023-10-30T12:42:40.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.940+0000 7f963ee9a640 1 -- 172.21.15.125:0/3222896495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96380c6f60 msgr2=0x7f96380c38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:40.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.940+0000 7f963ee9a640 1 --2- 172.21.15.125:0/3222896495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96380c6f60 0x7f96380c38a0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f962400c560 tx=0x7f962400ca20 comp rx=0 tx=0).stop 2023-10-30T12:42:40.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.940+0000 7f963ee9a640 1 -- 172.21.15.125:0/3222896495 shutdown_connections 2023-10-30T12:42:40.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.940+0000 7f963ee9a640 1 --2- 172.21.15.125:0/3222896495 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f961c0664a0 0x7f961c068960 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.940+0000 7f963ee9a640 1 --2- 172.21.15.125:0/3222896495 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f963813f4a0 0x7f96380aa2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.940+0000 7f963ee9a640 1 --2- 172.21.15.125:0/3222896495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f96380c6f60 0x7f96380c38a0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.940+0000 7f963ee9a640 1 --2- 172.21.15.125:0/3222896495 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f96380c6b80 0x7f96380c2210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:40.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.940+0000 7f963ee9a640 1 -- 172.21.15.125:0/3222896495 >> 172.21.15.125:0/3222896495 conn(0x7f96380c7910 msgr2=0x7f963806a970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:40.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.940+0000 7f963ee9a640 1 -- 172.21.15.125:0/3222896495 shutdown_connections 2023-10-30T12:42:40.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:40.940+0000 7f963ee9a640 1 -- 172.21.15.125:0/3222896495 wait complete. 2023-10-30T12:42:40.962 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 74, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 74, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5345": { "addr": "172.21.15.136:6839/3922559697", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3922559697, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3922559697, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5345, "incarnation": 70, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:clientreplay", "state_seq": 22 }, "gid_5433": { "addr": "172.21.15.125:6836/2595438444", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2595438444, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2595438444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5433, "incarnation": 62, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 69 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:42:38.511939+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5433, "mds_1": 5345 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:42:42.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:43.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe82bab640 1 Processor -- start 2023-10-30T12:42:43.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe82bab640 1 -- start start 2023-10-30T12:42:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe82bab640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efe7c151800 0x7efe7c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe82bab640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7efe7c0c8260 0x7efe7c0cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe82bab640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe7c06e090 con 0x7efe7c0c1ee0 2023-10-30T12:42:43.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe82bab640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe7c06dc70 con 0x7efe7c151800 2023-10-30T12:42:43.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe82bab640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe7c0c8a50 con 0x7efe7c0c8260 2023-10-30T12:42:43.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe80947640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efe7c151800 0x7efe7c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:43.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe80947640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efe7c151800 0x7efe7c06e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54774/0 (socket says 172.21.15.125:54774) 2023-10-30T12:42:43.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe80947640 1 -- 172.21.15.125:0/1685061002 learned_addr learned my addr 172.21.15.125:0/1685061002 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:43.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe73fff640 1 --1- 172.21.15.125:0/1685061002 >> v1:172.21.15.125:6789/0 conn(0x7efe7c0c1ee0 0x7efe7c0c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52216/0 (socket says 172.21.15.125:52216) 2023-10-30T12:42:43.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe81148640 1 --2- 172.21.15.125:0/1685061002 >> v2:172.21.15.136:3300/0 conn(0x7efe7c0c8260 0x7efe7c0cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:43.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe80947640 1 -- 172.21.15.125:0/1685061002 >> v2:172.21.15.136:3300/0 conn(0x7efe7c0c8260 msgr2=0x7efe7c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:43.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe80947640 1 --2- 172.21.15.125:0/1685061002 >> v2:172.21.15.136:3300/0 conn(0x7efe7c0c8260 0x7efe7c0cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:43.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe80947640 1 -- 172.21.15.125:0/1685061002 >> v1:172.21.15.125:6789/0 conn(0x7efe7c0c1ee0 legacy=0x7efe7c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:43.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.112+0000 7efe80947640 1 -- 172.21.15.125:0/1685061002 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe7c0c3b10 con 0x7efe7c151800 2023-10-30T12:42:43.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.116+0000 7efe80947640 1 --2- 172.21.15.125:0/1685061002 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe7c151800 0x7efe7c06e470 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7efe7c0c98d0 tx=0x7efe7801c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=978ba767c50b030e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:43.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.116+0000 7efe737fe640 1 -- 172.21.15.125:0/1685061002 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe780ac640 con 0x7efe7c151800 2023-10-30T12:42:43.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.116+0000 7efe737fe640 1 -- 172.21.15.125:0/1685061002 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe780ac7a0 con 0x7efe7c151800 2023-10-30T12:42:43.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.116+0000 7efe737fe640 1 -- 172.21.15.125:0/1685061002 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe780acb30 con 0x7efe7c151800 2023-10-30T12:42:43.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.116+0000 7efe82bab640 1 -- 172.21.15.125:0/1685061002 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe7c151800 msgr2=0x7efe7c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:43.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.116+0000 7efe82bab640 1 --2- 172.21.15.125:0/1685061002 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe7c151800 0x7efe7c06e470 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7efe7c0c98d0 tx=0x7efe7801c4d0 comp rx=0 tx=0).stop 2023-10-30T12:42:43.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.116+0000 7efe82bab640 1 -- 172.21.15.125:0/1685061002 shutdown_connections 2023-10-30T12:42:43.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.116+0000 7efe82bab640 1 --2- 172.21.15.125:0/1685061002 >> v2:172.21.15.136:3300/0 conn(0x7efe7c0c8260 0x7efe7c0cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:43.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.116+0000 7efe82bab640 1 --2- 172.21.15.125:0/1685061002 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe7c151800 0x7efe7c06e470 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:43.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.116+0000 7efe82bab640 1 -- 172.21.15.125:0/1685061002 >> 172.21.15.125:0/1685061002 conn(0x7efe7c071ea0 msgr2=0x7efe7c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:43.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.116+0000 7efe82bab640 1 -- 172.21.15.125:0/1685061002 shutdown_connections 2023-10-30T12:42:43.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.116+0000 7efe82bab640 1 -- 172.21.15.125:0/1685061002 wait complete. 2023-10-30T12:42:43.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe82bab640 1 Processor -- start 2023-10-30T12:42:43.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe82bab640 1 -- start start 2023-10-30T12:42:43.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe82bab640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe7c0c1ee0 0x7efe7c1f5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:43.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe82bab640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe7c0c8260 0x7efe7c156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:43.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe82bab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe7c151800 0x7efe7c06bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:43.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe82bab640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c145d90 con 0x7efe7c151800 2023-10-30T12:42:43.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe82bab640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c06c680 con 0x7efe7c0c1ee0 2023-10-30T12:42:43.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe82bab640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c06c960 con 0x7efe7c0c8260 2023-10-30T12:42:43.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe81148640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe7c151800 0x7efe7c06bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:43.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe81148640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe7c151800 0x7efe7c06bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54790/0 (socket says 172.21.15.125:54790) 2023-10-30T12:42:43.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe81148640 1 -- 172.21.15.125:0/2244675060 learned_addr learned my addr 172.21.15.125:0/2244675060 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:43.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe73fff640 1 --2- 172.21.15.125:0/2244675060 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe7c0c8260 0x7efe7c156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:43.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe80947640 1 --2- 172.21.15.125:0/2244675060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe7c0c1ee0 0x7efe7c1f5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:43.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe81148640 1 -- 172.21.15.125:0/2244675060 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe7c0c8260 msgr2=0x7efe7c156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:43.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe81148640 1 --2- 172.21.15.125:0/2244675060 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe7c0c8260 0x7efe7c156d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:43.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe81148640 1 -- 172.21.15.125:0/2244675060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe7c0c1ee0 msgr2=0x7efe7c1f5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:43.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe81148640 1 --2- 172.21.15.125:0/2244675060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe7c0c1ee0 0x7efe7c1f5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:43.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe81148640 1 -- 172.21.15.125:0/2244675060 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe780094d0 con 0x7efe7c151800 2023-10-30T12:42:43.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe81148640 1 --2- 172.21.15.125:0/2244675060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe7c151800 0x7efe7c06bbd0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7efe74007e60 tx=0x7efe7400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:43.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe717fa640 1 -- 172.21.15.125:0/2244675060 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe74014420 con 0x7efe7c151800 2023-10-30T12:42:43.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe717fa640 1 -- 172.21.15.125:0/2244675060 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe740145a0 con 0x7efe7c151800 2023-10-30T12:42:43.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe82bab640 1 -- 172.21.15.125:0/2244675060 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7c06cc40 con 0x7efe7c151800 2023-10-30T12:42:43.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe717fa640 1 -- 172.21.15.125:0/2244675060 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe74014950 con 0x7efe7c151800 2023-10-30T12:42:43.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.120+0000 7efe82bab640 1 -- 172.21.15.125:0/2244675060 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7c1f53d0 con 0x7efe7c151800 2023-10-30T12:42:43.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.124+0000 7efe82bab640 1 -- 172.21.15.125:0/2244675060 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe7c04e170 con 0x7efe7c151800 2023-10-30T12:42:43.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.124+0000 7efe717fa640 1 -- 172.21.15.125:0/2244675060 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efe74014ab0 con 0x7efe7c151800 2023-10-30T12:42:43.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.124+0000 7efe717fa640 1 --2- 172.21.15.125:0/2244675060 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe680664a0 0x7efe68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:43.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.124+0000 7efe717fa640 1 -- 172.21.15.125:0/2244675060 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7efe740849c0 con 0x7efe7c151800 2023-10-30T12:42:43.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.124+0000 7efe80947640 1 --2- 172.21.15.125:0/2244675060 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe680664a0 0x7efe68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:43.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.124+0000 7efe80947640 1 --2- 172.21.15.125:0/2244675060 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe680664a0 0x7efe68068960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7efe780ae000 tx=0x7efe780a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:43.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.128+0000 7efe717fa640 1 -- 172.21.15.125:0/2244675060 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efe74051ba0 con 0x7efe7c151800 2023-10-30T12:42:43.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.308+0000 7efe82bab640 1 -- 172.21.15.125:0/2244675060 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe7c0c6de0 con 0x7efe7c151800 2023-10-30T12:42:43.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.308+0000 7efe717fa640 1 -- 172.21.15.125:0/2244675060 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+5743 (secure 0 0 0) 0x7efe740551c0 con 0x7efe7c151800 2023-10-30T12:42:43.309 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:43.309 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":74,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:38.511939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:43.310 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 74 2023-10-30T12:42:43.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.312+0000 7efe82bab640 1 -- 172.21.15.125:0/2244675060 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe680664a0 msgr2=0x7efe68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:43.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.312+0000 7efe82bab640 1 --2- 172.21.15.125:0/2244675060 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe680664a0 0x7efe68068960 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7efe780ae000 tx=0x7efe780a7000 comp rx=0 tx=0).stop 2023-10-30T12:42:43.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.312+0000 7efe82bab640 1 -- 172.21.15.125:0/2244675060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe7c151800 msgr2=0x7efe7c06bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:43.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.312+0000 7efe82bab640 1 --2- 172.21.15.125:0/2244675060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe7c151800 0x7efe7c06bbd0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7efe74007e60 tx=0x7efe7400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:42:43.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.312+0000 7efe82bab640 1 -- 172.21.15.125:0/2244675060 shutdown_connections 2023-10-30T12:42:43.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.312+0000 7efe82bab640 1 --2- 172.21.15.125:0/2244675060 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe680664a0 0x7efe68068960 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:43.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.312+0000 7efe82bab640 1 --2- 172.21.15.125:0/2244675060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe7c151800 0x7efe7c06bbd0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:43.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.312+0000 7efe82bab640 1 --2- 172.21.15.125:0/2244675060 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe7c0c8260 0x7efe7c156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:43.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.312+0000 7efe82bab640 1 --2- 172.21.15.125:0/2244675060 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe7c0c1ee0 0x7efe7c1f5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:43.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.312+0000 7efe82bab640 1 -- 172.21.15.125:0/2244675060 >> 172.21.15.125:0/2244675060 conn(0x7efe7c071ea0 msgr2=0x7efe7c0c8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:43.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.312+0000 7efe82bab640 1 -- 172.21.15.125:0/2244675060 shutdown_connections 2023-10-30T12:42:43.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:43.312+0000 7efe82bab640 1 -- 172.21.15.125:0/2244675060 wait complete. 2023-10-30T12:42:43.331 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:42:45.253 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:42:45.252+0000 7feb93272640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:42:45.332 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:45.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd472617640 1 Processor -- start 2023-10-30T12:42:45.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd472617640 1 -- start start 2023-10-30T12:42:45.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd472617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd46c1538a0 0x7fd46c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:45.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd472617640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd46c0ca0e0 0x7fd46c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:45.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd472617640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd46c0cf160 0x7fd46c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:45.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd472617640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd46c0cb800 con 0x7fd46c0cf160 2023-10-30T12:42:45.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd472617640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd46c15dc30 con 0x7fd46c0ca0e0 2023-10-30T12:42:45.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd472617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c0cb330 con 0x7fd46c1538a0 2023-10-30T12:42:45.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd470bb4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd46c0cf160 0x7fd46c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:45.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd470bb4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd46c0cf160 0x7fd46c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54800/0 (socket says 172.21.15.125:54800) 2023-10-30T12:42:45.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd470bb4640 1 -- 172.21.15.125:0/2215643746 learned_addr learned my addr 172.21.15.125:0/2215643746 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:45.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd46bfff640 1 --2- 172.21.15.125:0/2215643746 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd46c1538a0 0x7fd46c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:45.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd46b7fe640 1 --2- 172.21.15.125:0/2215643746 >> v2:172.21.15.136:3300/0 conn(0x7fd46c0ca0e0 0x7fd46c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:45.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd470bb4640 1 -- 172.21.15.125:0/2215643746 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd46c1538a0 msgr2=0x7fd46c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:45.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd470bb4640 1 --2- 172.21.15.125:0/2215643746 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd46c1538a0 0x7fd46c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd470bb4640 1 -- 172.21.15.125:0/2215643746 >> v2:172.21.15.136:3300/0 conn(0x7fd46c0ca0e0 msgr2=0x7fd46c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:45.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd470bb4640 1 --2- 172.21.15.125:0/2215643746 >> v2:172.21.15.136:3300/0 conn(0x7fd46c0ca0e0 0x7fd46c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.476+0000 7fd470bb4640 1 -- 172.21.15.125:0/2215643746 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd46c0cad70 con 0x7fd46c0cf160 2023-10-30T12:42:45.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.480+0000 7fd470bb4640 1 --2- 172.21.15.125:0/2215643746 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd46c0cf160 0x7fd46c0cf590 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0ccd10 tx=0x7fd46401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3480c3c1116511f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:45.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.480+0000 7fd46affd640 1 -- 172.21.15.125:0/2215643746 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4640af400 con 0x7fd46c0cf160 2023-10-30T12:42:45.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.480+0000 7fd46affd640 1 -- 172.21.15.125:0/2215643746 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4640ab050 con 0x7fd46c0cf160 2023-10-30T12:42:45.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.480+0000 7fd46affd640 1 -- 172.21.15.125:0/2215643746 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4640af780 con 0x7fd46c0cf160 2023-10-30T12:42:45.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.480+0000 7fd472617640 1 -- 172.21.15.125:0/2215643746 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd46c0cf160 msgr2=0x7fd46c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:45.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.480+0000 7fd472617640 1 --2- 172.21.15.125:0/2215643746 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd46c0cf160 0x7fd46c0cf590 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd46c0ccd10 tx=0x7fd46401f1c0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.480+0000 7fd472617640 1 -- 172.21.15.125:0/2215643746 shutdown_connections 2023-10-30T12:42:45.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.480+0000 7fd472617640 1 --2- 172.21.15.125:0/2215643746 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd46c0cf160 0x7fd46c0cf590 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.480+0000 7fd472617640 1 --2- 172.21.15.125:0/2215643746 >> v2:172.21.15.136:3300/0 conn(0x7fd46c0ca0e0 0x7fd46c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.480+0000 7fd472617640 1 --2- 172.21.15.125:0/2215643746 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd46c1538a0 0x7fd46c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.480+0000 7fd472617640 1 -- 172.21.15.125:0/2215643746 >> 172.21.15.125:0/2215643746 conn(0x7fd46c06bfe0 msgr2=0x7fd46c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:45.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.480+0000 7fd472617640 1 -- 172.21.15.125:0/2215643746 shutdown_connections 2023-10-30T12:42:45.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.480+0000 7fd472617640 1 -- 172.21.15.125:0/2215643746 wait complete. 2023-10-30T12:42:45.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd472617640 1 Processor -- start 2023-10-30T12:42:45.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd472617640 1 -- start start 2023-10-30T12:42:45.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd472617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd46c0ca0e0 0x7fd46c0b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:45.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd472617640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd46c0cf160 0x7fd46c145a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:45.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd472617640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd46c1538a0 0x7fd46c145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:45.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd472617640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c1e3bf0 con 0x7fd46c0cf160 2023-10-30T12:42:45.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd472617640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c1e58e0 con 0x7fd46c1538a0 2023-10-30T12:42:45.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd472617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46c1e4ea0 con 0x7fd46c0ca0e0 2023-10-30T12:42:45.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd46b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd46c0cf160 0x7fd46c145a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:45.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd46b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd46c0cf160 0x7fd46c145a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54802/0 (socket says 172.21.15.125:54802) 2023-10-30T12:42:45.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd46b7fe640 1 -- 172.21.15.125:0/965007285 learned_addr learned my addr 172.21.15.125:0/965007285 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:45.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd46bfff640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd46c0ca0e0 0x7fd46c0b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:45.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd470bb4640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd46c1538a0 0x7fd46c145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:45.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd46b7fe640 1 -- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd46c0ca0e0 msgr2=0x7fd46c0b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:45.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd46b7fe640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd46c0ca0e0 0x7fd46c0b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd46b7fe640 1 -- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd46c1538a0 msgr2=0x7fd46c145fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:45.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd46b7fe640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd46c1538a0 0x7fd46c145fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd46b7fe640 1 -- 172.21.15.125:0/965007285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd464009b30 con 0x7fd46c0cf160 2023-10-30T12:42:45.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd46bfff640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd46c0ca0e0 0x7fd46c0b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:45.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd46b7fe640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd46c0cf160 0x7fd46c145a60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd46000d830 tx=0x7fd46000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:45.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd468ff9640 1 -- 172.21.15.125:0/965007285 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4600096a0 con 0x7fd46c0cf160 2023-10-30T12:42:45.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd468ff9640 1 -- 172.21.15.125:0/965007285 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd460009800 con 0x7fd46c0cf160 2023-10-30T12:42:45.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd472617640 1 -- 172.21.15.125:0/965007285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46c158350 con 0x7fd46c0cf160 2023-10-30T12:42:45.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd468ff9640 1 -- 172.21.15.125:0/965007285 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd460009b70 con 0x7fd46c0cf160 2023-10-30T12:42:45.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.484+0000 7fd472617640 1 -- 172.21.15.125:0/965007285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46c1e3910 con 0x7fd46c0cf160 2023-10-30T12:42:45.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.488+0000 7fd472617640 1 -- 172.21.15.125:0/965007285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd46c0b2a50 con 0x7fd46c0cf160 2023-10-30T12:42:45.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.488+0000 7fd468ff9640 1 -- 172.21.15.125:0/965007285 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd460009cd0 con 0x7fd46c0cf160 2023-10-30T12:42:45.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.488+0000 7fd468ff9640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4580664a0 0x7fd458068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:45.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.488+0000 7fd468ff9640 1 -- 172.21.15.125:0/965007285 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7fd460084810 con 0x7fd46c0cf160 2023-10-30T12:42:45.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.488+0000 7fd46bfff640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4580664a0 0x7fd458068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:45.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.488+0000 7fd46bfff640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4580664a0 0x7fd458068960 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00b9f0 tx=0x7fd45c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:45.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.492+0000 7fd468ff9640 1 -- 172.21.15.125:0/965007285 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd460051a60 con 0x7fd46c0cf160 2023-10-30T12:42:45.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.676+0000 7fd472617640 1 -- 172.21.15.125:0/965007285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd46c04e170 con 0x7fd46c0cf160 2023-10-30T12:42:45.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.676+0000 7fd468ff9640 1 -- 172.21.15.125:0/965007285 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+5737 (secure 0 0 0) 0x7fd460055080 con 0x7fd46c0cf160 2023-10-30T12:42:45.674 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:45.674 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:44.489464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:active","state_seq":24,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:45.675 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 75 2023-10-30T12:42:45.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.676+0000 7fd472617640 1 -- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4580664a0 msgr2=0x7fd458068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:45.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.676+0000 7fd472617640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4580664a0 0x7fd458068960 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fd45c00b9f0 tx=0x7fd45c008040 comp rx=0 tx=0).stop 2023-10-30T12:42:45.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.676+0000 7fd472617640 1 -- 172.21.15.125:0/965007285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd46c0cf160 msgr2=0x7fd46c145a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:45.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.676+0000 7fd472617640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd46c0cf160 0x7fd46c145a60 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd46000d830 tx=0x7fd46000dcf0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.676+0000 7fd472617640 1 -- 172.21.15.125:0/965007285 shutdown_connections 2023-10-30T12:42:45.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.676+0000 7fd472617640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4580664a0 0x7fd458068960 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.676+0000 7fd472617640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd46c1538a0 0x7fd46c145fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.676+0000 7fd472617640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd46c0cf160 0x7fd46c145a60 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.676+0000 7fd472617640 1 --2- 172.21.15.125:0/965007285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd46c0ca0e0 0x7fd46c0b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:45.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.676+0000 7fd472617640 1 -- 172.21.15.125:0/965007285 >> 172.21.15.125:0/965007285 conn(0x7fd46c06bfe0 msgr2=0x7fd46c150ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:45.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.676+0000 7fd472617640 1 -- 172.21.15.125:0/965007285 shutdown_connections 2023-10-30T12:42:45.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:45.676+0000 7fd472617640 1 -- 172.21.15.125:0/965007285 wait complete. 2023-10-30T12:42:45.695 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=1, replacing gid=5130 2023-10-30T12:42:45.695 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.c 2023-10-30T12:42:45.964 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:42:45.968 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:42:51.696 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T12:42:52.076 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T12:42:52.077 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T12:42:52.082 INFO:tasks.ceph.mds.c:Started 2023-10-30T12:42:52.082 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:42:52.181 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2023-10-30T12:42:52.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.232+0000 7fb2e6ec9640 1 Processor -- start 2023-10-30T12:42:52.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.232+0000 7fb2e6ec9640 1 -- start start 2023-10-30T12:42:52.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.232+0000 7fb2e6ec9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2e00c14f0 0x7fb2e00c42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:52.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.232+0000 7fb2e6ec9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb2e00c18d0 0x7fb2e00c4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:52.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.232+0000 7fb2e6ec9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb2e006c100 0x7fb2e015add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:52.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.232+0000 7fb2e6ec9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e01584c0 con 0x7fb2e006c100 2023-10-30T12:42:52.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.232+0000 7fb2e6ec9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2e0157610 con 0x7fb2e00c18d0 2023-10-30T12:42:52.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.232+0000 7fb2e6ec9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e01578b0 con 0x7fb2e00c14f0 2023-10-30T12:42:52.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.236+0000 7fb2e5466640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb2e006c100 0x7fb2e015add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:52.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.236+0000 7fb2e5466640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb2e006c100 0x7fb2e015add0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38602/0 (socket says 172.21.15.125:38602) 2023-10-30T12:42:52.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.236+0000 7fb2e5466640 1 -- 172.21.15.125:0/656664572 learned_addr learned my addr 172.21.15.125:0/656664572 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:52.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.236+0000 7fb2e4c65640 1 --2- 172.21.15.125:0/656664572 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2e00c14f0 0x7fb2e00c42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:52.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.236+0000 7fb2d7fff640 1 --2- 172.21.15.125:0/656664572 >> v2:172.21.15.136:3300/0 conn(0x7fb2e00c18d0 0x7fb2e00c4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:52.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.236+0000 7fb2e5466640 1 -- 172.21.15.125:0/656664572 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2e00c14f0 msgr2=0x7fb2e00c42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:52.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.236+0000 7fb2e5466640 1 --2- 172.21.15.125:0/656664572 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2e00c14f0 0x7fb2e00c42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.236+0000 7fb2e5466640 1 -- 172.21.15.125:0/656664572 >> v2:172.21.15.136:3300/0 conn(0x7fb2e00c18d0 msgr2=0x7fb2e00c4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:42:52.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.236+0000 7fb2e5466640 1 --2- 172.21.15.125:0/656664572 >> v2:172.21.15.136:3300/0 conn(0x7fb2e00c18d0 0x7fb2e00c4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.236+0000 7fb2e5466640 1 -- 172.21.15.125:0/656664572 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e0157b90 con 0x7fb2e006c100 2023-10-30T12:42:52.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.236+0000 7fb2e5466640 1 --2- 172.21.15.125:0/656664572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2e006c100 0x7fb2e015add0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8009e50 tx=0x7fb2d801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7ad6e0f7778e7e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:52.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.236+0000 7fb2d77fe640 1 -- 172.21.15.125:0/656664572 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d80af400 con 0x7fb2e006c100 2023-10-30T12:42:52.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.236+0000 7fb2d77fe640 1 -- 172.21.15.125:0/656664572 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d80ab050 con 0x7fb2e006c100 2023-10-30T12:42:52.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.236+0000 7fb2d77fe640 1 -- 172.21.15.125:0/656664572 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d80af780 con 0x7fb2e006c100 2023-10-30T12:42:52.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/656664572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2e006c100 msgr2=0x7fb2e015add0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:52.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 --2- 172.21.15.125:0/656664572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2e006c100 0x7fb2e015add0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8009e50 tx=0x7fb2d801f1c0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/656664572 shutdown_connections 2023-10-30T12:42:52.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 --2- 172.21.15.125:0/656664572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2e006c100 0x7fb2e015add0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 --2- 172.21.15.125:0/656664572 >> v2:172.21.15.136:3300/0 conn(0x7fb2e00c18d0 0x7fb2e00c4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 --2- 172.21.15.125:0/656664572 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2e00c14f0 0x7fb2e00c42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/656664572 >> 172.21.15.125:0/656664572 conn(0x7fb2e00cace0 msgr2=0x7fb2e00cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:52.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/656664572 shutdown_connections 2023-10-30T12:42:52.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/656664572 wait complete. 2023-10-30T12:42:52.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 Processor -- start 2023-10-30T12:42:52.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 -- start start 2023-10-30T12:42:52.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2e006c100 0x7fb2e015a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:52.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2e00c14f0 0x7fb2e01596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:52.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2e00c18d0 0x7fb2e0146930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:52.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e01592b0 con 0x7fb2e00c14f0 2023-10-30T12:42:52.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e00717d0 con 0x7fb2e006c100 2023-10-30T12:42:52.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e6ec9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2e01586c0 con 0x7fb2e00c18d0 2023-10-30T12:42:52.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2e00c14f0 0x7fb2e01596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:52.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2d7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2e00c14f0 0x7fb2e01596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38614/0 (socket says 172.21.15.125:38614) 2023-10-30T12:42:52.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2d7fff640 1 -- 172.21.15.125:0/4223008689 learned_addr learned my addr 172.21.15.125:0/4223008689 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:42:52.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e4c65640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2e006c100 0x7fb2e015a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:52.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e5466640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2e00c18d0 0x7fb2e0146930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:52.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2d7fff640 1 -- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2e00c18d0 msgr2=0x7fb2e0146930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:52.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2d7fff640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2e00c18d0 0x7fb2e0146930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2d7fff640 1 -- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2e006c100 msgr2=0x7fb2e015a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:52.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2d7fff640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2e006c100 0x7fb2e015a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2d7fff640 1 -- 172.21.15.125:0/4223008689 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d8009b30 con 0x7fb2e00c14f0 2023-10-30T12:42:52.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2e5466640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2e00c18d0 0x7fb2e0146930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:42:52.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.240+0000 7fb2d7fff640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2e00c14f0 0x7fb2e01596d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000d800 tx=0x7fb2d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:52.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.244+0000 7fb2d57fa640 1 -- 172.21.15.125:0/4223008689 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d000b6c0 con 0x7fb2e00c14f0 2023-10-30T12:42:52.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.244+0000 7fb2d57fa640 1 -- 172.21.15.125:0/4223008689 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d000b820 con 0x7fb2e00c14f0 2023-10-30T12:42:52.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.244+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/4223008689 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2e0070860 con 0x7fb2e00c14f0 2023-10-30T12:42:52.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.244+0000 7fb2d57fa640 1 -- 172.21.15.125:0/4223008689 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d000bbb0 con 0x7fb2e00c14f0 2023-10-30T12:42:52.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.244+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/4223008689 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2e015a620 con 0x7fb2e00c14f0 2023-10-30T12:42:52.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.244+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/4223008689 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2e004e170 con 0x7fb2e00c14f0 2023-10-30T12:42:52.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.244+0000 7fb2d57fa640 1 -- 172.21.15.125:0/4223008689 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb2d000bd10 con 0x7fb2e00c14f0 2023-10-30T12:42:52.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.244+0000 7fb2d57fa640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2cc0664a0 0x7fb2cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:42:52.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.244+0000 7fb2d57fa640 1 -- 172.21.15.125:0/4223008689 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7fb2d0084500 con 0x7fb2e00c14f0 2023-10-30T12:42:52.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.244+0000 7fb2e4c65640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2cc0664a0 0x7fb2cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:42:52.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.248+0000 7fb2e4c65640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2cc0664a0 0x7fb2cc068960 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0045f0 tx=0x7fb2dc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:42:52.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.248+0000 7fb2d57fa640 1 -- 172.21.15.125:0/4223008689 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb2d00516c0 con 0x7fb2e00c14f0 2023-10-30T12:42:52.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.448+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/4223008689 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2e006bb30 con 0x7fb2e00c14f0 2023-10-30T12:42:52.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.448+0000 7fb2d57fa640 1 -- 172.21.15.125:0/4223008689 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fb2d0054ce0 con 0x7fb2e00c14f0 2023-10-30T12:42:52.457 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:42:52.458 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5988,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:44.489464+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:active","state_seq":24,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:42:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 76 2023-10-30T12:42:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.448+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2cc0664a0 msgr2=0x7fb2cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.448+0000 7fb2e6ec9640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2cc0664a0 0x7fb2cc068960 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0045f0 tx=0x7fb2dc008040 comp rx=0 tx=0).stop 2023-10-30T12:42:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.448+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/4223008689 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2e00c14f0 msgr2=0x7fb2e01596d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:42:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.448+0000 7fb2e6ec9640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2e00c14f0 0x7fb2e01596d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000d800 tx=0x7fb2d000dcc0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.448+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/4223008689 shutdown_connections 2023-10-30T12:42:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.448+0000 7fb2e6ec9640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2cc0664a0 0x7fb2cc068960 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.448+0000 7fb2e6ec9640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2e00c18d0 0x7fb2e0146930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.448+0000 7fb2e6ec9640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2e00c14f0 0x7fb2e01596d0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.448+0000 7fb2e6ec9640 1 --2- 172.21.15.125:0/4223008689 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2e006c100 0x7fb2e015a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:42:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.448+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/4223008689 >> 172.21.15.125:0/4223008689 conn(0x7fb2e00cace0 msgr2=0x7fb2e00cb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:42:52.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.448+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/4223008689 shutdown_connections 2023-10-30T12:42:52.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:42:52.448+0000 7fb2e6ec9640 1 -- 172.21.15.125:0/4223008689 wait complete. 2023-10-30T12:42:52.468 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T12:42:52.468 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 52 secs before thrashing 2023-10-30T12:43:16.020 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:43:16.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:43:44.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:44.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.623+0000 7f802c994640 1 Processor -- start 2023-10-30T12:43:44.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.623+0000 7f802c994640 1 -- start start 2023-10-30T12:43:44.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.627+0000 7f802c994640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f80281538a0 0x7f802815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:44.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.627+0000 7f802c994640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80280cb800 con 0x7f80280cf160 2023-10-30T12:43:44.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.627+0000 7f802c994640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802815dc30 con 0x7f80280ca0e0 2023-10-30T12:43:44.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.627+0000 7f802c994640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80280cb330 con 0x7f80281538a0 2023-10-30T12:43:44.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.627+0000 7f8026d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f80280cf160 0x7f8028164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55160/0 (socket says 172.21.15.125:55160) 2023-10-30T12:43:44.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.627+0000 7f8026d9d640 1 -- 172.21.15.125:0/2408211023 learned_addr learned my addr 172.21.15.125:0/2408211023 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:43:44.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.627+0000 7f802659c640 1 --2- 172.21.15.125:0/2408211023 >> v2:172.21.15.136:3300/0 conn(0x7f80281538a0 0x7f802815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:44.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.627+0000 7f802559a640 1 -- 172.21.15.125:0/2408211023 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686655834 0 0) 0x7f80280cb800 con 0x7f80280cf160 2023-10-30T12:43:44.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.627+0000 7f802559a640 1 -- 172.21.15.125:0/2408211023 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8010003530 con 0x7f80280cf160 2023-10-30T12:43:44.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.627+0000 7f802559a640 1 -- 172.21.15.125:0/2408211023 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1349766028 0 0) 0x7f8010003530 con 0x7f80280cf160 2023-10-30T12:43:44.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.627+0000 7f802659c640 1 -- 172.21.15.125:0/2408211023 >> v1:172.21.15.136:6789/0 conn(0x7f80280ca0e0 legacy=0x7f802815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:44.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.627+0000 7f802659c640 1 -- 172.21.15.125:0/2408211023 >> v1:172.21.15.125:6789/0 conn(0x7f80280cf160 legacy=0x7f8028164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:44.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.627+0000 7f802659c640 1 -- 172.21.15.125:0/2408211023 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80280cad70 con 0x7f80281538a0 2023-10-30T12:43:44.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.631+0000 7f802659c640 1 --2- 172.21.15.125:0/2408211023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80281538a0 0x7f802815d6f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f80180097c0 tx=0x7f801801eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57d2e9a350879472 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:44.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.631+0000 7f802559a640 1 -- 172.21.15.125:0/2408211023 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8020001a00 con 0x7f80281538a0 2023-10-30T12:43:44.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.631+0000 7f802559a640 1 -- 172.21.15.125:0/2408211023 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80180a8610 con 0x7f80281538a0 2023-10-30T12:43:44.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.631+0000 7f802559a640 1 -- 172.21.15.125:0/2408211023 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80180a8980 con 0x7f80281538a0 2023-10-30T12:43:44.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.631+0000 7f802c994640 1 -- 172.21.15.125:0/2408211023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80281538a0 msgr2=0x7f802815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:44.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.631+0000 7f802c994640 1 --2- 172.21.15.125:0/2408211023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80281538a0 0x7f802815d6f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f80180097c0 tx=0x7f801801eb90 comp rx=0 tx=0).stop 2023-10-30T12:43:44.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.631+0000 7f802c994640 1 -- 172.21.15.125:0/2408211023 shutdown_connections 2023-10-30T12:43:44.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.631+0000 7f802c994640 1 --2- 172.21.15.125:0/2408211023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80281538a0 0x7f802815d6f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:44.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.631+0000 7f802c994640 1 -- 172.21.15.125:0/2408211023 >> 172.21.15.125:0/2408211023 conn(0x7f802806bfe0 msgr2=0x7f802806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:44.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.631+0000 7f802c994640 1 -- 172.21.15.125:0/2408211023 shutdown_connections 2023-10-30T12:43:44.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.631+0000 7f802c994640 1 -- 172.21.15.125:0/2408211023 wait complete. 2023-10-30T12:43:44.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.631+0000 7f802c994640 1 Processor -- start 2023-10-30T12:43:44.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.631+0000 7f802c994640 1 -- start start 2023-10-30T12:43:44.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f802c994640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f80280ca0e0 0x7f80280b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:44.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f802c994640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80280cf160 0x7f8028148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:44.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f802c994640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f80281538a0 0x7f80280cd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:44.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f802c994640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80280ce960 con 0x7f80281538a0 2023-10-30T12:43:44.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f802c994640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8028158420 con 0x7f80280cf160 2023-10-30T12:43:44.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f802c994640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80280b41e0 con 0x7f80280ca0e0 2023-10-30T12:43:44.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f8026d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f80281538a0 0x7f80280cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:44.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f8026d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f80281538a0 0x7f80280cd630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48776/0 (socket says 172.21.15.125:48776) 2023-10-30T12:43:44.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f8026d9d640 1 -- 172.21.15.125:0/3084505774 learned_addr learned my addr 172.21.15.125:0/3084505774 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:43:44.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f802659c640 1 --2- 172.21.15.125:0/3084505774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f80280ca0e0 0x7f80280b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:44.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f8025d9b640 1 --2- 172.21.15.125:0/3084505774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80280cf160 0x7f8028148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:44.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f8026d9d640 1 -- 172.21.15.125:0/3084505774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f80280ca0e0 msgr2=0x7f80280b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:44.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f8026d9d640 1 --2- 172.21.15.125:0/3084505774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f80280ca0e0 0x7f80280b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:44.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f8026d9d640 1 -- 172.21.15.125:0/3084505774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80280cf160 msgr2=0x7f8028148ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:44.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f8026d9d640 1 --2- 172.21.15.125:0/3084505774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80280cf160 0x7f8028148ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:44.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f8026d9d640 1 -- 172.21.15.125:0/3084505774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8018009420 con 0x7f80281538a0 2023-10-30T12:43:44.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f8026d9d640 1 --2- 172.21.15.125:0/3084505774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f80281538a0 0x7f80280cd630 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f802000cbf0 tx=0x7f802000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:44.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f8016ffd640 1 -- 172.21.15.125:0/3084505774 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802000ad50 con 0x7f80281538a0 2023-10-30T12:43:44.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f8016ffd640 1 -- 172.21.15.125:0/3084505774 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8020012400 con 0x7f80281538a0 2023-10-30T12:43:44.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f802c994640 1 -- 172.21.15.125:0/3084505774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80280b37e0 con 0x7f80281538a0 2023-10-30T12:43:44.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f8016ffd640 1 -- 172.21.15.125:0/3084505774 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80200127f0 con 0x7f80281538a0 2023-10-30T12:43:44.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f802c994640 1 -- 172.21.15.125:0/3084505774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802815d890 con 0x7f80281538a0 2023-10-30T12:43:44.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.635+0000 7f802c994640 1 -- 172.21.15.125:0/3084505774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f802804e170 con 0x7f80281538a0 2023-10-30T12:43:44.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.639+0000 7f8016ffd640 1 -- 172.21.15.125:0/3084505774 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8020012950 con 0x7f80281538a0 2023-10-30T12:43:44.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.639+0000 7f8016ffd640 1 --2- 172.21.15.125:0/3084505774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80080664a0 0x7f8008068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:44.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.639+0000 7f8016ffd640 1 -- 172.21.15.125:0/3084505774 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f80200846a0 con 0x7f80281538a0 2023-10-30T12:43:44.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.639+0000 7f802659c640 1 --2- 172.21.15.125:0/3084505774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80080664a0 0x7f8008068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:44.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.639+0000 7f802659c640 1 --2- 172.21.15.125:0/3084505774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80080664a0 0x7f8008068960 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f80180b2000 tx=0x7f80180aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:44.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.643+0000 7f8016ffd640 1 -- 172.21.15.125:0/3084505774 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8020051900 con 0x7f80281538a0 2023-10-30T12:43:44.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.827+0000 7f802c994640 1 -- 172.21.15.125:0/3084505774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80280b1dd0 con 0x7f80281538a0 2023-10-30T12:43:44.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.827+0000 7f8016ffd640 1 -- 172.21.15.125:0/3084505774 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f8020054f20 con 0x7f80281538a0 2023-10-30T12:43:44.827 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:43:44.828 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5988,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:59.575448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:active","state_seq":24,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:44.828 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 77 2023-10-30T12:43:44.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.831+0000 7f802c994640 1 -- 172.21.15.125:0/3084505774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80080664a0 msgr2=0x7f8008068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:44.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.831+0000 7f802c994640 1 --2- 172.21.15.125:0/3084505774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80080664a0 0x7f8008068960 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f80180b2000 tx=0x7f80180aa070 comp rx=0 tx=0).stop 2023-10-30T12:43:44.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.831+0000 7f802c994640 1 -- 172.21.15.125:0/3084505774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f80281538a0 msgr2=0x7f80280cd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:44.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.831+0000 7f802c994640 1 --2- 172.21.15.125:0/3084505774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f80281538a0 0x7f80280cd630 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f802000cbf0 tx=0x7f802000a430 comp rx=0 tx=0).stop 2023-10-30T12:43:44.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.831+0000 7f802c994640 1 -- 172.21.15.125:0/3084505774 shutdown_connections 2023-10-30T12:43:44.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.831+0000 7f802c994640 1 --2- 172.21.15.125:0/3084505774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80080664a0 0x7f8008068960 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:44.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.831+0000 7f802c994640 1 --2- 172.21.15.125:0/3084505774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f80281538a0 0x7f80280cd630 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:44.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.831+0000 7f802c994640 1 --2- 172.21.15.125:0/3084505774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f80280cf160 0x7f8028148ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:44.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.831+0000 7f802c994640 1 --2- 172.21.15.125:0/3084505774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f80280ca0e0 0x7f80280b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:44.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.831+0000 7f802c994640 1 -- 172.21.15.125:0/3084505774 >> 172.21.15.125:0/3084505774 conn(0x7f802806bfe0 msgr2=0x7f80281516c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:44.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.831+0000 7f802c994640 1 -- 172.21.15.125:0/3084505774 shutdown_connections 2023-10-30T12:43:44.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:44.831+0000 7f802c994640 1 -- 172.21.15.125:0/3084505774 wait complete. 2023-10-30T12:43:44.848 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=1) 2023-10-30T12:43:44.848 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T12:43:44.849 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:43:44.864 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T12:43:44.864 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:45.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.003+0000 7f3a1e009640 1 Processor -- start 2023-10-30T12:43:45.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.003+0000 7f3a1e009640 1 -- start start 2023-10-30T12:43:45.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a1e009640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3a180c05d0 0x7f3a1813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:45.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a1e009640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a180c0a50 con 0x7f3a180c6b80 2023-10-30T12:43:45.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a1e009640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1806f6e0 con 0x7f3a180c6f60 2023-10-30T12:43:45.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a1e009640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a1806f9c0 con 0x7f3a180c05d0 2023-10-30T12:43:45.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a177fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3a180c6b80 0x7f3a180bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55172/0 (socket says 172.21.15.125:55172) 2023-10-30T12:43:45.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a177fe640 1 -- 172.21.15.125:0/3785636047 learned_addr learned my addr 172.21.15.125:0/3785636047 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:43:45.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a16ffd640 1 --2- 172.21.15.125:0/3785636047 >> v2:172.21.15.136:3300/0 conn(0x7f3a180c05d0 0x7f3a1813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:45.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 -- 172.21.15.125:0/3785636047 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464175248 0 0) 0x7f3a180c0a50 con 0x7f3a180c6b80 2023-10-30T12:43:45.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 -- 172.21.15.125:0/3785636047 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a00005740 con 0x7f3a180c6b80 2023-10-30T12:43:45.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 -- 172.21.15.125:0/3785636047 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3861275852 0 0) 0x7f3a00005740 con 0x7f3a180c6b80 2023-10-30T12:43:45.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 -- 172.21.15.125:0/3785636047 >> v2:172.21.15.136:3300/0 conn(0x7f3a180c05d0 msgr2=0x7f3a1813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:45.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 --2- 172.21.15.125:0/3785636047 >> v2:172.21.15.136:3300/0 conn(0x7f3a180c05d0 0x7f3a1813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 -- 172.21.15.125:0/3785636047 >> v1:172.21.15.136:6789/0 conn(0x7f3a180c6f60 legacy=0x7f3a181566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:45.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 -- 172.21.15.125:0/3785636047 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1806fca0 con 0x7f3a180c6b80 2023-10-30T12:43:45.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 -- 172.21.15.125:0/3785636047 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f3a08001680 con 0x7f3a180c6b80 2023-10-30T12:43:45.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 -- 172.21.15.125:0/3785636047 >> v1:172.21.15.125:6789/0 conn(0x7f3a180c6b80 legacy=0x7f3a180bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:45.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 --2- 172.21.15.125:0/3785636047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a00005b90 0x7f3a00005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:45.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 --2- 172.21.15.125:0/3785636047 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a000069e0 0x7f3a00006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:45.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 --2- 172.21.15.125:0/3785636047 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a000073a0 0x7f3a0000dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:45.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 -- 172.21.15.125:0/3785636047 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1806f9c0 con 0x7f3a000069e0 2023-10-30T12:43:45.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 -- 172.21.15.125:0/3785636047 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0000e240 con 0x7f3a00005b90 2023-10-30T12:43:45.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a167fc640 1 -- 172.21.15.125:0/3785636047 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a0000e400 con 0x7f3a000073a0 2023-10-30T12:43:45.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a16ffd640 1 --2- 172.21.15.125:0/3785636047 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a000069e0 0x7f3a00006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:45.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a177fe640 1 --2- 172.21.15.125:0/3785636047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a00005b90 0x7f3a00005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:45.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a17fff640 1 --2- 172.21.15.125:0/3785636047 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a000073a0 0x7f3a0000dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:45.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a16ffd640 1 -- 172.21.15.125:0/3785636047 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a000073a0 msgr2=0x7f3a0000dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:45.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a16ffd640 1 --2- 172.21.15.125:0/3785636047 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a000073a0 0x7f3a0000dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a16ffd640 1 -- 172.21.15.125:0/3785636047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a00005b90 msgr2=0x7f3a00005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:45.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a16ffd640 1 --2- 172.21.15.125:0/3785636047 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a00005b90 0x7f3a00005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a16ffd640 1 -- 172.21.15.125:0/3785636047 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a00005740 con 0x7f3a000069e0 2023-10-30T12:43:45.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a17fff640 1 --2- 172.21.15.125:0/3785636047 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a000073a0 0x7f3a0000dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:43:45.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.007+0000 7f3a17fff640 1 -- 172.21.15.125:0/3785636047 reap_dead start 2023-10-30T12:43:45.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.011+0000 7f3a16ffd640 1 --2- 172.21.15.125:0/3785636047 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a000069e0 0x7f3a00006e60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009ab0 tx=0x7f3a0c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8943aba6ce6829f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:45.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.011+0000 7f3a167fc640 1 -- 172.21.15.125:0/3785636047 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c0a5510 con 0x7f3a000069e0 2023-10-30T12:43:45.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.011+0000 7f3a167fc640 1 -- 172.21.15.125:0/3785636047 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0c0a5670 con 0x7f3a000069e0 2023-10-30T12:43:45.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.011+0000 7f3a167fc640 1 -- 172.21.15.125:0/3785636047 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c0a5a20 con 0x7f3a000069e0 2023-10-30T12:43:45.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.011+0000 7f3a1e009640 1 -- 172.21.15.125:0/3785636047 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a000069e0 msgr2=0x7f3a00006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:45.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.011+0000 7f3a1e009640 1 --2- 172.21.15.125:0/3785636047 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a000069e0 0x7f3a00006e60 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c009ab0 tx=0x7f3a0c01ca30 comp rx=0 tx=0).stop 2023-10-30T12:43:45.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.011+0000 7f3a1e009640 1 -- 172.21.15.125:0/3785636047 shutdown_connections 2023-10-30T12:43:45.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.011+0000 7f3a1e009640 1 --2- 172.21.15.125:0/3785636047 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a000069e0 0x7f3a00006e60 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.011+0000 7f3a1e009640 1 -- 172.21.15.125:0/3785636047 >> 172.21.15.125:0/3785636047 conn(0x7f3a180c7910 msgr2=0x7f3a180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:45.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.011+0000 7f3a1e009640 1 -- 172.21.15.125:0/3785636047 shutdown_connections 2023-10-30T12:43:45.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.011+0000 7f3a1e009640 1 -- 172.21.15.125:0/3785636047 wait complete. 2023-10-30T12:43:45.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.011+0000 7f3a1e009640 1 Processor -- start 2023-10-30T12:43:45.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.011+0000 7f3a1e009640 1 -- start start 2023-10-30T12:43:45.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a1e009640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a000069e0 0x7f3a181f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:45.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a1e009640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a18136500 0x7f3a1806a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:45.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a1e009640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a1813c9d0 0x7f3a1813ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:45.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a1e009640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1813dfc0 con 0x7f3a000069e0 2023-10-30T12:43:45.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a1e009640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a181f2fa0 con 0x7f3a18136500 2023-10-30T12:43:45.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a1e009640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a18157790 con 0x7f3a1813c9d0 2023-10-30T12:43:45.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a177fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a000069e0 0x7f3a181f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:45.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a177fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a000069e0 0x7f3a181f2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48788/0 (socket says 172.21.15.125:48788) 2023-10-30T12:43:45.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a177fe640 1 -- 172.21.15.125:0/3736086797 learned_addr learned my addr 172.21.15.125:0/3736086797 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:43:45.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a16ffd640 1 --2- 172.21.15.125:0/3736086797 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a18136500 0x7f3a1806a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:45.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a17fff640 1 --2- 172.21.15.125:0/3736086797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a1813c9d0 0x7f3a1813ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:45.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a177fe640 1 -- 172.21.15.125:0/3736086797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a1813c9d0 msgr2=0x7f3a1813ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:45.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a177fe640 1 --2- 172.21.15.125:0/3736086797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a1813c9d0 0x7f3a1813ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a177fe640 1 -- 172.21.15.125:0/3736086797 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a18136500 msgr2=0x7f3a1806a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:45.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a177fe640 1 --2- 172.21.15.125:0/3736086797 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a18136500 0x7f3a1806a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a177fe640 1 -- 172.21.15.125:0/3736086797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1813d390 con 0x7f3a000069e0 2023-10-30T12:43:45.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a177fe640 1 --2- 172.21.15.125:0/3736086797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a000069e0 0x7f3a181f2a60 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3a08001ca0 tx=0x7f3a08011d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:45.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f39fffff640 1 -- 172.21.15.125:0/3736086797 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0801a030 con 0x7f3a000069e0 2023-10-30T12:43:45.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f39fffff640 1 -- 172.21.15.125:0/3736086797 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0800fb60 con 0x7f3a000069e0 2023-10-30T12:43:45.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a1e009640 1 -- 172.21.15.125:0/3736086797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a18139820 con 0x7f3a000069e0 2023-10-30T12:43:45.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f39fffff640 1 -- 172.21.15.125:0/3736086797 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a08015410 con 0x7f3a000069e0 2023-10-30T12:43:45.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a1e009640 1 -- 172.21.15.125:0/3736086797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a181579e0 con 0x7f3a000069e0 2023-10-30T12:43:45.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f3a1e009640 1 -- 172.21.15.125:0/3736086797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a1804e170 con 0x7f3a000069e0 2023-10-30T12:43:45.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.015+0000 7f39fffff640 1 -- 172.21.15.125:0/3736086797 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3a0801d070 con 0x7f3a000069e0 2023-10-30T12:43:45.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.019+0000 7f39fffff640 1 --2- 172.21.15.125:0/3736086797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3a040664d0 0x7f3a04068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:45.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.019+0000 7f39fffff640 1 -- 172.21.15.125:0/3736086797 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f3a0801b070 con 0x7f3a000069e0 2023-10-30T12:43:45.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.019+0000 7f3a16ffd640 1 --2- 172.21.15.125:0/3736086797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3a040664d0 0x7f3a04068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:45.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.019+0000 7f3a16ffd640 1 --2- 172.21.15.125:0/3736086797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3a040664d0 0x7f3a04068990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0af000 tx=0x7f3a0c0046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:45.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.023+0000 7f39fffff640 1 -- 172.21.15.125:0/3736086797 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3a080541c0 con 0x7f3a000069e0 2023-10-30T12:43:45.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.207+0000 7f3a1e009640 1 -- 172.21.15.125:0/3736086797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a1813c5f0 con 0x7f3a000069e0 2023-10-30T12:43:45.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.207+0000 7f39fffff640 1 -- 172.21.15.125:0/3736086797 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f3a080577e0 con 0x7f3a000069e0 2023-10-30T12:43:45.207 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:43:45.207 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5988,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:59.575448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:active","state_seq":24,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:45.208 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 77 2023-10-30T12:43:45.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.207+0000 7f3a1e009640 1 -- 172.21.15.125:0/3736086797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3a040664d0 msgr2=0x7f3a04068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:45.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.207+0000 7f3a1e009640 1 --2- 172.21.15.125:0/3736086797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3a040664d0 0x7f3a04068990 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0af000 tx=0x7f3a0c0046d0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.207+0000 7f3a1e009640 1 -- 172.21.15.125:0/3736086797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a000069e0 msgr2=0x7f3a181f2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:45.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.207+0000 7f3a1e009640 1 --2- 172.21.15.125:0/3736086797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a000069e0 0x7f3a181f2a60 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3a08001ca0 tx=0x7f3a08011d10 comp rx=0 tx=0).stop 2023-10-30T12:43:45.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.207+0000 7f3a1e009640 1 -- 172.21.15.125:0/3736086797 shutdown_connections 2023-10-30T12:43:45.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.207+0000 7f3a1e009640 1 --2- 172.21.15.125:0/3736086797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3a040664d0 0x7f3a04068990 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.207+0000 7f3a1e009640 1 --2- 172.21.15.125:0/3736086797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a1813c9d0 0x7f3a1813ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.207+0000 7f3a1e009640 1 --2- 172.21.15.125:0/3736086797 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a18136500 0x7f3a1806a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.207+0000 7f3a1e009640 1 --2- 172.21.15.125:0/3736086797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a000069e0 0x7f3a181f2a60 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:45.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.207+0000 7f3a1e009640 1 -- 172.21.15.125:0/3736086797 >> 172.21.15.125:0/3736086797 conn(0x7f3a180c7910 msgr2=0x7f3a1806b570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:45.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.207+0000 7f3a1e009640 1 -- 172.21.15.125:0/3736086797 shutdown_connections 2023-10-30T12:43:45.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:45.207+0000 7f3a1e009640 1 -- 172.21.15.125:0/3736086797 wait complete. 2023-10-30T12:43:45.228 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:43:46.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:43:46.083 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:43:47.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:47.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.375+0000 7f0e5cad6640 1 Processor -- start 2023-10-30T12:43:47.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.375+0000 7f0e5cad6640 1 -- start start 2023-10-30T12:43:47.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.375+0000 7f0e5cad6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0e580c6b80 0x7f0e580bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:47.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.375+0000 7f0e5cad6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0e580c6f60 0x7f0e580c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:47.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.375+0000 7f0e5cad6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e580c8430 0x7f0e5813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:47.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.375+0000 7f0e5cad6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e580c0a50 con 0x7f0e580c6b80 2023-10-30T12:43:47.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.375+0000 7f0e5cad6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e5806f6e0 con 0x7f0e580c6f60 2023-10-30T12:43:47.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.375+0000 7f0e5cad6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5806f9c0 con 0x7f0e580c8430 2023-10-30T12:43:47.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.379+0000 7f0e5659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0e580c6b80 0x7f0e580bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:47.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.379+0000 7f0e5659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0e580c6b80 0x7f0e580bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48804/0 (socket says 172.21.15.125:48804) 2023-10-30T12:43:47.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.379+0000 7f0e5659c640 1 -- 172.21.15.125:0/1720747869 learned_addr learned my addr 172.21.15.125:0/1720747869 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:43:47.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.379+0000 7f0e4ffff640 1 --2- 172.21.15.125:0/1720747869 >> v2:172.21.15.136:3300/0 conn(0x7f0e580c6f60 0x7f0e580c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:47.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.379+0000 7f0e56d9d640 1 --2- 172.21.15.125:0/1720747869 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e580c8430 0x7f0e5813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:47.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.379+0000 7f0e5659c640 1 -- 172.21.15.125:0/1720747869 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e580c8430 msgr2=0x7f0e5813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:47.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.379+0000 7f0e5659c640 1 --2- 172.21.15.125:0/1720747869 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e580c8430 0x7f0e5813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.379+0000 7f0e5659c640 1 -- 172.21.15.125:0/1720747869 >> v2:172.21.15.136:3300/0 conn(0x7f0e580c6f60 msgr2=0x7f0e580c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:47.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.379+0000 7f0e5659c640 1 --2- 172.21.15.125:0/1720747869 >> v2:172.21.15.136:3300/0 conn(0x7f0e580c6f60 0x7f0e580c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.379+0000 7f0e5659c640 1 -- 172.21.15.125:0/1720747869 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5806fca0 con 0x7f0e580c6b80 2023-10-30T12:43:47.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.379+0000 7f0e5659c640 1 --2- 172.21.15.125:0/1720747869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580c6b80 0x7f0e580bfec0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0e58159b90 tx=0x7f0e4801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c6e45f725c3047 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:47.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.379+0000 7f0e55d9b640 1 -- 172.21.15.125:0/1720747869 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e480ac630 con 0x7f0e580c6b80 2023-10-30T12:43:47.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.379+0000 7f0e55d9b640 1 -- 172.21.15.125:0/1720747869 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e480ac790 con 0x7f0e580c6b80 2023-10-30T12:43:47.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.379+0000 7f0e55d9b640 1 -- 172.21.15.125:0/1720747869 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e480acb20 con 0x7f0e580c6b80 2023-10-30T12:43:47.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1720747869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580c6b80 msgr2=0x7f0e580bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:47.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 --2- 172.21.15.125:0/1720747869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580c6b80 0x7f0e580bfec0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f0e58159b90 tx=0x7f0e4801c500 comp rx=0 tx=0).stop 2023-10-30T12:43:47.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1720747869 shutdown_connections 2023-10-30T12:43:47.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 --2- 172.21.15.125:0/1720747869 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e580c8430 0x7f0e5813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 --2- 172.21.15.125:0/1720747869 >> v2:172.21.15.136:3300/0 conn(0x7f0e580c6f60 0x7f0e580c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 --2- 172.21.15.125:0/1720747869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580c6b80 0x7f0e580bfec0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1720747869 >> 172.21.15.125:0/1720747869 conn(0x7f0e580c7910 msgr2=0x7f0e580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:47.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1720747869 shutdown_connections 2023-10-30T12:43:47.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1720747869 wait complete. 2023-10-30T12:43:47.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 Processor -- start 2023-10-30T12:43:47.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 -- start start 2023-10-30T12:43:47.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e580c6b80 0x7f0e58155880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:47.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580c6f60 0x7f0e58132ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:47.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e580c8430 0x7f0e58133410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:47.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e58157810 con 0x7f0e580c6f60 2023-10-30T12:43:47.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e58132a40 con 0x7f0e580c8430 2023-10-30T12:43:47.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5cad6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e58159970 con 0x7f0e580c6b80 2023-10-30T12:43:47.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e4ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580c6f60 0x7f0e58132ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:47.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e4ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580c6f60 0x7f0e58132ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48808/0 (socket says 172.21.15.125:48808) 2023-10-30T12:43:47.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e4ffff640 1 -- 172.21.15.125:0/1466569358 learned_addr learned my addr 172.21.15.125:0/1466569358 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:43:47.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e5659c640 1 --2- 172.21.15.125:0/1466569358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e580c6b80 0x7f0e58155880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:47.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e56d9d640 1 --2- 172.21.15.125:0/1466569358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e580c8430 0x7f0e58133410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:47.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e4ffff640 1 -- 172.21.15.125:0/1466569358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e580c6b80 msgr2=0x7f0e58155880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:47.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e4ffff640 1 --2- 172.21.15.125:0/1466569358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e580c6b80 0x7f0e58155880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e4ffff640 1 -- 172.21.15.125:0/1466569358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e580c8430 msgr2=0x7f0e58133410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:47.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e4ffff640 1 --2- 172.21.15.125:0/1466569358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e580c8430 0x7f0e58133410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.383+0000 7f0e4ffff640 1 -- 172.21.15.125:0/1466569358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e480094d0 con 0x7f0e580c6f60 2023-10-30T12:43:47.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.387+0000 7f0e4ffff640 1 --2- 172.21.15.125:0/1466569358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580c6f60 0x7f0e58132ed0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0e4000d830 tx=0x7f0e4000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:47.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.387+0000 7f0e4effd640 1 -- 172.21.15.125:0/1466569358 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e400096a0 con 0x7f0e580c6f60 2023-10-30T12:43:47.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.387+0000 7f0e4effd640 1 -- 172.21.15.125:0/1466569358 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e40009800 con 0x7f0e580c6f60 2023-10-30T12:43:47.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.387+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1466569358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e58157530 con 0x7f0e580c6f60 2023-10-30T12:43:47.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.387+0000 7f0e4effd640 1 -- 172.21.15.125:0/1466569358 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e40009b70 con 0x7f0e580c6f60 2023-10-30T12:43:47.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.387+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1466569358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e58159690 con 0x7f0e580c6f60 2023-10-30T12:43:47.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.387+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1466569358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e58132bf0 con 0x7f0e580c6f60 2023-10-30T12:43:47.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.387+0000 7f0e4effd640 1 -- 172.21.15.125:0/1466569358 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e40014410 con 0x7f0e580c6f60 2023-10-30T12:43:47.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.387+0000 7f0e4effd640 1 --2- 172.21.15.125:0/1466569358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e380664a0 0x7f0e38068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:47.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.391+0000 7f0e4effd640 1 -- 172.21.15.125:0/1466569358 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f0e40084aa0 con 0x7f0e580c6f60 2023-10-30T12:43:47.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.391+0000 7f0e5659c640 1 --2- 172.21.15.125:0/1466569358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e380664a0 0x7f0e38068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:47.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.391+0000 7f0e5659c640 1 --2- 172.21.15.125:0/1466569358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e380664a0 0x7f0e38068960 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0e480ae000 tx=0x7f0e48008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:47.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.395+0000 7f0e4effd640 1 -- 172.21.15.125:0/1466569358 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0e58132bf0 con 0x7f0e580c6f60 2023-10-30T12:43:47.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.575+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1466569358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e58132bf0 con 0x7f0e580c6f60 2023-10-30T12:43:47.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.575+0000 7f0e4effd640 1 -- 172.21.15.125:0/1466569358 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f0e58132bf0 con 0x7f0e580c6f60 2023-10-30T12:43:47.577 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:43:47.577 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5988,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:59.575448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:active","state_seq":24,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:47.577 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 77 2023-10-30T12:43:47.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.579+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1466569358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e380664a0 msgr2=0x7f0e38068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:47.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.579+0000 7f0e5cad6640 1 --2- 172.21.15.125:0/1466569358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e380664a0 0x7f0e38068960 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f0e480ae000 tx=0x7f0e48008040 comp rx=0 tx=0).stop 2023-10-30T12:43:47.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.579+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1466569358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580c6f60 msgr2=0x7f0e58132ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:47.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.579+0000 7f0e5cad6640 1 --2- 172.21.15.125:0/1466569358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580c6f60 0x7f0e58132ed0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f0e4000d830 tx=0x7f0e4000dcf0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.579+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1466569358 shutdown_connections 2023-10-30T12:43:47.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.579+0000 7f0e5cad6640 1 --2- 172.21.15.125:0/1466569358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e380664a0 0x7f0e38068960 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.579+0000 7f0e5cad6640 1 --2- 172.21.15.125:0/1466569358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e580c8430 0x7f0e58133410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.579+0000 7f0e5cad6640 1 --2- 172.21.15.125:0/1466569358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e580c6f60 0x7f0e58132ed0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.579+0000 7f0e5cad6640 1 --2- 172.21.15.125:0/1466569358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e580c6b80 0x7f0e58155880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:47.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.579+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1466569358 >> 172.21.15.125:0/1466569358 conn(0x7f0e580c7910 msgr2=0x7f0e580c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:47.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.579+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1466569358 shutdown_connections 2023-10-30T12:43:47.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:47.579+0000 7f0e5cad6640 1 -- 172.21.15.125:0/1466569358 wait complete. 2023-10-30T12:43:47.597 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:43:49.598 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:49.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.747+0000 7f5b416d6640 1 Processor -- start 2023-10-30T12:43:49.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.747+0000 7f5b416d6640 1 -- start start 2023-10-30T12:43:49.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.751+0000 7f5b416d6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5b3c0c6bd0 0x7f5b3c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:49.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.751+0000 7f5b416d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b3c066cf0 con 0x7f5b3c06f7a0 2023-10-30T12:43:49.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.751+0000 7f5b416d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b3c0c70d0 con 0x7f5b3c071110 2023-10-30T12:43:49.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.751+0000 7f5b416d6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b3c0700c0 con 0x7f5b3c0c6bd0 2023-10-30T12:43:49.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.751+0000 7f5b33fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5b3c0c6bd0 0x7f5b3c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:49.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.751+0000 7f5b33fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5b3c0c6bd0 0x7f5b3c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48812/0 (socket says 172.21.15.125:48812) 2023-10-30T12:43:49.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.751+0000 7f5b33fff640 1 -- 172.21.15.125:0/995839820 learned_addr learned my addr 172.21.15.125:0/995839820 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:43:49.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.751+0000 7f5b3affd640 1 --1- 172.21.15.125:0/995839820 >> v1:172.21.15.125:6789/0 conn(0x7f5b3c06f7a0 0x7f5b3c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55176/0 (socket says 172.21.15.125:55176) 2023-10-30T12:43:49.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.751+0000 7f5b33fff640 1 -- 172.21.15.125:0/995839820 >> v1:172.21.15.136:6789/0 conn(0x7f5b3c071110 legacy=0x7f5b3c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:49.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.751+0000 7f5b33fff640 1 -- 172.21.15.125:0/995839820 >> v1:172.21.15.125:6789/0 conn(0x7f5b3c06f7a0 legacy=0x7f5b3c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:49.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.751+0000 7f5b33fff640 1 -- 172.21.15.125:0/995839820 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3c0c7390 con 0x7f5b3c0c6bd0 2023-10-30T12:43:49.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.755+0000 7f5b33fff640 1 --2- 172.21.15.125:0/995839820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b3c0c6bd0 0x7f5b3c06fb80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f5b24009870 tx=0x7f5b2401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b0f837402c7c65 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:49.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.755+0000 7f5b3a7fc640 1 -- 172.21.15.125:0/995839820 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b240a8a30 con 0x7f5b3c0c6bd0 2023-10-30T12:43:49.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.755+0000 7f5b3a7fc640 1 -- 172.21.15.125:0/995839820 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b240a8b90 con 0x7f5b3c0c6bd0 2023-10-30T12:43:49.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.755+0000 7f5b3a7fc640 1 -- 172.21.15.125:0/995839820 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b240b0440 con 0x7f5b3c0c6bd0 2023-10-30T12:43:49.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.755+0000 7f5b416d6640 1 -- 172.21.15.125:0/995839820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b3c0c6bd0 msgr2=0x7f5b3c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:49.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.755+0000 7f5b416d6640 1 --2- 172.21.15.125:0/995839820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b3c0c6bd0 0x7f5b3c06fb80 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f5b24009870 tx=0x7f5b2401ed30 comp rx=0 tx=0).stop 2023-10-30T12:43:49.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.755+0000 7f5b416d6640 1 -- 172.21.15.125:0/995839820 shutdown_connections 2023-10-30T12:43:49.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.755+0000 7f5b416d6640 1 --2- 172.21.15.125:0/995839820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b3c0c6bd0 0x7f5b3c06fb80 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.755+0000 7f5b416d6640 1 -- 172.21.15.125:0/995839820 >> 172.21.15.125:0/995839820 conn(0x7f5b3c0c7910 msgr2=0x7f5b3c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:49.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.755+0000 7f5b416d6640 1 -- 172.21.15.125:0/995839820 shutdown_connections 2023-10-30T12:43:49.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.755+0000 7f5b416d6640 1 -- 172.21.15.125:0/995839820 wait complete. 2023-10-30T12:43:49.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.755+0000 7f5b416d6640 1 Processor -- start 2023-10-30T12:43:49.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.755+0000 7f5b416d6640 1 -- start start 2023-10-30T12:43:49.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b416d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b3c06f7a0 0x7f5b3c1e4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:49.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b416d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b3c071110 0x7f5b3c1f01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:49.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b416d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b3c0c6bd0 0x7f5b3c148b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:49.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b416d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3c1564a0 con 0x7f5b3c06f7a0 2023-10-30T12:43:49.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b416d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3c155a80 con 0x7f5b3c071110 2023-10-30T12:43:49.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b416d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3c153020 con 0x7f5b3c0c6bd0 2023-10-30T12:43:49.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b3affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b3c06f7a0 0x7f5b3c1e4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:49.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b3affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b3c06f7a0 0x7f5b3c1e4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48818/0 (socket says 172.21.15.125:48818) 2023-10-30T12:43:49.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b3affd640 1 -- 172.21.15.125:0/587690309 learned_addr learned my addr 172.21.15.125:0/587690309 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:43:49.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b3b7fe640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b3c0c6bd0 0x7f5b3c148b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:49.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b33fff640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b3c071110 0x7f5b3c1f01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:49.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b3affd640 1 -- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b3c0c6bd0 msgr2=0x7f5b3c148b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:49.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b3affd640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b3c0c6bd0 0x7f5b3c148b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b3affd640 1 -- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b3c071110 msgr2=0x7f5b3c1f01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:49.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b3affd640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b3c071110 0x7f5b3c1f01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b3affd640 1 -- 172.21.15.125:0/587690309 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240094d0 con 0x7f5b3c06f7a0 2023-10-30T12:43:49.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b3b7fe640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b3c0c6bd0 0x7f5b3c148b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:43:49.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b3affd640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b3c06f7a0 0x7f5b3c1e4100 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00c460 tx=0x7f5b2c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:49.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b337fe640 1 -- 172.21.15.125:0/587690309 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c007480 con 0x7f5b3c06f7a0 2023-10-30T12:43:49.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b337fe640 1 -- 172.21.15.125:0/587690309 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b2c0075e0 con 0x7f5b3c06f7a0 2023-10-30T12:43:49.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b337fe640 1 -- 172.21.15.125:0/587690309 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c004a10 con 0x7f5b3c06f7a0 2023-10-30T12:43:49.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b416d6640 1 -- 172.21.15.125:0/587690309 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b3c152a60 con 0x7f5b3c06f7a0 2023-10-30T12:43:49.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b416d6640 1 -- 172.21.15.125:0/587690309 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b3c155430 con 0x7f5b3c06f7a0 2023-10-30T12:43:49.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b416d6640 1 -- 172.21.15.125:0/587690309 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b3c04e170 con 0x7f5b3c06f7a0 2023-10-30T12:43:49.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.759+0000 7f5b337fe640 1 -- 172.21.15.125:0/587690309 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5b2c004b70 con 0x7f5b3c06f7a0 2023-10-30T12:43:49.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.763+0000 7f5b337fe640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5b1c0664a0 0x7f5b1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:49.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.763+0000 7f5b337fe640 1 -- 172.21.15.125:0/587690309 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f5b2c0831e0 con 0x7f5b3c06f7a0 2023-10-30T12:43:49.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.763+0000 7f5b33fff640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5b1c0664a0 0x7f5b1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:49.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.763+0000 7f5b33fff640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5b1c0664a0 0x7f5b1c068960 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5b240b2000 tx=0x7f5b240aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:49.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.767+0000 7f5b337fe640 1 -- 172.21.15.125:0/587690309 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5b2c050430 con 0x7f5b3c06f7a0 2023-10-30T12:43:49.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.947+0000 7f5b416d6640 1 -- 172.21.15.125:0/587690309 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b3c0c39d0 con 0x7f5b3c06f7a0 2023-10-30T12:43:49.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.947+0000 7f5b337fe640 1 -- 172.21.15.125:0/587690309 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f5b2c053a50 con 0x7f5b3c06f7a0 2023-10-30T12:43:49.949 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:43:49.949 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5988,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:59.575448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:active","state_seq":24,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:49.949 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 77 2023-10-30T12:43:49.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.951+0000 7f5b416d6640 1 -- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5b1c0664a0 msgr2=0x7f5b1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:49.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.951+0000 7f5b416d6640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5b1c0664a0 0x7f5b1c068960 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5b240b2000 tx=0x7f5b240aa070 comp rx=0 tx=0).stop 2023-10-30T12:43:49.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.951+0000 7f5b416d6640 1 -- 172.21.15.125:0/587690309 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b3c06f7a0 msgr2=0x7f5b3c1e4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:49.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.951+0000 7f5b416d6640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b3c06f7a0 0x7f5b3c1e4100 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00c460 tx=0x7f5b2c00c920 comp rx=0 tx=0).stop 2023-10-30T12:43:49.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.951+0000 7f5b416d6640 1 -- 172.21.15.125:0/587690309 shutdown_connections 2023-10-30T12:43:49.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.951+0000 7f5b416d6640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5b1c0664a0 0x7f5b1c068960 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.951+0000 7f5b416d6640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5b3c0c6bd0 0x7f5b3c148b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.951+0000 7f5b416d6640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5b3c071110 0x7f5b3c1f01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.951+0000 7f5b416d6640 1 --2- 172.21.15.125:0/587690309 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5b3c06f7a0 0x7f5b3c1e4100 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:49.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.951+0000 7f5b416d6640 1 -- 172.21.15.125:0/587690309 >> 172.21.15.125:0/587690309 conn(0x7f5b3c0c7910 msgr2=0x7f5b3c0c3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:49.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.951+0000 7f5b416d6640 1 -- 172.21.15.125:0/587690309 shutdown_connections 2023-10-30T12:43:49.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:49.951+0000 7f5b416d6640 1 -- 172.21.15.125:0/587690309 wait complete. 2023-10-30T12:43:49.969 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:43:51.970 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:52.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51cec17640 1 Processor -- start 2023-10-30T12:43:52.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51cec17640 1 -- start start 2023-10-30T12:43:52.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51cec17640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f51c806f7a0 0x7f51c806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:52.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51cec17640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f51c80c6bd0 0x7f51c8069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:52.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51cec17640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c806a3d0 con 0x7f51c8071110 2023-10-30T12:43:52.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51cec17640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c80700c0 con 0x7f51c80c6bd0 2023-10-30T12:43:52.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51cec17640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c8066cf0 con 0x7f51c806f7a0 2023-10-30T12:43:52.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51bffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f51c80c6bd0 0x7f51c8069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:52.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51cd1b4640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f51c8071110 0x7f51c8158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34594/0 (socket says 172.21.15.125:34594) 2023-10-30T12:43:52.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51cd1b4640 1 -- 172.21.15.125:0/1571415321 learned_addr learned my addr 172.21.15.125:0/1571415321 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:43:52.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51bffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f51c80c6bd0 0x7f51c8069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51138/0 (socket says 172.21.15.125:51138) 2023-10-30T12:43:52.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51cc9b3640 1 --2- 172.21.15.125:0/1571415321 >> v2:172.21.15.136:3300/0 conn(0x7f51c806f7a0 0x7f51c806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:52.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51bffff640 1 -- 172.21.15.125:0/1571415321 >> v2:172.21.15.136:3300/0 conn(0x7f51c806f7a0 msgr2=0x7f51c806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:52.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51bffff640 1 --2- 172.21.15.125:0/1571415321 >> v2:172.21.15.136:3300/0 conn(0x7f51c806f7a0 0x7f51c806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:52.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51bffff640 1 -- 172.21.15.125:0/1571415321 >> v1:172.21.15.125:6789/0 conn(0x7f51c8071110 legacy=0x7f51c8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:52.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51bffff640 1 -- 172.21.15.125:0/1571415321 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c80c70d0 con 0x7f51c80c6bd0 2023-10-30T12:43:52.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.119+0000 7f51bf7fe640 1 -- 172.21.15.125:0/1571415321 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260652646 0 0) 0x7f51c806a3d0 con 0x7f51c8071110 2023-10-30T12:43:52.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.123+0000 7f51bffff640 1 --2- 172.21.15.125:0/1571415321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51c80c6bd0 0x7f51c8069e90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f51c806c830 tx=0x7f51b801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24525ad2a448a0e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:52.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.123+0000 7f51bf7fe640 1 -- 172.21.15.125:0/1571415321 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b80a8950 con 0x7f51c80c6bd0 2023-10-30T12:43:52.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.123+0000 7f51bf7fe640 1 -- 172.21.15.125:0/1571415321 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b80a8ab0 con 0x7f51c80c6bd0 2023-10-30T12:43:52.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.123+0000 7f51bf7fe640 1 -- 172.21.15.125:0/1571415321 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b80a8e40 con 0x7f51c80c6bd0 2023-10-30T12:43:52.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.123+0000 7f51cec17640 1 -- 172.21.15.125:0/1571415321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51c80c6bd0 msgr2=0x7f51c8069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:52.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.123+0000 7f51cec17640 1 --2- 172.21.15.125:0/1571415321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51c80c6bd0 0x7f51c8069e90 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f51c806c830 tx=0x7f51b801ec50 comp rx=0 tx=0).stop 2023-10-30T12:43:52.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.123+0000 7f51cec17640 1 -- 172.21.15.125:0/1571415321 shutdown_connections 2023-10-30T12:43:52.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.123+0000 7f51cec17640 1 --2- 172.21.15.125:0/1571415321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51c80c6bd0 0x7f51c8069e90 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:52.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.123+0000 7f51cec17640 1 --2- 172.21.15.125:0/1571415321 >> v2:172.21.15.136:3300/0 conn(0x7f51c806f7a0 0x7f51c806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:52.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.123+0000 7f51cec17640 1 -- 172.21.15.125:0/1571415321 >> 172.21.15.125:0/1571415321 conn(0x7f51c80c7910 msgr2=0x7f51c80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:52.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.123+0000 7f51cec17640 1 -- 172.21.15.125:0/1571415321 shutdown_connections 2023-10-30T12:43:52.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.123+0000 7f51cec17640 1 -- 172.21.15.125:0/1571415321 wait complete. 2023-10-30T12:43:52.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cec17640 1 Processor -- start 2023-10-30T12:43:52.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cec17640 1 -- start start 2023-10-30T12:43:52.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cec17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51c806f7a0 0x7f51c8150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:52.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cec17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51c8071110 0x7f51c81572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:52.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cec17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51c80c6bd0 0x7f51c814bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:52.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cec17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c814d3c0 con 0x7f51c806f7a0 2023-10-30T12:43:52.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cec17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c814d6a0 con 0x7f51c8071110 2023-10-30T12:43:52.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cec17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c814a420 con 0x7f51c80c6bd0 2023-10-30T12:43:52.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cc9b3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51c806f7a0 0x7f51c8150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:52.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cc9b3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51c806f7a0 0x7f51c8150060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51144/0 (socket says 172.21.15.125:51144) 2023-10-30T12:43:52.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cc9b3640 1 -- 172.21.15.125:0/1272033448 learned_addr learned my addr 172.21.15.125:0/1272033448 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:43:52.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cd1b4640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51c80c6bd0 0x7f51c814bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:52.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51bffff640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51c8071110 0x7f51c81572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:52.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cc9b3640 1 -- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51c80c6bd0 msgr2=0x7f51c814bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:52.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cc9b3640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51c80c6bd0 0x7f51c814bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:52.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cc9b3640 1 -- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51c8071110 msgr2=0x7f51c81572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:52.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cc9b3640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51c8071110 0x7f51c81572e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:52.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cc9b3640 1 -- 172.21.15.125:0/1272033448 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51b8009420 con 0x7f51c806f7a0 2023-10-30T12:43:52.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51bffff640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51c8071110 0x7f51c81572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:43:52.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cc9b3640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51c806f7a0 0x7f51c8150060 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f51c400d5f0 tx=0x7f51c400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:52.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51bd7fa640 1 -- 172.21.15.125:0/1272033448 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c40125f0 con 0x7f51c806f7a0 2023-10-30T12:43:52.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51bd7fa640 1 -- 172.21.15.125:0/1272033448 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c4012750 con 0x7f51c806f7a0 2023-10-30T12:43:52.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cec17640 1 -- 172.21.15.125:0/1272033448 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c814a700 con 0x7f51c806f7a0 2023-10-30T12:43:52.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51bd7fa640 1 -- 172.21.15.125:0/1272033448 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c4012ae0 con 0x7f51c806f7a0 2023-10-30T12:43:52.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.127+0000 7f51cec17640 1 -- 172.21.15.125:0/1272033448 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c814acc0 con 0x7f51c806f7a0 2023-10-30T12:43:52.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.131+0000 7f51cec17640 1 -- 172.21.15.125:0/1272033448 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c804e170 con 0x7f51c806f7a0 2023-10-30T12:43:52.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.131+0000 7f51bd7fa640 1 -- 172.21.15.125:0/1272033448 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51c4012c40 con 0x7f51c806f7a0 2023-10-30T12:43:52.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.131+0000 7f51bd7fa640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51b40664a0 0x7f51b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:52.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.131+0000 7f51bd7fa640 1 -- 172.21.15.125:0/1272033448 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f51c40845b0 con 0x7f51c806f7a0 2023-10-30T12:43:52.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.131+0000 7f51bffff640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51b40664a0 0x7f51b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:52.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.131+0000 7f51bffff640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51b40664a0 0x7f51b4068960 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f51b80b2000 tx=0x7f51b80aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:52.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.135+0000 7f51bd7fa640 1 -- 172.21.15.125:0/1272033448 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51c4051800 con 0x7f51c806f7a0 2023-10-30T12:43:52.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.315+0000 7f51cec17640 1 -- 172.21.15.125:0/1272033448 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51c8069fc0 con 0x7f51c806f7a0 2023-10-30T12:43:52.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.315+0000 7f51bd7fa640 1 -- 172.21.15.125:0/1272033448 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f51c4007380 con 0x7f51c806f7a0 2023-10-30T12:43:52.316 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:43:52.316 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5988,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:59.575448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:active","state_seq":24,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:52.317 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 77 2023-10-30T12:43:52.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.319+0000 7f51cec17640 1 -- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51b40664a0 msgr2=0x7f51b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:52.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.319+0000 7f51cec17640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51b40664a0 0x7f51b4068960 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f51b80b2000 tx=0x7f51b80aa070 comp rx=0 tx=0).stop 2023-10-30T12:43:52.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.319+0000 7f51cec17640 1 -- 172.21.15.125:0/1272033448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51c806f7a0 msgr2=0x7f51c8150060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:52.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.319+0000 7f51cec17640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51c806f7a0 0x7f51c8150060 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f51c400d5f0 tx=0x7f51c400dab0 comp rx=0 tx=0).stop 2023-10-30T12:43:52.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.319+0000 7f51cec17640 1 -- 172.21.15.125:0/1272033448 shutdown_connections 2023-10-30T12:43:52.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.319+0000 7f51cec17640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51b40664a0 0x7f51b4068960 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:52.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.319+0000 7f51cec17640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51c80c6bd0 0x7f51c814bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:52.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.319+0000 7f51cec17640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51c8071110 0x7f51c81572e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:52.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.319+0000 7f51cec17640 1 --2- 172.21.15.125:0/1272033448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51c806f7a0 0x7f51c8150060 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:52.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.319+0000 7f51cec17640 1 -- 172.21.15.125:0/1272033448 >> 172.21.15.125:0/1272033448 conn(0x7f51c80c7910 msgr2=0x7f51c806bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:52.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.319+0000 7f51cec17640 1 -- 172.21.15.125:0/1272033448 shutdown_connections 2023-10-30T12:43:52.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:52.319+0000 7f51cec17640 1 -- 172.21.15.125:0/1272033448 wait complete. 2023-10-30T12:43:52.336 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:43:54.337 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:54.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.483+0000 7f0219bc5640 1 Processor -- start 2023-10-30T12:43:54.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.483+0000 7f0219bc5640 1 -- start start 2023-10-30T12:43:54.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.487+0000 7f0219bc5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f02140c6b80 0x7f02140bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.487+0000 7f0219bc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02140c6f60 0x7f02140c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.487+0000 7f0219bc5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02140c0a50 con 0x7f02140c8430 2023-10-30T12:43:54.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.487+0000 7f0219bc5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f021406f6e0 con 0x7f02140c6b80 2023-10-30T12:43:54.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.487+0000 7f0219bc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021406f9c0 con 0x7f02140c6f60 2023-10-30T12:43:54.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.487+0000 7f02137fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f02140c6b80 0x7f02140bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.487+0000 7f02137fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f02140c6b80 0x7f02140bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51148/0 (socket says 172.21.15.125:51148) 2023-10-30T12:43:54.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.487+0000 7f02137fe640 1 -- 172.21.15.125:0/1447711029 learned_addr learned my addr 172.21.15.125:0/1447711029 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:43:54.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.487+0000 7f0213fff640 1 --1- 172.21.15.125:0/1447711029 >> v1:172.21.15.125:6789/0 conn(0x7f02140c8430 0x7f021413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34608/0 (socket says 172.21.15.125:34608) 2023-10-30T12:43:54.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.487+0000 7f0212ffd640 1 --2- 172.21.15.125:0/1447711029 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02140c6f60 0x7f02140c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.487+0000 7f02137fe640 1 -- 172.21.15.125:0/1447711029 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02140c6f60 msgr2=0x7f02140c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:54.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.487+0000 7f02137fe640 1 --2- 172.21.15.125:0/1447711029 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02140c6f60 0x7f02140c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.487+0000 7f02137fe640 1 -- 172.21.15.125:0/1447711029 >> v1:172.21.15.125:6789/0 conn(0x7f02140c8430 legacy=0x7f021413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:54.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.487+0000 7f02137fe640 1 -- 172.21.15.125:0/1447711029 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021406fca0 con 0x7f02140c6b80 2023-10-30T12:43:54.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.491+0000 7f02137fe640 1 --2- 172.21.15.125:0/1447711029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02140c6b80 0x7f02140bfec0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f020400bb10 tx=0x7f020401eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93a69123ea6dc367 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:54.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.491+0000 7f02127fc640 1 -- 172.21.15.125:0/1447711029 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040a7b60 con 0x7f02140c6b80 2023-10-30T12:43:54.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.491+0000 7f02127fc640 1 -- 172.21.15.125:0/1447711029 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02040a7cc0 con 0x7f02140c6b80 2023-10-30T12:43:54.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.491+0000 7f02127fc640 1 -- 172.21.15.125:0/1447711029 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040af660 con 0x7f02140c6b80 2023-10-30T12:43:54.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.491+0000 7f0219bc5640 1 -- 172.21.15.125:0/1447711029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02140c6b80 msgr2=0x7f02140bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:54.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.491+0000 7f0219bc5640 1 --2- 172.21.15.125:0/1447711029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02140c6b80 0x7f02140bfec0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f020400bb10 tx=0x7f020401eb80 comp rx=0 tx=0).stop 2023-10-30T12:43:54.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.491+0000 7f0219bc5640 1 -- 172.21.15.125:0/1447711029 shutdown_connections 2023-10-30T12:43:54.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.491+0000 7f0219bc5640 1 --2- 172.21.15.125:0/1447711029 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02140c6f60 0x7f02140c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.491+0000 7f0219bc5640 1 --2- 172.21.15.125:0/1447711029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02140c6b80 0x7f02140bfec0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.491+0000 7f0219bc5640 1 -- 172.21.15.125:0/1447711029 >> 172.21.15.125:0/1447711029 conn(0x7f02140c7910 msgr2=0x7f02140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:54.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.491+0000 7f0219bc5640 1 -- 172.21.15.125:0/1447711029 shutdown_connections 2023-10-30T12:43:54.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.491+0000 7f0219bc5640 1 -- 172.21.15.125:0/1447711029 wait complete. 2023-10-30T12:43:54.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.491+0000 7f0219bc5640 1 Processor -- start 2023-10-30T12:43:54.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.491+0000 7f0219bc5640 1 -- start start 2023-10-30T12:43:54.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f0219bc5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02140c6b80 0x7f021413a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f0219bc5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02140c6f60 0x7f02141f2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f0219bc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02140c8430 0x7f02141377b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f0219bc5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02141f6440 con 0x7f02140c6b80 2023-10-30T12:43:54.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f0219bc5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0214155dc0 con 0x7f02140c6f60 2023-10-30T12:43:54.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f0219bc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02141f6160 con 0x7f02140c8430 2023-10-30T12:43:54.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f02137fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02140c6b80 0x7f021413a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f02137fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02140c6b80 0x7f021413a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51152/0 (socket says 172.21.15.125:51152) 2023-10-30T12:43:54.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f02137fe640 1 -- 172.21.15.125:0/2054701339 learned_addr learned my addr 172.21.15.125:0/2054701339 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:43:54.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f0213fff640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02140c8430 0x7f02141377b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f0212ffd640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02140c6f60 0x7f02141f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f02137fe640 1 -- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02140c8430 msgr2=0x7f02141377b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:54.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f02137fe640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02140c8430 0x7f02141377b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f02137fe640 1 -- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02140c6f60 msgr2=0x7f02141f2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:54.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f02137fe640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02140c6f60 0x7f02141f2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f02137fe640 1 -- 172.21.15.125:0/2054701339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020400b770 con 0x7f02140c6b80 2023-10-30T12:43:54.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f0212ffd640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02140c6f60 0x7f02141f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:43:54.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f02137fe640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02140c6b80 0x7f021413a7d0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f020400b2c0 tx=0x7f020401d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:54.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f01fbfff640 1 -- 172.21.15.125:0/2054701339 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040ba680 con 0x7f02140c6b80 2023-10-30T12:43:54.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f01fbfff640 1 -- 172.21.15.125:0/2054701339 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02040ba800 con 0x7f02140c6b80 2023-10-30T12:43:54.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f01fbfff640 1 -- 172.21.15.125:0/2054701339 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040babb0 con 0x7f02140c6b80 2023-10-30T12:43:54.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f0219bc5640 1 -- 172.21.15.125:0/2054701339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02140ca820 con 0x7f02140c6b80 2023-10-30T12:43:54.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f0219bc5640 1 -- 172.21.15.125:0/2054701339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02141f2270 con 0x7f02140c6b80 2023-10-30T12:43:54.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.495+0000 7f0219bc5640 1 -- 172.21.15.125:0/2054701339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02141f2460 con 0x7f02140c6b80 2023-10-30T12:43:54.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.499+0000 7f01fbfff640 1 -- 172.21.15.125:0/2054701339 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f02040bad10 con 0x7f02140c6b80 2023-10-30T12:43:54.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.499+0000 7f01fbfff640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f02000664a0 0x7f0200068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:54.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.499+0000 7f01fbfff640 1 -- 172.21.15.125:0/2054701339 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f020411fa00 con 0x7f02140c6b80 2023-10-30T12:43:54.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.499+0000 7f0212ffd640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f02000664a0 0x7f0200068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:54.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.499+0000 7f0212ffd640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f02000664a0 0x7f0200068960 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f021413bd60 tx=0x7f0208008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:54.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.503+0000 7f01fbfff640 1 -- 172.21.15.125:0/2054701339 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f02040ecbd0 con 0x7f02140c6b80 2023-10-30T12:43:54.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.683+0000 7f0219bc5640 1 -- 172.21.15.125:0/2054701339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f021404e170 con 0x7f02140c6b80 2023-10-30T12:43:54.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.683+0000 7f01fbfff640 1 -- 172.21.15.125:0/2054701339 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f02040b4030 con 0x7f02140c6b80 2023-10-30T12:43:54.683 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:43:54.683 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5988,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:59.575448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:active","state_seq":24,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:54.684 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 77 2023-10-30T12:43:54.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.683+0000 7f0219bc5640 1 -- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f02000664a0 msgr2=0x7f0200068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:54.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.683+0000 7f0219bc5640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f02000664a0 0x7f0200068960 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f021413bd60 tx=0x7f0208008040 comp rx=0 tx=0).stop 2023-10-30T12:43:54.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.683+0000 7f0219bc5640 1 -- 172.21.15.125:0/2054701339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02140c6b80 msgr2=0x7f021413a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:54.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.683+0000 7f0219bc5640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02140c6b80 0x7f021413a7d0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f020400b2c0 tx=0x7f020401d3c0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.683+0000 7f0219bc5640 1 -- 172.21.15.125:0/2054701339 shutdown_connections 2023-10-30T12:43:54.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.683+0000 7f0219bc5640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f02000664a0 0x7f0200068960 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.683+0000 7f0219bc5640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f02140c8430 0x7f02141377b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.683+0000 7f0219bc5640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f02140c6f60 0x7f02141f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.683+0000 7f0219bc5640 1 --2- 172.21.15.125:0/2054701339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f02140c6b80 0x7f021413a7d0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:54.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.683+0000 7f0219bc5640 1 -- 172.21.15.125:0/2054701339 >> 172.21.15.125:0/2054701339 conn(0x7f02140c7910 msgr2=0x7f02140c1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:54.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.687+0000 7f0219bc5640 1 -- 172.21.15.125:0/2054701339 shutdown_connections 2023-10-30T12:43:54.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:54.687+0000 7f0219bc5640 1 -- 172.21.15.125:0/2054701339 wait complete. 2023-10-30T12:43:54.704 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:43:56.705 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:56.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.851+0000 7f6f9b209640 1 Processor -- start 2023-10-30T12:43:56.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.851+0000 7f6f9b209640 1 -- start start 2023-10-30T12:43:56.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.851+0000 7f6f9b209640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f940c6b80 0x7f6f940bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:56.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.851+0000 7f6f9b209640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6f9413f4a0 0x7f6f940c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:56.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.851+0000 7f6f9b209640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f940c0a50 con 0x7f6f940c6f60 2023-10-30T12:43:56.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.855+0000 7f6f9b209640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f9406f6e0 con 0x7f6f9413f4a0 2023-10-30T12:43:56.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.855+0000 7f6f9b209640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9406f9c0 con 0x7f6f940c6b80 2023-10-30T12:43:56.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.855+0000 7f6f997a6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6f9413f4a0 0x7f6f940c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:56.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.855+0000 7f6f997a6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6f9413f4a0 0x7f6f940c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51166/0 (socket says 172.21.15.125:51166) 2023-10-30T12:43:56.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.855+0000 7f6f997a6640 1 -- 172.21.15.125:0/2826630708 learned_addr learned my addr 172.21.15.125:0/2826630708 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:43:56.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.855+0000 7f6f98fa5640 1 --2- 172.21.15.125:0/2826630708 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f940c6b80 0x7f6f940bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:56.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.855+0000 7f6f997a6640 1 -- 172.21.15.125:0/2826630708 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f940c6b80 msgr2=0x7f6f940bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:56.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.855+0000 7f6f997a6640 1 --2- 172.21.15.125:0/2826630708 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f940c6b80 0x7f6f940bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.855+0000 7f6f997a6640 1 -- 172.21.15.125:0/2826630708 >> v1:172.21.15.136:6789/0 conn(0x7f6f940c6f60 legacy=0x7f6f9413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:56.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.855+0000 7f6f997a6640 1 -- 172.21.15.125:0/2826630708 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9406fca0 con 0x7f6f9413f4a0 2023-10-30T12:43:56.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.855+0000 7f6f997a6640 1 --2- 172.21.15.125:0/2826630708 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9413f4a0 0x7f6f940c0400 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c009e50 tx=0x7f6f8c01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484988a1862d73ee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:56.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.855+0000 7f6f8b7fe640 1 -- 172.21.15.125:0/2826630708 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c0a57f0 con 0x7f6f9413f4a0 2023-10-30T12:43:56.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.855+0000 7f6f8b7fe640 1 -- 172.21.15.125:0/2826630708 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c0a5950 con 0x7f6f9413f4a0 2023-10-30T12:43:56.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.855+0000 7f6f8b7fe640 1 -- 172.21.15.125:0/2826630708 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c0a5ce0 con 0x7f6f9413f4a0 2023-10-30T12:43:56.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 -- 172.21.15.125:0/2826630708 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9413f4a0 msgr2=0x7f6f940c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:56.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 --2- 172.21.15.125:0/2826630708 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9413f4a0 0x7f6f940c0400 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c009e50 tx=0x7f6f8c01ca10 comp rx=0 tx=0).stop 2023-10-30T12:43:56.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 -- 172.21.15.125:0/2826630708 shutdown_connections 2023-10-30T12:43:56.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 --2- 172.21.15.125:0/2826630708 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9413f4a0 0x7f6f940c0400 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 --2- 172.21.15.125:0/2826630708 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f940c6b80 0x7f6f940bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 -- 172.21.15.125:0/2826630708 >> 172.21.15.125:0/2826630708 conn(0x7f6f940c7910 msgr2=0x7f6f940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:56.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 -- 172.21.15.125:0/2826630708 shutdown_connections 2023-10-30T12:43:56.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 -- 172.21.15.125:0/2826630708 wait complete. 2023-10-30T12:43:56.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 Processor -- start 2023-10-30T12:43:56.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 -- start start 2023-10-30T12:43:56.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f940c6b80 0x7f6f940c9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:56.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f940c6f60 0x7f6f940c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:56.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9413f4a0 0x7f6f941559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:56.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f940c8cf0 con 0x7f6f9413f4a0 2023-10-30T12:43:56.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9413dff0 con 0x7f6f940c6b80 2023-10-30T12:43:56.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f9b209640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f94151f60 con 0x7f6f940c6f60 2023-10-30T12:43:56.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f997a6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9413f4a0 0x7f6f941559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:56.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f997a6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9413f4a0 0x7f6f941559b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51180/0 (socket says 172.21.15.125:51180) 2023-10-30T12:43:56.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f997a6640 1 -- 172.21.15.125:0/1241028490 learned_addr learned my addr 172.21.15.125:0/1241028490 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:43:56.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f98fa5640 1 --2- 172.21.15.125:0/1241028490 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f940c6b80 0x7f6f940c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:56.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.859+0000 7f6f8bfff640 1 --2- 172.21.15.125:0/1241028490 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f940c6f60 0x7f6f940c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:56.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.863+0000 7f6f997a6640 1 -- 172.21.15.125:0/1241028490 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f940c6f60 msgr2=0x7f6f940c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:56.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.863+0000 7f6f997a6640 1 --2- 172.21.15.125:0/1241028490 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f940c6f60 0x7f6f940c9650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.863+0000 7f6f997a6640 1 -- 172.21.15.125:0/1241028490 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f940c6b80 msgr2=0x7f6f940c9110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:56.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.863+0000 7f6f997a6640 1 --2- 172.21.15.125:0/1241028490 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f940c6b80 0x7f6f940c9110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:56.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.863+0000 7f6f997a6640 1 -- 172.21.15.125:0/1241028490 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c009b30 con 0x7f6f9413f4a0 2023-10-30T12:43:56.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.863+0000 7f6f997a6640 1 --2- 172.21.15.125:0/1241028490 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9413f4a0 0x7f6f941559b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c01a640 tx=0x7f6f8c0197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:56.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.863+0000 7f6f897fa640 1 -- 172.21.15.125:0/1241028490 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c0b8880 con 0x7f6f9413f4a0 2023-10-30T12:43:56.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.863+0000 7f6f897fa640 1 -- 172.21.15.125:0/1241028490 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c0b89e0 con 0x7f6f9413f4a0 2023-10-30T12:43:56.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.863+0000 7f6f9b209640 1 -- 172.21.15.125:0/1241028490 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f94152c60 con 0x7f6f9413f4a0 2023-10-30T12:43:56.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.863+0000 7f6f897fa640 1 -- 172.21.15.125:0/1241028490 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c0b8d70 con 0x7f6f9413f4a0 2023-10-30T12:43:56.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.863+0000 7f6f9b209640 1 -- 172.21.15.125:0/1241028490 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f941521b0 con 0x7f6f9413f4a0 2023-10-30T12:43:56.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.863+0000 7f6f9b209640 1 -- 172.21.15.125:0/1241028490 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f941540d0 con 0x7f6f9413f4a0 2023-10-30T12:43:56.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.863+0000 7f6f897fa640 1 -- 172.21.15.125:0/1241028490 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6f8c0b6440 con 0x7f6f9413f4a0 2023-10-30T12:43:56.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.867+0000 7f6f897fa640 1 --2- 172.21.15.125:0/1241028490 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f800664a0 0x7f6f80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:56.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.867+0000 7f6f897fa640 1 -- 172.21.15.125:0/1241028490 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f6f8c11f8c0 con 0x7f6f9413f4a0 2023-10-30T12:43:56.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.867+0000 7f6f98fa5640 1 --2- 172.21.15.125:0/1241028490 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f800664a0 0x7f6f80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:56.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.867+0000 7f6f98fa5640 1 --2- 172.21.15.125:0/1241028490 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f800664a0 0x7f6f80068960 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6f90007a60 tx=0x7f6f90008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:56.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:56.871+0000 7f6f897fa640 1 -- 172.21.15.125:0/1241028490 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6f8c0ecce0 con 0x7f6f9413f4a0 2023-10-30T12:43:57.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:57.055+0000 7f6f9b209640 1 -- 172.21.15.125:0/1241028490 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f9404e170 con 0x7f6f9413f4a0 2023-10-30T12:43:57.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:57.055+0000 7f6f897fa640 1 -- 172.21.15.125:0/1241028490 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f6f8c0f0300 con 0x7f6f9413f4a0 2023-10-30T12:43:57.055 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:43:57.055 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5988,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:59.575448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:active","state_seq":24,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:57.056 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 77 2023-10-30T12:43:57.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:57.055+0000 7f6f9b209640 1 -- 172.21.15.125:0/1241028490 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f800664a0 msgr2=0x7f6f80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:57.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:57.055+0000 7f6f9b209640 1 --2- 172.21.15.125:0/1241028490 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f800664a0 0x7f6f80068960 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f6f90007a60 tx=0x7f6f90008040 comp rx=0 tx=0).stop 2023-10-30T12:43:57.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:57.055+0000 7f6f9b209640 1 -- 172.21.15.125:0/1241028490 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9413f4a0 msgr2=0x7f6f941559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:57.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:57.055+0000 7f6f9b209640 1 --2- 172.21.15.125:0/1241028490 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9413f4a0 0x7f6f941559b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c01a640 tx=0x7f6f8c0197b0 comp rx=0 tx=0).stop 2023-10-30T12:43:57.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:57.055+0000 7f6f9b209640 1 -- 172.21.15.125:0/1241028490 shutdown_connections 2023-10-30T12:43:57.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:57.055+0000 7f6f9b209640 1 --2- 172.21.15.125:0/1241028490 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f800664a0 0x7f6f80068960 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:57.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:57.055+0000 7f6f9b209640 1 --2- 172.21.15.125:0/1241028490 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f9413f4a0 0x7f6f941559b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:57.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:57.055+0000 7f6f9b209640 1 --2- 172.21.15.125:0/1241028490 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f940c6f60 0x7f6f940c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:57.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:57.055+0000 7f6f9b209640 1 --2- 172.21.15.125:0/1241028490 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f940c6b80 0x7f6f940c9110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:57.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:57.055+0000 7f6f9b209640 1 -- 172.21.15.125:0/1241028490 >> 172.21.15.125:0/1241028490 conn(0x7f6f940c7910 msgr2=0x7f6f940bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:57.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:57.055+0000 7f6f9b209640 1 -- 172.21.15.125:0/1241028490 shutdown_connections 2023-10-30T12:43:57.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:57.055+0000 7f6f9b209640 1 -- 172.21.15.125:0/1241028490 wait complete. 2023-10-30T12:43:57.075 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:43:59.077 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:43:59.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.223+0000 7f337613e640 1 Processor -- start 2023-10-30T12:43:59.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.223+0000 7f337613e640 1 -- start start 2023-10-30T12:43:59.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.223+0000 7f337613e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f337006f7a0 0x7f337006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:59.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.223+0000 7f337613e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f33700c6bd0 0x7f3370069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:59.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.223+0000 7f337613e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337006a3d0 con 0x7f3370071110 2023-10-30T12:43:59.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.223+0000 7f337613e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33700700c0 con 0x7f33700c6bd0 2023-10-30T12:43:59.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.223+0000 7f337613e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3370066cf0 con 0x7f337006f7a0 2023-10-30T12:43:59.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.223+0000 7f336effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f33700c6bd0 0x7f3370069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:59.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.223+0000 7f336effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f33700c6bd0 0x7f3370069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51194/0 (socket says 172.21.15.125:51194) 2023-10-30T12:43:59.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.223+0000 7f336effd640 1 -- 172.21.15.125:0/3016773760 learned_addr learned my addr 172.21.15.125:0/3016773760 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:43:59.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.223+0000 7f336f7fe640 1 --2- 172.21.15.125:0/3016773760 >> v2:172.21.15.136:3300/0 conn(0x7f337006f7a0 0x7f337006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:59.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f336effd640 1 -- 172.21.15.125:0/3016773760 >> v2:172.21.15.136:3300/0 conn(0x7f337006f7a0 msgr2=0x7f337006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:59.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f336effd640 1 --2- 172.21.15.125:0/3016773760 >> v2:172.21.15.136:3300/0 conn(0x7f337006f7a0 0x7f337006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f336effd640 1 -- 172.21.15.125:0/3016773760 >> v1:172.21.15.136:6789/0 conn(0x7f3370071110 legacy=0x7f3370158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:43:59.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f336effd640 1 -- 172.21.15.125:0/3016773760 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33700c70d0 con 0x7f33700c6bd0 2023-10-30T12:43:59.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f336effd640 1 --2- 172.21.15.125:0/3016773760 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33700c6bd0 0x7f3370069e90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f337006c830 tx=0x7f336401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c443e3077a8488 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:59.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f336e7fc640 1 -- 172.21.15.125:0/3016773760 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33640a89b0 con 0x7f33700c6bd0 2023-10-30T12:43:59.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f336e7fc640 1 -- 172.21.15.125:0/3016773760 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33640a8b10 con 0x7f33700c6bd0 2023-10-30T12:43:59.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f336e7fc640 1 -- 172.21.15.125:0/3016773760 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33640a8ea0 con 0x7f33700c6bd0 2023-10-30T12:43:59.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f337613e640 1 -- 172.21.15.125:0/3016773760 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33700c6bd0 msgr2=0x7f3370069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:59.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f337613e640 1 --2- 172.21.15.125:0/3016773760 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33700c6bd0 0x7f3370069e90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f337006c830 tx=0x7f336401ecb0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f337613e640 1 -- 172.21.15.125:0/3016773760 shutdown_connections 2023-10-30T12:43:59.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f337613e640 1 --2- 172.21.15.125:0/3016773760 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33700c6bd0 0x7f3370069e90 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f337613e640 1 --2- 172.21.15.125:0/3016773760 >> v2:172.21.15.136:3300/0 conn(0x7f337006f7a0 0x7f337006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f337613e640 1 -- 172.21.15.125:0/3016773760 >> 172.21.15.125:0/3016773760 conn(0x7f33700c7910 msgr2=0x7f33700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:59.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.227+0000 7f337613e640 1 -- 172.21.15.125:0/3016773760 shutdown_connections 2023-10-30T12:43:59.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f337613e640 1 -- 172.21.15.125:0/3016773760 wait complete. 2023-10-30T12:43:59.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f337613e640 1 Processor -- start 2023-10-30T12:43:59.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f337613e640 1 -- start start 2023-10-30T12:43:59.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f337613e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f337006f7a0 0x7f3370150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:59.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f337613e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3370071110 0x7f337006bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:59.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f337613e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33700c6bd0 0x7f33701ed000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:59.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f337613e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3370157030 con 0x7f3370071110 2023-10-30T12:43:59.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f337613e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3370158690 con 0x7f337006f7a0 2023-10-30T12:43:59.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f337613e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33701f1040 con 0x7f33700c6bd0 2023-10-30T12:43:59.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f336effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3370071110 0x7f337006bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:59.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f336effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3370071110 0x7f337006bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51198/0 (socket says 172.21.15.125:51198) 2023-10-30T12:43:59.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f336effd640 1 -- 172.21.15.125:0/3441693227 learned_addr learned my addr 172.21.15.125:0/3441693227 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:43:59.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f336f7fe640 1 --2- 172.21.15.125:0/3441693227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f337006f7a0 0x7f3370150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:59.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f336ffff640 1 --2- 172.21.15.125:0/3441693227 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33700c6bd0 0x7f33701ed000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:59.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f336effd640 1 -- 172.21.15.125:0/3441693227 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33700c6bd0 msgr2=0x7f33701ed000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:59.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f336effd640 1 --2- 172.21.15.125:0/3441693227 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33700c6bd0 0x7f33701ed000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f336effd640 1 -- 172.21.15.125:0/3441693227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f337006f7a0 msgr2=0x7f3370150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:59.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f336effd640 1 --2- 172.21.15.125:0/3441693227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f337006f7a0 0x7f3370150bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f336effd640 1 -- 172.21.15.125:0/3441693227 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33640094d0 con 0x7f3370071110 2023-10-30T12:43:59.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.231+0000 7f336effd640 1 --2- 172.21.15.125:0/3441693227 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3370071110 0x7f337006bf60 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f33640b3040 tx=0x7f336401ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:59.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.235+0000 7f3357fff640 1 -- 172.21.15.125:0/3441693227 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33640b9620 con 0x7f3370071110 2023-10-30T12:43:59.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.235+0000 7f3357fff640 1 -- 172.21.15.125:0/3441693227 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33640b97a0 con 0x7f3370071110 2023-10-30T12:43:59.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.235+0000 7f337613e640 1 -- 172.21.15.125:0/3441693227 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33701f14d0 con 0x7f3370071110 2023-10-30T12:43:59.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.235+0000 7f3357fff640 1 -- 172.21.15.125:0/3441693227 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33640b9b50 con 0x7f3370071110 2023-10-30T12:43:59.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.235+0000 7f337613e640 1 -- 172.21.15.125:0/3441693227 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33701f0710 con 0x7f3370071110 2023-10-30T12:43:59.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.235+0000 7f337613e640 1 -- 172.21.15.125:0/3441693227 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337004e170 con 0x7f3370071110 2023-10-30T12:43:59.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.235+0000 7f3357fff640 1 -- 172.21.15.125:0/3441693227 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f33640b7070 con 0x7f3370071110 2023-10-30T12:43:59.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.235+0000 7f3357fff640 1 --2- 172.21.15.125:0/3441693227 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f335c0664a0 0x7f335c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:43:59.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.235+0000 7f3357fff640 1 -- 172.21.15.125:0/3441693227 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6254+0+0 (secure 0 0 0) 0x7f3364121f00 con 0x7f3370071110 2023-10-30T12:43:59.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.235+0000 7f336f7fe640 1 --2- 172.21.15.125:0/3441693227 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f335c0664a0 0x7f335c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:43:59.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.239+0000 7f336f7fe640 1 --2- 172.21.15.125:0/3441693227 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f335c0664a0 0x7f335c068960 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f33600079c0 tx=0x7f3360008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:43:59.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.239+0000 7f3357fff640 1 -- 172.21.15.125:0/3441693227 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f33640ef150 con 0x7f3370071110 2023-10-30T12:43:59.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.423+0000 7f337613e640 1 -- 172.21.15.125:0/3441693227 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f337004e170 con 0x7f3370071110 2023-10-30T12:43:59.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.423+0000 7f3357fff640 1 -- 172.21.15.125:0/3441693227 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f33640f2770 con 0x7f3370071110 2023-10-30T12:43:59.423 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:43:59.423 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5988,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:42:59.575448+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5345},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5345":{"gid":5345,"name":"f","rank":1,"incarnation":70,"state":"up:active","state_seq":24,"addr":"172.21.15.136:6839/3922559697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3922559697},{"type":"v1","addr":"172.21.15.136:6839","nonce":3922559697}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:43:59.425 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 77 2023-10-30T12:43:59.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.423+0000 7f337613e640 1 -- 172.21.15.125:0/3441693227 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f335c0664a0 msgr2=0x7f335c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:59.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.423+0000 7f337613e640 1 --2- 172.21.15.125:0/3441693227 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f335c0664a0 0x7f335c068960 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f33600079c0 tx=0x7f3360008040 comp rx=0 tx=0).stop 2023-10-30T12:43:59.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.423+0000 7f337613e640 1 -- 172.21.15.125:0/3441693227 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3370071110 msgr2=0x7f337006bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:43:59.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.423+0000 7f337613e640 1 --2- 172.21.15.125:0/3441693227 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3370071110 0x7f337006bf60 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f33640b3040 tx=0x7f336401ba50 comp rx=0 tx=0).stop 2023-10-30T12:43:59.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.423+0000 7f337613e640 1 -- 172.21.15.125:0/3441693227 shutdown_connections 2023-10-30T12:43:59.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.423+0000 7f337613e640 1 --2- 172.21.15.125:0/3441693227 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f335c0664a0 0x7f335c068960 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.423+0000 7f337613e640 1 --2- 172.21.15.125:0/3441693227 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33700c6bd0 0x7f33701ed000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.423+0000 7f337613e640 1 --2- 172.21.15.125:0/3441693227 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3370071110 0x7f337006bf60 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.423+0000 7f337613e640 1 --2- 172.21.15.125:0/3441693227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f337006f7a0 0x7f3370150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:43:59.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.423+0000 7f337613e640 1 -- 172.21.15.125:0/3441693227 >> 172.21.15.125:0/3441693227 conn(0x7f33700c7910 msgr2=0x7f33700c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:43:59.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.423+0000 7f337613e640 1 -- 172.21.15.125:0/3441693227 shutdown_connections 2023-10-30T12:43:59.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:43:59.423+0000 7f337613e640 1 -- 172.21.15.125:0/3441693227 wait complete. 2023-10-30T12:43:59.444 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:44:01.445 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:01.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.595+0000 7f7195446640 1 Processor -- start 2023-10-30T12:44:01.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.595+0000 7f7195446640 1 -- start start 2023-10-30T12:44:01.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.595+0000 7f7195446640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f71900c6bd0 0x7f719006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.595+0000 7f7195446640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7190066cf0 con 0x7f719006f7a0 2023-10-30T12:44:01.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.595+0000 7f7195446640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71900c70d0 con 0x7f7190071110 2023-10-30T12:44:01.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.595+0000 7f7195446640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71900700c0 con 0x7f71900c6bd0 2023-10-30T12:44:01.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.595+0000 7f718effd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f719006f7a0 0x7f7190069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54198/0 (socket says 172.21.15.125:54198) 2023-10-30T12:44:01.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.595+0000 7f718effd640 1 -- 172.21.15.125:0/598705209 learned_addr learned my addr 172.21.15.125:0/598705209 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:44:01.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.595+0000 7f718e7fc640 1 --2- 172.21.15.125:0/598705209 >> v2:172.21.15.125:3300/0 conn(0x7f71900c6bd0 0x7f719006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.595+0000 7f718dffb640 1 -- 172.21.15.125:0/598705209 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3411417629 0 0) 0x7f7190066cf0 con 0x7f719006f7a0 2023-10-30T12:44:01.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.595+0000 7f718e7fc640 1 -- 172.21.15.125:0/598705209 >> v1:172.21.15.136:6789/0 conn(0x7f7190071110 legacy=0x7f7190158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:01.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.595+0000 7f718e7fc640 1 -- 172.21.15.125:0/598705209 >> v1:172.21.15.125:6789/0 conn(0x7f719006f7a0 legacy=0x7f7190069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.595+0000 7f718e7fc640 1 -- 172.21.15.125:0/598705209 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71900c7390 con 0x7f71900c6bd0 2023-10-30T12:44:01.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.599+0000 7f718e7fc640 1 --2- 172.21.15.125:0/598705209 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71900c6bd0 0x7f719006fb80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f71840097c0 tx=0x7f718401ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba4922f5460a87bd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:01.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.599+0000 7f718dffb640 1 -- 172.21.15.125:0/598705209 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71840a8930 con 0x7f71900c6bd0 2023-10-30T12:44:01.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.599+0000 7f718dffb640 1 -- 172.21.15.125:0/598705209 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71840a8a90 con 0x7f71900c6bd0 2023-10-30T12:44:01.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.599+0000 7f718dffb640 1 -- 172.21.15.125:0/598705209 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71840a8e20 con 0x7f71900c6bd0 2023-10-30T12:44:01.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.599+0000 7f7195446640 1 -- 172.21.15.125:0/598705209 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71900c6bd0 msgr2=0x7f719006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.599+0000 7f7195446640 1 --2- 172.21.15.125:0/598705209 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71900c6bd0 0x7f719006fb80 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f71840097c0 tx=0x7f718401ec30 comp rx=0 tx=0).stop 2023-10-30T12:44:01.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.599+0000 7f7195446640 1 -- 172.21.15.125:0/598705209 shutdown_connections 2023-10-30T12:44:01.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.599+0000 7f7195446640 1 --2- 172.21.15.125:0/598705209 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71900c6bd0 0x7f719006fb80 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.599+0000 7f7195446640 1 -- 172.21.15.125:0/598705209 >> 172.21.15.125:0/598705209 conn(0x7f71900c7910 msgr2=0x7f71900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:01.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.599+0000 7f7195446640 1 -- 172.21.15.125:0/598705209 shutdown_connections 2023-10-30T12:44:01.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.599+0000 7f7195446640 1 -- 172.21.15.125:0/598705209 wait complete. 2023-10-30T12:44:01.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f7195446640 1 Processor -- start 2023-10-30T12:44:01.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f7195446640 1 -- start start 2023-10-30T12:44:01.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f7195446640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f719006f7a0 0x7f7190154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f7195446640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7190071110 0x7f71901540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f7195446640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71900c6bd0 0x7f71900c3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f7195446640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71901f2aa0 con 0x7f71900c6bd0 2023-10-30T12:44:01.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f7195446640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71901550f0 con 0x7f7190071110 2023-10-30T12:44:01.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f7195446640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7190157260 con 0x7f719006f7a0 2023-10-30T12:44:01.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f718f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71900c6bd0 0x7f71900c3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f718f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71900c6bd0 0x7f71900c3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33682/0 (socket says 172.21.15.125:33682) 2023-10-30T12:44:01.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f718f7fe640 1 -- 172.21.15.125:0/1172186368 learned_addr learned my addr 172.21.15.125:0/1172186368 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:01.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f718effd640 1 --2- 172.21.15.125:0/1172186368 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f719006f7a0 0x7f7190154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f718e7fc640 1 --2- 172.21.15.125:0/1172186368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7190071110 0x7f71901540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f718f7fe640 1 -- 172.21.15.125:0/1172186368 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f719006f7a0 msgr2=0x7f7190154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f718f7fe640 1 --2- 172.21.15.125:0/1172186368 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f719006f7a0 0x7f7190154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f718f7fe640 1 -- 172.21.15.125:0/1172186368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7190071110 msgr2=0x7f71901540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f718f7fe640 1 --2- 172.21.15.125:0/1172186368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7190071110 0x7f71901540f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f718f7fe640 1 -- 172.21.15.125:0/1172186368 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7184009420 con 0x7f71900c6bd0 2023-10-30T12:44:01.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f718f7fe640 1 --2- 172.21.15.125:0/1172186368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71900c6bd0 0x7f71900c3060 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f718800b410 tx=0x7f718800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:01.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f71777fe640 1 -- 172.21.15.125:0/1172186368 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7188011830 con 0x7f71900c6bd0 2023-10-30T12:44:01.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f71777fe640 1 -- 172.21.15.125:0/1172186368 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71880119d0 con 0x7f71900c6bd0 2023-10-30T12:44:01.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f71777fe640 1 -- 172.21.15.125:0/1172186368 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7188011d80 con 0x7f71900c6bd0 2023-10-30T12:44:01.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f7195446640 1 -- 172.21.15.125:0/1172186368 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71901f1e90 con 0x7f71900c6bd0 2023-10-30T12:44:01.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f7195446640 1 -- 172.21.15.125:0/1172186368 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71901f2450 con 0x7f71900c6bd0 2023-10-30T12:44:01.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.603+0000 7f7195446640 1 -- 172.21.15.125:0/1172186368 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71901f2c10 con 0x7f71900c6bd0 2023-10-30T12:44:01.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.607+0000 7f71777fe640 1 -- 172.21.15.125:0/1172186368 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f71880193b0 con 0x7f71900c6bd0 2023-10-30T12:44:01.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.607+0000 7f71777fe640 1 --2- 172.21.15.125:0/1172186368 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f717c0664d0 0x7f717c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.607+0000 7f71777fe640 1 -- 172.21.15.125:0/1172186368 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7f7188083eb0 con 0x7f71900c6bd0 2023-10-30T12:44:01.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.607+0000 7f718effd640 1 --2- 172.21.15.125:0/1172186368 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f717c0664d0 0x7f717c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.607+0000 7f718effd640 1 --2- 172.21.15.125:0/1172186368 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f717c0664d0 0x7f717c068990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f71800012f0 tx=0x7f7180005ec0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:01.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.611+0000 7f71777fe640 1 -- 172.21.15.125:0/1172186368 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f71880510a0 con 0x7f71900c6bd0 2023-10-30T12:44:01.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.791+0000 7f7195446640 1 -- 172.21.15.125:0/1172186368 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f719004e170 con 0x7f71900c6bd0 2023-10-30T12:44:01.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.791+0000 7f71777fe640 1 -- 172.21.15.125:0/1172186368 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f71880546c0 con 0x7f71900c6bd0 2023-10-30T12:44:01.792 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:01.792 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:00.178277+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:01.793 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 78 2023-10-30T12:44:01.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.795+0000 7f7195446640 1 -- 172.21.15.125:0/1172186368 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f717c0664d0 msgr2=0x7f717c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.795+0000 7f7195446640 1 --2- 172.21.15.125:0/1172186368 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f717c0664d0 0x7f717c068990 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f71800012f0 tx=0x7f7180005ec0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.795+0000 7f7195446640 1 -- 172.21.15.125:0/1172186368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71900c6bd0 msgr2=0x7f71900c3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.795+0000 7f7195446640 1 --2- 172.21.15.125:0/1172186368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71900c6bd0 0x7f71900c3060 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f718800b410 tx=0x7f718800b8d0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.795+0000 7f7195446640 1 -- 172.21.15.125:0/1172186368 shutdown_connections 2023-10-30T12:44:01.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.795+0000 7f7195446640 1 --2- 172.21.15.125:0/1172186368 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f717c0664d0 0x7f717c068990 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.795+0000 7f7195446640 1 --2- 172.21.15.125:0/1172186368 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f71900c6bd0 0x7f71900c3060 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.795+0000 7f7195446640 1 --2- 172.21.15.125:0/1172186368 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7190071110 0x7f71901540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.795+0000 7f7195446640 1 --2- 172.21.15.125:0/1172186368 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f719006f7a0 0x7f7190154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.795+0000 7f7195446640 1 -- 172.21.15.125:0/1172186368 >> 172.21.15.125:0/1172186368 conn(0x7f71900c7910 msgr2=0x7f71900c7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:01.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.795+0000 7f7195446640 1 -- 172.21.15.125:0/1172186368 shutdown_connections 2023-10-30T12:44:01.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.795+0000 7f7195446640 1 -- 172.21.15.125:0/1172186368 wait complete. 2023-10-30T12:44:01.812 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T12:44:01.813 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:44:01.813 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:01.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd92ab0f640 1 Processor -- start 2023-10-30T12:44:01.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd92ab0f640 1 -- start start 2023-10-30T12:44:01.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd92ab0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd92415d6f0 0x7fd92415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd92ab0f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd9240cf160 0x7fd9240cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd92ab0f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9240cb800 con 0x7fd9241538a0 2023-10-30T12:44:01.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd92ab0f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9240cb330 con 0x7fd9240cf160 2023-10-30T12:44:01.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd92ab0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9240cad70 con 0x7fd92415d6f0 2023-10-30T12:44:01.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd9288ab640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd9241538a0 0x7fd9240ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54210/0 (socket says 172.21.15.125:54210) 2023-10-30T12:44:01.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd9288ab640 1 -- 172.21.15.125:0/1448462460 learned_addr learned my addr 172.21.15.125:0/1448462460 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:44:01.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd9290ac640 1 --2- 172.21.15.125:0/1448462460 >> v2:172.21.15.125:3300/0 conn(0x7fd9240cf160 0x7fd9240cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd91bfff640 1 --2- 172.21.15.125:0/1448462460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd92415d6f0 0x7fd92415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd9290ac640 1 -- 172.21.15.125:0/1448462460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd92415d6f0 msgr2=0x7fd92415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:01.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd9290ac640 1 --2- 172.21.15.125:0/1448462460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd92415d6f0 0x7fd92415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd9290ac640 1 -- 172.21.15.125:0/1448462460 >> v1:172.21.15.125:6789/0 conn(0x7fd9241538a0 legacy=0x7fd9240ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd9290ac640 1 -- 172.21.15.125:0/1448462460 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9240cb050 con 0x7fd9240cf160 2023-10-30T12:44:01.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd91b7fe640 1 -- 172.21.15.125:0/1448462460 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293279912 0 0) 0x7fd9240cb800 con 0x7fd9241538a0 2023-10-30T12:44:01.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.951+0000 7fd91bfff640 1 --2- 172.21.15.125:0/1448462460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd92415d6f0 0x7fd92415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:44:01.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.955+0000 7fd9290ac640 1 --2- 172.21.15.125:0/1448462460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9240cf160 0x7fd9240cf590 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd91c009d90 tx=0x7fd91c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae82cf7d621ae01 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:01.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.955+0000 7fd91b7fe640 1 -- 172.21.15.125:0/1448462460 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91c0af600 con 0x7fd9240cf160 2023-10-30T12:44:01.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.955+0000 7fd91b7fe640 1 -- 172.21.15.125:0/1448462460 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91c0ab050 con 0x7fd9240cf160 2023-10-30T12:44:01.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.955+0000 7fd91b7fe640 1 -- 172.21.15.125:0/1448462460 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91c0af980 con 0x7fd9240cf160 2023-10-30T12:44:01.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.955+0000 7fd92ab0f640 1 -- 172.21.15.125:0/1448462460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9240cf160 msgr2=0x7fd9240cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.955+0000 7fd92ab0f640 1 --2- 172.21.15.125:0/1448462460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9240cf160 0x7fd9240cf590 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fd91c009d90 tx=0x7fd91c01f130 comp rx=0 tx=0).stop 2023-10-30T12:44:01.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.955+0000 7fd92ab0f640 1 -- 172.21.15.125:0/1448462460 shutdown_connections 2023-10-30T12:44:01.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.955+0000 7fd92ab0f640 1 --2- 172.21.15.125:0/1448462460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9240cf160 0x7fd9240cf590 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.955+0000 7fd92ab0f640 1 --2- 172.21.15.125:0/1448462460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd92415d6f0 0x7fd92415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.955+0000 7fd92ab0f640 1 -- 172.21.15.125:0/1448462460 >> 172.21.15.125:0/1448462460 conn(0x7fd92406bfe0 msgr2=0x7fd92406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:01.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.955+0000 7fd92ab0f640 1 -- 172.21.15.125:0/1448462460 shutdown_connections 2023-10-30T12:44:01.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.955+0000 7fd92ab0f640 1 -- 172.21.15.125:0/1448462460 wait complete. 2023-10-30T12:44:01.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd92ab0f640 1 Processor -- start 2023-10-30T12:44:01.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd92ab0f640 1 -- start start 2023-10-30T12:44:01.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd92ab0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9240cf160 0x7fd92414db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd92ab0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9241538a0 0x7fd92414ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd92ab0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd92415d6f0 0x7fd92415b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd92ab0f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9241e36e0 con 0x7fd9241538a0 2023-10-30T12:44:01.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd92ab0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd924150b00 con 0x7fd92415d6f0 2023-10-30T12:44:01.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd92ab0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9241e2b60 con 0x7fd9240cf160 2023-10-30T12:44:01.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd91bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9241538a0 0x7fd92414ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd91bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9241538a0 0x7fd92414ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33700/0 (socket says 172.21.15.125:33700) 2023-10-30T12:44:01.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd91bfff640 1 -- 172.21.15.125:0/3192176841 learned_addr learned my addr 172.21.15.125:0/3192176841 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:01.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd9288ab640 1 --2- 172.21.15.125:0/3192176841 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9240cf160 0x7fd92414db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd9290ac640 1 --2- 172.21.15.125:0/3192176841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd92415d6f0 0x7fd92415b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd91bfff640 1 -- 172.21.15.125:0/3192176841 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9240cf160 msgr2=0x7fd92414db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd91bfff640 1 --2- 172.21.15.125:0/3192176841 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9240cf160 0x7fd92414db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd91bfff640 1 -- 172.21.15.125:0/3192176841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd92415d6f0 msgr2=0x7fd92415b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:01.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd91bfff640 1 --2- 172.21.15.125:0/3192176841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd92415d6f0 0x7fd92415b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:01.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd91bfff640 1 -- 172.21.15.125:0/3192176841 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd91c009a70 con 0x7fd9241538a0 2023-10-30T12:44:01.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd91bfff640 1 --2- 172.21.15.125:0/3192176841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9241538a0 0x7fd92414ef30 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd91400d800 tx=0x7fd91400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:01.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd9197fa640 1 -- 172.21.15.125:0/3192176841 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9140096c0 con 0x7fd9241538a0 2023-10-30T12:44:01.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd9197fa640 1 -- 172.21.15.125:0/3192176841 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd914009820 con 0x7fd9241538a0 2023-10-30T12:44:01.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd9197fa640 1 -- 172.21.15.125:0/3192176841 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd914009bd0 con 0x7fd9241538a0 2023-10-30T12:44:01.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd92ab0f640 1 -- 172.21.15.125:0/3192176841 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9241e2e40 con 0x7fd9241538a0 2023-10-30T12:44:01.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.959+0000 7fd92ab0f640 1 -- 172.21.15.125:0/3192176841 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9241e3400 con 0x7fd9241538a0 2023-10-30T12:44:01.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.963+0000 7fd92ab0f640 1 -- 172.21.15.125:0/3192176841 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9240b2a50 con 0x7fd9241538a0 2023-10-30T12:44:01.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.963+0000 7fd9197fa640 1 -- 172.21.15.125:0/3192176841 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd914009d30 con 0x7fd9241538a0 2023-10-30T12:44:01.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.963+0000 7fd9197fa640 1 --2- 172.21.15.125:0/3192176841 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9100664d0 0x7fd910068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:01.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.963+0000 7fd9197fa640 1 -- 172.21.15.125:0/3192176841 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7fd914084800 con 0x7fd9241538a0 2023-10-30T12:44:01.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.963+0000 7fd9288ab640 1 --2- 172.21.15.125:0/3192176841 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9100664d0 0x7fd910068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:01.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.963+0000 7fd9288ab640 1 --2- 172.21.15.125:0/3192176841 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9100664d0 0x7fd910068990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd920000f70 tx=0x7fd920007b60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:01.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:01.967+0000 7fd9197fa640 1 -- 172.21.15.125:0/3192176841 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd9140519f0 con 0x7fd9241538a0 2023-10-30T12:44:02.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:02.143+0000 7fd92ab0f640 1 -- 172.21.15.125:0/3192176841 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd92404e170 con 0x7fd9241538a0 2023-10-30T12:44:02.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:02.143+0000 7fd9197fa640 1 -- 172.21.15.125:0/3192176841 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5736 (secure 0 0 0) 0x7fd914055010 con 0x7fd9241538a0 2023-10-30T12:44:02.145 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:02.145 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:00.178277+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:02.145 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 78 2023-10-30T12:44:02.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:02.147+0000 7fd92ab0f640 1 -- 172.21.15.125:0/3192176841 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9100664d0 msgr2=0x7fd910068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:02.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:02.147+0000 7fd92ab0f640 1 --2- 172.21.15.125:0/3192176841 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9100664d0 0x7fd910068990 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd920000f70 tx=0x7fd920007b60 comp rx=0 tx=0).stop 2023-10-30T12:44:02.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:02.147+0000 7fd92ab0f640 1 -- 172.21.15.125:0/3192176841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9241538a0 msgr2=0x7fd92414ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:02.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:02.147+0000 7fd92ab0f640 1 --2- 172.21.15.125:0/3192176841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9241538a0 0x7fd92414ef30 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd91400d800 tx=0x7fd91400dcc0 comp rx=0 tx=0).stop 2023-10-30T12:44:02.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:02.147+0000 7fd92ab0f640 1 -- 172.21.15.125:0/3192176841 shutdown_connections 2023-10-30T12:44:02.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:02.147+0000 7fd92ab0f640 1 --2- 172.21.15.125:0/3192176841 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9100664d0 0x7fd910068990 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:02.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:02.147+0000 7fd92ab0f640 1 --2- 172.21.15.125:0/3192176841 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd92415d6f0 0x7fd92415b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:02.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:02.147+0000 7fd92ab0f640 1 --2- 172.21.15.125:0/3192176841 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9241538a0 0x7fd92414ef30 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:02.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:02.147+0000 7fd92ab0f640 1 --2- 172.21.15.125:0/3192176841 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9240cf160 0x7fd92414db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:02.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:02.147+0000 7fd92ab0f640 1 -- 172.21.15.125:0/3192176841 >> 172.21.15.125:0/3192176841 conn(0x7fd92406bfe0 msgr2=0x7fd9240ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:02.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:02.147+0000 7fd92ab0f640 1 -- 172.21.15.125:0/3192176841 shutdown_connections 2023-10-30T12:44:02.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:02.147+0000 7fd92ab0f640 1 -- 172.21.15.125:0/3192176841 wait complete. 2023-10-30T12:44:02.166 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 78, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 78, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5433": { "addr": "172.21.15.125:6836/2595438444", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2595438444, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2595438444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5433, "incarnation": 62, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 69 }, "gid_5988": { "addr": "172.21.15.125:6837/3156415951", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 3156415951, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 3156415951, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5988, "incarnation": 78, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:44:00.178277+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5433, "mds_1": 5988 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:44:04.167 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:04.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff61e254640 1 Processor -- start 2023-10-30T12:44:04.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff61e254640 1 -- start start 2023-10-30T12:44:04.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff61e254640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff61806e470 0x7ff61806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff61e254640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6180c8490 0x7ff6180c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff61e254640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61806afa0 con 0x7ff618153d70 2023-10-30T12:44:04.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff61e254640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff61806d780 con 0x7ff61806e470 2023-10-30T12:44:04.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff61e254640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff618154150 con 0x7ff6180c8490 2023-10-30T12:44:04.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff616ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff61806e470 0x7ff61806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff616ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff61806e470 0x7ff61806ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33704/0 (socket says 172.21.15.125:33704) 2023-10-30T12:44:04.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff616ffd640 1 -- 172.21.15.125:0/1823830389 learned_addr learned my addr 172.21.15.125:0/1823830389 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:04.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff617fff640 1 --2- 172.21.15.125:0/1823830389 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6180c8490 0x7ff6180c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff616ffd640 1 -- 172.21.15.125:0/1823830389 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6180c8490 msgr2=0x7ff6180c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:04.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff616ffd640 1 --2- 172.21.15.125:0/1823830389 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6180c8490 0x7ff6180c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff616ffd640 1 -- 172.21.15.125:0/1823830389 >> v1:172.21.15.125:6789/0 conn(0x7ff618153d70 legacy=0x7ff6180cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:04.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff616ffd640 1 -- 172.21.15.125:0/1823830389 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61806e030 con 0x7ff61806e470 2023-10-30T12:44:04.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.315+0000 7ff6167fc640 1 -- 172.21.15.125:0/1823830389 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129919918 0 0) 0x7ff61806afa0 con 0x7ff618153d70 2023-10-30T12:44:04.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.319+0000 7ff616ffd640 1 --2- 172.21.15.125:0/1823830389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff61806e470 0x7ff61806ca80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009870 tx=0x7ff60c01c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=191da2606ca8accf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:04.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.319+0000 7ff6167fc640 1 -- 172.21.15.125:0/1823830389 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c0aa670 con 0x7ff61806e470 2023-10-30T12:44:04.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.319+0000 7ff6167fc640 1 -- 172.21.15.125:0/1823830389 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60c0aa7d0 con 0x7ff61806e470 2023-10-30T12:44:04.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.319+0000 7ff61e254640 1 -- 172.21.15.125:0/1823830389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff61806e470 msgr2=0x7ff61806ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:04.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.319+0000 7ff61e254640 1 --2- 172.21.15.125:0/1823830389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff61806e470 0x7ff61806ca80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7ff60c009870 tx=0x7ff60c01c510 comp rx=0 tx=0).stop 2023-10-30T12:44:04.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.319+0000 7ff6167fc640 1 -- 172.21.15.125:0/1823830389 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60c0aab60 con 0x7ff61806e470 2023-10-30T12:44:04.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.319+0000 7ff61e254640 1 -- 172.21.15.125:0/1823830389 shutdown_connections 2023-10-30T12:44:04.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.319+0000 7ff61e254640 1 --2- 172.21.15.125:0/1823830389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff61806e470 0x7ff61806ca80 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.319+0000 7ff61e254640 1 --2- 172.21.15.125:0/1823830389 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6180c8490 0x7ff6180c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.319+0000 7ff61e254640 1 -- 172.21.15.125:0/1823830389 >> 172.21.15.125:0/1823830389 conn(0x7ff618071ea0 msgr2=0x7ff618065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:04.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.319+0000 7ff61e254640 1 -- 172.21.15.125:0/1823830389 shutdown_connections 2023-10-30T12:44:04.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.319+0000 7ff61e254640 1 -- 172.21.15.125:0/1823830389 wait complete. 2023-10-30T12:44:04.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.319+0000 7ff61e254640 1 Processor -- start 2023-10-30T12:44:04.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff61e254640 1 -- start start 2023-10-30T12:44:04.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff61e254640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff61806e470 0x7ff618141510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff61e254640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6180c8490 0x7ff61814d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff61e254640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff618153d70 0x7ff61814d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff61e254640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff618150c10 con 0x7ff61806e470 2023-10-30T12:44:04.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff61e254640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff618151100 con 0x7ff618153d70 2023-10-30T12:44:04.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff61e254640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61814ba90 con 0x7ff6180c8490 2023-10-30T12:44:04.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff6177fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff61806e470 0x7ff618141510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff6177fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff61806e470 0x7ff618141510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33706/0 (socket says 172.21.15.125:33706) 2023-10-30T12:44:04.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff6177fe640 1 -- 172.21.15.125:0/3281826548 learned_addr learned my addr 172.21.15.125:0/3281826548 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:04.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff616ffd640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6180c8490 0x7ff61814d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff617fff640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff618153d70 0x7ff61814d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff6177fe640 1 -- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6180c8490 msgr2=0x7ff61814d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:04.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff6177fe640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6180c8490 0x7ff61814d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff6177fe640 1 -- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff618153d70 msgr2=0x7ff61814d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:04.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff6177fe640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff618153d70 0x7ff61814d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff6177fe640 1 -- 172.21.15.125:0/3281826548 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c0094d0 con 0x7ff61806e470 2023-10-30T12:44:04.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff616ffd640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6180c8490 0x7ff61814d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:44:04.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff617fff640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff618153d70 0x7ff61814d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:44:04.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff6177fe640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff61806e470 0x7ff618141510 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff60800c440 tx=0x7ff60800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:04.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff5fffff640 1 -- 172.21.15.125:0/3281826548 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608005480 con 0x7ff61806e470 2023-10-30T12:44:04.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff5fffff640 1 -- 172.21.15.125:0/3281826548 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff608003600 con 0x7ff61806e470 2023-10-30T12:44:04.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff5fffff640 1 -- 172.21.15.125:0/3281826548 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6080039b0 con 0x7ff61806e470 2023-10-30T12:44:04.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff61e254640 1 -- 172.21.15.125:0/3281826548 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6181f2790 con 0x7ff61806e470 2023-10-30T12:44:04.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff61e254640 1 -- 172.21.15.125:0/3281826548 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6181408c0 con 0x7ff61806e470 2023-10-30T12:44:04.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.323+0000 7ff61e254640 1 -- 172.21.15.125:0/3281826548 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6181f25f0 con 0x7ff61806e470 2023-10-30T12:44:04.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.327+0000 7ff5fffff640 1 -- 172.21.15.125:0/3281826548 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff608003b10 con 0x7ff61806e470 2023-10-30T12:44:04.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.327+0000 7ff5fffff640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6040664d0 0x7ff604068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:04.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.327+0000 7ff5fffff640 1 -- 172.21.15.125:0/3281826548 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7ff608083d20 con 0x7ff61806e470 2023-10-30T12:44:04.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.327+0000 7ff616ffd640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6040664d0 0x7ff604068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:04.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.327+0000 7ff616ffd640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6040664d0 0x7ff604068990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0ae000 tx=0x7ff60c0a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:04.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.331+0000 7ff5fffff640 1 -- 172.21.15.125:0/3281826548 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff6181f25f0 con 0x7ff61806e470 2023-10-30T12:44:04.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.515+0000 7ff61e254640 1 -- 172.21.15.125:0/3281826548 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6181f25f0 con 0x7ff61806e470 2023-10-30T12:44:04.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.515+0000 7ff5fffff640 1 -- 172.21.15.125:0/3281826548 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5736 (secure 0 0 0) 0x7ff6181f25f0 con 0x7ff61806e470 2023-10-30T12:44:04.515 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:04.516 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:00.178277+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:04.516 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 78 2023-10-30T12:44:04.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.519+0000 7ff61e254640 1 -- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6040664d0 msgr2=0x7ff604068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:04.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.519+0000 7ff61e254640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6040664d0 0x7ff604068990 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0ae000 tx=0x7ff60c0a7000 comp rx=0 tx=0).stop 2023-10-30T12:44:04.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.519+0000 7ff61e254640 1 -- 172.21.15.125:0/3281826548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff61806e470 msgr2=0x7ff618141510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:04.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.519+0000 7ff61e254640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff61806e470 0x7ff618141510 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7ff60800c440 tx=0x7ff60800c900 comp rx=0 tx=0).stop 2023-10-30T12:44:04.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.519+0000 7ff61e254640 1 -- 172.21.15.125:0/3281826548 shutdown_connections 2023-10-30T12:44:04.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.519+0000 7ff61e254640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6040664d0 0x7ff604068990 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.519+0000 7ff61e254640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff618153d70 0x7ff61814d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.519+0000 7ff61e254640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6180c8490 0x7ff61814d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.519+0000 7ff61e254640 1 --2- 172.21.15.125:0/3281826548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff61806e470 0x7ff618141510 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:04.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.519+0000 7ff61e254640 1 -- 172.21.15.125:0/3281826548 >> 172.21.15.125:0/3281826548 conn(0x7ff618071ea0 msgr2=0x7ff6180cd260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:04.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.519+0000 7ff61e254640 1 -- 172.21.15.125:0/3281826548 shutdown_connections 2023-10-30T12:44:04.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:04.519+0000 7ff61e254640 1 -- 172.21.15.125:0/3281826548 wait complete. 2023-10-30T12:44:04.536 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:44:06.538 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:06.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb990f640 1 Processor -- start 2023-10-30T12:44:06.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb990f640 1 -- start start 2023-10-30T12:44:06.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb990f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2eb40c8cf0 0x7f2eb40c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:06.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb990f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2eb40ca340 0x7f2eb415d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:06.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb990f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb415ce30 con 0x7f2eb40c90d0 2023-10-30T12:44:06.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb990f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb4159880 con 0x7f2eb40ca340 2023-10-30T12:44:06.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb990f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb4159ae0 con 0x7f2eb40c8cf0 2023-10-30T12:44:06.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb27fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2eb40c90d0 0x7f2eb40cef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54242/0 (socket says 172.21.15.125:54242) 2023-10-30T12:44:06.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb27fc640 1 -- 172.21.15.125:0/3821017273 learned_addr learned my addr 172.21.15.125:0/3821017273 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:44:06.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb2ffd640 1 --2- 172.21.15.125:0/3821017273 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2eb40c8cf0 0x7f2eb40c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:06.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb37fe640 1 --2- 172.21.15.125:0/3821017273 >> v2:172.21.15.136:3300/0 conn(0x7f2eb40ca340 0x7f2eb415d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:06.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb1ffb640 1 -- 172.21.15.125:0/3821017273 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416122054 0 0) 0x7f2eb415ce30 con 0x7f2eb40c90d0 2023-10-30T12:44:06.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb1ffb640 1 -- 172.21.15.125:0/3821017273 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e9c003530 con 0x7f2eb40c90d0 2023-10-30T12:44:06.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb2ffd640 1 -- 172.21.15.125:0/3821017273 >> v2:172.21.15.136:3300/0 conn(0x7f2eb40ca340 msgr2=0x7f2eb415d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:06.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb2ffd640 1 --2- 172.21.15.125:0/3821017273 >> v2:172.21.15.136:3300/0 conn(0x7f2eb40ca340 0x7f2eb415d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:06.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb2ffd640 1 -- 172.21.15.125:0/3821017273 >> v1:172.21.15.125:6789/0 conn(0x7f2eb40c90d0 legacy=0x7f2eb40cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:06.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.683+0000 7f2eb2ffd640 1 -- 172.21.15.125:0/3821017273 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb4159dc0 con 0x7f2eb40c8cf0 2023-10-30T12:44:06.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.687+0000 7f2eb2ffd640 1 --2- 172.21.15.125:0/3821017273 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2eb40c8cf0 0x7f2eb40c9e00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40097c0 tx=0x7f2ea401c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82cb936afb4aafa3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:06.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.687+0000 7f2eb1ffb640 1 -- 172.21.15.125:0/3821017273 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea40ad660 con 0x7f2eb40c8cf0 2023-10-30T12:44:06.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.687+0000 7f2eb1ffb640 1 -- 172.21.15.125:0/3821017273 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea40ad7c0 con 0x7f2eb40c8cf0 2023-10-30T12:44:06.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.687+0000 7f2eb1ffb640 1 -- 172.21.15.125:0/3821017273 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea40adb50 con 0x7f2eb40c8cf0 2023-10-30T12:44:06.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.687+0000 7f2eb990f640 1 -- 172.21.15.125:0/3821017273 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2eb40c8cf0 msgr2=0x7f2eb40c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:06.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.687+0000 7f2eb990f640 1 --2- 172.21.15.125:0/3821017273 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2eb40c8cf0 0x7f2eb40c9e00 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40097c0 tx=0x7f2ea401c450 comp rx=0 tx=0).stop 2023-10-30T12:44:06.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.687+0000 7f2eb990f640 1 -- 172.21.15.125:0/3821017273 shutdown_connections 2023-10-30T12:44:06.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.687+0000 7f2eb990f640 1 --2- 172.21.15.125:0/3821017273 >> v2:172.21.15.136:3300/0 conn(0x7f2eb40ca340 0x7f2eb415d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:06.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.687+0000 7f2eb990f640 1 --2- 172.21.15.125:0/3821017273 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2eb40c8cf0 0x7f2eb40c9e00 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:06.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.687+0000 7f2eb990f640 1 -- 172.21.15.125:0/3821017273 >> 172.21.15.125:0/3821017273 conn(0x7f2eb406be40 msgr2=0x7f2eb406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:06.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.687+0000 7f2eb990f640 1 -- 172.21.15.125:0/3821017273 shutdown_connections 2023-10-30T12:44:06.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.687+0000 7f2eb990f640 1 -- 172.21.15.125:0/3821017273 wait complete. 2023-10-30T12:44:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb990f640 1 Processor -- start 2023-10-30T12:44:06.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb990f640 1 -- start start 2023-10-30T12:44:06.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb990f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2eb40c8cf0 0x7f2eb40c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:06.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb990f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eb40c90d0 0x7f2eb41f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:06.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb990f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2eb40ca340 0x7f2eb4155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:06.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb990f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb4153620 con 0x7f2eb40c90d0 2023-10-30T12:44:06.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb990f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb40b3a70 con 0x7f2eb40c8cf0 2023-10-30T12:44:06.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb990f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb4153870 con 0x7f2eb40ca340 2023-10-30T12:44:06.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb27fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eb40c90d0 0x7f2eb41f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:06.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb27fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eb40c90d0 0x7f2eb41f2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33716/0 (socket says 172.21.15.125:33716) 2023-10-30T12:44:06.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb27fc640 1 -- 172.21.15.125:0/882908424 learned_addr learned my addr 172.21.15.125:0/882908424 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:06.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb2ffd640 1 --2- 172.21.15.125:0/882908424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2eb40c8cf0 0x7f2eb40c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:06.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb37fe640 1 --2- 172.21.15.125:0/882908424 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2eb40ca340 0x7f2eb4155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:06.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb27fc640 1 -- 172.21.15.125:0/882908424 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2eb40ca340 msgr2=0x7f2eb4155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:06.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb27fc640 1 --2- 172.21.15.125:0/882908424 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2eb40ca340 0x7f2eb4155eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:06.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb27fc640 1 -- 172.21.15.125:0/882908424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2eb40c8cf0 msgr2=0x7f2eb40c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:06.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb27fc640 1 --2- 172.21.15.125:0/882908424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2eb40c8cf0 0x7f2eb40c6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:06.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb27fc640 1 -- 172.21.15.125:0/882908424 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea4009420 con 0x7f2eb40c90d0 2023-10-30T12:44:06.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb27fc640 1 --2- 172.21.15.125:0/882908424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eb40c90d0 0x7f2eb41f2960 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800c4d0 tx=0x7f2ea800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:06.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2e9bfff640 1 -- 172.21.15.125:0/882908424 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8005cf0 con 0x7f2eb40c90d0 2023-10-30T12:44:06.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2e9bfff640 1 -- 172.21.15.125:0/882908424 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea8005e50 con 0x7f2eb40c90d0 2023-10-30T12:44:06.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2e9bfff640 1 -- 172.21.15.125:0/882908424 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8007270 con 0x7f2eb40c90d0 2023-10-30T12:44:06.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb990f640 1 -- 172.21.15.125:0/882908424 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb4153b50 con 0x7f2eb40c90d0 2023-10-30T12:44:06.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.691+0000 7f2eb990f640 1 -- 172.21.15.125:0/882908424 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb41f2440 con 0x7f2eb40c90d0 2023-10-30T12:44:06.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.695+0000 7f2eb990f640 1 -- 172.21.15.125:0/882908424 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb41f2660 con 0x7f2eb40c90d0 2023-10-30T12:44:06.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.695+0000 7f2e9bfff640 1 -- 172.21.15.125:0/882908424 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ea80073d0 con 0x7f2eb40c90d0 2023-10-30T12:44:06.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.695+0000 7f2e9bfff640 1 --2- 172.21.15.125:0/882908424 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ea00664d0 0x7f2ea0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:06.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.695+0000 7f2e9bfff640 1 -- 172.21.15.125:0/882908424 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7f2ea8083610 con 0x7f2eb40c90d0 2023-10-30T12:44:06.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.695+0000 7f2eb2ffd640 1 --2- 172.21.15.125:0/882908424 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ea00664d0 0x7f2ea0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:06.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.699+0000 7f2eb2ffd640 1 --2- 172.21.15.125:0/882908424 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ea00664d0 0x7f2ea0068990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40af000 tx=0x7f2ea40a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:06.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.703+0000 7f2e9bfff640 1 -- 172.21.15.125:0/882908424 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2ea8050820 con 0x7f2eb40c90d0 2023-10-30T12:44:06.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.883+0000 7f2eb990f640 1 -- 172.21.15.125:0/882908424 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2eb404e170 con 0x7f2eb40c90d0 2023-10-30T12:44:06.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.883+0000 7f2e9bfff640 1 -- 172.21.15.125:0/882908424 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f2ea8053e40 con 0x7f2eb40c90d0 2023-10-30T12:44:06.882 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:06.882 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:00.178277+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:06.882 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 78 2023-10-30T12:44:06.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.883+0000 7f2eb990f640 1 -- 172.21.15.125:0/882908424 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ea00664d0 msgr2=0x7f2ea0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:06.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.883+0000 7f2eb990f640 1 --2- 172.21.15.125:0/882908424 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ea00664d0 0x7f2ea0068990 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2ea40af000 tx=0x7f2ea40a7000 comp rx=0 tx=0).stop 2023-10-30T12:44:06.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.883+0000 7f2eb990f640 1 -- 172.21.15.125:0/882908424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eb40c90d0 msgr2=0x7f2eb41f2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:06.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.883+0000 7f2eb990f640 1 --2- 172.21.15.125:0/882908424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eb40c90d0 0x7f2eb41f2960 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f2ea800c4d0 tx=0x7f2ea800c990 comp rx=0 tx=0).stop 2023-10-30T12:44:06.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.883+0000 7f2eb990f640 1 -- 172.21.15.125:0/882908424 shutdown_connections 2023-10-30T12:44:06.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.883+0000 7f2eb990f640 1 --2- 172.21.15.125:0/882908424 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ea00664d0 0x7f2ea0068990 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:06.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.883+0000 7f2eb990f640 1 --2- 172.21.15.125:0/882908424 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2eb40ca340 0x7f2eb4155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:06.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.883+0000 7f2eb990f640 1 --2- 172.21.15.125:0/882908424 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2eb40c90d0 0x7f2eb41f2960 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:06.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.883+0000 7f2eb990f640 1 --2- 172.21.15.125:0/882908424 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2eb40c8cf0 0x7f2eb40c6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:06.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.883+0000 7f2eb990f640 1 -- 172.21.15.125:0/882908424 >> 172.21.15.125:0/882908424 conn(0x7f2eb406be40 msgr2=0x7f2eb40b2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:06.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.883+0000 7f2eb990f640 1 -- 172.21.15.125:0/882908424 shutdown_connections 2023-10-30T12:44:06.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:06.883+0000 7f2eb990f640 1 -- 172.21.15.125:0/882908424 wait complete. 2023-10-30T12:44:06.902 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:44:08.904 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:09.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd73b71d640 1 Processor -- start 2023-10-30T12:44:09.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd73b71d640 1 -- start start 2023-10-30T12:44:09.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd73b71d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd734153d70 0x7fd73406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:09.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd73b71d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73406afa0 con 0x7fd7340c8490 2023-10-30T12:44:09.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd73b71d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd734154150 con 0x7fd73406e470 2023-10-30T12:44:09.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd73b71d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd73406d780 con 0x7fd734153d70 2023-10-30T12:44:09.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd739cba640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd7340c8490 0x7fd7340c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54252/0 (socket says 172.21.15.125:54252) 2023-10-30T12:44:09.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd739cba640 1 -- 172.21.15.125:0/3059150093 learned_addr learned my addr 172.21.15.125:0/3059150093 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:44:09.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd7394b9640 1 --2- 172.21.15.125:0/3059150093 >> v2:172.21.15.136:3300/0 conn(0x7fd734153d70 0x7fd73406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:09.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806904435 0 0) 0x7fd73406afa0 con 0x7fd7340c8490 2023-10-30T12:44:09.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd720003530 con 0x7fd7340c8490 2023-10-30T12:44:09.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414484708 0 0) 0x7fd734154150 con 0x7fd73406e470 2023-10-30T12:44:09.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd73406afa0 con 0x7fd73406e470 2023-10-30T12:44:09.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 952835290 0 0) 0x7fd720003530 con 0x7fd7340c8490 2023-10-30T12:44:09.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 >> v2:172.21.15.136:3300/0 conn(0x7fd734153d70 msgr2=0x7fd73406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:09.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 --2- 172.21.15.125:0/3059150093 >> v2:172.21.15.136:3300/0 conn(0x7fd734153d70 0x7fd73406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:09.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 >> v1:172.21.15.136:6789/0 conn(0x7fd73406e470 legacy=0x7fd7340cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:09.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73406e030 con 0x7fd7340c8490 2023-10-30T12:44:09.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fd72c001a00 con 0x7fd7340c8490 2023-10-30T12:44:09.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 >> v1:172.21.15.125:6789/0 conn(0x7fd7340c8490 legacy=0x7fd7340c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:09.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 --2- 172.21.15.125:0/3059150093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd720003b60 0x7fd720003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:09.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 --2- 172.21.15.125:0/3059150093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7200047d0 0x7fd720004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:09.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 --2- 172.21.15.125:0/3059150093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd720005190 0x7fd72000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:09.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73406d780 con 0x7fd720005190 2023-10-30T12:44:09.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72000c060 con 0x7fd720003b60 2023-10-30T12:44:09.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd72000c220 con 0x7fd7200047d0 2023-10-30T12:44:09.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd739cba640 1 --2- 172.21.15.125:0/3059150093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd720005190 0x7fd72000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:09.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd7394b9640 1 --2- 172.21.15.125:0/3059150093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd720003b60 0x7fd720003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:09.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd738cb8640 1 --2- 172.21.15.125:0/3059150093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7200047d0 0x7fd720004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:09.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd739cba640 1 -- 172.21.15.125:0/3059150093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7200047d0 msgr2=0x7fd720004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:09.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd739cba640 1 --2- 172.21.15.125:0/3059150093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7200047d0 0x7fd720004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:09.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd739cba640 1 -- 172.21.15.125:0/3059150093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd720003b60 msgr2=0x7fd720003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:09.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd739cba640 1 --2- 172.21.15.125:0/3059150093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd720003b60 0x7fd720003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:09.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.051+0000 7fd739cba640 1 -- 172.21.15.125:0/3059150093 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd720003530 con 0x7fd720005190 2023-10-30T12:44:09.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.055+0000 7fd739cba640 1 -- 172.21.15.125:0/3059150093 reap_dead start 2023-10-30T12:44:09.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.055+0000 7fd739cba640 1 --2- 172.21.15.125:0/3059150093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd720005190 0x7fd72000b990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd72c00a710 tx=0x7fd72c01f960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392d6429c1613017 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:09.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.055+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c0b0070 con 0x7fd720005190 2023-10-30T12:44:09.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.055+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72c0ac410 con 0x7fd720005190 2023-10-30T12:44:09.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.055+0000 7fd72bfff640 1 -- 172.21.15.125:0/3059150093 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd72c0ac7c0 con 0x7fd720005190 2023-10-30T12:44:09.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.055+0000 7fd73b71d640 1 -- 172.21.15.125:0/3059150093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd720005190 msgr2=0x7fd72000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:09.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.055+0000 7fd73b71d640 1 --2- 172.21.15.125:0/3059150093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd720005190 0x7fd72000b990 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd72c00a710 tx=0x7fd72c01f960 comp rx=0 tx=0).stop 2023-10-30T12:44:09.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.055+0000 7fd73b71d640 1 -- 172.21.15.125:0/3059150093 shutdown_connections 2023-10-30T12:44:09.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.055+0000 7fd73b71d640 1 --2- 172.21.15.125:0/3059150093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd720005190 0x7fd72000b990 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:09.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.055+0000 7fd73b71d640 1 -- 172.21.15.125:0/3059150093 >> 172.21.15.125:0/3059150093 conn(0x7fd734071ea0 msgr2=0x7fd734065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:09.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.055+0000 7fd73b71d640 1 -- 172.21.15.125:0/3059150093 shutdown_connections 2023-10-30T12:44:09.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.055+0000 7fd73b71d640 1 -- 172.21.15.125:0/3059150093 wait complete. 2023-10-30T12:44:09.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd73b71d640 1 Processor -- start 2023-10-30T12:44:09.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd73b71d640 1 -- start start 2023-10-30T12:44:09.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd73b71d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd720005190 0x7fd7341f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:09.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd73b71d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7341f2af0 0x7fd7341f2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:09.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd73b71d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7341ef040 0x7fd7341ef4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:09.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd73b71d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73415c590 con 0x7fd720005190 2023-10-30T12:44:09.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd73b71d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73415da40 con 0x7fd7341ef040 2023-10-30T12:44:09.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd73b71d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7341595f0 con 0x7fd7341f2af0 2023-10-30T12:44:09.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd7394b9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd720005190 0x7fd7341f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:09.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd7394b9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd720005190 0x7fd7341f25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33734/0 (socket says 172.21.15.125:33734) 2023-10-30T12:44:09.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd7394b9640 1 -- 172.21.15.125:0/3000925649 learned_addr learned my addr 172.21.15.125:0/3000925649 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:09.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd739cba640 1 --2- 172.21.15.125:0/3000925649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7341ef040 0x7fd7341ef4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:09.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd738cb8640 1 --2- 172.21.15.125:0/3000925649 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7341f2af0 0x7fd7341f2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:09.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd7394b9640 1 -- 172.21.15.125:0/3000925649 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7341f2af0 msgr2=0x7fd7341f2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:09.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd7394b9640 1 --2- 172.21.15.125:0/3000925649 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7341f2af0 0x7fd7341f2f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:09.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd7394b9640 1 -- 172.21.15.125:0/3000925649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7341ef040 msgr2=0x7fd7341ef4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:09.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd7394b9640 1 --2- 172.21.15.125:0/3000925649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7341ef040 0x7fd7341ef4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:09.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd7394b9640 1 -- 172.21.15.125:0/3000925649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd734159890 con 0x7fd720005190 2023-10-30T12:44:09.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd7394b9640 1 --2- 172.21.15.125:0/3000925649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd720005190 0x7fd7341f25b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd730011960 tx=0x7fd730011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:09.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd729ffb640 1 -- 172.21.15.125:0/3000925649 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd73000faf0 con 0x7fd720005190 2023-10-30T12:44:09.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd729ffb640 1 -- 172.21.15.125:0/3000925649 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd73000fc50 con 0x7fd720005190 2023-10-30T12:44:09.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd73b71d640 1 -- 172.21.15.125:0/3000925649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd734159b70 con 0x7fd720005190 2023-10-30T12:44:09.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.059+0000 7fd729ffb640 1 -- 172.21.15.125:0/3000925649 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd730014440 con 0x7fd720005190 2023-10-30T12:44:09.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.063+0000 7fd73b71d640 1 -- 172.21.15.125:0/3000925649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7341588d0 con 0x7fd720005190 2023-10-30T12:44:09.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.063+0000 7fd73b71d640 1 -- 172.21.15.125:0/3000925649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd73404e170 con 0x7fd720005190 2023-10-30T12:44:09.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.063+0000 7fd729ffb640 1 -- 172.21.15.125:0/3000925649 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd7300145a0 con 0x7fd720005190 2023-10-30T12:44:09.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.063+0000 7fd729ffb640 1 --2- 172.21.15.125:0/3000925649 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7180664d0 0x7fd718068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:09.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.063+0000 7fd729ffb640 1 -- 172.21.15.125:0/3000925649 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7fd7300848b0 con 0x7fd720005190 2023-10-30T12:44:09.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.063+0000 7fd738cb8640 1 --2- 172.21.15.125:0/3000925649 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7180664d0 0x7fd718068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:09.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.067+0000 7fd738cb8640 1 --2- 172.21.15.125:0/3000925649 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7180664d0 0x7fd718068990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd734157750 tx=0x7fd724009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:09.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.067+0000 7fd729ffb640 1 -- 172.21.15.125:0/3000925649 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd730051b20 con 0x7fd720005190 2023-10-30T12:44:09.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.251+0000 7fd73b71d640 1 -- 172.21.15.125:0/3000925649 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd73406c2a0 con 0x7fd720005190 2023-10-30T12:44:09.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.251+0000 7fd729ffb640 1 -- 172.21.15.125:0/3000925649 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fd730055140 con 0x7fd720005190 2023-10-30T12:44:09.252 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:09.252 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:06.704344+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:resolve","state_seq":20,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:09.253 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 79 2023-10-30T12:44:09.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.255+0000 7fd73b71d640 1 -- 172.21.15.125:0/3000925649 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7180664d0 msgr2=0x7fd718068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:09.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.255+0000 7fd73b71d640 1 --2- 172.21.15.125:0/3000925649 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7180664d0 0x7fd718068990 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd734157750 tx=0x7fd724009040 comp rx=0 tx=0).stop 2023-10-30T12:44:09.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.255+0000 7fd73b71d640 1 -- 172.21.15.125:0/3000925649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd720005190 msgr2=0x7fd7341f25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:09.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.255+0000 7fd73b71d640 1 --2- 172.21.15.125:0/3000925649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd720005190 0x7fd7341f25b0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd730011960 tx=0x7fd730011e20 comp rx=0 tx=0).stop 2023-10-30T12:44:09.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.255+0000 7fd73b71d640 1 -- 172.21.15.125:0/3000925649 shutdown_connections 2023-10-30T12:44:09.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.255+0000 7fd73b71d640 1 --2- 172.21.15.125:0/3000925649 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7180664d0 0x7fd718068990 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:09.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.255+0000 7fd73b71d640 1 --2- 172.21.15.125:0/3000925649 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7341ef040 0x7fd7341ef4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:09.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.255+0000 7fd73b71d640 1 --2- 172.21.15.125:0/3000925649 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7341f2af0 0x7fd7341f2f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:09.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.255+0000 7fd73b71d640 1 --2- 172.21.15.125:0/3000925649 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd720005190 0x7fd7341f25b0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:09.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.255+0000 7fd73b71d640 1 -- 172.21.15.125:0/3000925649 >> 172.21.15.125:0/3000925649 conn(0x7fd734071ea0 msgr2=0x7fd7340c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:09.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.255+0000 7fd73b71d640 1 -- 172.21.15.125:0/3000925649 shutdown_connections 2023-10-30T12:44:09.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:09.255+0000 7fd73b71d640 1 -- 172.21.15.125:0/3000925649 wait complete. 2023-10-30T12:44:09.272 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:44:11.272 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:11.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2f846640 1 Processor -- start 2023-10-30T12:44:11.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2f846640 1 -- start start 2023-10-30T12:44:11.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2f846640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbf28153d70 0x7fbf2806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:11.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2f846640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbf2806e470 0x7fbf280cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:11.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2f846640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2806afa0 con 0x7fbf280c8490 2023-10-30T12:44:11.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2f846640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf2806d780 con 0x7fbf2806e470 2023-10-30T12:44:11.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2f846640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf28154150 con 0x7fbf28153d70 2023-10-30T12:44:11.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2cde1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbf2806e470 0x7fbf280cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:11.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2cde1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbf2806e470 0x7fbf280cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43382/0 (socket says 172.21.15.125:43382) 2023-10-30T12:44:11.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2cde1640 1 -- 172.21.15.125:0/3889375183 learned_addr learned my addr 172.21.15.125:0/3889375183 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:11.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2d5e2640 1 --2- 172.21.15.125:0/3889375183 >> v2:172.21.15.136:3300/0 conn(0x7fbf28153d70 0x7fbf2806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:11.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2cde1640 1 -- 172.21.15.125:0/3889375183 >> v2:172.21.15.136:3300/0 conn(0x7fbf28153d70 msgr2=0x7fbf2806ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:11.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2cde1640 1 --2- 172.21.15.125:0/3889375183 >> v2:172.21.15.136:3300/0 conn(0x7fbf28153d70 0x7fbf2806ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:11.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2cde1640 1 -- 172.21.15.125:0/3889375183 >> v1:172.21.15.136:6789/0 conn(0x7fbf280c8490 legacy=0x7fbf280c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:11.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.419+0000 7fbf2cde1640 1 -- 172.21.15.125:0/3889375183 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2806e030 con 0x7fbf2806e470 2023-10-30T12:44:11.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.423+0000 7fbf2cde1640 1 --2- 172.21.15.125:0/3889375183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf2806e470 0x7fbf280cce00 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fbf2806d180 tx=0x7fbf1801c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244f4fc814c1105b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:11.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.423+0000 7fbf1ffff640 1 -- 172.21.15.125:0/3889375183 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf180aa5f0 con 0x7fbf2806e470 2023-10-30T12:44:11.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.423+0000 7fbf1ffff640 1 -- 172.21.15.125:0/3889375183 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf180aa750 con 0x7fbf2806e470 2023-10-30T12:44:11.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.423+0000 7fbf1ffff640 1 -- 172.21.15.125:0/3889375183 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf180aaae0 con 0x7fbf2806e470 2023-10-30T12:44:11.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.423+0000 7fbf2f846640 1 -- 172.21.15.125:0/3889375183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf2806e470 msgr2=0x7fbf280cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:11.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.423+0000 7fbf2f846640 1 --2- 172.21.15.125:0/3889375183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf2806e470 0x7fbf280cce00 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fbf2806d180 tx=0x7fbf1801c490 comp rx=0 tx=0).stop 2023-10-30T12:44:11.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.423+0000 7fbf2f846640 1 -- 172.21.15.125:0/3889375183 shutdown_connections 2023-10-30T12:44:11.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.423+0000 7fbf2f846640 1 --2- 172.21.15.125:0/3889375183 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf2806e470 0x7fbf280cce00 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:11.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.423+0000 7fbf2f846640 1 --2- 172.21.15.125:0/3889375183 >> v2:172.21.15.136:3300/0 conn(0x7fbf28153d70 0x7fbf2806ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:11.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.423+0000 7fbf2f846640 1 -- 172.21.15.125:0/3889375183 >> 172.21.15.125:0/3889375183 conn(0x7fbf28071ea0 msgr2=0x7fbf28065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:11.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.423+0000 7fbf2f846640 1 -- 172.21.15.125:0/3889375183 shutdown_connections 2023-10-30T12:44:11.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.423+0000 7fbf2f846640 1 -- 172.21.15.125:0/3889375183 wait complete. 2023-10-30T12:44:11.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2f846640 1 Processor -- start 2023-10-30T12:44:11.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2f846640 1 -- start start 2023-10-30T12:44:11.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2f846640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf2806e470 0x7fbf281f6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:11.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2f846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf280c8490 0x7fbf281427a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:11.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2f846640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf28153d70 0x7fbf280cc050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:11.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2f846640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf28144e50 con 0x7fbf28153d70 2023-10-30T12:44:11.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2f846640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf281f6ed0 con 0x7fbf2806e470 2023-10-30T12:44:11.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2f846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf281464d0 con 0x7fbf280c8490 2023-10-30T12:44:11.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2dde3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf28153d70 0x7fbf280cc050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:11.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2dde3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf28153d70 0x7fbf280cc050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43388/0 (socket says 172.21.15.125:43388) 2023-10-30T12:44:11.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2dde3640 1 -- 172.21.15.125:0/4244335606 learned_addr learned my addr 172.21.15.125:0/4244335606 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:11.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2d5e2640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf2806e470 0x7fbf281f6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:11.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2cde1640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf280c8490 0x7fbf281427a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:11.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2dde3640 1 -- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf280c8490 msgr2=0x7fbf281427a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:11.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2dde3640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf280c8490 0x7fbf281427a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:11.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2dde3640 1 -- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf2806e470 msgr2=0x7fbf281f6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:11.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2dde3640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf2806e470 0x7fbf281f6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:11.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2dde3640 1 -- 172.21.15.125:0/4244335606 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf180094d0 con 0x7fbf28153d70 2023-10-30T12:44:11.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2cde1640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf280c8490 0x7fbf281427a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:44:11.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2dde3640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf28153d70 0x7fbf280cc050 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000c780 tx=0x7fbf2000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:11.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf1dffb640 1 -- 172.21.15.125:0/4244335606 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20004b70 con 0x7fbf28153d70 2023-10-30T12:44:11.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf1dffb640 1 -- 172.21.15.125:0/4244335606 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2000ce90 con 0x7fbf28153d70 2023-10-30T12:44:11.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf1dffb640 1 -- 172.21.15.125:0/4244335606 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2000a850 con 0x7fbf28153d70 2023-10-30T12:44:11.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2f846640 1 -- 172.21.15.125:0/4244335606 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf28154ea0 con 0x7fbf28153d70 2023-10-30T12:44:11.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.427+0000 7fbf2f846640 1 -- 172.21.15.125:0/4244335606 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf28145ba0 con 0x7fbf28153d70 2023-10-30T12:44:11.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.431+0000 7fbf2f846640 1 -- 172.21.15.125:0/4244335606 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2804e170 con 0x7fbf28153d70 2023-10-30T12:44:11.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.431+0000 7fbf1dffb640 1 -- 172.21.15.125:0/4244335606 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbf2000a9b0 con 0x7fbf28153d70 2023-10-30T12:44:11.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.431+0000 7fbf1dffb640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbf0c0664d0 0x7fbf0c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:11.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.431+0000 7fbf1dffb640 1 -- 172.21.15.125:0/4244335606 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7fbf200841f0 con 0x7fbf28153d70 2023-10-30T12:44:11.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.431+0000 7fbf2d5e2640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbf0c0664d0 0x7fbf0c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:11.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.435+0000 7fbf2d5e2640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbf0c0664d0 0x7fbf0c068990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbf240079c0 tx=0x7fbf24008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:11.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.435+0000 7fbf1dffb640 1 -- 172.21.15.125:0/4244335606 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbf200513f0 con 0x7fbf28153d70 2023-10-30T12:44:11.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.615+0000 7fbf2f846640 1 -- 172.21.15.125:0/4244335606 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf28146610 con 0x7fbf28153d70 2023-10-30T12:44:11.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.615+0000 7fbf1dffb640 1 -- 172.21.15.125:0/4244335606 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fbf20054a10 con 0x7fbf28153d70 2023-10-30T12:44:11.616 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:11.616 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":79,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:06.704344+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:resolve","state_seq":20,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:11.616 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 79 2023-10-30T12:44:11.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.619+0000 7fbf2f846640 1 -- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbf0c0664d0 msgr2=0x7fbf0c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:11.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.619+0000 7fbf2f846640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbf0c0664d0 0x7fbf0c068990 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbf240079c0 tx=0x7fbf24008040 comp rx=0 tx=0).stop 2023-10-30T12:44:11.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.619+0000 7fbf2f846640 1 -- 172.21.15.125:0/4244335606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf28153d70 msgr2=0x7fbf280cc050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:11.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.619+0000 7fbf2f846640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf28153d70 0x7fbf280cc050 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000c780 tx=0x7fbf2000cc40 comp rx=0 tx=0).stop 2023-10-30T12:44:11.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.619+0000 7fbf2f846640 1 -- 172.21.15.125:0/4244335606 shutdown_connections 2023-10-30T12:44:11.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.619+0000 7fbf2f846640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbf0c0664d0 0x7fbf0c068990 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:11.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.619+0000 7fbf2f846640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf28153d70 0x7fbf280cc050 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:11.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.619+0000 7fbf2f846640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf280c8490 0x7fbf281427a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:11.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.619+0000 7fbf2f846640 1 --2- 172.21.15.125:0/4244335606 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf2806e470 0x7fbf281f6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:11.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.619+0000 7fbf2f846640 1 -- 172.21.15.125:0/4244335606 >> 172.21.15.125:0/4244335606 conn(0x7fbf28071ea0 msgr2=0x7fbf28144440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:11.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.619+0000 7fbf2f846640 1 -- 172.21.15.125:0/4244335606 shutdown_connections 2023-10-30T12:44:11.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:11.619+0000 7fbf2f846640 1 -- 172.21.15.125:0/4244335606 wait complete. 2023-10-30T12:44:11.636 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:44:13.637 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:13.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.783+0000 7fd65f8c9640 1 Processor -- start 2023-10-30T12:44:13.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.783+0000 7fd65f8c9640 1 -- start start 2023-10-30T12:44:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65f8c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd658070740 0x7fd6580c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65f8c9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6580c5810 con 0x7fd6581580e0 2023-10-30T12:44:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65f8c9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65815eb90 con 0x7fd65815e700 2023-10-30T12:44:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65f8c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd65806acb0 con 0x7fd658070740 2023-10-30T12:44:13.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65d665640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd6581580e0 0x7fd658069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54774/0 (socket says 172.21.15.125:54774) 2023-10-30T12:44:13.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65d665640 1 -- 172.21.15.125:0/1372579156 learned_addr learned my addr 172.21.15.125:0/1372579156 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:44:13.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65ce64640 1 --2- 172.21.15.125:0/1372579156 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd658070740 0x7fd6580c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:13.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 -- 172.21.15.125:0/1372579156 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551356509 0 0) 0x7fd6580c5810 con 0x7fd6581580e0 2023-10-30T12:44:13.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 -- 172.21.15.125:0/1372579156 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd644003530 con 0x7fd6581580e0 2023-10-30T12:44:13.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 -- 172.21.15.125:0/1372579156 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3319420515 0 0) 0x7fd644003530 con 0x7fd6581580e0 2023-10-30T12:44:13.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 -- 172.21.15.125:0/1372579156 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd658070740 msgr2=0x7fd6580c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:13.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 --2- 172.21.15.125:0/1372579156 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd658070740 0x7fd6580c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:13.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 -- 172.21.15.125:0/1372579156 >> v1:172.21.15.136:6789/0 conn(0x7fd65815e700 legacy=0x7fd65806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:13.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 -- 172.21.15.125:0/1372579156 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6580c6bb0 con 0x7fd6581580e0 2023-10-30T12:44:13.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 -- 172.21.15.125:0/1372579156 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fd654001680 con 0x7fd6581580e0 2023-10-30T12:44:13.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 -- 172.21.15.125:0/1372579156 >> v1:172.21.15.125:6789/0 conn(0x7fd6581580e0 legacy=0x7fd658069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:13.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 --2- 172.21.15.125:0/1372579156 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd658070740 0x7fd6440039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:13.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 --2- 172.21.15.125:0/1372579156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd644003f30 0x7fd6440047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:13.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 --2- 172.21.15.125:0/1372579156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd644004d10 0x7fd644005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:13.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 -- 172.21.15.125:0/1372579156 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64400ba40 con 0x7fd644004d10 2023-10-30T12:44:13.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 -- 172.21.15.125:0/1372579156 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64400bbb0 con 0x7fd644003f30 2023-10-30T12:44:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd64ffff640 1 -- 172.21.15.125:0/1372579156 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd64400bd70 con 0x7fd658070740 2023-10-30T12:44:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65de66640 1 --2- 172.21.15.125:0/1372579156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd644004d10 0x7fd644005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65d665640 1 --2- 172.21.15.125:0/1372579156 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd658070740 0x7fd6440039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65ce64640 1 --2- 172.21.15.125:0/1372579156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd644003f30 0x7fd6440047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:13.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65de66640 1 -- 172.21.15.125:0/1372579156 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd658070740 msgr2=0x7fd6440039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65de66640 1 --2- 172.21.15.125:0/1372579156 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd658070740 0x7fd6440039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65de66640 1 -- 172.21.15.125:0/1372579156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd644003f30 msgr2=0x7fd6440047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65de66640 1 --2- 172.21.15.125:0/1372579156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd644003f30 0x7fd6440047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65de66640 1 -- 172.21.15.125:0/1372579156 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd644003530 con 0x7fd644004d10 2023-10-30T12:44:13.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.787+0000 7fd65ce64640 1 --2- 172.21.15.125:0/1372579156 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd644003f30 0x7fd6440047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:44:13.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.791+0000 7fd65de66640 1 --2- 172.21.15.125:0/1372579156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd644004d10 0x7fd644005190 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd650001e70 tx=0x7fd65001c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0ea2cac989085bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:13.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.791+0000 7fd64ffff640 1 -- 172.21.15.125:0/1372579156 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd650003b80 con 0x7fd644004d10 2023-10-30T12:44:13.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.791+0000 7fd64ffff640 1 -- 172.21.15.125:0/1372579156 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd650003ce0 con 0x7fd644004d10 2023-10-30T12:44:13.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.791+0000 7fd64ffff640 1 -- 172.21.15.125:0/1372579156 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6500052c0 con 0x7fd644004d10 2023-10-30T12:44:13.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.791+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1372579156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd644004d10 msgr2=0x7fd644005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:13.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.791+0000 7fd65f8c9640 1 --2- 172.21.15.125:0/1372579156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd644004d10 0x7fd644005190 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fd650001e70 tx=0x7fd65001c760 comp rx=0 tx=0).stop 2023-10-30T12:44:13.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.791+0000 7fd65de66640 1 -- 172.21.15.125:0/1372579156 reap_dead start 2023-10-30T12:44:13.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.791+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1372579156 shutdown_connections 2023-10-30T12:44:13.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.791+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1372579156 >> 172.21.15.125:0/1372579156 conn(0x7fd6580c7910 msgr2=0x7fd6580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:13.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.791+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1372579156 shutdown_connections 2023-10-30T12:44:13.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.791+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1372579156 wait complete. 2023-10-30T12:44:13.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65f8c9640 1 Processor -- start 2023-10-30T12:44:13.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65f8c9640 1 -- start start 2023-10-30T12:44:13.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65f8c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd644004d10 0x7fd6580c10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:13.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65f8c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd65815a080 0x7fd65815a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:13.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65f8c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6581f68e0 0x7fd6581f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:13.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65f8c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6580c9c00 con 0x7fd6581f68e0 2023-10-30T12:44:13.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65f8c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6581595a0 con 0x7fd65815a080 2023-10-30T12:44:13.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65f8c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd65815aa40 con 0x7fd644004d10 2023-10-30T12:44:13.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65de66640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6581f68e0 0x7fd6581f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:13.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65de66640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6581f68e0 0x7fd6581f6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43406/0 (socket says 172.21.15.125:43406) 2023-10-30T12:44:13.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65de66640 1 -- 172.21.15.125:0/1876760787 learned_addr learned my addr 172.21.15.125:0/1876760787 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:13.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65d665640 1 --2- 172.21.15.125:0/1876760787 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd644004d10 0x7fd6580c10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:13.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65ce64640 1 --2- 172.21.15.125:0/1876760787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd65815a080 0x7fd65815a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:13.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65de66640 1 -- 172.21.15.125:0/1876760787 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd644004d10 msgr2=0x7fd6580c10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:13.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65de66640 1 --2- 172.21.15.125:0/1876760787 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd644004d10 0x7fd6580c10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:13.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65de66640 1 -- 172.21.15.125:0/1876760787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd65815a080 msgr2=0x7fd65815a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:13.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65de66640 1 --2- 172.21.15.125:0/1876760787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd65815a080 0x7fd65815a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:13.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65de66640 1 -- 172.21.15.125:0/1876760787 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6580c9920 con 0x7fd6581f68e0 2023-10-30T12:44:13.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65de66640 1 --2- 172.21.15.125:0/1876760787 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6581f68e0 0x7fd6581f6d60 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd650001e70 tx=0x7fd644005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:13.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd64dffb640 1 -- 172.21.15.125:0/1876760787 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd650003980 con 0x7fd6581f68e0 2023-10-30T12:44:13.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd64dffb640 1 -- 172.21.15.125:0/1876760787 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd650003ae0 con 0x7fd6581f68e0 2023-10-30T12:44:13.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd64dffb640 1 -- 172.21.15.125:0/1876760787 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6500a6660 con 0x7fd6581f68e0 2023-10-30T12:44:13.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1876760787 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6580c9f50 con 0x7fd6581f68e0 2023-10-30T12:44:13.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1876760787 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd658159b60 con 0x7fd6581f68e0 2023-10-30T12:44:13.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.795+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1876760787 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd658159d50 con 0x7fd6581f68e0 2023-10-30T12:44:13.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.799+0000 7fd64dffb640 1 -- 172.21.15.125:0/1876760787 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6500a67c0 con 0x7fd6581f68e0 2023-10-30T12:44:13.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.799+0000 7fd64dffb640 1 --2- 172.21.15.125:0/1876760787 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd63c0664d0 0x7fd63c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:13.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.799+0000 7fd64dffb640 1 -- 172.21.15.125:0/1876760787 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7fd65011da90 con 0x7fd6581f68e0 2023-10-30T12:44:13.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.799+0000 7fd65d665640 1 --2- 172.21.15.125:0/1876760787 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd63c0664d0 0x7fd63c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:13.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.799+0000 7fd65d665640 1 --2- 172.21.15.125:0/1876760787 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd63c0664d0 0x7fd63c068990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd654007d30 tx=0x7fd654009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:13.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.803+0000 7fd64dffb640 1 -- 172.21.15.125:0/1876760787 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6500eaad0 con 0x7fd6581f68e0 2023-10-30T12:44:13.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.983+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1876760787 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd65804e170 con 0x7fd6581f68e0 2023-10-30T12:44:13.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.983+0000 7fd64dffb640 1 -- 172.21.15.125:0/1876760787 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5740 (secure 0 0 0) 0x7fd6500ba070 con 0x7fd6581f68e0 2023-10-30T12:44:13.982 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:13.983 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:12.222588+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:13.983 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 80 2023-10-30T12:44:13.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.983+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1876760787 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd63c0664d0 msgr2=0x7fd63c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:13.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.983+0000 7fd65f8c9640 1 --2- 172.21.15.125:0/1876760787 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd63c0664d0 0x7fd63c068990 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd654007d30 tx=0x7fd654009040 comp rx=0 tx=0).stop 2023-10-30T12:44:13.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.983+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1876760787 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6581f68e0 msgr2=0x7fd6581f6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:13.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.983+0000 7fd65f8c9640 1 --2- 172.21.15.125:0/1876760787 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6581f68e0 0x7fd6581f6d60 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd650001e70 tx=0x7fd644005800 comp rx=0 tx=0).stop 2023-10-30T12:44:13.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.983+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1876760787 shutdown_connections 2023-10-30T12:44:13.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.983+0000 7fd65f8c9640 1 --2- 172.21.15.125:0/1876760787 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd63c0664d0 0x7fd63c068990 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:13.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.983+0000 7fd65f8c9640 1 --2- 172.21.15.125:0/1876760787 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6581f68e0 0x7fd6581f6d60 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:13.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.983+0000 7fd65f8c9640 1 --2- 172.21.15.125:0/1876760787 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd65815a080 0x7fd65815a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:13.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.983+0000 7fd65f8c9640 1 --2- 172.21.15.125:0/1876760787 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd644004d10 0x7fd6580c10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:13.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.983+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1876760787 >> 172.21.15.125:0/1876760787 conn(0x7fd6580c7910 msgr2=0x7fd6581560c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:13.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.983+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1876760787 shutdown_connections 2023-10-30T12:44:13.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:13.983+0000 7fd65f8c9640 1 -- 172.21.15.125:0/1876760787 wait complete. 2023-10-30T12:44:14.003 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 80, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 80, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5433": { "addr": "172.21.15.125:6836/2595438444", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2595438444, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2595438444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5433, "incarnation": 62, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 69 }, "gid_5988": { "addr": "172.21.15.125:6837/3156415951", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 3156415951, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 3156415951, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5988, "incarnation": 78, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:reconnect", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:44:12.222588+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5433, "mds_1": 5988 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:44:16.005 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:16.134 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:44:16.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:44:16.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.150+0000 7fb11030f640 1 Processor -- start 2023-10-30T12:44:16.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.150+0000 7fb11030f640 1 -- start start 2023-10-30T12:44:16.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb11030f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1081538a0 0x7fb10815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:16.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb11030f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb1080ca0e0 0x7fb10815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:16.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb11030f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1080cb800 con 0x7fb1080cf160 2023-10-30T12:44:16.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb11030f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb10815dc30 con 0x7fb1080ca0e0 2023-10-30T12:44:16.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb11030f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1080cb330 con 0x7fb1081538a0 2023-10-30T12:44:16.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb10e8ac640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb1080cf160 0x7fb1080cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54776/0 (socket says 172.21.15.125:54776) 2023-10-30T12:44:16.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb10e8ac640 1 -- 172.21.15.125:0/2452093523 learned_addr learned my addr 172.21.15.125:0/2452093523 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:44:16.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb10d8aa640 1 --2- 172.21.15.125:0/2452093523 >> v2:172.21.15.136:3300/0 conn(0x7fb1080ca0e0 0x7fb10815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:16.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb10e0ab640 1 --2- 172.21.15.125:0/2452093523 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1081538a0 0x7fb10815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:16.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb10d0a9640 1 -- 172.21.15.125:0/2452093523 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219172206 0 0) 0x7fb1080cb800 con 0x7fb1080cf160 2023-10-30T12:44:16.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb10d0a9640 1 -- 172.21.15.125:0/2452093523 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f4003530 con 0x7fb1080cf160 2023-10-30T12:44:16.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb10d8aa640 1 -- 172.21.15.125:0/2452093523 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1081538a0 msgr2=0x7fb10815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:16.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb10d8aa640 1 --2- 172.21.15.125:0/2452093523 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1081538a0 0x7fb10815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:16.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb10d8aa640 1 -- 172.21.15.125:0/2452093523 >> v1:172.21.15.125:6789/0 conn(0x7fb1080cf160 legacy=0x7fb1080cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:16.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb10d8aa640 1 -- 172.21.15.125:0/2452093523 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1080cad70 con 0x7fb1080ca0e0 2023-10-30T12:44:16.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.154+0000 7fb10e0ab640 1 --2- 172.21.15.125:0/2452093523 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1081538a0 0x7fb10815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:44:16.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.158+0000 7fb10d8aa640 1 --2- 172.21.15.125:0/2452093523 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1080ca0e0 0x7fb10815a090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb0f80097c0 tx=0x7fb0f801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2eb81d2bb864b4e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:16.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.158+0000 7fb10d0a9640 1 -- 172.21.15.125:0/2452093523 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f80a8970 con 0x7fb1080ca0e0 2023-10-30T12:44:16.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.158+0000 7fb10d0a9640 1 -- 172.21.15.125:0/2452093523 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0f80a8ad0 con 0x7fb1080ca0e0 2023-10-30T12:44:16.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.158+0000 7fb10d0a9640 1 -- 172.21.15.125:0/2452093523 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0f80a8e40 con 0x7fb1080ca0e0 2023-10-30T12:44:16.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.158+0000 7fb11030f640 1 -- 172.21.15.125:0/2452093523 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1080ca0e0 msgr2=0x7fb10815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:16.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.158+0000 7fb11030f640 1 --2- 172.21.15.125:0/2452093523 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1080ca0e0 0x7fb10815a090 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb0f80097c0 tx=0x7fb0f801ec70 comp rx=0 tx=0).stop 2023-10-30T12:44:16.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.158+0000 7fb11030f640 1 -- 172.21.15.125:0/2452093523 shutdown_connections 2023-10-30T12:44:16.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.158+0000 7fb11030f640 1 --2- 172.21.15.125:0/2452093523 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1080ca0e0 0x7fb10815a090 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:16.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.158+0000 7fb11030f640 1 --2- 172.21.15.125:0/2452093523 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1081538a0 0x7fb10815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:16.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.158+0000 7fb11030f640 1 -- 172.21.15.125:0/2452093523 >> 172.21.15.125:0/2452093523 conn(0x7fb10806bfe0 msgr2=0x7fb10806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:16.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.158+0000 7fb11030f640 1 -- 172.21.15.125:0/2452093523 shutdown_connections 2023-10-30T12:44:16.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.158+0000 7fb11030f640 1 -- 172.21.15.125:0/2452093523 wait complete. 2023-10-30T12:44:16.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb11030f640 1 Processor -- start 2023-10-30T12:44:16.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb11030f640 1 -- start start 2023-10-30T12:44:16.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb11030f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1080ca0e0 0x7fb10815cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:16.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb11030f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1080cf160 0x7fb10815d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:16.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb11030f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1081538a0 0x7fb10814ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:16.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb11030f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb108149980 con 0x7fb1080ca0e0 2023-10-30T12:44:16.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb11030f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1080c7690 con 0x7fb1080cf160 2023-10-30T12:44:16.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb11030f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1080c7930 con 0x7fb1081538a0 2023-10-30T12:44:16.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb10e0ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1080ca0e0 0x7fb10815cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:16.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb10e0ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1080ca0e0 0x7fb10815cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43416/0 (socket says 172.21.15.125:43416) 2023-10-30T12:44:16.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb10e0ab640 1 -- 172.21.15.125:0/353667551 learned_addr learned my addr 172.21.15.125:0/353667551 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:16.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb10d8aa640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1080cf160 0x7fb10815d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:16.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb10e8ac640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1081538a0 0x7fb10814ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:16.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb10e0ab640 1 -- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1081538a0 msgr2=0x7fb10814ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:16.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb10e0ab640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1081538a0 0x7fb10814ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:16.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb10e0ab640 1 -- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1080cf160 msgr2=0x7fb10815d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:16.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb10e0ab640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1080cf160 0x7fb10815d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:16.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb10e0ab640 1 -- 172.21.15.125:0/353667551 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb104009420 con 0x7fb1080ca0e0 2023-10-30T12:44:16.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb10e8ac640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1081538a0 0x7fb10814ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:44:16.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb10e0ab640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1080ca0e0 0x7fb10815cc90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fb10400ebe0 tx=0x7fb10400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:16.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb0feffd640 1 -- 172.21.15.125:0/353667551 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10400eee0 con 0x7fb1080ca0e0 2023-10-30T12:44:16.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb0feffd640 1 -- 172.21.15.125:0/353667551 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10400cc30 con 0x7fb1080ca0e0 2023-10-30T12:44:16.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb0feffd640 1 -- 172.21.15.125:0/353667551 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104013470 con 0x7fb1080ca0e0 2023-10-30T12:44:16.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb11030f640 1 -- 172.21.15.125:0/353667551 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f8009420 con 0x7fb1080ca0e0 2023-10-30T12:44:16.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.162+0000 7fb11030f640 1 -- 172.21.15.125:0/353667551 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1080c7ef0 con 0x7fb1080ca0e0 2023-10-30T12:44:16.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.166+0000 7fb11030f640 1 -- 172.21.15.125:0/353667551 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1080b2a50 con 0x7fb1080ca0e0 2023-10-30T12:44:16.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.166+0000 7fb0feffd640 1 -- 172.21.15.125:0/353667551 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb1040135d0 con 0x7fb1080ca0e0 2023-10-30T12:44:16.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.166+0000 7fb0feffd640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0ec0664a0 0x7fb0ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:16.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.166+0000 7fb0feffd640 1 -- 172.21.15.125:0/353667551 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7fb1040848d0 con 0x7fb1080ca0e0 2023-10-30T12:44:16.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.166+0000 7fb10d8aa640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0ec0664a0 0x7fb0ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:16.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.166+0000 7fb10d8aa640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0ec0664a0 0x7fb0ec068960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb0f80b2000 tx=0x7fb0f80aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:16.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.170+0000 7fb0feffd640 1 -- 172.21.15.125:0/353667551 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb1040519a0 con 0x7fb1080ca0e0 2023-10-30T12:44:16.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.354+0000 7fb11030f640 1 -- 172.21.15.125:0/353667551 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb10804e170 con 0x7fb1080ca0e0 2023-10-30T12:44:16.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.354+0000 7fb0feffd640 1 -- 172.21.15.125:0/353667551 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+5740 (secure 0 0 0) 0x7fb104054fc0 con 0x7fb1080ca0e0 2023-10-30T12:44:16.355 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:16.355 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:12.222588+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:16.356 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 80 2023-10-30T12:44:16.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.354+0000 7fb11030f640 1 -- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0ec0664a0 msgr2=0x7fb0ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:16.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.354+0000 7fb11030f640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0ec0664a0 0x7fb0ec068960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb0f80b2000 tx=0x7fb0f80aa070 comp rx=0 tx=0).stop 2023-10-30T12:44:16.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.354+0000 7fb11030f640 1 -- 172.21.15.125:0/353667551 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1080ca0e0 msgr2=0x7fb10815cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:16.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.354+0000 7fb11030f640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1080ca0e0 0x7fb10815cc90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fb10400ebe0 tx=0x7fb10400c520 comp rx=0 tx=0).stop 2023-10-30T12:44:16.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.354+0000 7fb11030f640 1 -- 172.21.15.125:0/353667551 shutdown_connections 2023-10-30T12:44:16.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.354+0000 7fb11030f640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0ec0664a0 0x7fb0ec068960 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:16.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.354+0000 7fb11030f640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1081538a0 0x7fb10814ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:16.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.354+0000 7fb11030f640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1080cf160 0x7fb10815d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:16.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.354+0000 7fb11030f640 1 --2- 172.21.15.125:0/353667551 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1080ca0e0 0x7fb10815cc90 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:16.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.354+0000 7fb11030f640 1 -- 172.21.15.125:0/353667551 >> 172.21.15.125:0/353667551 conn(0x7fb10806bfe0 msgr2=0x7fb1080cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:16.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.354+0000 7fb11030f640 1 -- 172.21.15.125:0/353667551 shutdown_connections 2023-10-30T12:44:16.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:16.358+0000 7fb11030f640 1 -- 172.21.15.125:0/353667551 wait complete. 2023-10-30T12:44:16.376 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:44:18.376 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:18.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.522+0000 7f173bed6640 1 Processor -- start 2023-10-30T12:44:18.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.522+0000 7f173bed6640 1 -- start start 2023-10-30T12:44:18.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.526+0000 7f173bed6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f17341570f0 0x7f173406ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:18.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.526+0000 7f173bed6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f173406b3b0 con 0x7f173406f370 2023-10-30T12:44:18.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.526+0000 7f173bed6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17340c38a0 con 0x7f17340b2300 2023-10-30T12:44:18.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.526+0000 7f173bed6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17340c3b20 con 0x7f17341570f0 2023-10-30T12:44:18.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.526+0000 7f1739c72640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f17341570f0 0x7f173406ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:18.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.526+0000 7f1739c72640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f17341570f0 0x7f173406ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43428/0 (socket says 172.21.15.125:43428) 2023-10-30T12:44:18.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.526+0000 7f173a473640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f173406f370 0x7f17341640d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54792/0 (socket says 172.21.15.125:54792) 2023-10-30T12:44:18.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.526+0000 7f1739c72640 1 -- 172.21.15.125:0/631303304 learned_addr learned my addr 172.21.15.125:0/631303304 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:18.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.526+0000 7f1739c72640 1 -- 172.21.15.125:0/631303304 >> v1:172.21.15.136:6789/0 conn(0x7f17340b2300 legacy=0x7f1734071b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:18.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.526+0000 7f1739c72640 1 -- 172.21.15.125:0/631303304 >> v1:172.21.15.125:6789/0 conn(0x7f173406f370 legacy=0x7f17341640d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:18.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.526+0000 7f1738c70640 1 -- 172.21.15.125:0/631303304 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270393300 0 0) 0x7f173406b3b0 con 0x7f173406f370 2023-10-30T12:44:18.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.526+0000 7f1739c72640 1 -- 172.21.15.125:0/631303304 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17340ca9f0 con 0x7f17341570f0 2023-10-30T12:44:18.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.530+0000 7f1739c72640 1 --2- 172.21.15.125:0/631303304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17341570f0 0x7f173406ca30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f1730009870 tx=0x7f173001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15288451596c4b6a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:18.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.530+0000 7f1738c70640 1 -- 172.21.15.125:0/631303304 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17300ab5f0 con 0x7f17341570f0 2023-10-30T12:44:18.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.530+0000 7f1738c70640 1 -- 172.21.15.125:0/631303304 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17300ab750 con 0x7f17341570f0 2023-10-30T12:44:18.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.530+0000 7f1738c70640 1 -- 172.21.15.125:0/631303304 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17300abae0 con 0x7f17341570f0 2023-10-30T12:44:18.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.530+0000 7f173bed6640 1 -- 172.21.15.125:0/631303304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17341570f0 msgr2=0x7f173406ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:18.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.530+0000 7f173bed6640 1 --2- 172.21.15.125:0/631303304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17341570f0 0x7f173406ca30 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f1730009870 tx=0x7f173001c500 comp rx=0 tx=0).stop 2023-10-30T12:44:18.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.530+0000 7f173bed6640 1 -- 172.21.15.125:0/631303304 shutdown_connections 2023-10-30T12:44:18.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.530+0000 7f173bed6640 1 --2- 172.21.15.125:0/631303304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17341570f0 0x7f173406ca30 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:18.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.530+0000 7f173bed6640 1 -- 172.21.15.125:0/631303304 >> 172.21.15.125:0/631303304 conn(0x7f1734069e90 msgr2=0x7f173406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:18.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.530+0000 7f173bed6640 1 -- 172.21.15.125:0/631303304 shutdown_connections 2023-10-30T12:44:18.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.530+0000 7f173bed6640 1 -- 172.21.15.125:0/631303304 wait complete. 2023-10-30T12:44:18.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.530+0000 7f173bed6640 1 Processor -- start 2023-10-30T12:44:18.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.530+0000 7f173bed6640 1 -- start start 2023-10-30T12:44:18.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173bed6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f173406f370 0x7f1734143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:18.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173bed6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f17340b2300 0x7f17341f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:18.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173bed6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17341570f0 0x7f17340c2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:18.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173bed6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17341f5a40 con 0x7f17341570f0 2023-10-30T12:44:18.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173bed6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17341482c0 con 0x7f173406f370 2023-10-30T12:44:18.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173bed6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1734147d00 con 0x7f17340b2300 2023-10-30T12:44:18.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173a473640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17341570f0 0x7f17340c2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:18.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173a473640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17341570f0 0x7f17340c2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43442/0 (socket says 172.21.15.125:43442) 2023-10-30T12:44:18.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173a473640 1 -- 172.21.15.125:0/1907131007 learned_addr learned my addr 172.21.15.125:0/1907131007 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:18.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f1739c72640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f173406f370 0x7f1734143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:18.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f1739471640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f17340b2300 0x7f17341f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:18.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173a473640 1 -- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f17340b2300 msgr2=0x7f17341f5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:18.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173a473640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f17340b2300 0x7f17341f5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:18.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173a473640 1 -- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f173406f370 msgr2=0x7f1734143bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:18.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173a473640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f173406f370 0x7f1734143bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:18.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173a473640 1 -- 172.21.15.125:0/1907131007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17300094d0 con 0x7f17341570f0 2023-10-30T12:44:18.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f1739471640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f17340b2300 0x7f17341f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:44:18.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f1739c72640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f173406f370 0x7f1734143bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:44:18.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173a473640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17341570f0 0x7f17340c2480 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f172c00c780 tx=0x7f172c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:18.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f172a7fc640 1 -- 172.21.15.125:0/1907131007 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c00ce90 con 0x7f17341570f0 2023-10-30T12:44:18.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f172a7fc640 1 -- 172.21.15.125:0/1907131007 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f172c00a640 con 0x7f17341570f0 2023-10-30T12:44:18.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173bed6640 1 -- 172.21.15.125:0/1907131007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1734147fe0 con 0x7f17341570f0 2023-10-30T12:44:18.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f172a7fc640 1 -- 172.21.15.125:0/1907131007 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f172c00a9d0 con 0x7f17341570f0 2023-10-30T12:44:18.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173bed6640 1 -- 172.21.15.125:0/1907131007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17341f6410 con 0x7f17341570f0 2023-10-30T12:44:18.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f173bed6640 1 -- 172.21.15.125:0/1907131007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17341f5bf0 con 0x7f17341570f0 2023-10-30T12:44:18.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f172a7fc640 1 -- 172.21.15.125:0/1907131007 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f172c00ab30 con 0x7f17341570f0 2023-10-30T12:44:18.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.534+0000 7f172a7fc640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f17180664a0 0x7f1718068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:18.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.538+0000 7f172a7fc640 1 -- 172.21.15.125:0/1907131007 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7f172c083650 con 0x7f17341570f0 2023-10-30T12:44:18.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.538+0000 7f1739c72640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f17180664a0 0x7f1718068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:18.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.538+0000 7f1739c72640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f17180664a0 0x7f1718068960 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f17300af000 tx=0x7f1730008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:18.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.538+0000 7f172a7fc640 1 -- 172.21.15.125:0/1907131007 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f17341f5bf0 con 0x7f17341570f0 2023-10-30T12:44:18.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.722+0000 7f173bed6640 1 -- 172.21.15.125:0/1907131007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f173406cf40 con 0x7f17341570f0 2023-10-30T12:44:18.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.722+0000 7f172a7fc640 1 -- 172.21.15.125:0/1907131007 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f172c018090 con 0x7f17341570f0 2023-10-30T12:44:18.722 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:18.722 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:17.334943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:18.723 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 81 2023-10-30T12:44:18.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.722+0000 7f173bed6640 1 -- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f17180664a0 msgr2=0x7f1718068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:18.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.722+0000 7f173bed6640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f17180664a0 0x7f1718068960 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f17300af000 tx=0x7f1730008040 comp rx=0 tx=0).stop 2023-10-30T12:44:18.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.722+0000 7f173bed6640 1 -- 172.21.15.125:0/1907131007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17341570f0 msgr2=0x7f17340c2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:18.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.722+0000 7f173bed6640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17341570f0 0x7f17340c2480 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f172c00c780 tx=0x7f172c00cc40 comp rx=0 tx=0).stop 2023-10-30T12:44:18.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.722+0000 7f173bed6640 1 -- 172.21.15.125:0/1907131007 shutdown_connections 2023-10-30T12:44:18.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.722+0000 7f173bed6640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f17180664a0 0x7f1718068960 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:18.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.722+0000 7f173bed6640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f17341570f0 0x7f17340c2480 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:18.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.722+0000 7f173bed6640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f17340b2300 0x7f17341f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:18.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.722+0000 7f173bed6640 1 --2- 172.21.15.125:0/1907131007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f173406f370 0x7f1734143bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:18.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.722+0000 7f173bed6640 1 -- 172.21.15.125:0/1907131007 >> 172.21.15.125:0/1907131007 conn(0x7f1734069e90 msgr2=0x7f17340c9010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:18.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.722+0000 7f173bed6640 1 -- 172.21.15.125:0/1907131007 shutdown_connections 2023-10-30T12:44:18.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:18.722+0000 7f173bed6640 1 -- 172.21.15.125:0/1907131007 wait complete. 2023-10-30T12:44:18.742 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:44:20.743 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:20.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f594a288640 1 Processor -- start 2023-10-30T12:44:20.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f594a288640 1 -- start start 2023-10-30T12:44:20.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f594a288640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f59441538f0 0x7f5944159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:20.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f594a288640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f59440cefa0 0x7f59440cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:20.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f594a288640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59440ca6a0 con 0x7f594415cd50 2023-10-30T12:44:20.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f594a288640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59440cf8f0 con 0x7f59441538f0 2023-10-30T12:44:20.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f594a288640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f594415d300 con 0x7f59440cefa0 2023-10-30T12:44:20.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f5949286640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f59441538f0 0x7f5944159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:20.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f5949286640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f59441538f0 0x7f5944159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33556/0 (socket says 172.21.15.125:33556) 2023-10-30T12:44:20.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f5949286640 1 -- 172.21.15.125:0/1542368608 learned_addr learned my addr 172.21.15.125:0/1542368608 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:20.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f5949a87640 1 --2- 172.21.15.125:0/1542368608 >> v2:172.21.15.136:3300/0 conn(0x7f59440cefa0 0x7f59440cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:20.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f5949286640 1 -- 172.21.15.125:0/1542368608 >> v2:172.21.15.136:3300/0 conn(0x7f59440cefa0 msgr2=0x7f59440cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:20.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f5949286640 1 --2- 172.21.15.125:0/1542368608 >> v2:172.21.15.136:3300/0 conn(0x7f59440cefa0 0x7f59440cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:20.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f5949286640 1 -- 172.21.15.125:0/1542368608 >> v1:172.21.15.136:6789/0 conn(0x7f594415cd50 legacy=0x7f59440c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:20.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f5949286640 1 -- 172.21.15.125:0/1542368608 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594415d5e0 con 0x7f59441538f0 2023-10-30T12:44:20.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.890+0000 7f5949a87640 1 --2- 172.21.15.125:0/1542368608 >> v2:172.21.15.136:3300/0 conn(0x7f59440cefa0 0x7f59440cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:44:20.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.894+0000 7f5949286640 1 --2- 172.21.15.125:0/1542368608 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59441538f0 0x7f5944159fb0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5944066a40 tx=0x7f594001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d42f45438634430 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:20.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.894+0000 7f593bfff640 1 -- 172.21.15.125:0/1542368608 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59400a89f0 con 0x7f59441538f0 2023-10-30T12:44:20.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.894+0000 7f593bfff640 1 -- 172.21.15.125:0/1542368608 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59400a8b50 con 0x7f59441538f0 2023-10-30T12:44:20.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.894+0000 7f593bfff640 1 -- 172.21.15.125:0/1542368608 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59400b0440 con 0x7f59441538f0 2023-10-30T12:44:20.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.894+0000 7f594a288640 1 -- 172.21.15.125:0/1542368608 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59441538f0 msgr2=0x7f5944159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:20.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.894+0000 7f594a288640 1 --2- 172.21.15.125:0/1542368608 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59441538f0 0x7f5944159fb0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f5944066a40 tx=0x7f594001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:44:20.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.894+0000 7f594a288640 1 -- 172.21.15.125:0/1542368608 shutdown_connections 2023-10-30T12:44:20.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.894+0000 7f594a288640 1 --2- 172.21.15.125:0/1542368608 >> v2:172.21.15.136:3300/0 conn(0x7f59440cefa0 0x7f59440cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:20.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.894+0000 7f594a288640 1 --2- 172.21.15.125:0/1542368608 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59441538f0 0x7f5944159fb0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:20.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.894+0000 7f594a288640 1 -- 172.21.15.125:0/1542368608 >> 172.21.15.125:0/1542368608 conn(0x7f594406a640 msgr2=0x7f5944065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:20.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.894+0000 7f594a288640 1 -- 172.21.15.125:0/1542368608 shutdown_connections 2023-10-30T12:44:20.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.894+0000 7f594a288640 1 -- 172.21.15.125:0/1542368608 wait complete. 2023-10-30T12:44:20.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.894+0000 7f594a288640 1 Processor -- start 2023-10-30T12:44:20.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f594a288640 1 -- start start 2023-10-30T12:44:20.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f594a288640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59440cefa0 0x7f5944148ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:20.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f594a288640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59441538f0 0x7f5944158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:20.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f594a288640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f594415cd50 0x7f59441575d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:20.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f594a288640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59441ef2c0 con 0x7f59440cefa0 2023-10-30T12:44:20.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f594a288640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59440b4af0 con 0x7f59441538f0 2023-10-30T12:44:20.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f594a288640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5944159850 con 0x7f594415cd50 2023-10-30T12:44:20.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f5949286640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59440cefa0 0x7f5944148ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:20.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f5949286640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59440cefa0 0x7f5944148ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33566/0 (socket says 172.21.15.125:33566) 2023-10-30T12:44:20.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f5949286640 1 -- 172.21.15.125:0/3082265416 learned_addr learned my addr 172.21.15.125:0/3082265416 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:20.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f5949a87640 1 --2- 172.21.15.125:0/3082265416 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f594415cd50 0x7f59441575d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:20.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f5948a85640 1 --2- 172.21.15.125:0/3082265416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59441538f0 0x7f5944158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:20.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f5949286640 1 -- 172.21.15.125:0/3082265416 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f594415cd50 msgr2=0x7f59441575d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:20.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f5949286640 1 --2- 172.21.15.125:0/3082265416 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f594415cd50 0x7f59441575d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:20.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f5949286640 1 -- 172.21.15.125:0/3082265416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59441538f0 msgr2=0x7f5944158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:20.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f5949286640 1 --2- 172.21.15.125:0/3082265416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59441538f0 0x7f5944158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:20.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f5949286640 1 -- 172.21.15.125:0/3082265416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59400094d0 con 0x7f59440cefa0 2023-10-30T12:44:20.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f5949286640 1 --2- 172.21.15.125:0/3082265416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59440cefa0 0x7f5944148ff0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f594001b900 tx=0x7f594001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:20.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f5939ffb640 1 -- 172.21.15.125:0/3082265416 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59400ba600 con 0x7f59440cefa0 2023-10-30T12:44:20.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f5939ffb640 1 -- 172.21.15.125:0/3082265416 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59400ba760 con 0x7f59440cefa0 2023-10-30T12:44:20.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f5939ffb640 1 -- 172.21.15.125:0/3082265416 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59400bab10 con 0x7f59440cefa0 2023-10-30T12:44:20.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f594a288640 1 -- 172.21.15.125:0/3082265416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5944149910 con 0x7f59440cefa0 2023-10-30T12:44:20.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f594a288640 1 -- 172.21.15.125:0/3082265416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59441f36c0 con 0x7f59440cefa0 2023-10-30T12:44:20.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.898+0000 7f594a288640 1 -- 172.21.15.125:0/3082265416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59441596a0 con 0x7f59440cefa0 2023-10-30T12:44:20.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.902+0000 7f5939ffb640 1 -- 172.21.15.125:0/3082265416 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f59400bac70 con 0x7f59440cefa0 2023-10-30T12:44:20.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.902+0000 7f5939ffb640 1 --2- 172.21.15.125:0/3082265416 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59280664a0 0x7f5928068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:20.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.902+0000 7f5939ffb640 1 -- 172.21.15.125:0/3082265416 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7f59401226d0 con 0x7f59440cefa0 2023-10-30T12:44:20.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.902+0000 7f5948a85640 1 --2- 172.21.15.125:0/3082265416 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59280664a0 0x7f5928068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:20.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.902+0000 7f5948a85640 1 --2- 172.21.15.125:0/3082265416 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59280664a0 0x7f5928068960 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f59440c5c10 tx=0x7f5934009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:20.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:20.906+0000 7f5939ffb640 1 -- 172.21.15.125:0/3082265416 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f59400ef8c0 con 0x7f59440cefa0 2023-10-30T12:44:21.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:21.094+0000 7f594a288640 1 -- 172.21.15.125:0/3082265416 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59441f3210 con 0x7f59440cefa0 2023-10-30T12:44:21.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:21.094+0000 7f5939ffb640 1 -- 172.21.15.125:0/3082265416 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f59400f2ee0 con 0x7f59440cefa0 2023-10-30T12:44:21.094 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:21.094 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:17.334943+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:rejoin","state_seq":24,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:21.095 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 81 2023-10-30T12:44:21.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:21.094+0000 7f594a288640 1 -- 172.21.15.125:0/3082265416 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59280664a0 msgr2=0x7f5928068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:21.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:21.094+0000 7f594a288640 1 --2- 172.21.15.125:0/3082265416 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59280664a0 0x7f5928068960 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f59440c5c10 tx=0x7f5934009040 comp rx=0 tx=0).stop 2023-10-30T12:44:21.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:21.094+0000 7f594a288640 1 -- 172.21.15.125:0/3082265416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59440cefa0 msgr2=0x7f5944148ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:21.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:21.094+0000 7f594a288640 1 --2- 172.21.15.125:0/3082265416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59440cefa0 0x7f5944148ff0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f594001b900 tx=0x7f594001ba90 comp rx=0 tx=0).stop 2023-10-30T12:44:21.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:21.094+0000 7f594a288640 1 -- 172.21.15.125:0/3082265416 shutdown_connections 2023-10-30T12:44:21.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:21.094+0000 7f594a288640 1 --2- 172.21.15.125:0/3082265416 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59280664a0 0x7f5928068960 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:21.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:21.094+0000 7f594a288640 1 --2- 172.21.15.125:0/3082265416 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f594415cd50 0x7f59441575d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:21.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:21.094+0000 7f594a288640 1 --2- 172.21.15.125:0/3082265416 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59441538f0 0x7f5944158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:21.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:21.094+0000 7f594a288640 1 --2- 172.21.15.125:0/3082265416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f59440cefa0 0x7f5944148ff0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:21.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:21.094+0000 7f594a288640 1 -- 172.21.15.125:0/3082265416 >> 172.21.15.125:0/3082265416 conn(0x7f594406a640 msgr2=0x7f59440c47c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:21.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:21.094+0000 7f594a288640 1 -- 172.21.15.125:0/3082265416 shutdown_connections 2023-10-30T12:44:21.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:21.094+0000 7f594a288640 1 -- 172.21.15.125:0/3082265416 wait complete. 2023-10-30T12:44:21.115 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:44:23.116 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:23.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.278+0000 7f999ae2f640 1 Processor -- start 2023-10-30T12:44:23.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.278+0000 7f999ae2f640 1 -- start start 2023-10-30T12:44:23.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.282+0000 7f999ae2f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f999415cd50 0x7f99940c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:23.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.282+0000 7f999ae2f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f99940cefa0 0x7f99940cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:23.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.282+0000 7f999ae2f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f999415a6c0 con 0x7f99941538f0 2023-10-30T12:44:23.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.282+0000 7f999ae2f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99940cf8f0 con 0x7f99940cefa0 2023-10-30T12:44:23.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.282+0000 7f999ae2f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f999415d300 con 0x7f999415cd50 2023-10-30T12:44:23.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.282+0000 7f999a62e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f99940cefa0 0x7f99940cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:23.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.282+0000 7f9999e2d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f99941538f0 0x7f9994159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46400/0 (socket says 172.21.15.125:46400) 2023-10-30T12:44:23.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.282+0000 7f9999e2d640 1 -- 172.21.15.125:0/478660550 learned_addr learned my addr 172.21.15.125:0/478660550 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:44:23.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.282+0000 7f999962c640 1 --2- 172.21.15.125:0/478660550 >> v2:172.21.15.136:3300/0 conn(0x7f999415cd50 0x7f99940c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:23.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.282+0000 7f999a62e640 1 -- 172.21.15.125:0/478660550 >> v2:172.21.15.136:3300/0 conn(0x7f999415cd50 msgr2=0x7f99940c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:23.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.282+0000 7f999a62e640 1 --2- 172.21.15.125:0/478660550 >> v2:172.21.15.136:3300/0 conn(0x7f999415cd50 0x7f99940c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:23.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.282+0000 7f999a62e640 1 -- 172.21.15.125:0/478660550 >> v1:172.21.15.125:6789/0 conn(0x7f99941538f0 legacy=0x7f9994159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:23.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.282+0000 7f9998e2b640 1 -- 172.21.15.125:0/478660550 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163973798 0 0) 0x7f999415a6c0 con 0x7f99941538f0 2023-10-30T12:44:23.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.282+0000 7f999a62e640 1 -- 172.21.15.125:0/478660550 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999415d5e0 con 0x7f99940cefa0 2023-10-30T12:44:23.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999a62e640 1 --2- 172.21.15.125:0/478660550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99940cefa0 0x7f99940cf3b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9994072e90 tx=0x7f998c01f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11d7e80aa46d0b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:23.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f9998e2b640 1 -- 172.21.15.125:0/478660550 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c0b0440 con 0x7f99940cefa0 2023-10-30T12:44:23.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f9998e2b640 1 -- 172.21.15.125:0/478660550 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998c0b05a0 con 0x7f99940cefa0 2023-10-30T12:44:23.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f9998e2b640 1 -- 172.21.15.125:0/478660550 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f998c0b0930 con 0x7f99940cefa0 2023-10-30T12:44:23.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 -- 172.21.15.125:0/478660550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99940cefa0 msgr2=0x7f99940cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:23.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 --2- 172.21.15.125:0/478660550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99940cefa0 0x7f99940cf3b0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9994072e90 tx=0x7f998c01f0f0 comp rx=0 tx=0).stop 2023-10-30T12:44:23.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 -- 172.21.15.125:0/478660550 shutdown_connections 2023-10-30T12:44:23.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 --2- 172.21.15.125:0/478660550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99940cefa0 0x7f99940cf3b0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:23.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 --2- 172.21.15.125:0/478660550 >> v2:172.21.15.136:3300/0 conn(0x7f999415cd50 0x7f99940c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:23.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 -- 172.21.15.125:0/478660550 >> 172.21.15.125:0/478660550 conn(0x7f999406a640 msgr2=0x7f9994065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:23.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 -- 172.21.15.125:0/478660550 shutdown_connections 2023-10-30T12:44:23.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 -- 172.21.15.125:0/478660550 wait complete. 2023-10-30T12:44:23.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 Processor -- start 2023-10-30T12:44:23.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 -- start start 2023-10-30T12:44:23.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99940cefa0 0x7f99941452c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:23.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99941538f0 0x7f9994151120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:23.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f999415cd50 0x7f9994152a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:23.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f999415cb40 con 0x7f99940cefa0 2023-10-30T12:44:23.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99940b3f30 con 0x7f99941538f0 2023-10-30T12:44:23.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.286+0000 7f999ae2f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99940b4d60 con 0x7f999415cd50 2023-10-30T12:44:23.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f9999e2d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99940cefa0 0x7f99941452c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:23.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f9999e2d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99940cefa0 0x7f99941452c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33582/0 (socket says 172.21.15.125:33582) 2023-10-30T12:44:23.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f9999e2d640 1 -- 172.21.15.125:0/861825328 learned_addr learned my addr 172.21.15.125:0/861825328 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:23.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f999a62e640 1 --2- 172.21.15.125:0/861825328 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f999415cd50 0x7f9994152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:23.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f999962c640 1 --2- 172.21.15.125:0/861825328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99941538f0 0x7f9994151120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:23.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f9999e2d640 1 -- 172.21.15.125:0/861825328 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f999415cd50 msgr2=0x7f9994152a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:23.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f9999e2d640 1 --2- 172.21.15.125:0/861825328 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f999415cd50 0x7f9994152a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:23.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f9999e2d640 1 -- 172.21.15.125:0/861825328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99941538f0 msgr2=0x7f9994151120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:23.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f9999e2d640 1 --2- 172.21.15.125:0/861825328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99941538f0 0x7f9994151120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:23.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f9999e2d640 1 -- 172.21.15.125:0/861825328 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c009a70 con 0x7f99940cefa0 2023-10-30T12:44:23.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f9999e2d640 1 --2- 172.21.15.125:0/861825328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99940cefa0 0x7f99941452c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f99900011f0 tx=0x7f999000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:23.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f998a7fc640 1 -- 172.21.15.125:0/861825328 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99900074e0 con 0x7f99940cefa0 2023-10-30T12:44:23.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f998a7fc640 1 -- 172.21.15.125:0/861825328 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9990007640 con 0x7f99940cefa0 2023-10-30T12:44:23.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f998a7fc640 1 -- 172.21.15.125:0/861825328 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9990004a10 con 0x7f99940cefa0 2023-10-30T12:44:23.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f999ae2f640 1 -- 172.21.15.125:0/861825328 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9994144e30 con 0x7f99940cefa0 2023-10-30T12:44:23.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f999ae2f640 1 -- 172.21.15.125:0/861825328 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f999415c600 con 0x7f99940cefa0 2023-10-30T12:44:23.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f999ae2f640 1 -- 172.21.15.125:0/861825328 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999415c8c0 con 0x7f99940cefa0 2023-10-30T12:44:23.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f998a7fc640 1 -- 172.21.15.125:0/861825328 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9990004b70 con 0x7f99940cefa0 2023-10-30T12:44:23.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.290+0000 7f998a7fc640 1 --2- 172.21.15.125:0/861825328 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99780664a0 0x7f9978068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:23.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.294+0000 7f998a7fc640 1 -- 172.21.15.125:0/861825328 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7f9990083330 con 0x7f99940cefa0 2023-10-30T12:44:23.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.294+0000 7f999962c640 1 --2- 172.21.15.125:0/861825328 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99780664a0 0x7f9978068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:23.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.294+0000 7f999962c640 1 --2- 172.21.15.125:0/861825328 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99780664a0 0x7f9978068960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f99940b1890 tx=0x7f9984008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:23.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.294+0000 7f998a7fc640 1 -- 172.21.15.125:0/861825328 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f99900505c0 con 0x7f99940cefa0 2023-10-30T12:44:23.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.482+0000 7f999ae2f640 1 -- 172.21.15.125:0/861825328 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f999404e170 con 0x7f99940cefa0 2023-10-30T12:44:23.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.482+0000 7f998a7fc640 1 -- 172.21.15.125:0/861825328 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f9990018090 con 0x7f99940cefa0 2023-10-30T12:44:23.483 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:23.484 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:23.258648+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:23.484 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 82 2023-10-30T12:44:23.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.482+0000 7f999ae2f640 1 -- 172.21.15.125:0/861825328 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99780664a0 msgr2=0x7f9978068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:23.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.482+0000 7f999ae2f640 1 --2- 172.21.15.125:0/861825328 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99780664a0 0x7f9978068960 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f99940b1890 tx=0x7f9984008040 comp rx=0 tx=0).stop 2023-10-30T12:44:23.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.486+0000 7f999ae2f640 1 -- 172.21.15.125:0/861825328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99940cefa0 msgr2=0x7f99941452c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:23.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.486+0000 7f999ae2f640 1 --2- 172.21.15.125:0/861825328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99940cefa0 0x7f99941452c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f99900011f0 tx=0x7f999000c9b0 comp rx=0 tx=0).stop 2023-10-30T12:44:23.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.486+0000 7f999ae2f640 1 -- 172.21.15.125:0/861825328 shutdown_connections 2023-10-30T12:44:23.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.486+0000 7f999ae2f640 1 --2- 172.21.15.125:0/861825328 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99780664a0 0x7f9978068960 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:23.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.486+0000 7f999ae2f640 1 --2- 172.21.15.125:0/861825328 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f999415cd50 0x7f9994152a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:23.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.486+0000 7f999ae2f640 1 --2- 172.21.15.125:0/861825328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99941538f0 0x7f9994151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:23.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.486+0000 7f999ae2f640 1 --2- 172.21.15.125:0/861825328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99940cefa0 0x7f99941452c0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:23.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.486+0000 7f999ae2f640 1 -- 172.21.15.125:0/861825328 >> 172.21.15.125:0/861825328 conn(0x7f999406a640 msgr2=0x7f99940c5120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:23.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.486+0000 7f999ae2f640 1 -- 172.21.15.125:0/861825328 shutdown_connections 2023-10-30T12:44:23.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:23.486+0000 7f999ae2f640 1 -- 172.21.15.125:0/861825328 wait complete. 2023-10-30T12:44:23.504 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:44:25.505 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:25.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.650+0000 7fb590701640 1 Processor -- start 2023-10-30T12:44:25.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.650+0000 7fb590701640 1 -- start start 2023-10-30T12:44:25.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.654+0000 7fb590701640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb588069e90 0x7fb58815a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:25.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.654+0000 7fb590701640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb58806cb80 0x7fb58806cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:25.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.654+0000 7fb590701640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb58806d530 con 0x7fb58806a270 2023-10-30T12:44:25.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.654+0000 7fb590701640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5880c8570 con 0x7fb58806cb80 2023-10-30T12:44:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.654+0000 7fb590701640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5880c6c10 con 0x7fb588069e90 2023-10-30T12:44:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.654+0000 7fb58ec9e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb58806cb80 0x7fb58806cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.654+0000 7fb58ec9e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb58806cb80 0x7fb58806cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33588/0 (socket says 172.21.15.125:33588) 2023-10-30T12:44:25.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.654+0000 7fb58ec9e640 1 -- 172.21.15.125:0/4022076600 learned_addr learned my addr 172.21.15.125:0/4022076600 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:25.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.654+0000 7fb58e49d640 1 --2- 172.21.15.125:0/4022076600 >> v2:172.21.15.136:3300/0 conn(0x7fb588069e90 0x7fb58815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.654+0000 7fb58ec9e640 1 -- 172.21.15.125:0/4022076600 >> v2:172.21.15.136:3300/0 conn(0x7fb588069e90 msgr2=0x7fb58815a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.654+0000 7fb58ec9e640 1 --2- 172.21.15.125:0/4022076600 >> v2:172.21.15.136:3300/0 conn(0x7fb588069e90 0x7fb58815a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.654+0000 7fb58ec9e640 1 -- 172.21.15.125:0/4022076600 >> v1:172.21.15.125:6789/0 conn(0x7fb58806a270 legacy=0x7fb5880c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:25.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.654+0000 7fb58ec9e640 1 -- 172.21.15.125:0/4022076600 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5880c6ef0 con 0x7fb58806cb80 2023-10-30T12:44:25.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.654+0000 7fb58d49b640 1 -- 172.21.15.125:0/4022076600 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16745934 0 0) 0x7fb58806d530 con 0x7fb58806a270 2023-10-30T12:44:25.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb58ec9e640 1 --2- 172.21.15.125:0/4022076600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb58806cb80 0x7fb58806cff0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb58806ed20 tx=0x7fb58001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8041b9bc350d3372 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:25.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb58d49b640 1 -- 172.21.15.125:0/4022076600 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5800a5780 con 0x7fb58806cb80 2023-10-30T12:44:25.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb58d49b640 1 -- 172.21.15.125:0/4022076600 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5800a58e0 con 0x7fb58806cb80 2023-10-30T12:44:25.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb58d49b640 1 -- 172.21.15.125:0/4022076600 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5800a5c70 con 0x7fb58806cb80 2023-10-30T12:44:25.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 -- 172.21.15.125:0/4022076600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb58806cb80 msgr2=0x7fb58806cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:25.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 --2- 172.21.15.125:0/4022076600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb58806cb80 0x7fb58806cff0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fb58806ed20 tx=0x7fb58001c9d0 comp rx=0 tx=0).stop 2023-10-30T12:44:25.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 -- 172.21.15.125:0/4022076600 shutdown_connections 2023-10-30T12:44:25.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 --2- 172.21.15.125:0/4022076600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb58806cb80 0x7fb58806cff0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:25.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 --2- 172.21.15.125:0/4022076600 >> v2:172.21.15.136:3300/0 conn(0x7fb588069e90 0x7fb58815a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:25.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 -- 172.21.15.125:0/4022076600 >> 172.21.15.125:0/4022076600 conn(0x7fb58806dfd0 msgr2=0x7fb58806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:25.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 -- 172.21.15.125:0/4022076600 shutdown_connections 2023-10-30T12:44:25.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 -- 172.21.15.125:0/4022076600 wait complete. 2023-10-30T12:44:25.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 Processor -- start 2023-10-30T12:44:25.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 -- start start 2023-10-30T12:44:25.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb588069e90 0x7fb5881f5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:25.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb58806a270 0x7fb58806b200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:25.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb58806cb80 0x7fb588158eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:25.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb588146330 con 0x7fb58806a270 2023-10-30T12:44:25.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5881f57b0 con 0x7fb58806cb80 2023-10-30T12:44:25.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.658+0000 7fb590701640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb588159870 con 0x7fb588069e90 2023-10-30T12:44:25.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb58dc9c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb58806a270 0x7fb58806b200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:25.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb58dc9c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb58806a270 0x7fb58806b200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33596/0 (socket says 172.21.15.125:33596) 2023-10-30T12:44:25.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb58dc9c640 1 -- 172.21.15.125:0/410730308 learned_addr learned my addr 172.21.15.125:0/410730308 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:25.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb58e49d640 1 --2- 172.21.15.125:0/410730308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb588069e90 0x7fb5881f5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:25.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb58ec9e640 1 --2- 172.21.15.125:0/410730308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb58806cb80 0x7fb588158eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:25.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb58dc9c640 1 -- 172.21.15.125:0/410730308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb588069e90 msgr2=0x7fb5881f5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:25.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb58dc9c640 1 --2- 172.21.15.125:0/410730308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb588069e90 0x7fb5881f5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:25.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb58dc9c640 1 -- 172.21.15.125:0/410730308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb58806cb80 msgr2=0x7fb588158eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:25.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb58dc9c640 1 --2- 172.21.15.125:0/410730308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb58806cb80 0x7fb588158eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:25.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb58dc9c640 1 -- 172.21.15.125:0/410730308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb580009b30 con 0x7fb58806a270 2023-10-30T12:44:25.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb58dc9c640 1 --2- 172.21.15.125:0/410730308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb58806a270 0x7fb58806b200 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb57800c4d0 tx=0x7fb57800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:25.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb57effd640 1 -- 172.21.15.125:0/410730308 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb578005480 con 0x7fb58806a270 2023-10-30T12:44:25.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb57effd640 1 -- 172.21.15.125:0/410730308 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb578003600 con 0x7fb58806a270 2023-10-30T12:44:25.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb590701640 1 -- 172.21.15.125:0/410730308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb588147f30 con 0x7fb58806a270 2023-10-30T12:44:25.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb57effd640 1 -- 172.21.15.125:0/410730308 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5780039b0 con 0x7fb58806a270 2023-10-30T12:44:25.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb590701640 1 -- 172.21.15.125:0/410730308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5881484b0 con 0x7fb58806a270 2023-10-30T12:44:25.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb590701640 1 -- 172.21.15.125:0/410730308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb58804e170 con 0x7fb58806a270 2023-10-30T12:44:25.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.662+0000 7fb57effd640 1 -- 172.21.15.125:0/410730308 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb578003b10 con 0x7fb58806a270 2023-10-30T12:44:25.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.666+0000 7fb57effd640 1 --2- 172.21.15.125:0/410730308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb56c0664a0 0x7fb56c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:25.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.666+0000 7fb57effd640 1 -- 172.21.15.125:0/410730308 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7fb578083630 con 0x7fb58806a270 2023-10-30T12:44:25.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.666+0000 7fb58e49d640 1 --2- 172.21.15.125:0/410730308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb56c0664a0 0x7fb56c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:25.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.666+0000 7fb58e49d640 1 --2- 172.21.15.125:0/410730308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb56c0664a0 0x7fb56c068960 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb5840079c0 tx=0x7fb584008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:25.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.670+0000 7fb57effd640 1 -- 172.21.15.125:0/410730308 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb5780509a0 con 0x7fb58806a270 2023-10-30T12:44:25.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.846+0000 7fb590701640 1 -- 172.21.15.125:0/410730308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb58806ef60 con 0x7fb58806a270 2023-10-30T12:44:25.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.846+0000 7fb57effd640 1 -- 172.21.15.125:0/410730308 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5743 (secure 0 0 0) 0x7fb578017090 con 0x7fb58806a270 2023-10-30T12:44:25.849 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:25.850 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:23.258648+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:25.852 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 82 2023-10-30T12:44:25.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.850+0000 7fb590701640 1 -- 172.21.15.125:0/410730308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb56c0664a0 msgr2=0x7fb56c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:25.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.850+0000 7fb590701640 1 --2- 172.21.15.125:0/410730308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb56c0664a0 0x7fb56c068960 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb5840079c0 tx=0x7fb584008040 comp rx=0 tx=0).stop 2023-10-30T12:44:25.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.850+0000 7fb590701640 1 -- 172.21.15.125:0/410730308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb58806a270 msgr2=0x7fb58806b200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:25.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.850+0000 7fb590701640 1 --2- 172.21.15.125:0/410730308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb58806a270 0x7fb58806b200 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fb57800c4d0 tx=0x7fb57800c990 comp rx=0 tx=0).stop 2023-10-30T12:44:25.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.850+0000 7fb590701640 1 -- 172.21.15.125:0/410730308 shutdown_connections 2023-10-30T12:44:25.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.850+0000 7fb590701640 1 --2- 172.21.15.125:0/410730308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb56c0664a0 0x7fb56c068960 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:25.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.850+0000 7fb590701640 1 --2- 172.21.15.125:0/410730308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb58806cb80 0x7fb588158eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:25.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.850+0000 7fb590701640 1 --2- 172.21.15.125:0/410730308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb58806a270 0x7fb58806b200 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:25.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.850+0000 7fb590701640 1 --2- 172.21.15.125:0/410730308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb588069e90 0x7fb5881f5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:25.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.850+0000 7fb590701640 1 -- 172.21.15.125:0/410730308 >> 172.21.15.125:0/410730308 conn(0x7fb58806dfd0 msgr2=0x7fb58806cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:25.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.850+0000 7fb590701640 1 -- 172.21.15.125:0/410730308 shutdown_connections 2023-10-30T12:44:25.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:25.850+0000 7fb590701640 1 -- 172.21.15.125:0/410730308 wait complete. 2023-10-30T12:44:25.871 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 82, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 82, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5433": { "addr": "172.21.15.125:6836/2595438444", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2595438444, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2595438444, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 5433, "incarnation": 62, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 69 }, "gid_5988": { "addr": "172.21.15.125:6837/3156415951", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 3156415951, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 3156415951, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5988, "incarnation": 78, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:clientreplay", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:44:23.258648+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5433, "mds_1": 5988 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:44:27.872 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:28.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.018+0000 7ff62b7c4640 1 Processor -- start 2023-10-30T12:44:28.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.018+0000 7ff62b7c4640 1 -- start start 2023-10-30T12:44:28.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff62b7c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6240c6b80 0x7ff6240bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:28.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff62b7c4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff62413f4a0 0x7ff6240c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:28.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff62b7c4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6240c0a50 con 0x7ff6240c6f60 2023-10-30T12:44:28.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff62b7c4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff62406f6e0 con 0x7ff62413f4a0 2023-10-30T12:44:28.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff62b7c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff62406f9c0 con 0x7ff6240c6b80 2023-10-30T12:44:28.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff629d61640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff62413f4a0 0x7ff6240c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:28.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff629560640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6240c6b80 0x7ff6240bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:28.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff629d61640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff62413f4a0 0x7ff6240c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:50706/0 (socket says 172.21.15.125:50706) 2023-10-30T12:44:28.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff629d61640 1 -- 172.21.15.125:0/990473038 learned_addr learned my addr 172.21.15.125:0/990473038 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:28.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff628d5f640 1 --1- 172.21.15.125:0/990473038 >> v1:172.21.15.136:6789/0 conn(0x7ff6240c6f60 0x7ff62413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:42968/0 (socket says 172.21.15.125:42968) 2023-10-30T12:44:28.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff629560640 1 -- 172.21.15.125:0/990473038 >> v2:172.21.15.136:3300/0 conn(0x7ff62413f4a0 msgr2=0x7ff6240c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:28.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff629560640 1 --2- 172.21.15.125:0/990473038 >> v2:172.21.15.136:3300/0 conn(0x7ff62413f4a0 0x7ff6240c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:28.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff629560640 1 -- 172.21.15.125:0/990473038 >> v1:172.21.15.136:6789/0 conn(0x7ff6240c6f60 legacy=0x7ff62413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:28.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff629560640 1 -- 172.21.15.125:0/990473038 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62406fca0 con 0x7ff6240c6b80 2023-10-30T12:44:28.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.022+0000 7ff629d61640 1 --2- 172.21.15.125:0/990473038 >> v2:172.21.15.136:3300/0 conn(0x7ff62413f4a0 0x7ff6240c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:44:28.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff629560640 1 --2- 172.21.15.125:0/990473038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6240c6b80 0x7ff6240bfec0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff6200097c0 tx=0x7ff62001c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ae266561fd035bc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:28.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff61bfff640 1 -- 172.21.15.125:0/990473038 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6200ad640 con 0x7ff6240c6b80 2023-10-30T12:44:28.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff61bfff640 1 -- 172.21.15.125:0/990473038 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6200ad7a0 con 0x7ff6240c6b80 2023-10-30T12:44:28.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff61bfff640 1 -- 172.21.15.125:0/990473038 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6200adb30 con 0x7ff6240c6b80 2023-10-30T12:44:28.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 -- 172.21.15.125:0/990473038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6240c6b80 msgr2=0x7ff6240bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:28.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 --2- 172.21.15.125:0/990473038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6240c6b80 0x7ff6240bfec0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff6200097c0 tx=0x7ff62001c480 comp rx=0 tx=0).stop 2023-10-30T12:44:28.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 -- 172.21.15.125:0/990473038 shutdown_connections 2023-10-30T12:44:28.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 --2- 172.21.15.125:0/990473038 >> v2:172.21.15.136:3300/0 conn(0x7ff62413f4a0 0x7ff6240c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:28.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 --2- 172.21.15.125:0/990473038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6240c6b80 0x7ff6240bfec0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:28.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 -- 172.21.15.125:0/990473038 >> 172.21.15.125:0/990473038 conn(0x7ff6240c7910 msgr2=0x7ff6240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:28.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 -- 172.21.15.125:0/990473038 shutdown_connections 2023-10-30T12:44:28.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 -- 172.21.15.125:0/990473038 wait complete. 2023-10-30T12:44:28.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 Processor -- start 2023-10-30T12:44:28.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 -- start start 2023-10-30T12:44:28.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6240c6b80 0x7ff6241f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:28.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6240c6f60 0x7ff6241f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:28.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff62413f4a0 0x7ff6240c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:28.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6241564f0 con 0x7ff6240c6b80 2023-10-30T12:44:28.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff624155fa0 con 0x7ff6240c6f60 2023-10-30T12:44:28.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.026+0000 7ff62b7c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6240c8db0 con 0x7ff62413f4a0 2023-10-30T12:44:28.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff629560640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6240c6b80 0x7ff6241f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:28.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff629560640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6240c6b80 0x7ff6241f5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33602/0 (socket says 172.21.15.125:33602) 2023-10-30T12:44:28.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff629560640 1 -- 172.21.15.125:0/2988424007 learned_addr learned my addr 172.21.15.125:0/2988424007 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:28.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff629d61640 1 --2- 172.21.15.125:0/2988424007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff62413f4a0 0x7ff6240c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:28.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff628d5f640 1 --2- 172.21.15.125:0/2988424007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6240c6f60 0x7ff6241f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:28.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff629560640 1 -- 172.21.15.125:0/2988424007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff62413f4a0 msgr2=0x7ff6240c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:28.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff629560640 1 --2- 172.21.15.125:0/2988424007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff62413f4a0 0x7ff6240c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:28.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff629560640 1 -- 172.21.15.125:0/2988424007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6240c6f60 msgr2=0x7ff6241f60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:28.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff629560640 1 --2- 172.21.15.125:0/2988424007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6240c6f60 0x7ff6241f60b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:28.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff629560640 1 -- 172.21.15.125:0/2988424007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c009b30 con 0x7ff6240c6b80 2023-10-30T12:44:28.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff629560640 1 --2- 172.21.15.125:0/2988424007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6240c6b80 0x7ff6241f5b70 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff6200b0040 tx=0x7ff620019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:28.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff619ffb640 1 -- 172.21.15.125:0/2988424007 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6200b6820 con 0x7ff6240c6b80 2023-10-30T12:44:28.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff619ffb640 1 -- 172.21.15.125:0/2988424007 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6200a6040 con 0x7ff6240c6b80 2023-10-30T12:44:28.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff62b7c4640 1 -- 172.21.15.125:0/2988424007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff620009420 con 0x7ff6240c6b80 2023-10-30T12:44:28.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff619ffb640 1 -- 172.21.15.125:0/2988424007 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6200b6be0 con 0x7ff6240c6b80 2023-10-30T12:44:28.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff62b7c4640 1 -- 172.21.15.125:0/2988424007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6240c9050 con 0x7ff6240c6b80 2023-10-30T12:44:28.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff62b7c4640 1 -- 172.21.15.125:0/2988424007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6240c8c10 con 0x7ff6240c6b80 2023-10-30T12:44:28.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.030+0000 7ff619ffb640 1 -- 172.21.15.125:0/2988424007 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff6200b4040 con 0x7ff6240c6b80 2023-10-30T12:44:28.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.034+0000 7ff619ffb640 1 --2- 172.21.15.125:0/2988424007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6080664a0 0x7ff608068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:28.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.034+0000 7ff619ffb640 1 -- 172.21.15.125:0/2988424007 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7ff62011e4c0 con 0x7ff6240c6b80 2023-10-30T12:44:28.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.034+0000 7ff628d5f640 1 --2- 172.21.15.125:0/2988424007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6080664a0 0x7ff608068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:28.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.034+0000 7ff628d5f640 1 --2- 172.21.15.125:0/2988424007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6080664a0 0x7ff608068960 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff62413e910 tx=0x7ff614007cb0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:28.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.034+0000 7ff619ffb640 1 -- 172.21.15.125:0/2988424007 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff6240c8c10 con 0x7ff6240c6b80 2023-10-30T12:44:28.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.218+0000 7ff62b7c4640 1 -- 172.21.15.125:0/2988424007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff6240c8c10 con 0x7ff6240c6b80 2023-10-30T12:44:28.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.218+0000 7ff619ffb640 1 -- 172.21.15.125:0/2988424007 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+5743 (secure 0 0 0) 0x7ff6240c8c10 con 0x7ff6240c6b80 2023-10-30T12:44:28.221 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:28.221 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:23.258648+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:clientreplay","state_seq":26,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:28.223 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 82 2023-10-30T12:44:28.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.222+0000 7ff62b7c4640 1 -- 172.21.15.125:0/2988424007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6080664a0 msgr2=0x7ff608068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:28.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.222+0000 7ff62b7c4640 1 --2- 172.21.15.125:0/2988424007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6080664a0 0x7ff608068960 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff62413e910 tx=0x7ff614007cb0 comp rx=0 tx=0).stop 2023-10-30T12:44:28.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.222+0000 7ff62b7c4640 1 -- 172.21.15.125:0/2988424007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6240c6b80 msgr2=0x7ff6241f5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:28.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.222+0000 7ff62b7c4640 1 --2- 172.21.15.125:0/2988424007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6240c6b80 0x7ff6241f5b70 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7ff6200b0040 tx=0x7ff620019220 comp rx=0 tx=0).stop 2023-10-30T12:44:28.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.222+0000 7ff62b7c4640 1 -- 172.21.15.125:0/2988424007 shutdown_connections 2023-10-30T12:44:28.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.222+0000 7ff62b7c4640 1 --2- 172.21.15.125:0/2988424007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6080664a0 0x7ff608068960 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:28.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.222+0000 7ff62b7c4640 1 --2- 172.21.15.125:0/2988424007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff62413f4a0 0x7ff6240c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:28.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.222+0000 7ff62b7c4640 1 --2- 172.21.15.125:0/2988424007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff6240c6f60 0x7ff6241f60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:28.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.222+0000 7ff62b7c4640 1 --2- 172.21.15.125:0/2988424007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6240c6b80 0x7ff6241f5b70 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:28.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.222+0000 7ff62b7c4640 1 -- 172.21.15.125:0/2988424007 >> 172.21.15.125:0/2988424007 conn(0x7ff6240c7910 msgr2=0x7ff624069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:28.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.222+0000 7ff62b7c4640 1 -- 172.21.15.125:0/2988424007 shutdown_connections 2023-10-30T12:44:28.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:28.222+0000 7ff62b7c4640 1 -- 172.21.15.125:0/2988424007 wait complete. 2023-10-30T12:44:28.241 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:44:30.242 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:30.258 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:44:30.258+0000 7feb93272640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:44:30.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.390+0000 7fd4fdf9d640 1 Processor -- start 2023-10-30T12:44:30.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.390+0000 7fd4fdf9d640 1 -- start start 2023-10-30T12:44:30.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.390+0000 7fd4fdf9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4f81538a0 0x7fd4f815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:30.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.390+0000 7fd4fdf9d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4f80cb800 con 0x7fd4f80ca0e0 2023-10-30T12:44:30.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.390+0000 7fd4fdf9d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4f815dc30 con 0x7fd4f80cf160 2023-10-30T12:44:30.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.390+0000 7fd4fdf9d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f80cb330 con 0x7fd4f81538a0 2023-10-30T12:44:30.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.390+0000 7fd4f77fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4f81538a0 0x7fd4f815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:30.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.390+0000 7fd4effff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd4f80ca0e0 0x7fd4f815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46414/0 (socket says 172.21.15.125:46414) 2023-10-30T12:44:30.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.390+0000 7fd4effff640 1 -- 172.21.15.125:0/3023402045 learned_addr learned my addr 172.21.15.125:0/3023402045 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:44:30.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.390+0000 7fd4f77fe640 1 -- 172.21.15.125:0/3023402045 >> v1:172.21.15.136:6789/0 conn(0x7fd4f80cf160 legacy=0x7fd4f8164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:30.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.390+0000 7fd4f77fe640 1 -- 172.21.15.125:0/3023402045 >> v1:172.21.15.125:6789/0 conn(0x7fd4f80ca0e0 legacy=0x7fd4f815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:30.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.390+0000 7fd4f77fe640 1 -- 172.21.15.125:0/3023402045 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f80cad70 con 0x7fd4f81538a0 2023-10-30T12:44:30.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.390+0000 7fd4f6ffd640 1 -- 172.21.15.125:0/3023402045 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061796381 0 0) 0x7fd4f80cb800 con 0x7fd4f80ca0e0 2023-10-30T12:44:30.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.394+0000 7fd4f77fe640 1 --2- 172.21.15.125:0/3023402045 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4f81538a0 0x7fd4f815d6f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd4e80097c0 tx=0x7fd4e801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5fef87ef4df892c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:30.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.394+0000 7fd4f6ffd640 1 -- 172.21.15.125:0/3023402045 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e80a8970 con 0x7fd4f81538a0 2023-10-30T12:44:30.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.394+0000 7fd4f6ffd640 1 -- 172.21.15.125:0/3023402045 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e80a8ad0 con 0x7fd4f81538a0 2023-10-30T12:44:30.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.394+0000 7fd4f6ffd640 1 -- 172.21.15.125:0/3023402045 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e80a8e60 con 0x7fd4f81538a0 2023-10-30T12:44:30.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.394+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3023402045 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4f81538a0 msgr2=0x7fd4f815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:30.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.394+0000 7fd4fdf9d640 1 --2- 172.21.15.125:0/3023402045 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4f81538a0 0x7fd4f815d6f0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fd4e80097c0 tx=0x7fd4e801ec70 comp rx=0 tx=0).stop 2023-10-30T12:44:30.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.394+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3023402045 shutdown_connections 2023-10-30T12:44:30.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.394+0000 7fd4fdf9d640 1 --2- 172.21.15.125:0/3023402045 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4f81538a0 0x7fd4f815d6f0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:30.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.394+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3023402045 >> 172.21.15.125:0/3023402045 conn(0x7fd4f806bfe0 msgr2=0x7fd4f806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:30.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.394+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3023402045 shutdown_connections 2023-10-30T12:44:30.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.394+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3023402045 wait complete. 2023-10-30T12:44:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4fdf9d640 1 Processor -- start 2023-10-30T12:44:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4fdf9d640 1 -- start start 2023-10-30T12:44:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4fdf9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4f80ca0e0 0x7fd4f80b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:30.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4fdf9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4f80cf160 0x7fd4f8148080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:30.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4fdf9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4f81538a0 0x7fd4f81448b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:30.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4fdf9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f81e5480 con 0x7fd4f81538a0 2023-10-30T12:44:30.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4fdf9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f80b4d10 con 0x7fd4f80cf160 2023-10-30T12:44:30.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4fdf9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f8144010 con 0x7fd4f80ca0e0 2023-10-30T12:44:30.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4f7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4f81538a0 0x7fd4f81448b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:30.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4f7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4f81538a0 0x7fd4f81448b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33628/0 (socket says 172.21.15.125:33628) 2023-10-30T12:44:30.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4f7fff640 1 -- 172.21.15.125:0/3921598716 learned_addr learned my addr 172.21.15.125:0/3921598716 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:30.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4effff640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4f80cf160 0x7fd4f8148080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4f77fe640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4f80ca0e0 0x7fd4f80b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4f7fff640 1 -- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4f80ca0e0 msgr2=0x7fd4f80b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4f7fff640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4f80ca0e0 0x7fd4f80b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4f7fff640 1 -- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4f80cf160 msgr2=0x7fd4f8148080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4f7fff640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4f80cf160 0x7fd4f8148080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4f7fff640 1 -- 172.21.15.125:0/3921598716 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e8009420 con 0x7fd4f81538a0 2023-10-30T12:44:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4f77fe640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4f80ca0e0 0x7fd4f80b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:44:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4effff640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4f80cf160 0x7fd4f8148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:44:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4f7fff640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4f81538a0 0x7fd4f81448b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000c6c0 tx=0x7fd4f000cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4f4ff9640 1 -- 172.21.15.125:0/3921598716 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f0003b80 con 0x7fd4f81538a0 2023-10-30T12:44:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4f4ff9640 1 -- 172.21.15.125:0/3921598716 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f0003ce0 con 0x7fd4f81538a0 2023-10-30T12:44:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4f4ff9640 1 -- 172.21.15.125:0/3921598716 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f00052f0 con 0x7fd4f81538a0 2023-10-30T12:44:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3921598716 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f8144420 con 0x7fd4f81538a0 2023-10-30T12:44:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3921598716 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f81e49e0 con 0x7fd4f81538a0 2023-10-30T12:44:30.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.398+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3921598716 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f80b4240 con 0x7fd4f81538a0 2023-10-30T12:44:30.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.402+0000 7fd4f4ff9640 1 -- 172.21.15.125:0/3921598716 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd4f0005450 con 0x7fd4f81538a0 2023-10-30T12:44:30.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.402+0000 7fd4f4ff9640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4d80664a0 0x7fd4d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:30.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.402+0000 7fd4f4ff9640 1 -- 172.21.15.125:0/3921598716 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7fd4f0083ce0 con 0x7fd4f81538a0 2023-10-30T12:44:30.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.402+0000 7fd4f77fe640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4d80664a0 0x7fd4d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:30.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.402+0000 7fd4f77fe640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4d80664a0 0x7fd4d8068960 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd4e80b2000 tx=0x7fd4e8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:30.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.406+0000 7fd4f4ff9640 1 -- 172.21.15.125:0/3921598716 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd4f0050ed0 con 0x7fd4f81538a0 2023-10-30T12:44:30.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.586+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3921598716 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4f815dc30 con 0x7fd4f81538a0 2023-10-30T12:44:30.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.586+0000 7fd4f4ff9640 1 -- 172.21.15.125:0/3921598716 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5737 (secure 0 0 0) 0x7fd4f815dc30 con 0x7fd4f81538a0 2023-10-30T12:44:30.589 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:30.589 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:28.903939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:30.589 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 83 2023-10-30T12:44:30.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.590+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4d80664a0 msgr2=0x7fd4d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:30.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.590+0000 7fd4fdf9d640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4d80664a0 0x7fd4d8068960 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd4e80b2000 tx=0x7fd4e8008040 comp rx=0 tx=0).stop 2023-10-30T12:44:30.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.590+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3921598716 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4f81538a0 msgr2=0x7fd4f81448b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:30.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.590+0000 7fd4fdf9d640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4f81538a0 0x7fd4f81448b0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd4f000c6c0 tx=0x7fd4f000cb80 comp rx=0 tx=0).stop 2023-10-30T12:44:30.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.590+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3921598716 shutdown_connections 2023-10-30T12:44:30.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.590+0000 7fd4fdf9d640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4d80664a0 0x7fd4d8068960 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:30.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.590+0000 7fd4fdf9d640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4f81538a0 0x7fd4f81448b0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:30.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.590+0000 7fd4fdf9d640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4f80cf160 0x7fd4f8148080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:30.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.590+0000 7fd4fdf9d640 1 --2- 172.21.15.125:0/3921598716 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4f80ca0e0 0x7fd4f80b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:30.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.590+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3921598716 >> 172.21.15.125:0/3921598716 conn(0x7fd4f806bfe0 msgr2=0x7fd4f8142980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:30.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.590+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3921598716 shutdown_connections 2023-10-30T12:44:30.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:30.590+0000 7fd4fdf9d640 1 -- 172.21.15.125:0/3921598716 wait complete. 2023-10-30T12:44:30.609 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=1, replacing gid=5345 2023-10-30T12:44:30.609 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.f 2023-10-30T12:44:31.610 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T12:44:31.610 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T12:44:31.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T12:44:31.613 INFO:tasks.ceph.mds.f:Started 2023-10-30T12:44:31.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:31.690 INFO:tasks.ceph.mds.f.smithi136.stdout:starting mds.f at 2023-10-30T12:44:31.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.758+0000 7febeb891640 1 Processor -- start 2023-10-30T12:44:31.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.758+0000 7febeb891640 1 -- start start 2023-10-30T12:44:31.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febeb891640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7febe4065a80 0x7febe40cb550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:31.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febeb891640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe41560a0 con 0x7febe4065f10 2023-10-30T12:44:31.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febeb891640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe4156380 con 0x7febe414fd50 2023-10-30T12:44:31.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febeb891640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe4156660 con 0x7febe4065a80 2023-10-30T12:44:31.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febe9e2e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7febe4065f10 0x7febe4071250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36428/0 (socket says 172.21.15.125:36428) 2023-10-30T12:44:31.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febe9e2e640 1 -- 172.21.15.125:0/1199606241 learned_addr learned my addr 172.21.15.125:0/1199606241 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:44:31.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febe8e2c640 1 --2- 172.21.15.125:0/1199606241 >> v2:172.21.15.136:3300/0 conn(0x7febe4065a80 0x7febe40cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:31.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febdbfff640 1 -- 172.21.15.125:0/1199606241 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288505799 0 0) 0x7febe41560a0 con 0x7febe4065f10 2023-10-30T12:44:31.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febdbfff640 1 -- 172.21.15.125:0/1199606241 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febd0003530 con 0x7febe4065f10 2023-10-30T12:44:31.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febdbfff640 1 -- 172.21.15.125:0/1199606241 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618451042 0 0) 0x7febe4156380 con 0x7febe414fd50 2023-10-30T12:44:31.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febdbfff640 1 -- 172.21.15.125:0/1199606241 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe41560a0 con 0x7febe414fd50 2023-10-30T12:44:31.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febdbfff640 1 -- 172.21.15.125:0/1199606241 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2423659891 0 0) 0x7febd0003530 con 0x7febe4065f10 2023-10-30T12:44:31.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febe8e2c640 1 -- 172.21.15.125:0/1199606241 >> v1:172.21.15.136:6789/0 conn(0x7febe414fd50 legacy=0x7febe40c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:31.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febe8e2c640 1 -- 172.21.15.125:0/1199606241 >> v1:172.21.15.125:6789/0 conn(0x7febe4065f10 legacy=0x7febe4071250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:31.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.762+0000 7febe8e2c640 1 -- 172.21.15.125:0/1199606241 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe4156940 con 0x7febe4065a80 2023-10-30T12:44:31.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.766+0000 7febe8e2c640 1 --2- 172.21.15.125:0/1199606241 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe4065a80 0x7febe40cb550 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7febd40097c0 tx=0x7febd401c340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25ba2463a86e1942 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:31.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.766+0000 7febdbfff640 1 -- 172.21.15.125:0/1199606241 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc001a00 con 0x7febe4065a80 2023-10-30T12:44:31.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.766+0000 7febdbfff640 1 -- 172.21.15.125:0/1199606241 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd40a7050 con 0x7febe4065a80 2023-10-30T12:44:31.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.766+0000 7febdbfff640 1 -- 172.21.15.125:0/1199606241 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd40ab600 con 0x7febe4065a80 2023-10-30T12:44:31.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.766+0000 7febeb891640 1 -- 172.21.15.125:0/1199606241 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe4065a80 msgr2=0x7febe40cb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:31.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.766+0000 7febeb891640 1 --2- 172.21.15.125:0/1199606241 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe4065a80 0x7febe40cb550 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7febd40097c0 tx=0x7febd401c340 comp rx=0 tx=0).stop 2023-10-30T12:44:31.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.766+0000 7febeb891640 1 -- 172.21.15.125:0/1199606241 shutdown_connections 2023-10-30T12:44:31.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.766+0000 7febeb891640 1 --2- 172.21.15.125:0/1199606241 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe4065a80 0x7febe40cb550 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:31.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.766+0000 7febeb891640 1 -- 172.21.15.125:0/1199606241 >> 172.21.15.125:0/1199606241 conn(0x7febe406c6e0 msgr2=0x7febe4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:31.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.766+0000 7febeb891640 1 -- 172.21.15.125:0/1199606241 shutdown_connections 2023-10-30T12:44:31.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.766+0000 7febeb891640 1 -- 172.21.15.125:0/1199606241 wait complete. 2023-10-30T12:44:31.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.766+0000 7febeb891640 1 Processor -- start 2023-10-30T12:44:31.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.766+0000 7febeb891640 1 -- start start 2023-10-30T12:44:31.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febeb891640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe4065a80 0x7febe40c3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:31.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febeb891640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febe4065f10 0x7febe41f0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:31.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febeb891640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe414fd50 0x7febe41434e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:31.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febeb891640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe40c5130 con 0x7febe4065f10 2023-10-30T12:44:31.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febeb891640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe40c4090 con 0x7febe414fd50 2023-10-30T12:44:31.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febeb891640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe40c4370 con 0x7febe4065a80 2023-10-30T12:44:31.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febe8e2c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febe4065f10 0x7febe41f0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:31.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febe8e2c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febe4065f10 0x7febe41f0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43620/0 (socket says 172.21.15.125:43620) 2023-10-30T12:44:31.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febe8e2c640 1 -- 172.21.15.125:0/3611905269 learned_addr learned my addr 172.21.15.125:0/3611905269 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:31.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febe962d640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe4065a80 0x7febe40c3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:31.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febe9e2e640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe414fd50 0x7febe41434e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:31.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febe8e2c640 1 -- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe4065a80 msgr2=0x7febe40c3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:31.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febe8e2c640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe4065a80 0x7febe40c3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:31.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febe8e2c640 1 -- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe414fd50 msgr2=0x7febe41434e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:31.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febe8e2c640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe414fd50 0x7febe41434e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:31.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febe8e2c640 1 -- 172.21.15.125:0/3611905269 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd4009420 con 0x7febe4065f10 2023-10-30T12:44:31.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febe962d640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe4065a80 0x7febe40c3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:44:31.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febe8e2c640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febe4065f10 0x7febe41f0940 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7febd4000c10 tx=0x7febd4019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:31.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febd9ffb640 1 -- 172.21.15.125:0/3611905269 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd40b9070 con 0x7febe4065f10 2023-10-30T12:44:31.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febd9ffb640 1 -- 172.21.15.125:0/3611905269 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd4003830 con 0x7febe4065f10 2023-10-30T12:44:31.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febd9ffb640 1 -- 172.21.15.125:0/3611905269 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4003ba0 con 0x7febe4065f10 2023-10-30T12:44:31.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febeb891640 1 -- 172.21.15.125:0/3611905269 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe40c4650 con 0x7febe4065f10 2023-10-30T12:44:31.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febeb891640 1 -- 172.21.15.125:0/3611905269 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe40c4c10 con 0x7febe4065f10 2023-10-30T12:44:31.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.770+0000 7febeb891640 1 -- 172.21.15.125:0/3611905269 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe40c4e00 con 0x7febe4065f10 2023-10-30T12:44:31.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.774+0000 7febd9ffb640 1 -- 172.21.15.125:0/3611905269 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7febd40b2040 con 0x7febe4065f10 2023-10-30T12:44:31.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.774+0000 7febd9ffb640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febc80664a0 0x7febc8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:31.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.774+0000 7febd9ffb640 1 -- 172.21.15.125:0/3611905269 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7febd411d8b0 con 0x7febe4065f10 2023-10-30T12:44:31.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.774+0000 7febe962d640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febc80664a0 0x7febc8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:31.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.774+0000 7febe962d640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febc80664a0 0x7febc8068960 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7febe00011f0 tx=0x7febe0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:31.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.778+0000 7febd9ffb640 1 -- 172.21.15.125:0/3611905269 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7febd40eab60 con 0x7febe4065f10 2023-10-30T12:44:31.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.958+0000 7febeb891640 1 -- 172.21.15.125:0/3611905269 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febe404e170 con 0x7febe4065f10 2023-10-30T12:44:31.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.962+0000 7febd9ffb640 1 -- 172.21.15.125:0/3611905269 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+5737 (secure 0 0 0) 0x7febd40ee180 con 0x7febe4065f10 2023-10-30T12:44:31.961 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:31.962 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:28.903939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:31.962 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 83 2023-10-30T12:44:31.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.962+0000 7febeb891640 1 -- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febc80664a0 msgr2=0x7febc8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:31.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.962+0000 7febeb891640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febc80664a0 0x7febc8068960 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7febe00011f0 tx=0x7febe0009040 comp rx=0 tx=0).stop 2023-10-30T12:44:31.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.962+0000 7febeb891640 1 -- 172.21.15.125:0/3611905269 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febe4065f10 msgr2=0x7febe41f0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:31.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.962+0000 7febeb891640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febe4065f10 0x7febe41f0940 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7febd4000c10 tx=0x7febd4019140 comp rx=0 tx=0).stop 2023-10-30T12:44:31.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.962+0000 7febeb891640 1 -- 172.21.15.125:0/3611905269 shutdown_connections 2023-10-30T12:44:31.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.962+0000 7febeb891640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febc80664a0 0x7febc8068960 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:31.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.962+0000 7febeb891640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe414fd50 0x7febe41434e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:31.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.962+0000 7febeb891640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febe4065f10 0x7febe41f0940 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:31.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.962+0000 7febeb891640 1 --2- 172.21.15.125:0/3611905269 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe4065a80 0x7febe40c3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:31.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.962+0000 7febeb891640 1 -- 172.21.15.125:0/3611905269 >> 172.21.15.125:0/3611905269 conn(0x7febe406c6e0 msgr2=0x7febe4152990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:31.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.962+0000 7febeb891640 1 -- 172.21.15.125:0/3611905269 shutdown_connections 2023-10-30T12:44:31.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:31.962+0000 7febeb891640 1 -- 172.21.15.125:0/3611905269 wait complete. 2023-10-30T12:44:31.982 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T12:44:33.983 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:44:34.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2c965640 1 Processor -- start 2023-10-30T12:44:34.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2c965640 1 -- start start 2023-10-30T12:44:34.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2c965640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac28069e90 0x7fac2815a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:34.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2c965640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac2806cb80 0x7fac2806d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:34.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2c965640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac2806d550 con 0x7fac2806a270 2023-10-30T12:44:34.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2c965640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac280c8570 con 0x7fac28069e90 2023-10-30T12:44:34.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2c965640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac280c6c10 con 0x7fac2806cb80 2023-10-30T12:44:34.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac28069e90 0x7fac2815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:34.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac28069e90 0x7fac2815a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43624/0 (socket says 172.21.15.125:43624) 2023-10-30T12:44:34.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2659c640 1 -- 172.21.15.125:0/767062773 learned_addr learned my addr 172.21.15.125:0/767062773 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:34.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac26d9d640 1 --2- 172.21.15.125:0/767062773 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac2806cb80 0x7fac2806d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:34.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2659c640 1 -- 172.21.15.125:0/767062773 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac2806cb80 msgr2=0x7fac2806d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:44:34.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2659c640 1 --2- 172.21.15.125:0/767062773 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac2806cb80 0x7fac2806d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:34.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2659c640 1 -- 172.21.15.125:0/767062773 >> v1:172.21.15.125:6789/0 conn(0x7fac2806a270 legacy=0x7fac280c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:34.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2659c640 1 -- 172.21.15.125:0/767062773 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac280c6ef0 con 0x7fac28069e90 2023-10-30T12:44:34.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.130+0000 7fac2559a640 1 -- 172.21.15.125:0/767062773 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764502847 0 0) 0x7fac2806d550 con 0x7fac2806a270 2023-10-30T12:44:34.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.134+0000 7fac2659c640 1 --2- 172.21.15.125:0/767062773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac28069e90 0x7fac2815a410 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fac18009870 tx=0x7fac1801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cce196c0a95d77ca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:34.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.134+0000 7fac2559a640 1 -- 172.21.15.125:0/767062773 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac180a7070 con 0x7fac28069e90 2023-10-30T12:44:34.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.134+0000 7fac2559a640 1 -- 172.21.15.125:0/767062773 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac180ab430 con 0x7fac28069e90 2023-10-30T12:44:34.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.134+0000 7fac2559a640 1 -- 172.21.15.125:0/767062773 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac180ab7c0 con 0x7fac28069e90 2023-10-30T12:44:34.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.134+0000 7fac2c965640 1 -- 172.21.15.125:0/767062773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac28069e90 msgr2=0x7fac2815a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:34.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.134+0000 7fac2c965640 1 --2- 172.21.15.125:0/767062773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac28069e90 0x7fac2815a410 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fac18009870 tx=0x7fac1801c540 comp rx=0 tx=0).stop 2023-10-30T12:44:34.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.134+0000 7fac2c965640 1 -- 172.21.15.125:0/767062773 shutdown_connections 2023-10-30T12:44:34.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.134+0000 7fac2c965640 1 --2- 172.21.15.125:0/767062773 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac2806cb80 0x7fac2806d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:34.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.134+0000 7fac2c965640 1 --2- 172.21.15.125:0/767062773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac28069e90 0x7fac2815a410 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:34.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.134+0000 7fac2c965640 1 -- 172.21.15.125:0/767062773 >> 172.21.15.125:0/767062773 conn(0x7fac2806dfd0 msgr2=0x7fac2806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:34.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.134+0000 7fac2c965640 1 -- 172.21.15.125:0/767062773 shutdown_connections 2023-10-30T12:44:34.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.134+0000 7fac2c965640 1 -- 172.21.15.125:0/767062773 wait complete. 2023-10-30T12:44:34.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2c965640 1 Processor -- start 2023-10-30T12:44:34.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2c965640 1 -- start start 2023-10-30T12:44:34.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2c965640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac28069e90 0x7fac28146f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:34.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2c965640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac2806a270 0x7fac281533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:34.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2c965640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac2806cb80 0x7fac28153910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:34.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2c965640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac281f4a30 con 0x7fac28069e90 2023-10-30T12:44:34.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2c965640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac280715a0 con 0x7fac2806a270 2023-10-30T12:44:34.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2c965640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac280c25c0 con 0x7fac2806cb80 2023-10-30T12:44:34.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac28069e90 0x7fac28146f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:34.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac28069e90 0x7fac28146f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43636/0 (socket says 172.21.15.125:43636) 2023-10-30T12:44:34.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2659c640 1 -- 172.21.15.125:0/339315576 learned_addr learned my addr 172.21.15.125:0/339315576 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:44:34.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac26d9d640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac2806cb80 0x7fac28153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:34.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac25d9b640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac2806a270 0x7fac281533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:34.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2659c640 1 -- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac2806cb80 msgr2=0x7fac28153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:34.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2659c640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac2806cb80 0x7fac28153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:34.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2659c640 1 -- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac2806a270 msgr2=0x7fac281533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:34.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2659c640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac2806a270 0x7fac281533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:34.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2659c640 1 -- 172.21.15.125:0/339315576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac180094d0 con 0x7fac28069e90 2023-10-30T12:44:34.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac25d9b640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac2806a270 0x7fac281533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:44:34.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2659c640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac28069e90 0x7fac28146f10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fac180ad040 tx=0x7fac180192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:34.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac1effd640 1 -- 172.21.15.125:0/339315576 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac180bc070 con 0x7fac28069e90 2023-10-30T12:44:34.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac1effd640 1 -- 172.21.15.125:0/339315576 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac180abe10 con 0x7fac28069e90 2023-10-30T12:44:34.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac1effd640 1 -- 172.21.15.125:0/339315576 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac180b4890 con 0x7fac28069e90 2023-10-30T12:44:34.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2c965640 1 -- 172.21.15.125:0/339315576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac28151a60 con 0x7fac28069e90 2023-10-30T12:44:34.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2c965640 1 -- 172.21.15.125:0/339315576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac2814e390 con 0x7fac28069e90 2023-10-30T12:44:34.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.138+0000 7fac2c965640 1 -- 172.21.15.125:0/339315576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac2806a650 con 0x7fac28069e90 2023-10-30T12:44:34.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.142+0000 7fac1effd640 1 -- 172.21.15.125:0/339315576 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fac180b49f0 con 0x7fac28069e90 2023-10-30T12:44:34.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.142+0000 7fac1effd640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac08066470 0x7fac08068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:44:34.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.142+0000 7fac1effd640 1 -- 172.21.15.125:0/339315576 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7fac1811dba0 con 0x7fac28069e90 2023-10-30T12:44:34.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.142+0000 7fac25d9b640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac08066470 0x7fac08068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:44:34.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.142+0000 7fac25d9b640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac08066470 0x7fac08068930 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fac2814a290 tx=0x7fac10009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:44:34.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.146+0000 7fac1effd640 1 -- 172.21.15.125:0/339315576 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fac2806a650 con 0x7fac28069e90 2023-10-30T12:44:34.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.326+0000 7fac2c965640 1 -- 172.21.15.125:0/339315576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac2806a650 con 0x7fac28069e90 2023-10-30T12:44:34.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.326+0000 7fac1effd640 1 -- 172.21.15.125:0/339315576 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fac2806a650 con 0x7fac28069e90 2023-10-30T12:44:34.329 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:44:34.329 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5935,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:28.903939+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:44:34.330 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 84 2023-10-30T12:44:34.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.330+0000 7fac2c965640 1 -- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac08066470 msgr2=0x7fac08068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:34.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.330+0000 7fac2c965640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac08066470 0x7fac08068930 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fac2814a290 tx=0x7fac10009040 comp rx=0 tx=0).stop 2023-10-30T12:44:34.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.330+0000 7fac2c965640 1 -- 172.21.15.125:0/339315576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac28069e90 msgr2=0x7fac28146f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:44:34.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.330+0000 7fac2c965640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac28069e90 0x7fac28146f10 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fac180ad040 tx=0x7fac180192e0 comp rx=0 tx=0).stop 2023-10-30T12:44:34.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.330+0000 7fac2c965640 1 -- 172.21.15.125:0/339315576 shutdown_connections 2023-10-30T12:44:34.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.330+0000 7fac2c965640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac08066470 0x7fac08068930 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:34.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.330+0000 7fac2c965640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac2806cb80 0x7fac28153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:34.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.330+0000 7fac2c965640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac2806a270 0x7fac281533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:34.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.330+0000 7fac2c965640 1 --2- 172.21.15.125:0/339315576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac28069e90 0x7fac28146f10 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:44:34.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.330+0000 7fac2c965640 1 -- 172.21.15.125:0/339315576 >> 172.21.15.125:0/339315576 conn(0x7fac2806dfd0 msgr2=0x7fac280718e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:44:34.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.330+0000 7fac2c965640 1 -- 172.21.15.125:0/339315576 shutdown_connections 2023-10-30T12:44:34.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:44:34.330+0000 7fac2c965640 1 -- 172.21.15.125:0/339315576 wait complete. 2023-10-30T12:44:34.349 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T12:44:34.349 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 98 secs before thrashing 2023-10-30T12:44:46.231 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:44:46.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:45:16.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:45:16.309 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:45:46.369 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:45:46.373 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:46:12.350 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:12.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb2da79640 1 Processor -- start 2023-10-30T12:46:12.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb2da79640 1 -- start start 2023-10-30T12:46:12.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb2da79640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb280cf160 0x7fcb280cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:12.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb2da79640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb280cb800 con 0x7fcb281538a0 2023-10-30T12:46:12.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb2da79640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb280cb330 con 0x7fcb2815d6f0 2023-10-30T12:46:12.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb2da79640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb280cad70 con 0x7fcb280cf160 2023-10-30T12:46:12.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb277fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcb281538a0 0x7fcb280ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37142/0 (socket says 172.21.15.125:37142) 2023-10-30T12:46:12.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb277fe640 1 -- 172.21.15.125:0/1788782663 learned_addr learned my addr 172.21.15.125:0/1788782663 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:46:12.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb27fff640 1 --2- 172.21.15.125:0/1788782663 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb280cf160 0x7fcb280cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:12.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb267fc640 1 -- 172.21.15.125:0/1788782663 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625919914 0 0) 0x7fcb280cb800 con 0x7fcb281538a0 2023-10-30T12:46:12.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb267fc640 1 -- 172.21.15.125:0/1788782663 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb10003530 con 0x7fcb281538a0 2023-10-30T12:46:12.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb267fc640 1 -- 172.21.15.125:0/1788782663 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1360796267 0 0) 0x7fcb10003530 con 0x7fcb281538a0 2023-10-30T12:46:12.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb267fc640 1 -- 172.21.15.125:0/1788782663 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb280cf160 msgr2=0x7fcb280cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:12.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb267fc640 1 --2- 172.21.15.125:0/1788782663 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb280cf160 0x7fcb280cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb267fc640 1 -- 172.21.15.125:0/1788782663 >> v1:172.21.15.136:6789/0 conn(0x7fcb2815d6f0 legacy=0x7fcb2815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:12.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb267fc640 1 -- 172.21.15.125:0/1788782663 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb280cb050 con 0x7fcb281538a0 2023-10-30T12:46:12.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb267fc640 1 -- 172.21.15.125:0/1788782663 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fcb18001680 con 0x7fcb281538a0 2023-10-30T12:46:12.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb267fc640 1 -- 172.21.15.125:0/1788782663 >> v1:172.21.15.125:6789/0 conn(0x7fcb281538a0 legacy=0x7fcb280ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:12.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb267fc640 1 --2- 172.21.15.125:0/1788782663 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb280cf160 0x7fcb100039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:12.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb267fc640 1 --2- 172.21.15.125:0/1788782663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb10003f30 0x7fcb100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:12.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb267fc640 1 --2- 172.21.15.125:0/1788782663 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb10004d10 0x7fcb10005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:12.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb267fc640 1 -- 172.21.15.125:0/1788782663 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb280cad70 con 0x7fcb10003f30 2023-10-30T12:46:12.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.489+0000 7fcb267fc640 1 -- 172.21.15.125:0/1788782663 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1000bb00 con 0x7fcb10004d10 2023-10-30T12:46:12.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.493+0000 7fcb267fc640 1 -- 172.21.15.125:0/1788782663 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb1000bcc0 con 0x7fcb280cf160 2023-10-30T12:46:12.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.493+0000 7fcb26ffd640 1 --2- 172.21.15.125:0/1788782663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb10003f30 0x7fcb100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:12.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.493+0000 7fcb27fff640 1 --2- 172.21.15.125:0/1788782663 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb10004d10 0x7fcb10005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:12.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.493+0000 7fcb277fe640 1 --2- 172.21.15.125:0/1788782663 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb280cf160 0x7fcb100039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:12.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.493+0000 7fcb26ffd640 1 -- 172.21.15.125:0/1788782663 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb280cf160 msgr2=0x7fcb100039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:12.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.493+0000 7fcb26ffd640 1 --2- 172.21.15.125:0/1788782663 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb280cf160 0x7fcb100039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.493+0000 7fcb26ffd640 1 -- 172.21.15.125:0/1788782663 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb10004d10 msgr2=0x7fcb10005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:12.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.493+0000 7fcb26ffd640 1 --2- 172.21.15.125:0/1788782663 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb10004d10 0x7fcb10005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.493+0000 7fcb26ffd640 1 -- 172.21.15.125:0/1788782663 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb10003530 con 0x7fcb10003f30 2023-10-30T12:46:12.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.493+0000 7fcb26ffd640 1 --2- 172.21.15.125:0/1788782663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb10003f30 0x7fcb100047d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c001780 tx=0x7fcb1c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fe38d31c90aed1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:12.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.493+0000 7fcb267fc640 1 -- 172.21.15.125:0/1788782663 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c0052d0 con 0x7fcb10003f30 2023-10-30T12:46:12.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.493+0000 7fcb267fc640 1 -- 172.21.15.125:0/1788782663 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb1c0a7050 con 0x7fcb10003f30 2023-10-30T12:46:12.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.493+0000 7fcb267fc640 1 -- 172.21.15.125:0/1788782663 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb1c003600 con 0x7fcb10003f30 2023-10-30T12:46:12.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb2da79640 1 -- 172.21.15.125:0/1788782663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb10003f30 msgr2=0x7fcb100047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:12.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb2da79640 1 --2- 172.21.15.125:0/1788782663 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb10003f30 0x7fcb100047d0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c001780 tx=0x7fcb1c01c400 comp rx=0 tx=0).stop 2023-10-30T12:46:12.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb27fff640 1 -- 172.21.15.125:0/1788782663 reap_dead start 2023-10-30T12:46:12.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb2da79640 1 -- 172.21.15.125:0/1788782663 shutdown_connections 2023-10-30T12:46:12.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb2da79640 1 -- 172.21.15.125:0/1788782663 >> 172.21.15.125:0/1788782663 conn(0x7fcb2806bfe0 msgr2=0x7fcb2806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:12.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb2da79640 1 -- 172.21.15.125:0/1788782663 shutdown_connections 2023-10-30T12:46:12.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb2da79640 1 -- 172.21.15.125:0/1788782663 wait complete. 2023-10-30T12:46:12.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb2da79640 1 Processor -- start 2023-10-30T12:46:12.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb2da79640 1 -- start start 2023-10-30T12:46:12.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb2da79640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb10003f30 0x7fcb281587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:12.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb2da79640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb28159120 0x7fcb28153fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:12.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb2da79640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb2814fb70 0x7fcb28154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:12.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb2da79640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb28156ed0 con 0x7fcb2814fb70 2023-10-30T12:46:12.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb2da79640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2815cd50 con 0x7fcb10003f30 2023-10-30T12:46:12.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb2da79640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2815c790 con 0x7fcb28159120 2023-10-30T12:46:12.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb27fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb2814fb70 0x7fcb28154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:12.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb27fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb2814fb70 0x7fcb28154520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55628/0 (socket says 172.21.15.125:55628) 2023-10-30T12:46:12.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb27fff640 1 -- 172.21.15.125:0/4035921953 learned_addr learned my addr 172.21.15.125:0/4035921953 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:12.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb277fe640 1 --2- 172.21.15.125:0/4035921953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb10003f30 0x7fcb281587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:12.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb26ffd640 1 --2- 172.21.15.125:0/4035921953 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb28159120 0x7fcb28153fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:12.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb27fff640 1 -- 172.21.15.125:0/4035921953 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb28159120 msgr2=0x7fcb28153fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:12.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb27fff640 1 --2- 172.21.15.125:0/4035921953 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb28159120 0x7fcb28153fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb27fff640 1 -- 172.21.15.125:0/4035921953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb10003f30 msgr2=0x7fcb281587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:12.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb27fff640 1 --2- 172.21.15.125:0/4035921953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb10003f30 0x7fcb281587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb27fff640 1 -- 172.21.15.125:0/4035921953 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2815ca70 con 0x7fcb2814fb70 2023-10-30T12:46:12.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.497+0000 7fcb27fff640 1 --2- 172.21.15.125:0/4035921953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb2814fb70 0x7fcb28154520 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fcb2806cb00 tx=0x7fcb200115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:12.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.501+0000 7fcb0ffff640 1 -- 172.21.15.125:0/4035921953 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2000e070 con 0x7fcb2814fb70 2023-10-30T12:46:12.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.501+0000 7fcb0ffff640 1 -- 172.21.15.125:0/4035921953 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb20007e70 con 0x7fcb2814fb70 2023-10-30T12:46:12.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.501+0000 7fcb2da79640 1 -- 172.21.15.125:0/4035921953 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb2815b360 con 0x7fcb2814fb70 2023-10-30T12:46:12.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.501+0000 7fcb0ffff640 1 -- 172.21.15.125:0/4035921953 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb20011e10 con 0x7fcb2814fb70 2023-10-30T12:46:12.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.501+0000 7fcb2da79640 1 -- 172.21.15.125:0/4035921953 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb281595f0 con 0x7fcb2814fb70 2023-10-30T12:46:12.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.501+0000 7fcb2da79640 1 -- 172.21.15.125:0/4035921953 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb2815b880 con 0x7fcb2814fb70 2023-10-30T12:46:12.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.501+0000 7fcb0ffff640 1 -- 172.21.15.125:0/4035921953 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcb200039a0 con 0x7fcb2814fb70 2023-10-30T12:46:12.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.501+0000 7fcb0ffff640 1 --2- 172.21.15.125:0/4035921953 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb140664a0 0x7fcb14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:12.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.501+0000 7fcb0ffff640 1 -- 172.21.15.125:0/4035921953 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7fcb20084810 con 0x7fcb2814fb70 2023-10-30T12:46:12.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.501+0000 7fcb277fe640 1 --2- 172.21.15.125:0/4035921953 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb140664a0 0x7fcb14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:12.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.505+0000 7fcb277fe640 1 --2- 172.21.15.125:0/4035921953 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb140664a0 0x7fcb14068960 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcb18001ca0 tx=0x7fcb18007040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:12.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.505+0000 7fcb0ffff640 1 -- 172.21.15.125:0/4035921953 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcb2815b880 con 0x7fcb2814fb70 2023-10-30T12:46:12.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.713+0000 7fcb2da79640 1 -- 172.21.15.125:0/4035921953 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb2815b880 con 0x7fcb2814fb70 2023-10-30T12:46:12.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.713+0000 7fcb0ffff640 1 -- 172.21.15.125:0/4035921953 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fcb2815b880 con 0x7fcb2814fb70 2023-10-30T12:46:12.716 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:12.717 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5935,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:44.430841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:12.717 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 85 2023-10-30T12:46:12.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.717+0000 7fcb2da79640 1 -- 172.21.15.125:0/4035921953 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb140664a0 msgr2=0x7fcb14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:12.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.717+0000 7fcb2da79640 1 --2- 172.21.15.125:0/4035921953 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb140664a0 0x7fcb14068960 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fcb18001ca0 tx=0x7fcb18007040 comp rx=0 tx=0).stop 2023-10-30T12:46:12.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.717+0000 7fcb2da79640 1 -- 172.21.15.125:0/4035921953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb2814fb70 msgr2=0x7fcb28154520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:12.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.717+0000 7fcb2da79640 1 --2- 172.21.15.125:0/4035921953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb2814fb70 0x7fcb28154520 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fcb2806cb00 tx=0x7fcb200115a0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.717+0000 7fcb2da79640 1 -- 172.21.15.125:0/4035921953 shutdown_connections 2023-10-30T12:46:12.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.717+0000 7fcb2da79640 1 --2- 172.21.15.125:0/4035921953 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb140664a0 0x7fcb14068960 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.717+0000 7fcb2da79640 1 --2- 172.21.15.125:0/4035921953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb2814fb70 0x7fcb28154520 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.717+0000 7fcb2da79640 1 --2- 172.21.15.125:0/4035921953 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb28159120 0x7fcb28153fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.717+0000 7fcb2da79640 1 --2- 172.21.15.125:0/4035921953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb10003f30 0x7fcb281587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.717+0000 7fcb2da79640 1 -- 172.21.15.125:0/4035921953 >> 172.21.15.125:0/4035921953 conn(0x7fcb2806bfe0 msgr2=0x7fcb281514e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:12.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.717+0000 7fcb2da79640 1 -- 172.21.15.125:0/4035921953 shutdown_connections 2023-10-30T12:46:12.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.717+0000 7fcb2da79640 1 -- 172.21.15.125:0/4035921953 wait complete. 2023-10-30T12:46:12.740 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T12:46:12.740 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T12:46:12.740 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:46:12.761 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T12:46:12.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:12.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.937+0000 7f2e60f72640 1 Processor -- start 2023-10-30T12:46:12.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.937+0000 7f2e60f72640 1 -- start start 2023-10-30T12:46:12.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e60f72640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2e5c153d70 0x7f2e5c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e60f72640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c06e470 0x7f2e5c0cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e60f72640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c06afa0 con 0x7f2e5c0c8490 2023-10-30T12:46:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e60f72640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e5c06d780 con 0x7f2e5c153d70 2023-10-30T12:46:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e60f72640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c154150 con 0x7f2e5c06e470 2023-10-30T12:46:12.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e5a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2e5c153d70 0x7f2e5c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e5a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2e5c153d70 0x7f2e5c06ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55630/0 (socket says 172.21.15.125:55630) 2023-10-30T12:46:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e5a59c640 1 -- 172.21.15.125:0/1214151705 learned_addr learned my addr 172.21.15.125:0/1214151705 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e5ad9d640 1 --1- 172.21.15.125:0/1214151705 >> v1:172.21.15.125:6789/0 conn(0x7f2e5c0c8490 0x7f2e5c0c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37144/0 (socket says 172.21.15.125:37144) 2023-10-30T12:46:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e51d9b640 1 --2- 172.21.15.125:0/1214151705 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c06e470 0x7f2e5c0cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:12.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e5a59c640 1 -- 172.21.15.125:0/1214151705 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c06e470 msgr2=0x7f2e5c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:12.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e5a59c640 1 --2- 172.21.15.125:0/1214151705 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c06e470 0x7f2e5c0cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e5a59c640 1 -- 172.21.15.125:0/1214151705 >> v1:172.21.15.125:6789/0 conn(0x7f2e5c0c8490 legacy=0x7f2e5c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:12.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e5a59c640 1 -- 172.21.15.125:0/1214151705 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5c06e030 con 0x7f2e5c153d70 2023-10-30T12:46:12.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.941+0000 7f2e59d9b640 1 -- 172.21.15.125:0/1214151705 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777567578 0 0) 0x7f2e5c06afa0 con 0x7f2e5c0c8490 2023-10-30T12:46:12.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.945+0000 7f2e5a59c640 1 --2- 172.21.15.125:0/1214151705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c153d70 0x7f2e5c06ca80 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c009e50 tx=0x7f2e4c01c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1b1800bd1d83cfe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:12.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.945+0000 7f2e59d9b640 1 -- 172.21.15.125:0/1214151705 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c0a58a0 con 0x7f2e5c153d70 2023-10-30T12:46:12.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.945+0000 7f2e59d9b640 1 -- 172.21.15.125:0/1214151705 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4c0a5a00 con 0x7f2e5c153d70 2023-10-30T12:46:12.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.945+0000 7f2e59d9b640 1 -- 172.21.15.125:0/1214151705 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c0a5d90 con 0x7f2e5c153d70 2023-10-30T12:46:12.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.945+0000 7f2e60f72640 1 -- 172.21.15.125:0/1214151705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c153d70 msgr2=0x7f2e5c06ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:12.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.945+0000 7f2e60f72640 1 --2- 172.21.15.125:0/1214151705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c153d70 0x7f2e5c06ca80 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c009e50 tx=0x7f2e4c01c9e0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.945+0000 7f2e60f72640 1 -- 172.21.15.125:0/1214151705 shutdown_connections 2023-10-30T12:46:12.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.945+0000 7f2e60f72640 1 --2- 172.21.15.125:0/1214151705 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c06e470 0x7f2e5c0cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.945+0000 7f2e60f72640 1 --2- 172.21.15.125:0/1214151705 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c153d70 0x7f2e5c06ca80 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.945+0000 7f2e60f72640 1 -- 172.21.15.125:0/1214151705 >> 172.21.15.125:0/1214151705 conn(0x7f2e5c071ea0 msgr2=0x7f2e5c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:12.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.945+0000 7f2e60f72640 1 -- 172.21.15.125:0/1214151705 shutdown_connections 2023-10-30T12:46:12.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.945+0000 7f2e60f72640 1 -- 172.21.15.125:0/1214151705 wait complete. 2023-10-30T12:46:12.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.945+0000 7f2e60f72640 1 Processor -- start 2023-10-30T12:46:12.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.945+0000 7f2e60f72640 1 -- start start 2023-10-30T12:46:12.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e60f72640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c06e470 0x7f2e5c149e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:12.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e60f72640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c0c8490 0x7f2e5c1f5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:12.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e60f72640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e5c153d70 0x7f2e5c1f6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:12.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e60f72640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c14bc50 con 0x7f2e5c0c8490 2023-10-30T12:46:12.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e60f72640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c0adbe0 con 0x7f2e5c153d70 2023-10-30T12:46:12.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e60f72640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c0ad970 con 0x7f2e5c06e470 2023-10-30T12:46:12.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e51d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c0c8490 0x7f2e5c1f5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:12.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e51d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c0c8490 0x7f2e5c1f5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55638/0 (socket says 172.21.15.125:55638) 2023-10-30T12:46:12.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e51d9b640 1 -- 172.21.15.125:0/3734395241 learned_addr learned my addr 172.21.15.125:0/3734395241 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:12.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e5a59c640 1 --2- 172.21.15.125:0/3734395241 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c06e470 0x7f2e5c149e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:12.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e5ad9d640 1 --2- 172.21.15.125:0/3734395241 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e5c153d70 0x7f2e5c1f6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:12.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e51d9b640 1 -- 172.21.15.125:0/3734395241 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c06e470 msgr2=0x7f2e5c149e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:12.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e51d9b640 1 --2- 172.21.15.125:0/3734395241 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c06e470 0x7f2e5c149e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e51d9b640 1 -- 172.21.15.125:0/3734395241 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e5c153d70 msgr2=0x7f2e5c1f6270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:12.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e51d9b640 1 --2- 172.21.15.125:0/3734395241 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e5c153d70 0x7f2e5c1f6270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:12.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e51d9b640 1 -- 172.21.15.125:0/3734395241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c009b30 con 0x7f2e5c0c8490 2023-10-30T12:46:12.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e51d9b640 1 --2- 172.21.15.125:0/3734395241 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c0c8490 0x7f2e5c1f5d30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400d620 tx=0x7f2e4400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:12.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e537fe640 1 -- 172.21.15.125:0/3734395241 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4400b620 con 0x7f2e5c0c8490 2023-10-30T12:46:12.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e537fe640 1 -- 172.21.15.125:0/3734395241 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4400b780 con 0x7f2e5c0c8490 2023-10-30T12:46:12.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e537fe640 1 -- 172.21.15.125:0/3734395241 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4400bb10 con 0x7f2e5c0c8490 2023-10-30T12:46:12.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e60f72640 1 -- 172.21.15.125:0/3734395241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e5c14b320 con 0x7f2e5c0c8490 2023-10-30T12:46:12.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e60f72640 1 -- 172.21.15.125:0/3734395241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e5c14b8e0 con 0x7f2e5c0c8490 2023-10-30T12:46:12.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e60f72640 1 -- 172.21.15.125:0/3734395241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e5c14be00 con 0x7f2e5c0c8490 2023-10-30T12:46:12.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.949+0000 7f2e537fe640 1 -- 172.21.15.125:0/3734395241 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2e4400bc70 con 0x7f2e5c0c8490 2023-10-30T12:46:12.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.953+0000 7f2e537fe640 1 --2- 172.21.15.125:0/3734395241 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e3c0664a0 0x7f2e3c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:12.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.953+0000 7f2e537fe640 1 -- 172.21.15.125:0/3734395241 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7f2e440845d0 con 0x7f2e5c0c8490 2023-10-30T12:46:12.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.953+0000 7f2e5a59c640 1 --2- 172.21.15.125:0/3734395241 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e3c0664a0 0x7f2e3c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:12.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.953+0000 7f2e5a59c640 1 --2- 172.21.15.125:0/3734395241 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e3c0664a0 0x7f2e3c068960 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0ae000 tx=0x7f2e4c0a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:12.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:12.957+0000 7f2e537fe640 1 -- 172.21.15.125:0/3734395241 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2e440517c0 con 0x7f2e5c0c8490 2023-10-30T12:46:13.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:13.141+0000 7f2e60f72640 1 -- 172.21.15.125:0/3734395241 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e5c04e170 con 0x7f2e5c0c8490 2023-10-30T12:46:13.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:13.141+0000 7f2e537fe640 1 -- 172.21.15.125:0/3734395241 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f2e44007380 con 0x7f2e5c0c8490 2023-10-30T12:46:13.143 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:13.144 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5935,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:44.430841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:13.144 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 85 2023-10-30T12:46:13.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:13.145+0000 7f2e60f72640 1 -- 172.21.15.125:0/3734395241 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e3c0664a0 msgr2=0x7f2e3c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:13.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:13.145+0000 7f2e60f72640 1 --2- 172.21.15.125:0/3734395241 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e3c0664a0 0x7f2e3c068960 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0ae000 tx=0x7f2e4c0a7070 comp rx=0 tx=0).stop 2023-10-30T12:46:13.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:13.145+0000 7f2e60f72640 1 -- 172.21.15.125:0/3734395241 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c0c8490 msgr2=0x7f2e5c1f5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:13.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:13.145+0000 7f2e60f72640 1 --2- 172.21.15.125:0/3734395241 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c0c8490 0x7f2e5c1f5d30 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400d620 tx=0x7f2e4400dae0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:13.145+0000 7f2e60f72640 1 -- 172.21.15.125:0/3734395241 shutdown_connections 2023-10-30T12:46:13.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:13.145+0000 7f2e60f72640 1 --2- 172.21.15.125:0/3734395241 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e3c0664a0 0x7f2e3c068960 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:13.145+0000 7f2e60f72640 1 --2- 172.21.15.125:0/3734395241 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e5c153d70 0x7f2e5c1f6270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:13.145+0000 7f2e60f72640 1 --2- 172.21.15.125:0/3734395241 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c0c8490 0x7f2e5c1f5d30 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:13.145+0000 7f2e60f72640 1 --2- 172.21.15.125:0/3734395241 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c06e470 0x7f2e5c149e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:13.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:13.145+0000 7f2e60f72640 1 -- 172.21.15.125:0/3734395241 >> 172.21.15.125:0/3734395241 conn(0x7f2e5c071ea0 msgr2=0x7f2e5c0c7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:13.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:13.145+0000 7f2e60f72640 1 -- 172.21.15.125:0/3734395241 shutdown_connections 2023-10-30T12:46:13.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:13.145+0000 7f2e60f72640 1 -- 172.21.15.125:0/3734395241 wait complete. 2023-10-30T12:46:13.165 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:46:15.166 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:15.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.305+0000 7f94ae10f640 1 Processor -- start 2023-10-30T12:46:15.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.305+0000 7f94ae10f640 1 -- start start 2023-10-30T12:46:15.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.305+0000 7f94ae10f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f94a80bfec0 0x7f94a80c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.305+0000 7f94ae10f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a8071290 con 0x7f94a806a720 2023-10-30T12:46:15.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.305+0000 7f94ae10f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a8071570 con 0x7f94a8159e60 2023-10-30T12:46:15.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.305+0000 7f94ae10f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a8071850 con 0x7f94a80bfec0 2023-10-30T12:46:15.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.305+0000 7f949ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f94a80bfec0 0x7f94a80c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.305+0000 7f94a7fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f94a806a720 0x7f94a8160610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37146/0 (socket says 172.21.15.125:37146) 2023-10-30T12:46:15.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.305+0000 7f949ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f94a80bfec0 0x7f94a80c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55648/0 (socket says 172.21.15.125:55648) 2023-10-30T12:46:15.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.305+0000 7f94a7fff640 1 -- 172.21.15.125:0/23611447 learned_addr learned my addr 172.21.15.125:0/23611447 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:46:15.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.305+0000 7f949ffff640 1 -- 172.21.15.125:0/23611447 >> v1:172.21.15.136:6789/0 conn(0x7f94a8159e60 legacy=0x7f94a806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:15.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.305+0000 7f949ffff640 1 -- 172.21.15.125:0/23611447 >> v1:172.21.15.125:6789/0 conn(0x7f94a806a720 legacy=0x7f94a8160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:15.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.305+0000 7f949ffff640 1 -- 172.21.15.125:0/23611447 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a8071b30 con 0x7f94a80bfec0 2023-10-30T12:46:15.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.305+0000 7f94a6ffd640 1 -- 172.21.15.125:0/23611447 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689872286 0 0) 0x7f94a8071290 con 0x7f94a806a720 2023-10-30T12:46:15.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.309+0000 7f949ffff640 1 --2- 172.21.15.125:0/23611447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a80bfec0 0x7f94a80c1a40 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f94900097c0 tx=0x7f949001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bedbbaf51c327c6e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:15.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.309+0000 7f94a6ffd640 1 -- 172.21.15.125:0/23611447 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94900a8970 con 0x7f94a80bfec0 2023-10-30T12:46:15.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.309+0000 7f94a6ffd640 1 -- 172.21.15.125:0/23611447 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94900a8ad0 con 0x7f94a80bfec0 2023-10-30T12:46:15.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.309+0000 7f94a6ffd640 1 -- 172.21.15.125:0/23611447 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94900a8e60 con 0x7f94a80bfec0 2023-10-30T12:46:15.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.309+0000 7f94ae10f640 1 -- 172.21.15.125:0/23611447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a80bfec0 msgr2=0x7f94a80c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:15.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.309+0000 7f94ae10f640 1 --2- 172.21.15.125:0/23611447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a80bfec0 0x7f94a80c1a40 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f94900097c0 tx=0x7f949001ec70 comp rx=0 tx=0).stop 2023-10-30T12:46:15.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.309+0000 7f94ae10f640 1 -- 172.21.15.125:0/23611447 shutdown_connections 2023-10-30T12:46:15.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.309+0000 7f94ae10f640 1 --2- 172.21.15.125:0/23611447 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a80bfec0 0x7f94a80c1a40 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.309+0000 7f94ae10f640 1 -- 172.21.15.125:0/23611447 >> 172.21.15.125:0/23611447 conn(0x7f94a80c7910 msgr2=0x7f94a80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:15.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.309+0000 7f94ae10f640 1 -- 172.21.15.125:0/23611447 shutdown_connections 2023-10-30T12:46:15.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.309+0000 7f94ae10f640 1 -- 172.21.15.125:0/23611447 wait complete. 2023-10-30T12:46:15.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94ae10f640 1 Processor -- start 2023-10-30T12:46:15.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94ae10f640 1 -- start start 2023-10-30T12:46:15.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94ae10f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94a806a720 0x7f94a81559e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94ae10f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a80bfec0 0x7f94a81585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94ae10f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94a8159e60 0x7f94a8158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94ae10f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a8155f20 con 0x7f94a80bfec0 2023-10-30T12:46:15.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94ae10f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a8155550 con 0x7f94a8159e60 2023-10-30T12:46:15.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94ae10f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a80b0ce0 con 0x7f94a806a720 2023-10-30T12:46:15.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f949ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a80bfec0 0x7f94a81585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f949ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a80bfec0 0x7f94a81585d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55658/0 (socket says 172.21.15.125:55658) 2023-10-30T12:46:15.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f949ffff640 1 -- 172.21.15.125:0/2710812917 learned_addr learned my addr 172.21.15.125:0/2710812917 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:15.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94a77fe640 1 --2- 172.21.15.125:0/2710812917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94a806a720 0x7f94a81559e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94a7fff640 1 --2- 172.21.15.125:0/2710812917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94a8159e60 0x7f94a8158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f949ffff640 1 -- 172.21.15.125:0/2710812917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94a806a720 msgr2=0x7f94a81559e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:15.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f949ffff640 1 --2- 172.21.15.125:0/2710812917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94a806a720 0x7f94a81559e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f949ffff640 1 -- 172.21.15.125:0/2710812917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94a8159e60 msgr2=0x7f94a8158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:15.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f949ffff640 1 --2- 172.21.15.125:0/2710812917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94a8159e60 0x7f94a8158b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f949ffff640 1 -- 172.21.15.125:0/2710812917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9490009420 con 0x7f94a80bfec0 2023-10-30T12:46:15.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f949ffff640 1 --2- 172.21.15.125:0/2710812917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a80bfec0 0x7f94a81585d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f94900097c0 tx=0x7f949001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:15.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94a4ff9640 1 -- 172.21.15.125:0/2710812917 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94900b9630 con 0x7f94a80bfec0 2023-10-30T12:46:15.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94a4ff9640 1 -- 172.21.15.125:0/2710812917 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94900b97b0 con 0x7f94a80bfec0 2023-10-30T12:46:15.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94a4ff9640 1 -- 172.21.15.125:0/2710812917 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94900b9b60 con 0x7f94a80bfec0 2023-10-30T12:46:15.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94ae10f640 1 -- 172.21.15.125:0/2710812917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a8154480 con 0x7f94a80bfec0 2023-10-30T12:46:15.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.313+0000 7f94ae10f640 1 -- 172.21.15.125:0/2710812917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a81f2b90 con 0x7f94a80bfec0 2023-10-30T12:46:15.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.317+0000 7f94ae10f640 1 -- 172.21.15.125:0/2710812917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a8155700 con 0x7f94a80bfec0 2023-10-30T12:46:15.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.317+0000 7f94a4ff9640 1 -- 172.21.15.125:0/2710812917 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94900b7040 con 0x7f94a80bfec0 2023-10-30T12:46:15.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.317+0000 7f94a4ff9640 1 --2- 172.21.15.125:0/2710812917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94880664a0 0x7f9488068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:15.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.317+0000 7f94a4ff9640 1 -- 172.21.15.125:0/2710812917 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7f9490122220 con 0x7f94a80bfec0 2023-10-30T12:46:15.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.317+0000 7f94a77fe640 1 --2- 172.21.15.125:0/2710812917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94880664a0 0x7f9488068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:15.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.317+0000 7f94a77fe640 1 --2- 172.21.15.125:0/2710812917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94880664a0 0x7f9488068960 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9498000f70 tx=0x7f9498009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:15.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.321+0000 7f94a4ff9640 1 -- 172.21.15.125:0/2710812917 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f94a8155700 con 0x7f94a80bfec0 2023-10-30T12:46:15.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.501+0000 7f94ae10f640 1 -- 172.21.15.125:0/2710812917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94a8155700 con 0x7f94a80bfec0 2023-10-30T12:46:15.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.501+0000 7f94a4ff9640 1 -- 172.21.15.125:0/2710812917 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f94a8155700 con 0x7f94a80bfec0 2023-10-30T12:46:15.504 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:15.505 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5935,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:44.430841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:15.505 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 85 2023-10-30T12:46:15.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.505+0000 7f94ae10f640 1 -- 172.21.15.125:0/2710812917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94880664a0 msgr2=0x7f9488068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:15.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.505+0000 7f94ae10f640 1 --2- 172.21.15.125:0/2710812917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94880664a0 0x7f9488068960 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9498000f70 tx=0x7f9498009040 comp rx=0 tx=0).stop 2023-10-30T12:46:15.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.505+0000 7f94ae10f640 1 -- 172.21.15.125:0/2710812917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a80bfec0 msgr2=0x7f94a81585d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:15.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.505+0000 7f94ae10f640 1 --2- 172.21.15.125:0/2710812917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a80bfec0 0x7f94a81585d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f94900097c0 tx=0x7f949001ba10 comp rx=0 tx=0).stop 2023-10-30T12:46:15.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.505+0000 7f94ae10f640 1 -- 172.21.15.125:0/2710812917 shutdown_connections 2023-10-30T12:46:15.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.505+0000 7f94ae10f640 1 --2- 172.21.15.125:0/2710812917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94880664a0 0x7f9488068960 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.505+0000 7f94ae10f640 1 --2- 172.21.15.125:0/2710812917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94a8159e60 0x7f94a8158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.505+0000 7f94ae10f640 1 --2- 172.21.15.125:0/2710812917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a80bfec0 0x7f94a81585d0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.505+0000 7f94ae10f640 1 --2- 172.21.15.125:0/2710812917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94a806a720 0x7f94a81559e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:15.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.505+0000 7f94ae10f640 1 -- 172.21.15.125:0/2710812917 >> 172.21.15.125:0/2710812917 conn(0x7f94a80c7910 msgr2=0x7f94a806f540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:15.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.505+0000 7f94ae10f640 1 -- 172.21.15.125:0/2710812917 shutdown_connections 2023-10-30T12:46:15.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:15.505+0000 7f94ae10f640 1 -- 172.21.15.125:0/2710812917 wait complete. 2023-10-30T12:46:15.525 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:46:16.441 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:46:16.445 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:46:17.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:17.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc654a46640 1 Processor -- start 2023-10-30T12:46:17.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc654a46640 1 -- start start 2023-10-30T12:46:17.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc654a46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc6501538a0 0x7fc65015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:17.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc654a46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc6500ca0e0 0x7fc65015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:17.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc654a46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6500cb800 con 0x7fc6500cf160 2023-10-30T12:46:17.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc654a46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc65015dc30 con 0x7fc6500ca0e0 2023-10-30T12:46:17.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc654a46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6500cb330 con 0x7fc6501538a0 2023-10-30T12:46:17.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc645d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc6500ca0e0 0x7fc65015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:17.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc645d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc6500ca0e0 0x7fc65015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55668/0 (socket says 172.21.15.125:55668) 2023-10-30T12:46:17.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc64ed9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc6500cf160 0x7fc6500cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37160/0 (socket says 172.21.15.125:37160) 2023-10-30T12:46:17.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc645d9b640 1 -- 172.21.15.125:0/835335781 learned_addr learned my addr 172.21.15.125:0/835335781 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:17.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc64e59c640 1 --2- 172.21.15.125:0/835335781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc6501538a0 0x7fc65015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:17.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc645d9b640 1 -- 172.21.15.125:0/835335781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc6501538a0 msgr2=0x7fc65015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:17.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc645d9b640 1 --2- 172.21.15.125:0/835335781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc6501538a0 0x7fc65015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:17.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc645d9b640 1 -- 172.21.15.125:0/835335781 >> v1:172.21.15.125:6789/0 conn(0x7fc6500cf160 legacy=0x7fc6500cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:17.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc645d9b640 1 -- 172.21.15.125:0/835335781 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6500cad70 con 0x7fc6500ca0e0 2023-10-30T12:46:17.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.673+0000 7fc64dd9b640 1 -- 172.21.15.125:0/835335781 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780987860 0 0) 0x7fc6500cb800 con 0x7fc6500cf160 2023-10-30T12:46:17.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.677+0000 7fc645d9b640 1 --2- 172.21.15.125:0/835335781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6500ca0e0 0x7fc65015a090 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc640009870 tx=0x7fc64001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8933ea1d39d0dba9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:17.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.677+0000 7fc64dd9b640 1 -- 172.21.15.125:0/835335781 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6400a8a30 con 0x7fc6500ca0e0 2023-10-30T12:46:17.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.677+0000 7fc64dd9b640 1 -- 172.21.15.125:0/835335781 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6400a8b90 con 0x7fc6500ca0e0 2023-10-30T12:46:17.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.677+0000 7fc64dd9b640 1 -- 172.21.15.125:0/835335781 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6400b0440 con 0x7fc6500ca0e0 2023-10-30T12:46:17.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.677+0000 7fc654a46640 1 -- 172.21.15.125:0/835335781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6500ca0e0 msgr2=0x7fc65015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:17.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.677+0000 7fc654a46640 1 --2- 172.21.15.125:0/835335781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6500ca0e0 0x7fc65015a090 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fc640009870 tx=0x7fc64001ed30 comp rx=0 tx=0).stop 2023-10-30T12:46:17.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.677+0000 7fc654a46640 1 -- 172.21.15.125:0/835335781 shutdown_connections 2023-10-30T12:46:17.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.677+0000 7fc654a46640 1 --2- 172.21.15.125:0/835335781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6500ca0e0 0x7fc65015a090 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:17.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.677+0000 7fc654a46640 1 --2- 172.21.15.125:0/835335781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc6501538a0 0x7fc65015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:17.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.677+0000 7fc654a46640 1 -- 172.21.15.125:0/835335781 >> 172.21.15.125:0/835335781 conn(0x7fc65006bfe0 msgr2=0x7fc65006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:17.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.677+0000 7fc654a46640 1 -- 172.21.15.125:0/835335781 shutdown_connections 2023-10-30T12:46:17.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.677+0000 7fc654a46640 1 -- 172.21.15.125:0/835335781 wait complete. 2023-10-30T12:46:17.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc654a46640 1 Processor -- start 2023-10-30T12:46:17.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc654a46640 1 -- start start 2023-10-30T12:46:17.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc654a46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc6500ca0e0 0x7fc6501ef500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:17.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc654a46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6500cf160 0x7fc6501e3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:17.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc654a46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6501538a0 0x7fc6501e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:17.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc654a46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6501e4450 con 0x7fc6501538a0 2023-10-30T12:46:17.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc654a46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6501e2ff0 con 0x7fc6500cf160 2023-10-30T12:46:17.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc654a46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6501efa40 con 0x7fc6500ca0e0 2023-10-30T12:46:17.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc64ed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6501538a0 0x7fc6501e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:17.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc64ed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6501538a0 0x7fc6501e5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55680/0 (socket says 172.21.15.125:55680) 2023-10-30T12:46:17.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc64ed9d640 1 -- 172.21.15.125:0/832131650 learned_addr learned my addr 172.21.15.125:0/832131650 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:17.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc64e59c640 1 --2- 172.21.15.125:0/832131650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc6500ca0e0 0x7fc6501ef500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:17.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc645d9b640 1 --2- 172.21.15.125:0/832131650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6500cf160 0x7fc6501e3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:17.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc64ed9d640 1 -- 172.21.15.125:0/832131650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc6500ca0e0 msgr2=0x7fc6501ef500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:17.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc64ed9d640 1 --2- 172.21.15.125:0/832131650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc6500ca0e0 0x7fc6501ef500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:17.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc64ed9d640 1 -- 172.21.15.125:0/832131650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6500cf160 msgr2=0x7fc6501e3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:17.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc64ed9d640 1 --2- 172.21.15.125:0/832131650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6500cf160 0x7fc6501e3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:17.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc64ed9d640 1 -- 172.21.15.125:0/832131650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6400094d0 con 0x7fc6501538a0 2023-10-30T12:46:17.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc64ed9d640 1 --2- 172.21.15.125:0/832131650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6501538a0 0x7fc6501e5260 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc648001570 tx=0x7fc64800cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:17.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc6477fe640 1 -- 172.21.15.125:0/832131650 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64800cea0 con 0x7fc6501538a0 2023-10-30T12:46:17.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc6477fe640 1 -- 172.21.15.125:0/832131650 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64800a5e0 con 0x7fc6501538a0 2023-10-30T12:46:17.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc654a46640 1 -- 172.21.15.125:0/832131650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6501e39c0 con 0x7fc6501538a0 2023-10-30T12:46:17.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc6477fe640 1 -- 172.21.15.125:0/832131650 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64800a990 con 0x7fc6501538a0 2023-10-30T12:46:17.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.681+0000 7fc654a46640 1 -- 172.21.15.125:0/832131650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6501eff90 con 0x7fc6501538a0 2023-10-30T12:46:17.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.685+0000 7fc654a46640 1 -- 172.21.15.125:0/832131650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6500b2a50 con 0x7fc6501538a0 2023-10-30T12:46:17.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.685+0000 7fc6477fe640 1 -- 172.21.15.125:0/832131650 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc64800aaf0 con 0x7fc6501538a0 2023-10-30T12:46:17.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.685+0000 7fc6477fe640 1 --2- 172.21.15.125:0/832131650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6300664a0 0x7fc630068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:17.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.685+0000 7fc6477fe640 1 -- 172.21.15.125:0/832131650 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7fc648083830 con 0x7fc6501538a0 2023-10-30T12:46:17.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.685+0000 7fc64e59c640 1 --2- 172.21.15.125:0/832131650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6300664a0 0x7fc630068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:17.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.685+0000 7fc64e59c640 1 --2- 172.21.15.125:0/832131650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6300664a0 0x7fc630068960 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc63800b930 tx=0x7fc638008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:17.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.689+0000 7fc6477fe640 1 -- 172.21.15.125:0/832131650 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc648050ae0 con 0x7fc6501538a0 2023-10-30T12:46:17.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.869+0000 7fc654a46640 1 -- 172.21.15.125:0/832131650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6501e3160 con 0x7fc6501538a0 2023-10-30T12:46:17.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.873+0000 7fc6477fe640 1 -- 172.21.15.125:0/832131650 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fc648018090 con 0x7fc6501538a0 2023-10-30T12:46:17.872 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:17.873 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5935,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:44.430841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:17.873 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 85 2023-10-30T12:46:17.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.873+0000 7fc654a46640 1 -- 172.21.15.125:0/832131650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6300664a0 msgr2=0x7fc630068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:17.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.873+0000 7fc654a46640 1 --2- 172.21.15.125:0/832131650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6300664a0 0x7fc630068960 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fc63800b930 tx=0x7fc638008040 comp rx=0 tx=0).stop 2023-10-30T12:46:17.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.873+0000 7fc654a46640 1 -- 172.21.15.125:0/832131650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6501538a0 msgr2=0x7fc6501e5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:17.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.873+0000 7fc654a46640 1 --2- 172.21.15.125:0/832131650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6501538a0 0x7fc6501e5260 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc648001570 tx=0x7fc64800cc50 comp rx=0 tx=0).stop 2023-10-30T12:46:17.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.873+0000 7fc654a46640 1 -- 172.21.15.125:0/832131650 shutdown_connections 2023-10-30T12:46:17.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.873+0000 7fc654a46640 1 --2- 172.21.15.125:0/832131650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6300664a0 0x7fc630068960 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:17.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.873+0000 7fc654a46640 1 --2- 172.21.15.125:0/832131650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc6501538a0 0x7fc6501e5260 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:17.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.873+0000 7fc654a46640 1 --2- 172.21.15.125:0/832131650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6500cf160 0x7fc6501e3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:17.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.873+0000 7fc654a46640 1 --2- 172.21.15.125:0/832131650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc6500ca0e0 0x7fc6501ef500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:17.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.873+0000 7fc654a46640 1 -- 172.21.15.125:0/832131650 >> 172.21.15.125:0/832131650 conn(0x7fc65006bfe0 msgr2=0x7fc650150f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:17.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.873+0000 7fc654a46640 1 -- 172.21.15.125:0/832131650 shutdown_connections 2023-10-30T12:46:17.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:17.873+0000 7fc654a46640 1 -- 172.21.15.125:0/832131650 wait complete. 2023-10-30T12:46:17.893 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:46:19.894 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:20.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed78254640 1 Processor -- start 2023-10-30T12:46:20.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed78254640 1 -- start start 2023-10-30T12:46:20.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed78254640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed701538a0 0x7fed7015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:20.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed78254640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed700cb800 con 0x7fed700cf160 2023-10-30T12:46:20.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed78254640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed7015dc30 con 0x7fed700ca0e0 2023-10-30T12:46:20.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed78254640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed700cb330 con 0x7fed701538a0 2023-10-30T12:46:20.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed767f1640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fed700cf160 0x7fed70164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37170/0 (socket says 172.21.15.125:37170) 2023-10-30T12:46:20.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed767f1640 1 -- 172.21.15.125:0/3416907591 learned_addr learned my addr 172.21.15.125:0/3416907591 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:46:20.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed75ff0640 1 --2- 172.21.15.125:0/3416907591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed701538a0 0x7fed7015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:20.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 -- 172.21.15.125:0/3416907591 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936882428 0 0) 0x7fed700cb800 con 0x7fed700cf160 2023-10-30T12:46:20.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 -- 172.21.15.125:0/3416907591 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed5c003530 con 0x7fed700cf160 2023-10-30T12:46:20.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 -- 172.21.15.125:0/3416907591 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1939121150 0 0) 0x7fed5c003530 con 0x7fed700cf160 2023-10-30T12:46:20.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 -- 172.21.15.125:0/3416907591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed701538a0 msgr2=0x7fed7015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:20.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 --2- 172.21.15.125:0/3416907591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed701538a0 0x7fed7015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:20.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 -- 172.21.15.125:0/3416907591 >> v1:172.21.15.136:6789/0 conn(0x7fed700ca0e0 legacy=0x7fed7015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:20.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 -- 172.21.15.125:0/3416907591 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed700cad70 con 0x7fed700cf160 2023-10-30T12:46:20.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 -- 172.21.15.125:0/3416907591 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fed68001a00 con 0x7fed700cf160 2023-10-30T12:46:20.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 -- 172.21.15.125:0/3416907591 >> v1:172.21.15.125:6789/0 conn(0x7fed700cf160 legacy=0x7fed70164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:20.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 --2- 172.21.15.125:0/3416907591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed5c0039b0 0x7fed5c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:20.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 --2- 172.21.15.125:0/3416907591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed5c0047d0 0x7fed5c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:20.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 --2- 172.21.15.125:0/3416907591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed701538a0 0x7fed5c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:20.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 -- 172.21.15.125:0/3416907591 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed700cb330 con 0x7fed5c0047d0 2023-10-30T12:46:20.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 -- 172.21.15.125:0/3416907591 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c00bad0 con 0x7fed5c0039b0 2023-10-30T12:46:20.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.041+0000 7fed74fee640 1 -- 172.21.15.125:0/3416907591 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed5c00bc90 con 0x7fed701538a0 2023-10-30T12:46:20.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.045+0000 7fed757ef640 1 --2- 172.21.15.125:0/3416907591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed5c0047d0 0x7fed5c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:20.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.045+0000 7fed75ff0640 1 --2- 172.21.15.125:0/3416907591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed5c0039b0 0x7fed5c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:20.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.045+0000 7fed767f1640 1 --2- 172.21.15.125:0/3416907591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed701538a0 0x7fed5c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:20.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.045+0000 7fed757ef640 1 -- 172.21.15.125:0/3416907591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed701538a0 msgr2=0x7fed5c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:20.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.045+0000 7fed757ef640 1 --2- 172.21.15.125:0/3416907591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed701538a0 0x7fed5c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:20.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.045+0000 7fed757ef640 1 -- 172.21.15.125:0/3416907591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed5c0039b0 msgr2=0x7fed5c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:20.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.045+0000 7fed757ef640 1 --2- 172.21.15.125:0/3416907591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed5c0039b0 0x7fed5c003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:20.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.045+0000 7fed757ef640 1 -- 172.21.15.125:0/3416907591 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c003530 con 0x7fed5c0047d0 2023-10-30T12:46:20.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.045+0000 7fed757ef640 1 --2- 172.21.15.125:0/3416907591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed5c0047d0 0x7fed5c004c50 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fed600015b0 tx=0x7fed6001c200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=101d9cfecfe630e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:20.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.045+0000 7fed74fee640 1 -- 172.21.15.125:0/3416907591 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed600052d0 con 0x7fed5c0047d0 2023-10-30T12:46:20.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.045+0000 7fed74fee640 1 -- 172.21.15.125:0/3416907591 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed600a7050 con 0x7fed5c0047d0 2023-10-30T12:46:20.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.045+0000 7fed74fee640 1 -- 172.21.15.125:0/3416907591 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed60003600 con 0x7fed5c0047d0 2023-10-30T12:46:20.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed78254640 1 -- 172.21.15.125:0/3416907591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed5c0047d0 msgr2=0x7fed5c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:20.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed78254640 1 --2- 172.21.15.125:0/3416907591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed5c0047d0 0x7fed5c004c50 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fed600015b0 tx=0x7fed6001c200 comp rx=0 tx=0).stop 2023-10-30T12:46:20.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed767f1640 1 -- 172.21.15.125:0/3416907591 reap_dead start 2023-10-30T12:46:20.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed78254640 1 -- 172.21.15.125:0/3416907591 shutdown_connections 2023-10-30T12:46:20.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed78254640 1 -- 172.21.15.125:0/3416907591 >> 172.21.15.125:0/3416907591 conn(0x7fed7006bfe0 msgr2=0x7fed7006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:20.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed78254640 1 -- 172.21.15.125:0/3416907591 shutdown_connections 2023-10-30T12:46:20.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed78254640 1 -- 172.21.15.125:0/3416907591 wait complete. 2023-10-30T12:46:20.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed78254640 1 Processor -- start 2023-10-30T12:46:20.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed78254640 1 -- start start 2023-10-30T12:46:20.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed78254640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed5c0047d0 0x7fed701efdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:20.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed78254640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed700c71f0 0x7fed701e6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:20.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed78254640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed701e6570 0x7fed7015c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:20.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed78254640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed701e69f0 con 0x7fed700c71f0 2023-10-30T12:46:20.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed78254640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7014e7a0 con 0x7fed701e6570 2023-10-30T12:46:20.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed78254640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed701e2ae0 con 0x7fed5c0047d0 2023-10-30T12:46:20.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed757ef640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed700c71f0 0x7fed701e6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:20.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed757ef640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed700c71f0 0x7fed701e6030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55704/0 (socket says 172.21.15.125:55704) 2023-10-30T12:46:20.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed757ef640 1 -- 172.21.15.125:0/4072812859 learned_addr learned my addr 172.21.15.125:0/4072812859 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:20.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed75ff0640 1 --2- 172.21.15.125:0/4072812859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed5c0047d0 0x7fed701efdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:20.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed767f1640 1 --2- 172.21.15.125:0/4072812859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed701e6570 0x7fed7015c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:20.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed757ef640 1 -- 172.21.15.125:0/4072812859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed5c0047d0 msgr2=0x7fed701efdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:20.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed757ef640 1 --2- 172.21.15.125:0/4072812859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed5c0047d0 0x7fed701efdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:20.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed757ef640 1 -- 172.21.15.125:0/4072812859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed701e6570 msgr2=0x7fed7015c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:20.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed757ef640 1 --2- 172.21.15.125:0/4072812859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed701e6570 0x7fed7015c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:20.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.049+0000 7fed757ef640 1 -- 172.21.15.125:0/4072812859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed701f02f0 con 0x7fed700c71f0 2023-10-30T12:46:20.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.053+0000 7fed757ef640 1 --2- 172.21.15.125:0/4072812859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed700c71f0 0x7fed701e6030 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fed600015b0 tx=0x7fed5c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:20.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.053+0000 7fed667fc640 1 -- 172.21.15.125:0/4072812859 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed60000ee0 con 0x7fed700c71f0 2023-10-30T12:46:20.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.053+0000 7fed667fc640 1 -- 172.21.15.125:0/4072812859 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed60007680 con 0x7fed700c71f0 2023-10-30T12:46:20.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.053+0000 7fed667fc640 1 -- 172.21.15.125:0/4072812859 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed60006a00 con 0x7fed700c71f0 2023-10-30T12:46:20.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.053+0000 7fed78254640 1 -- 172.21.15.125:0/4072812859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7015bb10 con 0x7fed700c71f0 2023-10-30T12:46:20.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.053+0000 7fed78254640 1 -- 172.21.15.125:0/4072812859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed701e2d30 con 0x7fed700c71f0 2023-10-30T12:46:20.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.053+0000 7fed78254640 1 -- 172.21.15.125:0/4072812859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed7004e170 con 0x7fed700c71f0 2023-10-30T12:46:20.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.053+0000 7fed667fc640 1 -- 172.21.15.125:0/4072812859 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fed60006b60 con 0x7fed700c71f0 2023-10-30T12:46:20.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.053+0000 7fed667fc640 1 --2- 172.21.15.125:0/4072812859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed540664a0 0x7fed54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:20.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.053+0000 7fed667fc640 1 -- 172.21.15.125:0/4072812859 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7fed6011dec0 con 0x7fed700c71f0 2023-10-30T12:46:20.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.053+0000 7fed75ff0640 1 --2- 172.21.15.125:0/4072812859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed540664a0 0x7fed54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:20.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.057+0000 7fed75ff0640 1 --2- 172.21.15.125:0/4072812859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed540664a0 0x7fed54068960 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009aa0 tx=0x7fed6c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:20.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.057+0000 7fed667fc640 1 -- 172.21.15.125:0/4072812859 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fed600eb210 con 0x7fed700c71f0 2023-10-30T12:46:20.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.249+0000 7fed78254640 1 -- 172.21.15.125:0/4072812859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed7015dc10 con 0x7fed700c71f0 2023-10-30T12:46:20.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.249+0000 7fed667fc640 1 -- 172.21.15.125:0/4072812859 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fed600ee830 con 0x7fed700c71f0 2023-10-30T12:46:20.251 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:20.251 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5935,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:44.430841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:20.251 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 85 2023-10-30T12:46:20.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.253+0000 7fed78254640 1 -- 172.21.15.125:0/4072812859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed540664a0 msgr2=0x7fed54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:20.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.253+0000 7fed78254640 1 --2- 172.21.15.125:0/4072812859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed540664a0 0x7fed54068960 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009aa0 tx=0x7fed6c008040 comp rx=0 tx=0).stop 2023-10-30T12:46:20.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.253+0000 7fed78254640 1 -- 172.21.15.125:0/4072812859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed700c71f0 msgr2=0x7fed701e6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:20.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.253+0000 7fed78254640 1 --2- 172.21.15.125:0/4072812859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed700c71f0 0x7fed701e6030 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fed600015b0 tx=0x7fed5c005800 comp rx=0 tx=0).stop 2023-10-30T12:46:20.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.253+0000 7fed78254640 1 -- 172.21.15.125:0/4072812859 shutdown_connections 2023-10-30T12:46:20.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.253+0000 7fed78254640 1 --2- 172.21.15.125:0/4072812859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed540664a0 0x7fed54068960 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:20.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.253+0000 7fed78254640 1 --2- 172.21.15.125:0/4072812859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fed701e6570 0x7fed7015c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:20.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.253+0000 7fed78254640 1 --2- 172.21.15.125:0/4072812859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fed700c71f0 0x7fed701e6030 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:20.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.253+0000 7fed78254640 1 --2- 172.21.15.125:0/4072812859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fed5c0047d0 0x7fed701efdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:20.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.253+0000 7fed78254640 1 -- 172.21.15.125:0/4072812859 >> 172.21.15.125:0/4072812859 conn(0x7fed7006bfe0 msgr2=0x7fed70150f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:20.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.253+0000 7fed78254640 1 -- 172.21.15.125:0/4072812859 shutdown_connections 2023-10-30T12:46:20.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:20.253+0000 7fed78254640 1 -- 172.21.15.125:0/4072812859 wait complete. 2023-10-30T12:46:20.271 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:46:22.272 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:22.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb482ec9640 1 Processor -- start 2023-10-30T12:46:22.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb482ec9640 1 -- start start 2023-10-30T12:46:22.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb482ec9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb47c06f7a0 0x7fb47c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:22.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb482ec9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb47c0c6bd0 0x7fb47c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:22.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb482ec9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb47c071110 0x7fb47c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:22.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb482ec9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c0700c0 con 0x7fb47c06f7a0 2023-10-30T12:46:22.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb482ec9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c06a3d0 con 0x7fb47c0c6bd0 2023-10-30T12:46:22.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb482ec9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c066cf0 con 0x7fb47c071110 2023-10-30T12:46:22.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb480c65640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb47c06f7a0 0x7fb47c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:22.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb480c65640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb47c06f7a0 0x7fb47c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50130/0 (socket says 172.21.15.125:50130) 2023-10-30T12:46:22.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb480c65640 1 -- 172.21.15.125:0/2932282584 learned_addr learned my addr 172.21.15.125:0/2932282584 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:22.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb473fff640 1 --2- 172.21.15.125:0/2932282584 >> v2:172.21.15.136:3300/0 conn(0x7fb47c0c6bd0 0x7fb47c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:22.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb481466640 1 --2- 172.21.15.125:0/2932282584 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb47c071110 0x7fb47c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:22.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb480c65640 1 -- 172.21.15.125:0/2932282584 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb47c071110 msgr2=0x7fb47c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:22.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.413+0000 7fb480c65640 1 --2- 172.21.15.125:0/2932282584 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb47c071110 0x7fb47c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:22.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb480c65640 1 -- 172.21.15.125:0/2932282584 >> v2:172.21.15.136:3300/0 conn(0x7fb47c0c6bd0 msgr2=0x7fb47c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:22.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb480c65640 1 --2- 172.21.15.125:0/2932282584 >> v2:172.21.15.136:3300/0 conn(0x7fb47c0c6bd0 0x7fb47c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:22.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb480c65640 1 -- 172.21.15.125:0/2932282584 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c0c70d0 con 0x7fb47c06f7a0 2023-10-30T12:46:22.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb480c65640 1 --2- 172.21.15.125:0/2932282584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb47c06f7a0 0x7fb47c06fb80 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb47c06f1c0 tx=0x7fb47801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457eda1300441060 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:22.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb4737fe640 1 -- 172.21.15.125:0/2932282584 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4780a89b0 con 0x7fb47c06f7a0 2023-10-30T12:46:22.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb4737fe640 1 -- 172.21.15.125:0/2932282584 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4780a8b10 con 0x7fb47c06f7a0 2023-10-30T12:46:22.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb4737fe640 1 -- 172.21.15.125:0/2932282584 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4780a8ea0 con 0x7fb47c06f7a0 2023-10-30T12:46:22.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb482ec9640 1 -- 172.21.15.125:0/2932282584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb47c06f7a0 msgr2=0x7fb47c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:22.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb482ec9640 1 --2- 172.21.15.125:0/2932282584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb47c06f7a0 0x7fb47c06fb80 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fb47c06f1c0 tx=0x7fb47801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:46:22.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb482ec9640 1 -- 172.21.15.125:0/2932282584 shutdown_connections 2023-10-30T12:46:22.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb482ec9640 1 --2- 172.21.15.125:0/2932282584 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb47c071110 0x7fb47c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:22.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb482ec9640 1 --2- 172.21.15.125:0/2932282584 >> v2:172.21.15.136:3300/0 conn(0x7fb47c0c6bd0 0x7fb47c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:22.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb482ec9640 1 --2- 172.21.15.125:0/2932282584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb47c06f7a0 0x7fb47c06fb80 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:22.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb482ec9640 1 -- 172.21.15.125:0/2932282584 >> 172.21.15.125:0/2932282584 conn(0x7fb47c0c7910 msgr2=0x7fb47c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:22.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb482ec9640 1 -- 172.21.15.125:0/2932282584 shutdown_connections 2023-10-30T12:46:22.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.417+0000 7fb482ec9640 1 -- 172.21.15.125:0/2932282584 wait complete. 2023-10-30T12:46:22.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb482ec9640 1 Processor -- start 2023-10-30T12:46:22.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb482ec9640 1 -- start start 2023-10-30T12:46:22.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb482ec9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb47c06f7a0 0x7fb47c14e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:22.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb482ec9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb47c071110 0x7fb47c14f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:22.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb482ec9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb47c0c6bd0 0x7fb47c14c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:22.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb482ec9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c156640 con 0x7fb47c071110 2023-10-30T12:46:22.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb482ec9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c14f290 con 0x7fb47c0c6bd0 2023-10-30T12:46:22.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb482ec9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c1563c0 con 0x7fb47c06f7a0 2023-10-30T12:46:22.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb473fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb47c071110 0x7fb47c14f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:22.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb473fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb47c071110 0x7fb47c14f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50136/0 (socket says 172.21.15.125:50136) 2023-10-30T12:46:22.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb473fff640 1 -- 172.21.15.125:0/2874505734 learned_addr learned my addr 172.21.15.125:0/2874505734 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:22.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb480c65640 1 --2- 172.21.15.125:0/2874505734 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb47c06f7a0 0x7fb47c14e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:22.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb481466640 1 --2- 172.21.15.125:0/2874505734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb47c0c6bd0 0x7fb47c14c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:22.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb473fff640 1 -- 172.21.15.125:0/2874505734 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb47c06f7a0 msgr2=0x7fb47c14e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:22.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb473fff640 1 --2- 172.21.15.125:0/2874505734 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb47c06f7a0 0x7fb47c14e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:22.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb473fff640 1 -- 172.21.15.125:0/2874505734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb47c0c6bd0 msgr2=0x7fb47c14c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:22.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb473fff640 1 --2- 172.21.15.125:0/2874505734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb47c0c6bd0 0x7fb47c14c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:22.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb473fff640 1 -- 172.21.15.125:0/2874505734 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4780094d0 con 0x7fb47c071110 2023-10-30T12:46:22.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb473fff640 1 --2- 172.21.15.125:0/2874505734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb47c071110 0x7fb47c14f720 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00d600 tx=0x7fb46c00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:22.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.421+0000 7fb4717fa640 1 -- 172.21.15.125:0/2874505734 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c004240 con 0x7fb47c071110 2023-10-30T12:46:22.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.425+0000 7fb4717fa640 1 -- 172.21.15.125:0/2874505734 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c0043a0 con 0x7fb47c071110 2023-10-30T12:46:22.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.425+0000 7fb482ec9640 1 -- 172.21.15.125:0/2874505734 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c1ef8f0 con 0x7fb47c071110 2023-10-30T12:46:22.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.425+0000 7fb4717fa640 1 -- 172.21.15.125:0/2874505734 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c003840 con 0x7fb47c071110 2023-10-30T12:46:22.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.425+0000 7fb482ec9640 1 -- 172.21.15.125:0/2874505734 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c14ef20 con 0x7fb47c071110 2023-10-30T12:46:22.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.425+0000 7fb482ec9640 1 -- 172.21.15.125:0/2874505734 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c04e170 con 0x7fb47c071110 2023-10-30T12:46:22.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.425+0000 7fb4717fa640 1 -- 172.21.15.125:0/2874505734 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb46c0039a0 con 0x7fb47c071110 2023-10-30T12:46:22.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.425+0000 7fb4717fa640 1 --2- 172.21.15.125:0/2874505734 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb4680664a0 0x7fb468068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:22.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.425+0000 7fb4717fa640 1 -- 172.21.15.125:0/2874505734 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7fb46c084430 con 0x7fb47c071110 2023-10-30T12:46:22.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.425+0000 7fb480c65640 1 --2- 172.21.15.125:0/2874505734 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb4680664a0 0x7fb468068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:22.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.429+0000 7fb480c65640 1 --2- 172.21.15.125:0/2874505734 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb4680664a0 0x7fb468068960 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb4780b2000 tx=0x7fb478008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:22.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.429+0000 7fb4717fa640 1 -- 172.21.15.125:0/2874505734 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb46c051500 con 0x7fb47c071110 2023-10-30T12:46:22.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.609+0000 7fb482ec9640 1 -- 172.21.15.125:0/2874505734 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb47c0c4990 con 0x7fb47c071110 2023-10-30T12:46:22.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.609+0000 7fb4717fa640 1 -- 172.21.15.125:0/2874505734 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fb46c054b20 con 0x7fb47c071110 2023-10-30T12:46:22.612 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:22.612 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5935,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:44.430841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:22.612 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 85 2023-10-30T12:46:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.613+0000 7fb482ec9640 1 -- 172.21.15.125:0/2874505734 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb4680664a0 msgr2=0x7fb468068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.613+0000 7fb482ec9640 1 --2- 172.21.15.125:0/2874505734 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb4680664a0 0x7fb468068960 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fb4780b2000 tx=0x7fb478008040 comp rx=0 tx=0).stop 2023-10-30T12:46:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.613+0000 7fb482ec9640 1 -- 172.21.15.125:0/2874505734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb47c071110 msgr2=0x7fb47c14f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.613+0000 7fb482ec9640 1 --2- 172.21.15.125:0/2874505734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb47c071110 0x7fb47c14f720 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00d600 tx=0x7fb46c00dac0 comp rx=0 tx=0).stop 2023-10-30T12:46:22.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.613+0000 7fb482ec9640 1 -- 172.21.15.125:0/2874505734 shutdown_connections 2023-10-30T12:46:22.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.613+0000 7fb482ec9640 1 --2- 172.21.15.125:0/2874505734 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb4680664a0 0x7fb468068960 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:22.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.613+0000 7fb482ec9640 1 --2- 172.21.15.125:0/2874505734 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb47c0c6bd0 0x7fb47c14c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:22.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.613+0000 7fb482ec9640 1 --2- 172.21.15.125:0/2874505734 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb47c071110 0x7fb47c14f720 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:22.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.613+0000 7fb482ec9640 1 --2- 172.21.15.125:0/2874505734 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb47c06f7a0 0x7fb47c14e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:22.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.613+0000 7fb482ec9640 1 -- 172.21.15.125:0/2874505734 >> 172.21.15.125:0/2874505734 conn(0x7fb47c0c7910 msgr2=0x7fb47c156cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:22.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.613+0000 7fb482ec9640 1 -- 172.21.15.125:0/2874505734 shutdown_connections 2023-10-30T12:46:22.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:22.613+0000 7fb482ec9640 1 -- 172.21.15.125:0/2874505734 wait complete. 2023-10-30T12:46:22.632 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:46:24.632 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:24.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.793+0000 7f6ce0df5640 1 Processor -- start 2023-10-30T12:46:24.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.793+0000 7f6ce0df5640 1 -- start start 2023-10-30T12:46:24.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.793+0000 7f6ce0df5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6cdc06f7a0 0x7f6cdc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:24.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.793+0000 7f6ce0df5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc066cf0 con 0x7f6cdc0c6bd0 2023-10-30T12:46:24.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.793+0000 7f6ce0df5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc0c70d0 con 0x7f6cdc071110 2023-10-30T12:46:24.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.793+0000 7f6ce0df5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cdc0700c0 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.797+0000 7f6cda59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6cdc06f7a0 0x7f6cdc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:24.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.797+0000 7f6cd9d9b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6cdc0c6bd0 0x7f6cdc069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52600/0 (socket says 172.21.15.125:52600) 2023-10-30T12:46:24.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.797+0000 7f6cda59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6cdc06f7a0 0x7f6cdc06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50150/0 (socket says 172.21.15.125:50150) 2023-10-30T12:46:24.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.797+0000 7f6cd9d9b640 1 -- 172.21.15.125:0/1485576728 learned_addr learned my addr 172.21.15.125:0/1485576728 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:46:24.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.797+0000 7f6cd959a640 1 -- 172.21.15.125:0/1485576728 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990928777 0 0) 0x7f6cdc066cf0 con 0x7f6cdc0c6bd0 2023-10-30T12:46:24.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.797+0000 7f6cd959a640 1 -- 172.21.15.125:0/1485576728 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc4003530 con 0x7f6cdc0c6bd0 2023-10-30T12:46:24.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.797+0000 7f6cda59c640 1 -- 172.21.15.125:0/1485576728 >> v1:172.21.15.136:6789/0 conn(0x7f6cdc071110 legacy=0x7f6cdc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:24.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.797+0000 7f6cda59c640 1 -- 172.21.15.125:0/1485576728 >> v1:172.21.15.125:6789/0 conn(0x7f6cdc0c6bd0 legacy=0x7f6cdc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:24.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.797+0000 7f6cda59c640 1 -- 172.21.15.125:0/1485576728 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc0c7390 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.797+0000 7f6cda59c640 1 --2- 172.21.15.125:0/1485576728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cdc06f7a0 0x7f6cdc06fb80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0097c0 tx=0x7f6ccc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ecdc29d924eb68 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:24.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.801+0000 7f6cd959a640 1 -- 172.21.15.125:0/1485576728 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc0a8970 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.801+0000 7f6cd959a640 1 -- 172.21.15.125:0/1485576728 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc0a8ad0 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.801+0000 7f6cd959a640 1 -- 172.21.15.125:0/1485576728 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc0a8e60 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.801+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1485576728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cdc06f7a0 msgr2=0x7f6cdc06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:24.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.801+0000 7f6ce0df5640 1 --2- 172.21.15.125:0/1485576728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cdc06f7a0 0x7f6cdc06fb80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0097c0 tx=0x7f6ccc01ec70 comp rx=0 tx=0).stop 2023-10-30T12:46:24.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.801+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1485576728 shutdown_connections 2023-10-30T12:46:24.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.801+0000 7f6ce0df5640 1 --2- 172.21.15.125:0/1485576728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cdc06f7a0 0x7f6cdc06fb80 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:24.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.801+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1485576728 >> 172.21.15.125:0/1485576728 conn(0x7f6cdc0c7910 msgr2=0x7f6cdc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:24.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.801+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1485576728 shutdown_connections 2023-10-30T12:46:24.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.801+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1485576728 wait complete. 2023-10-30T12:46:24.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.801+0000 7f6ce0df5640 1 Processor -- start 2023-10-30T12:46:24.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.801+0000 7f6ce0df5640 1 -- start start 2023-10-30T12:46:24.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6ce0df5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cdc06f7a0 0x7f6cdc1506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:24.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6ce0df5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cdc071110 0x7f6cdc0af640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:24.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6ce0df5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cdc0c6bd0 0x7f6cdc0ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:24.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6ce0df5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc14fd10 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6ce0df5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0af1b0 con 0x7f6cdc071110 2023-10-30T12:46:24.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6ce0df5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0ae2c0 con 0x7f6cdc0c6bd0 2023-10-30T12:46:24.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6cda59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cdc06f7a0 0x7f6cdc1506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:24.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6cda59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cdc06f7a0 0x7f6cdc1506f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50160/0 (socket says 172.21.15.125:50160) 2023-10-30T12:46:24.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6cda59c640 1 -- 172.21.15.125:0/1395499339 learned_addr learned my addr 172.21.15.125:0/1395499339 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:24.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6cd9d9b640 1 --2- 172.21.15.125:0/1395499339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cdc071110 0x7f6cdc0af640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:24.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6cdad9d640 1 --2- 172.21.15.125:0/1395499339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cdc0c6bd0 0x7f6cdc0ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:24.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6cda59c640 1 -- 172.21.15.125:0/1395499339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cdc0c6bd0 msgr2=0x7f6cdc0ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:24.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6cda59c640 1 --2- 172.21.15.125:0/1395499339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cdc0c6bd0 0x7f6cdc0ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:24.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6cda59c640 1 -- 172.21.15.125:0/1395499339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cdc071110 msgr2=0x7f6cdc0af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:24.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6cda59c640 1 --2- 172.21.15.125:0/1395499339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cdc071110 0x7f6cdc0af640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:24.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6cda59c640 1 -- 172.21.15.125:0/1395499339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ccc009420 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6cda59c640 1 --2- 172.21.15.125:0/1395499339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cdc06f7a0 0x7f6cdc1506f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0b3040 tx=0x7f6ccc01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:24.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6ccaffd640 1 -- 172.21.15.125:0/1395499339 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc0b9630 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6ccaffd640 1 -- 172.21.15.125:0/1395499339 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc0b97b0 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1395499339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc0ae5a0 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6ccaffd640 1 -- 172.21.15.125:0/1395499339 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc0b9b60 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1395499339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc0aeb60 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1395499339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cdc0c7d90 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.805+0000 7f6ccaffd640 1 -- 172.21.15.125:0/1395499339 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6ccc0b7040 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.809+0000 7f6ccaffd640 1 --2- 172.21.15.125:0/1395499339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cbc0664a0 0x7f6cbc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:24.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.809+0000 7f6ccaffd640 1 -- 172.21.15.125:0/1395499339 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7f6ccc122190 con 0x7f6cdc06f7a0 2023-10-30T12:46:24.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.809+0000 7f6cd9d9b640 1 --2- 172.21.15.125:0/1395499339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cbc0664a0 0x7f6cbc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:24.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.809+0000 7f6cd9d9b640 1 --2- 172.21.15.125:0/1395499339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cbc0664a0 0x7f6cbc068960 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc1f2f70 tx=0x7f6cd0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:24.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:24.813+0000 7f6ccaffd640 1 -- 172.21.15.125:0/1395499339 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6ccc0ef410 con 0x7f6cdc06f7a0 2023-10-30T12:46:25.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:25.001+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1395499339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cdc04e170 con 0x7f6cdc06f7a0 2023-10-30T12:46:25.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:25.005+0000 7f6ccaffd640 1 -- 172.21.15.125:0/1395499339 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f6ccc0f2a30 con 0x7f6cdc06f7a0 2023-10-30T12:46:25.004 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:25.005 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5935,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:44.430841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:25.005 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 85 2023-10-30T12:46:25.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:25.005+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1395499339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cbc0664a0 msgr2=0x7f6cbc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:25.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:25.005+0000 7f6ce0df5640 1 --2- 172.21.15.125:0/1395499339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cbc0664a0 0x7f6cbc068960 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc1f2f70 tx=0x7f6cd0009040 comp rx=0 tx=0).stop 2023-10-30T12:46:25.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:25.005+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1395499339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cdc06f7a0 msgr2=0x7f6cdc1506f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:25.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:25.005+0000 7f6ce0df5640 1 --2- 172.21.15.125:0/1395499339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cdc06f7a0 0x7f6cdc1506f0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc0b3040 tx=0x7f6ccc01ba10 comp rx=0 tx=0).stop 2023-10-30T12:46:25.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:25.005+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1395499339 shutdown_connections 2023-10-30T12:46:25.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:25.005+0000 7f6ce0df5640 1 --2- 172.21.15.125:0/1395499339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cbc0664a0 0x7f6cbc068960 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:25.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:25.005+0000 7f6ce0df5640 1 --2- 172.21.15.125:0/1395499339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cdc0c6bd0 0x7f6cdc0ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:25.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:25.005+0000 7f6ce0df5640 1 --2- 172.21.15.125:0/1395499339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cdc071110 0x7f6cdc0af640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:25.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:25.005+0000 7f6ce0df5640 1 --2- 172.21.15.125:0/1395499339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cdc06f7a0 0x7f6cdc1506f0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:25.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:25.005+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1395499339 >> 172.21.15.125:0/1395499339 conn(0x7f6cdc0c7910 msgr2=0x7f6cdc0c61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:25.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:25.005+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1395499339 shutdown_connections 2023-10-30T12:46:25.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:25.005+0000 7f6ce0df5640 1 -- 172.21.15.125:0/1395499339 wait complete. 2023-10-30T12:46:25.025 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:46:27.026 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:27.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.165+0000 7f9e7c617640 1 Processor -- start 2023-10-30T12:46:27.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.165+0000 7f9e7c617640 1 -- start start 2023-10-30T12:46:27.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.169+0000 7f9e7c617640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9e7415d6f0 0x7f9e7415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:27.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.169+0000 7f9e7c617640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9e740cf160 0x7f9e740cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:27.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.169+0000 7f9e7c617640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e740cb800 con 0x7f9e741538a0 2023-10-30T12:46:27.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.169+0000 7f9e7c617640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e740cb330 con 0x7f9e740cf160 2023-10-30T12:46:27.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.169+0000 7f9e7c617640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e740cad70 con 0x7f9e7415d6f0 2023-10-30T12:46:27.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.169+0000 7f9e7abb4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9e740cf160 0x7f9e740cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:27.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.169+0000 7f9e7abb4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9e740cf160 0x7f9e740cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50168/0 (socket says 172.21.15.125:50168) 2023-10-30T12:46:27.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.169+0000 7f9e7abb4640 1 -- 172.21.15.125:0/254413720 learned_addr learned my addr 172.21.15.125:0/254413720 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:27.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.169+0000 7f9e79bb2640 1 --2- 172.21.15.125:0/254413720 >> v2:172.21.15.136:3300/0 conn(0x7f9e7415d6f0 0x7f9e7415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:27.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.169+0000 7f9e7abb4640 1 -- 172.21.15.125:0/254413720 >> v2:172.21.15.136:3300/0 conn(0x7f9e7415d6f0 msgr2=0x7f9e7415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:27.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.169+0000 7f9e7abb4640 1 --2- 172.21.15.125:0/254413720 >> v2:172.21.15.136:3300/0 conn(0x7f9e7415d6f0 0x7f9e7415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:27.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.169+0000 7f9e7abb4640 1 -- 172.21.15.125:0/254413720 >> v1:172.21.15.136:6789/0 conn(0x7f9e741538a0 legacy=0x7f9e740ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:27.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.169+0000 7f9e7abb4640 1 -- 172.21.15.125:0/254413720 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e740cb050 con 0x7f9e740cf160 2023-10-30T12:46:27.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.173+0000 7f9e7abb4640 1 --2- 172.21.15.125:0/254413720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e740cf160 0x7f9e740cf590 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9e740b30a0 tx=0x7f9e6c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44b1c96c5577ca47 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:27.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.173+0000 7f9e793b1640 1 -- 172.21.15.125:0/254413720 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c0af400 con 0x7f9e740cf160 2023-10-30T12:46:27.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.173+0000 7f9e793b1640 1 -- 172.21.15.125:0/254413720 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c0ab050 con 0x7f9e740cf160 2023-10-30T12:46:27.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.173+0000 7f9e793b1640 1 -- 172.21.15.125:0/254413720 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c0af780 con 0x7f9e740cf160 2023-10-30T12:46:27.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.173+0000 7f9e7c617640 1 -- 172.21.15.125:0/254413720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e740cf160 msgr2=0x7f9e740cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:27.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.173+0000 7f9e7c617640 1 --2- 172.21.15.125:0/254413720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e740cf160 0x7f9e740cf590 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f9e740b30a0 tx=0x7f9e6c01f1c0 comp rx=0 tx=0).stop 2023-10-30T12:46:27.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.173+0000 7f9e7c617640 1 -- 172.21.15.125:0/254413720 shutdown_connections 2023-10-30T12:46:27.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.173+0000 7f9e7c617640 1 --2- 172.21.15.125:0/254413720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e740cf160 0x7f9e740cf590 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:27.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.173+0000 7f9e7c617640 1 --2- 172.21.15.125:0/254413720 >> v2:172.21.15.136:3300/0 conn(0x7f9e7415d6f0 0x7f9e7415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:27.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.173+0000 7f9e7c617640 1 -- 172.21.15.125:0/254413720 >> 172.21.15.125:0/254413720 conn(0x7f9e7406bfe0 msgr2=0x7f9e7406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:27.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.173+0000 7f9e7c617640 1 -- 172.21.15.125:0/254413720 shutdown_connections 2023-10-30T12:46:27.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.173+0000 7f9e7c617640 1 -- 172.21.15.125:0/254413720 wait complete. 2023-10-30T12:46:27.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.173+0000 7f9e7c617640 1 Processor -- start 2023-10-30T12:46:27.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.173+0000 7f9e7c617640 1 -- start start 2023-10-30T12:46:27.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7c617640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e740cf160 0x7f9e7414eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:27.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7c617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e741538a0 0x7f9e7414f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:27.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7c617640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e7415d6f0 0x7f9e741f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:27.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7c617640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740b43c0 con 0x7f9e7415d6f0 2023-10-30T12:46:27.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7c617640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e741f33b0 con 0x7f9e740cf160 2023-10-30T12:46:27.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7c617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740b4660 con 0x7f9e741538a0 2023-10-30T12:46:27.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7abb4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e7415d6f0 0x7f9e741f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:27.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7abb4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e7415d6f0 0x7f9e741f2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50176/0 (socket says 172.21.15.125:50176) 2023-10-30T12:46:27.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7abb4640 1 -- 172.21.15.125:0/248746569 learned_addr learned my addr 172.21.15.125:0/248746569 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:27.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7a3b3640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e740cf160 0x7f9e7414eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:27.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e79bb2640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e741538a0 0x7f9e7414f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:27.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7abb4640 1 -- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e741538a0 msgr2=0x7f9e7414f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:27.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7abb4640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e741538a0 0x7f9e7414f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:27.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7abb4640 1 -- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e740cf160 msgr2=0x7f9e7414eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:27.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7abb4640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e740cf160 0x7f9e7414eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:27.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7abb4640 1 -- 172.21.15.125:0/248746569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e6c009b30 con 0x7f9e7415d6f0 2023-10-30T12:46:27.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7a3b3640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e740cf160 0x7f9e7414eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:27.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7abb4640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e7415d6f0 0x7f9e741f2760 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0b2040 tx=0x7f9e6c01bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:27.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e6affd640 1 -- 172.21.15.125:0/248746569 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c0afd10 con 0x7f9e7415d6f0 2023-10-30T12:46:27.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e6affd640 1 -- 172.21.15.125:0/248746569 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c0afe90 con 0x7f9e7415d6f0 2023-10-30T12:46:27.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7c617640 1 -- 172.21.15.125:0/248746569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e740b4940 con 0x7f9e7415d6f0 2023-10-30T12:46:27.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e6affd640 1 -- 172.21.15.125:0/248746569 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c0ba660 con 0x7f9e7415d6f0 2023-10-30T12:46:27.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7c617640 1 -- 172.21.15.125:0/248746569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e740b4f00 con 0x7f9e7415d6f0 2023-10-30T12:46:27.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e7c617640 1 -- 172.21.15.125:0/248746569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e740b2a50 con 0x7f9e7415d6f0 2023-10-30T12:46:27.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.177+0000 7f9e6affd640 1 -- 172.21.15.125:0/248746569 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9e6c0b6040 con 0x7f9e7415d6f0 2023-10-30T12:46:27.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.181+0000 7f9e6affd640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9e580664a0 0x7f9e58068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:27.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.181+0000 7f9e6affd640 1 -- 172.21.15.125:0/248746569 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7f9e6c122350 con 0x7f9e7415d6f0 2023-10-30T12:46:27.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.181+0000 7f9e7a3b3640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9e580664a0 0x7f9e58068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:27.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.181+0000 7f9e7a3b3640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9e580664a0 0x7f9e58068960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9e70000f70 tx=0x7f9e70009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:27.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.185+0000 7f9e6affd640 1 -- 172.21.15.125:0/248746569 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9e6c0ef5d0 con 0x7f9e7415d6f0 2023-10-30T12:46:27.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.365+0000 7f9e7c617640 1 -- 172.21.15.125:0/248746569 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e7404e170 con 0x7f9e7415d6f0 2023-10-30T12:46:27.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.365+0000 7f9e6affd640 1 -- 172.21.15.125:0/248746569 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f9e6c009510 con 0x7f9e7415d6f0 2023-10-30T12:46:27.366 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:27.366 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5935,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:44.430841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:27.367 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 85 2023-10-30T12:46:27.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.365+0000 7f9e7c617640 1 -- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9e580664a0 msgr2=0x7f9e58068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:27.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.365+0000 7f9e7c617640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9e580664a0 0x7f9e58068960 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9e70000f70 tx=0x7f9e70009040 comp rx=0 tx=0).stop 2023-10-30T12:46:27.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.365+0000 7f9e7c617640 1 -- 172.21.15.125:0/248746569 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e7415d6f0 msgr2=0x7f9e741f2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:27.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.365+0000 7f9e7c617640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e7415d6f0 0x7f9e741f2760 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f9e6c0b2040 tx=0x7f9e6c01bf60 comp rx=0 tx=0).stop 2023-10-30T12:46:27.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.365+0000 7f9e7c617640 1 -- 172.21.15.125:0/248746569 shutdown_connections 2023-10-30T12:46:27.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.365+0000 7f9e7c617640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9e580664a0 0x7f9e58068960 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:27.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.365+0000 7f9e7c617640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e7415d6f0 0x7f9e741f2760 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:27.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.365+0000 7f9e7c617640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e741538a0 0x7f9e7414f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:27.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.365+0000 7f9e7c617640 1 --2- 172.21.15.125:0/248746569 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e740cf160 0x7f9e7414eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:27.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.365+0000 7f9e7c617640 1 -- 172.21.15.125:0/248746569 >> 172.21.15.125:0/248746569 conn(0x7f9e7406bfe0 msgr2=0x7f9e7415b680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:27.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.365+0000 7f9e7c617640 1 -- 172.21.15.125:0/248746569 shutdown_connections 2023-10-30T12:46:27.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:27.365+0000 7f9e7c617640 1 -- 172.21.15.125:0/248746569 wait complete. 2023-10-30T12:46:27.386 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:46:29.387 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:29.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.541+0000 7f497dbaa640 1 Processor -- start 2023-10-30T12:46:29.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.541+0000 7f497dbaa640 1 -- start start 2023-10-30T12:46:29.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.545+0000 7f497dbaa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f497806f7a0 0x7f497806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:29.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.545+0000 7f497dbaa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4978071110 0x7f4978158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:29.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.545+0000 7f497dbaa640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4978066cf0 con 0x7f49780c6bd0 2023-10-30T12:46:29.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.545+0000 7f497dbaa640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49780700c0 con 0x7f497806f7a0 2023-10-30T12:46:29.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.545+0000 7f497dbaa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49780c70d0 con 0x7f4978071110 2023-10-30T12:46:29.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.545+0000 7f49777fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f497806f7a0 0x7f497806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:29.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.545+0000 7f49777fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f497806f7a0 0x7f497806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50192/0 (socket says 172.21.15.125:50192) 2023-10-30T12:46:29.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.545+0000 7f49777fe640 1 -- 172.21.15.125:0/15778443 learned_addr learned my addr 172.21.15.125:0/15778443 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:29.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.545+0000 7f4977fff640 1 --2- 172.21.15.125:0/15778443 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4978071110 0x7f4978158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:29.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.545+0000 7f49777fe640 1 -- 172.21.15.125:0/15778443 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4978071110 msgr2=0x7f4978158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:29.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.545+0000 7f49777fe640 1 --2- 172.21.15.125:0/15778443 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4978071110 0x7f4978158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:29.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.545+0000 7f49777fe640 1 -- 172.21.15.125:0/15778443 >> v1:172.21.15.136:6789/0 conn(0x7f49780c6bd0 legacy=0x7f4978069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:29.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.545+0000 7f49777fe640 1 -- 172.21.15.125:0/15778443 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49780c7390 con 0x7f497806f7a0 2023-10-30T12:46:29.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.549+0000 7f49777fe640 1 --2- 172.21.15.125:0/15778443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f497806f7a0 0x7f497806fb80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4968009870 tx=0x7f496801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f49b1f5815ed5ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:29.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.549+0000 7f49767fc640 1 -- 172.21.15.125:0/15778443 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49680a8a30 con 0x7f497806f7a0 2023-10-30T12:46:29.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.549+0000 7f49767fc640 1 -- 172.21.15.125:0/15778443 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49680a8b90 con 0x7f497806f7a0 2023-10-30T12:46:29.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.549+0000 7f49767fc640 1 -- 172.21.15.125:0/15778443 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49680b0440 con 0x7f497806f7a0 2023-10-30T12:46:29.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.549+0000 7f497dbaa640 1 -- 172.21.15.125:0/15778443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f497806f7a0 msgr2=0x7f497806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:29.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.549+0000 7f497dbaa640 1 --2- 172.21.15.125:0/15778443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f497806f7a0 0x7f497806fb80 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f4968009870 tx=0x7f496801ed30 comp rx=0 tx=0).stop 2023-10-30T12:46:29.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.549+0000 7f497dbaa640 1 -- 172.21.15.125:0/15778443 shutdown_connections 2023-10-30T12:46:29.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.549+0000 7f497dbaa640 1 --2- 172.21.15.125:0/15778443 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4978071110 0x7f4978158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:29.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.549+0000 7f497dbaa640 1 --2- 172.21.15.125:0/15778443 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f497806f7a0 0x7f497806fb80 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:29.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.549+0000 7f497dbaa640 1 -- 172.21.15.125:0/15778443 >> 172.21.15.125:0/15778443 conn(0x7f49780c7910 msgr2=0x7f49780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:29.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.549+0000 7f497dbaa640 1 -- 172.21.15.125:0/15778443 shutdown_connections 2023-10-30T12:46:29.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.549+0000 7f497dbaa640 1 -- 172.21.15.125:0/15778443 wait complete. 2023-10-30T12:46:29.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.549+0000 7f497dbaa640 1 Processor -- start 2023-10-30T12:46:29.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.549+0000 7f497dbaa640 1 -- start start 2023-10-30T12:46:29.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f497dbaa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f497806f7a0 0x7f49780af4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:29.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f497dbaa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4978071110 0x7f49780c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:29.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f497dbaa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49780c6bd0 0x7f49781f1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:29.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f497dbaa640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49781ef140 con 0x7f49780c6bd0 2023-10-30T12:46:29.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f497dbaa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4978158020 con 0x7f497806f7a0 2023-10-30T12:46:29.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f497dbaa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4978158290 con 0x7f4978071110 2023-10-30T12:46:29.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f4977fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49780c6bd0 0x7f49781f1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:29.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f4977fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49780c6bd0 0x7f49781f1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50200/0 (socket says 172.21.15.125:50200) 2023-10-30T12:46:29.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f4977fff640 1 -- 172.21.15.125:0/3664257941 learned_addr learned my addr 172.21.15.125:0/3664257941 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:29.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f49777fe640 1 --2- 172.21.15.125:0/3664257941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f497806f7a0 0x7f49780af4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:29.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f4976ffd640 1 --2- 172.21.15.125:0/3664257941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4978071110 0x7f49780c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:29.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f4977fff640 1 -- 172.21.15.125:0/3664257941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4978071110 msgr2=0x7f49780c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:29.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f4977fff640 1 --2- 172.21.15.125:0/3664257941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4978071110 0x7f49780c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:29.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f4977fff640 1 -- 172.21.15.125:0/3664257941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f497806f7a0 msgr2=0x7f49780af4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:29.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f4977fff640 1 --2- 172.21.15.125:0/3664257941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f497806f7a0 0x7f49780af4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:29.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f4977fff640 1 -- 172.21.15.125:0/3664257941 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49680094d0 con 0x7f49780c6bd0 2023-10-30T12:46:29.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f4977fff640 1 --2- 172.21.15.125:0/3664257941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49780c6bd0 0x7f49781f1d50 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4970007e70 tx=0x7f497000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:29.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f495ffff640 1 -- 172.21.15.125:0/3664257941 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4970014620 con 0x7f49780c6bd0 2023-10-30T12:46:29.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f495ffff640 1 -- 172.21.15.125:0/3664257941 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49700147a0 con 0x7f49780c6bd0 2023-10-30T12:46:29.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f497dbaa640 1 -- 172.21.15.125:0/3664257941 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49781eeaf0 con 0x7f49780c6bd0 2023-10-30T12:46:29.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f495ffff640 1 -- 172.21.15.125:0/3664257941 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4970014b50 con 0x7f49780c6bd0 2023-10-30T12:46:29.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f497dbaa640 1 -- 172.21.15.125:0/3664257941 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49781ee7a0 con 0x7f49780c6bd0 2023-10-30T12:46:29.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f497dbaa640 1 -- 172.21.15.125:0/3664257941 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49781ef2f0 con 0x7f49780c6bd0 2023-10-30T12:46:29.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.553+0000 7f495ffff640 1 -- 172.21.15.125:0/3664257941 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4970014cb0 con 0x7f49780c6bd0 2023-10-30T12:46:29.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.557+0000 7f495ffff640 1 --2- 172.21.15.125:0/3664257941 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49640664d0 0x7f4964068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:29.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.557+0000 7f495ffff640 1 -- 172.21.15.125:0/3664257941 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6340+0+0 (secure 0 0 0) 0x7f49700849b0 con 0x7f49780c6bd0 2023-10-30T12:46:29.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.557+0000 7f49777fe640 1 --2- 172.21.15.125:0/3664257941 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49640664d0 0x7f4964068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:29.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.557+0000 7f49777fe640 1 --2- 172.21.15.125:0/3664257941 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49640664d0 0x7f4964068990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f49680b2000 tx=0x7f4968009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:29.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.561+0000 7f495ffff640 1 -- 172.21.15.125:0/3664257941 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f49781ef2f0 con 0x7f49780c6bd0 2023-10-30T12:46:29.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.737+0000 7f497dbaa640 1 -- 172.21.15.125:0/3664257941 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f497804e170 con 0x7f49780c6bd0 2023-10-30T12:46:29.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.741+0000 7f495ffff640 1 -- 172.21.15.125:0/3664257941 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f4970051bd0 con 0x7f49780c6bd0 2023-10-30T12:46:29.740 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:29.741 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":5935,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":85,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:44:44.430841+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5433,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5433":{"gid":5433,"name":"e","rank":0,"incarnation":62,"state":"up:active","state_seq":69,"addr":"172.21.15.125:6836/2595438444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2595438444},{"type":"v1","addr":"172.21.15.125:6836","nonce":2595438444}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:29.741 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 85 2023-10-30T12:46:29.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.741+0000 7f497dbaa640 1 -- 172.21.15.125:0/3664257941 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49640664d0 msgr2=0x7f4964068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:29.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.741+0000 7f497dbaa640 1 --2- 172.21.15.125:0/3664257941 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49640664d0 0x7f4964068990 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f49680b2000 tx=0x7f4968009040 comp rx=0 tx=0).stop 2023-10-30T12:46:29.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.741+0000 7f497dbaa640 1 -- 172.21.15.125:0/3664257941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49780c6bd0 msgr2=0x7f49781f1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:29.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.741+0000 7f497dbaa640 1 --2- 172.21.15.125:0/3664257941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49780c6bd0 0x7f49781f1d50 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f4970007e70 tx=0x7f497000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:46:29.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.741+0000 7f497dbaa640 1 -- 172.21.15.125:0/3664257941 shutdown_connections 2023-10-30T12:46:29.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.741+0000 7f497dbaa640 1 --2- 172.21.15.125:0/3664257941 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49640664d0 0x7f4964068990 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:29.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.741+0000 7f497dbaa640 1 --2- 172.21.15.125:0/3664257941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49780c6bd0 0x7f49781f1d50 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:29.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.741+0000 7f497dbaa640 1 --2- 172.21.15.125:0/3664257941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4978071110 0x7f49780c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:29.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.741+0000 7f497dbaa640 1 --2- 172.21.15.125:0/3664257941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f497806f7a0 0x7f49780af4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:29.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.741+0000 7f497dbaa640 1 -- 172.21.15.125:0/3664257941 >> 172.21.15.125:0/3664257941 conn(0x7f49780c7910 msgr2=0x7f49780c93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:29.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.741+0000 7f497dbaa640 1 -- 172.21.15.125:0/3664257941 shutdown_connections 2023-10-30T12:46:29.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:29.741+0000 7f497dbaa640 1 -- 172.21.15.125:0/3664257941 wait complete. 2023-10-30T12:46:29.761 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:46:31.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:31.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.909+0000 7fba45fc5640 1 Processor -- start 2023-10-30T12:46:31.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.909+0000 7fba45fc5640 1 -- start start 2023-10-30T12:46:31.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.913+0000 7fba45fc5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fba4015d6f0 0x7fba4015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:31.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.913+0000 7fba45fc5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fba400cf160 0x7fba400cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:31.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.913+0000 7fba45fc5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba400cb800 con 0x7fba401538a0 2023-10-30T12:46:31.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.913+0000 7fba45fc5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba400cb330 con 0x7fba400cf160 2023-10-30T12:46:31.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.913+0000 7fba45fc5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba400cad70 con 0x7fba4015d6f0 2023-10-30T12:46:31.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.913+0000 7fba3ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fba400cf160 0x7fba400cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:31.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.913+0000 7fba3ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fba400cf160 0x7fba400cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51768/0 (socket says 172.21.15.125:51768) 2023-10-30T12:46:31.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.913+0000 7fba3ffff640 1 -- 172.21.15.125:0/316974830 learned_addr learned my addr 172.21.15.125:0/316974830 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:31.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.913+0000 7fba3effd640 1 --2- 172.21.15.125:0/316974830 >> v2:172.21.15.136:3300/0 conn(0x7fba4015d6f0 0x7fba4015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:31.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.913+0000 7fba3ffff640 1 -- 172.21.15.125:0/316974830 >> v2:172.21.15.136:3300/0 conn(0x7fba4015d6f0 msgr2=0x7fba4015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:31.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.913+0000 7fba3ffff640 1 --2- 172.21.15.125:0/316974830 >> v2:172.21.15.136:3300/0 conn(0x7fba4015d6f0 0x7fba4015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:31.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.913+0000 7fba3ffff640 1 -- 172.21.15.125:0/316974830 >> v1:172.21.15.136:6789/0 conn(0x7fba401538a0 legacy=0x7fba400ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:31.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.913+0000 7fba3ffff640 1 -- 172.21.15.125:0/316974830 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba400cb050 con 0x7fba400cf160 2023-10-30T12:46:31.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.917+0000 7fba3ffff640 1 --2- 172.21.15.125:0/316974830 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba400cf160 0x7fba400cf590 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fba400b30a0 tx=0x7fba3801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46744f83e2509560 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:31.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.917+0000 7fba3e7fc640 1 -- 172.21.15.125:0/316974830 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba380af400 con 0x7fba400cf160 2023-10-30T12:46:31.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.917+0000 7fba3e7fc640 1 -- 172.21.15.125:0/316974830 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba380ab050 con 0x7fba400cf160 2023-10-30T12:46:31.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.917+0000 7fba3e7fc640 1 -- 172.21.15.125:0/316974830 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba380af780 con 0x7fba400cf160 2023-10-30T12:46:31.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.917+0000 7fba45fc5640 1 -- 172.21.15.125:0/316974830 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba400cf160 msgr2=0x7fba400cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:31.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.917+0000 7fba45fc5640 1 --2- 172.21.15.125:0/316974830 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba400cf160 0x7fba400cf590 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fba400b30a0 tx=0x7fba3801f1c0 comp rx=0 tx=0).stop 2023-10-30T12:46:31.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.917+0000 7fba45fc5640 1 -- 172.21.15.125:0/316974830 shutdown_connections 2023-10-30T12:46:31.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.917+0000 7fba45fc5640 1 --2- 172.21.15.125:0/316974830 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba400cf160 0x7fba400cf590 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:31.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.917+0000 7fba45fc5640 1 --2- 172.21.15.125:0/316974830 >> v2:172.21.15.136:3300/0 conn(0x7fba4015d6f0 0x7fba4015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:31.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.917+0000 7fba45fc5640 1 -- 172.21.15.125:0/316974830 >> 172.21.15.125:0/316974830 conn(0x7fba4006bfe0 msgr2=0x7fba4006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:31.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.917+0000 7fba45fc5640 1 -- 172.21.15.125:0/316974830 shutdown_connections 2023-10-30T12:46:31.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.917+0000 7fba45fc5640 1 -- 172.21.15.125:0/316974830 wait complete. 2023-10-30T12:46:31.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba45fc5640 1 Processor -- start 2023-10-30T12:46:31.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba45fc5640 1 -- start start 2023-10-30T12:46:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba45fc5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba400cf160 0x7fba4014dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba45fc5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba401538a0 0x7fba401591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba45fc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba4015d6f0 0x7fba401e5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:31.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba45fc5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba401e4e60 con 0x7fba401538a0 2023-10-30T12:46:31.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba45fc5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba401596f0 con 0x7fba400cf160 2023-10-30T12:46:31.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba45fc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba40158480 con 0x7fba4015d6f0 2023-10-30T12:46:31.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba3effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba401538a0 0x7fba401591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:31.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba3effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba401538a0 0x7fba401591b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51772/0 (socket says 172.21.15.125:51772) 2023-10-30T12:46:31.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba3effd640 1 -- 172.21.15.125:0/2182345798 learned_addr learned my addr 172.21.15.125:0/2182345798 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:31.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba3f7fe640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba400cf160 0x7fba4014dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:31.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba3ffff640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba4015d6f0 0x7fba401e5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:31.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba3effd640 1 -- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba4015d6f0 msgr2=0x7fba401e5100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:31.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba3effd640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba4015d6f0 0x7fba401e5100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:31.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba3effd640 1 -- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba400cf160 msgr2=0x7fba4014dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:31.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba3effd640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba400cf160 0x7fba4014dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:31.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba3effd640 1 -- 172.21.15.125:0/2182345798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba38009b30 con 0x7fba401538a0 2023-10-30T12:46:31.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba3ffff640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba4015d6f0 0x7fba401e5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:31.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba3f7fe640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba400cf160 0x7fba4014dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:31.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba3effd640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba401538a0 0x7fba401591b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fba3400e8a0 tx=0x7fba3400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:31.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba27fff640 1 -- 172.21.15.125:0/2182345798 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba34009510 con 0x7fba401538a0 2023-10-30T12:46:31.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba27fff640 1 -- 172.21.15.125:0/2182345798 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba3400c8e0 con 0x7fba401538a0 2023-10-30T12:46:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba27fff640 1 -- 172.21.15.125:0/2182345798 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba3400cc50 con 0x7fba401538a0 2023-10-30T12:46:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba45fc5640 1 -- 172.21.15.125:0/2182345798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba40158d20 con 0x7fba401538a0 2023-10-30T12:46:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.921+0000 7fba45fc5640 1 -- 172.21.15.125:0/2182345798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba401586d0 con 0x7fba401538a0 2023-10-30T12:46:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.925+0000 7fba45fc5640 1 -- 172.21.15.125:0/2182345798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba400b2a50 con 0x7fba401538a0 2023-10-30T12:46:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.925+0000 7fba27fff640 1 -- 172.21.15.125:0/2182345798 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fba34013470 con 0x7fba401538a0 2023-10-30T12:46:31.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.925+0000 7fba27fff640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fba2c0664a0 0x7fba2c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:31.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.925+0000 7fba27fff640 1 -- 172.21.15.125:0/2182345798 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7fba34084170 con 0x7fba401538a0 2023-10-30T12:46:31.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.925+0000 7fba3f7fe640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fba2c0664a0 0x7fba2c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:31.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.929+0000 7fba3f7fe640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fba2c0664a0 0x7fba2c068960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fba30000f70 tx=0x7fba30009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:31.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:31.929+0000 7fba27fff640 1 -- 172.21.15.125:0/2182345798 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fba340513b0 con 0x7fba401538a0 2023-10-30T12:46:32.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.117+0000 7fba45fc5640 1 -- 172.21.15.125:0/2182345798 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba40158eb0 con 0x7fba401538a0 2023-10-30T12:46:32.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.117+0000 7fba27fff640 1 -- 172.21.15.125:0/2182345798 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5736 (secure 0 0 0) 0x7fba340549d0 con 0x7fba401538a0 2023-10-30T12:46:32.119 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:32.119 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:30.228037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:32.119 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 86 2023-10-30T12:46:32.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.121+0000 7fba45fc5640 1 -- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fba2c0664a0 msgr2=0x7fba2c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:32.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.121+0000 7fba45fc5640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fba2c0664a0 0x7fba2c068960 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fba30000f70 tx=0x7fba30009040 comp rx=0 tx=0).stop 2023-10-30T12:46:32.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.121+0000 7fba45fc5640 1 -- 172.21.15.125:0/2182345798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba401538a0 msgr2=0x7fba401591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:32.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.121+0000 7fba45fc5640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba401538a0 0x7fba401591b0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fba3400e8a0 tx=0x7fba3400eca0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.121+0000 7fba45fc5640 1 -- 172.21.15.125:0/2182345798 shutdown_connections 2023-10-30T12:46:32.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.121+0000 7fba45fc5640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fba2c0664a0 0x7fba2c068960 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.121+0000 7fba45fc5640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba4015d6f0 0x7fba401e5100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.121+0000 7fba45fc5640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba401538a0 0x7fba401591b0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.121+0000 7fba45fc5640 1 --2- 172.21.15.125:0/2182345798 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba400cf160 0x7fba4014dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.121+0000 7fba45fc5640 1 -- 172.21.15.125:0/2182345798 >> 172.21.15.125:0/2182345798 conn(0x7fba4006bfe0 msgr2=0x7fba400c4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:32.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.121+0000 7fba45fc5640 1 -- 172.21.15.125:0/2182345798 shutdown_connections 2023-10-30T12:46:32.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.121+0000 7fba45fc5640 1 -- 172.21.15.125:0/2182345798 wait complete. 2023-10-30T12:46:32.140 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T12:46:32.140 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:46:32.140 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:32.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.277+0000 7fbf1ec17640 1 Processor -- start 2023-10-30T12:46:32.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.277+0000 7fbf1ec17640 1 -- start start 2023-10-30T12:46:32.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.277+0000 7fbf1ec17640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbf180c6b80 0x7fbf180bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:32.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.277+0000 7fbf1ec17640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbf180c6f60 0x7fbf180c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:32.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.277+0000 7fbf1ec17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf180c8430 0x7fbf1813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:32.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.277+0000 7fbf1ec17640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf180c0a50 con 0x7fbf180c6b80 2023-10-30T12:46:32.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.277+0000 7fbf1ec17640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf1806f6e0 con 0x7fbf180c6f60 2023-10-30T12:46:32.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.277+0000 7fbf1ec17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1806f9c0 con 0x7fbf180c8430 2023-10-30T12:46:32.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.277+0000 7fbf1c9b3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbf180c6b80 0x7fbf180bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:32.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.277+0000 7fbf1c9b3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbf180c6b80 0x7fbf180bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51786/0 (socket says 172.21.15.125:51786) 2023-10-30T12:46:32.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.277+0000 7fbf1c9b3640 1 -- 172.21.15.125:0/425485807 learned_addr learned my addr 172.21.15.125:0/425485807 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:32.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.277+0000 7fbf17fff640 1 --2- 172.21.15.125:0/425485807 >> v2:172.21.15.136:3300/0 conn(0x7fbf180c6f60 0x7fbf180c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:32.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.277+0000 7fbf1d1b4640 1 --2- 172.21.15.125:0/425485807 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf180c8430 0x7fbf1813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:32.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.281+0000 7fbf1c9b3640 1 -- 172.21.15.125:0/425485807 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf180c8430 msgr2=0x7fbf1813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:32.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.281+0000 7fbf1c9b3640 1 --2- 172.21.15.125:0/425485807 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf180c8430 0x7fbf1813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.281+0000 7fbf1c9b3640 1 -- 172.21.15.125:0/425485807 >> v2:172.21.15.136:3300/0 conn(0x7fbf180c6f60 msgr2=0x7fbf180c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:32.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.281+0000 7fbf1c9b3640 1 --2- 172.21.15.125:0/425485807 >> v2:172.21.15.136:3300/0 conn(0x7fbf180c6f60 0x7fbf180c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.281+0000 7fbf1c9b3640 1 -- 172.21.15.125:0/425485807 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1806fca0 con 0x7fbf180c6b80 2023-10-30T12:46:32.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.281+0000 7fbf1c9b3640 1 --2- 172.21.15.125:0/425485807 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf180c6b80 0x7fbf180bfec0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fbf18159b90 tx=0x7fbf0c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ee0f9a94cce841 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:32.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.281+0000 7fbf177fe640 1 -- 172.21.15.125:0/425485807 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c0ac630 con 0x7fbf180c6b80 2023-10-30T12:46:32.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.281+0000 7fbf177fe640 1 -- 172.21.15.125:0/425485807 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0c0ac790 con 0x7fbf180c6b80 2023-10-30T12:46:32.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.281+0000 7fbf177fe640 1 -- 172.21.15.125:0/425485807 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c0acb20 con 0x7fbf180c6b80 2023-10-30T12:46:32.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 -- 172.21.15.125:0/425485807 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf180c6b80 msgr2=0x7fbf180bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:32.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 --2- 172.21.15.125:0/425485807 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf180c6b80 0x7fbf180bfec0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fbf18159b90 tx=0x7fbf0c01c500 comp rx=0 tx=0).stop 2023-10-30T12:46:32.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 -- 172.21.15.125:0/425485807 shutdown_connections 2023-10-30T12:46:32.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 --2- 172.21.15.125:0/425485807 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf180c8430 0x7fbf1813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 --2- 172.21.15.125:0/425485807 >> v2:172.21.15.136:3300/0 conn(0x7fbf180c6f60 0x7fbf180c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 --2- 172.21.15.125:0/425485807 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf180c6b80 0x7fbf180bfec0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 -- 172.21.15.125:0/425485807 >> 172.21.15.125:0/425485807 conn(0x7fbf180c7910 msgr2=0x7fbf180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:32.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 -- 172.21.15.125:0/425485807 shutdown_connections 2023-10-30T12:46:32.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 -- 172.21.15.125:0/425485807 wait complete. 2023-10-30T12:46:32.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 Processor -- start 2023-10-30T12:46:32.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 -- start start 2023-10-30T12:46:32.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf180c6b80 0x7fbf1813b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:32.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf180c6f60 0x7fbf1806b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:32.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf180c8430 0x7fbf1813a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:32.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf18139ea0 con 0x7fbf180c8430 2023-10-30T12:46:32.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf181f2790 con 0x7fbf180c6b80 2023-10-30T12:46:32.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1ec17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf181f2a70 con 0x7fbf180c6f60 2023-10-30T12:46:32.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1d1b4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf180c8430 0x7fbf1813a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1d1b4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf180c8430 0x7fbf1813a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51800/0 (socket says 172.21.15.125:51800) 2023-10-30T12:46:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1d1b4640 1 -- 172.21.15.125:0/1906842949 learned_addr learned my addr 172.21.15.125:0/1906842949 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1c9b3640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf180c6b80 0x7fbf1813b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf17fff640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf180c6f60 0x7fbf1806b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:32.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1d1b4640 1 -- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf180c6f60 msgr2=0x7fbf1806b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1d1b4640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf180c6f60 0x7fbf1806b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1d1b4640 1 -- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf180c6b80 msgr2=0x7fbf1813b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1d1b4640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf180c6b80 0x7fbf1813b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1d1b4640 1 -- 172.21.15.125:0/1906842949 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c0094d0 con 0x7fbf180c8430 2023-10-30T12:46:32.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.285+0000 7fbf1c9b3640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf180c6b80 0x7fbf1813b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.289+0000 7fbf1d1b4640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf180c8430 0x7fbf1813a330 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fbf10007e60 tx=0x7fbf1000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.289+0000 7fbf157fa640 1 -- 172.21.15.125:0/1906842949 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf10014420 con 0x7fbf180c8430 2023-10-30T12:46:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.289+0000 7fbf157fa640 1 -- 172.21.15.125:0/1906842949 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf100145a0 con 0x7fbf180c8430 2023-10-30T12:46:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.289+0000 7fbf1ec17640 1 -- 172.21.15.125:0/1906842949 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf181f2d50 con 0x7fbf180c8430 2023-10-30T12:46:32.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.289+0000 7fbf157fa640 1 -- 172.21.15.125:0/1906842949 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf10014950 con 0x7fbf180c8430 2023-10-30T12:46:32.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.289+0000 7fbf1ec17640 1 -- 172.21.15.125:0/1906842949 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf180c1690 con 0x7fbf180c8430 2023-10-30T12:46:32.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.289+0000 7fbf1ec17640 1 -- 172.21.15.125:0/1906842949 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf18139c70 con 0x7fbf180c8430 2023-10-30T12:46:32.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.289+0000 7fbf157fa640 1 -- 172.21.15.125:0/1906842949 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbf10014ab0 con 0x7fbf180c8430 2023-10-30T12:46:32.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.289+0000 7fbf157fa640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbefc0664d0 0x7fbefc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:32.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.289+0000 7fbf157fa640 1 -- 172.21.15.125:0/1906842949 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7fbf10084ac0 con 0x7fbf180c8430 2023-10-30T12:46:32.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.289+0000 7fbf1c9b3640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbefc0664d0 0x7fbefc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:32.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.293+0000 7fbf1c9b3640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbefc0664d0 0x7fbefc068990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0ae000 tx=0x7fbf0c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:32.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.293+0000 7fbf157fa640 1 -- 172.21.15.125:0/1906842949 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbf18139c70 con 0x7fbf180c8430 2023-10-30T12:46:32.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.489+0000 7fbf1ec17640 1 -- 172.21.15.125:0/1906842949 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf18139c70 con 0x7fbf180c8430 2023-10-30T12:46:32.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.489+0000 7fbf157fa640 1 -- 172.21.15.125:0/1906842949 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5736 (secure 0 0 0) 0x7fbf18139c70 con 0x7fbf180c8430 2023-10-30T12:46:32.491 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:32.491 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:30.228037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:32.492 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 86 2023-10-30T12:46:32.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.493+0000 7fbf1ec17640 1 -- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbefc0664d0 msgr2=0x7fbefc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:32.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.493+0000 7fbf1ec17640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbefc0664d0 0x7fbefc068990 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0ae000 tx=0x7fbf0c008040 comp rx=0 tx=0).stop 2023-10-30T12:46:32.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.493+0000 7fbf1ec17640 1 -- 172.21.15.125:0/1906842949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf180c8430 msgr2=0x7fbf1813a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:32.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.493+0000 7fbf1ec17640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf180c8430 0x7fbf1813a330 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fbf10007e60 tx=0x7fbf1000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.493+0000 7fbf1ec17640 1 -- 172.21.15.125:0/1906842949 shutdown_connections 2023-10-30T12:46:32.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.493+0000 7fbf1ec17640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbefc0664d0 0x7fbefc068990 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.493+0000 7fbf1ec17640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf180c8430 0x7fbf1813a330 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.493+0000 7fbf1ec17640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf180c6f60 0x7fbf1806b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.493+0000 7fbf1ec17640 1 --2- 172.21.15.125:0/1906842949 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf180c6b80 0x7fbf1813b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:32.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.493+0000 7fbf1ec17640 1 -- 172.21.15.125:0/1906842949 >> 172.21.15.125:0/1906842949 conn(0x7fbf180c7910 msgr2=0x7fbf180c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:32.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.493+0000 7fbf1ec17640 1 -- 172.21.15.125:0/1906842949 shutdown_connections 2023-10-30T12:46:32.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:32.493+0000 7fbf1ec17640 1 -- 172.21.15.125:0/1906842949 wait complete. 2023-10-30T12:46:32.514 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 86, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 86, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5935": { "addr": "172.21.15.136:6839/1510290691", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 1510290691, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 1510290691, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5935, "incarnation": 86, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_5988": { "addr": "172.21.15.125:6837/3156415951", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 3156415951, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 3156415951, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5988, "incarnation": 78, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 28 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:46:30.228037+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5935, "mds_1": 5988 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:46:34.516 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:34.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.657+0000 7f5514b0e640 1 Processor -- start 2023-10-30T12:46:34.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.657+0000 7f5514b0e640 1 -- start start 2023-10-30T12:46:34.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f5514b0e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f55101538a0 0x7f551015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:34.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f5514b0e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55100cb800 con 0x7f55100cf160 2023-10-30T12:46:34.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f5514b0e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551015dc30 con 0x7f55100ca0e0 2023-10-30T12:46:34.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f5514b0e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55100cb330 con 0x7f55101538a0 2023-10-30T12:46:34.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550ed9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f55100cf160 0x7f5510164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45740/0 (socket says 172.21.15.125:45740) 2023-10-30T12:46:34.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550ed9d640 1 -- 172.21.15.125:0/3579587781 learned_addr learned my addr 172.21.15.125:0/3579587781 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:46:34.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550e59c640 1 --2- 172.21.15.125:0/3579587781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f55101538a0 0x7f551015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:34.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 -- 172.21.15.125:0/3579587781 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117473022 0 0) 0x7f55100cb800 con 0x7f55100cf160 2023-10-30T12:46:34.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 -- 172.21.15.125:0/3579587781 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54fc003530 con 0x7f55100cf160 2023-10-30T12:46:34.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 -- 172.21.15.125:0/3579587781 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1829359610 0 0) 0x7f54fc003530 con 0x7f55100cf160 2023-10-30T12:46:34.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 -- 172.21.15.125:0/3579587781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f55101538a0 msgr2=0x7f551015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:34.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 --2- 172.21.15.125:0/3579587781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f55101538a0 0x7f551015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:34.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 -- 172.21.15.125:0/3579587781 >> v1:172.21.15.136:6789/0 conn(0x7f55100ca0e0 legacy=0x7f551015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:34.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 -- 172.21.15.125:0/3579587781 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55100cad70 con 0x7f55100cf160 2023-10-30T12:46:34.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 -- 172.21.15.125:0/3579587781 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f5508001680 con 0x7f55100cf160 2023-10-30T12:46:34.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 -- 172.21.15.125:0/3579587781 >> v1:172.21.15.125:6789/0 conn(0x7f55100cf160 legacy=0x7f5510164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:34.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 --2- 172.21.15.125:0/3579587781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f54fc0039b0 0x7f54fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:34.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 --2- 172.21.15.125:0/3579587781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f54fc004360 0x7f54fc004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:34.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 --2- 172.21.15.125:0/3579587781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54fc004fe0 0x7f54fc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:34.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 -- 172.21.15.125:0/3579587781 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55100cb330 con 0x7f54fc004fe0 2023-10-30T12:46:34.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 -- 172.21.15.125:0/3579587781 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc00bed0 con 0x7f54fc0039b0 2023-10-30T12:46:34.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550dd9b640 1 -- 172.21.15.125:0/3579587781 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc00c010 con 0x7f54fc004360 2023-10-30T12:46:34.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550ed9d640 1 --2- 172.21.15.125:0/3579587781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54fc004fe0 0x7f54fc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:34.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f5505d9b640 1 --2- 172.21.15.125:0/3579587781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f54fc004360 0x7f54fc004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:34.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550e59c640 1 --2- 172.21.15.125:0/3579587781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f54fc0039b0 0x7f54fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:34.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550ed9d640 1 -- 172.21.15.125:0/3579587781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f54fc004360 msgr2=0x7f54fc004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:34.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550ed9d640 1 --2- 172.21.15.125:0/3579587781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f54fc004360 0x7f54fc004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:34.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550ed9d640 1 -- 172.21.15.125:0/3579587781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f54fc0039b0 msgr2=0x7f54fc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:34.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550ed9d640 1 --2- 172.21.15.125:0/3579587781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f54fc0039b0 0x7f54fc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:34.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.661+0000 7f550ed9d640 1 -- 172.21.15.125:0/3579587781 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54fc003530 con 0x7f54fc004fe0 2023-10-30T12:46:34.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.665+0000 7f550ed9d640 1 --2- 172.21.15.125:0/3579587781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54fc004fe0 0x7f54fc00b990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f550800c1c0 tx=0x7f550801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f629a33376905606 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:34.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.665+0000 7f550dd9b640 1 -- 172.21.15.125:0/3579587781 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55080a5b20 con 0x7f54fc004fe0 2023-10-30T12:46:34.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.665+0000 7f550dd9b640 1 -- 172.21.15.125:0/3579587781 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55080a5c80 con 0x7f54fc004fe0 2023-10-30T12:46:34.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.665+0000 7f550dd9b640 1 -- 172.21.15.125:0/3579587781 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55080058e0 con 0x7f54fc004fe0 2023-10-30T12:46:34.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.665+0000 7f5514b0e640 1 -- 172.21.15.125:0/3579587781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54fc004fe0 msgr2=0x7f54fc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:34.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.665+0000 7f5514b0e640 1 --2- 172.21.15.125:0/3579587781 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54fc004fe0 0x7f54fc00b990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f550800c1c0 tx=0x7f550801cc20 comp rx=0 tx=0).stop 2023-10-30T12:46:34.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.665+0000 7f550ed9d640 1 -- 172.21.15.125:0/3579587781 reap_dead start 2023-10-30T12:46:34.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.665+0000 7f5514b0e640 1 -- 172.21.15.125:0/3579587781 shutdown_connections 2023-10-30T12:46:34.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.665+0000 7f5514b0e640 1 -- 172.21.15.125:0/3579587781 >> 172.21.15.125:0/3579587781 conn(0x7f551006bfe0 msgr2=0x7f551006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:34.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.665+0000 7f5514b0e640 1 -- 172.21.15.125:0/3579587781 shutdown_connections 2023-10-30T12:46:34.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.665+0000 7f5514b0e640 1 -- 172.21.15.125:0/3579587781 wait complete. 2023-10-30T12:46:34.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f5514b0e640 1 Processor -- start 2023-10-30T12:46:34.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f5514b0e640 1 -- start start 2023-10-30T12:46:34.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f5514b0e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54fc004fe0 0x7f5510158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:34.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f5514b0e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f55100b19f0 0x7f5510158e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:34.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f5514b0e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f551015d1d0 0x7f551014f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:34.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f5514b0e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551014e560 con 0x7f54fc004fe0 2023-10-30T12:46:34.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f5514b0e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55101efb40 con 0x7f551015d1d0 2023-10-30T12:46:34.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f5514b0e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f551014e790 con 0x7f55100b19f0 2023-10-30T12:46:34.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f550e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54fc004fe0 0x7f5510158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:34.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f550e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54fc004fe0 0x7f5510158940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51824/0 (socket says 172.21.15.125:51824) 2023-10-30T12:46:34.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f550e59c640 1 -- 172.21.15.125:0/545094083 learned_addr learned my addr 172.21.15.125:0/545094083 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:34.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f5505d9b640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f55100b19f0 0x7f5510158e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:34.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f550ed9d640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f551015d1d0 0x7f551014f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f550e59c640 1 -- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f55100b19f0 msgr2=0x7f5510158e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f550e59c640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f55100b19f0 0x7f5510158e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f550e59c640 1 -- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f551015d1d0 msgr2=0x7f551014f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f550e59c640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f551015d1d0 0x7f551014f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f550e59c640 1 -- 172.21.15.125:0/545094083 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55101ef210 con 0x7f54fc004fe0 2023-10-30T12:46:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f550ed9d640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f551015d1d0 0x7f551014f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f550e59c640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54fc004fe0 0x7f5510158940 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f5500011cb0 tx=0x7f550000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f55077fe640 1 -- 172.21.15.125:0/545094083 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550000e040 con 0x7f54fc004fe0 2023-10-30T12:46:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f55077fe640 1 -- 172.21.15.125:0/545094083 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550000f040 con 0x7f54fc004fe0 2023-10-30T12:46:34.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f5514b0e640 1 -- 172.21.15.125:0/545094083 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55101ef4f0 con 0x7f54fc004fe0 2023-10-30T12:46:34.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f55077fe640 1 -- 172.21.15.125:0/545094083 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5500003bc0 con 0x7f54fc004fe0 2023-10-30T12:46:34.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.669+0000 7f5514b0e640 1 -- 172.21.15.125:0/545094083 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55101eef00 con 0x7f54fc004fe0 2023-10-30T12:46:34.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.673+0000 7f5514b0e640 1 -- 172.21.15.125:0/545094083 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f551004e170 con 0x7f54fc004fe0 2023-10-30T12:46:34.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.673+0000 7f55077fe640 1 -- 172.21.15.125:0/545094083 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5500003d20 con 0x7f54fc004fe0 2023-10-30T12:46:34.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.673+0000 7f55077fe640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f54f00664d0 0x7f54f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:34.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.673+0000 7f55077fe640 1 -- 172.21.15.125:0/545094083 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f5500084ac0 con 0x7f54fc004fe0 2023-10-30T12:46:34.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.673+0000 7f5505d9b640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f54f00664d0 0x7f54f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:34.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.673+0000 7f5505d9b640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f54f00664d0 0x7f54f0068990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f54f8001280 tx=0x7f54f8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:34.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.677+0000 7f55077fe640 1 -- 172.21.15.125:0/545094083 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5500051d00 con 0x7f54fc004fe0 2023-10-30T12:46:34.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.957+0000 7f5514b0e640 1 -- 172.21.15.125:0/545094083 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55100b1dd0 con 0x7f54fc004fe0 2023-10-30T12:46:34.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.957+0000 7f55077fe640 1 -- 172.21.15.125:0/545094083 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f5500023090 con 0x7f54fc004fe0 2023-10-30T12:46:34.960 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:34.960 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:30.228037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:34.961 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 86 2023-10-30T12:46:34.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.961+0000 7f5514b0e640 1 -- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f54f00664d0 msgr2=0x7f54f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:34.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.961+0000 7f5514b0e640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f54f00664d0 0x7f54f0068990 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f54f8001280 tx=0x7f54f8009040 comp rx=0 tx=0).stop 2023-10-30T12:46:34.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.961+0000 7f5514b0e640 1 -- 172.21.15.125:0/545094083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54fc004fe0 msgr2=0x7f5510158940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:34.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.961+0000 7f5514b0e640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54fc004fe0 0x7f5510158940 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f5500011cb0 tx=0x7f550000f580 comp rx=0 tx=0).stop 2023-10-30T12:46:34.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.961+0000 7f5514b0e640 1 -- 172.21.15.125:0/545094083 shutdown_connections 2023-10-30T12:46:34.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.961+0000 7f5514b0e640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f54f00664d0 0x7f54f0068990 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:34.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.961+0000 7f5514b0e640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f551015d1d0 0x7f551014f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:34.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.961+0000 7f5514b0e640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f55100b19f0 0x7f5510158e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:34.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.961+0000 7f5514b0e640 1 --2- 172.21.15.125:0/545094083 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f54fc004fe0 0x7f5510158940 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:34.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.961+0000 7f5514b0e640 1 -- 172.21.15.125:0/545094083 >> 172.21.15.125:0/545094083 conn(0x7f551006bfe0 msgr2=0x7f5510159f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:34.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.961+0000 7f5514b0e640 1 -- 172.21.15.125:0/545094083 shutdown_connections 2023-10-30T12:46:34.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:34.961+0000 7f5514b0e640 1 -- 172.21.15.125:0/545094083 wait complete. 2023-10-30T12:46:34.980 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:46:36.981 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:37.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.121+0000 7f379bf9d640 1 Processor -- start 2023-10-30T12:46:37.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.121+0000 7f379bf9d640 1 -- start start 2023-10-30T12:46:37.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f379bf9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f37940c6b80 0x7f37940bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:37.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f379bf9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6f60 0x7f37940c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:37.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f379bf9d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37940c0a50 con 0x7f37940c8430 2023-10-30T12:46:37.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f379bf9d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f379406f6e0 con 0x7f37940c6b80 2023-10-30T12:46:37.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f379bf9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379406f9c0 con 0x7f37940c6f60 2023-10-30T12:46:37.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f3799d39640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f37940c6b80 0x7f37940bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:37.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f379a53a640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f37940c8430 0x7f379413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:60834/0 (socket says 172.21.15.125:60834) 2023-10-30T12:46:37.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f379a53a640 1 -- 172.21.15.125:0/485675502 learned_addr learned my addr 172.21.15.125:0/485675502 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:46:37.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f3799538640 1 --2- 172.21.15.125:0/485675502 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6f60 0x7f37940c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:37.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f3798d37640 1 -- 172.21.15.125:0/485675502 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558358788 0 0) 0x7f37940c0a50 con 0x7f37940c8430 2023-10-30T12:46:37.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f3798d37640 1 -- 172.21.15.125:0/485675502 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3780003530 con 0x7f37940c8430 2023-10-30T12:46:37.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f3799d39640 1 -- 172.21.15.125:0/485675502 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6f60 msgr2=0x7f37940c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:37.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f3799d39640 1 --2- 172.21.15.125:0/485675502 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6f60 0x7f37940c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:37.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f3799d39640 1 -- 172.21.15.125:0/485675502 >> v1:172.21.15.136:6789/0 conn(0x7f37940c8430 legacy=0x7f379413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:37.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.125+0000 7f3799d39640 1 -- 172.21.15.125:0/485675502 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379406fca0 con 0x7f37940c6b80 2023-10-30T12:46:37.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f3799d39640 1 --2- 172.21.15.125:0/485675502 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37940c6b80 0x7f37940bfec0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f37880097c0 tx=0x7f378801c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1561ec02ca141974 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:37.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f3798d37640 1 -- 172.21.15.125:0/485675502 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37880ab600 con 0x7f37940c6b80 2023-10-30T12:46:37.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f3798d37640 1 -- 172.21.15.125:0/485675502 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37880a7050 con 0x7f37940c6b80 2023-10-30T12:46:37.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f3798d37640 1 -- 172.21.15.125:0/485675502 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37880ab960 con 0x7f37940c6b80 2023-10-30T12:46:37.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f379bf9d640 1 -- 172.21.15.125:0/485675502 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37940c6b80 msgr2=0x7f37940bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:37.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f379bf9d640 1 --2- 172.21.15.125:0/485675502 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37940c6b80 0x7f37940bfec0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f37880097c0 tx=0x7f378801c3f0 comp rx=0 tx=0).stop 2023-10-30T12:46:37.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f379bf9d640 1 -- 172.21.15.125:0/485675502 shutdown_connections 2023-10-30T12:46:37.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f379bf9d640 1 --2- 172.21.15.125:0/485675502 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6f60 0x7f37940c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:37.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f379bf9d640 1 --2- 172.21.15.125:0/485675502 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37940c6b80 0x7f37940bfec0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:37.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f379bf9d640 1 -- 172.21.15.125:0/485675502 >> 172.21.15.125:0/485675502 conn(0x7f37940c7910 msgr2=0x7f37940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:37.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f379bf9d640 1 -- 172.21.15.125:0/485675502 shutdown_connections 2023-10-30T12:46:37.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f379bf9d640 1 -- 172.21.15.125:0/485675502 wait complete. 2023-10-30T12:46:37.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f379bf9d640 1 Processor -- start 2023-10-30T12:46:37.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f379bf9d640 1 -- start start 2023-10-30T12:46:37.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f379bf9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37940c6b80 0x7f37941f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:37.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.129+0000 7f379bf9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6f60 0x7f37941f1350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:37.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379bf9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37940c8430 0x7f37940c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:37.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379bf9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37941ef5c0 con 0x7f37940c8430 2023-10-30T12:46:37.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379bf9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37941ef7b0 con 0x7f37940c6b80 2023-10-30T12:46:37.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379bf9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37941efa90 con 0x7f37940c6f60 2023-10-30T12:46:37.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379a53a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37940c8430 0x7f37940c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:37.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f3799d39640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37940c6b80 0x7f37941f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:37.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379a53a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37940c8430 0x7f37940c5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51836/0 (socket says 172.21.15.125:51836) 2023-10-30T12:46:37.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f3799538640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6f60 0x7f37941f1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:37.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379a53a640 1 -- 172.21.15.125:0/888572891 learned_addr learned my addr 172.21.15.125:0/888572891 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:37.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f3799d39640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37940c6b80 0x7f37941f0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:34820/0 (socket says 172.21.15.125:34820) 2023-10-30T12:46:37.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379a53a640 1 -- 172.21.15.125:0/888572891 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6f60 msgr2=0x7f37941f1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:37.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379a53a640 1 --2- 172.21.15.125:0/888572891 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6f60 0x7f37941f1350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:37.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379a53a640 1 -- 172.21.15.125:0/888572891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37940c6b80 msgr2=0x7f37941f0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:37.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379a53a640 1 --2- 172.21.15.125:0/888572891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37940c6b80 0x7f37941f0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:37.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379a53a640 1 -- 172.21.15.125:0/888572891 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3788009420 con 0x7f37940c8430 2023-10-30T12:46:37.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379a53a640 1 --2- 172.21.15.125:0/888572891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37940c8430 0x7f37940c5a70 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f378c00caa0 tx=0x7f378c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:37.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f37867fc640 1 -- 172.21.15.125:0/888572891 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c00ac00 con 0x7f37940c8430 2023-10-30T12:46:37.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f37867fc640 1 -- 172.21.15.125:0/888572891 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f378c00ad60 con 0x7f37940c8430 2023-10-30T12:46:37.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379bf9d640 1 -- 172.21.15.125:0/888572891 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37941efd70 con 0x7f37940c8430 2023-10-30T12:46:37.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f37867fc640 1 -- 172.21.15.125:0/888572891 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c012620 con 0x7f37940c8430 2023-10-30T12:46:37.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379bf9d640 1 -- 172.21.15.125:0/888572891 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37941f0330 con 0x7f37940c8430 2023-10-30T12:46:37.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f379bf9d640 1 -- 172.21.15.125:0/888572891 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37941f0550 con 0x7f37940c8430 2023-10-30T12:46:37.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.133+0000 7f37867fc640 1 -- 172.21.15.125:0/888572891 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f378c007070 con 0x7f37940c8430 2023-10-30T12:46:37.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.137+0000 7f37867fc640 1 --2- 172.21.15.125:0/888572891 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37780664a0 0x7f3778068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:37.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.137+0000 7f37867fc640 1 -- 172.21.15.125:0/888572891 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f378c084140 con 0x7f37940c8430 2023-10-30T12:46:37.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.137+0000 7f3799d39640 1 --2- 172.21.15.125:0/888572891 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37780664a0 0x7f3778068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:37.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.137+0000 7f3799d39640 1 --2- 172.21.15.125:0/888572891 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37780664a0 0x7f3778068960 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f37880ad000 tx=0x7f37880b6040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:37.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.141+0000 7f37867fc640 1 -- 172.21.15.125:0/888572891 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f378c051270 con 0x7f37940c8430 2023-10-30T12:46:37.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.357+0000 7f379bf9d640 1 -- 172.21.15.125:0/888572891 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f379404e170 con 0x7f37940c8430 2023-10-30T12:46:37.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.357+0000 7f37867fc640 1 -- 172.21.15.125:0/888572891 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f378c017090 con 0x7f37940c8430 2023-10-30T12:46:37.360 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:37.360 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:30.228037+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:37.361 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 86 2023-10-30T12:46:37.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.361+0000 7f379bf9d640 1 -- 172.21.15.125:0/888572891 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37780664a0 msgr2=0x7f3778068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:37.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.361+0000 7f379bf9d640 1 --2- 172.21.15.125:0/888572891 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37780664a0 0x7f3778068960 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f37880ad000 tx=0x7f37880b6040 comp rx=0 tx=0).stop 2023-10-30T12:46:37.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.361+0000 7f379bf9d640 1 -- 172.21.15.125:0/888572891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37940c8430 msgr2=0x7f37940c5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:37.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.361+0000 7f379bf9d640 1 --2- 172.21.15.125:0/888572891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37940c8430 0x7f37940c5a70 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f378c00caa0 tx=0x7f378c00cf60 comp rx=0 tx=0).stop 2023-10-30T12:46:37.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.361+0000 7f379bf9d640 1 -- 172.21.15.125:0/888572891 shutdown_connections 2023-10-30T12:46:37.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.361+0000 7f379bf9d640 1 --2- 172.21.15.125:0/888572891 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37780664a0 0x7f3778068960 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:37.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.361+0000 7f379bf9d640 1 --2- 172.21.15.125:0/888572891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37940c8430 0x7f37940c5a70 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:37.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.361+0000 7f379bf9d640 1 --2- 172.21.15.125:0/888572891 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6f60 0x7f37941f1350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:37.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.361+0000 7f379bf9d640 1 --2- 172.21.15.125:0/888572891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37940c6b80 0x7f37941f0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:37.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.361+0000 7f379bf9d640 1 -- 172.21.15.125:0/888572891 >> 172.21.15.125:0/888572891 conn(0x7f37940c7910 msgr2=0x7f37941532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:37.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.361+0000 7f379bf9d640 1 -- 172.21.15.125:0/888572891 shutdown_connections 2023-10-30T12:46:37.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:37.361+0000 7f379bf9d640 1 -- 172.21.15.125:0/888572891 wait complete. 2023-10-30T12:46:37.380 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:46:39.382 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:39.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.521+0000 7f1d4c032640 1 Processor -- start 2023-10-30T12:46:39.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.521+0000 7f1d4c032640 1 -- start start 2023-10-30T12:46:39.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.525+0000 7f1d4c032640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d441539a0 0x7f1d4415a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:39.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.525+0000 7f1d4c032640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1d44069e90 0x7f1d4406a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:39.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.525+0000 7f1d4c032640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d4415a8c0 con 0x7f1d44070a00 2023-10-30T12:46:39.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.525+0000 7f1d4c032640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d4406fbb0 con 0x7f1d44069e90 2023-10-30T12:46:39.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.525+0000 7f1d4c032640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d44153d80 con 0x7f1d441539a0 2023-10-30T12:46:39.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.525+0000 7f1d4a5cf640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1d44069e90 0x7f1d4406a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:39.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.525+0000 7f1d49dce640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d441539a0 0x7f1d4415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:39.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.525+0000 7f1d495cd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1d44070a00 0x7f1d44070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:60842/0 (socket says 172.21.15.125:60842) 2023-10-30T12:46:39.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.525+0000 7f1d495cd640 1 -- 172.21.15.125:0/4122643781 learned_addr learned my addr 172.21.15.125:0/4122643781 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:46:39.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.525+0000 7f1d4a5cf640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1d44069e90 0x7f1d4406a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:34834/0 (socket says 172.21.15.125:34834) 2023-10-30T12:46:39.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.525+0000 7f1d4a5cf640 1 -- 172.21.15.125:0/4122643781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d441539a0 msgr2=0x7f1d4415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:39.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.525+0000 7f1d4a5cf640 1 --2- 172.21.15.125:0/4122643781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d441539a0 0x7f1d4415a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:39.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.525+0000 7f1d4a5cf640 1 -- 172.21.15.125:0/4122643781 >> v1:172.21.15.136:6789/0 conn(0x7f1d44070a00 legacy=0x7f1d44070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:39.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.525+0000 7f1d4a5cf640 1 -- 172.21.15.125:0/4122643781 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d440c5c50 con 0x7f1d44069e90 2023-10-30T12:46:39.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.529+0000 7f1d4a5cf640 1 --2- 172.21.15.125:0/4122643781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d44069e90 0x7f1d4406a300 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c009d90 tx=0x7f1d3c01c890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99f4a6dab21cb5c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:39.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.529+0000 7f1d48dcc640 1 -- 172.21.15.125:0/4122643781 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c0a5860 con 0x7f1d44069e90 2023-10-30T12:46:39.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.529+0000 7f1d48dcc640 1 -- 172.21.15.125:0/4122643781 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d3c0a59c0 con 0x7f1d44069e90 2023-10-30T12:46:39.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.529+0000 7f1d48dcc640 1 -- 172.21.15.125:0/4122643781 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d3c0a5d50 con 0x7f1d44069e90 2023-10-30T12:46:39.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.529+0000 7f1d4c032640 1 -- 172.21.15.125:0/4122643781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d44069e90 msgr2=0x7f1d4406a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:39.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.529+0000 7f1d4c032640 1 --2- 172.21.15.125:0/4122643781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d44069e90 0x7f1d4406a300 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c009d90 tx=0x7f1d3c01c890 comp rx=0 tx=0).stop 2023-10-30T12:46:39.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.529+0000 7f1d4c032640 1 -- 172.21.15.125:0/4122643781 shutdown_connections 2023-10-30T12:46:39.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.529+0000 7f1d4c032640 1 --2- 172.21.15.125:0/4122643781 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d44069e90 0x7f1d4406a300 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:39.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.529+0000 7f1d4c032640 1 --2- 172.21.15.125:0/4122643781 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d441539a0 0x7f1d4415a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:39.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.529+0000 7f1d4c032640 1 -- 172.21.15.125:0/4122643781 >> 172.21.15.125:0/4122643781 conn(0x7f1d440c7910 msgr2=0x7f1d440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:39.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.529+0000 7f1d4c032640 1 -- 172.21.15.125:0/4122643781 shutdown_connections 2023-10-30T12:46:39.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.529+0000 7f1d4c032640 1 -- 172.21.15.125:0/4122643781 wait complete. 2023-10-30T12:46:39.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.529+0000 7f1d4c032640 1 Processor -- start 2023-10-30T12:46:39.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.529+0000 7f1d4c032640 1 -- start start 2023-10-30T12:46:39.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d4c032640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d44069e90 0x7f1d440c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:39.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d4c032640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d44070a00 0x7f1d44144c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:39.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d4c032640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d441539a0 0x7f1d441f2060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:39.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d4c032640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d44155d00 con 0x7f1d44069e90 2023-10-30T12:46:39.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d4c032640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d44154290 con 0x7f1d44070a00 2023-10-30T12:46:39.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d4c032640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d440c9150 con 0x7f1d441539a0 2023-10-30T12:46:39.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d49dce640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d44069e90 0x7f1d440c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:39.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d49dce640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d44069e90 0x7f1d440c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51850/0 (socket says 172.21.15.125:51850) 2023-10-30T12:46:39.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d49dce640 1 -- 172.21.15.125:0/3039620766 learned_addr learned my addr 172.21.15.125:0/3039620766 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:39.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d495cd640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d44070a00 0x7f1d44144c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:39.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d4a5cf640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d441539a0 0x7f1d441f2060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:39.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d49dce640 1 -- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d441539a0 msgr2=0x7f1d441f2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:39.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d49dce640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d441539a0 0x7f1d441f2060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:39.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d49dce640 1 -- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d44070a00 msgr2=0x7f1d44144c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:39.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d49dce640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d44070a00 0x7f1d44144c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:39.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d49dce640 1 -- 172.21.15.125:0/3039620766 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c009a70 con 0x7f1d44069e90 2023-10-30T12:46:39.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d495cd640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d44070a00 0x7f1d44144c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:39.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d49dce640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d44069e90 0x7f1d440c8c10 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f1d4000e870 tx=0x7f1d4000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:39.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d3a7fc640 1 -- 172.21.15.125:0/3039620766 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d40009510 con 0x7f1d44069e90 2023-10-30T12:46:39.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d3a7fc640 1 -- 172.21.15.125:0/3039620766 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d4000eec0 con 0x7f1d44069e90 2023-10-30T12:46:39.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d3a7fc640 1 -- 172.21.15.125:0/3039620766 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4000cb40 con 0x7f1d44069e90 2023-10-30T12:46:39.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d4c032640 1 -- 172.21.15.125:0/3039620766 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d44156990 con 0x7f1d44069e90 2023-10-30T12:46:39.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d4c032640 1 -- 172.21.15.125:0/3039620766 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d44154530 con 0x7f1d44069e90 2023-10-30T12:46:39.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.533+0000 7f1d4c032640 1 -- 172.21.15.125:0/3039620766 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d4404e170 con 0x7f1d44069e90 2023-10-30T12:46:39.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.537+0000 7f1d3a7fc640 1 -- 172.21.15.125:0/3039620766 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1d4000cca0 con 0x7f1d44069e90 2023-10-30T12:46:39.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.537+0000 7f1d3a7fc640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1d280664a0 0x7f1d28068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:39.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.537+0000 7f1d3a7fc640 1 -- 172.21.15.125:0/3039620766 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f1d40084310 con 0x7f1d44069e90 2023-10-30T12:46:39.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.537+0000 7f1d495cd640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1d280664a0 0x7f1d28068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:39.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.537+0000 7f1d495cd640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1d280664a0 0x7f1d28068960 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1d4406ff60 tx=0x7f1d34009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:39.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.541+0000 7f1d3a7fc640 1 -- 172.21.15.125:0/3039620766 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1d40051430 con 0x7f1d44069e90 2023-10-30T12:46:39.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.721+0000 7f1d4c032640 1 -- 172.21.15.125:0/3039620766 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d440c72f0 con 0x7f1d44069e90 2023-10-30T12:46:39.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.721+0000 7f1d3a7fc640 1 -- 172.21.15.125:0/3039620766 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f1d40054a50 con 0x7f1d44069e90 2023-10-30T12:46:39.724 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:39.724 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:39.169108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:resolve","state_seq":33,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:39.725 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 87 2023-10-30T12:46:39.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.725+0000 7f1d4c032640 1 -- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1d280664a0 msgr2=0x7f1d28068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:39.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.725+0000 7f1d4c032640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1d280664a0 0x7f1d28068960 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1d4406ff60 tx=0x7f1d34009040 comp rx=0 tx=0).stop 2023-10-30T12:46:39.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.725+0000 7f1d4c032640 1 -- 172.21.15.125:0/3039620766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d44069e90 msgr2=0x7f1d440c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:39.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.725+0000 7f1d4c032640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d44069e90 0x7f1d440c8c10 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f1d4000e870 tx=0x7f1d4000ec70 comp rx=0 tx=0).stop 2023-10-30T12:46:39.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.725+0000 7f1d4c032640 1 -- 172.21.15.125:0/3039620766 shutdown_connections 2023-10-30T12:46:39.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.725+0000 7f1d4c032640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1d280664a0 0x7f1d28068960 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:39.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.725+0000 7f1d4c032640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1d441539a0 0x7f1d441f2060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:39.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.725+0000 7f1d4c032640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1d44070a00 0x7f1d44144c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:39.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.725+0000 7f1d4c032640 1 --2- 172.21.15.125:0/3039620766 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1d44069e90 0x7f1d440c8c10 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:39.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.725+0000 7f1d4c032640 1 -- 172.21.15.125:0/3039620766 >> 172.21.15.125:0/3039620766 conn(0x7f1d440c7910 msgr2=0x7f1d44152c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:39.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.725+0000 7f1d4c032640 1 -- 172.21.15.125:0/3039620766 shutdown_connections 2023-10-30T12:46:39.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:39.725+0000 7f1d4c032640 1 -- 172.21.15.125:0/3039620766 wait complete. 2023-10-30T12:46:39.744 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:46:41.745 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:41.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.889+0000 7f0f1133e640 1 Processor -- start 2023-10-30T12:46:41.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.889+0000 7f0f1133e640 1 -- start start 2023-10-30T12:46:41.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f1133e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f0c0c9e40 0x7f0f0c153640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:41.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f1133e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0c06b6b0 con 0x7f0f0c0c9a60 2023-10-30T12:46:41.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f1133e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0c0b3040 con 0x7f0f0c0c9000 2023-10-30T12:46:41.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f1133e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c06bc30 con 0x7f0f0c0c9e40 2023-10-30T12:46:41.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f0affd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0f0c0c9a60 0x7f0f0c06afa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53046/0 (socket says 172.21.15.125:53046) 2023-10-30T12:46:41.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f0affd640 1 -- 172.21.15.125:0/1411838901 learned_addr learned my addr 172.21.15.125:0/1411838901 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:46:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f0a7fc640 1 --2- 172.21.15.125:0/1411838901 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f0c0c9e40 0x7f0f0c153640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601097552 0 0) 0x7f0f0c06b6b0 con 0x7f0f0c0c9a60 2023-10-30T12:46:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ef4003530 con 0x7f0f0c0c9a60 2023-10-30T12:46:41.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731528027 0 0) 0x7f0f0c0b3040 con 0x7f0f0c0c9000 2023-10-30T12:46:41.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f0c06b6b0 con 0x7f0f0c0c9000 2023-10-30T12:46:41.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2187750818 0 0) 0x7f0ef4003530 con 0x7f0f0c0c9a60 2023-10-30T12:46:41.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f0c0c9e40 msgr2=0x7f0f0c153640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:41.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 --2- 172.21.15.125:0/1411838901 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f0c0c9e40 0x7f0f0c153640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:41.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 >> v1:172.21.15.136:6789/0 conn(0x7f0f0c0c9000 legacy=0x7f0f0c159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:41.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c06b950 con 0x7f0f0c0c9a60 2023-10-30T12:46:41.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f0efc001680 con 0x7f0f0c0c9a60 2023-10-30T12:46:41.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 >> v1:172.21.15.125:6789/0 conn(0x7f0f0c0c9a60 legacy=0x7f0f0c06afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:41.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 --2- 172.21.15.125:0/1411838901 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ef4003b40 0x7f0ef4003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:41.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 --2- 172.21.15.125:0/1411838901 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f0c0c9e40 0x7f0ef40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:41.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 --2- 172.21.15.125:0/1411838901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ef4004d10 0x7f0ef4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:41.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c06bc30 con 0x7f0ef4003b40 2023-10-30T12:46:41.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef400ba70 con 0x7f0ef4004d10 2023-10-30T12:46:41.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef400bc30 con 0x7f0f0c0c9e40 2023-10-30T12:46:41.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f0affd640 1 --2- 172.21.15.125:0/1411838901 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ef4003b40 0x7f0ef4003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:41.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f0a7fc640 1 --2- 172.21.15.125:0/1411838901 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f0c0c9e40 0x7f0ef40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:41.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f0b7fe640 1 --2- 172.21.15.125:0/1411838901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ef4004d10 0x7f0ef4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:41.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f0affd640 1 -- 172.21.15.125:0/1411838901 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f0c0c9e40 msgr2=0x7f0ef40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:41.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f0affd640 1 --2- 172.21.15.125:0/1411838901 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f0c0c9e40 0x7f0ef40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:41.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f0affd640 1 -- 172.21.15.125:0/1411838901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ef4004d10 msgr2=0x7f0ef4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:41.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f0affd640 1 --2- 172.21.15.125:0/1411838901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ef4004d10 0x7f0ef4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:41.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f0affd640 1 -- 172.21.15.125:0/1411838901 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef4003530 con 0x7f0ef4003b40 2023-10-30T12:46:41.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.893+0000 7f0f0b7fe640 1 --2- 172.21.15.125:0/1411838901 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ef4004d10 0x7f0ef4005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:46:41.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.897+0000 7f0f0affd640 1 --2- 172.21.15.125:0/1411838901 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ef4003b40 0x7f0ef4003f40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f0efc007fa0 tx=0x7f0efc01cb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7865406d1d3e2a3f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:41.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.897+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc005ae0 con 0x7f0ef4003b40 2023-10-30T12:46:41.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.897+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0efc005c40 con 0x7f0ef4003b40 2023-10-30T12:46:41.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.897+0000 7f0f09ffb640 1 -- 172.21.15.125:0/1411838901 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc007070 con 0x7f0ef4003b40 2023-10-30T12:46:41.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.897+0000 7f0f1133e640 1 -- 172.21.15.125:0/1411838901 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ef4003b40 msgr2=0x7f0ef4003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:41.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.897+0000 7f0f1133e640 1 --2- 172.21.15.125:0/1411838901 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ef4003b40 0x7f0ef4003f40 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f0efc007fa0 tx=0x7f0efc01cb20 comp rx=0 tx=0).stop 2023-10-30T12:46:41.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.897+0000 7f0f0b7fe640 1 -- 172.21.15.125:0/1411838901 reap_dead start 2023-10-30T12:46:41.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.897+0000 7f0f1133e640 1 -- 172.21.15.125:0/1411838901 shutdown_connections 2023-10-30T12:46:41.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.897+0000 7f0f1133e640 1 -- 172.21.15.125:0/1411838901 >> 172.21.15.125:0/1411838901 conn(0x7f0f0c065a80 msgr2=0x7f0f0c065e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:41.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.897+0000 7f0f1133e640 1 -- 172.21.15.125:0/1411838901 shutdown_connections 2023-10-30T12:46:41.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.897+0000 7f0f1133e640 1 -- 172.21.15.125:0/1411838901 wait complete. 2023-10-30T12:46:41.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f1133e640 1 Processor -- start 2023-10-30T12:46:41.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f1133e640 1 -- start start 2023-10-30T12:46:41.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f1133e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ef4003b40 0x7f0f0c1e2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:41.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f1133e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f0c1e6390 0x7f0f0c0cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:41.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f1133e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f0c0ada90 0x7f0f0c1f2290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:41.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f1133e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c0c3c60 con 0x7f0ef4003b40 2023-10-30T12:46:41.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f1133e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c0ccdb0 con 0x7f0f0c0ada90 2023-10-30T12:46:41.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f1133e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0c0adf10 con 0x7f0f0c1e6390 2023-10-30T12:46:41.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f0affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ef4003b40 0x7f0f0c1e2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:41.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f0affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ef4003b40 0x7f0f0c1e2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35260/0 (socket says 172.21.15.125:35260) 2023-10-30T12:46:41.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f0affd640 1 -- 172.21.15.125:0/357289699 learned_addr learned my addr 172.21.15.125:0/357289699 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:41.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f0b7fe640 1 --2- 172.21.15.125:0/357289699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f0c0ada90 0x7f0f0c1f2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:41.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f0a7fc640 1 --2- 172.21.15.125:0/357289699 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f0c1e6390 0x7f0f0c0cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:41.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f0affd640 1 -- 172.21.15.125:0/357289699 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f0c1e6390 msgr2=0x7f0f0c0cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:41.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f0affd640 1 --2- 172.21.15.125:0/357289699 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f0c1e6390 0x7f0f0c0cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:41.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f0affd640 1 -- 172.21.15.125:0/357289699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f0c0ada90 msgr2=0x7f0f0c1f2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:41.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f0affd640 1 --2- 172.21.15.125:0/357289699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f0c0ada90 0x7f0f0c1f2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:41.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f0affd640 1 -- 172.21.15.125:0/357289699 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0c0c36a0 con 0x7f0ef4003b40 2023-10-30T12:46:41.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f0affd640 1 --2- 172.21.15.125:0/357289699 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ef4003b40 0x7f0f0c1e2940 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0efc01c950 tx=0x7f0ef4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:41.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0ef37fe640 1 -- 172.21.15.125:0/357289699 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc0b4070 con 0x7f0ef4003b40 2023-10-30T12:46:41.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0ef37fe640 1 -- 172.21.15.125:0/357289699 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0efc005e70 con 0x7f0ef4003b40 2023-10-30T12:46:41.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f1133e640 1 -- 172.21.15.125:0/357289699 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0c0c3980 con 0x7f0ef4003b40 2023-10-30T12:46:41.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0ef37fe640 1 -- 172.21.15.125:0/357289699 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0efc0072a0 con 0x7f0ef4003b40 2023-10-30T12:46:41.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f1133e640 1 -- 172.21.15.125:0/357289699 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0c0cca40 con 0x7f0ef4003b40 2023-10-30T12:46:41.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.901+0000 7f0f1133e640 1 -- 172.21.15.125:0/357289699 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0c04e170 con 0x7f0ef4003b40 2023-10-30T12:46:41.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.905+0000 7f0ef37fe640 1 -- 172.21.15.125:0/357289699 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0efc007400 con 0x7f0ef4003b40 2023-10-30T12:46:41.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.905+0000 7f0ef37fe640 1 --2- 172.21.15.125:0/357289699 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ef80664a0 0x7f0ef8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:41.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.905+0000 7f0ef37fe640 1 -- 172.21.15.125:0/357289699 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f0efc11ddd0 con 0x7f0ef4003b40 2023-10-30T12:46:41.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.905+0000 7f0f0a7fc640 1 --2- 172.21.15.125:0/357289699 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ef80664a0 0x7f0ef8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:41.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.905+0000 7f0f0a7fc640 1 --2- 172.21.15.125:0/357289699 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ef80664a0 0x7f0ef8068960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c1514a0 tx=0x7f0f00008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:41.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:41.909+0000 7f0ef37fe640 1 -- 172.21.15.125:0/357289699 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0efc0eaef0 con 0x7f0ef4003b40 2023-10-30T12:46:42.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:42.089+0000 7f0f1133e640 1 -- 172.21.15.125:0/357289699 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f0c14fb50 con 0x7f0ef4003b40 2023-10-30T12:46:42.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:42.089+0000 7f0ef37fe640 1 -- 172.21.15.125:0/357289699 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f0efc0076b0 con 0x7f0ef4003b40 2023-10-30T12:46:42.090 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:42.090 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":87,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:39.169108+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:resolve","state_seq":33,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:42.090 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 87 2023-10-30T12:46:42.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:42.089+0000 7f0f1133e640 1 -- 172.21.15.125:0/357289699 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ef80664a0 msgr2=0x7f0ef8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:42.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:42.089+0000 7f0f1133e640 1 --2- 172.21.15.125:0/357289699 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ef80664a0 0x7f0ef8068960 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c1514a0 tx=0x7f0f00008040 comp rx=0 tx=0).stop 2023-10-30T12:46:42.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:42.089+0000 7f0f1133e640 1 -- 172.21.15.125:0/357289699 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ef4003b40 msgr2=0x7f0f0c1e2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:42.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:42.089+0000 7f0f1133e640 1 --2- 172.21.15.125:0/357289699 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ef4003b40 0x7f0f0c1e2940 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f0efc01c950 tx=0x7f0ef4005800 comp rx=0 tx=0).stop 2023-10-30T12:46:42.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:42.089+0000 7f0f1133e640 1 -- 172.21.15.125:0/357289699 shutdown_connections 2023-10-30T12:46:42.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:42.089+0000 7f0f1133e640 1 --2- 172.21.15.125:0/357289699 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ef80664a0 0x7f0ef8068960 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:42.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:42.089+0000 7f0f1133e640 1 --2- 172.21.15.125:0/357289699 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f0c0ada90 0x7f0f0c1f2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:42.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:42.089+0000 7f0f1133e640 1 --2- 172.21.15.125:0/357289699 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f0c1e6390 0x7f0f0c0cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:42.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:42.089+0000 7f0f1133e640 1 --2- 172.21.15.125:0/357289699 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ef4003b40 0x7f0f0c1e2940 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:42.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:42.089+0000 7f0f1133e640 1 -- 172.21.15.125:0/357289699 >> 172.21.15.125:0/357289699 conn(0x7f0f0c065a80 msgr2=0x7f0f0c150350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:42.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:42.089+0000 7f0f1133e640 1 -- 172.21.15.125:0/357289699 shutdown_connections 2023-10-30T12:46:42.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:42.089+0000 7f0f1133e640 1 -- 172.21.15.125:0/357289699 wait complete. 2023-10-30T12:46:42.110 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:46:44.111 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:44.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.257+0000 7fdbd899d640 1 Processor -- start 2023-10-30T12:46:44.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.257+0000 7fdbd899d640 1 -- start start 2023-10-30T12:46:44.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.257+0000 7fdbd899d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbd4071110 0x7fdbd406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:44.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.257+0000 7fdbd899d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd4066cf0 con 0x7fdbd406f7a0 2023-10-30T12:46:44.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.257+0000 7fdbd899d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbd40c70d0 con 0x7fdbd40c6bd0 2023-10-30T12:46:44.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd899d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd40700c0 con 0x7fdbd4071110 2023-10-30T12:46:44.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd259c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdbd406f7a0 0x7fdbd4069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53050/0 (socket says 172.21.15.125:53050) 2023-10-30T12:46:44.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd259c640 1 -- 172.21.15.125:0/880747550 learned_addr learned my addr 172.21.15.125:0/880747550 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:46:44.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd2d9d640 1 --2- 172.21.15.125:0/880747550 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbd4071110 0x7fdbd406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:44.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 -- 172.21.15.125:0/880747550 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841587021 0 0) 0x7fdbd4066cf0 con 0x7fdbd406f7a0 2023-10-30T12:46:44.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 -- 172.21.15.125:0/880747550 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbc0003530 con 0x7fdbd406f7a0 2023-10-30T12:46:44.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 -- 172.21.15.125:0/880747550 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1295681926 0 0) 0x7fdbc0003530 con 0x7fdbd406f7a0 2023-10-30T12:46:44.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 -- 172.21.15.125:0/880747550 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbd4071110 msgr2=0x7fdbd406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:44.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 --2- 172.21.15.125:0/880747550 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbd4071110 0x7fdbd406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:44.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 -- 172.21.15.125:0/880747550 >> v1:172.21.15.136:6789/0 conn(0x7fdbd40c6bd0 legacy=0x7fdbd4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:44.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 -- 172.21.15.125:0/880747550 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd40c7390 con 0x7fdbd406f7a0 2023-10-30T12:46:44.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 -- 172.21.15.125:0/880747550 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fdbc4001680 con 0x7fdbd406f7a0 2023-10-30T12:46:44.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 -- 172.21.15.125:0/880747550 >> v1:172.21.15.125:6789/0 conn(0x7fdbd406f7a0 legacy=0x7fdbd4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:44.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 --2- 172.21.15.125:0/880747550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbc00039b0 0x7fdbc0003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:44.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 --2- 172.21.15.125:0/880747550 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbd4071110 0x7fdbc00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:44.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 --2- 172.21.15.125:0/880747550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbc0004d10 0x7fdbc0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:44.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 -- 172.21.15.125:0/880747550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd40700c0 con 0x7fdbc00039b0 2023-10-30T12:46:44.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 -- 172.21.15.125:0/880747550 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc000bad0 con 0x7fdbc0004d10 2023-10-30T12:46:44.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd1d9b640 1 -- 172.21.15.125:0/880747550 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc000bc90 con 0x7fdbd4071110 2023-10-30T12:46:44.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd259c640 1 --2- 172.21.15.125:0/880747550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbc00039b0 0x7fdbc0003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:44.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd2d9d640 1 --2- 172.21.15.125:0/880747550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbc0004d10 0x7fdbc0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:44.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbc9d9b640 1 --2- 172.21.15.125:0/880747550 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbd4071110 0x7fdbc00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:44.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd259c640 1 -- 172.21.15.125:0/880747550 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbd4071110 msgr2=0x7fdbc00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:44.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd259c640 1 --2- 172.21.15.125:0/880747550 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbd4071110 0x7fdbc00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:44.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd259c640 1 -- 172.21.15.125:0/880747550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbc0004d10 msgr2=0x7fdbc0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:44.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd259c640 1 --2- 172.21.15.125:0/880747550 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbc0004d10 0x7fdbc0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:44.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.261+0000 7fdbd259c640 1 -- 172.21.15.125:0/880747550 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc0003530 con 0x7fdbc00039b0 2023-10-30T12:46:44.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.265+0000 7fdbd259c640 1 --2- 172.21.15.125:0/880747550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbc00039b0 0x7fdbc0003dd0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400c210 tx=0x7fdbc401f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2686d52a220ec6f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:44.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.265+0000 7fdbd1d9b640 1 -- 172.21.15.125:0/880747550 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4005ae0 con 0x7fdbc00039b0 2023-10-30T12:46:44.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.265+0000 7fdbd1d9b640 1 -- 172.21.15.125:0/880747550 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc40ab070 con 0x7fdbc00039b0 2023-10-30T12:46:44.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.265+0000 7fdbd1d9b640 1 -- 172.21.15.125:0/880747550 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc4007070 con 0x7fdbc00039b0 2023-10-30T12:46:44.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.265+0000 7fdbd899d640 1 -- 172.21.15.125:0/880747550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbc00039b0 msgr2=0x7fdbc0003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:44.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.265+0000 7fdbd899d640 1 --2- 172.21.15.125:0/880747550 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbc00039b0 0x7fdbc0003dd0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400c210 tx=0x7fdbc401f370 comp rx=0 tx=0).stop 2023-10-30T12:46:44.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.265+0000 7fdbd2d9d640 1 -- 172.21.15.125:0/880747550 reap_dead start 2023-10-30T12:46:44.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.265+0000 7fdbd899d640 1 -- 172.21.15.125:0/880747550 shutdown_connections 2023-10-30T12:46:44.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.265+0000 7fdbd899d640 1 -- 172.21.15.125:0/880747550 >> 172.21.15.125:0/880747550 conn(0x7fdbd40c7910 msgr2=0x7fdbd40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:44.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.265+0000 7fdbd899d640 1 -- 172.21.15.125:0/880747550 shutdown_connections 2023-10-30T12:46:44.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.265+0000 7fdbd899d640 1 -- 172.21.15.125:0/880747550 wait complete. 2023-10-30T12:46:44.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.265+0000 7fdbd899d640 1 Processor -- start 2023-10-30T12:46:44.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.265+0000 7fdbd899d640 1 -- start start 2023-10-30T12:46:44.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbd899d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbc00039b0 0x7fdbd414da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:44.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbd899d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbd41ef8b0 0x7fdbd414e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:44.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbd899d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbd4156240 0x7fdbd414e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:44.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbd899d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd41f2bb0 con 0x7fdbd41ef8b0 2023-10-30T12:46:44.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbd899d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd41f30e0 con 0x7fdbc00039b0 2023-10-30T12:46:44.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbd899d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd41ef420 con 0x7fdbd4156240 2023-10-30T12:46:44.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbc9d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbd41ef8b0 0x7fdbd414e290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:44.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbc9d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbd41ef8b0 0x7fdbd414e290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35274/0 (socket says 172.21.15.125:35274) 2023-10-30T12:46:44.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbc9d9b640 1 -- 172.21.15.125:0/1672944521 learned_addr learned my addr 172.21.15.125:0/1672944521 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:44.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbd259c640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbc00039b0 0x7fdbd414da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:44.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbd2d9d640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbd4156240 0x7fdbd414e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:44.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbc9d9b640 1 -- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbd4156240 msgr2=0x7fdbd414e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:44.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbc9d9b640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbd4156240 0x7fdbd414e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:44.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbc9d9b640 1 -- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbc00039b0 msgr2=0x7fdbd414da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:44.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbc9d9b640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbc00039b0 0x7fdbd414da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:44.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbc9d9b640 1 -- 172.21.15.125:0/1672944521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd414dfc0 con 0x7fdbd41ef8b0 2023-10-30T12:46:44.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbd2d9d640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbd4156240 0x7fdbd414e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:44.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbc9d9b640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbd41ef8b0 0x7fdbd414e290 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc00e5b0 tx=0x7fdbbc00ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:44.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbcb7fe640 1 -- 172.21.15.125:0/1672944521 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc0052e0 con 0x7fdbd41ef8b0 2023-10-30T12:46:44.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbcb7fe640 1 -- 172.21.15.125:0/1672944521 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbbc005440 con 0x7fdbd41ef8b0 2023-10-30T12:46:44.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbd899d640 1 -- 172.21.15.125:0/1672944521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd41f2e00 con 0x7fdbd41ef8b0 2023-10-30T12:46:44.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbcb7fe640 1 -- 172.21.15.125:0/1672944521 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbbc003820 con 0x7fdbd41ef8b0 2023-10-30T12:46:44.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbd899d640 1 -- 172.21.15.125:0/1672944521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd41ef0b0 con 0x7fdbd41ef8b0 2023-10-30T12:46:44.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbd899d640 1 -- 172.21.15.125:0/1672944521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd41560b0 con 0x7fdbd41ef8b0 2023-10-30T12:46:44.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.269+0000 7fdbcb7fe640 1 -- 172.21.15.125:0/1672944521 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdbbc003980 con 0x7fdbd41ef8b0 2023-10-30T12:46:44.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.273+0000 7fdbcb7fe640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdbb40664a0 0x7fdbb4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:44.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.273+0000 7fdbcb7fe640 1 -- 172.21.15.125:0/1672944521 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7fdbbc083f20 con 0x7fdbd41ef8b0 2023-10-30T12:46:44.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.273+0000 7fdbd259c640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdbb40664a0 0x7fdbb4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:44.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.273+0000 7fdbd259c640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdbb40664a0 0x7fdbb4068960 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40af000 tx=0x7fdbc40aa000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:44.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.277+0000 7fdbcb7fe640 1 -- 172.21.15.125:0/1672944521 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdbbc050fb0 con 0x7fdbd41ef8b0 2023-10-30T12:46:44.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.457+0000 7fdbd899d640 1 -- 172.21.15.125:0/1672944521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbd41eec30 con 0x7fdbd41ef8b0 2023-10-30T12:46:44.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.461+0000 7fdbcb7fe640 1 -- 172.21.15.125:0/1672944521 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5740 (secure 0 0 0) 0x7fdbd41eec30 con 0x7fdbd41ef8b0 2023-10-30T12:46:44.460 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:44.461 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:44.291229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:44.461 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 88 2023-10-30T12:46:44.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.461+0000 7fdbd899d640 1 -- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdbb40664a0 msgr2=0x7fdbb4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:44.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.461+0000 7fdbd899d640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdbb40664a0 0x7fdbb4068960 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdbc40af000 tx=0x7fdbc40aa000 comp rx=0 tx=0).stop 2023-10-30T12:46:44.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.461+0000 7fdbd899d640 1 -- 172.21.15.125:0/1672944521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbd41ef8b0 msgr2=0x7fdbd414e290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:44.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.461+0000 7fdbd899d640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbd41ef8b0 0x7fdbd414e290 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc00e5b0 tx=0x7fdbbc00ea70 comp rx=0 tx=0).stop 2023-10-30T12:46:44.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.461+0000 7fdbd899d640 1 -- 172.21.15.125:0/1672944521 shutdown_connections 2023-10-30T12:46:44.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.461+0000 7fdbd899d640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdbb40664a0 0x7fdbb4068960 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:44.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.461+0000 7fdbd899d640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdbd4156240 0x7fdbd414e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:44.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.461+0000 7fdbd899d640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdbd41ef8b0 0x7fdbd414e290 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:44.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.461+0000 7fdbd899d640 1 --2- 172.21.15.125:0/1672944521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdbc00039b0 0x7fdbd414da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:44.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.461+0000 7fdbd899d640 1 -- 172.21.15.125:0/1672944521 >> 172.21.15.125:0/1672944521 conn(0x7fdbd40c7910 msgr2=0x7fdbd40c7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:44.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.461+0000 7fdbd899d640 1 -- 172.21.15.125:0/1672944521 shutdown_connections 2023-10-30T12:46:44.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:44.461+0000 7fdbd899d640 1 -- 172.21.15.125:0/1672944521 wait complete. 2023-10-30T12:46:44.481 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 88, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 88, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5935": { "addr": "172.21.15.136:6839/1510290691", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 1510290691, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 1510290691, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5935, "incarnation": 86, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:reconnect", "state_seq": 35 }, "gid_5988": { "addr": "172.21.15.125:6837/3156415951", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 3156415951, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 3156415951, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5988, "incarnation": 78, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 28 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:46:44.291229+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 5935, "mds_1": 5988 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:46:46.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:46.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f3974609640 1 Processor -- start 2023-10-30T12:46:46.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f3974609640 1 -- start start 2023-10-30T12:46:46.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f3974609640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f396c1538a0 0x7f396c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:46.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f3974609640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f396c0ca0e0 0x7f396c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:46.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f3974609640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396c0cb800 con 0x7f396c0cf160 2023-10-30T12:46:46.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f3974609640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f396c15dc30 con 0x7f396c1538a0 2023-10-30T12:46:46.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f3974609640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c0cb330 con 0x7f396c0ca0e0 2023-10-30T12:46:46.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f39723a5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f396c1538a0 0x7f396c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:46.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f3971ba4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f396c0ca0e0 0x7f396c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:46.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f3972ba6640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f396c0cf160 0x7f396c0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:60090/0 (socket says 172.21.15.125:60090) 2023-10-30T12:46:46.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f3972ba6640 1 -- 172.21.15.125:0/1374094080 learned_addr learned my addr 172.21.15.125:0/1374094080 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:46:46.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f39723a5640 1 -- 172.21.15.125:0/1374094080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f396c0ca0e0 msgr2=0x7f396c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:46.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f39723a5640 1 --2- 172.21.15.125:0/1374094080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f396c0ca0e0 0x7f396c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:46.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f39723a5640 1 -- 172.21.15.125:0/1374094080 >> v1:172.21.15.136:6789/0 conn(0x7f396c0cf160 legacy=0x7f396c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:46.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.629+0000 7f39723a5640 1 -- 172.21.15.125:0/1374094080 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c0cad70 con 0x7f396c1538a0 2023-10-30T12:46:46.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.633+0000 7f39723a5640 1 --2- 172.21.15.125:0/1374094080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c1538a0 0x7f396c15d6f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f39680097c0 tx=0x7f396801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e37329d82db75be1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:46.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.633+0000 7f39713a3640 1 -- 172.21.15.125:0/1374094080 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39680a8970 con 0x7f396c1538a0 2023-10-30T12:46:46.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.633+0000 7f39713a3640 1 -- 172.21.15.125:0/1374094080 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39680a8ad0 con 0x7f396c1538a0 2023-10-30T12:46:46.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.633+0000 7f39713a3640 1 -- 172.21.15.125:0/1374094080 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39680a8e60 con 0x7f396c1538a0 2023-10-30T12:46:46.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.633+0000 7f3974609640 1 -- 172.21.15.125:0/1374094080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c1538a0 msgr2=0x7f396c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:46.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.633+0000 7f3974609640 1 --2- 172.21.15.125:0/1374094080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c1538a0 0x7f396c15d6f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f39680097c0 tx=0x7f396801ec70 comp rx=0 tx=0).stop 2023-10-30T12:46:46.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 -- 172.21.15.125:0/1374094080 shutdown_connections 2023-10-30T12:46:46.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 --2- 172.21.15.125:0/1374094080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f396c0ca0e0 0x7f396c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:46.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 --2- 172.21.15.125:0/1374094080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c1538a0 0x7f396c15d6f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:46.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 -- 172.21.15.125:0/1374094080 >> 172.21.15.125:0/1374094080 conn(0x7f396c06bfe0 msgr2=0x7f396c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:46.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 -- 172.21.15.125:0/1374094080 shutdown_connections 2023-10-30T12:46:46.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 -- 172.21.15.125:0/1374094080 wait complete. 2023-10-30T12:46:46.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 Processor -- start 2023-10-30T12:46:46.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 -- start start 2023-10-30T12:46:46.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f396c0ca0e0 0x7f396c15b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:46.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c0cf160 0x7f396c1e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:46.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f396c1538a0 0x7f396c1e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:46.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c1e5580 con 0x7f396c0ca0e0 2023-10-30T12:46:46.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c15bc60 con 0x7f396c0cf160 2023-10-30T12:46:46.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c1e4670 con 0x7f396c1538a0 2023-10-30T12:46:46.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f39723a5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f396c0ca0e0 0x7f396c15b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:46.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f39723a5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f396c0ca0e0 0x7f396c15b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35276/0 (socket says 172.21.15.125:35276) 2023-10-30T12:46:46.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f39723a5640 1 -- 172.21.15.125:0/3196818344 learned_addr learned my addr 172.21.15.125:0/3196818344 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:46.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3971ba4640 1 --2- 172.21.15.125:0/3196818344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c0cf160 0x7f396c1e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:46.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3972ba6640 1 --2- 172.21.15.125:0/3196818344 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f396c1538a0 0x7f396c1e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:46.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f39723a5640 1 -- 172.21.15.125:0/3196818344 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f396c1538a0 msgr2=0x7f396c1e5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:46.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f39723a5640 1 --2- 172.21.15.125:0/3196818344 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f396c1538a0 0x7f396c1e5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:46.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f39723a5640 1 -- 172.21.15.125:0/3196818344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c0cf160 msgr2=0x7f396c1e4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:46.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f39723a5640 1 --2- 172.21.15.125:0/3196818344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c0cf160 0x7f396c1e4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:46.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f39723a5640 1 -- 172.21.15.125:0/3196818344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3968009420 con 0x7f396c0ca0e0 2023-10-30T12:46:46.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f39723a5640 1 --2- 172.21.15.125:0/3196818344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f396c0ca0e0 0x7f396c15b040 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f396801c8a0 tx=0x7f396801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:46.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3962ffd640 1 -- 172.21.15.125:0/3196818344 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39680b9630 con 0x7f396c0ca0e0 2023-10-30T12:46:46.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3962ffd640 1 -- 172.21.15.125:0/3196818344 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39680b97b0 con 0x7f396c0ca0e0 2023-10-30T12:46:46.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3962ffd640 1 -- 172.21.15.125:0/3196818344 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39680b9b60 con 0x7f396c0ca0e0 2023-10-30T12:46:46.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.637+0000 7f3974609640 1 -- 172.21.15.125:0/3196818344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396c158840 con 0x7f396c0ca0e0 2023-10-30T12:46:46.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.641+0000 7f3974609640 1 -- 172.21.15.125:0/3196818344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c158ae0 con 0x7f396c0ca0e0 2023-10-30T12:46:46.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.641+0000 7f3974609640 1 -- 172.21.15.125:0/3196818344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396c0b2a50 con 0x7f396c0ca0e0 2023-10-30T12:46:46.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.641+0000 7f3962ffd640 1 -- 172.21.15.125:0/3196818344 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f39680b7040 con 0x7f396c0ca0e0 2023-10-30T12:46:46.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.641+0000 7f3962ffd640 1 --2- 172.21.15.125:0/3196818344 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f39500664a0 0x7f3950068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:46.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.641+0000 7f3962ffd640 1 -- 172.21.15.125:0/3196818344 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f3968122160 con 0x7f396c0ca0e0 2023-10-30T12:46:46.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.641+0000 7f3971ba4640 1 --2- 172.21.15.125:0/3196818344 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f39500664a0 0x7f3950068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:46.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.645+0000 7f3971ba4640 1 --2- 172.21.15.125:0/3196818344 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f39500664a0 0x7f3950068960 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f396c1e58f0 tx=0x7f395c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:46.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.645+0000 7f3962ffd640 1 -- 172.21.15.125:0/3196818344 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f39680ef1e0 con 0x7f396c0ca0e0 2023-10-30T12:46:46.720 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:46:46.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:46:46.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.829+0000 7f3974609640 1 -- 172.21.15.125:0/3196818344 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f396c04e170 con 0x7f396c0ca0e0 2023-10-30T12:46:46.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.829+0000 7f3962ffd640 1 -- 172.21.15.125:0/3196818344 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f39680f2800 con 0x7f396c0ca0e0 2023-10-30T12:46:46.829 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:46.830 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:44.291229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:46.830 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 88 2023-10-30T12:46:46.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.829+0000 7f3974609640 1 -- 172.21.15.125:0/3196818344 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f39500664a0 msgr2=0x7f3950068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:46.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.829+0000 7f3974609640 1 --2- 172.21.15.125:0/3196818344 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f39500664a0 0x7f3950068960 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f396c1e58f0 tx=0x7f395c008040 comp rx=0 tx=0).stop 2023-10-30T12:46:46.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.829+0000 7f3974609640 1 -- 172.21.15.125:0/3196818344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f396c0ca0e0 msgr2=0x7f396c15b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:46.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.829+0000 7f3974609640 1 --2- 172.21.15.125:0/3196818344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f396c0ca0e0 0x7f396c15b040 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f396801c8a0 tx=0x7f396801ba10 comp rx=0 tx=0).stop 2023-10-30T12:46:46.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.829+0000 7f3974609640 1 -- 172.21.15.125:0/3196818344 shutdown_connections 2023-10-30T12:46:46.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.829+0000 7f3974609640 1 --2- 172.21.15.125:0/3196818344 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f39500664a0 0x7f3950068960 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:46.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.829+0000 7f3974609640 1 --2- 172.21.15.125:0/3196818344 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f396c1538a0 0x7f396c1e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:46.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.829+0000 7f3974609640 1 --2- 172.21.15.125:0/3196818344 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f396c0cf160 0x7f396c1e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:46.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.829+0000 7f3974609640 1 --2- 172.21.15.125:0/3196818344 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f396c0ca0e0 0x7f396c15b040 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:46.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.829+0000 7f3974609640 1 -- 172.21.15.125:0/3196818344 >> 172.21.15.125:0/3196818344 conn(0x7f396c06bfe0 msgr2=0x7f396c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:46.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.829+0000 7f3974609640 1 -- 172.21.15.125:0/3196818344 shutdown_connections 2023-10-30T12:46:46.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:46.829+0000 7f3974609640 1 -- 172.21.15.125:0/3196818344 wait complete. 2023-10-30T12:46:46.850 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:46:48.851 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:48.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:48.997+0000 7fa1cbb0f640 1 Processor -- start 2023-10-30T12:46:48.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:48.997+0000 7fa1cbb0f640 1 -- start start 2023-10-30T12:46:49.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.001+0000 7fa1cbb0f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa1c406f7a0 0x7fa1c406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:49.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.001+0000 7fa1cbb0f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c4066cf0 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.001+0000 7fa1cbb0f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1c40c70d0 con 0x7fa1c4071110 2023-10-30T12:46:49.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.001+0000 7fa1cbb0f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1c40700c0 con 0x7fa1c406f7a0 2023-10-30T12:46:49.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.001+0000 7fa1c98ab640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa1c406f7a0 0x7fa1c406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:49.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.001+0000 7fa1c90aa640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa1c40c6bd0 0x7fa1c4069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53062/0 (socket says 172.21.15.125:53062) 2023-10-30T12:46:49.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.001+0000 7fa1c90aa640 1 -- 172.21.15.125:0/1818372572 learned_addr learned my addr 172.21.15.125:0/1818372572 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:46:49.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.001+0000 7fa1c98ab640 1 -- 172.21.15.125:0/1818372572 >> v1:172.21.15.136:6789/0 conn(0x7fa1c4071110 legacy=0x7fa1c4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:49.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.001+0000 7fa1c98ab640 1 -- 172.21.15.125:0/1818372572 >> v1:172.21.15.125:6789/0 conn(0x7fa1c40c6bd0 legacy=0x7fa1c4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:49.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.001+0000 7fa1c88a9640 1 -- 172.21.15.125:0/1818372572 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566133831 0 0) 0x7fa1c4066cf0 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.001+0000 7fa1c98ab640 1 -- 172.21.15.125:0/1818372572 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c40c7390 con 0x7fa1c406f7a0 2023-10-30T12:46:49.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1c98ab640 1 --2- 172.21.15.125:0/1818372572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c406f7a0 0x7fa1c406fb80 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00097c0 tx=0x7fa1c001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3350b32921027cc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:49.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1c88a9640 1 -- 172.21.15.125:0/1818372572 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c00a8970 con 0x7fa1c406f7a0 2023-10-30T12:46:49.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1c88a9640 1 -- 172.21.15.125:0/1818372572 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1c00a8ad0 con 0x7fa1c406f7a0 2023-10-30T12:46:49.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1c88a9640 1 -- 172.21.15.125:0/1818372572 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c00a8e60 con 0x7fa1c406f7a0 2023-10-30T12:46:49.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/1818372572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c406f7a0 msgr2=0x7fa1c406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:49.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1cbb0f640 1 --2- 172.21.15.125:0/1818372572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c406f7a0 0x7fa1c406fb80 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00097c0 tx=0x7fa1c001ec70 comp rx=0 tx=0).stop 2023-10-30T12:46:49.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/1818372572 shutdown_connections 2023-10-30T12:46:49.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1cbb0f640 1 --2- 172.21.15.125:0/1818372572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c406f7a0 0x7fa1c406fb80 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:49.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/1818372572 >> 172.21.15.125:0/1818372572 conn(0x7fa1c40c7910 msgr2=0x7fa1c40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:49.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/1818372572 shutdown_connections 2023-10-30T12:46:49.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/1818372572 wait complete. 2023-10-30T12:46:49.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1cbb0f640 1 Processor -- start 2023-10-30T12:46:49.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1cbb0f640 1 -- start start 2023-10-30T12:46:49.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1cbb0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1c406f7a0 0x7fa1c4152350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:49.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1cbb0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1c4071110 0x7fa1c41de540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:49.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.005+0000 7fa1cbb0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c40c6bd0 0x7fa1c41ed010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:49.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1cbb0f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c41ecbc0 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1cbb0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c41f02b0 con 0x7fa1c4071110 2023-10-30T12:46:49.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1cbb0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c41f0590 con 0x7fa1c406f7a0 2023-10-30T12:46:49.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1ca0ac640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c40c6bd0 0x7fa1c41ed010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:49.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1ca0ac640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c40c6bd0 0x7fa1c41ed010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35300/0 (socket says 172.21.15.125:35300) 2023-10-30T12:46:49.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1ca0ac640 1 -- 172.21.15.125:0/3490650095 learned_addr learned my addr 172.21.15.125:0/3490650095 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:49.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1c98ab640 1 --2- 172.21.15.125:0/3490650095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1c406f7a0 0x7fa1c4152350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:49.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1c90aa640 1 --2- 172.21.15.125:0/3490650095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1c4071110 0x7fa1c41de540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:49.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1ca0ac640 1 -- 172.21.15.125:0/3490650095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1c406f7a0 msgr2=0x7fa1c4152350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:49.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1ca0ac640 1 --2- 172.21.15.125:0/3490650095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1c406f7a0 0x7fa1c4152350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:49.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1ca0ac640 1 -- 172.21.15.125:0/3490650095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1c4071110 msgr2=0x7fa1c41de540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:49.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1ca0ac640 1 --2- 172.21.15.125:0/3490650095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1c4071110 0x7fa1c41de540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:49.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1ca0ac640 1 -- 172.21.15.125:0/3490650095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c0009420 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1ca0ac640 1 --2- 172.21.15.125:0/3490650095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c40c6bd0 0x7fa1c41ed010 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00c730 tx=0x7fa1bc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:49.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1ba7fc640 1 -- 172.21.15.125:0/3490650095 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc003b80 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1ba7fc640 1 -- 172.21.15.125:0/3490650095 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1bc003ce0 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1ba7fc640 1 -- 172.21.15.125:0/3490650095 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1bc00aa30 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/3490650095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c41f0870 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/3490650095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c41f0e30 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/3490650095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c41ecd30 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.009+0000 7fa1ba7fc640 1 -- 172.21.15.125:0/3490650095 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa1bc00ab90 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.013+0000 7fa1ba7fc640 1 --2- 172.21.15.125:0/3490650095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1a80664a0 0x7fa1a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:49.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.013+0000 7fa1ba7fc640 1 -- 172.21.15.125:0/3490650095 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7fa1bc083dd0 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.013+0000 7fa1c98ab640 1 --2- 172.21.15.125:0/3490650095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1a80664a0 0x7fa1a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:49.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.013+0000 7fa1c98ab640 1 --2- 172.21.15.125:0/3490650095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1a80664a0 0x7fa1a8068960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00b2000 tx=0x7fa1c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:49.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.017+0000 7fa1ba7fc640 1 -- 172.21.15.125:0/3490650095 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa1c41ecd30 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.197+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/3490650095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1c41ecd00 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.197+0000 7fa1ba7fc640 1 -- 172.21.15.125:0/3490650095 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+5740 (secure 0 0 0) 0x7fa1bc051020 con 0x7fa1c40c6bd0 2023-10-30T12:46:49.198 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:49.198 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:44.291229+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:49.199 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 88 2023-10-30T12:46:49.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.197+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/3490650095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1a80664a0 msgr2=0x7fa1a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:49.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.197+0000 7fa1cbb0f640 1 --2- 172.21.15.125:0/3490650095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1a80664a0 0x7fa1a8068960 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00b2000 tx=0x7fa1c0008040 comp rx=0 tx=0).stop 2023-10-30T12:46:49.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.197+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/3490650095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c40c6bd0 msgr2=0x7fa1c41ed010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:49.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.197+0000 7fa1cbb0f640 1 --2- 172.21.15.125:0/3490650095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c40c6bd0 0x7fa1c41ed010 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00c730 tx=0x7fa1bc00cbf0 comp rx=0 tx=0).stop 2023-10-30T12:46:49.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.197+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/3490650095 shutdown_connections 2023-10-30T12:46:49.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.197+0000 7fa1cbb0f640 1 --2- 172.21.15.125:0/3490650095 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1a80664a0 0x7fa1a8068960 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:49.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.197+0000 7fa1cbb0f640 1 --2- 172.21.15.125:0/3490650095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1c40c6bd0 0x7fa1c41ed010 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:49.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.197+0000 7fa1cbb0f640 1 --2- 172.21.15.125:0/3490650095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1c4071110 0x7fa1c41de540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:49.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.197+0000 7fa1cbb0f640 1 --2- 172.21.15.125:0/3490650095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1c406f7a0 0x7fa1c4152350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:49.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.197+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/3490650095 >> 172.21.15.125:0/3490650095 conn(0x7fa1c40c7910 msgr2=0x7fa1c40c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:49.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.197+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/3490650095 shutdown_connections 2023-10-30T12:46:49.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:49.201+0000 7fa1cbb0f640 1 -- 172.21.15.125:0/3490650095 wait complete. 2023-10-30T12:46:49.219 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:46:51.220 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:51.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.365+0000 7fa2291d3640 1 Processor -- start 2023-10-30T12:46:51.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.365+0000 7fa2291d3640 1 -- start start 2023-10-30T12:46:51.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.369+0000 7fa2291d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa2240c05d0 0x7fa22413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:51.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.369+0000 7fa2291d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2240c6f60 0x7fa22413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:51.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.369+0000 7fa2291d3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2240c0a50 con 0x7fa2240c6b80 2023-10-30T12:46:51.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.369+0000 7fa2291d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa22406f6e0 con 0x7fa2240c05d0 2023-10-30T12:46:51.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.369+0000 7fa2291d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa22406f9c0 con 0x7fa2240c6f60 2023-10-30T12:46:51.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.369+0000 7fa21bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa2240c05d0 0x7fa22413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:51.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.369+0000 7fa21bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa2240c05d0 0x7fa22413ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49634/0 (socket says 172.21.15.125:49634) 2023-10-30T12:46:51.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.369+0000 7fa21bfff640 1 -- 172.21.15.125:0/4065340354 learned_addr learned my addr 172.21.15.125:0/4065340354 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:51.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.369+0000 7fa22359e640 1 --2- 172.21.15.125:0/4065340354 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2240c6f60 0x7fa22413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:51.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.369+0000 7fa21bfff640 1 -- 172.21.15.125:0/4065340354 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2240c6f60 msgr2=0x7fa22413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:51.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.369+0000 7fa21bfff640 1 --2- 172.21.15.125:0/4065340354 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2240c6f60 0x7fa22413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:51.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.369+0000 7fa21bfff640 1 -- 172.21.15.125:0/4065340354 >> v1:172.21.15.136:6789/0 conn(0x7fa2240c6b80 legacy=0x7fa2240bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:51.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.369+0000 7fa21bfff640 1 -- 172.21.15.125:0/4065340354 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa22406fca0 con 0x7fa2240c05d0 2023-10-30T12:46:51.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa21bfff640 1 --2- 172.21.15.125:0/4065340354 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2240c05d0 0x7fa22413ed90 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa20c009870 tx=0x7fa20c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1231284c69d6a9e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:51.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa22259c640 1 -- 172.21.15.125:0/4065340354 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c0ab650 con 0x7fa2240c05d0 2023-10-30T12:46:51.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa22259c640 1 -- 172.21.15.125:0/4065340354 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20c0ab7b0 con 0x7fa2240c05d0 2023-10-30T12:46:51.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa22259c640 1 -- 172.21.15.125:0/4065340354 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c0abb40 con 0x7fa2240c05d0 2023-10-30T12:46:51.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa2291d3640 1 -- 172.21.15.125:0/4065340354 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2240c05d0 msgr2=0x7fa22413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:51.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa2291d3640 1 --2- 172.21.15.125:0/4065340354 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2240c05d0 0x7fa22413ed90 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fa20c009870 tx=0x7fa20c01c540 comp rx=0 tx=0).stop 2023-10-30T12:46:51.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa2291d3640 1 -- 172.21.15.125:0/4065340354 shutdown_connections 2023-10-30T12:46:51.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa2291d3640 1 --2- 172.21.15.125:0/4065340354 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2240c6f60 0x7fa22413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:51.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa2291d3640 1 --2- 172.21.15.125:0/4065340354 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2240c05d0 0x7fa22413ed90 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:51.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa2291d3640 1 -- 172.21.15.125:0/4065340354 >> 172.21.15.125:0/4065340354 conn(0x7fa2240c7910 msgr2=0x7fa2240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:51.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa2291d3640 1 -- 172.21.15.125:0/4065340354 shutdown_connections 2023-10-30T12:46:51.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa2291d3640 1 -- 172.21.15.125:0/4065340354 wait complete. 2023-10-30T12:46:51.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa2291d3640 1 Processor -- start 2023-10-30T12:46:51.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa2291d3640 1 -- start start 2023-10-30T12:46:51.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa2291d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2240c05d0 0x7fa22414b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:51.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.373+0000 7fa2291d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2240c6b80 0x7fa224132490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:51.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa2291d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2240c6f60 0x7fa2241329d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:51.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa2291d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa224158710 con 0x7fa2240c6b80 2023-10-30T12:46:51.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa2291d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22414d3d0 con 0x7fa2240c6f60 2023-10-30T12:46:51.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa2291d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa22412f770 con 0x7fa2240c05d0 2023-10-30T12:46:51.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa21bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2240c6b80 0x7fa224132490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:51.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa21bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2240c6b80 0x7fa224132490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49646/0 (socket says 172.21.15.125:49646) 2023-10-30T12:46:51.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa21bfff640 1 -- 172.21.15.125:0/1179573044 learned_addr learned my addr 172.21.15.125:0/1179573044 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:51.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa222d9d640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2240c05d0 0x7fa22414b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:51.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa22359e640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2240c6f60 0x7fa2241329d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:51.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa21bfff640 1 -- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2240c05d0 msgr2=0x7fa22414b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:51.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa21bfff640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2240c05d0 0x7fa22414b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:51.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa21bfff640 1 -- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2240c6f60 msgr2=0x7fa2241329d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:51.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa21bfff640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2240c6f60 0x7fa2241329d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:51.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa21bfff640 1 -- 172.21.15.125:0/1179573044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20c0094d0 con 0x7fa2240c6b80 2023-10-30T12:46:51.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa22359e640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2240c6f60 0x7fa2241329d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:51.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa21bfff640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2240c6b80 0x7fa224132490 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0ad040 tx=0x7fa20c01ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:51.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa21b7fe640 1 -- 172.21.15.125:0/1179573044 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c0aa980 con 0x7fa2240c6b80 2023-10-30T12:46:51.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa21b7fe640 1 -- 172.21.15.125:0/1179573044 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20c0aab00 con 0x7fa2240c6b80 2023-10-30T12:46:51.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa2291d3640 1 -- 172.21.15.125:0/1179573044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa224157750 con 0x7fa2240c6b80 2023-10-30T12:46:51.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa21b7fe640 1 -- 172.21.15.125:0/1179573044 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa20c0aaeb0 con 0x7fa2240c6b80 2023-10-30T12:46:51.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa2291d3640 1 -- 172.21.15.125:0/1179573044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22412f160 con 0x7fa2240c6b80 2023-10-30T12:46:51.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa2291d3640 1 -- 172.21.15.125:0/1179573044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa22404e170 con 0x7fa2240c6b80 2023-10-30T12:46:51.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.377+0000 7fa21b7fe640 1 -- 172.21.15.125:0/1179573044 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa20c0b43f0 con 0x7fa2240c6b80 2023-10-30T12:46:51.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.381+0000 7fa21b7fe640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2040664d0 0x7fa204068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:51.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.381+0000 7fa21b7fe640 1 -- 172.21.15.125:0/1179573044 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7fa20c11ee40 con 0x7fa2240c6b80 2023-10-30T12:46:51.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.381+0000 7fa222d9d640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2040664d0 0x7fa204068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:51.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.381+0000 7fa222d9d640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2040664d0 0x7fa204068990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa214007fd0 tx=0x7fa214009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:51.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.385+0000 7fa21b7fe640 1 -- 172.21.15.125:0/1179573044 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa20c0ec000 con 0x7fa2240c6b80 2023-10-30T12:46:51.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.561+0000 7fa2291d3640 1 -- 172.21.15.125:0/1179573044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2241321f0 con 0x7fa2240c6b80 2023-10-30T12:46:51.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.565+0000 7fa21b7fe640 1 -- 172.21.15.125:0/1179573044 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+5737 (secure 0 0 0) 0x7fa20c0ef620 con 0x7fa2240c6b80 2023-10-30T12:46:51.564 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:51.565 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:49.460277+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:51.565 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 89 2023-10-30T12:46:51.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.565+0000 7fa2291d3640 1 -- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2040664d0 msgr2=0x7fa204068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:51.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.565+0000 7fa2291d3640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2040664d0 0x7fa204068990 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa214007fd0 tx=0x7fa214009040 comp rx=0 tx=0).stop 2023-10-30T12:46:51.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.565+0000 7fa2291d3640 1 -- 172.21.15.125:0/1179573044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2240c6b80 msgr2=0x7fa224132490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:51.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.565+0000 7fa2291d3640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2240c6b80 0x7fa224132490 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fa20c0ad040 tx=0x7fa20c01ad80 comp rx=0 tx=0).stop 2023-10-30T12:46:51.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.565+0000 7fa2291d3640 1 -- 172.21.15.125:0/1179573044 shutdown_connections 2023-10-30T12:46:51.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.565+0000 7fa2291d3640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2040664d0 0x7fa204068990 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:51.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.565+0000 7fa2291d3640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2240c6f60 0x7fa2241329d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:51.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.565+0000 7fa2291d3640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2240c6b80 0x7fa224132490 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:51.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.565+0000 7fa2291d3640 1 --2- 172.21.15.125:0/1179573044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2240c05d0 0x7fa22414b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:51.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.565+0000 7fa2291d3640 1 -- 172.21.15.125:0/1179573044 >> 172.21.15.125:0/1179573044 conn(0x7fa2240c7910 msgr2=0x7fa22406bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:51.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.565+0000 7fa2291d3640 1 -- 172.21.15.125:0/1179573044 shutdown_connections 2023-10-30T12:46:51.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:51.565+0000 7fa2291d3640 1 -- 172.21.15.125:0/1179573044 wait complete. 2023-10-30T12:46:51.585 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:46:53.586 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:53.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6faa3a0640 1 Processor -- start 2023-10-30T12:46:53.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6faa3a0640 1 -- start start 2023-10-30T12:46:53.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6faa3a0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fa41538f0 0x7f6fa4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:53.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6faa3a0640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fa40cefa0 0x7f6fa40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:53.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6faa3a0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fa40ca6a0 con 0x7f6fa415cd50 2023-10-30T12:46:53.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6faa3a0640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fa415d300 con 0x7f6fa41538f0 2023-10-30T12:46:53.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6faa3a0640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa415d5e0 con 0x7f6fa40cefa0 2023-10-30T12:46:53.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6fa939e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fa41538f0 0x7f6fa4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:53.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6fa939e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fa41538f0 0x7f6fa4159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49648/0 (socket says 172.21.15.125:49648) 2023-10-30T12:46:53.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6fa939e640 1 -- 172.21.15.125:0/1003699862 learned_addr learned my addr 172.21.15.125:0/1003699862 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:53.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6fa9b9f640 1 --2- 172.21.15.125:0/1003699862 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fa40cefa0 0x7f6fa40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:53.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6fa939e640 1 -- 172.21.15.125:0/1003699862 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fa40cefa0 msgr2=0x7f6fa40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:53.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6fa939e640 1 --2- 172.21.15.125:0/1003699862 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fa40cefa0 0x7f6fa40cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:53.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6fa939e640 1 -- 172.21.15.125:0/1003699862 >> v1:172.21.15.136:6789/0 conn(0x7f6fa415cd50 legacy=0x7f6fa40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:53.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.733+0000 7f6fa939e640 1 -- 172.21.15.125:0/1003699862 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa415d8c0 con 0x7f6fa41538f0 2023-10-30T12:46:53.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.737+0000 7f6fa939e640 1 --2- 172.21.15.125:0/1003699862 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa41538f0 0x7f6fa4159fb0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0009870 tx=0x7f6fa001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb7e7c014c081efb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:53.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.737+0000 7f6f9bfff640 1 -- 172.21.15.125:0/1003699862 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa00a8a30 con 0x7f6fa41538f0 2023-10-30T12:46:53.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.737+0000 7f6f9bfff640 1 -- 172.21.15.125:0/1003699862 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa00a8b90 con 0x7f6fa41538f0 2023-10-30T12:46:53.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.737+0000 7f6f9bfff640 1 -- 172.21.15.125:0/1003699862 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa00b0440 con 0x7f6fa41538f0 2023-10-30T12:46:53.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.737+0000 7f6faa3a0640 1 -- 172.21.15.125:0/1003699862 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa41538f0 msgr2=0x7f6fa4159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:53.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.737+0000 7f6faa3a0640 1 --2- 172.21.15.125:0/1003699862 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa41538f0 0x7f6fa4159fb0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f6fa0009870 tx=0x7f6fa001ed30 comp rx=0 tx=0).stop 2023-10-30T12:46:53.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.737+0000 7f6faa3a0640 1 -- 172.21.15.125:0/1003699862 shutdown_connections 2023-10-30T12:46:53.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.737+0000 7f6faa3a0640 1 --2- 172.21.15.125:0/1003699862 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fa40cefa0 0x7f6fa40cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:53.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.737+0000 7f6faa3a0640 1 --2- 172.21.15.125:0/1003699862 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa41538f0 0x7f6fa4159fb0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:53.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.737+0000 7f6faa3a0640 1 -- 172.21.15.125:0/1003699862 >> 172.21.15.125:0/1003699862 conn(0x7f6fa406a640 msgr2=0x7f6fa4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:53.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.737+0000 7f6faa3a0640 1 -- 172.21.15.125:0/1003699862 shutdown_connections 2023-10-30T12:46:53.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.737+0000 7f6faa3a0640 1 -- 172.21.15.125:0/1003699862 wait complete. 2023-10-30T12:46:53.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6faa3a0640 1 Processor -- start 2023-10-30T12:46:53.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6faa3a0640 1 -- start start 2023-10-30T12:46:53.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6faa3a0640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa40cefa0 0x7f6fa4148270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:53.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6faa3a0640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fa41538f0 0x7f6fa4152880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:53.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6faa3a0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fa415cd50 0x7f6fa4151330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:53.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6faa3a0640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa40ce9e0 con 0x7f6fa40cefa0 2023-10-30T12:46:53.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6faa3a0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa41ef600 con 0x7f6fa415cd50 2023-10-30T12:46:53.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6faa3a0640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa40b4e10 con 0x7f6fa41538f0 2023-10-30T12:46:53.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6fa939e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa40cefa0 0x7f6fa4148270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:53.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6fa939e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa40cefa0 0x7f6fa4148270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49658/0 (socket says 172.21.15.125:49658) 2023-10-30T12:46:53.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6fa939e640 1 -- 172.21.15.125:0/75475208 learned_addr learned my addr 172.21.15.125:0/75475208 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:53.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6fa9b9f640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fa415cd50 0x7f6fa4151330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:53.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6fa8b9d640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fa41538f0 0x7f6fa4152880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:53.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6fa939e640 1 -- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fa41538f0 msgr2=0x7f6fa4152880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:53.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6fa939e640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fa41538f0 0x7f6fa4152880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:53.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6fa939e640 1 -- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fa415cd50 msgr2=0x7f6fa4151330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:53.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6fa939e640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fa415cd50 0x7f6fa4151330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:53.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6fa939e640 1 -- 172.21.15.125:0/75475208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa00094d0 con 0x7f6fa40cefa0 2023-10-30T12:46:53.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6fa8b9d640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fa41538f0 0x7f6fa4152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:53.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6fa9b9f640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fa415cd50 0x7f6fa4151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:46:53.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6fa939e640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa40cefa0 0x7f6fa4148270 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f6fa001c960 tx=0x7f6fa001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:53.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6f99ffb640 1 -- 172.21.15.125:0/75475208 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa00b1d90 con 0x7f6fa40cefa0 2023-10-30T12:46:53.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6f99ffb640 1 -- 172.21.15.125:0/75475208 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa00b9440 con 0x7f6fa40cefa0 2023-10-30T12:46:53.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6faa3a0640 1 -- 172.21.15.125:0/75475208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa41ef870 con 0x7f6fa40cefa0 2023-10-30T12:46:53.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6f99ffb640 1 -- 172.21.15.125:0/75475208 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa00b97f0 con 0x7f6fa40cefa0 2023-10-30T12:46:53.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6faa3a0640 1 -- 172.21.15.125:0/75475208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa41efe30 con 0x7f6fa40cefa0 2023-10-30T12:46:53.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.741+0000 7f6faa3a0640 1 -- 172.21.15.125:0/75475208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa41f0020 con 0x7f6fa40cefa0 2023-10-30T12:46:53.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.745+0000 7f6f99ffb640 1 -- 172.21.15.125:0/75475208 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6fa00b7070 con 0x7f6fa40cefa0 2023-10-30T12:46:53.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.745+0000 7f6f99ffb640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f880664a0 0x7f6f88068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:53.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.745+0000 7f6f99ffb640 1 -- 172.21.15.125:0/75475208 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f6fa0121a80 con 0x7f6fa40cefa0 2023-10-30T12:46:53.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.745+0000 7f6fa8b9d640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f880664a0 0x7f6f88068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:53.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.745+0000 7f6fa8b9d640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f880664a0 0x7f6f88068960 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6fa40b46d0 tx=0x7f6f94009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:53.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.749+0000 7f6f99ffb640 1 -- 172.21.15.125:0/75475208 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6fa00eec20 con 0x7f6fa40cefa0 2023-10-30T12:46:53.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.929+0000 7f6faa3a0640 1 -- 172.21.15.125:0/75475208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fa404e170 con 0x7f6fa40cefa0 2023-10-30T12:46:53.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.929+0000 7f6f99ffb640 1 -- 172.21.15.125:0/75475208 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 89 v89) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f6fa00f2240 con 0x7f6fa40cefa0 2023-10-30T12:46:53.929 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:53.930 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":89,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":89,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:49.460277+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:53.930 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 89 2023-10-30T12:46:53.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.929+0000 7f6faa3a0640 1 -- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f880664a0 msgr2=0x7f6f88068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:53.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.929+0000 7f6faa3a0640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f880664a0 0x7f6f88068960 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f6fa40b46d0 tx=0x7f6f94009040 comp rx=0 tx=0).stop 2023-10-30T12:46:53.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.929+0000 7f6faa3a0640 1 -- 172.21.15.125:0/75475208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa40cefa0 msgr2=0x7f6fa4148270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:53.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.929+0000 7f6faa3a0640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa40cefa0 0x7f6fa4148270 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f6fa001c960 tx=0x7f6fa001bad0 comp rx=0 tx=0).stop 2023-10-30T12:46:53.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.929+0000 7f6faa3a0640 1 -- 172.21.15.125:0/75475208 shutdown_connections 2023-10-30T12:46:53.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.929+0000 7f6faa3a0640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6f880664a0 0x7f6f88068960 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:53.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.929+0000 7f6faa3a0640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fa415cd50 0x7f6fa4151330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:53.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.929+0000 7f6faa3a0640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fa41538f0 0x7f6fa4152880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:53.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.929+0000 7f6faa3a0640 1 --2- 172.21.15.125:0/75475208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fa40cefa0 0x7f6fa4148270 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:53.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.929+0000 7f6faa3a0640 1 -- 172.21.15.125:0/75475208 >> 172.21.15.125:0/75475208 conn(0x7f6fa406a640 msgr2=0x7f6fa406bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:53.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.929+0000 7f6faa3a0640 1 -- 172.21.15.125:0/75475208 shutdown_connections 2023-10-30T12:46:53.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:53.929+0000 7f6faa3a0640 1 -- 172.21.15.125:0/75475208 wait complete. 2023-10-30T12:46:53.951 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:46:55.952 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:46:56.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f365231d640 1 Processor -- start 2023-10-30T12:46:56.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f365231d640 1 -- start start 2023-10-30T12:46:56.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f365231d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f364c0c6f60 0x7f364c1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:56.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f365231d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c0c0a50 con 0x7f364c0c05d0 2023-10-30T12:46:56.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f365231d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f364c06f6e0 con 0x7f364c0c6b80 2023-10-30T12:46:56.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f365231d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364c06f9c0 con 0x7f364c0c6f60 2023-10-30T12:46:56.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364b7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f364c0c05d0 0x7f364c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39934/0 (socket says 172.21.15.125:39934) 2023-10-30T12:46:56.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364b7fe640 1 -- 172.21.15.125:0/484346576 learned_addr learned my addr 172.21.15.125:0/484346576 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:46:56.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f36508ba640 1 --2- 172.21.15.125:0/484346576 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f364c0c6f60 0x7f364c1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:56.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 -- 172.21.15.125:0/484346576 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224485387 0 0) 0x7f364c0c0a50 con 0x7f364c0c05d0 2023-10-30T12:46:56.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 -- 172.21.15.125:0/484346576 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3634003530 con 0x7f364c0c05d0 2023-10-30T12:46:56.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 -- 172.21.15.125:0/484346576 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 183102677 0 0) 0x7f3634003530 con 0x7f364c0c05d0 2023-10-30T12:46:56.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 -- 172.21.15.125:0/484346576 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f364c0c6f60 msgr2=0x7f364c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:56.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 --2- 172.21.15.125:0/484346576 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f364c0c6f60 0x7f364c1566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:56.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 -- 172.21.15.125:0/484346576 >> v1:172.21.15.136:6789/0 conn(0x7f364c0c6b80 legacy=0x7f364c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:56.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 -- 172.21.15.125:0/484346576 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c06fca0 con 0x7f364c0c05d0 2023-10-30T12:46:56.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 -- 172.21.15.125:0/484346576 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f3640001680 con 0x7f364c0c05d0 2023-10-30T12:46:56.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 -- 172.21.15.125:0/484346576 >> v1:172.21.15.125:6789/0 conn(0x7f364c0c05d0 legacy=0x7f364c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:56.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 --2- 172.21.15.125:0/484346576 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f364c0c6f60 0x7f36340039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:56.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 --2- 172.21.15.125:0/484346576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3634003ef0 0x7f36340047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:56.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 --2- 172.21.15.125:0/484346576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3634004d10 0x7f3634005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:56.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 -- 172.21.15.125:0/484346576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c06f9c0 con 0x7f3634004d10 2023-10-30T12:46:56.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 -- 172.21.15.125:0/484346576 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363400b9f0 con 0x7f3634003ef0 2023-10-30T12:46:56.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364affd640 1 -- 172.21.15.125:0/484346576 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363400ddc0 con 0x7f364c0c6f60 2023-10-30T12:46:56.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f36508ba640 1 --2- 172.21.15.125:0/484346576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3634004d10 0x7f3634005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:56.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364bfff640 1 --2- 172.21.15.125:0/484346576 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f364c0c6f60 0x7f36340039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:56.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f364b7fe640 1 --2- 172.21.15.125:0/484346576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3634003ef0 0x7f36340047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:56.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f36508ba640 1 -- 172.21.15.125:0/484346576 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f364c0c6f60 msgr2=0x7f36340039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:56.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f36508ba640 1 --2- 172.21.15.125:0/484346576 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f364c0c6f60 0x7f36340039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:56.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f36508ba640 1 -- 172.21.15.125:0/484346576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3634003ef0 msgr2=0x7f36340047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:46:56.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f36508ba640 1 --2- 172.21.15.125:0/484346576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3634003ef0 0x7f36340047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:56.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.101+0000 7f36508ba640 1 -- 172.21.15.125:0/484346576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3634003530 con 0x7f3634004d10 2023-10-30T12:46:56.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.105+0000 7f36508ba640 1 --2- 172.21.15.125:0/484346576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3634004d10 0x7f3634005190 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3644009fc0 tx=0x7f364401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ed8fb6673ba91 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:56.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.105+0000 7f364affd640 1 -- 172.21.15.125:0/484346576 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36440a5c70 con 0x7f3634004d10 2023-10-30T12:46:56.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.105+0000 7f364affd640 1 -- 172.21.15.125:0/484346576 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364400f040 con 0x7f3634004d10 2023-10-30T12:46:56.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.105+0000 7f364affd640 1 -- 172.21.15.125:0/484346576 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36440ad440 con 0x7f3634004d10 2023-10-30T12:46:56.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.105+0000 7f365231d640 1 -- 172.21.15.125:0/484346576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3634004d10 msgr2=0x7f3634005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:56.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.105+0000 7f365231d640 1 --2- 172.21.15.125:0/484346576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3634004d10 0x7f3634005190 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f3644009fc0 tx=0x7f364401cc60 comp rx=0 tx=0).stop 2023-10-30T12:46:56.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.105+0000 7f36508ba640 1 -- 172.21.15.125:0/484346576 reap_dead start 2023-10-30T12:46:56.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.105+0000 7f365231d640 1 -- 172.21.15.125:0/484346576 shutdown_connections 2023-10-30T12:46:56.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.105+0000 7f365231d640 1 -- 172.21.15.125:0/484346576 >> 172.21.15.125:0/484346576 conn(0x7f364c0c7910 msgr2=0x7f364c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:56.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.105+0000 7f365231d640 1 -- 172.21.15.125:0/484346576 shutdown_connections 2023-10-30T12:46:56.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.105+0000 7f365231d640 1 -- 172.21.15.125:0/484346576 wait complete. 2023-10-30T12:46:56.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f365231d640 1 Processor -- start 2023-10-30T12:46:56.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f365231d640 1 -- start start 2023-10-30T12:46:56.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f365231d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3634004d10 0x7f364c1dfe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:56.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f365231d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f364c0ca4c0 0x7f364c0c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:56.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f365231d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f364c0ca920 0x7f364c0c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:56.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f365231d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c0adc00 con 0x7f364c0ca920 2023-10-30T12:46:56.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f365231d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364c0c8e20 con 0x7f364c0ca4c0 2023-10-30T12:46:56.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f365231d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f364c1f21f0 con 0x7f3634004d10 2023-10-30T12:46:56.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f36508ba640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f364c0ca920 0x7f364c0c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:56.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f36508ba640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f364c0ca920 0x7f364c0c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49674/0 (socket says 172.21.15.125:49674) 2023-10-30T12:46:56.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f36508ba640 1 -- 172.21.15.125:0/1792106327 learned_addr learned my addr 172.21.15.125:0/1792106327 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:46:56.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f364b7fe640 1 --2- 172.21.15.125:0/1792106327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f364c0ca4c0 0x7f364c0c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:56.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f364bfff640 1 --2- 172.21.15.125:0/1792106327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3634004d10 0x7f364c1dfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:56.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f36508ba640 1 -- 172.21.15.125:0/1792106327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3634004d10 msgr2=0x7f364c1dfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:56.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f36508ba640 1 --2- 172.21.15.125:0/1792106327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3634004d10 0x7f364c1dfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:56.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f36508ba640 1 -- 172.21.15.125:0/1792106327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f364c0ca4c0 msgr2=0x7f364c0c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:56.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f36508ba640 1 --2- 172.21.15.125:0/1792106327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f364c0ca4c0 0x7f364c0c92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:56.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f36508ba640 1 -- 172.21.15.125:0/1792106327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f364c1f24d0 con 0x7f364c0ca920 2023-10-30T12:46:56.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f36508ba640 1 --2- 172.21.15.125:0/1792106327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f364c0ca920 0x7f364c0c97f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f36440b0040 tx=0x7f3644009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:56.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f3648ff9640 1 -- 172.21.15.125:0/1792106327 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36440bc070 con 0x7f364c0ca920 2023-10-30T12:46:56.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f3648ff9640 1 -- 172.21.15.125:0/1792106327 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36440039a0 con 0x7f364c0ca920 2023-10-30T12:46:56.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f365231d640 1 -- 172.21.15.125:0/1792106327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c1f27b0 con 0x7f364c0ca920 2023-10-30T12:46:56.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f3648ff9640 1 -- 172.21.15.125:0/1792106327 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3644003b30 con 0x7f364c0ca920 2023-10-30T12:46:56.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.109+0000 7f365231d640 1 -- 172.21.15.125:0/1792106327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364c1e0350 con 0x7f364c0ca920 2023-10-30T12:46:56.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.113+0000 7f365231d640 1 -- 172.21.15.125:0/1792106327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f364c1f2c60 con 0x7f364c0ca920 2023-10-30T12:46:56.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.113+0000 7f3648ff9640 1 -- 172.21.15.125:0/1792106327 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f36440b4040 con 0x7f364c0ca920 2023-10-30T12:46:56.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.113+0000 7f3648ff9640 1 --2- 172.21.15.125:0/1792106327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f36380664d0 0x7f3638068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:46:56.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.113+0000 7f3648ff9640 1 -- 172.21.15.125:0/1792106327 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f364411e3c0 con 0x7f364c0ca920 2023-10-30T12:46:56.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.113+0000 7f364bfff640 1 --2- 172.21.15.125:0/1792106327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f36380664d0 0x7f3638068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:46:56.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.113+0000 7f364bfff640 1 --2- 172.21.15.125:0/1792106327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f36380664d0 0x7f3638068990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f363c000c10 tx=0x7f363c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:46:56.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.117+0000 7f3648ff9640 1 -- 172.21.15.125:0/1792106327 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f36440eb4f0 con 0x7f364c0ca920 2023-10-30T12:46:56.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.301+0000 7f365231d640 1 -- 172.21.15.125:0/1792106327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f364c04e170 con 0x7f364c0ca920 2023-10-30T12:46:56.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.301+0000 7f3648ff9640 1 -- 172.21.15.125:0/1792106327 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f36440b2090 con 0x7f364c0ca920 2023-10-30T12:46:56.302 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:46:56.302 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:55.354589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:46:56.303 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 90 2023-10-30T12:46:56.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.301+0000 7f365231d640 1 -- 172.21.15.125:0/1792106327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f36380664d0 msgr2=0x7f3638068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:56.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.301+0000 7f365231d640 1 --2- 172.21.15.125:0/1792106327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f36380664d0 0x7f3638068990 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f363c000c10 tx=0x7f363c009040 comp rx=0 tx=0).stop 2023-10-30T12:46:56.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.301+0000 7f365231d640 1 -- 172.21.15.125:0/1792106327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f364c0ca920 msgr2=0x7f364c0c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:46:56.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.301+0000 7f365231d640 1 --2- 172.21.15.125:0/1792106327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f364c0ca920 0x7f364c0c97f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f36440b0040 tx=0x7f3644009750 comp rx=0 tx=0).stop 2023-10-30T12:46:56.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.301+0000 7f365231d640 1 -- 172.21.15.125:0/1792106327 shutdown_connections 2023-10-30T12:46:56.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.301+0000 7f365231d640 1 --2- 172.21.15.125:0/1792106327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f36380664d0 0x7f3638068990 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:56.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.301+0000 7f365231d640 1 --2- 172.21.15.125:0/1792106327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f364c0ca920 0x7f364c0c97f0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:56.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.301+0000 7f365231d640 1 --2- 172.21.15.125:0/1792106327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f364c0ca4c0 0x7f364c0c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:56.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.301+0000 7f365231d640 1 --2- 172.21.15.125:0/1792106327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3634004d10 0x7f364c1dfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:46:56.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.301+0000 7f365231d640 1 -- 172.21.15.125:0/1792106327 >> 172.21.15.125:0/1792106327 conn(0x7f364c0c7910 msgr2=0x7f364c0c0c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:46:56.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.305+0000 7f365231d640 1 -- 172.21.15.125:0/1792106327 shutdown_connections 2023-10-30T12:46:56.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:46:56.305+0000 7f365231d640 1 -- 172.21.15.125:0/1792106327 wait complete. 2023-10-30T12:46:56.323 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=5433 2023-10-30T12:46:56.323 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.e 2023-10-30T12:47:00.324 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T12:47:00.325 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T12:47:00.325 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T12:47:00.329 INFO:tasks.ceph.mds.e:Started 2023-10-30T12:47:00.329 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:00.420 INFO:tasks.ceph.mds.e.smithi125.stdout:starting mds.e at 2023-10-30T12:47:00.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e78d6640 1 Processor -- start 2023-10-30T12:47:00.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e78d6640 1 -- start start 2023-10-30T12:47:00.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e78d6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f60e006f7a0 0x7f60e006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:00.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e78d6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f60e00c6bd0 0x7f60e0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:00.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e78d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60e006a3d0 con 0x7f60e0071110 2023-10-30T12:47:00.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e78d6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e00700c0 con 0x7f60e006f7a0 2023-10-30T12:47:00.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e78d6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60e0066cf0 con 0x7f60e00c6bd0 2023-10-30T12:47:00.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e5672640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f60e006f7a0 0x7f60e006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:00.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e5672640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f60e006f7a0 0x7f60e006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49690/0 (socket says 172.21.15.125:49690) 2023-10-30T12:47:00.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e5672640 1 -- 172.21.15.125:0/1571711925 learned_addr learned my addr 172.21.15.125:0/1571711925 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:00.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e4e71640 1 --2- 172.21.15.125:0/1571711925 >> v2:172.21.15.136:3300/0 conn(0x7f60e00c6bd0 0x7f60e0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:00.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e5672640 1 -- 172.21.15.125:0/1571711925 >> v2:172.21.15.136:3300/0 conn(0x7f60e00c6bd0 msgr2=0x7f60e0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:00.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e5672640 1 --2- 172.21.15.125:0/1571711925 >> v2:172.21.15.136:3300/0 conn(0x7f60e00c6bd0 0x7f60e0069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:00.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e5672640 1 -- 172.21.15.125:0/1571711925 >> v1:172.21.15.136:6789/0 conn(0x7f60e0071110 legacy=0x7f60e0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:00.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.473+0000 7f60e5672640 1 -- 172.21.15.125:0/1571711925 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60e00c70d0 con 0x7f60e006f7a0 2023-10-30T12:47:00.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.477+0000 7f60e5672640 1 --2- 172.21.15.125:0/1571711925 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60e006f7a0 0x7f60e006fb80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f60e006f1c0 tx=0x7f60dc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8975a1a96dbcd00a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:00.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.477+0000 7f60d7fff640 1 -- 172.21.15.125:0/1571711925 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc0a89f0 con 0x7f60e006f7a0 2023-10-30T12:47:00.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.477+0000 7f60d7fff640 1 -- 172.21.15.125:0/1571711925 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60dc0a8b50 con 0x7f60e006f7a0 2023-10-30T12:47:00.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.477+0000 7f60d7fff640 1 -- 172.21.15.125:0/1571711925 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc0b0440 con 0x7f60e006f7a0 2023-10-30T12:47:00.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.477+0000 7f60e78d6640 1 -- 172.21.15.125:0/1571711925 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60e006f7a0 msgr2=0x7f60e006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:00.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.477+0000 7f60e78d6640 1 --2- 172.21.15.125:0/1571711925 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60e006f7a0 0x7f60e006fb80 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f60e006f1c0 tx=0x7f60dc01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:47:00.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.477+0000 7f60e78d6640 1 -- 172.21.15.125:0/1571711925 shutdown_connections 2023-10-30T12:47:00.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.477+0000 7f60e78d6640 1 --2- 172.21.15.125:0/1571711925 >> v2:172.21.15.136:3300/0 conn(0x7f60e00c6bd0 0x7f60e0069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:00.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.477+0000 7f60e78d6640 1 --2- 172.21.15.125:0/1571711925 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60e006f7a0 0x7f60e006fb80 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:00.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.477+0000 7f60e78d6640 1 -- 172.21.15.125:0/1571711925 >> 172.21.15.125:0/1571711925 conn(0x7f60e00c7910 msgr2=0x7f60e00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:00.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.477+0000 7f60e78d6640 1 -- 172.21.15.125:0/1571711925 shutdown_connections 2023-10-30T12:47:00.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.477+0000 7f60e78d6640 1 -- 172.21.15.125:0/1571711925 wait complete. 2023-10-30T12:47:00.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e78d6640 1 Processor -- start 2023-10-30T12:47:00.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e78d6640 1 -- start start 2023-10-30T12:47:00.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e78d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60e006f7a0 0x7f60e00c1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:00.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e78d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60e0071110 0x7f60e014e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:00.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e78d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60e00c6bd0 0x7f60e0156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:00.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e78d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e01e2fa0 con 0x7f60e00c6bd0 2023-10-30T12:47:00.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e78d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60e01ef5d0 con 0x7f60e006f7a0 2023-10-30T12:47:00.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e78d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60e01efb00 con 0x7f60e0071110 2023-10-30T12:47:00.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e5e73640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60e00c6bd0 0x7f60e0156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:00.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e5e73640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60e00c6bd0 0x7f60e0156d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49696/0 (socket says 172.21.15.125:49696) 2023-10-30T12:47:00.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e5e73640 1 -- 172.21.15.125:0/3574554642 learned_addr learned my addr 172.21.15.125:0/3574554642 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:00.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e5672640 1 --2- 172.21.15.125:0/3574554642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60e006f7a0 0x7f60e00c1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:00.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e4e71640 1 --2- 172.21.15.125:0/3574554642 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60e0071110 0x7f60e014e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:00.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e5e73640 1 -- 172.21.15.125:0/3574554642 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60e0071110 msgr2=0x7f60e014e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:00.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e5e73640 1 --2- 172.21.15.125:0/3574554642 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60e0071110 0x7f60e014e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:00.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e5e73640 1 -- 172.21.15.125:0/3574554642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60e006f7a0 msgr2=0x7f60e00c1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:00.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e5e73640 1 --2- 172.21.15.125:0/3574554642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60e006f7a0 0x7f60e00c1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:00.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e5e73640 1 -- 172.21.15.125:0/3574554642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60dc0094d0 con 0x7f60e00c6bd0 2023-10-30T12:47:00.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e5e73640 1 --2- 172.21.15.125:0/3574554642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60e00c6bd0 0x7f60e0156d40 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f60d800c760 tx=0x7f60d800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:00.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60d5ffb640 1 -- 172.21.15.125:0/3574554642 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d8003b80 con 0x7f60e00c6bd0 2023-10-30T12:47:00.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60d5ffb640 1 -- 172.21.15.125:0/3574554642 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60d8003ce0 con 0x7f60e00c6bd0 2023-10-30T12:47:00.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e78d6640 1 -- 172.21.15.125:0/3574554642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60e01e22c0 con 0x7f60e00c6bd0 2023-10-30T12:47:00.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60d5ffb640 1 -- 172.21.15.125:0/3574554642 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d800aa30 con 0x7f60e00c6bd0 2023-10-30T12:47:00.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e78d6640 1 -- 172.21.15.125:0/3574554642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60e01e1d00 con 0x7f60e00c6bd0 2023-10-30T12:47:00.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.481+0000 7f60e78d6640 1 -- 172.21.15.125:0/3574554642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60e01e30e0 con 0x7f60e00c6bd0 2023-10-30T12:47:00.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.485+0000 7f60d5ffb640 1 -- 172.21.15.125:0/3574554642 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f60d800ab90 con 0x7f60e00c6bd0 2023-10-30T12:47:00.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.485+0000 7f60d5ffb640 1 --2- 172.21.15.125:0/3574554642 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60c40664a0 0x7f60c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:00.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.485+0000 7f60d5ffb640 1 -- 172.21.15.125:0/3574554642 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f60d8083ea0 con 0x7f60e00c6bd0 2023-10-30T12:47:00.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.485+0000 7f60e5672640 1 --2- 172.21.15.125:0/3574554642 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60c40664a0 0x7f60c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:00.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.485+0000 7f60e5672640 1 --2- 172.21.15.125:0/3574554642 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60c40664a0 0x7f60c4068960 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f60dc0b2000 tx=0x7f60dc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:00.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.489+0000 7f60d5ffb640 1 -- 172.21.15.125:0/3574554642 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f60e01e30e0 con 0x7f60e00c6bd0 2023-10-30T12:47:00.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.669+0000 7f60e78d6640 1 -- 172.21.15.125:0/3574554642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60e01e30e0 con 0x7f60e00c6bd0 2023-10-30T12:47:00.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.669+0000 7f60d5ffb640 1 -- 172.21.15.125:0/3574554642 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f60e01e30e0 con 0x7f60e00c6bd0 2023-10-30T12:47:00.671 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:00.671 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:55.354589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:00.671 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 90 2023-10-30T12:47:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.673+0000 7f60e78d6640 1 -- 172.21.15.125:0/3574554642 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60c40664a0 msgr2=0x7f60c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.673+0000 7f60e78d6640 1 --2- 172.21.15.125:0/3574554642 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60c40664a0 0x7f60c4068960 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f60dc0b2000 tx=0x7f60dc008040 comp rx=0 tx=0).stop 2023-10-30T12:47:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.673+0000 7f60e78d6640 1 -- 172.21.15.125:0/3574554642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60e00c6bd0 msgr2=0x7f60e0156d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:00.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.673+0000 7f60e78d6640 1 --2- 172.21.15.125:0/3574554642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60e00c6bd0 0x7f60e0156d40 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f60d800c760 tx=0x7f60d800cc20 comp rx=0 tx=0).stop 2023-10-30T12:47:00.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.673+0000 7f60e78d6640 1 -- 172.21.15.125:0/3574554642 shutdown_connections 2023-10-30T12:47:00.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.673+0000 7f60e78d6640 1 --2- 172.21.15.125:0/3574554642 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60c40664a0 0x7f60c4068960 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:00.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.673+0000 7f60e78d6640 1 --2- 172.21.15.125:0/3574554642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60e00c6bd0 0x7f60e0156d40 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:00.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.673+0000 7f60e78d6640 1 --2- 172.21.15.125:0/3574554642 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60e0071110 0x7f60e014e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:00.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.673+0000 7f60e78d6640 1 --2- 172.21.15.125:0/3574554642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60e006f7a0 0x7f60e00c1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:00.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.673+0000 7f60e78d6640 1 -- 172.21.15.125:0/3574554642 >> 172.21.15.125:0/3574554642 conn(0x7f60e00c7910 msgr2=0x7f60e00c3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:00.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.673+0000 7f60e78d6640 1 -- 172.21.15.125:0/3574554642 shutdown_connections 2023-10-30T12:47:00.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:00.673+0000 7f60e78d6640 1 -- 172.21.15.125:0/3574554642 wait complete. 2023-10-30T12:47:00.691 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T12:47:02.692 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:02.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f59946d6640 1 Processor -- start 2023-10-30T12:47:02.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f59946d6640 1 -- start start 2023-10-30T12:47:02.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f59946d6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f598c153b10 0x7f598c15ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:02.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f59946d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598c0c74f0 con 0x7f598c0c5e70 2023-10-30T12:47:02.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f59946d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f598c0c77b0 con 0x7f598c0b0dc0 2023-10-30T12:47:02.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f59946d6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f598c0c1f20 con 0x7f598c153b10 2023-10-30T12:47:02.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5992c73640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f598c0c5e70 0x7f598c1641f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33792/0 (socket says 172.21.15.125:33792) 2023-10-30T12:47:02.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5992c73640 1 -- 172.21.15.125:0/4058523928 learned_addr learned my addr 172.21.15.125:0/4058523928 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:47:02.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5992472640 1 --2- 172.21.15.125:0/4058523928 >> v2:172.21.15.136:3300/0 conn(0x7f598c153b10 0x7f598c15ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:02.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 -- 172.21.15.125:0/4058523928 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164725536 0 0) 0x7f598c0c74f0 con 0x7f598c0c5e70 2023-10-30T12:47:02.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 -- 172.21.15.125:0/4058523928 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5978003530 con 0x7f598c0c5e70 2023-10-30T12:47:02.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 -- 172.21.15.125:0/4058523928 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2913406294 0 0) 0x7f5978003530 con 0x7f598c0c5e70 2023-10-30T12:47:02.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 -- 172.21.15.125:0/4058523928 >> v2:172.21.15.136:3300/0 conn(0x7f598c153b10 msgr2=0x7f598c15ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:02.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 --2- 172.21.15.125:0/4058523928 >> v2:172.21.15.136:3300/0 conn(0x7f598c153b10 0x7f598c15ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:02.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 -- 172.21.15.125:0/4058523928 >> v1:172.21.15.136:6789/0 conn(0x7f598c0b0dc0 legacy=0x7f598c0c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:02.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 -- 172.21.15.125:0/4058523928 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c159b80 con 0x7f598c0c5e70 2023-10-30T12:47:02.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 -- 172.21.15.125:0/4058523928 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f5984001a00 con 0x7f598c0c5e70 2023-10-30T12:47:02.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 -- 172.21.15.125:0/4058523928 >> v1:172.21.15.125:6789/0 conn(0x7f598c0c5e70 legacy=0x7f598c1641f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:02.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 --2- 172.21.15.125:0/4058523928 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5978003980 0x7f5978003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:02.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 --2- 172.21.15.125:0/4058523928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59780047d0 0x7f5978004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:02.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 --2- 172.21.15.125:0/4058523928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5978005190 0x7f597800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:02.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 -- 172.21.15.125:0/4058523928 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597800c010 con 0x7f5978005190 2023-10-30T12:47:02.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 -- 172.21.15.125:0/4058523928 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f597800c180 con 0x7f59780047d0 2023-10-30T12:47:02.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.837+0000 7f5991470640 1 -- 172.21.15.125:0/4058523928 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f597800c340 con 0x7f5978003980 2023-10-30T12:47:02.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.841+0000 7f5992c73640 1 --2- 172.21.15.125:0/4058523928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5978005190 0x7f597800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:02.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.841+0000 7f5991c71640 1 --2- 172.21.15.125:0/4058523928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59780047d0 0x7f5978004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:02.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.841+0000 7f5992472640 1 --2- 172.21.15.125:0/4058523928 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5978003980 0x7f5978003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:02.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.841+0000 7f5992c73640 1 -- 172.21.15.125:0/4058523928 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5978003980 msgr2=0x7f5978003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:02.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.841+0000 7f5992c73640 1 --2- 172.21.15.125:0/4058523928 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5978003980 0x7f5978003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:02.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.841+0000 7f5992c73640 1 -- 172.21.15.125:0/4058523928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59780047d0 msgr2=0x7f5978004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:02.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.841+0000 7f5992c73640 1 --2- 172.21.15.125:0/4058523928 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f59780047d0 0x7f5978004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:02.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.841+0000 7f5992c73640 1 -- 172.21.15.125:0/4058523928 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5978003530 con 0x7f5978005190 2023-10-30T12:47:02.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.841+0000 7f5992c73640 1 -- 172.21.15.125:0/4058523928 reap_dead start 2023-10-30T12:47:02.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.841+0000 7f5992c73640 1 --2- 172.21.15.125:0/4058523928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5978005190 0x7f597800b990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f598400a820 tx=0x7f598401d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e39b4a43c1e7bd9d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:02.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.841+0000 7f5991470640 1 -- 172.21.15.125:0/4058523928 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59840072d0 con 0x7f5978005190 2023-10-30T12:47:02.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.841+0000 7f5991470640 1 -- 172.21.15.125:0/4058523928 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59840a8070 con 0x7f5978005190 2023-10-30T12:47:02.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.841+0000 7f5991470640 1 -- 172.21.15.125:0/4058523928 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59840076b0 con 0x7f5978005190 2023-10-30T12:47:02.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 -- 172.21.15.125:0/4058523928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5978005190 msgr2=0x7f597800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:02.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 --2- 172.21.15.125:0/4058523928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5978005190 0x7f597800b990 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f598400a820 tx=0x7f598401d320 comp rx=0 tx=0).stop 2023-10-30T12:47:02.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 -- 172.21.15.125:0/4058523928 shutdown_connections 2023-10-30T12:47:02.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 --2- 172.21.15.125:0/4058523928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5978005190 0x7f597800b990 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:02.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 -- 172.21.15.125:0/4058523928 >> 172.21.15.125:0/4058523928 conn(0x7f598c06d770 msgr2=0x7f598c06db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:02.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 -- 172.21.15.125:0/4058523928 shutdown_connections 2023-10-30T12:47:02.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 -- 172.21.15.125:0/4058523928 wait complete. 2023-10-30T12:47:02.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 Processor -- start 2023-10-30T12:47:02.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 -- start start 2023-10-30T12:47:02.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5978005190 0x7f598c14fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:02.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f598c1ef1d0 0x7f598c14f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:02.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f598c0c9f70 0x7f598c0ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:02.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c150140 con 0x7f5978005190 2023-10-30T12:47:02.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598c14eb70 con 0x7f598c1ef1d0 2023-10-30T12:47:02.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f59946d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f598c14df60 con 0x7f598c0c9f70 2023-10-30T12:47:02.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f5992472640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5978005190 0x7f598c14fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:02.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f5992472640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5978005190 0x7f598c14fa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50728/0 (socket says 172.21.15.125:50728) 2023-10-30T12:47:02.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f5992472640 1 -- 172.21.15.125:0/1972704259 learned_addr learned my addr 172.21.15.125:0/1972704259 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:02.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f5991c71640 1 --2- 172.21.15.125:0/1972704259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f598c1ef1d0 0x7f598c14f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:02.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f5992c73640 1 --2- 172.21.15.125:0/1972704259 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f598c0c9f70 0x7f598c0ca3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:02.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f5992472640 1 -- 172.21.15.125:0/1972704259 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f598c0c9f70 msgr2=0x7f598c0ca3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:02.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f5992472640 1 --2- 172.21.15.125:0/1972704259 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f598c0c9f70 0x7f598c0ca3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:02.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f5992472640 1 -- 172.21.15.125:0/1972704259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f598c1ef1d0 msgr2=0x7f598c14f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:02.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f5992472640 1 --2- 172.21.15.125:0/1972704259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f598c1ef1d0 0x7f598c14f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:02.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f5992472640 1 -- 172.21.15.125:0/1972704259 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598c14e240 con 0x7f5978005190 2023-10-30T12:47:02.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.845+0000 7f5992472640 1 --2- 172.21.15.125:0/1972704259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5978005190 0x7f598c14fa40 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5988011960 tx=0x7f5988011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:02.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.849+0000 7f5982ffd640 1 -- 172.21.15.125:0/1972704259 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598800faf0 con 0x7f5978005190 2023-10-30T12:47:02.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.849+0000 7f5982ffd640 1 -- 172.21.15.125:0/1972704259 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f598800fc50 con 0x7f5978005190 2023-10-30T12:47:02.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.849+0000 7f5982ffd640 1 -- 172.21.15.125:0/1972704259 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5988014440 con 0x7f5978005190 2023-10-30T12:47:02.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.849+0000 7f59946d6640 1 -- 172.21.15.125:0/1972704259 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598c14e520 con 0x7f5978005190 2023-10-30T12:47:02.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.849+0000 7f59946d6640 1 -- 172.21.15.125:0/1972704259 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598c0caae0 con 0x7f5978005190 2023-10-30T12:47:02.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.849+0000 7f59946d6640 1 -- 172.21.15.125:0/1972704259 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f598c074250 con 0x7f5978005190 2023-10-30T12:47:02.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.849+0000 7f5982ffd640 1 -- 172.21.15.125:0/1972704259 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f59880145a0 con 0x7f5978005190 2023-10-30T12:47:02.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.849+0000 7f5982ffd640 1 --2- 172.21.15.125:0/1972704259 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59700664d0 0x7f5970068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:02.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.849+0000 7f5982ffd640 1 -- 172.21.15.125:0/1972704259 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f5988084a90 con 0x7f5978005190 2023-10-30T12:47:02.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.849+0000 7f5991c71640 1 --2- 172.21.15.125:0/1972704259 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59700664d0 0x7f5970068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:02.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.853+0000 7f5991c71640 1 --2- 172.21.15.125:0/1972704259 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59700664d0 0x7f5970068990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f598c072160 tx=0x7f597c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:02.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:02.853+0000 7f5982ffd640 1 -- 172.21.15.125:0/1972704259 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5988051b10 con 0x7f5978005190 2023-10-30T12:47:03.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:03.049+0000 7f59946d6640 1 -- 172.21.15.125:0/1972704259 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f598c04e170 con 0x7f5978005190 2023-10-30T12:47:03.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:03.049+0000 7f5982ffd640 1 -- 172.21.15.125:0/1972704259 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f5988022070 con 0x7f5978005190 2023-10-30T12:47:03.049 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:03.049 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6318,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:55.354589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:03.049 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 91 2023-10-30T12:47:03.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:03.049+0000 7f59946d6640 1 -- 172.21.15.125:0/1972704259 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59700664d0 msgr2=0x7f5970068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:03.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:03.049+0000 7f59946d6640 1 --2- 172.21.15.125:0/1972704259 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59700664d0 0x7f5970068990 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f598c072160 tx=0x7f597c009040 comp rx=0 tx=0).stop 2023-10-30T12:47:03.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:03.049+0000 7f59946d6640 1 -- 172.21.15.125:0/1972704259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5978005190 msgr2=0x7f598c14fa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:03.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:03.049+0000 7f59946d6640 1 --2- 172.21.15.125:0/1972704259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5978005190 0x7f598c14fa40 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5988011960 tx=0x7f5988011e20 comp rx=0 tx=0).stop 2023-10-30T12:47:03.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:03.049+0000 7f59946d6640 1 -- 172.21.15.125:0/1972704259 shutdown_connections 2023-10-30T12:47:03.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:03.049+0000 7f59946d6640 1 --2- 172.21.15.125:0/1972704259 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f59700664d0 0x7f5970068990 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:03.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:03.049+0000 7f59946d6640 1 --2- 172.21.15.125:0/1972704259 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f598c0c9f70 0x7f598c0ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:03.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:03.049+0000 7f59946d6640 1 --2- 172.21.15.125:0/1972704259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f598c1ef1d0 0x7f598c14f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:03.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:03.049+0000 7f59946d6640 1 --2- 172.21.15.125:0/1972704259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5978005190 0x7f598c14fa40 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:03.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:03.049+0000 7f59946d6640 1 -- 172.21.15.125:0/1972704259 >> 172.21.15.125:0/1972704259 conn(0x7f598c06d770 msgr2=0x7f598c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:03.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:03.049+0000 7f59946d6640 1 -- 172.21.15.125:0/1972704259 shutdown_connections 2023-10-30T12:47:03.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:03.049+0000 7f59946d6640 1 -- 172.21.15.125:0/1972704259 wait complete. 2023-10-30T12:47:03.071 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T12:47:03.071 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before thrashing 2023-10-30T12:47:03.445 INFO:tasks.ceph.mds.f.smithi136.stderr:2023-10-30T12:47:03.447+0000 7f04bca81640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:47:03.445 INFO:tasks.ceph.mds.f.smithi136.stderr:2023-10-30T12:47:03.447+0000 7f04bca81640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:47:10.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:10.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.220+0000 7f70f05f8640 1 Processor -- start 2023-10-30T12:47:10.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.220+0000 7f70f05f8640 1 -- start start 2023-10-30T12:47:10.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.220+0000 7f70f05f8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e81538a0 0x7f70e815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70f05f8640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f70e80ca0e0 0x7f70e815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70f05f8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f70e80cf160 0x7f70e80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70f05f8640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e80cb800 con 0x7f70e80ca0e0 2023-10-30T12:47:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70f05f8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70e815dc30 con 0x7f70e80cf160 2023-10-30T12:47:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70f05f8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e80cb330 con 0x7f70e81538a0 2023-10-30T12:47:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70edb93640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f70e80ca0e0 0x7f70e815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70edb93640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f70e80ca0e0 0x7f70e815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50736/0 (socket says 172.21.15.125:50736) 2023-10-30T12:47:10.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70edb93640 1 -- 172.21.15.125:0/2363079465 learned_addr learned my addr 172.21.15.125:0/2363079465 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:10.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70eeb95640 1 --2- 172.21.15.125:0/2363079465 >> v2:172.21.15.136:3300/0 conn(0x7f70e80cf160 0x7f70e80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70ee394640 1 --2- 172.21.15.125:0/2363079465 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e81538a0 0x7f70e815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70edb93640 1 -- 172.21.15.125:0/2363079465 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e81538a0 msgr2=0x7f70e815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:10.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70edb93640 1 --2- 172.21.15.125:0/2363079465 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e81538a0 0x7f70e815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70edb93640 1 -- 172.21.15.125:0/2363079465 >> v2:172.21.15.136:3300/0 conn(0x7f70e80cf160 msgr2=0x7f70e80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:10.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70edb93640 1 --2- 172.21.15.125:0/2363079465 >> v2:172.21.15.136:3300/0 conn(0x7f70e80cf160 0x7f70e80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70edb93640 1 -- 172.21.15.125:0/2363079465 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e80cad70 con 0x7f70e80ca0e0 2023-10-30T12:47:10.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70eeb95640 1 --2- 172.21.15.125:0/2363079465 >> v2:172.21.15.136:3300/0 conn(0x7f70e80cf160 0x7f70e80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:47:10.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70edb93640 1 --2- 172.21.15.125:0/2363079465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e80ca0e0 0x7f70e815a090 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f70e806cb00 tx=0x7f70d801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=542b10a45df859fc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:10.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.224+0000 7f70ed392640 1 -- 172.21.15.125:0/2363079465 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d80a89b0 con 0x7f70e80ca0e0 2023-10-30T12:47:10.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70ed392640 1 -- 172.21.15.125:0/2363079465 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70d80a8b10 con 0x7f70e80ca0e0 2023-10-30T12:47:10.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70ed392640 1 -- 172.21.15.125:0/2363079465 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70d80a8ea0 con 0x7f70e80ca0e0 2023-10-30T12:47:10.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 -- 172.21.15.125:0/2363079465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e80ca0e0 msgr2=0x7f70e815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:10.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 --2- 172.21.15.125:0/2363079465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e80ca0e0 0x7f70e815a090 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f70e806cb00 tx=0x7f70d801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 -- 172.21.15.125:0/2363079465 shutdown_connections 2023-10-30T12:47:10.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 --2- 172.21.15.125:0/2363079465 >> v2:172.21.15.136:3300/0 conn(0x7f70e80cf160 0x7f70e80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 --2- 172.21.15.125:0/2363079465 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e80ca0e0 0x7f70e815a090 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 --2- 172.21.15.125:0/2363079465 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e81538a0 0x7f70e815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 -- 172.21.15.125:0/2363079465 >> 172.21.15.125:0/2363079465 conn(0x7f70e806bfe0 msgr2=0x7f70e806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:10.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 -- 172.21.15.125:0/2363079465 shutdown_connections 2023-10-30T12:47:10.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 -- 172.21.15.125:0/2363079465 wait complete. 2023-10-30T12:47:10.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 Processor -- start 2023-10-30T12:47:10.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 -- start start 2023-10-30T12:47:10.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70e80ca0e0 0x7f70e81e5880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e80cf160 0x7f70e81e3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e81538a0 0x7f70e81ef8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e80ccfb0 con 0x7f70e81538a0 2023-10-30T12:47:10.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e80cd1c0 con 0x7f70e80ca0e0 2023-10-30T12:47:10.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70f05f8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e80cd4a0 con 0x7f70e80cf160 2023-10-30T12:47:10.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70eeb95640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e81538a0 0x7f70e81ef8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70eeb95640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e81538a0 0x7f70e81ef8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50748/0 (socket says 172.21.15.125:50748) 2023-10-30T12:47:10.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70eeb95640 1 -- 172.21.15.125:0/1839606595 learned_addr learned my addr 172.21.15.125:0/1839606595 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:10.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70ee394640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70e80ca0e0 0x7f70e81e5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.228+0000 7f70edb93640 1 --2- 172.21.15.125:0/1839606595 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e80cf160 0x7f70e81e3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.232+0000 7f70eeb95640 1 -- 172.21.15.125:0/1839606595 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e80cf160 msgr2=0x7f70e81e3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:10.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.232+0000 7f70eeb95640 1 --2- 172.21.15.125:0/1839606595 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e80cf160 0x7f70e81e3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.232+0000 7f70eeb95640 1 -- 172.21.15.125:0/1839606595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70e80ca0e0 msgr2=0x7f70e81e5880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:10.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.232+0000 7f70eeb95640 1 --2- 172.21.15.125:0/1839606595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70e80ca0e0 0x7f70e81e5880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.232+0000 7f70eeb95640 1 -- 172.21.15.125:0/1839606595 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70d80094d0 con 0x7f70e81538a0 2023-10-30T12:47:10.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.232+0000 7f70eeb95640 1 --2- 172.21.15.125:0/1839606595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e81538a0 0x7f70e81ef8b0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f70e000cd80 tx=0x7f70e000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:10.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.232+0000 7f70deffd640 1 -- 172.21.15.125:0/1839606595 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e000d840 con 0x7f70e81538a0 2023-10-30T12:47:10.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.232+0000 7f70deffd640 1 -- 172.21.15.125:0/1839606595 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e000d9a0 con 0x7f70e81538a0 2023-10-30T12:47:10.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.232+0000 7f70f05f8640 1 -- 172.21.15.125:0/1839606595 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70e80cd780 con 0x7f70e81538a0 2023-10-30T12:47:10.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.232+0000 7f70deffd640 1 -- 172.21.15.125:0/1839606595 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e000dd50 con 0x7f70e81538a0 2023-10-30T12:47:10.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.232+0000 7f70f05f8640 1 -- 172.21.15.125:0/1839606595 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70e81ef360 con 0x7f70e81538a0 2023-10-30T12:47:10.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.232+0000 7f70f05f8640 1 -- 172.21.15.125:0/1839606595 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70e80b2a50 con 0x7f70e81538a0 2023-10-30T12:47:10.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.232+0000 7f70deffd640 1 -- 172.21.15.125:0/1839606595 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f70e001e3b0 con 0x7f70e81538a0 2023-10-30T12:47:10.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.236+0000 7f70deffd640 1 --2- 172.21.15.125:0/1839606595 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70cc0664d0 0x7f70cc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.236+0000 7f70deffd640 1 -- 172.21.15.125:0/1839606595 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f70e0084a90 con 0x7f70e81538a0 2023-10-30T12:47:10.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.236+0000 7f70ee394640 1 --2- 172.21.15.125:0/1839606595 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70cc0664d0 0x7f70cc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.236+0000 7f70ee394640 1 --2- 172.21.15.125:0/1839606595 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70cc0664d0 0x7f70cc068990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f70e4006fd0 tx=0x7f70e4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:10.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.240+0000 7f70deffd640 1 -- 172.21.15.125:0/1839606595 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f70e0051c60 con 0x7f70e81538a0 2023-10-30T12:47:10.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.440+0000 7f70f05f8640 1 -- 172.21.15.125:0/1839606595 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70e804e170 con 0x7f70e81538a0 2023-10-30T12:47:10.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.440+0000 7f70deffd640 1 -- 172.21.15.125:0/1839606595 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f70e001a070 con 0x7f70e81538a0 2023-10-30T12:47:10.441 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:10.442 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6318,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:55.354589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:10.442 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 91 2023-10-30T12:47:10.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.440+0000 7f70f05f8640 1 -- 172.21.15.125:0/1839606595 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70cc0664d0 msgr2=0x7f70cc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:10.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.440+0000 7f70f05f8640 1 --2- 172.21.15.125:0/1839606595 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70cc0664d0 0x7f70cc068990 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f70e4006fd0 tx=0x7f70e4008040 comp rx=0 tx=0).stop 2023-10-30T12:47:10.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.440+0000 7f70f05f8640 1 -- 172.21.15.125:0/1839606595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e81538a0 msgr2=0x7f70e81ef8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:10.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.440+0000 7f70f05f8640 1 --2- 172.21.15.125:0/1839606595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e81538a0 0x7f70e81ef8b0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f70e000cd80 tx=0x7f70e000a710 comp rx=0 tx=0).stop 2023-10-30T12:47:10.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.440+0000 7f70f05f8640 1 -- 172.21.15.125:0/1839606595 shutdown_connections 2023-10-30T12:47:10.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.440+0000 7f70f05f8640 1 --2- 172.21.15.125:0/1839606595 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70cc0664d0 0x7f70cc068990 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.440+0000 7f70f05f8640 1 --2- 172.21.15.125:0/1839606595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70e81538a0 0x7f70e81ef8b0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.440+0000 7f70f05f8640 1 --2- 172.21.15.125:0/1839606595 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70e80cf160 0x7f70e81e3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.440+0000 7f70f05f8640 1 --2- 172.21.15.125:0/1839606595 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70e80ca0e0 0x7f70e81e5880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.440+0000 7f70f05f8640 1 -- 172.21.15.125:0/1839606595 >> 172.21.15.125:0/1839606595 conn(0x7f70e806bfe0 msgr2=0x7f70e8151770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:10.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.440+0000 7f70f05f8640 1 -- 172.21.15.125:0/1839606595 shutdown_connections 2023-10-30T12:47:10.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.440+0000 7f70f05f8640 1 -- 172.21.15.125:0/1839606595 wait complete. 2023-10-30T12:47:10.462 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2023-10-30T12:47:10.463 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T12:47:10.463 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:47:10.479 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T12:47:10.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:10.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.616+0000 7f4988901640 1 Processor -- start 2023-10-30T12:47:10.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.616+0000 7f4988901640 1 -- start start 2023-10-30T12:47:10.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.616+0000 7f4988901640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4984069e90 0x7f498415a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.616+0000 7f4988901640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f498406cb80 0x7f498406d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.616+0000 7f4988901640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f498406d550 con 0x7f498406a270 2023-10-30T12:47:10.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.616+0000 7f4988901640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49840c8570 con 0x7f4984069e90 2023-10-30T12:47:10.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.616+0000 7f4988901640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49840c6c10 con 0x7f498406cb80 2023-10-30T12:47:10.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.616+0000 7f4981d9b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f498406a270 0x7f49840c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33800/0 (socket says 172.21.15.125:33800) 2023-10-30T12:47:10.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.616+0000 7f4981d9b640 1 -- 172.21.15.125:0/1180545684 learned_addr learned my addr 172.21.15.125:0/1180545684 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:47:10.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.616+0000 7f4982d9d640 1 --2- 172.21.15.125:0/1180545684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f498406cb80 0x7f498406d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.616+0000 7f498259c640 1 --2- 172.21.15.125:0/1180545684 >> v2:172.21.15.136:3300/0 conn(0x7f4984069e90 0x7f498415a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 -- 172.21.15.125:0/1180545684 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392958949 0 0) 0x7f498406d550 con 0x7f498406a270 2023-10-30T12:47:10.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 -- 172.21.15.125:0/1180545684 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f496c003530 con 0x7f498406a270 2023-10-30T12:47:10.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 -- 172.21.15.125:0/1180545684 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1597017085 0 0) 0x7f496c003530 con 0x7f498406a270 2023-10-30T12:47:10.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 -- 172.21.15.125:0/1180545684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f498406cb80 msgr2=0x7f498406d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:10.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 --2- 172.21.15.125:0/1180545684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f498406cb80 0x7f498406d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 -- 172.21.15.125:0/1180545684 >> v2:172.21.15.136:3300/0 conn(0x7f4984069e90 msgr2=0x7f498415a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:10.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 --2- 172.21.15.125:0/1180545684 >> v2:172.21.15.136:3300/0 conn(0x7f4984069e90 0x7f498415a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 -- 172.21.15.125:0/1180545684 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49840c6ef0 con 0x7f498406a270 2023-10-30T12:47:10.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 -- 172.21.15.125:0/1180545684 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f4978001680 con 0x7f498406a270 2023-10-30T12:47:10.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 -- 172.21.15.125:0/1180545684 >> v1:172.21.15.125:6789/0 conn(0x7f498406a270 legacy=0x7f49840c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:10.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 --2- 172.21.15.125:0/1180545684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f496c003980 0x7f496c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 --2- 172.21.15.125:0/1180545684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f496c0047d0 0x7f496c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 --2- 172.21.15.125:0/1180545684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f498406cb80 0x7f496c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 -- 172.21.15.125:0/1180545684 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49840c8570 con 0x7f496c003980 2023-10-30T12:47:10.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 -- 172.21.15.125:0/1180545684 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49840c6c10 con 0x7f496c0047d0 2023-10-30T12:47:10.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498159a640 1 -- 172.21.15.125:0/1180545684 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f496c00b990 con 0x7f498406cb80 2023-10-30T12:47:10.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498259c640 1 --2- 172.21.15.125:0/1180545684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f496c003980 0x7f496c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f4981d9b640 1 --2- 172.21.15.125:0/1180545684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f496c0047d0 0x7f496c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f4982d9d640 1 --2- 172.21.15.125:0/1180545684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f498406cb80 0x7f496c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498259c640 1 -- 172.21.15.125:0/1180545684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f498406cb80 msgr2=0x7f496c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:10.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498259c640 1 --2- 172.21.15.125:0/1180545684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f498406cb80 0x7f496c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498259c640 1 -- 172.21.15.125:0/1180545684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f496c0047d0 msgr2=0x7f496c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:10.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498259c640 1 --2- 172.21.15.125:0/1180545684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f496c0047d0 0x7f496c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498259c640 1 -- 172.21.15.125:0/1180545684 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f496c003530 con 0x7f496c003980 2023-10-30T12:47:10.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.620+0000 7f498259c640 1 --2- 172.21.15.125:0/1180545684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f496c003980 0x7f496c003da0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f4974009ae0 tx=0x7f497401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=147fe947e1599d7e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:10.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.624+0000 7f498159a640 1 -- 172.21.15.125:0/1180545684 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49740b0400 con 0x7f496c003980 2023-10-30T12:47:10.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.624+0000 7f498159a640 1 -- 172.21.15.125:0/1180545684 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49740b0560 con 0x7f496c003980 2023-10-30T12:47:10.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.624+0000 7f498159a640 1 -- 172.21.15.125:0/1180545684 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49740b0910 con 0x7f496c003980 2023-10-30T12:47:10.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.624+0000 7f4988901640 1 -- 172.21.15.125:0/1180545684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f496c003980 msgr2=0x7f496c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:10.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.624+0000 7f4988901640 1 --2- 172.21.15.125:0/1180545684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f496c003980 0x7f496c003da0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f4974009ae0 tx=0x7f497401ca90 comp rx=0 tx=0).stop 2023-10-30T12:47:10.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.624+0000 7f4982d9d640 1 -- 172.21.15.125:0/1180545684 reap_dead start 2023-10-30T12:47:10.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.624+0000 7f4988901640 1 -- 172.21.15.125:0/1180545684 shutdown_connections 2023-10-30T12:47:10.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.624+0000 7f4988901640 1 -- 172.21.15.125:0/1180545684 >> 172.21.15.125:0/1180545684 conn(0x7f498406dfd0 msgr2=0x7f498406e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:10.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.624+0000 7f4988901640 1 -- 172.21.15.125:0/1180545684 shutdown_connections 2023-10-30T12:47:10.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.624+0000 7f4988901640 1 -- 172.21.15.125:0/1180545684 wait complete. 2023-10-30T12:47:10.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.624+0000 7f4988901640 1 Processor -- start 2023-10-30T12:47:10.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.624+0000 7f4988901640 1 -- start start 2023-10-30T12:47:10.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.624+0000 7f4988901640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f496c003980 0x7f49841f59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.624+0000 7f4988901640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f498415ade0 0x7f49841f2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4988901640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f498414ef40 0x7f498414b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4988901640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f498415da00 con 0x7f496c003980 2023-10-30T12:47:10.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4988901640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4984151ae0 con 0x7f498415ade0 2023-10-30T12:47:10.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4988901640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49841f1a30 con 0x7f498414ef40 2023-10-30T12:47:10.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f498259c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f496c003980 0x7f49841f59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f498259c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f496c003980 0x7f49841f59b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50764/0 (socket says 172.21.15.125:50764) 2023-10-30T12:47:10.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f498259c640 1 -- 172.21.15.125:0/839975249 learned_addr learned my addr 172.21.15.125:0/839975249 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:10.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4981d9b640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f498415ade0 0x7f49841f2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4982d9d640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f498414ef40 0x7f498414b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f498259c640 1 -- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f498414ef40 msgr2=0x7f498414b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:10.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f498259c640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f498414ef40 0x7f498414b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f498259c640 1 -- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f498415ade0 msgr2=0x7f49841f2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:10.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f498259c640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f498415ade0 0x7f49841f2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f498259c640 1 -- 172.21.15.125:0/839975249 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49841f5ef0 con 0x7f496c003980 2023-10-30T12:47:10.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4981d9b640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f498415ade0 0x7f49841f2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:47:10.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4982d9d640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f498414ef40 0x7f498414b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:47:10.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f498259c640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f496c003980 0x7f49841f59b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f497401c8c0 tx=0x7f4974019830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:10.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4972ffd640 1 -- 172.21.15.125:0/839975249 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497400f000 con 0x7f496c003980 2023-10-30T12:47:10.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4972ffd640 1 -- 172.21.15.125:0/839975249 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49740b0cd0 con 0x7f496c003980 2023-10-30T12:47:10.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4988901640 1 -- 172.21.15.125:0/839975249 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49841f2980 con 0x7f496c003980 2023-10-30T12:47:10.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4972ffd640 1 -- 172.21.15.125:0/839975249 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4974004220 con 0x7f496c003980 2023-10-30T12:47:10.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4988901640 1 -- 172.21.15.125:0/839975249 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4984151520 con 0x7f496c003980 2023-10-30T12:47:10.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4988901640 1 -- 172.21.15.125:0/839975249 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f498404e170 con 0x7f496c003980 2023-10-30T12:47:10.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4972ffd640 1 -- 172.21.15.125:0/839975249 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4974004380 con 0x7f496c003980 2023-10-30T12:47:10.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.628+0000 7f4972ffd640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49640664d0 0x7f4964068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:10.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.632+0000 7f4972ffd640 1 -- 172.21.15.125:0/839975249 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f497411e1e0 con 0x7f496c003980 2023-10-30T12:47:10.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.632+0000 7f4981d9b640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49640664d0 0x7f4964068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:10.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.632+0000 7f4981d9b640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49640664d0 0x7f4964068990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4978000c10 tx=0x7f4978000f40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:10.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.632+0000 7f4972ffd640 1 -- 172.21.15.125:0/839975249 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f49740eb380 con 0x7f496c003980 2023-10-30T12:47:10.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.812+0000 7f4988901640 1 -- 172.21.15.125:0/839975249 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49841f5ef0 con 0x7f496c003980 2023-10-30T12:47:10.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.812+0000 7f4972ffd640 1 -- 172.21.15.125:0/839975249 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f49841f5ef0 con 0x7f496c003980 2023-10-30T12:47:10.815 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:10.815 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6318,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:55.354589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:10.816 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 91 2023-10-30T12:47:10.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.816+0000 7f4988901640 1 -- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49640664d0 msgr2=0x7f4964068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:10.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.816+0000 7f4988901640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49640664d0 0x7f4964068990 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4978000c10 tx=0x7f4978000f40 comp rx=0 tx=0).stop 2023-10-30T12:47:10.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.816+0000 7f4988901640 1 -- 172.21.15.125:0/839975249 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f496c003980 msgr2=0x7f49841f59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:10.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.816+0000 7f4988901640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f496c003980 0x7f49841f59b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f497401c8c0 tx=0x7f4974019830 comp rx=0 tx=0).stop 2023-10-30T12:47:10.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.816+0000 7f4988901640 1 -- 172.21.15.125:0/839975249 shutdown_connections 2023-10-30T12:47:10.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.816+0000 7f4988901640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49640664d0 0x7f4964068990 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.816+0000 7f4988901640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f498414ef40 0x7f498414b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.816+0000 7f4988901640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f498415ade0 0x7f49841f2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.816+0000 7f4988901640 1 --2- 172.21.15.125:0/839975249 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f496c003980 0x7f49841f59b0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:10.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.816+0000 7f4988901640 1 -- 172.21.15.125:0/839975249 >> 172.21.15.125:0/839975249 conn(0x7f498406dfd0 msgr2=0x7f498406b410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:10.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.816+0000 7f4988901640 1 -- 172.21.15.125:0/839975249 shutdown_connections 2023-10-30T12:47:10.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:10.816+0000 7f4988901640 1 -- 172.21.15.125:0/839975249 wait complete. 2023-10-30T12:47:10.835 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:47:12.836 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:12.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.980+0000 7fb2b1854640 1 Processor -- start 2023-10-30T12:47:12.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.980+0000 7fb2b1854640 1 -- start start 2023-10-30T12:47:12.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.980+0000 7fb2b1854640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb2ac1538a0 0x7fb2ac15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:12.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.980+0000 7fb2b1854640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2ac0cf160 0x7fb2ac0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:12.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.980+0000 7fb2b1854640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ac0cb800 con 0x7fb2ac0ca0e0 2023-10-30T12:47:12.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.980+0000 7fb2b1854640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2ac15dc30 con 0x7fb2ac1538a0 2023-10-30T12:47:12.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.980+0000 7fb2b1854640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac0cb330 con 0x7fb2ac0cf160 2023-10-30T12:47:12.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.980+0000 7fb2a27fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb2ac0ca0e0 0x7fb2ac15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54732/0 (socket says 172.21.15.125:54732) 2023-10-30T12:47:12.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.980+0000 7fb2a27fc640 1 -- 172.21.15.125:0/57838203 learned_addr learned my addr 172.21.15.125:0/57838203 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:47:12.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.980+0000 7fb2aaffd640 1 --2- 172.21.15.125:0/57838203 >> v2:172.21.15.136:3300/0 conn(0x7fb2ac1538a0 0x7fb2ac15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:12.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.980+0000 7fb2ab7fe640 1 --2- 172.21.15.125:0/57838203 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2ac0cf160 0x7fb2ac0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:12.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.980+0000 7fb2aa7fc640 1 -- 172.21.15.125:0/57838203 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546065130 0 0) 0x7fb2ac0cb800 con 0x7fb2ac0ca0e0 2023-10-30T12:47:12.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.980+0000 7fb2aa7fc640 1 -- 172.21.15.125:0/57838203 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb298003530 con 0x7fb2ac0ca0e0 2023-10-30T12:47:12.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.984+0000 7fb2ab7fe640 1 -- 172.21.15.125:0/57838203 >> v2:172.21.15.136:3300/0 conn(0x7fb2ac1538a0 msgr2=0x7fb2ac15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:12.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.984+0000 7fb2ab7fe640 1 --2- 172.21.15.125:0/57838203 >> v2:172.21.15.136:3300/0 conn(0x7fb2ac1538a0 0x7fb2ac15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:12.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.984+0000 7fb2ab7fe640 1 -- 172.21.15.125:0/57838203 >> v1:172.21.15.125:6789/0 conn(0x7fb2ac0ca0e0 legacy=0x7fb2ac15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:12.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.984+0000 7fb2aa7fc640 1 -- 172.21.15.125:0/57838203 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3163092651 0 0) 0x7fb298003530 con 0x7fb2ac0ca0e0 2023-10-30T12:47:12.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.984+0000 7fb2ab7fe640 1 -- 172.21.15.125:0/57838203 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ac0cad70 con 0x7fb2ac0cf160 2023-10-30T12:47:12.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.984+0000 7fb2aaffd640 1 --2- 172.21.15.125:0/57838203 >> v2:172.21.15.136:3300/0 conn(0x7fb2ac1538a0 0x7fb2ac15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:47:12.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.984+0000 7fb2ab7fe640 1 --2- 172.21.15.125:0/57838203 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2ac0cf160 0x7fb2ac0cf590 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009d90 tx=0x7fb2a401f060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=252e6b1d803bb877 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:12.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.984+0000 7fb2aa7fc640 1 -- 172.21.15.125:0/57838203 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294001680 con 0x7fb2ac0cf160 2023-10-30T12:47:12.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.984+0000 7fb2aa7fc640 1 -- 172.21.15.125:0/57838203 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2a40a8bb0 con 0x7fb2ac0cf160 2023-10-30T12:47:12.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.984+0000 7fb2aa7fc640 1 -- 172.21.15.125:0/57838203 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a40b0440 con 0x7fb2ac0cf160 2023-10-30T12:47:12.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 -- 172.21.15.125:0/57838203 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2ac0cf160 msgr2=0x7fb2ac0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:12.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 --2- 172.21.15.125:0/57838203 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2ac0cf160 0x7fb2ac0cf590 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb2a4009d90 tx=0x7fb2a401f060 comp rx=0 tx=0).stop 2023-10-30T12:47:12.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 -- 172.21.15.125:0/57838203 shutdown_connections 2023-10-30T12:47:12.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 --2- 172.21.15.125:0/57838203 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2ac0cf160 0x7fb2ac0cf590 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:12.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 --2- 172.21.15.125:0/57838203 >> v2:172.21.15.136:3300/0 conn(0x7fb2ac1538a0 0x7fb2ac15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:12.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 -- 172.21.15.125:0/57838203 >> 172.21.15.125:0/57838203 conn(0x7fb2ac06bfe0 msgr2=0x7fb2ac06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:12.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 -- 172.21.15.125:0/57838203 shutdown_connections 2023-10-30T12:47:12.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 -- 172.21.15.125:0/57838203 wait complete. 2023-10-30T12:47:12.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 Processor -- start 2023-10-30T12:47:12.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 -- start start 2023-10-30T12:47:12.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2ac0ca0e0 0x7fb2ac14b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:12.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2ac0cf160 0x7fb2ac147f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:12.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2ac1538a0 0x7fb2ac1f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:12.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac0c9490 con 0x7fb2ac0cf160 2023-10-30T12:47:12.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac14ba50 con 0x7fb2ac1538a0 2023-10-30T12:47:12.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2b1854640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac0c8cf0 con 0x7fb2ac0ca0e0 2023-10-30T12:47:12.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2a27fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2ac0cf160 0x7fb2ac147f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:12.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2a27fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2ac0cf160 0x7fb2ac147f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38196/0 (socket says 172.21.15.125:38196) 2023-10-30T12:47:12.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2a27fc640 1 -- 172.21.15.125:0/552743856 learned_addr learned my addr 172.21.15.125:0/552743856 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:12.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2ab7fe640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2ac1538a0 0x7fb2ac1f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:12.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2aaffd640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2ac0ca0e0 0x7fb2ac14b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:12.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2a27fc640 1 -- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2ac0ca0e0 msgr2=0x7fb2ac14b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:12.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2a27fc640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2ac0ca0e0 0x7fb2ac14b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:12.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2a27fc640 1 -- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2ac1538a0 msgr2=0x7fb2ac1f2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:12.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2a27fc640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2ac1538a0 0x7fb2ac1f2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:12.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2a27fc640 1 -- 172.21.15.125:0/552743856 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c009420 con 0x7fb2ac0cf160 2023-10-30T12:47:12.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2ab7fe640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2ac1538a0 0x7fb2ac1f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:47:12.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.988+0000 7fb2aaffd640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2ac0ca0e0 0x7fb2ac14b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:47:12.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.992+0000 7fb2a27fc640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2ac0cf160 0x7fb2ac147f30 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb29400c750 tx=0x7fb29400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:12.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.992+0000 7fb2a8ff9640 1 -- 172.21.15.125:0/552743856 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294005ae0 con 0x7fb2ac0cf160 2023-10-30T12:47:12.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.992+0000 7fb2a8ff9640 1 -- 172.21.15.125:0/552743856 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb294005c40 con 0x7fb2ac0cf160 2023-10-30T12:47:12.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.992+0000 7fb2b1854640 1 -- 172.21.15.125:0/552743856 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a4009a70 con 0x7fb2ac0cf160 2023-10-30T12:47:12.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.992+0000 7fb2a8ff9640 1 -- 172.21.15.125:0/552743856 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb294007070 con 0x7fb2ac0cf160 2023-10-30T12:47:12.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.992+0000 7fb2b1854640 1 -- 172.21.15.125:0/552743856 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ac0c8860 con 0x7fb2ac0cf160 2023-10-30T12:47:12.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.992+0000 7fb2b1854640 1 -- 172.21.15.125:0/552743856 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2ac0b2a50 con 0x7fb2ac0cf160 2023-10-30T12:47:13.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.992+0000 7fb2a8ff9640 1 -- 172.21.15.125:0/552743856 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb2940071d0 con 0x7fb2ac0cf160 2023-10-30T12:47:13.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.992+0000 7fb2a8ff9640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb28c0664a0 0x7fb28c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:13.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.992+0000 7fb2a8ff9640 1 -- 172.21.15.125:0/552743856 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7fb2940837d0 con 0x7fb2ac0cf160 2023-10-30T12:47:13.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.996+0000 7fb2aaffd640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb28c0664a0 0x7fb28c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:13.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.996+0000 7fb2aaffd640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb28c0664a0 0x7fb28c068960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009bd0 tx=0x7fb29c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:13.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:12.996+0000 7fb2a8ff9640 1 -- 172.21.15.125:0/552743856 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb2940509f0 con 0x7fb2ac0cf160 2023-10-30T12:47:13.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:13.180+0000 7fb2b1854640 1 -- 172.21.15.125:0/552743856 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2ac04e170 con 0x7fb2ac0cf160 2023-10-30T12:47:13.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:13.180+0000 7fb2a8ff9640 1 -- 172.21.15.125:0/552743856 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fb294054010 con 0x7fb2ac0cf160 2023-10-30T12:47:13.182 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:13.182 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6318,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:55.354589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:13.183 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 91 2023-10-30T12:47:13.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:13.180+0000 7fb2b1854640 1 -- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb28c0664a0 msgr2=0x7fb28c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:13.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:13.180+0000 7fb2b1854640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb28c0664a0 0x7fb28c068960 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009bd0 tx=0x7fb29c008040 comp rx=0 tx=0).stop 2023-10-30T12:47:13.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:13.180+0000 7fb2b1854640 1 -- 172.21.15.125:0/552743856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2ac0cf160 msgr2=0x7fb2ac147f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:13.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:13.180+0000 7fb2b1854640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2ac0cf160 0x7fb2ac147f30 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fb29400c750 tx=0x7fb29400cc10 comp rx=0 tx=0).stop 2023-10-30T12:47:13.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:13.180+0000 7fb2b1854640 1 -- 172.21.15.125:0/552743856 shutdown_connections 2023-10-30T12:47:13.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:13.180+0000 7fb2b1854640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb28c0664a0 0x7fb28c068960 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:13.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:13.180+0000 7fb2b1854640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2ac1538a0 0x7fb2ac1f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:13.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:13.180+0000 7fb2b1854640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2ac0cf160 0x7fb2ac147f30 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:13.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:13.180+0000 7fb2b1854640 1 --2- 172.21.15.125:0/552743856 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2ac0ca0e0 0x7fb2ac14b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:13.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:13.180+0000 7fb2b1854640 1 -- 172.21.15.125:0/552743856 >> 172.21.15.125:0/552743856 conn(0x7fb2ac06bfe0 msgr2=0x7fb2ac066310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:13.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:13.180+0000 7fb2b1854640 1 -- 172.21.15.125:0/552743856 shutdown_connections 2023-10-30T12:47:13.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:13.180+0000 7fb2b1854640 1 -- 172.21.15.125:0/552743856 wait complete. 2023-10-30T12:47:13.202 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:47:15.204 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:15.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.348+0000 7fbf98172640 1 Processor -- start 2023-10-30T12:47:15.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.348+0000 7fbf98172640 1 -- start start 2023-10-30T12:47:15.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf98172640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf90153bf0 0x7fbf90071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:15.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf98172640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbf900c5f60 0x7fbf900cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:15.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf98172640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf900700b0 con 0x7fbf900c6fe0 2023-10-30T12:47:15.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf98172640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf900c6810 con 0x7fbf900c5f60 2023-10-30T12:47:15.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf98172640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf90159900 con 0x7fbf90153bf0 2023-10-30T12:47:15.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf95f0e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf90153bf0 0x7fbf90071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf9670f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbf900c5f60 0x7fbf900cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf95f0e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf90153bf0 0x7fbf90071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:51170/0 (socket says 172.21.15.125:51170) 2023-10-30T12:47:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf9570d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fbf900c6fe0 0x7fbf900c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:54304/0 (socket says 172.21.15.125:54304) 2023-10-30T12:47:15.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf95f0e640 1 -- 172.21.15.125:0/2098840646 learned_addr learned my addr 172.21.15.125:0/2098840646 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf95f0e640 1 -- 172.21.15.125:0/2098840646 >> v2:172.21.15.136:3300/0 conn(0x7fbf900c5f60 msgr2=0x7fbf900cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf95f0e640 1 --2- 172.21.15.125:0/2098840646 >> v2:172.21.15.136:3300/0 conn(0x7fbf900c5f60 0x7fbf900cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:15.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf95f0e640 1 -- 172.21.15.125:0/2098840646 >> v1:172.21.15.136:6789/0 conn(0x7fbf900c6fe0 legacy=0x7fbf900c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:15.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf95f0e640 1 -- 172.21.15.125:0/2098840646 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf900c6530 con 0x7fbf90153bf0 2023-10-30T12:47:15.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf9670f640 1 --2- 172.21.15.125:0/2098840646 >> v2:172.21.15.136:3300/0 conn(0x7fbf900c5f60 0x7fbf900cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:47:15.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.352+0000 7fbf94f0c640 1 -- 172.21.15.125:0/2098840646 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24750440 0 0) 0x7fbf900700b0 con 0x7fbf900c6fe0 2023-10-30T12:47:15.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.356+0000 7fbf95f0e640 1 --2- 172.21.15.125:0/2098840646 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf90153bf0 0x7fbf90071260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009870 tx=0x7fbf8c01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eae5abc46d4cf762 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:15.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.356+0000 7fbf94f0c640 1 -- 172.21.15.125:0/2098840646 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0a8a30 con 0x7fbf90153bf0 2023-10-30T12:47:15.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.356+0000 7fbf94f0c640 1 -- 172.21.15.125:0/2098840646 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8c0a8b90 con 0x7fbf90153bf0 2023-10-30T12:47:15.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.356+0000 7fbf94f0c640 1 -- 172.21.15.125:0/2098840646 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf8c0b0440 con 0x7fbf90153bf0 2023-10-30T12:47:15.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.356+0000 7fbf98172640 1 -- 172.21.15.125:0/2098840646 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf90153bf0 msgr2=0x7fbf90071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:15.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.356+0000 7fbf98172640 1 --2- 172.21.15.125:0/2098840646 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf90153bf0 0x7fbf90071260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009870 tx=0x7fbf8c01ed30 comp rx=0 tx=0).stop 2023-10-30T12:47:15.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.356+0000 7fbf98172640 1 -- 172.21.15.125:0/2098840646 shutdown_connections 2023-10-30T12:47:15.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.356+0000 7fbf98172640 1 --2- 172.21.15.125:0/2098840646 >> v2:172.21.15.136:3300/0 conn(0x7fbf900c5f60 0x7fbf900cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:15.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.356+0000 7fbf98172640 1 --2- 172.21.15.125:0/2098840646 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf90153bf0 0x7fbf90071260 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:15.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.356+0000 7fbf98172640 1 -- 172.21.15.125:0/2098840646 >> 172.21.15.125:0/2098840646 conn(0x7fbf9006c6e0 msgr2=0x7fbf90069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:15.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.356+0000 7fbf98172640 1 -- 172.21.15.125:0/2098840646 shutdown_connections 2023-10-30T12:47:15.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.356+0000 7fbf98172640 1 -- 172.21.15.125:0/2098840646 wait complete. 2023-10-30T12:47:15.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf98172640 1 Processor -- start 2023-10-30T12:47:15.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf98172640 1 -- start start 2023-10-30T12:47:15.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf98172640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf900c5f60 0x7fbf90150740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:15.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf98172640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf900c6fe0 0x7fbf9014cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:15.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf98172640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf90153bf0 0x7fbf9014dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:15.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf98172640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf901eefb0 con 0x7fbf90153bf0 2023-10-30T12:47:15.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf98172640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9014d1e0 con 0x7fbf900c6fe0 2023-10-30T12:47:15.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf98172640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf901eff80 con 0x7fbf900c5f60 2023-10-30T12:47:15.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf9670f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf90153bf0 0x7fbf9014dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:15.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf9670f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf90153bf0 0x7fbf9014dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38208/0 (socket says 172.21.15.125:38208) 2023-10-30T12:47:15.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf9670f640 1 -- 172.21.15.125:0/2149219797 learned_addr learned my addr 172.21.15.125:0/2149219797 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:15.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf9570d640 1 --2- 172.21.15.125:0/2149219797 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf900c6fe0 0x7fbf9014cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:15.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf95f0e640 1 --2- 172.21.15.125:0/2149219797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf900c5f60 0x7fbf90150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:15.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf9670f640 1 -- 172.21.15.125:0/2149219797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf900c5f60 msgr2=0x7fbf90150740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:15.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf9670f640 1 --2- 172.21.15.125:0/2149219797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf900c5f60 0x7fbf90150740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:15.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf9670f640 1 -- 172.21.15.125:0/2149219797 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf900c6fe0 msgr2=0x7fbf9014cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:15.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf9670f640 1 --2- 172.21.15.125:0/2149219797 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf900c6fe0 0x7fbf9014cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:15.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf9670f640 1 -- 172.21.15.125:0/2149219797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf88009a70 con 0x7fbf90153bf0 2023-10-30T12:47:15.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf9670f640 1 --2- 172.21.15.125:0/2149219797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf90153bf0 0x7fbf9014dea0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fbf88009f00 tx=0x7fbf8800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:15.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf867fc640 1 -- 172.21.15.125:0/2149219797 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf88009d70 con 0x7fbf90153bf0 2023-10-30T12:47:15.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf867fc640 1 -- 172.21.15.125:0/2149219797 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf8800f040 con 0x7fbf90153bf0 2023-10-30T12:47:15.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf867fc640 1 -- 172.21.15.125:0/2149219797 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf88003bc0 con 0x7fbf90153bf0 2023-10-30T12:47:15.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf98172640 1 -- 172.21.15.125:0/2149219797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf8c0094d0 con 0x7fbf90153bf0 2023-10-30T12:47:15.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.360+0000 7fbf98172640 1 -- 172.21.15.125:0/2149219797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf901510b0 con 0x7fbf90153bf0 2023-10-30T12:47:15.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.364+0000 7fbf98172640 1 -- 172.21.15.125:0/2149219797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf900673b0 con 0x7fbf90153bf0 2023-10-30T12:47:15.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.364+0000 7fbf867fc640 1 -- 172.21.15.125:0/2149219797 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbf88014480 con 0x7fbf90153bf0 2023-10-30T12:47:15.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.364+0000 7fbf867fc640 1 --2- 172.21.15.125:0/2149219797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbf740664a0 0x7fbf74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:15.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.364+0000 7fbf867fc640 1 -- 172.21.15.125:0/2149219797 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7fbf88084ac0 con 0x7fbf90153bf0 2023-10-30T12:47:15.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.364+0000 7fbf95f0e640 1 --2- 172.21.15.125:0/2149219797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbf740664a0 0x7fbf74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:15.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.364+0000 7fbf95f0e640 1 --2- 172.21.15.125:0/2149219797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbf740664a0 0x7fbf74068960 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0b2000 tx=0x7fbf8c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:15.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.368+0000 7fbf867fc640 1 -- 172.21.15.125:0/2149219797 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbf88051d20 con 0x7fbf90153bf0 2023-10-30T12:47:15.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.552+0000 7fbf98172640 1 -- 172.21.15.125:0/2149219797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf901510b0 con 0x7fbf90153bf0 2023-10-30T12:47:15.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.552+0000 7fbf867fc640 1 -- 172.21.15.125:0/2149219797 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fbf901510b0 con 0x7fbf90153bf0 2023-10-30T12:47:15.555 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:15.556 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6318,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:55.354589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:15.556 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 91 2023-10-30T12:47:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.556+0000 7fbf98172640 1 -- 172.21.15.125:0/2149219797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbf740664a0 msgr2=0x7fbf74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.556+0000 7fbf98172640 1 --2- 172.21.15.125:0/2149219797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbf740664a0 0x7fbf74068960 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c0b2000 tx=0x7fbf8c009040 comp rx=0 tx=0).stop 2023-10-30T12:47:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.556+0000 7fbf98172640 1 -- 172.21.15.125:0/2149219797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf90153bf0 msgr2=0x7fbf9014dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.556+0000 7fbf98172640 1 --2- 172.21.15.125:0/2149219797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf90153bf0 0x7fbf9014dea0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fbf88009f00 tx=0x7fbf8800c490 comp rx=0 tx=0).stop 2023-10-30T12:47:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.556+0000 7fbf98172640 1 -- 172.21.15.125:0/2149219797 shutdown_connections 2023-10-30T12:47:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.556+0000 7fbf98172640 1 --2- 172.21.15.125:0/2149219797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbf740664a0 0x7fbf74068960 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:15.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.556+0000 7fbf98172640 1 --2- 172.21.15.125:0/2149219797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbf90153bf0 0x7fbf9014dea0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:15.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.556+0000 7fbf98172640 1 --2- 172.21.15.125:0/2149219797 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbf900c6fe0 0x7fbf9014cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:15.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.556+0000 7fbf98172640 1 --2- 172.21.15.125:0/2149219797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbf900c5f60 0x7fbf90150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:15.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.556+0000 7fbf98172640 1 -- 172.21.15.125:0/2149219797 >> 172.21.15.125:0/2149219797 conn(0x7fbf9006c6e0 msgr2=0x7fbf90071160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:15.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.556+0000 7fbf98172640 1 -- 172.21.15.125:0/2149219797 shutdown_connections 2023-10-30T12:47:15.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:15.556+0000 7fbf98172640 1 -- 172.21.15.125:0/2149219797 wait complete. 2023-10-30T12:47:15.576 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:47:16.888 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:47:16.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:47:17.577 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:17.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.720+0000 7f98a08c9640 1 Processor -- start 2023-10-30T12:47:17.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.720+0000 7f98a08c9640 1 -- start start 2023-10-30T12:47:17.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f98a08c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f989c0c6b80 0x7f989c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:17.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f98a08c9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f989c13f4a0 0x7f989c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:17.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f98a08c9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f989c0c0a50 con 0x7f989c0c6f60 2023-10-30T12:47:17.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f98a08c9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f989c06f6e0 con 0x7f989c13f4a0 2023-10-30T12:47:17.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f98a08c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f989c06f9c0 con 0x7f989c0c6b80 2023-10-30T12:47:17.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9891d9b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f989c0c6f60 0x7f989c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54736/0 (socket says 172.21.15.125:54736) 2023-10-30T12:47:17.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9891d9b640 1 -- 172.21.15.125:0/3120126485 learned_addr learned my addr 172.21.15.125:0/3120126485 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:47:17.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f989ad9d640 1 --2- 172.21.15.125:0/3120126485 >> v2:172.21.15.136:3300/0 conn(0x7f989c13f4a0 0x7f989c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:17.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f989a59c640 1 --2- 172.21.15.125:0/3120126485 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f989c0c6b80 0x7f989c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:17.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 -- 172.21.15.125:0/3120126485 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435869568 0 0) 0x7f989c0c0a50 con 0x7f989c0c6f60 2023-10-30T12:47:17.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 -- 172.21.15.125:0/3120126485 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9888005740 con 0x7f989c0c6f60 2023-10-30T12:47:17.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 -- 172.21.15.125:0/3120126485 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 989768104 0 0) 0x7f9888005740 con 0x7f989c0c6f60 2023-10-30T12:47:17.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 -- 172.21.15.125:0/3120126485 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f989c0c6b80 msgr2=0x7f989c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:17.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 --2- 172.21.15.125:0/3120126485 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f989c0c6b80 0x7f989c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:17.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 -- 172.21.15.125:0/3120126485 >> v2:172.21.15.136:3300/0 conn(0x7f989c13f4a0 msgr2=0x7f989c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:17.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 --2- 172.21.15.125:0/3120126485 >> v2:172.21.15.136:3300/0 conn(0x7f989c13f4a0 0x7f989c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:17.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 -- 172.21.15.125:0/3120126485 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f989c06fca0 con 0x7f989c0c6f60 2023-10-30T12:47:17.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 -- 172.21.15.125:0/3120126485 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f9884001680 con 0x7f989c0c6f60 2023-10-30T12:47:17.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 -- 172.21.15.125:0/3120126485 >> v1:172.21.15.125:6789/0 conn(0x7f989c0c6f60 legacy=0x7f989c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:17.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 --2- 172.21.15.125:0/3120126485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9888005b90 0x7f9888005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:17.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 --2- 172.21.15.125:0/3120126485 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98880069e0 0x7f9888006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:17.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 --2- 172.21.15.125:0/3120126485 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f989c0c6b80 0x7f98880073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:17.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 -- 172.21.15.125:0/3120126485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c06f6e0 con 0x7f98880069e0 2023-10-30T12:47:17.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 -- 172.21.15.125:0/3120126485 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c06f9c0 con 0x7f9888005b90 2023-10-30T12:47:17.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9899d9b640 1 -- 172.21.15.125:0/3120126485 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988800dba0 con 0x7f989c0c6b80 2023-10-30T12:47:17.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9891d9b640 1 --2- 172.21.15.125:0/3120126485 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98880069e0 0x7f9888006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:17.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f989ad9d640 1 --2- 172.21.15.125:0/3120126485 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f989c0c6b80 0x7f98880073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:17.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f989a59c640 1 --2- 172.21.15.125:0/3120126485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9888005b90 0x7f9888005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:17.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9891d9b640 1 -- 172.21.15.125:0/3120126485 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f989c0c6b80 msgr2=0x7f98880073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:17.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9891d9b640 1 --2- 172.21.15.125:0/3120126485 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f989c0c6b80 0x7f98880073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:17.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9891d9b640 1 -- 172.21.15.125:0/3120126485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9888005b90 msgr2=0x7f9888005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:17.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9891d9b640 1 --2- 172.21.15.125:0/3120126485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9888005b90 0x7f9888005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:17.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f9891d9b640 1 -- 172.21.15.125:0/3120126485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9888005740 con 0x7f98880069e0 2023-10-30T12:47:17.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.724+0000 7f989a59c640 1 --2- 172.21.15.125:0/3120126485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9888005b90 0x7f9888005fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:17.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.728+0000 7f9891d9b640 1 --2- 172.21.15.125:0/3120126485 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98880069e0 0x7f9888006e60 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f988400c210 tx=0x7f988401f390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db9d8eff1bd4e95 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:17.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.728+0000 7f9899d9b640 1 -- 172.21.15.125:0/3120126485 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98840a8cb0 con 0x7f98880069e0 2023-10-30T12:47:17.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.728+0000 7f9899d9b640 1 -- 172.21.15.125:0/3120126485 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98840a8e10 con 0x7f98880069e0 2023-10-30T12:47:17.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.728+0000 7f9899d9b640 1 -- 172.21.15.125:0/3120126485 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9884005b00 con 0x7f98880069e0 2023-10-30T12:47:17.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.728+0000 7f98a08c9640 1 -- 172.21.15.125:0/3120126485 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98880069e0 msgr2=0x7f9888006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:17.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.728+0000 7f98a08c9640 1 --2- 172.21.15.125:0/3120126485 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98880069e0 0x7f9888006e60 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f988400c210 tx=0x7f988401f390 comp rx=0 tx=0).stop 2023-10-30T12:47:17.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.728+0000 7f989ad9d640 1 -- 172.21.15.125:0/3120126485 reap_dead start 2023-10-30T12:47:17.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.728+0000 7f98a08c9640 1 -- 172.21.15.125:0/3120126485 shutdown_connections 2023-10-30T12:47:17.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.728+0000 7f98a08c9640 1 -- 172.21.15.125:0/3120126485 >> 172.21.15.125:0/3120126485 conn(0x7f989c0c7910 msgr2=0x7f989c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:17.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.728+0000 7f98a08c9640 1 -- 172.21.15.125:0/3120126485 shutdown_connections 2023-10-30T12:47:17.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.728+0000 7f98a08c9640 1 -- 172.21.15.125:0/3120126485 wait complete. 2023-10-30T12:47:17.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f98a08c9640 1 Processor -- start 2023-10-30T12:47:17.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f98a08c9640 1 -- start start 2023-10-30T12:47:17.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f98a08c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98880069e0 0x7f989c0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:17.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f98a08c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f989c153bd0 0x7f989c0c0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:17.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f98a08c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f989c1ef030 0x7f989c1ef4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:17.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f98a08c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c1ef9f0 con 0x7f989c1ef030 2023-10-30T12:47:17.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f98a08c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989c0af830 con 0x7f98880069e0 2023-10-30T12:47:17.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f98a08c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f989c0ae110 con 0x7f989c153bd0 2023-10-30T12:47:17.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f989ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f989c1ef030 0x7f989c1ef4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:17.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f989ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f989c1ef030 0x7f989c1ef4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38220/0 (socket says 172.21.15.125:38220) 2023-10-30T12:47:17.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f989ad9d640 1 -- 172.21.15.125:0/1451748273 learned_addr learned my addr 172.21.15.125:0/1451748273 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:17.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f989a59c640 1 --2- 172.21.15.125:0/1451748273 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98880069e0 0x7f989c0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:17.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f9891d9b640 1 --2- 172.21.15.125:0/1451748273 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f989c153bd0 0x7f989c0c0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:17.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f989ad9d640 1 -- 172.21.15.125:0/1451748273 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f989c153bd0 msgr2=0x7f989c0c0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:17.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f989ad9d640 1 --2- 172.21.15.125:0/1451748273 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f989c153bd0 0x7f989c0c0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:17.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f989ad9d640 1 -- 172.21.15.125:0/1451748273 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98880069e0 msgr2=0x7f989c0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:17.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f989ad9d640 1 --2- 172.21.15.125:0/1451748273 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98880069e0 0x7f989c0ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:17.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f989ad9d640 1 -- 172.21.15.125:0/1451748273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f989c0b0370 con 0x7f989c1ef030 2023-10-30T12:47:17.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f989ad9d640 1 --2- 172.21.15.125:0/1451748273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f989c1ef030 0x7f989c1ef4b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f989c13f8d0 tx=0x7f989400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:17.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f98937fe640 1 -- 172.21.15.125:0/1451748273 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989400e040 con 0x7f989c1ef030 2023-10-30T12:47:17.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f98937fe640 1 -- 172.21.15.125:0/1451748273 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989400f040 con 0x7f989c1ef030 2023-10-30T12:47:17.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f98a08c9640 1 -- 172.21.15.125:0/1451748273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f989c0afad0 con 0x7f989c1ef030 2023-10-30T12:47:17.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f98937fe640 1 -- 172.21.15.125:0/1451748273 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9894003be0 con 0x7f989c1ef030 2023-10-30T12:47:17.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f98a08c9640 1 -- 172.21.15.125:0/1451748273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f989c0b0090 con 0x7f989c1ef030 2023-10-30T12:47:17.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.732+0000 7f98a08c9640 1 -- 172.21.15.125:0/1451748273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f989c04e170 con 0x7f989c1ef030 2023-10-30T12:47:17.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.736+0000 7f98937fe640 1 -- 172.21.15.125:0/1451748273 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9894003d40 con 0x7f989c1ef030 2023-10-30T12:47:17.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.736+0000 7f98937fe640 1 --2- 172.21.15.125:0/1451748273 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f987c0664a0 0x7f987c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:17.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.736+0000 7f98937fe640 1 -- 172.21.15.125:0/1451748273 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f9894084a70 con 0x7f989c1ef030 2023-10-30T12:47:17.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.736+0000 7f989a59c640 1 --2- 172.21.15.125:0/1451748273 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f987c0664a0 0x7f987c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:17.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.736+0000 7f989a59c640 1 --2- 172.21.15.125:0/1451748273 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f987c0664a0 0x7f987c068960 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f988c0098d0 tx=0x7f988c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:17.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.740+0000 7f98937fe640 1 -- 172.21.15.125:0/1451748273 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9894051af0 con 0x7f989c1ef030 2023-10-30T12:47:17.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.916+0000 7f98a08c9640 1 -- 172.21.15.125:0/1451748273 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f989c1eece0 con 0x7f989c1ef030 2023-10-30T12:47:17.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.920+0000 7f98937fe640 1 -- 172.21.15.125:0/1451748273 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f9894055110 con 0x7f989c1ef030 2023-10-30T12:47:17.920 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:17.920 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6318,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:55.354589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:17.921 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 91 2023-10-30T12:47:17.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.920+0000 7f98a08c9640 1 -- 172.21.15.125:0/1451748273 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f987c0664a0 msgr2=0x7f987c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:17.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.920+0000 7f98a08c9640 1 --2- 172.21.15.125:0/1451748273 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f987c0664a0 0x7f987c068960 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f988c0098d0 tx=0x7f988c008040 comp rx=0 tx=0).stop 2023-10-30T12:47:17.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.920+0000 7f98a08c9640 1 -- 172.21.15.125:0/1451748273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f989c1ef030 msgr2=0x7f989c1ef4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:17.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.920+0000 7f98a08c9640 1 --2- 172.21.15.125:0/1451748273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f989c1ef030 0x7f989c1ef4b0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f989c13f8d0 tx=0x7f989400f580 comp rx=0 tx=0).stop 2023-10-30T12:47:17.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.920+0000 7f98a08c9640 1 -- 172.21.15.125:0/1451748273 shutdown_connections 2023-10-30T12:47:17.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.920+0000 7f98a08c9640 1 --2- 172.21.15.125:0/1451748273 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f987c0664a0 0x7f987c068960 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:17.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.920+0000 7f98a08c9640 1 --2- 172.21.15.125:0/1451748273 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f989c1ef030 0x7f989c1ef4b0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:17.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.920+0000 7f98a08c9640 1 --2- 172.21.15.125:0/1451748273 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f989c153bd0 0x7f989c0c0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:17.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.920+0000 7f98a08c9640 1 --2- 172.21.15.125:0/1451748273 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98880069e0 0x7f989c0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:17.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.920+0000 7f98a08c9640 1 -- 172.21.15.125:0/1451748273 >> 172.21.15.125:0/1451748273 conn(0x7f989c0c7910 msgr2=0x7f989c1388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:17.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.920+0000 7f98a08c9640 1 -- 172.21.15.125:0/1451748273 shutdown_connections 2023-10-30T12:47:17.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:17.920+0000 7f98a08c9640 1 -- 172.21.15.125:0/1451748273 wait complete. 2023-10-30T12:47:17.941 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:47:19.942 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:20.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2f014d6640 1 Processor -- start 2023-10-30T12:47:20.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2f014d6640 1 -- start start 2023-10-30T12:47:20.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2f014d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0cf160 0x7f2efc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:20.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2f014d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2efc0cb800 con 0x7f2efc1538a0 2023-10-30T12:47:20.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2f014d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2efc0cb330 con 0x7f2efc15d6f0 2023-10-30T12:47:20.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2f014d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0cad70 con 0x7f2efc0cf160 2023-10-30T12:47:20.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2efaffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2efc1538a0 0x7f2efc0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54746/0 (socket says 172.21.15.125:54746) 2023-10-30T12:47:20.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2efaffd640 1 -- 172.21.15.125:0/2954830255 learned_addr learned my addr 172.21.15.125:0/2954830255 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:47:20.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2efb7fe640 1 --2- 172.21.15.125:0/2954830255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0cf160 0x7f2efc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:20.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2ef9ffb640 1 -- 172.21.15.125:0/2954830255 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100241772 0 0) 0x7f2efc0cb800 con 0x7f2efc1538a0 2023-10-30T12:47:20.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2ef9ffb640 1 -- 172.21.15.125:0/2954830255 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee4003530 con 0x7f2efc1538a0 2023-10-30T12:47:20.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2ef9ffb640 1 -- 172.21.15.125:0/2954830255 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812773650 0 0) 0x7f2efc0cb330 con 0x7f2efc15d6f0 2023-10-30T12:47:20.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2efb7fe640 1 -- 172.21.15.125:0/2954830255 >> v1:172.21.15.136:6789/0 conn(0x7f2efc15d6f0 legacy=0x7f2efc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:20.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2efb7fe640 1 -- 172.21.15.125:0/2954830255 >> v1:172.21.15.125:6789/0 conn(0x7f2efc1538a0 legacy=0x7f2efc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:20.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.092+0000 7f2efb7fe640 1 -- 172.21.15.125:0/2954830255 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc0cb050 con 0x7f2efc0cf160 2023-10-30T12:47:20.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.096+0000 7f2efb7fe640 1 --2- 172.21.15.125:0/2954830255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0cf160 0x7f2efc0cf590 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009dc0 tx=0x7f2ef401efa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d853ec127a13233c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:20.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.096+0000 7f2ef9ffb640 1 -- 172.21.15.125:0/2954830255 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec001680 con 0x7f2efc0cf160 2023-10-30T12:47:20.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.096+0000 7f2ef9ffb640 1 -- 172.21.15.125:0/2954830255 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef40a8b70 con 0x7f2efc0cf160 2023-10-30T12:47:20.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.096+0000 7f2ef9ffb640 1 -- 172.21.15.125:0/2954830255 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef40b0440 con 0x7f2efc0cf160 2023-10-30T12:47:20.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.096+0000 7f2f014d6640 1 -- 172.21.15.125:0/2954830255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0cf160 msgr2=0x7f2efc0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:20.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.096+0000 7f2f014d6640 1 --2- 172.21.15.125:0/2954830255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0cf160 0x7f2efc0cf590 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4009dc0 tx=0x7f2ef401efa0 comp rx=0 tx=0).stop 2023-10-30T12:47:20.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.096+0000 7f2f014d6640 1 -- 172.21.15.125:0/2954830255 shutdown_connections 2023-10-30T12:47:20.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.096+0000 7f2f014d6640 1 --2- 172.21.15.125:0/2954830255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0cf160 0x7f2efc0cf590 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:20.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.096+0000 7f2f014d6640 1 -- 172.21.15.125:0/2954830255 >> 172.21.15.125:0/2954830255 conn(0x7f2efc06bfe0 msgr2=0x7f2efc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:20.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.096+0000 7f2f014d6640 1 -- 172.21.15.125:0/2954830255 shutdown_connections 2023-10-30T12:47:20.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.096+0000 7f2f014d6640 1 -- 172.21.15.125:0/2954830255 wait complete. 2023-10-30T12:47:20.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2f014d6640 1 Processor -- start 2023-10-30T12:47:20.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2f014d6640 1 -- start start 2023-10-30T12:47:20.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2f014d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0cf160 0x7f2efc0cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:20.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2f014d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc1538a0 0x7f2efc0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:20.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2f014d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc15d6f0 0x7f2efc1f2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:20.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2f014d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0b5240 con 0x7f2efc15d6f0 2023-10-30T12:47:20.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2f014d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc152250 con 0x7f2efc1538a0 2023-10-30T12:47:20.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2f014d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0b3840 con 0x7f2efc0cf160 2023-10-30T12:47:20.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2efb7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc15d6f0 0x7f2efc1f2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:20.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2efb7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc15d6f0 0x7f2efc1f2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38226/0 (socket says 172.21.15.125:38226) 2023-10-30T12:47:20.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2efb7fe640 1 -- 172.21.15.125:0/3851819710 learned_addr learned my addr 172.21.15.125:0/3851819710 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:20.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2efaffd640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0cf160 0x7f2efc0cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:20.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2efa7fc640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc1538a0 0x7f2efc0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:20.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2efb7fe640 1 -- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0cf160 msgr2=0x7f2efc0cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:20.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2efb7fe640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0cf160 0x7f2efc0cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:20.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2efb7fe640 1 -- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc1538a0 msgr2=0x7f2efc0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:20.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2efb7fe640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc1538a0 0x7f2efc0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:20.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2efb7fe640 1 -- 172.21.15.125:0/3851819710 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef4009a70 con 0x7f2efc15d6f0 2023-10-30T12:47:20.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2efa7fc640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc1538a0 0x7f2efc0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:20.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2efb7fe640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc15d6f0 0x7f2efc1f2210 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2ef401d650 tx=0x7f2ef401bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:20.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2ee37fe640 1 -- 172.21.15.125:0/3851819710 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef40c0070 con 0x7f2efc15d6f0 2023-10-30T12:47:20.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2ee37fe640 1 -- 172.21.15.125:0/3851819710 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef4009450 con 0x7f2efc15d6f0 2023-10-30T12:47:20.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2ee37fe640 1 -- 172.21.15.125:0/3851819710 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4004450 con 0x7f2efc15d6f0 2023-10-30T12:47:20.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2f014d6640 1 -- 172.21.15.125:0/3851819710 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2efc149a40 con 0x7f2efc15d6f0 2023-10-30T12:47:20.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.100+0000 7f2f014d6640 1 -- 172.21.15.125:0/3851819710 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2efc0b3320 con 0x7f2efc15d6f0 2023-10-30T12:47:20.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.104+0000 7f2f014d6640 1 -- 172.21.15.125:0/3851819710 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2efc0b4dc0 con 0x7f2efc15d6f0 2023-10-30T12:47:20.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.104+0000 7f2ee37fe640 1 -- 172.21.15.125:0/3851819710 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ef40b7030 con 0x7f2efc15d6f0 2023-10-30T12:47:20.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.104+0000 7f2ee37fe640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ee80664a0 0x7f2ee8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:20.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.104+0000 7f2ee37fe640 1 -- 172.21.15.125:0/3851819710 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f2ef4121e70 con 0x7f2efc15d6f0 2023-10-30T12:47:20.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.104+0000 7f2efaffd640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ee80664a0 0x7f2ee8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:20.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.104+0000 7f2efaffd640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ee80664a0 0x7f2ee8068960 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2eec001d10 tx=0x7f2eec001ca0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:20.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.108+0000 7f2ee37fe640 1 -- 172.21.15.125:0/3851819710 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2efc0b4dc0 con 0x7f2efc15d6f0 2023-10-30T12:47:20.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.296+0000 7f2f014d6640 1 -- 172.21.15.125:0/3851819710 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2efc0b4dc0 con 0x7f2efc15d6f0 2023-10-30T12:47:20.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.296+0000 7f2ee37fe640 1 -- 172.21.15.125:0/3851819710 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f2efc0b4dc0 con 0x7f2efc15d6f0 2023-10-30T12:47:20.299 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:20.299 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6318,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:55.354589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:20.301 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 91 2023-10-30T12:47:20.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.300+0000 7f2f014d6640 1 -- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ee80664a0 msgr2=0x7f2ee8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:20.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.300+0000 7f2f014d6640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ee80664a0 0x7f2ee8068960 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f2eec001d10 tx=0x7f2eec001ca0 comp rx=0 tx=0).stop 2023-10-30T12:47:20.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.300+0000 7f2f014d6640 1 -- 172.21.15.125:0/3851819710 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc15d6f0 msgr2=0x7f2efc1f2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:20.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.300+0000 7f2f014d6640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc15d6f0 0x7f2efc1f2210 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f2ef401d650 tx=0x7f2ef401bd40 comp rx=0 tx=0).stop 2023-10-30T12:47:20.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.300+0000 7f2f014d6640 1 -- 172.21.15.125:0/3851819710 shutdown_connections 2023-10-30T12:47:20.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.300+0000 7f2f014d6640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ee80664a0 0x7f2ee8068960 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:20.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.300+0000 7f2f014d6640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc15d6f0 0x7f2efc1f2210 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:20.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.300+0000 7f2f014d6640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc1538a0 0x7f2efc0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:20.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.300+0000 7f2f014d6640 1 --2- 172.21.15.125:0/3851819710 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0cf160 0x7f2efc0cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:20.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.300+0000 7f2f014d6640 1 -- 172.21.15.125:0/3851819710 >> 172.21.15.125:0/3851819710 conn(0x7f2efc06bfe0 msgr2=0x7f2efc151ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:20.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.300+0000 7f2f014d6640 1 -- 172.21.15.125:0/3851819710 shutdown_connections 2023-10-30T12:47:20.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:20.300+0000 7f2f014d6640 1 -- 172.21.15.125:0/3851819710 wait complete. 2023-10-30T12:47:20.319 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:47:22.320 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:22.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.472+0000 7f811c79d640 1 Processor -- start 2023-10-30T12:47:22.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.472+0000 7f811c79d640 1 -- start start 2023-10-30T12:47:22.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f811c79d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81140c6b80 0x7f81140bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:22.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f811c79d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81140c0a50 con 0x7f811413f4a0 2023-10-30T12:47:22.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f811c79d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811406f6e0 con 0x7f81140c6f60 2023-10-30T12:47:22.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f811c79d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811406f9c0 con 0x7f81140c6b80 2023-10-30T12:47:22.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f811ad3a640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f811413f4a0 0x7f81141566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37670/0 (socket says 172.21.15.125:37670) 2023-10-30T12:47:22.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f811ad3a640 1 -- 172.21.15.125:0/3670358596 learned_addr learned my addr 172.21.15.125:0/3670358596 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:47:22.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f811a539640 1 --2- 172.21.15.125:0/3670358596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81140c6b80 0x7f81140bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:22.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2670483329 0 0) 0x7f81140c0a50 con 0x7f811413f4a0 2023-10-30T12:47:22.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8100003530 con 0x7f811413f4a0 2023-10-30T12:47:22.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1359256700 0 0) 0x7f811406f6e0 con 0x7f81140c6f60 2023-10-30T12:47:22.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81140c0a50 con 0x7f81140c6f60 2023-10-30T12:47:22.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 935420026 0 0) 0x7f8100003530 con 0x7f811413f4a0 2023-10-30T12:47:22.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81140c6b80 msgr2=0x7f81140bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:22.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 --2- 172.21.15.125:0/3670358596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81140c6b80 0x7f81140bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:22.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 >> v1:172.21.15.136:6789/0 conn(0x7f81140c6f60 legacy=0x7f811413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:22.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811406fca0 con 0x7f811413f4a0 2023-10-30T12:47:22.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f810c001a00 con 0x7f811413f4a0 2023-10-30T12:47:22.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 >> v1:172.21.15.125:6789/0 conn(0x7f811413f4a0 legacy=0x7f81141566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:22.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 --2- 172.21.15.125:0/3670358596 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8100003b40 0x7f8100003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:22.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 --2- 172.21.15.125:0/3670358596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81000047d0 0x7f8100004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:22.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 --2- 172.21.15.125:0/3670358596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81140c6b80 0x7f8100005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:22.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811406f9c0 con 0x7f8100003b40 2023-10-30T12:47:22.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810000dc50 con 0x7f81000047d0 2023-10-30T12:47:22.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810000de10 con 0x7f81140c6b80 2023-10-30T12:47:22.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f811a539640 1 --2- 172.21.15.125:0/3670358596 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8100003b40 0x7f8100003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:22.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f8119d38640 1 --2- 172.21.15.125:0/3670358596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81000047d0 0x7f8100004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:22.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f811ad3a640 1 --2- 172.21.15.125:0/3670358596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81140c6b80 0x7f8100005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:22.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f811a539640 1 -- 172.21.15.125:0/3670358596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81140c6b80 msgr2=0x7f8100005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:22.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f811a539640 1 --2- 172.21.15.125:0/3670358596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81140c6b80 0x7f8100005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:22.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f811a539640 1 -- 172.21.15.125:0/3670358596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81000047d0 msgr2=0x7f8100004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:22.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f811a539640 1 --2- 172.21.15.125:0/3670358596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f81000047d0 0x7f8100004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:22.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.476+0000 7f811a539640 1 -- 172.21.15.125:0/3670358596 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8100003530 con 0x7f8100003b40 2023-10-30T12:47:22.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.480+0000 7f811a539640 1 --2- 172.21.15.125:0/3670358596 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8100003b40 0x7f8100003f40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f8110009ae0 tx=0x7f811001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=499650d030cf1e34 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:22.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.480+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81100a5530 con 0x7f8100003b40 2023-10-30T12:47:22.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.480+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81100a5690 con 0x7f8100003b40 2023-10-30T12:47:22.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.480+0000 7f8119537640 1 -- 172.21.15.125:0/3670358596 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81100a5a40 con 0x7f8100003b40 2023-10-30T12:47:22.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.480+0000 7f811c79d640 1 -- 172.21.15.125:0/3670358596 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8100003b40 msgr2=0x7f8100003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:22.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.480+0000 7f811c79d640 1 --2- 172.21.15.125:0/3670358596 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8100003b40 0x7f8100003f40 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f8110009ae0 tx=0x7f811001ca50 comp rx=0 tx=0).stop 2023-10-30T12:47:22.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.480+0000 7f811ad3a640 1 -- 172.21.15.125:0/3670358596 reap_dead start 2023-10-30T12:47:22.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.480+0000 7f811c79d640 1 -- 172.21.15.125:0/3670358596 shutdown_connections 2023-10-30T12:47:22.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.480+0000 7f811c79d640 1 -- 172.21.15.125:0/3670358596 >> 172.21.15.125:0/3670358596 conn(0x7f81140c7910 msgr2=0x7f81140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:22.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.480+0000 7f811c79d640 1 -- 172.21.15.125:0/3670358596 shutdown_connections 2023-10-30T12:47:22.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.480+0000 7f811c79d640 1 -- 172.21.15.125:0/3670358596 wait complete. 2023-10-30T12:47:22.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811c79d640 1 Processor -- start 2023-10-30T12:47:22.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811c79d640 1 -- start start 2023-10-30T12:47:22.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811c79d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8100003b40 0x7f8114157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:22.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811c79d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8114159440 0x7f8114158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:22.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811c79d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81140c5100 0x7f8114158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:22.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811c79d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81141592c0 con 0x7f81140c5100 2023-10-30T12:47:22.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811c79d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81141e8560 con 0x7f8100003b40 2023-10-30T12:47:22.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811c79d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8114153f50 con 0x7f8114159440 2023-10-30T12:47:22.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811ad3a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81140c5100 0x7f8114158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:22.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811ad3a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81140c5100 0x7f8114158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48346/0 (socket says 172.21.15.125:48346) 2023-10-30T12:47:22.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811ad3a640 1 -- 172.21.15.125:0/3177134428 learned_addr learned my addr 172.21.15.125:0/3177134428 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:22.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811a539640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8100003b40 0x7f8114157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:22.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f8119d38640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8114159440 0x7f8114158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:22.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811ad3a640 1 -- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8114159440 msgr2=0x7f8114158250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:22.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811ad3a640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8114159440 0x7f8114158250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:22.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811ad3a640 1 -- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8100003b40 msgr2=0x7f8114157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:22.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811ad3a640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8100003b40 0x7f8114157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:22.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811ad3a640 1 -- 172.21.15.125:0/3177134428 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8114158fe0 con 0x7f81140c5100 2023-10-30T12:47:22.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f8119d38640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8114159440 0x7f8114158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:22.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811ad3a640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81140c5100 0x7f8114158790 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f811413f8d0 tx=0x7f810c00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:22.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f810affd640 1 -- 172.21.15.125:0/3177134428 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c007ae0 con 0x7f81140c5100 2023-10-30T12:47:22.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f810affd640 1 -- 172.21.15.125:0/3177134428 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810c00edc0 con 0x7f81140c5100 2023-10-30T12:47:22.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811c79d640 1 -- 172.21.15.125:0/3177134428 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81141f1e70 con 0x7f81140c5100 2023-10-30T12:47:22.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f810affd640 1 -- 172.21.15.125:0/3177134428 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810c014650 con 0x7f81140c5100 2023-10-30T12:47:22.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.484+0000 7f811c79d640 1 -- 172.21.15.125:0/3177134428 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8114158cd0 con 0x7f81140c5100 2023-10-30T12:47:22.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.488+0000 7f811c79d640 1 -- 172.21.15.125:0/3177134428 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f811413d590 con 0x7f81140c5100 2023-10-30T12:47:22.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.488+0000 7f810affd640 1 -- 172.21.15.125:0/3177134428 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f810c0147b0 con 0x7f81140c5100 2023-10-30T12:47:22.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.488+0000 7f810affd640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80f80664d0 0x7f80f8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:22.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.488+0000 7f810affd640 1 -- 172.21.15.125:0/3177134428 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7f810c083d90 con 0x7f81140c5100 2023-10-30T12:47:22.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.488+0000 7f811a539640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80f80664d0 0x7f80f8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:22.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.488+0000 7f811a539640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80f80664d0 0x7f80f8068990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f81100af000 tx=0x7f81100046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:22.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.492+0000 7f810affd640 1 -- 172.21.15.125:0/3177134428 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f811413d590 con 0x7f81140c5100 2023-10-30T12:47:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.672+0000 7f811c79d640 1 -- 172.21.15.125:0/3177134428 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f811413d590 con 0x7f81140c5100 2023-10-30T12:47:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.672+0000 7f810affd640 1 -- 172.21.15.125:0/3177134428 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f811413d590 con 0x7f81140c5100 2023-10-30T12:47:22.673 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:22.673 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6318,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:55.354589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 91 2023-10-30T12:47:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.672+0000 7f811c79d640 1 -- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80f80664d0 msgr2=0x7f80f8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.672+0000 7f811c79d640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80f80664d0 0x7f80f8068990 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f81100af000 tx=0x7f81100046d0 comp rx=0 tx=0).stop 2023-10-30T12:47:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.672+0000 7f811c79d640 1 -- 172.21.15.125:0/3177134428 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81140c5100 msgr2=0x7f8114158790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.672+0000 7f811c79d640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81140c5100 0x7f8114158790 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f811413f8d0 tx=0x7f810c00e530 comp rx=0 tx=0).stop 2023-10-30T12:47:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.672+0000 7f811c79d640 1 -- 172.21.15.125:0/3177134428 shutdown_connections 2023-10-30T12:47:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.672+0000 7f811c79d640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80f80664d0 0x7f80f8068990 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:22.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.672+0000 7f811c79d640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81140c5100 0x7f8114158790 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:22.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.672+0000 7f811c79d640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8114159440 0x7f8114158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:22.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.672+0000 7f811c79d640 1 --2- 172.21.15.125:0/3177134428 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8100003b40 0x7f8114157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:22.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.672+0000 7f811c79d640 1 -- 172.21.15.125:0/3177134428 >> 172.21.15.125:0/3177134428 conn(0x7f81140c7910 msgr2=0x7f81140c1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:22.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.672+0000 7f811c79d640 1 -- 172.21.15.125:0/3177134428 shutdown_connections 2023-10-30T12:47:22.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:22.672+0000 7f811c79d640 1 -- 172.21.15.125:0/3177134428 wait complete. 2023-10-30T12:47:22.693 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:47:24.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:24.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe3258d6640 1 Processor -- start 2023-10-30T12:47:24.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe3258d6640 1 -- start start 2023-10-30T12:47:24.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe3258d6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe3200c8cf0 0x7fe3200c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:24.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe3258d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe3200ca340 0x7fe32015d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:24.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe3258d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32015ce30 con 0x7fe3200c90d0 2023-10-30T12:47:24.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe3258d6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe320159880 con 0x7fe3200c8cf0 2023-10-30T12:47:24.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe3258d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe320159ae0 con 0x7fe3200ca340 2023-10-30T12:47:24.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe31effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe3200c8cf0 0x7fe3200c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:24.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe31effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe3200c8cf0 0x7fe3200c9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48348/0 (socket says 172.21.15.125:48348) 2023-10-30T12:47:24.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe31effd640 1 -- 172.21.15.125:0/1722958314 learned_addr learned my addr 172.21.15.125:0/1722958314 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:24.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe31f7fe640 1 --2- 172.21.15.125:0/1722958314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe3200ca340 0x7fe32015d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:24.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe31effd640 1 -- 172.21.15.125:0/1722958314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe3200ca340 msgr2=0x7fe32015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:24.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe31effd640 1 --2- 172.21.15.125:0/1722958314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe3200ca340 0x7fe32015d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:24.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe31effd640 1 -- 172.21.15.125:0/1722958314 >> v1:172.21.15.136:6789/0 conn(0x7fe3200c90d0 legacy=0x7fe3200cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:24.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.840+0000 7fe31effd640 1 -- 172.21.15.125:0/1722958314 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe320159dc0 con 0x7fe3200c8cf0 2023-10-30T12:47:24.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.844+0000 7fe31effd640 1 --2- 172.21.15.125:0/1722958314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3200c8cf0 0x7fe3200c9e00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fe310009870 tx=0x7fe31001c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6afdf676e28d2ee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:24.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.844+0000 7fe31e7fc640 1 -- 172.21.15.125:0/1722958314 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3100ab620 con 0x7fe3200c8cf0 2023-10-30T12:47:24.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.844+0000 7fe31e7fc640 1 -- 172.21.15.125:0/1722958314 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3100ab780 con 0x7fe3200c8cf0 2023-10-30T12:47:24.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.844+0000 7fe31e7fc640 1 -- 172.21.15.125:0/1722958314 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3100abb10 con 0x7fe3200c8cf0 2023-10-30T12:47:24.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.844+0000 7fe3258d6640 1 -- 172.21.15.125:0/1722958314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3200c8cf0 msgr2=0x7fe3200c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:24.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.844+0000 7fe3258d6640 1 --2- 172.21.15.125:0/1722958314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3200c8cf0 0x7fe3200c9e00 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fe310009870 tx=0x7fe31001c510 comp rx=0 tx=0).stop 2023-10-30T12:47:24.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.844+0000 7fe3258d6640 1 -- 172.21.15.125:0/1722958314 shutdown_connections 2023-10-30T12:47:24.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.844+0000 7fe3258d6640 1 --2- 172.21.15.125:0/1722958314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe3200ca340 0x7fe32015d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:24.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.844+0000 7fe3258d6640 1 --2- 172.21.15.125:0/1722958314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3200c8cf0 0x7fe3200c9e00 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:24.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.844+0000 7fe3258d6640 1 -- 172.21.15.125:0/1722958314 >> 172.21.15.125:0/1722958314 conn(0x7fe32006be40 msgr2=0x7fe32006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:24.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.844+0000 7fe3258d6640 1 -- 172.21.15.125:0/1722958314 shutdown_connections 2023-10-30T12:47:24.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.844+0000 7fe3258d6640 1 -- 172.21.15.125:0/1722958314 wait complete. 2023-10-30T12:47:24.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe3258d6640 1 Processor -- start 2023-10-30T12:47:24.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe3258d6640 1 -- start start 2023-10-30T12:47:24.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe3258d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3200c8cf0 0x7fe320151990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:24.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe3258d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3200c90d0 0x7fe3200cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:24.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe3258d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe3200ca340 0x7fe3201f5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:24.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe3258d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3200c4890 con 0x7fe3200c8cf0 2023-10-30T12:47:24.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe3258d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32015c790 con 0x7fe3200c90d0 2023-10-30T12:47:24.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe3258d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe32015c240 con 0x7fe3200ca340 2023-10-30T12:47:24.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe31effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3200c8cf0 0x7fe320151990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:24.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe31effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3200c8cf0 0x7fe320151990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48356/0 (socket says 172.21.15.125:48356) 2023-10-30T12:47:24.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe31effd640 1 -- 172.21.15.125:0/1528504275 learned_addr learned my addr 172.21.15.125:0/1528504275 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:24.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe3167fc640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3200c90d0 0x7fe3200cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:24.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe31f7fe640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe3200ca340 0x7fe3201f5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:24.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe31effd640 1 -- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe3200ca340 msgr2=0x7fe3201f5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:24.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe31effd640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe3200ca340 0x7fe3201f5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:24.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe31effd640 1 -- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3200c90d0 msgr2=0x7fe3200cdfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:24.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe31effd640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3200c90d0 0x7fe3200cdfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:24.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe31effd640 1 -- 172.21.15.125:0/1528504275 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3100094d0 con 0x7fe3200c8cf0 2023-10-30T12:47:24.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe31f7fe640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe3200ca340 0x7fe3201f5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:24.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe31effd640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3200c8cf0 0x7fe320151990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fe3100ad040 tx=0x7fe31001ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:24.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe31cff9640 1 -- 172.21.15.125:0/1528504275 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3100be3f0 con 0x7fe3200c8cf0 2023-10-30T12:47:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe31cff9640 1 -- 172.21.15.125:0/1528504275 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3100be570 con 0x7fe3200c8cf0 2023-10-30T12:47:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe3258d6640 1 -- 172.21.15.125:0/1528504275 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32015c4b0 con 0x7fe3200c8cf0 2023-10-30T12:47:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe31cff9640 1 -- 172.21.15.125:0/1528504275 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3100be920 con 0x7fe3200c8cf0 2023-10-30T12:47:24.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe3258d6640 1 -- 172.21.15.125:0/1528504275 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe320150d00 con 0x7fe3200c8cf0 2023-10-30T12:47:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.848+0000 7fe3258d6640 1 -- 172.21.15.125:0/1528504275 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe320150e90 con 0x7fe3200c8cf0 2023-10-30T12:47:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.852+0000 7fe31cff9640 1 -- 172.21.15.125:0/1528504275 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe3100b43f0 con 0x7fe3200c8cf0 2023-10-30T12:47:24.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.852+0000 7fe31cff9640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3000664a0 0x7fe300068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:24.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.852+0000 7fe31cff9640 1 -- 172.21.15.125:0/1528504275 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6426+0+0 (secure 0 0 0) 0x7fe31011e730 con 0x7fe3200c8cf0 2023-10-30T12:47:24.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.852+0000 7fe3167fc640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3000664a0 0x7fe300068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:24.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.852+0000 7fe3167fc640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3000664a0 0x7fe300068960 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe3201584c0 tx=0x7fe308009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:24.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:24.856+0000 7fe31cff9640 1 -- 172.21.15.125:0/1528504275 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe3100eb950 con 0x7fe3200c8cf0 2023-10-30T12:47:25.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:25.040+0000 7fe3258d6640 1 -- 172.21.15.125:0/1528504275 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe32004e170 con 0x7fe3200c8cf0 2023-10-30T12:47:25.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:25.040+0000 7fe31cff9640 1 -- 172.21.15.125:0/1528504275 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+6531 (secure 0 0 0) 0x7fe3100eef70 con 0x7fe3200c8cf0 2023-10-30T12:47:25.042 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:25.042 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6318,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":91}],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:46:55.354589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":5935,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5935":{"gid":5935,"name":"f","rank":0,"incarnation":86,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/1510290691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1510290691},{"type":"v1","addr":"172.21.15.136:6839","nonce":1510290691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:25.043 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 91 2023-10-30T12:47:25.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:25.040+0000 7fe3258d6640 1 -- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3000664a0 msgr2=0x7fe300068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:25.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:25.040+0000 7fe3258d6640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3000664a0 0x7fe300068960 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fe3201584c0 tx=0x7fe308009040 comp rx=0 tx=0).stop 2023-10-30T12:47:25.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:25.040+0000 7fe3258d6640 1 -- 172.21.15.125:0/1528504275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3200c8cf0 msgr2=0x7fe320151990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:25.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:25.040+0000 7fe3258d6640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3200c8cf0 0x7fe320151990 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fe3100ad040 tx=0x7fe31001ad50 comp rx=0 tx=0).stop 2023-10-30T12:47:25.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:25.040+0000 7fe3258d6640 1 -- 172.21.15.125:0/1528504275 shutdown_connections 2023-10-30T12:47:25.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:25.040+0000 7fe3258d6640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3000664a0 0x7fe300068960 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:25.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:25.040+0000 7fe3258d6640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe3200ca340 0x7fe3201f5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:25.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:25.040+0000 7fe3258d6640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe3200c90d0 0x7fe3200cdfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:25.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:25.040+0000 7fe3258d6640 1 --2- 172.21.15.125:0/1528504275 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe3200c8cf0 0x7fe320151990 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:25.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:25.040+0000 7fe3258d6640 1 -- 172.21.15.125:0/1528504275 >> 172.21.15.125:0/1528504275 conn(0x7fe32006be40 msgr2=0x7fe3200b3290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:25.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:25.040+0000 7fe3258d6640 1 -- 172.21.15.125:0/1528504275 shutdown_connections 2023-10-30T12:47:25.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:25.040+0000 7fe3258d6640 1 -- 172.21.15.125:0/1528504275 wait complete. 2023-10-30T12:47:25.062 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T12:47:27.063 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:27.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.208+0000 7f27da9d3640 1 Processor -- start 2023-10-30T12:47:27.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.208+0000 7f27da9d3640 1 -- start start 2023-10-30T12:47:27.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.208+0000 7f27da9d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f27d406f7a0 0x7f27d406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.208+0000 7f27da9d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27d40c6bd0 0x7f27d4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.208+0000 7f27da9d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f27d4071110 0x7f27d4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.208+0000 7f27da9d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27d40700c0 con 0x7f27d4071110 2023-10-30T12:47:27.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.208+0000 7f27da9d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27d406a3d0 con 0x7f27d406f7a0 2023-10-30T12:47:27.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.208+0000 7f27da9d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d4066cf0 con 0x7f27d40c6bd0 2023-10-30T12:47:27.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d8f70640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f27d4071110 0x7f27d4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d8f70640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f27d4071110 0x7f27d4158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48362/0 (socket says 172.21.15.125:48362) 2023-10-30T12:47:27.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d8f70640 1 -- 172.21.15.125:0/3259002014 learned_addr learned my addr 172.21.15.125:0/3259002014 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:27.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d3fff640 1 --2- 172.21.15.125:0/3259002014 >> v2:172.21.15.136:3300/0 conn(0x7f27d406f7a0 0x7f27d406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d37fe640 1 --2- 172.21.15.125:0/3259002014 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27d40c6bd0 0x7f27d4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d8f70640 1 -- 172.21.15.125:0/3259002014 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27d40c6bd0 msgr2=0x7f27d4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:27.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d8f70640 1 --2- 172.21.15.125:0/3259002014 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27d40c6bd0 0x7f27d4069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d8f70640 1 -- 172.21.15.125:0/3259002014 >> v2:172.21.15.136:3300/0 conn(0x7f27d406f7a0 msgr2=0x7f27d406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:27.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d8f70640 1 --2- 172.21.15.125:0/3259002014 >> v2:172.21.15.136:3300/0 conn(0x7f27d406f7a0 0x7f27d406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d8f70640 1 -- 172.21.15.125:0/3259002014 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d40c70d0 con 0x7f27d4071110 2023-10-30T12:47:27.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d3fff640 1 --2- 172.21.15.125:0/3259002014 >> v2:172.21.15.136:3300/0 conn(0x7f27d406f7a0 0x7f27d406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:27.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d8f70640 1 --2- 172.21.15.125:0/3259002014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d4071110 0x7f27d4158110 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f27d406c830 tx=0x7f27cc01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49db04a4ecb1190d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:27.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d2ffd640 1 -- 172.21.15.125:0/3259002014 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc0af400 con 0x7f27d4071110 2023-10-30T12:47:27.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d2ffd640 1 -- 172.21.15.125:0/3259002014 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27cc0ab050 con 0x7f27d4071110 2023-10-30T12:47:27.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.212+0000 7f27d2ffd640 1 -- 172.21.15.125:0/3259002014 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc0af780 con 0x7f27d4071110 2023-10-30T12:47:27.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 -- 172.21.15.125:0/3259002014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d4071110 msgr2=0x7f27d4158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:27.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 --2- 172.21.15.125:0/3259002014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d4071110 0x7f27d4158110 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f27d406c830 tx=0x7f27cc01f1c0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 -- 172.21.15.125:0/3259002014 shutdown_connections 2023-10-30T12:47:27.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 --2- 172.21.15.125:0/3259002014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d4071110 0x7f27d4158110 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 --2- 172.21.15.125:0/3259002014 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27d40c6bd0 0x7f27d4069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 --2- 172.21.15.125:0/3259002014 >> v2:172.21.15.136:3300/0 conn(0x7f27d406f7a0 0x7f27d406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 -- 172.21.15.125:0/3259002014 >> 172.21.15.125:0/3259002014 conn(0x7f27d40c7910 msgr2=0x7f27d40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:27.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 -- 172.21.15.125:0/3259002014 shutdown_connections 2023-10-30T12:47:27.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 -- 172.21.15.125:0/3259002014 wait complete. 2023-10-30T12:47:27.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 Processor -- start 2023-10-30T12:47:27.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 -- start start 2023-10-30T12:47:27.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27d406f7a0 0x7f27d4153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d4071110 0x7f27d40c1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27d40c6bd0 0x7f27d40c1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d40c66c0 con 0x7f27d4071110 2023-10-30T12:47:27.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d4151ea0 con 0x7f27d40c6bd0 2023-10-30T12:47:27.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27da9d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27d4152140 con 0x7f27d406f7a0 2023-10-30T12:47:27.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27d37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d4071110 0x7f27d40c1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27d37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d4071110 0x7f27d40c1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48366/0 (socket says 172.21.15.125:48366) 2023-10-30T12:47:27.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27d37fe640 1 -- 172.21.15.125:0/1097522295 learned_addr learned my addr 172.21.15.125:0/1097522295 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:27.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27d3fff640 1 --2- 172.21.15.125:0/1097522295 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27d406f7a0 0x7f27d4153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.216+0000 7f27d8f70640 1 --2- 172.21.15.125:0/1097522295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27d40c6bd0 0x7f27d40c1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.220+0000 7f27d37fe640 1 -- 172.21.15.125:0/1097522295 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27d406f7a0 msgr2=0x7f27d4153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:27.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.220+0000 7f27d37fe640 1 --2- 172.21.15.125:0/1097522295 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27d406f7a0 0x7f27d4153b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.220+0000 7f27d37fe640 1 -- 172.21.15.125:0/1097522295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27d40c6bd0 msgr2=0x7f27d40c1aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:27.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.220+0000 7f27d37fe640 1 --2- 172.21.15.125:0/1097522295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27d40c6bd0 0x7f27d40c1aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.220+0000 7f27d37fe640 1 -- 172.21.15.125:0/1097522295 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc009b30 con 0x7f27d4071110 2023-10-30T12:47:27.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.220+0000 7f27d37fe640 1 --2- 172.21.15.125:0/1097522295 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d4071110 0x7f27d40c1560 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f27c800d5f0 tx=0x7f27c800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:27.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.220+0000 7f27d0ff9640 1 -- 172.21.15.125:0/1097522295 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c80125f0 con 0x7f27d4071110 2023-10-30T12:47:27.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.220+0000 7f27d0ff9640 1 -- 172.21.15.125:0/1097522295 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c8012750 con 0x7f27d4071110 2023-10-30T12:47:27.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.220+0000 7f27d0ff9640 1 -- 172.21.15.125:0/1097522295 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c8012b00 con 0x7f27d4071110 2023-10-30T12:47:27.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.220+0000 7f27da9d3640 1 -- 172.21.15.125:0/1097522295 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d40ca230 con 0x7f27d4071110 2023-10-30T12:47:27.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.220+0000 7f27da9d3640 1 -- 172.21.15.125:0/1097522295 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d40ca7f0 con 0x7f27d4071110 2023-10-30T12:47:27.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.220+0000 7f27da9d3640 1 -- 172.21.15.125:0/1097522295 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d404e170 con 0x7f27d4071110 2023-10-30T12:47:27.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.220+0000 7f27d0ff9640 1 -- 172.21.15.125:0/1097522295 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f27c8012c60 con 0x7f27d4071110 2023-10-30T12:47:27.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.220+0000 7f27d0ff9640 1 --2- 172.21.15.125:0/1097522295 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f27c00664a0 0x7f27c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.224+0000 7f27d0ff9640 1 -- 172.21.15.125:0/1097522295 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f27c80846e0 con 0x7f27d4071110 2023-10-30T12:47:27.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.224+0000 7f27d3fff640 1 --2- 172.21.15.125:0/1097522295 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f27c00664a0 0x7f27c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.224+0000 7f27d3fff640 1 --2- 172.21.15.125:0/1097522295 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f27c00664a0 0x7f27c0068960 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f27c400b990 tx=0x7f27c4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:27.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.224+0000 7f27d0ff9640 1 -- 172.21.15.125:0/1097522295 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f27c8051850 con 0x7f27d4071110 2023-10-30T12:47:27.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.408+0000 7f27da9d3640 1 -- 172.21.15.125:0/1097522295 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27d40afa10 con 0x7f27d4071110 2023-10-30T12:47:27.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.408+0000 7f27d0ff9640 1 -- 172.21.15.125:0/1097522295 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f27c8054e70 con 0x7f27d4071110 2023-10-30T12:47:27.412 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:27.412 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:25.242485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:27.412 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 92 2023-10-30T12:47:27.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.412+0000 7f27da9d3640 1 -- 172.21.15.125:0/1097522295 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f27c00664a0 msgr2=0x7f27c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:27.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.412+0000 7f27da9d3640 1 --2- 172.21.15.125:0/1097522295 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f27c00664a0 0x7f27c0068960 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f27c400b990 tx=0x7f27c4008040 comp rx=0 tx=0).stop 2023-10-30T12:47:27.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.412+0000 7f27da9d3640 1 -- 172.21.15.125:0/1097522295 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d4071110 msgr2=0x7f27d40c1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:27.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.412+0000 7f27da9d3640 1 --2- 172.21.15.125:0/1097522295 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d4071110 0x7f27d40c1560 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f27c800d5f0 tx=0x7f27c800dab0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.412+0000 7f27da9d3640 1 -- 172.21.15.125:0/1097522295 shutdown_connections 2023-10-30T12:47:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.412+0000 7f27da9d3640 1 --2- 172.21.15.125:0/1097522295 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f27c00664a0 0x7f27c0068960 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.412+0000 7f27da9d3640 1 --2- 172.21.15.125:0/1097522295 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27d40c6bd0 0x7f27d40c1aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.412+0000 7f27da9d3640 1 --2- 172.21.15.125:0/1097522295 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27d4071110 0x7f27d40c1560 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.412+0000 7f27da9d3640 1 --2- 172.21.15.125:0/1097522295 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27d406f7a0 0x7f27d4153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.412+0000 7f27da9d3640 1 -- 172.21.15.125:0/1097522295 >> 172.21.15.125:0/1097522295 conn(0x7f27d40c7910 msgr2=0x7f27d40c5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:27.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.412+0000 7f27da9d3640 1 -- 172.21.15.125:0/1097522295 shutdown_connections 2023-10-30T12:47:27.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.412+0000 7f27da9d3640 1 -- 172.21.15.125:0/1097522295 wait complete. 2023-10-30T12:47:27.433 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T12:47:27.433 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:47:27.433 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:27.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.568+0000 7fb8ab2d6640 1 Processor -- start 2023-10-30T12:47:27.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8ab2d6640 1 -- start start 2023-10-30T12:47:27.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8ab2d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a40c6bd0 0x7fb8a406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8ab2d6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb8a4071110 0x7fb8a4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8ab2d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8a4066cf0 con 0x7fb8a406f7a0 2023-10-30T12:47:27.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8ab2d6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a40700c0 con 0x7fb8a4071110 2023-10-30T12:47:27.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8ab2d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a40c70d0 con 0x7fb8a40c6bd0 2023-10-30T12:47:27.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8a9072640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb8a406f7a0 0x7fb8a4069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37682/0 (socket says 172.21.15.125:37682) 2023-10-30T12:47:27.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8a9072640 1 -- 172.21.15.125:0/152462780 learned_addr learned my addr 172.21.15.125:0/152462780 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:47:27.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8a9873640 1 --2- 172.21.15.125:0/152462780 >> v2:172.21.15.136:3300/0 conn(0x7fb8a4071110 0x7fb8a4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8a8871640 1 --2- 172.21.15.125:0/152462780 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a40c6bd0 0x7fb8a406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 -- 172.21.15.125:0/152462780 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370033437 0 0) 0x7fb8a4066cf0 con 0x7fb8a406f7a0 2023-10-30T12:47:27.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 -- 172.21.15.125:0/152462780 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb890003530 con 0x7fb8a406f7a0 2023-10-30T12:47:27.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 -- 172.21.15.125:0/152462780 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2539820815 0 0) 0x7fb890003530 con 0x7fb8a406f7a0 2023-10-30T12:47:27.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 -- 172.21.15.125:0/152462780 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a40c6bd0 msgr2=0x7fb8a406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:27.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 --2- 172.21.15.125:0/152462780 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a40c6bd0 0x7fb8a406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 -- 172.21.15.125:0/152462780 >> v2:172.21.15.136:3300/0 conn(0x7fb8a4071110 msgr2=0x7fb8a4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:27.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 --2- 172.21.15.125:0/152462780 >> v2:172.21.15.136:3300/0 conn(0x7fb8a4071110 0x7fb8a4158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 -- 172.21.15.125:0/152462780 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a40c7390 con 0x7fb8a406f7a0 2023-10-30T12:47:27.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 -- 172.21.15.125:0/152462780 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fb8a0001680 con 0x7fb8a406f7a0 2023-10-30T12:47:27.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 -- 172.21.15.125:0/152462780 >> v1:172.21.15.125:6789/0 conn(0x7fb8a406f7a0 legacy=0x7fb8a4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:27.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 --2- 172.21.15.125:0/152462780 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a40c6bd0 0x7fb890003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 --2- 172.21.15.125:0/152462780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb890003f00 0x7fb8900047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 --2- 172.21.15.125:0/152462780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb890004d10 0x7fb890005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 -- 172.21.15.125:0/152462780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a40700c0 con 0x7fb890003f00 2023-10-30T12:47:27.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 -- 172.21.15.125:0/152462780 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a40c70d0 con 0x7fb890004d10 2023-10-30T12:47:27.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb89bfff640 1 -- 172.21.15.125:0/152462780 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89000b9c0 con 0x7fb8a40c6bd0 2023-10-30T12:47:27.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8a8871640 1 --2- 172.21.15.125:0/152462780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb890003f00 0x7fb8900047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8a9072640 1 --2- 172.21.15.125:0/152462780 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a40c6bd0 0x7fb890003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8a9873640 1 --2- 172.21.15.125:0/152462780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb890004d10 0x7fb890005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8a8871640 1 -- 172.21.15.125:0/152462780 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a40c6bd0 msgr2=0x7fb890003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:27.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8a8871640 1 --2- 172.21.15.125:0/152462780 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a40c6bd0 0x7fb890003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8a8871640 1 -- 172.21.15.125:0/152462780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb890004d10 msgr2=0x7fb890005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:27.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8a8871640 1 --2- 172.21.15.125:0/152462780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb890004d10 0x7fb890005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.572+0000 7fb8a8871640 1 -- 172.21.15.125:0/152462780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb890003530 con 0x7fb890003f00 2023-10-30T12:47:27.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.576+0000 7fb8a8871640 1 --2- 172.21.15.125:0/152462780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb890003f00 0x7fb8900047d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fb894009ae0 tx=0x7fb89401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db872cbea0fc1de2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:27.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.576+0000 7fb89bfff640 1 -- 172.21.15.125:0/152462780 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8940a5640 con 0x7fb890003f00 2023-10-30T12:47:27.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.576+0000 7fb89bfff640 1 -- 172.21.15.125:0/152462780 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8940a57a0 con 0x7fb890003f00 2023-10-30T12:47:27.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.576+0000 7fb89bfff640 1 -- 172.21.15.125:0/152462780 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8940a5b50 con 0x7fb890003f00 2023-10-30T12:47:27.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.576+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/152462780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb890003f00 msgr2=0x7fb8900047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:27.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.576+0000 7fb8ab2d6640 1 --2- 172.21.15.125:0/152462780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb890003f00 0x7fb8900047d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fb894009ae0 tx=0x7fb89401ca50 comp rx=0 tx=0).stop 2023-10-30T12:47:27.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.576+0000 7fb8a9873640 1 -- 172.21.15.125:0/152462780 reap_dead start 2023-10-30T12:47:27.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.576+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/152462780 shutdown_connections 2023-10-30T12:47:27.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.576+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/152462780 >> 172.21.15.125:0/152462780 conn(0x7fb8a40c7910 msgr2=0x7fb8a40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:27.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.576+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/152462780 shutdown_connections 2023-10-30T12:47:27.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.576+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/152462780 wait complete. 2023-10-30T12:47:27.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8ab2d6640 1 Processor -- start 2023-10-30T12:47:27.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8ab2d6640 1 -- start start 2023-10-30T12:47:27.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8ab2d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb890003f00 0x7fb8a41526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8ab2d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a4151ce0 0x7fb8a41ef6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8ab2d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a41e6160 0x7fb8a40c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8ab2d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a4153b00 con 0x7fb8a4151ce0 2023-10-30T12:47:27.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8ab2d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a41ef210 con 0x7fb890003f00 2023-10-30T12:47:27.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8ab2d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a40afff0 con 0x7fb8a41e6160 2023-10-30T12:47:27.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8a8871640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a4151ce0 0x7fb8a41ef6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8a8871640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a4151ce0 0x7fb8a41ef6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48380/0 (socket says 172.21.15.125:48380) 2023-10-30T12:47:27.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8a8871640 1 -- 172.21.15.125:0/1395945981 learned_addr learned my addr 172.21.15.125:0/1395945981 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:27.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8a9072640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb890003f00 0x7fb8a41526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8a9873640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a41e6160 0x7fb8a40c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8a8871640 1 -- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a41e6160 msgr2=0x7fb8a40c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:27.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8a8871640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a41e6160 0x7fb8a40c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8a8871640 1 -- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb890003f00 msgr2=0x7fb8a41526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:27.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8a8871640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb890003f00 0x7fb8a41526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8a8871640 1 -- 172.21.15.125:0/1395945981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a4154f70 con 0x7fb8a4151ce0 2023-10-30T12:47:27.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8a9072640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb890003f00 0x7fb8a41526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:27.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8a8871640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a4151ce0 0x7fb8a41ef6a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fb894009ae0 tx=0x7fb890005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:27.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb899ffb640 1 -- 172.21.15.125:0/1395945981 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb894004230 con 0x7fb8a4151ce0 2023-10-30T12:47:27.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb899ffb640 1 -- 172.21.15.125:0/1395945981 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb894004390 con 0x7fb8a4151ce0 2023-10-30T12:47:27.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/1395945981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a41eebe0 con 0x7fb8a4151ce0 2023-10-30T12:47:27.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb899ffb640 1 -- 172.21.15.125:0/1395945981 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8940ad850 con 0x7fb8a4151ce0 2023-10-30T12:47:27.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/1395945981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a41f2ee0 con 0x7fb8a4151ce0 2023-10-30T12:47:27.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.580+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/1395945981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a41585f0 con 0x7fb8a4151ce0 2023-10-30T12:47:27.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.584+0000 7fb899ffb640 1 -- 172.21.15.125:0/1395945981 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb8940b4040 con 0x7fb8a4151ce0 2023-10-30T12:47:27.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.584+0000 7fb899ffb640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8880664d0 0x7fb888068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:27.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.584+0000 7fb899ffb640 1 -- 172.21.15.125:0/1395945981 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7fb89411e050 con 0x7fb8a4151ce0 2023-10-30T12:47:27.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.584+0000 7fb8a9072640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8880664d0 0x7fb888068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:27.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.584+0000 7fb8a9072640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8880664d0 0x7fb888068990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0001640 tx=0x7fb8a0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:27.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.588+0000 7fb899ffb640 1 -- 172.21.15.125:0/1395945981 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb8940eb150 con 0x7fb8a4151ce0 2023-10-30T12:47:27.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.768+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/1395945981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8a41ef380 con 0x7fb8a4151ce0 2023-10-30T12:47:27.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.768+0000 7fb899ffb640 1 -- 172.21.15.125:0/1395945981 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5736 (secure 0 0 0) 0x7fb8940b2090 con 0x7fb8a4151ce0 2023-10-30T12:47:27.769 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:27.769 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:25.242485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:27.769 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 92 2023-10-30T12:47:27.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.768+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8880664d0 msgr2=0x7fb888068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:27.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.768+0000 7fb8ab2d6640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8880664d0 0x7fb888068990 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0001640 tx=0x7fb8a0009040 comp rx=0 tx=0).stop 2023-10-30T12:47:27.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.768+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/1395945981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a4151ce0 msgr2=0x7fb8a41ef6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:27.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.768+0000 7fb8ab2d6640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a4151ce0 0x7fb8a41ef6a0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fb894009ae0 tx=0x7fb890005800 comp rx=0 tx=0).stop 2023-10-30T12:47:27.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.768+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/1395945981 shutdown_connections 2023-10-30T12:47:27.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.768+0000 7fb8ab2d6640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8880664d0 0x7fb888068990 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.768+0000 7fb8ab2d6640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a41e6160 0x7fb8a40c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.768+0000 7fb8ab2d6640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a4151ce0 0x7fb8a41ef6a0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.768+0000 7fb8ab2d6640 1 --2- 172.21.15.125:0/1395945981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb890003f00 0x7fb8a41526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:27.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.768+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/1395945981 >> 172.21.15.125:0/1395945981 conn(0x7fb8a40c7910 msgr2=0x7fb8a40bffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:27.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.768+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/1395945981 shutdown_connections 2023-10-30T12:47:27.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:27.768+0000 7fb8ab2d6640 1 -- 172.21.15.125:0/1395945981 wait complete. 2023-10-30T12:47:27.790 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 92, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 92, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5988": { "addr": "172.21.15.125:6837/3156415951", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 3156415951, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 3156415951, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5988, "incarnation": 78, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 28 }, "gid_6318": { "addr": "172.21.15.125:6836/1721514617", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1721514617, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1721514617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6318, "incarnation": 92, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:47:25.242485+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6318, "mds_1": 5988 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:47:29.791 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:29.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.936+0000 7f2b0ca09640 1 Processor -- start 2023-10-30T12:47:29.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.936+0000 7f2b0ca09640 1 -- start start 2023-10-30T12:47:29.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.936+0000 7f2b0ca09640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2b080c6b80 0x7f2b080bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:29.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.936+0000 7f2b0ca09640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2b0813f4a0 0x7f2b080c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:29.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.936+0000 7f2b0ca09640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b080c0a50 con 0x7f2b080c6f60 2023-10-30T12:47:29.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.936+0000 7f2b0ca09640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b0806f6e0 con 0x7f2b0813f4a0 2023-10-30T12:47:29.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.936+0000 7f2b0ca09640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b0806f9c0 con 0x7f2b080c6b80 2023-10-30T12:47:29.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.936+0000 7f2b06d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2b0813f4a0 0x7f2b080c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:29.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.936+0000 7f2b06d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2b0813f4a0 0x7f2b080c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48386/0 (socket says 172.21.15.125:48386) 2023-10-30T12:47:29.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.936+0000 7f2b06d9d640 1 -- 172.21.15.125:0/2382042135 learned_addr learned my addr 172.21.15.125:0/2382042135 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:29.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.936+0000 7f2b0659c640 1 --2- 172.21.15.125:0/2382042135 >> v2:172.21.15.136:3300/0 conn(0x7f2b080c6b80 0x7f2b080bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:29.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b06d9d640 1 -- 172.21.15.125:0/2382042135 >> v2:172.21.15.136:3300/0 conn(0x7f2b080c6b80 msgr2=0x7f2b080bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:29.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b06d9d640 1 --2- 172.21.15.125:0/2382042135 >> v2:172.21.15.136:3300/0 conn(0x7f2b080c6b80 0x7f2b080bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:29.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b06d9d640 1 -- 172.21.15.125:0/2382042135 >> v1:172.21.15.136:6789/0 conn(0x7f2b080c6f60 legacy=0x7f2b0813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:29.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b06d9d640 1 -- 172.21.15.125:0/2382042135 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0806fca0 con 0x7f2b0813f4a0 2023-10-30T12:47:29.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b06d9d640 1 --2- 172.21.15.125:0/2382042135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b0813f4a0 0x7f2b080c0400 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f2b0806f1b0 tx=0x7f2b0001ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45dcf45442e6ea8a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:29.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b05d9b640 1 -- 172.21.15.125:0/2382042135 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b0000b2d0 con 0x7f2b0813f4a0 2023-10-30T12:47:29.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b05d9b640 1 -- 172.21.15.125:0/2382042135 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0000b430 con 0x7f2b0813f4a0 2023-10-30T12:47:29.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b05d9b640 1 -- 172.21.15.125:0/2382042135 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b000a7bd0 con 0x7f2b0813f4a0 2023-10-30T12:47:29.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b0ca09640 1 -- 172.21.15.125:0/2382042135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b0813f4a0 msgr2=0x7f2b080c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:29.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b0ca09640 1 --2- 172.21.15.125:0/2382042135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b0813f4a0 0x7f2b080c0400 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f2b0806f1b0 tx=0x7f2b0001ebd0 comp rx=0 tx=0).stop 2023-10-30T12:47:29.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b0ca09640 1 -- 172.21.15.125:0/2382042135 shutdown_connections 2023-10-30T12:47:29.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b0ca09640 1 --2- 172.21.15.125:0/2382042135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b0813f4a0 0x7f2b080c0400 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:29.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b0ca09640 1 --2- 172.21.15.125:0/2382042135 >> v2:172.21.15.136:3300/0 conn(0x7f2b080c6b80 0x7f2b080bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:29.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b0ca09640 1 -- 172.21.15.125:0/2382042135 >> 172.21.15.125:0/2382042135 conn(0x7f2b080c7910 msgr2=0x7f2b080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:29.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b0ca09640 1 -- 172.21.15.125:0/2382042135 shutdown_connections 2023-10-30T12:47:29.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.940+0000 7f2b0ca09640 1 -- 172.21.15.125:0/2382042135 wait complete. 2023-10-30T12:47:29.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0ca09640 1 Processor -- start 2023-10-30T12:47:29.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0ca09640 1 -- start start 2023-10-30T12:47:29.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0ca09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b080c6b80 0x7f2b08157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:29.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0ca09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b080c6f60 0x7f2b080c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:29.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0ca09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2b0813f4a0 0x7f2b080ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:29.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0ca09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b08134120 con 0x7f2b080c6b80 2023-10-30T12:47:29.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0ca09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b081333e0 con 0x7f2b080c6f60 2023-10-30T12:47:29.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0ca09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b080c1b30 con 0x7f2b0813f4a0 2023-10-30T12:47:29.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b080c6b80 0x7f2b08157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:29.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b080c6b80 0x7f2b08157790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48390/0 (socket says 172.21.15.125:48390) 2023-10-30T12:47:29.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0659c640 1 -- 172.21.15.125:0/1309068635 learned_addr learned my addr 172.21.15.125:0/1309068635 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:29.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b06d9d640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2b0813f4a0 0x7f2b080ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:29.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2afdd9b640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b080c6f60 0x7f2b080c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:29.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0659c640 1 -- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2b0813f4a0 msgr2=0x7f2b080ca400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:29.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0659c640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2b0813f4a0 0x7f2b080ca400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:29.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0659c640 1 -- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b080c6f60 msgr2=0x7f2b080c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:29.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0659c640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b080c6f60 0x7f2b080c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:29.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0659c640 1 -- 172.21.15.125:0/1309068635 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b0000bd40 con 0x7f2b080c6b80 2023-10-30T12:47:29.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b06d9d640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2b0813f4a0 0x7f2b080ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:29.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.944+0000 7f2b0659c640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b080c6b80 0x7f2b08157790 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2af800e930 tx=0x7f2af800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:29.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.948+0000 7f2aff7fe640 1 -- 172.21.15.125:0/1309068635 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af8009690 con 0x7f2b080c6b80 2023-10-30T12:47:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.948+0000 7f2aff7fe640 1 -- 172.21.15.125:0/1309068635 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af800ca00 con 0x7f2b080c6b80 2023-10-30T12:47:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.948+0000 7f2aff7fe640 1 -- 172.21.15.125:0/1309068635 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af800cd70 con 0x7f2b080c6b80 2023-10-30T12:47:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.948+0000 7f2b0ca09640 1 -- 172.21.15.125:0/1309068635 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b080ca940 con 0x7f2b080c6b80 2023-10-30T12:47:29.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.948+0000 7f2b0ca09640 1 -- 172.21.15.125:0/1309068635 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b08134370 con 0x7f2b080c6b80 2023-10-30T12:47:29.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.948+0000 7f2b0ca09640 1 -- 172.21.15.125:0/1309068635 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b0804e170 con 0x7f2b080c6b80 2023-10-30T12:47:29.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.948+0000 7f2aff7fe640 1 -- 172.21.15.125:0/1309068635 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2af8013470 con 0x7f2b080c6b80 2023-10-30T12:47:29.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.948+0000 7f2aff7fe640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ae80664d0 0x7f2ae8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:29.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.948+0000 7f2aff7fe640 1 -- 172.21.15.125:0/1309068635 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f2af80844e0 con 0x7f2b080c6b80 2023-10-30T12:47:29.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.948+0000 7f2afdd9b640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ae80664d0 0x7f2ae8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:29.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.952+0000 7f2afdd9b640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ae80664d0 0x7f2ae8068990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2b080c12f0 tx=0x7f2af0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:29.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:29.952+0000 7f2aff7fe640 1 -- 172.21.15.125:0/1309068635 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2af80515f0 con 0x7f2b080c6b80 2023-10-30T12:47:30.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:30.132+0000 7f2b0ca09640 1 -- 172.21.15.125:0/1309068635 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b08132170 con 0x7f2b080c6b80 2023-10-30T12:47:30.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:30.132+0000 7f2aff7fe640 1 -- 172.21.15.125:0/1309068635 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f2af8054c10 con 0x7f2b080c6b80 2023-10-30T12:47:30.134 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:30.135 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:25.242485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:30.135 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 92 2023-10-30T12:47:30.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:30.136+0000 7f2b0ca09640 1 -- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ae80664d0 msgr2=0x7f2ae8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:30.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:30.136+0000 7f2b0ca09640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ae80664d0 0x7f2ae8068990 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2b080c12f0 tx=0x7f2af0009040 comp rx=0 tx=0).stop 2023-10-30T12:47:30.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:30.136+0000 7f2b0ca09640 1 -- 172.21.15.125:0/1309068635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b080c6b80 msgr2=0x7f2b08157790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:30.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:30.136+0000 7f2b0ca09640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b080c6b80 0x7f2b08157790 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f2af800e930 tx=0x7f2af800edf0 comp rx=0 tx=0).stop 2023-10-30T12:47:30.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:30.136+0000 7f2b0ca09640 1 -- 172.21.15.125:0/1309068635 shutdown_connections 2023-10-30T12:47:30.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:30.136+0000 7f2b0ca09640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ae80664d0 0x7f2ae8068990 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:30.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:30.136+0000 7f2b0ca09640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2b0813f4a0 0x7f2b080ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:30.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:30.136+0000 7f2b0ca09640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2b080c6f60 0x7f2b080c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:30.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:30.136+0000 7f2b0ca09640 1 --2- 172.21.15.125:0/1309068635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2b080c6b80 0x7f2b08157790 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:30.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:30.136+0000 7f2b0ca09640 1 -- 172.21.15.125:0/1309068635 >> 172.21.15.125:0/1309068635 conn(0x7f2b080c7910 msgr2=0x7f2b080c1340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:30.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:30.136+0000 7f2b0ca09640 1 -- 172.21.15.125:0/1309068635 shutdown_connections 2023-10-30T12:47:30.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:30.136+0000 7f2b0ca09640 1 -- 172.21.15.125:0/1309068635 wait complete. 2023-10-30T12:47:30.155 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:47:32.156 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:32.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.300+0000 7f97193d3640 1 Processor -- start 2023-10-30T12:47:32.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.300+0000 7f97193d3640 1 -- start start 2023-10-30T12:47:32.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.300+0000 7f97193d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f97140c6b80 0x7f97140bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:32.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.300+0000 7f97193d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f971413f4a0 0x7f97140c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:32.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.300+0000 7f97193d3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97140c0a50 con 0x7f97140c6f60 2023-10-30T12:47:32.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.300+0000 7f97193d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f971406f6e0 con 0x7f971413f4a0 2023-10-30T12:47:32.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.300+0000 7f97193d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f971406f9c0 con 0x7f97140c6b80 2023-10-30T12:47:32.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.300+0000 7f97137fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f971413f4a0 0x7f97140c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:32.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.300+0000 7f97137fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f971413f4a0 0x7f97140c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42534/0 (socket says 172.21.15.125:42534) 2023-10-30T12:47:32.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.300+0000 7f97137fe640 1 -- 172.21.15.125:0/202456198 learned_addr learned my addr 172.21.15.125:0/202456198 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:32.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.300+0000 7f970bfff640 1 --1- 172.21.15.125:0/202456198 >> v1:172.21.15.125:6789/0 conn(0x7f97140c6f60 0x7f971413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:49264/0 (socket says 172.21.15.125:49264) 2023-10-30T12:47:32.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.304+0000 7f9712ffd640 1 --2- 172.21.15.125:0/202456198 >> v2:172.21.15.136:3300/0 conn(0x7f97140c6b80 0x7f97140bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:32.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.304+0000 7f97137fe640 1 -- 172.21.15.125:0/202456198 >> v2:172.21.15.136:3300/0 conn(0x7f97140c6b80 msgr2=0x7f97140bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:32.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.304+0000 7f97137fe640 1 --2- 172.21.15.125:0/202456198 >> v2:172.21.15.136:3300/0 conn(0x7f97140c6b80 0x7f97140bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:32.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.304+0000 7f97137fe640 1 -- 172.21.15.125:0/202456198 >> v1:172.21.15.125:6789/0 conn(0x7f97140c6f60 legacy=0x7f971413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:32.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.304+0000 7f97127fc640 1 -- 172.21.15.125:0/202456198 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019111291 0 0) 0x7f97140c0a50 con 0x7f97140c6f60 2023-10-30T12:47:32.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.304+0000 7f97137fe640 1 -- 172.21.15.125:0/202456198 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971406fca0 con 0x7f971413f4a0 2023-10-30T12:47:32.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.308+0000 7f97137fe640 1 --2- 172.21.15.125:0/202456198 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f971413f4a0 0x7f97140c0400 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f971406f1b0 tx=0x7f970c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=873c64bb7f0366f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:32.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.308+0000 7f97127fc640 1 -- 172.21.15.125:0/202456198 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c0a57a0 con 0x7f971413f4a0 2023-10-30T12:47:32.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.308+0000 7f97127fc640 1 -- 172.21.15.125:0/202456198 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970c0a5900 con 0x7f971413f4a0 2023-10-30T12:47:32.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.308+0000 7f97127fc640 1 -- 172.21.15.125:0/202456198 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c0a5c90 con 0x7f971413f4a0 2023-10-30T12:47:32.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.308+0000 7f97193d3640 1 -- 172.21.15.125:0/202456198 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f971413f4a0 msgr2=0x7f97140c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:32.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.308+0000 7f97193d3640 1 --2- 172.21.15.125:0/202456198 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f971413f4a0 0x7f97140c0400 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f971406f1b0 tx=0x7f970c01c9d0 comp rx=0 tx=0).stop 2023-10-30T12:47:32.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.308+0000 7f97193d3640 1 -- 172.21.15.125:0/202456198 shutdown_connections 2023-10-30T12:47:32.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.308+0000 7f97193d3640 1 --2- 172.21.15.125:0/202456198 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f971413f4a0 0x7f97140c0400 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:32.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.308+0000 7f97193d3640 1 --2- 172.21.15.125:0/202456198 >> v2:172.21.15.136:3300/0 conn(0x7f97140c6b80 0x7f97140bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:32.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.308+0000 7f97193d3640 1 -- 172.21.15.125:0/202456198 >> 172.21.15.125:0/202456198 conn(0x7f97140c7910 msgr2=0x7f97140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:32.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.308+0000 7f97193d3640 1 -- 172.21.15.125:0/202456198 shutdown_connections 2023-10-30T12:47:32.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.308+0000 7f97193d3640 1 -- 172.21.15.125:0/202456198 wait complete. 2023-10-30T12:47:32.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.308+0000 7f97193d3640 1 Processor -- start 2023-10-30T12:47:32.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f97193d3640 1 -- start start 2023-10-30T12:47:32.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f97193d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97140c6b80 0x7f97141f5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:32.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f97193d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f97140c6f60 0x7f9714157960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:32.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f97193d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f971413f4a0 0x7f9714153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:32.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f97193d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97141594a0 con 0x7f97140c6b80 2023-10-30T12:47:32.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f97193d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971415fea0 con 0x7f971413f4a0 2023-10-30T12:47:32.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f97193d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9714135720 con 0x7f97140c6f60 2023-10-30T12:47:32.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f9712ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97140c6b80 0x7f97141f5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:32.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f9712ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97140c6b80 0x7f97141f5b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42540/0 (socket says 172.21.15.125:42540) 2023-10-30T12:47:32.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f9712ffd640 1 -- 172.21.15.125:0/1926941917 learned_addr learned my addr 172.21.15.125:0/1926941917 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:32.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f970bfff640 1 --2- 172.21.15.125:0/1926941917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f97140c6f60 0x7f9714157960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:32.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f97137fe640 1 --2- 172.21.15.125:0/1926941917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f971413f4a0 0x7f9714153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:32.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f9712ffd640 1 -- 172.21.15.125:0/1926941917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f97140c6f60 msgr2=0x7f9714157960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:32.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f9712ffd640 1 --2- 172.21.15.125:0/1926941917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f97140c6f60 0x7f9714157960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:32.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f9712ffd640 1 -- 172.21.15.125:0/1926941917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f971413f4a0 msgr2=0x7f9714153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:32.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f9712ffd640 1 --2- 172.21.15.125:0/1926941917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f971413f4a0 0x7f9714153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:32.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f9712ffd640 1 -- 172.21.15.125:0/1926941917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f970c009b30 con 0x7f97140c6b80 2023-10-30T12:47:32.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f9712ffd640 1 --2- 172.21.15.125:0/1926941917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97140c6b80 0x7f97141f5b70 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f970400e930 tx=0x7f970400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:32.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f970b7fe640 1 -- 172.21.15.125:0/1926941917 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9704009690 con 0x7f97140c6b80 2023-10-30T12:47:32.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f970b7fe640 1 -- 172.21.15.125:0/1926941917 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f970400ca00 con 0x7f97140c6b80 2023-10-30T12:47:32.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f970b7fe640 1 -- 172.21.15.125:0/1926941917 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970400cd70 con 0x7f97140c6b80 2023-10-30T12:47:32.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f97193d3640 1 -- 172.21.15.125:0/1926941917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971415f8e0 con 0x7f97140c6b80 2023-10-30T12:47:32.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f97193d3640 1 -- 172.21.15.125:0/1926941917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97141f5370 con 0x7f97140c6b80 2023-10-30T12:47:32.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.312+0000 7f97193d3640 1 -- 172.21.15.125:0/1926941917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f971404e170 con 0x7f97140c6b80 2023-10-30T12:47:32.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.316+0000 7f970b7fe640 1 -- 172.21.15.125:0/1926941917 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9704013470 con 0x7f97140c6b80 2023-10-30T12:47:32.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.316+0000 7f970b7fe640 1 --2- 172.21.15.125:0/1926941917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f96f40664d0 0x7f96f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:32.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.316+0000 7f970b7fe640 1 -- 172.21.15.125:0/1926941917 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f97040843d0 con 0x7f97140c6b80 2023-10-30T12:47:32.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.316+0000 7f970bfff640 1 --2- 172.21.15.125:0/1926941917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f96f40664d0 0x7f96f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:32.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.316+0000 7f970bfff640 1 --2- 172.21.15.125:0/1926941917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f96f40664d0 0x7f96f4068990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f97140b0860 tx=0x7f96fc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:32.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.324+0000 7f970b7fe640 1 -- 172.21.15.125:0/1926941917 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97040515f0 con 0x7f97140c6b80 2023-10-30T12:47:32.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.516+0000 7f97193d3640 1 -- 172.21.15.125:0/1926941917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97140c0f80 con 0x7f97140c6b80 2023-10-30T12:47:32.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.516+0000 7f970b7fe640 1 -- 172.21.15.125:0/1926941917 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+5736 (secure 0 0 0) 0x7f9704054c10 con 0x7f97140c6b80 2023-10-30T12:47:32.517 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:32.518 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:25.242485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:32.518 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 92 2023-10-30T12:47:32.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.516+0000 7f97193d3640 1 -- 172.21.15.125:0/1926941917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f96f40664d0 msgr2=0x7f96f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:32.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.516+0000 7f97193d3640 1 --2- 172.21.15.125:0/1926941917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f96f40664d0 0x7f96f4068990 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f97140b0860 tx=0x7f96fc009040 comp rx=0 tx=0).stop 2023-10-30T12:47:32.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.516+0000 7f97193d3640 1 -- 172.21.15.125:0/1926941917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97140c6b80 msgr2=0x7f97141f5b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:32.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.516+0000 7f97193d3640 1 --2- 172.21.15.125:0/1926941917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97140c6b80 0x7f97141f5b70 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f970400e930 tx=0x7f970400edf0 comp rx=0 tx=0).stop 2023-10-30T12:47:32.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.516+0000 7f97193d3640 1 -- 172.21.15.125:0/1926941917 shutdown_connections 2023-10-30T12:47:32.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.516+0000 7f97193d3640 1 --2- 172.21.15.125:0/1926941917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f96f40664d0 0x7f96f4068990 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:32.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.516+0000 7f97193d3640 1 --2- 172.21.15.125:0/1926941917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f971413f4a0 0x7f9714153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:32.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.516+0000 7f97193d3640 1 --2- 172.21.15.125:0/1926941917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f97140c6f60 0x7f9714157960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:32.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.516+0000 7f97193d3640 1 --2- 172.21.15.125:0/1926941917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97140c6b80 0x7f97141f5b70 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:32.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.516+0000 7f97193d3640 1 -- 172.21.15.125:0/1926941917 >> 172.21.15.125:0/1926941917 conn(0x7f97140c7910 msgr2=0x7f97140c01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:32.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.520+0000 7f97193d3640 1 -- 172.21.15.125:0/1926941917 shutdown_connections 2023-10-30T12:47:32.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:32.520+0000 7f97193d3640 1 -- 172.21.15.125:0/1926941917 wait complete. 2023-10-30T12:47:32.546 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:47:34.548 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:34.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f418c91d640 1 Processor -- start 2023-10-30T12:47:34.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f418c91d640 1 -- start start 2023-10-30T12:47:34.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f418c91d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f41881538a0 0x7f418815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:34.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f418c91d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41880cf160 0x7f41880cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:34.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f418c91d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41880cb800 con 0x7f41880ca0e0 2023-10-30T12:47:34.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f418c91d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f418815dc30 con 0x7f41881538a0 2023-10-30T12:47:34.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f418c91d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41880cb330 con 0x7f41880cf160 2023-10-30T12:47:34.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f418659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f41881538a0 0x7f418815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:34.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f417ffff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f41880ca0e0 0x7f418815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:49268/0 (socket says 172.21.15.125:49268) 2023-10-30T12:47:34.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f417ffff640 1 -- 172.21.15.125:0/3067008124 learned_addr learned my addr 172.21.15.125:0/3067008124 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:47:34.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f4186d9d640 1 --2- 172.21.15.125:0/3067008124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41880cf160 0x7f41880cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:34.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f418659c640 1 -- 172.21.15.125:0/3067008124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41880cf160 msgr2=0x7f41880cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:34.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f418659c640 1 --2- 172.21.15.125:0/3067008124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41880cf160 0x7f41880cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:34.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f418659c640 1 -- 172.21.15.125:0/3067008124 >> v1:172.21.15.125:6789/0 conn(0x7f41880ca0e0 legacy=0x7f418815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:34.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f418659c640 1 -- 172.21.15.125:0/3067008124 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41880cad70 con 0x7f41881538a0 2023-10-30T12:47:34.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f4185d9b640 1 -- 172.21.15.125:0/3067008124 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098595199 0 0) 0x7f41880cb800 con 0x7f41880ca0e0 2023-10-30T12:47:34.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.692+0000 7f4186d9d640 1 --2- 172.21.15.125:0/3067008124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41880cf160 0x7f41880cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:47:34.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.696+0000 7f418659c640 1 --2- 172.21.15.125:0/3067008124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41881538a0 0x7f418815d6f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f41780097c0 tx=0x7f417801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb45aac348c53ca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:34.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.696+0000 7f4185d9b640 1 -- 172.21.15.125:0/3067008124 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41780a8970 con 0x7f41881538a0 2023-10-30T12:47:34.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.696+0000 7f4185d9b640 1 -- 172.21.15.125:0/3067008124 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41780a8ad0 con 0x7f41881538a0 2023-10-30T12:47:34.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.696+0000 7f4185d9b640 1 -- 172.21.15.125:0/3067008124 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41780a8e60 con 0x7f41881538a0 2023-10-30T12:47:34.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.696+0000 7f418c91d640 1 -- 172.21.15.125:0/3067008124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41881538a0 msgr2=0x7f418815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:34.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.696+0000 7f418c91d640 1 --2- 172.21.15.125:0/3067008124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41881538a0 0x7f418815d6f0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f41780097c0 tx=0x7f417801ec70 comp rx=0 tx=0).stop 2023-10-30T12:47:34.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.696+0000 7f418c91d640 1 -- 172.21.15.125:0/3067008124 shutdown_connections 2023-10-30T12:47:34.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.696+0000 7f418c91d640 1 --2- 172.21.15.125:0/3067008124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41880cf160 0x7f41880cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:34.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.696+0000 7f418c91d640 1 --2- 172.21.15.125:0/3067008124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41881538a0 0x7f418815d6f0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:34.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.696+0000 7f418c91d640 1 -- 172.21.15.125:0/3067008124 >> 172.21.15.125:0/3067008124 conn(0x7f418806bfe0 msgr2=0x7f418806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:34.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.696+0000 7f418c91d640 1 -- 172.21.15.125:0/3067008124 shutdown_connections 2023-10-30T12:47:34.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.696+0000 7f418c91d640 1 -- 172.21.15.125:0/3067008124 wait complete. 2023-10-30T12:47:34.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f418c91d640 1 Processor -- start 2023-10-30T12:47:34.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f418c91d640 1 -- start start 2023-10-30T12:47:34.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f418c91d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41880ca0e0 0x7f4188157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:34.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f418c91d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41880cf160 0x7f4188154000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:34.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f418c91d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41881538a0 0x7f41881e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:34.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f418c91d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4188157020 con 0x7f41881538a0 2023-10-30T12:47:34.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f418c91d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41881efb90 con 0x7f41880ca0e0 2023-10-30T12:47:34.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f418c91d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4188159630 con 0x7f41880cf160 2023-10-30T12:47:34.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f4186d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41881538a0 0x7f41881e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:34.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f4186d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41881538a0 0x7f41881e4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42572/0 (socket says 172.21.15.125:42572) 2023-10-30T12:47:34.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f4186d9d640 1 -- 172.21.15.125:0/3340959889 learned_addr learned my addr 172.21.15.125:0/3340959889 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:34.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f417ffff640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41880cf160 0x7f4188154000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:34.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f418659c640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41880ca0e0 0x7f4188157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:34.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f4186d9d640 1 -- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41880cf160 msgr2=0x7f4188154000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:34.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f4186d9d640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41880cf160 0x7f4188154000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:34.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f4186d9d640 1 -- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41880ca0e0 msgr2=0x7f4188157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:34.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f4186d9d640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41880ca0e0 0x7f4188157c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:34.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f4186d9d640 1 -- 172.21.15.125:0/3340959889 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4178009420 con 0x7f41881538a0 2023-10-30T12:47:34.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f417ffff640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41880cf160 0x7f4188154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:34.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f418659c640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41880ca0e0 0x7f4188157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:34.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f4186d9d640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41881538a0 0x7f41881e4b00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f418000ce40 tx=0x7f418000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:34.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f417effd640 1 -- 172.21.15.125:0/3340959889 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418000d840 con 0x7f41881538a0 2023-10-30T12:47:34.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f417effd640 1 -- 172.21.15.125:0/3340959889 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418000d9a0 con 0x7f41881538a0 2023-10-30T12:47:34.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f417effd640 1 -- 172.21.15.125:0/3340959889 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418000dd50 con 0x7f41881538a0 2023-10-30T12:47:34.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f418c91d640 1 -- 172.21.15.125:0/3340959889 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4188156b90 con 0x7f41881538a0 2023-10-30T12:47:34.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.700+0000 7f418c91d640 1 -- 172.21.15.125:0/3340959889 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41881590b0 con 0x7f41881538a0 2023-10-30T12:47:34.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.704+0000 7f418c91d640 1 -- 172.21.15.125:0/3340959889 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41880b2a50 con 0x7f41881538a0 2023-10-30T12:47:34.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.704+0000 7f417effd640 1 -- 172.21.15.125:0/3340959889 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f418001e3b0 con 0x7f41881538a0 2023-10-30T12:47:34.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.704+0000 7f417effd640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41680664a0 0x7f4168068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:34.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.704+0000 7f417effd640 1 -- 172.21.15.125:0/3340959889 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f4180084b70 con 0x7f41881538a0 2023-10-30T12:47:34.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.704+0000 7f418659c640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41680664a0 0x7f4168068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:34.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.704+0000 7f418659c640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41680664a0 0x7f4168068960 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f41780b2000 tx=0x7f41780aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:34.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.708+0000 7f417effd640 1 -- 172.21.15.125:0/3340959889 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4180051cd0 con 0x7f41881538a0 2023-10-30T12:47:34.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.888+0000 7f418c91d640 1 -- 172.21.15.125:0/3340959889 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4188156d20 con 0x7f41881538a0 2023-10-30T12:47:34.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.888+0000 7f417effd640 1 -- 172.21.15.125:0/3340959889 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f418001a020 con 0x7f41881538a0 2023-10-30T12:47:34.891 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:34.891 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:33.590499+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:resolve","state_seq":10,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:34.891 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 93 2023-10-30T12:47:34.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.892+0000 7f418c91d640 1 -- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41680664a0 msgr2=0x7f4168068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:34.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.892+0000 7f418c91d640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41680664a0 0x7f4168068960 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f41780b2000 tx=0x7f41780aa070 comp rx=0 tx=0).stop 2023-10-30T12:47:34.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.892+0000 7f418c91d640 1 -- 172.21.15.125:0/3340959889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41881538a0 msgr2=0x7f41881e4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:34.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.892+0000 7f418c91d640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41881538a0 0x7f41881e4b00 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f418000ce40 tx=0x7f418000a710 comp rx=0 tx=0).stop 2023-10-30T12:47:34.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.892+0000 7f418c91d640 1 -- 172.21.15.125:0/3340959889 shutdown_connections 2023-10-30T12:47:34.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.892+0000 7f418c91d640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41680664a0 0x7f4168068960 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:34.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.892+0000 7f418c91d640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f41881538a0 0x7f41881e4b00 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:34.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.892+0000 7f418c91d640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f41880cf160 0x7f4188154000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:34.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.892+0000 7f418c91d640 1 --2- 172.21.15.125:0/3340959889 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f41880ca0e0 0x7f4188157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:34.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.892+0000 7f418c91d640 1 -- 172.21.15.125:0/3340959889 >> 172.21.15.125:0/3340959889 conn(0x7f418806bfe0 msgr2=0x7f4188149180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:34.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.892+0000 7f418c91d640 1 -- 172.21.15.125:0/3340959889 shutdown_connections 2023-10-30T12:47:34.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:34.892+0000 7f418c91d640 1 -- 172.21.15.125:0/3340959889 wait complete. 2023-10-30T12:47:34.911 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:47:36.912 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:37.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.056+0000 7f655f679640 1 Processor -- start 2023-10-30T12:47:37.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.056+0000 7f655f679640 1 -- start start 2023-10-30T12:47:37.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655f679640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6558153bf0 0x7f6558071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:37.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655f679640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f65580c6fe0 0x7f65580c7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:37.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655f679640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65580cb550 0x7f65580cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:37.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655f679640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f655815d870 con 0x7f65580cb550 2023-10-30T12:47:37.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655f679640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65580c60e0 con 0x7f65580c6fe0 2023-10-30T12:47:37.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655f679640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65580c63a0 con 0x7f6558153bf0 2023-10-30T12:47:37.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655dc16640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65580cb550 0x7f65580cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:37.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655dc16640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65580cb550 0x7f65580cb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42588/0 (socket says 172.21.15.125:42588) 2023-10-30T12:47:37.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655dc16640 1 -- 172.21.15.125:0/286114686 learned_addr learned my addr 172.21.15.125:0/286114686 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:37.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655cc14640 1 --2- 172.21.15.125:0/286114686 >> v2:172.21.15.136:3300/0 conn(0x7f65580c6fe0 0x7f65580c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655d415640 1 --2- 172.21.15.125:0/286114686 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6558153bf0 0x7f6558071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655dc16640 1 -- 172.21.15.125:0/286114686 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6558153bf0 msgr2=0x7f6558071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655dc16640 1 --2- 172.21.15.125:0/286114686 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6558153bf0 0x7f6558071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655dc16640 1 -- 172.21.15.125:0/286114686 >> v2:172.21.15.136:3300/0 conn(0x7f65580c6fe0 msgr2=0x7f65580c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:37.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655dc16640 1 --2- 172.21.15.125:0/286114686 >> v2:172.21.15.136:3300/0 conn(0x7f65580c6fe0 0x7f65580c7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:37.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.060+0000 7f655dc16640 1 -- 172.21.15.125:0/286114686 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65580c6680 con 0x7f65580cb550 2023-10-30T12:47:37.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655dc16640 1 --2- 172.21.15.125:0/286114686 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65580cb550 0x7f65580cb9e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f65580c9180 tx=0x7f655001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f00474e7a973a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:37.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f654ffff640 1 -- 172.21.15.125:0/286114686 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65500a5820 con 0x7f65580cb550 2023-10-30T12:47:37.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f654ffff640 1 -- 172.21.15.125:0/286114686 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65500a5980 con 0x7f65580cb550 2023-10-30T12:47:37.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f654ffff640 1 -- 172.21.15.125:0/286114686 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65500a5d10 con 0x7f65580cb550 2023-10-30T12:47:37.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 -- 172.21.15.125:0/286114686 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65580cb550 msgr2=0x7f65580cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:37.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 --2- 172.21.15.125:0/286114686 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65580cb550 0x7f65580cb9e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f65580c9180 tx=0x7f655001c9a0 comp rx=0 tx=0).stop 2023-10-30T12:47:37.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 -- 172.21.15.125:0/286114686 shutdown_connections 2023-10-30T12:47:37.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 --2- 172.21.15.125:0/286114686 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65580cb550 0x7f65580cb9e0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:37.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 --2- 172.21.15.125:0/286114686 >> v2:172.21.15.136:3300/0 conn(0x7f65580c6fe0 0x7f65580c7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:37.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 --2- 172.21.15.125:0/286114686 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6558153bf0 0x7f6558071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:37.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 -- 172.21.15.125:0/286114686 >> 172.21.15.125:0/286114686 conn(0x7f655806c6e0 msgr2=0x7f6558069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:37.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 -- 172.21.15.125:0/286114686 shutdown_connections 2023-10-30T12:47:37.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 -- 172.21.15.125:0/286114686 wait complete. 2023-10-30T12:47:37.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 Processor -- start 2023-10-30T12:47:37.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 -- start start 2023-10-30T12:47:37.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65580c6fe0 0x7f655815c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:37.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65580cb550 0x7f65580c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:37.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6558153bf0 0x7f65580c47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:37.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65580c3d40 con 0x7f65580cb550 2023-10-30T12:47:37.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655815b330 con 0x7f6558153bf0 2023-10-30T12:47:37.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.064+0000 7f655f679640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6558147f40 con 0x7f65580c6fe0 2023-10-30T12:47:37.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f655cc14640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65580cb550 0x7f65580c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:37.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f655cc14640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65580cb550 0x7f65580c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42590/0 (socket says 172.21.15.125:42590) 2023-10-30T12:47:37.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f655cc14640 1 -- 172.21.15.125:0/4174080170 learned_addr learned my addr 172.21.15.125:0/4174080170 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:37.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f655d415640 1 --2- 172.21.15.125:0/4174080170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65580c6fe0 0x7f655815c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:37.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f655dc16640 1 --2- 172.21.15.125:0/4174080170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6558153bf0 0x7f65580c47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:37.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f655cc14640 1 -- 172.21.15.125:0/4174080170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65580c6fe0 msgr2=0x7f655815c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:37.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f655cc14640 1 --2- 172.21.15.125:0/4174080170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65580c6fe0 0x7f655815c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:37.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f655cc14640 1 -- 172.21.15.125:0/4174080170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6558153bf0 msgr2=0x7f65580c47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:37.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f655cc14640 1 --2- 172.21.15.125:0/4174080170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6558153bf0 0x7f65580c47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:37.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f655cc14640 1 -- 172.21.15.125:0/4174080170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6550009b30 con 0x7f65580cb550 2023-10-30T12:47:37.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f655cc14640 1 --2- 172.21.15.125:0/4174080170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65580cb550 0x7f65580c42a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f654800d7d0 tx=0x7f654800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:37.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f654dffb640 1 -- 172.21.15.125:0/4174080170 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6548004020 con 0x7f65580cb550 2023-10-30T12:47:37.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f654dffb640 1 -- 172.21.15.125:0/4174080170 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6548004180 con 0x7f65580cb550 2023-10-30T12:47:37.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f655f679640 1 -- 172.21.15.125:0/4174080170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6558152f30 con 0x7f65580cb550 2023-10-30T12:47:37.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f654dffb640 1 -- 172.21.15.125:0/4174080170 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6548004330 con 0x7f65580cb550 2023-10-30T12:47:37.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f655f679640 1 -- 172.21.15.125:0/4174080170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655815b5d0 con 0x7f65580cb550 2023-10-30T12:47:37.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f655f679640 1 -- 172.21.15.125:0/4174080170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f655804e170 con 0x7f65580cb550 2023-10-30T12:47:37.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f654dffb640 1 -- 172.21.15.125:0/4174080170 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f65480096a0 con 0x7f65580cb550 2023-10-30T12:47:37.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.068+0000 7f654dffb640 1 --2- 172.21.15.125:0/4174080170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f653c0664a0 0x7f653c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:37.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.072+0000 7f654dffb640 1 -- 172.21.15.125:0/4174080170 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f65480843c0 con 0x7f65580cb550 2023-10-30T12:47:37.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.072+0000 7f655d415640 1 --2- 172.21.15.125:0/4174080170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f653c0664a0 0x7f653c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:37.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.072+0000 7f655d415640 1 --2- 172.21.15.125:0/4174080170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f653c0664a0 0x7f653c068960 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f65540079c0 tx=0x7f6554008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:37.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.076+0000 7f654dffb640 1 -- 172.21.15.125:0/4174080170 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6548051550 con 0x7f65580cb550 2023-10-30T12:47:37.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.256+0000 7f655f679640 1 -- 172.21.15.125:0/4174080170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6558151200 con 0x7f65580cb550 2023-10-30T12:47:37.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.256+0000 7f654dffb640 1 -- 172.21.15.125:0/4174080170 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5738 (secure 0 0 0) 0x7f6548054b70 con 0x7f65580cb550 2023-10-30T12:47:37.260 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:37.260 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:33.590499+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:resolve","state_seq":10,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:37.260 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 93 2023-10-30T12:47:37.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.260+0000 7f655f679640 1 -- 172.21.15.125:0/4174080170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f653c0664a0 msgr2=0x7f653c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:37.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.260+0000 7f655f679640 1 --2- 172.21.15.125:0/4174080170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f653c0664a0 0x7f653c068960 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f65540079c0 tx=0x7f6554008040 comp rx=0 tx=0).stop 2023-10-30T12:47:37.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.260+0000 7f655f679640 1 -- 172.21.15.125:0/4174080170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65580cb550 msgr2=0x7f65580c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:37.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.260+0000 7f655f679640 1 --2- 172.21.15.125:0/4174080170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65580cb550 0x7f65580c42a0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f654800d7d0 tx=0x7f654800dc90 comp rx=0 tx=0).stop 2023-10-30T12:47:37.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.260+0000 7f655f679640 1 -- 172.21.15.125:0/4174080170 shutdown_connections 2023-10-30T12:47:37.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.260+0000 7f655f679640 1 --2- 172.21.15.125:0/4174080170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f653c0664a0 0x7f653c068960 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:37.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.260+0000 7f655f679640 1 --2- 172.21.15.125:0/4174080170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6558153bf0 0x7f65580c47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:37.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.260+0000 7f655f679640 1 --2- 172.21.15.125:0/4174080170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65580cb550 0x7f65580c42a0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:37.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.260+0000 7f655f679640 1 --2- 172.21.15.125:0/4174080170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65580c6fe0 0x7f655815c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:37.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.260+0000 7f655f679640 1 -- 172.21.15.125:0/4174080170 >> 172.21.15.125:0/4174080170 conn(0x7f655806c6e0 msgr2=0x7f655814a160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:37.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.260+0000 7f655f679640 1 -- 172.21.15.125:0/4174080170 shutdown_connections 2023-10-30T12:47:37.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:37.260+0000 7f655f679640 1 -- 172.21.15.125:0/4174080170 wait complete. 2023-10-30T12:47:37.280 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:47:39.281 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:39.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd67146b640 1 Processor -- start 2023-10-30T12:47:39.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd67146b640 1 -- start start 2023-10-30T12:47:39.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd67146b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd66c1538a0 0x7fd66c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:39.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd67146b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd66c0ca0e0 0x7fd66c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:39.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd67146b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd66c0cf160 0x7fd66c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:39.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd67146b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66c0cb800 con 0x7fd66c1538a0 2023-10-30T12:47:39.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd67146b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd66c15dc30 con 0x7fd66c0ca0e0 2023-10-30T12:47:39.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd67146b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0cb330 con 0x7fd66c0cf160 2023-10-30T12:47:39.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd66affd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd66c1538a0 0x7fd66c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:39.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd66affd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd66c1538a0 0x7fd66c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42596/0 (socket says 172.21.15.125:42596) 2023-10-30T12:47:39.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd66affd640 1 -- 172.21.15.125:0/1111969363 learned_addr learned my addr 172.21.15.125:0/1111969363 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:39.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd66b7fe640 1 --2- 172.21.15.125:0/1111969363 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd66c0cf160 0x7fd66c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:39.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd6627fc640 1 --2- 172.21.15.125:0/1111969363 >> v2:172.21.15.136:3300/0 conn(0x7fd66c0ca0e0 0x7fd66c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:39.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd66affd640 1 -- 172.21.15.125:0/1111969363 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd66c0cf160 msgr2=0x7fd66c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:39.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd66affd640 1 --2- 172.21.15.125:0/1111969363 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd66c0cf160 0x7fd66c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:39.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd66affd640 1 -- 172.21.15.125:0/1111969363 >> v2:172.21.15.136:3300/0 conn(0x7fd66c0ca0e0 msgr2=0x7fd66c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:39.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd66affd640 1 --2- 172.21.15.125:0/1111969363 >> v2:172.21.15.136:3300/0 conn(0x7fd66c0ca0e0 0x7fd66c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:39.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd66affd640 1 -- 172.21.15.125:0/1111969363 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd66c0cad70 con 0x7fd66c1538a0 2023-10-30T12:47:39.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.428+0000 7fd6627fc640 1 --2- 172.21.15.125:0/1111969363 >> v2:172.21.15.136:3300/0 conn(0x7fd66c0ca0e0 0x7fd66c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:47:39.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.432+0000 7fd66affd640 1 --2- 172.21.15.125:0/1111969363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd66c1538a0 0x7fd66c15d6f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0b30a0 tx=0x7fd65c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59285c036cc1c590 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:39.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.432+0000 7fd66a7fc640 1 -- 172.21.15.125:0/1111969363 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c0a89b0 con 0x7fd66c1538a0 2023-10-30T12:47:39.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.432+0000 7fd66a7fc640 1 -- 172.21.15.125:0/1111969363 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65c0a8b10 con 0x7fd66c1538a0 2023-10-30T12:47:39.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.432+0000 7fd66a7fc640 1 -- 172.21.15.125:0/1111969363 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd65c0a8ea0 con 0x7fd66c1538a0 2023-10-30T12:47:39.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.432+0000 7fd67146b640 1 -- 172.21.15.125:0/1111969363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd66c1538a0 msgr2=0x7fd66c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:39.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.432+0000 7fd67146b640 1 --2- 172.21.15.125:0/1111969363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd66c1538a0 0x7fd66c15d6f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fd66c0b30a0 tx=0x7fd65c01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:47:39.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.432+0000 7fd67146b640 1 -- 172.21.15.125:0/1111969363 shutdown_connections 2023-10-30T12:47:39.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.432+0000 7fd67146b640 1 --2- 172.21.15.125:0/1111969363 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd66c0cf160 0x7fd66c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:39.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.432+0000 7fd67146b640 1 --2- 172.21.15.125:0/1111969363 >> v2:172.21.15.136:3300/0 conn(0x7fd66c0ca0e0 0x7fd66c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:39.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.432+0000 7fd67146b640 1 --2- 172.21.15.125:0/1111969363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd66c1538a0 0x7fd66c15d6f0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:39.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.432+0000 7fd67146b640 1 -- 172.21.15.125:0/1111969363 >> 172.21.15.125:0/1111969363 conn(0x7fd66c06bfe0 msgr2=0x7fd66c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:39.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.432+0000 7fd67146b640 1 -- 172.21.15.125:0/1111969363 shutdown_connections 2023-10-30T12:47:39.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.432+0000 7fd67146b640 1 -- 172.21.15.125:0/1111969363 wait complete. 2023-10-30T12:47:39.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd67146b640 1 Processor -- start 2023-10-30T12:47:39.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd67146b640 1 -- start start 2023-10-30T12:47:39.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd67146b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd66c0ca0e0 0x7fd66c0b4c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:39.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd67146b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd66c0cf160 0x7fd66c158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:39.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd67146b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd66c1538a0 0x7fd66c148ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:39.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd67146b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0ce540 con 0x7fd66c1538a0 2023-10-30T12:47:39.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd67146b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0ccfb0 con 0x7fd66c0ca0e0 2023-10-30T12:47:39.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd67146b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd66c0ce7b0 con 0x7fd66c0cf160 2023-10-30T12:47:39.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd66b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd66c1538a0 0x7fd66c148ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:39.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd66b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd66c1538a0 0x7fd66c148ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42602/0 (socket says 172.21.15.125:42602) 2023-10-30T12:47:39.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd66b7fe640 1 -- 172.21.15.125:0/4247189378 learned_addr learned my addr 172.21.15.125:0/4247189378 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:39.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd66affd640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd66c0ca0e0 0x7fd66c0b4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:39.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd6627fc640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd66c0cf160 0x7fd66c158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:39.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd66b7fe640 1 -- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd66c0cf160 msgr2=0x7fd66c158aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:39.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd66b7fe640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd66c0cf160 0x7fd66c158aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:39.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd66b7fe640 1 -- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd66c0ca0e0 msgr2=0x7fd66c0b4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:39.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd66b7fe640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd66c0ca0e0 0x7fd66c0b4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:39.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd66b7fe640 1 -- 172.21.15.125:0/4247189378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65c0094d0 con 0x7fd66c1538a0 2023-10-30T12:47:39.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd6627fc640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd66c0cf160 0x7fd66c158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:39.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd66affd640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd66c0ca0e0 0x7fd66c0b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:39.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.436+0000 7fd66b7fe640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd66c1538a0 0x7fd66c148ed0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fd66400d5b0 tx=0x7fd66400da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:39.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.440+0000 7fd663fff640 1 -- 172.21.15.125:0/4247189378 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6640125f0 con 0x7fd66c1538a0 2023-10-30T12:47:39.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.440+0000 7fd663fff640 1 -- 172.21.15.125:0/4247189378 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd664012750 con 0x7fd66c1538a0 2023-10-30T12:47:39.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.440+0000 7fd663fff640 1 -- 172.21.15.125:0/4247189378 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd664012b00 con 0x7fd66c1538a0 2023-10-30T12:47:39.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.440+0000 7fd67146b640 1 -- 172.21.15.125:0/4247189378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd66c0cea90 con 0x7fd66c1538a0 2023-10-30T12:47:39.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.440+0000 7fd67146b640 1 -- 172.21.15.125:0/4247189378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd66c0b31a0 con 0x7fd66c1538a0 2023-10-30T12:47:39.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.440+0000 7fd67146b640 1 -- 172.21.15.125:0/4247189378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd66c0b2a50 con 0x7fd66c1538a0 2023-10-30T12:47:39.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.440+0000 7fd663fff640 1 -- 172.21.15.125:0/4247189378 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd664012c60 con 0x7fd66c1538a0 2023-10-30T12:47:39.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.444+0000 7fd663fff640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd64c0664f0 0x7fd64c0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:39.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.444+0000 7fd663fff640 1 -- 172.21.15.125:0/4247189378 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7fd6640845a0 con 0x7fd66c1538a0 2023-10-30T12:47:39.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.444+0000 7fd66affd640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd64c0664f0 0x7fd64c0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:39.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.444+0000 7fd66affd640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd64c0664f0 0x7fd64c0689b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0b2000 tx=0x7fd65c0aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:39.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.448+0000 7fd663fff640 1 -- 172.21.15.125:0/4247189378 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd664051790 con 0x7fd66c1538a0 2023-10-30T12:47:39.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.632+0000 7fd67146b640 1 -- 172.21.15.125:0/4247189378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd66c04e170 con 0x7fd66c1538a0 2023-10-30T12:47:39.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.632+0000 7fd663fff640 1 -- 172.21.15.125:0/4247189378 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+5738 (secure 0 0 0) 0x7fd664054db0 con 0x7fd66c1538a0 2023-10-30T12:47:39.633 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:39.633 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:33.590499+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:resolve","state_seq":10,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:39.634 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 93 2023-10-30T12:47:39.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.632+0000 7fd67146b640 1 -- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd64c0664f0 msgr2=0x7fd64c0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:39.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.632+0000 7fd67146b640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd64c0664f0 0x7fd64c0689b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd65c0b2000 tx=0x7fd65c0aa070 comp rx=0 tx=0).stop 2023-10-30T12:47:39.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.632+0000 7fd67146b640 1 -- 172.21.15.125:0/4247189378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd66c1538a0 msgr2=0x7fd66c148ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:39.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.632+0000 7fd67146b640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd66c1538a0 0x7fd66c148ed0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fd66400d5b0 tx=0x7fd66400da70 comp rx=0 tx=0).stop 2023-10-30T12:47:39.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.632+0000 7fd67146b640 1 -- 172.21.15.125:0/4247189378 shutdown_connections 2023-10-30T12:47:39.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.632+0000 7fd67146b640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd64c0664f0 0x7fd64c0689b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:39.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.632+0000 7fd67146b640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd66c1538a0 0x7fd66c148ed0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:39.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.632+0000 7fd67146b640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd66c0cf160 0x7fd66c158aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:39.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.632+0000 7fd67146b640 1 --2- 172.21.15.125:0/4247189378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd66c0ca0e0 0x7fd66c0b4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:39.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.632+0000 7fd67146b640 1 -- 172.21.15.125:0/4247189378 >> 172.21.15.125:0/4247189378 conn(0x7fd66c06bfe0 msgr2=0x7fd66c066410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:39.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.632+0000 7fd67146b640 1 -- 172.21.15.125:0/4247189378 shutdown_connections 2023-10-30T12:47:39.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:39.632+0000 7fd67146b640 1 -- 172.21.15.125:0/4247189378 wait complete. 2023-10-30T12:47:39.654 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 93, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 93, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5988": { "addr": "172.21.15.125:6837/3156415951", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 3156415951, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 3156415951, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5988, "incarnation": 78, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 28 }, "gid_6318": { "addr": "172.21.15.125:6836/1721514617", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1721514617, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1721514617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6318, "incarnation": 92, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:resolve", "state_seq": 10 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:47:33.590499+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6318, "mds_1": 5988 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:47:41.656 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:41.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.800+0000 7f5844232640 1 Processor -- start 2023-10-30T12:47:41.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f5844232640 1 -- start start 2023-10-30T12:47:41.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f5844232640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f583c0c8460 0x7f583c0715c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:41.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f5844232640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f583c070650 0x7f583c159810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:41.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f5844232640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f583c159d50 con 0x7f583c157010 2023-10-30T12:47:41.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f5844232640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f583c0c21b0 con 0x7f583c0c8460 2023-10-30T12:47:41.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f5844232640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583c0c1530 con 0x7f583c070650 2023-10-30T12:47:41.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f58417cd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f583c0c8460 0x7f583c0715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:41.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f58417cd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f583c0c8460 0x7f583c0715c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37846/0 (socket says 172.21.15.125:37846) 2023-10-30T12:47:41.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f58417cd640 1 -- 172.21.15.125:0/306991680 learned_addr learned my addr 172.21.15.125:0/306991680 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:41.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f58427cf640 1 --2- 172.21.15.125:0/306991680 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f583c070650 0x7f583c159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:41.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f58417cd640 1 -- 172.21.15.125:0/306991680 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f583c070650 msgr2=0x7f583c159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:41.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f58417cd640 1 --2- 172.21.15.125:0/306991680 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f583c070650 0x7f583c159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:41.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f58417cd640 1 -- 172.21.15.125:0/306991680 >> v1:172.21.15.125:6789/0 conn(0x7f583c157010 legacy=0x7f583c071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:41.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f5840fcc640 1 -- 172.21.15.125:0/306991680 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542201456 0 0) 0x7f583c159d50 con 0x7f583c157010 2023-10-30T12:47:41.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.804+0000 7f58417cd640 1 -- 172.21.15.125:0/306991680 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c0c0f70 con 0x7f583c0c8460 2023-10-30T12:47:41.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.808+0000 7f58417cd640 1 --2- 172.21.15.125:0/306991680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f583c0c8460 0x7f583c0715c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f582c009870 tx=0x7f582c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01e0c954923ec27 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:41.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.808+0000 7f5840fcc640 1 -- 172.21.15.125:0/306991680 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c0a7070 con 0x7f583c0c8460 2023-10-30T12:47:41.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.808+0000 7f5840fcc640 1 -- 172.21.15.125:0/306991680 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f582c0ab430 con 0x7f583c0c8460 2023-10-30T12:47:41.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.808+0000 7f5840fcc640 1 -- 172.21.15.125:0/306991680 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f582c0ab7c0 con 0x7f583c0c8460 2023-10-30T12:47:41.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.808+0000 7f5844232640 1 -- 172.21.15.125:0/306991680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f583c0c8460 msgr2=0x7f583c0715c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:41.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.808+0000 7f5844232640 1 --2- 172.21.15.125:0/306991680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f583c0c8460 0x7f583c0715c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f582c009870 tx=0x7f582c01c540 comp rx=0 tx=0).stop 2023-10-30T12:47:41.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.808+0000 7f5844232640 1 -- 172.21.15.125:0/306991680 shutdown_connections 2023-10-30T12:47:41.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.808+0000 7f5844232640 1 --2- 172.21.15.125:0/306991680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f583c0c8460 0x7f583c0715c0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:41.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.808+0000 7f5844232640 1 --2- 172.21.15.125:0/306991680 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f583c070650 0x7f583c159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:41.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.808+0000 7f5844232640 1 -- 172.21.15.125:0/306991680 >> 172.21.15.125:0/306991680 conn(0x7f583c0cbeb0 msgr2=0x7f583c0cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:41.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.808+0000 7f5844232640 1 -- 172.21.15.125:0/306991680 shutdown_connections 2023-10-30T12:47:41.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.808+0000 7f5844232640 1 -- 172.21.15.125:0/306991680 wait complete. 2023-10-30T12:47:41.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.808+0000 7f5844232640 1 Processor -- start 2023-10-30T12:47:41.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.808+0000 7f5844232640 1 -- start start 2023-10-30T12:47:41.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5844232640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f583c070650 0x7f583c143990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:41.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5844232640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f583c0c8460 0x7f583c1f1a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:41.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5844232640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f583c157010 0x7f583c14ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:41.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5844232640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c0c9fa0 con 0x7f583c070650 2023-10-30T12:47:41.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5844232640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583c0cb8d0 con 0x7f583c0c8460 2023-10-30T12:47:41.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5844232640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f583c0cbbb0 con 0x7f583c157010 2023-10-30T12:47:41.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5841fce640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f583c070650 0x7f583c143990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:41.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5841fce640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f583c070650 0x7f583c143990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37850/0 (socket says 172.21.15.125:37850) 2023-10-30T12:47:41.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5841fce640 1 -- 172.21.15.125:0/1501348827 learned_addr learned my addr 172.21.15.125:0/1501348827 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:41.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f58427cf640 1 --2- 172.21.15.125:0/1501348827 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f583c157010 0x7f583c14ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:41.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f58417cd640 1 --2- 172.21.15.125:0/1501348827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f583c0c8460 0x7f583c1f1a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:41.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5841fce640 1 -- 172.21.15.125:0/1501348827 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f583c157010 msgr2=0x7f583c14ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:41.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5841fce640 1 --2- 172.21.15.125:0/1501348827 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f583c157010 0x7f583c14ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:41.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5841fce640 1 -- 172.21.15.125:0/1501348827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f583c0c8460 msgr2=0x7f583c1f1a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:41.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5841fce640 1 --2- 172.21.15.125:0/1501348827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f583c0c8460 0x7f583c1f1a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:41.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5841fce640 1 -- 172.21.15.125:0/1501348827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582c0094d0 con 0x7f583c070650 2023-10-30T12:47:41.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5841fce640 1 --2- 172.21.15.125:0/1501348827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f583c070650 0x7f583c143990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f583800c4d0 tx=0x7f583800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:41.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f58327fc640 1 -- 172.21.15.125:0/1501348827 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5838005480 con 0x7f583c070650 2023-10-30T12:47:41.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f58327fc640 1 -- 172.21.15.125:0/1501348827 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5838003600 con 0x7f583c070650 2023-10-30T12:47:41.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f58327fc640 1 -- 172.21.15.125:0/1501348827 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58380039b0 con 0x7f583c070650 2023-10-30T12:47:41.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5844232640 1 -- 172.21.15.125:0/1501348827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583c0ca260 con 0x7f583c070650 2023-10-30T12:47:41.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5844232640 1 -- 172.21.15.125:0/1501348827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583c0ca820 con 0x7f583c070650 2023-10-30T12:47:41.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f5844232640 1 -- 172.21.15.125:0/1501348827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f583c04e170 con 0x7f583c070650 2023-10-30T12:47:41.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.812+0000 7f58327fc640 1 -- 172.21.15.125:0/1501348827 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5838003b10 con 0x7f583c070650 2023-10-30T12:47:41.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.816+0000 7f58327fc640 1 --2- 172.21.15.125:0/1501348827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58200664a0 0x7f5820068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:41.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.816+0000 7f58327fc640 1 -- 172.21.15.125:0/1501348827 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f5838083800 con 0x7f583c070650 2023-10-30T12:47:41.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.816+0000 7f58417cd640 1 --2- 172.21.15.125:0/1501348827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58200664a0 0x7f5820068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:41.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.816+0000 7f58417cd640 1 --2- 172.21.15.125:0/1501348827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58200664a0 0x7f5820068960 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f582c0af000 tx=0x7f582c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:41.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:41.820+0000 7f58327fc640 1 -- 172.21.15.125:0/1501348827 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f58380509a0 con 0x7f583c070650 2023-10-30T12:47:42.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:42.000+0000 7f5844232640 1 -- 172.21.15.125:0/1501348827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f583c0c8240 con 0x7f583c070650 2023-10-30T12:47:42.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:42.000+0000 7f58327fc640 1 -- 172.21.15.125:0/1501348827 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f5838017090 con 0x7f583c070650 2023-10-30T12:47:42.001 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:42.001 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:39.615192+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:42.001 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 94 2023-10-30T12:47:42.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:42.000+0000 7f5844232640 1 -- 172.21.15.125:0/1501348827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58200664a0 msgr2=0x7f5820068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:42.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:42.000+0000 7f5844232640 1 --2- 172.21.15.125:0/1501348827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58200664a0 0x7f5820068960 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f582c0af000 tx=0x7f582c008040 comp rx=0 tx=0).stop 2023-10-30T12:47:42.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:42.000+0000 7f5844232640 1 -- 172.21.15.125:0/1501348827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f583c070650 msgr2=0x7f583c143990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:42.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:42.000+0000 7f5844232640 1 --2- 172.21.15.125:0/1501348827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f583c070650 0x7f583c143990 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f583800c4d0 tx=0x7f583800c990 comp rx=0 tx=0).stop 2023-10-30T12:47:42.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:42.000+0000 7f5844232640 1 -- 172.21.15.125:0/1501348827 shutdown_connections 2023-10-30T12:47:42.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:42.000+0000 7f5844232640 1 --2- 172.21.15.125:0/1501348827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58200664a0 0x7f5820068960 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:42.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:42.000+0000 7f5844232640 1 --2- 172.21.15.125:0/1501348827 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f583c157010 0x7f583c14ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:42.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:42.000+0000 7f5844232640 1 --2- 172.21.15.125:0/1501348827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f583c0c8460 0x7f583c1f1a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:42.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:42.000+0000 7f5844232640 1 --2- 172.21.15.125:0/1501348827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f583c070650 0x7f583c143990 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:42.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:42.000+0000 7f5844232640 1 -- 172.21.15.125:0/1501348827 >> 172.21.15.125:0/1501348827 conn(0x7f583c0cbeb0 msgr2=0x7f583c1529c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:42.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:42.000+0000 7f5844232640 1 -- 172.21.15.125:0/1501348827 shutdown_connections 2023-10-30T12:47:42.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:42.000+0000 7f5844232640 1 -- 172.21.15.125:0/1501348827 wait complete. 2023-10-30T12:47:42.021 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:47:44.022 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:44.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.168+0000 7f3371c46640 1 Processor -- start 2023-10-30T12:47:44.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.168+0000 7f3371c46640 1 -- start start 2023-10-30T12:47:44.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.168+0000 7f3371c46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f336c06f7a0 0x7f336c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:44.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f3371c46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c066cf0 con 0x7f336c0c6bd0 2023-10-30T12:47:44.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f3371c46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c0c70d0 con 0x7f336c071110 2023-10-30T12:47:44.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f3371c46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336c0700c0 con 0x7f336c06f7a0 2023-10-30T12:47:44.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f3363fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f336c0c6bd0 0x7f336c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40208/0 (socket says 172.21.15.125:40208) 2023-10-30T12:47:44.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f3363fff640 1 -- 172.21.15.125:0/163852580 learned_addr learned my addr 172.21.15.125:0/163852580 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:47:44.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336b7fe640 1 --2- 172.21.15.125:0/163852580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f336c06f7a0 0x7f336c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:44.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 -- 172.21.15.125:0/163852580 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534085410 0 0) 0x7f336c066cf0 con 0x7f336c0c6bd0 2023-10-30T12:47:44.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 -- 172.21.15.125:0/163852580 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3358003530 con 0x7f336c0c6bd0 2023-10-30T12:47:44.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 -- 172.21.15.125:0/163852580 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3795814027 0 0) 0x7f3358003530 con 0x7f336c0c6bd0 2023-10-30T12:47:44.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 -- 172.21.15.125:0/163852580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f336c06f7a0 msgr2=0x7f336c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:44.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 --2- 172.21.15.125:0/163852580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f336c06f7a0 0x7f336c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:44.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 -- 172.21.15.125:0/163852580 >> v1:172.21.15.136:6789/0 conn(0x7f336c071110 legacy=0x7f336c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:44.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 -- 172.21.15.125:0/163852580 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c0c7390 con 0x7f336c0c6bd0 2023-10-30T12:47:44.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 -- 172.21.15.125:0/163852580 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f3354001680 con 0x7f336c0c6bd0 2023-10-30T12:47:44.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 -- 172.21.15.125:0/163852580 >> v1:172.21.15.125:6789/0 conn(0x7f336c0c6bd0 legacy=0x7f336c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 --2- 172.21.15.125:0/163852580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f336c06f7a0 0x7f33580039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 --2- 172.21.15.125:0/163852580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3358003f30 0x7f33580047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 --2- 172.21.15.125:0/163852580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3358004d10 0x7f3358005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 -- 172.21.15.125:0/163852580 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c0700c0 con 0x7f3358004d10 2023-10-30T12:47:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 -- 172.21.15.125:0/163852580 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335800bb00 con 0x7f3358003f30 2023-10-30T12:47:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336affd640 1 -- 172.21.15.125:0/163852580 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335800bcc0 con 0x7f336c06f7a0 2023-10-30T12:47:44.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336bfff640 1 --2- 172.21.15.125:0/163852580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3358004d10 0x7f3358005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:44.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f3363fff640 1 --2- 172.21.15.125:0/163852580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3358003f30 0x7f33580047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:44.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336b7fe640 1 --2- 172.21.15.125:0/163852580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f336c06f7a0 0x7f33580039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:44.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336bfff640 1 -- 172.21.15.125:0/163852580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f336c06f7a0 msgr2=0x7f33580039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336bfff640 1 --2- 172.21.15.125:0/163852580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f336c06f7a0 0x7f33580039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336bfff640 1 -- 172.21.15.125:0/163852580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3358003f30 msgr2=0x7f33580047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336bfff640 1 --2- 172.21.15.125:0/163852580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3358003f30 0x7f33580047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336bfff640 1 -- 172.21.15.125:0/163852580 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3358003530 con 0x7f3358004d10 2023-10-30T12:47:44.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.172+0000 7f336b7fe640 1 --2- 172.21.15.125:0/163852580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f336c06f7a0 0x7f33580039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:44.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.176+0000 7f336bfff640 1 --2- 172.21.15.125:0/163852580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3358004d10 0x7f3358005190 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f3364001e70 tx=0x7f336401c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ece66474600eae4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:44.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.176+0000 7f336affd640 1 -- 172.21.15.125:0/163852580 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364003b80 con 0x7f3358004d10 2023-10-30T12:47:44.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.176+0000 7f336affd640 1 -- 172.21.15.125:0/163852580 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3364003ce0 con 0x7f3358004d10 2023-10-30T12:47:44.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.176+0000 7f336affd640 1 -- 172.21.15.125:0/163852580 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33640052c0 con 0x7f3358004d10 2023-10-30T12:47:44.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.176+0000 7f3371c46640 1 -- 172.21.15.125:0/163852580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3358004d10 msgr2=0x7f3358005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:44.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.176+0000 7f3371c46640 1 --2- 172.21.15.125:0/163852580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3358004d10 0x7f3358005190 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f3364001e70 tx=0x7f336401c760 comp rx=0 tx=0).stop 2023-10-30T12:47:44.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.176+0000 7f336bfff640 1 -- 172.21.15.125:0/163852580 reap_dead start 2023-10-30T12:47:44.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.176+0000 7f3371c46640 1 -- 172.21.15.125:0/163852580 shutdown_connections 2023-10-30T12:47:44.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.176+0000 7f3371c46640 1 -- 172.21.15.125:0/163852580 >> 172.21.15.125:0/163852580 conn(0x7f336c0c7910 msgr2=0x7f336c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:44.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.176+0000 7f3371c46640 1 -- 172.21.15.125:0/163852580 shutdown_connections 2023-10-30T12:47:44.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.176+0000 7f3371c46640 1 -- 172.21.15.125:0/163852580 wait complete. 2023-10-30T12:47:44.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.176+0000 7f3371c46640 1 Processor -- start 2023-10-30T12:47:44.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.176+0000 7f3371c46640 1 -- start start 2023-10-30T12:47:44.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f3371c46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3358004d10 0x7f336c146b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:44.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f3371c46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f336c1588a0 0x7f336c14cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:44.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f3371c46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f336c1efa80 0x7f336c1eff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:44.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f3371c46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c147060 con 0x7f336c1efa80 2023-10-30T12:47:44.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f3371c46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c15b8b0 con 0x7f336c1588a0 2023-10-30T12:47:44.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f3371c46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336c15b420 con 0x7f3358004d10 2023-10-30T12:47:44.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f336bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f336c1efa80 0x7f336c1eff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:44.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f336bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f336c1efa80 0x7f336c1eff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37864/0 (socket says 172.21.15.125:37864) 2023-10-30T12:47:44.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f336bfff640 1 -- 172.21.15.125:0/1986235430 learned_addr learned my addr 172.21.15.125:0/1986235430 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:44.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f336b7fe640 1 --2- 172.21.15.125:0/1986235430 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3358004d10 0x7f336c146b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:44.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f3363fff640 1 --2- 172.21.15.125:0/1986235430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f336c1588a0 0x7f336c14cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:44.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f336bfff640 1 -- 172.21.15.125:0/1986235430 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3358004d10 msgr2=0x7f336c146b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:44.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f336bfff640 1 --2- 172.21.15.125:0/1986235430 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3358004d10 0x7f336c146b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:44.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f336bfff640 1 -- 172.21.15.125:0/1986235430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f336c1588a0 msgr2=0x7f336c14cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:44.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f336bfff640 1 --2- 172.21.15.125:0/1986235430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f336c1588a0 0x7f336c14cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:44.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f336bfff640 1 -- 172.21.15.125:0/1986235430 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c1590e0 con 0x7f336c1efa80 2023-10-30T12:47:44.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f336bfff640 1 --2- 172.21.15.125:0/1986235430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f336c1efa80 0x7f336c1eff00 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f336401cc40 tx=0x7f3358005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:44.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f3368ff9640 1 -- 172.21.15.125:0/1986235430 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364006d60 con 0x7f336c1efa80 2023-10-30T12:47:44.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f3368ff9640 1 -- 172.21.15.125:0/1986235430 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3364005820 con 0x7f336c1efa80 2023-10-30T12:47:44.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f3368ff9640 1 -- 172.21.15.125:0/1986235430 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364007230 con 0x7f336c1efa80 2023-10-30T12:47:44.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f3371c46640 1 -- 172.21.15.125:0/1986235430 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336c1593c0 con 0x7f336c1efa80 2023-10-30T12:47:44.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f3371c46640 1 -- 172.21.15.125:0/1986235430 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336c15add0 con 0x7f336c1efa80 2023-10-30T12:47:44.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f3371c46640 1 -- 172.21.15.125:0/1986235430 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336c15b590 con 0x7f336c1efa80 2023-10-30T12:47:44.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.180+0000 7f3368ff9640 1 -- 172.21.15.125:0/1986235430 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3364007390 con 0x7f336c1efa80 2023-10-30T12:47:44.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.184+0000 7f3368ff9640 1 --2- 172.21.15.125:0/1986235430 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f334c0664d0 0x7f334c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:44.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.184+0000 7f3368ff9640 1 -- 172.21.15.125:0/1986235430 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f336411db80 con 0x7f336c1efa80 2023-10-30T12:47:44.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.184+0000 7f336b7fe640 1 --2- 172.21.15.125:0/1986235430 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f334c0664d0 0x7f334c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:44.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.184+0000 7f336b7fe640 1 --2- 172.21.15.125:0/1986235430 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f334c0664d0 0x7f334c068990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f335c009ab0 tx=0x7f335c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:44.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.188+0000 7f3368ff9640 1 -- 172.21.15.125:0/1986235430 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f33640eac60 con 0x7f336c1efa80 2023-10-30T12:47:44.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.364+0000 7f3371c46640 1 -- 172.21.15.125:0/1986235430 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f336c04e170 con 0x7f336c1efa80 2023-10-30T12:47:44.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.364+0000 7f3368ff9640 1 -- 172.21.15.125:0/1986235430 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 94 v94) v1 ==== 94+0+5740 (secure 0 0 0) 0x7f336411de60 con 0x7f336c1efa80 2023-10-30T12:47:44.367 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:44.367 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":94,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":94,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:39.615192+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:44.368 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 94 2023-10-30T12:47:44.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.368+0000 7f3371c46640 1 -- 172.21.15.125:0/1986235430 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f334c0664d0 msgr2=0x7f334c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:44.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.368+0000 7f3371c46640 1 --2- 172.21.15.125:0/1986235430 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f334c0664d0 0x7f334c068990 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f335c009ab0 tx=0x7f335c008040 comp rx=0 tx=0).stop 2023-10-30T12:47:44.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.368+0000 7f3371c46640 1 -- 172.21.15.125:0/1986235430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f336c1efa80 msgr2=0x7f336c1eff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:44.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.368+0000 7f3371c46640 1 --2- 172.21.15.125:0/1986235430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f336c1efa80 0x7f336c1eff00 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f336401cc40 tx=0x7f3358005800 comp rx=0 tx=0).stop 2023-10-30T12:47:44.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.368+0000 7f3371c46640 1 -- 172.21.15.125:0/1986235430 shutdown_connections 2023-10-30T12:47:44.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.368+0000 7f3371c46640 1 --2- 172.21.15.125:0/1986235430 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f334c0664d0 0x7f334c068990 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:44.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.368+0000 7f3371c46640 1 --2- 172.21.15.125:0/1986235430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f336c1efa80 0x7f336c1eff00 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:44.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.368+0000 7f3371c46640 1 --2- 172.21.15.125:0/1986235430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f336c1588a0 0x7f336c14cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:44.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.368+0000 7f3371c46640 1 --2- 172.21.15.125:0/1986235430 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3358004d10 0x7f336c146b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:44.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.368+0000 7f3371c46640 1 -- 172.21.15.125:0/1986235430 >> 172.21.15.125:0/1986235430 conn(0x7f336c0c7910 msgr2=0x7f336c0c2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:44.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.368+0000 7f3371c46640 1 -- 172.21.15.125:0/1986235430 shutdown_connections 2023-10-30T12:47:44.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:44.368+0000 7f3371c46640 1 -- 172.21.15.125:0/1986235430 wait complete. 2023-10-30T12:47:44.387 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:47:46.388 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:46.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.532+0000 7f7f1ba17640 1 Processor -- start 2023-10-30T12:47:46.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.532+0000 7f7f1ba17640 1 -- start start 2023-10-30T12:47:46.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.532+0000 7f7f1ba17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f140c6b80 0x7f7f140bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:46.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f1ba17640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7f140c6f60 0x7f7f140c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:46.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f1ba17640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f140c0a50 con 0x7f7f140c8430 2023-10-30T12:47:46.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f1ba17640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f1406f6e0 con 0x7f7f140c6f60 2023-10-30T12:47:46.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f1ba17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1406f9c0 con 0x7f7f140c6b80 2023-10-30T12:47:46.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f18fb2640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7f140c6f60 0x7f7f140c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:46.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f18fb2640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7f140c6f60 0x7f7f140c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37874/0 (socket says 172.21.15.125:37874) 2023-10-30T12:47:46.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f18fb2640 1 -- 172.21.15.125:0/2857395189 learned_addr learned my addr 172.21.15.125:0/2857395189 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:46.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f197b3640 1 --2- 172.21.15.125:0/2857395189 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f140c6b80 0x7f7f140bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:46.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f18fb2640 1 -- 172.21.15.125:0/2857395189 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f140c6b80 msgr2=0x7f7f140bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:46.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f18fb2640 1 --2- 172.21.15.125:0/2857395189 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f140c6b80 0x7f7f140bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:46.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f18fb2640 1 -- 172.21.15.125:0/2857395189 >> v1:172.21.15.136:6789/0 conn(0x7f7f140c8430 legacy=0x7f7f1413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:46.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f18fb2640 1 -- 172.21.15.125:0/2857395189 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1406fca0 con 0x7f7f140c6f60 2023-10-30T12:47:46.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f18fb2640 1 --2- 172.21.15.125:0/2857395189 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140c6f60 0x7f7f140c0400 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009870 tx=0x7f7f0401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c1f2148af086903 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:46.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f0bfff640 1 -- 172.21.15.125:0/2857395189 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f040ab650 con 0x7f7f140c6f60 2023-10-30T12:47:46.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f0bfff640 1 -- 172.21.15.125:0/2857395189 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f040ab7b0 con 0x7f7f140c6f60 2023-10-30T12:47:46.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.536+0000 7f7f0bfff640 1 -- 172.21.15.125:0/2857395189 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f040abb40 con 0x7f7f140c6f60 2023-10-30T12:47:46.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 -- 172.21.15.125:0/2857395189 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140c6f60 msgr2=0x7f7f140c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:46.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 --2- 172.21.15.125:0/2857395189 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140c6f60 0x7f7f140c0400 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009870 tx=0x7f7f0401c540 comp rx=0 tx=0).stop 2023-10-30T12:47:46.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 -- 172.21.15.125:0/2857395189 shutdown_connections 2023-10-30T12:47:46.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 --2- 172.21.15.125:0/2857395189 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140c6f60 0x7f7f140c0400 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:46.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 --2- 172.21.15.125:0/2857395189 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f140c6b80 0x7f7f140bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:46.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 -- 172.21.15.125:0/2857395189 >> 172.21.15.125:0/2857395189 conn(0x7f7f140c7910 msgr2=0x7f7f140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:46.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 -- 172.21.15.125:0/2857395189 shutdown_connections 2023-10-30T12:47:46.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 -- 172.21.15.125:0/2857395189 wait complete. 2023-10-30T12:47:46.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 Processor -- start 2023-10-30T12:47:46.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 -- start start 2023-10-30T12:47:46.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140c6b80 0x7f7f140ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:46.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f140c6f60 0x7f7f140c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:46.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f140c8430 0x7f7f14153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:46.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140c9a20 con 0x7f7f140c6b80 2023-10-30T12:47:46.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140cac00 con 0x7f7f140c8430 2023-10-30T12:47:46.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f1ba17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f140c9740 con 0x7f7f140c6f60 2023-10-30T12:47:46.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f197b3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140c6b80 0x7f7f140ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:46.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f197b3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140c6b80 0x7f7f140ca6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37880/0 (socket says 172.21.15.125:37880) 2023-10-30T12:47:46.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f197b3640 1 -- 172.21.15.125:0/3955127414 learned_addr learned my addr 172.21.15.125:0/3955127414 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:46.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f18fb2640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f140c6f60 0x7f7f140c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:46.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.540+0000 7f7f19fb4640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f140c8430 0x7f7f14153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:46.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f197b3640 1 -- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f140c6f60 msgr2=0x7f7f140c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:46.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f197b3640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f140c6f60 0x7f7f140c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:46.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f197b3640 1 -- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f140c8430 msgr2=0x7f7f14153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:46.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f197b3640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f140c8430 0x7f7f14153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:46.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f197b3640 1 -- 172.21.15.125:0/3955127414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f040094d0 con 0x7f7f140c6b80 2023-10-30T12:47:46.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f18fb2640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f140c6f60 0x7f7f140c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:46.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f197b3640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140c6b80 0x7f7f140ca6c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000d800 tx=0x7f7f1000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:46.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f09ffb640 1 -- 172.21.15.125:0/3955127414 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1000b6c0 con 0x7f7f140c6b80 2023-10-30T12:47:46.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f09ffb640 1 -- 172.21.15.125:0/3955127414 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1000b820 con 0x7f7f140c6b80 2023-10-30T12:47:46.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f1ba17640 1 -- 172.21.15.125:0/3955127414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f141f5f60 con 0x7f7f140c6b80 2023-10-30T12:47:46.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f09ffb640 1 -- 172.21.15.125:0/3955127414 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1000bb90 con 0x7f7f140c6b80 2023-10-30T12:47:46.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f1ba17640 1 -- 172.21.15.125:0/3955127414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f140c9ed0 con 0x7f7f140c6b80 2023-10-30T12:47:46.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f1ba17640 1 -- 172.21.15.125:0/3955127414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1404e170 con 0x7f7f140c6b80 2023-10-30T12:47:46.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f09ffb640 1 -- 172.21.15.125:0/3955127414 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7f1000bcf0 con 0x7f7f140c6b80 2023-10-30T12:47:46.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f09ffb640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ef80664a0 0x7f7ef8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:46.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f09ffb640 1 -- 172.21.15.125:0/3955127414 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f7f100846e0 con 0x7f7f140c6b80 2023-10-30T12:47:46.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.544+0000 7f7f18fb2640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ef80664a0 0x7f7ef8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:46.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.548+0000 7f7f18fb2640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ef80664a0 0x7f7ef8068960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7f040af000 tx=0x7f7f040a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:46.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.548+0000 7f7f09ffb640 1 -- 172.21.15.125:0/3955127414 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7f100518d0 con 0x7f7f140c6b80 2023-10-30T12:47:46.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.728+0000 7f7f1ba17640 1 -- 172.21.15.125:0/3955127414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f140c7e00 con 0x7f7f140c6b80 2023-10-30T12:47:46.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.728+0000 7f7f09ffb640 1 -- 172.21.15.125:0/3955127414 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f7f10054ef0 con 0x7f7f140c6b80 2023-10-30T12:47:46.730 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:46.730 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:45.476891+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:46.731 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 95 2023-10-30T12:47:46.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.732+0000 7f7f1ba17640 1 -- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ef80664a0 msgr2=0x7f7ef8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:46.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.732+0000 7f7f1ba17640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ef80664a0 0x7f7ef8068960 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f7f040af000 tx=0x7f7f040a7000 comp rx=0 tx=0).stop 2023-10-30T12:47:46.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.732+0000 7f7f1ba17640 1 -- 172.21.15.125:0/3955127414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140c6b80 msgr2=0x7f7f140ca6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:46.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.732+0000 7f7f1ba17640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140c6b80 0x7f7f140ca6c0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f7f1000d800 tx=0x7f7f1000dcc0 comp rx=0 tx=0).stop 2023-10-30T12:47:46.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.732+0000 7f7f1ba17640 1 -- 172.21.15.125:0/3955127414 shutdown_connections 2023-10-30T12:47:46.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.732+0000 7f7f1ba17640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ef80664a0 0x7f7ef8068960 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:46.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.732+0000 7f7f1ba17640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f140c8430 0x7f7f14153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:46.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.732+0000 7f7f1ba17640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f140c6f60 0x7f7f140c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:46.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.732+0000 7f7f1ba17640 1 --2- 172.21.15.125:0/3955127414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f140c6b80 0x7f7f140ca6c0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:46.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.732+0000 7f7f1ba17640 1 -- 172.21.15.125:0/3955127414 >> 172.21.15.125:0/3955127414 conn(0x7f7f140c7910 msgr2=0x7f7f140bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:46.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.732+0000 7f7f1ba17640 1 -- 172.21.15.125:0/3955127414 shutdown_connections 2023-10-30T12:47:46.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:46.732+0000 7f7f1ba17640 1 -- 172.21.15.125:0/3955127414 wait complete. 2023-10-30T12:47:46.751 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:47:46.944 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:47:46.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:47:48.753 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:48.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a77aa640 1 Processor -- start 2023-10-30T12:47:48.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a77aa640 1 -- start start 2023-10-30T12:47:48.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a77aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62a01538a0 0x7f62a015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:48.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a77aa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f62a00ca0e0 0x7f62a015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:48.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a77aa640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a00cb800 con 0x7f62a00cf160 2023-10-30T12:47:48.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a77aa640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a015dc30 con 0x7f62a00ca0e0 2023-10-30T12:47:48.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a77aa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a00cb330 con 0x7f62a01538a0 2023-10-30T12:47:48.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a4d45640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f62a00ca0e0 0x7f62a015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:48.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a4d45640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f62a00ca0e0 0x7f62a015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37882/0 (socket says 172.21.15.125:37882) 2023-10-30T12:47:48.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a4d45640 1 -- 172.21.15.125:0/3018271571 learned_addr learned my addr 172.21.15.125:0/3018271571 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:48.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a5546640 1 --2- 172.21.15.125:0/3018271571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62a01538a0 0x7f62a015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:48.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a4d45640 1 -- 172.21.15.125:0/3018271571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62a01538a0 msgr2=0x7f62a015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:48.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a4d45640 1 --2- 172.21.15.125:0/3018271571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62a01538a0 0x7f62a015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:48.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a4d45640 1 -- 172.21.15.125:0/3018271571 >> v1:172.21.15.136:6789/0 conn(0x7f62a00cf160 legacy=0x7f62a00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:48.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a4d45640 1 -- 172.21.15.125:0/3018271571 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a00cad70 con 0x7f62a00ca0e0 2023-10-30T12:47:48.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.900+0000 7f62a5546640 1 --2- 172.21.15.125:0/3018271571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62a01538a0 0x7f62a015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:48.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.904+0000 7f62a4d45640 1 --2- 172.21.15.125:0/3018271571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62a00ca0e0 0x7f62a015a090 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f6290009870 tx=0x7f629001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8118cf5b834a12ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:48.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.904+0000 7f6297fff640 1 -- 172.21.15.125:0/3018271571 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62900a8a30 con 0x7f62a00ca0e0 2023-10-30T12:47:48.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.904+0000 7f6297fff640 1 -- 172.21.15.125:0/3018271571 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62900a8b90 con 0x7f62a00ca0e0 2023-10-30T12:47:48.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.904+0000 7f6297fff640 1 -- 172.21.15.125:0/3018271571 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62900b0440 con 0x7f62a00ca0e0 2023-10-30T12:47:48.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.904+0000 7f62a77aa640 1 -- 172.21.15.125:0/3018271571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62a00ca0e0 msgr2=0x7f62a015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:48.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.904+0000 7f62a77aa640 1 --2- 172.21.15.125:0/3018271571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62a00ca0e0 0x7f62a015a090 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f6290009870 tx=0x7f629001ed30 comp rx=0 tx=0).stop 2023-10-30T12:47:48.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.904+0000 7f62a77aa640 1 -- 172.21.15.125:0/3018271571 shutdown_connections 2023-10-30T12:47:48.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.904+0000 7f62a77aa640 1 --2- 172.21.15.125:0/3018271571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62a00ca0e0 0x7f62a015a090 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:48.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.904+0000 7f62a77aa640 1 --2- 172.21.15.125:0/3018271571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62a01538a0 0x7f62a015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:48.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.904+0000 7f62a77aa640 1 -- 172.21.15.125:0/3018271571 >> 172.21.15.125:0/3018271571 conn(0x7f62a006bfe0 msgr2=0x7f62a006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:48.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.904+0000 7f62a77aa640 1 -- 172.21.15.125:0/3018271571 shutdown_connections 2023-10-30T12:47:48.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.904+0000 7f62a77aa640 1 -- 172.21.15.125:0/3018271571 wait complete. 2023-10-30T12:47:48.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a77aa640 1 Processor -- start 2023-10-30T12:47:48.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a77aa640 1 -- start start 2023-10-30T12:47:48.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a77aa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62a00ca0e0 0x7f62a0147e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:48.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a77aa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62a00cf160 0x7f62a015c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:48.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a77aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62a01538a0 0x7f62a015ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:48.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a77aa640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a01f0c50 con 0x7f62a00ca0e0 2023-10-30T12:47:48.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a77aa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a01436f0 con 0x7f62a00cf160 2023-10-30T12:47:48.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a77aa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a015acf0 con 0x7f62a01538a0 2023-10-30T12:47:48.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a5546640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62a00ca0e0 0x7f62a0147e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:48.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a5546640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62a00ca0e0 0x7f62a0147e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37892/0 (socket says 172.21.15.125:37892) 2023-10-30T12:47:48.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a5546640 1 -- 172.21.15.125:0/2789773670 learned_addr learned my addr 172.21.15.125:0/2789773670 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:48.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a4d45640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62a00cf160 0x7f62a015c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:48.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a5d47640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62a01538a0 0x7f62a015ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:48.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a5546640 1 -- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62a01538a0 msgr2=0x7f62a015ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:48.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a5546640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62a01538a0 0x7f62a015ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:48.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a5546640 1 -- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62a00cf160 msgr2=0x7f62a015c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:48.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a5546640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62a00cf160 0x7f62a015c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:48.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a5546640 1 -- 172.21.15.125:0/2789773670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62900094d0 con 0x7f62a00ca0e0 2023-10-30T12:47:48.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a5d47640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62a01538a0 0x7f62a015ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:47:48.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a4d45640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62a00cf160 0x7f62a015c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:47:48.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a5546640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62a00ca0e0 0x7f62a0147e90 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f629c00d800 tx=0x7f629c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:48.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f6295ffb640 1 -- 172.21.15.125:0/2789773670 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c009660 con 0x7f62a00ca0e0 2023-10-30T12:47:48.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f6295ffb640 1 -- 172.21.15.125:0/2789773670 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629c0097c0 con 0x7f62a00ca0e0 2023-10-30T12:47:48.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a77aa640 1 -- 172.21.15.125:0/2789773670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62a015afd0 con 0x7f62a00ca0e0 2023-10-30T12:47:48.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f6295ffb640 1 -- 172.21.15.125:0/2789773670 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629c009b70 con 0x7f62a00ca0e0 2023-10-30T12:47:48.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a77aa640 1 -- 172.21.15.125:0/2789773670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a015b590 con 0x7f62a00ca0e0 2023-10-30T12:47:48.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.908+0000 7f62a77aa640 1 -- 172.21.15.125:0/2789773670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a00b2a50 con 0x7f62a00ca0e0 2023-10-30T12:47:48.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.912+0000 7f6295ffb640 1 -- 172.21.15.125:0/2789773670 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f629c009cd0 con 0x7f62a00ca0e0 2023-10-30T12:47:48.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.912+0000 7f6295ffb640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62840664a0 0x7f6284068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:48.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.912+0000 7f6295ffb640 1 -- 172.21.15.125:0/2789773670 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f629c084990 con 0x7f62a00ca0e0 2023-10-30T12:47:48.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.912+0000 7f62a4d45640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62840664a0 0x7f6284068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:48.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.912+0000 7f62a4d45640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62840664a0 0x7f6284068960 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f62900b2000 tx=0x7f62900aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:48.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:48.916+0000 7f6295ffb640 1 -- 172.21.15.125:0/2789773670 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f629c0519e0 con 0x7f62a00ca0e0 2023-10-30T12:47:49.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:49.104+0000 7f62a77aa640 1 -- 172.21.15.125:0/2789773670 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62a004e170 con 0x7f62a00ca0e0 2023-10-30T12:47:49.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:49.104+0000 7f6295ffb640 1 -- 172.21.15.125:0/2789773670 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f629c055000 con 0x7f62a00ca0e0 2023-10-30T12:47:49.105 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:49.105 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":95,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:45.476891+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:49.106 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 95 2023-10-30T12:47:49.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:49.104+0000 7f62a77aa640 1 -- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62840664a0 msgr2=0x7f6284068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:49.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:49.104+0000 7f62a77aa640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62840664a0 0x7f6284068960 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f62900b2000 tx=0x7f62900aa070 comp rx=0 tx=0).stop 2023-10-30T12:47:49.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:49.104+0000 7f62a77aa640 1 -- 172.21.15.125:0/2789773670 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62a00ca0e0 msgr2=0x7f62a0147e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:49.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:49.104+0000 7f62a77aa640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62a00ca0e0 0x7f62a0147e90 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f629c00d800 tx=0x7f629c00dcc0 comp rx=0 tx=0).stop 2023-10-30T12:47:49.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:49.104+0000 7f62a77aa640 1 -- 172.21.15.125:0/2789773670 shutdown_connections 2023-10-30T12:47:49.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:49.104+0000 7f62a77aa640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62840664a0 0x7f6284068960 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:49.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:49.104+0000 7f62a77aa640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62a01538a0 0x7f62a015ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:49.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:49.104+0000 7f62a77aa640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62a00cf160 0x7f62a015c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:49.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:49.104+0000 7f62a77aa640 1 --2- 172.21.15.125:0/2789773670 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62a00ca0e0 0x7f62a0147e90 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:49.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:49.104+0000 7f62a77aa640 1 -- 172.21.15.125:0/2789773670 >> 172.21.15.125:0/2789773670 conn(0x7f62a006bfe0 msgr2=0x7f62a00cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:49.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:49.104+0000 7f62a77aa640 1 -- 172.21.15.125:0/2789773670 shutdown_connections 2023-10-30T12:47:49.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:49.104+0000 7f62a77aa640 1 -- 172.21.15.125:0/2789773670 wait complete. 2023-10-30T12:47:49.126 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:47:51.127 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:51.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.272+0000 7f9782ad6640 1 Processor -- start 2023-10-30T12:47:51.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.272+0000 7f9782ad6640 1 -- start start 2023-10-30T12:47:51.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.272+0000 7f9782ad6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f977c0c6f60 0x7f977c1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:51.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.272+0000 7f9782ad6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c0c0a50 con 0x7f977c0c6b80 2023-10-30T12:47:51.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.272+0000 7f9782ad6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f977c06f6e0 con 0x7f977c0c05d0 2023-10-30T12:47:51.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.272+0000 7f9782ad6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f977c06f9c0 con 0x7f977c0c6f60 2023-10-30T12:47:51.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.272+0000 7f9780872640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f977c0c6b80 0x7f977c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55160/0 (socket says 172.21.15.125:55160) 2023-10-30T12:47:51.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.272+0000 7f9780872640 1 -- 172.21.15.125:0/2069963645 learned_addr learned my addr 172.21.15.125:0/2069963645 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:47:51.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.272+0000 7f9781073640 1 --2- 172.21.15.125:0/2069963645 >> v2:172.21.15.125:3300/0 conn(0x7f977c0c6f60 0x7f977c1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:51.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f977b7fe640 1 -- 172.21.15.125:0/2069963645 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074407782 0 0) 0x7f977c0c0a50 con 0x7f977c0c6b80 2023-10-30T12:47:51.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f977b7fe640 1 -- 172.21.15.125:0/2069963645 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f976c003530 con 0x7f977c0c6b80 2023-10-30T12:47:51.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f9781073640 1 -- 172.21.15.125:0/2069963645 >> v1:172.21.15.136:6789/0 conn(0x7f977c0c05d0 legacy=0x7f977c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:51.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f9781073640 1 -- 172.21.15.125:0/2069963645 >> v1:172.21.15.125:6789/0 conn(0x7f977c0c6b80 legacy=0x7f977c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:51.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f9781073640 1 -- 172.21.15.125:0/2069963645 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c06fca0 con 0x7f977c0c6f60 2023-10-30T12:47:51.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f9781073640 1 --2- 172.21.15.125:0/2069963645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f977c0c6f60 0x7f977c1566a0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f9774009d90 tx=0x7f977401c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0a15b3380536ff0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:51.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f977b7fe640 1 -- 172.21.15.125:0/2069963645 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97740a5920 con 0x7f977c0c6f60 2023-10-30T12:47:51.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f977b7fe640 1 -- 172.21.15.125:0/2069963645 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97740a5a80 con 0x7f977c0c6f60 2023-10-30T12:47:51.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f977b7fe640 1 -- 172.21.15.125:0/2069963645 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97740a5df0 con 0x7f977c0c6f60 2023-10-30T12:47:51.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f9782ad6640 1 -- 172.21.15.125:0/2069963645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f977c0c6f60 msgr2=0x7f977c1566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:51.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f9782ad6640 1 --2- 172.21.15.125:0/2069963645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f977c0c6f60 0x7f977c1566a0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f9774009d90 tx=0x7f977401c950 comp rx=0 tx=0).stop 2023-10-30T12:47:51.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f9782ad6640 1 -- 172.21.15.125:0/2069963645 shutdown_connections 2023-10-30T12:47:51.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f9782ad6640 1 --2- 172.21.15.125:0/2069963645 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f977c0c6f60 0x7f977c1566a0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:51.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f9782ad6640 1 -- 172.21.15.125:0/2069963645 >> 172.21.15.125:0/2069963645 conn(0x7f977c0c7910 msgr2=0x7f977c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:51.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f9782ad6640 1 -- 172.21.15.125:0/2069963645 shutdown_connections 2023-10-30T12:47:51.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.276+0000 7f9782ad6640 1 -- 172.21.15.125:0/2069963645 wait complete. 2023-10-30T12:47:51.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9782ad6640 1 Processor -- start 2023-10-30T12:47:51.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9782ad6640 1 -- start start 2023-10-30T12:47:51.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9782ad6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f977c0c05d0 0x7f977c138f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:51.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9782ad6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f977c0c6b80 0x7f977c152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:51.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9782ad6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f977c0c6f60 0x7f977c137760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:51.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9782ad6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0c94f0 con 0x7f977c0c6f60 2023-10-30T12:47:51.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9782ad6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0c8f50 con 0x7f977c0c6b80 2023-10-30T12:47:51.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9782ad6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f977c0c9210 con 0x7f977c0c05d0 2023-10-30T12:47:51.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9781073640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f977c0c6f60 0x7f977c137760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:51.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9781073640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f977c0c6f60 0x7f977c137760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56422/0 (socket says 172.21.15.125:56422) 2023-10-30T12:47:51.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9781073640 1 -- 172.21.15.125:0/813033983 learned_addr learned my addr 172.21.15.125:0/813033983 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:51.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f977bfff640 1 --2- 172.21.15.125:0/813033983 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f977c0c6b80 0x7f977c152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:51.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9780872640 1 --2- 172.21.15.125:0/813033983 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f977c0c05d0 0x7f977c138f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:51.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9781073640 1 -- 172.21.15.125:0/813033983 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f977c0c05d0 msgr2=0x7f977c138f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:51.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9781073640 1 --2- 172.21.15.125:0/813033983 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f977c0c05d0 0x7f977c138f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:51.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9781073640 1 -- 172.21.15.125:0/813033983 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f977c0c6b80 msgr2=0x7f977c152a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:51.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9781073640 1 --2- 172.21.15.125:0/813033983 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f977c0c6b80 0x7f977c152a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:51.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9781073640 1 -- 172.21.15.125:0/813033983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9774009a70 con 0x7f977c0c6f60 2023-10-30T12:47:51.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.280+0000 7f9781073640 1 --2- 172.21.15.125:0/813033983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f977c0c6f60 0x7f977c137760 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9774019560 tx=0x7f97740196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:51.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.284+0000 7f97797fa640 1 -- 172.21.15.125:0/813033983 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97740bf070 con 0x7f977c0c6f60 2023-10-30T12:47:51.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.284+0000 7f97797fa640 1 -- 172.21.15.125:0/813033983 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9774009450 con 0x7f977c0c6f60 2023-10-30T12:47:51.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.284+0000 7f9782ad6640 1 -- 172.21.15.125:0/813033983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c1f9b10 con 0x7f977c0c6f60 2023-10-30T12:47:51.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.284+0000 7f97797fa640 1 -- 172.21.15.125:0/813033983 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97740aee00 con 0x7f977c0c6f60 2023-10-30T12:47:51.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.284+0000 7f9782ad6640 1 -- 172.21.15.125:0/813033983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977c1fa0d0 con 0x7f977c0c6f60 2023-10-30T12:47:51.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.284+0000 7f9782ad6640 1 -- 172.21.15.125:0/813033983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977c1fa2c0 con 0x7f977c0c6f60 2023-10-30T12:47:51.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.284+0000 7f97797fa640 1 -- 172.21.15.125:0/813033983 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f97740b4050 con 0x7f977c0c6f60 2023-10-30T12:47:51.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.284+0000 7f97797fa640 1 --2- 172.21.15.125:0/813033983 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f97600664a0 0x7f9760068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:51.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.284+0000 7f97797fa640 1 -- 172.21.15.125:0/813033983 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f977411ed00 con 0x7f977c0c6f60 2023-10-30T12:47:51.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.284+0000 7f9780872640 1 --2- 172.21.15.125:0/813033983 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f97600664a0 0x7f9760068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:51.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.288+0000 7f9780872640 1 --2- 172.21.15.125:0/813033983 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f97600664a0 0x7f9760068960 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f97700011f0 tx=0x7f9770009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:51.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.292+0000 7f97797fa640 1 -- 172.21.15.125:0/813033983 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97740ebf00 con 0x7f977c0c6f60 2023-10-30T12:47:51.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.476+0000 7f9782ad6640 1 -- 172.21.15.125:0/813033983 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f977c1fa2c0 con 0x7f977c0c6f60 2023-10-30T12:47:51.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.476+0000 7f97797fa640 1 -- 172.21.15.125:0/813033983 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f97740b2020 con 0x7f977c0c6f60 2023-10-30T12:47:51.477 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:51.478 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:51.328568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:51.479 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 96 2023-10-30T12:47:51.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.476+0000 7f9782ad6640 1 -- 172.21.15.125:0/813033983 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f97600664a0 msgr2=0x7f9760068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:51.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.476+0000 7f9782ad6640 1 --2- 172.21.15.125:0/813033983 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f97600664a0 0x7f9760068960 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f97700011f0 tx=0x7f9770009040 comp rx=0 tx=0).stop 2023-10-30T12:47:51.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.476+0000 7f9782ad6640 1 -- 172.21.15.125:0/813033983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f977c0c6f60 msgr2=0x7f977c137760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:51.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.476+0000 7f9782ad6640 1 --2- 172.21.15.125:0/813033983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f977c0c6f60 0x7f977c137760 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f9774019560 tx=0x7f97740196f0 comp rx=0 tx=0).stop 2023-10-30T12:47:51.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.476+0000 7f9782ad6640 1 -- 172.21.15.125:0/813033983 shutdown_connections 2023-10-30T12:47:51.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.476+0000 7f9782ad6640 1 --2- 172.21.15.125:0/813033983 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f97600664a0 0x7f9760068960 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:51.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.476+0000 7f9782ad6640 1 --2- 172.21.15.125:0/813033983 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f977c0c6f60 0x7f977c137760 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:51.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.476+0000 7f9782ad6640 1 --2- 172.21.15.125:0/813033983 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f977c0c6b80 0x7f977c152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:51.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.476+0000 7f9782ad6640 1 --2- 172.21.15.125:0/813033983 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f977c0c05d0 0x7f977c138f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:51.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.476+0000 7f9782ad6640 1 -- 172.21.15.125:0/813033983 >> 172.21.15.125:0/813033983 conn(0x7f977c0c7910 msgr2=0x7f977c0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:51.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.476+0000 7f9782ad6640 1 -- 172.21.15.125:0/813033983 shutdown_connections 2023-10-30T12:47:51.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:51.476+0000 7f9782ad6640 1 -- 172.21.15.125:0/813033983 wait complete. 2023-10-30T12:47:51.498 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 96, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 96, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_5988": { "addr": "172.21.15.125:6837/3156415951", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 3156415951, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 3156415951, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 5988, "incarnation": 78, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 28 }, "gid_6318": { "addr": "172.21.15.125:6836/1721514617", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1721514617, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1721514617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6318, "incarnation": 92, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:clientreplay", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:47:51.328568+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6318, "mds_1": 5988 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:47:53.500 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:53.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f67054640 1 Processor -- start 2023-10-30T12:47:53.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f67054640 1 -- start start 2023-10-30T12:47:53.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f67054640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f600c6fe0 0x7f5f60071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:53.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f67054640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5f600c5f60 0x7f5f600cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:53.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f67054640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f600700b0 con 0x7f5f60153bf0 2023-10-30T12:47:53.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f67054640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f600c6810 con 0x7f5f600c5f60 2023-10-30T12:47:53.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f67054640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f60159900 con 0x7f5f600c6fe0 2023-10-30T12:47:53.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f655f1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5f600c5f60 0x7f5f600cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:53.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f655f1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5f600c5f60 0x7f5f600cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56434/0 (socket says 172.21.15.125:56434) 2023-10-30T12:47:53.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f655f1640 1 -- 172.21.15.125:0/2856328786 learned_addr learned my addr 172.21.15.125:0/2856328786 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:53.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f57fff640 1 --2- 172.21.15.125:0/2856328786 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f600c6fe0 0x7f5f60071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:53.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f655f1640 1 -- 172.21.15.125:0/2856328786 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f600c6fe0 msgr2=0x7f5f60071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:53.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f655f1640 1 --2- 172.21.15.125:0/2856328786 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f600c6fe0 0x7f5f60071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:53.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f655f1640 1 -- 172.21.15.125:0/2856328786 >> v1:172.21.15.136:6789/0 conn(0x7f5f60153bf0 legacy=0x7f5f600c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:53.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.648+0000 7f5f655f1640 1 -- 172.21.15.125:0/2856328786 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f600c6530 con 0x7f5f600c5f60 2023-10-30T12:47:53.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.652+0000 7f5f655f1640 1 --2- 172.21.15.125:0/2856328786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f600c5f60 0x7f5f600cb570 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009e50 tx=0x7f5f5801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e929cf2fcb1ef388 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:53.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.652+0000 7f5f577fe640 1 -- 172.21.15.125:0/2856328786 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f580af400 con 0x7f5f600c5f60 2023-10-30T12:47:53.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.652+0000 7f5f577fe640 1 -- 172.21.15.125:0/2856328786 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f580ab050 con 0x7f5f600c5f60 2023-10-30T12:47:53.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.652+0000 7f5f577fe640 1 -- 172.21.15.125:0/2856328786 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f580af780 con 0x7f5f600c5f60 2023-10-30T12:47:53.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.652+0000 7f5f67054640 1 -- 172.21.15.125:0/2856328786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f600c5f60 msgr2=0x7f5f600cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:53.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.652+0000 7f5f67054640 1 --2- 172.21.15.125:0/2856328786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f600c5f60 0x7f5f600cb570 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f5f58009e50 tx=0x7f5f5801f1f0 comp rx=0 tx=0).stop 2023-10-30T12:47:53.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.652+0000 7f5f67054640 1 -- 172.21.15.125:0/2856328786 shutdown_connections 2023-10-30T12:47:53.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.652+0000 7f5f67054640 1 --2- 172.21.15.125:0/2856328786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f600c5f60 0x7f5f600cb570 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:53.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.652+0000 7f5f67054640 1 --2- 172.21.15.125:0/2856328786 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f600c6fe0 0x7f5f60071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:53.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.652+0000 7f5f67054640 1 -- 172.21.15.125:0/2856328786 >> 172.21.15.125:0/2856328786 conn(0x7f5f6006c6e0 msgr2=0x7f5f60069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:53.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.652+0000 7f5f67054640 1 -- 172.21.15.125:0/2856328786 shutdown_connections 2023-10-30T12:47:53.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.652+0000 7f5f67054640 1 -- 172.21.15.125:0/2856328786 wait complete. 2023-10-30T12:47:53.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f67054640 1 Processor -- start 2023-10-30T12:47:53.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f67054640 1 -- start start 2023-10-30T12:47:53.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f67054640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f600c5f60 0x7f5f600c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:53.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f67054640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f600c6fe0 0x7f5f600c5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:53.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f67054640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f60153bf0 0x7f5f601ef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:53.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f67054640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f60154e10 con 0x7f5f60153bf0 2023-10-30T12:47:53.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f67054640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f601587f0 con 0x7f5f600c6fe0 2023-10-30T12:47:53.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f67054640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f601ef150 con 0x7f5f600c5f60 2023-10-30T12:47:53.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f655f1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f60153bf0 0x7f5f601ef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:53.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f655f1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f60153bf0 0x7f5f601ef5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56440/0 (socket says 172.21.15.125:56440) 2023-10-30T12:47:53.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f655f1640 1 -- 172.21.15.125:0/2901767656 learned_addr learned my addr 172.21.15.125:0/2901767656 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:53.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f57fff640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f600c6fe0 0x7f5f600c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:53.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f64df0640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f600c5f60 0x7f5f600c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:53.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f655f1640 1 -- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f600c5f60 msgr2=0x7f5f600c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:53.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f655f1640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f600c5f60 0x7f5f600c7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:53.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f655f1640 1 -- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f600c6fe0 msgr2=0x7f5f600c5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:53.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f655f1640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f600c6fe0 0x7f5f600c5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:53.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f655f1640 1 -- 172.21.15.125:0/2901767656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f58009b30 con 0x7f5f60153bf0 2023-10-30T12:47:53.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f57fff640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f600c6fe0 0x7f5f600c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:53.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f655f1640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f60153bf0 0x7f5f601ef5a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f5f5801ce20 tx=0x7f5f5801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:53.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f557fa640 1 -- 172.21.15.125:0/2901767656 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f580afde0 con 0x7f5f60153bf0 2023-10-30T12:47:53.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f557fa640 1 -- 172.21.15.125:0/2901767656 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f58009510 con 0x7f5f60153bf0 2023-10-30T12:47:53.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f557fa640 1 -- 172.21.15.125:0/2901767656 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f580ba660 con 0x7f5f60153bf0 2023-10-30T12:47:53.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f67054640 1 -- 172.21.15.125:0/2901767656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f601540b0 con 0x7f5f60153bf0 2023-10-30T12:47:53.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.656+0000 7f5f67054640 1 -- 172.21.15.125:0/2901767656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f60154300 con 0x7f5f60153bf0 2023-10-30T12:47:53.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.660+0000 7f5f67054640 1 -- 172.21.15.125:0/2901767656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f601ef2c0 con 0x7f5f60153bf0 2023-10-30T12:47:53.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.660+0000 7f5f557fa640 1 -- 172.21.15.125:0/2901767656 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5f580b6040 con 0x7f5f60153bf0 2023-10-30T12:47:53.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.660+0000 7f5f557fa640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5f4c0664a0 0x7f5f4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:53.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.660+0000 7f5f557fa640 1 -- 172.21.15.125:0/2901767656 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f5f581223d0 con 0x7f5f60153bf0 2023-10-30T12:47:53.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.660+0000 7f5f64df0640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5f4c0664a0 0x7f5f4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:53.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.660+0000 7f5f64df0640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5f4c0664a0 0x7f5f4c068960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c000f70 tx=0x7f5f5c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:53.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.664+0000 7f5f557fa640 1 -- 172.21.15.125:0/2901767656 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5f601ef2c0 con 0x7f5f60153bf0 2023-10-30T12:47:53.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.844+0000 7f5f67054640 1 -- 172.21.15.125:0/2901767656 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f601ef2c0 con 0x7f5f60153bf0 2023-10-30T12:47:53.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.844+0000 7f5f557fa640 1 -- 172.21.15.125:0/2901767656 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f5f601ef2c0 con 0x7f5f60153bf0 2023-10-30T12:47:53.848 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:53.848 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:51.328568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:53.848 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 96 2023-10-30T12:47:53.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.848+0000 7f5f67054640 1 -- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5f4c0664a0 msgr2=0x7f5f4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:53.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.848+0000 7f5f67054640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5f4c0664a0 0x7f5f4c068960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c000f70 tx=0x7f5f5c009040 comp rx=0 tx=0).stop 2023-10-30T12:47:53.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.848+0000 7f5f67054640 1 -- 172.21.15.125:0/2901767656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f60153bf0 msgr2=0x7f5f601ef5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:53.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.848+0000 7f5f67054640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f60153bf0 0x7f5f601ef5a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f5f5801ce20 tx=0x7f5f5801bf90 comp rx=0 tx=0).stop 2023-10-30T12:47:53.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.848+0000 7f5f67054640 1 -- 172.21.15.125:0/2901767656 shutdown_connections 2023-10-30T12:47:53.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.848+0000 7f5f67054640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5f4c0664a0 0x7f5f4c068960 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:53.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.848+0000 7f5f67054640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f60153bf0 0x7f5f601ef5a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:53.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.848+0000 7f5f67054640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f600c6fe0 0x7f5f600c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:53.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.848+0000 7f5f67054640 1 --2- 172.21.15.125:0/2901767656 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f600c5f60 0x7f5f600c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:53.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.848+0000 7f5f67054640 1 -- 172.21.15.125:0/2901767656 >> 172.21.15.125:0/2901767656 conn(0x7f5f6006c6e0 msgr2=0x7f5f60157e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:53.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.848+0000 7f5f67054640 1 -- 172.21.15.125:0/2901767656 shutdown_connections 2023-10-30T12:47:53.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:53.848+0000 7f5f67054640 1 -- 172.21.15.125:0/2901767656 wait complete. 2023-10-30T12:47:53.868 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:47:55.869 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:56.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.016+0000 7f6ec03c5640 1 Processor -- start 2023-10-30T12:47:56.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.016+0000 7f6ec03c5640 1 -- start start 2023-10-30T12:47:56.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.016+0000 7f6ec03c5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6eb80c93d0 0x7f6eb806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:56.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.016+0000 7f6ec03c5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb806afa0 con 0x7f6eb8153d70 2023-10-30T12:47:56.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.016+0000 7f6ec03c5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eb8154150 con 0x7f6eb806e470 2023-10-30T12:47:56.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.016+0000 7f6ec03c5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eb806d780 con 0x7f6eb80c93d0 2023-10-30T12:47:56.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.016+0000 7f6ebe962640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6eb80c93d0 0x7f6eb806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:56.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.016+0000 7f6ebe962640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6eb80c93d0 0x7f6eb806ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56448/0 (socket says 172.21.15.125:56448) 2023-10-30T12:47:56.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.016+0000 7f6ebe962640 1 -- 172.21.15.125:0/3410745962 learned_addr learned my addr 172.21.15.125:0/3410745962 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:56.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.016+0000 7f6ebe161640 1 --1- 172.21.15.125:0/3410745962 >> v1:172.21.15.125:6789/0 conn(0x7f6eb8153d70 0x7f6eb80cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55166/0 (socket says 172.21.15.125:55166) 2023-10-30T12:47:56.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.016+0000 7f6ebe962640 1 -- 172.21.15.125:0/3410745962 >> v1:172.21.15.136:6789/0 conn(0x7f6eb806e470 legacy=0x7f6eb80c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:56.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.016+0000 7f6ebe962640 1 -- 172.21.15.125:0/3410745962 >> v1:172.21.15.125:6789/0 conn(0x7f6eb8153d70 legacy=0x7f6eb80cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:56.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.016+0000 7f6ebe962640 1 -- 172.21.15.125:0/3410745962 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb806e030 con 0x7f6eb80c93d0 2023-10-30T12:47:56.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.016+0000 7f6ebd15f640 1 -- 172.21.15.125:0/3410745962 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510743481 0 0) 0x7f6eb806afa0 con 0x7f6eb8153d70 2023-10-30T12:47:56.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.020+0000 7f6ebe962640 1 --2- 172.21.15.125:0/3410745962 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6eb80c93d0 0x7f6eb806ca80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009e50 tx=0x7f6eb001c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8827afcbaa70cde server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:56.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.020+0000 7f6ebd15f640 1 -- 172.21.15.125:0/3410745962 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb00a58a0 con 0x7f6eb80c93d0 2023-10-30T12:47:56.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.020+0000 7f6ebd15f640 1 -- 172.21.15.125:0/3410745962 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb00a5a00 con 0x7f6eb80c93d0 2023-10-30T12:47:56.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.020+0000 7f6ebd15f640 1 -- 172.21.15.125:0/3410745962 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb00a5d90 con 0x7f6eb80c93d0 2023-10-30T12:47:56.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.020+0000 7f6ec03c5640 1 -- 172.21.15.125:0/3410745962 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6eb80c93d0 msgr2=0x7f6eb806ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:56.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.020+0000 7f6ec03c5640 1 --2- 172.21.15.125:0/3410745962 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6eb80c93d0 0x7f6eb806ca80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f6eb0009e50 tx=0x7f6eb001c9e0 comp rx=0 tx=0).stop 2023-10-30T12:47:56.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.020+0000 7f6ec03c5640 1 -- 172.21.15.125:0/3410745962 shutdown_connections 2023-10-30T12:47:56.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.020+0000 7f6ec03c5640 1 --2- 172.21.15.125:0/3410745962 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6eb80c93d0 0x7f6eb806ca80 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:56.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.020+0000 7f6ec03c5640 1 -- 172.21.15.125:0/3410745962 >> 172.21.15.125:0/3410745962 conn(0x7f6eb8071ea0 msgr2=0x7f6eb8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:56.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.020+0000 7f6ec03c5640 1 -- 172.21.15.125:0/3410745962 shutdown_connections 2023-10-30T12:47:56.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.020+0000 7f6ec03c5640 1 -- 172.21.15.125:0/3410745962 wait complete. 2023-10-30T12:47:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.020+0000 7f6ec03c5640 1 Processor -- start 2023-10-30T12:47:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ec03c5640 1 -- start start 2023-10-30T12:47:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ec03c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6eb806e470 0x7f6eb80cb140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:56.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ec03c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6eb80c93d0 0x7f6eb81f5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:56.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ec03c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eb8153d70 0x7f6eb81f55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:56.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ec03c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb81362c0 con 0x7f6eb806e470 2023-10-30T12:47:56.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ec03c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb81f6a00 con 0x7f6eb8153d70 2023-10-30T12:47:56.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ec03c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eb81356b0 con 0x7f6eb80c93d0 2023-10-30T12:47:56.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ebe161640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6eb806e470 0x7f6eb80cb140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:56.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ebe161640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6eb806e470 0x7f6eb80cb140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56464/0 (socket says 172.21.15.125:56464) 2023-10-30T12:47:56.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ebe161640 1 -- 172.21.15.125:0/2118203436 learned_addr learned my addr 172.21.15.125:0/2118203436 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:56.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ebe962640 1 --2- 172.21.15.125:0/2118203436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eb8153d70 0x7f6eb81f55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:56.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ebd960640 1 --2- 172.21.15.125:0/2118203436 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6eb80c93d0 0x7f6eb81f5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:56.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ebe161640 1 -- 172.21.15.125:0/2118203436 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6eb80c93d0 msgr2=0x7f6eb81f5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:56.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ebe161640 1 --2- 172.21.15.125:0/2118203436 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6eb80c93d0 0x7f6eb81f5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:56.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ebe161640 1 -- 172.21.15.125:0/2118203436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eb8153d70 msgr2=0x7f6eb81f55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:56.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ebe161640 1 --2- 172.21.15.125:0/2118203436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eb8153d70 0x7f6eb81f55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:56.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ebe161640 1 -- 172.21.15.125:0/2118203436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eb0009b30 con 0x7f6eb806e470 2023-10-30T12:47:56.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ebe161640 1 --2- 172.21.15.125:0/2118203436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6eb806e470 0x7f6eb80cb140 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400c410 tx=0x7f6eb400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:56.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6eaeffd640 1 -- 172.21.15.125:0/2118203436 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb40074e0 con 0x7f6eb806e470 2023-10-30T12:47:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6eaeffd640 1 -- 172.21.15.125:0/2118203436 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6eb4007640 con 0x7f6eb806e470 2023-10-30T12:47:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6eaeffd640 1 -- 172.21.15.125:0/2118203436 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4004a30 con 0x7f6eb806e470 2023-10-30T12:47:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ec03c5640 1 -- 172.21.15.125:0/2118203436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eb8135990 con 0x7f6eb806e470 2023-10-30T12:47:56.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ec03c5640 1 -- 172.21.15.125:0/2118203436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eb8135f50 con 0x7f6eb806e470 2023-10-30T12:47:56.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.024+0000 7f6ec03c5640 1 -- 172.21.15.125:0/2118203436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eb8136110 con 0x7f6eb806e470 2023-10-30T12:47:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.028+0000 7f6eaeffd640 1 -- 172.21.15.125:0/2118203436 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6eb4004b90 con 0x7f6eb806e470 2023-10-30T12:47:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.028+0000 7f6eaeffd640 1 --2- 172.21.15.125:0/2118203436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e9c0664a0 0x7f6e9c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.028+0000 7f6eaeffd640 1 -- 172.21.15.125:0/2118203436 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f6eb4084070 con 0x7f6eb806e470 2023-10-30T12:47:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.028+0000 7f6ebd960640 1 --2- 172.21.15.125:0/2118203436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e9c0664a0 0x7f6e9c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:56.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.028+0000 7f6ebd960640 1 --2- 172.21.15.125:0/2118203436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e9c0664a0 0x7f6e9c068960 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6eb80cbbb0 tx=0x7f6ea8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:56.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.032+0000 7f6eaeffd640 1 -- 172.21.15.125:0/2118203436 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6eb4051140 con 0x7f6eb806e470 2023-10-30T12:47:56.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.216+0000 7f6ec03c5640 1 -- 172.21.15.125:0/2118203436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6eb804e170 con 0x7f6eb806e470 2023-10-30T12:47:56.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.216+0000 7f6eaeffd640 1 -- 172.21.15.125:0/2118203436 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+5743 (secure 0 0 0) 0x7f6eb4054760 con 0x7f6eb806e470 2023-10-30T12:47:56.219 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:56.219 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:51.328568+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:clientreplay","state_seq":16,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:56.219 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 96 2023-10-30T12:47:56.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.220+0000 7f6ec03c5640 1 -- 172.21.15.125:0/2118203436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e9c0664a0 msgr2=0x7f6e9c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:56.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.220+0000 7f6ec03c5640 1 --2- 172.21.15.125:0/2118203436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e9c0664a0 0x7f6e9c068960 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f6eb80cbbb0 tx=0x7f6ea8009040 comp rx=0 tx=0).stop 2023-10-30T12:47:56.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.220+0000 7f6ec03c5640 1 -- 172.21.15.125:0/2118203436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6eb806e470 msgr2=0x7f6eb80cb140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:56.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.220+0000 7f6ec03c5640 1 --2- 172.21.15.125:0/2118203436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6eb806e470 0x7f6eb80cb140 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400c410 tx=0x7f6eb400c8d0 comp rx=0 tx=0).stop 2023-10-30T12:47:56.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.220+0000 7f6ec03c5640 1 -- 172.21.15.125:0/2118203436 shutdown_connections 2023-10-30T12:47:56.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.220+0000 7f6ec03c5640 1 --2- 172.21.15.125:0/2118203436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e9c0664a0 0x7f6e9c068960 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:56.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.220+0000 7f6ec03c5640 1 --2- 172.21.15.125:0/2118203436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6eb8153d70 0x7f6eb81f55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:56.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.220+0000 7f6ec03c5640 1 --2- 172.21.15.125:0/2118203436 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6eb80c93d0 0x7f6eb81f5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:56.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.220+0000 7f6ec03c5640 1 --2- 172.21.15.125:0/2118203436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6eb806e470 0x7f6eb80cb140 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:56.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.220+0000 7f6ec03c5640 1 -- 172.21.15.125:0/2118203436 >> 172.21.15.125:0/2118203436 conn(0x7f6eb8071ea0 msgr2=0x7f6eb8151370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:56.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.220+0000 7f6ec03c5640 1 -- 172.21.15.125:0/2118203436 shutdown_connections 2023-10-30T12:47:56.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:56.220+0000 7f6ec03c5640 1 -- 172.21.15.125:0/2118203436 wait complete. 2023-10-30T12:47:56.239 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:47:58.241 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:47:58.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.388+0000 7f9827d9d640 1 Processor -- start 2023-10-30T12:47:58.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.388+0000 7f9827d9d640 1 -- start start 2023-10-30T12:47:58.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.388+0000 7f9827d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f982006f7a0 0x7f982006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:58.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.388+0000 7f9827d9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98200c6bd0 0x7f9820069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:58.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.388+0000 7f9827d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9820071110 0x7f9820158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:58.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.388+0000 7f9827d9d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98200700c0 con 0x7f9820071110 2023-10-30T12:47:58.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.388+0000 7f9827d9d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f982006a3d0 con 0x7f982006f7a0 2023-10-30T12:47:58.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.388+0000 7f9827d9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9820066cf0 con 0x7f98200c6bd0 2023-10-30T12:47:58.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.388+0000 7f982633a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9820071110 0x7f9820158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:58.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.388+0000 7f982633a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9820071110 0x7f9820158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56470/0 (socket says 172.21.15.125:56470) 2023-10-30T12:47:58.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.388+0000 7f982633a640 1 -- 172.21.15.125:0/906193035 learned_addr learned my addr 172.21.15.125:0/906193035 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:58.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.388+0000 7f9825338640 1 --2- 172.21.15.125:0/906193035 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98200c6bd0 0x7f9820069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:58.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.388+0000 7f9825b39640 1 --2- 172.21.15.125:0/906193035 >> v2:172.21.15.136:3300/0 conn(0x7f982006f7a0 0x7f982006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:58.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.388+0000 7f982633a640 1 -- 172.21.15.125:0/906193035 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98200c6bd0 msgr2=0x7f9820069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:58.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f982633a640 1 --2- 172.21.15.125:0/906193035 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98200c6bd0 0x7f9820069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:58.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f982633a640 1 -- 172.21.15.125:0/906193035 >> v2:172.21.15.136:3300/0 conn(0x7f982006f7a0 msgr2=0x7f982006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:47:58.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f982633a640 1 --2- 172.21.15.125:0/906193035 >> v2:172.21.15.136:3300/0 conn(0x7f982006f7a0 0x7f982006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:58.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f982633a640 1 -- 172.21.15.125:0/906193035 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98200c70d0 con 0x7f9820071110 2023-10-30T12:47:58.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f9825338640 1 --2- 172.21.15.125:0/906193035 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98200c6bd0 0x7f9820069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:58.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f9825b39640 1 --2- 172.21.15.125:0/906193035 >> v2:172.21.15.136:3300/0 conn(0x7f982006f7a0 0x7f982006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:58.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f982633a640 1 --2- 172.21.15.125:0/906193035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9820071110 0x7f9820158110 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f982006c830 tx=0x7f981801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=952668374470eb14 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:58.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f9824b37640 1 -- 172.21.15.125:0/906193035 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98180af400 con 0x7f9820071110 2023-10-30T12:47:58.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f9824b37640 1 -- 172.21.15.125:0/906193035 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98180ab050 con 0x7f9820071110 2023-10-30T12:47:58.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f9824b37640 1 -- 172.21.15.125:0/906193035 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98180af780 con 0x7f9820071110 2023-10-30T12:47:58.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f9827d9d640 1 -- 172.21.15.125:0/906193035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9820071110 msgr2=0x7f9820158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:58.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f9827d9d640 1 --2- 172.21.15.125:0/906193035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9820071110 0x7f9820158110 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f982006c830 tx=0x7f981801f1c0 comp rx=0 tx=0).stop 2023-10-30T12:47:58.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f9827d9d640 1 -- 172.21.15.125:0/906193035 shutdown_connections 2023-10-30T12:47:58.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f9827d9d640 1 --2- 172.21.15.125:0/906193035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9820071110 0x7f9820158110 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:58.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f9827d9d640 1 --2- 172.21.15.125:0/906193035 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98200c6bd0 0x7f9820069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:58.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f9827d9d640 1 --2- 172.21.15.125:0/906193035 >> v2:172.21.15.136:3300/0 conn(0x7f982006f7a0 0x7f982006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:58.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f9827d9d640 1 -- 172.21.15.125:0/906193035 >> 172.21.15.125:0/906193035 conn(0x7f98200c7910 msgr2=0x7f98200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:58.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f9827d9d640 1 -- 172.21.15.125:0/906193035 shutdown_connections 2023-10-30T12:47:58.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.392+0000 7f9827d9d640 1 -- 172.21.15.125:0/906193035 wait complete. 2023-10-30T12:47:58.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9827d9d640 1 Processor -- start 2023-10-30T12:47:58.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9827d9d640 1 -- start start 2023-10-30T12:47:58.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9827d9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f982006f7a0 0x7f98201f2c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:58.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9827d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9820071110 0x7f982014ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:58.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9827d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98200c6bd0 0x7f982014dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:58.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9827d9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982014d8a0 con 0x7f9820071110 2023-10-30T12:47:58.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9827d9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982014f8a0 con 0x7f98200c6bd0 2023-10-30T12:47:58.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9827d9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98201f3160 con 0x7f982006f7a0 2023-10-30T12:47:58.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9825338640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9820071110 0x7f982014ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:58.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9825338640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9820071110 0x7f982014ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56474/0 (socket says 172.21.15.125:56474) 2023-10-30T12:47:58.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9825338640 1 -- 172.21.15.125:0/3443695141 learned_addr learned my addr 172.21.15.125:0/3443695141 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:47:58.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f982633a640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98200c6bd0 0x7f982014dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:58.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9825b39640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f982006f7a0 0x7f98201f2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:58.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9825338640 1 -- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f982006f7a0 msgr2=0x7f98201f2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:58.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9825338640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f982006f7a0 0x7f98201f2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:58.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9825338640 1 -- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98200c6bd0 msgr2=0x7f982014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:58.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9825338640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98200c6bd0 0x7f982014dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:58.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9825338640 1 -- 172.21.15.125:0/3443695141 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9818009b30 con 0x7f9820071110 2023-10-30T12:47:58.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f982633a640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98200c6bd0 0x7f982014dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:47:58.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9825338640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9820071110 0x7f982014ca80 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f981000d830 tx=0x7f981000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:58.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f98167fc640 1 -- 172.21.15.125:0/3443695141 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98100096c0 con 0x7f9820071110 2023-10-30T12:47:58.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f98167fc640 1 -- 172.21.15.125:0/3443695141 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9810009820 con 0x7f9820071110 2023-10-30T12:47:58.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f9827d9d640 1 -- 172.21.15.125:0/3443695141 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982014f000 con 0x7f9820071110 2023-10-30T12:47:58.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.396+0000 7f98167fc640 1 -- 172.21.15.125:0/3443695141 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810009bd0 con 0x7f9820071110 2023-10-30T12:47:58.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.400+0000 7f9827d9d640 1 -- 172.21.15.125:0/3443695141 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982014f5c0 con 0x7f9820071110 2023-10-30T12:47:58.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.400+0000 7f9827d9d640 1 -- 172.21.15.125:0/3443695141 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982014db10 con 0x7f9820071110 2023-10-30T12:47:58.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.400+0000 7f98167fc640 1 -- 172.21.15.125:0/3443695141 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9810014410 con 0x7f9820071110 2023-10-30T12:47:58.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.400+0000 7f98167fc640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98040664d0 0x7f9804068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:47:58.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.400+0000 7f98167fc640 1 -- 172.21.15.125:0/3443695141 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f9810084a30 con 0x7f9820071110 2023-10-30T12:47:58.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.400+0000 7f9825b39640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98040664d0 0x7f9804068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:47:58.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.400+0000 7f9825b39640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98040664d0 0x7f9804068990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f981c00b970 tx=0x7f981c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:47:58.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.404+0000 7f98167fc640 1 -- 172.21.15.125:0/3443695141 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9810051c20 con 0x7f9820071110 2023-10-30T12:47:58.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.584+0000 7f9827d9d640 1 -- 172.21.15.125:0/3443695141 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f982004e170 con 0x7f9820071110 2023-10-30T12:47:58.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.584+0000 7f98167fc640 1 -- 172.21.15.125:0/3443695141 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f9810055240 con 0x7f9820071110 2023-10-30T12:47:58.586 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:47:58.587 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:56.562576+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:47:58.587 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 97 2023-10-30T12:47:58.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.588+0000 7f9827d9d640 1 -- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98040664d0 msgr2=0x7f9804068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:58.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.588+0000 7f9827d9d640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98040664d0 0x7f9804068990 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f981c00b970 tx=0x7f981c008040 comp rx=0 tx=0).stop 2023-10-30T12:47:58.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.588+0000 7f9827d9d640 1 -- 172.21.15.125:0/3443695141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9820071110 msgr2=0x7f982014ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:47:58.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.588+0000 7f9827d9d640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9820071110 0x7f982014ca80 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f981000d830 tx=0x7f981000dcf0 comp rx=0 tx=0).stop 2023-10-30T12:47:58.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.588+0000 7f9827d9d640 1 -- 172.21.15.125:0/3443695141 shutdown_connections 2023-10-30T12:47:58.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.588+0000 7f9827d9d640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98040664d0 0x7f9804068990 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:58.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.588+0000 7f9827d9d640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98200c6bd0 0x7f982014dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:58.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.588+0000 7f9827d9d640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9820071110 0x7f982014ca80 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:58.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.588+0000 7f9827d9d640 1 --2- 172.21.15.125:0/3443695141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f982006f7a0 0x7f98201f2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:47:58.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.588+0000 7f9827d9d640 1 -- 172.21.15.125:0/3443695141 >> 172.21.15.125:0/3443695141 conn(0x7f98200c7910 msgr2=0x7f98200c3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:47:58.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.588+0000 7f9827d9d640 1 -- 172.21.15.125:0/3443695141 shutdown_connections 2023-10-30T12:47:58.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:47:58.588+0000 7f9827d9d640 1 -- 172.21.15.125:0/3443695141 wait complete. 2023-10-30T12:47:58.607 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=5935 2023-10-30T12:47:58.607 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.f 2023-10-30T12:48:00.608 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T12:48:00.608 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T12:48:00.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T12:48:00.611 INFO:tasks.ceph.mds.f:Started 2023-10-30T12:48:00.612 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:00.685 INFO:tasks.ceph.mds.f.smithi136.stdout:starting mds.f at 2023-10-30T12:48:00.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.748+0000 7f0df5817640 1 Processor -- start 2023-10-30T12:48:00.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0df5817640 1 -- start start 2023-10-30T12:48:00.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0df5817640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0df015cd50 0x7f0df00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:00.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0df5817640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0df00cefa0 0x7f0df00cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:00.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0df5817640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df015a6c0 con 0x7f0df01538f0 2023-10-30T12:48:00.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0df5817640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0df015d300 con 0x7f0df015cd50 2023-10-30T12:48:00.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0df5817640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df015d5e0 con 0x7f0df00cefa0 2023-10-30T12:48:00.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0deffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0df015cd50 0x7f0df00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:00.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0df4815640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0df01538f0 0x7f0df0159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:48202/0 (socket says 172.21.15.125:48202) 2023-10-30T12:48:00.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0df4815640 1 -- 172.21.15.125:0/1013950569 learned_addr learned my addr 172.21.15.125:0/1013950569 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:48:00.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0df5016640 1 --2- 172.21.15.125:0/1013950569 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0df00cefa0 0x7f0df00cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:00.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0def7fe640 1 -- 172.21.15.125:0/1013950569 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 573289645 0 0) 0x7f0df015a6c0 con 0x7f0df01538f0 2023-10-30T12:48:00.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0def7fe640 1 -- 172.21.15.125:0/1013950569 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0de0003530 con 0x7f0df01538f0 2023-10-30T12:48:00.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0df5016640 1 -- 172.21.15.125:0/1013950569 >> v2:172.21.15.136:3300/0 conn(0x7f0df015cd50 msgr2=0x7f0df00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:00.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0df5016640 1 --2- 172.21.15.125:0/1013950569 >> v2:172.21.15.136:3300/0 conn(0x7f0df015cd50 0x7f0df00c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0df5016640 1 -- 172.21.15.125:0/1013950569 >> v1:172.21.15.136:6789/0 conn(0x7f0df01538f0 legacy=0x7f0df0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:00.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.752+0000 7f0df5016640 1 -- 172.21.15.125:0/1013950569 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df015d8c0 con 0x7f0df00cefa0 2023-10-30T12:48:00.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.756+0000 7f0df5016640 1 --2- 172.21.15.125:0/1013950569 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0df00cefa0 0x7f0df00cf3d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009d90 tx=0x7f0de801f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84fa071e61ecbc9d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:00.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.756+0000 7f0def7fe640 1 -- 172.21.15.125:0/1013950569 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de80af600 con 0x7f0df00cefa0 2023-10-30T12:48:00.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.756+0000 7f0def7fe640 1 -- 172.21.15.125:0/1013950569 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de80ab050 con 0x7f0df00cefa0 2023-10-30T12:48:00.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.756+0000 7f0def7fe640 1 -- 172.21.15.125:0/1013950569 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de80af980 con 0x7f0df00cefa0 2023-10-30T12:48:00.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.756+0000 7f0df5817640 1 -- 172.21.15.125:0/1013950569 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0df00cefa0 msgr2=0x7f0df00cf3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:00.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.756+0000 7f0df5817640 1 --2- 172.21.15.125:0/1013950569 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0df00cefa0 0x7f0df00cf3d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009d90 tx=0x7f0de801f130 comp rx=0 tx=0).stop 2023-10-30T12:48:00.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.756+0000 7f0df5817640 1 -- 172.21.15.125:0/1013950569 shutdown_connections 2023-10-30T12:48:00.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.756+0000 7f0df5817640 1 --2- 172.21.15.125:0/1013950569 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0df00cefa0 0x7f0df00cf3d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.756+0000 7f0df5817640 1 --2- 172.21.15.125:0/1013950569 >> v2:172.21.15.136:3300/0 conn(0x7f0df015cd50 0x7f0df00c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.756+0000 7f0df5817640 1 -- 172.21.15.125:0/1013950569 >> 172.21.15.125:0/1013950569 conn(0x7f0df006a640 msgr2=0x7f0df0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:00.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.756+0000 7f0df5817640 1 -- 172.21.15.125:0/1013950569 shutdown_connections 2023-10-30T12:48:00.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.756+0000 7f0df5817640 1 -- 172.21.15.125:0/1013950569 wait complete. 2023-10-30T12:48:00.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5817640 1 Processor -- start 2023-10-30T12:48:00.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5817640 1 -- start start 2023-10-30T12:48:00.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5817640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0df00cefa0 0x7f0df0144b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:00.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5817640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0df01538f0 0x7f0df0152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:00.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5817640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0df015cd50 0x7f0df0142ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:00.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5817640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df00c4a20 con 0x7f0df015cd50 2023-10-30T12:48:00.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5817640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df00b52d0 con 0x7f0df00cefa0 2023-10-30T12:48:00.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5817640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0df0158bd0 con 0x7f0df01538f0 2023-10-30T12:48:00.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5016640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0df015cd50 0x7f0df0142ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:00.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5016640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0df015cd50 0x7f0df0142ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57520/0 (socket says 172.21.15.125:57520) 2023-10-30T12:48:00.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5016640 1 -- 172.21.15.125:0/1338906556 learned_addr learned my addr 172.21.15.125:0/1338906556 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:00.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df4815640 1 --2- 172.21.15.125:0/1338906556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0df00cefa0 0x7f0df0144b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:00.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0deffff640 1 --2- 172.21.15.125:0/1338906556 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0df01538f0 0x7f0df0152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:00.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5016640 1 -- 172.21.15.125:0/1338906556 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0df01538f0 msgr2=0x7f0df0152ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:00.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5016640 1 --2- 172.21.15.125:0/1338906556 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0df01538f0 0x7f0df0152ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5016640 1 -- 172.21.15.125:0/1338906556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0df00cefa0 msgr2=0x7f0df0144b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:00.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5016640 1 --2- 172.21.15.125:0/1338906556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0df00cefa0 0x7f0df0144b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5016640 1 -- 172.21.15.125:0/1338906556 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de8009a70 con 0x7f0df015cd50 2023-10-30T12:48:00.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5016640 1 --2- 172.21.15.125:0/1338906556 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0df015cd50 0x7f0df0142ba0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009d90 tx=0x7f0de801bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:00.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0ded7fa640 1 -- 172.21.15.125:0/1338906556 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de8009450 con 0x7f0df015cd50 2023-10-30T12:48:00.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0ded7fa640 1 -- 172.21.15.125:0/1338906556 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de80b9410 con 0x7f0df015cd50 2023-10-30T12:48:00.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0ded7fa640 1 -- 172.21.15.125:0/1338906556 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de80b97c0 con 0x7f0df015cd50 2023-10-30T12:48:00.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5817640 1 -- 172.21.15.125:0/1338906556 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df015b680 con 0x7f0df015cd50 2023-10-30T12:48:00.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5817640 1 -- 172.21.15.125:0/1338906556 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df00b4ff0 con 0x7f0df015cd50 2023-10-30T12:48:00.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.760+0000 7f0df5817640 1 -- 172.21.15.125:0/1338906556 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df015b810 con 0x7f0df015cd50 2023-10-30T12:48:00.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.764+0000 7f0ded7fa640 1 -- 172.21.15.125:0/1338906556 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0de80b6050 con 0x7f0df015cd50 2023-10-30T12:48:00.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.764+0000 7f0ded7fa640 1 --2- 172.21.15.125:0/1338906556 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0dd40664d0 0x7f0dd4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:00.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.764+0000 7f0ded7fa640 1 -- 172.21.15.125:0/1338906556 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f0de81225b0 con 0x7f0df015cd50 2023-10-30T12:48:00.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.764+0000 7f0df4815640 1 --2- 172.21.15.125:0/1338906556 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0dd40664d0 0x7f0dd4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:00.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.764+0000 7f0df4815640 1 --2- 172.21.15.125:0/1338906556 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0dd40664d0 0x7f0dd4068990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0de40011f0 tx=0x7f0de4007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:00.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.768+0000 7f0ded7fa640 1 -- 172.21.15.125:0/1338906556 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0de80ef6b0 con 0x7f0df015cd50 2023-10-30T12:48:00.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.948+0000 7f0df5817640 1 -- 172.21.15.125:0/1338906556 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0df004e170 con 0x7f0df015cd50 2023-10-30T12:48:00.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.948+0000 7f0ded7fa640 1 -- 172.21.15.125:0/1338906556 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+5737 (secure 0 0 0) 0x7f0de80b4090 con 0x7f0df015cd50 2023-10-30T12:48:00.952 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:00.952 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:56.562576+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:00.953 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 97 2023-10-30T12:48:00.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.952+0000 7f0df5817640 1 -- 172.21.15.125:0/1338906556 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0dd40664d0 msgr2=0x7f0dd4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:00.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.952+0000 7f0df5817640 1 --2- 172.21.15.125:0/1338906556 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0dd40664d0 0x7f0dd4068990 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0de40011f0 tx=0x7f0de4007d40 comp rx=0 tx=0).stop 2023-10-30T12:48:00.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.952+0000 7f0df5817640 1 -- 172.21.15.125:0/1338906556 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0df015cd50 msgr2=0x7f0df0142ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:00.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.952+0000 7f0df5817640 1 --2- 172.21.15.125:0/1338906556 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0df015cd50 0x7f0df0142ba0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f0de8009d90 tx=0x7f0de801bed0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.952+0000 7f0df5817640 1 -- 172.21.15.125:0/1338906556 shutdown_connections 2023-10-30T12:48:00.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.952+0000 7f0df5817640 1 --2- 172.21.15.125:0/1338906556 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0dd40664d0 0x7f0dd4068990 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.952+0000 7f0df5817640 1 --2- 172.21.15.125:0/1338906556 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0df015cd50 0x7f0df0142ba0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.952+0000 7f0df5817640 1 --2- 172.21.15.125:0/1338906556 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0df01538f0 0x7f0df0152ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.952+0000 7f0df5817640 1 --2- 172.21.15.125:0/1338906556 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0df00cefa0 0x7f0df0144b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:00.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.952+0000 7f0df5817640 1 -- 172.21.15.125:0/1338906556 >> 172.21.15.125:0/1338906556 conn(0x7f0df006a640 msgr2=0x7f0df00c4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:00.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.952+0000 7f0df5817640 1 -- 172.21.15.125:0/1338906556 shutdown_connections 2023-10-30T12:48:00.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:00.952+0000 7f0df5817640 1 -- 172.21.15.125:0/1338906556 wait complete. 2023-10-30T12:48:00.972 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T12:48:02.973 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:03.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c4e3d3640 1 Processor -- start 2023-10-30T12:48:03.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c4e3d3640 1 -- start start 2023-10-30T12:48:03.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c4e3d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2c480c6b80 0x7f2c480bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:03.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c4e3d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2c4813f4a0 0x7f2c480c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:03.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c4e3d3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c480c0a50 con 0x7f2c480c6f60 2023-10-30T12:48:03.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c4e3d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c4806f6e0 con 0x7f2c480c6b80 2023-10-30T12:48:03.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c4e3d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c4806f9c0 con 0x7f2c4813f4a0 2023-10-30T12:48:03.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c47fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2c480c6b80 0x7f2c480bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:03.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c47fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2c480c6b80 0x7f2c480bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57524/0 (socket says 172.21.15.125:57524) 2023-10-30T12:48:03.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c47fff640 1 -- 172.21.15.125:0/190061264 learned_addr learned my addr 172.21.15.125:0/190061264 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:03.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c4c970640 1 --2- 172.21.15.125:0/190061264 >> v2:172.21.15.136:3300/0 conn(0x7f2c4813f4a0 0x7f2c480c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:03.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c47fff640 1 -- 172.21.15.125:0/190061264 >> v2:172.21.15.136:3300/0 conn(0x7f2c4813f4a0 msgr2=0x7f2c480c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:03.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c47fff640 1 --2- 172.21.15.125:0/190061264 >> v2:172.21.15.136:3300/0 conn(0x7f2c4813f4a0 0x7f2c480c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:03.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c47fff640 1 -- 172.21.15.125:0/190061264 >> v1:172.21.15.125:6789/0 conn(0x7f2c480c6f60 legacy=0x7f2c4813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:03.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.112+0000 7f2c47fff640 1 -- 172.21.15.125:0/190061264 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4806fca0 con 0x7f2c480c6b80 2023-10-30T12:48:03.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.116+0000 7f2c47fff640 1 --2- 172.21.15.125:0/190061264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c480c6b80 0x7f2c480bfec0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f2c480c8980 tx=0x7f2c3801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd28c5b20af71144 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:03.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.116+0000 7f2c46ffd640 1 -- 172.21.15.125:0/190061264 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c380ab610 con 0x7f2c480c6b80 2023-10-30T12:48:03.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.116+0000 7f2c46ffd640 1 -- 172.21.15.125:0/190061264 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c380ab770 con 0x7f2c480c6b80 2023-10-30T12:48:03.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.116+0000 7f2c46ffd640 1 -- 172.21.15.125:0/190061264 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c380abb00 con 0x7f2c480c6b80 2023-10-30T12:48:03.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.116+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/190061264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c480c6b80 msgr2=0x7f2c480bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:03.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.116+0000 7f2c4e3d3640 1 --2- 172.21.15.125:0/190061264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c480c6b80 0x7f2c480bfec0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f2c480c8980 tx=0x7f2c3801c500 comp rx=0 tx=0).stop 2023-10-30T12:48:03.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.116+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/190061264 shutdown_connections 2023-10-30T12:48:03.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.116+0000 7f2c4e3d3640 1 --2- 172.21.15.125:0/190061264 >> v2:172.21.15.136:3300/0 conn(0x7f2c4813f4a0 0x7f2c480c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:03.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.116+0000 7f2c4e3d3640 1 --2- 172.21.15.125:0/190061264 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c480c6b80 0x7f2c480bfec0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:03.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.116+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/190061264 >> 172.21.15.125:0/190061264 conn(0x7f2c480c7910 msgr2=0x7f2c480c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:03.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.116+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/190061264 shutdown_connections 2023-10-30T12:48:03.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.116+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/190061264 wait complete. 2023-10-30T12:48:03.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c4e3d3640 1 Processor -- start 2023-10-30T12:48:03.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c4e3d3640 1 -- start start 2023-10-30T12:48:03.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c4e3d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c480c6b80 0x7f2c48150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:03.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c4e3d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c480c6f60 0x7f2c48135440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:03.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c4e3d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c4813f4a0 0x7f2c48135980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:03.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c4e3d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c48136ca0 con 0x7f2c480c6f60 2023-10-30T12:48:03.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c4e3d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4813e7f0 con 0x7f2c480c6b80 2023-10-30T12:48:03.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c4e3d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4813e230 con 0x7f2c4813f4a0 2023-10-30T12:48:03.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c477fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c480c6f60 0x7f2c48135440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:03.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c477fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c480c6f60 0x7f2c48135440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57540/0 (socket says 172.21.15.125:57540) 2023-10-30T12:48:03.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c477fe640 1 -- 172.21.15.125:0/218192185 learned_addr learned my addr 172.21.15.125:0/218192185 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:03.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c47fff640 1 --2- 172.21.15.125:0/218192185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c480c6b80 0x7f2c48150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:03.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c4c970640 1 --2- 172.21.15.125:0/218192185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c4813f4a0 0x7f2c48135980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:03.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c477fe640 1 -- 172.21.15.125:0/218192185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c4813f4a0 msgr2=0x7f2c48135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:03.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c477fe640 1 --2- 172.21.15.125:0/218192185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c4813f4a0 0x7f2c48135980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:03.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c477fe640 1 -- 172.21.15.125:0/218192185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c480c6b80 msgr2=0x7f2c48150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:03.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c477fe640 1 --2- 172.21.15.125:0/218192185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c480c6b80 0x7f2c48150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:03.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c477fe640 1 -- 172.21.15.125:0/218192185 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c380094d0 con 0x7f2c480c6f60 2023-10-30T12:48:03.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c477fe640 1 --2- 172.21.15.125:0/218192185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c480c6f60 0x7f2c48135440 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00c530 tx=0x7f2c3c00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:03.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c44ff9640 1 -- 172.21.15.125:0/218192185 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c005220 con 0x7f2c480c6f60 2023-10-30T12:48:03.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c44ff9640 1 -- 172.21.15.125:0/218192185 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c3c005380 con 0x7f2c480c6f60 2023-10-30T12:48:03.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c44ff9640 1 -- 172.21.15.125:0/218192185 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c3c007270 con 0x7f2c480c6f60 2023-10-30T12:48:03.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/218192185 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c481561e0 con 0x7f2c480c6f60 2023-10-30T12:48:03.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.120+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/218192185 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c4813e480 con 0x7f2c480c6f60 2023-10-30T12:48:03.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.124+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/218192185 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c4813e640 con 0x7f2c480c6f60 2023-10-30T12:48:03.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.124+0000 7f2c44ff9640 1 -- 172.21.15.125:0/218192185 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c3c0073d0 con 0x7f2c480c6f60 2023-10-30T12:48:03.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.124+0000 7f2c44ff9640 1 --2- 172.21.15.125:0/218192185 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c340664a0 0x7f2c34068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:03.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.124+0000 7f2c44ff9640 1 -- 172.21.15.125:0/218192185 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f2c3c083a10 con 0x7f2c480c6f60 2023-10-30T12:48:03.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.124+0000 7f2c47fff640 1 --2- 172.21.15.125:0/218192185 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c340664a0 0x7f2c34068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:03.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.124+0000 7f2c47fff640 1 --2- 172.21.15.125:0/218192185 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c340664a0 0x7f2c34068960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2c380af000 tx=0x7f2c380b6040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:03.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.128+0000 7f2c44ff9640 1 -- 172.21.15.125:0/218192185 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c3c050b80 con 0x7f2c480c6f60 2023-10-30T12:48:03.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.332+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/218192185 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c4804e170 con 0x7f2c480c6f60 2023-10-30T12:48:03.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.332+0000 7f2c44ff9640 1 -- 172.21.15.125:0/218192185 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+6531 (secure 0 0 0) 0x7f2c3c0541a0 con 0x7f2c480c6f60 2023-10-30T12:48:03.335 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:03.335 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:47:56.562576+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:03.336 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 98 2023-10-30T12:48:03.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.336+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/218192185 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c340664a0 msgr2=0x7f2c34068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:03.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.336+0000 7f2c4e3d3640 1 --2- 172.21.15.125:0/218192185 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c340664a0 0x7f2c34068960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f2c380af000 tx=0x7f2c380b6040 comp rx=0 tx=0).stop 2023-10-30T12:48:03.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.336+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/218192185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c480c6f60 msgr2=0x7f2c48135440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:03.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.336+0000 7f2c4e3d3640 1 --2- 172.21.15.125:0/218192185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c480c6f60 0x7f2c48135440 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f2c3c00c530 tx=0x7f2c3c00c9f0 comp rx=0 tx=0).stop 2023-10-30T12:48:03.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.336+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/218192185 shutdown_connections 2023-10-30T12:48:03.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.336+0000 7f2c4e3d3640 1 --2- 172.21.15.125:0/218192185 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c340664a0 0x7f2c34068960 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:03.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.336+0000 7f2c4e3d3640 1 --2- 172.21.15.125:0/218192185 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c4813f4a0 0x7f2c48135980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:03.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.336+0000 7f2c4e3d3640 1 --2- 172.21.15.125:0/218192185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c480c6f60 0x7f2c48135440 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:03.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.336+0000 7f2c4e3d3640 1 --2- 172.21.15.125:0/218192185 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c480c6b80 0x7f2c48150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:03.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.336+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/218192185 >> 172.21.15.125:0/218192185 conn(0x7f2c480c7910 msgr2=0x7f2c480c13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:03.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.336+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/218192185 shutdown_connections 2023-10-30T12:48:03.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:03.336+0000 7f2c4e3d3640 1 -- 172.21.15.125:0/218192185 wait complete. 2023-10-30T12:48:03.356 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T12:48:03.356 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before thrashing 2023-10-30T12:48:07.248 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:48:07.248+0000 7f2dd3c0c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:48:07.248 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:48:07.248+0000 7f2dd3c0c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:48:09.357 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:09.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.504+0000 7f031a0a9640 1 Processor -- start 2023-10-30T12:48:09.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.504+0000 7f031a0a9640 1 -- start start 2023-10-30T12:48:09.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f031a0a9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03140c05d0 0x7f031413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f031a0a9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f03140c6f60 0x7f031413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f031a0a9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03140c0a50 con 0x7f03140c6b80 2023-10-30T12:48:09.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f031a0a9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f031406f6e0 con 0x7f03140c6f60 2023-10-30T12:48:09.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f031a0a9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f031406f9c0 con 0x7f03140c05d0 2023-10-30T12:48:09.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f03137fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f03140c6b80 0x7f03140bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44518/0 (socket says 172.21.15.125:44518) 2023-10-30T12:48:09.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f03137fe640 1 -- 172.21.15.125:0/3267609614 learned_addr learned my addr 172.21.15.125:0/3267609614 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:48:09.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f0313fff640 1 --2- 172.21.15.125:0/3267609614 >> v2:172.21.15.125:3300/0 conn(0x7f03140c6f60 0x7f031413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f0312ffd640 1 --2- 172.21.15.125:0/3267609614 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03140c05d0 0x7f031413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f03127fc640 1 -- 172.21.15.125:0/3267609614 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736513884 0 0) 0x7f03140c0a50 con 0x7f03140c6b80 2023-10-30T12:48:09.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f03127fc640 1 -- 172.21.15.125:0/3267609614 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02fc005740 con 0x7f03140c6b80 2023-10-30T12:48:09.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f0313fff640 1 -- 172.21.15.125:0/3267609614 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03140c05d0 msgr2=0x7f031413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:09.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f0313fff640 1 --2- 172.21.15.125:0/3267609614 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03140c05d0 0x7f031413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f0313fff640 1 -- 172.21.15.125:0/3267609614 >> v1:172.21.15.125:6789/0 conn(0x7f03140c6b80 legacy=0x7f03140bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.508+0000 7f0313fff640 1 -- 172.21.15.125:0/3267609614 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031406fca0 con 0x7f03140c6f60 2023-10-30T12:48:09.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f0313fff640 1 --2- 172.21.15.125:0/3267609614 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03140c6f60 0x7f031413f2d0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f030c009d90 tx=0x7f030c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1977c7e0a7e845d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:09.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f03127fc640 1 -- 172.21.15.125:0/3267609614 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030c0a5920 con 0x7f03140c6f60 2023-10-30T12:48:09.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f03127fc640 1 -- 172.21.15.125:0/3267609614 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f030c0a5a80 con 0x7f03140c6f60 2023-10-30T12:48:09.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f03127fc640 1 -- 172.21.15.125:0/3267609614 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030c0a5e10 con 0x7f03140c6f60 2023-10-30T12:48:09.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 -- 172.21.15.125:0/3267609614 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03140c6f60 msgr2=0x7f031413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 --2- 172.21.15.125:0/3267609614 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03140c6f60 0x7f031413f2d0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f030c009d90 tx=0x7f030c01c950 comp rx=0 tx=0).stop 2023-10-30T12:48:09.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 -- 172.21.15.125:0/3267609614 shutdown_connections 2023-10-30T12:48:09.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 --2- 172.21.15.125:0/3267609614 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03140c6f60 0x7f031413f2d0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 --2- 172.21.15.125:0/3267609614 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03140c05d0 0x7f031413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 -- 172.21.15.125:0/3267609614 >> 172.21.15.125:0/3267609614 conn(0x7f03140c7910 msgr2=0x7f03140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:09.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 -- 172.21.15.125:0/3267609614 shutdown_connections 2023-10-30T12:48:09.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 -- 172.21.15.125:0/3267609614 wait complete. 2023-10-30T12:48:09.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 Processor -- start 2023-10-30T12:48:09.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 -- start start 2023-10-30T12:48:09.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03140c05d0 0x7f03141337d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03140c6b80 0x7f031414d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03140c6f60 0x7f031414df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0314133340 con 0x7f03140c05d0 2023-10-30T12:48:09.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0314133d10 con 0x7f03140c6f60 2023-10-30T12:48:09.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.512+0000 7f031a0a9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f031406c6d0 con 0x7f03140c6b80 2023-10-30T12:48:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f03137fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03140c05d0 0x7f03141337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f03137fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03140c05d0 0x7f03141337d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57570/0 (socket says 172.21.15.125:57570) 2023-10-30T12:48:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f03137fe640 1 -- 172.21.15.125:0/4042422391 learned_addr learned my addr 172.21.15.125:0/4042422391 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f0313fff640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03140c6f60 0x7f031414df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f0312ffd640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03140c6b80 0x7f031414d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f03137fe640 1 -- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03140c6b80 msgr2=0x7f031414d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f03137fe640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03140c6b80 0x7f031414d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f03137fe640 1 -- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03140c6f60 msgr2=0x7f031414df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f03137fe640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03140c6f60 0x7f031414df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f03137fe640 1 -- 172.21.15.125:0/4042422391 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f030c009a70 con 0x7f03140c05d0 2023-10-30T12:48:09.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f0313fff640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03140c6f60 0x7f031414df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:09.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f03137fe640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03140c05d0 0x7f03141337d0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f030400c450 tx=0x7f030400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:09.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f02fbfff640 1 -- 172.21.15.125:0/4042422391 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0304005cf0 con 0x7f03140c05d0 2023-10-30T12:48:09.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f02fbfff640 1 -- 172.21.15.125:0/4042422391 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0304005e50 con 0x7f03140c05d0 2023-10-30T12:48:09.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f02fbfff640 1 -- 172.21.15.125:0/4042422391 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0304007290 con 0x7f03140c05d0 2023-10-30T12:48:09.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f031a0a9640 1 -- 172.21.15.125:0/4042422391 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03140afe50 con 0x7f03140c05d0 2023-10-30T12:48:09.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f031a0a9640 1 -- 172.21.15.125:0/4042422391 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03140b0410 con 0x7f03140c05d0 2023-10-30T12:48:09.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f031a0a9640 1 -- 172.21.15.125:0/4042422391 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f031404e170 con 0x7f03140c05d0 2023-10-30T12:48:09.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f02fbfff640 1 -- 172.21.15.125:0/4042422391 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f03040073f0 con 0x7f03140c05d0 2023-10-30T12:48:09.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.516+0000 7f02fbfff640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f03000664a0 0x7f0300068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.520+0000 7f02fbfff640 1 -- 172.21.15.125:0/4042422391 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f0304083870 con 0x7f03140c05d0 2023-10-30T12:48:09.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.520+0000 7f0312ffd640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f03000664a0 0x7f0300068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.520+0000 7f0312ffd640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f03000664a0 0x7f0300068960 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0314160a90 tx=0x7f0308008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:09.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.520+0000 7f02fbfff640 1 -- 172.21.15.125:0/4042422391 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f03040508c0 con 0x7f03140c05d0 2023-10-30T12:48:09.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.704+0000 7f031a0a9640 1 -- 172.21.15.125:0/4042422391 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0314131f40 con 0x7f03140c05d0 2023-10-30T12:48:09.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.704+0000 7f02fbfff640 1 -- 172.21.15.125:0/4042422391 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f0304017020 con 0x7f03140c05d0 2023-10-30T12:48:09.706 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:09.706 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:07.294062+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:09.707 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 99 2023-10-30T12:48:09.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.704+0000 7f031a0a9640 1 -- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f03000664a0 msgr2=0x7f0300068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.704+0000 7f031a0a9640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f03000664a0 0x7f0300068960 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0314160a90 tx=0x7f0308008040 comp rx=0 tx=0).stop 2023-10-30T12:48:09.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.704+0000 7f031a0a9640 1 -- 172.21.15.125:0/4042422391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03140c05d0 msgr2=0x7f03141337d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.704+0000 7f031a0a9640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03140c05d0 0x7f03141337d0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f030400c450 tx=0x7f030400c910 comp rx=0 tx=0).stop 2023-10-30T12:48:09.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.704+0000 7f031a0a9640 1 -- 172.21.15.125:0/4042422391 shutdown_connections 2023-10-30T12:48:09.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.704+0000 7f031a0a9640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f03000664a0 0x7f0300068960 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.704+0000 7f031a0a9640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f03140c6f60 0x7f031414df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.704+0000 7f031a0a9640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03140c6b80 0x7f031414d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.704+0000 7f031a0a9640 1 --2- 172.21.15.125:0/4042422391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f03140c05d0 0x7f03141337d0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.704+0000 7f031a0a9640 1 -- 172.21.15.125:0/4042422391 >> 172.21.15.125:0/4042422391 conn(0x7f03140c7910 msgr2=0x7f0314131260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:09.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.704+0000 7f031a0a9640 1 -- 172.21.15.125:0/4042422391 shutdown_connections 2023-10-30T12:48:09.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.704+0000 7f031a0a9640 1 -- 172.21.15.125:0/4042422391 wait complete. 2023-10-30T12:48:09.726 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=1) 2023-10-30T12:48:09.726 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T12:48:09.727 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:48:09.788 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T12:48:09.789 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:09.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e43279640 1 Processor -- start 2023-10-30T12:48:09.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e43279640 1 -- start start 2023-10-30T12:48:09.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e43279640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1e3c15d6f0 0x7f1e3c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e43279640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e3c0cf160 0x7f1e3c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e43279640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e3c0cb800 con 0x7f1e3c1538a0 2023-10-30T12:48:09.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e43279640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e3c0cb330 con 0x7f1e3c15d6f0 2023-10-30T12:48:09.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e43279640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c0cad70 con 0x7f1e3c0cf160 2023-10-30T12:48:09.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e41015640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1e3c1538a0 0x7f1e3c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44532/0 (socket says 172.21.15.125:44532) 2023-10-30T12:48:09.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e41015640 1 -- 172.21.15.125:0/3087310610 learned_addr learned my addr 172.21.15.125:0/3087310610 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:48:09.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e40814640 1 --2- 172.21.15.125:0/3087310610 >> v2:172.21.15.136:3300/0 conn(0x7f1e3c15d6f0 0x7f1e3c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e41816640 1 --2- 172.21.15.125:0/3087310610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e3c0cf160 0x7f1e3c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 -- 172.21.15.125:0/3087310610 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060339348 0 0) 0x7f1e3c0cb800 con 0x7f1e3c1538a0 2023-10-30T12:48:09.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 -- 172.21.15.125:0/3087310610 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e28003530 con 0x7f1e3c1538a0 2023-10-30T12:48:09.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 -- 172.21.15.125:0/3087310610 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 223182654 0 0) 0x7f1e28003530 con 0x7f1e3c1538a0 2023-10-30T12:48:09.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 -- 172.21.15.125:0/3087310610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e3c0cf160 msgr2=0x7f1e3c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:09.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 --2- 172.21.15.125:0/3087310610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e3c0cf160 0x7f1e3c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 -- 172.21.15.125:0/3087310610 >> v2:172.21.15.136:3300/0 conn(0x7f1e3c15d6f0 msgr2=0x7f1e3c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:09.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 --2- 172.21.15.125:0/3087310610 >> v2:172.21.15.136:3300/0 conn(0x7f1e3c15d6f0 0x7f1e3c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 -- 172.21.15.125:0/3087310610 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3c0cb050 con 0x7f1e3c1538a0 2023-10-30T12:48:09.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 -- 172.21.15.125:0/3087310610 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f1e38001680 con 0x7f1e3c1538a0 2023-10-30T12:48:09.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 -- 172.21.15.125:0/3087310610 >> v1:172.21.15.125:6789/0 conn(0x7f1e3c1538a0 legacy=0x7f1e3c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 --2- 172.21.15.125:0/3087310610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e3c0cf160 0x7f1e28003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 --2- 172.21.15.125:0/3087310610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e28003f00 0x7f1e280047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 --2- 172.21.15.125:0/3087310610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e28004d10 0x7f1e28005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 -- 172.21.15.125:0/3087310610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c0cb330 con 0x7f1e28003f00 2023-10-30T12:48:09.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 -- 172.21.15.125:0/3087310610 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c0cad70 con 0x7f1e28004d10 2023-10-30T12:48:09.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e33fff640 1 -- 172.21.15.125:0/3087310610 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2800b9c0 con 0x7f1e3c0cf160 2023-10-30T12:48:09.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e40814640 1 --2- 172.21.15.125:0/3087310610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e28003f00 0x7f1e280047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e41015640 1 --2- 172.21.15.125:0/3087310610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e3c0cf160 0x7f1e28003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e41816640 1 --2- 172.21.15.125:0/3087310610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e28004d10 0x7f1e28005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e40814640 1 -- 172.21.15.125:0/3087310610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e3c0cf160 msgr2=0x7f1e28003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:09.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e40814640 1 --2- 172.21.15.125:0/3087310610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e3c0cf160 0x7f1e28003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e40814640 1 -- 172.21.15.125:0/3087310610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e28004d10 msgr2=0x7f1e28005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:09.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e40814640 1 --2- 172.21.15.125:0/3087310610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e28004d10 0x7f1e28005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.968+0000 7f1e40814640 1 -- 172.21.15.125:0/3087310610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e28003530 con 0x7f1e28003f00 2023-10-30T12:48:09.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.972+0000 7f1e40814640 1 --2- 172.21.15.125:0/3087310610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e28003f00 0x7f1e280047d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c009ae0 tx=0x7f1e2c01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f680129eca98811 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:09.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.972+0000 7f1e33fff640 1 -- 172.21.15.125:0/3087310610 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c0a5680 con 0x7f1e28003f00 2023-10-30T12:48:09.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.972+0000 7f1e33fff640 1 -- 172.21.15.125:0/3087310610 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2c0a57e0 con 0x7f1e28003f00 2023-10-30T12:48:09.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.972+0000 7f1e33fff640 1 -- 172.21.15.125:0/3087310610 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2c0a5b90 con 0x7f1e28003f00 2023-10-30T12:48:09.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.972+0000 7f1e43279640 1 -- 172.21.15.125:0/3087310610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e28003f00 msgr2=0x7f1e280047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.972+0000 7f1e43279640 1 --2- 172.21.15.125:0/3087310610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e28003f00 0x7f1e280047d0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c009ae0 tx=0x7f1e2c01ca90 comp rx=0 tx=0).stop 2023-10-30T12:48:09.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.972+0000 7f1e41816640 1 -- 172.21.15.125:0/3087310610 reap_dead start 2023-10-30T12:48:09.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.972+0000 7f1e43279640 1 -- 172.21.15.125:0/3087310610 shutdown_connections 2023-10-30T12:48:09.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.972+0000 7f1e43279640 1 -- 172.21.15.125:0/3087310610 >> 172.21.15.125:0/3087310610 conn(0x7f1e3c06bfe0 msgr2=0x7f1e3c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:09.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.972+0000 7f1e43279640 1 -- 172.21.15.125:0/3087310610 shutdown_connections 2023-10-30T12:48:09.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.972+0000 7f1e43279640 1 -- 172.21.15.125:0/3087310610 wait complete. 2023-10-30T12:48:09.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e43279640 1 Processor -- start 2023-10-30T12:48:09.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e43279640 1 -- start start 2023-10-30T12:48:09.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e43279640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e28003f00 0x7f1e3c152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e43279640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e3c1ef1d0 0x7f1e3c157be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e43279640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e3c1e2a40 0x7f1e3c158120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e43279640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c0c6780 con 0x7f1e3c1e2a40 2023-10-30T12:48:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e43279640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c157750 con 0x7f1e3c1ef1d0 2023-10-30T12:48:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e43279640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3c150b80 con 0x7f1e28003f00 2023-10-30T12:48:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e41816640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e3c1e2a40 0x7f1e3c158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e41816640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e3c1e2a40 0x7f1e3c158120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57598/0 (socket says 172.21.15.125:57598) 2023-10-30T12:48:09.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e41816640 1 -- 172.21.15.125:0/3524508592 learned_addr learned my addr 172.21.15.125:0/3524508592 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:09.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e41015640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e28003f00 0x7f1e3c152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e40814640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e3c1ef1d0 0x7f1e3c157be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e41816640 1 -- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e28003f00 msgr2=0x7f1e3c152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e41816640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e28003f00 0x7f1e3c152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e41816640 1 -- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e3c1ef1d0 msgr2=0x7f1e3c157be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:09.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e41816640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e3c1ef1d0 0x7f1e3c157be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:09.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e41816640 1 -- 172.21.15.125:0/3524508592 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3c1ef6c0 con 0x7f1e3c1e2a40 2023-10-30T12:48:09.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e41015640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e28003f00 0x7f1e3c152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:48:09.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e41816640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e3c1e2a40 0x7f1e3c158120 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c072ce0 tx=0x7f1e340115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:09.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e31ffb640 1 -- 172.21.15.125:0/3524508592 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3400e070 con 0x7f1e3c1e2a40 2023-10-30T12:48:09.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e31ffb640 1 -- 172.21.15.125:0/3524508592 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e34011de0 con 0x7f1e3c1e2a40 2023-10-30T12:48:09.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e43279640 1 -- 172.21.15.125:0/3524508592 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e3c1ef9a0 con 0x7f1e3c1e2a40 2023-10-30T12:48:09.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e31ffb640 1 -- 172.21.15.125:0/3524508592 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e340039a0 con 0x7f1e3c1e2a40 2023-10-30T12:48:09.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.976+0000 7f1e43279640 1 -- 172.21.15.125:0/3524508592 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e3c1eff60 con 0x7f1e3c1e2a40 2023-10-30T12:48:09.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.980+0000 7f1e43279640 1 -- 172.21.15.125:0/3524508592 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3c1f0150 con 0x7f1e3c1e2a40 2023-10-30T12:48:09.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.980+0000 7f1e31ffb640 1 -- 172.21.15.125:0/3524508592 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1e34003b00 con 0x7f1e3c1e2a40 2023-10-30T12:48:09.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.980+0000 7f1e31ffb640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1e200664d0 0x7f1e20068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:09.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.980+0000 7f1e31ffb640 1 -- 172.21.15.125:0/3524508592 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f1e340846c0 con 0x7f1e3c1e2a40 2023-10-30T12:48:09.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.980+0000 7f1e41015640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1e200664d0 0x7f1e20068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:09.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.980+0000 7f1e41015640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1e200664d0 0x7f1e20068990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1e38000f70 tx=0x7f1e38000ee0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:09.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:09.984+0000 7f1e31ffb640 1 -- 172.21.15.125:0/3524508592 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1e34051710 con 0x7f1e3c1e2a40 2023-10-30T12:48:10.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:10.164+0000 7f1e43279640 1 -- 172.21.15.125:0/3524508592 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e3c04e170 con 0x7f1e3c1e2a40 2023-10-30T12:48:10.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:10.164+0000 7f1e31ffb640 1 -- 172.21.15.125:0/3524508592 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f1e34054d30 con 0x7f1e3c1e2a40 2023-10-30T12:48:10.165 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:10.165 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:07.294062+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:10.166 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 99 2023-10-30T12:48:10.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:10.164+0000 7f1e43279640 1 -- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1e200664d0 msgr2=0x7f1e20068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:10.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:10.164+0000 7f1e43279640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1e200664d0 0x7f1e20068990 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f1e38000f70 tx=0x7f1e38000ee0 comp rx=0 tx=0).stop 2023-10-30T12:48:10.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:10.164+0000 7f1e43279640 1 -- 172.21.15.125:0/3524508592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e3c1e2a40 msgr2=0x7f1e3c158120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:10.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:10.164+0000 7f1e43279640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e3c1e2a40 0x7f1e3c158120 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f1e3c072ce0 tx=0x7f1e340115a0 comp rx=0 tx=0).stop 2023-10-30T12:48:10.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:10.164+0000 7f1e43279640 1 -- 172.21.15.125:0/3524508592 shutdown_connections 2023-10-30T12:48:10.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:10.164+0000 7f1e43279640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1e200664d0 0x7f1e20068990 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:10.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:10.164+0000 7f1e43279640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1e3c1e2a40 0x7f1e3c158120 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:10.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:10.164+0000 7f1e43279640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1e3c1ef1d0 0x7f1e3c157be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:10.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:10.164+0000 7f1e43279640 1 --2- 172.21.15.125:0/3524508592 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1e28003f00 0x7f1e3c152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:10.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:10.164+0000 7f1e43279640 1 -- 172.21.15.125:0/3524508592 >> 172.21.15.125:0/3524508592 conn(0x7f1e3c06bfe0 msgr2=0x7f1e3c0c4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:10.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:10.164+0000 7f1e43279640 1 -- 172.21.15.125:0/3524508592 shutdown_connections 2023-10-30T12:48:10.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:10.164+0000 7f1e43279640 1 -- 172.21.15.125:0/3524508592 wait complete. 2023-10-30T12:48:10.186 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:48:12.187 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:12.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.328+0000 7f1524ff2640 1 Processor -- start 2023-10-30T12:48:12.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f1524ff2640 1 -- start start 2023-10-30T12:48:12.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f1524ff2640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15201538f0 0x7f1520159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f1524ff2640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15200cefa0 0x7f15200cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f1524ff2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15200ca6a0 con 0x7f152015cd50 2023-10-30T12:48:12.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f1524ff2640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f152015d300 con 0x7f15201538f0 2023-10-30T12:48:12.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f1524ff2640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f152015d5e0 con 0x7f15200cefa0 2023-10-30T12:48:12.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f151f7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15201538f0 0x7f1520159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f151f7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15201538f0 0x7f1520159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55728/0 (socket says 172.21.15.125:55728) 2023-10-30T12:48:12.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f151f7fe640 1 -- 172.21.15.125:0/189623698 learned_addr learned my addr 172.21.15.125:0/189623698 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:12.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f151ffff640 1 --2- 172.21.15.125:0/189623698 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15200cefa0 0x7f15200cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f151f7fe640 1 -- 172.21.15.125:0/189623698 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15200cefa0 msgr2=0x7f15200cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:12.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f151f7fe640 1 --2- 172.21.15.125:0/189623698 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15200cefa0 0x7f15200cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f151f7fe640 1 -- 172.21.15.125:0/189623698 >> v1:172.21.15.136:6789/0 conn(0x7f152015cd50 legacy=0x7f15200c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:12.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f151f7fe640 1 -- 172.21.15.125:0/189623698 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152015d8c0 con 0x7f15201538f0 2023-10-30T12:48:12.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.332+0000 7f151ffff640 1 --2- 172.21.15.125:0/189623698 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15200cefa0 0x7f15200cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:48:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.336+0000 7f151f7fe640 1 --2- 172.21.15.125:0/189623698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15201538f0 0x7f1520159fb0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1510009870 tx=0x7f151001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5428fbd5504163 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.336+0000 7f151effd640 1 -- 172.21.15.125:0/189623698 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15100a8a30 con 0x7f15201538f0 2023-10-30T12:48:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.336+0000 7f151effd640 1 -- 172.21.15.125:0/189623698 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15100a8b90 con 0x7f15201538f0 2023-10-30T12:48:12.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.336+0000 7f151effd640 1 -- 172.21.15.125:0/189623698 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15100b0440 con 0x7f15201538f0 2023-10-30T12:48:12.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.336+0000 7f1524ff2640 1 -- 172.21.15.125:0/189623698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15201538f0 msgr2=0x7f1520159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:12.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.336+0000 7f1524ff2640 1 --2- 172.21.15.125:0/189623698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15201538f0 0x7f1520159fb0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f1510009870 tx=0x7f151001ed30 comp rx=0 tx=0).stop 2023-10-30T12:48:12.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.336+0000 7f1524ff2640 1 -- 172.21.15.125:0/189623698 shutdown_connections 2023-10-30T12:48:12.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.336+0000 7f1524ff2640 1 --2- 172.21.15.125:0/189623698 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15200cefa0 0x7f15200cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.336+0000 7f1524ff2640 1 --2- 172.21.15.125:0/189623698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15201538f0 0x7f1520159fb0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.336+0000 7f1524ff2640 1 -- 172.21.15.125:0/189623698 >> 172.21.15.125:0/189623698 conn(0x7f152006a640 msgr2=0x7f1520065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:12.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.336+0000 7f1524ff2640 1 -- 172.21.15.125:0/189623698 shutdown_connections 2023-10-30T12:48:12.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.336+0000 7f1524ff2640 1 -- 172.21.15.125:0/189623698 wait complete. 2023-10-30T12:48:12.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.336+0000 7f1524ff2640 1 Processor -- start 2023-10-30T12:48:12.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.336+0000 7f1524ff2640 1 -- start start 2023-10-30T12:48:12.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1524ff2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15200cefa0 0x7f152015b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1524ff2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15201538f0 0x7f1520147e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1524ff2640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f152015cd50 0x7f1520152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1524ff2640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15201efb50 con 0x7f15201538f0 2023-10-30T12:48:12.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1524ff2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f152013ed50 con 0x7f15200cefa0 2023-10-30T12:48:12.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1524ff2640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1520148370 con 0x7f152015cd50 2023-10-30T12:48:12.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1517fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15201538f0 0x7f1520147e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1517fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15201538f0 0x7f1520147e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55734/0 (socket says 172.21.15.125:55734) 2023-10-30T12:48:12.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1517fff640 1 -- 172.21.15.125:0/1910126033 learned_addr learned my addr 172.21.15.125:0/1910126033 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:12.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f151f7fe640 1 --2- 172.21.15.125:0/1910126033 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15200cefa0 0x7f152015b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f151ffff640 1 --2- 172.21.15.125:0/1910126033 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f152015cd50 0x7f1520152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1517fff640 1 -- 172.21.15.125:0/1910126033 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f152015cd50 msgr2=0x7f1520152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:12.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1517fff640 1 --2- 172.21.15.125:0/1910126033 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f152015cd50 0x7f1520152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1517fff640 1 -- 172.21.15.125:0/1910126033 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15200cefa0 msgr2=0x7f152015b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:12.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1517fff640 1 --2- 172.21.15.125:0/1910126033 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15200cefa0 0x7f152015b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1517fff640 1 -- 172.21.15.125:0/1910126033 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15100094d0 con 0x7f15201538f0 2023-10-30T12:48:12.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1517fff640 1 --2- 172.21.15.125:0/1910126033 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15201538f0 0x7f1520147e30 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f150800c3f0 tx=0x7f150800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f151cff9640 1 -- 172.21.15.125:0/1910126033 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15080052d0 con 0x7f15201538f0 2023-10-30T12:48:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f151cff9640 1 -- 172.21.15.125:0/1910126033 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1508005430 con 0x7f15201538f0 2023-10-30T12:48:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f151cff9640 1 -- 172.21.15.125:0/1910126033 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1508003840 con 0x7f15201538f0 2023-10-30T12:48:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1524ff2640 1 -- 172.21.15.125:0/1910126033 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152013de30 con 0x7f15201538f0 2023-10-30T12:48:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1524ff2640 1 -- 172.21.15.125:0/1910126033 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15201f0010 con 0x7f15201538f0 2023-10-30T12:48:12.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.340+0000 7f1524ff2640 1 -- 172.21.15.125:0/1910126033 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152013ee90 con 0x7f15201538f0 2023-10-30T12:48:12.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.344+0000 7f151cff9640 1 -- 172.21.15.125:0/1910126033 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f15080039a0 con 0x7f15201538f0 2023-10-30T12:48:12.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.344+0000 7f151cff9640 1 --2- 172.21.15.125:0/1910126033 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15000664a0 0x7f1500068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:12.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.344+0000 7f151cff9640 1 -- 172.21.15.125:0/1910126033 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f1508083780 con 0x7f15201538f0 2023-10-30T12:48:12.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.344+0000 7f151f7fe640 1 --2- 172.21.15.125:0/1910126033 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15000664a0 0x7f1500068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:12.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.344+0000 7f151f7fe640 1 --2- 172.21.15.125:0/1910126033 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15000664a0 0x7f1500068960 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f15100b2000 tx=0x7f1510008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:12.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.348+0000 7f151cff9640 1 -- 172.21.15.125:0/1910126033 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f152013ee90 con 0x7f15201538f0 2023-10-30T12:48:12.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.532+0000 7f1524ff2640 1 -- 172.21.15.125:0/1910126033 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f152013ee90 con 0x7f15201538f0 2023-10-30T12:48:12.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.532+0000 7f151cff9640 1 -- 172.21.15.125:0/1910126033 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f152013ee90 con 0x7f15201538f0 2023-10-30T12:48:12.534 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:12.535 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:07.294062+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:12.536 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 99 2023-10-30T12:48:12.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.532+0000 7f1524ff2640 1 -- 172.21.15.125:0/1910126033 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15000664a0 msgr2=0x7f1500068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:12.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.532+0000 7f1524ff2640 1 --2- 172.21.15.125:0/1910126033 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15000664a0 0x7f1500068960 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f15100b2000 tx=0x7f1510008040 comp rx=0 tx=0).stop 2023-10-30T12:48:12.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.532+0000 7f1524ff2640 1 -- 172.21.15.125:0/1910126033 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15201538f0 msgr2=0x7f1520147e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:12.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.532+0000 7f1524ff2640 1 --2- 172.21.15.125:0/1910126033 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15201538f0 0x7f1520147e30 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f150800c3f0 tx=0x7f150800c8b0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.532+0000 7f1524ff2640 1 -- 172.21.15.125:0/1910126033 shutdown_connections 2023-10-30T12:48:12.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.532+0000 7f1524ff2640 1 --2- 172.21.15.125:0/1910126033 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15000664a0 0x7f1500068960 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.532+0000 7f1524ff2640 1 --2- 172.21.15.125:0/1910126033 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f152015cd50 0x7f1520152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.532+0000 7f1524ff2640 1 --2- 172.21.15.125:0/1910126033 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15201538f0 0x7f1520147e30 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.532+0000 7f1524ff2640 1 --2- 172.21.15.125:0/1910126033 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15200cefa0 0x7f152015b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:12.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.532+0000 7f1524ff2640 1 -- 172.21.15.125:0/1910126033 >> 172.21.15.125:0/1910126033 conn(0x7f152006a640 msgr2=0x7f15200c9ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:12.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.532+0000 7f1524ff2640 1 -- 172.21.15.125:0/1910126033 shutdown_connections 2023-10-30T12:48:12.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:12.532+0000 7f1524ff2640 1 -- 172.21.15.125:0/1910126033 wait complete. 2023-10-30T12:48:12.553 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:48:14.555 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:14.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa033b8640 1 Processor -- start 2023-10-30T12:48:14.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa033b8640 1 -- start start 2023-10-30T12:48:14.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa033b8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff9fc1538a0 0x7ff9fc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:14.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa033b8640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff9fc0ca0e0 0x7ff9fc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:14.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa033b8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9fc0cb800 con 0x7ff9fc0cf160 2023-10-30T12:48:14.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa033b8640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9fc15dc30 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa033b8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9fc0cb330 con 0x7ff9fc1538a0 2023-10-30T12:48:14.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa00953640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff9fc0ca0e0 0x7ff9fc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:14.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa00953640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff9fc0ca0e0 0x7ff9fc15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55746/0 (socket says 172.21.15.125:55746) 2023-10-30T12:48:14.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa00953640 1 -- 172.21.15.125:0/4260599411 learned_addr learned my addr 172.21.15.125:0/4260599411 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:14.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa01154640 1 --2- 172.21.15.125:0/4260599411 >> v2:172.21.15.136:3300/0 conn(0x7ff9fc1538a0 0x7ff9fc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:14.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa00953640 1 -- 172.21.15.125:0/4260599411 >> v2:172.21.15.136:3300/0 conn(0x7ff9fc1538a0 msgr2=0x7ff9fc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:14.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa00953640 1 --2- 172.21.15.125:0/4260599411 >> v2:172.21.15.136:3300/0 conn(0x7ff9fc1538a0 0x7ff9fc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa00953640 1 -- 172.21.15.125:0/4260599411 >> v1:172.21.15.136:6789/0 conn(0x7ff9fc0cf160 legacy=0x7ff9fc0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:14.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.696+0000 7ffa00953640 1 -- 172.21.15.125:0/4260599411 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9fc0cad70 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.700+0000 7ffa00953640 1 --2- 172.21.15.125:0/4260599411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9fc0ca0e0 0x7ff9fc15a090 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc0cfd10 tx=0x7ff9ec01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85bd77841cef6ce2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:14.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.700+0000 7ff9f3fff640 1 -- 172.21.15.125:0/4260599411 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0a89f0 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.700+0000 7ff9f3fff640 1 -- 172.21.15.125:0/4260599411 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9ec0a8b50 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.700+0000 7ff9f3fff640 1 -- 172.21.15.125:0/4260599411 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9ec0b0440 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.700+0000 7ffa033b8640 1 -- 172.21.15.125:0/4260599411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9fc0ca0e0 msgr2=0x7ff9fc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:14.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.700+0000 7ffa033b8640 1 --2- 172.21.15.125:0/4260599411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9fc0ca0e0 0x7ff9fc15a090 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ff9fc0cfd10 tx=0x7ff9ec01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.700+0000 7ffa033b8640 1 -- 172.21.15.125:0/4260599411 shutdown_connections 2023-10-30T12:48:14.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.700+0000 7ffa033b8640 1 --2- 172.21.15.125:0/4260599411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9fc0ca0e0 0x7ff9fc15a090 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.700+0000 7ffa033b8640 1 --2- 172.21.15.125:0/4260599411 >> v2:172.21.15.136:3300/0 conn(0x7ff9fc1538a0 0x7ff9fc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.700+0000 7ffa033b8640 1 -- 172.21.15.125:0/4260599411 >> 172.21.15.125:0/4260599411 conn(0x7ff9fc06bfe0 msgr2=0x7ff9fc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:14.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.700+0000 7ffa033b8640 1 -- 172.21.15.125:0/4260599411 shutdown_connections 2023-10-30T12:48:14.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.700+0000 7ffa033b8640 1 -- 172.21.15.125:0/4260599411 wait complete. 2023-10-30T12:48:14.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa033b8640 1 Processor -- start 2023-10-30T12:48:14.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa033b8640 1 -- start start 2023-10-30T12:48:14.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa033b8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9fc0ca0e0 0x7ff9fc0ce630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:14.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa033b8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9fc0cf160 0x7ff9fc142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:14.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa033b8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9fc1538a0 0x7ff9fc1ef7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:14.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa033b8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc1584d0 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa033b8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc1ef390 con 0x7ff9fc1538a0 2023-10-30T12:48:14.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa033b8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9fc0cec20 con 0x7ff9fc0cf160 2023-10-30T12:48:14.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa01154640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9fc0ca0e0 0x7ff9fc0ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:14.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa01154640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9fc0ca0e0 0x7ff9fc0ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55754/0 (socket says 172.21.15.125:55754) 2023-10-30T12:48:14.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa01154640 1 -- 172.21.15.125:0/1439248936 learned_addr learned my addr 172.21.15.125:0/1439248936 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:14.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa00953640 1 --2- 172.21.15.125:0/1439248936 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9fc0cf160 0x7ff9fc142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:14.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa01955640 1 --2- 172.21.15.125:0/1439248936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9fc1538a0 0x7ff9fc1ef7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:14.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa01154640 1 -- 172.21.15.125:0/1439248936 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9fc0cf160 msgr2=0x7ff9fc142550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:14.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa01154640 1 --2- 172.21.15.125:0/1439248936 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9fc0cf160 0x7ff9fc142550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa01154640 1 -- 172.21.15.125:0/1439248936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9fc1538a0 msgr2=0x7ff9fc1ef7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:14.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa01154640 1 --2- 172.21.15.125:0/1439248936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9fc1538a0 0x7ff9fc1ef7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa01154640 1 -- 172.21.15.125:0/1439248936 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec0094d0 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa01154640 1 --2- 172.21.15.125:0/1439248936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9fc0ca0e0 0x7ff9fc0ce630 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800d5f0 tx=0x7ff9f800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:14.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ff9f1ffb640 1 -- 172.21.15.125:0/1439248936 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f80125f0 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ff9f1ffb640 1 -- 172.21.15.125:0/1439248936 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9f8012750 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa033b8640 1 -- 172.21.15.125:0/1439248936 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9fc0ceec0 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ff9f1ffb640 1 -- 172.21.15.125:0/1439248936 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9f8012b00 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa033b8640 1 -- 172.21.15.125:0/1439248936 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9fc149080 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa033b8640 1 -- 172.21.15.125:0/1439248936 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9fc0b2a50 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ff9f1ffb640 1 -- 172.21.15.125:0/1439248936 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff9f8012c60 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ff9f1ffb640 1 --2- 172.21.15.125:0/1439248936 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9e00664a0 0x7ff9e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:14.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ff9f1ffb640 1 -- 172.21.15.125:0/1439248936 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7ff9f8084740 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa00953640 1 --2- 172.21.15.125:0/1439248936 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9e00664a0 0x7ff9e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:14.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.708+0000 7ffa00953640 1 --2- 172.21.15.125:0/1439248936 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9e00664a0 0x7ff9e0068960 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0b2000 tx=0x7ff9ec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:14.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.712+0000 7ff9f1ffb640 1 -- 172.21.15.125:0/1439248936 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff9f8051820 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.896+0000 7ffa033b8640 1 -- 172.21.15.125:0/1439248936 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9fc04e170 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.896+0000 7ff9f1ffb640 1 -- 172.21.15.125:0/1439248936 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6532 (secure 0 0 0) 0x7ff9f8054e40 con 0x7ff9fc0ca0e0 2023-10-30T12:48:14.899 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:14.899 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:07.294062+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:14.900 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 99 2023-10-30T12:48:14.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.900+0000 7ffa033b8640 1 -- 172.21.15.125:0/1439248936 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9e00664a0 msgr2=0x7ff9e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:14.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.900+0000 7ffa033b8640 1 --2- 172.21.15.125:0/1439248936 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9e00664a0 0x7ff9e0068960 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec0b2000 tx=0x7ff9ec008040 comp rx=0 tx=0).stop 2023-10-30T12:48:14.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.900+0000 7ffa033b8640 1 -- 172.21.15.125:0/1439248936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9fc0ca0e0 msgr2=0x7ff9fc0ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:14.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.900+0000 7ffa033b8640 1 --2- 172.21.15.125:0/1439248936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9fc0ca0e0 0x7ff9fc0ce630 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7ff9f800d5f0 tx=0x7ff9f800dab0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.900+0000 7ffa033b8640 1 -- 172.21.15.125:0/1439248936 shutdown_connections 2023-10-30T12:48:14.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.900+0000 7ffa033b8640 1 --2- 172.21.15.125:0/1439248936 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9e00664a0 0x7ff9e0068960 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.900+0000 7ffa033b8640 1 --2- 172.21.15.125:0/1439248936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9fc1538a0 0x7ff9fc1ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.900+0000 7ffa033b8640 1 --2- 172.21.15.125:0/1439248936 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9fc0cf160 0x7ff9fc142550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.900+0000 7ffa033b8640 1 --2- 172.21.15.125:0/1439248936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9fc0ca0e0 0x7ff9fc0ce630 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:14.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.900+0000 7ffa033b8640 1 -- 172.21.15.125:0/1439248936 >> 172.21.15.125:0/1439248936 conn(0x7ff9fc06bfe0 msgr2=0x7ff9fc152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:14.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.900+0000 7ffa033b8640 1 -- 172.21.15.125:0/1439248936 shutdown_connections 2023-10-30T12:48:14.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:14.900+0000 7ffa033b8640 1 -- 172.21.15.125:0/1439248936 wait complete. 2023-10-30T12:48:14.920 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:48:16.921 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:17.000 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:48:17.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:48:17.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.072+0000 7f117d565640 1 Processor -- start 2023-10-30T12:48:17.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.072+0000 7f117d565640 1 -- start start 2023-10-30T12:48:17.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f117d565640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f11781538a0 0x7f117815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:17.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f117d565640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11780ca0e0 0x7f117815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:17.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f117d565640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11780cb800 con 0x7f11780cf160 2023-10-30T12:48:17.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f117d565640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117815dc30 con 0x7f11781538a0 2023-10-30T12:48:17.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f117d565640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11780cb330 con 0x7f11780ca0e0 2023-10-30T12:48:17.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f1176ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f11781538a0 0x7f117815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:17.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f1176ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f11781538a0 0x7f117815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:55912/0 (socket says 172.21.15.125:55912) 2023-10-30T12:48:17.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f116e7fc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11780ca0e0 0x7f117815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:17.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f1176ffd640 1 -- 172.21.15.125:0/1560022560 learned_addr learned my addr 172.21.15.125:0/1560022560 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:17.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f11767fc640 1 -- 172.21.15.125:0/1560022560 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740376171 0 0) 0x7f11780cb800 con 0x7f11780cf160 2023-10-30T12:48:17.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f11767fc640 1 -- 172.21.15.125:0/1560022560 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1164003530 con 0x7f11780cf160 2023-10-30T12:48:17.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f1176ffd640 1 -- 172.21.15.125:0/1560022560 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11780ca0e0 msgr2=0x7f117815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:17.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f1176ffd640 1 --2- 172.21.15.125:0/1560022560 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11780ca0e0 0x7f117815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:17.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f1176ffd640 1 -- 172.21.15.125:0/1560022560 >> v1:172.21.15.136:6789/0 conn(0x7f11780cf160 legacy=0x7f11780cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:17.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f1176ffd640 1 -- 172.21.15.125:0/1560022560 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11780cad70 con 0x7f11781538a0 2023-10-30T12:48:17.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.076+0000 7f116e7fc640 1 --2- 172.21.15.125:0/1560022560 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11780ca0e0 0x7f117815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:48:17.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.080+0000 7f1176ffd640 1 --2- 172.21.15.125:0/1560022560 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11781538a0 0x7f117815d6f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1168009870 tx=0x7f116801ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d72d3ad8584c193 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:17.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.080+0000 7f11767fc640 1 -- 172.21.15.125:0/1560022560 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11680a8a30 con 0x7f11781538a0 2023-10-30T12:48:17.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.080+0000 7f11767fc640 1 -- 172.21.15.125:0/1560022560 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11680a8b90 con 0x7f11781538a0 2023-10-30T12:48:17.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.080+0000 7f11767fc640 1 -- 172.21.15.125:0/1560022560 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11680b0440 con 0x7f11781538a0 2023-10-30T12:48:17.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.080+0000 7f117d565640 1 -- 172.21.15.125:0/1560022560 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11781538a0 msgr2=0x7f117815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:17.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.080+0000 7f117d565640 1 --2- 172.21.15.125:0/1560022560 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11781538a0 0x7f117815d6f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1168009870 tx=0x7f116801ed30 comp rx=0 tx=0).stop 2023-10-30T12:48:17.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.080+0000 7f117d565640 1 -- 172.21.15.125:0/1560022560 shutdown_connections 2023-10-30T12:48:17.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.080+0000 7f117d565640 1 --2- 172.21.15.125:0/1560022560 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11780ca0e0 0x7f117815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:17.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.080+0000 7f117d565640 1 --2- 172.21.15.125:0/1560022560 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11781538a0 0x7f117815d6f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:17.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.080+0000 7f117d565640 1 -- 172.21.15.125:0/1560022560 >> 172.21.15.125:0/1560022560 conn(0x7f117806bfe0 msgr2=0x7f117806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:17.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.080+0000 7f117d565640 1 -- 172.21.15.125:0/1560022560 shutdown_connections 2023-10-30T12:48:17.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.080+0000 7f117d565640 1 -- 172.21.15.125:0/1560022560 wait complete. 2023-10-30T12:48:17.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.080+0000 7f117d565640 1 Processor -- start 2023-10-30T12:48:17.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.080+0000 7f117d565640 1 -- start start 2023-10-30T12:48:17.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f117d565640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11780ca0e0 0x7f11781481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:17.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f117d565640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11780cf160 0x7f11781f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:17.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f117d565640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11781538a0 0x7f11781f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:17.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f117d565640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117815acf0 con 0x7f11780ca0e0 2023-10-30T12:48:17.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f117d565640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11780cef70 con 0x7f11781538a0 2023-10-30T12:48:17.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f117d565640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117815a950 con 0x7f11780cf160 2023-10-30T12:48:17.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f1176ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11780ca0e0 0x7f11781481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:17.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f1176ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11780ca0e0 0x7f11781481a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55758/0 (socket says 172.21.15.125:55758) 2023-10-30T12:48:17.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f1176ffd640 1 -- 172.21.15.125:0/2133650184 learned_addr learned my addr 172.21.15.125:0/2133650184 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:17.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f116e7fc640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11780cf160 0x7f11781f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:17.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f11777fe640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11781538a0 0x7f11781f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:17.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f1176ffd640 1 -- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11780cf160 msgr2=0x7f11781f2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:17.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f1176ffd640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11780cf160 0x7f11781f2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:17.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f1176ffd640 1 -- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11781538a0 msgr2=0x7f11781f2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:17.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f1176ffd640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11781538a0 0x7f11781f2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:17.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f1176ffd640 1 -- 172.21.15.125:0/2133650184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1160009a70 con 0x7f11780ca0e0 2023-10-30T12:48:17.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f11777fe640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11781538a0 0x7f11781f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:17.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f1176ffd640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11780ca0e0 0x7f11781481a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f11680b3040 tx=0x7f116801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:17.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f116ffff640 1 -- 172.21.15.125:0/2133650184 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11680b0d20 con 0x7f11780ca0e0 2023-10-30T12:48:17.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f116ffff640 1 -- 172.21.15.125:0/2133650184 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11680bb460 con 0x7f11780ca0e0 2023-10-30T12:48:17.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f116ffff640 1 -- 172.21.15.125:0/2133650184 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11680b0ea0 con 0x7f11780ca0e0 2023-10-30T12:48:17.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f117d565640 1 -- 172.21.15.125:0/2133650184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11680094d0 con 0x7f11780ca0e0 2023-10-30T12:48:17.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f117d565640 1 -- 172.21.15.125:0/2133650184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11780cec90 con 0x7f11780ca0e0 2023-10-30T12:48:17.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.084+0000 7f117d565640 1 -- 172.21.15.125:0/2133650184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11780b2a50 con 0x7f11780ca0e0 2023-10-30T12:48:17.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.088+0000 7f116ffff640 1 -- 172.21.15.125:0/2133650184 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f11680b83f0 con 0x7f11780ca0e0 2023-10-30T12:48:17.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.088+0000 7f116ffff640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11580664a0 0x7f1158068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:17.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.088+0000 7f116ffff640 1 -- 172.21.15.125:0/2133650184 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f1168121930 con 0x7f11780ca0e0 2023-10-30T12:48:17.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.088+0000 7f116e7fc640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11580664a0 0x7f1158068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:17.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.088+0000 7f116e7fc640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11580664a0 0x7f1158068960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1160009fd0 tx=0x7f116000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:17.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.092+0000 7f116ffff640 1 -- 172.21.15.125:0/2133650184 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f11680eeac0 con 0x7f11780ca0e0 2023-10-30T12:48:17.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.272+0000 7f117d565640 1 -- 172.21.15.125:0/2133650184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f117815ae30 con 0x7f11780ca0e0 2023-10-30T12:48:17.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.272+0000 7f116ffff640 1 -- 172.21.15.125:0/2133650184 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f11680c0090 con 0x7f11780ca0e0 2023-10-30T12:48:17.270 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:17.270 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:07.294062+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:17.271 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 99 2023-10-30T12:48:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.272+0000 7f117d565640 1 -- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11580664a0 msgr2=0x7f1158068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.272+0000 7f117d565640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11580664a0 0x7f1158068960 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f1160009fd0 tx=0x7f116000b040 comp rx=0 tx=0).stop 2023-10-30T12:48:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.272+0000 7f117d565640 1 -- 172.21.15.125:0/2133650184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11780ca0e0 msgr2=0x7f11781481a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:17.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.272+0000 7f117d565640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11780ca0e0 0x7f11781481a0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f11680b3040 tx=0x7f116801bad0 comp rx=0 tx=0).stop 2023-10-30T12:48:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.272+0000 7f117d565640 1 -- 172.21.15.125:0/2133650184 shutdown_connections 2023-10-30T12:48:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.272+0000 7f117d565640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11580664a0 0x7f1158068960 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.272+0000 7f117d565640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11781538a0 0x7f11781f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.272+0000 7f117d565640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11780cf160 0x7f11781f2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:17.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.272+0000 7f117d565640 1 --2- 172.21.15.125:0/2133650184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11780ca0e0 0x7f11781481a0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.272+0000 7f117d565640 1 -- 172.21.15.125:0/2133650184 >> 172.21.15.125:0/2133650184 conn(0x7f117806bfe0 msgr2=0x7f11781517f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.272+0000 7f117d565640 1 -- 172.21.15.125:0/2133650184 shutdown_connections 2023-10-30T12:48:17.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:17.272+0000 7f117d565640 1 -- 172.21.15.125:0/2133650184 wait complete. 2023-10-30T12:48:17.291 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:48:19.294 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:19.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.444+0000 7fa881209640 1 Processor -- start 2023-10-30T12:48:19.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.444+0000 7fa881209640 1 -- start start 2023-10-30T12:48:19.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa881209640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa87c0c6b80 0x7fa87c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:19.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa881209640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa87c13f4a0 0x7fa87c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:19.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa881209640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa87c0c0a50 con 0x7fa87c0c6f60 2023-10-30T12:48:19.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa881209640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa87c06f6e0 con 0x7fa87c13f4a0 2023-10-30T12:48:19.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa881209640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c06f9c0 con 0x7fa87c0c6b80 2023-10-30T12:48:19.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87259c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa87c0c6f60 0x7fa87c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44876/0 (socket says 172.21.15.125:44876) 2023-10-30T12:48:19.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87259c640 1 -- 172.21.15.125:0/1982841634 learned_addr learned my addr 172.21.15.125:0/1982841634 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:48:19.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87b59e640 1 --2- 172.21.15.125:0/1982841634 >> v2:172.21.15.136:3300/0 conn(0x7fa87c13f4a0 0x7fa87c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:19.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87ad9d640 1 --2- 172.21.15.125:0/1982841634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa87c0c6b80 0x7fa87c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:19.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 -- 172.21.15.125:0/1982841634 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918579855 0 0) 0x7fa87c0c0a50 con 0x7fa87c0c6f60 2023-10-30T12:48:19.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 -- 172.21.15.125:0/1982841634 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa868003530 con 0x7fa87c0c6f60 2023-10-30T12:48:19.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 -- 172.21.15.125:0/1982841634 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1663480878 0 0) 0x7fa868003530 con 0x7fa87c0c6f60 2023-10-30T12:48:19.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 -- 172.21.15.125:0/1982841634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa87c0c6b80 msgr2=0x7fa87c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:19.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 --2- 172.21.15.125:0/1982841634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa87c0c6b80 0x7fa87c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 -- 172.21.15.125:0/1982841634 >> v2:172.21.15.136:3300/0 conn(0x7fa87c13f4a0 msgr2=0x7fa87c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:19.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 --2- 172.21.15.125:0/1982841634 >> v2:172.21.15.136:3300/0 conn(0x7fa87c13f4a0 0x7fa87c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 -- 172.21.15.125:0/1982841634 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c06fca0 con 0x7fa87c0c6f60 2023-10-30T12:48:19.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 -- 172.21.15.125:0/1982841634 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fa864001a00 con 0x7fa87c0c6f60 2023-10-30T12:48:19.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 -- 172.21.15.125:0/1982841634 >> v1:172.21.15.125:6789/0 conn(0x7fa87c0c6f60 legacy=0x7fa87c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:19.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 --2- 172.21.15.125:0/1982841634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa87c0c6b80 0x7fa868003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:19.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 --2- 172.21.15.125:0/1982841634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa868003f00 0x7fa8680047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:19.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 --2- 172.21.15.125:0/1982841634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa868004d10 0x7fa868005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:19.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 -- 172.21.15.125:0/1982841634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c06f6e0 con 0x7fa868004d10 2023-10-30T12:48:19.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 -- 172.21.15.125:0/1982841634 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c06f9c0 con 0x7fa868003f00 2023-10-30T12:48:19.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87a59c640 1 -- 172.21.15.125:0/1982841634 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa86800b990 con 0x7fa87c0c6b80 2023-10-30T12:48:19.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87b59e640 1 --2- 172.21.15.125:0/1982841634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa868004d10 0x7fa868005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:19.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87ad9d640 1 --2- 172.21.15.125:0/1982841634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa87c0c6b80 0x7fa868003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:19.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87259c640 1 --2- 172.21.15.125:0/1982841634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa868003f00 0x7fa8680047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:19.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87b59e640 1 -- 172.21.15.125:0/1982841634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa87c0c6b80 msgr2=0x7fa868003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:19.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87b59e640 1 --2- 172.21.15.125:0/1982841634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa87c0c6b80 0x7fa868003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87b59e640 1 -- 172.21.15.125:0/1982841634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa868003f00 msgr2=0x7fa8680047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:19.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87b59e640 1 --2- 172.21.15.125:0/1982841634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa868003f00 0x7fa8680047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87b59e640 1 -- 172.21.15.125:0/1982841634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa868003530 con 0x7fa868004d10 2023-10-30T12:48:19.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.448+0000 7fa87259c640 1 --2- 172.21.15.125:0/1982841634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa868003f00 0x7fa8680047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:19.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.452+0000 7fa87b59e640 1 --2- 172.21.15.125:0/1982841634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa868004d10 0x7fa868005190 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fa87c06f4f0 tx=0x7fa87401ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f003f2e5668d975 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:19.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.452+0000 7fa87a59c640 1 -- 172.21.15.125:0/1982841634 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8740a5620 con 0x7fa868004d10 2023-10-30T12:48:19.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.452+0000 7fa87a59c640 1 -- 172.21.15.125:0/1982841634 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8740a5780 con 0x7fa868004d10 2023-10-30T12:48:19.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.452+0000 7fa87a59c640 1 -- 172.21.15.125:0/1982841634 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8740a5b30 con 0x7fa868004d10 2023-10-30T12:48:19.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.452+0000 7fa881209640 1 -- 172.21.15.125:0/1982841634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa868004d10 msgr2=0x7fa868005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:19.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.452+0000 7fa881209640 1 --2- 172.21.15.125:0/1982841634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa868004d10 0x7fa868005190 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fa87c06f4f0 tx=0x7fa87401ca30 comp rx=0 tx=0).stop 2023-10-30T12:48:19.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.452+0000 7fa87b59e640 1 -- 172.21.15.125:0/1982841634 reap_dead start 2023-10-30T12:48:19.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.452+0000 7fa881209640 1 -- 172.21.15.125:0/1982841634 shutdown_connections 2023-10-30T12:48:19.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.452+0000 7fa881209640 1 -- 172.21.15.125:0/1982841634 >> 172.21.15.125:0/1982841634 conn(0x7fa87c0c7910 msgr2=0x7fa87c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:19.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.452+0000 7fa881209640 1 -- 172.21.15.125:0/1982841634 shutdown_connections 2023-10-30T12:48:19.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.452+0000 7fa881209640 1 -- 172.21.15.125:0/1982841634 wait complete. 2023-10-30T12:48:19.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa881209640 1 Processor -- start 2023-10-30T12:48:19.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa881209640 1 -- start start 2023-10-30T12:48:19.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa881209640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa868004d10 0x7fa87c1eb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:19.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa881209640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa87c1559b0 0x7fa87c13cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:19.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa881209640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa87c13d8f0 0x7fa87c152fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:19.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa881209640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c156340 con 0x7fa87c1559b0 2023-10-30T12:48:19.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa881209640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c155520 con 0x7fa87c13d8f0 2023-10-30T12:48:19.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa881209640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa87c154c30 con 0x7fa868004d10 2023-10-30T12:48:19.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa87259c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa87c1559b0 0x7fa87c13cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:19.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa87259c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa87c1559b0 0x7fa87c13cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55774/0 (socket says 172.21.15.125:55774) 2023-10-30T12:48:19.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa87259c640 1 -- 172.21.15.125:0/524714397 learned_addr learned my addr 172.21.15.125:0/524714397 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:19.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa87ad9d640 1 --2- 172.21.15.125:0/524714397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa868004d10 0x7fa87c1eb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:19.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa87b59e640 1 --2- 172.21.15.125:0/524714397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa87c13d8f0 0x7fa87c152fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:19.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa87259c640 1 -- 172.21.15.125:0/524714397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa868004d10 msgr2=0x7fa87c1eb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:19.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa87259c640 1 --2- 172.21.15.125:0/524714397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa868004d10 0x7fa87c1eb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa87259c640 1 -- 172.21.15.125:0/524714397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa87c13d8f0 msgr2=0x7fa87c152fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:19.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa87259c640 1 --2- 172.21.15.125:0/524714397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa87c13d8f0 0x7fa87c152fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa87259c640 1 -- 172.21.15.125:0/524714397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c1eb040 con 0x7fa87c1559b0 2023-10-30T12:48:19.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa87259c640 1 --2- 172.21.15.125:0/524714397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa87c1559b0 0x7fa87c13cd70 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fa864007a60 tx=0x7fa86400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:19.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa873fff640 1 -- 172.21.15.125:0/524714397 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86400ee50 con 0x7fa87c1559b0 2023-10-30T12:48:19.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa873fff640 1 -- 172.21.15.125:0/524714397 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa864013420 con 0x7fa87c1559b0 2023-10-30T12:48:19.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa881209640 1 -- 172.21.15.125:0/524714397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa87c154ed0 con 0x7fa87c1559b0 2023-10-30T12:48:19.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa873fff640 1 -- 172.21.15.125:0/524714397 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8640137d0 con 0x7fa87c1559b0 2023-10-30T12:48:19.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa881209640 1 -- 172.21.15.125:0/524714397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87c1eb320 con 0x7fa87c1559b0 2023-10-30T12:48:19.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.456+0000 7fa881209640 1 -- 172.21.15.125:0/524714397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa87c155690 con 0x7fa87c1559b0 2023-10-30T12:48:19.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.460+0000 7fa873fff640 1 -- 172.21.15.125:0/524714397 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa864013930 con 0x7fa87c1559b0 2023-10-30T12:48:19.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.460+0000 7fa873fff640 1 --2- 172.21.15.125:0/524714397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa85c0664d0 0x7fa85c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:19.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.460+0000 7fa873fff640 1 -- 172.21.15.125:0/524714397 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7fa864084960 con 0x7fa87c1559b0 2023-10-30T12:48:19.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.460+0000 7fa87ad9d640 1 --2- 172.21.15.125:0/524714397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa85c0664d0 0x7fa85c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:19.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.460+0000 7fa87ad9d640 1 --2- 172.21.15.125:0/524714397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa85c0664d0 0x7fa85c068990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa86c009be0 tx=0x7fa86c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:19.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.464+0000 7fa873fff640 1 -- 172.21.15.125:0/524714397 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa864051ad0 con 0x7fa87c1559b0 2023-10-30T12:48:19.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.648+0000 7fa881209640 1 -- 172.21.15.125:0/524714397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa87c04e170 con 0x7fa87c1559b0 2023-10-30T12:48:19.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.648+0000 7fa873fff640 1 -- 172.21.15.125:0/524714397 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6532 (secure 0 0 0) 0x7fa8640550f0 con 0x7fa87c1559b0 2023-10-30T12:48:19.648 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:19.648 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:07.294062+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:19.649 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 99 2023-10-30T12:48:19.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.652+0000 7fa881209640 1 -- 172.21.15.125:0/524714397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa85c0664d0 msgr2=0x7fa85c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:19.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.652+0000 7fa881209640 1 --2- 172.21.15.125:0/524714397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa85c0664d0 0x7fa85c068990 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa86c009be0 tx=0x7fa86c009190 comp rx=0 tx=0).stop 2023-10-30T12:48:19.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.652+0000 7fa881209640 1 -- 172.21.15.125:0/524714397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa87c1559b0 msgr2=0x7fa87c13cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.652+0000 7fa881209640 1 --2- 172.21.15.125:0/524714397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa87c1559b0 0x7fa87c13cd70 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fa864007a60 tx=0x7fa86400e530 comp rx=0 tx=0).stop 2023-10-30T12:48:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.652+0000 7fa881209640 1 -- 172.21.15.125:0/524714397 shutdown_connections 2023-10-30T12:48:19.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.652+0000 7fa881209640 1 --2- 172.21.15.125:0/524714397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa85c0664d0 0x7fa85c068990 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.652+0000 7fa881209640 1 --2- 172.21.15.125:0/524714397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa87c13d8f0 0x7fa87c152fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.652+0000 7fa881209640 1 --2- 172.21.15.125:0/524714397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa87c1559b0 0x7fa87c13cd70 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.652+0000 7fa881209640 1 --2- 172.21.15.125:0/524714397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa868004d10 0x7fa87c1eb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:19.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.652+0000 7fa881209640 1 -- 172.21.15.125:0/524714397 >> 172.21.15.125:0/524714397 conn(0x7fa87c0c7910 msgr2=0x7fa87c13b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:19.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.652+0000 7fa881209640 1 -- 172.21.15.125:0/524714397 shutdown_connections 2023-10-30T12:48:19.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:19.652+0000 7fa881209640 1 -- 172.21.15.125:0/524714397 wait complete. 2023-10-30T12:48:19.668 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:48:21.669 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:21.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8f5f8640 1 Processor -- start 2023-10-30T12:48:21.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8f5f8640 1 -- start start 2023-10-30T12:48:21.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8f5f8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6b8815d6f0 0x7f6b8815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:21.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8f5f8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b880cf160 0x7f6b880cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:21.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8f5f8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b880cb800 con 0x7f6b881538a0 2023-10-30T12:48:21.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8f5f8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b880cb330 con 0x7f6b8815d6f0 2023-10-30T12:48:21.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8f5f8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b880cad70 con 0x7f6b880cf160 2023-10-30T12:48:21.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8cb93640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6b8815d6f0 0x7f6b8815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:21.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8db95640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b880cf160 0x7f6b880cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:21.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8cb93640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6b8815d6f0 0x7f6b8815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:39484/0 (socket says 172.21.15.125:39484) 2023-10-30T12:48:21.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8db95640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b880cf160 0x7f6b880cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:50044/0 (socket says 172.21.15.125:50044) 2023-10-30T12:48:21.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8d394640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f6b881538a0 0x7f6b880ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:35702/0 (socket says 172.21.15.125:35702) 2023-10-30T12:48:21.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8cb93640 1 -- 172.21.15.125:0/3296094935 learned_addr learned my addr 172.21.15.125:0/3296094935 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:21.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8db95640 1 -- 172.21.15.125:0/3296094935 >> v2:172.21.15.136:3300/0 conn(0x7f6b8815d6f0 msgr2=0x7f6b8815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:21.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8db95640 1 --2- 172.21.15.125:0/3296094935 >> v2:172.21.15.136:3300/0 conn(0x7f6b8815d6f0 0x7f6b8815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8db95640 1 -- 172.21.15.125:0/3296094935 >> v1:172.21.15.136:6789/0 conn(0x7f6b881538a0 legacy=0x7f6b880ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:21.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8db95640 1 -- 172.21.15.125:0/3296094935 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b880cb050 con 0x7f6b880cf160 2023-10-30T12:48:21.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.820+0000 7f6b8cb93640 1 --2- 172.21.15.125:0/3296094935 >> v2:172.21.15.136:3300/0 conn(0x7f6b8815d6f0 0x7f6b8815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:48:21.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.824+0000 7f6b8db95640 1 --2- 172.21.15.125:0/3296094935 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b880cf160 0x7f6b880cf590 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6b80009d90 tx=0x7f6b8001f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87b2b00b29bedaea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:21.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.824+0000 7f6b7ffff640 1 -- 172.21.15.125:0/3296094935 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b800af600 con 0x7f6b880cf160 2023-10-30T12:48:21.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.824+0000 7f6b7ffff640 1 -- 172.21.15.125:0/3296094935 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b800ab050 con 0x7f6b880cf160 2023-10-30T12:48:21.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.824+0000 7f6b7ffff640 1 -- 172.21.15.125:0/3296094935 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b800af980 con 0x7f6b880cf160 2023-10-30T12:48:21.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.824+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/3296094935 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b880cf160 msgr2=0x7f6b880cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:21.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.824+0000 7f6b8f5f8640 1 --2- 172.21.15.125:0/3296094935 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b880cf160 0x7f6b880cf590 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6b80009d90 tx=0x7f6b8001f130 comp rx=0 tx=0).stop 2023-10-30T12:48:21.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.824+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/3296094935 shutdown_connections 2023-10-30T12:48:21.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.824+0000 7f6b8f5f8640 1 --2- 172.21.15.125:0/3296094935 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b880cf160 0x7f6b880cf590 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.824+0000 7f6b8f5f8640 1 --2- 172.21.15.125:0/3296094935 >> v2:172.21.15.136:3300/0 conn(0x7f6b8815d6f0 0x7f6b8815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.824+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/3296094935 >> 172.21.15.125:0/3296094935 conn(0x7f6b8806bfe0 msgr2=0x7f6b8806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:21.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.824+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/3296094935 shutdown_connections 2023-10-30T12:48:21.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.824+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/3296094935 wait complete. 2023-10-30T12:48:21.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8f5f8640 1 Processor -- start 2023-10-30T12:48:21.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8f5f8640 1 -- start start 2023-10-30T12:48:21.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8f5f8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b880cf160 0x7f6b8814e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:21.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8f5f8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b881538a0 0x7f6b880b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:21.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8f5f8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b8815d6f0 0x7f6b880cd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:21.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8f5f8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b881efd70 con 0x7f6b881538a0 2023-10-30T12:48:21.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8f5f8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b881f0260 con 0x7f6b880cf160 2023-10-30T12:48:21.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8f5f8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b880ce7b0 con 0x7f6b8815d6f0 2023-10-30T12:48:21.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8cb93640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b881538a0 0x7f6b880b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:21.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8cb93640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b881538a0 0x7f6b880b49f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44548/0 (socket says 172.21.15.125:44548) 2023-10-30T12:48:21.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8cb93640 1 -- 172.21.15.125:0/695077785 learned_addr learned my addr 172.21.15.125:0/695077785 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:21.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8d394640 1 --2- 172.21.15.125:0/695077785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b880cf160 0x7f6b8814e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:21.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8db95640 1 --2- 172.21.15.125:0/695077785 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b8815d6f0 0x7f6b880cd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:21.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8cb93640 1 -- 172.21.15.125:0/695077785 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b8815d6f0 msgr2=0x7f6b880cd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:21.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8cb93640 1 --2- 172.21.15.125:0/695077785 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b8815d6f0 0x7f6b880cd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8cb93640 1 -- 172.21.15.125:0/695077785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b880cf160 msgr2=0x7f6b8814e220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:21.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8cb93640 1 --2- 172.21.15.125:0/695077785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b880cf160 0x7f6b8814e220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:21.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8cb93640 1 -- 172.21.15.125:0/695077785 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b780094d0 con 0x7f6b881538a0 2023-10-30T12:48:21.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8cb93640 1 --2- 172.21.15.125:0/695077785 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b881538a0 0x7f6b880b49f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800e930 tx=0x7f6b7800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:21.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b7dffb640 1 -- 172.21.15.125:0/695077785 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b7800ca00 con 0x7f6b881538a0 2023-10-30T12:48:21.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b7dffb640 1 -- 172.21.15.125:0/695077785 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b7800cb60 con 0x7f6b881538a0 2023-10-30T12:48:21.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b7dffb640 1 -- 172.21.15.125:0/695077785 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b78013470 con 0x7f6b881538a0 2023-10-30T12:48:21.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/695077785 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b80009a70 con 0x7f6b881538a0 2023-10-30T12:48:21.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.828+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/695077785 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b881eff80 con 0x7f6b881538a0 2023-10-30T12:48:21.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.832+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/695077785 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b880b2a50 con 0x7f6b881538a0 2023-10-30T12:48:21.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.832+0000 7f6b7dffb640 1 -- 172.21.15.125:0/695077785 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6b780135d0 con 0x7f6b881538a0 2023-10-30T12:48:21.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.832+0000 7f6b7dffb640 1 --2- 172.21.15.125:0/695077785 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6b6c0664d0 0x7f6b6c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:21.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.832+0000 7f6b7dffb640 1 -- 172.21.15.125:0/695077785 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f6b78084320 con 0x7f6b881538a0 2023-10-30T12:48:21.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.832+0000 7f6b8d394640 1 --2- 172.21.15.125:0/695077785 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6b6c0664d0 0x7f6b6c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:21.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.832+0000 7f6b8d394640 1 --2- 172.21.15.125:0/695077785 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6b6c0664d0 0x7f6b6c068990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6b84000f70 tx=0x7f6b84007e50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:21.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:21.836+0000 7f6b7dffb640 1 -- 172.21.15.125:0/695077785 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6b78051540 con 0x7f6b881538a0 2023-10-30T12:48:22.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:22.016+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/695077785 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b8804e170 con 0x7f6b881538a0 2023-10-30T12:48:22.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:22.020+0000 7f6b7dffb640 1 -- 172.21.15.125:0/695077785 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f6b78054b60 con 0x7f6b881538a0 2023-10-30T12:48:22.016 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:22.016 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:07.294062+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:22.017 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 99 2023-10-30T12:48:22.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:22.020+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/695077785 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6b6c0664d0 msgr2=0x7f6b6c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:22.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:22.020+0000 7f6b8f5f8640 1 --2- 172.21.15.125:0/695077785 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6b6c0664d0 0x7f6b6c068990 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6b84000f70 tx=0x7f6b84007e50 comp rx=0 tx=0).stop 2023-10-30T12:48:22.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:22.020+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/695077785 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b881538a0 msgr2=0x7f6b880b49f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:22.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:22.020+0000 7f6b8f5f8640 1 --2- 172.21.15.125:0/695077785 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b881538a0 0x7f6b880b49f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f6b7800e930 tx=0x7f6b7800edf0 comp rx=0 tx=0).stop 2023-10-30T12:48:22.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:22.020+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/695077785 shutdown_connections 2023-10-30T12:48:22.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:22.020+0000 7f6b8f5f8640 1 --2- 172.21.15.125:0/695077785 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6b6c0664d0 0x7f6b6c068990 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:22.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:22.020+0000 7f6b8f5f8640 1 --2- 172.21.15.125:0/695077785 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b8815d6f0 0x7f6b880cd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:22.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:22.020+0000 7f6b8f5f8640 1 --2- 172.21.15.125:0/695077785 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b881538a0 0x7f6b880b49f0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:22.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:22.020+0000 7f6b8f5f8640 1 --2- 172.21.15.125:0/695077785 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b880cf160 0x7f6b8814e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:22.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:22.020+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/695077785 >> 172.21.15.125:0/695077785 conn(0x7f6b8806bfe0 msgr2=0x7f6b880c48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:22.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:22.020+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/695077785 shutdown_connections 2023-10-30T12:48:22.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:22.020+0000 7f6b8f5f8640 1 -- 172.21.15.125:0/695077785 wait complete. 2023-10-30T12:48:22.037 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:48:24.038 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:24.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f641e301640 1 Processor -- start 2023-10-30T12:48:24.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f641e301640 1 -- start start 2023-10-30T12:48:24.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f641e301640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f641815d6f0 0x7f641815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:24.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f641e301640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f64180cf160 0x7f64180cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:24.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f641e301640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64180cb800 con 0x7f64181538a0 2023-10-30T12:48:24.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f641e301640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64180cb330 con 0x7f64180cf160 2023-10-30T12:48:24.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f641e301640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64180cad70 con 0x7f641815d6f0 2023-10-30T12:48:24.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f6417fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f64181538a0 0x7f64180ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36518/0 (socket says 172.21.15.125:36518) 2023-10-30T12:48:24.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f6417fff640 1 -- 172.21.15.125:0/1526541525 learned_addr learned my addr 172.21.15.125:0/1526541525 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:48:24.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f641c89e640 1 --2- 172.21.15.125:0/1526541525 >> v2:172.21.15.125:3300/0 conn(0x7f64180cf160 0x7f64180cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:24.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f64177fe640 1 --2- 172.21.15.125:0/1526541525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f641815d6f0 0x7f641815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:24.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f641c89e640 1 -- 172.21.15.125:0/1526541525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f641815d6f0 msgr2=0x7f641815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:24.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f641c89e640 1 --2- 172.21.15.125:0/1526541525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f641815d6f0 0x7f641815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f641c89e640 1 -- 172.21.15.125:0/1526541525 >> v1:172.21.15.125:6789/0 conn(0x7f64181538a0 legacy=0x7f64180ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:24.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f641c89e640 1 -- 172.21.15.125:0/1526541525 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64180cb050 con 0x7f64180cf160 2023-10-30T12:48:24.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.188+0000 7f64177fe640 1 --2- 172.21.15.125:0/1526541525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f641815d6f0 0x7f641815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:24.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.192+0000 7f641c89e640 1 --2- 172.21.15.125:0/1526541525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64180cf160 0x7f64180cf590 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6410009d90 tx=0x7f641001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=831e847e04a11876 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:24.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.192+0000 7f6416ffd640 1 -- 172.21.15.125:0/1526541525 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64100af600 con 0x7f64180cf160 2023-10-30T12:48:24.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.192+0000 7f6416ffd640 1 -- 172.21.15.125:0/1526541525 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64100ab050 con 0x7f64180cf160 2023-10-30T12:48:24.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.192+0000 7f6416ffd640 1 -- 172.21.15.125:0/1526541525 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64100af980 con 0x7f64180cf160 2023-10-30T12:48:24.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.192+0000 7f641e301640 1 -- 172.21.15.125:0/1526541525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64180cf160 msgr2=0x7f64180cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:24.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.192+0000 7f641e301640 1 --2- 172.21.15.125:0/1526541525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64180cf160 0x7f64180cf590 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6410009d90 tx=0x7f641001f130 comp rx=0 tx=0).stop 2023-10-30T12:48:24.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.192+0000 7f641e301640 1 -- 172.21.15.125:0/1526541525 shutdown_connections 2023-10-30T12:48:24.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.192+0000 7f641e301640 1 --2- 172.21.15.125:0/1526541525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64180cf160 0x7f64180cf590 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.192+0000 7f641e301640 1 --2- 172.21.15.125:0/1526541525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f641815d6f0 0x7f641815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.192+0000 7f641e301640 1 -- 172.21.15.125:0/1526541525 >> 172.21.15.125:0/1526541525 conn(0x7f641806bfe0 msgr2=0x7f641806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:24.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.192+0000 7f641e301640 1 -- 172.21.15.125:0/1526541525 shutdown_connections 2023-10-30T12:48:24.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.192+0000 7f641e301640 1 -- 172.21.15.125:0/1526541525 wait complete. 2023-10-30T12:48:24.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f641e301640 1 Processor -- start 2023-10-30T12:48:24.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f641e301640 1 -- start start 2023-10-30T12:48:24.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f641e301640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64180cf160 0x7f64181e49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:24.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f641e301640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64181538a0 0x7f64180b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:24.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f641e301640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f641815d6f0 0x7f64181f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:24.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f641e301640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64181e3180 con 0x7f64181538a0 2023-10-30T12:48:24.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f641e301640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64181e2a10 con 0x7f64180cf160 2023-10-30T12:48:24.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f641e301640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64181e2cf0 con 0x7f641815d6f0 2023-10-30T12:48:24.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f64177fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64181538a0 0x7f64180b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:24.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f64177fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64181538a0 0x7f64180b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44566/0 (socket says 172.21.15.125:44566) 2023-10-30T12:48:24.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f64177fe640 1 -- 172.21.15.125:0/2760953474 learned_addr learned my addr 172.21.15.125:0/2760953474 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:24.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f641c89e640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f641815d6f0 0x7f64181f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:24.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f6417fff640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64180cf160 0x7f64181e49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:24.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f64177fe640 1 -- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f641815d6f0 msgr2=0x7f64181f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:24.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f64177fe640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f641815d6f0 0x7f64181f2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f64177fe640 1 -- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64180cf160 msgr2=0x7f64181e49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:24.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f64177fe640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64180cf160 0x7f64181e49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f64177fe640 1 -- 172.21.15.125:0/2760953474 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6410009a70 con 0x7f64181538a0 2023-10-30T12:48:24.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f641c89e640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f641815d6f0 0x7f64181f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:24.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f64177fe640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64181538a0 0x7f64180b31a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f640c00d800 tx=0x7f640c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:24.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f6414ff9640 1 -- 172.21.15.125:0/2760953474 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c0096c0 con 0x7f64181538a0 2023-10-30T12:48:24.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f6414ff9640 1 -- 172.21.15.125:0/2760953474 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c009820 con 0x7f64181538a0 2023-10-30T12:48:24.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f6414ff9640 1 -- 172.21.15.125:0/2760953474 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f640c009bd0 con 0x7f64181538a0 2023-10-30T12:48:24.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f641e301640 1 -- 172.21.15.125:0/2760953474 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64180b4a90 con 0x7f64181538a0 2023-10-30T12:48:24.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.196+0000 7f641e301640 1 -- 172.21.15.125:0/2760953474 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64180b4d30 con 0x7f64181538a0 2023-10-30T12:48:24.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.200+0000 7f641e301640 1 -- 172.21.15.125:0/2760953474 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64180b2a50 con 0x7f64181538a0 2023-10-30T12:48:24.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.200+0000 7f6414ff9640 1 -- 172.21.15.125:0/2760953474 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f640c009d30 con 0x7f64181538a0 2023-10-30T12:48:24.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.200+0000 7f6414ff9640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64040664a0 0x7f6404068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:24.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.200+0000 7f6414ff9640 1 -- 172.21.15.125:0/2760953474 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f640c084970 con 0x7f64181538a0 2023-10-30T12:48:24.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.200+0000 7f6417fff640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64040664a0 0x7f6404068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:24.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.200+0000 7f6417fff640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64040664a0 0x7f6404068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6408005890 tx=0x7f6408009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:24.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.204+0000 7f6414ff9640 1 -- 172.21.15.125:0/2760953474 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f640c051a40 con 0x7f64181538a0 2023-10-30T12:48:24.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.388+0000 7f641e301640 1 -- 172.21.15.125:0/2760953474 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64181e2e60 con 0x7f64181538a0 2023-10-30T12:48:24.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.388+0000 7f6414ff9640 1 -- 172.21.15.125:0/2760953474 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+6532 (secure 0 0 0) 0x7f640c055060 con 0x7f64181538a0 2023-10-30T12:48:24.385 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:24.385 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6223,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":98}],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:07.294062+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":5988},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_5988":{"gid":5988,"name":"c","rank":1,"incarnation":78,"state":"up:active","state_seq":28,"addr":"172.21.15.125:6837/3156415951","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3156415951},{"type":"v1","addr":"172.21.15.125:6837","nonce":3156415951}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:24.385 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 99 2023-10-30T12:48:24.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.388+0000 7f641e301640 1 -- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64040664a0 msgr2=0x7f6404068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:24.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.388+0000 7f641e301640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64040664a0 0x7f6404068960 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f6408005890 tx=0x7f6408009040 comp rx=0 tx=0).stop 2023-10-30T12:48:24.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.388+0000 7f641e301640 1 -- 172.21.15.125:0/2760953474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64181538a0 msgr2=0x7f64180b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:24.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.388+0000 7f641e301640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64181538a0 0x7f64180b31a0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f640c00d800 tx=0x7f640c00dcc0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.388+0000 7f641e301640 1 -- 172.21.15.125:0/2760953474 shutdown_connections 2023-10-30T12:48:24.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.388+0000 7f641e301640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64040664a0 0x7f6404068960 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.388+0000 7f641e301640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f641815d6f0 0x7f64181f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.388+0000 7f641e301640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64181538a0 0x7f64180b31a0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.388+0000 7f641e301640 1 --2- 172.21.15.125:0/2760953474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64180cf160 0x7f64181e49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:24.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.388+0000 7f641e301640 1 -- 172.21.15.125:0/2760953474 >> 172.21.15.125:0/2760953474 conn(0x7f641806bfe0 msgr2=0x7f6418151cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:24.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.388+0000 7f641e301640 1 -- 172.21.15.125:0/2760953474 shutdown_connections 2023-10-30T12:48:24.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:24.388+0000 7f641e301640 1 -- 172.21.15.125:0/2760953474 wait complete. 2023-10-30T12:48:24.406 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:48:26.408 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:26.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bffb0f640 1 Processor -- start 2023-10-30T12:48:26.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bffb0f640 1 -- start start 2023-10-30T12:48:26.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bffb0f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8bf81538a0 0x7f8bf815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bffb0f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8bf80ca0e0 0x7f8bf815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bffb0f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bf80cb800 con 0x7f8bf80cf160 2023-10-30T12:48:26.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bffb0f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf815dc30 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bffb0f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bf80cb330 con 0x7f8bf81538a0 2023-10-30T12:48:26.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bfd0aa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8bf80ca0e0 0x7f8bf815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bfd0aa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8bf80ca0e0 0x7f8bf815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44568/0 (socket says 172.21.15.125:44568) 2023-10-30T12:48:26.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bfd0aa640 1 -- 172.21.15.125:0/1139576806 learned_addr learned my addr 172.21.15.125:0/1139576806 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:26.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bfd8ab640 1 --2- 172.21.15.125:0/1139576806 >> v2:172.21.15.136:3300/0 conn(0x7f8bf81538a0 0x7f8bf815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bfd0aa640 1 -- 172.21.15.125:0/1139576806 >> v2:172.21.15.136:3300/0 conn(0x7f8bf81538a0 msgr2=0x7f8bf815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:26.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bfd0aa640 1 --2- 172.21.15.125:0/1139576806 >> v2:172.21.15.136:3300/0 conn(0x7f8bf81538a0 0x7f8bf815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bfd0aa640 1 -- 172.21.15.125:0/1139576806 >> v1:172.21.15.125:6789/0 conn(0x7f8bf80cf160 legacy=0x7f8bf80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.560+0000 7f8bfd0aa640 1 -- 172.21.15.125:0/1139576806 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bf80cad70 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.564+0000 7f8bfd0aa640 1 --2- 172.21.15.125:0/1139576806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bf80ca0e0 0x7f8bf815a090 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80cfd10 tx=0x7f8be801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead362b18c9f1ed1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:26.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.564+0000 7f8bfc8a9640 1 -- 172.21.15.125:0/1139576806 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be80a89f0 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.564+0000 7f8bfc8a9640 1 -- 172.21.15.125:0/1139576806 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8be80a8b50 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.564+0000 7f8bfc8a9640 1 -- 172.21.15.125:0/1139576806 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8be80b0440 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.564+0000 7f8bffb0f640 1 -- 172.21.15.125:0/1139576806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bf80ca0e0 msgr2=0x7f8bf815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.564+0000 7f8bffb0f640 1 --2- 172.21.15.125:0/1139576806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bf80ca0e0 0x7f8bf815a090 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f8bf80cfd10 tx=0x7f8be801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.564+0000 7f8bffb0f640 1 -- 172.21.15.125:0/1139576806 shutdown_connections 2023-10-30T12:48:26.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.564+0000 7f8bffb0f640 1 --2- 172.21.15.125:0/1139576806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bf80ca0e0 0x7f8bf815a090 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.564+0000 7f8bffb0f640 1 --2- 172.21.15.125:0/1139576806 >> v2:172.21.15.136:3300/0 conn(0x7f8bf81538a0 0x7f8bf815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.564+0000 7f8bffb0f640 1 -- 172.21.15.125:0/1139576806 >> 172.21.15.125:0/1139576806 conn(0x7f8bf806bfe0 msgr2=0x7f8bf806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:26.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.564+0000 7f8bffb0f640 1 -- 172.21.15.125:0/1139576806 shutdown_connections 2023-10-30T12:48:26.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.564+0000 7f8bffb0f640 1 -- 172.21.15.125:0/1139576806 wait complete. 2023-10-30T12:48:26.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bffb0f640 1 Processor -- start 2023-10-30T12:48:26.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bffb0f640 1 -- start start 2023-10-30T12:48:26.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bffb0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bf80ca0e0 0x7f8bf81ef570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bffb0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8bf80cf160 0x7f8bf80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bffb0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8bf81538a0 0x7f8bf81585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bffb0f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf80b3110 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bffb0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf81efab0 con 0x7f8bf80cf160 2023-10-30T12:48:26.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bffb0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bf80c4420 con 0x7f8bf81538a0 2023-10-30T12:48:26.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bfd8ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bf80ca0e0 0x7f8bf81ef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bfd8ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bf80ca0e0 0x7f8bf81ef570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44570/0 (socket says 172.21.15.125:44570) 2023-10-30T12:48:26.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bfd8ab640 1 -- 172.21.15.125:0/755824764 learned_addr learned my addr 172.21.15.125:0/755824764 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:26.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bfd0aa640 1 --2- 172.21.15.125:0/755824764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8bf80cf160 0x7f8bf80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bfe0ac640 1 --2- 172.21.15.125:0/755824764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8bf81538a0 0x7f8bf81585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bfd8ab640 1 -- 172.21.15.125:0/755824764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8bf81538a0 msgr2=0x7f8bf81585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bfd8ab640 1 --2- 172.21.15.125:0/755824764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8bf81538a0 0x7f8bf81585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bfd8ab640 1 -- 172.21.15.125:0/755824764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8bf80cf160 msgr2=0x7f8bf80cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bfd8ab640 1 --2- 172.21.15.125:0/755824764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8bf80cf160 0x7f8bf80cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bfd8ab640 1 -- 172.21.15.125:0/755824764 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be80094d0 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bfd8ab640 1 --2- 172.21.15.125:0/755824764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bf80ca0e0 0x7f8bf81ef570 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400d5f0 tx=0x7f8bf400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:26.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bee7fc640 1 -- 172.21.15.125:0/755824764 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf40125f0 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bee7fc640 1 -- 172.21.15.125:0/755824764 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bf4012750 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bee7fc640 1 -- 172.21.15.125:0/755824764 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bf4012ae0 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bffb0f640 1 -- 172.21.15.125:0/755824764 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bf81efd20 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bffb0f640 1 -- 172.21.15.125:0/755824764 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bf81f02c0 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.568+0000 7f8bffb0f640 1 -- 172.21.15.125:0/755824764 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bf80b2a50 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.572+0000 7f8bee7fc640 1 -- 172.21.15.125:0/755824764 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8bf4012c40 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.572+0000 7f8bee7fc640 1 --2- 172.21.15.125:0/755824764 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8bdc0664d0 0x7f8bdc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.572+0000 7f8bee7fc640 1 -- 172.21.15.125:0/755824764 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f8bf40846c0 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.572+0000 7f8bfd0aa640 1 --2- 172.21.15.125:0/755824764 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8bdc0664d0 0x7f8bdc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.572+0000 7f8bfd0aa640 1 --2- 172.21.15.125:0/755824764 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8bdc0664d0 0x7f8bdc068990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8be80b2000 tx=0x7f8be8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:26.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.576+0000 7f8bee7fc640 1 -- 172.21.15.125:0/755824764 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8bf4051720 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.756+0000 7f8bffb0f640 1 -- 172.21.15.125:0/755824764 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bf8159860 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.756+0000 7f8bee7fc640 1 -- 172.21.15.125:0/755824764 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5739 (secure 0 0 0) 0x7f8bf4004650 con 0x7f8bf80ca0e0 2023-10-30T12:48:26.756 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:26.756 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:25.270309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:26.759 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 100 2023-10-30T12:48:26.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.760+0000 7f8bffb0f640 1 -- 172.21.15.125:0/755824764 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8bdc0664d0 msgr2=0x7f8bdc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.760+0000 7f8bffb0f640 1 --2- 172.21.15.125:0/755824764 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8bdc0664d0 0x7f8bdc068990 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f8be80b2000 tx=0x7f8be8009040 comp rx=0 tx=0).stop 2023-10-30T12:48:26.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.760+0000 7f8bffb0f640 1 -- 172.21.15.125:0/755824764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bf80ca0e0 msgr2=0x7f8bf81ef570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.760+0000 7f8bffb0f640 1 --2- 172.21.15.125:0/755824764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bf80ca0e0 0x7f8bf81ef570 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f8bf400d5f0 tx=0x7f8bf400dab0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.760+0000 7f8bffb0f640 1 -- 172.21.15.125:0/755824764 shutdown_connections 2023-10-30T12:48:26.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.760+0000 7f8bffb0f640 1 --2- 172.21.15.125:0/755824764 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8bdc0664d0 0x7f8bdc068990 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.760+0000 7f8bffb0f640 1 --2- 172.21.15.125:0/755824764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8bf81538a0 0x7f8bf81585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.760+0000 7f8bffb0f640 1 --2- 172.21.15.125:0/755824764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8bf80cf160 0x7f8bf80cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.760+0000 7f8bffb0f640 1 --2- 172.21.15.125:0/755824764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8bf80ca0e0 0x7f8bf81ef570 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.760+0000 7f8bffb0f640 1 -- 172.21.15.125:0/755824764 >> 172.21.15.125:0/755824764 conn(0x7f8bf806bfe0 msgr2=0x7f8bf8148dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:26.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.760+0000 7f8bffb0f640 1 -- 172.21.15.125:0/755824764 shutdown_connections 2023-10-30T12:48:26.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.760+0000 7f8bffb0f640 1 -- 172.21.15.125:0/755824764 wait complete. 2023-10-30T12:48:26.775 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T12:48:26.775 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:48:26.776 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:26.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ecc10f640 1 Processor -- start 2023-10-30T12:48:26.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ecc10f640 1 -- start start 2023-10-30T12:48:26.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ecc10f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1ec4153d70 0x7f1ec406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ecc10f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ec40c8490 0x7f1ec40c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ecc10f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec406afa0 con 0x7f1ec406e470 2023-10-30T12:48:26.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ecc10f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ec406d780 con 0x7f1ec4153d70 2023-10-30T12:48:26.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ecc10f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec4154150 con 0x7f1ec40c8490 2023-10-30T12:48:26.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec96aa640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1ec406e470 0x7f1ec40cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36542/0 (socket says 172.21.15.125:36542) 2023-10-30T12:48:26.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec96aa640 1 -- 172.21.15.125:0/1200632299 learned_addr learned my addr 172.21.15.125:0/1200632299 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:48:26.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1eca6ac640 1 --2- 172.21.15.125:0/1200632299 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ec40c8490 0x7f1ec40c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec9eab640 1 --2- 172.21.15.125:0/1200632299 >> v2:172.21.15.136:3300/0 conn(0x7f1ec4153d70 0x7f1ec406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 -- 172.21.15.125:0/1200632299 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911112130 0 0) 0x7f1ec406afa0 con 0x7f1ec406e470 2023-10-30T12:48:26.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 -- 172.21.15.125:0/1200632299 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eb0003530 con 0x7f1ec406e470 2023-10-30T12:48:26.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 -- 172.21.15.125:0/1200632299 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2720256374 0 0) 0x7f1eb0003530 con 0x7f1ec406e470 2023-10-30T12:48:26.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 -- 172.21.15.125:0/1200632299 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ec40c8490 msgr2=0x7f1ec40c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:26.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 --2- 172.21.15.125:0/1200632299 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ec40c8490 0x7f1ec40c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 -- 172.21.15.125:0/1200632299 >> v2:172.21.15.136:3300/0 conn(0x7f1ec4153d70 msgr2=0x7f1ec406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:26.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 --2- 172.21.15.125:0/1200632299 >> v2:172.21.15.136:3300/0 conn(0x7f1ec4153d70 0x7f1ec406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 -- 172.21.15.125:0/1200632299 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec406e030 con 0x7f1ec406e470 2023-10-30T12:48:26.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 -- 172.21.15.125:0/1200632299 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f1eb4001680 con 0x7f1ec406e470 2023-10-30T12:48:26.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 -- 172.21.15.125:0/1200632299 >> v1:172.21.15.125:6789/0 conn(0x7f1ec406e470 legacy=0x7f1ec40cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 --2- 172.21.15.125:0/1200632299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1eb0003980 0x7f1eb0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 --2- 172.21.15.125:0/1200632299 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1eb00047d0 0x7f1eb0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 --2- 172.21.15.125:0/1200632299 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ec40c8490 0x7f1eb0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 -- 172.21.15.125:0/1200632299 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec406d780 con 0x7f1eb00047d0 2023-10-30T12:48:26.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 -- 172.21.15.125:0/1200632299 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec4154150 con 0x7f1eb0003980 2023-10-30T12:48:26.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec8ea9640 1 -- 172.21.15.125:0/1200632299 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb000b990 con 0x7f1ec40c8490 2023-10-30T12:48:26.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.916+0000 7f1ec96aa640 1 --2- 172.21.15.125:0/1200632299 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1eb00047d0 0x7f1eb0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1eca6ac640 1 --2- 172.21.15.125:0/1200632299 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ec40c8490 0x7f1eb0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1ec9eab640 1 --2- 172.21.15.125:0/1200632299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1eb0003980 0x7f1eb0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1ec96aa640 1 -- 172.21.15.125:0/1200632299 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ec40c8490 msgr2=0x7f1eb0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:26.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1ec96aa640 1 --2- 172.21.15.125:0/1200632299 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ec40c8490 0x7f1eb0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1ec96aa640 1 -- 172.21.15.125:0/1200632299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1eb0003980 msgr2=0x7f1eb0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:26.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1ec96aa640 1 --2- 172.21.15.125:0/1200632299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1eb0003980 0x7f1eb0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1ec96aa640 1 -- 172.21.15.125:0/1200632299 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb0003530 con 0x7f1eb00047d0 2023-10-30T12:48:26.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1ec9eab640 1 --2- 172.21.15.125:0/1200632299 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1eb0003980 0x7f1eb0003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:26.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1ec96aa640 1 --2- 172.21.15.125:0/1200632299 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1eb00047d0 0x7f1eb0004c50 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400c290 tx=0x7f1eb401f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75946825ee334dd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:26.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1ec8ea9640 1 -- 172.21.15.125:0/1200632299 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb40a8cd0 con 0x7f1eb00047d0 2023-10-30T12:48:26.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1ec8ea9640 1 -- 172.21.15.125:0/1200632299 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eb40a8e30 con 0x7f1eb00047d0 2023-10-30T12:48:26.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1ec8ea9640 1 -- 172.21.15.125:0/1200632299 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb4005b20 con 0x7f1eb00047d0 2023-10-30T12:48:26.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1200632299 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1eb00047d0 msgr2=0x7f1eb0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1ecc10f640 1 --2- 172.21.15.125:0/1200632299 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1eb00047d0 0x7f1eb0004c50 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f1eb400c290 tx=0x7f1eb401f430 comp rx=0 tx=0).stop 2023-10-30T12:48:26.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.920+0000 7f1eca6ac640 1 -- 172.21.15.125:0/1200632299 reap_dead start 2023-10-30T12:48:26.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1200632299 shutdown_connections 2023-10-30T12:48:26.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1200632299 >> 172.21.15.125:0/1200632299 conn(0x7f1ec4071ea0 msgr2=0x7f1ec4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:26.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1200632299 shutdown_connections 2023-10-30T12:48:26.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1200632299 wait complete. 2023-10-30T12:48:26.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ecc10f640 1 Processor -- start 2023-10-30T12:48:26.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ecc10f640 1 -- start start 2023-10-30T12:48:26.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ecc10f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1eb00047d0 0x7f1ec414aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ecc10f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ec4152c70 0x7f1ec41e6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ecc10f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ec414c820 0x7f1ec414cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ecc10f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec414bfb0 con 0x7f1ec414c820 2023-10-30T12:48:26.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ecc10f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec4149640 con 0x7f1ec4152c70 2023-10-30T12:48:26.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ecc10f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec414b110 con 0x7f1eb00047d0 2023-10-30T12:48:26.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1eca6ac640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ec414c820 0x7f1ec414cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1eca6ac640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ec414c820 0x7f1ec414cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44586/0 (socket says 172.21.15.125:44586) 2023-10-30T12:48:26.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1eca6ac640 1 -- 172.21.15.125:0/1315221700 learned_addr learned my addr 172.21.15.125:0/1315221700 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:26.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ec9eab640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1eb00047d0 0x7f1ec414aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ec96aa640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ec4152c70 0x7f1ec41e6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1eca6ac640 1 -- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1eb00047d0 msgr2=0x7f1ec414aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1eca6ac640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1eb00047d0 0x7f1ec414aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1eca6ac640 1 -- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ec4152c70 msgr2=0x7f1ec41e6260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:26.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1eca6ac640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ec4152c70 0x7f1ec41e6260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:26.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1eca6ac640 1 -- 172.21.15.125:0/1315221700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec41f2810 con 0x7f1ec414c820 2023-10-30T12:48:26.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1ec9eab640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1eb00047d0 0x7f1ec414aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:26.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.924+0000 7f1eca6ac640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ec414c820 0x7f1ec414cca0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f1ec40c8440 tx=0x7f1ebc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:26.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.928+0000 7f1eba7fc640 1 -- 172.21.15.125:0/1315221700 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc00e040 con 0x7f1ec414c820 2023-10-30T12:48:26.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.928+0000 7f1eba7fc640 1 -- 172.21.15.125:0/1315221700 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ebc00f040 con 0x7f1ec414c820 2023-10-30T12:48:26.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.928+0000 7f1eba7fc640 1 -- 172.21.15.125:0/1315221700 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ebc015430 con 0x7f1ec414c820 2023-10-30T12:48:26.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.928+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1315221700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec414d1e0 con 0x7f1ec414c820 2023-10-30T12:48:26.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.928+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1315221700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec41f2d40 con 0x7f1ec414c820 2023-10-30T12:48:26.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.928+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1315221700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec404e170 con 0x7f1ec414c820 2023-10-30T12:48:26.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.928+0000 7f1eba7fc640 1 -- 172.21.15.125:0/1315221700 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1ebc015590 con 0x7f1ec414c820 2023-10-30T12:48:26.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.928+0000 7f1eba7fc640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ea80664a0 0x7f1ea8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:26.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.928+0000 7f1eba7fc640 1 -- 172.21.15.125:0/1315221700 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f1ebc084970 con 0x7f1ec414c820 2023-10-30T12:48:26.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.928+0000 7f1ec9eab640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ea80664a0 0x7f1ea8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:26.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.932+0000 7f1ec9eab640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ea80664a0 0x7f1ea8068960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0009af0 tx=0x7f1ec0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:26.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:26.932+0000 7f1eba7fc640 1 -- 172.21.15.125:0/1315221700 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1ebc0519d0 con 0x7f1ec414c820 2023-10-30T12:48:27.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:27.116+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1315221700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ec414c600 con 0x7f1ec414c820 2023-10-30T12:48:27.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:27.116+0000 7f1eba7fc640 1 -- 172.21.15.125:0/1315221700 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5739 (secure 0 0 0) 0x7f1ebc054ff0 con 0x7f1ec414c820 2023-10-30T12:48:27.114 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:27.114 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:25.270309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:27.114 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 100 2023-10-30T12:48:27.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:27.116+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ea80664a0 msgr2=0x7f1ea8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:27.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:27.116+0000 7f1ecc10f640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ea80664a0 0x7f1ea8068960 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1ec0009af0 tx=0x7f1ec0008040 comp rx=0 tx=0).stop 2023-10-30T12:48:27.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:27.116+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1315221700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ec414c820 msgr2=0x7f1ec414cca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:27.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:27.116+0000 7f1ecc10f640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ec414c820 0x7f1ec414cca0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f1ec40c8440 tx=0x7f1ebc00f580 comp rx=0 tx=0).stop 2023-10-30T12:48:27.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:27.116+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1315221700 shutdown_connections 2023-10-30T12:48:27.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:27.116+0000 7f1ecc10f640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ea80664a0 0x7f1ea8068960 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:27.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:27.116+0000 7f1ecc10f640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ec414c820 0x7f1ec414cca0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:27.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:27.116+0000 7f1ecc10f640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ec4152c70 0x7f1ec41e6260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:27.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:27.116+0000 7f1ecc10f640 1 --2- 172.21.15.125:0/1315221700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1eb00047d0 0x7f1ec414aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:27.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:27.116+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1315221700 >> 172.21.15.125:0/1315221700 conn(0x7f1ec4071ea0 msgr2=0x7f1ec414e980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:27.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:27.116+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1315221700 shutdown_connections 2023-10-30T12:48:27.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:27.116+0000 7f1ecc10f640 1 -- 172.21.15.125:0/1315221700 wait complete. 2023-10-30T12:48:27.135 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 100, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 100, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6223": { "addr": "172.21.15.136:6839/3479573317", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3479573317, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3479573317, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6223, "incarnation": 100, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:replay", "state_seq": 1 }, "gid_6318": { "addr": "172.21.15.125:6836/1721514617", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1721514617, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1721514617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 6318, "incarnation": 92, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:48:25.270309+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6318, "mds_1": 6223 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:48:29.136 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:29.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa64759e640 1 Processor -- start 2023-10-30T12:48:29.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa64759e640 1 -- start start 2023-10-30T12:48:29.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa64759e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa64814ff10 0x7fa6480c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:29.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa64759e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa64806fc90 0x7fa6481566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:29.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa64759e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6480c1460 0x7fa6480c1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:29.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa64759e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa648156220 con 0x7fa64806fc90 2023-10-30T12:48:29.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa64759e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa648155bf0 con 0x7fa64814ff10 2023-10-30T12:48:29.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa64759e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa648070110 con 0x7fa6480c1460 2023-10-30T12:48:29.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa645d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa64806fc90 0x7fa6481566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:29.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa645d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa64806fc90 0x7fa6481566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44598/0 (socket says 172.21.15.125:44598) 2023-10-30T12:48:29.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa645d9b640 1 -- 172.21.15.125:0/829550112 learned_addr learned my addr 172.21.15.125:0/829550112 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:29.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa646d9d640 1 --2- 172.21.15.125:0/829550112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6480c1460 0x7fa6480c1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:29.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa64659c640 1 --2- 172.21.15.125:0/829550112 >> v2:172.21.15.136:3300/0 conn(0x7fa64814ff10 0x7fa6480c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:29.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa645d9b640 1 -- 172.21.15.125:0/829550112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6480c1460 msgr2=0x7fa6480c1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:29.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa645d9b640 1 --2- 172.21.15.125:0/829550112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6480c1460 0x7fa6480c1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa645d9b640 1 -- 172.21.15.125:0/829550112 >> v2:172.21.15.136:3300/0 conn(0x7fa64814ff10 msgr2=0x7fa6480c54d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:29.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa645d9b640 1 --2- 172.21.15.125:0/829550112 >> v2:172.21.15.136:3300/0 conn(0x7fa64814ff10 0x7fa6480c54d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa645d9b640 1 -- 172.21.15.125:0/829550112 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6480703f0 con 0x7fa64806fc90 2023-10-30T12:48:29.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa646d9d640 1 --2- 172.21.15.125:0/829550112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6480c1460 0x7fa6480c1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:29.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.288+0000 7fa64659c640 1 --2- 172.21.15.125:0/829550112 >> v2:172.21.15.136:3300/0 conn(0x7fa64814ff10 0x7fa6480c54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:29.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.292+0000 7fa645d9b640 1 --2- 172.21.15.125:0/829550112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64806fc90 0x7fa6481566b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa6480668b0 tx=0x7fa63c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb91511e94882425 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:29.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.292+0000 7fa64559a640 1 -- 172.21.15.125:0/829550112 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c0ac410 con 0x7fa64806fc90 2023-10-30T12:48:29.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.292+0000 7fa64559a640 1 -- 172.21.15.125:0/829550112 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa63c0ac570 con 0x7fa64806fc90 2023-10-30T12:48:29.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.292+0000 7fa64559a640 1 -- 172.21.15.125:0/829550112 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63c0ac900 con 0x7fa64806fc90 2023-10-30T12:48:29.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.292+0000 7fa64759e640 1 -- 172.21.15.125:0/829550112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64806fc90 msgr2=0x7fa6481566b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:29.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.292+0000 7fa64759e640 1 --2- 172.21.15.125:0/829550112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64806fc90 0x7fa6481566b0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa6480668b0 tx=0x7fa63c01c500 comp rx=0 tx=0).stop 2023-10-30T12:48:29.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.292+0000 7fa64759e640 1 -- 172.21.15.125:0/829550112 shutdown_connections 2023-10-30T12:48:29.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.292+0000 7fa64759e640 1 --2- 172.21.15.125:0/829550112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6480c1460 0x7fa6480c1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.292+0000 7fa64759e640 1 --2- 172.21.15.125:0/829550112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64806fc90 0x7fa6481566b0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.292+0000 7fa64759e640 1 --2- 172.21.15.125:0/829550112 >> v2:172.21.15.136:3300/0 conn(0x7fa64814ff10 0x7fa6480c54d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.292+0000 7fa64759e640 1 -- 172.21.15.125:0/829550112 >> 172.21.15.125:0/829550112 conn(0x7fa6480c75d0 msgr2=0x7fa6480c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:29.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.292+0000 7fa64759e640 1 -- 172.21.15.125:0/829550112 shutdown_connections 2023-10-30T12:48:29.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.292+0000 7fa64759e640 1 -- 172.21.15.125:0/829550112 wait complete. 2023-10-30T12:48:29.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa64759e640 1 Processor -- start 2023-10-30T12:48:29.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa64759e640 1 -- start start 2023-10-30T12:48:29.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa64759e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa64806fc90 0x7fa6481f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:29.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa64759e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6480c1460 0x7fa6480c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:29.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa64759e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64814ff10 0x7fa6480c2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:29.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa64759e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6481f1a80 con 0x7fa64814ff10 2023-10-30T12:48:29.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa64759e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa648147180 con 0x7fa64806fc90 2023-10-30T12:48:29.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa64759e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa648147460 con 0x7fa6480c1460 2023-10-30T12:48:29.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa646d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64814ff10 0x7fa6480c2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:29.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa646d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64814ff10 0x7fa6480c2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44606/0 (socket says 172.21.15.125:44606) 2023-10-30T12:48:29.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa646d9d640 1 -- 172.21.15.125:0/295328821 learned_addr learned my addr 172.21.15.125:0/295328821 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:29.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa645d9b640 1 --2- 172.21.15.125:0/295328821 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6480c1460 0x7fa6480c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:29.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa64659c640 1 --2- 172.21.15.125:0/295328821 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa64806fc90 0x7fa6481f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:29.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa646d9d640 1 -- 172.21.15.125:0/295328821 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6480c1460 msgr2=0x7fa6480c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:29.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa646d9d640 1 --2- 172.21.15.125:0/295328821 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6480c1460 0x7fa6480c36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa646d9d640 1 -- 172.21.15.125:0/295328821 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa64806fc90 msgr2=0x7fa6481f2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:29.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa646d9d640 1 --2- 172.21.15.125:0/295328821 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa64806fc90 0x7fa6481f2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa646d9d640 1 -- 172.21.15.125:0/295328821 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63c0094d0 con 0x7fa64814ff10 2023-10-30T12:48:29.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa646d9d640 1 --2- 172.21.15.125:0/295328821 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64814ff10 0x7fa6480c2e20 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fa640007e10 tx=0x7fa64000d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:29.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa636ffd640 1 -- 172.21.15.125:0/295328821 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa640003ba0 con 0x7fa64814ff10 2023-10-30T12:48:29.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa636ffd640 1 -- 172.21.15.125:0/295328821 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64000f040 con 0x7fa64814ff10 2023-10-30T12:48:29.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa636ffd640 1 -- 172.21.15.125:0/295328821 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa64000de80 con 0x7fa64814ff10 2023-10-30T12:48:29.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa64759e640 1 -- 172.21.15.125:0/295328821 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6481f1d20 con 0x7fa64814ff10 2023-10-30T12:48:29.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa64759e640 1 -- 172.21.15.125:0/295328821 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa648147740 con 0x7fa64814ff10 2023-10-30T12:48:29.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.296+0000 7fa64759e640 1 -- 172.21.15.125:0/295328821 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6481f2240 con 0x7fa64814ff10 2023-10-30T12:48:29.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.300+0000 7fa636ffd640 1 -- 172.21.15.125:0/295328821 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa640004030 con 0x7fa64814ff10 2023-10-30T12:48:29.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.300+0000 7fa636ffd640 1 --2- 172.21.15.125:0/295328821 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6280664d0 0x7fa628068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:29.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.300+0000 7fa636ffd640 1 -- 172.21.15.125:0/295328821 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fa640084420 con 0x7fa64814ff10 2023-10-30T12:48:29.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.300+0000 7fa64659c640 1 --2- 172.21.15.125:0/295328821 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6280664d0 0x7fa628068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:29.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.300+0000 7fa64659c640 1 --2- 172.21.15.125:0/295328821 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6280664d0 0x7fa628068990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa638006fd0 tx=0x7fa638008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:29.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.304+0000 7fa636ffd640 1 -- 172.21.15.125:0/295328821 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6481f2240 con 0x7fa64814ff10 2023-10-30T12:48:29.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.484+0000 7fa64759e640 1 -- 172.21.15.125:0/295328821 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6481f2240 con 0x7fa64814ff10 2023-10-30T12:48:29.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.488+0000 7fa636ffd640 1 -- 172.21.15.125:0/295328821 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5739 (secure 0 0 0) 0x7fa6481f2240 con 0x7fa64814ff10 2023-10-30T12:48:29.484 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:29.484 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:25.270309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:29.486 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 100 2023-10-30T12:48:29.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.488+0000 7fa64759e640 1 -- 172.21.15.125:0/295328821 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6280664d0 msgr2=0x7fa628068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:29.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.488+0000 7fa64759e640 1 --2- 172.21.15.125:0/295328821 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6280664d0 0x7fa628068990 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa638006fd0 tx=0x7fa638008040 comp rx=0 tx=0).stop 2023-10-30T12:48:29.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.488+0000 7fa64759e640 1 -- 172.21.15.125:0/295328821 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64814ff10 msgr2=0x7fa6480c2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:29.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.488+0000 7fa64759e640 1 --2- 172.21.15.125:0/295328821 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64814ff10 0x7fa6480c2e20 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fa640007e10 tx=0x7fa64000d4a0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.488+0000 7fa64759e640 1 -- 172.21.15.125:0/295328821 shutdown_connections 2023-10-30T12:48:29.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.488+0000 7fa64759e640 1 --2- 172.21.15.125:0/295328821 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6280664d0 0x7fa628068990 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.488+0000 7fa64759e640 1 --2- 172.21.15.125:0/295328821 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa64814ff10 0x7fa6480c2e20 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.488+0000 7fa64759e640 1 --2- 172.21.15.125:0/295328821 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6480c1460 0x7fa6480c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.488+0000 7fa64759e640 1 --2- 172.21.15.125:0/295328821 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa64806fc90 0x7fa6481f2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:29.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.488+0000 7fa64759e640 1 -- 172.21.15.125:0/295328821 >> 172.21.15.125:0/295328821 conn(0x7fa6480c75d0 msgr2=0x7fa64815efd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:29.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.488+0000 7fa64759e640 1 -- 172.21.15.125:0/295328821 shutdown_connections 2023-10-30T12:48:29.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:29.488+0000 7fa64759e640 1 -- 172.21.15.125:0/295328821 wait complete. 2023-10-30T12:48:29.504 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:31.505 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:31.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f8502d2a640 1 Processor -- start 2023-10-30T12:48:31.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f8502d2a640 1 -- start start 2023-10-30T12:48:31.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f8502d2a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84fc1538a0 0x7f84fc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:31.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f8502d2a640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84fc0cb800 con 0x7f84fc0ca0e0 2023-10-30T12:48:31.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f8502d2a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84fc15dc30 con 0x7f84fc0cf160 2023-10-30T12:48:31.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f8502d2a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc0cb330 con 0x7f84fc1538a0 2023-10-30T12:48:31.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f3fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f84fc0ca0e0 0x7f84fc15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:49422/0 (socket says 172.21.15.125:49422) 2023-10-30T12:48:31.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f3fff640 1 -- 172.21.15.125:0/303653739 learned_addr learned my addr 172.21.15.125:0/303653739 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:48:31.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f8500ac6640 1 --2- 172.21.15.125:0/303653739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84fc1538a0 0x7f84fc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:31.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 -- 172.21.15.125:0/303653739 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518508325 0 0) 0x7f84fc0cb800 con 0x7f84fc0ca0e0 2023-10-30T12:48:31.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 -- 172.21.15.125:0/303653739 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e4003530 con 0x7f84fc0ca0e0 2023-10-30T12:48:31.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 -- 172.21.15.125:0/303653739 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 436280629 0 0) 0x7f84e4003530 con 0x7f84fc0ca0e0 2023-10-30T12:48:31.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 -- 172.21.15.125:0/303653739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84fc1538a0 msgr2=0x7f84fc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:31.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 --2- 172.21.15.125:0/303653739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84fc1538a0 0x7f84fc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 -- 172.21.15.125:0/303653739 >> v1:172.21.15.136:6789/0 conn(0x7f84fc0cf160 legacy=0x7f84fc164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:31.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 -- 172.21.15.125:0/303653739 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc0cad70 con 0x7f84fc0ca0e0 2023-10-30T12:48:31.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 -- 172.21.15.125:0/303653739 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f84ec001680 con 0x7f84fc0ca0e0 2023-10-30T12:48:31.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 -- 172.21.15.125:0/303653739 >> v1:172.21.15.125:6789/0 conn(0x7f84fc0ca0e0 legacy=0x7f84fc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:31.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 --2- 172.21.15.125:0/303653739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84fc1538a0 0x7f84e40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:31.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 --2- 172.21.15.125:0/303653739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e4003ef0 0x7f84e40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:31.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 --2- 172.21.15.125:0/303653739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84e4004d10 0x7f84e4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:31.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 -- 172.21.15.125:0/303653739 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc0cb330 con 0x7f84e4003ef0 2023-10-30T12:48:31.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 -- 172.21.15.125:0/303653739 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e400b9f0 con 0x7f84e4004d10 2023-10-30T12:48:31.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f37fe640 1 -- 172.21.15.125:0/303653739 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e400bbb0 con 0x7f84fc1538a0 2023-10-30T12:48:31.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f3fff640 1 --2- 172.21.15.125:0/303653739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e4003ef0 0x7f84e40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:31.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f8500ac6640 1 --2- 172.21.15.125:0/303653739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84fc1538a0 0x7f84e40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:31.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f85012c7640 1 --2- 172.21.15.125:0/303653739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84e4004d10 0x7f84e4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:31.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.648+0000 7f84f3fff640 1 -- 172.21.15.125:0/303653739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84fc1538a0 msgr2=0x7f84e40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:31.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f84f3fff640 1 --2- 172.21.15.125:0/303653739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84fc1538a0 0x7f84e40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f84f3fff640 1 -- 172.21.15.125:0/303653739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84e4004d10 msgr2=0x7f84e4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:31.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f84f3fff640 1 --2- 172.21.15.125:0/303653739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84e4004d10 0x7f84e4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f84f3fff640 1 -- 172.21.15.125:0/303653739 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e4003530 con 0x7f84e4003ef0 2023-10-30T12:48:31.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f84f3fff640 1 --2- 172.21.15.125:0/303653739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e4003ef0 0x7f84e40047d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00c1c0 tx=0x7f84ec01f2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d44d15ecd08d432 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:31.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f84f37fe640 1 -- 172.21.15.125:0/303653739 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec0058e0 con 0x7f84e4003ef0 2023-10-30T12:48:31.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f84f37fe640 1 -- 172.21.15.125:0/303653739 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ec005a40 con 0x7f84e4003ef0 2023-10-30T12:48:31.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f84f37fe640 1 -- 172.21.15.125:0/303653739 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ec005e30 con 0x7f84e4003ef0 2023-10-30T12:48:31.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f8502d2a640 1 -- 172.21.15.125:0/303653739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e4003ef0 msgr2=0x7f84e40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:31.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f8502d2a640 1 --2- 172.21.15.125:0/303653739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84e4003ef0 0x7f84e40047d0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00c1c0 tx=0x7f84ec01f2d0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f85012c7640 1 -- 172.21.15.125:0/303653739 reap_dead start 2023-10-30T12:48:31.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f8502d2a640 1 -- 172.21.15.125:0/303653739 shutdown_connections 2023-10-30T12:48:31.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f8502d2a640 1 -- 172.21.15.125:0/303653739 >> 172.21.15.125:0/303653739 conn(0x7f84fc06bfe0 msgr2=0x7f84fc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:31.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f8502d2a640 1 -- 172.21.15.125:0/303653739 shutdown_connections 2023-10-30T12:48:31.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.652+0000 7f8502d2a640 1 -- 172.21.15.125:0/303653739 wait complete. 2023-10-30T12:48:31.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f8502d2a640 1 Processor -- start 2023-10-30T12:48:31.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f8502d2a640 1 -- start start 2023-10-30T12:48:31.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f8502d2a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84e4003ef0 0x7f84fc0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:31.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f8502d2a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84fc157520 0x7f84fc1527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:31.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f8502d2a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84fc0cd630 0x7f84fc0cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:31.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f8502d2a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc159750 con 0x7f84fc0cd630 2023-10-30T12:48:31.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f8502d2a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc0b5120 con 0x7f84fc157520 2023-10-30T12:48:31.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f8502d2a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84fc0cdff0 con 0x7f84e4003ef0 2023-10-30T12:48:31.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f85012c7640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84fc0cd630 0x7f84fc0cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:31.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f8500ac6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84e4003ef0 0x7f84fc0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:31.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f85012c7640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84fc0cd630 0x7f84fc0cdab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39536/0 (socket says 172.21.15.125:39536) 2023-10-30T12:48:31.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f84f3fff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84fc157520 0x7f84fc1527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:31.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f85012c7640 1 -- 172.21.15.125:0/3301146668 learned_addr learned my addr 172.21.15.125:0/3301146668 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:31.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f85012c7640 1 -- 172.21.15.125:0/3301146668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84e4003ef0 msgr2=0x7f84fc0b41f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:31.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f85012c7640 1 --2- 172.21.15.125:0/3301146668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84e4003ef0 0x7f84fc0b41f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f85012c7640 1 -- 172.21.15.125:0/3301146668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84fc157520 msgr2=0x7f84fc1527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:31.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f85012c7640 1 --2- 172.21.15.125:0/3301146668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84fc157520 0x7f84fc1527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f85012c7640 1 -- 172.21.15.125:0/3301146668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84fc159190 con 0x7f84fc0cd630 2023-10-30T12:48:31.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f85012c7640 1 --2- 172.21.15.125:0/3301146668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84fc0cd630 0x7f84fc0cdab0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f84f4007fd0 tx=0x7f84f400feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:31.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.656+0000 7f84f17fa640 1 -- 172.21.15.125:0/3301146668 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4006d70 con 0x7f84fc0cd630 2023-10-30T12:48:31.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.660+0000 7f84f17fa640 1 -- 172.21.15.125:0/3301146668 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f4003980 con 0x7f84fc0cd630 2023-10-30T12:48:31.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.660+0000 7f8502d2a640 1 -- 172.21.15.125:0/3301146668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84fc159470 con 0x7f84fc0cd630 2023-10-30T12:48:31.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.660+0000 7f84f17fa640 1 -- 172.21.15.125:0/3301146668 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4003d30 con 0x7f84fc0cd630 2023-10-30T12:48:31.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.660+0000 7f8502d2a640 1 -- 172.21.15.125:0/3301146668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84fc0b4e40 con 0x7f84fc0cd630 2023-10-30T12:48:31.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.660+0000 7f8502d2a640 1 -- 172.21.15.125:0/3301146668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84fc0b49c0 con 0x7f84fc0cd630 2023-10-30T12:48:31.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.660+0000 7f84f17fa640 1 -- 172.21.15.125:0/3301146668 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f84f40050d0 con 0x7f84fc0cd630 2023-10-30T12:48:31.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.660+0000 7f84f17fa640 1 --2- 172.21.15.125:0/3301146668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f84e80664d0 0x7f84e8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:31.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.660+0000 7f84f17fa640 1 -- 172.21.15.125:0/3301146668 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f84f4083220 con 0x7f84fc0cd630 2023-10-30T12:48:31.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.660+0000 7f8500ac6640 1 --2- 172.21.15.125:0/3301146668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f84e80664d0 0x7f84e8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:31.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.660+0000 7f8500ac6640 1 --2- 172.21.15.125:0/3301146668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f84e80664d0 0x7f84e8068990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009be0 tx=0x7f84f8009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:31.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.664+0000 7f84f17fa640 1 -- 172.21.15.125:0/3301146668 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f84fc0b49c0 con 0x7f84fc0cd630 2023-10-30T12:48:31.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.848+0000 7f8502d2a640 1 -- 172.21.15.125:0/3301146668 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84fc0cd350 con 0x7f84fc0cd630 2023-10-30T12:48:31.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.848+0000 7f84f17fa640 1 -- 172.21.15.125:0/3301146668 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+5739 (secure 0 0 0) 0x7f84fc0cd350 con 0x7f84fc0cd630 2023-10-30T12:48:31.846 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:31.846 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":100,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":100,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:25.270309+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:31.846 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 100 2023-10-30T12:48:31.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.848+0000 7f8502d2a640 1 -- 172.21.15.125:0/3301146668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f84e80664d0 msgr2=0x7f84e8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:31.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.848+0000 7f8502d2a640 1 --2- 172.21.15.125:0/3301146668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f84e80664d0 0x7f84e8068990 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f84f8009be0 tx=0x7f84f8009190 comp rx=0 tx=0).stop 2023-10-30T12:48:31.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.848+0000 7f8502d2a640 1 -- 172.21.15.125:0/3301146668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84fc0cd630 msgr2=0x7f84fc0cdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:31.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.848+0000 7f8502d2a640 1 --2- 172.21.15.125:0/3301146668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84fc0cd630 0x7f84fc0cdab0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f84f4007fd0 tx=0x7f84f400feb0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.848+0000 7f8502d2a640 1 -- 172.21.15.125:0/3301146668 shutdown_connections 2023-10-30T12:48:31.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.848+0000 7f8502d2a640 1 --2- 172.21.15.125:0/3301146668 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f84e80664d0 0x7f84e8068990 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.848+0000 7f8502d2a640 1 --2- 172.21.15.125:0/3301146668 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84fc0cd630 0x7f84fc0cdab0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.848+0000 7f8502d2a640 1 --2- 172.21.15.125:0/3301146668 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f84fc157520 0x7f84fc1527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.848+0000 7f8502d2a640 1 --2- 172.21.15.125:0/3301146668 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84e4003ef0 0x7f84fc0b41f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.848+0000 7f8502d2a640 1 -- 172.21.15.125:0/3301146668 >> 172.21.15.125:0/3301146668 conn(0x7f84fc06bfe0 msgr2=0x7f84fc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.848+0000 7f8502d2a640 1 -- 172.21.15.125:0/3301146668 shutdown_connections 2023-10-30T12:48:31.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:31.852+0000 7f8502d2a640 1 -- 172.21.15.125:0/3301146668 wait complete. 2023-10-30T12:48:31.866 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:33.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:34.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff5927d3640 1 Processor -- start 2023-10-30T12:48:34.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff5927d3640 1 -- start start 2023-10-30T12:48:34.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff5927d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff58c1538a0 0x7ff58c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:34.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff5927d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff58c0cf160 0x7ff58c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:34.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff5927d3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58c0cb800 con 0x7ff58c0ca0e0 2023-10-30T12:48:34.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff5927d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58c15dc30 con 0x7ff58c1538a0 2023-10-30T12:48:34.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff5927d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58c0cb330 con 0x7ff58c0cf160 2023-10-30T12:48:34.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff58bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff58c1538a0 0x7ff58c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:34.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff58bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff58c1538a0 0x7ff58c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39548/0 (socket says 172.21.15.125:39548) 2023-10-30T12:48:34.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff58bfff640 1 -- 172.21.15.125:0/29747775 learned_addr learned my addr 172.21.15.125:0/29747775 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:34.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff590d70640 1 --2- 172.21.15.125:0/29747775 >> v2:172.21.15.136:3300/0 conn(0x7ff58c0cf160 0x7ff58c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:34.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff58bfff640 1 -- 172.21.15.125:0/29747775 >> v2:172.21.15.136:3300/0 conn(0x7ff58c0cf160 msgr2=0x7ff58c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:34.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff58bfff640 1 --2- 172.21.15.125:0/29747775 >> v2:172.21.15.136:3300/0 conn(0x7ff58c0cf160 0x7ff58c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:34.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff58bfff640 1 -- 172.21.15.125:0/29747775 >> v1:172.21.15.136:6789/0 conn(0x7ff58c0ca0e0 legacy=0x7ff58c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:34.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.016+0000 7ff58bfff640 1 -- 172.21.15.125:0/29747775 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c0cad70 con 0x7ff58c1538a0 2023-10-30T12:48:34.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.020+0000 7ff58bfff640 1 --2- 172.21.15.125:0/29747775 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff58c1538a0 0x7ff58c15d6f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff58c0b5330 tx=0x7ff57c01ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=256134f30c3250e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:34.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.020+0000 7ff58b7fe640 1 -- 172.21.15.125:0/29747775 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c0a89b0 con 0x7ff58c1538a0 2023-10-30T12:48:34.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.020+0000 7ff58b7fe640 1 -- 172.21.15.125:0/29747775 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c0a8b10 con 0x7ff58c1538a0 2023-10-30T12:48:34.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.020+0000 7ff58b7fe640 1 -- 172.21.15.125:0/29747775 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c0a8ea0 con 0x7ff58c1538a0 2023-10-30T12:48:34.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.020+0000 7ff5927d3640 1 -- 172.21.15.125:0/29747775 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff58c1538a0 msgr2=0x7ff58c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:34.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.020+0000 7ff5927d3640 1 --2- 172.21.15.125:0/29747775 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff58c1538a0 0x7ff58c15d6f0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7ff58c0b5330 tx=0x7ff57c01ecb0 comp rx=0 tx=0).stop 2023-10-30T12:48:34.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.020+0000 7ff5927d3640 1 -- 172.21.15.125:0/29747775 shutdown_connections 2023-10-30T12:48:34.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.020+0000 7ff5927d3640 1 --2- 172.21.15.125:0/29747775 >> v2:172.21.15.136:3300/0 conn(0x7ff58c0cf160 0x7ff58c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:34.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.020+0000 7ff5927d3640 1 --2- 172.21.15.125:0/29747775 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff58c1538a0 0x7ff58c15d6f0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:34.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.020+0000 7ff5927d3640 1 -- 172.21.15.125:0/29747775 >> 172.21.15.125:0/29747775 conn(0x7ff58c06bfe0 msgr2=0x7ff58c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:34.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.020+0000 7ff5927d3640 1 -- 172.21.15.125:0/29747775 shutdown_connections 2023-10-30T12:48:34.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.020+0000 7ff5927d3640 1 -- 172.21.15.125:0/29747775 wait complete. 2023-10-30T12:48:34.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff5927d3640 1 Processor -- start 2023-10-30T12:48:34.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff5927d3640 1 -- start start 2023-10-30T12:48:34.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff5927d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff58c0ca0e0 0x7ff58c0c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:34.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff5927d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff58c0cf160 0x7ff58c157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:34.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff5927d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff58c1538a0 0x7ff58c1e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:34.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff5927d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c1ef4c0 con 0x7ff58c1538a0 2023-10-30T12:48:34.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff5927d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c0c9180 con 0x7ff58c0cf160 2023-10-30T12:48:34.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff5927d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c1ef730 con 0x7ff58c0ca0e0 2023-10-30T12:48:34.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff590d70640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff58c1538a0 0x7ff58c1e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:34.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff590d70640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff58c1538a0 0x7ff58c1e5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39564/0 (socket says 172.21.15.125:39564) 2023-10-30T12:48:34.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff590d70640 1 -- 172.21.15.125:0/1148873364 learned_addr learned my addr 172.21.15.125:0/1148873364 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:34.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff583fff640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff58c0cf160 0x7ff58c157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:34.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff58bfff640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff58c0ca0e0 0x7ff58c0c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:34.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff590d70640 1 -- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff58c0ca0e0 msgr2=0x7ff58c0c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:34.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff590d70640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff58c0ca0e0 0x7ff58c0c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:34.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff590d70640 1 -- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff58c0cf160 msgr2=0x7ff58c157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:34.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff590d70640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff58c0cf160 0x7ff58c157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:34.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff590d70640 1 -- 172.21.15.125:0/1148873364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff57c0094d0 con 0x7ff58c1538a0 2023-10-30T12:48:34.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff58bfff640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff58c0ca0e0 0x7ff58c0c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:34.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff590d70640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff58c1538a0 0x7ff58c1e5040 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff584007d90 tx=0x7ff58400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:34.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff5897fa640 1 -- 172.21.15.125:0/1148873364 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584014420 con 0x7ff58c1538a0 2023-10-30T12:48:34.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff5897fa640 1 -- 172.21.15.125:0/1148873364 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5840145a0 con 0x7ff58c1538a0 2023-10-30T12:48:34.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff5897fa640 1 -- 172.21.15.125:0/1148873364 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff584014950 con 0x7ff58c1538a0 2023-10-30T12:48:34.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff5927d3640 1 -- 172.21.15.125:0/1148873364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff58c1efa10 con 0x7ff58c1538a0 2023-10-30T12:48:34.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.024+0000 7ff5927d3640 1 -- 172.21.15.125:0/1148873364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58c0c9460 con 0x7ff58c1538a0 2023-10-30T12:48:34.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.028+0000 7ff5927d3640 1 -- 172.21.15.125:0/1148873364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58c0b2a50 con 0x7ff58c1538a0 2023-10-30T12:48:34.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.028+0000 7ff5897fa640 1 -- 172.21.15.125:0/1148873364 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff584014ab0 con 0x7ff58c1538a0 2023-10-30T12:48:34.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.028+0000 7ff5897fa640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff56c0664a0 0x7ff56c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:34.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.028+0000 7ff5897fa640 1 -- 172.21.15.125:0/1148873364 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7ff584084bd0 con 0x7ff58c1538a0 2023-10-30T12:48:34.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.028+0000 7ff58bfff640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff56c0664a0 0x7ff56c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:34.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.028+0000 7ff58bfff640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff56c0664a0 0x7ff56c068960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0b2000 tx=0x7ff57c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:34.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.032+0000 7ff5897fa640 1 -- 172.21.15.125:0/1148873364 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff584051ba0 con 0x7ff58c1538a0 2023-10-30T12:48:34.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.212+0000 7ff5927d3640 1 -- 172.21.15.125:0/1148873364 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff58c04e170 con 0x7ff58c1538a0 2023-10-30T12:48:34.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.212+0000 7ff5897fa640 1 -- 172.21.15.125:0/1148873364 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5741 (secure 0 0 0) 0x7ff5840551c0 con 0x7ff58c1538a0 2023-10-30T12:48:34.210 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:34.210 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:33.179042+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:resolve","state_seq":10,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:34.213 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 101 2023-10-30T12:48:34.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.216+0000 7ff5927d3640 1 -- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff56c0664a0 msgr2=0x7ff56c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:34.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.216+0000 7ff5927d3640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff56c0664a0 0x7ff56c068960 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0b2000 tx=0x7ff57c008040 comp rx=0 tx=0).stop 2023-10-30T12:48:34.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.216+0000 7ff5927d3640 1 -- 172.21.15.125:0/1148873364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff58c1538a0 msgr2=0x7ff58c1e5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:34.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.216+0000 7ff5927d3640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff58c1538a0 0x7ff58c1e5040 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff584007d90 tx=0x7ff58400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:48:34.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.216+0000 7ff5927d3640 1 -- 172.21.15.125:0/1148873364 shutdown_connections 2023-10-30T12:48:34.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.216+0000 7ff5927d3640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff56c0664a0 0x7ff56c068960 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:34.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.216+0000 7ff5927d3640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff58c1538a0 0x7ff58c1e5040 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:34.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.216+0000 7ff5927d3640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff58c0cf160 0x7ff58c157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:34.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.216+0000 7ff5927d3640 1 --2- 172.21.15.125:0/1148873364 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff58c0ca0e0 0x7ff58c0c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:34.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.216+0000 7ff5927d3640 1 -- 172.21.15.125:0/1148873364 >> 172.21.15.125:0/1148873364 conn(0x7ff58c06bfe0 msgr2=0x7ff58c1519e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:34.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.216+0000 7ff5927d3640 1 -- 172.21.15.125:0/1148873364 shutdown_connections 2023-10-30T12:48:34.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:34.216+0000 7ff5927d3640 1 -- 172.21.15.125:0/1148873364 wait complete. 2023-10-30T12:48:34.230 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:36.232 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:36.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a8ad640 1 Processor -- start 2023-10-30T12:48:36.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a8ad640 1 -- start start 2023-10-30T12:48:36.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a8ad640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46841538f0 0x7f4684159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:36.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a8ad640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f468415cd50 0x7f46840c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:36.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a8ad640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f46840cefa0 0x7f46840cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:36.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a8ad640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f468415d300 con 0x7f46840cefa0 2023-10-30T12:48:36.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a8ad640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f468415d5e0 con 0x7f468415cd50 2023-10-30T12:48:36.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a8ad640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468415d8c0 con 0x7f46841538f0 2023-10-30T12:48:36.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a0ac640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f46840cefa0 0x7f46840cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:36.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a0ac640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f46840cefa0 0x7f46840cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39580/0 (socket says 172.21.15.125:39580) 2023-10-30T12:48:36.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a0ac640 1 -- 172.21.15.125:0/1563562963 learned_addr learned my addr 172.21.15.125:0/1563562963 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:36.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f46898ab640 1 --2- 172.21.15.125:0/1563562963 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46841538f0 0x7f4684159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:36.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f46890aa640 1 --2- 172.21.15.125:0/1563562963 >> v2:172.21.15.136:3300/0 conn(0x7f468415cd50 0x7f46840c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:36.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a0ac640 1 -- 172.21.15.125:0/1563562963 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46841538f0 msgr2=0x7f4684159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:36.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a0ac640 1 --2- 172.21.15.125:0/1563562963 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46841538f0 0x7f4684159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a0ac640 1 -- 172.21.15.125:0/1563562963 >> v2:172.21.15.136:3300/0 conn(0x7f468415cd50 msgr2=0x7f46840c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:36.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a0ac640 1 --2- 172.21.15.125:0/1563562963 >> v2:172.21.15.136:3300/0 conn(0x7f468415cd50 0x7f46840c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f468a0ac640 1 -- 172.21.15.125:0/1563562963 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468415dba0 con 0x7f46840cefa0 2023-10-30T12:48:36.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.383+0000 7f46890aa640 1 --2- 172.21.15.125:0/1563562963 >> v2:172.21.15.136:3300/0 conn(0x7f468415cd50 0x7f46840c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:36.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.387+0000 7f468a0ac640 1 --2- 172.21.15.125:0/1563562963 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840cefa0 0x7f46840cf3f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4684066410 tx=0x7f467c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df764e9aa26f7f83 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:36.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.387+0000 7f46888a9640 1 -- 172.21.15.125:0/1563562963 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c0af400 con 0x7f46840cefa0 2023-10-30T12:48:36.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.387+0000 7f46888a9640 1 -- 172.21.15.125:0/1563562963 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467c0ab050 con 0x7f46840cefa0 2023-10-30T12:48:36.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.387+0000 7f46888a9640 1 -- 172.21.15.125:0/1563562963 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467c0af780 con 0x7f46840cefa0 2023-10-30T12:48:36.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.387+0000 7f468a8ad640 1 -- 172.21.15.125:0/1563562963 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840cefa0 msgr2=0x7f46840cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:36.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.387+0000 7f468a8ad640 1 --2- 172.21.15.125:0/1563562963 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840cefa0 0x7f46840cf3f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f4684066410 tx=0x7f467c01f1c0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.387+0000 7f468a8ad640 1 -- 172.21.15.125:0/1563562963 shutdown_connections 2023-10-30T12:48:36.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.387+0000 7f468a8ad640 1 --2- 172.21.15.125:0/1563562963 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840cefa0 0x7f46840cf3f0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.387+0000 7f468a8ad640 1 --2- 172.21.15.125:0/1563562963 >> v2:172.21.15.136:3300/0 conn(0x7f468415cd50 0x7f46840c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.387+0000 7f468a8ad640 1 --2- 172.21.15.125:0/1563562963 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46841538f0 0x7f4684159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.387+0000 7f468a8ad640 1 -- 172.21.15.125:0/1563562963 >> 172.21.15.125:0/1563562963 conn(0x7f468406a640 msgr2=0x7f4684065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:36.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.387+0000 7f468a8ad640 1 -- 172.21.15.125:0/1563562963 shutdown_connections 2023-10-30T12:48:36.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.387+0000 7f468a8ad640 1 -- 172.21.15.125:0/1563562963 wait complete. 2023-10-30T12:48:36.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f468a8ad640 1 Processor -- start 2023-10-30T12:48:36.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f468a8ad640 1 -- start start 2023-10-30T12:48:36.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f468a8ad640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840cefa0 0x7f4684144640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:36.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f468a8ad640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46841538f0 0x7f4684152490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:36.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f468a8ad640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f468415cd50 0x7f4684145530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:36.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f468a8ad640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46841effd0 con 0x7f46840cefa0 2023-10-30T12:48:36.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f468a8ad640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4684144d90 con 0x7f468415cd50 2023-10-30T12:48:36.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f468a8ad640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46841efa50 con 0x7f46841538f0 2023-10-30T12:48:36.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f46898ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840cefa0 0x7f4684144640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:36.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f46898ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840cefa0 0x7f4684144640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39582/0 (socket says 172.21.15.125:39582) 2023-10-30T12:48:36.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f46898ab640 1 -- 172.21.15.125:0/1876433843 learned_addr learned my addr 172.21.15.125:0/1876433843 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:36.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f46890aa640 1 --2- 172.21.15.125:0/1876433843 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46841538f0 0x7f4684152490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:36.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f468a0ac640 1 --2- 172.21.15.125:0/1876433843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f468415cd50 0x7f4684145530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:36.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f46898ab640 1 -- 172.21.15.125:0/1876433843 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46841538f0 msgr2=0x7f4684152490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:36.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f46898ab640 1 --2- 172.21.15.125:0/1876433843 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46841538f0 0x7f4684152490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f46898ab640 1 -- 172.21.15.125:0/1876433843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f468415cd50 msgr2=0x7f4684145530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:36.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f46898ab640 1 --2- 172.21.15.125:0/1876433843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f468415cd50 0x7f4684145530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f46898ab640 1 -- 172.21.15.125:0/1876433843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f467c009b30 con 0x7f46840cefa0 2023-10-30T12:48:36.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f46898ab640 1 --2- 172.21.15.125:0/1876433843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840cefa0 0x7f4684144640 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f468000e8b0 tx=0x7f468000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:36.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f467a7fc640 1 -- 172.21.15.125:0/1876433843 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468000c940 con 0x7f46840cefa0 2023-10-30T12:48:36.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f467a7fc640 1 -- 172.21.15.125:0/1876433843 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468000caa0 con 0x7f46840cefa0 2023-10-30T12:48:36.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f468a8ad640 1 -- 172.21.15.125:0/1876433843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46841efcf0 con 0x7f46840cefa0 2023-10-30T12:48:36.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f467a7fc640 1 -- 172.21.15.125:0/1876433843 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468000ce30 con 0x7f46840cefa0 2023-10-30T12:48:36.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.391+0000 7f468a8ad640 1 -- 172.21.15.125:0/1876433843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4684146b80 con 0x7f46840cefa0 2023-10-30T12:48:36.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.395+0000 7f468a8ad640 1 -- 172.21.15.125:0/1876433843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468404e170 con 0x7f46840cefa0 2023-10-30T12:48:36.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.395+0000 7f467a7fc640 1 -- 172.21.15.125:0/1876433843 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4680013470 con 0x7f46840cefa0 2023-10-30T12:48:36.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.395+0000 7f467a7fc640 1 --2- 172.21.15.125:0/1876433843 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46680664a0 0x7f4668068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:36.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.395+0000 7f467a7fc640 1 -- 172.21.15.125:0/1876433843 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f4680084840 con 0x7f46840cefa0 2023-10-30T12:48:36.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.395+0000 7f46890aa640 1 --2- 172.21.15.125:0/1876433843 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46680664a0 0x7f4668068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:36.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.395+0000 7f46890aa640 1 --2- 172.21.15.125:0/1876433843 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46680664a0 0x7f4668068960 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f468406c7f0 tx=0x7f4674008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:36.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.399+0000 7f467a7fc640 1 -- 172.21.15.125:0/1876433843 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4680051810 con 0x7f46840cefa0 2023-10-30T12:48:36.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.579+0000 7f468a8ad640 1 -- 172.21.15.125:0/1876433843 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46840c5690 con 0x7f46840cefa0 2023-10-30T12:48:36.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.579+0000 7f467a7fc640 1 -- 172.21.15.125:0/1876433843 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5741 (secure 0 0 0) 0x7f4680054e30 con 0x7f46840cefa0 2023-10-30T12:48:36.579 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:36.579 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:33.179042+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:resolve","state_seq":10,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:36.579 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 101 2023-10-30T12:48:36.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.583+0000 7f468a8ad640 1 -- 172.21.15.125:0/1876433843 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46680664a0 msgr2=0x7f4668068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:36.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.583+0000 7f468a8ad640 1 --2- 172.21.15.125:0/1876433843 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46680664a0 0x7f4668068960 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f468406c7f0 tx=0x7f4674008040 comp rx=0 tx=0).stop 2023-10-30T12:48:36.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.583+0000 7f468a8ad640 1 -- 172.21.15.125:0/1876433843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840cefa0 msgr2=0x7f4684144640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:36.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.583+0000 7f468a8ad640 1 --2- 172.21.15.125:0/1876433843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840cefa0 0x7f4684144640 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f468000e8b0 tx=0x7f468000ed70 comp rx=0 tx=0).stop 2023-10-30T12:48:36.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.583+0000 7f468a8ad640 1 -- 172.21.15.125:0/1876433843 shutdown_connections 2023-10-30T12:48:36.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.583+0000 7f468a8ad640 1 --2- 172.21.15.125:0/1876433843 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46680664a0 0x7f4668068960 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.583+0000 7f468a8ad640 1 --2- 172.21.15.125:0/1876433843 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f468415cd50 0x7f4684145530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.583+0000 7f468a8ad640 1 --2- 172.21.15.125:0/1876433843 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f46841538f0 0x7f4684152490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.583+0000 7f468a8ad640 1 --2- 172.21.15.125:0/1876433843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46840cefa0 0x7f4684144640 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:36.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.583+0000 7f468a8ad640 1 -- 172.21.15.125:0/1876433843 >> 172.21.15.125:0/1876433843 conn(0x7f468406a640 msgr2=0x7f46840cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:36.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.583+0000 7f468a8ad640 1 -- 172.21.15.125:0/1876433843 shutdown_connections 2023-10-30T12:48:36.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:36.583+0000 7f468a8ad640 1 -- 172.21.15.125:0/1876433843 wait complete. 2023-10-30T12:48:36.599 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:38.601 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:38.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff16ea17640 1 Processor -- start 2023-10-30T12:48:38.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff16ea17640 1 -- start start 2023-10-30T12:48:38.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff16ea17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff168153bf0 0x7ff168071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:38.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff16ea17640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff1680c6fe0 0x7ff1680c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:38.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff16ea17640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1680700b0 con 0x7ff1680c79e0 2023-10-30T12:48:38.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff16ea17640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1680c6810 con 0x7ff1680c6fe0 2023-10-30T12:48:38.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff16ea17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff168159900 con 0x7ff168153bf0 2023-10-30T12:48:38.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff1677fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff1680c6fe0 0x7ff1680c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:38.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff1677fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff1680c6fe0 0x7ff1680c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39586/0 (socket says 172.21.15.125:39586) 2023-10-30T12:48:38.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff1677fe640 1 -- 172.21.15.125:0/3138955472 learned_addr learned my addr 172.21.15.125:0/3138955472 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:38.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff167fff640 1 --2- 172.21.15.125:0/3138955472 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff168153bf0 0x7ff168071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:38.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff1677fe640 1 -- 172.21.15.125:0/3138955472 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff168153bf0 msgr2=0x7ff168071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:38.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff1677fe640 1 --2- 172.21.15.125:0/3138955472 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff168153bf0 0x7ff168071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff1677fe640 1 -- 172.21.15.125:0/3138955472 >> v1:172.21.15.125:6789/0 conn(0x7ff1680c79e0 legacy=0x7ff1680cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:38.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.751+0000 7ff1677fe640 1 -- 172.21.15.125:0/3138955472 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1680c6530 con 0x7ff1680c6fe0 2023-10-30T12:48:38.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.755+0000 7ff1677fe640 1 --2- 172.21.15.125:0/3138955472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1680c6fe0 0x7ff1680c74a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ff15c009870 tx=0x7ff15c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4080f278a1a9c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:38.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.755+0000 7ff166ffd640 1 -- 172.21.15.125:0/3138955472 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c0a8a30 con 0x7ff1680c6fe0 2023-10-30T12:48:38.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.755+0000 7ff166ffd640 1 -- 172.21.15.125:0/3138955472 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15c0a8b90 con 0x7ff1680c6fe0 2023-10-30T12:48:38.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.755+0000 7ff166ffd640 1 -- 172.21.15.125:0/3138955472 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c0b0440 con 0x7ff1680c6fe0 2023-10-30T12:48:38.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.755+0000 7ff16ea17640 1 -- 172.21.15.125:0/3138955472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1680c6fe0 msgr2=0x7ff1680c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:38.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.755+0000 7ff16ea17640 1 --2- 172.21.15.125:0/3138955472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1680c6fe0 0x7ff1680c74a0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ff15c009870 tx=0x7ff15c01ed30 comp rx=0 tx=0).stop 2023-10-30T12:48:38.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.755+0000 7ff16ea17640 1 -- 172.21.15.125:0/3138955472 shutdown_connections 2023-10-30T12:48:38.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.755+0000 7ff16ea17640 1 --2- 172.21.15.125:0/3138955472 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1680c6fe0 0x7ff1680c74a0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.755+0000 7ff16ea17640 1 --2- 172.21.15.125:0/3138955472 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff168153bf0 0x7ff168071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.755+0000 7ff16ea17640 1 -- 172.21.15.125:0/3138955472 >> 172.21.15.125:0/3138955472 conn(0x7ff16806c6e0 msgr2=0x7ff168069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:38.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.755+0000 7ff16ea17640 1 -- 172.21.15.125:0/3138955472 shutdown_connections 2023-10-30T12:48:38.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.755+0000 7ff16ea17640 1 -- 172.21.15.125:0/3138955472 wait complete. 2023-10-30T12:48:38.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff16ea17640 1 Processor -- start 2023-10-30T12:48:38.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff16ea17640 1 -- start start 2023-10-30T12:48:38.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff16ea17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1680c6fe0 0x7ff1680cac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:38.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff16ea17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1680c79e0 0x7ff1681eed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:38.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff16ea17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff168153bf0 0x7ff1681ef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:38.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff16ea17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff168145860 con 0x7ff1680c79e0 2023-10-30T12:48:38.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff16ea17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff168158dd0 con 0x7ff1680c6fe0 2023-10-30T12:48:38.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff16ea17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff16814a4b0 con 0x7ff168153bf0 2023-10-30T12:48:38.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff1677fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1680c79e0 0x7ff1681eed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:38.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff1677fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1680c79e0 0x7ff1681eed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39588/0 (socket says 172.21.15.125:39588) 2023-10-30T12:48:38.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff1677fe640 1 -- 172.21.15.125:0/2620723296 learned_addr learned my addr 172.21.15.125:0/2620723296 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:38.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff167fff640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1680c6fe0 0x7ff1680cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:38.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff16cfb4640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff168153bf0 0x7ff1681ef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:38.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff1677fe640 1 -- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff168153bf0 msgr2=0x7ff1681ef2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:38.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff1677fe640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff168153bf0 0x7ff1681ef2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff1677fe640 1 -- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1680c6fe0 msgr2=0x7ff1680cac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:38.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff1677fe640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1680c6fe0 0x7ff1680cac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff1677fe640 1 -- 172.21.15.125:0/2620723296 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff15c0094d0 con 0x7ff1680c79e0 2023-10-30T12:48:38.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff16cfb4640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff168153bf0 0x7ff1681ef2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:48:38.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff1677fe640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1680c79e0 0x7ff1681eed80 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff15c0b3040 tx=0x7ff15c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:38.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff164ff9640 1 -- 172.21.15.125:0/2620723296 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c0ba600 con 0x7ff1680c79e0 2023-10-30T12:48:38.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff164ff9640 1 -- 172.21.15.125:0/2620723296 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15c0ba780 con 0x7ff1680c79e0 2023-10-30T12:48:38.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff16ea17640 1 -- 172.21.15.125:0/2620723296 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff168149dd0 con 0x7ff1680c79e0 2023-10-30T12:48:38.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff164ff9640 1 -- 172.21.15.125:0/2620723296 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c0bab30 con 0x7ff1680c79e0 2023-10-30T12:48:38.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff16ea17640 1 -- 172.21.15.125:0/2620723296 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff16814a770 con 0x7ff1680c79e0 2023-10-30T12:48:38.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.759+0000 7ff16ea17640 1 -- 172.21.15.125:0/2620723296 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff16804e170 con 0x7ff1680c79e0 2023-10-30T12:48:38.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.763+0000 7ff164ff9640 1 -- 172.21.15.125:0/2620723296 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff15c0bac90 con 0x7ff1680c79e0 2023-10-30T12:48:38.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.763+0000 7ff164ff9640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1540664d0 0x7ff154068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:38.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.763+0000 7ff164ff9640 1 -- 172.21.15.125:0/2620723296 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7ff15c122790 con 0x7ff1680c79e0 2023-10-30T12:48:38.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.763+0000 7ff167fff640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1540664d0 0x7ff154068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:38.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.763+0000 7ff167fff640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1540664d0 0x7ff154068990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff15800b930 tx=0x7ff158008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:38.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.767+0000 7ff164ff9640 1 -- 172.21.15.125:0/2620723296 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff15c0ef880 con 0x7ff1680c79e0 2023-10-30T12:48:38.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.947+0000 7ff16ea17640 1 -- 172.21.15.125:0/2620723296 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff16815a0a0 con 0x7ff1680c79e0 2023-10-30T12:48:38.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.947+0000 7ff164ff9640 1 -- 172.21.15.125:0/2620723296 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+5741 (secure 0 0 0) 0x7ff15c0f2ea0 con 0x7ff1680c79e0 2023-10-30T12:48:38.945 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:38.946 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:33.179042+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:resolve","state_seq":10,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:38.947 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 101 2023-10-30T12:48:38.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.947+0000 7ff16ea17640 1 -- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1540664d0 msgr2=0x7ff154068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:38.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.947+0000 7ff16ea17640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1540664d0 0x7ff154068990 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7ff15800b930 tx=0x7ff158008040 comp rx=0 tx=0).stop 2023-10-30T12:48:38.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.947+0000 7ff16ea17640 1 -- 172.21.15.125:0/2620723296 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1680c79e0 msgr2=0x7ff1681eed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:38.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.947+0000 7ff16ea17640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1680c79e0 0x7ff1681eed80 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff15c0b3040 tx=0x7ff15c01bad0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.947+0000 7ff16ea17640 1 -- 172.21.15.125:0/2620723296 shutdown_connections 2023-10-30T12:48:38.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.947+0000 7ff16ea17640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1540664d0 0x7ff154068990 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.947+0000 7ff16ea17640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff168153bf0 0x7ff1681ef2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.947+0000 7ff16ea17640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1680c79e0 0x7ff1681eed80 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.947+0000 7ff16ea17640 1 --2- 172.21.15.125:0/2620723296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1680c6fe0 0x7ff1680cac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:38.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.947+0000 7ff16ea17640 1 -- 172.21.15.125:0/2620723296 >> 172.21.15.125:0/2620723296 conn(0x7ff16806c6e0 msgr2=0x7ff168159020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:38.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.947+0000 7ff16ea17640 1 -- 172.21.15.125:0/2620723296 shutdown_connections 2023-10-30T12:48:38.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:38.947+0000 7ff16ea17640 1 -- 172.21.15.125:0/2620723296 wait complete. 2023-10-30T12:48:38.967 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 101, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 101, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6223": { "addr": "172.21.15.136:6839/3479573317", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3479573317, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3479573317, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6223, "incarnation": 100, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:resolve", "state_seq": 10 }, "gid_6318": { "addr": "172.21.15.125:6836/1721514617", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1721514617, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1721514617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 6318, "incarnation": 92, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 2, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:48:33.179042+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6318, "mds_1": 6223 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:48:40.969 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:41.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.115+0000 7ff0f4cc9640 1 Processor -- start 2023-10-30T12:48:41.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.115+0000 7ff0f4cc9640 1 -- start start 2023-10-30T12:48:41.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0f4cc9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f01538a0 0x7ff0f015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:41.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0f4cc9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff0f00cf160 0x7ff0f00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:41.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0f4cc9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0f00cb800 con 0x7ff0f00ca0e0 2023-10-30T12:48:41.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0f4cc9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0f015dc30 con 0x7ff0f00cf160 2023-10-30T12:48:41.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0f4cc9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00cb330 con 0x7ff0f01538a0 2023-10-30T12:48:41.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0e5d9b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff0f00ca0e0 0x7ff0f015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43506/0 (socket says 172.21.15.125:43506) 2023-10-30T12:48:41.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0e5d9b640 1 -- 172.21.15.125:0/3822513584 learned_addr learned my addr 172.21.15.125:0/3822513584 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:48:41.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0eed9d640 1 --2- 172.21.15.125:0/3822513584 >> v2:172.21.15.136:3300/0 conn(0x7ff0f00cf160 0x7ff0f00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:41.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0ee59c640 1 --2- 172.21.15.125:0/3822513584 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f01538a0 0x7ff0f015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:41.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 -- 172.21.15.125:0/3822513584 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958560725 0 0) 0x7ff0f00cb800 con 0x7ff0f00ca0e0 2023-10-30T12:48:41.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 -- 172.21.15.125:0/3822513584 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0dc003530 con 0x7ff0f00ca0e0 2023-10-30T12:48:41.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 -- 172.21.15.125:0/3822513584 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3445475345 0 0) 0x7ff0dc003530 con 0x7ff0f00ca0e0 2023-10-30T12:48:41.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 -- 172.21.15.125:0/3822513584 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f01538a0 msgr2=0x7ff0f015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:41.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 --2- 172.21.15.125:0/3822513584 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f01538a0 0x7ff0f015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:41.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 -- 172.21.15.125:0/3822513584 >> v2:172.21.15.136:3300/0 conn(0x7ff0f00cf160 msgr2=0x7ff0f00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:41.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 --2- 172.21.15.125:0/3822513584 >> v2:172.21.15.136:3300/0 conn(0x7ff0f00cf160 0x7ff0f00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:41.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 -- 172.21.15.125:0/3822513584 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f00cad70 con 0x7ff0f00ca0e0 2023-10-30T12:48:41.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 -- 172.21.15.125:0/3822513584 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7ff0d8001680 con 0x7ff0f00ca0e0 2023-10-30T12:48:41.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 -- 172.21.15.125:0/3822513584 >> v1:172.21.15.125:6789/0 conn(0x7ff0f00ca0e0 legacy=0x7ff0f015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:41.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 --2- 172.21.15.125:0/3822513584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0dc003980 0x7ff0dc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:41.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 --2- 172.21.15.125:0/3822513584 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f01538a0 0x7ff0dc0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:41.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 --2- 172.21.15.125:0/3822513584 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0dc004d10 0x7ff0dc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 -- 172.21.15.125:0/3822513584 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f015dc30 con 0x7ff0dc003980 2023-10-30T12:48:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 -- 172.21.15.125:0/3822513584 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00cb330 con 0x7ff0dc004d10 2023-10-30T12:48:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0edd9b640 1 -- 172.21.15.125:0/3822513584 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc00b990 con 0x7ff0f01538a0 2023-10-30T12:48:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0ee59c640 1 --2- 172.21.15.125:0/3822513584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0dc003980 0x7ff0dc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:41.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0e5d9b640 1 --2- 172.21.15.125:0/3822513584 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f01538a0 0x7ff0dc0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:41.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0eed9d640 1 --2- 172.21.15.125:0/3822513584 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0dc004d10 0x7ff0dc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:41.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0ee59c640 1 -- 172.21.15.125:0/3822513584 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f01538a0 msgr2=0x7ff0dc0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:41.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0ee59c640 1 --2- 172.21.15.125:0/3822513584 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f01538a0 0x7ff0dc0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:41.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0ee59c640 1 -- 172.21.15.125:0/3822513584 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0dc004d10 msgr2=0x7ff0dc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:41.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0ee59c640 1 --2- 172.21.15.125:0/3822513584 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0dc004d10 0x7ff0dc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:41.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.119+0000 7ff0ee59c640 1 -- 172.21.15.125:0/3822513584 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc003530 con 0x7ff0dc003980 2023-10-30T12:48:41.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.123+0000 7ff0ee59c640 1 --2- 172.21.15.125:0/3822513584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0dc003980 0x7ff0dc003da0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009ae0 tx=0x7ff0e001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0aa2d00a6433ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:41.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.123+0000 7ff0edd9b640 1 -- 172.21.15.125:0/3822513584 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e00a5530 con 0x7ff0dc003980 2023-10-30T12:48:41.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.123+0000 7ff0edd9b640 1 -- 172.21.15.125:0/3822513584 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0e00a5690 con 0x7ff0dc003980 2023-10-30T12:48:41.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.123+0000 7ff0edd9b640 1 -- 172.21.15.125:0/3822513584 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0e00a5a40 con 0x7ff0dc003980 2023-10-30T12:48:41.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.123+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/3822513584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0dc003980 msgr2=0x7ff0dc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:41.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.123+0000 7ff0f4cc9640 1 --2- 172.21.15.125:0/3822513584 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0dc003980 0x7ff0dc003da0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009ae0 tx=0x7ff0e001ca50 comp rx=0 tx=0).stop 2023-10-30T12:48:41.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.123+0000 7ff0eed9d640 1 -- 172.21.15.125:0/3822513584 reap_dead start 2023-10-30T12:48:41.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.123+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/3822513584 shutdown_connections 2023-10-30T12:48:41.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.123+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/3822513584 >> 172.21.15.125:0/3822513584 conn(0x7ff0f006bfe0 msgr2=0x7ff0f006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:41.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.123+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/3822513584 shutdown_connections 2023-10-30T12:48:41.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.123+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/3822513584 wait complete. 2023-10-30T12:48:41.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0f4cc9640 1 Processor -- start 2023-10-30T12:48:41.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0f4cc9640 1 -- start start 2023-10-30T12:48:41.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0f4cc9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0dc003980 0x7ff0f0147dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:41.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0f4cc9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0f0156ec0 0x7ff0f01573d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:41.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0f4cc9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f00cebf0 0x7ff0f015cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:41.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0f4cc9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f01f01d0 con 0x7ff0f0156ec0 2023-10-30T12:48:41.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0f4cc9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00b3250 con 0x7ff0dc003980 2023-10-30T12:48:41.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0f4cc9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f00cd9d0 con 0x7ff0f00cebf0 2023-10-30T12:48:41.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0e5d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0f0156ec0 0x7ff0f01573d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:41.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0e5d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0f0156ec0 0x7ff0f01573d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50370/0 (socket says 172.21.15.125:50370) 2023-10-30T12:48:41.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0e5d9b640 1 -- 172.21.15.125:0/4126502939 learned_addr learned my addr 172.21.15.125:0/4126502939 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:41.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0ee59c640 1 --2- 172.21.15.125:0/4126502939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0dc003980 0x7ff0f0147dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:41.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0eed9d640 1 --2- 172.21.15.125:0/4126502939 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f00cebf0 0x7ff0f015cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:41.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0e5d9b640 1 -- 172.21.15.125:0/4126502939 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f00cebf0 msgr2=0x7ff0f015cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:41.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0e5d9b640 1 --2- 172.21.15.125:0/4126502939 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f00cebf0 0x7ff0f015cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:41.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0e5d9b640 1 -- 172.21.15.125:0/4126502939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0dc003980 msgr2=0x7ff0f0147dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:41.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0e5d9b640 1 --2- 172.21.15.125:0/4126502939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0dc003980 0x7ff0f0147dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:41.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0e5d9b640 1 -- 172.21.15.125:0/4126502939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f01f06c0 con 0x7ff0f0156ec0 2023-10-30T12:48:41.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0e5d9b640 1 --2- 172.21.15.125:0/4126502939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0f0156ec0 0x7ff0f01573d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8011950 tx=0x7ff0d8011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:41.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0e77fe640 1 -- 172.21.15.125:0/4126502939 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d800fb60 con 0x7ff0f0156ec0 2023-10-30T12:48:41.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0e77fe640 1 -- 172.21.15.125:0/4126502939 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d800fce0 con 0x7ff0f0156ec0 2023-10-30T12:48:41.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/4126502939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f00cd0e0 con 0x7ff0f0156ec0 2023-10-30T12:48:41.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0e77fe640 1 -- 172.21.15.125:0/4126502939 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8015650 con 0x7ff0f0156ec0 2023-10-30T12:48:41.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.127+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/4126502939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f00cd380 con 0x7ff0f0156ec0 2023-10-30T12:48:41.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.131+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/4126502939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f00cdb80 con 0x7ff0f0156ec0 2023-10-30T12:48:41.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.131+0000 7ff0e77fe640 1 -- 172.21.15.125:0/4126502939 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff0d80157b0 con 0x7ff0f0156ec0 2023-10-30T12:48:41.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.131+0000 7ff0e77fe640 1 --2- 172.21.15.125:0/4126502939 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0d00664d0 0x7ff0d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:41.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.131+0000 7ff0e77fe640 1 -- 172.21.15.125:0/4126502939 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7ff0d8087480 con 0x7ff0f0156ec0 2023-10-30T12:48:41.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.131+0000 7ff0ee59c640 1 --2- 172.21.15.125:0/4126502939 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0d00664d0 0x7ff0d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:41.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.131+0000 7ff0ee59c640 1 --2- 172.21.15.125:0/4126502939 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0d00664d0 0x7ff0d0068990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff0e00af000 tx=0x7ff0e00046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:41.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.135+0000 7ff0e77fe640 1 -- 172.21.15.125:0/4126502939 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff0f00cdb80 con 0x7ff0f0156ec0 2023-10-30T12:48:41.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.315+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/4126502939 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0f00cdb40 con 0x7ff0f0156ec0 2023-10-30T12:48:41.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.315+0000 7ff0e77fe640 1 -- 172.21.15.125:0/4126502939 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5743 (secure 0 0 0) 0x7ff0d80580b0 con 0x7ff0f0156ec0 2023-10-30T12:48:41.315 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:41.315 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:38.414758+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:41.316 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 102 2023-10-30T12:48:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.319+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/4126502939 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0d00664d0 msgr2=0x7ff0d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.319+0000 7ff0f4cc9640 1 --2- 172.21.15.125:0/4126502939 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0d00664d0 0x7ff0d0068990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff0e00af000 tx=0x7ff0e00046d0 comp rx=0 tx=0).stop 2023-10-30T12:48:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.319+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/4126502939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0f0156ec0 msgr2=0x7ff0f01573d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.319+0000 7ff0f4cc9640 1 --2- 172.21.15.125:0/4126502939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0f0156ec0 0x7ff0f01573d0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8011950 tx=0x7ff0d8011e10 comp rx=0 tx=0).stop 2023-10-30T12:48:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.319+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/4126502939 shutdown_connections 2023-10-30T12:48:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.319+0000 7ff0f4cc9640 1 --2- 172.21.15.125:0/4126502939 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0d00664d0 0x7ff0d0068990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.319+0000 7ff0f4cc9640 1 --2- 172.21.15.125:0/4126502939 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0f00cebf0 0x7ff0f015cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.319+0000 7ff0f4cc9640 1 --2- 172.21.15.125:0/4126502939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0f0156ec0 0x7ff0f01573d0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:41.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.319+0000 7ff0f4cc9640 1 --2- 172.21.15.125:0/4126502939 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0dc003980 0x7ff0f0147dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:41.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.319+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/4126502939 >> 172.21.15.125:0/4126502939 conn(0x7ff0f006bfe0 msgr2=0x7ff0f0152660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:41.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.319+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/4126502939 shutdown_connections 2023-10-30T12:48:41.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:41.319+0000 7ff0f4cc9640 1 -- 172.21.15.125:0/4126502939 wait complete. 2023-10-30T12:48:41.336 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:43.337 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:43.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.487+0000 7f346f1c5640 1 Processor -- start 2023-10-30T12:48:43.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.487+0000 7f346f1c5640 1 -- start start 2023-10-30T12:48:43.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.487+0000 7f346f1c5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f34681580e0 0x7f3468070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:43.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.487+0000 7f346f1c5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f346815e700 0x7f346806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:43.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.487+0000 7f346f1c5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34680c5810 con 0x7f34680c6540 2023-10-30T12:48:43.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.487+0000 7f346f1c5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34680c6990 con 0x7f346815e700 2023-10-30T12:48:43.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.487+0000 7f346f1c5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f346815eb90 con 0x7f34681580e0 2023-10-30T12:48:43.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.487+0000 7f346d762640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f346815e700 0x7f346806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:43.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.487+0000 7f346d762640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f346815e700 0x7f346806a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50374/0 (socket says 172.21.15.125:50374) 2023-10-30T12:48:43.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.491+0000 7f346d762640 1 -- 172.21.15.125:0/2774534974 learned_addr learned my addr 172.21.15.125:0/2774534974 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:43.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.491+0000 7f346cf61640 1 --2- 172.21.15.125:0/2774534974 >> v2:172.21.15.136:3300/0 conn(0x7f34681580e0 0x7f3468070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:43.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.491+0000 7f346d762640 1 -- 172.21.15.125:0/2774534974 >> v2:172.21.15.136:3300/0 conn(0x7f34681580e0 msgr2=0x7f3468070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:43.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.491+0000 7f346d762640 1 --2- 172.21.15.125:0/2774534974 >> v2:172.21.15.136:3300/0 conn(0x7f34681580e0 0x7f3468070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.491+0000 7f346d762640 1 -- 172.21.15.125:0/2774534974 >> v1:172.21.15.136:6789/0 conn(0x7f34680c6540 legacy=0x7f3468069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:43.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.491+0000 7f346d762640 1 -- 172.21.15.125:0/2774534974 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34680c6bb0 con 0x7f346815e700 2023-10-30T12:48:43.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.491+0000 7f346d762640 1 --2- 172.21.15.125:0/2774534974 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f346815e700 0x7f346806a5a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f346806eea0 tx=0x7f346001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=198bcf966da460dc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:43.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.491+0000 7f34677fe640 1 -- 172.21.15.125:0/2774534974 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34600a5860 con 0x7f346815e700 2023-10-30T12:48:43.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.491+0000 7f34677fe640 1 -- 172.21.15.125:0/2774534974 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34600a59c0 con 0x7f346815e700 2023-10-30T12:48:43.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.491+0000 7f34677fe640 1 -- 172.21.15.125:0/2774534974 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34600a5d50 con 0x7f346815e700 2023-10-30T12:48:43.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 -- 172.21.15.125:0/2774534974 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f346815e700 msgr2=0x7f346806a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:43.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 --2- 172.21.15.125:0/2774534974 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f346815e700 0x7f346806a5a0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f346806eea0 tx=0x7f346001c9a0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 -- 172.21.15.125:0/2774534974 shutdown_connections 2023-10-30T12:48:43.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 --2- 172.21.15.125:0/2774534974 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f346815e700 0x7f346806a5a0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 --2- 172.21.15.125:0/2774534974 >> v2:172.21.15.136:3300/0 conn(0x7f34681580e0 0x7f3468070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 -- 172.21.15.125:0/2774534974 >> 172.21.15.125:0/2774534974 conn(0x7f34680c7910 msgr2=0x7f34680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:43.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 -- 172.21.15.125:0/2774534974 shutdown_connections 2023-10-30T12:48:43.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 -- 172.21.15.125:0/2774534974 wait complete. 2023-10-30T12:48:43.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 Processor -- start 2023-10-30T12:48:43.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 -- start start 2023-10-30T12:48:43.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34680c6540 0x7f34681487f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:43.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34681580e0 0x7f34681492c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:43.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f346815e700 0x7f3468149800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:43.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346813cb40 con 0x7f34681580e0 2023-10-30T12:48:43.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346813db70 con 0x7f34680c6540 2023-10-30T12:48:43.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346f1c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f346813de50 con 0x7f346815e700 2023-10-30T12:48:43.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f3467fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34681580e0 0x7f34681492c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:43.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f3467fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34681580e0 0x7f34681492c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50376/0 (socket says 172.21.15.125:50376) 2023-10-30T12:48:43.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f3467fff640 1 -- 172.21.15.125:0/4039358737 learned_addr learned my addr 172.21.15.125:0/4039358737 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:43.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346cf61640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34680c6540 0x7f34681487f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:43.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346d762640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f346815e700 0x7f3468149800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:43.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f3467fff640 1 -- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f346815e700 msgr2=0x7f3468149800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:43.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f3467fff640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f346815e700 0x7f3468149800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f3467fff640 1 -- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34680c6540 msgr2=0x7f34681487f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:43.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f3467fff640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34680c6540 0x7f34681487f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f3467fff640 1 -- 172.21.15.125:0/4039358737 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3460009b30 con 0x7f34681580e0 2023-10-30T12:48:43.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.495+0000 7f346cf61640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34680c6540 0x7f34681487f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:43.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.499+0000 7f3467fff640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34681580e0 0x7f34681492c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f345400c500 tx=0x7f345400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:43.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.499+0000 7f34657fa640 1 -- 172.21.15.125:0/4039358737 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454005220 con 0x7f34681580e0 2023-10-30T12:48:43.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.499+0000 7f34657fa640 1 -- 172.21.15.125:0/4039358737 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3454005380 con 0x7f34681580e0 2023-10-30T12:48:43.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.499+0000 7f34657fa640 1 -- 172.21.15.125:0/4039358737 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3454007270 con 0x7f34681580e0 2023-10-30T12:48:43.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.499+0000 7f346f1c5640 1 -- 172.21.15.125:0/4039358737 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f346813e130 con 0x7f34681580e0 2023-10-30T12:48:43.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.499+0000 7f346f1c5640 1 -- 172.21.15.125:0/4039358737 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f346813e6f0 con 0x7f34681580e0 2023-10-30T12:48:43.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.499+0000 7f346f1c5640 1 -- 172.21.15.125:0/4039358737 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f346804e170 con 0x7f34681580e0 2023-10-30T12:48:43.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.499+0000 7f34657fa640 1 -- 172.21.15.125:0/4039358737 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34540073d0 con 0x7f34681580e0 2023-10-30T12:48:43.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.499+0000 7f34657fa640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f344c0664d0 0x7f344c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:43.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.499+0000 7f34657fa640 1 -- 172.21.15.125:0/4039358737 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f3454083bc0 con 0x7f34681580e0 2023-10-30T12:48:43.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.499+0000 7f346cf61640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f344c0664d0 0x7f344c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:43.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.503+0000 7f346cf61640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f344c0664d0 0x7f344c068990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f345c007a70 tx=0x7f345c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:43.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.503+0000 7f34657fa640 1 -- 172.21.15.125:0/4039358737 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3454050c30 con 0x7f34681580e0 2023-10-30T12:48:43.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.687+0000 7f346f1c5640 1 -- 172.21.15.125:0/4039358737 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3468148140 con 0x7f34681580e0 2023-10-30T12:48:43.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.687+0000 7f34657fa640 1 -- 172.21.15.125:0/4039358737 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+5743 (secure 0 0 0) 0x7f3454054250 con 0x7f34681580e0 2023-10-30T12:48:43.687 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:43.687 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:38.414758+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:reconnect","state_seq":12,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 102 2023-10-30T12:48:43.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.687+0000 7f346f1c5640 1 -- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f344c0664d0 msgr2=0x7f344c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:43.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.687+0000 7f346f1c5640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f344c0664d0 0x7f344c068990 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f345c007a70 tx=0x7f345c008040 comp rx=0 tx=0).stop 2023-10-30T12:48:43.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.687+0000 7f346f1c5640 1 -- 172.21.15.125:0/4039358737 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34681580e0 msgr2=0x7f34681492c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:43.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.687+0000 7f346f1c5640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34681580e0 0x7f34681492c0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f345400c500 tx=0x7f345400c9c0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.687+0000 7f346f1c5640 1 -- 172.21.15.125:0/4039358737 shutdown_connections 2023-10-30T12:48:43.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.687+0000 7f346f1c5640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f344c0664d0 0x7f344c068990 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.687+0000 7f346f1c5640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f346815e700 0x7f3468149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.687+0000 7f346f1c5640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f34681580e0 0x7f34681492c0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.687+0000 7f346f1c5640 1 --2- 172.21.15.125:0/4039358737 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f34680c6540 0x7f34681487f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:43.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.687+0000 7f346f1c5640 1 -- 172.21.15.125:0/4039358737 >> 172.21.15.125:0/4039358737 conn(0x7f34680c7910 msgr2=0x7f346814a7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:43.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.687+0000 7f346f1c5640 1 -- 172.21.15.125:0/4039358737 shutdown_connections 2023-10-30T12:48:43.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:43.687+0000 7f346f1c5640 1 -- 172.21.15.125:0/4039358737 wait complete. 2023-10-30T12:48:43.706 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:45.707 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:45.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff08d50f640 1 Processor -- start 2023-10-30T12:48:45.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff08d50f640 1 -- start start 2023-10-30T12:48:45.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff08d50f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff0881538a0 0x7ff08815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:45.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff08d50f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff0880cf160 0x7ff0880cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:45.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff08d50f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0880cb800 con 0x7ff0880ca0e0 2023-10-30T12:48:45.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff08d50f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff08815dc30 con 0x7ff0881538a0 2023-10-30T12:48:45.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff08d50f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0880cb330 con 0x7ff0880cf160 2023-10-30T12:48:45.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff086ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff0881538a0 0x7ff08815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:45.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff086ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff0881538a0 0x7ff08815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50380/0 (socket says 172.21.15.125:50380) 2023-10-30T12:48:45.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff086ffd640 1 -- 172.21.15.125:0/985237357 learned_addr learned my addr 172.21.15.125:0/985237357 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:45.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff0877fe640 1 --2- 172.21.15.125:0/985237357 >> v2:172.21.15.136:3300/0 conn(0x7ff0880cf160 0x7ff0880cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:45.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff086ffd640 1 -- 172.21.15.125:0/985237357 >> v2:172.21.15.136:3300/0 conn(0x7ff0880cf160 msgr2=0x7ff0880cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:45.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff086ffd640 1 --2- 172.21.15.125:0/985237357 >> v2:172.21.15.136:3300/0 conn(0x7ff0880cf160 0x7ff0880cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:45.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff086ffd640 1 -- 172.21.15.125:0/985237357 >> v1:172.21.15.136:6789/0 conn(0x7ff0880ca0e0 legacy=0x7ff08815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:45.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.855+0000 7ff086ffd640 1 -- 172.21.15.125:0/985237357 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0880cad70 con 0x7ff0881538a0 2023-10-30T12:48:45.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.859+0000 7ff086ffd640 1 --2- 172.21.15.125:0/985237357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0881538a0 0x7ff08815d6f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7ff0880b5330 tx=0x7ff07001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df9e7eaebece9d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:45.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.859+0000 7ff0867fc640 1 -- 172.21.15.125:0/985237357 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0700a89b0 con 0x7ff0881538a0 2023-10-30T12:48:45.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.859+0000 7ff0867fc640 1 -- 172.21.15.125:0/985237357 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0700a8b10 con 0x7ff0881538a0 2023-10-30T12:48:45.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.859+0000 7ff0867fc640 1 -- 172.21.15.125:0/985237357 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0700a8ea0 con 0x7ff0881538a0 2023-10-30T12:48:45.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.859+0000 7ff08d50f640 1 -- 172.21.15.125:0/985237357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0881538a0 msgr2=0x7ff08815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:45.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.859+0000 7ff08d50f640 1 --2- 172.21.15.125:0/985237357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0881538a0 0x7ff08815d6f0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7ff0880b5330 tx=0x7ff07001ecb0 comp rx=0 tx=0).stop 2023-10-30T12:48:45.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.859+0000 7ff08d50f640 1 -- 172.21.15.125:0/985237357 shutdown_connections 2023-10-30T12:48:45.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.859+0000 7ff08d50f640 1 --2- 172.21.15.125:0/985237357 >> v2:172.21.15.136:3300/0 conn(0x7ff0880cf160 0x7ff0880cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:45.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.859+0000 7ff08d50f640 1 --2- 172.21.15.125:0/985237357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0881538a0 0x7ff08815d6f0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:45.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.859+0000 7ff08d50f640 1 -- 172.21.15.125:0/985237357 >> 172.21.15.125:0/985237357 conn(0x7ff08806bfe0 msgr2=0x7ff08806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:45.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.859+0000 7ff08d50f640 1 -- 172.21.15.125:0/985237357 shutdown_connections 2023-10-30T12:48:45.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.859+0000 7ff08d50f640 1 -- 172.21.15.125:0/985237357 wait complete. 2023-10-30T12:48:45.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff08d50f640 1 Processor -- start 2023-10-30T12:48:45.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff08d50f640 1 -- start start 2023-10-30T12:48:45.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff08d50f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0880ca0e0 0x7ff0880c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:45.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff08d50f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0880cf160 0x7ff088157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:45.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff08d50f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0881538a0 0x7ff0881e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:45.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff08d50f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0881ef4c0 con 0x7ff0880cf160 2023-10-30T12:48:45.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff08d50f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0880c9180 con 0x7ff0881538a0 2023-10-30T12:48:45.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff08d50f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0881ef730 con 0x7ff0880ca0e0 2023-10-30T12:48:45.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff07e7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0880cf160 0x7ff088157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:45.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff07e7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0880cf160 0x7ff088157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50382/0 (socket says 172.21.15.125:50382) 2023-10-30T12:48:45.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff07e7fc640 1 -- 172.21.15.125:0/2808376061 learned_addr learned my addr 172.21.15.125:0/2808376061 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:45.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff0877fe640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0881538a0 0x7ff0881e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:45.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff086ffd640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0880ca0e0 0x7ff0880c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:45.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff07e7fc640 1 -- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0880ca0e0 msgr2=0x7ff0880c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:45.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff07e7fc640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0880ca0e0 0x7ff0880c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:45.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff07e7fc640 1 -- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0881538a0 msgr2=0x7ff0881e5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:45.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff07e7fc640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0881538a0 0x7ff0881e5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:45.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff07e7fc640 1 -- 172.21.15.125:0/2808376061 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0700094d0 con 0x7ff0880cf160 2023-10-30T12:48:45.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff086ffd640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0880ca0e0 0x7ff0880c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:45.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff0877fe640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0881538a0 0x7ff0881e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:45.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff07e7fc640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0880cf160 0x7ff088157560 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ff07800b410 tx=0x7ff07800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:45.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff07ffff640 1 -- 172.21.15.125:0/2808376061 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078004020 con 0x7ff0880cf160 2023-10-30T12:48:45.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff07ffff640 1 -- 172.21.15.125:0/2808376061 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff078004180 con 0x7ff0880cf160 2023-10-30T12:48:45.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff08d50f640 1 -- 172.21.15.125:0/2808376061 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0881efa10 con 0x7ff0880cf160 2023-10-30T12:48:45.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff07ffff640 1 -- 172.21.15.125:0/2808376061 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff078011630 con 0x7ff0880cf160 2023-10-30T12:48:45.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.863+0000 7ff08d50f640 1 -- 172.21.15.125:0/2808376061 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0880c9460 con 0x7ff0880cf160 2023-10-30T12:48:45.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.867+0000 7ff08d50f640 1 -- 172.21.15.125:0/2808376061 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0880b2a50 con 0x7ff0880cf160 2023-10-30T12:48:45.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.867+0000 7ff07ffff640 1 -- 172.21.15.125:0/2808376061 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff078011790 con 0x7ff0880cf160 2023-10-30T12:48:45.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.867+0000 7ff07ffff640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0680664a0 0x7ff068068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:45.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.867+0000 7ff07ffff640 1 -- 172.21.15.125:0/2808376061 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7ff0780840d0 con 0x7ff0880cf160 2023-10-30T12:48:45.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.867+0000 7ff086ffd640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0680664a0 0x7ff068068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:45.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.867+0000 7ff086ffd640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0680664a0 0x7ff068068960 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff0700b2000 tx=0x7ff070008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:45.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:45.871+0000 7ff07ffff640 1 -- 172.21.15.125:0/2808376061 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff0780510a0 con 0x7ff0880cf160 2023-10-30T12:48:46.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:46.055+0000 7ff08d50f640 1 -- 172.21.15.125:0/2808376061 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0881eff30 con 0x7ff0880cf160 2023-10-30T12:48:46.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:46.055+0000 7ff07ffff640 1 -- 172.21.15.125:0/2808376061 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5740 (secure 0 0 0) 0x7ff0881eff30 con 0x7ff0880cf160 2023-10-30T12:48:46.053 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:46.053 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:44.405738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:46.054 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 103 2023-10-30T12:48:46.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:46.055+0000 7ff08d50f640 1 -- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0680664a0 msgr2=0x7ff068068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:46.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:46.055+0000 7ff08d50f640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0680664a0 0x7ff068068960 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff0700b2000 tx=0x7ff070008040 comp rx=0 tx=0).stop 2023-10-30T12:48:46.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:46.055+0000 7ff08d50f640 1 -- 172.21.15.125:0/2808376061 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0880cf160 msgr2=0x7ff088157560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:46.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:46.055+0000 7ff08d50f640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0880cf160 0x7ff088157560 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7ff07800b410 tx=0x7ff07800b8d0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:46.055+0000 7ff08d50f640 1 -- 172.21.15.125:0/2808376061 shutdown_connections 2023-10-30T12:48:46.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:46.055+0000 7ff08d50f640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0680664a0 0x7ff068068960 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:46.055+0000 7ff08d50f640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0881538a0 0x7ff0881e5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:46.055+0000 7ff08d50f640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0880cf160 0x7ff088157560 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:46.055+0000 7ff08d50f640 1 --2- 172.21.15.125:0/2808376061 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0880ca0e0 0x7ff0880c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:46.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:46.055+0000 7ff08d50f640 1 -- 172.21.15.125:0/2808376061 >> 172.21.15.125:0/2808376061 conn(0x7ff08806bfe0 msgr2=0x7ff0881519e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:46.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:46.055+0000 7ff08d50f640 1 -- 172.21.15.125:0/2808376061 shutdown_connections 2023-10-30T12:48:46.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:46.055+0000 7ff08d50f640 1 -- 172.21.15.125:0/2808376061 wait complete. 2023-10-30T12:48:46.074 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:47.065 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:48:47.069 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:48:48.075 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:48.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.223+0000 7fd0d6417640 1 Processor -- start 2023-10-30T12:48:48.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.223+0000 7fd0d6417640 1 -- start start 2023-10-30T12:48:48.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d6417640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0d00c6b80 0x7fd0d00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:48.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d6417640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd0d00c6f60 0x7fd0d00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:48.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d6417640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd0d00c8430 0x7fd0d013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:48.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d6417640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d00c0a50 con 0x7fd0d00c8430 2023-10-30T12:48:48.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d6417640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d006f6e0 con 0x7fd0d00c6f60 2023-10-30T12:48:48.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d6417640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d006f9c0 con 0x7fd0d00c6b80 2023-10-30T12:48:48.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d49b4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd0d00c8430 0x7fd0d013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:48.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d49b4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd0d00c8430 0x7fd0d013ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50390/0 (socket says 172.21.15.125:50390) 2023-10-30T12:48:48.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d49b4640 1 -- 172.21.15.125:0/3948493141 learned_addr learned my addr 172.21.15.125:0/3948493141 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:48.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0cf7fe640 1 --2- 172.21.15.125:0/3948493141 >> v2:172.21.15.136:3300/0 conn(0x7fd0d00c6f60 0x7fd0d00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:48.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0cffff640 1 --2- 172.21.15.125:0/3948493141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0d00c6b80 0x7fd0d00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:48.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d49b4640 1 -- 172.21.15.125:0/3948493141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0d00c6b80 msgr2=0x7fd0d00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:48.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d49b4640 1 --2- 172.21.15.125:0/3948493141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0d00c6b80 0x7fd0d00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:48.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d49b4640 1 -- 172.21.15.125:0/3948493141 >> v2:172.21.15.136:3300/0 conn(0x7fd0d00c6f60 msgr2=0x7fd0d00c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:48.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d49b4640 1 --2- 172.21.15.125:0/3948493141 >> v2:172.21.15.136:3300/0 conn(0x7fd0d00c6f60 0x7fd0d00c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:48.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d49b4640 1 -- 172.21.15.125:0/3948493141 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d006fca0 con 0x7fd0d00c8430 2023-10-30T12:48:48.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0cf7fe640 1 --2- 172.21.15.125:0/3948493141 >> v2:172.21.15.136:3300/0 conn(0x7fd0d00c6f60 0x7fd0d00c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:48.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0cffff640 1 --2- 172.21.15.125:0/3948493141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0d00c6b80 0x7fd0d00bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:48.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.227+0000 7fd0d49b4640 1 --2- 172.21.15.125:0/3948493141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d00c8430 0x7fd0d013ed90 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd0d006c830 tx=0x7fd0c801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2652ce5df38a6bc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:48.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0ceffd640 1 -- 172.21.15.125:0/3948493141 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c80a5760 con 0x7fd0d00c8430 2023-10-30T12:48:48.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0ceffd640 1 -- 172.21.15.125:0/3948493141 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c80a58c0 con 0x7fd0d00c8430 2023-10-30T12:48:48.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0ceffd640 1 -- 172.21.15.125:0/3948493141 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c80a5c50 con 0x7fd0d00c8430 2023-10-30T12:48:48.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 -- 172.21.15.125:0/3948493141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d00c8430 msgr2=0x7fd0d013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:48.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 --2- 172.21.15.125:0/3948493141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d00c8430 0x7fd0d013ed90 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd0d006c830 tx=0x7fd0c801c9d0 comp rx=0 tx=0).stop 2023-10-30T12:48:48.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 -- 172.21.15.125:0/3948493141 shutdown_connections 2023-10-30T12:48:48.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 --2- 172.21.15.125:0/3948493141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d00c8430 0x7fd0d013ed90 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:48.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 --2- 172.21.15.125:0/3948493141 >> v2:172.21.15.136:3300/0 conn(0x7fd0d00c6f60 0x7fd0d00c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:48.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 --2- 172.21.15.125:0/3948493141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0d00c6b80 0x7fd0d00bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:48.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 -- 172.21.15.125:0/3948493141 >> 172.21.15.125:0/3948493141 conn(0x7fd0d00c7910 msgr2=0x7fd0d00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:48.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 -- 172.21.15.125:0/3948493141 shutdown_connections 2023-10-30T12:48:48.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 -- 172.21.15.125:0/3948493141 wait complete. 2023-10-30T12:48:48.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 Processor -- start 2023-10-30T12:48:48.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 -- start start 2023-10-30T12:48:48.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0d00c6b80 0x7fd0d01363f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:48.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0d00c6f60 0x7fd0d01502f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:48.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d00c8430 0x7fd0d0150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:48.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d0133160 con 0x7fd0d00c8430 2023-10-30T12:48:48.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d0132be0 con 0x7fd0d00c6f60 2023-10-30T12:48:48.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d6417640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d00c0730 con 0x7fd0d00c6b80 2023-10-30T12:48:48.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d49b4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d00c8430 0x7fd0d0150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:48.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d49b4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d00c8430 0x7fd0d0150830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50404/0 (socket says 172.21.15.125:50404) 2023-10-30T12:48:48.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.231+0000 7fd0d49b4640 1 -- 172.21.15.125:0/3607687052 learned_addr learned my addr 172.21.15.125:0/3607687052 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:48.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0cffff640 1 --2- 172.21.15.125:0/3607687052 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0d00c6b80 0x7fd0d01363f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:48.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0cf7fe640 1 --2- 172.21.15.125:0/3607687052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0d00c6f60 0x7fd0d01502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:48.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0d49b4640 1 -- 172.21.15.125:0/3607687052 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0d00c6b80 msgr2=0x7fd0d01363f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0d49b4640 1 --2- 172.21.15.125:0/3607687052 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0d00c6b80 0x7fd0d01363f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0d49b4640 1 -- 172.21.15.125:0/3607687052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0d00c6f60 msgr2=0x7fd0d01502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0d49b4640 1 --2- 172.21.15.125:0/3607687052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0d00c6f60 0x7fd0d01502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0d49b4640 1 -- 172.21.15.125:0/3607687052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c8009b30 con 0x7fd0d00c8430 2023-10-30T12:48:48.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0d49b4640 1 --2- 172.21.15.125:0/3607687052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d00c8430 0x7fd0d0150830 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fd0c80af040 tx=0x7fd0c8019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:48.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0ccff9640 1 -- 172.21.15.125:0/3607687052 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c80b8880 con 0x7fd0d00c8430 2023-10-30T12:48:48.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0ccff9640 1 -- 172.21.15.125:0/3607687052 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c80b8a00 con 0x7fd0d00c8430 2023-10-30T12:48:48.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0d6417640 1 -- 172.21.15.125:0/3607687052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d01f0910 con 0x7fd0d00c8430 2023-10-30T12:48:48.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0ccff9640 1 -- 172.21.15.125:0/3607687052 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c80b8db0 con 0x7fd0d00c8430 2023-10-30T12:48:48.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0d6417640 1 -- 172.21.15.125:0/3607687052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d014fdd0 con 0x7fd0d00c8430 2023-10-30T12:48:48.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0d6417640 1 -- 172.21.15.125:0/3607687052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d014fc30 con 0x7fd0d00c8430 2023-10-30T12:48:48.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0ccff9640 1 -- 172.21.15.125:0/3607687052 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0c80c3460 con 0x7fd0d00c8430 2023-10-30T12:48:48.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0ccff9640 1 --2- 172.21.15.125:0/3607687052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0bc0664a0 0x7fd0bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:48.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.235+0000 7fd0ccff9640 1 -- 172.21.15.125:0/3607687052 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fd0c811faf0 con 0x7fd0d00c8430 2023-10-30T12:48:48.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.239+0000 7fd0cffff640 1 --2- 172.21.15.125:0/3607687052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0bc0664a0 0x7fd0bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:48.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.239+0000 7fd0cffff640 1 --2- 172.21.15.125:0/3607687052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0bc0664a0 0x7fd0bc068960 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000c880 tx=0x7fd0c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:48.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.239+0000 7fd0ccff9640 1 -- 172.21.15.125:0/3607687052 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd0d014fc30 con 0x7fd0d00c8430 2023-10-30T12:48:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.419+0000 7fd0d6417640 1 -- 172.21.15.125:0/3607687052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0d014fc30 con 0x7fd0d00c8430 2023-10-30T12:48:48.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.423+0000 7fd0ccff9640 1 -- 172.21.15.125:0/3607687052 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+5740 (secure 0 0 0) 0x7fd0d014fc30 con 0x7fd0d00c8430 2023-10-30T12:48:48.420 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:48.420 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:44.405738+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:48.421 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 103 2023-10-30T12:48:48.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.423+0000 7fd0d6417640 1 -- 172.21.15.125:0/3607687052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0bc0664a0 msgr2=0x7fd0bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:48.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.423+0000 7fd0d6417640 1 --2- 172.21.15.125:0/3607687052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0bc0664a0 0x7fd0bc068960 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd0c000c880 tx=0x7fd0c0008040 comp rx=0 tx=0).stop 2023-10-30T12:48:48.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.423+0000 7fd0d6417640 1 -- 172.21.15.125:0/3607687052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d00c8430 msgr2=0x7fd0d0150830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:48.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.423+0000 7fd0d6417640 1 --2- 172.21.15.125:0/3607687052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d00c8430 0x7fd0d0150830 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fd0c80af040 tx=0x7fd0c8019770 comp rx=0 tx=0).stop 2023-10-30T12:48:48.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.423+0000 7fd0d6417640 1 -- 172.21.15.125:0/3607687052 shutdown_connections 2023-10-30T12:48:48.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.423+0000 7fd0d6417640 1 --2- 172.21.15.125:0/3607687052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0bc0664a0 0x7fd0bc068960 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:48.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.423+0000 7fd0d6417640 1 --2- 172.21.15.125:0/3607687052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0d00c8430 0x7fd0d0150830 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:48.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.423+0000 7fd0d6417640 1 --2- 172.21.15.125:0/3607687052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0d00c6f60 0x7fd0d01502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:48.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.423+0000 7fd0d6417640 1 --2- 172.21.15.125:0/3607687052 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0d00c6b80 0x7fd0d01363f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:48.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.423+0000 7fd0d6417640 1 -- 172.21.15.125:0/3607687052 >> 172.21.15.125:0/3607687052 conn(0x7fd0d00c7910 msgr2=0x7fd0d006bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:48.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.423+0000 7fd0d6417640 1 -- 172.21.15.125:0/3607687052 shutdown_connections 2023-10-30T12:48:48.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:48.423+0000 7fd0d6417640 1 -- 172.21.15.125:0/3607687052 wait complete. 2023-10-30T12:48:48.441 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:48:50.442 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:50.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f3187c46640 1 Processor -- start 2023-10-30T12:48:50.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f3187c46640 1 -- start start 2023-10-30T12:48:50.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f3187c46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f318014fd50 0x7f31800c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:50.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f3187c46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31800cb550 0x7f31800cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:50.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f3187c46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31801560a0 con 0x7f3180071250 2023-10-30T12:48:50.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f3187c46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3180156380 con 0x7f31800cb550 2023-10-30T12:48:50.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f3187c46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3180156660 con 0x7f318014fd50 2023-10-30T12:48:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f31851e1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31800cb550 0x7f31800cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f31851e1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31800cb550 0x7f31800cb9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50410/0 (socket says 172.21.15.125:50410) 2023-10-30T12:48:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f31851e1640 1 -- 172.21.15.125:0/1822004646 learned_addr learned my addr 172.21.15.125:0/1822004646 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f31859e2640 1 --2- 172.21.15.125:0/1822004646 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f318014fd50 0x7f31800c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:50.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f31851e1640 1 -- 172.21.15.125:0/1822004646 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f318014fd50 msgr2=0x7f31800c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:50.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f31851e1640 1 --2- 172.21.15.125:0/1822004646 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f318014fd50 0x7f31800c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:50.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f31851e1640 1 -- 172.21.15.125:0/1822004646 >> v1:172.21.15.136:6789/0 conn(0x7f3180071250 legacy=0x7f31800716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:50.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.587+0000 7f31851e1640 1 -- 172.21.15.125:0/1822004646 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3180156940 con 0x7f31800cb550 2023-10-30T12:48:50.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.591+0000 7f31851e1640 1 --2- 172.21.15.125:0/1822004646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31800cb550 0x7f31800cb9b0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3170009870 tx=0x7f317001d700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758df6e8a0641988 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:50.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.591+0000 7f31849e0640 1 -- 172.21.15.125:0/1822004646 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31700a76e0 con 0x7f31800cb550 2023-10-30T12:48:50.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.591+0000 7f31849e0640 1 -- 172.21.15.125:0/1822004646 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31700a7840 con 0x7f31800cb550 2023-10-30T12:48:50.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.591+0000 7f31849e0640 1 -- 172.21.15.125:0/1822004646 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31700a7bd0 con 0x7f31800cb550 2023-10-30T12:48:50.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.591+0000 7f3187c46640 1 -- 172.21.15.125:0/1822004646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31800cb550 msgr2=0x7f31800cb9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:50.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.591+0000 7f3187c46640 1 --2- 172.21.15.125:0/1822004646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31800cb550 0x7f31800cb9b0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f3170009870 tx=0x7f317001d700 comp rx=0 tx=0).stop 2023-10-30T12:48:50.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.591+0000 7f3187c46640 1 -- 172.21.15.125:0/1822004646 shutdown_connections 2023-10-30T12:48:50.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.591+0000 7f3187c46640 1 --2- 172.21.15.125:0/1822004646 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31800cb550 0x7f31800cb9b0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:50.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.591+0000 7f3187c46640 1 --2- 172.21.15.125:0/1822004646 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f318014fd50 0x7f31800c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:50.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.591+0000 7f3187c46640 1 -- 172.21.15.125:0/1822004646 >> 172.21.15.125:0/1822004646 conn(0x7f318006c6e0 msgr2=0x7f3180069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:50.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.591+0000 7f3187c46640 1 -- 172.21.15.125:0/1822004646 shutdown_connections 2023-10-30T12:48:50.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.591+0000 7f3187c46640 1 -- 172.21.15.125:0/1822004646 wait complete. 2023-10-30T12:48:50.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f3187c46640 1 Processor -- start 2023-10-30T12:48:50.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f3187c46640 1 -- start start 2023-10-30T12:48:50.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f3187c46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3180071250 0x7f31800c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:50.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f3187c46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31800cb550 0x7f31800c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:50.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f3187c46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f318014fd50 0x7f3180144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:50.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f3187c46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31800c7b00 con 0x7f31800cb550 2023-10-30T12:48:50.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f3187c46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31800c7480 con 0x7f318014fd50 2023-10-30T12:48:50.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f3187c46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31801f5b10 con 0x7f3180071250 2023-10-30T12:48:50.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31851e1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31800cb550 0x7f31800c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:50.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31851e1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31800cb550 0x7f31800c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50426/0 (socket says 172.21.15.125:50426) 2023-10-30T12:48:50.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31851e1640 1 -- 172.21.15.125:0/2202511281 learned_addr learned my addr 172.21.15.125:0/2202511281 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:50.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31861e3640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f318014fd50 0x7f3180144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:50.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31859e2640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3180071250 0x7f31800c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:50.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31851e1640 1 -- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3180071250 msgr2=0x7f31800c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:50.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31851e1640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3180071250 0x7f31800c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:50.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31851e1640 1 -- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f318014fd50 msgr2=0x7f3180144fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:50.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31851e1640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f318014fd50 0x7f3180144fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:50.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31851e1640 1 -- 172.21.15.125:0/2202511281 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31700094d0 con 0x7f31800cb550 2023-10-30T12:48:50.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31861e3640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f318014fd50 0x7f3180144fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:48:50.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31851e1640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31800cb550 0x7f31800c4cd0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f31700b1040 tx=0x7f317001a4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:50.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31767fc640 1 -- 172.21.15.125:0/2202511281 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31700af830 con 0x7f31800cb550 2023-10-30T12:48:50.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31767fc640 1 -- 172.21.15.125:0/2202511281 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31700af9b0 con 0x7f31800cb550 2023-10-30T12:48:50.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f3187c46640 1 -- 172.21.15.125:0/2202511281 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31800c7720 con 0x7f31800cb550 2023-10-30T12:48:50.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.595+0000 7f31767fc640 1 -- 172.21.15.125:0/2202511281 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31700afd60 con 0x7f31800cb550 2023-10-30T12:48:50.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.599+0000 7f3187c46640 1 -- 172.21.15.125:0/2202511281 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3180153420 con 0x7f31800cb550 2023-10-30T12:48:50.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.599+0000 7f3187c46640 1 -- 172.21.15.125:0/2202511281 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31800c78b0 con 0x7f31800cb550 2023-10-30T12:48:50.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.599+0000 7f31767fc640 1 -- 172.21.15.125:0/2202511281 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f31700b9410 con 0x7f31800cb550 2023-10-30T12:48:50.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.599+0000 7f31767fc640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31640664a0 0x7f3164068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:50.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.599+0000 7f31767fc640 1 -- 172.21.15.125:0/2202511281 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f31700ab070 con 0x7f31800cb550 2023-10-30T12:48:50.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.599+0000 7f31859e2640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31640664a0 0x7f3164068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:50.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.599+0000 7f31859e2640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31640664a0 0x7f3164068960 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f317c00b9d0 tx=0x7f317c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:50.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.603+0000 7f31767fc640 1 -- 172.21.15.125:0/2202511281 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f31700ebe00 con 0x7f31800cb550 2023-10-30T12:48:50.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.791+0000 7f3187c46640 1 -- 172.21.15.125:0/2202511281 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f318004e170 con 0x7f31800cb550 2023-10-30T12:48:50.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.795+0000 7f31767fc640 1 -- 172.21.15.125:0/2202511281 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5740 (secure 0 0 0) 0x7f31700be070 con 0x7f31800cb550 2023-10-30T12:48:50.792 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:50.792 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:49.544257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:50.793 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 104 2023-10-30T12:48:50.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.795+0000 7f3187c46640 1 -- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31640664a0 msgr2=0x7f3164068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:50.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.795+0000 7f3187c46640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31640664a0 0x7f3164068960 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f317c00b9d0 tx=0x7f317c008040 comp rx=0 tx=0).stop 2023-10-30T12:48:50.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.795+0000 7f3187c46640 1 -- 172.21.15.125:0/2202511281 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31800cb550 msgr2=0x7f31800c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:50.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.795+0000 7f3187c46640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31800cb550 0x7f31800c4cd0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f31700b1040 tx=0x7f317001a4a0 comp rx=0 tx=0).stop 2023-10-30T12:48:50.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.795+0000 7f3187c46640 1 -- 172.21.15.125:0/2202511281 shutdown_connections 2023-10-30T12:48:50.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.795+0000 7f3187c46640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31640664a0 0x7f3164068960 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:50.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.795+0000 7f3187c46640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f318014fd50 0x7f3180144fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:50.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.795+0000 7f3187c46640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31800cb550 0x7f31800c4cd0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:50.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.795+0000 7f3187c46640 1 --2- 172.21.15.125:0/2202511281 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3180071250 0x7f31800c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:50.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.795+0000 7f3187c46640 1 -- 172.21.15.125:0/2202511281 >> 172.21.15.125:0/2202511281 conn(0x7f318006c6e0 msgr2=0x7f3180073480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:50.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.795+0000 7f3187c46640 1 -- 172.21.15.125:0/2202511281 shutdown_connections 2023-10-30T12:48:50.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:50.795+0000 7f3187c46640 1 -- 172.21.15.125:0/2202511281 wait complete. 2023-10-30T12:48:50.815 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=1, replacing gid=5988 2023-10-30T12:48:50.815 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.c 2023-10-30T12:48:52.250 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:48:52.251+0000 7f2dd3c0c640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T12:48:54.816 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T12:48:54.816 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T12:48:54.817 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T12:48:54.820 INFO:tasks.ceph.mds.c:Started 2023-10-30T12:48:54.821 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:54.919 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2023-10-30T12:48:54.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f157b09c640 1 Processor -- start 2023-10-30T12:48:54.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f157b09c640 1 -- start start 2023-10-30T12:48:54.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f157b09c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15741538a0 0x7f157415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:54.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f157b09c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f15740ca0e0 0x7f157415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:54.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f157b09c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15740cb800 con 0x7f15740cf160 2023-10-30T12:48:54.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f157b09c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f157415dc30 con 0x7f15740ca0e0 2023-10-30T12:48:54.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f157b09c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15740cb330 con 0x7f15741538a0 2023-10-30T12:48:54.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f1573fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f15740ca0e0 0x7f157415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:54.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f1578e38640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15741538a0 0x7f157415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:54.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f1573fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f15740ca0e0 0x7f157415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:60670/0 (socket says 172.21.15.125:60670) 2023-10-30T12:48:54.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f1573fff640 1 -- 172.21.15.125:0/1018378564 learned_addr learned my addr 172.21.15.125:0/1018378564 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:54.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f1579639640 1 --1- 172.21.15.125:0/1018378564 >> v1:172.21.15.136:6789/0 conn(0x7f15740cf160 0x7f15740cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:46374/0 (socket says 172.21.15.125:46374) 2023-10-30T12:48:55.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f1578e38640 1 -- 172.21.15.125:0/1018378564 >> v2:172.21.15.136:3300/0 conn(0x7f15740ca0e0 msgr2=0x7f157415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:55.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f1578e38640 1 --2- 172.21.15.125:0/1018378564 >> v2:172.21.15.136:3300/0 conn(0x7f15740ca0e0 0x7f157415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:55.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f1578e38640 1 -- 172.21.15.125:0/1018378564 >> v1:172.21.15.136:6789/0 conn(0x7f15740cf160 legacy=0x7f15740cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:55.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f1578e38640 1 -- 172.21.15.125:0/1018378564 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15740cad70 con 0x7f15741538a0 2023-10-30T12:48:55.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:54.999+0000 7f1573fff640 1 --2- 172.21.15.125:0/1018378564 >> v2:172.21.15.136:3300/0 conn(0x7f15740ca0e0 0x7f157415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:48:55.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.003+0000 7f1578e38640 1 --2- 172.21.15.125:0/1018378564 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15741538a0 0x7f157415d6f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f15680097c0 tx=0x7f156801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4a7a8a3e3c51fd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:55.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.003+0000 7f15737fe640 1 -- 172.21.15.125:0/1018378564 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15680a8970 con 0x7f15741538a0 2023-10-30T12:48:55.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.003+0000 7f15737fe640 1 -- 172.21.15.125:0/1018378564 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15680a8ad0 con 0x7f15741538a0 2023-10-30T12:48:55.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.003+0000 7f15737fe640 1 -- 172.21.15.125:0/1018378564 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15680a8e60 con 0x7f15741538a0 2023-10-30T12:48:55.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.003+0000 7f157b09c640 1 -- 172.21.15.125:0/1018378564 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15741538a0 msgr2=0x7f157415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:55.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.003+0000 7f157b09c640 1 --2- 172.21.15.125:0/1018378564 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15741538a0 0x7f157415d6f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f15680097c0 tx=0x7f156801ec70 comp rx=0 tx=0).stop 2023-10-30T12:48:55.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.003+0000 7f157b09c640 1 -- 172.21.15.125:0/1018378564 shutdown_connections 2023-10-30T12:48:55.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.003+0000 7f157b09c640 1 --2- 172.21.15.125:0/1018378564 >> v2:172.21.15.136:3300/0 conn(0x7f15740ca0e0 0x7f157415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:55.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.003+0000 7f157b09c640 1 --2- 172.21.15.125:0/1018378564 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15741538a0 0x7f157415d6f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:55.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.003+0000 7f157b09c640 1 -- 172.21.15.125:0/1018378564 >> 172.21.15.125:0/1018378564 conn(0x7f157406bfe0 msgr2=0x7f157406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:55.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.003+0000 7f157b09c640 1 -- 172.21.15.125:0/1018378564 shutdown_connections 2023-10-30T12:48:55.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.003+0000 7f157b09c640 1 -- 172.21.15.125:0/1018378564 wait complete. 2023-10-30T12:48:55.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f157b09c640 1 Processor -- start 2023-10-30T12:48:55.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f157b09c640 1 -- start start 2023-10-30T12:48:55.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f157b09c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15740ca0e0 0x7f15740ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:55.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f157b09c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15740cf160 0x7f15740b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:55.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f157b09c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15741538a0 0x7f15740b42f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:55.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f157b09c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15741efcc0 con 0x7f15740cf160 2023-10-30T12:48:55.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f157b09c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15741f01f0 con 0x7f15741538a0 2023-10-30T12:48:55.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f157b09c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15741eff10 con 0x7f15740ca0e0 2023-10-30T12:48:55.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f1573fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15740cf160 0x7f15740b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:55.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f1573fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15740cf160 0x7f15740b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51130/0 (socket says 172.21.15.125:51130) 2023-10-30T12:48:55.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f1573fff640 1 -- 172.21.15.125:0/455545586 learned_addr learned my addr 172.21.15.125:0/455545586 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:55.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f1578e38640 1 --2- 172.21.15.125:0/455545586 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15740ca0e0 0x7f15740ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:55.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f1579639640 1 --2- 172.21.15.125:0/455545586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15741538a0 0x7f15740b42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:55.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f1573fff640 1 -- 172.21.15.125:0/455545586 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15740ca0e0 msgr2=0x7f15740ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:55.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f1573fff640 1 --2- 172.21.15.125:0/455545586 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15740ca0e0 0x7f15740ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:55.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f1573fff640 1 -- 172.21.15.125:0/455545586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15741538a0 msgr2=0x7f15740b42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:55.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f1573fff640 1 --2- 172.21.15.125:0/455545586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15741538a0 0x7f15740b42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:55.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f1573fff640 1 -- 172.21.15.125:0/455545586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15600094d0 con 0x7f15740cf160 2023-10-30T12:48:55.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f1573fff640 1 --2- 172.21.15.125:0/455545586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15740cf160 0x7f15740b3db0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f156000e9d0 tx=0x7f156000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:55.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f15717fa640 1 -- 172.21.15.125:0/455545586 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156000ca60 con 0x7f15740cf160 2023-10-30T12:48:55.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f15717fa640 1 -- 172.21.15.125:0/455545586 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156000cbc0 con 0x7f15740cf160 2023-10-30T12:48:55.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f15717fa640 1 -- 172.21.15.125:0/455545586 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1560013470 con 0x7f15740cf160 2023-10-30T12:48:55.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f157b09c640 1 -- 172.21.15.125:0/455545586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1568009420 con 0x7f15740cf160 2023-10-30T12:48:55.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.007+0000 7f157b09c640 1 -- 172.21.15.125:0/455545586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15740b35b0 con 0x7f15740cf160 2023-10-30T12:48:55.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.011+0000 7f157b09c640 1 -- 172.21.15.125:0/455545586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15740b2a50 con 0x7f15740cf160 2023-10-30T12:48:55.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.011+0000 7f15717fa640 1 -- 172.21.15.125:0/455545586 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f15600135d0 con 0x7f15740cf160 2023-10-30T12:48:55.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.011+0000 7f15717fa640 1 --2- 172.21.15.125:0/455545586 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15580664d0 0x7f1558068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:55.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.011+0000 7f15717fa640 1 -- 172.21.15.125:0/455545586 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f1560084840 con 0x7f15740cf160 2023-10-30T12:48:55.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.011+0000 7f1578e38640 1 --2- 172.21.15.125:0/455545586 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15580664d0 0x7f1558068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:55.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.011+0000 7f1578e38640 1 --2- 172.21.15.125:0/455545586 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15580664d0 0x7f1558068990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f15680b2000 tx=0x7f15680aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:55.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.015+0000 7f15717fa640 1 -- 172.21.15.125:0/455545586 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1560051930 con 0x7f15740cf160 2023-10-30T12:48:55.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.195+0000 7f157b09c640 1 -- 172.21.15.125:0/455545586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f157404e170 con 0x7f15740cf160 2023-10-30T12:48:55.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.195+0000 7f15717fa640 1 -- 172.21.15.125:0/455545586 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+5740 (secure 0 0 0) 0x7f1560054f50 con 0x7f15740cf160 2023-10-30T12:48:55.195 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:55.195 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:49.544257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:55.196 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 104 2023-10-30T12:48:55.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.199+0000 7f157b09c640 1 -- 172.21.15.125:0/455545586 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15580664d0 msgr2=0x7f1558068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:55.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.199+0000 7f157b09c640 1 --2- 172.21.15.125:0/455545586 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15580664d0 0x7f1558068990 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f15680b2000 tx=0x7f15680aa070 comp rx=0 tx=0).stop 2023-10-30T12:48:55.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.199+0000 7f157b09c640 1 -- 172.21.15.125:0/455545586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15740cf160 msgr2=0x7f15740b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:55.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.199+0000 7f157b09c640 1 --2- 172.21.15.125:0/455545586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15740cf160 0x7f15740b3db0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f156000e9d0 tx=0x7f156000ee90 comp rx=0 tx=0).stop 2023-10-30T12:48:55.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.199+0000 7f157b09c640 1 -- 172.21.15.125:0/455545586 shutdown_connections 2023-10-30T12:48:55.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.199+0000 7f157b09c640 1 --2- 172.21.15.125:0/455545586 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15580664d0 0x7f1558068990 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:55.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.199+0000 7f157b09c640 1 --2- 172.21.15.125:0/455545586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15741538a0 0x7f15740b42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:55.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.199+0000 7f157b09c640 1 --2- 172.21.15.125:0/455545586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15740cf160 0x7f15740b3db0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:55.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.199+0000 7f157b09c640 1 --2- 172.21.15.125:0/455545586 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15740ca0e0 0x7f15740ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:55.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.199+0000 7f157b09c640 1 -- 172.21.15.125:0/455545586 >> 172.21.15.125:0/455545586 conn(0x7f157406bfe0 msgr2=0x7f1574152870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:55.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.199+0000 7f157b09c640 1 -- 172.21.15.125:0/455545586 shutdown_connections 2023-10-30T12:48:55.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:55.199+0000 7f157b09c640 1 -- 172.21.15.125:0/455545586 wait complete. 2023-10-30T12:48:55.217 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T12:48:57.217 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:48:57.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.371+0000 7f99e2a09640 1 Processor -- start 2023-10-30T12:48:57.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.371+0000 7f99e2a09640 1 -- start start 2023-10-30T12:48:57.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.371+0000 7f99e2a09640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f99dc0c05d0 0x7f99dc13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:57.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.371+0000 7f99e2a09640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99dc0c0a50 con 0x7f99dc0c6f60 2023-10-30T12:48:57.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.371+0000 7f99e2a09640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99dc06f6e0 con 0x7f99dc0c6b80 2023-10-30T12:48:57.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.371+0000 7f99e2a09640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99dc06f9c0 con 0x7f99dc0c05d0 2023-10-30T12:48:57.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.371+0000 7f99e0fa6640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f99dc0c6f60 0x7f99dc1566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36926/0 (socket says 172.21.15.125:36926) 2023-10-30T12:48:57.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.371+0000 7f99e0fa6640 1 -- 172.21.15.125:0/2075710662 learned_addr learned my addr 172.21.15.125:0/2075710662 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:48:57.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.371+0000 7f99db7fe640 1 --2- 172.21.15.125:0/2075710662 >> v2:172.21.15.136:3300/0 conn(0x7f99dc0c05d0 0x7f99dc13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:57.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.371+0000 7f99daffd640 1 -- 172.21.15.125:0/2075710662 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178023074 0 0) 0x7f99dc0c0a50 con 0x7f99dc0c6f60 2023-10-30T12:48:57.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99daffd640 1 -- 172.21.15.125:0/2075710662 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99c4005740 con 0x7f99dc0c6f60 2023-10-30T12:48:57.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99daffd640 1 -- 172.21.15.125:0/2075710662 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3325948273 0 0) 0x7f99c4005740 con 0x7f99dc0c6f60 2023-10-30T12:48:57.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99daffd640 1 -- 172.21.15.125:0/2075710662 >> v2:172.21.15.136:3300/0 conn(0x7f99dc0c05d0 msgr2=0x7f99dc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:57.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99daffd640 1 --2- 172.21.15.125:0/2075710662 >> v2:172.21.15.136:3300/0 conn(0x7f99dc0c05d0 0x7f99dc13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:57.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99daffd640 1 -- 172.21.15.125:0/2075710662 >> v1:172.21.15.136:6789/0 conn(0x7f99dc0c6b80 legacy=0x7f99dc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:57.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99daffd640 1 -- 172.21.15.125:0/2075710662 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99dc06fca0 con 0x7f99dc0c6f60 2023-10-30T12:48:57.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99daffd640 1 -- 172.21.15.125:0/2075710662 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f99d4001a00 con 0x7f99dc0c6f60 2023-10-30T12:48:57.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99daffd640 1 -- 172.21.15.125:0/2075710662 >> v1:172.21.15.125:6789/0 conn(0x7f99dc0c6f60 legacy=0x7f99dc1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:57.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99daffd640 1 --2- 172.21.15.125:0/2075710662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99c4005b90 0x7f99c4005f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:57.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99daffd640 1 --2- 172.21.15.125:0/2075710662 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99c4006590 0x7f99c4006a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:57.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99daffd640 1 --2- 172.21.15.125:0/2075710662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99c4007260 0x7f99c400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:57.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99daffd640 1 -- 172.21.15.125:0/2075710662 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc06f9c0 con 0x7f99c4007260 2023-10-30T12:48:57.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99daffd640 1 -- 172.21.15.125:0/2075710662 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c400e1c0 con 0x7f99c4005b90 2023-10-30T12:48:57.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99daffd640 1 -- 172.21.15.125:0/2075710662 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c400e380 con 0x7f99c4006590 2023-10-30T12:48:57.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99e0fa6640 1 --2- 172.21.15.125:0/2075710662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99c4007260 0x7f99c400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:57.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99dbfff640 1 --2- 172.21.15.125:0/2075710662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99c4005b90 0x7f99c4005f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:57.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99db7fe640 1 --2- 172.21.15.125:0/2075710662 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99c4006590 0x7f99c4006a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:57.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99e0fa6640 1 -- 172.21.15.125:0/2075710662 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99c4006590 msgr2=0x7f99c4006a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:57.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99e0fa6640 1 --2- 172.21.15.125:0/2075710662 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99c4006590 0x7f99c4006a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:57.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99e0fa6640 1 -- 172.21.15.125:0/2075710662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99c4005b90 msgr2=0x7f99c4005f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:48:57.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99e0fa6640 1 --2- 172.21.15.125:0/2075710662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99c4005b90 0x7f99c4005f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:57.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99e0fa6640 1 -- 172.21.15.125:0/2075710662 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c4005740 con 0x7f99c4007260 2023-10-30T12:48:57.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99e0fa6640 1 -- 172.21.15.125:0/2075710662 reap_dead start 2023-10-30T12:48:57.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.375+0000 7f99e0fa6640 1 --2- 172.21.15.125:0/2075710662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99c4007260 0x7f99c400dba0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f99d400a7d0 tx=0x7f99d401fa20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd63ea515fd911f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:57.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99daffd640 1 -- 172.21.15.125:0/2075710662 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d40af070 con 0x7f99c4007260 2023-10-30T12:48:57.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99daffd640 1 -- 172.21.15.125:0/2075710662 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d40aa450 con 0x7f99c4007260 2023-10-30T12:48:57.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99daffd640 1 -- 172.21.15.125:0/2075710662 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d40aa820 con 0x7f99c4007260 2023-10-30T12:48:57.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 -- 172.21.15.125:0/2075710662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99c4007260 msgr2=0x7f99c400dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:57.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 --2- 172.21.15.125:0/2075710662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99c4007260 0x7f99c400dba0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f99d400a7d0 tx=0x7f99d401fa20 comp rx=0 tx=0).stop 2023-10-30T12:48:57.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 -- 172.21.15.125:0/2075710662 shutdown_connections 2023-10-30T12:48:57.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 --2- 172.21.15.125:0/2075710662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99c4007260 0x7f99c400dba0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:57.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 -- 172.21.15.125:0/2075710662 >> 172.21.15.125:0/2075710662 conn(0x7f99dc0c7910 msgr2=0x7f99dc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:57.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 -- 172.21.15.125:0/2075710662 shutdown_connections 2023-10-30T12:48:57.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 -- 172.21.15.125:0/2075710662 wait complete. 2023-10-30T12:48:57.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 Processor -- start 2023-10-30T12:48:57.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 -- start start 2023-10-30T12:48:57.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99c4007260 0x7f99dc1368d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:57.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99dc1ee2c0 0x7f99dc1ea770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:57.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99dc151650 0x7f99dc1eacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:57.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc139250 con 0x7f99c4007260 2023-10-30T12:48:57.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc1ee800 con 0x7f99dc151650 2023-10-30T12:48:57.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.379+0000 7f99e2a09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc13a410 con 0x7f99dc1ee2c0 2023-10-30T12:48:57.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99dbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99c4007260 0x7f99dc1368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:57.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99dbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99c4007260 0x7f99dc1368d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51158/0 (socket says 172.21.15.125:51158) 2023-10-30T12:48:57.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99dbfff640 1 -- 172.21.15.125:0/2743109182 learned_addr learned my addr 172.21.15.125:0/2743109182 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:48:57.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99e0fa6640 1 --2- 172.21.15.125:0/2743109182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99dc151650 0x7f99dc1eacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:57.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99db7fe640 1 --2- 172.21.15.125:0/2743109182 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99dc1ee2c0 0x7f99dc1ea770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:57.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99dbfff640 1 -- 172.21.15.125:0/2743109182 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99dc1ee2c0 msgr2=0x7f99dc1ea770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99dbfff640 1 --2- 172.21.15.125:0/2743109182 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99dc1ee2c0 0x7f99dc1ea770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99dbfff640 1 -- 172.21.15.125:0/2743109182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99dc151650 msgr2=0x7f99dc1eacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99dbfff640 1 --2- 172.21.15.125:0/2743109182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99dc151650 0x7f99dc1eacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:57.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99dbfff640 1 -- 172.21.15.125:0/2743109182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99dc0c10d0 con 0x7f99c4007260 2023-10-30T12:48:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99dbfff640 1 --2- 172.21.15.125:0/2743109182 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99c4007260 0x7f99dc1368d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00e5d0 tx=0x7f99cc00ea90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99d8ff9640 1 -- 172.21.15.125:0/2743109182 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc0052e0 con 0x7f99c4007260 2023-10-30T12:48:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99d8ff9640 1 -- 172.21.15.125:0/2743109182 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc005440 con 0x7f99c4007260 2023-10-30T12:48:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99e2a09640 1 -- 172.21.15.125:0/2743109182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99dc13a6b0 con 0x7f99c4007260 2023-10-30T12:48:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99d8ff9640 1 -- 172.21.15.125:0/2743109182 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc003840 con 0x7f99c4007260 2023-10-30T12:48:57.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99e2a09640 1 -- 172.21.15.125:0/2743109182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99dc136e10 con 0x7f99c4007260 2023-10-30T12:48:57.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99e2a09640 1 -- 172.21.15.125:0/2743109182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99dc04e170 con 0x7f99c4007260 2023-10-30T12:48:57.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.383+0000 7f99d8ff9640 1 -- 172.21.15.125:0/2743109182 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f99cc0039a0 con 0x7f99c4007260 2023-10-30T12:48:57.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.387+0000 7f99d8ff9640 1 --2- 172.21.15.125:0/2743109182 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99c8066470 0x7f99c8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:48:57.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.387+0000 7f99d8ff9640 1 -- 172.21.15.125:0/2743109182 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f99cc084ad0 con 0x7f99c4007260 2023-10-30T12:48:57.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.387+0000 7f99db7fe640 1 --2- 172.21.15.125:0/2743109182 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99c8066470 0x7f99c8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:48:57.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.387+0000 7f99db7fe640 1 --2- 172.21.15.125:0/2743109182 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99c8066470 0x7f99c8068930 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f99dc13e8d0 tx=0x7f99d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:48:57.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.387+0000 7f99d8ff9640 1 -- 172.21.15.125:0/2743109182 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f99cc051c40 con 0x7f99c4007260 2023-10-30T12:48:57.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.575+0000 7f99e2a09640 1 -- 172.21.15.125:0/2743109182 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99dc1ea390 con 0x7f99c4007260 2023-10-30T12:48:57.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.575+0000 7f99d8ff9640 1 -- 172.21.15.125:0/2743109182 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+6535 (secure 0 0 0) 0x7f99cc055260 con 0x7f99c4007260 2023-10-30T12:48:57.575 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:48:57.575 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":105,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:48:49.544257+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:48:57.577 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 105 2023-10-30T12:48:57.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.579+0000 7f99e2a09640 1 -- 172.21.15.125:0/2743109182 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99c8066470 msgr2=0x7f99c8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:57.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.579+0000 7f99e2a09640 1 --2- 172.21.15.125:0/2743109182 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99c8066470 0x7f99c8068930 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f99dc13e8d0 tx=0x7f99d0008040 comp rx=0 tx=0).stop 2023-10-30T12:48:57.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.579+0000 7f99e2a09640 1 -- 172.21.15.125:0/2743109182 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99c4007260 msgr2=0x7f99dc1368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:48:57.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.579+0000 7f99e2a09640 1 --2- 172.21.15.125:0/2743109182 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99c4007260 0x7f99dc1368d0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00e5d0 tx=0x7f99cc00ea90 comp rx=0 tx=0).stop 2023-10-30T12:48:57.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.579+0000 7f99e2a09640 1 -- 172.21.15.125:0/2743109182 shutdown_connections 2023-10-30T12:48:57.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.579+0000 7f99e2a09640 1 --2- 172.21.15.125:0/2743109182 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99c8066470 0x7f99c8068930 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:57.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.579+0000 7f99e2a09640 1 --2- 172.21.15.125:0/2743109182 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99dc151650 0x7f99dc1eacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:57.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.579+0000 7f99e2a09640 1 --2- 172.21.15.125:0/2743109182 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99dc1ee2c0 0x7f99dc1ea770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:57.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.579+0000 7f99e2a09640 1 --2- 172.21.15.125:0/2743109182 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99c4007260 0x7f99dc1368d0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:48:57.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.579+0000 7f99e2a09640 1 -- 172.21.15.125:0/2743109182 >> 172.21.15.125:0/2743109182 conn(0x7f99dc0c7910 msgr2=0x7f99dc069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:48:57.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.579+0000 7f99e2a09640 1 -- 172.21.15.125:0/2743109182 shutdown_connections 2023-10-30T12:48:57.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:48:57.579+0000 7f99e2a09640 1 -- 172.21.15.125:0/2743109182 wait complete. 2023-10-30T12:48:57.597 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T12:48:57.597 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 14 secs before thrashing 2023-10-30T12:49:11.598 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.735+0000 7f707e1d0640 1 Processor -- start 2023-10-30T12:49:11.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707e1d0640 1 -- start start 2023-10-30T12:49:11.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707e1d0640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f707815cd50 0x7f70780c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707e1d0640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f707815a6c0 con 0x7f70781538f0 2023-10-30T12:49:11.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707e1d0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f707815d300 con 0x7f70780cefa0 2023-10-30T12:49:11.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707e1d0640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707815d5e0 con 0x7f707815cd50 2023-10-30T12:49:11.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707d1ce640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f70781538f0 0x7f7078159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60652/0 (socket says 172.21.15.125:60652) 2023-10-30T12:49:11.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707d1ce640 1 -- 172.21.15.125:0/246396814 learned_addr learned my addr 172.21.15.125:0/246396814 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:49:11.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707c9cd640 1 --2- 172.21.15.125:0/246396814 >> v2:172.21.15.136:3300/0 conn(0x7f707815cd50 0x7f70780c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2852601753 0 0) 0x7f707815a6c0 con 0x7f70781538f0 2023-10-30T12:49:11.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7064003530 con 0x7f70781538f0 2023-10-30T12:49:11.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1890374124 0 0) 0x7f707815d300 con 0x7f70780cefa0 2023-10-30T12:49:11.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f707815a6c0 con 0x7f70780cefa0 2023-10-30T12:49:11.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3996750886 0 0) 0x7f7064003530 con 0x7f70781538f0 2023-10-30T12:49:11.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 >> v2:172.21.15.136:3300/0 conn(0x7f707815cd50 msgr2=0x7f70780c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:11.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 --2- 172.21.15.125:0/246396814 >> v2:172.21.15.136:3300/0 conn(0x7f707815cd50 0x7f70780c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 >> v1:172.21.15.136:6789/0 conn(0x7f70780cefa0 legacy=0x7f70781640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:11.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707815d8c0 con 0x7f70781538f0 2023-10-30T12:49:11.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f7074001680 con 0x7f70781538f0 2023-10-30T12:49:11.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 >> v1:172.21.15.125:6789/0 conn(0x7f70781538f0 legacy=0x7f7078159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:11.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 --2- 172.21.15.125:0/246396814 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7064003b10 0x7f7064003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 --2- 172.21.15.125:0/246396814 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70640047d0 0x7f7064004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 --2- 172.21.15.125:0/246396814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7064005190 0x7f706400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707815d5e0 con 0x7f7064003b10 2023-10-30T12:49:11.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f706400bf80 con 0x7f7064005190 2023-10-30T12:49:11.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f706400c140 con 0x7f70640047d0 2023-10-30T12:49:11.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707d1ce640 1 --2- 172.21.15.125:0/246396814 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7064003b10 0x7f7064003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707c9cd640 1 --2- 172.21.15.125:0/246396814 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70640047d0 0x7f7064004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707d9cf640 1 --2- 172.21.15.125:0/246396814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7064005190 0x7f706400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707d1ce640 1 -- 172.21.15.125:0/246396814 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70640047d0 msgr2=0x7f7064004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:11.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707d1ce640 1 --2- 172.21.15.125:0/246396814 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70640047d0 0x7f7064004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707d1ce640 1 -- 172.21.15.125:0/246396814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7064005190 msgr2=0x7f706400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:11.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707d1ce640 1 --2- 172.21.15.125:0/246396814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7064005190 0x7f706400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707d1ce640 1 -- 172.21.15.125:0/246396814 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7064003530 con 0x7f7064003b10 2023-10-30T12:49:11.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707d9cf640 1 -- 172.21.15.125:0/246396814 reap_dead start 2023-10-30T12:49:11.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.739+0000 7f707c9cd640 1 --2- 172.21.15.125:0/246396814 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70640047d0 0x7f7064004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:11.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.743+0000 7f707d1ce640 1 --2- 172.21.15.125:0/246396814 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7064003b10 0x7f7064003f10 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7074001ec0 tx=0x7f707401cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a2168b1c63cccea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:11.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.743+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70740058e0 con 0x7f7064003b10 2023-10-30T12:49:11.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.743+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7074005a60 con 0x7f7064003b10 2023-10-30T12:49:11.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.743+0000 7f706ffff640 1 -- 172.21.15.125:0/246396814 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7074005e10 con 0x7f7064003b10 2023-10-30T12:49:11.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.743+0000 7f707e1d0640 1 -- 172.21.15.125:0/246396814 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7064003b10 msgr2=0x7f7064003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:11.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.743+0000 7f707e1d0640 1 --2- 172.21.15.125:0/246396814 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7064003b10 0x7f7064003f10 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f7074001ec0 tx=0x7f707401cba0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.743+0000 7f707e1d0640 1 -- 172.21.15.125:0/246396814 shutdown_connections 2023-10-30T12:49:11.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.743+0000 7f707e1d0640 1 --2- 172.21.15.125:0/246396814 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7064003b10 0x7f7064003f10 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.743+0000 7f707e1d0640 1 -- 172.21.15.125:0/246396814 >> 172.21.15.125:0/246396814 conn(0x7f707806a640 msgr2=0x7f7078065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:11.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.743+0000 7f707e1d0640 1 -- 172.21.15.125:0/246396814 shutdown_connections 2023-10-30T12:49:11.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.743+0000 7f707e1d0640 1 -- 172.21.15.125:0/246396814 wait complete. 2023-10-30T12:49:11.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707e1d0640 1 Processor -- start 2023-10-30T12:49:11.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707e1d0640 1 -- start start 2023-10-30T12:49:11.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707e1d0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7064003b10 0x7f70780b3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707e1d0640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7078148490 0x7f70780ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707e1d0640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7078146890 0x7f7078147330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707e1d0640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70780b43a0 con 0x7f7078146890 2023-10-30T12:49:11.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707e1d0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70780b3d60 con 0x7f7064003b10 2023-10-30T12:49:11.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707e1d0640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70781505b0 con 0x7f7078148490 2023-10-30T12:49:11.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707d9cf640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7078146890 0x7f7078147330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707d9cf640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7078146890 0x7f7078147330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36662/0 (socket says 172.21.15.125:36662) 2023-10-30T12:49:11.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707d9cf640 1 -- 172.21.15.125:0/3214069508 learned_addr learned my addr 172.21.15.125:0/3214069508 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:11.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707c9cd640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7078148490 0x7f70780ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707d1ce640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7064003b10 0x7f70780b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707d9cf640 1 -- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7078148490 msgr2=0x7f70780ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:11.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707d9cf640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7078148490 0x7f70780ce1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707d9cf640 1 -- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7064003b10 msgr2=0x7f70780b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:11.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707d9cf640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7064003b10 0x7f70780b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707d9cf640 1 -- 172.21.15.125:0/3214069508 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70781efa10 con 0x7f7078146890 2023-10-30T12:49:11.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707c9cd640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7078148490 0x7f70780ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:11.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707d9cf640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7078146890 0x7f7078147330 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f70780b1ae0 tx=0x7f7070011f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:11.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f706dffb640 1 -- 172.21.15.125:0/3214069508 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f707000fd10 con 0x7f7078146890 2023-10-30T12:49:11.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f706dffb640 1 -- 172.21.15.125:0/3214069508 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f707000fe90 con 0x7f7078146890 2023-10-30T12:49:11.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707e1d0640 1 -- 172.21.15.125:0/3214069508 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70781507e0 con 0x7f7078146890 2023-10-30T12:49:11.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f706dffb640 1 -- 172.21.15.125:0/3214069508 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7070015650 con 0x7f7078146890 2023-10-30T12:49:11.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.747+0000 7f707e1d0640 1 -- 172.21.15.125:0/3214069508 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70781ef3c0 con 0x7f7078146890 2023-10-30T12:49:11.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.751+0000 7f707e1d0640 1 -- 172.21.15.125:0/3214069508 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70781efba0 con 0x7f7078146890 2023-10-30T12:49:11.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.751+0000 7f706dffb640 1 -- 172.21.15.125:0/3214069508 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f70700157b0 con 0x7f7078146890 2023-10-30T12:49:11.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.751+0000 7f706dffb640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f705c0664a0 0x7f705c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:11.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.751+0000 7f706dffb640 1 -- 172.21.15.125:0/3214069508 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f70700876b0 con 0x7f7078146890 2023-10-30T12:49:11.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.751+0000 7f707d1ce640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f705c0664a0 0x7f705c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:11.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.751+0000 7f707d1ce640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f705c0664a0 0x7f705c068960 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7074001ec0 tx=0x7f7074001a00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:11.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.755+0000 7f706dffb640 1 -- 172.21.15.125:0/3214069508 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7070054840 con 0x7f7078146890 2023-10-30T12:49:11.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.939+0000 7f707e1d0640 1 -- 172.21.15.125:0/3214069508 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f707804e170 con 0x7f7078146890 2023-10-30T12:49:11.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.939+0000 7f706dffb640 1 -- 172.21.15.125:0/3214069508 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+6536 (secure 0 0 0) 0x7f7070057e60 con 0x7f7078146890 2023-10-30T12:49:11.940 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:11.940 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:05.510099+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":2,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:active","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:11.942 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 106 2023-10-30T12:49:11.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.943+0000 7f707e1d0640 1 -- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f705c0664a0 msgr2=0x7f705c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:11.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.943+0000 7f707e1d0640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f705c0664a0 0x7f705c068960 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7074001ec0 tx=0x7f7074001a00 comp rx=0 tx=0).stop 2023-10-30T12:49:11.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.943+0000 7f707e1d0640 1 -- 172.21.15.125:0/3214069508 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7078146890 msgr2=0x7f7078147330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:11.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.943+0000 7f707e1d0640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7078146890 0x7f7078147330 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f70780b1ae0 tx=0x7f7070011f90 comp rx=0 tx=0).stop 2023-10-30T12:49:11.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.943+0000 7f707e1d0640 1 -- 172.21.15.125:0/3214069508 shutdown_connections 2023-10-30T12:49:11.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.943+0000 7f707e1d0640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f705c0664a0 0x7f705c068960 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.943+0000 7f707e1d0640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7078146890 0x7f7078147330 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.943+0000 7f707e1d0640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7078148490 0x7f70780ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.943+0000 7f707e1d0640 1 --2- 172.21.15.125:0/3214069508 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7064003b10 0x7f70780b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:11.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.943+0000 7f707e1d0640 1 -- 172.21.15.125:0/3214069508 >> 172.21.15.125:0/3214069508 conn(0x7f707806a640 msgr2=0x7f70780caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:11.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.943+0000 7f707e1d0640 1 -- 172.21.15.125:0/3214069508 shutdown_connections 2023-10-30T12:49:11.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:11.943+0000 7f707e1d0640 1 -- 172.21.15.125:0/3214069508 wait complete. 2023-10-30T12:49:11.960 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 2 -> 1 2023-10-30T12:49:11.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 1 2023-10-30T12:49:12.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.103+0000 7fbb471d3640 1 Processor -- start 2023-10-30T12:49:12.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.103+0000 7fbb471d3640 1 -- start start 2023-10-30T12:49:12.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.103+0000 7fbb471d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb40159930 0x7fbb400cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:12.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.103+0000 7fbb471d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb400cf6e0 0x7fbb4015d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:12.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.103+0000 7fbb471d3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4015a920 con 0x7fbb40153980 2023-10-30T12:49:12.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.103+0000 7fbb471d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb400cb930 con 0x7fbb400cf6e0 2023-10-30T12:49:12.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.103+0000 7fbb471d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb40159dc0 con 0x7fbb40159930 2023-10-30T12:49:12.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.103+0000 7fbb45770640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb400cf6e0 0x7fbb4015d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:12.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.103+0000 7fbb45770640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb400cf6e0 0x7fbb4015d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36678/0 (socket says 172.21.15.125:36678) 2023-10-30T12:49:12.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.103+0000 7fbb44f6f640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fbb40153980 0x7fbb400caca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60662/0 (socket says 172.21.15.125:60662) 2023-10-30T12:49:12.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.103+0000 7fbb45770640 1 -- 172.21.15.125:0/617367612 learned_addr learned my addr 172.21.15.125:0/617367612 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:12.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb37fff640 1 --2- 172.21.15.125:0/617367612 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb40159930 0x7fbb400cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:12.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb45770640 1 -- 172.21.15.125:0/617367612 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb40159930 msgr2=0x7fbb400cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:12.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb45770640 1 --2- 172.21.15.125:0/617367612 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb40159930 0x7fbb400cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb45770640 1 -- 172.21.15.125:0/617367612 >> v1:172.21.15.125:6789/0 conn(0x7fbb40153980 legacy=0x7fbb400caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb45770640 1 -- 172.21.15.125:0/617367612 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4015a080 con 0x7fbb400cf6e0 2023-10-30T12:49:12.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb45770640 1 --2- 172.21.15.125:0/617367612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb400cf6e0 0x7fbb4015d570 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fbb38009e50 tx=0x7fbb3801c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c1cfec1d6c14558 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:12.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb377fe640 1 -- 172.21.15.125:0/617367612 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb380a57c0 con 0x7fbb400cf6e0 2023-10-30T12:49:12.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb377fe640 1 -- 172.21.15.125:0/617367612 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb380a5920 con 0x7fbb400cf6e0 2023-10-30T12:49:12.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb377fe640 1 -- 172.21.15.125:0/617367612 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb380a5cb0 con 0x7fbb400cf6e0 2023-10-30T12:49:12.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb471d3640 1 -- 172.21.15.125:0/617367612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb400cf6e0 msgr2=0x7fbb4015d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:12.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb471d3640 1 --2- 172.21.15.125:0/617367612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb400cf6e0 0x7fbb4015d570 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fbb38009e50 tx=0x7fbb3801c9e0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb471d3640 1 -- 172.21.15.125:0/617367612 shutdown_connections 2023-10-30T12:49:12.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb471d3640 1 --2- 172.21.15.125:0/617367612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb400cf6e0 0x7fbb4015d570 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb471d3640 1 --2- 172.21.15.125:0/617367612 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb40159930 0x7fbb400cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.107+0000 7fbb471d3640 1 -- 172.21.15.125:0/617367612 >> 172.21.15.125:0/617367612 conn(0x7fbb4006c0f0 msgr2=0x7fbb4006c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:12.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb471d3640 1 -- 172.21.15.125:0/617367612 shutdown_connections 2023-10-30T12:49:12.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb471d3640 1 -- 172.21.15.125:0/617367612 wait complete. 2023-10-30T12:49:12.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb471d3640 1 Processor -- start 2023-10-30T12:49:12.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb471d3640 1 -- start start 2023-10-30T12:49:12.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb471d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb400cf6e0 0x7fbb4015c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:12.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb471d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb40153980 0x7fbb4015c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:12.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb471d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb4015aaf0 0x7fbb40148240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:12.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb471d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb400cee60 con 0x7fbb4015aaf0 2023-10-30T12:49:12.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb471d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb40145b20 con 0x7fbb40153980 2023-10-30T12:49:12.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb471d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb401477f0 con 0x7fbb400cf6e0 2023-10-30T12:49:12.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb45770640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb4015aaf0 0x7fbb40148240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:12.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb45770640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb4015aaf0 0x7fbb40148240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36690/0 (socket says 172.21.15.125:36690) 2023-10-30T12:49:12.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb45770640 1 -- 172.21.15.125:0/1717050085 learned_addr learned my addr 172.21.15.125:0/1717050085 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:12.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb44f6f640 1 --2- 172.21.15.125:0/1717050085 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb400cf6e0 0x7fbb4015c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:12.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb37fff640 1 --2- 172.21.15.125:0/1717050085 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb40153980 0x7fbb4015c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:12.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb45770640 1 -- 172.21.15.125:0/1717050085 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb400cf6e0 msgr2=0x7fbb4015c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:12.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb45770640 1 --2- 172.21.15.125:0/1717050085 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb400cf6e0 0x7fbb4015c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb45770640 1 -- 172.21.15.125:0/1717050085 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb40153980 msgr2=0x7fbb4015c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:12.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb45770640 1 --2- 172.21.15.125:0/1717050085 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb40153980 0x7fbb4015c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb45770640 1 -- 172.21.15.125:0/1717050085 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb38009b30 con 0x7fbb4015aaf0 2023-10-30T12:49:12.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.111+0000 7fbb45770640 1 --2- 172.21.15.125:0/1717050085 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb4015aaf0 0x7fbb40148240 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fbb38009e50 tx=0x7fbb38019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:12.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.115+0000 7fbb357fa640 1 -- 172.21.15.125:0/1717050085 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb380bd070 con 0x7fbb4015aaf0 2023-10-30T12:49:12.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.115+0000 7fbb357fa640 1 -- 172.21.15.125:0/1717050085 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb380b8470 con 0x7fbb4015aaf0 2023-10-30T12:49:12.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.115+0000 7fbb471d3640 1 -- 172.21.15.125:0/1717050085 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb40147db0 con 0x7fbb4015aaf0 2023-10-30T12:49:12.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.115+0000 7fbb357fa640 1 -- 172.21.15.125:0/1717050085 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb380b8800 con 0x7fbb4015aaf0 2023-10-30T12:49:12.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.115+0000 7fbb471d3640 1 -- 172.21.15.125:0/1717050085 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb40152000 con 0x7fbb4015aaf0 2023-10-30T12:49:12.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.115+0000 7fbb471d3640 1 -- 172.21.15.125:0/1717050085 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb40147f40 con 0x7fbb4015aaf0 2023-10-30T12:49:12.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.115+0000 7fbb357fa640 1 -- 172.21.15.125:0/1717050085 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb380b8960 con 0x7fbb4015aaf0 2023-10-30T12:49:12.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.115+0000 7fbb357fa640 1 --2- 172.21.15.125:0/1717050085 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb2c0664d0 0x7fbb2c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:12.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.115+0000 7fbb357fa640 1 -- 172.21.15.125:0/1717050085 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fbb3811f330 con 0x7fbb4015aaf0 2023-10-30T12:49:12.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.115+0000 7fbb44f6f640 1 --2- 172.21.15.125:0/1717050085 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb2c0664d0 0x7fbb2c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:12.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.119+0000 7fbb44f6f640 1 --2- 172.21.15.125:0/1717050085 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb2c0664d0 0x7fbb2c068990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c000f70 tx=0x7fbb3c007c90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:12.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.119+0000 7fbb357fa640 1 -- 172.21.15.125:0/1717050085 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbb380ec4b0 con 0x7fbb4015aaf0 2023-10-30T12:49:12.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.307+0000 7fbb471d3640 1 -- 172.21.15.125:0/1717050085 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fbb4004e170 con 0x7fbb4015aaf0 2023-10-30T12:49:12.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.323+0000 7fbb357fa640 1 -- 172.21.15.125:0/1717050085 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v107) v1 ==== 105+0+0 (secure 0 0 0) 0x7fbb380efad0 con 0x7fbb4015aaf0 2023-10-30T12:49:12.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.323+0000 7fbb471d3640 1 -- 172.21.15.125:0/1717050085 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb2c0664d0 msgr2=0x7fbb2c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:12.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.323+0000 7fbb471d3640 1 --2- 172.21.15.125:0/1717050085 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb2c0664d0 0x7fbb2c068990 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c000f70 tx=0x7fbb3c007c90 comp rx=0 tx=0).stop 2023-10-30T12:49:12.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.323+0000 7fbb471d3640 1 -- 172.21.15.125:0/1717050085 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb4015aaf0 msgr2=0x7fbb40148240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:12.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.323+0000 7fbb471d3640 1 --2- 172.21.15.125:0/1717050085 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb4015aaf0 0x7fbb40148240 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fbb38009e50 tx=0x7fbb38019780 comp rx=0 tx=0).stop 2023-10-30T12:49:12.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.323+0000 7fbb471d3640 1 -- 172.21.15.125:0/1717050085 shutdown_connections 2023-10-30T12:49:12.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.323+0000 7fbb471d3640 1 --2- 172.21.15.125:0/1717050085 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb2c0664d0 0x7fbb2c068990 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.323+0000 7fbb471d3640 1 --2- 172.21.15.125:0/1717050085 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb4015aaf0 0x7fbb40148240 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.323+0000 7fbb471d3640 1 --2- 172.21.15.125:0/1717050085 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb40153980 0x7fbb4015c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.323+0000 7fbb471d3640 1 --2- 172.21.15.125:0/1717050085 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb400cf6e0 0x7fbb4015c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.323+0000 7fbb471d3640 1 -- 172.21.15.125:0/1717050085 >> 172.21.15.125:0/1717050085 conn(0x7fbb4006c0f0 msgr2=0x7fbb40158470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:12.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.323+0000 7fbb471d3640 1 -- 172.21.15.125:0/1717050085 shutdown_connections 2023-10-30T12:49:12.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.323+0000 7fbb471d3640 1 -- 172.21.15.125:0/1717050085 wait complete. 2023-10-30T12:49:12.344 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:49:12.345 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:12.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.487+0000 7fa6017c5640 1 Processor -- start 2023-10-30T12:49:12.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.487+0000 7fa6017c5640 1 -- start start 2023-10-30T12:49:12.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.487+0000 7fa6017c5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa5fc1538a0 0x7fa5fc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:12.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.487+0000 7fa6017c5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc0cb800 con 0x7fa5fc0cf160 2023-10-30T12:49:12.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.487+0000 7fa6017c5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc15dc30 con 0x7fa5fc0ca0e0 2023-10-30T12:49:12.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.487+0000 7fa6017c5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5fc0cb330 con 0x7fa5fc1538a0 2023-10-30T12:49:12.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.487+0000 7fa5faffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa5fc1538a0 0x7fa5fc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:12.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.487+0000 7fa5fb7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa5fc0cf160 0x7fa5fc164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60664/0 (socket says 172.21.15.125:60664) 2023-10-30T12:49:12.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.487+0000 7fa5faffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa5fc1538a0 0x7fa5fc15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36696/0 (socket says 172.21.15.125:36696) 2023-10-30T12:49:12.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.487+0000 7fa5fb7fe640 1 -- 172.21.15.125:0/4209693500 learned_addr learned my addr 172.21.15.125:0/4209693500 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:49:12.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.487+0000 7fa5faffd640 1 -- 172.21.15.125:0/4209693500 >> v1:172.21.15.136:6789/0 conn(0x7fa5fc0ca0e0 legacy=0x7fa5fc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:12.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.487+0000 7fa5faffd640 1 -- 172.21.15.125:0/4209693500 >> v1:172.21.15.125:6789/0 conn(0x7fa5fc0cf160 legacy=0x7fa5fc164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:12.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.487+0000 7fa5faffd640 1 -- 172.21.15.125:0/4209693500 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc0cad70 con 0x7fa5fc1538a0 2023-10-30T12:49:12.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.487+0000 7fa5fa7fc640 1 -- 172.21.15.125:0/4209693500 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022932210 0 0) 0x7fa5fc0cb800 con 0x7fa5fc0cf160 2023-10-30T12:49:12.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.491+0000 7fa5faffd640 1 --2- 172.21.15.125:0/4209693500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5fc1538a0 0x7fa5fc15d6f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec0097c0 tx=0x7fa5ec01ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dfae7458ee00ca4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:12.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.491+0000 7fa5fa7fc640 1 -- 172.21.15.125:0/4209693500 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec0a8930 con 0x7fa5fc1538a0 2023-10-30T12:49:12.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.491+0000 7fa5fa7fc640 1 -- 172.21.15.125:0/4209693500 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec0a8a90 con 0x7fa5fc1538a0 2023-10-30T12:49:12.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.491+0000 7fa5fa7fc640 1 -- 172.21.15.125:0/4209693500 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec0a8e20 con 0x7fa5fc1538a0 2023-10-30T12:49:12.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.491+0000 7fa6017c5640 1 -- 172.21.15.125:0/4209693500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5fc1538a0 msgr2=0x7fa5fc15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:12.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.491+0000 7fa6017c5640 1 --2- 172.21.15.125:0/4209693500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5fc1538a0 0x7fa5fc15d6f0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec0097c0 tx=0x7fa5ec01ec30 comp rx=0 tx=0).stop 2023-10-30T12:49:12.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.491+0000 7fa6017c5640 1 -- 172.21.15.125:0/4209693500 shutdown_connections 2023-10-30T12:49:12.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.491+0000 7fa6017c5640 1 --2- 172.21.15.125:0/4209693500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5fc1538a0 0x7fa5fc15d6f0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.491+0000 7fa6017c5640 1 -- 172.21.15.125:0/4209693500 >> 172.21.15.125:0/4209693500 conn(0x7fa5fc06bfe0 msgr2=0x7fa5fc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:12.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.491+0000 7fa6017c5640 1 -- 172.21.15.125:0/4209693500 shutdown_connections 2023-10-30T12:49:12.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.491+0000 7fa6017c5640 1 -- 172.21.15.125:0/4209693500 wait complete. 2023-10-30T12:49:12.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.491+0000 7fa6017c5640 1 Processor -- start 2023-10-30T12:49:12.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.491+0000 7fa6017c5640 1 -- start start 2023-10-30T12:49:12.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa6017c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5fc0ca0e0 0x7fa5fc0b4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:12.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa6017c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5fc0cf160 0x7fa5fc147780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:12.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa6017c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc1538a0 0x7fa5fc1f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:12.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa6017c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0ce150 con 0x7fa5fc0ca0e0 2023-10-30T12:49:12.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa6017c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0ce9b0 con 0x7fa5fc1538a0 2023-10-30T12:49:12.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa6017c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc0b4e90 con 0x7fa5fc0cf160 2023-10-30T12:49:12.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa5faffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5fc0ca0e0 0x7fa5fc0b4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:12.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa5faffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5fc0ca0e0 0x7fa5fc0b4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36702/0 (socket says 172.21.15.125:36702) 2023-10-30T12:49:12.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa5faffd640 1 -- 172.21.15.125:0/763338163 learned_addr learned my addr 172.21.15.125:0/763338163 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:12.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa5f3fff640 1 --2- 172.21.15.125:0/763338163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5fc0cf160 0x7fa5fc147780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:12.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa5fb7fe640 1 --2- 172.21.15.125:0/763338163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc1538a0 0x7fa5fc1f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:12.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa5faffd640 1 -- 172.21.15.125:0/763338163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5fc0cf160 msgr2=0x7fa5fc147780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:12.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa5faffd640 1 --2- 172.21.15.125:0/763338163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5fc0cf160 0x7fa5fc147780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa5faffd640 1 -- 172.21.15.125:0/763338163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc1538a0 msgr2=0x7fa5fc1f2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:12.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa5faffd640 1 --2- 172.21.15.125:0/763338163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc1538a0 0x7fa5fc1f2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa5faffd640 1 -- 172.21.15.125:0/763338163 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5ec009420 con 0x7fa5fc0ca0e0 2023-10-30T12:49:12.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa5faffd640 1 --2- 172.21.15.125:0/763338163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5fc0ca0e0 0x7fa5fc0b4950 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec0b3040 tx=0x7fa5ec01b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:12.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa5f37fe640 1 -- 172.21.15.125:0/763338163 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec0a8930 con 0x7fa5fc0ca0e0 2023-10-30T12:49:12.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa5f37fe640 1 -- 172.21.15.125:0/763338163 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ec0b9410 con 0x7fa5fc0ca0e0 2023-10-30T12:49:12.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa5f37fe640 1 -- 172.21.15.125:0/763338163 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec0b97e0 con 0x7fa5fc0ca0e0 2023-10-30T12:49:12.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa6017c5640 1 -- 172.21.15.125:0/763338163 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5fc0ce360 con 0x7fa5fc0ca0e0 2023-10-30T12:49:12.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa6017c5640 1 -- 172.21.15.125:0/763338163 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5fc0b3c40 con 0x7fa5fc0ca0e0 2023-10-30T12:49:12.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.495+0000 7fa6017c5640 1 -- 172.21.15.125:0/763338163 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5fc04e170 con 0x7fa5fc0ca0e0 2023-10-30T12:49:12.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.499+0000 7fa5f37fe640 1 -- 172.21.15.125:0/763338163 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa5ec0b7040 con 0x7fa5fc0ca0e0 2023-10-30T12:49:12.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.499+0000 7fa5f37fe640 1 --2- 172.21.15.125:0/763338163 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa5dc0664a0 0x7fa5dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:12.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.499+0000 7fa5f37fe640 1 -- 172.21.15.125:0/763338163 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fa5ec121f90 con 0x7fa5fc0ca0e0 2023-10-30T12:49:12.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.499+0000 7fa5f3fff640 1 --2- 172.21.15.125:0/763338163 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa5dc0664a0 0x7fa5dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:12.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.499+0000 7fa5f3fff640 1 --2- 172.21.15.125:0/763338163 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa5dc0664a0 0x7fa5dc068960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0cf600 tx=0x7fa5e4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:12.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.503+0000 7fa5f37fe640 1 -- 172.21.15.125:0/763338163 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa5ec0eef60 con 0x7fa5fc0ca0e0 2023-10-30T12:49:12.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.687+0000 7fa6017c5640 1 -- 172.21.15.125:0/763338163 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5fc15dc30 con 0x7fa5fc0ca0e0 2023-10-30T12:49:12.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.687+0000 7fa5f37fe640 1 -- 172.21.15.125:0/763338163 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7fa5fc15dc30 con 0x7fa5fc0ca0e0 2023-10-30T12:49:12.685 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:12.685 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:12.686 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:12.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.687+0000 7fa6017c5640 1 -- 172.21.15.125:0/763338163 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa5dc0664a0 msgr2=0x7fa5dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:12.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.687+0000 7fa6017c5640 1 --2- 172.21.15.125:0/763338163 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa5dc0664a0 0x7fa5dc068960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc0cf600 tx=0x7fa5e4008040 comp rx=0 tx=0).stop 2023-10-30T12:49:12.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.687+0000 7fa6017c5640 1 -- 172.21.15.125:0/763338163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5fc0ca0e0 msgr2=0x7fa5fc0b4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:12.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.687+0000 7fa6017c5640 1 --2- 172.21.15.125:0/763338163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5fc0ca0e0 0x7fa5fc0b4950 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec0b3040 tx=0x7fa5ec01b9d0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.687+0000 7fa6017c5640 1 -- 172.21.15.125:0/763338163 shutdown_connections 2023-10-30T12:49:12.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.687+0000 7fa6017c5640 1 --2- 172.21.15.125:0/763338163 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa5dc0664a0 0x7fa5dc068960 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.687+0000 7fa6017c5640 1 --2- 172.21.15.125:0/763338163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa5fc1538a0 0x7fa5fc1f2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.687+0000 7fa6017c5640 1 --2- 172.21.15.125:0/763338163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5fc0cf160 0x7fa5fc147780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.687+0000 7fa6017c5640 1 --2- 172.21.15.125:0/763338163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5fc0ca0e0 0x7fa5fc0b4950 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:12.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.687+0000 7fa6017c5640 1 -- 172.21.15.125:0/763338163 >> 172.21.15.125:0/763338163 conn(0x7fa5fc06bfe0 msgr2=0x7fa5fc06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:12.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.687+0000 7fa6017c5640 1 -- 172.21.15.125:0/763338163 shutdown_connections 2023-10-30T12:49:12.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:12.687+0000 7fa6017c5640 1 -- 172.21.15.125:0/763338163 wait complete. 2023-10-30T12:49:12.705 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T12:49:12.706 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6223": { "addr": "172.21.15.136:6839/3479573317", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3479573317, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3479573317, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6223, "incarnation": 100, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:stopping", "state_seq": 16 }, "gid_6318": { "addr": "172.21.15.125:6836/1721514617", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1721514617, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1721514617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 6318, "incarnation": 92, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:49:12.327796+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6318, "mds_1": 6223 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6837/2408701650", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2408701650, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2408701650, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 105, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6627, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:49:14.707 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:14.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.851+0000 7f4c32cea640 1 Processor -- start 2023-10-30T12:49:14.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.851+0000 7f4c32cea640 1 -- start start 2023-10-30T12:49:14.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c32cea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c2c0c6f60 0x7f4c2c1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:14.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c32cea640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c0c0a50 con 0x7f4c2c0c05d0 2023-10-30T12:49:14.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c32cea640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c2c06f6e0 con 0x7f4c2c0c6b80 2023-10-30T12:49:14.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c32cea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c06f9c0 con 0x7f4c2c0c6f60 2023-10-30T12:49:14.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c23fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4c2c0c05d0 0x7f4c2c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60672/0 (socket says 172.21.15.125:60672) 2023-10-30T12:49:14.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c23fff640 1 -- 172.21.15.125:0/1430652950 learned_addr learned my addr 172.21.15.125:0/1430652950 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:49:14.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c31287640 1 --2- 172.21.15.125:0/1430652950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c2c0c6f60 0x7f4c2c1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:14.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 -- 172.21.15.125:0/1430652950 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081807246 0 0) 0x7f4c2c0c0a50 con 0x7f4c2c0c05d0 2023-10-30T12:49:14.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 -- 172.21.15.125:0/1430652950 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c14005740 con 0x7f4c2c0c05d0 2023-10-30T12:49:14.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 -- 172.21.15.125:0/1430652950 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2795607993 0 0) 0x7f4c14005740 con 0x7f4c2c0c05d0 2023-10-30T12:49:14.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 -- 172.21.15.125:0/1430652950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c2c0c6f60 msgr2=0x7f4c2c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:14.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 --2- 172.21.15.125:0/1430652950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c2c0c6f60 0x7f4c2c1566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:14.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 -- 172.21.15.125:0/1430652950 >> v1:172.21.15.136:6789/0 conn(0x7f4c2c0c6b80 legacy=0x7f4c2c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:14.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 -- 172.21.15.125:0/1430652950 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c06fca0 con 0x7f4c2c0c05d0 2023-10-30T12:49:14.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 -- 172.21.15.125:0/1430652950 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f4c1c001680 con 0x7f4c2c0c05d0 2023-10-30T12:49:14.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 -- 172.21.15.125:0/1430652950 >> v1:172.21.15.125:6789/0 conn(0x7f4c2c0c05d0 legacy=0x7f4c2c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:14.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 --2- 172.21.15.125:0/1430652950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c14005bc0 0x7f4c14005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:14.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 --2- 172.21.15.125:0/1430652950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c14006570 0x7f4c14006a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:14.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 --2- 172.21.15.125:0/1430652950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c14007240 0x7f4c1400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:14.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 -- 172.21.15.125:0/1430652950 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c06f9c0 con 0x7f4c14005bc0 2023-10-30T12:49:14.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 -- 172.21.15.125:0/1430652950 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1400e110 con 0x7f4c14006570 2023-10-30T12:49:14.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c237fe640 1 -- 172.21.15.125:0/1430652950 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c1400e2d0 con 0x7f4c14007240 2023-10-30T12:49:14.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c31287640 1 --2- 172.21.15.125:0/1430652950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c2c0c6f60 0x7f4c2c1566a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:49:14.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c30a86640 1 --2- 172.21.15.125:0/1430652950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c14005bc0 0x7f4c14005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:14.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c23fff640 1 --2- 172.21.15.125:0/1430652950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c14006570 0x7f4c14006a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:14.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c31287640 1 --2- 172.21.15.125:0/1430652950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c14007240 0x7f4c1400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:14.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c30a86640 1 -- 172.21.15.125:0/1430652950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c14007240 msgr2=0x7f4c1400dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:14.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c30a86640 1 --2- 172.21.15.125:0/1430652950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c14007240 0x7f4c1400dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:14.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c30a86640 1 -- 172.21.15.125:0/1430652950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c14006570 msgr2=0x7f4c14006a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:14.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c30a86640 1 --2- 172.21.15.125:0/1430652950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c14006570 0x7f4c14006a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:14.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c30a86640 1 -- 172.21.15.125:0/1430652950 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c14005740 con 0x7f4c14005bc0 2023-10-30T12:49:14.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.855+0000 7f4c31287640 1 --2- 172.21.15.125:0/1430652950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c14007240 0x7f4c1400dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:49:14.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.859+0000 7f4c30a86640 1 --2- 172.21.15.125:0/1430652950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c14005bc0 0x7f4c14005fa0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f4c28001860 tx=0x7f4c2801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=412ba36aab3e05bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:14.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.859+0000 7f4c237fe640 1 -- 172.21.15.125:0/1430652950 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c280ab600 con 0x7f4c14005bc0 2023-10-30T12:49:14.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.859+0000 7f4c237fe640 1 -- 172.21.15.125:0/1430652950 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c280a7050 con 0x7f4c14005bc0 2023-10-30T12:49:14.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.859+0000 7f4c237fe640 1 -- 172.21.15.125:0/1430652950 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c280ab9a0 con 0x7f4c14005bc0 2023-10-30T12:49:14.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.859+0000 7f4c32cea640 1 -- 172.21.15.125:0/1430652950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c14005bc0 msgr2=0x7f4c14005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:14.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.859+0000 7f4c32cea640 1 --2- 172.21.15.125:0/1430652950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c14005bc0 0x7f4c14005fa0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f4c28001860 tx=0x7f4c2801c500 comp rx=0 tx=0).stop 2023-10-30T12:49:14.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.859+0000 7f4c31287640 1 -- 172.21.15.125:0/1430652950 reap_dead start 2023-10-30T12:49:14.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.859+0000 7f4c32cea640 1 -- 172.21.15.125:0/1430652950 shutdown_connections 2023-10-30T12:49:14.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.859+0000 7f4c32cea640 1 -- 172.21.15.125:0/1430652950 >> 172.21.15.125:0/1430652950 conn(0x7f4c2c0c7910 msgr2=0x7f4c2c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:14.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.859+0000 7f4c32cea640 1 -- 172.21.15.125:0/1430652950 shutdown_connections 2023-10-30T12:49:14.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.859+0000 7f4c32cea640 1 -- 172.21.15.125:0/1430652950 wait complete. 2023-10-30T12:49:14.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.859+0000 7f4c32cea640 1 Processor -- start 2023-10-30T12:49:14.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c32cea640 1 -- start start 2023-10-30T12:49:14.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c32cea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c14005bc0 0x7f4c2c158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:14.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c32cea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c2c155670 0x7f4c2c1f0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:14.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c32cea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c2c1533f0 0x7f4c2c14ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:14.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c32cea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c155250 con 0x7f4c2c1533f0 2023-10-30T12:49:14.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c32cea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c1599c0 con 0x7f4c2c155670 2023-10-30T12:49:14.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c32cea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c2c158df0 con 0x7f4c14005bc0 2023-10-30T12:49:14.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c31287640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c2c1533f0 0x7f4c2c14ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:14.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c31287640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c2c1533f0 0x7f4c2c14ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36718/0 (socket says 172.21.15.125:36718) 2023-10-30T12:49:14.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c31287640 1 -- 172.21.15.125:0/2028915731 learned_addr learned my addr 172.21.15.125:0/2028915731 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:14.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c30a86640 1 --2- 172.21.15.125:0/2028915731 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c14005bc0 0x7f4c2c158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:14.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c23fff640 1 --2- 172.21.15.125:0/2028915731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c2c155670 0x7f4c2c1f0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:14.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c31287640 1 -- 172.21.15.125:0/2028915731 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c14005bc0 msgr2=0x7f4c2c158760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c31287640 1 --2- 172.21.15.125:0/2028915731 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c14005bc0 0x7f4c2c158760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c31287640 1 -- 172.21.15.125:0/2028915731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c2c155670 msgr2=0x7f4c2c1f0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c31287640 1 --2- 172.21.15.125:0/2028915731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c2c155670 0x7f4c2c1f0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c31287640 1 -- 172.21.15.125:0/2028915731 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c2c1f1330 con 0x7f4c2c1533f0 2023-10-30T12:49:14.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c31287640 1 --2- 172.21.15.125:0/2028915731 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c2c1533f0 0x7f4c2c14ef40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0c8980 tx=0x7f4c2400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:14.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c217fa640 1 -- 172.21.15.125:0/2028915731 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24017600 con 0x7f4c2c1533f0 2023-10-30T12:49:14.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c217fa640 1 -- 172.21.15.125:0/2028915731 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c24011040 con 0x7f4c2c1533f0 2023-10-30T12:49:14.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c217fa640 1 -- 172.21.15.125:0/2028915731 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24017980 con 0x7f4c2c1533f0 2023-10-30T12:49:14.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c32cea640 1 -- 172.21.15.125:0/2028915731 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c2c159120 con 0x7f4c2c1533f0 2023-10-30T12:49:14.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c32cea640 1 -- 172.21.15.125:0/2028915731 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c2c1596e0 con 0x7f4c2c1533f0 2023-10-30T12:49:14.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.863+0000 7f4c32cea640 1 -- 172.21.15.125:0/2028915731 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c2c155390 con 0x7f4c2c1533f0 2023-10-30T12:49:14.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.867+0000 7f4c217fa640 1 -- 172.21.15.125:0/2028915731 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4c24017ae0 con 0x7f4c2c1533f0 2023-10-30T12:49:14.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.867+0000 7f4c217fa640 1 --2- 172.21.15.125:0/2028915731 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4c180664d0 0x7f4c18068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:14.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.867+0000 7f4c217fa640 1 -- 172.21.15.125:0/2028915731 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f4c24084470 con 0x7f4c2c1533f0 2023-10-30T12:49:14.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.867+0000 7f4c30a86640 1 --2- 172.21.15.125:0/2028915731 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4c180664d0 0x7f4c18068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:14.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.867+0000 7f4c30a86640 1 --2- 172.21.15.125:0/2028915731 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4c180664d0 0x7f4c18068990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4c280ad000 tx=0x7f4c280a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:14.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:14.871+0000 7f4c217fa640 1 -- 172.21.15.125:0/2028915731 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4c2c155390 con 0x7f4c2c1533f0 2023-10-30T12:49:15.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:15.051+0000 7f4c32cea640 1 -- 172.21.15.125:0/2028915731 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c2c155390 con 0x7f4c2c1533f0 2023-10-30T12:49:15.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:15.051+0000 7f4c217fa640 1 -- 172.21.15.125:0/2028915731 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7f4c2c155390 con 0x7f4c2c1533f0 2023-10-30T12:49:15.050 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:15.050 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:15.051 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:15.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:15.051+0000 7f4c32cea640 1 -- 172.21.15.125:0/2028915731 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4c180664d0 msgr2=0x7f4c18068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:15.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:15.051+0000 7f4c32cea640 1 --2- 172.21.15.125:0/2028915731 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4c180664d0 0x7f4c18068990 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f4c280ad000 tx=0x7f4c280a6000 comp rx=0 tx=0).stop 2023-10-30T12:49:15.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:15.051+0000 7f4c32cea640 1 -- 172.21.15.125:0/2028915731 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c2c1533f0 msgr2=0x7f4c2c14ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:15.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:15.051+0000 7f4c32cea640 1 --2- 172.21.15.125:0/2028915731 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c2c1533f0 0x7f4c2c14ef40 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c0c8980 tx=0x7f4c2400f790 comp rx=0 tx=0).stop 2023-10-30T12:49:15.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:15.051+0000 7f4c32cea640 1 -- 172.21.15.125:0/2028915731 shutdown_connections 2023-10-30T12:49:15.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:15.051+0000 7f4c32cea640 1 --2- 172.21.15.125:0/2028915731 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4c180664d0 0x7f4c18068990 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:15.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:15.051+0000 7f4c32cea640 1 --2- 172.21.15.125:0/2028915731 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4c2c1533f0 0x7f4c2c14ef40 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:15.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:15.051+0000 7f4c32cea640 1 --2- 172.21.15.125:0/2028915731 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4c2c155670 0x7f4c2c1f0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:15.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:15.051+0000 7f4c32cea640 1 --2- 172.21.15.125:0/2028915731 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4c14005bc0 0x7f4c2c158760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:15.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:15.051+0000 7f4c32cea640 1 -- 172.21.15.125:0/2028915731 >> 172.21.15.125:0/2028915731 conn(0x7f4c2c0c7910 msgr2=0x7f4c2c1362b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:15.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:15.055+0000 7f4c32cea640 1 -- 172.21.15.125:0/2028915731 shutdown_connections 2023-10-30T12:49:15.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:15.055+0000 7f4c32cea640 1 -- 172.21.15.125:0/2028915731 wait complete. 2023-10-30T12:49:15.070 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:17.071 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:17.132 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:49:17.136 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:49:17.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.219+0000 7f6294846640 1 Processor -- start 2023-10-30T12:49:17.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.223+0000 7f6294846640 1 -- start start 2023-10-30T12:49:17.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.223+0000 7f6294846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62900c6b80 0x7f62900bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:17.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.223+0000 7f6294846640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f629013f4a0 0x7f62900c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:17.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.223+0000 7f6294846640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62900c0a50 con 0x7f62900c6f60 2023-10-30T12:49:17.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.223+0000 7f6294846640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f629006f6e0 con 0x7f629013f4a0 2023-10-30T12:49:17.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.223+0000 7f6294846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629006f9c0 con 0x7f62900c6b80 2023-10-30T12:49:17.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.223+0000 7f628ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f629013f4a0 0x7f62900c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:17.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.223+0000 7f628ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f629013f4a0 0x7f62900c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36734/0 (socket says 172.21.15.125:36734) 2023-10-30T12:49:17.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.223+0000 7f628ed9d640 1 -- 172.21.15.125:0/4246031308 learned_addr learned my addr 172.21.15.125:0/4246031308 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:17.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.223+0000 7f628e59c640 1 --2- 172.21.15.125:0/4246031308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62900c6b80 0x7f62900bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:17.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.223+0000 7f628ed9d640 1 -- 172.21.15.125:0/4246031308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62900c6b80 msgr2=0x7f62900bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:17.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.223+0000 7f628ed9d640 1 --2- 172.21.15.125:0/4246031308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62900c6b80 0x7f62900bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:17.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.223+0000 7f628ed9d640 1 -- 172.21.15.125:0/4246031308 >> v1:172.21.15.136:6789/0 conn(0x7f62900c6f60 legacy=0x7f629013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:17.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.223+0000 7f628ed9d640 1 -- 172.21.15.125:0/4246031308 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f629006fca0 con 0x7f629013f4a0 2023-10-30T12:49:17.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.227+0000 7f628ed9d640 1 --2- 172.21.15.125:0/4246031308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f629013f4a0 0x7f62900c0400 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f6288009e50 tx=0x7f628801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3435b61493f1a59 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:17.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.227+0000 7f628dd9b640 1 -- 172.21.15.125:0/4246031308 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62880a57f0 con 0x7f629013f4a0 2023-10-30T12:49:17.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.227+0000 7f628dd9b640 1 -- 172.21.15.125:0/4246031308 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62880a5950 con 0x7f629013f4a0 2023-10-30T12:49:17.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.227+0000 7f628dd9b640 1 -- 172.21.15.125:0/4246031308 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62880a5ce0 con 0x7f629013f4a0 2023-10-30T12:49:17.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.227+0000 7f6294846640 1 -- 172.21.15.125:0/4246031308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f629013f4a0 msgr2=0x7f62900c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:17.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.227+0000 7f6294846640 1 --2- 172.21.15.125:0/4246031308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f629013f4a0 0x7f62900c0400 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f6288009e50 tx=0x7f628801ca10 comp rx=0 tx=0).stop 2023-10-30T12:49:17.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.227+0000 7f6294846640 1 -- 172.21.15.125:0/4246031308 shutdown_connections 2023-10-30T12:49:17.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.227+0000 7f6294846640 1 --2- 172.21.15.125:0/4246031308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f629013f4a0 0x7f62900c0400 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:17.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.227+0000 7f6294846640 1 --2- 172.21.15.125:0/4246031308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62900c6b80 0x7f62900bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:17.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.227+0000 7f6294846640 1 -- 172.21.15.125:0/4246031308 >> 172.21.15.125:0/4246031308 conn(0x7f62900c7910 msgr2=0x7f62900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:17.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.227+0000 7f6294846640 1 -- 172.21.15.125:0/4246031308 shutdown_connections 2023-10-30T12:49:17.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.227+0000 7f6294846640 1 -- 172.21.15.125:0/4246031308 wait complete. 2023-10-30T12:49:17.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.227+0000 7f6294846640 1 Processor -- start 2023-10-30T12:49:17.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.227+0000 7f6294846640 1 -- start start 2023-10-30T12:49:17.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f6294846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62900c6b80 0x7f62901392b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:17.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f6294846640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62900c6f60 0x7f629006a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:17.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f6294846640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f629013f4a0 0x7f62900c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:17.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f6294846640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6290152640 con 0x7f629013f4a0 2023-10-30T12:49:17.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f6294846640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6290151d50 con 0x7f62900c6f60 2023-10-30T12:49:17.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f6294846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62900ca220 con 0x7f62900c6b80 2023-10-30T12:49:17.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f628ed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f629013f4a0 0x7f62900c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:17.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f628ed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f629013f4a0 0x7f62900c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36750/0 (socket says 172.21.15.125:36750) 2023-10-30T12:49:17.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f628ed9d640 1 -- 172.21.15.125:0/2619671981 learned_addr learned my addr 172.21.15.125:0/2619671981 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:17.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f6285d9b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62900c6f60 0x7f629006a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:17.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f628e59c640 1 --2- 172.21.15.125:0/2619671981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62900c6b80 0x7f62901392b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:17.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f628ed9d640 1 -- 172.21.15.125:0/2619671981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62900c6b80 msgr2=0x7f62901392b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:17.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f628ed9d640 1 --2- 172.21.15.125:0/2619671981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62900c6b80 0x7f62901392b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:17.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f628ed9d640 1 -- 172.21.15.125:0/2619671981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62900c6f60 msgr2=0x7f629006a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:17.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f628ed9d640 1 --2- 172.21.15.125:0/2619671981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62900c6f60 0x7f629006a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:17.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f628ed9d640 1 -- 172.21.15.125:0/2619671981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6288009b30 con 0x7f629013f4a0 2023-10-30T12:49:17.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f628ed9d640 1 --2- 172.21.15.125:0/2619671981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f629013f4a0 0x7f62900c5810 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f628801a640 tx=0x7f62880197b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:17.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f62877fe640 1 -- 172.21.15.125:0/2619671981 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62880b8880 con 0x7f629013f4a0 2023-10-30T12:49:17.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f62877fe640 1 -- 172.21.15.125:0/2619671981 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62880b89e0 con 0x7f629013f4a0 2023-10-30T12:49:17.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f6294846640 1 -- 172.21.15.125:0/2619671981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6290151ff0 con 0x7f629013f4a0 2023-10-30T12:49:17.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f62877fe640 1 -- 172.21.15.125:0/2619671981 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62880b8d90 con 0x7f629013f4a0 2023-10-30T12:49:17.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f6294846640 1 -- 172.21.15.125:0/2619671981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6290151800 con 0x7f629013f4a0 2023-10-30T12:49:17.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.231+0000 7f6294846640 1 -- 172.21.15.125:0/2619671981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62901527b0 con 0x7f629013f4a0 2023-10-30T12:49:17.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.235+0000 7f62877fe640 1 -- 172.21.15.125:0/2619671981 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f62880c3460 con 0x7f629013f4a0 2023-10-30T12:49:17.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.235+0000 7f62877fe640 1 --2- 172.21.15.125:0/2619671981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62700664a0 0x7f6270068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:17.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.235+0000 7f62877fe640 1 -- 172.21.15.125:0/2619671981 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f628811fa20 con 0x7f629013f4a0 2023-10-30T12:49:17.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.235+0000 7f628e59c640 1 --2- 172.21.15.125:0/2619671981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62700664a0 0x7f6270068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:17.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.235+0000 7f628e59c640 1 --2- 172.21.15.125:0/2619671981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62700664a0 0x7f6270068960 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6280006aa0 tx=0x7f6280008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:17.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.239+0000 7f62877fe640 1 -- 172.21.15.125:0/2619671981 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f62880ecc50 con 0x7f629013f4a0 2023-10-30T12:49:17.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.455+0000 7f6294846640 1 -- 172.21.15.125:0/2619671981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f629004e170 con 0x7f629013f4a0 2023-10-30T12:49:17.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.455+0000 7f62877fe640 1 -- 172.21.15.125:0/2619671981 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7f62880f0270 con 0x7f629013f4a0 2023-10-30T12:49:17.454 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:17.454 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:17.454 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:17.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.459+0000 7f6294846640 1 -- 172.21.15.125:0/2619671981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62700664a0 msgr2=0x7f6270068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:17.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.459+0000 7f6294846640 1 --2- 172.21.15.125:0/2619671981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62700664a0 0x7f6270068960 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6280006aa0 tx=0x7f6280008040 comp rx=0 tx=0).stop 2023-10-30T12:49:17.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.459+0000 7f6294846640 1 -- 172.21.15.125:0/2619671981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f629013f4a0 msgr2=0x7f62900c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:17.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.459+0000 7f6294846640 1 --2- 172.21.15.125:0/2619671981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f629013f4a0 0x7f62900c5810 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f628801a640 tx=0x7f62880197b0 comp rx=0 tx=0).stop 2023-10-30T12:49:17.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.459+0000 7f6294846640 1 -- 172.21.15.125:0/2619671981 shutdown_connections 2023-10-30T12:49:17.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.459+0000 7f6294846640 1 --2- 172.21.15.125:0/2619671981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62700664a0 0x7f6270068960 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:17.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.459+0000 7f6294846640 1 --2- 172.21.15.125:0/2619671981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f629013f4a0 0x7f62900c5810 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:17.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.459+0000 7f6294846640 1 --2- 172.21.15.125:0/2619671981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62900c6f60 0x7f629006a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:17.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.459+0000 7f6294846640 1 --2- 172.21.15.125:0/2619671981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62900c6b80 0x7f62901392b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:17.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.459+0000 7f6294846640 1 -- 172.21.15.125:0/2619671981 >> 172.21.15.125:0/2619671981 conn(0x7f62900c7910 msgr2=0x7f62900b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:17.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.459+0000 7f6294846640 1 -- 172.21.15.125:0/2619671981 shutdown_connections 2023-10-30T12:49:17.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:17.459+0000 7f6294846640 1 -- 172.21.15.125:0/2619671981 wait complete. 2023-10-30T12:49:17.477 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:19.477 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:19.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.619+0000 7f2f01b9d640 1 Processor -- start 2023-10-30T12:49:19.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.619+0000 7f2f01b9d640 1 -- start start 2023-10-30T12:49:19.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.623+0000 7f2f01b9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2efc0c6bd0 0x7f2efc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:19.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.623+0000 7f2f01b9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc071110 0x7f2efc158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:19.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.623+0000 7f2f01b9d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2efc066cf0 con 0x7f2efc06f7a0 2023-10-30T12:49:19.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.623+0000 7f2f01b9d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2efc0700c0 con 0x7f2efc0c6bd0 2023-10-30T12:49:19.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.623+0000 7f2f01b9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0c70d0 con 0x7f2efc071110 2023-10-30T12:49:19.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.623+0000 7f2ef3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2efc0c6bd0 0x7f2efc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:19.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.623+0000 7f2efb7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2efc06f7a0 0x7f2efc069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60686/0 (socket says 172.21.15.125:60686) 2023-10-30T12:49:19.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.623+0000 7f2efb7fe640 1 -- 172.21.15.125:0/1029871877 learned_addr learned my addr 172.21.15.125:0/1029871877 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:49:19.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.623+0000 7f2efbfff640 1 --2- 172.21.15.125:0/1029871877 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc071110 0x7f2efc158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:19.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.623+0000 7f2ef3fff640 1 -- 172.21.15.125:0/1029871877 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc071110 msgr2=0x7f2efc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:19.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.623+0000 7f2ef3fff640 1 --2- 172.21.15.125:0/1029871877 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc071110 0x7f2efc158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:19.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.623+0000 7f2ef3fff640 1 -- 172.21.15.125:0/1029871877 >> v1:172.21.15.125:6789/0 conn(0x7f2efc06f7a0 legacy=0x7f2efc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:19.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.623+0000 7f2ef3fff640 1 -- 172.21.15.125:0/1029871877 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc0c7390 con 0x7f2efc0c6bd0 2023-10-30T12:49:19.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.623+0000 7f2efaffd640 1 -- 172.21.15.125:0/1029871877 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392603406 0 0) 0x7f2efc066cf0 con 0x7f2efc06f7a0 2023-10-30T12:49:19.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.627+0000 7f2ef3fff640 1 --2- 172.21.15.125:0/1029871877 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc0c6bd0 0x7f2efc06fb80 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40097c0 tx=0x7f2ee401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fdbdc395740d01b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:19.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.627+0000 7f2efaffd640 1 -- 172.21.15.125:0/1029871877 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee40a8970 con 0x7f2efc0c6bd0 2023-10-30T12:49:19.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.627+0000 7f2efaffd640 1 -- 172.21.15.125:0/1029871877 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee40a8ad0 con 0x7f2efc0c6bd0 2023-10-30T12:49:19.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.627+0000 7f2efaffd640 1 -- 172.21.15.125:0/1029871877 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee40a8e60 con 0x7f2efc0c6bd0 2023-10-30T12:49:19.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.627+0000 7f2f01b9d640 1 -- 172.21.15.125:0/1029871877 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc0c6bd0 msgr2=0x7f2efc06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:19.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.627+0000 7f2f01b9d640 1 --2- 172.21.15.125:0/1029871877 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc0c6bd0 0x7f2efc06fb80 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f2ee40097c0 tx=0x7f2ee401ec70 comp rx=0 tx=0).stop 2023-10-30T12:49:19.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.627+0000 7f2f01b9d640 1 -- 172.21.15.125:0/1029871877 shutdown_connections 2023-10-30T12:49:19.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.627+0000 7f2f01b9d640 1 --2- 172.21.15.125:0/1029871877 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc0c6bd0 0x7f2efc06fb80 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:19.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.627+0000 7f2f01b9d640 1 --2- 172.21.15.125:0/1029871877 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc071110 0x7f2efc158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:19.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.627+0000 7f2f01b9d640 1 -- 172.21.15.125:0/1029871877 >> 172.21.15.125:0/1029871877 conn(0x7f2efc0c7910 msgr2=0x7f2efc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:19.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.627+0000 7f2f01b9d640 1 -- 172.21.15.125:0/1029871877 shutdown_connections 2023-10-30T12:49:19.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.627+0000 7f2f01b9d640 1 -- 172.21.15.125:0/1029871877 wait complete. 2023-10-30T12:49:19.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.627+0000 7f2f01b9d640 1 Processor -- start 2023-10-30T12:49:19.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.627+0000 7f2f01b9d640 1 -- start start 2023-10-30T12:49:19.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2f01b9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc06f7a0 0x7f2efc1eced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:19.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2f01b9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc071110 0x7f2efc0aafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:19.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2f01b9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0c6bd0 0x7f2efc0c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:19.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2f01b9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc1ed410 con 0x7f2efc071110 2023-10-30T12:49:19.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2f01b9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc154d80 con 0x7f2efc06f7a0 2023-10-30T12:49:19.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2f01b9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0a9560 con 0x7f2efc0c6bd0 2023-10-30T12:49:19.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2ef3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc071110 0x7f2efc0aafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:19.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2ef3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc071110 0x7f2efc0aafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36772/0 (socket says 172.21.15.125:36772) 2023-10-30T12:49:19.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2ef3fff640 1 -- 172.21.15.125:0/3473167062 learned_addr learned my addr 172.21.15.125:0/3473167062 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:19.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2efbfff640 1 --2- 172.21.15.125:0/3473167062 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0c6bd0 0x7f2efc0c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:19.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2efb7fe640 1 --2- 172.21.15.125:0/3473167062 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc06f7a0 0x7f2efc1eced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:19.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2ef3fff640 1 -- 172.21.15.125:0/3473167062 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0c6bd0 msgr2=0x7f2efc0c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:19.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2ef3fff640 1 --2- 172.21.15.125:0/3473167062 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0c6bd0 0x7f2efc0c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:19.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2ef3fff640 1 -- 172.21.15.125:0/3473167062 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc06f7a0 msgr2=0x7f2efc1eced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:19.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2ef3fff640 1 --2- 172.21.15.125:0/3473167062 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc06f7a0 0x7f2efc1eced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:19.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2ef3fff640 1 -- 172.21.15.125:0/3473167062 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee4009420 con 0x7f2efc071110 2023-10-30T12:49:19.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2ef3fff640 1 --2- 172.21.15.125:0/3473167062 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc071110 0x7f2efc0aafe0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f2ee401d320 tx=0x7f2ee401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:19.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2ef8ff9640 1 -- 172.21.15.125:0/3473167062 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee40bb640 con 0x7f2efc071110 2023-10-30T12:49:19.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2ef8ff9640 1 -- 172.21.15.125:0/3473167062 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee40bb7c0 con 0x7f2efc071110 2023-10-30T12:49:19.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2ef8ff9640 1 -- 172.21.15.125:0/3473167062 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee40bbb70 con 0x7f2efc071110 2023-10-30T12:49:19.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2f01b9d640 1 -- 172.21.15.125:0/3473167062 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2efc0a9840 con 0x7f2efc071110 2023-10-30T12:49:19.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2f01b9d640 1 -- 172.21.15.125:0/3473167062 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2efc0a9e00 con 0x7f2efc071110 2023-10-30T12:49:19.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2f01b9d640 1 -- 172.21.15.125:0/3473167062 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2efc04e170 con 0x7f2efc071110 2023-10-30T12:49:19.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.631+0000 7f2ef8ff9640 1 -- 172.21.15.125:0/3473167062 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ee40b7040 con 0x7f2efc071110 2023-10-30T12:49:19.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.635+0000 7f2ef8ff9640 1 --2- 172.21.15.125:0/3473167062 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2edc0664d0 0x7f2edc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:19.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.635+0000 7f2ef8ff9640 1 -- 172.21.15.125:0/3473167062 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f2ee4122550 con 0x7f2efc071110 2023-10-30T12:49:19.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.635+0000 7f2efb7fe640 1 --2- 172.21.15.125:0/3473167062 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2edc0664d0 0x7f2edc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:19.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.635+0000 7f2efb7fe640 1 --2- 172.21.15.125:0/3473167062 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2edc0664d0 0x7f2edc068990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2eec000f70 tx=0x7f2eec007c10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:19.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.635+0000 7f2ef8ff9640 1 -- 172.21.15.125:0/3473167062 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2ee40ef640 con 0x7f2efc071110 2023-10-30T12:49:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.815+0000 7f2f01b9d640 1 -- 172.21.15.125:0/3473167062 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2efc06cbc0 con 0x7f2efc071110 2023-10-30T12:49:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.819+0000 7f2ef8ff9640 1 -- 172.21.15.125:0/3473167062 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7f2ee40f2c60 con 0x7f2efc071110 2023-10-30T12:49:19.816 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:19.817 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.819+0000 7f2f01b9d640 1 -- 172.21.15.125:0/3473167062 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2edc0664d0 msgr2=0x7f2edc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.819+0000 7f2f01b9d640 1 --2- 172.21.15.125:0/3473167062 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2edc0664d0 0x7f2edc068990 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f2eec000f70 tx=0x7f2eec007c10 comp rx=0 tx=0).stop 2023-10-30T12:49:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.819+0000 7f2f01b9d640 1 -- 172.21.15.125:0/3473167062 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc071110 msgr2=0x7f2efc0aafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.819+0000 7f2f01b9d640 1 --2- 172.21.15.125:0/3473167062 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc071110 0x7f2efc0aafe0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f2ee401d320 tx=0x7f2ee401ba10 comp rx=0 tx=0).stop 2023-10-30T12:49:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.819+0000 7f2f01b9d640 1 -- 172.21.15.125:0/3473167062 shutdown_connections 2023-10-30T12:49:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.819+0000 7f2f01b9d640 1 --2- 172.21.15.125:0/3473167062 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2edc0664d0 0x7f2edc068990 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.819+0000 7f2f01b9d640 1 --2- 172.21.15.125:0/3473167062 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0c6bd0 0x7f2efc0c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.819+0000 7f2f01b9d640 1 --2- 172.21.15.125:0/3473167062 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2efc071110 0x7f2efc0aafe0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.819+0000 7f2f01b9d640 1 --2- 172.21.15.125:0/3473167062 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc06f7a0 0x7f2efc1eced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.819+0000 7f2f01b9d640 1 -- 172.21.15.125:0/3473167062 >> 172.21.15.125:0/3473167062 conn(0x7f2efc0c7910 msgr2=0x7f2efc0ca5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.819+0000 7f2f01b9d640 1 -- 172.21.15.125:0/3473167062 shutdown_connections 2023-10-30T12:49:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:19.819+0000 7f2f01b9d640 1 -- 172.21.15.125:0/3473167062 wait complete. 2023-10-30T12:49:19.836 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:21.754 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 0% (95/12861) Updating files: 1% (129/12861) Updating files: 1% (138/12861) Updating files: 1% (168/12861) Updating files: 2% (258/12861) Updating files: 2% (293/12861) Updating files: 2% (294/12861) Updating files: 2% (296/12861) Updating files: 3% (386/12861) Updating files: 3% (505/12861) Updating files: 4% (515/12861) Updating files: 4% (539/12861) Updating files: 4% (587/12861) Updating files: 4% (611/12861) Updating files: 4% (630/12861) Updating files: 4% (637/12861) Updating files: 5% (644/12861) Updating files: 5% (682/12861) Updating files: 5% (694/12861) Updating files: 5% (731/12861) Updating files: 6% (772/12861) Updating files: 6% (797/12861) Updating files: 6% (861/12861) Updating files: 7% (901/12861) Updating files: 7% (912/12861) Updating files: 7% (954/12861) Updating files: 7% (987/12861) Updating files: 8% (1029/12861) Updating files: 8% (1096/12861) Updating files: 8% (1128/12861) Updating files: 9% (1158/12861) Updating files: 9% (1178/12861) Updating files: 9% (1228/12861) Updating files: 9% (1281/12861) Updating files: 10% (1287/12861) Updating files: 10% (1294/12861) Updating files: 10% (1321/12861) Updating files: 10% (1406/12861) Updating files: 10% (1412/12861) Updating files: 11% (1415/12861) Updating files: 11% (1423/12861) Updating files: 11% (1427/12861) Updating files: 11% (1469/12861) Updating files: 11% (1474/12861) Updating files: 11% (1485/12861) Updating files: 11% (1540/12861) Updating files: 12% (1544/12861) Updating files: 12% (1597/12861) Updating files: 13% (1672/12861) Updating files: 13% (1680/12861) Updating files: 13% (1740/12861) Updating files: 13% (1797/12861) Updating files: 14% (1801/12861) Updating files: 14% (1858/12861) Updating files: 14% (1928/12861) Updating files: 15% (1930/12861) Updating files: 15% (1969/12861) Updating files: 15% (2023/12861) Updating files: 15% (2024/12861) Updating files: 16% (2058/12861) Updating files: 16% (2064/12861) Updating files: 16% (2080/12861) Updating files: 16% (2119/12861) Updating files: 16% (2124/12861) Updating files: 16% (2161/12861) Updating files: 16% (2186/12861) Updating files: 17% (2187/12861) Updating files: 17% (2192/12861) Updating files: 17% (2206/12861) Updating files: 17% (2314/12861) Updating files: 18% (2315/12861) Updating files: 18% (2320/12861) Updating files: 18% (2348/12861) Updating files: 18% (2376/12861) Updating files: 19% (2444/12861) Updating files: 19% (2463/12861) Updating files: 19% (2498/12861) Updating files: 19% (2513/12861) Updating files: 20% (2573/12861) Updating files: 20% (2588/12861) Updating files: 20% (2617/12861) Updating files: 21% (2701/12861) Updating files: 21% (2707/12861) Updating files: 21% (2723/12861) Updating files: 21% (2761/12861) Updating files: 21% (2772/12861) Updating files: 22% (2830/12861) Updating files: 22% (2839/12861) Updating files: 22% (2869/12861) Updating files: 22% (2915/12861) Updating files: 22% (2950/12861) Updating files: 23% (2959/12861) Updating files: 23% (2980/12861) Updating files: 23% (3024/12861) Updating files: 23% (3052/12861) Updating files: 24% (3087/12861) Updating files: 24% (3102/12861) Updating files: 25% (3216/12861) Updating files: 26% (3344/12861) Updating files: 26% (3355/12861) Updating files: 26% (3442/12861) Updating files: 26% (3450/12861) Updating files: 26% (3466/12861) Updating files: 27% (3473/12861) Updating files: 27% (3525/12861) Updating files: 27% (3552/12861) Updating files: 28% (3602/12861) Updating files: 28% (3614/12861) Updating files: 28% (3618/12861) Updating files: 28% (3683/12861) Updating files: 29% (3730/12861) Updating files: 29% (3753/12861) Updating files: 29% (3768/12861) Updating files: 29% (3803/12861) Updating files: 29% (3810/12861) Updating files: 30% (3859/12861) Updating files: 30% (3894/12861) Updating files: 30% (3969/12861) Updating files: 31% (3987/12861) Updating files: 31% (4084/12861) Updating files: 32% (4116/12861) Updating files: 32% (4143/12861) Updating files: 32% (4183/12861) Updating files: 32% (4219/12861) Updating files: 33% (4245/12861) Updating files: 33% (4295/12861) Updating files: 33% (4327/12861) Updating files: 34% (4373/12861) Updating files: 34% (4391/12861) Updating files: 34% (4463/12861) Updating files: 35% (4502/12861) Updating files: 35% (4503/12861) Updating files: 35% (4527/12861) Updating files: 35% (4621/12861) Updating files: 36% (4630/12861) Updating files: 36% (4648/12861) Updating files: 36% (4658/12861) Updating files: 36% (4669/12861) Updating files: 36% (4710/12861) Updating files: 37% (4759/12861) Updating files: 37% (4774/12861) Updating files: 37% (4877/12861) Updating files: 38% (4888/12861) Updating files: 38% (5003/12861) Updating files: 39% (5016/12861) Updating files: 39% (5035/12861) Updating files: 39% (5069/12861) Updating files: 40% (5145/12861) Updating files: 40% (5165/12861) Updating files: 40% (5204/12861) Updating files: 40% (5237/12861) Updating files: 40% (5242/12861) Updating files: 40% (5264/12861) Updating files: 41% (5274/12861) Updating files: 41% (5304/12861) Updating files: 41% (5335/12861) Updating files: 42% (5402/12861) Updating files: 43% (5531/12861) Updating files: 43% (5533/12861) Updating files: 43% (5622/12861) Updating files: 44% (5659/12861) Updating files: 44% (5664/12861) Updating files: 44% (5746/12861) Updating files: 45% (5788/12861) Updating files: 45% (5816/12861) Updating files: 45% (5890/12861) Updating files: 46% (5917/12861) Updating files: 46% (5927/12861) Updating files: 46% (6042/12861) Updating files: 47% (6045/12861) Updating files: 47% (6050/12861) Updating files: 47% (6163/12861) Updating files: 48% (6174/12861) Updating files: 49% (6302/12861) Updating files: 49% (6335/12861) Updating files: 49% (6361/12861) Updating files: 49% (6426/12861) Updating files: 50% (6431/12861) Updating files: 50% (6495/12861) Updating files: 51% (6560/12861) Updating files: 51% (6616/12861) Updating files: 51% (6641/12861) Updating files: 52% (6688/12861) Updating files: 52% (6723/12861) Updating files: 53% (6817/12861) Updating files: 53% (6870/12861) Updating files: 53% (6943/12861) Updating files: 54% (6945/12861) Updating files: 54% (7023/12861) Updating files: 55% (7074/12861) Updating files: 56% (7203/12861) Updating files: 56% (7221/12861) Updating files: 56% (7274/12861) Updating files: 56% (7318/12861) Updating files: 57% (7331/12861) Updating files: 57% (7386/12861) Updating files: 58% (7460/12861) Updating files: 58% (7467/12861) Updating files: 59% (7588/12861) Updating files: 59% (7610/12861) Updating files: 59% (7681/12861) Updating files: 60% (7717/12861) Updating files: 60% (7741/12861) Updating files: 60% (7746/12861) Updating files: 60% (7817/12861) Updating files: 61% (7846/12861) Updating files: 61% (7848/12861) Updating files: 61% (7880/12861) Updating files: 61% (7964/12861) Updating files: 62% (7974/12861) Updating files: 62% (8010/12861) Updating files: 62% (8034/12861) Updating files: 62% (8069/12861) Updating files: 62% (8073/12861) Updating files: 62% (8097/12861) Updating files: 62% (8098/12861) Updating files: 63% (8103/12861) Updating files: 63% (8194/12861) Updating files: 64% (8232/12861) Updating files: 64% (8248/12861) Updating files: 65% (8360/12861) Updating files: 65% (8377/12861) Updating files: 65% (8403/12861) Updating files: 65% (8406/12861) Updating files: 65% (8459/12861) Updating files: 65% (8482/12861) Updating files: 65% (8487/12861) Updating files: 66% (8489/12861) Updating files: 66% (8595/12861) Updating files: 67% (8617/12861) Updating files: 67% (8653/12861) Updating files: 68% (8746/12861) Updating files: 68% (8753/12861) Updating files: 69% (8875/12861) Updating files: 69% (8881/12861) Updating files: 69% (8884/12861) Updating files: 69% (8900/12861) Updating files: 69% (8972/12861) Updating files: 69% (8997/12861) Updating files: 70% (9003/12861) Updating files: 71% (9132/12861) Updating files: 71% (9135/12861) Updating files: 71% (9194/12861) Updating files: 72% (9260/12861) Updating files: 72% (9264/12861) Updating files: 72% (9299/12861) Updating files: 73% (9389/12861) Updating files: 73% (9454/12861) Updating files: 74% (9518/12861) Updating files: 74% (9520/12861) Updating files: 74% (9639/12861) Updating files: 75% (9646/12861) Updating files: 75% (9676/12861) Updating files: 75% (9737/12861) Updating files: 76% (9775/12861) Updating files: 76% (9791/12861) Updating files: 76% (9876/12861) Updating files: 77% (9903/12861) Updating files: 77% (9920/12861) Updating files: 77% (9928/12861) Updating files: 77% (9972/12861) Updating files: 78% (10032/12861) Updating files: 78% (10051/12861) Updating files: 78% (10075/12861) Updating files: 78% (10116/12861) Updating files: 79% (10161/12861) Updating files: 79% (10162/12861) Updating files: 79% (10263/12861) Updating files: 80% (10289/12861) Updating files: 80% (10300/12861) Updating files: 80% (10322/12861) Updating files: 80% (10331/12861) Updating files: 80% (10345/12861) Updating files: 81% (10418/12861) Updating files: 81% (10465/12861) Updating files: 81% (10479/12861) Updating files: 81% (10542/12861) Updating files: 82% (10547/12861) Updating files: 82% (10644/12861) Updating files: 83% (10675/12861) Updating files: 84% (10804/12861) Updating files: 84% (10819/12861) Updating files: 84% (10895/12861) Updating files: 85% (10932/12861) Updating files: 85% (10953/12861) Updating files: 85% (11029/12861) Updating files: 86% (11061/12861) Updating files: 86% (11183/12861) Updating files: 87% (11190/12861) Updating files: 88% (11318/12861) Updating files: 88% (11358/12861) Updating files: 88% (11442/12861) Updating files: 89% (11447/12861) Updating files: 89% (11448/12861) Updating files: 90% (11575/12861) Updating files: 90% (11580/12861) Updating files: 90% (11626/12861) Updating files: 90% (11638/12861) Updating files: 91% (11704/12861) Updating files: 91% (11719/12861) Updating files: 92% (11833/12861) Updating files: 92% (11847/12861) Updating files: 92% (11874/12861) Updating files: 93% (11961/12861) Updating files: 93% (11970/12861) Updating files: 93% (12047/12861) Updating files: 94% (12090/12861) Updating files: 95% (12218/12861) Updating files: 95% (12316/12861) Updating files: 96% (12347/12861) Updating files: 96% (12413/12861) Updating files: 96% (12427/12861) Updating files: 96% (12435/12861) Updating files: 96% (12440/12861) Updating files: 96% (12461/12861) Updating files: 97% (12476/12861) Updating files: 97% (12508/12861) Updating files: 98% (12604/12861) Updating files: 99% (12733/12861) Updating files: 99% (12735/12861) Updating files: 100% (12861/12861) Updating files: 100% (12861/12861), done. 2023-10-30T12:49:21.838 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:21.881 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 204c175eea3 msg: blobhash on entity_name_t causes strange badness... should look into that, but avoid for now 2023-10-30T12:49:21.889 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.1 2023-10-30T12:49:21.923 INFO:tasks.workunit.client.0.smithi125.stdout:v0.2 2023-10-30T12:49:21.924 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 2 -eq 48 ] 2023-10-30T12:49:21.924 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.2 2023-10-30T12:49:21.924 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.2 2023-10-30T12:49:21.925 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.2 2023-10-30T12:49:21.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.995+0000 7f7e80580640 1 Processor -- start 2023-10-30T12:49:21.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.995+0000 7f7e80580640 1 -- start start 2023-10-30T12:49:21.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.995+0000 7f7e80580640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7e7806f7a0 0x7f7e7806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:21.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.995+0000 7f7e80580640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e78066cf0 con 0x7f7e78071110 2023-10-30T12:49:21.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.995+0000 7f7e80580640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e780c70d0 con 0x7f7e780c6bd0 2023-10-30T12:49:21.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.995+0000 7f7e80580640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e780700c0 con 0x7f7e7806f7a0 2023-10-30T12:49:21.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.995+0000 7f7e7eb1d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7e78071110 0x7f7e78158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43590/0 (socket says 172.21.15.125:43590) 2023-10-30T12:49:21.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.995+0000 7f7e7eb1d640 1 -- 172.21.15.125:0/3329313806 learned_addr learned my addr 172.21.15.125:0/3329313806 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:49:21.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.995+0000 7f7e7e31c640 1 --2- 172.21.15.125:0/3329313806 >> v2:172.21.15.136:3300/0 conn(0x7f7e7806f7a0 0x7f7e7806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 -- 172.21.15.125:0/3329313806 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854038446 0 0) 0x7f7e78066cf0 con 0x7f7e78071110 2023-10-30T12:49:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 -- 172.21.15.125:0/3329313806 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e64003530 con 0x7f7e78071110 2023-10-30T12:49:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 -- 172.21.15.125:0/3329313806 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1505302492 0 0) 0x7f7e64003530 con 0x7f7e78071110 2023-10-30T12:49:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 -- 172.21.15.125:0/3329313806 >> v2:172.21.15.136:3300/0 conn(0x7f7e7806f7a0 msgr2=0x7f7e7806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:21.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 --2- 172.21.15.125:0/3329313806 >> v2:172.21.15.136:3300/0 conn(0x7f7e7806f7a0 0x7f7e7806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:21.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 -- 172.21.15.125:0/3329313806 >> v1:172.21.15.136:6789/0 conn(0x7f7e780c6bd0 legacy=0x7f7e78069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:21.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 -- 172.21.15.125:0/3329313806 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e780c7390 con 0x7f7e78071110 2023-10-30T12:49:21.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 -- 172.21.15.125:0/3329313806 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f7e70001a00 con 0x7f7e78071110 2023-10-30T12:49:21.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 -- 172.21.15.125:0/3329313806 >> v1:172.21.15.125:6789/0 conn(0x7f7e78071110 legacy=0x7f7e78158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 --2- 172.21.15.125:0/3329313806 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e64003980 0x7f7e64003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 --2- 172.21.15.125:0/3329313806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e640047d0 0x7f7e64004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 --2- 172.21.15.125:0/3329313806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e64005190 0x7f7e6400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 -- 172.21.15.125:0/3329313806 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e780700c0 con 0x7f7e640047d0 2023-10-30T12:49:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 -- 172.21.15.125:0/3329313806 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6400c040 con 0x7f7e64005190 2023-10-30T12:49:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7d31a640 1 -- 172.21.15.125:0/3329313806 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e6400c200 con 0x7f7e64003980 2023-10-30T12:49:22.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7db1b640 1 --2- 172.21.15.125:0/3329313806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e640047d0 0x7f7e64004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:22.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7e31c640 1 --2- 172.21.15.125:0/3329313806 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e64003980 0x7f7e64003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:22.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7eb1d640 1 --2- 172.21.15.125:0/3329313806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e64005190 0x7f7e6400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:22.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7db1b640 1 -- 172.21.15.125:0/3329313806 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e64003980 msgr2=0x7f7e64003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:22.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7db1b640 1 --2- 172.21.15.125:0/3329313806 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e64003980 0x7f7e64003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:22.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7db1b640 1 -- 172.21.15.125:0/3329313806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e64005190 msgr2=0x7f7e6400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:22.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7db1b640 1 --2- 172.21.15.125:0/3329313806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e64005190 0x7f7e6400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:22.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7db1b640 1 -- 172.21.15.125:0/3329313806 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e64003530 con 0x7f7e640047d0 2023-10-30T12:49:22.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7eb1d640 1 --2- 172.21.15.125:0/3329313806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e64005190 0x7f7e6400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:22.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:21.999+0000 7f7e7eb1d640 1 -- 172.21.15.125:0/3329313806 reap_dead start 2023-10-30T12:49:22.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.003+0000 7f7e7db1b640 1 --2- 172.21.15.125:0/3329313806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e640047d0 0x7f7e64004c50 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7e68001750 tx=0x7f7e6801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=844eed3da2395f89 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:22.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.003+0000 7f7e7d31a640 1 -- 172.21.15.125:0/3329313806 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e680052d0 con 0x7f7e640047d0 2023-10-30T12:49:22.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.003+0000 7f7e7d31a640 1 -- 172.21.15.125:0/3329313806 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e680a7050 con 0x7f7e640047d0 2023-10-30T12:49:22.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.003+0000 7f7e7d31a640 1 -- 172.21.15.125:0/3329313806 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e68003600 con 0x7f7e640047d0 2023-10-30T12:49:22.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.003+0000 7f7e80580640 1 -- 172.21.15.125:0/3329313806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e640047d0 msgr2=0x7f7e64004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:22.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.003+0000 7f7e80580640 1 --2- 172.21.15.125:0/3329313806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e640047d0 0x7f7e64004c50 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f7e68001750 tx=0x7f7e6801c3a0 comp rx=0 tx=0).stop 2023-10-30T12:49:22.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.003+0000 7f7e80580640 1 -- 172.21.15.125:0/3329313806 shutdown_connections 2023-10-30T12:49:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.003+0000 7f7e80580640 1 --2- 172.21.15.125:0/3329313806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e640047d0 0x7f7e64004c50 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.003+0000 7f7e80580640 1 -- 172.21.15.125:0/3329313806 >> 172.21.15.125:0/3329313806 conn(0x7f7e780c7910 msgr2=0x7f7e780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.003+0000 7f7e80580640 1 -- 172.21.15.125:0/3329313806 shutdown_connections 2023-10-30T12:49:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.003+0000 7f7e80580640 1 -- 172.21.15.125:0/3329313806 wait complete. 2023-10-30T12:49:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.003+0000 7f7e80580640 1 Processor -- start 2023-10-30T12:49:22.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.003+0000 7f7e80580640 1 -- start start 2023-10-30T12:49:22.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e80580640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e640047d0 0x7f7e7815a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:22.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e80580640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e780c0a10 0x7f7e7815b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:22.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e80580640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e780c3bd0 0x7f7e78143760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:22.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e80580640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e7815a3a0 con 0x7f7e640047d0 2023-10-30T12:49:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e80580640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e78158c70 con 0x7f7e780c3bd0 2023-10-30T12:49:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e80580640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e780ca820 con 0x7f7e780c0a10 2023-10-30T12:49:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e7e31c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e640047d0 0x7f7e7815a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e7e31c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e640047d0 0x7f7e7815a7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45410/0 (socket says 172.21.15.125:45410) 2023-10-30T12:49:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e7e31c640 1 -- 172.21.15.125:0/2107362630 learned_addr learned my addr 172.21.15.125:0/2107362630 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:22.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e7db1b640 1 --2- 172.21.15.125:0/2107362630 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e780c0a10 0x7f7e7815b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:22.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e7eb1d640 1 --2- 172.21.15.125:0/2107362630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e780c3bd0 0x7f7e78143760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:22.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e7e31c640 1 -- 172.21.15.125:0/2107362630 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e780c0a10 msgr2=0x7f7e7815b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:22.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e7e31c640 1 --2- 172.21.15.125:0/2107362630 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e780c0a10 0x7f7e7815b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:22.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e7e31c640 1 -- 172.21.15.125:0/2107362630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e780c3bd0 msgr2=0x7f7e78143760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:22.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e7e31c640 1 --2- 172.21.15.125:0/2107362630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e780c3bd0 0x7f7e78143760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:22.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e7e31c640 1 -- 172.21.15.125:0/2107362630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e78158ea0 con 0x7f7e640047d0 2023-10-30T12:49:22.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e7e31c640 1 --2- 172.21.15.125:0/2107362630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e640047d0 0x7f7e7815a7c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f7e74011aa0 tx=0x7f7e74011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:22.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e6effd640 1 -- 172.21.15.125:0/2107362630 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7400fb80 con 0x7f7e640047d0 2023-10-30T12:49:22.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e6effd640 1 -- 172.21.15.125:0/2107362630 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7400fce0 con 0x7f7e640047d0 2023-10-30T12:49:22.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e6effd640 1 -- 172.21.15.125:0/2107362630 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e74014680 con 0x7f7e640047d0 2023-10-30T12:49:22.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e80580640 1 -- 172.21.15.125:0/2107362630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e780c0e70 con 0x7f7e640047d0 2023-10-30T12:49:22.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e80580640 1 -- 172.21.15.125:0/2107362630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e78159ce0 con 0x7f7e640047d0 2023-10-30T12:49:22.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e80580640 1 -- 172.21.15.125:0/2107362630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e78159b40 con 0x7f7e640047d0 2023-10-30T12:49:22.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.007+0000 7f7e6effd640 1 -- 172.21.15.125:0/2107362630 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7e740147e0 con 0x7f7e640047d0 2023-10-30T12:49:22.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.011+0000 7f7e6effd640 1 --2- 172.21.15.125:0/2107362630 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7e5c0664d0 0x7f7e5c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:22.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.011+0000 7f7e6effd640 1 -- 172.21.15.125:0/2107362630 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f7e74084bd0 con 0x7f7e640047d0 2023-10-30T12:49:22.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.011+0000 7f7e7db1b640 1 --2- 172.21.15.125:0/2107362630 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7e5c0664d0 0x7f7e5c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:22.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.011+0000 7f7e7db1b640 1 --2- 172.21.15.125:0/2107362630 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7e5c0664d0 0x7f7e5c068990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7e780c7000 tx=0x7f7e680011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:22.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.015+0000 7f7e6effd640 1 -- 172.21.15.125:0/2107362630 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7e78159b40 con 0x7f7e640047d0 2023-10-30T12:49:22.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.219+0000 7f7e80580640 1 -- 172.21.15.125:0/2107362630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e78159b40 con 0x7f7e640047d0 2023-10-30T12:49:22.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.219+0000 7f7e6effd640 1 -- 172.21.15.125:0/2107362630 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7f7e78159b40 con 0x7f7e640047d0 2023-10-30T12:49:22.220 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:22.220 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:22.220 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:22.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.223+0000 7f7e80580640 1 -- 172.21.15.125:0/2107362630 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7e5c0664d0 msgr2=0x7f7e5c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:22.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.223+0000 7f7e80580640 1 --2- 172.21.15.125:0/2107362630 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7e5c0664d0 0x7f7e5c068990 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f7e780c7000 tx=0x7f7e680011a0 comp rx=0 tx=0).stop 2023-10-30T12:49:22.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.223+0000 7f7e80580640 1 -- 172.21.15.125:0/2107362630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e640047d0 msgr2=0x7f7e7815a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:22.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.223+0000 7f7e80580640 1 --2- 172.21.15.125:0/2107362630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e640047d0 0x7f7e7815a7c0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f7e74011aa0 tx=0x7f7e74011f60 comp rx=0 tx=0).stop 2023-10-30T12:49:22.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.223+0000 7f7e80580640 1 -- 172.21.15.125:0/2107362630 shutdown_connections 2023-10-30T12:49:22.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.223+0000 7f7e80580640 1 --2- 172.21.15.125:0/2107362630 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7e5c0664d0 0x7f7e5c068990 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:22.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.223+0000 7f7e80580640 1 --2- 172.21.15.125:0/2107362630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7e780c3bd0 0x7f7e78143760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:22.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.223+0000 7f7e80580640 1 --2- 172.21.15.125:0/2107362630 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7e780c0a10 0x7f7e7815b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:22.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.223+0000 7f7e80580640 1 --2- 172.21.15.125:0/2107362630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7e640047d0 0x7f7e7815a7c0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:22.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.223+0000 7f7e80580640 1 -- 172.21.15.125:0/2107362630 >> 172.21.15.125:0/2107362630 conn(0x7f7e780c7910 msgr2=0x7f7e780b0910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:22.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.223+0000 7f7e80580640 1 -- 172.21.15.125:0/2107362630 shutdown_connections 2023-10-30T12:49:22.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:22.223+0000 7f7e80580640 1 -- 172.21.15.125:0/2107362630 wait complete. 2023-10-30T12:49:22.240 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:24.241 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:24.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea5e565640 1 Processor -- start 2023-10-30T12:49:24.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea5e565640 1 -- start start 2023-10-30T12:49:24.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea5e565640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fea581538a0 0x7fea5815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:24.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea5e565640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fea580ca0e0 0x7fea5815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:24.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea5e565640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fea580cf160 0x7fea580cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:24.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea5e565640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea580cb800 con 0x7fea581538a0 2023-10-30T12:49:24.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea5e565640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea5815dc30 con 0x7fea580ca0e0 2023-10-30T12:49:24.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea5e565640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea580cb330 con 0x7fea580cf160 2023-10-30T12:49:24.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea57fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fea581538a0 0x7fea5815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:24.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea57fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fea581538a0 0x7fea5815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45424/0 (socket says 172.21.15.125:45424) 2023-10-30T12:49:24.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea57fff640 1 -- 172.21.15.125:0/1126323812 learned_addr learned my addr 172.21.15.125:0/1126323812 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:24.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea5cb02640 1 --2- 172.21.15.125:0/1126323812 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fea580cf160 0x7fea580cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:24.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea577fe640 1 --2- 172.21.15.125:0/1126323812 >> v2:172.21.15.136:3300/0 conn(0x7fea580ca0e0 0x7fea5815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:24.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea57fff640 1 -- 172.21.15.125:0/1126323812 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fea580cf160 msgr2=0x7fea580cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:24.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea57fff640 1 --2- 172.21.15.125:0/1126323812 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fea580cf160 0x7fea580cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:24.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea57fff640 1 -- 172.21.15.125:0/1126323812 >> v2:172.21.15.136:3300/0 conn(0x7fea580ca0e0 msgr2=0x7fea5815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:24.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea57fff640 1 --2- 172.21.15.125:0/1126323812 >> v2:172.21.15.136:3300/0 conn(0x7fea580ca0e0 0x7fea5815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:24.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.383+0000 7fea57fff640 1 -- 172.21.15.125:0/1126323812 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea580cad70 con 0x7fea581538a0 2023-10-30T12:49:24.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.387+0000 7fea57fff640 1 --2- 172.21.15.125:0/1126323812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea581538a0 0x7fea5815d6f0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fea580b30a0 tx=0x7fea4801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecebc8d123ef899 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:24.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.387+0000 7fea56ffd640 1 -- 172.21.15.125:0/1126323812 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea480a89b0 con 0x7fea581538a0 2023-10-30T12:49:24.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.387+0000 7fea56ffd640 1 -- 172.21.15.125:0/1126323812 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea480a8b10 con 0x7fea581538a0 2023-10-30T12:49:24.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.387+0000 7fea56ffd640 1 -- 172.21.15.125:0/1126323812 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea480a8ea0 con 0x7fea581538a0 2023-10-30T12:49:24.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.387+0000 7fea5e565640 1 -- 172.21.15.125:0/1126323812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea581538a0 msgr2=0x7fea5815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:24.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.387+0000 7fea5e565640 1 --2- 172.21.15.125:0/1126323812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea581538a0 0x7fea5815d6f0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7fea580b30a0 tx=0x7fea4801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:49:24.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.387+0000 7fea5e565640 1 -- 172.21.15.125:0/1126323812 shutdown_connections 2023-10-30T12:49:24.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.387+0000 7fea5e565640 1 --2- 172.21.15.125:0/1126323812 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fea580cf160 0x7fea580cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:24.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.387+0000 7fea5e565640 1 --2- 172.21.15.125:0/1126323812 >> v2:172.21.15.136:3300/0 conn(0x7fea580ca0e0 0x7fea5815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:24.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.387+0000 7fea5e565640 1 --2- 172.21.15.125:0/1126323812 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea581538a0 0x7fea5815d6f0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:24.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.387+0000 7fea5e565640 1 -- 172.21.15.125:0/1126323812 >> 172.21.15.125:0/1126323812 conn(0x7fea5806bfe0 msgr2=0x7fea5806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:24.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.387+0000 7fea5e565640 1 -- 172.21.15.125:0/1126323812 shutdown_connections 2023-10-30T12:49:24.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.387+0000 7fea5e565640 1 -- 172.21.15.125:0/1126323812 wait complete. 2023-10-30T12:49:24.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea5e565640 1 Processor -- start 2023-10-30T12:49:24.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea5e565640 1 -- start start 2023-10-30T12:49:24.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea5e565640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea580ca0e0 0x7fea58142d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:24.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea5e565640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fea580cf160 0x7fea581432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:24.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea5e565640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea581538a0 0x7fea580b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:24.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea5e565640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea58158bb0 con 0x7fea580ca0e0 2023-10-30T12:49:24.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea5e565640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea58159520 con 0x7fea581538a0 2023-10-30T12:49:24.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea5e565640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea581508e0 con 0x7fea580cf160 2023-10-30T12:49:24.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea57fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea580ca0e0 0x7fea58142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:24.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea57fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea580ca0e0 0x7fea58142d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45436/0 (socket says 172.21.15.125:45436) 2023-10-30T12:49:24.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea57fff640 1 -- 172.21.15.125:0/1278573259 learned_addr learned my addr 172.21.15.125:0/1278573259 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:24.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea5cb02640 1 --2- 172.21.15.125:0/1278573259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea581538a0 0x7fea580b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:24.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea577fe640 1 --2- 172.21.15.125:0/1278573259 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fea580cf160 0x7fea581432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:24.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea57fff640 1 -- 172.21.15.125:0/1278573259 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fea580cf160 msgr2=0x7fea581432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:24.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea57fff640 1 --2- 172.21.15.125:0/1278573259 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fea580cf160 0x7fea581432a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:24.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea57fff640 1 -- 172.21.15.125:0/1278573259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea581538a0 msgr2=0x7fea580b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:24.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea57fff640 1 --2- 172.21.15.125:0/1278573259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea581538a0 0x7fea580b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:24.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea57fff640 1 -- 172.21.15.125:0/1278573259 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea480094d0 con 0x7fea580ca0e0 2023-10-30T12:49:24.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea57fff640 1 --2- 172.21.15.125:0/1278573259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea580ca0e0 0x7fea58142d60 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fea4801d3a0 tx=0x7fea4801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:24.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea54ff9640 1 -- 172.21.15.125:0/1278573259 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea480ba600 con 0x7fea580ca0e0 2023-10-30T12:49:24.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea54ff9640 1 -- 172.21.15.125:0/1278573259 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea480ba760 con 0x7fea580ca0e0 2023-10-30T12:49:24.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea5e565640 1 -- 172.21.15.125:0/1278573259 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea58150bc0 con 0x7fea580ca0e0 2023-10-30T12:49:24.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea54ff9640 1 -- 172.21.15.125:0/1278573259 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea480bab10 con 0x7fea580ca0e0 2023-10-30T12:49:24.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.391+0000 7fea5e565640 1 -- 172.21.15.125:0/1278573259 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea58158e00 con 0x7fea580ca0e0 2023-10-30T12:49:24.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.395+0000 7fea5e565640 1 -- 172.21.15.125:0/1278573259 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea580b2a50 con 0x7fea580ca0e0 2023-10-30T12:49:24.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.395+0000 7fea54ff9640 1 -- 172.21.15.125:0/1278573259 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fea480bac70 con 0x7fea580ca0e0 2023-10-30T12:49:24.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.395+0000 7fea54ff9640 1 --2- 172.21.15.125:0/1278573259 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fea440664a0 0x7fea44068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:24.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.395+0000 7fea54ff9640 1 -- 172.21.15.125:0/1278573259 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fea481227e0 con 0x7fea580ca0e0 2023-10-30T12:49:24.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.395+0000 7fea577fe640 1 --2- 172.21.15.125:0/1278573259 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fea440664a0 0x7fea44068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:24.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.395+0000 7fea577fe640 1 --2- 172.21.15.125:0/1278573259 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fea440664a0 0x7fea44068960 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fea580cea10 tx=0x7fea4c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:24.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.399+0000 7fea54ff9640 1 -- 172.21.15.125:0/1278573259 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fea480ef8d0 con 0x7fea580ca0e0 2023-10-30T12:49:24.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.579+0000 7fea5e565640 1 -- 172.21.15.125:0/1278573259 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea58159290 con 0x7fea580ca0e0 2023-10-30T12:49:24.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.579+0000 7fea54ff9640 1 -- 172.21.15.125:0/1278573259 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7fea480f2ef0 con 0x7fea580ca0e0 2023-10-30T12:49:24.579 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:24.580 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:24.581 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:24.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.583+0000 7fea5e565640 1 -- 172.21.15.125:0/1278573259 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fea440664a0 msgr2=0x7fea44068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:24.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.583+0000 7fea5e565640 1 --2- 172.21.15.125:0/1278573259 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fea440664a0 0x7fea44068960 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fea580cea10 tx=0x7fea4c008040 comp rx=0 tx=0).stop 2023-10-30T12:49:24.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.583+0000 7fea5e565640 1 -- 172.21.15.125:0/1278573259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea580ca0e0 msgr2=0x7fea58142d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:24.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.583+0000 7fea5e565640 1 --2- 172.21.15.125:0/1278573259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea580ca0e0 0x7fea58142d60 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7fea4801d3a0 tx=0x7fea4801ba90 comp rx=0 tx=0).stop 2023-10-30T12:49:24.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.583+0000 7fea5e565640 1 -- 172.21.15.125:0/1278573259 shutdown_connections 2023-10-30T12:49:24.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.583+0000 7fea5e565640 1 --2- 172.21.15.125:0/1278573259 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fea440664a0 0x7fea44068960 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:24.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.583+0000 7fea5e565640 1 --2- 172.21.15.125:0/1278573259 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fea581538a0 0x7fea580b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:24.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.583+0000 7fea5e565640 1 --2- 172.21.15.125:0/1278573259 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fea580cf160 0x7fea581432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:24.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.583+0000 7fea5e565640 1 --2- 172.21.15.125:0/1278573259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fea580ca0e0 0x7fea58142d60 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:24.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.583+0000 7fea5e565640 1 -- 172.21.15.125:0/1278573259 >> 172.21.15.125:0/1278573259 conn(0x7fea5806bfe0 msgr2=0x7fea58066410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:24.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.583+0000 7fea5e565640 1 -- 172.21.15.125:0/1278573259 shutdown_connections 2023-10-30T12:49:24.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:24.583+0000 7fea5e565640 1 -- 172.21.15.125:0/1278573259 wait complete. 2023-10-30T12:49:24.600 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T12:49:24.601 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6223": { "addr": "172.21.15.136:6839/3479573317", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3479573317, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3479573317, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6223, "incarnation": 100, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:stopping", "state_seq": 16 }, "gid_6318": { "addr": "172.21.15.125:6836/1721514617", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1721514617, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1721514617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 6318, "incarnation": 92, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:49:12.327796+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6318, "mds_1": 6223 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6837/2408701650", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2408701650, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2408701650, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 105, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6627, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:49:26.602 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:26.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.751+0000 7f9cce0dc640 1 Processor -- start 2023-10-30T12:49:26.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.751+0000 7f9cce0dc640 1 -- start start 2023-10-30T12:49:26.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.755+0000 7f9cce0dc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9cc815cd50 0x7f9cc80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:26.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.755+0000 7f9cce0dc640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc815a6c0 con 0x7f9cc81538f0 2023-10-30T12:49:26.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.755+0000 7f9cce0dc640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc815d300 con 0x7f9cc80cefa0 2023-10-30T12:49:26.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.755+0000 7f9cce0dc640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc815d5e0 con 0x7f9cc815cd50 2023-10-30T12:49:26.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.755+0000 7f9ccc8d9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9cc815cd50 0x7f9cc80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:26.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.755+0000 7f9ccc8d9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9cc815cd50 0x7f9cc80c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45444/0 (socket says 172.21.15.125:45444) 2023-10-30T12:49:26.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.755+0000 7f9ccc8d9640 1 -- 172.21.15.125:0/1349196740 learned_addr learned my addr 172.21.15.125:0/1349196740 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:26.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.755+0000 7f9ccd0da640 1 --1- 172.21.15.125:0/1349196740 >> v1:172.21.15.125:6789/0 conn(0x7f9cc81538f0 0x7f9cc8159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43594/0 (socket says 172.21.15.125:43594) 2023-10-30T12:49:26.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.755+0000 7f9ccc8d9640 1 -- 172.21.15.125:0/1349196740 >> v1:172.21.15.136:6789/0 conn(0x7f9cc80cefa0 legacy=0x7f9cc81640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:26.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.755+0000 7f9ccc8d9640 1 -- 172.21.15.125:0/1349196740 >> v1:172.21.15.125:6789/0 conn(0x7f9cc81538f0 legacy=0x7f9cc8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:26.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.755+0000 7f9ccc8d9640 1 -- 172.21.15.125:0/1349196740 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc815d8c0 con 0x7f9cc815cd50 2023-10-30T12:49:26.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.759+0000 7f9ccc8d9640 1 --2- 172.21.15.125:0/1349196740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc815cd50 0x7f9cc80c9f90 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8009870 tx=0x7f9cb801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c247f59c7b7f6c83 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:26.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.759+0000 7f9cbffff640 1 -- 172.21.15.125:0/1349196740 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb80a8a30 con 0x7f9cc815cd50 2023-10-30T12:49:26.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.759+0000 7f9cbffff640 1 -- 172.21.15.125:0/1349196740 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb80a8b90 con 0x7f9cc815cd50 2023-10-30T12:49:26.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.759+0000 7f9cbffff640 1 -- 172.21.15.125:0/1349196740 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb80b0440 con 0x7f9cc815cd50 2023-10-30T12:49:26.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.759+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1349196740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc815cd50 msgr2=0x7f9cc80c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:26.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.759+0000 7f9cce0dc640 1 --2- 172.21.15.125:0/1349196740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc815cd50 0x7f9cc80c9f90 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f9cb8009870 tx=0x7f9cb801ed30 comp rx=0 tx=0).stop 2023-10-30T12:49:26.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.759+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1349196740 shutdown_connections 2023-10-30T12:49:26.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.759+0000 7f9cce0dc640 1 --2- 172.21.15.125:0/1349196740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc815cd50 0x7f9cc80c9f90 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:26.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.759+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1349196740 >> 172.21.15.125:0/1349196740 conn(0x7f9cc806a640 msgr2=0x7f9cc8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:26.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.759+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1349196740 shutdown_connections 2023-10-30T12:49:26.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.759+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1349196740 wait complete. 2023-10-30T12:49:26.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.759+0000 7f9cce0dc640 1 Processor -- start 2023-10-30T12:49:26.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.759+0000 7f9cce0dc640 1 -- start start 2023-10-30T12:49:26.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9cce0dc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc80cefa0 0x7f9cc8147e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:26.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9cce0dc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc81538f0 0x7f9cc81ef490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:26.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9cce0dc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9cc815cd50 0x7f9cc81ef9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:26.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9cce0dc640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc8152e90 con 0x7f9cc81538f0 2023-10-30T12:49:26.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9cce0dc640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc81f31f0 con 0x7f9cc80cefa0 2023-10-30T12:49:26.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9cce0dc640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc815b1c0 con 0x7f9cc815cd50 2023-10-30T12:49:26.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9ccc8d9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc81538f0 0x7f9cc81ef490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:26.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9ccc8d9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc81538f0 0x7f9cc81ef490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45448/0 (socket says 172.21.15.125:45448) 2023-10-30T12:49:26.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9ccc8d9640 1 -- 172.21.15.125:0/1585539659 learned_addr learned my addr 172.21.15.125:0/1585539659 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:26.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9ccd0da640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc80cefa0 0x7f9cc8147e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:26.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9ccc8d9640 1 -- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9cc815cd50 msgr2=0x7f9cc81ef9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:26.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9ccd8db640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9cc815cd50 0x7f9cc81ef9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:26.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9ccc8d9640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9cc815cd50 0x7f9cc81ef9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:26.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9ccc8d9640 1 -- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc80cefa0 msgr2=0x7f9cc8147e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:26.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9ccc8d9640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc80cefa0 0x7f9cc8147e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:26.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9ccc8d9640 1 -- 172.21.15.125:0/1585539659 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb80094d0 con 0x7f9cc81538f0 2023-10-30T12:49:26.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9ccd8db640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9cc815cd50 0x7f9cc81ef9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:49:26.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9ccc8d9640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc81538f0 0x7f9cc81ef490 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80b3040 tx=0x7f9cb801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:26.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9cbdffb640 1 -- 172.21.15.125:0/1585539659 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb80ba600 con 0x7f9cc81538f0 2023-10-30T12:49:26.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9cbdffb640 1 -- 172.21.15.125:0/1585539659 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb80ba7a0 con 0x7f9cc81538f0 2023-10-30T12:49:26.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1585539659 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc815b4a0 con 0x7f9cc81538f0 2023-10-30T12:49:26.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9cbdffb640 1 -- 172.21.15.125:0/1585539659 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb80bab50 con 0x7f9cc81538f0 2023-10-30T12:49:26.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1585539659 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc81f34d0 con 0x7f9cc81538f0 2023-10-30T12:49:26.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1585539659 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc804e170 con 0x7f9cc81538f0 2023-10-30T12:49:26.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.763+0000 7f9cbdffb640 1 -- 172.21.15.125:0/1585539659 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9cb80bacb0 con 0x7f9cc81538f0 2023-10-30T12:49:26.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.767+0000 7f9cbdffb640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9cac0664a0 0x7f9cac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:26.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.767+0000 7f9cbdffb640 1 -- 172.21.15.125:0/1585539659 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f9cb8122790 con 0x7f9cc81538f0 2023-10-30T12:49:26.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.767+0000 7f9ccd0da640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9cac0664a0 0x7f9cac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:26.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.767+0000 7f9ccd0da640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9cac0664a0 0x7f9cac068960 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4000f70 tx=0x7f9cc4007c90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:26.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.771+0000 7f9cbdffb640 1 -- 172.21.15.125:0/1585539659 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9cb80ef880 con 0x7f9cc81538f0 2023-10-30T12:49:26.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.951+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1585539659 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cc80b2290 con 0x7f9cc81538f0 2023-10-30T12:49:26.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.951+0000 7f9cbdffb640 1 -- 172.21.15.125:0/1585539659 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7f9cb80f2ea0 con 0x7f9cc81538f0 2023-10-30T12:49:26.950 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:26.950 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:26.951 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:26.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.955+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9cac0664a0 msgr2=0x7f9cac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:26.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.955+0000 7f9cce0dc640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9cac0664a0 0x7f9cac068960 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4000f70 tx=0x7f9cc4007c90 comp rx=0 tx=0).stop 2023-10-30T12:49:26.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.955+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1585539659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc81538f0 msgr2=0x7f9cc81ef490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:26.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.955+0000 7f9cce0dc640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc81538f0 0x7f9cc81ef490 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f9cb80b3040 tx=0x7f9cb801bad0 comp rx=0 tx=0).stop 2023-10-30T12:49:26.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.955+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1585539659 shutdown_connections 2023-10-30T12:49:26.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.955+0000 7f9cce0dc640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9cac0664a0 0x7f9cac068960 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:26.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.955+0000 7f9cce0dc640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9cc815cd50 0x7f9cc81ef9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:26.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.955+0000 7f9cce0dc640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9cc81538f0 0x7f9cc81ef490 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:26.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.955+0000 7f9cce0dc640 1 --2- 172.21.15.125:0/1585539659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9cc80cefa0 0x7f9cc8147e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:26.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.955+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1585539659 >> 172.21.15.125:0/1585539659 conn(0x7f9cc806a640 msgr2=0x7f9cc80cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:26.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.955+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1585539659 shutdown_connections 2023-10-30T12:49:26.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:26.955+0000 7f9cce0dc640 1 -- 172.21.15.125:0/1585539659 wait complete. 2023-10-30T12:49:26.970 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:28.972 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:29.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc705417640 1 Processor -- start 2023-10-30T12:49:29.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc705417640 1 -- start start 2023-10-30T12:49:29.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc705417640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc700158360 0x7fc7000c7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:29.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc705417640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70006eaf0 0x7fc700069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:29.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc705417640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc700157ed0 con 0x7fc7001517c0 2023-10-30T12:49:29.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc705417640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7000c77b0 con 0x7fc700158360 2023-10-30T12:49:29.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc705417640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7001575c0 con 0x7fc70006eaf0 2023-10-30T12:49:29.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc6f67fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc700158360 0x7fc7000c7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:29.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc6f67fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc700158360 0x7fc7000c7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45454/0 (socket says 172.21.15.125:45454) 2023-10-30T12:49:29.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc6f67fc640 1 -- 172.21.15.125:0/754047610 learned_addr learned my addr 172.21.15.125:0/754047610 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:29.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc6ff7fe640 1 --2- 172.21.15.125:0/754047610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70006eaf0 0x7fc700069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:29.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc6f67fc640 1 -- 172.21.15.125:0/754047610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70006eaf0 msgr2=0x7fc700069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:29.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc6f67fc640 1 --2- 172.21.15.125:0/754047610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70006eaf0 0x7fc700069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:29.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc6f67fc640 1 -- 172.21.15.125:0/754047610 >> v1:172.21.15.136:6789/0 conn(0x7fc7001517c0 legacy=0x7fc70006c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:29.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.115+0000 7fc6f67fc640 1 -- 172.21.15.125:0/754047610 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7000ca400 con 0x7fc700158360 2023-10-30T12:49:29.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.119+0000 7fc6f67fc640 1 --2- 172.21.15.125:0/754047610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc700158360 0x7fc7000c7270 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8009870 tx=0x7fc6e801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846502a8f5919333 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:29.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.119+0000 7fc6fe7fc640 1 -- 172.21.15.125:0/754047610 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e80a8a30 con 0x7fc700158360 2023-10-30T12:49:29.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.119+0000 7fc6fe7fc640 1 -- 172.21.15.125:0/754047610 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e80a8b90 con 0x7fc700158360 2023-10-30T12:49:29.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.119+0000 7fc6fe7fc640 1 -- 172.21.15.125:0/754047610 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e80b0440 con 0x7fc700158360 2023-10-30T12:49:29.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.119+0000 7fc705417640 1 -- 172.21.15.125:0/754047610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc700158360 msgr2=0x7fc7000c7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:29.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.119+0000 7fc705417640 1 --2- 172.21.15.125:0/754047610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc700158360 0x7fc7000c7270 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8009870 tx=0x7fc6e801ed30 comp rx=0 tx=0).stop 2023-10-30T12:49:29.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.119+0000 7fc705417640 1 -- 172.21.15.125:0/754047610 shutdown_connections 2023-10-30T12:49:29.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.119+0000 7fc705417640 1 --2- 172.21.15.125:0/754047610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc70006eaf0 0x7fc700069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:29.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.119+0000 7fc705417640 1 --2- 172.21.15.125:0/754047610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc700158360 0x7fc7000c7270 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:29.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.119+0000 7fc705417640 1 -- 172.21.15.125:0/754047610 >> 172.21.15.125:0/754047610 conn(0x7fc70006dfd0 msgr2=0x7fc70006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:29.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.119+0000 7fc705417640 1 -- 172.21.15.125:0/754047610 shutdown_connections 2023-10-30T12:49:29.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.119+0000 7fc705417640 1 -- 172.21.15.125:0/754047610 wait complete. 2023-10-30T12:49:29.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc705417640 1 Processor -- start 2023-10-30T12:49:29.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc705417640 1 -- start start 2023-10-30T12:49:29.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc705417640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70006eaf0 0x7fc70014e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:29.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc705417640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7001517c0 0x7fc70014e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:29.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc705417640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc700158360 0x7fc7001f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:29.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc705417640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70014c630 con 0x7fc70006eaf0 2023-10-30T12:49:29.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc705417640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7001f1ff0 con 0x7fc7001517c0 2023-10-30T12:49:29.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc705417640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70014d610 con 0x7fc700158360 2023-10-30T12:49:29.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc6feffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70006eaf0 0x7fc70014e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:29.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc6feffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70006eaf0 0x7fc70014e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45468/0 (socket says 172.21.15.125:45468) 2023-10-30T12:49:29.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc6feffd640 1 -- 172.21.15.125:0/3181659317 learned_addr learned my addr 172.21.15.125:0/3181659317 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:29.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc6f67fc640 1 --2- 172.21.15.125:0/3181659317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7001517c0 0x7fc70014e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:29.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc6ff7fe640 1 --2- 172.21.15.125:0/3181659317 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc700158360 0x7fc7001f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:29.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc6feffd640 1 -- 172.21.15.125:0/3181659317 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc700158360 msgr2=0x7fc7001f24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:29.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc6feffd640 1 --2- 172.21.15.125:0/3181659317 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc700158360 0x7fc7001f24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:29.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc6feffd640 1 -- 172.21.15.125:0/3181659317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7001517c0 msgr2=0x7fc70014e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:29.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc6feffd640 1 --2- 172.21.15.125:0/3181659317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7001517c0 0x7fc70014e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:29.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc6feffd640 1 -- 172.21.15.125:0/3181659317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6e80094d0 con 0x7fc70006eaf0 2023-10-30T12:49:29.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc6feffd640 1 --2- 172.21.15.125:0/3181659317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70006eaf0 0x7fc70014e390 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc6f000c7f0 tx=0x7fc6f000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:29.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc6f7fff640 1 -- 172.21.15.125:0/3181659317 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f0003b80 con 0x7fc70006eaf0 2023-10-30T12:49:29.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc6f7fff640 1 -- 172.21.15.125:0/3181659317 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6f0003ce0 con 0x7fc70006eaf0 2023-10-30T12:49:29.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc6f7fff640 1 -- 172.21.15.125:0/3181659317 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6f000aa90 con 0x7fc70006eaf0 2023-10-30T12:49:29.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc705417640 1 -- 172.21.15.125:0/3181659317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7001f1d10 con 0x7fc70006eaf0 2023-10-30T12:49:29.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.123+0000 7fc705417640 1 -- 172.21.15.125:0/3181659317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc70014db90 con 0x7fc70006eaf0 2023-10-30T12:49:29.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.127+0000 7fc705417640 1 -- 172.21.15.125:0/3181659317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70004e170 con 0x7fc70006eaf0 2023-10-30T12:49:29.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.127+0000 7fc6f7fff640 1 -- 172.21.15.125:0/3181659317 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6f000abf0 con 0x7fc70006eaf0 2023-10-30T12:49:29.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.127+0000 7fc6f7fff640 1 --2- 172.21.15.125:0/3181659317 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6e00664d0 0x7fc6e0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:29.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.127+0000 7fc6f7fff640 1 -- 172.21.15.125:0/3181659317 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fc6f0083ff0 con 0x7fc70006eaf0 2023-10-30T12:49:29.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.127+0000 7fc6f67fc640 1 --2- 172.21.15.125:0/3181659317 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6e00664d0 0x7fc6e0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:29.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.131+0000 7fc6f67fc640 1 --2- 172.21.15.125:0/3181659317 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6e00664d0 0x7fc6e0068990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc6e80b2000 tx=0x7fc6e80aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:29.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.131+0000 7fc6f7fff640 1 -- 172.21.15.125:0/3181659317 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc6f0051100 con 0x7fc70006eaf0 2023-10-30T12:49:29.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.347+0000 7fc705417640 1 -- 172.21.15.125:0/3181659317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc700157880 con 0x7fc70006eaf0 2023-10-30T12:49:29.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.347+0000 7fc6f7fff640 1 -- 172.21.15.125:0/3181659317 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7fc6f0054720 con 0x7fc70006eaf0 2023-10-30T12:49:29.346 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:29.347 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:29.347 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:29.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.351+0000 7fc705417640 1 -- 172.21.15.125:0/3181659317 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6e00664d0 msgr2=0x7fc6e0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:29.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.351+0000 7fc705417640 1 --2- 172.21.15.125:0/3181659317 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6e00664d0 0x7fc6e0068990 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fc6e80b2000 tx=0x7fc6e80aa070 comp rx=0 tx=0).stop 2023-10-30T12:49:29.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.351+0000 7fc705417640 1 -- 172.21.15.125:0/3181659317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70006eaf0 msgr2=0x7fc70014e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:29.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.351+0000 7fc705417640 1 --2- 172.21.15.125:0/3181659317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70006eaf0 0x7fc70014e390 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fc6f000c7f0 tx=0x7fc6f000ccb0 comp rx=0 tx=0).stop 2023-10-30T12:49:29.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.351+0000 7fc705417640 1 -- 172.21.15.125:0/3181659317 shutdown_connections 2023-10-30T12:49:29.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.351+0000 7fc705417640 1 --2- 172.21.15.125:0/3181659317 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc6e00664d0 0x7fc6e0068990 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:29.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.351+0000 7fc705417640 1 --2- 172.21.15.125:0/3181659317 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc700158360 0x7fc7001f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:29.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.351+0000 7fc705417640 1 --2- 172.21.15.125:0/3181659317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7001517c0 0x7fc70014e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:29.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.351+0000 7fc705417640 1 --2- 172.21.15.125:0/3181659317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc70006eaf0 0x7fc70014e390 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:29.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.351+0000 7fc705417640 1 -- 172.21.15.125:0/3181659317 >> 172.21.15.125:0/3181659317 conn(0x7fc70006dfd0 msgr2=0x7fc70006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:29.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.351+0000 7fc705417640 1 -- 172.21.15.125:0/3181659317 shutdown_connections 2023-10-30T12:49:29.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:29.351+0000 7fc705417640 1 -- 172.21.15.125:0/3181659317 wait complete. 2023-10-30T12:49:29.367 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:31.368 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:31.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.518+0000 7fb37a2c9640 1 Processor -- start 2023-10-30T12:49:31.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.518+0000 7fb37a2c9640 1 -- start start 2023-10-30T12:49:31.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb37a2c9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb3740c6b80 0x7fb3740bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:31.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb37a2c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3740c6f60 0x7fb3740c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:31.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb37a2c9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb3740c8430 0x7fb37413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:31.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb37a2c9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3740c0a50 con 0x7fb3740c6b80 2023-10-30T12:49:31.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb37a2c9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb37406f6e0 con 0x7fb3740c8430 2023-10-30T12:49:31.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb37a2c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37406f9c0 con 0x7fb3740c6f60 2023-10-30T12:49:31.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb373fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb3740c6b80 0x7fb3740bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:31.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb373fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb3740c6b80 0x7fb3740bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60952/0 (socket says 172.21.15.125:60952) 2023-10-30T12:49:31.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb373fff640 1 -- 172.21.15.125:0/241927077 learned_addr learned my addr 172.21.15.125:0/241927077 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:31.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb3737fe640 1 --2- 172.21.15.125:0/241927077 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3740c6f60 0x7fb3740c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:31.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb378866640 1 --2- 172.21.15.125:0/241927077 >> v2:172.21.15.136:3300/0 conn(0x7fb3740c8430 0x7fb37413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:31.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb373fff640 1 -- 172.21.15.125:0/241927077 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3740c6f60 msgr2=0x7fb3740c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:31.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb373fff640 1 --2- 172.21.15.125:0/241927077 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3740c6f60 0x7fb3740c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:31.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb373fff640 1 -- 172.21.15.125:0/241927077 >> v2:172.21.15.136:3300/0 conn(0x7fb3740c8430 msgr2=0x7fb37413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:31.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb373fff640 1 --2- 172.21.15.125:0/241927077 >> v2:172.21.15.136:3300/0 conn(0x7fb3740c8430 0x7fb37413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:31.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb373fff640 1 -- 172.21.15.125:0/241927077 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb37406fca0 con 0x7fb3740c6b80 2023-10-30T12:49:31.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.522+0000 7fb373fff640 1 --2- 172.21.15.125:0/241927077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740c6b80 0x7fb3740bfec0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fb3740b0860 tx=0x7fb36401c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586ef9e6d9f6d315 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:31.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb372ffd640 1 -- 172.21.15.125:0/241927077 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3640ab600 con 0x7fb3740c6b80 2023-10-30T12:49:31.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb372ffd640 1 -- 172.21.15.125:0/241927077 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3640a7050 con 0x7fb3740c6b80 2023-10-30T12:49:31.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb372ffd640 1 -- 172.21.15.125:0/241927077 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3640ab980 con 0x7fb3740c6b80 2023-10-30T12:49:31.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 -- 172.21.15.125:0/241927077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740c6b80 msgr2=0x7fb3740bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:31.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 --2- 172.21.15.125:0/241927077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740c6b80 0x7fb3740bfec0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fb3740b0860 tx=0x7fb36401c470 comp rx=0 tx=0).stop 2023-10-30T12:49:31.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 -- 172.21.15.125:0/241927077 shutdown_connections 2023-10-30T12:49:31.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 --2- 172.21.15.125:0/241927077 >> v2:172.21.15.136:3300/0 conn(0x7fb3740c8430 0x7fb37413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:31.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 --2- 172.21.15.125:0/241927077 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3740c6f60 0x7fb3740c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:31.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 --2- 172.21.15.125:0/241927077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740c6b80 0x7fb3740bfec0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:31.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 -- 172.21.15.125:0/241927077 >> 172.21.15.125:0/241927077 conn(0x7fb3740c7910 msgr2=0x7fb3740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:31.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 -- 172.21.15.125:0/241927077 shutdown_connections 2023-10-30T12:49:31.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 -- 172.21.15.125:0/241927077 wait complete. 2023-10-30T12:49:31.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 Processor -- start 2023-10-30T12:49:31.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 -- start start 2023-10-30T12:49:31.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740c6b80 0x7fb37414d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:31.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3740c6f60 0x7fb37414fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:31.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3740c8430 0x7fb374150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:31.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3741596e0 con 0x7fb3740c6b80 2023-10-30T12:49:31.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb374154470 con 0x7fb3740c8430 2023-10-30T12:49:31.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb37a2c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb37413b000 con 0x7fb3740c6f60 2023-10-30T12:49:31.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb373fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740c6b80 0x7fb37414d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:31.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb373fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740c6b80 0x7fb37414d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60962/0 (socket says 172.21.15.125:60962) 2023-10-30T12:49:31.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.526+0000 7fb373fff640 1 -- 172.21.15.125:0/828527333 learned_addr learned my addr 172.21.15.125:0/828527333 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:31.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb3737fe640 1 --2- 172.21.15.125:0/828527333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3740c6f60 0x7fb37414fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:31.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb378866640 1 --2- 172.21.15.125:0/828527333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3740c8430 0x7fb374150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:31.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb373fff640 1 -- 172.21.15.125:0/828527333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3740c6f60 msgr2=0x7fb37414fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:31.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb373fff640 1 --2- 172.21.15.125:0/828527333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3740c6f60 0x7fb37414fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:31.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb373fff640 1 -- 172.21.15.125:0/828527333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3740c8430 msgr2=0x7fb374150320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:31.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb373fff640 1 --2- 172.21.15.125:0/828527333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3740c8430 0x7fb374150320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:31.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb373fff640 1 -- 172.21.15.125:0/828527333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3640094d0 con 0x7fb3740c6b80 2023-10-30T12:49:31.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb373fff640 1 --2- 172.21.15.125:0/828527333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740c6b80 0x7fb37414d4d0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb36401cd10 tx=0x7fb364019210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:31.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb370ff9640 1 -- 172.21.15.125:0/828527333 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3640b6650 con 0x7fb3740c6b80 2023-10-30T12:49:31.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb370ff9640 1 -- 172.21.15.125:0/828527333 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3640b67d0 con 0x7fb3740c6b80 2023-10-30T12:49:31.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb370ff9640 1 -- 172.21.15.125:0/828527333 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3640b6b80 con 0x7fb3740c6b80 2023-10-30T12:49:31.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb37a2c9640 1 -- 172.21.15.125:0/828527333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb374158d40 con 0x7fb3740c6b80 2023-10-30T12:49:31.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb37a2c9640 1 -- 172.21.15.125:0/828527333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb374159300 con 0x7fb3740c6b80 2023-10-30T12:49:31.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb37a2c9640 1 -- 172.21.15.125:0/828527333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb374159820 con 0x7fb3740c6b80 2023-10-30T12:49:31.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb370ff9640 1 -- 172.21.15.125:0/828527333 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb3640b2040 con 0x7fb3740c6b80 2023-10-30T12:49:31.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb370ff9640 1 --2- 172.21.15.125:0/828527333 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb3600664a0 0x7fb360068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:31.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.530+0000 7fb370ff9640 1 -- 172.21.15.125:0/828527333 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fb36411e700 con 0x7fb3740c6b80 2023-10-30T12:49:31.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.534+0000 7fb3737fe640 1 --2- 172.21.15.125:0/828527333 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb3600664a0 0x7fb360068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:31.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.534+0000 7fb3737fe640 1 --2- 172.21.15.125:0/828527333 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb3600664a0 0x7fb360068960 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb37413b9a0 tx=0x7fb368008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:31.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.538+0000 7fb370ff9640 1 -- 172.21.15.125:0/828527333 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb3640eb7f0 con 0x7fb3740c6b80 2023-10-30T12:49:31.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.718+0000 7fb37a2c9640 1 -- 172.21.15.125:0/828527333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb37404e170 con 0x7fb3740c6b80 2023-10-30T12:49:31.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.718+0000 7fb370ff9640 1 -- 172.21.15.125:0/828527333 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7fb3640eee10 con 0x7fb3740c6b80 2023-10-30T12:49:31.717 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:31.717 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:31.718 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:31.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.718+0000 7fb37a2c9640 1 -- 172.21.15.125:0/828527333 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb3600664a0 msgr2=0x7fb360068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:31.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.718+0000 7fb37a2c9640 1 --2- 172.21.15.125:0/828527333 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb3600664a0 0x7fb360068960 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fb37413b9a0 tx=0x7fb368008040 comp rx=0 tx=0).stop 2023-10-30T12:49:31.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.718+0000 7fb37a2c9640 1 -- 172.21.15.125:0/828527333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740c6b80 msgr2=0x7fb37414d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:31.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.718+0000 7fb37a2c9640 1 --2- 172.21.15.125:0/828527333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740c6b80 0x7fb37414d4d0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fb36401cd10 tx=0x7fb364019210 comp rx=0 tx=0).stop 2023-10-30T12:49:31.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.718+0000 7fb37a2c9640 1 -- 172.21.15.125:0/828527333 shutdown_connections 2023-10-30T12:49:31.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.718+0000 7fb37a2c9640 1 --2- 172.21.15.125:0/828527333 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb3600664a0 0x7fb360068960 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:31.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.718+0000 7fb37a2c9640 1 --2- 172.21.15.125:0/828527333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3740c8430 0x7fb374150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:31.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.718+0000 7fb37a2c9640 1 --2- 172.21.15.125:0/828527333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3740c6f60 0x7fb37414fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:31.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.718+0000 7fb37a2c9640 1 --2- 172.21.15.125:0/828527333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3740c6b80 0x7fb37414d4d0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:31.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.718+0000 7fb37a2c9640 1 -- 172.21.15.125:0/828527333 >> 172.21.15.125:0/828527333 conn(0x7fb3740c7910 msgr2=0x7fb3740c0ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:31.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.718+0000 7fb37a2c9640 1 -- 172.21.15.125:0/828527333 shutdown_connections 2023-10-30T12:49:31.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:31.718+0000 7fb37a2c9640 1 -- 172.21.15.125:0/828527333 wait complete. 2023-10-30T12:49:31.738 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:33.739 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74de994640 1 Processor -- start 2023-10-30T12:49:33.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74de994640 1 -- start start 2023-10-30T12:49:33.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74de994640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f74d8069e90 0x7f74d815a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:33.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74de994640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d806a270 0x7f74d80c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74de994640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d80c6c10 con 0x7f74d806cb80 2023-10-30T12:49:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74de994640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d80c8570 con 0x7f74d8069e90 2023-10-30T12:49:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74de994640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d80c6ef0 con 0x7f74d806a270 2023-10-30T12:49:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74d7fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f74d8069e90 0x7f74d815a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74d7fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f74d8069e90 0x7f74d815a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60974/0 (socket says 172.21.15.125:60974) 2023-10-30T12:49:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74d7fff640 1 -- 172.21.15.125:0/832864065 learned_addr learned my addr 172.21.15.125:0/832864065 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74dcf31640 1 --1- 172.21.15.125:0/832864065 >> v1:172.21.15.125:6789/0 conn(0x7f74d806cb80 0x7f74d806d010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47886/0 (socket says 172.21.15.125:47886) 2023-10-30T12:49:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74d77fe640 1 --2- 172.21.15.125:0/832864065 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d806a270 0x7f74d80c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74d7fff640 1 -- 172.21.15.125:0/832864065 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d806a270 msgr2=0x7f74d80c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74d7fff640 1 --2- 172.21.15.125:0/832864065 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d806a270 0x7f74d80c9cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:33.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74d7fff640 1 -- 172.21.15.125:0/832864065 >> v1:172.21.15.125:6789/0 conn(0x7f74d806cb80 legacy=0x7f74d806d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:33.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74d7fff640 1 -- 172.21.15.125:0/832864065 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d80c71d0 con 0x7f74d8069e90 2023-10-30T12:49:33.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.886+0000 7f74d6ffd640 1 -- 172.21.15.125:0/832864065 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091381231 0 0) 0x7f74d80c6c10 con 0x7f74d806cb80 2023-10-30T12:49:33.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.890+0000 7f74d7fff640 1 --2- 172.21.15.125:0/832864065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d8069e90 0x7f74d815a410 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f74c8009870 tx=0x7f74c801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2a33f55cc9c675a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:33.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.890+0000 7f74d6ffd640 1 -- 172.21.15.125:0/832864065 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c80a7070 con 0x7f74d8069e90 2023-10-30T12:49:33.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.890+0000 7f74d6ffd640 1 -- 172.21.15.125:0/832864065 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c80ab430 con 0x7f74d8069e90 2023-10-30T12:49:33.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.890+0000 7f74d6ffd640 1 -- 172.21.15.125:0/832864065 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74c80ab7c0 con 0x7f74d8069e90 2023-10-30T12:49:33.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.890+0000 7f74de994640 1 -- 172.21.15.125:0/832864065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d8069e90 msgr2=0x7f74d815a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:33.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.890+0000 7f74de994640 1 --2- 172.21.15.125:0/832864065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d8069e90 0x7f74d815a410 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f74c8009870 tx=0x7f74c801c540 comp rx=0 tx=0).stop 2023-10-30T12:49:33.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.890+0000 7f74de994640 1 -- 172.21.15.125:0/832864065 shutdown_connections 2023-10-30T12:49:33.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.890+0000 7f74de994640 1 --2- 172.21.15.125:0/832864065 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d806a270 0x7f74d80c9cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:33.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.890+0000 7f74de994640 1 --2- 172.21.15.125:0/832864065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d8069e90 0x7f74d815a410 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:33.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.890+0000 7f74de994640 1 -- 172.21.15.125:0/832864065 >> 172.21.15.125:0/832864065 conn(0x7f74d806dfd0 msgr2=0x7f74d806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:33.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.890+0000 7f74de994640 1 -- 172.21.15.125:0/832864065 shutdown_connections 2023-10-30T12:49:33.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.890+0000 7f74de994640 1 -- 172.21.15.125:0/832864065 wait complete. 2023-10-30T12:49:33.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74de994640 1 Processor -- start 2023-10-30T12:49:33.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74de994640 1 -- start start 2023-10-30T12:49:33.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74de994640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74d8069e90 0x7f74d814cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:33.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74de994640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d806a270 0x7f74d80c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:33.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74de994640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d806cb80 0x7f74d80c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:33.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74de994640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d81f1a90 con 0x7f74d806cb80 2023-10-30T12:49:33.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74de994640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d81f1d50 con 0x7f74d8069e90 2023-10-30T12:49:33.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74de994640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d81f2030 con 0x7f74d806a270 2023-10-30T12:49:33.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74dcf31640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d806cb80 0x7f74d80c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:33.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74dcf31640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d806cb80 0x7f74d80c4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60990/0 (socket says 172.21.15.125:60990) 2023-10-30T12:49:33.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74dcf31640 1 -- 172.21.15.125:0/125436192 learned_addr learned my addr 172.21.15.125:0/125436192 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:33.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74d77fe640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d806a270 0x7f74d80c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:33.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74d7fff640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74d8069e90 0x7f74d814cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:33.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74dcf31640 1 -- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d806a270 msgr2=0x7f74d80c4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:33.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74dcf31640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d806a270 0x7f74d80c4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:33.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74dcf31640 1 -- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74d8069e90 msgr2=0x7f74d814cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:33.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74dcf31640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74d8069e90 0x7f74d814cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:33.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74dcf31640 1 -- 172.21.15.125:0/125436192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c80094d0 con 0x7f74d806cb80 2023-10-30T12:49:33.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74d7fff640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74d8069e90 0x7f74d814cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:33.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74d77fe640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d806a270 0x7f74d80c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:33.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74dcf31640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d806cb80 0x7f74d80c4f60 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f74d000c780 tx=0x7f74d000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:33.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74d4ff9640 1 -- 172.21.15.125:0/125436192 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d000ce90 con 0x7f74d806cb80 2023-10-30T12:49:33.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74d4ff9640 1 -- 172.21.15.125:0/125436192 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74d000a640 con 0x7f74d806cb80 2023-10-30T12:49:33.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74de994640 1 -- 172.21.15.125:0/125436192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d81f2310 con 0x7f74d806cb80 2023-10-30T12:49:33.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74d4ff9640 1 -- 172.21.15.125:0/125436192 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d000a9d0 con 0x7f74d806cb80 2023-10-30T12:49:33.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74de994640 1 -- 172.21.15.125:0/125436192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d81f28d0 con 0x7f74d806cb80 2023-10-30T12:49:33.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.894+0000 7f74de994640 1 -- 172.21.15.125:0/125436192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d804e170 con 0x7f74d806cb80 2023-10-30T12:49:33.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.898+0000 7f74d4ff9640 1 -- 172.21.15.125:0/125436192 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f74d000ab30 con 0x7f74d806cb80 2023-10-30T12:49:33.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.898+0000 7f74d4ff9640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f74c40664d0 0x7f74c4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:33.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.898+0000 7f74d4ff9640 1 -- 172.21.15.125:0/125436192 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f74d0083af0 con 0x7f74d806cb80 2023-10-30T12:49:33.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.898+0000 7f74d7fff640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f74c40664d0 0x7f74c4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:33.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.898+0000 7f74d7fff640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f74c40664d0 0x7f74c4068990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f74c80af000 tx=0x7f74c8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:33.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:33.902+0000 7f74d4ff9640 1 -- 172.21.15.125:0/125436192 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f74d0050ac0 con 0x7f74d806cb80 2023-10-30T12:49:34.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:34.086+0000 7f74de994640 1 -- 172.21.15.125:0/125436192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f74d8070aa0 con 0x7f74d806cb80 2023-10-30T12:49:34.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:34.086+0000 7f74d4ff9640 1 -- 172.21.15.125:0/125436192 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7f74d0018070 con 0x7f74d806cb80 2023-10-30T12:49:34.086 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:34.086 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:34.088 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:34.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:34.086+0000 7f74de994640 1 -- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f74c40664d0 msgr2=0x7f74c4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:34.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:34.086+0000 7f74de994640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f74c40664d0 0x7f74c4068990 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f74c80af000 tx=0x7f74c8008040 comp rx=0 tx=0).stop 2023-10-30T12:49:34.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:34.086+0000 7f74de994640 1 -- 172.21.15.125:0/125436192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d806cb80 msgr2=0x7f74d80c4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:34.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:34.086+0000 7f74de994640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d806cb80 0x7f74d80c4f60 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f74d000c780 tx=0x7f74d000cc40 comp rx=0 tx=0).stop 2023-10-30T12:49:34.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:34.086+0000 7f74de994640 1 -- 172.21.15.125:0/125436192 shutdown_connections 2023-10-30T12:49:34.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:34.086+0000 7f74de994640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f74c40664d0 0x7f74c4068990 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:34.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:34.086+0000 7f74de994640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f74d806cb80 0x7f74d80c4f60 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:34.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:34.086+0000 7f74de994640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f74d806a270 0x7f74d80c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:34.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:34.086+0000 7f74de994640 1 --2- 172.21.15.125:0/125436192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f74d8069e90 0x7f74d814cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:34.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:34.086+0000 7f74de994640 1 -- 172.21.15.125:0/125436192 >> 172.21.15.125:0/125436192 conn(0x7f74d806dfd0 msgr2=0x7f74d814f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:34.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:34.086+0000 7f74de994640 1 -- 172.21.15.125:0/125436192 shutdown_connections 2023-10-30T12:49:34.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:34.086+0000 7f74de994640 1 -- 172.21.15.125:0/125436192 wait complete. 2023-10-30T12:49:34.105 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:36.106 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:36.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.254+0000 7f7fe6a6b640 1 Processor -- start 2023-10-30T12:49:36.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.254+0000 7f7fe6a6b640 1 -- start start 2023-10-30T12:49:36.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.254+0000 7f7fe6a6b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7fe01538a0 0x7f7fe015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:36.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.254+0000 7f7fe6a6b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe00cb800 con 0x7f7fe00ca0e0 2023-10-30T12:49:36.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.254+0000 7f7fe6a6b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe015dc30 con 0x7f7fe00cf160 2023-10-30T12:49:36.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.254+0000 7f7fe6a6b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe00cb330 con 0x7f7fe01538a0 2023-10-30T12:49:36.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.254+0000 7f7fe4807640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7fe01538a0 0x7f7fe015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:36.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.254+0000 7f7fe4807640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7fe01538a0 0x7f7fe015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:32768/0 (socket says 172.21.15.125:32768) 2023-10-30T12:49:36.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.254+0000 7f7fd7fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7fe00ca0e0 0x7f7fe015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47890/0 (socket says 172.21.15.125:47890) 2023-10-30T12:49:36.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.254+0000 7f7fe4807640 1 -- 172.21.15.125:0/1744094786 learned_addr learned my addr 172.21.15.125:0/1744094786 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:36.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.254+0000 7f7fe4807640 1 -- 172.21.15.125:0/1744094786 >> v1:172.21.15.136:6789/0 conn(0x7f7fe00cf160 legacy=0x7f7fe0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:36.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.254+0000 7f7fe4807640 1 -- 172.21.15.125:0/1744094786 >> v1:172.21.15.125:6789/0 conn(0x7f7fe00ca0e0 legacy=0x7f7fe015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:36.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.254+0000 7f7fe4807640 1 -- 172.21.15.125:0/1744094786 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe00cad70 con 0x7f7fe01538a0 2023-10-30T12:49:36.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.258+0000 7f7fe4807640 1 --2- 172.21.15.125:0/1744094786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fe01538a0 0x7f7fe015d6f0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009870 tx=0x7f7fdc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=981957386ef78764 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:36.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.258+0000 7f7fd77fe640 1 -- 172.21.15.125:0/1744094786 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc0a89f0 con 0x7f7fe01538a0 2023-10-30T12:49:36.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.258+0000 7f7fd77fe640 1 -- 172.21.15.125:0/1744094786 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fdc0a8b50 con 0x7f7fe01538a0 2023-10-30T12:49:36.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.258+0000 7f7fd77fe640 1 -- 172.21.15.125:0/1744094786 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fdc0b0440 con 0x7f7fe01538a0 2023-10-30T12:49:36.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.258+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/1744094786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fe01538a0 msgr2=0x7f7fe015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:36.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.258+0000 7f7fe6a6b640 1 --2- 172.21.15.125:0/1744094786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fe01538a0 0x7f7fe015d6f0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009870 tx=0x7f7fdc01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:49:36.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.258+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/1744094786 shutdown_connections 2023-10-30T12:49:36.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.258+0000 7f7fe6a6b640 1 --2- 172.21.15.125:0/1744094786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fe01538a0 0x7f7fe015d6f0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:36.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.258+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/1744094786 >> 172.21.15.125:0/1744094786 conn(0x7f7fe006bfe0 msgr2=0x7f7fe006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:36.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.258+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/1744094786 shutdown_connections 2023-10-30T12:49:36.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.258+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/1744094786 wait complete. 2023-10-30T12:49:36.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe6a6b640 1 Processor -- start 2023-10-30T12:49:36.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe6a6b640 1 -- start start 2023-10-30T12:49:36.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe6a6b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fe00ca0e0 0x7f7fe0151b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:36.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe6a6b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fe00cf160 0x7f7fe01e3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:36.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe6a6b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fe01538a0 0x7f7fe00b4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:36.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe6a6b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe01f25a0 con 0x7f7fe01538a0 2023-10-30T12:49:36.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe6a6b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe0158a20 con 0x7f7fe00cf160 2023-10-30T12:49:36.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe6a6b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe01f27f0 con 0x7f7fe00ca0e0 2023-10-30T12:49:36.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe5008640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fe01538a0 0x7f7fe00b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:36.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe5008640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fe01538a0 0x7f7fe00b4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:32770/0 (socket says 172.21.15.125:32770) 2023-10-30T12:49:36.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe5008640 1 -- 172.21.15.125:0/960766073 learned_addr learned my addr 172.21.15.125:0/960766073 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:36.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fd7fff640 1 --2- 172.21.15.125:0/960766073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fe00cf160 0x7f7fe01e3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:36.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe4807640 1 --2- 172.21.15.125:0/960766073 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fe00ca0e0 0x7f7fe0151b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:36.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe5008640 1 -- 172.21.15.125:0/960766073 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fe00ca0e0 msgr2=0x7f7fe0151b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:36.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe5008640 1 --2- 172.21.15.125:0/960766073 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fe00ca0e0 0x7f7fe0151b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:36.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe5008640 1 -- 172.21.15.125:0/960766073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fe00cf160 msgr2=0x7f7fe01e3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:36.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe5008640 1 --2- 172.21.15.125:0/960766073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fe00cf160 0x7f7fe01e3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:36.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe5008640 1 -- 172.21.15.125:0/960766073 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc0094d0 con 0x7f7fe01538a0 2023-10-30T12:49:36.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe5008640 1 --2- 172.21.15.125:0/960766073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fe01538a0 0x7f7fe00b4320 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800c780 tx=0x7f7fd800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:36.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fd57fa640 1 -- 172.21.15.125:0/960766073 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd8004b20 con 0x7f7fe01538a0 2023-10-30T12:49:36.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fd57fa640 1 -- 172.21.15.125:0/960766073 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd8004c80 con 0x7f7fe01538a0 2023-10-30T12:49:36.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fd57fa640 1 -- 172.21.15.125:0/960766073 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fd800aa90 con 0x7f7fe01538a0 2023-10-30T12:49:36.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/960766073 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe01f2ad0 con 0x7f7fe01538a0 2023-10-30T12:49:36.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.262+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/960766073 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe0158610 con 0x7f7fe01538a0 2023-10-30T12:49:36.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.266+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/960766073 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe004e170 con 0x7f7fe01538a0 2023-10-30T12:49:36.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.266+0000 7f7fd57fa640 1 -- 172.21.15.125:0/960766073 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7fd800abf0 con 0x7f7fe01538a0 2023-10-30T12:49:36.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.266+0000 7f7fd57fa640 1 --2- 172.21.15.125:0/960766073 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fcc0664a0 0x7f7fcc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:36.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.266+0000 7f7fd57fa640 1 -- 172.21.15.125:0/960766073 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f7fd80841f0 con 0x7f7fe01538a0 2023-10-30T12:49:36.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.266+0000 7f7fe4807640 1 --2- 172.21.15.125:0/960766073 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fcc0664a0 0x7f7fcc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:36.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.266+0000 7f7fe4807640 1 --2- 172.21.15.125:0/960766073 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fcc0664a0 0x7f7fcc068960 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0b2000 tx=0x7f7fdc0aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:36.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.270+0000 7f7fd57fa640 1 -- 172.21.15.125:0/960766073 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7fd8051300 con 0x7f7fe01538a0 2023-10-30T12:49:36.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.458+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/960766073 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7fe01f2f80 con 0x7f7fe01538a0 2023-10-30T12:49:36.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.458+0000 7f7fd57fa640 1 -- 172.21.15.125:0/960766073 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7f7fd8054920 con 0x7f7fe01538a0 2023-10-30T12:49:36.457 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:36.458 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:36.458 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:36.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.458+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/960766073 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fcc0664a0 msgr2=0x7f7fcc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:36.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.458+0000 7f7fe6a6b640 1 --2- 172.21.15.125:0/960766073 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fcc0664a0 0x7f7fcc068960 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc0b2000 tx=0x7f7fdc0aa070 comp rx=0 tx=0).stop 2023-10-30T12:49:36.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.458+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/960766073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fe01538a0 msgr2=0x7f7fe00b4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:36.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.458+0000 7f7fe6a6b640 1 --2- 172.21.15.125:0/960766073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fe01538a0 0x7f7fe00b4320 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f7fd800c780 tx=0x7f7fd800cc40 comp rx=0 tx=0).stop 2023-10-30T12:49:36.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.458+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/960766073 shutdown_connections 2023-10-30T12:49:36.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.458+0000 7f7fe6a6b640 1 --2- 172.21.15.125:0/960766073 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fcc0664a0 0x7f7fcc068960 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:36.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.458+0000 7f7fe6a6b640 1 --2- 172.21.15.125:0/960766073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fe01538a0 0x7f7fe00b4320 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:36.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.458+0000 7f7fe6a6b640 1 --2- 172.21.15.125:0/960766073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fe00cf160 0x7f7fe01e3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:36.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.458+0000 7f7fe6a6b640 1 --2- 172.21.15.125:0/960766073 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7fe00ca0e0 0x7f7fe0151b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:36.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.458+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/960766073 >> 172.21.15.125:0/960766073 conn(0x7f7fe006bfe0 msgr2=0x7f7fe00c4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:36.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.458+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/960766073 shutdown_connections 2023-10-30T12:49:36.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:36.458+0000 7f7fe6a6b640 1 -- 172.21.15.125:0/960766073 wait complete. 2023-10-30T12:49:36.478 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T12:49:36.479 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6223": { "addr": "172.21.15.136:6839/3479573317", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3479573317, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3479573317, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6223, "incarnation": 100, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:stopping", "state_seq": 16 }, "gid_6318": { "addr": "172.21.15.125:6836/1721514617", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1721514617, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1721514617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 6318, "incarnation": 92, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:49:12.327796+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6318, "mds_1": 6223 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6837/2408701650", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2408701650, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2408701650, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 105, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6627, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:49:38.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:38.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.630+0000 7f857c0d6640 1 Processor -- start 2023-10-30T12:49:38.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.630+0000 7f857c0d6640 1 -- start start 2023-10-30T12:49:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f857c0d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8574070740 0x7f85740c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f857c0d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85740c5810 con 0x7f85741580e0 2023-10-30T12:49:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f857c0d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f857415eb90 con 0x7f857415e700 2023-10-30T12:49:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f857c0d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857406acb0 con 0x7f8574070740 2023-10-30T12:49:38.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8579e72640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f85741580e0 0x7f8574069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47898/0 (socket says 172.21.15.125:47898) 2023-10-30T12:49:38.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8579e72640 1 -- 172.21.15.125:0/709104575 learned_addr learned my addr 172.21.15.125:0/709104575 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:49:38.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8579671640 1 --2- 172.21.15.125:0/709104575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8574070740 0x7f85740c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:38.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 -- 172.21.15.125:0/709104575 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477407500 0 0) 0x7f85740c5810 con 0x7f85741580e0 2023-10-30T12:49:38.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 -- 172.21.15.125:0/709104575 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8560003530 con 0x7f85741580e0 2023-10-30T12:49:38.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 -- 172.21.15.125:0/709104575 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 21548118 0 0) 0x7f8560003530 con 0x7f85741580e0 2023-10-30T12:49:38.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 -- 172.21.15.125:0/709104575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8574070740 msgr2=0x7f85740c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:38.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 --2- 172.21.15.125:0/709104575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8574070740 0x7f85740c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:38.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 -- 172.21.15.125:0/709104575 >> v1:172.21.15.136:6789/0 conn(0x7f857415e700 legacy=0x7f857406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:38.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 -- 172.21.15.125:0/709104575 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85740c6bb0 con 0x7f85741580e0 2023-10-30T12:49:38.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 -- 172.21.15.125:0/709104575 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f8570001680 con 0x7f85741580e0 2023-10-30T12:49:38.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 -- 172.21.15.125:0/709104575 >> v1:172.21.15.125:6789/0 conn(0x7f85741580e0 legacy=0x7f8574069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:38.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 --2- 172.21.15.125:0/709104575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8574070740 0x7f85600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:38.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 --2- 172.21.15.125:0/709104575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8560003ef0 0x7f85600047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:38.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 --2- 172.21.15.125:0/709104575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8560004d10 0x7f8560005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:38.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 -- 172.21.15.125:0/709104575 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856000b9c0 con 0x7f8560003ef0 2023-10-30T12:49:38.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 -- 172.21.15.125:0/709104575 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f856000bb30 con 0x7f8560004d10 2023-10-30T12:49:38.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8578e70640 1 -- 172.21.15.125:0/709104575 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f856000bcf0 con 0x7f8574070740 2023-10-30T12:49:38.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8579671640 1 --2- 172.21.15.125:0/709104575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8560003ef0 0x7f85600047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:38.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8579e72640 1 --2- 172.21.15.125:0/709104575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8574070740 0x7f85600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:38.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f857a673640 1 --2- 172.21.15.125:0/709104575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8560004d10 0x7f8560005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:38.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8579671640 1 -- 172.21.15.125:0/709104575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8574070740 msgr2=0x7f85600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:38.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8579671640 1 --2- 172.21.15.125:0/709104575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8574070740 0x7f85600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:38.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8579671640 1 -- 172.21.15.125:0/709104575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8560004d10 msgr2=0x7f8560005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:38.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8579671640 1 --2- 172.21.15.125:0/709104575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8560004d10 0x7f8560005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:38.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.634+0000 7f8579671640 1 -- 172.21.15.125:0/709104575 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8560003530 con 0x7f8560003ef0 2023-10-30T12:49:38.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.638+0000 7f8579671640 1 --2- 172.21.15.125:0/709104575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8560003ef0 0x7f85600047d0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f8564009ae0 tx=0x7f856401c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87348bf2f9084c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:38.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.638+0000 7f8578e70640 1 -- 172.21.15.125:0/709104575 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85640a5580 con 0x7f8560003ef0 2023-10-30T12:49:38.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.638+0000 7f8578e70640 1 -- 172.21.15.125:0/709104575 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85640a56e0 con 0x7f8560003ef0 2023-10-30T12:49:38.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.638+0000 7f8578e70640 1 -- 172.21.15.125:0/709104575 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85640a5a90 con 0x7f8560003ef0 2023-10-30T12:49:38.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.638+0000 7f857c0d6640 1 -- 172.21.15.125:0/709104575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8560003ef0 msgr2=0x7f85600047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:38.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.638+0000 7f857c0d6640 1 --2- 172.21.15.125:0/709104575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8560003ef0 0x7f85600047d0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f8564009ae0 tx=0x7f856401c990 comp rx=0 tx=0).stop 2023-10-30T12:49:38.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.638+0000 7f857a673640 1 -- 172.21.15.125:0/709104575 reap_dead start 2023-10-30T12:49:38.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.638+0000 7f857c0d6640 1 -- 172.21.15.125:0/709104575 shutdown_connections 2023-10-30T12:49:38.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.638+0000 7f857c0d6640 1 -- 172.21.15.125:0/709104575 >> 172.21.15.125:0/709104575 conn(0x7f85740c7910 msgr2=0x7f85740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:38.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.638+0000 7f857c0d6640 1 -- 172.21.15.125:0/709104575 shutdown_connections 2023-10-30T12:49:38.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.638+0000 7f857c0d6640 1 -- 172.21.15.125:0/709104575 wait complete. 2023-10-30T12:49:38.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.638+0000 7f857c0d6640 1 Processor -- start 2023-10-30T12:49:38.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.638+0000 7f857c0d6640 1 -- start start 2023-10-30T12:49:38.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f857c0d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8560003ef0 0x7f85740c2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:38.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f857c0d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f857415f5b0 0x7f85740c30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:38.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f857c0d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85740c4340 0x7f85740c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:38.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f857c0d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85740c21d0 con 0x7f8560003ef0 2023-10-30T12:49:38.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f857c0d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85740c2790 con 0x7f85740c4340 2023-10-30T12:49:38.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f857c0d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85740c5040 con 0x7f857415f5b0 2023-10-30T12:49:38.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f8579e72640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8560003ef0 0x7f85740c2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:38.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f8579e72640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8560003ef0 0x7f85740c2bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:32810/0 (socket says 172.21.15.125:32810) 2023-10-30T12:49:38.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f8579e72640 1 -- 172.21.15.125:0/706984208 learned_addr learned my addr 172.21.15.125:0/706984208 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:38.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f8579671640 1 --2- 172.21.15.125:0/706984208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f857415f5b0 0x7f85740c30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:38.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f857a673640 1 --2- 172.21.15.125:0/706984208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85740c4340 0x7f85740c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:38.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f8579e72640 1 -- 172.21.15.125:0/706984208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f857415f5b0 msgr2=0x7f85740c30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:38.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f8579e72640 1 --2- 172.21.15.125:0/706984208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f857415f5b0 0x7f85740c30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:38.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f8579e72640 1 -- 172.21.15.125:0/706984208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85740c4340 msgr2=0x7f85740c47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:38.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f8579e72640 1 --2- 172.21.15.125:0/706984208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85740c4340 0x7f85740c47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:38.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f8579e72640 1 -- 172.21.15.125:0/706984208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857414bc20 con 0x7f8560003ef0 2023-10-30T12:49:38.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f8579e72640 1 --2- 172.21.15.125:0/706984208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8560003ef0 0x7f85740c2bb0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f857000e7d0 tx=0x7f857000ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:38.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f856a7fc640 1 -- 172.21.15.125:0/706984208 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857000eee0 con 0x7f8560003ef0 2023-10-30T12:49:38.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f856a7fc640 1 -- 172.21.15.125:0/706984208 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857000c8d0 con 0x7f8560003ef0 2023-10-30T12:49:38.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f856a7fc640 1 -- 172.21.15.125:0/706984208 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857000cc80 con 0x7f8560003ef0 2023-10-30T12:49:38.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f857c0d6640 1 -- 172.21.15.125:0/706984208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85740c5240 con 0x7f8560003ef0 2023-10-30T12:49:38.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f857c0d6640 1 -- 172.21.15.125:0/706984208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8574161440 con 0x7f8560003ef0 2023-10-30T12:49:38.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.642+0000 7f857c0d6640 1 -- 172.21.15.125:0/706984208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85740c28d0 con 0x7f8560003ef0 2023-10-30T12:49:38.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.646+0000 7f856a7fc640 1 -- 172.21.15.125:0/706984208 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f857001d460 con 0x7f8560003ef0 2023-10-30T12:49:38.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.646+0000 7f856a7fc640 1 --2- 172.21.15.125:0/706984208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85580664d0 0x7f8558068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:38.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.646+0000 7f856a7fc640 1 -- 172.21.15.125:0/706984208 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f85700840c0 con 0x7f8560003ef0 2023-10-30T12:49:38.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.646+0000 7f8579671640 1 --2- 172.21.15.125:0/706984208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85580664d0 0x7f8558068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:38.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.646+0000 7f8579671640 1 --2- 172.21.15.125:0/706984208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85580664d0 0x7f8558068990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f85640af000 tx=0x7f85640046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:38.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.650+0000 7f856a7fc640 1 -- 172.21.15.125:0/706984208 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f85740c28d0 con 0x7f8560003ef0 2023-10-30T12:49:38.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.830+0000 7f857c0d6640 1 -- 172.21.15.125:0/706984208 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85740c28d0 con 0x7f8560003ef0 2023-10-30T12:49:38.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.830+0000 7f856a7fc640 1 -- 172.21.15.125:0/706984208 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7f85740c28d0 con 0x7f8560003ef0 2023-10-30T12:49:38.832 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:38.832 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:38.832 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:38.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.834+0000 7f857c0d6640 1 -- 172.21.15.125:0/706984208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85580664d0 msgr2=0x7f8558068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:38.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.834+0000 7f857c0d6640 1 --2- 172.21.15.125:0/706984208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85580664d0 0x7f8558068990 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f85640af000 tx=0x7f85640046d0 comp rx=0 tx=0).stop 2023-10-30T12:49:38.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.834+0000 7f857c0d6640 1 -- 172.21.15.125:0/706984208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8560003ef0 msgr2=0x7f85740c2bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:38.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.834+0000 7f857c0d6640 1 --2- 172.21.15.125:0/706984208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8560003ef0 0x7f85740c2bb0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f857000e7d0 tx=0x7f857000ec90 comp rx=0 tx=0).stop 2023-10-30T12:49:38.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.834+0000 7f857c0d6640 1 -- 172.21.15.125:0/706984208 shutdown_connections 2023-10-30T12:49:38.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.834+0000 7f857c0d6640 1 --2- 172.21.15.125:0/706984208 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85580664d0 0x7f8558068990 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:38.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.834+0000 7f857c0d6640 1 --2- 172.21.15.125:0/706984208 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85740c4340 0x7f85740c47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:38.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.834+0000 7f857c0d6640 1 --2- 172.21.15.125:0/706984208 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f857415f5b0 0x7f85740c30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:38.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.834+0000 7f857c0d6640 1 --2- 172.21.15.125:0/706984208 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8560003ef0 0x7f85740c2bb0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:38.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.834+0000 7f857c0d6640 1 -- 172.21.15.125:0/706984208 >> 172.21.15.125:0/706984208 conn(0x7f85740c7910 msgr2=0x7f8574071740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:38.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.834+0000 7f857c0d6640 1 -- 172.21.15.125:0/706984208 shutdown_connections 2023-10-30T12:49:38.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:38.834+0000 7f857c0d6640 1 -- 172.21.15.125:0/706984208 wait complete. 2023-10-30T12:49:38.852 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:40.853 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:41.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.002+0000 7f2043a54640 1 Processor -- start 2023-10-30T12:49:41.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.002+0000 7f2043a54640 1 -- start start 2023-10-30T12:49:41.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.002+0000 7f2043a54640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f203c1580e0 0x7f203c070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:41.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.002+0000 7f2043a54640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c0c6540 0x7f203c15e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:41.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.002+0000 7f2043a54640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c0c5810 con 0x7f203c069e90 2023-10-30T12:49:41.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.002+0000 7f2043a54640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f203c0c6960 con 0x7f203c1580e0 2023-10-30T12:49:41.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.002+0000 7f2043a54640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203c0c6bb0 con 0x7f203c0c6540 2023-10-30T12:49:41.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.002+0000 7f2040fef640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c0c6540 0x7f203c15e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:41.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.002+0000 7f20417f0640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f203c1580e0 0x7f203c070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:41.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.002+0000 7f2040fef640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c0c6540 0x7f203c15e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:45722/0 (socket says 172.21.15.125:45722) 2023-10-30T12:49:41.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.002+0000 7f2040fef640 1 -- 172.21.15.125:0/1370851023 learned_addr learned my addr 172.21.15.125:0/1370851023 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.006+0000 7f2040fef640 1 -- 172.21.15.125:0/1370851023 >> v2:172.21.15.136:3300/0 conn(0x7f203c1580e0 msgr2=0x7f203c070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.006+0000 7f2040fef640 1 --2- 172.21.15.125:0/1370851023 >> v2:172.21.15.136:3300/0 conn(0x7f203c1580e0 0x7f203c070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.006+0000 7f2040fef640 1 -- 172.21.15.125:0/1370851023 >> v1:172.21.15.136:6789/0 conn(0x7f203c069e90 legacy=0x7f203c06a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:41.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.006+0000 7f2040fef640 1 -- 172.21.15.125:0/1370851023 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c0c6e70 con 0x7f203c0c6540 2023-10-30T12:49:41.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.006+0000 7f20417f0640 1 --2- 172.21.15.125:0/1370851023 >> v2:172.21.15.136:3300/0 conn(0x7f203c1580e0 0x7f203c070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:49:41.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.006+0000 7f2040fef640 1 --2- 172.21.15.125:0/1370851023 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c0c6540 0x7f203c15e700 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f202c009870 tx=0x7f202c01c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaccdeb33531bbb9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:41.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.006+0000 7f2033fff640 1 -- 172.21.15.125:0/1370851023 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c0a9400 con 0x7f203c0c6540 2023-10-30T12:49:41.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.006+0000 7f2033fff640 1 -- 172.21.15.125:0/1370851023 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c0a9560 con 0x7f203c0c6540 2023-10-30T12:49:41.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.006+0000 7f2033fff640 1 -- 172.21.15.125:0/1370851023 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c0a98f0 con 0x7f203c0c6540 2023-10-30T12:49:41.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 -- 172.21.15.125:0/1370851023 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c0c6540 msgr2=0x7f203c15e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:41.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 --2- 172.21.15.125:0/1370851023 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c0c6540 0x7f203c15e700 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f202c009870 tx=0x7f202c01c540 comp rx=0 tx=0).stop 2023-10-30T12:49:41.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 -- 172.21.15.125:0/1370851023 shutdown_connections 2023-10-30T12:49:41.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 --2- 172.21.15.125:0/1370851023 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c0c6540 0x7f203c15e700 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:41.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 --2- 172.21.15.125:0/1370851023 >> v2:172.21.15.136:3300/0 conn(0x7f203c1580e0 0x7f203c070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:41.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 -- 172.21.15.125:0/1370851023 >> 172.21.15.125:0/1370851023 conn(0x7f203c0c7910 msgr2=0x7f203c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:41.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 -- 172.21.15.125:0/1370851023 shutdown_connections 2023-10-30T12:49:41.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 -- 172.21.15.125:0/1370851023 wait complete. 2023-10-30T12:49:41.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 Processor -- start 2023-10-30T12:49:41.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 -- start start 2023-10-30T12:49:41.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f203c069e90 0x7f203c0c9e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:41.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c0c6540 0x7f203c0c4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:41.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f203c1580e0 0x7f203c0c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:41.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c147a70 con 0x7f203c069e90 2023-10-30T12:49:41.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c0ca370 con 0x7f203c1580e0 2023-10-30T12:49:41.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f2043a54640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203c1f7e70 con 0x7f203c0c6540 2023-10-30T12:49:41.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.010+0000 7f20417f0640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f203c069e90 0x7f203c0c9e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:41.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f20417f0640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f203c069e90 0x7f203c0c9e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55242/0 (socket says 172.21.15.125:55242) 2023-10-30T12:49:41.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f20417f0640 1 -- 172.21.15.125:0/867787966 learned_addr learned my addr 172.21.15.125:0/867787966 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:41.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f2041ff1640 1 --2- 172.21.15.125:0/867787966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f203c1580e0 0x7f203c0c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:41.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f20417f0640 1 -- 172.21.15.125:0/867787966 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c0c6540 msgr2=0x7f203c0c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:41.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f20417f0640 1 --2- 172.21.15.125:0/867787966 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c0c6540 0x7f203c0c4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:41.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f20417f0640 1 -- 172.21.15.125:0/867787966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f203c1580e0 msgr2=0x7f203c0c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:41.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f20417f0640 1 --2- 172.21.15.125:0/867787966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f203c1580e0 0x7f203c0c12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:41.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f20417f0640 1 -- 172.21.15.125:0/867787966 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2038009420 con 0x7f203c069e90 2023-10-30T12:49:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f20417f0640 1 --2- 172.21.15.125:0/867787966 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f203c069e90 0x7f203c0c9e30 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f203800e9b0 tx=0x7f203800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f2031ffb640 1 -- 172.21.15.125:0/867787966 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f203800ca00 con 0x7f203c069e90 2023-10-30T12:49:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f2031ffb640 1 -- 172.21.15.125:0/867787966 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f203800cb60 con 0x7f203c069e90 2023-10-30T12:49:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f2031ffb640 1 -- 172.21.15.125:0/867787966 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2038013470 con 0x7f203c069e90 2023-10-30T12:49:41.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f2043a54640 1 -- 172.21.15.125:0/867787966 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202c0094d0 con 0x7f203c069e90 2023-10-30T12:49:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f2043a54640 1 -- 172.21.15.125:0/867787966 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f203c0c9870 con 0x7f203c069e90 2023-10-30T12:49:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f2043a54640 1 -- 172.21.15.125:0/867787966 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203c04e170 con 0x7f203c069e90 2023-10-30T12:49:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.014+0000 7f2031ffb640 1 -- 172.21.15.125:0/867787966 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f20380135d0 con 0x7f203c069e90 2023-10-30T12:49:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.018+0000 7f2031ffb640 1 --2- 172.21.15.125:0/867787966 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f20200664d0 0x7f2020068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:41.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.018+0000 7f2031ffb640 1 -- 172.21.15.125:0/867787966 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f2038084740 con 0x7f203c069e90 2023-10-30T12:49:41.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.018+0000 7f2040fef640 1 --2- 172.21.15.125:0/867787966 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f20200664d0 0x7f2020068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:41.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.018+0000 7f2040fef640 1 --2- 172.21.15.125:0/867787966 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f20200664d0 0x7f2020068990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f202c0ae000 tx=0x7f202c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:41.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.022+0000 7f2031ffb640 1 -- 172.21.15.125:0/867787966 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f20380518d0 con 0x7f203c069e90 2023-10-30T12:49:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.202+0000 7f2043a54640 1 -- 172.21.15.125:0/867787966 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f203c0c5ff0 con 0x7f203c069e90 2023-10-30T12:49:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.202+0000 7f2031ffb640 1 -- 172.21.15.125:0/867787966 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7f2038054ef0 con 0x7f203c069e90 2023-10-30T12:49:41.201 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:41.201 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:41.202 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:41.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.202+0000 7f2043a54640 1 -- 172.21.15.125:0/867787966 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f20200664d0 msgr2=0x7f2020068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:41.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.202+0000 7f2043a54640 1 --2- 172.21.15.125:0/867787966 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f20200664d0 0x7f2020068990 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f202c0ae000 tx=0x7f202c008040 comp rx=0 tx=0).stop 2023-10-30T12:49:41.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.202+0000 7f2043a54640 1 -- 172.21.15.125:0/867787966 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f203c069e90 msgr2=0x7f203c0c9e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:41.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.202+0000 7f2043a54640 1 --2- 172.21.15.125:0/867787966 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f203c069e90 0x7f203c0c9e30 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f203800e9b0 tx=0x7f203800ee70 comp rx=0 tx=0).stop 2023-10-30T12:49:41.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.202+0000 7f2043a54640 1 -- 172.21.15.125:0/867787966 shutdown_connections 2023-10-30T12:49:41.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.202+0000 7f2043a54640 1 --2- 172.21.15.125:0/867787966 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f20200664d0 0x7f2020068990 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:41.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.202+0000 7f2043a54640 1 --2- 172.21.15.125:0/867787966 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f203c1580e0 0x7f203c0c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.202+0000 7f2043a54640 1 --2- 172.21.15.125:0/867787966 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c0c6540 0x7f203c0c4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.202+0000 7f2043a54640 1 --2- 172.21.15.125:0/867787966 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f203c069e90 0x7f203c0c9e30 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.202+0000 7f2043a54640 1 -- 172.21.15.125:0/867787966 >> 172.21.15.125:0/867787966 conn(0x7f203c0c7910 msgr2=0x7f203c155f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.202+0000 7f2043a54640 1 -- 172.21.15.125:0/867787966 shutdown_connections 2023-10-30T12:49:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:41.202+0000 7f2043a54640 1 -- 172.21.15.125:0/867787966 wait complete. 2023-10-30T12:49:41.221 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:43.222 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:43.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.370+0000 7f90c097f640 1 Processor -- start 2023-10-30T12:49:43.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.370+0000 7f90c097f640 1 -- start start 2023-10-30T12:49:43.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.374+0000 7f90c097f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f90bc1538f0 0x7f90bc159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:43.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.374+0000 7f90c097f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f90bc15cd50 0x7f90bc0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:43.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.374+0000 7f90c097f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90bc15d300 con 0x7f90bc0cefa0 2023-10-30T12:49:43.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.374+0000 7f90c097f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90bc15d5e0 con 0x7f90bc15cd50 2023-10-30T12:49:43.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.374+0000 7f90c097f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90bc15d8c0 con 0x7f90bc1538f0 2023-10-30T12:49:43.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.374+0000 7f90ba59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f90bc15cd50 0x7f90bc0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:43.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.374+0000 7f90ba59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f90bc15cd50 0x7f90bc0c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55248/0 (socket says 172.21.15.125:55248) 2023-10-30T12:49:43.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.374+0000 7f90ba59c640 1 -- 172.21.15.125:0/3166033321 learned_addr learned my addr 172.21.15.125:0/3166033321 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:43.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.374+0000 7f90bad9d640 1 --2- 172.21.15.125:0/3166033321 >> v2:172.21.15.136:3300/0 conn(0x7f90bc1538f0 0x7f90bc159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:43.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.374+0000 7f90ba59c640 1 -- 172.21.15.125:0/3166033321 >> v2:172.21.15.136:3300/0 conn(0x7f90bc1538f0 msgr2=0x7f90bc159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:43.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.374+0000 7f90ba59c640 1 --2- 172.21.15.125:0/3166033321 >> v2:172.21.15.136:3300/0 conn(0x7f90bc1538f0 0x7f90bc159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:43.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.374+0000 7f90ba59c640 1 -- 172.21.15.125:0/3166033321 >> v1:172.21.15.136:6789/0 conn(0x7f90bc0cefa0 legacy=0x7f90bc0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:43.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.374+0000 7f90ba59c640 1 -- 172.21.15.125:0/3166033321 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90bc15dba0 con 0x7f90bc15cd50 2023-10-30T12:49:43.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.378+0000 7f90ba59c640 1 --2- 172.21.15.125:0/3166033321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90bc15cd50 0x7f90bc0c9f90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0c9cb0 tx=0x7f90b001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1050201ab797298b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:43.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.378+0000 7f90b9d9b640 1 -- 172.21.15.125:0/3166033321 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b00a89b0 con 0x7f90bc15cd50 2023-10-30T12:49:43.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.378+0000 7f90b9d9b640 1 -- 172.21.15.125:0/3166033321 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b00a8b10 con 0x7f90bc15cd50 2023-10-30T12:49:43.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.378+0000 7f90b9d9b640 1 -- 172.21.15.125:0/3166033321 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b00a8ea0 con 0x7f90bc15cd50 2023-10-30T12:49:43.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.378+0000 7f90c097f640 1 -- 172.21.15.125:0/3166033321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90bc15cd50 msgr2=0x7f90bc0c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:43.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.378+0000 7f90c097f640 1 --2- 172.21.15.125:0/3166033321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90bc15cd50 0x7f90bc0c9f90 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f90bc0c9cb0 tx=0x7f90b001ecb0 comp rx=0 tx=0).stop 2023-10-30T12:49:43.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.378+0000 7f90c097f640 1 -- 172.21.15.125:0/3166033321 shutdown_connections 2023-10-30T12:49:43.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.378+0000 7f90c097f640 1 --2- 172.21.15.125:0/3166033321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90bc15cd50 0x7f90bc0c9f90 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:43.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.378+0000 7f90c097f640 1 --2- 172.21.15.125:0/3166033321 >> v2:172.21.15.136:3300/0 conn(0x7f90bc1538f0 0x7f90bc159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:43.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.378+0000 7f90c097f640 1 -- 172.21.15.125:0/3166033321 >> 172.21.15.125:0/3166033321 conn(0x7f90bc06a640 msgr2=0x7f90bc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:43.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.378+0000 7f90c097f640 1 -- 172.21.15.125:0/3166033321 shutdown_connections 2023-10-30T12:49:43.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.378+0000 7f90c097f640 1 -- 172.21.15.125:0/3166033321 wait complete. 2023-10-30T12:49:43.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.378+0000 7f90c097f640 1 Processor -- start 2023-10-30T12:49:43.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90c097f640 1 -- start start 2023-10-30T12:49:43.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90c097f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90bc0cefa0 0x7f90bc147a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:43.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90c097f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90bc1538f0 0x7f90bc147f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:43.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90c097f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90bc15cd50 0x7f90bc1579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:43.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90c097f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc158fd0 con 0x7f90bc15cd50 2023-10-30T12:49:43.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90c097f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc0b4a00 con 0x7f90bc0cefa0 2023-10-30T12:49:43.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90c097f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90bc0b3c20 con 0x7f90bc1538f0 2023-10-30T12:49:43.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90bb59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90bc15cd50 0x7f90bc1579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:43.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90bb59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90bc15cd50 0x7f90bc1579f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55256/0 (socket says 172.21.15.125:55256) 2023-10-30T12:49:43.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90bb59e640 1 -- 172.21.15.125:0/559067792 learned_addr learned my addr 172.21.15.125:0/559067792 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:43.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90bad9d640 1 --2- 172.21.15.125:0/559067792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90bc0cefa0 0x7f90bc147a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:43.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90ba59c640 1 --2- 172.21.15.125:0/559067792 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90bc1538f0 0x7f90bc147f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:43.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90bb59e640 1 -- 172.21.15.125:0/559067792 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90bc1538f0 msgr2=0x7f90bc147f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:43.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90bb59e640 1 --2- 172.21.15.125:0/559067792 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90bc1538f0 0x7f90bc147f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:43.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90bb59e640 1 -- 172.21.15.125:0/559067792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90bc0cefa0 msgr2=0x7f90bc147a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:43.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90bb59e640 1 --2- 172.21.15.125:0/559067792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90bc0cefa0 0x7f90bc147a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:43.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90bb59e640 1 -- 172.21.15.125:0/559067792 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90b00094d0 con 0x7f90bc15cd50 2023-10-30T12:49:43.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90bb59e640 1 --2- 172.21.15.125:0/559067792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90bc15cd50 0x7f90bc1579f0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f90b400c780 tx=0x7f90b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:43.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90a37fe640 1 -- 172.21.15.125:0/559067792 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b4004b70 con 0x7f90bc15cd50 2023-10-30T12:49:43.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90a37fe640 1 -- 172.21.15.125:0/559067792 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90b400ce90 con 0x7f90bc15cd50 2023-10-30T12:49:43.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90a37fe640 1 -- 172.21.15.125:0/559067792 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90b400a850 con 0x7f90bc15cd50 2023-10-30T12:49:43.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90c097f640 1 -- 172.21.15.125:0/559067792 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90bc0b3ec0 con 0x7f90bc15cd50 2023-10-30T12:49:43.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90c097f640 1 -- 172.21.15.125:0/559067792 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90bc1530e0 con 0x7f90bc15cd50 2023-10-30T12:49:43.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.382+0000 7f90c097f640 1 -- 172.21.15.125:0/559067792 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90bc04e170 con 0x7f90bc15cd50 2023-10-30T12:49:43.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.386+0000 7f90a37fe640 1 -- 172.21.15.125:0/559067792 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f90b400a9b0 con 0x7f90bc15cd50 2023-10-30T12:49:43.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.386+0000 7f90a37fe640 1 --2- 172.21.15.125:0/559067792 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90a80664a0 0x7f90a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:43.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.386+0000 7f90a37fe640 1 -- 172.21.15.125:0/559067792 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f90b4084350 con 0x7f90bc15cd50 2023-10-30T12:49:43.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.386+0000 7f90bad9d640 1 --2- 172.21.15.125:0/559067792 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90a80664a0 0x7f90a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:43.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.386+0000 7f90bad9d640 1 --2- 172.21.15.125:0/559067792 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90a80664a0 0x7f90a8068960 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f90ac007a60 tx=0x7f90ac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:43.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.390+0000 7f90a37fe640 1 -- 172.21.15.125:0/559067792 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f90b4051460 con 0x7f90bc15cd50 2023-10-30T12:49:43.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.570+0000 7f90c097f640 1 -- 172.21.15.125:0/559067792 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90bc0cb130 con 0x7f90bc15cd50 2023-10-30T12:49:43.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.570+0000 7f90a37fe640 1 -- 172.21.15.125:0/559067792 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7f90b4054a80 con 0x7f90bc15cd50 2023-10-30T12:49:43.570 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:43.571 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:43.571 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:43.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.574+0000 7f90c097f640 1 -- 172.21.15.125:0/559067792 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90a80664a0 msgr2=0x7f90a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:43.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.574+0000 7f90c097f640 1 --2- 172.21.15.125:0/559067792 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90a80664a0 0x7f90a8068960 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f90ac007a60 tx=0x7f90ac008040 comp rx=0 tx=0).stop 2023-10-30T12:49:43.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.574+0000 7f90c097f640 1 -- 172.21.15.125:0/559067792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90bc15cd50 msgr2=0x7f90bc1579f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:43.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.574+0000 7f90c097f640 1 --2- 172.21.15.125:0/559067792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90bc15cd50 0x7f90bc1579f0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f90b400c780 tx=0x7f90b400cc40 comp rx=0 tx=0).stop 2023-10-30T12:49:43.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.574+0000 7f90c097f640 1 -- 172.21.15.125:0/559067792 shutdown_connections 2023-10-30T12:49:43.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.574+0000 7f90c097f640 1 --2- 172.21.15.125:0/559067792 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90a80664a0 0x7f90a8068960 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:43.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.574+0000 7f90c097f640 1 --2- 172.21.15.125:0/559067792 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90bc15cd50 0x7f90bc1579f0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:43.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.574+0000 7f90c097f640 1 --2- 172.21.15.125:0/559067792 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90bc1538f0 0x7f90bc147f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:43.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.574+0000 7f90c097f640 1 --2- 172.21.15.125:0/559067792 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90bc0cefa0 0x7f90bc147a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:43.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.574+0000 7f90c097f640 1 -- 172.21.15.125:0/559067792 >> 172.21.15.125:0/559067792 conn(0x7f90bc06a640 msgr2=0x7f90bc0c4450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:43.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.574+0000 7f90c097f640 1 -- 172.21.15.125:0/559067792 shutdown_connections 2023-10-30T12:49:43.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:43.574+0000 7f90c097f640 1 -- 172.21.15.125:0/559067792 wait complete. 2023-10-30T12:49:43.591 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:45.592 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f4513e640 1 Processor -- start 2023-10-30T12:49:45.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f4513e640 1 -- start start 2023-10-30T12:49:45.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f4513e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5f400c6b80 0x7f5f400bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:45.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f4513e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5f400c6f60 0x7f5f400c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:45.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f4513e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f400c0a50 con 0x7f5f400c8430 2023-10-30T12:49:45.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f4513e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4006f6e0 con 0x7f5f400c6b80 2023-10-30T12:49:45.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f4513e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4006f9c0 con 0x7f5f400c6f60 2023-10-30T12:49:45.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f3ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5f400c6b80 0x7f5f400bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:45.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f3ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5f400c6b80 0x7f5f400bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55268/0 (socket says 172.21.15.125:55268) 2023-10-30T12:49:45.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f3ed9d640 1 -- 172.21.15.125:0/3428524500 learned_addr learned my addr 172.21.15.125:0/3428524500 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:45.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f3659c640 1 --2- 172.21.15.125:0/3428524500 >> v2:172.21.15.136:3300/0 conn(0x7f5f400c6f60 0x7f5f400c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:45.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f3ed9d640 1 -- 172.21.15.125:0/3428524500 >> v2:172.21.15.136:3300/0 conn(0x7f5f400c6f60 msgr2=0x7f5f400c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:45.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f3ed9d640 1 --2- 172.21.15.125:0/3428524500 >> v2:172.21.15.136:3300/0 conn(0x7f5f400c6f60 0x7f5f400c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f3ed9d640 1 -- 172.21.15.125:0/3428524500 >> v1:172.21.15.136:6789/0 conn(0x7f5f400c8430 legacy=0x7f5f4013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.738+0000 7f5f3ed9d640 1 -- 172.21.15.125:0/3428524500 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4006fca0 con 0x7f5f400c6b80 2023-10-30T12:49:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.742+0000 7f5f3ed9d640 1 --2- 172.21.15.125:0/3428524500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f400c6b80 0x7f5f400bfec0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f5f40159b90 tx=0x7f5f3001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a5afd628e15b81 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.742+0000 7f5f3e59c640 1 -- 172.21.15.125:0/3428524500 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f300a57a0 con 0x7f5f400c6b80 2023-10-30T12:49:45.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.742+0000 7f5f3e59c640 1 -- 172.21.15.125:0/3428524500 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f300a5900 con 0x7f5f400c6b80 2023-10-30T12:49:45.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.742+0000 7f5f3e59c640 1 -- 172.21.15.125:0/3428524500 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f300a5c90 con 0x7f5f400c6b80 2023-10-30T12:49:45.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.742+0000 7f5f4513e640 1 -- 172.21.15.125:0/3428524500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f400c6b80 msgr2=0x7f5f400bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:45.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.742+0000 7f5f4513e640 1 --2- 172.21.15.125:0/3428524500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f400c6b80 0x7f5f400bfec0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f5f40159b90 tx=0x7f5f3001c9d0 comp rx=0 tx=0).stop 2023-10-30T12:49:45.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.742+0000 7f5f4513e640 1 -- 172.21.15.125:0/3428524500 shutdown_connections 2023-10-30T12:49:45.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.742+0000 7f5f4513e640 1 --2- 172.21.15.125:0/3428524500 >> v2:172.21.15.136:3300/0 conn(0x7f5f400c6f60 0x7f5f400c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:45.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.742+0000 7f5f4513e640 1 --2- 172.21.15.125:0/3428524500 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f400c6b80 0x7f5f400bfec0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:45.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.742+0000 7f5f4513e640 1 -- 172.21.15.125:0/3428524500 >> 172.21.15.125:0/3428524500 conn(0x7f5f400c7910 msgr2=0x7f5f400c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:45.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.742+0000 7f5f4513e640 1 -- 172.21.15.125:0/3428524500 shutdown_connections 2023-10-30T12:49:45.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.742+0000 7f5f4513e640 1 -- 172.21.15.125:0/3428524500 wait complete. 2023-10-30T12:49:45.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f4513e640 1 Processor -- start 2023-10-30T12:49:45.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f4513e640 1 -- start start 2023-10-30T12:49:45.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f4513e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f400c6b80 0x7f5f40154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:45.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f4513e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f400c6f60 0x7f5f40155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:45.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f4513e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f400c8430 0x7f5f401560b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:45.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f4513e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f4013f210 con 0x7f5f400c6f60 2023-10-30T12:49:45.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f4513e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f400c9fe0 con 0x7f5f400c6b80 2023-10-30T12:49:45.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f4513e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f400c5010 con 0x7f5f400c8430 2023-10-30T12:49:45.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f3659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f400c6f60 0x7f5f40155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:45.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f3659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f400c6f60 0x7f5f40155b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55274/0 (socket says 172.21.15.125:55274) 2023-10-30T12:49:45.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f3659c640 1 -- 172.21.15.125:0/877894180 learned_addr learned my addr 172.21.15.125:0/877894180 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:45.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f3ed9d640 1 --2- 172.21.15.125:0/877894180 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f400c6b80 0x7f5f40154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:45.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f3f59e640 1 --2- 172.21.15.125:0/877894180 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f400c8430 0x7f5f401560b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:45.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f3659c640 1 -- 172.21.15.125:0/877894180 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f400c8430 msgr2=0x7f5f401560b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:45.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f3659c640 1 --2- 172.21.15.125:0/877894180 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f400c8430 0x7f5f401560b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:45.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f3659c640 1 -- 172.21.15.125:0/877894180 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f400c6b80 msgr2=0x7f5f40154fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:45.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f3659c640 1 --2- 172.21.15.125:0/877894180 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f400c6b80 0x7f5f40154fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:45.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f3659c640 1 -- 172.21.15.125:0/877894180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f30009b30 con 0x7f5f400c6f60 2023-10-30T12:49:45.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f3659c640 1 --2- 172.21.15.125:0/877894180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f400c6f60 0x7f5f40155b70 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800d830 tx=0x7f5f2800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:45.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f37fff640 1 -- 172.21.15.125:0/877894180 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2800b660 con 0x7f5f400c6f60 2023-10-30T12:49:45.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f37fff640 1 -- 172.21.15.125:0/877894180 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2800b7c0 con 0x7f5f400c6f60 2023-10-30T12:49:45.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f37fff640 1 -- 172.21.15.125:0/877894180 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2800bb30 con 0x7f5f400c6f60 2023-10-30T12:49:45.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f4513e640 1 -- 172.21.15.125:0/877894180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f40155510 con 0x7f5f400c6f60 2023-10-30T12:49:45.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.746+0000 7f5f4513e640 1 -- 172.21.15.125:0/877894180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f4013e330 con 0x7f5f400c6f60 2023-10-30T12:49:45.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.750+0000 7f5f4513e640 1 -- 172.21.15.125:0/877894180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f4004e170 con 0x7f5f400c6f60 2023-10-30T12:49:45.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.750+0000 7f5f37fff640 1 -- 172.21.15.125:0/877894180 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5f2801e460 con 0x7f5f400c6f60 2023-10-30T12:49:45.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.750+0000 7f5f37fff640 1 --2- 172.21.15.125:0/877894180 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5f200664a0 0x7f5f20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:45.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.750+0000 7f5f37fff640 1 -- 172.21.15.125:0/877894180 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f5f28084a60 con 0x7f5f400c6f60 2023-10-30T12:49:45.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.750+0000 7f5f3ed9d640 1 --2- 172.21.15.125:0/877894180 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5f200664a0 0x7f5f20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:45.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.750+0000 7f5f3ed9d640 1 --2- 172.21.15.125:0/877894180 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5f200664a0 0x7f5f20068960 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5f300ae000 tx=0x7f5f300a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:45.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.754+0000 7f5f37fff640 1 -- 172.21.15.125:0/877894180 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5f28051a30 con 0x7f5f400c6f60 2023-10-30T12:49:45.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.934+0000 7f5f4513e640 1 -- 172.21.15.125:0/877894180 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f400c9d00 con 0x7f5f400c6f60 2023-10-30T12:49:45.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.938+0000 7f5f37fff640 1 -- 172.21.15.125:0/877894180 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7f5f28055050 con 0x7f5f400c6f60 2023-10-30T12:49:45.936 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:45.937 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:45.937 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:45.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.938+0000 7f5f4513e640 1 -- 172.21.15.125:0/877894180 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5f200664a0 msgr2=0x7f5f20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:45.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.938+0000 7f5f4513e640 1 --2- 172.21.15.125:0/877894180 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5f200664a0 0x7f5f20068960 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5f300ae000 tx=0x7f5f300a7070 comp rx=0 tx=0).stop 2023-10-30T12:49:45.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.938+0000 7f5f4513e640 1 -- 172.21.15.125:0/877894180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f400c6f60 msgr2=0x7f5f40155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:45.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.938+0000 7f5f4513e640 1 --2- 172.21.15.125:0/877894180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f400c6f60 0x7f5f40155b70 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f5f2800d830 tx=0x7f5f2800dcf0 comp rx=0 tx=0).stop 2023-10-30T12:49:45.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.938+0000 7f5f4513e640 1 -- 172.21.15.125:0/877894180 shutdown_connections 2023-10-30T12:49:45.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.938+0000 7f5f4513e640 1 --2- 172.21.15.125:0/877894180 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5f200664a0 0x7f5f20068960 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:45.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.938+0000 7f5f4513e640 1 --2- 172.21.15.125:0/877894180 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5f400c8430 0x7f5f401560b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:45.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.938+0000 7f5f4513e640 1 --2- 172.21.15.125:0/877894180 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5f400c6f60 0x7f5f40155b70 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:45.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.938+0000 7f5f4513e640 1 --2- 172.21.15.125:0/877894180 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5f400c6b80 0x7f5f40154fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:45.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.938+0000 7f5f4513e640 1 -- 172.21.15.125:0/877894180 >> 172.21.15.125:0/877894180 conn(0x7f5f400c7910 msgr2=0x7f5f400c17d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:45.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.938+0000 7f5f4513e640 1 -- 172.21.15.125:0/877894180 shutdown_connections 2023-10-30T12:49:45.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:45.938+0000 7f5f4513e640 1 -- 172.21.15.125:0/877894180 wait complete. 2023-10-30T12:49:45.958 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:47.438 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:49:47.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:49:47.959 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:48.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.106+0000 7fb3526dc640 1 Processor -- start 2023-10-30T12:49:48.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.106+0000 7fb3526dc640 1 -- start start 2023-10-30T12:49:48.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb3526dc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb34c06fef0 0x7fb34c161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:48.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb3526dc640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb34c161f60 con 0x7fb34c06ab20 2023-10-30T12:49:48.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb3526dc640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb34c1616c0 con 0x7fb34c157c80 2023-10-30T12:49:48.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb3526dc640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c15ef30 con 0x7fb34c06fef0 2023-10-30T12:49:48.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb351edb640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb34c06ab20 0x7fb34c070c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59094/0 (socket says 172.21.15.125:59094) 2023-10-30T12:49:48.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb351edb640 1 -- 172.21.15.125:0/3597895616 learned_addr learned my addr 172.21.15.125:0/3597895616 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:49:48.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb350ed9640 1 --2- 172.21.15.125:0/3597895616 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb34c06fef0 0x7fb34c161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:48.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 -- 172.21.15.125:0/3597895616 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2537531708 0 0) 0x7fb34c161f60 con 0x7fb34c06ab20 2023-10-30T12:49:48.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 -- 172.21.15.125:0/3597895616 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb338003530 con 0x7fb34c06ab20 2023-10-30T12:49:48.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 -- 172.21.15.125:0/3597895616 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2371703174 0 0) 0x7fb338003530 con 0x7fb34c06ab20 2023-10-30T12:49:48.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 -- 172.21.15.125:0/3597895616 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb34c06fef0 msgr2=0x7fb34c161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:48.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 --2- 172.21.15.125:0/3597895616 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb34c06fef0 0x7fb34c161030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:48.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 -- 172.21.15.125:0/3597895616 >> v1:172.21.15.136:6789/0 conn(0x7fb34c157c80 legacy=0x7fb34c06f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:48.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 -- 172.21.15.125:0/3597895616 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c1619a0 con 0x7fb34c06ab20 2023-10-30T12:49:48.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 -- 172.21.15.125:0/3597895616 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fb344001a00 con 0x7fb34c06ab20 2023-10-30T12:49:48.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 -- 172.21.15.125:0/3597895616 >> v1:172.21.15.125:6789/0 conn(0x7fb34c06ab20 legacy=0x7fb34c070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:48.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 --2- 172.21.15.125:0/3597895616 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb34c06fef0 0x7fb3380039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:48.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 --2- 172.21.15.125:0/3597895616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb338003f30 0x7fb3380047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:48.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 --2- 172.21.15.125:0/3597895616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb338004d10 0x7fb338005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:48.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 -- 172.21.15.125:0/3597895616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33800bad0 con 0x7fb338003f30 2023-10-30T12:49:48.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 -- 172.21.15.125:0/3597895616 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33800bc40 con 0x7fb338004d10 2023-10-30T12:49:48.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb343fff640 1 -- 172.21.15.125:0/3597895616 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33800be00 con 0x7fb34c06fef0 2023-10-30T12:49:48.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb350ed9640 1 --2- 172.21.15.125:0/3597895616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb338003f30 0x7fb3380047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:48.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb3516da640 1 --2- 172.21.15.125:0/3597895616 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb34c06fef0 0x7fb3380039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:48.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb351edb640 1 --2- 172.21.15.125:0/3597895616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb338004d10 0x7fb338005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:48.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb350ed9640 1 -- 172.21.15.125:0/3597895616 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb34c06fef0 msgr2=0x7fb3380039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:48.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb350ed9640 1 --2- 172.21.15.125:0/3597895616 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb34c06fef0 0x7fb3380039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:48.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb350ed9640 1 -- 172.21.15.125:0/3597895616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb338004d10 msgr2=0x7fb338005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:48.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb350ed9640 1 --2- 172.21.15.125:0/3597895616 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb338004d10 0x7fb338005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:48.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb350ed9640 1 -- 172.21.15.125:0/3597895616 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb338003530 con 0x7fb338003f30 2023-10-30T12:49:48.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.110+0000 7fb3516da640 1 --2- 172.21.15.125:0/3597895616 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb34c06fef0 0x7fb3380039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:48.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.114+0000 7fb350ed9640 1 --2- 172.21.15.125:0/3597895616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb338003f30 0x7fb3380047d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fb33c009ab0 tx=0x7fb33c01ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5949a20dfd37643b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:48.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.114+0000 7fb343fff640 1 -- 172.21.15.125:0/3597895616 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c0a5620 con 0x7fb338003f30 2023-10-30T12:49:48.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.114+0000 7fb343fff640 1 -- 172.21.15.125:0/3597895616 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33c0a5780 con 0x7fb338003f30 2023-10-30T12:49:48.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.114+0000 7fb343fff640 1 -- 172.21.15.125:0/3597895616 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33c0a5b30 con 0x7fb338003f30 2023-10-30T12:49:48.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.114+0000 7fb3526dc640 1 -- 172.21.15.125:0/3597895616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb338003f30 msgr2=0x7fb3380047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:48.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.114+0000 7fb3526dc640 1 --2- 172.21.15.125:0/3597895616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb338003f30 0x7fb3380047d0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fb33c009ab0 tx=0x7fb33c01ca30 comp rx=0 tx=0).stop 2023-10-30T12:49:48.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.114+0000 7fb351edb640 1 -- 172.21.15.125:0/3597895616 reap_dead start 2023-10-30T12:49:48.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.114+0000 7fb3526dc640 1 -- 172.21.15.125:0/3597895616 shutdown_connections 2023-10-30T12:49:48.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.114+0000 7fb3526dc640 1 -- 172.21.15.125:0/3597895616 >> 172.21.15.125:0/3597895616 conn(0x7fb34c0c75d0 msgr2=0x7fb34c0c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:48.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.114+0000 7fb3526dc640 1 -- 172.21.15.125:0/3597895616 shutdown_connections 2023-10-30T12:49:48.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.114+0000 7fb3526dc640 1 -- 172.21.15.125:0/3597895616 wait complete. 2023-10-30T12:49:48.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.114+0000 7fb3526dc640 1 Processor -- start 2023-10-30T12:49:48.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.114+0000 7fb3526dc640 1 -- start start 2023-10-30T12:49:48.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb3526dc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb338003f30 0x7fb34c0c0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:48.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb3526dc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb34c154fa0 0x7fb34c1f29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:48.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb3526dc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb34c1f2f20 0x7fb34c06f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:48.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb3526dc640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c15d7a0 con 0x7fb34c1f2f20 2023-10-30T12:49:48.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb3526dc640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c1f6a60 con 0x7fb338003f30 2023-10-30T12:49:48.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb3526dc640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c1518b0 con 0x7fb34c154fa0 2023-10-30T12:49:48.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb351edb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb34c1f2f20 0x7fb34c06f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:48.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb351edb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb34c1f2f20 0x7fb34c06f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55280/0 (socket says 172.21.15.125:55280) 2023-10-30T12:49:48.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb351edb640 1 -- 172.21.15.125:0/1755992470 learned_addr learned my addr 172.21.15.125:0/1755992470 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:48.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb350ed9640 1 --2- 172.21.15.125:0/1755992470 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb34c154fa0 0x7fb34c1f29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:48.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb3516da640 1 --2- 172.21.15.125:0/1755992470 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb338003f30 0x7fb34c0c0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:48.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb351edb640 1 -- 172.21.15.125:0/1755992470 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb34c154fa0 msgr2=0x7fb34c1f29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:48.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb351edb640 1 --2- 172.21.15.125:0/1755992470 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb34c154fa0 0x7fb34c1f29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:48.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb351edb640 1 -- 172.21.15.125:0/1755992470 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb338003f30 msgr2=0x7fb34c0c0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:48.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb351edb640 1 --2- 172.21.15.125:0/1755992470 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb338003f30 0x7fb34c0c0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:48.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb351edb640 1 -- 172.21.15.125:0/1755992470 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c1f6e80 con 0x7fb34c1f2f20 2023-10-30T12:49:48.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb351edb640 1 --2- 172.21.15.125:0/1755992470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb34c1f2f20 0x7fb34c06f340 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb34c06b310 tx=0x7fb34400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:48.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb341ffb640 1 -- 172.21.15.125:0/1755992470 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb34400eb90 con 0x7fb34c1f2f20 2023-10-30T12:49:48.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb341ffb640 1 -- 172.21.15.125:0/1755992470 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb34400ecf0 con 0x7fb34c1f2f20 2023-10-30T12:49:48.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb341ffb640 1 -- 172.21.15.125:0/1755992470 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb344013640 con 0x7fb34c1f2f20 2023-10-30T12:49:48.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb3526dc640 1 -- 172.21.15.125:0/1755992470 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34c15e9c0 con 0x7fb34c1f2f20 2023-10-30T12:49:48.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb3526dc640 1 -- 172.21.15.125:0/1755992470 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34c15e6e0 con 0x7fb34c1f2f20 2023-10-30T12:49:48.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb3526dc640 1 -- 172.21.15.125:0/1755992470 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34c04e170 con 0x7fb34c1f2f20 2023-10-30T12:49:48.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.118+0000 7fb341ffb640 1 -- 172.21.15.125:0/1755992470 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb3440137a0 con 0x7fb34c1f2f20 2023-10-30T12:49:48.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.122+0000 7fb341ffb640 1 --2- 172.21.15.125:0/1755992470 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb3300664a0 0x7fb330068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:48.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.122+0000 7fb341ffb640 1 -- 172.21.15.125:0/1755992470 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fb344084680 con 0x7fb34c1f2f20 2023-10-30T12:49:48.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.122+0000 7fb3516da640 1 --2- 172.21.15.125:0/1755992470 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb3300664a0 0x7fb330068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:48.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.122+0000 7fb3516da640 1 --2- 172.21.15.125:0/1755992470 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb3300664a0 0x7fb330068960 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb348005630 tx=0x7fb348009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:48.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.126+0000 7fb341ffb640 1 -- 172.21.15.125:0/1755992470 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb344055200 con 0x7fb34c1f2f20 2023-10-30T12:49:48.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.302+0000 7fb3526dc640 1 -- 172.21.15.125:0/1755992470 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb34c0c89d0 con 0x7fb34c1f2f20 2023-10-30T12:49:48.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.302+0000 7fb341ffb640 1 -- 172.21.15.125:0/1755992470 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7fb344054d90 con 0x7fb34c1f2f20 2023-10-30T12:49:48.304 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:48.304 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:48.304 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:48.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.306+0000 7fb3526dc640 1 -- 172.21.15.125:0/1755992470 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb3300664a0 msgr2=0x7fb330068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:48.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.306+0000 7fb3526dc640 1 --2- 172.21.15.125:0/1755992470 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb3300664a0 0x7fb330068960 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb348005630 tx=0x7fb348009040 comp rx=0 tx=0).stop 2023-10-30T12:49:48.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.306+0000 7fb3526dc640 1 -- 172.21.15.125:0/1755992470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb34c1f2f20 msgr2=0x7fb34c06f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:48.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.306+0000 7fb3526dc640 1 --2- 172.21.15.125:0/1755992470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb34c1f2f20 0x7fb34c06f340 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fb34c06b310 tx=0x7fb34400e530 comp rx=0 tx=0).stop 2023-10-30T12:49:48.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.306+0000 7fb3526dc640 1 -- 172.21.15.125:0/1755992470 shutdown_connections 2023-10-30T12:49:48.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.306+0000 7fb3526dc640 1 --2- 172.21.15.125:0/1755992470 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb3300664a0 0x7fb330068960 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:48.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.306+0000 7fb3526dc640 1 --2- 172.21.15.125:0/1755992470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb34c1f2f20 0x7fb34c06f340 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:48.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.306+0000 7fb3526dc640 1 --2- 172.21.15.125:0/1755992470 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb34c154fa0 0x7fb34c1f29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:48.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.306+0000 7fb3526dc640 1 --2- 172.21.15.125:0/1755992470 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb338003f30 0x7fb34c0c0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:48.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.306+0000 7fb3526dc640 1 -- 172.21.15.125:0/1755992470 >> 172.21.15.125:0/1755992470 conn(0x7fb34c0c75d0 msgr2=0x7fb34c0ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:48.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.306+0000 7fb3526dc640 1 -- 172.21.15.125:0/1755992470 shutdown_connections 2023-10-30T12:49:48.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:48.306+0000 7fb3526dc640 1 -- 172.21.15.125:0/1755992470 wait complete. 2023-10-30T12:49:48.324 INFO:tasks.mds_thrash.fs.[cephfs]:cluster is considered unstable while MDS are in up:stopping (!thrash_while_stopping) 2023-10-30T12:49:48.325 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 108, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 108, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_6223": { "addr": "172.21.15.136:6839/3479573317", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3479573317, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3479573317, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 6223, "incarnation": 100, "join_fscid": -1, "name": "f", "rank": 1, "state": "up:stopping", "state_seq": 16 }, "gid_6318": { "addr": "172.21.15.125:6836/1721514617", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1721514617, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1721514617, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1 ], "features": 4540138322906710015, "flags": 0, "gid": 6318, "incarnation": 92, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:49:12.327796+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 6318, "mds_1": 6223 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6837/2408701650", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2408701650, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2408701650, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 105, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6627, "incarnation": 0, "join_fscid": -1, "name": "c", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:49:50.326 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:50.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf72646640 1 Processor -- start 2023-10-30T12:49:50.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf72646640 1 -- start start 2023-10-30T12:49:50.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf72646640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faf6c0c05d0 0x7faf6c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:50.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf72646640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7faf6c0c6f60 0x7faf6c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:50.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf72646640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf6c0c0a50 con 0x7faf6c0c6b80 2023-10-30T12:49:50.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf72646640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf6c06f6e0 con 0x7faf6c0c05d0 2023-10-30T12:49:50.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf72646640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf6c06f9c0 con 0x7faf6c0c6f60 2023-10-30T12:49:50.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf6b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faf6c0c05d0 0x7faf6c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:50.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf6b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faf6c0c05d0 0x7faf6c13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55290/0 (socket says 172.21.15.125:55290) 2023-10-30T12:49:50.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf6bfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7faf6c0c6b80 0x7faf6c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59100/0 (socket says 172.21.15.125:59100) 2023-10-30T12:49:50.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf6b7fe640 1 -- 172.21.15.125:0/1383424682 learned_addr learned my addr 172.21.15.125:0/1383424682 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:50.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf70be3640 1 --2- 172.21.15.125:0/1383424682 >> v2:172.21.15.136:3300/0 conn(0x7faf6c0c6f60 0x7faf6c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:50.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf6b7fe640 1 -- 172.21.15.125:0/1383424682 >> v2:172.21.15.136:3300/0 conn(0x7faf6c0c6f60 msgr2=0x7faf6c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:50.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf6b7fe640 1 --2- 172.21.15.125:0/1383424682 >> v2:172.21.15.136:3300/0 conn(0x7faf6c0c6f60 0x7faf6c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:50.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf6b7fe640 1 -- 172.21.15.125:0/1383424682 >> v1:172.21.15.125:6789/0 conn(0x7faf6c0c6b80 legacy=0x7faf6c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:50.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf6b7fe640 1 -- 172.21.15.125:0/1383424682 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c06fca0 con 0x7faf6c0c05d0 2023-10-30T12:49:50.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.478+0000 7faf6affd640 1 -- 172.21.15.125:0/1383424682 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566107983 0 0) 0x7faf6c0c0a50 con 0x7faf6c0c6b80 2023-10-30T12:49:50.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.482+0000 7faf6b7fe640 1 --2- 172.21.15.125:0/1383424682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf6c0c05d0 0x7faf6c13ed90 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7faf6c0c89b0 tx=0x7faf6001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=656ebe12fd485ac8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:50.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.482+0000 7faf6affd640 1 -- 172.21.15.125:0/1383424682 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf600ab610 con 0x7faf6c0c05d0 2023-10-30T12:49:50.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.482+0000 7faf6affd640 1 -- 172.21.15.125:0/1383424682 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf600ab770 con 0x7faf6c0c05d0 2023-10-30T12:49:50.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.482+0000 7faf6affd640 1 -- 172.21.15.125:0/1383424682 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf600abb00 con 0x7faf6c0c05d0 2023-10-30T12:49:50.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.482+0000 7faf72646640 1 -- 172.21.15.125:0/1383424682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf6c0c05d0 msgr2=0x7faf6c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:50.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.482+0000 7faf72646640 1 --2- 172.21.15.125:0/1383424682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf6c0c05d0 0x7faf6c13ed90 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7faf6c0c89b0 tx=0x7faf6001c500 comp rx=0 tx=0).stop 2023-10-30T12:49:50.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.482+0000 7faf72646640 1 -- 172.21.15.125:0/1383424682 shutdown_connections 2023-10-30T12:49:50.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.482+0000 7faf72646640 1 --2- 172.21.15.125:0/1383424682 >> v2:172.21.15.136:3300/0 conn(0x7faf6c0c6f60 0x7faf6c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:50.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.482+0000 7faf72646640 1 --2- 172.21.15.125:0/1383424682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf6c0c05d0 0x7faf6c13ed90 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:50.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.482+0000 7faf72646640 1 -- 172.21.15.125:0/1383424682 >> 172.21.15.125:0/1383424682 conn(0x7faf6c0c7910 msgr2=0x7faf6c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:50.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.482+0000 7faf72646640 1 -- 172.21.15.125:0/1383424682 shutdown_connections 2023-10-30T12:49:50.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.482+0000 7faf72646640 1 -- 172.21.15.125:0/1383424682 wait complete. 2023-10-30T12:49:50.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf72646640 1 Processor -- start 2023-10-30T12:49:50.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf72646640 1 -- start start 2023-10-30T12:49:50.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf72646640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf6c0c05d0 0x7faf6c153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:50.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf72646640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf6c0c6b80 0x7faf6c134960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:50.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf72646640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf6c0c6f60 0x7faf6c134ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:50.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf72646640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c134540 con 0x7faf6c0c6b80 2023-10-30T12:49:50.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf72646640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c135710 con 0x7faf6c0c05d0 2023-10-30T12:49:50.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf72646640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf6c135980 con 0x7faf6c0c6f60 2023-10-30T12:49:50.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf6b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf6c0c6b80 0x7faf6c134960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:50.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf6b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf6c0c6b80 0x7faf6c134960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55296/0 (socket says 172.21.15.125:55296) 2023-10-30T12:49:50.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf6b7fe640 1 -- 172.21.15.125:0/89205305 learned_addr learned my addr 172.21.15.125:0/89205305 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:50.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf6bfff640 1 --2- 172.21.15.125:0/89205305 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf6c0c05d0 0x7faf6c153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:50.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf70be3640 1 --2- 172.21.15.125:0/89205305 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf6c0c6f60 0x7faf6c134ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:50.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf6b7fe640 1 -- 172.21.15.125:0/89205305 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf6c0c6f60 msgr2=0x7faf6c134ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:50.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf6b7fe640 1 --2- 172.21.15.125:0/89205305 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf6c0c6f60 0x7faf6c134ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:50.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf6b7fe640 1 -- 172.21.15.125:0/89205305 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf6c0c05d0 msgr2=0x7faf6c153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:50.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf6b7fe640 1 --2- 172.21.15.125:0/89205305 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf6c0c05d0 0x7faf6c153510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:50.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf6b7fe640 1 -- 172.21.15.125:0/89205305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf600094d0 con 0x7faf6c0c6b80 2023-10-30T12:49:50.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf6b7fe640 1 --2- 172.21.15.125:0/89205305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf6c0c6b80 0x7faf6c134960 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7faf60019110 tx=0x7faf6001ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:50.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf68ff9640 1 -- 172.21.15.125:0/89205305 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf600b4800 con 0x7faf6c0c6b80 2023-10-30T12:49:50.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf68ff9640 1 -- 172.21.15.125:0/89205305 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf600b4960 con 0x7faf6c0c6b80 2023-10-30T12:49:50.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf68ff9640 1 -- 172.21.15.125:0/89205305 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf600b4d10 con 0x7faf6c0c6b80 2023-10-30T12:49:50.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf72646640 1 -- 172.21.15.125:0/89205305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf6c135c60 con 0x7faf6c0c6b80 2023-10-30T12:49:50.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.486+0000 7faf72646640 1 -- 172.21.15.125:0/89205305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf6c0ca6f0 con 0x7faf6c0c6b80 2023-10-30T12:49:50.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.490+0000 7faf72646640 1 -- 172.21.15.125:0/89205305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6c1342a0 con 0x7faf6c0c6b80 2023-10-30T12:49:50.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.490+0000 7faf68ff9640 1 -- 172.21.15.125:0/89205305 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faf600b2410 con 0x7faf6c0c6b80 2023-10-30T12:49:50.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.490+0000 7faf68ff9640 1 --2- 172.21.15.125:0/89205305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf580664d0 0x7faf58068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:50.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.490+0000 7faf68ff9640 1 -- 172.21.15.125:0/89205305 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7faf6011ea10 con 0x7faf6c0c6b80 2023-10-30T12:49:50.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.490+0000 7faf6bfff640 1 --2- 172.21.15.125:0/89205305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf580664d0 0x7faf58068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:50.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.494+0000 7faf6bfff640 1 --2- 172.21.15.125:0/89205305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf580664d0 0x7faf58068990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7faf5c007fd0 tx=0x7faf5c007d60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:50.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.494+0000 7faf68ff9640 1 -- 172.21.15.125:0/89205305 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faf6c1342a0 con 0x7faf6c0c6b80 2023-10-30T12:49:50.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.678+0000 7faf72646640 1 -- 172.21.15.125:0/89205305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf6c134680 con 0x7faf6c0c6b80 2023-10-30T12:49:50.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.678+0000 7faf68ff9640 1 -- 172.21.15.125:0/89205305 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7faf6c134680 con 0x7faf6c0c6b80 2023-10-30T12:49:50.677 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:50.677 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:50.678 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:50.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.678+0000 7faf72646640 1 -- 172.21.15.125:0/89205305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf580664d0 msgr2=0x7faf58068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:50.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.678+0000 7faf72646640 1 --2- 172.21.15.125:0/89205305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf580664d0 0x7faf58068990 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7faf5c007fd0 tx=0x7faf5c007d60 comp rx=0 tx=0).stop 2023-10-30T12:49:50.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.678+0000 7faf72646640 1 -- 172.21.15.125:0/89205305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf6c0c6b80 msgr2=0x7faf6c134960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:50.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.678+0000 7faf72646640 1 --2- 172.21.15.125:0/89205305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf6c0c6b80 0x7faf6c134960 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7faf60019110 tx=0x7faf6001ad40 comp rx=0 tx=0).stop 2023-10-30T12:49:50.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.678+0000 7faf72646640 1 -- 172.21.15.125:0/89205305 shutdown_connections 2023-10-30T12:49:50.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.678+0000 7faf72646640 1 --2- 172.21.15.125:0/89205305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf580664d0 0x7faf58068990 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:50.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.678+0000 7faf72646640 1 --2- 172.21.15.125:0/89205305 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf6c0c6f60 0x7faf6c134ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:50.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.678+0000 7faf72646640 1 --2- 172.21.15.125:0/89205305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf6c0c6b80 0x7faf6c134960 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:50.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.678+0000 7faf72646640 1 --2- 172.21.15.125:0/89205305 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf6c0c05d0 0x7faf6c153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:50.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.678+0000 7faf72646640 1 -- 172.21.15.125:0/89205305 >> 172.21.15.125:0/89205305 conn(0x7faf6c0c7910 msgr2=0x7faf6c0b0880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:50.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.678+0000 7faf72646640 1 -- 172.21.15.125:0/89205305 shutdown_connections 2023-10-30T12:49:50.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:50.678+0000 7faf72646640 1 -- 172.21.15.125:0/89205305 wait complete. 2023-10-30T12:49:50.698 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:52.700 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:52.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.846+0000 7f0570165640 1 Processor -- start 2023-10-30T12:49:52.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.846+0000 7f0570165640 1 -- start start 2023-10-30T12:49:52.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.846+0000 7f0570165640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0568070740 0x7f05680c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:52.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.846+0000 7f0570165640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f056815e700 0x7f056806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:52.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.846+0000 7f0570165640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05680c5810 con 0x7f05681580e0 2023-10-30T12:49:52.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.846+0000 7f0570165640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f056815eb70 con 0x7f0568070740 2023-10-30T12:49:52.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.846+0000 7f0570165640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05680c6bb0 con 0x7f056815e700 2023-10-30T12:49:52.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.846+0000 7f056d700640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0568070740 0x7f05680c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:52.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.846+0000 7f056d700640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0568070740 0x7f05680c6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59172/0 (socket says 172.21.15.125:59172) 2023-10-30T12:49:52.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.846+0000 7f056d700640 1 -- 172.21.15.125:0/325731221 learned_addr learned my addr 172.21.15.125:0/325731221 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:52.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.850+0000 7f056e702640 1 --2- 172.21.15.125:0/325731221 >> v2:172.21.15.136:3300/0 conn(0x7f056815e700 0x7f056806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:52.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.850+0000 7f056d700640 1 -- 172.21.15.125:0/325731221 >> v2:172.21.15.136:3300/0 conn(0x7f056815e700 msgr2=0x7f056806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:52.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.850+0000 7f056d700640 1 --2- 172.21.15.125:0/325731221 >> v2:172.21.15.136:3300/0 conn(0x7f056815e700 0x7f056806a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:52.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.850+0000 7f056d700640 1 -- 172.21.15.125:0/325731221 >> v1:172.21.15.125:6789/0 conn(0x7f05681580e0 legacy=0x7f0568069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:52.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.850+0000 7f056d700640 1 -- 172.21.15.125:0/325731221 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05680c6e70 con 0x7f0568070740 2023-10-30T12:49:52.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.850+0000 7f056d700640 1 --2- 172.21.15.125:0/325731221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0568070740 0x7f05680c6540 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f05680cbeb0 tx=0x7f055801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b029b808455082f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:52.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.850+0000 7f056ceff640 1 -- 172.21.15.125:0/325731221 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05580ab5f0 con 0x7f0568070740 2023-10-30T12:49:52.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.850+0000 7f056ceff640 1 -- 172.21.15.125:0/325731221 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05580ab750 con 0x7f0568070740 2023-10-30T12:49:52.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.850+0000 7f056ceff640 1 -- 172.21.15.125:0/325731221 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05580abae0 con 0x7f0568070740 2023-10-30T12:49:52.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 -- 172.21.15.125:0/325731221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0568070740 msgr2=0x7f05680c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:52.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 --2- 172.21.15.125:0/325731221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0568070740 0x7f05680c6540 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f05680cbeb0 tx=0x7f055801c500 comp rx=0 tx=0).stop 2023-10-30T12:49:52.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 -- 172.21.15.125:0/325731221 shutdown_connections 2023-10-30T12:49:52.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 --2- 172.21.15.125:0/325731221 >> v2:172.21.15.136:3300/0 conn(0x7f056815e700 0x7f056806a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:52.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 --2- 172.21.15.125:0/325731221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0568070740 0x7f05680c6540 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:52.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 -- 172.21.15.125:0/325731221 >> 172.21.15.125:0/325731221 conn(0x7f05680c7910 msgr2=0x7f05680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:52.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 -- 172.21.15.125:0/325731221 shutdown_connections 2023-10-30T12:49:52.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 -- 172.21.15.125:0/325731221 wait complete. 2023-10-30T12:49:52.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 Processor -- start 2023-10-30T12:49:52.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 -- start start 2023-10-30T12:49:52.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0568070740 0x7f0568155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:52.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05681580e0 0x7f05680c1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:52.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f056815e700 0x7f05680c19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:52.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05680c8f10 con 0x7f0568070740 2023-10-30T12:49:52.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05681f9b20 con 0x7f05681580e0 2023-10-30T12:49:52.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f0570165640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05681f9e00 con 0x7f056815e700 2023-10-30T12:49:52.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f056df01640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0568070740 0x7f0568155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:52.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f056df01640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0568070740 0x7f0568155b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59178/0 (socket says 172.21.15.125:59178) 2023-10-30T12:49:52.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f056df01640 1 -- 172.21.15.125:0/3562792717 learned_addr learned my addr 172.21.15.125:0/3562792717 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:52.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f056d700640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05681580e0 0x7f05680c1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:52.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f056e702640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f056815e700 0x7f05680c19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:52.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f056df01640 1 -- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f056815e700 msgr2=0x7f05680c19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:52.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f056df01640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f056815e700 0x7f05680c19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:52.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f056df01640 1 -- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05681580e0 msgr2=0x7f05680c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:52.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f056df01640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05681580e0 0x7f05680c1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:52.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f056df01640 1 -- 172.21.15.125:0/3562792717 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05580094d0 con 0x7f0568070740 2023-10-30T12:49:52.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.854+0000 7f056e702640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f056815e700 0x7f05680c19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:49:52.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.858+0000 7f056df01640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0568070740 0x7f0568155b30 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f056400c500 tx=0x7f056400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:52.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.858+0000 7f055e7fc640 1 -- 172.21.15.125:0/3562792717 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564005220 con 0x7f0568070740 2023-10-30T12:49:52.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.858+0000 7f055e7fc640 1 -- 172.21.15.125:0/3562792717 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0564005380 con 0x7f0568070740 2023-10-30T12:49:52.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.858+0000 7f055e7fc640 1 -- 172.21.15.125:0/3562792717 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0564007270 con 0x7f0568070740 2023-10-30T12:49:52.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.858+0000 7f0570165640 1 -- 172.21.15.125:0/3562792717 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05681fa0e0 con 0x7f0568070740 2023-10-30T12:49:52.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.858+0000 7f0570165640 1 -- 172.21.15.125:0/3562792717 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05681fa6a0 con 0x7f0568070740 2023-10-30T12:49:52.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.858+0000 7f0570165640 1 -- 172.21.15.125:0/3562792717 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f056804e170 con 0x7f0568070740 2023-10-30T12:49:52.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.858+0000 7f055e7fc640 1 -- 172.21.15.125:0/3562792717 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f05640073d0 con 0x7f0568070740 2023-10-30T12:49:52.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.858+0000 7f055e7fc640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f054c0664d0 0x7f054c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:52.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.858+0000 7f055e7fc640 1 -- 172.21.15.125:0/3562792717 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f0564083b30 con 0x7f0568070740 2023-10-30T12:49:52.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.858+0000 7f056d700640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f054c0664d0 0x7f054c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:52.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.862+0000 7f056d700640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f054c0664d0 0x7f054c068990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f05580af000 tx=0x7f05580a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:52.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:52.862+0000 7f055e7fc640 1 -- 172.21.15.125:0/3562792717 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0564050ba0 con 0x7f0568070740 2023-10-30T12:49:53.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:53.046+0000 7f0570165640 1 -- 172.21.15.125:0/3562792717 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f056806aed0 con 0x7f0568070740 2023-10-30T12:49:53.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:53.046+0000 7f055e7fc640 1 -- 172.21.15.125:0/3562792717 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7f05640541c0 con 0x7f0568070740 2023-10-30T12:49:53.045 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:53.045 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:53.046 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:53.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:53.046+0000 7f0570165640 1 -- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f054c0664d0 msgr2=0x7f054c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:53.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:53.046+0000 7f0570165640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f054c0664d0 0x7f054c068990 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f05580af000 tx=0x7f05580a7000 comp rx=0 tx=0).stop 2023-10-30T12:49:53.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:53.046+0000 7f0570165640 1 -- 172.21.15.125:0/3562792717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0568070740 msgr2=0x7f0568155b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:53.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:53.046+0000 7f0570165640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0568070740 0x7f0568155b30 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f056400c500 tx=0x7f056400c9c0 comp rx=0 tx=0).stop 2023-10-30T12:49:53.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:53.046+0000 7f0570165640 1 -- 172.21.15.125:0/3562792717 shutdown_connections 2023-10-30T12:49:53.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:53.046+0000 7f0570165640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f054c0664d0 0x7f054c068990 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:53.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:53.046+0000 7f0570165640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f056815e700 0x7f05680c19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:53.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:53.046+0000 7f0570165640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05681580e0 0x7f05680c1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:53.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:53.046+0000 7f0570165640 1 --2- 172.21.15.125:0/3562792717 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0568070740 0x7f0568155b30 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:53.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:53.046+0000 7f0570165640 1 -- 172.21.15.125:0/3562792717 >> 172.21.15.125:0/3562792717 conn(0x7f05680c7910 msgr2=0x7f0568156f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:53.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:53.046+0000 7f0570165640 1 -- 172.21.15.125:0/3562792717 shutdown_connections 2023-10-30T12:49:53.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:53.046+0000 7f0570165640 1 -- 172.21.15.125:0/3562792717 wait complete. 2023-10-30T12:49:53.065 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:55.066 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:55.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb8a57d3640 1 Processor -- start 2023-10-30T12:49:55.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb8a57d3640 1 -- start start 2023-10-30T12:49:55.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb8a57d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb8a0159e60 0x7fb8a006f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:55.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb8a57d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a00bfec0 0x7fb8a00c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:55.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb8a57d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb8a006a720 0x7fb8a006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:55.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb8a57d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a006fa80 con 0x7fb8a006a720 2023-10-30T12:49:55.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb8a57d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8a0071290 con 0x7fb8a0159e60 2023-10-30T12:49:55.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb8a57d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a0071570 con 0x7fb8a00bfec0 2023-10-30T12:49:55.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb89f7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb8a006a720 0x7fb8a006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:55.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb89f7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb8a006a720 0x7fb8a006abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59186/0 (socket says 172.21.15.125:59186) 2023-10-30T12:49:55.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb89f7fe640 1 -- 172.21.15.125:0/1816723797 learned_addr learned my addr 172.21.15.125:0/1816723797 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:55.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb8967fc640 1 --2- 172.21.15.125:0/1816723797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a00bfec0 0x7fb8a00c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:55.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb89effd640 1 --2- 172.21.15.125:0/1816723797 >> v2:172.21.15.136:3300/0 conn(0x7fb8a0159e60 0x7fb8a006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:55.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb89f7fe640 1 -- 172.21.15.125:0/1816723797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a00bfec0 msgr2=0x7fb8a00c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:55.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb89f7fe640 1 --2- 172.21.15.125:0/1816723797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a00bfec0 0x7fb8a00c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:55.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb89f7fe640 1 -- 172.21.15.125:0/1816723797 >> v2:172.21.15.136:3300/0 conn(0x7fb8a0159e60 msgr2=0x7fb8a006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:55.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb89f7fe640 1 --2- 172.21.15.125:0/1816723797 >> v2:172.21.15.136:3300/0 conn(0x7fb8a0159e60 0x7fb8a006f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:55.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.214+0000 7fb89f7fe640 1 -- 172.21.15.125:0/1816723797 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a0071850 con 0x7fb8a006a720 2023-10-30T12:49:55.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.218+0000 7fb89f7fe640 1 --2- 172.21.15.125:0/1816723797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a006a720 0x7fb8a006abb0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00c8950 tx=0x7fb89801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd54d19fc1f03ee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:55.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.218+0000 7fb89e7fc640 1 -- 172.21.15.125:0/1816723797 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8980a89b0 con 0x7fb8a006a720 2023-10-30T12:49:55.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.218+0000 7fb89e7fc640 1 -- 172.21.15.125:0/1816723797 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8980a8b10 con 0x7fb8a006a720 2023-10-30T12:49:55.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.218+0000 7fb89e7fc640 1 -- 172.21.15.125:0/1816723797 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8980a8ea0 con 0x7fb8a006a720 2023-10-30T12:49:55.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.218+0000 7fb8a57d3640 1 -- 172.21.15.125:0/1816723797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a006a720 msgr2=0x7fb8a006abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:55.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.218+0000 7fb8a57d3640 1 --2- 172.21.15.125:0/1816723797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a006a720 0x7fb8a006abb0 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00c8950 tx=0x7fb89801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:49:55.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.218+0000 7fb8a57d3640 1 -- 172.21.15.125:0/1816723797 shutdown_connections 2023-10-30T12:49:55.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.218+0000 7fb8a57d3640 1 --2- 172.21.15.125:0/1816723797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a006a720 0x7fb8a006abb0 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:55.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.218+0000 7fb8a57d3640 1 --2- 172.21.15.125:0/1816723797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a00bfec0 0x7fb8a00c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:55.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.218+0000 7fb8a57d3640 1 --2- 172.21.15.125:0/1816723797 >> v2:172.21.15.136:3300/0 conn(0x7fb8a0159e60 0x7fb8a006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:55.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.218+0000 7fb8a57d3640 1 -- 172.21.15.125:0/1816723797 >> 172.21.15.125:0/1816723797 conn(0x7fb8a00c7910 msgr2=0x7fb8a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:55.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.218+0000 7fb8a57d3640 1 -- 172.21.15.125:0/1816723797 shutdown_connections 2023-10-30T12:49:55.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.218+0000 7fb8a57d3640 1 -- 172.21.15.125:0/1816723797 wait complete. 2023-10-30T12:49:55.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb8a57d3640 1 Processor -- start 2023-10-30T12:49:55.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb8a57d3640 1 -- start start 2023-10-30T12:49:55.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb8a57d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8a006a720 0x7fb8a014ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:55.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb8a57d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a00bfec0 0x7fb8a0153d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:55.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb8a57d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a0159e60 0x7fb8a01eedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:55.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb8a57d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a01efad0 con 0x7fb8a0159e60 2023-10-30T12:49:55.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb8a57d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a01ef7f0 con 0x7fb8a006a720 2023-10-30T12:49:55.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb8a57d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a0156a90 con 0x7fb8a00bfec0 2023-10-30T12:49:55.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb89f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a0159e60 0x7fb8a01eedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:55.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb89f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a0159e60 0x7fb8a01eedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59194/0 (socket says 172.21.15.125:59194) 2023-10-30T12:49:55.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb89f7fe640 1 -- 172.21.15.125:0/3039797286 learned_addr learned my addr 172.21.15.125:0/3039797286 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:55.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb89effd640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8a006a720 0x7fb8a014ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:55.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb8967fc640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a00bfec0 0x7fb8a0153d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:55.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb89f7fe640 1 -- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a00bfec0 msgr2=0x7fb8a0153d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:55.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb89f7fe640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a00bfec0 0x7fb8a0153d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:55.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb89f7fe640 1 -- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8a006a720 msgr2=0x7fb8a014ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:55.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb89f7fe640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8a006a720 0x7fb8a014ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:55.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb89f7fe640 1 -- 172.21.15.125:0/3039797286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8980094d0 con 0x7fb8a0159e60 2023-10-30T12:49:55.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb89effd640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8a006a720 0x7fb8a014ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:55.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb89f7fe640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a0159e60 0x7fb8a01eedd0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb8980b3040 tx=0x7fb89801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:55.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb897fff640 1 -- 172.21.15.125:0/3039797286 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8980b0da0 con 0x7fb8a0159e60 2023-10-30T12:49:55.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb897fff640 1 -- 172.21.15.125:0/3039797286 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8980bb470 con 0x7fb8a0159e60 2023-10-30T12:49:55.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb8a57d3640 1 -- 172.21.15.125:0/3039797286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a0156d70 con 0x7fb8a0159e60 2023-10-30T12:49:55.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb897fff640 1 -- 172.21.15.125:0/3039797286 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8980bb7e0 con 0x7fb8a0159e60 2023-10-30T12:49:55.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.222+0000 7fb8a57d3640 1 -- 172.21.15.125:0/3039797286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a014e1e0 con 0x7fb8a0159e60 2023-10-30T12:49:55.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.226+0000 7fb8a57d3640 1 -- 172.21.15.125:0/3039797286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a014e750 con 0x7fb8a0159e60 2023-10-30T12:49:55.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.226+0000 7fb897fff640 1 -- 172.21.15.125:0/3039797286 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb8980b7070 con 0x7fb8a0159e60 2023-10-30T12:49:55.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.226+0000 7fb897fff640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8800664a0 0x7fb880068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:55.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.226+0000 7fb897fff640 1 -- 172.21.15.125:0/3039797286 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fb8981226a0 con 0x7fb8a0159e60 2023-10-30T12:49:55.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.226+0000 7fb89effd640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8800664a0 0x7fb880068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:55.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.226+0000 7fb89effd640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8800664a0 0x7fb880068960 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb89000b9f0 tx=0x7fb890008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:55.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.230+0000 7fb897fff640 1 -- 172.21.15.125:0/3039797286 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb8980ef670 con 0x7fb8a0159e60 2023-10-30T12:49:55.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.414+0000 7fb8a57d3640 1 -- 172.21.15.125:0/3039797286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8a004e170 con 0x7fb8a0159e60 2023-10-30T12:49:55.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.414+0000 7fb897fff640 1 -- 172.21.15.125:0/3039797286 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+6538 (secure 0 0 0) 0x7fb8980b5020 con 0x7fb8a0159e60 2023-10-30T12:49:55.414 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:55.415 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:12.327796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0,1],"up":{"mds_0":6318,"mds_1":6223},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_6223":{"gid":6223,"name":"f","rank":1,"incarnation":100,"state":"up:stopping","state_seq":16,"addr":"172.21.15.136:6839/3479573317","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3479573317},{"type":"v1","addr":"172.21.15.136:6839","nonce":3479573317}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:55.415 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 108 2023-10-30T12:49:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.418+0000 7fb8a57d3640 1 -- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8800664a0 msgr2=0x7fb880068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:55.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.418+0000 7fb8a57d3640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8800664a0 0x7fb880068960 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fb89000b9f0 tx=0x7fb890008040 comp rx=0 tx=0).stop 2023-10-30T12:49:55.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.418+0000 7fb8a57d3640 1 -- 172.21.15.125:0/3039797286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a0159e60 msgr2=0x7fb8a01eedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:55.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.418+0000 7fb8a57d3640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a0159e60 0x7fb8a01eedd0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb8980b3040 tx=0x7fb89801ba90 comp rx=0 tx=0).stop 2023-10-30T12:49:55.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.418+0000 7fb8a57d3640 1 -- 172.21.15.125:0/3039797286 shutdown_connections 2023-10-30T12:49:55.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.418+0000 7fb8a57d3640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8800664a0 0x7fb880068960 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:55.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.418+0000 7fb8a57d3640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8a0159e60 0x7fb8a01eedd0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:55.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.418+0000 7fb8a57d3640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8a00bfec0 0x7fb8a0153d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:55.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.418+0000 7fb8a57d3640 1 --2- 172.21.15.125:0/3039797286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8a006a720 0x7fb8a014ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:55.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.418+0000 7fb8a57d3640 1 -- 172.21.15.125:0/3039797286 >> 172.21.15.125:0/3039797286 conn(0x7fb8a00c7910 msgr2=0x7fb8a0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:55.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.418+0000 7fb8a57d3640 1 -- 172.21.15.125:0/3039797286 shutdown_connections 2023-10-30T12:49:55.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:55.418+0000 7fb8a57d3640 1 -- 172.21.15.125:0/3039797286 wait complete. 2023-10-30T12:49:55.435 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:49:57.435 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:57.462 INFO:tasks.ceph.mds.f.smithi136.stdout:starting mds.f at 2023-10-30T12:49:57.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e89606640 1 Processor -- start 2023-10-30T12:49:57.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e89606640 1 -- start start 2023-10-30T12:49:57.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e89606640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8e8415cd50 0x7f8e840c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:57.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e89606640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e840cefa0 0x7f8e840cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:57.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e89606640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e8415a6c0 con 0x7f8e841538f0 2023-10-30T12:49:57.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e89606640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8415d300 con 0x7f8e8415cd50 2023-10-30T12:49:57.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e89606640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8415d5e0 con 0x7f8e840cefa0 2023-10-30T12:49:57.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e837fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8e8415cd50 0x7f8e840c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:57.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e837fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8e8415cd50 0x7f8e840c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59208/0 (socket says 172.21.15.125:59208) 2023-10-30T12:49:57.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e837fe640 1 -- 172.21.15.125:0/1292537341 learned_addr learned my addr 172.21.15.125:0/1292537341 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:57.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e88e05640 1 --2- 172.21.15.125:0/1292537341 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e840cefa0 0x7f8e840cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:57.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e837fe640 1 -- 172.21.15.125:0/1292537341 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e840cefa0 msgr2=0x7f8e840cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:57.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e837fe640 1 --2- 172.21.15.125:0/1292537341 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e840cefa0 0x7f8e840cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e837fe640 1 -- 172.21.15.125:0/1292537341 >> v1:172.21.15.136:6789/0 conn(0x7f8e841538f0 legacy=0x7f8e84159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:57.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.578+0000 7f8e837fe640 1 -- 172.21.15.125:0/1292537341 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8415d8c0 con 0x7f8e8415cd50 2023-10-30T12:49:57.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.582+0000 7f8e837fe640 1 --2- 172.21.15.125:0/1292537341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e8415cd50 0x7f8e840c9f90 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f8e78009870 tx=0x7f8e7801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723b12a0bf9d82f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:57.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.582+0000 7f8e82ffd640 1 -- 172.21.15.125:0/1292537341 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e780a8a30 con 0x7f8e8415cd50 2023-10-30T12:49:57.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.582+0000 7f8e82ffd640 1 -- 172.21.15.125:0/1292537341 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e780a8b90 con 0x7f8e8415cd50 2023-10-30T12:49:57.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.582+0000 7f8e82ffd640 1 -- 172.21.15.125:0/1292537341 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e780b0440 con 0x7f8e8415cd50 2023-10-30T12:49:57.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.582+0000 7f8e89606640 1 -- 172.21.15.125:0/1292537341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e8415cd50 msgr2=0x7f8e840c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:57.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.582+0000 7f8e89606640 1 --2- 172.21.15.125:0/1292537341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e8415cd50 0x7f8e840c9f90 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f8e78009870 tx=0x7f8e7801ed30 comp rx=0 tx=0).stop 2023-10-30T12:49:57.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.582+0000 7f8e89606640 1 -- 172.21.15.125:0/1292537341 shutdown_connections 2023-10-30T12:49:57.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.582+0000 7f8e89606640 1 --2- 172.21.15.125:0/1292537341 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e840cefa0 0x7f8e840cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.582+0000 7f8e89606640 1 --2- 172.21.15.125:0/1292537341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e8415cd50 0x7f8e840c9f90 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.582+0000 7f8e89606640 1 -- 172.21.15.125:0/1292537341 >> 172.21.15.125:0/1292537341 conn(0x7f8e8406a640 msgr2=0x7f8e84065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:57.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.582+0000 7f8e89606640 1 -- 172.21.15.125:0/1292537341 shutdown_connections 2023-10-30T12:49:57.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.582+0000 7f8e89606640 1 -- 172.21.15.125:0/1292537341 wait complete. 2023-10-30T12:49:57.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.582+0000 7f8e89606640 1 Processor -- start 2023-10-30T12:49:57.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e89606640 1 -- start start 2023-10-30T12:49:57.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e89606640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e840cefa0 0x7f8e84158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:57.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e89606640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e841538f0 0x7f8e840b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:57.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e89606640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e8415cd50 0x7f8e84144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:57.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e89606640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e840b4300 con 0x7f8e841538f0 2023-10-30T12:49:57.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e89606640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e840b2ff0 con 0x7f8e8415cd50 2023-10-30T12:49:57.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e89606640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e84150f90 con 0x7f8e840cefa0 2023-10-30T12:49:57.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e837fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e841538f0 0x7f8e840b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:57.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e837fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e841538f0 0x7f8e840b37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59222/0 (socket says 172.21.15.125:59222) 2023-10-30T12:49:57.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e837fe640 1 -- 172.21.15.125:0/3187307876 learned_addr learned my addr 172.21.15.125:0/3187307876 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:57.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e83fff640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e840cefa0 0x7f8e84158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:57.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e88e05640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e8415cd50 0x7f8e84144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:57.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e837fe640 1 -- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e840cefa0 msgr2=0x7f8e84158f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:57.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e837fe640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e840cefa0 0x7f8e84158f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e837fe640 1 -- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e8415cd50 msgr2=0x7f8e84144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:57.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e837fe640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e8415cd50 0x7f8e84144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e837fe640 1 -- 172.21.15.125:0/3187307876 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e780094d0 con 0x7f8e841538f0 2023-10-30T12:49:57.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e88e05640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e8415cd50 0x7f8e84144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:57.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e837fe640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e841538f0 0x7f8e840b37b0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f8e780b3040 tx=0x7f8e7801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:57.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e80ff9640 1 -- 172.21.15.125:0/3187307876 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e780be070 con 0x7f8e841538f0 2023-10-30T12:49:57.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e80ff9640 1 -- 172.21.15.125:0/3187307876 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e780ba400 con 0x7f8e841538f0 2023-10-30T12:49:57.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e80ff9640 1 -- 172.21.15.125:0/3187307876 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e780ba7d0 con 0x7f8e841538f0 2023-10-30T12:49:57.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e89606640 1 -- 172.21.15.125:0/3187307876 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e84151270 con 0x7f8e841538f0 2023-10-30T12:49:57.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e89606640 1 -- 172.21.15.125:0/3187307876 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e84151550 con 0x7f8e841538f0 2023-10-30T12:49:57.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.586+0000 7f8e89606640 1 -- 172.21.15.125:0/3187307876 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8404e170 con 0x7f8e841538f0 2023-10-30T12:49:57.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.590+0000 7f8e80ff9640 1 -- 172.21.15.125:0/3187307876 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e780c3410 con 0x7f8e841538f0 2023-10-30T12:49:57.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.590+0000 7f8e80ff9640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8e700664d0 0x7f8e70068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:57.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.590+0000 7f8e80ff9640 1 -- 172.21.15.125:0/3187307876 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f8e78121c90 con 0x7f8e841538f0 2023-10-30T12:49:57.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.590+0000 7f8e83fff640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8e700664d0 0x7f8e70068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:57.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.590+0000 7f8e83fff640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8e700664d0 0x7f8e70068990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8e74007fd0 tx=0x7f8e74009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:57.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.594+0000 7f8e80ff9640 1 -- 172.21.15.125:0/3187307876 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e780eed80 con 0x7f8e841538f0 2023-10-30T12:49:57.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.774+0000 7f8e89606640 1 -- 172.21.15.125:0/3187307876 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e840b3460 con 0x7f8e841538f0 2023-10-30T12:49:57.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.774+0000 7f8e80ff9640 1 -- 172.21.15.125:0/3187307876 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f8e780b5090 con 0x7f8e841538f0 2023-10-30T12:49:57.776 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:57.776 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:57.338574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:57.776 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 109 2023-10-30T12:49:57.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.778+0000 7f8e89606640 1 -- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8e700664d0 msgr2=0x7f8e70068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:57.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.778+0000 7f8e89606640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8e700664d0 0x7f8e70068990 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f8e74007fd0 tx=0x7f8e74009040 comp rx=0 tx=0).stop 2023-10-30T12:49:57.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.778+0000 7f8e89606640 1 -- 172.21.15.125:0/3187307876 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e841538f0 msgr2=0x7f8e840b37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:57.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.778+0000 7f8e89606640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e841538f0 0x7f8e840b37b0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f8e780b3040 tx=0x7f8e7801bad0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.778+0000 7f8e89606640 1 -- 172.21.15.125:0/3187307876 shutdown_connections 2023-10-30T12:49:57.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.778+0000 7f8e89606640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8e700664d0 0x7f8e70068990 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.778+0000 7f8e89606640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e8415cd50 0x7f8e84144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.778+0000 7f8e89606640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e841538f0 0x7f8e840b37b0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.778+0000 7f8e89606640 1 --2- 172.21.15.125:0/3187307876 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e840cefa0 0x7f8e84158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.778+0000 7f8e89606640 1 -- 172.21.15.125:0/3187307876 >> 172.21.15.125:0/3187307876 conn(0x7f8e8406a640 msgr2=0x7f8e840cf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:57.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.778+0000 7f8e89606640 1 -- 172.21.15.125:0/3187307876 shutdown_connections 2023-10-30T12:49:57.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.778+0000 7f8e89606640 1 -- 172.21.15.125:0/3187307876 wait complete. 2023-10-30T12:49:57.796 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 1 alive and active, now stable! 2023-10-30T12:49:57.796 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=1) 2023-10-30T12:49:57.797 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T12:49:57.797 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:49:57.811 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T12:49:57.811 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:57.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc93730f640 1 Processor -- start 2023-10-30T12:49:57.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc93730f640 1 -- start start 2023-10-30T12:49:57.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc93730f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc930153d70 0x7fc93006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:57.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc93730f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc93006e470 0x7fc9300cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:57.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc93730f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300c8490 0x7fc9300cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:57.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc93730f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc93006d780 con 0x7fc93006e470 2023-10-30T12:49:57.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc93730f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc93006afa0 con 0x7fc930153d70 2023-10-30T12:49:57.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc93730f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc930154150 con 0x7fc9300c8490 2023-10-30T12:49:57.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc9348aa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc93006e470 0x7fc9300cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:57.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc9348aa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc93006e470 0x7fc9300cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59234/0 (socket says 172.21.15.125:59234) 2023-10-30T12:49:57.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc9348aa640 1 -- 172.21.15.125:0/960506790 learned_addr learned my addr 172.21.15.125:0/960506790 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:57.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc9350ab640 1 --2- 172.21.15.125:0/960506790 >> v2:172.21.15.136:3300/0 conn(0x7fc930153d70 0x7fc93006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:57.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc9358ac640 1 --2- 172.21.15.125:0/960506790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300c8490 0x7fc9300cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:57.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc9348aa640 1 -- 172.21.15.125:0/960506790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300c8490 msgr2=0x7fc9300cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:57.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc9348aa640 1 --2- 172.21.15.125:0/960506790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300c8490 0x7fc9300cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc9348aa640 1 -- 172.21.15.125:0/960506790 >> v2:172.21.15.136:3300/0 conn(0x7fc930153d70 msgr2=0x7fc93006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:57.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc9348aa640 1 --2- 172.21.15.125:0/960506790 >> v2:172.21.15.136:3300/0 conn(0x7fc930153d70 0x7fc93006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc9348aa640 1 -- 172.21.15.125:0/960506790 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93006e030 con 0x7fc93006e470 2023-10-30T12:49:57.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc9350ab640 1 --2- 172.21.15.125:0/960506790 >> v2:172.21.15.136:3300/0 conn(0x7fc930153d70 0x7fc93006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:57.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.950+0000 7fc9358ac640 1 --2- 172.21.15.125:0/960506790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300c8490 0x7fc9300cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:49:57.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.954+0000 7fc9348aa640 1 --2- 172.21.15.125:0/960506790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006e470 0x7fc9300cce00 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fc93006d180 tx=0x7fc92001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6a077bda7ac4391 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:57.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.954+0000 7fc927fff640 1 -- 172.21.15.125:0/960506790 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9200aa5f0 con 0x7fc93006e470 2023-10-30T12:49:57.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.954+0000 7fc927fff640 1 -- 172.21.15.125:0/960506790 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9200aa750 con 0x7fc93006e470 2023-10-30T12:49:57.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.954+0000 7fc927fff640 1 -- 172.21.15.125:0/960506790 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9200aaae0 con 0x7fc93006e470 2023-10-30T12:49:57.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.954+0000 7fc93730f640 1 -- 172.21.15.125:0/960506790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006e470 msgr2=0x7fc9300cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:57.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.954+0000 7fc93730f640 1 --2- 172.21.15.125:0/960506790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006e470 0x7fc9300cce00 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7fc93006d180 tx=0x7fc92001c4d0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.954+0000 7fc93730f640 1 -- 172.21.15.125:0/960506790 shutdown_connections 2023-10-30T12:49:57.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.954+0000 7fc93730f640 1 --2- 172.21.15.125:0/960506790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300c8490 0x7fc9300cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.954+0000 7fc93730f640 1 --2- 172.21.15.125:0/960506790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006e470 0x7fc9300cce00 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.954+0000 7fc93730f640 1 --2- 172.21.15.125:0/960506790 >> v2:172.21.15.136:3300/0 conn(0x7fc930153d70 0x7fc93006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.954+0000 7fc93730f640 1 -- 172.21.15.125:0/960506790 >> 172.21.15.125:0/960506790 conn(0x7fc930071ea0 msgr2=0x7fc930065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:57.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.954+0000 7fc93730f640 1 -- 172.21.15.125:0/960506790 shutdown_connections 2023-10-30T12:49:57.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.954+0000 7fc93730f640 1 -- 172.21.15.125:0/960506790 wait complete. 2023-10-30T12:49:57.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc93730f640 1 Processor -- start 2023-10-30T12:49:57.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc93730f640 1 -- start start 2023-10-30T12:49:57.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc93730f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006e470 0x7fc93014aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:57.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc93730f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300c8490 0x7fc9301f6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:57.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc93730f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc930153d70 0x7fc930150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc93730f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc930155ba0 con 0x7fc93006e470 2023-10-30T12:49:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc93730f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93014afa0 con 0x7fc930153d70 2023-10-30T12:49:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc93730f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9301470b0 con 0x7fc9300c8490 2023-10-30T12:49:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc9350ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006e470 0x7fc93014aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:57.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc9350ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006e470 0x7fc93014aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59250/0 (socket says 172.21.15.125:59250) 2023-10-30T12:49:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc9350ab640 1 -- 172.21.15.125:0/2661225881 learned_addr learned my addr 172.21.15.125:0/2661225881 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc9348aa640 1 --2- 172.21.15.125:0/2661225881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300c8490 0x7fc9301f6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc9358ac640 1 --2- 172.21.15.125:0/2661225881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc930153d70 0x7fc930150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:57.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc9350ab640 1 -- 172.21.15.125:0/2661225881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300c8490 msgr2=0x7fc9301f6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:57.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc9350ab640 1 --2- 172.21.15.125:0/2661225881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300c8490 0x7fc9301f6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc9350ab640 1 -- 172.21.15.125:0/2661225881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc930153d70 msgr2=0x7fc930150a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:57.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc9350ab640 1 --2- 172.21.15.125:0/2661225881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc930153d70 0x7fc930150a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:57.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc9350ab640 1 -- 172.21.15.125:0/2661225881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9200094d0 con 0x7fc93006e470 2023-10-30T12:49:57.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc9350ab640 1 --2- 172.21.15.125:0/2661225881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006e470 0x7fc93014aa60 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00d800 tx=0x7fc92c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:57.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc925ffb640 1 -- 172.21.15.125:0/2661225881 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c00b6c0 con 0x7fc93006e470 2023-10-30T12:49:57.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc925ffb640 1 -- 172.21.15.125:0/2661225881 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc92c00b820 con 0x7fc93006e470 2023-10-30T12:49:57.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc925ffb640 1 -- 172.21.15.125:0/2661225881 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c00bbd0 con 0x7fc93006e470 2023-10-30T12:49:57.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc93730f640 1 -- 172.21.15.125:0/2661225881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc930155300 con 0x7fc93006e470 2023-10-30T12:49:57.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc93730f640 1 -- 172.21.15.125:0/2661225881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9301558c0 con 0x7fc93006e470 2023-10-30T12:49:57.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.958+0000 7fc93730f640 1 -- 172.21.15.125:0/2661225881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93004e170 con 0x7fc93006e470 2023-10-30T12:49:57.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.962+0000 7fc925ffb640 1 -- 172.21.15.125:0/2661225881 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc92c00bd30 con 0x7fc93006e470 2023-10-30T12:49:57.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.962+0000 7fc925ffb640 1 --2- 172.21.15.125:0/2661225881 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9140664d0 0x7fc914068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:57.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.962+0000 7fc925ffb640 1 -- 172.21.15.125:0/2661225881 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fc92c0846f0 con 0x7fc93006e470 2023-10-30T12:49:57.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.962+0000 7fc9348aa640 1 --2- 172.21.15.125:0/2661225881 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9140664d0 0x7fc914068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:57.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.962+0000 7fc9348aa640 1 --2- 172.21.15.125:0/2661225881 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9140664d0 0x7fc914068990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc9200ae000 tx=0x7fc9200a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:57.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:57.966+0000 7fc925ffb640 1 -- 172.21.15.125:0/2661225881 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc92c051760 con 0x7fc93006e470 2023-10-30T12:49:58.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.186+0000 7fc93730f640 1 -- 172.21.15.125:0/2661225881 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc93006c2a0 con 0x7fc93006e470 2023-10-30T12:49:58.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.186+0000 7fc925ffb640 1 -- 172.21.15.125:0/2661225881 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+5727 (secure 0 0 0) 0x7fc92c054d80 con 0x7fc93006e470 2023-10-30T12:49:58.186 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:58.186 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:57.338574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:58.187 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 109 2023-10-30T12:49:58.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.186+0000 7fc93730f640 1 -- 172.21.15.125:0/2661225881 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9140664d0 msgr2=0x7fc914068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:58.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.186+0000 7fc93730f640 1 --2- 172.21.15.125:0/2661225881 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9140664d0 0x7fc914068990 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc9200ae000 tx=0x7fc9200a7000 comp rx=0 tx=0).stop 2023-10-30T12:49:58.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.186+0000 7fc93730f640 1 -- 172.21.15.125:0/2661225881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006e470 msgr2=0x7fc93014aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:58.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.186+0000 7fc93730f640 1 --2- 172.21.15.125:0/2661225881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006e470 0x7fc93014aa60 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00d800 tx=0x7fc92c00dcc0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.190+0000 7fc93730f640 1 -- 172.21.15.125:0/2661225881 shutdown_connections 2023-10-30T12:49:58.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.190+0000 7fc93730f640 1 --2- 172.21.15.125:0/2661225881 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9140664d0 0x7fc914068990 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.190+0000 7fc93730f640 1 --2- 172.21.15.125:0/2661225881 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc930153d70 0x7fc930150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.190+0000 7fc93730f640 1 --2- 172.21.15.125:0/2661225881 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9300c8490 0x7fc9301f6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.190+0000 7fc93730f640 1 --2- 172.21.15.125:0/2661225881 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc93006e470 0x7fc93014aa60 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.190+0000 7fc93730f640 1 -- 172.21.15.125:0/2661225881 >> 172.21.15.125:0/2661225881 conn(0x7fc930071ea0 msgr2=0x7fc93014f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:58.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.190+0000 7fc93730f640 1 -- 172.21.15.125:0/2661225881 shutdown_connections 2023-10-30T12:49:58.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.190+0000 7fc93730f640 1 -- 172.21.15.125:0/2661225881 wait complete. 2023-10-30T12:49:58.207 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T12:49:58.207 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:49:58.207 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:49:58.306 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 21% (48/222) Updating files: 22% (49/222) Updating files: 23% (52/222) Updating files: 24% (54/222) Updating files: 25% (56/222) Updating files: 26% (58/222) Updating files: 27% (60/222) Updating files: 28% (63/222) Updating files: 29% (65/222) Updating files: 30% (67/222) Updating files: 31% (69/222) Updating files: 32% (72/222) Updating files: 33% (74/222) Updating files: 34% (76/222) Updating files: 35% (78/222) Updating files: 36% (80/222) Updating files: 37% (83/222) Updating files: 38% (85/222) Updating files: 39% (87/222) Updating files: 40% (89/222) Updating files: 40% (90/222) Updating files: 41% (92/222) Updating files: 42% (94/222) Updating files: 43% (96/222) Updating files: 44% (98/222) Updating files: 45% (100/222) Updating files: 46% (103/222) Updating files: 46% (104/222) Updating files: 47% (105/222) Updating files: 47% (106/222) Updating files: 48% (107/222) Updating files: 49% (109/222) Updating files: 50% (111/222) Updating files: 51% (114/222) Updating files: 52% (116/222) Updating files: 53% (118/222) Updating files: 54% (120/222) Updating files: 55% (123/222) Updating files: 56% (125/222) Updating files: 57% (127/222) Updating files: 58% (129/222) Updating files: 59% (131/222) Updating files: 60% (134/222) Updating files: 61% (136/222) Updating files: 62% (138/222) Updating files: 63% (140/222) Updating files: 64% (143/222) Updating files: 65% (145/222) Updating files: 66% (147/222) Updating files: 67% (149/222) Updating files: 68% (151/222) Updating files: 69% (154/222) Updating files: 70% (156/222) Updating files: 70% (157/222) Updating files: 71% (158/222) Updating files: 72% (160/222) Updating files: 73% (163/222) Updating files: 74% (165/222) Updating files: 75% (167/222) Updating files: 76% (169/222) Updating files: 77% (171/222) Updating files: 78% (174/222) Updating files: 79% (176/222) Updating files: 80% (178/222) Updating files: 81% (180/222) Updating files: 82% (183/222) Updating files: 83% (185/222) Updating files: 84% (187/222) Updating files: 85% (189/222) Updating files: 86% (191/222) Updating files: 87% (194/222) Updating files: 88% (196/222) Updating files: 89% (198/222) Updating files: 89% (199/222) Updating files: 90% (200/222) Updating files: 91% (203/222) Updating files: 92% (205/222) Updating files: 93% (207/222) Updating files: 94% (209/222) Updating files: 95% (211/222) Updating files: 96% (214/222) Updating files: 97% (216/222) Updating files: 98% (218/222) Updating files: 99% (220/222) Updating files: 100% (222/222) Updating files: 100% (222/222), done. 2023-10-30T12:49:58.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374d0ea640 1 Processor -- start 2023-10-30T12:49:58.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374d0ea640 1 -- start start 2023-10-30T12:49:58.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374d0ea640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3748153ca0 0x7f37480c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:58.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374d0ea640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37481598f0 con 0x7f37480c7510 2023-10-30T12:49:58.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374d0ea640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3748159b60 con 0x7f37480c0a40 2023-10-30T12:49:58.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374d0ea640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3748159e40 con 0x7f3748153ca0 2023-10-30T12:49:58.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374659c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f37480c7510 0x7f37480c69d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53886/0 (socket says 172.21.15.125:53886) 2023-10-30T12:49:58.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374659c640 1 -- 172.21.15.125:0/2299289859 learned_addr learned my addr 172.21.15.125:0/2299289859 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:49:58.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3746d9d640 1 --2- 172.21.15.125:0/2299289859 >> v2:172.21.15.136:3300/0 conn(0x7f3748153ca0 0x7f37480c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:58.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 -- 172.21.15.125:0/2299289859 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090491970 0 0) 0x7f37481598f0 con 0x7f37480c7510 2023-10-30T12:49:58.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 -- 172.21.15.125:0/2299289859 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3730003530 con 0x7f37480c7510 2023-10-30T12:49:58.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 -- 172.21.15.125:0/2299289859 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2127830310 0 0) 0x7f3730003530 con 0x7f37480c7510 2023-10-30T12:49:58.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 -- 172.21.15.125:0/2299289859 >> v2:172.21.15.136:3300/0 conn(0x7f3748153ca0 msgr2=0x7f37480c5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:58.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 --2- 172.21.15.125:0/2299289859 >> v2:172.21.15.136:3300/0 conn(0x7f3748153ca0 0x7f37480c5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 -- 172.21.15.125:0/2299289859 >> v1:172.21.15.136:6789/0 conn(0x7f37480c0a40 legacy=0x7f37480c0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:58.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 -- 172.21.15.125:0/2299289859 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374815a120 con 0x7f37480c7510 2023-10-30T12:49:58.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 -- 172.21.15.125:0/2299289859 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f373c001680 con 0x7f37480c7510 2023-10-30T12:49:58.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 -- 172.21.15.125:0/2299289859 >> v1:172.21.15.125:6789/0 conn(0x7f37480c7510 legacy=0x7f37480c69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:58.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 --2- 172.21.15.125:0/2299289859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3730003980 0x7f3730003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:58.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 --2- 172.21.15.125:0/2299289859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37300047d0 0x7f3730004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:58.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 --2- 172.21.15.125:0/2299289859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3730005190 0x7f373000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:58.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 -- 172.21.15.125:0/2299289859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3748159e40 con 0x7f3730005190 2023-10-30T12:49:58.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 -- 172.21.15.125:0/2299289859 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373000c040 con 0x7f37300047d0 2023-10-30T12:49:58.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3745d9b640 1 -- 172.21.15.125:0/2299289859 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373000c200 con 0x7f3730003980 2023-10-30T12:49:58.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374759e640 1 --2- 172.21.15.125:0/2299289859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3730005190 0x7f373000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:58.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f3746d9d640 1 --2- 172.21.15.125:0/2299289859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3730003980 0x7f3730003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:58.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374659c640 1 --2- 172.21.15.125:0/2299289859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37300047d0 0x7f3730004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:58.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374759e640 1 -- 172.21.15.125:0/2299289859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3730003980 msgr2=0x7f3730003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:58.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374759e640 1 --2- 172.21.15.125:0/2299289859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3730003980 0x7f3730003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374759e640 1 -- 172.21.15.125:0/2299289859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37300047d0 msgr2=0x7f3730004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:49:58.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374759e640 1 --2- 172.21.15.125:0/2299289859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37300047d0 0x7f3730004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374759e640 1 -- 172.21.15.125:0/2299289859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3730003530 con 0x7f3730005190 2023-10-30T12:49:58.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.350+0000 7f374659c640 1 --2- 172.21.15.125:0/2299289859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37300047d0 0x7f3730004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:58.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.354+0000 7f374759e640 1 -- 172.21.15.125:0/2299289859 reap_dead start 2023-10-30T12:49:58.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.354+0000 7f374759e640 1 --2- 172.21.15.125:0/2299289859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3730005190 0x7f373000b990 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f3740001e70 tx=0x7f374001c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936d72cf9f65c13f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:58.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.354+0000 7f3745d9b640 1 -- 172.21.15.125:0/2299289859 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3740003b80 con 0x7f3730005190 2023-10-30T12:49:58.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.354+0000 7f3745d9b640 1 -- 172.21.15.125:0/2299289859 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3740003d20 con 0x7f3730005190 2023-10-30T12:49:58.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.354+0000 7f3745d9b640 1 -- 172.21.15.125:0/2299289859 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3740005260 con 0x7f3730005190 2023-10-30T12:49:58.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.354+0000 7f374d0ea640 1 -- 172.21.15.125:0/2299289859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3730005190 msgr2=0x7f373000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:58.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.354+0000 7f374d0ea640 1 --2- 172.21.15.125:0/2299289859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3730005190 0x7f373000b990 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f3740001e70 tx=0x7f374001c760 comp rx=0 tx=0).stop 2023-10-30T12:49:58.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.354+0000 7f374d0ea640 1 -- 172.21.15.125:0/2299289859 shutdown_connections 2023-10-30T12:49:58.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.354+0000 7f374d0ea640 1 --2- 172.21.15.125:0/2299289859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3730005190 0x7f373000b990 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.354+0000 7f374d0ea640 1 -- 172.21.15.125:0/2299289859 >> 172.21.15.125:0/2299289859 conn(0x7f374806d770 msgr2=0x7f374806db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:58.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.354+0000 7f374d0ea640 1 -- 172.21.15.125:0/2299289859 shutdown_connections 2023-10-30T12:49:58.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.354+0000 7f374d0ea640 1 -- 172.21.15.125:0/2299289859 wait complete. 2023-10-30T12:49:58.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374d0ea640 1 Processor -- start 2023-10-30T12:49:58.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374d0ea640 1 -- start start 2023-10-30T12:49:58.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374d0ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3730005190 0x7f37480c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:58.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374d0ea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37480c2460 0x7f37480c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:58.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374d0ea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f374814db30 0x7f374814dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:58.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374d0ea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37480c2a60 con 0x7f37480c2460 2023-10-30T12:49:58.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374d0ea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37480ae450 con 0x7f374814db30 2023-10-30T12:49:58.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374d0ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37480c5370 con 0x7f3730005190 2023-10-30T12:49:58.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37480c2460 0x7f37480c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:58.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37480c2460 0x7f37480c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59280/0 (socket says 172.21.15.125:59280) 2023-10-30T12:49:58.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374659c640 1 -- 172.21.15.125:0/897237953 learned_addr learned my addr 172.21.15.125:0/897237953 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:49:58.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f3746d9d640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3730005190 0x7f37480c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:58.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374759e640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f374814db30 0x7f374814dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:58.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374659c640 1 -- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3730005190 msgr2=0x7f37480c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:58.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374659c640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3730005190 0x7f37480c4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374659c640 1 -- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f374814db30 msgr2=0x7f374814dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:58.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374659c640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f374814db30 0x7f374814dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374659c640 1 -- 172.21.15.125:0/897237953 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37480ad8d0 con 0x7f37480c2460 2023-10-30T12:49:58.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374759e640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f374814db30 0x7f374814dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:49:58.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374659c640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37480c2460 0x7f37480c6c30 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f373c00e730 tx=0x7f373c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:58.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f372f7fe640 1 -- 172.21.15.125:0/897237953 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c00ee40 con 0x7f37480c2460 2023-10-30T12:49:58.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f372f7fe640 1 -- 172.21.15.125:0/897237953 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373c00c870 con 0x7f37480c2460 2023-10-30T12:49:58.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f372f7fe640 1 -- 172.21.15.125:0/897237953 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373c00cbe0 con 0x7f37480c2460 2023-10-30T12:49:58.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374d0ea640 1 -- 172.21.15.125:0/897237953 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37480adbb0 con 0x7f37480c2460 2023-10-30T12:49:58.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.358+0000 7f374d0ea640 1 -- 172.21.15.125:0/897237953 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37480ae170 con 0x7f37480c2460 2023-10-30T12:49:58.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.362+0000 7f374d0ea640 1 -- 172.21.15.125:0/897237953 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3748074250 con 0x7f37480c2460 2023-10-30T12:49:58.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.362+0000 7f372f7fe640 1 -- 172.21.15.125:0/897237953 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f373c00cd40 con 0x7f37480c2460 2023-10-30T12:49:58.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.362+0000 7f372f7fe640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37340664d0 0x7f3734068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:49:58.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.362+0000 7f372f7fe640 1 -- 172.21.15.125:0/897237953 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f373c083dd0 con 0x7f37480c2460 2023-10-30T12:49:58.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.362+0000 7f3746d9d640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37340664d0 0x7f3734068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:49:58.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.362+0000 7f3746d9d640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37340664d0 0x7f3734068990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3738009aa0 tx=0x7f3738008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:49:58.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.366+0000 7f372f7fe640 1 -- 172.21.15.125:0/897237953 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f373c054950 con 0x7f37480c2460 2023-10-30T12:49:58.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.550+0000 7f374d0ea640 1 -- 172.21.15.125:0/897237953 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f374804e170 con 0x7f37480c2460 2023-10-30T12:49:58.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.550+0000 7f372f7fe640 1 -- 172.21.15.125:0/897237953 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6522 (secure 0 0 0) 0x7f373c0014e0 con 0x7f37480c2460 2023-10-30T12:49:58.551 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:49:58.551 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1874762894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1874762894},{"type":"v1","addr":"172.21.15.136:6839","nonce":1874762894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":110},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:57.338574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:49:58.551 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 110 2023-10-30T12:49:58.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.554+0000 7f374d0ea640 1 -- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37340664d0 msgr2=0x7f3734068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:58.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.554+0000 7f374d0ea640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37340664d0 0x7f3734068990 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3738009aa0 tx=0x7f3738008040 comp rx=0 tx=0).stop 2023-10-30T12:49:58.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.554+0000 7f374d0ea640 1 -- 172.21.15.125:0/897237953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37480c2460 msgr2=0x7f37480c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:49:58.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.554+0000 7f374d0ea640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37480c2460 0x7f37480c6c30 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f373c00e730 tx=0x7f373c00ebf0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.554+0000 7f374d0ea640 1 -- 172.21.15.125:0/897237953 shutdown_connections 2023-10-30T12:49:58.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.554+0000 7f374d0ea640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37340664d0 0x7f3734068990 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.554+0000 7f374d0ea640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f374814db30 0x7f374814dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.554+0000 7f374d0ea640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f37480c2460 0x7f37480c6c30 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.554+0000 7f374d0ea640 1 --2- 172.21.15.125:0/897237953 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3730005190 0x7f37480c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:49:58.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.554+0000 7f374d0ea640 1 -- 172.21.15.125:0/897237953 >> 172.21.15.125:0/897237953 conn(0x7f374806d770 msgr2=0x7f3748152f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:49:58.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.554+0000 7f374d0ea640 1 -- 172.21.15.125:0/897237953 shutdown_connections 2023-10-30T12:49:58.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:49:58.554+0000 7f374d0ea640 1 -- 172.21.15.125:0/897237953 wait complete. 2023-10-30T12:49:58.571 INFO:tasks.mds_thrash.fs.[cephfs]:cluster has 1 actives (max_mds is 1), no MDS can replace rank 1 2023-10-30T12:49:58.571 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.f 2023-10-30T12:49:59.376 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 2b5f76f48e9 readme 2023-10-30T12:49:59.382 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.2 2023-10-30T12:49:59.397 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 3 -eq 48 ] 2023-10-30T12:49:59.397 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.3 2023-10-30T12:49:59.397 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.3 2023-10-30T12:49:59.398 INFO:tasks.workunit.client.0.smithi125.stdout:v0.3 2023-10-30T12:49:59.398 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.3 2023-10-30T12:50:01.031 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 58% (174/295) Updating files: 59% (175/295) Updating files: 60% (177/295) Updating files: 61% (180/295) Updating files: 62% (183/295) Updating files: 63% (186/295) Updating files: 64% (189/295) Updating files: 65% (192/295) Updating files: 66% (195/295) Updating files: 67% (198/295) Updating files: 68% (201/295) Updating files: 69% (204/295) Updating files: 70% (207/295) Updating files: 71% (210/295) Updating files: 72% (213/295) Updating files: 73% (216/295) Updating files: 74% (219/295) Updating files: 75% (222/295) Updating files: 76% (225/295) Updating files: 77% (228/295) Updating files: 78% (231/295) Updating files: 79% (234/295) Updating files: 80% (236/295) Updating files: 81% (239/295) Updating files: 82% (242/295) Updating files: 83% (245/295) Updating files: 84% (248/295) Updating files: 85% (251/295) Updating files: 86% (254/295) Updating files: 87% (257/295) Updating files: 88% (260/295) Updating files: 89% (263/295) Updating files: 90% (266/295) Updating files: 91% (269/295) Updating files: 92% (272/295) Updating files: 93% (275/295) Updating files: 94% (278/295) Updating files: 95% (281/295) Updating files: 96% (284/295) Updating files: 97% (287/295) Updating files: 98% (290/295) Updating files: 99% (293/295) Updating files: 100% (295/295) Updating files: 100% (295/295), done. 2023-10-30T12:50:01.055 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 2e986fe9ddf include ceph.spec in configure.ac 2023-10-30T12:50:01.064 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.3 2023-10-30T12:50:01.724 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 4 -eq 48 ] 2023-10-30T12:50:01.724 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.4 2023-10-30T12:50:01.725 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.4 2023-10-30T12:50:01.725 INFO:tasks.workunit.client.0.smithi125.stdout:v0.4 2023-10-30T12:50:01.725 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.4 2023-10-30T12:50:02.572 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T12:50:02.572 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T12:50:02.573 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T12:50:02.576 INFO:tasks.ceph.mds.f:Started 2023-10-30T12:50:02.577 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:50:02.663 INFO:tasks.ceph.mds.f.smithi136.stdout:starting mds.f at 2023-10-30T12:50:02.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.794+0000 7fa3fe7f2640 1 Processor -- start 2023-10-30T12:50:02.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.798+0000 7fa3fe7f2640 1 -- start start 2023-10-30T12:50:02.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.798+0000 7fa3fe7f2640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa3f81538f0 0x7fa3f8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:02.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.798+0000 7fa3fe7f2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa3f815cd50 0x7fa3f80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:02.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.798+0000 7fa3fe7f2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f815d300 con 0x7fa3f80cefa0 2023-10-30T12:50:02.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.798+0000 7fa3fe7f2640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f815d5e0 con 0x7fa3f81538f0 2023-10-30T12:50:02.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.798+0000 7fa3fe7f2640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f815d8c0 con 0x7fa3f815cd50 2023-10-30T12:50:02.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.798+0000 7fa3fd7f0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa3f81538f0 0x7fa3f8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:02.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.798+0000 7fa3fd7f0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa3f81538f0 0x7fa3f8159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47020/0 (socket says 172.21.15.125:47020) 2023-10-30T12:50:02.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.798+0000 7fa3fd7f0640 1 -- 172.21.15.125:0/1333102989 learned_addr learned my addr 172.21.15.125:0/1333102989 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:50:02.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.798+0000 7fa3fcfef640 1 --2- 172.21.15.125:0/1333102989 >> v2:172.21.15.136:3300/0 conn(0x7fa3f815cd50 0x7fa3f80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:02.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.798+0000 7fa3fd7f0640 1 -- 172.21.15.125:0/1333102989 >> v2:172.21.15.136:3300/0 conn(0x7fa3f815cd50 msgr2=0x7fa3f80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:50:02.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.798+0000 7fa3fd7f0640 1 --2- 172.21.15.125:0/1333102989 >> v2:172.21.15.136:3300/0 conn(0x7fa3f815cd50 0x7fa3f80c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:02.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.798+0000 7fa3fd7f0640 1 -- 172.21.15.125:0/1333102989 >> v1:172.21.15.136:6789/0 conn(0x7fa3f80cefa0 legacy=0x7fa3f80cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:02.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.798+0000 7fa3fd7f0640 1 -- 172.21.15.125:0/1333102989 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f815dba0 con 0x7fa3f81538f0 2023-10-30T12:50:02.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.802+0000 7fa3fd7f0640 1 --2- 172.21.15.125:0/1333102989 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3f81538f0 0x7fa3f8159fb0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8072e90 tx=0x7fa3f401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47ca5d0bce1d180f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:02.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.802+0000 7fa3effff640 1 -- 172.21.15.125:0/1333102989 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40a89f0 con 0x7fa3f81538f0 2023-10-30T12:50:02.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.802+0000 7fa3effff640 1 -- 172.21.15.125:0/1333102989 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f40a8b50 con 0x7fa3f81538f0 2023-10-30T12:50:02.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.802+0000 7fa3effff640 1 -- 172.21.15.125:0/1333102989 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f40b0440 con 0x7fa3f81538f0 2023-10-30T12:50:02.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.802+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/1333102989 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3f81538f0 msgr2=0x7fa3f8159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:02.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.802+0000 7fa3fe7f2640 1 --2- 172.21.15.125:0/1333102989 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3f81538f0 0x7fa3f8159fb0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fa3f8072e90 tx=0x7fa3f401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:50:02.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.802+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/1333102989 shutdown_connections 2023-10-30T12:50:02.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.802+0000 7fa3fe7f2640 1 --2- 172.21.15.125:0/1333102989 >> v2:172.21.15.136:3300/0 conn(0x7fa3f815cd50 0x7fa3f80c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:02.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.802+0000 7fa3fe7f2640 1 --2- 172.21.15.125:0/1333102989 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3f81538f0 0x7fa3f8159fb0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:02.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.802+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/1333102989 >> 172.21.15.125:0/1333102989 conn(0x7fa3f806a640 msgr2=0x7fa3f8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:02.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.802+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/1333102989 shutdown_connections 2023-10-30T12:50:02.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.802+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/1333102989 wait complete. 2023-10-30T12:50:02.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fe7f2640 1 Processor -- start 2023-10-30T12:50:02.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fe7f2640 1 -- start start 2023-10-30T12:50:02.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fe7f2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3f80cefa0 0x7fa3f81573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:02.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fe7f2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3f81538f0 0x7fa3f8145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:02.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fe7f2640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3f815cd50 0x7fa3f81f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:02.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fe7f2640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f8152ad0 con 0x7fa3f81538f0 2023-10-30T12:50:02.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fe7f2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f8146490 con 0x7fa3f80cefa0 2023-10-30T12:50:02.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fe7f2640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f8157ff0 con 0x7fa3f815cd50 2023-10-30T12:50:02.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fcfef640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3f81538f0 0x7fa3f8145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:02.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fcfef640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3f81538f0 0x7fa3f8145f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47026/0 (socket says 172.21.15.125:47026) 2023-10-30T12:50:02.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fcfef640 1 -- 172.21.15.125:0/2171742041 learned_addr learned my addr 172.21.15.125:0/2171742041 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:50:02.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fd7f0640 1 --2- 172.21.15.125:0/2171742041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3f80cefa0 0x7fa3f81573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:02.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fdff1640 1 --2- 172.21.15.125:0/2171742041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3f815cd50 0x7fa3f81f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:02.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fcfef640 1 -- 172.21.15.125:0/2171742041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3f815cd50 msgr2=0x7fa3f81f2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:02.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fcfef640 1 --2- 172.21.15.125:0/2171742041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3f815cd50 0x7fa3f81f2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:02.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fcfef640 1 -- 172.21.15.125:0/2171742041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3f80cefa0 msgr2=0x7fa3f81573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:02.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fcfef640 1 --2- 172.21.15.125:0/2171742041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3f80cefa0 0x7fa3f81573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:02.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fcfef640 1 -- 172.21.15.125:0/2171742041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f40094d0 con 0x7fa3f81538f0 2023-10-30T12:50:02.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fcfef640 1 --2- 172.21.15.125:0/2171742041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3f81538f0 0x7fa3f8145f50 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800e870 tx=0x7fa3e800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:02.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3edffb640 1 -- 172.21.15.125:0/2171742041 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e8009510 con 0x7fa3f81538f0 2023-10-30T12:50:02.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3edffb640 1 -- 172.21.15.125:0/2171742041 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e800eec0 con 0x7fa3f81538f0 2023-10-30T12:50:02.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3edffb640 1 -- 172.21.15.125:0/2171742041 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e800cb40 con 0x7fa3f81538f0 2023-10-30T12:50:02.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/2171742041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3f8157a30 con 0x7fa3f81538f0 2023-10-30T12:50:02.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.806+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/2171742041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f8152480 con 0x7fa3f81538f0 2023-10-30T12:50:02.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.810+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/2171742041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f8152c80 con 0x7fa3f81538f0 2023-10-30T12:50:02.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.810+0000 7fa3edffb640 1 -- 172.21.15.125:0/2171742041 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa3e800cca0 con 0x7fa3f81538f0 2023-10-30T12:50:02.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.810+0000 7fa3edffb640 1 --2- 172.21.15.125:0/2171742041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa3dc066520 0x7fa3dc0689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:50:02.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.810+0000 7fa3edffb640 1 -- 172.21.15.125:0/2171742041 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fa3e8084160 con 0x7fa3f81538f0 2023-10-30T12:50:02.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.810+0000 7fa3fd7f0640 1 --2- 172.21.15.125:0/2171742041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa3dc066520 0x7fa3dc0689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:50:02.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.814+0000 7fa3fd7f0640 1 --2- 172.21.15.125:0/2171742041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa3dc066520 0x7fa3dc0689e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40b2000 tx=0x7fa3f4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:50:02.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.814+0000 7fa3edffb640 1 -- 172.21.15.125:0/2171742041 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa3f8152c80 con 0x7fa3f81538f0 2023-10-30T12:50:02.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.994+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/2171742041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3f804e170 con 0x7fa3f81538f0 2023-10-30T12:50:02.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.994+0000 7fa3edffb640 1 -- 172.21.15.125:0/2171742041 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 110 v110) v1 ==== 95+0+6522 (secure 0 0 0) 0x7fa3e8051360 con 0x7fa3f81538f0 2023-10-30T12:50:02.996 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:50:02.996 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":110,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/1874762894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":1874762894},{"type":"v1","addr":"172.21.15.136:6839","nonce":1874762894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":110},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:57.338574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:50:02.997 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 110 2023-10-30T12:50:02.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.998+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/2171742041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa3dc066520 msgr2=0x7fa3dc0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:02.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.998+0000 7fa3fe7f2640 1 --2- 172.21.15.125:0/2171742041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa3dc066520 0x7fa3dc0689e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40b2000 tx=0x7fa3f4009040 comp rx=0 tx=0).stop 2023-10-30T12:50:02.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.998+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/2171742041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3f81538f0 msgr2=0x7fa3f8145f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:50:02.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.998+0000 7fa3fe7f2640 1 --2- 172.21.15.125:0/2171742041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3f81538f0 0x7fa3f8145f50 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fa3e800e870 tx=0x7fa3e800ec70 comp rx=0 tx=0).stop 2023-10-30T12:50:02.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.998+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/2171742041 shutdown_connections 2023-10-30T12:50:02.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.998+0000 7fa3fe7f2640 1 --2- 172.21.15.125:0/2171742041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa3dc066520 0x7fa3dc0689e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:02.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.998+0000 7fa3fe7f2640 1 --2- 172.21.15.125:0/2171742041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa3f815cd50 0x7fa3f81f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:02.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.998+0000 7fa3fe7f2640 1 --2- 172.21.15.125:0/2171742041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa3f81538f0 0x7fa3f8145f50 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:03.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.998+0000 7fa3fe7f2640 1 --2- 172.21.15.125:0/2171742041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa3f80cefa0 0x7fa3f81573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:50:03.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.998+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/2171742041 >> 172.21.15.125:0/2171742041 conn(0x7fa3f806a640 msgr2=0x7fa3f8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:50:03.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.998+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/2171742041 shutdown_connections 2023-10-30T12:50:03.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:50:02.998+0000 7fa3fe7f2640 1 -- 172.21.15.125:0/2171742041 wait complete. 2023-10-30T12:50:03.016 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T12:50:03.017 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 57 secs before thrashing 2023-10-30T12:50:05.801 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 7% (18/247) Updating files: 8% (20/247) Updating files: 9% (23/247) Updating files: 10% (25/247) Updating files: 11% (28/247) Updating files: 12% (30/247) Updating files: 13% (33/247) Updating files: 14% (35/247) Updating files: 15% (38/247) Updating files: 16% (40/247) Updating files: 17% (42/247) Updating files: 18% (45/247) Updating files: 19% (47/247) Updating files: 20% (50/247) Updating files: 21% (52/247) Updating files: 22% (55/247) Updating files: 23% (57/247) Updating files: 24% (60/247) Updating files: 25% (62/247) Updating files: 26% (65/247) Updating files: 27% (67/247) Updating files: 28% (70/247) Updating files: 29% (72/247) Updating files: 30% (75/247) Updating files: 31% (77/247) Updating files: 31% (79/247) Updating files: 32% (80/247) Updating files: 33% (82/247) Updating files: 34% (84/247) Updating files: 35% (87/247) Updating files: 36% (89/247) Updating files: 37% (92/247) Updating files: 38% (94/247) Updating files: 39% (97/247) Updating files: 40% (99/247) Updating files: 41% (102/247) Updating files: 42% (104/247) Updating files: 43% (107/247) Updating files: 44% (109/247) Updating files: 45% (112/247) Updating files: 46% (114/247) Updating files: 47% (117/247) Updating files: 48% (119/247) Updating files: 49% (122/247) Updating files: 50% (124/247) Updating files: 51% (126/247) Updating files: 52% (129/247) Updating files: 52% (130/247) Updating files: 53% (131/247) Updating files: 54% (134/247) Updating files: 55% (136/247) Updating files: 56% (139/247) Updating files: 57% (141/247) Updating files: 58% (144/247) Updating files: 59% (146/247) Updating files: 60% (149/247) Updating files: 61% (151/247) Updating files: 62% (154/247) Updating files: 63% (156/247) Updating files: 64% (159/247) Updating files: 65% (161/247) Updating files: 66% (164/247) Updating files: 67% (166/247) Updating files: 68% (168/247) Updating files: 69% (171/247) Updating files: 70% (173/247) Updating files: 71% (176/247) Updating files: 72% (178/247) Updating files: 73% (181/247) Updating files: 74% (183/247) Updating files: 75% (186/247) Updating files: 76% (188/247) Updating files: 77% (191/247) Updating files: 78% (193/247) Updating files: 79% (196/247) Updating files: 80% (198/247) Updating files: 81% (201/247) Updating files: 82% (203/247) Updating files: 83% (206/247) Updating files: 84% (208/247) Updating files: 85% (210/247) Updating files: 86% (213/247) Updating files: 87% (215/247) Updating files: 88% (218/247) Updating files: 89% (220/247) Updating files: 90% (223/247) Updating files: 91% (225/247) Updating files: 92% (228/247) Updating files: 93% (230/247) Updating files: 94% (233/247) Updating files: 95% (235/247) Updating files: 96% (238/247) Updating files: 97% (240/247) Updating files: 98% (243/247) Updating files: 99% (245/247) Updating files: 100% (247/247) Updating files: 100% (247/247), done. 2023-10-30T12:50:05.818 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 13b1bf7c492 kclient: initializing kaddr (merge fix) 2023-10-30T12:50:05.825 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.4 2023-10-30T12:50:06.111 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 5 -eq 48 ] 2023-10-30T12:50:06.111 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.5 2023-10-30T12:50:06.112 INFO:tasks.workunit.client.0.smithi125.stdout:v0.5 2023-10-30T12:50:06.112 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.5 2023-10-30T12:50:06.112 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.5 2023-10-30T12:50:11.184 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 35% (87/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 70% (173/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 97% (239/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2023-10-30T12:50:11.915 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 1dd420957f9 journal: detect size of raw block devices properly 2023-10-30T12:50:11.920 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.5 2023-10-30T12:50:12.737 INFO:tasks.workunit.client.0.smithi125.stdout:v0.6 2023-10-30T12:50:12.737 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 6 -eq 48 ] 2023-10-30T12:50:12.737 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.6 2023-10-30T12:50:12.738 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.6 2023-10-30T12:50:12.738 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.6 2023-10-30T12:50:17.501 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:50:17.505 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:50:31.176 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 11% (27/227) Updating files: 12% (28/227) Updating files: 13% (30/227) Updating files: 14% (32/227) Updating files: 15% (35/227) Updating files: 16% (37/227) Updating files: 17% (39/227) Updating files: 18% (41/227) Updating files: 19% (44/227) Updating files: 20% (46/227) Updating files: 21% (48/227) Updating files: 22% (50/227) Updating files: 23% (53/227) Updating files: 24% (55/227) Updating files: 25% (57/227) Updating files: 26% (60/227) Updating files: 27% (62/227) Updating files: 28% (64/227) Updating files: 29% (66/227) Updating files: 30% (69/227) Updating files: 31% (71/227) Updating files: 32% (73/227) Updating files: 33% (75/227) Updating files: 34% (78/227) Updating files: 35% (80/227) Updating files: 36% (82/227) Updating files: 37% (84/227) Updating files: 37% (86/227) Updating files: 38% (87/227) Updating files: 39% (89/227) Updating files: 40% (91/227) Updating files: 41% (94/227) Updating files: 42% (96/227) Updating files: 43% (98/227) Updating files: 44% (100/227) Updating files: 45% (103/227) Updating files: 46% (105/227) Updating files: 47% (107/227) Updating files: 48% (109/227) Updating files: 49% (112/227) Updating files: 50% (114/227) Updating files: 51% (116/227) Updating files: 52% (119/227) Updating files: 53% (121/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 68% (156/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 89% (204/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2023-10-30T12:50:31.201 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at a4c752df713 v0.6 2023-10-30T12:50:31.206 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.6 2023-10-30T12:50:31.712 INFO:tasks.workunit.client.0.smithi125.stdout:v0.7 2023-10-30T12:50:31.713 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 7 -eq 48 ] 2023-10-30T12:50:31.713 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.7 2023-10-30T12:50:31.713 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.7 2023-10-30T12:50:31.714 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.7 2023-10-30T12:50:34.168 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 16% (34/201) Updating files: 17% (35/201) Updating files: 18% (37/201) Updating files: 19% (39/201) Updating files: 20% (41/201) Updating files: 21% (43/201) Updating files: 22% (45/201) Updating files: 23% (47/201) Updating files: 24% (49/201) Updating files: 25% (51/201) Updating files: 26% (53/201) Updating files: 27% (55/201) Updating files: 28% (57/201) Updating files: 29% (59/201) Updating files: 30% (61/201) Updating files: 31% (63/201) Updating files: 32% (65/201) Updating files: 33% (67/201) Updating files: 34% (69/201) Updating files: 35% (71/201) Updating files: 36% (73/201) Updating files: 37% (75/201) Updating files: 38% (77/201) Updating files: 39% (79/201) Updating files: 40% (81/201) Updating files: 41% (83/201) Updating files: 42% (85/201) Updating files: 43% (87/201) Updating files: 44% (89/201) Updating files: 45% (91/201) Updating files: 46% (93/201) Updating files: 47% (95/201) Updating files: 48% (97/201) Updating files: 49% (99/201) Updating files: 50% (101/201) Updating files: 51% (103/201) Updating files: 52% (105/201) Updating files: 53% (107/201) Updating files: 54% (109/201) Updating files: 55% (111/201) Updating files: 56% (113/201) Updating files: 57% (115/201) Updating files: 58% (117/201) Updating files: 59% (119/201) Updating files: 60% (121/201) Updating files: 61% (123/201) Updating files: 62% (125/201) Updating files: 63% (127/201) Updating files: 64% (129/201) Updating files: 65% (131/201) Updating files: 66% (133/201) Updating files: 67% (135/201) Updating files: 68% (137/201) Updating files: 69% (139/201) Updating files: 70% (141/201) Updating files: 70% (142/201) Updating files: 71% (143/201) Updating files: 72% (145/201) Updating files: 73% (147/201) Updating files: 74% (149/201) Updating files: 75% (151/201) Updating files: 76% (153/201) Updating files: 77% (155/201) Updating files: 78% (157/201) Updating files: 79% (159/201) Updating files: 80% (161/201) Updating files: 81% (163/201) Updating files: 82% (165/201) Updating files: 83% (167/201) Updating files: 84% (169/201) Updating files: 85% (171/201) Updating files: 86% (173/201) Updating files: 87% (175/201) Updating files: 88% (177/201) Updating files: 89% (179/201) Updating files: 90% (181/201) Updating files: 91% (183/201) Updating files: 92% (185/201) Updating files: 93% (187/201) Updating files: 94% (189/201) Updating files: 95% (191/201) Updating files: 96% (193/201) Updating files: 97% (195/201) Updating files: 98% (197/201) Updating files: 99% (199/201) Updating files: 100% (201/201) Updating files: 100% (201/201), done. 2023-10-30T12:50:34.196 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at c46e1b320c6 mkcephfs: btrfsctl -a between mkfs and mount 2023-10-30T12:50:34.201 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.7 2023-10-30T12:50:34.395 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 8 -eq 48 ] 2023-10-30T12:50:34.396 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.8 2023-10-30T12:50:34.396 INFO:tasks.workunit.client.0.smithi125.stdout:v0.8 2023-10-30T12:50:34.396 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.8 2023-10-30T12:50:34.397 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.8 2023-10-30T12:50:37.558 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 54% (122/225) Updating files: 55% (124/225) Updating files: 56% (126/225) Updating files: 57% (129/225) Updating files: 58% (131/225) Updating files: 59% (133/225) Updating files: 60% (135/225) Updating files: 61% (138/225) Updating files: 62% (140/225) Updating files: 63% (142/225) Updating files: 64% (144/225) Updating files: 65% (147/225) Updating files: 66% (149/225) Updating files: 67% (151/225) Updating files: 68% (153/225) Updating files: 69% (156/225) Updating files: 70% (158/225) Updating files: 71% (160/225) Updating files: 72% (162/225) Updating files: 73% (165/225) Updating files: 74% (167/225) Updating files: 75% (169/225) Updating files: 76% (171/225) Updating files: 77% (174/225) Updating files: 78% (176/225) Updating files: 79% (178/225) Updating files: 80% (180/225) Updating files: 81% (183/225) Updating files: 82% (185/225) Updating files: 83% (187/225) Updating files: 84% (189/225) Updating files: 85% (192/225) Updating files: 86% (194/225) Updating files: 87% (196/225) Updating files: 88% (198/225) Updating files: 89% (201/225) Updating files: 90% (203/225) Updating files: 91% (205/225) Updating files: 92% (207/225) Updating files: 93% (210/225) Updating files: 94% (212/225) Updating files: 95% (214/225) Updating files: 96% (216/225) Updating files: 97% (219/225) Updating files: 98% (221/225) Updating files: 99% (223/225) Updating files: 100% (225/225) Updating files: 100% (225/225), done. 2023-10-30T12:50:37.592 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 284161c00ab kclient: kill dput/d_drop debug cruft 2023-10-30T12:50:37.598 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.8 2023-10-30T12:50:37.664 INFO:tasks.workunit.client.0.smithi125.stdout:v0.9 2023-10-30T12:50:37.665 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 9 -eq 48 ] 2023-10-30T12:50:37.665 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.9 2023-10-30T12:50:37.665 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.9 2023-10-30T12:50:37.666 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.9 2023-10-30T12:50:47.566 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:50:47.570 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:50:57.677 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 62% (120/193) Updating files: 63% (122/193) Updating files: 64% (124/193) Updating files: 65% (126/193) Updating files: 66% (128/193) Updating files: 67% (130/193) Updating files: 68% (132/193) Updating files: 68% (133/193) Updating files: 69% (134/193) Updating files: 70% (136/193) Updating files: 71% (138/193) Updating files: 72% (139/193) Updating files: 73% (141/193) Updating files: 74% (143/193) Updating files: 75% (145/193) Updating files: 76% (147/193) Updating files: 77% (149/193) Updating files: 78% (151/193) Updating files: 79% (153/193) Updating files: 80% (155/193) Updating files: 81% (157/193) Updating files: 82% (159/193) Updating files: 83% (161/193) Updating files: 83% (162/193) Updating files: 84% (163/193) Updating files: 85% (165/193) Updating files: 86% (166/193) Updating files: 87% (168/193) Updating files: 88% (170/193) Updating files: 89% (172/193) Updating files: 90% (174/193) Updating files: 91% (176/193) Updating files: 92% (178/193) Updating files: 93% (180/193) Updating files: 94% (182/193) Updating files: 95% (184/193) Updating files: 96% (186/193) Updating files: 97% (188/193) Updating files: 98% (190/193) Updating files: 99% (192/193) Updating files: 100% (193/193) Updating files: 100% (193/193), done. 2023-10-30T12:50:57.697 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 41481853924 kclient: strip out kernel version compatibility cruft 2023-10-30T12:50:57.702 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.9 2023-10-30T12:50:57.758 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 10 -eq 48 ] 2023-10-30T12:50:57.758 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.10 2023-10-30T12:50:57.758 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.10 2023-10-30T12:50:57.759 INFO:tasks.workunit.client.0.smithi125.stdout:v0.10 2023-10-30T12:50:57.759 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.10 2023-10-30T12:50:59.768 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 24% (40/164) Updating files: 25% (41/164) Updating files: 26% (43/164) Updating files: 27% (45/164) Updating files: 28% (46/164) Updating files: 29% (48/164) Updating files: 30% (50/164) Updating files: 31% (51/164) Updating files: 32% (53/164) Updating files: 33% (55/164) Updating files: 34% (56/164) Updating files: 35% (58/164) Updating files: 36% (60/164) Updating files: 37% (61/164) Updating files: 38% (63/164) Updating files: 39% (64/164) Updating files: 40% (66/164) Updating files: 41% (68/164) Updating files: 42% (69/164) Updating files: 43% (71/164) Updating files: 44% (73/164) Updating files: 45% (74/164) Updating files: 46% (76/164) Updating files: 47% (78/164) Updating files: 48% (79/164) Updating files: 49% (81/164) Updating files: 50% (82/164) Updating files: 51% (84/164) Updating files: 52% (86/164) Updating files: 53% (87/164) Updating files: 54% (89/164) Updating files: 55% (91/164) Updating files: 56% (92/164) Updating files: 57% (94/164) Updating files: 58% (96/164) Updating files: 59% (97/164) Updating files: 60% (99/164) Updating files: 61% (101/164) Updating files: 62% (102/164) Updating files: 63% (104/164) Updating files: 64% (105/164) Updating files: 65% (107/164) Updating files: 66% (109/164) Updating files: 67% (110/164) Updating files: 68% (112/164) Updating files: 69% (114/164) Updating files: 70% (115/164) Updating files: 71% (117/164) Updating files: 72% (119/164) Updating files: 73% (120/164) Updating files: 74% (122/164) Updating files: 75% (123/164) Updating files: 76% (125/164) Updating files: 77% (127/164) Updating files: 78% (128/164) Updating files: 79% (130/164) Updating files: 80% (132/164) Updating files: 81% (133/164) Updating files: 82% (135/164) Updating files: 83% (137/164) Updating files: 84% (138/164) Updating files: 85% (140/164) Updating files: 86% (142/164) Updating files: 87% (143/164) Updating files: 88% (145/164) Updating files: 89% (146/164) Updating files: 90% (148/164) Updating files: 91% (150/164) Updating files: 92% (151/164) Updating files: 93% (153/164) Updating files: 94% (155/164) Updating files: 95% (156/164) Updating files: 96% (158/164) Updating files: 97% (160/164) Updating files: 98% (161/164) Updating files: 99% (163/164) Updating files: 100% (164/164) Updating files: 100% (164/164), done. 2023-10-30T12:50:59.796 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 007313fe844 v0.10 2023-10-30T12:50:59.801 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.10 2023-10-30T12:51:00.017 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:00.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4e393e640 1 Processor -- start 2023-10-30T12:51:00.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4e393e640 1 -- start start 2023-10-30T12:51:00.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4e393e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4dc0cf160 0x7fa4dc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:00.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4e393e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4dc0cb800 con 0x7fa4dc1538a0 2023-10-30T12:51:00.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4e393e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4dc0cb330 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4e393e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc0cad70 con 0x7fa4dc0cf160 2023-10-30T12:51:00.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4e16da640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa4dc1538a0 0x7fa4dc0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37414/0 (socket says 172.21.15.125:37414) 2023-10-30T12:51:00.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4e16da640 1 -- 172.21.15.125:0/2471962508 learned_addr learned my addr 172.21.15.125:0/2471962508 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:51:00.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4e1edb640 1 --2- 172.21.15.125:0/2471962508 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4dc0cf160 0x7fa4dc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:00.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4d3fff640 1 -- 172.21.15.125:0/2471962508 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325011208 0 0) 0x7fa4dc0cb800 con 0x7fa4dc1538a0 2023-10-30T12:51:00.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4d3fff640 1 -- 172.21.15.125:0/2471962508 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c8003530 con 0x7fa4dc1538a0 2023-10-30T12:51:00.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4d3fff640 1 -- 172.21.15.125:0/2471962508 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4117001615 0 0) 0x7fa4dc0cb330 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4e1edb640 1 -- 172.21.15.125:0/2471962508 >> v1:172.21.15.136:6789/0 conn(0x7fa4dc15d6f0 legacy=0x7fa4dc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4e1edb640 1 -- 172.21.15.125:0/2471962508 >> v1:172.21.15.125:6789/0 conn(0x7fa4dc1538a0 legacy=0x7fa4dc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.156+0000 7fa4e1edb640 1 -- 172.21.15.125:0/2471962508 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4dc0cb050 con 0x7fa4dc0cf160 2023-10-30T12:51:00.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.160+0000 7fa4e1edb640 1 --2- 172.21.15.125:0/2471962508 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4dc0cf160 0x7fa4dc0cf590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4009dc0 tx=0x7fa4d401f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8b0a866293f99b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:00.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.160+0000 7fa4d3fff640 1 -- 172.21.15.125:0/2471962508 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d40af600 con 0x7fa4dc0cf160 2023-10-30T12:51:00.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.160+0000 7fa4d3fff640 1 -- 172.21.15.125:0/2471962508 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d40ab050 con 0x7fa4dc0cf160 2023-10-30T12:51:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.160+0000 7fa4d3fff640 1 -- 172.21.15.125:0/2471962508 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d40af960 con 0x7fa4dc0cf160 2023-10-30T12:51:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.160+0000 7fa4e393e640 1 -- 172.21.15.125:0/2471962508 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4dc0cf160 msgr2=0x7fa4dc0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.160+0000 7fa4e393e640 1 --2- 172.21.15.125:0/2471962508 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4dc0cf160 0x7fa4dc0cf590 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa4d4009dc0 tx=0x7fa4d401f150 comp rx=0 tx=0).stop 2023-10-30T12:51:00.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.160+0000 7fa4e393e640 1 -- 172.21.15.125:0/2471962508 shutdown_connections 2023-10-30T12:51:00.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.160+0000 7fa4e393e640 1 --2- 172.21.15.125:0/2471962508 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4dc0cf160 0x7fa4dc0cf590 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.160+0000 7fa4e393e640 1 -- 172.21.15.125:0/2471962508 >> 172.21.15.125:0/2471962508 conn(0x7fa4dc06bfe0 msgr2=0x7fa4dc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:00.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.160+0000 7fa4e393e640 1 -- 172.21.15.125:0/2471962508 shutdown_connections 2023-10-30T12:51:00.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.160+0000 7fa4e393e640 1 -- 172.21.15.125:0/2471962508 wait complete. 2023-10-30T12:51:00.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e393e640 1 Processor -- start 2023-10-30T12:51:00.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e393e640 1 -- start start 2023-10-30T12:51:00.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e393e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4dc0cf160 0x7fa4dc14ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:00.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e393e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4dc1538a0 0x7fa4dc15ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:00.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e393e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4dc15d6f0 0x7fa4dc1507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:00.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e393e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc1572b0 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e393e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc14e1b0 con 0x7fa4dc0cf160 2023-10-30T12:51:00.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e393e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4dc1551c0 con 0x7fa4dc1538a0 2023-10-30T12:51:00.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e1edb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4dc15d6f0 0x7fa4dc1507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:00.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e1edb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4dc15d6f0 0x7fa4dc1507a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55602/0 (socket says 172.21.15.125:55602) 2023-10-30T12:51:00.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e1edb640 1 -- 172.21.15.125:0/95705704 learned_addr learned my addr 172.21.15.125:0/95705704 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:00.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e0ed9640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4dc1538a0 0x7fa4dc15ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:00.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e16da640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4dc0cf160 0x7fa4dc14ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:00.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e1edb640 1 -- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4dc1538a0 msgr2=0x7fa4dc15ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e1edb640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4dc1538a0 0x7fa4dc15ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e1edb640 1 -- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4dc0cf160 msgr2=0x7fa4dc14ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e1edb640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4dc0cf160 0x7fa4dc14ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e1edb640 1 -- 172.21.15.125:0/95705704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d4009a70 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e0ed9640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4dc1538a0 0x7fa4dc15ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:51:00.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e1edb640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4dc15d6f0 0x7fa4dc1507a0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fa4d401cd80 tx=0x7fa4d401bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:00.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4d1ffb640 1 -- 172.21.15.125:0/95705704 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d40bf070 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4d1ffb640 1 -- 172.21.15.125:0/95705704 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d4009450 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4e393e640 1 -- 172.21.15.125:0/95705704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4dc14dbf0 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.164+0000 7fa4d1ffb640 1 -- 172.21.15.125:0/95705704 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d4004450 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.168+0000 7fa4e393e640 1 -- 172.21.15.125:0/95705704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4dc15a9e0 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.168+0000 7fa4e393e640 1 -- 172.21.15.125:0/95705704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4dc04e170 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.168+0000 7fa4d1ffb640 1 -- 172.21.15.125:0/95705704 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa4d40045b0 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.168+0000 7fa4d1ffb640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4c00664d0 0x7fa4c0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:00.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.168+0000 7fa4d1ffb640 1 -- 172.21.15.125:0/95705704 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fa4d4122340 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.168+0000 7fa4e16da640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4c00664d0 0x7fa4c0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:00.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.172+0000 7fa4e16da640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4c00664d0 0x7fa4c0068990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8007fd0 tx=0x7fa4d8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:00.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.172+0000 7fa4d1ffb640 1 -- 172.21.15.125:0/95705704 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa4d40f2ee0 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.181 INFO:tasks.workunit.client.0.smithi125.stdout:v0.11 2023-10-30T12:51:00.181 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 11 -eq 48 ] 2023-10-30T12:51:00.182 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.11 2023-10-30T12:51:00.182 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.11 2023-10-30T12:51:00.182 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.11 2023-10-30T12:51:00.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.456+0000 7fa4e393e640 1 -- 172.21.15.125:0/95705704 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4dc0b1dd0 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.456+0000 7fa4d1ffb640 1 -- 172.21.15.125:0/95705704 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fa4d40f2a70 con 0x7fa4dc15d6f0 2023-10-30T12:51:00.460 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:00.460 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:57.338574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:00.461 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 111 2023-10-30T12:51:00.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.460+0000 7fa4e393e640 1 -- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4c00664d0 msgr2=0x7fa4c0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.460+0000 7fa4e393e640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4c00664d0 0x7fa4c0068990 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8007fd0 tx=0x7fa4d8009040 comp rx=0 tx=0).stop 2023-10-30T12:51:00.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.460+0000 7fa4e393e640 1 -- 172.21.15.125:0/95705704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4dc15d6f0 msgr2=0x7fa4dc1507a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.460+0000 7fa4e393e640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4dc15d6f0 0x7fa4dc1507a0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fa4d401cd80 tx=0x7fa4d401bef0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.460+0000 7fa4e393e640 1 -- 172.21.15.125:0/95705704 shutdown_connections 2023-10-30T12:51:00.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.460+0000 7fa4e393e640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4c00664d0 0x7fa4c0068990 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.460+0000 7fa4e393e640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa4dc15d6f0 0x7fa4dc1507a0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.460+0000 7fa4e393e640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa4dc1538a0 0x7fa4dc15ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.460+0000 7fa4e393e640 1 --2- 172.21.15.125:0/95705704 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa4dc0cf160 0x7fa4dc14ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.460+0000 7fa4e393e640 1 -- 172.21.15.125:0/95705704 >> 172.21.15.125:0/95705704 conn(0x7fa4dc06bfe0 msgr2=0x7fa4dc0c5640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:00.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.460+0000 7fa4e393e640 1 -- 172.21.15.125:0/95705704 shutdown_connections 2023-10-30T12:51:00.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.460+0000 7fa4e393e640 1 -- 172.21.15.125:0/95705704 wait complete. 2023-10-30T12:51:00.484 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T12:51:00.484 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T12:51:00.485 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:51:00.505 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T12:51:00.505 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:00.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa12c846640 1 Processor -- start 2023-10-30T12:51:00.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa12c846640 1 -- start start 2023-10-30T12:51:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa12c846640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa1280c6b80 0x7fa1280bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa12c846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1280c6f60 0x7fa1280c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa12c846640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1280c0a50 con 0x7fa1280c8430 2023-10-30T12:51:00.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa12c846640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa12806f6e0 con 0x7fa1280c6b80 2023-10-30T12:51:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa12c846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa12806f9c0 con 0x7fa1280c6f60 2023-10-30T12:51:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa12659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa1280c6b80 0x7fa1280bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa12659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa1280c6b80 0x7fa1280bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55612/0 (socket says 172.21.15.125:55612) 2023-10-30T12:51:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa12659c640 1 -- 172.21.15.125:0/3974161166 learned_addr learned my addr 172.21.15.125:0/3974161166 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:00.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa125d9b640 1 --2- 172.21.15.125:0/3974161166 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1280c6f60 0x7fa1280c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:00.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa12659c640 1 -- 172.21.15.125:0/3974161166 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1280c6f60 msgr2=0x7fa1280c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:00.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa12659c640 1 --2- 172.21.15.125:0/3974161166 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1280c6f60 0x7fa1280c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa12659c640 1 -- 172.21.15.125:0/3974161166 >> v1:172.21.15.136:6789/0 conn(0x7fa1280c8430 legacy=0x7fa12813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:00.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.684+0000 7fa12659c640 1 -- 172.21.15.125:0/3974161166 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12806fca0 con 0x7fa1280c6b80 2023-10-30T12:51:00.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.688+0000 7fa12659c640 1 --2- 172.21.15.125:0/3974161166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1280c6b80 0x7fa1280bfec0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fa118009870 tx=0x7fa11801c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377675f5a69bb74d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:00.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.688+0000 7fa12559a640 1 -- 172.21.15.125:0/3974161166 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1180ab600 con 0x7fa1280c6b80 2023-10-30T12:51:00.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.688+0000 7fa12559a640 1 -- 172.21.15.125:0/3974161166 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1180a7050 con 0x7fa1280c6b80 2023-10-30T12:51:00.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.688+0000 7fa12559a640 1 -- 172.21.15.125:0/3974161166 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1180ab980 con 0x7fa1280c6b80 2023-10-30T12:51:00.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.688+0000 7fa12c846640 1 -- 172.21.15.125:0/3974161166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1280c6b80 msgr2=0x7fa1280bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.688+0000 7fa12c846640 1 --2- 172.21.15.125:0/3974161166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1280c6b80 0x7fa1280bfec0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fa118009870 tx=0x7fa11801c470 comp rx=0 tx=0).stop 2023-10-30T12:51:00.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.688+0000 7fa12c846640 1 -- 172.21.15.125:0/3974161166 shutdown_connections 2023-10-30T12:51:00.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.688+0000 7fa12c846640 1 --2- 172.21.15.125:0/3974161166 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1280c6f60 0x7fa1280c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.688+0000 7fa12c846640 1 --2- 172.21.15.125:0/3974161166 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1280c6b80 0x7fa1280bfec0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.688+0000 7fa12c846640 1 -- 172.21.15.125:0/3974161166 >> 172.21.15.125:0/3974161166 conn(0x7fa1280c7910 msgr2=0x7fa1280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:00.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.688+0000 7fa12c846640 1 -- 172.21.15.125:0/3974161166 shutdown_connections 2023-10-30T12:51:00.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.688+0000 7fa12c846640 1 -- 172.21.15.125:0/3974161166 wait complete. 2023-10-30T12:51:00.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa12c846640 1 Processor -- start 2023-10-30T12:51:00.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa12c846640 1 -- start start 2023-10-30T12:51:00.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa12c846640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1280c6b80 0x7fa1281f3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:00.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa12c846640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1280c6f60 0x7fa1280c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:00.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa12c846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1280c8430 0x7fa128152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:00.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa12c846640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280c9620 con 0x7fa1280c6f60 2023-10-30T12:51:00.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa12c846640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1281f3a50 con 0x7fa1280c6b80 2023-10-30T12:51:00.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa12c846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1280c54e0 con 0x7fa1280c8430 2023-10-30T12:51:00.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa125d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1280c6f60 0x7fa1280c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:00.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa125d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1280c6f60 0x7fa1280c4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55626/0 (socket says 172.21.15.125:55626) 2023-10-30T12:51:00.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa125d9b640 1 -- 172.21.15.125:0/479221859 learned_addr learned my addr 172.21.15.125:0/479221859 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:00.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa12659c640 1 --2- 172.21.15.125:0/479221859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1280c6b80 0x7fa1281f3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:00.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa126d9d640 1 --2- 172.21.15.125:0/479221859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1280c8430 0x7fa128152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:00.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa125d9b640 1 -- 172.21.15.125:0/479221859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1280c8430 msgr2=0x7fa128152f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa125d9b640 1 --2- 172.21.15.125:0/479221859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1280c8430 0x7fa128152f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa125d9b640 1 -- 172.21.15.125:0/479221859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1280c6b80 msgr2=0x7fa1281f3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa125d9b640 1 --2- 172.21.15.125:0/479221859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1280c6b80 0x7fa1281f3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa125d9b640 1 -- 172.21.15.125:0/479221859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1180094d0 con 0x7fa1280c6f60 2023-10-30T12:51:00.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa125d9b640 1 --2- 172.21.15.125:0/479221859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1280c6f60 0x7fa1280c4fa0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00d830 tx=0x7fa11c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:00.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa116ffd640 1 -- 172.21.15.125:0/479221859 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c00b6c0 con 0x7fa1280c6f60 2023-10-30T12:51:00.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa116ffd640 1 -- 172.21.15.125:0/479221859 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11c00b820 con 0x7fa1280c6f60 2023-10-30T12:51:00.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa12c846640 1 -- 172.21.15.125:0/479221859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa128155860 con 0x7fa1280c6f60 2023-10-30T12:51:00.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa116ffd640 1 -- 172.21.15.125:0/479221859 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11c00bb90 con 0x7fa1280c6f60 2023-10-30T12:51:00.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa12c846640 1 -- 172.21.15.125:0/479221859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1280c9b50 con 0x7fa1280c6f60 2023-10-30T12:51:00.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.692+0000 7fa12c846640 1 -- 172.21.15.125:0/479221859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa128155620 con 0x7fa1280c6f60 2023-10-30T12:51:00.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.696+0000 7fa116ffd640 1 -- 172.21.15.125:0/479221859 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa11c01e460 con 0x7fa1280c6f60 2023-10-30T12:51:00.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.696+0000 7fa116ffd640 1 --2- 172.21.15.125:0/479221859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1080664a0 0x7fa108068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:00.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.696+0000 7fa116ffd640 1 -- 172.21.15.125:0/479221859 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fa11c084ac0 con 0x7fa1280c6f60 2023-10-30T12:51:00.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.696+0000 7fa12659c640 1 --2- 172.21.15.125:0/479221859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1080664a0 0x7fa108068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:00.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.696+0000 7fa12659c640 1 --2- 172.21.15.125:0/479221859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1080664a0 0x7fa108068960 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa1280b0860 tx=0x7fa118009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:00.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.700+0000 7fa116ffd640 1 -- 172.21.15.125:0/479221859 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa128155620 con 0x7fa1280c6f60 2023-10-30T12:51:00.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.880+0000 7fa12c846640 1 -- 172.21.15.125:0/479221859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa128155620 con 0x7fa1280c6f60 2023-10-30T12:51:00.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.880+0000 7fa116ffd640 1 -- 172.21.15.125:0/479221859 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fa128155620 con 0x7fa1280c6f60 2023-10-30T12:51:00.883 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:00.884 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:57.338574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:00.884 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 111 2023-10-30T12:51:00.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.884+0000 7fa12c846640 1 -- 172.21.15.125:0/479221859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1080664a0 msgr2=0x7fa108068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.884+0000 7fa12c846640 1 --2- 172.21.15.125:0/479221859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1080664a0 0x7fa108068960 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa1280b0860 tx=0x7fa118009040 comp rx=0 tx=0).stop 2023-10-30T12:51:00.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.884+0000 7fa12c846640 1 -- 172.21.15.125:0/479221859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1280c6f60 msgr2=0x7fa1280c4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:00.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.884+0000 7fa12c846640 1 --2- 172.21.15.125:0/479221859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1280c6f60 0x7fa1280c4fa0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00d830 tx=0x7fa11c00dcf0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.884+0000 7fa12c846640 1 -- 172.21.15.125:0/479221859 shutdown_connections 2023-10-30T12:51:00.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.884+0000 7fa12c846640 1 --2- 172.21.15.125:0/479221859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1080664a0 0x7fa108068960 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.884+0000 7fa12c846640 1 --2- 172.21.15.125:0/479221859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1280c8430 0x7fa128152f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.884+0000 7fa12c846640 1 --2- 172.21.15.125:0/479221859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1280c6f60 0x7fa1280c4fa0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.884+0000 7fa12c846640 1 --2- 172.21.15.125:0/479221859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1280c6b80 0x7fa1281f3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:00.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.884+0000 7fa12c846640 1 -- 172.21.15.125:0/479221859 >> 172.21.15.125:0/479221859 conn(0x7fa1280c7910 msgr2=0x7fa1280bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:00.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.884+0000 7fa12c846640 1 -- 172.21.15.125:0/479221859 shutdown_connections 2023-10-30T12:51:00.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:00.884+0000 7fa12c846640 1 -- 172.21.15.125:0/479221859 wait complete. 2023-10-30T12:51:00.904 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:51:02.905 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:03.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f3935874640 1 Processor -- start 2023-10-30T12:51:03.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f3935874640 1 -- start start 2023-10-30T12:51:03.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f3935874640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39301538f0 0x7f3930159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:03.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f3935874640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f393015cd50 0x7f39300c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:03.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f3935874640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f393015d300 con 0x7f39300cefa0 2023-10-30T12:51:03.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f3935874640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393015d5e0 con 0x7f393015cd50 2023-10-30T12:51:03.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f3935874640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f393015d8c0 con 0x7f39301538f0 2023-10-30T12:51:03.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f392ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f393015cd50 0x7f39300c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:03.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f392ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f393015cd50 0x7f39300c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44012/0 (socket says 172.21.15.125:44012) 2023-10-30T12:51:03.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f392ffff640 1 -- 172.21.15.125:0/2558913432 learned_addr learned my addr 172.21.15.125:0/2558913432 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:03.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f3934872640 1 --2- 172.21.15.125:0/2558913432 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39301538f0 0x7f3930159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:03.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f392ffff640 1 -- 172.21.15.125:0/2558913432 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39301538f0 msgr2=0x7f3930159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:03.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f392ffff640 1 --2- 172.21.15.125:0/2558913432 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39301538f0 0x7f3930159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f392ffff640 1 -- 172.21.15.125:0/2558913432 >> v1:172.21.15.136:6789/0 conn(0x7f39300cefa0 legacy=0x7f39300cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:03.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.052+0000 7f392ffff640 1 -- 172.21.15.125:0/2558913432 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393015dba0 con 0x7f393015cd50 2023-10-30T12:51:03.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.056+0000 7f392ffff640 1 --2- 172.21.15.125:0/2558913432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f393015cd50 0x7f39300c9f90 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f391c009870 tx=0x7f391c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d1e2f399ce516c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:03.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.056+0000 7f392f7fe640 1 -- 172.21.15.125:0/2558913432 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c0a8a30 con 0x7f393015cd50 2023-10-30T12:51:03.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.056+0000 7f392f7fe640 1 -- 172.21.15.125:0/2558913432 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f391c0a8b90 con 0x7f393015cd50 2023-10-30T12:51:03.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.056+0000 7f392f7fe640 1 -- 172.21.15.125:0/2558913432 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c0b0440 con 0x7f393015cd50 2023-10-30T12:51:03.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.056+0000 7f3935874640 1 -- 172.21.15.125:0/2558913432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f393015cd50 msgr2=0x7f39300c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:03.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.056+0000 7f3935874640 1 --2- 172.21.15.125:0/2558913432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f393015cd50 0x7f39300c9f90 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f391c009870 tx=0x7f391c01ed30 comp rx=0 tx=0).stop 2023-10-30T12:51:03.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.056+0000 7f3935874640 1 -- 172.21.15.125:0/2558913432 shutdown_connections 2023-10-30T12:51:03.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.056+0000 7f3935874640 1 --2- 172.21.15.125:0/2558913432 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f393015cd50 0x7f39300c9f90 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.056+0000 7f3935874640 1 --2- 172.21.15.125:0/2558913432 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39301538f0 0x7f3930159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.056+0000 7f3935874640 1 -- 172.21.15.125:0/2558913432 >> 172.21.15.125:0/2558913432 conn(0x7f393006a640 msgr2=0x7f3930065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:03.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.056+0000 7f3935874640 1 -- 172.21.15.125:0/2558913432 shutdown_connections 2023-10-30T12:51:03.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.056+0000 7f3935874640 1 -- 172.21.15.125:0/2558913432 wait complete. 2023-10-30T12:51:03.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.056+0000 7f3935874640 1 Processor -- start 2023-10-30T12:51:03.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.056+0000 7f3935874640 1 -- start start 2023-10-30T12:51:03.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935874640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39300cefa0 0x7f39301484d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:03.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935874640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39301538f0 0x7f39300ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:03.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935874640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f393015cd50 0x7f39301f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:03.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935874640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39301f26e0 con 0x7f393015cd50 2023-10-30T12:51:03.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935874640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f393014d750 con 0x7f39301538f0 2023-10-30T12:51:03.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935874640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3930147f50 con 0x7f39300cefa0 2023-10-30T12:51:03.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935073640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f393015cd50 0x7f39301f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:03.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935073640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f393015cd50 0x7f39301f2b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44020/0 (socket says 172.21.15.125:44020) 2023-10-30T12:51:03.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935073640 1 -- 172.21.15.125:0/2075158538 learned_addr learned my addr 172.21.15.125:0/2075158538 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:03.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3934872640 1 --2- 172.21.15.125:0/2075158538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39300cefa0 0x7f39301484d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:03.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f392ffff640 1 --2- 172.21.15.125:0/2075158538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39301538f0 0x7f39300ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:03.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935073640 1 -- 172.21.15.125:0/2075158538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39300cefa0 msgr2=0x7f39301484d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:03.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935073640 1 --2- 172.21.15.125:0/2075158538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39300cefa0 0x7f39301484d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935073640 1 -- 172.21.15.125:0/2075158538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39301538f0 msgr2=0x7f39300ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:03.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935073640 1 --2- 172.21.15.125:0/2075158538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39301538f0 0x7f39300ce1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935073640 1 -- 172.21.15.125:0/2075158538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f391c0094d0 con 0x7f393015cd50 2023-10-30T12:51:03.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935073640 1 --2- 172.21.15.125:0/2075158538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f393015cd50 0x7f39301f2b70 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f392800c770 tx=0x7f392800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:03.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f392d7fa640 1 -- 172.21.15.125:0/2075158538 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3928003b80 con 0x7f393015cd50 2023-10-30T12:51:03.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f392d7fa640 1 -- 172.21.15.125:0/2075158538 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3928003ce0 con 0x7f393015cd50 2023-10-30T12:51:03.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935874640 1 -- 172.21.15.125:0/2075158538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39301f23b0 con 0x7f393015cd50 2023-10-30T12:51:03.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f392d7fa640 1 -- 172.21.15.125:0/2075158538 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f392800aa90 con 0x7f393015cd50 2023-10-30T12:51:03.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935874640 1 -- 172.21.15.125:0/2075158538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39301481f0 con 0x7f393015cd50 2023-10-30T12:51:03.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.060+0000 7f3935874640 1 -- 172.21.15.125:0/2075158538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39301f2850 con 0x7f393015cd50 2023-10-30T12:51:03.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.064+0000 7f392d7fa640 1 -- 172.21.15.125:0/2075158538 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f392800abf0 con 0x7f393015cd50 2023-10-30T12:51:03.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.064+0000 7f392d7fa640 1 --2- 172.21.15.125:0/2075158538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f39140664a0 0x7f3914068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:03.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.064+0000 7f392d7fa640 1 -- 172.21.15.125:0/2075158538 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f39280840f0 con 0x7f393015cd50 2023-10-30T12:51:03.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.064+0000 7f3934872640 1 --2- 172.21.15.125:0/2075158538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f39140664a0 0x7f3914068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:03.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.064+0000 7f3934872640 1 --2- 172.21.15.125:0/2075158538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f39140664a0 0x7f3914068960 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f392400b9d0 tx=0x7f3924008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:03.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.068+0000 7f392d7fa640 1 -- 172.21.15.125:0/2075158538 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3928051160 con 0x7f393015cd50 2023-10-30T12:51:03.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.248+0000 7f3935874640 1 -- 172.21.15.125:0/2075158538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f393004e170 con 0x7f393015cd50 2023-10-30T12:51:03.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.248+0000 7f392d7fa640 1 -- 172.21.15.125:0/2075158538 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f3928054780 con 0x7f393015cd50 2023-10-30T12:51:03.252 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:03.252 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:57.338574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:03.253 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 111 2023-10-30T12:51:03.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.252+0000 7f3935874640 1 -- 172.21.15.125:0/2075158538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f39140664a0 msgr2=0x7f3914068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:03.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.252+0000 7f3935874640 1 --2- 172.21.15.125:0/2075158538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f39140664a0 0x7f3914068960 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f392400b9d0 tx=0x7f3924008040 comp rx=0 tx=0).stop 2023-10-30T12:51:03.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.252+0000 7f3935874640 1 -- 172.21.15.125:0/2075158538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f393015cd50 msgr2=0x7f39301f2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:03.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.252+0000 7f3935874640 1 --2- 172.21.15.125:0/2075158538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f393015cd50 0x7f39301f2b70 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f392800c770 tx=0x7f392800cc30 comp rx=0 tx=0).stop 2023-10-30T12:51:03.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.252+0000 7f3935874640 1 -- 172.21.15.125:0/2075158538 shutdown_connections 2023-10-30T12:51:03.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.252+0000 7f3935874640 1 --2- 172.21.15.125:0/2075158538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f39140664a0 0x7f3914068960 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.252+0000 7f3935874640 1 --2- 172.21.15.125:0/2075158538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f393015cd50 0x7f39301f2b70 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.252+0000 7f3935874640 1 --2- 172.21.15.125:0/2075158538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f39301538f0 0x7f39300ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.252+0000 7f3935874640 1 --2- 172.21.15.125:0/2075158538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39300cefa0 0x7f39301484d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:03.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.252+0000 7f3935874640 1 -- 172.21.15.125:0/2075158538 >> 172.21.15.125:0/2075158538 conn(0x7f393006a640 msgr2=0x7f39300c4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:03.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.252+0000 7f3935874640 1 -- 172.21.15.125:0/2075158538 shutdown_connections 2023-10-30T12:51:03.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:03.252+0000 7f3935874640 1 -- 172.21.15.125:0/2075158538 wait complete. 2023-10-30T12:51:03.272 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:51:05.273 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:05.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.420+0000 7f2ed5765640 1 Processor -- start 2023-10-30T12:51:05.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.420+0000 7f2ed5765640 1 -- start start 2023-10-30T12:51:05.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.420+0000 7f2ed5765640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ed0153d70 0x7f2ed006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:05.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.420+0000 7f2ed5765640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2ed00c8490 0x7f2ed00c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:05.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.420+0000 7f2ed5765640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed006afa0 con 0x7f2ed006e470 2023-10-30T12:51:05.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.420+0000 7f2ed5765640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed006d780 con 0x7f2ed00c8490 2023-10-30T12:51:05.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.420+0000 7f2ed5765640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed0154150 con 0x7f2ed0153d70 2023-10-30T12:51:05.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.420+0000 7f2eceffd640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ed0153d70 0x7f2ed006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:05.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.420+0000 7f2eceffd640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ed0153d70 0x7f2ed006ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38538/0 (socket says 172.21.15.125:38538) 2023-10-30T12:51:05.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.420+0000 7f2eceffd640 1 -- 172.21.15.125:0/68342230 learned_addr learned my addr 172.21.15.125:0/68342230 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:05.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.420+0000 7f2ecf7fe640 1 --2- 172.21.15.125:0/68342230 >> v2:172.21.15.136:3300/0 conn(0x7f2ed00c8490 0x7f2ed00c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:05.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.424+0000 7f2eceffd640 1 -- 172.21.15.125:0/68342230 >> v2:172.21.15.136:3300/0 conn(0x7f2ed00c8490 msgr2=0x7f2ed00c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:05.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.424+0000 7f2eceffd640 1 --2- 172.21.15.125:0/68342230 >> v2:172.21.15.136:3300/0 conn(0x7f2ed00c8490 0x7f2ed00c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.424+0000 7f2eceffd640 1 -- 172.21.15.125:0/68342230 >> v1:172.21.15.136:6789/0 conn(0x7f2ed006e470 legacy=0x7f2ed00cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:05.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.424+0000 7f2eceffd640 1 -- 172.21.15.125:0/68342230 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed006e030 con 0x7f2ed0153d70 2023-10-30T12:51:05.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.424+0000 7f2eceffd640 1 --2- 172.21.15.125:0/68342230 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ed0153d70 0x7f2ed006ca80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0009870 tx=0x7f2ec001c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e50231cd5222094c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:05.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.424+0000 7f2ece7fc640 1 -- 172.21.15.125:0/68342230 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00aa670 con 0x7f2ed0153d70 2023-10-30T12:51:05.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.424+0000 7f2ece7fc640 1 -- 172.21.15.125:0/68342230 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec00aa7d0 con 0x7f2ed0153d70 2023-10-30T12:51:05.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.424+0000 7f2ece7fc640 1 -- 172.21.15.125:0/68342230 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00aab60 con 0x7f2ed0153d70 2023-10-30T12:51:05.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 -- 172.21.15.125:0/68342230 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ed0153d70 msgr2=0x7f2ed006ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:05.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 --2- 172.21.15.125:0/68342230 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ed0153d70 0x7f2ed006ca80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0009870 tx=0x7f2ec001c510 comp rx=0 tx=0).stop 2023-10-30T12:51:05.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 -- 172.21.15.125:0/68342230 shutdown_connections 2023-10-30T12:51:05.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 --2- 172.21.15.125:0/68342230 >> v2:172.21.15.136:3300/0 conn(0x7f2ed00c8490 0x7f2ed00c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 --2- 172.21.15.125:0/68342230 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ed0153d70 0x7f2ed006ca80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 -- 172.21.15.125:0/68342230 >> 172.21.15.125:0/68342230 conn(0x7f2ed0071ea0 msgr2=0x7f2ed0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:05.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 -- 172.21.15.125:0/68342230 shutdown_connections 2023-10-30T12:51:05.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 -- 172.21.15.125:0/68342230 wait complete. 2023-10-30T12:51:05.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 Processor -- start 2023-10-30T12:51:05.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 -- start start 2023-10-30T12:51:05.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ed006e470 0x7f2ed01f6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:05.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ed00c8490 0x7f2ed00caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:05.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ed0153d70 0x7f2ed014c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:05.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed014b030 con 0x7f2ed0153d70 2023-10-30T12:51:05.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed014b910 con 0x7f2ed006e470 2023-10-30T12:51:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ed5765640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed00cb950 con 0x7f2ed00c8490 2023-10-30T12:51:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ecf7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ed0153d70 0x7f2ed014c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ecf7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ed0153d70 0x7f2ed014c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44032/0 (socket says 172.21.15.125:44032) 2023-10-30T12:51:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ecf7fe640 1 -- 172.21.15.125:0/1923732774 learned_addr learned my addr 172.21.15.125:0/1923732774 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:05.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2eceffd640 1 --2- 172.21.15.125:0/1923732774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ed006e470 0x7f2ed01f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:05.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ec7fff640 1 --2- 172.21.15.125:0/1923732774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ed00c8490 0x7f2ed00caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:05.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ecf7fe640 1 -- 172.21.15.125:0/1923732774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ed00c8490 msgr2=0x7f2ed00caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:05.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ecf7fe640 1 --2- 172.21.15.125:0/1923732774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ed00c8490 0x7f2ed00caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ecf7fe640 1 -- 172.21.15.125:0/1923732774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ed006e470 msgr2=0x7f2ed01f6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:05.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ecf7fe640 1 --2- 172.21.15.125:0/1923732774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ed006e470 0x7f2ed01f6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.428+0000 7f2ecf7fe640 1 -- 172.21.15.125:0/1923732774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec00094d0 con 0x7f2ed0153d70 2023-10-30T12:51:05.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.432+0000 7f2ecf7fe640 1 --2- 172.21.15.125:0/1923732774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ed0153d70 0x7f2ed014c9b0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2ec800efc0 tx=0x7f2ec800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:05.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.432+0000 7f2ec77fe640 1 -- 172.21.15.125:0/1923732774 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8014480 con 0x7f2ed0153d70 2023-10-30T12:51:05.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.432+0000 7f2ec77fe640 1 -- 172.21.15.125:0/1923732774 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec800f040 con 0x7f2ed0153d70 2023-10-30T12:51:05.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.432+0000 7f2ec77fe640 1 -- 172.21.15.125:0/1923732774 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8014820 con 0x7f2ed0153d70 2023-10-30T12:51:05.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.432+0000 7f2ed5765640 1 -- 172.21.15.125:0/1923732774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed00cbc30 con 0x7f2ed0153d70 2023-10-30T12:51:05.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.432+0000 7f2ed5765640 1 -- 172.21.15.125:0/1923732774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed014c1b0 con 0x7f2ed0153d70 2023-10-30T12:51:05.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.432+0000 7f2ed5765640 1 -- 172.21.15.125:0/1923732774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed004e170 con 0x7f2ed0153d70 2023-10-30T12:51:05.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.432+0000 7f2ec77fe640 1 -- 172.21.15.125:0/1923732774 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ec8014980 con 0x7f2ed0153d70 2023-10-30T12:51:05.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.432+0000 7f2ec77fe640 1 --2- 172.21.15.125:0/1923732774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2eb00664a0 0x7f2eb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.432+0000 7f2ec77fe640 1 -- 172.21.15.125:0/1923732774 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f2ec8085070 con 0x7f2ed0153d70 2023-10-30T12:51:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.432+0000 7f2eceffd640 1 --2- 172.21.15.125:0/1923732774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2eb00664a0 0x7f2eb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.436+0000 7f2eceffd640 1 --2- 172.21.15.125:0/1923732774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2eb00664a0 0x7f2eb0068960 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00ae000 tx=0x7f2ec00b5040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.436+0000 7f2ec77fe640 1 -- 172.21.15.125:0/1923732774 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2ec80b6090 con 0x7f2ed0153d70 2023-10-30T12:51:05.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.620+0000 7f2ed5765640 1 -- 172.21.15.125:0/1923732774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ed0157360 con 0x7f2ed0153d70 2023-10-30T12:51:05.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.620+0000 7f2ec77fe640 1 -- 172.21.15.125:0/1923732774 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f2ec8018020 con 0x7f2ed0153d70 2023-10-30T12:51:05.623 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:05.624 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:57.338574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:05.624 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 111 2023-10-30T12:51:05.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.624+0000 7f2ed5765640 1 -- 172.21.15.125:0/1923732774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2eb00664a0 msgr2=0x7f2eb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:05.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.624+0000 7f2ed5765640 1 --2- 172.21.15.125:0/1923732774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2eb00664a0 0x7f2eb0068960 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f2ec00ae000 tx=0x7f2ec00b5040 comp rx=0 tx=0).stop 2023-10-30T12:51:05.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.624+0000 7f2ed5765640 1 -- 172.21.15.125:0/1923732774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ed0153d70 msgr2=0x7f2ed014c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:05.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.624+0000 7f2ed5765640 1 --2- 172.21.15.125:0/1923732774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ed0153d70 0x7f2ed014c9b0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f2ec800efc0 tx=0x7f2ec800c490 comp rx=0 tx=0).stop 2023-10-30T12:51:05.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.624+0000 7f2ed5765640 1 -- 172.21.15.125:0/1923732774 shutdown_connections 2023-10-30T12:51:05.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.624+0000 7f2ed5765640 1 --2- 172.21.15.125:0/1923732774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2eb00664a0 0x7f2eb0068960 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.624+0000 7f2ed5765640 1 --2- 172.21.15.125:0/1923732774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ed0153d70 0x7f2ed014c9b0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.624+0000 7f2ed5765640 1 --2- 172.21.15.125:0/1923732774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ed00c8490 0x7f2ed00caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.624+0000 7f2ed5765640 1 --2- 172.21.15.125:0/1923732774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ed006e470 0x7f2ed01f6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:05.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.624+0000 7f2ed5765640 1 -- 172.21.15.125:0/1923732774 >> 172.21.15.125:0/1923732774 conn(0x7f2ed0071ea0 msgr2=0x7f2ed006ecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:05.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.624+0000 7f2ed5765640 1 -- 172.21.15.125:0/1923732774 shutdown_connections 2023-10-30T12:51:05.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:05.624+0000 7f2ed5765640 1 -- 172.21.15.125:0/1923732774 wait complete. 2023-10-30T12:51:05.644 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:51:07.646 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:07.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.792+0000 7fd999bd3640 1 Processor -- start 2023-10-30T12:51:07.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.792+0000 7fd999bd3640 1 -- start start 2023-10-30T12:51:07.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd999bd3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd9940bfec0 0x7fd9940c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:07.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd999bd3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd994071290 con 0x7fd994159e60 2023-10-30T12:51:07.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd999bd3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd994071570 con 0x7fd99406a720 2023-10-30T12:51:07.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd999bd3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd994071850 con 0x7fd9940bfec0 2023-10-30T12:51:07.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd9937fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd994159e60 0x7fd99406f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40134/0 (socket says 172.21.15.125:40134) 2023-10-30T12:51:07.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd9937fe640 1 -- 172.21.15.125:0/2862137961 learned_addr learned my addr 172.21.15.125:0/2862137961 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:51:07.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd98bfff640 1 --2- 172.21.15.125:0/2862137961 >> v2:172.21.15.136:3300/0 conn(0x7fd9940bfec0 0x7fd9940c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:07.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 -- 172.21.15.125:0/2862137961 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061592630 0 0) 0x7fd994071290 con 0x7fd994159e60 2023-10-30T12:51:07.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 -- 172.21.15.125:0/2862137961 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd980003530 con 0x7fd994159e60 2023-10-30T12:51:07.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 -- 172.21.15.125:0/2862137961 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 894201178 0 0) 0x7fd980003530 con 0x7fd994159e60 2023-10-30T12:51:07.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 -- 172.21.15.125:0/2862137961 >> v2:172.21.15.136:3300/0 conn(0x7fd9940bfec0 msgr2=0x7fd9940c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:07.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 --2- 172.21.15.125:0/2862137961 >> v2:172.21.15.136:3300/0 conn(0x7fd9940bfec0 0x7fd9940c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:07.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 -- 172.21.15.125:0/2862137961 >> v1:172.21.15.136:6789/0 conn(0x7fd99406a720 legacy=0x7fd994160610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:07.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 -- 172.21.15.125:0/2862137961 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd994071b30 con 0x7fd994159e60 2023-10-30T12:51:07.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 -- 172.21.15.125:0/2862137961 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fd984001680 con 0x7fd994159e60 2023-10-30T12:51:07.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 -- 172.21.15.125:0/2862137961 >> v1:172.21.15.125:6789/0 conn(0x7fd994159e60 legacy=0x7fd99406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:07.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 --2- 172.21.15.125:0/2862137961 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd980003980 0x7fd980003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:07.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 --2- 172.21.15.125:0/2862137961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd980004380 0x7fd980004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:07.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 --2- 172.21.15.125:0/2862137961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd980005050 0x7fd98000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:07.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 -- 172.21.15.125:0/2862137961 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd994071850 con 0x7fd980004380 2023-10-30T12:51:07.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 -- 172.21.15.125:0/2862137961 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98000bf00 con 0x7fd980005050 2023-10-30T12:51:07.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd992ffd640 1 -- 172.21.15.125:0/2862137961 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd98000e2d0 con 0x7fd980003980 2023-10-30T12:51:07.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd98bfff640 1 --2- 172.21.15.125:0/2862137961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd980004380 0x7fd980004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:07.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd993fff640 1 --2- 172.21.15.125:0/2862137961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd980005050 0x7fd98000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:07.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd9937fe640 1 --2- 172.21.15.125:0/2862137961 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd980003980 0x7fd980003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:07.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd98bfff640 1 -- 172.21.15.125:0/2862137961 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd980003980 msgr2=0x7fd980003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:07.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd98bfff640 1 --2- 172.21.15.125:0/2862137961 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd980003980 0x7fd980003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:07.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd98bfff640 1 -- 172.21.15.125:0/2862137961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd980005050 msgr2=0x7fd98000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:07.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd98bfff640 1 --2- 172.21.15.125:0/2862137961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd980005050 0x7fd98000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:07.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd98bfff640 1 -- 172.21.15.125:0/2862137961 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd980003530 con 0x7fd980004380 2023-10-30T12:51:07.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd993fff640 1 -- 172.21.15.125:0/2862137961 reap_dead start 2023-10-30T12:51:07.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.796+0000 7fd9937fe640 1 --2- 172.21.15.125:0/2862137961 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd980003980 0x7fd980003d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:51:07.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.800+0000 7fd98bfff640 1 --2- 172.21.15.125:0/2862137961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd980004380 0x7fd980004850 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00aa60 tx=0x7fd97c01cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b8a669a68adb82 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:07.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.800+0000 7fd992ffd640 1 -- 172.21.15.125:0/2862137961 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c0a5c50 con 0x7fd980004380 2023-10-30T12:51:07.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.800+0000 7fd992ffd640 1 -- 172.21.15.125:0/2862137961 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97c0a5db0 con 0x7fd980004380 2023-10-30T12:51:07.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.800+0000 7fd992ffd640 1 -- 172.21.15.125:0/2862137961 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c0ad680 con 0x7fd980004380 2023-10-30T12:51:07.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.800+0000 7fd999bd3640 1 -- 172.21.15.125:0/2862137961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd980004380 msgr2=0x7fd980004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:07.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.800+0000 7fd999bd3640 1 --2- 172.21.15.125:0/2862137961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd980004380 0x7fd980004850 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fd97c00aa60 tx=0x7fd97c01cc10 comp rx=0 tx=0).stop 2023-10-30T12:51:07.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.800+0000 7fd999bd3640 1 -- 172.21.15.125:0/2862137961 shutdown_connections 2023-10-30T12:51:07.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.800+0000 7fd999bd3640 1 --2- 172.21.15.125:0/2862137961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd980004380 0x7fd980004850 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:07.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.800+0000 7fd999bd3640 1 -- 172.21.15.125:0/2862137961 >> 172.21.15.125:0/2862137961 conn(0x7fd9940c7910 msgr2=0x7fd9940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:07.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.800+0000 7fd999bd3640 1 -- 172.21.15.125:0/2862137961 shutdown_connections 2023-10-30T12:51:07.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.800+0000 7fd999bd3640 1 -- 172.21.15.125:0/2862137961 wait complete. 2023-10-30T12:51:07.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.800+0000 7fd999bd3640 1 Processor -- start 2023-10-30T12:51:07.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.800+0000 7fd999bd3640 1 -- start start 2023-10-30T12:51:07.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd999bd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd980004380 0x7fd9940c3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:07.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd999bd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd99414e110 0x7fd9941ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:07.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd999bd3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9941577b0 0x7fd9941467a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:07.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd999bd3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9941eeec0 con 0x7fd9941577b0 2023-10-30T12:51:07.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd999bd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99414f3e0 con 0x7fd980004380 2023-10-30T12:51:07.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd999bd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99414f5d0 con 0x7fd99414e110 2023-10-30T12:51:07.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd993fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9941577b0 0x7fd9941467a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:07.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd993fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9941577b0 0x7fd9941467a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44042/0 (socket says 172.21.15.125:44042) 2023-10-30T12:51:07.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd993fff640 1 -- 172.21.15.125:0/4077648801 learned_addr learned my addr 172.21.15.125:0/4077648801 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:07.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd98bfff640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd99414e110 0x7fd9941ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:07.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd9937fe640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd980004380 0x7fd9940c3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:07.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd993fff640 1 -- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd99414e110 msgr2=0x7fd9941ee1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:07.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd993fff640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd99414e110 0x7fd9941ee1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:07.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd993fff640 1 -- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd980004380 msgr2=0x7fd9940c3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:07.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd993fff640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd980004380 0x7fd9940c3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:07.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd993fff640 1 -- 172.21.15.125:0/4077648801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9941eebe0 con 0x7fd9941577b0 2023-10-30T12:51:07.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd98bfff640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd99414e110 0x7fd9941ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:07.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd993fff640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9941577b0 0x7fd9941467a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fd99406f1b0 tx=0x7fd98c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:07.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd990ff9640 1 -- 172.21.15.125:0/4077648801 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c003980 con 0x7fd9941577b0 2023-10-30T12:51:07.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd990ff9640 1 -- 172.21.15.125:0/4077648801 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd98c003ae0 con 0x7fd9941577b0 2023-10-30T12:51:07.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd999bd3640 1 -- 172.21.15.125:0/4077648801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd99414ee50 con 0x7fd9941577b0 2023-10-30T12:51:07.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd990ff9640 1 -- 172.21.15.125:0/4077648801 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c003e90 con 0x7fd9941577b0 2023-10-30T12:51:07.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd999bd3640 1 -- 172.21.15.125:0/4077648801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9940c4250 con 0x7fd9941577b0 2023-10-30T12:51:07.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd999bd3640 1 -- 172.21.15.125:0/4077648801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99414ecb0 con 0x7fd9941577b0 2023-10-30T12:51:07.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.804+0000 7fd990ff9640 1 -- 172.21.15.125:0/4077648801 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd98c0050d0 con 0x7fd9941577b0 2023-10-30T12:51:07.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.808+0000 7fd990ff9640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9740664a0 0x7fd974068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:07.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.808+0000 7fd990ff9640 1 -- 172.21.15.125:0/4077648801 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fd98c0841b0 con 0x7fd9941577b0 2023-10-30T12:51:07.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.808+0000 7fd9937fe640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9740664a0 0x7fd974068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:07.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.808+0000 7fd9937fe640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9740664a0 0x7fd974068960 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd984004880 tx=0x7fd984001480 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:07.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.812+0000 7fd990ff9640 1 -- 172.21.15.125:0/4077648801 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd99414ecb0 con 0x7fd9941577b0 2023-10-30T12:51:07.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.992+0000 7fd999bd3640 1 -- 172.21.15.125:0/4077648801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd99414ecb0 con 0x7fd9941577b0 2023-10-30T12:51:07.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.992+0000 7fd990ff9640 1 -- 172.21.15.125:0/4077648801 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fd99414ecb0 con 0x7fd9941577b0 2023-10-30T12:51:07.990 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:07.991 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:57.338574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:07.991 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 111 2023-10-30T12:51:07.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.992+0000 7fd999bd3640 1 -- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9740664a0 msgr2=0x7fd974068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:07.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.992+0000 7fd999bd3640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9740664a0 0x7fd974068960 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fd984004880 tx=0x7fd984001480 comp rx=0 tx=0).stop 2023-10-30T12:51:07.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.992+0000 7fd999bd3640 1 -- 172.21.15.125:0/4077648801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9941577b0 msgr2=0x7fd9941467a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:07.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.992+0000 7fd999bd3640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9941577b0 0x7fd9941467a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fd99406f1b0 tx=0x7fd98c00eb20 comp rx=0 tx=0).stop 2023-10-30T12:51:07.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.992+0000 7fd999bd3640 1 -- 172.21.15.125:0/4077648801 shutdown_connections 2023-10-30T12:51:07.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.992+0000 7fd999bd3640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9740664a0 0x7fd974068960 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:07.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.992+0000 7fd999bd3640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9941577b0 0x7fd9941467a0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:07.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.992+0000 7fd999bd3640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd99414e110 0x7fd9941ee1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:07.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.992+0000 7fd999bd3640 1 --2- 172.21.15.125:0/4077648801 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd980004380 0x7fd9940c3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:07.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.992+0000 7fd999bd3640 1 -- 172.21.15.125:0/4077648801 >> 172.21.15.125:0/4077648801 conn(0x7fd9940c7910 msgr2=0x7fd9940c6a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:07.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.992+0000 7fd999bd3640 1 -- 172.21.15.125:0/4077648801 shutdown_connections 2023-10-30T12:51:07.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:07.992+0000 7fd999bd3640 1 -- 172.21.15.125:0/4077648801 wait complete. 2023-10-30T12:51:08.012 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:51:10.013 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:10.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.160+0000 7f7ad22d6640 1 Processor -- start 2023-10-30T12:51:10.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.160+0000 7f7ad22d6640 1 -- start start 2023-10-30T12:51:10.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7ad22d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7acc06f7a0 0x7f7acc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:10.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7ad22d6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7acc0c6bd0 0x7f7acc069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:10.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7ad22d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7acc06a3d0 con 0x7f7acc071110 2023-10-30T12:51:10.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7ad22d6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7acc0700c0 con 0x7f7acc0c6bd0 2023-10-30T12:51:10.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7ad22d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc066cf0 con 0x7f7acc06f7a0 2023-10-30T12:51:10.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7ad0873640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7acc071110 0x7f7acc158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40146/0 (socket says 172.21.15.125:40146) 2023-10-30T12:51:10.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7ad0873640 1 -- 172.21.15.125:0/3213358863 learned_addr learned my addr 172.21.15.125:0/3213358863 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:51:10.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7acbfff640 1 --2- 172.21.15.125:0/3213358863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7acc06f7a0 0x7f7acc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:10.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7acb7fe640 1 --2- 172.21.15.125:0/3213358863 >> v2:172.21.15.136:3300/0 conn(0x7f7acc0c6bd0 0x7f7acc069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:10.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7acaffd640 1 -- 172.21.15.125:0/3213358863 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 921796005 0 0) 0x7f7acc06a3d0 con 0x7f7acc071110 2023-10-30T12:51:10.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7acaffd640 1 -- 172.21.15.125:0/3213358863 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ab4003530 con 0x7f7acc071110 2023-10-30T12:51:10.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7acbfff640 1 -- 172.21.15.125:0/3213358863 >> v2:172.21.15.136:3300/0 conn(0x7f7acc0c6bd0 msgr2=0x7f7acc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:10.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7acbfff640 1 --2- 172.21.15.125:0/3213358863 >> v2:172.21.15.136:3300/0 conn(0x7f7acc0c6bd0 0x7f7acc069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7acaffd640 1 -- 172.21.15.125:0/3213358863 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3737248613 0 0) 0x7f7ab4003530 con 0x7f7acc071110 2023-10-30T12:51:10.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7acbfff640 1 -- 172.21.15.125:0/3213358863 >> v1:172.21.15.125:6789/0 conn(0x7f7acc071110 legacy=0x7f7acc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:10.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.164+0000 7f7acbfff640 1 -- 172.21.15.125:0/3213358863 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7acc0c70d0 con 0x7f7acc06f7a0 2023-10-30T12:51:10.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.168+0000 7f7acbfff640 1 --2- 172.21.15.125:0/3213358863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7acc06f7a0 0x7f7acc06fb80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0097c0 tx=0x7f7abc01eb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6197d54d680bbb9b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:10.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.168+0000 7f7acaffd640 1 -- 172.21.15.125:0/3213358863 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4001a00 con 0x7f7acc06f7a0 2023-10-30T12:51:10.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.168+0000 7f7acaffd640 1 -- 172.21.15.125:0/3213358863 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7abc0a85d0 con 0x7f7acc06f7a0 2023-10-30T12:51:10.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.168+0000 7f7acaffd640 1 -- 172.21.15.125:0/3213358863 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7abc0a8940 con 0x7f7acc06f7a0 2023-10-30T12:51:10.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.168+0000 7f7ad22d6640 1 -- 172.21.15.125:0/3213358863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7acc06f7a0 msgr2=0x7f7acc06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:10.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.168+0000 7f7ad22d6640 1 --2- 172.21.15.125:0/3213358863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7acc06f7a0 0x7f7acc06fb80 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0097c0 tx=0x7f7abc01eb90 comp rx=0 tx=0).stop 2023-10-30T12:51:10.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.168+0000 7f7ad22d6640 1 -- 172.21.15.125:0/3213358863 shutdown_connections 2023-10-30T12:51:10.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.168+0000 7f7ad22d6640 1 --2- 172.21.15.125:0/3213358863 >> v2:172.21.15.136:3300/0 conn(0x7f7acc0c6bd0 0x7f7acc069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.168+0000 7f7ad22d6640 1 --2- 172.21.15.125:0/3213358863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7acc06f7a0 0x7f7acc06fb80 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.168+0000 7f7ad22d6640 1 -- 172.21.15.125:0/3213358863 >> 172.21.15.125:0/3213358863 conn(0x7f7acc0c7910 msgr2=0x7f7acc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:10.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.168+0000 7f7ad22d6640 1 -- 172.21.15.125:0/3213358863 shutdown_connections 2023-10-30T12:51:10.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.168+0000 7f7ad22d6640 1 -- 172.21.15.125:0/3213358863 wait complete. 2023-10-30T12:51:10.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.168+0000 7f7ad22d6640 1 Processor -- start 2023-10-30T12:51:10.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.168+0000 7f7ad22d6640 1 -- start start 2023-10-30T12:51:10.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad22d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7acc06f7a0 0x7f7acc1f0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:10.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad22d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7acc071110 0x7f7acc0ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:10.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad22d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7acc0c6bd0 0x7f7acc14a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:10.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad22d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc14c130 con 0x7f7acc0c6bd0 2023-10-30T12:51:10.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad22d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc1f0d60 con 0x7f7acc06f7a0 2023-10-30T12:51:10.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad22d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7acc1569b0 con 0x7f7acc071110 2023-10-30T12:51:10.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad0873640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7acc0c6bd0 0x7f7acc14a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:10.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad0873640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7acc0c6bd0 0x7f7acc14a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44058/0 (socket says 172.21.15.125:44058) 2023-10-30T12:51:10.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad0873640 1 -- 172.21.15.125:0/300519271 learned_addr learned my addr 172.21.15.125:0/300519271 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:10.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7acb7fe640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7acc071110 0x7f7acc0ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:10.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7acbfff640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7acc06f7a0 0x7f7acc1f0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:10.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad0873640 1 -- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7acc071110 msgr2=0x7f7acc0ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:10.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad0873640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7acc071110 0x7f7acc0ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad0873640 1 -- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7acc06f7a0 msgr2=0x7f7acc1f0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:10.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad0873640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7acc06f7a0 0x7f7acc1f0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad0873640 1 -- 172.21.15.125:0/300519271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7abc009420 con 0x7f7acc0c6bd0 2023-10-30T12:51:10.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7acbfff640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7acc06f7a0 0x7f7acc1f0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:10.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad0873640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7acc0c6bd0 0x7f7acc14a6b0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400cc30 tx=0x7f7ac400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:10.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ac8ff9640 1 -- 172.21.15.125:0/300519271 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac400ce40 con 0x7f7acc0c6bd0 2023-10-30T12:51:10.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ac8ff9640 1 -- 172.21.15.125:0/300519271 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ac4012400 con 0x7f7acc0c6bd0 2023-10-30T12:51:10.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad22d6640 1 -- 172.21.15.125:0/300519271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7acc1efdd0 con 0x7f7acc0c6bd0 2023-10-30T12:51:10.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ac8ff9640 1 -- 172.21.15.125:0/300519271 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac40127b0 con 0x7f7acc0c6bd0 2023-10-30T12:51:10.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad22d6640 1 -- 172.21.15.125:0/300519271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7acc1f0390 con 0x7f7acc0c6bd0 2023-10-30T12:51:10.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ad22d6640 1 -- 172.21.15.125:0/300519271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7acc1f0520 con 0x7f7acc0c6bd0 2023-10-30T12:51:10.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.172+0000 7f7ac8ff9640 1 -- 172.21.15.125:0/300519271 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7ac4012910 con 0x7f7acc0c6bd0 2023-10-30T12:51:10.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.176+0000 7f7ac8ff9640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ab80664d0 0x7f7ab8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:10.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.176+0000 7f7ac8ff9640 1 -- 172.21.15.125:0/300519271 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f7ac4084710 con 0x7f7acc0c6bd0 2023-10-30T12:51:10.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.176+0000 7f7acbfff640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ab80664d0 0x7f7ab8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:10.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.176+0000 7f7acbfff640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ab80664d0 0x7f7ab8068990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0b2000 tx=0x7f7abc0aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:10.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.180+0000 7f7ac8ff9640 1 -- 172.21.15.125:0/300519271 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7ac40518a0 con 0x7f7acc0c6bd0 2023-10-30T12:51:10.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.372+0000 7f7ad22d6640 1 -- 172.21.15.125:0/300519271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7acc04e170 con 0x7f7acc0c6bd0 2023-10-30T12:51:10.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.372+0000 7f7ac8ff9640 1 -- 172.21.15.125:0/300519271 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f7ac4054ec0 con 0x7f7acc0c6bd0 2023-10-30T12:51:10.371 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:10.371 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:57.338574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:10.372 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 111 2023-10-30T12:51:10.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.376+0000 7f7ad22d6640 1 -- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ab80664d0 msgr2=0x7f7ab8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:10.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.376+0000 7f7ad22d6640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ab80664d0 0x7f7ab8068990 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7abc0b2000 tx=0x7f7abc0aa070 comp rx=0 tx=0).stop 2023-10-30T12:51:10.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.376+0000 7f7ad22d6640 1 -- 172.21.15.125:0/300519271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7acc0c6bd0 msgr2=0x7f7acc14a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:10.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.376+0000 7f7ad22d6640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7acc0c6bd0 0x7f7acc14a6b0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400cc30 tx=0x7f7ac400a430 comp rx=0 tx=0).stop 2023-10-30T12:51:10.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.376+0000 7f7ad22d6640 1 -- 172.21.15.125:0/300519271 shutdown_connections 2023-10-30T12:51:10.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.376+0000 7f7ad22d6640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ab80664d0 0x7f7ab8068990 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.376+0000 7f7ad22d6640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7acc0c6bd0 0x7f7acc14a6b0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.376+0000 7f7ad22d6640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7acc071110 0x7f7acc0ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.376+0000 7f7ad22d6640 1 --2- 172.21.15.125:0/300519271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7acc06f7a0 0x7f7acc1f0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:10.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.376+0000 7f7ad22d6640 1 -- 172.21.15.125:0/300519271 >> 172.21.15.125:0/300519271 conn(0x7f7acc0c7910 msgr2=0x7f7acc0c3430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:10.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.376+0000 7f7ad22d6640 1 -- 172.21.15.125:0/300519271 shutdown_connections 2023-10-30T12:51:10.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:10.376+0000 7f7ad22d6640 1 -- 172.21.15.125:0/300519271 wait complete. 2023-10-30T12:51:10.392 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:51:12.394 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:12.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a717aa640 1 Processor -- start 2023-10-30T12:51:12.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a717aa640 1 -- start start 2023-10-30T12:51:12.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a717aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0a6c153c30 0x7f0a6c06a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:12.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a717aa640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6c06ade0 con 0x7f0a6c0b1ea0 2023-10-30T12:51:12.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a717aa640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a6c1598f0 con 0x7f0a6c0cc0b0 2023-10-30T12:51:12.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a717aa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c1596e0 con 0x7f0a6c153c30 2023-10-30T12:51:12.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a627fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0a6c0b1ea0 0x7f0a6c0cb9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44996/0 (socket says 172.21.15.125:44996) 2023-10-30T12:51:12.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a627fc640 1 -- 172.21.15.125:0/3360571301 learned_addr learned my addr 172.21.15.125:0/3360571301 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:51:12.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6affd640 1 --2- 172.21.15.125:0/3360571301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0a6c153c30 0x7f0a6c06a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:12.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 -- 172.21.15.125:0/3360571301 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010602683 0 0) 0x7f0a5c0012a0 con 0x7f0a6c0b1ea0 2023-10-30T12:51:12.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 -- 172.21.15.125:0/3360571301 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a58003530 con 0x7f0a6c0b1ea0 2023-10-30T12:51:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 -- 172.21.15.125:0/3360571301 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4187440056 0 0) 0x7f0a58003530 con 0x7f0a6c0b1ea0 2023-10-30T12:51:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 -- 172.21.15.125:0/3360571301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0a6c153c30 msgr2=0x7f0a6c06a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:12.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 --2- 172.21.15.125:0/3360571301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0a6c153c30 0x7f0a6c06a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 -- 172.21.15.125:0/3360571301 >> v1:172.21.15.136:6789/0 conn(0x7f0a6c0cc0b0 legacy=0x7f0a6c0cabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 -- 172.21.15.125:0/3360571301 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c159b90 con 0x7f0a6c0b1ea0 2023-10-30T12:51:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 -- 172.21.15.125:0/3360571301 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f0a5c001990 con 0x7f0a6c0b1ea0 2023-10-30T12:51:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 -- 172.21.15.125:0/3360571301 >> v1:172.21.15.125:6789/0 conn(0x7f0a6c0b1ea0 legacy=0x7f0a6c0cb9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 --2- 172.21.15.125:0/3360571301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0a6c153c30 0x7f0a580039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:12.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 --2- 172.21.15.125:0/3360571301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a58003fe0 0x7f0a580047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:12.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 --2- 172.21.15.125:0/3360571301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a58004d10 0x7f0a58005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:12.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 -- 172.21.15.125:0/3360571301 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c1596e0 con 0x7f0a58004d10 2023-10-30T12:51:12.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 -- 172.21.15.125:0/3360571301 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5800bbe0 con 0x7f0a58003fe0 2023-10-30T12:51:12.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6a7fc640 1 -- 172.21.15.125:0/3360571301 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5800bda0 con 0x7f0a6c153c30 2023-10-30T12:51:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6b7fe640 1 --2- 172.21.15.125:0/3360571301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a58004d10 0x7f0a58005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a627fc640 1 --2- 172.21.15.125:0/3360571301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a58003fe0 0x7f0a580047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6b7fe640 1 -- 172.21.15.125:0/3360571301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0a6c153c30 msgr2=0x7f0a580039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.544+0000 7f0a6affd640 1 --2- 172.21.15.125:0/3360571301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0a6c153c30 0x7f0a580039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:12.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a6b7fe640 1 --2- 172.21.15.125:0/3360571301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0a6c153c30 0x7f0a580039b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:12.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a6b7fe640 1 -- 172.21.15.125:0/3360571301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a58003fe0 msgr2=0x7f0a580047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:12.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a6b7fe640 1 --2- 172.21.15.125:0/3360571301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a58003fe0 0x7f0a580047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:12.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a6b7fe640 1 -- 172.21.15.125:0/3360571301 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a58003530 con 0x7f0a58004d10 2023-10-30T12:51:12.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a6b7fe640 1 --2- 172.21.15.125:0/3360571301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a58004d10 0x7f0a58005190 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0a64001e40 tx=0x7f0a6401c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=142a28c2670ba60f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:12.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a6a7fc640 1 -- 172.21.15.125:0/3360571301 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a64003b80 con 0x7f0a58004d10 2023-10-30T12:51:12.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a6a7fc640 1 -- 172.21.15.125:0/3360571301 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a64003ce0 con 0x7f0a58004d10 2023-10-30T12:51:12.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a6a7fc640 1 -- 172.21.15.125:0/3360571301 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a640052c0 con 0x7f0a58004d10 2023-10-30T12:51:12.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a717aa640 1 -- 172.21.15.125:0/3360571301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a58004d10 msgr2=0x7f0a58005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:12.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a717aa640 1 --2- 172.21.15.125:0/3360571301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a58004d10 0x7f0a58005190 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0a64001e40 tx=0x7f0a6401c740 comp rx=0 tx=0).stop 2023-10-30T12:51:12.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a6b7fe640 1 -- 172.21.15.125:0/3360571301 reap_dead start 2023-10-30T12:51:12.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a717aa640 1 -- 172.21.15.125:0/3360571301 shutdown_connections 2023-10-30T12:51:12.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a717aa640 1 -- 172.21.15.125:0/3360571301 >> 172.21.15.125:0/3360571301 conn(0x7f0a6c06bf40 msgr2=0x7f0a6c06c350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:12.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a717aa640 1 -- 172.21.15.125:0/3360571301 shutdown_connections 2023-10-30T12:51:12.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.548+0000 7f0a717aa640 1 -- 172.21.15.125:0/3360571301 wait complete. 2023-10-30T12:51:12.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a717aa640 1 Processor -- start 2023-10-30T12:51:12.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a717aa640 1 -- start start 2023-10-30T12:51:12.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a717aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0a58004d10 0x7f0a6c0c70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:12.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a717aa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a6c154e50 0x7f0a6c1f24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:12.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a717aa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a6c1f2a10 0x7f0a6c1463b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:12.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a717aa640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c156350 con 0x7f0a6c154e50 2023-10-30T12:51:12.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a717aa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0c8e00 con 0x7f0a6c1f2a10 2023-10-30T12:51:12.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a717aa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c1f3010 con 0x7f0a58004d10 2023-10-30T12:51:12.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a627fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a6c154e50 0x7f0a6c1f24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:12.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a627fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a6c154e50 0x7f0a6c1f24d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58122/0 (socket says 172.21.15.125:58122) 2023-10-30T12:51:12.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a627fc640 1 -- 172.21.15.125:0/49117481 learned_addr learned my addr 172.21.15.125:0/49117481 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:12.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a6affd640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0a58004d10 0x7f0a6c0c70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:12.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a6b7fe640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a6c1f2a10 0x7f0a6c1463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:12.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a627fc640 1 -- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0a58004d10 msgr2=0x7f0a6c0c70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:12.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a627fc640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0a58004d10 0x7f0a6c0c70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:12.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a627fc640 1 -- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a6c1f2a10 msgr2=0x7f0a6c1463b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a627fc640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a6c1f2a10 0x7f0a6c1463b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a627fc640 1 -- 172.21.15.125:0/49117481 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c0c7600 con 0x7f0a6c154e50 2023-10-30T12:51:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a6affd640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0a58004d10 0x7f0a6c0c70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a6b7fe640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a6c1f2a10 0x7f0a6c1463b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a627fc640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a6c154e50 0x7f0a6c1f24d0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c001ac0 tx=0x7f0a5c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a63fff640 1 -- 172.21.15.125:0/49117481 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c00f7b0 con 0x7f0a6c154e50 2023-10-30T12:51:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a63fff640 1 -- 172.21.15.125:0/49117481 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5c00f910 con 0x7f0a6c154e50 2023-10-30T12:51:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a63fff640 1 -- 172.21.15.125:0/49117481 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c00fcc0 con 0x7f0a6c154e50 2023-10-30T12:51:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a717aa640 1 -- 172.21.15.125:0/49117481 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6c0ccf20 con 0x7f0a6c154e50 2023-10-30T12:51:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.552+0000 7f0a717aa640 1 -- 172.21.15.125:0/49117481 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6c0c5b20 con 0x7f0a6c154e50 2023-10-30T12:51:12.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.556+0000 7f0a717aa640 1 -- 172.21.15.125:0/49117481 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6c04e170 con 0x7f0a6c154e50 2023-10-30T12:51:12.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.556+0000 7f0a63fff640 1 -- 172.21.15.125:0/49117481 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0a5c01c400 con 0x7f0a6c154e50 2023-10-30T12:51:12.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.556+0000 7f0a63fff640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a4c0664a0 0x7f0a4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:12.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.556+0000 7f0a63fff640 1 -- 172.21.15.125:0/49117481 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f0a5c084760 con 0x7f0a6c154e50 2023-10-30T12:51:12.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.556+0000 7f0a6affd640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a4c0664a0 0x7f0a4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:12.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.560+0000 7f0a6affd640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a4c0664a0 0x7f0a4c068960 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400c8c0 tx=0x7f0a54008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:12.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.560+0000 7f0a63fff640 1 -- 172.21.15.125:0/49117481 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0a5c055300 con 0x7f0a6c154e50 2023-10-30T12:51:12.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.740+0000 7f0a717aa640 1 -- 172.21.15.125:0/49117481 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a6c06ade0 con 0x7f0a6c154e50 2023-10-30T12:51:12.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.740+0000 7f0a63fff640 1 -- 172.21.15.125:0/49117481 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f0a6c06ade0 con 0x7f0a6c154e50 2023-10-30T12:51:12.742 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:12.743 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:57.338574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:12.744 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 111 2023-10-30T12:51:12.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.744+0000 7f0a717aa640 1 -- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a4c0664a0 msgr2=0x7f0a4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:12.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.744+0000 7f0a717aa640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a4c0664a0 0x7f0a4c068960 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0a5400c8c0 tx=0x7f0a54008040 comp rx=0 tx=0).stop 2023-10-30T12:51:12.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.744+0000 7f0a717aa640 1 -- 172.21.15.125:0/49117481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a6c154e50 msgr2=0x7f0a6c1f24d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:12.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.744+0000 7f0a717aa640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a6c154e50 0x7f0a6c1f24d0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c001ac0 tx=0x7f0a5c007f90 comp rx=0 tx=0).stop 2023-10-30T12:51:12.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.744+0000 7f0a717aa640 1 -- 172.21.15.125:0/49117481 shutdown_connections 2023-10-30T12:51:12.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.744+0000 7f0a717aa640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a4c0664a0 0x7f0a4c068960 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:12.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.744+0000 7f0a717aa640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0a6c1f2a10 0x7f0a6c1463b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:12.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.744+0000 7f0a717aa640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0a6c154e50 0x7f0a6c1f24d0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:12.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.744+0000 7f0a717aa640 1 --2- 172.21.15.125:0/49117481 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0a58004d10 0x7f0a6c0c70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:12.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.744+0000 7f0a717aa640 1 -- 172.21.15.125:0/49117481 >> 172.21.15.125:0/49117481 conn(0x7f0a6c06bf40 msgr2=0x7f0a6c0c4880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:12.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.744+0000 7f0a717aa640 1 -- 172.21.15.125:0/49117481 shutdown_connections 2023-10-30T12:51:12.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:12.744+0000 7f0a717aa640 1 -- 172.21.15.125:0/49117481 wait complete. 2023-10-30T12:51:12.762 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:51:14.763 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:14.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.911+0000 7fe2744ea640 1 Processor -- start 2023-10-30T12:51:14.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.911+0000 7fe2744ea640 1 -- start start 2023-10-30T12:51:14.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.911+0000 7fe2744ea640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe26c15d6f0 0x7fe26c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:14.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.911+0000 7fe2744ea640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c0cb800 con 0x7fe26c0cf160 2023-10-30T12:51:14.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.911+0000 7fe2744ea640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe26c0cb330 con 0x7fe26c1538a0 2023-10-30T12:51:14.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.911+0000 7fe2744ea640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe26c0cad70 con 0x7fe26c15d6f0 2023-10-30T12:51:14.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.911+0000 7fe271a85640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe26c15d6f0 0x7fe26c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:14.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.911+0000 7fe271a85640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe26c15d6f0 0x7fe26c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58136/0 (socket says 172.21.15.125:58136) 2023-10-30T12:51:14.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.911+0000 7fe271a85640 1 -- 172.21.15.125:0/4059373483 learned_addr learned my addr 172.21.15.125:0/4059373483 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:14.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.911+0000 7fe272a87640 1 --1- 172.21.15.125:0/4059373483 >> v1:172.21.15.125:6789/0 conn(0x7fe26c0cf160 0x7fe26c164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:45000/0 (socket says 172.21.15.125:45000) 2023-10-30T12:51:14.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.911+0000 7fe271a85640 1 -- 172.21.15.125:0/4059373483 >> v1:172.21.15.136:6789/0 conn(0x7fe26c1538a0 legacy=0x7fe26c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:14.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.911+0000 7fe271a85640 1 -- 172.21.15.125:0/4059373483 >> v1:172.21.15.125:6789/0 conn(0x7fe26c0cf160 legacy=0x7fe26c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:14.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.911+0000 7fe271a85640 1 -- 172.21.15.125:0/4059373483 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26c0cb050 con 0x7fe26c15d6f0 2023-10-30T12:51:14.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.915+0000 7fe271a85640 1 --2- 172.21.15.125:0/4059373483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c15d6f0 0x7fe26c15a090 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009870 tx=0x7fe25c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a1acb9ca1f14359 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:14.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.915+0000 7fe271284640 1 -- 172.21.15.125:0/4059373483 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c0a89f0 con 0x7fe26c15d6f0 2023-10-30T12:51:14.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.915+0000 7fe271284640 1 -- 172.21.15.125:0/4059373483 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe25c0a8b50 con 0x7fe26c15d6f0 2023-10-30T12:51:14.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.915+0000 7fe271284640 1 -- 172.21.15.125:0/4059373483 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe25c0b0440 con 0x7fe26c15d6f0 2023-10-30T12:51:14.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.915+0000 7fe2744ea640 1 -- 172.21.15.125:0/4059373483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c15d6f0 msgr2=0x7fe26c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:14.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.915+0000 7fe2744ea640 1 --2- 172.21.15.125:0/4059373483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c15d6f0 0x7fe26c15a090 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fe25c009870 tx=0x7fe25c01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:51:14.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.915+0000 7fe2744ea640 1 -- 172.21.15.125:0/4059373483 shutdown_connections 2023-10-30T12:51:14.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.915+0000 7fe2744ea640 1 --2- 172.21.15.125:0/4059373483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c15d6f0 0x7fe26c15a090 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:14.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.915+0000 7fe2744ea640 1 -- 172.21.15.125:0/4059373483 >> 172.21.15.125:0/4059373483 conn(0x7fe26c06bfe0 msgr2=0x7fe26c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:14.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.915+0000 7fe2744ea640 1 -- 172.21.15.125:0/4059373483 shutdown_connections 2023-10-30T12:51:14.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.915+0000 7fe2744ea640 1 -- 172.21.15.125:0/4059373483 wait complete. 2023-10-30T12:51:14.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe2744ea640 1 Processor -- start 2023-10-30T12:51:14.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe2744ea640 1 -- start start 2023-10-30T12:51:14.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe2744ea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe26c0cf160 0x7fe26c159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:14.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe2744ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe26c1538a0 0x7fe26c15bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:14.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe2744ea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c15d6f0 0x7fe26c15c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:14.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe2744ea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c1e4030 con 0x7fe26c15d6f0 2023-10-30T12:51:14.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe2744ea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c1e56b0 con 0x7fe26c0cf160 2023-10-30T12:51:14.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe2744ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe26c151200 con 0x7fe26c1538a0 2023-10-30T12:51:14.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe272a87640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c15d6f0 0x7fe26c15c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:14.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe272a87640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c15d6f0 0x7fe26c15c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58152/0 (socket says 172.21.15.125:58152) 2023-10-30T12:51:14.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe272a87640 1 -- 172.21.15.125:0/3145903291 learned_addr learned my addr 172.21.15.125:0/3145903291 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:14.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe271a85640 1 --2- 172.21.15.125:0/3145903291 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe26c1538a0 0x7fe26c15bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:14.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe272286640 1 --2- 172.21.15.125:0/3145903291 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe26c0cf160 0x7fe26c159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:14.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe272a87640 1 -- 172.21.15.125:0/3145903291 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe26c1538a0 msgr2=0x7fe26c15bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:14.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe272a87640 1 --2- 172.21.15.125:0/3145903291 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe26c1538a0 0x7fe26c15bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:14.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe272a87640 1 -- 172.21.15.125:0/3145903291 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe26c0cf160 msgr2=0x7fe26c159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:14.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe272a87640 1 --2- 172.21.15.125:0/3145903291 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe26c0cf160 0x7fe26c159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:14.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe272a87640 1 -- 172.21.15.125:0/3145903291 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c0094d0 con 0x7fe26c15d6f0 2023-10-30T12:51:14.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe272a87640 1 --2- 172.21.15.125:0/3145903291 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c15d6f0 0x7fe26c15c180 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fe26400c730 tx=0x7fe26400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:14.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe262ffd640 1 -- 172.21.15.125:0/3145903291 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe264004d70 con 0x7fe26c15d6f0 2023-10-30T12:51:14.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe262ffd640 1 -- 172.21.15.125:0/3145903291 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe26400a640 con 0x7fe26c15d6f0 2023-10-30T12:51:14.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe2744ea640 1 -- 172.21.15.125:0/3145903291 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26c0c4c30 con 0x7fe26c15d6f0 2023-10-30T12:51:14.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe262ffd640 1 -- 172.21.15.125:0/3145903291 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe26400a9d0 con 0x7fe26c15d6f0 2023-10-30T12:51:14.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.919+0000 7fe2744ea640 1 -- 172.21.15.125:0/3145903291 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26c14abf0 con 0x7fe26c15d6f0 2023-10-30T12:51:14.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.923+0000 7fe2744ea640 1 -- 172.21.15.125:0/3145903291 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe26c1e57f0 con 0x7fe26c15d6f0 2023-10-30T12:51:14.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.923+0000 7fe262ffd640 1 -- 172.21.15.125:0/3145903291 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe26400ad40 con 0x7fe26c15d6f0 2023-10-30T12:51:14.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.923+0000 7fe262ffd640 1 --2- 172.21.15.125:0/3145903291 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2500664d0 0x7fe250068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:14.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.923+0000 7fe262ffd640 1 -- 172.21.15.125:0/3145903291 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6598+0+0 (secure 0 0 0) 0x7fe264083af0 con 0x7fe26c15d6f0 2023-10-30T12:51:14.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.923+0000 7fe272286640 1 --2- 172.21.15.125:0/3145903291 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2500664d0 0x7fe250068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:14.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.923+0000 7fe272286640 1 --2- 172.21.15.125:0/3145903291 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2500664d0 0x7fe250068990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe268004e30 tx=0x7fe268009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:14.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:14.927+0000 7fe262ffd640 1 -- 172.21.15.125:0/3145903291 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe26c1e57f0 con 0x7fe26c15d6f0 2023-10-30T12:51:15.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:15.107+0000 7fe2744ea640 1 -- 172.21.15.125:0/3145903291 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe26c1e57f0 con 0x7fe26c15d6f0 2023-10-30T12:51:15.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:15.107+0000 7fe262ffd640 1 -- 172.21.15.125:0/3145903291 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fe26c1e57f0 con 0x7fe26c15d6f0 2023-10-30T12:51:15.107 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:15.107 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6627,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":105}],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:49:57.338574+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6318},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6318":{"gid":6318,"name":"e","rank":0,"incarnation":92,"state":"up:active","state_seq":18,"addr":"172.21.15.125:6836/1721514617","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1721514617},{"type":"v1","addr":"172.21.15.125:6836","nonce":1721514617}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:15.107 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 111 2023-10-30T12:51:15.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:15.107+0000 7fe2744ea640 1 -- 172.21.15.125:0/3145903291 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2500664d0 msgr2=0x7fe250068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:15.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:15.107+0000 7fe2744ea640 1 --2- 172.21.15.125:0/3145903291 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2500664d0 0x7fe250068990 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fe268004e30 tx=0x7fe268009040 comp rx=0 tx=0).stop 2023-10-30T12:51:15.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:15.107+0000 7fe2744ea640 1 -- 172.21.15.125:0/3145903291 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c15d6f0 msgr2=0x7fe26c15c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:15.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:15.107+0000 7fe2744ea640 1 --2- 172.21.15.125:0/3145903291 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c15d6f0 0x7fe26c15c180 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fe26400c730 tx=0x7fe26400cbf0 comp rx=0 tx=0).stop 2023-10-30T12:51:15.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:15.107+0000 7fe2744ea640 1 -- 172.21.15.125:0/3145903291 shutdown_connections 2023-10-30T12:51:15.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:15.107+0000 7fe2744ea640 1 --2- 172.21.15.125:0/3145903291 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2500664d0 0x7fe250068990 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:15.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:15.107+0000 7fe2744ea640 1 --2- 172.21.15.125:0/3145903291 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe26c15d6f0 0x7fe26c15c180 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:15.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:15.107+0000 7fe2744ea640 1 --2- 172.21.15.125:0/3145903291 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe26c1538a0 0x7fe26c15bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:15.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:15.107+0000 7fe2744ea640 1 --2- 172.21.15.125:0/3145903291 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe26c0cf160 0x7fe26c159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:15.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:15.107+0000 7fe2744ea640 1 -- 172.21.15.125:0/3145903291 >> 172.21.15.125:0/3145903291 conn(0x7fe26c06bfe0 msgr2=0x7fe26c0ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:15.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:15.107+0000 7fe2744ea640 1 -- 172.21.15.125:0/3145903291 shutdown_connections 2023-10-30T12:51:15.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:15.107+0000 7fe2744ea640 1 -- 172.21.15.125:0/3145903291 wait complete. 2023-10-30T12:51:15.128 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:51:17.129 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:17.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.275+0000 7f5feb13e640 1 Processor -- start 2023-10-30T12:51:17.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.275+0000 7f5feb13e640 1 -- start start 2023-10-30T12:51:17.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5feb13e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fe4153bf0 0x7f5fe4071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:17.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5feb13e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5fe40c6fe0 0x7f5fe40c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:17.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5feb13e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe40700b0 con 0x7f5fe40c79e0 2023-10-30T12:51:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5feb13e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe40c6810 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5feb13e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe4159900 con 0x7f5fe4153bf0 2023-10-30T12:51:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5fdbfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5fe40c6fe0 0x7f5fe40c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5fdbfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5fe40c6fe0 0x7f5fe40c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58168/0 (socket says 172.21.15.125:58168) 2023-10-30T12:51:17.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5fdbfff640 1 -- 172.21.15.125:0/2148812124 learned_addr learned my addr 172.21.15.125:0/2148812124 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:17.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5fe8eda640 1 --2- 172.21.15.125:0/2148812124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fe4153bf0 0x7f5fe4071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:17.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5fdbfff640 1 -- 172.21.15.125:0/2148812124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fe4153bf0 msgr2=0x7f5fe4071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:17.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5fdbfff640 1 --2- 172.21.15.125:0/2148812124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fe4153bf0 0x7f5fe4071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5fdbfff640 1 -- 172.21.15.125:0/2148812124 >> v1:172.21.15.136:6789/0 conn(0x7f5fe40c79e0 legacy=0x7f5fe40cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:17.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5fdbfff640 1 -- 172.21.15.125:0/2148812124 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe40c6530 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5fe8eda640 1 --2- 172.21.15.125:0/2148812124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fe4153bf0 0x7f5fe4071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:17.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.279+0000 7f5fdbfff640 1 --2- 172.21.15.125:0/2148812124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe40c6fe0 0x7f5fe40c74a0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009870 tx=0x7f5fd401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8275d029ffcda14 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:17.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5fdb7fe640 1 -- 172.21.15.125:0/2148812124 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd40a8a30 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5fdb7fe640 1 -- 172.21.15.125:0/2148812124 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fd40a8b90 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5fdb7fe640 1 -- 172.21.15.125:0/2148812124 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd40b0440 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 -- 172.21.15.125:0/2148812124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe40c6fe0 msgr2=0x7f5fe40c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:17.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 --2- 172.21.15.125:0/2148812124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe40c6fe0 0x7f5fe40c74a0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4009870 tx=0x7f5fd401ed30 comp rx=0 tx=0).stop 2023-10-30T12:51:17.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 -- 172.21.15.125:0/2148812124 shutdown_connections 2023-10-30T12:51:17.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 --2- 172.21.15.125:0/2148812124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe40c6fe0 0x7f5fe40c74a0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 --2- 172.21.15.125:0/2148812124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fe4153bf0 0x7f5fe4071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 -- 172.21.15.125:0/2148812124 >> 172.21.15.125:0/2148812124 conn(0x7f5fe406c6e0 msgr2=0x7f5fe4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:17.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 -- 172.21.15.125:0/2148812124 shutdown_connections 2023-10-30T12:51:17.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 -- 172.21.15.125:0/2148812124 wait complete. 2023-10-30T12:51:17.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 Processor -- start 2023-10-30T12:51:17.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 -- start start 2023-10-30T12:51:17.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe40c6fe0 0x7f5fe40cac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:17.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe40c79e0 0x7f5fe41eed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:17.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fe4153bf0 0x7f5fe41ef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:17.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe4145860 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe4158dd0 con 0x7f5fe40c79e0 2023-10-30T12:51:17.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5feb13e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe414a4b0 con 0x7f5fe4153bf0 2023-10-30T12:51:17.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5fe8eda640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe40c6fe0 0x7f5fe40cac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:17.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5fe8eda640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe40c6fe0 0x7f5fe40cac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58180/0 (socket says 172.21.15.125:58180) 2023-10-30T12:51:17.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.283+0000 7f5fe8eda640 1 -- 172.21.15.125:0/3775237080 learned_addr learned my addr 172.21.15.125:0/3775237080 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:17.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5fe96db640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fe4153bf0 0x7f5fe41ef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:17.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5fdbfff640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe40c79e0 0x7f5fe41eed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:17.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5fe8eda640 1 -- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fe4153bf0 msgr2=0x7f5fe41ef2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:17.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5fe8eda640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fe4153bf0 0x7f5fe41ef2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5fe8eda640 1 -- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe40c79e0 msgr2=0x7f5fe41eed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:17.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5fe8eda640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe40c79e0 0x7f5fe41eed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5fe8eda640 1 -- 172.21.15.125:0/3775237080 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd40094d0 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5fdbfff640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe40c79e0 0x7f5fe41eed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:17.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5fe8eda640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe40c6fe0 0x7f5fe40cac50 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f5fe000d800 tx=0x7f5fe000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:17.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5fd97fa640 1 -- 172.21.15.125:0/3775237080 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe00096c0 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5fd97fa640 1 -- 172.21.15.125:0/3775237080 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe0009820 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5fd97fa640 1 -- 172.21.15.125:0/3775237080 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0009bd0 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5feb13e640 1 -- 172.21.15.125:0/3775237080 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe4149dd0 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5feb13e640 1 -- 172.21.15.125:0/3775237080 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe414a770 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5feb13e640 1 -- 172.21.15.125:0/3775237080 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe404e170 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5fd97fa640 1 -- 172.21.15.125:0/3775237080 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5fe0009d30 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.287+0000 7f5fd97fa640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd00664d0 0x7f5fd0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:17.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.291+0000 7f5fd97fa640 1 -- 172.21.15.125:0/3775237080 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7f5fe0084820 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.291+0000 7f5fdbfff640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd00664d0 0x7f5fd0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:17.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.291+0000 7f5fdbfff640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd00664d0 0x7f5fd0068990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5fd40b2000 tx=0x7f5fd4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:17.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.291+0000 7f5fd97fa640 1 -- 172.21.15.125:0/3775237080 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5fe0051960 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.475+0000 7f5feb13e640 1 -- 172.21.15.125:0/3775237080 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fe415a0a0 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.475+0000 7f5fd97fa640 1 -- 172.21.15.125:0/3775237080 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f5fe0054f80 con 0x7f5fe40c6fe0 2023-10-30T12:51:17.475 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:17.475 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:15.314380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:17.476 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 112 2023-10-30T12:51:17.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.475+0000 7f5feb13e640 1 -- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd00664d0 msgr2=0x7f5fd0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:17.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.475+0000 7f5feb13e640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd00664d0 0x7f5fd0068990 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f5fd40b2000 tx=0x7f5fd4008040 comp rx=0 tx=0).stop 2023-10-30T12:51:17.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.475+0000 7f5feb13e640 1 -- 172.21.15.125:0/3775237080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe40c6fe0 msgr2=0x7f5fe40cac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:17.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.475+0000 7f5feb13e640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe40c6fe0 0x7f5fe40cac50 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f5fe000d800 tx=0x7f5fe000dcc0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.479+0000 7f5feb13e640 1 -- 172.21.15.125:0/3775237080 shutdown_connections 2023-10-30T12:51:17.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.479+0000 7f5feb13e640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd00664d0 0x7f5fd0068990 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.479+0000 7f5feb13e640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fe4153bf0 0x7f5fe41ef2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.479+0000 7f5feb13e640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fe40c79e0 0x7f5fe41eed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.479+0000 7f5feb13e640 1 --2- 172.21.15.125:0/3775237080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fe40c6fe0 0x7f5fe40cac50 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.479+0000 7f5feb13e640 1 -- 172.21.15.125:0/3775237080 >> 172.21.15.125:0/3775237080 conn(0x7f5fe406c6e0 msgr2=0x7f5fe4159020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:17.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.479+0000 7f5feb13e640 1 -- 172.21.15.125:0/3775237080 shutdown_connections 2023-10-30T12:51:17.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.479+0000 7f5feb13e640 1 -- 172.21.15.125:0/3775237080 wait complete. 2023-10-30T12:51:17.496 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T12:51:17.496 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:51:17.496 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:17.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.635+0000 7f754a10f640 1 Processor -- start 2023-10-30T12:51:17.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.635+0000 7f754a10f640 1 -- start start 2023-10-30T12:51:17.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.635+0000 7f754a10f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75440c6b80 0x7f75440bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:17.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.635+0000 7f754a10f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f75440c6f60 0x7f75440c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:17.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.635+0000 7f754a10f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75440c0a50 con 0x7f75440c8430 2023-10-30T12:51:17.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.635+0000 7f754a10f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f754406f6e0 con 0x7f75440c6f60 2023-10-30T12:51:17.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.635+0000 7f754a10f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754406f9c0 con 0x7f75440c6b80 2023-10-30T12:51:17.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.635+0000 7f7542ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f75440c6f60 0x7f75440c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:17.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.635+0000 7f7543fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f75440c8430 0x7f754413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:56672/0 (socket says 172.21.15.125:56672) 2023-10-30T12:51:17.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.635+0000 7f7543fff640 1 -- 172.21.15.125:0/3516947740 learned_addr learned my addr 172.21.15.125:0/3516947740 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:51:17.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.635+0000 7f75437fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75440c6b80 0x7f75440bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:17.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.635+0000 7f7542ffd640 1 --2- 172.21.15.125:0/3516947740 >> v2:172.21.15.136:3300/0 conn(0x7f75440c6f60 0x7f75440c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:36312/0 (socket says 172.21.15.125:36312) 2023-10-30T12:51:17.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.639+0000 7f75427fc640 1 -- 172.21.15.125:0/3516947740 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771442500 0 0) 0x7f75440c0a50 con 0x7f75440c8430 2023-10-30T12:51:17.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.639+0000 7f75427fc640 1 -- 172.21.15.125:0/3516947740 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c003530 con 0x7f75440c8430 2023-10-30T12:51:17.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.639+0000 7f75437fe640 1 -- 172.21.15.125:0/3516947740 >> v2:172.21.15.136:3300/0 conn(0x7f75440c6f60 msgr2=0x7f75440c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:17.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.639+0000 7f75437fe640 1 --2- 172.21.15.125:0/3516947740 >> v2:172.21.15.136:3300/0 conn(0x7f75440c6f60 0x7f75440c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.639+0000 7f75437fe640 1 -- 172.21.15.125:0/3516947740 >> v1:172.21.15.136:6789/0 conn(0x7f75440c8430 legacy=0x7f754413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:17.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.639+0000 7f75437fe640 1 -- 172.21.15.125:0/3516947740 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754406fca0 con 0x7f75440c6b80 2023-10-30T12:51:17.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.639+0000 7f75437fe640 1 --2- 172.21.15.125:0/3516947740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75440c6b80 0x7f75440bfec0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f75340097c0 tx=0x7f753401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f983ba5988737c72 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:17.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f75427fc640 1 -- 172.21.15.125:0/3516947740 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75340ad640 con 0x7f75440c6b80 2023-10-30T12:51:17.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f75427fc640 1 -- 172.21.15.125:0/3516947740 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75340ad7a0 con 0x7f75440c6b80 2023-10-30T12:51:17.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f75427fc640 1 -- 172.21.15.125:0/3516947740 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75340adb10 con 0x7f75440c6b80 2023-10-30T12:51:17.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 -- 172.21.15.125:0/3516947740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75440c6b80 msgr2=0x7f75440bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:17.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 --2- 172.21.15.125:0/3516947740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75440c6b80 0x7f75440bfec0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f75340097c0 tx=0x7f753401c480 comp rx=0 tx=0).stop 2023-10-30T12:51:17.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 -- 172.21.15.125:0/3516947740 shutdown_connections 2023-10-30T12:51:17.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 --2- 172.21.15.125:0/3516947740 >> v2:172.21.15.136:3300/0 conn(0x7f75440c6f60 0x7f75440c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 --2- 172.21.15.125:0/3516947740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75440c6b80 0x7f75440bfec0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 -- 172.21.15.125:0/3516947740 >> 172.21.15.125:0/3516947740 conn(0x7f75440c7910 msgr2=0x7f75440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:17.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 -- 172.21.15.125:0/3516947740 shutdown_connections 2023-10-30T12:51:17.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 -- 172.21.15.125:0/3516947740 wait complete. 2023-10-30T12:51:17.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 Processor -- start 2023-10-30T12:51:17.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 -- start start 2023-10-30T12:51:17.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75440c6b80 0x7f7544134e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:17.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75440c6f60 0x7f754413cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:17.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75440c8430 0x7f754413d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:17.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7544158780 con 0x7f75440c8430 2023-10-30T12:51:17.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75441343e0 con 0x7f75440c6f60 2023-10-30T12:51:17.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.643+0000 7f754a10f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7544135820 con 0x7f75440c6b80 2023-10-30T12:51:17.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f7543fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75440c8430 0x7f754413d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:17.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f7543fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75440c8430 0x7f754413d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58196/0 (socket says 172.21.15.125:58196) 2023-10-30T12:51:17.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f7543fff640 1 -- 172.21.15.125:0/2439775068 learned_addr learned my addr 172.21.15.125:0/2439775068 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:17.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f75437fe640 1 --2- 172.21.15.125:0/2439775068 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75440c6b80 0x7f7544134e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:17.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f7542ffd640 1 --2- 172.21.15.125:0/2439775068 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75440c6f60 0x7f754413cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f7543fff640 1 -- 172.21.15.125:0/2439775068 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75440c6b80 msgr2=0x7f7544134e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f7543fff640 1 --2- 172.21.15.125:0/2439775068 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75440c6b80 0x7f7544134e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f7543fff640 1 -- 172.21.15.125:0/2439775068 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75440c6f60 msgr2=0x7f754413cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f7543fff640 1 --2- 172.21.15.125:0/2439775068 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75440c6f60 0x7f754413cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f7543fff640 1 -- 172.21.15.125:0/2439775068 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7534009420 con 0x7f75440c8430 2023-10-30T12:51:17.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f7543fff640 1 --2- 172.21.15.125:0/2439775068 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75440c8430 0x7f754413d420 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f753c00caa0 tx=0x7f753c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:17.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f752bfff640 1 -- 172.21.15.125:0/2439775068 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c00ac00 con 0x7f75440c8430 2023-10-30T12:51:17.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f752bfff640 1 -- 172.21.15.125:0/2439775068 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753c00ad60 con 0x7f75440c8430 2023-10-30T12:51:17.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f754a10f640 1 -- 172.21.15.125:0/2439775068 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75441584a0 con 0x7f75440c8430 2023-10-30T12:51:17.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f752bfff640 1 -- 172.21.15.125:0/2439775068 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c012620 con 0x7f75440c8430 2023-10-30T12:51:17.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f754a10f640 1 -- 172.21.15.125:0/2439775068 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7544134630 con 0x7f75440c8430 2023-10-30T12:51:17.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f754a10f640 1 -- 172.21.15.125:0/2439775068 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f754413c820 con 0x7f75440c8430 2023-10-30T12:51:17.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f752bfff640 1 -- 172.21.15.125:0/2439775068 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f753c007070 con 0x7f75440c8430 2023-10-30T12:51:17.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.647+0000 7f752bfff640 1 --2- 172.21.15.125:0/2439775068 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75300664a0 0x7f7530068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:17.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.651+0000 7f752bfff640 1 -- 172.21.15.125:0/2439775068 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7f753c084260 con 0x7f75440c8430 2023-10-30T12:51:17.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.651+0000 7f75437fe640 1 --2- 172.21.15.125:0/2439775068 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75300664a0 0x7f7530068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:17.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.651+0000 7f75437fe640 1 --2- 172.21.15.125:0/2439775068 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75300664a0 0x7f7530068960 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f75340af000 tx=0x7f75340a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:17.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.651+0000 7f752bfff640 1 -- 172.21.15.125:0/2439775068 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f754413c820 con 0x7f75440c8430 2023-10-30T12:51:17.714 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:51:17.718 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:51:17.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.839+0000 7f754a10f640 1 -- 172.21.15.125:0/2439775068 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f754413cc00 con 0x7f75440c8430 2023-10-30T12:51:17.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.839+0000 7f752bfff640 1 -- 172.21.15.125:0/2439775068 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f754413cc00 con 0x7f75440c8430 2023-10-30T12:51:17.840 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:17.840 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:15.314380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:17.840 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 112 2023-10-30T12:51:17.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.843+0000 7f754a10f640 1 -- 172.21.15.125:0/2439775068 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75300664a0 msgr2=0x7f7530068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:17.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.843+0000 7f754a10f640 1 --2- 172.21.15.125:0/2439775068 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75300664a0 0x7f7530068960 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f75340af000 tx=0x7f75340a7070 comp rx=0 tx=0).stop 2023-10-30T12:51:17.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.843+0000 7f754a10f640 1 -- 172.21.15.125:0/2439775068 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75440c8430 msgr2=0x7f754413d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:17.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.843+0000 7f754a10f640 1 --2- 172.21.15.125:0/2439775068 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75440c8430 0x7f754413d420 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f753c00caa0 tx=0x7f753c00cf60 comp rx=0 tx=0).stop 2023-10-30T12:51:17.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.843+0000 7f754a10f640 1 -- 172.21.15.125:0/2439775068 shutdown_connections 2023-10-30T12:51:17.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.843+0000 7f754a10f640 1 --2- 172.21.15.125:0/2439775068 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75300664a0 0x7f7530068960 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.843+0000 7f754a10f640 1 --2- 172.21.15.125:0/2439775068 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75440c8430 0x7f754413d420 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.843+0000 7f754a10f640 1 --2- 172.21.15.125:0/2439775068 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75440c6f60 0x7f754413cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.843+0000 7f754a10f640 1 --2- 172.21.15.125:0/2439775068 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75440c6b80 0x7f7544134e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:17.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.843+0000 7f754a10f640 1 -- 172.21.15.125:0/2439775068 >> 172.21.15.125:0/2439775068 conn(0x7f75440c7910 msgr2=0x7f75441389a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:17.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.843+0000 7f754a10f640 1 -- 172.21.15.125:0/2439775068 shutdown_connections 2023-10-30T12:51:17.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:17.843+0000 7f754a10f640 1 -- 172.21.15.125:0/2439775068 wait complete. 2023-10-30T12:51:17.861 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 112, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 112, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6627": { "addr": "172.21.15.125:6837/2408701650", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2408701650, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2408701650, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6627, "incarnation": 112, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:51:15.314380+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6627 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:51:19.862 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:20.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.007+0000 7f5dc793e640 1 Processor -- start 2023-10-30T12:51:20.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.007+0000 7f5dc793e640 1 -- start start 2023-10-30T12:51:20.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.011+0000 7f5dc793e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dc01580e0 0x7f5dc0070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:20.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.011+0000 7f5dc793e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5dc00c6540 0x7f5dc015e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:20.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.011+0000 7f5dc793e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc00c5810 con 0x7f5dc0069e90 2023-10-30T12:51:20.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.011+0000 7f5dc793e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dc00c6960 con 0x7f5dc00c6540 2023-10-30T12:51:20.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.011+0000 7f5dc793e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00c6bb0 con 0x7f5dc01580e0 2023-10-30T12:51:20.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.011+0000 7f5dc56da640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dc01580e0 0x7f5dc0070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:20.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.011+0000 7f5dc4ed9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5dc00c6540 0x7f5dc015e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:20.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.011+0000 7f5dc56da640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dc01580e0 0x7f5dc0070740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:34308/0 (socket says 172.21.15.125:34308) 2023-10-30T12:51:20.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.011+0000 7f5dc56da640 1 -- 172.21.15.125:0/3911840471 learned_addr learned my addr 172.21.15.125:0/3911840471 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:20.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.011+0000 7f5dc56da640 1 -- 172.21.15.125:0/3911840471 >> v2:172.21.15.136:3300/0 conn(0x7f5dc00c6540 msgr2=0x7f5dc015e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:20.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.011+0000 7f5dc56da640 1 --2- 172.21.15.125:0/3911840471 >> v2:172.21.15.136:3300/0 conn(0x7f5dc00c6540 0x7f5dc015e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:20.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.011+0000 7f5dc56da640 1 -- 172.21.15.125:0/3911840471 >> v1:172.21.15.136:6789/0 conn(0x7f5dc0069e90 legacy=0x7f5dc006a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:20.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.011+0000 7f5dc56da640 1 -- 172.21.15.125:0/3911840471 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc00c6e70 con 0x7f5dc01580e0 2023-10-30T12:51:20.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.015+0000 7f5dc56da640 1 --2- 172.21.15.125:0/3911840471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dc01580e0 0x7f5dc0070740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009870 tx=0x7f5dbc01c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58ff42ccd8f99825 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:20.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.015+0000 7f5db7fff640 1 -- 172.21.15.125:0/3911840471 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc0a9400 con 0x7f5dc01580e0 2023-10-30T12:51:20.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.015+0000 7f5db7fff640 1 -- 172.21.15.125:0/3911840471 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dbc0a9560 con 0x7f5dc01580e0 2023-10-30T12:51:20.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.015+0000 7f5db7fff640 1 -- 172.21.15.125:0/3911840471 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc0a98f0 con 0x7f5dc01580e0 2023-10-30T12:51:20.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.015+0000 7f5dc793e640 1 -- 172.21.15.125:0/3911840471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dc01580e0 msgr2=0x7f5dc0070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:20.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.015+0000 7f5dc793e640 1 --2- 172.21.15.125:0/3911840471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dc01580e0 0x7f5dc0070740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009870 tx=0x7f5dbc01c540 comp rx=0 tx=0).stop 2023-10-30T12:51:20.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.015+0000 7f5dc793e640 1 -- 172.21.15.125:0/3911840471 shutdown_connections 2023-10-30T12:51:20.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.015+0000 7f5dc793e640 1 --2- 172.21.15.125:0/3911840471 >> v2:172.21.15.136:3300/0 conn(0x7f5dc00c6540 0x7f5dc015e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:20.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.015+0000 7f5dc793e640 1 --2- 172.21.15.125:0/3911840471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dc01580e0 0x7f5dc0070740 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:20.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.015+0000 7f5dc793e640 1 -- 172.21.15.125:0/3911840471 >> 172.21.15.125:0/3911840471 conn(0x7f5dc00c7910 msgr2=0x7f5dc00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:20.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.015+0000 7f5dc793e640 1 -- 172.21.15.125:0/3911840471 shutdown_connections 2023-10-30T12:51:20.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.015+0000 7f5dc793e640 1 -- 172.21.15.125:0/3911840471 wait complete. 2023-10-30T12:51:20.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.015+0000 7f5dc793e640 1 Processor -- start 2023-10-30T12:51:20.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.015+0000 7f5dc793e640 1 -- start start 2023-10-30T12:51:20.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc793e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dc0069e90 0x7f5dc015f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:20.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc793e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5dc00c6540 0x7f5dc014e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:20.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc793e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dc01580e0 0x7f5dc0148b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:20.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc793e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc015fd90 con 0x7f5dc00c6540 2023-10-30T12:51:20.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc793e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc0157050 con 0x7f5dc01580e0 2023-10-30T12:51:20.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc793e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dc00c8cb0 con 0x7f5dc0069e90 2023-10-30T12:51:20.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc56da640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dc0069e90 0x7f5dc015f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:20.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc4ed9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5dc00c6540 0x7f5dc014e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:20.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc4ed9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5dc00c6540 0x7f5dc014e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58206/0 (socket says 172.21.15.125:58206) 2023-10-30T12:51:20.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc4ed9640 1 -- 172.21.15.125:0/2526059856 learned_addr learned my addr 172.21.15.125:0/2526059856 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:20.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc5edb640 1 --2- 172.21.15.125:0/2526059856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dc01580e0 0x7f5dc0148b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:20.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc4ed9640 1 -- 172.21.15.125:0/2526059856 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dc0069e90 msgr2=0x7f5dc015f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:20.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc4ed9640 1 --2- 172.21.15.125:0/2526059856 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dc0069e90 0x7f5dc015f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:20.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc4ed9640 1 -- 172.21.15.125:0/2526059856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dc01580e0 msgr2=0x7f5dc0148b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:20.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc4ed9640 1 --2- 172.21.15.125:0/2526059856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dc01580e0 0x7f5dc0148b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:20.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc4ed9640 1 -- 172.21.15.125:0/2526059856 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dbc0094d0 con 0x7f5dc00c6540 2023-10-30T12:51:20.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc4ed9640 1 --2- 172.21.15.125:0/2526059856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5dc00c6540 0x7f5dc014e220 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f5db000e930 tx=0x7f5db000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:20.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5db5ffb640 1 -- 172.21.15.125:0/2526059856 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db0009690 con 0x7f5dc00c6540 2023-10-30T12:51:20.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5db5ffb640 1 -- 172.21.15.125:0/2526059856 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5db000ca00 con 0x7f5dc00c6540 2023-10-30T12:51:20.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5db5ffb640 1 -- 172.21.15.125:0/2526059856 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5db000cd70 con 0x7f5dc00c6540 2023-10-30T12:51:20.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc793e640 1 -- 172.21.15.125:0/2526059856 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc00c8f20 con 0x7f5dc00c6540 2023-10-30T12:51:20.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc793e640 1 -- 172.21.15.125:0/2526059856 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc00c94e0 con 0x7f5dc00c6540 2023-10-30T12:51:20.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5dc793e640 1 -- 172.21.15.125:0/2526059856 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc004e170 con 0x7f5dc00c6540 2023-10-30T12:51:20.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.019+0000 7f5db5ffb640 1 -- 172.21.15.125:0/2526059856 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5db0013470 con 0x7f5dc00c6540 2023-10-30T12:51:20.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.023+0000 7f5db5ffb640 1 --2- 172.21.15.125:0/2526059856 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5da40664a0 0x7f5da4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:20.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.023+0000 7f5db5ffb640 1 -- 172.21.15.125:0/2526059856 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7f5db0084470 con 0x7f5dc00c6540 2023-10-30T12:51:20.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.023+0000 7f5dc56da640 1 --2- 172.21.15.125:0/2526059856 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5da40664a0 0x7f5da4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:20.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.023+0000 7f5dc56da640 1 --2- 172.21.15.125:0/2526059856 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5da40664a0 0x7f5da4068960 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0ae000 tx=0x7f5dbc0b3040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:20.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.027+0000 7f5db5ffb640 1 -- 172.21.15.125:0/2526059856 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5db00515f0 con 0x7f5dc00c6540 2023-10-30T12:51:20.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.207+0000 7f5dc793e640 1 -- 172.21.15.125:0/2526059856 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dc006aed0 con 0x7f5dc00c6540 2023-10-30T12:51:20.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.207+0000 7f5db5ffb640 1 -- 172.21.15.125:0/2526059856 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f5db0054c10 con 0x7f5dc00c6540 2023-10-30T12:51:20.206 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:20.206 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:15.314380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:20.207 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 112 2023-10-30T12:51:20.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.207+0000 7f5dc793e640 1 -- 172.21.15.125:0/2526059856 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5da40664a0 msgr2=0x7f5da4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:20.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.207+0000 7f5dc793e640 1 --2- 172.21.15.125:0/2526059856 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5da40664a0 0x7f5da4068960 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc0ae000 tx=0x7f5dbc0b3040 comp rx=0 tx=0).stop 2023-10-30T12:51:20.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.207+0000 7f5dc793e640 1 -- 172.21.15.125:0/2526059856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5dc00c6540 msgr2=0x7f5dc014e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:20.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.207+0000 7f5dc793e640 1 --2- 172.21.15.125:0/2526059856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5dc00c6540 0x7f5dc014e220 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f5db000e930 tx=0x7f5db000edf0 comp rx=0 tx=0).stop 2023-10-30T12:51:20.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.207+0000 7f5dc793e640 1 -- 172.21.15.125:0/2526059856 shutdown_connections 2023-10-30T12:51:20.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.207+0000 7f5dc793e640 1 --2- 172.21.15.125:0/2526059856 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5da40664a0 0x7f5da4068960 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:20.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.207+0000 7f5dc793e640 1 --2- 172.21.15.125:0/2526059856 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dc01580e0 0x7f5dc0148b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:20.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.207+0000 7f5dc793e640 1 --2- 172.21.15.125:0/2526059856 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5dc00c6540 0x7f5dc014e220 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:20.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.207+0000 7f5dc793e640 1 --2- 172.21.15.125:0/2526059856 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dc0069e90 0x7f5dc015f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:20.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.207+0000 7f5dc793e640 1 -- 172.21.15.125:0/2526059856 >> 172.21.15.125:0/2526059856 conn(0x7f5dc00c7910 msgr2=0x7f5dc0156610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:20.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.207+0000 7f5dc793e640 1 -- 172.21.15.125:0/2526059856 shutdown_connections 2023-10-30T12:51:20.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:20.207+0000 7f5dc793e640 1 -- 172.21.15.125:0/2526059856 wait complete. 2023-10-30T12:51:20.227 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:51:22.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:22.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.371+0000 7ff33359e640 1 Processor -- start 2023-10-30T12:51:22.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.375+0000 7ff33359e640 1 -- start start 2023-10-30T12:51:22.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.375+0000 7ff33359e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff3341539a0 0x7ff3340717c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:22.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.375+0000 7ff33359e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff33406ded0 0x7ff33406e340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:22.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.375+0000 7ff33359e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3340c6d00 con 0x7ff334070450 2023-10-30T12:51:22.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.375+0000 7ff33359e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff334071fe0 con 0x7ff3341539a0 2023-10-30T12:51:22.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.375+0000 7ff33359e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3340722c0 con 0x7ff33406ded0 2023-10-30T12:51:22.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.375+0000 7ff33259c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff3341539a0 0x7ff3340717c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:22.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.375+0000 7ff33259c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff3341539a0 0x7ff3340717c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60670/0 (socket says 172.21.15.125:60670) 2023-10-30T12:51:22.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.375+0000 7ff33259c640 1 -- 172.21.15.125:0/3197214612 learned_addr learned my addr 172.21.15.125:0/3197214612 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:22.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.375+0000 7ff332d9d640 1 --2- 172.21.15.125:0/3197214612 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff33406ded0 0x7ff33406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:22.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.375+0000 7ff33259c640 1 -- 172.21.15.125:0/3197214612 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff33406ded0 msgr2=0x7ff33406e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:22.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.375+0000 7ff33259c640 1 --2- 172.21.15.125:0/3197214612 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff33406ded0 0x7ff33406e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.375+0000 7ff33259c640 1 -- 172.21.15.125:0/3197214612 >> v1:172.21.15.136:6789/0 conn(0x7ff334070450 legacy=0x7ff3340c65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:22.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.375+0000 7ff33259c640 1 -- 172.21.15.125:0/3197214612 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33415cb00 con 0x7ff3341539a0 2023-10-30T12:51:22.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.379+0000 7ff33259c640 1 --2- 172.21.15.125:0/3197214612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3341539a0 0x7ff3340717c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ff324009870 tx=0x7ff32401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b113d8e25b3369 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:22.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.379+0000 7ff33159a640 1 -- 172.21.15.125:0/3197214612 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3240ac620 con 0x7ff3341539a0 2023-10-30T12:51:22.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.379+0000 7ff33159a640 1 -- 172.21.15.125:0/3197214612 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3240ac780 con 0x7ff3341539a0 2023-10-30T12:51:22.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.379+0000 7ff33159a640 1 -- 172.21.15.125:0/3197214612 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3240acb10 con 0x7ff3341539a0 2023-10-30T12:51:22.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.379+0000 7ff33359e640 1 -- 172.21.15.125:0/3197214612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3341539a0 msgr2=0x7ff3340717c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:22.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.379+0000 7ff33359e640 1 --2- 172.21.15.125:0/3197214612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3341539a0 0x7ff3340717c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ff324009870 tx=0x7ff32401c4b0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.379+0000 7ff33359e640 1 -- 172.21.15.125:0/3197214612 shutdown_connections 2023-10-30T12:51:22.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.379+0000 7ff33359e640 1 --2- 172.21.15.125:0/3197214612 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff33406ded0 0x7ff33406e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.379+0000 7ff33359e640 1 --2- 172.21.15.125:0/3197214612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3341539a0 0x7ff3340717c0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.379+0000 7ff33359e640 1 -- 172.21.15.125:0/3197214612 >> 172.21.15.125:0/3197214612 conn(0x7ff3340cbe40 msgr2=0x7ff3340cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:22.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.379+0000 7ff33359e640 1 -- 172.21.15.125:0/3197214612 shutdown_connections 2023-10-30T12:51:22.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.379+0000 7ff33359e640 1 -- 172.21.15.125:0/3197214612 wait complete. 2023-10-30T12:51:22.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33359e640 1 Processor -- start 2023-10-30T12:51:22.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33359e640 1 -- start start 2023-10-30T12:51:22.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33359e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff33406ded0 0x7ff3340c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:22.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33359e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff334070450 0x7ff33414dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:22.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33359e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3341539a0 0x7ff33414e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:22.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33359e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff334151ca0 con 0x7ff33406ded0 2023-10-30T12:51:22.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33359e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff334152ea0 con 0x7ff334070450 2023-10-30T12:51:22.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33359e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3341f2300 con 0x7ff3341539a0 2023-10-30T12:51:22.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33259c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff33406ded0 0x7ff3340c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:22.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33259c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff33406ded0 0x7ff3340c4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60686/0 (socket says 172.21.15.125:60686) 2023-10-30T12:51:22.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33259c640 1 -- 172.21.15.125:0/3635364076 learned_addr learned my addr 172.21.15.125:0/3635364076 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:22.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff332d9d640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3341539a0 0x7ff33414e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:22.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff331d9b640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff334070450 0x7ff33414dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:22.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33259c640 1 -- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3341539a0 msgr2=0x7ff33414e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:22.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33259c640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3341539a0 0x7ff33414e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33259c640 1 -- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff334070450 msgr2=0x7ff33414dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:22.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33259c640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff334070450 0x7ff33414dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33259c640 1 -- 172.21.15.125:0/3635364076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3240094d0 con 0x7ff33406ded0 2023-10-30T12:51:22.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff331d9b640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff334070450 0x7ff33414dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:22.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33259c640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff33406ded0 0x7ff3340c4a30 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7ff3240af040 tx=0x7ff32401acf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:22.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff322ffd640 1 -- 172.21.15.125:0/3635364076 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3240b7620 con 0x7ff33406ded0 2023-10-30T12:51:22.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff322ffd640 1 -- 172.21.15.125:0/3635364076 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3240b77a0 con 0x7ff33406ded0 2023-10-30T12:51:22.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff322ffd640 1 -- 172.21.15.125:0/3635364076 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3240b7b50 con 0x7ff33406ded0 2023-10-30T12:51:22.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.383+0000 7ff33359e640 1 -- 172.21.15.125:0/3635364076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3341f2830 con 0x7ff33406ded0 2023-10-30T12:51:22.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.387+0000 7ff33359e640 1 -- 172.21.15.125:0/3635364076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33414db50 con 0x7ff33406ded0 2023-10-30T12:51:22.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.387+0000 7ff33359e640 1 -- 172.21.15.125:0/3635364076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff33404e170 con 0x7ff33406ded0 2023-10-30T12:51:22.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.387+0000 7ff322ffd640 1 -- 172.21.15.125:0/3635364076 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff3240b3040 con 0x7ff33406ded0 2023-10-30T12:51:22.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.387+0000 7ff322ffd640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff3140664a0 0x7ff314068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:22.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.387+0000 7ff322ffd640 1 -- 172.21.15.125:0/3635364076 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7ff32411e4b0 con 0x7ff33406ded0 2023-10-30T12:51:22.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.387+0000 7ff331d9b640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff3140664a0 0x7ff314068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:22.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.391+0000 7ff331d9b640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff3140664a0 0x7ff314068960 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff3341ef4b0 tx=0x7ff328009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:22.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.391+0000 7ff322ffd640 1 -- 172.21.15.125:0/3635364076 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff3240eb7c0 con 0x7ff33406ded0 2023-10-30T12:51:22.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.583+0000 7ff33359e640 1 -- 172.21.15.125:0/3635364076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3340c28f0 con 0x7ff33406ded0 2023-10-30T12:51:22.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.583+0000 7ff322ffd640 1 -- 172.21.15.125:0/3635364076 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5723 (secure 0 0 0) 0x7ff3240ac5e0 con 0x7ff33406ded0 2023-10-30T12:51:22.583 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:22.584 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:15.314380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:22.584 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 112 2023-10-30T12:51:22.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.587+0000 7ff33359e640 1 -- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff3140664a0 msgr2=0x7ff314068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:22.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.587+0000 7ff33359e640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff3140664a0 0x7ff314068960 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7ff3341ef4b0 tx=0x7ff328009040 comp rx=0 tx=0).stop 2023-10-30T12:51:22.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.587+0000 7ff33359e640 1 -- 172.21.15.125:0/3635364076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff33406ded0 msgr2=0x7ff3340c4a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:22.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.587+0000 7ff33359e640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff33406ded0 0x7ff3340c4a30 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7ff3240af040 tx=0x7ff32401acf0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.587+0000 7ff33359e640 1 -- 172.21.15.125:0/3635364076 shutdown_connections 2023-10-30T12:51:22.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.587+0000 7ff33359e640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff3140664a0 0x7ff314068960 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.587+0000 7ff33359e640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3341539a0 0x7ff33414e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.587+0000 7ff33359e640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff334070450 0x7ff33414dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.587+0000 7ff33359e640 1 --2- 172.21.15.125:0/3635364076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff33406ded0 0x7ff3340c4a30 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.587+0000 7ff33359e640 1 -- 172.21.15.125:0/3635364076 >> 172.21.15.125:0/3635364076 conn(0x7ff3340cbe40 msgr2=0x7ff33414bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:22.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.587+0000 7ff33359e640 1 -- 172.21.15.125:0/3635364076 shutdown_connections 2023-10-30T12:51:22.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:22.587+0000 7ff33359e640 1 -- 172.21.15.125:0/3635364076 wait complete. 2023-10-30T12:51:22.613 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:51:24.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:24.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.759+0000 7f4b76fb8640 1 Processor -- start 2023-10-30T12:51:24.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.759+0000 7f4b76fb8640 1 -- start start 2023-10-30T12:51:24.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b76fb8640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4b700bfec0 0x7f4b700c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:24.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b76fb8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b7006a720 0x7f4b7006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:24.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b76fb8640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b70071290 con 0x7f4b70159e60 2023-10-30T12:51:24.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b76fb8640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b70071570 con 0x7f4b700bfec0 2023-10-30T12:51:24.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b76fb8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70071850 con 0x7f4b7006a720 2023-10-30T12:51:24.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b6ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4b700bfec0 0x7f4b700c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:24.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b6ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4b700bfec0 0x7f4b700c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60702/0 (socket says 172.21.15.125:60702) 2023-10-30T12:51:24.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b6ffff640 1 -- 172.21.15.125:0/3846426415 learned_addr learned my addr 172.21.15.125:0/3846426415 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:24.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b75555640 1 --2- 172.21.15.125:0/3846426415 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b7006a720 0x7f4b7006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:24.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b6ffff640 1 -- 172.21.15.125:0/3846426415 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b7006a720 msgr2=0x7f4b7006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:24.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b6ffff640 1 --2- 172.21.15.125:0/3846426415 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b7006a720 0x7f4b7006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b6ffff640 1 -- 172.21.15.125:0/3846426415 >> v1:172.21.15.125:6789/0 conn(0x7f4b70159e60 legacy=0x7f4b7006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:24.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b6ffff640 1 -- 172.21.15.125:0/3846426415 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b70071b30 con 0x7f4b700bfec0 2023-10-30T12:51:24.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b6f7fe640 1 -- 172.21.15.125:0/3846426415 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1033736786 0 0) 0x7f4b70071290 con 0x7f4b70159e60 2023-10-30T12:51:24.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.763+0000 7f4b75555640 1 --2- 172.21.15.125:0/3846426415 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b7006a720 0x7f4b7006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:24.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.767+0000 7f4b6ffff640 1 --2- 172.21.15.125:0/3846426415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b700bfec0 0x7f4b700c1a40 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c006fd0 tx=0x7f4b5c0211c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a442c8a2a7fbdc27 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:24.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.767+0000 7f4b6f7fe640 1 -- 172.21.15.125:0/3846426415 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c004030 con 0x7f4b700bfec0 2023-10-30T12:51:24.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.767+0000 7f4b6f7fe640 1 -- 172.21.15.125:0/3846426415 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b5c0ad030 con 0x7f4b700bfec0 2023-10-30T12:51:24.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.767+0000 7f4b6f7fe640 1 -- 172.21.15.125:0/3846426415 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b5c0043d0 con 0x7f4b700bfec0 2023-10-30T12:51:24.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.767+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3846426415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b700bfec0 msgr2=0x7f4b700c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:24.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.767+0000 7f4b76fb8640 1 --2- 172.21.15.125:0/3846426415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b700bfec0 0x7f4b700c1a40 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c006fd0 tx=0x7f4b5c0211c0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.767+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3846426415 shutdown_connections 2023-10-30T12:51:24.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.767+0000 7f4b76fb8640 1 --2- 172.21.15.125:0/3846426415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b700bfec0 0x7f4b700c1a40 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.767+0000 7f4b76fb8640 1 --2- 172.21.15.125:0/3846426415 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b7006a720 0x7f4b7006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.767+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3846426415 >> 172.21.15.125:0/3846426415 conn(0x7f4b700c7910 msgr2=0x7f4b700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:24.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.767+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3846426415 shutdown_connections 2023-10-30T12:51:24.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.767+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3846426415 wait complete. 2023-10-30T12:51:24.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b76fb8640 1 Processor -- start 2023-10-30T12:51:24.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b76fb8640 1 -- start start 2023-10-30T12:51:24.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b76fb8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7006a720 0x7f4b701eb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:24.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b76fb8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b700bfec0 0x7f4b701ebe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:24.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b76fb8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b70159e60 0x7f4b701ee920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:24.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b76fb8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701eee60 con 0x7f4b7006a720 2023-10-30T12:51:24.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b76fb8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b700c4190 con 0x7f4b700bfec0 2023-10-30T12:51:24.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b76fb8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b700c4470 con 0x7f4b70159e60 2023-10-30T12:51:24.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b74d54640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7006a720 0x7f4b701eb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:24.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b74d54640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7006a720 0x7f4b701eb590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60716/0 (socket says 172.21.15.125:60716) 2023-10-30T12:51:24.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b74d54640 1 -- 172.21.15.125:0/3071851279 learned_addr learned my addr 172.21.15.125:0/3071851279 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:24.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b75555640 1 --2- 172.21.15.125:0/3071851279 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b70159e60 0x7f4b701ee920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:24.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b6ffff640 1 --2- 172.21.15.125:0/3071851279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b700bfec0 0x7f4b701ebe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:24.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b74d54640 1 -- 172.21.15.125:0/3071851279 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b70159e60 msgr2=0x7f4b701ee920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:24.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b74d54640 1 --2- 172.21.15.125:0/3071851279 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b70159e60 0x7f4b701ee920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b74d54640 1 -- 172.21.15.125:0/3071851279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b700bfec0 msgr2=0x7f4b701ebe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:24.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b74d54640 1 --2- 172.21.15.125:0/3071851279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b700bfec0 0x7f4b701ebe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b74d54640 1 -- 172.21.15.125:0/3071851279 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c00b780 con 0x7f4b7006a720 2023-10-30T12:51:24.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b74d54640 1 --2- 172.21.15.125:0/3071851279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7006a720 0x7f4b701eb590 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4b64001ec0 tx=0x7f4b6400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:24.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b6d7fa640 1 -- 172.21.15.125:0/3071851279 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b640054e0 con 0x7f4b7006a720 2023-10-30T12:51:24.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b6d7fa640 1 -- 172.21.15.125:0/3071851279 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b64003600 con 0x7f4b7006a720 2023-10-30T12:51:24.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b6d7fa640 1 -- 172.21.15.125:0/3071851279 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b640039b0 con 0x7f4b7006a720 2023-10-30T12:51:24.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3071851279 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b700c4750 con 0x7f4b7006a720 2023-10-30T12:51:24.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.771+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3071851279 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b701ead90 con 0x7f4b7006a720 2023-10-30T12:51:24.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.775+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3071851279 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b7004e170 con 0x7f4b7006a720 2023-10-30T12:51:24.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.775+0000 7f4b6d7fa640 1 -- 172.21.15.125:0/3071851279 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4b64003b10 con 0x7f4b7006a720 2023-10-30T12:51:24.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.775+0000 7f4b6d7fa640 1 --2- 172.21.15.125:0/3071851279 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b540664a0 0x7f4b54068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:24.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.775+0000 7f4b6d7fa640 1 -- 172.21.15.125:0/3071851279 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7f4b640838b0 con 0x7f4b7006a720 2023-10-30T12:51:24.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.775+0000 7f4b6ffff640 1 --2- 172.21.15.125:0/3071851279 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b540664a0 0x7f4b54068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:24.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.779+0000 7f4b6ffff640 1 --2- 172.21.15.125:0/3071851279 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b540664a0 0x7f4b54068960 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c0b4000 tx=0x7f4b5c0ac000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:24.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.783+0000 7f4b6d7fa640 1 -- 172.21.15.125:0/3071851279 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4b640508e0 con 0x7f4b7006a720 2023-10-30T12:51:24.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.975+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3071851279 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b700c1240 con 0x7f4b7006a720 2023-10-30T12:51:24.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.975+0000 7f4b6d7fa640 1 -- 172.21.15.125:0/3071851279 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f4b64017070 con 0x7f4b7006a720 2023-10-30T12:51:24.975 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:24.975 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":112,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:15.314380+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:24.975 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 112 2023-10-30T12:51:24.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.979+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3071851279 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b540664a0 msgr2=0x7f4b54068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:24.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.979+0000 7f4b76fb8640 1 --2- 172.21.15.125:0/3071851279 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b540664a0 0x7f4b54068960 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f4b5c0b4000 tx=0x7f4b5c0ac000 comp rx=0 tx=0).stop 2023-10-30T12:51:24.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.979+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3071851279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7006a720 msgr2=0x7f4b701eb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:24.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.979+0000 7f4b76fb8640 1 --2- 172.21.15.125:0/3071851279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7006a720 0x7f4b701eb590 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f4b64001ec0 tx=0x7f4b6400c8e0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.979+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3071851279 shutdown_connections 2023-10-30T12:51:24.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.979+0000 7f4b76fb8640 1 --2- 172.21.15.125:0/3071851279 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b540664a0 0x7f4b54068960 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.979+0000 7f4b76fb8640 1 --2- 172.21.15.125:0/3071851279 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b70159e60 0x7f4b701ee920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.979+0000 7f4b76fb8640 1 --2- 172.21.15.125:0/3071851279 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b700bfec0 0x7f4b701ebe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.979+0000 7f4b76fb8640 1 --2- 172.21.15.125:0/3071851279 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b7006a720 0x7f4b701eb590 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:24.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.979+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3071851279 >> 172.21.15.125:0/3071851279 conn(0x7f4b700c7910 msgr2=0x7f4b70073480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:24.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.979+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3071851279 shutdown_connections 2023-10-30T12:51:24.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:24.979+0000 7f4b76fb8640 1 -- 172.21.15.125:0/3071851279 wait complete. 2023-10-30T12:51:24.998 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:51:26.999 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:27.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.143+0000 7f94091d3640 1 Processor -- start 2023-10-30T12:51:27.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.143+0000 7f94091d3640 1 -- start start 2023-10-30T12:51:27.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f94091d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f940415d6f0 0x7f940415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:27.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f94091d3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94040cb800 con 0x7f94041538a0 2023-10-30T12:51:27.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f94091d3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94040cb330 con 0x7f94040cf160 2023-10-30T12:51:27.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f94091d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94040cad70 con 0x7f940415d6f0 2023-10-30T12:51:27.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9402d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f94041538a0 0x7f94040ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57682/0 (socket says 172.21.15.125:57682) 2023-10-30T12:51:27.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9402d9d640 1 -- 172.21.15.125:0/859854306 learned_addr learned my addr 172.21.15.125:0/859854306 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:51:27.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f940259c640 1 --2- 172.21.15.125:0/859854306 >> v2:172.21.15.136:3300/0 conn(0x7f940415d6f0 0x7f940415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:27.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 -- 172.21.15.125:0/859854306 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076549173 0 0) 0x7f94040cb800 con 0x7f94041538a0 2023-10-30T12:51:27.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 -- 172.21.15.125:0/859854306 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93ec003530 con 0x7f94041538a0 2023-10-30T12:51:27.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 -- 172.21.15.125:0/859854306 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3677491490 0 0) 0x7f93ec003530 con 0x7f94041538a0 2023-10-30T12:51:27.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 -- 172.21.15.125:0/859854306 >> v2:172.21.15.136:3300/0 conn(0x7f940415d6f0 msgr2=0x7f940415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:27.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 --2- 172.21.15.125:0/859854306 >> v2:172.21.15.136:3300/0 conn(0x7f940415d6f0 0x7f940415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:27.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 -- 172.21.15.125:0/859854306 >> v1:172.21.15.136:6789/0 conn(0x7f94040cf160 legacy=0x7f9404164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:27.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 -- 172.21.15.125:0/859854306 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94040cb050 con 0x7f94041538a0 2023-10-30T12:51:27.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 -- 172.21.15.125:0/859854306 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f93f4001680 con 0x7f94041538a0 2023-10-30T12:51:27.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 -- 172.21.15.125:0/859854306 >> v1:172.21.15.125:6789/0 conn(0x7f94041538a0 legacy=0x7f94040ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:27.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 --2- 172.21.15.125:0/859854306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec003980 0x7f93ec003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:27.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 --2- 172.21.15.125:0/859854306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93ec004380 0x7f93ec004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:27.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 --2- 172.21.15.125:0/859854306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93ec005050 0x7f93ec00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:27.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 -- 172.21.15.125:0/859854306 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94040cad70 con 0x7f93ec003980 2023-10-30T12:51:27.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 -- 172.21.15.125:0/859854306 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec00bf00 con 0x7f93ec004380 2023-10-30T12:51:27.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9401d9b640 1 -- 172.21.15.125:0/859854306 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ec00c0c0 con 0x7f93ec005050 2023-10-30T12:51:27.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9402d9d640 1 --2- 172.21.15.125:0/859854306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec003980 0x7f93ec003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:27.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f940259c640 1 --2- 172.21.15.125:0/859854306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93ec004380 0x7f93ec004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:27.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f940359e640 1 --2- 172.21.15.125:0/859854306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93ec005050 0x7f93ec00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:27.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9402d9d640 1 -- 172.21.15.125:0/859854306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93ec005050 msgr2=0x7f93ec00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:27.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9402d9d640 1 --2- 172.21.15.125:0/859854306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f93ec005050 0x7f93ec00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:27.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9402d9d640 1 -- 172.21.15.125:0/859854306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93ec004380 msgr2=0x7f93ec004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:27.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9402d9d640 1 --2- 172.21.15.125:0/859854306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93ec004380 0x7f93ec004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:27.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f9402d9d640 1 -- 172.21.15.125:0/859854306 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ec003530 con 0x7f93ec003980 2023-10-30T12:51:27.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f940259c640 1 --2- 172.21.15.125:0/859854306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93ec004380 0x7f93ec004850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:27.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.147+0000 7f940359e640 1 -- 172.21.15.125:0/859854306 reap_dead start 2023-10-30T12:51:27.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.151+0000 7f9402d9d640 1 --2- 172.21.15.125:0/859854306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec003980 0x7f93ec003d60 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f93f400c190 tx=0x7f93f401cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fb666c32150319b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:27.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.151+0000 7f9401d9b640 1 -- 172.21.15.125:0/859854306 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f4005ae0 con 0x7f93ec003980 2023-10-30T12:51:27.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.151+0000 7f9401d9b640 1 -- 172.21.15.125:0/859854306 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f4005c60 con 0x7f93ec003980 2023-10-30T12:51:27.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.151+0000 7f9401d9b640 1 -- 172.21.15.125:0/859854306 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f40a5ae0 con 0x7f93ec003980 2023-10-30T12:51:27.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.151+0000 7f94091d3640 1 -- 172.21.15.125:0/859854306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec003980 msgr2=0x7f93ec003d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:27.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.151+0000 7f94091d3640 1 --2- 172.21.15.125:0/859854306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec003980 0x7f93ec003d60 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f93f400c190 tx=0x7f93f401cba0 comp rx=0 tx=0).stop 2023-10-30T12:51:27.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.151+0000 7f94091d3640 1 -- 172.21.15.125:0/859854306 shutdown_connections 2023-10-30T12:51:27.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.151+0000 7f94091d3640 1 --2- 172.21.15.125:0/859854306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec003980 0x7f93ec003d60 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:27.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.151+0000 7f94091d3640 1 -- 172.21.15.125:0/859854306 >> 172.21.15.125:0/859854306 conn(0x7f940406bfe0 msgr2=0x7f940406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:27.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.151+0000 7f94091d3640 1 -- 172.21.15.125:0/859854306 shutdown_connections 2023-10-30T12:51:27.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.151+0000 7f94091d3640 1 -- 172.21.15.125:0/859854306 wait complete. 2023-10-30T12:51:27.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f94091d3640 1 Processor -- start 2023-10-30T12:51:27.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f94091d3640 1 -- start start 2023-10-30T12:51:27.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f94091d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec003980 0x7f94041f2610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:27.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f94091d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f940414e560 0x7f94040ceac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:27.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f94091d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94040b30d0 0x7f9404065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:27.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f94091d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94040c9630 con 0x7f93ec003980 2023-10-30T12:51:27.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f94091d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9404150170 con 0x7f94040b30d0 2023-10-30T12:51:27.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f94091d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94040c8e00 con 0x7f940414e560 2023-10-30T12:51:27.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f9402d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec003980 0x7f94041f2610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:27.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f9402d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec003980 0x7f94041f2610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60744/0 (socket says 172.21.15.125:60744) 2023-10-30T12:51:27.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f9402d9d640 1 -- 172.21.15.125:0/2989908763 learned_addr learned my addr 172.21.15.125:0/2989908763 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:27.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f940259c640 1 --2- 172.21.15.125:0/2989908763 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f940414e560 0x7f94040ceac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:27.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f940359e640 1 --2- 172.21.15.125:0/2989908763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94040b30d0 0x7f9404065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:27.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f9402d9d640 1 -- 172.21.15.125:0/2989908763 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f940414e560 msgr2=0x7f94040ceac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:27.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f9402d9d640 1 --2- 172.21.15.125:0/2989908763 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f940414e560 0x7f94040ceac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:27.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f9402d9d640 1 -- 172.21.15.125:0/2989908763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94040b30d0 msgr2=0x7f9404065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:27.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f9402d9d640 1 --2- 172.21.15.125:0/2989908763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94040b30d0 0x7f9404065ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:27.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f9402d9d640 1 -- 172.21.15.125:0/2989908763 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94040c9070 con 0x7f93ec003980 2023-10-30T12:51:27.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f9402d9d640 1 --2- 172.21.15.125:0/2989908763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec003980 0x7f94041f2610 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f93f401c9d0 tx=0x7f93ec005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:27.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f93eb7fe640 1 -- 172.21.15.125:0/2989908763 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f40b4070 con 0x7f93ec003980 2023-10-30T12:51:27.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f93eb7fe640 1 -- 172.21.15.125:0/2989908763 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93f4000f20 con 0x7f93ec003980 2023-10-30T12:51:27.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f94091d3640 1 -- 172.21.15.125:0/2989908763 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94040c9350 con 0x7f93ec003980 2023-10-30T12:51:27.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f93eb7fe640 1 -- 172.21.15.125:0/2989908763 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93f40a5cb0 con 0x7f93ec003980 2023-10-30T12:51:27.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f94091d3640 1 -- 172.21.15.125:0/2989908763 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9404066810 con 0x7f93ec003980 2023-10-30T12:51:27.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.155+0000 7f94091d3640 1 -- 172.21.15.125:0/2989908763 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9404159830 con 0x7f93ec003980 2023-10-30T12:51:27.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.159+0000 7f93eb7fe640 1 -- 172.21.15.125:0/2989908763 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f93f4007070 con 0x7f93ec003980 2023-10-30T12:51:27.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.159+0000 7f93eb7fe640 1 --2- 172.21.15.125:0/2989908763 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93f00664d0 0x7f93f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:27.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.159+0000 7f93eb7fe640 1 -- 172.21.15.125:0/2989908763 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7f93f411daa0 con 0x7f93ec003980 2023-10-30T12:51:27.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.159+0000 7f940259c640 1 --2- 172.21.15.125:0/2989908763 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93f00664d0 0x7f93f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:27.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.159+0000 7f940259c640 1 --2- 172.21.15.125:0/2989908763 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93f00664d0 0x7f93f0068990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f940406c5c0 tx=0x7f93f8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:27.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.163+0000 7f93eb7fe640 1 -- 172.21.15.125:0/2989908763 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f93f40eaab0 con 0x7f93ec003980 2023-10-30T12:51:27.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.343+0000 7f94091d3640 1 -- 172.21.15.125:0/2989908763 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f940404e170 con 0x7f93ec003980 2023-10-30T12:51:27.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.343+0000 7f93eb7fe640 1 -- 172.21.15.125:0/2989908763 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f93f40ee0d0 con 0x7f93ec003980 2023-10-30T12:51:27.344 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:27.344 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:25.028391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:reconnect","state_seq":39,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:27.345 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 113 2023-10-30T12:51:27.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.347+0000 7f94091d3640 1 -- 172.21.15.125:0/2989908763 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93f00664d0 msgr2=0x7f93f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:27.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.347+0000 7f94091d3640 1 --2- 172.21.15.125:0/2989908763 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93f00664d0 0x7f93f0068990 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f940406c5c0 tx=0x7f93f8008040 comp rx=0 tx=0).stop 2023-10-30T12:51:27.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.347+0000 7f94091d3640 1 -- 172.21.15.125:0/2989908763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec003980 msgr2=0x7f94041f2610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:27.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.347+0000 7f94091d3640 1 --2- 172.21.15.125:0/2989908763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec003980 0x7f94041f2610 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f93f401c9d0 tx=0x7f93ec005800 comp rx=0 tx=0).stop 2023-10-30T12:51:27.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.347+0000 7f94091d3640 1 -- 172.21.15.125:0/2989908763 shutdown_connections 2023-10-30T12:51:27.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.347+0000 7f94091d3640 1 --2- 172.21.15.125:0/2989908763 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93f00664d0 0x7f93f0068990 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:27.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.347+0000 7f94091d3640 1 --2- 172.21.15.125:0/2989908763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94040b30d0 0x7f9404065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:27.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.347+0000 7f94091d3640 1 --2- 172.21.15.125:0/2989908763 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f940414e560 0x7f94040ceac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:27.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.347+0000 7f94091d3640 1 --2- 172.21.15.125:0/2989908763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93ec003980 0x7f94041f2610 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:27.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.347+0000 7f94091d3640 1 -- 172.21.15.125:0/2989908763 >> 172.21.15.125:0/2989908763 conn(0x7f940406bfe0 msgr2=0x7f9404151020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:27.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.347+0000 7f94091d3640 1 -- 172.21.15.125:0/2989908763 shutdown_connections 2023-10-30T12:51:27.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:27.347+0000 7f94091d3640 1 -- 172.21.15.125:0/2989908763 wait complete. 2023-10-30T12:51:27.365 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:51:29.366 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:29.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.515+0000 7f5ff9d9d640 1 Processor -- start 2023-10-30T12:51:29.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.515+0000 7f5ff9d9d640 1 -- start start 2023-10-30T12:51:29.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff9d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ff406f7a0 0x7f5ff406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff9d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5ff40c6bd0 0x7f5ff4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff9d9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ff4071110 0x7f5ff4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff9d9d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff40700c0 con 0x7f5ff406f7a0 2023-10-30T12:51:29.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff9d9d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ff406a3d0 con 0x7f5ff40c6bd0 2023-10-30T12:51:29.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff9d9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff4066cf0 con 0x7f5ff4071110 2023-10-30T12:51:29.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ff406f7a0 0x7f5ff406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ff406f7a0 0x7f5ff406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60748/0 (socket says 172.21.15.125:60748) 2023-10-30T12:51:29.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff37fe640 1 -- 172.21.15.125:0/3095373702 learned_addr learned my addr 172.21.15.125:0/3095373702 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:29.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff3fff640 1 --2- 172.21.15.125:0/3095373702 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ff4071110 0x7f5ff4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5febfff640 1 --2- 172.21.15.125:0/3095373702 >> v2:172.21.15.136:3300/0 conn(0x7f5ff40c6bd0 0x7f5ff4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff37fe640 1 -- 172.21.15.125:0/3095373702 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ff4071110 msgr2=0x7f5ff4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:29.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff37fe640 1 --2- 172.21.15.125:0/3095373702 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ff4071110 0x7f5ff4158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff37fe640 1 -- 172.21.15.125:0/3095373702 >> v2:172.21.15.136:3300/0 conn(0x7f5ff40c6bd0 msgr2=0x7f5ff4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:29.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff37fe640 1 --2- 172.21.15.125:0/3095373702 >> v2:172.21.15.136:3300/0 conn(0x7f5ff40c6bd0 0x7f5ff4069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff37fe640 1 -- 172.21.15.125:0/3095373702 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff40c70d0 con 0x7f5ff406f7a0 2023-10-30T12:51:29.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.519+0000 7f5ff3fff640 1 --2- 172.21.15.125:0/3095373702 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ff4071110 0x7f5ff4158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:29.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff37fe640 1 --2- 172.21.15.125:0/3095373702 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ff406f7a0 0x7f5ff406fb80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f5ff406f1c0 tx=0x7f5fe401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=500278496445ba6a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:29.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff2ffd640 1 -- 172.21.15.125:0/3095373702 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe40a89b0 con 0x7f5ff406f7a0 2023-10-30T12:51:29.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff2ffd640 1 -- 172.21.15.125:0/3095373702 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe40a8b10 con 0x7f5ff406f7a0 2023-10-30T12:51:29.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff2ffd640 1 -- 172.21.15.125:0/3095373702 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe40a8ea0 con 0x7f5ff406f7a0 2023-10-30T12:51:29.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/3095373702 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ff406f7a0 msgr2=0x7f5ff406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff9d9d640 1 --2- 172.21.15.125:0/3095373702 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ff406f7a0 0x7f5ff406fb80 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f5ff406f1c0 tx=0x7f5fe401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/3095373702 shutdown_connections 2023-10-30T12:51:29.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff9d9d640 1 --2- 172.21.15.125:0/3095373702 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ff4071110 0x7f5ff4158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff9d9d640 1 --2- 172.21.15.125:0/3095373702 >> v2:172.21.15.136:3300/0 conn(0x7f5ff40c6bd0 0x7f5ff4069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff9d9d640 1 --2- 172.21.15.125:0/3095373702 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ff406f7a0 0x7f5ff406fb80 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/3095373702 >> 172.21.15.125:0/3095373702 conn(0x7f5ff40c7910 msgr2=0x7f5ff40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:29.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/3095373702 shutdown_connections 2023-10-30T12:51:29.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/3095373702 wait complete. 2023-10-30T12:51:29.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff9d9d640 1 Processor -- start 2023-10-30T12:51:29.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.523+0000 7f5ff9d9d640 1 -- start start 2023-10-30T12:51:29.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff9d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ff406f7a0 0x7f5ff41f32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff9d9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ff4071110 0x7f5ff4153610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff9d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ff40c6bd0 0x7f5ff41ef1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff9d9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff4153b50 con 0x7f5ff406f7a0 2023-10-30T12:51:29.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff9d9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff41ef700 con 0x7f5ff40c6bd0 2023-10-30T12:51:29.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff9d9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff40af2e0 con 0x7f5ff4071110 2023-10-30T12:51:29.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ff406f7a0 0x7f5ff41f32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ff406f7a0 0x7f5ff41f32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60764/0 (socket says 172.21.15.125:60764) 2023-10-30T12:51:29.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff37fe640 1 -- 172.21.15.125:0/1841874285 learned_addr learned my addr 172.21.15.125:0/1841874285 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:29.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5febfff640 1 --2- 172.21.15.125:0/1841874285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ff4071110 0x7f5ff4153610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff3fff640 1 --2- 172.21.15.125:0/1841874285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ff40c6bd0 0x7f5ff41ef1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff37fe640 1 -- 172.21.15.125:0/1841874285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ff4071110 msgr2=0x7f5ff4153610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff37fe640 1 --2- 172.21.15.125:0/1841874285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ff4071110 0x7f5ff4153610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff37fe640 1 -- 172.21.15.125:0/1841874285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ff40c6bd0 msgr2=0x7f5ff41ef1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff37fe640 1 --2- 172.21.15.125:0/1841874285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ff40c6bd0 0x7f5ff41ef1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff37fe640 1 -- 172.21.15.125:0/1841874285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe40094d0 con 0x7f5ff406f7a0 2023-10-30T12:51:29.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff37fe640 1 --2- 172.21.15.125:0/1841874285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ff406f7a0 0x7f5ff41f32b0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f5fe401d3a0 tx=0x7f5fe401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:29.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff0ff9640 1 -- 172.21.15.125:0/1841874285 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe40ba600 con 0x7f5ff406f7a0 2023-10-30T12:51:29.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff0ff9640 1 -- 172.21.15.125:0/1841874285 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe40ba760 con 0x7f5ff406f7a0 2023-10-30T12:51:29.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff0ff9640 1 -- 172.21.15.125:0/1841874285 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe40bab10 con 0x7f5ff406f7a0 2023-10-30T12:51:29.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/1841874285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff40af580 con 0x7f5ff406f7a0 2023-10-30T12:51:29.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/1841874285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff41f27d0 con 0x7f5ff406f7a0 2023-10-30T12:51:29.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.527+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/1841874285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff404e170 con 0x7f5ff406f7a0 2023-10-30T12:51:29.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.531+0000 7f5ff0ff9640 1 -- 172.21.15.125:0/1841874285 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5fe40bac70 con 0x7f5ff406f7a0 2023-10-30T12:51:29.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.531+0000 7f5ff0ff9640 1 --2- 172.21.15.125:0/1841874285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd40664a0 0x7f5fd4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:29.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.531+0000 7f5ff0ff9640 1 -- 172.21.15.125:0/1841874285 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7f5fe4122840 con 0x7f5ff406f7a0 2023-10-30T12:51:29.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.531+0000 7f5febfff640 1 --2- 172.21.15.125:0/1841874285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd40664a0 0x7f5fd4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:29.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.531+0000 7f5febfff640 1 --2- 172.21.15.125:0/1841874285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd40664a0 0x7f5fd4068960 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5ff41564c0 tx=0x7f5fdc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:29.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.535+0000 7f5ff0ff9640 1 -- 172.21.15.125:0/1841874285 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5fe40ef850 con 0x7f5ff406f7a0 2023-10-30T12:51:29.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.715+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/1841874285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ff4070620 con 0x7f5ff406f7a0 2023-10-30T12:51:29.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.715+0000 7f5ff0ff9640 1 -- 172.21.15.125:0/1841874285 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f5fe40f2e70 con 0x7f5ff406f7a0 2023-10-30T12:51:29.715 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:29.716 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:25.028391+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:reconnect","state_seq":39,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:29.716 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 113 2023-10-30T12:51:29.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.715+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/1841874285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd40664a0 msgr2=0x7f5fd4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.715+0000 7f5ff9d9d640 1 --2- 172.21.15.125:0/1841874285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd40664a0 0x7f5fd4068960 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f5ff41564c0 tx=0x7f5fdc008040 comp rx=0 tx=0).stop 2023-10-30T12:51:29.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.715+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/1841874285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ff406f7a0 msgr2=0x7f5ff41f32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:29.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.715+0000 7f5ff9d9d640 1 --2- 172.21.15.125:0/1841874285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ff406f7a0 0x7f5ff41f32b0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f5fe401d3a0 tx=0x7f5fe401ba90 comp rx=0 tx=0).stop 2023-10-30T12:51:29.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.719+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/1841874285 shutdown_connections 2023-10-30T12:51:29.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.719+0000 7f5ff9d9d640 1 --2- 172.21.15.125:0/1841874285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd40664a0 0x7f5fd4068960 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.719+0000 7f5ff9d9d640 1 --2- 172.21.15.125:0/1841874285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ff40c6bd0 0x7f5ff41ef1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.719+0000 7f5ff9d9d640 1 --2- 172.21.15.125:0/1841874285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ff4071110 0x7f5ff4153610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.719+0000 7f5ff9d9d640 1 --2- 172.21.15.125:0/1841874285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ff406f7a0 0x7f5ff41f32b0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:29.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.719+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/1841874285 >> 172.21.15.125:0/1841874285 conn(0x7f5ff40c7910 msgr2=0x7f5ff4156bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:29.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.719+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/1841874285 shutdown_connections 2023-10-30T12:51:29.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:29.719+0000 7f5ff9d9d640 1 -- 172.21.15.125:0/1841874285 wait complete. 2023-10-30T12:51:29.737 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 113, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 113, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6627": { "addr": "172.21.15.125:6837/2408701650", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2408701650, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2408701650, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6627, "incarnation": 112, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 39 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:51:25.028391+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6627 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:51:31.738 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:31.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.883+0000 7f8f1511d640 1 Processor -- start 2023-10-30T12:51:31.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.883+0000 7f8f1511d640 1 -- start start 2023-10-30T12:51:31.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.883+0000 7f8f1511d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8f100c6f60 0x7f8f101566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.883+0000 7f8f1511d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f100c0a50 con 0x7f8f100c6b80 2023-10-30T12:51:31.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.883+0000 7f8f1511d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f1006f6e0 con 0x7f8f100c05d0 2023-10-30T12:51:31.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.883+0000 7f8f1511d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f1006f9c0 con 0x7f8f100c6f60 2023-10-30T12:51:31.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.883+0000 7f8f0ed9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8f100c6b80 0x7f8f100bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48816/0 (socket says 172.21.15.125:48816) 2023-10-30T12:51:31.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.883+0000 7f8f0ed9d640 1 -- 172.21.15.125:0/214437769 learned_addr learned my addr 172.21.15.125:0/214437769 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:51:31.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0f59e640 1 --2- 172.21.15.125:0/214437769 >> v2:172.21.15.136:3300/0 conn(0x7f8f100c6f60 0x7f8f101566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 -- 172.21.15.125:0/214437769 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120528729 0 0) 0x7f8f100c0a50 con 0x7f8f100c6b80 2023-10-30T12:51:31.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 -- 172.21.15.125:0/214437769 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ef8003530 con 0x7f8f100c6b80 2023-10-30T12:51:31.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 -- 172.21.15.125:0/214437769 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1586901983 0 0) 0x7f8ef8003530 con 0x7f8f100c6b80 2023-10-30T12:51:31.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 -- 172.21.15.125:0/214437769 >> v2:172.21.15.136:3300/0 conn(0x7f8f100c6f60 msgr2=0x7f8f101566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:31.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 --2- 172.21.15.125:0/214437769 >> v2:172.21.15.136:3300/0 conn(0x7f8f100c6f60 0x7f8f101566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 -- 172.21.15.125:0/214437769 >> v1:172.21.15.136:6789/0 conn(0x7f8f100c05d0 legacy=0x7f8f1013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 -- 172.21.15.125:0/214437769 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f1006fca0 con 0x7f8f100c6b80 2023-10-30T12:51:31.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 -- 172.21.15.125:0/214437769 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f8f00001680 con 0x7f8f100c6b80 2023-10-30T12:51:31.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 -- 172.21.15.125:0/214437769 >> v1:172.21.15.125:6789/0 conn(0x7f8f100c6b80 legacy=0x7f8f100bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 --2- 172.21.15.125:0/214437769 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ef8003980 0x7f8ef8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 --2- 172.21.15.125:0/214437769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ef80047d0 0x7f8ef8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 --2- 172.21.15.125:0/214437769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ef8005190 0x7f8ef800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 -- 172.21.15.125:0/214437769 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f1006f9c0 con 0x7f8ef80047d0 2023-10-30T12:51:31.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 -- 172.21.15.125:0/214437769 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef800c030 con 0x7f8ef8005190 2023-10-30T12:51:31.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0dd9b640 1 -- 172.21.15.125:0/214437769 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef800c1f0 con 0x7f8ef8003980 2023-10-30T12:51:31.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0e59c640 1 --2- 172.21.15.125:0/214437769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ef80047d0 0x7f8ef8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0ed9d640 1 --2- 172.21.15.125:0/214437769 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ef8003980 0x7f8ef8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0f59e640 1 --2- 172.21.15.125:0/214437769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ef8005190 0x7f8ef800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0e59c640 1 -- 172.21.15.125:0/214437769 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ef8003980 msgr2=0x7f8ef8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:31.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0e59c640 1 --2- 172.21.15.125:0/214437769 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ef8003980 0x7f8ef8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0e59c640 1 -- 172.21.15.125:0/214437769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ef8005190 msgr2=0x7f8ef800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:31.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0e59c640 1 --2- 172.21.15.125:0/214437769 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ef8005190 0x7f8ef800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0e59c640 1 -- 172.21.15.125:0/214437769 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef8003530 con 0x7f8ef80047d0 2023-10-30T12:51:31.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.887+0000 7f8f0f59e640 1 -- 172.21.15.125:0/214437769 reap_dead start 2023-10-30T12:51:31.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.891+0000 7f8f0e59c640 1 --2- 172.21.15.125:0/214437769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ef80047d0 0x7f8ef8004c50 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8f04001750 tx=0x7f8f0401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fc38a2ffe3d4694 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:31.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.891+0000 7f8f0dd9b640 1 -- 172.21.15.125:0/214437769 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f040052d0 con 0x7f8ef80047d0 2023-10-30T12:51:31.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.891+0000 7f8f0dd9b640 1 -- 172.21.15.125:0/214437769 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f040a7050 con 0x7f8ef80047d0 2023-10-30T12:51:31.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.891+0000 7f8f0dd9b640 1 -- 172.21.15.125:0/214437769 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04003600 con 0x7f8ef80047d0 2023-10-30T12:51:31.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.891+0000 7f8f1511d640 1 -- 172.21.15.125:0/214437769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ef80047d0 msgr2=0x7f8ef8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.891+0000 7f8f1511d640 1 --2- 172.21.15.125:0/214437769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ef80047d0 0x7f8ef8004c50 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f8f04001750 tx=0x7f8f0401c3a0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.891+0000 7f8f1511d640 1 -- 172.21.15.125:0/214437769 shutdown_connections 2023-10-30T12:51:31.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.891+0000 7f8f1511d640 1 --2- 172.21.15.125:0/214437769 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ef80047d0 0x7f8ef8004c50 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.891+0000 7f8f1511d640 1 -- 172.21.15.125:0/214437769 >> 172.21.15.125:0/214437769 conn(0x7f8f100c7910 msgr2=0x7f8f100c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:31.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.891+0000 7f8f1511d640 1 -- 172.21.15.125:0/214437769 shutdown_connections 2023-10-30T12:51:31.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.891+0000 7f8f1511d640 1 -- 172.21.15.125:0/214437769 wait complete. 2023-10-30T12:51:31.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.891+0000 7f8f1511d640 1 Processor -- start 2023-10-30T12:51:31.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.891+0000 7f8f1511d640 1 -- start start 2023-10-30T12:51:31.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f1511d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ef80047d0 0x7f8f1013a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f1511d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f1013e420 0x7f8f10157790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f1511d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f101f2690 0x7f8f101f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f1511d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101f2270 con 0x7f8ef80047d0 2023-10-30T12:51:31.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f1511d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101efb00 con 0x7f8f1013e420 2023-10-30T12:51:31.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f1511d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f101f3050 con 0x7f8f101f2690 2023-10-30T12:51:31.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f0ed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ef80047d0 0x7f8f1013a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f0ed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ef80047d0 0x7f8f1013a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51278/0 (socket says 172.21.15.125:51278) 2023-10-30T12:51:31.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f0ed9d640 1 -- 172.21.15.125:0/433682772 learned_addr learned my addr 172.21.15.125:0/433682772 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:31.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f0f59e640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f101f2690 0x7f8f101f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f0e59c640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f1013e420 0x7f8f10157790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f0ed9d640 1 -- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f101f2690 msgr2=0x7f8f101f2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f0ed9d640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f101f2690 0x7f8f101f2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f0ed9d640 1 -- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f1013e420 msgr2=0x7f8f10157790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:31.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f0ed9d640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f1013e420 0x7f8f10157790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:31.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f0ed9d640 1 -- 172.21.15.125:0/433682772 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f100aff90 con 0x7f8ef80047d0 2023-10-30T12:51:31.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f0f59e640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f101f2690 0x7f8f101f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:31.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f0e59c640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f1013e420 0x7f8f10157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:31.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f0ed9d640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ef80047d0 0x7f8f1013a710 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f8f000017b0 tx=0x7f8f00012920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:31.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8ef77fe640 1 -- 172.21.15.125:0/433682772 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00014810 con 0x7f8ef80047d0 2023-10-30T12:51:31.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8ef77fe640 1 -- 172.21.15.125:0/433682772 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f00014970 con 0x7f8ef80047d0 2023-10-30T12:51:31.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8ef77fe640 1 -- 172.21.15.125:0/433682772 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f00014ce0 con 0x7f8ef80047d0 2023-10-30T12:51:31.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f1511d640 1 -- 172.21.15.125:0/433682772 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f100b0510 con 0x7f8ef80047d0 2023-10-30T12:51:31.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f1511d640 1 -- 172.21.15.125:0/433682772 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f100b0230 con 0x7f8ef80047d0 2023-10-30T12:51:31.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8f1511d640 1 -- 172.21.15.125:0/433682772 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f101f23b0 con 0x7f8ef80047d0 2023-10-30T12:51:31.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.895+0000 7f8ef77fe640 1 -- 172.21.15.125:0/433682772 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f000153e0 con 0x7f8ef80047d0 2023-10-30T12:51:31.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.899+0000 7f8ef77fe640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8efc0664d0 0x7f8efc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:31.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.899+0000 7f8ef77fe640 1 -- 172.21.15.125:0/433682772 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7f8f00086c20 con 0x7f8ef80047d0 2023-10-30T12:51:31.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.899+0000 7f8f0e59c640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8efc0664d0 0x7f8efc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:31.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.899+0000 7f8f0e59c640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8efc0664d0 0x7f8efc068990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8f1006c9a0 tx=0x7f8f040011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:31.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:31.903+0000 7f8ef77fe640 1 -- 172.21.15.125:0/433682772 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8f101f23b0 con 0x7f8ef80047d0 2023-10-30T12:51:32.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:32.079+0000 7f8f1511d640 1 -- 172.21.15.125:0/433682772 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f101f23b0 con 0x7f8ef80047d0 2023-10-30T12:51:32.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:32.079+0000 7f8ef77fe640 1 -- 172.21.15.125:0/433682772 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f8f101f23b0 con 0x7f8ef80047d0 2023-10-30T12:51:32.081 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:32.081 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:30.421349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:32.082 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 114 2023-10-30T12:51:32.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:32.083+0000 7f8f1511d640 1 -- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8efc0664d0 msgr2=0x7f8efc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:32.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:32.083+0000 7f8f1511d640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8efc0664d0 0x7f8efc068990 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f8f1006c9a0 tx=0x7f8f040011a0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:32.083+0000 7f8f1511d640 1 -- 172.21.15.125:0/433682772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ef80047d0 msgr2=0x7f8f1013a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:32.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:32.083+0000 7f8f1511d640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ef80047d0 0x7f8f1013a710 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f8f000017b0 tx=0x7f8f00012920 comp rx=0 tx=0).stop 2023-10-30T12:51:32.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:32.083+0000 7f8f1511d640 1 -- 172.21.15.125:0/433682772 shutdown_connections 2023-10-30T12:51:32.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:32.083+0000 7f8f1511d640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8efc0664d0 0x7f8efc068990 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:32.083+0000 7f8f1511d640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f101f2690 0x7f8f101f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:32.083+0000 7f8f1511d640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f1013e420 0x7f8f10157790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:32.083+0000 7f8f1511d640 1 --2- 172.21.15.125:0/433682772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ef80047d0 0x7f8f1013a710 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:32.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:32.083+0000 7f8f1511d640 1 -- 172.21.15.125:0/433682772 >> 172.21.15.125:0/433682772 conn(0x7f8f100c7910 msgr2=0x7f8f100b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:32.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:32.083+0000 7f8f1511d640 1 -- 172.21.15.125:0/433682772 shutdown_connections 2023-10-30T12:51:32.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:32.083+0000 7f8f1511d640 1 -- 172.21.15.125:0/433682772 wait complete. 2023-10-30T12:51:32.101 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:51:34.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:34.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.247+0000 7f10dbb0f640 1 Processor -- start 2023-10-30T12:51:34.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.247+0000 7f10dbb0f640 1 -- start start 2023-10-30T12:51:34.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.251+0000 7f10dbb0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f10d40c6b80 0x7f10d40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.251+0000 7f10dbb0f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f10d40c6f60 0x7f10d40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.251+0000 7f10dbb0f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d40c0a50 con 0x7f10d40c8430 2023-10-30T12:51:34.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.251+0000 7f10dbb0f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d406f6e0 con 0x7f10d40c6f60 2023-10-30T12:51:34.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.251+0000 7f10dbb0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d406f9c0 con 0x7f10d40c6b80 2023-10-30T12:51:34.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.251+0000 7f10d90aa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f10d40c6f60 0x7f10d40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.251+0000 7f10d90aa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f10d40c6f60 0x7f10d40c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51288/0 (socket says 172.21.15.125:51288) 2023-10-30T12:51:34.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.251+0000 7f10d90aa640 1 -- 172.21.15.125:0/1822750155 learned_addr learned my addr 172.21.15.125:0/1822750155 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:34.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.251+0000 7f10d98ab640 1 --2- 172.21.15.125:0/1822750155 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f10d40c6b80 0x7f10d40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.251+0000 7f10d90aa640 1 -- 172.21.15.125:0/1822750155 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f10d40c6b80 msgr2=0x7f10d40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:34.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.251+0000 7f10d90aa640 1 --2- 172.21.15.125:0/1822750155 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f10d40c6b80 0x7f10d40bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.251+0000 7f10d90aa640 1 -- 172.21.15.125:0/1822750155 >> v1:172.21.15.136:6789/0 conn(0x7f10d40c8430 legacy=0x7f10d413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.251+0000 7f10d90aa640 1 -- 172.21.15.125:0/1822750155 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d406fca0 con 0x7f10d40c6f60 2023-10-30T12:51:34.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.255+0000 7f10d90aa640 1 --2- 172.21.15.125:0/1822750155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d40c6f60 0x7f10d40c0400 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f10c4009870 tx=0x7f10c401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ad0a8af86c965b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.255+0000 7f10d88a9640 1 -- 172.21.15.125:0/1822750155 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c40ab650 con 0x7f10d40c6f60 2023-10-30T12:51:34.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.255+0000 7f10d88a9640 1 -- 172.21.15.125:0/1822750155 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c40ab7b0 con 0x7f10d40c6f60 2023-10-30T12:51:34.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.255+0000 7f10d88a9640 1 -- 172.21.15.125:0/1822750155 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10c40abb40 con 0x7f10d40c6f60 2023-10-30T12:51:34.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.255+0000 7f10dbb0f640 1 -- 172.21.15.125:0/1822750155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d40c6f60 msgr2=0x7f10d40c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.255+0000 7f10dbb0f640 1 --2- 172.21.15.125:0/1822750155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d40c6f60 0x7f10d40c0400 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f10c4009870 tx=0x7f10c401c540 comp rx=0 tx=0).stop 2023-10-30T12:51:34.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.255+0000 7f10dbb0f640 1 -- 172.21.15.125:0/1822750155 shutdown_connections 2023-10-30T12:51:34.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.255+0000 7f10dbb0f640 1 --2- 172.21.15.125:0/1822750155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d40c6f60 0x7f10d40c0400 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.255+0000 7f10dbb0f640 1 --2- 172.21.15.125:0/1822750155 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f10d40c6b80 0x7f10d40bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.255+0000 7f10dbb0f640 1 -- 172.21.15.125:0/1822750155 >> 172.21.15.125:0/1822750155 conn(0x7f10d40c7910 msgr2=0x7f10d40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:34.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.255+0000 7f10dbb0f640 1 -- 172.21.15.125:0/1822750155 shutdown_connections 2023-10-30T12:51:34.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.255+0000 7f10dbb0f640 1 -- 172.21.15.125:0/1822750155 wait complete. 2023-10-30T12:51:34.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.255+0000 7f10dbb0f640 1 Processor -- start 2023-10-30T12:51:34.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.255+0000 7f10dbb0f640 1 -- start start 2023-10-30T12:51:34.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10dbb0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10d40c6b80 0x7f10d4150630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10dbb0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f10d40c6f60 0x7f10d414ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10dbb0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d40c8430 0x7f10d414f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10dbb0f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d4158ec0 con 0x7f10d40c8430 2023-10-30T12:51:34.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10dbb0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d414e640 con 0x7f10d40c6b80 2023-10-30T12:51:34.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10dbb0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d414e050 con 0x7f10d40c6f60 2023-10-30T12:51:34.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10da0ac640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d40c8430 0x7f10d414f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10da0ac640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d40c8430 0x7f10d414f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51304/0 (socket says 172.21.15.125:51304) 2023-10-30T12:51:34.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10da0ac640 1 -- 172.21.15.125:0/56744975 learned_addr learned my addr 172.21.15.125:0/56744975 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:34.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10d90aa640 1 --2- 172.21.15.125:0/56744975 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f10d40c6f60 0x7f10d414ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10d98ab640 1 --2- 172.21.15.125:0/56744975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10d40c6b80 0x7f10d4150630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10da0ac640 1 -- 172.21.15.125:0/56744975 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f10d40c6f60 msgr2=0x7f10d414ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10da0ac640 1 --2- 172.21.15.125:0/56744975 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f10d40c6f60 0x7f10d414ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10da0ac640 1 -- 172.21.15.125:0/56744975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10d40c6b80 msgr2=0x7f10d4150630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10da0ac640 1 --2- 172.21.15.125:0/56744975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10d40c6b80 0x7f10d4150630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10da0ac640 1 -- 172.21.15.125:0/56744975 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c40094d0 con 0x7f10d40c8430 2023-10-30T12:51:34.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10da0ac640 1 --2- 172.21.15.125:0/56744975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d40c8430 0x7f10d414f010 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00c760 tx=0x7f10cc00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10ca7fc640 1 -- 172.21.15.125:0/56744975 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc003b80 con 0x7f10d40c8430 2023-10-30T12:51:34.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10ca7fc640 1 -- 172.21.15.125:0/56744975 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc003ce0 con 0x7f10d40c8430 2023-10-30T12:51:34.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10ca7fc640 1 -- 172.21.15.125:0/56744975 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc00aa30 con 0x7f10d40c8430 2023-10-30T12:51:34.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10dbb0f640 1 -- 172.21.15.125:0/56744975 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d4153cf0 con 0x7f10d40c8430 2023-10-30T12:51:34.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10dbb0f640 1 -- 172.21.15.125:0/56744975 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d414e2d0 con 0x7f10d40c8430 2023-10-30T12:51:34.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10dbb0f640 1 -- 172.21.15.125:0/56744975 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d414e7b0 con 0x7f10d40c8430 2023-10-30T12:51:34.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.259+0000 7f10ca7fc640 1 -- 172.21.15.125:0/56744975 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f10cc00ab90 con 0x7f10d40c8430 2023-10-30T12:51:34.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.263+0000 7f10ca7fc640 1 --2- 172.21.15.125:0/56744975 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f10b80664a0 0x7f10b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:34.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.263+0000 7f10ca7fc640 1 -- 172.21.15.125:0/56744975 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7f10cc083e30 con 0x7f10d40c8430 2023-10-30T12:51:34.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.263+0000 7f10d98ab640 1 --2- 172.21.15.125:0/56744975 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f10b80664a0 0x7f10b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:34.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.263+0000 7f10d98ab640 1 --2- 172.21.15.125:0/56744975 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f10b80664a0 0x7f10b8068960 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f10d0009b60 tx=0x7f10d0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:34.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.267+0000 7f10ca7fc640 1 -- 172.21.15.125:0/56744975 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f10cc051020 con 0x7f10d40c8430 2023-10-30T12:51:34.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.447+0000 7f10dbb0f640 1 -- 172.21.15.125:0/56744975 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10d404e170 con 0x7f10d40c8430 2023-10-30T12:51:34.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.447+0000 7f10ca7fc640 1 -- 172.21.15.125:0/56744975 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f10cc054640 con 0x7f10d40c8430 2023-10-30T12:51:34.447 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:34.448 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:30.421349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:34.448 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 114 2023-10-30T12:51:34.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.447+0000 7f10dbb0f640 1 -- 172.21.15.125:0/56744975 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f10b80664a0 msgr2=0x7f10b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.447+0000 7f10dbb0f640 1 --2- 172.21.15.125:0/56744975 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f10b80664a0 0x7f10b8068960 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f10d0009b60 tx=0x7f10d0009190 comp rx=0 tx=0).stop 2023-10-30T12:51:34.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.447+0000 7f10dbb0f640 1 -- 172.21.15.125:0/56744975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d40c8430 msgr2=0x7f10d414f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:34.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.447+0000 7f10dbb0f640 1 --2- 172.21.15.125:0/56744975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d40c8430 0x7f10d414f010 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00c760 tx=0x7f10cc00cc20 comp rx=0 tx=0).stop 2023-10-30T12:51:34.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.451+0000 7f10dbb0f640 1 -- 172.21.15.125:0/56744975 shutdown_connections 2023-10-30T12:51:34.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.451+0000 7f10dbb0f640 1 --2- 172.21.15.125:0/56744975 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f10b80664a0 0x7f10b8068960 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.451+0000 7f10dbb0f640 1 --2- 172.21.15.125:0/56744975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f10d40c8430 0x7f10d414f010 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.451+0000 7f10dbb0f640 1 --2- 172.21.15.125:0/56744975 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f10d40c6f60 0x7f10d414ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.451+0000 7f10dbb0f640 1 --2- 172.21.15.125:0/56744975 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f10d40c6b80 0x7f10d4150630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:34.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.451+0000 7f10dbb0f640 1 -- 172.21.15.125:0/56744975 >> 172.21.15.125:0/56744975 conn(0x7f10d40c7910 msgr2=0x7f10d406a170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:34.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.451+0000 7f10dbb0f640 1 -- 172.21.15.125:0/56744975 shutdown_connections 2023-10-30T12:51:34.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:34.451+0000 7f10dbb0f640 1 -- 172.21.15.125:0/56744975 wait complete. 2023-10-30T12:51:34.468 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:51:36.468 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:36.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.607+0000 7f2cc926b640 1 Processor -- start 2023-10-30T12:51:36.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.607+0000 7f2cc926b640 1 -- start start 2023-10-30T12:51:36.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.607+0000 7f2cc926b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2cc4153d70 0x7f2cc406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.607+0000 7f2cc926b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2cc40c8490 0x7f2cc40c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.607+0000 7f2cc926b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc406afa0 con 0x7f2cc406e470 2023-10-30T12:51:36.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.607+0000 7f2cc926b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc406d780 con 0x7f2cc40c8490 2023-10-30T12:51:36.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.607+0000 7f2cc926b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc4154150 con 0x7f2cc4153d70 2023-10-30T12:51:36.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.607+0000 7f2cc37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2cc40c8490 0x7f2cc40c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.611+0000 7f2cba7fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2cc406e470 0x7f2cc40cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48832/0 (socket says 172.21.15.125:48832) 2023-10-30T12:51:36.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.611+0000 7f2cba7fc640 1 -- 172.21.15.125:0/1243360450 learned_addr learned my addr 172.21.15.125:0/1243360450 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:51:36.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.611+0000 7f2cc2ffd640 1 --2- 172.21.15.125:0/1243360450 >> v2:172.21.15.136:3300/0 conn(0x7f2cc4153d70 0x7f2cc406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.611+0000 7f2cc37fe640 1 -- 172.21.15.125:0/1243360450 >> v2:172.21.15.136:3300/0 conn(0x7f2cc4153d70 msgr2=0x7f2cc406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:36.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.611+0000 7f2cc37fe640 1 --2- 172.21.15.125:0/1243360450 >> v2:172.21.15.136:3300/0 conn(0x7f2cc4153d70 0x7f2cc406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.611+0000 7f2cc37fe640 1 -- 172.21.15.125:0/1243360450 >> v1:172.21.15.125:6789/0 conn(0x7f2cc406e470 legacy=0x7f2cc40cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.611+0000 7f2cc37fe640 1 -- 172.21.15.125:0/1243360450 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc406e030 con 0x7f2cc40c8490 2023-10-30T12:51:36.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.611+0000 7f2cc37fe640 1 --2- 172.21.15.125:0/1243360450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40c8490 0x7f2cc40c8cc0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f2cc406d180 tx=0x7f2cbc01c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a1b0b80e2b6bfe8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.611+0000 7f2cc27fc640 1 -- 172.21.15.125:0/1243360450 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc0a59a0 con 0x7f2cc40c8490 2023-10-30T12:51:36.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.611+0000 7f2cc27fc640 1 -- 172.21.15.125:0/1243360450 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cbc0a5b00 con 0x7f2cc40c8490 2023-10-30T12:51:36.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.611+0000 7f2cc27fc640 1 -- 172.21.15.125:0/1243360450 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc0a5e90 con 0x7f2cc40c8490 2023-10-30T12:51:36.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 -- 172.21.15.125:0/1243360450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40c8490 msgr2=0x7f2cc40c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 --2- 172.21.15.125:0/1243360450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40c8490 0x7f2cc40c8cc0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f2cc406d180 tx=0x7f2cbc01c8e0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 -- 172.21.15.125:0/1243360450 shutdown_connections 2023-10-30T12:51:36.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 --2- 172.21.15.125:0/1243360450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40c8490 0x7f2cc40c8cc0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 --2- 172.21.15.125:0/1243360450 >> v2:172.21.15.136:3300/0 conn(0x7f2cc4153d70 0x7f2cc406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 -- 172.21.15.125:0/1243360450 >> 172.21.15.125:0/1243360450 conn(0x7f2cc4071ea0 msgr2=0x7f2cc4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:36.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 -- 172.21.15.125:0/1243360450 shutdown_connections 2023-10-30T12:51:36.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 -- 172.21.15.125:0/1243360450 wait complete. 2023-10-30T12:51:36.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 Processor -- start 2023-10-30T12:51:36.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 -- start start 2023-10-30T12:51:36.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cc406e470 0x7f2cc4154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40c8490 0x7f2cc41f65c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cc4153d70 0x7f2cc41468a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc41f6b00 con 0x7f2cc40c8490 2023-10-30T12:51:36.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc4154b90 con 0x7f2cc406e470 2023-10-30T12:51:36.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc926b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc4156e60 con 0x7f2cc4153d70 2023-10-30T12:51:36.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cba7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40c8490 0x7f2cc41f65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cba7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40c8490 0x7f2cc41f65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51328/0 (socket says 172.21.15.125:51328) 2023-10-30T12:51:36.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cba7fc640 1 -- 172.21.15.125:0/2042896229 learned_addr learned my addr 172.21.15.125:0/2042896229 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:36.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc37fe640 1 --2- 172.21.15.125:0/2042896229 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cc4153d70 0x7f2cc41468a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cc2ffd640 1 --2- 172.21.15.125:0/2042896229 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cc406e470 0x7f2cc4154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cba7fc640 1 -- 172.21.15.125:0/2042896229 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cc4153d70 msgr2=0x7f2cc41468a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cba7fc640 1 --2- 172.21.15.125:0/2042896229 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cc4153d70 0x7f2cc41468a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cba7fc640 1 -- 172.21.15.125:0/2042896229 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cc406e470 msgr2=0x7f2cc4154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cba7fc640 1 --2- 172.21.15.125:0/2042896229 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cc406e470 0x7f2cc4154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cba7fc640 1 -- 172.21.15.125:0/2042896229 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc009a70 con 0x7f2cc40c8490 2023-10-30T12:51:36.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.615+0000 7f2cba7fc640 1 --2- 172.21.15.125:0/2042896229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40c8490 0x7f2cc41f65c0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400c400 tx=0x7f2cb400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.619+0000 7f2cbbfff640 1 -- 172.21.15.125:0/2042896229 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4005cf0 con 0x7f2cc40c8490 2023-10-30T12:51:36.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.619+0000 7f2cbbfff640 1 -- 172.21.15.125:0/2042896229 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb4005e50 con 0x7f2cc40c8490 2023-10-30T12:51:36.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.619+0000 7f2cc926b640 1 -- 172.21.15.125:0/2042896229 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc41548b0 con 0x7f2cc40c8490 2023-10-30T12:51:36.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.619+0000 7f2cbbfff640 1 -- 172.21.15.125:0/2042896229 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb40072d0 con 0x7f2cc40c8490 2023-10-30T12:51:36.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.619+0000 7f2cc926b640 1 -- 172.21.15.125:0/2042896229 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc4155c10 con 0x7f2cc40c8490 2023-10-30T12:51:36.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.619+0000 7f2cc926b640 1 -- 172.21.15.125:0/2042896229 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc404e170 con 0x7f2cc40c8490 2023-10-30T12:51:36.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.619+0000 7f2cbbfff640 1 -- 172.21.15.125:0/2042896229 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2cb4007430 con 0x7f2cc40c8490 2023-10-30T12:51:36.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.619+0000 7f2cbbfff640 1 --2- 172.21.15.125:0/2042896229 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ca40664a0 0x7f2ca4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:36.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.619+0000 7f2cbbfff640 1 -- 172.21.15.125:0/2042896229 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7f2cb40836e0 con 0x7f2cc40c8490 2023-10-30T12:51:36.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.619+0000 7f2cc2ffd640 1 --2- 172.21.15.125:0/2042896229 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ca40664a0 0x7f2ca4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:36.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.623+0000 7f2cc2ffd640 1 --2- 172.21.15.125:0/2042896229 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ca40664a0 0x7f2ca4068960 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00b9d0 tx=0x7f2cac008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:36.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.623+0000 7f2cbbfff640 1 -- 172.21.15.125:0/2042896229 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2cb4050840 con 0x7f2cc40c8490 2023-10-30T12:51:36.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.807+0000 7f2cc926b640 1 -- 172.21.15.125:0/2042896229 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cc4151030 con 0x7f2cc40c8490 2023-10-30T12:51:36.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.807+0000 7f2cbbfff640 1 -- 172.21.15.125:0/2042896229 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 114 v114) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f2cb4053e60 con 0x7f2cc40c8490 2023-10-30T12:51:36.806 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:36.806 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":114,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":114,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:30.421349+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:rejoin","state_seq":41,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:36.808 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 114 2023-10-30T12:51:36.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.807+0000 7f2cc926b640 1 -- 172.21.15.125:0/2042896229 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ca40664a0 msgr2=0x7f2ca4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.807+0000 7f2cc926b640 1 --2- 172.21.15.125:0/2042896229 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ca40664a0 0x7f2ca4068960 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2cac00b9d0 tx=0x7f2cac008040 comp rx=0 tx=0).stop 2023-10-30T12:51:36.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.807+0000 7f2cc926b640 1 -- 172.21.15.125:0/2042896229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40c8490 msgr2=0x7f2cc41f65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:36.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.807+0000 7f2cc926b640 1 --2- 172.21.15.125:0/2042896229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40c8490 0x7f2cc41f65c0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f2cb400c400 tx=0x7f2cb400c8c0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.807+0000 7f2cc926b640 1 -- 172.21.15.125:0/2042896229 shutdown_connections 2023-10-30T12:51:36.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.807+0000 7f2cc926b640 1 --2- 172.21.15.125:0/2042896229 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ca40664a0 0x7f2ca4068960 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.807+0000 7f2cc926b640 1 --2- 172.21.15.125:0/2042896229 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cc4153d70 0x7f2cc41468a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.807+0000 7f2cc926b640 1 --2- 172.21.15.125:0/2042896229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cc40c8490 0x7f2cc41f65c0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.807+0000 7f2cc926b640 1 --2- 172.21.15.125:0/2042896229 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cc406e470 0x7f2cc4154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:36.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.807+0000 7f2cc926b640 1 -- 172.21.15.125:0/2042896229 >> 172.21.15.125:0/2042896229 conn(0x7f2cc4071ea0 msgr2=0x7f2cc4143770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:36.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.807+0000 7f2cc926b640 1 -- 172.21.15.125:0/2042896229 shutdown_connections 2023-10-30T12:51:36.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:36.807+0000 7f2cc926b640 1 -- 172.21.15.125:0/2042896229 wait complete. 2023-10-30T12:51:36.827 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:51:38.828 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:38.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.975+0000 7ff637f3e640 1 Processor -- start 2023-10-30T12:51:38.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.975+0000 7ff637f3e640 1 -- start start 2023-10-30T12:51:38.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.975+0000 7ff637f3e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff6300cf160 0x7ff6300cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:38.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.975+0000 7ff637f3e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6300cb800 con 0x7ff6301538a0 2023-10-30T12:51:38.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.975+0000 7ff637f3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6300cb330 con 0x7ff63015d6f0 2023-10-30T12:51:38.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.975+0000 7ff637f3e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6300cad70 con 0x7ff6300cf160 2023-10-30T12:51:38.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.975+0000 7ff635cda640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff6301538a0 0x7ff6300ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48842/0 (socket says 172.21.15.125:48842) 2023-10-30T12:51:38.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.975+0000 7ff6364db640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff6300cf160 0x7ff6300cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:38.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.975+0000 7ff635cda640 1 -- 172.21.15.125:0/3153144419 learned_addr learned my addr 172.21.15.125:0/3153144419 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:51:38.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.975+0000 7ff6364db640 1 -- 172.21.15.125:0/3153144419 >> v1:172.21.15.136:6789/0 conn(0x7ff63015d6f0 legacy=0x7ff63015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:38.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.975+0000 7ff6364db640 1 -- 172.21.15.125:0/3153144419 >> v1:172.21.15.125:6789/0 conn(0x7ff6301538a0 legacy=0x7ff6300ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:38.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.975+0000 7ff6364db640 1 -- 172.21.15.125:0/3153144419 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6300cb050 con 0x7ff6300cf160 2023-10-30T12:51:38.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.975+0000 7ff634cd8640 1 -- 172.21.15.125:0/3153144419 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658140223 0 0) 0x7ff6300cb800 con 0x7ff6301538a0 2023-10-30T12:51:38.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.979+0000 7ff6364db640 1 --2- 172.21.15.125:0/3153144419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6300cf160 0x7ff6300cf590 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7ff628009d90 tx=0x7ff62801f100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c1303a678645b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:38.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.979+0000 7ff634cd8640 1 -- 172.21.15.125:0/3153144419 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6280af600 con 0x7ff6300cf160 2023-10-30T12:51:38.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.979+0000 7ff634cd8640 1 -- 172.21.15.125:0/3153144419 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6280ab050 con 0x7ff6300cf160 2023-10-30T12:51:38.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.979+0000 7ff634cd8640 1 -- 172.21.15.125:0/3153144419 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6280af980 con 0x7ff6300cf160 2023-10-30T12:51:38.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.979+0000 7ff637f3e640 1 -- 172.21.15.125:0/3153144419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6300cf160 msgr2=0x7ff6300cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:38.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.979+0000 7ff637f3e640 1 --2- 172.21.15.125:0/3153144419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6300cf160 0x7ff6300cf590 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7ff628009d90 tx=0x7ff62801f100 comp rx=0 tx=0).stop 2023-10-30T12:51:38.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.979+0000 7ff637f3e640 1 -- 172.21.15.125:0/3153144419 shutdown_connections 2023-10-30T12:51:38.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.979+0000 7ff637f3e640 1 --2- 172.21.15.125:0/3153144419 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6300cf160 0x7ff6300cf590 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:38.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.979+0000 7ff637f3e640 1 -- 172.21.15.125:0/3153144419 >> 172.21.15.125:0/3153144419 conn(0x7ff63006bfe0 msgr2=0x7ff63006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:38.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.979+0000 7ff637f3e640 1 -- 172.21.15.125:0/3153144419 shutdown_connections 2023-10-30T12:51:38.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.979+0000 7ff637f3e640 1 -- 172.21.15.125:0/3153144419 wait complete. 2023-10-30T12:51:38.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff637f3e640 1 Processor -- start 2023-10-30T12:51:38.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff637f3e640 1 -- start start 2023-10-30T12:51:38.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff637f3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6300cf160 0x7ff63014fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:38.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff637f3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6301538a0 0x7ff6300ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:38.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff637f3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff63015d6f0 0x7ff6300b3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:38.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff637f3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6301f3530 con 0x7ff6300cf160 2023-10-30T12:51:38.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff637f3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff630150400 con 0x7ff63015d6f0 2023-10-30T12:51:38.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff637f3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63014eb10 con 0x7ff6301538a0 2023-10-30T12:51:38.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff635cda640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6300cf160 0x7ff63014fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:38.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff635cda640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6300cf160 0x7ff63014fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51344/0 (socket says 172.21.15.125:51344) 2023-10-30T12:51:38.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff635cda640 1 -- 172.21.15.125:0/2817253775 learned_addr learned my addr 172.21.15.125:0/2817253775 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:38.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff6354d9640 1 --2- 172.21.15.125:0/2817253775 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6301538a0 0x7ff6300ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:38.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff6364db640 1 --2- 172.21.15.125:0/2817253775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff63015d6f0 0x7ff6300b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:38.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff635cda640 1 -- 172.21.15.125:0/2817253775 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6301538a0 msgr2=0x7ff6300ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:38.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff635cda640 1 --2- 172.21.15.125:0/2817253775 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6301538a0 0x7ff6300ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:38.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff635cda640 1 -- 172.21.15.125:0/2817253775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff63015d6f0 msgr2=0x7ff6300b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:38.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff635cda640 1 --2- 172.21.15.125:0/2817253775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff63015d6f0 0x7ff6300b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:38.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff635cda640 1 -- 172.21.15.125:0/2817253775 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff628009a70 con 0x7ff6300cf160 2023-10-30T12:51:38.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff635cda640 1 --2- 172.21.15.125:0/2817253775 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6300cf160 0x7ff63014fc70 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7ff62c0011f0 tx=0x7ff62c00c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:38.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff6267fc640 1 -- 172.21.15.125:0/2817253775 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62c0074e0 con 0x7ff6300cf160 2023-10-30T12:51:38.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff6267fc640 1 -- 172.21.15.125:0/2817253775 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff62c007640 con 0x7ff6300cf160 2023-10-30T12:51:38.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff637f3e640 1 -- 172.21.15.125:0/2817253775 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff63014df00 con 0x7ff6300cf160 2023-10-30T12:51:38.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff6267fc640 1 -- 172.21.15.125:0/2817253775 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff62c004a10 con 0x7ff6300cf160 2023-10-30T12:51:38.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.983+0000 7ff637f3e640 1 -- 172.21.15.125:0/2817253775 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff63014e4c0 con 0x7ff6300cf160 2023-10-30T12:51:38.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.987+0000 7ff637f3e640 1 -- 172.21.15.125:0/2817253775 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63015db00 con 0x7ff6300cf160 2023-10-30T12:51:38.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.987+0000 7ff6267fc640 1 -- 172.21.15.125:0/2817253775 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff62c004b70 con 0x7ff6300cf160 2023-10-30T12:51:38.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.987+0000 7ff6267fc640 1 --2- 172.21.15.125:0/2817253775 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6140664a0 0x7ff614068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:38.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.987+0000 7ff6267fc640 1 -- 172.21.15.125:0/2817253775 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7ff62c0836c0 con 0x7ff6300cf160 2023-10-30T12:51:38.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.987+0000 7ff6354d9640 1 --2- 172.21.15.125:0/2817253775 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6140664a0 0x7ff614068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:38.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.987+0000 7ff6354d9640 1 --2- 172.21.15.125:0/2817253775 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6140664a0 0x7ff614068960 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff6301f01a0 tx=0x7ff620009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:38.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:38.991+0000 7ff6267fc640 1 -- 172.21.15.125:0/2817253775 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff63015db00 con 0x7ff6300cf160 2023-10-30T12:51:39.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:39.171+0000 7ff637f3e640 1 -- 172.21.15.125:0/2817253775 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff63015db00 con 0x7ff6300cf160 2023-10-30T12:51:39.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:39.171+0000 7ff6267fc640 1 -- 172.21.15.125:0/2817253775 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5730 (secure 0 0 0) 0x7ff63015db00 con 0x7ff6300cf160 2023-10-30T12:51:39.170 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:39.170 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:36.423470+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:clientreplay","state_seq":43,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:39.170 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 115 2023-10-30T12:51:39.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:39.171+0000 7ff637f3e640 1 -- 172.21.15.125:0/2817253775 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6140664a0 msgr2=0x7ff614068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:39.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:39.171+0000 7ff637f3e640 1 --2- 172.21.15.125:0/2817253775 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6140664a0 0x7ff614068960 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff6301f01a0 tx=0x7ff620009060 comp rx=0 tx=0).stop 2023-10-30T12:51:39.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:39.171+0000 7ff637f3e640 1 -- 172.21.15.125:0/2817253775 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6300cf160 msgr2=0x7ff63014fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:39.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:39.171+0000 7ff637f3e640 1 --2- 172.21.15.125:0/2817253775 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6300cf160 0x7ff63014fc70 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7ff62c0011f0 tx=0x7ff62c00c9b0 comp rx=0 tx=0).stop 2023-10-30T12:51:39.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:39.171+0000 7ff637f3e640 1 -- 172.21.15.125:0/2817253775 shutdown_connections 2023-10-30T12:51:39.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:39.171+0000 7ff637f3e640 1 --2- 172.21.15.125:0/2817253775 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6140664a0 0x7ff614068960 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:39.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:39.171+0000 7ff637f3e640 1 --2- 172.21.15.125:0/2817253775 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff63015d6f0 0x7ff6300b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:39.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:39.171+0000 7ff637f3e640 1 --2- 172.21.15.125:0/2817253775 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff6301538a0 0x7ff6300ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:39.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:39.171+0000 7ff637f3e640 1 --2- 172.21.15.125:0/2817253775 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6300cf160 0x7ff63014fc70 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:39.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:39.171+0000 7ff637f3e640 1 -- 172.21.15.125:0/2817253775 >> 172.21.15.125:0/2817253775 conn(0x7ff63006bfe0 msgr2=0x7ff63015a090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:39.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:39.171+0000 7ff637f3e640 1 -- 172.21.15.125:0/2817253775 shutdown_connections 2023-10-30T12:51:39.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:39.171+0000 7ff637f3e640 1 -- 172.21.15.125:0/2817253775 wait complete. 2023-10-30T12:51:39.190 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:51:41.191 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:41.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.339+0000 7fe879b3e640 1 Processor -- start 2023-10-30T12:51:41.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.339+0000 7fe879b3e640 1 -- start start 2023-10-30T12:51:41.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.339+0000 7fe879b3e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe8741538a0 0x7fe87415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:41.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.339+0000 7fe879b3e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8740cb800 con 0x7fe8740cf160 2023-10-30T12:51:41.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.339+0000 7fe879b3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe87415dc30 con 0x7fe8740ca0e0 2023-10-30T12:51:41.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.339+0000 7fe879b3e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8740cb330 con 0x7fe8741538a0 2023-10-30T12:51:41.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.339+0000 7fe8737fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe8741538a0 0x7fe87415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:41.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.339+0000 7fe873fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe8740cf160 0x7fe874164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42274/0 (socket says 172.21.15.125:42274) 2023-10-30T12:51:41.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.339+0000 7fe873fff640 1 -- 172.21.15.125:0/1337025594 learned_addr learned my addr 172.21.15.125:0/1337025594 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:51:41.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.339+0000 7fe8737fe640 1 -- 172.21.15.125:0/1337025594 >> v1:172.21.15.136:6789/0 conn(0x7fe8740ca0e0 legacy=0x7fe87415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:41.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.339+0000 7fe8737fe640 1 -- 172.21.15.125:0/1337025594 >> v1:172.21.15.125:6789/0 conn(0x7fe8740cf160 legacy=0x7fe874164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:41.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.339+0000 7fe8737fe640 1 -- 172.21.15.125:0/1337025594 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8740cad70 con 0x7fe8741538a0 2023-10-30T12:51:41.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.339+0000 7fe8727fc640 1 -- 172.21.15.125:0/1337025594 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860179060 0 0) 0x7fe8740cb800 con 0x7fe8740cf160 2023-10-30T12:51:41.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.343+0000 7fe8737fe640 1 --2- 172.21.15.125:0/1337025594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8741538a0 0x7fe87415d6f0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fe8640097c0 tx=0x7fe86401ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3f218d8c6cc452 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:41.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.343+0000 7fe8727fc640 1 -- 172.21.15.125:0/1337025594 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8640a8970 con 0x7fe8741538a0 2023-10-30T12:51:41.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.343+0000 7fe8727fc640 1 -- 172.21.15.125:0/1337025594 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8640a8ad0 con 0x7fe8741538a0 2023-10-30T12:51:41.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.343+0000 7fe8727fc640 1 -- 172.21.15.125:0/1337025594 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8640a8e60 con 0x7fe8741538a0 2023-10-30T12:51:41.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.343+0000 7fe879b3e640 1 -- 172.21.15.125:0/1337025594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8741538a0 msgr2=0x7fe87415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:41.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.343+0000 7fe879b3e640 1 --2- 172.21.15.125:0/1337025594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8741538a0 0x7fe87415d6f0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7fe8640097c0 tx=0x7fe86401ec70 comp rx=0 tx=0).stop 2023-10-30T12:51:41.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.343+0000 7fe879b3e640 1 -- 172.21.15.125:0/1337025594 shutdown_connections 2023-10-30T12:51:41.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.343+0000 7fe879b3e640 1 --2- 172.21.15.125:0/1337025594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8741538a0 0x7fe87415d6f0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:41.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.343+0000 7fe879b3e640 1 -- 172.21.15.125:0/1337025594 >> 172.21.15.125:0/1337025594 conn(0x7fe87406bfe0 msgr2=0x7fe87406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:41.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.343+0000 7fe879b3e640 1 -- 172.21.15.125:0/1337025594 shutdown_connections 2023-10-30T12:51:41.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.343+0000 7fe879b3e640 1 -- 172.21.15.125:0/1337025594 wait complete. 2023-10-30T12:51:41.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe879b3e640 1 Processor -- start 2023-10-30T12:51:41.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe879b3e640 1 -- start start 2023-10-30T12:51:41.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe879b3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8740ca0e0 0x7fe8740c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:41.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe879b3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8740cf160 0x7fe8741507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:41.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe879b3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe8741538a0 0x7fe8740b42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:41.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe879b3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe874157830 con 0x7fe8740cf160 2023-10-30T12:51:41.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe879b3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8741575c0 con 0x7fe8740ca0e0 2023-10-30T12:51:41.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe879b3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8740b3d90 con 0x7fe8741538a0 2023-10-30T12:51:41.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe872ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8740cf160 0x7fe8741507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:41.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe872ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8740cf160 0x7fe8741507d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37160/0 (socket says 172.21.15.125:37160) 2023-10-30T12:51:41.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe872ffd640 1 -- 172.21.15.125:0/962369958 learned_addr learned my addr 172.21.15.125:0/962369958 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:41.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe8737fe640 1 --2- 172.21.15.125:0/962369958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8740ca0e0 0x7fe8740c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:41.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe873fff640 1 --2- 172.21.15.125:0/962369958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe8741538a0 0x7fe8740b42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:41.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe872ffd640 1 -- 172.21.15.125:0/962369958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe8741538a0 msgr2=0x7fe8740b42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:41.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe872ffd640 1 --2- 172.21.15.125:0/962369958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe8741538a0 0x7fe8740b42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:41.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe872ffd640 1 -- 172.21.15.125:0/962369958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8740ca0e0 msgr2=0x7fe8740c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:41.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe872ffd640 1 --2- 172.21.15.125:0/962369958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8740ca0e0 0x7fe8740c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:41.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe872ffd640 1 -- 172.21.15.125:0/962369958 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe864009420 con 0x7fe8740cf160 2023-10-30T12:51:41.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe872ffd640 1 --2- 172.21.15.125:0/962369958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8740cf160 0x7fe8741507d0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fe86800c3c0 tx=0x7fe86800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:41.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe85bfff640 1 -- 172.21.15.125:0/962369958 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8680052d0 con 0x7fe8740cf160 2023-10-30T12:51:41.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe85bfff640 1 -- 172.21.15.125:0/962369958 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe868005430 con 0x7fe8740cf160 2023-10-30T12:51:41.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe879b3e640 1 -- 172.21.15.125:0/962369958 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe874159040 con 0x7fe8740cf160 2023-10-30T12:51:41.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe85bfff640 1 -- 172.21.15.125:0/962369958 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe868003840 con 0x7fe8740cf160 2023-10-30T12:51:41.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.347+0000 7fe879b3e640 1 -- 172.21.15.125:0/962369958 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8740b4010 con 0x7fe8740cf160 2023-10-30T12:51:41.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.351+0000 7fe879b3e640 1 -- 172.21.15.125:0/962369958 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe87404e170 con 0x7fe8740cf160 2023-10-30T12:51:41.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.351+0000 7fe85bfff640 1 -- 172.21.15.125:0/962369958 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe8680039a0 con 0x7fe8740cf160 2023-10-30T12:51:41.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.351+0000 7fe85bfff640 1 --2- 172.21.15.125:0/962369958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe8600664a0 0x7fe860068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:41.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.351+0000 7fe85bfff640 1 -- 172.21.15.125:0/962369958 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7fe8680836b0 con 0x7fe8740cf160 2023-10-30T12:51:41.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.351+0000 7fe8737fe640 1 --2- 172.21.15.125:0/962369958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe8600664a0 0x7fe860068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:41.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.351+0000 7fe8737fe640 1 --2- 172.21.15.125:0/962369958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe8600664a0 0x7fe860068960 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe8640b2000 tx=0x7fe864008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:41.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.355+0000 7fe85bfff640 1 -- 172.21.15.125:0/962369958 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe8680506c0 con 0x7fe8740cf160 2023-10-30T12:51:41.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.539+0000 7fe879b3e640 1 -- 172.21.15.125:0/962369958 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe874159b40 con 0x7fe8740cf160 2023-10-30T12:51:41.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.539+0000 7fe85bfff640 1 -- 172.21.15.125:0/962369958 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+5730 (secure 0 0 0) 0x7fe868053ce0 con 0x7fe8740cf160 2023-10-30T12:51:41.539 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:41.539 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:36.423470+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:clientreplay","state_seq":43,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:41.540 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 115 2023-10-30T12:51:41.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.539+0000 7fe879b3e640 1 -- 172.21.15.125:0/962369958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe8600664a0 msgr2=0x7fe860068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:41.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.539+0000 7fe879b3e640 1 --2- 172.21.15.125:0/962369958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe8600664a0 0x7fe860068960 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7fe8640b2000 tx=0x7fe864008040 comp rx=0 tx=0).stop 2023-10-30T12:51:41.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.539+0000 7fe879b3e640 1 -- 172.21.15.125:0/962369958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8740cf160 msgr2=0x7fe8741507d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:41.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.539+0000 7fe879b3e640 1 --2- 172.21.15.125:0/962369958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8740cf160 0x7fe8741507d0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fe86800c3c0 tx=0x7fe86800c880 comp rx=0 tx=0).stop 2023-10-30T12:51:41.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.543+0000 7fe879b3e640 1 -- 172.21.15.125:0/962369958 shutdown_connections 2023-10-30T12:51:41.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.543+0000 7fe879b3e640 1 --2- 172.21.15.125:0/962369958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe8600664a0 0x7fe860068960 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:41.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.543+0000 7fe879b3e640 1 --2- 172.21.15.125:0/962369958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe8741538a0 0x7fe8740b42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:41.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.543+0000 7fe879b3e640 1 --2- 172.21.15.125:0/962369958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe8740cf160 0x7fe8741507d0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:41.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.543+0000 7fe879b3e640 1 --2- 172.21.15.125:0/962369958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe8740ca0e0 0x7fe8740c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:41.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.543+0000 7fe879b3e640 1 -- 172.21.15.125:0/962369958 >> 172.21.15.125:0/962369958 conn(0x7fe87406bfe0 msgr2=0x7fe874152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:41.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.543+0000 7fe879b3e640 1 -- 172.21.15.125:0/962369958 shutdown_connections 2023-10-30T12:51:41.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:41.543+0000 7fe879b3e640 1 -- 172.21.15.125:0/962369958 wait complete. 2023-10-30T12:51:41.560 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 115, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 115, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6627": { "addr": "172.21.15.125:6837/2408701650", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2408701650, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2408701650, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6627, "incarnation": 112, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:clientreplay", "state_seq": 43 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:51:36.423470+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6627 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:51:43.562 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:43.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.699+0000 7fae13e9c640 1 Processor -- start 2023-10-30T12:51:43.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.699+0000 7fae13e9c640 1 -- start start 2023-10-30T12:51:43.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.703+0000 7fae13e9c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae0c0c6b80 0x7fae0c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:43.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.703+0000 7fae13e9c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fae0c13f4a0 0x7fae0c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:43.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.703+0000 7fae13e9c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae0c0c0a50 con 0x7fae0c0c6f60 2023-10-30T12:51:43.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.703+0000 7fae13e9c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae0c06f6e0 con 0x7fae0c13f4a0 2023-10-30T12:51:43.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.703+0000 7fae13e9c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c06f9c0 con 0x7fae0c0c6b80 2023-10-30T12:51:43.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.703+0000 7fae12439640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fae0c13f4a0 0x7fae0c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:43.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.703+0000 7fae12439640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fae0c13f4a0 0x7fae0c0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37168/0 (socket says 172.21.15.125:37168) 2023-10-30T12:51:43.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.703+0000 7fae12439640 1 -- 172.21.15.125:0/340980898 learned_addr learned my addr 172.21.15.125:0/340980898 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:43.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.703+0000 7fae11c38640 1 --2- 172.21.15.125:0/340980898 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae0c0c6b80 0x7fae0c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:43.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.703+0000 7fae12439640 1 -- 172.21.15.125:0/340980898 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae0c0c6b80 msgr2=0x7fae0c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:43.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.703+0000 7fae12439640 1 --2- 172.21.15.125:0/340980898 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae0c0c6b80 0x7fae0c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:43.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.703+0000 7fae12439640 1 -- 172.21.15.125:0/340980898 >> v1:172.21.15.136:6789/0 conn(0x7fae0c0c6f60 legacy=0x7fae0c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:43.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.703+0000 7fae12439640 1 -- 172.21.15.125:0/340980898 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0c06fca0 con 0x7fae0c13f4a0 2023-10-30T12:51:43.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.703+0000 7fae12439640 1 --2- 172.21.15.125:0/340980898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae0c13f4a0 0x7fae0c0c0400 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fae04009e50 tx=0x7fae0401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a412664bd7ace9cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:43.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae10c36640 1 -- 172.21.15.125:0/340980898 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae040a57f0 con 0x7fae0c13f4a0 2023-10-30T12:51:43.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae10c36640 1 -- 172.21.15.125:0/340980898 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae040a5950 con 0x7fae0c13f4a0 2023-10-30T12:51:43.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae10c36640 1 -- 172.21.15.125:0/340980898 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae040a5ce0 con 0x7fae0c13f4a0 2023-10-30T12:51:43.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 -- 172.21.15.125:0/340980898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae0c13f4a0 msgr2=0x7fae0c0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:43.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 --2- 172.21.15.125:0/340980898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae0c13f4a0 0x7fae0c0c0400 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fae04009e50 tx=0x7fae0401ca10 comp rx=0 tx=0).stop 2023-10-30T12:51:43.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 -- 172.21.15.125:0/340980898 shutdown_connections 2023-10-30T12:51:43.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 --2- 172.21.15.125:0/340980898 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae0c13f4a0 0x7fae0c0c0400 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:43.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 --2- 172.21.15.125:0/340980898 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae0c0c6b80 0x7fae0c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:43.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 -- 172.21.15.125:0/340980898 >> 172.21.15.125:0/340980898 conn(0x7fae0c0c7910 msgr2=0x7fae0c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:43.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 -- 172.21.15.125:0/340980898 shutdown_connections 2023-10-30T12:51:43.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 -- 172.21.15.125:0/340980898 wait complete. 2023-10-30T12:51:43.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 Processor -- start 2023-10-30T12:51:43.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 -- start start 2023-10-30T12:51:43.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae0c0c6b80 0x7fae0c06a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:43.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae0c0c6f60 0x7fae0c1559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:43.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c13f4a0 0x7fae0c155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:43.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c0c9a10 con 0x7fae0c0c6b80 2023-10-30T12:51:43.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c155530 con 0x7fae0c13f4a0 2023-10-30T12:51:43.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae13e9c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0c0c9730 con 0x7fae0c0c6f60 2023-10-30T12:51:43.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae11c38640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae0c0c6b80 0x7fae0c06a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:43.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae11c38640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae0c0c6b80 0x7fae0c06a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37174/0 (socket says 172.21.15.125:37174) 2023-10-30T12:51:43.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae11c38640 1 -- 172.21.15.125:0/3735737955 learned_addr learned my addr 172.21.15.125:0/3735737955 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:43.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.707+0000 7fae12439640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c13f4a0 0x7fae0c155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:43.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae11437640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae0c0c6f60 0x7fae0c1559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:43.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae11c38640 1 -- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae0c0c6f60 msgr2=0x7fae0c1559c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:43.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae11c38640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae0c0c6f60 0x7fae0c1559c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:43.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae11c38640 1 -- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c13f4a0 msgr2=0x7fae0c155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:43.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae11c38640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c13f4a0 0x7fae0c155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:43.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae11c38640 1 -- 172.21.15.125:0/3735737955 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae04009b30 con 0x7fae0c0c6b80 2023-10-30T12:51:43.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae12439640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c13f4a0 0x7fae0c155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:43.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae11c38640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae0c0c6b80 0x7fae0c06a0f0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fae0800d800 tx=0x7fae0800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:43.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae027fc640 1 -- 172.21.15.125:0/3735737955 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0800b6c0 con 0x7fae0c0c6b80 2023-10-30T12:51:43.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae027fc640 1 -- 172.21.15.125:0/3735737955 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0800b820 con 0x7fae0c0c6b80 2023-10-30T12:51:43.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae027fc640 1 -- 172.21.15.125:0/3735737955 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0800bb90 con 0x7fae0c0c6b80 2023-10-30T12:51:43.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae13e9c640 1 -- 172.21.15.125:0/3735737955 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0c15f8c0 con 0x7fae0c0c6b80 2023-10-30T12:51:43.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae13e9c640 1 -- 172.21.15.125:0/3735737955 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0c15fe80 con 0x7fae0c0c6b80 2023-10-30T12:51:43.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae13e9c640 1 -- 172.21.15.125:0/3735737955 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae0c04e170 con 0x7fae0c0c6b80 2023-10-30T12:51:43.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae027fc640 1 -- 172.21.15.125:0/3735737955 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fae0800bcf0 con 0x7fae0c0c6b80 2023-10-30T12:51:43.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae027fc640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fadf00664d0 0x7fadf0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:43.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.711+0000 7fae027fc640 1 -- 172.21.15.125:0/3735737955 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7fae08084780 con 0x7fae0c0c6b80 2023-10-30T12:51:43.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.715+0000 7fae11437640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fadf00664d0 0x7fadf0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:43.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.715+0000 7fae11437640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fadf00664d0 0x7fadf0068990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fae0c06a9c0 tx=0x7fadfc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:43.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.715+0000 7fae027fc640 1 -- 172.21.15.125:0/3735737955 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fae08051830 con 0x7fae0c0c6b80 2023-10-30T12:51:43.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.899+0000 7fae13e9c640 1 -- 172.21.15.125:0/3735737955 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae0c150d80 con 0x7fae0c0c6b80 2023-10-30T12:51:43.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.899+0000 7fae027fc640 1 -- 172.21.15.125:0/3735737955 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+5724 (secure 0 0 0) 0x7fae08054e50 con 0x7fae0c0c6b80 2023-10-30T12:51:43.898 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:43.898 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:43.301579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":45,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:43.899 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 116 2023-10-30T12:51:43.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.899+0000 7fae13e9c640 1 -- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fadf00664d0 msgr2=0x7fadf0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:43.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.899+0000 7fae13e9c640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fadf00664d0 0x7fadf0068990 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fae0c06a9c0 tx=0x7fadfc009040 comp rx=0 tx=0).stop 2023-10-30T12:51:43.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.899+0000 7fae13e9c640 1 -- 172.21.15.125:0/3735737955 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae0c0c6b80 msgr2=0x7fae0c06a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:43.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.899+0000 7fae13e9c640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae0c0c6b80 0x7fae0c06a0f0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fae0800d800 tx=0x7fae0800dcc0 comp rx=0 tx=0).stop 2023-10-30T12:51:43.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.899+0000 7fae13e9c640 1 -- 172.21.15.125:0/3735737955 shutdown_connections 2023-10-30T12:51:43.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.899+0000 7fae13e9c640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fadf00664d0 0x7fadf0068990 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:43.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.899+0000 7fae13e9c640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae0c13f4a0 0x7fae0c155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:43.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.899+0000 7fae13e9c640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae0c0c6f60 0x7fae0c1559c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:43.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.899+0000 7fae13e9c640 1 --2- 172.21.15.125:0/3735737955 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae0c0c6b80 0x7fae0c06a0f0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:43.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.899+0000 7fae13e9c640 1 -- 172.21.15.125:0/3735737955 >> 172.21.15.125:0/3735737955 conn(0x7fae0c0c7910 msgr2=0x7fae0c13ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:43.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.899+0000 7fae13e9c640 1 -- 172.21.15.125:0/3735737955 shutdown_connections 2023-10-30T12:51:43.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:43.899+0000 7fae13e9c640 1 -- 172.21.15.125:0/3735737955 wait complete. 2023-10-30T12:51:43.918 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=6318 2023-10-30T12:51:43.919 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.e 2023-10-30T12:51:47.953 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:51:47.958 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:51:49.446 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 92% (64/69) Updating files: 94% (65/69) Updating files: 95% (66/69) Updating files: 97% (67/69) Updating files: 98% (68/69) Updating files: 100% (69/69) Updating files: 100% (69/69), done. 2023-10-30T12:51:50.613 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 9fa6f5e7d09 v0.11 2023-10-30T12:51:50.617 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.11 2023-10-30T12:51:50.632 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 12 -eq 48 ] 2023-10-30T12:51:50.632 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.12 2023-10-30T12:51:50.633 INFO:tasks.workunit.client.0.smithi125.stdout:v0.12 2023-10-30T12:51:50.633 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.12 2023-10-30T12:51:50.633 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.12 2023-10-30T12:51:51.530 INFO:tasks.ceph.mds.c.smithi125.stderr:2023-10-30T12:51:51.531+0000 7f0808235640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:51:52.919 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T12:51:52.920 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T12:51:52.920 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T12:51:52.924 INFO:tasks.ceph.mds.e:Started 2023-10-30T12:51:52.925 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:53.016 INFO:tasks.ceph.mds.e.smithi125.stdout:starting mds.e at 2023-10-30T12:51:53.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.067+0000 7faa841d3640 1 Processor -- start 2023-10-30T12:51:53.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.067+0000 7faa841d3640 1 -- start start 2023-10-30T12:51:53.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.071+0000 7faa841d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faa7c0c6f60 0x7faa7c1566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:53.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.071+0000 7faa841d3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c0c0a50 con 0x7faa7c0c05d0 2023-10-30T12:51:53.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.071+0000 7faa841d3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c06f6e0 con 0x7faa7c0c6b80 2023-10-30T12:51:53.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.071+0000 7faa841d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7c06f9c0 con 0x7faa7c0c6f60 2023-10-30T12:51:53.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.071+0000 7faa82770640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faa7c0c6f60 0x7faa7c1566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:53.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.071+0000 7faa8176e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7faa7c0c05d0 0x7faa7c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55526/0 (socket says 172.21.15.125:55526) 2023-10-30T12:51:53.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.071+0000 7faa8176e640 1 -- 172.21.15.125:0/3072950726 learned_addr learned my addr 172.21.15.125:0/3072950726 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:51:53.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.071+0000 7faa82770640 1 -- 172.21.15.125:0/3072950726 >> v1:172.21.15.136:6789/0 conn(0x7faa7c0c6b80 legacy=0x7faa7c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:53.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.071+0000 7faa82770640 1 -- 172.21.15.125:0/3072950726 >> v1:172.21.15.125:6789/0 conn(0x7faa7c0c05d0 legacy=0x7faa7c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:53.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.071+0000 7faa82770640 1 -- 172.21.15.125:0/3072950726 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c06fca0 con 0x7faa7c0c6f60 2023-10-30T12:51:53.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.071+0000 7faa82770640 1 --2- 172.21.15.125:0/3072950726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa7c0c6f60 0x7faa7c1566a0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7faa74009d90 tx=0x7faa7401eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b390c062fa53b5e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:53.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.071+0000 7faa80f6d640 1 -- 172.21.15.125:0/3072950726 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa740a7a40 con 0x7faa7c0c6f60 2023-10-30T12:51:53.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.071+0000 7faa80f6d640 1 -- 172.21.15.125:0/3072950726 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa740a7ba0 con 0x7faa7c0c6f60 2023-10-30T12:51:53.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.071+0000 7faa80f6d640 1 -- 172.21.15.125:0/3072950726 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa740af440 con 0x7faa7c0c6f60 2023-10-30T12:51:53.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 -- 172.21.15.125:0/3072950726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa7c0c6f60 msgr2=0x7faa7c1566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:53.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 --2- 172.21.15.125:0/3072950726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa7c0c6f60 0x7faa7c1566a0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7faa74009d90 tx=0x7faa7401eb60 comp rx=0 tx=0).stop 2023-10-30T12:51:53.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 -- 172.21.15.125:0/3072950726 shutdown_connections 2023-10-30T12:51:53.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 --2- 172.21.15.125:0/3072950726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa7c0c6f60 0x7faa7c1566a0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:53.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 -- 172.21.15.125:0/3072950726 >> 172.21.15.125:0/3072950726 conn(0x7faa7c0c7910 msgr2=0x7faa7c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:53.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 -- 172.21.15.125:0/3072950726 shutdown_connections 2023-10-30T12:51:53.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 -- 172.21.15.125:0/3072950726 wait complete. 2023-10-30T12:51:53.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 Processor -- start 2023-10-30T12:51:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 -- start start 2023-10-30T12:51:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa7c0c05d0 0x7faa7c136ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa7c0c6b80 0x7faa7c1f5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faa7c0c6f60 0x7faa7c153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:53.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c13e140 con 0x7faa7c0c05d0 2023-10-30T12:51:53.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c0ca040 con 0x7faa7c0c6b80 2023-10-30T12:51:53.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa841d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c0c9a80 con 0x7faa7c0c6f60 2023-10-30T12:51:53.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa81f6f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa7c0c05d0 0x7faa7c136ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:53.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa81f6f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa7c0c05d0 0x7faa7c136ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37300/0 (socket says 172.21.15.125:37300) 2023-10-30T12:51:53.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa81f6f640 1 -- 172.21.15.125:0/2928842773 learned_addr learned my addr 172.21.15.125:0/2928842773 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:53.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa8176e640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa7c0c6b80 0x7faa7c1f5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:53.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.075+0000 7faa82770640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faa7c0c6f60 0x7faa7c153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:53.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa81f6f640 1 -- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faa7c0c6f60 msgr2=0x7faa7c153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:53.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa81f6f640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faa7c0c6f60 0x7faa7c153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:53.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa81f6f640 1 -- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa7c0c6b80 msgr2=0x7faa7c1f5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:53.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa81f6f640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa7c0c6b80 0x7faa7c1f5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:53.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa81f6f640 1 -- 172.21.15.125:0/2928842773 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa74009a70 con 0x7faa7c0c05d0 2023-10-30T12:51:53.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa8176e640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa7c0c6b80 0x7faa7c1f5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:51:53.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa81f6f640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa7c0c05d0 0x7faa7c136ab0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7faa7800c500 tx=0x7faa7800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:53.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa727fc640 1 -- 172.21.15.125:0/2928842773 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa78005220 con 0x7faa7c0c05d0 2023-10-30T12:51:53.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa727fc640 1 -- 172.21.15.125:0/2928842773 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa78005380 con 0x7faa7c0c05d0 2023-10-30T12:51:53.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa727fc640 1 -- 172.21.15.125:0/2928842773 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa78007270 con 0x7faa7c0c05d0 2023-10-30T12:51:53.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa841d3640 1 -- 172.21.15.125:0/2928842773 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7c0c9d60 con 0x7faa7c0c05d0 2023-10-30T12:51:53.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa841d3640 1 -- 172.21.15.125:0/2928842773 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7c13ddd0 con 0x7faa7c0c05d0 2023-10-30T12:51:53.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa841d3640 1 -- 172.21.15.125:0/2928842773 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7c13df90 con 0x7faa7c0c05d0 2023-10-30T12:51:53.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa727fc640 1 -- 172.21.15.125:0/2928842773 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faa780073d0 con 0x7faa7c0c05d0 2023-10-30T12:51:53.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa727fc640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faa600664a0 0x7faa60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:53.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa727fc640 1 -- 172.21.15.125:0/2928842773 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7faa78083bd0 con 0x7faa7c0c05d0 2023-10-30T12:51:53.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.079+0000 7faa8176e640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faa600664a0 0x7faa60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:53.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.083+0000 7faa8176e640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faa600664a0 0x7faa60068960 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0c54c0 tx=0x7faa6c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:53.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.083+0000 7faa727fc640 1 -- 172.21.15.125:0/2928842773 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faa78050bf0 con 0x7faa7c0c05d0 2023-10-30T12:51:53.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.262+0000 7faa841d3640 1 -- 172.21.15.125:0/2928842773 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa7c04e170 con 0x7faa7c0c05d0 2023-10-30T12:51:53.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.262+0000 7faa727fc640 1 -- 172.21.15.125:0/2928842773 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+5724 (secure 0 0 0) 0x7faa78054210 con 0x7faa7c0c05d0 2023-10-30T12:51:53.265 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:53.265 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:43.301579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":45,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:53.266 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 116 2023-10-30T12:51:53.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.266+0000 7faa841d3640 1 -- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faa600664a0 msgr2=0x7faa60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:53.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.266+0000 7faa841d3640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faa600664a0 0x7faa60068960 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0c54c0 tx=0x7faa6c009040 comp rx=0 tx=0).stop 2023-10-30T12:51:53.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.266+0000 7faa841d3640 1 -- 172.21.15.125:0/2928842773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa7c0c05d0 msgr2=0x7faa7c136ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:53.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.266+0000 7faa841d3640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa7c0c05d0 0x7faa7c136ab0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7faa7800c500 tx=0x7faa7800c9c0 comp rx=0 tx=0).stop 2023-10-30T12:51:53.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.266+0000 7faa841d3640 1 -- 172.21.15.125:0/2928842773 shutdown_connections 2023-10-30T12:51:53.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.266+0000 7faa841d3640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faa600664a0 0x7faa60068960 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:53.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.266+0000 7faa841d3640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faa7c0c6f60 0x7faa7c153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:53.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.266+0000 7faa841d3640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa7c0c6b80 0x7faa7c1f5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:53.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.266+0000 7faa841d3640 1 --2- 172.21.15.125:0/2928842773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa7c0c05d0 0x7faa7c136ab0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:53.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.266+0000 7faa841d3640 1 -- 172.21.15.125:0/2928842773 >> 172.21.15.125:0/2928842773 conn(0x7faa7c0c7910 msgr2=0x7faa7c06a880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:53.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.266+0000 7faa841d3640 1 -- 172.21.15.125:0/2928842773 shutdown_connections 2023-10-30T12:51:53.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:53.266+0000 7faa841d3640 1 -- 172.21.15.125:0/2928842773 wait complete. 2023-10-30T12:51:53.285 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T12:51:55.286 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:51:55.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.430+0000 7f69de046640 1 Processor -- start 2023-10-30T12:51:55.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69de046640 1 -- start start 2023-10-30T12:51:55.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69de046640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69d80c6b80 0x7f69d80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:55.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69de046640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f69d80c6f60 0x7f69d80c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:55.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69de046640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f69d80c8430 0x7f69d813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:55.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69de046640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69d80c0a50 con 0x7f69d80c8430 2023-10-30T12:51:55.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69de046640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69d806f6e0 con 0x7f69d80c6f60 2023-10-30T12:51:55.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69de046640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69d806f9c0 con 0x7f69d80c6b80 2023-10-30T12:51:55.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69d7fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f69d80c8430 0x7f69d813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:55.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69d7fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f69d80c8430 0x7f69d813ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37316/0 (socket says 172.21.15.125:37316) 2023-10-30T12:51:55.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69d7fff640 1 -- 172.21.15.125:0/2905508405 learned_addr learned my addr 172.21.15.125:0/2905508405 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:55.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69d77fe640 1 --2- 172.21.15.125:0/2905508405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69d80c6b80 0x7f69d80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:55.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69d6ffd640 1 --2- 172.21.15.125:0/2905508405 >> v2:172.21.15.136:3300/0 conn(0x7f69d80c6f60 0x7f69d80c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:55.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69d7fff640 1 -- 172.21.15.125:0/2905508405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69d80c6b80 msgr2=0x7f69d80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:55.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69d7fff640 1 --2- 172.21.15.125:0/2905508405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69d80c6b80 0x7f69d80bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:55.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69d7fff640 1 -- 172.21.15.125:0/2905508405 >> v2:172.21.15.136:3300/0 conn(0x7f69d80c6f60 msgr2=0x7f69d80c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:51:55.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69d7fff640 1 --2- 172.21.15.125:0/2905508405 >> v2:172.21.15.136:3300/0 conn(0x7f69d80c6f60 0x7f69d80c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:55.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69d7fff640 1 -- 172.21.15.125:0/2905508405 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d806fca0 con 0x7f69d80c8430 2023-10-30T12:51:55.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.434+0000 7f69d6ffd640 1 --2- 172.21.15.125:0/2905508405 >> v2:172.21.15.136:3300/0 conn(0x7f69d80c6f60 0x7f69d80c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:55.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.438+0000 7f69d7fff640 1 --2- 172.21.15.125:0/2905508405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69d80c8430 0x7f69d813ed90 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f69d806c830 tx=0x7f69d001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d74ab1e977c21a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:55.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.438+0000 7f69d67fc640 1 -- 172.21.15.125:0/2905508405 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d00a5760 con 0x7f69d80c8430 2023-10-30T12:51:55.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.438+0000 7f69d67fc640 1 -- 172.21.15.125:0/2905508405 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d00a58c0 con 0x7f69d80c8430 2023-10-30T12:51:55.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.438+0000 7f69d67fc640 1 -- 172.21.15.125:0/2905508405 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d00a5c50 con 0x7f69d80c8430 2023-10-30T12:51:55.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.438+0000 7f69de046640 1 -- 172.21.15.125:0/2905508405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69d80c8430 msgr2=0x7f69d813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:55.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.438+0000 7f69de046640 1 --2- 172.21.15.125:0/2905508405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69d80c8430 0x7f69d813ed90 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f69d806c830 tx=0x7f69d001c9d0 comp rx=0 tx=0).stop 2023-10-30T12:51:55.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.438+0000 7f69de046640 1 -- 172.21.15.125:0/2905508405 shutdown_connections 2023-10-30T12:51:55.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.438+0000 7f69de046640 1 --2- 172.21.15.125:0/2905508405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69d80c8430 0x7f69d813ed90 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:55.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.438+0000 7f69de046640 1 --2- 172.21.15.125:0/2905508405 >> v2:172.21.15.136:3300/0 conn(0x7f69d80c6f60 0x7f69d80c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:55.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.438+0000 7f69de046640 1 --2- 172.21.15.125:0/2905508405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69d80c6b80 0x7f69d80bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:55.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.438+0000 7f69de046640 1 -- 172.21.15.125:0/2905508405 >> 172.21.15.125:0/2905508405 conn(0x7f69d80c7910 msgr2=0x7f69d80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:55.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.438+0000 7f69de046640 1 -- 172.21.15.125:0/2905508405 shutdown_connections 2023-10-30T12:51:55.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.438+0000 7f69de046640 1 -- 172.21.15.125:0/2905508405 wait complete. 2023-10-30T12:51:55.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.438+0000 7f69de046640 1 Processor -- start 2023-10-30T12:51:55.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69de046640 1 -- start start 2023-10-30T12:51:55.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69de046640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69d80c6b80 0x7f69d8155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:55.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69de046640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69d80c6f60 0x7f69d8156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:55.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69de046640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69d80c8430 0x7f69d80c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:55.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69de046640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d80c9bc0 con 0x7f69d80c8430 2023-10-30T12:51:55.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69de046640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69d813a450 con 0x7f69d80c6b80 2023-10-30T12:51:55.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69de046640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69d813a6c0 con 0x7f69d80c6f60 2023-10-30T12:51:55.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69d7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69d80c8430 0x7f69d80c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:55.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69d7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69d80c8430 0x7f69d80c8e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37332/0 (socket says 172.21.15.125:37332) 2023-10-30T12:51:55.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69d7fff640 1 -- 172.21.15.125:0/453473037 learned_addr learned my addr 172.21.15.125:0/453473037 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:51:55.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69d77fe640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69d80c6b80 0x7f69d8155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:55.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69d6ffd640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69d80c6f60 0x7f69d8156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:55.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69d7fff640 1 -- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69d80c6f60 msgr2=0x7f69d8156120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:55.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69d7fff640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69d80c6f60 0x7f69d8156120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:55.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69d7fff640 1 -- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69d80c6b80 msgr2=0x7f69d8155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:55.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69d7fff640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69d80c6b80 0x7f69d8155a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:55.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69d7fff640 1 -- 172.21.15.125:0/453473037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69d0009b30 con 0x7f69d80c8430 2023-10-30T12:51:55.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69d6ffd640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69d80c6f60 0x7f69d8156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:51:55.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69d7fff640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69d80c8430 0x7f69d80c8e70 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f69d00af040 tx=0x7f69d0019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:55.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69bffff640 1 -- 172.21.15.125:0/453473037 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d00b8880 con 0x7f69d80c8430 2023-10-30T12:51:55.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69bffff640 1 -- 172.21.15.125:0/453473037 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69d00b8a20 con 0x7f69d80c8430 2023-10-30T12:51:55.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69bffff640 1 -- 172.21.15.125:0/453473037 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69d00b8dd0 con 0x7f69d80c8430 2023-10-30T12:51:55.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69de046640 1 -- 172.21.15.125:0/453473037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69d80c9920 con 0x7f69d80c8430 2023-10-30T12:51:55.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69de046640 1 -- 172.21.15.125:0/453473037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69d8154900 con 0x7f69d80c8430 2023-10-30T12:51:55.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.442+0000 7f69de046640 1 -- 172.21.15.125:0/453473037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69d80c7710 con 0x7f69d80c8430 2023-10-30T12:51:55.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.446+0000 7f69bffff640 1 -- 172.21.15.125:0/453473037 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f69d00c3460 con 0x7f69d80c8430 2023-10-30T12:51:55.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.446+0000 7f69bffff640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69c40664a0 0x7f69c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:51:55.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.446+0000 7f69bffff640 1 -- 172.21.15.125:0/453473037 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7f69d011f8d0 con 0x7f69d80c8430 2023-10-30T12:51:55.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.446+0000 7f69d77fe640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69c40664a0 0x7f69c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:51:55.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.446+0000 7f69d77fe640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69c40664a0 0x7f69c4068960 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f69c800bd80 tx=0x7f69c800b3a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:51:55.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.450+0000 7f69bffff640 1 -- 172.21.15.125:0/453473037 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f69d00ec970 con 0x7f69d80c8430 2023-10-30T12:51:55.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.642+0000 7f69de046640 1 -- 172.21.15.125:0/453473037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69d8154480 con 0x7f69d80c8430 2023-10-30T12:51:55.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.642+0000 7f69bffff640 1 -- 172.21.15.125:0/453473037 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f69d8154480 con 0x7f69d80c8430 2023-10-30T12:51:55.643 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:51:55.643 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6954,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:43.301579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":45,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:51:55.643 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 117 2023-10-30T12:51:55.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.646+0000 7f69de046640 1 -- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69c40664a0 msgr2=0x7f69c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:55.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.646+0000 7f69de046640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69c40664a0 0x7f69c4068960 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f69c800bd80 tx=0x7f69c800b3a0 comp rx=0 tx=0).stop 2023-10-30T12:51:55.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.646+0000 7f69de046640 1 -- 172.21.15.125:0/453473037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69d80c8430 msgr2=0x7f69d80c8e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:51:55.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.646+0000 7f69de046640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69d80c8430 0x7f69d80c8e70 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f69d00af040 tx=0x7f69d0019770 comp rx=0 tx=0).stop 2023-10-30T12:51:55.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.646+0000 7f69de046640 1 -- 172.21.15.125:0/453473037 shutdown_connections 2023-10-30T12:51:55.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.646+0000 7f69de046640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69c40664a0 0x7f69c4068960 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:55.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.646+0000 7f69de046640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69d80c8430 0x7f69d80c8e70 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:55.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.646+0000 7f69de046640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69d80c6f60 0x7f69d8156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:55.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.646+0000 7f69de046640 1 --2- 172.21.15.125:0/453473037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f69d80c6b80 0x7f69d8155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:51:55.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.646+0000 7f69de046640 1 -- 172.21.15.125:0/453473037 >> 172.21.15.125:0/453473037 conn(0x7f69d80c7910 msgr2=0x7f69d806a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:51:55.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.646+0000 7f69de046640 1 -- 172.21.15.125:0/453473037 shutdown_connections 2023-10-30T12:51:55.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:51:55.646+0000 7f69de046640 1 -- 172.21.15.125:0/453473037 wait complete. 2023-10-30T12:51:55.666 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T12:51:55.666 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 21 secs before thrashing 2023-10-30T12:51:55.672 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 24% (18/75) Updating files: 25% (19/75) Updating files: 26% (20/75) Updating files: 28% (21/75) Updating files: 29% (22/75) Updating files: 30% (23/75) Updating files: 32% (24/75) Updating files: 33% (25/75) Updating files: 34% (26/75) Updating files: 36% (27/75) Updating files: 37% (28/75) Updating files: 38% (29/75) Updating files: 40% (30/75) Updating files: 41% (31/75) Updating files: 42% (32/75) Updating files: 44% (33/75) Updating files: 45% (34/75) Updating files: 46% (35/75) Updating files: 48% (36/75) Updating files: 49% (37/75) Updating files: 50% (38/75) Updating files: 52% (39/75) Updating files: 53% (40/75) Updating files: 54% (41/75) Updating files: 56% (42/75) Updating files: 57% (43/75) Updating files: 58% (44/75) Updating files: 60% (45/75) Updating files: 61% (46/75) Updating files: 62% (47/75) Updating files: 64% (48/75) Updating files: 65% (49/75) Updating files: 66% (50/75) Updating files: 68% (51/75) Updating files: 69% (52/75) Updating files: 70% (53/75) Updating files: 72% (54/75) Updating files: 73% (55/75) Updating files: 74% (56/75) Updating files: 76% (57/75) Updating files: 77% (58/75) Updating files: 78% (59/75) Updating files: 80% (60/75) Updating files: 81% (61/75) Updating files: 82% (62/75) Updating files: 84% (63/75) Updating files: 85% (64/75) Updating files: 86% (65/75) Updating files: 88% (66/75) Updating files: 89% (67/75) Updating files: 90% (68/75) Updating files: 92% (69/75) Updating files: 93% (70/75) Updating files: 94% (71/75) Updating files: 96% (72/75) Updating files: 97% (73/75) Updating files: 98% (74/75) Updating files: 100% (75/75) Updating files: 100% (75/75), done. 2023-10-30T12:51:55.699 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at f524150822a v0.12 2023-10-30T12:51:55.703 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.12 2023-10-30T12:51:55.798 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 13 -eq 48 ] 2023-10-30T12:51:55.798 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.13 2023-10-30T12:51:55.798 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.13 2023-10-30T12:51:55.799 INFO:tasks.workunit.client.0.smithi125.stdout:v0.13 2023-10-30T12:51:55.799 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.13 2023-10-30T12:51:56.242 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 2b32297267d v0.13 2023-10-30T12:51:56.246 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.13 2023-10-30T12:51:56.407 INFO:tasks.workunit.client.0.smithi125.stdout:v0.14 2023-10-30T12:51:56.408 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 14 -eq 48 ] 2023-10-30T12:51:56.409 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.14 2023-10-30T12:51:56.409 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.14 2023-10-30T12:51:56.409 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.14 2023-10-30T12:51:58.631 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 68% (103/151) Updating files: 69% (105/151) Updating files: 70% (106/151) Updating files: 71% (108/151) Updating files: 72% (109/151) Updating files: 73% (111/151) Updating files: 74% (112/151) Updating files: 75% (114/151) Updating files: 76% (115/151) Updating files: 77% (117/151) Updating files: 78% (118/151) Updating files: 79% (120/151) Updating files: 80% (121/151) Updating files: 81% (123/151) Updating files: 82% (124/151) Updating files: 83% (126/151) Updating files: 84% (127/151) Updating files: 85% (129/151) Updating files: 86% (130/151) Updating files: 87% (132/151) Updating files: 88% (133/151) Updating files: 89% (135/151) Updating files: 90% (136/151) Updating files: 91% (138/151) Updating files: 92% (139/151) Updating files: 93% (141/151) Updating files: 94% (142/151) Updating files: 95% (144/151) Updating files: 96% (145/151) Updating files: 97% (147/151) Updating files: 98% (148/151) Updating files: 99% (150/151) Updating files: 100% (151/151) Updating files: 100% (151/151), done. 2023-10-30T12:51:58.661 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 1ec1e8d6fb6 v0.14 2023-10-30T12:51:58.665 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.14 2023-10-30T12:51:58.725 INFO:tasks.workunit.client.0.smithi125.stdout:v0.15 2023-10-30T12:51:58.726 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 15 -eq 48 ] 2023-10-30T12:51:58.726 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.15 2023-10-30T12:51:58.726 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.15 2023-10-30T12:51:58.727 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.15 2023-10-30T12:51:59.612 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at ab3c421eb95 v0.15 2023-10-30T12:51:59.615 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.15 2023-10-30T12:51:59.695 INFO:tasks.workunit.client.0.smithi125.stdout:v0.16 2023-10-30T12:51:59.695 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 16 -eq 48 ] 2023-10-30T12:51:59.696 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.16 2023-10-30T12:51:59.696 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.16 2023-10-30T12:51:59.696 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.16 2023-10-30T12:52:14.496 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 34% (37/107) Updating files: 35% (38/107) Updating files: 36% (39/107) Updating files: 37% (40/107) Updating files: 38% (41/107) Updating files: 39% (42/107) Updating files: 40% (43/107) Updating files: 41% (44/107) Updating files: 42% (45/107) Updating files: 43% (47/107) Updating files: 44% (48/107) Updating files: 45% (49/107) Updating files: 46% (50/107) Updating files: 47% (51/107) Updating files: 48% (52/107) Updating files: 49% (53/107) Updating files: 50% (54/107) Updating files: 51% (55/107) Updating files: 52% (56/107) Updating files: 53% (57/107) Updating files: 54% (58/107) Updating files: 55% (59/107) Updating files: 56% (60/107) Updating files: 57% (61/107) Updating files: 58% (63/107) Updating files: 59% (64/107) Updating files: 60% (65/107) Updating files: 61% (66/107) Updating files: 62% (67/107) Updating files: 63% (68/107) Updating files: 64% (69/107) Updating files: 65% (70/107) Updating files: 66% (71/107) Updating files: 67% (72/107) Updating files: 68% (73/107) Updating files: 69% (74/107) Updating files: 70% (75/107) Updating files: 71% (76/107) Updating files: 72% (78/107) Updating files: 73% (79/107) Updating files: 74% (80/107) Updating files: 75% (81/107) Updating files: 76% (82/107) Updating files: 77% (83/107) Updating files: 78% (84/107) Updating files: 79% (85/107) Updating files: 80% (86/107) Updating files: 81% (87/107) Updating files: 82% (88/107) Updating files: 83% (89/107) Updating files: 84% (90/107) Updating files: 85% (91/107) Updating files: 86% (93/107) Updating files: 87% (94/107) Updating files: 88% (95/107) Updating files: 89% (96/107) Updating files: 90% (97/107) Updating files: 91% (98/107) Updating files: 92% (99/107) Updating files: 93% (100/107) Updating files: 94% (101/107) Updating files: 95% (102/107) Updating files: 96% (103/107) Updating files: 97% (104/107) Updating files: 98% (105/107) Updating files: 99% (106/107) Updating files: 100% (107/107) Updating files: 100% (107/107), done. 2023-10-30T12:52:14.566 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 38dbaa5175c v0.16 2023-10-30T12:52:14.571 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.16 2023-10-30T12:52:14.644 INFO:tasks.workunit.client.0.smithi125.stdout:v0.17 2023-10-30T12:52:14.645 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 17 -eq 48 ] 2023-10-30T12:52:14.645 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.17 2023-10-30T12:52:14.645 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.17 2023-10-30T12:52:14.646 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.17 2023-10-30T12:52:16.055 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2023-10-30T12:52:16.087 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at dac3a8c35fb v0.17 2023-10-30T12:52:16.091 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.17 2023-10-30T12:52:16.668 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:16.739 INFO:tasks.workunit.client.0.smithi125.stdout:v0.18 2023-10-30T12:52:16.739 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 18 -eq 48 ] 2023-10-30T12:52:16.739 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.18 2023-10-30T12:52:16.740 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.18 2023-10-30T12:52:16.740 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.18 2023-10-30T12:52:16.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.818+0000 7f696850f640 1 Processor -- start 2023-10-30T12:52:16.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.818+0000 7f696850f640 1 -- start start 2023-10-30T12:52:16.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f696850f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69600c6b80 0x7f69600bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:16.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f696850f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f696013f4a0 0x7f69600c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:16.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f696850f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69600c0a50 con 0x7f69600c6f60 2023-10-30T12:52:16.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f696850f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696006f6e0 con 0x7f696013f4a0 2023-10-30T12:52:16.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f696850f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696006f9c0 con 0x7f69600c6b80 2023-10-30T12:52:16.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f6965aaa640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f69600c6f60 0x7f696013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47712/0 (socket says 172.21.15.125:47712) 2023-10-30T12:52:16.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f6965aaa640 1 -- 172.21.15.125:0/1608995314 learned_addr learned my addr 172.21.15.125:0/1608995314 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:52:16.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69662ab640 1 --2- 172.21.15.125:0/1608995314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69600c6b80 0x7f69600bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:16.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f6966aac640 1 --2- 172.21.15.125:0/1608995314 >> v2:172.21.15.136:3300/0 conn(0x7f696013f4a0 0x7f69600c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:16.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 -- 172.21.15.125:0/1608995314 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1609637530 0 0) 0x7f69600c0a50 con 0x7f69600c6f60 2023-10-30T12:52:16.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 -- 172.21.15.125:0/1608995314 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f694c005740 con 0x7f69600c6f60 2023-10-30T12:52:16.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 -- 172.21.15.125:0/1608995314 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2878601170 0 0) 0x7f694c005740 con 0x7f69600c6f60 2023-10-30T12:52:16.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 -- 172.21.15.125:0/1608995314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69600c6b80 msgr2=0x7f69600bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:16.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 --2- 172.21.15.125:0/1608995314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69600c6b80 0x7f69600bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:16.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 -- 172.21.15.125:0/1608995314 >> v2:172.21.15.136:3300/0 conn(0x7f696013f4a0 msgr2=0x7f69600c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:16.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 --2- 172.21.15.125:0/1608995314 >> v2:172.21.15.136:3300/0 conn(0x7f696013f4a0 0x7f69600c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:16.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 -- 172.21.15.125:0/1608995314 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696006fca0 con 0x7f69600c6f60 2023-10-30T12:52:16.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 -- 172.21.15.125:0/1608995314 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f6950001680 con 0x7f69600c6f60 2023-10-30T12:52:16.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 -- 172.21.15.125:0/1608995314 >> v1:172.21.15.125:6789/0 conn(0x7f69600c6f60 legacy=0x7f696013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:16.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 --2- 172.21.15.125:0/1608995314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f694c005b90 0x7f694c005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:16.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 --2- 172.21.15.125:0/1608995314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69600c6b80 0x7f694c0069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:16.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 --2- 172.21.15.125:0/1608995314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f694c006f20 0x7f694c0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:16.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 -- 172.21.15.125:0/1608995314 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696006f6e0 con 0x7f694c005b90 2023-10-30T12:52:16.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 -- 172.21.15.125:0/1608995314 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696006f9c0 con 0x7f694c006f20 2023-10-30T12:52:16.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69652a9640 1 -- 172.21.15.125:0/1608995314 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f694c00dba0 con 0x7f69600c6b80 2023-10-30T12:52:16.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69662ab640 1 --2- 172.21.15.125:0/1608995314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f694c005b90 0x7f694c005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:16.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f6965aaa640 1 --2- 172.21.15.125:0/1608995314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69600c6b80 0x7f694c0069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:16.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f6966aac640 1 --2- 172.21.15.125:0/1608995314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f694c006f20 0x7f694c0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:16.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69662ab640 1 -- 172.21.15.125:0/1608995314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69600c6b80 msgr2=0x7f694c0069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:16.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69662ab640 1 --2- 172.21.15.125:0/1608995314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69600c6b80 0x7f694c0069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:16.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69662ab640 1 -- 172.21.15.125:0/1608995314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f694c006f20 msgr2=0x7f694c0073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:16.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69662ab640 1 --2- 172.21.15.125:0/1608995314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f694c006f20 0x7f694c0073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:16.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.822+0000 7f69662ab640 1 -- 172.21.15.125:0/1608995314 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f694c005740 con 0x7f694c005b90 2023-10-30T12:52:16.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.826+0000 7f69662ab640 1 --2- 172.21.15.125:0/1608995314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f694c005b90 0x7f694c005fb0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f695c009ae0 tx=0x7f695c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692dfd706ffd8827 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:16.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.826+0000 7f69652a9640 1 -- 172.21.15.125:0/1608995314 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c0a5530 con 0x7f694c005b90 2023-10-30T12:52:16.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.826+0000 7f69652a9640 1 -- 172.21.15.125:0/1608995314 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695c0a5690 con 0x7f694c005b90 2023-10-30T12:52:16.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.826+0000 7f69652a9640 1 -- 172.21.15.125:0/1608995314 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c0a5a40 con 0x7f694c005b90 2023-10-30T12:52:16.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.826+0000 7f696850f640 1 -- 172.21.15.125:0/1608995314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f694c005b90 msgr2=0x7f694c005fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:16.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.826+0000 7f696850f640 1 --2- 172.21.15.125:0/1608995314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f694c005b90 0x7f694c005fb0 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f695c009ae0 tx=0x7f695c01ca50 comp rx=0 tx=0).stop 2023-10-30T12:52:16.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.826+0000 7f6966aac640 1 -- 172.21.15.125:0/1608995314 reap_dead start 2023-10-30T12:52:16.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.826+0000 7f696850f640 1 -- 172.21.15.125:0/1608995314 shutdown_connections 2023-10-30T12:52:16.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.826+0000 7f696850f640 1 -- 172.21.15.125:0/1608995314 >> 172.21.15.125:0/1608995314 conn(0x7f69600c7910 msgr2=0x7f69600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:16.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.826+0000 7f696850f640 1 -- 172.21.15.125:0/1608995314 shutdown_connections 2023-10-30T12:52:16.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.826+0000 7f696850f640 1 -- 172.21.15.125:0/1608995314 wait complete. 2023-10-30T12:52:16.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f696850f640 1 Processor -- start 2023-10-30T12:52:16.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f696850f640 1 -- start start 2023-10-30T12:52:16.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f696850f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f694c005b90 0x7f6960155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:16.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f696850f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69600c4fa0 0x7f6960158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:16.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f696850f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69601f12a0 0x7f6960136540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:16.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f696850f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6960155430 con 0x7f69600c4fa0 2023-10-30T12:52:16.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f696850f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6960154510 con 0x7f694c005b90 2023-10-30T12:52:16.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f696850f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69601547b0 con 0x7f69601f12a0 2023-10-30T12:52:16.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f6965aaa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69600c4fa0 0x7f6960158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:16.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f6965aaa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69600c4fa0 0x7f6960158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49850/0 (socket says 172.21.15.125:49850) 2023-10-30T12:52:16.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f6965aaa640 1 -- 172.21.15.125:0/2435371044 learned_addr learned my addr 172.21.15.125:0/2435371044 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:16.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f69662ab640 1 --2- 172.21.15.125:0/2435371044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f694c005b90 0x7f6960155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:16.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f6966aac640 1 --2- 172.21.15.125:0/2435371044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69601f12a0 0x7f6960136540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:16.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f6965aaa640 1 -- 172.21.15.125:0/2435371044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69601f12a0 msgr2=0x7f6960136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:16.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f6965aaa640 1 --2- 172.21.15.125:0/2435371044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69601f12a0 0x7f6960136540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:16.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f6965aaa640 1 -- 172.21.15.125:0/2435371044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f694c005b90 msgr2=0x7f6960155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:16.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f6965aaa640 1 --2- 172.21.15.125:0/2435371044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f694c005b90 0x7f6960155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:16.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f6965aaa640 1 -- 172.21.15.125:0/2435371044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6960154a90 con 0x7f69600c4fa0 2023-10-30T12:52:16.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f6965aaa640 1 --2- 172.21.15.125:0/2435371044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69600c4fa0 0x7f6960158d80 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f695000e780 tx=0x7f695000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:16.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f6956ffd640 1 -- 172.21.15.125:0/2435371044 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695000ee30 con 0x7f69600c4fa0 2023-10-30T12:52:16.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f6956ffd640 1 -- 172.21.15.125:0/2435371044 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695000c8d0 con 0x7f69600c4fa0 2023-10-30T12:52:16.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f6956ffd640 1 -- 172.21.15.125:0/2435371044 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695000cc80 con 0x7f69600c4fa0 2023-10-30T12:52:16.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f696850f640 1 -- 172.21.15.125:0/2435371044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6960154d70 con 0x7f69600c4fa0 2023-10-30T12:52:16.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.830+0000 7f696850f640 1 -- 172.21.15.125:0/2435371044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6960155d90 con 0x7f69600c4fa0 2023-10-30T12:52:16.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.834+0000 7f696850f640 1 -- 172.21.15.125:0/2435371044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696004e170 con 0x7f69600c4fa0 2023-10-30T12:52:16.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.834+0000 7f6956ffd640 1 -- 172.21.15.125:0/2435371044 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f695001d460 con 0x7f69600c4fa0 2023-10-30T12:52:16.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.834+0000 7f6956ffd640 1 --2- 172.21.15.125:0/2435371044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69440664a0 0x7f6944068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:16.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.834+0000 7f6956ffd640 1 -- 172.21.15.125:0/2435371044 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7f6950084170 con 0x7f69600c4fa0 2023-10-30T12:52:16.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.834+0000 7f69662ab640 1 --2- 172.21.15.125:0/2435371044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69440664a0 0x7f6944068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:16.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.834+0000 7f69662ab640 1 --2- 172.21.15.125:0/2435371044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69440664a0 0x7f6944068960 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f695c0af000 tx=0x7f695c0046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:16.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:16.838+0000 7f6956ffd640 1 -- 172.21.15.125:0/2435371044 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6950051210 con 0x7f69600c4fa0 2023-10-30T12:52:17.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.030+0000 7f696850f640 1 -- 172.21.15.125:0/2435371044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f696013e790 con 0x7f69600c4fa0 2023-10-30T12:52:17.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.030+0000 7f6956ffd640 1 -- 172.21.15.125:0/2435371044 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f6950054830 con 0x7f69600c4fa0 2023-10-30T12:52:17.033 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:17.033 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6954,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:43.301579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":45,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:17.033 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 117 2023-10-30T12:52:17.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.034+0000 7f696850f640 1 -- 172.21.15.125:0/2435371044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69440664a0 msgr2=0x7f6944068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:17.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.034+0000 7f696850f640 1 --2- 172.21.15.125:0/2435371044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69440664a0 0x7f6944068960 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f695c0af000 tx=0x7f695c0046d0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.034+0000 7f696850f640 1 -- 172.21.15.125:0/2435371044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69600c4fa0 msgr2=0x7f6960158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:17.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.034+0000 7f696850f640 1 --2- 172.21.15.125:0/2435371044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69600c4fa0 0x7f6960158d80 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f695000e780 tx=0x7f695000ec40 comp rx=0 tx=0).stop 2023-10-30T12:52:17.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.034+0000 7f696850f640 1 -- 172.21.15.125:0/2435371044 shutdown_connections 2023-10-30T12:52:17.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.034+0000 7f696850f640 1 --2- 172.21.15.125:0/2435371044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69440664a0 0x7f6944068960 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.034+0000 7f696850f640 1 --2- 172.21.15.125:0/2435371044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69601f12a0 0x7f6960136540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.034+0000 7f696850f640 1 --2- 172.21.15.125:0/2435371044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f69600c4fa0 0x7f6960158d80 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.034+0000 7f696850f640 1 --2- 172.21.15.125:0/2435371044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f694c005b90 0x7f6960155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.034+0000 7f696850f640 1 -- 172.21.15.125:0/2435371044 >> 172.21.15.125:0/2435371044 conn(0x7f69600c7910 msgr2=0x7f69600c1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:17.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.034+0000 7f696850f640 1 -- 172.21.15.125:0/2435371044 shutdown_connections 2023-10-30T12:52:17.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.034+0000 7f696850f640 1 -- 172.21.15.125:0/2435371044 wait complete. 2023-10-30T12:52:17.054 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T12:52:17.055 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T12:52:17.055 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:52:17.076 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T12:52:17.076 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:17.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.250+0000 7ff3b359e640 1 Processor -- start 2023-10-30T12:52:17.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.254+0000 7ff3b359e640 1 -- start start 2023-10-30T12:52:17.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.254+0000 7ff3b359e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff3b41538f0 0x7ff3b4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:17.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.254+0000 7ff3b359e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3b40cefa0 0x7ff3b40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:17.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.254+0000 7ff3b359e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3b40ca6a0 con 0x7ff3b415cd50 2023-10-30T12:52:17.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.254+0000 7ff3b359e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3b415d300 con 0x7ff3b41538f0 2023-10-30T12:52:17.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.254+0000 7ff3b359e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b415d5e0 con 0x7ff3b40cefa0 2023-10-30T12:52:17.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.254+0000 7ff3b259c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff3b41538f0 0x7ff3b4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:17.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.254+0000 7ff3b259c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff3b41538f0 0x7ff3b4159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49864/0 (socket says 172.21.15.125:49864) 2023-10-30T12:52:17.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.254+0000 7ff3b259c640 1 -- 172.21.15.125:0/1698518870 learned_addr learned my addr 172.21.15.125:0/1698518870 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:17.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.254+0000 7ff3b2d9d640 1 --2- 172.21.15.125:0/1698518870 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3b40cefa0 0x7ff3b40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:17.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.254+0000 7ff3b259c640 1 -- 172.21.15.125:0/1698518870 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3b40cefa0 msgr2=0x7ff3b40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:17.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.254+0000 7ff3b259c640 1 --2- 172.21.15.125:0/1698518870 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3b40cefa0 0x7ff3b40cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.254+0000 7ff3b259c640 1 -- 172.21.15.125:0/1698518870 >> v1:172.21.15.125:6789/0 conn(0x7ff3b415cd50 legacy=0x7ff3b40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:17.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.254+0000 7ff3b259c640 1 -- 172.21.15.125:0/1698518870 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b415d8c0 con 0x7ff3b41538f0 2023-10-30T12:52:17.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.258+0000 7ff3b259c640 1 --2- 172.21.15.125:0/1698518870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3b41538f0 0x7ff3b4159fb0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4009870 tx=0x7ff3a401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbeb76666faa0eca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:17.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.258+0000 7ff3b1d9b640 1 -- 172.21.15.125:0/1698518870 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a40a8a30 con 0x7ff3b41538f0 2023-10-30T12:52:17.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.258+0000 7ff3b1d9b640 1 -- 172.21.15.125:0/1698518870 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a40a8b90 con 0x7ff3b41538f0 2023-10-30T12:52:17.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.258+0000 7ff3b1d9b640 1 -- 172.21.15.125:0/1698518870 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a40b0440 con 0x7ff3b41538f0 2023-10-30T12:52:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.258+0000 7ff3b359e640 1 -- 172.21.15.125:0/1698518870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3b41538f0 msgr2=0x7ff3b4159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.258+0000 7ff3b359e640 1 --2- 172.21.15.125:0/1698518870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3b41538f0 0x7ff3b4159fb0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4009870 tx=0x7ff3a401ed30 comp rx=0 tx=0).stop 2023-10-30T12:52:17.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.258+0000 7ff3b359e640 1 -- 172.21.15.125:0/1698518870 shutdown_connections 2023-10-30T12:52:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.258+0000 7ff3b359e640 1 --2- 172.21.15.125:0/1698518870 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3b40cefa0 0x7ff3b40cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.258+0000 7ff3b359e640 1 --2- 172.21.15.125:0/1698518870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3b41538f0 0x7ff3b4159fb0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.258+0000 7ff3b359e640 1 -- 172.21.15.125:0/1698518870 >> 172.21.15.125:0/1698518870 conn(0x7ff3b406a640 msgr2=0x7ff3b4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.258+0000 7ff3b359e640 1 -- 172.21.15.125:0/1698518870 shutdown_connections 2023-10-30T12:52:17.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.258+0000 7ff3b359e640 1 -- 172.21.15.125:0/1698518870 wait complete. 2023-10-30T12:52:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b359e640 1 Processor -- start 2023-10-30T12:52:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b359e640 1 -- start start 2023-10-30T12:52:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b359e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3b40cefa0 0x7ff3b40ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b359e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3b41538f0 0x7ff3b41ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:17.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b359e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3b415cd50 0x7ff3b4145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:17.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b359e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b415a9c0 con 0x7ff3b40cefa0 2023-10-30T12:52:17.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b359e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b41590c0 con 0x7ff3b415cd50 2023-10-30T12:52:17.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b359e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b415ac60 con 0x7ff3b41538f0 2023-10-30T12:52:17.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b259c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3b40cefa0 0x7ff3b40ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:17.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b259c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3b40cefa0 0x7ff3b40ce1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49868/0 (socket says 172.21.15.125:49868) 2023-10-30T12:52:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b259c640 1 -- 172.21.15.125:0/2140622266 learned_addr learned my addr 172.21.15.125:0/2140622266 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3a9d9b640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3b41538f0 0x7ff3b41ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b2d9d640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3b415cd50 0x7ff3b4145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b259c640 1 -- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3b41538f0 msgr2=0x7ff3b41ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:17.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b259c640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3b41538f0 0x7ff3b41ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b259c640 1 -- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3b415cd50 msgr2=0x7ff3b4145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b259c640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3b415cd50 0x7ff3b4145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b259c640 1 -- 172.21.15.125:0/2140622266 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a40094d0 con 0x7ff3b40cefa0 2023-10-30T12:52:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3a9d9b640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3b41538f0 0x7ff3b41ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:52:17.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b259c640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3b40cefa0 0x7ff3b40ce1c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7ff3a40b3040 tx=0x7ff3a401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:17.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3ab7fe640 1 -- 172.21.15.125:0/2140622266 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a40be070 con 0x7ff3b40cefa0 2023-10-30T12:52:17.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3ab7fe640 1 -- 172.21.15.125:0/2140622266 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a40ba400 con 0x7ff3b40cefa0 2023-10-30T12:52:17.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b359e640 1 -- 172.21.15.125:0/2140622266 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b415af40 con 0x7ff3b40cefa0 2023-10-30T12:52:17.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3ab7fe640 1 -- 172.21.15.125:0/2140622266 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a40ba7d0 con 0x7ff3b40cefa0 2023-10-30T12:52:17.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b359e640 1 -- 172.21.15.125:0/2140622266 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b41ef420 con 0x7ff3b40cefa0 2023-10-30T12:52:17.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.262+0000 7ff3b359e640 1 -- 172.21.15.125:0/2140622266 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3b404e170 con 0x7ff3b40cefa0 2023-10-30T12:52:17.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.266+0000 7ff3ab7fe640 1 -- 172.21.15.125:0/2140622266 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff3a40c3410 con 0x7ff3b40cefa0 2023-10-30T12:52:17.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.266+0000 7ff3ab7fe640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff3940664a0 0x7ff394068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:17.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.266+0000 7ff3ab7fe640 1 -- 172.21.15.125:0/2140622266 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7ff3a4121e60 con 0x7ff3b40cefa0 2023-10-30T12:52:17.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.266+0000 7ff3a9d9b640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff3940664a0 0x7ff394068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:17.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.266+0000 7ff3a9d9b640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff3940664a0 0x7ff394068960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff3b415cc40 tx=0x7ff39c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:17.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.270+0000 7ff3ab7fe640 1 -- 172.21.15.125:0/2140622266 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff3a40eef00 con 0x7ff3b40cefa0 2023-10-30T12:52:17.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.450+0000 7ff3b359e640 1 -- 172.21.15.125:0/2140622266 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3b41ef610 con 0x7ff3b40cefa0 2023-10-30T12:52:17.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.454+0000 7ff3ab7fe640 1 -- 172.21.15.125:0/2140622266 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+6519 (secure 0 0 0) 0x7ff3a40f2520 con 0x7ff3b40cefa0 2023-10-30T12:52:17.454 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:17.454 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6954,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:43.301579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":45,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:17.454 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 117 2023-10-30T12:52:17.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.454+0000 7ff3b359e640 1 -- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff3940664a0 msgr2=0x7ff394068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:17.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.454+0000 7ff3b359e640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff3940664a0 0x7ff394068960 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ff3b415cc40 tx=0x7ff39c009040 comp rx=0 tx=0).stop 2023-10-30T12:52:17.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.454+0000 7ff3b359e640 1 -- 172.21.15.125:0/2140622266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3b40cefa0 msgr2=0x7ff3b40ce1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:17.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.454+0000 7ff3b359e640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3b40cefa0 0x7ff3b40ce1c0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7ff3a40b3040 tx=0x7ff3a401bad0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.454+0000 7ff3b359e640 1 -- 172.21.15.125:0/2140622266 shutdown_connections 2023-10-30T12:52:17.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.454+0000 7ff3b359e640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff3940664a0 0x7ff394068960 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.454+0000 7ff3b359e640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff3b415cd50 0x7ff3b4145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.454+0000 7ff3b359e640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff3b41538f0 0x7ff3b41ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.454+0000 7ff3b359e640 1 --2- 172.21.15.125:0/2140622266 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff3b40cefa0 0x7ff3b40ce1c0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:17.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.454+0000 7ff3b359e640 1 -- 172.21.15.125:0/2140622266 >> 172.21.15.125:0/2140622266 conn(0x7ff3b406a640 msgr2=0x7ff3b4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:17.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.454+0000 7ff3b359e640 1 -- 172.21.15.125:0/2140622266 shutdown_connections 2023-10-30T12:52:17.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:17.454+0000 7ff3b359e640 1 -- 172.21.15.125:0/2140622266 wait complete. 2023-10-30T12:52:17.474 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:52:18.030 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:52:18.035 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:52:19.475 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:19.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5fbd9d640 1 Processor -- start 2023-10-30T12:52:19.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5fbd9d640 1 -- start start 2023-10-30T12:52:19.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5fbd9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5f41538a0 0x7ff5f415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:19.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5fbd9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff5f40ca0e0 0x7ff5f415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:19.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5fbd9d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f40cb800 con 0x7ff5f40cf160 2023-10-30T12:52:19.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5fbd9d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f415dc30 con 0x7ff5f40ca0e0 2023-10-30T12:52:19.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5fbd9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40cb330 con 0x7ff5f41538a0 2023-10-30T12:52:19.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5fa33a640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff5f40cf160 0x7ff5f40cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47736/0 (socket says 172.21.15.125:47736) 2023-10-30T12:52:19.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5fa33a640 1 -- 172.21.15.125:0/378178714 learned_addr learned my addr 172.21.15.125:0/378178714 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:52:19.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f9b39640 1 --2- 172.21.15.125:0/378178714 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5f41538a0 0x7ff5f415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:19.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f9338640 1 --2- 172.21.15.125:0/378178714 >> v2:172.21.15.136:3300/0 conn(0x7ff5f40ca0e0 0x7ff5f415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:19.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 -- 172.21.15.125:0/378178714 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12819184 0 0) 0x7ff5f40cb800 con 0x7ff5f40cf160 2023-10-30T12:52:19.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 -- 172.21.15.125:0/378178714 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5e0003530 con 0x7ff5f40cf160 2023-10-30T12:52:19.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 -- 172.21.15.125:0/378178714 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3714574482 0 0) 0x7ff5e0003530 con 0x7ff5f40cf160 2023-10-30T12:52:19.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 -- 172.21.15.125:0/378178714 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5f41538a0 msgr2=0x7ff5f415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:19.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 --2- 172.21.15.125:0/378178714 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5f41538a0 0x7ff5f415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:19.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 -- 172.21.15.125:0/378178714 >> v2:172.21.15.136:3300/0 conn(0x7ff5f40ca0e0 msgr2=0x7ff5f415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:19.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 --2- 172.21.15.125:0/378178714 >> v2:172.21.15.136:3300/0 conn(0x7ff5f40ca0e0 0x7ff5f415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:19.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 -- 172.21.15.125:0/378178714 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f40cad70 con 0x7ff5f40cf160 2023-10-30T12:52:19.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 -- 172.21.15.125:0/378178714 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7ff5ec001a00 con 0x7ff5f40cf160 2023-10-30T12:52:19.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 -- 172.21.15.125:0/378178714 >> v1:172.21.15.125:6789/0 conn(0x7ff5f40cf160 legacy=0x7ff5f40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:19.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 --2- 172.21.15.125:0/378178714 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5f41538a0 0x7ff5e0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:19.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 --2- 172.21.15.125:0/378178714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e0003f00 0x7ff5e00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:19.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 --2- 172.21.15.125:0/378178714 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5e0004d10 0x7ff5e0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:19.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 -- 172.21.15.125:0/378178714 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f415dc30 con 0x7ff5e0004d10 2023-10-30T12:52:19.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 -- 172.21.15.125:0/378178714 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f40cb330 con 0x7ff5e0003f00 2023-10-30T12:52:19.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f8b37640 1 -- 172.21.15.125:0/378178714 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e000b9c0 con 0x7ff5f41538a0 2023-10-30T12:52:19.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5fa33a640 1 --2- 172.21.15.125:0/378178714 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5e0004d10 0x7ff5e0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:19.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f9b39640 1 --2- 172.21.15.125:0/378178714 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5f41538a0 0x7ff5e0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:19.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.626+0000 7ff5f9338640 1 --2- 172.21.15.125:0/378178714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e0003f00 0x7ff5e00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:19.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.630+0000 7ff5fa33a640 1 -- 172.21.15.125:0/378178714 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5f41538a0 msgr2=0x7ff5e0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:19.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.630+0000 7ff5fa33a640 1 --2- 172.21.15.125:0/378178714 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5f41538a0 0x7ff5e0003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:19.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.630+0000 7ff5fa33a640 1 -- 172.21.15.125:0/378178714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e0003f00 msgr2=0x7ff5e00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:19.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.630+0000 7ff5fa33a640 1 --2- 172.21.15.125:0/378178714 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5e0003f00 0x7ff5e00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:19.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.630+0000 7ff5fa33a640 1 -- 172.21.15.125:0/378178714 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e0003530 con 0x7ff5e0004d10 2023-10-30T12:52:19.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.630+0000 7ff5fa33a640 1 --2- 172.21.15.125:0/378178714 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5e0004d10 0x7ff5e0005190 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00a860 tx=0x7ff5ec01d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6da20b9ba06f17 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:19.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.630+0000 7ff5f8b37640 1 -- 172.21.15.125:0/378178714 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec0ad630 con 0x7ff5e0004d10 2023-10-30T12:52:19.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.630+0000 7ff5f8b37640 1 -- 172.21.15.125:0/378178714 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec0a8070 con 0x7ff5e0004d10 2023-10-30T12:52:19.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.630+0000 7ff5f8b37640 1 -- 172.21.15.125:0/378178714 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec0ada30 con 0x7ff5e0004d10 2023-10-30T12:52:19.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.630+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/378178714 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5e0004d10 msgr2=0x7ff5e0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:19.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.630+0000 7ff5fbd9d640 1 --2- 172.21.15.125:0/378178714 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5e0004d10 0x7ff5e0005190 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00a860 tx=0x7ff5ec01d360 comp rx=0 tx=0).stop 2023-10-30T12:52:19.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.630+0000 7ff5fa33a640 1 -- 172.21.15.125:0/378178714 reap_dead start 2023-10-30T12:52:19.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.630+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/378178714 shutdown_connections 2023-10-30T12:52:19.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.630+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/378178714 >> 172.21.15.125:0/378178714 conn(0x7ff5f406bfe0 msgr2=0x7ff5f406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:19.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/378178714 shutdown_connections 2023-10-30T12:52:19.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/378178714 wait complete. 2023-10-30T12:52:19.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fbd9d640 1 Processor -- start 2023-10-30T12:52:19.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fbd9d640 1 -- start start 2023-10-30T12:52:19.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fbd9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5f40b43c0 0x7ff5f414e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:19.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fbd9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5f40b47a0 0x7ff5f41ef1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:19.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fbd9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5f41ef720 0x7ff5f4145960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:19.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fbd9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f415a8b0 con 0x7ff5f41ef720 2023-10-30T12:52:19.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fbd9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f415b150 con 0x7ff5f40b43c0 2023-10-30T12:52:19.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fbd9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f41efcc0 con 0x7ff5f40b47a0 2023-10-30T12:52:19.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fa33a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5f41ef720 0x7ff5f4145960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fa33a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5f41ef720 0x7ff5f4145960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49888/0 (socket says 172.21.15.125:49888) 2023-10-30T12:52:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fa33a640 1 -- 172.21.15.125:0/1658839543 learned_addr learned my addr 172.21.15.125:0/1658839543 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5f9338640 1 --2- 172.21.15.125:0/1658839543 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5f40b47a0 0x7ff5f41ef1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:19.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5f9b39640 1 --2- 172.21.15.125:0/1658839543 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5f40b43c0 0x7ff5f414e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:19.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fa33a640 1 -- 172.21.15.125:0/1658839543 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5f40b47a0 msgr2=0x7ff5f41ef1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:19.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fa33a640 1 --2- 172.21.15.125:0/1658839543 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5f40b47a0 0x7ff5f41ef1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:19.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fa33a640 1 -- 172.21.15.125:0/1658839543 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5f40b43c0 msgr2=0x7ff5f414e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:19.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fa33a640 1 --2- 172.21.15.125:0/1658839543 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5f40b43c0 0x7ff5f414e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:19.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.634+0000 7ff5fa33a640 1 -- 172.21.15.125:0/1658839543 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f41effa0 con 0x7ff5f41ef720 2023-10-30T12:52:19.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.638+0000 7ff5fa33a640 1 --2- 172.21.15.125:0/1658839543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5f41ef720 0x7ff5f4145960 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec01cdd0 tx=0x7ff5e0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:19.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.638+0000 7ff5ea7fc640 1 -- 172.21.15.125:0/1658839543 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec0019d0 con 0x7ff5f41ef720 2023-10-30T12:52:19.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.638+0000 7ff5ea7fc640 1 -- 172.21.15.125:0/1658839543 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec0ad430 con 0x7ff5f41ef720 2023-10-30T12:52:19.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.638+0000 7ff5ea7fc640 1 -- 172.21.15.125:0/1658839543 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec0ad7a0 con 0x7ff5f41ef720 2023-10-30T12:52:19.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.638+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/1658839543 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f415ab00 con 0x7ff5f41ef720 2023-10-30T12:52:19.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.638+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/1658839543 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f415b5e0 con 0x7ff5f41ef720 2023-10-30T12:52:19.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.638+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/1658839543 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f40b5010 con 0x7ff5f41ef720 2023-10-30T12:52:19.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.638+0000 7ff5ea7fc640 1 -- 172.21.15.125:0/1658839543 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5ec0aa030 con 0x7ff5f41ef720 2023-10-30T12:52:19.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.638+0000 7ff5ea7fc640 1 --2- 172.21.15.125:0/1658839543 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5d80664d0 0x7ff5d8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:19.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.638+0000 7ff5ea7fc640 1 -- 172.21.15.125:0/1658839543 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7ff5ec11e6d0 con 0x7ff5f41ef720 2023-10-30T12:52:19.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.638+0000 7ff5f9b39640 1 --2- 172.21.15.125:0/1658839543 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5d80664d0 0x7ff5d8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:19.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.642+0000 7ff5f9b39640 1 --2- 172.21.15.125:0/1658839543 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5d80664d0 0x7ff5d8068990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009b20 tx=0x7ff5f0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:19.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.642+0000 7ff5ea7fc640 1 -- 172.21.15.125:0/1658839543 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5ec0eb770 con 0x7ff5f41ef720 2023-10-30T12:52:19.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.822+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/1658839543 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5f40b5010 con 0x7ff5f41ef720 2023-10-30T12:52:19.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.822+0000 7ff5ea7fc640 1 -- 172.21.15.125:0/1658839543 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+6519 (secure 0 0 0) 0x7ff5ec0eed90 con 0x7ff5f41ef720 2023-10-30T12:52:19.826 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:19.826 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6954,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:43.301579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":45,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:19.828 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 117 2023-10-30T12:52:19.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.826+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/1658839543 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5d80664d0 msgr2=0x7ff5d8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:19.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.826+0000 7ff5fbd9d640 1 --2- 172.21.15.125:0/1658839543 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5d80664d0 0x7ff5d8068990 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009b20 tx=0x7ff5f0008040 comp rx=0 tx=0).stop 2023-10-30T12:52:19.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.826+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/1658839543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5f41ef720 msgr2=0x7ff5f4145960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:19.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.826+0000 7ff5fbd9d640 1 --2- 172.21.15.125:0/1658839543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5f41ef720 0x7ff5f4145960 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec01cdd0 tx=0x7ff5e0005800 comp rx=0 tx=0).stop 2023-10-30T12:52:19.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.826+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/1658839543 shutdown_connections 2023-10-30T12:52:19.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.826+0000 7ff5fbd9d640 1 --2- 172.21.15.125:0/1658839543 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5d80664d0 0x7ff5d8068990 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:19.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.826+0000 7ff5fbd9d640 1 --2- 172.21.15.125:0/1658839543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5f41ef720 0x7ff5f4145960 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:19.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.826+0000 7ff5fbd9d640 1 --2- 172.21.15.125:0/1658839543 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5f40b47a0 0x7ff5f41ef1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:19.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.826+0000 7ff5fbd9d640 1 --2- 172.21.15.125:0/1658839543 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5f40b43c0 0x7ff5f414e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:19.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.826+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/1658839543 >> 172.21.15.125:0/1658839543 conn(0x7ff5f406bfe0 msgr2=0x7ff5f4151c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:19.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.826+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/1658839543 shutdown_connections 2023-10-30T12:52:19.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:19.826+0000 7ff5fbd9d640 1 -- 172.21.15.125:0/1658839543 wait complete. 2023-10-30T12:52:19.846 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:52:21.847 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:21.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.990+0000 7f14ce009640 1 Processor -- start 2023-10-30T12:52:21.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.994+0000 7f14ce009640 1 -- start start 2023-10-30T12:52:21.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.994+0000 7f14ce009640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f14c81538f0 0x7f14c8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:21.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.994+0000 7f14ce009640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f14c80cefa0 0x7f14c80cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:21.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.994+0000 7f14ce009640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c80ca6a0 con 0x7f14c815cd50 2023-10-30T12:52:21.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.994+0000 7f14ce009640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c80cf8f0 con 0x7f14c81538f0 2023-10-30T12:52:21.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.994+0000 7f14ce009640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14c815d300 con 0x7f14c80cefa0 2023-10-30T12:52:21.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.994+0000 7f14cd007640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f14c81538f0 0x7f14c8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:21.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.994+0000 7f14cd007640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f14c81538f0 0x7f14c8159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52196/0 (socket says 172.21.15.125:52196) 2023-10-30T12:52:21.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.994+0000 7f14cd007640 1 -- 172.21.15.125:0/3557849070 learned_addr learned my addr 172.21.15.125:0/3557849070 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:21.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.994+0000 7f14cd808640 1 --2- 172.21.15.125:0/3557849070 >> v2:172.21.15.136:3300/0 conn(0x7f14c80cefa0 0x7f14c80cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:21.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.994+0000 7f14cd007640 1 -- 172.21.15.125:0/3557849070 >> v2:172.21.15.136:3300/0 conn(0x7f14c80cefa0 msgr2=0x7f14c80cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:21.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.994+0000 7f14cd007640 1 --2- 172.21.15.125:0/3557849070 >> v2:172.21.15.136:3300/0 conn(0x7f14c80cefa0 0x7f14c80cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:21.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.994+0000 7f14cd007640 1 -- 172.21.15.125:0/3557849070 >> v1:172.21.15.136:6789/0 conn(0x7f14c815cd50 legacy=0x7f14c80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:21.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.994+0000 7f14cd007640 1 -- 172.21.15.125:0/3557849070 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c815d5e0 con 0x7f14c81538f0 2023-10-30T12:52:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.998+0000 7f14cd007640 1 --2- 172.21.15.125:0/3557849070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14c81538f0 0x7f14c8159fb0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f14c8066a40 tx=0x7f14c401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1936749da76aeb30 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.998+0000 7f14bffff640 1 -- 172.21.15.125:0/3557849070 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c40a89b0 con 0x7f14c81538f0 2023-10-30T12:52:21.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.998+0000 7f14bffff640 1 -- 172.21.15.125:0/3557849070 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c40a8b10 con 0x7f14c81538f0 2023-10-30T12:52:21.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.998+0000 7f14bffff640 1 -- 172.21.15.125:0/3557849070 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c40a8ea0 con 0x7f14c81538f0 2023-10-30T12:52:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.998+0000 7f14ce009640 1 -- 172.21.15.125:0/3557849070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14c81538f0 msgr2=0x7f14c8159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.998+0000 7f14ce009640 1 --2- 172.21.15.125:0/3557849070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14c81538f0 0x7f14c8159fb0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f14c8066a40 tx=0x7f14c401ecb0 comp rx=0 tx=0).stop 2023-10-30T12:52:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.998+0000 7f14ce009640 1 -- 172.21.15.125:0/3557849070 shutdown_connections 2023-10-30T12:52:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.998+0000 7f14ce009640 1 --2- 172.21.15.125:0/3557849070 >> v2:172.21.15.136:3300/0 conn(0x7f14c80cefa0 0x7f14c80cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:22.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.998+0000 7f14ce009640 1 --2- 172.21.15.125:0/3557849070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14c81538f0 0x7f14c8159fb0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:22.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.998+0000 7f14ce009640 1 -- 172.21.15.125:0/3557849070 >> 172.21.15.125:0/3557849070 conn(0x7f14c806a640 msgr2=0x7f14c8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:22.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.998+0000 7f14ce009640 1 -- 172.21.15.125:0/3557849070 shutdown_connections 2023-10-30T12:52:22.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.998+0000 7f14ce009640 1 -- 172.21.15.125:0/3557849070 wait complete. 2023-10-30T12:52:22.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.998+0000 7f14ce009640 1 Processor -- start 2023-10-30T12:52:22.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:21.998+0000 7f14ce009640 1 -- start start 2023-10-30T12:52:22.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14ce009640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14c80cefa0 0x7f14c8149660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14ce009640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14c81538f0 0x7f14c81583e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14ce009640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14c815cd50 0x7f14c8158920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14ce009640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c8152ae0 con 0x7f14c815cd50 2023-10-30T12:52:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14ce009640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c81efbc0 con 0x7f14c80cefa0 2023-10-30T12:52:22.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14ce009640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14c81f00f0 con 0x7f14c81538f0 2023-10-30T12:52:22.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14cd808640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14c815cd50 0x7f14c8158920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:22.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14cd808640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14c815cd50 0x7f14c8158920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52202/0 (socket says 172.21.15.125:52202) 2023-10-30T12:52:22.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14cd808640 1 -- 172.21.15.125:0/3895120339 learned_addr learned my addr 172.21.15.125:0/3895120339 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:22.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14cd007640 1 --2- 172.21.15.125:0/3895120339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14c80cefa0 0x7f14c8149660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:22.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14cc806640 1 --2- 172.21.15.125:0/3895120339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14c81538f0 0x7f14c81583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:22.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14cd808640 1 -- 172.21.15.125:0/3895120339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14c81538f0 msgr2=0x7f14c81583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:22.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14cd808640 1 --2- 172.21.15.125:0/3895120339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14c81538f0 0x7f14c81583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:22.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14cd808640 1 -- 172.21.15.125:0/3895120339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14c80cefa0 msgr2=0x7f14c8149660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:22.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14cd808640 1 --2- 172.21.15.125:0/3895120339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14c80cefa0 0x7f14c8149660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:22.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14cd808640 1 -- 172.21.15.125:0/3895120339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c40094d0 con 0x7f14c815cd50 2023-10-30T12:52:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14cd808640 1 --2- 172.21.15.125:0/3895120339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14c815cd50 0x7f14c8158920 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f14c0007e40 tx=0x7f14c000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14bdffb640 1 -- 172.21.15.125:0/3895120339 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c0014620 con 0x7f14c815cd50 2023-10-30T12:52:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14bdffb640 1 -- 172.21.15.125:0/3895120339 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c000ee90 con 0x7f14c815cd50 2023-10-30T12:52:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14bdffb640 1 -- 172.21.15.125:0/3895120339 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14c00149e0 con 0x7f14c815cd50 2023-10-30T12:52:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14ce009640 1 -- 172.21.15.125:0/3895120339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c8157f50 con 0x7f14c815cd50 2023-10-30T12:52:22.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14ce009640 1 -- 172.21.15.125:0/3895120339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c8157900 con 0x7f14c815cd50 2023-10-30T12:52:22.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.002+0000 7f14ce009640 1 -- 172.21.15.125:0/3895120339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c81580e0 con 0x7f14c815cd50 2023-10-30T12:52:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.006+0000 7f14bdffb640 1 -- 172.21.15.125:0/3895120339 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f14c0014b40 con 0x7f14c815cd50 2023-10-30T12:52:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.006+0000 7f14bdffb640 1 --2- 172.21.15.125:0/3895120339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14ac0664a0 0x7f14ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.006+0000 7f14bdffb640 1 -- 172.21.15.125:0/3895120339 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7f14c0084be0 con 0x7f14c815cd50 2023-10-30T12:52:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.006+0000 7f14cd007640 1 --2- 172.21.15.125:0/3895120339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14ac0664a0 0x7f14ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:22.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.006+0000 7f14cd007640 1 --2- 172.21.15.125:0/3895120339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14ac0664a0 0x7f14ac068960 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f14c40b2000 tx=0x7f14c4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:22.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.010+0000 7f14bdffb640 1 -- 172.21.15.125:0/3895120339 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f14c0051bf0 con 0x7f14c815cd50 2023-10-30T12:52:22.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.190+0000 7f14ce009640 1 -- 172.21.15.125:0/3895120339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14c804e170 con 0x7f14c815cd50 2023-10-30T12:52:22.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.194+0000 7f14bdffb640 1 -- 172.21.15.125:0/3895120339 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f14c0055210 con 0x7f14c815cd50 2023-10-30T12:52:22.194 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:22.194 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6954,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:43.301579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":45,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:22.195 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 117 2023-10-30T12:52:22.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.194+0000 7f14ce009640 1 -- 172.21.15.125:0/3895120339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14ac0664a0 msgr2=0x7f14ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:22.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.194+0000 7f14ce009640 1 --2- 172.21.15.125:0/3895120339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14ac0664a0 0x7f14ac068960 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f14c40b2000 tx=0x7f14c4009040 comp rx=0 tx=0).stop 2023-10-30T12:52:22.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.194+0000 7f14ce009640 1 -- 172.21.15.125:0/3895120339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14c815cd50 msgr2=0x7f14c8158920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:22.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.194+0000 7f14ce009640 1 --2- 172.21.15.125:0/3895120339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14c815cd50 0x7f14c8158920 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f14c0007e40 tx=0x7f14c000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:52:22.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.194+0000 7f14ce009640 1 -- 172.21.15.125:0/3895120339 shutdown_connections 2023-10-30T12:52:22.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.194+0000 7f14ce009640 1 --2- 172.21.15.125:0/3895120339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f14ac0664a0 0x7f14ac068960 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:22.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.194+0000 7f14ce009640 1 --2- 172.21.15.125:0/3895120339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f14c815cd50 0x7f14c8158920 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:22.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.194+0000 7f14ce009640 1 --2- 172.21.15.125:0/3895120339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f14c81538f0 0x7f14c81583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:22.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.194+0000 7f14ce009640 1 --2- 172.21.15.125:0/3895120339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f14c80cefa0 0x7f14c8149660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:22.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.194+0000 7f14ce009640 1 -- 172.21.15.125:0/3895120339 >> 172.21.15.125:0/3895120339 conn(0x7f14c806a640 msgr2=0x7f14c80ce610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:22.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.194+0000 7f14ce009640 1 -- 172.21.15.125:0/3895120339 shutdown_connections 2023-10-30T12:52:22.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:22.194+0000 7f14ce009640 1 -- 172.21.15.125:0/3895120339 wait complete. 2023-10-30T12:52:22.215 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:52:24.216 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:24.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc591cf7640 1 Processor -- start 2023-10-30T12:52:24.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc591cf7640 1 -- start start 2023-10-30T12:52:24.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc591cf7640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c05d0 0x7fc58c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:24.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc591cf7640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc58c0c6f60 0x7fc58c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:24.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc591cf7640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c0c0a50 con 0x7fc58c0c6b80 2023-10-30T12:52:24.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc591cf7640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58c06f6e0 con 0x7fc58c0c6f60 2023-10-30T12:52:24.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc591cf7640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c06f9c0 con 0x7fc58c0c05d0 2023-10-30T12:52:24.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc58bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc58c0c6f60 0x7fc58c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:24.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc58bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc58c0c6f60 0x7fc58c13f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52208/0 (socket says 172.21.15.125:52208) 2023-10-30T12:52:24.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc58bfff640 1 -- 172.21.15.125:0/751686834 learned_addr learned my addr 172.21.15.125:0/751686834 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:24.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc58affd640 1 --2- 172.21.15.125:0/751686834 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c05d0 0x7fc58c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:24.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc58bfff640 1 -- 172.21.15.125:0/751686834 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c05d0 msgr2=0x7fc58c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:24.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc58bfff640 1 --2- 172.21.15.125:0/751686834 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c05d0 0x7fc58c13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:24.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc58bfff640 1 -- 172.21.15.125:0/751686834 >> v1:172.21.15.125:6789/0 conn(0x7fc58c0c6b80 legacy=0x7fc58c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:24.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.362+0000 7fc58bfff640 1 -- 172.21.15.125:0/751686834 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c06fca0 con 0x7fc58c0c6f60 2023-10-30T12:52:24.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.366+0000 7fc58bfff640 1 --2- 172.21.15.125:0/751686834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c6f60 0x7fc58c13f2d0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fc584009e50 tx=0x7fc58401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9459692122a6aa40 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:24.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.366+0000 7fc58a7fc640 1 -- 172.21.15.125:0/751686834 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5840a57f0 con 0x7fc58c0c6f60 2023-10-30T12:52:24.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.366+0000 7fc58a7fc640 1 -- 172.21.15.125:0/751686834 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5840a5950 con 0x7fc58c0c6f60 2023-10-30T12:52:24.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.366+0000 7fc58a7fc640 1 -- 172.21.15.125:0/751686834 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5840a5ce0 con 0x7fc58c0c6f60 2023-10-30T12:52:24.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.366+0000 7fc591cf7640 1 -- 172.21.15.125:0/751686834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c6f60 msgr2=0x7fc58c13f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:24.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.366+0000 7fc591cf7640 1 --2- 172.21.15.125:0/751686834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c6f60 0x7fc58c13f2d0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fc584009e50 tx=0x7fc58401ca10 comp rx=0 tx=0).stop 2023-10-30T12:52:24.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.366+0000 7fc591cf7640 1 -- 172.21.15.125:0/751686834 shutdown_connections 2023-10-30T12:52:24.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.366+0000 7fc591cf7640 1 --2- 172.21.15.125:0/751686834 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c6f60 0x7fc58c13f2d0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:24.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.366+0000 7fc591cf7640 1 --2- 172.21.15.125:0/751686834 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c05d0 0x7fc58c13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:24.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.366+0000 7fc591cf7640 1 -- 172.21.15.125:0/751686834 >> 172.21.15.125:0/751686834 conn(0x7fc58c0c7910 msgr2=0x7fc58c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:24.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.366+0000 7fc591cf7640 1 -- 172.21.15.125:0/751686834 shutdown_connections 2023-10-30T12:52:24.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.366+0000 7fc591cf7640 1 -- 172.21.15.125:0/751686834 wait complete. 2023-10-30T12:52:24.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc591cf7640 1 Processor -- start 2023-10-30T12:52:24.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc591cf7640 1 -- start start 2023-10-30T12:52:24.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc591cf7640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c05d0 0x7fc58c06ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:24.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc591cf7640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c6b80 0x7fc58c1f50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:24.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc591cf7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58c0c6f60 0x7fc58c163790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:24.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc591cf7640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c1330a0 con 0x7fc58c0c6b80 2023-10-30T12:52:24.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc591cf7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c1f5730 con 0x7fc58c0c6f60 2023-10-30T12:52:24.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc591cf7640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c1f5fb0 con 0x7fc58c0c05d0 2023-10-30T12:52:24.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc58affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c6b80 0x7fc58c1f50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:24.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc58affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c6b80 0x7fc58c1f50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52218/0 (socket says 172.21.15.125:52218) 2023-10-30T12:52:24.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc58affd640 1 -- 172.21.15.125:0/1275361680 learned_addr learned my addr 172.21.15.125:0/1275361680 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:24.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc58bfff640 1 --2- 172.21.15.125:0/1275361680 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58c0c6f60 0x7fc58c163790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:24.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc58b7fe640 1 --2- 172.21.15.125:0/1275361680 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c05d0 0x7fc58c06ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:24.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc58affd640 1 -- 172.21.15.125:0/1275361680 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c05d0 msgr2=0x7fc58c06ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:24.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc58affd640 1 --2- 172.21.15.125:0/1275361680 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c05d0 0x7fc58c06ac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:24.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc58affd640 1 -- 172.21.15.125:0/1275361680 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58c0c6f60 msgr2=0x7fc58c163790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:24.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc58affd640 1 --2- 172.21.15.125:0/1275361680 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58c0c6f60 0x7fc58c163790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:24.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc58affd640 1 -- 172.21.15.125:0/1275361680 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc584009b30 con 0x7fc58c0c6b80 2023-10-30T12:52:24.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc58affd640 1 --2- 172.21.15.125:0/1275361680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c6b80 0x7fc58c1f50c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fc58000d800 tx=0x7fc58000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:24.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc573fff640 1 -- 172.21.15.125:0/1275361680 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58000b6c0 con 0x7fc58c0c6b80 2023-10-30T12:52:24.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc573fff640 1 -- 172.21.15.125:0/1275361680 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58000b820 con 0x7fc58c0c6b80 2023-10-30T12:52:24.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc573fff640 1 -- 172.21.15.125:0/1275361680 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58000bb90 con 0x7fc58c0c6b80 2023-10-30T12:52:24.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc591cf7640 1 -- 172.21.15.125:0/1275361680 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58c1f59f0 con 0x7fc58c0c6b80 2023-10-30T12:52:24.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc591cf7640 1 -- 172.21.15.125:0/1275361680 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58c154540 con 0x7fc58c0c6b80 2023-10-30T12:52:24.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.370+0000 7fc591cf7640 1 -- 172.21.15.125:0/1275361680 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58c04e170 con 0x7fc58c0c6b80 2023-10-30T12:52:24.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.374+0000 7fc573fff640 1 -- 172.21.15.125:0/1275361680 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc58000bcf0 con 0x7fc58c0c6b80 2023-10-30T12:52:24.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.374+0000 7fc573fff640 1 --2- 172.21.15.125:0/1275361680 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5780664a0 0x7fc578068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:24.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.374+0000 7fc573fff640 1 -- 172.21.15.125:0/1275361680 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7fc5800848f0 con 0x7fc58c0c6b80 2023-10-30T12:52:24.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.374+0000 7fc58b7fe640 1 --2- 172.21.15.125:0/1275361680 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5780664a0 0x7fc578068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:24.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.374+0000 7fc58b7fe640 1 --2- 172.21.15.125:0/1275361680 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5780664a0 0x7fc578068960 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc57c000f70 tx=0x7fc57c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:24.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.378+0000 7fc573fff640 1 -- 172.21.15.125:0/1275361680 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc580051880 con 0x7fc58c0c6b80 2023-10-30T12:52:24.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.558+0000 7fc591cf7640 1 -- 172.21.15.125:0/1275361680 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc58c06bb30 con 0x7fc58c0c6b80 2023-10-30T12:52:24.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.558+0000 7fc573fff640 1 -- 172.21.15.125:0/1275361680 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fc580054ea0 con 0x7fc58c0c6b80 2023-10-30T12:52:24.560 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:24.560 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6954,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:43.301579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":45,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:24.561 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 117 2023-10-30T12:52:24.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.558+0000 7fc591cf7640 1 -- 172.21.15.125:0/1275361680 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5780664a0 msgr2=0x7fc578068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:24.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.558+0000 7fc591cf7640 1 --2- 172.21.15.125:0/1275361680 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5780664a0 0x7fc578068960 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fc57c000f70 tx=0x7fc57c009040 comp rx=0 tx=0).stop 2023-10-30T12:52:24.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.558+0000 7fc591cf7640 1 -- 172.21.15.125:0/1275361680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c6b80 msgr2=0x7fc58c1f50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:24.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.558+0000 7fc591cf7640 1 --2- 172.21.15.125:0/1275361680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c6b80 0x7fc58c1f50c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fc58000d800 tx=0x7fc58000dcc0 comp rx=0 tx=0).stop 2023-10-30T12:52:24.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.558+0000 7fc591cf7640 1 -- 172.21.15.125:0/1275361680 shutdown_connections 2023-10-30T12:52:24.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.558+0000 7fc591cf7640 1 --2- 172.21.15.125:0/1275361680 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5780664a0 0x7fc578068960 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:24.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.558+0000 7fc591cf7640 1 --2- 172.21.15.125:0/1275361680 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc58c0c6f60 0x7fc58c163790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:24.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.558+0000 7fc591cf7640 1 --2- 172.21.15.125:0/1275361680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc58c0c6b80 0x7fc58c1f50c0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:24.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.558+0000 7fc591cf7640 1 --2- 172.21.15.125:0/1275361680 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc58c0c05d0 0x7fc58c06ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:24.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.558+0000 7fc591cf7640 1 -- 172.21.15.125:0/1275361680 >> 172.21.15.125:0/1275361680 conn(0x7fc58c0c7910 msgr2=0x7fc58c0c1320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:24.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.562+0000 7fc591cf7640 1 -- 172.21.15.125:0/1275361680 shutdown_connections 2023-10-30T12:52:24.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:24.562+0000 7fc591cf7640 1 -- 172.21.15.125:0/1275361680 wait complete. 2023-10-30T12:52:24.581 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:52:26.582 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:26.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb23165640 1 Processor -- start 2023-10-30T12:52:26.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb23165640 1 -- start start 2023-10-30T12:52:26.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb23165640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb1c159e60 0x7fbb1c06f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:26.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb23165640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb1c06a720 0x7fbb1c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:26.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb23165640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb1c06fa80 con 0x7fbb1c0bfec0 2023-10-30T12:52:26.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb23165640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb1c071290 con 0x7fbb1c159e60 2023-10-30T12:52:26.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb23165640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c071570 con 0x7fbb1c06a720 2023-10-30T12:52:26.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb20f01640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb1c159e60 0x7fbb1c06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:26.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb20f01640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb1c159e60 0x7fbb1c06f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52230/0 (socket says 172.21.15.125:52230) 2023-10-30T12:52:26.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb20f01640 1 -- 172.21.15.125:0/4048458865 learned_addr learned my addr 172.21.15.125:0/4048458865 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:26.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb21702640 1 --2- 172.21.15.125:0/4048458865 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb1c06a720 0x7fbb1c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:26.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb20f01640 1 -- 172.21.15.125:0/4048458865 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb1c06a720 msgr2=0x7fbb1c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:26.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb20f01640 1 --2- 172.21.15.125:0/4048458865 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb1c06a720 0x7fbb1c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:26.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb20f01640 1 -- 172.21.15.125:0/4048458865 >> v1:172.21.15.125:6789/0 conn(0x7fbb1c0bfec0 legacy=0x7fbb1c0c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:26.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.730+0000 7fbb20f01640 1 -- 172.21.15.125:0/4048458865 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1c071850 con 0x7fbb1c159e60 2023-10-30T12:52:26.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.734+0000 7fbb20f01640 1 --2- 172.21.15.125:0/4048458865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb1c159e60 0x7fbb1c06f540 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000aa10 tx=0x7fbb10021200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcdee84fe9538a2c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:26.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.734+0000 7fbb1b7fe640 1 -- 172.21.15.125:0/4048458865 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb100b1600 con 0x7fbb1c159e60 2023-10-30T12:52:26.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.734+0000 7fbb1b7fe640 1 -- 172.21.15.125:0/4048458865 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb100ad050 con 0x7fbb1c159e60 2023-10-30T12:52:26.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.734+0000 7fbb1b7fe640 1 -- 172.21.15.125:0/4048458865 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb100b1980 con 0x7fbb1c159e60 2023-10-30T12:52:26.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.734+0000 7fbb23165640 1 -- 172.21.15.125:0/4048458865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb1c159e60 msgr2=0x7fbb1c06f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:26.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.734+0000 7fbb23165640 1 --2- 172.21.15.125:0/4048458865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb1c159e60 0x7fbb1c06f540 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000aa10 tx=0x7fbb10021200 comp rx=0 tx=0).stop 2023-10-30T12:52:26.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.734+0000 7fbb23165640 1 -- 172.21.15.125:0/4048458865 shutdown_connections 2023-10-30T12:52:26.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.734+0000 7fbb23165640 1 --2- 172.21.15.125:0/4048458865 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb1c06a720 0x7fbb1c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:26.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.734+0000 7fbb23165640 1 --2- 172.21.15.125:0/4048458865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb1c159e60 0x7fbb1c06f540 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:26.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.734+0000 7fbb23165640 1 -- 172.21.15.125:0/4048458865 >> 172.21.15.125:0/4048458865 conn(0x7fbb1c0c7910 msgr2=0x7fbb1c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:26.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.734+0000 7fbb23165640 1 -- 172.21.15.125:0/4048458865 shutdown_connections 2023-10-30T12:52:26.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.734+0000 7fbb23165640 1 -- 172.21.15.125:0/4048458865 wait complete. 2023-10-30T12:52:26.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.734+0000 7fbb23165640 1 Processor -- start 2023-10-30T12:52:26.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb23165640 1 -- start start 2023-10-30T12:52:26.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb23165640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb1c06a720 0x7fbb1c1ef7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:26.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb23165640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb1c0bfec0 0x7fbb1c1f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:26.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb23165640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb1c159e60 0x7fbb1c1ecbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:26.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb23165640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c0b0560 con 0x7fbb1c06a720 2023-10-30T12:52:26.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb23165640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c1e1400 con 0x7fbb1c159e60 2023-10-30T12:52:26.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb23165640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c1e1930 con 0x7fbb1c0bfec0 2023-10-30T12:52:26.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb20f01640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb1c06a720 0x7fbb1c1ef7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:26.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb20f01640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb1c06a720 0x7fbb1c1ef7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52246/0 (socket says 172.21.15.125:52246) 2023-10-30T12:52:26.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb20f01640 1 -- 172.21.15.125:0/2130257339 learned_addr learned my addr 172.21.15.125:0/2130257339 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:26.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb1bfff640 1 --2- 172.21.15.125:0/2130257339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb1c0bfec0 0x7fbb1c1f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:26.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb21702640 1 --2- 172.21.15.125:0/2130257339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb1c159e60 0x7fbb1c1ecbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:26.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb20f01640 1 -- 172.21.15.125:0/2130257339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb1c0bfec0 msgr2=0x7fbb1c1f2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:26.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb20f01640 1 --2- 172.21.15.125:0/2130257339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb1c0bfec0 0x7fbb1c1f2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:26.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb20f01640 1 -- 172.21.15.125:0/2130257339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb1c159e60 msgr2=0x7fbb1c1ecbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:26.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb20f01640 1 --2- 172.21.15.125:0/2130257339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb1c159e60 0x7fbb1c1ecbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:26.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb20f01640 1 -- 172.21.15.125:0/2130257339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1000a670 con 0x7fbb1c06a720 2023-10-30T12:52:26.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb20f01640 1 --2- 172.21.15.125:0/2130257339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb1c06a720 0x7fbb1c1ef7e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000a1c0 tx=0x7fbb1001dfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:26.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb197fa640 1 -- 172.21.15.125:0/2130257339 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb10006ee0 con 0x7fbb1c06a720 2023-10-30T12:52:26.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb197fa640 1 -- 172.21.15.125:0/2130257339 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb100ba460 con 0x7fbb1c06a720 2023-10-30T12:52:26.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb23165640 1 -- 172.21.15.125:0/2130257339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1c1e1650 con 0x7fbb1c06a720 2023-10-30T12:52:26.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb197fa640 1 -- 172.21.15.125:0/2130257339 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb100ba830 con 0x7fbb1c06a720 2023-10-30T12:52:26.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb23165640 1 -- 172.21.15.125:0/2130257339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1c1eefe0 con 0x7fbb1c06a720 2023-10-30T12:52:26.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.738+0000 7fbb23165640 1 -- 172.21.15.125:0/2130257339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb1c071b10 con 0x7fbb1c06a720 2023-10-30T12:52:26.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.742+0000 7fbb197fa640 1 -- 172.21.15.125:0/2130257339 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb100b8070 con 0x7fbb1c06a720 2023-10-30T12:52:26.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.742+0000 7fbb197fa640 1 --2- 172.21.15.125:0/2130257339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb000664a0 0x7fbb00068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:26.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.742+0000 7fbb197fa640 1 -- 172.21.15.125:0/2130257339 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7fbb101220d0 con 0x7fbb1c06a720 2023-10-30T12:52:26.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.742+0000 7fbb1bfff640 1 --2- 172.21.15.125:0/2130257339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb000664a0 0x7fbb00068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:26.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.742+0000 7fbb1bfff640 1 --2- 172.21.15.125:0/2130257339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb000664a0 0x7fbb00068960 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c0cab50 tx=0x7fbb08009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:26.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.746+0000 7fbb197fa640 1 -- 172.21.15.125:0/2130257339 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbb1c071b10 con 0x7fbb1c06a720 2023-10-30T12:52:26.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.926+0000 7fbb23165640 1 -- 172.21.15.125:0/2130257339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb1c071b10 con 0x7fbb1c06a720 2023-10-30T12:52:26.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.926+0000 7fbb197fa640 1 -- 172.21.15.125:0/2130257339 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fbb1c071b10 con 0x7fbb1c06a720 2023-10-30T12:52:26.928 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:26.928 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6954,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:43.301579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":45,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:26.929 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 117 2023-10-30T12:52:26.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.930+0000 7fbb23165640 1 -- 172.21.15.125:0/2130257339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb000664a0 msgr2=0x7fbb00068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:26.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.930+0000 7fbb23165640 1 --2- 172.21.15.125:0/2130257339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb000664a0 0x7fbb00068960 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c0cab50 tx=0x7fbb08009040 comp rx=0 tx=0).stop 2023-10-30T12:52:26.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.930+0000 7fbb23165640 1 -- 172.21.15.125:0/2130257339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb1c06a720 msgr2=0x7fbb1c1ef7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:26.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.930+0000 7fbb23165640 1 --2- 172.21.15.125:0/2130257339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb1c06a720 0x7fbb1c1ef7e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fbb1000a1c0 tx=0x7fbb1001dfa0 comp rx=0 tx=0).stop 2023-10-30T12:52:26.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.930+0000 7fbb23165640 1 -- 172.21.15.125:0/2130257339 shutdown_connections 2023-10-30T12:52:26.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.930+0000 7fbb23165640 1 --2- 172.21.15.125:0/2130257339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb000664a0 0x7fbb00068960 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:26.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.930+0000 7fbb23165640 1 --2- 172.21.15.125:0/2130257339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb1c159e60 0x7fbb1c1ecbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:26.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.930+0000 7fbb23165640 1 --2- 172.21.15.125:0/2130257339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb1c0bfec0 0x7fbb1c1f2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:26.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.930+0000 7fbb23165640 1 --2- 172.21.15.125:0/2130257339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb1c06a720 0x7fbb1c1ef7e0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:26.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.930+0000 7fbb23165640 1 -- 172.21.15.125:0/2130257339 >> 172.21.15.125:0/2130257339 conn(0x7fbb1c0c7910 msgr2=0x7fbb1c157060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:26.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.930+0000 7fbb23165640 1 -- 172.21.15.125:0/2130257339 shutdown_connections 2023-10-30T12:52:26.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:26.930+0000 7fbb23165640 1 -- 172.21.15.125:0/2130257339 wait complete. 2023-10-30T12:52:26.949 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:52:28.950 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:29.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.098+0000 7f7f7a9c4640 1 Processor -- start 2023-10-30T12:52:29.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.098+0000 7f7f7a9c4640 1 -- start start 2023-10-30T12:52:29.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.098+0000 7f7f7a9c4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7f7406f7a0 0x7f7f7406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:29.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.098+0000 7f7f7a9c4640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f74066cf0 con 0x7f7f74071110 2023-10-30T12:52:29.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.098+0000 7f7f7a9c4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f740c70d0 con 0x7f7f740c6bd0 2023-10-30T12:52:29.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.098+0000 7f7f7a9c4640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f740700c0 con 0x7f7f7406f7a0 2023-10-30T12:52:29.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.102+0000 7f7f73fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7f7406f7a0 0x7f7f7406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:29.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.102+0000 7f7f73fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7f7406f7a0 0x7f7f7406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52256/0 (socket says 172.21.15.125:52256) 2023-10-30T12:52:29.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.102+0000 7f7f73fff640 1 -- 172.21.15.125:0/1913433298 learned_addr learned my addr 172.21.15.125:0/1913433298 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:29.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.102+0000 7f7f73fff640 1 -- 172.21.15.125:0/1913433298 >> v1:172.21.15.136:6789/0 conn(0x7f7f740c6bd0 legacy=0x7f7f74069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:29.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.102+0000 7f7f73fff640 1 -- 172.21.15.125:0/1913433298 >> v1:172.21.15.125:6789/0 conn(0x7f7f74071110 legacy=0x7f7f74158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:29.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.102+0000 7f7f73fff640 1 -- 172.21.15.125:0/1913433298 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f740c7390 con 0x7f7f7406f7a0 2023-10-30T12:52:29.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.102+0000 7f7f73fff640 1 --2- 172.21.15.125:0/1913433298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f7406f7a0 0x7f7f7406fb80 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f7f64009870 tx=0x7f7f6401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f8be26d534f36d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:29.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.102+0000 7f7f72ffd640 1 -- 172.21.15.125:0/1913433298 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f640a89f0 con 0x7f7f7406f7a0 2023-10-30T12:52:29.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.102+0000 7f7f72ffd640 1 -- 172.21.15.125:0/1913433298 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f640a8b50 con 0x7f7f7406f7a0 2023-10-30T12:52:29.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.102+0000 7f7f72ffd640 1 -- 172.21.15.125:0/1913433298 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f640b0440 con 0x7f7f7406f7a0 2023-10-30T12:52:29.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1913433298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f7406f7a0 msgr2=0x7f7f7406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:29.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 --2- 172.21.15.125:0/1913433298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f7406f7a0 0x7f7f7406fb80 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f7f64009870 tx=0x7f7f6401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:52:29.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1913433298 shutdown_connections 2023-10-30T12:52:29.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 --2- 172.21.15.125:0/1913433298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f7406f7a0 0x7f7f7406fb80 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:29.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1913433298 >> 172.21.15.125:0/1913433298 conn(0x7f7f740c7910 msgr2=0x7f7f740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:29.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1913433298 shutdown_connections 2023-10-30T12:52:29.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1913433298 wait complete. 2023-10-30T12:52:29.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 Processor -- start 2023-10-30T12:52:29.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 -- start start 2023-10-30T12:52:29.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f7406f7a0 0x7f7f7414f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:29.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f74071110 0x7f7f7414feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:29.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f740c6bd0 0x7f7f741e8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:29.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7414ec40 con 0x7f7f740c6bd0 2023-10-30T12:52:29.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f741557c0 con 0x7f7f74071110 2023-10-30T12:52:29.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f7a9c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f74154a70 con 0x7f7f7406f7a0 2023-10-30T12:52:29.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f78f61640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f740c6bd0 0x7f7f741e8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:29.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f78f61640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f740c6bd0 0x7f7f741e8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52260/0 (socket says 172.21.15.125:52260) 2023-10-30T12:52:29.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f78f61640 1 -- 172.21.15.125:0/1479991214 learned_addr learned my addr 172.21.15.125:0/1479991214 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:29.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f73fff640 1 --2- 172.21.15.125:0/1479991214 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f7406f7a0 0x7f7f7414f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:29.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f737fe640 1 --2- 172.21.15.125:0/1479991214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f74071110 0x7f7f7414feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:29.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f78f61640 1 -- 172.21.15.125:0/1479991214 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f7406f7a0 msgr2=0x7f7f7414f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:29.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f78f61640 1 --2- 172.21.15.125:0/1479991214 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f7406f7a0 0x7f7f7414f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:29.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f78f61640 1 -- 172.21.15.125:0/1479991214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f74071110 msgr2=0x7f7f7414feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:29.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f78f61640 1 --2- 172.21.15.125:0/1479991214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f74071110 0x7f7f7414feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:29.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.106+0000 7f7f78f61640 1 -- 172.21.15.125:0/1479991214 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f640094d0 con 0x7f7f740c6bd0 2023-10-30T12:52:29.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.110+0000 7f7f78f61640 1 --2- 172.21.15.125:0/1479991214 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f740c6bd0 0x7f7f741e8cd0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c00c730 tx=0x7f7f6c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:29.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.110+0000 7f7f70ff9640 1 -- 172.21.15.125:0/1479991214 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c003b80 con 0x7f7f740c6bd0 2023-10-30T12:52:29.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.110+0000 7f7f70ff9640 1 -- 172.21.15.125:0/1479991214 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f6c003ce0 con 0x7f7f740c6bd0 2023-10-30T12:52:29.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.110+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1479991214 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f74154d50 con 0x7f7f740c6bd0 2023-10-30T12:52:29.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.110+0000 7f7f70ff9640 1 -- 172.21.15.125:0/1479991214 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f6c00aa30 con 0x7f7f740c6bd0 2023-10-30T12:52:29.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.110+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1479991214 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f7414f170 con 0x7f7f740c6bd0 2023-10-30T12:52:29.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.110+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1479991214 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f7404e170 con 0x7f7f740c6bd0 2023-10-30T12:52:29.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.110+0000 7f7f70ff9640 1 -- 172.21.15.125:0/1479991214 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7f6c00ab90 con 0x7f7f740c6bd0 2023-10-30T12:52:29.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.110+0000 7f7f70ff9640 1 --2- 172.21.15.125:0/1479991214 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f600664a0 0x7f7f60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:29.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.110+0000 7f7f70ff9640 1 -- 172.21.15.125:0/1479991214 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6684+0+0 (secure 0 0 0) 0x7f7f6c083fe0 con 0x7f7f740c6bd0 2023-10-30T12:52:29.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.110+0000 7f7f73fff640 1 --2- 172.21.15.125:0/1479991214 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f600664a0 0x7f7f60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:29.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.114+0000 7f7f73fff640 1 --2- 172.21.15.125:0/1479991214 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f600664a0 0x7f7f60068960 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7f640b2000 tx=0x7f7f64008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:29.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.114+0000 7f7f70ff9640 1 -- 172.21.15.125:0/1479991214 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7f6c051030 con 0x7f7f740c6bd0 2023-10-30T12:52:29.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.298+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1479991214 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f740c47f0 con 0x7f7f740c6bd0 2023-10-30T12:52:29.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.298+0000 7f7f70ff9640 1 -- 172.21.15.125:0/1479991214 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f7f6c054650 con 0x7f7f740c6bd0 2023-10-30T12:52:29.300 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:29.300 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":6954,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:51:43.301579+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6627},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6627":{"gid":6627,"name":"c","rank":0,"incarnation":112,"state":"up:active","state_seq":45,"addr":"172.21.15.125:6837/2408701650","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2408701650},{"type":"v1","addr":"172.21.15.125:6837","nonce":2408701650}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:29.302 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 117 2023-10-30T12:52:29.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.298+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1479991214 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f600664a0 msgr2=0x7f7f60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:29.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.298+0000 7f7f7a9c4640 1 --2- 172.21.15.125:0/1479991214 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f600664a0 0x7f7f60068960 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f7f640b2000 tx=0x7f7f64008040 comp rx=0 tx=0).stop 2023-10-30T12:52:29.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.298+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1479991214 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f740c6bd0 msgr2=0x7f7f741e8cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:29.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.298+0000 7f7f7a9c4640 1 --2- 172.21.15.125:0/1479991214 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f740c6bd0 0x7f7f741e8cd0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f7f6c00c730 tx=0x7f7f6c00cbf0 comp rx=0 tx=0).stop 2023-10-30T12:52:29.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.298+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1479991214 shutdown_connections 2023-10-30T12:52:29.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.298+0000 7f7f7a9c4640 1 --2- 172.21.15.125:0/1479991214 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f600664a0 0x7f7f60068960 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:29.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.298+0000 7f7f7a9c4640 1 --2- 172.21.15.125:0/1479991214 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f740c6bd0 0x7f7f741e8cd0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:29.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.298+0000 7f7f7a9c4640 1 --2- 172.21.15.125:0/1479991214 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f74071110 0x7f7f7414feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:29.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.298+0000 7f7f7a9c4640 1 --2- 172.21.15.125:0/1479991214 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f7406f7a0 0x7f7f7414f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:29.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.298+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1479991214 >> 172.21.15.125:0/1479991214 conn(0x7f7f740c7910 msgr2=0x7f7f740c94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:29.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.298+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1479991214 shutdown_connections 2023-10-30T12:52:29.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:29.298+0000 7f7f7a9c4640 1 -- 172.21.15.125:0/1479991214 wait complete. 2023-10-30T12:52:29.320 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:52:31.322 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:31.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.470+0000 7f75fae54640 1 Processor -- start 2023-10-30T12:52:31.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.470+0000 7f75fae54640 1 -- start start 2023-10-30T12:52:31.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.474+0000 7f75fae54640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f75f41538a0 0x7f75f415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:31.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.474+0000 7f75fae54640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75f40ca0e0 0x7f75f415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:31.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.474+0000 7f75fae54640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f40cb800 con 0x7f75f40cf160 2023-10-30T12:52:31.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.474+0000 7f75fae54640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f415dc30 con 0x7f75f40ca0e0 2023-10-30T12:52:31.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.474+0000 7f75fae54640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f40cb330 con 0x7f75f41538a0 2023-10-30T12:52:31.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.474+0000 7f75ebfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75f40ca0e0 0x7f75f415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:31.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.474+0000 7f75ebfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75f40ca0e0 0x7f75f415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40120/0 (socket says 172.21.15.125:40120) 2023-10-30T12:52:31.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.474+0000 7f75ebfff640 1 -- 172.21.15.125:0/426262542 learned_addr learned my addr 172.21.15.125:0/426262542 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:31.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.474+0000 7f75f8bf0640 1 --2- 172.21.15.125:0/426262542 >> v2:172.21.15.136:3300/0 conn(0x7f75f41538a0 0x7f75f415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:31.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.474+0000 7f75ebfff640 1 -- 172.21.15.125:0/426262542 >> v2:172.21.15.136:3300/0 conn(0x7f75f41538a0 msgr2=0x7f75f415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:31.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.474+0000 7f75ebfff640 1 --2- 172.21.15.125:0/426262542 >> v2:172.21.15.136:3300/0 conn(0x7f75f41538a0 0x7f75f415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.474+0000 7f75ebfff640 1 -- 172.21.15.125:0/426262542 >> v1:172.21.15.136:6789/0 conn(0x7f75f40cf160 legacy=0x7f75f40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:31.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.474+0000 7f75ebfff640 1 -- 172.21.15.125:0/426262542 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f40cad70 con 0x7f75f40ca0e0 2023-10-30T12:52:31.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.474+0000 7f75ebfff640 1 --2- 172.21.15.125:0/426262542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f40ca0e0 0x7f75f415a090 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f75f40cfd10 tx=0x7f75e401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249541177374801e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:31.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75eb7fe640 1 -- 172.21.15.125:0/426262542 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e40a89f0 con 0x7f75f40ca0e0 2023-10-30T12:52:31.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75eb7fe640 1 -- 172.21.15.125:0/426262542 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e40a8b50 con 0x7f75f40ca0e0 2023-10-30T12:52:31.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75eb7fe640 1 -- 172.21.15.125:0/426262542 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e40b0440 con 0x7f75f40ca0e0 2023-10-30T12:52:31.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 -- 172.21.15.125:0/426262542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f40ca0e0 msgr2=0x7f75f415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:31.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 --2- 172.21.15.125:0/426262542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f40ca0e0 0x7f75f415a090 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f75f40cfd10 tx=0x7f75e401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 -- 172.21.15.125:0/426262542 shutdown_connections 2023-10-30T12:52:31.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 --2- 172.21.15.125:0/426262542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f40ca0e0 0x7f75f415a090 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 --2- 172.21.15.125:0/426262542 >> v2:172.21.15.136:3300/0 conn(0x7f75f41538a0 0x7f75f415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 -- 172.21.15.125:0/426262542 >> 172.21.15.125:0/426262542 conn(0x7f75f406bfe0 msgr2=0x7f75f406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:31.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 -- 172.21.15.125:0/426262542 shutdown_connections 2023-10-30T12:52:31.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 -- 172.21.15.125:0/426262542 wait complete. 2023-10-30T12:52:31.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 Processor -- start 2023-10-30T12:52:31.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 -- start start 2023-10-30T12:52:31.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f40ca0e0 0x7f75f40ce630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:31.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75f40cf160 0x7f75f4142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:31.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f41538a0 0x7f75f41ef7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:31.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f41584d0 con 0x7f75f40ca0e0 2023-10-30T12:52:31.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f41ef390 con 0x7f75f41538a0 2023-10-30T12:52:31.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.478+0000 7f75fae54640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f40cec20 con 0x7f75f40cf160 2023-10-30T12:52:31.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75f8bf0640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f40ca0e0 0x7f75f40ce630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:31.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75f8bf0640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f40ca0e0 0x7f75f40ce630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40132/0 (socket says 172.21.15.125:40132) 2023-10-30T12:52:31.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75f8bf0640 1 -- 172.21.15.125:0/1260864936 learned_addr learned my addr 172.21.15.125:0/1260864936 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:31.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75ebfff640 1 --2- 172.21.15.125:0/1260864936 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75f40cf160 0x7f75f4142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:31.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75f93f1640 1 --2- 172.21.15.125:0/1260864936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f41538a0 0x7f75f41ef7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:31.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75f8bf0640 1 -- 172.21.15.125:0/1260864936 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75f40cf160 msgr2=0x7f75f4142550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:31.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75f8bf0640 1 --2- 172.21.15.125:0/1260864936 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75f40cf160 0x7f75f4142550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75f8bf0640 1 -- 172.21.15.125:0/1260864936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f41538a0 msgr2=0x7f75f41ef7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:31.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75f8bf0640 1 --2- 172.21.15.125:0/1260864936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f41538a0 0x7f75f41ef7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75f8bf0640 1 -- 172.21.15.125:0/1260864936 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e40094d0 con 0x7f75f40ca0e0 2023-10-30T12:52:31.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75f8bf0640 1 --2- 172.21.15.125:0/1260864936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f40ca0e0 0x7f75f40ce630 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f75f000e870 tx=0x7f75f000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:31.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75e97fa640 1 -- 172.21.15.125:0/1260864936 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f0009530 con 0x7f75f40ca0e0 2023-10-30T12:52:31.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75e97fa640 1 -- 172.21.15.125:0/1260864936 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f000c940 con 0x7f75f40ca0e0 2023-10-30T12:52:31.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75e97fa640 1 -- 172.21.15.125:0/1260864936 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f000ccd0 con 0x7f75f40ca0e0 2023-10-30T12:52:31.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75fae54640 1 -- 172.21.15.125:0/1260864936 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f40ceec0 con 0x7f75f40ca0e0 2023-10-30T12:52:31.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75fae54640 1 -- 172.21.15.125:0/1260864936 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f4149080 con 0x7f75f40ca0e0 2023-10-30T12:52:31.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75fae54640 1 -- 172.21.15.125:0/1260864936 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f40b2a50 con 0x7f75f40ca0e0 2023-10-30T12:52:31.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75e97fa640 1 -- 172.21.15.125:0/1260864936 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f75f0013470 con 0x7f75f40ca0e0 2023-10-30T12:52:31.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.482+0000 7f75e97fa640 1 --2- 172.21.15.125:0/1260864936 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75e00664a0 0x7f75e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:31.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.486+0000 7f75e97fa640 1 -- 172.21.15.125:0/1260864936 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7f75f0084320 con 0x7f75f40ca0e0 2023-10-30T12:52:31.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.486+0000 7f75ebfff640 1 --2- 172.21.15.125:0/1260864936 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75e00664a0 0x7f75e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:31.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.486+0000 7f75ebfff640 1 --2- 172.21.15.125:0/1260864936 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75e00664a0 0x7f75e0068960 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f75e40b2000 tx=0x7f75e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:31.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.486+0000 7f75e97fa640 1 -- 172.21.15.125:0/1260864936 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f75f0051400 con 0x7f75f40ca0e0 2023-10-30T12:52:31.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.670+0000 7f75fae54640 1 -- 172.21.15.125:0/1260864936 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75f404e170 con 0x7f75f40ca0e0 2023-10-30T12:52:31.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.670+0000 7f75e97fa640 1 -- 172.21.15.125:0/1260864936 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f75f0054a20 con 0x7f75f40ca0e0 2023-10-30T12:52:31.671 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:31.672 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:30.335199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:31.672 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 118 2023-10-30T12:52:31.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.670+0000 7f75fae54640 1 -- 172.21.15.125:0/1260864936 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75e00664a0 msgr2=0x7f75e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:31.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.670+0000 7f75fae54640 1 --2- 172.21.15.125:0/1260864936 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75e00664a0 0x7f75e0068960 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f75e40b2000 tx=0x7f75e4009040 comp rx=0 tx=0).stop 2023-10-30T12:52:31.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.670+0000 7f75fae54640 1 -- 172.21.15.125:0/1260864936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f40ca0e0 msgr2=0x7f75f40ce630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:31.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.670+0000 7f75fae54640 1 --2- 172.21.15.125:0/1260864936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f40ca0e0 0x7f75f40ce630 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f75f000e870 tx=0x7f75f000ed30 comp rx=0 tx=0).stop 2023-10-30T12:52:31.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.670+0000 7f75fae54640 1 -- 172.21.15.125:0/1260864936 shutdown_connections 2023-10-30T12:52:31.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.670+0000 7f75fae54640 1 --2- 172.21.15.125:0/1260864936 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75e00664a0 0x7f75e0068960 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.670+0000 7f75fae54640 1 --2- 172.21.15.125:0/1260864936 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f41538a0 0x7f75f41ef7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.670+0000 7f75fae54640 1 --2- 172.21.15.125:0/1260864936 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75f40cf160 0x7f75f4142550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.670+0000 7f75fae54640 1 --2- 172.21.15.125:0/1260864936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f40ca0e0 0x7f75f40ce630 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.670+0000 7f75fae54640 1 -- 172.21.15.125:0/1260864936 >> 172.21.15.125:0/1260864936 conn(0x7f75f406bfe0 msgr2=0x7f75f4152310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:31.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.674+0000 7f75fae54640 1 -- 172.21.15.125:0/1260864936 shutdown_connections 2023-10-30T12:52:31.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.674+0000 7f75fae54640 1 -- 172.21.15.125:0/1260864936 wait complete. 2023-10-30T12:52:31.693 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T12:52:31.693 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:52:31.693 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:31.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c2409e640 1 Processor -- start 2023-10-30T12:52:31.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c2409e640 1 -- start start 2023-10-30T12:52:31.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c2409e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c1c1538a0 0x7f2c1c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:31.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c2409e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1c0cb800 con 0x7f2c1c0cf160 2023-10-30T12:52:31.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c2409e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c1c15dc30 con 0x7f2c1c0ca0e0 2023-10-30T12:52:31.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c2409e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0cb330 con 0x7f2c1c1538a0 2023-10-30T12:52:31.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c2263b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2c1c0cf160 0x7f2c1c164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37660/0 (socket says 172.21.15.125:37660) 2023-10-30T12:52:31.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c2263b640 1 -- 172.21.15.125:0/3451094277 learned_addr learned my addr 172.21.15.125:0/3451094277 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:52:31.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c21e3a640 1 --2- 172.21.15.125:0/3451094277 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c1c1538a0 0x7f2c1c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:31.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c20e38640 1 -- 172.21.15.125:0/3451094277 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195759374 0 0) 0x7f2c1c0cb800 con 0x7f2c1c0cf160 2023-10-30T12:52:31.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c20e38640 1 -- 172.21.15.125:0/3451094277 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c08003530 con 0x7f2c1c0cf160 2023-10-30T12:52:31.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c20e38640 1 -- 172.21.15.125:0/3451094277 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2937118452 0 0) 0x7f2c08003530 con 0x7f2c1c0cf160 2023-10-30T12:52:31.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c20e38640 1 -- 172.21.15.125:0/3451094277 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c1c1538a0 msgr2=0x7f2c1c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:31.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c20e38640 1 --2- 172.21.15.125:0/3451094277 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c1c1538a0 0x7f2c1c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c20e38640 1 -- 172.21.15.125:0/3451094277 >> v1:172.21.15.136:6789/0 conn(0x7f2c1c0ca0e0 legacy=0x7f2c1c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:31.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c20e38640 1 -- 172.21.15.125:0/3451094277 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c0cad70 con 0x7f2c1c0cf160 2023-10-30T12:52:31.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c20e38640 1 -- 172.21.15.125:0/3451094277 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f2c14001a00 con 0x7f2c1c0cf160 2023-10-30T12:52:31.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c20e38640 1 -- 172.21.15.125:0/3451094277 >> v1:172.21.15.125:6789/0 conn(0x7f2c1c0cf160 legacy=0x7f2c1c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:31.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.830+0000 7f2c20e38640 1 --2- 172.21.15.125:0/3451094277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c080039b0 0x7f2c08003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:31.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c20e38640 1 --2- 172.21.15.125:0/3451094277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c080047d0 0x7f2c08004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:31.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c20e38640 1 --2- 172.21.15.125:0/3451094277 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c1c1538a0 0x7f2c08005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:31.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c20e38640 1 -- 172.21.15.125:0/3451094277 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c0cb330 con 0x7f2c080047d0 2023-10-30T12:52:31.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c20e38640 1 -- 172.21.15.125:0/3451094277 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0800bad0 con 0x7f2c080039b0 2023-10-30T12:52:31.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c20e38640 1 -- 172.21.15.125:0/3451094277 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c0800bc90 con 0x7f2c1c1538a0 2023-10-30T12:52:31.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c21639640 1 --2- 172.21.15.125:0/3451094277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c080047d0 0x7f2c08004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:31.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c21e3a640 1 --2- 172.21.15.125:0/3451094277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c080039b0 0x7f2c08003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:31.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c2263b640 1 --2- 172.21.15.125:0/3451094277 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c1c1538a0 0x7f2c08005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:31.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c21639640 1 -- 172.21.15.125:0/3451094277 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c1c1538a0 msgr2=0x7f2c08005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:31.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c21639640 1 --2- 172.21.15.125:0/3451094277 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c1c1538a0 0x7f2c08005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c21639640 1 -- 172.21.15.125:0/3451094277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c080039b0 msgr2=0x7f2c08003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:31.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c21639640 1 --2- 172.21.15.125:0/3451094277 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c080039b0 0x7f2c08003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c21639640 1 -- 172.21.15.125:0/3451094277 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c08003530 con 0x7f2c080047d0 2023-10-30T12:52:31.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c21639640 1 --2- 172.21.15.125:0/3451094277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c080047d0 0x7f2c08004c50 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0015b0 tx=0x7f2c0c01c200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33c2045b3e5298f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:31.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c20e38640 1 -- 172.21.15.125:0/3451094277 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c0052d0 con 0x7f2c080047d0 2023-10-30T12:52:31.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c20e38640 1 -- 172.21.15.125:0/3451094277 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c0c0a7050 con 0x7f2c080047d0 2023-10-30T12:52:31.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.834+0000 7f2c20e38640 1 -- 172.21.15.125:0/3451094277 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c0c003600 con 0x7f2c080047d0 2023-10-30T12:52:31.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2409e640 1 -- 172.21.15.125:0/3451094277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c080047d0 msgr2=0x7f2c08004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:31.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2409e640 1 --2- 172.21.15.125:0/3451094277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c080047d0 0x7f2c08004c50 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c0015b0 tx=0x7f2c0c01c200 comp rx=0 tx=0).stop 2023-10-30T12:52:31.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2263b640 1 -- 172.21.15.125:0/3451094277 reap_dead start 2023-10-30T12:52:31.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2409e640 1 -- 172.21.15.125:0/3451094277 shutdown_connections 2023-10-30T12:52:31.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2409e640 1 -- 172.21.15.125:0/3451094277 >> 172.21.15.125:0/3451094277 conn(0x7f2c1c06bfe0 msgr2=0x7f2c1c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:31.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2409e640 1 -- 172.21.15.125:0/3451094277 shutdown_connections 2023-10-30T12:52:31.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2409e640 1 -- 172.21.15.125:0/3451094277 wait complete. 2023-10-30T12:52:31.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2409e640 1 Processor -- start 2023-10-30T12:52:31.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2409e640 1 -- start start 2023-10-30T12:52:31.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2409e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c080047d0 0x7f2c1c1efdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:31.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2409e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c1c1e6770 0x7f2c1c0b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:31.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2409e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c1c1e31b0 0x7f2c1c145b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:31.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2409e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c1ef350 con 0x7f2c1c1e31b0 2023-10-30T12:52:31.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2409e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c1f3360 con 0x7f2c080047d0 2023-10-30T12:52:31.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2409e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1c1ef5f0 con 0x7f2c1c1e6770 2023-10-30T12:52:31.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2263b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c1c1e31b0 0x7f2c1c145b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:31.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2263b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c1c1e31b0 0x7f2c1c145b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40146/0 (socket says 172.21.15.125:40146) 2023-10-30T12:52:31.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c2263b640 1 -- 172.21.15.125:0/3323222712 learned_addr learned my addr 172.21.15.125:0/3323222712 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:31.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c21639640 1 --2- 172.21.15.125:0/3323222712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c1c1e6770 0x7f2c1c0b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:31.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.838+0000 7f2c21e3a640 1 --2- 172.21.15.125:0/3323222712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c080047d0 0x7f2c1c1efdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:31.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c2263b640 1 -- 172.21.15.125:0/3323222712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c1c1e6770 msgr2=0x7f2c1c0b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:31.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c2263b640 1 --2- 172.21.15.125:0/3323222712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c1c1e6770 0x7f2c1c0b4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c2263b640 1 -- 172.21.15.125:0/3323222712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c080047d0 msgr2=0x7f2c1c1efdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:31.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c2263b640 1 --2- 172.21.15.125:0/3323222712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c080047d0 0x7f2c1c1efdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:31.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c2263b640 1 -- 172.21.15.125:0/3323222712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c1c1ef8d0 con 0x7f2c1c1e31b0 2023-10-30T12:52:31.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c2263b640 1 --2- 172.21.15.125:0/3323222712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c1c1e31b0 0x7f2c1c145b60 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c06b0e0 tx=0x7f2c1400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c127fc640 1 -- 172.21.15.125:0/3323222712 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1400eb90 con 0x7f2c1c1e31b0 2023-10-30T12:52:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c127fc640 1 -- 172.21.15.125:0/3323222712 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1400ecf0 con 0x7f2c1c1e31b0 2023-10-30T12:52:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c2409e640 1 -- 172.21.15.125:0/3323222712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c1c1f3640 con 0x7f2c1c1e31b0 2023-10-30T12:52:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c127fc640 1 -- 172.21.15.125:0/3323222712 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c14013660 con 0x7f2c1c1e31b0 2023-10-30T12:52:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c2409e640 1 -- 172.21.15.125:0/3323222712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c1c0b3910 con 0x7f2c1c1e31b0 2023-10-30T12:52:31.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c2409e640 1 -- 172.21.15.125:0/3323222712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c1c04e170 con 0x7f2c1c1e31b0 2023-10-30T12:52:31.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c127fc640 1 -- 172.21.15.125:0/3323222712 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c140137c0 con 0x7f2c1c1e31b0 2023-10-30T12:52:31.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c127fc640 1 --2- 172.21.15.125:0/3323222712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c000664d0 0x7f2c00068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:31.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c127fc640 1 -- 172.21.15.125:0/3323222712 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7f2c140845c0 con 0x7f2c1c1e31b0 2023-10-30T12:52:31.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.842+0000 7f2c21e3a640 1 --2- 172.21.15.125:0/3323222712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c000664d0 0x7f2c00068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:31.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.846+0000 7f2c21e3a640 1 --2- 172.21.15.125:0/3323222712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c000664d0 0x7f2c00068990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2c18009be0 tx=0x7f2c18009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:31.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:31.846+0000 7f2c127fc640 1 -- 172.21.15.125:0/3323222712 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c140514e0 con 0x7f2c1c1e31b0 2023-10-30T12:52:32.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:32.026+0000 7f2c2409e640 1 -- 172.21.15.125:0/3323222712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c1c0ca9b0 con 0x7f2c1c1e31b0 2023-10-30T12:52:32.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:32.026+0000 7f2c127fc640 1 -- 172.21.15.125:0/3323222712 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f2c14054b00 con 0x7f2c1c1e31b0 2023-10-30T12:52:32.028 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:32.029 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:30.335199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:32.030 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 118 2023-10-30T12:52:32.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:32.030+0000 7f2c2409e640 1 -- 172.21.15.125:0/3323222712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c000664d0 msgr2=0x7f2c00068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:32.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:32.030+0000 7f2c2409e640 1 --2- 172.21.15.125:0/3323222712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c000664d0 0x7f2c00068990 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2c18009be0 tx=0x7f2c18009190 comp rx=0 tx=0).stop 2023-10-30T12:52:32.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:32.030+0000 7f2c2409e640 1 -- 172.21.15.125:0/3323222712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c1c1e31b0 msgr2=0x7f2c1c145b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:32.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:32.030+0000 7f2c2409e640 1 --2- 172.21.15.125:0/3323222712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c1c1e31b0 0x7f2c1c145b60 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c06b0e0 tx=0x7f2c1400e530 comp rx=0 tx=0).stop 2023-10-30T12:52:32.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:32.030+0000 7f2c2409e640 1 -- 172.21.15.125:0/3323222712 shutdown_connections 2023-10-30T12:52:32.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:32.030+0000 7f2c2409e640 1 --2- 172.21.15.125:0/3323222712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c000664d0 0x7f2c00068990 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:32.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:32.030+0000 7f2c2409e640 1 --2- 172.21.15.125:0/3323222712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c1c1e31b0 0x7f2c1c145b60 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:32.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:32.030+0000 7f2c2409e640 1 --2- 172.21.15.125:0/3323222712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c1c1e6770 0x7f2c1c0b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:32.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:32.030+0000 7f2c2409e640 1 --2- 172.21.15.125:0/3323222712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c080047d0 0x7f2c1c1efdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:32.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:32.030+0000 7f2c2409e640 1 -- 172.21.15.125:0/3323222712 >> 172.21.15.125:0/3323222712 conn(0x7f2c1c06bfe0 msgr2=0x7f2c1c1526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:32.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:32.030+0000 7f2c2409e640 1 -- 172.21.15.125:0/3323222712 shutdown_connections 2023-10-30T12:52:32.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:32.030+0000 7f2c2409e640 1 -- 172.21.15.125:0/3323222712 wait complete. 2023-10-30T12:52:32.049 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 118, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 118, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6954": { "addr": "172.21.15.125:6836/1163622496", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1163622496, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1163622496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6954, "incarnation": 118, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:52:30.335199+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6954 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:52:34.050 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:34.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.194+0000 7fca02ed6640 1 Processor -- start 2023-10-30T12:52:34.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.194+0000 7fca02ed6640 1 -- start start 2023-10-30T12:52:34.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca02ed6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc9fc0c6b80 0x7fc9fc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:34.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca02ed6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9fc0c6f60 0x7fc9fc0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:34.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca02ed6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc9fc0c8430 0x7fc9fc13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:34.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca02ed6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9fc0c0a50 con 0x7fc9fc0c6b80 2023-10-30T12:52:34.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca02ed6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9fc06f6e0 con 0x7fc9fc0c8430 2023-10-30T12:52:34.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca02ed6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc06f9c0 con 0x7fc9fc0c6f60 2023-10-30T12:52:34.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca00c72640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc9fc0c6b80 0x7fc9fc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:34.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca00c72640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc9fc0c6b80 0x7fc9fc0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40154/0 (socket says 172.21.15.125:40154) 2023-10-30T12:52:34.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca00c72640 1 -- 172.21.15.125:0/1934467503 learned_addr learned my addr 172.21.15.125:0/1934467503 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:34.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fc9fbfff640 1 --2- 172.21.15.125:0/1934467503 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9fc0c6f60 0x7fc9fc0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:34.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca01473640 1 --2- 172.21.15.125:0/1934467503 >> v2:172.21.15.136:3300/0 conn(0x7fc9fc0c8430 0x7fc9fc13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:34.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca00c72640 1 -- 172.21.15.125:0/1934467503 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9fc0c6f60 msgr2=0x7fc9fc0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:34.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca00c72640 1 --2- 172.21.15.125:0/1934467503 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9fc0c6f60 0x7fc9fc0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:34.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca00c72640 1 -- 172.21.15.125:0/1934467503 >> v2:172.21.15.136:3300/0 conn(0x7fc9fc0c8430 msgr2=0x7fc9fc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:34.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca00c72640 1 --2- 172.21.15.125:0/1934467503 >> v2:172.21.15.136:3300/0 conn(0x7fc9fc0c8430 0x7fc9fc13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:34.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.198+0000 7fca00c72640 1 -- 172.21.15.125:0/1934467503 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9fc06fca0 con 0x7fc9fc0c6b80 2023-10-30T12:52:34.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fca00c72640 1 --2- 172.21.15.125:0/1934467503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9fc0c6b80 0x7fc9fc0bfec0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc0b0860 tx=0x7fc9f001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c782f537af629f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:34.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fc9fb7fe640 1 -- 172.21.15.125:0/1934467503 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f00ab600 con 0x7fc9fc0c6b80 2023-10-30T12:52:34.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fc9fb7fe640 1 -- 172.21.15.125:0/1934467503 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f00a7050 con 0x7fc9fc0c6b80 2023-10-30T12:52:34.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fc9fb7fe640 1 -- 172.21.15.125:0/1934467503 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f00ab980 con 0x7fc9fc0c6b80 2023-10-30T12:52:34.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fca02ed6640 1 -- 172.21.15.125:0/1934467503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9fc0c6b80 msgr2=0x7fc9fc0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:34.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fca02ed6640 1 --2- 172.21.15.125:0/1934467503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9fc0c6b80 0x7fc9fc0bfec0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc0b0860 tx=0x7fc9f001c470 comp rx=0 tx=0).stop 2023-10-30T12:52:34.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fca02ed6640 1 -- 172.21.15.125:0/1934467503 shutdown_connections 2023-10-30T12:52:34.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fca02ed6640 1 --2- 172.21.15.125:0/1934467503 >> v2:172.21.15.136:3300/0 conn(0x7fc9fc0c8430 0x7fc9fc13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:34.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fca02ed6640 1 --2- 172.21.15.125:0/1934467503 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9fc0c6f60 0x7fc9fc0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:34.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fca02ed6640 1 --2- 172.21.15.125:0/1934467503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9fc0c6b80 0x7fc9fc0bfec0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:34.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fca02ed6640 1 -- 172.21.15.125:0/1934467503 >> 172.21.15.125:0/1934467503 conn(0x7fc9fc0c7910 msgr2=0x7fc9fc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:34.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fca02ed6640 1 -- 172.21.15.125:0/1934467503 shutdown_connections 2023-10-30T12:52:34.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fca02ed6640 1 -- 172.21.15.125:0/1934467503 wait complete. 2023-10-30T12:52:34.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fca02ed6640 1 Processor -- start 2023-10-30T12:52:34.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.202+0000 7fca02ed6640 1 -- start start 2023-10-30T12:52:34.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca02ed6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9fc0c6b80 0x7fc9fc13d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:34.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca02ed6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9fc0c6f60 0x7fc9fc158310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:34.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca02ed6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9fc0c8430 0x7fc9fc154d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:34.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca02ed6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc157ef0 con 0x7fc9fc0c8430 2023-10-30T12:52:34.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca02ed6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc138520 con 0x7fc9fc0c6f60 2023-10-30T12:52:34.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca02ed6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9fc0c8d20 con 0x7fc9fc0c6b80 2023-10-30T12:52:34.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca01473640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9fc0c8430 0x7fc9fc154d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:34.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca00c72640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9fc0c6b80 0x7fc9fc13d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:34.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca01473640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9fc0c8430 0x7fc9fc154d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40166/0 (socket says 172.21.15.125:40166) 2023-10-30T12:52:34.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca01473640 1 -- 172.21.15.125:0/695963347 learned_addr learned my addr 172.21.15.125:0/695963347 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:34.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fc9fbfff640 1 --2- 172.21.15.125:0/695963347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9fc0c6f60 0x7fc9fc158310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:34.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca01473640 1 -- 172.21.15.125:0/695963347 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9fc0c6b80 msgr2=0x7fc9fc13d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:34.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca01473640 1 --2- 172.21.15.125:0/695963347 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9fc0c6b80 0x7fc9fc13d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:34.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca01473640 1 -- 172.21.15.125:0/695963347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9fc0c6f60 msgr2=0x7fc9fc158310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:34.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca01473640 1 --2- 172.21.15.125:0/695963347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9fc0c6f60 0x7fc9fc158310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:34.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca01473640 1 -- 172.21.15.125:0/695963347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f00094d0 con 0x7fc9fc0c8430 2023-10-30T12:52:34.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca01473640 1 --2- 172.21.15.125:0/695963347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9fc0c8430 0x7fc9fc154d60 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4007e60 tx=0x7fc9f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:34.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fc9f97fa640 1 -- 172.21.15.125:0/695963347 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4014420 con 0x7fc9fc0c8430 2023-10-30T12:52:34.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fc9f97fa640 1 -- 172.21.15.125:0/695963347 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9f40145a0 con 0x7fc9fc0c8430 2023-10-30T12:52:34.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca02ed6640 1 -- 172.21.15.125:0/695963347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9fc0c8fe0 con 0x7fc9fc0c8430 2023-10-30T12:52:34.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fc9f97fa640 1 -- 172.21.15.125:0/695963347 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9f4014950 con 0x7fc9fc0c8430 2023-10-30T12:52:34.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca02ed6640 1 -- 172.21.15.125:0/695963347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9fc157830 con 0x7fc9fc0c8430 2023-10-30T12:52:34.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fca02ed6640 1 -- 172.21.15.125:0/695963347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9fc04e170 con 0x7fc9fc0c8430 2023-10-30T12:52:34.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.206+0000 7fc9f97fa640 1 -- 172.21.15.125:0/695963347 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc9f4014ab0 con 0x7fc9fc0c8430 2023-10-30T12:52:34.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.210+0000 7fc9f97fa640 1 --2- 172.21.15.125:0/695963347 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9e00664a0 0x7fc9e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:34.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.210+0000 7fc9f97fa640 1 -- 172.21.15.125:0/695963347 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7fc9f4084b40 con 0x7fc9fc0c8430 2023-10-30T12:52:34.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.210+0000 7fca00c72640 1 --2- 172.21.15.125:0/695963347 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9e00664a0 0x7fc9e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:34.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.210+0000 7fca00c72640 1 --2- 172.21.15.125:0/695963347 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9e00664a0 0x7fc9e0068960 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc06f1b0 tx=0x7fc9f0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:34.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.214+0000 7fc9f97fa640 1 -- 172.21.15.125:0/695963347 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc9f4051ba0 con 0x7fc9fc0c8430 2023-10-30T12:52:34.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.390+0000 7fca02ed6640 1 -- 172.21.15.125:0/695963347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9fc0c1730 con 0x7fc9fc0c8430 2023-10-30T12:52:34.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.390+0000 7fc9f97fa640 1 -- 172.21.15.125:0/695963347 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5723 (secure 0 0 0) 0x7fc9f40551c0 con 0x7fc9fc0c8430 2023-10-30T12:52:34.392 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:34.393 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:30.335199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:34.393 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 118 2023-10-30T12:52:34.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.394+0000 7fca02ed6640 1 -- 172.21.15.125:0/695963347 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9e00664a0 msgr2=0x7fc9e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:34.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.394+0000 7fca02ed6640 1 --2- 172.21.15.125:0/695963347 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9e00664a0 0x7fc9e0068960 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc06f1b0 tx=0x7fc9f0008040 comp rx=0 tx=0).stop 2023-10-30T12:52:34.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.394+0000 7fca02ed6640 1 -- 172.21.15.125:0/695963347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9fc0c8430 msgr2=0x7fc9fc154d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:34.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.394+0000 7fca02ed6640 1 --2- 172.21.15.125:0/695963347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9fc0c8430 0x7fc9fc154d60 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fc9f4007e60 tx=0x7fc9f400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:52:34.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.394+0000 7fca02ed6640 1 -- 172.21.15.125:0/695963347 shutdown_connections 2023-10-30T12:52:34.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.394+0000 7fca02ed6640 1 --2- 172.21.15.125:0/695963347 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9e00664a0 0x7fc9e0068960 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.394+0000 7fca02ed6640 1 --2- 172.21.15.125:0/695963347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9fc0c8430 0x7fc9fc154d60 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.394+0000 7fca02ed6640 1 --2- 172.21.15.125:0/695963347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9fc0c6f60 0x7fc9fc158310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.394+0000 7fca02ed6640 1 --2- 172.21.15.125:0/695963347 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9fc0c6b80 0x7fc9fc13d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:34.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.394+0000 7fca02ed6640 1 -- 172.21.15.125:0/695963347 >> 172.21.15.125:0/695963347 conn(0x7fc9fc0c7910 msgr2=0x7fc9fc0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:34.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.394+0000 7fca02ed6640 1 -- 172.21.15.125:0/695963347 shutdown_connections 2023-10-30T12:52:34.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:34.394+0000 7fca02ed6640 1 -- 172.21.15.125:0/695963347 wait complete. 2023-10-30T12:52:34.414 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:52:36.416 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:36.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.561+0000 7f6dadfa3640 1 Processor -- start 2023-10-30T12:52:36.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.561+0000 7f6dadfa3640 1 -- start start 2023-10-30T12:52:36.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.561+0000 7f6dadfa3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6da81538f0 0x7f6da8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:36.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.561+0000 7f6dadfa3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6da815cd50 0x7f6da80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:36.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.561+0000 7f6dadfa3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6da815d300 con 0x7f6da80cefa0 2023-10-30T12:52:36.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.561+0000 7f6dadfa3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6da815d5e0 con 0x7f6da815cd50 2023-10-30T12:52:36.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.561+0000 7f6dadfa3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da815d8c0 con 0x7f6da81538f0 2023-10-30T12:52:36.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.561+0000 7f6dad7a2640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6da80cefa0 0x7f6da80cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37662/0 (socket says 172.21.15.125:37662) 2023-10-30T12:52:36.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.561+0000 7f6dad7a2640 1 -- 172.21.15.125:0/1649817311 learned_addr learned my addr 172.21.15.125:0/1649817311 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:52:36.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.561+0000 7f6dacfa1640 1 --2- 172.21.15.125:0/1649817311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6da81538f0 0x7f6da8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:36.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.561+0000 7f6da7fff640 1 --2- 172.21.15.125:0/1649817311 >> v2:172.21.15.136:3300/0 conn(0x7f6da815cd50 0x7f6da80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:36.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 -- 172.21.15.125:0/1649817311 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065140554 0 0) 0x7f6da815d300 con 0x7f6da80cefa0 2023-10-30T12:52:36.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 -- 172.21.15.125:0/1649817311 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d98003530 con 0x7f6da80cefa0 2023-10-30T12:52:36.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 -- 172.21.15.125:0/1649817311 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4014616930 0 0) 0x7f6d98003530 con 0x7f6da80cefa0 2023-10-30T12:52:36.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 -- 172.21.15.125:0/1649817311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6da81538f0 msgr2=0x7f6da8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:36.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 --2- 172.21.15.125:0/1649817311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6da81538f0 0x7f6da8159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:36.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 -- 172.21.15.125:0/1649817311 >> v2:172.21.15.136:3300/0 conn(0x7f6da815cd50 msgr2=0x7f6da80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:36.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 --2- 172.21.15.125:0/1649817311 >> v2:172.21.15.136:3300/0 conn(0x7f6da815cd50 0x7f6da80c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:36.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 -- 172.21.15.125:0/1649817311 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da815dba0 con 0x7f6da80cefa0 2023-10-30T12:52:36.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 -- 172.21.15.125:0/1649817311 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f6da0001a00 con 0x7f6da80cefa0 2023-10-30T12:52:36.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 -- 172.21.15.125:0/1649817311 >> v1:172.21.15.125:6789/0 conn(0x7f6da80cefa0 legacy=0x7f6da80cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:36.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 --2- 172.21.15.125:0/1649817311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d98003980 0x7f6d98003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:36.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 --2- 172.21.15.125:0/1649817311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6da81538f0 0x7f6d980047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:36.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 --2- 172.21.15.125:0/1649817311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d98004d10 0x7f6d98005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:36.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 -- 172.21.15.125:0/1649817311 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da815d5e0 con 0x7f6d98003980 2023-10-30T12:52:36.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 -- 172.21.15.125:0/1649817311 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da815d8c0 con 0x7f6d98004d10 2023-10-30T12:52:36.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da77fe640 1 -- 172.21.15.125:0/1649817311 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9800b990 con 0x7f6da81538f0 2023-10-30T12:52:36.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6dacfa1640 1 --2- 172.21.15.125:0/1649817311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d98003980 0x7f6d98003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:36.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6da7fff640 1 --2- 172.21.15.125:0/1649817311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6da81538f0 0x7f6d980047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:36.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6dad7a2640 1 --2- 172.21.15.125:0/1649817311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d98004d10 0x7f6d98005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:36.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6dacfa1640 1 -- 172.21.15.125:0/1649817311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6da81538f0 msgr2=0x7f6d980047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:36.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6dacfa1640 1 --2- 172.21.15.125:0/1649817311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6da81538f0 0x7f6d980047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:36.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6dacfa1640 1 -- 172.21.15.125:0/1649817311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d98004d10 msgr2=0x7f6d98005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:36.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6dacfa1640 1 --2- 172.21.15.125:0/1649817311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d98004d10 0x7f6d98005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:36.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6dacfa1640 1 -- 172.21.15.125:0/1649817311 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d98003530 con 0x7f6d98003980 2023-10-30T12:52:36.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.565+0000 7f6dad7a2640 1 --2- 172.21.15.125:0/1649817311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d98004d10 0x7f6d98005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:52:36.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.569+0000 7f6dacfa1640 1 --2- 172.21.15.125:0/1649817311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d98003980 0x7f6d98003da0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009ae0 tx=0x7f6d9c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f8764aac5e09ef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:36.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.569+0000 7f6da77fe640 1 -- 172.21.15.125:0/1649817311 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c0a5640 con 0x7f6d98003980 2023-10-30T12:52:36.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.569+0000 7f6da77fe640 1 -- 172.21.15.125:0/1649817311 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9c0a57a0 con 0x7f6d98003980 2023-10-30T12:52:36.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.569+0000 7f6da77fe640 1 -- 172.21.15.125:0/1649817311 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c0a5b50 con 0x7f6d98003980 2023-10-30T12:52:36.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.569+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1649817311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d98003980 msgr2=0x7f6d98003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:36.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.569+0000 7f6dadfa3640 1 --2- 172.21.15.125:0/1649817311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d98003980 0x7f6d98003da0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009ae0 tx=0x7f6d9c01ca50 comp rx=0 tx=0).stop 2023-10-30T12:52:36.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.569+0000 7f6dad7a2640 1 -- 172.21.15.125:0/1649817311 reap_dead start 2023-10-30T12:52:36.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.569+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1649817311 shutdown_connections 2023-10-30T12:52:36.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.569+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1649817311 >> 172.21.15.125:0/1649817311 conn(0x7f6da806a640 msgr2=0x7f6da8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:36.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.569+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1649817311 shutdown_connections 2023-10-30T12:52:36.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.569+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1649817311 wait complete. 2023-10-30T12:52:36.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.569+0000 7f6dadfa3640 1 Processor -- start 2023-10-30T12:52:36.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.569+0000 7f6dadfa3640 1 -- start start 2023-10-30T12:52:36.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dadfa3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d98003980 0x7f6da81f0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:36.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dadfa3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6da815b610 0x7f6da81480e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:36.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dadfa3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6da8150860 0x7f6da815bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:36.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dadfa3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da8151f60 con 0x7f6d98003980 2023-10-30T12:52:36.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dadfa3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da815a8e0 con 0x7f6da8150860 2023-10-30T12:52:36.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dadfa3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6da815b180 con 0x7f6da815b610 2023-10-30T12:52:36.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dacfa1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d98003980 0x7f6da81f0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:36.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dacfa1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d98003980 0x7f6da81f0a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40182/0 (socket says 172.21.15.125:40182) 2023-10-30T12:52:36.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dacfa1640 1 -- 172.21.15.125:0/1136963301 learned_addr learned my addr 172.21.15.125:0/1136963301 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:36.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dad7a2640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6da8150860 0x7f6da815bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:36.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6da7fff640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6da815b610 0x7f6da81480e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:36.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dacfa1640 1 -- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6da815b610 msgr2=0x7f6da81480e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:36.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dacfa1640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6da815b610 0x7f6da81480e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:36.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dacfa1640 1 -- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6da8150860 msgr2=0x7f6da815bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:36.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dacfa1640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6da8150860 0x7f6da815bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:36.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dacfa1640 1 -- 172.21.15.125:0/1136963301 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da815c680 con 0x7f6d98003980 2023-10-30T12:52:36.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6da7fff640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6da815b610 0x7f6da81480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:52:36.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dacfa1640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d98003980 0x7f6da81f0a60 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009ae0 tx=0x7f6d98005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:36.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6da57fa640 1 -- 172.21.15.125:0/1136963301 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c004230 con 0x7f6d98003980 2023-10-30T12:52:36.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6da57fa640 1 -- 172.21.15.125:0/1136963301 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d9c004390 con 0x7f6d98003980 2023-10-30T12:52:36.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1136963301 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da815c960 con 0x7f6d98003980 2023-10-30T12:52:36.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6da57fa640 1 -- 172.21.15.125:0/1136963301 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c0ad850 con 0x7f6d98003980 2023-10-30T12:52:36.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1136963301 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da815ae10 con 0x7f6d98003980 2023-10-30T12:52:36.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.573+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1136963301 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6da8159a70 con 0x7f6d98003980 2023-10-30T12:52:36.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.577+0000 7f6da57fa640 1 -- 172.21.15.125:0/1136963301 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6d9c0b4040 con 0x7f6d98003980 2023-10-30T12:52:36.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.577+0000 7f6da57fa640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6d8c0664a0 0x7f6d8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:36.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.577+0000 7f6da57fa640 1 -- 172.21.15.125:0/1136963301 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7f6d9c11ef60 con 0x7f6d98003980 2023-10-30T12:52:36.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.577+0000 7f6da7fff640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6d8c0664a0 0x7f6d8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:36.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.577+0000 7f6da7fff640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6d8c0664a0 0x7f6d8c068960 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6da8149590 tx=0x7f6d94008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:36.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.585+0000 7f6da57fa640 1 -- 172.21.15.125:0/1136963301 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6d9c0ec020 con 0x7f6d98003980 2023-10-30T12:52:36.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.785+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1136963301 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6da8159a70 con 0x7f6d98003980 2023-10-30T12:52:36.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.785+0000 7f6da57fa640 1 -- 172.21.15.125:0/1136963301 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f6da8159a70 con 0x7f6d98003980 2023-10-30T12:52:36.788 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:36.788 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:30.335199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:36.788 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 118 2023-10-30T12:52:36.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.785+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6d8c0664a0 msgr2=0x7f6d8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.785+0000 7f6dadfa3640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6d8c0664a0 0x7f6d8c068960 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f6da8149590 tx=0x7f6d94008040 comp rx=0 tx=0).stop 2023-10-30T12:52:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.785+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1136963301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d98003980 msgr2=0x7f6da81f0a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.785+0000 7f6dadfa3640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d98003980 0x7f6da81f0a60 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c009ae0 tx=0x7f6d98005800 comp rx=0 tx=0).stop 2023-10-30T12:52:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.789+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1136963301 shutdown_connections 2023-10-30T12:52:36.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.789+0000 7f6dadfa3640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6d8c0664a0 0x7f6d8c068960 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:36.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.789+0000 7f6dadfa3640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6da8150860 0x7f6da815bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:36.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.789+0000 7f6dadfa3640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6da815b610 0x7f6da81480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:36.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.789+0000 7f6dadfa3640 1 --2- 172.21.15.125:0/1136963301 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d98003980 0x7f6da81f0a60 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:36.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.789+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1136963301 >> 172.21.15.125:0/1136963301 conn(0x7f6da806a640 msgr2=0x7f6da80caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:36.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.789+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1136963301 shutdown_connections 2023-10-30T12:52:36.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:36.789+0000 7f6dadfa3640 1 -- 172.21.15.125:0/1136963301 wait complete. 2023-10-30T12:52:36.809 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:52:38.810 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:38.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.953+0000 7fab22a46640 1 Processor -- start 2023-10-30T12:52:38.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.953+0000 7fab22a46640 1 -- start start 2023-10-30T12:52:38.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.957+0000 7fab22a46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fab1c15d6f0 0x7fab1c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:38.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.957+0000 7fab22a46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab1c0cb800 con 0x7fab1c1538a0 2023-10-30T12:52:38.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.957+0000 7fab22a46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab1c0cb330 con 0x7fab1c0cf160 2023-10-30T12:52:38.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.957+0000 7fab22a46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab1c0cad70 con 0x7fab1c15d6f0 2023-10-30T12:52:38.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.957+0000 7fab1b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fab1c15d6f0 0x7fab1c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:38.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.957+0000 7fab1b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fab1c15d6f0 0x7fab1c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40192/0 (socket says 172.21.15.125:40192) 2023-10-30T12:52:38.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.957+0000 7fab1b7fe640 1 -- 172.21.15.125:0/3545046060 learned_addr learned my addr 172.21.15.125:0/3545046060 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:38.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.957+0000 7fab1b7fe640 1 -- 172.21.15.125:0/3545046060 >> v1:172.21.15.136:6789/0 conn(0x7fab1c0cf160 legacy=0x7fab1c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:38.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.957+0000 7fab1b7fe640 1 -- 172.21.15.125:0/3545046060 >> v1:172.21.15.125:6789/0 conn(0x7fab1c1538a0 legacy=0x7fab1c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:38.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.957+0000 7fab1b7fe640 1 -- 172.21.15.125:0/3545046060 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1c0cb050 con 0x7fab1c15d6f0 2023-10-30T12:52:38.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.961+0000 7fab1b7fe640 1 --2- 172.21.15.125:0/3545046060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab1c15d6f0 0x7fab1c15a090 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fab10009870 tx=0x7fab1001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10f18a8c7fe0c9a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:38.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.961+0000 7fab1affd640 1 -- 172.21.15.125:0/3545046060 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab100a8a30 con 0x7fab1c15d6f0 2023-10-30T12:52:38.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.961+0000 7fab1affd640 1 -- 172.21.15.125:0/3545046060 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab100a8b90 con 0x7fab1c15d6f0 2023-10-30T12:52:38.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.961+0000 7fab1affd640 1 -- 172.21.15.125:0/3545046060 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab100b0440 con 0x7fab1c15d6f0 2023-10-30T12:52:38.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.961+0000 7fab22a46640 1 -- 172.21.15.125:0/3545046060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab1c15d6f0 msgr2=0x7fab1c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:38.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.961+0000 7fab22a46640 1 --2- 172.21.15.125:0/3545046060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab1c15d6f0 0x7fab1c15a090 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fab10009870 tx=0x7fab1001ed30 comp rx=0 tx=0).stop 2023-10-30T12:52:38.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.961+0000 7fab22a46640 1 -- 172.21.15.125:0/3545046060 shutdown_connections 2023-10-30T12:52:38.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.961+0000 7fab22a46640 1 --2- 172.21.15.125:0/3545046060 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab1c15d6f0 0x7fab1c15a090 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:38.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.961+0000 7fab22a46640 1 -- 172.21.15.125:0/3545046060 >> 172.21.15.125:0/3545046060 conn(0x7fab1c06bfe0 msgr2=0x7fab1c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:38.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.961+0000 7fab22a46640 1 -- 172.21.15.125:0/3545046060 shutdown_connections 2023-10-30T12:52:38.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.961+0000 7fab22a46640 1 -- 172.21.15.125:0/3545046060 wait complete. 2023-10-30T12:52:38.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.961+0000 7fab22a46640 1 Processor -- start 2023-10-30T12:52:38.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab22a46640 1 -- start start 2023-10-30T12:52:38.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab22a46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab1c0cf160 0x7fab1c0c6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:38.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab22a46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab1c1538a0 0x7fab1c0ce550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:38.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab22a46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab1c15d6f0 0x7fab1c1e40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:38.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab22a46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c1592a0 con 0x7fab1c15d6f0 2023-10-30T12:52:38.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab22a46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c1e3980 con 0x7fab1c0cf160 2023-10-30T12:52:38.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab22a46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c1e3c60 con 0x7fab1c1538a0 2023-10-30T12:52:38.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab20fe3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab1c15d6f0 0x7fab1c1e40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:38.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab1bfff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab1c0cf160 0x7fab1c0c6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:38.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab20fe3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab1c15d6f0 0x7fab1c1e40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40204/0 (socket says 172.21.15.125:40204) 2023-10-30T12:52:38.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab20fe3640 1 -- 172.21.15.125:0/2256735920 learned_addr learned my addr 172.21.15.125:0/2256735920 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:38.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab1b7fe640 1 --2- 172.21.15.125:0/2256735920 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab1c1538a0 0x7fab1c0ce550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:38.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab20fe3640 1 -- 172.21.15.125:0/2256735920 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab1c1538a0 msgr2=0x7fab1c0ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:38.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab20fe3640 1 --2- 172.21.15.125:0/2256735920 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab1c1538a0 0x7fab1c0ce550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:38.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab20fe3640 1 -- 172.21.15.125:0/2256735920 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab1c0cf160 msgr2=0x7fab1c0c6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:38.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab20fe3640 1 --2- 172.21.15.125:0/2256735920 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab1c0cf160 0x7fab1c0c6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:38.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab20fe3640 1 -- 172.21.15.125:0/2256735920 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab100094d0 con 0x7fab1c15d6f0 2023-10-30T12:52:38.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab20fe3640 1 --2- 172.21.15.125:0/2256735920 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab1c15d6f0 0x7fab1c1e40f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fab1400c760 tx=0x7fab1400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:38.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab18ff9640 1 -- 172.21.15.125:0/2256735920 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab14003b80 con 0x7fab1c15d6f0 2023-10-30T12:52:38.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab18ff9640 1 -- 172.21.15.125:0/2256735920 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab14003ce0 con 0x7fab1c15d6f0 2023-10-30T12:52:38.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab18ff9640 1 -- 172.21.15.125:0/2256735920 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1400aa90 con 0x7fab1c15d6f0 2023-10-30T12:52:38.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab22a46640 1 -- 172.21.15.125:0/2256735920 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1c1e5690 con 0x7fab1c15d6f0 2023-10-30T12:52:38.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab22a46640 1 -- 172.21.15.125:0/2256735920 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1c1594f0 con 0x7fab1c15d6f0 2023-10-30T12:52:38.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.965+0000 7fab22a46640 1 -- 172.21.15.125:0/2256735920 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab1c04e170 con 0x7fab1c15d6f0 2023-10-30T12:52:38.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.969+0000 7fab18ff9640 1 -- 172.21.15.125:0/2256735920 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fab1400abf0 con 0x7fab1c15d6f0 2023-10-30T12:52:38.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.969+0000 7fab18ff9640 1 --2- 172.21.15.125:0/2256735920 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fab080664a0 0x7fab08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:38.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.969+0000 7fab18ff9640 1 -- 172.21.15.125:0/2256735920 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7fab14083e20 con 0x7fab1c15d6f0 2023-10-30T12:52:38.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.969+0000 7fab1bfff640 1 --2- 172.21.15.125:0/2256735920 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fab080664a0 0x7fab08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:38.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.969+0000 7fab1bfff640 1 --2- 172.21.15.125:0/2256735920 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fab080664a0 0x7fab08068960 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fab0c007fd0 tx=0x7fab0c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:38.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:38.973+0000 7fab18ff9640 1 -- 172.21.15.125:0/2256735920 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fab14050fb0 con 0x7fab1c15d6f0 2023-10-30T12:52:39.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:39.161+0000 7fab22a46640 1 -- 172.21.15.125:0/2256735920 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab1c0ca9b0 con 0x7fab1c15d6f0 2023-10-30T12:52:39.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:39.165+0000 7fab18ff9640 1 -- 172.21.15.125:0/2256735920 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+5723 (secure 0 0 0) 0x7fab140545d0 con 0x7fab1c15d6f0 2023-10-30T12:52:39.166 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:39.166 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:30.335199+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:39.166 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 118 2023-10-30T12:52:39.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:39.165+0000 7fab22a46640 1 -- 172.21.15.125:0/2256735920 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fab080664a0 msgr2=0x7fab08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:39.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:39.165+0000 7fab22a46640 1 --2- 172.21.15.125:0/2256735920 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fab080664a0 0x7fab08068960 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fab0c007fd0 tx=0x7fab0c009040 comp rx=0 tx=0).stop 2023-10-30T12:52:39.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:39.165+0000 7fab22a46640 1 -- 172.21.15.125:0/2256735920 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab1c15d6f0 msgr2=0x7fab1c1e40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:39.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:39.165+0000 7fab22a46640 1 --2- 172.21.15.125:0/2256735920 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab1c15d6f0 0x7fab1c1e40f0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fab1400c760 tx=0x7fab1400cc20 comp rx=0 tx=0).stop 2023-10-30T12:52:39.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:39.165+0000 7fab22a46640 1 -- 172.21.15.125:0/2256735920 shutdown_connections 2023-10-30T12:52:39.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:39.165+0000 7fab22a46640 1 --2- 172.21.15.125:0/2256735920 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fab080664a0 0x7fab08068960 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:39.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:39.165+0000 7fab22a46640 1 --2- 172.21.15.125:0/2256735920 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab1c15d6f0 0x7fab1c1e40f0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:39.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:39.165+0000 7fab22a46640 1 --2- 172.21.15.125:0/2256735920 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab1c1538a0 0x7fab1c0ce550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:39.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:39.165+0000 7fab22a46640 1 --2- 172.21.15.125:0/2256735920 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab1c0cf160 0x7fab1c0c6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:39.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:39.165+0000 7fab22a46640 1 -- 172.21.15.125:0/2256735920 >> 172.21.15.125:0/2256735920 conn(0x7fab1c06bfe0 msgr2=0x7fab1c1510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:39.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:39.165+0000 7fab22a46640 1 -- 172.21.15.125:0/2256735920 shutdown_connections 2023-10-30T12:52:39.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:39.165+0000 7fab22a46640 1 -- 172.21.15.125:0/2256735920 wait complete. 2023-10-30T12:52:39.187 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:52:41.188 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:41.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47b9a46640 1 Processor -- start 2023-10-30T12:52:41.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47b9a46640 1 -- start start 2023-10-30T12:52:41.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47b9a46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f47b40c05d0 0x7f47b413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:41.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47b9a46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47b40c6f60 0x7f47b413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:41.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47b9a46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b40c0a50 con 0x7f47b40c6b80 2023-10-30T12:52:41.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47b9a46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b406f6e0 con 0x7f47b40c05d0 2023-10-30T12:52:41.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47b9a46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b406f9c0 con 0x7f47b40c6f60 2023-10-30T12:52:41.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47aa7fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f47b40c05d0 0x7f47b413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:41.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47aa7fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f47b40c05d0 0x7f47b413ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33740/0 (socket says 172.21.15.125:33740) 2023-10-30T12:52:41.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47aa7fc640 1 -- 172.21.15.125:0/3047850121 learned_addr learned my addr 172.21.15.125:0/3047850121 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:41.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47b37fe640 1 --2- 172.21.15.125:0/3047850121 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47b40c6f60 0x7f47b413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:41.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47aa7fc640 1 -- 172.21.15.125:0/3047850121 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47b40c6f60 msgr2=0x7f47b413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:41.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47aa7fc640 1 --2- 172.21.15.125:0/3047850121 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47b40c6f60 0x7f47b413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:41.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47aa7fc640 1 -- 172.21.15.125:0/3047850121 >> v1:172.21.15.136:6789/0 conn(0x7f47b40c6b80 legacy=0x7f47b40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:41.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.329+0000 7f47aa7fc640 1 -- 172.21.15.125:0/3047850121 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b406fca0 con 0x7f47b40c05d0 2023-10-30T12:52:41.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.333+0000 7f47aa7fc640 1 --2- 172.21.15.125:0/3047850121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b40c05d0 0x7f47b413ed90 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f47a400bb10 tx=0x7f47a401ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da17628b02330d60 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:41.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.333+0000 7f47b27fc640 1 -- 172.21.15.125:0/3047850121 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a40a7bf0 con 0x7f47b40c05d0 2023-10-30T12:52:41.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.333+0000 7f47b27fc640 1 -- 172.21.15.125:0/3047850121 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a40a7d50 con 0x7f47b40c05d0 2023-10-30T12:52:41.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.333+0000 7f47b27fc640 1 -- 172.21.15.125:0/3047850121 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47a40af660 con 0x7f47b40c05d0 2023-10-30T12:52:41.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.333+0000 7f47b9a46640 1 -- 172.21.15.125:0/3047850121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b40c05d0 msgr2=0x7f47b413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:41.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.333+0000 7f47b9a46640 1 --2- 172.21.15.125:0/3047850121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b40c05d0 0x7f47b413ed90 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f47a400bb10 tx=0x7f47a401ec10 comp rx=0 tx=0).stop 2023-10-30T12:52:41.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.333+0000 7f47b9a46640 1 -- 172.21.15.125:0/3047850121 shutdown_connections 2023-10-30T12:52:41.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.333+0000 7f47b9a46640 1 --2- 172.21.15.125:0/3047850121 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47b40c6f60 0x7f47b413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:41.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.333+0000 7f47b9a46640 1 --2- 172.21.15.125:0/3047850121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b40c05d0 0x7f47b413ed90 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:41.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.333+0000 7f47b9a46640 1 -- 172.21.15.125:0/3047850121 >> 172.21.15.125:0/3047850121 conn(0x7f47b40c7910 msgr2=0x7f47b40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:41.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.333+0000 7f47b9a46640 1 -- 172.21.15.125:0/3047850121 shutdown_connections 2023-10-30T12:52:41.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.333+0000 7f47b9a46640 1 -- 172.21.15.125:0/3047850121 wait complete. 2023-10-30T12:52:41.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b9a46640 1 Processor -- start 2023-10-30T12:52:41.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b9a46640 1 -- start start 2023-10-30T12:52:41.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b9a46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47b40c05d0 0x7f47b4134c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:41.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b9a46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47b40c6b80 0x7f47b41302d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:41.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b9a46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b40c6f60 0x7f47b4130810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:41.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b9a46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b4154480 con 0x7f47b40c6f60 2023-10-30T12:52:41.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b9a46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b4134650 con 0x7f47b40c05d0 2023-10-30T12:52:41.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b9a46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b4133ef0 con 0x7f47b40c6b80 2023-10-30T12:52:41.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b40c6f60 0x7f47b4130810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:41.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b40c6f60 0x7f47b4130810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33742/0 (socket says 172.21.15.125:33742) 2023-10-30T12:52:41.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b37fe640 1 -- 172.21.15.125:0/1316648297 learned_addr learned my addr 172.21.15.125:0/1316648297 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:41.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47aa7fc640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47b40c6b80 0x7f47b41302d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:41.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b2ffd640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47b40c05d0 0x7f47b4134c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:41.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b37fe640 1 -- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47b40c6b80 msgr2=0x7f47b41302d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:41.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b37fe640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47b40c6b80 0x7f47b41302d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:41.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b37fe640 1 -- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47b40c05d0 msgr2=0x7f47b4134c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:41.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b37fe640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47b40c05d0 0x7f47b4134c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:41.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b37fe640 1 -- 172.21.15.125:0/1316648297 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47a400b770 con 0x7f47b40c6f60 2023-10-30T12:52:41.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b2ffd640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47b40c05d0 0x7f47b4134c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:52:41.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.337+0000 7f47b37fe640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b40c6f60 0x7f47b4130810 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f47ac007e90 tx=0x7f47ac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:41.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.341+0000 7f47b0ff9640 1 -- 172.21.15.125:0/1316648297 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac014420 con 0x7f47b40c6f60 2023-10-30T12:52:41.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.341+0000 7f47b0ff9640 1 -- 172.21.15.125:0/1316648297 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ac014580 con 0x7f47b40c6f60 2023-10-30T12:52:41.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.341+0000 7f47b0ff9640 1 -- 172.21.15.125:0/1316648297 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac014930 con 0x7f47b40c6f60 2023-10-30T12:52:41.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.341+0000 7f47b9a46640 1 -- 172.21.15.125:0/1316648297 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b4132890 con 0x7f47b40c6f60 2023-10-30T12:52:41.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.341+0000 7f47b9a46640 1 -- 172.21.15.125:0/1316648297 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b41546d0 con 0x7f47b40c6f60 2023-10-30T12:52:41.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.345+0000 7f47b9a46640 1 -- 172.21.15.125:0/1316648297 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b404e170 con 0x7f47b40c6f60 2023-10-30T12:52:41.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.345+0000 7f47b0ff9640 1 -- 172.21.15.125:0/1316648297 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f47ac014a90 con 0x7f47b40c6f60 2023-10-30T12:52:41.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.345+0000 7f47b0ff9640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f47940664d0 0x7f4794068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:41.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.345+0000 7f47b0ff9640 1 -- 172.21.15.125:0/1316648297 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7f47ac084bf0 con 0x7f47b40c6f60 2023-10-30T12:52:41.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.345+0000 7f47b2ffd640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f47940664d0 0x7f4794068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:41.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.345+0000 7f47b2ffd640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f47940664d0 0x7f4794068990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f479c000f70 tx=0x7f479c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:41.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.349+0000 7f47b0ff9640 1 -- 172.21.15.125:0/1316648297 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f47ac051ba0 con 0x7f47b40c6f60 2023-10-30T12:52:41.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.533+0000 7f47b9a46640 1 -- 172.21.15.125:0/1316648297 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47b413acb0 con 0x7f47b40c6f60 2023-10-30T12:52:41.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.533+0000 7f47b0ff9640 1 -- 172.21.15.125:0/1316648297 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f47ac0551c0 con 0x7f47b40c6f60 2023-10-30T12:52:41.535 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:41.535 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:40.388143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:41.536 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 119 2023-10-30T12:52:41.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.533+0000 7f47b9a46640 1 -- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f47940664d0 msgr2=0x7f4794068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:41.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.533+0000 7f47b9a46640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f47940664d0 0x7f4794068990 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f479c000f70 tx=0x7f479c009040 comp rx=0 tx=0).stop 2023-10-30T12:52:41.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.533+0000 7f47b9a46640 1 -- 172.21.15.125:0/1316648297 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b40c6f60 msgr2=0x7f47b4130810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:41.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.533+0000 7f47b9a46640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b40c6f60 0x7f47b4130810 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f47ac007e90 tx=0x7f47ac00e4b0 comp rx=0 tx=0).stop 2023-10-30T12:52:41.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.533+0000 7f47b9a46640 1 -- 172.21.15.125:0/1316648297 shutdown_connections 2023-10-30T12:52:41.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.533+0000 7f47b9a46640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f47940664d0 0x7f4794068990 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:41.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.533+0000 7f47b9a46640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47b40c6f60 0x7f47b4130810 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:41.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.533+0000 7f47b9a46640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47b40c6b80 0x7f47b41302d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:41.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.533+0000 7f47b9a46640 1 --2- 172.21.15.125:0/1316648297 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47b40c05d0 0x7f47b4134c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:41.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.533+0000 7f47b9a46640 1 -- 172.21.15.125:0/1316648297 >> 172.21.15.125:0/1316648297 conn(0x7f47b40c7910 msgr2=0x7f47b40bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:41.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.533+0000 7f47b9a46640 1 -- 172.21.15.125:0/1316648297 shutdown_connections 2023-10-30T12:52:41.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:41.533+0000 7f47b9a46640 1 -- 172.21.15.125:0/1316648297 wait complete. 2023-10-30T12:52:41.556 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:52:43.557 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:43.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.701+0000 7f7281365640 1 Processor -- start 2023-10-30T12:52:43.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.701+0000 7f7281365640 1 -- start start 2023-10-30T12:52:43.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.701+0000 7f7281365640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f727c153d70 0x7f727c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:43.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.701+0000 7f7281365640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f727c06e470 0x7f727c0cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:43.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f7281365640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f727c06afa0 con 0x7f727c0c8490 2023-10-30T12:52:43.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f7281365640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f727c06d780 con 0x7f727c153d70 2023-10-30T12:52:43.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f7281365640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f727c154150 con 0x7f727c06e470 2023-10-30T12:52:43.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f727affd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f727c153d70 0x7f727c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:43.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f727affd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f727c153d70 0x7f727c06ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33744/0 (socket says 172.21.15.125:33744) 2023-10-30T12:52:43.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f727affd640 1 -- 172.21.15.125:0/1127589014 learned_addr learned my addr 172.21.15.125:0/1127589014 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:43.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f727a7fc640 1 --2- 172.21.15.125:0/1127589014 >> v2:172.21.15.136:3300/0 conn(0x7f727c06e470 0x7f727c0cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:43.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f727affd640 1 -- 172.21.15.125:0/1127589014 >> v2:172.21.15.136:3300/0 conn(0x7f727c06e470 msgr2=0x7f727c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:43.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f727affd640 1 --2- 172.21.15.125:0/1127589014 >> v2:172.21.15.136:3300/0 conn(0x7f727c06e470 0x7f727c0cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:43.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f727affd640 1 -- 172.21.15.125:0/1127589014 >> v1:172.21.15.136:6789/0 conn(0x7f727c0c8490 legacy=0x7f727c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:43.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f727affd640 1 -- 172.21.15.125:0/1127589014 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f727c06e030 con 0x7f727c153d70 2023-10-30T12:52:43.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f727affd640 1 --2- 172.21.15.125:0/1127589014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c153d70 0x7f727c06ca80 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f727c070230 tx=0x7f726c01c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=517499c6b6666107 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:43.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f7279ffb640 1 -- 172.21.15.125:0/1127589014 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c0aa5f0 con 0x7f727c153d70 2023-10-30T12:52:43.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f7279ffb640 1 -- 172.21.15.125:0/1127589014 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726c0aa750 con 0x7f727c153d70 2023-10-30T12:52:43.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.705+0000 7f7279ffb640 1 -- 172.21.15.125:0/1127589014 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c0aaae0 con 0x7f727c153d70 2023-10-30T12:52:43.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 -- 172.21.15.125:0/1127589014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c153d70 msgr2=0x7f727c06ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:43.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 --2- 172.21.15.125:0/1127589014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c153d70 0x7f727c06ca80 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f727c070230 tx=0x7f726c01c490 comp rx=0 tx=0).stop 2023-10-30T12:52:43.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 -- 172.21.15.125:0/1127589014 shutdown_connections 2023-10-30T12:52:43.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 --2- 172.21.15.125:0/1127589014 >> v2:172.21.15.136:3300/0 conn(0x7f727c06e470 0x7f727c0cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:43.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 --2- 172.21.15.125:0/1127589014 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c153d70 0x7f727c06ca80 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:43.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 -- 172.21.15.125:0/1127589014 >> 172.21.15.125:0/1127589014 conn(0x7f727c071ea0 msgr2=0x7f727c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:43.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 -- 172.21.15.125:0/1127589014 shutdown_connections 2023-10-30T12:52:43.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 -- 172.21.15.125:0/1127589014 wait complete. 2023-10-30T12:52:43.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 Processor -- start 2023-10-30T12:52:43.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 -- start start 2023-10-30T12:52:43.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c06e470 0x7f727c1f6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:43.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f727c0c8490 0x7f727c14ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:43.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f727c153d70 0x7f727c14d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:43.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727c148da0 con 0x7f727c06e470 2023-10-30T12:52:43.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f727c0aecd0 con 0x7f727c153d70 2023-10-30T12:52:43.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f7281365640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f727c1f6c80 con 0x7f727c0c8490 2023-10-30T12:52:43.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f727affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c06e470 0x7f727c1f6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:43.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f727affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c06e470 0x7f727c1f6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33758/0 (socket says 172.21.15.125:33758) 2023-10-30T12:52:43.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f727affd640 1 -- 172.21.15.125:0/1392804911 learned_addr learned my addr 172.21.15.125:0/1392804911 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:43.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f727a7fc640 1 --2- 172.21.15.125:0/1392804911 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f727c0c8490 0x7f727c14ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:43.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f727b7fe640 1 --2- 172.21.15.125:0/1392804911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f727c153d70 0x7f727c14d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:43.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f727affd640 1 -- 172.21.15.125:0/1392804911 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f727c0c8490 msgr2=0x7f727c14ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:43.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f727affd640 1 --2- 172.21.15.125:0/1392804911 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f727c0c8490 0x7f727c14ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:43.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f727affd640 1 -- 172.21.15.125:0/1392804911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f727c153d70 msgr2=0x7f727c14d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:43.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f727affd640 1 --2- 172.21.15.125:0/1392804911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f727c153d70 0x7f727c14d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:43.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.709+0000 7f727affd640 1 -- 172.21.15.125:0/1392804911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726c0094d0 con 0x7f727c06e470 2023-10-30T12:52:43.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.713+0000 7f727affd640 1 --2- 172.21.15.125:0/1392804911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c06e470 0x7f727c1f6740 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f726c0ac040 tx=0x7f726c01acd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:43.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.713+0000 7f72637fe640 1 -- 172.21.15.125:0/1392804911 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c0aa5f0 con 0x7f727c06e470 2023-10-30T12:52:43.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.713+0000 7f72637fe640 1 -- 172.21.15.125:0/1392804911 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f726c0b3800 con 0x7f727c06e470 2023-10-30T12:52:43.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.713+0000 7f72637fe640 1 -- 172.21.15.125:0/1392804911 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f726c0b3b90 con 0x7f727c06e470 2023-10-30T12:52:43.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.713+0000 7f7281365640 1 -- 172.21.15.125:0/1392804911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f727c14c5e0 con 0x7f727c06e470 2023-10-30T12:52:43.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.713+0000 7f7281365640 1 -- 172.21.15.125:0/1392804911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f727c163fc0 con 0x7f727c06e470 2023-10-30T12:52:43.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.713+0000 7f7281365640 1 -- 172.21.15.125:0/1392804911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f727c06b4b0 con 0x7f727c06e470 2023-10-30T12:52:43.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.713+0000 7f72637fe640 1 -- 172.21.15.125:0/1392804911 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f726c0b3cf0 con 0x7f727c06e470 2023-10-30T12:52:43.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.713+0000 7f72637fe640 1 --2- 172.21.15.125:0/1392804911 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72680664d0 0x7f7268068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:43.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.713+0000 7f72637fe640 1 -- 172.21.15.125:0/1392804911 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7f726c0a7070 con 0x7f727c06e470 2023-10-30T12:52:43.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.717+0000 7f727a7fc640 1 --2- 172.21.15.125:0/1392804911 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72680664d0 0x7f7268068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:43.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.717+0000 7f727a7fc640 1 --2- 172.21.15.125:0/1392804911 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72680664d0 0x7f7268068990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f727c14f7a0 tx=0x7f7270008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:43.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.717+0000 7f72637fe640 1 -- 172.21.15.125:0/1392804911 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f727c06b4b0 con 0x7f727c06e470 2023-10-30T12:52:43.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.897+0000 7f7281365640 1 -- 172.21.15.125:0/1392804911 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f727c06b4b0 con 0x7f727c06e470 2023-10-30T12:52:43.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.901+0000 7f72637fe640 1 -- 172.21.15.125:0/1392804911 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f727c06b4b0 con 0x7f727c06e470 2023-10-30T12:52:43.902 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:43.902 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:40.388143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:43.903 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 119 2023-10-30T12:52:43.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.901+0000 7f7281365640 1 -- 172.21.15.125:0/1392804911 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72680664d0 msgr2=0x7f7268068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:43.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.901+0000 7f7281365640 1 --2- 172.21.15.125:0/1392804911 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72680664d0 0x7f7268068990 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f727c14f7a0 tx=0x7f7270008040 comp rx=0 tx=0).stop 2023-10-30T12:52:43.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.901+0000 7f7281365640 1 -- 172.21.15.125:0/1392804911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c06e470 msgr2=0x7f727c1f6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:43.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.901+0000 7f7281365640 1 --2- 172.21.15.125:0/1392804911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c06e470 0x7f727c1f6740 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f726c0ac040 tx=0x7f726c01acd0 comp rx=0 tx=0).stop 2023-10-30T12:52:43.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.901+0000 7f7281365640 1 -- 172.21.15.125:0/1392804911 shutdown_connections 2023-10-30T12:52:43.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.901+0000 7f7281365640 1 --2- 172.21.15.125:0/1392804911 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72680664d0 0x7f7268068990 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:43.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.901+0000 7f7281365640 1 --2- 172.21.15.125:0/1392804911 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f727c153d70 0x7f727c14d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:43.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.901+0000 7f7281365640 1 --2- 172.21.15.125:0/1392804911 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f727c0c8490 0x7f727c14ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:43.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.901+0000 7f7281365640 1 --2- 172.21.15.125:0/1392804911 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727c06e470 0x7f727c1f6740 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:43.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.901+0000 7f7281365640 1 -- 172.21.15.125:0/1392804911 >> 172.21.15.125:0/1392804911 conn(0x7f727c071ea0 msgr2=0x7f727c152d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:43.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.901+0000 7f7281365640 1 -- 172.21.15.125:0/1392804911 shutdown_connections 2023-10-30T12:52:43.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:43.901+0000 7f7281365640 1 -- 172.21.15.125:0/1392804911 wait complete. 2023-10-30T12:52:43.923 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 119, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 119, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6954": { "addr": "172.21.15.125:6836/1163622496", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1163622496, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1163622496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6954, "incarnation": 118, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:reconnect", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:52:40.388143+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6954 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:52:45.924 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:46.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e3f94640 1 Processor -- start 2023-10-30T12:52:46.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e3f94640 1 -- start start 2023-10-30T12:52:46.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e3f94640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f58dc15d6f0 0x7f58dc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:46.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e3f94640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58dc0cb800 con 0x7f58dc0cf160 2023-10-30T12:52:46.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e3f94640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58dc0cb330 con 0x7f58dc1538a0 2023-10-30T12:52:46.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e3f94640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58dc0cad70 con 0x7f58dc15d6f0 2023-10-30T12:52:46.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e2531640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f58dc0cf160 0x7f58dc164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33506/0 (socket says 172.21.15.125:33506) 2023-10-30T12:52:46.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e2531640 1 -- 172.21.15.125:0/4180421582 learned_addr learned my addr 172.21.15.125:0/4180421582 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:52:46.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e152f640 1 --2- 172.21.15.125:0/4180421582 >> v2:172.21.15.136:3300/0 conn(0x7f58dc15d6f0 0x7f58dc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:46.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 -- 172.21.15.125:0/4180421582 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279272717 0 0) 0x7f58dc0cb800 con 0x7f58dc0cf160 2023-10-30T12:52:46.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 -- 172.21.15.125:0/4180421582 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c8003530 con 0x7f58dc0cf160 2023-10-30T12:52:46.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 -- 172.21.15.125:0/4180421582 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2053354863 0 0) 0x7f58c8003530 con 0x7f58dc0cf160 2023-10-30T12:52:46.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 -- 172.21.15.125:0/4180421582 >> v2:172.21.15.136:3300/0 conn(0x7f58dc15d6f0 msgr2=0x7f58dc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:46.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 --2- 172.21.15.125:0/4180421582 >> v2:172.21.15.136:3300/0 conn(0x7f58dc15d6f0 0x7f58dc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:46.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 -- 172.21.15.125:0/4180421582 >> v1:172.21.15.136:6789/0 conn(0x7f58dc1538a0 legacy=0x7f58dc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:46.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 -- 172.21.15.125:0/4180421582 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc0cb050 con 0x7f58dc0cf160 2023-10-30T12:52:46.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 -- 172.21.15.125:0/4180421582 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f58d4001a00 con 0x7f58dc0cf160 2023-10-30T12:52:46.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 -- 172.21.15.125:0/4180421582 >> v1:172.21.15.125:6789/0 conn(0x7f58dc0cf160 legacy=0x7f58dc164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:46.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 --2- 172.21.15.125:0/4180421582 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c8003980 0x7f58c8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:46.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 --2- 172.21.15.125:0/4180421582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58c80047d0 0x7f58c8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:46.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 --2- 172.21.15.125:0/4180421582 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58c8005190 0x7f58c800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:46.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 -- 172.21.15.125:0/4180421582 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc0cad70 con 0x7f58c8003980 2023-10-30T12:52:46.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 -- 172.21.15.125:0/4180421582 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c800c040 con 0x7f58c80047d0 2023-10-30T12:52:46.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e0d2e640 1 -- 172.21.15.125:0/4180421582 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c800c200 con 0x7f58c8005190 2023-10-30T12:52:46.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.069+0000 7f58e1d30640 1 --2- 172.21.15.125:0/4180421582 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c8003980 0x7f58c8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:46.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.073+0000 7f58e152f640 1 --2- 172.21.15.125:0/4180421582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58c80047d0 0x7f58c8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:46.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.073+0000 7f58e2531640 1 --2- 172.21.15.125:0/4180421582 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58c8005190 0x7f58c800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:46.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.073+0000 7f58e1d30640 1 -- 172.21.15.125:0/4180421582 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58c8005190 msgr2=0x7f58c800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:46.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.073+0000 7f58e1d30640 1 --2- 172.21.15.125:0/4180421582 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58c8005190 0x7f58c800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:46.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.073+0000 7f58e1d30640 1 -- 172.21.15.125:0/4180421582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58c80047d0 msgr2=0x7f58c8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:46.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.073+0000 7f58e1d30640 1 --2- 172.21.15.125:0/4180421582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58c80047d0 0x7f58c8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:46.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.073+0000 7f58e1d30640 1 -- 172.21.15.125:0/4180421582 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c8003530 con 0x7f58c8003980 2023-10-30T12:52:46.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.073+0000 7f58e2531640 1 -- 172.21.15.125:0/4180421582 reap_dead start 2023-10-30T12:52:46.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.073+0000 7f58e1d30640 1 --2- 172.21.15.125:0/4180421582 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c8003980 0x7f58c8003da0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f58d8001780 tx=0x7f58d801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471a3a987992491d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:46.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.073+0000 7f58e0d2e640 1 -- 172.21.15.125:0/4180421582 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d80a4ea0 con 0x7f58c8003980 2023-10-30T12:52:46.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.073+0000 7f58e0d2e640 1 -- 172.21.15.125:0/4180421582 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d80a7050 con 0x7f58c8003980 2023-10-30T12:52:46.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.073+0000 7f58e0d2e640 1 -- 172.21.15.125:0/4180421582 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8003600 con 0x7f58c8003980 2023-10-30T12:52:46.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.073+0000 7f58e3f94640 1 -- 172.21.15.125:0/4180421582 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c8003980 msgr2=0x7f58c8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:46.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.073+0000 7f58e3f94640 1 --2- 172.21.15.125:0/4180421582 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c8003980 0x7f58c8003da0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f58d8001780 tx=0x7f58d801c3e0 comp rx=0 tx=0).stop 2023-10-30T12:52:46.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e3f94640 1 -- 172.21.15.125:0/4180421582 shutdown_connections 2023-10-30T12:52:46.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e3f94640 1 --2- 172.21.15.125:0/4180421582 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c8003980 0x7f58c8003da0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:46.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e3f94640 1 -- 172.21.15.125:0/4180421582 >> 172.21.15.125:0/4180421582 conn(0x7f58dc06bfe0 msgr2=0x7f58dc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:46.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e3f94640 1 -- 172.21.15.125:0/4180421582 shutdown_connections 2023-10-30T12:52:46.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e3f94640 1 -- 172.21.15.125:0/4180421582 wait complete. 2023-10-30T12:52:46.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e3f94640 1 Processor -- start 2023-10-30T12:52:46.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e3f94640 1 -- start start 2023-10-30T12:52:46.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e3f94640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c8003980 0x7f58dc158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:46.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e3f94640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58dc1e5530 0x7f58dc0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:46.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e3f94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58dc0cdc20 0x7f58dc1e3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:46.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e3f94640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc0c43d0 con 0x7f58c8003980 2023-10-30T12:52:46.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e3f94640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc0ccfb0 con 0x7f58dc1e5530 2023-10-30T12:52:46.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e3f94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58dc0c41a0 con 0x7f58dc0cdc20 2023-10-30T12:52:46.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e1d30640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c8003980 0x7f58dc158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:46.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e1d30640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c8003980 0x7f58dc158810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33774/0 (socket says 172.21.15.125:33774) 2023-10-30T12:52:46.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e1d30640 1 -- 172.21.15.125:0/191004200 learned_addr learned my addr 172.21.15.125:0/191004200 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:46.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e152f640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58dc1e5530 0x7f58dc0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:46.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e2531640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58dc0cdc20 0x7f58dc1e3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:46.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e1d30640 1 -- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58dc0cdc20 msgr2=0x7f58dc1e3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:46.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e1d30640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58dc0cdc20 0x7f58dc1e3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:46.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e1d30640 1 -- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58dc1e5530 msgr2=0x7f58dc0cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:46.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e1d30640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58dc1e5530 0x7f58dc0cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:46.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e1d30640 1 -- 172.21.15.125:0/191004200 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58dc0ce9f0 con 0x7f58c8003980 2023-10-30T12:52:46.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e152f640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58dc1e5530 0x7f58dc0cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:52:46.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.077+0000 7f58e1d30640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c8003980 0x7f58dc158810 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f58d801c210 tx=0x7f58c8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:46.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.081+0000 7f58d27fc640 1 -- 172.21.15.125:0/191004200 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8003e40 con 0x7f58c8003980 2023-10-30T12:52:46.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.081+0000 7f58d27fc640 1 -- 172.21.15.125:0/191004200 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d80a4ea0 con 0x7f58c8003980 2023-10-30T12:52:46.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.081+0000 7f58e3f94640 1 -- 172.21.15.125:0/191004200 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58dc0cd250 con 0x7f58c8003980 2023-10-30T12:52:46.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.081+0000 7f58d27fc640 1 -- 172.21.15.125:0/191004200 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d8007270 con 0x7f58c8003980 2023-10-30T12:52:46.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.081+0000 7f58e3f94640 1 -- 172.21.15.125:0/191004200 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58dc0663e0 con 0x7f58c8003980 2023-10-30T12:52:46.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.081+0000 7f58e3f94640 1 -- 172.21.15.125:0/191004200 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58dc04e170 con 0x7f58c8003980 2023-10-30T12:52:46.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.081+0000 7f58d27fc640 1 -- 172.21.15.125:0/191004200 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f58d80073d0 con 0x7f58c8003980 2023-10-30T12:52:46.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.081+0000 7f58d27fc640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58c00664a0 0x7f58c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:46.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.081+0000 7f58d27fc640 1 -- 172.21.15.125:0/191004200 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7f58d811e780 con 0x7f58c8003980 2023-10-30T12:52:46.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.081+0000 7f58e152f640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58c00664a0 0x7f58c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:46.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.085+0000 7f58e152f640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58c00664a0 0x7f58c0068960 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f58dc15a260 tx=0x7f58cc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:46.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.085+0000 7f58d27fc640 1 -- 172.21.15.125:0/191004200 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f58d80eb7c0 con 0x7f58c8003980 2023-10-30T12:52:46.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.265+0000 7f58e3f94640 1 -- 172.21.15.125:0/191004200 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58dc0b1dd0 con 0x7f58c8003980 2023-10-30T12:52:46.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.265+0000 7f58d27fc640 1 -- 172.21.15.125:0/191004200 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 119 v119) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f58d8007a40 con 0x7f58c8003980 2023-10-30T12:52:46.271 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:46.271 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":119,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":119,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:40.388143+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:reconnect","state_seq":13,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:46.272 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 119 2023-10-30T12:52:46.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.269+0000 7f58e3f94640 1 -- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58c00664a0 msgr2=0x7f58c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:46.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.269+0000 7f58e3f94640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58c00664a0 0x7f58c0068960 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f58dc15a260 tx=0x7f58cc008040 comp rx=0 tx=0).stop 2023-10-30T12:52:46.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.269+0000 7f58e3f94640 1 -- 172.21.15.125:0/191004200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c8003980 msgr2=0x7f58dc158810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:46.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.269+0000 7f58e3f94640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c8003980 0x7f58dc158810 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f58d801c210 tx=0x7f58c8005800 comp rx=0 tx=0).stop 2023-10-30T12:52:46.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.269+0000 7f58e3f94640 1 -- 172.21.15.125:0/191004200 shutdown_connections 2023-10-30T12:52:46.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.269+0000 7f58e3f94640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58c00664a0 0x7f58c0068960 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:46.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.269+0000 7f58e3f94640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58dc0cdc20 0x7f58dc1e3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:46.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.269+0000 7f58e3f94640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58dc1e5530 0x7f58dc0cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:46.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.269+0000 7f58e3f94640 1 --2- 172.21.15.125:0/191004200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58c8003980 0x7f58dc158810 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:46.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.269+0000 7f58e3f94640 1 -- 172.21.15.125:0/191004200 >> 172.21.15.125:0/191004200 conn(0x7f58dc06bfe0 msgr2=0x7f58dc15b1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:46.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.269+0000 7f58e3f94640 1 -- 172.21.15.125:0/191004200 shutdown_connections 2023-10-30T12:52:46.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:46.269+0000 7f58e3f94640 1 -- 172.21.15.125:0/191004200 wait complete. 2023-10-30T12:52:46.289 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:52:48.084 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:52:48.088 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:52:48.290 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:48.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.433+0000 7fdfa6cbb640 1 Processor -- start 2023-10-30T12:52:48.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa6cbb640 1 -- start start 2023-10-30T12:52:48.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa6cbb640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdfa01538f0 0x7fdfa0159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:48.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa6cbb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdfa015cd50 0x7fdfa00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:48.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa6cbb640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdfa00cefa0 0x7fdfa00cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:48.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa6cbb640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa015d300 con 0x7fdfa01538f0 2023-10-30T12:52:48.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa6cbb640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfa015d5e0 con 0x7fdfa00cefa0 2023-10-30T12:52:48.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa6cbb640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa015d8c0 con 0x7fdfa015cd50 2023-10-30T12:52:48.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa5cb9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdfa01538f0 0x7fdfa0159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:48.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa5cb9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdfa01538f0 0x7fdfa0159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33782/0 (socket says 172.21.15.125:33782) 2023-10-30T12:52:48.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa5cb9640 1 -- 172.21.15.125:0/522545950 learned_addr learned my addr 172.21.15.125:0/522545950 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:48.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa64ba640 1 --2- 172.21.15.125:0/522545950 >> v2:172.21.15.136:3300/0 conn(0x7fdfa00cefa0 0x7fdfa00cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:48.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa54b8640 1 --2- 172.21.15.125:0/522545950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdfa015cd50 0x7fdfa00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:48.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa5cb9640 1 -- 172.21.15.125:0/522545950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdfa015cd50 msgr2=0x7fdfa00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:48.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa5cb9640 1 --2- 172.21.15.125:0/522545950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdfa015cd50 0x7fdfa00c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:48.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa5cb9640 1 -- 172.21.15.125:0/522545950 >> v2:172.21.15.136:3300/0 conn(0x7fdfa00cefa0 msgr2=0x7fdfa00cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:48.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa5cb9640 1 --2- 172.21.15.125:0/522545950 >> v2:172.21.15.136:3300/0 conn(0x7fdfa00cefa0 0x7fdfa00cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:48.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.437+0000 7fdfa5cb9640 1 -- 172.21.15.125:0/522545950 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa015dba0 con 0x7fdfa01538f0 2023-10-30T12:52:48.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa5cb9640 1 --2- 172.21.15.125:0/522545950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa01538f0 0x7fdfa0159fb0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00b1ae0 tx=0x7fdf9c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbae1f3ffa5629ea server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:48.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa4cb7640 1 -- 172.21.15.125:0/522545950 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c0a89b0 con 0x7fdfa01538f0 2023-10-30T12:52:48.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa4cb7640 1 -- 172.21.15.125:0/522545950 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf9c0a8b10 con 0x7fdfa01538f0 2023-10-30T12:52:48.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa4cb7640 1 -- 172.21.15.125:0/522545950 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf9c0a8ea0 con 0x7fdfa01538f0 2023-10-30T12:52:48.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/522545950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa01538f0 msgr2=0x7fdfa0159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:48.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa6cbb640 1 --2- 172.21.15.125:0/522545950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa01538f0 0x7fdfa0159fb0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fdfa00b1ae0 tx=0x7fdf9c01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:52:48.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/522545950 shutdown_connections 2023-10-30T12:52:48.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa6cbb640 1 --2- 172.21.15.125:0/522545950 >> v2:172.21.15.136:3300/0 conn(0x7fdfa00cefa0 0x7fdfa00cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:48.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa6cbb640 1 --2- 172.21.15.125:0/522545950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdfa015cd50 0x7fdfa00c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:48.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa6cbb640 1 --2- 172.21.15.125:0/522545950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa01538f0 0x7fdfa0159fb0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:48.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/522545950 >> 172.21.15.125:0/522545950 conn(0x7fdfa006a640 msgr2=0x7fdfa0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:48.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/522545950 shutdown_connections 2023-10-30T12:52:48.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/522545950 wait complete. 2023-10-30T12:52:48.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa6cbb640 1 Processor -- start 2023-10-30T12:52:48.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.441+0000 7fdfa6cbb640 1 -- start start 2023-10-30T12:52:48.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa6cbb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa00cefa0 0x7fdfa00cd920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:48.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa6cbb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdfa01538f0 0x7fdfa0147310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:48.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa6cbb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa015cd50 0x7fdfa01f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:48.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa6cbb640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa00cedd0 con 0x7fdfa015cd50 2023-10-30T12:52:48.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa6cbb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa0151bf0 con 0x7fdfa00cefa0 2023-10-30T12:52:48.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa6cbb640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfa0152080 con 0x7fdfa01538f0 2023-10-30T12:52:48.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa64ba640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa015cd50 0x7fdfa01f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:48.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa64ba640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa015cd50 0x7fdfa01f27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33798/0 (socket says 172.21.15.125:33798) 2023-10-30T12:52:48.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa64ba640 1 -- 172.21.15.125:0/3531845575 learned_addr learned my addr 172.21.15.125:0/3531845575 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:48.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa54b8640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdfa01538f0 0x7fdfa0147310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:48.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa5cb9640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa00cefa0 0x7fdfa00cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:48.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa64ba640 1 -- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdfa01538f0 msgr2=0x7fdfa0147310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:48.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa64ba640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdfa01538f0 0x7fdfa0147310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:48.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa64ba640 1 -- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa00cefa0 msgr2=0x7fdfa00cd920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:48.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa64ba640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa00cefa0 0x7fdfa00cd920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:48.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa64ba640 1 -- 172.21.15.125:0/3531845575 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf9c0094d0 con 0x7fdfa015cd50 2023-10-30T12:52:48.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa54b8640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdfa01538f0 0x7fdfa0147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:52:48.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa5cb9640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa00cefa0 0x7fdfa00cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:52:48.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa64ba640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa015cd50 0x7fdfa01f27c0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fdf98007ec0 tx=0x7fdf9800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:48.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdf967fc640 1 -- 172.21.15.125:0/3531845575 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf98014620 con 0x7fdfa015cd50 2023-10-30T12:52:48.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdf967fc640 1 -- 172.21.15.125:0/3531845575 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf980147a0 con 0x7fdfa015cd50 2023-10-30T12:52:48.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdf967fc640 1 -- 172.21.15.125:0/3531845575 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf98014b50 con 0x7fdfa015cd50 2023-10-30T12:52:48.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/3531845575 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfa0147c40 con 0x7fdfa015cd50 2023-10-30T12:52:48.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/3531845575 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfa01512c0 con 0x7fdfa015cd50 2023-10-30T12:52:48.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/3531845575 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfa004e170 con 0x7fdfa015cd50 2023-10-30T12:52:48.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.445+0000 7fdf967fc640 1 -- 172.21.15.125:0/3531845575 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdf98014cb0 con 0x7fdfa015cd50 2023-10-30T12:52:48.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.449+0000 7fdf967fc640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdf840664d0 0x7fdf84068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:48.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.449+0000 7fdf967fc640 1 -- 172.21.15.125:0/3531845575 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7fdf98084c60 con 0x7fdfa015cd50 2023-10-30T12:52:48.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.449+0000 7fdfa5cb9640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdf840664d0 0x7fdf84068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:48.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.449+0000 7fdfa5cb9640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdf840664d0 0x7fdf84068990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0b2000 tx=0x7fdf9c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:48.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.453+0000 7fdf967fc640 1 -- 172.21.15.125:0/3531845575 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdf98051c10 con 0x7fdfa015cd50 2023-10-30T12:52:48.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.633+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/3531845575 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfa0151d30 con 0x7fdfa015cd50 2023-10-30T12:52:48.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.633+0000 7fdf967fc640 1 -- 172.21.15.125:0/3531845575 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5724 (secure 0 0 0) 0x7fdf98055230 con 0x7fdfa015cd50 2023-10-30T12:52:48.637 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:48.637 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:46.367136+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:48.637 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 120 2023-10-30T12:52:48.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.637+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdf840664d0 msgr2=0x7fdf84068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:48.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.637+0000 7fdfa6cbb640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdf840664d0 0x7fdf84068990 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fdf9c0b2000 tx=0x7fdf9c008040 comp rx=0 tx=0).stop 2023-10-30T12:52:48.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.637+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/3531845575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa015cd50 msgr2=0x7fdfa01f27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:48.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.637+0000 7fdfa6cbb640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa015cd50 0x7fdfa01f27c0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fdf98007ec0 tx=0x7fdf9800e4b0 comp rx=0 tx=0).stop 2023-10-30T12:52:48.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.637+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/3531845575 shutdown_connections 2023-10-30T12:52:48.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.637+0000 7fdfa6cbb640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdf840664d0 0x7fdf84068990 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:48.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.637+0000 7fdfa6cbb640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdfa015cd50 0x7fdfa01f27c0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:48.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.637+0000 7fdfa6cbb640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdfa01538f0 0x7fdfa0147310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:48.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.637+0000 7fdfa6cbb640 1 --2- 172.21.15.125:0/3531845575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdfa00cefa0 0x7fdfa00cd920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:48.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.637+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/3531845575 >> 172.21.15.125:0/3531845575 conn(0x7fdfa006a640 msgr2=0x7fdfa00c4350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:48.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.637+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/3531845575 shutdown_connections 2023-10-30T12:52:48.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:48.637+0000 7fdfa6cbb640 1 -- 172.21.15.125:0/3531845575 wait complete. 2023-10-30T12:52:48.657 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:52:50.658 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:50.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb6f846640 1 Processor -- start 2023-10-30T12:52:50.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb6f846640 1 -- start start 2023-10-30T12:52:50.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb6f846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb6815d6f0 0x7ffb6815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:50.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb6f846640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ffb680cf160 0x7ffb680cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:50.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb6f846640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb680cb800 con 0x7ffb681538a0 2023-10-30T12:52:50.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb6f846640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb680cb330 con 0x7ffb680cf160 2023-10-30T12:52:50.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb6f846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb680cad70 con 0x7ffb6815d6f0 2023-10-30T12:52:50.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb6d5e2640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ffb681538a0 0x7ffb680ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59728/0 (socket says 172.21.15.125:59728) 2023-10-30T12:52:50.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb6d5e2640 1 -- 172.21.15.125:0/1231665711 learned_addr learned my addr 172.21.15.125:0/1231665711 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:52:50.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb6cde1640 1 --2- 172.21.15.125:0/1231665711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb6815d6f0 0x7ffb6815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:50.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb6dde3640 1 --2- 172.21.15.125:0/1231665711 >> v2:172.21.15.136:3300/0 conn(0x7ffb680cf160 0x7ffb680cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:50.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 -- 172.21.15.125:0/1231665711 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2031727998 0 0) 0x7ffb680cb800 con 0x7ffb681538a0 2023-10-30T12:52:50.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 -- 172.21.15.125:0/1231665711 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb54003530 con 0x7ffb681538a0 2023-10-30T12:52:50.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 -- 172.21.15.125:0/1231665711 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 620958609 0 0) 0x7ffb54003530 con 0x7ffb681538a0 2023-10-30T12:52:50.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 -- 172.21.15.125:0/1231665711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb6815d6f0 msgr2=0x7ffb6815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:50.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 --2- 172.21.15.125:0/1231665711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb6815d6f0 0x7ffb6815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:50.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 -- 172.21.15.125:0/1231665711 >> v2:172.21.15.136:3300/0 conn(0x7ffb680cf160 msgr2=0x7ffb680cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:50.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 --2- 172.21.15.125:0/1231665711 >> v2:172.21.15.136:3300/0 conn(0x7ffb680cf160 0x7ffb680cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:50.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 -- 172.21.15.125:0/1231665711 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb680cb050 con 0x7ffb681538a0 2023-10-30T12:52:50.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 -- 172.21.15.125:0/1231665711 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7ffb64001680 con 0x7ffb681538a0 2023-10-30T12:52:50.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 -- 172.21.15.125:0/1231665711 >> v1:172.21.15.125:6789/0 conn(0x7ffb681538a0 legacy=0x7ffb680ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:50.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 --2- 172.21.15.125:0/1231665711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb54003980 0x7ffb54003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:50.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 --2- 172.21.15.125:0/1231665711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb540047d0 0x7ffb54004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:50.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 --2- 172.21.15.125:0/1231665711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb6815d6f0 0x7ffb54005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:50.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 -- 172.21.15.125:0/1231665711 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb680cb330 con 0x7ffb54003980 2023-10-30T12:52:50.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 -- 172.21.15.125:0/1231665711 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb680cad70 con 0x7ffb540047d0 2023-10-30T12:52:50.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb5ffff640 1 -- 172.21.15.125:0/1231665711 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5400b9c0 con 0x7ffb6815d6f0 2023-10-30T12:52:50.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.805+0000 7ffb6d5e2640 1 --2- 172.21.15.125:0/1231665711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb54003980 0x7ffb54003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:50.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.809+0000 7ffb6cde1640 1 --2- 172.21.15.125:0/1231665711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb540047d0 0x7ffb54004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:50.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.809+0000 7ffb6dde3640 1 --2- 172.21.15.125:0/1231665711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb6815d6f0 0x7ffb54005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:50.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.809+0000 7ffb6d5e2640 1 -- 172.21.15.125:0/1231665711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb6815d6f0 msgr2=0x7ffb54005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:50.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.809+0000 7ffb6d5e2640 1 --2- 172.21.15.125:0/1231665711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb6815d6f0 0x7ffb54005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:50.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.809+0000 7ffb6d5e2640 1 -- 172.21.15.125:0/1231665711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb540047d0 msgr2=0x7ffb54004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:50.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.809+0000 7ffb6d5e2640 1 --2- 172.21.15.125:0/1231665711 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb540047d0 0x7ffb54004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:50.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.809+0000 7ffb6d5e2640 1 -- 172.21.15.125:0/1231665711 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb54003530 con 0x7ffb54003980 2023-10-30T12:52:50.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.809+0000 7ffb6d5e2640 1 --2- 172.21.15.125:0/1231665711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb54003980 0x7ffb54003da0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7ffb6400c210 tx=0x7ffb6401f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da6f38980e22250 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:50.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.809+0000 7ffb5ffff640 1 -- 172.21.15.125:0/1231665711 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb640a8ce0 con 0x7ffb54003980 2023-10-30T12:52:50.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.809+0000 7ffb5ffff640 1 -- 172.21.15.125:0/1231665711 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb640a8e40 con 0x7ffb54003980 2023-10-30T12:52:50.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.809+0000 7ffb5ffff640 1 -- 172.21.15.125:0/1231665711 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64005b00 con 0x7ffb54003980 2023-10-30T12:52:50.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6f846640 1 -- 172.21.15.125:0/1231665711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb54003980 msgr2=0x7ffb54003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:50.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6f846640 1 --2- 172.21.15.125:0/1231665711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb54003980 0x7ffb54003da0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7ffb6400c210 tx=0x7ffb6401f3b0 comp rx=0 tx=0).stop 2023-10-30T12:52:50.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6dde3640 1 -- 172.21.15.125:0/1231665711 reap_dead start 2023-10-30T12:52:50.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6f846640 1 -- 172.21.15.125:0/1231665711 shutdown_connections 2023-10-30T12:52:50.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6f846640 1 -- 172.21.15.125:0/1231665711 >> 172.21.15.125:0/1231665711 conn(0x7ffb6806bfe0 msgr2=0x7ffb6806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:50.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6f846640 1 -- 172.21.15.125:0/1231665711 shutdown_connections 2023-10-30T12:52:50.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6f846640 1 -- 172.21.15.125:0/1231665711 wait complete. 2023-10-30T12:52:50.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6f846640 1 Processor -- start 2023-10-30T12:52:50.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6f846640 1 -- start start 2023-10-30T12:52:50.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6f846640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb54003980 0x7ffb6815c210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:50.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6f846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb68150820 0x7ffb681f2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:50.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6f846640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb68143940 0x7ffb68143df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:50.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6f846640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb680b4bc0 con 0x7ffb54003980 2023-10-30T12:52:50.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6f846640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681f24e0 con 0x7ffb68143940 2023-10-30T12:52:50.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6f846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb681f21b0 con 0x7ffb68150820 2023-10-30T12:52:50.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6d5e2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb54003980 0x7ffb6815c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:50.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6d5e2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb54003980 0x7ffb6815c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49148/0 (socket says 172.21.15.125:49148) 2023-10-30T12:52:50.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6d5e2640 1 -- 172.21.15.125:0/1730923115 learned_addr learned my addr 172.21.15.125:0/1730923115 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:50.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6dde3640 1 --2- 172.21.15.125:0/1730923115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb68143940 0x7ffb68143df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:50.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6cde1640 1 --2- 172.21.15.125:0/1730923115 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb68150820 0x7ffb681f2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:50.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6d5e2640 1 -- 172.21.15.125:0/1730923115 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb68150820 msgr2=0x7ffb681f2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:50.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6d5e2640 1 --2- 172.21.15.125:0/1730923115 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb68150820 0x7ffb681f2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:50.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6d5e2640 1 -- 172.21.15.125:0/1730923115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb68143940 msgr2=0x7ffb68143df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:50.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6d5e2640 1 --2- 172.21.15.125:0/1730923115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb68143940 0x7ffb68143df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:50.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.813+0000 7ffb6d5e2640 1 -- 172.21.15.125:0/1730923115 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb680b4230 con 0x7ffb54003980 2023-10-30T12:52:50.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.817+0000 7ffb6d5e2640 1 --2- 172.21.15.125:0/1730923115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb54003980 0x7ffb6815c210 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7ffb640ae040 tx=0x7ffb54005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:50.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.817+0000 7ffb5dffb640 1 -- 172.21.15.125:0/1730923115 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb640049f0 con 0x7ffb54003980 2023-10-30T12:52:50.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.817+0000 7ffb5dffb640 1 -- 172.21.15.125:0/1730923115 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb640b6070 con 0x7ffb54003980 2023-10-30T12:52:50.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.817+0000 7ffb6f846640 1 -- 172.21.15.125:0/1730923115 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb681e2de0 con 0x7ffb54003980 2023-10-30T12:52:50.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.817+0000 7ffb5dffb640 1 -- 172.21.15.125:0/1730923115 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64004d50 con 0x7ffb54003980 2023-10-30T12:52:50.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.817+0000 7ffb6f846640 1 -- 172.21.15.125:0/1730923115 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb680b4e10 con 0x7ffb54003980 2023-10-30T12:52:50.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.817+0000 7ffb6f846640 1 -- 172.21.15.125:0/1730923115 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6804e170 con 0x7ffb54003980 2023-10-30T12:52:50.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.817+0000 7ffb5dffb640 1 -- 172.21.15.125:0/1730923115 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffb640070d0 con 0x7ffb54003980 2023-10-30T12:52:50.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.817+0000 7ffb5dffb640 1 --2- 172.21.15.125:0/1730923115 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffb4c0664d0 0x7ffb4c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:50.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.817+0000 7ffb5dffb640 1 -- 172.21.15.125:0/1730923115 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7ffb640b8070 con 0x7ffb54003980 2023-10-30T12:52:50.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.821+0000 7ffb6cde1640 1 --2- 172.21.15.125:0/1730923115 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffb4c0664d0 0x7ffb4c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:50.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.821+0000 7ffb6cde1640 1 --2- 172.21.15.125:0/1730923115 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffb4c0664d0 0x7ffb4c068990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ffb681e5300 tx=0x7ffb58008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:50.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:50.821+0000 7ffb5dffb640 1 -- 172.21.15.125:0/1730923115 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffb640ee710 con 0x7ffb54003980 2023-10-30T12:52:51.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:51.001+0000 7ffb6f846640 1 -- 172.21.15.125:0/1730923115 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb680ca9b0 con 0x7ffb54003980 2023-10-30T12:52:51.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:51.005+0000 7ffb5dffb640 1 -- 172.21.15.125:0/1730923115 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5724 (secure 0 0 0) 0x7ffb640c2090 con 0x7ffb54003980 2023-10-30T12:52:51.006 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:51.006 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:46.367136+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:51.006 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 120 2023-10-30T12:52:51.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:51.005+0000 7ffb6f846640 1 -- 172.21.15.125:0/1730923115 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffb4c0664d0 msgr2=0x7ffb4c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:51.005+0000 7ffb6f846640 1 --2- 172.21.15.125:0/1730923115 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffb4c0664d0 0x7ffb4c068990 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7ffb681e5300 tx=0x7ffb58008040 comp rx=0 tx=0).stop 2023-10-30T12:52:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:51.005+0000 7ffb6f846640 1 -- 172.21.15.125:0/1730923115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb54003980 msgr2=0x7ffb6815c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:51.005+0000 7ffb6f846640 1 --2- 172.21.15.125:0/1730923115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb54003980 0x7ffb6815c210 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7ffb640ae040 tx=0x7ffb54005800 comp rx=0 tx=0).stop 2023-10-30T12:52:51.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:51.005+0000 7ffb6f846640 1 -- 172.21.15.125:0/1730923115 shutdown_connections 2023-10-30T12:52:51.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:51.005+0000 7ffb6f846640 1 --2- 172.21.15.125:0/1730923115 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffb4c0664d0 0x7ffb4c068990 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:51.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:51.005+0000 7ffb6f846640 1 --2- 172.21.15.125:0/1730923115 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb68143940 0x7ffb68143df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:51.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:51.005+0000 7ffb6f846640 1 --2- 172.21.15.125:0/1730923115 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb68150820 0x7ffb681f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:51.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:51.005+0000 7ffb6f846640 1 --2- 172.21.15.125:0/1730923115 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb54003980 0x7ffb6815c210 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:51.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:51.005+0000 7ffb6f846640 1 -- 172.21.15.125:0/1730923115 >> 172.21.15.125:0/1730923115 conn(0x7ffb6806bfe0 msgr2=0x7ffb681510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:51.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:51.005+0000 7ffb6f846640 1 -- 172.21.15.125:0/1730923115 shutdown_connections 2023-10-30T12:52:51.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:51.005+0000 7ffb6f846640 1 -- 172.21.15.125:0/1730923115 wait complete. 2023-10-30T12:52:51.026 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:52:53.028 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:53.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.169+0000 7f0f961c5640 1 Processor -- start 2023-10-30T12:52:53.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.169+0000 7f0f961c5640 1 -- start start 2023-10-30T12:52:53.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.169+0000 7f0f961c5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0f900c05d0 0x7f0f9013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:53.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.169+0000 7f0f961c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f900c6f60 0x7f0f9013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:53.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f961c5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f900c0a50 con 0x7f0f900c6b80 2023-10-30T12:52:53.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f961c5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f9006f6e0 con 0x7f0f900c05d0 2023-10-30T12:52:53.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f961c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9006f9c0 con 0x7f0f900c6f60 2023-10-30T12:52:53.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f8effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0f900c05d0 0x7f0f9013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:53.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f8effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0f900c05d0 0x7f0f9013ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49160/0 (socket says 172.21.15.125:49160) 2023-10-30T12:52:53.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f8effd640 1 -- 172.21.15.125:0/3133517159 learned_addr learned my addr 172.21.15.125:0/3133517159 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:53.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f8ffff640 1 --2- 172.21.15.125:0/3133517159 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f900c6f60 0x7f0f9013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:53.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f8effd640 1 -- 172.21.15.125:0/3133517159 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f900c6f60 msgr2=0x7f0f9013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:53.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f8effd640 1 --2- 172.21.15.125:0/3133517159 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f900c6f60 0x7f0f9013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:53.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f8effd640 1 -- 172.21.15.125:0/3133517159 >> v1:172.21.15.136:6789/0 conn(0x7f0f900c6b80 legacy=0x7f0f900bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:53.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f8effd640 1 -- 172.21.15.125:0/3133517159 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9006fca0 con 0x7f0f900c05d0 2023-10-30T12:52:53.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f8effd640 1 --2- 172.21.15.125:0/3133517159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f900c05d0 0x7f0f9013ed90 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f0f84009870 tx=0x7f0f8401e750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be91f77ec7da5861 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:53.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f8e7fc640 1 -- 172.21.15.125:0/3133517159 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f840a8630 con 0x7f0f900c05d0 2023-10-30T12:52:53.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f8e7fc640 1 -- 172.21.15.125:0/3133517159 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f840a8790 con 0x7f0f900c05d0 2023-10-30T12:52:53.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.173+0000 7f0f8e7fc640 1 -- 172.21.15.125:0/3133517159 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f840a8b20 con 0x7f0f900c05d0 2023-10-30T12:52:53.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 -- 172.21.15.125:0/3133517159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f900c05d0 msgr2=0x7f0f9013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:53.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 --2- 172.21.15.125:0/3133517159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f900c05d0 0x7f0f9013ed90 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f0f84009870 tx=0x7f0f8401e750 comp rx=0 tx=0).stop 2023-10-30T12:52:53.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 -- 172.21.15.125:0/3133517159 shutdown_connections 2023-10-30T12:52:53.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 --2- 172.21.15.125:0/3133517159 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f900c6f60 0x7f0f9013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:53.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 --2- 172.21.15.125:0/3133517159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f900c05d0 0x7f0f9013ed90 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:53.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 -- 172.21.15.125:0/3133517159 >> 172.21.15.125:0/3133517159 conn(0x7f0f900c7910 msgr2=0x7f0f900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:53.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 -- 172.21.15.125:0/3133517159 shutdown_connections 2023-10-30T12:52:53.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 -- 172.21.15.125:0/3133517159 wait complete. 2023-10-30T12:52:53.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 Processor -- start 2023-10-30T12:52:53.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 -- start start 2023-10-30T12:52:53.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f900c05d0 0x7f0f9006b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:53.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f900c6b80 0x7f0f9012f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:53.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f900c6f60 0x7f0f9012e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:53.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9014d010 con 0x7f0f900c6f60 2023-10-30T12:52:53.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f90135140 con 0x7f0f900c05d0 2023-10-30T12:52:53.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f961c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f90139750 con 0x7f0f900c6b80 2023-10-30T12:52:53.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f8ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f900c6f60 0x7f0f9012e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:53.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f8ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f900c6f60 0x7f0f9012e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49176/0 (socket says 172.21.15.125:49176) 2023-10-30T12:52:53.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f8ffff640 1 -- 172.21.15.125:0/3092752286 learned_addr learned my addr 172.21.15.125:0/3092752286 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:53.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.177+0000 7f0f8f7fe640 1 --2- 172.21.15.125:0/3092752286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f900c05d0 0x7f0f9006b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:53.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f8effd640 1 --2- 172.21.15.125:0/3092752286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f900c6b80 0x7f0f9012f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:53.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f8ffff640 1 -- 172.21.15.125:0/3092752286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f900c6b80 msgr2=0x7f0f9012f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:53.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f8ffff640 1 --2- 172.21.15.125:0/3092752286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f900c6b80 0x7f0f9012f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:53.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f8ffff640 1 -- 172.21.15.125:0/3092752286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f900c05d0 msgr2=0x7f0f9006b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:53.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f8ffff640 1 --2- 172.21.15.125:0/3092752286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f900c05d0 0x7f0f9006b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:53.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f8ffff640 1 -- 172.21.15.125:0/3092752286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f840094d0 con 0x7f0f900c6f60 2023-10-30T12:52:53.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f8ffff640 1 --2- 172.21.15.125:0/3092752286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f900c6f60 0x7f0f9012e510 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800efc0 tx=0x7f0f8800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:53.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f77fff640 1 -- 172.21.15.125:0/3092752286 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88009ba0 con 0x7f0f900c6f60 2023-10-30T12:52:53.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f77fff640 1 -- 172.21.15.125:0/3092752286 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8800f040 con 0x7f0f900c6f60 2023-10-30T12:52:53.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f77fff640 1 -- 172.21.15.125:0/3092752286 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f880146c0 con 0x7f0f900c6f60 2023-10-30T12:52:53.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f961c5640 1 -- 172.21.15.125:0/3092752286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9014d4a0 con 0x7f0f900c6f60 2023-10-30T12:52:53.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f961c5640 1 -- 172.21.15.125:0/3092752286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f90139470 con 0x7f0f900c6f60 2023-10-30T12:52:53.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f961c5640 1 -- 172.21.15.125:0/3092752286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9014d180 con 0x7f0f900c6f60 2023-10-30T12:52:53.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f77fff640 1 -- 172.21.15.125:0/3092752286 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0f88014820 con 0x7f0f900c6f60 2023-10-30T12:52:53.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f77fff640 1 --2- 172.21.15.125:0/3092752286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0f7c0664d0 0x7f0f7c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:53.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.181+0000 7f0f77fff640 1 -- 172.21.15.125:0/3092752286 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7f0f88084c40 con 0x7f0f900c6f60 2023-10-30T12:52:53.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.185+0000 7f0f8f7fe640 1 --2- 172.21.15.125:0/3092752286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0f7c0664d0 0x7f0f7c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:53.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.185+0000 7f0f8f7fe640 1 --2- 172.21.15.125:0/3092752286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0f7c0664d0 0x7f0f7c068990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0f80007fd0 tx=0x7f0f80009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:53.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.185+0000 7f0f77fff640 1 -- 172.21.15.125:0/3092752286 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0f88051d30 con 0x7f0f900c6f60 2023-10-30T12:52:53.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.365+0000 7f0f961c5640 1 -- 172.21.15.125:0/3092752286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f9004e170 con 0x7f0f900c6f60 2023-10-30T12:52:53.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.369+0000 7f0f77fff640 1 -- 172.21.15.125:0/3092752286 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f0f88018090 con 0x7f0f900c6f60 2023-10-30T12:52:53.370 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:53.370 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:46.367136+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:53.371 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 120 2023-10-30T12:52:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.369+0000 7f0f961c5640 1 -- 172.21.15.125:0/3092752286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0f7c0664d0 msgr2=0x7f0f7c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.369+0000 7f0f961c5640 1 --2- 172.21.15.125:0/3092752286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0f7c0664d0 0x7f0f7c068990 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f0f80007fd0 tx=0x7f0f80009040 comp rx=0 tx=0).stop 2023-10-30T12:52:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.369+0000 7f0f961c5640 1 -- 172.21.15.125:0/3092752286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f900c6f60 msgr2=0x7f0f9012e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.369+0000 7f0f961c5640 1 --2- 172.21.15.125:0/3092752286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f900c6f60 0x7f0f9012e510 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800efc0 tx=0x7f0f8800c490 comp rx=0 tx=0).stop 2023-10-30T12:52:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.369+0000 7f0f961c5640 1 -- 172.21.15.125:0/3092752286 shutdown_connections 2023-10-30T12:52:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.369+0000 7f0f961c5640 1 --2- 172.21.15.125:0/3092752286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0f7c0664d0 0x7f0f7c068990 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.369+0000 7f0f961c5640 1 --2- 172.21.15.125:0/3092752286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0f900c6f60 0x7f0f9012e510 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.369+0000 7f0f961c5640 1 --2- 172.21.15.125:0/3092752286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0f900c6b80 0x7f0f9012f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.369+0000 7f0f961c5640 1 --2- 172.21.15.125:0/3092752286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0f900c05d0 0x7f0f9006b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:53.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.369+0000 7f0f961c5640 1 -- 172.21.15.125:0/3092752286 >> 172.21.15.125:0/3092752286 conn(0x7f0f900c7910 msgr2=0x7f0f900c1970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:53.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.369+0000 7f0f961c5640 1 -- 172.21.15.125:0/3092752286 shutdown_connections 2023-10-30T12:52:53.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:53.369+0000 7f0f961c5640 1 -- 172.21.15.125:0/3092752286 wait complete. 2023-10-30T12:52:53.391 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:52:55.391 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:55.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.537+0000 7ff88c1f5640 1 Processor -- start 2023-10-30T12:52:55.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.537+0000 7ff88c1f5640 1 -- start start 2023-10-30T12:52:55.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.541+0000 7ff88c1f5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff884157080 0x7ff884071b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:55.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.541+0000 7ff88c1f5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff8840c7990 0x7ff884072090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:55.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.541+0000 7ff88c1f5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8840c8da0 con 0x7ff88406fdb0 2023-10-30T12:52:55.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.541+0000 7ff88c1f5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff884157460 con 0x7ff884157080 2023-10-30T12:52:55.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.541+0000 7ff88c1f5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff88406cb60 con 0x7ff8840c7990 2023-10-30T12:52:55.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.541+0000 7ff889f91640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff884157080 0x7ff884071b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:55.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.541+0000 7ff889f91640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff884157080 0x7ff884071b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49184/0 (socket says 172.21.15.125:49184) 2023-10-30T12:52:55.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.541+0000 7ff889f91640 1 -- 172.21.15.125:0/1655109118 learned_addr learned my addr 172.21.15.125:0/1655109118 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:55.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.541+0000 7ff889790640 1 --2- 172.21.15.125:0/1655109118 >> v2:172.21.15.136:3300/0 conn(0x7ff8840c7990 0x7ff884072090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:55.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.541+0000 7ff889f91640 1 -- 172.21.15.125:0/1655109118 >> v2:172.21.15.136:3300/0 conn(0x7ff8840c7990 msgr2=0x7ff884072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:55.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.541+0000 7ff889f91640 1 --2- 172.21.15.125:0/1655109118 >> v2:172.21.15.136:3300/0 conn(0x7ff8840c7990 0x7ff884072090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:55.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.541+0000 7ff889f91640 1 -- 172.21.15.125:0/1655109118 >> v1:172.21.15.136:6789/0 conn(0x7ff88406fdb0 legacy=0x7ff8840c7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:55.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.541+0000 7ff889f91640 1 -- 172.21.15.125:0/1655109118 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff88406ce20 con 0x7ff884157080 2023-10-30T12:52:55.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.541+0000 7ff889f91640 1 --2- 172.21.15.125:0/1655109118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff884157080 0x7ff884071b50 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7ff884072f50 tx=0x7ff88001c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea7efcdcc4429d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:55.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff888f8f640 1 -- 172.21.15.125:0/1655109118 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8800ab600 con 0x7ff884157080 2023-10-30T12:52:55.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff888f8f640 1 -- 172.21.15.125:0/1655109118 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8800a7050 con 0x7ff884157080 2023-10-30T12:52:55.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff888f8f640 1 -- 172.21.15.125:0/1655109118 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8800ab980 con 0x7ff884157080 2023-10-30T12:52:55.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1655109118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff884157080 msgr2=0x7ff884071b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:55.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 --2- 172.21.15.125:0/1655109118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff884157080 0x7ff884071b50 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7ff884072f50 tx=0x7ff88001c3e0 comp rx=0 tx=0).stop 2023-10-30T12:52:55.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1655109118 shutdown_connections 2023-10-30T12:52:55.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 --2- 172.21.15.125:0/1655109118 >> v2:172.21.15.136:3300/0 conn(0x7ff8840c7990 0x7ff884072090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:55.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 --2- 172.21.15.125:0/1655109118 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff884157080 0x7ff884071b50 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:55.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1655109118 >> 172.21.15.125:0/1655109118 conn(0x7ff884069e90 msgr2=0x7ff88406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:55.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1655109118 shutdown_connections 2023-10-30T12:52:55.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1655109118 wait complete. 2023-10-30T12:52:55.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 Processor -- start 2023-10-30T12:52:55.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 -- start start 2023-10-30T12:52:55.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff88406fdb0 0x7ff884148040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:55.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8840c7990 0x7ff8841eb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:55.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff884157080 0x7ff8840cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:55.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8841f6600 con 0x7ff884157080 2023-10-30T12:52:55.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff884147bb0 con 0x7ff88406fdb0 2023-10-30T12:52:55.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88c1f5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8840cd280 con 0x7ff8840c7990 2023-10-30T12:52:55.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.545+0000 7ff88a792640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff884157080 0x7ff8840cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:55.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff88a792640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff884157080 0x7ff8840cb810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49198/0 (socket says 172.21.15.125:49198) 2023-10-30T12:52:55.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff88a792640 1 -- 172.21.15.125:0/1853528851 learned_addr learned my addr 172.21.15.125:0/1853528851 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:55.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff889790640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8840c7990 0x7ff8841eb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:55.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff889f91640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff88406fdb0 0x7ff884148040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:55.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff88a792640 1 -- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8840c7990 msgr2=0x7ff8841eb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:55.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff88a792640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8840c7990 0x7ff8841eb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:55.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff88a792640 1 -- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff88406fdb0 msgr2=0x7ff884148040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:55.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff88a792640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff88406fdb0 0x7ff884148040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:55.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff88a792640 1 -- 172.21.15.125:0/1853528851 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8800094d0 con 0x7ff884157080 2023-10-30T12:52:55.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff889790640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8840c7990 0x7ff8841eb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:52:55.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff88a792640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff884157080 0x7ff8840cb810 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00c770 tx=0x7ff87c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:55.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff87a7fc640 1 -- 172.21.15.125:0/1853528851 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c003b80 con 0x7ff884157080 2023-10-30T12:52:55.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff87a7fc640 1 -- 172.21.15.125:0/1853528851 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87c003ce0 con 0x7ff884157080 2023-10-30T12:52:55.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1853528851 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8841f2b50 con 0x7ff884157080 2023-10-30T12:52:55.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff87a7fc640 1 -- 172.21.15.125:0/1853528851 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c00aa90 con 0x7ff884157080 2023-10-30T12:52:55.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1853528851 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff884147560 con 0x7ff884157080 2023-10-30T12:52:55.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1853528851 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff884147d20 con 0x7ff884157080 2023-10-30T12:52:55.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff87a7fc640 1 -- 172.21.15.125:0/1853528851 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff87c00abf0 con 0x7ff884157080 2023-10-30T12:52:55.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff87a7fc640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff8680664a0 0x7ff868068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:55.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.549+0000 7ff87a7fc640 1 -- 172.21.15.125:0/1853528851 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7ff87c084180 con 0x7ff884157080 2023-10-30T12:52:55.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.553+0000 7ff889f91640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff8680664a0 0x7ff868068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:55.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.553+0000 7ff889f91640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff8680664a0 0x7ff868068960 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff8800ad000 tx=0x7ff880008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:55.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.553+0000 7ff87a7fc640 1 -- 172.21.15.125:0/1853528851 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff87c0510a0 con 0x7ff884157080 2023-10-30T12:52:55.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.737+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1853528851 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff88404e170 con 0x7ff884157080 2023-10-30T12:52:55.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.737+0000 7ff87a7fc640 1 -- 172.21.15.125:0/1853528851 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+5730 (secure 0 0 0) 0x7ff87c0546c0 con 0x7ff884157080 2023-10-30T12:52:55.739 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:55.740 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:52.439960+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:55.741 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 121 2023-10-30T12:52:55.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.737+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff8680664a0 msgr2=0x7ff868068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:55.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.737+0000 7ff88c1f5640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff8680664a0 0x7ff868068960 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff8800ad000 tx=0x7ff880008040 comp rx=0 tx=0).stop 2023-10-30T12:52:55.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.737+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1853528851 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff884157080 msgr2=0x7ff8840cb810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:55.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.737+0000 7ff88c1f5640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff884157080 0x7ff8840cb810 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00c770 tx=0x7ff87c00cc30 comp rx=0 tx=0).stop 2023-10-30T12:52:55.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.737+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1853528851 shutdown_connections 2023-10-30T12:52:55.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.737+0000 7ff88c1f5640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff8680664a0 0x7ff868068960 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:55.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.737+0000 7ff88c1f5640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff884157080 0x7ff8840cb810 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:55.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.737+0000 7ff88c1f5640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8840c7990 0x7ff8841eb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:55.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.737+0000 7ff88c1f5640 1 --2- 172.21.15.125:0/1853528851 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff88406fdb0 0x7ff884148040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:55.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.737+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1853528851 >> 172.21.15.125:0/1853528851 conn(0x7ff884069e90 msgr2=0x7ff8840c7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:55.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.737+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1853528851 shutdown_connections 2023-10-30T12:52:55.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:55.737+0000 7ff88c1f5640 1 -- 172.21.15.125:0/1853528851 wait complete. 2023-10-30T12:52:55.760 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 121, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 121, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6954": { "addr": "172.21.15.125:6836/1163622496", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1163622496, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1163622496, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6954, "incarnation": 118, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:clientreplay", "state_seq": 17 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:52:52.439960+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 6954 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:52:57.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:52:57.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.905+0000 7faf9dd0e640 1 Processor -- start 2023-10-30T12:52:57.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.905+0000 7faf9dd0e640 1 -- start start 2023-10-30T12:52:57.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.905+0000 7faf9dd0e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7faf980c6540 0x7faf9815e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:57.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.905+0000 7faf9dd0e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf980c5810 con 0x7faf981580e0 2023-10-30T12:52:57.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf9dd0e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf980c69b0 con 0x7faf98070740 2023-10-30T12:52:57.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf9dd0e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf9806acb0 con 0x7faf980c6540 2023-10-30T12:52:57.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf977fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7faf981580e0 0x7faf98069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59734/0 (socket says 172.21.15.125:59734) 2023-10-30T12:52:57.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf977fe640 1 -- 172.21.15.125:0/2212629067 learned_addr learned my addr 172.21.15.125:0/2212629067 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:52:57.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf97fff640 1 --2- 172.21.15.125:0/2212629067 >> v2:172.21.15.136:3300/0 conn(0x7faf980c6540 0x7faf9815e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:57.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 -- 172.21.15.125:0/2212629067 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347678370 0 0) 0x7faf980c5810 con 0x7faf981580e0 2023-10-30T12:52:57.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 -- 172.21.15.125:0/2212629067 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf84003530 con 0x7faf981580e0 2023-10-30T12:52:57.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 -- 172.21.15.125:0/2212629067 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1644828113 0 0) 0x7faf84003530 con 0x7faf981580e0 2023-10-30T12:52:57.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 -- 172.21.15.125:0/2212629067 >> v2:172.21.15.136:3300/0 conn(0x7faf980c6540 msgr2=0x7faf9815e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:57.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 --2- 172.21.15.125:0/2212629067 >> v2:172.21.15.136:3300/0 conn(0x7faf980c6540 0x7faf9815e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:57.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 -- 172.21.15.125:0/2212629067 >> v1:172.21.15.136:6789/0 conn(0x7faf98070740 legacy=0x7faf9806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:57.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 -- 172.21.15.125:0/2212629067 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf980c6bb0 con 0x7faf981580e0 2023-10-30T12:52:57.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 -- 172.21.15.125:0/2212629067 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7faf88001680 con 0x7faf981580e0 2023-10-30T12:52:57.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 -- 172.21.15.125:0/2212629067 >> v1:172.21.15.125:6789/0 conn(0x7faf981580e0 legacy=0x7faf98069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:57.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 --2- 172.21.15.125:0/2212629067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf84003980 0x7faf84003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:57.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 --2- 172.21.15.125:0/2212629067 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf840047d0 0x7faf84004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:57.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 --2- 172.21.15.125:0/2212629067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf84005190 0x7faf8400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:57.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 -- 172.21.15.125:0/2212629067 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8400c000 con 0x7faf84003980 2023-10-30T12:52:57.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 -- 172.21.15.125:0/2212629067 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8400c170 con 0x7faf84005190 2023-10-30T12:52:57.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf96ffd640 1 -- 172.21.15.125:0/2212629067 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8400c330 con 0x7faf840047d0 2023-10-30T12:52:57.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf977fe640 1 --2- 172.21.15.125:0/2212629067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf84003980 0x7faf84003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:57.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf97fff640 1 --2- 172.21.15.125:0/2212629067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf84005190 0x7faf8400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:57.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf8ffff640 1 --2- 172.21.15.125:0/2212629067 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf840047d0 0x7faf84004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:57.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf977fe640 1 -- 172.21.15.125:0/2212629067 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf840047d0 msgr2=0x7faf84004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:57.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf977fe640 1 --2- 172.21.15.125:0/2212629067 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf840047d0 0x7faf84004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:57.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf977fe640 1 -- 172.21.15.125:0/2212629067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf84005190 msgr2=0x7faf8400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:52:57.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf977fe640 1 --2- 172.21.15.125:0/2212629067 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf84005190 0x7faf8400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:57.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf977fe640 1 -- 172.21.15.125:0/2212629067 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf84003530 con 0x7faf84003980 2023-10-30T12:52:57.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf97fff640 1 -- 172.21.15.125:0/2212629067 reap_dead start 2023-10-30T12:52:57.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.909+0000 7faf8ffff640 1 --2- 172.21.15.125:0/2212629067 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf840047d0 0x7faf84004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:52:57.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.913+0000 7faf977fe640 1 --2- 172.21.15.125:0/2212629067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf84003980 0x7faf84003da0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7faf8800c240 tx=0x7faf8801cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f35c93d4b98d89 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:57.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.913+0000 7faf96ffd640 1 -- 172.21.15.125:0/2212629067 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf88005ae0 con 0x7faf84003980 2023-10-30T12:52:57.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.913+0000 7faf96ffd640 1 -- 172.21.15.125:0/2212629067 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf88005c60 con 0x7faf84003980 2023-10-30T12:52:57.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.913+0000 7faf96ffd640 1 -- 172.21.15.125:0/2212629067 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf880a5790 con 0x7faf84003980 2023-10-30T12:52:57.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.913+0000 7faf9dd0e640 1 -- 172.21.15.125:0/2212629067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf84003980 msgr2=0x7faf84003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:57.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.913+0000 7faf9dd0e640 1 --2- 172.21.15.125:0/2212629067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf84003980 0x7faf84003da0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7faf8800c240 tx=0x7faf8801cbc0 comp rx=0 tx=0).stop 2023-10-30T12:52:57.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.913+0000 7faf9dd0e640 1 -- 172.21.15.125:0/2212629067 shutdown_connections 2023-10-30T12:52:57.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.913+0000 7faf9dd0e640 1 --2- 172.21.15.125:0/2212629067 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf84003980 0x7faf84003da0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:57.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.913+0000 7faf9dd0e640 1 -- 172.21.15.125:0/2212629067 >> 172.21.15.125:0/2212629067 conn(0x7faf980c7910 msgr2=0x7faf980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:57.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.913+0000 7faf9dd0e640 1 -- 172.21.15.125:0/2212629067 shutdown_connections 2023-10-30T12:52:57.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.913+0000 7faf9dd0e640 1 -- 172.21.15.125:0/2212629067 wait complete. 2023-10-30T12:52:57.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.913+0000 7faf9dd0e640 1 Processor -- start 2023-10-30T12:52:57.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.913+0000 7faf9dd0e640 1 -- start start 2023-10-30T12:52:57.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf9dd0e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf84003980 0x7faf9814e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:57.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf9dd0e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf980c5440 0x7faf9815f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:57.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf9dd0e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf981619a0 0x7faf981608a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:57.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf9dd0e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9815fce0 con 0x7faf981619a0 2023-10-30T12:52:57.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf9dd0e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf9815ffa0 con 0x7faf980c5440 2023-10-30T12:52:57.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf9dd0e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf98160280 con 0x7faf84003980 2023-10-30T12:52:57.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf97fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf981619a0 0x7faf981608a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:57.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf97fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf981619a0 0x7faf981608a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49202/0 (socket says 172.21.15.125:49202) 2023-10-30T12:52:57.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf97fff640 1 -- 172.21.15.125:0/3344534248 learned_addr learned my addr 172.21.15.125:0/3344534248 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:52:57.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf977fe640 1 --2- 172.21.15.125:0/3344534248 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf84003980 0x7faf9814e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:57.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf8ffff640 1 --2- 172.21.15.125:0/3344534248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf980c5440 0x7faf9815f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:57.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf97fff640 1 -- 172.21.15.125:0/3344534248 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf84003980 msgr2=0x7faf9814e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:57.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf97fff640 1 --2- 172.21.15.125:0/3344534248 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf84003980 0x7faf9814e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:57.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf97fff640 1 -- 172.21.15.125:0/3344534248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf980c5440 msgr2=0x7faf9815f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:57.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf97fff640 1 --2- 172.21.15.125:0/3344534248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf980c5440 0x7faf9815f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:57.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf97fff640 1 -- 172.21.15.125:0/3344534248 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf980c23b0 con 0x7faf981619a0 2023-10-30T12:52:57.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf97fff640 1 --2- 172.21.15.125:0/3344534248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf981619a0 0x7faf981608a0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7faf9806f4f0 tx=0x7faf9000f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:57.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf94ff9640 1 -- 172.21.15.125:0/3344534248 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9000e040 con 0x7faf981619a0 2023-10-30T12:52:57.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf94ff9640 1 -- 172.21.15.125:0/3344534248 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9000f040 con 0x7faf981619a0 2023-10-30T12:52:57.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf94ff9640 1 -- 172.21.15.125:0/3344534248 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf90015430 con 0x7faf981619a0 2023-10-30T12:52:57.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf9dd0e640 1 -- 172.21.15.125:0/3344534248 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf980c2690 con 0x7faf981619a0 2023-10-30T12:52:57.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf9dd0e640 1 -- 172.21.15.125:0/3344534248 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf98160de0 con 0x7faf981619a0 2023-10-30T12:52:57.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf9dd0e640 1 -- 172.21.15.125:0/3344534248 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf980c21d0 con 0x7faf981619a0 2023-10-30T12:52:57.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.917+0000 7faf94ff9640 1 -- 172.21.15.125:0/3344534248 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faf90015590 con 0x7faf981619a0 2023-10-30T12:52:57.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.921+0000 7faf94ff9640 1 --2- 172.21.15.125:0/3344534248 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf780664d0 0x7faf78068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:52:57.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.921+0000 7faf94ff9640 1 -- 172.21.15.125:0/3344534248 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7faf90084950 con 0x7faf981619a0 2023-10-30T12:52:57.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.921+0000 7faf977fe640 1 --2- 172.21.15.125:0/3344534248 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf780664d0 0x7faf78068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:52:57.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.921+0000 7faf977fe640 1 --2- 172.21.15.125:0/3344534248 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf780664d0 0x7faf78068990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7faf880ab000 tx=0x7faf880a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:52:57.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:57.925+0000 7faf94ff9640 1 -- 172.21.15.125:0/3344534248 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faf90051900 con 0x7faf981619a0 2023-10-30T12:52:58.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:58.105+0000 7faf9dd0e640 1 -- 172.21.15.125:0/3344534248 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf9804e170 con 0x7faf981619a0 2023-10-30T12:52:58.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:58.105+0000 7faf94ff9640 1 -- 172.21.15.125:0/3344534248 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+5730 (secure 0 0 0) 0x7faf90054f20 con 0x7faf981619a0 2023-10-30T12:52:58.107 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:52:58.107 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:52.439960+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:clientreplay","state_seq":17,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:52:58.108 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 121 2023-10-30T12:52:58.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:58.105+0000 7faf9dd0e640 1 -- 172.21.15.125:0/3344534248 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf780664d0 msgr2=0x7faf78068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:58.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:58.105+0000 7faf9dd0e640 1 --2- 172.21.15.125:0/3344534248 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf780664d0 0x7faf78068990 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7faf880ab000 tx=0x7faf880a7070 comp rx=0 tx=0).stop 2023-10-30T12:52:58.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:58.105+0000 7faf9dd0e640 1 -- 172.21.15.125:0/3344534248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf981619a0 msgr2=0x7faf981608a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:52:58.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:58.105+0000 7faf9dd0e640 1 --2- 172.21.15.125:0/3344534248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf981619a0 0x7faf981608a0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7faf9806f4f0 tx=0x7faf9000f1b0 comp rx=0 tx=0).stop 2023-10-30T12:52:58.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:58.105+0000 7faf9dd0e640 1 -- 172.21.15.125:0/3344534248 shutdown_connections 2023-10-30T12:52:58.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:58.105+0000 7faf9dd0e640 1 --2- 172.21.15.125:0/3344534248 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf780664d0 0x7faf78068990 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:58.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:58.105+0000 7faf9dd0e640 1 --2- 172.21.15.125:0/3344534248 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf981619a0 0x7faf981608a0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:58.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:58.105+0000 7faf9dd0e640 1 --2- 172.21.15.125:0/3344534248 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf980c5440 0x7faf9815f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:58.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:58.105+0000 7faf9dd0e640 1 --2- 172.21.15.125:0/3344534248 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf84003980 0x7faf9814e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:52:58.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:58.105+0000 7faf9dd0e640 1 -- 172.21.15.125:0/3344534248 >> 172.21.15.125:0/3344534248 conn(0x7faf980c7910 msgr2=0x7faf98071dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:52:58.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:58.105+0000 7faf9dd0e640 1 -- 172.21.15.125:0/3344534248 shutdown_connections 2023-10-30T12:52:58.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:52:58.105+0000 7faf9dd0e640 1 -- 172.21.15.125:0/3344534248 wait complete. 2023-10-30T12:52:58.128 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:53:00.129 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:53:00.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.265+0000 7f29a9dc4640 1 Processor -- start 2023-10-30T12:53:00.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.265+0000 7f29a9dc4640 1 -- start start 2023-10-30T12:53:00.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.265+0000 7f29a9dc4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f29a40c05d0 0x7f29a413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.269+0000 7f29a9dc4640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a40c0a50 con 0x7f29a40c6b80 2023-10-30T12:53:00.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.269+0000 7f29a9dc4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a406f6e0 con 0x7f29a40c6f60 2023-10-30T12:53:00.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.269+0000 7f29a9dc4640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a406f9c0 con 0x7f29a40c05d0 2023-10-30T12:53:00.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.269+0000 7f29a37fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f29a40c6b80 0x7f29a40bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59750/0 (socket says 172.21.15.125:59750) 2023-10-30T12:53:00.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.269+0000 7f29a37fe640 1 -- 172.21.15.125:0/2518775827 learned_addr learned my addr 172.21.15.125:0/2518775827 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:53:00.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.269+0000 7f29a2ffd640 1 --2- 172.21.15.125:0/2518775827 >> v2:172.21.15.125:3300/0 conn(0x7f29a40c05d0 0x7f29a413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.269+0000 7f29a2ffd640 1 -- 172.21.15.125:0/2518775827 >> v1:172.21.15.136:6789/0 conn(0x7f29a40c6f60 legacy=0x7f29a41566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.269+0000 7f29a2ffd640 1 -- 172.21.15.125:0/2518775827 >> v1:172.21.15.125:6789/0 conn(0x7f29a40c6b80 legacy=0x7f29a40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.269+0000 7f29a2ffd640 1 -- 172.21.15.125:0/2518775827 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a406fca0 con 0x7f29a40c05d0 2023-10-30T12:53:00.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.269+0000 7f29a2ffd640 1 --2- 172.21.15.125:0/2518775827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29a40c05d0 0x7f29a413ed90 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f29980097c0 tx=0x7f299801e690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f713bc12885555 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:00.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.269+0000 7f29a27fc640 1 -- 172.21.15.125:0/2518775827 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29980ac610 con 0x7f29a40c05d0 2023-10-30T12:53:00.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.269+0000 7f29a27fc640 1 -- 172.21.15.125:0/2518775827 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29980a8040 con 0x7f29a40c05d0 2023-10-30T12:53:00.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.269+0000 7f29a27fc640 1 -- 172.21.15.125:0/2518775827 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29980ac990 con 0x7f29a40c05d0 2023-10-30T12:53:00.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2518775827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29a40c05d0 msgr2=0x7f29a413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 --2- 172.21.15.125:0/2518775827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29a40c05d0 0x7f29a413ed90 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f29980097c0 tx=0x7f299801e690 comp rx=0 tx=0).stop 2023-10-30T12:53:00.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2518775827 shutdown_connections 2023-10-30T12:53:00.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 --2- 172.21.15.125:0/2518775827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29a40c05d0 0x7f29a413ed90 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2518775827 >> 172.21.15.125:0/2518775827 conn(0x7f29a40c7910 msgr2=0x7f29a40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:00.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2518775827 shutdown_connections 2023-10-30T12:53:00.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2518775827 wait complete. 2023-10-30T12:53:00.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 Processor -- start 2023-10-30T12:53:00.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 -- start start 2023-10-30T12:53:00.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29a40c05d0 0x7f29a413d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29a40c6b80 0x7f29a40c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29a40c6f60 0x7f29a40c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a4156560 con 0x7f29a40c6f60 2023-10-30T12:53:00.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a4154f10 con 0x7f29a40c6b80 2023-10-30T12:53:00.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a9dc4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a4155c00 con 0x7f29a40c05d0 2023-10-30T12:53:00.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29a40c6f60 0x7f29a40c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29a40c6f60 0x7f29a40c9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49214/0 (socket says 172.21.15.125:49214) 2023-10-30T12:53:00.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a3fff640 1 -- 172.21.15.125:0/2068706314 learned_addr learned my addr 172.21.15.125:0/2068706314 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:53:00.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a37fe640 1 --2- 172.21.15.125:0/2068706314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29a40c05d0 0x7f29a413d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.273+0000 7f29a2ffd640 1 --2- 172.21.15.125:0/2068706314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29a40c6b80 0x7f29a40c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f29a3fff640 1 -- 172.21.15.125:0/2068706314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29a40c05d0 msgr2=0x7f29a413d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f29a3fff640 1 --2- 172.21.15.125:0/2068706314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29a40c05d0 0x7f29a413d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f29a3fff640 1 -- 172.21.15.125:0/2068706314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29a40c6b80 msgr2=0x7f29a40c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f29a3fff640 1 --2- 172.21.15.125:0/2068706314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29a40c6b80 0x7f29a40c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f29a3fff640 1 -- 172.21.15.125:0/2068706314 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2998009420 con 0x7f29a40c6f60 2023-10-30T12:53:00.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f29a3fff640 1 --2- 172.21.15.125:0/2068706314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29a40c6f60 0x7f29a40c9250 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f299c00c730 tx=0x7f299c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:00.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f298bfff640 1 -- 172.21.15.125:0/2068706314 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c003b80 con 0x7f29a40c6f60 2023-10-30T12:53:00.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f298bfff640 1 -- 172.21.15.125:0/2068706314 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299c003ce0 con 0x7f29a40c6f60 2023-10-30T12:53:00.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2068706314 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a41389d0 con 0x7f29a40c6f60 2023-10-30T12:53:00.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f298bfff640 1 -- 172.21.15.125:0/2068706314 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c00aa30 con 0x7f29a40c6f60 2023-10-30T12:53:00.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2068706314 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a413e1c0 con 0x7f29a40c6f60 2023-10-30T12:53:00.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2068706314 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a41566a0 con 0x7f29a40c6f60 2023-10-30T12:53:00.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f298bfff640 1 -- 172.21.15.125:0/2068706314 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f299c00ab90 con 0x7f29a40c6f60 2023-10-30T12:53:00.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f298bfff640 1 --2- 172.21.15.125:0/2068706314 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29900664a0 0x7f2990068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:00.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f298bfff640 1 -- 172.21.15.125:0/2068706314 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7f299c084090 con 0x7f29a40c6f60 2023-10-30T12:53:00.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.277+0000 7f29a37fe640 1 --2- 172.21.15.125:0/2068706314 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29900664a0 0x7f2990068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:00.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.281+0000 7f29a37fe640 1 --2- 172.21.15.125:0/2068706314 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29900664a0 0x7f2990068960 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2994005890 tx=0x7f2994009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:00.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.281+0000 7f298bfff640 1 -- 172.21.15.125:0/2068706314 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f299c051060 con 0x7f29a40c6f60 2023-10-30T12:53:00.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.465+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2068706314 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29a404e170 con 0x7f29a40c6f60 2023-10-30T12:53:00.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.465+0000 7f298bfff640 1 -- 172.21.15.125:0/2068706314 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f299c054680 con 0x7f29a40c6f60 2023-10-30T12:53:00.468 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:53:00.468 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:59.039508+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:active","state_seq":19,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:53:00.469 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 122 2023-10-30T12:53:00.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.469+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2068706314 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29900664a0 msgr2=0x7f2990068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.469+0000 7f29a9dc4640 1 --2- 172.21.15.125:0/2068706314 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29900664a0 0x7f2990068960 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2994005890 tx=0x7f2994009040 comp rx=0 tx=0).stop 2023-10-30T12:53:00.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.469+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2068706314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29a40c6f60 msgr2=0x7f29a40c9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:00.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.469+0000 7f29a9dc4640 1 --2- 172.21.15.125:0/2068706314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29a40c6f60 0x7f29a40c9250 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f299c00c730 tx=0x7f299c00cbf0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.469+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2068706314 shutdown_connections 2023-10-30T12:53:00.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.469+0000 7f29a9dc4640 1 --2- 172.21.15.125:0/2068706314 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29900664a0 0x7f2990068960 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.469+0000 7f29a9dc4640 1 --2- 172.21.15.125:0/2068706314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29a40c6f60 0x7f29a40c9250 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.469+0000 7f29a9dc4640 1 --2- 172.21.15.125:0/2068706314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29a40c6b80 0x7f29a40c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.469+0000 7f29a9dc4640 1 --2- 172.21.15.125:0/2068706314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29a40c05d0 0x7f29a413d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:00.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.469+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2068706314 >> 172.21.15.125:0/2068706314 conn(0x7f29a40c7910 msgr2=0x7f29a414cf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:00.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.469+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2068706314 shutdown_connections 2023-10-30T12:53:00.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:00.469+0000 7f29a9dc4640 1 -- 172.21.15.125:0/2068706314 wait complete. 2023-10-30T12:53:00.489 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=6627 2023-10-30T12:53:00.490 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.c 2023-10-30T12:53:09.425 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:53:09.421+0000 7f7bd8019640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:53:09.491 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T12:53:09.491 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T12:53:09.491 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T12:53:09.495 INFO:tasks.ceph.mds.c:Started 2023-10-30T12:53:09.495 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:53:09.640 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2023-10-30T12:53:09.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.701+0000 7f98acbc5640 1 Processor -- start 2023-10-30T12:53:09.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.701+0000 7f98acbc5640 1 -- start start 2023-10-30T12:53:09.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98acbc5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f98a80c6f60 0x7f98a81566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:09.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98acbc5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a80c0a50 con 0x7f98a80c05d0 2023-10-30T12:53:09.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98acbc5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98a806f6e0 con 0x7f98a80c6b80 2023-10-30T12:53:09.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98acbc5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98a806f9c0 con 0x7f98a80c6f60 2023-10-30T12:53:09.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a5d9b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f98a80c05d0 0x7f98a813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53480/0 (socket says 172.21.15.125:53480) 2023-10-30T12:53:09.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a5d9b640 1 -- 172.21.15.125:0/1218441828 learned_addr learned my addr 172.21.15.125:0/1218441828 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:53:09.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a6d9d640 1 --2- 172.21.15.125:0/1218441828 >> v2:172.21.15.136:3300/0 conn(0x7f98a80c6f60 0x7f98a81566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:09.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206540550 0 0) 0x7f98a80c0a50 con 0x7f98a80c05d0 2023-10-30T12:53:09.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9890005740 con 0x7f98a80c05d0 2023-10-30T12:53:09.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249143978 0 0) 0x7f98a806f6e0 con 0x7f98a80c6b80 2023-10-30T12:53:09.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98a80c0a50 con 0x7f98a80c6b80 2023-10-30T12:53:09.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1722609022 0 0) 0x7f9890005740 con 0x7f98a80c05d0 2023-10-30T12:53:09.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 >> v2:172.21.15.136:3300/0 conn(0x7f98a80c6f60 msgr2=0x7f98a81566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:09.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 --2- 172.21.15.125:0/1218441828 >> v2:172.21.15.136:3300/0 conn(0x7f98a80c6f60 0x7f98a81566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:09.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 >> v1:172.21.15.136:6789/0 conn(0x7f98a80c6b80 legacy=0x7f98a80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:09.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a806fca0 con 0x7f98a80c05d0 2023-10-30T12:53:09.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f989c001680 con 0x7f98a80c05d0 2023-10-30T12:53:09.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 >> v1:172.21.15.125:6789/0 conn(0x7f98a80c05d0 legacy=0x7f98a813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:09.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 --2- 172.21.15.125:0/1218441828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9890005d20 0x7f9890006120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:09.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 --2- 172.21.15.125:0/1218441828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98900069e0 0x7f9890006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:09.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 --2- 172.21.15.125:0/1218441828 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98900073a0 0x7f989000dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:09.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a806f9c0 con 0x7f98900069e0 2023-10-30T12:53:09.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989000e220 con 0x7f9890005d20 2023-10-30T12:53:09.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f989000e3e0 con 0x7f98900073a0 2023-10-30T12:53:09.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a5d9b640 1 --2- 172.21.15.125:0/1218441828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98900069e0 0x7f9890006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:09.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a659c640 1 --2- 172.21.15.125:0/1218441828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9890005d20 0x7f9890006120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:09.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a6d9d640 1 --2- 172.21.15.125:0/1218441828 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98900073a0 0x7f989000dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:09.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a5d9b640 1 -- 172.21.15.125:0/1218441828 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98900073a0 msgr2=0x7f989000dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:09.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a5d9b640 1 --2- 172.21.15.125:0/1218441828 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98900073a0 0x7f989000dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:09.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a5d9b640 1 -- 172.21.15.125:0/1218441828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9890005d20 msgr2=0x7f9890006120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:53:09.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a5d9b640 1 --2- 172.21.15.125:0/1218441828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9890005d20 0x7f9890006120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:09.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a5d9b640 1 -- 172.21.15.125:0/1218441828 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9890005740 con 0x7f98900069e0 2023-10-30T12:53:09.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.705+0000 7f98a6d9d640 1 -- 172.21.15.125:0/1218441828 reap_dead start 2023-10-30T12:53:09.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.709+0000 7f98a5d9b640 1 --2- 172.21.15.125:0/1218441828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98900069e0 0x7f9890006e60 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f98a806c9a0 tx=0x7f989c01cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce209d6545fb921e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:09.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.709+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c0058e0 con 0x7f98900069e0 2023-10-30T12:53:09.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.709+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f989c005a60 con 0x7f98900069e0 2023-10-30T12:53:09.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.709+0000 7f98a559a640 1 -- 172.21.15.125:0/1218441828 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989c005e50 con 0x7f98900069e0 2023-10-30T12:53:09.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.709+0000 7f98acbc5640 1 -- 172.21.15.125:0/1218441828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98900069e0 msgr2=0x7f9890006e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:09.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.709+0000 7f98acbc5640 1 --2- 172.21.15.125:0/1218441828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98900069e0 0x7f9890006e60 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f98a806c9a0 tx=0x7f989c01cb40 comp rx=0 tx=0).stop 2023-10-30T12:53:09.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.709+0000 7f98acbc5640 1 -- 172.21.15.125:0/1218441828 shutdown_connections 2023-10-30T12:53:09.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.709+0000 7f98acbc5640 1 --2- 172.21.15.125:0/1218441828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98900069e0 0x7f9890006e60 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:09.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.709+0000 7f98acbc5640 1 -- 172.21.15.125:0/1218441828 >> 172.21.15.125:0/1218441828 conn(0x7f98a80c7910 msgr2=0x7f98a80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:09.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.709+0000 7f98acbc5640 1 -- 172.21.15.125:0/1218441828 shutdown_connections 2023-10-30T12:53:09.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.709+0000 7f98acbc5640 1 -- 172.21.15.125:0/1218441828 wait complete. 2023-10-30T12:53:09.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98acbc5640 1 Processor -- start 2023-10-30T12:53:09.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98acbc5640 1 -- start start 2023-10-30T12:53:09.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98acbc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98900069e0 0x7f98a81e0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:09.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98acbc5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98a81595d0 0x7f98a8158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:09.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98acbc5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98a8157790 0x7f98a8158570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:09.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98acbc5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a80ca590 con 0x7f98a8157790 2023-10-30T12:53:09.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98acbc5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98a8158ab0 con 0x7f98a81595d0 2023-10-30T12:53:09.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98acbc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98a81f2800 con 0x7f98900069e0 2023-10-30T12:53:09.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98a6d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98a8157790 0x7f98a8158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:09.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98a6d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98a8157790 0x7f98a8158570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41000/0 (socket says 172.21.15.125:41000) 2023-10-30T12:53:09.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98a6d9d640 1 -- 172.21.15.125:0/952131436 learned_addr learned my addr 172.21.15.125:0/952131436 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:53:09.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98a659c640 1 --2- 172.21.15.125:0/952131436 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98900069e0 0x7f98a81e0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:09.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98a5d9b640 1 --2- 172.21.15.125:0/952131436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98a81595d0 0x7f98a8158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:09.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98a6d9d640 1 -- 172.21.15.125:0/952131436 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98900069e0 msgr2=0x7f98a81e0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:09.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98a6d9d640 1 --2- 172.21.15.125:0/952131436 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98900069e0 0x7f98a81e0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:09.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98a6d9d640 1 -- 172.21.15.125:0/952131436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98a81595d0 msgr2=0x7f98a8158030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:09.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98a6d9d640 1 --2- 172.21.15.125:0/952131436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98a81595d0 0x7f98a8158030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:09.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98a6d9d640 1 -- 172.21.15.125:0/952131436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a80adee0 con 0x7f98a8157790 2023-10-30T12:53:09.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98a6d9d640 1 --2- 172.21.15.125:0/952131436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98a8157790 0x7f98a8158570 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f9890006960 tx=0x7f98a00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:09.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f9896ffd640 1 -- 172.21.15.125:0/952131436 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a000e070 con 0x7f98a8157790 2023-10-30T12:53:09.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f9896ffd640 1 -- 172.21.15.125:0/952131436 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a0011e40 con 0x7f98a8157790 2023-10-30T12:53:09.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f9896ffd640 1 -- 172.21.15.125:0/952131436 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a00039a0 con 0x7f98a8157790 2023-10-30T12:53:09.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98acbc5640 1 -- 172.21.15.125:0/952131436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98a80c9fc0 con 0x7f98a8157790 2023-10-30T12:53:09.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98acbc5640 1 -- 172.21.15.125:0/952131436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98a81f1ed0 con 0x7f98a8157790 2023-10-30T12:53:09.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.713+0000 7f98acbc5640 1 -- 172.21.15.125:0/952131436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98a804e170 con 0x7f98a8157790 2023-10-30T12:53:09.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.717+0000 7f9896ffd640 1 -- 172.21.15.125:0/952131436 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f98a0003b00 con 0x7f98a8157790 2023-10-30T12:53:09.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.717+0000 7f9896ffd640 1 --2- 172.21.15.125:0/952131436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98880664f0 0x7f98880689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:09.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.717+0000 7f9896ffd640 1 -- 172.21.15.125:0/952131436 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7f98a00846c0 con 0x7f98a8157790 2023-10-30T12:53:09.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.717+0000 7f98a659c640 1 --2- 172.21.15.125:0/952131436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98880664f0 0x7f98880689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:09.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.717+0000 7f98a659c640 1 --2- 172.21.15.125:0/952131436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98880664f0 0x7f98880689b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9898005630 tx=0x7f9898009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:09.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.721+0000 7f9896ffd640 1 -- 172.21.15.125:0/952131436 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f98a00517f0 con 0x7f98a8157790 2023-10-30T12:53:09.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.901+0000 7f98acbc5640 1 -- 172.21.15.125:0/952131436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98a81e4b20 con 0x7f98a8157790 2023-10-30T12:53:09.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.905+0000 7f9896ffd640 1 -- 172.21.15.125:0/952131436 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f98a0054e10 con 0x7f98a8157790 2023-10-30T12:53:09.906 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:53:09.906 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:59.039508+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:active","state_seq":19,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:53:09.907 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 122 2023-10-30T12:53:09.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.905+0000 7f98acbc5640 1 -- 172.21.15.125:0/952131436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98880664f0 msgr2=0x7f98880689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:09.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.905+0000 7f98acbc5640 1 --2- 172.21.15.125:0/952131436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98880664f0 0x7f98880689b0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9898005630 tx=0x7f9898009040 comp rx=0 tx=0).stop 2023-10-30T12:53:09.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.905+0000 7f98acbc5640 1 -- 172.21.15.125:0/952131436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98a8157790 msgr2=0x7f98a8158570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:09.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.905+0000 7f98acbc5640 1 --2- 172.21.15.125:0/952131436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98a8157790 0x7f98a8158570 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f9890006960 tx=0x7f98a00115a0 comp rx=0 tx=0).stop 2023-10-30T12:53:09.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.905+0000 7f98acbc5640 1 -- 172.21.15.125:0/952131436 shutdown_connections 2023-10-30T12:53:09.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.905+0000 7f98acbc5640 1 --2- 172.21.15.125:0/952131436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98880664f0 0x7f98880689b0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:09.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.905+0000 7f98acbc5640 1 --2- 172.21.15.125:0/952131436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f98a8157790 0x7f98a8158570 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:09.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.905+0000 7f98acbc5640 1 --2- 172.21.15.125:0/952131436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f98a81595d0 0x7f98a8158030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:09.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.905+0000 7f98acbc5640 1 --2- 172.21.15.125:0/952131436 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98900069e0 0x7f98a81e0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:09.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.905+0000 7f98acbc5640 1 -- 172.21.15.125:0/952131436 >> 172.21.15.125:0/952131436 conn(0x7f98a80c7910 msgr2=0x7f98a80bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:09.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.905+0000 7f98acbc5640 1 -- 172.21.15.125:0/952131436 shutdown_connections 2023-10-30T12:53:09.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:09.905+0000 7f98acbc5640 1 -- 172.21.15.125:0/952131436 wait complete. 2023-10-30T12:53:09.926 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T12:53:11.927 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:53:12.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.089+0000 7f2290da7640 1 Processor -- start 2023-10-30T12:53:12.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.089+0000 7f2290da7640 1 -- start start 2023-10-30T12:53:12.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f2290da7640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f228c1538f0 0x7f228c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:12.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f2290da7640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c0ca6a0 con 0x7f228c0cefa0 2023-10-30T12:53:12.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f2290da7640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f228c15d300 con 0x7f228c15cd50 2023-10-30T12:53:12.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f2290da7640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c15d5e0 con 0x7f228c1538f0 2023-10-30T12:53:12.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f228bfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f228c0cefa0 0x7f228c1640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40972/0 (socket says 172.21.15.125:40972) 2023-10-30T12:53:12.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f228bfff640 1 -- 172.21.15.125:0/3208688361 learned_addr learned my addr 172.21.15.125:0/3208688361 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:53:12.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f228b7fe640 1 --2- 172.21.15.125:0/3208688361 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f228c1538f0 0x7f228c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:12.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f228a7fc640 1 -- 172.21.15.125:0/3208688361 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114151765 0 0) 0x7f228c0ca6a0 con 0x7f228c0cefa0 2023-10-30T12:53:12.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f228a7fc640 1 -- 172.21.15.125:0/3208688361 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2274003530 con 0x7f228c0cefa0 2023-10-30T12:53:12.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f228a7fc640 1 -- 172.21.15.125:0/3208688361 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148342954 0 0) 0x7f228c15d300 con 0x7f228c15cd50 2023-10-30T12:53:12.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f228a7fc640 1 -- 172.21.15.125:0/3208688361 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f228c0ca6a0 con 0x7f228c15cd50 2023-10-30T12:53:12.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f228a7fc640 1 -- 172.21.15.125:0/3208688361 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3690015048 0 0) 0x7f2274003530 con 0x7f228c0cefa0 2023-10-30T12:53:12.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f228b7fe640 1 -- 172.21.15.125:0/3208688361 >> v1:172.21.15.136:6789/0 conn(0x7f228c15cd50 legacy=0x7f228c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:12.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f228b7fe640 1 -- 172.21.15.125:0/3208688361 >> v1:172.21.15.125:6789/0 conn(0x7f228c0cefa0 legacy=0x7f228c1640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:12.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.093+0000 7f228b7fe640 1 -- 172.21.15.125:0/3208688361 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f228c15d8c0 con 0x7f228c1538f0 2023-10-30T12:53:12.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.097+0000 7f228b7fe640 1 --2- 172.21.15.125:0/3208688361 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f228c1538f0 0x7f228c159fb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f227c0097f0 tx=0x7f227c01ebd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7501f6445397fd91 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:12.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.097+0000 7f228a7fc640 1 -- 172.21.15.125:0/3208688361 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2284001a00 con 0x7f228c1538f0 2023-10-30T12:53:12.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.097+0000 7f228a7fc640 1 -- 172.21.15.125:0/3208688361 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227c0a8650 con 0x7f228c1538f0 2023-10-30T12:53:12.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.097+0000 7f228a7fc640 1 -- 172.21.15.125:0/3208688361 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227c0a89c0 con 0x7f228c1538f0 2023-10-30T12:53:12.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.097+0000 7f2290da7640 1 -- 172.21.15.125:0/3208688361 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f228c1538f0 msgr2=0x7f228c159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:12.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.097+0000 7f2290da7640 1 --2- 172.21.15.125:0/3208688361 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f228c1538f0 0x7f228c159fb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f227c0097f0 tx=0x7f227c01ebd0 comp rx=0 tx=0).stop 2023-10-30T12:53:12.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.097+0000 7f2290da7640 1 -- 172.21.15.125:0/3208688361 shutdown_connections 2023-10-30T12:53:12.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.097+0000 7f2290da7640 1 --2- 172.21.15.125:0/3208688361 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f228c1538f0 0x7f228c159fb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:12.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.097+0000 7f2290da7640 1 -- 172.21.15.125:0/3208688361 >> 172.21.15.125:0/3208688361 conn(0x7f228c06a640 msgr2=0x7f228c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:12.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.097+0000 7f2290da7640 1 -- 172.21.15.125:0/3208688361 shutdown_connections 2023-10-30T12:53:12.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.097+0000 7f2290da7640 1 -- 172.21.15.125:0/3208688361 wait complete. 2023-10-30T12:53:12.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f2290da7640 1 Processor -- start 2023-10-30T12:53:12.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f2290da7640 1 -- start start 2023-10-30T12:53:12.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f2290da7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f228c0cefa0 0x7f228c152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:12.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f2290da7640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f228c1538f0 0x7f228c15bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:12.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f2290da7640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f228c15cd50 0x7f228c1455f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:12.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f2290da7640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c0b3050 con 0x7f228c1538f0 2023-10-30T12:53:12.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f2290da7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228c1ef580 con 0x7f228c0cefa0 2023-10-30T12:53:12.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f2290da7640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228c15b910 con 0x7f228c15cd50 2023-10-30T12:53:12.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f228affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f228c1538f0 0x7f228c15bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f228b7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f228c0cefa0 0x7f228c152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f228bfff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f228c15cd50 0x7f228c1455f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f228b7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f228c0cefa0 0x7f228c152860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:55668/0 (socket says 172.21.15.125:55668) 2023-10-30T12:53:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f228affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f228c1538f0 0x7f228c15bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52128/0 (socket says 172.21.15.125:52128) 2023-10-30T12:53:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f228b7fe640 1 -- 172.21.15.125:0/1127857437 learned_addr learned my addr 172.21.15.125:0/1127857437 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:53:12.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f228affd640 1 -- 172.21.15.125:0/1127857437 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f228c15cd50 msgr2=0x7f228c1455f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:12.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f228affd640 1 --2- 172.21.15.125:0/1127857437 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f228c15cd50 0x7f228c1455f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:12.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f228affd640 1 -- 172.21.15.125:0/1127857437 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f228c0cefa0 msgr2=0x7f228c152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:12.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f228affd640 1 --2- 172.21.15.125:0/1127857437 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f228c0cefa0 0x7f228c152860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:12.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f228affd640 1 -- 172.21.15.125:0/1127857437 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227c009420 con 0x7f228c1538f0 2023-10-30T12:53:12.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f228affd640 1 --2- 172.21.15.125:0/1127857437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f228c1538f0 0x7f228c15bda0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f228000c3f0 tx=0x7f228000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:12.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f2273fff640 1 -- 172.21.15.125:0/1127857437 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22800052d0 con 0x7f228c1538f0 2023-10-30T12:53:12.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f2273fff640 1 -- 172.21.15.125:0/1127857437 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2280005430 con 0x7f228c1538f0 2023-10-30T12:53:12.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f2290da7640 1 -- 172.21.15.125:0/1127857437 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228c15ab20 con 0x7f228c1538f0 2023-10-30T12:53:12.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f2273fff640 1 -- 172.21.15.125:0/1127857437 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2280003840 con 0x7f228c1538f0 2023-10-30T12:53:12.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.101+0000 7f2290da7640 1 -- 172.21.15.125:0/1127857437 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228c1ef2c0 con 0x7f228c1538f0 2023-10-30T12:53:12.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.105+0000 7f2290da7640 1 -- 172.21.15.125:0/1127857437 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228c0cf8f0 con 0x7f228c1538f0 2023-10-30T12:53:12.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.105+0000 7f2273fff640 1 -- 172.21.15.125:0/1127857437 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f22800039a0 con 0x7f228c1538f0 2023-10-30T12:53:12.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.109+0000 7f2273fff640 1 --2- 172.21.15.125:0/1127857437 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22780664a0 0x7f2278068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:53:12.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.109+0000 7f2273fff640 1 -- 172.21.15.125:0/1127857437 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7f2280083820 con 0x7f228c1538f0 2023-10-30T12:53:12.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.109+0000 7f228b7fe640 1 --2- 172.21.15.125:0/1127857437 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22780664a0 0x7f2278068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:53:12.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.109+0000 7f228b7fe640 1 --2- 172.21.15.125:0/1127857437 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22780664a0 0x7f2278068960 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f227c0097f0 tx=0x7f227c0aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:53:12.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.113+0000 7f2273fff640 1 -- 172.21.15.125:0/1127857437 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2280050880 con 0x7f228c1538f0 2023-10-30T12:53:12.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.309+0000 7f2290da7640 1 -- 172.21.15.125:0/1127857437 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f228c04e170 con 0x7f228c1538f0 2023-10-30T12:53:12.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.309+0000 7f2273fff640 1 -- 172.21.15.125:0/1127857437 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f2280053ea0 con 0x7f228c1538f0 2023-10-30T12:53:12.312 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:53:12.312 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7113,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:59.039508+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:active","state_seq":19,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:53:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 123 2023-10-30T12:53:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.313+0000 7f2290da7640 1 -- 172.21.15.125:0/1127857437 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22780664a0 msgr2=0x7f2278068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.313+0000 7f2290da7640 1 --2- 172.21.15.125:0/1127857437 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22780664a0 0x7f2278068960 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f227c0097f0 tx=0x7f227c0aa070 comp rx=0 tx=0).stop 2023-10-30T12:53:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.313+0000 7f2290da7640 1 -- 172.21.15.125:0/1127857437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f228c1538f0 msgr2=0x7f228c15bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:53:12.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.313+0000 7f2290da7640 1 --2- 172.21.15.125:0/1127857437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f228c1538f0 0x7f228c15bda0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f228000c3f0 tx=0x7f228000c8b0 comp rx=0 tx=0).stop 2023-10-30T12:53:12.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.313+0000 7f2290da7640 1 -- 172.21.15.125:0/1127857437 shutdown_connections 2023-10-30T12:53:12.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.313+0000 7f2290da7640 1 --2- 172.21.15.125:0/1127857437 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f22780664a0 0x7f2278068960 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:12.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.313+0000 7f2290da7640 1 --2- 172.21.15.125:0/1127857437 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f228c15cd50 0x7f228c1455f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:12.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.313+0000 7f2290da7640 1 --2- 172.21.15.125:0/1127857437 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f228c1538f0 0x7f228c15bda0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:12.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.313+0000 7f2290da7640 1 --2- 172.21.15.125:0/1127857437 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f228c0cefa0 0x7f228c152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:53:12.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.313+0000 7f2290da7640 1 -- 172.21.15.125:0/1127857437 >> 172.21.15.125:0/1127857437 conn(0x7f228c06a640 msgr2=0x7f228c1590c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:53:12.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.313+0000 7f2290da7640 1 -- 172.21.15.125:0/1127857437 shutdown_connections 2023-10-30T12:53:12.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:53:12.313+0000 7f2290da7640 1 -- 172.21.15.125:0/1127857437 wait complete. 2023-10-30T12:53:12.333 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T12:53:12.333 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 109 secs before thrashing 2023-10-30T12:53:16.762 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 7% (56/779) Updating files: 8% (63/779) Updating files: 9% (71/779) Updating files: 10% (78/779) Updating files: 11% (86/779) Updating files: 12% (94/779) Updating files: 13% (102/779) Updating files: 14% (110/779) Updating files: 15% (117/779) Updating files: 15% (121/779) Updating files: 16% (125/779) Updating files: 17% (133/779) Updating files: 18% (141/779) Updating files: 19% (149/779) Updating files: 20% (156/779) Updating files: 21% (164/779) Updating files: 22% (172/779) Updating files: 23% (180/779) Updating files: 24% (187/779) Updating files: 25% (195/779) Updating files: 26% (203/779) Updating files: 27% (211/779) Updating files: 27% (216/779) Updating files: 28% (219/779) Updating files: 29% (226/779) Updating files: 30% (234/779) Updating files: 30% (235/779) Updating files: 31% (242/779) Updating files: 32% (250/779) Updating files: 33% (258/779) Updating files: 33% (260/779) Updating files: 34% (265/779) Updating files: 35% (273/779) Updating files: 36% (281/779) Updating files: 37% (289/779) Updating files: 38% (297/779) Updating files: 39% (304/779) Updating files: 40% (312/779) Updating files: 41% (320/779) Updating files: 42% (328/779) Updating files: 42% (332/779) Updating files: 43% (335/779) Updating files: 44% (343/779) Updating files: 45% (351/779) Updating files: 46% (359/779) Updating files: 47% (367/779) Updating files: 48% (374/779) Updating files: 49% (382/779) Updating files: 50% (390/779) Updating files: 51% (398/779) Updating files: 52% (406/779) Updating files: 53% (413/779) Updating files: 53% (414/779) Updating files: 54% (421/779) Updating files: 55% (429/779) Updating files: 56% (437/779) Updating files: 56% (443/779) Updating files: 57% (445/779) Updating files: 58% (452/779) Updating files: 59% (460/779) Updating files: 59% (461/779) Updating files: 60% (468/779) Updating files: 61% (476/779) Updating files: 62% (483/779) Updating files: 63% (491/779) Updating files: 64% (499/779) Updating files: 65% (507/779) Updating files: 66% (515/779) Updating files: 67% (522/779) Updating files: 68% (530/779) Updating files: 69% (538/779) Updating files: 70% (546/779) Updating files: 70% (548/779) Updating files: 71% (554/779) Updating files: 72% (561/779) Updating files: 73% (569/779) Updating files: 74% (577/779) Updating files: 75% (585/779) Updating files: 76% (593/779) Updating files: 77% (600/779) Updating files: 78% (608/779) Updating files: 79% (616/779) Updating files: 79% (619/779) Updating files: 80% (624/779) Updating files: 81% (631/779) Updating files: 82% (639/779) Updating files: 83% (647/779) Updating files: 84% (655/779) Updating files: 84% (658/779) Updating files: 85% (663/779) Updating files: 86% (670/779) Updating files: 87% (678/779) Updating files: 88% (686/779) Updating files: 89% (694/779) Updating files: 90% (702/779) Updating files: 91% (709/779) Updating files: 92% (717/779) Updating files: 92% (723/779) Updating files: 93% (725/779) Updating files: 94% (733/779) Updating files: 95% (741/779) Updating files: 96% (748/779) Updating files: 97% (756/779) Updating files: 98% (764/779) Updating files: 99% (772/779) Updating files: 100% (779/779) Updating files: 100% (779/779), done. 2023-10-30T12:53:17.726 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 2f10137986a v0.18 2023-10-30T12:53:17.731 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.18 2023-10-30T12:53:17.755 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 19 -eq 48 ] 2023-10-30T12:53:17.755 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.19 2023-10-30T12:53:17.755 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.19 2023-10-30T12:53:17.756 INFO:tasks.workunit.client.0.smithi125.stdout:v0.19 2023-10-30T12:53:17.756 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.19 2023-10-30T12:53:18.146 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:53:18.150 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:53:20.601 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 73% (159/217) Updating files: 74% (161/217) Updating files: 75% (163/217) Updating files: 76% (165/217) Updating files: 77% (168/217) Updating files: 78% (170/217) Updating files: 79% (172/217) Updating files: 80% (174/217) Updating files: 81% (176/217) Updating files: 82% (178/217) Updating files: 83% (181/217) Updating files: 84% (183/217) Updating files: 85% (185/217) Updating files: 86% (187/217) Updating files: 87% (189/217) Updating files: 88% (191/217) Updating files: 89% (194/217) Updating files: 90% (196/217) Updating files: 91% (198/217) Updating files: 92% (200/217) Updating files: 93% (202/217) Updating files: 94% (204/217) Updating files: 95% (207/217) Updating files: 96% (209/217) Updating files: 97% (211/217) Updating files: 98% (213/217) Updating files: 99% (215/217) Updating files: 100% (217/217) Updating files: 100% (217/217), done. 2023-10-30T12:53:20.638 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 98f5be53f85 v0.19 2023-10-30T12:53:20.642 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.19 2023-10-30T12:53:20.744 INFO:tasks.workunit.client.0.smithi125.stdout:v0.20 2023-10-30T12:53:20.745 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 20 -eq 48 ] 2023-10-30T12:53:20.745 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.20 2023-10-30T12:53:20.745 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.20 2023-10-30T12:53:20.746 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.20 2023-10-30T12:53:24.751 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 21% (53/245) Updating files: 22% (54/245) Updating files: 23% (57/245) Updating files: 24% (59/245) Updating files: 25% (62/245) Updating files: 26% (64/245) Updating files: 27% (67/245) Updating files: 28% (69/245) Updating files: 29% (72/245) Updating files: 30% (74/245) Updating files: 31% (76/245) Updating files: 32% (79/245) Updating files: 33% (81/245) Updating files: 34% (84/245) Updating files: 35% (86/245) Updating files: 36% (89/245) Updating files: 37% (91/245) Updating files: 38% (94/245) Updating files: 39% (96/245) Updating files: 40% (98/245) Updating files: 41% (101/245) Updating files: 42% (103/245) Updating files: 43% (106/245) Updating files: 44% (108/245) Updating files: 45% (111/245) Updating files: 46% (113/245) Updating files: 47% (116/245) Updating files: 48% (118/245) Updating files: 49% (121/245) Updating files: 50% (123/245) Updating files: 51% (125/245) Updating files: 52% (128/245) Updating files: 53% (130/245) Updating files: 54% (133/245) Updating files: 55% (135/245) Updating files: 56% (138/245) Updating files: 56% (139/245) Updating files: 57% (140/245) Updating files: 58% (143/245) Updating files: 59% (145/245) Updating files: 60% (147/245) Updating files: 61% (150/245) Updating files: 62% (152/245) Updating files: 63% (155/245) Updating files: 64% (157/245) Updating files: 65% (160/245) Updating files: 66% (162/245) Updating files: 67% (165/245) Updating files: 68% (167/245) Updating files: 69% (170/245) Updating files: 70% (172/245) Updating files: 71% (174/245) Updating files: 72% (177/245) Updating files: 73% (179/245) Updating files: 74% (182/245) Updating files: 75% (184/245) Updating files: 75% (186/245) Updating files: 76% (187/245) Updating files: 77% (189/245) Updating files: 78% (192/245) Updating files: 79% (194/245) Updating files: 80% (196/245) Updating files: 81% (199/245) Updating files: 82% (201/245) Updating files: 83% (204/245) Updating files: 84% (206/245) Updating files: 85% (209/245) Updating files: 86% (211/245) Updating files: 87% (214/245) Updating files: 88% (216/245) Updating files: 89% (219/245) Updating files: 90% (221/245) Updating files: 91% (223/245) Updating files: 92% (226/245) Updating files: 93% (228/245) Updating files: 94% (231/245) Updating files: 95% (233/245) Updating files: 96% (236/245) Updating files: 97% (238/245) Updating files: 98% (241/245) Updating files: 99% (243/245) Updating files: 100% (245/245) Updating files: 100% (245/245), done. 2023-10-30T12:53:24.774 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 266d2076b6c debian: fix up debian scripts 2023-10-30T12:53:24.777 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.20 2023-10-30T12:53:24.812 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 21 -eq 48 ] 2023-10-30T12:53:24.813 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.21 2023-10-30T12:53:24.813 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.21 2023-10-30T12:53:24.813 INFO:tasks.workunit.client.0.smithi125.stdout:v0.21 2023-10-30T12:53:24.814 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.21 2023-10-30T12:53:29.636 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 23% (128/537) Updating files: 24% (129/537) Updating files: 25% (135/537) Updating files: 26% (140/537) Updating files: 27% (145/537) Updating files: 28% (151/537) Updating files: 29% (156/537) Updating files: 30% (162/537) Updating files: 31% (167/537) Updating files: 32% (172/537) Updating files: 32% (175/537) Updating files: 33% (178/537) Updating files: 34% (183/537) Updating files: 35% (188/537) Updating files: 36% (194/537) Updating files: 37% (199/537) Updating files: 38% (205/537) Updating files: 39% (210/537) Updating files: 40% (215/537) Updating files: 41% (221/537) Updating files: 42% (226/537) Updating files: 43% (231/537) Updating files: 44% (237/537) Updating files: 45% (242/537) Updating files: 46% (248/537) Updating files: 47% (253/537) Updating files: 47% (254/537) Updating files: 48% (258/537) Updating files: 49% (264/537) Updating files: 50% (269/537) Updating files: 51% (274/537) Updating files: 52% (280/537) Updating files: 53% (285/537) Updating files: 54% (290/537) Updating files: 55% (296/537) Updating files: 56% (301/537) Updating files: 57% (307/537) Updating files: 58% (312/537) Updating files: 59% (317/537) Updating files: 60% (323/537) Updating files: 61% (328/537) Updating files: 62% (333/537) Updating files: 63% (339/537) Updating files: 63% (343/537) Updating files: 64% (344/537) Updating files: 65% (350/537) Updating files: 66% (355/537) Updating files: 67% (360/537) Updating files: 68% (366/537) Updating files: 69% (371/537) Updating files: 70% (376/537) Updating files: 71% (382/537) Updating files: 72% (387/537) Updating files: 73% (393/537) Updating files: 74% (398/537) Updating files: 75% (403/537) Updating files: 76% (409/537) Updating files: 77% (414/537) Updating files: 78% (419/537) Updating files: 79% (425/537) Updating files: 80% (430/537) Updating files: 81% (435/537) Updating files: 82% (441/537) Updating files: 83% (446/537) Updating files: 84% (452/537) Updating files: 85% (457/537) Updating files: 86% (462/537) Updating files: 87% (468/537) Updating files: 88% (473/537) Updating files: 89% (478/537) Updating files: 90% (484/537) Updating files: 91% (489/537) Updating files: 92% (495/537) Updating files: 93% (500/537) Updating files: 94% (505/537) Updating files: 95% (511/537) Updating files: 96% (516/537) Updating files: 97% (521/537) Updating files: 98% (527/537) Updating files: 99% (532/537) Updating files: 100% (537/537) Updating files: 100% (537/537), done. 2023-10-30T12:53:29.679 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 090436f56ac v0.21 2023-10-30T12:53:29.696 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.21 2023-10-30T12:53:29.822 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 22 -eq 48 ] 2023-10-30T12:53:29.822 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.22 2023-10-30T12:53:29.822 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.22 2023-10-30T12:53:29.823 INFO:tasks.workunit.client.0.smithi125.stdout:v0.22 2023-10-30T12:53:29.823 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.22 2023-10-30T12:53:31.188 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 92% (189/204) Updating files: 93% (190/204) Updating files: 94% (192/204) Updating files: 95% (194/204) Updating files: 96% (196/204) Updating files: 97% (198/204) Updating files: 98% (200/204) Updating files: 99% (202/204) Updating files: 100% (204/204) Updating files: 100% (204/204), done. 2023-10-30T12:53:31.480 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 8a7c95f60ad v0.22 2023-10-30T12:53:31.485 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.22 2023-10-30T12:53:31.658 INFO:tasks.workunit.client.0.smithi125.stdout:v0.23 2023-10-30T12:53:31.659 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 23 -eq 48 ] 2023-10-30T12:53:31.659 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.23 2023-10-30T12:53:31.659 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.23 2023-10-30T12:53:31.660 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.23 2023-10-30T12:53:48.211 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:53:48.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:53:50.106 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 14% (23/163) Updating files: 15% (25/163) Updating files: 16% (27/163) Updating files: 17% (28/163) Updating files: 18% (30/163) Updating files: 19% (31/163) Updating files: 20% (33/163) Updating files: 21% (35/163) Updating files: 22% (36/163) Updating files: 23% (38/163) Updating files: 24% (40/163) Updating files: 25% (41/163) Updating files: 26% (43/163) Updating files: 27% (45/163) Updating files: 28% (46/163) Updating files: 29% (48/163) Updating files: 30% (49/163) Updating files: 31% (51/163) Updating files: 32% (53/163) Updating files: 33% (54/163) Updating files: 34% (56/163) Updating files: 35% (58/163) Updating files: 36% (59/163) Updating files: 37% (61/163) Updating files: 38% (62/163) Updating files: 38% (63/163) Updating files: 39% (64/163) Updating files: 40% (66/163) Updating files: 41% (67/163) Updating files: 42% (69/163) Updating files: 43% (71/163) Updating files: 44% (72/163) Updating files: 45% (74/163) Updating files: 46% (75/163) Updating files: 47% (77/163) Updating files: 48% (79/163) Updating files: 49% (80/163) Updating files: 50% (82/163) Updating files: 51% (84/163) Updating files: 52% (85/163) Updating files: 53% (87/163) Updating files: 54% (89/163) Updating files: 55% (90/163) Updating files: 56% (92/163) Updating files: 57% (93/163) Updating files: 58% (95/163) Updating files: 59% (97/163) Updating files: 60% (98/163) Updating files: 61% (100/163) Updating files: 62% (102/163) Updating files: 63% (103/163) Updating files: 64% (105/163) Updating files: 65% (106/163) Updating files: 66% (108/163) Updating files: 67% (110/163) Updating files: 68% (111/163) Updating files: 69% (113/163) Updating files: 70% (115/163) Updating files: 71% (116/163) Updating files: 72% (118/163) Updating files: 73% (119/163) Updating files: 74% (121/163) Updating files: 75% (123/163) Updating files: 76% (124/163) Updating files: 77% (126/163) Updating files: 78% (128/163) Updating files: 79% (129/163) Updating files: 80% (131/163) Updating files: 81% (133/163) Updating files: 82% (134/163) Updating files: 83% (136/163) Updating files: 84% (137/163) Updating files: 85% (139/163) Updating files: 86% (141/163) Updating files: 87% (142/163) Updating files: 88% (144/163) Updating files: 89% (146/163) Updating files: 90% (147/163) Updating files: 91% (149/163) Updating files: 92% (150/163) Updating files: 93% (152/163) Updating files: 94% (154/163) Updating files: 95% (155/163) Updating files: 96% (157/163) Updating files: 97% (159/163) Updating files: 98% (160/163) Updating files: 99% (162/163) Updating files: 100% (163/163) Updating files: 100% (163/163), done. 2023-10-30T12:53:50.137 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 5d1d8d0c460 v0.23 2023-10-30T12:53:50.142 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.23 2023-10-30T12:53:50.995 INFO:tasks.workunit.client.0.smithi125.stdout:v0.24 2023-10-30T12:53:50.996 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 24 -eq 48 ] 2023-10-30T12:53:50.996 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.24 2023-10-30T12:53:50.996 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.24 2023-10-30T12:53:50.996 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.24 2023-10-30T12:53:54.687 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 74% (137/185) Updating files: 75% (139/185) Updating files: 76% (141/185) Updating files: 77% (143/185) Updating files: 78% (145/185) Updating files: 79% (147/185) Updating files: 80% (148/185) Updating files: 81% (150/185) Updating files: 82% (152/185) Updating files: 83% (154/185) Updating files: 84% (156/185) Updating files: 84% (157/185) Updating files: 85% (158/185) Updating files: 86% (160/185) Updating files: 87% (161/185) Updating files: 88% (163/185) Updating files: 89% (165/185) Updating files: 90% (167/185) Updating files: 91% (169/185) Updating files: 92% (171/185) Updating files: 93% (173/185) Updating files: 94% (174/185) Updating files: 95% (176/185) Updating files: 96% (178/185) Updating files: 97% (180/185) Updating files: 98% (182/185) Updating files: 98% (183/185) Updating files: 99% (184/185) Updating files: 100% (185/185) Updating files: 100% (185/185), done. 2023-10-30T12:53:54.721 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 180a4176035 v0.24 2023-10-30T12:53:54.726 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.24 2023-10-30T12:53:55.660 INFO:tasks.workunit.client.0.smithi125.stdout:v0.25 2023-10-30T12:53:55.660 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 25 -eq 48 ] 2023-10-30T12:53:55.661 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.25 2023-10-30T12:53:55.661 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.25 2023-10-30T12:53:55.661 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.25 2023-10-30T12:54:08.117 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 18% (116/622) Updating files: 19% (119/622) Updating files: 20% (125/622) Updating files: 21% (131/622) Updating files: 22% (137/622) Updating files: 23% (144/622) Updating files: 24% (150/622) Updating files: 25% (156/622) Updating files: 26% (162/622) Updating files: 27% (168/622) Updating files: 28% (175/622) Updating files: 29% (181/622) Updating files: 30% (187/622) Updating files: 31% (193/622) Updating files: 32% (200/622) Updating files: 33% (206/622) Updating files: 34% (212/622) Updating files: 35% (218/622) Updating files: 36% (224/622) Updating files: 37% (231/622) Updating files: 38% (237/622) Updating files: 39% (243/622) Updating files: 40% (249/622) Updating files: 41% (256/622) Updating files: 42% (262/622) Updating files: 43% (268/622) Updating files: 43% (271/622) Updating files: 44% (274/622) Updating files: 45% (280/622) Updating files: 46% (287/622) Updating files: 47% (293/622) Updating files: 48% (299/622) Updating files: 49% (305/622) Updating files: 50% (311/622) Updating files: 51% (318/622) Updating files: 52% (324/622) Updating files: 53% (330/622) Updating files: 54% (336/622) Updating files: 55% (343/622) Updating files: 56% (349/622) Updating files: 57% (355/622) Updating files: 58% (361/622) Updating files: 58% (365/622) Updating files: 59% (367/622) Updating files: 60% (374/622) Updating files: 61% (380/622) Updating files: 62% (386/622) Updating files: 63% (392/622) Updating files: 64% (399/622) Updating files: 65% (405/622) Updating files: 66% (411/622) Updating files: 67% (417/622) Updating files: 68% (423/622) Updating files: 69% (430/622) Updating files: 70% (436/622) Updating files: 71% (442/622) Updating files: 72% (448/622) Updating files: 73% (455/622) Updating files: 73% (457/622) Updating files: 74% (461/622) Updating files: 75% (467/622) Updating files: 76% (473/622) Updating files: 77% (479/622) Updating files: 78% (486/622) Updating files: 79% (492/622) Updating files: 80% (498/622) Updating files: 80% (499/622) Updating files: 81% (504/622) Updating files: 82% (511/622) Updating files: 83% (517/622) Updating files: 84% (523/622) Updating files: 85% (529/622) Updating files: 86% (535/622) Updating files: 86% (538/622) Updating files: 87% (542/622) Updating files: 88% (548/622) Updating files: 89% (554/622) Updating files: 90% (560/622) Updating files: 91% (567/622) Updating files: 92% (573/622) Updating files: 93% (579/622) Updating files: 94% (585/622) Updating files: 95% (591/622) Updating files: 96% (598/622) Updating files: 97% (604/622) Updating files: 98% (610/622) Updating files: 99% (616/622) Updating files: 100% (622/622) Updating files: 100% (622/622), done. 2023-10-30T12:54:08.145 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 3fa121bf0a8 v0.25 2023-10-30T12:54:08.150 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.25 2023-10-30T12:54:08.294 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 26 -eq 48 ] 2023-10-30T12:54:08.295 INFO:tasks.workunit.client.0.smithi125.stdout:v0.26 2023-10-30T12:54:08.295 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.26 2023-10-30T12:54:08.296 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.26 2023-10-30T12:54:08.296 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.26 2023-10-30T12:54:10.634 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 46% (111/239) Updating files: 47% (113/239) Updating files: 48% (115/239) Updating files: 49% (118/239) Updating files: 50% (120/239) Updating files: 51% (122/239) Updating files: 52% (125/239) Updating files: 53% (127/239) Updating files: 54% (130/239) Updating files: 55% (132/239) Updating files: 56% (134/239) Updating files: 57% (137/239) Updating files: 58% (139/239) Updating files: 59% (142/239) Updating files: 60% (144/239) Updating files: 61% (146/239) Updating files: 62% (149/239) Updating files: 63% (151/239) Updating files: 64% (153/239) Updating files: 65% (156/239) Updating files: 66% (158/239) Updating files: 67% (161/239) Updating files: 68% (163/239) Updating files: 69% (165/239) Updating files: 70% (168/239) Updating files: 71% (170/239) Updating files: 72% (173/239) Updating files: 73% (175/239) Updating files: 74% (177/239) Updating files: 75% (180/239) Updating files: 76% (182/239) Updating files: 77% (185/239) Updating files: 78% (187/239) Updating files: 79% (189/239) Updating files: 80% (192/239) Updating files: 81% (194/239) Updating files: 82% (196/239) Updating files: 83% (199/239) Updating files: 84% (201/239) Updating files: 85% (204/239) Updating files: 86% (206/239) Updating files: 87% (208/239) Updating files: 88% (211/239) Updating files: 89% (213/239) Updating files: 89% (215/239) Updating files: 90% (216/239) Updating files: 91% (218/239) Updating files: 92% (220/239) Updating files: 93% (223/239) Updating files: 94% (225/239) Updating files: 95% (228/239) Updating files: 96% (230/239) Updating files: 97% (232/239) Updating files: 98% (235/239) Updating files: 99% (237/239) Updating files: 100% (239/239) Updating files: 100% (239/239), done. 2023-10-30T12:54:10.919 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 9981ff90968 v0.26 2023-10-30T12:54:10.925 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.26 2023-10-30T12:54:11.499 INFO:tasks.workunit.client.0.smithi125.stdout:v0.27 2023-10-30T12:54:11.499 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 27 -eq 48 ] 2023-10-30T12:54:11.499 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.27 2023-10-30T12:54:11.500 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.27 2023-10-30T12:54:11.500 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.27 2023-10-30T12:54:14.310 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 46% (117/253) Updating files: 47% (119/253) Updating files: 48% (122/253) Updating files: 49% (124/253) Updating files: 50% (127/253) Updating files: 51% (130/253) Updating files: 52% (132/253) Updating files: 53% (135/253) Updating files: 54% (137/253) Updating files: 55% (140/253) Updating files: 56% (142/253) Updating files: 57% (145/253) Updating files: 58% (147/253) Updating files: 59% (150/253) Updating files: 60% (152/253) Updating files: 61% (155/253) Updating files: 62% (157/253) Updating files: 63% (160/253) Updating files: 64% (162/253) Updating files: 65% (165/253) Updating files: 66% (167/253) Updating files: 67% (170/253) Updating files: 68% (173/253) Updating files: 69% (175/253) Updating files: 70% (178/253) Updating files: 71% (180/253) Updating files: 72% (183/253) Updating files: 73% (185/253) Updating files: 74% (188/253) Updating files: 75% (190/253) Updating files: 76% (193/253) Updating files: 77% (195/253) Updating files: 78% (198/253) Updating files: 79% (200/253) Updating files: 80% (203/253) Updating files: 81% (205/253) Updating files: 82% (208/253) Updating files: 83% (210/253) Updating files: 84% (213/253) Updating files: 85% (216/253) Updating files: 86% (218/253) Updating files: 87% (221/253) Updating files: 88% (223/253) Updating files: 89% (226/253) Updating files: 90% (228/253) Updating files: 91% (231/253) Updating files: 92% (233/253) Updating files: 93% (236/253) Updating files: 94% (238/253) Updating files: 95% (241/253) Updating files: 96% (243/253) Updating files: 97% (246/253) Updating files: 98% (248/253) Updating files: 99% (251/253) Updating files: 100% (253/253) Updating files: 100% (253/253), done. 2023-10-30T12:54:14.344 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 793034c62c8 v0.27 2023-10-30T12:54:14.349 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.27 2023-10-30T12:54:14.719 INFO:tasks.workunit.client.0.smithi125.stdout:v0.28 2023-10-30T12:54:14.719 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 28 -eq 48 ] 2023-10-30T12:54:14.720 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.28 2023-10-30T12:54:14.720 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.28 2023-10-30T12:54:14.720 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.28 2023-10-30T12:54:18.267 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:54:18.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:54:30.773 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 53% (122/227) Updating files: 54% (123/227) Updating files: 55% (125/227) Updating files: 56% (128/227) Updating files: 57% (130/227) Updating files: 58% (132/227) Updating files: 59% (134/227) Updating files: 60% (137/227) Updating files: 61% (139/227) Updating files: 62% (141/227) Updating files: 63% (144/227) Updating files: 64% (146/227) Updating files: 65% (148/227) Updating files: 66% (150/227) Updating files: 67% (153/227) Updating files: 68% (155/227) Updating files: 69% (157/227) Updating files: 70% (159/227) Updating files: 71% (162/227) Updating files: 72% (164/227) Updating files: 73% (166/227) Updating files: 74% (168/227) Updating files: 75% (171/227) Updating files: 76% (173/227) Updating files: 77% (175/227) Updating files: 78% (178/227) Updating files: 79% (180/227) Updating files: 80% (182/227) Updating files: 81% (184/227) Updating files: 82% (187/227) Updating files: 83% (189/227) Updating files: 84% (191/227) Updating files: 85% (193/227) Updating files: 86% (196/227) Updating files: 87% (198/227) Updating files: 88% (200/227) Updating files: 89% (203/227) Updating files: 90% (205/227) Updating files: 91% (207/227) Updating files: 92% (209/227) Updating files: 93% (212/227) Updating files: 93% (213/227) Updating files: 94% (214/227) Updating files: 95% (216/227) Updating files: 96% (218/227) Updating files: 97% (221/227) Updating files: 98% (223/227) Updating files: 99% (225/227) Updating files: 100% (227/227) Updating files: 100% (227/227), done. 2023-10-30T12:54:30.809 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 071881d7e55 v0.28 2023-10-30T12:54:30.821 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.28 2023-10-30T12:54:30.887 INFO:tasks.workunit.client.0.smithi125.stdout:v0.29 2023-10-30T12:54:30.888 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 29 -eq 48 ] 2023-10-30T12:54:30.888 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.29 2023-10-30T12:54:30.888 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.29 2023-10-30T12:54:30.888 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.29 2023-10-30T12:54:32.878 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 71% (114/159) Updating files: 72% (115/159) Updating files: 73% (117/159) Updating files: 74% (118/159) Updating files: 75% (120/159) Updating files: 76% (121/159) Updating files: 77% (123/159) Updating files: 78% (125/159) Updating files: 79% (126/159) Updating files: 80% (128/159) Updating files: 81% (129/159) Updating files: 82% (131/159) Updating files: 83% (132/159) Updating files: 84% (134/159) Updating files: 85% (136/159) Updating files: 86% (137/159) Updating files: 87% (139/159) Updating files: 88% (140/159) Updating files: 89% (142/159) Updating files: 90% (144/159) Updating files: 91% (145/159) Updating files: 92% (147/159) Updating files: 93% (148/159) Updating files: 94% (150/159) Updating files: 95% (152/159) Updating files: 96% (153/159) Updating files: 97% (155/159) Updating files: 98% (156/159) Updating files: 99% (158/159) Updating files: 100% (159/159) Updating files: 100% (159/159), done. 2023-10-30T12:54:32.914 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 8e69c39f699 v0.29 2023-10-30T12:54:32.920 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.29 2023-10-30T12:54:33.023 INFO:tasks.workunit.client.0.smithi125.stdout:v0.30 2023-10-30T12:54:33.023 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 30 -eq 48 ] 2023-10-30T12:54:33.024 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.30 2023-10-30T12:54:33.024 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.30 2023-10-30T12:54:33.024 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.30 2023-10-30T12:54:36.548 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 74% (213/285) Updating files: 75% (214/285) Updating files: 76% (217/285) Updating files: 77% (220/285) Updating files: 78% (223/285) Updating files: 79% (226/285) Updating files: 80% (228/285) Updating files: 81% (231/285) Updating files: 82% (234/285) Updating files: 83% (237/285) Updating files: 84% (240/285) Updating files: 85% (243/285) Updating files: 86% (246/285) Updating files: 87% (248/285) Updating files: 88% (251/285) Updating files: 89% (254/285) Updating files: 90% (257/285) Updating files: 91% (260/285) Updating files: 92% (263/285) Updating files: 93% (266/285) Updating files: 94% (268/285) Updating files: 95% (271/285) Updating files: 96% (274/285) Updating files: 97% (277/285) Updating files: 98% (280/285) Updating files: 99% (283/285) Updating files: 100% (285/285) Updating files: 100% (285/285), done. 2023-10-30T12:54:36.857 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 64b1b2c70f0 v0.30 2023-10-30T12:54:36.861 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.30 2023-10-30T12:54:36.897 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 31 -eq 48 ] 2023-10-30T12:54:36.898 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.31 2023-10-30T12:54:36.898 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.31 2023-10-30T12:54:36.898 INFO:tasks.workunit.client.0.smithi125.stdout:v0.31 2023-10-30T12:54:36.899 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.31 2023-10-30T12:54:48.330 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:54:48.335 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:54:51.426 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 18% (72/398) Updating files: 19% (76/398) Updating files: 20% (80/398) Updating files: 21% (84/398) Updating files: 22% (88/398) Updating files: 23% (92/398) Updating files: 24% (96/398) Updating files: 25% (100/398) Updating files: 26% (104/398) Updating files: 27% (108/398) Updating files: 28% (112/398) Updating files: 29% (116/398) Updating files: 30% (120/398) Updating files: 30% (123/398) Updating files: 31% (124/398) Updating files: 32% (128/398) Updating files: 33% (132/398) Updating files: 34% (136/398) Updating files: 35% (140/398) Updating files: 36% (144/398) Updating files: 37% (148/398) Updating files: 38% (152/398) Updating files: 39% (156/398) Updating files: 40% (160/398) Updating files: 41% (164/398) Updating files: 42% (168/398) Updating files: 43% (172/398) Updating files: 44% (176/398) Updating files: 45% (180/398) Updating files: 46% (184/398) Updating files: 47% (188/398) Updating files: 48% (192/398) Updating files: 49% (196/398) Updating files: 50% (199/398) Updating files: 51% (203/398) Updating files: 52% (207/398) Updating files: 53% (211/398) Updating files: 54% (215/398) Updating files: 55% (219/398) Updating files: 56% (223/398) Updating files: 57% (227/398) Updating files: 58% (231/398) Updating files: 59% (235/398) Updating files: 60% (239/398) Updating files: 61% (243/398) Updating files: 62% (247/398) Updating files: 63% (251/398) Updating files: 64% (255/398) Updating files: 65% (259/398) Updating files: 66% (263/398) Updating files: 67% (267/398) Updating files: 68% (271/398) Updating files: 68% (272/398) Updating files: 69% (275/398) Updating files: 70% (279/398) Updating files: 71% (283/398) Updating files: 72% (287/398) Updating files: 73% (291/398) Updating files: 74% (295/398) Updating files: 75% (299/398) Updating files: 76% (303/398) Updating files: 77% (307/398) Updating files: 78% (311/398) Updating files: 79% (315/398) Updating files: 80% (319/398) Updating files: 81% (323/398) Updating files: 82% (327/398) Updating files: 83% (331/398) Updating files: 84% (335/398) Updating files: 85% (339/398) Updating files: 86% (343/398) Updating files: 87% (347/398) Updating files: 88% (351/398) Updating files: 88% (353/398) Updating files: 89% (355/398) Updating files: 90% (359/398) Updating files: 91% (363/398) Updating files: 92% (367/398) Updating files: 93% (371/398) Updating files: 94% (375/398) Updating files: 95% (379/398) Updating files: 96% (383/398) Updating files: 97% (387/398) Updating files: 98% (391/398) Updating files: 99% (395/398) Updating files: 100% (398/398) Updating files: 100% (398/398), done. 2023-10-30T12:54:51.457 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 9019c6ce640 v0.31 2023-10-30T12:54:51.466 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.31 2023-10-30T12:54:51.740 INFO:tasks.workunit.client.0.smithi125.stdout:v0.32 2023-10-30T12:54:51.740 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 32 -eq 48 ] 2023-10-30T12:54:51.740 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.32 2023-10-30T12:54:51.741 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.32 2023-10-30T12:54:51.741 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.32 2023-10-30T12:54:54.757 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 51% (100/195) Updating files: 52% (102/195) Updating files: 53% (104/195) Updating files: 54% (106/195) Updating files: 55% (108/195) Updating files: 56% (110/195) Updating files: 57% (112/195) Updating files: 58% (114/195) Updating files: 59% (116/195) Updating files: 60% (117/195) Updating files: 61% (119/195) Updating files: 62% (121/195) Updating files: 63% (123/195) Updating files: 64% (125/195) Updating files: 65% (127/195) Updating files: 66% (129/195) Updating files: 67% (131/195) Updating files: 68% (133/195) Updating files: 69% (135/195) Updating files: 70% (137/195) Updating files: 71% (139/195) Updating files: 72% (141/195) Updating files: 73% (143/195) Updating files: 74% (145/195) Updating files: 75% (147/195) Updating files: 76% (149/195) Updating files: 77% (151/195) Updating files: 78% (153/195) Updating files: 79% (155/195) Updating files: 80% (156/195) Updating files: 81% (158/195) Updating files: 82% (160/195) Updating files: 83% (162/195) Updating files: 84% (164/195) Updating files: 85% (166/195) Updating files: 86% (168/195) Updating files: 87% (170/195) Updating files: 88% (172/195) Updating files: 89% (174/195) Updating files: 90% (176/195) Updating files: 91% (178/195) Updating files: 92% (180/195) Updating files: 93% (182/195) Updating files: 94% (184/195) Updating files: 95% (186/195) Updating files: 96% (188/195) Updating files: 97% (190/195) Updating files: 98% (192/195) Updating files: 98% (193/195) Updating files: 99% (194/195) Updating files: 100% (195/195) Updating files: 100% (195/195), done. 2023-10-30T12:54:54.798 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at c08d08baa6a v0.32 2023-10-30T12:54:54.802 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.32 2023-10-30T12:54:54.838 INFO:tasks.workunit.client.0.smithi125.stdout:v0.33 2023-10-30T12:54:54.839 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 33 -eq 48 ] 2023-10-30T12:54:54.839 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.33 2023-10-30T12:54:54.839 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.33 2023-10-30T12:54:54.839 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.33 2023-10-30T12:55:01.334 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:01.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.482+0000 7fcc4c3a1640 1 Processor -- start 2023-10-30T12:55:01.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.482+0000 7fcc4c3a1640 1 -- start start 2023-10-30T12:55:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4c3a1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fcc440c6b80 0x7fcc440bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4c3a1640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc440c0a50 con 0x7fcc440c6f60 2023-10-30T12:55:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4c3a1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4406f6e0 con 0x7fcc4413f4a0 2023-10-30T12:55:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4c3a1640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4406f9c0 con 0x7fcc440c6b80 2023-10-30T12:55:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4993c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcc440c6f60 0x7fcc4413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37694/0 (socket says 172.21.15.125:37694) 2023-10-30T12:55:01.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4993c640 1 -- 172.21.15.125:0/433719056 learned_addr learned my addr 172.21.15.125:0/433719056 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:55:01.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4a13d640 1 --2- 172.21.15.125:0/433719056 >> v2:172.21.15.136:3300/0 conn(0x7fcc440c6b80 0x7fcc440bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:01.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 -- 172.21.15.125:0/433719056 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192682600 0 0) 0x7fcc440c0a50 con 0x7fcc440c6f60 2023-10-30T12:55:01.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 -- 172.21.15.125:0/433719056 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc30003530 con 0x7fcc440c6f60 2023-10-30T12:55:01.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 -- 172.21.15.125:0/433719056 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1481241350 0 0) 0x7fcc30003530 con 0x7fcc440c6f60 2023-10-30T12:55:01.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 -- 172.21.15.125:0/433719056 >> v2:172.21.15.136:3300/0 conn(0x7fcc440c6b80 msgr2=0x7fcc440bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:01.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 --2- 172.21.15.125:0/433719056 >> v2:172.21.15.136:3300/0 conn(0x7fcc440c6b80 0x7fcc440bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 -- 172.21.15.125:0/433719056 >> v1:172.21.15.136:6789/0 conn(0x7fcc4413f4a0 legacy=0x7fcc441566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:01.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 -- 172.21.15.125:0/433719056 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4406fca0 con 0x7fcc440c6f60 2023-10-30T12:55:01.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 -- 172.21.15.125:0/433719056 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fcc34001680 con 0x7fcc440c6f60 2023-10-30T12:55:01.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 -- 172.21.15.125:0/433719056 >> v1:172.21.15.125:6789/0 conn(0x7fcc440c6f60 legacy=0x7fcc4413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:01.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 --2- 172.21.15.125:0/433719056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc30003980 0x7fcc30003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:01.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 --2- 172.21.15.125:0/433719056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc300047d0 0x7fcc30004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:01.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 --2- 172.21.15.125:0/433719056 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc30005190 0x7fcc3000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:01.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 -- 172.21.15.125:0/433719056 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4406f9c0 con 0x7fcc300047d0 2023-10-30T12:55:01.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 -- 172.21.15.125:0/433719056 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3000c030 con 0x7fcc30003980 2023-10-30T12:55:01.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4913b640 1 -- 172.21.15.125:0/433719056 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3000c1f0 con 0x7fcc30005190 2023-10-30T12:55:01.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4993c640 1 --2- 172.21.15.125:0/433719056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc300047d0 0x7fcc30004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:01.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4a13d640 1 --2- 172.21.15.125:0/433719056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc30003980 0x7fcc30003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:01.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4a93e640 1 --2- 172.21.15.125:0/433719056 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc30005190 0x7fcc3000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:01.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4993c640 1 -- 172.21.15.125:0/433719056 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc30005190 msgr2=0x7fcc3000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:01.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4993c640 1 --2- 172.21.15.125:0/433719056 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc30005190 0x7fcc3000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4993c640 1 -- 172.21.15.125:0/433719056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc30003980 msgr2=0x7fcc30003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:01.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4993c640 1 --2- 172.21.15.125:0/433719056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc30003980 0x7fcc30003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4993c640 1 -- 172.21.15.125:0/433719056 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc30003530 con 0x7fcc300047d0 2023-10-30T12:55:01.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.486+0000 7fcc4a93e640 1 -- 172.21.15.125:0/433719056 reap_dead start 2023-10-30T12:55:01.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.490+0000 7fcc4993c640 1 --2- 172.21.15.125:0/433719056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc300047d0 0x7fcc30004c50 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400c240 tx=0x7fcc3401f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210566e64f3815a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:01.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.490+0000 7fcc4913b640 1 -- 172.21.15.125:0/433719056 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34005ae0 con 0x7fcc300047d0 2023-10-30T12:55:01.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.490+0000 7fcc4913b640 1 -- 172.21.15.125:0/433719056 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc340ab070 con 0x7fcc300047d0 2023-10-30T12:55:01.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.490+0000 7fcc4913b640 1 -- 172.21.15.125:0/433719056 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34007070 con 0x7fcc300047d0 2023-10-30T12:55:01.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.490+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/433719056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc300047d0 msgr2=0x7fcc30004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:01.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.490+0000 7fcc4c3a1640 1 --2- 172.21.15.125:0/433719056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc300047d0 0x7fcc30004c50 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fcc3400c240 tx=0x7fcc3401f3b0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.490+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/433719056 shutdown_connections 2023-10-30T12:55:01.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.490+0000 7fcc4c3a1640 1 --2- 172.21.15.125:0/433719056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc300047d0 0x7fcc30004c50 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.490+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/433719056 >> 172.21.15.125:0/433719056 conn(0x7fcc440c7910 msgr2=0x7fcc440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:01.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.490+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/433719056 shutdown_connections 2023-10-30T12:55:01.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.490+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/433719056 wait complete. 2023-10-30T12:55:01.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.490+0000 7fcc4c3a1640 1 Processor -- start 2023-10-30T12:55:01.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4c3a1640 1 -- start start 2023-10-30T12:55:01.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4c3a1640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc300047d0 0x7fcc4413b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:01.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4c3a1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc441f0890 0x7fcc441ded20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:01.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4c3a1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc440c9190 0x7fcc440c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:01.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4c3a1640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc441390f0 con 0x7fcc441f0890 2023-10-30T12:55:01.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4c3a1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4413b940 con 0x7fcc440c9190 2023-10-30T12:55:01.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4c3a1640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc440c9b80 con 0x7fcc300047d0 2023-10-30T12:55:01.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4993c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc441f0890 0x7fcc441ded20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:01.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4993c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc441f0890 0x7fcc441ded20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37688/0 (socket says 172.21.15.125:37688) 2023-10-30T12:55:01.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4993c640 1 -- 172.21.15.125:0/3210385924 learned_addr learned my addr 172.21.15.125:0/3210385924 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:01.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4a13d640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc300047d0 0x7fcc4413b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:01.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4a93e640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc440c9190 0x7fcc440c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:01.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4993c640 1 -- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc300047d0 msgr2=0x7fcc4413b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:01.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4993c640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc300047d0 0x7fcc4413b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4993c640 1 -- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc440c9190 msgr2=0x7fcc440c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:01.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4993c640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc440c9190 0x7fcc440c9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4993c640 1 -- 172.21.15.125:0/3210385924 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc441ecbd0 con 0x7fcc441f0890 2023-10-30T12:55:01.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4a93e640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc440c9190 0x7fcc440c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:01.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4993c640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc441f0890 0x7fcc441ded20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fcc34001940 tx=0x7fcc30005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:01.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc3affd640 1 -- 172.21.15.125:0/3210385924 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34001640 con 0x7fcc441f0890 2023-10-30T12:55:01.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc3affd640 1 -- 172.21.15.125:0/3210385924 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc34007070 con 0x7fcc441f0890 2023-10-30T12:55:01.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc3affd640 1 -- 172.21.15.125:0/3210385924 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34007420 con 0x7fcc441f0890 2023-10-30T12:55:01.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/3210385924 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc441eceb0 con 0x7fcc441f0890 2023-10-30T12:55:01.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/3210385924 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc441ed470 con 0x7fcc441f0890 2023-10-30T12:55:01.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.494+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/3210385924 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc4404e170 con 0x7fcc441f0890 2023-10-30T12:55:01.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.498+0000 7fcc3affd640 1 -- 172.21.15.125:0/3210385924 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcc340b9030 con 0x7fcc441f0890 2023-10-30T12:55:01.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.498+0000 7fcc3affd640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc280664d0 0x7fcc28068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:01.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.498+0000 7fcc3affd640 1 -- 172.21.15.125:0/3210385924 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7fcc340b7070 con 0x7fcc441f0890 2023-10-30T12:55:01.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.498+0000 7fcc4a13d640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc280664d0 0x7fcc28068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:01.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.498+0000 7fcc4a13d640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc280664d0 0x7fcc28068990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcc40009c60 tx=0x7fcc40009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:01.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.502+0000 7fcc3affd640 1 -- 172.21.15.125:0/3210385924 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcc340ee1f0 con 0x7fcc441f0890 2023-10-30T12:55:01.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.682+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/3210385924 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc440b0be0 con 0x7fcc441f0890 2023-10-30T12:55:01.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.682+0000 7fcc3affd640 1 -- 172.21.15.125:0/3210385924 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fcc340c1020 con 0x7fcc441f0890 2023-10-30T12:55:01.685 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:01.685 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7113,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:59.039508+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:active","state_seq":19,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:01.685 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 123 2023-10-30T12:55:01.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.686+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc280664d0 msgr2=0x7fcc28068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:01.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.686+0000 7fcc4c3a1640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc280664d0 0x7fcc28068990 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcc40009c60 tx=0x7fcc40009190 comp rx=0 tx=0).stop 2023-10-30T12:55:01.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.686+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/3210385924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc441f0890 msgr2=0x7fcc441ded20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:01.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.686+0000 7fcc4c3a1640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc441f0890 0x7fcc441ded20 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fcc34001940 tx=0x7fcc30005800 comp rx=0 tx=0).stop 2023-10-30T12:55:01.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.686+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/3210385924 shutdown_connections 2023-10-30T12:55:01.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.686+0000 7fcc4c3a1640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc280664d0 0x7fcc28068990 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.686+0000 7fcc4c3a1640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc440c9190 0x7fcc440c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.686+0000 7fcc4c3a1640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc441f0890 0x7fcc441ded20 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.686+0000 7fcc4c3a1640 1 --2- 172.21.15.125:0/3210385924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc300047d0 0x7fcc4413b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.686+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/3210385924 >> 172.21.15.125:0/3210385924 conn(0x7fcc440c7910 msgr2=0x7fcc4413ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:01.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.686+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/3210385924 shutdown_connections 2023-10-30T12:55:01.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.686+0000 7fcc4c3a1640 1 -- 172.21.15.125:0/3210385924 wait complete. 2023-10-30T12:55:01.705 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T12:55:01.705 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T12:55:01.706 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:55:01.733 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T12:55:01.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:01.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe554b9d640 1 Processor -- start 2023-10-30T12:55:01.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe554b9d640 1 -- start start 2023-10-30T12:55:01.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe554b9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe5501538a0 0x7fe55015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:01.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe554b9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe5500cf160 0x7fe5500cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:01.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe554b9d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5500cb800 con 0x7fe5500ca0e0 2023-10-30T12:55:01.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe554b9d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe55015dc30 con 0x7fe5501538a0 2023-10-30T12:55:01.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe554b9d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5500cb330 con 0x7fe5500cf160 2023-10-30T12:55:01.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe54e59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe5501538a0 0x7fe55015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:01.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe54e59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe5501538a0 0x7fe55015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37694/0 (socket says 172.21.15.125:37694) 2023-10-30T12:55:01.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe54e59c640 1 -- 172.21.15.125:0/2677354838 learned_addr learned my addr 172.21.15.125:0/2677354838 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:01.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe54dd9b640 1 --1- 172.21.15.125:0/2677354838 >> v1:172.21.15.125:6789/0 conn(0x7fe5500ca0e0 0x7fe55015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37696/0 (socket says 172.21.15.125:37696) 2023-10-30T12:55:01.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe54ed9d640 1 --2- 172.21.15.125:0/2677354838 >> v2:172.21.15.136:3300/0 conn(0x7fe5500cf160 0x7fe5500cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:01.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe54e59c640 1 -- 172.21.15.125:0/2677354838 >> v2:172.21.15.136:3300/0 conn(0x7fe5500cf160 msgr2=0x7fe5500cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:01.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe54e59c640 1 --2- 172.21.15.125:0/2677354838 >> v2:172.21.15.136:3300/0 conn(0x7fe5500cf160 0x7fe5500cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe54e59c640 1 -- 172.21.15.125:0/2677354838 >> v1:172.21.15.125:6789/0 conn(0x7fe5500ca0e0 legacy=0x7fe55015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:01.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.914+0000 7fe54e59c640 1 -- 172.21.15.125:0/2677354838 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5500cad70 con 0x7fe5501538a0 2023-10-30T12:55:01.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.918+0000 7fe54e59c640 1 --2- 172.21.15.125:0/2677354838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5501538a0 0x7fe55015d6f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fe5500b5330 tx=0x7fe54001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d631aaa6d4b23579 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:01.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.918+0000 7fe54d59a640 1 -- 172.21.15.125:0/2677354838 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5400a89f0 con 0x7fe5501538a0 2023-10-30T12:55:01.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.918+0000 7fe54d59a640 1 -- 172.21.15.125:0/2677354838 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5400a8b50 con 0x7fe5501538a0 2023-10-30T12:55:01.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.918+0000 7fe54d59a640 1 -- 172.21.15.125:0/2677354838 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5400b0440 con 0x7fe5501538a0 2023-10-30T12:55:01.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.918+0000 7fe554b9d640 1 -- 172.21.15.125:0/2677354838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5501538a0 msgr2=0x7fe55015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:01.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.918+0000 7fe554b9d640 1 --2- 172.21.15.125:0/2677354838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5501538a0 0x7fe55015d6f0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fe5500b5330 tx=0x7fe54001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.918+0000 7fe554b9d640 1 -- 172.21.15.125:0/2677354838 shutdown_connections 2023-10-30T12:55:01.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.918+0000 7fe554b9d640 1 --2- 172.21.15.125:0/2677354838 >> v2:172.21.15.136:3300/0 conn(0x7fe5500cf160 0x7fe5500cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.918+0000 7fe554b9d640 1 --2- 172.21.15.125:0/2677354838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5501538a0 0x7fe55015d6f0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.918+0000 7fe554b9d640 1 -- 172.21.15.125:0/2677354838 >> 172.21.15.125:0/2677354838 conn(0x7fe55006bfe0 msgr2=0x7fe55006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:01.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.918+0000 7fe554b9d640 1 -- 172.21.15.125:0/2677354838 shutdown_connections 2023-10-30T12:55:01.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.918+0000 7fe554b9d640 1 -- 172.21.15.125:0/2677354838 wait complete. 2023-10-30T12:55:01.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe554b9d640 1 Processor -- start 2023-10-30T12:55:01.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe554b9d640 1 -- start start 2023-10-30T12:55:01.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe554b9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5500ca0e0 0x7fe550158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:01.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe554b9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5500cf160 0x7fe5501e40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:01.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe554b9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5501538a0 0x7fe5501e4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:01.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe554b9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55015d9f0 con 0x7fe5500cf160 2023-10-30T12:55:01.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe554b9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55015d710 con 0x7fe5500ca0e0 2023-10-30T12:55:01.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe554b9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5501e3600 con 0x7fe5501538a0 2023-10-30T12:55:01.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe54dd9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5500cf160 0x7fe5501e40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:01.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe54dd9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5500cf160 0x7fe5501e40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37698/0 (socket says 172.21.15.125:37698) 2023-10-30T12:55:01.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe54dd9b640 1 -- 172.21.15.125:0/3571997946 learned_addr learned my addr 172.21.15.125:0/3571997946 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:01.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe54e59c640 1 --2- 172.21.15.125:0/3571997946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5500ca0e0 0x7fe550158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:01.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe54ed9d640 1 --2- 172.21.15.125:0/3571997946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5501538a0 0x7fe5501e4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:01.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe54dd9b640 1 -- 172.21.15.125:0/3571997946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5501538a0 msgr2=0x7fe5501e4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:01.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe54dd9b640 1 --2- 172.21.15.125:0/3571997946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5501538a0 0x7fe5501e4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe54dd9b640 1 -- 172.21.15.125:0/3571997946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5500ca0e0 msgr2=0x7fe550158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:01.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe54dd9b640 1 --2- 172.21.15.125:0/3571997946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5500ca0e0 0x7fe550158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:01.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe54dd9b640 1 -- 172.21.15.125:0/3571997946 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5400094d0 con 0x7fe5500cf160 2023-10-30T12:55:01.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe54dd9b640 1 --2- 172.21.15.125:0/3571997946 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5500cf160 0x7fe5501e40e0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fe54400c430 tx=0x7fe54400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:01.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe53effd640 1 -- 172.21.15.125:0/3571997946 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5440074e0 con 0x7fe5500cf160 2023-10-30T12:55:01.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe53effd640 1 -- 172.21.15.125:0/3571997946 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe544007640 con 0x7fe5500cf160 2023-10-30T12:55:01.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe554b9d640 1 -- 172.21.15.125:0/3571997946 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5501e38e0 con 0x7fe5500cf160 2023-10-30T12:55:01.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe53effd640 1 -- 172.21.15.125:0/3571997946 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe544004a10 con 0x7fe5500cf160 2023-10-30T12:55:01.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe554b9d640 1 -- 172.21.15.125:0/3571997946 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5501595d0 con 0x7fe5500cf160 2023-10-30T12:55:01.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.922+0000 7fe554b9d640 1 -- 172.21.15.125:0/3571997946 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5500b2a50 con 0x7fe5500cf160 2023-10-30T12:55:01.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.926+0000 7fe53effd640 1 -- 172.21.15.125:0/3571997946 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe544004b70 con 0x7fe5500cf160 2023-10-30T12:55:01.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.926+0000 7fe53effd640 1 --2- 172.21.15.125:0/3571997946 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5300664a0 0x7fe530068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:01.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.926+0000 7fe53effd640 1 -- 172.21.15.125:0/3571997946 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7fe544083cc0 con 0x7fe5500cf160 2023-10-30T12:55:01.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.926+0000 7fe54e59c640 1 --2- 172.21.15.125:0/3571997946 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5300664a0 0x7fe530068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:01.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.926+0000 7fe54e59c640 1 --2- 172.21.15.125:0/3571997946 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5300664a0 0x7fe530068960 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe5400b2000 tx=0x7fe540009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:01.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:01.930+0000 7fe53effd640 1 -- 172.21.15.125:0/3571997946 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe54408a080 con 0x7fe5500cf160 2023-10-30T12:55:02.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:02.110+0000 7fe554b9d640 1 -- 172.21.15.125:0/3571997946 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe55004e170 con 0x7fe5500cf160 2023-10-30T12:55:02.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:02.110+0000 7fe53effd640 1 -- 172.21.15.125:0/3571997946 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fe544050c70 con 0x7fe5500cf160 2023-10-30T12:55:02.112 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:02.113 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7113,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:59.039508+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:active","state_seq":19,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:02.114 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 123 2023-10-30T12:55:02.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:02.110+0000 7fe554b9d640 1 -- 172.21.15.125:0/3571997946 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5300664a0 msgr2=0x7fe530068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:02.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:02.110+0000 7fe554b9d640 1 --2- 172.21.15.125:0/3571997946 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5300664a0 0x7fe530068960 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe5400b2000 tx=0x7fe540009040 comp rx=0 tx=0).stop 2023-10-30T12:55:02.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:02.110+0000 7fe554b9d640 1 -- 172.21.15.125:0/3571997946 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5500cf160 msgr2=0x7fe5501e40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:02.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:02.110+0000 7fe554b9d640 1 --2- 172.21.15.125:0/3571997946 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5500cf160 0x7fe5501e40e0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fe54400c430 tx=0x7fe54400c8f0 comp rx=0 tx=0).stop 2023-10-30T12:55:02.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:02.110+0000 7fe554b9d640 1 -- 172.21.15.125:0/3571997946 shutdown_connections 2023-10-30T12:55:02.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:02.110+0000 7fe554b9d640 1 --2- 172.21.15.125:0/3571997946 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5300664a0 0x7fe530068960 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:02.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:02.110+0000 7fe554b9d640 1 --2- 172.21.15.125:0/3571997946 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5501538a0 0x7fe5501e4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:02.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:02.110+0000 7fe554b9d640 1 --2- 172.21.15.125:0/3571997946 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5500cf160 0x7fe5501e40e0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:02.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:02.110+0000 7fe554b9d640 1 --2- 172.21.15.125:0/3571997946 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5500ca0e0 0x7fe550158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:02.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:02.110+0000 7fe554b9d640 1 -- 172.21.15.125:0/3571997946 >> 172.21.15.125:0/3571997946 conn(0x7fe55006bfe0 msgr2=0x7fe550065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:02.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:02.110+0000 7fe554b9d640 1 -- 172.21.15.125:0/3571997946 shutdown_connections 2023-10-30T12:55:02.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:02.114+0000 7fe554b9d640 1 -- 172.21.15.125:0/3571997946 wait complete. 2023-10-30T12:55:02.134 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:55:04.134 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:04.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.278+0000 7fc34bdd3640 1 Processor -- start 2023-10-30T12:55:04.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34bdd3640 1 -- start start 2023-10-30T12:55:04.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34bdd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3441538a0 0x7fc34415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:04.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34bdd3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc3440ca0e0 0x7fc34415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:04.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34bdd3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc3440cf160 0x7fc3440cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:04.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34bdd3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3440cb800 con 0x7fc3440ca0e0 2023-10-30T12:55:04.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34bdd3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc34415dc30 con 0x7fc3440cf160 2023-10-30T12:55:04.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34bdd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3440cb330 con 0x7fc3441538a0 2023-10-30T12:55:04.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34936e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc3440ca0e0 0x7fc34415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:04.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34936e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc3440ca0e0 0x7fc34415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37714/0 (socket says 172.21.15.125:37714) 2023-10-30T12:55:04.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34936e640 1 -- 172.21.15.125:0/3513094041 learned_addr learned my addr 172.21.15.125:0/3513094041 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:04.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc349b6f640 1 --2- 172.21.15.125:0/3513094041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3441538a0 0x7fc34415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:04.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34a370640 1 --2- 172.21.15.125:0/3513094041 >> v2:172.21.15.136:3300/0 conn(0x7fc3440cf160 0x7fc3440cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:04.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34936e640 1 -- 172.21.15.125:0/3513094041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3441538a0 msgr2=0x7fc34415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:04.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34936e640 1 --2- 172.21.15.125:0/3513094041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3441538a0 0x7fc34415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:04.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34936e640 1 -- 172.21.15.125:0/3513094041 >> v2:172.21.15.136:3300/0 conn(0x7fc3440cf160 msgr2=0x7fc3440cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:04.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34936e640 1 --2- 172.21.15.125:0/3513094041 >> v2:172.21.15.136:3300/0 conn(0x7fc3440cf160 0x7fc3440cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:04.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc34936e640 1 -- 172.21.15.125:0/3513094041 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3440cad70 con 0x7fc3440ca0e0 2023-10-30T12:55:04.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.282+0000 7fc349b6f640 1 --2- 172.21.15.125:0/3513094041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3441538a0 0x7fc34415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:04.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc34936e640 1 --2- 172.21.15.125:0/3513094041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3440ca0e0 0x7fc34415a090 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fc34406cb00 tx=0x7fc33401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd2566c9d06be399 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:04.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc348b6d640 1 -- 172.21.15.125:0/3513094041 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3340a89b0 con 0x7fc3440ca0e0 2023-10-30T12:55:04.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc348b6d640 1 -- 172.21.15.125:0/3513094041 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3340a8b10 con 0x7fc3440ca0e0 2023-10-30T12:55:04.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc348b6d640 1 -- 172.21.15.125:0/3513094041 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3340a8ea0 con 0x7fc3440ca0e0 2023-10-30T12:55:04.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc34bdd3640 1 -- 172.21.15.125:0/3513094041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3440ca0e0 msgr2=0x7fc34415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:04.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc34bdd3640 1 --2- 172.21.15.125:0/3513094041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3440ca0e0 0x7fc34415a090 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fc34406cb00 tx=0x7fc33401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:55:04.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc34bdd3640 1 -- 172.21.15.125:0/3513094041 shutdown_connections 2023-10-30T12:55:04.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc34bdd3640 1 --2- 172.21.15.125:0/3513094041 >> v2:172.21.15.136:3300/0 conn(0x7fc3440cf160 0x7fc3440cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:04.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc34bdd3640 1 --2- 172.21.15.125:0/3513094041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3440ca0e0 0x7fc34415a090 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:04.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc34bdd3640 1 --2- 172.21.15.125:0/3513094041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3441538a0 0x7fc34415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:04.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc34bdd3640 1 -- 172.21.15.125:0/3513094041 >> 172.21.15.125:0/3513094041 conn(0x7fc34406bfe0 msgr2=0x7fc34406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:04.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc34bdd3640 1 -- 172.21.15.125:0/3513094041 shutdown_connections 2023-10-30T12:55:04.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc34bdd3640 1 -- 172.21.15.125:0/3513094041 wait complete. 2023-10-30T12:55:04.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc34bdd3640 1 Processor -- start 2023-10-30T12:55:04.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.286+0000 7fc34bdd3640 1 -- start start 2023-10-30T12:55:04.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc34bdd3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3440ca0e0 0x7fc34415d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:04.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc34bdd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3440cf160 0x7fc34414a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:04.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc34bdd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3441538a0 0x7fc3441f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:04.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc34bdd3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3441f2fd0 con 0x7fc3440ca0e0 2023-10-30T12:55:04.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc34bdd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34415ad70 con 0x7fc3441538a0 2023-10-30T12:55:04.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc34bdd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34414bab0 con 0x7fc3440cf160 2023-10-30T12:55:04.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc349b6f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3440ca0e0 0x7fc34415d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:04.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc349b6f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3440ca0e0 0x7fc34415d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37730/0 (socket says 172.21.15.125:37730) 2023-10-30T12:55:04.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc349b6f640 1 -- 172.21.15.125:0/1287453176 learned_addr learned my addr 172.21.15.125:0/1287453176 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:04.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc34936e640 1 --2- 172.21.15.125:0/1287453176 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3440cf160 0x7fc34414a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc34a370640 1 --2- 172.21.15.125:0/1287453176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3441538a0 0x7fc3441f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc349b6f640 1 -- 172.21.15.125:0/1287453176 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3440cf160 msgr2=0x7fc34414a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc349b6f640 1 --2- 172.21.15.125:0/1287453176 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3440cf160 0x7fc34414a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc349b6f640 1 -- 172.21.15.125:0/1287453176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3441538a0 msgr2=0x7fc3441f33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc349b6f640 1 --2- 172.21.15.125:0/1287453176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3441538a0 0x7fc3441f33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:04.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc349b6f640 1 -- 172.21.15.125:0/1287453176 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3340094d0 con 0x7fc3440ca0e0 2023-10-30T12:55:04.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc349b6f640 1 --2- 172.21.15.125:0/1287453176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3440ca0e0 0x7fc34415d410 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fc34000d830 tx=0x7fc34000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:04.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc33a7fc640 1 -- 172.21.15.125:0/1287453176 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3400096c0 con 0x7fc3440ca0e0 2023-10-30T12:55:04.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc33a7fc640 1 -- 172.21.15.125:0/1287453176 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc340009820 con 0x7fc3440ca0e0 2023-10-30T12:55:04.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc34bdd3640 1 -- 172.21.15.125:0/1287453176 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3440ce130 con 0x7fc3440ca0e0 2023-10-30T12:55:04.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc33a7fc640 1 -- 172.21.15.125:0/1287453176 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc340009bd0 con 0x7fc3440ca0e0 2023-10-30T12:55:04.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc34bdd3640 1 -- 172.21.15.125:0/1287453176 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3440ce380 con 0x7fc3440ca0e0 2023-10-30T12:55:04.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc34bdd3640 1 -- 172.21.15.125:0/1287453176 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3440b2a50 con 0x7fc3440ca0e0 2023-10-30T12:55:04.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.290+0000 7fc33a7fc640 1 -- 172.21.15.125:0/1287453176 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc340014410 con 0x7fc3440ca0e0 2023-10-30T12:55:04.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.294+0000 7fc33a7fc640 1 --2- 172.21.15.125:0/1287453176 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc328066520 0x7fc3280689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:04.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.294+0000 7fc33a7fc640 1 -- 172.21.15.125:0/1287453176 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7fc340084c50 con 0x7fc3440ca0e0 2023-10-30T12:55:04.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.294+0000 7fc34936e640 1 --2- 172.21.15.125:0/1287453176 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc328066520 0x7fc3280689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:04.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.294+0000 7fc34936e640 1 --2- 172.21.15.125:0/1287453176 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc328066520 0x7fc3280689e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc3340b2000 tx=0x7fc334008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:04.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.298+0000 7fc33a7fc640 1 -- 172.21.15.125:0/1287453176 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc340051bf0 con 0x7fc3440ca0e0 2023-10-30T12:55:04.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.478+0000 7fc34bdd3640 1 -- 172.21.15.125:0/1287453176 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc34404e170 con 0x7fc3440ca0e0 2023-10-30T12:55:04.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.478+0000 7fc33a7fc640 1 -- 172.21.15.125:0/1287453176 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fc340055210 con 0x7fc3440ca0e0 2023-10-30T12:55:04.481 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:04.481 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7113,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:59.039508+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:active","state_seq":19,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:04.482 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 123 2023-10-30T12:55:04.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.482+0000 7fc34bdd3640 1 -- 172.21.15.125:0/1287453176 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc328066520 msgr2=0x7fc3280689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:04.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.482+0000 7fc34bdd3640 1 --2- 172.21.15.125:0/1287453176 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc328066520 0x7fc3280689e0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fc3340b2000 tx=0x7fc334008040 comp rx=0 tx=0).stop 2023-10-30T12:55:04.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.482+0000 7fc34bdd3640 1 -- 172.21.15.125:0/1287453176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3440ca0e0 msgr2=0x7fc34415d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:04.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.482+0000 7fc34bdd3640 1 --2- 172.21.15.125:0/1287453176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3440ca0e0 0x7fc34415d410 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7fc34000d830 tx=0x7fc34000dcf0 comp rx=0 tx=0).stop 2023-10-30T12:55:04.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.482+0000 7fc34bdd3640 1 -- 172.21.15.125:0/1287453176 shutdown_connections 2023-10-30T12:55:04.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.482+0000 7fc34bdd3640 1 --2- 172.21.15.125:0/1287453176 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc328066520 0x7fc3280689e0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:04.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.482+0000 7fc34bdd3640 1 --2- 172.21.15.125:0/1287453176 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc3441538a0 0x7fc3441f33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:04.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.482+0000 7fc34bdd3640 1 --2- 172.21.15.125:0/1287453176 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc3440cf160 0x7fc34414a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:04.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.482+0000 7fc34bdd3640 1 --2- 172.21.15.125:0/1287453176 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc3440ca0e0 0x7fc34415d410 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:04.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.482+0000 7fc34bdd3640 1 -- 172.21.15.125:0/1287453176 >> 172.21.15.125:0/1287453176 conn(0x7fc34406bfe0 msgr2=0x7fc3441483a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:04.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.482+0000 7fc34bdd3640 1 -- 172.21.15.125:0/1287453176 shutdown_connections 2023-10-30T12:55:04.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:04.482+0000 7fc34bdd3640 1 -- 172.21.15.125:0/1287453176 wait complete. 2023-10-30T12:55:04.502 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:55:06.503 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:06.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe667cea640 1 Processor -- start 2023-10-30T12:55:06.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe667cea640 1 -- start start 2023-10-30T12:55:06.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe667cea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe66006e470 0x7fe66006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:06.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe667cea640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe66006afa0 con 0x7fe6600c8490 2023-10-30T12:55:06.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe667cea640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe660154150 con 0x7fe660153d70 2023-10-30T12:55:06.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe667cea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe66006d780 con 0x7fe66006e470 2023-10-30T12:55:06.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe666287640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe6600c8490 0x7fe6600c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37700/0 (socket says 172.21.15.125:37700) 2023-10-30T12:55:06.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe666287640 1 -- 172.21.15.125:0/59488036 learned_addr learned my addr 172.21.15.125:0/59488036 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:55:06.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe665285640 1 --2- 172.21.15.125:0/59488036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe66006e470 0x7fe66006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:06.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758509680 0 0) 0x7fe66006afa0 con 0x7fe6600c8490 2023-10-30T12:55:06.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe64c003530 con 0x7fe6600c8490 2023-10-30T12:55:06.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472094326 0 0) 0x7fe660154150 con 0x7fe660153d70 2023-10-30T12:55:06.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe66006afa0 con 0x7fe660153d70 2023-10-30T12:55:06.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3817582640 0 0) 0x7fe64c003530 con 0x7fe6600c8490 2023-10-30T12:55:06.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe66006e470 msgr2=0x7fe66006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:06.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 --2- 172.21.15.125:0/59488036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe66006e470 0x7fe66006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:06.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 >> v1:172.21.15.136:6789/0 conn(0x7fe660153d70 legacy=0x7fe6600cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:06.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66006e030 con 0x7fe6600c8490 2023-10-30T12:55:06.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fe658001a00 con 0x7fe6600c8490 2023-10-30T12:55:06.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 >> v1:172.21.15.125:6789/0 conn(0x7fe6600c8490 legacy=0x7fe6600c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:06.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 --2- 172.21.15.125:0/59488036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe64c003b40 0x7fe64c003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:06.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 --2- 172.21.15.125:0/59488036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe66006e470 0x7fe64c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:06.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 --2- 172.21.15.125:0/59488036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe64c004d10 0x7fe64c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:06.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe66006d780 con 0x7fe64c003b40 2023-10-30T12:55:06.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c00ba40 con 0x7fe64c004d10 2023-10-30T12:55:06.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c00bc00 con 0x7fe66006e470 2023-10-30T12:55:06.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe665a86640 1 --2- 172.21.15.125:0/59488036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe64c003b40 0x7fe64c003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:06.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe665285640 1 --2- 172.21.15.125:0/59488036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe66006e470 0x7fe64c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:06.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe666287640 1 --2- 172.21.15.125:0/59488036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe64c004d10 0x7fe64c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:06.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe665a86640 1 -- 172.21.15.125:0/59488036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe66006e470 msgr2=0x7fe64c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:06.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe665a86640 1 --2- 172.21.15.125:0/59488036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe66006e470 0x7fe64c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:06.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe665a86640 1 -- 172.21.15.125:0/59488036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe64c004d10 msgr2=0x7fe64c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:06.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe665a86640 1 --2- 172.21.15.125:0/59488036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe64c004d10 0x7fe64c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:06.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe665a86640 1 -- 172.21.15.125:0/59488036 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe64c003530 con 0x7fe64c003b40 2023-10-30T12:55:06.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.650+0000 7fe666287640 1 --2- 172.21.15.125:0/59488036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe64c004d10 0x7fe64c005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:55:06.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.654+0000 7fe665a86640 1 --2- 172.21.15.125:0/59488036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe64c003b40 0x7fe64c003f40 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0018d0 tx=0x7fe65c01c530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9054127b3dc4a824 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:06.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.654+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c0052d0 con 0x7fe64c003b40 2023-10-30T12:55:06.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.654+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c005450 con 0x7fe64c003b40 2023-10-30T12:55:06.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.654+0000 7fe664a84640 1 -- 172.21.15.125:0/59488036 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c003840 con 0x7fe64c003b40 2023-10-30T12:55:06.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.654+0000 7fe667cea640 1 -- 172.21.15.125:0/59488036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe64c003b40 msgr2=0x7fe64c003f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:06.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.654+0000 7fe667cea640 1 --2- 172.21.15.125:0/59488036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe64c003b40 0x7fe64c003f40 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0018d0 tx=0x7fe65c01c530 comp rx=0 tx=0).stop 2023-10-30T12:55:06.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.654+0000 7fe666287640 1 -- 172.21.15.125:0/59488036 reap_dead start 2023-10-30T12:55:06.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.654+0000 7fe667cea640 1 -- 172.21.15.125:0/59488036 shutdown_connections 2023-10-30T12:55:06.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.654+0000 7fe667cea640 1 -- 172.21.15.125:0/59488036 >> 172.21.15.125:0/59488036 conn(0x7fe660071ea0 msgr2=0x7fe660065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:06.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.654+0000 7fe667cea640 1 -- 172.21.15.125:0/59488036 shutdown_connections 2023-10-30T12:55:06.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.654+0000 7fe667cea640 1 -- 172.21.15.125:0/59488036 wait complete. 2023-10-30T12:55:06.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe667cea640 1 Processor -- start 2023-10-30T12:55:06.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe667cea640 1 -- start start 2023-10-30T12:55:06.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe667cea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe64c003b40 0x7fe660157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:06.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe667cea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6600cbcc0 0x7fe660163b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:06.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe667cea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe660159790 0x7fe660159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:06.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe667cea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6600ccbe0 con 0x7fe64c003b40 2023-10-30T12:55:06.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe667cea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe660159300 con 0x7fe6600cbcc0 2023-10-30T12:55:06.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe667cea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe660158a40 con 0x7fe660159790 2023-10-30T12:55:06.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe665a86640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe64c003b40 0x7fe660157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:06.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe665a86640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe64c003b40 0x7fe660157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37750/0 (socket says 172.21.15.125:37750) 2023-10-30T12:55:06.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe665a86640 1 -- 172.21.15.125:0/2260176112 learned_addr learned my addr 172.21.15.125:0/2260176112 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:06.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe666287640 1 --2- 172.21.15.125:0/2260176112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe660159790 0x7fe660159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:06.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe665285640 1 --2- 172.21.15.125:0/2260176112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6600cbcc0 0x7fe660163b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:06.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe665a86640 1 -- 172.21.15.125:0/2260176112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe660159790 msgr2=0x7fe660159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:06.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe665a86640 1 --2- 172.21.15.125:0/2260176112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe660159790 0x7fe660159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:06.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe665a86640 1 -- 172.21.15.125:0/2260176112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6600cbcc0 msgr2=0x7fe660163b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:06.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe665a86640 1 --2- 172.21.15.125:0/2260176112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6600cbcc0 0x7fe660163b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:06.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe665a86640 1 -- 172.21.15.125:0/2260176112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe66015a150 con 0x7fe64c003b40 2023-10-30T12:55:06.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe665a86640 1 --2- 172.21.15.125:0/2260176112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe64c003b40 0x7fe660157d10 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0ae040 tx=0x7fe64c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:06.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe6567fc640 1 -- 172.21.15.125:0/2260176112 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c0050d0 con 0x7fe64c003b40 2023-10-30T12:55:06.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe6567fc640 1 -- 172.21.15.125:0/2260176112 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe65c005230 con 0x7fe64c003b40 2023-10-30T12:55:06.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe667cea640 1 -- 172.21.15.125:0/2260176112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6600cce10 con 0x7fe64c003b40 2023-10-30T12:55:06.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe6567fc640 1 -- 172.21.15.125:0/2260176112 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe65c006a20 con 0x7fe64c003b40 2023-10-30T12:55:06.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.658+0000 7fe667cea640 1 -- 172.21.15.125:0/2260176112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe660158cb0 con 0x7fe64c003b40 2023-10-30T12:55:06.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.662+0000 7fe667cea640 1 -- 172.21.15.125:0/2260176112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe66004e170 con 0x7fe64c003b40 2023-10-30T12:55:06.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.662+0000 7fe6567fc640 1 -- 172.21.15.125:0/2260176112 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe65c0b2040 con 0x7fe64c003b40 2023-10-30T12:55:06.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.662+0000 7fe6567fc640 1 --2- 172.21.15.125:0/2260176112 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6440664d0 0x7fe644068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:06.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.662+0000 7fe6567fc640 1 -- 172.21.15.125:0/2260176112 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7fe65c11dd90 con 0x7fe64c003b40 2023-10-30T12:55:06.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.662+0000 7fe665285640 1 --2- 172.21.15.125:0/2260176112 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6440664d0 0x7fe644068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:06.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.662+0000 7fe665285640 1 --2- 172.21.15.125:0/2260176112 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6440664d0 0x7fe644068990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe660154530 tx=0x7fe650008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:06.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.666+0000 7fe6567fc640 1 -- 172.21.15.125:0/2260176112 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe65c0eae80 con 0x7fe64c003b40 2023-10-30T12:55:06.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.846+0000 7fe667cea640 1 -- 172.21.15.125:0/2260176112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6600c6fa0 con 0x7fe64c003b40 2023-10-30T12:55:06.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.846+0000 7fe6567fc640 1 -- 172.21.15.125:0/2260176112 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fe65c0c0090 con 0x7fe64c003b40 2023-10-30T12:55:06.851 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:06.851 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7113,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:59.039508+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:active","state_seq":19,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:06.853 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 123 2023-10-30T12:55:06.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.850+0000 7fe667cea640 1 -- 172.21.15.125:0/2260176112 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6440664d0 msgr2=0x7fe644068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:06.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.850+0000 7fe667cea640 1 --2- 172.21.15.125:0/2260176112 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6440664d0 0x7fe644068990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe660154530 tx=0x7fe650008040 comp rx=0 tx=0).stop 2023-10-30T12:55:06.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.850+0000 7fe667cea640 1 -- 172.21.15.125:0/2260176112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe64c003b40 msgr2=0x7fe660157d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:06.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.850+0000 7fe667cea640 1 --2- 172.21.15.125:0/2260176112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe64c003b40 0x7fe660157d10 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0ae040 tx=0x7fe64c005800 comp rx=0 tx=0).stop 2023-10-30T12:55:06.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.850+0000 7fe667cea640 1 -- 172.21.15.125:0/2260176112 shutdown_connections 2023-10-30T12:55:06.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.850+0000 7fe667cea640 1 --2- 172.21.15.125:0/2260176112 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6440664d0 0x7fe644068990 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:06.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.850+0000 7fe667cea640 1 --2- 172.21.15.125:0/2260176112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe660159790 0x7fe660159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:06.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.850+0000 7fe667cea640 1 --2- 172.21.15.125:0/2260176112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6600cbcc0 0x7fe660163b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:06.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.850+0000 7fe667cea640 1 --2- 172.21.15.125:0/2260176112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe64c003b40 0x7fe660157d10 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:06.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.850+0000 7fe667cea640 1 -- 172.21.15.125:0/2260176112 >> 172.21.15.125:0/2260176112 conn(0x7fe660071ea0 msgr2=0x7fe6601522b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:06.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.850+0000 7fe667cea640 1 -- 172.21.15.125:0/2260176112 shutdown_connections 2023-10-30T12:55:06.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:06.850+0000 7fe667cea640 1 -- 172.21.15.125:0/2260176112 wait complete. 2023-10-30T12:55:06.871 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:55:08.872 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:09.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.018+0000 7f487f59e640 1 Processor -- start 2023-10-30T12:55:09.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.018+0000 7f487f59e640 1 -- start start 2023-10-30T12:55:09.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487f59e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f48801538f0 0x7f4880159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:09.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487f59e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f488015cd50 0x7f48800c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:09.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487f59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f48800cefa0 0x7f48800cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:09.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487f59e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488015d300 con 0x7f48801538f0 2023-10-30T12:55:09.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487f59e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488015d5e0 con 0x7f488015cd50 2023-10-30T12:55:09.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487f59e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488015d8c0 con 0x7f48800cefa0 2023-10-30T12:55:09.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487e59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f48801538f0 0x7f4880159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:09.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487e59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f48801538f0 0x7f4880159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37764/0 (socket says 172.21.15.125:37764) 2023-10-30T12:55:09.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487e59c640 1 -- 172.21.15.125:0/1048821751 learned_addr learned my addr 172.21.15.125:0/1048821751 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:09.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487dd9b640 1 --2- 172.21.15.125:0/1048821751 >> v2:172.21.15.136:3300/0 conn(0x7f488015cd50 0x7f48800c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:09.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487ed9d640 1 --2- 172.21.15.125:0/1048821751 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f48800cefa0 0x7f48800cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:09.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487e59c640 1 -- 172.21.15.125:0/1048821751 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f48800cefa0 msgr2=0x7f48800cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:09.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487e59c640 1 --2- 172.21.15.125:0/1048821751 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f48800cefa0 0x7f48800cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:09.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487e59c640 1 -- 172.21.15.125:0/1048821751 >> v2:172.21.15.136:3300/0 conn(0x7f488015cd50 msgr2=0x7f48800c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:09.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487e59c640 1 --2- 172.21.15.125:0/1048821751 >> v2:172.21.15.136:3300/0 conn(0x7f488015cd50 0x7f48800c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:09.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487e59c640 1 -- 172.21.15.125:0/1048821751 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488015dba0 con 0x7f48801538f0 2023-10-30T12:55:09.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.022+0000 7f487dd9b640 1 --2- 172.21.15.125:0/1048821751 >> v2:172.21.15.136:3300/0 conn(0x7f488015cd50 0x7f48800c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:09.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487e59c640 1 --2- 172.21.15.125:0/1048821751 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48801538f0 0x7f4880159fb0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f4880072e90 tx=0x7f487001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33fbd86d668d50c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:09.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487d59a640 1 -- 172.21.15.125:0/1048821751 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48700a89b0 con 0x7f48801538f0 2023-10-30T12:55:09.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487d59a640 1 -- 172.21.15.125:0/1048821751 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48700a8b10 con 0x7f48801538f0 2023-10-30T12:55:09.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487d59a640 1 -- 172.21.15.125:0/1048821751 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48700a8ea0 con 0x7f48801538f0 2023-10-30T12:55:09.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487f59e640 1 -- 172.21.15.125:0/1048821751 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48801538f0 msgr2=0x7f4880159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:09.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487f59e640 1 --2- 172.21.15.125:0/1048821751 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48801538f0 0x7f4880159fb0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f4880072e90 tx=0x7f487001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:55:09.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487f59e640 1 -- 172.21.15.125:0/1048821751 shutdown_connections 2023-10-30T12:55:09.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487f59e640 1 --2- 172.21.15.125:0/1048821751 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f48800cefa0 0x7f48800cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:09.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487f59e640 1 --2- 172.21.15.125:0/1048821751 >> v2:172.21.15.136:3300/0 conn(0x7f488015cd50 0x7f48800c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:09.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487f59e640 1 --2- 172.21.15.125:0/1048821751 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48801538f0 0x7f4880159fb0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:09.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487f59e640 1 -- 172.21.15.125:0/1048821751 >> 172.21.15.125:0/1048821751 conn(0x7f488006a640 msgr2=0x7f4880065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:09.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487f59e640 1 -- 172.21.15.125:0/1048821751 shutdown_connections 2023-10-30T12:55:09.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487f59e640 1 -- 172.21.15.125:0/1048821751 wait complete. 2023-10-30T12:55:09.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487f59e640 1 Processor -- start 2023-10-30T12:55:09.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.026+0000 7f487f59e640 1 -- start start 2023-10-30T12:55:09.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487f59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48800cefa0 0x7f488014dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:09.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487f59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48801538f0 0x7f488014e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:09.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487f59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488015cd50 0x7f4880147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:09.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487f59e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48801459e0 con 0x7f48800cefa0 2023-10-30T12:55:09.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487f59e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488014fbf0 con 0x7f48801538f0 2023-10-30T12:55:09.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487f59e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48801ef200 con 0x7f488015cd50 2023-10-30T12:55:09.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48800cefa0 0x7f488014dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:09.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48800cefa0 0x7f488014dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37766/0 (socket says 172.21.15.125:37766) 2023-10-30T12:55:09.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487e59c640 1 -- 172.21.15.125:0/1616044918 learned_addr learned my addr 172.21.15.125:0/1616044918 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:09.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487dd9b640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48801538f0 0x7f488014e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:09.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487ed9d640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488015cd50 0x7f4880147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:09.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487e59c640 1 -- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488015cd50 msgr2=0x7f4880147870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:09.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487e59c640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488015cd50 0x7f4880147870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:09.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487e59c640 1 -- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48801538f0 msgr2=0x7f488014e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:09.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487e59c640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48801538f0 0x7f488014e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:09.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487e59c640 1 -- 172.21.15.125:0/1616044918 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48700094d0 con 0x7f48800cefa0 2023-10-30T12:55:09.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487dd9b640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48801538f0 0x7f488014e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:09.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487ed9d640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488015cd50 0x7f4880147870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:09.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487e59c640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48800cefa0 0x7f488014dba0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f487001d3a0 tx=0x7f487001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:09.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f486effd640 1 -- 172.21.15.125:0/1616044918 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48700ba600 con 0x7f48800cefa0 2023-10-30T12:55:09.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f486effd640 1 -- 172.21.15.125:0/1616044918 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48700ba760 con 0x7f48800cefa0 2023-10-30T12:55:09.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487f59e640 1 -- 172.21.15.125:0/1616044918 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488014f630 con 0x7f48800cefa0 2023-10-30T12:55:09.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f486effd640 1 -- 172.21.15.125:0/1616044918 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48700bab10 con 0x7f48800cefa0 2023-10-30T12:55:09.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487f59e640 1 -- 172.21.15.125:0/1616044918 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48801ef4a0 con 0x7f48800cefa0 2023-10-30T12:55:09.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.030+0000 7f487f59e640 1 -- 172.21.15.125:0/1616044918 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48801ef060 con 0x7f48800cefa0 2023-10-30T12:55:09.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.034+0000 7f486effd640 1 -- 172.21.15.125:0/1616044918 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f48700bac70 con 0x7f48800cefa0 2023-10-30T12:55:09.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.034+0000 7f486effd640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f48600664a0 0x7f4860068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:09.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.034+0000 7f486effd640 1 -- 172.21.15.125:0/1616044918 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7f48701228b0 con 0x7f48800cefa0 2023-10-30T12:55:09.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.034+0000 7f487dd9b640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f48600664a0 0x7f4860068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:09.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.034+0000 7f487dd9b640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f48600664a0 0x7f4860068960 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f48800cfec0 tx=0x7f4874008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:09.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.038+0000 7f486effd640 1 -- 172.21.15.125:0/1616044918 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f48801ef060 con 0x7f48800cefa0 2023-10-30T12:55:09.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.218+0000 7f487f59e640 1 -- 172.21.15.125:0/1616044918 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48801ef060 con 0x7f48800cefa0 2023-10-30T12:55:09.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.218+0000 7f486effd640 1 -- 172.21.15.125:0/1616044918 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f48801ef060 con 0x7f48800cefa0 2023-10-30T12:55:09.222 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:09.223 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7113,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:59.039508+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:active","state_seq":19,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:09.224 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 123 2023-10-30T12:55:09.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.218+0000 7f487f59e640 1 -- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f48600664a0 msgr2=0x7f4860068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:09.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.218+0000 7f487f59e640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f48600664a0 0x7f4860068960 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f48800cfec0 tx=0x7f4874008040 comp rx=0 tx=0).stop 2023-10-30T12:55:09.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.218+0000 7f487f59e640 1 -- 172.21.15.125:0/1616044918 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48800cefa0 msgr2=0x7f488014dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:09.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.218+0000 7f487f59e640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48800cefa0 0x7f488014dba0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f487001d3a0 tx=0x7f487001ba90 comp rx=0 tx=0).stop 2023-10-30T12:55:09.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.218+0000 7f487f59e640 1 -- 172.21.15.125:0/1616044918 shutdown_connections 2023-10-30T12:55:09.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.218+0000 7f487f59e640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f48600664a0 0x7f4860068960 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:09.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.218+0000 7f487f59e640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f488015cd50 0x7f4880147870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:09.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.218+0000 7f487f59e640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48801538f0 0x7f488014e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:09.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.218+0000 7f487f59e640 1 --2- 172.21.15.125:0/1616044918 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48800cefa0 0x7f488014dba0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:09.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.218+0000 7f487f59e640 1 -- 172.21.15.125:0/1616044918 >> 172.21.15.125:0/1616044918 conn(0x7f488006a640 msgr2=0x7f48800ca080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:09.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.218+0000 7f487f59e640 1 -- 172.21.15.125:0/1616044918 shutdown_connections 2023-10-30T12:55:09.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:09.218+0000 7f487f59e640 1 -- 172.21.15.125:0/1616044918 wait complete. 2023-10-30T12:55:09.241 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:55:11.242 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:11.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.386+0000 7f5ff1f65640 1 Processor -- start 2023-10-30T12:55:11.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.386+0000 7f5ff1f65640 1 -- start start 2023-10-30T12:55:11.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.386+0000 7f5ff1f65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fec0c6b80 0x7f5fec0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.386+0000 7f5ff1f65640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec0c0a50 con 0x7f5fec0c6f60 2023-10-30T12:55:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.386+0000 7f5ff1f65640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec06f6e0 con 0x7f5fec13f4a0 2023-10-30T12:55:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.386+0000 7f5ff1f65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec06f9c0 con 0x7f5fec0c6b80 2023-10-30T12:55:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.386+0000 7f5feaffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5fec0c6f60 0x7f5fec13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54422/0 (socket says 172.21.15.125:54422) 2023-10-30T12:55:11.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.386+0000 7f5feaffd640 1 -- 172.21.15.125:0/65418730 learned_addr learned my addr 172.21.15.125:0/65418730 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:55:11.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5feb7fe640 1 --2- 172.21.15.125:0/65418730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fec0c6b80 0x7f5fec0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:11.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 -- 172.21.15.125:0/65418730 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508203379 0 0) 0x7f5fec0c0a50 con 0x7f5fec0c6f60 2023-10-30T12:55:11.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 -- 172.21.15.125:0/65418730 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd4005740 con 0x7f5fec0c6f60 2023-10-30T12:55:11.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 -- 172.21.15.125:0/65418730 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2632535393 0 0) 0x7f5fd4005740 con 0x7f5fec0c6f60 2023-10-30T12:55:11.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 -- 172.21.15.125:0/65418730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fec0c6b80 msgr2=0x7f5fec0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:11.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 --2- 172.21.15.125:0/65418730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fec0c6b80 0x7f5fec0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:11.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 -- 172.21.15.125:0/65418730 >> v1:172.21.15.136:6789/0 conn(0x7f5fec13f4a0 legacy=0x7f5fec1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:11.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 -- 172.21.15.125:0/65418730 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec06fca0 con 0x7f5fec0c6f60 2023-10-30T12:55:11.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 -- 172.21.15.125:0/65418730 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f5fe0001680 con 0x7f5fec0c6f60 2023-10-30T12:55:11.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 -- 172.21.15.125:0/65418730 >> v1:172.21.15.125:6789/0 conn(0x7f5fec0c6f60 legacy=0x7f5fec13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:11.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 --2- 172.21.15.125:0/65418730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fd4005bc0 0x7f5fd4005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:11.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 --2- 172.21.15.125:0/65418730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd40069e0 0x7f5fd4006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 --2- 172.21.15.125:0/65418730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fec0c6b80 0x7f5fd40073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 -- 172.21.15.125:0/65418730 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec06f9c0 con 0x7f5fd4005bc0 2023-10-30T12:55:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 -- 172.21.15.125:0/65418730 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd400dc50 con 0x7f5fd40069e0 2023-10-30T12:55:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5fea7fc640 1 -- 172.21.15.125:0/65418730 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd400de10 con 0x7f5fec0c6b80 2023-10-30T12:55:11.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5feb7fe640 1 --2- 172.21.15.125:0/65418730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fd4005bc0 0x7f5fd4005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:11.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5febfff640 1 --2- 172.21.15.125:0/65418730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fec0c6b80 0x7f5fd40073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:11.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5feaffd640 1 --2- 172.21.15.125:0/65418730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd40069e0 0x7f5fd4006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:11.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5feb7fe640 1 -- 172.21.15.125:0/65418730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fec0c6b80 msgr2=0x7f5fd40073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:11.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5feb7fe640 1 --2- 172.21.15.125:0/65418730 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fec0c6b80 0x7f5fd40073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:11.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5feb7fe640 1 -- 172.21.15.125:0/65418730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd40069e0 msgr2=0x7f5fd4006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:11.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5feb7fe640 1 --2- 172.21.15.125:0/65418730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd40069e0 0x7f5fd4006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:11.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5feb7fe640 1 -- 172.21.15.125:0/65418730 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd4005740 con 0x7f5fd4005bc0 2023-10-30T12:55:11.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.390+0000 7f5feaffd640 1 --2- 172.21.15.125:0/65418730 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd40069e0 0x7f5fd4006e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:55:11.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5feb7fe640 1 --2- 172.21.15.125:0/65418730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fd4005bc0 0x7f5fd4005fe0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc009ab0 tx=0x7f5fdc01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9941586fbd27491f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:11.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5fea7fc640 1 -- 172.21.15.125:0/65418730 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc0a6500 con 0x7f5fd4005bc0 2023-10-30T12:55:11.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5fea7fc640 1 -- 172.21.15.125:0/65418730 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fdc0a6660 con 0x7f5fd4005bc0 2023-10-30T12:55:11.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5fea7fc640 1 -- 172.21.15.125:0/65418730 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc0a6a10 con 0x7f5fd4005bc0 2023-10-30T12:55:11.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5ff1f65640 1 -- 172.21.15.125:0/65418730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fd4005bc0 msgr2=0x7f5fd4005fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:11.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5ff1f65640 1 --2- 172.21.15.125:0/65418730 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fd4005bc0 0x7f5fd4005fe0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc009ab0 tx=0x7f5fdc01ca10 comp rx=0 tx=0).stop 2023-10-30T12:55:11.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5febfff640 1 -- 172.21.15.125:0/65418730 reap_dead start 2023-10-30T12:55:11.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5ff1f65640 1 -- 172.21.15.125:0/65418730 shutdown_connections 2023-10-30T12:55:11.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5ff1f65640 1 -- 172.21.15.125:0/65418730 >> 172.21.15.125:0/65418730 conn(0x7f5fec0c7910 msgr2=0x7f5fec0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:11.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5ff1f65640 1 -- 172.21.15.125:0/65418730 shutdown_connections 2023-10-30T12:55:11.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5ff1f65640 1 -- 172.21.15.125:0/65418730 wait complete. 2023-10-30T12:55:11.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5ff1f65640 1 Processor -- start 2023-10-30T12:55:11.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5ff1f65640 1 -- start start 2023-10-30T12:55:11.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5ff1f65640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd4005bc0 0x7f5fec1594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:11.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5ff1f65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fec0adcb0 0x7f5fec0ae110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:11.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5ff1f65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fec13d590 0x7f5fec13da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:11.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5ff1f65640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0ae650 con 0x7f5fec0adcb0 2023-10-30T12:55:11.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5ff1f65640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec158770 con 0x7f5fd4005bc0 2023-10-30T12:55:11.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.394+0000 7f5ff1f65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec158a50 con 0x7f5fec13d590 2023-10-30T12:55:11.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5feaffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fec0adcb0 0x7f5fec0ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:11.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5feaffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fec0adcb0 0x7f5fec0ae110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36278/0 (socket says 172.21.15.125:36278) 2023-10-30T12:55:11.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5feaffd640 1 -- 172.21.15.125:0/4253661255 learned_addr learned my addr 172.21.15.125:0/4253661255 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:11.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5febfff640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fec13d590 0x7f5fec13da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:11.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5feb7fe640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd4005bc0 0x7f5fec1594c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:11.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5feaffd640 1 -- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fec13d590 msgr2=0x7f5fec13da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:11.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5feaffd640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fec13d590 0x7f5fec13da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:11.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5feaffd640 1 -- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd4005bc0 msgr2=0x7f5fec1594c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:11.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5feaffd640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd4005bc0 0x7f5fec1594c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:11.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5feaffd640 1 -- 172.21.15.125:0/4253661255 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec158d30 con 0x7f5fec0adcb0 2023-10-30T12:55:11.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5feb7fe640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd4005bc0 0x7f5fec1594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:11.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5feaffd640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fec0adcb0 0x7f5fec0ae110 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f5fe000d7e0 tx=0x7f5fe000dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:11.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5fd3fff640 1 -- 172.21.15.125:0/4253661255 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe00074e0 con 0x7f5fec0adcb0 2023-10-30T12:55:11.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5fd3fff640 1 -- 172.21.15.125:0/4253661255 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe0007640 con 0x7f5fec0adcb0 2023-10-30T12:55:11.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5fd3fff640 1 -- 172.21.15.125:0/4253661255 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe00049f0 con 0x7f5fec0adcb0 2023-10-30T12:55:11.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5ff1f65640 1 -- 172.21.15.125:0/4253661255 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fec159010 con 0x7f5fec0adcb0 2023-10-30T12:55:11.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5ff1f65640 1 -- 172.21.15.125:0/4253661255 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec0b04e0 con 0x7f5fec0adcb0 2023-10-30T12:55:11.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5ff1f65640 1 -- 172.21.15.125:0/4253661255 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fec04e170 con 0x7f5fec0adcb0 2023-10-30T12:55:11.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5fd3fff640 1 -- 172.21.15.125:0/4253661255 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5fe0004b50 con 0x7f5fec0adcb0 2023-10-30T12:55:11.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.398+0000 7f5fd3fff640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd80664a0 0x7f5fd8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:11.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.402+0000 7f5fd3fff640 1 -- 172.21.15.125:0/4253661255 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7f5fe0083c30 con 0x7f5fec0adcb0 2023-10-30T12:55:11.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.402+0000 7f5feb7fe640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd80664a0 0x7f5fd8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:11.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.402+0000 7f5feb7fe640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd80664a0 0x7f5fd8068960 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0af000 tx=0x7f5fdc0046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:11.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.406+0000 7f5fd3fff640 1 -- 172.21.15.125:0/4253661255 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5fe0050c00 con 0x7f5fec0adcb0 2023-10-30T12:55:11.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.586+0000 7f5ff1f65640 1 -- 172.21.15.125:0/4253661255 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fec0ad8d0 con 0x7f5fec0adcb0 2023-10-30T12:55:11.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.590+0000 7f5fd3fff640 1 -- 172.21.15.125:0/4253661255 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f5fe0019020 con 0x7f5fec0adcb0 2023-10-30T12:55:11.591 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:11.591 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7113,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:59.039508+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:active","state_seq":19,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:11.591 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 123 2023-10-30T12:55:11.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.590+0000 7f5ff1f65640 1 -- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd80664a0 msgr2=0x7f5fd8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:11.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.590+0000 7f5ff1f65640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd80664a0 0x7f5fd8068960 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc0af000 tx=0x7f5fdc0046d0 comp rx=0 tx=0).stop 2023-10-30T12:55:11.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.590+0000 7f5ff1f65640 1 -- 172.21.15.125:0/4253661255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fec0adcb0 msgr2=0x7f5fec0ae110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:11.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.590+0000 7f5ff1f65640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fec0adcb0 0x7f5fec0ae110 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f5fe000d7e0 tx=0x7f5fe000dca0 comp rx=0 tx=0).stop 2023-10-30T12:55:11.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.590+0000 7f5ff1f65640 1 -- 172.21.15.125:0/4253661255 shutdown_connections 2023-10-30T12:55:11.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.590+0000 7f5ff1f65640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5fd80664a0 0x7f5fd8068960 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:11.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.590+0000 7f5ff1f65640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5fec13d590 0x7f5fec13da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:11.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.590+0000 7f5ff1f65640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5fec0adcb0 0x7f5fec0ae110 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:11.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.590+0000 7f5ff1f65640 1 --2- 172.21.15.125:0/4253661255 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5fd4005bc0 0x7f5fec1594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:11.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.590+0000 7f5ff1f65640 1 -- 172.21.15.125:0/4253661255 >> 172.21.15.125:0/4253661255 conn(0x7f5fec0c7910 msgr2=0x7f5fec0b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:11.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.590+0000 7f5ff1f65640 1 -- 172.21.15.125:0/4253661255 shutdown_connections 2023-10-30T12:55:11.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:11.590+0000 7f5ff1f65640 1 -- 172.21.15.125:0/4253661255 wait complete. 2023-10-30T12:55:11.611 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:55:13.613 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:13.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.758+0000 7f94c019d640 1 Processor -- start 2023-10-30T12:55:13.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.758+0000 7f94c019d640 1 -- start start 2023-10-30T12:55:13.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.758+0000 7f94c019d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f94b80c6bd0 0x7f94b806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:13.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.758+0000 7f94c019d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94b8066cf0 con 0x7f94b8071110 2023-10-30T12:55:13.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.758+0000 7f94c019d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94b80c70d0 con 0x7f94b806f7a0 2023-10-30T12:55:13.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.758+0000 7f94c019d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b80700c0 con 0x7f94b80c6bd0 2023-10-30T12:55:13.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94be73a640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f94b8071110 0x7f94b8158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54426/0 (socket says 172.21.15.125:54426) 2023-10-30T12:55:13.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94be73a640 1 -- 172.21.15.125:0/2650724095 learned_addr learned my addr 172.21.15.125:0/2650724095 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:55:13.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bd738640 1 --2- 172.21.15.125:0/2650724095 >> v2:172.21.15.136:3300/0 conn(0x7f94b80c6bd0 0x7f94b806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:13.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 -- 172.21.15.125:0/2650724095 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637974642 0 0) 0x7f94b8066cf0 con 0x7f94b8071110 2023-10-30T12:55:13.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 -- 172.21.15.125:0/2650724095 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a4003530 con 0x7f94b8071110 2023-10-30T12:55:13.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 -- 172.21.15.125:0/2650724095 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2171838668 0 0) 0x7f94a4003530 con 0x7f94b8071110 2023-10-30T12:55:13.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 -- 172.21.15.125:0/2650724095 >> v2:172.21.15.136:3300/0 conn(0x7f94b80c6bd0 msgr2=0x7f94b806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:13.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 --2- 172.21.15.125:0/2650724095 >> v2:172.21.15.136:3300/0 conn(0x7f94b80c6bd0 0x7f94b806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:13.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 -- 172.21.15.125:0/2650724095 >> v1:172.21.15.136:6789/0 conn(0x7f94b806f7a0 legacy=0x7f94b8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:13.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 -- 172.21.15.125:0/2650724095 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b80c7390 con 0x7f94b8071110 2023-10-30T12:55:13.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 -- 172.21.15.125:0/2650724095 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f94b0001a00 con 0x7f94b8071110 2023-10-30T12:55:13.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 -- 172.21.15.125:0/2650724095 >> v1:172.21.15.125:6789/0 conn(0x7f94b8071110 legacy=0x7f94b8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:13.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 --2- 172.21.15.125:0/2650724095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a4003980 0x7f94a4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:13.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 --2- 172.21.15.125:0/2650724095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94a40047d0 0x7f94a4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:13.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 --2- 172.21.15.125:0/2650724095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94a4005190 0x7f94a400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:13.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 -- 172.21.15.125:0/2650724095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b80700c0 con 0x7f94a4003980 2023-10-30T12:55:13.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 -- 172.21.15.125:0/2650724095 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a400c040 con 0x7f94a40047d0 2023-10-30T12:55:13.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bcf37640 1 -- 172.21.15.125:0/2650724095 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a400c200 con 0x7f94a4005190 2023-10-30T12:55:13.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bdf39640 1 --2- 172.21.15.125:0/2650724095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a4003980 0x7f94a4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:13.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94be73a640 1 --2- 172.21.15.125:0/2650724095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94a4005190 0x7f94a400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:13.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bd738640 1 --2- 172.21.15.125:0/2650724095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94a40047d0 0x7f94a4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:13.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bdf39640 1 -- 172.21.15.125:0/2650724095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94a4005190 msgr2=0x7f94a400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:13.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bdf39640 1 --2- 172.21.15.125:0/2650724095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94a4005190 0x7f94a400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:13.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bdf39640 1 -- 172.21.15.125:0/2650724095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94a40047d0 msgr2=0x7f94a4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:13.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bdf39640 1 --2- 172.21.15.125:0/2650724095 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94a40047d0 0x7f94a4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:13.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94bdf39640 1 -- 172.21.15.125:0/2650724095 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a4003530 con 0x7f94a4003980 2023-10-30T12:55:13.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.762+0000 7f94be73a640 1 -- 172.21.15.125:0/2650724095 reap_dead start 2023-10-30T12:55:13.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.766+0000 7f94bdf39640 1 --2- 172.21.15.125:0/2650724095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a4003980 0x7f94a4003da0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f94b4001780 tx=0x7f94b401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b309f725da5ca7d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:13.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.766+0000 7f94bcf37640 1 -- 172.21.15.125:0/2650724095 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b40a4ea0 con 0x7f94a4003980 2023-10-30T12:55:13.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.766+0000 7f94bcf37640 1 -- 172.21.15.125:0/2650724095 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b40a7050 con 0x7f94a4003980 2023-10-30T12:55:13.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.766+0000 7f94bcf37640 1 -- 172.21.15.125:0/2650724095 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b4003600 con 0x7f94a4003980 2023-10-30T12:55:13.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.766+0000 7f94c019d640 1 -- 172.21.15.125:0/2650724095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a4003980 msgr2=0x7f94a4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:13.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.766+0000 7f94c019d640 1 --2- 172.21.15.125:0/2650724095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a4003980 0x7f94a4003da0 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f94b4001780 tx=0x7f94b401c3e0 comp rx=0 tx=0).stop 2023-10-30T12:55:13.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.766+0000 7f94c019d640 1 -- 172.21.15.125:0/2650724095 shutdown_connections 2023-10-30T12:55:13.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.766+0000 7f94c019d640 1 --2- 172.21.15.125:0/2650724095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a4003980 0x7f94a4003da0 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:13.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.766+0000 7f94c019d640 1 -- 172.21.15.125:0/2650724095 >> 172.21.15.125:0/2650724095 conn(0x7f94b80c7910 msgr2=0x7f94b80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:13.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.766+0000 7f94c019d640 1 -- 172.21.15.125:0/2650724095 shutdown_connections 2023-10-30T12:55:13.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.766+0000 7f94c019d640 1 -- 172.21.15.125:0/2650724095 wait complete. 2023-10-30T12:55:13.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.766+0000 7f94c019d640 1 Processor -- start 2023-10-30T12:55:13.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.766+0000 7f94c019d640 1 -- start start 2023-10-30T12:55:13.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94c019d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a4003980 0x7f94b80c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:13.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94c019d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94b814dee0 0x7f94b80c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:13.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94c019d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94b814c7a0 0x7f94b814cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:13.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94c019d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b81562f0 con 0x7f94a4003980 2023-10-30T12:55:13.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94c019d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94b8156590 con 0x7f94b814dee0 2023-10-30T12:55:13.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94c019d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b814e890 con 0x7f94b814c7a0 2023-10-30T12:55:13.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94bdf39640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a4003980 0x7f94b80c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:13.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94bdf39640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a4003980 0x7f94b80c36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36288/0 (socket says 172.21.15.125:36288) 2023-10-30T12:55:13.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94bdf39640 1 -- 172.21.15.125:0/3814534565 learned_addr learned my addr 172.21.15.125:0/3814534565 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:13.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94be73a640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94b814c7a0 0x7f94b814cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:13.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94bd738640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94b814dee0 0x7f94b80c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:13.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94bdf39640 1 -- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94b814c7a0 msgr2=0x7f94b814cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:13.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94bdf39640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94b814c7a0 0x7f94b814cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:13.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94bdf39640 1 -- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94b814dee0 msgr2=0x7f94b80c3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:13.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94bdf39640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94b814dee0 0x7f94b80c3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:13.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94bdf39640 1 -- 172.21.15.125:0/3814534565 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b814eb70 con 0x7f94a4003980 2023-10-30T12:55:13.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94bd738640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94b814dee0 0x7f94b80c3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:55:13.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94bdf39640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a4003980 0x7f94b80c36d0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f94b401c210 tx=0x7f94a4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:13.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94ae7fc640 1 -- 172.21.15.125:0/3814534565 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b4003e40 con 0x7f94a4003980 2023-10-30T12:55:13.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94ae7fc640 1 -- 172.21.15.125:0/3814534565 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b40a4ea0 con 0x7f94a4003980 2023-10-30T12:55:13.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94ae7fc640 1 -- 172.21.15.125:0/3814534565 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b4007270 con 0x7f94a4003980 2023-10-30T12:55:13.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94c019d640 1 -- 172.21.15.125:0/3814534565 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b814ee50 con 0x7f94a4003980 2023-10-30T12:55:13.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94c019d640 1 -- 172.21.15.125:0/3814534565 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b814f410 con 0x7f94a4003980 2023-10-30T12:55:13.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94c019d640 1 -- 172.21.15.125:0/3814534565 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94b804e170 con 0x7f94a4003980 2023-10-30T12:55:13.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.770+0000 7f94ae7fc640 1 -- 172.21.15.125:0/3814534565 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94b40073d0 con 0x7f94a4003980 2023-10-30T12:55:13.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.774+0000 7f94ae7fc640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f949c0664a0 0x7f949c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:13.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.774+0000 7f94ae7fc640 1 -- 172.21.15.125:0/3814534565 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6770+0+0 (secure 0 0 0) 0x7f94b411e740 con 0x7f94a4003980 2023-10-30T12:55:13.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.774+0000 7f94bd738640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f949c0664a0 0x7f949c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:13.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.774+0000 7f94bd738640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f949c0664a0 0x7f949c068960 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f94b814c3f0 tx=0x7f94a8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:13.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.778+0000 7f94ae7fc640 1 -- 172.21.15.125:0/3814534565 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f94b40ef0f0 con 0x7f94a4003980 2023-10-30T12:55:13.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.954+0000 7f94c019d640 1 -- 172.21.15.125:0/3814534565 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94b806c3a0 con 0x7f94a4003980 2023-10-30T12:55:13.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.954+0000 7f94ae7fc640 1 -- 172.21.15.125:0/3814534565 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f94b4007a40 con 0x7f94a4003980 2023-10-30T12:55:13.958 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:13.958 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7113,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":123}],"filesystems":[{"mdsmap":{"epoch":122,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:52:59.039508+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6954},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_6954":{"gid":6954,"name":"e","rank":0,"incarnation":118,"state":"up:active","state_seq":19,"addr":"172.21.15.125:6836/1163622496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1163622496},{"type":"v1","addr":"172.21.15.125:6836","nonce":1163622496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:13.959 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 123 2023-10-30T12:55:13.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.958+0000 7f94c019d640 1 -- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f949c0664a0 msgr2=0x7f949c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.958+0000 7f94c019d640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f949c0664a0 0x7f949c068960 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f94b814c3f0 tx=0x7f94a8008040 comp rx=0 tx=0).stop 2023-10-30T12:55:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.958+0000 7f94c019d640 1 -- 172.21.15.125:0/3814534565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a4003980 msgr2=0x7f94b80c36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.958+0000 7f94c019d640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a4003980 0x7f94b80c36d0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f94b401c210 tx=0x7f94a4005800 comp rx=0 tx=0).stop 2023-10-30T12:55:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.958+0000 7f94c019d640 1 -- 172.21.15.125:0/3814534565 shutdown_connections 2023-10-30T12:55:13.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.958+0000 7f94c019d640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f949c0664a0 0x7f949c068960 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:13.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.958+0000 7f94c019d640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94b814c7a0 0x7f94b814cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:13.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.958+0000 7f94c019d640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94b814dee0 0x7f94b80c3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:13.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.958+0000 7f94c019d640 1 --2- 172.21.15.125:0/3814534565 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94a4003980 0x7f94b80c36d0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:13.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.958+0000 7f94c019d640 1 -- 172.21.15.125:0/3814534565 >> 172.21.15.125:0/3814534565 conn(0x7f94b80c7910 msgr2=0x7f94b80c1190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:13.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.958+0000 7f94c019d640 1 -- 172.21.15.125:0/3814534565 shutdown_connections 2023-10-30T12:55:13.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:13.958+0000 7f94c019d640 1 -- 172.21.15.125:0/3814534565 wait complete. 2023-10-30T12:55:13.979 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:55:15.979 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:16.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.126+0000 7faf80c68640 1 Processor -- start 2023-10-30T12:55:16.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.126+0000 7faf80c68640 1 -- start start 2023-10-30T12:55:16.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.130+0000 7faf80c68640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faf7c0c6b80 0x7faf7c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:16.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.130+0000 7faf80c68640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7faf7c0c6f60 0x7faf7c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:16.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.130+0000 7faf80c68640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c0c0a50 con 0x7faf7c0c8430 2023-10-30T12:55:16.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.130+0000 7faf80c68640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7c06f6e0 con 0x7faf7c0c6b80 2023-10-30T12:55:16.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.130+0000 7faf80c68640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7c06f9c0 con 0x7faf7c0c6f60 2023-10-30T12:55:16.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.130+0000 7faf7a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faf7c0c6b80 0x7faf7c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:16.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.130+0000 7faf7a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faf7c0c6b80 0x7faf7c0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36290/0 (socket says 172.21.15.125:36290) 2023-10-30T12:55:16.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.130+0000 7faf7a59c640 1 -- 172.21.15.125:0/882170848 learned_addr learned my addr 172.21.15.125:0/882170848 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:16.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.130+0000 7faf79d9b640 1 --2- 172.21.15.125:0/882170848 >> v2:172.21.15.136:3300/0 conn(0x7faf7c0c6f60 0x7faf7c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:16.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.130+0000 7faf7a59c640 1 -- 172.21.15.125:0/882170848 >> v2:172.21.15.136:3300/0 conn(0x7faf7c0c6f60 msgr2=0x7faf7c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:16.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.130+0000 7faf7a59c640 1 --2- 172.21.15.125:0/882170848 >> v2:172.21.15.136:3300/0 conn(0x7faf7c0c6f60 0x7faf7c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.130+0000 7faf7a59c640 1 -- 172.21.15.125:0/882170848 >> v1:172.21.15.136:6789/0 conn(0x7faf7c0c8430 legacy=0x7faf7c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:16.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.130+0000 7faf7a59c640 1 -- 172.21.15.125:0/882170848 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c06fca0 con 0x7faf7c0c6b80 2023-10-30T12:55:16.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.134+0000 7faf7a59c640 1 --2- 172.21.15.125:0/882170848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf7c0c6b80 0x7faf7c0bfec0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7faf7c159b90 tx=0x7faf6c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb5863450296d9e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:16.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.134+0000 7faf7959a640 1 -- 172.21.15.125:0/882170848 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c0ab610 con 0x7faf7c0c6b80 2023-10-30T12:55:16.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.134+0000 7faf7959a640 1 -- 172.21.15.125:0/882170848 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6c0ab770 con 0x7faf7c0c6b80 2023-10-30T12:55:16.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.134+0000 7faf7959a640 1 -- 172.21.15.125:0/882170848 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c0abb00 con 0x7faf7c0c6b80 2023-10-30T12:55:16.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.134+0000 7faf80c68640 1 -- 172.21.15.125:0/882170848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf7c0c6b80 msgr2=0x7faf7c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:16.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.134+0000 7faf80c68640 1 --2- 172.21.15.125:0/882170848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf7c0c6b80 0x7faf7c0bfec0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7faf7c159b90 tx=0x7faf6c01c500 comp rx=0 tx=0).stop 2023-10-30T12:55:16.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.134+0000 7faf80c68640 1 -- 172.21.15.125:0/882170848 shutdown_connections 2023-10-30T12:55:16.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.134+0000 7faf80c68640 1 --2- 172.21.15.125:0/882170848 >> v2:172.21.15.136:3300/0 conn(0x7faf7c0c6f60 0x7faf7c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.134+0000 7faf80c68640 1 --2- 172.21.15.125:0/882170848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf7c0c6b80 0x7faf7c0bfec0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.134+0000 7faf80c68640 1 -- 172.21.15.125:0/882170848 >> 172.21.15.125:0/882170848 conn(0x7faf7c0c7910 msgr2=0x7faf7c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:16.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.134+0000 7faf80c68640 1 -- 172.21.15.125:0/882170848 shutdown_connections 2023-10-30T12:55:16.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.134+0000 7faf80c68640 1 -- 172.21.15.125:0/882170848 wait complete. 2023-10-30T12:55:16.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.134+0000 7faf80c68640 1 Processor -- start 2023-10-30T12:55:16.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.134+0000 7faf80c68640 1 -- start start 2023-10-30T12:55:16.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf80c68640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf7c0c6b80 0x7faf7c160610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:16.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf80c68640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf7c0c6f60 0x7faf7c15f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:16.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf80c68640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf7c0c8430 0x7faf7c15f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:16.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf80c68640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0c92c0 con 0x7faf7c0c6b80 2023-10-30T12:55:16.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf80c68640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c0c8cb0 con 0x7faf7c0c6f60 2023-10-30T12:55:16.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf80c68640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c136c80 con 0x7faf7c0c8430 2023-10-30T12:55:16.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf7c0c6b80 0x7faf7c160610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:16.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf7a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf7c0c6b80 0x7faf7c160610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36294/0 (socket says 172.21.15.125:36294) 2023-10-30T12:55:16.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf7a59c640 1 -- 172.21.15.125:0/2696447425 learned_addr learned my addr 172.21.15.125:0/2696447425 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:16.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf7ad9d640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf7c0c8430 0x7faf7c15f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:16.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf79d9b640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf7c0c6f60 0x7faf7c15f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:16.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf7a59c640 1 -- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf7c0c8430 msgr2=0x7faf7c15f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:16.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf7a59c640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf7c0c8430 0x7faf7c15f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf7a59c640 1 -- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf7c0c6f60 msgr2=0x7faf7c15f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:16.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf7a59c640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf7c0c6f60 0x7faf7c15f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf7a59c640 1 -- 172.21.15.125:0/2696447425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf6c0094d0 con 0x7faf7c0c6b80 2023-10-30T12:55:16.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf7ad9d640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf7c0c8430 0x7faf7c15f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:16.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf7a59c640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf7c0c6b80 0x7faf7c160610 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7faf6c019110 tx=0x7faf6c01ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:16.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf6affd640 1 -- 172.21.15.125:0/2696447425 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c0b4800 con 0x7faf7c0c6b80 2023-10-30T12:55:16.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf6affd640 1 -- 172.21.15.125:0/2696447425 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6c0b4960 con 0x7faf7c0c6b80 2023-10-30T12:55:16.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf80c68640 1 -- 172.21.15.125:0/2696447425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7c136f60 con 0x7faf7c0c6b80 2023-10-30T12:55:16.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf6affd640 1 -- 172.21.15.125:0/2696447425 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf6c0b4cf0 con 0x7faf7c0c6b80 2023-10-30T12:55:16.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf80c68640 1 -- 172.21.15.125:0/2696447425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7c137240 con 0x7faf7c0c6b80 2023-10-30T12:55:16.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.138+0000 7faf80c68640 1 -- 172.21.15.125:0/2696447425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7c04e170 con 0x7faf7c0c6b80 2023-10-30T12:55:16.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.142+0000 7faf6affd640 1 -- 172.21.15.125:0/2696447425 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faf6c0c7390 con 0x7faf7c0c6b80 2023-10-30T12:55:16.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.142+0000 7faf6affd640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf5c0664a0 0x7faf5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:16.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.142+0000 7faf6affd640 1 -- 172.21.15.125:0/2696447425 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7faf6c11ee50 con 0x7faf7c0c6b80 2023-10-30T12:55:16.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.142+0000 7faf79d9b640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf5c0664a0 0x7faf5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:16.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.142+0000 7faf79d9b640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf5c0664a0 0x7faf5c068960 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7faf7c0cad00 tx=0x7faf70009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:16.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.146+0000 7faf6affd640 1 -- 172.21.15.125:0/2696447425 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faf6c0ebe60 con 0x7faf7c0c6b80 2023-10-30T12:55:16.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.326+0000 7faf80c68640 1 -- 172.21.15.125:0/2696447425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf7c133da0 con 0x7faf7c0c6b80 2023-10-30T12:55:16.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.326+0000 7faf6affd640 1 -- 172.21.15.125:0/2696447425 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+5723 (secure 0 0 0) 0x7faf6c0ef480 con 0x7faf7c0c6b80 2023-10-30T12:55:16.329 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:16.329 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:15.400801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:16.330 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 124 2023-10-30T12:55:16.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.330+0000 7faf80c68640 1 -- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf5c0664a0 msgr2=0x7faf5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:16.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.330+0000 7faf80c68640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf5c0664a0 0x7faf5c068960 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7faf7c0cad00 tx=0x7faf70009190 comp rx=0 tx=0).stop 2023-10-30T12:55:16.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.330+0000 7faf80c68640 1 -- 172.21.15.125:0/2696447425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf7c0c6b80 msgr2=0x7faf7c160610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:16.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.330+0000 7faf80c68640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf7c0c6b80 0x7faf7c160610 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7faf6c019110 tx=0x7faf6c01ad40 comp rx=0 tx=0).stop 2023-10-30T12:55:16.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.330+0000 7faf80c68640 1 -- 172.21.15.125:0/2696447425 shutdown_connections 2023-10-30T12:55:16.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.330+0000 7faf80c68640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf5c0664a0 0x7faf5c068960 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.330+0000 7faf80c68640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faf7c0c8430 0x7faf7c15f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.330+0000 7faf80c68640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faf7c0c6f60 0x7faf7c15f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.330+0000 7faf80c68640 1 --2- 172.21.15.125:0/2696447425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faf7c0c6b80 0x7faf7c160610 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.330+0000 7faf80c68640 1 -- 172.21.15.125:0/2696447425 >> 172.21.15.125:0/2696447425 conn(0x7faf7c0c7910 msgr2=0x7faf7c0c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:16.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.330+0000 7faf80c68640 1 -- 172.21.15.125:0/2696447425 shutdown_connections 2023-10-30T12:55:16.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.330+0000 7faf80c68640 1 -- 172.21.15.125:0/2696447425 wait complete. 2023-10-30T12:55:16.350 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T12:55:16.350 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:55:16.351 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:16.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.486+0000 7ff45d49c640 1 Processor -- start 2023-10-30T12:55:16.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.486+0000 7ff45d49c640 1 -- start start 2023-10-30T12:55:16.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.486+0000 7ff45d49c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff4581538a0 0x7ff45815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:16.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.486+0000 7ff45d49c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4580cf160 0x7ff4580cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:16.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff45d49c640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4580cb800 con 0x7ff4580ca0e0 2023-10-30T12:55:16.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff45d49c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff45815dc30 con 0x7ff4581538a0 2023-10-30T12:55:16.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff45d49c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4580cb330 con 0x7ff4580cf160 2023-10-30T12:55:16.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff456ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff4581538a0 0x7ff45815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:16.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff44e7fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff4580ca0e0 0x7ff45815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54434/0 (socket says 172.21.15.125:54434) 2023-10-30T12:55:16.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff44e7fc640 1 -- 172.21.15.125:0/1967659825 learned_addr learned my addr 172.21.15.125:0/1967659825 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:55:16.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff4577fe640 1 --2- 172.21.15.125:0/1967659825 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4580cf160 0x7ff4580cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:16.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff456ffd640 1 -- 172.21.15.125:0/1967659825 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4580cf160 msgr2=0x7ff4580cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:16.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff456ffd640 1 --2- 172.21.15.125:0/1967659825 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4580cf160 0x7ff4580cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff456ffd640 1 -- 172.21.15.125:0/1967659825 >> v1:172.21.15.125:6789/0 conn(0x7ff4580ca0e0 legacy=0x7ff45815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:16.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff456ffd640 1 -- 172.21.15.125:0/1967659825 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4580cad70 con 0x7ff4581538a0 2023-10-30T12:55:16.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff4567fc640 1 -- 172.21.15.125:0/1967659825 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362814832 0 0) 0x7ff4580cb800 con 0x7ff4580ca0e0 2023-10-30T12:55:16.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff456ffd640 1 --2- 172.21.15.125:0/1967659825 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4581538a0 0x7ff45815d6f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff4400097c0 tx=0x7ff44001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03525b8e1adf889 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:16.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff4567fc640 1 -- 172.21.15.125:0/1967659825 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4400a8970 con 0x7ff4581538a0 2023-10-30T12:55:16.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff4567fc640 1 -- 172.21.15.125:0/1967659825 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4400a8ad0 con 0x7ff4581538a0 2023-10-30T12:55:16.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.490+0000 7ff4567fc640 1 -- 172.21.15.125:0/1967659825 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4400a8e60 con 0x7ff4581538a0 2023-10-30T12:55:16.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 -- 172.21.15.125:0/1967659825 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4581538a0 msgr2=0x7ff45815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:16.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 --2- 172.21.15.125:0/1967659825 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4581538a0 0x7ff45815d6f0 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7ff4400097c0 tx=0x7ff44001ec70 comp rx=0 tx=0).stop 2023-10-30T12:55:16.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 -- 172.21.15.125:0/1967659825 shutdown_connections 2023-10-30T12:55:16.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 --2- 172.21.15.125:0/1967659825 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4580cf160 0x7ff4580cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 --2- 172.21.15.125:0/1967659825 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4581538a0 0x7ff45815d6f0 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 -- 172.21.15.125:0/1967659825 >> 172.21.15.125:0/1967659825 conn(0x7ff45806bfe0 msgr2=0x7ff45806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:16.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 -- 172.21.15.125:0/1967659825 shutdown_connections 2023-10-30T12:55:16.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 -- 172.21.15.125:0/1967659825 wait complete. 2023-10-30T12:55:16.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 Processor -- start 2023-10-30T12:55:16.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 -- start start 2023-10-30T12:55:16.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4580ca0e0 0x7ff45815aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:16.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4580cf160 0x7ff4580ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:16.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4581538a0 0x7ff4581ef320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:16.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4581f3160 con 0x7ff4580cf160 2023-10-30T12:55:16.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45815b3e0 con 0x7ff4581538a0 2023-10-30T12:55:16.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff45d49c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4581f2890 con 0x7ff4580ca0e0 2023-10-30T12:55:16.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff44e7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4580cf160 0x7ff4580ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:16.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff44e7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4580cf160 0x7ff4580ce930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36324/0 (socket says 172.21.15.125:36324) 2023-10-30T12:55:16.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff44e7fc640 1 -- 172.21.15.125:0/1727629052 learned_addr learned my addr 172.21.15.125:0/1727629052 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:16.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff456ffd640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4580ca0e0 0x7ff45815aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:16.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.494+0000 7ff4577fe640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4581538a0 0x7ff4581ef320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:16.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.498+0000 7ff44e7fc640 1 -- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4580ca0e0 msgr2=0x7ff45815aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:16.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.498+0000 7ff44e7fc640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4580ca0e0 0x7ff45815aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.498+0000 7ff44e7fc640 1 -- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4581538a0 msgr2=0x7ff4581ef320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:16.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.498+0000 7ff44e7fc640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4581538a0 0x7ff4581ef320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.498+0000 7ff44e7fc640 1 -- 172.21.15.125:0/1727629052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff440009420 con 0x7ff4580cf160 2023-10-30T12:55:16.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.498+0000 7ff4577fe640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4581538a0 0x7ff4581ef320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:16.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.498+0000 7ff44e7fc640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4580cf160 0x7ff4580ce930 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7ff4480011f0 tx=0x7ff44800c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:16.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.498+0000 7ff44ffff640 1 -- 172.21.15.125:0/1727629052 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4480074e0 con 0x7ff4580cf160 2023-10-30T12:55:16.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.498+0000 7ff44ffff640 1 -- 172.21.15.125:0/1727629052 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff448007640 con 0x7ff4580cf160 2023-10-30T12:55:16.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.498+0000 7ff45d49c640 1 -- 172.21.15.125:0/1727629052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4580b51f0 con 0x7ff4580cf160 2023-10-30T12:55:16.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.498+0000 7ff44ffff640 1 -- 172.21.15.125:0/1727629052 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff448004a10 con 0x7ff4580cf160 2023-10-30T12:55:16.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.498+0000 7ff45d49c640 1 -- 172.21.15.125:0/1727629052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4581f2aa0 con 0x7ff4580cf160 2023-10-30T12:55:16.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.498+0000 7ff45d49c640 1 -- 172.21.15.125:0/1727629052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4580b2a50 con 0x7ff4580cf160 2023-10-30T12:55:16.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.502+0000 7ff44ffff640 1 -- 172.21.15.125:0/1727629052 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff448004b70 con 0x7ff4580cf160 2023-10-30T12:55:16.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.502+0000 7ff44ffff640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4380664a0 0x7ff438068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:16.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.502+0000 7ff44ffff640 1 -- 172.21.15.125:0/1727629052 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7ff448083440 con 0x7ff4580cf160 2023-10-30T12:55:16.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.502+0000 7ff456ffd640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4380664a0 0x7ff438068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:16.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.502+0000 7ff456ffd640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4380664a0 0x7ff438068960 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff4400b2000 tx=0x7ff440008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:16.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.506+0000 7ff44ffff640 1 -- 172.21.15.125:0/1727629052 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff448050490 con 0x7ff4580cf160 2023-10-30T12:55:16.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.686+0000 7ff45d49c640 1 -- 172.21.15.125:0/1727629052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4580b37a0 con 0x7ff4580cf160 2023-10-30T12:55:16.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.686+0000 7ff44ffff640 1 -- 172.21.15.125:0/1727629052 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+5723 (secure 0 0 0) 0x7ff4580b37a0 con 0x7ff4580cf160 2023-10-30T12:55:16.690 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:16.690 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:15.400801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:16.691 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 124 2023-10-30T12:55:16.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.690+0000 7ff45d49c640 1 -- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4380664a0 msgr2=0x7ff438068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:16.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.690+0000 7ff45d49c640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4380664a0 0x7ff438068960 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff4400b2000 tx=0x7ff440008040 comp rx=0 tx=0).stop 2023-10-30T12:55:16.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.690+0000 7ff45d49c640 1 -- 172.21.15.125:0/1727629052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4580cf160 msgr2=0x7ff4580ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:16.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.690+0000 7ff45d49c640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4580cf160 0x7ff4580ce930 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7ff4480011f0 tx=0x7ff44800c9b0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.690+0000 7ff45d49c640 1 -- 172.21.15.125:0/1727629052 shutdown_connections 2023-10-30T12:55:16.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.690+0000 7ff45d49c640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4380664a0 0x7ff438068960 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.690+0000 7ff45d49c640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4581538a0 0x7ff4581ef320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.690+0000 7ff45d49c640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4580cf160 0x7ff4580ce930 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.690+0000 7ff45d49c640 1 --2- 172.21.15.125:0/1727629052 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4580ca0e0 0x7ff45815aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:16.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.690+0000 7ff45d49c640 1 -- 172.21.15.125:0/1727629052 >> 172.21.15.125:0/1727629052 conn(0x7ff45806bfe0 msgr2=0x7ff45814db50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:16.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.690+0000 7ff45d49c640 1 -- 172.21.15.125:0/1727629052 shutdown_connections 2023-10-30T12:55:16.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:16.690+0000 7ff45d49c640 1 -- 172.21.15.125:0/1727629052 wait complete. 2023-10-30T12:55:16.712 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 124, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 124, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7113": { "addr": "172.21.15.125:6837/1348687922", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 1348687922, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 1348687922, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7113, "incarnation": 124, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:55:15.400801+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7113 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:55:18.395 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:55:18.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:55:18.713 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:18.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd328009640 1 Processor -- start 2023-10-30T12:55:18.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd328009640 1 -- start start 2023-10-30T12:55:18.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd328009640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd3200c05d0 0x7fd32013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:18.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd328009640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3200c6f60 0x7fd32013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:18.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd328009640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3200c0a50 con 0x7fd3200c6b80 2023-10-30T12:55:18.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd328009640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd32006f6e0 con 0x7fd3200c05d0 2023-10-30T12:55:18.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd328009640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32006f9c0 con 0x7fd3200c6f60 2023-10-30T12:55:18.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd325da5640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd3200c6b80 0x7fd3200bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54444/0 (socket says 172.21.15.125:54444) 2023-10-30T12:55:18.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd325da5640 1 -- 172.21.15.125:0/158858943 learned_addr learned my addr 172.21.15.125:0/158858943 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:55:18.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd3255a4640 1 --2- 172.21.15.125:0/158858943 >> v2:172.21.15.125:3300/0 conn(0x7fd3200c05d0 0x7fd32013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:18.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd3265a6640 1 --2- 172.21.15.125:0/158858943 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3200c6f60 0x7fd32013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:18.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd324da3640 1 -- 172.21.15.125:0/158858943 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326555020 0 0) 0x7fd3200c0a50 con 0x7fd3200c6b80 2023-10-30T12:55:18.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd324da3640 1 -- 172.21.15.125:0/158858943 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd30c003530 con 0x7fd3200c6b80 2023-10-30T12:55:18.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd3255a4640 1 -- 172.21.15.125:0/158858943 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3200c6f60 msgr2=0x7fd32013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:18.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd3255a4640 1 --2- 172.21.15.125:0/158858943 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3200c6f60 0x7fd32013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:18.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd3255a4640 1 -- 172.21.15.125:0/158858943 >> v1:172.21.15.125:6789/0 conn(0x7fd3200c6b80 legacy=0x7fd3200bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:18.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.854+0000 7fd3255a4640 1 -- 172.21.15.125:0/158858943 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32006fca0 con 0x7fd3200c05d0 2023-10-30T12:55:18.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.858+0000 7fd3255a4640 1 --2- 172.21.15.125:0/158858943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3200c05d0 0x7fd32013ed90 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fd3100097c0 tx=0x7fd31001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df711ef9f99057f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:18.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.858+0000 7fd324da3640 1 -- 172.21.15.125:0/158858943 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3100ad640 con 0x7fd3200c05d0 2023-10-30T12:55:18.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.858+0000 7fd324da3640 1 -- 172.21.15.125:0/158858943 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3100ad7a0 con 0x7fd3200c05d0 2023-10-30T12:55:18.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.858+0000 7fd324da3640 1 -- 172.21.15.125:0/158858943 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3100adb30 con 0x7fd3200c05d0 2023-10-30T12:55:18.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.858+0000 7fd328009640 1 -- 172.21.15.125:0/158858943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3200c05d0 msgr2=0x7fd32013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:18.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.858+0000 7fd328009640 1 --2- 172.21.15.125:0/158858943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3200c05d0 0x7fd32013ed90 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fd3100097c0 tx=0x7fd31001c480 comp rx=0 tx=0).stop 2023-10-30T12:55:18.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.858+0000 7fd328009640 1 -- 172.21.15.125:0/158858943 shutdown_connections 2023-10-30T12:55:18.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.858+0000 7fd328009640 1 --2- 172.21.15.125:0/158858943 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3200c05d0 0x7fd32013ed90 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:18.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.858+0000 7fd328009640 1 --2- 172.21.15.125:0/158858943 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3200c6f60 0x7fd32013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:18.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.858+0000 7fd328009640 1 -- 172.21.15.125:0/158858943 >> 172.21.15.125:0/158858943 conn(0x7fd3200c7910 msgr2=0x7fd3200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:18.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.858+0000 7fd328009640 1 -- 172.21.15.125:0/158858943 shutdown_connections 2023-10-30T12:55:18.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.858+0000 7fd328009640 1 -- 172.21.15.125:0/158858943 wait complete. 2023-10-30T12:55:18.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd328009640 1 Processor -- start 2023-10-30T12:55:18.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd328009640 1 -- start start 2023-10-30T12:55:18.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd328009640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3200c05d0 0x7fd32014b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:18.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd328009640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3200c6b80 0x7fd32014ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd328009640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3200c6f60 0x7fd3201f3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd328009640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32014c5c0 con 0x7fd3200c6b80 2023-10-30T12:55:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd328009640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32014d310 con 0x7fd3200c6f60 2023-10-30T12:55:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd328009640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd320139050 con 0x7fd3200c05d0 2023-10-30T12:55:18.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd3255a4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3200c6b80 0x7fd32014ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd3255a4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3200c6b80 0x7fd32014ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36334/0 (socket says 172.21.15.125:36334) 2023-10-30T12:55:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd3255a4640 1 -- 172.21.15.125:0/1116348256 learned_addr learned my addr 172.21.15.125:0/1116348256 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd325da5640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3200c05d0 0x7fd32014b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd3265a6640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3200c6f60 0x7fd3201f3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:18.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd3255a4640 1 -- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3200c05d0 msgr2=0x7fd32014b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:18.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd3255a4640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3200c05d0 0x7fd32014b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:18.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd3255a4640 1 -- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3200c6f60 msgr2=0x7fd3201f3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:18.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd3255a4640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3200c6f60 0x7fd3201f3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:18.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd3255a4640 1 -- 172.21.15.125:0/1116348256 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd310009420 con 0x7fd3200c6b80 2023-10-30T12:55:18.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd3265a6640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3200c6f60 0x7fd3201f3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:18.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd325da5640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3200c05d0 0x7fd32014b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:18.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd3255a4640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3200c6b80 0x7fd32014ddc0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fd3100b0040 tx=0x7fd310019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:18.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd3167fc640 1 -- 172.21.15.125:0/1116348256 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3100b6820 con 0x7fd3200c6b80 2023-10-30T12:55:18.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd3167fc640 1 -- 172.21.15.125:0/1116348256 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3100a6040 con 0x7fd3200c6b80 2023-10-30T12:55:18.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd3167fc640 1 -- 172.21.15.125:0/1116348256 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3100b6be0 con 0x7fd3200c6b80 2023-10-30T12:55:18.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd328009640 1 -- 172.21.15.125:0/1116348256 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd320138d70 con 0x7fd3200c6b80 2023-10-30T12:55:18.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.862+0000 7fd328009640 1 -- 172.21.15.125:0/1116348256 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd320158c80 con 0x7fd3200c6b80 2023-10-30T12:55:18.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.866+0000 7fd328009640 1 -- 172.21.15.125:0/1116348256 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32004e170 con 0x7fd3200c6b80 2023-10-30T12:55:18.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.866+0000 7fd3167fc640 1 -- 172.21.15.125:0/1116348256 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd3100b4040 con 0x7fd3200c6b80 2023-10-30T12:55:18.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.866+0000 7fd3167fc640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3040664a0 0x7fd304068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:18.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.866+0000 7fd3167fc640 1 -- 172.21.15.125:0/1116348256 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7fd31011e6e0 con 0x7fd3200c6b80 2023-10-30T12:55:18.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.866+0000 7fd325da5640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3040664a0 0x7fd304068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:18.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.866+0000 7fd325da5640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3040664a0 0x7fd304068960 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0011f0 tx=0x7fd31c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:18.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:18.870+0000 7fd3167fc640 1 -- 172.21.15.125:0/1116348256 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd3100eb640 con 0x7fd3200c6b80 2023-10-30T12:55:19.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:19.050+0000 7fd328009640 1 -- 172.21.15.125:0/1116348256 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd32014b9e0 con 0x7fd3200c6b80 2023-10-30T12:55:19.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:19.050+0000 7fd3167fc640 1 -- 172.21.15.125:0/1116348256 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+5723 (secure 0 0 0) 0x7fd3100b2090 con 0x7fd3200c6b80 2023-10-30T12:55:19.052 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:19.052 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:15.400801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:19.053 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 124 2023-10-30T12:55:19.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:19.050+0000 7fd328009640 1 -- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3040664a0 msgr2=0x7fd304068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:19.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:19.050+0000 7fd328009640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3040664a0 0x7fd304068960 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0011f0 tx=0x7fd31c009040 comp rx=0 tx=0).stop 2023-10-30T12:55:19.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:19.050+0000 7fd328009640 1 -- 172.21.15.125:0/1116348256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3200c6b80 msgr2=0x7fd32014ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:19.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:19.050+0000 7fd328009640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3200c6b80 0x7fd32014ddc0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fd3100b0040 tx=0x7fd310019220 comp rx=0 tx=0).stop 2023-10-30T12:55:19.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:19.050+0000 7fd328009640 1 -- 172.21.15.125:0/1116348256 shutdown_connections 2023-10-30T12:55:19.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:19.050+0000 7fd328009640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3040664a0 0x7fd304068960 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:19.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:19.050+0000 7fd328009640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3200c6f60 0x7fd3201f3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:19.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:19.050+0000 7fd328009640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3200c6b80 0x7fd32014ddc0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:19.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:19.050+0000 7fd328009640 1 --2- 172.21.15.125:0/1116348256 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3200c05d0 0x7fd32014b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:19.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:19.050+0000 7fd328009640 1 -- 172.21.15.125:0/1116348256 >> 172.21.15.125:0/1116348256 conn(0x7fd3200c7910 msgr2=0x7fd3201346e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:19.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:19.050+0000 7fd328009640 1 -- 172.21.15.125:0/1116348256 shutdown_connections 2023-10-30T12:55:19.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:19.050+0000 7fd328009640 1 -- 172.21.15.125:0/1116348256 wait complete. 2023-10-30T12:55:19.073 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:55:21.074 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:21.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d1633e640 1 Processor -- start 2023-10-30T12:55:21.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d1633e640 1 -- start start 2023-10-30T12:55:21.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d1633e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6d101538a0 0x7f6d1015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:21.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d1633e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6d100ca0e0 0x7f6d1015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:21.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d1633e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d100cb800 con 0x7f6d100cf160 2023-10-30T12:55:21.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d1633e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d1015dc30 con 0x7f6d100ca0e0 2023-10-30T12:55:21.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d1633e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d100cb330 con 0x7f6d101538a0 2023-10-30T12:55:21.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d0f7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6d100ca0e0 0x7f6d1015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:21.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d0f7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6d100ca0e0 0x7f6d1015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39036/0 (socket says 172.21.15.125:39036) 2023-10-30T12:55:21.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d0f7fe640 1 -- 172.21.15.125:0/4231167998 learned_addr learned my addr 172.21.15.125:0/4231167998 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:21.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d148db640 1 --1- 172.21.15.125:0/4231167998 >> v1:172.21.15.125:6789/0 conn(0x7f6d100cf160 0x7f6d100cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36612/0 (socket says 172.21.15.125:36612) 2023-10-30T12:55:21.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d0ffff640 1 --2- 172.21.15.125:0/4231167998 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6d101538a0 0x7f6d1015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:21.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d0f7fe640 1 -- 172.21.15.125:0/4231167998 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6d101538a0 msgr2=0x7f6d1015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:21.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d0f7fe640 1 --2- 172.21.15.125:0/4231167998 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6d101538a0 0x7f6d1015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:21.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d0f7fe640 1 -- 172.21.15.125:0/4231167998 >> v1:172.21.15.125:6789/0 conn(0x7f6d100cf160 legacy=0x7f6d100cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:21.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d0f7fe640 1 -- 172.21.15.125:0/4231167998 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d100cad70 con 0x7f6d100ca0e0 2023-10-30T12:55:21.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.218+0000 7f6d0effd640 1 -- 172.21.15.125:0/4231167998 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829856543 0 0) 0x7f6d100cb800 con 0x7f6d100cf160 2023-10-30T12:55:21.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.222+0000 7f6d0f7fe640 1 --2- 172.21.15.125:0/4231167998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d100ca0e0 0x7f6d1015a090 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009870 tx=0x7f6d0401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2422c9953f362c6f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:21.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.222+0000 7f6d0effd640 1 -- 172.21.15.125:0/4231167998 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d040a8a30 con 0x7f6d100ca0e0 2023-10-30T12:55:21.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.222+0000 7f6d0effd640 1 -- 172.21.15.125:0/4231167998 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d040a8b90 con 0x7f6d100ca0e0 2023-10-30T12:55:21.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.222+0000 7f6d0effd640 1 -- 172.21.15.125:0/4231167998 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d040b0440 con 0x7f6d100ca0e0 2023-10-30T12:55:21.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.222+0000 7f6d1633e640 1 -- 172.21.15.125:0/4231167998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d100ca0e0 msgr2=0x7f6d1015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:21.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.222+0000 7f6d1633e640 1 --2- 172.21.15.125:0/4231167998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d100ca0e0 0x7f6d1015a090 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f6d04009870 tx=0x7f6d0401ed30 comp rx=0 tx=0).stop 2023-10-30T12:55:21.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.222+0000 7f6d1633e640 1 -- 172.21.15.125:0/4231167998 shutdown_connections 2023-10-30T12:55:21.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.222+0000 7f6d1633e640 1 --2- 172.21.15.125:0/4231167998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d100ca0e0 0x7f6d1015a090 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:21.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.222+0000 7f6d1633e640 1 --2- 172.21.15.125:0/4231167998 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6d101538a0 0x7f6d1015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:21.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.222+0000 7f6d1633e640 1 -- 172.21.15.125:0/4231167998 >> 172.21.15.125:0/4231167998 conn(0x7f6d1006bfe0 msgr2=0x7f6d1006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:21.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.222+0000 7f6d1633e640 1 -- 172.21.15.125:0/4231167998 shutdown_connections 2023-10-30T12:55:21.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.222+0000 7f6d1633e640 1 -- 172.21.15.125:0/4231167998 wait complete. 2023-10-30T12:55:21.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d1633e640 1 Processor -- start 2023-10-30T12:55:21.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d1633e640 1 -- start start 2023-10-30T12:55:21.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d1633e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d100ca0e0 0x7f6d10158780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:21.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d1633e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6d100cf160 0x7f6d1015b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:21.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d1633e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d101538a0 0x7f6d101458d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:21.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d1633e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d1015b4c0 con 0x7f6d101538a0 2023-10-30T12:55:21.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d1633e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d101e5640 con 0x7f6d100ca0e0 2023-10-30T12:55:21.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d1633e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d101e5900 con 0x7f6d100cf160 2023-10-30T12:55:21.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d148db640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d101538a0 0x7f6d101458d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:21.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d148db640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d101538a0 0x7f6d101458d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39044/0 (socket says 172.21.15.125:39044) 2023-10-30T12:55:21.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d148db640 1 -- 172.21.15.125:0/3858948538 learned_addr learned my addr 172.21.15.125:0/3858948538 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:21.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d0f7fe640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6d100cf160 0x7f6d1015b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:21.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d0ffff640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d100ca0e0 0x7f6d10158780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:21.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d148db640 1 -- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6d100cf160 msgr2=0x7f6d1015b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:21.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d148db640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6d100cf160 0x7f6d1015b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:21.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d148db640 1 -- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d100ca0e0 msgr2=0x7f6d10158780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:21.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d148db640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d100ca0e0 0x7f6d10158780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:21.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d148db640 1 -- 172.21.15.125:0/3858948538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d040094d0 con 0x7f6d101538a0 2023-10-30T12:55:21.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d0ffff640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d100ca0e0 0x7f6d10158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:21.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d148db640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d101538a0 0x7f6d101458d0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800c780 tx=0x7f6d0800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:21.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d0cff9640 1 -- 172.21.15.125:0/3858948538 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0800ce30 con 0x7f6d101538a0 2023-10-30T12:55:21.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d0cff9640 1 -- 172.21.15.125:0/3858948538 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d0800a640 con 0x7f6d101538a0 2023-10-30T12:55:21.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d0cff9640 1 -- 172.21.15.125:0/3858948538 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d0800aa30 con 0x7f6d101538a0 2023-10-30T12:55:21.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d1633e640 1 -- 172.21.15.125:0/3858948538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d1015a8b0 con 0x7f6d101538a0 2023-10-30T12:55:21.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.226+0000 7f6d1633e640 1 -- 172.21.15.125:0/3858948538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d1015ae70 con 0x7f6d101538a0 2023-10-30T12:55:21.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.230+0000 7f6d1633e640 1 -- 172.21.15.125:0/3858948538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d100b2a50 con 0x7f6d101538a0 2023-10-30T12:55:21.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.230+0000 7f6d0cff9640 1 -- 172.21.15.125:0/3858948538 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6d0800ab90 con 0x7f6d101538a0 2023-10-30T12:55:21.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.230+0000 7f6d0cff9640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cfc0664a0 0x7f6cfc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:21.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.230+0000 7f6d0cff9640 1 -- 172.21.15.125:0/3858948538 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7f6d080836d0 con 0x7f6d101538a0 2023-10-30T12:55:21.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.230+0000 7f6d0ffff640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cfc0664a0 0x7f6cfc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:21.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.230+0000 7f6d0ffff640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cfc0664a0 0x7f6cfc068960 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009b10 tx=0x7f6d00008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:21.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.238+0000 7f6d0cff9640 1 -- 172.21.15.125:0/3858948538 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6d08050770 con 0x7f6d101538a0 2023-10-30T12:55:21.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.438+0000 7f6d1633e640 1 -- 172.21.15.125:0/3858948538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d10159860 con 0x7f6d101538a0 2023-10-30T12:55:21.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.442+0000 7f6d0cff9640 1 -- 172.21.15.125:0/3858948538 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f6d08001770 con 0x7f6d101538a0 2023-10-30T12:55:21.442 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:21.442 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:15.400801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:21.443 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 124 2023-10-30T12:55:21.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.442+0000 7f6d1633e640 1 -- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cfc0664a0 msgr2=0x7f6cfc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:21.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.442+0000 7f6d1633e640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cfc0664a0 0x7f6cfc068960 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f6d00009b10 tx=0x7f6d00008040 comp rx=0 tx=0).stop 2023-10-30T12:55:21.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.442+0000 7f6d1633e640 1 -- 172.21.15.125:0/3858948538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d101538a0 msgr2=0x7f6d101458d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:21.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.442+0000 7f6d1633e640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d101538a0 0x7f6d101458d0 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f6d0800c780 tx=0x7f6d0800cc40 comp rx=0 tx=0).stop 2023-10-30T12:55:21.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.442+0000 7f6d1633e640 1 -- 172.21.15.125:0/3858948538 shutdown_connections 2023-10-30T12:55:21.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.442+0000 7f6d1633e640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cfc0664a0 0x7f6cfc068960 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:21.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.442+0000 7f6d1633e640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6d101538a0 0x7f6d101458d0 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:21.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.442+0000 7f6d1633e640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6d100cf160 0x7f6d1015b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:21.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.442+0000 7f6d1633e640 1 --2- 172.21.15.125:0/3858948538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6d100ca0e0 0x7f6d10158780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:21.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.442+0000 7f6d1633e640 1 -- 172.21.15.125:0/3858948538 >> 172.21.15.125:0/3858948538 conn(0x7f6d1006bfe0 msgr2=0x7f6d10151a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:21.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.442+0000 7f6d1633e640 1 -- 172.21.15.125:0/3858948538 shutdown_connections 2023-10-30T12:55:21.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:21.442+0000 7f6d1633e640 1 -- 172.21.15.125:0/3858948538 wait complete. 2023-10-30T12:55:21.463 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:55:23.464 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:23.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f358dcea640 1 Processor -- start 2023-10-30T12:55:23.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f358dcea640 1 -- start start 2023-10-30T12:55:23.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f358dcea640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f35881538a0 0x7f358815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:23.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f358dcea640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f35880cf160 0x7f35880cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:23.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f358dcea640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35880cb800 con 0x7f35880ca0e0 2023-10-30T12:55:23.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f358dcea640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f358815dc30 con 0x7f35881538a0 2023-10-30T12:55:23.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f358dcea640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35880cb330 con 0x7f35880cf160 2023-10-30T12:55:23.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f35877fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f35881538a0 0x7f358815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:23.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f35877fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f35881538a0 0x7f358815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39056/0 (socket says 172.21.15.125:39056) 2023-10-30T12:55:23.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f35877fe640 1 -- 172.21.15.125:0/2381090061 learned_addr learned my addr 172.21.15.125:0/2381090061 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:23.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f3587fff640 1 --2- 172.21.15.125:0/2381090061 >> v2:172.21.15.136:3300/0 conn(0x7f35880cf160 0x7f35880cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:23.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f35867fc640 1 -- 172.21.15.125:0/2381090061 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324715627 0 0) 0x7f35880cb800 con 0x7f35880ca0e0 2023-10-30T12:55:23.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f35867fc640 1 -- 172.21.15.125:0/2381090061 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3570003530 con 0x7f35880ca0e0 2023-10-30T12:55:23.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f3587fff640 1 -- 172.21.15.125:0/2381090061 >> v2:172.21.15.125:3300/0 conn(0x7f35881538a0 msgr2=0x7f358815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:23.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f3587fff640 1 --2- 172.21.15.125:0/2381090061 >> v2:172.21.15.125:3300/0 conn(0x7f35881538a0 0x7f358815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:23.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f3587fff640 1 -- 172.21.15.125:0/2381090061 >> v1:172.21.15.125:6789/0 conn(0x7f35880ca0e0 legacy=0x7f358815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:23.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.610+0000 7f3587fff640 1 -- 172.21.15.125:0/2381090061 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35880cad70 con 0x7f35880cf160 2023-10-30T12:55:23.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.614+0000 7f3587fff640 1 --2- 172.21.15.125:0/2381090061 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35880cf160 0x7f35880cf590 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f35880b5330 tx=0x7f358001f0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d385d473a069ed58 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.614+0000 7f35867fc640 1 -- 172.21.15.125:0/2381090061 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35800b0440 con 0x7f35880cf160 2023-10-30T12:55:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.614+0000 7f35867fc640 1 -- 172.21.15.125:0/2381090061 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35800b05a0 con 0x7f35880cf160 2023-10-30T12:55:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.614+0000 7f35867fc640 1 -- 172.21.15.125:0/2381090061 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35800b0910 con 0x7f35880cf160 2023-10-30T12:55:23.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.614+0000 7f358dcea640 1 -- 172.21.15.125:0/2381090061 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35880cf160 msgr2=0x7f35880cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:23.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.614+0000 7f358dcea640 1 --2- 172.21.15.125:0/2381090061 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35880cf160 0x7f35880cf590 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f35880b5330 tx=0x7f358001f0f0 comp rx=0 tx=0).stop 2023-10-30T12:55:23.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.614+0000 7f358dcea640 1 -- 172.21.15.125:0/2381090061 shutdown_connections 2023-10-30T12:55:23.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.614+0000 7f358dcea640 1 --2- 172.21.15.125:0/2381090061 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35880cf160 0x7f35880cf590 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:23.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.614+0000 7f358dcea640 1 --2- 172.21.15.125:0/2381090061 >> v2:172.21.15.125:3300/0 conn(0x7f35881538a0 0x7f358815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:23.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.614+0000 7f358dcea640 1 -- 172.21.15.125:0/2381090061 >> 172.21.15.125:0/2381090061 conn(0x7f358806bfe0 msgr2=0x7f358806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:23.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.614+0000 7f358dcea640 1 -- 172.21.15.125:0/2381090061 shutdown_connections 2023-10-30T12:55:23.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.614+0000 7f358dcea640 1 -- 172.21.15.125:0/2381090061 wait complete. 2023-10-30T12:55:23.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.618+0000 7f358dcea640 1 Processor -- start 2023-10-30T12:55:23.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.618+0000 7f358dcea640 1 -- start start 2023-10-30T12:55:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f358dcea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35880ca0e0 0x7f3588153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f358dcea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35880cf160 0x7f35881589b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f358dcea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35881538a0 0x7f358815cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f358dcea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358815c6b0 con 0x7f35880ca0e0 2023-10-30T12:55:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f358dcea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35881566c0 con 0x7f35880cf160 2023-10-30T12:55:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f358dcea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3588159500 con 0x7f35881538a0 2023-10-30T12:55:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f35877fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35880ca0e0 0x7f3588153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f35877fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35880ca0e0 0x7f3588153e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39060/0 (socket says 172.21.15.125:39060) 2023-10-30T12:55:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f35877fe640 1 -- 172.21.15.125:0/2970434800 learned_addr learned my addr 172.21.15.125:0/2970434800 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f3587fff640 1 --2- 172.21.15.125:0/2970434800 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35881538a0 0x7f358815cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f3586ffd640 1 --2- 172.21.15.125:0/2970434800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35880cf160 0x7f35881589b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f35877fe640 1 -- 172.21.15.125:0/2970434800 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35881538a0 msgr2=0x7f358815cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f35877fe640 1 --2- 172.21.15.125:0/2970434800 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35881538a0 0x7f358815cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f35877fe640 1 -- 172.21.15.125:0/2970434800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35880cf160 msgr2=0x7f35881589b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:23.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f35877fe640 1 --2- 172.21.15.125:0/2970434800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35880cf160 0x7f35881589b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:23.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f35877fe640 1 -- 172.21.15.125:0/2970434800 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3580009a70 con 0x7f35880ca0e0 2023-10-30T12:55:23.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f35877fe640 1 --2- 172.21.15.125:0/2970434800 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35880ca0e0 0x7f3588153e10 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f357800e890 tx=0x7f357800ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:23.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f356ffff640 1 -- 172.21.15.125:0/2970434800 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3578009620 con 0x7f35880ca0e0 2023-10-30T12:55:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f356ffff640 1 -- 172.21.15.125:0/2970434800 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357800c940 con 0x7f35880ca0e0 2023-10-30T12:55:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f358dcea640 1 -- 172.21.15.125:0/2970434800 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f358815bd10 con 0x7f35880ca0e0 2023-10-30T12:55:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f356ffff640 1 -- 172.21.15.125:0/2970434800 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357800ccf0 con 0x7f35880ca0e0 2023-10-30T12:55:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f358dcea640 1 -- 172.21.15.125:0/2970434800 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f358815c2d0 con 0x7f35880ca0e0 2023-10-30T12:55:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f358dcea640 1 -- 172.21.15.125:0/2970434800 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35880b2a50 con 0x7f35880ca0e0 2023-10-30T12:55:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.622+0000 7f356ffff640 1 -- 172.21.15.125:0/2970434800 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3578013470 con 0x7f35880ca0e0 2023-10-30T12:55:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.626+0000 7f356ffff640 1 --2- 172.21.15.125:0/2970434800 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35740664a0 0x7f3574068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.626+0000 7f356ffff640 1 -- 172.21.15.125:0/2970434800 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7f3578084560 con 0x7f35880ca0e0 2023-10-30T12:55:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.626+0000 7f3586ffd640 1 --2- 172.21.15.125:0/2970434800 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35740664a0 0x7f3574068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:23.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.626+0000 7f3586ffd640 1 --2- 172.21.15.125:0/2970434800 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35740664a0 0x7f3574068960 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3588071590 tx=0x7f357c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:23.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.630+0000 7f356ffff640 1 -- 172.21.15.125:0/2970434800 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f35780514e0 con 0x7f35880ca0e0 2023-10-30T12:55:23.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.822+0000 7f358dcea640 1 -- 172.21.15.125:0/2970434800 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f358815c920 con 0x7f35880ca0e0 2023-10-30T12:55:23.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.822+0000 7f356ffff640 1 -- 172.21.15.125:0/2970434800 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f3578054b00 con 0x7f35880ca0e0 2023-10-30T12:55:23.826 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:23.826 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:15.400801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:23.826 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 124 2023-10-30T12:55:23.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.826+0000 7f358dcea640 1 -- 172.21.15.125:0/2970434800 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35740664a0 msgr2=0x7f3574068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:23.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.826+0000 7f358dcea640 1 --2- 172.21.15.125:0/2970434800 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35740664a0 0x7f3574068960 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f3588071590 tx=0x7f357c009040 comp rx=0 tx=0).stop 2023-10-30T12:55:23.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.826+0000 7f358dcea640 1 -- 172.21.15.125:0/2970434800 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35880ca0e0 msgr2=0x7f3588153e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:23.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.826+0000 7f358dcea640 1 --2- 172.21.15.125:0/2970434800 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35880ca0e0 0x7f3588153e10 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f357800e890 tx=0x7f357800ed50 comp rx=0 tx=0).stop 2023-10-30T12:55:23.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.826+0000 7f358dcea640 1 -- 172.21.15.125:0/2970434800 shutdown_connections 2023-10-30T12:55:23.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.826+0000 7f358dcea640 1 --2- 172.21.15.125:0/2970434800 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f35740664a0 0x7f3574068960 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:23.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.826+0000 7f358dcea640 1 --2- 172.21.15.125:0/2970434800 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f35881538a0 0x7f358815cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:23.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.826+0000 7f358dcea640 1 --2- 172.21.15.125:0/2970434800 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f35880cf160 0x7f35881589b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:23.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.826+0000 7f358dcea640 1 --2- 172.21.15.125:0/2970434800 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f35880ca0e0 0x7f3588153e10 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:23.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.826+0000 7f358dcea640 1 -- 172.21.15.125:0/2970434800 >> 172.21.15.125:0/2970434800 conn(0x7f358806bfe0 msgr2=0x7f3588151910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:23.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.826+0000 7f358dcea640 1 -- 172.21.15.125:0/2970434800 shutdown_connections 2023-10-30T12:55:23.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:23.826+0000 7f358dcea640 1 -- 172.21.15.125:0/2970434800 wait complete. 2023-10-30T12:55:23.848 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:55:25.850 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:25.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa50bff5640 1 Processor -- start 2023-10-30T12:55:25.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa50bff5640 1 -- start start 2023-10-30T12:55:25.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa50bff5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa5040c6b80 0x7fa5040bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:25.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa50bff5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa50413f4a0 0x7fa5040c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:25.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa50bff5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5040c0a50 con 0x7fa5040c6f60 2023-10-30T12:55:25.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa50bff5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa50406f6e0 con 0x7fa5040c6b80 2023-10-30T12:55:25.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa50bff5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50406f9c0 con 0x7fa50413f4a0 2023-10-30T12:55:25.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa509d91640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa5040c6b80 0x7fa5040bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:25.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa509590640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa5040c6f60 0x7fa50413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36636/0 (socket says 172.21.15.125:36636) 2023-10-30T12:55:25.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa509590640 1 -- 172.21.15.125:0/1875313157 learned_addr learned my addr 172.21.15.125:0/1875313157 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:55:25.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa50a592640 1 --2- 172.21.15.125:0/1875313157 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa50413f4a0 0x7fa5040c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:25.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa509d91640 1 -- 172.21.15.125:0/1875313157 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa50413f4a0 msgr2=0x7fa5040c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:25.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa509d91640 1 --2- 172.21.15.125:0/1875313157 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa50413f4a0 0x7fa5040c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:25.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa509d91640 1 -- 172.21.15.125:0/1875313157 >> v1:172.21.15.125:6789/0 conn(0x7fa5040c6f60 legacy=0x7fa50413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:25.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa509d91640 1 -- 172.21.15.125:0/1875313157 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50406fca0 con 0x7fa5040c6b80 2023-10-30T12:55:26.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa508d8f640 1 -- 172.21.15.125:0/1875313157 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049041167 0 0) 0x7fa5040c0a50 con 0x7fa5040c6f60 2023-10-30T12:55:26.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.994+0000 7fa50a592640 1 --2- 172.21.15.125:0/1875313157 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa50413f4a0 0x7fa5040c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:26.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.998+0000 7fa509d91640 1 --2- 172.21.15.125:0/1875313157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5040c6b80 0x7fa5040bfec0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fa5000097c0 tx=0x7fa50001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab98e664d831133 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:26.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.998+0000 7fa508d8f640 1 -- 172.21.15.125:0/1875313157 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5000ad640 con 0x7fa5040c6b80 2023-10-30T12:55:26.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.998+0000 7fa508d8f640 1 -- 172.21.15.125:0/1875313157 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5000ad7a0 con 0x7fa5040c6b80 2023-10-30T12:55:26.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.998+0000 7fa508d8f640 1 -- 172.21.15.125:0/1875313157 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5000adb30 con 0x7fa5040c6b80 2023-10-30T12:55:26.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.998+0000 7fa50bff5640 1 -- 172.21.15.125:0/1875313157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5040c6b80 msgr2=0x7fa5040bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:26.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.998+0000 7fa50bff5640 1 --2- 172.21.15.125:0/1875313157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5040c6b80 0x7fa5040bfec0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fa5000097c0 tx=0x7fa50001c480 comp rx=0 tx=0).stop 2023-10-30T12:55:26.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.998+0000 7fa50bff5640 1 -- 172.21.15.125:0/1875313157 shutdown_connections 2023-10-30T12:55:26.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.998+0000 7fa50bff5640 1 --2- 172.21.15.125:0/1875313157 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa50413f4a0 0x7fa5040c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:26.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.998+0000 7fa50bff5640 1 --2- 172.21.15.125:0/1875313157 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5040c6b80 0x7fa5040bfec0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:26.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.998+0000 7fa50bff5640 1 -- 172.21.15.125:0/1875313157 >> 172.21.15.125:0/1875313157 conn(0x7fa5040c7910 msgr2=0x7fa5040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:26.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.998+0000 7fa50bff5640 1 -- 172.21.15.125:0/1875313157 shutdown_connections 2023-10-30T12:55:26.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:25.998+0000 7fa50bff5640 1 -- 172.21.15.125:0/1875313157 wait complete. 2023-10-30T12:55:26.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa50bff5640 1 Processor -- start 2023-10-30T12:55:26.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa50bff5640 1 -- start start 2023-10-30T12:55:26.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa50bff5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5040c6b80 0x7fa5040c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:26.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa50bff5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5040c6f60 0x7fa5041342a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:26.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa50bff5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa50413f4a0 0x7fa504136b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:26.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa50bff5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa504155330 con 0x7fa5040c6b80 2023-10-30T12:55:26.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa50bff5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5040ca210 con 0x7fa50413f4a0 2023-10-30T12:55:26.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa50bff5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5040c8c10 con 0x7fa5040c6f60 2023-10-30T12:55:26.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa509d91640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5040c6b80 0x7fa5040c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:26.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa509d91640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5040c6b80 0x7fa5040c9a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39076/0 (socket says 172.21.15.125:39076) 2023-10-30T12:55:26.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa509d91640 1 -- 172.21.15.125:0/1504010125 learned_addr learned my addr 172.21.15.125:0/1504010125 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:26.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa509590640 1 --2- 172.21.15.125:0/1504010125 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5040c6f60 0x7fa5041342a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:26.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa509d91640 1 -- 172.21.15.125:0/1504010125 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5040c6f60 msgr2=0x7fa5041342a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:26.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa509d91640 1 --2- 172.21.15.125:0/1504010125 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5040c6f60 0x7fa5041342a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:26.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa509d91640 1 -- 172.21.15.125:0/1504010125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa50413f4a0 msgr2=0x7fa504136b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:26.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa509d91640 1 --2- 172.21.15.125:0/1504010125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa50413f4a0 0x7fa504136b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:26.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa509d91640 1 -- 172.21.15.125:0/1504010125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa500009420 con 0x7fa5040c6b80 2023-10-30T12:55:26.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa509590640 1 --2- 172.21.15.125:0/1504010125 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5040c6f60 0x7fa5041342a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:55:26.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa509d91640 1 --2- 172.21.15.125:0/1504010125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5040c6b80 0x7fa5040c9a80 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fa50001a0b0 tx=0x7fa500019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:26.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa4fa7fc640 1 -- 172.21.15.125:0/1504010125 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5000b68e0 con 0x7fa5040c6b80 2023-10-30T12:55:26.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa4fa7fc640 1 -- 172.21.15.125:0/1504010125 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5000a6040 con 0x7fa5040c6b80 2023-10-30T12:55:26.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa4fa7fc640 1 -- 172.21.15.125:0/1504010125 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5000b6ca0 con 0x7fa5040c6b80 2023-10-30T12:55:26.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa50bff5640 1 -- 172.21.15.125:0/1504010125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5041347e0 con 0x7fa5040c6b80 2023-10-30T12:55:26.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.002+0000 7fa50bff5640 1 -- 172.21.15.125:0/1504010125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5040ca870 con 0x7fa5040c6b80 2023-10-30T12:55:26.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.006+0000 7fa50bff5640 1 -- 172.21.15.125:0/1504010125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa50404e170 con 0x7fa5040c6b80 2023-10-30T12:55:26.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.006+0000 7fa4fa7fc640 1 -- 172.21.15.125:0/1504010125 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa5000b4040 con 0x7fa5040c6b80 2023-10-30T12:55:26.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.006+0000 7fa4fa7fc640 1 --2- 172.21.15.125:0/1504010125 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4e80664a0 0x7fa4e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:26.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.006+0000 7fa4fa7fc640 1 -- 172.21.15.125:0/1504010125 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7fa50011e980 con 0x7fa5040c6b80 2023-10-30T12:55:26.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.006+0000 7fa509590640 1 --2- 172.21.15.125:0/1504010125 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4e80664a0 0x7fa4e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:26.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.006+0000 7fa509590640 1 --2- 172.21.15.125:0/1504010125 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4e80664a0 0x7fa4e8068960 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa5040b0cc0 tx=0x7fa4f4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:26.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.010+0000 7fa4fa7fc640 1 -- 172.21.15.125:0/1504010125 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa5000eb8f0 con 0x7fa5040c6b80 2023-10-30T12:55:26.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.230+0000 7fa50bff5640 1 -- 172.21.15.125:0/1504010125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa50413bfc0 con 0x7fa5040c6b80 2023-10-30T12:55:26.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.230+0000 7fa4fa7fc640 1 -- 172.21.15.125:0/1504010125 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 124 v124) v1 ==== 95+0+5723 (secure 0 0 0) 0x7fa5000eef10 con 0x7fa5040c6b80 2023-10-30T12:55:26.233 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:26.233 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":124,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":124,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:15.400801+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:26.233 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 124 2023-10-30T12:55:26.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.234+0000 7fa50bff5640 1 -- 172.21.15.125:0/1504010125 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4e80664a0 msgr2=0x7fa4e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:26.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.234+0000 7fa50bff5640 1 --2- 172.21.15.125:0/1504010125 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4e80664a0 0x7fa4e8068960 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fa5040b0cc0 tx=0x7fa4f4009040 comp rx=0 tx=0).stop 2023-10-30T12:55:26.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.234+0000 7fa50bff5640 1 -- 172.21.15.125:0/1504010125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5040c6b80 msgr2=0x7fa5040c9a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:26.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.234+0000 7fa50bff5640 1 --2- 172.21.15.125:0/1504010125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5040c6b80 0x7fa5040c9a80 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fa50001a0b0 tx=0x7fa500019220 comp rx=0 tx=0).stop 2023-10-30T12:55:26.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.234+0000 7fa50bff5640 1 -- 172.21.15.125:0/1504010125 shutdown_connections 2023-10-30T12:55:26.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.234+0000 7fa50bff5640 1 --2- 172.21.15.125:0/1504010125 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa4e80664a0 0x7fa4e8068960 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:26.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.234+0000 7fa50bff5640 1 --2- 172.21.15.125:0/1504010125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa50413f4a0 0x7fa504136b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:26.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.234+0000 7fa50bff5640 1 --2- 172.21.15.125:0/1504010125 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5040c6f60 0x7fa5041342a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:26.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.234+0000 7fa50bff5640 1 --2- 172.21.15.125:0/1504010125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa5040c6b80 0x7fa5040c9a80 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:26.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.234+0000 7fa50bff5640 1 -- 172.21.15.125:0/1504010125 >> 172.21.15.125:0/1504010125 conn(0x7fa5040c7910 msgr2=0x7fa5040c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:26.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.234+0000 7fa50bff5640 1 -- 172.21.15.125:0/1504010125 shutdown_connections 2023-10-30T12:55:26.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:26.234+0000 7fa50bff5640 1 -- 172.21.15.125:0/1504010125 wait complete. 2023-10-30T12:55:26.253 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:55:28.254 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:28.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.398+0000 7fafb386b640 1 Processor -- start 2023-10-30T12:55:28.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.398+0000 7fafb386b640 1 -- start start 2023-10-30T12:55:28.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.402+0000 7fafb386b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fafac0c6b80 0x7fafac0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:28.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.402+0000 7fafb386b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fafac13f4a0 0x7fafac0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:28.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.402+0000 7fafb386b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafac0c0a50 con 0x7fafac0c6f60 2023-10-30T12:55:28.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.402+0000 7fafb386b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafac06f6e0 con 0x7fafac13f4a0 2023-10-30T12:55:28.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.402+0000 7fafb386b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafac06f9c0 con 0x7fafac0c6b80 2023-10-30T12:55:28.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.402+0000 7fafb1e08640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fafac13f4a0 0x7fafac0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:28.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.402+0000 7fafb1e08640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fafac13f4a0 0x7fafac0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39090/0 (socket says 172.21.15.125:39090) 2023-10-30T12:55:28.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.402+0000 7fafb1e08640 1 -- 172.21.15.125:0/2038287011 learned_addr learned my addr 172.21.15.125:0/2038287011 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:28.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.402+0000 7fafb1607640 1 --2- 172.21.15.125:0/2038287011 >> v2:172.21.15.136:3300/0 conn(0x7fafac0c6b80 0x7fafac0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:28.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.402+0000 7fafb1e08640 1 -- 172.21.15.125:0/2038287011 >> v2:172.21.15.136:3300/0 conn(0x7fafac0c6b80 msgr2=0x7fafac0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:28.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.402+0000 7fafb1e08640 1 --2- 172.21.15.125:0/2038287011 >> v2:172.21.15.136:3300/0 conn(0x7fafac0c6b80 0x7fafac0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:28.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.402+0000 7fafb1e08640 1 -- 172.21.15.125:0/2038287011 >> v1:172.21.15.136:6789/0 conn(0x7fafac0c6f60 legacy=0x7fafac13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:28.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.402+0000 7fafb1e08640 1 -- 172.21.15.125:0/2038287011 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac06fca0 con 0x7fafac13f4a0 2023-10-30T12:55:28.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.406+0000 7fafb1e08640 1 --2- 172.21.15.125:0/2038287011 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac13f4a0 0x7fafac0c0400 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fafac06f1b0 tx=0x7fafa401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c49dbc538cfa84ce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:28.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.406+0000 7fafa3fff640 1 -- 172.21.15.125:0/2038287011 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa40a57a0 con 0x7fafac13f4a0 2023-10-30T12:55:28.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.406+0000 7fafa3fff640 1 -- 172.21.15.125:0/2038287011 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa40a5900 con 0x7fafac13f4a0 2023-10-30T12:55:28.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.406+0000 7fafa3fff640 1 -- 172.21.15.125:0/2038287011 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa40a5c90 con 0x7fafac13f4a0 2023-10-30T12:55:28.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.406+0000 7fafb386b640 1 -- 172.21.15.125:0/2038287011 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac13f4a0 msgr2=0x7fafac0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:28.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.406+0000 7fafb386b640 1 --2- 172.21.15.125:0/2038287011 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac13f4a0 0x7fafac0c0400 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fafac06f1b0 tx=0x7fafa401c9d0 comp rx=0 tx=0).stop 2023-10-30T12:55:28.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.406+0000 7fafb386b640 1 -- 172.21.15.125:0/2038287011 shutdown_connections 2023-10-30T12:55:28.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.406+0000 7fafb386b640 1 --2- 172.21.15.125:0/2038287011 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac13f4a0 0x7fafac0c0400 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:28.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.406+0000 7fafb386b640 1 --2- 172.21.15.125:0/2038287011 >> v2:172.21.15.136:3300/0 conn(0x7fafac0c6b80 0x7fafac0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:28.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.406+0000 7fafb386b640 1 -- 172.21.15.125:0/2038287011 >> 172.21.15.125:0/2038287011 conn(0x7fafac0c7910 msgr2=0x7fafac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:28.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.406+0000 7fafb386b640 1 -- 172.21.15.125:0/2038287011 shutdown_connections 2023-10-30T12:55:28.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.406+0000 7fafb386b640 1 -- 172.21.15.125:0/2038287011 wait complete. 2023-10-30T12:55:28.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.406+0000 7fafb386b640 1 Processor -- start 2023-10-30T12:55:28.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.406+0000 7fafb386b640 1 -- start start 2023-10-30T12:55:28.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb386b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac0c6b80 0x7fafac13cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:28.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb386b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fafac0c6f60 0x7fafac13d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:28.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb386b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafac13f4a0 0x7fafac0c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:28.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb386b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac1f03c0 con 0x7fafac0c6b80 2023-10-30T12:55:28.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb386b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac13caa0 con 0x7fafac13f4a0 2023-10-30T12:55:28.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb386b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafac155f40 con 0x7fafac0c6f60 2023-10-30T12:55:28.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb1607640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac0c6b80 0x7fafac13cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:28.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb1607640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac0c6b80 0x7fafac13cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39092/0 (socket says 172.21.15.125:39092) 2023-10-30T12:55:28.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb1607640 1 -- 172.21.15.125:0/2874463568 learned_addr learned my addr 172.21.15.125:0/2874463568 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:28.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb0e06640 1 --2- 172.21.15.125:0/2874463568 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fafac0c6f60 0x7fafac13d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:28.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb1e08640 1 --2- 172.21.15.125:0/2874463568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafac13f4a0 0x7fafac0c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:28.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb1607640 1 -- 172.21.15.125:0/2874463568 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fafac0c6f60 msgr2=0x7fafac13d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:28.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb1607640 1 --2- 172.21.15.125:0/2874463568 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fafac0c6f60 0x7fafac13d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:28.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb1607640 1 -- 172.21.15.125:0/2874463568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafac13f4a0 msgr2=0x7fafac0c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:28.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb1607640 1 --2- 172.21.15.125:0/2874463568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafac13f4a0 0x7fafac0c4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:28.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb1607640 1 -- 172.21.15.125:0/2874463568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa4009b30 con 0x7fafac0c6b80 2023-10-30T12:55:28.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb1607640 1 --2- 172.21.15.125:0/2874463568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac0c6b80 0x7fafac13cf30 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fafa800e9e0 tx=0x7fafa800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:28.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafa1ffb640 1 -- 172.21.15.125:0/2874463568 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa80096b0 con 0x7fafac0c6b80 2023-10-30T12:55:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafa1ffb640 1 -- 172.21.15.125:0/2874463568 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa800c9d0 con 0x7fafac0c6b80 2023-10-30T12:55:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafa1ffb640 1 -- 172.21.15.125:0/2874463568 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa800cd60 con 0x7fafac0c6b80 2023-10-30T12:55:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb386b640 1 -- 172.21.15.125:0/2874463568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafac1f08b0 con 0x7fafac0c6b80 2023-10-30T12:55:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb386b640 1 -- 172.21.15.125:0/2874463568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafac136200 con 0x7fafac0c6b80 2023-10-30T12:55:28.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.410+0000 7fafb386b640 1 -- 172.21.15.125:0/2874463568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafac04e170 con 0x7fafac0c6b80 2023-10-30T12:55:28.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.414+0000 7fafa1ffb640 1 -- 172.21.15.125:0/2874463568 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fafa8013470 con 0x7fafac0c6b80 2023-10-30T12:55:28.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.414+0000 7fafa1ffb640 1 --2- 172.21.15.125:0/2874463568 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf900664a0 0x7faf90068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:28.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.414+0000 7fafa1ffb640 1 -- 172.21.15.125:0/2874463568 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7fafa80845d0 con 0x7fafac0c6b80 2023-10-30T12:55:28.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.414+0000 7fafb0e06640 1 --2- 172.21.15.125:0/2874463568 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf900664a0 0x7faf90068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:28.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.414+0000 7fafb0e06640 1 --2- 172.21.15.125:0/2874463568 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf900664a0 0x7faf90068960 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fafac13f140 tx=0x7faf9c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:28.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.418+0000 7fafa1ffb640 1 -- 172.21.15.125:0/2874463568 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fafa80515e0 con 0x7fafac0c6b80 2023-10-30T12:55:28.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.598+0000 7fafb386b640 1 -- 172.21.15.125:0/2874463568 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafac13c330 con 0x7fafac0c6b80 2023-10-30T12:55:28.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.598+0000 7fafa1ffb640 1 -- 172.21.15.125:0/2874463568 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5727 (secure 0 0 0) 0x7fafa8054c00 con 0x7fafac0c6b80 2023-10-30T12:55:28.602 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:28.602 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:25.975685+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:28.603 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 125 2023-10-30T12:55:28.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.602+0000 7fafb386b640 1 -- 172.21.15.125:0/2874463568 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf900664a0 msgr2=0x7faf90068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:28.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.602+0000 7fafb386b640 1 --2- 172.21.15.125:0/2874463568 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf900664a0 0x7faf90068960 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fafac13f140 tx=0x7faf9c009040 comp rx=0 tx=0).stop 2023-10-30T12:55:28.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.602+0000 7fafb386b640 1 -- 172.21.15.125:0/2874463568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac0c6b80 msgr2=0x7fafac13cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:28.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.602+0000 7fafb386b640 1 --2- 172.21.15.125:0/2874463568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac0c6b80 0x7fafac13cf30 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fafa800e9e0 tx=0x7fafa800eea0 comp rx=0 tx=0).stop 2023-10-30T12:55:28.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.602+0000 7fafb386b640 1 -- 172.21.15.125:0/2874463568 shutdown_connections 2023-10-30T12:55:28.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.602+0000 7fafb386b640 1 --2- 172.21.15.125:0/2874463568 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf900664a0 0x7faf90068960 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:28.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.602+0000 7fafb386b640 1 --2- 172.21.15.125:0/2874463568 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafac13f4a0 0x7fafac0c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:28.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.602+0000 7fafb386b640 1 --2- 172.21.15.125:0/2874463568 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fafac0c6f60 0x7fafac13d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:28.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.602+0000 7fafb386b640 1 --2- 172.21.15.125:0/2874463568 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac0c6b80 0x7fafac13cf30 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:28.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.602+0000 7fafb386b640 1 -- 172.21.15.125:0/2874463568 >> 172.21.15.125:0/2874463568 conn(0x7fafac0c7910 msgr2=0x7fafac0c0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:28.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.602+0000 7fafb386b640 1 -- 172.21.15.125:0/2874463568 shutdown_connections 2023-10-30T12:55:28.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:28.602+0000 7fafb386b640 1 -- 172.21.15.125:0/2874463568 wait complete. 2023-10-30T12:55:28.623 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 125, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 125, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7113": { "addr": "172.21.15.125:6837/1348687922", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 1348687922, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 1348687922, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7113, "incarnation": 124, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:55:25.975685+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7113 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:55:30.625 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:30.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.770+0000 7f007d017640 1 Processor -- start 2023-10-30T12:55:30.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.770+0000 7f007d017640 1 -- start start 2023-10-30T12:55:30.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.774+0000 7f007d017640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00780c74b0 0x7f0078069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:30.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.774+0000 7f007d017640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0078153f80 con 0x7f0078153ba0 2023-10-30T12:55:30.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.774+0000 7f007d017640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00780b0e60 con 0x7f00780c7df0 2023-10-30T12:55:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.774+0000 7f007d017640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00780c1f50 con 0x7f00780c74b0 2023-10-30T12:55:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.774+0000 7f006ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00780c74b0 0x7f0078069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.774+0000 7f006ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00780c74b0 0x7f0078069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50124/0 (socket says 172.21.15.125:50124) 2023-10-30T12:55:30.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.774+0000 7f006ffff640 1 -- 172.21.15.125:0/3807774121 learned_addr learned my addr 172.21.15.125:0/3807774121 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:30.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.774+0000 7f006ffff640 1 -- 172.21.15.125:0/3807774121 >> v1:172.21.15.136:6789/0 conn(0x7f00780c7df0 legacy=0x7f0078164160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:30.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.774+0000 7f006ffff640 1 -- 172.21.15.125:0/3807774121 >> v1:172.21.15.125:6789/0 conn(0x7f0078153ba0 legacy=0x7f007815a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:30.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.774+0000 7f006ffff640 1 -- 172.21.15.125:0/3807774121 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00780b1140 con 0x7f00780c74b0 2023-10-30T12:55:30.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.774+0000 7f007659c640 1 -- 172.21.15.125:0/3807774121 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103807686 0 0) 0x7f0078153f80 con 0x7f0078153ba0 2023-10-30T12:55:30.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.778+0000 7f006ffff640 1 --2- 172.21.15.125:0/3807774121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00780c74b0 0x7f0078069e90 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f0060009870 tx=0x7f006001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad85f87c36e40706 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:30.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.778+0000 7f007659c640 1 -- 172.21.15.125:0/3807774121 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00600ab640 con 0x7f00780c74b0 2023-10-30T12:55:30.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.778+0000 7f007659c640 1 -- 172.21.15.125:0/3807774121 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00600a7070 con 0x7f00780c74b0 2023-10-30T12:55:30.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.778+0000 7f007659c640 1 -- 172.21.15.125:0/3807774121 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00600ab9c0 con 0x7f00780c74b0 2023-10-30T12:55:30.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.778+0000 7f007d017640 1 -- 172.21.15.125:0/3807774121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00780c74b0 msgr2=0x7f0078069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:30.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.778+0000 7f007d017640 1 --2- 172.21.15.125:0/3807774121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00780c74b0 0x7f0078069e90 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f0060009870 tx=0x7f006001c440 comp rx=0 tx=0).stop 2023-10-30T12:55:30.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.778+0000 7f007d017640 1 -- 172.21.15.125:0/3807774121 shutdown_connections 2023-10-30T12:55:30.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.778+0000 7f007d017640 1 --2- 172.21.15.125:0/3807774121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00780c74b0 0x7f0078069e90 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:30.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.778+0000 7f007d017640 1 -- 172.21.15.125:0/3807774121 >> 172.21.15.125:0/3807774121 conn(0x7f007806d770 msgr2=0x7f007806db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:30.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.778+0000 7f007d017640 1 -- 172.21.15.125:0/3807774121 shutdown_connections 2023-10-30T12:55:30.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.778+0000 7f007d017640 1 -- 172.21.15.125:0/3807774121 wait complete. 2023-10-30T12:55:30.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.778+0000 7f007d017640 1 Processor -- start 2023-10-30T12:55:30.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.778+0000 7f007d017640 1 -- start start 2023-10-30T12:55:30.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007d017640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00780c74b0 0x7f00781509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:30.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007d017640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00780c7df0 0x7f007814fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:30.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007d017640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0078153ba0 0x7f00781529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:30.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007d017640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00781525c0 con 0x7f0078153ba0 2023-10-30T12:55:30.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007d017640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0078152270 con 0x7f00780c7df0 2023-10-30T12:55:30.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007d017640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00781f2830 con 0x7f00780c74b0 2023-10-30T12:55:30.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007759e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0078153ba0 0x7f00781529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:30.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007759e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0078153ba0 0x7f00781529e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50126/0 (socket says 172.21.15.125:50126) 2023-10-30T12:55:30.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007759e640 1 -- 172.21.15.125:0/3918266338 learned_addr learned my addr 172.21.15.125:0/3918266338 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:30.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f0076d9d640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00780c74b0 0x7f00781509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:30.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f006ffff640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00780c7df0 0x7f007814fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:30.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007759e640 1 -- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00780c74b0 msgr2=0x7f00781509a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:30.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007759e640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00780c74b0 0x7f00781509a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:30.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007759e640 1 -- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00780c7df0 msgr2=0x7f007814fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:30.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007759e640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00780c7df0 0x7f007814fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:30.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007759e640 1 -- 172.21.15.125:0/3918266338 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00600094d0 con 0x7f0078153ba0 2023-10-30T12:55:30.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f0076d9d640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00780c74b0 0x7f00781509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:30.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f006ffff640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00780c7df0 0x7f007814fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:30.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007759e640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0078153ba0 0x7f00781529e0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f007000c780 tx=0x7f007000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:30.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f006f7fe640 1 -- 172.21.15.125:0/3918266338 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0070004b70 con 0x7f0078153ba0 2023-10-30T12:55:30.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f006f7fe640 1 -- 172.21.15.125:0/3918266338 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f007000ce90 con 0x7f0078153ba0 2023-10-30T12:55:30.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f006f7fe640 1 -- 172.21.15.125:0/3918266338 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007000a850 con 0x7f0078153ba0 2023-10-30T12:55:30.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007d017640 1 -- 172.21.15.125:0/3918266338 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00781f2b10 con 0x7f0078153ba0 2023-10-30T12:55:30.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007d017640 1 -- 172.21.15.125:0/3918266338 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00781f30d0 con 0x7f0078153ba0 2023-10-30T12:55:30.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f007d017640 1 -- 172.21.15.125:0/3918266338 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0078074250 con 0x7f0078153ba0 2023-10-30T12:55:30.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.782+0000 7f006f7fe640 1 -- 172.21.15.125:0/3918266338 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f007000a9b0 con 0x7f0078153ba0 2023-10-30T12:55:30.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.786+0000 7f006f7fe640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00580664a0 0x7f0058068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:30.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.786+0000 7f006f7fe640 1 -- 172.21.15.125:0/3918266338 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7f0070084570 con 0x7f0078153ba0 2023-10-30T12:55:30.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.786+0000 7f0076d9d640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00580664a0 0x7f0058068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:30.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.786+0000 7f0076d9d640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00580664a0 0x7f0058068960 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0068000f70 tx=0x7f0068009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:30.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.790+0000 7f006f7fe640 1 -- 172.21.15.125:0/3918266338 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f00700514e0 con 0x7f0078153ba0 2023-10-30T12:55:30.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.974+0000 7f007d017640 1 -- 172.21.15.125:0/3918266338 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00780c68f0 con 0x7f0078153ba0 2023-10-30T12:55:30.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.974+0000 7f006f7fe640 1 -- 172.21.15.125:0/3918266338 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 125 v125) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f00780c68f0 con 0x7f0078153ba0 2023-10-30T12:55:30.975 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:30.976 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":125,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":125,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:25.975685+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:reconnect","state_seq":36,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:30.976 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 125 2023-10-30T12:55:30.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.974+0000 7f007d017640 1 -- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00580664a0 msgr2=0x7f0058068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:30.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.974+0000 7f007d017640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00580664a0 0x7f0058068960 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f0068000f70 tx=0x7f0068009040 comp rx=0 tx=0).stop 2023-10-30T12:55:30.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.974+0000 7f007d017640 1 -- 172.21.15.125:0/3918266338 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0078153ba0 msgr2=0x7f00781529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:30.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.974+0000 7f007d017640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0078153ba0 0x7f00781529e0 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f007000c780 tx=0x7f007000cc40 comp rx=0 tx=0).stop 2023-10-30T12:55:30.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.974+0000 7f007d017640 1 -- 172.21.15.125:0/3918266338 shutdown_connections 2023-10-30T12:55:30.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.974+0000 7f007d017640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00580664a0 0x7f0058068960 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:30.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.974+0000 7f007d017640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0078153ba0 0x7f00781529e0 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:30.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.974+0000 7f007d017640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00780c7df0 0x7f007814fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:30.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.974+0000 7f007d017640 1 --2- 172.21.15.125:0/3918266338 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00780c74b0 0x7f00781509a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:30.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.974+0000 7f007d017640 1 -- 172.21.15.125:0/3918266338 >> 172.21.15.125:0/3918266338 conn(0x7f007806d770 msgr2=0x7f00780c9240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:30.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.974+0000 7f007d017640 1 -- 172.21.15.125:0/3918266338 shutdown_connections 2023-10-30T12:55:30.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:30.974+0000 7f007d017640 1 -- 172.21.15.125:0/3918266338 wait complete. 2023-10-30T12:55:30.996 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:55:32.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:33.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.150+0000 7f6fca017640 1 Processor -- start 2023-10-30T12:55:33.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.150+0000 7f6fca017640 1 -- start start 2023-10-30T12:55:33.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.150+0000 7f6fca017640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fc40c6bd0 0x7f6fc406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:33.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.150+0000 7f6fca017640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc4066cf0 con 0x7f6fc4071110 2023-10-30T12:55:33.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.154+0000 7f6fca017640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fc40c70d0 con 0x7f6fc406f7a0 2023-10-30T12:55:33.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.154+0000 7f6fca017640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fc40700c0 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.154+0000 7f6fc2ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fc40c6bd0 0x7f6fc406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:33.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.154+0000 7f6fc3fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6fc4071110 0x7f6fc4158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35674/0 (socket says 172.21.15.125:35674) 2023-10-30T12:55:33.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.154+0000 7f6fc2ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6fc40c6bd0 0x7f6fc406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50130/0 (socket says 172.21.15.125:50130) 2023-10-30T12:55:33.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.154+0000 7f6fc3fff640 1 -- 172.21.15.125:0/2239353357 learned_addr learned my addr 172.21.15.125:0/2239353357 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:55:33.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.154+0000 7f6fc2ffd640 1 -- 172.21.15.125:0/2239353357 >> v1:172.21.15.136:6789/0 conn(0x7f6fc406f7a0 legacy=0x7f6fc4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:33.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.154+0000 7f6fc2ffd640 1 -- 172.21.15.125:0/2239353357 >> v1:172.21.15.125:6789/0 conn(0x7f6fc4071110 legacy=0x7f6fc4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:33.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.154+0000 7f6fc27fc640 1 -- 172.21.15.125:0/2239353357 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 193508291 0 0) 0x7f6fc4066cf0 con 0x7f6fc4071110 2023-10-30T12:55:33.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.154+0000 7f6fc2ffd640 1 -- 172.21.15.125:0/2239353357 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fc40c7390 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.154+0000 7f6fc2ffd640 1 --2- 172.21.15.125:0/2239353357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc40c6bd0 0x7f6fc406fb80 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80097c0 tx=0x7f6fb801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f83ca0aec85d6c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:33.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.154+0000 7f6fc27fc640 1 -- 172.21.15.125:0/2239353357 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb80a8970 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.154+0000 7f6fc27fc640 1 -- 172.21.15.125:0/2239353357 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb80a8ad0 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.154+0000 7f6fc27fc640 1 -- 172.21.15.125:0/2239353357 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb80a8e60 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 -- 172.21.15.125:0/2239353357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc40c6bd0 msgr2=0x7f6fc406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:33.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 --2- 172.21.15.125:0/2239353357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc40c6bd0 0x7f6fc406fb80 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f6fb80097c0 tx=0x7f6fb801ec70 comp rx=0 tx=0).stop 2023-10-30T12:55:33.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 -- 172.21.15.125:0/2239353357 shutdown_connections 2023-10-30T12:55:33.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 --2- 172.21.15.125:0/2239353357 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc40c6bd0 0x7f6fc406fb80 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:33.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 -- 172.21.15.125:0/2239353357 >> 172.21.15.125:0/2239353357 conn(0x7f6fc40c7910 msgr2=0x7f6fc40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:33.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 -- 172.21.15.125:0/2239353357 shutdown_connections 2023-10-30T12:55:33.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 -- 172.21.15.125:0/2239353357 wait complete. 2023-10-30T12:55:33.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 Processor -- start 2023-10-30T12:55:33.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 -- start start 2023-10-30T12:55:33.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fc406f7a0 0x7f6fc41eeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:33.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fc4071110 0x7f6fc40ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:33.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc40c6bd0 0x7f6fc41489e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:33.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc41569c0 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc41f2b00 con 0x7f6fc4071110 2023-10-30T12:55:33.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fca017640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fc41585c0 con 0x7f6fc406f7a0 2023-10-30T12:55:33.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fc3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc40c6bd0 0x7f6fc41489e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:33.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fc3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc40c6bd0 0x7f6fc41489e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50146/0 (socket says 172.21.15.125:50146) 2023-10-30T12:55:33.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fc3fff640 1 -- 172.21.15.125:0/1585095325 learned_addr learned my addr 172.21.15.125:0/1585095325 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:33.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fc2ffd640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fc4071110 0x7f6fc40ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:33.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.158+0000 7f6fc37fe640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fc406f7a0 0x7f6fc41eeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:33.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fc3fff640 1 -- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fc406f7a0 msgr2=0x7f6fc41eeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:33.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fc3fff640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fc406f7a0 0x7f6fc41eeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:33.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fc3fff640 1 -- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fc4071110 msgr2=0x7f6fc40ca120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:33.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fc3fff640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fc4071110 0x7f6fc40ca120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:33.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fc3fff640 1 -- 172.21.15.125:0/1585095325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb8009420 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fc2ffd640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fc4071110 0x7f6fc40ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:33.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fc3fff640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc40c6bd0 0x7f6fc41489e0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00c7b0 tx=0x7f6fbc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:33.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fabfff640 1 -- 172.21.15.125:0/1585095325 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00a640 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fabfff640 1 -- 172.21.15.125:0/1585095325 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc00a7c0 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fca017640 1 -- 172.21.15.125:0/1585095325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fc4158000 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fabfff640 1 -- 172.21.15.125:0/1585095325 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc00ab70 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fca017640 1 -- 172.21.15.125:0/1585095325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fc41f2440 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fca017640 1 -- 172.21.15.125:0/1585095325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fc41f2c40 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fabfff640 1 -- 172.21.15.125:0/1585095325 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6fbc00acd0 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fabfff640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fb00664a0 0x7f6fb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:33.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fabfff640 1 -- 172.21.15.125:0/1585095325 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7f6fbc083960 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.162+0000 7f6fc37fe640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fb00664a0 0x7f6fb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:33.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.166+0000 7f6fc37fe640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fb00664a0 0x7f6fb0068960 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4007fd0 tx=0x7f6fb4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:33.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.166+0000 7f6fabfff640 1 -- 172.21.15.125:0/1585095325 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6fc41f2c40 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.346+0000 7f6fca017640 1 -- 172.21.15.125:0/1585095325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fc41f2c40 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.346+0000 7f6fabfff640 1 -- 172.21.15.125:0/1585095325 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f6fc41f2c40 con 0x7f6fc40c6bd0 2023-10-30T12:55:33.350 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:33.350 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:31.461386+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:33.350 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 126 2023-10-30T12:55:33.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.350+0000 7f6fca017640 1 -- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fb00664a0 msgr2=0x7f6fb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:33.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.350+0000 7f6fca017640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fb00664a0 0x7f6fb0068960 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f6fb4007fd0 tx=0x7f6fb4009040 comp rx=0 tx=0).stop 2023-10-30T12:55:33.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.350+0000 7f6fca017640 1 -- 172.21.15.125:0/1585095325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc40c6bd0 msgr2=0x7f6fc41489e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:33.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.350+0000 7f6fca017640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc40c6bd0 0x7f6fc41489e0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc00c7b0 tx=0x7f6fbc00cc70 comp rx=0 tx=0).stop 2023-10-30T12:55:33.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.350+0000 7f6fca017640 1 -- 172.21.15.125:0/1585095325 shutdown_connections 2023-10-30T12:55:33.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.350+0000 7f6fca017640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fb00664a0 0x7f6fb0068960 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:33.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.350+0000 7f6fca017640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fc40c6bd0 0x7f6fc41489e0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:33.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.350+0000 7f6fca017640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fc4071110 0x7f6fc40ca120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:33.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.350+0000 7f6fca017640 1 --2- 172.21.15.125:0/1585095325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fc406f7a0 0x7f6fc41eeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:33.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.350+0000 7f6fca017640 1 -- 172.21.15.125:0/1585095325 >> 172.21.15.125:0/1585095325 conn(0x7f6fc40c7910 msgr2=0x7f6fc40c9270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:33.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.350+0000 7f6fca017640 1 -- 172.21.15.125:0/1585095325 shutdown_connections 2023-10-30T12:55:33.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:33.350+0000 7f6fca017640 1 -- 172.21.15.125:0/1585095325 wait complete. 2023-10-30T12:55:33.371 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:55:35.373 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:35.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.518+0000 7fb020409640 1 Processor -- start 2023-10-30T12:55:35.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.518+0000 7fb020409640 1 -- start start 2023-10-30T12:55:35.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.522+0000 7fb020409640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0180c6bd0 0x7fb01806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:35.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.522+0000 7fb020409640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb018071110 0x7fb018158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:35.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.522+0000 7fb020409640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb018066cf0 con 0x7fb01806f7a0 2023-10-30T12:55:35.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.522+0000 7fb020409640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0180700c0 con 0x7fb018071110 2023-10-30T12:55:35.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.522+0000 7fb020409640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180c70d0 con 0x7fb0180c6bd0 2023-10-30T12:55:35.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.522+0000 7fb01e9a6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb018071110 0x7fb018158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:35.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.522+0000 7fb01e9a6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb018071110 0x7fb018158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50158/0 (socket says 172.21.15.125:50158) 2023-10-30T12:55:35.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.522+0000 7fb01e9a6640 1 -- 172.21.15.125:0/3890782104 learned_addr learned my addr 172.21.15.125:0/3890782104 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:35.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.522+0000 7fb01d9a4640 1 --2- 172.21.15.125:0/3890782104 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0180c6bd0 0x7fb01806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:35.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.522+0000 7fb01e9a6640 1 -- 172.21.15.125:0/3890782104 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0180c6bd0 msgr2=0x7fb01806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:35.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.522+0000 7fb01e9a6640 1 --2- 172.21.15.125:0/3890782104 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0180c6bd0 0x7fb01806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:35.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.522+0000 7fb01e9a6640 1 -- 172.21.15.125:0/3890782104 >> v1:172.21.15.136:6789/0 conn(0x7fb01806f7a0 legacy=0x7fb018069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:35.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.522+0000 7fb01e9a6640 1 -- 172.21.15.125:0/3890782104 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180c7390 con 0x7fb018071110 2023-10-30T12:55:35.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.526+0000 7fb01e9a6640 1 --2- 172.21.15.125:0/3890782104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb018071110 0x7fb018158110 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fb010009e50 tx=0x7fb01001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af925f2e2706ce94 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:35.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.526+0000 7fb01d1a3640 1 -- 172.21.15.125:0/3890782104 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0100af400 con 0x7fb018071110 2023-10-30T12:55:35.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.526+0000 7fb01d1a3640 1 -- 172.21.15.125:0/3890782104 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0100ab050 con 0x7fb018071110 2023-10-30T12:55:35.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.526+0000 7fb01d1a3640 1 -- 172.21.15.125:0/3890782104 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0100af780 con 0x7fb018071110 2023-10-30T12:55:35.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.526+0000 7fb020409640 1 -- 172.21.15.125:0/3890782104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb018071110 msgr2=0x7fb018158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:35.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.526+0000 7fb020409640 1 --2- 172.21.15.125:0/3890782104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb018071110 0x7fb018158110 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fb010009e50 tx=0x7fb01001f1f0 comp rx=0 tx=0).stop 2023-10-30T12:55:35.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.526+0000 7fb020409640 1 -- 172.21.15.125:0/3890782104 shutdown_connections 2023-10-30T12:55:35.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.526+0000 7fb020409640 1 --2- 172.21.15.125:0/3890782104 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb018071110 0x7fb018158110 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:35.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.526+0000 7fb020409640 1 --2- 172.21.15.125:0/3890782104 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0180c6bd0 0x7fb01806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:35.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.526+0000 7fb020409640 1 -- 172.21.15.125:0/3890782104 >> 172.21.15.125:0/3890782104 conn(0x7fb0180c7910 msgr2=0x7fb0180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:35.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.526+0000 7fb020409640 1 -- 172.21.15.125:0/3890782104 shutdown_connections 2023-10-30T12:55:35.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.526+0000 7fb020409640 1 -- 172.21.15.125:0/3890782104 wait complete. 2023-10-30T12:55:35.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.526+0000 7fb020409640 1 Processor -- start 2023-10-30T12:55:35.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.526+0000 7fb020409640 1 -- start start 2023-10-30T12:55:35.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb020409640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb01806f7a0 0x7fb0180c3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:35.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb020409640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb018071110 0x7fb0180ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:35.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb020409640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0180c6bd0 0x7fb0180aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:35.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb020409640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180af8f0 con 0x7fb018071110 2023-10-30T12:55:35.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb020409640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0181ef380 con 0x7fb01806f7a0 2023-10-30T12:55:35.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb020409640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0181ef8b0 con 0x7fb0180c6bd0 2023-10-30T12:55:35.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb01d9a4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb018071110 0x7fb0180ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:35.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb01d9a4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb018071110 0x7fb0180ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50170/0 (socket says 172.21.15.125:50170) 2023-10-30T12:55:35.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb01d9a4640 1 -- 172.21.15.125:0/1995610857 learned_addr learned my addr 172.21.15.125:0/1995610857 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:35.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb01e9a6640 1 --2- 172.21.15.125:0/1995610857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0180c6bd0 0x7fb0180aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:35.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb01e1a5640 1 --2- 172.21.15.125:0/1995610857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb01806f7a0 0x7fb0180c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:35.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb01d9a4640 1 -- 172.21.15.125:0/1995610857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0180c6bd0 msgr2=0x7fb0180aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:35.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb01d9a4640 1 --2- 172.21.15.125:0/1995610857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0180c6bd0 0x7fb0180aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:35.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb01d9a4640 1 -- 172.21.15.125:0/1995610857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb01806f7a0 msgr2=0x7fb0180c3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:35.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb01d9a4640 1 --2- 172.21.15.125:0/1995610857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb01806f7a0 0x7fb0180c3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:35.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb01d9a4640 1 -- 172.21.15.125:0/1995610857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb010009b30 con 0x7fb018071110 2023-10-30T12:55:35.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb01d9a4640 1 --2- 172.21.15.125:0/1995610857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb018071110 0x7fb0180ae640 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fb00800d800 tx=0x7fb00800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:35.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb00effd640 1 -- 172.21.15.125:0/1995610857 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00800b6c0 con 0x7fb018071110 2023-10-30T12:55:35.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb00effd640 1 -- 172.21.15.125:0/1995610857 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00800b820 con 0x7fb018071110 2023-10-30T12:55:35.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb020409640 1 -- 172.21.15.125:0/1995610857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb01814f270 con 0x7fb018071110 2023-10-30T12:55:35.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb00effd640 1 -- 172.21.15.125:0/1995610857 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00800bb90 con 0x7fb018071110 2023-10-30T12:55:35.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb020409640 1 -- 172.21.15.125:0/1995610857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb01814f510 con 0x7fb018071110 2023-10-30T12:55:35.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb020409640 1 -- 172.21.15.125:0/1995610857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01804e170 con 0x7fb018071110 2023-10-30T12:55:35.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.530+0000 7fb00effd640 1 -- 172.21.15.125:0/1995610857 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb00800bcf0 con 0x7fb018071110 2023-10-30T12:55:35.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.534+0000 7fb00effd640 1 --2- 172.21.15.125:0/1995610857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faffc0664d0 0x7faffc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:35.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.534+0000 7fb00effd640 1 -- 172.21.15.125:0/1995610857 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7fb0080848d0 con 0x7fb018071110 2023-10-30T12:55:35.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.534+0000 7fb01e1a5640 1 --2- 172.21.15.125:0/1995610857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faffc0664d0 0x7faffc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:35.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.534+0000 7fb01e1a5640 1 --2- 172.21.15.125:0/1995610857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faffc0664d0 0x7faffc068990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb014000f70 tx=0x7fb014009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:35.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.538+0000 7fb00effd640 1 -- 172.21.15.125:0/1995610857 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb008051830 con 0x7fb018071110 2023-10-30T12:55:35.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.718+0000 7fb020409640 1 -- 172.21.15.125:0/1995610857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0180c6160 con 0x7fb018071110 2023-10-30T12:55:35.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.718+0000 7fb00effd640 1 -- 172.21.15.125:0/1995610857 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5724 (secure 0 0 0) 0x7fb008054e50 con 0x7fb018071110 2023-10-30T12:55:35.722 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:35.722 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:31.461386+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:35.722 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 126 2023-10-30T12:55:35.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.722+0000 7fb020409640 1 -- 172.21.15.125:0/1995610857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faffc0664d0 msgr2=0x7faffc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:35.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.722+0000 7fb020409640 1 --2- 172.21.15.125:0/1995610857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faffc0664d0 0x7faffc068990 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fb014000f70 tx=0x7fb014009040 comp rx=0 tx=0).stop 2023-10-30T12:55:35.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.722+0000 7fb020409640 1 -- 172.21.15.125:0/1995610857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb018071110 msgr2=0x7fb0180ae640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:35.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.722+0000 7fb020409640 1 --2- 172.21.15.125:0/1995610857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb018071110 0x7fb0180ae640 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fb00800d800 tx=0x7fb00800dcc0 comp rx=0 tx=0).stop 2023-10-30T12:55:35.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.722+0000 7fb020409640 1 -- 172.21.15.125:0/1995610857 shutdown_connections 2023-10-30T12:55:35.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.722+0000 7fb020409640 1 --2- 172.21.15.125:0/1995610857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faffc0664d0 0x7faffc068990 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:35.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.722+0000 7fb020409640 1 --2- 172.21.15.125:0/1995610857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0180c6bd0 0x7fb0180aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.722+0000 7fb020409640 1 --2- 172.21.15.125:0/1995610857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb018071110 0x7fb0180ae640 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.722+0000 7fb020409640 1 --2- 172.21.15.125:0/1995610857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb01806f7a0 0x7fb0180c3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.722+0000 7fb020409640 1 -- 172.21.15.125:0/1995610857 >> 172.21.15.125:0/1995610857 conn(0x7fb0180c7910 msgr2=0x7fb0180c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.722+0000 7fb020409640 1 -- 172.21.15.125:0/1995610857 shutdown_connections 2023-10-30T12:55:35.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:35.722+0000 7fb020409640 1 -- 172.21.15.125:0/1995610857 wait complete. 2023-10-30T12:55:35.742 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:55:37.743 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:37.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.878+0000 7fd49011d640 1 Processor -- start 2023-10-30T12:55:37.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.882+0000 7fd49011d640 1 -- start start 2023-10-30T12:55:37.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.882+0000 7fd49011d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4881538a0 0x7fd48815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:37.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.882+0000 7fd49011d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4880ca0e0 0x7fd48815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:37.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.882+0000 7fd49011d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4880cb800 con 0x7fd4880cf160 2023-10-30T12:55:37.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.882+0000 7fd49011d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd48815dc30 con 0x7fd4881538a0 2023-10-30T12:55:37.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.882+0000 7fd49011d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4880cb330 con 0x7fd4880ca0e0 2023-10-30T12:55:37.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.882+0000 7fd48deb9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4881538a0 0x7fd48815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:37.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.882+0000 7fd48deb9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4881538a0 0x7fd48815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50172/0 (socket says 172.21.15.125:50172) 2023-10-30T12:55:37.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.882+0000 7fd48deb9640 1 -- 172.21.15.125:0/607101463 learned_addr learned my addr 172.21.15.125:0/607101463 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:37.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.882+0000 7fd48d6b8640 1 --2- 172.21.15.125:0/607101463 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4880ca0e0 0x7fd48815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:37.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.882+0000 7fd48deb9640 1 -- 172.21.15.125:0/607101463 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4880ca0e0 msgr2=0x7fd48815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:37.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.882+0000 7fd48deb9640 1 --2- 172.21.15.125:0/607101463 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4880ca0e0 0x7fd48815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:37.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.882+0000 7fd48deb9640 1 -- 172.21.15.125:0/607101463 >> v1:172.21.15.136:6789/0 conn(0x7fd4880cf160 legacy=0x7fd4880cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:37.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.882+0000 7fd48deb9640 1 -- 172.21.15.125:0/607101463 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4880cad70 con 0x7fd4881538a0 2023-10-30T12:55:37.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.886+0000 7fd48deb9640 1 --2- 172.21.15.125:0/607101463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4881538a0 0x7fd48815d6f0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fd484009870 tx=0x7fd48401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b000a125b2a78710 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:37.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.886+0000 7fd48ceb7640 1 -- 172.21.15.125:0/607101463 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4840a8a30 con 0x7fd4881538a0 2023-10-30T12:55:37.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.886+0000 7fd48ceb7640 1 -- 172.21.15.125:0/607101463 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4840a8b90 con 0x7fd4881538a0 2023-10-30T12:55:37.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.886+0000 7fd48ceb7640 1 -- 172.21.15.125:0/607101463 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4840b0440 con 0x7fd4881538a0 2023-10-30T12:55:37.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.886+0000 7fd49011d640 1 -- 172.21.15.125:0/607101463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4881538a0 msgr2=0x7fd48815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:37.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.886+0000 7fd49011d640 1 --2- 172.21.15.125:0/607101463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4881538a0 0x7fd48815d6f0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7fd484009870 tx=0x7fd48401ed30 comp rx=0 tx=0).stop 2023-10-30T12:55:37.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.886+0000 7fd49011d640 1 -- 172.21.15.125:0/607101463 shutdown_connections 2023-10-30T12:55:37.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.886+0000 7fd49011d640 1 --2- 172.21.15.125:0/607101463 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4880ca0e0 0x7fd48815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:37.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.886+0000 7fd49011d640 1 --2- 172.21.15.125:0/607101463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4881538a0 0x7fd48815d6f0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:37.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.886+0000 7fd49011d640 1 -- 172.21.15.125:0/607101463 >> 172.21.15.125:0/607101463 conn(0x7fd48806bfe0 msgr2=0x7fd48806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:37.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.886+0000 7fd49011d640 1 -- 172.21.15.125:0/607101463 shutdown_connections 2023-10-30T12:55:37.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.886+0000 7fd49011d640 1 -- 172.21.15.125:0/607101463 wait complete. 2023-10-30T12:55:37.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.886+0000 7fd49011d640 1 Processor -- start 2023-10-30T12:55:37.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.886+0000 7fd49011d640 1 -- start start 2023-10-30T12:55:37.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd49011d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4880ca0e0 0x7fd48815b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:37.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd49011d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4880cf160 0x7fd4881e4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:37.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd49011d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4881538a0 0x7fd4881e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:37.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd49011d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4881e5580 con 0x7fd4881538a0 2023-10-30T12:55:37.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd49011d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48815bc60 con 0x7fd4880cf160 2023-10-30T12:55:37.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd49011d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4881e4670 con 0x7fd4880ca0e0 2023-10-30T12:55:37.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd48e6ba640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4881538a0 0x7fd4881e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:37.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd48e6ba640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4881538a0 0x7fd4881e5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50176/0 (socket says 172.21.15.125:50176) 2023-10-30T12:55:37.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd48e6ba640 1 -- 172.21.15.125:0/4244966433 learned_addr learned my addr 172.21.15.125:0/4244966433 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:37.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd48d6b8640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4880cf160 0x7fd4881e4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:37.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd48deb9640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4880ca0e0 0x7fd48815b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:37.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd48e6ba640 1 -- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4880ca0e0 msgr2=0x7fd48815b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:37.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd48e6ba640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4880ca0e0 0x7fd48815b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:37.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd48e6ba640 1 -- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4880cf160 msgr2=0x7fd4881e4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:37.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd48e6ba640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4880cf160 0x7fd4881e4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:37.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd48e6ba640 1 -- 172.21.15.125:0/4244966433 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4840094d0 con 0x7fd4881538a0 2023-10-30T12:55:37.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd48deb9640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4880ca0e0 0x7fd48815b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:37.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd48d6b8640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4880cf160 0x7fd4881e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:55:37.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd48e6ba640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4881538a0 0x7fd4881e5040 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fd48000c720 tx=0x7fd48000cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:37.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd47e7fc640 1 -- 172.21.15.125:0/4244966433 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd480003b80 con 0x7fd4881538a0 2023-10-30T12:55:37.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd47e7fc640 1 -- 172.21.15.125:0/4244966433 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd480003ce0 con 0x7fd4881538a0 2023-10-30T12:55:37.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd47e7fc640 1 -- 172.21.15.125:0/4244966433 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd48000aa90 con 0x7fd4881538a0 2023-10-30T12:55:37.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd49011d640 1 -- 172.21.15.125:0/4244966433 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd488158840 con 0x7fd4881538a0 2023-10-30T12:55:37.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd49011d640 1 -- 172.21.15.125:0/4244966433 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd488158ae0 con 0x7fd4881538a0 2023-10-30T12:55:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd49011d640 1 -- 172.21.15.125:0/4244966433 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4880b2a50 con 0x7fd4881538a0 2023-10-30T12:55:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.890+0000 7fd47e7fc640 1 -- 172.21.15.125:0/4244966433 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd48000abf0 con 0x7fd4881538a0 2023-10-30T12:55:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.894+0000 7fd47e7fc640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd46c0664a0 0x7fd46c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.894+0000 7fd47e7fc640 1 -- 172.21.15.125:0/4244966433 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7fd480084180 con 0x7fd4881538a0 2023-10-30T12:55:37.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.894+0000 7fd48deb9640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd46c0664a0 0x7fd46c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:37.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.894+0000 7fd48deb9640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd46c0664a0 0x7fd46c068960 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd4840b2000 tx=0x7fd484008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:37.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:37.898+0000 7fd47e7fc640 1 -- 172.21.15.125:0/4244966433 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd480051050 con 0x7fd4881538a0 2023-10-30T12:55:38.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:38.078+0000 7fd49011d640 1 -- 172.21.15.125:0/4244966433 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4881e4820 con 0x7fd4881538a0 2023-10-30T12:55:38.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:38.078+0000 7fd47e7fc640 1 -- 172.21.15.125:0/4244966433 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+5724 (secure 0 0 0) 0x7fd4881e4820 con 0x7fd4881538a0 2023-10-30T12:55:38.080 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:38.080 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:31.461386+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:rejoin","state_seq":38,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:38.081 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 126 2023-10-30T12:55:38.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:38.078+0000 7fd49011d640 1 -- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd46c0664a0 msgr2=0x7fd46c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:38.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:38.078+0000 7fd49011d640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd46c0664a0 0x7fd46c068960 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fd4840b2000 tx=0x7fd484008040 comp rx=0 tx=0).stop 2023-10-30T12:55:38.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:38.078+0000 7fd49011d640 1 -- 172.21.15.125:0/4244966433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4881538a0 msgr2=0x7fd4881e5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:38.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:38.078+0000 7fd49011d640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4881538a0 0x7fd4881e5040 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fd48000c720 tx=0x7fd48000cbe0 comp rx=0 tx=0).stop 2023-10-30T12:55:38.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:38.082+0000 7fd49011d640 1 -- 172.21.15.125:0/4244966433 shutdown_connections 2023-10-30T12:55:38.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:38.082+0000 7fd49011d640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd46c0664a0 0x7fd46c068960 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:38.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:38.082+0000 7fd49011d640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4881538a0 0x7fd4881e5040 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:38.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:38.082+0000 7fd49011d640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4880cf160 0x7fd4881e4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:38.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:38.082+0000 7fd49011d640 1 --2- 172.21.15.125:0/4244966433 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4880ca0e0 0x7fd48815b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:38.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:38.082+0000 7fd49011d640 1 -- 172.21.15.125:0/4244966433 >> 172.21.15.125:0/4244966433 conn(0x7fd48806bfe0 msgr2=0x7fd488065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:38.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:38.082+0000 7fd49011d640 1 -- 172.21.15.125:0/4244966433 shutdown_connections 2023-10-30T12:55:38.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:38.082+0000 7fd49011d640 1 -- 172.21.15.125:0/4244966433 wait complete. 2023-10-30T12:55:38.101 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:55:40.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:40.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.254+0000 7fa6797c5640 1 Processor -- start 2023-10-30T12:55:40.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.254+0000 7fa6797c5640 1 -- start start 2023-10-30T12:55:40.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.254+0000 7fa6797c5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa67414fc30 0x7fa6740c3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:40.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.254+0000 7fa6797c5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa674155ea0 con 0x7fa67406a0f0 2023-10-30T12:55:40.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.254+0000 7fa6797c5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa674156180 con 0x7fa674155a10 2023-10-30T12:55:40.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.254+0000 7fa6797c5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa67406b350 con 0x7fa67414fc30 2023-10-30T12:55:40.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.254+0000 7fa6737fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa67414fc30 0x7fa6740c3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:40.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.254+0000 7fa672ffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa67406a0f0 0x7fa67414f520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35684/0 (socket says 172.21.15.125:35684) 2023-10-30T12:55:40.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.254+0000 7fa672ffd640 1 -- 172.21.15.125:0/1160300061 learned_addr learned my addr 172.21.15.125:0/1160300061 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:55:40.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.254+0000 7fa6737fe640 1 -- 172.21.15.125:0/1160300061 >> v1:172.21.15.136:6789/0 conn(0x7fa674155a10 legacy=0x7fa674071ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:40.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.254+0000 7fa6727fc640 1 -- 172.21.15.125:0/1160300061 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572220737 0 0) 0x7fa674155ea0 con 0x7fa67406a0f0 2023-10-30T12:55:40.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.254+0000 7fa6737fe640 1 -- 172.21.15.125:0/1160300061 >> v1:172.21.15.125:6789/0 conn(0x7fa67406a0f0 legacy=0x7fa67414f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:40.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.254+0000 7fa6737fe640 1 -- 172.21.15.125:0/1160300061 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa674156460 con 0x7fa67414fc30 2023-10-30T12:55:40.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.258+0000 7fa6737fe640 1 --2- 172.21.15.125:0/1160300061 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa67414fc30 0x7fa6740c3640 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fa66c009d90 tx=0x7fa66c01eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a5ebc5e79c821b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:40.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.258+0000 7fa6727fc640 1 -- 172.21.15.125:0/1160300061 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c0a7c30 con 0x7fa67414fc30 2023-10-30T12:55:40.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.258+0000 7fa6727fc640 1 -- 172.21.15.125:0/1160300061 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa66c0a7d90 con 0x7fa67414fc30 2023-10-30T12:55:40.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.258+0000 7fa6727fc640 1 -- 172.21.15.125:0/1160300061 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c0af660 con 0x7fa67414fc30 2023-10-30T12:55:40.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.258+0000 7fa6797c5640 1 -- 172.21.15.125:0/1160300061 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa67414fc30 msgr2=0x7fa6740c3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:40.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.258+0000 7fa6797c5640 1 --2- 172.21.15.125:0/1160300061 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa67414fc30 0x7fa6740c3640 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fa66c009d90 tx=0x7fa66c01eb60 comp rx=0 tx=0).stop 2023-10-30T12:55:40.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.258+0000 7fa6797c5640 1 -- 172.21.15.125:0/1160300061 shutdown_connections 2023-10-30T12:55:40.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.258+0000 7fa6797c5640 1 --2- 172.21.15.125:0/1160300061 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa67414fc30 0x7fa6740c3640 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:40.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.258+0000 7fa6797c5640 1 -- 172.21.15.125:0/1160300061 >> 172.21.15.125:0/1160300061 conn(0x7fa6740cac50 msgr2=0x7fa6740cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:40.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.258+0000 7fa6797c5640 1 -- 172.21.15.125:0/1160300061 shutdown_connections 2023-10-30T12:55:40.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.258+0000 7fa6797c5640 1 -- 172.21.15.125:0/1160300061 wait complete. 2023-10-30T12:55:40.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.258+0000 7fa6797c5640 1 Processor -- start 2023-10-30T12:55:40.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.258+0000 7fa6797c5640 1 -- start start 2023-10-30T12:55:40.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa6797c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa67406a0f0 0x7fa6741436c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:40.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa6797c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa67414fc30 0x7fa674071540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:40.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa6797c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6741f2540 0x7fa674071a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:40.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa6797c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6741f2ad0 con 0x7fa67406a0f0 2023-10-30T12:55:40.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa6797c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6740c6ed0 con 0x7fa6741f2540 2023-10-30T12:55:40.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa6797c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6740c7170 con 0x7fa67414fc30 2023-10-30T12:55:40.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa672ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa67406a0f0 0x7fa6741436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:40.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa672ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa67406a0f0 0x7fa6741436c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50186/0 (socket says 172.21.15.125:50186) 2023-10-30T12:55:40.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa672ffd640 1 -- 172.21.15.125:0/264490888 learned_addr learned my addr 172.21.15.125:0/264490888 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:40.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa66a7fc640 1 --2- 172.21.15.125:0/264490888 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa67414fc30 0x7fa674071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:40.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa6737fe640 1 --2- 172.21.15.125:0/264490888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6741f2540 0x7fa674071a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:40.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa672ffd640 1 -- 172.21.15.125:0/264490888 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa67414fc30 msgr2=0x7fa674071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:40.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa672ffd640 1 --2- 172.21.15.125:0/264490888 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa67414fc30 0x7fa674071540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:40.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa672ffd640 1 -- 172.21.15.125:0/264490888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6741f2540 msgr2=0x7fa674071a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:40.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa672ffd640 1 --2- 172.21.15.125:0/264490888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6741f2540 0x7fa674071a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:40.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa672ffd640 1 -- 172.21.15.125:0/264490888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c009a70 con 0x7fa67406a0f0 2023-10-30T12:55:40.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa672ffd640 1 --2- 172.21.15.125:0/264490888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa67406a0f0 0x7fa6741436c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fa65c00c4a0 tx=0x7fa65c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:40.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa66bfff640 1 -- 172.21.15.125:0/264490888 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c0074e0 con 0x7fa67406a0f0 2023-10-30T12:55:40.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa66bfff640 1 -- 172.21.15.125:0/264490888 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa65c007640 con 0x7fa67406a0f0 2023-10-30T12:55:40.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa66bfff640 1 -- 172.21.15.125:0/264490888 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa65c004a30 con 0x7fa67406a0f0 2023-10-30T12:55:40.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa6797c5640 1 -- 172.21.15.125:0/264490888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6740c50d0 con 0x7fa67406a0f0 2023-10-30T12:55:40.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa6797c5640 1 -- 172.21.15.125:0/264490888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa674143c00 con 0x7fa67406a0f0 2023-10-30T12:55:40.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.262+0000 7fa6797c5640 1 -- 172.21.15.125:0/264490888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa67404e170 con 0x7fa67406a0f0 2023-10-30T12:55:40.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.266+0000 7fa66bfff640 1 -- 172.21.15.125:0/264490888 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa65c004b90 con 0x7fa67406a0f0 2023-10-30T12:55:40.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.266+0000 7fa66bfff640 1 --2- 172.21.15.125:0/264490888 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6540664a0 0x7fa654068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:40.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.266+0000 7fa66bfff640 1 -- 172.21.15.125:0/264490888 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7fa65c0836c0 con 0x7fa67406a0f0 2023-10-30T12:55:40.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.266+0000 7fa66a7fc640 1 --2- 172.21.15.125:0/264490888 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6540664a0 0x7fa654068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:40.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.266+0000 7fa66a7fc640 1 --2- 172.21.15.125:0/264490888 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6540664a0 0x7fa654068960 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa67414e5b0 tx=0x7fa664009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:40.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.270+0000 7fa66bfff640 1 -- 172.21.15.125:0/264490888 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa65c050590 con 0x7fa67406a0f0 2023-10-30T12:55:40.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.454+0000 7fa6797c5640 1 -- 172.21.15.125:0/264490888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa674153330 con 0x7fa67406a0f0 2023-10-30T12:55:40.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.454+0000 7fa66bfff640 1 -- 172.21.15.125:0/264490888 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+5724 (secure 0 0 0) 0x7fa65c018020 con 0x7fa67406a0f0 2023-10-30T12:55:40.455 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:40.455 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:37.637700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":40,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:40.456 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 127 2023-10-30T12:55:40.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.454+0000 7fa6797c5640 1 -- 172.21.15.125:0/264490888 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6540664a0 msgr2=0x7fa654068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:40.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.454+0000 7fa6797c5640 1 --2- 172.21.15.125:0/264490888 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6540664a0 0x7fa654068960 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa67414e5b0 tx=0x7fa664009040 comp rx=0 tx=0).stop 2023-10-30T12:55:40.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.454+0000 7fa6797c5640 1 -- 172.21.15.125:0/264490888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa67406a0f0 msgr2=0x7fa6741436c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:40.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.454+0000 7fa6797c5640 1 --2- 172.21.15.125:0/264490888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa67406a0f0 0x7fa6741436c0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fa65c00c4a0 tx=0x7fa65c00c960 comp rx=0 tx=0).stop 2023-10-30T12:55:40.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.454+0000 7fa6797c5640 1 -- 172.21.15.125:0/264490888 shutdown_connections 2023-10-30T12:55:40.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.454+0000 7fa6797c5640 1 --2- 172.21.15.125:0/264490888 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6540664a0 0x7fa654068960 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:40.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.454+0000 7fa6797c5640 1 --2- 172.21.15.125:0/264490888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6741f2540 0x7fa674071a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:40.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.454+0000 7fa6797c5640 1 --2- 172.21.15.125:0/264490888 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa67414fc30 0x7fa674071540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:40.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.454+0000 7fa6797c5640 1 --2- 172.21.15.125:0/264490888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa67406a0f0 0x7fa6741436c0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:40.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.454+0000 7fa6797c5640 1 -- 172.21.15.125:0/264490888 >> 172.21.15.125:0/264490888 conn(0x7fa6740cac50 msgr2=0x7fa67414ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:40.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.454+0000 7fa6797c5640 1 -- 172.21.15.125:0/264490888 shutdown_connections 2023-10-30T12:55:40.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:40.454+0000 7fa6797c5640 1 -- 172.21.15.125:0/264490888 wait complete. 2023-10-30T12:55:40.476 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=6954 2023-10-30T12:55:40.476 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.e 2023-10-30T12:55:46.372 INFO:tasks.ceph.mds.c.smithi125.stderr:2023-10-30T12:55:46.370+0000 7f5bcfd06640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:55:47.476 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T12:55:47.477 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T12:55:47.477 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T12:55:47.481 INFO:tasks.ceph.mds.e:Started 2023-10-30T12:55:47.481 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:47.566 INFO:tasks.ceph.mds.e.smithi125.stdout:starting mds.e at 2023-10-30T12:55:47.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.618+0000 7f3263a79640 1 Processor -- start 2023-10-30T12:55:47.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.618+0000 7f3263a79640 1 -- start start 2023-10-30T12:55:47.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.618+0000 7f3263a79640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f325c0c6b80 0x7f325c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:47.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.618+0000 7f3263a79640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325c0c0a50 con 0x7f325c0c6f60 2023-10-30T12:55:47.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.618+0000 7f3263a79640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f325c06f6e0 con 0x7f325c13f4a0 2023-10-30T12:55:47.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.618+0000 7f3263a79640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f325c06f9c0 con 0x7f325c0c6b80 2023-10-30T12:55:47.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3261014640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f325c0c6f60 0x7f325c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40590/0 (socket says 172.21.15.125:40590) 2023-10-30T12:55:47.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3261014640 1 -- 172.21.15.125:0/2022476961 learned_addr learned my addr 172.21.15.125:0/2022476961 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:55:47.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3261815640 1 --2- 172.21.15.125:0/2022476961 >> v2:172.21.15.136:3300/0 conn(0x7f325c0c6b80 0x7f325c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:47.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627981656 0 0) 0x7f325c0c0a50 con 0x7f325c0c6f60 2023-10-30T12:55:47.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3248003530 con 0x7f325c0c6f60 2023-10-30T12:55:47.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956825147 0 0) 0x7f325c06f6e0 con 0x7f325c13f4a0 2023-10-30T12:55:47.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f325c0c0a50 con 0x7f325c13f4a0 2023-10-30T12:55:47.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1985341323 0 0) 0x7f3248003530 con 0x7f325c0c6f60 2023-10-30T12:55:47.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 >> v2:172.21.15.136:3300/0 conn(0x7f325c0c6b80 msgr2=0x7f325c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:47.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 --2- 172.21.15.125:0/2022476961 >> v2:172.21.15.136:3300/0 conn(0x7f325c0c6b80 0x7f325c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:47.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 >> v1:172.21.15.136:6789/0 conn(0x7f325c13f4a0 legacy=0x7f325c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:47.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c06fca0 con 0x7f325c0c6f60 2023-10-30T12:55:47.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f324c001680 con 0x7f325c0c6f60 2023-10-30T12:55:47.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 >> v1:172.21.15.125:6789/0 conn(0x7f325c0c6f60 legacy=0x7f325c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:47.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 --2- 172.21.15.125:0/2022476961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3248003b10 0x7f3248003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:47.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 --2- 172.21.15.125:0/2022476961 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32480047d0 0x7f3248004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:47.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 --2- 172.21.15.125:0/2022476961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3248005190 0x7f324800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:47.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c06f9c0 con 0x7f3248003b10 2023-10-30T12:55:47.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f324800c010 con 0x7f3248005190 2023-10-30T12:55:47.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f324800c1d0 con 0x7f32480047d0 2023-10-30T12:55:47.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3261815640 1 --2- 172.21.15.125:0/2022476961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3248003b10 0x7f3248003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:47.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3262016640 1 --2- 172.21.15.125:0/2022476961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3248005190 0x7f324800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:47.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3261014640 1 --2- 172.21.15.125:0/2022476961 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32480047d0 0x7f3248004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:47.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3261815640 1 -- 172.21.15.125:0/2022476961 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32480047d0 msgr2=0x7f3248004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:47.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3261815640 1 --2- 172.21.15.125:0/2022476961 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32480047d0 0x7f3248004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:47.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3261815640 1 -- 172.21.15.125:0/2022476961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3248005190 msgr2=0x7f324800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:47.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3261815640 1 --2- 172.21.15.125:0/2022476961 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3248005190 0x7f324800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:47.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3261815640 1 -- 172.21.15.125:0/2022476961 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3248003530 con 0x7f3248003b10 2023-10-30T12:55:47.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.622+0000 7f3262016640 1 -- 172.21.15.125:0/2022476961 reap_dead start 2023-10-30T12:55:47.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.626+0000 7f3261815640 1 --2- 172.21.15.125:0/2022476961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3248003b10 0x7f3248003f10 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f3258009ae0 tx=0x7f325801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4832b6dad23bfb50 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:47.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.626+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32580a5530 con 0x7f3248003b10 2023-10-30T12:55:47.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.626+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32580a5690 con 0x7f3248003b10 2023-10-30T12:55:47.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.626+0000 7f3260813640 1 -- 172.21.15.125:0/2022476961 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32580a5a40 con 0x7f3248003b10 2023-10-30T12:55:47.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.626+0000 7f3263a79640 1 -- 172.21.15.125:0/2022476961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3248003b10 msgr2=0x7f3248003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:47.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.626+0000 7f3263a79640 1 --2- 172.21.15.125:0/2022476961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3248003b10 0x7f3248003f10 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f3258009ae0 tx=0x7f325801ca50 comp rx=0 tx=0).stop 2023-10-30T12:55:47.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.626+0000 7f3263a79640 1 -- 172.21.15.125:0/2022476961 shutdown_connections 2023-10-30T12:55:47.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.626+0000 7f3263a79640 1 --2- 172.21.15.125:0/2022476961 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3248003b10 0x7f3248003f10 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:47.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.626+0000 7f3263a79640 1 -- 172.21.15.125:0/2022476961 >> 172.21.15.125:0/2022476961 conn(0x7f325c0c7910 msgr2=0x7f325c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:47.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.626+0000 7f3263a79640 1 -- 172.21.15.125:0/2022476961 shutdown_connections 2023-10-30T12:55:47.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.626+0000 7f3263a79640 1 -- 172.21.15.125:0/2022476961 wait complete. 2023-10-30T12:55:47.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.626+0000 7f3263a79640 1 Processor -- start 2023-10-30T12:55:47.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.626+0000 7f3263a79640 1 -- start start 2023-10-30T12:55:47.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3263a79640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3248003b10 0x7f325c137fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:47.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3263a79640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f325c1547b0 0x7f325c138510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:47.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3263a79640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f325c13b660 0x7f325c136630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:47.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3263a79640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c156150 con 0x7f325c1547b0 2023-10-30T12:55:47.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3263a79640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325c1375c0 con 0x7f325c13b660 2023-10-30T12:55:47.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3263a79640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f325c13ce50 con 0x7f3248003b10 2023-10-30T12:55:47.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3261014640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f325c1547b0 0x7f325c138510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:47.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3261014640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f325c1547b0 0x7f325c138510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45584/0 (socket says 172.21.15.125:45584) 2023-10-30T12:55:47.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3261014640 1 -- 172.21.15.125:0/986123193 learned_addr learned my addr 172.21.15.125:0/986123193 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:47.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3261815640 1 --2- 172.21.15.125:0/986123193 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3248003b10 0x7f325c137fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:47.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3262016640 1 --2- 172.21.15.125:0/986123193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f325c13b660 0x7f325c136630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:47.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3261014640 1 -- 172.21.15.125:0/986123193 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3248003b10 msgr2=0x7f325c137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:47.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3261014640 1 --2- 172.21.15.125:0/986123193 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3248003b10 0x7f325c137fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:47.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3261014640 1 -- 172.21.15.125:0/986123193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f325c13b660 msgr2=0x7f325c136630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:47.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3261014640 1 --2- 172.21.15.125:0/986123193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f325c13b660 0x7f325c136630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:47.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3261014640 1 -- 172.21.15.125:0/986123193 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f325c06b020 con 0x7f325c1547b0 2023-10-30T12:55:47.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3261014640 1 --2- 172.21.15.125:0/986123193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f325c1547b0 0x7f325c138510 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f324c001060 tx=0x7f324c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:47.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f32527fc640 1 -- 172.21.15.125:0/986123193 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c00ee60 con 0x7f325c1547b0 2023-10-30T12:55:47.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f32527fc640 1 -- 172.21.15.125:0/986123193 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f324c00c870 con 0x7f325c1547b0 2023-10-30T12:55:47.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f32527fc640 1 -- 172.21.15.125:0/986123193 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324c00cc20 con 0x7f325c1547b0 2023-10-30T12:55:47.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3263a79640 1 -- 172.21.15.125:0/986123193 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325c13bc30 con 0x7f325c1547b0 2023-10-30T12:55:47.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3263a79640 1 -- 172.21.15.125:0/986123193 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325c137b40 con 0x7f325c1547b0 2023-10-30T12:55:47.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f3263a79640 1 -- 172.21.15.125:0/986123193 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f325c137cd0 con 0x7f325c1547b0 2023-10-30T12:55:47.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.630+0000 7f32527fc640 1 -- 172.21.15.125:0/986123193 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f324c01d460 con 0x7f325c1547b0 2023-10-30T12:55:47.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.634+0000 7f32527fc640 1 --2- 172.21.15.125:0/986123193 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32400664a0 0x7f3240068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:47.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.634+0000 7f32527fc640 1 -- 172.21.15.125:0/986123193 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7f324c0841f0 con 0x7f325c1547b0 2023-10-30T12:55:47.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.634+0000 7f3261815640 1 --2- 172.21.15.125:0/986123193 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32400664a0 0x7f3240068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:47.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.634+0000 7f3261815640 1 --2- 172.21.15.125:0/986123193 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32400664a0 0x7f3240068960 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f32580af000 tx=0x7f3258008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:47.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.638+0000 7f32527fc640 1 -- 172.21.15.125:0/986123193 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f324c0511f0 con 0x7f325c1547b0 2023-10-30T12:55:47.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.818+0000 7f3263a79640 1 -- 172.21.15.125:0/986123193 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f325c04e170 con 0x7f325c1547b0 2023-10-30T12:55:47.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.818+0000 7f32527fc640 1 -- 172.21.15.125:0/986123193 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f324c054810 con 0x7f325c1547b0 2023-10-30T12:55:47.819 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:47.820 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:37.637700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":40,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:47.820 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 127 2023-10-30T12:55:47.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.818+0000 7f3263a79640 1 -- 172.21.15.125:0/986123193 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32400664a0 msgr2=0x7f3240068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:47.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.818+0000 7f3263a79640 1 --2- 172.21.15.125:0/986123193 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32400664a0 0x7f3240068960 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f32580af000 tx=0x7f3258008040 comp rx=0 tx=0).stop 2023-10-30T12:55:47.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.818+0000 7f3263a79640 1 -- 172.21.15.125:0/986123193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f325c1547b0 msgr2=0x7f325c138510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:47.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.818+0000 7f3263a79640 1 --2- 172.21.15.125:0/986123193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f325c1547b0 0x7f325c138510 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f324c001060 tx=0x7f324c00ec10 comp rx=0 tx=0).stop 2023-10-30T12:55:47.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.818+0000 7f3263a79640 1 -- 172.21.15.125:0/986123193 shutdown_connections 2023-10-30T12:55:47.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.818+0000 7f3263a79640 1 --2- 172.21.15.125:0/986123193 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32400664a0 0x7f3240068960 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:47.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.818+0000 7f3263a79640 1 --2- 172.21.15.125:0/986123193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f325c13b660 0x7f325c136630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:47.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.818+0000 7f3263a79640 1 --2- 172.21.15.125:0/986123193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f325c1547b0 0x7f325c138510 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:47.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.818+0000 7f3263a79640 1 --2- 172.21.15.125:0/986123193 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3248003b10 0x7f325c137fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:47.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.818+0000 7f3263a79640 1 -- 172.21.15.125:0/986123193 >> 172.21.15.125:0/986123193 conn(0x7f325c0c7910 msgr2=0x7f325c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:47.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.818+0000 7f3263a79640 1 -- 172.21.15.125:0/986123193 shutdown_connections 2023-10-30T12:55:47.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:47.818+0000 7f3263a79640 1 -- 172.21.15.125:0/986123193 wait complete. 2023-10-30T12:55:47.839 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T12:55:48.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:55:48.463 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:55:48.830 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 4% (7/160) Updating files: 5% (8/160) Updating files: 6% (10/160) Updating files: 7% (12/160) Updating files: 8% (13/160) Updating files: 9% (15/160) Updating files: 10% (16/160) Updating files: 11% (18/160) Updating files: 12% (20/160) Updating files: 13% (21/160) Updating files: 14% (23/160) Updating files: 15% (24/160) Updating files: 16% (26/160) Updating files: 17% (28/160) Updating files: 18% (29/160) Updating files: 19% (31/160) Updating files: 20% (32/160) Updating files: 21% (34/160) Updating files: 22% (36/160) Updating files: 23% (37/160) Updating files: 24% (39/160) Updating files: 25% (40/160) Updating files: 26% (42/160) Updating files: 27% (44/160) Updating files: 28% (45/160) Updating files: 28% (46/160) Updating files: 29% (47/160) Updating files: 30% (48/160) Updating files: 31% (50/160) Updating files: 32% (52/160) Updating files: 33% (53/160) Updating files: 34% (55/160) Updating files: 35% (56/160) Updating files: 36% (58/160) Updating files: 37% (60/160) Updating files: 38% (61/160) Updating files: 39% (63/160) Updating files: 40% (64/160) Updating files: 41% (66/160) Updating files: 42% (68/160) Updating files: 43% (69/160) Updating files: 44% (71/160) Updating files: 45% (72/160) Updating files: 46% (74/160) Updating files: 47% (76/160) Updating files: 48% (77/160) Updating files: 49% (79/160) Updating files: 50% (80/160) Updating files: 51% (82/160) Updating files: 52% (84/160) Updating files: 53% (85/160) Updating files: 54% (87/160) Updating files: 55% (88/160) Updating files: 56% (90/160) Updating files: 57% (92/160) Updating files: 58% (93/160) Updating files: 59% (95/160) Updating files: 60% (96/160) Updating files: 61% (98/160) Updating files: 62% (100/160) Updating files: 63% (101/160) Updating files: 64% (103/160) Updating files: 65% (104/160) Updating files: 66% (106/160) Updating files: 67% (108/160) Updating files: 68% (109/160) Updating files: 69% (111/160) Updating files: 70% (112/160) Updating files: 71% (114/160) Updating files: 72% (116/160) Updating files: 73% (117/160) Updating files: 74% (119/160) Updating files: 75% (120/160) Updating files: 76% (122/160) Updating files: 77% (124/160) Updating files: 78% (125/160) Updating files: 79% (127/160) Updating files: 80% (128/160) Updating files: 81% (130/160) Updating files: 82% (132/160) Updating files: 83% (133/160) Updating files: 84% (135/160) Updating files: 85% (136/160) Updating files: 86% (138/160) Updating files: 87% (140/160) Updating files: 88% (141/160) Updating files: 89% (143/160) Updating files: 90% (144/160) Updating files: 91% (146/160) Updating files: 92% (148/160) Updating files: 93% (149/160) Updating files: 94% (151/160) Updating files: 95% (152/160) Updating files: 96% (154/160) Updating files: 97% (156/160) Updating files: 98% (157/160) Updating files: 99% (159/160) Updating files: 100% (160/160) Updating files: 100% (160/160), done. 2023-10-30T12:55:48.915 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 340b4348bf6 v0.33 2023-10-30T12:55:48.920 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.33 2023-10-30T12:55:48.942 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 34 -eq 48 ] 2023-10-30T12:55:48.943 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.34 2023-10-30T12:55:48.943 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.34 2023-10-30T12:55:48.943 INFO:tasks.workunit.client.0.smithi125.stdout:v0.34 2023-10-30T12:55:48.944 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.34 2023-10-30T12:55:49.841 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:55:49.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.986+0000 7fb31ce09640 1 Processor -- start 2023-10-30T12:55:49.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.986+0000 7fb31ce09640 1 -- start start 2023-10-30T12:55:49.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb31ce09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3180c05d0 0x7fb31813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:49.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb31ce09640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb3180c6f60 0x7fb31813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:49.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb31ce09640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3180c0a50 con 0x7fb3180c6b80 2023-10-30T12:55:49.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb31ce09640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31806f6e0 con 0x7fb3180c6f60 2023-10-30T12:55:49.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb31ce09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31806f9c0 con 0x7fb3180c05d0 2023-10-30T12:55:49.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb31659c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb3180c6b80 0x7fb3180bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40602/0 (socket says 172.21.15.125:40602) 2023-10-30T12:55:49.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb31659c640 1 -- 172.21.15.125:0/4017693092 learned_addr learned my addr 172.21.15.125:0/4017693092 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:55:49.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb316d9d640 1 --2- 172.21.15.125:0/4017693092 >> v2:172.21.15.125:3300/0 conn(0x7fb3180c6f60 0x7fb31813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:49.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb315d9b640 1 --2- 172.21.15.125:0/4017693092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3180c05d0 0x7fb31813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:49.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb31559a640 1 -- 172.21.15.125:0/4017693092 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855050006 0 0) 0x7fb3180c0a50 con 0x7fb3180c6b80 2023-10-30T12:55:49.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb31559a640 1 -- 172.21.15.125:0/4017693092 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb300005740 con 0x7fb3180c6b80 2023-10-30T12:55:49.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb316d9d640 1 -- 172.21.15.125:0/4017693092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3180c05d0 msgr2=0x7fb31813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:55:49.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb316d9d640 1 --2- 172.21.15.125:0/4017693092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3180c05d0 0x7fb31813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:49.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb316d9d640 1 -- 172.21.15.125:0/4017693092 >> v1:172.21.15.125:6789/0 conn(0x7fb3180c6b80 legacy=0x7fb3180bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:49.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.990+0000 7fb316d9d640 1 -- 172.21.15.125:0/4017693092 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb31806fca0 con 0x7fb3180c6f60 2023-10-30T12:55:49.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.994+0000 7fb316d9d640 1 --2- 172.21.15.125:0/4017693092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3180c6f60 0x7fb31813f2d0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fb310009d90 tx=0x7fb31001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e510910f3108586f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:49.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.994+0000 7fb31559a640 1 -- 172.21.15.125:0/4017693092 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3100a5920 con 0x7fb3180c6f60 2023-10-30T12:55:49.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.994+0000 7fb31559a640 1 -- 172.21.15.125:0/4017693092 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3100a5a80 con 0x7fb3180c6f60 2023-10-30T12:55:49.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.994+0000 7fb31559a640 1 -- 172.21.15.125:0/4017693092 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3100a5e10 con 0x7fb3180c6f60 2023-10-30T12:55:49.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.994+0000 7fb31ce09640 1 -- 172.21.15.125:0/4017693092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3180c6f60 msgr2=0x7fb31813f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:49.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.994+0000 7fb31ce09640 1 --2- 172.21.15.125:0/4017693092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3180c6f60 0x7fb31813f2d0 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fb310009d90 tx=0x7fb31001c950 comp rx=0 tx=0).stop 2023-10-30T12:55:49.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.994+0000 7fb31ce09640 1 -- 172.21.15.125:0/4017693092 shutdown_connections 2023-10-30T12:55:49.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.994+0000 7fb31ce09640 1 --2- 172.21.15.125:0/4017693092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3180c6f60 0x7fb31813f2d0 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:49.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.994+0000 7fb31ce09640 1 --2- 172.21.15.125:0/4017693092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3180c05d0 0x7fb31813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:49.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.994+0000 7fb31ce09640 1 -- 172.21.15.125:0/4017693092 >> 172.21.15.125:0/4017693092 conn(0x7fb3180c7910 msgr2=0x7fb3180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:49.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.994+0000 7fb31ce09640 1 -- 172.21.15.125:0/4017693092 shutdown_connections 2023-10-30T12:55:49.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.994+0000 7fb31ce09640 1 -- 172.21.15.125:0/4017693092 wait complete. 2023-10-30T12:55:50.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.994+0000 7fb31ce09640 1 Processor -- start 2023-10-30T12:55:50.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.994+0000 7fb31ce09640 1 -- start start 2023-10-30T12:55:50.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb31ce09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3180c05d0 0x7fb318138b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:50.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb31ce09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3180c6b80 0x7fb31813cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:50.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb31ce09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3180c6f60 0x7fb31813d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:50.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb31ce09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31813e010 con 0x7fb3180c6f60 2023-10-30T12:55:50.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb31ce09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb318154b50 con 0x7fb3180c05d0 2023-10-30T12:55:50.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb31ce09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31813da50 con 0x7fb3180c6b80 2023-10-30T12:55:50.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb316d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3180c6f60 0x7fb31813d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:50.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb316d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3180c6f60 0x7fb31813d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45604/0 (socket says 172.21.15.125:45604) 2023-10-30T12:55:50.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb316d9d640 1 -- 172.21.15.125:0/2135894347 learned_addr learned my addr 172.21.15.125:0/2135894347 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:55:50.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb315d9b640 1 --2- 172.21.15.125:0/2135894347 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3180c6b80 0x7fb31813cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:50.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb31659c640 1 --2- 172.21.15.125:0/2135894347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3180c05d0 0x7fb318138b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:50.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb316d9d640 1 -- 172.21.15.125:0/2135894347 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3180c6b80 msgr2=0x7fb31813cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:50.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb316d9d640 1 --2- 172.21.15.125:0/2135894347 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3180c6b80 0x7fb31813cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:50.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb316d9d640 1 -- 172.21.15.125:0/2135894347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3180c05d0 msgr2=0x7fb318138b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:50.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb316d9d640 1 --2- 172.21.15.125:0/2135894347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3180c05d0 0x7fb318138b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:50.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb316d9d640 1 -- 172.21.15.125:0/2135894347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb310009a70 con 0x7fb3180c6f60 2023-10-30T12:55:50.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb316d9d640 1 --2- 172.21.15.125:0/2135894347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3180c6f60 0x7fb31813d440 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fb31001a580 tx=0x7fb3100196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:50.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb306ffd640 1 -- 172.21.15.125:0/2135894347 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3100a5920 con 0x7fb3180c6f60 2023-10-30T12:55:50.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb306ffd640 1 -- 172.21.15.125:0/2135894347 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb310009450 con 0x7fb3180c6f60 2023-10-30T12:55:50.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb31ce09640 1 -- 172.21.15.125:0/2135894347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb31813dd30 con 0x7fb3180c6f60 2023-10-30T12:55:50.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb306ffd640 1 -- 172.21.15.125:0/2135894347 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3100b7650 con 0x7fb3180c6f60 2023-10-30T12:55:50.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb31ce09640 1 -- 172.21.15.125:0/2135894347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3181f2cd0 con 0x7fb3180c6f60 2023-10-30T12:55:50.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:49.998+0000 7fb31ce09640 1 -- 172.21.15.125:0/2135894347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31806b0c0 con 0x7fb3180c6f60 2023-10-30T12:55:50.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.002+0000 7fb306ffd640 1 -- 172.21.15.125:0/2135894347 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb3100b4050 con 0x7fb3180c6f60 2023-10-30T12:55:50.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.002+0000 7fb306ffd640 1 --2- 172.21.15.125:0/2135894347 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2f80664a0 0x7fb2f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:55:50.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.002+0000 7fb306ffd640 1 -- 172.21.15.125:0/2135894347 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7fb31011f660 con 0x7fb3180c6f60 2023-10-30T12:55:50.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.002+0000 7fb31659c640 1 --2- 172.21.15.125:0/2135894347 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2f80664a0 0x7fb2f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:55:50.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.002+0000 7fb31659c640 1 --2- 172.21.15.125:0/2135894347 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2f80664a0 0x7fb2f8068960 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb308005890 tx=0x7fb308009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:55:50.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.006+0000 7fb306ffd640 1 -- 172.21.15.125:0/2135894347 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb3100ec660 con 0x7fb3180c6f60 2023-10-30T12:55:50.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.318+0000 7fb31ce09640 1 -- 172.21.15.125:0/2135894347 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb318154cc0 con 0x7fb3180c6f60 2023-10-30T12:55:50.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.318+0000 7fb306ffd640 1 -- 172.21.15.125:0/2135894347 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+6516 (secure 0 0 0) 0x7fb3100efc80 con 0x7fb3180c6f60 2023-10-30T12:55:50.319 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:55:50.320 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7269,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:37.637700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":40,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:55:50.320 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 128 2023-10-30T12:55:50.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.318+0000 7fb31ce09640 1 -- 172.21.15.125:0/2135894347 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2f80664a0 msgr2=0x7fb2f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:50.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.318+0000 7fb31ce09640 1 --2- 172.21.15.125:0/2135894347 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2f80664a0 0x7fb2f8068960 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fb308005890 tx=0x7fb308009040 comp rx=0 tx=0).stop 2023-10-30T12:55:50.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.318+0000 7fb31ce09640 1 -- 172.21.15.125:0/2135894347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3180c6f60 msgr2=0x7fb31813d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:55:50.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.318+0000 7fb31ce09640 1 --2- 172.21.15.125:0/2135894347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3180c6f60 0x7fb31813d440 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fb31001a580 tx=0x7fb3100196f0 comp rx=0 tx=0).stop 2023-10-30T12:55:50.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.318+0000 7fb31ce09640 1 -- 172.21.15.125:0/2135894347 shutdown_connections 2023-10-30T12:55:50.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.318+0000 7fb31ce09640 1 --2- 172.21.15.125:0/2135894347 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2f80664a0 0x7fb2f8068960 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:50.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.318+0000 7fb31ce09640 1 --2- 172.21.15.125:0/2135894347 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb3180c6f60 0x7fb31813d440 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:50.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.318+0000 7fb31ce09640 1 --2- 172.21.15.125:0/2135894347 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb3180c6b80 0x7fb31813cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:50.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.318+0000 7fb31ce09640 1 --2- 172.21.15.125:0/2135894347 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb3180c05d0 0x7fb318138b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:55:50.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.318+0000 7fb31ce09640 1 -- 172.21.15.125:0/2135894347 >> 172.21.15.125:0/2135894347 conn(0x7fb3180c7910 msgr2=0x7fb31813ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:55:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.318+0000 7fb31ce09640 1 -- 172.21.15.125:0/2135894347 shutdown_connections 2023-10-30T12:55:50.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:55:50.318+0000 7fb31ce09640 1 -- 172.21.15.125:0/2135894347 wait complete. 2023-10-30T12:55:50.347 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T12:55:50.347 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 28 secs before thrashing 2023-10-30T12:55:56.203 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 44% (58/129) Updating files: 45% (59/129) Updating files: 46% (60/129) Updating files: 47% (61/129) Updating files: 48% (62/129) Updating files: 49% (64/129) Updating files: 50% (65/129) Updating files: 51% (66/129) Updating files: 52% (68/129) Updating files: 53% (69/129) Updating files: 54% (70/129) Updating files: 55% (71/129) Updating files: 56% (73/129) Updating files: 57% (74/129) Updating files: 58% (75/129) Updating files: 59% (77/129) Updating files: 60% (78/129) Updating files: 61% (79/129) Updating files: 62% (80/129) Updating files: 63% (82/129) Updating files: 64% (83/129) Updating files: 65% (84/129) Updating files: 66% (86/129) Updating files: 67% (87/129) Updating files: 68% (88/129) Updating files: 69% (90/129) Updating files: 70% (91/129) Updating files: 71% (92/129) Updating files: 72% (93/129) Updating files: 73% (95/129) Updating files: 74% (96/129) Updating files: 75% (97/129) Updating files: 76% (99/129) Updating files: 77% (100/129) Updating files: 78% (101/129) Updating files: 79% (102/129) Updating files: 80% (104/129) Updating files: 81% (105/129) Updating files: 82% (106/129) Updating files: 83% (108/129) Updating files: 84% (109/129) Updating files: 85% (110/129) Updating files: 86% (111/129) Updating files: 87% (113/129) Updating files: 88% (114/129) Updating files: 89% (115/129) Updating files: 90% (117/129) Updating files: 91% (118/129) Updating files: 92% (119/129) Updating files: 93% (120/129) Updating files: 94% (122/129) Updating files: 95% (123/129) Updating files: 96% (124/129) Updating files: 97% (126/129) Updating files: 98% (127/129) Updating files: 99% (128/129) Updating files: 100% (129/129) Updating files: 100% (129/129), done. 2023-10-30T12:55:56.234 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 2f039eeeb74 v0.34 2023-10-30T12:55:56.237 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.34 2023-10-30T12:55:56.314 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 35 -eq 48 ] 2023-10-30T12:55:56.315 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.35 2023-10-30T12:55:56.315 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.35 2023-10-30T12:55:56.316 INFO:tasks.workunit.client.0.smithi125.stdout:v0.35 2023-10-30T12:55:56.316 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.35 2023-10-30T12:56:00.780 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 36% (122/334) Updating files: 37% (124/334) Updating files: 38% (127/334) Updating files: 39% (131/334) Updating files: 40% (134/334) Updating files: 41% (137/334) Updating files: 42% (141/334) Updating files: 43% (144/334) Updating files: 44% (147/334) Updating files: 45% (151/334) Updating files: 46% (154/334) Updating files: 47% (157/334) Updating files: 48% (161/334) Updating files: 49% (164/334) Updating files: 50% (167/334) Updating files: 51% (171/334) Updating files: 52% (174/334) Updating files: 53% (178/334) Updating files: 54% (181/334) Updating files: 55% (184/334) Updating files: 56% (188/334) Updating files: 57% (191/334) Updating files: 58% (194/334) Updating files: 59% (198/334) Updating files: 60% (201/334) Updating files: 61% (204/334) Updating files: 62% (208/334) Updating files: 63% (211/334) Updating files: 64% (214/334) Updating files: 65% (218/334) Updating files: 66% (221/334) Updating files: 67% (224/334) Updating files: 68% (228/334) Updating files: 69% (231/334) Updating files: 70% (234/334) Updating files: 71% (238/334) Updating files: 72% (241/334) Updating files: 72% (243/334) Updating files: 73% (244/334) Updating files: 74% (248/334) Updating files: 75% (251/334) Updating files: 76% (254/334) Updating files: 77% (258/334) Updating files: 78% (261/334) Updating files: 79% (264/334) Updating files: 80% (268/334) Updating files: 81% (271/334) Updating files: 82% (274/334) Updating files: 83% (278/334) Updating files: 84% (281/334) Updating files: 85% (284/334) Updating files: 86% (288/334) Updating files: 87% (291/334) Updating files: 88% (294/334) Updating files: 89% (298/334) Updating files: 89% (300/334) Updating files: 90% (301/334) Updating files: 91% (304/334) Updating files: 92% (308/334) Updating files: 93% (311/334) Updating files: 94% (314/334) Updating files: 95% (318/334) Updating files: 96% (321/334) Updating files: 97% (324/334) Updating files: 98% (328/334) Updating files: 99% (331/334) Updating files: 100% (334/334) Updating files: 100% (334/334), done. 2023-10-30T12:56:00.811 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 0afda378124 v0.35 2023-10-30T12:56:00.814 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.35 2023-10-30T12:56:00.886 INFO:tasks.workunit.client.0.smithi125.stdout:v0.36 2023-10-30T12:56:00.887 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 36 -eq 48 ] 2023-10-30T12:56:00.887 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.36 2023-10-30T12:56:00.887 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.36 2023-10-30T12:56:00.887 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.36 2023-10-30T12:56:03.151 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 877cacbbdbb v0.36 2023-10-30T12:56:03.153 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.36 2023-10-30T12:56:03.282 INFO:tasks.workunit.client.0.smithi125.stdout:v0.37 2023-10-30T12:56:03.282 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 37 -eq 48 ] 2023-10-30T12:56:03.283 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.37 2023-10-30T12:56:03.283 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.37 2023-10-30T12:56:03.283 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.37 2023-10-30T12:56:16.034 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 19% (36/181) Updating files: 20% (37/181) Updating files: 21% (39/181) Updating files: 22% (40/181) Updating files: 23% (42/181) Updating files: 24% (44/181) Updating files: 25% (46/181) Updating files: 26% (48/181) Updating files: 27% (49/181) Updating files: 28% (51/181) Updating files: 29% (53/181) Updating files: 30% (55/181) Updating files: 31% (57/181) Updating files: 32% (58/181) Updating files: 33% (60/181) Updating files: 34% (62/181) Updating files: 35% (64/181) Updating files: 36% (66/181) Updating files: 37% (67/181) Updating files: 38% (69/181) Updating files: 39% (71/181) Updating files: 40% (73/181) Updating files: 41% (75/181) Updating files: 42% (77/181) Updating files: 43% (78/181) Updating files: 44% (80/181) Updating files: 45% (82/181) Updating files: 46% (84/181) Updating files: 47% (86/181) Updating files: 48% (87/181) Updating files: 49% (89/181) Updating files: 50% (91/181) Updating files: 51% (93/181) Updating files: 52% (95/181) Updating files: 53% (96/181) Updating files: 54% (98/181) Updating files: 55% (100/181) Updating files: 56% (102/181) Updating files: 57% (104/181) Updating files: 58% (105/181) Updating files: 59% (107/181) Updating files: 60% (109/181) Updating files: 61% (111/181) Updating files: 62% (113/181) Updating files: 63% (115/181) Updating files: 64% (116/181) Updating files: 65% (118/181) Updating files: 66% (120/181) Updating files: 67% (122/181) Updating files: 68% (124/181) Updating files: 69% (125/181) Updating files: 70% (127/181) Updating files: 71% (129/181) Updating files: 72% (131/181) Updating files: 73% (133/181) Updating files: 74% (134/181) Updating files: 75% (136/181) Updating files: 76% (138/181) Updating files: 77% (140/181) Updating files: 78% (142/181) Updating files: 79% (143/181) Updating files: 80% (145/181) Updating files: 81% (147/181) Updating files: 82% (149/181) Updating files: 83% (151/181) Updating files: 84% (153/181) Updating files: 85% (154/181) Updating files: 86% (156/181) Updating files: 87% (158/181) Updating files: 88% (160/181) Updating files: 89% (162/181) Updating files: 90% (163/181) Updating files: 91% (165/181) Updating files: 92% (167/181) Updating files: 93% (169/181) Updating files: 94% (171/181) Updating files: 95% (172/181) Updating files: 96% (174/181) Updating files: 97% (176/181) Updating files: 98% (178/181) Updating files: 99% (180/181) Updating files: 100% (181/181) Updating files: 100% (181/181), done. 2023-10-30T12:56:16.081 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at a6f3bbb744a v0.37 2023-10-30T12:56:16.084 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.37 2023-10-30T12:56:16.191 INFO:tasks.workunit.client.0.smithi125.stdout:v0.38 2023-10-30T12:56:16.191 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 38 -eq 48 ] 2023-10-30T12:56:16.191 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.38 2023-10-30T12:56:16.191 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.38 2023-10-30T12:56:16.192 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.38 2023-10-30T12:56:17.958 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 69% (119/171) Updating files: 70% (120/171) Updating files: 71% (122/171) Updating files: 72% (124/171) Updating files: 73% (125/171) Updating files: 74% (127/171) Updating files: 75% (129/171) Updating files: 76% (130/171) Updating files: 77% (132/171) Updating files: 78% (134/171) Updating files: 79% (136/171) Updating files: 80% (137/171) Updating files: 81% (139/171) Updating files: 82% (141/171) Updating files: 83% (142/171) Updating files: 84% (144/171) Updating files: 85% (146/171) Updating files: 86% (148/171) Updating files: 87% (149/171) Updating files: 88% (151/171) Updating files: 89% (153/171) Updating files: 90% (154/171) Updating files: 91% (156/171) Updating files: 92% (158/171) Updating files: 93% (160/171) Updating files: 94% (161/171) Updating files: 95% (163/171) Updating files: 96% (165/171) Updating files: 97% (166/171) Updating files: 98% (168/171) Updating files: 99% (170/171) Updating files: 100% (171/171) Updating files: 100% (171/171), done. 2023-10-30T12:56:18.348 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:18.358 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at b600ec2ac7c v0.38 2023-10-30T12:56:18.361 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.38 2023-10-30T12:56:18.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.497+0000 7f6cf166b640 1 Processor -- start 2023-10-30T12:56:18.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.497+0000 7f6cf166b640 1 -- start start 2023-10-30T12:56:18.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6cf166b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cec1538a0 0x7f6cec15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:18.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6cf166b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6cec0ca0e0 0x7f6cec15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:18.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6cf166b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6cec0cf160 0x7f6cec0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:18.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6cf166b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cec0cb800 con 0x7f6cec0cf160 2023-10-30T12:56:18.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6cf166b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cec15dc30 con 0x7f6cec0ca0e0 2023-10-30T12:56:18.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6cf166b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec0cb330 con 0x7f6cec1538a0 2023-10-30T12:56:18.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6ceb7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6cec0cf160 0x7f6cec0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:18.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6ceb7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6cec0cf160 0x7f6cec0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48106/0 (socket says 172.21.15.125:48106) 2023-10-30T12:56:18.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6ceb7fe640 1 -- 172.21.15.125:0/2721945528 learned_addr learned my addr 172.21.15.125:0/2721945528 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:18.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6ceaffd640 1 --2- 172.21.15.125:0/2721945528 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cec1538a0 0x7f6cec15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:18.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6ce27fc640 1 --2- 172.21.15.125:0/2721945528 >> v2:172.21.15.136:3300/0 conn(0x7f6cec0ca0e0 0x7f6cec15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:18.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6ceb7fe640 1 -- 172.21.15.125:0/2721945528 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cec1538a0 msgr2=0x7f6cec15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:18.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6ceb7fe640 1 --2- 172.21.15.125:0/2721945528 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cec1538a0 0x7f6cec15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6ceb7fe640 1 -- 172.21.15.125:0/2721945528 >> v2:172.21.15.136:3300/0 conn(0x7f6cec0ca0e0 msgr2=0x7f6cec15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:18.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6ceb7fe640 1 --2- 172.21.15.125:0/2721945528 >> v2:172.21.15.136:3300/0 conn(0x7f6cec0ca0e0 0x7f6cec15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.501+0000 7f6ceb7fe640 1 -- 172.21.15.125:0/2721945528 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cec0cad70 con 0x7f6cec0cf160 2023-10-30T12:56:18.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6ceb7fe640 1 --2- 172.21.15.125:0/2721945528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cec0cf160 0x7f6cec0cf590 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f6cec0ccd10 tx=0x7f6ce401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a34498f54c1c4a7b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:18.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6cea7fc640 1 -- 172.21.15.125:0/2721945528 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce40a89b0 con 0x7f6cec0cf160 2023-10-30T12:56:18.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6cea7fc640 1 -- 172.21.15.125:0/2721945528 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce40a8b10 con 0x7f6cec0cf160 2023-10-30T12:56:18.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6cea7fc640 1 -- 172.21.15.125:0/2721945528 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce40a8ea0 con 0x7f6cec0cf160 2023-10-30T12:56:18.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6cf166b640 1 -- 172.21.15.125:0/2721945528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cec0cf160 msgr2=0x7f6cec0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:18.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6cf166b640 1 --2- 172.21.15.125:0/2721945528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cec0cf160 0x7f6cec0cf590 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f6cec0ccd10 tx=0x7f6ce401ecf0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6cf166b640 1 -- 172.21.15.125:0/2721945528 shutdown_connections 2023-10-30T12:56:18.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6cf166b640 1 --2- 172.21.15.125:0/2721945528 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cec0cf160 0x7f6cec0cf590 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6cf166b640 1 --2- 172.21.15.125:0/2721945528 >> v2:172.21.15.136:3300/0 conn(0x7f6cec0ca0e0 0x7f6cec15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6cf166b640 1 --2- 172.21.15.125:0/2721945528 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cec1538a0 0x7f6cec15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6cf166b640 1 -- 172.21.15.125:0/2721945528 >> 172.21.15.125:0/2721945528 conn(0x7f6cec06bfe0 msgr2=0x7f6cec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:18.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6cf166b640 1 -- 172.21.15.125:0/2721945528 shutdown_connections 2023-10-30T12:56:18.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6cf166b640 1 -- 172.21.15.125:0/2721945528 wait complete. 2023-10-30T12:56:18.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6cf166b640 1 Processor -- start 2023-10-30T12:56:18.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.505+0000 7f6cf166b640 1 -- start start 2023-10-30T12:56:18.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6cf166b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cec0ca0e0 0x7f6cec15d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:18.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6cf166b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cec0cf160 0x7f6cec0ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:18.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6cf166b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cec1538a0 0x7f6cec15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:18.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6cf166b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec159280 con 0x7f6cec0cf160 2023-10-30T12:56:18.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6cf166b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec159560 con 0x7f6cec1538a0 2023-10-30T12:56:18.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6cf166b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cec1f2430 con 0x7f6cec0ca0e0 2023-10-30T12:56:18.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ce27fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cec0cf160 0x7f6cec0ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:18.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ce27fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cec0cf160 0x7f6cec0ce290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48112/0 (socket says 172.21.15.125:48112) 2023-10-30T12:56:18.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ce27fc640 1 -- 172.21.15.125:0/794807958 learned_addr learned my addr 172.21.15.125:0/794807958 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:18.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ceb7fe640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cec1538a0 0x7f6cec15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:18.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ceaffd640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cec0ca0e0 0x7f6cec15d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:18.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ce27fc640 1 -- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cec0ca0e0 msgr2=0x7f6cec15d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:18.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ce27fc640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cec0ca0e0 0x7f6cec15d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ce27fc640 1 -- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cec1538a0 msgr2=0x7f6cec15a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:18.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ce27fc640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cec1538a0 0x7f6cec15a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ce27fc640 1 -- 172.21.15.125:0/794807958 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce40094d0 con 0x7f6cec0cf160 2023-10-30T12:56:18.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ceaffd640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cec0ca0e0 0x7f6cec15d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:56:18.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ce27fc640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cec0cf160 0x7f6cec0ce290 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4007e60 tx=0x7f6cd400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:18.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ce3fff640 1 -- 172.21.15.125:0/794807958 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd4014620 con 0x7f6cec0cf160 2023-10-30T12:56:18.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ce3fff640 1 -- 172.21.15.125:0/794807958 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd400ee90 con 0x7f6cec0cf160 2023-10-30T12:56:18.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6cf166b640 1 -- 172.21.15.125:0/794807958 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cec1f2710 con 0x7f6cec0cf160 2023-10-30T12:56:18.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ce3fff640 1 -- 172.21.15.125:0/794807958 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd40149c0 con 0x7f6cec0cf160 2023-10-30T12:56:18.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6cf166b640 1 -- 172.21.15.125:0/794807958 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cec0cee60 con 0x7f6cec0cf160 2023-10-30T12:56:18.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6cf166b640 1 -- 172.21.15.125:0/794807958 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cec0b2a50 con 0x7f6cec0cf160 2023-10-30T12:56:18.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.509+0000 7f6ce3fff640 1 -- 172.21.15.125:0/794807958 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6cd4014b20 con 0x7f6cec0cf160 2023-10-30T12:56:18.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.513+0000 7f6ce3fff640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ccc0664a0 0x7f6ccc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:18.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.513+0000 7f6ce3fff640 1 -- 172.21.15.125:0/794807958 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7f6cd4084c70 con 0x7f6cec0cf160 2023-10-30T12:56:18.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.513+0000 7f6ceaffd640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ccc0664a0 0x7f6ccc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:18.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.513+0000 7f6ceaffd640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ccc0664a0 0x7f6ccc068960 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00b9f0 tx=0x7f6cdc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:18.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.517+0000 7f6ce3fff640 1 -- 172.21.15.125:0/794807958 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6cd4051bd0 con 0x7f6cec0cf160 2023-10-30T12:56:18.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.697+0000 7f6cf166b640 1 -- 172.21.15.125:0/794807958 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cec1f2f10 con 0x7f6cec0cf160 2023-10-30T12:56:18.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.697+0000 7f6ce3fff640 1 -- 172.21.15.125:0/794807958 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+6516 (secure 0 0 0) 0x7f6cec1f2f10 con 0x7f6cec0cf160 2023-10-30T12:56:18.701 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:18.702 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7269,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:37.637700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":40,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:18.702 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 128 2023-10-30T12:56:18.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.701+0000 7f6cf166b640 1 -- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ccc0664a0 msgr2=0x7f6ccc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:18.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.701+0000 7f6cf166b640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ccc0664a0 0x7f6ccc068960 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc00b9f0 tx=0x7f6cdc008040 comp rx=0 tx=0).stop 2023-10-30T12:56:18.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.701+0000 7f6cf166b640 1 -- 172.21.15.125:0/794807958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cec0cf160 msgr2=0x7f6cec0ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:18.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.701+0000 7f6cf166b640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cec0cf160 0x7f6cec0ce290 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4007e60 tx=0x7f6cd400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.701+0000 7f6cf166b640 1 -- 172.21.15.125:0/794807958 shutdown_connections 2023-10-30T12:56:18.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.701+0000 7f6cf166b640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ccc0664a0 0x7f6ccc068960 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.701+0000 7f6cf166b640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cec1538a0 0x7f6cec15a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.701+0000 7f6cf166b640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cec0cf160 0x7f6cec0ce290 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.701+0000 7f6cf166b640 1 --2- 172.21.15.125:0/794807958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cec0ca0e0 0x7f6cec15d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.701+0000 7f6cf166b640 1 -- 172.21.15.125:0/794807958 >> 172.21.15.125:0/794807958 conn(0x7f6cec06bfe0 msgr2=0x7f6cec150ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:18.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.701+0000 7f6cf166b640 1 -- 172.21.15.125:0/794807958 shutdown_connections 2023-10-30T12:56:18.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.701+0000 7f6cf166b640 1 -- 172.21.15.125:0/794807958 wait complete. 2023-10-30T12:56:18.722 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T12:56:18.723 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T12:56:18.723 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:56:18.756 INFO:tasks.workunit.client.0.smithi125.stdout:v0.39 2023-10-30T12:56:18.757 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 39 -eq 48 ] 2023-10-30T12:56:18.757 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.39 2023-10-30T12:56:18.757 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.39 2023-10-30T12:56:18.758 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.39 2023-10-30T12:56:18.802 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T12:56:18.802 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:18.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.937+0000 7f00c9291640 1 Processor -- start 2023-10-30T12:56:18.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.937+0000 7f00c9291640 1 -- start start 2023-10-30T12:56:18.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.937+0000 7f00c9291640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00c4071110 0x7f00c406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:18.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.937+0000 7f00c9291640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c4066cf0 con 0x7f00c406f7a0 2023-10-30T12:56:18.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.937+0000 7f00c9291640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c40c70d0 con 0x7f00c40c6bd0 2023-10-30T12:56:18.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.937+0000 7f00c9291640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00c40700c0 con 0x7f00c4071110 2023-10-30T12:56:18.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.937+0000 7f00c37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00c4071110 0x7f00c406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:18.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.937+0000 7f00c37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00c4071110 0x7f00c406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48128/0 (socket says 172.21.15.125:48128) 2023-10-30T12:56:18.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.937+0000 7f00c37fe640 1 -- 172.21.15.125:0/3349106755 learned_addr learned my addr 172.21.15.125:0/3349106755 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:18.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.937+0000 7f00c37fe640 1 -- 172.21.15.125:0/3349106755 >> v1:172.21.15.136:6789/0 conn(0x7f00c40c6bd0 legacy=0x7f00c4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:18.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.937+0000 7f00c37fe640 1 -- 172.21.15.125:0/3349106755 >> v1:172.21.15.125:6789/0 conn(0x7f00c406f7a0 legacy=0x7f00c4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:18.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.937+0000 7f00c37fe640 1 -- 172.21.15.125:0/3349106755 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c40c7390 con 0x7f00c4071110 2023-10-30T12:56:18.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.941+0000 7f00c37fe640 1 --2- 172.21.15.125:0/3349106755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00c4071110 0x7f00c406fb80 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009e50 tx=0x7f00bc01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6b10383e3fa9a7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:18.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.941+0000 7f00c27fc640 1 -- 172.21.15.125:0/3349106755 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc0af400 con 0x7f00c4071110 2023-10-30T12:56:18.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.941+0000 7f00c27fc640 1 -- 172.21.15.125:0/3349106755 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00bc0ab050 con 0x7f00c4071110 2023-10-30T12:56:18.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.941+0000 7f00c27fc640 1 -- 172.21.15.125:0/3349106755 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc0af780 con 0x7f00c4071110 2023-10-30T12:56:18.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.941+0000 7f00c9291640 1 -- 172.21.15.125:0/3349106755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00c4071110 msgr2=0x7f00c406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:18.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.941+0000 7f00c9291640 1 --2- 172.21.15.125:0/3349106755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00c4071110 0x7f00c406fb80 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009e50 tx=0x7f00bc01f1c0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.941+0000 7f00c9291640 1 -- 172.21.15.125:0/3349106755 shutdown_connections 2023-10-30T12:56:18.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.941+0000 7f00c9291640 1 --2- 172.21.15.125:0/3349106755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00c4071110 0x7f00c406fb80 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.941+0000 7f00c9291640 1 -- 172.21.15.125:0/3349106755 >> 172.21.15.125:0/3349106755 conn(0x7f00c40c7910 msgr2=0x7f00c40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:18.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.941+0000 7f00c9291640 1 -- 172.21.15.125:0/3349106755 shutdown_connections 2023-10-30T12:56:18.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.941+0000 7f00c9291640 1 -- 172.21.15.125:0/3349106755 wait complete. 2023-10-30T12:56:18.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c9291640 1 Processor -- start 2023-10-30T12:56:18.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c9291640 1 -- start start 2023-10-30T12:56:18.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c9291640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00c406f7a0 0x7f00c4150b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:18.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c9291640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c4071110 0x7f00c41eefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:18.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c9291640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00c40c6bd0 0x7f00c41ef510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:18.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c9291640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c41f3140 con 0x7f00c40c6bd0 2023-10-30T12:56:18.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c9291640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c40ae920 con 0x7f00c4071110 2023-10-30T12:56:18.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c9291640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c41f24e0 con 0x7f00c406f7a0 2023-10-30T12:56:18.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00c40c6bd0 0x7f00c41ef510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:18.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00c40c6bd0 0x7f00c41ef510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48130/0 (socket says 172.21.15.125:48130) 2023-10-30T12:56:18.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c37fe640 1 -- 172.21.15.125:0/3178000612 learned_addr learned my addr 172.21.15.125:0/3178000612 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:18.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c2ffd640 1 --2- 172.21.15.125:0/3178000612 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00c406f7a0 0x7f00c4150b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:18.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00bbfff640 1 --2- 172.21.15.125:0/3178000612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c4071110 0x7f00c41eefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:18.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c37fe640 1 -- 172.21.15.125:0/3178000612 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00c406f7a0 msgr2=0x7f00c4150b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:18.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c37fe640 1 --2- 172.21.15.125:0/3178000612 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00c406f7a0 0x7f00c4150b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c37fe640 1 -- 172.21.15.125:0/3178000612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c4071110 msgr2=0x7f00c41eefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:18.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c37fe640 1 --2- 172.21.15.125:0/3178000612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c4071110 0x7f00c41eefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:18.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c37fe640 1 -- 172.21.15.125:0/3178000612 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00bc009b30 con 0x7f00c40c6bd0 2023-10-30T12:56:18.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c37fe640 1 --2- 172.21.15.125:0/3178000612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00c40c6bd0 0x7f00c41ef510 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009e50 tx=0x7f00bc01bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:18.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00bb7fe640 1 -- 172.21.15.125:0/3178000612 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc0afde0 con 0x7f00c40c6bd0 2023-10-30T12:56:18.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00bb7fe640 1 -- 172.21.15.125:0/3178000612 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00bc009510 con 0x7f00c40c6bd0 2023-10-30T12:56:18.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00bb7fe640 1 -- 172.21.15.125:0/3178000612 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc0b8640 con 0x7f00c40c6bd0 2023-10-30T12:56:18.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c9291640 1 -- 172.21.15.125:0/3178000612 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c41f27a0 con 0x7f00c40c6bd0 2023-10-30T12:56:18.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.945+0000 7f00c9291640 1 -- 172.21.15.125:0/3178000612 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c41f2d60 con 0x7f00c40c6bd0 2023-10-30T12:56:18.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.949+0000 7f00c9291640 1 -- 172.21.15.125:0/3178000612 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c41f3280 con 0x7f00c40c6bd0 2023-10-30T12:56:18.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.949+0000 7f00bb7fe640 1 -- 172.21.15.125:0/3178000612 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f00bc0b6040 con 0x7f00c40c6bd0 2023-10-30T12:56:18.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.949+0000 7f00bb7fe640 1 --2- 172.21.15.125:0/3178000612 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00a40664a0 0x7f00a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:18.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.949+0000 7f00bb7fe640 1 -- 172.21.15.125:0/3178000612 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7f00bc122420 con 0x7f00c40c6bd0 2023-10-30T12:56:18.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.949+0000 7f00c2ffd640 1 --2- 172.21.15.125:0/3178000612 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00a40664a0 0x7f00a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:18.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.949+0000 7f00c2ffd640 1 --2- 172.21.15.125:0/3178000612 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00a40664a0 0x7f00a4068960 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f00b4000f70 tx=0x7f00b4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:18.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:18.953+0000 7f00bb7fe640 1 -- 172.21.15.125:0/3178000612 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f00c41f3280 con 0x7f00c40c6bd0 2023-10-30T12:56:19.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:19.133+0000 7f00c9291640 1 -- 172.21.15.125:0/3178000612 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00c41f3280 con 0x7f00c40c6bd0 2023-10-30T12:56:19.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:19.133+0000 7f00bb7fe640 1 -- 172.21.15.125:0/3178000612 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+6516 (secure 0 0 0) 0x7f00c41f3280 con 0x7f00c40c6bd0 2023-10-30T12:56:19.137 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:19.137 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7269,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:37.637700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":40,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:19.138 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 128 2023-10-30T12:56:19.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:19.137+0000 7f00c9291640 1 -- 172.21.15.125:0/3178000612 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00a40664a0 msgr2=0x7f00a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:19.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:19.137+0000 7f00c9291640 1 --2- 172.21.15.125:0/3178000612 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00a40664a0 0x7f00a4068960 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f00b4000f70 tx=0x7f00b4009040 comp rx=0 tx=0).stop 2023-10-30T12:56:19.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:19.137+0000 7f00c9291640 1 -- 172.21.15.125:0/3178000612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00c40c6bd0 msgr2=0x7f00c41ef510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:19.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:19.137+0000 7f00c9291640 1 --2- 172.21.15.125:0/3178000612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00c40c6bd0 0x7f00c41ef510 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009e50 tx=0x7f00bc01bf60 comp rx=0 tx=0).stop 2023-10-30T12:56:19.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:19.137+0000 7f00c9291640 1 -- 172.21.15.125:0/3178000612 shutdown_connections 2023-10-30T12:56:19.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:19.137+0000 7f00c9291640 1 --2- 172.21.15.125:0/3178000612 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00a40664a0 0x7f00a4068960 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:19.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:19.137+0000 7f00c9291640 1 --2- 172.21.15.125:0/3178000612 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00c40c6bd0 0x7f00c41ef510 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:19.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:19.137+0000 7f00c9291640 1 --2- 172.21.15.125:0/3178000612 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00c4071110 0x7f00c41eefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:19.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:19.137+0000 7f00c9291640 1 --2- 172.21.15.125:0/3178000612 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00c406f7a0 0x7f00c4150b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:19.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:19.137+0000 7f00c9291640 1 -- 172.21.15.125:0/3178000612 >> 172.21.15.125:0/3178000612 conn(0x7f00c40c7910 msgr2=0x7f00c40ca0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:19.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:19.137+0000 7f00c9291640 1 -- 172.21.15.125:0/3178000612 shutdown_connections 2023-10-30T12:56:19.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:19.137+0000 7f00c9291640 1 -- 172.21.15.125:0/3178000612 wait complete. 2023-10-30T12:56:19.159 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:56:19.542 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:56:19.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:56:21.160 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:21.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e7fd3640 1 Processor -- start 2023-10-30T12:56:21.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e7fd3640 1 -- start start 2023-10-30T12:56:21.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e7fd3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87e01538a0 0x7f87e015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:21.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e7fd3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f87e00ca0e0 0x7f87e015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:21.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e7fd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87e00cf160 0x7f87e00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:21.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e7fd3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e00cb800 con 0x7f87e01538a0 2023-10-30T12:56:21.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e7fd3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87e015dc30 con 0x7f87e00ca0e0 2023-10-30T12:56:21.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e7fd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e00cb330 con 0x7f87e00cf160 2023-10-30T12:56:21.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e5d6f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87e01538a0 0x7f87e015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:21.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e5d6f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f87e01538a0 0x7f87e015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47926/0 (socket says 172.21.15.125:47926) 2023-10-30T12:56:21.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e5d6f640 1 -- 172.21.15.125:0/1725794525 learned_addr learned my addr 172.21.15.125:0/1725794525 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:21.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e556e640 1 --2- 172.21.15.125:0/1725794525 >> v2:172.21.15.136:3300/0 conn(0x7f87e00ca0e0 0x7f87e015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:21.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e6570640 1 --2- 172.21.15.125:0/1725794525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87e00cf160 0x7f87e00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:21.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e5d6f640 1 -- 172.21.15.125:0/1725794525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87e00cf160 msgr2=0x7f87e00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:21.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e5d6f640 1 --2- 172.21.15.125:0/1725794525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87e00cf160 0x7f87e00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:21.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e5d6f640 1 -- 172.21.15.125:0/1725794525 >> v2:172.21.15.136:3300/0 conn(0x7f87e00ca0e0 msgr2=0x7f87e015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:21.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e5d6f640 1 --2- 172.21.15.125:0/1725794525 >> v2:172.21.15.136:3300/0 conn(0x7f87e00ca0e0 0x7f87e015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:21.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.305+0000 7f87e5d6f640 1 -- 172.21.15.125:0/1725794525 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e00cad70 con 0x7f87e01538a0 2023-10-30T12:56:21.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.309+0000 7f87e5d6f640 1 --2- 172.21.15.125:0/1725794525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e01538a0 0x7f87e015d6f0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f87e00b30a0 tx=0x7f87dc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a1eb595e404d54 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:21.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.309+0000 7f87e4d6d640 1 -- 172.21.15.125:0/1725794525 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc0a89b0 con 0x7f87e01538a0 2023-10-30T12:56:21.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.309+0000 7f87e4d6d640 1 -- 172.21.15.125:0/1725794525 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc0a8b10 con 0x7f87e01538a0 2023-10-30T12:56:21.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.309+0000 7f87e4d6d640 1 -- 172.21.15.125:0/1725794525 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc0a8ea0 con 0x7f87e01538a0 2023-10-30T12:56:21.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.309+0000 7f87e7fd3640 1 -- 172.21.15.125:0/1725794525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e01538a0 msgr2=0x7f87e015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:21.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.309+0000 7f87e7fd3640 1 --2- 172.21.15.125:0/1725794525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e01538a0 0x7f87e015d6f0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f87e00b30a0 tx=0x7f87dc01ecf0 comp rx=0 tx=0).stop 2023-10-30T12:56:21.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.309+0000 7f87e7fd3640 1 -- 172.21.15.125:0/1725794525 shutdown_connections 2023-10-30T12:56:21.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.309+0000 7f87e7fd3640 1 --2- 172.21.15.125:0/1725794525 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87e00cf160 0x7f87e00cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:21.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.309+0000 7f87e7fd3640 1 --2- 172.21.15.125:0/1725794525 >> v2:172.21.15.136:3300/0 conn(0x7f87e00ca0e0 0x7f87e015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:21.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.309+0000 7f87e7fd3640 1 --2- 172.21.15.125:0/1725794525 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e01538a0 0x7f87e015d6f0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:21.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.309+0000 7f87e7fd3640 1 -- 172.21.15.125:0/1725794525 >> 172.21.15.125:0/1725794525 conn(0x7f87e006bfe0 msgr2=0x7f87e006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:21.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.309+0000 7f87e7fd3640 1 -- 172.21.15.125:0/1725794525 shutdown_connections 2023-10-30T12:56:21.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.309+0000 7f87e7fd3640 1 -- 172.21.15.125:0/1725794525 wait complete. 2023-10-30T12:56:21.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e7fd3640 1 Processor -- start 2023-10-30T12:56:21.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e7fd3640 1 -- start start 2023-10-30T12:56:21.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e7fd3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e00ca0e0 0x7f87e0142d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:21.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e7fd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87e00cf160 0x7f87e01432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:21.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e7fd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87e01538a0 0x7f87e00b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:21.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e7fd3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e0158bb0 con 0x7f87e00ca0e0 2023-10-30T12:56:21.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e7fd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87e0159520 con 0x7f87e00cf160 2023-10-30T12:56:21.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e7fd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87e01508e0 con 0x7f87e01538a0 2023-10-30T12:56:21.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e5d6f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e00ca0e0 0x7f87e0142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:21.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e5d6f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e00ca0e0 0x7f87e0142d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47934/0 (socket says 172.21.15.125:47934) 2023-10-30T12:56:21.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e5d6f640 1 -- 172.21.15.125:0/3851277379 learned_addr learned my addr 172.21.15.125:0/3851277379 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:21.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e556e640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87e00cf160 0x7f87e01432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:21.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e6570640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87e01538a0 0x7f87e00b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:21.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e5d6f640 1 -- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87e01538a0 msgr2=0x7f87e00b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:21.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e5d6f640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87e01538a0 0x7f87e00b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:21.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e5d6f640 1 -- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87e00cf160 msgr2=0x7f87e01432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:21.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e5d6f640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87e00cf160 0x7f87e01432a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:21.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e5d6f640 1 -- 172.21.15.125:0/3851277379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc0094d0 con 0x7f87e00ca0e0 2023-10-30T12:56:21.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e556e640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87e00cf160 0x7f87e01432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:56:21.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e5d6f640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e00ca0e0 0x7f87e0142d60 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f87dc01d3a0 tx=0x7f87dc01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:21.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87d67fc640 1 -- 172.21.15.125:0/3851277379 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc0ba600 con 0x7f87e00ca0e0 2023-10-30T12:56:21.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87d67fc640 1 -- 172.21.15.125:0/3851277379 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc0ba760 con 0x7f87e00ca0e0 2023-10-30T12:56:21.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e7fd3640 1 -- 172.21.15.125:0/3851277379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87e0150bc0 con 0x7f87e00ca0e0 2023-10-30T12:56:21.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87d67fc640 1 -- 172.21.15.125:0/3851277379 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc0bab10 con 0x7f87e00ca0e0 2023-10-30T12:56:21.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e7fd3640 1 -- 172.21.15.125:0/3851277379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87e0158e00 con 0x7f87e00ca0e0 2023-10-30T12:56:21.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.313+0000 7f87e7fd3640 1 -- 172.21.15.125:0/3851277379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87e00b2a50 con 0x7f87e00ca0e0 2023-10-30T12:56:21.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.317+0000 7f87d67fc640 1 -- 172.21.15.125:0/3851277379 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f87dc0bac70 con 0x7f87e00ca0e0 2023-10-30T12:56:21.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.317+0000 7f87d67fc640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87c40664a0 0x7f87c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:21.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.317+0000 7f87d67fc640 1 -- 172.21.15.125:0/3851277379 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7f87dc122840 con 0x7f87e00ca0e0 2023-10-30T12:56:21.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.317+0000 7f87e556e640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87c40664a0 0x7f87c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:21.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.317+0000 7f87e556e640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87c40664a0 0x7f87c4068960 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f87e00cea10 tx=0x7f87d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:21.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.321+0000 7f87d67fc640 1 -- 172.21.15.125:0/3851277379 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f87dc0ef9f0 con 0x7f87e00ca0e0 2023-10-30T12:56:21.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.501+0000 7f87e7fd3640 1 -- 172.21.15.125:0/3851277379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87e0150d50 con 0x7f87e00ca0e0 2023-10-30T12:56:21.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.501+0000 7f87d67fc640 1 -- 172.21.15.125:0/3851277379 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+6516 (secure 0 0 0) 0x7f87dc0f3010 con 0x7f87e00ca0e0 2023-10-30T12:56:21.505 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:21.505 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7269,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:37.637700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":40,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:21.506 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 128 2023-10-30T12:56:21.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.505+0000 7f87e7fd3640 1 -- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87c40664a0 msgr2=0x7f87c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:21.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.505+0000 7f87e7fd3640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87c40664a0 0x7f87c4068960 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f87e00cea10 tx=0x7f87d0008040 comp rx=0 tx=0).stop 2023-10-30T12:56:21.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.505+0000 7f87e7fd3640 1 -- 172.21.15.125:0/3851277379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e00ca0e0 msgr2=0x7f87e0142d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:21.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.505+0000 7f87e7fd3640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e00ca0e0 0x7f87e0142d60 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f87dc01d3a0 tx=0x7f87dc01ba90 comp rx=0 tx=0).stop 2023-10-30T12:56:21.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.505+0000 7f87e7fd3640 1 -- 172.21.15.125:0/3851277379 shutdown_connections 2023-10-30T12:56:21.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.505+0000 7f87e7fd3640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87c40664a0 0x7f87c4068960 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:21.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.505+0000 7f87e7fd3640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87e01538a0 0x7f87e00b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:21.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.505+0000 7f87e7fd3640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87e00cf160 0x7f87e01432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:21.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.505+0000 7f87e7fd3640 1 --2- 172.21.15.125:0/3851277379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87e00ca0e0 0x7f87e0142d60 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:21.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.505+0000 7f87e7fd3640 1 -- 172.21.15.125:0/3851277379 >> 172.21.15.125:0/3851277379 conn(0x7f87e006bfe0 msgr2=0x7f87e0066410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:21.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.505+0000 7f87e7fd3640 1 -- 172.21.15.125:0/3851277379 shutdown_connections 2023-10-30T12:56:21.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:21.505+0000 7f87e7fd3640 1 -- 172.21.15.125:0/3851277379 wait complete. 2023-10-30T12:56:21.526 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:56:23.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:23.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.669+0000 7f943ba6b640 1 Processor -- start 2023-10-30T12:56:23.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f943ba6b640 1 -- start start 2023-10-30T12:56:23.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f943ba6b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f94340c6b80 0x7f94340bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:23.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f943ba6b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94340c0a50 con 0x7f94340c6f60 2023-10-30T12:56:23.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f943ba6b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f943406f6e0 con 0x7f943413f4a0 2023-10-30T12:56:23.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f943ba6b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f943406f9c0 con 0x7f94340c6b80 2023-10-30T12:56:23.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9439006640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f94340c6f60 0x7f943413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33962/0 (socket says 172.21.15.125:33962) 2023-10-30T12:56:23.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9439006640 1 -- 172.21.15.125:0/1622979578 learned_addr learned my addr 172.21.15.125:0/1622979578 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:56:23.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9439807640 1 --2- 172.21.15.125:0/1622979578 >> v2:172.21.15.136:3300/0 conn(0x7f94340c6b80 0x7f94340bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:23.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 -- 172.21.15.125:0/1622979578 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1455735866 0 0) 0x7f94340c0a50 con 0x7f94340c6f60 2023-10-30T12:56:23.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 -- 172.21.15.125:0/1622979578 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9420003530 con 0x7f94340c6f60 2023-10-30T12:56:23.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 -- 172.21.15.125:0/1622979578 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2916745970 0 0) 0x7f9420003530 con 0x7f94340c6f60 2023-10-30T12:56:23.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 -- 172.21.15.125:0/1622979578 >> v2:172.21.15.136:3300/0 conn(0x7f94340c6b80 msgr2=0x7f94340bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:23.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 --2- 172.21.15.125:0/1622979578 >> v2:172.21.15.136:3300/0 conn(0x7f94340c6b80 0x7f94340bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:23.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 -- 172.21.15.125:0/1622979578 >> v1:172.21.15.136:6789/0 conn(0x7f943413f4a0 legacy=0x7f94341566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:23.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 -- 172.21.15.125:0/1622979578 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f943406fca0 con 0x7f94340c6f60 2023-10-30T12:56:23.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 -- 172.21.15.125:0/1622979578 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f9424001680 con 0x7f94340c6f60 2023-10-30T12:56:23.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9439807640 1 --2- 172.21.15.125:0/1622979578 >> v2:172.21.15.136:3300/0 conn(0x7f94340c6b80 0x7f94340bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:56:23.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 -- 172.21.15.125:0/1622979578 >> v1:172.21.15.125:6789/0 conn(0x7f94340c6f60 legacy=0x7f943413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:23.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 --2- 172.21.15.125:0/1622979578 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9420003980 0x7f9420003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:23.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 --2- 172.21.15.125:0/1622979578 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94200047d0 0x7f9420004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:23.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 --2- 172.21.15.125:0/1622979578 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9420005190 0x7f942000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:23.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 -- 172.21.15.125:0/1622979578 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943406f9c0 con 0x7f9420005190 2023-10-30T12:56:23.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 -- 172.21.15.125:0/1622979578 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942000c030 con 0x7f9420003980 2023-10-30T12:56:23.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9438805640 1 -- 172.21.15.125:0/1622979578 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f942000c1f0 con 0x7f94200047d0 2023-10-30T12:56:23.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f943a008640 1 --2- 172.21.15.125:0/1622979578 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9420005190 0x7f942000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:23.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9439807640 1 --2- 172.21.15.125:0/1622979578 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9420003980 0x7f9420003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:23.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9439006640 1 --2- 172.21.15.125:0/1622979578 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94200047d0 0x7f9420004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:23.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f943a008640 1 -- 172.21.15.125:0/1622979578 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94200047d0 msgr2=0x7f9420004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:23.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f943a008640 1 --2- 172.21.15.125:0/1622979578 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94200047d0 0x7f9420004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:23.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f943a008640 1 -- 172.21.15.125:0/1622979578 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9420003980 msgr2=0x7f9420003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:23.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f943a008640 1 --2- 172.21.15.125:0/1622979578 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9420003980 0x7f9420003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:23.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f943a008640 1 -- 172.21.15.125:0/1622979578 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9420003530 con 0x7f9420005190 2023-10-30T12:56:23.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.673+0000 7f9439006640 1 --2- 172.21.15.125:0/1622979578 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94200047d0 0x7f9420004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:56:23.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.677+0000 7f943a008640 1 -- 172.21.15.125:0/1622979578 reap_dead start 2023-10-30T12:56:23.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.677+0000 7f943a008640 1 --2- 172.21.15.125:0/1622979578 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9420005190 0x7f942000b990 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f942c001e70 tx=0x7f942c01c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed994ac01f93a15a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:23.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.677+0000 7f9438805640 1 -- 172.21.15.125:0/1622979578 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c003b80 con 0x7f9420005190 2023-10-30T12:56:23.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.677+0000 7f9438805640 1 -- 172.21.15.125:0/1622979578 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942c003d00 con 0x7f9420005190 2023-10-30T12:56:23.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.677+0000 7f9438805640 1 -- 172.21.15.125:0/1622979578 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c005260 con 0x7f9420005190 2023-10-30T12:56:23.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.677+0000 7f943ba6b640 1 -- 172.21.15.125:0/1622979578 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9420005190 msgr2=0x7f942000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:23.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.677+0000 7f943ba6b640 1 --2- 172.21.15.125:0/1622979578 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9420005190 0x7f942000b990 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f942c001e70 tx=0x7f942c01c760 comp rx=0 tx=0).stop 2023-10-30T12:56:23.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.677+0000 7f943ba6b640 1 -- 172.21.15.125:0/1622979578 shutdown_connections 2023-10-30T12:56:23.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.677+0000 7f943ba6b640 1 --2- 172.21.15.125:0/1622979578 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9420005190 0x7f942000b990 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:23.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.677+0000 7f943ba6b640 1 -- 172.21.15.125:0/1622979578 >> 172.21.15.125:0/1622979578 conn(0x7f94340c7910 msgr2=0x7f94340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:23.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.677+0000 7f943ba6b640 1 -- 172.21.15.125:0/1622979578 shutdown_connections 2023-10-30T12:56:23.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.677+0000 7f943ba6b640 1 -- 172.21.15.125:0/1622979578 wait complete. 2023-10-30T12:56:23.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f943ba6b640 1 Processor -- start 2023-10-30T12:56:23.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f943ba6b640 1 -- start start 2023-10-30T12:56:23.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f943ba6b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9420005190 0x7f943413d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:23.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f943ba6b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f943413dc70 0x7f94340c1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:23.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f943ba6b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94341ecff0 0x7f94341ed470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:23.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f943ba6b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9434139eb0 con 0x7f9420005190 2023-10-30T12:56:23.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f943ba6b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943413daf0 con 0x7f943413dc70 2023-10-30T12:56:23.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f943ba6b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f943413ce40 con 0x7f94341ecff0 2023-10-30T12:56:23.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f9439807640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9420005190 0x7f943413d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:23.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f9439807640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9420005190 0x7f943413d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47950/0 (socket says 172.21.15.125:47950) 2023-10-30T12:56:23.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f9439807640 1 -- 172.21.15.125:0/847388116 learned_addr learned my addr 172.21.15.125:0/847388116 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:23.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f9439006640 1 --2- 172.21.15.125:0/847388116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f943413dc70 0x7f94340c1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:23.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f943a008640 1 --2- 172.21.15.125:0/847388116 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94341ecff0 0x7f94341ed470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:23.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f9439807640 1 -- 172.21.15.125:0/847388116 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94341ecff0 msgr2=0x7f94341ed470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:23.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f9439807640 1 --2- 172.21.15.125:0/847388116 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94341ecff0 0x7f94341ed470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:23.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f9439807640 1 -- 172.21.15.125:0/847388116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f943413dc70 msgr2=0x7f94340c1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:23.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f9439807640 1 --2- 172.21.15.125:0/847388116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f943413dc70 0x7f94340c1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:23.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f9439807640 1 -- 172.21.15.125:0/847388116 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94341ecb60 con 0x7f9420005190 2023-10-30T12:56:23.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f9439807640 1 --2- 172.21.15.125:0/847388116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9420005190 0x7f943413d5b0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f9430011a00 tx=0x7f9430011ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:23.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f942a7fc640 1 -- 172.21.15.125:0/847388116 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f943000fbb0 con 0x7f9420005190 2023-10-30T12:56:23.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f942a7fc640 1 -- 172.21.15.125:0/847388116 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f943000fd10 con 0x7f9420005190 2023-10-30T12:56:23.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f942a7fc640 1 -- 172.21.15.125:0/847388116 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9430004260 con 0x7f9420005190 2023-10-30T12:56:23.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f943ba6b640 1 -- 172.21.15.125:0/847388116 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f943413e360 con 0x7f9420005190 2023-10-30T12:56:23.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.681+0000 7f943ba6b640 1 -- 172.21.15.125:0/847388116 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f943413e600 con 0x7f9420005190 2023-10-30T12:56:23.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.685+0000 7f943ba6b640 1 -- 172.21.15.125:0/847388116 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f943413e820 con 0x7f9420005190 2023-10-30T12:56:23.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.685+0000 7f942a7fc640 1 -- 172.21.15.125:0/847388116 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94300043c0 con 0x7f9420005190 2023-10-30T12:56:23.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.685+0000 7f942a7fc640 1 --2- 172.21.15.125:0/847388116 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94180664a0 0x7f9418068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:23.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.685+0000 7f942a7fc640 1 -- 172.21.15.125:0/847388116 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7f94300848c0 con 0x7f9420005190 2023-10-30T12:56:23.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.685+0000 7f9439006640 1 --2- 172.21.15.125:0/847388116 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94180664a0 0x7f9418068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:23.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.685+0000 7f9439006640 1 --2- 172.21.15.125:0/847388116 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94180664a0 0x7f9418068960 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9424001000 tx=0x7f9424009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:23.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.689+0000 7f942a7fc640 1 -- 172.21.15.125:0/847388116 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f94300518b0 con 0x7f9420005190 2023-10-30T12:56:23.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.873+0000 7f943ba6b640 1 -- 172.21.15.125:0/847388116 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f943404e170 con 0x7f9420005190 2023-10-30T12:56:23.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.873+0000 7f942a7fc640 1 -- 172.21.15.125:0/847388116 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+6516 (secure 0 0 0) 0x7f9430022090 con 0x7f9420005190 2023-10-30T12:56:23.876 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:23.876 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7269,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:37.637700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":40,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:23.877 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 128 2023-10-30T12:56:23.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.873+0000 7f943ba6b640 1 -- 172.21.15.125:0/847388116 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94180664a0 msgr2=0x7f9418068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:23.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.873+0000 7f943ba6b640 1 --2- 172.21.15.125:0/847388116 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94180664a0 0x7f9418068960 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9424001000 tx=0x7f9424009040 comp rx=0 tx=0).stop 2023-10-30T12:56:23.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.873+0000 7f943ba6b640 1 -- 172.21.15.125:0/847388116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9420005190 msgr2=0x7f943413d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:23.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.873+0000 7f943ba6b640 1 --2- 172.21.15.125:0/847388116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9420005190 0x7f943413d5b0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f9430011a00 tx=0x7f9430011ec0 comp rx=0 tx=0).stop 2023-10-30T12:56:23.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.873+0000 7f943ba6b640 1 -- 172.21.15.125:0/847388116 shutdown_connections 2023-10-30T12:56:23.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.873+0000 7f943ba6b640 1 --2- 172.21.15.125:0/847388116 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94180664a0 0x7f9418068960 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:23.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.873+0000 7f943ba6b640 1 --2- 172.21.15.125:0/847388116 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94341ecff0 0x7f94341ed470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:23.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.873+0000 7f943ba6b640 1 --2- 172.21.15.125:0/847388116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f943413dc70 0x7f94340c1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:23.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.873+0000 7f943ba6b640 1 --2- 172.21.15.125:0/847388116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9420005190 0x7f943413d5b0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:23.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.873+0000 7f943ba6b640 1 -- 172.21.15.125:0/847388116 >> 172.21.15.125:0/847388116 conn(0x7f94340c7910 msgr2=0x7f943413ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:23.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.873+0000 7f943ba6b640 1 -- 172.21.15.125:0/847388116 shutdown_connections 2023-10-30T12:56:23.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:23.873+0000 7f943ba6b640 1 -- 172.21.15.125:0/847388116 wait complete. 2023-10-30T12:56:23.896 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:56:25.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:26.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.041+0000 7fb77b59e640 1 Processor -- start 2023-10-30T12:56:26.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.045+0000 7fb77b59e640 1 -- start start 2023-10-30T12:56:26.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.045+0000 7fb77b59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77c0c6840 0x7fb77c15a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:26.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.045+0000 7fb77b59e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb77c0c3640 0x7fb77c156590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:26.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.045+0000 7fb77b59e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb77c0c3b30 con 0x7fb77c0c6c20 2023-10-30T12:56:26.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.045+0000 7fb77b59e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77c0c2aa0 con 0x7fb77c0c3640 2023-10-30T12:56:26.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.045+0000 7fb77b59e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c0c3df0 con 0x7fb77c0c6840 2023-10-30T12:56:26.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.045+0000 7fb77a59c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77c0c6840 0x7fb77c15a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:26.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.045+0000 7fb77a59c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77c0c6840 0x7fb77c15a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:41388/0 (socket says 172.21.15.125:41388) 2023-10-30T12:56:26.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.045+0000 7fb77a59c640 1 -- 172.21.15.125:0/2160077339 learned_addr learned my addr 172.21.15.125:0/2160077339 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:26.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.045+0000 7fb771d9b640 1 --2- 172.21.15.125:0/2160077339 >> v2:172.21.15.136:3300/0 conn(0x7fb77c0c3640 0x7fb77c156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:26.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.045+0000 7fb77a59c640 1 -- 172.21.15.125:0/2160077339 >> v2:172.21.15.136:3300/0 conn(0x7fb77c0c3640 msgr2=0x7fb77c156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:26.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.045+0000 7fb77a59c640 1 --2- 172.21.15.125:0/2160077339 >> v2:172.21.15.136:3300/0 conn(0x7fb77c0c3640 0x7fb77c156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:26.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.045+0000 7fb77a59c640 1 -- 172.21.15.125:0/2160077339 >> v1:172.21.15.136:6789/0 conn(0x7fb77c0c6c20 legacy=0x7fb77c153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:26.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.045+0000 7fb77a59c640 1 -- 172.21.15.125:0/2160077339 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77c159ba0 con 0x7fb77c0c6840 2023-10-30T12:56:26.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.049+0000 7fb77a59c640 1 --2- 172.21.15.125:0/2160077339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77c0c6840 0x7fb77c15a680 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb76c006fd0 tx=0x7fb76c01c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3028ac2abc13df server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:26.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.049+0000 7fb779d9b640 1 -- 172.21.15.125:0/2160077339 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0a7070 con 0x7fb77c0c6840 2023-10-30T12:56:26.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.049+0000 7fb779d9b640 1 -- 172.21.15.125:0/2160077339 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76c0ab430 con 0x7fb77c0c6840 2023-10-30T12:56:26.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.049+0000 7fb779d9b640 1 -- 172.21.15.125:0/2160077339 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb76c0ab7c0 con 0x7fb77c0c6840 2023-10-30T12:56:26.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.049+0000 7fb77b59e640 1 -- 172.21.15.125:0/2160077339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77c0c6840 msgr2=0x7fb77c15a680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:26.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.049+0000 7fb77b59e640 1 --2- 172.21.15.125:0/2160077339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77c0c6840 0x7fb77c15a680 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb76c006fd0 tx=0x7fb76c01c520 comp rx=0 tx=0).stop 2023-10-30T12:56:26.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.049+0000 7fb77b59e640 1 -- 172.21.15.125:0/2160077339 shutdown_connections 2023-10-30T12:56:26.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.049+0000 7fb77b59e640 1 --2- 172.21.15.125:0/2160077339 >> v2:172.21.15.136:3300/0 conn(0x7fb77c0c3640 0x7fb77c156590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:26.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.049+0000 7fb77b59e640 1 --2- 172.21.15.125:0/2160077339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77c0c6840 0x7fb77c15a680 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:26.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.049+0000 7fb77b59e640 1 -- 172.21.15.125:0/2160077339 >> 172.21.15.125:0/2160077339 conn(0x7fb77c0c75d0 msgr2=0x7fb77c0c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:26.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.049+0000 7fb77b59e640 1 -- 172.21.15.125:0/2160077339 shutdown_connections 2023-10-30T12:56:26.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.049+0000 7fb77b59e640 1 -- 172.21.15.125:0/2160077339 wait complete. 2023-10-30T12:56:26.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77b59e640 1 Processor -- start 2023-10-30T12:56:26.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77b59e640 1 -- start start 2023-10-30T12:56:26.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77b59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77c0c3640 0x7fb77c15eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:26.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77b59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77c0c6840 0x7fb77c152010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:26.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77b59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb77c0c6c20 0x7fb77c1f22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:26.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77b59e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c15f3e0 con 0x7fb77c0c6c20 2023-10-30T12:56:26.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77b59e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c1f5300 con 0x7fb77c0c3640 2023-10-30T12:56:26.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77b59e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77c1f5550 con 0x7fb77c0c6840 2023-10-30T12:56:26.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb77c0c6c20 0x7fb77c1f22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:26.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb77c0c6c20 0x7fb77c1f22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47966/0 (socket says 172.21.15.125:47966) 2023-10-30T12:56:26.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77ad9d640 1 -- 172.21.15.125:0/836716471 learned_addr learned my addr 172.21.15.125:0/836716471 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:26.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77a59c640 1 --2- 172.21.15.125:0/836716471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77c0c3640 0x7fb77c15eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:26.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb771d9b640 1 --2- 172.21.15.125:0/836716471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77c0c6840 0x7fb77c152010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:26.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77ad9d640 1 -- 172.21.15.125:0/836716471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77c0c6840 msgr2=0x7fb77c152010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:26.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77ad9d640 1 --2- 172.21.15.125:0/836716471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77c0c6840 0x7fb77c152010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:26.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77ad9d640 1 -- 172.21.15.125:0/836716471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77c0c3640 msgr2=0x7fb77c15eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:26.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77ad9d640 1 --2- 172.21.15.125:0/836716471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77c0c3640 0x7fb77c15eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:26.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77ad9d640 1 -- 172.21.15.125:0/836716471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c009480 con 0x7fb77c0c6c20 2023-10-30T12:56:26.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77ad9d640 1 --2- 172.21.15.125:0/836716471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb77c0c6c20 0x7fb77c1f22e0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb77400c730 tx=0x7fb77400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:26.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb7737fe640 1 -- 172.21.15.125:0/836716471 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774003b80 con 0x7fb77c0c6c20 2023-10-30T12:56:26.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb7737fe640 1 -- 172.21.15.125:0/836716471 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb774003ce0 con 0x7fb77c0c6c20 2023-10-30T12:56:26.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb7737fe640 1 -- 172.21.15.125:0/836716471 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb774005310 con 0x7fb77c0c6c20 2023-10-30T12:56:26.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77b59e640 1 -- 172.21.15.125:0/836716471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb77c1f5830 con 0x7fb77c0c6c20 2023-10-30T12:56:26.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77b59e640 1 -- 172.21.15.125:0/836716471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77c1f5df0 con 0x7fb77c0c6c20 2023-10-30T12:56:26.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.053+0000 7fb77b59e640 1 -- 172.21.15.125:0/836716471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77c074a20 con 0x7fb77c0c6c20 2023-10-30T12:56:26.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.057+0000 7fb7737fe640 1 -- 172.21.15.125:0/836716471 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb774005680 con 0x7fb77c0c6c20 2023-10-30T12:56:26.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.057+0000 7fb7737fe640 1 --2- 172.21.15.125:0/836716471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb75c0664a0 0x7fb75c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:26.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.057+0000 7fb7737fe640 1 -- 172.21.15.125:0/836716471 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7fb774084210 con 0x7fb77c0c6c20 2023-10-30T12:56:26.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.057+0000 7fb77a59c640 1 --2- 172.21.15.125:0/836716471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb75c0664a0 0x7fb75c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:26.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.057+0000 7fb77a59c640 1 --2- 172.21.15.125:0/836716471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb75c0664a0 0x7fb75c068960 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0af000 tx=0x7fb76c0a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:26.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.061+0000 7fb7737fe640 1 -- 172.21.15.125:0/836716471 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb774051230 con 0x7fb77c0c6c20 2023-10-30T12:56:26.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.241+0000 7fb77b59e640 1 -- 172.21.15.125:0/836716471 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb77c0c1f20 con 0x7fb77c0c6c20 2023-10-30T12:56:26.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.241+0000 7fb7737fe640 1 -- 172.21.15.125:0/836716471 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+6516 (secure 0 0 0) 0x7fb774054850 con 0x7fb77c0c6c20 2023-10-30T12:56:26.246 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:26.246 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7269,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:37.637700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":40,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:26.247 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 128 2023-10-30T12:56:26.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.245+0000 7fb77b59e640 1 -- 172.21.15.125:0/836716471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb75c0664a0 msgr2=0x7fb75c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:26.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.245+0000 7fb77b59e640 1 --2- 172.21.15.125:0/836716471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb75c0664a0 0x7fb75c068960 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb76c0af000 tx=0x7fb76c0a7000 comp rx=0 tx=0).stop 2023-10-30T12:56:26.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.245+0000 7fb77b59e640 1 -- 172.21.15.125:0/836716471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb77c0c6c20 msgr2=0x7fb77c1f22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:26.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.245+0000 7fb77b59e640 1 --2- 172.21.15.125:0/836716471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb77c0c6c20 0x7fb77c1f22e0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fb77400c730 tx=0x7fb77400cbf0 comp rx=0 tx=0).stop 2023-10-30T12:56:26.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.245+0000 7fb77b59e640 1 -- 172.21.15.125:0/836716471 shutdown_connections 2023-10-30T12:56:26.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.245+0000 7fb77b59e640 1 --2- 172.21.15.125:0/836716471 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb75c0664a0 0x7fb75c068960 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:26.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.245+0000 7fb77b59e640 1 --2- 172.21.15.125:0/836716471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb77c0c6c20 0x7fb77c1f22e0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:26.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.245+0000 7fb77b59e640 1 --2- 172.21.15.125:0/836716471 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb77c0c6840 0x7fb77c152010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:26.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.245+0000 7fb77b59e640 1 --2- 172.21.15.125:0/836716471 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb77c0c3640 0x7fb77c15eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:26.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.245+0000 7fb77b59e640 1 -- 172.21.15.125:0/836716471 >> 172.21.15.125:0/836716471 conn(0x7fb77c0c75d0 msgr2=0x7fb77c154b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:26.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.245+0000 7fb77b59e640 1 -- 172.21.15.125:0/836716471 shutdown_connections 2023-10-30T12:56:26.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:26.245+0000 7fb77b59e640 1 -- 172.21.15.125:0/836716471 wait complete. 2023-10-30T12:56:26.266 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:56:28.268 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:28.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe544f3e640 1 Processor -- start 2023-10-30T12:56:28.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe544f3e640 1 -- start start 2023-10-30T12:56:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe544f3e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe54006f7a0 0x7fe54006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe544f3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5400c6bd0 0x7fe540069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:28.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe544f3e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe540071110 0x7fe540158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:28.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe544f3e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5400700c0 con 0x7fe540071110 2023-10-30T12:56:28.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe544f3e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54006a3d0 con 0x7fe54006f7a0 2023-10-30T12:56:28.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe544f3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe540066cf0 con 0x7fe5400c6bd0 2023-10-30T12:56:28.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe53ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe540071110 0x7fe540158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:28.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe53ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe540071110 0x7fe540158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47980/0 (socket says 172.21.15.125:47980) 2023-10-30T12:56:28.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe53ed9d640 1 -- 172.21.15.125:0/978207470 learned_addr learned my addr 172.21.15.125:0/978207470 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:28.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe535d9b640 1 --2- 172.21.15.125:0/978207470 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5400c6bd0 0x7fe540069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:28.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe53e59c640 1 --2- 172.21.15.125:0/978207470 >> v2:172.21.15.136:3300/0 conn(0x7fe54006f7a0 0x7fe54006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:28.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe53ed9d640 1 -- 172.21.15.125:0/978207470 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5400c6bd0 msgr2=0x7fe540069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:28.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe53ed9d640 1 --2- 172.21.15.125:0/978207470 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5400c6bd0 0x7fe540069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:28.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe53ed9d640 1 -- 172.21.15.125:0/978207470 >> v2:172.21.15.136:3300/0 conn(0x7fe54006f7a0 msgr2=0x7fe54006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:28.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe53ed9d640 1 --2- 172.21.15.125:0/978207470 >> v2:172.21.15.136:3300/0 conn(0x7fe54006f7a0 0x7fe54006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:28.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.413+0000 7fe53ed9d640 1 -- 172.21.15.125:0/978207470 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5400c70d0 con 0x7fe540071110 2023-10-30T12:56:28.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.417+0000 7fe53ed9d640 1 --2- 172.21.15.125:0/978207470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe540071110 0x7fe540158110 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fe54006c830 tx=0x7fe53801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4c625e2e6c4187 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:28.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.417+0000 7fe53dd9b640 1 -- 172.21.15.125:0/978207470 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5380a89b0 con 0x7fe540071110 2023-10-30T12:56:28.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.417+0000 7fe53dd9b640 1 -- 172.21.15.125:0/978207470 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5380a8b10 con 0x7fe540071110 2023-10-30T12:56:28.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.417+0000 7fe53dd9b640 1 -- 172.21.15.125:0/978207470 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5380a8ea0 con 0x7fe540071110 2023-10-30T12:56:28.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.417+0000 7fe544f3e640 1 -- 172.21.15.125:0/978207470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe540071110 msgr2=0x7fe540158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:28.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.417+0000 7fe544f3e640 1 --2- 172.21.15.125:0/978207470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe540071110 0x7fe540158110 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fe54006c830 tx=0x7fe53801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:56:28.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.417+0000 7fe544f3e640 1 -- 172.21.15.125:0/978207470 shutdown_connections 2023-10-30T12:56:28.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.417+0000 7fe544f3e640 1 --2- 172.21.15.125:0/978207470 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe540071110 0x7fe540158110 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:28.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.417+0000 7fe544f3e640 1 --2- 172.21.15.125:0/978207470 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5400c6bd0 0x7fe540069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:28.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.417+0000 7fe544f3e640 1 --2- 172.21.15.125:0/978207470 >> v2:172.21.15.136:3300/0 conn(0x7fe54006f7a0 0x7fe54006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:28.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.417+0000 7fe544f3e640 1 -- 172.21.15.125:0/978207470 >> 172.21.15.125:0/978207470 conn(0x7fe5400c7910 msgr2=0x7fe5400c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:28.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.417+0000 7fe544f3e640 1 -- 172.21.15.125:0/978207470 shutdown_connections 2023-10-30T12:56:28.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.417+0000 7fe544f3e640 1 -- 172.21.15.125:0/978207470 wait complete. 2023-10-30T12:56:28.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe544f3e640 1 Processor -- start 2023-10-30T12:56:28.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe544f3e640 1 -- start start 2023-10-30T12:56:28.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe544f3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54006f7a0 0x7fe540159590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:28.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe544f3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe540071110 0x7fe54014f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:28.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe544f3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5400c6bd0 0x7fe54014fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:28.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe544f3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54015a280 con 0x7fe54006f7a0 2023-10-30T12:56:28.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe544f3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5400c0fb0 con 0x7fe540071110 2023-10-30T12:56:28.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe544f3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe540159fa0 con 0x7fe5400c6bd0 2023-10-30T12:56:28.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe53e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54006f7a0 0x7fe540159590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:28.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe53e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54006f7a0 0x7fe540159590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47996/0 (socket says 172.21.15.125:47996) 2023-10-30T12:56:28.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe53e59c640 1 -- 172.21.15.125:0/496669495 learned_addr learned my addr 172.21.15.125:0/496669495 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:28.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe535d9b640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe540071110 0x7fe54014f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:28.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe53ed9d640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5400c6bd0 0x7fe54014fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:28.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe53e59c640 1 -- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5400c6bd0 msgr2=0x7fe54014fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:28.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe53e59c640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5400c6bd0 0x7fe54014fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:28.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe53e59c640 1 -- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe540071110 msgr2=0x7fe54014f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:28.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe53e59c640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe540071110 0x7fe54014f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:28.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe53e59c640 1 -- 172.21.15.125:0/496669495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5380094d0 con 0x7fe54006f7a0 2023-10-30T12:56:28.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe535d9b640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe540071110 0x7fe54014f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:56:28.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe53e59c640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54006f7a0 0x7fe540159590 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fe530007e60 tx=0x7fe53000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:28.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe5377fe640 1 -- 172.21.15.125:0/496669495 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530014620 con 0x7fe54006f7a0 2023-10-30T12:56:28.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe5377fe640 1 -- 172.21.15.125:0/496669495 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53000ee90 con 0x7fe54006f7a0 2023-10-30T12:56:28.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe5377fe640 1 -- 172.21.15.125:0/496669495 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5300149a0 con 0x7fe54006f7a0 2023-10-30T12:56:28.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe544f3e640 1 -- 172.21.15.125:0/496669495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5401ef670 con 0x7fe54006f7a0 2023-10-30T12:56:28.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.421+0000 7fe544f3e640 1 -- 172.21.15.125:0/496669495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe540156910 con 0x7fe54006f7a0 2023-10-30T12:56:28.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.425+0000 7fe544f3e640 1 -- 172.21.15.125:0/496669495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5401ef4d0 con 0x7fe54006f7a0 2023-10-30T12:56:28.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.425+0000 7fe5377fe640 1 -- 172.21.15.125:0/496669495 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe530014b00 con 0x7fe54006f7a0 2023-10-30T12:56:28.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.425+0000 7fe5377fe640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5200664d0 0x7fe520068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:28.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.425+0000 7fe5377fe640 1 -- 172.21.15.125:0/496669495 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7fe530084b90 con 0x7fe54006f7a0 2023-10-30T12:56:28.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.425+0000 7fe535d9b640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5200664d0 0x7fe520068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:28.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.425+0000 7fe535d9b640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5200664d0 0x7fe520068990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe5400c8520 tx=0x7fe528008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:28.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.429+0000 7fe5377fe640 1 -- 172.21.15.125:0/496669495 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe5401ef4d0 con 0x7fe54006f7a0 2023-10-30T12:56:28.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.609+0000 7fe544f3e640 1 -- 172.21.15.125:0/496669495 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5400c1120 con 0x7fe54006f7a0 2023-10-30T12:56:28.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.613+0000 7fe5377fe640 1 -- 172.21.15.125:0/496669495 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+6516 (secure 0 0 0) 0x7fe5400c1120 con 0x7fe54006f7a0 2023-10-30T12:56:28.614 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:28.615 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7269,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:37.637700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":40,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:28.616 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 128 2023-10-30T12:56:28.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.613+0000 7fe544f3e640 1 -- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5200664d0 msgr2=0x7fe520068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:28.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.613+0000 7fe544f3e640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5200664d0 0x7fe520068990 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe5400c8520 tx=0x7fe528008040 comp rx=0 tx=0).stop 2023-10-30T12:56:28.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.613+0000 7fe544f3e640 1 -- 172.21.15.125:0/496669495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54006f7a0 msgr2=0x7fe540159590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:28.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.613+0000 7fe544f3e640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54006f7a0 0x7fe540159590 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7fe530007e60 tx=0x7fe53000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:56:28.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.613+0000 7fe544f3e640 1 -- 172.21.15.125:0/496669495 shutdown_connections 2023-10-30T12:56:28.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.613+0000 7fe544f3e640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5200664d0 0x7fe520068990 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:28.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.613+0000 7fe544f3e640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5400c6bd0 0x7fe54014fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:28.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.613+0000 7fe544f3e640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe540071110 0x7fe54014f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:28.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.613+0000 7fe544f3e640 1 --2- 172.21.15.125:0/496669495 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe54006f7a0 0x7fe540159590 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:28.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.613+0000 7fe544f3e640 1 -- 172.21.15.125:0/496669495 >> 172.21.15.125:0/496669495 conn(0x7fe5400c7910 msgr2=0x7fe5400b0750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:28.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.613+0000 7fe544f3e640 1 -- 172.21.15.125:0/496669495 shutdown_connections 2023-10-30T12:56:28.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:28.613+0000 7fe544f3e640 1 -- 172.21.15.125:0/496669495 wait complete. 2023-10-30T12:56:28.635 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:56:30.637 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:30.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.781+0000 7f295613e640 1 Processor -- start 2023-10-30T12:56:30.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.781+0000 7f295613e640 1 -- start start 2023-10-30T12:56:30.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.781+0000 7f295613e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f29500c05d0 0x7f295013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:30.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.781+0000 7f295613e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29500c0a50 con 0x7f29500c6f60 2023-10-30T12:56:30.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.781+0000 7f295613e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f295006f6e0 con 0x7f29500c6b80 2023-10-30T12:56:30.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.781+0000 7f295613e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f295006f9c0 con 0x7f29500c05d0 2023-10-30T12:56:30.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.781+0000 7f294ffff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f29500c6f60 0x7f29501566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37560/0 (socket says 172.21.15.125:37560) 2023-10-30T12:56:30.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.781+0000 7f294ffff640 1 -- 172.21.15.125:0/3725114884 learned_addr learned my addr 172.21.15.125:0/3725114884 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:56:30.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.781+0000 7f294effd640 1 --2- 172.21.15.125:0/3725114884 >> v2:172.21.15.136:3300/0 conn(0x7f29500c05d0 0x7f295013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:30.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.781+0000 7f294e7fc640 1 -- 172.21.15.125:0/3725114884 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205809618 0 0) 0x7f29500c0a50 con 0x7f29500c6f60 2023-10-30T12:56:30.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.781+0000 7f294e7fc640 1 -- 172.21.15.125:0/3725114884 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2938003530 con 0x7f29500c6f60 2023-10-30T12:56:30.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.785+0000 7f294effd640 1 -- 172.21.15.125:0/3725114884 >> v1:172.21.15.136:6789/0 conn(0x7f29500c6b80 legacy=0x7f29500bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:30.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.785+0000 7f294effd640 1 -- 172.21.15.125:0/3725114884 >> v1:172.21.15.125:6789/0 conn(0x7f29500c6f60 legacy=0x7f29501566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:30.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.785+0000 7f294effd640 1 -- 172.21.15.125:0/3725114884 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295006fca0 con 0x7f29500c05d0 2023-10-30T12:56:30.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.785+0000 7f294effd640 1 --2- 172.21.15.125:0/3725114884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29500c05d0 0x7f295013ed90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f29440097c0 tx=0x7f294401c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30779bc237480117 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:30.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.785+0000 7f294e7fc640 1 -- 172.21.15.125:0/3725114884 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29440a8050 con 0x7f29500c05d0 2023-10-30T12:56:30.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.785+0000 7f294e7fc640 1 -- 172.21.15.125:0/3725114884 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29440ac5f0 con 0x7f29500c05d0 2023-10-30T12:56:30.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.785+0000 7f294e7fc640 1 -- 172.21.15.125:0/3725114884 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29440ac960 con 0x7f29500c05d0 2023-10-30T12:56:30.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.785+0000 7f295613e640 1 -- 172.21.15.125:0/3725114884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29500c05d0 msgr2=0x7f295013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:30.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f295613e640 1 --2- 172.21.15.125:0/3725114884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29500c05d0 0x7f295013ed90 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f29440097c0 tx=0x7f294401c480 comp rx=0 tx=0).stop 2023-10-30T12:56:30.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f295613e640 1 -- 172.21.15.125:0/3725114884 shutdown_connections 2023-10-30T12:56:30.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f295613e640 1 --2- 172.21.15.125:0/3725114884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29500c05d0 0x7f295013ed90 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:30.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f295613e640 1 -- 172.21.15.125:0/3725114884 >> 172.21.15.125:0/3725114884 conn(0x7f29500c7910 msgr2=0x7f29500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:30.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f295613e640 1 -- 172.21.15.125:0/3725114884 shutdown_connections 2023-10-30T12:56:30.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f295613e640 1 -- 172.21.15.125:0/3725114884 wait complete. 2023-10-30T12:56:30.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f295613e640 1 Processor -- start 2023-10-30T12:56:30.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f295613e640 1 -- start start 2023-10-30T12:56:30.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f295613e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29500c05d0 0x7f29500afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:30.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f295613e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29500c6b80 0x7f2950150370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:30.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f295613e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29500c6f60 0x7f29501508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:30.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f295613e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2950150df0 con 0x7f29500c6b80 2023-10-30T12:56:30.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f295613e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295013e560 con 0x7f29500c6f60 2023-10-30T12:56:30.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f295613e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29500c9050 con 0x7f29500c05d0 2023-10-30T12:56:30.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f294effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29500c6b80 0x7f2950150370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:30.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f294effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29500c6b80 0x7f2950150370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34552/0 (socket says 172.21.15.125:34552) 2023-10-30T12:56:30.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f294effd640 1 -- 172.21.15.125:0/3922755303 learned_addr learned my addr 172.21.15.125:0/3922755303 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:30.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f294f7fe640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29500c05d0 0x7f29500afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:30.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f294ffff640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29500c6f60 0x7f29501508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:30.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f294effd640 1 -- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29500c05d0 msgr2=0x7f29500afce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:30.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f294effd640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29500c05d0 0x7f29500afce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:30.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f294effd640 1 -- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29500c6f60 msgr2=0x7f29501508b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:30.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f294effd640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29500c6f60 0x7f29501508b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:30.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f294effd640 1 -- 172.21.15.125:0/3922755303 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2944009420 con 0x7f29500c6b80 2023-10-30T12:56:30.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f294f7fe640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29500c05d0 0x7f29500afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:56:30.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.789+0000 7f294ffff640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29500c6f60 0x7f29501508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:56:30.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.793+0000 7f294effd640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29500c6b80 0x7f2950150370 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f2944000c10 tx=0x7f2944019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:30.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.793+0000 7f2937fff640 1 -- 172.21.15.125:0/3922755303 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29440acb70 con 0x7f29500c6b80 2023-10-30T12:56:30.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.793+0000 7f2937fff640 1 -- 172.21.15.125:0/3922755303 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29440accd0 con 0x7f29500c6b80 2023-10-30T12:56:30.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.793+0000 7f2937fff640 1 -- 172.21.15.125:0/3922755303 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2944003820 con 0x7f29500c6b80 2023-10-30T12:56:30.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.793+0000 7f295613e640 1 -- 172.21.15.125:0/3922755303 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2950071c50 con 0x7f29500c6b80 2023-10-30T12:56:30.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.793+0000 7f295613e640 1 -- 172.21.15.125:0/3922755303 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29501598b0 con 0x7f29500c6b80 2023-10-30T12:56:30.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.793+0000 7f295613e640 1 -- 172.21.15.125:0/3922755303 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2950071e10 con 0x7f29500c6b80 2023-10-30T12:56:30.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.793+0000 7f2937fff640 1 -- 172.21.15.125:0/3922755303 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29440b3040 con 0x7f29500c6b80 2023-10-30T12:56:30.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.793+0000 7f2937fff640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f293c0664d0 0x7f293c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:30.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.793+0000 7f2937fff640 1 -- 172.21.15.125:0/3922755303 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7f294411e2f0 con 0x7f29500c6b80 2023-10-30T12:56:30.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.797+0000 7f294f7fe640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f293c0664d0 0x7f293c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:30.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.797+0000 7f294f7fe640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f293c0664d0 0x7f293c068990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2940001710 tx=0x7f2940009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:30.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.801+0000 7f2937fff640 1 -- 172.21.15.125:0/3922755303 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2950071e10 con 0x7f29500c6b80 2023-10-30T12:56:30.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.977+0000 7f295613e640 1 -- 172.21.15.125:0/3922755303 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29501595d0 con 0x7f29500c6b80 2023-10-30T12:56:30.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.977+0000 7f2937fff640 1 -- 172.21.15.125:0/3922755303 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+6516 (secure 0 0 0) 0x7f29501595d0 con 0x7f29500c6b80 2023-10-30T12:56:30.982 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:30.982 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7269,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:37.637700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":40,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:30.983 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 128 2023-10-30T12:56:30.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.981+0000 7f295613e640 1 -- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f293c0664d0 msgr2=0x7f293c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:30.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.981+0000 7f295613e640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f293c0664d0 0x7f293c068990 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f2940001710 tx=0x7f2940009040 comp rx=0 tx=0).stop 2023-10-30T12:56:30.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.981+0000 7f295613e640 1 -- 172.21.15.125:0/3922755303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29500c6b80 msgr2=0x7f2950150370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:30.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.981+0000 7f295613e640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29500c6b80 0x7f2950150370 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f2944000c10 tx=0x7f2944019220 comp rx=0 tx=0).stop 2023-10-30T12:56:30.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.981+0000 7f295613e640 1 -- 172.21.15.125:0/3922755303 shutdown_connections 2023-10-30T12:56:30.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.981+0000 7f295613e640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f293c0664d0 0x7f293c068990 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:30.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.981+0000 7f295613e640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29500c6f60 0x7f29501508b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:30.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.981+0000 7f295613e640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29500c6b80 0x7f2950150370 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:30.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.981+0000 7f295613e640 1 --2- 172.21.15.125:0/3922755303 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29500c05d0 0x7f29500afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:30.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.981+0000 7f295613e640 1 -- 172.21.15.125:0/3922755303 >> 172.21.15.125:0/3922755303 conn(0x7f29500c7910 msgr2=0x7f29500c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:30.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.981+0000 7f295613e640 1 -- 172.21.15.125:0/3922755303 shutdown_connections 2023-10-30T12:56:30.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:30.981+0000 7f295613e640 1 -- 172.21.15.125:0/3922755303 wait complete. 2023-10-30T12:56:31.003 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:56:33.004 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:33.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.149+0000 7f85935aa640 1 Processor -- start 2023-10-30T12:56:33.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.149+0000 7f85935aa640 1 -- start start 2023-10-30T12:56:33.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.149+0000 7f85935aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f858c06f7a0 0x7f858c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:33.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.149+0000 7f85935aa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f858c0c6bd0 0x7f858c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:33.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.149+0000 7f85935aa640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f858c06a3d0 con 0x7f858c071110 2023-10-30T12:56:33.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.149+0000 7f85935aa640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f858c0700c0 con 0x7f858c0c6bd0 2023-10-30T12:56:33.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.149+0000 7f85935aa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f858c066cf0 con 0x7f858c06f7a0 2023-10-30T12:56:33.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.153+0000 7f8590b45640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f858c0c6bd0 0x7f858c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:33.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.153+0000 7f8590b45640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f858c0c6bd0 0x7f858c069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34562/0 (socket says 172.21.15.125:34562) 2023-10-30T12:56:33.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.153+0000 7f8590b45640 1 -- 172.21.15.125:0/2437731963 learned_addr learned my addr 172.21.15.125:0/2437731963 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:33.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.153+0000 7f8591346640 1 --2- 172.21.15.125:0/2437731963 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f858c06f7a0 0x7f858c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:33.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.153+0000 7f8590b45640 1 -- 172.21.15.125:0/2437731963 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f858c06f7a0 msgr2=0x7f858c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:33.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.153+0000 7f8590b45640 1 --2- 172.21.15.125:0/2437731963 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f858c06f7a0 0x7f858c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:33.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.153+0000 7f8590b45640 1 -- 172.21.15.125:0/2437731963 >> v1:172.21.15.125:6789/0 conn(0x7f858c071110 legacy=0x7f858c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:33.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.153+0000 7f8590b45640 1 -- 172.21.15.125:0/2437731963 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c0c70d0 con 0x7f858c0c6bd0 2023-10-30T12:56:33.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.153+0000 7f8590b45640 1 --2- 172.21.15.125:0/2437731963 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f858c0c6bd0 0x7f858c069e90 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f857c009870 tx=0x7f857c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe2c4f8e2588c18c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:33.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.153+0000 7f8583fff640 1 -- 172.21.15.125:0/2437731963 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c0a8a30 con 0x7f858c0c6bd0 2023-10-30T12:56:33.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f8583fff640 1 -- 172.21.15.125:0/2437731963 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857c0a8b90 con 0x7f858c0c6bd0 2023-10-30T12:56:33.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f8583fff640 1 -- 172.21.15.125:0/2437731963 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c0b0440 con 0x7f858c0c6bd0 2023-10-30T12:56:33.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 -- 172.21.15.125:0/2437731963 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f858c0c6bd0 msgr2=0x7f858c069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:33.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 --2- 172.21.15.125:0/2437731963 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f858c0c6bd0 0x7f858c069e90 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f857c009870 tx=0x7f857c01ed30 comp rx=0 tx=0).stop 2023-10-30T12:56:33.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 -- 172.21.15.125:0/2437731963 shutdown_connections 2023-10-30T12:56:33.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 --2- 172.21.15.125:0/2437731963 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f858c0c6bd0 0x7f858c069e90 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:33.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 --2- 172.21.15.125:0/2437731963 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f858c06f7a0 0x7f858c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:33.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 -- 172.21.15.125:0/2437731963 >> 172.21.15.125:0/2437731963 conn(0x7f858c0c7910 msgr2=0x7f858c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:33.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 -- 172.21.15.125:0/2437731963 shutdown_connections 2023-10-30T12:56:33.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 -- 172.21.15.125:0/2437731963 wait complete. 2023-10-30T12:56:33.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 Processor -- start 2023-10-30T12:56:33.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 -- start start 2023-10-30T12:56:33.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f858c06f7a0 0x7f858c0c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:33.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f858c071110 0x7f858c0aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:33.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f858c0c6bd0 0x7f858c0af020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:33.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c1efe00 con 0x7f858c06f7a0 2023-10-30T12:56:33.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858c155040 con 0x7f858c0c6bd0 2023-10-30T12:56:33.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f85935aa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f858c1eef00 con 0x7f858c071110 2023-10-30T12:56:33.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f8591346640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f858c06f7a0 0x7f858c0c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:33.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f8591346640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f858c06f7a0 0x7f858c0c9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34576/0 (socket says 172.21.15.125:34576) 2023-10-30T12:56:33.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f8591346640 1 -- 172.21.15.125:0/3804680885 learned_addr learned my addr 172.21.15.125:0/3804680885 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:33.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f8590b45640 1 --2- 172.21.15.125:0/3804680885 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f858c071110 0x7f858c0aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:33.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.157+0000 7f8591b47640 1 --2- 172.21.15.125:0/3804680885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f858c0c6bd0 0x7f858c0af020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:33.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f8591346640 1 -- 172.21.15.125:0/3804680885 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f858c071110 msgr2=0x7f858c0aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:33.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f8591346640 1 --2- 172.21.15.125:0/3804680885 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f858c071110 0x7f858c0aeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:33.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f8591346640 1 -- 172.21.15.125:0/3804680885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f858c0c6bd0 msgr2=0x7f858c0af020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:33.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f8591346640 1 --2- 172.21.15.125:0/3804680885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f858c0c6bd0 0x7f858c0af020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:33.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f8591346640 1 -- 172.21.15.125:0/3804680885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857c0094d0 con 0x7f858c06f7a0 2023-10-30T12:56:33.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f8591346640 1 --2- 172.21.15.125:0/3804680885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f858c06f7a0 0x7f858c0c9ec0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f858800e8a0 tx=0x7f858800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:33.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f8581ffb640 1 -- 172.21.15.125:0/3804680885 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8588009510 con 0x7f858c06f7a0 2023-10-30T12:56:33.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f8581ffb640 1 -- 172.21.15.125:0/3804680885 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f858800c940 con 0x7f858c06f7a0 2023-10-30T12:56:33.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f85935aa640 1 -- 172.21.15.125:0/3804680885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f858c1ef180 con 0x7f858c06f7a0 2023-10-30T12:56:33.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f8581ffb640 1 -- 172.21.15.125:0/3804680885 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858800ccb0 con 0x7f858c06f7a0 2023-10-30T12:56:33.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f85935aa640 1 -- 172.21.15.125:0/3804680885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f858c1ef740 con 0x7f858c06f7a0 2023-10-30T12:56:33.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f85935aa640 1 -- 172.21.15.125:0/3804680885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f858c0ae930 con 0x7f858c06f7a0 2023-10-30T12:56:33.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f8581ffb640 1 -- 172.21.15.125:0/3804680885 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8588013470 con 0x7f858c06f7a0 2023-10-30T12:56:33.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f8581ffb640 1 --2- 172.21.15.125:0/3804680885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85700664a0 0x7f8570068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:33.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f8581ffb640 1 -- 172.21.15.125:0/3804680885 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6856+0+0 (secure 0 0 0) 0x7f8588084360 con 0x7f858c06f7a0 2023-10-30T12:56:33.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.161+0000 7f8590b45640 1 --2- 172.21.15.125:0/3804680885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85700664a0 0x7f8570068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:33.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.165+0000 7f8590b45640 1 --2- 172.21.15.125:0/3804680885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85700664a0 0x7f8570068960 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f857c0b2000 tx=0x7f857c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:33.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.165+0000 7f8581ffb640 1 -- 172.21.15.125:0/3804680885 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f85880513f0 con 0x7f858c06f7a0 2023-10-30T12:56:33.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.345+0000 7f85935aa640 1 -- 172.21.15.125:0/3804680885 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f858c04e170 con 0x7f858c06f7a0 2023-10-30T12:56:33.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.345+0000 7f8581ffb640 1 -- 172.21.15.125:0/3804680885 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+6516 (secure 0 0 0) 0x7f8588054a10 con 0x7f858c06f7a0 2023-10-30T12:56:33.349 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:33.350 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7269,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":128}],"filesystems":[{"mdsmap":{"epoch":127,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:55:37.637700+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7113},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7113":{"gid":7113,"name":"c","rank":0,"incarnation":124,"state":"up:active","state_seq":40,"addr":"172.21.15.125:6837/1348687922","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":1348687922},{"type":"v1","addr":"172.21.15.125:6837","nonce":1348687922}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:33.350 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 128 2023-10-30T12:56:33.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.349+0000 7f85935aa640 1 -- 172.21.15.125:0/3804680885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85700664a0 msgr2=0x7f8570068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:33.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.349+0000 7f85935aa640 1 --2- 172.21.15.125:0/3804680885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85700664a0 0x7f8570068960 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f857c0b2000 tx=0x7f857c008040 comp rx=0 tx=0).stop 2023-10-30T12:56:33.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.349+0000 7f85935aa640 1 -- 172.21.15.125:0/3804680885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f858c06f7a0 msgr2=0x7f858c0c9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:33.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.349+0000 7f85935aa640 1 --2- 172.21.15.125:0/3804680885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f858c06f7a0 0x7f858c0c9ec0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f858800e8a0 tx=0x7f858800eca0 comp rx=0 tx=0).stop 2023-10-30T12:56:33.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.349+0000 7f85935aa640 1 -- 172.21.15.125:0/3804680885 shutdown_connections 2023-10-30T12:56:33.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.349+0000 7f85935aa640 1 --2- 172.21.15.125:0/3804680885 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85700664a0 0x7f8570068960 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:33.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.349+0000 7f85935aa640 1 --2- 172.21.15.125:0/3804680885 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f858c0c6bd0 0x7f858c0af020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:33.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.349+0000 7f85935aa640 1 --2- 172.21.15.125:0/3804680885 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f858c071110 0x7f858c0aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:33.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.349+0000 7f85935aa640 1 --2- 172.21.15.125:0/3804680885 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f858c06f7a0 0x7f858c0c9ec0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:33.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.349+0000 7f85935aa640 1 -- 172.21.15.125:0/3804680885 >> 172.21.15.125:0/3804680885 conn(0x7f858c0c7910 msgr2=0x7f858c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:33.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.349+0000 7f85935aa640 1 -- 172.21.15.125:0/3804680885 shutdown_connections 2023-10-30T12:56:33.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:33.349+0000 7f85935aa640 1 -- 172.21.15.125:0/3804680885 wait complete. 2023-10-30T12:56:33.370 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:56:35.372 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:35.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.509+0000 7fe93186b640 1 Processor -- start 2023-10-30T12:56:35.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.509+0000 7fe93186b640 1 -- start start 2023-10-30T12:56:35.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.509+0000 7fe93186b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe92c0c6bd0 0x7fe92c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.509+0000 7fe93186b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe92c071110 0x7fe92c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.509+0000 7fe93186b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe92c066cf0 con 0x7fe92c06f7a0 2023-10-30T12:56:35.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.509+0000 7fe93186b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92c0700c0 con 0x7fe92c0c6bd0 2023-10-30T12:56:35.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.509+0000 7fe93186b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe92c0c70d0 con 0x7fe92c071110 2023-10-30T12:56:35.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.513+0000 7fe92a7fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe92c0c6bd0 0x7fe92c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.513+0000 7fe92a7fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe92c0c6bd0 0x7fe92c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34578/0 (socket says 172.21.15.125:34578) 2023-10-30T12:56:35.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.513+0000 7fe92a7fc640 1 -- 172.21.15.125:0/1333741980 learned_addr learned my addr 172.21.15.125:0/1333741980 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:35.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.513+0000 7fe92affd640 1 --1- 172.21.15.125:0/1333741980 >> v1:172.21.15.125:6789/0 conn(0x7fe92c06f7a0 0x7fe92c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37580/0 (socket says 172.21.15.125:37580) 2023-10-30T12:56:35.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.513+0000 7fe92b7fe640 1 --2- 172.21.15.125:0/1333741980 >> v2:172.21.15.136:3300/0 conn(0x7fe92c071110 0x7fe92c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.513+0000 7fe92a7fc640 1 -- 172.21.15.125:0/1333741980 >> v2:172.21.15.136:3300/0 conn(0x7fe92c071110 msgr2=0x7fe92c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:35.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.513+0000 7fe92a7fc640 1 --2- 172.21.15.125:0/1333741980 >> v2:172.21.15.136:3300/0 conn(0x7fe92c071110 0x7fe92c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.513+0000 7fe92a7fc640 1 -- 172.21.15.125:0/1333741980 >> v1:172.21.15.125:6789/0 conn(0x7fe92c06f7a0 legacy=0x7fe92c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:35.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.513+0000 7fe92a7fc640 1 -- 172.21.15.125:0/1333741980 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe92c0c7390 con 0x7fe92c0c6bd0 2023-10-30T12:56:35.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.513+0000 7fe92a7fc640 1 --2- 172.21.15.125:0/1333741980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c0c6bd0 0x7fe92c06fb80 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe92c06cdd0 tx=0x7fe92001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39cbe4088aa1028a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:35.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.513+0000 7fe929ffb640 1 -- 172.21.15.125:0/1333741980 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9200a89f0 con 0x7fe92c0c6bd0 2023-10-30T12:56:35.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.513+0000 7fe929ffb640 1 -- 172.21.15.125:0/1333741980 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9200a8b50 con 0x7fe92c0c6bd0 2023-10-30T12:56:35.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.513+0000 7fe929ffb640 1 -- 172.21.15.125:0/1333741980 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9200b0440 con 0x7fe92c0c6bd0 2023-10-30T12:56:35.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 -- 172.21.15.125:0/1333741980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c0c6bd0 msgr2=0x7fe92c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:35.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 --2- 172.21.15.125:0/1333741980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c0c6bd0 0x7fe92c06fb80 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fe92c06cdd0 tx=0x7fe92001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 -- 172.21.15.125:0/1333741980 shutdown_connections 2023-10-30T12:56:35.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 --2- 172.21.15.125:0/1333741980 >> v2:172.21.15.136:3300/0 conn(0x7fe92c071110 0x7fe92c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 --2- 172.21.15.125:0/1333741980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c0c6bd0 0x7fe92c06fb80 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 -- 172.21.15.125:0/1333741980 >> 172.21.15.125:0/1333741980 conn(0x7fe92c0c7910 msgr2=0x7fe92c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:35.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 -- 172.21.15.125:0/1333741980 shutdown_connections 2023-10-30T12:56:35.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 -- 172.21.15.125:0/1333741980 wait complete. 2023-10-30T12:56:35.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 Processor -- start 2023-10-30T12:56:35.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 -- start start 2023-10-30T12:56:35.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c06f7a0 0x7fe92c0b00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe92c071110 0x7fe92c1eef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe92c0c6bd0 0x7fe92c0c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c0c1c00 con 0x7fe92c06f7a0 2023-10-30T12:56:35.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c1eead0 con 0x7fe92c0c6bd0 2023-10-30T12:56:35.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe93186b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe92c1f2980 con 0x7fe92c071110 2023-10-30T12:56:35.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe92affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c06f7a0 0x7fe92c0b00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe92affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c06f7a0 0x7fe92c0b00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34582/0 (socket says 172.21.15.125:34582) 2023-10-30T12:56:35.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe92affd640 1 -- 172.21.15.125:0/3650877628 learned_addr learned my addr 172.21.15.125:0/3650877628 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:35.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe92b7fe640 1 --2- 172.21.15.125:0/3650877628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe92c0c6bd0 0x7fe92c0c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe92a7fc640 1 --2- 172.21.15.125:0/3650877628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe92c071110 0x7fe92c1eef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe92affd640 1 -- 172.21.15.125:0/3650877628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe92c071110 msgr2=0x7fe92c1eef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:35.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe92affd640 1 --2- 172.21.15.125:0/3650877628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe92c071110 0x7fe92c1eef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe92affd640 1 -- 172.21.15.125:0/3650877628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe92c0c6bd0 msgr2=0x7fe92c0c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:35.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe92affd640 1 --2- 172.21.15.125:0/3650877628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe92c0c6bd0 0x7fe92c0c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe92affd640 1 -- 172.21.15.125:0/3650877628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9200094d0 con 0x7fe92c06f7a0 2023-10-30T12:56:35.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.517+0000 7fe92affd640 1 --2- 172.21.15.125:0/3650877628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c06f7a0 0x7fe92c0b00d0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fe91c00c430 tx=0x7fe91c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:35.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.521+0000 7fe913fff640 1 -- 172.21.15.125:0/3650877628 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c0074e0 con 0x7fe92c06f7a0 2023-10-30T12:56:35.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.521+0000 7fe913fff640 1 -- 172.21.15.125:0/3650877628 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe91c007640 con 0x7fe92c06f7a0 2023-10-30T12:56:35.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.521+0000 7fe93186b640 1 -- 172.21.15.125:0/3650877628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe92c0c1e50 con 0x7fe92c06f7a0 2023-10-30T12:56:35.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.521+0000 7fe913fff640 1 -- 172.21.15.125:0/3650877628 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe91c004a10 con 0x7fe92c06f7a0 2023-10-30T12:56:35.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.521+0000 7fe93186b640 1 -- 172.21.15.125:0/3650877628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe92c1ee760 con 0x7fe92c06f7a0 2023-10-30T12:56:35.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.521+0000 7fe93186b640 1 -- 172.21.15.125:0/3650877628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe92c04e170 con 0x7fe92c06f7a0 2023-10-30T12:56:35.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.521+0000 7fe913fff640 1 -- 172.21.15.125:0/3650877628 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe91c004b70 con 0x7fe92c06f7a0 2023-10-30T12:56:35.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.521+0000 7fe913fff640 1 --2- 172.21.15.125:0/3650877628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe9180664a0 0x7fe918068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.521+0000 7fe913fff640 1 -- 172.21.15.125:0/3650877628 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7fe91c083ea0 con 0x7fe92c06f7a0 2023-10-30T12:56:35.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.521+0000 7fe92a7fc640 1 --2- 172.21.15.125:0/3650877628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe9180664a0 0x7fe918068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.525+0000 7fe92a7fc640 1 --2- 172.21.15.125:0/3650877628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe9180664a0 0x7fe918068960 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe9200b2000 tx=0x7fe9200aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:35.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.525+0000 7fe913fff640 1 -- 172.21.15.125:0/3650877628 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe91c050d20 con 0x7fe92c06f7a0 2023-10-30T12:56:35.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.705+0000 7fe93186b640 1 -- 172.21.15.125:0/3650877628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe92c066510 con 0x7fe92c06f7a0 2023-10-30T12:56:35.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.705+0000 7fe913fff640 1 -- 172.21.15.125:0/3650877628 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5720 (secure 0 0 0) 0x7fe91c054340 con 0x7fe92c06f7a0 2023-10-30T12:56:35.710 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:35.710 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:35.422971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:35.710 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 129 2023-10-30T12:56:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.709+0000 7fe93186b640 1 -- 172.21.15.125:0/3650877628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe9180664a0 msgr2=0x7fe918068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.709+0000 7fe93186b640 1 --2- 172.21.15.125:0/3650877628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe9180664a0 0x7fe918068960 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fe9200b2000 tx=0x7fe9200aa070 comp rx=0 tx=0).stop 2023-10-30T12:56:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.709+0000 7fe93186b640 1 -- 172.21.15.125:0/3650877628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c06f7a0 msgr2=0x7fe92c0b00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.709+0000 7fe93186b640 1 --2- 172.21.15.125:0/3650877628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c06f7a0 0x7fe92c0b00d0 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7fe91c00c430 tx=0x7fe91c00c8f0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.709+0000 7fe93186b640 1 -- 172.21.15.125:0/3650877628 shutdown_connections 2023-10-30T12:56:35.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.709+0000 7fe93186b640 1 --2- 172.21.15.125:0/3650877628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe9180664a0 0x7fe918068960 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.709+0000 7fe93186b640 1 --2- 172.21.15.125:0/3650877628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe92c0c6bd0 0x7fe92c0c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.709+0000 7fe93186b640 1 --2- 172.21.15.125:0/3650877628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe92c071110 0x7fe92c1eef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.709+0000 7fe93186b640 1 --2- 172.21.15.125:0/3650877628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe92c06f7a0 0x7fe92c0b00d0 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.709+0000 7fe93186b640 1 -- 172.21.15.125:0/3650877628 >> 172.21.15.125:0/3650877628 conn(0x7fe92c0c7910 msgr2=0x7fe92c0c99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:35.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.709+0000 7fe93186b640 1 -- 172.21.15.125:0/3650877628 shutdown_connections 2023-10-30T12:56:35.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.709+0000 7fe93186b640 1 -- 172.21.15.125:0/3650877628 wait complete. 2023-10-30T12:56:35.731 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T12:56:35.731 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:56:35.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:35.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.865+0000 7fd03ff9d640 1 Processor -- start 2023-10-30T12:56:35.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.865+0000 7fd03ff9d640 1 -- start start 2023-10-30T12:56:35.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03ff9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd0380c6b80 0x7fd0380bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03ff9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd03813f4a0 0x7fd0380c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03ff9d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0380c0a50 con 0x7fd0380c6f60 2023-10-30T12:56:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03ff9d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd03806f6e0 con 0x7fd0380c6b80 2023-10-30T12:56:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03ff9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd03806f9c0 con 0x7fd03813f4a0 2023-10-30T12:56:35.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03d538640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd0380c6f60 0x7fd03813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37594/0 (socket says 172.21.15.125:37594) 2023-10-30T12:56:35.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03d538640 1 -- 172.21.15.125:0/1087651651 learned_addr learned my addr 172.21.15.125:0/1087651651 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:56:35.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03e53a640 1 --2- 172.21.15.125:0/1087651651 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd03813f4a0 0x7fd0380c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03dd39640 1 --2- 172.21.15.125:0/1087651651 >> v2:172.21.15.136:3300/0 conn(0x7fd0380c6b80 0x7fd0380bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 -- 172.21.15.125:0/1087651651 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657633513 0 0) 0x7fd0380c0a50 con 0x7fd0380c6f60 2023-10-30T12:56:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 -- 172.21.15.125:0/1087651651 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd024003530 con 0x7fd0380c6f60 2023-10-30T12:56:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 -- 172.21.15.125:0/1087651651 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 766980517 0 0) 0x7fd024003530 con 0x7fd0380c6f60 2023-10-30T12:56:35.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 -- 172.21.15.125:0/1087651651 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd03813f4a0 msgr2=0x7fd0380c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:35.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 --2- 172.21.15.125:0/1087651651 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd03813f4a0 0x7fd0380c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 -- 172.21.15.125:0/1087651651 >> v2:172.21.15.136:3300/0 conn(0x7fd0380c6b80 msgr2=0x7fd0380bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:35.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 --2- 172.21.15.125:0/1087651651 >> v2:172.21.15.136:3300/0 conn(0x7fd0380c6b80 0x7fd0380bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 -- 172.21.15.125:0/1087651651 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03806fca0 con 0x7fd0380c6f60 2023-10-30T12:56:35.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 -- 172.21.15.125:0/1087651651 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fd028001680 con 0x7fd0380c6f60 2023-10-30T12:56:35.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 -- 172.21.15.125:0/1087651651 >> v1:172.21.15.125:6789/0 conn(0x7fd0380c6f60 legacy=0x7fd03813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:35.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 --2- 172.21.15.125:0/1087651651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd024003980 0x7fd024003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 --2- 172.21.15.125:0/1087651651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0240047d0 0x7fd024004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 --2- 172.21.15.125:0/1087651651 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd03813f4a0 0x7fd024005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 -- 172.21.15.125:0/1087651651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03806f6e0 con 0x7fd0240047d0 2023-10-30T12:56:35.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 -- 172.21.15.125:0/1087651651 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03806f9c0 con 0x7fd024003980 2023-10-30T12:56:35.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03cd37640 1 -- 172.21.15.125:0/1087651651 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd02400dba0 con 0x7fd03813f4a0 2023-10-30T12:56:35.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03d538640 1 --2- 172.21.15.125:0/1087651651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0240047d0 0x7fd024004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03dd39640 1 --2- 172.21.15.125:0/1087651651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd024003980 0x7fd024003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03e53a640 1 --2- 172.21.15.125:0/1087651651 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd03813f4a0 0x7fd024005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03d538640 1 -- 172.21.15.125:0/1087651651 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd03813f4a0 msgr2=0x7fd024005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03d538640 1 --2- 172.21.15.125:0/1087651651 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd03813f4a0 0x7fd024005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03d538640 1 -- 172.21.15.125:0/1087651651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd024003980 msgr2=0x7fd024003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03d538640 1 --2- 172.21.15.125:0/1087651651 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd024003980 0x7fd024003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.869+0000 7fd03d538640 1 -- 172.21.15.125:0/1087651651 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd024003530 con 0x7fd0240047d0 2023-10-30T12:56:35.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.873+0000 7fd03d538640 1 --2- 172.21.15.125:0/1087651651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0240047d0 0x7fd024004c50 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fd02800c240 tx=0x7fd02801f3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f4c700aa96e65f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:35.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.873+0000 7fd03cd37640 1 -- 172.21.15.125:0/1087651651 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0280a8d00 con 0x7fd0240047d0 2023-10-30T12:56:35.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.873+0000 7fd03cd37640 1 -- 172.21.15.125:0/1087651651 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0280a8e60 con 0x7fd0240047d0 2023-10-30T12:56:35.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.873+0000 7fd03cd37640 1 -- 172.21.15.125:0/1087651651 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd028005ae0 con 0x7fd0240047d0 2023-10-30T12:56:35.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.873+0000 7fd03ff9d640 1 -- 172.21.15.125:0/1087651651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0240047d0 msgr2=0x7fd024004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:35.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.873+0000 7fd03ff9d640 1 --2- 172.21.15.125:0/1087651651 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0240047d0 0x7fd024004c50 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7fd02800c240 tx=0x7fd02801f3d0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.873+0000 7fd03e53a640 1 -- 172.21.15.125:0/1087651651 reap_dead start 2023-10-30T12:56:35.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.873+0000 7fd03ff9d640 1 -- 172.21.15.125:0/1087651651 shutdown_connections 2023-10-30T12:56:35.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.873+0000 7fd03ff9d640 1 -- 172.21.15.125:0/1087651651 >> 172.21.15.125:0/1087651651 conn(0x7fd0380c7910 msgr2=0x7fd0380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:35.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.873+0000 7fd03ff9d640 1 -- 172.21.15.125:0/1087651651 shutdown_connections 2023-10-30T12:56:35.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.873+0000 7fd03ff9d640 1 -- 172.21.15.125:0/1087651651 wait complete. 2023-10-30T12:56:35.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03ff9d640 1 Processor -- start 2023-10-30T12:56:35.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03ff9d640 1 -- start start 2023-10-30T12:56:35.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03ff9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0240047d0 0x7fd0381f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03ff9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd038139170 0x7fd0381efdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03ff9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0381f02f0 0x7fd0381373f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03ff9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0381f0770 con 0x7fd0381f02f0 2023-10-30T12:56:35.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03ff9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0380c2de0 con 0x7fd0240047d0 2023-10-30T12:56:35.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03ff9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0381f27c0 con 0x7fd038139170 2023-10-30T12:56:35.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03e53a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0381f02f0 0x7fd0381373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03e53a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0381f02f0 0x7fd0381373f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34600/0 (socket says 172.21.15.125:34600) 2023-10-30T12:56:35.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03e53a640 1 -- 172.21.15.125:0/4129608632 learned_addr learned my addr 172.21.15.125:0/4129608632 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:35.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03d538640 1 --2- 172.21.15.125:0/4129608632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd038139170 0x7fd0381efdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03dd39640 1 --2- 172.21.15.125:0/4129608632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0240047d0 0x7fd0381f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03e53a640 1 -- 172.21.15.125:0/4129608632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd038139170 msgr2=0x7fd0381efdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:35.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03e53a640 1 --2- 172.21.15.125:0/4129608632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd038139170 0x7fd0381efdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03e53a640 1 -- 172.21.15.125:0/4129608632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0240047d0 msgr2=0x7fd0381f1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:35.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03e53a640 1 --2- 172.21.15.125:0/4129608632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0240047d0 0x7fd0381f1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:35.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03e53a640 1 -- 172.21.15.125:0/4129608632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0381f2aa0 con 0x7fd0381f02f0 2023-10-30T12:56:35.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03e53a640 1 --2- 172.21.15.125:0/4129608632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0381f02f0 0x7fd0381373f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fd03806f4d0 tx=0x7fd0300115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:35.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd02e7fc640 1 -- 172.21.15.125:0/4129608632 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03000e070 con 0x7fd0381f02f0 2023-10-30T12:56:35.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd02e7fc640 1 -- 172.21.15.125:0/4129608632 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd030011ed0 con 0x7fd0381f02f0 2023-10-30T12:56:35.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03ff9d640 1 -- 172.21.15.125:0/4129608632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd03813a2a0 con 0x7fd0381f02f0 2023-10-30T12:56:35.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd02e7fc640 1 -- 172.21.15.125:0/4129608632 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0300039a0 con 0x7fd0381f02f0 2023-10-30T12:56:35.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03ff9d640 1 -- 172.21.15.125:0/4129608632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03813a860 con 0x7fd0381f02f0 2023-10-30T12:56:35.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.877+0000 7fd03ff9d640 1 -- 172.21.15.125:0/4129608632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd03806b0c0 con 0x7fd0381f02f0 2023-10-30T12:56:35.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.881+0000 7fd02e7fc640 1 -- 172.21.15.125:0/4129608632 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd030003b20 con 0x7fd0381f02f0 2023-10-30T12:56:35.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.881+0000 7fd02e7fc640 1 --2- 172.21.15.125:0/4129608632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd01c0664a0 0x7fd01c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:35.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.881+0000 7fd02e7fc640 1 -- 172.21.15.125:0/4129608632 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7fd030084b90 con 0x7fd0381f02f0 2023-10-30T12:56:35.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.881+0000 7fd03dd39640 1 --2- 172.21.15.125:0/4129608632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd01c0664a0 0x7fd01c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:35.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.881+0000 7fd03dd39640 1 --2- 172.21.15.125:0/4129608632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd01c0664a0 0x7fd01c068960 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd0340099b0 tx=0x7fd034008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:35.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:35.885+0000 7fd02e7fc640 1 -- 172.21.15.125:0/4129608632 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd030051b30 con 0x7fd0381f02f0 2023-10-30T12:56:36.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:36.065+0000 7fd03ff9d640 1 -- 172.21.15.125:0/4129608632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd03804e170 con 0x7fd0381f02f0 2023-10-30T12:56:36.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:36.065+0000 7fd02e7fc640 1 -- 172.21.15.125:0/4129608632 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5720 (secure 0 0 0) 0x7fd030055150 con 0x7fd0381f02f0 2023-10-30T12:56:36.068 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:36.068 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:35.422971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:36.069 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 129 2023-10-30T12:56:36.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:36.065+0000 7fd03ff9d640 1 -- 172.21.15.125:0/4129608632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd01c0664a0 msgr2=0x7fd01c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:36.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:36.065+0000 7fd03ff9d640 1 --2- 172.21.15.125:0/4129608632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd01c0664a0 0x7fd01c068960 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fd0340099b0 tx=0x7fd034008040 comp rx=0 tx=0).stop 2023-10-30T12:56:36.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:36.065+0000 7fd03ff9d640 1 -- 172.21.15.125:0/4129608632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0381f02f0 msgr2=0x7fd0381373f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:36.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:36.065+0000 7fd03ff9d640 1 --2- 172.21.15.125:0/4129608632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0381f02f0 0x7fd0381373f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7fd03806f4d0 tx=0x7fd0300115a0 comp rx=0 tx=0).stop 2023-10-30T12:56:36.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:36.065+0000 7fd03ff9d640 1 -- 172.21.15.125:0/4129608632 shutdown_connections 2023-10-30T12:56:36.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:36.065+0000 7fd03ff9d640 1 --2- 172.21.15.125:0/4129608632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd01c0664a0 0x7fd01c068960 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:36.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:36.065+0000 7fd03ff9d640 1 --2- 172.21.15.125:0/4129608632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0381f02f0 0x7fd0381373f0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:36.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:36.065+0000 7fd03ff9d640 1 --2- 172.21.15.125:0/4129608632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd038139170 0x7fd0381efdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:36.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:36.065+0000 7fd03ff9d640 1 --2- 172.21.15.125:0/4129608632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0240047d0 0x7fd0381f1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:36.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:36.065+0000 7fd03ff9d640 1 -- 172.21.15.125:0/4129608632 >> 172.21.15.125:0/4129608632 conn(0x7fd0380c7910 msgr2=0x7fd03813ec40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:36.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:36.065+0000 7fd03ff9d640 1 -- 172.21.15.125:0/4129608632 shutdown_connections 2023-10-30T12:56:36.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:36.065+0000 7fd03ff9d640 1 -- 172.21.15.125:0/4129608632 wait complete. 2023-10-30T12:56:36.091 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 129, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 129, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7269": { "addr": "172.21.15.125:6836/739564283", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 739564283, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 739564283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7269, "incarnation": 129, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:56:35.422971+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7269 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:56:38.092 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:38.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65ba267640 1 Processor -- start 2023-10-30T12:56:38.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65ba267640 1 -- start start 2023-10-30T12:56:38.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65ba267640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65b41538f0 0x7f65b4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:38.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65ba267640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b415cd50 0x7f65b40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:38.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65ba267640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f65b40cefa0 0x7f65b40cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:38.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65ba267640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b415d300 con 0x7f65b41538f0 2023-10-30T12:56:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65ba267640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65b415d5e0 con 0x7f65b40cefa0 2023-10-30T12:56:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65ba267640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b415d8c0 con 0x7f65b415cd50 2023-10-30T12:56:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65b9265640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65b41538f0 0x7f65b4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65b9265640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f65b41538f0 0x7f65b4159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34602/0 (socket says 172.21.15.125:34602) 2023-10-30T12:56:38.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65b9265640 1 -- 172.21.15.125:0/2386013377 learned_addr learned my addr 172.21.15.125:0/2386013377 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:38.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65b8a64640 1 --2- 172.21.15.125:0/2386013377 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b415cd50 0x7f65b40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:38.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65b9a66640 1 --2- 172.21.15.125:0/2386013377 >> v2:172.21.15.136:3300/0 conn(0x7f65b40cefa0 0x7f65b40cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:38.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65b9265640 1 -- 172.21.15.125:0/2386013377 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b415cd50 msgr2=0x7f65b40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:38.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65b9265640 1 --2- 172.21.15.125:0/2386013377 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b415cd50 0x7f65b40c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:38.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65b9265640 1 -- 172.21.15.125:0/2386013377 >> v2:172.21.15.136:3300/0 conn(0x7f65b40cefa0 msgr2=0x7f65b40cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:38.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65b9265640 1 --2- 172.21.15.125:0/2386013377 >> v2:172.21.15.136:3300/0 conn(0x7f65b40cefa0 0x7f65b40cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:38.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65b9265640 1 -- 172.21.15.125:0/2386013377 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b415dba0 con 0x7f65b41538f0 2023-10-30T12:56:38.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.237+0000 7f65b9a66640 1 --2- 172.21.15.125:0/2386013377 >> v2:172.21.15.136:3300/0 conn(0x7f65b40cefa0 0x7f65b40cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:56:38.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.241+0000 7f65b9265640 1 --2- 172.21.15.125:0/2386013377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b41538f0 0x7f65b4159fb0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f65b40b1ae0 tx=0x7f65b001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c24fb5f0f84233 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:38.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.241+0000 7f65abfff640 1 -- 172.21.15.125:0/2386013377 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b00a89b0 con 0x7f65b41538f0 2023-10-30T12:56:38.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.241+0000 7f65abfff640 1 -- 172.21.15.125:0/2386013377 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b00a8b10 con 0x7f65b41538f0 2023-10-30T12:56:38.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.241+0000 7f65abfff640 1 -- 172.21.15.125:0/2386013377 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b00a8ea0 con 0x7f65b41538f0 2023-10-30T12:56:38.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.241+0000 7f65ba267640 1 -- 172.21.15.125:0/2386013377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b41538f0 msgr2=0x7f65b4159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:38.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.241+0000 7f65ba267640 1 --2- 172.21.15.125:0/2386013377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b41538f0 0x7f65b4159fb0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f65b40b1ae0 tx=0x7f65b001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:56:38.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.241+0000 7f65ba267640 1 -- 172.21.15.125:0/2386013377 shutdown_connections 2023-10-30T12:56:38.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.241+0000 7f65ba267640 1 --2- 172.21.15.125:0/2386013377 >> v2:172.21.15.136:3300/0 conn(0x7f65b40cefa0 0x7f65b40cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:38.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.241+0000 7f65ba267640 1 --2- 172.21.15.125:0/2386013377 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b415cd50 0x7f65b40c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:38.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.241+0000 7f65ba267640 1 --2- 172.21.15.125:0/2386013377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b41538f0 0x7f65b4159fb0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:38.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.241+0000 7f65ba267640 1 -- 172.21.15.125:0/2386013377 >> 172.21.15.125:0/2386013377 conn(0x7f65b406a640 msgr2=0x7f65b4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:38.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.241+0000 7f65ba267640 1 -- 172.21.15.125:0/2386013377 shutdown_connections 2023-10-30T12:56:38.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.241+0000 7f65ba267640 1 -- 172.21.15.125:0/2386013377 wait complete. 2023-10-30T12:56:38.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65ba267640 1 Processor -- start 2023-10-30T12:56:38.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65ba267640 1 -- start start 2023-10-30T12:56:38.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65ba267640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b40cefa0 0x7f65b41eefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:38.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65ba267640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65b41538f0 0x7f65b414f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:38.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65ba267640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b415cd50 0x7f65b41f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:38.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65ba267640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b4147e90 con 0x7f65b415cd50 2023-10-30T12:56:38.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65ba267640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65b414fd60 con 0x7f65b41538f0 2023-10-30T12:56:38.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65ba267640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b41f3490 con 0x7f65b40cefa0 2023-10-30T12:56:38.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65b9a66640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b415cd50 0x7f65b41f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:38.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65b9a66640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b415cd50 0x7f65b41f2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34612/0 (socket says 172.21.15.125:34612) 2023-10-30T12:56:38.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65b9a66640 1 -- 172.21.15.125:0/4248868641 learned_addr learned my addr 172.21.15.125:0/4248868641 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:38.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65b9265640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b40cefa0 0x7f65b41eefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:38.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65b8a64640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65b41538f0 0x7f65b414f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:38.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65b9a66640 1 -- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b40cefa0 msgr2=0x7f65b41eefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:38.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65b9a66640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b40cefa0 0x7f65b41eefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:38.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65b9a66640 1 -- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65b41538f0 msgr2=0x7f65b414f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:38.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65b9a66640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65b41538f0 0x7f65b414f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:38.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65b9a66640 1 -- 172.21.15.125:0/4248868641 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b00094d0 con 0x7f65b415cd50 2023-10-30T12:56:38.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65b8a64640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65b41538f0 0x7f65b414f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:56:38.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65b9a66640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b415cd50 0x7f65b41f2f50 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f65ac007e40 tx=0x7f65ac00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:38.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65a9ffb640 1 -- 172.21.15.125:0/4248868641 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac003ba0 con 0x7f65b415cd50 2023-10-30T12:56:38.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65a9ffb640 1 -- 172.21.15.125:0/4248868641 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ac00f040 con 0x7f65b415cd50 2023-10-30T12:56:38.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65ba267640 1 -- 172.21.15.125:0/4248868641 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65b414dba0 con 0x7f65b415cd50 2023-10-30T12:56:38.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65a9ffb640 1 -- 172.21.15.125:0/4248868641 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ac00de80 con 0x7f65b415cd50 2023-10-30T12:56:38.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65ba267640 1 -- 172.21.15.125:0/4248868641 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65b414ddf0 con 0x7f65b415cd50 2023-10-30T12:56:38.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.245+0000 7f65ba267640 1 -- 172.21.15.125:0/4248868641 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65b4147fd0 con 0x7f65b415cd50 2023-10-30T12:56:38.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.249+0000 7f65a9ffb640 1 -- 172.21.15.125:0/4248868641 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f65ac004030 con 0x7f65b415cd50 2023-10-30T12:56:38.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.249+0000 7f65a9ffb640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65980664d0 0x7f6598068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:38.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.249+0000 7f65a9ffb640 1 -- 172.21.15.125:0/4248868641 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7f65ac084620 con 0x7f65b415cd50 2023-10-30T12:56:38.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.249+0000 7f65b9265640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65980664d0 0x7f6598068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:38.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.249+0000 7f65b9265640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65980664d0 0x7f6598068990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f65b00b2000 tx=0x7f65b0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:38.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.253+0000 7f65a9ffb640 1 -- 172.21.15.125:0/4248868641 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f65ac0515e0 con 0x7f65b415cd50 2023-10-30T12:56:38.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.433+0000 7f65ba267640 1 -- 172.21.15.125:0/4248868641 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65b4159730 con 0x7f65b415cd50 2023-10-30T12:56:38.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.433+0000 7f65a9ffb640 1 -- 172.21.15.125:0/4248868641 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f65ac054c00 con 0x7f65b415cd50 2023-10-30T12:56:38.437 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:38.437 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:35.422971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:38.438 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 129 2023-10-30T12:56:38.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.437+0000 7f65ba267640 1 -- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65980664d0 msgr2=0x7f6598068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:38.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.437+0000 7f65ba267640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65980664d0 0x7f6598068990 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f65b00b2000 tx=0x7f65b0009040 comp rx=0 tx=0).stop 2023-10-30T12:56:38.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.437+0000 7f65ba267640 1 -- 172.21.15.125:0/4248868641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b415cd50 msgr2=0x7f65b41f2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:38.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.437+0000 7f65ba267640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b415cd50 0x7f65b41f2f50 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f65ac007e40 tx=0x7f65ac00d4a0 comp rx=0 tx=0).stop 2023-10-30T12:56:38.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.437+0000 7f65ba267640 1 -- 172.21.15.125:0/4248868641 shutdown_connections 2023-10-30T12:56:38.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.437+0000 7f65ba267640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65980664d0 0x7f6598068990 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:38.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.437+0000 7f65ba267640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65b415cd50 0x7f65b41f2f50 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:38.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.437+0000 7f65ba267640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65b41538f0 0x7f65b414f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:38.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.437+0000 7f65ba267640 1 --2- 172.21.15.125:0/4248868641 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65b40cefa0 0x7f65b41eefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:38.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.437+0000 7f65ba267640 1 -- 172.21.15.125:0/4248868641 >> 172.21.15.125:0/4248868641 conn(0x7f65b406a640 msgr2=0x7f65b40c5300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:38.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.437+0000 7f65ba267640 1 -- 172.21.15.125:0/4248868641 shutdown_connections 2023-10-30T12:56:38.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:38.437+0000 7f65ba267640 1 -- 172.21.15.125:0/4248868641 wait complete. 2023-10-30T12:56:38.458 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:56:40.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:40.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.609+0000 7f662db0f640 1 Processor -- start 2023-10-30T12:56:40.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.609+0000 7f662db0f640 1 -- start start 2023-10-30T12:56:40.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f662db0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f662806d290 0x7f662806a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:40.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f662db0f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66280c6c10 con 0x7f6628069e90 2023-10-30T12:56:40.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f662db0f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66280c6ef0 con 0x7f662815a410 2023-10-30T12:56:40.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f662db0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66280c8570 con 0x7f662806d290 2023-10-30T12:56:40.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f66277fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6628069e90 0x7f66280c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37596/0 (socket says 172.21.15.125:37596) 2023-10-30T12:56:40.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f66277fe640 1 -- 172.21.15.125:0/3291568268 learned_addr learned my addr 172.21.15.125:0/3291568268 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:56:40.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6627fff640 1 --2- 172.21.15.125:0/3291568268 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f662806d290 0x7f662806a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:40.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 -- 172.21.15.125:0/3291568268 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157984119 0 0) 0x7f66280c6c10 con 0x7f6628069e90 2023-10-30T12:56:40.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 -- 172.21.15.125:0/3291568268 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6614003530 con 0x7f6628069e90 2023-10-30T12:56:40.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 -- 172.21.15.125:0/3291568268 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3772819814 0 0) 0x7f6614003530 con 0x7f6628069e90 2023-10-30T12:56:40.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 -- 172.21.15.125:0/3291568268 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f662806d290 msgr2=0x7f662806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:40.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 --2- 172.21.15.125:0/3291568268 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f662806d290 0x7f662806a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:40.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 -- 172.21.15.125:0/3291568268 >> v1:172.21.15.136:6789/0 conn(0x7f662815a410 legacy=0x7f662806cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:40.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 -- 172.21.15.125:0/3291568268 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66280c71d0 con 0x7f6628069e90 2023-10-30T12:56:40.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 -- 172.21.15.125:0/3291568268 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f6618001680 con 0x7f6628069e90 2023-10-30T12:56:40.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 -- 172.21.15.125:0/3291568268 >> v1:172.21.15.125:6789/0 conn(0x7f6628069e90 legacy=0x7f66280c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:40.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 --2- 172.21.15.125:0/3291568268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66140039b0 0x7f6614003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:40.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 --2- 172.21.15.125:0/3291568268 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f662806d290 0x7f66140047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:40.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 --2- 172.21.15.125:0/3291568268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6614004d10 0x7f6614005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:40.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 -- 172.21.15.125:0/3291568268 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66280c8570 con 0x7f6614004d10 2023-10-30T12:56:40.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 -- 172.21.15.125:0/3291568268 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661400b990 con 0x7f66140039b0 2023-10-30T12:56:40.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6626ffd640 1 -- 172.21.15.125:0/3291568268 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f661400bb50 con 0x7f662806d290 2023-10-30T12:56:40.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6627fff640 1 --2- 172.21.15.125:0/3291568268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6614004d10 0x7f6614005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:40.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f66277fe640 1 --2- 172.21.15.125:0/3291568268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66140039b0 0x7f6614003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:40.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f661ffff640 1 --2- 172.21.15.125:0/3291568268 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f662806d290 0x7f66140047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:40.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6627fff640 1 -- 172.21.15.125:0/3291568268 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f662806d290 msgr2=0x7f66140047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:40.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6627fff640 1 --2- 172.21.15.125:0/3291568268 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f662806d290 0x7f66140047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:40.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6627fff640 1 -- 172.21.15.125:0/3291568268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66140039b0 msgr2=0x7f6614003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:40.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6627fff640 1 --2- 172.21.15.125:0/3291568268 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66140039b0 0x7f6614003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:40.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.613+0000 7f6627fff640 1 -- 172.21.15.125:0/3291568268 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6614003530 con 0x7f6614004d10 2023-10-30T12:56:40.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.617+0000 7f6627fff640 1 --2- 172.21.15.125:0/3291568268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6614004d10 0x7f6614005190 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f6620009fc0 tx=0x7f662001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f8dd26bb1ad7fd5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:40.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.617+0000 7f6626ffd640 1 -- 172.21.15.125:0/3291568268 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6618007ea0 con 0x7f6614004d10 2023-10-30T12:56:40.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.617+0000 7f6626ffd640 1 -- 172.21.15.125:0/3291568268 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662000f040 con 0x7f6614004d10 2023-10-30T12:56:40.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.617+0000 7f6626ffd640 1 -- 172.21.15.125:0/3291568268 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66200ad440 con 0x7f6614004d10 2023-10-30T12:56:40.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.617+0000 7f662db0f640 1 -- 172.21.15.125:0/3291568268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6614004d10 msgr2=0x7f6614005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:40.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.617+0000 7f662db0f640 1 --2- 172.21.15.125:0/3291568268 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6614004d10 0x7f6614005190 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f6620009fc0 tx=0x7f662001cc60 comp rx=0 tx=0).stop 2023-10-30T12:56:40.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.617+0000 7f6627fff640 1 -- 172.21.15.125:0/3291568268 reap_dead start 2023-10-30T12:56:40.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.617+0000 7f662db0f640 1 -- 172.21.15.125:0/3291568268 shutdown_connections 2023-10-30T12:56:40.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.617+0000 7f662db0f640 1 -- 172.21.15.125:0/3291568268 >> 172.21.15.125:0/3291568268 conn(0x7f662806dfd0 msgr2=0x7f662806e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:40.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.617+0000 7f662db0f640 1 -- 172.21.15.125:0/3291568268 shutdown_connections 2023-10-30T12:56:40.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.617+0000 7f662db0f640 1 -- 172.21.15.125:0/3291568268 wait complete. 2023-10-30T12:56:40.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f662db0f640 1 Processor -- start 2023-10-30T12:56:40.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f662db0f640 1 -- start start 2023-10-30T12:56:40.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f662db0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6614004d10 0x7f6628154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:40.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f662db0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f662815d8f0 0x7f66280736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:40.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f662db0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66281f1910 0x7f66281f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:40.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f662db0f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6628156d30 con 0x7f66281f1910 2023-10-30T12:56:40.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f662db0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66281552c0 con 0x7f6614004d10 2023-10-30T12:56:40.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f662db0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66281ee320 con 0x7f662815d8f0 2023-10-30T12:56:40.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f6627fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66281f1910 0x7f66281f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:40.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f6627fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66281f1910 0x7f66281f1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34624/0 (socket says 172.21.15.125:34624) 2023-10-30T12:56:40.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f6627fff640 1 -- 172.21.15.125:0/194589875 learned_addr learned my addr 172.21.15.125:0/194589875 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:40.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f661ffff640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f662815d8f0 0x7f66280736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:40.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f66277fe640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6614004d10 0x7f6628154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:40.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f6627fff640 1 -- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f662815d8f0 msgr2=0x7f66280736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:40.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f6627fff640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f662815d8f0 0x7f66280736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:40.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f6627fff640 1 -- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6614004d10 msgr2=0x7f6628154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:40.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f6627fff640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6614004d10 0x7f6628154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:40.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f6627fff640 1 -- 172.21.15.125:0/194589875 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66281eee60 con 0x7f66281f1910 2023-10-30T12:56:40.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f66277fe640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6614004d10 0x7f6628154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:56:40.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f6627fff640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66281f1910 0x7f66281f1d90 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f66200b0040 tx=0x7f6620009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:40.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f6624ff9640 1 -- 172.21.15.125:0/194589875 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66200039a0 con 0x7f66281f1910 2023-10-30T12:56:40.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f6624ff9640 1 -- 172.21.15.125:0/194589875 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6620003b00 con 0x7f66281f1910 2023-10-30T12:56:40.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f6624ff9640 1 -- 172.21.15.125:0/194589875 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6620003e70 con 0x7f66281f1910 2023-10-30T12:56:40.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f662db0f640 1 -- 172.21.15.125:0/194589875 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6628155020 con 0x7f66281f1910 2023-10-30T12:56:40.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.621+0000 7f662db0f640 1 -- 172.21.15.125:0/194589875 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66281ee810 con 0x7f66281f1910 2023-10-30T12:56:40.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.625+0000 7f662db0f640 1 -- 172.21.15.125:0/194589875 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66281eeff0 con 0x7f66281f1910 2023-10-30T12:56:40.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.625+0000 7f6624ff9640 1 -- 172.21.15.125:0/194589875 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f66200b4040 con 0x7f66281f1910 2023-10-30T12:56:40.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.625+0000 7f6624ff9640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66080664a0 0x7f6608068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:40.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.625+0000 7f6624ff9640 1 -- 172.21.15.125:0/194589875 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7f662011dfd0 con 0x7f66281f1910 2023-10-30T12:56:40.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.625+0000 7f66277fe640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66080664a0 0x7f6608068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:40.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.625+0000 7f66277fe640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66080664a0 0x7f6608068960 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6618001060 tx=0x7f6618009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:40.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.629+0000 7f6624ff9640 1 -- 172.21.15.125:0/194589875 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f66200eb020 con 0x7f66281f1910 2023-10-30T12:56:40.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.833+0000 7f662db0f640 1 -- 172.21.15.125:0/194589875 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66280c7760 con 0x7f66281f1910 2023-10-30T12:56:40.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.833+0000 7f6624ff9640 1 -- 172.21.15.125:0/194589875 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f66200b2090 con 0x7f66281f1910 2023-10-30T12:56:40.837 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:40.838 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:35.422971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:40.838 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 129 2023-10-30T12:56:40.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.837+0000 7f662db0f640 1 -- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66080664a0 msgr2=0x7f6608068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:40.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.837+0000 7f662db0f640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66080664a0 0x7f6608068960 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6618001060 tx=0x7f6618009040 comp rx=0 tx=0).stop 2023-10-30T12:56:40.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.837+0000 7f662db0f640 1 -- 172.21.15.125:0/194589875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66281f1910 msgr2=0x7f66281f1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:40.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.837+0000 7f662db0f640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66281f1910 0x7f66281f1d90 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f66200b0040 tx=0x7f6620009750 comp rx=0 tx=0).stop 2023-10-30T12:56:40.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.837+0000 7f662db0f640 1 -- 172.21.15.125:0/194589875 shutdown_connections 2023-10-30T12:56:40.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.837+0000 7f662db0f640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66080664a0 0x7f6608068960 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:40.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.837+0000 7f662db0f640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66281f1910 0x7f66281f1d90 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:40.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.837+0000 7f662db0f640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f662815d8f0 0x7f66280736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:40.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.837+0000 7f662db0f640 1 --2- 172.21.15.125:0/194589875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6614004d10 0x7f6628154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:40.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.837+0000 7f662db0f640 1 -- 172.21.15.125:0/194589875 >> 172.21.15.125:0/194589875 conn(0x7f662806dfd0 msgr2=0x7f66280c1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:40.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.837+0000 7f662db0f640 1 -- 172.21.15.125:0/194589875 shutdown_connections 2023-10-30T12:56:40.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:40.837+0000 7f662db0f640 1 -- 172.21.15.125:0/194589875 wait complete. 2023-10-30T12:56:40.860 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:56:42.861 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:43.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c4365640 1 Processor -- start 2023-10-30T12:56:43.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c4365640 1 -- start start 2023-10-30T12:56:43.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c4365640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f47bc1538a0 0x7f47bc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:43.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c4365640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f47bc0cf160 0x7f47bc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:43.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c4365640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47bc0cb800 con 0x7f47bc0ca0e0 2023-10-30T12:56:43.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c4365640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47bc15dc30 con 0x7f47bc0cf160 2023-10-30T12:56:43.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c4365640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47bc0cb330 con 0x7f47bc1538a0 2023-10-30T12:56:43.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c2902640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f47bc0cf160 0x7f47bc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:43.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c2902640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f47bc0cf160 0x7f47bc0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45814/0 (socket says 172.21.15.125:45814) 2023-10-30T12:56:43.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c2902640 1 -- 172.21.15.125:0/3953374146 learned_addr learned my addr 172.21.15.125:0/3953374146 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:43.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c2101640 1 --2- 172.21.15.125:0/3953374146 >> v2:172.21.15.136:3300/0 conn(0x7f47bc1538a0 0x7f47bc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:43.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c2902640 1 -- 172.21.15.125:0/3953374146 >> v2:172.21.15.136:3300/0 conn(0x7f47bc1538a0 msgr2=0x7f47bc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:43.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c2902640 1 --2- 172.21.15.125:0/3953374146 >> v2:172.21.15.136:3300/0 conn(0x7f47bc1538a0 0x7f47bc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:43.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c2902640 1 -- 172.21.15.125:0/3953374146 >> v1:172.21.15.136:6789/0 conn(0x7f47bc0ca0e0 legacy=0x7f47bc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:43.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.009+0000 7f47c2902640 1 -- 172.21.15.125:0/3953374146 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47bc0cad70 con 0x7f47bc0cf160 2023-10-30T12:56:43.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c2902640 1 --2- 172.21.15.125:0/3953374146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47bc0cf160 0x7f47bc0cf590 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f47bc0cfd10 tx=0x7f47b401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce364905632bb35 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:43.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c10ff640 1 -- 172.21.15.125:0/3953374146 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b40af400 con 0x7f47bc0cf160 2023-10-30T12:56:43.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c10ff640 1 -- 172.21.15.125:0/3953374146 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b40ab050 con 0x7f47bc0cf160 2023-10-30T12:56:43.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c10ff640 1 -- 172.21.15.125:0/3953374146 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b40af780 con 0x7f47bc0cf160 2023-10-30T12:56:43.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c4365640 1 -- 172.21.15.125:0/3953374146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47bc0cf160 msgr2=0x7f47bc0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:43.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c4365640 1 --2- 172.21.15.125:0/3953374146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47bc0cf160 0x7f47bc0cf590 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f47bc0cfd10 tx=0x7f47b401f1c0 comp rx=0 tx=0).stop 2023-10-30T12:56:43.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c4365640 1 -- 172.21.15.125:0/3953374146 shutdown_connections 2023-10-30T12:56:43.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c4365640 1 --2- 172.21.15.125:0/3953374146 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47bc0cf160 0x7f47bc0cf590 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:43.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c4365640 1 --2- 172.21.15.125:0/3953374146 >> v2:172.21.15.136:3300/0 conn(0x7f47bc1538a0 0x7f47bc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:43.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c4365640 1 -- 172.21.15.125:0/3953374146 >> 172.21.15.125:0/3953374146 conn(0x7f47bc06bfe0 msgr2=0x7f47bc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:43.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c4365640 1 -- 172.21.15.125:0/3953374146 shutdown_connections 2023-10-30T12:56:43.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c4365640 1 -- 172.21.15.125:0/3953374146 wait complete. 2023-10-30T12:56:43.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c4365640 1 Processor -- start 2023-10-30T12:56:43.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c4365640 1 -- start start 2023-10-30T12:56:43.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c4365640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47bc0ca0e0 0x7f47bc1e2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:43.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.017+0000 7f47c4365640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47bc0cf160 0x7f47bc0ce9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:43.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c4365640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47bc1538a0 0x7f47bc0b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:43.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c4365640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc1e3cc0 con 0x7f47bc0cf160 2023-10-30T12:56:43.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c4365640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc0b1bf0 con 0x7f47bc0ca0e0 2023-10-30T12:56:43.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c4365640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47bc1e4880 con 0x7f47bc1538a0 2023-10-30T12:56:43.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c1900640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47bc0cf160 0x7f47bc0ce9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:43.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c1900640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47bc0cf160 0x7f47bc0ce9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45824/0 (socket says 172.21.15.125:45824) 2023-10-30T12:56:43.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c1900640 1 -- 172.21.15.125:0/73413425 learned_addr learned my addr 172.21.15.125:0/73413425 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:43.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c2101640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47bc0ca0e0 0x7f47bc1e2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:43.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c2902640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47bc1538a0 0x7f47bc0b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:43.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c1900640 1 -- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47bc1538a0 msgr2=0x7f47bc0b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:43.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c1900640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47bc1538a0 0x7f47bc0b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:43.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c1900640 1 -- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47bc0ca0e0 msgr2=0x7f47bc1e2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:43.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c1900640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47bc0ca0e0 0x7f47bc1e2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:43.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c1900640 1 -- 172.21.15.125:0/73413425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b4009b30 con 0x7f47bc0cf160 2023-10-30T12:56:43.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c2902640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47bc1538a0 0x7f47bc0b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:56:43.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c1900640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47bc0cf160 0x7f47bc0ce9c0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f47ac00c3c0 tx=0x7f47ac00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:43.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47b2ffd640 1 -- 172.21.15.125:0/73413425 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac0052d0 con 0x7f47bc0cf160 2023-10-30T12:56:43.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47b2ffd640 1 -- 172.21.15.125:0/73413425 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47ac005430 con 0x7f47bc0cf160 2023-10-30T12:56:43.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47b2ffd640 1 -- 172.21.15.125:0/73413425 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47ac003840 con 0x7f47bc0cf160 2023-10-30T12:56:43.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c4365640 1 -- 172.21.15.125:0/73413425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47bc1f3620 con 0x7f47bc0cf160 2023-10-30T12:56:43.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c4365640 1 -- 172.21.15.125:0/73413425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47bc1f3ab0 con 0x7f47bc0cf160 2023-10-30T12:56:43.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.021+0000 7f47c4365640 1 -- 172.21.15.125:0/73413425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47bc0b2a50 con 0x7f47bc0cf160 2023-10-30T12:56:43.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.025+0000 7f47b2ffd640 1 -- 172.21.15.125:0/73413425 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f47ac0039a0 con 0x7f47bc0cf160 2023-10-30T12:56:43.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.025+0000 7f47b2ffd640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f47a00664d0 0x7f47a0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:43.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.025+0000 7f47b2ffd640 1 -- 172.21.15.125:0/73413425 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7f47ac083730 con 0x7f47bc0cf160 2023-10-30T12:56:43.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.025+0000 7f47c2101640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f47a00664d0 0x7f47a0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:43.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.025+0000 7f47c2101640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f47a00664d0 0x7f47a0068990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f47b8006aa0 tx=0x7f47b8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:43.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.033+0000 7f47b2ffd640 1 -- 172.21.15.125:0/73413425 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f47ac050820 con 0x7f47bc0cf160 2023-10-30T12:56:43.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.233+0000 7f47c4365640 1 -- 172.21.15.125:0/73413425 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47bc04e170 con 0x7f47bc0cf160 2023-10-30T12:56:43.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.233+0000 7f47b2ffd640 1 -- 172.21.15.125:0/73413425 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f47ac053e40 con 0x7f47bc0cf160 2023-10-30T12:56:43.238 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:43.238 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:35.422971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:43.239 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 129 2023-10-30T12:56:43.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.237+0000 7f47c4365640 1 -- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f47a00664d0 msgr2=0x7f47a0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:43.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.237+0000 7f47c4365640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f47a00664d0 0x7f47a0068990 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f47b8006aa0 tx=0x7f47b8008040 comp rx=0 tx=0).stop 2023-10-30T12:56:43.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.237+0000 7f47c4365640 1 -- 172.21.15.125:0/73413425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47bc0cf160 msgr2=0x7f47bc0ce9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:43.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.237+0000 7f47c4365640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47bc0cf160 0x7f47bc0ce9c0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f47ac00c3c0 tx=0x7f47ac00c880 comp rx=0 tx=0).stop 2023-10-30T12:56:43.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.237+0000 7f47c4365640 1 -- 172.21.15.125:0/73413425 shutdown_connections 2023-10-30T12:56:43.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.237+0000 7f47c4365640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f47a00664d0 0x7f47a0068990 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:43.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.237+0000 7f47c4365640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f47bc1538a0 0x7f47bc0b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:43.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.237+0000 7f47c4365640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f47bc0cf160 0x7f47bc0ce9c0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:43.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.237+0000 7f47c4365640 1 --2- 172.21.15.125:0/73413425 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f47bc0ca0e0 0x7f47bc1e2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:43.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.237+0000 7f47c4365640 1 -- 172.21.15.125:0/73413425 >> 172.21.15.125:0/73413425 conn(0x7f47bc06bfe0 msgr2=0x7f47bc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:43.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.237+0000 7f47c4365640 1 -- 172.21.15.125:0/73413425 shutdown_connections 2023-10-30T12:56:43.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:43.237+0000 7f47c4365640 1 -- 172.21.15.125:0/73413425 wait complete. 2023-10-30T12:56:43.259 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:56:45.261 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:45.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c87b8640 1 Processor -- start 2023-10-30T12:56:45.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c87b8640 1 -- start start 2023-10-30T12:56:45.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c87b8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f85c015d6f0 0x7f85c015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:45.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c87b8640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c00cb800 con 0x7f85c00cf160 2023-10-30T12:56:45.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c87b8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c00cb330 con 0x7f85c01538a0 2023-10-30T12:56:45.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c87b8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c00cad70 con 0x7f85c015d6f0 2023-10-30T12:56:45.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c6d55640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f85c00cf160 0x7f85c0164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59236/0 (socket says 172.21.15.125:59236) 2023-10-30T12:56:45.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c6d55640 1 -- 172.21.15.125:0/184745107 learned_addr learned my addr 172.21.15.125:0/184745107 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:56:45.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c5d53640 1 --2- 172.21.15.125:0/184745107 >> v2:172.21.15.136:3300/0 conn(0x7f85c015d6f0 0x7f85c015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:45.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c5552640 1 -- 172.21.15.125:0/184745107 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1518228367 0 0) 0x7f85c00cb800 con 0x7f85c00cf160 2023-10-30T12:56:45.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c5552640 1 -- 172.21.15.125:0/184745107 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85ac003530 con 0x7f85c00cf160 2023-10-30T12:56:45.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c5552640 1 -- 172.21.15.125:0/184745107 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2326778890 0 0) 0x7f85ac003530 con 0x7f85c00cf160 2023-10-30T12:56:45.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c5552640 1 -- 172.21.15.125:0/184745107 >> v2:172.21.15.136:3300/0 conn(0x7f85c015d6f0 msgr2=0x7f85c015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:45.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c5552640 1 --2- 172.21.15.125:0/184745107 >> v2:172.21.15.136:3300/0 conn(0x7f85c015d6f0 0x7f85c015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:45.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c5552640 1 -- 172.21.15.125:0/184745107 >> v1:172.21.15.136:6789/0 conn(0x7f85c01538a0 legacy=0x7f85c00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:45.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c5552640 1 -- 172.21.15.125:0/184745107 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c00cb050 con 0x7f85c00cf160 2023-10-30T12:56:45.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c5552640 1 -- 172.21.15.125:0/184745107 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f85b8001680 con 0x7f85c00cf160 2023-10-30T12:56:45.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c5d53640 1 --2- 172.21.15.125:0/184745107 >> v2:172.21.15.136:3300/0 conn(0x7f85c015d6f0 0x7f85c015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:56:45.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c5552640 1 -- 172.21.15.125:0/184745107 >> v1:172.21.15.125:6789/0 conn(0x7f85c00cf160 legacy=0x7f85c0164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:45.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.397+0000 7f85c5552640 1 --2- 172.21.15.125:0/184745107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85ac003980 0x7f85ac003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:45.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c5552640 1 --2- 172.21.15.125:0/184745107 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ac0047d0 0x7f85ac004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:45.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c5552640 1 --2- 172.21.15.125:0/184745107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ac005190 0x7f85ac00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:45.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c5552640 1 -- 172.21.15.125:0/184745107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00cad70 con 0x7f85ac005190 2023-10-30T12:56:45.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c5552640 1 -- 172.21.15.125:0/184745107 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac00c040 con 0x7f85ac003980 2023-10-30T12:56:45.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c5552640 1 -- 172.21.15.125:0/184745107 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85ac00c200 con 0x7f85ac0047d0 2023-10-30T12:56:45.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c6d55640 1 --2- 172.21.15.125:0/184745107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ac005190 0x7f85ac00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:45.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c6554640 1 --2- 172.21.15.125:0/184745107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85ac003980 0x7f85ac003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:45.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c5d53640 1 --2- 172.21.15.125:0/184745107 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ac0047d0 0x7f85ac004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:45.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c6d55640 1 -- 172.21.15.125:0/184745107 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ac0047d0 msgr2=0x7f85ac004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:45.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c6d55640 1 --2- 172.21.15.125:0/184745107 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85ac0047d0 0x7f85ac004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:45.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c6d55640 1 -- 172.21.15.125:0/184745107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85ac003980 msgr2=0x7f85ac003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:45.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c6d55640 1 --2- 172.21.15.125:0/184745107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85ac003980 0x7f85ac003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:45.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c6d55640 1 -- 172.21.15.125:0/184745107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85ac003530 con 0x7f85ac005190 2023-10-30T12:56:45.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c6d55640 1 -- 172.21.15.125:0/184745107 reap_dead start 2023-10-30T12:56:45.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c6d55640 1 --2- 172.21.15.125:0/184745107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ac005190 0x7f85ac00b990 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f85b800c240 tx=0x7f85b801cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86287d5d800e23f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:45.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c5552640 1 -- 172.21.15.125:0/184745107 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b80a5b40 con 0x7f85ac005190 2023-10-30T12:56:45.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c5552640 1 -- 172.21.15.125:0/184745107 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b80a5ce0 con 0x7f85ac005190 2023-10-30T12:56:45.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.401+0000 7f85c5552640 1 -- 172.21.15.125:0/184745107 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b8005b20 con 0x7f85ac005190 2023-10-30T12:56:45.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 -- 172.21.15.125:0/184745107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ac005190 msgr2=0x7f85ac00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:45.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 --2- 172.21.15.125:0/184745107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ac005190 0x7f85ac00b990 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7f85b800c240 tx=0x7f85b801cc40 comp rx=0 tx=0).stop 2023-10-30T12:56:45.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 -- 172.21.15.125:0/184745107 shutdown_connections 2023-10-30T12:56:45.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 --2- 172.21.15.125:0/184745107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ac005190 0x7f85ac00b990 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:45.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 -- 172.21.15.125:0/184745107 >> 172.21.15.125:0/184745107 conn(0x7f85c006bfe0 msgr2=0x7f85c006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:45.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 -- 172.21.15.125:0/184745107 shutdown_connections 2023-10-30T12:56:45.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 -- 172.21.15.125:0/184745107 wait complete. 2023-10-30T12:56:45.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 Processor -- start 2023-10-30T12:56:45.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 -- start start 2023-10-30T12:56:45.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ac005190 0x7f85c00b4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:45.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85c01ef860 0x7f85c01f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:45.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85c01e3030 0x7f85c014fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:45.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c01f3260 con 0x7f85ac005190 2023-10-30T12:56:45.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c00c50d0 con 0x7f85c01ef860 2023-10-30T12:56:45.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c87b8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c01efca0 con 0x7f85c01e3030 2023-10-30T12:56:45.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c6554640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ac005190 0x7f85c00b4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:45.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c6554640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ac005190 0x7f85c00b4130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45850/0 (socket says 172.21.15.125:45850) 2023-10-30T12:56:45.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c6554640 1 -- 172.21.15.125:0/2884254304 learned_addr learned my addr 172.21.15.125:0/2884254304 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:45.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c5d53640 1 --2- 172.21.15.125:0/2884254304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85c01ef860 0x7f85c01f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:45.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c6d55640 1 --2- 172.21.15.125:0/2884254304 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85c01e3030 0x7f85c014fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:45.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c6554640 1 -- 172.21.15.125:0/2884254304 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85c01e3030 msgr2=0x7f85c014fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:45.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c6554640 1 --2- 172.21.15.125:0/2884254304 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85c01e3030 0x7f85c014fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:45.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c6554640 1 -- 172.21.15.125:0/2884254304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85c01ef860 msgr2=0x7f85c01f2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:45.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c6554640 1 --2- 172.21.15.125:0/2884254304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85c01ef860 0x7f85c01f2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:45.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c6554640 1 -- 172.21.15.125:0/2884254304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c01eeda0 con 0x7f85ac005190 2023-10-30T12:56:45.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.405+0000 7f85c6554640 1 --2- 172.21.15.125:0/2884254304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ac005190 0x7f85c00b4130 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f85bc00e920 tx=0x7f85bc00ede0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:45.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.409+0000 7f85b6ffd640 1 -- 172.21.15.125:0/2884254304 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85bc003b90 con 0x7f85ac005190 2023-10-30T12:56:45.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.409+0000 7f85b6ffd640 1 -- 172.21.15.125:0/2884254304 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85bc003cf0 con 0x7f85ac005190 2023-10-30T12:56:45.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.409+0000 7f85b6ffd640 1 -- 172.21.15.125:0/2884254304 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85bc0052f0 con 0x7f85ac005190 2023-10-30T12:56:45.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.409+0000 7f85c87b8640 1 -- 172.21.15.125:0/2884254304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c00c5320 con 0x7f85ac005190 2023-10-30T12:56:45.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.409+0000 7f85c87b8640 1 -- 172.21.15.125:0/2884254304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c00c58e0 con 0x7f85ac005190 2023-10-30T12:56:45.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.409+0000 7f85c87b8640 1 -- 172.21.15.125:0/2884254304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c0159870 con 0x7f85ac005190 2023-10-30T12:56:45.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.409+0000 7f85b6ffd640 1 -- 172.21.15.125:0/2884254304 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f85bc005450 con 0x7f85ac005190 2023-10-30T12:56:45.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.409+0000 7f85b6ffd640 1 --2- 172.21.15.125:0/2884254304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85a40664d0 0x7f85a4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:45.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.409+0000 7f85b6ffd640 1 -- 172.21.15.125:0/2884254304 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7f85bc0844f0 con 0x7f85ac005190 2023-10-30T12:56:45.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.409+0000 7f85c5d53640 1 --2- 172.21.15.125:0/2884254304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85a40664d0 0x7f85a4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:45.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.413+0000 7f85c5d53640 1 --2- 172.21.15.125:0/2884254304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85a40664d0 0x7f85a4068990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f85c00cf040 tx=0x7f85b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:45.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.413+0000 7f85b6ffd640 1 -- 172.21.15.125:0/2884254304 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f85c0159870 con 0x7f85ac005190 2023-10-30T12:56:45.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.597+0000 7f85c87b8640 1 -- 172.21.15.125:0/2884254304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85c0159870 con 0x7f85ac005190 2023-10-30T12:56:45.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.601+0000 7f85b6ffd640 1 -- 172.21.15.125:0/2884254304 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+5720 (secure 0 0 0) 0x7f85c0159870 con 0x7f85ac005190 2023-10-30T12:56:45.602 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:45.603 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:35.422971+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:45.604 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 129 2023-10-30T12:56:45.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.601+0000 7f85c87b8640 1 -- 172.21.15.125:0/2884254304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85a40664d0 msgr2=0x7f85a4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:45.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.601+0000 7f85c87b8640 1 --2- 172.21.15.125:0/2884254304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85a40664d0 0x7f85a4068990 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f85c00cf040 tx=0x7f85b0008040 comp rx=0 tx=0).stop 2023-10-30T12:56:45.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.601+0000 7f85c87b8640 1 -- 172.21.15.125:0/2884254304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ac005190 msgr2=0x7f85c00b4130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:45.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.601+0000 7f85c87b8640 1 --2- 172.21.15.125:0/2884254304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ac005190 0x7f85c00b4130 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f85bc00e920 tx=0x7f85bc00ede0 comp rx=0 tx=0).stop 2023-10-30T12:56:45.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.601+0000 7f85c87b8640 1 -- 172.21.15.125:0/2884254304 shutdown_connections 2023-10-30T12:56:45.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.601+0000 7f85c87b8640 1 --2- 172.21.15.125:0/2884254304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85a40664d0 0x7f85a4068990 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:45.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.601+0000 7f85c87b8640 1 --2- 172.21.15.125:0/2884254304 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85c01e3030 0x7f85c014fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:45.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.601+0000 7f85c87b8640 1 --2- 172.21.15.125:0/2884254304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85c01ef860 0x7f85c01f2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:45.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.601+0000 7f85c87b8640 1 --2- 172.21.15.125:0/2884254304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85ac005190 0x7f85c00b4130 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:45.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.601+0000 7f85c87b8640 1 -- 172.21.15.125:0/2884254304 >> 172.21.15.125:0/2884254304 conn(0x7f85c006bfe0 msgr2=0x7f85c0066090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:45.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.601+0000 7f85c87b8640 1 -- 172.21.15.125:0/2884254304 shutdown_connections 2023-10-30T12:56:45.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:45.601+0000 7f85c87b8640 1 -- 172.21.15.125:0/2884254304 wait complete. 2023-10-30T12:56:45.623 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:56:47.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:47.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.757+0000 7fe41c90e640 1 Processor -- start 2023-10-30T12:56:47.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.761+0000 7fe41c90e640 1 -- start start 2023-10-30T12:56:47.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.761+0000 7fe41c90e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe4181538a0 0x7fe41815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:47.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.761+0000 7fe41c90e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe4180cf160 0x7fe4180cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:47.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.761+0000 7fe41c90e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4180cb800 con 0x7fe4180ca0e0 2023-10-30T12:56:47.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.761+0000 7fe41c90e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe41815dc30 con 0x7fe4181538a0 2023-10-30T12:56:47.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.761+0000 7fe41c90e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4180cb330 con 0x7fe4180cf160 2023-10-30T12:56:47.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.761+0000 7fe41659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe4181538a0 0x7fe41815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:47.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.761+0000 7fe41659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe4181538a0 0x7fe41815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45856/0 (socket says 172.21.15.125:45856) 2023-10-30T12:56:47.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.761+0000 7fe41659c640 1 -- 172.21.15.125:0/743325680 learned_addr learned my addr 172.21.15.125:0/743325680 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:47.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.761+0000 7fe416d9d640 1 --2- 172.21.15.125:0/743325680 >> v2:172.21.15.136:3300/0 conn(0x7fe4180cf160 0x7fe4180cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:47.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.761+0000 7fe41659c640 1 -- 172.21.15.125:0/743325680 >> v2:172.21.15.136:3300/0 conn(0x7fe4180cf160 msgr2=0x7fe4180cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:47.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.761+0000 7fe41659c640 1 --2- 172.21.15.125:0/743325680 >> v2:172.21.15.136:3300/0 conn(0x7fe4180cf160 0x7fe4180cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:47.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.761+0000 7fe41659c640 1 -- 172.21.15.125:0/743325680 >> v1:172.21.15.136:6789/0 conn(0x7fe4180ca0e0 legacy=0x7fe41815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:47.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.761+0000 7fe41659c640 1 -- 172.21.15.125:0/743325680 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4180cad70 con 0x7fe4181538a0 2023-10-30T12:56:47.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.765+0000 7fe41659c640 1 --2- 172.21.15.125:0/743325680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4181538a0 0x7fe41815d6f0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fe4180b5330 tx=0x7fe40801ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5680ea06f9069a5a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:47.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.765+0000 7fe415d9b640 1 -- 172.21.15.125:0/743325680 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4080a89b0 con 0x7fe4181538a0 2023-10-30T12:56:47.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.765+0000 7fe415d9b640 1 -- 172.21.15.125:0/743325680 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4080a8b10 con 0x7fe4181538a0 2023-10-30T12:56:47.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.765+0000 7fe415d9b640 1 -- 172.21.15.125:0/743325680 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4080a8ea0 con 0x7fe4181538a0 2023-10-30T12:56:47.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.765+0000 7fe41c90e640 1 -- 172.21.15.125:0/743325680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4181538a0 msgr2=0x7fe41815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:47.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.765+0000 7fe41c90e640 1 --2- 172.21.15.125:0/743325680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4181538a0 0x7fe41815d6f0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fe4180b5330 tx=0x7fe40801ecb0 comp rx=0 tx=0).stop 2023-10-30T12:56:47.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.765+0000 7fe41c90e640 1 -- 172.21.15.125:0/743325680 shutdown_connections 2023-10-30T12:56:47.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.765+0000 7fe41c90e640 1 --2- 172.21.15.125:0/743325680 >> v2:172.21.15.136:3300/0 conn(0x7fe4180cf160 0x7fe4180cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:47.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.765+0000 7fe41c90e640 1 --2- 172.21.15.125:0/743325680 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4181538a0 0x7fe41815d6f0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:47.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.765+0000 7fe41c90e640 1 -- 172.21.15.125:0/743325680 >> 172.21.15.125:0/743325680 conn(0x7fe41806bfe0 msgr2=0x7fe41806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:47.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.765+0000 7fe41c90e640 1 -- 172.21.15.125:0/743325680 shutdown_connections 2023-10-30T12:56:47.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.765+0000 7fe41c90e640 1 -- 172.21.15.125:0/743325680 wait complete. 2023-10-30T12:56:47.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe41c90e640 1 Processor -- start 2023-10-30T12:56:47.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe41c90e640 1 -- start start 2023-10-30T12:56:47.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe41c90e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe4180ca0e0 0x7fe4180c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:47.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe41c90e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4180cf160 0x7fe418157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:47.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe41c90e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4181538a0 0x7fe4181e5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:47.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe41c90e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4181ef4c0 con 0x7fe4181538a0 2023-10-30T12:56:47.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe41c90e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4180c9180 con 0x7fe4180cf160 2023-10-30T12:56:47.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe41c90e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4181ef730 con 0x7fe4180ca0e0 2023-10-30T12:56:47.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe416d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4181538a0 0x7fe4181e5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:47.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe41659c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe4180ca0e0 0x7fe4180c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:47.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe416d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4181538a0 0x7fe4181e5040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45862/0 (socket says 172.21.15.125:45862) 2023-10-30T12:56:47.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe41659c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe4180ca0e0 0x7fe4180c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:46342/0 (socket says 172.21.15.125:46342) 2023-10-30T12:56:47.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe416d9d640 1 -- 172.21.15.125:0/2526808392 learned_addr learned my addr 172.21.15.125:0/2526808392 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:47.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe40dd9b640 1 --2- 172.21.15.125:0/2526808392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4180cf160 0x7fe418157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:47.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe416d9d640 1 -- 172.21.15.125:0/2526808392 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe4180ca0e0 msgr2=0x7fe4180c64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:47.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe416d9d640 1 --2- 172.21.15.125:0/2526808392 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe4180ca0e0 0x7fe4180c64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:47.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe416d9d640 1 -- 172.21.15.125:0/2526808392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4180cf160 msgr2=0x7fe418157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:47.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe416d9d640 1 --2- 172.21.15.125:0/2526808392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4180cf160 0x7fe418157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:47.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe416d9d640 1 -- 172.21.15.125:0/2526808392 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4080094d0 con 0x7fe4181538a0 2023-10-30T12:56:47.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe416d9d640 1 --2- 172.21.15.125:0/2526808392 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4181538a0 0x7fe4181e5040 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fe410007d90 tx=0x7fe41000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:47.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe40f7fe640 1 -- 172.21.15.125:0/2526808392 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe410014420 con 0x7fe4181538a0 2023-10-30T12:56:47.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe40f7fe640 1 -- 172.21.15.125:0/2526808392 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4100145a0 con 0x7fe4181538a0 2023-10-30T12:56:47.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe41c90e640 1 -- 172.21.15.125:0/2526808392 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4181efa10 con 0x7fe4181538a0 2023-10-30T12:56:47.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe40f7fe640 1 -- 172.21.15.125:0/2526808392 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe410014950 con 0x7fe4181538a0 2023-10-30T12:56:47.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe41c90e640 1 -- 172.21.15.125:0/2526808392 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4180c9460 con 0x7fe4181538a0 2023-10-30T12:56:47.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.769+0000 7fe41c90e640 1 -- 172.21.15.125:0/2526808392 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4180b2a50 con 0x7fe4181538a0 2023-10-30T12:56:47.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.773+0000 7fe40f7fe640 1 -- 172.21.15.125:0/2526808392 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe410014ab0 con 0x7fe4181538a0 2023-10-30T12:56:47.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.773+0000 7fe40f7fe640 1 --2- 172.21.15.125:0/2526808392 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3f80664a0 0x7fe3f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:47.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.773+0000 7fe40f7fe640 1 -- 172.21.15.125:0/2526808392 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7fe410084bf0 con 0x7fe4181538a0 2023-10-30T12:56:47.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.773+0000 7fe41659c640 1 --2- 172.21.15.125:0/2526808392 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3f80664a0 0x7fe3f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:47.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.773+0000 7fe41659c640 1 --2- 172.21.15.125:0/2526808392 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3f80664a0 0x7fe3f8068960 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe4080b2000 tx=0x7fe408009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:47.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.777+0000 7fe40f7fe640 1 -- 172.21.15.125:0/2526808392 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe410051ba0 con 0x7fe4181538a0 2023-10-30T12:56:47.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.957+0000 7fe41c90e640 1 -- 172.21.15.125:0/2526808392 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe41804e170 con 0x7fe4181538a0 2023-10-30T12:56:47.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.957+0000 7fe40f7fe640 1 -- 172.21.15.125:0/2526808392 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+5724 (secure 0 0 0) 0x7fe4100551c0 con 0x7fe4181538a0 2023-10-30T12:56:47.961 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:47.962 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:46.638216+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:47.963 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 130 2023-10-30T12:56:47.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.961+0000 7fe41c90e640 1 -- 172.21.15.125:0/2526808392 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3f80664a0 msgr2=0x7fe3f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:47.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.961+0000 7fe41c90e640 1 --2- 172.21.15.125:0/2526808392 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3f80664a0 0x7fe3f8068960 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe4080b2000 tx=0x7fe408009040 comp rx=0 tx=0).stop 2023-10-30T12:56:47.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.961+0000 7fe41c90e640 1 -- 172.21.15.125:0/2526808392 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4181538a0 msgr2=0x7fe4181e5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:47.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.961+0000 7fe41c90e640 1 --2- 172.21.15.125:0/2526808392 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4181538a0 0x7fe4181e5040 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fe410007d90 tx=0x7fe41000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:56:47.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.961+0000 7fe41c90e640 1 -- 172.21.15.125:0/2526808392 shutdown_connections 2023-10-30T12:56:47.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.961+0000 7fe41c90e640 1 --2- 172.21.15.125:0/2526808392 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe3f80664a0 0x7fe3f8068960 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:47.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.961+0000 7fe41c90e640 1 --2- 172.21.15.125:0/2526808392 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe4181538a0 0x7fe4181e5040 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:47.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.961+0000 7fe41c90e640 1 --2- 172.21.15.125:0/2526808392 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe4180cf160 0x7fe418157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:47.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.961+0000 7fe41c90e640 1 --2- 172.21.15.125:0/2526808392 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe4180ca0e0 0x7fe4180c64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:47.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.961+0000 7fe41c90e640 1 -- 172.21.15.125:0/2526808392 >> 172.21.15.125:0/2526808392 conn(0x7fe41806bfe0 msgr2=0x7fe4181519e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:47.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.961+0000 7fe41c90e640 1 -- 172.21.15.125:0/2526808392 shutdown_connections 2023-10-30T12:56:47.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:47.961+0000 7fe41c90e640 1 -- 172.21.15.125:0/2526808392 wait complete. 2023-10-30T12:56:47.982 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 130, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 130, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7269": { "addr": "172.21.15.125:6836/739564283", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 739564283, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 739564283, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7269, "incarnation": 129, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:reconnect", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:56:46.638216+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7269 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:56:49.598 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:56:49.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:56:49.983 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:50.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.125+0000 7f8b48409640 1 Processor -- start 2023-10-30T12:56:50.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.125+0000 7f8b48409640 1 -- start start 2023-10-30T12:56:50.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b48409640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8b400c6bd0 0x7f8b4006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:50.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b48409640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b40071110 0x7f8b40158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:50.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b48409640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b40066cf0 con 0x7f8b4006f7a0 2023-10-30T12:56:50.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b48409640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b400700c0 con 0x7f8b400c6bd0 2023-10-30T12:56:50.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b48409640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b400c70d0 con 0x7f8b40071110 2023-10-30T12:56:50.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b461a5640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8b4006f7a0 0x7f8b40069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59244/0 (socket says 172.21.15.125:59244) 2023-10-30T12:56:50.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b461a5640 1 -- 172.21.15.125:0/925504141 learned_addr learned my addr 172.21.15.125:0/925504141 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:56:50.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b459a4640 1 --2- 172.21.15.125:0/925504141 >> v2:172.21.15.136:3300/0 conn(0x7f8b400c6bd0 0x7f8b4006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:50.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b469a6640 1 --2- 172.21.15.125:0/925504141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b40071110 0x7f8b40158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:50.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 -- 172.21.15.125:0/925504141 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151589333 0 0) 0x7f8b40066cf0 con 0x7f8b4006f7a0 2023-10-30T12:56:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 -- 172.21.15.125:0/925504141 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b2c003530 con 0x7f8b4006f7a0 2023-10-30T12:56:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 -- 172.21.15.125:0/925504141 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3826696911 0 0) 0x7f8b2c003530 con 0x7f8b4006f7a0 2023-10-30T12:56:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 -- 172.21.15.125:0/925504141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b40071110 msgr2=0x7f8b40158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 --2- 172.21.15.125:0/925504141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b40071110 0x7f8b40158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:50.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 -- 172.21.15.125:0/925504141 >> v2:172.21.15.136:3300/0 conn(0x7f8b400c6bd0 msgr2=0x7f8b4006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:50.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 --2- 172.21.15.125:0/925504141 >> v2:172.21.15.136:3300/0 conn(0x7f8b400c6bd0 0x7f8b4006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:50.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 -- 172.21.15.125:0/925504141 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b400c7390 con 0x7f8b4006f7a0 2023-10-30T12:56:50.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 -- 172.21.15.125:0/925504141 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f8b3c001680 con 0x7f8b4006f7a0 2023-10-30T12:56:50.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 -- 172.21.15.125:0/925504141 >> v1:172.21.15.125:6789/0 conn(0x7f8b4006f7a0 legacy=0x7f8b40069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:50.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 --2- 172.21.15.125:0/925504141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b40071110 0x7f8b2c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:50.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 --2- 172.21.15.125:0/925504141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b2c003f00 0x7f8b2c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:50.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 --2- 172.21.15.125:0/925504141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b2c004d10 0x7f8b2c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:50.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 -- 172.21.15.125:0/925504141 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b400700c0 con 0x7f8b2c004d10 2023-10-30T12:56:50.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 -- 172.21.15.125:0/925504141 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b400c70d0 con 0x7f8b2c003f00 2023-10-30T12:56:50.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b451a3640 1 -- 172.21.15.125:0/925504141 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c00b9c0 con 0x7f8b40071110 2023-10-30T12:56:50.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b469a6640 1 --2- 172.21.15.125:0/925504141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b2c004d10 0x7f8b2c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:50.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b459a4640 1 --2- 172.21.15.125:0/925504141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b2c003f00 0x7f8b2c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:50.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b461a5640 1 --2- 172.21.15.125:0/925504141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b40071110 0x7f8b2c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:50.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b469a6640 1 -- 172.21.15.125:0/925504141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b40071110 msgr2=0x7f8b2c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:50.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b469a6640 1 --2- 172.21.15.125:0/925504141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b40071110 0x7f8b2c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b469a6640 1 -- 172.21.15.125:0/925504141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b2c003f00 msgr2=0x7f8b2c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b469a6640 1 --2- 172.21.15.125:0/925504141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b2c003f00 0x7f8b2c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b469a6640 1 -- 172.21.15.125:0/925504141 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2c003530 con 0x7f8b2c004d10 2023-10-30T12:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b459a4640 1 --2- 172.21.15.125:0/925504141 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b2c003f00 0x7f8b2c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:56:50.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.129+0000 7f8b461a5640 1 --2- 172.21.15.125:0/925504141 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b40071110 0x7f8b2c003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:56:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.133+0000 7f8b469a6640 1 --2- 172.21.15.125:0/925504141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b2c004d10 0x7f8b2c005190 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f8b38009fc0 tx=0x7f8b3801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5416cc95f9e00c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.133+0000 7f8b451a3640 1 -- 172.21.15.125:0/925504141 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b380a5c70 con 0x7f8b2c004d10 2023-10-30T12:56:50.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.133+0000 7f8b451a3640 1 -- 172.21.15.125:0/925504141 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3800f040 con 0x7f8b2c004d10 2023-10-30T12:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.133+0000 7f8b451a3640 1 -- 172.21.15.125:0/925504141 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b380ad440 con 0x7f8b2c004d10 2023-10-30T12:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.133+0000 7f8b48409640 1 -- 172.21.15.125:0/925504141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b2c004d10 msgr2=0x7f8b2c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.133+0000 7f8b48409640 1 --2- 172.21.15.125:0/925504141 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b2c004d10 0x7f8b2c005190 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f8b38009fc0 tx=0x7f8b3801cc60 comp rx=0 tx=0).stop 2023-10-30T12:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.133+0000 7f8b469a6640 1 -- 172.21.15.125:0/925504141 reap_dead start 2023-10-30T12:56:50.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.133+0000 7f8b48409640 1 -- 172.21.15.125:0/925504141 shutdown_connections 2023-10-30T12:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.133+0000 7f8b48409640 1 -- 172.21.15.125:0/925504141 >> 172.21.15.125:0/925504141 conn(0x7f8b400c7910 msgr2=0x7f8b400c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.133+0000 7f8b48409640 1 -- 172.21.15.125:0/925504141 shutdown_connections 2023-10-30T12:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.133+0000 7f8b48409640 1 -- 172.21.15.125:0/925504141 wait complete. 2023-10-30T12:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.133+0000 7f8b48409640 1 Processor -- start 2023-10-30T12:56:50.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b48409640 1 -- start start 2023-10-30T12:56:50.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b48409640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b2c004d10 0x7f8b40147680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:50.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b48409640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b401ddcd0 0x7f8b40147bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:50.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b48409640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b400bfec0 0x7f8b401efeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:50.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b48409640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4014b220 con 0x7f8b401ddcd0 2023-10-30T12:56:50.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b48409640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b400c13a0 con 0x7f8b400bfec0 2023-10-30T12:56:50.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b48409640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4014a5a0 con 0x7f8b2c004d10 2023-10-30T12:56:50.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b459a4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b401ddcd0 0x7f8b40147bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:50.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b459a4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b401ddcd0 0x7f8b40147bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45880/0 (socket says 172.21.15.125:45880) 2023-10-30T12:56:50.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b459a4640 1 -- 172.21.15.125:0/1879296985 learned_addr learned my addr 172.21.15.125:0/1879296985 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:50.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b461a5640 1 --2- 172.21.15.125:0/1879296985 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b2c004d10 0x7f8b40147680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b469a6640 1 --2- 172.21.15.125:0/1879296985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b400bfec0 0x7f8b401efeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b459a4640 1 -- 172.21.15.125:0/1879296985 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b2c004d10 msgr2=0x7f8b40147680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b459a4640 1 --2- 172.21.15.125:0/1879296985 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b2c004d10 0x7f8b40147680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b459a4640 1 -- 172.21.15.125:0/1879296985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b400bfec0 msgr2=0x7f8b401efeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b459a4640 1 --2- 172.21.15.125:0/1879296985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b400bfec0 0x7f8b401efeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:50.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b459a4640 1 -- 172.21.15.125:0/1879296985 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4014a880 con 0x7f8b401ddcd0 2023-10-30T12:56:50.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b459a4640 1 --2- 172.21.15.125:0/1879296985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b401ddcd0 0x7f8b40147bc0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f8b30010870 tx=0x7f8b30010d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:50.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b36ffd640 1 -- 172.21.15.125:0/1879296985 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3000eb60 con 0x7f8b401ddcd0 2023-10-30T12:56:50.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b36ffd640 1 -- 172.21.15.125:0/1879296985 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3000ecc0 con 0x7f8b401ddcd0 2023-10-30T12:56:50.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b48409640 1 -- 172.21.15.125:0/1879296985 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b4014ab60 con 0x7f8b401ddcd0 2023-10-30T12:56:50.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b36ffd640 1 -- 172.21.15.125:0/1879296985 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3001c400 con 0x7f8b401ddcd0 2023-10-30T12:56:50.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b48409640 1 -- 172.21.15.125:0/1879296985 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b4006a5e0 con 0x7f8b401ddcd0 2023-10-30T12:56:50.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.137+0000 7f8b48409640 1 -- 172.21.15.125:0/1879296985 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4004e170 con 0x7f8b401ddcd0 2023-10-30T12:56:50.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.141+0000 7f8b36ffd640 1 -- 172.21.15.125:0/1879296985 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8b3001c560 con 0x7f8b401ddcd0 2023-10-30T12:56:50.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.141+0000 7f8b36ffd640 1 --2- 172.21.15.125:0/1879296985 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b240664a0 0x7f8b24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:50.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.141+0000 7f8b36ffd640 1 -- 172.21.15.125:0/1879296985 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7f8b30084900 con 0x7f8b401ddcd0 2023-10-30T12:56:50.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.141+0000 7f8b461a5640 1 --2- 172.21.15.125:0/1879296985 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b240664a0 0x7f8b24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:50.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.141+0000 7f8b461a5640 1 --2- 172.21.15.125:0/1879296985 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b240664a0 0x7f8b24068960 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c001060 tx=0x7f8b3c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:50.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.145+0000 7f8b36ffd640 1 -- 172.21.15.125:0/1879296985 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8b30051890 con 0x7f8b401ddcd0 2023-10-30T12:56:50.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.325+0000 7f8b48409640 1 -- 172.21.15.125:0/1879296985 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b40071900 con 0x7f8b401ddcd0 2023-10-30T12:56:50.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.325+0000 7f8b36ffd640 1 -- 172.21.15.125:0/1879296985 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 130 v130) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f8b30054eb0 con 0x7f8b401ddcd0 2023-10-30T12:56:50.328 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:50.328 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":130,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":130,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:46.638216+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:50.329 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 130 2023-10-30T12:56:50.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.325+0000 7f8b48409640 1 -- 172.21.15.125:0/1879296985 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b240664a0 msgr2=0x7f8b24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:50.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.325+0000 7f8b48409640 1 --2- 172.21.15.125:0/1879296985 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b240664a0 0x7f8b24068960 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f8b3c001060 tx=0x7f8b3c009040 comp rx=0 tx=0).stop 2023-10-30T12:56:50.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.325+0000 7f8b48409640 1 -- 172.21.15.125:0/1879296985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b401ddcd0 msgr2=0x7f8b40147bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:50.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.325+0000 7f8b48409640 1 --2- 172.21.15.125:0/1879296985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b401ddcd0 0x7f8b40147bc0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f8b30010870 tx=0x7f8b30010d30 comp rx=0 tx=0).stop 2023-10-30T12:56:50.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.325+0000 7f8b48409640 1 -- 172.21.15.125:0/1879296985 shutdown_connections 2023-10-30T12:56:50.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.325+0000 7f8b48409640 1 --2- 172.21.15.125:0/1879296985 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b240664a0 0x7f8b24068960 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:50.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.325+0000 7f8b48409640 1 --2- 172.21.15.125:0/1879296985 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b400bfec0 0x7f8b401efeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:50.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.325+0000 7f8b48409640 1 --2- 172.21.15.125:0/1879296985 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b401ddcd0 0x7f8b40147bc0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:50.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.325+0000 7f8b48409640 1 --2- 172.21.15.125:0/1879296985 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b2c004d10 0x7f8b40147680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:50.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.325+0000 7f8b48409640 1 -- 172.21.15.125:0/1879296985 >> 172.21.15.125:0/1879296985 conn(0x7f8b400c7910 msgr2=0x7f8b400c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:50.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.325+0000 7f8b48409640 1 -- 172.21.15.125:0/1879296985 shutdown_connections 2023-10-30T12:56:50.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:50.325+0000 7f8b48409640 1 -- 172.21.15.125:0/1879296985 wait complete. 2023-10-30T12:56:50.347 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:56:52.349 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:52.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe99ce9c640 1 Processor -- start 2023-10-30T12:56:52.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe99ce9c640 1 -- start start 2023-10-30T12:56:52.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe99ce9c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe9981538a0 0x7fe99815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:52.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe99ce9c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe9980ca0e0 0x7fe99815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:52.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe99ce9c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe9980cf160 0x7fe9980cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:52.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe99ce9c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9980cb800 con 0x7fe9980cf160 2023-10-30T12:56:52.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe99ce9c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe99815dc30 con 0x7fe9980ca0e0 2023-10-30T12:56:52.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe99ce9c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9980cb330 con 0x7fe9981538a0 2023-10-30T12:56:52.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe996d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe9980cf160 0x7fe9980cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:52.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe996d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe9980cf160 0x7fe9980cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47850/0 (socket says 172.21.15.125:47850) 2023-10-30T12:56:52.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe996d9d640 1 -- 172.21.15.125:0/2058780008 learned_addr learned my addr 172.21.15.125:0/2058780008 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:52.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe995d9b640 1 --2- 172.21.15.125:0/2058780008 >> v2:172.21.15.136:3300/0 conn(0x7fe9980ca0e0 0x7fe99815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:52.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe99659c640 1 --2- 172.21.15.125:0/2058780008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe9981538a0 0x7fe99815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:52.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe996d9d640 1 -- 172.21.15.125:0/2058780008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe9981538a0 msgr2=0x7fe99815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:52.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe996d9d640 1 --2- 172.21.15.125:0/2058780008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe9981538a0 0x7fe99815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:52.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe996d9d640 1 -- 172.21.15.125:0/2058780008 >> v2:172.21.15.136:3300/0 conn(0x7fe9980ca0e0 msgr2=0x7fe99815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:52.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe996d9d640 1 --2- 172.21.15.125:0/2058780008 >> v2:172.21.15.136:3300/0 conn(0x7fe9980ca0e0 0x7fe99815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:52.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe996d9d640 1 -- 172.21.15.125:0/2058780008 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9980cad70 con 0x7fe9980cf160 2023-10-30T12:56:52.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.497+0000 7fe99659c640 1 --2- 172.21.15.125:0/2058780008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe9981538a0 0x7fe99815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:56:52.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.501+0000 7fe996d9d640 1 --2- 172.21.15.125:0/2058780008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9980cf160 0x7fe9980cf590 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fe9980ccd10 tx=0x7fe99001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6888a030a178d2fd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:52.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.501+0000 7fe99559a640 1 -- 172.21.15.125:0/2058780008 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9900af400 con 0x7fe9980cf160 2023-10-30T12:56:52.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.501+0000 7fe99559a640 1 -- 172.21.15.125:0/2058780008 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9900ab050 con 0x7fe9980cf160 2023-10-30T12:56:52.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.501+0000 7fe99559a640 1 -- 172.21.15.125:0/2058780008 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9900af780 con 0x7fe9980cf160 2023-10-30T12:56:52.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.501+0000 7fe99ce9c640 1 -- 172.21.15.125:0/2058780008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9980cf160 msgr2=0x7fe9980cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:52.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.501+0000 7fe99ce9c640 1 --2- 172.21.15.125:0/2058780008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9980cf160 0x7fe9980cf590 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fe9980ccd10 tx=0x7fe99001f1c0 comp rx=0 tx=0).stop 2023-10-30T12:56:52.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.501+0000 7fe99ce9c640 1 -- 172.21.15.125:0/2058780008 shutdown_connections 2023-10-30T12:56:52.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.501+0000 7fe99ce9c640 1 --2- 172.21.15.125:0/2058780008 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9980cf160 0x7fe9980cf590 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.501+0000 7fe99ce9c640 1 --2- 172.21.15.125:0/2058780008 >> v2:172.21.15.136:3300/0 conn(0x7fe9980ca0e0 0x7fe99815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.501+0000 7fe99ce9c640 1 --2- 172.21.15.125:0/2058780008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe9981538a0 0x7fe99815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.501+0000 7fe99ce9c640 1 -- 172.21.15.125:0/2058780008 >> 172.21.15.125:0/2058780008 conn(0x7fe99806bfe0 msgr2=0x7fe99806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.501+0000 7fe99ce9c640 1 -- 172.21.15.125:0/2058780008 shutdown_connections 2023-10-30T12:56:52.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.501+0000 7fe99ce9c640 1 -- 172.21.15.125:0/2058780008 wait complete. 2023-10-30T12:56:52.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe99ce9c640 1 Processor -- start 2023-10-30T12:56:52.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe99ce9c640 1 -- start start 2023-10-30T12:56:52.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe99ce9c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9980ca0e0 0x7fe99815d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:52.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe99ce9c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9980cf160 0x7fe9980ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:52.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe99ce9c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe9981538a0 0x7fe99815a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:52.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe99ce9c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe998159280 con 0x7fe9980cf160 2023-10-30T12:56:52.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe99ce9c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe998159560 con 0x7fe9980ca0e0 2023-10-30T12:56:52.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe99ce9c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9981f2430 con 0x7fe9981538a0 2023-10-30T12:56:52.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe995d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9980cf160 0x7fe9980ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:52.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe995d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9980cf160 0x7fe9980ce290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47862/0 (socket says 172.21.15.125:47862) 2023-10-30T12:56:52.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe995d9b640 1 -- 172.21.15.125:0/3640110573 learned_addr learned my addr 172.21.15.125:0/3640110573 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:52.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe99659c640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9980ca0e0 0x7fe99815d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:52.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe996d9d640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe9981538a0 0x7fe99815a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:52.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe995d9b640 1 -- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe9981538a0 msgr2=0x7fe99815a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:52.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe995d9b640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe9981538a0 0x7fe99815a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:52.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe995d9b640 1 -- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9980ca0e0 msgr2=0x7fe99815d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:52.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe995d9b640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9980ca0e0 0x7fe99815d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:52.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe995d9b640 1 -- 172.21.15.125:0/3640110573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe990009b30 con 0x7fe9980cf160 2023-10-30T12:56:52.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe99659c640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9980ca0e0 0x7fe99815d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:56:52.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe995d9b640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9980cf160 0x7fe9980ce290 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fe98c00e8b0 tx=0x7fe98c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:52.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe986ffd640 1 -- 172.21.15.125:0/3640110573 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c00c940 con 0x7fe9980cf160 2023-10-30T12:56:52.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe986ffd640 1 -- 172.21.15.125:0/3640110573 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe98c00caa0 con 0x7fe9980cf160 2023-10-30T12:56:52.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe986ffd640 1 -- 172.21.15.125:0/3640110573 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c00ce30 con 0x7fe9980cf160 2023-10-30T12:56:52.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe99ce9c640 1 -- 172.21.15.125:0/3640110573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9981f2710 con 0x7fe9980cf160 2023-10-30T12:56:52.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.505+0000 7fe99ce9c640 1 -- 172.21.15.125:0/3640110573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9980cee60 con 0x7fe9980cf160 2023-10-30T12:56:52.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.509+0000 7fe99ce9c640 1 -- 172.21.15.125:0/3640110573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9980b2a50 con 0x7fe9980cf160 2023-10-30T12:56:52.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.509+0000 7fe986ffd640 1 -- 172.21.15.125:0/3640110573 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe98c013470 con 0x7fe9980cf160 2023-10-30T12:56:52.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.509+0000 7fe986ffd640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe9780664a0 0x7fe978068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:52.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.509+0000 7fe986ffd640 1 -- 172.21.15.125:0/3640110573 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7fe98c084620 con 0x7fe9980cf160 2023-10-30T12:56:52.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.509+0000 7fe99659c640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe9780664a0 0x7fe978068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:52.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.509+0000 7fe99659c640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe9780664a0 0x7fe978068960 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe98800b990 tx=0x7fe988008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:52.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.513+0000 7fe986ffd640 1 -- 172.21.15.125:0/3640110573 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe98c051530 con 0x7fe9980cf160 2023-10-30T12:56:52.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.693+0000 7fe99ce9c640 1 -- 172.21.15.125:0/3640110573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe9981f2f10 con 0x7fe9980cf160 2023-10-30T12:56:52.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.693+0000 7fe986ffd640 1 -- 172.21.15.125:0/3640110573 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+5721 (secure 0 0 0) 0x7fe9981f2f10 con 0x7fe9980cf160 2023-10-30T12:56:52.698 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:52.698 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:52.466503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:52.699 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 131 2023-10-30T12:56:52.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.697+0000 7fe99ce9c640 1 -- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe9780664a0 msgr2=0x7fe978068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.697+0000 7fe99ce9c640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe9780664a0 0x7fe978068960 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fe98800b990 tx=0x7fe988008040 comp rx=0 tx=0).stop 2023-10-30T12:56:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.697+0000 7fe99ce9c640 1 -- 172.21.15.125:0/3640110573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9980cf160 msgr2=0x7fe9980ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.697+0000 7fe99ce9c640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9980cf160 0x7fe9980ce290 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fe98c00e8b0 tx=0x7fe98c00ed70 comp rx=0 tx=0).stop 2023-10-30T12:56:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.697+0000 7fe99ce9c640 1 -- 172.21.15.125:0/3640110573 shutdown_connections 2023-10-30T12:56:52.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.697+0000 7fe99ce9c640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe9780664a0 0x7fe978068960 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:52.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.697+0000 7fe99ce9c640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe9981538a0 0x7fe99815a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:52.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.697+0000 7fe99ce9c640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe9980cf160 0x7fe9980ce290 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:52.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.697+0000 7fe99ce9c640 1 --2- 172.21.15.125:0/3640110573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe9980ca0e0 0x7fe99815d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:52.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.697+0000 7fe99ce9c640 1 -- 172.21.15.125:0/3640110573 >> 172.21.15.125:0/3640110573 conn(0x7fe99806bfe0 msgr2=0x7fe998150ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:52.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.697+0000 7fe99ce9c640 1 -- 172.21.15.125:0/3640110573 shutdown_connections 2023-10-30T12:56:52.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:52.697+0000 7fe99ce9c640 1 -- 172.21.15.125:0/3640110573 wait complete. 2023-10-30T12:56:52.718 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:56:54.719 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:54.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.869+0000 7fc2a70bb640 1 Processor -- start 2023-10-30T12:56:54.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.869+0000 7fc2a70bb640 1 -- start start 2023-10-30T12:56:54.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.869+0000 7fc2a70bb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a006ab20 0x7fc2a0070970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:54.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.869+0000 7fc2a70bb640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a00c1cc0 con 0x7fc2a006a4d0 2023-10-30T12:56:54.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.869+0000 7fc2a70bb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a0070490 con 0x7fc2a006a0f0 2023-10-30T12:56:54.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.869+0000 7fc2a70bb640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a0166770 con 0x7fc2a006ab20 2023-10-30T12:56:54.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.869+0000 7fc2a58b8640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc2a006a4d0 0x7fc2a0163520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59384/0 (socket says 172.21.15.125:59384) 2023-10-30T12:56:54.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.869+0000 7fc2a58b8640 1 -- 172.21.15.125:0/3383428694 learned_addr learned my addr 172.21.15.125:0/3383428694 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:56:54.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.869+0000 7fc2a68ba640 1 --2- 172.21.15.125:0/3383428694 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a006ab20 0x7fc2a0070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:54.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.869+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353216539 0 0) 0x7fc2a00c1cc0 con 0x7fc2a006a4d0 2023-10-30T12:56:54.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc28c003510 con 0x7fc2a006a4d0 2023-10-30T12:56:54.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732045352 0 0) 0x7fc2a0070490 con 0x7fc2a006a0f0 2023-10-30T12:56:54.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a00c1cc0 con 0x7fc2a006a0f0 2023-10-30T12:56:54.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3833409696 0 0) 0x7fc28c003510 con 0x7fc2a006a4d0 2023-10-30T12:56:54.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a006ab20 msgr2=0x7fc2a0070970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:54.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 --2- 172.21.15.125:0/3383428694 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a006ab20 0x7fc2a0070970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:54.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 >> v1:172.21.15.136:6789/0 conn(0x7fc2a006a0f0 legacy=0x7fc2a016a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:54.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a0166a50 con 0x7fc2a006a4d0 2023-10-30T12:56:54.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fc290001680 con 0x7fc2a006a4d0 2023-10-30T12:56:54.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 >> v1:172.21.15.125:6789/0 conn(0x7fc2a006a4d0 legacy=0x7fc2a0163520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:54.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 --2- 172.21.15.125:0/3383428694 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a006ab20 0x7fc28c0038a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:54.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 --2- 172.21.15.125:0/3383428694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc28c003de0 0x7fc28c004260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:54.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 --2- 172.21.15.125:0/3383428694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc28c004d80 0x7fc28c0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:54.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a0166770 con 0x7fc28c004d80 2023-10-30T12:56:54.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c005a20 con 0x7fc28c003de0 2023-10-30T12:56:54.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c005be0 con 0x7fc2a006ab20 2023-10-30T12:56:54.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a68ba640 1 --2- 172.21.15.125:0/3383428694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc28c004d80 0x7fc28c0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:54.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.953+0000 7fc2a58b8640 1 --2- 172.21.15.125:0/3383428694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc28c003de0 0x7fc28c004260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:54.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.957+0000 7fc2a60b9640 1 --2- 172.21.15.125:0/3383428694 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a006ab20 0x7fc28c0038a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:54.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.957+0000 7fc2a68ba640 1 -- 172.21.15.125:0/3383428694 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a006ab20 msgr2=0x7fc28c0038a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:54.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.957+0000 7fc2a68ba640 1 --2- 172.21.15.125:0/3383428694 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a006ab20 0x7fc28c0038a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:54.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.957+0000 7fc2a68ba640 1 -- 172.21.15.125:0/3383428694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc28c003de0 msgr2=0x7fc28c004260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:54.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.957+0000 7fc2a68ba640 1 --2- 172.21.15.125:0/3383428694 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc28c003de0 0x7fc28c004260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:54.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.957+0000 7fc2a68ba640 1 -- 172.21.15.125:0/3383428694 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c003510 con 0x7fc28c004d80 2023-10-30T12:56:54.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.961+0000 7fc2a68ba640 1 --2- 172.21.15.125:0/3383428694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc28c004d80 0x7fc28c0051e0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fc29800e6b0 tx=0x7fc29801d3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d07eb5731cd13a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:54.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.961+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc290007ea0 con 0x7fc28c004d80 2023-10-30T12:56:54.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.961+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc298007dc0 con 0x7fc28c004d80 2023-10-30T12:56:54.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.961+0000 7fc2a50b7640 1 -- 172.21.15.125:0/3383428694 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29800c040 con 0x7fc28c004d80 2023-10-30T12:56:54.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.961+0000 7fc2a70bb640 1 -- 172.21.15.125:0/3383428694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc28c004d80 msgr2=0x7fc28c0051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:54.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.961+0000 7fc2a70bb640 1 --2- 172.21.15.125:0/3383428694 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc28c004d80 0x7fc28c0051e0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fc29800e6b0 tx=0x7fc29801d3c0 comp rx=0 tx=0).stop 2023-10-30T12:56:54.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.961+0000 7fc2a68ba640 1 -- 172.21.15.125:0/3383428694 reap_dead start 2023-10-30T12:56:54.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.961+0000 7fc2a70bb640 1 -- 172.21.15.125:0/3383428694 shutdown_connections 2023-10-30T12:56:54.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.961+0000 7fc2a70bb640 1 -- 172.21.15.125:0/3383428694 >> 172.21.15.125:0/3383428694 conn(0x7fc2a00c75d0 msgr2=0x7fc2a00c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:54.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.961+0000 7fc2a70bb640 1 -- 172.21.15.125:0/3383428694 shutdown_connections 2023-10-30T12:56:54.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.961+0000 7fc2a70bb640 1 -- 172.21.15.125:0/3383428694 wait complete. 2023-10-30T12:56:54.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a70bb640 1 Processor -- start 2023-10-30T12:56:54.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a70bb640 1 -- start start 2023-10-30T12:56:54.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a70bb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc28c004d80 0x7fc2a00c5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:54.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a70bb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a008b6d0 0x7fc2a01ff260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:54.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a70bb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2a01fb450 0x7fc2a015d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:54.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a70bb640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a015c060 con 0x7fc2a008b6d0 2023-10-30T12:56:54.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a70bb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a0159a00 con 0x7fc2a01fb450 2023-10-30T12:56:54.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a70bb640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a00c6a70 con 0x7fc28c004d80 2023-10-30T12:56:54.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a58b8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a008b6d0 0x7fc2a01ff260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:54.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a58b8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a008b6d0 0x7fc2a01ff260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47882/0 (socket says 172.21.15.125:47882) 2023-10-30T12:56:54.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a58b8640 1 -- 172.21.15.125:0/2774409479 learned_addr learned my addr 172.21.15.125:0/2774409479 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:54.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a60b9640 1 --2- 172.21.15.125:0/2774409479 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc28c004d80 0x7fc2a00c5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:54.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a68ba640 1 --2- 172.21.15.125:0/2774409479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2a01fb450 0x7fc2a015d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:54.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a58b8640 1 -- 172.21.15.125:0/2774409479 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc28c004d80 msgr2=0x7fc2a00c5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:54.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a58b8640 1 --2- 172.21.15.125:0/2774409479 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc28c004d80 0x7fc2a00c5de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:54.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a58b8640 1 -- 172.21.15.125:0/2774409479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2a01fb450 msgr2=0x7fc2a015d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:54.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a58b8640 1 --2- 172.21.15.125:0/2774409479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2a01fb450 0x7fc2a015d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:54.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.965+0000 7fc2a58b8640 1 -- 172.21.15.125:0/2774409479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a00c6d50 con 0x7fc2a008b6d0 2023-10-30T12:56:54.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.969+0000 7fc2a58b8640 1 --2- 172.21.15.125:0/2774409479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a008b6d0 0x7fc2a01ff260 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fc290012510 tx=0x7fc290012910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:54.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.969+0000 7fc296ffd640 1 -- 172.21.15.125:0/2774409479 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc290014750 con 0x7fc2a008b6d0 2023-10-30T12:56:54.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.969+0000 7fc296ffd640 1 -- 172.21.15.125:0/2774409479 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2900148b0 con 0x7fc2a008b6d0 2023-10-30T12:56:54.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.969+0000 7fc2a70bb640 1 -- 172.21.15.125:0/2774409479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a00c7030 con 0x7fc2a008b6d0 2023-10-30T12:56:54.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.969+0000 7fc296ffd640 1 -- 172.21.15.125:0/2774409479 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc290014c60 con 0x7fc2a008b6d0 2023-10-30T12:56:54.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.969+0000 7fc2a70bb640 1 -- 172.21.15.125:0/2774409479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a01fea60 con 0x7fc2a008b6d0 2023-10-30T12:56:54.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.969+0000 7fc2a70bb640 1 -- 172.21.15.125:0/2774409479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a004e170 con 0x7fc2a008b6d0 2023-10-30T12:56:54.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.969+0000 7fc296ffd640 1 -- 172.21.15.125:0/2774409479 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc2900153e0 con 0x7fc2a008b6d0 2023-10-30T12:56:54.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.973+0000 7fc296ffd640 1 --2- 172.21.15.125:0/2774409479 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2840664a0 0x7fc284068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:54.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.973+0000 7fc296ffd640 1 -- 172.21.15.125:0/2774409479 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7fc290086e30 con 0x7fc2a008b6d0 2023-10-30T12:56:54.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.973+0000 7fc2a60b9640 1 --2- 172.21.15.125:0/2774409479 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2840664a0 0x7fc284068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:54.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.973+0000 7fc2a60b9640 1 --2- 172.21.15.125:0/2774409479 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2840664a0 0x7fc284068960 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0018d0 tx=0x7fc29c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:54.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:54.977+0000 7fc296ffd640 1 -- 172.21.15.125:0/2774409479 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc290053e00 con 0x7fc2a008b6d0 2023-10-30T12:56:55.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:55.177+0000 7fc2a70bb640 1 -- 172.21.15.125:0/2774409479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2a00c17b0 con 0x7fc2a008b6d0 2023-10-30T12:56:55.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:55.177+0000 7fc296ffd640 1 -- 172.21.15.125:0/2774409479 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+5721 (secure 0 0 0) 0x7fc29001c070 con 0x7fc2a008b6d0 2023-10-30T12:56:55.181 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:55.182 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:52.466503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:55.182 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 131 2023-10-30T12:56:55.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:55.181+0000 7fc2a70bb640 1 -- 172.21.15.125:0/2774409479 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2840664a0 msgr2=0x7fc284068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:55.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:55.181+0000 7fc2a70bb640 1 --2- 172.21.15.125:0/2774409479 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2840664a0 0x7fc284068960 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fc29c0018d0 tx=0x7fc29c009040 comp rx=0 tx=0).stop 2023-10-30T12:56:55.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:55.181+0000 7fc2a70bb640 1 -- 172.21.15.125:0/2774409479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a008b6d0 msgr2=0x7fc2a01ff260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:55.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:55.181+0000 7fc2a70bb640 1 --2- 172.21.15.125:0/2774409479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a008b6d0 0x7fc2a01ff260 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fc290012510 tx=0x7fc290012910 comp rx=0 tx=0).stop 2023-10-30T12:56:55.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:55.181+0000 7fc2a70bb640 1 -- 172.21.15.125:0/2774409479 shutdown_connections 2023-10-30T12:56:55.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:55.181+0000 7fc2a70bb640 1 --2- 172.21.15.125:0/2774409479 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2840664a0 0x7fc284068960 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:55.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:55.181+0000 7fc2a70bb640 1 --2- 172.21.15.125:0/2774409479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2a01fb450 0x7fc2a015d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:55.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:55.181+0000 7fc2a70bb640 1 --2- 172.21.15.125:0/2774409479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a008b6d0 0x7fc2a01ff260 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:55.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:55.181+0000 7fc2a70bb640 1 --2- 172.21.15.125:0/2774409479 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc28c004d80 0x7fc2a00c5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:55.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:55.181+0000 7fc2a70bb640 1 -- 172.21.15.125:0/2774409479 >> 172.21.15.125:0/2774409479 conn(0x7fc2a00c75d0 msgr2=0x7fc2a01632f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:55.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:55.181+0000 7fc2a70bb640 1 -- 172.21.15.125:0/2774409479 shutdown_connections 2023-10-30T12:56:55.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:55.181+0000 7fc2a70bb640 1 -- 172.21.15.125:0/2774409479 wait complete. 2023-10-30T12:56:55.202 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:56:57.204 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:57.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff51f29c640 1 Processor -- start 2023-10-30T12:56:57.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff51f29c640 1 -- start start 2023-10-30T12:56:57.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff51f29c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff518071110 0x7ff51806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:57.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff51f29c640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff518066cf0 con 0x7ff51806f7a0 2023-10-30T12:56:57.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff51f29c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5180c70d0 con 0x7ff5180c6bd0 2023-10-30T12:56:57.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff51f29c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5180700c0 con 0x7ff518071110 2023-10-30T12:56:57.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff51d038640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff51806f7a0 0x7ff518069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59400/0 (socket says 172.21.15.125:59400) 2023-10-30T12:56:57.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff51d038640 1 -- 172.21.15.125:0/566685767 learned_addr learned my addr 172.21.15.125:0/566685767 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:56:57.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff51d839640 1 --2- 172.21.15.125:0/566685767 >> v2:172.21.15.136:3300/0 conn(0x7ff518071110 0x7ff51806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:57.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 -- 172.21.15.125:0/566685767 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183700260 0 0) 0x7ff518066cf0 con 0x7ff51806f7a0 2023-10-30T12:56:57.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 -- 172.21.15.125:0/566685767 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff504003530 con 0x7ff51806f7a0 2023-10-30T12:56:57.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 -- 172.21.15.125:0/566685767 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 702001556 0 0) 0x7ff504003530 con 0x7ff51806f7a0 2023-10-30T12:56:57.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 -- 172.21.15.125:0/566685767 >> v2:172.21.15.136:3300/0 conn(0x7ff518071110 msgr2=0x7ff51806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:57.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 --2- 172.21.15.125:0/566685767 >> v2:172.21.15.136:3300/0 conn(0x7ff518071110 0x7ff51806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:57.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 -- 172.21.15.125:0/566685767 >> v1:172.21.15.136:6789/0 conn(0x7ff5180c6bd0 legacy=0x7ff518158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:57.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 -- 172.21.15.125:0/566685767 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5180c7390 con 0x7ff51806f7a0 2023-10-30T12:56:57.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 -- 172.21.15.125:0/566685767 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7ff514001680 con 0x7ff51806f7a0 2023-10-30T12:56:57.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 -- 172.21.15.125:0/566685767 >> v1:172.21.15.125:6789/0 conn(0x7ff51806f7a0 legacy=0x7ff518069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:57.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 --2- 172.21.15.125:0/566685767 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff504003980 0x7ff504003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:57.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 --2- 172.21.15.125:0/566685767 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5040047d0 0x7ff504004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:57.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 --2- 172.21.15.125:0/566685767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff504005190 0x7ff50400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:57.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 -- 172.21.15.125:0/566685767 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5180700c0 con 0x7ff504005190 2023-10-30T12:56:57.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 -- 172.21.15.125:0/566685767 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50400c040 con 0x7ff504003980 2023-10-30T12:56:57.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff50ffff640 1 -- 172.21.15.125:0/566685767 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50400c200 con 0x7ff5040047d0 2023-10-30T12:56:57.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff51d839640 1 --2- 172.21.15.125:0/566685767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff504005190 0x7ff50400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:57.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff51c837640 1 --2- 172.21.15.125:0/566685767 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5040047d0 0x7ff504004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:57.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.348+0000 7ff51d038640 1 --2- 172.21.15.125:0/566685767 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff504003980 0x7ff504003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:57.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff51d839640 1 -- 172.21.15.125:0/566685767 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5040047d0 msgr2=0x7ff504004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:57.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff51d839640 1 --2- 172.21.15.125:0/566685767 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5040047d0 0x7ff504004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:57.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff51d839640 1 -- 172.21.15.125:0/566685767 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff504003980 msgr2=0x7ff504003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:57.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff51d839640 1 --2- 172.21.15.125:0/566685767 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff504003980 0x7ff504003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:57.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff51d839640 1 -- 172.21.15.125:0/566685767 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff504003530 con 0x7ff504005190 2023-10-30T12:56:57.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff51d839640 1 -- 172.21.15.125:0/566685767 reap_dead start 2023-10-30T12:56:57.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff51d839640 1 --2- 172.21.15.125:0/566685767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff504005190 0x7ff50400b990 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7ff510009fc0 tx=0x7ff51001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77adb80efd68a457 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:57.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff50ffff640 1 -- 172.21.15.125:0/566685767 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5100a5cb0 con 0x7ff504005190 2023-10-30T12:56:57.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff50ffff640 1 -- 172.21.15.125:0/566685767 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff51000f040 con 0x7ff504005190 2023-10-30T12:56:57.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff50ffff640 1 -- 172.21.15.125:0/566685767 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff510003be0 con 0x7ff504005190 2023-10-30T12:56:57.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff51f29c640 1 -- 172.21.15.125:0/566685767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff504005190 msgr2=0x7ff50400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:57.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff51f29c640 1 --2- 172.21.15.125:0/566685767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff504005190 0x7ff50400b990 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7ff510009fc0 tx=0x7ff51001cca0 comp rx=0 tx=0).stop 2023-10-30T12:56:57.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff51f29c640 1 -- 172.21.15.125:0/566685767 shutdown_connections 2023-10-30T12:56:57.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff51f29c640 1 --2- 172.21.15.125:0/566685767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff504005190 0x7ff50400b990 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:57.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.352+0000 7ff51f29c640 1 -- 172.21.15.125:0/566685767 >> 172.21.15.125:0/566685767 conn(0x7ff5180c7910 msgr2=0x7ff5180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:57.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51f29c640 1 -- 172.21.15.125:0/566685767 shutdown_connections 2023-10-30T12:56:57.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51f29c640 1 -- 172.21.15.125:0/566685767 wait complete. 2023-10-30T12:56:57.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51f29c640 1 Processor -- start 2023-10-30T12:56:57.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51f29c640 1 -- start start 2023-10-30T12:56:57.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51f29c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff504005190 0x7ff51814caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:57.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51f29c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5180afaa0 0x7ff5180afee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:57.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51f29c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5181588a0 0x7ff518158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:57.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51f29c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff518159260 con 0x7ff5181588a0 2023-10-30T12:56:57.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51f29c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5180b0e30 con 0x7ff504005190 2023-10-30T12:56:57.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51f29c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5181f2d10 con 0x7ff5180afaa0 2023-10-30T12:56:57.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51d839640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5181588a0 0x7ff518158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:57.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51d839640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5181588a0 0x7ff518158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47904/0 (socket says 172.21.15.125:47904) 2023-10-30T12:56:57.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51d839640 1 -- 172.21.15.125:0/2939438658 learned_addr learned my addr 172.21.15.125:0/2939438658 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:57.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51c837640 1 --2- 172.21.15.125:0/2939438658 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5180afaa0 0x7ff5180afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:57.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51d038640 1 --2- 172.21.15.125:0/2939438658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff504005190 0x7ff51814caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:57.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51d839640 1 -- 172.21.15.125:0/2939438658 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5180afaa0 msgr2=0x7ff5180afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:57.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51d839640 1 --2- 172.21.15.125:0/2939438658 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5180afaa0 0x7ff5180afee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:57.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51d839640 1 -- 172.21.15.125:0/2939438658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff504005190 msgr2=0x7ff51814caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:57.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51d839640 1 --2- 172.21.15.125:0/2939438658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff504005190 0x7ff51814caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:57.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51d839640 1 -- 172.21.15.125:0/2939438658 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51814d030 con 0x7ff5181588a0 2023-10-30T12:56:57.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.356+0000 7ff51d839640 1 --2- 172.21.15.125:0/2939438658 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5181588a0 0x7ff518158d20 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7ff5100987d0 tx=0x7ff510009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:57.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.360+0000 7ff50dffb640 1 -- 172.21.15.125:0/2939438658 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5100aed40 con 0x7ff5181588a0 2023-10-30T12:56:57.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.360+0000 7ff50dffb640 1 -- 172.21.15.125:0/2939438658 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5100aeec0 con 0x7ff5181588a0 2023-10-30T12:56:57.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.360+0000 7ff50dffb640 1 -- 172.21.15.125:0/2939438658 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff510003be0 con 0x7ff5181588a0 2023-10-30T12:56:57.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.360+0000 7ff51f29c640 1 -- 172.21.15.125:0/2939438658 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5181f1eb0 con 0x7ff5181588a0 2023-10-30T12:56:57.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.360+0000 7ff51f29c640 1 -- 172.21.15.125:0/2939438658 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5181f2470 con 0x7ff5181588a0 2023-10-30T12:56:57.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.360+0000 7ff51f29c640 1 -- 172.21.15.125:0/2939438658 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5180b0420 con 0x7ff5181588a0 2023-10-30T12:56:57.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.360+0000 7ff50dffb640 1 -- 172.21.15.125:0/2939438658 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff510003d40 con 0x7ff5181588a0 2023-10-30T12:56:57.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.360+0000 7ff50dffb640 1 --2- 172.21.15.125:0/2939438658 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4fc0664a0 0x7ff4fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:57.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.360+0000 7ff50dffb640 1 -- 172.21.15.125:0/2939438658 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7ff5100a6e40 con 0x7ff5181588a0 2023-10-30T12:56:57.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.360+0000 7ff51d038640 1 --2- 172.21.15.125:0/2939438658 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4fc0664a0 0x7ff4fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:57.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.364+0000 7ff51d038640 1 --2- 172.21.15.125:0/2939438658 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4fc0664a0 0x7ff4fc068960 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff5140011f0 tx=0x7ff514001180 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:57.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.364+0000 7ff50dffb640 1 -- 172.21.15.125:0/2939438658 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5100ec5b0 con 0x7ff5181588a0 2023-10-30T12:56:57.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.576+0000 7ff51f29c640 1 -- 172.21.15.125:0/2939438658 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff51804e170 con 0x7ff5181588a0 2023-10-30T12:56:57.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.580+0000 7ff50dffb640 1 -- 172.21.15.125:0/2939438658 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+5721 (secure 0 0 0) 0x7ff5100be090 con 0x7ff5181588a0 2023-10-30T12:56:57.583 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:57.583 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:52.466503+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:57.583 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 131 2023-10-30T12:56:57.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.580+0000 7ff51f29c640 1 -- 172.21.15.125:0/2939438658 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4fc0664a0 msgr2=0x7ff4fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:57.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.580+0000 7ff51f29c640 1 --2- 172.21.15.125:0/2939438658 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4fc0664a0 0x7ff4fc068960 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff5140011f0 tx=0x7ff514001180 comp rx=0 tx=0).stop 2023-10-30T12:56:57.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.580+0000 7ff51f29c640 1 -- 172.21.15.125:0/2939438658 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5181588a0 msgr2=0x7ff518158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:57.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.580+0000 7ff51f29c640 1 --2- 172.21.15.125:0/2939438658 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5181588a0 0x7ff518158d20 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7ff5100987d0 tx=0x7ff510009750 comp rx=0 tx=0).stop 2023-10-30T12:56:57.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.580+0000 7ff51f29c640 1 -- 172.21.15.125:0/2939438658 shutdown_connections 2023-10-30T12:56:57.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.580+0000 7ff51f29c640 1 --2- 172.21.15.125:0/2939438658 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4fc0664a0 0x7ff4fc068960 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:57.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.580+0000 7ff51f29c640 1 --2- 172.21.15.125:0/2939438658 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5181588a0 0x7ff518158d20 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:57.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.580+0000 7ff51f29c640 1 --2- 172.21.15.125:0/2939438658 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5180afaa0 0x7ff5180afee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:57.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.580+0000 7ff51f29c640 1 --2- 172.21.15.125:0/2939438658 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff504005190 0x7ff51814caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:57.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.580+0000 7ff51f29c640 1 -- 172.21.15.125:0/2939438658 >> 172.21.15.125:0/2939438658 conn(0x7ff5180c7910 msgr2=0x7ff5180ca540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:57.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.580+0000 7ff51f29c640 1 -- 172.21.15.125:0/2939438658 shutdown_connections 2023-10-30T12:56:57.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:57.580+0000 7ff51f29c640 1 -- 172.21.15.125:0/2939438658 wait complete. 2023-10-30T12:56:57.604 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:56:59.605 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:56:59.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.748+0000 7f0102246640 1 Processor -- start 2023-10-30T12:56:59.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.748+0000 7f0102246640 1 -- start start 2023-10-30T12:56:59.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f0102246640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f00fc0c6b80 0x7f00fc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:59.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f0102246640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00fc0c6f60 0x7f00fc0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:59.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f0102246640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00fc0c8430 0x7f00fc13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:59.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f0102246640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00fc0c0a50 con 0x7f00fc0c6f60 2023-10-30T12:56:59.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f0102246640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00fc06f6e0 con 0x7f00fc0c6b80 2023-10-30T12:56:59.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f0102246640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc06f9c0 con 0x7f00fc0c8430 2023-10-30T12:56:59.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f00f3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00fc0c6f60 0x7f00fc0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:59.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f00f3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00fc0c6f60 0x7f00fc0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47920/0 (socket says 172.21.15.125:47920) 2023-10-30T12:56:59.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f00f3fff640 1 -- 172.21.15.125:0/812091936 learned_addr learned my addr 172.21.15.125:0/812091936 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:59.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f00fb7fe640 1 --2- 172.21.15.125:0/812091936 >> v2:172.21.15.136:3300/0 conn(0x7f00fc0c6b80 0x7f00fc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:59.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f00fbfff640 1 --2- 172.21.15.125:0/812091936 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00fc0c8430 0x7f00fc13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:59.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f00f3fff640 1 -- 172.21.15.125:0/812091936 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00fc0c8430 msgr2=0x7f00fc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:59.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f00f3fff640 1 --2- 172.21.15.125:0/812091936 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00fc0c8430 0x7f00fc13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:59.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f00f3fff640 1 -- 172.21.15.125:0/812091936 >> v2:172.21.15.136:3300/0 conn(0x7f00fc0c6b80 msgr2=0x7f00fc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:56:59.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f00f3fff640 1 --2- 172.21.15.125:0/812091936 >> v2:172.21.15.136:3300/0 conn(0x7f00fc0c6b80 0x7f00fc0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:59.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f00f3fff640 1 -- 172.21.15.125:0/812091936 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00fc06fca0 con 0x7f00fc0c6f60 2023-10-30T12:56:59.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f00f3fff640 1 --2- 172.21.15.125:0/812091936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00fc0c6f60 0x7f00fc0c0400 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f00fc06f1b0 tx=0x7f00e401ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7225ecef6736c910 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:59.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f00faffd640 1 -- 172.21.15.125:0/812091936 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e40a7b70 con 0x7f00fc0c6f60 2023-10-30T12:56:59.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f00faffd640 1 -- 172.21.15.125:0/812091936 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e40a7cd0 con 0x7f00fc0c6f60 2023-10-30T12:56:59.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.752+0000 7f00faffd640 1 -- 172.21.15.125:0/812091936 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e40af660 con 0x7f00fc0c6f60 2023-10-30T12:56:59.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 -- 172.21.15.125:0/812091936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00fc0c6f60 msgr2=0x7f00fc0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:59.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 --2- 172.21.15.125:0/812091936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00fc0c6f60 0x7f00fc0c0400 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f00fc06f1b0 tx=0x7f00e401ebd0 comp rx=0 tx=0).stop 2023-10-30T12:56:59.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 -- 172.21.15.125:0/812091936 shutdown_connections 2023-10-30T12:56:59.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 --2- 172.21.15.125:0/812091936 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00fc0c8430 0x7f00fc13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:59.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 --2- 172.21.15.125:0/812091936 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00fc0c6f60 0x7f00fc0c0400 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:59.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 --2- 172.21.15.125:0/812091936 >> v2:172.21.15.136:3300/0 conn(0x7f00fc0c6b80 0x7f00fc0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:59.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 -- 172.21.15.125:0/812091936 >> 172.21.15.125:0/812091936 conn(0x7f00fc0c7910 msgr2=0x7f00fc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:59.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 -- 172.21.15.125:0/812091936 shutdown_connections 2023-10-30T12:56:59.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 -- 172.21.15.125:0/812091936 wait complete. 2023-10-30T12:56:59.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 Processor -- start 2023-10-30T12:56:59.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 -- start start 2023-10-30T12:56:59.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00fc0c6b80 0x7f00fc15d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:59.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00fc0c6f60 0x7f00fc15b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:59.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00fc0c8430 0x7f00fc15bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:59.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc1f64e0 con 0x7f00fc0c8430 2023-10-30T12:56:59.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc15d560 con 0x7f00fc0c6f60 2023-10-30T12:56:59.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f0102246640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00fc1562f0 con 0x7f00fc0c6b80 2023-10-30T12:56:59.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f00fbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00fc0c8430 0x7f00fc15bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:59.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f00fbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00fc0c8430 0x7f00fc15bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47930/0 (socket says 172.21.15.125:47930) 2023-10-30T12:56:59.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f00fbfff640 1 -- 172.21.15.125:0/376328678 learned_addr learned my addr 172.21.15.125:0/376328678 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:56:59.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f00fb7fe640 1 --2- 172.21.15.125:0/376328678 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00fc0c6b80 0x7f00fc15d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:59.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.756+0000 7f00f3fff640 1 --2- 172.21.15.125:0/376328678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00fc0c6f60 0x7f00fc15b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:59.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f00fbfff640 1 -- 172.21.15.125:0/376328678 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00fc0c6b80 msgr2=0x7f00fc15d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:59.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f00fbfff640 1 --2- 172.21.15.125:0/376328678 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00fc0c6b80 0x7f00fc15d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:59.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f00fbfff640 1 -- 172.21.15.125:0/376328678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00fc0c6f60 msgr2=0x7f00fc15b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:59.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f00fbfff640 1 --2- 172.21.15.125:0/376328678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00fc0c6f60 0x7f00fc15b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:59.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f00fbfff640 1 -- 172.21.15.125:0/376328678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e400b770 con 0x7f00fc0c8430 2023-10-30T12:56:59.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f00fbfff640 1 --2- 172.21.15.125:0/376328678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00fc0c8430 0x7f00fc15bd90 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f00f4007e90 tx=0x7f00f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:59.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f00f8ff9640 1 -- 172.21.15.125:0/376328678 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f4014420 con 0x7f00fc0c8430 2023-10-30T12:56:59.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f00f8ff9640 1 -- 172.21.15.125:0/376328678 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f4014580 con 0x7f00fc0c8430 2023-10-30T12:56:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f0102246640 1 -- 172.21.15.125:0/376328678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00fc13ef10 con 0x7f00fc0c8430 2023-10-30T12:56:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f00f8ff9640 1 -- 172.21.15.125:0/376328678 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f4014930 con 0x7f00fc0c8430 2023-10-30T12:56:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f0102246640 1 -- 172.21.15.125:0/376328678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00fc15b050 con 0x7f00fc0c8430 2023-10-30T12:56:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f0102246640 1 -- 172.21.15.125:0/376328678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00fc15aeb0 con 0x7f00fc0c8430 2023-10-30T12:56:59.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f00f8ff9640 1 -- 172.21.15.125:0/376328678 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f00f4014a90 con 0x7f00fc0c8430 2023-10-30T12:56:59.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f00f8ff9640 1 --2- 172.21.15.125:0/376328678 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00dc0664d0 0x7f00dc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:56:59.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f00f8ff9640 1 -- 172.21.15.125:0/376328678 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7f00f4084c70 con 0x7f00fc0c8430 2023-10-30T12:56:59.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.760+0000 7f00fb7fe640 1 --2- 172.21.15.125:0/376328678 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00dc0664d0 0x7f00dc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:56:59.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.764+0000 7f00fb7fe640 1 --2- 172.21.15.125:0/376328678 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00dc0664d0 0x7f00dc068990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0045f0 tx=0x7f00ec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:56:59.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.764+0000 7f00f8ff9640 1 -- 172.21.15.125:0/376328678 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f00fc15aeb0 con 0x7f00fc0c8430 2023-10-30T12:56:59.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.944+0000 7f0102246640 1 -- 172.21.15.125:0/376328678 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00fc15aeb0 con 0x7f00fc0c8430 2023-10-30T12:56:59.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.944+0000 7f00f8ff9640 1 -- 172.21.15.125:0/376328678 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5721 (secure 0 0 0) 0x7f00fc15aeb0 con 0x7f00fc0c8430 2023-10-30T12:56:59.948 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:56:59.948 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:57.659452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:56:59.949 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 132 2023-10-30T12:56:59.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.944+0000 7f0102246640 1 -- 172.21.15.125:0/376328678 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00dc0664d0 msgr2=0x7f00dc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:59.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.944+0000 7f0102246640 1 --2- 172.21.15.125:0/376328678 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00dc0664d0 0x7f00dc068990 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f00ec0045f0 tx=0x7f00ec008040 comp rx=0 tx=0).stop 2023-10-30T12:56:59.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.944+0000 7f0102246640 1 -- 172.21.15.125:0/376328678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00fc0c8430 msgr2=0x7f00fc15bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:56:59.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.944+0000 7f0102246640 1 --2- 172.21.15.125:0/376328678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00fc0c8430 0x7f00fc15bd90 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f00f4007e90 tx=0x7f00f400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:56:59.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.944+0000 7f0102246640 1 -- 172.21.15.125:0/376328678 shutdown_connections 2023-10-30T12:56:59.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.944+0000 7f0102246640 1 --2- 172.21.15.125:0/376328678 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00dc0664d0 0x7f00dc068990 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:59.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.944+0000 7f0102246640 1 --2- 172.21.15.125:0/376328678 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00fc0c8430 0x7f00fc15bd90 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:59.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.944+0000 7f0102246640 1 --2- 172.21.15.125:0/376328678 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00fc0c6f60 0x7f00fc15b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:59.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.944+0000 7f0102246640 1 --2- 172.21.15.125:0/376328678 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00fc0c6b80 0x7f00fc15d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:56:59.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.944+0000 7f0102246640 1 -- 172.21.15.125:0/376328678 >> 172.21.15.125:0/376328678 conn(0x7f00fc0c7910 msgr2=0x7f00fc06a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:56:59.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.944+0000 7f0102246640 1 -- 172.21.15.125:0/376328678 shutdown_connections 2023-10-30T12:56:59.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:56:59.944+0000 7f0102246640 1 -- 172.21.15.125:0/376328678 wait complete. 2023-10-30T12:56:59.969 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=7113 2023-10-30T12:56:59.969 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.c 2023-10-30T12:57:04.449 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:57:04.444+0000 7f0d883bf640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:57:04.969 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T12:57:04.970 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T12:57:04.970 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T12:57:04.974 INFO:tasks.ceph.mds.c:Started 2023-10-30T12:57:04.974 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:57:05.079 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2023-10-30T12:57:05.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.128+0000 7f0e59794640 1 Processor -- start 2023-10-30T12:57:05.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.128+0000 7f0e59794640 1 -- start start 2023-10-30T12:57:05.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.128+0000 7f0e59794640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0e5415d6f0 0x7f0e5415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:05.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.128+0000 7f0e59794640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0e540cf160 0x7f0e540cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:05.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.128+0000 7f0e59794640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e540cb800 con 0x7f0e541538a0 2023-10-30T12:57:05.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.128+0000 7f0e59794640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e540cb330 con 0x7f0e540cf160 2023-10-30T12:57:05.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.128+0000 7f0e59794640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e540cad70 con 0x7f0e5415d6f0 2023-10-30T12:57:05.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.128+0000 7f0e537fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0e540cf160 0x7f0e540cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:05.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.128+0000 7f0e537fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0e540cf160 0x7f0e540cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57160/0 (socket says 172.21.15.125:57160) 2023-10-30T12:57:05.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.128+0000 7f0e537fe640 1 -- 172.21.15.125:0/929909543 learned_addr learned my addr 172.21.15.125:0/929909543 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:05.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.132+0000 7f0e527fc640 1 --2- 172.21.15.125:0/929909543 >> v2:172.21.15.136:3300/0 conn(0x7f0e5415d6f0 0x7f0e5415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:05.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.132+0000 7f0e537fe640 1 -- 172.21.15.125:0/929909543 >> v2:172.21.15.136:3300/0 conn(0x7f0e5415d6f0 msgr2=0x7f0e5415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:05.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.132+0000 7f0e537fe640 1 --2- 172.21.15.125:0/929909543 >> v2:172.21.15.136:3300/0 conn(0x7f0e5415d6f0 0x7f0e5415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:05.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.132+0000 7f0e537fe640 1 -- 172.21.15.125:0/929909543 >> v1:172.21.15.136:6789/0 conn(0x7f0e541538a0 legacy=0x7f0e540ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:05.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.132+0000 7f0e537fe640 1 -- 172.21.15.125:0/929909543 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e540cb050 con 0x7f0e540cf160 2023-10-30T12:57:05.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.132+0000 7f0e537fe640 1 --2- 172.21.15.125:0/929909543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e540cf160 0x7f0e540cf590 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f0e540b30a0 tx=0x7f0e4c01f180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b97a42f32f0c3b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:05.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.132+0000 7f0e51ffb640 1 -- 172.21.15.125:0/929909543 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0af400 con 0x7f0e540cf160 2023-10-30T12:57:05.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.132+0000 7f0e51ffb640 1 -- 172.21.15.125:0/929909543 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c0ab050 con 0x7f0e540cf160 2023-10-30T12:57:05.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.132+0000 7f0e51ffb640 1 -- 172.21.15.125:0/929909543 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0af780 con 0x7f0e540cf160 2023-10-30T12:57:05.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.132+0000 7f0e59794640 1 -- 172.21.15.125:0/929909543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e540cf160 msgr2=0x7f0e540cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:05.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.132+0000 7f0e59794640 1 --2- 172.21.15.125:0/929909543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e540cf160 0x7f0e540cf590 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f0e540b30a0 tx=0x7f0e4c01f180 comp rx=0 tx=0).stop 2023-10-30T12:57:05.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.136+0000 7f0e59794640 1 -- 172.21.15.125:0/929909543 shutdown_connections 2023-10-30T12:57:05.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.136+0000 7f0e59794640 1 --2- 172.21.15.125:0/929909543 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e540cf160 0x7f0e540cf590 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:05.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.136+0000 7f0e59794640 1 --2- 172.21.15.125:0/929909543 >> v2:172.21.15.136:3300/0 conn(0x7f0e5415d6f0 0x7f0e5415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:05.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.136+0000 7f0e59794640 1 -- 172.21.15.125:0/929909543 >> 172.21.15.125:0/929909543 conn(0x7f0e5406bfe0 msgr2=0x7f0e5406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:05.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.136+0000 7f0e59794640 1 -- 172.21.15.125:0/929909543 shutdown_connections 2023-10-30T12:57:05.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.136+0000 7f0e59794640 1 -- 172.21.15.125:0/929909543 wait complete. 2023-10-30T12:57:05.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.136+0000 7f0e59794640 1 Processor -- start 2023-10-30T12:57:05.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.136+0000 7f0e59794640 1 -- start start 2023-10-30T12:57:05.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e59794640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e540cf160 0x7f0e541ef890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:05.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e59794640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e541538a0 0x7f0e540b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:05.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e59794640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e5415d6f0 0x7f0e541e3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:05.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e59794640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e541e4e10 con 0x7f0e540cf160 2023-10-30T12:57:05.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e59794640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5414d680 con 0x7f0e541538a0 2023-10-30T12:57:05.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e59794640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e541e2a40 con 0x7f0e5415d6f0 2023-10-30T12:57:05.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e52ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e540cf160 0x7f0e541ef890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:05.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e52ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e540cf160 0x7f0e541ef890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57168/0 (socket says 172.21.15.125:57168) 2023-10-30T12:57:05.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e52ffd640 1 -- 172.21.15.125:0/204324022 learned_addr learned my addr 172.21.15.125:0/204324022 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:05.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e537fe640 1 --2- 172.21.15.125:0/204324022 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e5415d6f0 0x7f0e541e3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:05.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e527fc640 1 --2- 172.21.15.125:0/204324022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e541538a0 0x7f0e540b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:05.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e52ffd640 1 -- 172.21.15.125:0/204324022 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e5415d6f0 msgr2=0x7f0e541e3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:05.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e52ffd640 1 --2- 172.21.15.125:0/204324022 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e5415d6f0 0x7f0e541e3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:05.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e52ffd640 1 -- 172.21.15.125:0/204324022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e541538a0 msgr2=0x7f0e540b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:05.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e52ffd640 1 --2- 172.21.15.125:0/204324022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e541538a0 0x7f0e540b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:05.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e52ffd640 1 -- 172.21.15.125:0/204324022 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4c009b30 con 0x7f0e540cf160 2023-10-30T12:57:05.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e52ffd640 1 --2- 172.21.15.125:0/204324022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e540cf160 0x7f0e541ef890 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400c480 tx=0x7f0e4400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:05.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e3b7fe640 1 -- 172.21.15.125:0/204324022 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e44012830 con 0x7f0e540cf160 2023-10-30T12:57:05.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e3b7fe640 1 -- 172.21.15.125:0/204324022 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e440129b0 con 0x7f0e540cf160 2023-10-30T12:57:05.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e3b7fe640 1 -- 172.21.15.125:0/204324022 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e44012d60 con 0x7f0e540cf160 2023-10-30T12:57:05.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e59794640 1 -- 172.21.15.125:0/204324022 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e54158f70 con 0x7f0e540cf160 2023-10-30T12:57:05.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e59794640 1 -- 172.21.15.125:0/204324022 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e541594f0 con 0x7f0e540cf160 2023-10-30T12:57:05.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.140+0000 7f0e59794640 1 -- 172.21.15.125:0/204324022 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e540b2a50 con 0x7f0e540cf160 2023-10-30T12:57:05.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.144+0000 7f0e3b7fe640 1 -- 172.21.15.125:0/204324022 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0e4401c460 con 0x7f0e540cf160 2023-10-30T12:57:05.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.144+0000 7f0e3b7fe640 1 --2- 172.21.15.125:0/204324022 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e400664a0 0x7f0e40068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:05.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.144+0000 7f0e3b7fe640 1 -- 172.21.15.125:0/204324022 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7f0e44084150 con 0x7f0e540cf160 2023-10-30T12:57:05.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.144+0000 7f0e527fc640 1 --2- 172.21.15.125:0/204324022 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e400664a0 0x7f0e40068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:05.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.144+0000 7f0e527fc640 1 --2- 172.21.15.125:0/204324022 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e400664a0 0x7f0e40068960 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0e540c48a0 tx=0x7f0e48008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:05.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.148+0000 7f0e3b7fe640 1 -- 172.21.15.125:0/204324022 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0e44051190 con 0x7f0e540cf160 2023-10-30T12:57:05.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.380+0000 7f0e59794640 1 -- 172.21.15.125:0/204324022 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0e5406c3c0 con 0x7f0e540cf160 2023-10-30T12:57:05.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.380+0000 7f0e3b7fe640 1 -- 172.21.15.125:0/204324022 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+5721 (secure 0 0 0) 0x7f0e440547b0 con 0x7f0e540cf160 2023-10-30T12:57:05.384 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:57:05.384 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:57.659452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:57:05.384 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 132 2023-10-30T12:57:05.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.380+0000 7f0e59794640 1 -- 172.21.15.125:0/204324022 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e400664a0 msgr2=0x7f0e40068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:05.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.380+0000 7f0e59794640 1 --2- 172.21.15.125:0/204324022 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e400664a0 0x7f0e40068960 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0e540c48a0 tx=0x7f0e48008040 comp rx=0 tx=0).stop 2023-10-30T12:57:05.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.380+0000 7f0e59794640 1 -- 172.21.15.125:0/204324022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e540cf160 msgr2=0x7f0e541ef890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:05.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.380+0000 7f0e59794640 1 --2- 172.21.15.125:0/204324022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e540cf160 0x7f0e541ef890 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400c480 tx=0x7f0e4400c940 comp rx=0 tx=0).stop 2023-10-30T12:57:05.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.380+0000 7f0e59794640 1 -- 172.21.15.125:0/204324022 shutdown_connections 2023-10-30T12:57:05.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.380+0000 7f0e59794640 1 --2- 172.21.15.125:0/204324022 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e400664a0 0x7f0e40068960 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:05.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.380+0000 7f0e59794640 1 --2- 172.21.15.125:0/204324022 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0e5415d6f0 0x7f0e541e3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:05.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.380+0000 7f0e59794640 1 --2- 172.21.15.125:0/204324022 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0e541538a0 0x7f0e540b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:05.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.380+0000 7f0e59794640 1 --2- 172.21.15.125:0/204324022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0e540cf160 0x7f0e541ef890 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:05.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.380+0000 7f0e59794640 1 -- 172.21.15.125:0/204324022 >> 172.21.15.125:0/204324022 conn(0x7f0e5406bfe0 msgr2=0x7f0e540c4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:05.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.380+0000 7f0e59794640 1 -- 172.21.15.125:0/204324022 shutdown_connections 2023-10-30T12:57:05.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:05.380+0000 7f0e59794640 1 -- 172.21.15.125:0/204324022 wait complete. 2023-10-30T12:57:05.404 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T12:57:07.405 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:57:07.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.540+0000 7f0bc4017640 1 Processor -- start 2023-10-30T12:57:07.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.540+0000 7f0bc4017640 1 -- start start 2023-10-30T12:57:07.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.540+0000 7f0bc4017640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0bbc0c6b80 0x7f0bbc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:07.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.540+0000 7f0bc4017640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0bbc13f4a0 0x7f0bbc0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:07.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.540+0000 7f0bc4017640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bbc0c0a50 con 0x7f0bbc0c6f60 2023-10-30T12:57:07.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.540+0000 7f0bc4017640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bbc06f6e0 con 0x7f0bbc0c6b80 2023-10-30T12:57:07.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.540+0000 7f0bc4017640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bbc06f9c0 con 0x7f0bbc13f4a0 2023-10-30T12:57:07.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.540+0000 7f0bc1db3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0bbc0c6b80 0x7f0bbc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:07.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.540+0000 7f0bc1db3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0bbc0c6b80 0x7f0bbc0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57180/0 (socket says 172.21.15.125:57180) 2023-10-30T12:57:07.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.540+0000 7f0bc1db3640 1 -- 172.21.15.125:0/2494581981 learned_addr learned my addr 172.21.15.125:0/2494581981 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:07.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.540+0000 7f0bc25b4640 1 --2- 172.21.15.125:0/2494581981 >> v2:172.21.15.136:3300/0 conn(0x7f0bbc13f4a0 0x7f0bbc0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:07.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc1db3640 1 -- 172.21.15.125:0/2494581981 >> v2:172.21.15.136:3300/0 conn(0x7f0bbc13f4a0 msgr2=0x7f0bbc0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:07.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc1db3640 1 --2- 172.21.15.125:0/2494581981 >> v2:172.21.15.136:3300/0 conn(0x7f0bbc13f4a0 0x7f0bbc0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:07.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc1db3640 1 -- 172.21.15.125:0/2494581981 >> v1:172.21.15.136:6789/0 conn(0x7f0bbc0c6f60 legacy=0x7f0bbc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:07.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc1db3640 1 -- 172.21.15.125:0/2494581981 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc06fca0 con 0x7f0bbc0c6b80 2023-10-30T12:57:07.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc1db3640 1 --2- 172.21.15.125:0/2494581981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bbc0c6b80 0x7f0bbc0bfec0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0c8980 tx=0x7f0bb801e6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dcc6dec1959e736 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:07.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc0db1640 1 -- 172.21.15.125:0/2494581981 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb80aa610 con 0x7f0bbc0c6b80 2023-10-30T12:57:07.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc0db1640 1 -- 172.21.15.125:0/2494581981 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb80aa770 con 0x7f0bbc0c6b80 2023-10-30T12:57:07.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc0db1640 1 -- 172.21.15.125:0/2494581981 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb80aab00 con 0x7f0bbc0c6b80 2023-10-30T12:57:07.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc4017640 1 -- 172.21.15.125:0/2494581981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bbc0c6b80 msgr2=0x7f0bbc0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:07.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc4017640 1 --2- 172.21.15.125:0/2494581981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bbc0c6b80 0x7f0bbc0bfec0 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc0c8980 tx=0x7f0bb801e6d0 comp rx=0 tx=0).stop 2023-10-30T12:57:07.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc4017640 1 -- 172.21.15.125:0/2494581981 shutdown_connections 2023-10-30T12:57:07.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc4017640 1 --2- 172.21.15.125:0/2494581981 >> v2:172.21.15.136:3300/0 conn(0x7f0bbc13f4a0 0x7f0bbc0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:07.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc4017640 1 --2- 172.21.15.125:0/2494581981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bbc0c6b80 0x7f0bbc0bfec0 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:07.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc4017640 1 -- 172.21.15.125:0/2494581981 >> 172.21.15.125:0/2494581981 conn(0x7f0bbc0c7910 msgr2=0x7f0bbc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:07.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc4017640 1 -- 172.21.15.125:0/2494581981 shutdown_connections 2023-10-30T12:57:07.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.544+0000 7f0bc4017640 1 -- 172.21.15.125:0/2494581981 wait complete. 2023-10-30T12:57:07.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc4017640 1 Processor -- start 2023-10-30T12:57:07.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc4017640 1 -- start start 2023-10-30T12:57:07.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc4017640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bbc0c6b80 0x7f0bbc1f1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:07.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc4017640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bbc0c6f60 0x7f0bbc0ca310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc4017640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bbc13f4a0 0x7f0bbc13b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc4017640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc0c9e80 con 0x7f0bbc0c6f60 2023-10-30T12:57:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc4017640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc153f80 con 0x7f0bbc13f4a0 2023-10-30T12:57:07.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc4017640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bbc158500 con 0x7f0bbc0c6b80 2023-10-30T12:57:07.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc15b2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bbc0c6f60 0x7f0bbc0ca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc15b2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bbc0c6f60 0x7f0bbc0ca310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57184/0 (socket says 172.21.15.125:57184) 2023-10-30T12:57:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc15b2640 1 -- 172.21.15.125:0/2065444025 learned_addr learned my addr 172.21.15.125:0/2065444025 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc25b4640 1 --2- 172.21.15.125:0/2065444025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bbc13f4a0 0x7f0bbc13b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:07.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc1db3640 1 --2- 172.21.15.125:0/2065444025 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bbc0c6b80 0x7f0bbc1f1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:07.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc15b2640 1 -- 172.21.15.125:0/2065444025 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bbc0c6b80 msgr2=0x7f0bbc1f1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:07.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc15b2640 1 --2- 172.21.15.125:0/2065444025 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bbc0c6b80 0x7f0bbc1f1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc15b2640 1 -- 172.21.15.125:0/2065444025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bbc13f4a0 msgr2=0x7f0bbc13b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc15b2640 1 --2- 172.21.15.125:0/2065444025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bbc13f4a0 0x7f0bbc13b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:07.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc15b2640 1 -- 172.21.15.125:0/2065444025 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb80094d0 con 0x7f0bbc0c6f60 2023-10-30T12:57:07.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc15b2640 1 --2- 172.21.15.125:0/2065444025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bbc0c6f60 0x7f0bbc0ca310 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00c450 tx=0x7f0bac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:07.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bb27fc640 1 -- 172.21.15.125:0/2065444025 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac012830 con 0x7f0bbc0c6f60 2023-10-30T12:57:07.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bb27fc640 1 -- 172.21.15.125:0/2065444025 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac0129b0 con 0x7f0bbc0c6f60 2023-10-30T12:57:07.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc4017640 1 -- 172.21.15.125:0/2065444025 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bbc154c00 con 0x7f0bbc0c6f60 2023-10-30T12:57:07.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bb27fc640 1 -- 172.21.15.125:0/2065444025 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac012d60 con 0x7f0bbc0c6f60 2023-10-30T12:57:07.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.548+0000 7f0bc4017640 1 -- 172.21.15.125:0/2065444025 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bbc1596c0 con 0x7f0bbc0c6f60 2023-10-30T12:57:07.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.552+0000 7f0bc4017640 1 -- 172.21.15.125:0/2065444025 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bbc1540f0 con 0x7f0bbc0c6f60 2023-10-30T12:57:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.552+0000 7f0bb27fc640 1 -- 172.21.15.125:0/2065444025 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0bac01c460 con 0x7f0bbc0c6f60 2023-10-30T12:57:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.552+0000 7f0bb27fc640 1 --2- 172.21.15.125:0/2065444025 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ba00664d0 0x7f0ba0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.552+0000 7f0bb27fc640 1 -- 172.21.15.125:0/2065444025 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7f0bac084110 con 0x7f0bbc0c6f60 2023-10-30T12:57:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.552+0000 7f0bc1db3640 1 --2- 172.21.15.125:0/2065444025 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ba00664d0 0x7f0ba0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.552+0000 7f0bc1db3640 1 --2- 172.21.15.125:0/2065444025 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ba00664d0 0x7f0ba0068990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80b1000 tx=0x7f0bb8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:07.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.556+0000 7f0bb27fc640 1 -- 172.21.15.125:0/2065444025 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0bbc1540f0 con 0x7f0bbc0c6f60 2023-10-30T12:57:07.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.736+0000 7f0bc4017640 1 -- 172.21.15.125:0/2065444025 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bbc1540f0 con 0x7f0bbc0c6f60 2023-10-30T12:57:07.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.740+0000 7f0bb27fc640 1 -- 172.21.15.125:0/2065444025 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6516 (secure 0 0 0) 0x7f0bbc1540f0 con 0x7f0bbc0c6f60 2023-10-30T12:57:07.743 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:57:07.743 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7413,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:57.659452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:57:07.744 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 133 2023-10-30T12:57:07.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.740+0000 7f0bc4017640 1 -- 172.21.15.125:0/2065444025 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ba00664d0 msgr2=0x7f0ba0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:07.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.740+0000 7f0bc4017640 1 --2- 172.21.15.125:0/2065444025 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ba00664d0 0x7f0ba0068990 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f0bb80b1000 tx=0x7f0bb8008040 comp rx=0 tx=0).stop 2023-10-30T12:57:07.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.740+0000 7f0bc4017640 1 -- 172.21.15.125:0/2065444025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bbc0c6f60 msgr2=0x7f0bbc0ca310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:07.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.740+0000 7f0bc4017640 1 --2- 172.21.15.125:0/2065444025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bbc0c6f60 0x7f0bbc0ca310 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00c450 tx=0x7f0bac00c910 comp rx=0 tx=0).stop 2023-10-30T12:57:07.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.740+0000 7f0bc4017640 1 -- 172.21.15.125:0/2065444025 shutdown_connections 2023-10-30T12:57:07.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.740+0000 7f0bc4017640 1 --2- 172.21.15.125:0/2065444025 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ba00664d0 0x7f0ba0068990 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:07.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.740+0000 7f0bc4017640 1 --2- 172.21.15.125:0/2065444025 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bbc13f4a0 0x7f0bbc13b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:07.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.740+0000 7f0bc4017640 1 --2- 172.21.15.125:0/2065444025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bbc0c6f60 0x7f0bbc0ca310 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:07.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.740+0000 7f0bc4017640 1 --2- 172.21.15.125:0/2065444025 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bbc0c6b80 0x7f0bbc1f1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:07.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.740+0000 7f0bc4017640 1 -- 172.21.15.125:0/2065444025 >> 172.21.15.125:0/2065444025 conn(0x7f0bbc0c7910 msgr2=0x7f0bbc0c11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:07.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.740+0000 7f0bc4017640 1 -- 172.21.15.125:0/2065444025 shutdown_connections 2023-10-30T12:57:07.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:07.740+0000 7f0bc4017640 1 -- 172.21.15.125:0/2065444025 wait complete. 2023-10-30T12:57:07.763 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T12:57:07.763 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 29 secs before thrashing 2023-10-30T12:57:12.037 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 22% (59/258) Updating files: 23% (60/258) Updating files: 24% (62/258) Updating files: 25% (65/258) Updating files: 26% (68/258) Updating files: 27% (70/258) Updating files: 28% (73/258) Updating files: 29% (75/258) Updating files: 30% (78/258) Updating files: 31% (80/258) Updating files: 32% (83/258) Updating files: 33% (86/258) Updating files: 34% (88/258) Updating files: 35% (91/258) Updating files: 36% (93/258) Updating files: 37% (96/258) Updating files: 38% (99/258) Updating files: 39% (101/258) Updating files: 40% (104/258) Updating files: 41% (106/258) Updating files: 42% (109/258) Updating files: 43% (111/258) Updating files: 44% (114/258) Updating files: 45% (117/258) Updating files: 46% (119/258) Updating files: 47% (122/258) Updating files: 48% (124/258) Updating files: 49% (127/258) Updating files: 50% (129/258) Updating files: 51% (132/258) Updating files: 52% (135/258) Updating files: 53% (137/258) Updating files: 54% (140/258) Updating files: 55% (142/258) Updating files: 56% (145/258) Updating files: 57% (148/258) Updating files: 58% (150/258) Updating files: 59% (153/258) Updating files: 60% (155/258) Updating files: 61% (158/258) Updating files: 62% (160/258) Updating files: 63% (163/258) Updating files: 64% (166/258) Updating files: 65% (168/258) Updating files: 66% (171/258) Updating files: 67% (173/258) Updating files: 68% (176/258) Updating files: 69% (179/258) Updating files: 70% (181/258) Updating files: 71% (184/258) Updating files: 72% (186/258) Updating files: 73% (189/258) Updating files: 74% (191/258) Updating files: 75% (194/258) Updating files: 76% (197/258) Updating files: 77% (199/258) Updating files: 78% (202/258) Updating files: 79% (204/258) Updating files: 80% (207/258) Updating files: 81% (209/258) Updating files: 82% (212/258) Updating files: 83% (215/258) Updating files: 84% (217/258) Updating files: 85% (220/258) Updating files: 86% (222/258) Updating files: 87% (225/258) Updating files: 88% (228/258) Updating files: 89% (230/258) Updating files: 90% (233/258) Updating files: 91% (235/258) Updating files: 92% (238/258) Updating files: 93% (240/258) Updating files: 94% (243/258) Updating files: 95% (246/258) Updating files: 96% (248/258) Updating files: 97% (251/258) Updating files: 98% (253/258) Updating files: 99% (256/258) Updating files: 100% (258/258) Updating files: 100% (258/258), done. 2023-10-30T12:57:12.615 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 321ecdaba2c v0.39 2023-10-30T12:57:12.619 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.39 2023-10-30T12:57:12.887 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 40 -eq 48 ] 2023-10-30T12:57:12.887 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.40 2023-10-30T12:57:12.887 INFO:tasks.workunit.client.0.smithi125.stdout:v0.40 2023-10-30T12:57:12.888 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.40 2023-10-30T12:57:12.888 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.40 2023-10-30T12:57:18.244 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 32% (73/226) Updating files: 33% (75/226) Updating files: 34% (77/226) Updating files: 35% (80/226) Updating files: 36% (82/226) Updating files: 37% (84/226) Updating files: 38% (86/226) Updating files: 39% (89/226) Updating files: 40% (91/226) Updating files: 41% (93/226) Updating files: 42% (95/226) Updating files: 43% (98/226) Updating files: 44% (100/226) Updating files: 45% (102/226) Updating files: 46% (104/226) Updating files: 47% (107/226) Updating files: 48% (109/226) Updating files: 49% (111/226) Updating files: 50% (113/226) Updating files: 51% (116/226) Updating files: 52% (118/226) Updating files: 53% (120/226) Updating files: 54% (123/226) Updating files: 55% (125/226) Updating files: 56% (127/226) Updating files: 57% (129/226) Updating files: 58% (132/226) Updating files: 59% (134/226) Updating files: 60% (136/226) Updating files: 61% (138/226) Updating files: 62% (141/226) Updating files: 63% (143/226) Updating files: 64% (145/226) Updating files: 65% (147/226) Updating files: 66% (150/226) Updating files: 67% (152/226) Updating files: 68% (154/226) Updating files: 69% (156/226) Updating files: 70% (159/226) Updating files: 71% (161/226) Updating files: 72% (163/226) Updating files: 73% (165/226) Updating files: 74% (168/226) Updating files: 75% (170/226) Updating files: 76% (172/226) Updating files: 77% (175/226) Updating files: 78% (177/226) Updating files: 79% (179/226) Updating files: 80% (181/226) Updating files: 81% (184/226) Updating files: 82% (186/226) Updating files: 83% (188/226) Updating files: 84% (190/226) Updating files: 85% (193/226) Updating files: 86% (195/226) Updating files: 87% (197/226) Updating files: 88% (199/226) Updating files: 89% (202/226) Updating files: 90% (204/226) Updating files: 91% (206/226) Updating files: 92% (208/226) Updating files: 93% (211/226) Updating files: 94% (213/226) Updating files: 95% (215/226) Updating files: 96% (217/226) Updating files: 97% (220/226) Updating files: 98% (222/226) Updating files: 99% (224/226) Updating files: 100% (226/226) Updating files: 100% (226/226), done. 2023-10-30T12:57:18.339 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 7eea40ea37f v0.40 2023-10-30T12:57:18.344 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.40 2023-10-30T12:57:18.422 INFO:tasks.workunit.client.0.smithi125.stdout:v0.41 2023-10-30T12:57:18.423 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 41 -eq 48 ] 2023-10-30T12:57:18.423 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.41 2023-10-30T12:57:18.423 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.41 2023-10-30T12:57:18.424 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.41 2023-10-30T12:57:19.671 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:57:19.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:57:20.248 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 58% (51/87) Updating files: 59% (52/87) Updating files: 60% (53/87) Updating files: 62% (54/87) Updating files: 63% (55/87) Updating files: 64% (56/87) Updating files: 65% (57/87) Updating files: 66% (58/87) Updating files: 67% (59/87) Updating files: 68% (60/87) Updating files: 70% (61/87) Updating files: 71% (62/87) Updating files: 72% (63/87) Updating files: 73% (64/87) Updating files: 74% (65/87) Updating files: 75% (66/87) Updating files: 77% (67/87) Updating files: 78% (68/87) Updating files: 79% (69/87) Updating files: 80% (70/87) Updating files: 81% (71/87) Updating files: 82% (72/87) Updating files: 83% (73/87) Updating files: 85% (74/87) Updating files: 86% (75/87) Updating files: 87% (76/87) Updating files: 88% (77/87) Updating files: 89% (78/87) Updating files: 90% (79/87) Updating files: 91% (80/87) Updating files: 93% (81/87) Updating files: 94% (82/87) Updating files: 95% (83/87) Updating files: 96% (84/87) Updating files: 97% (85/87) Updating files: 98% (86/87) Updating files: 100% (87/87) Updating files: 100% (87/87), done. 2023-10-30T12:57:20.299 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at c1345f7136a v0.41 2023-10-30T12:57:20.302 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.41 2023-10-30T12:57:20.357 INFO:tasks.workunit.client.0.smithi125.stdout:v0.42 2023-10-30T12:57:20.357 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 42 -eq 48 ] 2023-10-30T12:57:20.357 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.42 2023-10-30T12:57:20.358 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.42 2023-10-30T12:57:20.358 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.42 2023-10-30T12:57:29.636 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 37% (94/250) Updating files: 38% (95/250) Updating files: 39% (98/250) Updating files: 40% (100/250) Updating files: 41% (103/250) Updating files: 42% (105/250) Updating files: 43% (108/250) Updating files: 44% (110/250) Updating files: 45% (113/250) Updating files: 46% (115/250) Updating files: 47% (118/250) Updating files: 48% (120/250) Updating files: 49% (123/250) Updating files: 50% (125/250) Updating files: 51% (128/250) Updating files: 52% (130/250) Updating files: 53% (133/250) Updating files: 54% (135/250) Updating files: 55% (138/250) Updating files: 56% (140/250) Updating files: 57% (143/250) Updating files: 57% (144/250) Updating files: 58% (145/250) Updating files: 59% (148/250) Updating files: 60% (150/250) Updating files: 61% (153/250) Updating files: 62% (155/250) Updating files: 63% (158/250) Updating files: 64% (160/250) Updating files: 65% (163/250) Updating files: 66% (165/250) Updating files: 67% (168/250) Updating files: 68% (170/250) Updating files: 69% (173/250) Updating files: 70% (175/250) Updating files: 71% (178/250) Updating files: 72% (180/250) Updating files: 73% (183/250) Updating files: 74% (185/250) Updating files: 75% (188/250) Updating files: 76% (190/250) Updating files: 77% (193/250) Updating files: 78% (195/250) Updating files: 79% (198/250) Updating files: 80% (200/250) Updating files: 81% (203/250) Updating files: 82% (205/250) Updating files: 83% (208/250) Updating files: 84% (210/250) Updating files: 85% (213/250) Updating files: 86% (215/250) Updating files: 87% (218/250) Updating files: 88% (220/250) Updating files: 89% (223/250) Updating files: 90% (225/250) Updating files: 91% (228/250) Updating files: 92% (230/250) Updating files: 92% (231/250) Updating files: 93% (233/250) Updating files: 94% (235/250) Updating files: 95% (238/250) Updating files: 96% (240/250) Updating files: 97% (243/250) Updating files: 98% (245/250) Updating files: 99% (248/250) Updating files: 100% (250/250) Updating files: 100% (250/250), done. 2023-10-30T12:57:29.708 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at b205c64c687 v0.42 2023-10-30T12:57:29.711 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.42 2023-10-30T12:57:29.741 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 43 -eq 48 ] 2023-10-30T12:57:29.742 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.43 2023-10-30T12:57:29.742 INFO:tasks.workunit.client.0.smithi125.stdout:v0.43 2023-10-30T12:57:29.743 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.43 2023-10-30T12:57:29.743 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.43 2023-10-30T12:57:30.612 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 9fa8781c014 v0.43 2023-10-30T12:57:30.614 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.43 2023-10-30T12:57:30.736 INFO:tasks.workunit.client.0.smithi125.stdout:v0.44 2023-10-30T12:57:30.737 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 44 -eq 48 ] 2023-10-30T12:57:30.737 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.44 2023-10-30T12:57:30.737 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.44 2023-10-30T12:57:30.737 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.44 2023-10-30T12:57:35.283 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 21% (40/186) Updating files: 22% (41/186) Updating files: 23% (43/186) Updating files: 24% (45/186) Updating files: 25% (47/186) Updating files: 26% (49/186) Updating files: 27% (51/186) Updating files: 28% (53/186) Updating files: 29% (54/186) Updating files: 30% (56/186) Updating files: 31% (58/186) Updating files: 32% (60/186) Updating files: 33% (62/186) Updating files: 34% (64/186) Updating files: 35% (66/186) Updating files: 36% (67/186) Updating files: 37% (69/186) Updating files: 38% (71/186) Updating files: 39% (73/186) Updating files: 40% (75/186) Updating files: 41% (77/186) Updating files: 41% (78/186) Updating files: 42% (79/186) Updating files: 43% (80/186) Updating files: 44% (82/186) Updating files: 45% (84/186) Updating files: 46% (86/186) Updating files: 47% (88/186) Updating files: 48% (90/186) Updating files: 49% (92/186) Updating files: 50% (93/186) Updating files: 51% (95/186) Updating files: 52% (97/186) Updating files: 53% (99/186) Updating files: 54% (101/186) Updating files: 55% (103/186) Updating files: 56% (105/186) Updating files: 57% (107/186) Updating files: 58% (108/186) Updating files: 59% (110/186) Updating files: 60% (112/186) Updating files: 61% (114/186) Updating files: 62% (116/186) Updating files: 63% (118/186) Updating files: 64% (120/186) Updating files: 65% (121/186) Updating files: 66% (123/186) Updating files: 67% (125/186) Updating files: 68% (127/186) Updating files: 69% (129/186) Updating files: 70% (131/186) Updating files: 71% (133/186) Updating files: 72% (134/186) Updating files: 73% (136/186) Updating files: 74% (138/186) Updating files: 75% (140/186) Updating files: 76% (142/186) Updating files: 77% (144/186) Updating files: 78% (146/186) Updating files: 79% (147/186) Updating files: 80% (149/186) Updating files: 81% (151/186) Updating files: 82% (153/186) Updating files: 83% (155/186) Updating files: 84% (157/186) Updating files: 85% (159/186) Updating files: 86% (160/186) Updating files: 87% (162/186) Updating files: 88% (164/186) Updating files: 89% (166/186) Updating files: 90% (168/186) Updating files: 91% (170/186) Updating files: 92% (172/186) Updating files: 93% (173/186) Updating files: 94% (175/186) Updating files: 95% (177/186) Updating files: 96% (179/186) Updating files: 97% (181/186) Updating files: 98% (183/186) Updating files: 99% (185/186) Updating files: 100% (186/186) Updating files: 100% (186/186), done. 2023-10-30T12:57:35.327 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at cdd5298d276 v0.44 2023-10-30T12:57:35.330 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.44 2023-10-30T12:57:35.617 INFO:tasks.workunit.client.0.smithi125.stdout:v0.45 2023-10-30T12:57:35.617 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 45 -eq 48 ] 2023-10-30T12:57:35.618 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.45 2023-10-30T12:57:35.618 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.45 2023-10-30T12:57:35.618 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.45 2023-10-30T12:57:36.765 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:57:36.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f278cf94640 1 Processor -- start 2023-10-30T12:57:36.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f278cf94640 1 -- start start 2023-10-30T12:57:36.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f278cf94640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f278815d6f0 0x7f278815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:36.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f278cf94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27880cf160 0x7f27880cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:36.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f278cf94640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27880cb800 con 0x7f27881538a0 2023-10-30T12:57:36.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f278cf94640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27880cb330 con 0x7f278815d6f0 2023-10-30T12:57:36.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f278cf94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27880cad70 con 0x7f27880cf160 2023-10-30T12:57:36.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f278659c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f27881538a0 0x7f27880ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:56274/0 (socket says 172.21.15.125:56274) 2023-10-30T12:57:36.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f278659c640 1 -- 172.21.15.125:0/1300530325 learned_addr learned my addr 172.21.15.125:0/1300530325 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:57:36.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f277ffff640 1 --2- 172.21.15.125:0/1300530325 >> v2:172.21.15.136:3300/0 conn(0x7f278815d6f0 0x7f278815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:36.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2786d9d640 1 --2- 172.21.15.125:0/1300530325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27880cf160 0x7f27880cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:36.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 -- 172.21.15.125:0/1300530325 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966686153 0 0) 0x7f27880cb800 con 0x7f27881538a0 2023-10-30T12:57:36.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 -- 172.21.15.125:0/1300530325 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2774003530 con 0x7f27881538a0 2023-10-30T12:57:36.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 -- 172.21.15.125:0/1300530325 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2057125224 0 0) 0x7f2774003530 con 0x7f27881538a0 2023-10-30T12:57:36.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 -- 172.21.15.125:0/1300530325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27880cf160 msgr2=0x7f27880cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:36.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 --2- 172.21.15.125:0/1300530325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27880cf160 0x7f27880cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:36.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 -- 172.21.15.125:0/1300530325 >> v2:172.21.15.136:3300/0 conn(0x7f278815d6f0 msgr2=0x7f278815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:36.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 --2- 172.21.15.125:0/1300530325 >> v2:172.21.15.136:3300/0 conn(0x7f278815d6f0 0x7f278815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:36.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 -- 172.21.15.125:0/1300530325 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27880cb050 con 0x7f27881538a0 2023-10-30T12:57:36.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 -- 172.21.15.125:0/1300530325 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f2778001680 con 0x7f27881538a0 2023-10-30T12:57:36.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 -- 172.21.15.125:0/1300530325 >> v1:172.21.15.125:6789/0 conn(0x7f27881538a0 legacy=0x7f27880ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:36.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 --2- 172.21.15.125:0/1300530325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2774003980 0x7f2774003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:36.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 --2- 172.21.15.125:0/1300530325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27740047d0 0x7f2774004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:36.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 --2- 172.21.15.125:0/1300530325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27880cf160 0x7f2774005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:36.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 -- 172.21.15.125:0/1300530325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27880cb330 con 0x7f27740047d0 2023-10-30T12:57:36.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 -- 172.21.15.125:0/1300530325 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27880cad70 con 0x7f2774003980 2023-10-30T12:57:36.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2785d9b640 1 -- 172.21.15.125:0/1300530325 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f277400b990 con 0x7f27880cf160 2023-10-30T12:57:36.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f277ffff640 1 --2- 172.21.15.125:0/1300530325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27740047d0 0x7f2774004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:36.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f2786d9d640 1 --2- 172.21.15.125:0/1300530325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27880cf160 0x7f2774005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:36.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f278659c640 1 --2- 172.21.15.125:0/1300530325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2774003980 0x7f2774003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:36.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f277ffff640 1 -- 172.21.15.125:0/1300530325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27880cf160 msgr2=0x7f2774005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:36.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f277ffff640 1 --2- 172.21.15.125:0/1300530325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f27880cf160 0x7f2774005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:36.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f277ffff640 1 -- 172.21.15.125:0/1300530325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2774003980 msgr2=0x7f2774003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:36.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f277ffff640 1 --2- 172.21.15.125:0/1300530325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2774003980 0x7f2774003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:36.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.900+0000 7f277ffff640 1 -- 172.21.15.125:0/1300530325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2774003530 con 0x7f27740047d0 2023-10-30T12:57:36.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.904+0000 7f277ffff640 1 --2- 172.21.15.125:0/1300530325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27740047d0 0x7f2774004c50 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f2770009ae0 tx=0x7f277001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c730d49190ed295 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:36.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.904+0000 7f2785d9b640 1 -- 172.21.15.125:0/1300530325 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27700a55a0 con 0x7f27740047d0 2023-10-30T12:57:36.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.904+0000 7f2785d9b640 1 -- 172.21.15.125:0/1300530325 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27700a5700 con 0x7f27740047d0 2023-10-30T12:57:36.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.904+0000 7f2785d9b640 1 -- 172.21.15.125:0/1300530325 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27700a5ab0 con 0x7f27740047d0 2023-10-30T12:57:36.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.904+0000 7f278cf94640 1 -- 172.21.15.125:0/1300530325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27740047d0 msgr2=0x7f2774004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:36.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.904+0000 7f278cf94640 1 --2- 172.21.15.125:0/1300530325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f27740047d0 0x7f2774004c50 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f2770009ae0 tx=0x7f277001ca90 comp rx=0 tx=0).stop 2023-10-30T12:57:36.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.904+0000 7f2786d9d640 1 -- 172.21.15.125:0/1300530325 reap_dead start 2023-10-30T12:57:36.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.904+0000 7f278cf94640 1 -- 172.21.15.125:0/1300530325 shutdown_connections 2023-10-30T12:57:36.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.904+0000 7f278cf94640 1 -- 172.21.15.125:0/1300530325 >> 172.21.15.125:0/1300530325 conn(0x7f278806bfe0 msgr2=0x7f278806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:36.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.904+0000 7f278cf94640 1 -- 172.21.15.125:0/1300530325 shutdown_connections 2023-10-30T12:57:36.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.904+0000 7f278cf94640 1 -- 172.21.15.125:0/1300530325 wait complete. 2023-10-30T12:57:36.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f278cf94640 1 Processor -- start 2023-10-30T12:57:36.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f278cf94640 1 -- start start 2023-10-30T12:57:36.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f278cf94640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27740047d0 0x7f278815bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:36.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f278cf94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f278815c260 0x7f2788158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:36.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f278cf94640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2788159060 0x7f27881f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:36.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f278cf94640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27880cd0b0 con 0x7f2788159060 2023-10-30T12:57:36.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f278cf94640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f278815b2c0 con 0x7f27740047d0 2023-10-30T12:57:36.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f278cf94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27881efd80 con 0x7f278815c260 2023-10-30T12:57:36.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f2786d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2788159060 0x7f27881f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:36.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f2786d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2788159060 0x7f27881f2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47024/0 (socket says 172.21.15.125:47024) 2023-10-30T12:57:36.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f2786d9d640 1 -- 172.21.15.125:0/4202193914 learned_addr learned my addr 172.21.15.125:0/4202193914 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:36.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f277ffff640 1 --2- 172.21.15.125:0/4202193914 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f278815c260 0x7f2788158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:36.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f278659c640 1 --2- 172.21.15.125:0/4202193914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27740047d0 0x7f278815bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:36.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f2786d9d640 1 -- 172.21.15.125:0/4202193914 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f278815c260 msgr2=0x7f2788158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:36.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f2786d9d640 1 --2- 172.21.15.125:0/4202193914 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f278815c260 0x7f2788158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:36.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f2786d9d640 1 -- 172.21.15.125:0/4202193914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27740047d0 msgr2=0x7f278815bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:36.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f2786d9d640 1 --2- 172.21.15.125:0/4202193914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27740047d0 0x7f278815bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:36.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f2786d9d640 1 -- 172.21.15.125:0/4202193914 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278815a920 con 0x7f2788159060 2023-10-30T12:57:36.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f2786d9d640 1 --2- 172.21.15.125:0/4202193914 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2788159060 0x7f27881f2380 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f2788072ce0 tx=0x7f27800115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:36.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f277effd640 1 -- 172.21.15.125:0/4202193914 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278000e070 con 0x7f2788159060 2023-10-30T12:57:36.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f277effd640 1 -- 172.21.15.125:0/4202193914 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2780011e40 con 0x7f2788159060 2023-10-30T12:57:36.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f278cf94640 1 -- 172.21.15.125:0/4202193914 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278815c6a0 con 0x7f2788159060 2023-10-30T12:57:36.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f277effd640 1 -- 172.21.15.125:0/4202193914 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27800039a0 con 0x7f2788159060 2023-10-30T12:57:36.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.908+0000 7f278cf94640 1 -- 172.21.15.125:0/4202193914 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f278815b510 con 0x7f2788159060 2023-10-30T12:57:36.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.912+0000 7f278cf94640 1 -- 172.21.15.125:0/4202193914 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f278815b700 con 0x7f2788159060 2023-10-30T12:57:36.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.912+0000 7f277effd640 1 -- 172.21.15.125:0/4202193914 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2780003b20 con 0x7f2788159060 2023-10-30T12:57:36.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.912+0000 7f277effd640 1 --2- 172.21.15.125:0/4202193914 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f27680664d0 0x7f2768068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:36.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.912+0000 7f277effd640 1 -- 172.21.15.125:0/4202193914 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7f2780084af0 con 0x7f2788159060 2023-10-30T12:57:36.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.912+0000 7f278659c640 1 --2- 172.21.15.125:0/4202193914 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f27680664d0 0x7f2768068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:36.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.912+0000 7f278659c640 1 --2- 172.21.15.125:0/4202193914 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f27680664d0 0x7f2768068990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f2778005a40 tx=0x7f2778009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:36.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:36.916+0000 7f277effd640 1 -- 172.21.15.125:0/4202193914 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2780051a00 con 0x7f2788159060 2023-10-30T12:57:37.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.096+0000 7f278cf94640 1 -- 172.21.15.125:0/4202193914 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f278804e170 con 0x7f2788159060 2023-10-30T12:57:37.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.100+0000 7f277effd640 1 -- 172.21.15.125:0/4202193914 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6516 (secure 0 0 0) 0x7f2780055020 con 0x7f2788159060 2023-10-30T12:57:37.103 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:57:37.103 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7413,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:57.659452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:57:37.103 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 133 2023-10-30T12:57:37.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.100+0000 7f278cf94640 1 -- 172.21.15.125:0/4202193914 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f27680664d0 msgr2=0x7f2768068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:37.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.100+0000 7f278cf94640 1 --2- 172.21.15.125:0/4202193914 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f27680664d0 0x7f2768068990 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f2778005a40 tx=0x7f2778009040 comp rx=0 tx=0).stop 2023-10-30T12:57:37.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.100+0000 7f278cf94640 1 -- 172.21.15.125:0/4202193914 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2788159060 msgr2=0x7f27881f2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:37.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.100+0000 7f278cf94640 1 --2- 172.21.15.125:0/4202193914 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2788159060 0x7f27881f2380 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f2788072ce0 tx=0x7f27800115a0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.100+0000 7f278cf94640 1 -- 172.21.15.125:0/4202193914 shutdown_connections 2023-10-30T12:57:37.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.100+0000 7f278cf94640 1 --2- 172.21.15.125:0/4202193914 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f27680664d0 0x7f2768068990 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.100+0000 7f278cf94640 1 --2- 172.21.15.125:0/4202193914 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2788159060 0x7f27881f2380 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.100+0000 7f278cf94640 1 --2- 172.21.15.125:0/4202193914 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f278815c260 0x7f2788158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.100+0000 7f278cf94640 1 --2- 172.21.15.125:0/4202193914 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f27740047d0 0x7f278815bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.100+0000 7f278cf94640 1 -- 172.21.15.125:0/4202193914 >> 172.21.15.125:0/4202193914 conn(0x7f278806bfe0 msgr2=0x7f2788151870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:37.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.100+0000 7f278cf94640 1 -- 172.21.15.125:0/4202193914 shutdown_connections 2023-10-30T12:57:37.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.100+0000 7f278cf94640 1 -- 172.21.15.125:0/4202193914 wait complete. 2023-10-30T12:57:37.123 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T12:57:37.123 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T12:57:37.124 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:57:37.152 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T12:57:37.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:57:37.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b71846640 1 Processor -- start 2023-10-30T12:57:37.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b71846640 1 -- start start 2023-10-30T12:57:37.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b71846640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0b6c15d6f0 0x7f0b6c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:37.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b71846640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6c0cb800 con 0x7f0b6c1538a0 2023-10-30T12:57:37.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b71846640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b6c0cb330 con 0x7f0b6c0cf160 2023-10-30T12:57:37.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b71846640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b6c0cad70 con 0x7f0b6c15d6f0 2023-10-30T12:57:37.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b6affd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0b6c1538a0 0x7f0b6c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:56288/0 (socket says 172.21.15.125:56288) 2023-10-30T12:57:37.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b6affd640 1 -- 172.21.15.125:0/1537606709 learned_addr learned my addr 172.21.15.125:0/1537606709 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:57:37.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b6a7fc640 1 --2- 172.21.15.125:0/1537606709 >> v2:172.21.15.136:3300/0 conn(0x7f0b6c15d6f0 0x7f0b6c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:37.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 -- 172.21.15.125:0/1537606709 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2973531643 0 0) 0x7f0b6c0cb800 con 0x7f0b6c1538a0 2023-10-30T12:57:37.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 -- 172.21.15.125:0/1537606709 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b54003530 con 0x7f0b6c1538a0 2023-10-30T12:57:37.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 -- 172.21.15.125:0/1537606709 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3285853043 0 0) 0x7f0b54003530 con 0x7f0b6c1538a0 2023-10-30T12:57:37.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 -- 172.21.15.125:0/1537606709 >> v2:172.21.15.136:3300/0 conn(0x7f0b6c15d6f0 msgr2=0x7f0b6c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:37.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 --2- 172.21.15.125:0/1537606709 >> v2:172.21.15.136:3300/0 conn(0x7f0b6c15d6f0 0x7f0b6c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 -- 172.21.15.125:0/1537606709 >> v1:172.21.15.136:6789/0 conn(0x7f0b6c0cf160 legacy=0x7f0b6c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:37.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 -- 172.21.15.125:0/1537606709 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c0cb050 con 0x7f0b6c1538a0 2023-10-30T12:57:37.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 -- 172.21.15.125:0/1537606709 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f0b5c001680 con 0x7f0b6c1538a0 2023-10-30T12:57:37.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 -- 172.21.15.125:0/1537606709 >> v1:172.21.15.125:6789/0 conn(0x7f0b6c1538a0 legacy=0x7f0b6c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:37.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 --2- 172.21.15.125:0/1537606709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b54003980 0x7f0b54003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:37.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 --2- 172.21.15.125:0/1537606709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b540047d0 0x7f0b54004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:37.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 --2- 172.21.15.125:0/1537606709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b54005190 0x7f0b5400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:37.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 -- 172.21.15.125:0/1537606709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c0cad70 con 0x7f0b54003980 2023-10-30T12:57:37.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 -- 172.21.15.125:0/1537606709 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5400c040 con 0x7f0b54005190 2023-10-30T12:57:37.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b69ffb640 1 -- 172.21.15.125:0/1537606709 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5400c200 con 0x7f0b540047d0 2023-10-30T12:57:37.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b6affd640 1 --2- 172.21.15.125:0/1537606709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b54003980 0x7f0b54003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:37.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b6b7fe640 1 --2- 172.21.15.125:0/1537606709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b54005190 0x7f0b5400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:37.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.328+0000 7f0b6a7fc640 1 --2- 172.21.15.125:0/1537606709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b540047d0 0x7f0b54004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:37.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b6affd640 1 -- 172.21.15.125:0/1537606709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b540047d0 msgr2=0x7f0b54004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:37.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b6affd640 1 --2- 172.21.15.125:0/1537606709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b540047d0 0x7f0b54004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b6affd640 1 -- 172.21.15.125:0/1537606709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b54005190 msgr2=0x7f0b5400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:37.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b6affd640 1 --2- 172.21.15.125:0/1537606709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b54005190 0x7f0b5400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b6affd640 1 -- 172.21.15.125:0/1537606709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b54003530 con 0x7f0b54003980 2023-10-30T12:57:37.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b6b7fe640 1 -- 172.21.15.125:0/1537606709 reap_dead start 2023-10-30T12:57:37.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b6affd640 1 --2- 172.21.15.125:0/1537606709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b54003980 0x7f0b54003da0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c00c240 tx=0x7f0b5c01f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d973f56f3223427 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:37.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b69ffb640 1 -- 172.21.15.125:0/1537606709 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c005ae0 con 0x7f0b54003980 2023-10-30T12:57:37.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b69ffb640 1 -- 172.21.15.125:0/1537606709 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5c0ab070 con 0x7f0b54003980 2023-10-30T12:57:37.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b69ffb640 1 -- 172.21.15.125:0/1537606709 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c007070 con 0x7f0b54003980 2023-10-30T12:57:37.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b71846640 1 -- 172.21.15.125:0/1537606709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b54003980 msgr2=0x7f0b54003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:37.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b71846640 1 --2- 172.21.15.125:0/1537606709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b54003980 0x7f0b54003da0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c00c240 tx=0x7f0b5c01f3b0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b71846640 1 -- 172.21.15.125:0/1537606709 shutdown_connections 2023-10-30T12:57:37.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b71846640 1 --2- 172.21.15.125:0/1537606709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b54003980 0x7f0b54003da0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b71846640 1 -- 172.21.15.125:0/1537606709 >> 172.21.15.125:0/1537606709 conn(0x7f0b6c06bfe0 msgr2=0x7f0b6c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:37.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b71846640 1 -- 172.21.15.125:0/1537606709 shutdown_connections 2023-10-30T12:57:37.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.332+0000 7f0b71846640 1 -- 172.21.15.125:0/1537606709 wait complete. 2023-10-30T12:57:37.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b71846640 1 Processor -- start 2023-10-30T12:57:37.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b71846640 1 -- start start 2023-10-30T12:57:37.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b71846640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b54003980 0x7f0b6c1f0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:37.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b71846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b6c1f01b0 0x7f0b6c0b4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:37.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b71846640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b6c158790 0x7f0b6c1f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:37.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b71846640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c1e55b0 con 0x7f0b6c158790 2023-10-30T12:57:37.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b71846640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c1ef4c0 con 0x7f0b54003980 2023-10-30T12:57:37.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b71846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6c1f3400 con 0x7f0b6c1f01b0 2023-10-30T12:57:37.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b6b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b6c158790 0x7f0b6c1f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:37.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b6b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b6c158790 0x7f0b6c1f2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47042/0 (socket says 172.21.15.125:47042) 2023-10-30T12:57:37.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b6b7fe640 1 -- 172.21.15.125:0/4214955923 learned_addr learned my addr 172.21.15.125:0/4214955923 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:37.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b6affd640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b54003980 0x7f0b6c1f0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:37.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b6a7fc640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b6c1f01b0 0x7f0b6c0b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:37.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b6b7fe640 1 -- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b6c1f01b0 msgr2=0x7f0b6c0b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:37.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b6b7fe640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b6c1f01b0 0x7f0b6c0b4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b6b7fe640 1 -- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b54003980 msgr2=0x7f0b6c1f0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:37.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b6b7fe640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b54003980 0x7f0b6c1f0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b6b7fe640 1 -- 172.21.15.125:0/4214955923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c1f36e0 con 0x7f0b6c158790 2023-10-30T12:57:37.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b6a7fc640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b6c1f01b0 0x7f0b6c0b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:57:37.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b6affd640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b54003980 0x7f0b6c1f0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:57:37.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b6b7fe640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b6c158790 0x7f0b6c1f2a30 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c06cb00 tx=0x7f0b6400e930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:37.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b53fff640 1 -- 172.21.15.125:0/4214955923 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64006e30 con 0x7f0b6c158790 2023-10-30T12:57:37.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b53fff640 1 -- 172.21.15.125:0/4214955923 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b64003980 con 0x7f0b6c158790 2023-10-30T12:57:37.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b53fff640 1 -- 172.21.15.125:0/4214955923 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b64003d30 con 0x7f0b6c158790 2023-10-30T12:57:37.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.336+0000 7f0b71846640 1 -- 172.21.15.125:0/4214955923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b6c1ef6d0 con 0x7f0b6c158790 2023-10-30T12:57:37.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.340+0000 7f0b71846640 1 -- 172.21.15.125:0/4214955923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b6c1efc90 con 0x7f0b6c158790 2023-10-30T12:57:37.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.340+0000 7f0b71846640 1 -- 172.21.15.125:0/4214955923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b6c1efeb0 con 0x7f0b6c158790 2023-10-30T12:57:37.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.340+0000 7f0b53fff640 1 -- 172.21.15.125:0/4214955923 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0b64005070 con 0x7f0b6c158790 2023-10-30T12:57:37.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.340+0000 7f0b53fff640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0b580664d0 0x7f0b58068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:37.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.340+0000 7f0b53fff640 1 -- 172.21.15.125:0/4214955923 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7f0b64083b80 con 0x7f0b6c158790 2023-10-30T12:57:37.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.340+0000 7f0b6affd640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0b580664d0 0x7f0b58068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:37.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.340+0000 7f0b6affd640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0b580664d0 0x7f0b58068990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c0af000 tx=0x7f0b5c0aa000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:37.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.344+0000 7f0b53fff640 1 -- 172.21.15.125:0/4214955923 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0b64050a00 con 0x7f0b6c158790 2023-10-30T12:57:37.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.528+0000 7f0b71846640 1 -- 172.21.15.125:0/4214955923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b6c04e170 con 0x7f0b6c158790 2023-10-30T12:57:37.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.528+0000 7f0b53fff640 1 -- 172.21.15.125:0/4214955923 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6516 (secure 0 0 0) 0x7f0b64019090 con 0x7f0b6c158790 2023-10-30T12:57:37.532 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:57:37.532 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7413,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:57.659452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:57:37.532 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 133 2023-10-30T12:57:37.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.528+0000 7f0b71846640 1 -- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0b580664d0 msgr2=0x7f0b58068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:37.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.528+0000 7f0b71846640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0b580664d0 0x7f0b58068990 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c0af000 tx=0x7f0b5c0aa000 comp rx=0 tx=0).stop 2023-10-30T12:57:37.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.528+0000 7f0b71846640 1 -- 172.21.15.125:0/4214955923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b6c158790 msgr2=0x7f0b6c1f2a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:37.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.528+0000 7f0b71846640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b6c158790 0x7f0b6c1f2a30 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f0b6c06cb00 tx=0x7f0b6400e930 comp rx=0 tx=0).stop 2023-10-30T12:57:37.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.528+0000 7f0b71846640 1 -- 172.21.15.125:0/4214955923 shutdown_connections 2023-10-30T12:57:37.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.528+0000 7f0b71846640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0b580664d0 0x7f0b58068990 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.528+0000 7f0b71846640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b6c158790 0x7f0b6c1f2a30 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.528+0000 7f0b71846640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b6c1f01b0 0x7f0b6c0b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.528+0000 7f0b71846640 1 --2- 172.21.15.125:0/4214955923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b54003980 0x7f0b6c1f0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:37.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.528+0000 7f0b71846640 1 -- 172.21.15.125:0/4214955923 >> 172.21.15.125:0/4214955923 conn(0x7f0b6c06bfe0 msgr2=0x7f0b6c06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:37.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.532+0000 7f0b71846640 1 -- 172.21.15.125:0/4214955923 shutdown_connections 2023-10-30T12:57:37.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:37.532+0000 7f0b71846640 1 -- 172.21.15.125:0/4214955923 wait complete. 2023-10-30T12:57:37.553 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:57:39.554 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:57:39.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.696+0000 7fc8aef3e640 1 Processor -- start 2023-10-30T12:57:39.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.696+0000 7fc8aef3e640 1 -- start start 2023-10-30T12:57:39.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.696+0000 7fc8aef3e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc8a80cf160 0x7fc8a80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:39.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.696+0000 7fc8aef3e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a80cb800 con 0x7fc8a81538a0 2023-10-30T12:57:39.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.696+0000 7fc8aef3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a80cb330 con 0x7fc8a815d6f0 2023-10-30T12:57:39.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.696+0000 7fc8aef3e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8a80cad70 con 0x7fc8a80cf160 2023-10-30T12:57:39.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.696+0000 7fc8accda640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc8a81538a0 0x7fc8a80ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:56302/0 (socket says 172.21.15.125:56302) 2023-10-30T12:57:39.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.696+0000 7fc8accda640 1 -- 172.21.15.125:0/3777020112 learned_addr learned my addr 172.21.15.125:0/3777020112 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:57:39.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.696+0000 7fc8ad4db640 1 --2- 172.21.15.125:0/3777020112 >> v2:172.21.15.136:3300/0 conn(0x7fc8a80cf160 0x7fc8a80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:39.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.696+0000 7fc8a77fe640 1 -- 172.21.15.125:0/3777020112 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086325439 0 0) 0x7fc8a80cb800 con 0x7fc8a81538a0 2023-10-30T12:57:39.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.696+0000 7fc8a77fe640 1 -- 172.21.15.125:0/3777020112 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc898003530 con 0x7fc8a81538a0 2023-10-30T12:57:39.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 -- 172.21.15.125:0/3777020112 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3837765699 0 0) 0x7fc898003530 con 0x7fc8a81538a0 2023-10-30T12:57:39.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 -- 172.21.15.125:0/3777020112 >> v2:172.21.15.136:3300/0 conn(0x7fc8a80cf160 msgr2=0x7fc8a80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:39.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 --2- 172.21.15.125:0/3777020112 >> v2:172.21.15.136:3300/0 conn(0x7fc8a80cf160 0x7fc8a80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:39.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 -- 172.21.15.125:0/3777020112 >> v1:172.21.15.136:6789/0 conn(0x7fc8a815d6f0 legacy=0x7fc8a815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:39.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 -- 172.21.15.125:0/3777020112 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a80cb050 con 0x7fc8a81538a0 2023-10-30T12:57:39.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 -- 172.21.15.125:0/3777020112 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fc89c001680 con 0x7fc8a81538a0 2023-10-30T12:57:39.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 -- 172.21.15.125:0/3777020112 >> v1:172.21.15.125:6789/0 conn(0x7fc8a81538a0 legacy=0x7fc8a80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:39.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 --2- 172.21.15.125:0/3777020112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc898003980 0x7fc898003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:39.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 --2- 172.21.15.125:0/3777020112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8980047d0 0x7fc898004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:39.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 --2- 172.21.15.125:0/3777020112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc898005190 0x7fc89800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:39.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 -- 172.21.15.125:0/3777020112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a80cad70 con 0x7fc8980047d0 2023-10-30T12:57:39.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 -- 172.21.15.125:0/3777020112 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89800c040 con 0x7fc898003980 2023-10-30T12:57:39.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 -- 172.21.15.125:0/3777020112 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc89800c200 con 0x7fc898005190 2023-10-30T12:57:39.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a7fff640 1 --2- 172.21.15.125:0/3777020112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8980047d0 0x7fc898004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:39.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8accda640 1 --2- 172.21.15.125:0/3777020112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc898003980 0x7fc898003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:39.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8ad4db640 1 --2- 172.21.15.125:0/3777020112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc898005190 0x7fc89800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a7fff640 1 -- 172.21.15.125:0/3777020112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc898005190 msgr2=0x7fc89800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a7fff640 1 --2- 172.21.15.125:0/3777020112 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc898005190 0x7fc89800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:39.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a7fff640 1 -- 172.21.15.125:0/3777020112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc898003980 msgr2=0x7fc898003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:39.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a7fff640 1 --2- 172.21.15.125:0/3777020112 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc898003980 0x7fc898003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:39.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a7fff640 1 -- 172.21.15.125:0/3777020112 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc898003530 con 0x7fc8980047d0 2023-10-30T12:57:39.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8ad4db640 1 -- 172.21.15.125:0/3777020112 reap_dead start 2023-10-30T12:57:39.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a7fff640 1 --2- 172.21.15.125:0/3777020112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8980047d0 0x7fc898004c50 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fc894001780 tx=0x7fc89401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb9607a87f63d58 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:39.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 -- 172.21.15.125:0/3777020112 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8940052d0 con 0x7fc8980047d0 2023-10-30T12:57:39.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 -- 172.21.15.125:0/3777020112 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8940a7050 con 0x7fc8980047d0 2023-10-30T12:57:39.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.700+0000 7fc8a77fe640 1 -- 172.21.15.125:0/3777020112 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc894003600 con 0x7fc8980047d0 2023-10-30T12:57:39.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 -- 172.21.15.125:0/3777020112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8980047d0 msgr2=0x7fc898004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:39.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 --2- 172.21.15.125:0/3777020112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8980047d0 0x7fc898004c50 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7fc894001780 tx=0x7fc89401c400 comp rx=0 tx=0).stop 2023-10-30T12:57:39.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 -- 172.21.15.125:0/3777020112 shutdown_connections 2023-10-30T12:57:39.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 --2- 172.21.15.125:0/3777020112 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8980047d0 0x7fc898004c50 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:39.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 -- 172.21.15.125:0/3777020112 >> 172.21.15.125:0/3777020112 conn(0x7fc8a806bfe0 msgr2=0x7fc8a806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:39.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 -- 172.21.15.125:0/3777020112 shutdown_connections 2023-10-30T12:57:39.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 -- 172.21.15.125:0/3777020112 wait complete. 2023-10-30T12:57:39.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 Processor -- start 2023-10-30T12:57:39.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 -- start start 2023-10-30T12:57:39.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8980047d0 0x7fc8a814e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:39.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8a81e5550 0x7fc8a814f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:39.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8a814ea30 0x7fc8a814fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:39.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a8150790 con 0x7fc8a81e5550 2023-10-30T12:57:39.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a81f0190 con 0x7fc8980047d0 2023-10-30T12:57:39.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8aef3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a81ef350 con 0x7fc8a814ea30 2023-10-30T12:57:39.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8a7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8a81e5550 0x7fc8a814f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:39.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8a7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8a81e5550 0x7fc8a814f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47062/0 (socket says 172.21.15.125:47062) 2023-10-30T12:57:39.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8a7fff640 1 -- 172.21.15.125:0/2906022004 learned_addr learned my addr 172.21.15.125:0/2906022004 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:39.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8ad4db640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8a814ea30 0x7fc8a814fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:39.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.704+0000 7fc8accda640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8980047d0 0x7fc8a814e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:39.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8a7fff640 1 -- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8a814ea30 msgr2=0x7fc8a814fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:39.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8a7fff640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8a814ea30 0x7fc8a814fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:39.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8a7fff640 1 -- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8980047d0 msgr2=0x7fc8a814e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:39.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8a7fff640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8980047d0 0x7fc8a814e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:39.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8a7fff640 1 -- 172.21.15.125:0/2906022004 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a81ef610 con 0x7fc8a81e5550 2023-10-30T12:57:39.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8ad4db640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8a814ea30 0x7fc8a814fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:57:39.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8accda640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8980047d0 0x7fc8a814e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:57:39.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8a7fff640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8a81e5550 0x7fc8a814f6d0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fc89401c230 tx=0x7fc898005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:39.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8a57fa640 1 -- 172.21.15.125:0/2906022004 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc894007500 con 0x7fc8a81e5550 2023-10-30T12:57:39.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8a57fa640 1 -- 172.21.15.125:0/2906022004 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc894007660 con 0x7fc8a81e5550 2023-10-30T12:57:39.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8aef3e640 1 -- 172.21.15.125:0/2906022004 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a81ef8f0 con 0x7fc8a81e5550 2023-10-30T12:57:39.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8a57fa640 1 -- 172.21.15.125:0/2906022004 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc894006a20 con 0x7fc8a81e5550 2023-10-30T12:57:39.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8aef3e640 1 -- 172.21.15.125:0/2906022004 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a81efeb0 con 0x7fc8a81e5550 2023-10-30T12:57:39.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8aef3e640 1 -- 172.21.15.125:0/2906022004 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8a804e170 con 0x7fc8a81e5550 2023-10-30T12:57:39.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8a57fa640 1 -- 172.21.15.125:0/2906022004 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc894006b80 con 0x7fc8a81e5550 2023-10-30T12:57:39.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8a57fa640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc88c0664a0 0x7fc88c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:39.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.708+0000 7fc8a57fa640 1 -- 172.21.15.125:0/2906022004 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7fc89411e580 con 0x7fc8a81e5550 2023-10-30T12:57:39.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.712+0000 7fc8accda640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc88c0664a0 0x7fc88c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:39.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.712+0000 7fc8accda640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc88c0664a0 0x7fc88c068960 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc89c001ca0 tx=0x7fc89c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:39.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.712+0000 7fc8a57fa640 1 -- 172.21.15.125:0/2906022004 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc8940eefb0 con 0x7fc8a81e5550 2023-10-30T12:57:39.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.892+0000 7fc8aef3e640 1 -- 172.21.15.125:0/2906022004 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8a80b1dd0 con 0x7fc8a81e5550 2023-10-30T12:57:39.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.892+0000 7fc8a57fa640 1 -- 172.21.15.125:0/2906022004 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6516 (secure 0 0 0) 0x7fc8940eeb40 con 0x7fc8a81e5550 2023-10-30T12:57:39.897 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:57:39.897 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7413,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:57.659452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:57:39.898 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 133 2023-10-30T12:57:39.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.896+0000 7fc8aef3e640 1 -- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc88c0664a0 msgr2=0x7fc88c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:39.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.896+0000 7fc8aef3e640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc88c0664a0 0x7fc88c068960 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fc89c001ca0 tx=0x7fc89c009040 comp rx=0 tx=0).stop 2023-10-30T12:57:39.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.896+0000 7fc8aef3e640 1 -- 172.21.15.125:0/2906022004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8a81e5550 msgr2=0x7fc8a814f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:39.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.896+0000 7fc8aef3e640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8a81e5550 0x7fc8a814f6d0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7fc89401c230 tx=0x7fc898005800 comp rx=0 tx=0).stop 2023-10-30T12:57:39.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.896+0000 7fc8aef3e640 1 -- 172.21.15.125:0/2906022004 shutdown_connections 2023-10-30T12:57:39.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.896+0000 7fc8aef3e640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc88c0664a0 0x7fc88c068960 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:39.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.896+0000 7fc8aef3e640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8a814ea30 0x7fc8a814fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:39.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.896+0000 7fc8aef3e640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8a81e5550 0x7fc8a814f6d0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:39.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.896+0000 7fc8aef3e640 1 --2- 172.21.15.125:0/2906022004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8980047d0 0x7fc8a814e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:39.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.896+0000 7fc8aef3e640 1 -- 172.21.15.125:0/2906022004 >> 172.21.15.125:0/2906022004 conn(0x7fc8a806bfe0 msgr2=0x7fc8a8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:39.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.896+0000 7fc8aef3e640 1 -- 172.21.15.125:0/2906022004 shutdown_connections 2023-10-30T12:57:39.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:39.896+0000 7fc8aef3e640 1 -- 172.21.15.125:0/2906022004 wait complete. 2023-10-30T12:57:39.918 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:57:41.919 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:57:42.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.060+0000 7f2c09372640 1 Processor -- start 2023-10-30T12:57:42.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.060+0000 7f2c09372640 1 -- start start 2023-10-30T12:57:42.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.060+0000 7f2c09372640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2c040c05d0 0x7f2c0413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:42.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.060+0000 7f2c09372640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c040c0a50 con 0x7f2c040c6b80 2023-10-30T12:57:42.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.060+0000 7f2c09372640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c0406f6e0 con 0x7f2c040c6f60 2023-10-30T12:57:42.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.060+0000 7f2c09372640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c0406f9c0 con 0x7f2c040c05d0 2023-10-30T12:57:42.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.060+0000 7f2c02ffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2c040c6b80 0x7f2c040bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42306/0 (socket says 172.21.15.125:42306) 2023-10-30T12:57:42.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.060+0000 7f2c02ffd640 1 -- 172.21.15.125:0/3002212448 learned_addr learned my addr 172.21.15.125:0/3002212448 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:57:42.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.060+0000 7f2bfa7fc640 1 --2- 172.21.15.125:0/3002212448 >> v2:172.21.15.125:3300/0 conn(0x7f2c040c05d0 0x7f2c0413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:42.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.064+0000 7f2c027fc640 1 -- 172.21.15.125:0/3002212448 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800577815 0 0) 0x7f2c040c0a50 con 0x7f2c040c6b80 2023-10-30T12:57:42.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.064+0000 7f2c027fc640 1 -- 172.21.15.125:0/3002212448 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bf0003530 con 0x7f2c040c6b80 2023-10-30T12:57:42.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.064+0000 7f2bfa7fc640 1 -- 172.21.15.125:0/3002212448 >> v1:172.21.15.136:6789/0 conn(0x7f2c040c6f60 legacy=0x7f2c041566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:42.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.064+0000 7f2bfa7fc640 1 -- 172.21.15.125:0/3002212448 >> v1:172.21.15.125:6789/0 conn(0x7f2c040c6b80 legacy=0x7f2c040bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:42.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.064+0000 7f2bfa7fc640 1 -- 172.21.15.125:0/3002212448 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c0406fca0 con 0x7f2c040c05d0 2023-10-30T12:57:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.064+0000 7f2bfa7fc640 1 --2- 172.21.15.125:0/3002212448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c040c05d0 0x7f2c0413ed90 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40097c0 tx=0x7f2bf401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e33b499b05e69be server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:42.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.064+0000 7f2c027fc640 1 -- 172.21.15.125:0/3002212448 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf40a8050 con 0x7f2c040c05d0 2023-10-30T12:57:42.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.064+0000 7f2c027fc640 1 -- 172.21.15.125:0/3002212448 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf40ac5f0 con 0x7f2c040c05d0 2023-10-30T12:57:42.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.064+0000 7f2c027fc640 1 -- 172.21.15.125:0/3002212448 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf40ac980 con 0x7f2c040c05d0 2023-10-30T12:57:42.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.064+0000 7f2c09372640 1 -- 172.21.15.125:0/3002212448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c040c05d0 msgr2=0x7f2c0413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:42.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.064+0000 7f2c09372640 1 --2- 172.21.15.125:0/3002212448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c040c05d0 0x7f2c0413ed90 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40097c0 tx=0x7f2bf401c480 comp rx=0 tx=0).stop 2023-10-30T12:57:42.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c09372640 1 -- 172.21.15.125:0/3002212448 shutdown_connections 2023-10-30T12:57:42.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c09372640 1 --2- 172.21.15.125:0/3002212448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c040c05d0 0x7f2c0413ed90 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:42.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c09372640 1 -- 172.21.15.125:0/3002212448 >> 172.21.15.125:0/3002212448 conn(0x7f2c040c7910 msgr2=0x7f2c040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:42.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c09372640 1 -- 172.21.15.125:0/3002212448 shutdown_connections 2023-10-30T12:57:42.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c09372640 1 -- 172.21.15.125:0/3002212448 wait complete. 2023-10-30T12:57:42.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c09372640 1 Processor -- start 2023-10-30T12:57:42.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c09372640 1 -- start start 2023-10-30T12:57:42.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c09372640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c040c05d0 0x7f2c04138640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:42.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c09372640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c040c6b80 0x7f2c040c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:42.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c09372640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c040c6f60 0x7f2c0413a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:42.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c09372640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c04135000 con 0x7f2c040c6b80 2023-10-30T12:57:42.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c09372640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c04133be0 con 0x7f2c040c6f60 2023-10-30T12:57:42.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c09372640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c04137d80 con 0x7f2c040c05d0 2023-10-30T12:57:42.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2bfa7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c040c6b80 0x7f2c040c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:42.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2bfa7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c040c6b80 0x7f2c040c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45438/0 (socket says 172.21.15.125:45438) 2023-10-30T12:57:42.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2bfa7fc640 1 -- 172.21.15.125:0/3200834965 learned_addr learned my addr 172.21.15.125:0/3200834965 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:42.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c02ffd640 1 --2- 172.21.15.125:0/3200834965 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c040c05d0 0x7f2c04138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:42.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2c037fe640 1 --2- 172.21.15.125:0/3200834965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c040c6f60 0x7f2c0413a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:42.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2bfa7fc640 1 -- 172.21.15.125:0/3200834965 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c040c05d0 msgr2=0x7f2c04138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:42.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2bfa7fc640 1 --2- 172.21.15.125:0/3200834965 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c040c05d0 0x7f2c04138640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:42.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2bfa7fc640 1 -- 172.21.15.125:0/3200834965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c040c6f60 msgr2=0x7f2c0413a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:42.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2bfa7fc640 1 --2- 172.21.15.125:0/3200834965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c040c6f60 0x7f2c0413a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:42.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2bfa7fc640 1 -- 172.21.15.125:0/3200834965 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf4009420 con 0x7f2c040c6b80 2023-10-30T12:57:42.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2bfa7fc640 1 --2- 172.21.15.125:0/3200834965 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c040c6b80 0x7f2c040c8c10 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40af040 tx=0x7f2bf4019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:42.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.068+0000 7f2bfbfff640 1 -- 172.21.15.125:0/3200834965 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf4006e20 con 0x7f2c040c6b80 2023-10-30T12:57:42.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.072+0000 7f2bfbfff640 1 -- 172.21.15.125:0/3200834965 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bf40b5620 con 0x7f2c040c6b80 2023-10-30T12:57:42.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.072+0000 7f2bfbfff640 1 -- 172.21.15.125:0/3200834965 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bf40b59f0 con 0x7f2c040c6b80 2023-10-30T12:57:42.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.072+0000 7f2c09372640 1 -- 172.21.15.125:0/3200834965 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c04138060 con 0x7f2c040c6b80 2023-10-30T12:57:42.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.072+0000 7f2c09372640 1 -- 172.21.15.125:0/3200834965 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c04133590 con 0x7f2c040c6b80 2023-10-30T12:57:42.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.072+0000 7f2c09372640 1 -- 172.21.15.125:0/3200834965 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c0404e170 con 0x7f2c040c6b80 2023-10-30T12:57:42.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.072+0000 7f2bfbfff640 1 -- 172.21.15.125:0/3200834965 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2bf40b3040 con 0x7f2c040c6b80 2023-10-30T12:57:42.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.072+0000 7f2bfbfff640 1 --2- 172.21.15.125:0/3200834965 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2be40664a0 0x7f2be4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:42.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.072+0000 7f2bfbfff640 1 -- 172.21.15.125:0/3200834965 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7f2bf411e080 con 0x7f2c040c6b80 2023-10-30T12:57:42.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.072+0000 7f2c02ffd640 1 --2- 172.21.15.125:0/3200834965 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2be40664a0 0x7f2be4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:42.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.076+0000 7f2c02ffd640 1 --2- 172.21.15.125:0/3200834965 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2be40664a0 0x7f2be4068960 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2bec0011f0 tx=0x7f2bec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:42.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.076+0000 7f2bfbfff640 1 -- 172.21.15.125:0/3200834965 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2bf40eaf00 con 0x7f2c040c6b80 2023-10-30T12:57:42.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.268+0000 7f2c09372640 1 -- 172.21.15.125:0/3200834965 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c04156750 con 0x7f2c040c6b80 2023-10-30T12:57:42.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.268+0000 7f2bfbfff640 1 -- 172.21.15.125:0/3200834965 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6516 (secure 0 0 0) 0x7f2bf40b1020 con 0x7f2c040c6b80 2023-10-30T12:57:42.272 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:57:42.272 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7413,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:57.659452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:57:42.272 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 133 2023-10-30T12:57:42.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.268+0000 7f2c09372640 1 -- 172.21.15.125:0/3200834965 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2be40664a0 msgr2=0x7f2be4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:42.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.268+0000 7f2c09372640 1 --2- 172.21.15.125:0/3200834965 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2be40664a0 0x7f2be4068960 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f2bec0011f0 tx=0x7f2bec009040 comp rx=0 tx=0).stop 2023-10-30T12:57:42.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.268+0000 7f2c09372640 1 -- 172.21.15.125:0/3200834965 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c040c6b80 msgr2=0x7f2c040c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:42.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.268+0000 7f2c09372640 1 --2- 172.21.15.125:0/3200834965 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c040c6b80 0x7f2c040c8c10 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f2bf40af040 tx=0x7f2bf4019220 comp rx=0 tx=0).stop 2023-10-30T12:57:42.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.268+0000 7f2c09372640 1 -- 172.21.15.125:0/3200834965 shutdown_connections 2023-10-30T12:57:42.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.268+0000 7f2c09372640 1 --2- 172.21.15.125:0/3200834965 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2be40664a0 0x7f2be4068960 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:42.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.268+0000 7f2c09372640 1 --2- 172.21.15.125:0/3200834965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c040c6f60 0x7f2c0413a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:42.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.268+0000 7f2c09372640 1 --2- 172.21.15.125:0/3200834965 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c040c6b80 0x7f2c040c8c10 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:42.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.268+0000 7f2c09372640 1 --2- 172.21.15.125:0/3200834965 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c040c05d0 0x7f2c04138640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:42.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.268+0000 7f2c09372640 1 -- 172.21.15.125:0/3200834965 >> 172.21.15.125:0/3200834965 conn(0x7f2c040c7910 msgr2=0x7f2c040c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:42.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.268+0000 7f2c09372640 1 -- 172.21.15.125:0/3200834965 shutdown_connections 2023-10-30T12:57:42.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:42.268+0000 7f2c09372640 1 -- 172.21.15.125:0/3200834965 wait complete. 2023-10-30T12:57:42.292 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:57:44.294 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:57:44.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f7a013c5640 1 Processor -- start 2023-10-30T12:57:44.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f7a013c5640 1 -- start start 2023-10-30T12:57:44.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f7a013c5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f79fc14fe40 0x7f79fc159a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:44.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f7a013c5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f79fc06ac60 0x7f79fc06bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:44.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f7a013c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79fc06bff0 0x7f79fc069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:44.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f7a013c5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79fc06dee0 con 0x7f79fc14fe40 2023-10-30T12:57:44.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f7a013c5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79fc06c680 con 0x7f79fc06ac60 2023-10-30T12:57:44.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f7a013c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc0cd0c0 con 0x7f79fc06bff0 2023-10-30T12:57:44.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f79faffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f79fc14fe40 0x7f79fc159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:44.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f79faffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f79fc14fe40 0x7f79fc159a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45452/0 (socket says 172.21.15.125:45452) 2023-10-30T12:57:44.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f79faffd640 1 -- 172.21.15.125:0/4222033935 learned_addr learned my addr 172.21.15.125:0/4222033935 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:44.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f79fbfff640 1 --2- 172.21.15.125:0/4222033935 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79fc06bff0 0x7f79fc069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:44.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f79fa7fc640 1 --2- 172.21.15.125:0/4222033935 >> v2:172.21.15.136:3300/0 conn(0x7f79fc06ac60 0x7f79fc06bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:44.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f79faffd640 1 -- 172.21.15.125:0/4222033935 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79fc06bff0 msgr2=0x7f79fc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:44.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f79faffd640 1 --2- 172.21.15.125:0/4222033935 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79fc06bff0 0x7f79fc069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:44.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f79faffd640 1 -- 172.21.15.125:0/4222033935 >> v2:172.21.15.136:3300/0 conn(0x7f79fc06ac60 msgr2=0x7f79fc06bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:44.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f79faffd640 1 --2- 172.21.15.125:0/4222033935 >> v2:172.21.15.136:3300/0 conn(0x7f79fc06ac60 0x7f79fc06bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:44.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.440+0000 7f79faffd640 1 -- 172.21.15.125:0/4222033935 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79fc159fb0 con 0x7f79fc14fe40 2023-10-30T12:57:44.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.444+0000 7f79faffd640 1 --2- 172.21.15.125:0/4222033935 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79fc14fe40 0x7f79fc159a70 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f79fc06ca60 tx=0x7f79ec0213c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4da97487299e1a48 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:44.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.444+0000 7f79f9ffb640 1 -- 172.21.15.125:0/4222033935 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec0b1600 con 0x7f79fc14fe40 2023-10-30T12:57:44.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.444+0000 7f79f9ffb640 1 -- 172.21.15.125:0/4222033935 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ec0ad050 con 0x7f79fc14fe40 2023-10-30T12:57:44.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.444+0000 7f79f9ffb640 1 -- 172.21.15.125:0/4222033935 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec0b1980 con 0x7f79fc14fe40 2023-10-30T12:57:44.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.444+0000 7f7a013c5640 1 -- 172.21.15.125:0/4222033935 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79fc14fe40 msgr2=0x7f79fc159a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:44.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.444+0000 7f7a013c5640 1 --2- 172.21.15.125:0/4222033935 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79fc14fe40 0x7f79fc159a70 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f79fc06ca60 tx=0x7f79ec0213c0 comp rx=0 tx=0).stop 2023-10-30T12:57:44.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.444+0000 7f7a013c5640 1 -- 172.21.15.125:0/4222033935 shutdown_connections 2023-10-30T12:57:44.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.444+0000 7f7a013c5640 1 --2- 172.21.15.125:0/4222033935 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79fc06bff0 0x7f79fc069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:44.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.444+0000 7f7a013c5640 1 --2- 172.21.15.125:0/4222033935 >> v2:172.21.15.136:3300/0 conn(0x7f79fc06ac60 0x7f79fc06bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:44.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.444+0000 7f7a013c5640 1 --2- 172.21.15.125:0/4222033935 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79fc14fe40 0x7f79fc159a70 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:44.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.444+0000 7f7a013c5640 1 -- 172.21.15.125:0/4222033935 >> 172.21.15.125:0/4222033935 conn(0x7f79fc0cc410 msgr2=0x7f79fc0cc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:44.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.444+0000 7f7a013c5640 1 -- 172.21.15.125:0/4222033935 shutdown_connections 2023-10-30T12:57:44.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.444+0000 7f7a013c5640 1 -- 172.21.15.125:0/4222033935 wait complete. 2023-10-30T12:57:44.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f7a013c5640 1 Processor -- start 2023-10-30T12:57:44.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f7a013c5640 1 -- start start 2023-10-30T12:57:44.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f7a013c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79fc06ac60 0x7f79fc0c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:44.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f7a013c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79fc06bff0 0x7f79fc151f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:44.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f7a013c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79fc14fe40 0x7f79fc14cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:44.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f7a013c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc06a7e0 con 0x7f79fc14fe40 2023-10-30T12:57:44.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f7a013c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc1eebb0 con 0x7f79fc06bff0 2023-10-30T12:57:44.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f7a013c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc151200 con 0x7f79fc06ac60 2023-10-30T12:57:44.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79fbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79fc14fe40 0x7f79fc14cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:44.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79faffd640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79fc06ac60 0x7f79fc0c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:44.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79fbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79fc14fe40 0x7f79fc14cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45456/0 (socket says 172.21.15.125:45456) 2023-10-30T12:57:44.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79fbfff640 1 -- 172.21.15.125:0/3188587420 learned_addr learned my addr 172.21.15.125:0/3188587420 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:44.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79fa7fc640 1 --2- 172.21.15.125:0/3188587420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79fc06bff0 0x7f79fc151f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:44.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79fbfff640 1 -- 172.21.15.125:0/3188587420 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79fc06ac60 msgr2=0x7f79fc0c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:44.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79fbfff640 1 --2- 172.21.15.125:0/3188587420 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79fc06ac60 0x7f79fc0c4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:44.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79fbfff640 1 -- 172.21.15.125:0/3188587420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79fc06bff0 msgr2=0x7f79fc151f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:44.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79fbfff640 1 --2- 172.21.15.125:0/3188587420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79fc06bff0 0x7f79fc151f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:44.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79fbfff640 1 -- 172.21.15.125:0/3188587420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ec00b770 con 0x7f79fc14fe40 2023-10-30T12:57:44.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79fa7fc640 1 --2- 172.21.15.125:0/3188587420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79fc06bff0 0x7f79fc151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:57:44.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79fbfff640 1 --2- 172.21.15.125:0/3188587420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79fc14fe40 0x7f79fc14cc40 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f79f4007e40 tx=0x7f79f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:44.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79e37fe640 1 -- 172.21.15.125:0/3188587420 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4014620 con 0x7f79fc14fe40 2023-10-30T12:57:44.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79e37fe640 1 -- 172.21.15.125:0/3188587420 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79f400ee90 con 0x7f79fc14fe40 2023-10-30T12:57:44.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f7a013c5640 1 -- 172.21.15.125:0/3188587420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79fc151aa0 con 0x7f79fc14fe40 2023-10-30T12:57:44.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f79e37fe640 1 -- 172.21.15.125:0/3188587420 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f40149e0 con 0x7f79fc14fe40 2023-10-30T12:57:44.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.448+0000 7f7a013c5640 1 -- 172.21.15.125:0/3188587420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79fc1eedc0 con 0x7f79fc14fe40 2023-10-30T12:57:44.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.452+0000 7f7a013c5640 1 -- 172.21.15.125:0/3188587420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79fc151c30 con 0x7f79fc14fe40 2023-10-30T12:57:44.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.452+0000 7f79e37fe640 1 -- 172.21.15.125:0/3188587420 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f79f4014b40 con 0x7f79fc14fe40 2023-10-30T12:57:44.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.452+0000 7f79e37fe640 1 --2- 172.21.15.125:0/3188587420 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f79e80664a0 0x7f79e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:44.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.452+0000 7f79e37fe640 1 -- 172.21.15.125:0/3188587420 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7f79f4084ce0 con 0x7f79fc14fe40 2023-10-30T12:57:44.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.452+0000 7f79faffd640 1 --2- 172.21.15.125:0/3188587420 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f79e80664a0 0x7f79e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:44.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.452+0000 7f79faffd640 1 --2- 172.21.15.125:0/3188587420 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f79e80664a0 0x7f79e8068960 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0b3000 tx=0x7f79ec0ac000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:44.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.456+0000 7f79e37fe640 1 -- 172.21.15.125:0/3188587420 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f79f4051bf0 con 0x7f79fc14fe40 2023-10-30T12:57:44.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.640+0000 7f7a013c5640 1 -- 172.21.15.125:0/3188587420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79fc06e440 con 0x7f79fc14fe40 2023-10-30T12:57:44.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.640+0000 7f79e37fe640 1 -- 172.21.15.125:0/3188587420 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6516 (secure 0 0 0) 0x7f79f4055210 con 0x7f79fc14fe40 2023-10-30T12:57:44.643 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:57:44.643 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7413,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:57.659452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:57:44.644 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 133 2023-10-30T12:57:44.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.640+0000 7f7a013c5640 1 -- 172.21.15.125:0/3188587420 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f79e80664a0 msgr2=0x7f79e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:44.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.640+0000 7f7a013c5640 1 --2- 172.21.15.125:0/3188587420 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f79e80664a0 0x7f79e8068960 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0b3000 tx=0x7f79ec0ac000 comp rx=0 tx=0).stop 2023-10-30T12:57:44.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.640+0000 7f7a013c5640 1 -- 172.21.15.125:0/3188587420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79fc14fe40 msgr2=0x7f79fc14cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:44.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.640+0000 7f7a013c5640 1 --2- 172.21.15.125:0/3188587420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79fc14fe40 0x7f79fc14cc40 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f79f4007e40 tx=0x7f79f400e4b0 comp rx=0 tx=0).stop 2023-10-30T12:57:44.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.640+0000 7f7a013c5640 1 -- 172.21.15.125:0/3188587420 shutdown_connections 2023-10-30T12:57:44.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.640+0000 7f7a013c5640 1 --2- 172.21.15.125:0/3188587420 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f79e80664a0 0x7f79e8068960 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:44.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.640+0000 7f7a013c5640 1 --2- 172.21.15.125:0/3188587420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f79fc14fe40 0x7f79fc14cc40 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:44.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.640+0000 7f7a013c5640 1 --2- 172.21.15.125:0/3188587420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f79fc06bff0 0x7f79fc151f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:44.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.640+0000 7f7a013c5640 1 --2- 172.21.15.125:0/3188587420 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f79fc06ac60 0x7f79fc0c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:44.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.640+0000 7f7a013c5640 1 -- 172.21.15.125:0/3188587420 >> 172.21.15.125:0/3188587420 conn(0x7f79fc0cc410 msgr2=0x7f79fc158ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:44.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.640+0000 7f7a013c5640 1 -- 172.21.15.125:0/3188587420 shutdown_connections 2023-10-30T12:57:44.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:44.640+0000 7f7a013c5640 1 -- 172.21.15.125:0/3188587420 wait complete. 2023-10-30T12:57:44.664 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:57:46.665 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:57:46.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f582989e640 1 Processor -- start 2023-10-30T12:57:46.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f582989e640 1 -- start start 2023-10-30T12:57:46.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f582989e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f582406f7a0 0x7f582406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:46.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f582989e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5824071110 0x7f5824158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:46.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f582989e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5824066cf0 con 0x7f58240c6bd0 2023-10-30T12:57:46.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f582989e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58240700c0 con 0x7f582406f7a0 2023-10-30T12:57:46.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f582989e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58240c70d0 con 0x7f5824071110 2023-10-30T12:57:46.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f581a7fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f58240c6bd0 0x7f5824069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42310/0 (socket says 172.21.15.125:42310) 2023-10-30T12:57:46.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f581a7fc640 1 -- 172.21.15.125:0/357590537 learned_addr learned my addr 172.21.15.125:0/357590537 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:57:46.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58237fe640 1 --2- 172.21.15.125:0/357590537 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5824071110 0x7f5824158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:46.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f5822ffd640 1 --2- 172.21.15.125:0/357590537 >> v2:172.21.15.136:3300/0 conn(0x7f582406f7a0 0x7f582406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:46.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 -- 172.21.15.125:0/357590537 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849338120 0 0) 0x7f5824066cf0 con 0x7f58240c6bd0 2023-10-30T12:57:46.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 -- 172.21.15.125:0/357590537 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5810003530 con 0x7f58240c6bd0 2023-10-30T12:57:46.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 -- 172.21.15.125:0/357590537 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3677962207 0 0) 0x7f5810003530 con 0x7f58240c6bd0 2023-10-30T12:57:46.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 -- 172.21.15.125:0/357590537 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5824071110 msgr2=0x7f5824158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:46.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 --2- 172.21.15.125:0/357590537 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5824071110 0x7f5824158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:46.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 -- 172.21.15.125:0/357590537 >> v2:172.21.15.136:3300/0 conn(0x7f582406f7a0 msgr2=0x7f582406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:46.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 --2- 172.21.15.125:0/357590537 >> v2:172.21.15.136:3300/0 conn(0x7f582406f7a0 0x7f582406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:46.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 -- 172.21.15.125:0/357590537 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58240c7390 con 0x7f58240c6bd0 2023-10-30T12:57:46.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 -- 172.21.15.125:0/357590537 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f580c001a00 con 0x7f58240c6bd0 2023-10-30T12:57:46.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 -- 172.21.15.125:0/357590537 >> v1:172.21.15.125:6789/0 conn(0x7f58240c6bd0 legacy=0x7f5824069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:46.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 --2- 172.21.15.125:0/357590537 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5810003980 0x7f5810003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:46.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 --2- 172.21.15.125:0/357590537 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5824071110 0x7f58100047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:46.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 --2- 172.21.15.125:0/357590537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5810004d10 0x7f5810005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:46.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 -- 172.21.15.125:0/357590537 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58240700c0 con 0x7f5810004d10 2023-10-30T12:57:46.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 -- 172.21.15.125:0/357590537 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58240c70d0 con 0x7f5810003980 2023-10-30T12:57:46.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58227fc640 1 -- 172.21.15.125:0/357590537 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f581000b990 con 0x7f5824071110 2023-10-30T12:57:46.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58237fe640 1 --2- 172.21.15.125:0/357590537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5810004d10 0x7f5810005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:46.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f581a7fc640 1 --2- 172.21.15.125:0/357590537 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5824071110 0x7f58100047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:46.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f5822ffd640 1 --2- 172.21.15.125:0/357590537 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5810003980 0x7f5810003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:46.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58237fe640 1 -- 172.21.15.125:0/357590537 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5824071110 msgr2=0x7f58100047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:46.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58237fe640 1 --2- 172.21.15.125:0/357590537 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5824071110 0x7f58100047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:46.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58237fe640 1 -- 172.21.15.125:0/357590537 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5810003980 msgr2=0x7f5810003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:46.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58237fe640 1 --2- 172.21.15.125:0/357590537 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5810003980 0x7f5810003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:46.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.812+0000 7f58237fe640 1 -- 172.21.15.125:0/357590537 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5810003530 con 0x7f5810004d10 2023-10-30T12:57:46.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.816+0000 7f58237fe640 1 --2- 172.21.15.125:0/357590537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5810004d10 0x7f5810005190 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f581c009ae0 tx=0x7f581c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b08a46e24a412740 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:46.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.816+0000 7f58227fc640 1 -- 172.21.15.125:0/357590537 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c0a5640 con 0x7f5810004d10 2023-10-30T12:57:46.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.816+0000 7f58227fc640 1 -- 172.21.15.125:0/357590537 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c0a57a0 con 0x7f5810004d10 2023-10-30T12:57:46.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.816+0000 7f58227fc640 1 -- 172.21.15.125:0/357590537 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c0a5b50 con 0x7f5810004d10 2023-10-30T12:57:46.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.816+0000 7f582989e640 1 -- 172.21.15.125:0/357590537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5810004d10 msgr2=0x7f5810005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:46.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.816+0000 7f582989e640 1 --2- 172.21.15.125:0/357590537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5810004d10 0x7f5810005190 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f581c009ae0 tx=0x7f581c01ca50 comp rx=0 tx=0).stop 2023-10-30T12:57:46.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.816+0000 7f58237fe640 1 -- 172.21.15.125:0/357590537 reap_dead start 2023-10-30T12:57:46.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.816+0000 7f582989e640 1 -- 172.21.15.125:0/357590537 shutdown_connections 2023-10-30T12:57:46.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.816+0000 7f582989e640 1 -- 172.21.15.125:0/357590537 >> 172.21.15.125:0/357590537 conn(0x7f58240c7910 msgr2=0x7f58240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:46.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.816+0000 7f582989e640 1 -- 172.21.15.125:0/357590537 shutdown_connections 2023-10-30T12:57:46.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.816+0000 7f582989e640 1 -- 172.21.15.125:0/357590537 wait complete. 2023-10-30T12:57:46.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.816+0000 7f582989e640 1 Processor -- start 2023-10-30T12:57:46.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.816+0000 7f582989e640 1 -- start start 2023-10-30T12:57:46.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f582989e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5810004d10 0x7f58240af6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:46.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f582989e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5824154c70 0x7f58241f04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:46.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f582989e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58241e5fd0 0x7f58241f0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:46.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f582989e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58240b0390 con 0x7f58241e5fd0 2023-10-30T12:57:46.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f582989e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5824152f10 con 0x7f5824154c70 2023-10-30T12:57:46.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f582989e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5824153180 con 0x7f5810004d10 2023-10-30T12:57:46.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f58237fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58241e5fd0 0x7f58241f0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:46.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f58237fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58241e5fd0 0x7f58241f0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45476/0 (socket says 172.21.15.125:45476) 2023-10-30T12:57:46.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f58237fe640 1 -- 172.21.15.125:0/885985852 learned_addr learned my addr 172.21.15.125:0/885985852 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:46.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f581a7fc640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5824154c70 0x7f58241f04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:46.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f5822ffd640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5810004d10 0x7f58240af6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:46.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f58237fe640 1 -- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5810004d10 msgr2=0x7f58240af6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:46.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f58237fe640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5810004d10 0x7f58240af6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:46.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f58237fe640 1 -- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5824154c70 msgr2=0x7f58241f04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:46.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f58237fe640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5824154c70 0x7f58241f04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:46.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f58237fe640 1 -- 172.21.15.125:0/885985852 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5824153460 con 0x7f58241e5fd0 2023-10-30T12:57:46.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f581a7fc640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5824154c70 0x7f58241f04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:57:46.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f58237fe640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58241e5fd0 0x7f58241f0a00 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f581c0b0040 tx=0x7f5810005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:46.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f5820ff9640 1 -- 172.21.15.125:0/885985852 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c004220 con 0x7f58241e5fd0 2023-10-30T12:57:46.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f5820ff9640 1 -- 172.21.15.125:0/885985852 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f581c004380 con 0x7f58241e5fd0 2023-10-30T12:57:46.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f582989e640 1 -- 172.21.15.125:0/885985852 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58241effa0 con 0x7f58241e5fd0 2023-10-30T12:57:46.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f5820ff9640 1 -- 172.21.15.125:0/885985852 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f581c003860 con 0x7f58241e5fd0 2023-10-30T12:57:46.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f582989e640 1 -- 172.21.15.125:0/885985852 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5824151be0 con 0x7f58241e5fd0 2023-10-30T12:57:46.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f582989e640 1 -- 172.21.15.125:0/885985852 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58241efe00 con 0x7f58241e5fd0 2023-10-30T12:57:46.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.820+0000 7f5820ff9640 1 -- 172.21.15.125:0/885985852 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f581c0b4070 con 0x7f58241e5fd0 2023-10-30T12:57:46.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.824+0000 7f5820ff9640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58040664d0 0x7f5804068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:46.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.824+0000 7f5820ff9640 1 -- 172.21.15.125:0/885985852 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7f581c11e570 con 0x7f58241e5fd0 2023-10-30T12:57:46.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.824+0000 7f5822ffd640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58040664d0 0x7f5804068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:46.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.824+0000 7f5822ffd640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58040664d0 0x7f5804068990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5814009b00 tx=0x7f5814008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:46.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:46.828+0000 7f5820ff9640 1 -- 172.21.15.125:0/885985852 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f58241efe00 con 0x7f58241e5fd0 2023-10-30T12:57:47.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:47.004+0000 7f582989e640 1 -- 172.21.15.125:0/885985852 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58241efe00 con 0x7f58241e5fd0 2023-10-30T12:57:47.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:47.004+0000 7f5820ff9640 1 -- 172.21.15.125:0/885985852 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6516 (secure 0 0 0) 0x7f58241efe00 con 0x7f58241e5fd0 2023-10-30T12:57:47.010 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:57:47.010 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7413,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:57.659452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:57:47.011 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 133 2023-10-30T12:57:47.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:47.008+0000 7f582989e640 1 -- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58040664d0 msgr2=0x7f5804068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:47.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:47.008+0000 7f582989e640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58040664d0 0x7f5804068990 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f5814009b00 tx=0x7f5814008040 comp rx=0 tx=0).stop 2023-10-30T12:57:47.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:47.008+0000 7f582989e640 1 -- 172.21.15.125:0/885985852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58241e5fd0 msgr2=0x7f58241f0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:47.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:47.008+0000 7f582989e640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58241e5fd0 0x7f58241f0a00 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f581c0b0040 tx=0x7f5810005800 comp rx=0 tx=0).stop 2023-10-30T12:57:47.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:47.008+0000 7f582989e640 1 -- 172.21.15.125:0/885985852 shutdown_connections 2023-10-30T12:57:47.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:47.008+0000 7f582989e640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58040664d0 0x7f5804068990 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:47.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:47.008+0000 7f582989e640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58241e5fd0 0x7f58241f0a00 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:47.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:47.008+0000 7f582989e640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5824154c70 0x7f58241f04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:47.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:47.008+0000 7f582989e640 1 --2- 172.21.15.125:0/885985852 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5810004d10 0x7f58240af6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:47.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:47.008+0000 7f582989e640 1 -- 172.21.15.125:0/885985852 >> 172.21.15.125:0/885985852 conn(0x7f58240c7910 msgr2=0x7f58240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:47.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:47.008+0000 7f582989e640 1 -- 172.21.15.125:0/885985852 shutdown_connections 2023-10-30T12:57:47.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:47.008+0000 7f582989e640 1 -- 172.21.15.125:0/885985852 wait complete. 2023-10-30T12:57:47.030 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:57:49.031 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:57:49.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.176+0000 7f4087df5640 1 Processor -- start 2023-10-30T12:57:49.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.176+0000 7f4087df5640 1 -- start start 2023-10-30T12:57:49.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4087df5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4080153ba0 0x7f408015a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:49.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4087df5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f40800c74b0 0x7f4080069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:49.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4087df5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4080153f80 con 0x7f40800c7df0 2023-10-30T12:57:49.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4087df5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40800c1f50 con 0x7f40800c74b0 2023-10-30T12:57:49.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4087df5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40800b0e60 con 0x7f4080153ba0 2023-10-30T12:57:49.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4086392640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f40800c7df0 0x7f408015cdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42320/0 (socket says 172.21.15.125:42320) 2023-10-30T12:57:49.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4086392640 1 -- 172.21.15.125:0/896455863 learned_addr learned my addr 172.21.15.125:0/896455863 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:57:49.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4085b91640 1 --2- 172.21.15.125:0/896455863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4080153ba0 0x7f408015a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:49.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4085390640 1 --2- 172.21.15.125:0/896455863 >> v2:172.21.15.136:3300/0 conn(0x7f40800c74b0 0x7f4080069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:49.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 -- 172.21.15.125:0/896455863 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882568184 0 0) 0x7f4080153f80 con 0x7f40800c7df0 2023-10-30T12:57:49.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 -- 172.21.15.125:0/896455863 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f406c003530 con 0x7f40800c7df0 2023-10-30T12:57:49.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 -- 172.21.15.125:0/896455863 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4024308836 0 0) 0x7f406c003530 con 0x7f40800c7df0 2023-10-30T12:57:49.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 -- 172.21.15.125:0/896455863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4080153ba0 msgr2=0x7f408015a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:49.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 --2- 172.21.15.125:0/896455863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4080153ba0 0x7f408015a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:49.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 -- 172.21.15.125:0/896455863 >> v2:172.21.15.136:3300/0 conn(0x7f40800c74b0 msgr2=0x7f4080069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:49.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 --2- 172.21.15.125:0/896455863 >> v2:172.21.15.136:3300/0 conn(0x7f40800c74b0 0x7f4080069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:49.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 -- 172.21.15.125:0/896455863 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40800b1140 con 0x7f40800c7df0 2023-10-30T12:57:49.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 -- 172.21.15.125:0/896455863 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f4078001a00 con 0x7f40800c7df0 2023-10-30T12:57:49.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 -- 172.21.15.125:0/896455863 >> v1:172.21.15.125:6789/0 conn(0x7f40800c7df0 legacy=0x7f408015cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:49.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 --2- 172.21.15.125:0/896455863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f406c003980 0x7f406c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:49.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 --2- 172.21.15.125:0/896455863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4080153ba0 0x7f406c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:49.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 --2- 172.21.15.125:0/896455863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f406c004d10 0x7f406c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:49.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 -- 172.21.15.125:0/896455863 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40800c1f50 con 0x7f406c003980 2023-10-30T12:57:49.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 -- 172.21.15.125:0/896455863 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40800b0e60 con 0x7f406c004d10 2023-10-30T12:57:49.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4084b8f640 1 -- 172.21.15.125:0/896455863 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c00b990 con 0x7f4080153ba0 2023-10-30T12:57:49.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4085b91640 1 --2- 172.21.15.125:0/896455863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f406c003980 0x7f406c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:49.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4085390640 1 --2- 172.21.15.125:0/896455863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4080153ba0 0x7f406c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:49.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4086392640 1 --2- 172.21.15.125:0/896455863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f406c004d10 0x7f406c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:49.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4085b91640 1 -- 172.21.15.125:0/896455863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4080153ba0 msgr2=0x7f406c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:49.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4085b91640 1 --2- 172.21.15.125:0/896455863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4080153ba0 0x7f406c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:49.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4085b91640 1 -- 172.21.15.125:0/896455863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f406c004d10 msgr2=0x7f406c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:49.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4085b91640 1 --2- 172.21.15.125:0/896455863 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f406c004d10 0x7f406c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:49.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4085b91640 1 -- 172.21.15.125:0/896455863 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c003530 con 0x7f406c003980 2023-10-30T12:57:49.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.180+0000 7f4085390640 1 --2- 172.21.15.125:0/896455863 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4080153ba0 0x7f406c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:57:49.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.184+0000 7f4085b91640 1 --2- 172.21.15.125:0/896455863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f406c003980 0x7f406c003da0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f407c009ae0 tx=0x7f407c01ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8268b509471d490 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:49.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.184+0000 7f4084b8f640 1 -- 172.21.15.125:0/896455863 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c0a5530 con 0x7f406c003980 2023-10-30T12:57:49.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.184+0000 7f4084b8f640 1 -- 172.21.15.125:0/896455863 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407c0a5690 con 0x7f406c003980 2023-10-30T12:57:49.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.184+0000 7f4084b8f640 1 -- 172.21.15.125:0/896455863 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407c0a5a40 con 0x7f406c003980 2023-10-30T12:57:49.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.184+0000 7f4087df5640 1 -- 172.21.15.125:0/896455863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f406c003980 msgr2=0x7f406c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:49.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.184+0000 7f4087df5640 1 --2- 172.21.15.125:0/896455863 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f406c003980 0x7f406c003da0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f407c009ae0 tx=0x7f407c01ca50 comp rx=0 tx=0).stop 2023-10-30T12:57:49.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.184+0000 7f4086392640 1 -- 172.21.15.125:0/896455863 reap_dead start 2023-10-30T12:57:49.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.184+0000 7f4087df5640 1 -- 172.21.15.125:0/896455863 shutdown_connections 2023-10-30T12:57:49.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.184+0000 7f4087df5640 1 -- 172.21.15.125:0/896455863 >> 172.21.15.125:0/896455863 conn(0x7f408006d770 msgr2=0x7f408006db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:49.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.184+0000 7f4087df5640 1 -- 172.21.15.125:0/896455863 shutdown_connections 2023-10-30T12:57:49.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.184+0000 7f4087df5640 1 -- 172.21.15.125:0/896455863 wait complete. 2023-10-30T12:57:49.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4087df5640 1 Processor -- start 2023-10-30T12:57:49.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4087df5640 1 -- start start 2023-10-30T12:57:49.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4087df5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f406c003980 0x7f4080159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:49.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4087df5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f408014e1a0 0x7f4080070500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:49.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4087df5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4080070a40 0x7f40801588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:49.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4087df5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408015d980 con 0x7f4080070a40 2023-10-30T12:57:49.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4087df5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408014c890 con 0x7f408014e1a0 2023-10-30T12:57:49.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4087df5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408015b6d0 con 0x7f406c003980 2023-10-30T12:57:49.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4086392640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4080070a40 0x7f40801588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:49.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4086392640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4080070a40 0x7f40801588e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45498/0 (socket says 172.21.15.125:45498) 2023-10-30T12:57:49.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4086392640 1 -- 172.21.15.125:0/1994852548 learned_addr learned my addr 172.21.15.125:0/1994852548 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:49.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4085390640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f408014e1a0 0x7f4080070500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:49.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4085b91640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f406c003980 0x7f4080159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:49.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4086392640 1 -- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f406c003980 msgr2=0x7f4080159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:49.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4086392640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f406c003980 0x7f4080159b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:49.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4086392640 1 -- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f408014e1a0 msgr2=0x7f4080070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:49.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4086392640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f408014e1a0 0x7f4080070500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:49.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4086392640 1 -- 172.21.15.125:0/1994852548 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408015b9b0 con 0x7f4080070a40 2023-10-30T12:57:49.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4085b91640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f406c003980 0x7f4080159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:57:49.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4085390640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f408014e1a0 0x7f4080070500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:57:49.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4086392640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4080070a40 0x7f40801588e0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f4078002040 tx=0x7f4078011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:49.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f40767fc640 1 -- 172.21.15.125:0/1994852548 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407800f840 con 0x7f4080070a40 2023-10-30T12:57:49.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f40767fc640 1 -- 172.21.15.125:0/1994852548 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407800f9c0 con 0x7f4080070a40 2023-10-30T12:57:49.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f40767fc640 1 -- 172.21.15.125:0/1994852548 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407800fd70 con 0x7f4080070a40 2023-10-30T12:57:49.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4087df5640 1 -- 172.21.15.125:0/1994852548 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408015bc90 con 0x7f4080070a40 2023-10-30T12:57:49.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4087df5640 1 -- 172.21.15.125:0/1994852548 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408015c250 con 0x7f4080070a40 2023-10-30T12:57:49.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.188+0000 7f4087df5640 1 -- 172.21.15.125:0/1994852548 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4080074250 con 0x7f4080070a40 2023-10-30T12:57:49.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.192+0000 7f40767fc640 1 -- 172.21.15.125:0/1994852548 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f407801f460 con 0x7f4080070a40 2023-10-30T12:57:49.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.192+0000 7f40767fc640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f40640664a0 0x7f4064068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:49.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.192+0000 7f40767fc640 1 -- 172.21.15.125:0/1994852548 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6942+0+0 (secure 0 0 0) 0x7f4078087360 con 0x7f4080070a40 2023-10-30T12:57:49.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.192+0000 7f4085b91640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f40640664a0 0x7f4064068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:49.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.192+0000 7f4085b91640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f40640664a0 0x7f4064068960 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f407c0af000 tx=0x7f407c0046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:49.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.196+0000 7f40767fc640 1 -- 172.21.15.125:0/1994852548 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4078057d90 con 0x7f4080070a40 2023-10-30T12:57:49.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.376+0000 7f4087df5640 1 -- 172.21.15.125:0/1994852548 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f408004e170 con 0x7f4080070a40 2023-10-30T12:57:49.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.376+0000 7f40767fc640 1 -- 172.21.15.125:0/1994852548 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+6516 (secure 0 0 0) 0x7f4078057920 con 0x7f4080070a40 2023-10-30T12:57:49.376 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:57:49.376 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7413,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[{"mdsmap":{"epoch":132,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:56:57.659452+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7269},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7269":{"gid":7269,"name":"e","rank":0,"incarnation":129,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6836/739564283","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":739564283},{"type":"v1","addr":"172.21.15.125:6836","nonce":739564283}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:57:49.376 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 133 2023-10-30T12:57:49.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.376+0000 7f4087df5640 1 -- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f40640664a0 msgr2=0x7f4064068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:49.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.376+0000 7f4087df5640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f40640664a0 0x7f4064068960 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f407c0af000 tx=0x7f407c0046d0 comp rx=0 tx=0).stop 2023-10-30T12:57:49.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.376+0000 7f4087df5640 1 -- 172.21.15.125:0/1994852548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4080070a40 msgr2=0x7f40801588e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:49.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.376+0000 7f4087df5640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4080070a40 0x7f40801588e0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f4078002040 tx=0x7f4078011c20 comp rx=0 tx=0).stop 2023-10-30T12:57:49.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.376+0000 7f4087df5640 1 -- 172.21.15.125:0/1994852548 shutdown_connections 2023-10-30T12:57:49.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.376+0000 7f4087df5640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f40640664a0 0x7f4064068960 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:49.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.376+0000 7f4087df5640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4080070a40 0x7f40801588e0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:49.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.376+0000 7f4087df5640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f408014e1a0 0x7f4080070500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:49.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.376+0000 7f4087df5640 1 --2- 172.21.15.125:0/1994852548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f406c003980 0x7f4080159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:49.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.376+0000 7f4087df5640 1 -- 172.21.15.125:0/1994852548 >> 172.21.15.125:0/1994852548 conn(0x7f408006d770 msgr2=0x7f4080151a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:49.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.376+0000 7f4087df5640 1 -- 172.21.15.125:0/1994852548 shutdown_connections 2023-10-30T12:57:49.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:49.376+0000 7f4087df5640 1 -- 172.21.15.125:0/1994852548 wait complete. 2023-10-30T12:57:49.396 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T12:57:49.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:57:49.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:57:51.398 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:57:51.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f20409d3640 1 Processor -- start 2023-10-30T12:57:51.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f20409d3640 1 -- start start 2023-10-30T12:57:51.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f20409d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c1538a0 0x7f203c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:51.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f20409d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f203c0ca0e0 0x7f203c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:51.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f20409d3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f203c0cb800 con 0x7f203c0cf160 2023-10-30T12:57:51.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f20409d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f203c15dc30 con 0x7f203c0ca0e0 2023-10-30T12:57:51.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f20409d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203c0cb330 con 0x7f203c1538a0 2023-10-30T12:57:51.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f203a59c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c1538a0 0x7f203c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:51.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f2033fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f203c0ca0e0 0x7f203c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:51.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f203a59c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c1538a0 0x7f203c15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:42478/0 (socket says 172.21.15.125:42478) 2023-10-30T12:57:51.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f203a59c640 1 -- 172.21.15.125:0/1242129818 learned_addr learned my addr 172.21.15.125:0/1242129818 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:51.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f203a59c640 1 -- 172.21.15.125:0/1242129818 >> v2:172.21.15.136:3300/0 conn(0x7f203c0ca0e0 msgr2=0x7f203c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:51.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f203a59c640 1 --2- 172.21.15.125:0/1242129818 >> v2:172.21.15.136:3300/0 conn(0x7f203c0ca0e0 0x7f203c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:51.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f203a59c640 1 -- 172.21.15.125:0/1242129818 >> v1:172.21.15.136:6789/0 conn(0x7f203c0cf160 legacy=0x7f203c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:51.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f203a59c640 1 -- 172.21.15.125:0/1242129818 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f203c0cad70 con 0x7f203c1538a0 2023-10-30T12:57:51.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.548+0000 7f2033fff640 1 --2- 172.21.15.125:0/1242129818 >> v2:172.21.15.136:3300/0 conn(0x7f203c0ca0e0 0x7f203c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.552+0000 7f203a59c640 1 --2- 172.21.15.125:0/1242129818 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c1538a0 0x7f203c15d6f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f202c009870 tx=0x7f202c01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e835448c6187b86 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:51.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.552+0000 7f2039d9b640 1 -- 172.21.15.125:0/1242129818 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c0a8a30 con 0x7f203c1538a0 2023-10-30T12:57:51.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.552+0000 7f2039d9b640 1 -- 172.21.15.125:0/1242129818 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202c0a8b90 con 0x7f203c1538a0 2023-10-30T12:57:51.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.552+0000 7f2039d9b640 1 -- 172.21.15.125:0/1242129818 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202c0b0440 con 0x7f203c1538a0 2023-10-30T12:57:51.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.552+0000 7f20409d3640 1 -- 172.21.15.125:0/1242129818 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c1538a0 msgr2=0x7f203c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:51.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.552+0000 7f20409d3640 1 --2- 172.21.15.125:0/1242129818 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c1538a0 0x7f203c15d6f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f202c009870 tx=0x7f202c01ed30 comp rx=0 tx=0).stop 2023-10-30T12:57:51.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.552+0000 7f20409d3640 1 -- 172.21.15.125:0/1242129818 shutdown_connections 2023-10-30T12:57:51.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.552+0000 7f20409d3640 1 --2- 172.21.15.125:0/1242129818 >> v2:172.21.15.136:3300/0 conn(0x7f203c0ca0e0 0x7f203c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:51.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.552+0000 7f20409d3640 1 --2- 172.21.15.125:0/1242129818 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c1538a0 0x7f203c15d6f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:51.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.552+0000 7f20409d3640 1 -- 172.21.15.125:0/1242129818 >> 172.21.15.125:0/1242129818 conn(0x7f203c06bfe0 msgr2=0x7f203c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:51.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.552+0000 7f20409d3640 1 -- 172.21.15.125:0/1242129818 shutdown_connections 2023-10-30T12:57:51.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.552+0000 7f20409d3640 1 -- 172.21.15.125:0/1242129818 wait complete. 2023-10-30T12:57:51.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f20409d3640 1 Processor -- start 2023-10-30T12:57:51.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f20409d3640 1 -- start start 2023-10-30T12:57:51.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f20409d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f203c0ca0e0 0x7f203c15b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:51.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f20409d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f203c0cf160 0x7f203c0ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:51.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f20409d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c1538a0 0x7f203c1e3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:51.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f20409d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c1e2ae0 con 0x7f203c0cf160 2023-10-30T12:57:51.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f20409d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f203c15cf60 con 0x7f203c0ca0e0 2023-10-30T12:57:51.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f20409d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f203c15cc30 con 0x7f203c1538a0 2023-10-30T12:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f2033fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f203c0cf160 0x7f203c0ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f2033fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f203c0cf160 0x7f203c0ce390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35502/0 (socket says 172.21.15.125:35502) 2023-10-30T12:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f2033fff640 1 -- 172.21.15.125:0/1727255566 learned_addr learned my addr 172.21.15.125:0/1727255566 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f203ad9d640 1 --2- 172.21.15.125:0/1727255566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c1538a0 0x7f203c1e3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:51.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f203a59c640 1 --2- 172.21.15.125:0/1727255566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f203c0ca0e0 0x7f203c15b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:51.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f2033fff640 1 -- 172.21.15.125:0/1727255566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c1538a0 msgr2=0x7f203c1e3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:51.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f2033fff640 1 --2- 172.21.15.125:0/1727255566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c1538a0 0x7f203c1e3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:51.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f2033fff640 1 -- 172.21.15.125:0/1727255566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f203c0ca0e0 msgr2=0x7f203c15b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:51.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f2033fff640 1 --2- 172.21.15.125:0/1727255566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f203c0ca0e0 0x7f203c15b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:51.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f2033fff640 1 -- 172.21.15.125:0/1727255566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2024009420 con 0x7f203c0cf160 2023-10-30T12:57:51.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f2033fff640 1 --2- 172.21.15.125:0/1727255566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f203c0cf160 0x7f203c0ce390 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f202400e9b0 tx=0x7f202400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:51.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f2032ffd640 1 -- 172.21.15.125:0/1727255566 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f202400ca00 con 0x7f203c0cf160 2023-10-30T12:57:51.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f2032ffd640 1 -- 172.21.15.125:0/1727255566 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f202400cb60 con 0x7f203c0cf160 2023-10-30T12:57:51.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f2032ffd640 1 -- 172.21.15.125:0/1727255566 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2024013470 con 0x7f203c0cf160 2023-10-30T12:57:51.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f20409d3640 1 -- 172.21.15.125:0/1727255566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202c0094d0 con 0x7f203c0cf160 2023-10-30T12:57:51.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.556+0000 7f20409d3640 1 -- 172.21.15.125:0/1727255566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f203c1e2cf0 con 0x7f203c0cf160 2023-10-30T12:57:51.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.560+0000 7f20409d3640 1 -- 172.21.15.125:0/1727255566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f203c0b2a50 con 0x7f203c0cf160 2023-10-30T12:57:51.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.560+0000 7f2032ffd640 1 -- 172.21.15.125:0/1727255566 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f20240135d0 con 0x7f203c0cf160 2023-10-30T12:57:51.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.560+0000 7f2032ffd640 1 --2- 172.21.15.125:0/1727255566 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f201c0664a0 0x7f201c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:51.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.560+0000 7f2032ffd640 1 -- 172.21.15.125:0/1727255566 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f20240846a0 con 0x7f203c0cf160 2023-10-30T12:57:51.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.560+0000 7f203a59c640 1 --2- 172.21.15.125:0/1727255566 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f201c0664a0 0x7f201c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:51.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.560+0000 7f203a59c640 1 --2- 172.21.15.125:0/1727255566 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f201c0664a0 0x7f201c068960 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f202c0b2000 tx=0x7f202c0aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:51.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.564+0000 7f2032ffd640 1 -- 172.21.15.125:0/1727255566 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2024051570 con 0x7f203c0cf160 2023-10-30T12:57:51.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.744+0000 7f20409d3640 1 -- 172.21.15.125:0/1727255566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f203c159850 con 0x7f203c0cf160 2023-10-30T12:57:51.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.744+0000 7f2032ffd640 1 -- 172.21.15.125:0/1727255566 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f2024054b90 con 0x7f203c0cf160 2023-10-30T12:57:51.766 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:57:51.766 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:57:50.441755+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:57:51.768 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 134 2023-10-30T12:57:51.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.748+0000 7f20409d3640 1 -- 172.21.15.125:0/1727255566 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f201c0664a0 msgr2=0x7f201c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:51.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.748+0000 7f20409d3640 1 --2- 172.21.15.125:0/1727255566 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f201c0664a0 0x7f201c068960 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f202c0b2000 tx=0x7f202c0aa070 comp rx=0 tx=0).stop 2023-10-30T12:57:51.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.748+0000 7f20409d3640 1 -- 172.21.15.125:0/1727255566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f203c0cf160 msgr2=0x7f203c0ce390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:51.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.748+0000 7f20409d3640 1 --2- 172.21.15.125:0/1727255566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f203c0cf160 0x7f203c0ce390 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f202400e9b0 tx=0x7f202400ee70 comp rx=0 tx=0).stop 2023-10-30T12:57:51.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.748+0000 7f20409d3640 1 -- 172.21.15.125:0/1727255566 shutdown_connections 2023-10-30T12:57:51.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.748+0000 7f20409d3640 1 --2- 172.21.15.125:0/1727255566 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f201c0664a0 0x7f201c068960 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:51.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.748+0000 7f20409d3640 1 --2- 172.21.15.125:0/1727255566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f203c1538a0 0x7f203c1e3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:51.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.748+0000 7f20409d3640 1 --2- 172.21.15.125:0/1727255566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f203c0cf160 0x7f203c0ce390 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:51.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.748+0000 7f20409d3640 1 --2- 172.21.15.125:0/1727255566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f203c0ca0e0 0x7f203c15b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:51.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.748+0000 7f20409d3640 1 -- 172.21.15.125:0/1727255566 >> 172.21.15.125:0/1727255566 conn(0x7f203c06bfe0 msgr2=0x7f203c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:51.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.748+0000 7f20409d3640 1 -- 172.21.15.125:0/1727255566 shutdown_connections 2023-10-30T12:57:51.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.748+0000 7f20409d3640 1 -- 172.21.15.125:0/1727255566 wait complete. 2023-10-30T12:57:51.771 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T12:57:51.771 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:57:51.771 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:57:51.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.908+0000 7f25999f5640 1 Processor -- start 2023-10-30T12:57:51.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.908+0000 7f25999f5640 1 -- start start 2023-10-30T12:57:51.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.912+0000 7f25999f5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25941538a0 0x7f259415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:51.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.912+0000 7f25999f5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25940cb800 con 0x7f25940cf160 2023-10-30T12:57:51.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.912+0000 7f25999f5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259415dc30 con 0x7f25940ca0e0 2023-10-30T12:57:51.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.912+0000 7f25999f5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25940cb330 con 0x7f25941538a0 2023-10-30T12:57:51.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.912+0000 7f25937fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f25940cf160 0x7f2594164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53928/0 (socket says 172.21.15.125:53928) 2023-10-30T12:57:51.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.912+0000 7f25937fe640 1 -- 172.21.15.125:0/1099651776 learned_addr learned my addr 172.21.15.125:0/1099651776 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:57:51.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.912+0000 7f2592ffd640 1 --2- 172.21.15.125:0/1099651776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25941538a0 0x7f259415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:51.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.912+0000 7f25927fc640 1 -- 172.21.15.125:0/1099651776 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775386808 0 0) 0x7f25940cb800 con 0x7f25940cf160 2023-10-30T12:57:51.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.912+0000 7f25927fc640 1 -- 172.21.15.125:0/1099651776 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2580003530 con 0x7f25940cf160 2023-10-30T12:57:51.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.912+0000 7f2592ffd640 1 -- 172.21.15.125:0/1099651776 >> v1:172.21.15.136:6789/0 conn(0x7f25940ca0e0 legacy=0x7f259415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:51.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.912+0000 7f25927fc640 1 -- 172.21.15.125:0/1099651776 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774087935 0 0) 0x7f257c001120 con 0x7f25940ca0e0 2023-10-30T12:57:51.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.912+0000 7f2592ffd640 1 -- 172.21.15.125:0/1099651776 >> v1:172.21.15.125:6789/0 conn(0x7f25940cf160 legacy=0x7f2594164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:51.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.912+0000 7f2592ffd640 1 -- 172.21.15.125:0/1099651776 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25940cad70 con 0x7f25941538a0 2023-10-30T12:57:51.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f2592ffd640 1 --2- 172.21.15.125:0/1099651776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25941538a0 0x7f259415d6f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f25840097f0 tx=0x7f258401eaf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec9fc169262a6834 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:51.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25927fc640 1 -- 172.21.15.125:0/1099651776 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c001980 con 0x7f25941538a0 2023-10-30T12:57:51.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25927fc640 1 -- 172.21.15.125:0/1099651776 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25840a85b0 con 0x7f25941538a0 2023-10-30T12:57:51.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25927fc640 1 -- 172.21.15.125:0/1099651776 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25840a8940 con 0x7f25941538a0 2023-10-30T12:57:51.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 -- 172.21.15.125:0/1099651776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25941538a0 msgr2=0x7f259415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:51.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 --2- 172.21.15.125:0/1099651776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25941538a0 0x7f259415d6f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f25840097f0 tx=0x7f258401eaf0 comp rx=0 tx=0).stop 2023-10-30T12:57:51.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 -- 172.21.15.125:0/1099651776 shutdown_connections 2023-10-30T12:57:51.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 --2- 172.21.15.125:0/1099651776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25941538a0 0x7f259415d6f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:51.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 -- 172.21.15.125:0/1099651776 >> 172.21.15.125:0/1099651776 conn(0x7f259406bfe0 msgr2=0x7f259406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:51.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 -- 172.21.15.125:0/1099651776 shutdown_connections 2023-10-30T12:57:51.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 -- 172.21.15.125:0/1099651776 wait complete. 2023-10-30T12:57:51.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 Processor -- start 2023-10-30T12:57:51.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 -- start start 2023-10-30T12:57:51.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25940ca0e0 0x7f25940b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:51.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25940cf160 0x7f25941efd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:51.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25941538a0 0x7f25941482a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:51.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2594158980 con 0x7f25941538a0 2023-10-30T12:57:51.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25941ef8b0 con 0x7f25940ca0e0 2023-10-30T12:57:51.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.916+0000 7f25999f5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25941ef280 con 0x7f25940cf160 2023-10-30T12:57:51.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f25937fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25941538a0 0x7f25941482a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:51.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f25937fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25941538a0 0x7f25941482a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35514/0 (socket says 172.21.15.125:35514) 2023-10-30T12:57:51.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f25937fe640 1 -- 172.21.15.125:0/2138055900 learned_addr learned my addr 172.21.15.125:0/2138055900 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:51.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f258bfff640 1 --2- 172.21.15.125:0/2138055900 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25940cf160 0x7f25941efd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:51.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f2592ffd640 1 --2- 172.21.15.125:0/2138055900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25940ca0e0 0x7f25940b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:51.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f25937fe640 1 -- 172.21.15.125:0/2138055900 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25940cf160 msgr2=0x7f25941efd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:51.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f25937fe640 1 --2- 172.21.15.125:0/2138055900 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25940cf160 0x7f25941efd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:51.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f25937fe640 1 -- 172.21.15.125:0/2138055900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25940ca0e0 msgr2=0x7f25940b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:51.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f25937fe640 1 --2- 172.21.15.125:0/2138055900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25940ca0e0 0x7f25940b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:51.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f25937fe640 1 -- 172.21.15.125:0/2138055900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2584009420 con 0x7f25941538a0 2023-10-30T12:57:51.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f25937fe640 1 --2- 172.21.15.125:0/2138055900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25941538a0 0x7f25941482a0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f258c00cac0 tx=0x7f258c00cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:51.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f2590ff9640 1 -- 172.21.15.125:0/2138055900 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c00a700 con 0x7f25941538a0 2023-10-30T12:57:51.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f2590ff9640 1 -- 172.21.15.125:0/2138055900 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c00a860 con 0x7f25941538a0 2023-10-30T12:57:51.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f2590ff9640 1 -- 172.21.15.125:0/2138055900 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258c00abf0 con 0x7f25941538a0 2023-10-30T12:57:51.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f25999f5640 1 -- 172.21.15.125:0/2138055900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25940b49a0 con 0x7f25941538a0 2023-10-30T12:57:51.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f25999f5640 1 -- 172.21.15.125:0/2138055900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25940b4f60 con 0x7f25941538a0 2023-10-30T12:57:51.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f25999f5640 1 -- 172.21.15.125:0/2138055900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259404e170 con 0x7f25941538a0 2023-10-30T12:57:51.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f2590ff9640 1 -- 172.21.15.125:0/2138055900 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f258c00ad50 con 0x7f25941538a0 2023-10-30T12:57:51.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.920+0000 7f2590ff9640 1 --2- 172.21.15.125:0/2138055900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f25740664d0 0x7f2574068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:51.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.924+0000 7f2590ff9640 1 -- 172.21.15.125:0/2138055900 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f258c0839d0 con 0x7f25941538a0 2023-10-30T12:57:51.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.924+0000 7f2592ffd640 1 --2- 172.21.15.125:0/2138055900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f25740664d0 0x7f2574068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:51.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.924+0000 7f2592ffd640 1 --2- 172.21.15.125:0/2138055900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f25740664d0 0x7f2574068990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f25840b2000 tx=0x7f25840aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:51.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:51.924+0000 7f2590ff9640 1 -- 172.21.15.125:0/2138055900 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f258c0509d0 con 0x7f25941538a0 2023-10-30T12:57:52.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:52.108+0000 7f25999f5640 1 -- 172.21.15.125:0/2138055900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25941ef9f0 con 0x7f25941538a0 2023-10-30T12:57:52.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:52.108+0000 7f2590ff9640 1 -- 172.21.15.125:0/2138055900 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f258c018090 con 0x7f25941538a0 2023-10-30T12:57:52.110 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:57:52.110 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:57:50.441755+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:57:52.110 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 134 2023-10-30T12:57:52.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:52.112+0000 7f25999f5640 1 -- 172.21.15.125:0/2138055900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f25740664d0 msgr2=0x7f2574068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:52.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:52.112+0000 7f25999f5640 1 --2- 172.21.15.125:0/2138055900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f25740664d0 0x7f2574068990 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f25840b2000 tx=0x7f25840aa070 comp rx=0 tx=0).stop 2023-10-30T12:57:52.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:52.112+0000 7f25999f5640 1 -- 172.21.15.125:0/2138055900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25941538a0 msgr2=0x7f25941482a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:52.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:52.112+0000 7f25999f5640 1 --2- 172.21.15.125:0/2138055900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25941538a0 0x7f25941482a0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f258c00cac0 tx=0x7f258c00cf80 comp rx=0 tx=0).stop 2023-10-30T12:57:52.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:52.112+0000 7f25999f5640 1 -- 172.21.15.125:0/2138055900 shutdown_connections 2023-10-30T12:57:52.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:52.112+0000 7f25999f5640 1 --2- 172.21.15.125:0/2138055900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f25740664d0 0x7f2574068990 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:52.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:52.112+0000 7f25999f5640 1 --2- 172.21.15.125:0/2138055900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25941538a0 0x7f25941482a0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:52.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:52.112+0000 7f25999f5640 1 --2- 172.21.15.125:0/2138055900 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25940cf160 0x7f25941efd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:52.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:52.112+0000 7f25999f5640 1 --2- 172.21.15.125:0/2138055900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f25940ca0e0 0x7f25940b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:52.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:52.112+0000 7f25999f5640 1 -- 172.21.15.125:0/2138055900 >> 172.21.15.125:0/2138055900 conn(0x7f259406bfe0 msgr2=0x7f2594065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:52.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:52.112+0000 7f25999f5640 1 -- 172.21.15.125:0/2138055900 shutdown_connections 2023-10-30T12:57:52.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:52.112+0000 7f25999f5640 1 -- 172.21.15.125:0/2138055900 wait complete. 2023-10-30T12:57:52.132 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 134, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 134, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7413": { "addr": "172.21.15.125:6837/3906359913", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 3906359913, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 3906359913, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7413, "incarnation": 134, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 69, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:57:50.441755+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7413 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:57:54.133 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:57:54.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.280+0000 7f771dc9a640 1 Processor -- start 2023-10-30T12:57:54.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.280+0000 7f771dc9a640 1 -- start start 2023-10-30T12:57:54.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.284+0000 7f771dc9a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f77180c6b80 0x7f77180bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:54.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.284+0000 7f771dc9a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f77180c6f60 0x7f77180c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:54.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.284+0000 7f771dc9a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77180c0a50 con 0x7f77180c8430 2023-10-30T12:57:54.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.284+0000 7f771dc9a640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771806f6e0 con 0x7f77180c6b80 2023-10-30T12:57:54.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.284+0000 7f771dc9a640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f771806f9c0 con 0x7f77180c6f60 2023-10-30T12:57:54.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.284+0000 7f77177fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f77180c6b80 0x7f77180bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:54.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.284+0000 7f77177fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f77180c6b80 0x7f77180bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35518/0 (socket says 172.21.15.125:35518) 2023-10-30T12:57:54.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.284+0000 7f77177fe640 1 -- 172.21.15.125:0/1555137156 learned_addr learned my addr 172.21.15.125:0/1555137156 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:54.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.284+0000 7f7716ffd640 1 --2- 172.21.15.125:0/1555137156 >> v2:172.21.15.136:3300/0 conn(0x7f77180c6f60 0x7f77180c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:54.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.284+0000 7f77177fe640 1 -- 172.21.15.125:0/1555137156 >> v2:172.21.15.136:3300/0 conn(0x7f77180c6f60 msgr2=0x7f77180c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:54.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.284+0000 7f77177fe640 1 --2- 172.21.15.125:0/1555137156 >> v2:172.21.15.136:3300/0 conn(0x7f77180c6f60 0x7f77180c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:54.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.284+0000 7f77177fe640 1 -- 172.21.15.125:0/1555137156 >> v1:172.21.15.136:6789/0 conn(0x7f77180c8430 legacy=0x7f771813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:54.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.284+0000 7f77177fe640 1 -- 172.21.15.125:0/1555137156 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771806fca0 con 0x7f77180c6b80 2023-10-30T12:57:54.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.284+0000 7f7716ffd640 1 --2- 172.21.15.125:0/1555137156 >> v2:172.21.15.136:3300/0 conn(0x7f77180c6f60 0x7f77180c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:57:54.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f77177fe640 1 --2- 172.21.15.125:0/1555137156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77180c6b80 0x7f77180bfec0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f7718159b90 tx=0x7f770801c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d82fc8d8f476ec server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:54.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f77167fc640 1 -- 172.21.15.125:0/1555137156 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77080ac630 con 0x7f77180c6b80 2023-10-30T12:57:54.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f77167fc640 1 -- 172.21.15.125:0/1555137156 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77080ac790 con 0x7f77180c6b80 2023-10-30T12:57:54.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f77167fc640 1 -- 172.21.15.125:0/1555137156 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77080acb20 con 0x7f77180c6b80 2023-10-30T12:57:54.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 -- 172.21.15.125:0/1555137156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77180c6b80 msgr2=0x7f77180bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:54.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 --2- 172.21.15.125:0/1555137156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77180c6b80 0x7f77180bfec0 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f7718159b90 tx=0x7f770801c4c0 comp rx=0 tx=0).stop 2023-10-30T12:57:54.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 -- 172.21.15.125:0/1555137156 shutdown_connections 2023-10-30T12:57:54.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 --2- 172.21.15.125:0/1555137156 >> v2:172.21.15.136:3300/0 conn(0x7f77180c6f60 0x7f77180c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:54.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 --2- 172.21.15.125:0/1555137156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77180c6b80 0x7f77180bfec0 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:54.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 -- 172.21.15.125:0/1555137156 >> 172.21.15.125:0/1555137156 conn(0x7f77180c7910 msgr2=0x7f77180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:54.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 -- 172.21.15.125:0/1555137156 shutdown_connections 2023-10-30T12:57:54.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 -- 172.21.15.125:0/1555137156 wait complete. 2023-10-30T12:57:54.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 Processor -- start 2023-10-30T12:57:54.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 -- start start 2023-10-30T12:57:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77180c6b80 0x7f77181546b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f77180c6f60 0x7f771806a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77180c8430 0x7f77181f4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771813dd30 con 0x7f77180c6b80 2023-10-30T12:57:54.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77181355d0 con 0x7f77180c8430 2023-10-30T12:57:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.288+0000 7f771dc9a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77181346e0 con 0x7f77180c6f60 2023-10-30T12:57:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f77177fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77180c6b80 0x7f77181546b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f77177fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77180c6b80 0x7f77181546b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35532/0 (socket says 172.21.15.125:35532) 2023-10-30T12:57:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f77177fe640 1 -- 172.21.15.125:0/716407346 learned_addr learned my addr 172.21.15.125:0/716407346 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:54.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f7717fff640 1 --2- 172.21.15.125:0/716407346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77180c8430 0x7f77181f4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:54.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f7716ffd640 1 --2- 172.21.15.125:0/716407346 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f77180c6f60 0x7f771806a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:54.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f77177fe640 1 -- 172.21.15.125:0/716407346 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f77180c6f60 msgr2=0x7f771806a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:54.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f77177fe640 1 --2- 172.21.15.125:0/716407346 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f77180c6f60 0x7f771806a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:54.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f77177fe640 1 -- 172.21.15.125:0/716407346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77180c8430 msgr2=0x7f77181f4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:54.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f77177fe640 1 --2- 172.21.15.125:0/716407346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77180c8430 0x7f77181f4f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:54.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f77177fe640 1 -- 172.21.15.125:0/716407346 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77080094d0 con 0x7f77180c6b80 2023-10-30T12:57:54.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f77177fe640 1 --2- 172.21.15.125:0/716407346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77180c6b80 0x7f77181546b0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f77080af040 tx=0x7f770801ad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:54.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f76fffff640 1 -- 172.21.15.125:0/716407346 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77080ab8e0 con 0x7f77180c6b80 2023-10-30T12:57:54.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f76fffff640 1 -- 172.21.15.125:0/716407346 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77080aba60 con 0x7f77180c6b80 2023-10-30T12:57:54.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f771dc9a640 1 -- 172.21.15.125:0/716407346 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77181349c0 con 0x7f77180c6b80 2023-10-30T12:57:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f76fffff640 1 -- 172.21.15.125:0/716407346 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77080abe10 con 0x7f77180c6b80 2023-10-30T12:57:54.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f771dc9a640 1 -- 172.21.15.125:0/716407346 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7718134f80 con 0x7f77180c6b80 2023-10-30T12:57:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f771dc9a640 1 -- 172.21.15.125:0/716407346 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771804e170 con 0x7f77180c6b80 2023-10-30T12:57:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f76fffff640 1 -- 172.21.15.125:0/716407346 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f77080b3040 con 0x7f77180c6b80 2023-10-30T12:57:54.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f76fffff640 1 --2- 172.21.15.125:0/716407346 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f77040664a0 0x7f7704068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:54.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.292+0000 7f76fffff640 1 -- 172.21.15.125:0/716407346 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f770811ec10 con 0x7f77180c6b80 2023-10-30T12:57:54.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.296+0000 7f7716ffd640 1 --2- 172.21.15.125:0/716407346 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f77040664a0 0x7f7704068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:54.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.296+0000 7f7716ffd640 1 --2- 172.21.15.125:0/716407346 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f77040664a0 0x7f7704068960 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f77180af8c0 tx=0x7f770c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:54.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.296+0000 7f76fffff640 1 -- 172.21.15.125:0/716407346 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f77080eba30 con 0x7f77180c6b80 2023-10-30T12:57:54.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.480+0000 7f771dc9a640 1 -- 172.21.15.125:0/716407346 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7718157490 con 0x7f77180c6b80 2023-10-30T12:57:54.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.480+0000 7f76fffff640 1 -- 172.21.15.125:0/716407346 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f77080ef050 con 0x7f77180c6b80 2023-10-30T12:57:54.481 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:57:54.481 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:57:50.441755+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:57:54.482 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 134 2023-10-30T12:57:54.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.484+0000 7f771dc9a640 1 -- 172.21.15.125:0/716407346 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f77040664a0 msgr2=0x7f7704068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:54.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.484+0000 7f771dc9a640 1 --2- 172.21.15.125:0/716407346 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f77040664a0 0x7f7704068960 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f77180af8c0 tx=0x7f770c008040 comp rx=0 tx=0).stop 2023-10-30T12:57:54.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.484+0000 7f771dc9a640 1 -- 172.21.15.125:0/716407346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77180c6b80 msgr2=0x7f77181546b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:54.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.484+0000 7f771dc9a640 1 --2- 172.21.15.125:0/716407346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77180c6b80 0x7f77181546b0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f77080af040 tx=0x7f770801ad00 comp rx=0 tx=0).stop 2023-10-30T12:57:54.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.484+0000 7f771dc9a640 1 -- 172.21.15.125:0/716407346 shutdown_connections 2023-10-30T12:57:54.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.484+0000 7f771dc9a640 1 --2- 172.21.15.125:0/716407346 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f77040664a0 0x7f7704068960 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:54.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.484+0000 7f771dc9a640 1 --2- 172.21.15.125:0/716407346 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f77180c8430 0x7f77181f4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:54.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.484+0000 7f771dc9a640 1 --2- 172.21.15.125:0/716407346 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f77180c6f60 0x7f771806a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:54.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.484+0000 7f771dc9a640 1 --2- 172.21.15.125:0/716407346 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f77180c6b80 0x7f77181546b0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:54.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.484+0000 7f771dc9a640 1 -- 172.21.15.125:0/716407346 >> 172.21.15.125:0/716407346 conn(0x7f77180c7910 msgr2=0x7f77180c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:54.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.484+0000 7f771dc9a640 1 -- 172.21.15.125:0/716407346 shutdown_connections 2023-10-30T12:57:54.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:54.484+0000 7f771dc9a640 1 -- 172.21.15.125:0/716407346 wait complete. 2023-10-30T12:57:54.502 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:57:56.503 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:57:56.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.656+0000 7f6741f80640 1 Processor -- start 2023-10-30T12:57:56.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.656+0000 7f6741f80640 1 -- start start 2023-10-30T12:57:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.656+0000 7f6741f80640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f673c15d6f0 0x7f673c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.656+0000 7f6741f80640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f673c0cf160 0x7f673c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.656+0000 7f6741f80640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f673c0cb800 con 0x7f673c1538a0 2023-10-30T12:57:56.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.656+0000 7f6741f80640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673c0cb330 con 0x7f673c0cf160 2023-10-30T12:57:56.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.656+0000 7f6741f80640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673c0cad70 con 0x7f673c15d6f0 2023-10-30T12:57:56.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.660+0000 7f673bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f673c0cf160 0x7f673c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:56.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.660+0000 7f673b7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f673c1538a0 0x7f673c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53940/0 (socket says 172.21.15.125:53940) 2023-10-30T12:57:56.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.660+0000 7f673b7fe640 1 -- 172.21.15.125:0/423420452 learned_addr learned my addr 172.21.15.125:0/423420452 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:57:56.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.660+0000 7f673affd640 1 --2- 172.21.15.125:0/423420452 >> v2:172.21.15.136:3300/0 conn(0x7f673c15d6f0 0x7f673c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:56.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.660+0000 7f673bfff640 1 -- 172.21.15.125:0/423420452 >> v2:172.21.15.136:3300/0 conn(0x7f673c15d6f0 msgr2=0x7f673c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:56.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.660+0000 7f673bfff640 1 --2- 172.21.15.125:0/423420452 >> v2:172.21.15.136:3300/0 conn(0x7f673c15d6f0 0x7f673c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:56.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.660+0000 7f673bfff640 1 -- 172.21.15.125:0/423420452 >> v1:172.21.15.125:6789/0 conn(0x7f673c1538a0 legacy=0x7f673c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.660+0000 7f673bfff640 1 -- 172.21.15.125:0/423420452 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673c0cb050 con 0x7f673c0cf160 2023-10-30T12:57:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.660+0000 7f673a7fc640 1 -- 172.21.15.125:0/423420452 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227659452 0 0) 0x7f673c0cb800 con 0x7f673c1538a0 2023-10-30T12:57:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.660+0000 7f673bfff640 1 --2- 172.21.15.125:0/423420452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f673c0cf160 0x7f673c0cf590 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f673c0b30a0 tx=0x7f673401f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d43c334ea7ab6a61 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f673a7fc640 1 -- 172.21.15.125:0/423420452 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67340b0440 con 0x7f673c0cf160 2023-10-30T12:57:56.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f673a7fc640 1 -- 172.21.15.125:0/423420452 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67340b05a0 con 0x7f673c0cf160 2023-10-30T12:57:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f673a7fc640 1 -- 172.21.15.125:0/423420452 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67340b0930 con 0x7f673c0cf160 2023-10-30T12:57:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 -- 172.21.15.125:0/423420452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f673c0cf160 msgr2=0x7f673c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 --2- 172.21.15.125:0/423420452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f673c0cf160 0x7f673c0cf590 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f673c0b30a0 tx=0x7f673401f0f0 comp rx=0 tx=0).stop 2023-10-30T12:57:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 -- 172.21.15.125:0/423420452 shutdown_connections 2023-10-30T12:57:56.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 --2- 172.21.15.125:0/423420452 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f673c0cf160 0x7f673c0cf590 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:56.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 --2- 172.21.15.125:0/423420452 >> v2:172.21.15.136:3300/0 conn(0x7f673c15d6f0 0x7f673c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:56.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 -- 172.21.15.125:0/423420452 >> 172.21.15.125:0/423420452 conn(0x7f673c06bfe0 msgr2=0x7f673c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:56.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 -- 172.21.15.125:0/423420452 shutdown_connections 2023-10-30T12:57:56.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 -- 172.21.15.125:0/423420452 wait complete. 2023-10-30T12:57:56.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 Processor -- start 2023-10-30T12:57:56.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 -- start start 2023-10-30T12:57:56.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f673c0cf160 0x7f673c0c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:56.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f673c1538a0 0x7f673c0ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:56.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f673c15d6f0 0x7f673c1f2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:56.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0b3ed0 con 0x7f673c15d6f0 2023-10-30T12:57:56.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673c159500 con 0x7f673c1538a0 2023-10-30T12:57:56.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.664+0000 7f6741f80640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673c0b4100 con 0x7f673c0cf160 2023-10-30T12:57:56.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f673bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f673c15d6f0 0x7f673c1f2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:56.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f673bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f673c15d6f0 0x7f673c1f2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35552/0 (socket says 172.21.15.125:35552) 2023-10-30T12:57:56.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f673bfff640 1 -- 172.21.15.125:0/2795639922 learned_addr learned my addr 172.21.15.125:0/2795639922 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:56.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f673b7fe640 1 --2- 172.21.15.125:0/2795639922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f673c0cf160 0x7f673c0c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:56.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f673affd640 1 --2- 172.21.15.125:0/2795639922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f673c1538a0 0x7f673c0ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:56.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f673bfff640 1 -- 172.21.15.125:0/2795639922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f673c0cf160 msgr2=0x7f673c0c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:56.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f673bfff640 1 --2- 172.21.15.125:0/2795639922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f673c0cf160 0x7f673c0c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:56.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f673bfff640 1 -- 172.21.15.125:0/2795639922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f673c1538a0 msgr2=0x7f673c0ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:56.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f673bfff640 1 --2- 172.21.15.125:0/2795639922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f673c1538a0 0x7f673c0ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:56.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f673bfff640 1 -- 172.21.15.125:0/2795639922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6734009a70 con 0x7f673c15d6f0 2023-10-30T12:57:56.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f673bfff640 1 --2- 172.21.15.125:0/2795639922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f673c15d6f0 0x7f673c1f2ba0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f673401bd00 tx=0x7f673401be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:56.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f6723fff640 1 -- 172.21.15.125:0/2795639922 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67340b0440 con 0x7f673c15d6f0 2023-10-30T12:57:56.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f6723fff640 1 -- 172.21.15.125:0/2795639922 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67340ba820 con 0x7f673c15d6f0 2023-10-30T12:57:56.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f6741f80640 1 -- 172.21.15.125:0/2795639922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f673c0b4f30 con 0x7f673c15d6f0 2023-10-30T12:57:56.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f6723fff640 1 -- 172.21.15.125:0/2795639922 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67340bab90 con 0x7f673c15d6f0 2023-10-30T12:57:56.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f6741f80640 1 -- 172.21.15.125:0/2795639922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c1575c0 con 0x7f673c15d6f0 2023-10-30T12:57:56.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f6741f80640 1 -- 172.21.15.125:0/2795639922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673c0b2a50 con 0x7f673c15d6f0 2023-10-30T12:57:56.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.668+0000 7f6723fff640 1 -- 172.21.15.125:0/2795639922 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f67340b7050 con 0x7f673c15d6f0 2023-10-30T12:57:56.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.672+0000 7f6723fff640 1 --2- 172.21.15.125:0/2795639922 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67280664a0 0x7f6728068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:56.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.672+0000 7f6723fff640 1 -- 172.21.15.125:0/2795639922 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f6734122860 con 0x7f673c15d6f0 2023-10-30T12:57:56.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.672+0000 7f673b7fe640 1 --2- 172.21.15.125:0/2795639922 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67280664a0 0x7f6728068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:56.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.672+0000 7f673b7fe640 1 --2- 172.21.15.125:0/2795639922 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67280664a0 0x7f6728068960 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f672c007fd0 tx=0x7f672c007d60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:56.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.676+0000 7f6723fff640 1 -- 172.21.15.125:0/2795639922 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f67340efa10 con 0x7f673c15d6f0 2023-10-30T12:57:56.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.868+0000 7f6741f80640 1 -- 172.21.15.125:0/2795639922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f673c04e170 con 0x7f673c15d6f0 2023-10-30T12:57:56.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.868+0000 7f6723fff640 1 -- 172.21.15.125:0/2795639922 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f67340b5090 con 0x7f673c15d6f0 2023-10-30T12:57:56.869 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:57:56.869 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:57:50.441755+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:57:56.870 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 134 2023-10-30T12:57:56.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.872+0000 7f6741f80640 1 -- 172.21.15.125:0/2795639922 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67280664a0 msgr2=0x7f6728068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:56.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.872+0000 7f6741f80640 1 --2- 172.21.15.125:0/2795639922 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67280664a0 0x7f6728068960 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f672c007fd0 tx=0x7f672c007d60 comp rx=0 tx=0).stop 2023-10-30T12:57:56.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.872+0000 7f6741f80640 1 -- 172.21.15.125:0/2795639922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f673c15d6f0 msgr2=0x7f673c1f2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:56.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.872+0000 7f6741f80640 1 --2- 172.21.15.125:0/2795639922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f673c15d6f0 0x7f673c1f2ba0 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f673401bd00 tx=0x7f673401be90 comp rx=0 tx=0).stop 2023-10-30T12:57:56.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.872+0000 7f6741f80640 1 -- 172.21.15.125:0/2795639922 shutdown_connections 2023-10-30T12:57:56.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.872+0000 7f6741f80640 1 --2- 172.21.15.125:0/2795639922 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67280664a0 0x7f6728068960 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:56.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.872+0000 7f6741f80640 1 --2- 172.21.15.125:0/2795639922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f673c15d6f0 0x7f673c1f2ba0 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:56.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.872+0000 7f6741f80640 1 --2- 172.21.15.125:0/2795639922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f673c1538a0 0x7f673c0ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:56.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.872+0000 7f6741f80640 1 --2- 172.21.15.125:0/2795639922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f673c0cf160 0x7f673c0c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:56.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.872+0000 7f6741f80640 1 -- 172.21.15.125:0/2795639922 >> 172.21.15.125:0/2795639922 conn(0x7f673c06bfe0 msgr2=0x7f673c15b560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:56.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.872+0000 7f6741f80640 1 -- 172.21.15.125:0/2795639922 shutdown_connections 2023-10-30T12:57:56.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:56.872+0000 7f6741f80640 1 -- 172.21.15.125:0/2795639922 wait complete. 2023-10-30T12:57:56.898 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:57:58.899 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:57:59.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1d091d640 1 Processor -- start 2023-10-30T12:57:59.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1d091d640 1 -- start start 2023-10-30T12:57:59.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1d091d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe1cc15d6f0 0x7fe1cc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:59.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1d091d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe1cc0cf160 0x7fe1cc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:59.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1d091d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1cc0cb800 con 0x7fe1cc1538a0 2023-10-30T12:57:59.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1d091d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1cc0cb330 con 0x7fe1cc15d6f0 2023-10-30T12:57:59.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1d091d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc0cad70 con 0x7fe1cc0cf160 2023-10-30T12:57:59.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1c3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe1cc15d6f0 0x7fe1cc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:59.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1c3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe1cc15d6f0 0x7fe1cc15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35560/0 (socket says 172.21.15.125:35560) 2023-10-30T12:57:59.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1c3fff640 1 -- 172.21.15.125:0/2627525397 learned_addr learned my addr 172.21.15.125:0/2627525397 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:59.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1cad9d640 1 --2- 172.21.15.125:0/2627525397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe1cc0cf160 0x7fe1cc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:59.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1c3fff640 1 -- 172.21.15.125:0/2627525397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe1cc0cf160 msgr2=0x7fe1cc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:57:59.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1c3fff640 1 --2- 172.21.15.125:0/2627525397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe1cc0cf160 0x7fe1cc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:59.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1c3fff640 1 -- 172.21.15.125:0/2627525397 >> v1:172.21.15.136:6789/0 conn(0x7fe1cc1538a0 legacy=0x7fe1cc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:59.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.048+0000 7fe1c3fff640 1 -- 172.21.15.125:0/2627525397 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1cc0cb050 con 0x7fe1cc15d6f0 2023-10-30T12:57:59.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.052+0000 7fe1c3fff640 1 --2- 172.21.15.125:0/2627525397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1cc15d6f0 0x7fe1cc15a090 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4009870 tx=0x7fe1b401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a30469e26177f5bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:59.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.052+0000 7fe1c9d9b640 1 -- 172.21.15.125:0/2627525397 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b40a8a30 con 0x7fe1cc15d6f0 2023-10-30T12:57:59.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.052+0000 7fe1c9d9b640 1 -- 172.21.15.125:0/2627525397 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1b40a8b90 con 0x7fe1cc15d6f0 2023-10-30T12:57:59.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.052+0000 7fe1c9d9b640 1 -- 172.21.15.125:0/2627525397 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1b40b0440 con 0x7fe1cc15d6f0 2023-10-30T12:57:59.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.052+0000 7fe1d091d640 1 -- 172.21.15.125:0/2627525397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1cc15d6f0 msgr2=0x7fe1cc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:59.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.052+0000 7fe1d091d640 1 --2- 172.21.15.125:0/2627525397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1cc15d6f0 0x7fe1cc15a090 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7fe1b4009870 tx=0x7fe1b401ed30 comp rx=0 tx=0).stop 2023-10-30T12:57:59.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.052+0000 7fe1d091d640 1 -- 172.21.15.125:0/2627525397 shutdown_connections 2023-10-30T12:57:59.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.052+0000 7fe1d091d640 1 --2- 172.21.15.125:0/2627525397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe1cc0cf160 0x7fe1cc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:59.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.052+0000 7fe1d091d640 1 --2- 172.21.15.125:0/2627525397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1cc15d6f0 0x7fe1cc15a090 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:59.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.052+0000 7fe1d091d640 1 -- 172.21.15.125:0/2627525397 >> 172.21.15.125:0/2627525397 conn(0x7fe1cc06bfe0 msgr2=0x7fe1cc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:59.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.052+0000 7fe1d091d640 1 -- 172.21.15.125:0/2627525397 shutdown_connections 2023-10-30T12:57:59.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.052+0000 7fe1d091d640 1 -- 172.21.15.125:0/2627525397 wait complete. 2023-10-30T12:57:59.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1d091d640 1 Processor -- start 2023-10-30T12:57:59.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1d091d640 1 -- start start 2023-10-30T12:57:59.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1d091d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1cc0cf160 0x7fe1cc159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:59.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1d091d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1cc1538a0 0x7fe1cc0c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1d091d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe1cc15d6f0 0x7fe1cc1e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1d091d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc1e3bf0 con 0x7fe1cc0cf160 2023-10-30T12:57:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1d091d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc1580f0 con 0x7fe1cc1538a0 2023-10-30T12:57:59.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1d091d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1cc0c59a0 con 0x7fe1cc15d6f0 2023-10-30T12:57:59.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1ca59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1cc0cf160 0x7fe1cc159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:59.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1ca59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1cc0cf160 0x7fe1cc159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35568/0 (socket says 172.21.15.125:35568) 2023-10-30T12:57:59.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1ca59c640 1 -- 172.21.15.125:0/4051400433 learned_addr learned my addr 172.21.15.125:0/4051400433 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:57:59.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1c3fff640 1 --2- 172.21.15.125:0/4051400433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1cc1538a0 0x7fe1cc0c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:59.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1cad9d640 1 --2- 172.21.15.125:0/4051400433 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe1cc15d6f0 0x7fe1cc1e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:59.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1ca59c640 1 -- 172.21.15.125:0/4051400433 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe1cc15d6f0 msgr2=0x7fe1cc1e2a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:59.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1ca59c640 1 --2- 172.21.15.125:0/4051400433 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe1cc15d6f0 0x7fe1cc1e2a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:59.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1ca59c640 1 -- 172.21.15.125:0/4051400433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1cc1538a0 msgr2=0x7fe1cc0c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:59.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1ca59c640 1 --2- 172.21.15.125:0/4051400433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1cc1538a0 0x7fe1cc0c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:59.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1ca59c640 1 -- 172.21.15.125:0/4051400433 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b40094d0 con 0x7fe1cc0cf160 2023-10-30T12:57:59.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1ca59c640 1 --2- 172.21.15.125:0/4051400433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1cc0cf160 0x7fe1cc159300 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc00c500 tx=0x7fe1bc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:59.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1c2ffd640 1 -- 172.21.15.125:0/4051400433 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc005220 con 0x7fe1cc0cf160 2023-10-30T12:57:59.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1c2ffd640 1 -- 172.21.15.125:0/4051400433 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1bc005380 con 0x7fe1cc0cf160 2023-10-30T12:57:59.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1c2ffd640 1 -- 172.21.15.125:0/4051400433 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1bc007270 con 0x7fe1cc0cf160 2023-10-30T12:57:59.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1d091d640 1 -- 172.21.15.125:0/4051400433 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1cc1584d0 con 0x7fe1cc0cf160 2023-10-30T12:57:59.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.056+0000 7fe1d091d640 1 -- 172.21.15.125:0/4051400433 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1cc0c4f60 con 0x7fe1cc0cf160 2023-10-30T12:57:59.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.060+0000 7fe1d091d640 1 -- 172.21.15.125:0/4051400433 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1cc0b2a50 con 0x7fe1cc0cf160 2023-10-30T12:57:59.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.060+0000 7fe1c2ffd640 1 -- 172.21.15.125:0/4051400433 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe1bc0073d0 con 0x7fe1cc0cf160 2023-10-30T12:57:59.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.060+0000 7fe1c2ffd640 1 --2- 172.21.15.125:0/4051400433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe1ac0664d0 0x7fe1ac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:57:59.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.060+0000 7fe1c2ffd640 1 -- 172.21.15.125:0/4051400433 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7fe1bc083c60 con 0x7fe1cc0cf160 2023-10-30T12:57:59.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.060+0000 7fe1c3fff640 1 --2- 172.21.15.125:0/4051400433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe1ac0664d0 0x7fe1ac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:57:59.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.064+0000 7fe1c3fff640 1 --2- 172.21.15.125:0/4051400433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe1ac0664d0 0x7fe1ac068990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe1b40b2000 tx=0x7fe1b4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:57:59.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.064+0000 7fe1c2ffd640 1 -- 172.21.15.125:0/4051400433 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe1bc050b20 con 0x7fe1cc0cf160 2023-10-30T12:57:59.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.268+0000 7fe1d091d640 1 -- 172.21.15.125:0/4051400433 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1cc04e170 con 0x7fe1cc0cf160 2023-10-30T12:57:59.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.268+0000 7fe1c2ffd640 1 -- 172.21.15.125:0/4051400433 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 134 v134) v1 ==== 95+0+5723 (secure 0 0 0) 0x7fe1bc054140 con 0x7fe1cc0cf160 2023-10-30T12:57:59.269 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:57:59.270 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":134,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":134,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:57:50.441755+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:57:59.270 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 134 2023-10-30T12:57:59.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.272+0000 7fe1d091d640 1 -- 172.21.15.125:0/4051400433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe1ac0664d0 msgr2=0x7fe1ac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:59.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.272+0000 7fe1d091d640 1 --2- 172.21.15.125:0/4051400433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe1ac0664d0 0x7fe1ac068990 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fe1b40b2000 tx=0x7fe1b4009040 comp rx=0 tx=0).stop 2023-10-30T12:57:59.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.272+0000 7fe1d091d640 1 -- 172.21.15.125:0/4051400433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1cc0cf160 msgr2=0x7fe1cc159300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:57:59.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.272+0000 7fe1d091d640 1 --2- 172.21.15.125:0/4051400433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1cc0cf160 0x7fe1cc159300 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc00c500 tx=0x7fe1bc00c9c0 comp rx=0 tx=0).stop 2023-10-30T12:57:59.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.272+0000 7fe1d091d640 1 -- 172.21.15.125:0/4051400433 shutdown_connections 2023-10-30T12:57:59.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.272+0000 7fe1d091d640 1 --2- 172.21.15.125:0/4051400433 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe1ac0664d0 0x7fe1ac068990 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:59.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.272+0000 7fe1d091d640 1 --2- 172.21.15.125:0/4051400433 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe1cc15d6f0 0x7fe1cc1e2a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:59.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.272+0000 7fe1d091d640 1 --2- 172.21.15.125:0/4051400433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1cc1538a0 0x7fe1cc0c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:59.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.272+0000 7fe1d091d640 1 --2- 172.21.15.125:0/4051400433 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe1cc0cf160 0x7fe1cc159300 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:57:59.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.272+0000 7fe1d091d640 1 -- 172.21.15.125:0/4051400433 >> 172.21.15.125:0/4051400433 conn(0x7fe1cc06bfe0 msgr2=0x7fe1cc14ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:57:59.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.272+0000 7fe1d091d640 1 -- 172.21.15.125:0/4051400433 shutdown_connections 2023-10-30T12:57:59.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:57:59.272+0000 7fe1d091d640 1 -- 172.21.15.125:0/4051400433 wait complete. 2023-10-30T12:57:59.291 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:58:01.292 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:58:01.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.431+0000 7f8142f9d640 1 Processor -- start 2023-10-30T12:58:01.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.431+0000 7f8142f9d640 1 -- start start 2023-10-30T12:58:01.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.431+0000 7f8142f9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f813c0c6bd0 0x7f813c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:01.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.431+0000 7f8142f9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f813c071110 0x7f813c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:01.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.431+0000 7f8142f9d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c066cf0 con 0x7f813c06f7a0 2023-10-30T12:58:01.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.431+0000 7f8142f9d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813c0700c0 con 0x7f813c071110 2023-10-30T12:58:01.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.431+0000 7f8142f9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c0c70d0 con 0x7f813c0c6bd0 2023-10-30T12:58:01.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.435+0000 7f814153a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f813c071110 0x7f813c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:01.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.435+0000 7f814153a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f813c071110 0x7f813c158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37076/0 (socket says 172.21.15.125:37076) 2023-10-30T12:58:01.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.435+0000 7f814153a640 1 -- 172.21.15.125:0/1916923396 learned_addr learned my addr 172.21.15.125:0/1916923396 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:01.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.435+0000 7f8133fff640 1 --2- 172.21.15.125:0/1916923396 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f813c0c6bd0 0x7f813c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:01.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.435+0000 7f814153a640 1 -- 172.21.15.125:0/1916923396 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f813c0c6bd0 msgr2=0x7f813c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:01.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.435+0000 7f814153a640 1 --2- 172.21.15.125:0/1916923396 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f813c0c6bd0 0x7f813c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:01.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.435+0000 7f814153a640 1 -- 172.21.15.125:0/1916923396 >> v1:172.21.15.136:6789/0 conn(0x7f813c06f7a0 legacy=0x7f813c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:01.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.435+0000 7f814153a640 1 -- 172.21.15.125:0/1916923396 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c0c7390 con 0x7f813c071110 2023-10-30T12:58:01.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.435+0000 7f814153a640 1 --2- 172.21.15.125:0/1916923396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f813c071110 0x7f813c158110 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f8134009e50 tx=0x7f813401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11b288c77999f3f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:01.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.435+0000 7f81337fe640 1 -- 172.21.15.125:0/1916923396 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81340af400 con 0x7f813c071110 2023-10-30T12:58:01.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.435+0000 7f81337fe640 1 -- 172.21.15.125:0/1916923396 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81340ab050 con 0x7f813c071110 2023-10-30T12:58:01.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.435+0000 7f81337fe640 1 -- 172.21.15.125:0/1916923396 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81340af780 con 0x7f813c071110 2023-10-30T12:58:01.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 -- 172.21.15.125:0/1916923396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f813c071110 msgr2=0x7f813c158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:01.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 --2- 172.21.15.125:0/1916923396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f813c071110 0x7f813c158110 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f8134009e50 tx=0x7f813401f1f0 comp rx=0 tx=0).stop 2023-10-30T12:58:01.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 -- 172.21.15.125:0/1916923396 shutdown_connections 2023-10-30T12:58:01.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 --2- 172.21.15.125:0/1916923396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f813c071110 0x7f813c158110 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:01.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 --2- 172.21.15.125:0/1916923396 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f813c0c6bd0 0x7f813c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:01.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 -- 172.21.15.125:0/1916923396 >> 172.21.15.125:0/1916923396 conn(0x7f813c0c7910 msgr2=0x7f813c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:01.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 -- 172.21.15.125:0/1916923396 shutdown_connections 2023-10-30T12:58:01.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 -- 172.21.15.125:0/1916923396 wait complete. 2023-10-30T12:58:01.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 Processor -- start 2023-10-30T12:58:01.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 -- start start 2023-10-30T12:58:01.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f813c06f7a0 0x7f813c0c3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:01.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f813c071110 0x7f813c0ae640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:01.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f813c0c6bd0 0x7f813c0aeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:01.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c0af8f0 con 0x7f813c071110 2023-10-30T12:58:01.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c1ef380 con 0x7f813c0c6bd0 2023-10-30T12:58:01.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8142f9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c1ef8b0 con 0x7f813c06f7a0 2023-10-30T12:58:01.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8133fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f813c071110 0x7f813c0ae640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:01.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8133fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f813c071110 0x7f813c0ae640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37080/0 (socket says 172.21.15.125:37080) 2023-10-30T12:58:01.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8133fff640 1 -- 172.21.15.125:0/2128073192 learned_addr learned my addr 172.21.15.125:0/2128073192 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:01.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f8140d39640 1 --2- 172.21.15.125:0/2128073192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f813c06f7a0 0x7f813c0c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:01.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.439+0000 7f814153a640 1 --2- 172.21.15.125:0/2128073192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f813c0c6bd0 0x7f813c0aeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:01.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f8133fff640 1 -- 172.21.15.125:0/2128073192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f813c06f7a0 msgr2=0x7f813c0c3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:01.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f8133fff640 1 --2- 172.21.15.125:0/2128073192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f813c06f7a0 0x7f813c0c3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:01.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f8133fff640 1 -- 172.21.15.125:0/2128073192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f813c0c6bd0 msgr2=0x7f813c0aeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:01.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f8133fff640 1 --2- 172.21.15.125:0/2128073192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f813c0c6bd0 0x7f813c0aeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:01.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f8133fff640 1 -- 172.21.15.125:0/2128073192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8134009b30 con 0x7f813c071110 2023-10-30T12:58:01.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f8133fff640 1 --2- 172.21.15.125:0/2128073192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f813c071110 0x7f813c0ae640 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f812c00d800 tx=0x7f812c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:01.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f81317fa640 1 -- 172.21.15.125:0/2128073192 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c00b6c0 con 0x7f813c071110 2023-10-30T12:58:01.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f81317fa640 1 -- 172.21.15.125:0/2128073192 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c00b820 con 0x7f813c071110 2023-10-30T12:58:01.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f81317fa640 1 -- 172.21.15.125:0/2128073192 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c00bb90 con 0x7f813c071110 2023-10-30T12:58:01.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f8142f9d640 1 -- 172.21.15.125:0/2128073192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813c14f270 con 0x7f813c071110 2023-10-30T12:58:01.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f8142f9d640 1 -- 172.21.15.125:0/2128073192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813c14f510 con 0x7f813c071110 2023-10-30T12:58:01.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f8142f9d640 1 -- 172.21.15.125:0/2128073192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813c04e170 con 0x7f813c071110 2023-10-30T12:58:01.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f81317fa640 1 -- 172.21.15.125:0/2128073192 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f812c00bcf0 con 0x7f813c071110 2023-10-30T12:58:01.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f81317fa640 1 --2- 172.21.15.125:0/2128073192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f81280664d0 0x7f8128068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:01.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f81317fa640 1 -- 172.21.15.125:0/2128073192 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f812c0848d0 con 0x7f813c071110 2023-10-30T12:58:01.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.443+0000 7f8140d39640 1 --2- 172.21.15.125:0/2128073192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f81280664d0 0x7f8128068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:01.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.447+0000 7f8140d39640 1 --2- 172.21.15.125:0/2128073192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f81280664d0 0x7f8128068990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8138007fd0 tx=0x7f8138009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:01.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.447+0000 7f81317fa640 1 -- 172.21.15.125:0/2128073192 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f812c051780 con 0x7f813c071110 2023-10-30T12:58:01.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.635+0000 7f8142f9d640 1 -- 172.21.15.125:0/2128073192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f813c0c6160 con 0x7f813c071110 2023-10-30T12:58:01.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.635+0000 7f81317fa640 1 -- 172.21.15.125:0/2128073192 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f812c054da0 con 0x7f813c071110 2023-10-30T12:58:01.637 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:58:01.637 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:00.887562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:58:01.638 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 135 2023-10-30T12:58:01.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.639+0000 7f8142f9d640 1 -- 172.21.15.125:0/2128073192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f81280664d0 msgr2=0x7f8128068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:01.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.639+0000 7f8142f9d640 1 --2- 172.21.15.125:0/2128073192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f81280664d0 0x7f8128068990 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f8138007fd0 tx=0x7f8138009040 comp rx=0 tx=0).stop 2023-10-30T12:58:01.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.639+0000 7f8142f9d640 1 -- 172.21.15.125:0/2128073192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f813c071110 msgr2=0x7f813c0ae640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:01.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.639+0000 7f8142f9d640 1 --2- 172.21.15.125:0/2128073192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f813c071110 0x7f813c0ae640 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f812c00d800 tx=0x7f812c00dcc0 comp rx=0 tx=0).stop 2023-10-30T12:58:01.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.639+0000 7f8142f9d640 1 -- 172.21.15.125:0/2128073192 shutdown_connections 2023-10-30T12:58:01.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.639+0000 7f8142f9d640 1 --2- 172.21.15.125:0/2128073192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f81280664d0 0x7f8128068990 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:01.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.639+0000 7f8142f9d640 1 --2- 172.21.15.125:0/2128073192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f813c0c6bd0 0x7f813c0aeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:01.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.639+0000 7f8142f9d640 1 --2- 172.21.15.125:0/2128073192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f813c071110 0x7f813c0ae640 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:01.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.639+0000 7f8142f9d640 1 --2- 172.21.15.125:0/2128073192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f813c06f7a0 0x7f813c0c3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:01.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.639+0000 7f8142f9d640 1 -- 172.21.15.125:0/2128073192 >> 172.21.15.125:0/2128073192 conn(0x7f813c0c7910 msgr2=0x7f813c0c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:01.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.639+0000 7f8142f9d640 1 -- 172.21.15.125:0/2128073192 shutdown_connections 2023-10-30T12:58:01.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:01.639+0000 7f8142f9d640 1 -- 172.21.15.125:0/2128073192 wait complete. 2023-10-30T12:58:01.657 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:58:03.658 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:58:03.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.807+0000 7f6f0d8c9640 1 Processor -- start 2023-10-30T12:58:03.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.807+0000 7f6f0d8c9640 1 -- start start 2023-10-30T12:58:03.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f0d8c9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6f080c5f60 0x7f6f08071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:03.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f0d8c9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080700b0 con 0x7f6f080c6fe0 2023-10-30T12:58:03.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f0d8c9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080c6530 con 0x7f6f08153bf0 2023-10-30T12:58:03.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f0d8c9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f080c6810 con 0x7f6f080c5f60 2023-10-30T12:58:03.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6efe7fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6f080c6fe0 0x7f6f080cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37444/0 (socket says 172.21.15.125:37444) 2023-10-30T12:58:03.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6efe7fc640 1 -- 172.21.15.125:0/2494196459 learned_addr learned my addr 172.21.15.125:0/2494196459 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:58:03.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f077fe640 1 --2- 172.21.15.125:0/2494196459 >> v2:172.21.15.136:3300/0 conn(0x7f6f080c5f60 0x7f6f08071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:03.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 -- 172.21.15.125:0/2494196459 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3318895702 0 0) 0x7f6f080700b0 con 0x7f6f080c6fe0 2023-10-30T12:58:03.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 -- 172.21.15.125:0/2494196459 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef4003530 con 0x7f6f080c6fe0 2023-10-30T12:58:03.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 -- 172.21.15.125:0/2494196459 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 397800534 0 0) 0x7f6ef4003530 con 0x7f6f080c6fe0 2023-10-30T12:58:03.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 -- 172.21.15.125:0/2494196459 >> v2:172.21.15.136:3300/0 conn(0x7f6f080c5f60 msgr2=0x7f6f08071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:03.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 --2- 172.21.15.125:0/2494196459 >> v2:172.21.15.136:3300/0 conn(0x7f6f080c5f60 0x7f6f08071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:03.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 -- 172.21.15.125:0/2494196459 >> v1:172.21.15.136:6789/0 conn(0x7f6f08153bf0 legacy=0x7f6f080c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:03.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 -- 172.21.15.125:0/2494196459 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f08159b10 con 0x7f6f080c6fe0 2023-10-30T12:58:03.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 -- 172.21.15.125:0/2494196459 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f6ef8001680 con 0x7f6f080c6fe0 2023-10-30T12:58:03.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 -- 172.21.15.125:0/2494196459 >> v1:172.21.15.125:6789/0 conn(0x7f6f080c6fe0 legacy=0x7f6f080cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:03.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 --2- 172.21.15.125:0/2494196459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ef4003980 0x7f6ef4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:03.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 --2- 172.21.15.125:0/2494196459 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ef40047d0 0x7f6ef4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:03.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 --2- 172.21.15.125:0/2494196459 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ef4005190 0x7f6ef400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:03.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 -- 172.21.15.125:0/2494196459 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f080c6810 con 0x7f6ef40047d0 2023-10-30T12:58:03.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 -- 172.21.15.125:0/2494196459 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef400c040 con 0x7f6ef4003980 2023-10-30T12:58:03.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f067fc640 1 -- 172.21.15.125:0/2494196459 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ef400c200 con 0x7f6ef4005190 2023-10-30T12:58:03.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6efe7fc640 1 --2- 172.21.15.125:0/2494196459 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ef40047d0 0x7f6ef4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:03.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f077fe640 1 --2- 172.21.15.125:0/2494196459 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ef4005190 0x7f6ef400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:03.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f06ffd640 1 --2- 172.21.15.125:0/2494196459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ef4003980 0x7f6ef4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:03.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6efe7fc640 1 -- 172.21.15.125:0/2494196459 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ef4005190 msgr2=0x7f6ef400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:03.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6efe7fc640 1 --2- 172.21.15.125:0/2494196459 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ef4005190 0x7f6ef400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:03.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6efe7fc640 1 -- 172.21.15.125:0/2494196459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ef4003980 msgr2=0x7f6ef4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:03.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6efe7fc640 1 --2- 172.21.15.125:0/2494196459 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ef4003980 0x7f6ef4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:03.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6efe7fc640 1 -- 172.21.15.125:0/2494196459 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef4003530 con 0x7f6ef40047d0 2023-10-30T12:58:03.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.811+0000 7f6f077fe640 1 -- 172.21.15.125:0/2494196459 reap_dead start 2023-10-30T12:58:03.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.815+0000 7f6efe7fc640 1 --2- 172.21.15.125:0/2494196459 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ef40047d0 0x7f6ef4004c50 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800c1c0 tx=0x7f6ef801cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e6b0fb7f43aca5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:03.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.815+0000 7f6f067fc640 1 -- 172.21.15.125:0/2494196459 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8005ae0 con 0x7f6ef40047d0 2023-10-30T12:58:03.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.815+0000 7f6f067fc640 1 -- 172.21.15.125:0/2494196459 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef8005c60 con 0x7f6ef40047d0 2023-10-30T12:58:03.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.815+0000 7f6f067fc640 1 -- 172.21.15.125:0/2494196459 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef80a58a0 con 0x7f6ef40047d0 2023-10-30T12:58:03.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.815+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/2494196459 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ef40047d0 msgr2=0x7f6ef4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:03.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.815+0000 7f6f0d8c9640 1 --2- 172.21.15.125:0/2494196459 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ef40047d0 0x7f6ef4004c50 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800c1c0 tx=0x7f6ef801cba0 comp rx=0 tx=0).stop 2023-10-30T12:58:03.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.815+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/2494196459 shutdown_connections 2023-10-30T12:58:03.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.815+0000 7f6f0d8c9640 1 --2- 172.21.15.125:0/2494196459 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ef40047d0 0x7f6ef4004c50 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:03.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.815+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/2494196459 >> 172.21.15.125:0/2494196459 conn(0x7f6f0806c6e0 msgr2=0x7f6f08069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:03.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.815+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/2494196459 shutdown_connections 2023-10-30T12:58:03.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.815+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/2494196459 wait complete. 2023-10-30T12:58:03.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.815+0000 7f6f0d8c9640 1 Processor -- start 2023-10-30T12:58:03.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f0d8c9640 1 -- start start 2023-10-30T12:58:03.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f0d8c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ef40047d0 0x7f6f0814f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:03.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f0d8c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f080c5760 0x7f6f080c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:03.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f0d8c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f08150020 0x7f6f08149560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:03.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f0d8c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f08154d40 con 0x7f6ef40047d0 2023-10-30T12:58:03.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f0d8c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0814ea20 con 0x7f6f080c5760 2023-10-30T12:58:03.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f0d8c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0814e760 con 0x7f6f08150020 2023-10-30T12:58:03.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f06ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ef40047d0 0x7f6f0814f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:03.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f06ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ef40047d0 0x7f6f0814f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37096/0 (socket says 172.21.15.125:37096) 2023-10-30T12:58:03.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f06ffd640 1 -- 172.21.15.125:0/634096379 learned_addr learned my addr 172.21.15.125:0/634096379 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:03.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6efe7fc640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f080c5760 0x7f6f080c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:03.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f077fe640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f08150020 0x7f6f08149560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:03.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f06ffd640 1 -- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f08150020 msgr2=0x7f6f08149560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:03.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f06ffd640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f08150020 0x7f6f08149560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:03.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f06ffd640 1 -- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f080c5760 msgr2=0x7f6f080c4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:03.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f06ffd640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f080c5760 0x7f6f080c4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:03.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f06ffd640 1 -- 172.21.15.125:0/634096379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f081543a0 con 0x7f6ef40047d0 2023-10-30T12:58:03.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f077fe640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f08150020 0x7f6f08149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:58:03.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6efe7fc640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f080c5760 0x7f6f080c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:58:03.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f06ffd640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ef40047d0 0x7f6f0814f8b0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f6ef000e4d0 tx=0x7f6ef000e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:03.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f04ff9640 1 -- 172.21.15.125:0/634096379 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef00052e0 con 0x7f6ef40047d0 2023-10-30T12:58:03.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f04ff9640 1 -- 172.21.15.125:0/634096379 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef0005440 con 0x7f6ef40047d0 2023-10-30T12:58:03.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/634096379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f08154680 con 0x7f6ef40047d0 2023-10-30T12:58:03.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f04ff9640 1 -- 172.21.15.125:0/634096379 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef0003820 con 0x7f6ef40047d0 2023-10-30T12:58:03.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/634096379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f080c4940 con 0x7f6ef40047d0 2023-10-30T12:58:03.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.819+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/634096379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f08154fb0 con 0x7f6ef40047d0 2023-10-30T12:58:03.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.823+0000 7f6f04ff9640 1 -- 172.21.15.125:0/634096379 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6ef0003980 con 0x7f6ef40047d0 2023-10-30T12:58:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.823+0000 7f6f04ff9640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ee80664a0 0x7f6ee8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.823+0000 7f6f04ff9640 1 -- 172.21.15.125:0/634096379 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f6ef0084040 con 0x7f6ef40047d0 2023-10-30T12:58:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.823+0000 7f6efe7fc640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ee80664a0 0x7f6ee8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.823+0000 7f6efe7fc640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ee80664a0 0x7f6ee8068960 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80ab000 tx=0x7f6ef80b1040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:03.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:03.827+0000 7f6f04ff9640 1 -- 172.21.15.125:0/634096379 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6ef0051020 con 0x7f6ef40047d0 2023-10-30T12:58:04.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:04.007+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/634096379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f08159dd0 con 0x7f6ef40047d0 2023-10-30T12:58:04.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:04.007+0000 7f6f04ff9640 1 -- 172.21.15.125:0/634096379 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f6f08159dd0 con 0x7f6ef40047d0 2023-10-30T12:58:04.009 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:58:04.009 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:00.887562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:58:04.009 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 135 2023-10-30T12:58:04.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:04.011+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ee80664a0 msgr2=0x7f6ee8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:04.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:04.011+0000 7f6f0d8c9640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ee80664a0 0x7f6ee8068960 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80ab000 tx=0x7f6ef80b1040 comp rx=0 tx=0).stop 2023-10-30T12:58:04.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:04.011+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/634096379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ef40047d0 msgr2=0x7f6f0814f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:04.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:04.011+0000 7f6f0d8c9640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ef40047d0 0x7f6f0814f8b0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f6ef000e4d0 tx=0x7f6ef000e990 comp rx=0 tx=0).stop 2023-10-30T12:58:04.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:04.011+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/634096379 shutdown_connections 2023-10-30T12:58:04.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:04.011+0000 7f6f0d8c9640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ee80664a0 0x7f6ee8068960 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:04.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:04.011+0000 7f6f0d8c9640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f08150020 0x7f6f08149560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:04.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:04.011+0000 7f6f0d8c9640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f080c5760 0x7f6f080c4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:04.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:04.011+0000 7f6f0d8c9640 1 --2- 172.21.15.125:0/634096379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ef40047d0 0x7f6f0814f8b0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:04.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:04.011+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/634096379 >> 172.21.15.125:0/634096379 conn(0x7f6f0806c6e0 msgr2=0x7f6f080c94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:04.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:04.011+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/634096379 shutdown_connections 2023-10-30T12:58:04.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:04.011+0000 7f6f0d8c9640 1 -- 172.21.15.125:0/634096379 wait complete. 2023-10-30T12:58:04.029 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 135, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 135, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7413": { "addr": "172.21.15.125:6837/3906359913", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 3906359913, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 3906359913, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7413, "incarnation": 134, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 69, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:58:00.887562+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7413 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:58:06.031 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:58:06.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.175+0000 7f4e49cd6640 1 Processor -- start 2023-10-30T12:58:06.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.175+0000 7f4e49cd6640 1 -- start start 2023-10-30T12:58:06.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e49cd6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e440c6b80 0x7f4e440bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:06.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e49cd6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4e440c6f60 0x7f4e440c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:06.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e49cd6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e440c0a50 con 0x7f4e440c8430 2023-10-30T12:58:06.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e49cd6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e4406f6e0 con 0x7f4e440c6f60 2023-10-30T12:58:06.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e49cd6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4406f9c0 con 0x7f4e440c6b80 2023-10-30T12:58:06.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e43fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4e440c8430 0x7f4e4413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37450/0 (socket says 172.21.15.125:37450) 2023-10-30T12:58:06.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e43fff640 1 -- 172.21.15.125:0/3595312109 learned_addr learned my addr 172.21.15.125:0/3595312109 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:58:06.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e437fe640 1 --2- 172.21.15.125:0/3595312109 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e440c6b80 0x7f4e440bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:06.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e42ffd640 1 --2- 172.21.15.125:0/3595312109 >> v2:172.21.15.136:3300/0 conn(0x7f4e440c6f60 0x7f4e440c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:06.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 -- 172.21.15.125:0/3595312109 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390594479 0 0) 0x7f4e440c0a50 con 0x7f4e440c8430 2023-10-30T12:58:06.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 -- 172.21.15.125:0/3595312109 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e2c003530 con 0x7f4e440c8430 2023-10-30T12:58:06.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 -- 172.21.15.125:0/3595312109 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2776053155 0 0) 0x7f4e2c003530 con 0x7f4e440c8430 2023-10-30T12:58:06.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 -- 172.21.15.125:0/3595312109 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e440c6b80 msgr2=0x7f4e440bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:06.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 --2- 172.21.15.125:0/3595312109 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e440c6b80 0x7f4e440bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:06.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 -- 172.21.15.125:0/3595312109 >> v2:172.21.15.136:3300/0 conn(0x7f4e440c6f60 msgr2=0x7f4e440c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:06.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 --2- 172.21.15.125:0/3595312109 >> v2:172.21.15.136:3300/0 conn(0x7f4e440c6f60 0x7f4e440c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:06.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 -- 172.21.15.125:0/3595312109 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4406fca0 con 0x7f4e440c8430 2023-10-30T12:58:06.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 -- 172.21.15.125:0/3595312109 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f4e3c001a00 con 0x7f4e440c8430 2023-10-30T12:58:06.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 -- 172.21.15.125:0/3595312109 >> v1:172.21.15.125:6789/0 conn(0x7f4e440c8430 legacy=0x7f4e4413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:06.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 --2- 172.21.15.125:0/3595312109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e2c003980 0x7f4e2c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:06.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 --2- 172.21.15.125:0/3595312109 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e440c6b80 0x7f4e2c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:06.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 --2- 172.21.15.125:0/3595312109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e2c004d10 0x7f4e2c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:06.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 -- 172.21.15.125:0/3595312109 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4406f6e0 con 0x7f4e2c004d10 2023-10-30T12:58:06.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 -- 172.21.15.125:0/3595312109 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4406f9c0 con 0x7f4e2c003980 2023-10-30T12:58:06.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e427fc640 1 -- 172.21.15.125:0/3595312109 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c00dba0 con 0x7f4e440c6b80 2023-10-30T12:58:06.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e43fff640 1 --2- 172.21.15.125:0/3595312109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e2c004d10 0x7f4e2c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:06.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e437fe640 1 --2- 172.21.15.125:0/3595312109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e2c003980 0x7f4e2c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:06.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e42ffd640 1 --2- 172.21.15.125:0/3595312109 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e440c6b80 0x7f4e2c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:06.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e43fff640 1 -- 172.21.15.125:0/3595312109 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e440c6b80 msgr2=0x7f4e2c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:06.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e43fff640 1 --2- 172.21.15.125:0/3595312109 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e440c6b80 0x7f4e2c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:06.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e43fff640 1 -- 172.21.15.125:0/3595312109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e2c003980 msgr2=0x7f4e2c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:06.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e43fff640 1 --2- 172.21.15.125:0/3595312109 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e2c003980 0x7f4e2c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:06.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.179+0000 7f4e43fff640 1 -- 172.21.15.125:0/3595312109 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c003530 con 0x7f4e2c004d10 2023-10-30T12:58:06.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.183+0000 7f4e43fff640 1 --2- 172.21.15.125:0/3595312109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e2c004d10 0x7f4e2c005190 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00a860 tx=0x7f4e3c01d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc84f277a6849f51 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:06.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.183+0000 7f4e427fc640 1 -- 172.21.15.125:0/3595312109 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c0ad630 con 0x7f4e2c004d10 2023-10-30T12:58:06.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.183+0000 7f4e427fc640 1 -- 172.21.15.125:0/3595312109 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c0a8070 con 0x7f4e2c004d10 2023-10-30T12:58:06.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.183+0000 7f4e427fc640 1 -- 172.21.15.125:0/3595312109 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c0ad9f0 con 0x7f4e2c004d10 2023-10-30T12:58:06.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.183+0000 7f4e49cd6640 1 -- 172.21.15.125:0/3595312109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e2c004d10 msgr2=0x7f4e2c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:06.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.183+0000 7f4e49cd6640 1 --2- 172.21.15.125:0/3595312109 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e2c004d10 0x7f4e2c005190 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00a860 tx=0x7f4e3c01d360 comp rx=0 tx=0).stop 2023-10-30T12:58:06.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.183+0000 7f4e43fff640 1 -- 172.21.15.125:0/3595312109 reap_dead start 2023-10-30T12:58:06.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.183+0000 7f4e49cd6640 1 -- 172.21.15.125:0/3595312109 shutdown_connections 2023-10-30T12:58:06.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.183+0000 7f4e49cd6640 1 -- 172.21.15.125:0/3595312109 >> 172.21.15.125:0/3595312109 conn(0x7f4e440c7910 msgr2=0x7f4e440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:06.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.183+0000 7f4e49cd6640 1 -- 172.21.15.125:0/3595312109 shutdown_connections 2023-10-30T12:58:06.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.183+0000 7f4e49cd6640 1 -- 172.21.15.125:0/3595312109 wait complete. 2023-10-30T12:58:06.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e49cd6640 1 Processor -- start 2023-10-30T12:58:06.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e49cd6640 1 -- start start 2023-10-30T12:58:06.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e49cd6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e4413cd40 0x7f4e4413a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:06.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e49cd6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e4413d120 0x7f4e441ea7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:06.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e49cd6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e441ead00 0x7f4e440bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:06.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e49cd6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e441ee810 con 0x7f4e441ead00 2023-10-30T12:58:06.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e49cd6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e440c26d0 con 0x7f4e4413d120 2023-10-30T12:58:06.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e49cd6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4413b180 con 0x7f4e4413cd40 2023-10-30T12:58:06.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e43fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e441ead00 0x7f4e440bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:06.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e43fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e441ead00 0x7f4e440bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37128/0 (socket says 172.21.15.125:37128) 2023-10-30T12:58:06.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e43fff640 1 -- 172.21.15.125:0/431624633 learned_addr learned my addr 172.21.15.125:0/431624633 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:06.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e437fe640 1 --2- 172.21.15.125:0/431624633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e4413cd40 0x7f4e4413a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:06.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e42ffd640 1 --2- 172.21.15.125:0/431624633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e4413d120 0x7f4e441ea7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:06.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e43fff640 1 -- 172.21.15.125:0/431624633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e4413cd40 msgr2=0x7f4e4413a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:06.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e43fff640 1 --2- 172.21.15.125:0/431624633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e4413cd40 0x7f4e4413a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:06.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e43fff640 1 -- 172.21.15.125:0/431624633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e4413d120 msgr2=0x7f4e441ea7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:06.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e43fff640 1 --2- 172.21.15.125:0/431624633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e4413d120 0x7f4e441ea7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:06.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e43fff640 1 -- 172.21.15.125:0/431624633 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e441eeca0 con 0x7f4e441ead00 2023-10-30T12:58:06.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e43fff640 1 --2- 172.21.15.125:0/431624633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e441ead00 0x7f4e440bfec0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00a860 tx=0x7f4e2c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:06.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e2bfff640 1 -- 172.21.15.125:0/431624633 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c0019d0 con 0x7f4e441ead00 2023-10-30T12:58:06.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e2bfff640 1 -- 172.21.15.125:0/431624633 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c0ad430 con 0x7f4e441ead00 2023-10-30T12:58:06.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e49cd6640 1 -- 172.21.15.125:0/431624633 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4413aea0 con 0x7f4e441ead00 2023-10-30T12:58:06.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e2bfff640 1 -- 172.21.15.125:0/431624633 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c0ad7a0 con 0x7f4e441ead00 2023-10-30T12:58:06.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e49cd6640 1 -- 172.21.15.125:0/431624633 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e441ee1c0 con 0x7f4e441ead00 2023-10-30T12:58:06.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.187+0000 7f4e49cd6640 1 -- 172.21.15.125:0/431624633 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e441ee9c0 con 0x7f4e441ead00 2023-10-30T12:58:06.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.191+0000 7f4e2bfff640 1 -- 172.21.15.125:0/431624633 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4e3c0aa030 con 0x7f4e441ead00 2023-10-30T12:58:06.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.191+0000 7f4e2bfff640 1 --2- 172.21.15.125:0/431624633 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4e300664d0 0x7f4e30068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:06.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.191+0000 7f4e2bfff640 1 -- 172.21.15.125:0/431624633 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f4e3c11dfa0 con 0x7f4e441ead00 2023-10-30T12:58:06.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.191+0000 7f4e437fe640 1 --2- 172.21.15.125:0/431624633 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4e300664d0 0x7f4e30068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:06.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.191+0000 7f4e437fe640 1 --2- 172.21.15.125:0/431624633 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4e300664d0 0x7f4e30068990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009aa0 tx=0x7f4e34008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:06.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.195+0000 7f4e2bfff640 1 -- 172.21.15.125:0/431624633 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4e441ee9c0 con 0x7f4e441ead00 2023-10-30T12:58:06.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.379+0000 7f4e49cd6640 1 -- 172.21.15.125:0/431624633 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e441ee9c0 con 0x7f4e441ead00 2023-10-30T12:58:06.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.379+0000 7f4e2bfff640 1 -- 172.21.15.125:0/431624633 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f4e441ee9c0 con 0x7f4e441ead00 2023-10-30T12:58:06.381 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:58:06.381 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:00.887562+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:58:06.381 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 135 2023-10-30T12:58:06.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.383+0000 7f4e49cd6640 1 -- 172.21.15.125:0/431624633 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4e300664d0 msgr2=0x7f4e30068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.383+0000 7f4e49cd6640 1 --2- 172.21.15.125:0/431624633 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4e300664d0 0x7f4e30068990 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009aa0 tx=0x7f4e34008040 comp rx=0 tx=0).stop 2023-10-30T12:58:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.383+0000 7f4e49cd6640 1 -- 172.21.15.125:0/431624633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e441ead00 msgr2=0x7f4e440bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.383+0000 7f4e49cd6640 1 --2- 172.21.15.125:0/431624633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e441ead00 0x7f4e440bfec0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c00a860 tx=0x7f4e2c005800 comp rx=0 tx=0).stop 2023-10-30T12:58:06.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.383+0000 7f4e49cd6640 1 -- 172.21.15.125:0/431624633 shutdown_connections 2023-10-30T12:58:06.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.383+0000 7f4e49cd6640 1 --2- 172.21.15.125:0/431624633 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4e300664d0 0x7f4e30068990 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:06.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.383+0000 7f4e49cd6640 1 --2- 172.21.15.125:0/431624633 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4e441ead00 0x7f4e440bfec0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:06.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.383+0000 7f4e49cd6640 1 --2- 172.21.15.125:0/431624633 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4e4413d120 0x7f4e441ea7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:06.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.383+0000 7f4e49cd6640 1 --2- 172.21.15.125:0/431624633 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4e4413cd40 0x7f4e4413a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:06.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.383+0000 7f4e49cd6640 1 -- 172.21.15.125:0/431624633 >> 172.21.15.125:0/431624633 conn(0x7f4e440c7910 msgr2=0x7f4e44138da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:06.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.383+0000 7f4e49cd6640 1 -- 172.21.15.125:0/431624633 shutdown_connections 2023-10-30T12:58:06.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:06.383+0000 7f4e49cd6640 1 -- 172.21.15.125:0/431624633 wait complete. 2023-10-30T12:58:06.401 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:58:08.402 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:58:08.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650759e640 1 Processor -- start 2023-10-30T12:58:08.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650759e640 1 -- start start 2023-10-30T12:58:08.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650759e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f65080cefa0 0x7f65080cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:08.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650759e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f650815a6c0 con 0x7f650815cd50 2023-10-30T12:58:08.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650759e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65080ca6a0 con 0x7f65081538f0 2023-10-30T12:58:08.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650759e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65080cf8f0 con 0x7f65080cefa0 2023-10-30T12:58:08.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f6505d9b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f650815cd50 0x7f65080c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37464/0 (socket says 172.21.15.125:37464) 2023-10-30T12:58:08.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f6505d9b640 1 -- 172.21.15.125:0/1464412504 learned_addr learned my addr 172.21.15.125:0/1464412504 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:58:08.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f6506d9d640 1 --2- 172.21.15.125:0/1464412504 >> v2:172.21.15.136:3300/0 conn(0x7f65080cefa0 0x7f65080cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:08.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 -- 172.21.15.125:0/1464412504 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346928257 0 0) 0x7f650815a6c0 con 0x7f650815cd50 2023-10-30T12:58:08.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 -- 172.21.15.125:0/1464412504 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64f0003530 con 0x7f650815cd50 2023-10-30T12:58:08.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 -- 172.21.15.125:0/1464412504 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3005077893 0 0) 0x7f64f0003530 con 0x7f650815cd50 2023-10-30T12:58:08.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 -- 172.21.15.125:0/1464412504 >> v2:172.21.15.136:3300/0 conn(0x7f65080cefa0 msgr2=0x7f65080cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:08.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 --2- 172.21.15.125:0/1464412504 >> v2:172.21.15.136:3300/0 conn(0x7f65080cefa0 0x7f65080cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:08.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 -- 172.21.15.125:0/1464412504 >> v1:172.21.15.136:6789/0 conn(0x7f65081538f0 legacy=0x7f6508159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:08.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 -- 172.21.15.125:0/1464412504 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f650815d300 con 0x7f650815cd50 2023-10-30T12:58:08.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 -- 172.21.15.125:0/1464412504 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f64fc001680 con 0x7f650815cd50 2023-10-30T12:58:08.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 -- 172.21.15.125:0/1464412504 >> v1:172.21.15.125:6789/0 conn(0x7f650815cd50 legacy=0x7f65080c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:08.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 --2- 172.21.15.125:0/1464412504 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f64f0003980 0x7f64f0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:08.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 --2- 172.21.15.125:0/1464412504 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f00047d0 0x7f64f0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:08.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 --2- 172.21.15.125:0/1464412504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64f0005190 0x7f64f000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:08.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 -- 172.21.15.125:0/1464412504 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65080cf8f0 con 0x7f64f00047d0 2023-10-30T12:58:08.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 -- 172.21.15.125:0/1464412504 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f000c040 con 0x7f64f0005190 2023-10-30T12:58:08.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.551+0000 7f650559a640 1 -- 172.21.15.125:0/1464412504 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f000c200 con 0x7f64f0003980 2023-10-30T12:58:08.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.555+0000 7f6505d9b640 1 --2- 172.21.15.125:0/1464412504 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f00047d0 0x7f64f0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:08.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.555+0000 7f650659c640 1 --2- 172.21.15.125:0/1464412504 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f64f0003980 0x7f64f0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:08.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.555+0000 7f6506d9d640 1 --2- 172.21.15.125:0/1464412504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64f0005190 0x7f64f000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:08.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.555+0000 7f6505d9b640 1 -- 172.21.15.125:0/1464412504 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f64f0003980 msgr2=0x7f64f0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:08.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.555+0000 7f6505d9b640 1 --2- 172.21.15.125:0/1464412504 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f64f0003980 0x7f64f0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:08.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.555+0000 7f6505d9b640 1 -- 172.21.15.125:0/1464412504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64f0005190 msgr2=0x7f64f000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:08.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.555+0000 7f6505d9b640 1 --2- 172.21.15.125:0/1464412504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64f0005190 0x7f64f000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:08.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.555+0000 7f6505d9b640 1 -- 172.21.15.125:0/1464412504 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f0003530 con 0x7f64f00047d0 2023-10-30T12:58:08.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.555+0000 7f6506d9d640 1 --2- 172.21.15.125:0/1464412504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64f0005190 0x7f64f000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:58:08.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.555+0000 7f6506d9d640 1 -- 172.21.15.125:0/1464412504 reap_dead start 2023-10-30T12:58:08.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.555+0000 7f6505d9b640 1 --2- 172.21.15.125:0/1464412504 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f00047d0 0x7f64f0004c50 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00c240 tx=0x7f64fc01cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c83c4d03bcda6b64 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:08.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.555+0000 7f650559a640 1 -- 172.21.15.125:0/1464412504 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64fc0a5b00 con 0x7f64f00047d0 2023-10-30T12:58:08.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.555+0000 7f650559a640 1 -- 172.21.15.125:0/1464412504 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64fc0a5c60 con 0x7f64f00047d0 2023-10-30T12:58:08.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.555+0000 7f650559a640 1 -- 172.21.15.125:0/1464412504 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64fc005b20 con 0x7f64f00047d0 2023-10-30T12:58:08.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 -- 172.21.15.125:0/1464412504 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f00047d0 msgr2=0x7f64f0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:08.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 --2- 172.21.15.125:0/1464412504 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f00047d0 0x7f64f0004c50 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00c240 tx=0x7f64fc01cc00 comp rx=0 tx=0).stop 2023-10-30T12:58:08.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 -- 172.21.15.125:0/1464412504 shutdown_connections 2023-10-30T12:58:08.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 --2- 172.21.15.125:0/1464412504 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f00047d0 0x7f64f0004c50 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:08.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 -- 172.21.15.125:0/1464412504 >> 172.21.15.125:0/1464412504 conn(0x7f650806a640 msgr2=0x7f6508065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:08.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 -- 172.21.15.125:0/1464412504 shutdown_connections 2023-10-30T12:58:08.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 -- 172.21.15.125:0/1464412504 wait complete. 2023-10-30T12:58:08.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 Processor -- start 2023-10-30T12:58:08.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 -- start start 2023-10-30T12:58:08.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f00047d0 0x7f6508156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:08.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65080b3de0 0x7f65080b3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:08.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65080b4230 0x7f6508158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:08.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65080b4cd0 con 0x7f64f00047d0 2023-10-30T12:58:08.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6508156080 con 0x7f65080b3de0 2023-10-30T12:58:08.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650759e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6508155b10 con 0x7f65080b4230 2023-10-30T12:58:08.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f00047d0 0x7f6508156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:08.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f00047d0 0x7f6508156510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37150/0 (socket says 172.21.15.125:37150) 2023-10-30T12:58:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650659c640 1 -- 172.21.15.125:0/1287839165 learned_addr learned my addr 172.21.15.125:0/1287839165 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f6505d9b640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65080b3de0 0x7f65080b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f6506d9d640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65080b4230 0x7f6508158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650659c640 1 -- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65080b4230 msgr2=0x7f6508158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:08.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650659c640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65080b4230 0x7f6508158860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:08.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650659c640 1 -- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65080b3de0 msgr2=0x7f65080b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:08.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650659c640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65080b3de0 0x7f65080b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:08.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f650659c640 1 -- 172.21.15.125:0/1287839165 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65080b4f20 con 0x7f64f00047d0 2023-10-30T12:58:08.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.559+0000 7f6505d9b640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65080b3de0 0x7f65080b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:58:08.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.563+0000 7f650659c640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f00047d0 0x7f6508156510 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f64f800f7f0 tx=0x7f64f800fcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:08.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.563+0000 7f64f6ffd640 1 -- 172.21.15.125:0/1287839165 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f80050d0 con 0x7f64f00047d0 2023-10-30T12:58:08.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.563+0000 7f64f6ffd640 1 -- 172.21.15.125:0/1287839165 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64f8005230 con 0x7f64f00047d0 2023-10-30T12:58:08.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.563+0000 7f64f6ffd640 1 -- 172.21.15.125:0/1287839165 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f800d6d0 con 0x7f64f00047d0 2023-10-30T12:58:08.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.563+0000 7f650759e640 1 -- 172.21.15.125:0/1287839165 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65080b5200 con 0x7f64f00047d0 2023-10-30T12:58:08.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.563+0000 7f650759e640 1 -- 172.21.15.125:0/1287839165 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650815c690 con 0x7f64f00047d0 2023-10-30T12:58:08.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.563+0000 7f650759e640 1 -- 172.21.15.125:0/1287839165 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f650804e170 con 0x7f64f00047d0 2023-10-30T12:58:08.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.563+0000 7f64f6ffd640 1 -- 172.21.15.125:0/1287839165 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64f800d830 con 0x7f64f00047d0 2023-10-30T12:58:08.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.563+0000 7f64f6ffd640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64e80664a0 0x7f64e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:08.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.563+0000 7f64f6ffd640 1 -- 172.21.15.125:0/1287839165 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f64f8084350 con 0x7f64f00047d0 2023-10-30T12:58:08.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.563+0000 7f6505d9b640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64e80664a0 0x7f64e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:08.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.567+0000 7f6505d9b640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64e80664a0 0x7f64e8068960 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f64fc0ab000 tx=0x7f64fc006040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:08.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.567+0000 7f64f6ffd640 1 -- 172.21.15.125:0/1287839165 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64f8051290 con 0x7f64f00047d0 2023-10-30T12:58:08.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.751+0000 7f650759e640 1 -- 172.21.15.125:0/1287839165 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65080b2290 con 0x7f64f00047d0 2023-10-30T12:58:08.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.751+0000 7f64f6ffd640 1 -- 172.21.15.125:0/1287839165 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f64f80548b0 con 0x7f64f00047d0 2023-10-30T12:58:08.753 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:58:08.753 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:06.512188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:58:08.754 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 136 2023-10-30T12:58:08.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.755+0000 7f650759e640 1 -- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64e80664a0 msgr2=0x7f64e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:08.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.755+0000 7f650759e640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64e80664a0 0x7f64e8068960 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f64fc0ab000 tx=0x7f64fc006040 comp rx=0 tx=0).stop 2023-10-30T12:58:08.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.755+0000 7f650759e640 1 -- 172.21.15.125:0/1287839165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f00047d0 msgr2=0x7f6508156510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:08.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.755+0000 7f650759e640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f00047d0 0x7f6508156510 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7f64f800f7f0 tx=0x7f64f800fcb0 comp rx=0 tx=0).stop 2023-10-30T12:58:08.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.755+0000 7f650759e640 1 -- 172.21.15.125:0/1287839165 shutdown_connections 2023-10-30T12:58:08.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.755+0000 7f650759e640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64e80664a0 0x7f64e8068960 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:08.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.755+0000 7f650759e640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65080b4230 0x7f6508158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:08.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.755+0000 7f650759e640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65080b3de0 0x7f65080b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:08.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.755+0000 7f650759e640 1 --2- 172.21.15.125:0/1287839165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64f00047d0 0x7f6508156510 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:08.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.755+0000 7f650759e640 1 -- 172.21.15.125:0/1287839165 >> 172.21.15.125:0/1287839165 conn(0x7f650806a640 msgr2=0x7f6508065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:08.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.755+0000 7f650759e640 1 -- 172.21.15.125:0/1287839165 shutdown_connections 2023-10-30T12:58:08.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:08.755+0000 7f650759e640 1 -- 172.21.15.125:0/1287839165 wait complete. 2023-10-30T12:58:08.773 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:58:10.775 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:58:10.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.919+0000 7f3dd5bf8640 1 Processor -- start 2023-10-30T12:58:10.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.919+0000 7f3dd5bf8640 1 -- start start 2023-10-30T12:58:10.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dd5bf8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3dd0071110 0x7f3dd006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:10.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dd5bf8640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd0066cf0 con 0x7f3dd00c6bd0 2023-10-30T12:58:10.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dd5bf8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd00c70d0 con 0x7f3dd006f7a0 2023-10-30T12:58:10.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dd5bf8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd00700c0 con 0x7f3dd0071110 2023-10-30T12:58:10.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dceffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3dd00c6bd0 0x7f3dd0158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58132/0 (socket says 172.21.15.125:58132) 2023-10-30T12:58:10.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dceffd640 1 -- 172.21.15.125:0/2256983553 learned_addr learned my addr 172.21.15.125:0/2256983553 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:58:10.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dcffff640 1 --2- 172.21.15.125:0/2256983553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3dd0071110 0x7f3dd006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:10.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 -- 172.21.15.125:0/2256983553 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47952736 0 0) 0x7f3dd0066cf0 con 0x7f3dd00c6bd0 2023-10-30T12:58:10.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 -- 172.21.15.125:0/2256983553 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db8003530 con 0x7f3dd00c6bd0 2023-10-30T12:58:10.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 -- 172.21.15.125:0/2256983553 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2893674904 0 0) 0x7f3db8003530 con 0x7f3dd00c6bd0 2023-10-30T12:58:10.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 -- 172.21.15.125:0/2256983553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3dd0071110 msgr2=0x7f3dd006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:10.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 --2- 172.21.15.125:0/2256983553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3dd0071110 0x7f3dd006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:10.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 -- 172.21.15.125:0/2256983553 >> v1:172.21.15.136:6789/0 conn(0x7f3dd006f7a0 legacy=0x7f3dd0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:10.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 -- 172.21.15.125:0/2256983553 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd00c7390 con 0x7f3dd00c6bd0 2023-10-30T12:58:10.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 -- 172.21.15.125:0/2256983553 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f3dc4001680 con 0x7f3dd00c6bd0 2023-10-30T12:58:10.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 -- 172.21.15.125:0/2256983553 >> v1:172.21.15.125:6789/0 conn(0x7f3dd00c6bd0 legacy=0x7f3dd0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:10.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 --2- 172.21.15.125:0/2256983553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3db80039b0 0x7f3db8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:10.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 --2- 172.21.15.125:0/2256983553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3dd0071110 0x7f3db80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:10.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 --2- 172.21.15.125:0/2256983553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3db8004d10 0x7f3db8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:10.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 -- 172.21.15.125:0/2256983553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd00700c0 con 0x7f3db8004d10 2023-10-30T12:58:10.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 -- 172.21.15.125:0/2256983553 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db800bad0 con 0x7f3db80039b0 2023-10-30T12:58:10.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dce7fc640 1 -- 172.21.15.125:0/2256983553 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db800bc90 con 0x7f3dd0071110 2023-10-30T12:58:10.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dcffff640 1 --2- 172.21.15.125:0/2256983553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3db8004d10 0x7f3db8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:10.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dceffd640 1 --2- 172.21.15.125:0/2256983553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3dd0071110 0x7f3db80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:10.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dcf7fe640 1 --2- 172.21.15.125:0/2256983553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3db80039b0 0x7f3db8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:10.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dcffff640 1 -- 172.21.15.125:0/2256983553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3dd0071110 msgr2=0x7f3db80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:10.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dcffff640 1 --2- 172.21.15.125:0/2256983553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3dd0071110 0x7f3db80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:10.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dcffff640 1 -- 172.21.15.125:0/2256983553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3db80039b0 msgr2=0x7f3db8003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:10.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dcffff640 1 --2- 172.21.15.125:0/2256983553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3db80039b0 0x7f3db8003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:10.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dcffff640 1 -- 172.21.15.125:0/2256983553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db8003530 con 0x7f3db8004d10 2023-10-30T12:58:10.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dceffd640 1 --2- 172.21.15.125:0/2256983553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3dd0071110 0x7f3db80047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:58:10.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.923+0000 7f3dcf7fe640 1 --2- 172.21.15.125:0/2256983553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3db80039b0 0x7f3db8003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:58:10.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.927+0000 7f3dcffff640 1 --2- 172.21.15.125:0/2256983553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3db8004d10 0x7f3db8005190 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8009f90 tx=0x7f3dc801cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efd38d4214736337 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:10.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.927+0000 7f3dce7fc640 1 -- 172.21.15.125:0/2256983553 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc80a5c30 con 0x7f3db8004d10 2023-10-30T12:58:10.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.927+0000 7f3dce7fc640 1 -- 172.21.15.125:0/2256983553 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc800f040 con 0x7f3db8004d10 2023-10-30T12:58:10.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.927+0000 7f3dce7fc640 1 -- 172.21.15.125:0/2256983553 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc80ad440 con 0x7f3db8004d10 2023-10-30T12:58:10.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.927+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2256983553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3db8004d10 msgr2=0x7f3db8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:10.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.927+0000 7f3dd5bf8640 1 --2- 172.21.15.125:0/2256983553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3db8004d10 0x7f3db8005190 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f3dc8009f90 tx=0x7f3dc801cc20 comp rx=0 tx=0).stop 2023-10-30T12:58:10.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.927+0000 7f3dcffff640 1 -- 172.21.15.125:0/2256983553 reap_dead start 2023-10-30T12:58:10.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.927+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2256983553 shutdown_connections 2023-10-30T12:58:10.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.927+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2256983553 >> 172.21.15.125:0/2256983553 conn(0x7f3dd00c7910 msgr2=0x7f3dd00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:10.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.927+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2256983553 shutdown_connections 2023-10-30T12:58:10.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.927+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2256983553 wait complete. 2023-10-30T12:58:10.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.927+0000 7f3dd5bf8640 1 Processor -- start 2023-10-30T12:58:10.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.927+0000 7f3dd5bf8640 1 -- start start 2023-10-30T12:58:10.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dd5bf8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3dd0152380 0x7f3dd0154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:10.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dd5bf8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dd0152760 0x7f3dd0156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:10.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dd5bf8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dd01504a0 0x7f3dd0157080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:10.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dd5bf8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd01ed760 con 0x7f3dd01504a0 2023-10-30T12:58:10.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dd5bf8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd0152c20 con 0x7f3dd0152760 2023-10-30T12:58:10.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dd5bf8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd0150a00 con 0x7f3dd0152380 2023-10-30T12:58:10.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dcffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dd01504a0 0x7f3dd0157080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:10.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dcffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dd01504a0 0x7f3dd0157080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56378/0 (socket says 172.21.15.125:56378) 2023-10-30T12:58:10.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dcffff640 1 -- 172.21.15.125:0/2667701272 learned_addr learned my addr 172.21.15.125:0/2667701272 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:10.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dcf7fe640 1 --2- 172.21.15.125:0/2667701272 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3dd0152380 0x7f3dd0154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:10.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dceffd640 1 --2- 172.21.15.125:0/2667701272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dd0152760 0x7f3dd0156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:10.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dcffff640 1 -- 172.21.15.125:0/2667701272 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3dd0152380 msgr2=0x7f3dd0154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:10.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dcffff640 1 --2- 172.21.15.125:0/2667701272 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3dd0152380 0x7f3dd0154ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:10.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dcffff640 1 -- 172.21.15.125:0/2667701272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dd0152760 msgr2=0x7f3dd0156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:10.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dcffff640 1 --2- 172.21.15.125:0/2667701272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dd0152760 0x7f3dd0156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:10.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dcffff640 1 -- 172.21.15.125:0/2667701272 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd01ece30 con 0x7f3dd01504a0 2023-10-30T12:58:10.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dcffff640 1 --2- 172.21.15.125:0/2667701272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dd01504a0 0x7f3dd0157080 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f3dc80b0040 tx=0x7f3dc8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:10.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3db7fff640 1 -- 172.21.15.125:0/2667701272 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc80039a0 con 0x7f3dd01504a0 2023-10-30T12:58:10.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3db7fff640 1 -- 172.21.15.125:0/2667701272 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc8003b00 con 0x7f3dd01504a0 2023-10-30T12:58:10.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2667701272 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd01ed110 con 0x7f3dd01504a0 2023-10-30T12:58:10.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3db7fff640 1 -- 172.21.15.125:0/2667701272 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc80ae890 con 0x7f3dd01504a0 2023-10-30T12:58:10.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2667701272 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd01ecae0 con 0x7f3dd01504a0 2023-10-30T12:58:10.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2667701272 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd01ed8d0 con 0x7f3dd01504a0 2023-10-30T12:58:10.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.931+0000 7f3db7fff640 1 -- 172.21.15.125:0/2667701272 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3dc80b4040 con 0x7f3dd01504a0 2023-10-30T12:58:10.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.935+0000 7f3db7fff640 1 --2- 172.21.15.125:0/2667701272 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3dbc0664d0 0x7f3dbc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:10.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.935+0000 7f3db7fff640 1 -- 172.21.15.125:0/2667701272 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f3dc811def0 con 0x7f3dd01504a0 2023-10-30T12:58:10.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.935+0000 7f3dcf7fe640 1 --2- 172.21.15.125:0/2667701272 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3dbc0664d0 0x7f3dbc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:10.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.935+0000 7f3dcf7fe640 1 --2- 172.21.15.125:0/2667701272 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3dbc0664d0 0x7f3dbc068990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0005450 tx=0x7f3dc0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:10.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:10.939+0000 7f3db7fff640 1 -- 172.21.15.125:0/2667701272 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3dc80eae50 con 0x7f3dd01504a0 2023-10-30T12:58:11.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:11.119+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2667701272 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3dd004e170 con 0x7f3dd01504a0 2023-10-30T12:58:11.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:11.119+0000 7f3db7fff640 1 -- 172.21.15.125:0/2667701272 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f3dc80be090 con 0x7f3dd01504a0 2023-10-30T12:58:11.120 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:58:11.121 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:06.512188+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:58:11.121 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 136 2023-10-30T12:58:11.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:11.119+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2667701272 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3dbc0664d0 msgr2=0x7f3dbc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:11.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:11.119+0000 7f3dd5bf8640 1 --2- 172.21.15.125:0/2667701272 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3dbc0664d0 0x7f3dbc068990 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3dc0005450 tx=0x7f3dc0009040 comp rx=0 tx=0).stop 2023-10-30T12:58:11.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:11.119+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2667701272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dd01504a0 msgr2=0x7f3dd0157080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:11.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:11.119+0000 7f3dd5bf8640 1 --2- 172.21.15.125:0/2667701272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dd01504a0 0x7f3dd0157080 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f3dc80b0040 tx=0x7f3dc8009750 comp rx=0 tx=0).stop 2023-10-30T12:58:11.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:11.123+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2667701272 shutdown_connections 2023-10-30T12:58:11.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:11.123+0000 7f3dd5bf8640 1 --2- 172.21.15.125:0/2667701272 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3dbc0664d0 0x7f3dbc068990 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:11.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:11.123+0000 7f3dd5bf8640 1 --2- 172.21.15.125:0/2667701272 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3dd01504a0 0x7f3dd0157080 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:11.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:11.123+0000 7f3dd5bf8640 1 --2- 172.21.15.125:0/2667701272 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3dd0152760 0x7f3dd0156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:11.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:11.123+0000 7f3dd5bf8640 1 --2- 172.21.15.125:0/2667701272 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3dd0152380 0x7f3dd0154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:11.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:11.123+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2667701272 >> 172.21.15.125:0/2667701272 conn(0x7f3dd00c7910 msgr2=0x7f3dd00c8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:11.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:11.123+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2667701272 shutdown_connections 2023-10-30T12:58:11.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:11.123+0000 7f3dd5bf8640 1 -- 172.21.15.125:0/2667701272 wait complete. 2023-10-30T12:58:11.141 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:58:13.142 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:58:13.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.287+0000 7f7ef293e640 1 Processor -- start 2023-10-30T12:58:13.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.287+0000 7f7ef293e640 1 -- start start 2023-10-30T12:58:13.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef293e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7eec0c6b80 0x7f7eec0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:13.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef293e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7eec0c6f60 0x7f7eec0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:13.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef293e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7eec0c8430 0x7f7eec13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:13.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef293e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eec0c0a50 con 0x7f7eec0c8430 2023-10-30T12:58:13.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef293e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eec06f6e0 con 0x7f7eec0c6f60 2023-10-30T12:58:13.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef293e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec06f9c0 con 0x7f7eec0c6b80 2023-10-30T12:58:13.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef0edb640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7eec0c8430 0x7f7eec13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:13.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef0edb640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7eec0c8430 0x7f7eec13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56390/0 (socket says 172.21.15.125:56390) 2023-10-30T12:58:13.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef0edb640 1 -- 172.21.15.125:0/1056789387 learned_addr learned my addr 172.21.15.125:0/1056789387 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:13.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7eeb7fe640 1 --2- 172.21.15.125:0/1056789387 >> v2:172.21.15.136:3300/0 conn(0x7f7eec0c6f60 0x7f7eec0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:13.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7eebfff640 1 --2- 172.21.15.125:0/1056789387 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7eec0c6b80 0x7f7eec0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:13.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef0edb640 1 -- 172.21.15.125:0/1056789387 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7eec0c6b80 msgr2=0x7f7eec0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:13.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef0edb640 1 --2- 172.21.15.125:0/1056789387 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7eec0c6b80 0x7f7eec0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:13.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef0edb640 1 -- 172.21.15.125:0/1056789387 >> v2:172.21.15.136:3300/0 conn(0x7f7eec0c6f60 msgr2=0x7f7eec0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:13.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef0edb640 1 --2- 172.21.15.125:0/1056789387 >> v2:172.21.15.136:3300/0 conn(0x7f7eec0c6f60 0x7f7eec0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:13.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef0edb640 1 -- 172.21.15.125:0/1056789387 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec06fca0 con 0x7f7eec0c8430 2023-10-30T12:58:13.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.291+0000 7f7ef0edb640 1 --2- 172.21.15.125:0/1056789387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7eec0c8430 0x7f7eec13ed90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f7eec06c830 tx=0x7f7ee401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170fc92bbe83cfa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:13.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7eeaffd640 1 -- 172.21.15.125:0/1056789387 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee40a5760 con 0x7f7eec0c8430 2023-10-30T12:58:13.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7eeaffd640 1 -- 172.21.15.125:0/1056789387 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee40a58c0 con 0x7f7eec0c8430 2023-10-30T12:58:13.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7eeaffd640 1 -- 172.21.15.125:0/1056789387 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee40a5c50 con 0x7f7eec0c8430 2023-10-30T12:58:13.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 -- 172.21.15.125:0/1056789387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7eec0c8430 msgr2=0x7f7eec13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:13.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 --2- 172.21.15.125:0/1056789387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7eec0c8430 0x7f7eec13ed90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f7eec06c830 tx=0x7f7ee401c9d0 comp rx=0 tx=0).stop 2023-10-30T12:58:13.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 -- 172.21.15.125:0/1056789387 shutdown_connections 2023-10-30T12:58:13.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 --2- 172.21.15.125:0/1056789387 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7eec0c8430 0x7f7eec13ed90 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:13.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 --2- 172.21.15.125:0/1056789387 >> v2:172.21.15.136:3300/0 conn(0x7f7eec0c6f60 0x7f7eec0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:13.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 --2- 172.21.15.125:0/1056789387 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7eec0c6b80 0x7f7eec0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:13.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 -- 172.21.15.125:0/1056789387 >> 172.21.15.125:0/1056789387 conn(0x7f7eec0c7910 msgr2=0x7f7eec0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:13.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 -- 172.21.15.125:0/1056789387 shutdown_connections 2023-10-30T12:58:13.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 -- 172.21.15.125:0/1056789387 wait complete. 2023-10-30T12:58:13.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 Processor -- start 2023-10-30T12:58:13.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 -- start start 2023-10-30T12:58:13.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7eec0c6b80 0x7f7eec13e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:13.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7eec0c6f60 0x7f7eec0c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:13.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7eec0c8430 0x7f7eec0c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:13.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec0c8e20 con 0x7f7eec0c8430 2023-10-30T12:58:13.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec1efdb0 con 0x7f7eec0c6b80 2023-10-30T12:58:13.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.295+0000 7f7ef293e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec1f0090 con 0x7f7eec0c6f60 2023-10-30T12:58:13.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ef0edb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7eec0c8430 0x7f7eec0c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:13.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ef0edb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7eec0c8430 0x7f7eec0c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56406/0 (socket says 172.21.15.125:56406) 2023-10-30T12:58:13.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ef0edb640 1 -- 172.21.15.125:0/1455891154 learned_addr learned my addr 172.21.15.125:0/1455891154 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:13.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7eebfff640 1 --2- 172.21.15.125:0/1455891154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7eec0c6b80 0x7f7eec13e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:13.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7eeb7fe640 1 --2- 172.21.15.125:0/1455891154 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7eec0c6f60 0x7f7eec0c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:13.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ef0edb640 1 -- 172.21.15.125:0/1455891154 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7eec0c6f60 msgr2=0x7f7eec0c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:13.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ef0edb640 1 --2- 172.21.15.125:0/1455891154 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7eec0c6f60 0x7f7eec0c92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:13.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ef0edb640 1 -- 172.21.15.125:0/1455891154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7eec0c6b80 msgr2=0x7f7eec13e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:13.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ef0edb640 1 --2- 172.21.15.125:0/1455891154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7eec0c6b80 0x7f7eec13e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:13.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ef0edb640 1 -- 172.21.15.125:0/1455891154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee4009b30 con 0x7f7eec0c8430 2023-10-30T12:58:13.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ef0edb640 1 --2- 172.21.15.125:0/1455891154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7eec0c8430 0x7f7eec0c97f0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40af040 tx=0x7f7ee4019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:13.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ee8ff9640 1 -- 172.21.15.125:0/1455891154 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee40b8880 con 0x7f7eec0c8430 2023-10-30T12:58:13.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ee8ff9640 1 -- 172.21.15.125:0/1455891154 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee40b8a00 con 0x7f7eec0c8430 2023-10-30T12:58:13.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ef293e640 1 -- 172.21.15.125:0/1455891154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eec1f0370 con 0x7f7eec0c8430 2023-10-30T12:58:13.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ee8ff9640 1 -- 172.21.15.125:0/1455891154 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee40b8db0 con 0x7f7eec0c8430 2023-10-30T12:58:13.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ef293e640 1 -- 172.21.15.125:0/1455891154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eec13e6b0 con 0x7f7eec0c8430 2023-10-30T12:58:13.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ef293e640 1 -- 172.21.15.125:0/1455891154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eec0c8c10 con 0x7f7eec0c8430 2023-10-30T12:58:13.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ee8ff9640 1 -- 172.21.15.125:0/1455891154 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7ee40c3460 con 0x7f7eec0c8430 2023-10-30T12:58:13.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.299+0000 7f7ee8ff9640 1 --2- 172.21.15.125:0/1455891154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ed80664a0 0x7f7ed8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:13.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.303+0000 7f7ee8ff9640 1 -- 172.21.15.125:0/1455891154 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f7ee411fc40 con 0x7f7eec0c8430 2023-10-30T12:58:13.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.303+0000 7f7eebfff640 1 --2- 172.21.15.125:0/1455891154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ed80664a0 0x7f7ed8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:13.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.303+0000 7f7eebfff640 1 --2- 172.21.15.125:0/1455891154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ed80664a0 0x7f7ed8068960 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00ba30 tx=0x7f7edc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:13.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.303+0000 7f7ee8ff9640 1 -- 172.21.15.125:0/1455891154 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7ee40ecb90 con 0x7f7eec0c8430 2023-10-30T12:58:13.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.487+0000 7f7ef293e640 1 -- 172.21.15.125:0/1455891154 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7eec1f0820 con 0x7f7eec0c8430 2023-10-30T12:58:13.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.487+0000 7f7ee8ff9640 1 -- 172.21.15.125:0/1455891154 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f7ee4009510 con 0x7f7eec0c8430 2023-10-30T12:58:13.487 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:58:13.487 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:11.736307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:clientreplay","state_seq":19,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:58:13.488 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 137 2023-10-30T12:58:13.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.487+0000 7f7ef293e640 1 -- 172.21.15.125:0/1455891154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ed80664a0 msgr2=0x7f7ed8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:13.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.487+0000 7f7ef293e640 1 --2- 172.21.15.125:0/1455891154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ed80664a0 0x7f7ed8068960 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00ba30 tx=0x7f7edc008040 comp rx=0 tx=0).stop 2023-10-30T12:58:13.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.487+0000 7f7ef293e640 1 -- 172.21.15.125:0/1455891154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7eec0c8430 msgr2=0x7f7eec0c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:13.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.487+0000 7f7ef293e640 1 --2- 172.21.15.125:0/1455891154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7eec0c8430 0x7f7eec0c97f0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40af040 tx=0x7f7ee4019770 comp rx=0 tx=0).stop 2023-10-30T12:58:13.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.487+0000 7f7ef293e640 1 -- 172.21.15.125:0/1455891154 shutdown_connections 2023-10-30T12:58:13.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.487+0000 7f7ef293e640 1 --2- 172.21.15.125:0/1455891154 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ed80664a0 0x7f7ed8068960 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:13.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.487+0000 7f7ef293e640 1 --2- 172.21.15.125:0/1455891154 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7eec0c8430 0x7f7eec0c97f0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:13.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.487+0000 7f7ef293e640 1 --2- 172.21.15.125:0/1455891154 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7eec0c6f60 0x7f7eec0c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:13.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.487+0000 7f7ef293e640 1 --2- 172.21.15.125:0/1455891154 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7eec0c6b80 0x7f7eec13e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:13.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.487+0000 7f7ef293e640 1 -- 172.21.15.125:0/1455891154 >> 172.21.15.125:0/1455891154 conn(0x7f7eec0c7910 msgr2=0x7f7eec0b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:13.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.487+0000 7f7ef293e640 1 -- 172.21.15.125:0/1455891154 shutdown_connections 2023-10-30T12:58:13.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:13.487+0000 7f7ef293e640 1 -- 172.21.15.125:0/1455891154 wait complete. 2023-10-30T12:58:13.508 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:58:15.510 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:58:15.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.655+0000 7f12fb73e640 1 Processor -- start 2023-10-30T12:58:15.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.655+0000 7f12fb73e640 1 -- start start 2023-10-30T12:58:15.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12fb73e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12f40c6540 0x7f12f415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:15.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12fb73e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f40c69d0 con 0x7f12f41580e0 2023-10-30T12:58:15.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12fb73e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12f40c5810 con 0x7f12f4070740 2023-10-30T12:58:15.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12fb73e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f406acb0 con 0x7f12f40c6540 2023-10-30T12:58:15.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12f94da640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f12f41580e0 0x7f12f4069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58138/0 (socket says 172.21.15.125:58138) 2023-10-30T12:58:15.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12f94da640 1 -- 172.21.15.125:0/3228998679 learned_addr learned my addr 172.21.15.125:0/3228998679 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:58:15.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12f9cdb640 1 --2- 172.21.15.125:0/3228998679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12f40c6540 0x7f12f415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:15.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 -- 172.21.15.125:0/3228998679 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689969098 0 0) 0x7f12f40c69d0 con 0x7f12f41580e0 2023-10-30T12:58:15.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 -- 172.21.15.125:0/3228998679 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12e0003530 con 0x7f12f41580e0 2023-10-30T12:58:15.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 -- 172.21.15.125:0/3228998679 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3261796162 0 0) 0x7f12e0003530 con 0x7f12f41580e0 2023-10-30T12:58:15.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 -- 172.21.15.125:0/3228998679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12f40c6540 msgr2=0x7f12f415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:15.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 --2- 172.21.15.125:0/3228998679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12f40c6540 0x7f12f415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:15.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 -- 172.21.15.125:0/3228998679 >> v1:172.21.15.136:6789/0 conn(0x7f12f4070740 legacy=0x7f12f406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:15.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 -- 172.21.15.125:0/3228998679 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f40c6bb0 con 0x7f12f41580e0 2023-10-30T12:58:15.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 -- 172.21.15.125:0/3228998679 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f12f0001680 con 0x7f12f41580e0 2023-10-30T12:58:15.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 -- 172.21.15.125:0/3228998679 >> v1:172.21.15.125:6789/0 conn(0x7f12f41580e0 legacy=0x7f12f4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:15.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 --2- 172.21.15.125:0/3228998679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12f40c6540 0x7f12e00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:15.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 --2- 172.21.15.125:0/3228998679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e0003f30 0x7f12e00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:15.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 --2- 172.21.15.125:0/3228998679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12e0004d10 0x7f12e0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:15.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 -- 172.21.15.125:0/3228998679 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e000ba40 con 0x7f12e0003f30 2023-10-30T12:58:15.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 -- 172.21.15.125:0/3228998679 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e000bbb0 con 0x7f12e0004d10 2023-10-30T12:58:15.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12ebfff640 1 -- 172.21.15.125:0/3228998679 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e000bd70 con 0x7f12f40c6540 2023-10-30T12:58:15.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12f8cd9640 1 --2- 172.21.15.125:0/3228998679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e0003f30 0x7f12e00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:15.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12f94da640 1 --2- 172.21.15.125:0/3228998679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12f40c6540 0x7f12e00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:15.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12f9cdb640 1 --2- 172.21.15.125:0/3228998679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12e0004d10 0x7f12e0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:15.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12f8cd9640 1 -- 172.21.15.125:0/3228998679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12f40c6540 msgr2=0x7f12e00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:15.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12f8cd9640 1 --2- 172.21.15.125:0/3228998679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12f40c6540 0x7f12e00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:15.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12f8cd9640 1 -- 172.21.15.125:0/3228998679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12e0004d10 msgr2=0x7f12e0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:15.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12f8cd9640 1 --2- 172.21.15.125:0/3228998679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12e0004d10 0x7f12e0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:15.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12f8cd9640 1 -- 172.21.15.125:0/3228998679 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e0003530 con 0x7f12e0003f30 2023-10-30T12:58:15.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.659+0000 7f12f9cdb640 1 --2- 172.21.15.125:0/3228998679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12e0004d10 0x7f12e0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:58:15.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.663+0000 7f12f8cd9640 1 --2- 172.21.15.125:0/3228998679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e0003f30 0x7f12e00047d0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f12e4001780 tx=0x7f12e401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c7eb0cb91514f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:15.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.663+0000 7f12ebfff640 1 -- 172.21.15.125:0/3228998679 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e40052d0 con 0x7f12e0003f30 2023-10-30T12:58:15.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.663+0000 7f12ebfff640 1 -- 172.21.15.125:0/3228998679 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e40a7050 con 0x7f12e0003f30 2023-10-30T12:58:15.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.663+0000 7f12ebfff640 1 -- 172.21.15.125:0/3228998679 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e4003600 con 0x7f12e0003f30 2023-10-30T12:58:15.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.663+0000 7f12fb73e640 1 -- 172.21.15.125:0/3228998679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e0003f30 msgr2=0x7f12e00047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:15.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.663+0000 7f12fb73e640 1 --2- 172.21.15.125:0/3228998679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12e0003f30 0x7f12e00047d0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f12e4001780 tx=0x7f12e401c400 comp rx=0 tx=0).stop 2023-10-30T12:58:15.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.663+0000 7f12f9cdb640 1 -- 172.21.15.125:0/3228998679 reap_dead start 2023-10-30T12:58:15.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.663+0000 7f12fb73e640 1 -- 172.21.15.125:0/3228998679 shutdown_connections 2023-10-30T12:58:15.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.663+0000 7f12fb73e640 1 -- 172.21.15.125:0/3228998679 >> 172.21.15.125:0/3228998679 conn(0x7f12f40c7910 msgr2=0x7f12f40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:15.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.663+0000 7f12fb73e640 1 -- 172.21.15.125:0/3228998679 shutdown_connections 2023-10-30T12:58:15.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.663+0000 7f12fb73e640 1 -- 172.21.15.125:0/3228998679 wait complete. 2023-10-30T12:58:15.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12fb73e640 1 Processor -- start 2023-10-30T12:58:15.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12fb73e640 1 -- start start 2023-10-30T12:58:15.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12fb73e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12e0003f30 0x7f12f41f6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:15.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12fb73e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f41504d0 0x7f12f414ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:15.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12fb73e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12f40c3b50 0x7f12f40ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:15.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12fb73e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f40c3420 con 0x7f12f41504d0 2023-10-30T12:58:15.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12fb73e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f40c3fd0 con 0x7f12f40c3b50 2023-10-30T12:58:15.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12fb73e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f40c47c0 con 0x7f12e0003f30 2023-10-30T12:58:15.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12f8cd9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f41504d0 0x7f12f414ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:15.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12f8cd9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f41504d0 0x7f12f414ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56424/0 (socket says 172.21.15.125:56424) 2023-10-30T12:58:15.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12f8cd9640 1 -- 172.21.15.125:0/813215521 learned_addr learned my addr 172.21.15.125:0/813215521 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:15.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12f94da640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12e0003f30 0x7f12f41f6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:15.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12f9cdb640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12f40c3b50 0x7f12f40ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:15.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12f8cd9640 1 -- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12e0003f30 msgr2=0x7f12f41f6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:15.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12f8cd9640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12e0003f30 0x7f12f41f6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:15.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12f8cd9640 1 -- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12f40c3b50 msgr2=0x7f12f40ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:15.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12f8cd9640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12f40c3b50 0x7f12f40ca500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:15.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12f8cd9640 1 -- 172.21.15.125:0/813215521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f40c4aa0 con 0x7f12f41504d0 2023-10-30T12:58:15.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12f94da640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12e0003f30 0x7f12f41f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:58:15.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12f9cdb640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12f40c3b50 0x7f12f40ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:58:15.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12f8cd9640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f41504d0 0x7f12f414ee10 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f12e4005040 tx=0x7f12e0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:15.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12e9ffb640 1 -- 172.21.15.125:0/813215521 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e4007390 con 0x7f12f41504d0 2023-10-30T12:58:15.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12e9ffb640 1 -- 172.21.15.125:0/813215521 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e40074f0 con 0x7f12f41504d0 2023-10-30T12:58:15.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12e9ffb640 1 -- 172.21.15.125:0/813215521 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e4006a00 con 0x7f12f41504d0 2023-10-30T12:58:15.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12fb73e640 1 -- 172.21.15.125:0/813215521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f40c4d80 con 0x7f12f41504d0 2023-10-30T12:58:15.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12fb73e640 1 -- 172.21.15.125:0/813215521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f414f4c0 con 0x7f12f41504d0 2023-10-30T12:58:15.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.667+0000 7f12fb73e640 1 -- 172.21.15.125:0/813215521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f40c5a40 con 0x7f12f41504d0 2023-10-30T12:58:15.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.671+0000 7f12e9ffb640 1 -- 172.21.15.125:0/813215521 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f12e4006b60 con 0x7f12f41504d0 2023-10-30T12:58:15.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.671+0000 7f12e9ffb640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12d80664a0 0x7f12d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:15.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.671+0000 7f12e9ffb640 1 -- 172.21.15.125:0/813215521 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f12e411e870 con 0x7f12f41504d0 2023-10-30T12:58:15.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.671+0000 7f12f94da640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12d80664a0 0x7f12d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:15.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.671+0000 7f12f94da640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12d80664a0 0x7f12d8068960 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f12f0001060 tx=0x7f12f0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:15.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.675+0000 7f12e9ffb640 1 -- 172.21.15.125:0/813215521 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f12f40c5a40 con 0x7f12f41504d0 2023-10-30T12:58:15.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.855+0000 7f12fb73e640 1 -- 172.21.15.125:0/813215521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12f40c5a40 con 0x7f12f41504d0 2023-10-30T12:58:15.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.855+0000 7f12e9ffb640 1 -- 172.21.15.125:0/813215521 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f12f40c5a40 con 0x7f12f41504d0 2023-10-30T12:58:15.856 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:58:15.857 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:11.736307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:clientreplay","state_seq":19,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:58:15.857 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 137 2023-10-30T12:58:15.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.859+0000 7f12fb73e640 1 -- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12d80664a0 msgr2=0x7f12d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:15.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.859+0000 7f12fb73e640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12d80664a0 0x7f12d8068960 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f12f0001060 tx=0x7f12f0009040 comp rx=0 tx=0).stop 2023-10-30T12:58:15.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.859+0000 7f12fb73e640 1 -- 172.21.15.125:0/813215521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f41504d0 msgr2=0x7f12f414ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:15.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.859+0000 7f12fb73e640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f41504d0 0x7f12f414ee10 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f12e4005040 tx=0x7f12e0005800 comp rx=0 tx=0).stop 2023-10-30T12:58:15.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.859+0000 7f12fb73e640 1 -- 172.21.15.125:0/813215521 shutdown_connections 2023-10-30T12:58:15.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.859+0000 7f12fb73e640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12d80664a0 0x7f12d8068960 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:15.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.859+0000 7f12fb73e640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12f40c3b50 0x7f12f40ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:15.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.859+0000 7f12fb73e640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12f41504d0 0x7f12f414ee10 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:15.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.859+0000 7f12fb73e640 1 --2- 172.21.15.125:0/813215521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f12e0003f30 0x7f12f41f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:15.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.859+0000 7f12fb73e640 1 -- 172.21.15.125:0/813215521 >> 172.21.15.125:0/813215521 conn(0x7f12f40c7910 msgr2=0x7f12f406b440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:15.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.859+0000 7f12fb73e640 1 -- 172.21.15.125:0/813215521 shutdown_connections 2023-10-30T12:58:15.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:15.859+0000 7f12fb73e640 1 -- 172.21.15.125:0/813215521 wait complete. 2023-10-30T12:58:15.877 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 137, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 137, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7413": { "addr": "172.21.15.125:6837/3906359913", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 3906359913, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 3906359913, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7413, "incarnation": 134, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:clientreplay", "state_seq": 19 } }, "last_failure": 0, "last_failure_osd_epoch": 69, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:58:11.736307+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7413 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:58:17.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:58:18.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.035+0000 7f0b1291d640 1 Processor -- start 2023-10-30T12:58:18.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.035+0000 7f0b1291d640 1 -- start start 2023-10-30T12:58:18.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.035+0000 7f0b1291d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0b0c06f7a0 0x7f0b0c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:18.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.035+0000 7f0b1291d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0c066cf0 con 0x7f0b0c071110 2023-10-30T12:58:18.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.035+0000 7f0b1291d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b0c0c70d0 con 0x7f0b0c0c6bd0 2023-10-30T12:58:18.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.035+0000 7f0b1291d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b0c0700c0 con 0x7f0b0c06f7a0 2023-10-30T12:58:18.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b10eba640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0b0c071110 0x7f0b0c158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58144/0 (socket says 172.21.15.125:58144) 2023-10-30T12:58:18.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b10eba640 1 -- 172.21.15.125:0/614153415 learned_addr learned my addr 172.21.15.125:0/614153415 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:58:18.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0bfff640 1 --2- 172.21.15.125:0/614153415 >> v2:172.21.15.136:3300/0 conn(0x7f0b0c06f7a0 0x7f0b0c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:18.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 -- 172.21.15.125:0/614153415 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658585792 0 0) 0x7f0b0c066cf0 con 0x7f0b0c071110 2023-10-30T12:58:18.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 -- 172.21.15.125:0/614153415 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0af4003530 con 0x7f0b0c071110 2023-10-30T12:58:18.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 -- 172.21.15.125:0/614153415 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1693309361 0 0) 0x7f0af4003530 con 0x7f0b0c071110 2023-10-30T12:58:18.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 -- 172.21.15.125:0/614153415 >> v2:172.21.15.136:3300/0 conn(0x7f0b0c06f7a0 msgr2=0x7f0b0c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:18.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 --2- 172.21.15.125:0/614153415 >> v2:172.21.15.136:3300/0 conn(0x7f0b0c06f7a0 0x7f0b0c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:18.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 -- 172.21.15.125:0/614153415 >> v1:172.21.15.136:6789/0 conn(0x7f0b0c0c6bd0 legacy=0x7f0b0c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:18.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 -- 172.21.15.125:0/614153415 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c0c7390 con 0x7f0b0c071110 2023-10-30T12:58:18.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 -- 172.21.15.125:0/614153415 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f0b04001a00 con 0x7f0b0c071110 2023-10-30T12:58:18.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 -- 172.21.15.125:0/614153415 >> v1:172.21.15.125:6789/0 conn(0x7f0b0c071110 legacy=0x7f0b0c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:18.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 --2- 172.21.15.125:0/614153415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0af4003980 0x7f0af4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:18.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 --2- 172.21.15.125:0/614153415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0af40047d0 0x7f0af4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:18.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 --2- 172.21.15.125:0/614153415 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0af4005190 0x7f0af400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:18.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 -- 172.21.15.125:0/614153415 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c0700c0 con 0x7f0af40047d0 2023-10-30T12:58:18.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 -- 172.21.15.125:0/614153415 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af400c040 con 0x7f0af4003980 2023-10-30T12:58:18.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0affd640 1 -- 172.21.15.125:0/614153415 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0af400c200 con 0x7f0af4005190 2023-10-30T12:58:18.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0b7fe640 1 --2- 172.21.15.125:0/614153415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0af40047d0 0x7f0af4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:18.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0bfff640 1 --2- 172.21.15.125:0/614153415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0af4003980 0x7f0af4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:18.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b10eba640 1 --2- 172.21.15.125:0/614153415 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0af4005190 0x7f0af400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:18.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0b7fe640 1 -- 172.21.15.125:0/614153415 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0af4005190 msgr2=0x7f0af400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:18.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0b7fe640 1 --2- 172.21.15.125:0/614153415 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0af4005190 0x7f0af400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:18.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0b7fe640 1 -- 172.21.15.125:0/614153415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0af4003980 msgr2=0x7f0af4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:18.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0b7fe640 1 --2- 172.21.15.125:0/614153415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0af4003980 0x7f0af4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:18.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0b7fe640 1 -- 172.21.15.125:0/614153415 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af4003530 con 0x7f0af40047d0 2023-10-30T12:58:18.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b10eba640 1 -- 172.21.15.125:0/614153415 reap_dead start 2023-10-30T12:58:18.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.039+0000 7f0b0bfff640 1 --2- 172.21.15.125:0/614153415 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0af4003980 0x7f0af4003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:58:18.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.043+0000 7f0b0b7fe640 1 --2- 172.21.15.125:0/614153415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0af40047d0 0x7f0af4004c50 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f0b00001750 tx=0x7f0b0001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced6539276bb8480 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:18.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.043+0000 7f0b0affd640 1 -- 172.21.15.125:0/614153415 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b000052d0 con 0x7f0af40047d0 2023-10-30T12:58:18.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.043+0000 7f0b0affd640 1 -- 172.21.15.125:0/614153415 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b000a7050 con 0x7f0af40047d0 2023-10-30T12:58:18.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.043+0000 7f0b0affd640 1 -- 172.21.15.125:0/614153415 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b00003600 con 0x7f0af40047d0 2023-10-30T12:58:18.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.043+0000 7f0b1291d640 1 -- 172.21.15.125:0/614153415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0af40047d0 msgr2=0x7f0af4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:18.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.043+0000 7f0b1291d640 1 --2- 172.21.15.125:0/614153415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0af40047d0 0x7f0af4004c50 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f0b00001750 tx=0x7f0b0001c3c0 comp rx=0 tx=0).stop 2023-10-30T12:58:18.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.043+0000 7f0b1291d640 1 -- 172.21.15.125:0/614153415 shutdown_connections 2023-10-30T12:58:18.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.043+0000 7f0b1291d640 1 --2- 172.21.15.125:0/614153415 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0af40047d0 0x7f0af4004c50 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:18.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.043+0000 7f0b1291d640 1 -- 172.21.15.125:0/614153415 >> 172.21.15.125:0/614153415 conn(0x7f0b0c0c7910 msgr2=0x7f0b0c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:18.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.043+0000 7f0b1291d640 1 -- 172.21.15.125:0/614153415 shutdown_connections 2023-10-30T12:58:18.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.043+0000 7f0b1291d640 1 -- 172.21.15.125:0/614153415 wait complete. 2023-10-30T12:58:18.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.043+0000 7f0b1291d640 1 Processor -- start 2023-10-30T12:58:18.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.043+0000 7f0b1291d640 1 -- start start 2023-10-30T12:58:18.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b1291d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0af40047d0 0x7f0b0c1f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:18.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b1291d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b0c15b180 0x7f0b0c15a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:18.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b1291d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c159dc0 0x7f0b0c1f13f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:18.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b1291d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c158130 con 0x7f0af40047d0 2023-10-30T12:58:18.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b1291d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c1586a0 con 0x7f0b0c159dc0 2023-10-30T12:58:18.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b1291d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0c1f3070 con 0x7f0b0c15b180 2023-10-30T12:58:18.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b0bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0af40047d0 0x7f0b0c1f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:18.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b0bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0af40047d0 0x7f0b0c1f2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56436/0 (socket says 172.21.15.125:56436) 2023-10-30T12:58:18.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b0bfff640 1 -- 172.21.15.125:0/3761186597 learned_addr learned my addr 172.21.15.125:0/3761186597 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:18.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b10eba640 1 --2- 172.21.15.125:0/3761186597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c159dc0 0x7f0b0c1f13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:18.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b0b7fe640 1 --2- 172.21.15.125:0/3761186597 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b0c15b180 0x7f0b0c15a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:18.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b0bfff640 1 -- 172.21.15.125:0/3761186597 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b0c15b180 msgr2=0x7f0b0c15a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:18.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b0bfff640 1 --2- 172.21.15.125:0/3761186597 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b0c15b180 0x7f0b0c15a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:18.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b0bfff640 1 -- 172.21.15.125:0/3761186597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c159dc0 msgr2=0x7f0b0c1f13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:18.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b0bfff640 1 --2- 172.21.15.125:0/3761186597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c159dc0 0x7f0b0c1f13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:18.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b0bfff640 1 -- 172.21.15.125:0/3761186597 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b0c158a40 con 0x7f0af40047d0 2023-10-30T12:58:18.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b0bfff640 1 --2- 172.21.15.125:0/3761186597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0af40047d0 0x7f0b0c1f2b30 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f0afc011540 tx=0x7f0afc011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:18.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b08ff9640 1 -- 172.21.15.125:0/3761186597 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc004220 con 0x7f0af40047d0 2023-10-30T12:58:18.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b08ff9640 1 -- 172.21.15.125:0/3761186597 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0afc004380 con 0x7f0af40047d0 2023-10-30T12:58:18.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b08ff9640 1 -- 172.21.15.125:0/3761186597 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0afc015890 con 0x7f0af40047d0 2023-10-30T12:58:18.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b1291d640 1 -- 172.21.15.125:0/3761186597 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b0c158d20 con 0x7f0af40047d0 2023-10-30T12:58:18.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b1291d640 1 -- 172.21.15.125:0/3761186597 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0c1592e0 con 0x7f0af40047d0 2023-10-30T12:58:18.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b1291d640 1 -- 172.21.15.125:0/3761186597 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b0c04e170 con 0x7f0af40047d0 2023-10-30T12:58:18.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.047+0000 7f0b08ff9640 1 -- 172.21.15.125:0/3761186597 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0afc0159f0 con 0x7f0af40047d0 2023-10-30T12:58:18.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.051+0000 7f0b08ff9640 1 --2- 172.21.15.125:0/3761186597 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0af80664d0 0x7f0af8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:18.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.051+0000 7f0b08ff9640 1 -- 172.21.15.125:0/3761186597 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f0afc0848d0 con 0x7f0af40047d0 2023-10-30T12:58:18.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.051+0000 7f0b0b7fe640 1 --2- 172.21.15.125:0/3761186597 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0af80664d0 0x7f0af8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:18.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.051+0000 7f0b0b7fe640 1 --2- 172.21.15.125:0/3761186597 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0af80664d0 0x7f0af8068990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0c7000 tx=0x7f0b000011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:18.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.055+0000 7f0b08ff9640 1 -- 172.21.15.125:0/3761186597 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0afc0517a0 con 0x7f0af40047d0 2023-10-30T12:58:18.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.247+0000 7f0b1291d640 1 -- 172.21.15.125:0/3761186597 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b0c0c8660 con 0x7f0af40047d0 2023-10-30T12:58:18.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.247+0000 7f0b08ff9640 1 -- 172.21.15.125:0/3761186597 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f0afc022090 con 0x7f0af40047d0 2023-10-30T12:58:18.248 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:58:18.249 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":137,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:11.736307+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:clientreplay","state_seq":19,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:58:18.249 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 137 2023-10-30T12:58:18.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.251+0000 7f0b1291d640 1 -- 172.21.15.125:0/3761186597 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0af80664d0 msgr2=0x7f0af8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:18.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.251+0000 7f0b1291d640 1 --2- 172.21.15.125:0/3761186597 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0af80664d0 0x7f0af8068990 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c0c7000 tx=0x7f0b000011a0 comp rx=0 tx=0).stop 2023-10-30T12:58:18.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.251+0000 7f0b1291d640 1 -- 172.21.15.125:0/3761186597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0af40047d0 msgr2=0x7f0b0c1f2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:18.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.251+0000 7f0b1291d640 1 --2- 172.21.15.125:0/3761186597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0af40047d0 0x7f0b0c1f2b30 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f0afc011540 tx=0x7f0afc011a00 comp rx=0 tx=0).stop 2023-10-30T12:58:18.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.251+0000 7f0b1291d640 1 -- 172.21.15.125:0/3761186597 shutdown_connections 2023-10-30T12:58:18.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.251+0000 7f0b1291d640 1 --2- 172.21.15.125:0/3761186597 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0af80664d0 0x7f0af8068990 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:18.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.251+0000 7f0b1291d640 1 --2- 172.21.15.125:0/3761186597 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b0c159dc0 0x7f0b0c1f13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:18.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.251+0000 7f0b1291d640 1 --2- 172.21.15.125:0/3761186597 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b0c15b180 0x7f0b0c15a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:18.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.251+0000 7f0b1291d640 1 --2- 172.21.15.125:0/3761186597 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0af40047d0 0x7f0b0c1f2b30 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:18.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.251+0000 7f0b1291d640 1 -- 172.21.15.125:0/3761186597 >> 172.21.15.125:0/3761186597 conn(0x7f0b0c0c7910 msgr2=0x7f0b0c0ca6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:18.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.251+0000 7f0b1291d640 1 -- 172.21.15.125:0/3761186597 shutdown_connections 2023-10-30T12:58:18.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:18.251+0000 7f0b1291d640 1 -- 172.21.15.125:0/3761186597 wait complete. 2023-10-30T12:58:18.277 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:58:19.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:58:19.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:58:20.278 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:58:20.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f65a4a46640 1 Processor -- start 2023-10-30T12:58:20.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f65a4a46640 1 -- start start 2023-10-30T12:58:20.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f65a4a46640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f65a00c05d0 0x7f65a013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:20.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f65a4a46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65a00c6f60 0x7f65a013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:20.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f65a4a46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65a00c0a50 con 0x7f65a00c6b80 2023-10-30T12:58:20.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f65a4a46640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65a006f6e0 con 0x7f65a00c05d0 2023-10-30T12:58:20.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f65a4a46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a006f9c0 con 0x7f65a00c6f60 2023-10-30T12:58:20.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f6595d9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f65a00c05d0 0x7f65a013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:20.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f659ed9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65a00c6f60 0x7f65a013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:20.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f6595d9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f65a00c05d0 0x7f65a013ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:38478/0 (socket says 172.21.15.125:38478) 2023-10-30T12:58:20.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f6595d9b640 1 -- 172.21.15.125:0/451534296 learned_addr learned my addr 172.21.15.125:0/451534296 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:20.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f659e59c640 1 --1- 172.21.15.125:0/451534296 >> v1:172.21.15.136:6789/0 conn(0x7f65a00c6b80 0x7f65a00bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:47460/0 (socket says 172.21.15.125:47460) 2023-10-30T12:58:20.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f6595d9b640 1 -- 172.21.15.125:0/451534296 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65a00c6f60 msgr2=0x7f65a013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:20.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f6595d9b640 1 --2- 172.21.15.125:0/451534296 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65a00c6f60 0x7f65a013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:20.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f6595d9b640 1 -- 172.21.15.125:0/451534296 >> v1:172.21.15.136:6789/0 conn(0x7f65a00c6b80 legacy=0x7f65a00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:20.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.423+0000 7f6595d9b640 1 -- 172.21.15.125:0/451534296 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a006fca0 con 0x7f65a00c05d0 2023-10-30T12:58:20.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.427+0000 7f6595d9b640 1 --2- 172.21.15.125:0/451534296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a00c05d0 0x7f65a013ed90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6588009870 tx=0x7f658801c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68cb3c7c4ab4a394 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:20.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.427+0000 7f659dd9b640 1 -- 172.21.15.125:0/451534296 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65880ab650 con 0x7f65a00c05d0 2023-10-30T12:58:20.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.427+0000 7f659dd9b640 1 -- 172.21.15.125:0/451534296 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65880ab7b0 con 0x7f65a00c05d0 2023-10-30T12:58:20.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.427+0000 7f659dd9b640 1 -- 172.21.15.125:0/451534296 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65880abb40 con 0x7f65a00c05d0 2023-10-30T12:58:20.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.427+0000 7f65a4a46640 1 -- 172.21.15.125:0/451534296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a00c05d0 msgr2=0x7f65a013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:20.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.427+0000 7f65a4a46640 1 --2- 172.21.15.125:0/451534296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a00c05d0 0x7f65a013ed90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6588009870 tx=0x7f658801c540 comp rx=0 tx=0).stop 2023-10-30T12:58:20.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.427+0000 7f65a4a46640 1 -- 172.21.15.125:0/451534296 shutdown_connections 2023-10-30T12:58:20.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.427+0000 7f65a4a46640 1 --2- 172.21.15.125:0/451534296 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65a00c6f60 0x7f65a013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:20.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.427+0000 7f65a4a46640 1 --2- 172.21.15.125:0/451534296 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a00c05d0 0x7f65a013ed90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:20.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.427+0000 7f65a4a46640 1 -- 172.21.15.125:0/451534296 >> 172.21.15.125:0/451534296 conn(0x7f65a00c7910 msgr2=0x7f65a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:20.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.427+0000 7f65a4a46640 1 -- 172.21.15.125:0/451534296 shutdown_connections 2023-10-30T12:58:20.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.427+0000 7f65a4a46640 1 -- 172.21.15.125:0/451534296 wait complete. 2023-10-30T12:58:20.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f65a4a46640 1 Processor -- start 2023-10-30T12:58:20.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f65a4a46640 1 -- start start 2023-10-30T12:58:20.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f65a4a46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65a00c05d0 0x7f65a013ab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:20.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f65a4a46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65a00c6b80 0x7f65a00c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:20.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f65a4a46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a00c6f60 0x7f65a0152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:20.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f65a4a46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a01599f0 con 0x7f65a00c6b80 2023-10-30T12:58:20.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f65a4a46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a0159750 con 0x7f65a00c6f60 2023-10-30T12:58:20.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f65a4a46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65a01551c0 con 0x7f65a00c05d0 2023-10-30T12:58:20.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f6595d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65a00c6b80 0x7f65a00c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:20.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f6595d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65a00c6b80 0x7f65a00c8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56438/0 (socket says 172.21.15.125:56438) 2023-10-30T12:58:20.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f6595d9b640 1 -- 172.21.15.125:0/3029540236 learned_addr learned my addr 172.21.15.125:0/3029540236 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:20.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f659e59c640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65a00c05d0 0x7f65a013ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:20.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f659ed9d640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a00c6f60 0x7f65a0152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:20.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f6595d9b640 1 -- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65a00c05d0 msgr2=0x7f65a013ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:20.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f6595d9b640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65a00c05d0 0x7f65a013ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:20.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f6595d9b640 1 -- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a00c6f60 msgr2=0x7f65a0152440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:20.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f6595d9b640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a00c6f60 0x7f65a0152440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:20.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f6595d9b640 1 -- 172.21.15.125:0/3029540236 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65880094d0 con 0x7f65a00c6b80 2023-10-30T12:58:20.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f659e59c640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65a00c05d0 0x7f65a013ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:58:20.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f6595d9b640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65a00c6b80 0x7f65a00c8c10 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f658801a170 tx=0x7f658801ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:20.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f65977fe640 1 -- 172.21.15.125:0/3029540236 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65880b4800 con 0x7f65a00c6b80 2023-10-30T12:58:20.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f65977fe640 1 -- 172.21.15.125:0/3029540236 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65880b4960 con 0x7f65a00c6b80 2023-10-30T12:58:20.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f65a4a46640 1 -- 172.21.15.125:0/3029540236 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a0158d60 con 0x7f65a00c6b80 2023-10-30T12:58:20.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f65977fe640 1 -- 172.21.15.125:0/3029540236 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65880b4cf0 con 0x7f65a00c6b80 2023-10-30T12:58:20.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.431+0000 7f65a4a46640 1 -- 172.21.15.125:0/3029540236 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a0159320 con 0x7f65a00c6b80 2023-10-30T12:58:20.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.435+0000 7f65a4a46640 1 -- 172.21.15.125:0/3029540236 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65a004e170 con 0x7f65a00c6b80 2023-10-30T12:58:20.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.435+0000 7f65977fe640 1 -- 172.21.15.125:0/3029540236 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f65880aa430 con 0x7f65a00c6b80 2023-10-30T12:58:20.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.435+0000 7f65977fe640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65800664a0 0x7f6580068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:20.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.435+0000 7f65977fe640 1 -- 172.21.15.125:0/3029540236 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f658811e6c0 con 0x7f65a00c6b80 2023-10-30T12:58:20.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.435+0000 7f659e59c640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65800664a0 0x7f6580068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:20.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.435+0000 7f659e59c640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65800664a0 0x7f6580068960 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f65900012f0 tx=0x7f6590005be0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:20.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.439+0000 7f65977fe640 1 -- 172.21.15.125:0/3029540236 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f65880eb570 con 0x7f65a00c6b80 2023-10-30T12:58:20.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.623+0000 7f65a4a46640 1 -- 172.21.15.125:0/3029540236 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65a00af400 con 0x7f65a00c6b80 2023-10-30T12:58:20.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.623+0000 7f65977fe640 1 -- 172.21.15.125:0/3029540236 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f65880eeb90 con 0x7f65a00c6b80 2023-10-30T12:58:20.626 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:58:20.626 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:19.758581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":22,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:58:20.627 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 138 2023-10-30T12:58:20.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.627+0000 7f65a4a46640 1 -- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65800664a0 msgr2=0x7f6580068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:20.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.627+0000 7f65a4a46640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65800664a0 0x7f6580068960 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f65900012f0 tx=0x7f6590005be0 comp rx=0 tx=0).stop 2023-10-30T12:58:20.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.627+0000 7f65a4a46640 1 -- 172.21.15.125:0/3029540236 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65a00c6b80 msgr2=0x7f65a00c8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:20.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.627+0000 7f65a4a46640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65a00c6b80 0x7f65a00c8c10 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f658801a170 tx=0x7f658801ad80 comp rx=0 tx=0).stop 2023-10-30T12:58:20.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.627+0000 7f65a4a46640 1 -- 172.21.15.125:0/3029540236 shutdown_connections 2023-10-30T12:58:20.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.627+0000 7f65a4a46640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f65800664a0 0x7f6580068960 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:20.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.627+0000 7f65a4a46640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f65a00c6f60 0x7f65a0152440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:20.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.627+0000 7f65a4a46640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f65a00c6b80 0x7f65a00c8c10 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:20.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.627+0000 7f65a4a46640 1 --2- 172.21.15.125:0/3029540236 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f65a00c05d0 0x7f65a013ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:20.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.627+0000 7f65a4a46640 1 -- 172.21.15.125:0/3029540236 >> 172.21.15.125:0/3029540236 conn(0x7f65a00c7910 msgr2=0x7f65a006a720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:20.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.627+0000 7f65a4a46640 1 -- 172.21.15.125:0/3029540236 shutdown_connections 2023-10-30T12:58:20.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:20.627+0000 7f65a4a46640 1 -- 172.21.15.125:0/3029540236 wait complete. 2023-10-30T12:58:20.646 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=7269 2023-10-30T12:58:20.646 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.e 2023-10-30T12:58:21.647 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T12:58:21.648 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T12:58:21.648 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T12:58:21.652 INFO:tasks.ceph.mds.e:Started 2023-10-30T12:58:21.653 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:58:21.750 INFO:tasks.ceph.mds.e.smithi125.stdout:starting mds.e at 2023-10-30T12:58:21.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.803+0000 7fec0d09c640 1 Processor -- start 2023-10-30T12:58:21.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.803+0000 7fec0d09c640 1 -- start start 2023-10-30T12:58:21.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.803+0000 7fec0d09c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec081538a0 0x7fec0815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:21.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.803+0000 7fec0d09c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fec080ca0e0 0x7fec0815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:21.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.803+0000 7fec0d09c640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec080cb800 con 0x7fec080cf160 2023-10-30T12:58:21.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.803+0000 7fec0d09c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec0815dc30 con 0x7fec080ca0e0 2023-10-30T12:58:21.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.803+0000 7fec0d09c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec080cb330 con 0x7fec081538a0 2023-10-30T12:58:21.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec0759e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fec080cf160 0x7fec080cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58100/0 (socket says 172.21.15.125:58100) 2023-10-30T12:58:21.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec0759e640 1 -- 172.21.15.125:0/2633466374 learned_addr learned my addr 172.21.15.125:0/2633466374 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:58:21.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec06d9d640 1 --2- 172.21.15.125:0/2633466374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec081538a0 0x7fec0815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:21.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec0659c640 1 --2- 172.21.15.125:0/2633466374 >> v2:172.21.15.136:3300/0 conn(0x7fec080ca0e0 0x7fec0815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:21.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 -- 172.21.15.125:0/2633466374 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261986842 0 0) 0x7fec080cb800 con 0x7fec080cf160 2023-10-30T12:58:21.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 -- 172.21.15.125:0/2633466374 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf0003530 con 0x7fec080cf160 2023-10-30T12:58:21.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 -- 172.21.15.125:0/2633466374 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1620056992 0 0) 0x7febf0003530 con 0x7fec080cf160 2023-10-30T12:58:21.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 -- 172.21.15.125:0/2633466374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec081538a0 msgr2=0x7fec0815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:21.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 --2- 172.21.15.125:0/2633466374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec081538a0 0x7fec0815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:21.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 -- 172.21.15.125:0/2633466374 >> v2:172.21.15.136:3300/0 conn(0x7fec080ca0e0 msgr2=0x7fec0815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:21.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 --2- 172.21.15.125:0/2633466374 >> v2:172.21.15.136:3300/0 conn(0x7fec080ca0e0 0x7fec0815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:21.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 -- 172.21.15.125:0/2633466374 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec080cad70 con 0x7fec080cf160 2023-10-30T12:58:21.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 -- 172.21.15.125:0/2633466374 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fec00001a00 con 0x7fec080cf160 2023-10-30T12:58:21.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 -- 172.21.15.125:0/2633466374 >> v1:172.21.15.125:6789/0 conn(0x7fec080cf160 legacy=0x7fec080cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:21.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 --2- 172.21.15.125:0/2633466374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec081538a0 0x7febf0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:21.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 --2- 172.21.15.125:0/2633466374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febf0003f00 0x7febf00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:21.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 --2- 172.21.15.125:0/2633466374 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febf0004d10 0x7febf0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:21.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 -- 172.21.15.125:0/2633466374 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec0815dc30 con 0x7febf0004d10 2023-10-30T12:58:21.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 -- 172.21.15.125:0/2633466374 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec080cb330 con 0x7febf0003f00 2023-10-30T12:58:21.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec05d9b640 1 -- 172.21.15.125:0/2633466374 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf000b9c0 con 0x7fec081538a0 2023-10-30T12:58:21.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec0759e640 1 --2- 172.21.15.125:0/2633466374 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febf0004d10 0x7febf0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:21.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec0659c640 1 --2- 172.21.15.125:0/2633466374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febf0003f00 0x7febf00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:21.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec06d9d640 1 --2- 172.21.15.125:0/2633466374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec081538a0 0x7febf0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:21.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec0759e640 1 -- 172.21.15.125:0/2633466374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec081538a0 msgr2=0x7febf0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:21.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec0759e640 1 --2- 172.21.15.125:0/2633466374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fec081538a0 0x7febf0003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:21.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec0759e640 1 -- 172.21.15.125:0/2633466374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febf0003f00 msgr2=0x7febf00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:21.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec0759e640 1 --2- 172.21.15.125:0/2633466374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febf0003f00 0x7febf00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:21.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.807+0000 7fec0759e640 1 -- 172.21.15.125:0/2633466374 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf0003530 con 0x7febf0004d10 2023-10-30T12:58:21.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.811+0000 7fec0759e640 1 --2- 172.21.15.125:0/2633466374 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febf0004d10 0x7febf0005190 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fec0000a860 tx=0x7fec0001d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c67d6a198dab8af server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:21.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.811+0000 7fec05d9b640 1 -- 172.21.15.125:0/2633466374 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec000ad630 con 0x7febf0004d10 2023-10-30T12:58:21.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.811+0000 7fec05d9b640 1 -- 172.21.15.125:0/2633466374 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec000a8070 con 0x7febf0004d10 2023-10-30T12:58:21.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.811+0000 7fec05d9b640 1 -- 172.21.15.125:0/2633466374 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec000ada30 con 0x7febf0004d10 2023-10-30T12:58:21.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.811+0000 7fec0d09c640 1 -- 172.21.15.125:0/2633466374 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febf0004d10 msgr2=0x7febf0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:21.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.811+0000 7fec0d09c640 1 --2- 172.21.15.125:0/2633466374 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febf0004d10 0x7febf0005190 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fec0000a860 tx=0x7fec0001d360 comp rx=0 tx=0).stop 2023-10-30T12:58:21.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.811+0000 7fec0759e640 1 -- 172.21.15.125:0/2633466374 reap_dead start 2023-10-30T12:58:21.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.811+0000 7fec0d09c640 1 -- 172.21.15.125:0/2633466374 shutdown_connections 2023-10-30T12:58:21.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.811+0000 7fec0d09c640 1 -- 172.21.15.125:0/2633466374 >> 172.21.15.125:0/2633466374 conn(0x7fec0806bfe0 msgr2=0x7fec0806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:21.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.811+0000 7fec0d09c640 1 -- 172.21.15.125:0/2633466374 shutdown_connections 2023-10-30T12:58:21.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.811+0000 7fec0d09c640 1 -- 172.21.15.125:0/2633466374 wait complete. 2023-10-30T12:58:21.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.811+0000 7fec0d09c640 1 Processor -- start 2023-10-30T12:58:21.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.811+0000 7fec0d09c640 1 -- start start 2023-10-30T12:58:21.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0d09c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febf0004d10 0x7fec081f0120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:21.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0d09c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec08151540 0x7fec080c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:21.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0d09c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec081e52f0 0x7fec08145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:21.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0d09c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec081e2ae0 con 0x7fec08151540 2023-10-30T12:58:21.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0d09c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec081e3660 con 0x7fec081e52f0 2023-10-30T12:58:21.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0d09c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec081e5800 con 0x7febf0004d10 2023-10-30T12:58:21.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec08151540 0x7fec080c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:21.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec08151540 0x7fec080c8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37598/0 (socket says 172.21.15.125:37598) 2023-10-30T12:58:21.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0659c640 1 -- 172.21.15.125:0/4136849627 learned_addr learned my addr 172.21.15.125:0/4136849627 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:21.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec06d9d640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febf0004d10 0x7fec081f0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:21.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0759e640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec081e52f0 0x7fec08145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:21.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0659c640 1 -- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febf0004d10 msgr2=0x7fec081f0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:21.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0659c640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febf0004d10 0x7fec081f0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:21.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0659c640 1 -- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec081e52f0 msgr2=0x7fec08145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:21.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0659c640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec081e52f0 0x7fec08145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:21.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0659c640 1 -- 172.21.15.125:0/4136849627 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec081e2d30 con 0x7fec08151540 2023-10-30T12:58:21.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec06d9d640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febf0004d10 0x7fec081f0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:58:21.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0759e640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec081e52f0 0x7fec08145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:58:21.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0659c640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec08151540 0x7fec080c8bf0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7febfc0119b0 tx=0x7febfc011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:21.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7febef7fe640 1 -- 172.21.15.125:0/4136849627 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc00faf0 con 0x7fec08151540 2023-10-30T12:58:21.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7febef7fe640 1 -- 172.21.15.125:0/4136849627 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febfc00fc50 con 0x7fec08151540 2023-10-30T12:58:21.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7febef7fe640 1 -- 172.21.15.125:0/4136849627 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febfc014440 con 0x7fec08151540 2023-10-30T12:58:21.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0d09c640 1 -- 172.21.15.125:0/4136849627 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec081e3010 con 0x7fec08151540 2023-10-30T12:58:21.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0d09c640 1 -- 172.21.15.125:0/4136849627 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec081f0660 con 0x7fec08151540 2023-10-30T12:58:21.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.815+0000 7fec0d09c640 1 -- 172.21.15.125:0/4136849627 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec0804e170 con 0x7fec08151540 2023-10-30T12:58:21.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.819+0000 7febef7fe640 1 -- 172.21.15.125:0/4136849627 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7febfc0145a0 con 0x7fec08151540 2023-10-30T12:58:21.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.819+0000 7febef7fe640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febf40664d0 0x7febf4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:21.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.819+0000 7febef7fe640 1 -- 172.21.15.125:0/4136849627 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7febfc084ce0 con 0x7fec08151540 2023-10-30T12:58:21.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.819+0000 7fec06d9d640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febf40664d0 0x7febf4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:21.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.819+0000 7fec06d9d640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febf40664d0 0x7febf4068990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7febf8009be0 tx=0x7febf8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:21.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:21.823+0000 7febef7fe640 1 -- 172.21.15.125:0/4136849627 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7febfc051b90 con 0x7fec08151540 2023-10-30T12:58:22.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:22.003+0000 7fec0d09c640 1 -- 172.21.15.125:0/4136849627 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec080b1dd0 con 0x7fec08151540 2023-10-30T12:58:22.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:22.003+0000 7febef7fe640 1 -- 172.21.15.125:0/4136849627 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+5724 (secure 0 0 0) 0x7febfc022020 con 0x7fec08151540 2023-10-30T12:58:22.006 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:58:22.006 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:19.758581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":22,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:58:22.007 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 138 2023-10-30T12:58:22.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:22.007+0000 7fec0d09c640 1 -- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febf40664d0 msgr2=0x7febf4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:22.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:22.007+0000 7fec0d09c640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febf40664d0 0x7febf4068990 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7febf8009be0 tx=0x7febf8008040 comp rx=0 tx=0).stop 2023-10-30T12:58:22.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:22.007+0000 7fec0d09c640 1 -- 172.21.15.125:0/4136849627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec08151540 msgr2=0x7fec080c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:22.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:22.007+0000 7fec0d09c640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec08151540 0x7fec080c8bf0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7febfc0119b0 tx=0x7febfc011e70 comp rx=0 tx=0).stop 2023-10-30T12:58:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:22.007+0000 7fec0d09c640 1 -- 172.21.15.125:0/4136849627 shutdown_connections 2023-10-30T12:58:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:22.007+0000 7fec0d09c640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febf40664d0 0x7febf4068990 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:22.007+0000 7fec0d09c640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fec081e52f0 0x7fec08145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:22.007+0000 7fec0d09c640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fec08151540 0x7fec080c8bf0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:22.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:22.007+0000 7fec0d09c640 1 --2- 172.21.15.125:0/4136849627 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febf0004d10 0x7fec081f0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:22.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:22.007+0000 7fec0d09c640 1 -- 172.21.15.125:0/4136849627 >> 172.21.15.125:0/4136849627 conn(0x7fec0806bfe0 msgr2=0x7fec080c5700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:22.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:22.007+0000 7fec0d09c640 1 -- 172.21.15.125:0/4136849627 shutdown_connections 2023-10-30T12:58:22.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:22.007+0000 7fec0d09c640 1 -- 172.21.15.125:0/4136849627 wait complete. 2023-10-30T12:58:22.027 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T12:58:24.028 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:58:24.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.175+0000 7f6f102c9640 1 Processor -- start 2023-10-30T12:58:24.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.175+0000 7f6f102c9640 1 -- start start 2023-10-30T12:58:24.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.175+0000 7f6f102c9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6f080cf160 0x7f6f080cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:24.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.175+0000 7f6f102c9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080cb800 con 0x7f6f081538a0 2023-10-30T12:58:24.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.175+0000 7f6f102c9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f080cb330 con 0x7f6f0815d6f0 2023-10-30T12:58:24.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.175+0000 7f6f102c9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f080cad70 con 0x7f6f080cf160 2023-10-30T12:58:24.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.175+0000 7f6f0e065640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6f081538a0 0x7f6f080ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58110/0 (socket says 172.21.15.125:58110) 2023-10-30T12:58:24.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.175+0000 7f6f0e065640 1 -- 172.21.15.125:0/2585352121 learned_addr learned my addr 172.21.15.125:0/2585352121 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:58:24.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.175+0000 7f6f0e866640 1 --2- 172.21.15.125:0/2585352121 >> v2:172.21.15.125:3300/0 conn(0x7f6f080cf160 0x7f6f080cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:24.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.175+0000 7f6f0d063640 1 -- 172.21.15.125:0/2585352121 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831617429 0 0) 0x7f6f080cb800 con 0x7f6f081538a0 2023-10-30T12:58:24.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.175+0000 7f6f0d063640 1 -- 172.21.15.125:0/2585352121 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ef4003530 con 0x7f6f081538a0 2023-10-30T12:58:24.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.175+0000 7f6f0e866640 1 -- 172.21.15.125:0/2585352121 >> v1:172.21.15.136:6789/0 conn(0x7f6f0815d6f0 legacy=0x7f6f0815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:24.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.175+0000 7f6f0e866640 1 -- 172.21.15.125:0/2585352121 >> v1:172.21.15.125:6789/0 conn(0x7f6f081538a0 legacy=0x7f6f080ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:24.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.175+0000 7f6f0e866640 1 -- 172.21.15.125:0/2585352121 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f080cb050 con 0x7f6f080cf160 2023-10-30T12:58:24.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.179+0000 7f6f0e866640 1 --2- 172.21.15.125:0/2585352121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f080cf160 0x7f6f080cf590 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009d90 tx=0x7f6f0001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307bb90ed376e05e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:24.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.179+0000 7f6f0d063640 1 -- 172.21.15.125:0/2585352121 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f000af600 con 0x7f6f080cf160 2023-10-30T12:58:24.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.179+0000 7f6f0d063640 1 -- 172.21.15.125:0/2585352121 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f000ab050 con 0x7f6f080cf160 2023-10-30T12:58:24.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.179+0000 7f6f0d063640 1 -- 172.21.15.125:0/2585352121 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f000af980 con 0x7f6f080cf160 2023-10-30T12:58:24.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.179+0000 7f6f102c9640 1 -- 172.21.15.125:0/2585352121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f080cf160 msgr2=0x7f6f080cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:24.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.179+0000 7f6f102c9640 1 --2- 172.21.15.125:0/2585352121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f080cf160 0x7f6f080cf590 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009d90 tx=0x7f6f0001f130 comp rx=0 tx=0).stop 2023-10-30T12:58:24.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.179+0000 7f6f102c9640 1 -- 172.21.15.125:0/2585352121 shutdown_connections 2023-10-30T12:58:24.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.179+0000 7f6f102c9640 1 --2- 172.21.15.125:0/2585352121 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f080cf160 0x7f6f080cf590 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:24.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.179+0000 7f6f102c9640 1 -- 172.21.15.125:0/2585352121 >> 172.21.15.125:0/2585352121 conn(0x7f6f0806bfe0 msgr2=0x7f6f0806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:24.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.179+0000 7f6f102c9640 1 -- 172.21.15.125:0/2585352121 shutdown_connections 2023-10-30T12:58:24.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.179+0000 7f6f102c9640 1 -- 172.21.15.125:0/2585352121 wait complete. 2023-10-30T12:58:24.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f102c9640 1 Processor -- start 2023-10-30T12:58:24.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f102c9640 1 -- start start 2023-10-30T12:58:24.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f102c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f080cf160 0x7f6f080ce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:24.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f102c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f081538a0 0x7f6f080b3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:24.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f102c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f0815d6f0 0x7f6f08148690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:24.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f102c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f08149a30 con 0x7f6f0815d6f0 2023-10-30T12:58:24.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f102c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f080ce3a0 con 0x7f6f080cf160 2023-10-30T12:58:24.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f102c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f08149cd0 con 0x7f6f081538a0 2023-10-30T12:58:24.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f0e866640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f0815d6f0 0x7f6f08148690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:24.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f0e866640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f0815d6f0 0x7f6f08148690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37628/0 (socket says 172.21.15.125:37628) 2023-10-30T12:58:24.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f0e866640 1 -- 172.21.15.125:0/2237114671 learned_addr learned my addr 172.21.15.125:0/2237114671 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:24.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f0e065640 1 --2- 172.21.15.125:0/2237114671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f080cf160 0x7f6f080ce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:24.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f0d864640 1 --2- 172.21.15.125:0/2237114671 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f081538a0 0x7f6f080b3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:24.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f0e866640 1 -- 172.21.15.125:0/2237114671 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f081538a0 msgr2=0x7f6f080b3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:24.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f0e866640 1 --2- 172.21.15.125:0/2237114671 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f081538a0 0x7f6f080b3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:24.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f0e866640 1 -- 172.21.15.125:0/2237114671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f080cf160 msgr2=0x7f6f080ce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:24.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f0e866640 1 --2- 172.21.15.125:0/2237114671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f080cf160 0x7f6f080ce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:24.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f0e866640 1 -- 172.21.15.125:0/2237114671 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f00009a70 con 0x7f6f0815d6f0 2023-10-30T12:58:24.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f0e866640 1 --2- 172.21.15.125:0/2237114671 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f0815d6f0 0x7f6f08148690 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f6f000b2040 tx=0x7f6f0001bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:24.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6efeffd640 1 -- 172.21.15.125:0/2237114671 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f00009450 con 0x7f6f0815d6f0 2023-10-30T12:58:24.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6efeffd640 1 -- 172.21.15.125:0/2237114671 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f000b9410 con 0x7f6f0815d6f0 2023-10-30T12:58:24.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f102c9640 1 -- 172.21.15.125:0/2237114671 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f08149fb0 con 0x7f6f0815d6f0 2023-10-30T12:58:24.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6efeffd640 1 -- 172.21.15.125:0/2237114671 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f000b97c0 con 0x7f6f0815d6f0 2023-10-30T12:58:24.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f102c9640 1 -- 172.21.15.125:0/2237114671 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f080ce680 con 0x7f6f0815d6f0 2023-10-30T12:58:24.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.183+0000 7f6f102c9640 1 -- 172.21.15.125:0/2237114671 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f081483b0 con 0x7f6f0815d6f0 2023-10-30T12:58:24.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.187+0000 7f6efeffd640 1 -- 172.21.15.125:0/2237114671 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6f000b6050 con 0x7f6f0815d6f0 2023-10-30T12:58:24.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.187+0000 7f6efeffd640 1 --2- 172.21.15.125:0/2237114671 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6eec0664a0 0x7f6eec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:24.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.187+0000 7f6efeffd640 1 -- 172.21.15.125:0/2237114671 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f6f00122870 con 0x7f6f0815d6f0 2023-10-30T12:58:24.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.187+0000 7f6f0e065640 1 --2- 172.21.15.125:0/2237114671 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6eec0664a0 0x7f6eec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:24.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.187+0000 7f6f0e065640 1 --2- 172.21.15.125:0/2237114671 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6eec0664a0 0x7f6eec068960 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6f040011f0 tx=0x7f6f04009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:24.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.191+0000 7f6efeffd640 1 -- 172.21.15.125:0/2237114671 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6f081483b0 con 0x7f6f0815d6f0 2023-10-30T12:58:24.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.371+0000 7f6f102c9640 1 -- 172.21.15.125:0/2237114671 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f081483b0 con 0x7f6f0815d6f0 2023-10-30T12:58:24.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.371+0000 7f6efeffd640 1 -- 172.21.15.125:0/2237114671 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f6f081483b0 con 0x7f6f0815d6f0 2023-10-30T12:58:24.371 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:58:24.371 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7554,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:19.758581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":22,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:58:24.372 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 139 2023-10-30T12:58:24.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.371+0000 7f6f102c9640 1 -- 172.21.15.125:0/2237114671 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6eec0664a0 msgr2=0x7f6eec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:24.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.371+0000 7f6f102c9640 1 --2- 172.21.15.125:0/2237114671 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6eec0664a0 0x7f6eec068960 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f6f040011f0 tx=0x7f6f04009040 comp rx=0 tx=0).stop 2023-10-30T12:58:24.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.371+0000 7f6f102c9640 1 -- 172.21.15.125:0/2237114671 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f0815d6f0 msgr2=0x7f6f08148690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:24.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.371+0000 7f6f102c9640 1 --2- 172.21.15.125:0/2237114671 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f0815d6f0 0x7f6f08148690 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f6f000b2040 tx=0x7f6f0001bed0 comp rx=0 tx=0).stop 2023-10-30T12:58:24.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.371+0000 7f6f102c9640 1 -- 172.21.15.125:0/2237114671 shutdown_connections 2023-10-30T12:58:24.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.371+0000 7f6f102c9640 1 --2- 172.21.15.125:0/2237114671 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6eec0664a0 0x7f6eec068960 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:24.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.371+0000 7f6f102c9640 1 --2- 172.21.15.125:0/2237114671 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6f0815d6f0 0x7f6f08148690 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:24.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.371+0000 7f6f102c9640 1 --2- 172.21.15.125:0/2237114671 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6f081538a0 0x7f6f080b3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:24.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.371+0000 7f6f102c9640 1 --2- 172.21.15.125:0/2237114671 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6f080cf160 0x7f6f080ce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:24.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.371+0000 7f6f102c9640 1 -- 172.21.15.125:0/2237114671 >> 172.21.15.125:0/2237114671 conn(0x7f6f0806bfe0 msgr2=0x7f6f080c5610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:24.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.371+0000 7f6f102c9640 1 -- 172.21.15.125:0/2237114671 shutdown_connections 2023-10-30T12:58:24.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:24.371+0000 7f6f102c9640 1 -- 172.21.15.125:0/2237114671 wait complete. 2023-10-30T12:58:24.391 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T12:58:24.391 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 33 secs before thrashing 2023-10-30T12:58:26.678 INFO:tasks.ceph.mds.c.smithi125.stderr:2023-10-30T12:58:26.679+0000 7fdb9571c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:58:28.035 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 48% (75/154) Updating files: 49% (76/154) Updating files: 50% (77/154) Updating files: 51% (79/154) Updating files: 52% (81/154) Updating files: 53% (82/154) Updating files: 54% (84/154) Updating files: 55% (85/154) Updating files: 56% (87/154) Updating files: 57% (88/154) Updating files: 58% (90/154) Updating files: 59% (91/154) Updating files: 60% (93/154) Updating files: 61% (94/154) Updating files: 62% (96/154) Updating files: 63% (98/154) Updating files: 64% (99/154) Updating files: 65% (101/154) Updating files: 66% (102/154) Updating files: 67% (104/154) Updating files: 68% (105/154) Updating files: 69% (107/154) Updating files: 70% (108/154) Updating files: 71% (110/154) Updating files: 72% (111/154) Updating files: 73% (113/154) Updating files: 74% (114/154) Updating files: 75% (116/154) Updating files: 76% (118/154) Updating files: 77% (119/154) Updating files: 78% (121/154) Updating files: 79% (122/154) Updating files: 80% (124/154) Updating files: 81% (125/154) Updating files: 82% (127/154) Updating files: 83% (128/154) Updating files: 84% (130/154) Updating files: 85% (131/154) Updating files: 86% (133/154) Updating files: 87% (134/154) Updating files: 88% (136/154) Updating files: 89% (138/154) Updating files: 90% (139/154) Updating files: 91% (141/154) Updating files: 92% (142/154) Updating files: 93% (144/154) Updating files: 94% (145/154) Updating files: 95% (147/154) Updating files: 96% (148/154) Updating files: 97% (150/154) Updating files: 98% (151/154) Updating files: 99% (153/154) Updating files: 100% (154/154) Updating files: 100% (154/154), done. 2023-10-30T12:58:28.941 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 0aea1cb1df5 v0.45 2023-10-30T12:58:28.944 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.45 2023-10-30T12:58:29.098 INFO:tasks.workunit.client.0.smithi125.stdout:v0.46 2023-10-30T12:58:29.098 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 46 -eq 48 ] 2023-10-30T12:58:29.099 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.46 2023-10-30T12:58:29.099 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.46 2023-10-30T12:58:29.099 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.46 2023-10-30T12:58:37.262 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 36% (113/309) Updating files: 37% (115/309) Updating files: 38% (118/309) Updating files: 39% (121/309) Updating files: 40% (124/309) Updating files: 41% (127/309) Updating files: 42% (130/309) Updating files: 43% (133/309) Updating files: 44% (136/309) Updating files: 45% (140/309) Updating files: 46% (143/309) Updating files: 47% (146/309) Updating files: 48% (149/309) Updating files: 49% (152/309) Updating files: 50% (155/309) Updating files: 51% (158/309) Updating files: 52% (161/309) Updating files: 53% (164/309) Updating files: 54% (167/309) Updating files: 55% (170/309) Updating files: 56% (174/309) Updating files: 57% (177/309) Updating files: 58% (180/309) Updating files: 59% (183/309) Updating files: 60% (186/309) Updating files: 61% (189/309) Updating files: 62% (192/309) Updating files: 63% (195/309) Updating files: 64% (198/309) Updating files: 65% (201/309) Updating files: 66% (204/309) Updating files: 67% (208/309) Updating files: 68% (211/309) Updating files: 69% (214/309) Updating files: 70% (217/309) Updating files: 71% (220/309) Updating files: 72% (223/309) Updating files: 73% (226/309) Updating files: 74% (229/309) Updating files: 75% (232/309) Updating files: 76% (235/309) Updating files: 77% (238/309) Updating files: 78% (242/309) Updating files: 79% (245/309) Updating files: 80% (248/309) Updating files: 81% (251/309) Updating files: 82% (254/309) Updating files: 83% (257/309) Updating files: 84% (260/309) Updating files: 85% (263/309) Updating files: 86% (266/309) Updating files: 87% (269/309) Updating files: 88% (272/309) Updating files: 89% (276/309) Updating files: 90% (279/309) Updating files: 91% (282/309) Updating files: 92% (285/309) Updating files: 93% (288/309) Updating files: 94% (291/309) Updating files: 95% (294/309) Updating files: 96% (297/309) Updating files: 97% (300/309) Updating files: 98% (303/309) Updating files: 99% (306/309) Updating files: 100% (309/309) Updating files: 100% (309/309), done. 2023-10-30T12:58:37.959 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at cb7f1c9c752 v0.46 2023-10-30T12:58:37.963 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.46 2023-10-30T12:58:38.099 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 47 -eq 48 ] 2023-10-30T12:58:38.099 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.47 2023-10-30T12:58:38.100 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.47 2023-10-30T12:58:38.100 INFO:tasks.workunit.client.0.smithi125.stdout:v0.47 2023-10-30T12:58:38.100 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.47 2023-10-30T12:58:40.686 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 84% (185/219) Updating files: 85% (187/219) Updating files: 86% (189/219) Updating files: 87% (191/219) Updating files: 88% (193/219) Updating files: 89% (195/219) Updating files: 90% (198/219) Updating files: 91% (200/219) Updating files: 92% (202/219) Updating files: 93% (204/219) Updating files: 94% (206/219) Updating files: 95% (209/219) Updating files: 96% (211/219) Updating files: 97% (213/219) Updating files: 98% (215/219) Updating files: 99% (217/219) Updating files: 100% (219/219) Updating files: 100% (219/219), done. 2023-10-30T12:58:40.973 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at ad663d58c89 v0.47 2023-10-30T12:58:40.976 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.47 2023-10-30T12:58:41.018 INFO:tasks.workunit.client.0.smithi125.stdout:v0.49 2023-10-30T12:58:41.018 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 48 -eq 48 ] 2023-10-30T12:58:41.018 INFO:tasks.workunit.client.0.smithi125.stderr:+ continue 2023-10-30T12:58:41.019 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 49 -eq 48 ] 2023-10-30T12:58:41.019 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.49 2023-10-30T12:58:41.019 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.49 2023-10-30T12:58:41.019 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.49 2023-10-30T12:58:50.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:58:50.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:58:52.812 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 38% (187/481) Updating files: 39% (188/481) Updating files: 40% (193/481) Updating files: 41% (198/481) Updating files: 42% (203/481) Updating files: 43% (207/481) Updating files: 44% (212/481) Updating files: 45% (217/481) Updating files: 46% (222/481) Updating files: 47% (227/481) Updating files: 48% (231/481) Updating files: 49% (236/481) Updating files: 49% (240/481) Updating files: 50% (241/481) Updating files: 51% (246/481) Updating files: 52% (251/481) Updating files: 53% (255/481) Updating files: 54% (260/481) Updating files: 55% (265/481) Updating files: 56% (270/481) Updating files: 57% (275/481) Updating files: 58% (279/481) Updating files: 59% (284/481) Updating files: 60% (289/481) Updating files: 61% (294/481) Updating files: 62% (299/481) Updating files: 63% (304/481) Updating files: 64% (308/481) Updating files: 65% (313/481) Updating files: 66% (318/481) Updating files: 67% (323/481) Updating files: 68% (328/481) Updating files: 69% (332/481) Updating files: 70% (337/481) Updating files: 71% (342/481) Updating files: 71% (344/481) Updating files: 72% (347/481) Updating files: 73% (352/481) Updating files: 74% (356/481) Updating files: 74% (358/481) Updating files: 75% (361/481) Updating files: 76% (366/481) Updating files: 77% (371/481) Updating files: 78% (376/481) Updating files: 79% (380/481) Updating files: 80% (385/481) Updating files: 81% (390/481) Updating files: 82% (395/481) Updating files: 83% (400/481) Updating files: 84% (405/481) Updating files: 85% (409/481) Updating files: 86% (414/481) Updating files: 87% (419/481) Updating files: 88% (424/481) Updating files: 89% (429/481) Updating files: 90% (433/481) Updating files: 91% (438/481) Updating files: 92% (443/481) Updating files: 93% (448/481) Updating files: 93% (451/481) Updating files: 94% (453/481) Updating files: 95% (457/481) Updating files: 96% (462/481) Updating files: 97% (467/481) Updating files: 98% (472/481) Updating files: 99% (477/481) Updating files: 100% (481/481) Updating files: 100% (481/481), done. 2023-10-30T12:58:53.308 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at ca6265d0f4d v0.49 2023-10-30T12:58:53.312 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.49 2023-10-30T12:58:53.351 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 50 -eq 48 ] 2023-10-30T12:58:53.352 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.50 2023-10-30T12:58:53.352 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.50 2023-10-30T12:58:53.352 INFO:tasks.workunit.client.0.smithi125.stdout:v0.50 2023-10-30T12:58:53.353 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.50 2023-10-30T12:58:57.393 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:58:57.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.535+0000 7f29c04c9640 1 Processor -- start 2023-10-30T12:58:57.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.535+0000 7f29c04c9640 1 -- start start 2023-10-30T12:58:57.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29c04c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29b81538a0 0x7f29b815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:57.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29c04c9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f29b80cf160 0x7f29b80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:57.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29c04c9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b80cb800 con 0x7f29b80ca0e0 2023-10-30T12:58:57.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29c04c9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b815dc30 con 0x7f29b80cf160 2023-10-30T12:58:57.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29c04c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b80cb330 con 0x7f29b81538a0 2023-10-30T12:58:57.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29bea66640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f29b80cf160 0x7f29b80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:57.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29bea66640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f29b80cf160 0x7f29b80cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:34950/0 (socket says 172.21.15.125:34950) 2023-10-30T12:58:57.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29bea66640 1 -- 172.21.15.125:0/2620136310 learned_addr learned my addr 172.21.15.125:0/2620136310 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:57.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29bda64640 1 --1- 172.21.15.125:0/2620136310 >> v1:172.21.15.136:6789/0 conn(0x7f29b80ca0e0 0x7f29b815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:50978/0 (socket says 172.21.15.125:50978) 2023-10-30T12:58:57.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29be265640 1 --2- 172.21.15.125:0/2620136310 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29b81538a0 0x7f29b815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:57.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29bea66640 1 -- 172.21.15.125:0/2620136310 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29b81538a0 msgr2=0x7f29b815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:57.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29bea66640 1 --2- 172.21.15.125:0/2620136310 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29b81538a0 0x7f29b815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:57.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29bea66640 1 -- 172.21.15.125:0/2620136310 >> v1:172.21.15.136:6789/0 conn(0x7f29b80ca0e0 legacy=0x7f29b815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:57.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29bea66640 1 -- 172.21.15.125:0/2620136310 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b80cad70 con 0x7f29b80cf160 2023-10-30T12:58:57.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.539+0000 7f29be265640 1 --2- 172.21.15.125:0/2620136310 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29b81538a0 0x7f29b815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:58:57.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.543+0000 7f29bea66640 1 --2- 172.21.15.125:0/2620136310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29b80cf160 0x7f29b80cf590 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009e50 tx=0x7f29b001f1f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e68c1f6058aac9b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:57.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.543+0000 7f29bd263640 1 -- 172.21.15.125:0/2620136310 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b00af400 con 0x7f29b80cf160 2023-10-30T12:58:57.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.543+0000 7f29bd263640 1 -- 172.21.15.125:0/2620136310 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b00ab050 con 0x7f29b80cf160 2023-10-30T12:58:57.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.543+0000 7f29bd263640 1 -- 172.21.15.125:0/2620136310 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b00af780 con 0x7f29b80cf160 2023-10-30T12:58:57.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.543+0000 7f29c04c9640 1 -- 172.21.15.125:0/2620136310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29b80cf160 msgr2=0x7f29b80cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:57.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.543+0000 7f29c04c9640 1 --2- 172.21.15.125:0/2620136310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29b80cf160 0x7f29b80cf590 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f29b0009e50 tx=0x7f29b001f1f0 comp rx=0 tx=0).stop 2023-10-30T12:58:57.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.543+0000 7f29c04c9640 1 -- 172.21.15.125:0/2620136310 shutdown_connections 2023-10-30T12:58:57.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.543+0000 7f29c04c9640 1 --2- 172.21.15.125:0/2620136310 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29b80cf160 0x7f29b80cf590 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.543+0000 7f29c04c9640 1 --2- 172.21.15.125:0/2620136310 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29b81538a0 0x7f29b815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.543+0000 7f29c04c9640 1 -- 172.21.15.125:0/2620136310 >> 172.21.15.125:0/2620136310 conn(0x7f29b806bfe0 msgr2=0x7f29b806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.543+0000 7f29c04c9640 1 -- 172.21.15.125:0/2620136310 shutdown_connections 2023-10-30T12:58:57.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.543+0000 7f29c04c9640 1 -- 172.21.15.125:0/2620136310 wait complete. 2023-10-30T12:58:57.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.547+0000 7f29c04c9640 1 Processor -- start 2023-10-30T12:58:57.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.547+0000 7f29c04c9640 1 -- start start 2023-10-30T12:58:57.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.547+0000 7f29c04c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29b80ca0e0 0x7f29b81efd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:57.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.547+0000 7f29c04c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29b80cf160 0x7f29b81483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:57.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.547+0000 7f29c04c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29b81538a0 0x7f29b8142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:57.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.547+0000 7f29c04c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b81ef930 con 0x7f29b80ca0e0 2023-10-30T12:58:57.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.547+0000 7f29c04c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b81455c0 con 0x7f29b80cf160 2023-10-30T12:58:57.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.547+0000 7f29c04c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b8145910 con 0x7f29b81538a0 2023-10-30T12:58:57.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.547+0000 7f29be265640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29b80ca0e0 0x7f29b81efd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:57.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.547+0000 7f29be265640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29b80ca0e0 0x7f29b81efd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38012/0 (socket says 172.21.15.125:38012) 2023-10-30T12:58:57.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.547+0000 7f29be265640 1 -- 172.21.15.125:0/287721912 learned_addr learned my addr 172.21.15.125:0/287721912 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:57.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.547+0000 7f29bda64640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29b80cf160 0x7f29b81483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:57.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.547+0000 7f29bea66640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29b81538a0 0x7f29b8142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:57.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.551+0000 7f29be265640 1 -- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29b81538a0 msgr2=0x7f29b8142cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:57.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.551+0000 7f29be265640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29b81538a0 0x7f29b8142cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:57.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.551+0000 7f29be265640 1 -- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29b80cf160 msgr2=0x7f29b81483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:57.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.551+0000 7f29be265640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29b80cf160 0x7f29b81483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:57.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.551+0000 7f29be265640 1 -- 172.21.15.125:0/287721912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b4009420 con 0x7f29b80ca0e0 2023-10-30T12:58:57.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.551+0000 7f29bea66640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29b81538a0 0x7f29b8142cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:58:57.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.551+0000 7f29be265640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29b80ca0e0 0x7f29b81efd50 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f29b400e9b0 tx=0x7f29b400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:57.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.551+0000 7f29aeffd640 1 -- 172.21.15.125:0/287721912 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b400ca00 con 0x7f29b80ca0e0 2023-10-30T12:58:57.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.551+0000 7f29aeffd640 1 -- 172.21.15.125:0/287721912 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29b400cb60 con 0x7f29b80ca0e0 2023-10-30T12:58:57.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.551+0000 7f29aeffd640 1 -- 172.21.15.125:0/287721912 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29b4013470 con 0x7f29b80ca0e0 2023-10-30T12:58:57.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.551+0000 7f29c04c9640 1 -- 172.21.15.125:0/287721912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b0009b30 con 0x7f29b80ca0e0 2023-10-30T12:58:57.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.551+0000 7f29c04c9640 1 -- 172.21.15.125:0/287721912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b815bf70 con 0x7f29b80ca0e0 2023-10-30T12:58:57.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.551+0000 7f29c04c9640 1 -- 172.21.15.125:0/287721912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b81efba0 con 0x7f29b80ca0e0 2023-10-30T12:58:57.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.551+0000 7f29aeffd640 1 -- 172.21.15.125:0/287721912 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29b40135d0 con 0x7f29b80ca0e0 2023-10-30T12:58:57.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.555+0000 7f29aeffd640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f299c0664a0 0x7f299c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:57.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.555+0000 7f29aeffd640 1 -- 172.21.15.125:0/287721912 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f29b4084730 con 0x7f29b80ca0e0 2023-10-30T12:58:57.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.555+0000 7f29bda64640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f299c0664a0 0x7f299c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:57.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.555+0000 7f29bda64640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f299c0664a0 0x7f299c068960 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f29b8148e00 tx=0x7f29a8007cf0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:57.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.559+0000 7f29aeffd640 1 -- 172.21.15.125:0/287721912 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29b4051550 con 0x7f29b80ca0e0 2023-10-30T12:58:57.561 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 10% (14/137) Updating files: 11% (16/137) Updating files: 12% (17/137) Updating files: 13% (18/137) Updating files: 14% (20/137) Updating files: 15% (21/137) Updating files: 16% (22/137) Updating files: 17% (24/137) Updating files: 18% (25/137) Updating files: 19% (27/137) Updating files: 20% (28/137) Updating files: 21% (29/137) Updating files: 22% (31/137) Updating files: 23% (32/137) Updating files: 24% (33/137) Updating files: 25% (35/137) Updating files: 26% (36/137) Updating files: 27% (37/137) Updating files: 28% (39/137) Updating files: 29% (40/137) Updating files: 30% (42/137) Updating files: 31% (43/137) Updating files: 32% (44/137) Updating files: 33% (46/137) Updating files: 34% (47/137) Updating files: 35% (48/137) Updating files: 36% (50/137) Updating files: 37% (51/137) Updating files: 38% (53/137) Updating files: 39% (54/137) Updating files: 40% (55/137) Updating files: 41% (57/137) Updating files: 42% (58/137) Updating files: 43% (59/137) Updating files: 44% (61/137) Updating files: 45% (62/137) Updating files: 46% (64/137) Updating files: 47% (65/137) Updating files: 48% (66/137) Updating files: 49% (68/137) Updating files: 50% (69/137) Updating files: 51% (70/137) Updating files: 52% (72/137) Updating files: 53% (73/137) Updating files: 54% (74/137) Updating files: 55% (76/137) Updating files: 56% (77/137) Updating files: 57% (79/137) Updating files: 58% (80/137) Updating files: 59% (81/137) Updating files: 60% (83/137) Updating files: 61% (84/137) Updating files: 62% (85/137) Updating files: 63% (87/137) Updating files: 64% (88/137) Updating files: 65% (90/137) Updating files: 66% (91/137) Updating files: 67% (92/137) Updating files: 68% (94/137) Updating files: 69% (95/137) Updating files: 70% (96/137) Updating files: 71% (98/137) Updating files: 72% (99/137) Updating files: 73% (101/137) Updating files: 74% (102/137) Updating files: 75% (103/137) Updating files: 76% (105/137) Updating files: 77% (106/137) Updating files: 78% (107/137) Updating files: 79% (109/137) Updating files: 80% (110/137) Updating files: 81% (111/137) Updating files: 82% (113/137) Updating files: 83% (114/137) Updating files: 84% (116/137) Updating files: 85% (117/137) Updating files: 86% (118/137) Updating files: 87% (120/137) Updating files: 88% (121/137) Updating files: 89% (122/137) Updating files: 90% (124/137) Updating files: 91% (125/137) Updating files: 92% (127/137) Updating files: 93% (128/137) Updating files: 94% (129/137) Updating files: 95% (131/137) Updating files: 96% (132/137) Updating files: 97% (133/137) Updating files: 98% (135/137) Updating files: 99% (136/137) Updating files: 100% (137/137) Updating files: 100% (137/137), done. 2023-10-30T12:58:57.600 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at a4428bd0e19 v0.50 2023-10-30T12:58:57.603 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.50 2023-10-30T12:58:57.699 INFO:tasks.workunit.client.0.smithi125.stdout:v0.51 2023-10-30T12:58:57.700 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 51 -eq 48 ] 2023-10-30T12:58:57.700 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.51 2023-10-30T12:58:57.700 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.51 2023-10-30T12:58:57.700 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.51 2023-10-30T12:58:57.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.851+0000 7f29c04c9640 1 -- 172.21.15.125:0/287721912 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29b80b2a50 con 0x7f29b80ca0e0 2023-10-30T12:58:57.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.851+0000 7f29aeffd640 1 -- 172.21.15.125:0/287721912 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f29b4054b70 con 0x7f29b80ca0e0 2023-10-30T12:58:57.855 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:58:57.855 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7554,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:19.758581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":22,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:58:57.857 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 139 2023-10-30T12:58:57.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.855+0000 7f29c04c9640 1 -- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f299c0664a0 msgr2=0x7f299c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:57.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.855+0000 7f29c04c9640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f299c0664a0 0x7f299c068960 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f29b8148e00 tx=0x7f29a8007cf0 comp rx=0 tx=0).stop 2023-10-30T12:58:57.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.855+0000 7f29c04c9640 1 -- 172.21.15.125:0/287721912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29b80ca0e0 msgr2=0x7f29b81efd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:57.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.855+0000 7f29c04c9640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29b80ca0e0 0x7f29b81efd50 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f29b400e9b0 tx=0x7f29b400ee70 comp rx=0 tx=0).stop 2023-10-30T12:58:57.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.855+0000 7f29c04c9640 1 -- 172.21.15.125:0/287721912 shutdown_connections 2023-10-30T12:58:57.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.855+0000 7f29c04c9640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f299c0664a0 0x7f299c068960 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:57.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.855+0000 7f29c04c9640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29b81538a0 0x7f29b8142cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:57.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.855+0000 7f29c04c9640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29b80cf160 0x7f29b81483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:57.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.855+0000 7f29c04c9640 1 --2- 172.21.15.125:0/287721912 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29b80ca0e0 0x7f29b81efd50 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:57.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.855+0000 7f29c04c9640 1 -- 172.21.15.125:0/287721912 >> 172.21.15.125:0/287721912 conn(0x7f29b806bfe0 msgr2=0x7f29b80c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:57.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.855+0000 7f29c04c9640 1 -- 172.21.15.125:0/287721912 shutdown_connections 2023-10-30T12:58:57.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:57.855+0000 7f29c04c9640 1 -- 172.21.15.125:0/287721912 wait complete. 2023-10-30T12:58:57.877 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T12:58:57.877 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T12:58:57.878 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T12:58:57.906 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T12:58:57.907 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:58:58.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f296753e640 1 Processor -- start 2023-10-30T12:58:58.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f296753e640 1 -- start start 2023-10-30T12:58:58.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f296753e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f29601539a0 0x7f296015a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:58.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f296753e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2960070a00 0x7f2960070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:58.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f296753e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296015a8c0 con 0x7f2960069e90 2023-10-30T12:58:58.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f296753e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f296006fbb0 con 0x7f29601539a0 2023-10-30T12:58:58.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f296753e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29600713b0 con 0x7f2960070a00 2023-10-30T12:58:58.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f29652da640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f29601539a0 0x7f296015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:58.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f29652da640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f29601539a0 0x7f296015a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38022/0 (socket says 172.21.15.125:38022) 2023-10-30T12:58:58.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f29652da640 1 -- 172.21.15.125:0/4059151051 learned_addr learned my addr 172.21.15.125:0/4059151051 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:58.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f2964ad9640 1 --2- 172.21.15.125:0/4059151051 >> v2:172.21.15.136:3300/0 conn(0x7f2960070a00 0x7f2960070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:58.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f29652da640 1 -- 172.21.15.125:0/4059151051 >> v2:172.21.15.136:3300/0 conn(0x7f2960070a00 msgr2=0x7f2960070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:58.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f29652da640 1 --2- 172.21.15.125:0/4059151051 >> v2:172.21.15.136:3300/0 conn(0x7f2960070a00 0x7f2960070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:58.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f29652da640 1 -- 172.21.15.125:0/4059151051 >> v1:172.21.15.136:6789/0 conn(0x7f2960069e90 legacy=0x7f296006a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:58:58.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.091+0000 7f29652da640 1 -- 172.21.15.125:0/4059151051 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2960153d80 con 0x7f29601539a0 2023-10-30T12:58:58.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.095+0000 7f29652da640 1 --2- 172.21.15.125:0/4059151051 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29601539a0 0x7f296015a380 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f2960072640 tx=0x7f295c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=388e3ed023439715 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:58.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.095+0000 7f2957fff640 1 -- 172.21.15.125:0/4059151051 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c0ab600 con 0x7f29601539a0 2023-10-30T12:58:58.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.095+0000 7f2957fff640 1 -- 172.21.15.125:0/4059151051 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c0ab760 con 0x7f29601539a0 2023-10-30T12:58:58.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.095+0000 7f2957fff640 1 -- 172.21.15.125:0/4059151051 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c0abaf0 con 0x7f29601539a0 2023-10-30T12:58:58.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.095+0000 7f296753e640 1 -- 172.21.15.125:0/4059151051 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29601539a0 msgr2=0x7f296015a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:58.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.095+0000 7f296753e640 1 --2- 172.21.15.125:0/4059151051 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29601539a0 0x7f296015a380 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f2960072640 tx=0x7f295c01c400 comp rx=0 tx=0).stop 2023-10-30T12:58:58.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.095+0000 7f296753e640 1 -- 172.21.15.125:0/4059151051 shutdown_connections 2023-10-30T12:58:58.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.095+0000 7f296753e640 1 --2- 172.21.15.125:0/4059151051 >> v2:172.21.15.136:3300/0 conn(0x7f2960070a00 0x7f2960070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:58.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.095+0000 7f296753e640 1 --2- 172.21.15.125:0/4059151051 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29601539a0 0x7f296015a380 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:58.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.095+0000 7f296753e640 1 -- 172.21.15.125:0/4059151051 >> 172.21.15.125:0/4059151051 conn(0x7f29600c7910 msgr2=0x7f29600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:58.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.095+0000 7f296753e640 1 -- 172.21.15.125:0/4059151051 shutdown_connections 2023-10-30T12:58:58.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.095+0000 7f296753e640 1 -- 172.21.15.125:0/4059151051 wait complete. 2023-10-30T12:58:58.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f296753e640 1 Processor -- start 2023-10-30T12:58:58.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f296753e640 1 -- start start 2023-10-30T12:58:58.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f296753e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2960069e90 0x7f2960147b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:58.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f296753e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2960070a00 0x7f29600736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:58.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f296753e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29601539a0 0x7f29601f5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:58.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f296753e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29601582f0 con 0x7f29601539a0 2023-10-30T12:58:58.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f296753e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2960148080 con 0x7f2960069e90 2023-10-30T12:58:58.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f296753e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29601482f0 con 0x7f2960070a00 2023-10-30T12:58:58.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f2965adb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29601539a0 0x7f29601f5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:58.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f2965adb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29601539a0 0x7f29601f5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38032/0 (socket says 172.21.15.125:38032) 2023-10-30T12:58:58.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f2965adb640 1 -- 172.21.15.125:0/2041403434 learned_addr learned my addr 172.21.15.125:0/2041403434 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:58:58.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f29652da640 1 --2- 172.21.15.125:0/2041403434 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2960069e90 0x7f2960147b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:58.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f2964ad9640 1 --2- 172.21.15.125:0/2041403434 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2960070a00 0x7f29600736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:58.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f2965adb640 1 -- 172.21.15.125:0/2041403434 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2960070a00 msgr2=0x7f29600736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:58.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f2965adb640 1 --2- 172.21.15.125:0/2041403434 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2960070a00 0x7f29600736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:58.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f2965adb640 1 -- 172.21.15.125:0/2041403434 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2960069e90 msgr2=0x7f2960147b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:58.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f2965adb640 1 --2- 172.21.15.125:0/2041403434 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2960069e90 0x7f2960147b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:58.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f2965adb640 1 -- 172.21.15.125:0/2041403434 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c0094d0 con 0x7f29601539a0 2023-10-30T12:58:58.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f2965adb640 1 --2- 172.21.15.125:0/2041403434 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29601539a0 0x7f29601f5c40 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f295800c780 tx=0x7f295800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:58.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f2955ffb640 1 -- 172.21.15.125:0/2041403434 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2958004b70 con 0x7f29601539a0 2023-10-30T12:58:58.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f2955ffb640 1 -- 172.21.15.125:0/2041403434 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295800ce90 con 0x7f29601539a0 2023-10-30T12:58:58.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f2955ffb640 1 -- 172.21.15.125:0/2041403434 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295800a850 con 0x7f29601539a0 2023-10-30T12:58:58.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f296753e640 1 -- 172.21.15.125:0/2041403434 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29601595c0 con 0x7f29601539a0 2023-10-30T12:58:58.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f296753e640 1 -- 172.21.15.125:0/2041403434 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2960157950 con 0x7f29601539a0 2023-10-30T12:58:58.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.099+0000 7f296753e640 1 -- 172.21.15.125:0/2041403434 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296004e170 con 0x7f29601539a0 2023-10-30T12:58:58.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.103+0000 7f2955ffb640 1 -- 172.21.15.125:0/2041403434 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f295800a9b0 con 0x7f29601539a0 2023-10-30T12:58:58.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.103+0000 7f2955ffb640 1 --2- 172.21.15.125:0/2041403434 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29440664a0 0x7f2944068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:58:58.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.103+0000 7f2955ffb640 1 -- 172.21.15.125:0/2041403434 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f2958084530 con 0x7f29601539a0 2023-10-30T12:58:58.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.103+0000 7f29652da640 1 --2- 172.21.15.125:0/2041403434 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29440664a0 0x7f2944068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:58:58.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.103+0000 7f29652da640 1 --2- 172.21.15.125:0/2041403434 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29440664a0 0x7f2944068960 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f295c0af000 tx=0x7f295c0a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:58:58.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.107+0000 7f2955ffb640 1 -- 172.21.15.125:0/2041403434 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2958051470 con 0x7f29601539a0 2023-10-30T12:58:58.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.283+0000 7f296753e640 1 -- 172.21.15.125:0/2041403434 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f296014def0 con 0x7f29601539a0 2023-10-30T12:58:58.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.287+0000 7f2955ffb640 1 -- 172.21.15.125:0/2041403434 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f2958054a90 con 0x7f29601539a0 2023-10-30T12:58:58.287 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:58:58.287 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7554,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:19.758581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":22,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:58:58.287 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 139 2023-10-30T12:58:58.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.287+0000 7f296753e640 1 -- 172.21.15.125:0/2041403434 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29440664a0 msgr2=0x7f2944068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:58.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.287+0000 7f296753e640 1 --2- 172.21.15.125:0/2041403434 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29440664a0 0x7f2944068960 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f295c0af000 tx=0x7f295c0a7000 comp rx=0 tx=0).stop 2023-10-30T12:58:58.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.287+0000 7f296753e640 1 -- 172.21.15.125:0/2041403434 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29601539a0 msgr2=0x7f29601f5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:58:58.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.287+0000 7f296753e640 1 --2- 172.21.15.125:0/2041403434 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29601539a0 0x7f29601f5c40 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f295800c780 tx=0x7f295800cc40 comp rx=0 tx=0).stop 2023-10-30T12:58:58.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.287+0000 7f296753e640 1 -- 172.21.15.125:0/2041403434 shutdown_connections 2023-10-30T12:58:58.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.287+0000 7f296753e640 1 --2- 172.21.15.125:0/2041403434 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29440664a0 0x7f2944068960 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:58.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.287+0000 7f296753e640 1 --2- 172.21.15.125:0/2041403434 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29601539a0 0x7f29601f5c40 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:58.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.287+0000 7f296753e640 1 --2- 172.21.15.125:0/2041403434 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2960070a00 0x7f29600736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:58.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.287+0000 7f296753e640 1 --2- 172.21.15.125:0/2041403434 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2960069e90 0x7f2960147b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:58:58.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.287+0000 7f296753e640 1 -- 172.21.15.125:0/2041403434 >> 172.21.15.125:0/2041403434 conn(0x7f29600c7910 msgr2=0x7f296014e110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:58:58.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.287+0000 7f296753e640 1 -- 172.21.15.125:0/2041403434 shutdown_connections 2023-10-30T12:58:58.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:58:58.287+0000 7f296753e640 1 -- 172.21.15.125:0/2041403434 wait complete. 2023-10-30T12:58:58.307 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:59:00.309 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:00.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc520899640 1 Processor -- start 2023-10-30T12:59:00.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc520899640 1 -- start start 2023-10-30T12:59:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc520899640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc51c1538f0 0x7fc51c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc520899640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc51c15cd50 0x7fc51c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc520899640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc51c0cefa0 0x7fc51c0cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:00.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc520899640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc51c15d300 con 0x7fc51c15cd50 2023-10-30T12:59:00.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc520899640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc51c15d5e0 con 0x7fc51c0cefa0 2023-10-30T12:59:00.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc520899640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c15d8c0 con 0x7fc51c1538f0 2023-10-30T12:59:00.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc51a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc51c15cd50 0x7fc51c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:00.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc51a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc51c15cd50 0x7fc51c0c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38048/0 (socket says 172.21.15.125:38048) 2023-10-30T12:59:00.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc51a59c640 1 -- 172.21.15.125:0/1126200329 learned_addr learned my addr 172.21.15.125:0/1126200329 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:00.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc51ad9d640 1 --2- 172.21.15.125:0/1126200329 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc51c1538f0 0x7fc51c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:00.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc51b59e640 1 --2- 172.21.15.125:0/1126200329 >> v2:172.21.15.136:3300/0 conn(0x7fc51c0cefa0 0x7fc51c0cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:00.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc51a59c640 1 -- 172.21.15.125:0/1126200329 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc51c1538f0 msgr2=0x7fc51c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:00.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc51a59c640 1 --2- 172.21.15.125:0/1126200329 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc51c1538f0 0x7fc51c159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:00.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc51a59c640 1 -- 172.21.15.125:0/1126200329 >> v2:172.21.15.136:3300/0 conn(0x7fc51c0cefa0 msgr2=0x7fc51c0cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:00.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc51a59c640 1 --2- 172.21.15.125:0/1126200329 >> v2:172.21.15.136:3300/0 conn(0x7fc51c0cefa0 0x7fc51c0cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:00.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc51a59c640 1 -- 172.21.15.125:0/1126200329 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc51c15dba0 con 0x7fc51c15cd50 2023-10-30T12:59:00.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc51ad9d640 1 --2- 172.21.15.125:0/1126200329 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc51c1538f0 0x7fc51c159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:59:00.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.451+0000 7fc51b59e640 1 --2- 172.21.15.125:0/1126200329 >> v2:172.21.15.136:3300/0 conn(0x7fc51c0cefa0 0x7fc51c0cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:59:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.455+0000 7fc51a59c640 1 --2- 172.21.15.125:0/1126200329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc51c15cd50 0x7fc51c0c9f90 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0b1ae0 tx=0x7fc51001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b8d19332aaf014d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.455+0000 7fc519d9b640 1 -- 172.21.15.125:0/1126200329 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5100a89b0 con 0x7fc51c15cd50 2023-10-30T12:59:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.455+0000 7fc519d9b640 1 -- 172.21.15.125:0/1126200329 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5100a8b10 con 0x7fc51c15cd50 2023-10-30T12:59:00.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.455+0000 7fc519d9b640 1 -- 172.21.15.125:0/1126200329 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5100a8ea0 con 0x7fc51c15cd50 2023-10-30T12:59:00.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.455+0000 7fc520899640 1 -- 172.21.15.125:0/1126200329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc51c15cd50 msgr2=0x7fc51c0c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:00.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.455+0000 7fc520899640 1 --2- 172.21.15.125:0/1126200329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc51c15cd50 0x7fc51c0c9f90 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fc51c0b1ae0 tx=0x7fc51001ecf0 comp rx=0 tx=0).stop 2023-10-30T12:59:00.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.455+0000 7fc520899640 1 -- 172.21.15.125:0/1126200329 shutdown_connections 2023-10-30T12:59:00.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.455+0000 7fc520899640 1 --2- 172.21.15.125:0/1126200329 >> v2:172.21.15.136:3300/0 conn(0x7fc51c0cefa0 0x7fc51c0cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:00.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.455+0000 7fc520899640 1 --2- 172.21.15.125:0/1126200329 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc51c15cd50 0x7fc51c0c9f90 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:00.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.455+0000 7fc520899640 1 --2- 172.21.15.125:0/1126200329 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc51c1538f0 0x7fc51c159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:00.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.455+0000 7fc520899640 1 -- 172.21.15.125:0/1126200329 >> 172.21.15.125:0/1126200329 conn(0x7fc51c06a640 msgr2=0x7fc51c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:00.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.455+0000 7fc520899640 1 -- 172.21.15.125:0/1126200329 shutdown_connections 2023-10-30T12:59:00.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.455+0000 7fc520899640 1 -- 172.21.15.125:0/1126200329 wait complete. 2023-10-30T12:59:00.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.455+0000 7fc520899640 1 Processor -- start 2023-10-30T12:59:00.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc520899640 1 -- start start 2023-10-30T12:59:00.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc520899640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc51c0cefa0 0x7fc51c14df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:00.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc520899640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc51c1538f0 0x7fc51c0b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:00.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc520899640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc51c15cd50 0x7fc51c0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:00.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc520899640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c0b3930 con 0x7fc51c1538f0 2023-10-30T12:59:00.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc520899640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c15c480 con 0x7fc51c15cd50 2023-10-30T12:59:00.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc520899640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc51c15c6e0 con 0x7fc51c0cefa0 2023-10-30T12:59:00.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc51a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc51c1538f0 0x7fc51c0b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:00.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc51a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc51c1538f0 0x7fc51c0b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38052/0 (socket says 172.21.15.125:38052) 2023-10-30T12:59:00.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc51a59c640 1 -- 172.21.15.125:0/2444993634 learned_addr learned my addr 172.21.15.125:0/2444993634 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:00.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc51b59e640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc51c15cd50 0x7fc51c0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:00.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc51ad9d640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc51c0cefa0 0x7fc51c14df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:00.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc51a59c640 1 -- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc51c0cefa0 msgr2=0x7fc51c14df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:00.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc51a59c640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc51c0cefa0 0x7fc51c14df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:00.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc51a59c640 1 -- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc51c15cd50 msgr2=0x7fc51c0b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:00.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc51a59c640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc51c15cd50 0x7fc51c0b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:00.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc51a59c640 1 -- 172.21.15.125:0/2444993634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5100094d0 con 0x7fc51c1538f0 2023-10-30T12:59:00.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc51b59e640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc51c15cd50 0x7fc51c0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:59:00.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.459+0000 7fc51a59c640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc51c1538f0 0x7fc51c0b3dc0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fc5100b3040 tx=0x7fc51001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:00.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.463+0000 7fc5037fe640 1 -- 172.21.15.125:0/2444993634 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5100b9620 con 0x7fc51c1538f0 2023-10-30T12:59:00.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.463+0000 7fc5037fe640 1 -- 172.21.15.125:0/2444993634 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5100b97a0 con 0x7fc51c1538f0 2023-10-30T12:59:00.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.463+0000 7fc520899640 1 -- 172.21.15.125:0/2444993634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc51c1efc80 con 0x7fc51c1538f0 2023-10-30T12:59:00.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.463+0000 7fc5037fe640 1 -- 172.21.15.125:0/2444993634 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5100b9b50 con 0x7fc51c1538f0 2023-10-30T12:59:00.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.463+0000 7fc520899640 1 -- 172.21.15.125:0/2444993634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51c0b32e0 con 0x7fc51c1538f0 2023-10-30T12:59:00.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.463+0000 7fc520899640 1 -- 172.21.15.125:0/2444993634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc51c0b3aa0 con 0x7fc51c1538f0 2023-10-30T12:59:00.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.463+0000 7fc5037fe640 1 -- 172.21.15.125:0/2444993634 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc5100b7070 con 0x7fc51c1538f0 2023-10-30T12:59:00.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.463+0000 7fc5037fe640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5080664a0 0x7fc508068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:00.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.463+0000 7fc5037fe640 1 -- 172.21.15.125:0/2444993634 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7fc510122460 con 0x7fc51c1538f0 2023-10-30T12:59:00.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.463+0000 7fc51ad9d640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5080664a0 0x7fc508068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:00.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.467+0000 7fc51ad9d640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5080664a0 0x7fc508068960 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00a9a0 tx=0x7fc50c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:00.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.471+0000 7fc5037fe640 1 -- 172.21.15.125:0/2444993634 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc5100ef310 con 0x7fc51c1538f0 2023-10-30T12:59:00.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.651+0000 7fc520899640 1 -- 172.21.15.125:0/2444993634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc51c04e170 con 0x7fc51c1538f0 2023-10-30T12:59:00.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.651+0000 7fc5037fe640 1 -- 172.21.15.125:0/2444993634 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fc5100f7e60 con 0x7fc51c1538f0 2023-10-30T12:59:00.652 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:00.652 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7554,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:19.758581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":22,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:00.653 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 139 2023-10-30T12:59:00.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.651+0000 7fc520899640 1 -- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5080664a0 msgr2=0x7fc508068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:00.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.651+0000 7fc520899640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5080664a0 0x7fc508068960 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc50c00a9a0 tx=0x7fc50c008040 comp rx=0 tx=0).stop 2023-10-30T12:59:00.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.651+0000 7fc520899640 1 -- 172.21.15.125:0/2444993634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc51c1538f0 msgr2=0x7fc51c0b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:00.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.651+0000 7fc520899640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc51c1538f0 0x7fc51c0b3dc0 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7fc5100b3040 tx=0x7fc51001ba90 comp rx=0 tx=0).stop 2023-10-30T12:59:00.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.651+0000 7fc520899640 1 -- 172.21.15.125:0/2444993634 shutdown_connections 2023-10-30T12:59:00.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.651+0000 7fc520899640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc5080664a0 0x7fc508068960 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:00.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.651+0000 7fc520899640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc51c15cd50 0x7fc51c0b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:00.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.651+0000 7fc520899640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc51c1538f0 0x7fc51c0b3dc0 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:00.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.651+0000 7fc520899640 1 --2- 172.21.15.125:0/2444993634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc51c0cefa0 0x7fc51c14df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:00.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.651+0000 7fc520899640 1 -- 172.21.15.125:0/2444993634 >> 172.21.15.125:0/2444993634 conn(0x7fc51c06a640 msgr2=0x7fc51c158070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:00.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.651+0000 7fc520899640 1 -- 172.21.15.125:0/2444993634 shutdown_connections 2023-10-30T12:59:00.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:00.651+0000 7fc520899640 1 -- 172.21.15.125:0/2444993634 wait complete. 2023-10-30T12:59:00.673 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:59:02.674 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:02.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2ee2dd9640 1 Processor -- start 2023-10-30T12:59:02.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2ee2dd9640 1 -- start start 2023-10-30T12:59:02.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2ee2dd9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2edc06f7a0 0x7f2edc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:02.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2ee2dd9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc066cf0 con 0x7f2edc071110 2023-10-30T12:59:02.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2ee2dd9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2edc0c70d0 con 0x7f2edc0c6bd0 2023-10-30T12:59:02.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2ee2dd9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2edc0700c0 con 0x7f2edc06f7a0 2023-10-30T12:59:02.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2ee1376640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2edc071110 0x7f2edc158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51712/0 (socket says 172.21.15.125:51712) 2023-10-30T12:59:02.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2ee1376640 1 -- 172.21.15.125:0/3163757378 learned_addr learned my addr 172.21.15.125:0/3163757378 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:59:02.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2ee0b75640 1 --2- 172.21.15.125:0/3163757378 >> v2:172.21.15.136:3300/0 conn(0x7f2edc06f7a0 0x7f2edc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:02.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 -- 172.21.15.125:0/3163757378 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987665769 0 0) 0x7f2edc066cf0 con 0x7f2edc071110 2023-10-30T12:59:02.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 -- 172.21.15.125:0/3163757378 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ecc003530 con 0x7f2edc071110 2023-10-30T12:59:02.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 -- 172.21.15.125:0/3163757378 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1996338479 0 0) 0x7f2ecc003530 con 0x7f2edc071110 2023-10-30T12:59:02.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 -- 172.21.15.125:0/3163757378 >> v2:172.21.15.136:3300/0 conn(0x7f2edc06f7a0 msgr2=0x7f2edc06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:02.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 --2- 172.21.15.125:0/3163757378 >> v2:172.21.15.136:3300/0 conn(0x7f2edc06f7a0 0x7f2edc06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:02.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 -- 172.21.15.125:0/3163757378 >> v1:172.21.15.136:6789/0 conn(0x7f2edc0c6bd0 legacy=0x7f2edc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:02.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 -- 172.21.15.125:0/3163757378 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc0c7390 con 0x7f2edc071110 2023-10-30T12:59:02.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 -- 172.21.15.125:0/3163757378 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f2ed4001a00 con 0x7f2edc071110 2023-10-30T12:59:02.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 -- 172.21.15.125:0/3163757378 >> v1:172.21.15.125:6789/0 conn(0x7f2edc071110 legacy=0x7f2edc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:02.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 --2- 172.21.15.125:0/3163757378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ecc003980 0x7f2ecc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:02.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 --2- 172.21.15.125:0/3163757378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ecc0047d0 0x7f2ecc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:02.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 --2- 172.21.15.125:0/3163757378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ecc005190 0x7f2ecc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:02.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 -- 172.21.15.125:0/3163757378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc0700c0 con 0x7f2ecc0047d0 2023-10-30T12:59:02.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 -- 172.21.15.125:0/3163757378 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ecc00c040 con 0x7f2ecc003980 2023-10-30T12:59:02.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edb7fe640 1 -- 172.21.15.125:0/3163757378 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ecc00c200 con 0x7f2ecc005190 2023-10-30T12:59:02.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edbfff640 1 --2- 172.21.15.125:0/3163757378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ecc0047d0 0x7f2ecc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:02.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2ee0b75640 1 --2- 172.21.15.125:0/3163757378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ecc003980 0x7f2ecc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:02.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2ee1376640 1 --2- 172.21.15.125:0/3163757378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ecc005190 0x7f2ecc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:02.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edbfff640 1 -- 172.21.15.125:0/3163757378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ecc005190 msgr2=0x7f2ecc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:02.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edbfff640 1 --2- 172.21.15.125:0/3163757378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2ecc005190 0x7f2ecc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:02.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edbfff640 1 -- 172.21.15.125:0/3163757378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ecc003980 msgr2=0x7f2ecc003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:02.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edbfff640 1 --2- 172.21.15.125:0/3163757378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ecc003980 0x7f2ecc003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:02.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2edbfff640 1 -- 172.21.15.125:0/3163757378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ecc003530 con 0x7f2ecc0047d0 2023-10-30T12:59:02.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.822+0000 7f2ee1376640 1 -- 172.21.15.125:0/3163757378 reap_dead start 2023-10-30T12:59:02.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.826+0000 7f2edbfff640 1 --2- 172.21.15.125:0/3163757378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ecc0047d0 0x7f2ecc004c50 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8001750 tx=0x7f2ec801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b7404f08e113f4d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:02.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.826+0000 7f2edb7fe640 1 -- 172.21.15.125:0/3163757378 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec80052d0 con 0x7f2ecc0047d0 2023-10-30T12:59:02.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.826+0000 7f2edb7fe640 1 -- 172.21.15.125:0/3163757378 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec80a7050 con 0x7f2ecc0047d0 2023-10-30T12:59:02.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.826+0000 7f2edb7fe640 1 -- 172.21.15.125:0/3163757378 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec8003600 con 0x7f2ecc0047d0 2023-10-30T12:59:02.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.826+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/3163757378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ecc0047d0 msgr2=0x7f2ecc004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:02.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.826+0000 7f2ee2dd9640 1 --2- 172.21.15.125:0/3163757378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ecc0047d0 0x7f2ecc004c50 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f2ec8001750 tx=0x7f2ec801c3c0 comp rx=0 tx=0).stop 2023-10-30T12:59:02.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.826+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/3163757378 shutdown_connections 2023-10-30T12:59:02.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.826+0000 7f2ee2dd9640 1 --2- 172.21.15.125:0/3163757378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ecc0047d0 0x7f2ecc004c50 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:02.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.826+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/3163757378 >> 172.21.15.125:0/3163757378 conn(0x7f2edc0c7910 msgr2=0x7f2edc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:02.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.826+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/3163757378 shutdown_connections 2023-10-30T12:59:02.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.826+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/3163757378 wait complete. 2023-10-30T12:59:02.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee2dd9640 1 Processor -- start 2023-10-30T12:59:02.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee2dd9640 1 -- start start 2023-10-30T12:59:02.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee2dd9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ecc0047d0 0x7f2edc147010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:02.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee2dd9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2edc1462e0 0x7f2edc148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:02.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee2dd9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2edc144d60 0x7f2edc1436d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:02.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee2dd9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc15bc40 con 0x7f2ecc0047d0 2023-10-30T12:59:02.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee2dd9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc15b770 con 0x7f2edc144d60 2023-10-30T12:59:02.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee2dd9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2edc15b960 con 0x7f2edc1462e0 2023-10-30T12:59:02.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee0b75640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ecc0047d0 0x7f2edc147010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:02.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee0b75640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ecc0047d0 0x7f2edc147010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42192/0 (socket says 172.21.15.125:42192) 2023-10-30T12:59:02.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee0b75640 1 -- 172.21.15.125:0/1323974553 learned_addr learned my addr 172.21.15.125:0/1323974553 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:02.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2edbfff640 1 --2- 172.21.15.125:0/1323974553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2edc1462e0 0x7f2edc148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:02.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee1376640 1 --2- 172.21.15.125:0/1323974553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2edc144d60 0x7f2edc1436d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:02.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee0b75640 1 -- 172.21.15.125:0/1323974553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2edc1462e0 msgr2=0x7f2edc148c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:02.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee0b75640 1 --2- 172.21.15.125:0/1323974553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2edc1462e0 0x7f2edc148c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:02.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee0b75640 1 -- 172.21.15.125:0/1323974553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2edc144d60 msgr2=0x7f2edc1436d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:02.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee0b75640 1 --2- 172.21.15.125:0/1323974553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2edc144d60 0x7f2edc1436d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:02.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee0b75640 1 -- 172.21.15.125:0/1323974553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2edc14aaa0 con 0x7f2ecc0047d0 2023-10-30T12:59:02.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee0b75640 1 --2- 172.21.15.125:0/1323974553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ecc0047d0 0x7f2edc147010 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0011aa0 tx=0x7f2ed0011f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:02.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ed97fa640 1 -- 172.21.15.125:0/1323974553 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed000fb20 con 0x7f2ecc0047d0 2023-10-30T12:59:02.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ed97fa640 1 -- 172.21.15.125:0/1323974553 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed000fc80 con 0x7f2ecc0047d0 2023-10-30T12:59:02.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/1323974553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2edc14ad80 con 0x7f2ecc0047d0 2023-10-30T12:59:02.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.830+0000 7f2ed97fa640 1 -- 172.21.15.125:0/1323974553 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0014440 con 0x7f2ecc0047d0 2023-10-30T12:59:02.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.834+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/1323974553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2edc144840 con 0x7f2ecc0047d0 2023-10-30T12:59:02.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.834+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/1323974553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2edc04e170 con 0x7f2ecc0047d0 2023-10-30T12:59:02.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.834+0000 7f2ed97fa640 1 -- 172.21.15.125:0/1323974553 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2ed00145c0 con 0x7f2ecc0047d0 2023-10-30T12:59:02.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.834+0000 7f2ed97fa640 1 --2- 172.21.15.125:0/1323974553 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ec00664d0 0x7f2ec0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:02.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.834+0000 7f2ed97fa640 1 -- 172.21.15.125:0/1323974553 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f2ed0084da0 con 0x7f2ecc0047d0 2023-10-30T12:59:02.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.834+0000 7f2edbfff640 1 --2- 172.21.15.125:0/1323974553 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ec00664d0 0x7f2ec0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:02.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.834+0000 7f2edbfff640 1 --2- 172.21.15.125:0/1323974553 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ec00664d0 0x7f2ec0068990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0c7000 tx=0x7f2ec80011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:02.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:02.838+0000 7f2ed97fa640 1 -- 172.21.15.125:0/1323974553 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2ed0051bc0 con 0x7f2ecc0047d0 2023-10-30T12:59:03.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:03.018+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/1323974553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2edc0708e0 con 0x7f2ecc0047d0 2023-10-30T12:59:03.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:03.018+0000 7f2ed97fa640 1 -- 172.21.15.125:0/1323974553 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f2ed00551e0 con 0x7f2ecc0047d0 2023-10-30T12:59:03.022 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:03.022 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7554,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:19.758581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":22,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:03.022 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 139 2023-10-30T12:59:03.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:03.022+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/1323974553 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ec00664d0 msgr2=0x7f2ec0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:03.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:03.022+0000 7f2ee2dd9640 1 --2- 172.21.15.125:0/1323974553 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ec00664d0 0x7f2ec0068990 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f2edc0c7000 tx=0x7f2ec80011a0 comp rx=0 tx=0).stop 2023-10-30T12:59:03.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:03.022+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/1323974553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ecc0047d0 msgr2=0x7f2edc147010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:03.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:03.022+0000 7f2ee2dd9640 1 --2- 172.21.15.125:0/1323974553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ecc0047d0 0x7f2edc147010 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f2ed0011aa0 tx=0x7f2ed0011f60 comp rx=0 tx=0).stop 2023-10-30T12:59:03.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:03.022+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/1323974553 shutdown_connections 2023-10-30T12:59:03.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:03.022+0000 7f2ee2dd9640 1 --2- 172.21.15.125:0/1323974553 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ec00664d0 0x7f2ec0068990 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:03.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:03.022+0000 7f2ee2dd9640 1 --2- 172.21.15.125:0/1323974553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2edc144d60 0x7f2edc1436d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:03.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:03.022+0000 7f2ee2dd9640 1 --2- 172.21.15.125:0/1323974553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2edc1462e0 0x7f2edc148c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:03.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:03.022+0000 7f2ee2dd9640 1 --2- 172.21.15.125:0/1323974553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ecc0047d0 0x7f2edc147010 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:03.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:03.022+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/1323974553 >> 172.21.15.125:0/1323974553 conn(0x7f2edc0c7910 msgr2=0x7f2edc0c9960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:03.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:03.022+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/1323974553 shutdown_connections 2023-10-30T12:59:03.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:03.022+0000 7f2ee2dd9640 1 -- 172.21.15.125:0/1323974553 wait complete. 2023-10-30T12:59:03.043 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:59:05.044 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:05.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.190+0000 7f38eecc9640 1 Processor -- start 2023-10-30T12:59:05.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.190+0000 7f38eecc9640 1 -- start start 2023-10-30T12:59:05.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.190+0000 7f38eecc9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38e8159e60 0x7f38e806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:05.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.190+0000 7f38eecc9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f38e80bfec0 0x7f38e80c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:05.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.190+0000 7f38eecc9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38e806fa80 con 0x7f38e806a720 2023-10-30T12:59:05.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.190+0000 7f38eecc9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38e8071290 con 0x7f38e80bfec0 2023-10-30T12:59:05.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.190+0000 7f38eecc9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e8071570 con 0x7f38e8159e60 2023-10-30T12:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.190+0000 7f38eca65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38e8159e60 0x7f38e806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.190+0000 7f38e7fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f38e80bfec0 0x7f38e80c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.190+0000 7f38eca65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38e8159e60 0x7f38e806f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:41776/0 (socket says 172.21.15.125:41776) 2023-10-30T12:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.190+0000 7f38eca65640 1 -- 172.21.15.125:0/4034027450 learned_addr learned my addr 172.21.15.125:0/4034027450 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:05.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.194+0000 7f38eca65640 1 -- 172.21.15.125:0/4034027450 >> v2:172.21.15.136:3300/0 conn(0x7f38e80bfec0 msgr2=0x7f38e80c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:05.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.194+0000 7f38eca65640 1 --2- 172.21.15.125:0/4034027450 >> v2:172.21.15.136:3300/0 conn(0x7f38e80bfec0 0x7f38e80c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:05.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.194+0000 7f38eca65640 1 -- 172.21.15.125:0/4034027450 >> v1:172.21.15.136:6789/0 conn(0x7f38e806a720 legacy=0x7f38e8155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:05.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.194+0000 7f38eca65640 1 -- 172.21.15.125:0/4034027450 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38e8071850 con 0x7f38e8159e60 2023-10-30T12:59:05.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.194+0000 7f38eca65640 1 --2- 172.21.15.125:0/4034027450 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38e8159e60 0x7f38e806f540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f38dc009870 tx=0x7f38dc01ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=881fa04f082f1b02 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:05.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.194+0000 7f38e77fe640 1 -- 172.21.15.125:0/4034027450 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc0a8a30 con 0x7f38e8159e60 2023-10-30T12:59:05.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.194+0000 7f38e77fe640 1 -- 172.21.15.125:0/4034027450 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38dc0a8b90 con 0x7f38e8159e60 2023-10-30T12:59:05.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.194+0000 7f38e77fe640 1 -- 172.21.15.125:0/4034027450 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38dc0b0440 con 0x7f38e8159e60 2023-10-30T12:59:05.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 -- 172.21.15.125:0/4034027450 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38e8159e60 msgr2=0x7f38e806f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:05.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 --2- 172.21.15.125:0/4034027450 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38e8159e60 0x7f38e806f540 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f38dc009870 tx=0x7f38dc01ed30 comp rx=0 tx=0).stop 2023-10-30T12:59:05.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 -- 172.21.15.125:0/4034027450 shutdown_connections 2023-10-30T12:59:05.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 --2- 172.21.15.125:0/4034027450 >> v2:172.21.15.136:3300/0 conn(0x7f38e80bfec0 0x7f38e80c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:05.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 --2- 172.21.15.125:0/4034027450 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38e8159e60 0x7f38e806f540 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:05.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 -- 172.21.15.125:0/4034027450 >> 172.21.15.125:0/4034027450 conn(0x7f38e80c7910 msgr2=0x7f38e80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:05.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 -- 172.21.15.125:0/4034027450 shutdown_connections 2023-10-30T12:59:05.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 -- 172.21.15.125:0/4034027450 wait complete. 2023-10-30T12:59:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 Processor -- start 2023-10-30T12:59:05.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 -- start start 2023-10-30T12:59:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38e806a720 0x7f38e81f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38e80bfec0 0x7f38e81df480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38e8159e60 0x7f38e81e22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e81e42d0 con 0x7f38e80bfec0 2023-10-30T12:59:05.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38e81e31e0 con 0x7f38e806a720 2023-10-30T12:59:05.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eecc9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38e81e3da0 con 0x7f38e8159e60 2023-10-30T12:59:05.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38e7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38e80bfec0 0x7f38e81df480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:05.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38e7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38e80bfec0 0x7f38e81df480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42206/0 (socket says 172.21.15.125:42206) 2023-10-30T12:59:05.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38e7fff640 1 -- 172.21.15.125:0/23994135 learned_addr learned my addr 172.21.15.125:0/23994135 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:05.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38eca65640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38e806a720 0x7f38e81f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:05.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38ed266640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38e8159e60 0x7f38e81e22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:05.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38e7fff640 1 -- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38e8159e60 msgr2=0x7f38e81e22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:05.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38e7fff640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38e8159e60 0x7f38e81e22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:05.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38e7fff640 1 -- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38e806a720 msgr2=0x7f38e81f2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:05.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38e7fff640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38e806a720 0x7f38e81f2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:05.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38e7fff640 1 -- 172.21.15.125:0/23994135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38dc0094d0 con 0x7f38e80bfec0 2023-10-30T12:59:05.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.198+0000 7f38ed266640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38e8159e60 0x7f38e81e22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T12:59:05.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.202+0000 7f38e7fff640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38e80bfec0 0x7f38e81df480 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f38d400e910 tx=0x7f38d400edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:05.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.202+0000 7f38e57fa640 1 -- 172.21.15.125:0/23994135 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d400cc40 con 0x7f38e80bfec0 2023-10-30T12:59:05.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.202+0000 7f38e57fa640 1 -- 172.21.15.125:0/23994135 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d400cda0 con 0x7f38e80bfec0 2023-10-30T12:59:05.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.202+0000 7f38eecc9640 1 -- 172.21.15.125:0/23994135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38e81596c0 con 0x7f38e80bfec0 2023-10-30T12:59:05.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.202+0000 7f38e57fa640 1 -- 172.21.15.125:0/23994135 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d4011610 con 0x7f38e80bfec0 2023-10-30T12:59:05.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.202+0000 7f38eecc9640 1 -- 172.21.15.125:0/23994135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38e81e3ff0 con 0x7f38e80bfec0 2023-10-30T12:59:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.202+0000 7f38eecc9640 1 -- 172.21.15.125:0/23994135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e804e170 con 0x7f38e80bfec0 2023-10-30T12:59:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.202+0000 7f38e57fa640 1 -- 172.21.15.125:0/23994135 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f38d4011770 con 0x7f38e80bfec0 2023-10-30T12:59:05.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.202+0000 7f38e57fa640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38cc0664a0 0x7f38cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:05.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.202+0000 7f38e57fa640 1 -- 172.21.15.125:0/23994135 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f38d40842a0 con 0x7f38e80bfec0 2023-10-30T12:59:05.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.202+0000 7f38eca65640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38cc0664a0 0x7f38cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:05.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.206+0000 7f38eca65640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38cc0664a0 0x7f38cc068960 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f38dc0b2000 tx=0x7f38dc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:05.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.206+0000 7f38e57fa640 1 -- 172.21.15.125:0/23994135 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f38d4051160 con 0x7f38e80bfec0 2023-10-30T12:59:05.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.386+0000 7f38eecc9640 1 -- 172.21.15.125:0/23994135 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38e81df0a0 con 0x7f38e80bfec0 2023-10-30T12:59:05.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.386+0000 7f38e57fa640 1 -- 172.21.15.125:0/23994135 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f38d4054780 con 0x7f38e80bfec0 2023-10-30T12:59:05.390 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:05.390 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7554,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:19.758581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":22,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:05.391 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 139 2023-10-30T12:59:05.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.390+0000 7f38eecc9640 1 -- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38cc0664a0 msgr2=0x7f38cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:05.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.390+0000 7f38eecc9640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38cc0664a0 0x7f38cc068960 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f38dc0b2000 tx=0x7f38dc008040 comp rx=0 tx=0).stop 2023-10-30T12:59:05.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.390+0000 7f38eecc9640 1 -- 172.21.15.125:0/23994135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38e80bfec0 msgr2=0x7f38e81df480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:05.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.390+0000 7f38eecc9640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38e80bfec0 0x7f38e81df480 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f38d400e910 tx=0x7f38d400edd0 comp rx=0 tx=0).stop 2023-10-30T12:59:05.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.390+0000 7f38eecc9640 1 -- 172.21.15.125:0/23994135 shutdown_connections 2023-10-30T12:59:05.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.390+0000 7f38eecc9640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38cc0664a0 0x7f38cc068960 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:05.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.390+0000 7f38eecc9640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38e8159e60 0x7f38e81e22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:05.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.390+0000 7f38eecc9640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38e80bfec0 0x7f38e81df480 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:05.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.390+0000 7f38eecc9640 1 --2- 172.21.15.125:0/23994135 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38e806a720 0x7f38e81f2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:05.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.390+0000 7f38eecc9640 1 -- 172.21.15.125:0/23994135 >> 172.21.15.125:0/23994135 conn(0x7f38e80c7910 msgr2=0x7f38e814f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:05.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.390+0000 7f38eecc9640 1 -- 172.21.15.125:0/23994135 shutdown_connections 2023-10-30T12:59:05.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:05.390+0000 7f38eecc9640 1 -- 172.21.15.125:0/23994135 wait complete. 2023-10-30T12:59:05.411 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:59:07.412 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:07.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.558+0000 7fee71a46640 1 Processor -- start 2023-10-30T12:59:07.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.558+0000 7fee71a46640 1 -- start start 2023-10-30T12:59:07.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.558+0000 7fee71a46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fee6c0c6b80 0x7fee6c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:07.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.558+0000 7fee71a46640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fee6c0c6f60 0x7fee6c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:07.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.558+0000 7fee71a46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee6c0c0a50 con 0x7fee6c0c8430 2023-10-30T12:59:07.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.558+0000 7fee71a46640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee6c06f6e0 con 0x7fee6c0c6f60 2023-10-30T12:59:07.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.558+0000 7fee71a46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c06f9c0 con 0x7fee6c0c6b80 2023-10-30T12:59:07.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.558+0000 7fee6b7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fee6c0c8430 0x7fee6c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51720/0 (socket says 172.21.15.125:51720) 2023-10-30T12:59:07.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.558+0000 7fee6b7fe640 1 -- 172.21.15.125:0/2241582483 learned_addr learned my addr 172.21.15.125:0/2241582483 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:59:07.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.558+0000 7fee627fc640 1 --2- 172.21.15.125:0/2241582483 >> v2:172.21.15.136:3300/0 conn(0x7fee6c0c6f60 0x7fee6c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:07.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.558+0000 7fee6affd640 1 --2- 172.21.15.125:0/2241582483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fee6c0c6b80 0x7fee6c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.558+0000 7fee6a7fc640 1 -- 172.21.15.125:0/2241582483 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281932104 0 0) 0x7fee6c0c0a50 con 0x7fee6c0c8430 2023-10-30T12:59:07.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.558+0000 7fee6a7fc640 1 -- 172.21.15.125:0/2241582483 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee58007940 con 0x7fee6c0c8430 2023-10-30T12:59:07.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.558+0000 7fee6a7fc640 1 -- 172.21.15.125:0/2241582483 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2726093056 0 0) 0x7fee58007940 con 0x7fee6c0c8430 2023-10-30T12:59:07.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 -- 172.21.15.125:0/2241582483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fee6c0c6b80 msgr2=0x7fee6c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:07.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 --2- 172.21.15.125:0/2241582483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fee6c0c6b80 0x7fee6c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:07.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 -- 172.21.15.125:0/2241582483 >> v2:172.21.15.136:3300/0 conn(0x7fee6c0c6f60 msgr2=0x7fee6c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 --2- 172.21.15.125:0/2241582483 >> v2:172.21.15.136:3300/0 conn(0x7fee6c0c6f60 0x7fee6c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 -- 172.21.15.125:0/2241582483 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6c06fca0 con 0x7fee6c0c8430 2023-10-30T12:59:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 -- 172.21.15.125:0/2241582483 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fee64001680 con 0x7fee6c0c8430 2023-10-30T12:59:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 -- 172.21.15.125:0/2241582483 >> v1:172.21.15.125:6789/0 conn(0x7fee6c0c8430 legacy=0x7fee6c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 --2- 172.21.15.125:0/2241582483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fee6c0c6b80 0x7fee58007d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:07.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 --2- 172.21.15.125:0/2241582483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee58008310 0x7fee58008be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:07.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 --2- 172.21.15.125:0/2241582483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee58009120 0x7fee580095a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:07.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 -- 172.21.15.125:0/2241582483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c06f6e0 con 0x7fee58009120 2023-10-30T12:59:07.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 -- 172.21.15.125:0/2241582483 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c06f9c0 con 0x7fee58008310 2023-10-30T12:59:07.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 -- 172.21.15.125:0/2241582483 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee5800fda0 con 0x7fee6c0c6b80 2023-10-30T12:59:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6b7fe640 1 --2- 172.21.15.125:0/2241582483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee58009120 0x7fee580095a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6affd640 1 --2- 172.21.15.125:0/2241582483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fee6c0c6b80 0x7fee58007d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee627fc640 1 --2- 172.21.15.125:0/2241582483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee58008310 0x7fee58008be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6b7fe640 1 -- 172.21.15.125:0/2241582483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fee6c0c6b80 msgr2=0x7fee58007d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:07.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6b7fe640 1 --2- 172.21.15.125:0/2241582483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fee6c0c6b80 0x7fee58007d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:07.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6b7fe640 1 -- 172.21.15.125:0/2241582483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee58008310 msgr2=0x7fee58008be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:07.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6b7fe640 1 --2- 172.21.15.125:0/2241582483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee58008310 0x7fee58008be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:07.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6b7fe640 1 -- 172.21.15.125:0/2241582483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee58007940 con 0x7fee58009120 2023-10-30T12:59:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6b7fe640 1 --2- 172.21.15.125:0/2241582483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee58009120 0x7fee580095a0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fee6400c240 tx=0x7fee6401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=762cc94093efef9b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 -- 172.21.15.125:0/2241582483 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee640a5960 con 0x7fee58009120 2023-10-30T12:59:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 -- 172.21.15.125:0/2241582483 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee640a5ac0 con 0x7fee58009120 2023-10-30T12:59:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.562+0000 7fee6a7fc640 1 -- 172.21.15.125:0/2241582483 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee640058e0 con 0x7fee58009120 2023-10-30T12:59:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee71a46640 1 -- 172.21.15.125:0/2241582483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee58009120 msgr2=0x7fee580095a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee71a46640 1 --2- 172.21.15.125:0/2241582483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee58009120 0x7fee580095a0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fee6400c240 tx=0x7fee6401cc00 comp rx=0 tx=0).stop 2023-10-30T12:59:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee6b7fe640 1 -- 172.21.15.125:0/2241582483 reap_dead start 2023-10-30T12:59:07.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee71a46640 1 -- 172.21.15.125:0/2241582483 shutdown_connections 2023-10-30T12:59:07.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee71a46640 1 -- 172.21.15.125:0/2241582483 >> 172.21.15.125:0/2241582483 conn(0x7fee6c0c7910 msgr2=0x7fee6c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:07.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee71a46640 1 -- 172.21.15.125:0/2241582483 shutdown_connections 2023-10-30T12:59:07.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee71a46640 1 -- 172.21.15.125:0/2241582483 wait complete. 2023-10-30T12:59:07.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee71a46640 1 Processor -- start 2023-10-30T12:59:07.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee71a46640 1 -- start start 2023-10-30T12:59:07.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee71a46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fee58009120 0x7fee6c134c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:07.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee71a46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee6c0b0910 0x7fee6c1527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:07.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee71a46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee6c1341a0 0x7fee6c134620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:07.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee71a46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c1eaf50 con 0x7fee6c1341a0 2023-10-30T12:59:07.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee71a46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c1ee600 con 0x7fee6c0b0910 2023-10-30T12:59:07.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee71a46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee6c1ea620 con 0x7fee58009120 2023-10-30T12:59:07.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee6b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee6c1341a0 0x7fee6c134620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:07.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee6b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee6c1341a0 0x7fee6c134620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42214/0 (socket says 172.21.15.125:42214) 2023-10-30T12:59:07.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee6b7fe640 1 -- 172.21.15.125:0/3828419923 learned_addr learned my addr 172.21.15.125:0/3828419923 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:07.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.566+0000 7fee6affd640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fee58009120 0x7fee6c134c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:07.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee627fc640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee6c0b0910 0x7fee6c1527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:07.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee6b7fe640 1 -- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fee58009120 msgr2=0x7fee6c134c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:07.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee6b7fe640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fee58009120 0x7fee6c134c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:07.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee6b7fe640 1 -- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee6c0b0910 msgr2=0x7fee6c1527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:07.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee6b7fe640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee6c0b0910 0x7fee6c1527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:07.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee6b7fe640 1 -- 172.21.15.125:0/3828419923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee6c1ea900 con 0x7fee6c1341a0 2023-10-30T12:59:07.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee627fc640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee6c0b0910 0x7fee6c1527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:59:07.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee6b7fe640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee6c1341a0 0x7fee6c134620 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fee6400c240 tx=0x7fee58009c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:07.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee68ff9640 1 -- 172.21.15.125:0/3828419923 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee640072d0 con 0x7fee6c1341a0 2023-10-30T12:59:07.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee68ff9640 1 -- 172.21.15.125:0/3828419923 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee640b3070 con 0x7fee6c1341a0 2023-10-30T12:59:07.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee71a46640 1 -- 172.21.15.125:0/3828419923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee6c1eabe0 con 0x7fee6c1341a0 2023-10-30T12:59:07.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee68ff9640 1 -- 172.21.15.125:0/3828419923 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee64007630 con 0x7fee6c1341a0 2023-10-30T12:59:07.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee71a46640 1 -- 172.21.15.125:0/3828419923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee6c0c1820 con 0x7fee6c1341a0 2023-10-30T12:59:07.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee71a46640 1 -- 172.21.15.125:0/3828419923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee6c04e170 con 0x7fee6c1341a0 2023-10-30T12:59:07.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee68ff9640 1 -- 172.21.15.125:0/3828419923 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fee640af440 con 0x7fee6c1341a0 2023-10-30T12:59:07.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee68ff9640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fee4c0664a0 0x7fee4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:07.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.570+0000 7fee68ff9640 1 -- 172.21.15.125:0/3828419923 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7fee640b5070 con 0x7fee6c1341a0 2023-10-30T12:59:07.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.574+0000 7fee6affd640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fee4c0664a0 0x7fee4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:07.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.574+0000 7fee6affd640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fee4c0664a0 0x7fee4c068960 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fee5c014510 tx=0x7fee5c00b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:07.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.574+0000 7fee68ff9640 1 -- 172.21.15.125:0/3828419923 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fee640eb6c0 con 0x7fee6c1341a0 2023-10-30T12:59:07.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.758+0000 7fee71a46640 1 -- 172.21.15.125:0/3828419923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee6c138ed0 con 0x7fee6c1341a0 2023-10-30T12:59:07.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.758+0000 7fee68ff9640 1 -- 172.21.15.125:0/3828419923 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fee64001c60 con 0x7fee6c1341a0 2023-10-30T12:59:07.760 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:07.761 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7554,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:19.758581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":22,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:07.761 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 139 2023-10-30T12:59:07.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.762+0000 7fee71a46640 1 -- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fee4c0664a0 msgr2=0x7fee4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:07.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.762+0000 7fee71a46640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fee4c0664a0 0x7fee4c068960 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fee5c014510 tx=0x7fee5c00b040 comp rx=0 tx=0).stop 2023-10-30T12:59:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.762+0000 7fee71a46640 1 -- 172.21.15.125:0/3828419923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee6c1341a0 msgr2=0x7fee6c134620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.762+0000 7fee71a46640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee6c1341a0 0x7fee6c134620 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fee6400c240 tx=0x7fee58009c10 comp rx=0 tx=0).stop 2023-10-30T12:59:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.762+0000 7fee71a46640 1 -- 172.21.15.125:0/3828419923 shutdown_connections 2023-10-30T12:59:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.762+0000 7fee71a46640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fee4c0664a0 0x7fee4c068960 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:07.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.762+0000 7fee71a46640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fee6c1341a0 0x7fee6c134620 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:07.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.762+0000 7fee71a46640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fee6c0b0910 0x7fee6c1527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:07.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.762+0000 7fee71a46640 1 --2- 172.21.15.125:0/3828419923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fee58009120 0x7fee6c134c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:07.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.762+0000 7fee71a46640 1 -- 172.21.15.125:0/3828419923 >> 172.21.15.125:0/3828419923 conn(0x7fee6c0c7910 msgr2=0x7fee6c13bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:07.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.762+0000 7fee71a46640 1 -- 172.21.15.125:0/3828419923 shutdown_connections 2023-10-30T12:59:07.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:07.762+0000 7fee71a46640 1 -- 172.21.15.125:0/3828419923 wait complete. 2023-10-30T12:59:07.782 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:59:09.783 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:09.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.926+0000 7f9b6c39d640 1 Processor -- start 2023-10-30T12:59:09.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.926+0000 7f9b6c39d640 1 -- start start 2023-10-30T12:59:09.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6c39d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b64156f80 0x7f9b64159880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:09.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6c39d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9b64070650 0x7f9b64071bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:09.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6c39d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b640c2f20 con 0x7f9b64070c90 2023-10-30T12:59:09.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6c39d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b64157360 con 0x7f9b64070650 2023-10-30T12:59:09.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6c39d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b640c85c0 con 0x7f9b64156f80 2023-10-30T12:59:09.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6a93a640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9b64070c90 0x7f9b6415cdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51728/0 (socket says 172.21.15.125:51728) 2023-10-30T12:59:09.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6a93a640 1 -- 172.21.15.125:0/27222510 learned_addr learned my addr 172.21.15.125:0/27222510 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:59:09.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69938640 1 --2- 172.21.15.125:0/27222510 >> v2:172.21.15.136:3300/0 conn(0x7f9b64070650 0x7f9b64071bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:09.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6a139640 1 --2- 172.21.15.125:0/27222510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b64156f80 0x7f9b64159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:09.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 -- 172.21.15.125:0/27222510 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630946551 0 0) 0x7f9b640c2f20 con 0x7f9b64070c90 2023-10-30T12:59:09.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 -- 172.21.15.125:0/27222510 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b50003530 con 0x7f9b64070c90 2023-10-30T12:59:09.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 -- 172.21.15.125:0/27222510 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 842259156 0 0) 0x7f9b50003530 con 0x7f9b64070c90 2023-10-30T12:59:09.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 -- 172.21.15.125:0/27222510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b64156f80 msgr2=0x7f9b64159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:09.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 --2- 172.21.15.125:0/27222510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b64156f80 0x7f9b64159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:09.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 -- 172.21.15.125:0/27222510 >> v2:172.21.15.136:3300/0 conn(0x7f9b64070650 msgr2=0x7f9b64071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:09.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 --2- 172.21.15.125:0/27222510 >> v2:172.21.15.136:3300/0 conn(0x7f9b64070650 0x7f9b64071bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:09.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 -- 172.21.15.125:0/27222510 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b64071230 con 0x7f9b64070c90 2023-10-30T12:59:09.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 -- 172.21.15.125:0/27222510 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f9b5c001a00 con 0x7f9b64070c90 2023-10-30T12:59:09.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 -- 172.21.15.125:0/27222510 >> v1:172.21.15.125:6789/0 conn(0x7f9b64070c90 legacy=0x7f9b6415cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:09.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 --2- 172.21.15.125:0/27222510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b50003980 0x7f9b50003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:09.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 --2- 172.21.15.125:0/27222510 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b500047d0 0x7f9b50004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:09.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 --2- 172.21.15.125:0/27222510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b64156f80 0x7f9b50005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:09.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 -- 172.21.15.125:0/27222510 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b64157360 con 0x7f9b50003980 2023-10-30T12:59:09.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 -- 172.21.15.125:0/27222510 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b640c85c0 con 0x7f9b500047d0 2023-10-30T12:59:09.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69137640 1 -- 172.21.15.125:0/27222510 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5000b990 con 0x7f9b64156f80 2023-10-30T12:59:09.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6a139640 1 --2- 172.21.15.125:0/27222510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b50003980 0x7f9b50003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:09.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b69938640 1 --2- 172.21.15.125:0/27222510 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b500047d0 0x7f9b50004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:09.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6a93a640 1 --2- 172.21.15.125:0/27222510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b64156f80 0x7f9b50005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:09.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6a139640 1 -- 172.21.15.125:0/27222510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b64156f80 msgr2=0x7f9b50005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:09.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6a139640 1 --2- 172.21.15.125:0/27222510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b64156f80 0x7f9b50005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:09.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6a139640 1 -- 172.21.15.125:0/27222510 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b500047d0 msgr2=0x7f9b50004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:09.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6a139640 1 --2- 172.21.15.125:0/27222510 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b500047d0 0x7f9b50004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:09.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.930+0000 7f9b6a139640 1 -- 172.21.15.125:0/27222510 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b50003530 con 0x7f9b50003980 2023-10-30T12:59:09.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.934+0000 7f9b6a139640 1 --2- 172.21.15.125:0/27222510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b50003980 0x7f9b50003da0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f9b60009ae0 tx=0x7f9b6001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853c85d6fbfe095c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:09.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.934+0000 7f9b69137640 1 -- 172.21.15.125:0/27222510 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b600a64e0 con 0x7f9b50003980 2023-10-30T12:59:09.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.934+0000 7f9b69137640 1 -- 172.21.15.125:0/27222510 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b600a6640 con 0x7f9b50003980 2023-10-30T12:59:09.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.934+0000 7f9b69137640 1 -- 172.21.15.125:0/27222510 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b600a69f0 con 0x7f9b50003980 2023-10-30T12:59:09.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.934+0000 7f9b6c39d640 1 -- 172.21.15.125:0/27222510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b50003980 msgr2=0x7f9b50003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:09.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.934+0000 7f9b6c39d640 1 --2- 172.21.15.125:0/27222510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b50003980 0x7f9b50003da0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f9b60009ae0 tx=0x7f9b6001ca50 comp rx=0 tx=0).stop 2023-10-30T12:59:09.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.934+0000 7f9b6a93a640 1 -- 172.21.15.125:0/27222510 reap_dead start 2023-10-30T12:59:09.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.934+0000 7f9b6c39d640 1 -- 172.21.15.125:0/27222510 shutdown_connections 2023-10-30T12:59:09.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.934+0000 7f9b6c39d640 1 -- 172.21.15.125:0/27222510 >> 172.21.15.125:0/27222510 conn(0x7f9b640cbeb0 msgr2=0x7f9b640cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:09.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.934+0000 7f9b6c39d640 1 -- 172.21.15.125:0/27222510 shutdown_connections 2023-10-30T12:59:09.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.934+0000 7f9b6c39d640 1 -- 172.21.15.125:0/27222510 wait complete. 2023-10-30T12:59:09.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b6c39d640 1 Processor -- start 2023-10-30T12:59:09.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b6c39d640 1 -- start start 2023-10-30T12:59:09.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b6c39d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b50003980 0x7f9b641546d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:09.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b6c39d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b6414fba0 0x7f9b6414cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:09.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b6c39d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b641f5510 0x7f9b6414e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:09.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b6c39d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b64151170 con 0x7f9b6414fba0 2023-10-30T12:59:09.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b6c39d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b64152660 con 0x7f9b50003980 2023-10-30T12:59:09.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b6c39d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b641579d0 con 0x7f9b641f5510 2023-10-30T12:59:09.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b69938640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b6414fba0 0x7f9b6414cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:09.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b69938640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b6414fba0 0x7f9b6414cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42238/0 (socket says 172.21.15.125:42238) 2023-10-30T12:59:09.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b69938640 1 -- 172.21.15.125:0/2969090874 learned_addr learned my addr 172.21.15.125:0/2969090874 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:09.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b6a139640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b50003980 0x7f9b641546d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:09.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b6a93a640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b641f5510 0x7f9b6414e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:09.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b69938640 1 -- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b641f5510 msgr2=0x7f9b6414e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:09.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b69938640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b641f5510 0x7f9b6414e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:09.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b69938640 1 -- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b50003980 msgr2=0x7f9b641546d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:09.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b69938640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b50003980 0x7f9b641546d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:09.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b69938640 1 -- 172.21.15.125:0/2969090874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b64150ef0 con 0x7f9b6414fba0 2023-10-30T12:59:09.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b6a139640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b50003980 0x7f9b641546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:59:09.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b69938640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b6414fba0 0x7f9b6414cfe0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f9b540116d0 tx=0x7f9b54011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:09.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b5affd640 1 -- 172.21.15.125:0/2969090874 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5400f9c0 con 0x7f9b6414fba0 2023-10-30T12:59:09.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b5affd640 1 -- 172.21.15.125:0/2969090874 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b5400fb20 con 0x7f9b6414fba0 2023-10-30T12:59:09.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b6c39d640 1 -- 172.21.15.125:0/2969090874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b64152a80 con 0x7f9b6414fba0 2023-10-30T12:59:09.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b5affd640 1 -- 172.21.15.125:0/2969090874 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b5400feb0 con 0x7f9b6414fba0 2023-10-30T12:59:09.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b6c39d640 1 -- 172.21.15.125:0/2969090874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b6414d520 con 0x7f9b6414fba0 2023-10-30T12:59:09.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.938+0000 7f9b6c39d640 1 -- 172.21.15.125:0/2969090874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b6404e170 con 0x7f9b6414fba0 2023-10-30T12:59:09.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.942+0000 7f9b5affd640 1 -- 172.21.15.125:0/2969090874 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9b54015460 con 0x7f9b6414fba0 2023-10-30T12:59:09.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.942+0000 7f9b5affd640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b480664a0 0x7f9b48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:09.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.942+0000 7f9b5affd640 1 -- 172.21.15.125:0/2969090874 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f9b54084b60 con 0x7f9b6414fba0 2023-10-30T12:59:09.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.942+0000 7f9b6a139640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b480664a0 0x7f9b48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:09.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.942+0000 7f9b6a139640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b480664a0 0x7f9b48068960 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9b600af000 tx=0x7f9b600046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:09.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:09.946+0000 7f9b5affd640 1 -- 172.21.15.125:0/2969090874 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9b54051b20 con 0x7f9b6414fba0 2023-10-30T12:59:10.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:10.130+0000 7f9b6c39d640 1 -- 172.21.15.125:0/2969090874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b640c8220 con 0x7f9b6414fba0 2023-10-30T12:59:10.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:10.130+0000 7f9b5affd640 1 -- 172.21.15.125:0/2969090874 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f9b54055140 con 0x7f9b6414fba0 2023-10-30T12:59:10.131 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:10.132 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7554,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:19.758581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":22,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:10.132 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 139 2023-10-30T12:59:10.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:10.130+0000 7f9b6c39d640 1 -- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b480664a0 msgr2=0x7f9b48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:10.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:10.130+0000 7f9b6c39d640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b480664a0 0x7f9b48068960 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9b600af000 tx=0x7f9b600046d0 comp rx=0 tx=0).stop 2023-10-30T12:59:10.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:10.130+0000 7f9b6c39d640 1 -- 172.21.15.125:0/2969090874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b6414fba0 msgr2=0x7f9b6414cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:10.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:10.130+0000 7f9b6c39d640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b6414fba0 0x7f9b6414cfe0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f9b540116d0 tx=0x7f9b54011b90 comp rx=0 tx=0).stop 2023-10-30T12:59:10.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:10.130+0000 7f9b6c39d640 1 -- 172.21.15.125:0/2969090874 shutdown_connections 2023-10-30T12:59:10.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:10.130+0000 7f9b6c39d640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b480664a0 0x7f9b48068960 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:10.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:10.130+0000 7f9b6c39d640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b641f5510 0x7f9b6414e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:10.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:10.130+0000 7f9b6c39d640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b6414fba0 0x7f9b6414cfe0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:10.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:10.130+0000 7f9b6c39d640 1 --2- 172.21.15.125:0/2969090874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b50003980 0x7f9b641546d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:10.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:10.130+0000 7f9b6c39d640 1 -- 172.21.15.125:0/2969090874 >> 172.21.15.125:0/2969090874 conn(0x7f9b640cbeb0 msgr2=0x7f9b640cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:10.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:10.130+0000 7f9b6c39d640 1 -- 172.21.15.125:0/2969090874 shutdown_connections 2023-10-30T12:59:10.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:10.130+0000 7f9b6c39d640 1 -- 172.21.15.125:0/2969090874 wait complete. 2023-10-30T12:59:10.152 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:59:12.154 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:12.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.298+0000 7f41a36d6640 1 Processor -- start 2023-10-30T12:59:12.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.298+0000 7f41a36d6640 1 -- start start 2023-10-30T12:59:12.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.298+0000 7f41a36d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f419c06f7a0 0x7f419c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:12.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.298+0000 7f41a36d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c066cf0 con 0x7f419c0c6bd0 2023-10-30T12:59:12.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.298+0000 7f41a36d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c0c70d0 con 0x7f419c071110 2023-10-30T12:59:12.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.298+0000 7f41a36d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419c0700c0 con 0x7f419c06f7a0 2023-10-30T12:59:12.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.298+0000 7f41a0c71640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f419c0c6bd0 0x7f419c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36924/0 (socket says 172.21.15.125:36924) 2023-10-30T12:59:12.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.298+0000 7f41a0c71640 1 -- 172.21.15.125:0/2250579816 learned_addr learned my addr 172.21.15.125:0/2250579816 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:59:12.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f41a1472640 1 --2- 172.21.15.125:0/2250579816 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f419c06f7a0 0x7f419c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:12.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 -- 172.21.15.125:0/2250579816 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528084363 0 0) 0x7f419c066cf0 con 0x7f419c0c6bd0 2023-10-30T12:59:12.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 -- 172.21.15.125:0/2250579816 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4188003530 con 0x7f419c0c6bd0 2023-10-30T12:59:12.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 -- 172.21.15.125:0/2250579816 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1947682830 0 0) 0x7f4188003530 con 0x7f419c0c6bd0 2023-10-30T12:59:12.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 -- 172.21.15.125:0/2250579816 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f419c06f7a0 msgr2=0x7f419c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:12.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 --2- 172.21.15.125:0/2250579816 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f419c06f7a0 0x7f419c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:12.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 -- 172.21.15.125:0/2250579816 >> v1:172.21.15.136:6789/0 conn(0x7f419c071110 legacy=0x7f419c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:12.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 -- 172.21.15.125:0/2250579816 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c0c7390 con 0x7f419c0c6bd0 2023-10-30T12:59:12.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 -- 172.21.15.125:0/2250579816 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f418c001680 con 0x7f419c0c6bd0 2023-10-30T12:59:12.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 -- 172.21.15.125:0/2250579816 >> v1:172.21.15.125:6789/0 conn(0x7f419c0c6bd0 legacy=0x7f419c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:12.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 --2- 172.21.15.125:0/2250579816 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f419c06f7a0 0x7f41880039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:12.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 --2- 172.21.15.125:0/2250579816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4188003ef0 0x7f41880047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:12.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 --2- 172.21.15.125:0/2250579816 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4188004d10 0x7f4188005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:12.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 -- 172.21.15.125:0/2250579816 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c0700c0 con 0x7f4188004d10 2023-10-30T12:59:12.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 -- 172.21.15.125:0/2250579816 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f418800b9f0 con 0x7f4188003ef0 2023-10-30T12:59:12.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f4193fff640 1 -- 172.21.15.125:0/2250579816 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f418800bbb0 con 0x7f419c06f7a0 2023-10-30T12:59:12.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f41a1c73640 1 --2- 172.21.15.125:0/2250579816 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4188004d10 0x7f4188005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:12.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f41a0c71640 1 --2- 172.21.15.125:0/2250579816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4188003ef0 0x7f41880047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:12.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f41a1472640 1 --2- 172.21.15.125:0/2250579816 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f419c06f7a0 0x7f41880039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:12.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f41a1c73640 1 -- 172.21.15.125:0/2250579816 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f419c06f7a0 msgr2=0x7f41880039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:12.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f41a1c73640 1 --2- 172.21.15.125:0/2250579816 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f419c06f7a0 0x7f41880039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:12.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f41a1c73640 1 -- 172.21.15.125:0/2250579816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4188003ef0 msgr2=0x7f41880047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:12.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f41a1c73640 1 --2- 172.21.15.125:0/2250579816 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4188003ef0 0x7f41880047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:12.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.302+0000 7f41a1c73640 1 -- 172.21.15.125:0/2250579816 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4188003530 con 0x7f4188004d10 2023-10-30T12:59:12.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a1c73640 1 --2- 172.21.15.125:0/2250579816 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4188004d10 0x7f4188005190 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f4194001f60 tx=0x7f419401c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41b29a7fe2b13157 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:12.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f4193fff640 1 -- 172.21.15.125:0/2250579816 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41940a5750 con 0x7f4188004d10 2023-10-30T12:59:12.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f4193fff640 1 -- 172.21.15.125:0/2250579816 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41940a58b0 con 0x7f4188004d10 2023-10-30T12:59:12.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f4193fff640 1 -- 172.21.15.125:0/2250579816 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41940a5ca0 con 0x7f4188004d10 2023-10-30T12:59:12.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a36d6640 1 -- 172.21.15.125:0/2250579816 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4188004d10 msgr2=0x7f4188005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:12.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a36d6640 1 --2- 172.21.15.125:0/2250579816 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4188004d10 0x7f4188005190 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f4194001f60 tx=0x7f419401c860 comp rx=0 tx=0).stop 2023-10-30T12:59:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a1c73640 1 -- 172.21.15.125:0/2250579816 reap_dead start 2023-10-30T12:59:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a36d6640 1 -- 172.21.15.125:0/2250579816 shutdown_connections 2023-10-30T12:59:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a36d6640 1 -- 172.21.15.125:0/2250579816 >> 172.21.15.125:0/2250579816 conn(0x7f419c0c7910 msgr2=0x7f419c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a36d6640 1 -- 172.21.15.125:0/2250579816 shutdown_connections 2023-10-30T12:59:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a36d6640 1 -- 172.21.15.125:0/2250579816 wait complete. 2023-10-30T12:59:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a36d6640 1 Processor -- start 2023-10-30T12:59:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a36d6640 1 -- start start 2023-10-30T12:59:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a36d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f419c0c3a60 0x7f419c0c1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a36d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f419c0c3e40 0x7f419c15a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:12.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a36d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f419c15ae60 0x7f419c146470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:12.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a36d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c1ee350 con 0x7f419c0c3e40 2023-10-30T12:59:12.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a36d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419c1f2850 con 0x7f419c0c3a60 2023-10-30T12:59:12.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.306+0000 7f41a36d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419c153200 con 0x7f419c15ae60 2023-10-30T12:59:12.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f41a0c71640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f419c0c3e40 0x7f419c15a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:12.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f41a0c71640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f419c0c3e40 0x7f419c15a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50052/0 (socket says 172.21.15.125:50052) 2023-10-30T12:59:12.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f41a0c71640 1 -- 172.21.15.125:0/1562300004 learned_addr learned my addr 172.21.15.125:0/1562300004 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:12.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f41a1c73640 1 --2- 172.21.15.125:0/1562300004 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f419c15ae60 0x7f419c146470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:12.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f41a1472640 1 --2- 172.21.15.125:0/1562300004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f419c0c3a60 0x7f419c0c1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:12.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f41a0c71640 1 -- 172.21.15.125:0/1562300004 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f419c15ae60 msgr2=0x7f419c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:12.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f41a0c71640 1 --2- 172.21.15.125:0/1562300004 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f419c15ae60 0x7f419c146470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:12.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f41a0c71640 1 -- 172.21.15.125:0/1562300004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f419c0c3a60 msgr2=0x7f419c0c1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:12.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f41a0c71640 1 --2- 172.21.15.125:0/1562300004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f419c0c3a60 0x7f419c0c1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:12.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f41a0c71640 1 -- 172.21.15.125:0/1562300004 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c1f22d0 con 0x7f419c0c3e40 2023-10-30T12:59:12.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f41a0c71640 1 --2- 172.21.15.125:0/1562300004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f419c0c3e40 0x7f419c15a920 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f418c0072c0 tx=0x7f418c004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:12.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f4191ffb640 1 -- 172.21.15.125:0/1562300004 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c00f9c0 con 0x7f419c0c3e40 2023-10-30T12:59:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f4191ffb640 1 -- 172.21.15.125:0/1562300004 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418c00fb20 con 0x7f419c0c3e40 2023-10-30T12:59:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f4191ffb640 1 -- 172.21.15.125:0/1562300004 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c00fe90 con 0x7f419c0c3e40 2023-10-30T12:59:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f41a36d6640 1 -- 172.21.15.125:0/1562300004 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419c1f2570 con 0x7f419c0c3e40 2023-10-30T12:59:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f41a36d6640 1 -- 172.21.15.125:0/1562300004 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419c1ee820 con 0x7f419c0c3e40 2023-10-30T12:59:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f41a36d6640 1 -- 172.21.15.125:0/1562300004 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419c04e170 con 0x7f419c0c3e40 2023-10-30T12:59:12.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f4191ffb640 1 -- 172.21.15.125:0/1562300004 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f418c01c410 con 0x7f419c0c3e40 2023-10-30T12:59:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f4191ffb640 1 --2- 172.21.15.125:0/1562300004 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41800664d0 0x7f4180068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.310+0000 7f4191ffb640 1 -- 172.21.15.125:0/1562300004 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f418c0842c0 con 0x7f419c0c3e40 2023-10-30T12:59:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.314+0000 7f41a1472640 1 --2- 172.21.15.125:0/1562300004 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41800664d0 0x7f4180068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.314+0000 7f41a1472640 1 --2- 172.21.15.125:0/1562300004 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41800664d0 0x7f4180068990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f4198009b20 tx=0x7f4198008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:12.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.318+0000 7f4191ffb640 1 -- 172.21.15.125:0/1562300004 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f418c051180 con 0x7f419c0c3e40 2023-10-30T12:59:12.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.494+0000 7f41a36d6640 1 -- 172.21.15.125:0/1562300004 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f419c066510 con 0x7f419c0c3e40 2023-10-30T12:59:12.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.494+0000 7f4191ffb640 1 -- 172.21.15.125:0/1562300004 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f418c0547a0 con 0x7f419c0c3e40 2023-10-30T12:59:12.498 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:12.498 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7554,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:19.758581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":22,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:12.499 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 139 2023-10-30T12:59:12.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.498+0000 7f41a36d6640 1 -- 172.21.15.125:0/1562300004 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41800664d0 msgr2=0x7f4180068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:12.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.498+0000 7f41a36d6640 1 --2- 172.21.15.125:0/1562300004 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41800664d0 0x7f4180068990 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f4198009b20 tx=0x7f4198008040 comp rx=0 tx=0).stop 2023-10-30T12:59:12.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.498+0000 7f41a36d6640 1 -- 172.21.15.125:0/1562300004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f419c0c3e40 msgr2=0x7f419c15a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:12.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.498+0000 7f41a36d6640 1 --2- 172.21.15.125:0/1562300004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f419c0c3e40 0x7f419c15a920 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f418c0072c0 tx=0x7f418c004f90 comp rx=0 tx=0).stop 2023-10-30T12:59:12.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.498+0000 7f41a36d6640 1 -- 172.21.15.125:0/1562300004 shutdown_connections 2023-10-30T12:59:12.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.498+0000 7f41a36d6640 1 --2- 172.21.15.125:0/1562300004 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f41800664d0 0x7f4180068990 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:12.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.498+0000 7f41a36d6640 1 --2- 172.21.15.125:0/1562300004 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f419c15ae60 0x7f419c146470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:12.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.498+0000 7f41a36d6640 1 --2- 172.21.15.125:0/1562300004 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f419c0c3e40 0x7f419c15a920 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:12.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.498+0000 7f41a36d6640 1 --2- 172.21.15.125:0/1562300004 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f419c0c3a60 0x7f419c0c1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:12.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.498+0000 7f41a36d6640 1 -- 172.21.15.125:0/1562300004 >> 172.21.15.125:0/1562300004 conn(0x7f419c0c7910 msgr2=0x7f419c0c0340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:12.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.498+0000 7f41a36d6640 1 -- 172.21.15.125:0/1562300004 shutdown_connections 2023-10-30T12:59:12.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:12.498+0000 7f41a36d6640 1 -- 172.21.15.125:0/1562300004 wait complete. 2023-10-30T12:59:12.520 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:59:14.521 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:14.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61ed6dc640 1 Processor -- start 2023-10-30T12:59:14.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61ed6dc640 1 -- start start 2023-10-30T12:59:14.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61ed6dc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f61e81538f0 0x7f61e8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:14.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61ed6dc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f61e815cd50 0x7f61e80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:14.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61ed6dc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61e80cefa0 0x7f61e80cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:14.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61ed6dc640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e815d300 con 0x7f61e81538f0 2023-10-30T12:59:14.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61ed6dc640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e815d5e0 con 0x7f61e815cd50 2023-10-30T12:59:14.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61ed6dc640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e815d8c0 con 0x7f61e80cefa0 2023-10-30T12:59:14.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61e7fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f61e81538f0 0x7f61e8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61e7fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f61e81538f0 0x7f61e8159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50056/0 (socket says 172.21.15.125:50056) 2023-10-30T12:59:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61e7fff640 1 -- 172.21.15.125:0/226330253 learned_addr learned my addr 172.21.15.125:0/226330253 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61ecedb640 1 --2- 172.21.15.125:0/226330253 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61e80cefa0 0x7f61e80cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61e77fe640 1 --2- 172.21.15.125:0/226330253 >> v2:172.21.15.136:3300/0 conn(0x7f61e815cd50 0x7f61e80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:14.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61e7fff640 1 -- 172.21.15.125:0/226330253 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61e80cefa0 msgr2=0x7f61e80cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:14.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61e7fff640 1 --2- 172.21.15.125:0/226330253 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61e80cefa0 0x7f61e80cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:14.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61e7fff640 1 -- 172.21.15.125:0/226330253 >> v2:172.21.15.136:3300/0 conn(0x7f61e815cd50 msgr2=0x7f61e80c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:14.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61e7fff640 1 --2- 172.21.15.125:0/226330253 >> v2:172.21.15.136:3300/0 conn(0x7f61e815cd50 0x7f61e80c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:14.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61e7fff640 1 -- 172.21.15.125:0/226330253 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e815dba0 con 0x7f61e81538f0 2023-10-30T12:59:14.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.674+0000 7f61e77fe640 1 --2- 172.21.15.125:0/226330253 >> v2:172.21.15.136:3300/0 conn(0x7f61e815cd50 0x7f61e80c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:59:14.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.678+0000 7f61e7fff640 1 --2- 172.21.15.125:0/226330253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61e81538f0 0x7f61e8159fb0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f61e8072e90 tx=0x7f61d801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb021a60a23b8da server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:14.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.678+0000 7f61e6ffd640 1 -- 172.21.15.125:0/226330253 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d80a89b0 con 0x7f61e81538f0 2023-10-30T12:59:14.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.678+0000 7f61e6ffd640 1 -- 172.21.15.125:0/226330253 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d80a8b10 con 0x7f61e81538f0 2023-10-30T12:59:14.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.678+0000 7f61e6ffd640 1 -- 172.21.15.125:0/226330253 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d80a8ea0 con 0x7f61e81538f0 2023-10-30T12:59:14.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.678+0000 7f61ed6dc640 1 -- 172.21.15.125:0/226330253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61e81538f0 msgr2=0x7f61e8159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:14.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.678+0000 7f61ed6dc640 1 --2- 172.21.15.125:0/226330253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61e81538f0 0x7f61e8159fb0 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f61e8072e90 tx=0x7f61d801ecf0 comp rx=0 tx=0).stop 2023-10-30T12:59:14.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.678+0000 7f61ed6dc640 1 -- 172.21.15.125:0/226330253 shutdown_connections 2023-10-30T12:59:14.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.678+0000 7f61ed6dc640 1 --2- 172.21.15.125:0/226330253 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61e80cefa0 0x7f61e80cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:14.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.678+0000 7f61ed6dc640 1 --2- 172.21.15.125:0/226330253 >> v2:172.21.15.136:3300/0 conn(0x7f61e815cd50 0x7f61e80c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:14.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.678+0000 7f61ed6dc640 1 --2- 172.21.15.125:0/226330253 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61e81538f0 0x7f61e8159fb0 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:14.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.678+0000 7f61ed6dc640 1 -- 172.21.15.125:0/226330253 >> 172.21.15.125:0/226330253 conn(0x7f61e806a640 msgr2=0x7f61e8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:14.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.678+0000 7f61ed6dc640 1 -- 172.21.15.125:0/226330253 shutdown_connections 2023-10-30T12:59:14.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.678+0000 7f61ed6dc640 1 -- 172.21.15.125:0/226330253 wait complete. 2023-10-30T12:59:14.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ed6dc640 1 Processor -- start 2023-10-30T12:59:14.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ed6dc640 1 -- start start 2023-10-30T12:59:14.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ed6dc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61e80cefa0 0x7f61e814dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:14.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ed6dc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61e81538f0 0x7f61e814e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:14.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ed6dc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61e815cd50 0x7f61e8147870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:14.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ed6dc640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e81459e0 con 0x7f61e815cd50 2023-10-30T12:59:14.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ed6dc640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e814fbf0 con 0x7f61e81538f0 2023-10-30T12:59:14.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ed6dc640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e81ef200 con 0x7f61e80cefa0 2023-10-30T12:59:14.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ecedb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61e815cd50 0x7f61e8147870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:14.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ecedb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61e815cd50 0x7f61e8147870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50064/0 (socket says 172.21.15.125:50064) 2023-10-30T12:59:14.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ecedb640 1 -- 172.21.15.125:0/492499052 learned_addr learned my addr 172.21.15.125:0/492499052 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:14.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61e7fff640 1 --2- 172.21.15.125:0/492499052 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61e80cefa0 0x7f61e814dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:14.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61e77fe640 1 --2- 172.21.15.125:0/492499052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61e81538f0 0x7f61e814e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:14.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ecedb640 1 -- 172.21.15.125:0/492499052 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61e80cefa0 msgr2=0x7f61e814dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:14.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ecedb640 1 --2- 172.21.15.125:0/492499052 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61e80cefa0 0x7f61e814dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:14.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ecedb640 1 -- 172.21.15.125:0/492499052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61e81538f0 msgr2=0x7f61e814e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:14.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ecedb640 1 --2- 172.21.15.125:0/492499052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61e81538f0 0x7f61e814e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:14.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ecedb640 1 -- 172.21.15.125:0/492499052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61d80094d0 con 0x7f61e815cd50 2023-10-30T12:59:14.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ecedb640 1 --2- 172.21.15.125:0/492499052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61e815cd50 0x7f61e8147870 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f61e0007e60 tx=0x7f61e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:14.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61e4ff9640 1 -- 172.21.15.125:0/492499052 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e0014620 con 0x7f61e815cd50 2023-10-30T12:59:14.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61e4ff9640 1 -- 172.21.15.125:0/492499052 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e000ee90 con 0x7f61e815cd50 2023-10-30T12:59:14.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61e4ff9640 1 -- 172.21.15.125:0/492499052 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61e00149e0 con 0x7f61e815cd50 2023-10-30T12:59:14.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ed6dc640 1 -- 172.21.15.125:0/492499052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e814f630 con 0x7f61e815cd50 2023-10-30T12:59:14.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ed6dc640 1 -- 172.21.15.125:0/492499052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e81ef4a0 con 0x7f61e815cd50 2023-10-30T12:59:14.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.682+0000 7f61ed6dc640 1 -- 172.21.15.125:0/492499052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e81ef060 con 0x7f61e815cd50 2023-10-30T12:59:14.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.686+0000 7f61e4ff9640 1 -- 172.21.15.125:0/492499052 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f61e0014b40 con 0x7f61e815cd50 2023-10-30T12:59:14.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.686+0000 7f61e4ff9640 1 --2- 172.21.15.125:0/492499052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f61d40664a0 0x7f61d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:14.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.686+0000 7f61e4ff9640 1 -- 172.21.15.125:0/492499052 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7028+0+0 (secure 0 0 0) 0x7f61e0084bb0 con 0x7f61e815cd50 2023-10-30T12:59:14.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.686+0000 7f61e7fff640 1 --2- 172.21.15.125:0/492499052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f61d40664a0 0x7f61d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:14.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.686+0000 7f61e7fff640 1 --2- 172.21.15.125:0/492499052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f61d40664a0 0x7f61d4068960 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f61d80b2000 tx=0x7f61d8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:14.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.690+0000 7f61e4ff9640 1 -- 172.21.15.125:0/492499052 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f61e81ef060 con 0x7f61e815cd50 2023-10-30T12:59:14.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.870+0000 7f61ed6dc640 1 -- 172.21.15.125:0/492499052 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61e81ef060 con 0x7f61e815cd50 2023-10-30T12:59:14.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.870+0000 7f61e4ff9640 1 -- 172.21.15.125:0/492499052 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f61e81ef060 con 0x7f61e815cd50 2023-10-30T12:59:14.874 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:14.874 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":139,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7554,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":139}],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:58:19.758581+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7413},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7413":{"gid":7413,"name":"c","rank":0,"incarnation":134,"state":"up:active","state_seq":22,"addr":"172.21.15.125:6837/3906359913","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3906359913},{"type":"v1","addr":"172.21.15.125:6837","nonce":3906359913}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:14.875 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 139 2023-10-30T12:59:14.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.874+0000 7f61ed6dc640 1 -- 172.21.15.125:0/492499052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f61d40664a0 msgr2=0x7f61d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:14.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.874+0000 7f61ed6dc640 1 --2- 172.21.15.125:0/492499052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f61d40664a0 0x7f61d4068960 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f61d80b2000 tx=0x7f61d8009040 comp rx=0 tx=0).stop 2023-10-30T12:59:14.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.874+0000 7f61ed6dc640 1 -- 172.21.15.125:0/492499052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61e815cd50 msgr2=0x7f61e8147870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:14.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.874+0000 7f61ed6dc640 1 --2- 172.21.15.125:0/492499052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61e815cd50 0x7f61e8147870 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f61e0007e60 tx=0x7f61e000e4b0 comp rx=0 tx=0).stop 2023-10-30T12:59:14.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.874+0000 7f61ed6dc640 1 -- 172.21.15.125:0/492499052 shutdown_connections 2023-10-30T12:59:14.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.874+0000 7f61ed6dc640 1 --2- 172.21.15.125:0/492499052 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f61d40664a0 0x7f61d4068960 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:14.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.874+0000 7f61ed6dc640 1 --2- 172.21.15.125:0/492499052 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61e815cd50 0x7f61e8147870 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:14.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.874+0000 7f61ed6dc640 1 --2- 172.21.15.125:0/492499052 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61e81538f0 0x7f61e814e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:14.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.874+0000 7f61ed6dc640 1 --2- 172.21.15.125:0/492499052 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f61e80cefa0 0x7f61e814dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:14.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.874+0000 7f61ed6dc640 1 -- 172.21.15.125:0/492499052 >> 172.21.15.125:0/492499052 conn(0x7f61e806a640 msgr2=0x7f61e80ca080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:14.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.874+0000 7f61ed6dc640 1 -- 172.21.15.125:0/492499052 shutdown_connections 2023-10-30T12:59:14.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:14.874+0000 7f61ed6dc640 1 -- 172.21.15.125:0/492499052 wait complete. 2023-10-30T12:59:14.895 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T12:59:16.896 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:17.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.042+0000 7f3fe5ad6640 1 Processor -- start 2023-10-30T12:59:17.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.042+0000 7f3fe5ad6640 1 -- start start 2023-10-30T12:59:17.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.042+0000 7f3fe5ad6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3fe0153d70 0x7f3fe006ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:17.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.042+0000 7f3fe5ad6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3fe006e470 0x7f3fe00cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:17.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.042+0000 7f3fe5ad6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3fe00c8490 0x7f3fe00cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:17.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.042+0000 7f3fe5ad6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe006d780 con 0x7f3fe006e470 2023-10-30T12:59:17.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.042+0000 7f3fe5ad6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe006afa0 con 0x7f3fe0153d70 2023-10-30T12:59:17.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.042+0000 7f3fe5ad6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0154150 con 0x7f3fe00c8490 2023-10-30T12:59:17.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.042+0000 7f3fdeffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3fe006e470 0x7f3fe00cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:17.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.042+0000 7f3fdeffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3fe006e470 0x7f3fe00cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50066/0 (socket says 172.21.15.125:50066) 2023-10-30T12:59:17.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.042+0000 7f3fdeffd640 1 -- 172.21.15.125:0/187787648 learned_addr learned my addr 172.21.15.125:0/187787648 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:17.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.042+0000 7f3fdffff640 1 --2- 172.21.15.125:0/187787648 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3fe00c8490 0x7f3fe00cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:17.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.042+0000 7f3fdf7fe640 1 --2- 172.21.15.125:0/187787648 >> v2:172.21.15.136:3300/0 conn(0x7f3fe0153d70 0x7f3fe006ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:17.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fdeffd640 1 -- 172.21.15.125:0/187787648 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3fe00c8490 msgr2=0x7f3fe00cd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:17.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fdeffd640 1 --2- 172.21.15.125:0/187787648 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3fe00c8490 0x7f3fe00cd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fdeffd640 1 -- 172.21.15.125:0/187787648 >> v2:172.21.15.136:3300/0 conn(0x7f3fe0153d70 msgr2=0x7f3fe006ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:17.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fdeffd640 1 --2- 172.21.15.125:0/187787648 >> v2:172.21.15.136:3300/0 conn(0x7f3fe0153d70 0x7f3fe006ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fdeffd640 1 -- 172.21.15.125:0/187787648 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe006e030 con 0x7f3fe006e470 2023-10-30T12:59:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fdeffd640 1 --2- 172.21.15.125:0/187787648 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fe006e470 0x7f3fe00cce00 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f3fe006d180 tx=0x7f3fd401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=257b8e3d8f814006 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fde7fc640 1 -- 172.21.15.125:0/187787648 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd40aa5f0 con 0x7f3fe006e470 2023-10-30T12:59:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fde7fc640 1 -- 172.21.15.125:0/187787648 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd40aa750 con 0x7f3fe006e470 2023-10-30T12:59:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fde7fc640 1 -- 172.21.15.125:0/187787648 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd40aaae0 con 0x7f3fe006e470 2023-10-30T12:59:17.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/187787648 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fe006e470 msgr2=0x7f3fe00cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:17.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fe5ad6640 1 --2- 172.21.15.125:0/187787648 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fe006e470 0x7f3fe00cce00 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f3fe006d180 tx=0x7f3fd401c4d0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/187787648 shutdown_connections 2023-10-30T12:59:17.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fe5ad6640 1 --2- 172.21.15.125:0/187787648 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3fe00c8490 0x7f3fe00cd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fe5ad6640 1 --2- 172.21.15.125:0/187787648 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fe006e470 0x7f3fe00cce00 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fe5ad6640 1 --2- 172.21.15.125:0/187787648 >> v2:172.21.15.136:3300/0 conn(0x7f3fe0153d70 0x7f3fe006ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/187787648 >> 172.21.15.125:0/187787648 conn(0x7f3fe0071ea0 msgr2=0x7f3fe0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/187787648 shutdown_connections 2023-10-30T12:59:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.046+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/187787648 wait complete. 2023-10-30T12:59:17.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fe5ad6640 1 Processor -- start 2023-10-30T12:59:17.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fe5ad6640 1 -- start start 2023-10-30T12:59:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fe5ad6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fe006e470 0x7f3fe0156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fe5ad6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3fe00c8490 0x7f3fe014d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fe5ad6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fe0153d70 0x7f3fe00c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fe5ad6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0156a70 con 0x7f3fe0153d70 2023-10-30T12:59:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fe5ad6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0154330 con 0x7f3fe006e470 2023-10-30T12:59:17.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fe5ad6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0154b40 con 0x7f3fe00c8490 2023-10-30T12:59:17.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fdffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fe0153d70 0x7f3fe00c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:17.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fdffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fe0153d70 0x7f3fe00c7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50074/0 (socket says 172.21.15.125:50074) 2023-10-30T12:59:17.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fdffff640 1 -- 172.21.15.125:0/3731586893 learned_addr learned my addr 172.21.15.125:0/3731586893 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:17.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fdeffd640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3fe00c8490 0x7f3fe014d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:17.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fdf7fe640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fe006e470 0x7f3fe0156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:17.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fdffff640 1 -- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3fe00c8490 msgr2=0x7f3fe014d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:17.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fdffff640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3fe00c8490 0x7f3fe014d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fdffff640 1 -- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fe006e470 msgr2=0x7f3fe0156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:17.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fdffff640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fe006e470 0x7f3fe0156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fdffff640 1 -- 172.21.15.125:0/3731586893 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd40094d0 con 0x7f3fe0153d70 2023-10-30T12:59:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fdeffd640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3fe00c8490 0x7f3fe014d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:59:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fdffff640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fe0153d70 0x7f3fe00c7780 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8007e40 tx=0x7f3fd800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fc7fff640 1 -- 172.21.15.125:0/3731586893 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd8014620 con 0x7f3fe0153d70 2023-10-30T12:59:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fc7fff640 1 -- 172.21.15.125:0/3731586893 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd800ee90 con 0x7f3fe0153d70 2023-10-30T12:59:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fc7fff640 1 -- 172.21.15.125:0/3731586893 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd80149e0 con 0x7f3fe0153d70 2023-10-30T12:59:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.050+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/3731586893 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe00ccaf0 con 0x7f3fe0153d70 2023-10-30T12:59:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.054+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/3731586893 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe0154580 con 0x7f3fe0153d70 2023-10-30T12:59:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.054+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/3731586893 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe004e170 con 0x7f3fe0153d70 2023-10-30T12:59:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.054+0000 7f3fc7fff640 1 -- 172.21.15.125:0/3731586893 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3fd8014b40 con 0x7f3fe0153d70 2023-10-30T12:59:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.054+0000 7f3fc7fff640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3fcc0664a0 0x7f3fcc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:17.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.054+0000 7f3fc7fff640 1 -- 172.21.15.125:0/3731586893 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f3fd8084e60 con 0x7f3fe0153d70 2023-10-30T12:59:17.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.054+0000 7f3fdf7fe640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3fcc0664a0 0x7f3fcc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:17.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.054+0000 7f3fdf7fe640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3fcc0664a0 0x7f3fcc068960 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3fe015aa00 tx=0x7f3fd0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:17.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.058+0000 7f3fc7fff640 1 -- 172.21.15.125:0/3731586893 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3fd8051ce0 con 0x7f3fe0153d70 2023-10-30T12:59:17.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.238+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/3731586893 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3fe006dd20 con 0x7f3fe0153d70 2023-10-30T12:59:17.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.238+0000 7f3fc7fff640 1 -- 172.21.15.125:0/3731586893 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f3fd8055300 con 0x7f3fe0153d70 2023-10-30T12:59:17.240 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:17.241 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:15.462772+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:17.241 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 140 2023-10-30T12:59:17.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.238+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3fcc0664a0 msgr2=0x7f3fcc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:17.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.238+0000 7f3fe5ad6640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3fcc0664a0 0x7f3fcc068960 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f3fe015aa00 tx=0x7f3fd0008040 comp rx=0 tx=0).stop 2023-10-30T12:59:17.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.242+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/3731586893 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fe0153d70 msgr2=0x7f3fe00c7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:17.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.242+0000 7f3fe5ad6640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fe0153d70 0x7f3fe00c7780 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f3fd8007e40 tx=0x7f3fd800e4b0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.242+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/3731586893 shutdown_connections 2023-10-30T12:59:17.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.242+0000 7f3fe5ad6640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3fcc0664a0 0x7f3fcc068960 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.242+0000 7f3fe5ad6640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3fe0153d70 0x7f3fe00c7780 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.242+0000 7f3fe5ad6640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3fe00c8490 0x7f3fe014d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.242+0000 7f3fe5ad6640 1 --2- 172.21.15.125:0/3731586893 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3fe006e470 0x7f3fe0156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.242+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/3731586893 >> 172.21.15.125:0/3731586893 conn(0x7f3fe0071ea0 msgr2=0x7f3fe006f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:17.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.242+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/3731586893 shutdown_connections 2023-10-30T12:59:17.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.242+0000 7f3fe5ad6640 1 -- 172.21.15.125:0/3731586893 wait complete. 2023-10-30T12:59:17.262 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T12:59:17.262 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T12:59:17.262 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:17.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.430+0000 7f822b93e640 1 Processor -- start 2023-10-30T12:59:17.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.430+0000 7f822b93e640 1 -- start start 2023-10-30T12:59:17.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f822b93e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f82240c6b80 0x7f82240bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:17.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f822b93e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f822413f4a0 0x7f82240c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:17.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f822b93e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82240c0a50 con 0x7f82240c6f60 2023-10-30T12:59:17.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f822b93e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f822406f6e0 con 0x7f82240c6b80 2023-10-30T12:59:17.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f822b93e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f822406f9c0 con 0x7f822413f4a0 2023-10-30T12:59:17.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f8229edb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f822413f4a0 0x7f82240c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:17.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f82296da640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f82240c6b80 0x7f82240bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:17.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f8229edb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f822413f4a0 0x7f82240c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:48156/0 (socket says 172.21.15.125:48156) 2023-10-30T12:59:17.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f8229edb640 1 -- 172.21.15.125:0/2991145551 learned_addr learned my addr 172.21.15.125:0/2991145551 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:17.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f8228ed9640 1 --1- 172.21.15.125:0/2991145551 >> v1:172.21.15.136:6789/0 conn(0x7f82240c6f60 0x7f822413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:42338/0 (socket says 172.21.15.125:42338) 2023-10-30T12:59:17.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f82296da640 1 -- 172.21.15.125:0/2991145551 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f822413f4a0 msgr2=0x7f82240c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:17.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f82296da640 1 --2- 172.21.15.125:0/2991145551 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f822413f4a0 0x7f82240c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f82296da640 1 -- 172.21.15.125:0/2991145551 >> v1:172.21.15.136:6789/0 conn(0x7f82240c6f60 legacy=0x7f822413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:17.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f82296da640 1 -- 172.21.15.125:0/2991145551 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f822406fca0 con 0x7f82240c6b80 2023-10-30T12:59:17.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.434+0000 7f8229edb640 1 --2- 172.21.15.125:0/2991145551 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f822413f4a0 0x7f82240c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T12:59:17.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.438+0000 7f82296da640 1 --2- 172.21.15.125:0/2991145551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82240c6b80 0x7f82240bfec0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f82200097c0 tx=0x7f822001c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=486e717573fd7f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:17.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.438+0000 7f821bfff640 1 -- 172.21.15.125:0/2991145551 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82200ad640 con 0x7f82240c6b80 2023-10-30T12:59:17.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.438+0000 7f821bfff640 1 -- 172.21.15.125:0/2991145551 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82200ad7a0 con 0x7f82240c6b80 2023-10-30T12:59:17.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.438+0000 7f821bfff640 1 -- 172.21.15.125:0/2991145551 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82200adb30 con 0x7f82240c6b80 2023-10-30T12:59:17.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.438+0000 7f822b93e640 1 -- 172.21.15.125:0/2991145551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82240c6b80 msgr2=0x7f82240bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:17.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.438+0000 7f822b93e640 1 --2- 172.21.15.125:0/2991145551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82240c6b80 0x7f82240bfec0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f82200097c0 tx=0x7f822001c480 comp rx=0 tx=0).stop 2023-10-30T12:59:17.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.438+0000 7f822b93e640 1 -- 172.21.15.125:0/2991145551 shutdown_connections 2023-10-30T12:59:17.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.438+0000 7f822b93e640 1 --2- 172.21.15.125:0/2991145551 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f822413f4a0 0x7f82240c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.438+0000 7f822b93e640 1 --2- 172.21.15.125:0/2991145551 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82240c6b80 0x7f82240bfec0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.438+0000 7f822b93e640 1 -- 172.21.15.125:0/2991145551 >> 172.21.15.125:0/2991145551 conn(0x7f82240c7910 msgr2=0x7f82240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:17.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.438+0000 7f822b93e640 1 -- 172.21.15.125:0/2991145551 shutdown_connections 2023-10-30T12:59:17.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.438+0000 7f822b93e640 1 -- 172.21.15.125:0/2991145551 wait complete. 2023-10-30T12:59:17.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.438+0000 7f822b93e640 1 Processor -- start 2023-10-30T12:59:17.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.438+0000 7f822b93e640 1 -- start start 2023-10-30T12:59:17.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f822b93e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82240c6b80 0x7f82241356a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:17.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f822b93e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82240c6f60 0x7f82241f5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:17.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f822b93e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822413f4a0 0x7f82240c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:17.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f822b93e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82241f59e0 con 0x7f82240c6b80 2023-10-30T12:59:17.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f822b93e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8224159400 con 0x7f822413f4a0 2023-10-30T12:59:17.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f822b93e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82241f5620 con 0x7f82240c6f60 2023-10-30T12:59:17.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f82296da640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82240c6b80 0x7f82241356a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:17.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f82296da640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82240c6b80 0x7f82241356a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50088/0 (socket says 172.21.15.125:50088) 2023-10-30T12:59:17.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f82296da640 1 -- 172.21.15.125:0/610711170 learned_addr learned my addr 172.21.15.125:0/610711170 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:17.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f8228ed9640 1 --2- 172.21.15.125:0/610711170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82240c6f60 0x7f82241f5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:17.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f8229edb640 1 --2- 172.21.15.125:0/610711170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822413f4a0 0x7f82240c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:17.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f82296da640 1 -- 172.21.15.125:0/610711170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82240c6f60 msgr2=0x7f82241f5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:17.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f82296da640 1 --2- 172.21.15.125:0/610711170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82240c6f60 0x7f82241f5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f82296da640 1 -- 172.21.15.125:0/610711170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822413f4a0 msgr2=0x7f82240c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:17.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f82296da640 1 --2- 172.21.15.125:0/610711170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822413f4a0 0x7f82240c5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f82296da640 1 -- 172.21.15.125:0/610711170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f821c009b30 con 0x7f82240c6b80 2023-10-30T12:59:17.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f82296da640 1 --2- 172.21.15.125:0/610711170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82240c6b80 0x7f82241356a0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f822001a0b0 tx=0x7f8220019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:17.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f8219ffb640 1 -- 172.21.15.125:0/610711170 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82200b6840 con 0x7f82240c6b80 2023-10-30T12:59:17.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f8219ffb640 1 -- 172.21.15.125:0/610711170 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82200a6050 con 0x7f82240c6b80 2023-10-30T12:59:17.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f822b93e640 1 -- 172.21.15.125:0/610711170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8220009420 con 0x7f82240c6b80 2023-10-30T12:59:17.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f8219ffb640 1 -- 172.21.15.125:0/610711170 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82200b6bc0 con 0x7f82240c6b80 2023-10-30T12:59:17.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f822b93e640 1 -- 172.21.15.125:0/610711170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822413e180 con 0x7f82240c6b80 2023-10-30T12:59:17.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f822b93e640 1 -- 172.21.15.125:0/610711170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f822404e170 con 0x7f82240c6b80 2023-10-30T12:59:17.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.442+0000 7f8219ffb640 1 -- 172.21.15.125:0/610711170 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82200b4040 con 0x7f82240c6b80 2023-10-30T12:59:17.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.446+0000 7f8219ffb640 1 --2- 172.21.15.125:0/610711170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82080664a0 0x7f8208068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:17.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.446+0000 7f8219ffb640 1 -- 172.21.15.125:0/610711170 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f822011ec90 con 0x7f82240c6b80 2023-10-30T12:59:17.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.446+0000 7f8228ed9640 1 --2- 172.21.15.125:0/610711170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82080664a0 0x7f8208068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:17.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.446+0000 7f8228ed9640 1 --2- 172.21.15.125:0/610711170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82080664a0 0x7f8208068960 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8224159900 tx=0x7f8214007cd0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:17.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.450+0000 7f8219ffb640 1 -- 172.21.15.125:0/610711170 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82200ebc00 con 0x7f82240c6b80 2023-10-30T12:59:17.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.630+0000 7f822b93e640 1 -- 172.21.15.125:0/610711170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82240c1c50 con 0x7f82240c6b80 2023-10-30T12:59:17.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.630+0000 7f8219ffb640 1 -- 172.21.15.125:0/610711170 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f82200ba090 con 0x7f82240c6b80 2023-10-30T12:59:17.634 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:17.634 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:15.462772+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:17.635 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 140 2023-10-30T12:59:17.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.634+0000 7f822b93e640 1 -- 172.21.15.125:0/610711170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82080664a0 msgr2=0x7f8208068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:17.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.634+0000 7f822b93e640 1 --2- 172.21.15.125:0/610711170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82080664a0 0x7f8208068960 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f8224159900 tx=0x7f8214007cd0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.634+0000 7f822b93e640 1 -- 172.21.15.125:0/610711170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82240c6b80 msgr2=0x7f82241356a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:17.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.634+0000 7f822b93e640 1 --2- 172.21.15.125:0/610711170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82240c6b80 0x7f82241356a0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f822001a0b0 tx=0x7f8220019220 comp rx=0 tx=0).stop 2023-10-30T12:59:17.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.634+0000 7f822b93e640 1 -- 172.21.15.125:0/610711170 shutdown_connections 2023-10-30T12:59:17.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.634+0000 7f822b93e640 1 --2- 172.21.15.125:0/610711170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82080664a0 0x7f8208068960 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.634+0000 7f822b93e640 1 --2- 172.21.15.125:0/610711170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f822413f4a0 0x7f82240c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.634+0000 7f822b93e640 1 --2- 172.21.15.125:0/610711170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82240c6f60 0x7f82241f5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.634+0000 7f822b93e640 1 --2- 172.21.15.125:0/610711170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82240c6b80 0x7f82241356a0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:17.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.634+0000 7f822b93e640 1 -- 172.21.15.125:0/610711170 >> 172.21.15.125:0/610711170 conn(0x7f82240c7910 msgr2=0x7f82240c10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:17.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.634+0000 7f822b93e640 1 -- 172.21.15.125:0/610711170 shutdown_connections 2023-10-30T12:59:17.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:17.634+0000 7f822b93e640 1 -- 172.21.15.125:0/610711170 wait complete. 2023-10-30T12:59:17.655 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 140, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 140, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7554": { "addr": "172.21.15.125:6836/1944683138", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1944683138, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1944683138, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7554, "incarnation": 140, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 70, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:59:15.462772+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7554 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:59:19.656 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:19.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.802+0000 7f76473a7640 1 Processor -- start 2023-10-30T12:59:19.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.802+0000 7f76473a7640 1 -- start start 2023-10-30T12:59:19.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.802+0000 7f76473a7640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f76401538f0 0x7f7640159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:19.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.802+0000 7f76473a7640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76400cefa0 0x7f76400cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:19.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.802+0000 7f76473a7640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76400ca6a0 con 0x7f764015cd50 2023-10-30T12:59:19.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.802+0000 7f76473a7640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f764015d300 con 0x7f76401538f0 2023-10-30T12:59:19.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.802+0000 7f76473a7640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f764015d5e0 con 0x7f76400cefa0 2023-10-30T12:59:19.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.802+0000 7f76463a5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f76401538f0 0x7f7640159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:19.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.802+0000 7f7646ba6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76400cefa0 0x7f76400cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:19.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.802+0000 7f76463a5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f76401538f0 0x7f7640159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:44552/0 (socket says 172.21.15.125:44552) 2023-10-30T12:59:19.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.802+0000 7f76463a5640 1 -- 172.21.15.125:0/4178545676 learned_addr learned my addr 172.21.15.125:0/4178545676 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:19.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.802+0000 7f7645ba4640 1 --1- 172.21.15.125:0/4178545676 >> v1:172.21.15.136:6789/0 conn(0x7f764015cd50 0x7f76400c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:42344/0 (socket says 172.21.15.125:42344) 2023-10-30T12:59:19.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.802+0000 7f76463a5640 1 -- 172.21.15.125:0/4178545676 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76400cefa0 msgr2=0x7f76400cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:19.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.802+0000 7f76463a5640 1 --2- 172.21.15.125:0/4178545676 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76400cefa0 0x7f76400cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:19.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.806+0000 7f76463a5640 1 -- 172.21.15.125:0/4178545676 >> v1:172.21.15.136:6789/0 conn(0x7f764015cd50 legacy=0x7f76400c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:19.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.806+0000 7f76463a5640 1 -- 172.21.15.125:0/4178545676 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f764015d8c0 con 0x7f76401538f0 2023-10-30T12:59:19.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.806+0000 7f76463a5640 1 --2- 172.21.15.125:0/4178545676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76401538f0 0x7f7640159fb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f763c009870 tx=0x7f763c01ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e07f07a3e5b99f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:19.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.806+0000 7f76453a3640 1 -- 172.21.15.125:0/4178545676 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c0a8a30 con 0x7f76401538f0 2023-10-30T12:59:19.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.806+0000 7f76453a3640 1 -- 172.21.15.125:0/4178545676 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763c0a8b90 con 0x7f76401538f0 2023-10-30T12:59:19.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.806+0000 7f76453a3640 1 -- 172.21.15.125:0/4178545676 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c0b0440 con 0x7f76401538f0 2023-10-30T12:59:19.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.806+0000 7f76473a7640 1 -- 172.21.15.125:0/4178545676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76401538f0 msgr2=0x7f7640159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:19.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.806+0000 7f76473a7640 1 --2- 172.21.15.125:0/4178545676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76401538f0 0x7f7640159fb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f763c009870 tx=0x7f763c01ed30 comp rx=0 tx=0).stop 2023-10-30T12:59:19.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76473a7640 1 -- 172.21.15.125:0/4178545676 shutdown_connections 2023-10-30T12:59:19.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76473a7640 1 --2- 172.21.15.125:0/4178545676 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76400cefa0 0x7f76400cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:19.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76473a7640 1 --2- 172.21.15.125:0/4178545676 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f76401538f0 0x7f7640159fb0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:19.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76473a7640 1 -- 172.21.15.125:0/4178545676 >> 172.21.15.125:0/4178545676 conn(0x7f764006a640 msgr2=0x7f7640065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:19.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76473a7640 1 -- 172.21.15.125:0/4178545676 shutdown_connections 2023-10-30T12:59:19.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76473a7640 1 -- 172.21.15.125:0/4178545676 wait complete. 2023-10-30T12:59:19.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76473a7640 1 Processor -- start 2023-10-30T12:59:19.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76473a7640 1 -- start start 2023-10-30T12:59:19.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76473a7640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f76400cefa0 0x7f76400ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:19.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76473a7640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76401538f0 0x7f76401ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76473a7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f764015cd50 0x7f7640145a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76473a7640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f764015a9c0 con 0x7f76400cefa0 2023-10-30T12:59:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76473a7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76401590c0 con 0x7f764015cd50 2023-10-30T12:59:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76473a7640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f764015ac60 con 0x7f76401538f0 2023-10-30T12:59:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76463a5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f76400cefa0 0x7f76400ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76463a5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f76400cefa0 0x7f76400ce1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50098/0 (socket says 172.21.15.125:50098) 2023-10-30T12:59:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76463a5640 1 -- 172.21.15.125:0/1093035634 learned_addr learned my addr 172.21.15.125:0/1093035634 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f7645ba4640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76401538f0 0x7f76401ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f7646ba6640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f764015cd50 0x7f7640145a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76463a5640 1 -- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76401538f0 msgr2=0x7f76401ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76463a5640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76401538f0 0x7f76401ef940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76463a5640 1 -- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f764015cd50 msgr2=0x7f7640145a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76463a5640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f764015cd50 0x7f7640145a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76463a5640 1 -- 172.21.15.125:0/1093035634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763c0094d0 con 0x7f76400cefa0 2023-10-30T12:59:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f7645ba4640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76401538f0 0x7f76401ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:59:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f76463a5640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f76400cefa0 0x7f76400ce1c0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f763c0b3040 tx=0x7f763c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.810+0000 7f7636ffd640 1 -- 172.21.15.125:0/1093035634 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c0b0dd0 con 0x7f76400cefa0 2023-10-30T12:59:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.814+0000 7f7636ffd640 1 -- 172.21.15.125:0/1093035634 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f763c0bb460 con 0x7f76400cefa0 2023-10-30T12:59:19.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.814+0000 7f76473a7640 1 -- 172.21.15.125:0/1093035634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f764015af40 con 0x7f76400cefa0 2023-10-30T12:59:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.814+0000 7f7636ffd640 1 -- 172.21.15.125:0/1093035634 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f763c0bb810 con 0x7f76400cefa0 2023-10-30T12:59:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.814+0000 7f76473a7640 1 -- 172.21.15.125:0/1093035634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76401ef420 con 0x7f76400cefa0 2023-10-30T12:59:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.814+0000 7f76473a7640 1 -- 172.21.15.125:0/1093035634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f764004e170 con 0x7f76400cefa0 2023-10-30T12:59:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.814+0000 7f7636ffd640 1 -- 172.21.15.125:0/1093035634 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f763c0b83f0 con 0x7f76400cefa0 2023-10-30T12:59:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.814+0000 7f7636ffd640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f76240664a0 0x7f7624068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.814+0000 7f7636ffd640 1 -- 172.21.15.125:0/1093035634 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f763c121ce0 con 0x7f76400cefa0 2023-10-30T12:59:19.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.814+0000 7f7645ba4640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f76240664a0 0x7f7624068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:19.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.818+0000 7f7645ba4640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f76240664a0 0x7f7624068960 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f764015cc40 tx=0x7f7630007d70 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:19.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.818+0000 7f7636ffd640 1 -- 172.21.15.125:0/1093035634 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f763c0eec10 con 0x7f76400cefa0 2023-10-30T12:59:20.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.998+0000 7f76473a7640 1 -- 172.21.15.125:0/1093035634 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76401ef610 con 0x7f76400cefa0 2023-10-30T12:59:20.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:19.998+0000 7f7636ffd640 1 -- 172.21.15.125:0/1093035634 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f763c0c0090 con 0x7f76400cefa0 2023-10-30T12:59:20.001 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:20.001 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:15.462772+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:20.001 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 140 2023-10-30T12:59:20.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:20.002+0000 7f76473a7640 1 -- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f76240664a0 msgr2=0x7f7624068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:20.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:20.002+0000 7f76473a7640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f76240664a0 0x7f7624068960 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f764015cc40 tx=0x7f7630007d70 comp rx=0 tx=0).stop 2023-10-30T12:59:20.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:20.002+0000 7f76473a7640 1 -- 172.21.15.125:0/1093035634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f76400cefa0 msgr2=0x7f76400ce1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:20.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:20.002+0000 7f76473a7640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f76400cefa0 0x7f76400ce1c0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f763c0b3040 tx=0x7f763c01bad0 comp rx=0 tx=0).stop 2023-10-30T12:59:20.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:20.002+0000 7f76473a7640 1 -- 172.21.15.125:0/1093035634 shutdown_connections 2023-10-30T12:59:20.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:20.002+0000 7f76473a7640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f76240664a0 0x7f7624068960 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:20.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:20.002+0000 7f76473a7640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f764015cd50 0x7f7640145a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:20.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:20.002+0000 7f76473a7640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f76401538f0 0x7f76401ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:20.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:20.002+0000 7f76473a7640 1 --2- 172.21.15.125:0/1093035634 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f76400cefa0 0x7f76400ce1c0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:20.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:20.002+0000 7f76473a7640 1 -- 172.21.15.125:0/1093035634 >> 172.21.15.125:0/1093035634 conn(0x7f764006a640 msgr2=0x7f7640065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:20.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:20.002+0000 7f76473a7640 1 -- 172.21.15.125:0/1093035634 shutdown_connections 2023-10-30T12:59:20.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:20.002+0000 7f76473a7640 1 -- 172.21.15.125:0/1093035634 wait complete. 2023-10-30T12:59:20.022 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:59:21.255 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:59:21.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:59:22.023 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:22.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.166+0000 7fa007846640 1 Processor -- start 2023-10-30T12:59:22.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.170+0000 7fa007846640 1 -- start start 2023-10-30T12:59:22.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.170+0000 7fa007846640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa00015d6f0 0x7fa00015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:22.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.170+0000 7fa007846640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0000cb800 con 0x7fa0001538a0 2023-10-30T12:59:22.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.170+0000 7fa007846640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0000cb330 con 0x7fa0000cf160 2023-10-30T12:59:22.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.170+0000 7fa007846640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0000cad70 con 0x7fa00015d6f0 2023-10-30T12:59:22.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.170+0000 7fa004de1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa00015d6f0 0x7fa00015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:22.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.170+0000 7fa0055e2640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa0001538a0 0x7fa0000ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54860/0 (socket says 172.21.15.125:54860) 2023-10-30T12:59:22.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.170+0000 7fa004de1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa00015d6f0 0x7fa00015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33682/0 (socket says 172.21.15.125:33682) 2023-10-30T12:59:22.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.170+0000 7fa0055e2640 1 -- 172.21.15.125:0/4193110600 learned_addr learned my addr 172.21.15.125:0/4193110600 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:59:22.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.170+0000 7fa004de1640 1 -- 172.21.15.125:0/4193110600 >> v1:172.21.15.136:6789/0 conn(0x7fa0000cf160 legacy=0x7fa000164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:22.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.170+0000 7fa004de1640 1 -- 172.21.15.125:0/4193110600 >> v1:172.21.15.125:6789/0 conn(0x7fa0001538a0 legacy=0x7fa0000ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:22.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.170+0000 7fa004de1640 1 -- 172.21.15.125:0/4193110600 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0000cb050 con 0x7fa00015d6f0 2023-10-30T12:59:22.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.170+0000 7f9ff7fff640 1 -- 172.21.15.125:0/4193110600 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332034169 0 0) 0x7fa0000cb800 con 0x7fa0001538a0 2023-10-30T12:59:22.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.174+0000 7fa004de1640 1 --2- 172.21.15.125:0/4193110600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa00015d6f0 0x7fa00015a090 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f9ff00097c0 tx=0x7f9ff001ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf869323da401f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:22.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.174+0000 7f9ff7fff640 1 -- 172.21.15.125:0/4193110600 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00a8930 con 0x7fa00015d6f0 2023-10-30T12:59:22.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.174+0000 7f9ff7fff640 1 -- 172.21.15.125:0/4193110600 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff00a8a90 con 0x7fa00015d6f0 2023-10-30T12:59:22.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.174+0000 7f9ff7fff640 1 -- 172.21.15.125:0/4193110600 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00a8e20 con 0x7fa00015d6f0 2023-10-30T12:59:22.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 -- 172.21.15.125:0/4193110600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa00015d6f0 msgr2=0x7fa00015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:22.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 --2- 172.21.15.125:0/4193110600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa00015d6f0 0x7fa00015a090 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f9ff00097c0 tx=0x7f9ff001ec30 comp rx=0 tx=0).stop 2023-10-30T12:59:22.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 -- 172.21.15.125:0/4193110600 shutdown_connections 2023-10-30T12:59:22.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 --2- 172.21.15.125:0/4193110600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa00015d6f0 0x7fa00015a090 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:22.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 -- 172.21.15.125:0/4193110600 >> 172.21.15.125:0/4193110600 conn(0x7fa00006bfe0 msgr2=0x7fa00006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:22.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 -- 172.21.15.125:0/4193110600 shutdown_connections 2023-10-30T12:59:22.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 -- 172.21.15.125:0/4193110600 wait complete. 2023-10-30T12:59:22.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 Processor -- start 2023-10-30T12:59:22.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 -- start start 2023-10-30T12:59:22.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0000cf160 0x7fa0001e5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:22.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0001538a0 0x7fa00014ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:22.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa00015d6f0 0x7fa0000cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:22.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0001e4c80 con 0x7fa00015d6f0 2023-10-30T12:59:22.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0001e46c0 con 0x7fa0001538a0 2023-10-30T12:59:22.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa007846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0001e49a0 con 0x7fa0000cf160 2023-10-30T12:59:22.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa005de3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa00015d6f0 0x7fa0000cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:22.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa005de3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa00015d6f0 0x7fa0000cbbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33696/0 (socket says 172.21.15.125:33696) 2023-10-30T12:59:22.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa005de3640 1 -- 172.21.15.125:0/2572708942 learned_addr learned my addr 172.21.15.125:0/2572708942 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:22.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa0055e2640 1 --2- 172.21.15.125:0/2572708942 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0000cf160 0x7fa0001e5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:22.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.178+0000 7fa004de1640 1 --2- 172.21.15.125:0/2572708942 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0001538a0 0x7fa00014ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:22.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.182+0000 7fa005de3640 1 -- 172.21.15.125:0/2572708942 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0000cf160 msgr2=0x7fa0001e5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:22.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.182+0000 7fa005de3640 1 --2- 172.21.15.125:0/2572708942 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0000cf160 0x7fa0001e5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:22.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.182+0000 7fa005de3640 1 -- 172.21.15.125:0/2572708942 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0001538a0 msgr2=0x7fa00014ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:22.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.182+0000 7fa005de3640 1 --2- 172.21.15.125:0/2572708942 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0001538a0 0x7fa00014ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:22.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.182+0000 7fa005de3640 1 -- 172.21.15.125:0/2572708942 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff0009420 con 0x7fa00015d6f0 2023-10-30T12:59:22.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.182+0000 7fa005de3640 1 --2- 172.21.15.125:0/2572708942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa00015d6f0 0x7fa0000cbbe0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800b9e0 tx=0x7f9ff800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:22.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.182+0000 7f9ff5ffb640 1 -- 172.21.15.125:0/2572708942 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8003b80 con 0x7fa00015d6f0 2023-10-30T12:59:22.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.182+0000 7f9ff5ffb640 1 -- 172.21.15.125:0/2572708942 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff8003ce0 con 0x7fa00015d6f0 2023-10-30T12:59:22.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.182+0000 7f9ff5ffb640 1 -- 172.21.15.125:0/2572708942 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff800cae0 con 0x7fa00015d6f0 2023-10-30T12:59:22.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.182+0000 7fa007846640 1 -- 172.21.15.125:0/2572708942 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0001f0510 con 0x7fa00015d6f0 2023-10-30T12:59:22.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.182+0000 7fa007846640 1 -- 172.21.15.125:0/2572708942 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0001f0ad0 con 0x7fa00015d6f0 2023-10-30T12:59:22.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.182+0000 7fa007846640 1 -- 172.21.15.125:0/2572708942 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa00004e170 con 0x7fa00015d6f0 2023-10-30T12:59:22.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.182+0000 7f9ff5ffb640 1 -- 172.21.15.125:0/2572708942 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9ff800cc40 con 0x7fa00015d6f0 2023-10-30T12:59:22.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.186+0000 7f9ff5ffb640 1 --2- 172.21.15.125:0/2572708942 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fe40664a0 0x7f9fe4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:22.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.186+0000 7f9ff5ffb640 1 -- 172.21.15.125:0/2572708942 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f9ff8083fe0 con 0x7fa00015d6f0 2023-10-30T12:59:22.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.186+0000 7fa0055e2640 1 --2- 172.21.15.125:0/2572708942 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fe40664a0 0x7f9fe4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:22.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.186+0000 7fa0055e2640 1 --2- 172.21.15.125:0/2572708942 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fe40664a0 0x7f9fe4068960 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc007fd0 tx=0x7f9ffc007d60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:22.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.190+0000 7f9ff5ffb640 1 -- 172.21.15.125:0/2572708942 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9ff8050fe0 con 0x7fa00015d6f0 2023-10-30T12:59:22.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.386+0000 7fa007846640 1 -- 172.21.15.125:0/2572708942 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0000b1dd0 con 0x7fa00015d6f0 2023-10-30T12:59:22.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.386+0000 7f9ff5ffb640 1 -- 172.21.15.125:0/2572708942 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f9ff8054600 con 0x7fa00015d6f0 2023-10-30T12:59:22.388 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:22.388 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:15.462772+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:22.389 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 140 2023-10-30T12:59:22.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.386+0000 7fa007846640 1 -- 172.21.15.125:0/2572708942 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fe40664a0 msgr2=0x7f9fe4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:22.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.386+0000 7fa007846640 1 --2- 172.21.15.125:0/2572708942 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fe40664a0 0x7f9fe4068960 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f9ffc007fd0 tx=0x7f9ffc007d60 comp rx=0 tx=0).stop 2023-10-30T12:59:22.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.386+0000 7fa007846640 1 -- 172.21.15.125:0/2572708942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa00015d6f0 msgr2=0x7fa0000cbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:22.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.386+0000 7fa007846640 1 --2- 172.21.15.125:0/2572708942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa00015d6f0 0x7fa0000cbbe0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f9ff800b9e0 tx=0x7f9ff800bea0 comp rx=0 tx=0).stop 2023-10-30T12:59:22.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.386+0000 7fa007846640 1 -- 172.21.15.125:0/2572708942 shutdown_connections 2023-10-30T12:59:22.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.386+0000 7fa007846640 1 --2- 172.21.15.125:0/2572708942 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fe40664a0 0x7f9fe4068960 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:22.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.386+0000 7fa007846640 1 --2- 172.21.15.125:0/2572708942 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa00015d6f0 0x7fa0000cbbe0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:22.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.386+0000 7fa007846640 1 --2- 172.21.15.125:0/2572708942 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0001538a0 0x7fa00014ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:22.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.386+0000 7fa007846640 1 --2- 172.21.15.125:0/2572708942 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0000cf160 0x7fa0001e5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:22.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.386+0000 7fa007846640 1 -- 172.21.15.125:0/2572708942 >> 172.21.15.125:0/2572708942 conn(0x7fa00006bfe0 msgr2=0x7fa000152a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:22.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.390+0000 7fa007846640 1 -- 172.21.15.125:0/2572708942 shutdown_connections 2023-10-30T12:59:22.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:22.390+0000 7fa007846640 1 -- 172.21.15.125:0/2572708942 wait complete. 2023-10-30T12:59:22.410 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:59:24.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:24.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d9c74640 1 Processor -- start 2023-10-30T12:59:24.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d9c74640 1 -- start start 2023-10-30T12:59:24.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d9c74640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42d415cd50 0x7f42d40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:24.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d9c74640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d40cefa0 0x7f42d40cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:24.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d9c74640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d415a6c0 con 0x7f42d41538f0 2023-10-30T12:59:24.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d9c74640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42d415d300 con 0x7f42d415cd50 2023-10-30T12:59:24.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d9c74640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d415d5e0 con 0x7f42d40cefa0 2023-10-30T12:59:24.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42d415cd50 0x7f42d40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:24.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42d415cd50 0x7f42d40c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33704/0 (socket says 172.21.15.125:33704) 2023-10-30T12:59:24.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d3fff640 1 -- 172.21.15.125:0/3516194700 learned_addr learned my addr 172.21.15.125:0/3516194700 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:24.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d9473640 1 --2- 172.21.15.125:0/3516194700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d40cefa0 0x7f42d40cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:24.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d3fff640 1 -- 172.21.15.125:0/3516194700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d40cefa0 msgr2=0x7f42d40cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:24.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d3fff640 1 --2- 172.21.15.125:0/3516194700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d40cefa0 0x7f42d40cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:24.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d3fff640 1 -- 172.21.15.125:0/3516194700 >> v1:172.21.15.136:6789/0 conn(0x7f42d41538f0 legacy=0x7f42d4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:24.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.558+0000 7f42d3fff640 1 -- 172.21.15.125:0/3516194700 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d415d8c0 con 0x7f42d415cd50 2023-10-30T12:59:24.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.562+0000 7f42d3fff640 1 --2- 172.21.15.125:0/3516194700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d415cd50 0x7f42d40c9f90 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009870 tx=0x7f42c001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a9fad5fe81d1fc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:24.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.562+0000 7f42d37fe640 1 -- 172.21.15.125:0/3516194700 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c00a8a30 con 0x7f42d415cd50 2023-10-30T12:59:24.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.562+0000 7f42d37fe640 1 -- 172.21.15.125:0/3516194700 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c00a8b90 con 0x7f42d415cd50 2023-10-30T12:59:24.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.562+0000 7f42d37fe640 1 -- 172.21.15.125:0/3516194700 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c00b0440 con 0x7f42d415cd50 2023-10-30T12:59:24.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.562+0000 7f42d9c74640 1 -- 172.21.15.125:0/3516194700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d415cd50 msgr2=0x7f42d40c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:24.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.562+0000 7f42d9c74640 1 --2- 172.21.15.125:0/3516194700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d415cd50 0x7f42d40c9f90 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f42c0009870 tx=0x7f42c001ed30 comp rx=0 tx=0).stop 2023-10-30T12:59:24.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.562+0000 7f42d9c74640 1 -- 172.21.15.125:0/3516194700 shutdown_connections 2023-10-30T12:59:24.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.562+0000 7f42d9c74640 1 --2- 172.21.15.125:0/3516194700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d40cefa0 0x7f42d40cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:24.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.562+0000 7f42d9c74640 1 --2- 172.21.15.125:0/3516194700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d415cd50 0x7f42d40c9f90 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:24.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.562+0000 7f42d9c74640 1 -- 172.21.15.125:0/3516194700 >> 172.21.15.125:0/3516194700 conn(0x7f42d406a640 msgr2=0x7f42d4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:24.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.562+0000 7f42d9c74640 1 -- 172.21.15.125:0/3516194700 shutdown_connections 2023-10-30T12:59:24.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.562+0000 7f42d9c74640 1 -- 172.21.15.125:0/3516194700 wait complete. 2023-10-30T12:59:24.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d9c74640 1 Processor -- start 2023-10-30T12:59:24.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d9c74640 1 -- start start 2023-10-30T12:59:24.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d9c74640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d40cefa0 0x7f42d4158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:24.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d9c74640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d41538f0 0x7f42d40b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:24.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d9c74640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d415cd50 0x7f42d4144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:24.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d9c74640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d40b4300 con 0x7f42d40cefa0 2023-10-30T12:59:24.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d9c74640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d40b2ff0 con 0x7f42d415cd50 2023-10-30T12:59:24.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d9c74640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d4150f90 con 0x7f42d41538f0 2023-10-30T12:59:24.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d8c72640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d40cefa0 0x7f42d4158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:24.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d8c72640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d40cefa0 0x7f42d4158f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33708/0 (socket says 172.21.15.125:33708) 2023-10-30T12:59:24.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d8c72640 1 -- 172.21.15.125:0/272095494 learned_addr learned my addr 172.21.15.125:0/272095494 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:24.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d9473640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d415cd50 0x7f42d4144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:24.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d3fff640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d41538f0 0x7f42d40b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:24.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d8c72640 1 -- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d41538f0 msgr2=0x7f42d40b37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:24.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d8c72640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d41538f0 0x7f42d40b37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:24.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d8c72640 1 -- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d415cd50 msgr2=0x7f42d4144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:24.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d8c72640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d415cd50 0x7f42d4144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:24.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d8c72640 1 -- 172.21.15.125:0/272095494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c00094d0 con 0x7f42d40cefa0 2023-10-30T12:59:24.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d9473640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d415cd50 0x7f42d4144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:59:24.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d8c72640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d40cefa0 0x7f42d4158f90 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f42c800c500 tx=0x7f42c800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:24.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d17fa640 1 -- 172.21.15.125:0/272095494 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8005220 con 0x7f42d40cefa0 2023-10-30T12:59:24.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d17fa640 1 -- 172.21.15.125:0/272095494 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c8005380 con 0x7f42d40cefa0 2023-10-30T12:59:24.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.566+0000 7f42d9c74640 1 -- 172.21.15.125:0/272095494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d4151270 con 0x7f42d40cefa0 2023-10-30T12:59:24.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.570+0000 7f42d17fa640 1 -- 172.21.15.125:0/272095494 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8007270 con 0x7f42d40cefa0 2023-10-30T12:59:24.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.570+0000 7f42d9c74640 1 -- 172.21.15.125:0/272095494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d4151550 con 0x7f42d40cefa0 2023-10-30T12:59:24.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.570+0000 7f42d9c74640 1 -- 172.21.15.125:0/272095494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d40b34d0 con 0x7f42d40cefa0 2023-10-30T12:59:24.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.570+0000 7f42d17fa640 1 -- 172.21.15.125:0/272095494 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f42c80073d0 con 0x7f42d40cefa0 2023-10-30T12:59:24.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.570+0000 7f42d17fa640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42b80664d0 0x7f42b8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:24.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.570+0000 7f42d17fa640 1 -- 172.21.15.125:0/272095494 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f42c8083bf0 con 0x7f42d40cefa0 2023-10-30T12:59:24.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.574+0000 7f42d3fff640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42b80664d0 0x7f42b8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:24.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.574+0000 7f42d3fff640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42b80664d0 0x7f42b8068990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f42c00b2000 tx=0x7f42c0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:24.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.578+0000 7f42d17fa640 1 -- 172.21.15.125:0/272095494 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f42d40b34d0 con 0x7f42d40cefa0 2023-10-30T12:59:24.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.782+0000 7f42d9c74640 1 -- 172.21.15.125:0/272095494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42d40b34d0 con 0x7f42d40cefa0 2023-10-30T12:59:24.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.782+0000 7f42d17fa640 1 -- 172.21.15.125:0/272095494 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f42d40b34d0 con 0x7f42d40cefa0 2023-10-30T12:59:24.784 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:24.784 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:15.462772+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:24.785 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 140 2023-10-30T12:59:24.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.786+0000 7f42d9c74640 1 -- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42b80664d0 msgr2=0x7f42b8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:24.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.786+0000 7f42d9c74640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42b80664d0 0x7f42b8068990 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f42c00b2000 tx=0x7f42c0009040 comp rx=0 tx=0).stop 2023-10-30T12:59:24.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.786+0000 7f42d9c74640 1 -- 172.21.15.125:0/272095494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d40cefa0 msgr2=0x7f42d4158f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:24.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.786+0000 7f42d9c74640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d40cefa0 0x7f42d4158f90 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f42c800c500 tx=0x7f42c800c9c0 comp rx=0 tx=0).stop 2023-10-30T12:59:24.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.786+0000 7f42d9c74640 1 -- 172.21.15.125:0/272095494 shutdown_connections 2023-10-30T12:59:24.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.786+0000 7f42d9c74640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42b80664d0 0x7f42b8068990 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:24.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.786+0000 7f42d9c74640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d415cd50 0x7f42d4144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:24.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.786+0000 7f42d9c74640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d41538f0 0x7f42d40b37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:24.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.786+0000 7f42d9c74640 1 --2- 172.21.15.125:0/272095494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d40cefa0 0x7f42d4158f90 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:24.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.786+0000 7f42d9c74640 1 -- 172.21.15.125:0/272095494 >> 172.21.15.125:0/272095494 conn(0x7f42d406a640 msgr2=0x7f42d40cf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:24.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.786+0000 7f42d9c74640 1 -- 172.21.15.125:0/272095494 shutdown_connections 2023-10-30T12:59:24.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:24.786+0000 7f42d9c74640 1 -- 172.21.15.125:0/272095494 wait complete. 2023-10-30T12:59:24.816 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:59:26.817 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:26.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.962+0000 7faaae1c4640 1 Processor -- start 2023-10-30T12:59:26.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.962+0000 7faaae1c4640 1 -- start start 2023-10-30T12:59:26.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.962+0000 7faaae1c4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7faaa806f7a0 0x7faaa806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:26.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.962+0000 7faaae1c4640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa8066cf0 con 0x7faaa8071110 2023-10-30T12:59:26.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.962+0000 7faaae1c4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa80c70d0 con 0x7faaa80c6bd0 2023-10-30T12:59:26.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.962+0000 7faaae1c4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa80700c0 con 0x7faaa806f7a0 2023-10-30T12:59:26.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.962+0000 7faaa7fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7faaa8071110 0x7faaa8158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54862/0 (socket says 172.21.15.125:54862) 2023-10-30T12:59:26.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.962+0000 7faaa7fff640 1 -- 172.21.15.125:0/1913729780 learned_addr learned my addr 172.21.15.125:0/1913729780 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:59:26.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.962+0000 7faaa77fe640 1 --2- 172.21.15.125:0/1913729780 >> v2:172.21.15.136:3300/0 conn(0x7faaa806f7a0 0x7faaa806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:26.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.962+0000 7faaa67fc640 1 -- 172.21.15.125:0/1913729780 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144503326 0 0) 0x7faaa8066cf0 con 0x7faaa8071110 2023-10-30T12:59:26.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.962+0000 7faaa67fc640 1 -- 172.21.15.125:0/1913729780 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa90003530 con 0x7faaa8071110 2023-10-30T12:59:26.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa67fc640 1 -- 172.21.15.125:0/1913729780 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1285996265 0 0) 0x7faa90003530 con 0x7faaa8071110 2023-10-30T12:59:26.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa67fc640 1 -- 172.21.15.125:0/1913729780 >> v2:172.21.15.136:3300/0 conn(0x7faaa806f7a0 msgr2=0x7faaa806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:26.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa67fc640 1 --2- 172.21.15.125:0/1913729780 >> v2:172.21.15.136:3300/0 conn(0x7faaa806f7a0 0x7faaa806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:26.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa67fc640 1 -- 172.21.15.125:0/1913729780 >> v1:172.21.15.136:6789/0 conn(0x7faaa80c6bd0 legacy=0x7faaa8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:26.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa67fc640 1 -- 172.21.15.125:0/1913729780 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa80c7390 con 0x7faaa8071110 2023-10-30T12:59:26.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa67fc640 1 -- 172.21.15.125:0/1913729780 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7faaa0001a00 con 0x7faaa8071110 2023-10-30T12:59:26.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa67fc640 1 -- 172.21.15.125:0/1913729780 >> v1:172.21.15.125:6789/0 conn(0x7faaa8071110 legacy=0x7faaa8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:26.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa67fc640 1 --2- 172.21.15.125:0/1913729780 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faa90003980 0x7faa90003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:26.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa67fc640 1 --2- 172.21.15.125:0/1913729780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa900047d0 0x7faa90004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:26.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa67fc640 1 --2- 172.21.15.125:0/1913729780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa90005190 0x7faa9000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:26.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa67fc640 1 -- 172.21.15.125:0/1913729780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa80700c0 con 0x7faa90005190 2023-10-30T12:59:26.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa67fc640 1 -- 172.21.15.125:0/1913729780 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9000c040 con 0x7faa900047d0 2023-10-30T12:59:26.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa67fc640 1 -- 172.21.15.125:0/1913729780 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9000c200 con 0x7faa90003980 2023-10-30T12:59:26.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa7fff640 1 --2- 172.21.15.125:0/1913729780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa90005190 0x7faa9000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:26.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa6ffd640 1 --2- 172.21.15.125:0/1913729780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa900047d0 0x7faa90004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:26.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa77fe640 1 --2- 172.21.15.125:0/1913729780 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faa90003980 0x7faa90003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:26.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa7fff640 1 -- 172.21.15.125:0/1913729780 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faa90003980 msgr2=0x7faa90003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:26.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa7fff640 1 --2- 172.21.15.125:0/1913729780 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faa90003980 0x7faa90003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:26.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa7fff640 1 -- 172.21.15.125:0/1913729780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa900047d0 msgr2=0x7faa90004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:26.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa7fff640 1 --2- 172.21.15.125:0/1913729780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa900047d0 0x7faa90004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:26.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa7fff640 1 -- 172.21.15.125:0/1913729780 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa90003530 con 0x7faa90005190 2023-10-30T12:59:26.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa6ffd640 1 --2- 172.21.15.125:0/1913729780 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faa900047d0 0x7faa90004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:59:26.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa77fe640 1 --2- 172.21.15.125:0/1913729780 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faa90003980 0x7faa90003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:59:26.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa7fff640 1 -- 172.21.15.125:0/1913729780 reap_dead start 2023-10-30T12:59:26.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.966+0000 7faaa7fff640 1 --2- 172.21.15.125:0/1913729780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa90005190 0x7faa9000b990 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7faaa000a830 tx=0x7faaa001d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b0b55ad7e1fe03a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:26.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaa67fc640 1 -- 172.21.15.125:0/1913729780 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa00072d0 con 0x7faa90005190 2023-10-30T12:59:26.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaa67fc640 1 -- 172.21.15.125:0/1913729780 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa00a8070 con 0x7faa90005190 2023-10-30T12:59:26.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaa67fc640 1 -- 172.21.15.125:0/1913729780 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa00076b0 con 0x7faa90005190 2023-10-30T12:59:26.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 -- 172.21.15.125:0/1913729780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa90005190 msgr2=0x7faa9000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:26.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 --2- 172.21.15.125:0/1913729780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa90005190 0x7faa9000b990 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7faaa000a830 tx=0x7faaa001d340 comp rx=0 tx=0).stop 2023-10-30T12:59:26.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 -- 172.21.15.125:0/1913729780 shutdown_connections 2023-10-30T12:59:26.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 --2- 172.21.15.125:0/1913729780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa90005190 0x7faa9000b990 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:26.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 -- 172.21.15.125:0/1913729780 >> 172.21.15.125:0/1913729780 conn(0x7faaa80c7910 msgr2=0x7faaa80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:26.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 -- 172.21.15.125:0/1913729780 shutdown_connections 2023-10-30T12:59:26.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 -- 172.21.15.125:0/1913729780 wait complete. 2023-10-30T12:59:26.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 Processor -- start 2023-10-30T12:59:26.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 -- start start 2023-10-30T12:59:26.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa90005190 0x7faaa81ef8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:26.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaa80c9540 0x7faaa815b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:26.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaa8156940 0x7faaa8156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:26.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa8156520 con 0x7faa90005190 2023-10-30T12:59:26.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaa81efde0 con 0x7faaa80c9540 2023-10-30T12:59:26.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaae1c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa81580e0 con 0x7faaa8156940 2023-10-30T12:59:26.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaa77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa90005190 0x7faaa81ef8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:26.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaa77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa90005190 0x7faaa81ef8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33716/0 (socket says 172.21.15.125:33716) 2023-10-30T12:59:26.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.970+0000 7faaa77fe640 1 -- 172.21.15.125:0/112721024 learned_addr learned my addr 172.21.15.125:0/112721024 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:26.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faaa7fff640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaa8156940 0x7faaa8156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:26.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faaa6ffd640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaa80c9540 0x7faaa815b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:26.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faaa77fe640 1 -- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaa8156940 msgr2=0x7faaa8156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:26.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faaa77fe640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaa8156940 0x7faaa8156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:26.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faaa77fe640 1 -- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaa80c9540 msgr2=0x7faaa815b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:26.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faaa77fe640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaa80c9540 0x7faaa815b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:26.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faaa77fe640 1 -- 172.21.15.125:0/112721024 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa8155600 con 0x7faa90005190 2023-10-30T12:59:26.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faaa7fff640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaa8156940 0x7faaa8156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:59:26.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faaa77fe640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa90005190 0x7faaa81ef8a0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7faa98011560 tx=0x7faa98011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:26.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faa8ffff640 1 -- 172.21.15.125:0/112721024 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa98004220 con 0x7faa90005190 2023-10-30T12:59:26.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faa8ffff640 1 -- 172.21.15.125:0/112721024 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa98004380 con 0x7faa90005190 2023-10-30T12:59:26.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faaae1c4640 1 -- 172.21.15.125:0/112721024 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa81558a0 con 0x7faa90005190 2023-10-30T12:59:26.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faa8ffff640 1 -- 172.21.15.125:0/112721024 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa98015870 con 0x7faa90005190 2023-10-30T12:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faaae1c4640 1 -- 172.21.15.125:0/112721024 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa8155e60 con 0x7faa90005190 2023-10-30T12:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faaae1c4640 1 -- 172.21.15.125:0/112721024 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa804e170 con 0x7faa90005190 2023-10-30T12:59:26.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.974+0000 7faa8ffff640 1 -- 172.21.15.125:0/112721024 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faa980159d0 con 0x7faa90005190 2023-10-30T12:59:26.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.978+0000 7faa8ffff640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faa940664d0 0x7faa94068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:26.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.978+0000 7faa8ffff640 1 -- 172.21.15.125:0/112721024 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7faa98084950 con 0x7faa90005190 2023-10-30T12:59:26.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.978+0000 7faaa6ffd640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faa940664d0 0x7faa94068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:26.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.978+0000 7faaa6ffd640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faa940664d0 0x7faa94068990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7faa9c000c10 tx=0x7faa9c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:26.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:26.982+0000 7faa8ffff640 1 -- 172.21.15.125:0/112721024 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faa98051730 con 0x7faa90005190 2023-10-30T12:59:27.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:27.162+0000 7faaae1c4640 1 -- 172.21.15.125:0/112721024 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faaa8156790 con 0x7faa90005190 2023-10-30T12:59:27.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:27.162+0000 7faa8ffff640 1 -- 172.21.15.125:0/112721024 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+5727 (secure 0 0 0) 0x7faa98054d50 con 0x7faa90005190 2023-10-30T12:59:27.164 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:27.165 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:25.790911+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:27.166 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 141 2023-10-30T12:59:27.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:27.162+0000 7faaae1c4640 1 -- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faa940664d0 msgr2=0x7faa94068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:27.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:27.162+0000 7faaae1c4640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faa940664d0 0x7faa94068990 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7faa9c000c10 tx=0x7faa9c009040 comp rx=0 tx=0).stop 2023-10-30T12:59:27.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:27.162+0000 7faaae1c4640 1 -- 172.21.15.125:0/112721024 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa90005190 msgr2=0x7faaa81ef8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:27.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:27.162+0000 7faaae1c4640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa90005190 0x7faaa81ef8a0 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7faa98011560 tx=0x7faa98011a20 comp rx=0 tx=0).stop 2023-10-30T12:59:27.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:27.162+0000 7faaae1c4640 1 -- 172.21.15.125:0/112721024 shutdown_connections 2023-10-30T12:59:27.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:27.162+0000 7faaae1c4640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faa940664d0 0x7faa94068990 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:27.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:27.162+0000 7faaae1c4640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaa8156940 0x7faaa8156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:27.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:27.162+0000 7faaae1c4640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaa80c9540 0x7faaa815b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:27.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:27.162+0000 7faaae1c4640 1 --2- 172.21.15.125:0/112721024 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faa90005190 0x7faaa81ef8a0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:27.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:27.162+0000 7faaae1c4640 1 -- 172.21.15.125:0/112721024 >> 172.21.15.125:0/112721024 conn(0x7faaa80c7910 msgr2=0x7faaa80c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:27.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:27.162+0000 7faaae1c4640 1 -- 172.21.15.125:0/112721024 shutdown_connections 2023-10-30T12:59:27.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:27.162+0000 7faaae1c4640 1 -- 172.21.15.125:0/112721024 wait complete. 2023-10-30T12:59:27.184 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:59:29.185 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:29.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe19510640 1 Processor -- start 2023-10-30T12:59:29.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe19510640 1 -- start start 2023-10-30T12:59:29.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe19510640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7efe140cefa0 0x7efe140cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:29.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe19510640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1415a6c0 con 0x7efe1415cd50 2023-10-30T12:59:29.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe19510640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe140ca6a0 con 0x7efe141538f0 2023-10-30T12:59:29.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe19510640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe140cf8f0 con 0x7efe140cefa0 2023-10-30T12:59:29.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe137fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7efe1415cd50 0x7efe140c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54876/0 (socket says 172.21.15.125:54876) 2023-10-30T12:59:29.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe137fe640 1 -- 172.21.15.125:0/1558438583 learned_addr learned my addr 172.21.15.125:0/1558438583 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:59:29.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe18d0f640 1 --2- 172.21.15.125:0/1558438583 >> v2:172.21.15.136:3300/0 conn(0x7efe140cefa0 0x7efe140cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:29.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 -- 172.21.15.125:0/1558438583 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215595195 0 0) 0x7efe1415a6c0 con 0x7efe1415cd50 2023-10-30T12:59:29.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 -- 172.21.15.125:0/1558438583 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdfc003530 con 0x7efe1415cd50 2023-10-30T12:59:29.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 -- 172.21.15.125:0/1558438583 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2153971121 0 0) 0x7efdfc003530 con 0x7efe1415cd50 2023-10-30T12:59:29.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 -- 172.21.15.125:0/1558438583 >> v2:172.21.15.136:3300/0 conn(0x7efe140cefa0 msgr2=0x7efe140cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:29.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 --2- 172.21.15.125:0/1558438583 >> v2:172.21.15.136:3300/0 conn(0x7efe140cefa0 0x7efe140cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:29.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 -- 172.21.15.125:0/1558438583 >> v1:172.21.15.136:6789/0 conn(0x7efe141538f0 legacy=0x7efe14159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:29.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 -- 172.21.15.125:0/1558438583 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1415d300 con 0x7efe1415cd50 2023-10-30T12:59:29.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 -- 172.21.15.125:0/1558438583 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7efe08001680 con 0x7efe1415cd50 2023-10-30T12:59:29.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 -- 172.21.15.125:0/1558438583 >> v1:172.21.15.125:6789/0 conn(0x7efe1415cd50 legacy=0x7efe140c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:29.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 --2- 172.21.15.125:0/1558438583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdfc003980 0x7efdfc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:29.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 --2- 172.21.15.125:0/1558438583 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdfc0047d0 0x7efdfc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:29.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 --2- 172.21.15.125:0/1558438583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdfc005190 0x7efdfc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:29.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 -- 172.21.15.125:0/1558438583 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe140cf8f0 con 0x7efdfc003980 2023-10-30T12:59:29.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 -- 172.21.15.125:0/1558438583 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc00c040 con 0x7efdfc005190 2023-10-30T12:59:29.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe12ffd640 1 -- 172.21.15.125:0/1558438583 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc00c200 con 0x7efdfc0047d0 2023-10-30T12:59:29.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe13fff640 1 --2- 172.21.15.125:0/1558438583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdfc003980 0x7efdfc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:29.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe18d0f640 1 --2- 172.21.15.125:0/1558438583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdfc005190 0x7efdfc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:29.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe137fe640 1 --2- 172.21.15.125:0/1558438583 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdfc0047d0 0x7efdfc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:29.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe13fff640 1 -- 172.21.15.125:0/1558438583 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdfc0047d0 msgr2=0x7efdfc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:29.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe13fff640 1 --2- 172.21.15.125:0/1558438583 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdfc0047d0 0x7efdfc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:29.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe13fff640 1 -- 172.21.15.125:0/1558438583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdfc005190 msgr2=0x7efdfc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:29.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe13fff640 1 --2- 172.21.15.125:0/1558438583 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdfc005190 0x7efdfc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:29.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe13fff640 1 -- 172.21.15.125:0/1558438583 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdfc003530 con 0x7efdfc003980 2023-10-30T12:59:29.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.334+0000 7efe18d0f640 1 -- 172.21.15.125:0/1558438583 reap_dead start 2023-10-30T12:59:29.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.338+0000 7efe13fff640 1 --2- 172.21.15.125:0/1558438583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdfc003980 0x7efdfc003da0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7efe04001780 tx=0x7efe0401c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70336702bc95538 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:29.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.338+0000 7efe12ffd640 1 -- 172.21.15.125:0/1558438583 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe040a4ea0 con 0x7efdfc003980 2023-10-30T12:59:29.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.338+0000 7efe12ffd640 1 -- 172.21.15.125:0/1558438583 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe040a7050 con 0x7efdfc003980 2023-10-30T12:59:29.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.338+0000 7efe12ffd640 1 -- 172.21.15.125:0/1558438583 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe04003600 con 0x7efdfc003980 2023-10-30T12:59:29.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.338+0000 7efe19510640 1 -- 172.21.15.125:0/1558438583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdfc003980 msgr2=0x7efdfc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:29.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.338+0000 7efe19510640 1 --2- 172.21.15.125:0/1558438583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdfc003980 0x7efdfc003da0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7efe04001780 tx=0x7efe0401c3e0 comp rx=0 tx=0).stop 2023-10-30T12:59:29.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.338+0000 7efe19510640 1 -- 172.21.15.125:0/1558438583 shutdown_connections 2023-10-30T12:59:29.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.338+0000 7efe19510640 1 --2- 172.21.15.125:0/1558438583 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdfc003980 0x7efdfc003da0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:29.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.338+0000 7efe19510640 1 -- 172.21.15.125:0/1558438583 >> 172.21.15.125:0/1558438583 conn(0x7efe1406a640 msgr2=0x7efe14065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:29.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.338+0000 7efe19510640 1 -- 172.21.15.125:0/1558438583 shutdown_connections 2023-10-30T12:59:29.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe19510640 1 -- 172.21.15.125:0/1558438583 wait complete. 2023-10-30T12:59:29.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe19510640 1 Processor -- start 2023-10-30T12:59:29.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe19510640 1 -- start start 2023-10-30T12:59:29.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe19510640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdfc003980 0x7efe1414f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:29.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe19510640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe1414fad0 0x7efe140b4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:29.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe19510640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe14155610 0x7efe14155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:29.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe19510640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe1414ff50 con 0x7efe1414fad0 2023-10-30T12:59:29.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe19510640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe14154c90 con 0x7efdfc003980 2023-10-30T12:59:29.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe19510640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe14154870 con 0x7efe14155610 2023-10-30T12:59:29.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe137fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe1414fad0 0x7efe140b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:29.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe137fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe1414fad0 0x7efe140b4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33722/0 (socket says 172.21.15.125:33722) 2023-10-30T12:59:29.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe137fe640 1 -- 172.21.15.125:0/2400189628 learned_addr learned my addr 172.21.15.125:0/2400189628 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:29.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe13fff640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdfc003980 0x7efe1414f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:29.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe18d0f640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe14155610 0x7efe14155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:29.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe137fe640 1 -- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe14155610 msgr2=0x7efe14155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:29.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe137fe640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe14155610 0x7efe14155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:29.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe137fe640 1 -- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdfc003980 msgr2=0x7efe1414f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:29.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe137fe640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdfc003980 0x7efe1414f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:29.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe137fe640 1 -- 172.21.15.125:0/2400189628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe14155fd0 con 0x7efe1414fad0 2023-10-30T12:59:29.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe18d0f640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe14155610 0x7efe14155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:59:29.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe137fe640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe1414fad0 0x7efe140b4150 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7efe08011810 tx=0x7efe08011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:29.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.342+0000 7efe10ff9640 1 -- 172.21.15.125:0/2400189628 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe0800fb60 con 0x7efe1414fad0 2023-10-30T12:59:29.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.346+0000 7efe10ff9640 1 -- 172.21.15.125:0/2400189628 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe0800fcc0 con 0x7efe1414fad0 2023-10-30T12:59:29.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.346+0000 7efe10ff9640 1 -- 172.21.15.125:0/2400189628 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe08015630 con 0x7efe1414fad0 2023-10-30T12:59:29.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.346+0000 7efe19510640 1 -- 172.21.15.125:0/2400189628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe1414e750 con 0x7efe1414fad0 2023-10-30T12:59:29.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.346+0000 7efe19510640 1 -- 172.21.15.125:0/2400189628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe1414ed10 con 0x7efe1414fad0 2023-10-30T12:59:29.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.346+0000 7efe19510640 1 -- 172.21.15.125:0/2400189628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe1404e170 con 0x7efe1414fad0 2023-10-30T12:59:29.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.346+0000 7efe10ff9640 1 -- 172.21.15.125:0/2400189628 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efe08015790 con 0x7efe1414fad0 2023-10-30T12:59:29.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.346+0000 7efe10ff9640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe000664a0 0x7efe00068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:29.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.346+0000 7efe10ff9640 1 -- 172.21.15.125:0/2400189628 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7efe08087650 con 0x7efe1414fad0 2023-10-30T12:59:29.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.346+0000 7efe13fff640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe000664a0 0x7efe00068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:29.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.350+0000 7efe13fff640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe000664a0 0x7efe00068960 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7efe04001750 tx=0x7efe040011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:29.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.350+0000 7efe10ff9640 1 -- 172.21.15.125:0/2400189628 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efe08057f40 con 0x7efe1414fad0 2023-10-30T12:59:29.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.550+0000 7efe19510640 1 -- 172.21.15.125:0/2400189628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe1414d980 con 0x7efe1414fad0 2023-10-30T12:59:29.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.550+0000 7efe10ff9640 1 -- 172.21.15.125:0/2400189628 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+5727 (secure 0 0 0) 0x7efe08011e60 con 0x7efe1414fad0 2023-10-30T12:59:29.552 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:29.553 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:25.790911+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:29.553 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 141 2023-10-30T12:59:29.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.550+0000 7efe19510640 1 -- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe000664a0 msgr2=0x7efe00068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:29.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.550+0000 7efe19510640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe000664a0 0x7efe00068960 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7efe04001750 tx=0x7efe040011a0 comp rx=0 tx=0).stop 2023-10-30T12:59:29.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.550+0000 7efe19510640 1 -- 172.21.15.125:0/2400189628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe1414fad0 msgr2=0x7efe140b4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:29.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.550+0000 7efe19510640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe1414fad0 0x7efe140b4150 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7efe08011810 tx=0x7efe08011c10 comp rx=0 tx=0).stop 2023-10-30T12:59:29.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.550+0000 7efe19510640 1 -- 172.21.15.125:0/2400189628 shutdown_connections 2023-10-30T12:59:29.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.550+0000 7efe19510640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe000664a0 0x7efe00068960 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:29.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.550+0000 7efe19510640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe14155610 0x7efe14155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:29.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.550+0000 7efe19510640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe1414fad0 0x7efe140b4150 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:29.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.550+0000 7efe19510640 1 --2- 172.21.15.125:0/2400189628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdfc003980 0x7efe1414f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:29.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.550+0000 7efe19510640 1 -- 172.21.15.125:0/2400189628 >> 172.21.15.125:0/2400189628 conn(0x7efe1406a640 msgr2=0x7efe140c3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:29.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.550+0000 7efe19510640 1 -- 172.21.15.125:0/2400189628 shutdown_connections 2023-10-30T12:59:29.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:29.554+0000 7efe19510640 1 -- 172.21.15.125:0/2400189628 wait complete. 2023-10-30T12:59:29.573 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 141, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 141, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7554": { "addr": "172.21.15.125:6836/1944683138", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1944683138, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1944683138, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7554, "incarnation": 140, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:reconnect", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 70, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:59:25.790911+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7554 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:59:31.574 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:31.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.718+0000 7ff435ba1640 1 Processor -- start 2023-10-30T12:59:31.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.718+0000 7ff435ba1640 1 -- start start 2023-10-30T12:59:31.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff435ba1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff430069e90 0x7ff43015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:31.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff435ba1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff43006a270 0x7ff4300c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:31.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff435ba1640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4300c6c10 con 0x7ff43006cb80 2023-10-30T12:59:31.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff435ba1640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4300c8570 con 0x7ff43006a270 2023-10-30T12:59:31.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff435ba1640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4300c6ef0 con 0x7ff430069e90 2023-10-30T12:59:31.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff42effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff43006a270 0x7ff4300c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:31.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff42effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff43006a270 0x7ff4300c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53112/0 (socket says 172.21.15.125:53112) 2023-10-30T12:59:31.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff42effd640 1 -- 172.21.15.125:0/2712794644 learned_addr learned my addr 172.21.15.125:0/2712794644 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:31.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff42ffff640 1 --1- 172.21.15.125:0/2712794644 >> v1:172.21.15.125:6789/0 conn(0x7ff43006cb80 0x7ff43006cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37550/0 (socket says 172.21.15.125:37550) 2023-10-30T12:59:31.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff42f7fe640 1 --2- 172.21.15.125:0/2712794644 >> v2:172.21.15.136:3300/0 conn(0x7ff430069e90 0x7ff43015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:31.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff42effd640 1 -- 172.21.15.125:0/2712794644 >> v2:172.21.15.136:3300/0 conn(0x7ff430069e90 msgr2=0x7ff43015a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:31.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff42effd640 1 --2- 172.21.15.125:0/2712794644 >> v2:172.21.15.136:3300/0 conn(0x7ff430069e90 0x7ff43015a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:31.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff42effd640 1 -- 172.21.15.125:0/2712794644 >> v1:172.21.15.125:6789/0 conn(0x7ff43006cb80 legacy=0x7ff43006cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:31.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff42effd640 1 -- 172.21.15.125:0/2712794644 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4300c71d0 con 0x7ff43006a270 2023-10-30T12:59:31.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff42effd640 1 --2- 172.21.15.125:0/2712794644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43006a270 0x7ff4300c9cb0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7ff43006ed20 tx=0x7ff42401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f29fc35fe34f86a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:31.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff42e7fc640 1 -- 172.21.15.125:0/2712794644 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4240ab5f0 con 0x7ff43006a270 2023-10-30T12:59:31.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff42e7fc640 1 -- 172.21.15.125:0/2712794644 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4240ab750 con 0x7ff43006a270 2023-10-30T12:59:31.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.722+0000 7ff42e7fc640 1 -- 172.21.15.125:0/2712794644 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4240abae0 con 0x7ff43006a270 2023-10-30T12:59:31.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.726+0000 7ff435ba1640 1 -- 172.21.15.125:0/2712794644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43006a270 msgr2=0x7ff4300c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:31.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.726+0000 7ff435ba1640 1 --2- 172.21.15.125:0/2712794644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43006a270 0x7ff4300c9cb0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7ff43006ed20 tx=0x7ff42401c500 comp rx=0 tx=0).stop 2023-10-30T12:59:31.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.726+0000 7ff435ba1640 1 -- 172.21.15.125:0/2712794644 shutdown_connections 2023-10-30T12:59:31.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.726+0000 7ff435ba1640 1 --2- 172.21.15.125:0/2712794644 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43006a270 0x7ff4300c9cb0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:31.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.726+0000 7ff435ba1640 1 --2- 172.21.15.125:0/2712794644 >> v2:172.21.15.136:3300/0 conn(0x7ff430069e90 0x7ff43015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:31.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.726+0000 7ff435ba1640 1 -- 172.21.15.125:0/2712794644 >> 172.21.15.125:0/2712794644 conn(0x7ff43006dfd0 msgr2=0x7ff43006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:31.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.726+0000 7ff435ba1640 1 -- 172.21.15.125:0/2712794644 shutdown_connections 2023-10-30T12:59:31.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.726+0000 7ff435ba1640 1 -- 172.21.15.125:0/2712794644 wait complete. 2023-10-30T12:59:31.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.726+0000 7ff435ba1640 1 Processor -- start 2023-10-30T12:59:31.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.726+0000 7ff435ba1640 1 -- start start 2023-10-30T12:59:31.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff435ba1640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff430069e90 0x7ff43014a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:31.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff435ba1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43006a270 0x7ff430152540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:31.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff435ba1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43006cb80 0x7ff43014c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:31.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff435ba1640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff430152e40 con 0x7ff43006a270 2023-10-30T12:59:31.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff435ba1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43014aaf0 con 0x7ff43006cb80 2023-10-30T12:59:31.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff435ba1640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff430152a80 con 0x7ff430069e90 2023-10-30T12:59:31.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff42effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43006a270 0x7ff430152540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:31.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff42effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43006a270 0x7ff430152540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53122/0 (socket says 172.21.15.125:53122) 2023-10-30T12:59:31.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff42effd640 1 -- 172.21.15.125:0/1062893874 learned_addr learned my addr 172.21.15.125:0/1062893874 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:31.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff42effd640 1 -- 172.21.15.125:0/1062893874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff430069e90 msgr2=0x7ff43014a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:31.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff42effd640 1 --2- 172.21.15.125:0/1062893874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff430069e90 0x7ff43014a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:31.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff42effd640 1 -- 172.21.15.125:0/1062893874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43006cb80 msgr2=0x7ff43014c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:31.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff42effd640 1 --2- 172.21.15.125:0/1062893874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43006cb80 0x7ff43014c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:31.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff42effd640 1 -- 172.21.15.125:0/1062893874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4240094d0 con 0x7ff43006a270 2023-10-30T12:59:31.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff42effd640 1 --2- 172.21.15.125:0/1062893874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43006a270 0x7ff430152540 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7ff424019110 tx=0x7ff42401ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:31.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.730+0000 7ff417fff640 1 -- 172.21.15.125:0/1062893874 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4240be5f0 con 0x7ff43006a270 2023-10-30T12:59:31.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.734+0000 7ff417fff640 1 -- 172.21.15.125:0/1062893874 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4240be750 con 0x7ff43006a270 2023-10-30T12:59:31.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.734+0000 7ff417fff640 1 -- 172.21.15.125:0/1062893874 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4240beb00 con 0x7ff43006a270 2023-10-30T12:59:31.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.734+0000 7ff435ba1640 1 -- 172.21.15.125:0/1062893874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4301538a0 con 0x7ff43006a270 2023-10-30T12:59:31.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.734+0000 7ff435ba1640 1 -- 172.21.15.125:0/1062893874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff430153330 con 0x7ff43006a270 2023-10-30T12:59:31.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.734+0000 7ff435ba1640 1 -- 172.21.15.125:0/1062893874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff430153520 con 0x7ff43006a270 2023-10-30T12:59:31.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.734+0000 7ff417fff640 1 -- 172.21.15.125:0/1062893874 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4240b43f0 con 0x7ff43006a270 2023-10-30T12:59:31.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.738+0000 7ff417fff640 1 --2- 172.21.15.125:0/1062893874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff41c066520 0x7ff41c0689e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:31.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.738+0000 7ff417fff640 1 -- 172.21.15.125:0/1062893874 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7ff4240a8050 con 0x7ff43006a270 2023-10-30T12:59:31.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.738+0000 7ff42f7fe640 1 --2- 172.21.15.125:0/1062893874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff41c066520 0x7ff41c0689e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:31.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.738+0000 7ff42f7fe640 1 --2- 172.21.15.125:0/1062893874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff41c066520 0x7ff41c0689e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff4200098f0 tx=0x7ff420008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:31.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.742+0000 7ff417fff640 1 -- 172.21.15.125:0/1062893874 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff4240ec420 con 0x7ff43006a270 2023-10-30T12:59:31.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.926+0000 7ff435ba1640 1 -- 172.21.15.125:0/1062893874 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff43004e170 con 0x7ff43006a270 2023-10-30T12:59:31.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.926+0000 7ff417fff640 1 -- 172.21.15.125:0/1062893874 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+5727 (secure 0 0 0) 0x7ff4240efa40 con 0x7ff43006a270 2023-10-30T12:59:31.927 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:31.928 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:25.790911+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 141 2023-10-30T12:59:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.926+0000 7ff435ba1640 1 -- 172.21.15.125:0/1062893874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff41c066520 msgr2=0x7ff41c0689e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.926+0000 7ff435ba1640 1 --2- 172.21.15.125:0/1062893874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff41c066520 0x7ff41c0689e0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff4200098f0 tx=0x7ff420008040 comp rx=0 tx=0).stop 2023-10-30T12:59:31.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.926+0000 7ff435ba1640 1 -- 172.21.15.125:0/1062893874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43006a270 msgr2=0x7ff430152540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:31.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.926+0000 7ff435ba1640 1 --2- 172.21.15.125:0/1062893874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43006a270 0x7ff430152540 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7ff424019110 tx=0x7ff42401ad40 comp rx=0 tx=0).stop 2023-10-30T12:59:31.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.926+0000 7ff435ba1640 1 -- 172.21.15.125:0/1062893874 shutdown_connections 2023-10-30T12:59:31.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.926+0000 7ff435ba1640 1 --2- 172.21.15.125:0/1062893874 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff41c066520 0x7ff41c0689e0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:31.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.926+0000 7ff435ba1640 1 --2- 172.21.15.125:0/1062893874 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff43006cb80 0x7ff43014c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:31.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.926+0000 7ff435ba1640 1 --2- 172.21.15.125:0/1062893874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff43006a270 0x7ff430152540 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:31.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.926+0000 7ff435ba1640 1 --2- 172.21.15.125:0/1062893874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff430069e90 0x7ff43014a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:31.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.926+0000 7ff435ba1640 1 -- 172.21.15.125:0/1062893874 >> 172.21.15.125:0/1062893874 conn(0x7ff43006dfd0 msgr2=0x7ff4300ca5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:31.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.926+0000 7ff435ba1640 1 -- 172.21.15.125:0/1062893874 shutdown_connections 2023-10-30T12:59:31.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:31.926+0000 7ff435ba1640 1 -- 172.21.15.125:0/1062893874 wait complete. 2023-10-30T12:59:31.948 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:59:33.948 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:34.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.094+0000 7f8ae553e640 1 Processor -- start 2023-10-30T12:59:34.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.094+0000 7f8ae553e640 1 -- start start 2023-10-30T12:59:34.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.094+0000 7f8ae553e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8ae00c90d0 0x7f8ae00cef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:34.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.094+0000 7f8ae553e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae015ce30 con 0x7f8ae00c8cf0 2023-10-30T12:59:34.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.094+0000 7f8ae553e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae0159880 con 0x7f8ae00cf4d0 2023-10-30T12:59:34.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.094+0000 7f8ae553e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae0159ae0 con 0x7f8ae00c90d0 2023-10-30T12:59:34.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.094+0000 7f8ade7fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8ae00c90d0 0x7f8ae00cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:34.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.094+0000 7f8adeffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8ae00c8cf0 0x7f8ae00c9e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37562/0 (socket says 172.21.15.125:37562) 2023-10-30T12:59:34.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.094+0000 7f8adeffd640 1 -- 172.21.15.125:0/3430246732 learned_addr learned my addr 172.21.15.125:0/3430246732 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:59:34.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.094+0000 7f8ade7fc640 1 -- 172.21.15.125:0/3430246732 >> v1:172.21.15.136:6789/0 conn(0x7f8ae00cf4d0 legacy=0x7f8ae015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:34.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.094+0000 7f8ade7fc640 1 -- 172.21.15.125:0/3430246732 >> v1:172.21.15.125:6789/0 conn(0x7f8ae00c8cf0 legacy=0x7f8ae00c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:34.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.094+0000 7f8ade7fc640 1 -- 172.21.15.125:0/3430246732 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae0159dc0 con 0x7f8ae00c90d0 2023-10-30T12:59:34.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.094+0000 7f8addffb640 1 -- 172.21.15.125:0/3430246732 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426398596 0 0) 0x7f8ae015ce30 con 0x7f8ae00c8cf0 2023-10-30T12:59:34.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.098+0000 7f8ade7fc640 1 --2- 172.21.15.125:0/3430246732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ae00c90d0 0x7f8ae00cef90 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f8ad40097c0 tx=0x7f8ad401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0eeb522cbb407cc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:34.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.098+0000 7f8addffb640 1 -- 172.21.15.125:0/3430246732 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad40ad660 con 0x7f8ae00c90d0 2023-10-30T12:59:34.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.098+0000 7f8addffb640 1 -- 172.21.15.125:0/3430246732 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad40ad7c0 con 0x7f8ae00c90d0 2023-10-30T12:59:34.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.098+0000 7f8addffb640 1 -- 172.21.15.125:0/3430246732 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad40adb50 con 0x7f8ae00c90d0 2023-10-30T12:59:34.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.098+0000 7f8ae553e640 1 -- 172.21.15.125:0/3430246732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ae00c90d0 msgr2=0x7f8ae00cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:34.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.098+0000 7f8ae553e640 1 --2- 172.21.15.125:0/3430246732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ae00c90d0 0x7f8ae00cef90 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f8ad40097c0 tx=0x7f8ad401c450 comp rx=0 tx=0).stop 2023-10-30T12:59:34.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.098+0000 7f8ae553e640 1 -- 172.21.15.125:0/3430246732 shutdown_connections 2023-10-30T12:59:34.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.098+0000 7f8ae553e640 1 --2- 172.21.15.125:0/3430246732 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ae00c90d0 0x7f8ae00cef90 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:34.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.098+0000 7f8ae553e640 1 -- 172.21.15.125:0/3430246732 >> 172.21.15.125:0/3430246732 conn(0x7f8ae006be40 msgr2=0x7f8ae006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:34.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.098+0000 7f8ae553e640 1 -- 172.21.15.125:0/3430246732 shutdown_connections 2023-10-30T12:59:34.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.098+0000 7f8ae553e640 1 -- 172.21.15.125:0/3430246732 wait complete. 2023-10-30T12:59:34.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.098+0000 7f8ae553e640 1 Processor -- start 2023-10-30T12:59:34.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.098+0000 7f8ae553e640 1 -- start start 2023-10-30T12:59:34.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ae553e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ae00c8cf0 0x7f8ae01544b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:34.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ae553e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ae00c90d0 0x7f8ae014f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:34.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ae553e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ae00cf4d0 0x7f8ae00b2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:34.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ae553e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae0153d70 con 0x7f8ae00cf4d0 2023-10-30T12:59:34.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ae553e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae00b3df0 con 0x7f8ae00c8cf0 2023-10-30T12:59:34.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ae553e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae01f2550 con 0x7f8ae00c90d0 2023-10-30T12:59:34.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8adf7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ae00cf4d0 0x7f8ae00b2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:34.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8adf7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ae00cf4d0 0x7f8ae00b2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53144/0 (socket says 172.21.15.125:53144) 2023-10-30T12:59:34.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8adf7fe640 1 -- 172.21.15.125:0/110884553 learned_addr learned my addr 172.21.15.125:0/110884553 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:34.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8adeffd640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ae00c8cf0 0x7f8ae01544b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:34.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ade7fc640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ae00c90d0 0x7f8ae014f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:34.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8adf7fe640 1 -- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ae00c90d0 msgr2=0x7f8ae014f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:34.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8adf7fe640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ae00c90d0 0x7f8ae014f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:34.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8adf7fe640 1 -- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ae00c8cf0 msgr2=0x7f8ae01544b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:34.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8adf7fe640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ae00c8cf0 0x7f8ae01544b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:34.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8adf7fe640 1 -- 172.21.15.125:0/110884553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad4009420 con 0x7f8ae00cf4d0 2023-10-30T12:59:34.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ade7fc640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ae00c90d0 0x7f8ae014f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T12:59:34.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8adf7fe640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ae00cf4d0 0x7f8ae00b2c00 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800c7d0 tx=0x7f8ad800cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:34.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ac77fe640 1 -- 172.21.15.125:0/110884553 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad8003b80 con 0x7f8ae00cf4d0 2023-10-30T12:59:34.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ac77fe640 1 -- 172.21.15.125:0/110884553 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad8003ce0 con 0x7f8ae00cf4d0 2023-10-30T12:59:34.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ac77fe640 1 -- 172.21.15.125:0/110884553 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad800aa90 con 0x7f8ae00cf4d0 2023-10-30T12:59:34.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ae553e640 1 -- 172.21.15.125:0/110884553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae01f2a80 con 0x7f8ae00cf4d0 2023-10-30T12:59:34.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ae553e640 1 -- 172.21.15.125:0/110884553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae0151530 con 0x7f8ae00cf4d0 2023-10-30T12:59:34.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ae553e640 1 -- 172.21.15.125:0/110884553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae004e170 con 0x7f8ae00cf4d0 2023-10-30T12:59:34.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.102+0000 7f8ac77fe640 1 -- 172.21.15.125:0/110884553 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8ad800abf0 con 0x7f8ae00cf4d0 2023-10-30T12:59:34.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.106+0000 7f8ac77fe640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8acc0664d0 0x7f8acc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:34.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.106+0000 7f8ac77fe640 1 -- 172.21.15.125:0/110884553 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f8ad8084250 con 0x7f8ae00cf4d0 2023-10-30T12:59:34.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.106+0000 7f8adeffd640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8acc0664d0 0x7f8acc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:34.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.106+0000 7f8adeffd640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8acc0664d0 0x7f8acc068990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0000f70 tx=0x7f8ad0007d10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:34.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.110+0000 7f8ac77fe640 1 -- 172.21.15.125:0/110884553 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8ad8051140 con 0x7f8ae00cf4d0 2023-10-30T12:59:34.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.290+0000 7f8ae553e640 1 -- 172.21.15.125:0/110884553 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ae00ca9d0 con 0x7f8ae00cf4d0 2023-10-30T12:59:34.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.290+0000 7f8ac77fe640 1 -- 172.21.15.125:0/110884553 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f8ad8054760 con 0x7f8ae00cf4d0 2023-10-30T12:59:34.293 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:34.293 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:31.749139+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:34.295 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 142 2023-10-30T12:59:34.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.294+0000 7f8ae553e640 1 -- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8acc0664d0 msgr2=0x7f8acc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:34.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.294+0000 7f8ae553e640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8acc0664d0 0x7f8acc068990 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0000f70 tx=0x7f8ad0007d10 comp rx=0 tx=0).stop 2023-10-30T12:59:34.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.294+0000 7f8ae553e640 1 -- 172.21.15.125:0/110884553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ae00cf4d0 msgr2=0x7f8ae00b2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:34.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.294+0000 7f8ae553e640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ae00cf4d0 0x7f8ae00b2c00 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f8ad800c7d0 tx=0x7f8ad800cc90 comp rx=0 tx=0).stop 2023-10-30T12:59:34.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.294+0000 7f8ae553e640 1 -- 172.21.15.125:0/110884553 shutdown_connections 2023-10-30T12:59:34.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.294+0000 7f8ae553e640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8acc0664d0 0x7f8acc068990 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:34.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.294+0000 7f8ae553e640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ae00cf4d0 0x7f8ae00b2c00 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:34.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.294+0000 7f8ae553e640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ae00c90d0 0x7f8ae014f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:34.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.294+0000 7f8ae553e640 1 --2- 172.21.15.125:0/110884553 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ae00c8cf0 0x7f8ae01544b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:34.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.294+0000 7f8ae553e640 1 -- 172.21.15.125:0/110884553 >> 172.21.15.125:0/110884553 conn(0x7f8ae006be40 msgr2=0x7f8ae00c9d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:34.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.294+0000 7f8ae553e640 1 -- 172.21.15.125:0/110884553 shutdown_connections 2023-10-30T12:59:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:34.294+0000 7f8ae553e640 1 -- 172.21.15.125:0/110884553 wait complete. 2023-10-30T12:59:34.313 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:59:36.314 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:36.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.458+0000 7fe8047c4640 1 Processor -- start 2023-10-30T12:59:36.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.458+0000 7fe8047c4640 1 -- start start 2023-10-30T12:59:36.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.458+0000 7fe8047c4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe7fc0c6b80 0x7fe7fc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.458+0000 7fe8047c4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe7fc13f4a0 0x7fe7fc0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe8047c4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7fc0c0a50 con 0x7fe7fc0c6f60 2023-10-30T12:59:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe8047c4640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7fc06f6e0 con 0x7fe7fc0c6b80 2023-10-30T12:59:36.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe8047c4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7fc06f9c0 con 0x7fe7fc13f4a0 2023-10-30T12:59:36.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe802560640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe7fc0c6b80 0x7fe7fc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:36.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe802560640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe7fc0c6b80 0x7fe7fc0bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53152/0 (socket says 172.21.15.125:53152) 2023-10-30T12:59:36.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe802560640 1 -- 172.21.15.125:0/1070448848 learned_addr learned my addr 172.21.15.125:0/1070448848 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:36.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe802d61640 1 --2- 172.21.15.125:0/1070448848 >> v2:172.21.15.136:3300/0 conn(0x7fe7fc13f4a0 0x7fe7fc0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:36.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe802560640 1 -- 172.21.15.125:0/1070448848 >> v2:172.21.15.136:3300/0 conn(0x7fe7fc13f4a0 msgr2=0x7fe7fc0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:36.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe802560640 1 --2- 172.21.15.125:0/1070448848 >> v2:172.21.15.136:3300/0 conn(0x7fe7fc13f4a0 0x7fe7fc0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:36.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe802560640 1 -- 172.21.15.125:0/1070448848 >> v1:172.21.15.136:6789/0 conn(0x7fe7fc0c6f60 legacy=0x7fe7fc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:36.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe802560640 1 -- 172.21.15.125:0/1070448848 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc06fca0 con 0x7fe7fc0c6b80 2023-10-30T12:59:36.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe802560640 1 --2- 172.21.15.125:0/1070448848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7fc0c6b80 0x7fe7fc0bfec0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0c8980 tx=0x7fe7f801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26648ee30e30214c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:36.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe80155e640 1 -- 172.21.15.125:0/1070448848 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f80ab610 con 0x7fe7fc0c6b80 2023-10-30T12:59:36.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe80155e640 1 -- 172.21.15.125:0/1070448848 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f80ab770 con 0x7fe7fc0c6b80 2023-10-30T12:59:36.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.462+0000 7fe80155e640 1 -- 172.21.15.125:0/1070448848 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f80abb00 con 0x7fe7fc0c6b80 2023-10-30T12:59:36.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 -- 172.21.15.125:0/1070448848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7fc0c6b80 msgr2=0x7fe7fc0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:36.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 --2- 172.21.15.125:0/1070448848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7fc0c6b80 0x7fe7fc0bfec0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc0c8980 tx=0x7fe7f801c500 comp rx=0 tx=0).stop 2023-10-30T12:59:36.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 -- 172.21.15.125:0/1070448848 shutdown_connections 2023-10-30T12:59:36.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 --2- 172.21.15.125:0/1070448848 >> v2:172.21.15.136:3300/0 conn(0x7fe7fc13f4a0 0x7fe7fc0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:36.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 --2- 172.21.15.125:0/1070448848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7fc0c6b80 0x7fe7fc0bfec0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:36.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 -- 172.21.15.125:0/1070448848 >> 172.21.15.125:0/1070448848 conn(0x7fe7fc0c7910 msgr2=0x7fe7fc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:36.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 -- 172.21.15.125:0/1070448848 shutdown_connections 2023-10-30T12:59:36.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 -- 172.21.15.125:0/1070448848 wait complete. 2023-10-30T12:59:36.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 Processor -- start 2023-10-30T12:59:36.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 -- start start 2023-10-30T12:59:36.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe7fc0c6b80 0x7fe7fc150350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:36.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7fc0c6f60 0x7fe7fc135440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:36.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7fc13f4a0 0x7fe7fc135980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:36.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc136ca0 con 0x7fe7fc0c6f60 2023-10-30T12:59:36.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc13e7f0 con 0x7fe7fc13f4a0 2023-10-30T12:59:36.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe8047c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc13e230 con 0x7fe7fc0c6b80 2023-10-30T12:59:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe801d5f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7fc0c6f60 0x7fe7fc135440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe801d5f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7fc0c6f60 0x7fe7fc135440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53156/0 (socket says 172.21.15.125:53156) 2023-10-30T12:59:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe801d5f640 1 -- 172.21.15.125:0/231644000 learned_addr learned my addr 172.21.15.125:0/231644000 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe802560640 1 --2- 172.21.15.125:0/231644000 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe7fc0c6b80 0x7fe7fc150350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:36.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe802d61640 1 --2- 172.21.15.125:0/231644000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7fc13f4a0 0x7fe7fc135980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:36.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe801d5f640 1 -- 172.21.15.125:0/231644000 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe7fc0c6b80 msgr2=0x7fe7fc150350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:36.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.466+0000 7fe801d5f640 1 --2- 172.21.15.125:0/231644000 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe7fc0c6b80 0x7fe7fc150350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:36.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.470+0000 7fe801d5f640 1 -- 172.21.15.125:0/231644000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7fc13f4a0 msgr2=0x7fe7fc135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:36.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.470+0000 7fe801d5f640 1 --2- 172.21.15.125:0/231644000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7fc13f4a0 0x7fe7fc135980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:36.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.470+0000 7fe801d5f640 1 -- 172.21.15.125:0/231644000 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f80094d0 con 0x7fe7fc0c6f60 2023-10-30T12:59:36.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.470+0000 7fe801d5f640 1 --2- 172.21.15.125:0/231644000 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7fc0c6f60 0x7fe7fc135440 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00c530 tx=0x7fe7ec00c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:36.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.470+0000 7fe7f2ffd640 1 -- 172.21.15.125:0/231644000 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec005220 con 0x7fe7fc0c6f60 2023-10-30T12:59:36.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.470+0000 7fe7f2ffd640 1 -- 172.21.15.125:0/231644000 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7ec005380 con 0x7fe7fc0c6f60 2023-10-30T12:59:36.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.470+0000 7fe7f2ffd640 1 -- 172.21.15.125:0/231644000 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7ec007270 con 0x7fe7fc0c6f60 2023-10-30T12:59:36.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.470+0000 7fe8047c4640 1 -- 172.21.15.125:0/231644000 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc1561e0 con 0x7fe7fc0c6f60 2023-10-30T12:59:36.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.470+0000 7fe8047c4640 1 -- 172.21.15.125:0/231644000 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc13e480 con 0x7fe7fc0c6f60 2023-10-30T12:59:36.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.470+0000 7fe8047c4640 1 -- 172.21.15.125:0/231644000 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc13e640 con 0x7fe7fc0c6f60 2023-10-30T12:59:36.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.470+0000 7fe7f2ffd640 1 -- 172.21.15.125:0/231644000 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe7ec0073d0 con 0x7fe7fc0c6f60 2023-10-30T12:59:36.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.470+0000 7fe7f2ffd640 1 --2- 172.21.15.125:0/231644000 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe7e00664a0 0x7fe7e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:36.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.470+0000 7fe7f2ffd640 1 -- 172.21.15.125:0/231644000 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7fe7ec083cb0 con 0x7fe7fc0c6f60 2023-10-30T12:59:36.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.470+0000 7fe802560640 1 --2- 172.21.15.125:0/231644000 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe7e00664a0 0x7fe7e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:36.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.474+0000 7fe802560640 1 --2- 172.21.15.125:0/231644000 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe7e00664a0 0x7fe7e0068960 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fe7f80af000 tx=0x7fe7f80a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:36.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.474+0000 7fe7f2ffd640 1 -- 172.21.15.125:0/231644000 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe7ec050c40 con 0x7fe7fc0c6f60 2023-10-30T12:59:36.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.658+0000 7fe8047c4640 1 -- 172.21.15.125:0/231644000 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe7fc04e170 con 0x7fe7fc0c6f60 2023-10-30T12:59:36.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.658+0000 7fe7f2ffd640 1 -- 172.21.15.125:0/231644000 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+5724 (secure 0 0 0) 0x7fe7ec054260 con 0x7fe7fc0c6f60 2023-10-30T12:59:36.660 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:36.660 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":142,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:31.749139+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:36.662 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 142 2023-10-30T12:59:36.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.658+0000 7fe8047c4640 1 -- 172.21.15.125:0/231644000 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe7e00664a0 msgr2=0x7fe7e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:36.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.658+0000 7fe8047c4640 1 --2- 172.21.15.125:0/231644000 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe7e00664a0 0x7fe7e0068960 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fe7f80af000 tx=0x7fe7f80a7000 comp rx=0 tx=0).stop 2023-10-30T12:59:36.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.658+0000 7fe8047c4640 1 -- 172.21.15.125:0/231644000 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7fc0c6f60 msgr2=0x7fe7fc135440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:36.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.658+0000 7fe8047c4640 1 --2- 172.21.15.125:0/231644000 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7fc0c6f60 0x7fe7fc135440 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fe7ec00c530 tx=0x7fe7ec00c9f0 comp rx=0 tx=0).stop 2023-10-30T12:59:36.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.658+0000 7fe8047c4640 1 -- 172.21.15.125:0/231644000 shutdown_connections 2023-10-30T12:59:36.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.658+0000 7fe8047c4640 1 --2- 172.21.15.125:0/231644000 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe7e00664a0 0x7fe7e0068960 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:36.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.658+0000 7fe8047c4640 1 --2- 172.21.15.125:0/231644000 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe7fc13f4a0 0x7fe7fc135980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:36.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.658+0000 7fe8047c4640 1 --2- 172.21.15.125:0/231644000 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe7fc0c6f60 0x7fe7fc135440 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:36.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.658+0000 7fe8047c4640 1 --2- 172.21.15.125:0/231644000 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe7fc0c6b80 0x7fe7fc150350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:36.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.658+0000 7fe8047c4640 1 -- 172.21.15.125:0/231644000 >> 172.21.15.125:0/231644000 conn(0x7fe7fc0c7910 msgr2=0x7fe7fc0c13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:36.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.658+0000 7fe8047c4640 1 -- 172.21.15.125:0/231644000 shutdown_connections 2023-10-30T12:59:36.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:36.658+0000 7fe8047c4640 1 -- 172.21.15.125:0/231644000 wait complete. 2023-10-30T12:59:36.680 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:59:38.682 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:38.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.818+0000 7f4440846640 1 Processor -- start 2023-10-30T12:59:38.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.818+0000 7f4440846640 1 -- start start 2023-10-30T12:59:38.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.822+0000 7f4440846640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f443c0c05d0 0x7f443c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.822+0000 7f4440846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f443c0c6f60 0x7f443c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.822+0000 7f4440846640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f443c0c0a50 con 0x7f443c0c6b80 2023-10-30T12:59:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.822+0000 7f4440846640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f443c06f6e0 con 0x7f443c0c05d0 2023-10-30T12:59:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.822+0000 7f4440846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443c06f9c0 con 0x7f443c0c6f60 2023-10-30T12:59:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.822+0000 7f443ad9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f443c0c6f60 0x7f443c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:38.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.822+0000 7f443ad9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f443c0c6f60 0x7f443c13f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:42048/0 (socket says 172.21.15.125:42048) 2023-10-30T12:59:38.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.822+0000 7f443ad9d640 1 -- 172.21.15.125:0/3908752420 learned_addr learned my addr 172.21.15.125:0/3908752420 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:38.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.822+0000 7f4433fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f443c0c05d0 0x7f443c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:38.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.822+0000 7f443ad9d640 1 -- 172.21.15.125:0/3908752420 >> v2:172.21.15.136:3300/0 conn(0x7f443c0c05d0 msgr2=0x7f443c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:38.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.822+0000 7f443ad9d640 1 --2- 172.21.15.125:0/3908752420 >> v2:172.21.15.136:3300/0 conn(0x7f443c0c05d0 0x7f443c13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:38.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.822+0000 7f443ad9d640 1 -- 172.21.15.125:0/3908752420 >> v1:172.21.15.136:6789/0 conn(0x7f443c0c6b80 legacy=0x7f443c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:38.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.822+0000 7f443ad9d640 1 -- 172.21.15.125:0/3908752420 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f443c06fca0 con 0x7f443c0c6f60 2023-10-30T12:59:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.826+0000 7f443ad9d640 1 --2- 172.21.15.125:0/3908752420 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f443c0c6f60 0x7f443c13f2d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4434009e50 tx=0x7f443401ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ec70a625fda1b41 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.826+0000 7f4439d9b640 1 -- 172.21.15.125:0/3908752420 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44340a57f0 con 0x7f443c0c6f60 2023-10-30T12:59:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.826+0000 7f4439d9b640 1 -- 172.21.15.125:0/3908752420 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44340a5950 con 0x7f443c0c6f60 2023-10-30T12:59:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.826+0000 7f4439d9b640 1 -- 172.21.15.125:0/3908752420 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44340a5ce0 con 0x7f443c0c6f60 2023-10-30T12:59:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.826+0000 7f4440846640 1 -- 172.21.15.125:0/3908752420 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f443c0c6f60 msgr2=0x7f443c13f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.826+0000 7f4440846640 1 --2- 172.21.15.125:0/3908752420 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f443c0c6f60 0x7f443c13f2d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4434009e50 tx=0x7f443401ca10 comp rx=0 tx=0).stop 2023-10-30T12:59:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.826+0000 7f4440846640 1 -- 172.21.15.125:0/3908752420 shutdown_connections 2023-10-30T12:59:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.826+0000 7f4440846640 1 --2- 172.21.15.125:0/3908752420 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f443c0c6f60 0x7f443c13f2d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:38.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.826+0000 7f4440846640 1 --2- 172.21.15.125:0/3908752420 >> v2:172.21.15.136:3300/0 conn(0x7f443c0c05d0 0x7f443c13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:38.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.826+0000 7f4440846640 1 -- 172.21.15.125:0/3908752420 >> 172.21.15.125:0/3908752420 conn(0x7f443c0c7910 msgr2=0x7f443c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:38.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.826+0000 7f4440846640 1 -- 172.21.15.125:0/3908752420 shutdown_connections 2023-10-30T12:59:38.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.826+0000 7f4440846640 1 -- 172.21.15.125:0/3908752420 wait complete. 2023-10-30T12:59:38.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.826+0000 7f4440846640 1 Processor -- start 2023-10-30T12:59:38.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.826+0000 7f4440846640 1 -- start start 2023-10-30T12:59:38.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f4440846640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f443c0c05d0 0x7f443c151300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:38.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f4440846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f443c0c6b80 0x7f443c1521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:38.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f4440846640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f443c0c6f60 0x7f443c1f3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:38.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f4440846640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c1f3750 con 0x7f443c0c05d0 2023-10-30T12:59:38.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f4440846640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443c1f33b0 con 0x7f443c0c6f60 2023-10-30T12:59:38.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f4440846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443c138640 con 0x7f443c0c6b80 2023-10-30T12:59:38.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f443a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f443c0c05d0 0x7f443c151300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:38.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f443a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f443c0c05d0 0x7f443c151300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53162/0 (socket says 172.21.15.125:53162) 2023-10-30T12:59:38.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f443a59c640 1 -- 172.21.15.125:0/3446777162 learned_addr learned my addr 172.21.15.125:0/3446777162 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:38.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f443ad9d640 1 --2- 172.21.15.125:0/3446777162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f443c0c6f60 0x7f443c1f3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:38.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f4433fff640 1 --2- 172.21.15.125:0/3446777162 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f443c0c6b80 0x7f443c1521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:38.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f443a59c640 1 -- 172.21.15.125:0/3446777162 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f443c0c6b80 msgr2=0x7f443c1521e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:38.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f443a59c640 1 --2- 172.21.15.125:0/3446777162 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f443c0c6b80 0x7f443c1521e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:38.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f443a59c640 1 -- 172.21.15.125:0/3446777162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f443c0c6f60 msgr2=0x7f443c1f3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:38.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f443a59c640 1 --2- 172.21.15.125:0/3446777162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f443c0c6f60 0x7f443c1f3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:38.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f443a59c640 1 -- 172.21.15.125:0/3446777162 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4434009b30 con 0x7f443c0c05d0 2023-10-30T12:59:38.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f443a59c640 1 --2- 172.21.15.125:0/3446777162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f443c0c05d0 0x7f443c151300 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f442c00c500 tx=0x7f442c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:38.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f4432ffd640 1 -- 172.21.15.125:0/3446777162 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c005220 con 0x7f443c0c05d0 2023-10-30T12:59:38.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f4432ffd640 1 -- 172.21.15.125:0/3446777162 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442c005380 con 0x7f443c0c05d0 2023-10-30T12:59:38.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f4432ffd640 1 -- 172.21.15.125:0/3446777162 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c007270 con 0x7f443c0c05d0 2023-10-30T12:59:38.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f4440846640 1 -- 172.21.15.125:0/3446777162 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443c138360 con 0x7f443c0c05d0 2023-10-30T12:59:38.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f4440846640 1 -- 172.21.15.125:0/3446777162 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443c0c02b0 con 0x7f443c0c05d0 2023-10-30T12:59:38.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f4440846640 1 -- 172.21.15.125:0/3446777162 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f443c04e170 con 0x7f443c0c05d0 2023-10-30T12:59:38.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.830+0000 7f4432ffd640 1 -- 172.21.15.125:0/3446777162 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f442c0073d0 con 0x7f443c0c05d0 2023-10-30T12:59:38.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.834+0000 7f4432ffd640 1 --2- 172.21.15.125:0/3446777162 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f441c0664a0 0x7f441c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:38.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.834+0000 7f4432ffd640 1 -- 172.21.15.125:0/3446777162 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f442c083bd0 con 0x7f443c0c05d0 2023-10-30T12:59:38.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.834+0000 7f4433fff640 1 --2- 172.21.15.125:0/3446777162 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f441c0664a0 0x7f441c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:38.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.834+0000 7f4433fff640 1 --2- 172.21.15.125:0/3446777162 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f441c0664a0 0x7f441c068960 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f443c13e910 tx=0x7f4424009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:38.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:38.838+0000 7f4432ffd640 1 -- 172.21.15.125:0/3446777162 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f442c050b30 con 0x7f443c0c05d0 2023-10-30T12:59:39.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:39.014+0000 7f4440846640 1 -- 172.21.15.125:0/3446777162 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f443c137190 con 0x7f443c0c05d0 2023-10-30T12:59:39.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:39.018+0000 7f4432ffd640 1 -- 172.21.15.125:0/3446777162 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f442c054150 con 0x7f443c0c05d0 2023-10-30T12:59:39.019 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:39.019 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:38.170580+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:39.019 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 143 2023-10-30T12:59:39.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:39.018+0000 7f4440846640 1 -- 172.21.15.125:0/3446777162 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f441c0664a0 msgr2=0x7f441c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:39.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:39.018+0000 7f4440846640 1 --2- 172.21.15.125:0/3446777162 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f441c0664a0 0x7f441c068960 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f443c13e910 tx=0x7f4424009190 comp rx=0 tx=0).stop 2023-10-30T12:59:39.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:39.018+0000 7f4440846640 1 -- 172.21.15.125:0/3446777162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f443c0c05d0 msgr2=0x7f443c151300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:39.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:39.018+0000 7f4440846640 1 --2- 172.21.15.125:0/3446777162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f443c0c05d0 0x7f443c151300 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f442c00c500 tx=0x7f442c00c9c0 comp rx=0 tx=0).stop 2023-10-30T12:59:39.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:39.018+0000 7f4440846640 1 -- 172.21.15.125:0/3446777162 shutdown_connections 2023-10-30T12:59:39.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:39.018+0000 7f4440846640 1 --2- 172.21.15.125:0/3446777162 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f441c0664a0 0x7f441c068960 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:39.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:39.018+0000 7f4440846640 1 --2- 172.21.15.125:0/3446777162 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f443c0c6f60 0x7f443c1f3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:39.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:39.018+0000 7f4440846640 1 --2- 172.21.15.125:0/3446777162 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f443c0c6b80 0x7f443c1521e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:39.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:39.018+0000 7f4440846640 1 --2- 172.21.15.125:0/3446777162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f443c0c05d0 0x7f443c151300 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:39.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:39.018+0000 7f4440846640 1 -- 172.21.15.125:0/3446777162 >> 172.21.15.125:0/3446777162 conn(0x7f443c0c7910 msgr2=0x7f443c1532b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:39.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:39.018+0000 7f4440846640 1 -- 172.21.15.125:0/3446777162 shutdown_connections 2023-10-30T12:59:39.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:39.018+0000 7f4440846640 1 -- 172.21.15.125:0/3446777162 wait complete. 2023-10-30T12:59:39.039 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:59:41.040 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:41.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.182+0000 7f24ed780640 1 Processor -- start 2023-10-30T12:59:41.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24ed780640 1 -- start start 2023-10-30T12:59:41.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24ed780640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f24e806f7a0 0x7f24e806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:41.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24ed780640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e8066cf0 con 0x7f24e80c6bd0 2023-10-30T12:59:41.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24ed780640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e80c70d0 con 0x7f24e8071110 2023-10-30T12:59:41.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24ed780640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e80700c0 con 0x7f24e806f7a0 2023-10-30T12:59:41.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24de7fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f24e80c6bd0 0x7f24e8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58350/0 (socket says 172.21.15.125:58350) 2023-10-30T12:59:41.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24de7fc640 1 -- 172.21.15.125:0/1203596163 learned_addr learned my addr 172.21.15.125:0/1203596163 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:59:41.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e6ffd640 1 --2- 172.21.15.125:0/1203596163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f24e806f7a0 0x7f24e806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:41.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 -- 172.21.15.125:0/1203596163 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3189683181 0 0) 0x7f24e8066cf0 con 0x7f24e80c6bd0 2023-10-30T12:59:41.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 -- 172.21.15.125:0/1203596163 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24d4003530 con 0x7f24e80c6bd0 2023-10-30T12:59:41.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 -- 172.21.15.125:0/1203596163 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3851337460 0 0) 0x7f24d4003530 con 0x7f24e80c6bd0 2023-10-30T12:59:41.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 -- 172.21.15.125:0/1203596163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f24e806f7a0 msgr2=0x7f24e806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:41.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 --2- 172.21.15.125:0/1203596163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f24e806f7a0 0x7f24e806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:41.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 -- 172.21.15.125:0/1203596163 >> v1:172.21.15.136:6789/0 conn(0x7f24e8071110 legacy=0x7f24e8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:41.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 -- 172.21.15.125:0/1203596163 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e80c7390 con 0x7f24e80c6bd0 2023-10-30T12:59:41.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 -- 172.21.15.125:0/1203596163 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f24d8001680 con 0x7f24e80c6bd0 2023-10-30T12:59:41.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 -- 172.21.15.125:0/1203596163 >> v1:172.21.15.125:6789/0 conn(0x7f24e80c6bd0 legacy=0x7f24e8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:41.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 --2- 172.21.15.125:0/1203596163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f24e806f7a0 0x7f24d40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:41.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 --2- 172.21.15.125:0/1203596163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24d4003ef0 0x7f24d40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:41.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 --2- 172.21.15.125:0/1203596163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24d4004d10 0x7f24d4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:41.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 -- 172.21.15.125:0/1203596163 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e80700c0 con 0x7f24d4003ef0 2023-10-30T12:59:41.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 -- 172.21.15.125:0/1203596163 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d400b9f0 con 0x7f24d4004d10 2023-10-30T12:59:41.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e67fc640 1 -- 172.21.15.125:0/1203596163 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24d400bbb0 con 0x7f24e806f7a0 2023-10-30T12:59:41.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24de7fc640 1 --2- 172.21.15.125:0/1203596163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24d4003ef0 0x7f24d40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:41.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e6ffd640 1 --2- 172.21.15.125:0/1203596163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f24e806f7a0 0x7f24d40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:41.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24e77fe640 1 --2- 172.21.15.125:0/1203596163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24d4004d10 0x7f24d4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:41.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24de7fc640 1 -- 172.21.15.125:0/1203596163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f24e806f7a0 msgr2=0x7f24d40039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:41.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24de7fc640 1 --2- 172.21.15.125:0/1203596163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f24e806f7a0 0x7f24d40039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:41.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24de7fc640 1 -- 172.21.15.125:0/1203596163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24d4004d10 msgr2=0x7f24d4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:41.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24de7fc640 1 --2- 172.21.15.125:0/1203596163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24d4004d10 0x7f24d4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:41.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.186+0000 7f24de7fc640 1 -- 172.21.15.125:0/1203596163 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d4003530 con 0x7f24d4003ef0 2023-10-30T12:59:41.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.190+0000 7f24de7fc640 1 --2- 172.21.15.125:0/1203596163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24d4003ef0 0x7f24d40047d0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f24d800c190 tx=0x7f24d801f250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac7980e20842c79 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:41.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.190+0000 7f24e67fc640 1 -- 172.21.15.125:0/1203596163 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d80a8e70 con 0x7f24d4003ef0 2023-10-30T12:59:41.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.190+0000 7f24e67fc640 1 -- 172.21.15.125:0/1203596163 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d80058e0 con 0x7f24d4003ef0 2023-10-30T12:59:41.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.190+0000 7f24e67fc640 1 -- 172.21.15.125:0/1203596163 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d8005cf0 con 0x7f24d4003ef0 2023-10-30T12:59:41.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.190+0000 7f24ed780640 1 -- 172.21.15.125:0/1203596163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24d4003ef0 msgr2=0x7f24d40047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:41.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.190+0000 7f24ed780640 1 --2- 172.21.15.125:0/1203596163 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24d4003ef0 0x7f24d40047d0 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f24d800c190 tx=0x7f24d801f250 comp rx=0 tx=0).stop 2023-10-30T12:59:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.190+0000 7f24e77fe640 1 -- 172.21.15.125:0/1203596163 reap_dead start 2023-10-30T12:59:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.190+0000 7f24ed780640 1 -- 172.21.15.125:0/1203596163 shutdown_connections 2023-10-30T12:59:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.190+0000 7f24ed780640 1 -- 172.21.15.125:0/1203596163 >> 172.21.15.125:0/1203596163 conn(0x7f24e80c7910 msgr2=0x7f24e80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.190+0000 7f24ed780640 1 -- 172.21.15.125:0/1203596163 shutdown_connections 2023-10-30T12:59:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.190+0000 7f24ed780640 1 -- 172.21.15.125:0/1203596163 wait complete. 2023-10-30T12:59:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24ed780640 1 Processor -- start 2023-10-30T12:59:41.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24ed780640 1 -- start start 2023-10-30T12:59:41.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24ed780640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f24d4003ef0 0x7f24e814cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:41.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24ed780640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24e81f23f0 0x7f24e80c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:41.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24ed780640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24e81f2870 0x7f24e80c3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:41.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24ed780640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e815a810 con 0x7f24e81f2870 2023-10-30T12:59:41.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24ed780640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e80c17f0 con 0x7f24e81f23f0 2023-10-30T12:59:41.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24ed780640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e815b540 con 0x7f24d4003ef0 2023-10-30T12:59:41.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24e77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24e81f2870 0x7f24e80c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:41.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24e77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24e81f2870 0x7f24e80c3890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53008/0 (socket says 172.21.15.125:53008) 2023-10-30T12:59:41.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24e77fe640 1 -- 172.21.15.125:0/2056959107 learned_addr learned my addr 172.21.15.125:0/2056959107 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:41.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24de7fc640 1 --2- 172.21.15.125:0/2056959107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24e81f23f0 0x7f24e80c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:41.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24e6ffd640 1 --2- 172.21.15.125:0/2056959107 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f24d4003ef0 0x7f24e814cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:41.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24e77fe640 1 -- 172.21.15.125:0/2056959107 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f24d4003ef0 msgr2=0x7f24e814cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:41.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24e77fe640 1 --2- 172.21.15.125:0/2056959107 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f24d4003ef0 0x7f24e814cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:41.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24e77fe640 1 -- 172.21.15.125:0/2056959107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24e81f23f0 msgr2=0x7f24e80c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:41.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24e77fe640 1 --2- 172.21.15.125:0/2056959107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24e81f23f0 0x7f24e80c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:41.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24e77fe640 1 -- 172.21.15.125:0/2056959107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e80c1ab0 con 0x7f24e81f2870 2023-10-30T12:59:41.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24e77fe640 1 --2- 172.21.15.125:0/2056959107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24e81f2870 0x7f24e80c3890 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f24e806f1c0 tx=0x7f24e000eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:41.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24dffff640 1 -- 172.21.15.125:0/2056959107 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e0003980 con 0x7f24e81f2870 2023-10-30T12:59:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24dffff640 1 -- 172.21.15.125:0/2056959107 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e0003b00 con 0x7f24e81f2870 2023-10-30T12:59:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24ed780640 1 -- 172.21.15.125:0/2056959107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e80c1d90 con 0x7f24e81f2870 2023-10-30T12:59:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24dffff640 1 -- 172.21.15.125:0/2056959107 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e00050d0 con 0x7f24e81f2870 2023-10-30T12:59:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.194+0000 7f24ed780640 1 -- 172.21.15.125:0/2056959107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e81f1ed0 con 0x7f24e81f2870 2023-10-30T12:59:41.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.198+0000 7f24ed780640 1 -- 172.21.15.125:0/2056959107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e804e170 con 0x7f24e81f2870 2023-10-30T12:59:41.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.198+0000 7f24dffff640 1 -- 172.21.15.125:0/2056959107 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f24e0005230 con 0x7f24e81f2870 2023-10-30T12:59:41.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.198+0000 7f24dffff640 1 --2- 172.21.15.125:0/2056959107 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f24c80664d0 0x7f24c8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:41.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.198+0000 7f24dffff640 1 -- 172.21.15.125:0/2056959107 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f24e0084190 con 0x7f24e81f2870 2023-10-30T12:59:41.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.198+0000 7f24e6ffd640 1 --2- 172.21.15.125:0/2056959107 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f24c80664d0 0x7f24c8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:41.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.198+0000 7f24e6ffd640 1 --2- 172.21.15.125:0/2056959107 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f24c80664d0 0x7f24c8068990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f24d0009b20 tx=0x7f24d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:41.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.202+0000 7f24dffff640 1 -- 172.21.15.125:0/2056959107 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f24e0051090 con 0x7f24e81f2870 2023-10-30T12:59:41.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.386+0000 7f24ed780640 1 -- 172.21.15.125:0/2056959107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24e80c7f70 con 0x7f24e81f2870 2023-10-30T12:59:41.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.386+0000 7f24dffff640 1 -- 172.21.15.125:0/2056959107 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f24e00546b0 con 0x7f24e81f2870 2023-10-30T12:59:41.389 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:41.389 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:38.170580+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:41.390 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 143 2023-10-30T12:59:41.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.390+0000 7f24ed780640 1 -- 172.21.15.125:0/2056959107 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f24c80664d0 msgr2=0x7f24c8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:41.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.390+0000 7f24ed780640 1 --2- 172.21.15.125:0/2056959107 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f24c80664d0 0x7f24c8068990 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f24d0009b20 tx=0x7f24d0008040 comp rx=0 tx=0).stop 2023-10-30T12:59:41.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.390+0000 7f24ed780640 1 -- 172.21.15.125:0/2056959107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24e81f2870 msgr2=0x7f24e80c3890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:41.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.390+0000 7f24ed780640 1 --2- 172.21.15.125:0/2056959107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24e81f2870 0x7f24e80c3890 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f24e806f1c0 tx=0x7f24e000eb60 comp rx=0 tx=0).stop 2023-10-30T12:59:41.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.390+0000 7f24ed780640 1 -- 172.21.15.125:0/2056959107 shutdown_connections 2023-10-30T12:59:41.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.390+0000 7f24ed780640 1 --2- 172.21.15.125:0/2056959107 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f24c80664d0 0x7f24c8068990 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:41.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.390+0000 7f24ed780640 1 --2- 172.21.15.125:0/2056959107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f24e81f2870 0x7f24e80c3890 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:41.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.390+0000 7f24ed780640 1 --2- 172.21.15.125:0/2056959107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f24e81f23f0 0x7f24e80c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:41.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.390+0000 7f24ed780640 1 --2- 172.21.15.125:0/2056959107 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f24d4003ef0 0x7f24e814cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:41.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.390+0000 7f24ed780640 1 -- 172.21.15.125:0/2056959107 >> 172.21.15.125:0/2056959107 conn(0x7f24e80c7910 msgr2=0x7f24e80c62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:41.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.390+0000 7f24ed780640 1 -- 172.21.15.125:0/2056959107 shutdown_connections 2023-10-30T12:59:41.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:41.390+0000 7f24ed780640 1 -- 172.21.15.125:0/2056959107 wait complete. 2023-10-30T12:59:41.409 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 143, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 143, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7554": { "addr": "172.21.15.125:6836/1944683138", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1944683138, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1944683138, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7554, "incarnation": 140, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:clientreplay", "state_seq": 22 } }, "last_failure": 0, "last_failure_osd_epoch": 70, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T12:59:38.170580+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7554 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T12:59:43.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:43.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d10e9e640 1 Processor -- start 2023-10-30T12:59:43.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d10e9e640 1 -- start start 2023-10-30T12:59:43.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d10e9e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d0c1538a0 0x7f7d0c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:43.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d10e9e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7d0c0ca0e0 0x7f7d0c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:43.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d10e9e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0c0cb800 con 0x7f7d0c0cf160 2023-10-30T12:59:43.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d10e9e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d0c15dc30 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d10e9e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c0cb330 con 0x7f7d0c1538a0 2023-10-30T12:59:43.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d0ad9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7d0c0cf160 0x7f7d0c0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58356/0 (socket says 172.21.15.125:58356) 2023-10-30T12:59:43.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d0ad9d640 1 -- 172.21.15.125:0/3136379923 learned_addr learned my addr 172.21.15.125:0/3136379923 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:59:43.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d09d9b640 1 --2- 172.21.15.125:0/3136379923 >> v2:172.21.15.136:3300/0 conn(0x7f7d0c0ca0e0 0x7f7d0c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:43.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d0a59c640 1 --2- 172.21.15.125:0/3136379923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d0c1538a0 0x7f7d0c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:43.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d0959a640 1 -- 172.21.15.125:0/3136379923 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2618666163 0 0) 0x7f7d0c0cb800 con 0x7f7d0c0cf160 2023-10-30T12:59:43.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d0959a640 1 -- 172.21.15.125:0/3136379923 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf4003530 con 0x7f7d0c0cf160 2023-10-30T12:59:43.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d09d9b640 1 -- 172.21.15.125:0/3136379923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d0c1538a0 msgr2=0x7f7d0c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:43.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d09d9b640 1 --2- 172.21.15.125:0/3136379923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d0c1538a0 0x7f7d0c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:43.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d09d9b640 1 -- 172.21.15.125:0/3136379923 >> v1:172.21.15.125:6789/0 conn(0x7f7d0c0cf160 legacy=0x7f7d0c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:43.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.570+0000 7f7d09d9b640 1 -- 172.21.15.125:0/3136379923 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0c0cad70 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.574+0000 7f7d09d9b640 1 --2- 172.21.15.125:0/3136379923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d0c0ca0e0 0x7f7d0c15a090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7d000097c0 tx=0x7f7d0001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8083c7bc9586e57 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:43.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.574+0000 7f7d0959a640 1 -- 172.21.15.125:0/3136379923 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d000a8970 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.574+0000 7f7d0959a640 1 -- 172.21.15.125:0/3136379923 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d000a8ad0 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.574+0000 7f7d0959a640 1 -- 172.21.15.125:0/3136379923 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d000a8e60 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.578+0000 7f7d10e9e640 1 -- 172.21.15.125:0/3136379923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d0c0ca0e0 msgr2=0x7f7d0c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:43.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.578+0000 7f7d10e9e640 1 --2- 172.21.15.125:0/3136379923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d0c0ca0e0 0x7f7d0c15a090 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f7d000097c0 tx=0x7f7d0001ec70 comp rx=0 tx=0).stop 2023-10-30T12:59:43.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.578+0000 7f7d10e9e640 1 -- 172.21.15.125:0/3136379923 shutdown_connections 2023-10-30T12:59:43.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.578+0000 7f7d10e9e640 1 --2- 172.21.15.125:0/3136379923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d0c0ca0e0 0x7f7d0c15a090 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:43.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.578+0000 7f7d10e9e640 1 --2- 172.21.15.125:0/3136379923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d0c1538a0 0x7f7d0c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:43.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.578+0000 7f7d10e9e640 1 -- 172.21.15.125:0/3136379923 >> 172.21.15.125:0/3136379923 conn(0x7f7d0c06bfe0 msgr2=0x7f7d0c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:43.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.578+0000 7f7d10e9e640 1 -- 172.21.15.125:0/3136379923 shutdown_connections 2023-10-30T12:59:43.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.578+0000 7f7d10e9e640 1 -- 172.21.15.125:0/3136379923 wait complete. 2023-10-30T12:59:43.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.578+0000 7f7d10e9e640 1 Processor -- start 2023-10-30T12:59:43.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.578+0000 7f7d10e9e640 1 -- start start 2023-10-30T12:59:43.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d10e9e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d0c0ca0e0 0x7f7d0c158760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:43.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d10e9e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d0c0cf160 0x7f7d0c142fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:43.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d10e9e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d0c1538a0 0x7f7d0c157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:43.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d10e9e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c1ef8d0 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d10e9e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c15d830 con 0x7f7d0c1538a0 2023-10-30T12:59:43.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d10e9e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0c15dad0 con 0x7f7d0c0cf160 2023-10-30T12:59:43.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d0a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d0c0ca0e0 0x7f7d0c158760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:43.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d0a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d0c0ca0e0 0x7f7d0c158760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53010/0 (socket says 172.21.15.125:53010) 2023-10-30T12:59:43.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d0a59c640 1 -- 172.21.15.125:0/2099449894 learned_addr learned my addr 172.21.15.125:0/2099449894 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:43.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d09d9b640 1 --2- 172.21.15.125:0/2099449894 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d0c0cf160 0x7f7d0c142fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:43.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d0ad9d640 1 --2- 172.21.15.125:0/2099449894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d0c1538a0 0x7f7d0c157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:43.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d0a59c640 1 -- 172.21.15.125:0/2099449894 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d0c0cf160 msgr2=0x7f7d0c142fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:43.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d0a59c640 1 --2- 172.21.15.125:0/2099449894 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d0c0cf160 0x7f7d0c142fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:43.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d0a59c640 1 -- 172.21.15.125:0/2099449894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d0c1538a0 msgr2=0x7f7d0c157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:43.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d0a59c640 1 --2- 172.21.15.125:0/2099449894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d0c1538a0 0x7f7d0c157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:43.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d0a59c640 1 -- 172.21.15.125:0/2099449894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d00009420 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d0a59c640 1 --2- 172.21.15.125:0/2099449894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d0c0ca0e0 0x7f7d0c158760 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00e930 tx=0x7f7cfc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:43.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7cfaffd640 1 -- 172.21.15.125:0/2099449894 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc009690 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7cfaffd640 1 -- 172.21.15.125:0/2099449894 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cfc00ca00 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7cfaffd640 1 -- 172.21.15.125:0/2099449894 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc00cd70 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d10e9e640 1 -- 172.21.15.125:0/2099449894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0c1efcf0 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.582+0000 7f7d10e9e640 1 -- 172.21.15.125:0/2099449894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d0c15b3d0 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.586+0000 7f7d10e9e640 1 -- 172.21.15.125:0/2099449894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0c0b2a50 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.586+0000 7f7cfaffd640 1 -- 172.21.15.125:0/2099449894 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7cfc013470 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.586+0000 7f7cfaffd640 1 --2- 172.21.15.125:0/2099449894 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7cec0664a0 0x7f7cec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:43.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.586+0000 7f7cfaffd640 1 -- 172.21.15.125:0/2099449894 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f7cfc0846c0 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.586+0000 7f7d09d9b640 1 --2- 172.21.15.125:0/2099449894 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7cec0664a0 0x7f7cec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:43.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.590+0000 7f7d09d9b640 1 --2- 172.21.15.125:0/2099449894 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7cec0664a0 0x7f7cec068960 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7d000b2000 tx=0x7f7d000aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:43.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.590+0000 7f7cfaffd640 1 -- 172.21.15.125:0/2099449894 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7cfc051690 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.826+0000 7f7d10e9e640 1 -- 172.21.15.125:0/2099449894 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d0c143500 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.830+0000 7f7cfaffd640 1 -- 172.21.15.125:0/2099449894 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f7cfc054cb0 con 0x7f7d0c0ca0e0 2023-10-30T12:59:43.831 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:43.831 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:38.170580+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:clientreplay","state_seq":22,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:43.832 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 143 2023-10-30T12:59:43.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.830+0000 7f7d10e9e640 1 -- 172.21.15.125:0/2099449894 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7cec0664a0 msgr2=0x7f7cec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:43.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.830+0000 7f7d10e9e640 1 --2- 172.21.15.125:0/2099449894 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7cec0664a0 0x7f7cec068960 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7d000b2000 tx=0x7f7d000aa070 comp rx=0 tx=0).stop 2023-10-30T12:59:43.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.830+0000 7f7d10e9e640 1 -- 172.21.15.125:0/2099449894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d0c0ca0e0 msgr2=0x7f7d0c158760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:43.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.830+0000 7f7d10e9e640 1 --2- 172.21.15.125:0/2099449894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d0c0ca0e0 0x7f7d0c158760 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc00e930 tx=0x7f7cfc00edf0 comp rx=0 tx=0).stop 2023-10-30T12:59:43.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.830+0000 7f7d10e9e640 1 -- 172.21.15.125:0/2099449894 shutdown_connections 2023-10-30T12:59:43.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.830+0000 7f7d10e9e640 1 --2- 172.21.15.125:0/2099449894 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7cec0664a0 0x7f7cec068960 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:43.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.830+0000 7f7d10e9e640 1 --2- 172.21.15.125:0/2099449894 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d0c1538a0 0x7f7d0c157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:43.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.830+0000 7f7d10e9e640 1 --2- 172.21.15.125:0/2099449894 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d0c0cf160 0x7f7d0c142fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:43.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.830+0000 7f7d10e9e640 1 --2- 172.21.15.125:0/2099449894 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d0c0ca0e0 0x7f7d0c158760 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:43.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.830+0000 7f7d10e9e640 1 -- 172.21.15.125:0/2099449894 >> 172.21.15.125:0/2099449894 conn(0x7f7d0c06bfe0 msgr2=0x7f7d0c151360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:43.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.830+0000 7f7d10e9e640 1 -- 172.21.15.125:0/2099449894 shutdown_connections 2023-10-30T12:59:43.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:43.830+0000 7f7d10e9e640 1 -- 172.21.15.125:0/2099449894 wait complete. 2023-10-30T12:59:43.858 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T12:59:45.860 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:46.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.001+0000 7ff5cac46640 1 Processor -- start 2023-10-30T12:59:46.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.001+0000 7ff5cac46640 1 -- start start 2023-10-30T12:59:46.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5cac46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5c41538a0 0x7ff5c415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:46.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5cac46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff5c40cf160 0x7ff5c40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:46.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5cac46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5c40cb800 con 0x7ff5c40ca0e0 2023-10-30T12:59:46.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5cac46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5c415dc30 con 0x7ff5c40cf160 2023-10-30T12:59:46.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5cac46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c40cb330 con 0x7ff5c41538a0 2023-10-30T12:59:46.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5c91e3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff5c40cf160 0x7ff5c40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:46.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5bbfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff5c40ca0e0 0x7ff5c415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58360/0 (socket says 172.21.15.125:58360) 2023-10-30T12:59:46.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5bbfff640 1 -- 172.21.15.125:0/3642231718 learned_addr learned my addr 172.21.15.125:0/3642231718 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T12:59:46.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5c89e2640 1 --2- 172.21.15.125:0/3642231718 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5c41538a0 0x7ff5c415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:46.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5c91e3640 1 -- 172.21.15.125:0/3642231718 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5c41538a0 msgr2=0x7ff5c415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:46.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5c91e3640 1 --2- 172.21.15.125:0/3642231718 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5c41538a0 0x7ff5c415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:46.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5c91e3640 1 -- 172.21.15.125:0/3642231718 >> v1:172.21.15.125:6789/0 conn(0x7ff5c40ca0e0 legacy=0x7ff5c415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:46.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5c91e3640 1 -- 172.21.15.125:0/3642231718 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5c40cad70 con 0x7ff5c40cf160 2023-10-30T12:59:46.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5bb7fe640 1 -- 172.21.15.125:0/3642231718 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487266407 0 0) 0x7ff5c40cb800 con 0x7ff5c40ca0e0 2023-10-30T12:59:46.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5c91e3640 1 --2- 172.21.15.125:0/3642231718 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5c40cf160 0x7ff5c40cf590 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009d90 tx=0x7ff5bc01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36326e1df8ede990 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:46.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.005+0000 7ff5bb7fe640 1 -- 172.21.15.125:0/3642231718 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc0af600 con 0x7ff5c40cf160 2023-10-30T12:59:46.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5bb7fe640 1 -- 172.21.15.125:0/3642231718 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5bc0ab050 con 0x7ff5c40cf160 2023-10-30T12:59:46.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5bb7fe640 1 -- 172.21.15.125:0/3642231718 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc0af980 con 0x7ff5c40cf160 2023-10-30T12:59:46.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 -- 172.21.15.125:0/3642231718 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5c40cf160 msgr2=0x7ff5c40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:46.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 --2- 172.21.15.125:0/3642231718 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5c40cf160 0x7ff5c40cf590 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc009d90 tx=0x7ff5bc01f130 comp rx=0 tx=0).stop 2023-10-30T12:59:46.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 -- 172.21.15.125:0/3642231718 shutdown_connections 2023-10-30T12:59:46.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 --2- 172.21.15.125:0/3642231718 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5c40cf160 0x7ff5c40cf590 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:46.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 --2- 172.21.15.125:0/3642231718 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5c41538a0 0x7ff5c415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:46.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 -- 172.21.15.125:0/3642231718 >> 172.21.15.125:0/3642231718 conn(0x7ff5c406bfe0 msgr2=0x7ff5c406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:46.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 -- 172.21.15.125:0/3642231718 shutdown_connections 2023-10-30T12:59:46.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 -- 172.21.15.125:0/3642231718 wait complete. 2023-10-30T12:59:46.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 Processor -- start 2023-10-30T12:59:46.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 -- start start 2023-10-30T12:59:46.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5c40ca0e0 0x7ff5c40b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:46.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5c40cf160 0x7ff5c415bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:46.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5c41538a0 0x7ff5c415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:46.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c415b840 con 0x7ff5c41538a0 2023-10-30T12:59:46.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c415a950 con 0x7ff5c40ca0e0 2023-10-30T12:59:46.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5cac46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5c415ac30 con 0x7ff5c40cf160 2023-10-30T12:59:46.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5c91e3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5c41538a0 0x7ff5c415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:46.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5c91e3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5c41538a0 0x7ff5c415c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53016/0 (socket says 172.21.15.125:53016) 2023-10-30T12:59:46.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5c91e3640 1 -- 172.21.15.125:0/1120006010 learned_addr learned my addr 172.21.15.125:0/1120006010 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:46.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5bbfff640 1 --2- 172.21.15.125:0/1120006010 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5c40cf160 0x7ff5c415bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:46.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.009+0000 7ff5c89e2640 1 --2- 172.21.15.125:0/1120006010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5c40ca0e0 0x7ff5c40b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:46.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.013+0000 7ff5c91e3640 1 -- 172.21.15.125:0/1120006010 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5c40cf160 msgr2=0x7ff5c415bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:46.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.013+0000 7ff5c91e3640 1 --2- 172.21.15.125:0/1120006010 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5c40cf160 0x7ff5c415bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:46.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.013+0000 7ff5c91e3640 1 -- 172.21.15.125:0/1120006010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5c40ca0e0 msgr2=0x7ff5c40b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:46.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.013+0000 7ff5c91e3640 1 --2- 172.21.15.125:0/1120006010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5c40ca0e0 0x7ff5c40b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:46.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.013+0000 7ff5c91e3640 1 -- 172.21.15.125:0/1120006010 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5bc009a70 con 0x7ff5c41538a0 2023-10-30T12:59:46.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.013+0000 7ff5c91e3640 1 --2- 172.21.15.125:0/1120006010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5c41538a0 0x7ff5c415c2a0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0b2040 tx=0x7ff5bc01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:46.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.013+0000 7ff5b97fa640 1 -- 172.21.15.125:0/1120006010 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc009450 con 0x7ff5c41538a0 2023-10-30T12:59:46.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.013+0000 7ff5b97fa640 1 -- 172.21.15.125:0/1120006010 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5bc0b9410 con 0x7ff5c41538a0 2023-10-30T12:59:46.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.013+0000 7ff5cac46640 1 -- 172.21.15.125:0/1120006010 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5c415af10 con 0x7ff5c41538a0 2023-10-30T12:59:46.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.013+0000 7ff5b97fa640 1 -- 172.21.15.125:0/1120006010 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc0b97c0 con 0x7ff5c41538a0 2023-10-30T12:59:46.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.013+0000 7ff5cac46640 1 -- 172.21.15.125:0/1120006010 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5c415b4d0 con 0x7ff5c41538a0 2023-10-30T12:59:46.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.013+0000 7ff5cac46640 1 -- 172.21.15.125:0/1120006010 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5c40b2a50 con 0x7ff5c41538a0 2023-10-30T12:59:46.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.013+0000 7ff5b97fa640 1 -- 172.21.15.125:0/1120006010 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5bc0b6050 con 0x7ff5c41538a0 2023-10-30T12:59:46.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.017+0000 7ff5b97fa640 1 --2- 172.21.15.125:0/1120006010 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5b00664a0 0x7ff5b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:46.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.017+0000 7ff5b97fa640 1 -- 172.21.15.125:0/1120006010 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7ff5bc122a10 con 0x7ff5c41538a0 2023-10-30T12:59:46.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.017+0000 7ff5c89e2640 1 --2- 172.21.15.125:0/1120006010 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5b00664a0 0x7ff5b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:46.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.017+0000 7ff5c89e2640 1 --2- 172.21.15.125:0/1120006010 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5b00664a0 0x7ff5b0068960 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0009a50 tx=0x7ff5c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:46.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.021+0000 7ff5b97fa640 1 -- 172.21.15.125:0/1120006010 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5bc0efa20 con 0x7ff5c41538a0 2023-10-30T12:59:46.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.197+0000 7ff5cac46640 1 -- 172.21.15.125:0/1120006010 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff5c404e170 con 0x7ff5c41538a0 2023-10-30T12:59:46.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.197+0000 7ff5b97fa640 1 -- 172.21.15.125:0/1120006010 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+5724 (secure 0 0 0) 0x7ff5bc0b4090 con 0x7ff5c41538a0 2023-10-30T12:59:46.202 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:46.202 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:44.626686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:active","state_seq":24,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:46.202 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 144 2023-10-30T12:59:46.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.201+0000 7ff5cac46640 1 -- 172.21.15.125:0/1120006010 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5b00664a0 msgr2=0x7ff5b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:46.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.201+0000 7ff5cac46640 1 --2- 172.21.15.125:0/1120006010 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5b00664a0 0x7ff5b0068960 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7ff5c0009a50 tx=0x7ff5c0008040 comp rx=0 tx=0).stop 2023-10-30T12:59:46.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.201+0000 7ff5cac46640 1 -- 172.21.15.125:0/1120006010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5c41538a0 msgr2=0x7ff5c415c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:46.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.201+0000 7ff5cac46640 1 --2- 172.21.15.125:0/1120006010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5c41538a0 0x7ff5c415c2a0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc0b2040 tx=0x7ff5bc01bed0 comp rx=0 tx=0).stop 2023-10-30T12:59:46.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.201+0000 7ff5cac46640 1 -- 172.21.15.125:0/1120006010 shutdown_connections 2023-10-30T12:59:46.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.201+0000 7ff5cac46640 1 --2- 172.21.15.125:0/1120006010 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5b00664a0 0x7ff5b0068960 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:46.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.201+0000 7ff5cac46640 1 --2- 172.21.15.125:0/1120006010 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5c41538a0 0x7ff5c415c2a0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:46.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.201+0000 7ff5cac46640 1 --2- 172.21.15.125:0/1120006010 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5c40cf160 0x7ff5c415bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:46.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.201+0000 7ff5cac46640 1 --2- 172.21.15.125:0/1120006010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5c40ca0e0 0x7ff5c40b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:46.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.201+0000 7ff5cac46640 1 -- 172.21.15.125:0/1120006010 >> 172.21.15.125:0/1120006010 conn(0x7ff5c406bfe0 msgr2=0x7ff5c41523a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:46.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.201+0000 7ff5cac46640 1 -- 172.21.15.125:0/1120006010 shutdown_connections 2023-10-30T12:59:46.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:46.201+0000 7ff5cac46640 1 -- 172.21.15.125:0/1120006010 wait complete. 2023-10-30T12:59:46.222 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=7413 2023-10-30T12:59:46.223 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.c 2023-10-30T12:59:51.307 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:59:51.311 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T12:59:53.102 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 5% (6/105) Updating files: 6% (7/105) Updating files: 7% (8/105) Updating files: 8% (9/105) Updating files: 9% (10/105) Updating files: 10% (11/105) Updating files: 11% (12/105) Updating files: 12% (13/105) Updating files: 13% (14/105) Updating files: 14% (15/105) Updating files: 15% (16/105) Updating files: 16% (17/105) Updating files: 17% (18/105) Updating files: 18% (19/105) Updating files: 19% (20/105) Updating files: 20% (21/105) Updating files: 21% (23/105) Updating files: 22% (24/105) Updating files: 23% (25/105) Updating files: 24% (26/105) Updating files: 25% (27/105) Updating files: 26% (28/105) Updating files: 27% (29/105) Updating files: 28% (30/105) Updating files: 29% (31/105) Updating files: 30% (32/105) Updating files: 31% (33/105) Updating files: 32% (34/105) Updating files: 33% (35/105) Updating files: 34% (36/105) Updating files: 35% (37/105) Updating files: 36% (38/105) Updating files: 37% (39/105) Updating files: 38% (40/105) Updating files: 39% (41/105) Updating files: 40% (42/105) Updating files: 41% (44/105) Updating files: 42% (45/105) Updating files: 43% (46/105) Updating files: 44% (47/105) Updating files: 45% (48/105) Updating files: 46% (49/105) Updating files: 47% (50/105) Updating files: 48% (51/105) Updating files: 49% (52/105) Updating files: 50% (53/105) Updating files: 51% (54/105) Updating files: 52% (55/105) Updating files: 53% (56/105) Updating files: 54% (57/105) Updating files: 55% (58/105) Updating files: 56% (59/105) Updating files: 57% (60/105) Updating files: 58% (61/105) Updating files: 59% (62/105) Updating files: 60% (63/105) Updating files: 61% (65/105) Updating files: 62% (66/105) Updating files: 63% (67/105) Updating files: 64% (68/105) Updating files: 65% (69/105) Updating files: 66% (70/105) Updating files: 67% (71/105) Updating files: 68% (72/105) Updating files: 69% (73/105) Updating files: 70% (74/105) Updating files: 71% (75/105) Updating files: 72% (76/105) Updating files: 73% (77/105) Updating files: 74% (78/105) Updating files: 75% (79/105) Updating files: 76% (80/105) Updating files: 77% (81/105) Updating files: 78% (82/105) Updating files: 79% (83/105) Updating files: 80% (84/105) Updating files: 81% (86/105) Updating files: 82% (87/105) Updating files: 83% (88/105) Updating files: 84% (89/105) Updating files: 85% (90/105) Updating files: 86% (91/105) Updating files: 87% (92/105) Updating files: 88% (93/105) Updating files: 89% (94/105) Updating files: 90% (95/105) Updating files: 91% (96/105) Updating files: 92% (97/105) Updating files: 93% (98/105) Updating files: 94% (99/105) Updating files: 95% (100/105) Updating files: 96% (101/105) Updating files: 97% (102/105) Updating files: 98% (103/105) Updating files: 99% (104/105) Updating files: 100% (105/105) Updating files: 100% (105/105), done. 2023-10-30T12:59:53.213 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at c03ca95d235 v0.51 2023-10-30T12:59:53.216 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.51 2023-10-30T12:59:53.223 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T12:59:53.224 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T12:59:53.224 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T12:59:53.227 INFO:tasks.ceph.mds.c:Started 2023-10-30T12:59:53.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T12:59:53.315 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2023-10-30T12:59:53.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.365+0000 7f6a72ac9640 1 Processor -- start 2023-10-30T12:59:53.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.369+0000 7f6a72ac9640 1 -- start start 2023-10-30T12:59:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.369+0000 7f6a72ac9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a6c06f7a0 0x7f6a6c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.369+0000 7f6a72ac9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6a6c0c6bd0 0x7f6a6c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.369+0000 7f6a72ac9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6c06a3d0 con 0x7f6a6c071110 2023-10-30T12:59:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.369+0000 7f6a72ac9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6c0700c0 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.369+0000 7f6a72ac9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c066cf0 con 0x7f6a6c06f7a0 2023-10-30T12:59:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.369+0000 7f6a6bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6a6c0c6bd0 0x7f6a6c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.369+0000 7f6a6bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6a6c0c6bd0 0x7f6a6c069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40092/0 (socket says 172.21.15.125:40092) 2023-10-30T12:59:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.369+0000 7f6a6bfff640 1 -- 172.21.15.125:0/2750336857 learned_addr learned my addr 172.21.15.125:0/2750336857 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:53.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.369+0000 7f6a70865640 1 --2- 172.21.15.125:0/2750336857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a6c06f7a0 0x7f6a6c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:53.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.369+0000 7f6a6bfff640 1 -- 172.21.15.125:0/2750336857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a6c06f7a0 msgr2=0x7f6a6c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T12:59:53.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.369+0000 7f6a6bfff640 1 --2- 172.21.15.125:0/2750336857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a6c06f7a0 0x7f6a6c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:53.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.369+0000 7f6a6bfff640 1 -- 172.21.15.125:0/2750336857 >> v1:172.21.15.136:6789/0 conn(0x7f6a6c071110 legacy=0x7f6a6c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:53.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.369+0000 7f6a6bfff640 1 -- 172.21.15.125:0/2750336857 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c0c70d0 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.373+0000 7f6a6bfff640 1 --2- 172.21.15.125:0/2750336857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a6c0c6bd0 0x7f6a6c069e90 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009870 tx=0x7f6a5801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a4d4894267c5ba7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:53.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.373+0000 7f6a6b7fe640 1 -- 172.21.15.125:0/2750336857 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a580a8a30 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.373+0000 7f6a6b7fe640 1 -- 172.21.15.125:0/2750336857 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a580a8b90 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.373+0000 7f6a6b7fe640 1 -- 172.21.15.125:0/2750336857 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a580b0440 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.373+0000 7f6a72ac9640 1 -- 172.21.15.125:0/2750336857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a6c0c6bd0 msgr2=0x7f6a6c069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:53.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.373+0000 7f6a72ac9640 1 --2- 172.21.15.125:0/2750336857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a6c0c6bd0 0x7f6a6c069e90 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f6a58009870 tx=0x7f6a5801ed30 comp rx=0 tx=0).stop 2023-10-30T12:59:53.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.373+0000 7f6a72ac9640 1 -- 172.21.15.125:0/2750336857 shutdown_connections 2023-10-30T12:59:53.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.373+0000 7f6a72ac9640 1 --2- 172.21.15.125:0/2750336857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a6c0c6bd0 0x7f6a6c069e90 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:53.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.373+0000 7f6a72ac9640 1 --2- 172.21.15.125:0/2750336857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a6c06f7a0 0x7f6a6c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:53.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.373+0000 7f6a72ac9640 1 -- 172.21.15.125:0/2750336857 >> 172.21.15.125:0/2750336857 conn(0x7f6a6c0c7910 msgr2=0x7f6a6c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:53.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.373+0000 7f6a72ac9640 1 -- 172.21.15.125:0/2750336857 shutdown_connections 2023-10-30T12:59:53.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.373+0000 7f6a72ac9640 1 -- 172.21.15.125:0/2750336857 wait complete. 2023-10-30T12:59:53.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.373+0000 7f6a72ac9640 1 Processor -- start 2023-10-30T12:59:53.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.373+0000 7f6a72ac9640 1 -- start start 2023-10-30T12:59:53.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a72ac9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a6c06f7a0 0x7f6a6c14ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:53.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a72ac9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a6c071110 0x7f6a6c0ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:53.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a72ac9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a6c0c6bd0 0x7f6a6c15a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:53.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a72ac9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c1592b0 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a72ac9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c0c9d00 con 0x7f6a6c06f7a0 2023-10-30T12:59:53.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a72ac9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a6c159490 con 0x7f6a6c071110 2023-10-30T12:59:53.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a71066640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a6c0c6bd0 0x7f6a6c15a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:53.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a71066640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a6c0c6bd0 0x7f6a6c15a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40094/0 (socket says 172.21.15.125:40094) 2023-10-30T12:59:53.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a71066640 1 -- 172.21.15.125:0/4152197111 learned_addr learned my addr 172.21.15.125:0/4152197111 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T12:59:53.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a70865640 1 --2- 172.21.15.125:0/4152197111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a6c06f7a0 0x7f6a6c14ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:53.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a6bfff640 1 --2- 172.21.15.125:0/4152197111 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a6c071110 0x7f6a6c0ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:53.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a71066640 1 -- 172.21.15.125:0/4152197111 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a6c071110 msgr2=0x7f6a6c0ca770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:53.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a71066640 1 --2- 172.21.15.125:0/4152197111 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a6c071110 0x7f6a6c0ca770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:53.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a71066640 1 -- 172.21.15.125:0/4152197111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a6c06f7a0 msgr2=0x7f6a6c14ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:53.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a71066640 1 --2- 172.21.15.125:0/4152197111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a6c06f7a0 0x7f6a6c14ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:53.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a71066640 1 -- 172.21.15.125:0/4152197111 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a580094d0 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a71066640 1 --2- 172.21.15.125:0/4152197111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a6c0c6bd0 0x7f6a6c15a230 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400c730 tx=0x7f6a6400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:53.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a697fa640 1 -- 172.21.15.125:0/4152197111 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a64003b80 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a697fa640 1 -- 172.21.15.125:0/4152197111 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a64003ce0 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a72ac9640 1 -- 172.21.15.125:0/4152197111 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6c0c9f70 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a697fa640 1 -- 172.21.15.125:0/4152197111 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a6400aa30 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a72ac9640 1 -- 172.21.15.125:0/4152197111 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6c159770 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a72ac9640 1 -- 172.21.15.125:0/4152197111 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6c0b0e50 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.377+0000 7f6a697fa640 1 -- 172.21.15.125:0/4152197111 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6a6400ab90 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.381+0000 7f6a697fa640 1 --2- 172.21.15.125:0/4152197111 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6a500664a0 0x7f6a50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T12:59:53.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.381+0000 7f6a697fa640 1 -- 172.21.15.125:0/4152197111 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f6a64084080 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.381+0000 7f6a70865640 1 --2- 172.21.15.125:0/4152197111 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6a500664a0 0x7f6a50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T12:59:53.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.381+0000 7f6a70865640 1 --2- 172.21.15.125:0/4152197111 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6a500664a0 0x7f6a50068960 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009ac0 tx=0x7f6a60009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T12:59:53.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.385+0000 7f6a697fa640 1 -- 172.21.15.125:0/4152197111 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6a640b5090 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.439 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 52 -eq 48 ] 2023-10-30T12:59:53.440 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.52 2023-10-30T12:59:53.440 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.52 2023-10-30T12:59:53.440 INFO:tasks.workunit.client.0.smithi125.stdout:v0.52 2023-10-30T12:59:53.441 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.52 2023-10-30T12:59:53.530 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T12:59:53.525+0000 7faf26db4640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T12:59:53.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.577+0000 7f6a72ac9640 1 -- 172.21.15.125:0/4152197111 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a6c04e170 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.577+0000 7f6a697fa640 1 -- 172.21.15.125:0/4152197111 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f6a64051020 con 0x7f6a6c0c6bd0 2023-10-30T12:59:53.581 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T12:59:53.581 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7707,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:44.626686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:active","state_seq":24,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T12:59:53.581 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 145 2023-10-30T12:59:53.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.577+0000 7f6a72ac9640 1 -- 172.21.15.125:0/4152197111 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6a500664a0 msgr2=0x7f6a50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:53.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.577+0000 7f6a72ac9640 1 --2- 172.21.15.125:0/4152197111 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6a500664a0 0x7f6a50068960 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009ac0 tx=0x7f6a60009190 comp rx=0 tx=0).stop 2023-10-30T12:59:53.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.577+0000 7f6a72ac9640 1 -- 172.21.15.125:0/4152197111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a6c0c6bd0 msgr2=0x7f6a6c15a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T12:59:53.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.577+0000 7f6a72ac9640 1 --2- 172.21.15.125:0/4152197111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a6c0c6bd0 0x7f6a6c15a230 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7f6a6400c730 tx=0x7f6a6400cbf0 comp rx=0 tx=0).stop 2023-10-30T12:59:53.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.581+0000 7f6a72ac9640 1 -- 172.21.15.125:0/4152197111 shutdown_connections 2023-10-30T12:59:53.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.581+0000 7f6a72ac9640 1 --2- 172.21.15.125:0/4152197111 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6a500664a0 0x7f6a50068960 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:53.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.581+0000 7f6a72ac9640 1 --2- 172.21.15.125:0/4152197111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a6c0c6bd0 0x7f6a6c15a230 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:53.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.581+0000 7f6a72ac9640 1 --2- 172.21.15.125:0/4152197111 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a6c071110 0x7f6a6c0ca770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:53.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.581+0000 7f6a72ac9640 1 --2- 172.21.15.125:0/4152197111 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a6c06f7a0 0x7f6a6c14ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T12:59:53.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.581+0000 7f6a72ac9640 1 -- 172.21.15.125:0/4152197111 >> 172.21.15.125:0/4152197111 conn(0x7f6a6c0c7910 msgr2=0x7f6a6c157f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T12:59:53.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.581+0000 7f6a72ac9640 1 -- 172.21.15.125:0/4152197111 shutdown_connections 2023-10-30T12:59:53.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T12:59:53.581+0000 7f6a72ac9640 1 -- 172.21.15.125:0/4152197111 wait complete. 2023-10-30T12:59:53.601 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T12:59:53.601 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 27 secs before thrashing 2023-10-30T13:00:00.476 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 26% (49/183) Updating files: 27% (50/183) Updating files: 28% (52/183) Updating files: 29% (54/183) Updating files: 30% (55/183) Updating files: 31% (57/183) Updating files: 32% (59/183) Updating files: 33% (61/183) Updating files: 34% (63/183) Updating files: 35% (65/183) Updating files: 36% (66/183) Updating files: 37% (68/183) Updating files: 38% (70/183) Updating files: 39% (72/183) Updating files: 40% (74/183) Updating files: 41% (76/183) Updating files: 42% (77/183) Updating files: 43% (79/183) Updating files: 44% (81/183) Updating files: 45% (83/183) Updating files: 46% (85/183) Updating files: 47% (87/183) Updating files: 48% (88/183) Updating files: 49% (90/183) Updating files: 50% (92/183) Updating files: 51% (94/183) Updating files: 52% (96/183) Updating files: 53% (97/183) Updating files: 54% (99/183) Updating files: 55% (101/183) Updating files: 56% (103/183) Updating files: 57% (105/183) Updating files: 58% (107/183) Updating files: 59% (108/183) Updating files: 60% (110/183) Updating files: 61% (112/183) Updating files: 62% (114/183) Updating files: 63% (116/183) Updating files: 64% (118/183) Updating files: 65% (119/183) Updating files: 66% (121/183) Updating files: 67% (123/183) Updating files: 68% (125/183) Updating files: 69% (127/183) Updating files: 70% (129/183) Updating files: 71% (130/183) Updating files: 71% (131/183) Updating files: 72% (132/183) Updating files: 73% (134/183) Updating files: 74% (136/183) Updating files: 75% (138/183) Updating files: 76% (140/183) Updating files: 77% (141/183) Updating files: 78% (143/183) Updating files: 79% (145/183) Updating files: 80% (147/183) Updating files: 81% (149/183) Updating files: 82% (151/183) Updating files: 83% (152/183) Updating files: 84% (154/183) Updating files: 85% (156/183) Updating files: 86% (158/183) Updating files: 87% (160/183) Updating files: 88% (162/183) Updating files: 89% (163/183) Updating files: 90% (165/183) Updating files: 91% (167/183) Updating files: 92% (169/183) Updating files: 93% (171/183) Updating files: 94% (173/183) Updating files: 95% (174/183) Updating files: 96% (176/183) Updating files: 97% (178/183) Updating files: 98% (180/183) Updating files: 99% (182/183) Updating files: 100% (183/183) Updating files: 100% (183/183), done. 2023-10-30T13:00:00.784 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at e48859474c4 v0.52 2023-10-30T13:00:00.789 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.52 2023-10-30T13:00:01.553 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 53 -eq 48 ] 2023-10-30T13:00:01.554 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.53 2023-10-30T13:00:01.554 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.53 2023-10-30T13:00:01.555 INFO:tasks.workunit.client.0.smithi125.stdout:v0.53 2023-10-30T13:00:01.555 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.53 2023-10-30T13:00:10.369 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 47% (183/386) Updating files: 48% (186/386) Updating files: 49% (190/386) Updating files: 50% (193/386) Updating files: 51% (197/386) Updating files: 52% (201/386) Updating files: 53% (205/386) Updating files: 54% (209/386) Updating files: 55% (213/386) Updating files: 56% (217/386) Updating files: 56% (219/386) Updating files: 57% (221/386) Updating files: 58% (224/386) Updating files: 59% (228/386) Updating files: 60% (232/386) Updating files: 61% (236/386) Updating files: 62% (240/386) Updating files: 63% (244/386) Updating files: 63% (247/386) Updating files: 64% (248/386) Updating files: 65% (251/386) Updating files: 66% (255/386) Updating files: 67% (259/386) Updating files: 68% (263/386) Updating files: 69% (267/386) Updating files: 70% (271/386) Updating files: 71% (275/386) Updating files: 72% (278/386) Updating files: 73% (282/386) Updating files: 74% (286/386) Updating files: 75% (290/386) Updating files: 76% (294/386) Updating files: 77% (298/386) Updating files: 78% (302/386) Updating files: 79% (305/386) Updating files: 80% (309/386) Updating files: 80% (311/386) Updating files: 81% (313/386) Updating files: 82% (317/386) Updating files: 83% (321/386) Updating files: 84% (325/386) Updating files: 85% (329/386) Updating files: 86% (332/386) Updating files: 87% (336/386) Updating files: 87% (339/386) Updating files: 88% (340/386) Updating files: 89% (344/386) Updating files: 90% (348/386) Updating files: 91% (352/386) Updating files: 92% (356/386) Updating files: 93% (359/386) Updating files: 94% (363/386) Updating files: 95% (367/386) Updating files: 96% (371/386) Updating files: 97% (375/386) Updating files: 98% (379/386) Updating files: 99% (383/386) Updating files: 100% (386/386) Updating files: 100% (386/386), done. 2023-10-30T13:00:10.457 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 2528b5ee105 v0.43 2023-10-30T13:00:10.462 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.53 2023-10-30T13:00:10.733 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 54 -eq 48 ] 2023-10-30T13:00:10.733 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.54 2023-10-30T13:00:10.734 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.54 2023-10-30T13:00:10.734 INFO:tasks.workunit.client.0.smithi125.stdout:v0.54 2023-10-30T13:00:10.735 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.54 2023-10-30T13:00:14.684 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 74% (157/212) Updating files: 75% (159/212) Updating files: 76% (162/212) Updating files: 77% (164/212) Updating files: 78% (166/212) Updating files: 79% (168/212) Updating files: 80% (170/212) Updating files: 81% (172/212) Updating files: 82% (174/212) Updating files: 83% (176/212) Updating files: 84% (179/212) Updating files: 85% (181/212) Updating files: 86% (183/212) Updating files: 87% (185/212) Updating files: 88% (187/212) Updating files: 89% (189/212) Updating files: 90% (191/212) Updating files: 91% (193/212) Updating files: 92% (196/212) Updating files: 93% (198/212) Updating files: 94% (200/212) Updating files: 95% (202/212) Updating files: 96% (204/212) Updating files: 97% (206/212) Updating files: 98% (208/212) Updating files: 99% (210/212) Updating files: 100% (212/212) Updating files: 100% (212/212), done. 2023-10-30T13:00:14.733 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 60b84b095b1 v0.54 2023-10-30T13:00:14.740 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.54 2023-10-30T13:00:15.749 INFO:tasks.workunit.client.0.smithi125.stdout:v0.55 2023-10-30T13:00:15.749 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 55 -eq 48 ] 2023-10-30T13:00:15.750 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.55 2023-10-30T13:00:15.750 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.55 2023-10-30T13:00:15.750 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.55 2023-10-30T13:00:20.602 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:20.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a2a9f640 1 Processor -- start 2023-10-30T13:00:20.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a2a9f640 1 -- start start 2023-10-30T13:00:20.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a2a9f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f349c1538f0 0x7f349c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:20.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a2a9f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f349c15cd50 0x7f349c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:20.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a2a9f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f349c15d300 con 0x7f349c0cefa0 2023-10-30T13:00:20.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a2a9f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f349c15d5e0 con 0x7f349c1538f0 2023-10-30T13:00:20.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a2a9f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349c15d8c0 con 0x7f349c15cd50 2023-10-30T13:00:20.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a229e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f349c0cefa0 0x7f349c0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43392/0 (socket says 172.21.15.125:43392) 2023-10-30T13:00:20.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a229e640 1 -- 172.21.15.125:0/3628788514 learned_addr learned my addr 172.21.15.125:0/3628788514 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:00:20.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a1a9d640 1 --2- 172.21.15.125:0/3628788514 >> v2:172.21.15.136:3300/0 conn(0x7f349c1538f0 0x7f349c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:20.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a129c640 1 --2- 172.21.15.125:0/3628788514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f349c15cd50 0x7f349c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:20.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 -- 172.21.15.125:0/3628788514 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913395711 0 0) 0x7f349c15d300 con 0x7f349c0cefa0 2023-10-30T13:00:20.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 -- 172.21.15.125:0/3628788514 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3488003530 con 0x7f349c0cefa0 2023-10-30T13:00:20.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 -- 172.21.15.125:0/3628788514 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3299283914 0 0) 0x7f3488003530 con 0x7f349c0cefa0 2023-10-30T13:00:20.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 -- 172.21.15.125:0/3628788514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f349c15cd50 msgr2=0x7f349c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:20.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 --2- 172.21.15.125:0/3628788514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f349c15cd50 0x7f349c0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:20.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 -- 172.21.15.125:0/3628788514 >> v2:172.21.15.136:3300/0 conn(0x7f349c1538f0 msgr2=0x7f349c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:20.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 --2- 172.21.15.125:0/3628788514 >> v2:172.21.15.136:3300/0 conn(0x7f349c1538f0 0x7f349c159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:20.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 -- 172.21.15.125:0/3628788514 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c15dba0 con 0x7f349c0cefa0 2023-10-30T13:00:20.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 -- 172.21.15.125:0/3628788514 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f3494001a00 con 0x7f349c0cefa0 2023-10-30T13:00:20.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 -- 172.21.15.125:0/3628788514 >> v1:172.21.15.125:6789/0 conn(0x7f349c0cefa0 legacy=0x7f349c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:20.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 --2- 172.21.15.125:0/3628788514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3488003980 0x7f3488003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:20.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 --2- 172.21.15.125:0/3628788514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f349c15cd50 0x7f34880047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:20.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 --2- 172.21.15.125:0/3628788514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3488004d10 0x7f3488005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:20.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 -- 172.21.15.125:0/3628788514 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c15d5e0 con 0x7f3488003980 2023-10-30T13:00:20.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 -- 172.21.15.125:0/3628788514 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c15d8c0 con 0x7f3488004d10 2023-10-30T13:00:20.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a0a9b640 1 -- 172.21.15.125:0/3628788514 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348800b990 con 0x7f349c15cd50 2023-10-30T13:00:20.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a1a9d640 1 --2- 172.21.15.125:0/3628788514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3488003980 0x7f3488003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:20.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a229e640 1 --2- 172.21.15.125:0/3628788514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3488004d10 0x7f3488005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:20.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.777+0000 7f34a129c640 1 --2- 172.21.15.125:0/3628788514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f349c15cd50 0x7f34880047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:20.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.781+0000 7f34a1a9d640 1 -- 172.21.15.125:0/3628788514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f349c15cd50 msgr2=0x7f34880047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:20.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.781+0000 7f34a1a9d640 1 --2- 172.21.15.125:0/3628788514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f349c15cd50 0x7f34880047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:20.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.781+0000 7f34a1a9d640 1 -- 172.21.15.125:0/3628788514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3488004d10 msgr2=0x7f3488005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:20.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.781+0000 7f34a1a9d640 1 --2- 172.21.15.125:0/3628788514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3488004d10 0x7f3488005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:20.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.781+0000 7f34a1a9d640 1 -- 172.21.15.125:0/3628788514 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3488003530 con 0x7f3488003980 2023-10-30T13:00:20.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.781+0000 7f34a1a9d640 1 --2- 172.21.15.125:0/3628788514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3488003980 0x7f3488003da0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f3498009ae0 tx=0x7f349801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60fdfc6d01986a3e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:20.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.781+0000 7f34a0a9b640 1 -- 172.21.15.125:0/3628788514 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34980a5570 con 0x7f3488003980 2023-10-30T13:00:20.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.781+0000 7f34a0a9b640 1 -- 172.21.15.125:0/3628788514 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34980a56d0 con 0x7f3488003980 2023-10-30T13:00:20.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.781+0000 7f34a0a9b640 1 -- 172.21.15.125:0/3628788514 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34980a5a80 con 0x7f3488003980 2023-10-30T13:00:20.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.785+0000 7f34a2a9f640 1 -- 172.21.15.125:0/3628788514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3488003980 msgr2=0x7f3488003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:20.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.785+0000 7f34a2a9f640 1 --2- 172.21.15.125:0/3628788514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3488003980 0x7f3488003da0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7f3498009ae0 tx=0x7f349801ca90 comp rx=0 tx=0).stop 2023-10-30T13:00:20.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.785+0000 7f34a229e640 1 -- 172.21.15.125:0/3628788514 reap_dead start 2023-10-30T13:00:20.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.785+0000 7f34a2a9f640 1 -- 172.21.15.125:0/3628788514 shutdown_connections 2023-10-30T13:00:20.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.785+0000 7f34a2a9f640 1 -- 172.21.15.125:0/3628788514 >> 172.21.15.125:0/3628788514 conn(0x7f349c06a640 msgr2=0x7f349c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:20.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.785+0000 7f34a2a9f640 1 -- 172.21.15.125:0/3628788514 shutdown_connections 2023-10-30T13:00:20.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.785+0000 7f34a2a9f640 1 -- 172.21.15.125:0/3628788514 wait complete. 2023-10-30T13:00:20.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.785+0000 7f34a2a9f640 1 Processor -- start 2023-10-30T13:00:20.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.785+0000 7f34a2a9f640 1 -- start start 2023-10-30T13:00:20.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a2a9f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3488003980 0x7f349c157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:20.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a2a9f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f349c14f880 0x7f349c14dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:20.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a2a9f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f349c14e0e0 0x7f349c15c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:20.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a2a9f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c146460 con 0x7f349c14e0e0 2023-10-30T13:00:20.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a2a9f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349c14f5c0 con 0x7f3488003980 2023-10-30T13:00:20.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a2a9f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349c1f24c0 con 0x7f349c14f880 2023-10-30T13:00:20.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a229e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f349c14e0e0 0x7f349c15c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:20.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a229e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f349c14e0e0 0x7f349c15c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43514/0 (socket says 172.21.15.125:43514) 2023-10-30T13:00:20.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a229e640 1 -- 172.21.15.125:0/2644286765 learned_addr learned my addr 172.21.15.125:0/2644286765 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:20.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a129c640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f349c14f880 0x7f349c14dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:20.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a1a9d640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3488003980 0x7f349c157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:20.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a229e640 1 -- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f349c14f880 msgr2=0x7f349c14dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:20.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a229e640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f349c14f880 0x7f349c14dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:20.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a229e640 1 -- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3488003980 msgr2=0x7f349c157ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:20.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a229e640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3488003980 0x7f349c157ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:20.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a229e640 1 -- 172.21.15.125:0/2644286765 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f349c1f27a0 con 0x7f349c14e0e0 2023-10-30T13:00:20.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a129c640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f349c14f880 0x7f349c14dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:00:20.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a229e640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f349c14e0e0 0x7f349c15c250 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f34940013e0 tx=0x7f34940115c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:20.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34927fc640 1 -- 172.21.15.125:0/2644286765 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3494011df0 con 0x7f349c14e0e0 2023-10-30T13:00:20.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34927fc640 1 -- 172.21.15.125:0/2644286765 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3494004b70 con 0x7f349c14e0e0 2023-10-30T13:00:20.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34927fc640 1 -- 172.21.15.125:0/2644286765 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3494006070 con 0x7f349c14e0e0 2023-10-30T13:00:20.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a2a9f640 1 -- 172.21.15.125:0/2644286765 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f349c1f2a80 con 0x7f349c14e0e0 2023-10-30T13:00:20.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a2a9f640 1 -- 172.21.15.125:0/2644286765 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f349c1f3040 con 0x7f349c14e0e0 2023-10-30T13:00:20.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34a2a9f640 1 -- 172.21.15.125:0/2644286765 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f349c15b1a0 con 0x7f349c14e0e0 2023-10-30T13:00:20.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.789+0000 7f34927fc640 1 -- 172.21.15.125:0/2644286765 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f34940061d0 con 0x7f349c14e0e0 2023-10-30T13:00:20.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.793+0000 7f34927fc640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f34800664a0 0x7f3480068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:20.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.793+0000 7f34927fc640 1 -- 172.21.15.125:0/2644286765 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f34940871a0 con 0x7f349c14e0e0 2023-10-30T13:00:20.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.793+0000 7f34a1a9d640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f34800664a0 0x7f3480068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:20.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.793+0000 7f34a1a9d640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f34800664a0 0x7f3480068960 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f34980af000 tx=0x7f34980046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:20.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.797+0000 7f34927fc640 1 -- 172.21.15.125:0/2644286765 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f349c15b1a0 con 0x7f349c14e0e0 2023-10-30T13:00:20.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.985+0000 7f34a2a9f640 1 -- 172.21.15.125:0/2644286765 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f349c15b1a0 con 0x7f349c14e0e0 2023-10-30T13:00:20.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.985+0000 7f34927fc640 1 -- 172.21.15.125:0/2644286765 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f349c15b1a0 con 0x7f349c14e0e0 2023-10-30T13:00:20.989 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:20.990 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7707,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:44.626686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:active","state_seq":24,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:20.990 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 145 2023-10-30T13:00:20.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.989+0000 7f34a2a9f640 1 -- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f34800664a0 msgr2=0x7f3480068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:20.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.989+0000 7f34a2a9f640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f34800664a0 0x7f3480068960 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f34980af000 tx=0x7f34980046d0 comp rx=0 tx=0).stop 2023-10-30T13:00:20.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.989+0000 7f34a2a9f640 1 -- 172.21.15.125:0/2644286765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f349c14e0e0 msgr2=0x7f349c15c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:20.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.989+0000 7f34a2a9f640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f349c14e0e0 0x7f349c15c250 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f34940013e0 tx=0x7f34940115c0 comp rx=0 tx=0).stop 2023-10-30T13:00:20.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.989+0000 7f34a2a9f640 1 -- 172.21.15.125:0/2644286765 shutdown_connections 2023-10-30T13:00:20.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.989+0000 7f34a2a9f640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f34800664a0 0x7f3480068960 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:20.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.989+0000 7f34a2a9f640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f349c14e0e0 0x7f349c15c250 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:20.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.989+0000 7f34a2a9f640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f349c14f880 0x7f349c14dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:20.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.989+0000 7f34a2a9f640 1 --2- 172.21.15.125:0/2644286765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3488003980 0x7f349c157ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:20.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.989+0000 7f34a2a9f640 1 -- 172.21.15.125:0/2644286765 >> 172.21.15.125:0/2644286765 conn(0x7f349c06a640 msgr2=0x7f349c0cf550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:20.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.989+0000 7f34a2a9f640 1 -- 172.21.15.125:0/2644286765 shutdown_connections 2023-10-30T13:00:20.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:20.989+0000 7f34a2a9f640 1 -- 172.21.15.125:0/2644286765 wait complete. 2023-10-30T13:00:21.010 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T13:00:21.010 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T13:00:21.011 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:00:21.043 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T13:00:21.043 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:21.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc80e17640 1 Processor -- start 2023-10-30T13:00:21.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc80e17640 1 -- start start 2023-10-30T13:00:21.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc80e17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c1538a0 0x7fcc7c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:21.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc80e17640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c0cb800 con 0x7fcc7c0ca0e0 2023-10-30T13:00:21.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc80e17640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c15dc30 con 0x7fcc7c0cf160 2023-10-30T13:00:21.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc80e17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c0cb330 con 0x7fcc7c1538a0 2023-10-30T13:00:21.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc73fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcc7c0ca0e0 0x7fcc7c15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43396/0 (socket says 172.21.15.125:43396) 2023-10-30T13:00:21.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc73fff640 1 -- 172.21.15.125:0/2624496204 learned_addr learned my addr 172.21.15.125:0/2624496204 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:00:21.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc7a59c640 1 --2- 172.21.15.125:0/2624496204 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c1538a0 0x7fcc7c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:21.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 -- 172.21.15.125:0/2624496204 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225818812 0 0) 0x7fcc7c0cb800 con 0x7fcc7c0ca0e0 2023-10-30T13:00:21.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 -- 172.21.15.125:0/2624496204 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc68003530 con 0x7fcc7c0ca0e0 2023-10-30T13:00:21.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 -- 172.21.15.125:0/2624496204 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4259800257 0 0) 0x7fcc68003530 con 0x7fcc7c0ca0e0 2023-10-30T13:00:21.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 -- 172.21.15.125:0/2624496204 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c1538a0 msgr2=0x7fcc7c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:21.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 --2- 172.21.15.125:0/2624496204 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c1538a0 0x7fcc7c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:21.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 -- 172.21.15.125:0/2624496204 >> v1:172.21.15.136:6789/0 conn(0x7fcc7c0cf160 legacy=0x7fcc7c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:21.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 -- 172.21.15.125:0/2624496204 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c0cad70 con 0x7fcc7c0ca0e0 2023-10-30T13:00:21.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 -- 172.21.15.125:0/2624496204 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fcc64001680 con 0x7fcc7c0ca0e0 2023-10-30T13:00:21.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 -- 172.21.15.125:0/2624496204 >> v1:172.21.15.125:6789/0 conn(0x7fcc7c0ca0e0 legacy=0x7fcc7c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:21.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 --2- 172.21.15.125:0/2624496204 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c1538a0 0x7fcc680039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:21.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 --2- 172.21.15.125:0/2624496204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc68003f30 0x7fcc680047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:21.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 --2- 172.21.15.125:0/2624496204 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc68004d10 0x7fcc68005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:21.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 -- 172.21.15.125:0/2624496204 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c0cb330 con 0x7fcc68003f30 2023-10-30T13:00:21.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 -- 172.21.15.125:0/2624496204 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6800bb00 con 0x7fcc68004d10 2023-10-30T13:00:21.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc79d9b640 1 -- 172.21.15.125:0/2624496204 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6800bcc0 con 0x7fcc7c1538a0 2023-10-30T13:00:21.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.225+0000 7fcc73fff640 1 --2- 172.21.15.125:0/2624496204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc68003f30 0x7fcc680047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:21.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.229+0000 7fcc7a59c640 1 --2- 172.21.15.125:0/2624496204 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c1538a0 0x7fcc680039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:21.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.229+0000 7fcc7ad9d640 1 --2- 172.21.15.125:0/2624496204 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc68004d10 0x7fcc68005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:21.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.229+0000 7fcc73fff640 1 -- 172.21.15.125:0/2624496204 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c1538a0 msgr2=0x7fcc680039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:21.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.229+0000 7fcc73fff640 1 --2- 172.21.15.125:0/2624496204 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c1538a0 0x7fcc680039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:21.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.229+0000 7fcc73fff640 1 -- 172.21.15.125:0/2624496204 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc68004d10 msgr2=0x7fcc68005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:21.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.229+0000 7fcc73fff640 1 --2- 172.21.15.125:0/2624496204 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc68004d10 0x7fcc68005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:21.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.229+0000 7fcc73fff640 1 -- 172.21.15.125:0/2624496204 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc68003530 con 0x7fcc68003f30 2023-10-30T13:00:21.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.229+0000 7fcc7ad9d640 1 --2- 172.21.15.125:0/2624496204 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc68004d10 0x7fcc68005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:00:21.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.229+0000 7fcc7a59c640 1 --2- 172.21.15.125:0/2624496204 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c1538a0 0x7fcc680039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:00:21.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.229+0000 7fcc73fff640 1 --2- 172.21.15.125:0/2624496204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc68003f30 0x7fcc680047d0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400c1c0 tx=0x7fcc6401cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e79a91610b6e5f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:21.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.229+0000 7fcc79d9b640 1 -- 172.21.15.125:0/2624496204 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc640a5b20 con 0x7fcc68003f30 2023-10-30T13:00:21.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.229+0000 7fcc79d9b640 1 -- 172.21.15.125:0/2624496204 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc640a5c80 con 0x7fcc68003f30 2023-10-30T13:00:21.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.229+0000 7fcc79d9b640 1 -- 172.21.15.125:0/2624496204 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc64005b20 con 0x7fcc68003f30 2023-10-30T13:00:21.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc80e17640 1 -- 172.21.15.125:0/2624496204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc68003f30 msgr2=0x7fcc680047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:21.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc80e17640 1 --2- 172.21.15.125:0/2624496204 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc68003f30 0x7fcc680047d0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7fcc6400c1c0 tx=0x7fcc6401cc20 comp rx=0 tx=0).stop 2023-10-30T13:00:21.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc7ad9d640 1 -- 172.21.15.125:0/2624496204 reap_dead start 2023-10-30T13:00:21.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc80e17640 1 -- 172.21.15.125:0/2624496204 shutdown_connections 2023-10-30T13:00:21.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc80e17640 1 -- 172.21.15.125:0/2624496204 >> 172.21.15.125:0/2624496204 conn(0x7fcc7c06bfe0 msgr2=0x7fcc7c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:21.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc80e17640 1 -- 172.21.15.125:0/2624496204 shutdown_connections 2023-10-30T13:00:21.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc80e17640 1 -- 172.21.15.125:0/2624496204 wait complete. 2023-10-30T13:00:21.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc80e17640 1 Processor -- start 2023-10-30T13:00:21.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc80e17640 1 -- start start 2023-10-30T13:00:21.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc80e17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc68003f30 0x7fcc7c0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:21.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc80e17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc7c0b4750 0x7fcc7c0b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:21.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc80e17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c0ce040 0x7fcc7c0ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:21.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc80e17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c1ef240 con 0x7fcc68003f30 2023-10-30T13:00:21.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc80e17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c0cea00 con 0x7fcc7c0b4750 2023-10-30T13:00:21.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc80e17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c1ef450 con 0x7fcc7c0ce040 2023-10-30T13:00:21.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc7a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc68003f30 0x7fcc7c0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:21.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc7a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc68003f30 0x7fcc7c0cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43538/0 (socket says 172.21.15.125:43538) 2023-10-30T13:00:21.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc7a59c640 1 -- 172.21.15.125:0/621545590 learned_addr learned my addr 172.21.15.125:0/621545590 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:21.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc7ad9d640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c0ce040 0x7fcc7c0ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:21.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc73fff640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc7c0b4750 0x7fcc7c0b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:21.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc7a59c640 1 -- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c0ce040 msgr2=0x7fcc7c0ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:21.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc7a59c640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c0ce040 0x7fcc7c0ce4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:21.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc7a59c640 1 -- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc7c0b4750 msgr2=0x7fcc7c0b4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:21.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc7a59c640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc7c0b4750 0x7fcc7c0b4bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:21.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc7a59c640 1 -- 172.21.15.125:0/621545590 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c1ef730 con 0x7fcc68003f30 2023-10-30T13:00:21.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc7ad9d640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c0ce040 0x7fcc7c0ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:00:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.233+0000 7fcc73fff640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc7c0b4750 0x7fcc7c0b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:00:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.237+0000 7fcc7a59c640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc68003f30 0x7fcc7c0cd210 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009fd0 tx=0x7fcc6c011b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.237+0000 7fcc72ffd640 1 -- 172.21.15.125:0/621545590 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c004220 con 0x7fcc68003f30 2023-10-30T13:00:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.237+0000 7fcc72ffd640 1 -- 172.21.15.125:0/621545590 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c004380 con 0x7fcc68003f30 2023-10-30T13:00:21.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.237+0000 7fcc80e17640 1 -- 172.21.15.125:0/621545590 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7c1efa10 con 0x7fcc68003f30 2023-10-30T13:00:21.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.237+0000 7fcc72ffd640 1 -- 172.21.15.125:0/621545590 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc6c00f9e0 con 0x7fcc68003f30 2023-10-30T13:00:21.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.237+0000 7fcc80e17640 1 -- 172.21.15.125:0/621545590 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7c1effd0 con 0x7fcc68003f30 2023-10-30T13:00:21.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.237+0000 7fcc80e17640 1 -- 172.21.15.125:0/621545590 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7c1f0160 con 0x7fcc68003f30 2023-10-30T13:00:21.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.237+0000 7fcc72ffd640 1 -- 172.21.15.125:0/621545590 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcc6c00fb40 con 0x7fcc68003f30 2023-10-30T13:00:21.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.237+0000 7fcc72ffd640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc5c0664a0 0x7fcc5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:21.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.237+0000 7fcc72ffd640 1 -- 172.21.15.125:0/621545590 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7fcc6c084ab0 con 0x7fcc68003f30 2023-10-30T13:00:21.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.237+0000 7fcc73fff640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc5c0664a0 0x7fcc5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:21.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.241+0000 7fcc73fff640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc5c0664a0 0x7fcc5c068960 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fcc640ab000 tx=0x7fcc640a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:21.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.241+0000 7fcc72ffd640 1 -- 172.21.15.125:0/621545590 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcc6c051920 con 0x7fcc68003f30 2023-10-30T13:00:21.376 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:00:21.380 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:00:21.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.425+0000 7fcc80e17640 1 -- 172.21.15.125:0/621545590 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc7c04e170 con 0x7fcc68003f30 2023-10-30T13:00:21.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.425+0000 7fcc72ffd640 1 -- 172.21.15.125:0/621545590 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fcc6c022090 con 0x7fcc68003f30 2023-10-30T13:00:21.427 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:21.428 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7707,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:44.626686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:active","state_seq":24,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:21.428 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 145 2023-10-30T13:00:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.425+0000 7fcc80e17640 1 -- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc5c0664a0 msgr2=0x7fcc5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:21.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.425+0000 7fcc80e17640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc5c0664a0 0x7fcc5c068960 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7fcc640ab000 tx=0x7fcc640a7070 comp rx=0 tx=0).stop 2023-10-30T13:00:21.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.425+0000 7fcc80e17640 1 -- 172.21.15.125:0/621545590 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc68003f30 msgr2=0x7fcc7c0cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:21.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.425+0000 7fcc80e17640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc68003f30 0x7fcc7c0cd210 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fcc6c009fd0 tx=0x7fcc6c011b30 comp rx=0 tx=0).stop 2023-10-30T13:00:21.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.425+0000 7fcc80e17640 1 -- 172.21.15.125:0/621545590 shutdown_connections 2023-10-30T13:00:21.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.425+0000 7fcc80e17640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc5c0664a0 0x7fcc5c068960 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:21.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.425+0000 7fcc80e17640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc7c0ce040 0x7fcc7c0ce4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:21.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.425+0000 7fcc80e17640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc7c0b4750 0x7fcc7c0b4bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:21.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.425+0000 7fcc80e17640 1 --2- 172.21.15.125:0/621545590 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc68003f30 0x7fcc7c0cd210 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:21.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.425+0000 7fcc80e17640 1 -- 172.21.15.125:0/621545590 >> 172.21.15.125:0/621545590 conn(0x7fcc7c06bfe0 msgr2=0x7fcc7c151a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:21.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.425+0000 7fcc80e17640 1 -- 172.21.15.125:0/621545590 shutdown_connections 2023-10-30T13:00:21.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:21.425+0000 7fcc80e17640 1 -- 172.21.15.125:0/621545590 wait complete. 2023-10-30T13:00:21.449 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:00:23.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:23.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.597+0000 7fa055ed6640 1 Processor -- start 2023-10-30T13:00:23.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.597+0000 7fa055ed6640 1 -- start start 2023-10-30T13:00:23.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa055ed6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa050069e90 0x7fa05015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:23.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa055ed6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0500c6c10 con 0x7fa05006a270 2023-10-30T13:00:23.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa055ed6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0500c6ef0 con 0x7fa05006cb80 2023-10-30T13:00:23.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa055ed6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0500c8570 con 0x7fa050069e90 2023-10-30T13:00:23.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa047fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa05006a270 0x7fa0500c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43410/0 (socket says 172.21.15.125:43410) 2023-10-30T13:00:23.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa047fff640 1 -- 172.21.15.125:0/786370981 learned_addr learned my addr 172.21.15.125:0/786370981 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:00:23.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa04f7fe640 1 --2- 172.21.15.125:0/786370981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa050069e90 0x7fa05015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:23.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa04effd640 1 -- 172.21.15.125:0/786370981 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308164861 0 0) 0x7fa0500c6c10 con 0x7fa05006a270 2023-10-30T13:00:23.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa04effd640 1 -- 172.21.15.125:0/786370981 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa03c003530 con 0x7fa05006a270 2023-10-30T13:00:23.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa04effd640 1 -- 172.21.15.125:0/786370981 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867210097 0 0) 0x7fa0500c6ef0 con 0x7fa05006cb80 2023-10-30T13:00:23.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa04effd640 1 -- 172.21.15.125:0/786370981 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0500c6c10 con 0x7fa05006cb80 2023-10-30T13:00:23.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa04effd640 1 -- 172.21.15.125:0/786370981 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1117950639 0 0) 0x7fa03c003530 con 0x7fa05006a270 2023-10-30T13:00:23.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa04f7fe640 1 -- 172.21.15.125:0/786370981 >> v1:172.21.15.136:6789/0 conn(0x7fa05006cb80 legacy=0x7fa05006d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:23.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa04f7fe640 1 -- 172.21.15.125:0/786370981 >> v1:172.21.15.125:6789/0 conn(0x7fa05006a270 legacy=0x7fa0500c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:23.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.601+0000 7fa04f7fe640 1 -- 172.21.15.125:0/786370981 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0500c71d0 con 0x7fa050069e90 2023-10-30T13:00:23.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.605+0000 7fa04f7fe640 1 --2- 172.21.15.125:0/786370981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa050069e90 0x7fa05015a410 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa0400097f0 tx=0x7fa04001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47095d5c537c600e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:23.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.605+0000 7fa04effd640 1 -- 172.21.15.125:0/786370981 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa038001600 con 0x7fa050069e90 2023-10-30T13:00:23.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.605+0000 7fa04effd640 1 -- 172.21.15.125:0/786370981 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0400a7050 con 0x7fa050069e90 2023-10-30T13:00:23.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.605+0000 7fa04effd640 1 -- 172.21.15.125:0/786370981 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0400ab600 con 0x7fa050069e90 2023-10-30T13:00:23.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.605+0000 7fa055ed6640 1 -- 172.21.15.125:0/786370981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa050069e90 msgr2=0x7fa05015a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:23.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.605+0000 7fa055ed6640 1 --2- 172.21.15.125:0/786370981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa050069e90 0x7fa05015a410 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa0400097f0 tx=0x7fa04001c3e0 comp rx=0 tx=0).stop 2023-10-30T13:00:23.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.605+0000 7fa055ed6640 1 -- 172.21.15.125:0/786370981 shutdown_connections 2023-10-30T13:00:23.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.605+0000 7fa055ed6640 1 --2- 172.21.15.125:0/786370981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa050069e90 0x7fa05015a410 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:23.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.605+0000 7fa055ed6640 1 -- 172.21.15.125:0/786370981 >> 172.21.15.125:0/786370981 conn(0x7fa05006dfd0 msgr2=0x7fa05006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:23.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.605+0000 7fa055ed6640 1 -- 172.21.15.125:0/786370981 shutdown_connections 2023-10-30T13:00:23.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.605+0000 7fa055ed6640 1 -- 172.21.15.125:0/786370981 wait complete. 2023-10-30T13:00:23.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.605+0000 7fa055ed6640 1 Processor -- start 2023-10-30T13:00:23.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.605+0000 7fa055ed6640 1 -- start start 2023-10-30T13:00:23.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa055ed6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa050069e90 0x7fa05013ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:23.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa055ed6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa05006a270 0x7fa050142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:23.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa055ed6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa05006cb80 0x7fa0501f4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:23.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa055ed6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa050144a70 con 0x7fa05006cb80 2023-10-30T13:00:23.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa055ed6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa05013f120 con 0x7fa050069e90 2023-10-30T13:00:23.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa055ed6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa05014dff0 con 0x7fa05006a270 2023-10-30T13:00:23.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa04ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa05006cb80 0x7fa0501f4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:23.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa04ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa05006cb80 0x7fa0501f4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43552/0 (socket says 172.21.15.125:43552) 2023-10-30T13:00:23.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa04ffff640 1 -- 172.21.15.125:0/1548520971 learned_addr learned my addr 172.21.15.125:0/1548520971 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:23.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa047fff640 1 --2- 172.21.15.125:0/1548520971 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa05006a270 0x7fa050142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:23.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa04f7fe640 1 --2- 172.21.15.125:0/1548520971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa050069e90 0x7fa05013ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:23.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa04ffff640 1 -- 172.21.15.125:0/1548520971 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa05006a270 msgr2=0x7fa050142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:23.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa04ffff640 1 --2- 172.21.15.125:0/1548520971 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa05006a270 0x7fa050142ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:23.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa04ffff640 1 -- 172.21.15.125:0/1548520971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa050069e90 msgr2=0x7fa05013ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:23.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa04ffff640 1 --2- 172.21.15.125:0/1548520971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa050069e90 0x7fa05013ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:23.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa04ffff640 1 -- 172.21.15.125:0/1548520971 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa040009420 con 0x7fa05006cb80 2023-10-30T13:00:23.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa04ffff640 1 --2- 172.21.15.125:0/1548520971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa05006cb80 0x7fa0501f4ac0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fa04800c7a0 tx=0x7fa04800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:23.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa04cff9640 1 -- 172.21.15.125:0/1548520971 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa048003b80 con 0x7fa05006cb80 2023-10-30T13:00:23.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa04cff9640 1 -- 172.21.15.125:0/1548520971 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa048003ce0 con 0x7fa05006cb80 2023-10-30T13:00:23.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa055ed6640 1 -- 172.21.15.125:0/1548520971 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa050144790 con 0x7fa05006cb80 2023-10-30T13:00:23.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa04cff9640 1 -- 172.21.15.125:0/1548520971 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04800aa30 con 0x7fa05006cb80 2023-10-30T13:00:23.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa055ed6640 1 -- 172.21.15.125:0/1548520971 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa05014dc80 con 0x7fa05006cb80 2023-10-30T13:00:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa055ed6640 1 -- 172.21.15.125:0/1548520971 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0501f58c0 con 0x7fa05006cb80 2023-10-30T13:00:23.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.609+0000 7fa04cff9640 1 -- 172.21.15.125:0/1548520971 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa04800ab90 con 0x7fa05006cb80 2023-10-30T13:00:23.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.613+0000 7fa04cff9640 1 --2- 172.21.15.125:0/1548520971 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0300664a0 0x7fa030068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:23.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.613+0000 7fa04cff9640 1 -- 172.21.15.125:0/1548520971 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7fa048084300 con 0x7fa05006cb80 2023-10-30T13:00:23.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.613+0000 7fa04f7fe640 1 --2- 172.21.15.125:0/1548520971 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0300664a0 0x7fa030068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:23.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.613+0000 7fa04f7fe640 1 --2- 172.21.15.125:0/1548520971 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0300664a0 0x7fa030068960 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa0400ad000 tx=0x7fa0400a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:23.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.617+0000 7fa04cff9640 1 -- 172.21.15.125:0/1548520971 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa0501f58c0 con 0x7fa05006cb80 2023-10-30T13:00:23.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.797+0000 7fa055ed6640 1 -- 172.21.15.125:0/1548520971 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa05014dae0 con 0x7fa05006cb80 2023-10-30T13:00:23.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.801+0000 7fa04cff9640 1 -- 172.21.15.125:0/1548520971 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fa05014dae0 con 0x7fa05006cb80 2023-10-30T13:00:23.803 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:23.803 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7707,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:44.626686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:active","state_seq":24,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:23.805 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 145 2023-10-30T13:00:23.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.801+0000 7fa055ed6640 1 -- 172.21.15.125:0/1548520971 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0300664a0 msgr2=0x7fa030068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:23.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.801+0000 7fa055ed6640 1 --2- 172.21.15.125:0/1548520971 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0300664a0 0x7fa030068960 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fa0400ad000 tx=0x7fa0400a6000 comp rx=0 tx=0).stop 2023-10-30T13:00:23.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.801+0000 7fa055ed6640 1 -- 172.21.15.125:0/1548520971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa05006cb80 msgr2=0x7fa0501f4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:23.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.801+0000 7fa055ed6640 1 --2- 172.21.15.125:0/1548520971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa05006cb80 0x7fa0501f4ac0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fa04800c7a0 tx=0x7fa04800cc60 comp rx=0 tx=0).stop 2023-10-30T13:00:23.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.801+0000 7fa055ed6640 1 -- 172.21.15.125:0/1548520971 shutdown_connections 2023-10-30T13:00:23.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.801+0000 7fa055ed6640 1 --2- 172.21.15.125:0/1548520971 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa0300664a0 0x7fa030068960 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:23.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.801+0000 7fa055ed6640 1 --2- 172.21.15.125:0/1548520971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa05006cb80 0x7fa0501f4ac0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:23.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.801+0000 7fa055ed6640 1 --2- 172.21.15.125:0/1548520971 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa05006a270 0x7fa050142ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:23.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.801+0000 7fa055ed6640 1 --2- 172.21.15.125:0/1548520971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa050069e90 0x7fa05013ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:23.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.801+0000 7fa055ed6640 1 -- 172.21.15.125:0/1548520971 >> 172.21.15.125:0/1548520971 conn(0x7fa05006dfd0 msgr2=0x7fa05014e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:23.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.801+0000 7fa055ed6640 1 -- 172.21.15.125:0/1548520971 shutdown_connections 2023-10-30T13:00:23.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:23.801+0000 7fa055ed6640 1 -- 172.21.15.125:0/1548520971 wait complete. 2023-10-30T13:00:23.824 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:00:25.825 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:25.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f6056891640 1 Processor -- start 2023-10-30T13:00:25.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f6056891640 1 -- start start 2023-10-30T13:00:25.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f6056891640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f60501539a0 0x7f605015a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:25.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f6056891640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6050069e90 0x7f605006a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:25.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f6056891640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605015a8c0 con 0x7f6050070a00 2023-10-30T13:00:25.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f6056891640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f605006fbb0 con 0x7f60501539a0 2023-10-30T13:00:25.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f6056891640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6050153d80 con 0x7f6050069e90 2023-10-30T13:00:25.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f604ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f60501539a0 0x7f605015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:25.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f604ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f60501539a0 0x7f605015a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43556/0 (socket says 172.21.15.125:43556) 2023-10-30T13:00:25.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f604ffff640 1 -- 172.21.15.125:0/8809636 learned_addr learned my addr 172.21.15.125:0/8809636 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:25.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f6054e2e640 1 --2- 172.21.15.125:0/8809636 >> v2:172.21.15.136:3300/0 conn(0x7f6050069e90 0x7f605006a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:25.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f604ffff640 1 -- 172.21.15.125:0/8809636 >> v2:172.21.15.136:3300/0 conn(0x7f6050069e90 msgr2=0x7f605006a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:25.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f604ffff640 1 --2- 172.21.15.125:0/8809636 >> v2:172.21.15.136:3300/0 conn(0x7f6050069e90 0x7f605006a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:25.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f604ffff640 1 -- 172.21.15.125:0/8809636 >> v1:172.21.15.136:6789/0 conn(0x7f6050070a00 legacy=0x7f6050070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:25.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.969+0000 7f604ffff640 1 -- 172.21.15.125:0/8809636 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60500c5c50 con 0x7f60501539a0 2023-10-30T13:00:25.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.973+0000 7f604ffff640 1 --2- 172.21.15.125:0/8809636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60501539a0 0x7f605015a380 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f60500c8900 tx=0x7f604001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc0383e6c5a05c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:25.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.973+0000 7f604effd640 1 -- 172.21.15.125:0/8809636 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040004230 con 0x7f60501539a0 2023-10-30T13:00:25.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.973+0000 7f604effd640 1 -- 172.21.15.125:0/8809636 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60400a7050 con 0x7f60501539a0 2023-10-30T13:00:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.973+0000 7f604effd640 1 -- 172.21.15.125:0/8809636 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040003620 con 0x7f60501539a0 2023-10-30T13:00:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.973+0000 7f6056891640 1 -- 172.21.15.125:0/8809636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60501539a0 msgr2=0x7f605015a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.973+0000 7f6056891640 1 --2- 172.21.15.125:0/8809636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60501539a0 0x7f605015a380 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7f60500c8900 tx=0x7f604001c440 comp rx=0 tx=0).stop 2023-10-30T13:00:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.973+0000 7f6056891640 1 -- 172.21.15.125:0/8809636 shutdown_connections 2023-10-30T13:00:25.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.973+0000 7f6056891640 1 --2- 172.21.15.125:0/8809636 >> v2:172.21.15.136:3300/0 conn(0x7f6050069e90 0x7f605006a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:25.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.973+0000 7f6056891640 1 --2- 172.21.15.125:0/8809636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60501539a0 0x7f605015a380 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:25.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.973+0000 7f6056891640 1 -- 172.21.15.125:0/8809636 >> 172.21.15.125:0/8809636 conn(0x7f60500c7910 msgr2=0x7f60500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:25.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.973+0000 7f6056891640 1 -- 172.21.15.125:0/8809636 shutdown_connections 2023-10-30T13:00:25.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.973+0000 7f6056891640 1 -- 172.21.15.125:0/8809636 wait complete. 2023-10-30T13:00:25.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f6056891640 1 Processor -- start 2023-10-30T13:00:25.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f6056891640 1 -- start start 2023-10-30T13:00:25.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f6056891640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6050069e90 0x7f605006b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:25.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f6056891640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6050070a00 0x7f60500c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:25.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f6056891640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60501539a0 0x7f605014f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:25.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f6056891640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60500c9fa0 con 0x7f6050069e90 2023-10-30T13:00:25.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f6056891640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60500ca260 con 0x7f60501539a0 2023-10-30T13:00:25.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f6056891640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60500ca540 con 0x7f6050070a00 2023-10-30T13:00:25.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f604ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6050069e90 0x7f605006b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:25.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f604ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6050069e90 0x7f605006b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43560/0 (socket says 172.21.15.125:43560) 2023-10-30T13:00:25.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f604ffff640 1 -- 172.21.15.125:0/579233149 learned_addr learned my addr 172.21.15.125:0/579233149 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:25.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f604f7fe640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6050070a00 0x7f60500c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:25.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f6054e2e640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60501539a0 0x7f605014f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:25.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f604ffff640 1 -- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6050070a00 msgr2=0x7f60500c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:25.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f604ffff640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6050070a00 0x7f60500c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:25.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f604ffff640 1 -- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60501539a0 msgr2=0x7f605014f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:25.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f604ffff640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60501539a0 0x7f605014f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:25.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f604ffff640 1 -- 172.21.15.125:0/579233149 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60400094d0 con 0x7f6050069e90 2023-10-30T13:00:25.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.977+0000 7f6054e2e640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60501539a0 0x7f605014f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:00:25.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.981+0000 7f604ffff640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6050069e90 0x7f605006b7c0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f60400ae040 tx=0x7f60400191e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:25.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.981+0000 7f604cff9640 1 -- 172.21.15.125:0/579233149 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6040003bf0 con 0x7f6050069e90 2023-10-30T13:00:25.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.981+0000 7f604cff9640 1 -- 172.21.15.125:0/579233149 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6040003d70 con 0x7f6050069e90 2023-10-30T13:00:25.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.981+0000 7f604cff9640 1 -- 172.21.15.125:0/579233149 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60400ab680 con 0x7f6050069e90 2023-10-30T13:00:25.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.981+0000 7f6056891640 1 -- 172.21.15.125:0/579233149 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6050143360 con 0x7f6050069e90 2023-10-30T13:00:25.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.981+0000 7f6056891640 1 -- 172.21.15.125:0/579233149 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6050143010 con 0x7f6050069e90 2023-10-30T13:00:25.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.981+0000 7f6056891640 1 -- 172.21.15.125:0/579233149 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f605004e170 con 0x7f6050069e90 2023-10-30T13:00:25.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.981+0000 7f604cff9640 1 -- 172.21.15.125:0/579233149 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f60400b2040 con 0x7f6050069e90 2023-10-30T13:00:25.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.981+0000 7f604cff9640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f603c0664a0 0x7f603c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:25.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.981+0000 7f604cff9640 1 -- 172.21.15.125:0/579233149 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f604011e640 con 0x7f6050069e90 2023-10-30T13:00:25.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.981+0000 7f604f7fe640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f603c0664a0 0x7f603c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:25.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.985+0000 7f604f7fe640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f603c0664a0 0x7f603c068960 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6050142950 tx=0x7f6044009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:25.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:25.985+0000 7f604cff9640 1 -- 172.21.15.125:0/579233149 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f60400eb5d0 con 0x7f6050069e90 2023-10-30T13:00:26.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:26.165+0000 7f6056891640 1 -- 172.21.15.125:0/579233149 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f605014f370 con 0x7f6050069e90 2023-10-30T13:00:26.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:26.165+0000 7f604cff9640 1 -- 172.21.15.125:0/579233149 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f60400eebf0 con 0x7f6050069e90 2023-10-30T13:00:26.170 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:26.170 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7707,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:44.626686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:active","state_seq":24,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:26.171 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 145 2023-10-30T13:00:26.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:26.169+0000 7f6056891640 1 -- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f603c0664a0 msgr2=0x7f603c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:26.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:26.169+0000 7f6056891640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f603c0664a0 0x7f603c068960 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6050142950 tx=0x7f6044009040 comp rx=0 tx=0).stop 2023-10-30T13:00:26.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:26.169+0000 7f6056891640 1 -- 172.21.15.125:0/579233149 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6050069e90 msgr2=0x7f605006b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:26.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:26.169+0000 7f6056891640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6050069e90 0x7f605006b7c0 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f60400ae040 tx=0x7f60400191e0 comp rx=0 tx=0).stop 2023-10-30T13:00:26.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:26.169+0000 7f6056891640 1 -- 172.21.15.125:0/579233149 shutdown_connections 2023-10-30T13:00:26.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:26.169+0000 7f6056891640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f603c0664a0 0x7f603c068960 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:26.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:26.169+0000 7f6056891640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60501539a0 0x7f605014f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:26.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:26.169+0000 7f6056891640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6050070a00 0x7f60500c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:26.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:26.169+0000 7f6056891640 1 --2- 172.21.15.125:0/579233149 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6050069e90 0x7f605006b7c0 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:26.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:26.169+0000 7f6056891640 1 -- 172.21.15.125:0/579233149 >> 172.21.15.125:0/579233149 conn(0x7f60500c7910 msgr2=0x7f6050070e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:26.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:26.169+0000 7f6056891640 1 -- 172.21.15.125:0/579233149 shutdown_connections 2023-10-30T13:00:26.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:26.169+0000 7f6056891640 1 -- 172.21.15.125:0/579233149 wait complete. 2023-10-30T13:00:26.191 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:00:28.192 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:28.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb8a46640 1 Processor -- start 2023-10-30T13:00:28.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb8a46640 1 -- start start 2023-10-30T13:00:28.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb8a46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0cb40c05d0 0x7f0cb413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:28.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb8a46640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0cb40c6f60 0x7f0cb413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:28.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb8a46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb40c0a50 con 0x7f0cb40c6b80 2023-10-30T13:00:28.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb8a46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb406f6e0 con 0x7f0cb40c05d0 2023-10-30T13:00:28.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb8a46640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb406f9c0 con 0x7f0cb40c6f60 2023-10-30T13:00:28.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb259c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0cb40c6b80 0x7f0cb40bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43424/0 (socket says 172.21.15.125:43424) 2023-10-30T13:00:28.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb259c640 1 -- 172.21.15.125:0/1886341259 learned_addr learned my addr 172.21.15.125:0/1886341259 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:00:28.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb1d9b640 1 --2- 172.21.15.125:0/1886341259 >> v2:172.21.15.125:3300/0 conn(0x7f0cb40c05d0 0x7f0cb413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:28.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb2d9d640 1 --2- 172.21.15.125:0/1886341259 >> v2:172.21.15.136:3300/0 conn(0x7f0cb40c6f60 0x7f0cb413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:28.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb159a640 1 -- 172.21.15.125:0/1886341259 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160306978 0 0) 0x7f0cb40c0a50 con 0x7f0cb40c6b80 2023-10-30T13:00:28.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb159a640 1 -- 172.21.15.125:0/1886341259 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c9c003530 con 0x7f0cb40c6b80 2023-10-30T13:00:28.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb1d9b640 1 -- 172.21.15.125:0/1886341259 >> v2:172.21.15.136:3300/0 conn(0x7f0cb40c6f60 msgr2=0x7f0cb413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:28.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb1d9b640 1 --2- 172.21.15.125:0/1886341259 >> v2:172.21.15.136:3300/0 conn(0x7f0cb40c6f60 0x7f0cb413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:28.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb1d9b640 1 -- 172.21.15.125:0/1886341259 >> v1:172.21.15.125:6789/0 conn(0x7f0cb40c6b80 legacy=0x7f0cb40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:28.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.337+0000 7f0cb1d9b640 1 -- 172.21.15.125:0/1886341259 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb406fca0 con 0x7f0cb40c05d0 2023-10-30T13:00:28.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.341+0000 7f0cb1d9b640 1 --2- 172.21.15.125:0/1886341259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c05d0 0x7f0cb413ed90 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40c89b0 tx=0x7f0ca801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c108cf274e923a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:28.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.341+0000 7f0cb159a640 1 -- 172.21.15.125:0/1886341259 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca80ab600 con 0x7f0cb40c05d0 2023-10-30T13:00:28.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.341+0000 7f0cb159a640 1 -- 172.21.15.125:0/1886341259 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca80a7050 con 0x7f0cb40c05d0 2023-10-30T13:00:28.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.341+0000 7f0cb159a640 1 -- 172.21.15.125:0/1886341259 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca80ab980 con 0x7f0cb40c05d0 2023-10-30T13:00:28.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.341+0000 7f0cb8a46640 1 -- 172.21.15.125:0/1886341259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c05d0 msgr2=0x7f0cb413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:28.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.341+0000 7f0cb8a46640 1 --2- 172.21.15.125:0/1886341259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c05d0 0x7f0cb413ed90 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40c89b0 tx=0x7f0ca801c460 comp rx=0 tx=0).stop 2023-10-30T13:00:28.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.341+0000 7f0cb8a46640 1 -- 172.21.15.125:0/1886341259 shutdown_connections 2023-10-30T13:00:28.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.341+0000 7f0cb8a46640 1 --2- 172.21.15.125:0/1886341259 >> v2:172.21.15.136:3300/0 conn(0x7f0cb40c6f60 0x7f0cb413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:28.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.341+0000 7f0cb8a46640 1 --2- 172.21.15.125:0/1886341259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c05d0 0x7f0cb413ed90 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:28.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.341+0000 7f0cb8a46640 1 -- 172.21.15.125:0/1886341259 >> 172.21.15.125:0/1886341259 conn(0x7f0cb40c7910 msgr2=0x7f0cb40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:28.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.341+0000 7f0cb8a46640 1 -- 172.21.15.125:0/1886341259 shutdown_connections 2023-10-30T13:00:28.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.341+0000 7f0cb8a46640 1 -- 172.21.15.125:0/1886341259 wait complete. 2023-10-30T13:00:28.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb8a46640 1 Processor -- start 2023-10-30T13:00:28.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb8a46640 1 -- start start 2023-10-30T13:00:28.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb8a46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cb40c05d0 0x7f0cb414dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:28.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb8a46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cb40c6b80 0x7f0cb414ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:28.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb8a46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6f60 0x7f0cb41f4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:28.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb8a46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb41ebd40 con 0x7f0cb40c6f60 2023-10-30T13:00:28.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb8a46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb4157910 con 0x7f0cb40c6b80 2023-10-30T13:00:28.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb8a46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb4157bf0 con 0x7f0cb40c05d0 2023-10-30T13:00:28.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb2d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6f60 0x7f0cb41f4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:28.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb2d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6f60 0x7f0cb41f4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43566/0 (socket says 172.21.15.125:43566) 2023-10-30T13:00:28.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb2d9d640 1 -- 172.21.15.125:0/3145679315 learned_addr learned my addr 172.21.15.125:0/3145679315 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:28.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb259c640 1 --2- 172.21.15.125:0/3145679315 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cb40c05d0 0x7f0cb414dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:28.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb1d9b640 1 --2- 172.21.15.125:0/3145679315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cb40c6b80 0x7f0cb414ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:28.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb2d9d640 1 -- 172.21.15.125:0/3145679315 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cb40c05d0 msgr2=0x7f0cb414dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:28.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb2d9d640 1 --2- 172.21.15.125:0/3145679315 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cb40c05d0 0x7f0cb414dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:28.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb2d9d640 1 -- 172.21.15.125:0/3145679315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cb40c6b80 msgr2=0x7f0cb414ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:28.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb2d9d640 1 --2- 172.21.15.125:0/3145679315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cb40c6b80 0x7f0cb414ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:28.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb2d9d640 1 -- 172.21.15.125:0/3145679315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca8009420 con 0x7f0cb40c6f60 2023-10-30T13:00:28.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb2d9d640 1 --2- 172.21.15.125:0/3145679315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6f60 0x7f0cb41f4cd0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f0cac007e60 tx=0x7f0cac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:28.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0ca2ffd640 1 -- 172.21.15.125:0/3145679315 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac014420 con 0x7f0cb40c6f60 2023-10-30T13:00:28.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0ca2ffd640 1 -- 172.21.15.125:0/3145679315 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cac0145a0 con 0x7f0cb40c6f60 2023-10-30T13:00:28.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0ca2ffd640 1 -- 172.21.15.125:0/3145679315 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac014950 con 0x7f0cb40c6f60 2023-10-30T13:00:28.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb8a46640 1 -- 172.21.15.125:0/3145679315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb4157ed0 con 0x7f0cb40c6f60 2023-10-30T13:00:28.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb8a46640 1 -- 172.21.15.125:0/3145679315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb4158490 con 0x7f0cb40c6f60 2023-10-30T13:00:28.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.345+0000 7f0cb8a46640 1 -- 172.21.15.125:0/3145679315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb4158680 con 0x7f0cb40c6f60 2023-10-30T13:00:28.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.349+0000 7f0ca2ffd640 1 -- 172.21.15.125:0/3145679315 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0cac014ab0 con 0x7f0cb40c6f60 2023-10-30T13:00:28.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.349+0000 7f0ca2ffd640 1 --2- 172.21.15.125:0/3145679315 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c940664a0 0x7f0c94068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:28.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.349+0000 7f0ca2ffd640 1 -- 172.21.15.125:0/3145679315 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f0cac084d20 con 0x7f0cb40c6f60 2023-10-30T13:00:28.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.349+0000 7f0cb259c640 1 --2- 172.21.15.125:0/3145679315 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c940664a0 0x7f0c94068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:28.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.349+0000 7f0cb259c640 1 --2- 172.21.15.125:0/3145679315 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c940664a0 0x7f0c94068960 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40011f0 tx=0x7f0ca4007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:28.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.353+0000 7f0ca2ffd640 1 -- 172.21.15.125:0/3145679315 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0cac051ba0 con 0x7f0cb40c6f60 2023-10-30T13:00:28.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.533+0000 7f0cb8a46640 1 -- 172.21.15.125:0/3145679315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cb404e170 con 0x7f0cb40c6f60 2023-10-30T13:00:28.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.533+0000 7f0ca2ffd640 1 -- 172.21.15.125:0/3145679315 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f0cac0551c0 con 0x7f0cb40c6f60 2023-10-30T13:00:28.538 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:28.539 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7707,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:44.626686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:active","state_seq":24,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:28.539 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 145 2023-10-30T13:00:28.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.537+0000 7f0cb8a46640 1 -- 172.21.15.125:0/3145679315 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c940664a0 msgr2=0x7f0c94068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:28.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.537+0000 7f0cb8a46640 1 --2- 172.21.15.125:0/3145679315 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c940664a0 0x7f0c94068960 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40011f0 tx=0x7f0ca4007d40 comp rx=0 tx=0).stop 2023-10-30T13:00:28.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.537+0000 7f0cb8a46640 1 -- 172.21.15.125:0/3145679315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6f60 msgr2=0x7f0cb41f4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:28.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.537+0000 7f0cb8a46640 1 --2- 172.21.15.125:0/3145679315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6f60 0x7f0cb41f4cd0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f0cac007e60 tx=0x7f0cac00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:00:28.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.537+0000 7f0cb8a46640 1 -- 172.21.15.125:0/3145679315 shutdown_connections 2023-10-30T13:00:28.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.537+0000 7f0cb8a46640 1 --2- 172.21.15.125:0/3145679315 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c940664a0 0x7f0c94068960 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:28.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.537+0000 7f0cb8a46640 1 --2- 172.21.15.125:0/3145679315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6f60 0x7f0cb41f4cd0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:28.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.537+0000 7f0cb8a46640 1 --2- 172.21.15.125:0/3145679315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cb40c6b80 0x7f0cb414ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:28.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.537+0000 7f0cb8a46640 1 --2- 172.21.15.125:0/3145679315 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cb40c05d0 0x7f0cb414dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:28.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.537+0000 7f0cb8a46640 1 -- 172.21.15.125:0/3145679315 >> 172.21.15.125:0/3145679315 conn(0x7f0cb40c7910 msgr2=0x7f0cb406be40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:28.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.537+0000 7f0cb8a46640 1 -- 172.21.15.125:0/3145679315 shutdown_connections 2023-10-30T13:00:28.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:28.537+0000 7f0cb8a46640 1 -- 172.21.15.125:0/3145679315 wait complete. 2023-10-30T13:00:28.559 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:00:30.561 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:30.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3be986b640 1 Processor -- start 2023-10-30T13:00:30.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3be986b640 1 -- start start 2023-10-30T13:00:30.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3be986b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3be415d6f0 0x7f3be415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:30.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3be986b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3be40cf160 0x7f3be40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:30.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3be986b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be40cb800 con 0x7f3be41538a0 2023-10-30T13:00:30.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3be986b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be40cb330 con 0x7f3be40cf160 2023-10-30T13:00:30.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3be986b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3be40cad70 con 0x7f3be415d6f0 2023-10-30T13:00:30.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3be37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3be40cf160 0x7f3be40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:30.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3be2ffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3be41538a0 0x7f3be40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43436/0 (socket says 172.21.15.125:43436) 2023-10-30T13:00:30.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3be2ffd640 1 -- 172.21.15.125:0/1071410592 learned_addr learned my addr 172.21.15.125:0/1071410592 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:00:30.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3bdbfff640 1 --2- 172.21.15.125:0/1071410592 >> v2:172.21.15.136:3300/0 conn(0x7f3be415d6f0 0x7f3be415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:30.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3be37fe640 1 -- 172.21.15.125:0/1071410592 >> v2:172.21.15.136:3300/0 conn(0x7f3be415d6f0 msgr2=0x7f3be415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:30.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3be37fe640 1 --2- 172.21.15.125:0/1071410592 >> v2:172.21.15.136:3300/0 conn(0x7f3be415d6f0 0x7f3be415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:30.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3be37fe640 1 -- 172.21.15.125:0/1071410592 >> v1:172.21.15.125:6789/0 conn(0x7f3be41538a0 legacy=0x7f3be40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:30.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.705+0000 7f3be27fc640 1 -- 172.21.15.125:0/1071410592 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1104440839 0 0) 0x7f3be40cb800 con 0x7f3be41538a0 2023-10-30T13:00:30.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.709+0000 7f3be37fe640 1 -- 172.21.15.125:0/1071410592 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be40cb050 con 0x7f3be40cf160 2023-10-30T13:00:30.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.709+0000 7f3be37fe640 1 --2- 172.21.15.125:0/1071410592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3be40cf160 0x7f3be40cf590 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f3be40b30a0 tx=0x7f3bdc01f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2220d682ab436f87 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:30.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.709+0000 7f3be27fc640 1 -- 172.21.15.125:0/1071410592 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc0b0440 con 0x7f3be40cf160 2023-10-30T13:00:30.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.709+0000 7f3be27fc640 1 -- 172.21.15.125:0/1071410592 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bdc0b05a0 con 0x7f3be40cf160 2023-10-30T13:00:30.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.709+0000 7f3be27fc640 1 -- 172.21.15.125:0/1071410592 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bdc0b0930 con 0x7f3be40cf160 2023-10-30T13:00:30.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.709+0000 7f3be986b640 1 -- 172.21.15.125:0/1071410592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3be40cf160 msgr2=0x7f3be40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:30.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.709+0000 7f3be986b640 1 --2- 172.21.15.125:0/1071410592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3be40cf160 0x7f3be40cf590 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f3be40b30a0 tx=0x7f3bdc01f0f0 comp rx=0 tx=0).stop 2023-10-30T13:00:30.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.709+0000 7f3be986b640 1 -- 172.21.15.125:0/1071410592 shutdown_connections 2023-10-30T13:00:30.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.709+0000 7f3be986b640 1 --2- 172.21.15.125:0/1071410592 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3be40cf160 0x7f3be40cf590 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:30.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.709+0000 7f3be986b640 1 --2- 172.21.15.125:0/1071410592 >> v2:172.21.15.136:3300/0 conn(0x7f3be415d6f0 0x7f3be415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:30.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.709+0000 7f3be986b640 1 -- 172.21.15.125:0/1071410592 >> 172.21.15.125:0/1071410592 conn(0x7f3be406bfe0 msgr2=0x7f3be406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:30.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.709+0000 7f3be986b640 1 -- 172.21.15.125:0/1071410592 shutdown_connections 2023-10-30T13:00:30.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.709+0000 7f3be986b640 1 -- 172.21.15.125:0/1071410592 wait complete. 2023-10-30T13:00:30.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be986b640 1 Processor -- start 2023-10-30T13:00:30.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be986b640 1 -- start start 2023-10-30T13:00:30.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be986b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3be40cf160 0x7f3be41542c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:30.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be986b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3be41538a0 0x7f3be40b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:30.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be986b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be415d6f0 0x7f3be41ef880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:30.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be986b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be4156c20 con 0x7f3be40cf160 2023-10-30T13:00:30.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be986b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3be41f3690 con 0x7f3be415d6f0 2023-10-30T13:00:30.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be986b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3be41f3110 con 0x7f3be41538a0 2023-10-30T13:00:30.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be2ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3be40cf160 0x7f3be41542c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:30.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be2ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3be40cf160 0x7f3be41542c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43590/0 (socket says 172.21.15.125:43590) 2023-10-30T13:00:30.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be2ffd640 1 -- 172.21.15.125:0/2180293049 learned_addr learned my addr 172.21.15.125:0/2180293049 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:30.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be37fe640 1 --2- 172.21.15.125:0/2180293049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be415d6f0 0x7f3be41ef880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:30.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3bdbfff640 1 --2- 172.21.15.125:0/2180293049 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3be41538a0 0x7f3be40b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:30.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be2ffd640 1 -- 172.21.15.125:0/2180293049 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3be41538a0 msgr2=0x7f3be40b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:30.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be2ffd640 1 --2- 172.21.15.125:0/2180293049 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3be41538a0 0x7f3be40b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:30.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be2ffd640 1 -- 172.21.15.125:0/2180293049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be415d6f0 msgr2=0x7f3be41ef880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:30.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be2ffd640 1 --2- 172.21.15.125:0/2180293049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be415d6f0 0x7f3be41ef880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:30.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be2ffd640 1 -- 172.21.15.125:0/2180293049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc009a70 con 0x7f3be40cf160 2023-10-30T13:00:30.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be2ffd640 1 --2- 172.21.15.125:0/2180293049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3be40cf160 0x7f3be41542c0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40011f0 tx=0x7f3bd400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:30.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be0ff9640 1 -- 172.21.15.125:0/2180293049 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd40074e0 con 0x7f3be40cf160 2023-10-30T13:00:30.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be0ff9640 1 -- 172.21.15.125:0/2180293049 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd4007640 con 0x7f3be40cf160 2023-10-30T13:00:30.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be0ff9640 1 -- 172.21.15.125:0/2180293049 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd4004a10 con 0x7f3be40cf160 2023-10-30T13:00:30.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be986b640 1 -- 172.21.15.125:0/2180293049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be4154800 con 0x7f3be40cf160 2023-10-30T13:00:30.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.713+0000 7f3be986b640 1 -- 172.21.15.125:0/2180293049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be41f33b0 con 0x7f3be40cf160 2023-10-30T13:00:30.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.717+0000 7f3be986b640 1 -- 172.21.15.125:0/2180293049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be40b2a50 con 0x7f3be40cf160 2023-10-30T13:00:30.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.717+0000 7f3be0ff9640 1 -- 172.21.15.125:0/2180293049 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3bd4004b70 con 0x7f3be40cf160 2023-10-30T13:00:30.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.717+0000 7f3be0ff9640 1 --2- 172.21.15.125:0/2180293049 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3bc40664a0 0x7f3bc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:30.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.717+0000 7f3be0ff9640 1 -- 172.21.15.125:0/2180293049 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f3bd40837b0 con 0x7f3be40cf160 2023-10-30T13:00:30.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.717+0000 7f3bdbfff640 1 --2- 172.21.15.125:0/2180293049 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3bc40664a0 0x7f3bc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:30.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.721+0000 7f3bdbfff640 1 --2- 172.21.15.125:0/2180293049 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3bc40664a0 0x7f3bc4068960 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3be40b5330 tx=0x7f3bcc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:30.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.721+0000 7f3be0ff9640 1 -- 172.21.15.125:0/2180293049 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3bd4050610 con 0x7f3be40cf160 2023-10-30T13:00:30.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.901+0000 7f3be986b640 1 -- 172.21.15.125:0/2180293049 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3be404e170 con 0x7f3be40cf160 2023-10-30T13:00:30.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.901+0000 7f3be0ff9640 1 -- 172.21.15.125:0/2180293049 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f3bd4018090 con 0x7f3be40cf160 2023-10-30T13:00:30.906 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:30.906 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7707,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:44.626686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:active","state_seq":24,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:30.908 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 145 2023-10-30T13:00:30.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.905+0000 7f3be986b640 1 -- 172.21.15.125:0/2180293049 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3bc40664a0 msgr2=0x7f3bc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:30.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.905+0000 7f3be986b640 1 --2- 172.21.15.125:0/2180293049 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3bc40664a0 0x7f3bc4068960 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f3be40b5330 tx=0x7f3bcc008040 comp rx=0 tx=0).stop 2023-10-30T13:00:30.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.905+0000 7f3be986b640 1 -- 172.21.15.125:0/2180293049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3be40cf160 msgr2=0x7f3be41542c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:30.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.905+0000 7f3be986b640 1 --2- 172.21.15.125:0/2180293049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3be40cf160 0x7f3be41542c0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7f3bd40011f0 tx=0x7f3bd400c9b0 comp rx=0 tx=0).stop 2023-10-30T13:00:30.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.905+0000 7f3be986b640 1 -- 172.21.15.125:0/2180293049 shutdown_connections 2023-10-30T13:00:30.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.905+0000 7f3be986b640 1 --2- 172.21.15.125:0/2180293049 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3bc40664a0 0x7f3bc4068960 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:30.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.905+0000 7f3be986b640 1 --2- 172.21.15.125:0/2180293049 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3be415d6f0 0x7f3be41ef880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:30.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.905+0000 7f3be986b640 1 --2- 172.21.15.125:0/2180293049 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3be41538a0 0x7f3be40b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:30.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.905+0000 7f3be986b640 1 --2- 172.21.15.125:0/2180293049 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3be40cf160 0x7f3be41542c0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:30.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.905+0000 7f3be986b640 1 -- 172.21.15.125:0/2180293049 >> 172.21.15.125:0/2180293049 conn(0x7f3be406bfe0 msgr2=0x7f3be415aac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:30.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.905+0000 7f3be986b640 1 -- 172.21.15.125:0/2180293049 shutdown_connections 2023-10-30T13:00:30.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:30.905+0000 7f3be986b640 1 -- 172.21.15.125:0/2180293049 wait complete. 2023-10-30T13:00:30.927 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:00:32.928 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:33.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.076+0000 7f7a46d3e640 1 Processor -- start 2023-10-30T13:00:33.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.076+0000 7f7a46d3e640 1 -- start start 2023-10-30T13:00:33.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.076+0000 7f7a46d3e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7a40153bf0 0x7f7a40071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:33.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.076+0000 7f7a46d3e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a400700b0 con 0x7f7a400c5f60 2023-10-30T13:00:33.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.076+0000 7f7a46d3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a400c6530 con 0x7f7a400c6fe0 2023-10-30T13:00:33.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.076+0000 7f7a46d3e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a400c6810 con 0x7f7a40153bf0 2023-10-30T13:00:33.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.076+0000 7f7a452db640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7a400c5f60 0x7f7a400cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38228/0 (socket says 172.21.15.125:38228) 2023-10-30T13:00:33.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.076+0000 7f7a452db640 1 -- 172.21.15.125:0/3892176653 learned_addr learned my addr 172.21.15.125:0/3892176653 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:00:33.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a44ada640 1 --2- 172.21.15.125:0/3892176653 >> v2:172.21.15.136:3300/0 conn(0x7f7a40153bf0 0x7f7a40071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:33.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83239109 0 0) 0x7f7a400700b0 con 0x7f7a400c5f60 2023-10-30T13:00:33.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a30003530 con 0x7f7a400c5f60 2023-10-30T13:00:33.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316130471 0 0) 0x7f7a400c6530 con 0x7f7a400c6fe0 2023-10-30T13:00:33.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a400700b0 con 0x7f7a400c6fe0 2023-10-30T13:00:33.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1026429467 0 0) 0x7f7a30003530 con 0x7f7a400c5f60 2023-10-30T13:00:33.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 >> v2:172.21.15.136:3300/0 conn(0x7f7a40153bf0 msgr2=0x7f7a40071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:33.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 --2- 172.21.15.125:0/3892176653 >> v2:172.21.15.136:3300/0 conn(0x7f7a40153bf0 0x7f7a40071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:33.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 >> v1:172.21.15.136:6789/0 conn(0x7f7a400c6fe0 legacy=0x7f7a400c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:33.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a40159b10 con 0x7f7a400c5f60 2023-10-30T13:00:33.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f7a38001a00 con 0x7f7a400c5f60 2023-10-30T13:00:33.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 >> v1:172.21.15.125:6789/0 conn(0x7f7a400c5f60 legacy=0x7f7a400cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:33.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 --2- 172.21.15.125:0/3892176653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7a30003b60 0x7f7a30003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:33.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 --2- 172.21.15.125:0/3892176653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a300047d0 0x7f7a30004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:33.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 --2- 172.21.15.125:0/3892176653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a30005190 0x7f7a3000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:33.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a400c6810 con 0x7f7a30005190 2023-10-30T13:00:33.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3000bfb0 con 0x7f7a300047d0 2023-10-30T13:00:33.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3000c170 con 0x7f7a30003b60 2023-10-30T13:00:33.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a452db640 1 --2- 172.21.15.125:0/3892176653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a30005190 0x7f7a3000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:33.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a44ada640 1 --2- 172.21.15.125:0/3892176653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7a30003b60 0x7f7a30003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:33.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3ffff640 1 --2- 172.21.15.125:0/3892176653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a300047d0 0x7f7a30004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:33.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a452db640 1 -- 172.21.15.125:0/3892176653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7a30003b60 msgr2=0x7f7a30003f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:33.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a452db640 1 --2- 172.21.15.125:0/3892176653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7a30003b60 0x7f7a30003f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:33.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a452db640 1 -- 172.21.15.125:0/3892176653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a300047d0 msgr2=0x7f7a30004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:33.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a452db640 1 --2- 172.21.15.125:0/3892176653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a300047d0 0x7f7a30004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:33.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a452db640 1 -- 172.21.15.125:0/3892176653 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a30003530 con 0x7f7a30005190 2023-10-30T13:00:33.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.080+0000 7f7a3ffff640 1 --2- 172.21.15.125:0/3892176653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a300047d0 0x7f7a30004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:00:33.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.084+0000 7f7a452db640 1 -- 172.21.15.125:0/3892176653 reap_dead start 2023-10-30T13:00:33.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.084+0000 7f7a452db640 1 --2- 172.21.15.125:0/3892176653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a30005190 0x7f7a3000b990 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800a710 tx=0x7f7a3801d1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d32c69fec5bebd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:33.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.084+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a380ad630 con 0x7f7a30005190 2023-10-30T13:00:33.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.084+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a380a8070 con 0x7f7a30005190 2023-10-30T13:00:33.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.084+0000 7f7a3f7fe640 1 -- 172.21.15.125:0/3892176653 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a380ada10 con 0x7f7a30005190 2023-10-30T13:00:33.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.084+0000 7f7a46d3e640 1 -- 172.21.15.125:0/3892176653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a30005190 msgr2=0x7f7a3000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:33.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.084+0000 7f7a46d3e640 1 --2- 172.21.15.125:0/3892176653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a30005190 0x7f7a3000b990 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800a710 tx=0x7f7a3801d1e0 comp rx=0 tx=0).stop 2023-10-30T13:00:33.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.084+0000 7f7a46d3e640 1 -- 172.21.15.125:0/3892176653 shutdown_connections 2023-10-30T13:00:33.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.084+0000 7f7a46d3e640 1 --2- 172.21.15.125:0/3892176653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a30005190 0x7f7a3000b990 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:33.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.084+0000 7f7a46d3e640 1 -- 172.21.15.125:0/3892176653 >> 172.21.15.125:0/3892176653 conn(0x7f7a4006c6e0 msgr2=0x7f7a40069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:33.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.084+0000 7f7a46d3e640 1 -- 172.21.15.125:0/3892176653 shutdown_connections 2023-10-30T13:00:33.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.084+0000 7f7a46d3e640 1 -- 172.21.15.125:0/3892176653 wait complete. 2023-10-30T13:00:33.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.084+0000 7f7a46d3e640 1 Processor -- start 2023-10-30T13:00:33.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.084+0000 7f7a46d3e640 1 -- start start 2023-10-30T13:00:33.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a46d3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a30005190 0x7f7a400c43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:33.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a46d3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a40154d30 0x7f7a400cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:33.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a46d3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7a4014c190 0x7f7a4014c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:33.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a46d3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a40150680 con 0x7f7a30005190 2023-10-30T13:00:33.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a46d3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a401f20e0 con 0x7f7a40154d30 2023-10-30T13:00:33.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a46d3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a40154a70 con 0x7f7a4014c190 2023-10-30T13:00:33.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a44ada640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a30005190 0x7f7a400c43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:33.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a44ada640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a30005190 0x7f7a400c43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35716/0 (socket says 172.21.15.125:35716) 2023-10-30T13:00:33.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a44ada640 1 -- 172.21.15.125:0/168606610 learned_addr learned my addr 172.21.15.125:0/168606610 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:33.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a452db640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7a4014c190 0x7f7a4014c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:33.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a3ffff640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a40154d30 0x7f7a400cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:33.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a44ada640 1 -- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7a4014c190 msgr2=0x7f7a4014c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:33.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a44ada640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7a4014c190 0x7f7a4014c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:33.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a44ada640 1 -- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a40154d30 msgr2=0x7f7a400cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:33.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a44ada640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a40154d30 0x7f7a400cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:33.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a44ada640 1 -- 172.21.15.125:0/168606610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a40156ec0 con 0x7f7a30005190 2023-10-30T13:00:33.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a452db640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7a4014c190 0x7f7a4014c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:00:33.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a44ada640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a30005190 0x7f7a400c43b0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f7a34011960 tx=0x7f7a34011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:33.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a3d7fa640 1 -- 172.21.15.125:0/168606610 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3400fa90 con 0x7f7a30005190 2023-10-30T13:00:33.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a3d7fa640 1 -- 172.21.15.125:0/168606610 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a3400fbf0 con 0x7f7a30005190 2023-10-30T13:00:33.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a46d3e640 1 -- 172.21.15.125:0/168606610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a401571a0 con 0x7f7a30005190 2023-10-30T13:00:33.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a3d7fa640 1 -- 172.21.15.125:0/168606610 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a34014440 con 0x7f7a30005190 2023-10-30T13:00:33.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a46d3e640 1 -- 172.21.15.125:0/168606610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a401f25f0 con 0x7f7a30005190 2023-10-30T13:00:33.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a46d3e640 1 -- 172.21.15.125:0/168606610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4004e170 con 0x7f7a30005190 2023-10-30T13:00:33.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.088+0000 7f7a3d7fa640 1 -- 172.21.15.125:0/168606610 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7a340145a0 con 0x7f7a30005190 2023-10-30T13:00:33.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.092+0000 7f7a3d7fa640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7a240664d0 0x7f7a24068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:33.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.092+0000 7f7a3d7fa640 1 -- 172.21.15.125:0/168606610 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f7a34084c90 con 0x7f7a30005190 2023-10-30T13:00:33.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.092+0000 7f7a3ffff640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7a240664d0 0x7f7a24068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:33.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.092+0000 7f7a3ffff640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7a240664d0 0x7f7a24068990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7a400c31e0 tx=0x7f7a2c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:33.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.096+0000 7f7a3d7fa640 1 -- 172.21.15.125:0/168606610 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7a34051a60 con 0x7f7a30005190 2023-10-30T13:00:33.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.276+0000 7f7a46d3e640 1 -- 172.21.15.125:0/168606610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a40159dc0 con 0x7f7a30005190 2023-10-30T13:00:33.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.276+0000 7f7a3d7fa640 1 -- 172.21.15.125:0/168606610 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f7a34055080 con 0x7f7a30005190 2023-10-30T13:00:33.283 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:33.283 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7707,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:44.626686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:active","state_seq":24,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:33.284 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 145 2023-10-30T13:00:33.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.280+0000 7f7a46d3e640 1 -- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7a240664d0 msgr2=0x7f7a24068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:33.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.280+0000 7f7a46d3e640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7a240664d0 0x7f7a24068990 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f7a400c31e0 tx=0x7f7a2c009040 comp rx=0 tx=0).stop 2023-10-30T13:00:33.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.280+0000 7f7a46d3e640 1 -- 172.21.15.125:0/168606610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a30005190 msgr2=0x7f7a400c43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:33.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.280+0000 7f7a46d3e640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a30005190 0x7f7a400c43b0 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f7a34011960 tx=0x7f7a34011e20 comp rx=0 tx=0).stop 2023-10-30T13:00:33.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.280+0000 7f7a46d3e640 1 -- 172.21.15.125:0/168606610 shutdown_connections 2023-10-30T13:00:33.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.280+0000 7f7a46d3e640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7a240664d0 0x7f7a24068990 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:33.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.280+0000 7f7a46d3e640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7a4014c190 0x7f7a4014c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:33.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.280+0000 7f7a46d3e640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7a40154d30 0x7f7a400cacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:33.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.280+0000 7f7a46d3e640 1 --2- 172.21.15.125:0/168606610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7a30005190 0x7f7a400c43b0 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:33.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.280+0000 7f7a46d3e640 1 -- 172.21.15.125:0/168606610 >> 172.21.15.125:0/168606610 conn(0x7f7a4006c6e0 msgr2=0x7f7a400c94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:33.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.280+0000 7f7a46d3e640 1 -- 172.21.15.125:0/168606610 shutdown_connections 2023-10-30T13:00:33.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:33.280+0000 7f7a46d3e640 1 -- 172.21.15.125:0/168606610 wait complete. 2023-10-30T13:00:33.303 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:00:35.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:35.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdf4971640 1 Processor -- start 2023-10-30T13:00:35.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdf4971640 1 -- start start 2023-10-30T13:00:35.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdf4971640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdf00cefa0 0x7efdf00cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:35.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdf4971640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf015a6c0 con 0x7efdf01538f0 2023-10-30T13:00:35.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdf4971640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf00ca6a0 con 0x7efdf015cd50 2023-10-30T13:00:35.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdf4971640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf015d300 con 0x7efdf00cefa0 2023-10-30T13:00:35.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdef7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7efdf01538f0 0x7efdf0159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38242/0 (socket says 172.21.15.125:38242) 2023-10-30T13:00:35.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdef7fe640 1 -- 172.21.15.125:0/745880390 learned_addr learned my addr 172.21.15.125:0/745880390 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:00:35.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdeffff640 1 --2- 172.21.15.125:0/745880390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdf00cefa0 0x7efdf00cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:35.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 -- 172.21.15.125:0/745880390 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594525256 0 0) 0x7efdf015a6c0 con 0x7efdf01538f0 2023-10-30T13:00:35.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 -- 172.21.15.125:0/745880390 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdd8003530 con 0x7efdf01538f0 2023-10-30T13:00:35.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 -- 172.21.15.125:0/745880390 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2107541455 0 0) 0x7efdd8003530 con 0x7efdf01538f0 2023-10-30T13:00:35.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 -- 172.21.15.125:0/745880390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdf00cefa0 msgr2=0x7efdf00cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:35.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 --2- 172.21.15.125:0/745880390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdf00cefa0 0x7efdf00cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:35.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 -- 172.21.15.125:0/745880390 >> v1:172.21.15.136:6789/0 conn(0x7efdf015cd50 legacy=0x7efdf00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:35.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 -- 172.21.15.125:0/745880390 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf015d5e0 con 0x7efdf01538f0 2023-10-30T13:00:35.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 -- 172.21.15.125:0/745880390 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7efde0001680 con 0x7efdf01538f0 2023-10-30T13:00:35.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 -- 172.21.15.125:0/745880390 >> v1:172.21.15.125:6789/0 conn(0x7efdf01538f0 legacy=0x7efdf0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:35.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 --2- 172.21.15.125:0/745880390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdf00cefa0 0x7efdd80039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:35.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 --2- 172.21.15.125:0/745880390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdd8003f30 0x7efdd80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:35.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 --2- 172.21.15.125:0/745880390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdd8004d10 0x7efdd8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:35.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 -- 172.21.15.125:0/745880390 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf015d300 con 0x7efdd8003f30 2023-10-30T13:00:35.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 -- 172.21.15.125:0/745880390 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd800bb00 con 0x7efdd8004d10 2023-10-30T13:00:35.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdee7fc640 1 -- 172.21.15.125:0/745880390 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdd800bcc0 con 0x7efdf00cefa0 2023-10-30T13:00:35.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdeeffd640 1 --2- 172.21.15.125:0/745880390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdd8003f30 0x7efdd80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:35.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdef7fe640 1 --2- 172.21.15.125:0/745880390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdf00cefa0 0x7efdd80039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:35.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdeffff640 1 --2- 172.21.15.125:0/745880390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdd8004d10 0x7efdd8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:35.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdeeffd640 1 -- 172.21.15.125:0/745880390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdf00cefa0 msgr2=0x7efdd80039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:35.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdeeffd640 1 --2- 172.21.15.125:0/745880390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdf00cefa0 0x7efdd80039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:35.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdeeffd640 1 -- 172.21.15.125:0/745880390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdd8004d10 msgr2=0x7efdd8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:35.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdeeffd640 1 --2- 172.21.15.125:0/745880390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdd8004d10 0x7efdd8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:35.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.448+0000 7efdeeffd640 1 -- 172.21.15.125:0/745880390 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdd8003530 con 0x7efdd8003f30 2023-10-30T13:00:35.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.452+0000 7efdeeffd640 1 --2- 172.21.15.125:0/745880390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdd8003f30 0x7efdd80047d0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7efde4001780 tx=0x7efde401c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b24c4dc295dbc5cd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:35.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.452+0000 7efdee7fc640 1 -- 172.21.15.125:0/745880390 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde40052d0 con 0x7efdd8003f30 2023-10-30T13:00:35.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.452+0000 7efdee7fc640 1 -- 172.21.15.125:0/745880390 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde40a7050 con 0x7efdd8003f30 2023-10-30T13:00:35.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.452+0000 7efdee7fc640 1 -- 172.21.15.125:0/745880390 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde4003600 con 0x7efdd8003f30 2023-10-30T13:00:35.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.452+0000 7efdf4971640 1 -- 172.21.15.125:0/745880390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdd8003f30 msgr2=0x7efdd80047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:35.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.452+0000 7efdf4971640 1 --2- 172.21.15.125:0/745880390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdd8003f30 0x7efdd80047d0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7efde4001780 tx=0x7efde401c400 comp rx=0 tx=0).stop 2023-10-30T13:00:35.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.452+0000 7efdeffff640 1 -- 172.21.15.125:0/745880390 reap_dead start 2023-10-30T13:00:35.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.452+0000 7efdf4971640 1 -- 172.21.15.125:0/745880390 shutdown_connections 2023-10-30T13:00:35.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.452+0000 7efdf4971640 1 -- 172.21.15.125:0/745880390 >> 172.21.15.125:0/745880390 conn(0x7efdf006a640 msgr2=0x7efdf0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:35.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.452+0000 7efdf4971640 1 -- 172.21.15.125:0/745880390 shutdown_connections 2023-10-30T13:00:35.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.452+0000 7efdf4971640 1 -- 172.21.15.125:0/745880390 wait complete. 2023-10-30T13:00:35.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdf4971640 1 Processor -- start 2023-10-30T13:00:35.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdf4971640 1 -- start start 2023-10-30T13:00:35.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdf4971640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdd8003f30 0x7efdf00b2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:35.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdf4971640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdf0159170 0x7efdf0157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:35.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdf4971640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdf0157d90 0x7efdf013e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:35.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdf4971640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf00b3b90 con 0x7efdd8003f30 2023-10-30T13:00:35.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdf4971640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf0152170 con 0x7efdf0157d90 2023-10-30T13:00:35.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdf4971640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf00b4d80 con 0x7efdf0159170 2023-10-30T13:00:35.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdef7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdd8003f30 0x7efdf00b2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:35.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdef7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdd8003f30 0x7efdf00b2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35738/0 (socket says 172.21.15.125:35738) 2023-10-30T13:00:35.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdef7fe640 1 -- 172.21.15.125:0/1774750355 learned_addr learned my addr 172.21.15.125:0/1774750355 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:35.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdeeffd640 1 --2- 172.21.15.125:0/1774750355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdf0159170 0x7efdf0157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:35.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdeffff640 1 --2- 172.21.15.125:0/1774750355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdf0157d90 0x7efdf013e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:35.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdef7fe640 1 -- 172.21.15.125:0/1774750355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdf0159170 msgr2=0x7efdf0157850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:35.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdef7fe640 1 --2- 172.21.15.125:0/1774750355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdf0159170 0x7efdf0157850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:35.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdef7fe640 1 -- 172.21.15.125:0/1774750355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdf0157d90 msgr2=0x7efdf013e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:35.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdef7fe640 1 --2- 172.21.15.125:0/1774750355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdf0157d90 0x7efdf013e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:35.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdef7fe640 1 -- 172.21.15.125:0/1774750355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf00b5060 con 0x7efdd8003f30 2023-10-30T13:00:35.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdef7fe640 1 --2- 172.21.15.125:0/1774750355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdd8003f30 0x7efdf00b2ea0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7efde00072c0 tx=0x7efde0007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:35.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdd7fff640 1 -- 172.21.15.125:0/1774750355 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde000f940 con 0x7efdd8003f30 2023-10-30T13:00:35.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdd7fff640 1 -- 172.21.15.125:0/1774750355 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde000faa0 con 0x7efdd8003f30 2023-10-30T13:00:35.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdf4971640 1 -- 172.21.15.125:0/1774750355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdf0152380 con 0x7efdd8003f30 2023-10-30T13:00:35.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdd7fff640 1 -- 172.21.15.125:0/1774750355 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde000fe30 con 0x7efdd8003f30 2023-10-30T13:00:35.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.456+0000 7efdf4971640 1 -- 172.21.15.125:0/1774750355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf0152940 con 0x7efdd8003f30 2023-10-30T13:00:35.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.460+0000 7efdf4971640 1 -- 172.21.15.125:0/1774750355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf004e170 con 0x7efdd8003f30 2023-10-30T13:00:35.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.460+0000 7efdd7fff640 1 -- 172.21.15.125:0/1774750355 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efde001d460 con 0x7efdd8003f30 2023-10-30T13:00:35.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.460+0000 7efdd7fff640 1 --2- 172.21.15.125:0/1774750355 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efddc0664d0 0x7efddc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:35.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.460+0000 7efdd7fff640 1 -- 172.21.15.125:0/1774750355 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7efde00846e0 con 0x7efdd8003f30 2023-10-30T13:00:35.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.460+0000 7efdeeffd640 1 --2- 172.21.15.125:0/1774750355 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efddc0664d0 0x7efddc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:35.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.460+0000 7efdeeffd640 1 --2- 172.21.15.125:0/1774750355 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efddc0664d0 0x7efddc068990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7efde4004000 tx=0x7efde40011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:35.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.464+0000 7efdd7fff640 1 -- 172.21.15.125:0/1774750355 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efde00514b0 con 0x7efdd8003f30 2023-10-30T13:00:35.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.648+0000 7efdf4971640 1 -- 172.21.15.125:0/1774750355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efdf0159a30 con 0x7efdd8003f30 2023-10-30T13:00:35.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.648+0000 7efdd7fff640 1 -- 172.21.15.125:0/1774750355 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6519 (secure 0 0 0) 0x7efde0054ad0 con 0x7efdd8003f30 2023-10-30T13:00:35.653 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:35.653 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7707,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:44.626686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:active","state_seq":24,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:35.654 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 145 2023-10-30T13:00:35.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.648+0000 7efdf4971640 1 -- 172.21.15.125:0/1774750355 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efddc0664d0 msgr2=0x7efddc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:35.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.648+0000 7efdf4971640 1 --2- 172.21.15.125:0/1774750355 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efddc0664d0 0x7efddc068990 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7efde4004000 tx=0x7efde40011a0 comp rx=0 tx=0).stop 2023-10-30T13:00:35.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.648+0000 7efdf4971640 1 -- 172.21.15.125:0/1774750355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdd8003f30 msgr2=0x7efdf00b2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:35.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.648+0000 7efdf4971640 1 --2- 172.21.15.125:0/1774750355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdd8003f30 0x7efdf00b2ea0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7efde00072c0 tx=0x7efde0007780 comp rx=0 tx=0).stop 2023-10-30T13:00:35.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.648+0000 7efdf4971640 1 -- 172.21.15.125:0/1774750355 shutdown_connections 2023-10-30T13:00:35.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.648+0000 7efdf4971640 1 --2- 172.21.15.125:0/1774750355 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efddc0664d0 0x7efddc068990 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:35.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.648+0000 7efdf4971640 1 --2- 172.21.15.125:0/1774750355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efdf0157d90 0x7efdf013e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:35.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.652+0000 7efdf4971640 1 --2- 172.21.15.125:0/1774750355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efdf0159170 0x7efdf0157850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:35.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.652+0000 7efdf4971640 1 --2- 172.21.15.125:0/1774750355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efdd8003f30 0x7efdf00b2ea0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:35.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.652+0000 7efdf4971640 1 -- 172.21.15.125:0/1774750355 >> 172.21.15.125:0/1774750355 conn(0x7efdf006a640 msgr2=0x7efdf00c47a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:35.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.652+0000 7efdf4971640 1 -- 172.21.15.125:0/1774750355 shutdown_connections 2023-10-30T13:00:35.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:35.652+0000 7efdf4971640 1 -- 172.21.15.125:0/1774750355 wait complete. 2023-10-30T13:00:35.674 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:00:37.674 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:37.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.816+0000 7fc742c9c640 1 Processor -- start 2023-10-30T13:00:37.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.816+0000 7fc742c9c640 1 -- start start 2023-10-30T13:00:37.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.820+0000 7fc742c9c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc73c159e60 0x7fc73c06f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:37.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.820+0000 7fc742c9c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc73c06a720 0x7fc73c06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:37.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.820+0000 7fc742c9c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc73c06fa80 con 0x7fc73c0bfec0 2023-10-30T13:00:37.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.820+0000 7fc742c9c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc73c071290 con 0x7fc73c159e60 2023-10-30T13:00:37.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.820+0000 7fc742c9c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c071570 con 0x7fc73c06a720 2023-10-30T13:00:37.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.820+0000 7fc740a38640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc73c159e60 0x7fc73c06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:37.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.820+0000 7fc740a38640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc73c159e60 0x7fc73c06f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35746/0 (socket says 172.21.15.125:35746) 2023-10-30T13:00:37.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.820+0000 7fc740a38640 1 -- 172.21.15.125:0/2587793289 learned_addr learned my addr 172.21.15.125:0/2587793289 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:37.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.820+0000 7fc741239640 1 --2- 172.21.15.125:0/2587793289 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc73c06a720 0x7fc73c06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:37.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.820+0000 7fc740a38640 1 -- 172.21.15.125:0/2587793289 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc73c06a720 msgr2=0x7fc73c06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:37.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.820+0000 7fc740a38640 1 --2- 172.21.15.125:0/2587793289 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc73c06a720 0x7fc73c06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:37.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.820+0000 7fc740a38640 1 -- 172.21.15.125:0/2587793289 >> v1:172.21.15.136:6789/0 conn(0x7fc73c0bfec0 legacy=0x7fc73c0c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:37.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.820+0000 7fc740a38640 1 -- 172.21.15.125:0/2587793289 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73c071850 con 0x7fc73c159e60 2023-10-30T13:00:37.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.824+0000 7fc740a38640 1 --2- 172.21.15.125:0/2587793289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc73c159e60 0x7fc73c06f540 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fc738006fd0 tx=0x7fc7380211c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=417b367b7a803bf7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:37.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.824+0000 7fc7337fe640 1 -- 172.21.15.125:0/2587793289 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738004030 con 0x7fc73c159e60 2023-10-30T13:00:37.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.824+0000 7fc7337fe640 1 -- 172.21.15.125:0/2587793289 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7380ad030 con 0x7fc73c159e60 2023-10-30T13:00:37.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.824+0000 7fc7337fe640 1 -- 172.21.15.125:0/2587793289 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc738004410 con 0x7fc73c159e60 2023-10-30T13:00:37.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.824+0000 7fc742c9c640 1 -- 172.21.15.125:0/2587793289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc73c159e60 msgr2=0x7fc73c06f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:37.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.824+0000 7fc742c9c640 1 --2- 172.21.15.125:0/2587793289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc73c159e60 0x7fc73c06f540 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fc738006fd0 tx=0x7fc7380211c0 comp rx=0 tx=0).stop 2023-10-30T13:00:37.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.824+0000 7fc742c9c640 1 -- 172.21.15.125:0/2587793289 shutdown_connections 2023-10-30T13:00:37.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.824+0000 7fc742c9c640 1 --2- 172.21.15.125:0/2587793289 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc73c06a720 0x7fc73c06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:37.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.824+0000 7fc742c9c640 1 --2- 172.21.15.125:0/2587793289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc73c159e60 0x7fc73c06f540 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:37.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.824+0000 7fc742c9c640 1 -- 172.21.15.125:0/2587793289 >> 172.21.15.125:0/2587793289 conn(0x7fc73c0c7910 msgr2=0x7fc73c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:37.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.824+0000 7fc742c9c640 1 -- 172.21.15.125:0/2587793289 shutdown_connections 2023-10-30T13:00:37.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.824+0000 7fc742c9c640 1 -- 172.21.15.125:0/2587793289 wait complete. 2023-10-30T13:00:37.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.824+0000 7fc742c9c640 1 Processor -- start 2023-10-30T13:00:37.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.824+0000 7fc742c9c640 1 -- start start 2023-10-30T13:00:37.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc742c9c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc73c06a720 0x7fc73c153990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:37.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc742c9c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc73c0bfec0 0x7fc73c14ee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:37.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc742c9c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc73c159e60 0x7fc73c14f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:37.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc742c9c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c06a330 con 0x7fc73c0bfec0 2023-10-30T13:00:37.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc742c9c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c0c60d0 con 0x7fc73c06a720 2023-10-30T13:00:37.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc742c9c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73c0c5df0 con 0x7fc73c159e60 2023-10-30T13:00:37.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc733fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc73c0bfec0 0x7fc73c14ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:37.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc740a38640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc73c06a720 0x7fc73c153990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:37.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc733fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc73c0bfec0 0x7fc73c14ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35754/0 (socket says 172.21.15.125:35754) 2023-10-30T13:00:37.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc740a38640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc73c06a720 0x7fc73c153990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:53236/0 (socket says 172.21.15.125:53236) 2023-10-30T13:00:37.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc733fff640 1 -- 172.21.15.125:0/1092525355 learned_addr learned my addr 172.21.15.125:0/1092525355 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:37.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc741239640 1 --2- 172.21.15.125:0/1092525355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc73c159e60 0x7fc73c14f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:37.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc733fff640 1 -- 172.21.15.125:0/1092525355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc73c159e60 msgr2=0x7fc73c14f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:37.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc733fff640 1 --2- 172.21.15.125:0/1092525355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc73c159e60 0x7fc73c14f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:37.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc733fff640 1 -- 172.21.15.125:0/1092525355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc73c06a720 msgr2=0x7fc73c153990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:37.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc733fff640 1 --2- 172.21.15.125:0/1092525355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc73c06a720 0x7fc73c153990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:37.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc733fff640 1 -- 172.21.15.125:0/1092525355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc73800b780 con 0x7fc73c0bfec0 2023-10-30T13:00:37.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc733fff640 1 --2- 172.21.15.125:0/1092525355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc73c0bfec0 0x7fc73c14ee90 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00c500 tx=0x7fc72c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:37.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc7317fa640 1 -- 172.21.15.125:0/1092525355 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c005220 con 0x7fc73c0bfec0 2023-10-30T13:00:37.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc7317fa640 1 -- 172.21.15.125:0/1092525355 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc72c005380 con 0x7fc73c0bfec0 2023-10-30T13:00:37.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc742c9c640 1 -- 172.21.15.125:0/1092525355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc73c1564d0 con 0x7fc73c0bfec0 2023-10-30T13:00:37.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc7317fa640 1 -- 172.21.15.125:0/1092525355 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc72c007270 con 0x7fc73c0bfec0 2023-10-30T13:00:37.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc742c9c640 1 -- 172.21.15.125:0/1092525355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc73c1e0a20 con 0x7fc73c0bfec0 2023-10-30T13:00:37.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc742c9c640 1 -- 172.21.15.125:0/1092525355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc73c04e170 con 0x7fc73c0bfec0 2023-10-30T13:00:37.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.828+0000 7fc7317fa640 1 -- 172.21.15.125:0/1092525355 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc72c0073d0 con 0x7fc73c0bfec0 2023-10-30T13:00:37.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.832+0000 7fc7317fa640 1 --2- 172.21.15.125:0/1092525355 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7280664a0 0x7fc728068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:37.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.832+0000 7fc7317fa640 1 -- 172.21.15.125:0/1092525355 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7fc72c083c10 con 0x7fc73c0bfec0 2023-10-30T13:00:37.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.832+0000 7fc740a38640 1 --2- 172.21.15.125:0/1092525355 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7280664a0 0x7fc728068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:37.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.832+0000 7fc740a38640 1 --2- 172.21.15.125:0/1092525355 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7280664a0 0x7fc728068960 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc7380b4000 tx=0x7fc7380ac000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:37.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:37.836+0000 7fc7317fa640 1 -- 172.21.15.125:0/1092525355 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc72c050b30 con 0x7fc73c0bfec0 2023-10-30T13:00:38.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:38.016+0000 7fc742c9c640 1 -- 172.21.15.125:0/1092525355 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc73c073e10 con 0x7fc73c0bfec0 2023-10-30T13:00:38.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:38.016+0000 7fc7317fa640 1 -- 172.21.15.125:0/1092525355 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fc72c054150 con 0x7fc73c0bfec0 2023-10-30T13:00:38.021 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:38.022 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7707,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:44.626686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:active","state_seq":24,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:38.022 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 145 2023-10-30T13:00:38.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:38.020+0000 7fc742c9c640 1 -- 172.21.15.125:0/1092525355 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7280664a0 msgr2=0x7fc728068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:38.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:38.020+0000 7fc742c9c640 1 --2- 172.21.15.125:0/1092525355 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7280664a0 0x7fc728068960 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc7380b4000 tx=0x7fc7380ac000 comp rx=0 tx=0).stop 2023-10-30T13:00:38.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:38.020+0000 7fc742c9c640 1 -- 172.21.15.125:0/1092525355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc73c0bfec0 msgr2=0x7fc73c14ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:38.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:38.020+0000 7fc742c9c640 1 --2- 172.21.15.125:0/1092525355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc73c0bfec0 0x7fc73c14ee90 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fc72c00c500 tx=0x7fc72c00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:00:38.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:38.020+0000 7fc742c9c640 1 -- 172.21.15.125:0/1092525355 shutdown_connections 2023-10-30T13:00:38.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:38.020+0000 7fc742c9c640 1 --2- 172.21.15.125:0/1092525355 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc7280664a0 0x7fc728068960 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:38.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:38.020+0000 7fc742c9c640 1 --2- 172.21.15.125:0/1092525355 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc73c159e60 0x7fc73c14f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:38.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:38.020+0000 7fc742c9c640 1 --2- 172.21.15.125:0/1092525355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc73c0bfec0 0x7fc73c14ee90 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:38.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:38.020+0000 7fc742c9c640 1 --2- 172.21.15.125:0/1092525355 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc73c06a720 0x7fc73c153990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:38.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:38.020+0000 7fc742c9c640 1 -- 172.21.15.125:0/1092525355 >> 172.21.15.125:0/1092525355 conn(0x7fc73c0c7910 msgr2=0x7fc73c0c9bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:38.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:38.020+0000 7fc742c9c640 1 -- 172.21.15.125:0/1092525355 shutdown_connections 2023-10-30T13:00:38.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:38.020+0000 7fc742c9c640 1 -- 172.21.15.125:0/1092525355 wait complete. 2023-10-30T13:00:38.043 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:00:40.044 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:40.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.184+0000 7f26f569c640 1 Processor -- start 2023-10-30T13:00:40.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.184+0000 7f26f569c640 1 -- start start 2023-10-30T13:00:40.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26f569c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f26f01538a0 0x7f26f015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:40.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26f569c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26f00ca0e0 0x7f26f015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:40.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26f569c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f26f00cf160 0x7f26f00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:40.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26f569c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f00cb800 con 0x7f26f01538a0 2023-10-30T13:00:40.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26f569c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26f015dc30 con 0x7f26f00cf160 2023-10-30T13:00:40.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26f569c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00cb330 con 0x7f26f00ca0e0 2023-10-30T13:00:40.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26eeffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f26f01538a0 0x7f26f015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:40.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26eeffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f26f01538a0 0x7f26f015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35760/0 (socket says 172.21.15.125:35760) 2023-10-30T13:00:40.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26eeffd640 1 -- 172.21.15.125:0/3656484195 learned_addr learned my addr 172.21.15.125:0/3656484195 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:40.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26e67fc640 1 --2- 172.21.15.125:0/3656484195 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26f00ca0e0 0x7f26f015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:40.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26ef7fe640 1 --2- 172.21.15.125:0/3656484195 >> v2:172.21.15.136:3300/0 conn(0x7f26f00cf160 0x7f26f00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:40.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26eeffd640 1 -- 172.21.15.125:0/3656484195 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26f00ca0e0 msgr2=0x7f26f015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:40.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26eeffd640 1 --2- 172.21.15.125:0/3656484195 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26f00ca0e0 0x7f26f015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:40.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26eeffd640 1 -- 172.21.15.125:0/3656484195 >> v2:172.21.15.136:3300/0 conn(0x7f26f00cf160 msgr2=0x7f26f00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:40.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26eeffd640 1 --2- 172.21.15.125:0/3656484195 >> v2:172.21.15.136:3300/0 conn(0x7f26f00cf160 0x7f26f00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:40.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.188+0000 7f26eeffd640 1 -- 172.21.15.125:0/3656484195 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f00cad70 con 0x7f26f01538a0 2023-10-30T13:00:40.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26eeffd640 1 --2- 172.21.15.125:0/3656484195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f01538a0 0x7f26f015d6f0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f26f006cb00 tx=0x7f26e001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6236ed5dff596429 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:40.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26ee7fc640 1 -- 172.21.15.125:0/3656484195 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e00af400 con 0x7f26f01538a0 2023-10-30T13:00:40.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26ee7fc640 1 -- 172.21.15.125:0/3656484195 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e00ab050 con 0x7f26f01538a0 2023-10-30T13:00:40.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26ee7fc640 1 -- 172.21.15.125:0/3656484195 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e00af780 con 0x7f26f01538a0 2023-10-30T13:00:40.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26f569c640 1 -- 172.21.15.125:0/3656484195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f01538a0 msgr2=0x7f26f015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:40.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26f569c640 1 --2- 172.21.15.125:0/3656484195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f01538a0 0x7f26f015d6f0 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7f26f006cb00 tx=0x7f26e001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:00:40.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26f569c640 1 -- 172.21.15.125:0/3656484195 shutdown_connections 2023-10-30T13:00:40.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26f569c640 1 --2- 172.21.15.125:0/3656484195 >> v2:172.21.15.136:3300/0 conn(0x7f26f00cf160 0x7f26f00cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:40.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26f569c640 1 --2- 172.21.15.125:0/3656484195 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26f00ca0e0 0x7f26f015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:40.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26f569c640 1 --2- 172.21.15.125:0/3656484195 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f01538a0 0x7f26f015d6f0 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:40.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26f569c640 1 -- 172.21.15.125:0/3656484195 >> 172.21.15.125:0/3656484195 conn(0x7f26f006bfe0 msgr2=0x7f26f006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:40.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26f569c640 1 -- 172.21.15.125:0/3656484195 shutdown_connections 2023-10-30T13:00:40.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26f569c640 1 -- 172.21.15.125:0/3656484195 wait complete. 2023-10-30T13:00:40.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26f569c640 1 Processor -- start 2023-10-30T13:00:40.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.192+0000 7f26f569c640 1 -- start start 2023-10-30T13:00:40.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26f569c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26f00ca0e0 0x7f26f00ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:40.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26f569c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26f00cf160 0x7f26f0153c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:40.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26f569c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f01538a0 0x7f26f01549a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:40.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26f569c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00b3550 con 0x7f26f01538a0 2023-10-30T13:00:40.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26f569c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00b32b0 con 0x7f26f00cf160 2023-10-30T13:00:40.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26f569c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26f00b4240 con 0x7f26f00ca0e0 2023-10-30T13:00:40.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26ef7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f01538a0 0x7f26f01549a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:40.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26ef7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f01538a0 0x7f26f01549a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35770/0 (socket says 172.21.15.125:35770) 2023-10-30T13:00:40.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26ef7fe640 1 -- 172.21.15.125:0/2045184311 learned_addr learned my addr 172.21.15.125:0/2045184311 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:40.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26e67fc640 1 --2- 172.21.15.125:0/2045184311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26f00cf160 0x7f26f0153c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:40.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26eeffd640 1 --2- 172.21.15.125:0/2045184311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26f00ca0e0 0x7f26f00ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:40.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26ef7fe640 1 -- 172.21.15.125:0/2045184311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26f00ca0e0 msgr2=0x7f26f00ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:40.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26ef7fe640 1 --2- 172.21.15.125:0/2045184311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26f00ca0e0 0x7f26f00ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:40.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26ef7fe640 1 -- 172.21.15.125:0/2045184311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26f00cf160 msgr2=0x7f26f0153c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:40.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26ef7fe640 1 --2- 172.21.15.125:0/2045184311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26f00cf160 0x7f26f0153c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:40.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26ef7fe640 1 -- 172.21.15.125:0/2045184311 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e0009b30 con 0x7f26f01538a0 2023-10-30T13:00:40.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26ef7fe640 1 --2- 172.21.15.125:0/2045184311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f01538a0 0x7f26f01549a0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f26e800d580 tx=0x7f26e800da40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:40.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26e7fff640 1 -- 172.21.15.125:0/2045184311 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e80125f0 con 0x7f26f01538a0 2023-10-30T13:00:40.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26e7fff640 1 -- 172.21.15.125:0/2045184311 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26e8012750 con 0x7f26f01538a0 2023-10-30T13:00:40.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26e7fff640 1 -- 172.21.15.125:0/2045184311 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26e8012b00 con 0x7f26f01538a0 2023-10-30T13:00:40.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26f569c640 1 -- 172.21.15.125:0/2045184311 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26f00b4520 con 0x7f26f01538a0 2023-10-30T13:00:40.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26f569c640 1 -- 172.21.15.125:0/2045184311 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26f00b4ae0 con 0x7f26f01538a0 2023-10-30T13:00:40.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.196+0000 7f26f569c640 1 -- 172.21.15.125:0/2045184311 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26f00b2a50 con 0x7f26f01538a0 2023-10-30T13:00:40.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.200+0000 7f26e7fff640 1 -- 172.21.15.125:0/2045184311 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f26e8012c60 con 0x7f26f01538a0 2023-10-30T13:00:40.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.200+0000 7f26e7fff640 1 --2- 172.21.15.125:0/2045184311 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26d00664a0 0x7f26d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:40.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.200+0000 7f26e7fff640 1 -- 172.21.15.125:0/2045184311 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7114+0+0 (secure 0 0 0) 0x7f26e80847c0 con 0x7f26f01538a0 2023-10-30T13:00:40.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.200+0000 7f26eeffd640 1 --2- 172.21.15.125:0/2045184311 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26d00664a0 0x7f26d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:40.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.200+0000 7f26eeffd640 1 --2- 172.21.15.125:0/2045184311 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26d00664a0 0x7f26d0068960 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f26e00b1000 tx=0x7f26e00aa000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:40.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.204+0000 7f26e7fff640 1 -- 172.21.15.125:0/2045184311 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f26e80516e0 con 0x7f26f01538a0 2023-10-30T13:00:40.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.384+0000 7f26f569c640 1 -- 172.21.15.125:0/2045184311 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26f004e170 con 0x7f26f01538a0 2023-10-30T13:00:40.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.388+0000 7f26e7fff640 1 -- 172.21.15.125:0/2045184311 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 145 v145) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f26e8054d00 con 0x7f26f01538a0 2023-10-30T13:00:40.391 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:40.391 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":145,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7707,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":145}],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T12:59:44.626686+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7554},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7554":{"gid":7554,"name":"e","rank":0,"incarnation":140,"state":"up:active","state_seq":24,"addr":"172.21.15.125:6836/1944683138","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1944683138},{"type":"v1","addr":"172.21.15.125:6836","nonce":1944683138}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:40.391 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 145 2023-10-30T13:00:40.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.388+0000 7f26f569c640 1 -- 172.21.15.125:0/2045184311 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26d00664a0 msgr2=0x7f26d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:40.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.388+0000 7f26f569c640 1 --2- 172.21.15.125:0/2045184311 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26d00664a0 0x7f26d0068960 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f26e00b1000 tx=0x7f26e00aa000 comp rx=0 tx=0).stop 2023-10-30T13:00:40.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.388+0000 7f26f569c640 1 -- 172.21.15.125:0/2045184311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f01538a0 msgr2=0x7f26f01549a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:40.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.388+0000 7f26f569c640 1 --2- 172.21.15.125:0/2045184311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f01538a0 0x7f26f01549a0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f26e800d580 tx=0x7f26e800da40 comp rx=0 tx=0).stop 2023-10-30T13:00:40.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.388+0000 7f26f569c640 1 -- 172.21.15.125:0/2045184311 shutdown_connections 2023-10-30T13:00:40.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.388+0000 7f26f569c640 1 --2- 172.21.15.125:0/2045184311 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26d00664a0 0x7f26d0068960 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:40.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.388+0000 7f26f569c640 1 --2- 172.21.15.125:0/2045184311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26f01538a0 0x7f26f01549a0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:40.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.388+0000 7f26f569c640 1 --2- 172.21.15.125:0/2045184311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26f00cf160 0x7f26f0153c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:40.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.388+0000 7f26f569c640 1 --2- 172.21.15.125:0/2045184311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26f00ca0e0 0x7f26f00ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:40.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.388+0000 7f26f569c640 1 -- 172.21.15.125:0/2045184311 >> 172.21.15.125:0/2045184311 conn(0x7f26f006bfe0 msgr2=0x7f26f00cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:40.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.388+0000 7f26f569c640 1 -- 172.21.15.125:0/2045184311 shutdown_connections 2023-10-30T13:00:40.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:40.388+0000 7f26f569c640 1 -- 172.21.15.125:0/2045184311 wait complete. 2023-10-30T13:00:40.411 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:00:42.413 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:42.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb5a1a68640 1 Processor -- start 2023-10-30T13:00:42.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb5a1a68640 1 -- start start 2023-10-30T13:00:42.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb5a1a68640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb59c0c05d0 0x7fb59c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:42.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb5a1a68640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c0c0a50 con 0x7fb59c0c6b80 2023-10-30T13:00:42.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb5a1a68640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59c06f6e0 con 0x7fb59c0c6f60 2023-10-30T13:00:42.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb5a1a68640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c06f9c0 con 0x7fb59c0c05d0 2023-10-30T13:00:42.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59b7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb59c0c6b80 0x7fb59c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55614/0 (socket says 172.21.15.125:55614) 2023-10-30T13:00:42.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59b7fe640 1 -- 172.21.15.125:0/3337885631 learned_addr learned my addr 172.21.15.125:0/3337885631 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:00:42.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59affd640 1 --2- 172.21.15.125:0/3337885631 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb59c0c05d0 0x7fb59c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:42.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 -- 172.21.15.125:0/3337885631 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036010431 0 0) 0x7fb59c0c0a50 con 0x7fb59c0c6b80 2023-10-30T13:00:42.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 -- 172.21.15.125:0/3337885631 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb584003530 con 0x7fb59c0c6b80 2023-10-30T13:00:42.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 -- 172.21.15.125:0/3337885631 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3396077516 0 0) 0x7fb584003530 con 0x7fb59c0c6b80 2023-10-30T13:00:42.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 -- 172.21.15.125:0/3337885631 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb59c0c05d0 msgr2=0x7fb59c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:42.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 --2- 172.21.15.125:0/3337885631 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb59c0c05d0 0x7fb59c13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:42.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 -- 172.21.15.125:0/3337885631 >> v1:172.21.15.136:6789/0 conn(0x7fb59c0c6f60 legacy=0x7fb59c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:42.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 -- 172.21.15.125:0/3337885631 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c06fca0 con 0x7fb59c0c6b80 2023-10-30T13:00:42.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 -- 172.21.15.125:0/3337885631 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fb58c001680 con 0x7fb59c0c6b80 2023-10-30T13:00:42.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 -- 172.21.15.125:0/3337885631 >> v1:172.21.15.125:6789/0 conn(0x7fb59c0c6b80 legacy=0x7fb59c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:42.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 --2- 172.21.15.125:0/3337885631 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb59c0c05d0 0x7fb5840039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:42.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 --2- 172.21.15.125:0/3337885631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb584003f30 0x7fb5840047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:42.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 --2- 172.21.15.125:0/3337885631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb584004d10 0x7fb584005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:42.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 -- 172.21.15.125:0/3337885631 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c06f9c0 con 0x7fb584003f30 2023-10-30T13:00:42.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 -- 172.21.15.125:0/3337885631 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58400ba70 con 0x7fb584004d10 2023-10-30T13:00:42.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.556+0000 7fb59a7fc640 1 -- 172.21.15.125:0/3337885631 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb58400bc30 con 0x7fb59c0c05d0 2023-10-30T13:00:42.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb59affd640 1 --2- 172.21.15.125:0/3337885631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb584003f30 0x7fb5840047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:42.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb59b7fe640 1 --2- 172.21.15.125:0/3337885631 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb59c0c05d0 0x7fb5840039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:42.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb59bfff640 1 --2- 172.21.15.125:0/3337885631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb584004d10 0x7fb584005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:42.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb59affd640 1 -- 172.21.15.125:0/3337885631 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb59c0c05d0 msgr2=0x7fb5840039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:42.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb59affd640 1 --2- 172.21.15.125:0/3337885631 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb59c0c05d0 0x7fb5840039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:42.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb59affd640 1 -- 172.21.15.125:0/3337885631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb584004d10 msgr2=0x7fb584005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:42.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb59affd640 1 --2- 172.21.15.125:0/3337885631 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb584004d10 0x7fb584005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:42.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb59affd640 1 -- 172.21.15.125:0/3337885631 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb584003530 con 0x7fb584003f30 2023-10-30T13:00:42.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb59affd640 1 --2- 172.21.15.125:0/3337885631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb584003f30 0x7fb5840047d0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fb590009ae0 tx=0x7fb59001ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a392061781567e44 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:42.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb59a7fc640 1 -- 172.21.15.125:0/3337885631 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5900a79e0 con 0x7fb584003f30 2023-10-30T13:00:42.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb59a7fc640 1 -- 172.21.15.125:0/3337885631 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5900a7b40 con 0x7fb584003f30 2023-10-30T13:00:42.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb59a7fc640 1 -- 172.21.15.125:0/3337885631 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5900af440 con 0x7fb584003f30 2023-10-30T13:00:42.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb5a1a68640 1 -- 172.21.15.125:0/3337885631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb584003f30 msgr2=0x7fb5840047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:42.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb5a1a68640 1 --2- 172.21.15.125:0/3337885631 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb584003f30 0x7fb5840047d0 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fb590009ae0 tx=0x7fb59001ece0 comp rx=0 tx=0).stop 2023-10-30T13:00:42.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.560+0000 7fb59bfff640 1 -- 172.21.15.125:0/3337885631 reap_dead start 2023-10-30T13:00:42.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb5a1a68640 1 -- 172.21.15.125:0/3337885631 shutdown_connections 2023-10-30T13:00:42.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb5a1a68640 1 -- 172.21.15.125:0/3337885631 >> 172.21.15.125:0/3337885631 conn(0x7fb59c0c7910 msgr2=0x7fb59c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:42.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb5a1a68640 1 -- 172.21.15.125:0/3337885631 shutdown_connections 2023-10-30T13:00:42.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb5a1a68640 1 -- 172.21.15.125:0/3337885631 wait complete. 2023-10-30T13:00:42.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb5a1a68640 1 Processor -- start 2023-10-30T13:00:42.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb5a1a68640 1 -- start start 2023-10-30T13:00:42.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb5a1a68640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb584003f30 0x7fb59c1ddb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:42.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb5a1a68640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb59c0ca430 0x7fb59c1dc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:42.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb5a1a68640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb59c0c38b0 0x7fb59c1edd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:42.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb5a1a68640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c0c2ec0 con 0x7fb584003f30 2023-10-30T13:00:42.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb5a1a68640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c1de300 con 0x7fb59c0c38b0 2023-10-30T13:00:42.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb5a1a68640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c0cabe0 con 0x7fb59c0ca430 2023-10-30T13:00:42.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb59b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb584003f30 0x7fb59c1ddb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:42.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb59b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb584003f30 0x7fb59c1ddb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34498/0 (socket says 172.21.15.125:34498) 2023-10-30T13:00:42.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb59b7fe640 1 -- 172.21.15.125:0/323660468 learned_addr learned my addr 172.21.15.125:0/323660468 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:42.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb59affd640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb59c0ca430 0x7fb59c1dc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:42.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb59bfff640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb59c0c38b0 0x7fb59c1edd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:42.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb59b7fe640 1 -- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb59c0ca430 msgr2=0x7fb59c1dc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:42.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb59b7fe640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb59c0ca430 0x7fb59c1dc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:42.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb59b7fe640 1 -- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb59c0c38b0 msgr2=0x7fb59c1edd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:42.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb59b7fe640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb59c0c38b0 0x7fb59c1edd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:42.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb59b7fe640 1 -- 172.21.15.125:0/323660468 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c13e280 con 0x7fb584003f30 2023-10-30T13:00:42.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb59bfff640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb59c0c38b0 0x7fb59c1edd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:00:42.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb59b7fe640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb584003f30 0x7fb59c1ddb70 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fb58c004880 tx=0x7fb58c012c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:42.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.564+0000 7fb583fff640 1 -- 172.21.15.125:0/323660468 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c010960 con 0x7fb584003f30 2023-10-30T13:00:42.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.568+0000 7fb583fff640 1 -- 172.21.15.125:0/323660468 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58c010ac0 con 0x7fb584003f30 2023-10-30T13:00:42.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.568+0000 7fb5a1a68640 1 -- 172.21.15.125:0/323660468 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb59c0c22b0 con 0x7fb584003f30 2023-10-30T13:00:42.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.568+0000 7fb583fff640 1 -- 172.21.15.125:0/323660468 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58c010e70 con 0x7fb584003f30 2023-10-30T13:00:42.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.568+0000 7fb5a1a68640 1 -- 172.21.15.125:0/323660468 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb59c0c2870 con 0x7fb584003f30 2023-10-30T13:00:42.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.568+0000 7fb5a1a68640 1 -- 172.21.15.125:0/323660468 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb59c04e170 con 0x7fb584003f30 2023-10-30T13:00:42.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.568+0000 7fb583fff640 1 -- 172.21.15.125:0/323660468 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb58c016480 con 0x7fb584003f30 2023-10-30T13:00:42.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.568+0000 7fb583fff640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5880664a0 0x7fb588068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:42.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.568+0000 7fb583fff640 1 -- 172.21.15.125:0/323660468 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7fb58c087910 con 0x7fb584003f30 2023-10-30T13:00:42.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.568+0000 7fb59affd640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5880664a0 0x7fb588068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:42.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.572+0000 7fb59affd640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5880664a0 0x7fb588068960 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb5900b1000 tx=0x7fb5900046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:42.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.572+0000 7fb583fff640 1 -- 172.21.15.125:0/323660468 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb58c0546b0 con 0x7fb584003f30 2023-10-30T13:00:42.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.752+0000 7fb5a1a68640 1 -- 172.21.15.125:0/323660468 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb59c0c81a0 con 0x7fb584003f30 2023-10-30T13:00:42.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.756+0000 7fb583fff640 1 -- 172.21.15.125:0/323660468 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+5723 (secure 0 0 0) 0x7fb58c057cd0 con 0x7fb584003f30 2023-10-30T13:00:42.759 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:42.759 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:00:40.486046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:42.760 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 146 2023-10-30T13:00:42.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.756+0000 7fb5a1a68640 1 -- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5880664a0 msgr2=0x7fb588068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:42.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.756+0000 7fb5a1a68640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5880664a0 0x7fb588068960 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fb5900b1000 tx=0x7fb5900046d0 comp rx=0 tx=0).stop 2023-10-30T13:00:42.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.756+0000 7fb5a1a68640 1 -- 172.21.15.125:0/323660468 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb584003f30 msgr2=0x7fb59c1ddb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:42.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.756+0000 7fb5a1a68640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb584003f30 0x7fb59c1ddb70 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7fb58c004880 tx=0x7fb58c012c90 comp rx=0 tx=0).stop 2023-10-30T13:00:42.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.756+0000 7fb5a1a68640 1 -- 172.21.15.125:0/323660468 shutdown_connections 2023-10-30T13:00:42.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.756+0000 7fb5a1a68640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5880664a0 0x7fb588068960 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:42.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.756+0000 7fb5a1a68640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb59c0c38b0 0x7fb59c1edd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:42.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.756+0000 7fb5a1a68640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb59c0ca430 0x7fb59c1dc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:42.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.756+0000 7fb5a1a68640 1 --2- 172.21.15.125:0/323660468 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb584003f30 0x7fb59c1ddb70 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:42.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.756+0000 7fb5a1a68640 1 -- 172.21.15.125:0/323660468 >> 172.21.15.125:0/323660468 conn(0x7fb59c0c7910 msgr2=0x7fb59c0c19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:42.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.756+0000 7fb5a1a68640 1 -- 172.21.15.125:0/323660468 shutdown_connections 2023-10-30T13:00:42.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.756+0000 7fb5a1a68640 1 -- 172.21.15.125:0/323660468 wait complete. 2023-10-30T13:00:42.780 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T13:00:42.780 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:00:42.781 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:42.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.916+0000 7f4af5772640 1 Processor -- start 2023-10-30T13:00:42.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.916+0000 7f4af5772640 1 -- start start 2023-10-30T13:00:42.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.916+0000 7f4af5772640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4af00c90d0 0x7f4af00cef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:42.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.916+0000 7f4af5772640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af015ce30 con 0x7f4af00cf4d0 2023-10-30T13:00:42.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.916+0000 7f4af5772640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af0159880 con 0x7f4af00c8cf0 2023-10-30T13:00:42.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.916+0000 7f4af5772640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af0159ae0 con 0x7f4af00c90d0 2023-10-30T13:00:42.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.916+0000 7f4aee7fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4af00c90d0 0x7f4af00cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:42.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.916+0000 7f4aef7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4af00cf4d0 0x7f4af015d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55622/0 (socket says 172.21.15.125:55622) 2023-10-30T13:00:42.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.916+0000 7f4aef7fe640 1 -- 172.21.15.125:0/3885453020 learned_addr learned my addr 172.21.15.125:0/3885453020 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:00:42.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.916+0000 7f4aee7fc640 1 -- 172.21.15.125:0/3885453020 >> v1:172.21.15.136:6789/0 conn(0x7f4af00c8cf0 legacy=0x7f4af00c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:42.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.916+0000 7f4aee7fc640 1 -- 172.21.15.125:0/3885453020 >> v1:172.21.15.125:6789/0 conn(0x7f4af00cf4d0 legacy=0x7f4af015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:42.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.916+0000 7f4aee7fc640 1 -- 172.21.15.125:0/3885453020 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af0159dc0 con 0x7f4af00c90d0 2023-10-30T13:00:42.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.920+0000 7f4aee7fc640 1 --2- 172.21.15.125:0/3885453020 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4af00c90d0 0x7f4af00cef90 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40097c0 tx=0x7f4ae401c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b48e714f15451c6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:42.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.920+0000 7f4aedffb640 1 -- 172.21.15.125:0/3885453020 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae40ad660 con 0x7f4af00c90d0 2023-10-30T13:00:42.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.920+0000 7f4aedffb640 1 -- 172.21.15.125:0/3885453020 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae40ad7c0 con 0x7f4af00c90d0 2023-10-30T13:00:42.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.920+0000 7f4aedffb640 1 -- 172.21.15.125:0/3885453020 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae40adb50 con 0x7f4af00c90d0 2023-10-30T13:00:42.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.920+0000 7f4af5772640 1 -- 172.21.15.125:0/3885453020 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4af00c90d0 msgr2=0x7f4af00cef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:42.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.920+0000 7f4af5772640 1 --2- 172.21.15.125:0/3885453020 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4af00c90d0 0x7f4af00cef90 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40097c0 tx=0x7f4ae401c450 comp rx=0 tx=0).stop 2023-10-30T13:00:42.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.920+0000 7f4af5772640 1 -- 172.21.15.125:0/3885453020 shutdown_connections 2023-10-30T13:00:42.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.920+0000 7f4af5772640 1 --2- 172.21.15.125:0/3885453020 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4af00c90d0 0x7f4af00cef90 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:42.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.920+0000 7f4af5772640 1 -- 172.21.15.125:0/3885453020 >> 172.21.15.125:0/3885453020 conn(0x7f4af006be40 msgr2=0x7f4af006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:42.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.920+0000 7f4af5772640 1 -- 172.21.15.125:0/3885453020 shutdown_connections 2023-10-30T13:00:42.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.920+0000 7f4af5772640 1 -- 172.21.15.125:0/3885453020 wait complete. 2023-10-30T13:00:42.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.920+0000 7f4af5772640 1 Processor -- start 2023-10-30T13:00:42.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4af5772640 1 -- start start 2023-10-30T13:00:42.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4af5772640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4af00c8cf0 0x7f4af01f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:42.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4af5772640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4af00c90d0 0x7f4af0150060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:42.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4af5772640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4af00cf4d0 0x7f4af015c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:42.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4af5772640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af015ca20 con 0x7f4af00c8cf0 2023-10-30T13:00:42.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4af5772640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af0151f90 con 0x7f4af00c90d0 2023-10-30T13:00:42.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4af5772640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af015b5b0 con 0x7f4af00cf4d0 2023-10-30T13:00:42.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4aeeffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4af00c8cf0 0x7f4af01f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:42.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4aeeffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4af00c8cf0 0x7f4af01f2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34516/0 (socket says 172.21.15.125:34516) 2023-10-30T13:00:42.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4aeeffd640 1 -- 172.21.15.125:0/1587218776 learned_addr learned my addr 172.21.15.125:0/1587218776 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:42.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4aef7fe640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4af00cf4d0 0x7f4af015c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:42.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4aee7fc640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4af00c90d0 0x7f4af0150060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:42.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4aeeffd640 1 -- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4af00cf4d0 msgr2=0x7f4af015c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:42.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4aeeffd640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4af00cf4d0 0x7f4af015c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:42.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4aeeffd640 1 -- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4af00c90d0 msgr2=0x7f4af0150060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:42.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4aeeffd640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4af00c90d0 0x7f4af0150060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:42.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4aeeffd640 1 -- 172.21.15.125:0/1587218776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae4009420 con 0x7f4af00c8cf0 2023-10-30T13:00:42.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4aef7fe640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4af00cf4d0 0x7f4af015c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:00:42.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4aee7fc640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4af00c90d0 0x7f4af0150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:00:42.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4aeeffd640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4af00c8cf0 0x7f4af01f2f40 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f4ae000c3c0 tx=0x7f4ae000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:42.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4ad77fe640 1 -- 172.21.15.125:0/1587218776 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae00052d0 con 0x7f4af00c8cf0 2023-10-30T13:00:42.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4ad77fe640 1 -- 172.21.15.125:0/1587218776 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae0005430 con 0x7f4af00c8cf0 2023-10-30T13:00:42.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4af5772640 1 -- 172.21.15.125:0/1587218776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af0150cd0 con 0x7f4af00c8cf0 2023-10-30T13:00:42.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4ad77fe640 1 -- 172.21.15.125:0/1587218776 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae0003840 con 0x7f4af00c8cf0 2023-10-30T13:00:42.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4af5772640 1 -- 172.21.15.125:0/1587218776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af01509f0 con 0x7f4af00c8cf0 2023-10-30T13:00:42.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.924+0000 7f4af5772640 1 -- 172.21.15.125:0/1587218776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af015b410 con 0x7f4af00c8cf0 2023-10-30T13:00:42.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.928+0000 7f4ad77fe640 1 -- 172.21.15.125:0/1587218776 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4ae00039a0 con 0x7f4af00c8cf0 2023-10-30T13:00:42.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.928+0000 7f4ad77fe640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4adc0664a0 0x7f4adc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:42.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.928+0000 7f4ad77fe640 1 -- 172.21.15.125:0/1587218776 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7f4ae0083a30 con 0x7f4af00c8cf0 2023-10-30T13:00:42.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.928+0000 7f4aee7fc640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4adc0664a0 0x7f4adc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:42.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.928+0000 7f4aee7fc640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4adc0664a0 0x7f4adc068960 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40af000 tx=0x7f4ae4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:42.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:42.932+0000 7f4ad77fe640 1 -- 172.21.15.125:0/1587218776 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4af015b410 con 0x7f4af00c8cf0 2023-10-30T13:00:43.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:43.112+0000 7f4af5772640 1 -- 172.21.15.125:0/1587218776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4af015b410 con 0x7f4af00c8cf0 2023-10-30T13:00:43.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:43.112+0000 7f4ad77fe640 1 -- 172.21.15.125:0/1587218776 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f4af015b410 con 0x7f4af00c8cf0 2023-10-30T13:00:43.118 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:43.118 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:00:40.486046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:43.118 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 146 2023-10-30T13:00:43.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:43.116+0000 7f4af5772640 1 -- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4adc0664a0 msgr2=0x7f4adc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:43.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:43.116+0000 7f4af5772640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4adc0664a0 0x7f4adc068960 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40af000 tx=0x7f4ae4008040 comp rx=0 tx=0).stop 2023-10-30T13:00:43.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:43.116+0000 7f4af5772640 1 -- 172.21.15.125:0/1587218776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4af00c8cf0 msgr2=0x7f4af01f2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:43.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:43.116+0000 7f4af5772640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4af00c8cf0 0x7f4af01f2f40 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f4ae000c3c0 tx=0x7f4ae000c880 comp rx=0 tx=0).stop 2023-10-30T13:00:43.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:43.116+0000 7f4af5772640 1 -- 172.21.15.125:0/1587218776 shutdown_connections 2023-10-30T13:00:43.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:43.116+0000 7f4af5772640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4adc0664a0 0x7f4adc068960 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:43.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:43.116+0000 7f4af5772640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4af00cf4d0 0x7f4af015c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:43.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:43.116+0000 7f4af5772640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4af00c90d0 0x7f4af0150060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:43.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:43.116+0000 7f4af5772640 1 --2- 172.21.15.125:0/1587218776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4af00c8cf0 0x7f4af01f2f40 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:43.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:43.116+0000 7f4af5772640 1 -- 172.21.15.125:0/1587218776 >> 172.21.15.125:0/1587218776 conn(0x7f4af006be40 msgr2=0x7f4af00b2b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:43.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:43.116+0000 7f4af5772640 1 -- 172.21.15.125:0/1587218776 shutdown_connections 2023-10-30T13:00:43.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:43.116+0000 7f4af5772640 1 -- 172.21.15.125:0/1587218776 wait complete. 2023-10-30T13:00:43.139 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 146, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 146, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7707": { "addr": "172.21.15.125:6837/3762807620", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 3762807620, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 3762807620, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7707, "incarnation": 146, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:00:40.486046+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7707 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:00:45.140 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:45.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c7db01640 1 Processor -- start 2023-10-30T13:00:45.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c7db01640 1 -- start start 2023-10-30T13:00:45.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c7db01640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2c78159e60 0x7f2c7806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:45.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c7db01640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2c7806a720 0x7f2c7806ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:45.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c7db01640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7806fa80 con 0x7f2c780bfec0 2023-10-30T13:00:45.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c7db01640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c78071290 con 0x7f2c78159e60 2023-10-30T13:00:45.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c7db01640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c78071570 con 0x7f2c7806a720 2023-10-30T13:00:45.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c777fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2c78159e60 0x7f2c7806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:45.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c777fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2c78159e60 0x7f2c7806f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34518/0 (socket says 172.21.15.125:34518) 2023-10-30T13:00:45.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c777fe640 1 -- 172.21.15.125:0/2082097793 learned_addr learned my addr 172.21.15.125:0/2082097793 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:45.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c77fff640 1 --2- 172.21.15.125:0/2082097793 >> v2:172.21.15.136:3300/0 conn(0x7f2c7806a720 0x7f2c7806ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:45.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c777fe640 1 -- 172.21.15.125:0/2082097793 >> v2:172.21.15.136:3300/0 conn(0x7f2c7806a720 msgr2=0x7f2c7806ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:45.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c777fe640 1 --2- 172.21.15.125:0/2082097793 >> v2:172.21.15.136:3300/0 conn(0x7f2c7806a720 0x7f2c7806ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:45.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c777fe640 1 -- 172.21.15.125:0/2082097793 >> v1:172.21.15.136:6789/0 conn(0x7f2c780bfec0 legacy=0x7f2c780c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:45.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.284+0000 7f2c777fe640 1 -- 172.21.15.125:0/2082097793 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c78071850 con 0x7f2c78159e60 2023-10-30T13:00:45.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.288+0000 7f2c777fe640 1 --2- 172.21.15.125:0/2082097793 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c78159e60 0x7f2c7806f540 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f2c7806f4d0 tx=0x7f2c6801fe00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc8867d2ccb3629 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:45.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.288+0000 7f2c76ffd640 1 -- 172.21.15.125:0/2082097793 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c680a9b00 con 0x7f2c78159e60 2023-10-30T13:00:45.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.288+0000 7f2c76ffd640 1 -- 172.21.15.125:0/2082097793 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c680a9c60 con 0x7f2c78159e60 2023-10-30T13:00:45.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.288+0000 7f2c76ffd640 1 -- 172.21.15.125:0/2082097793 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c680b1440 con 0x7f2c78159e60 2023-10-30T13:00:45.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.288+0000 7f2c7db01640 1 -- 172.21.15.125:0/2082097793 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c78159e60 msgr2=0x7f2c7806f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:45.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.288+0000 7f2c7db01640 1 --2- 172.21.15.125:0/2082097793 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c78159e60 0x7f2c7806f540 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f2c7806f4d0 tx=0x7f2c6801fe00 comp rx=0 tx=0).stop 2023-10-30T13:00:45.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.288+0000 7f2c7db01640 1 -- 172.21.15.125:0/2082097793 shutdown_connections 2023-10-30T13:00:45.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.288+0000 7f2c7db01640 1 --2- 172.21.15.125:0/2082097793 >> v2:172.21.15.136:3300/0 conn(0x7f2c7806a720 0x7f2c7806ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:45.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.288+0000 7f2c7db01640 1 --2- 172.21.15.125:0/2082097793 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c78159e60 0x7f2c7806f540 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:45.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.288+0000 7f2c7db01640 1 -- 172.21.15.125:0/2082097793 >> 172.21.15.125:0/2082097793 conn(0x7f2c780c7910 msgr2=0x7f2c780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:45.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.288+0000 7f2c7db01640 1 -- 172.21.15.125:0/2082097793 shutdown_connections 2023-10-30T13:00:45.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.288+0000 7f2c7db01640 1 -- 172.21.15.125:0/2082097793 wait complete. 2023-10-30T13:00:45.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c7db01640 1 Processor -- start 2023-10-30T13:00:45.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c7db01640 1 -- start start 2023-10-30T13:00:45.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c7db01640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c7806a720 0x7f2c780c23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:45.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c7db01640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c780bfec0 0x7f2c781f0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:45.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c7db01640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c78159e60 0x7f2c781ede30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:45.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c7db01640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c781ee370 con 0x7f2c78159e60 2023-10-30T13:00:45.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c7db01640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c780c2a20 con 0x7f2c7806a720 2023-10-30T13:00:45.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c7db01640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c780c2d00 con 0x7f2c780bfec0 2023-10-30T13:00:45.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c77fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c78159e60 0x7f2c781ede30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:45.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c77fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c78159e60 0x7f2c781ede30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34522/0 (socket says 172.21.15.125:34522) 2023-10-30T13:00:45.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c77fff640 1 -- 172.21.15.125:0/2454269059 learned_addr learned my addr 172.21.15.125:0/2454269059 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:45.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c777fe640 1 --2- 172.21.15.125:0/2454269059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c7806a720 0x7f2c780c23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:45.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c6ffff640 1 --2- 172.21.15.125:0/2454269059 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c780bfec0 0x7f2c781f0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:45.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c77fff640 1 -- 172.21.15.125:0/2454269059 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c780bfec0 msgr2=0x7f2c781f0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:45.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c77fff640 1 --2- 172.21.15.125:0/2454269059 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c780bfec0 0x7f2c781f0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:45.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c77fff640 1 -- 172.21.15.125:0/2454269059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c7806a720 msgr2=0x7f2c780c23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:45.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c77fff640 1 --2- 172.21.15.125:0/2454269059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c7806a720 0x7f2c780c23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:45.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c77fff640 1 -- 172.21.15.125:0/2454269059 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c680094d0 con 0x7f2c78159e60 2023-10-30T13:00:45.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c77fff640 1 --2- 172.21.15.125:0/2454269059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c78159e60 0x7f2c781ede30 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f2c70007e40 tx=0x7f2c7000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:45.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c74ff9640 1 -- 172.21.15.125:0/2454269059 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c70014620 con 0x7f2c78159e60 2023-10-30T13:00:45.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c74ff9640 1 -- 172.21.15.125:0/2454269059 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7000ee90 con 0x7f2c78159e60 2023-10-30T13:00:45.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c7db01640 1 -- 172.21.15.125:0/2454269059 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c781ecb60 con 0x7f2c78159e60 2023-10-30T13:00:45.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c74ff9640 1 -- 172.21.15.125:0/2454269059 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c700149e0 con 0x7f2c78159e60 2023-10-30T13:00:45.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.292+0000 7f2c7db01640 1 -- 172.21.15.125:0/2454269059 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c781ed120 con 0x7f2c78159e60 2023-10-30T13:00:45.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.296+0000 7f2c7db01640 1 -- 172.21.15.125:0/2454269059 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c781ec9c0 con 0x7f2c78159e60 2023-10-30T13:00:45.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.296+0000 7f2c74ff9640 1 -- 172.21.15.125:0/2454269059 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c70014b40 con 0x7f2c78159e60 2023-10-30T13:00:45.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.296+0000 7f2c74ff9640 1 --2- 172.21.15.125:0/2454269059 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c580664a0 0x7f2c58068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:45.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.296+0000 7f2c74ff9640 1 -- 172.21.15.125:0/2454269059 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7f2c70084e80 con 0x7f2c78159e60 2023-10-30T13:00:45.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.296+0000 7f2c777fe640 1 --2- 172.21.15.125:0/2454269059 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c580664a0 0x7f2c58068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:45.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.296+0000 7f2c777fe640 1 --2- 172.21.15.125:0/2454269059 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c580664a0 0x7f2c58068960 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2c680b3000 tx=0x7f2c680ab070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:45.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.300+0000 7f2c74ff9640 1 -- 172.21.15.125:0/2454269059 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c781ec9c0 con 0x7f2c78159e60 2023-10-30T13:00:45.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.480+0000 7f2c7db01640 1 -- 172.21.15.125:0/2454269059 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c781ec9c0 con 0x7f2c78159e60 2023-10-30T13:00:45.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.480+0000 7f2c74ff9640 1 -- 172.21.15.125:0/2454269059 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f2c781ec9c0 con 0x7f2c78159e60 2023-10-30T13:00:45.485 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:45.485 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:00:40.486046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:45.486 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 146 2023-10-30T13:00:45.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.480+0000 7f2c7db01640 1 -- 172.21.15.125:0/2454269059 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c580664a0 msgr2=0x7f2c58068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:45.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.480+0000 7f2c7db01640 1 --2- 172.21.15.125:0/2454269059 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c580664a0 0x7f2c58068960 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2c680b3000 tx=0x7f2c680ab070 comp rx=0 tx=0).stop 2023-10-30T13:00:45.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.484+0000 7f2c7db01640 1 -- 172.21.15.125:0/2454269059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c78159e60 msgr2=0x7f2c781ede30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:45.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.484+0000 7f2c7db01640 1 --2- 172.21.15.125:0/2454269059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c78159e60 0x7f2c781ede30 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f2c70007e40 tx=0x7f2c7000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:00:45.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.484+0000 7f2c7db01640 1 -- 172.21.15.125:0/2454269059 shutdown_connections 2023-10-30T13:00:45.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.484+0000 7f2c7db01640 1 --2- 172.21.15.125:0/2454269059 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c580664a0 0x7f2c58068960 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:45.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.484+0000 7f2c7db01640 1 --2- 172.21.15.125:0/2454269059 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c78159e60 0x7f2c781ede30 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:45.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.484+0000 7f2c7db01640 1 --2- 172.21.15.125:0/2454269059 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c780bfec0 0x7f2c781f0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:45.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.484+0000 7f2c7db01640 1 --2- 172.21.15.125:0/2454269059 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c7806a720 0x7f2c780c23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:45.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.484+0000 7f2c7db01640 1 -- 172.21.15.125:0/2454269059 >> 172.21.15.125:0/2454269059 conn(0x7f2c780c7910 msgr2=0x7f2c780c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:45.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.484+0000 7f2c7db01640 1 -- 172.21.15.125:0/2454269059 shutdown_connections 2023-10-30T13:00:45.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:45.484+0000 7f2c7db01640 1 -- 172.21.15.125:0/2454269059 wait complete. 2023-10-30T13:00:45.508 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:00:47.509 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:47.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd43c4640 1 Processor -- start 2023-10-30T13:00:47.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd43c4640 1 -- start start 2023-10-30T13:00:47.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd43c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fcc15d6f0 0x7f6fcc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:47.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd43c4640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fcc0cb800 con 0x7f6fcc0cf160 2023-10-30T13:00:47.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd43c4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fcc0cb330 con 0x7f6fcc1538a0 2023-10-30T13:00:47.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd43c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc0cad70 con 0x7f6fcc15d6f0 2023-10-30T13:00:47.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd2961640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6fcc0cf160 0x7f6fcc164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55624/0 (socket says 172.21.15.125:55624) 2023-10-30T13:00:47.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd2961640 1 -- 172.21.15.125:0/2549617314 learned_addr learned my addr 172.21.15.125:0/2549617314 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:00:47.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd195f640 1 --2- 172.21.15.125:0/2549617314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fcc15d6f0 0x7f6fcc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:47.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd115e640 1 -- 172.21.15.125:0/2549617314 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185070647 0 0) 0x7f6fcc0cb800 con 0x7f6fcc0cf160 2023-10-30T13:00:47.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd115e640 1 -- 172.21.15.125:0/2549617314 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fb8003530 con 0x7f6fcc0cf160 2023-10-30T13:00:47.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd115e640 1 -- 172.21.15.125:0/2549617314 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 124656019 0 0) 0x7f6fcc0cb330 con 0x7f6fcc1538a0 2023-10-30T13:00:47.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd115e640 1 -- 172.21.15.125:0/2549617314 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fcc0cb800 con 0x7f6fcc1538a0 2023-10-30T13:00:47.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd195f640 1 -- 172.21.15.125:0/2549617314 >> v1:172.21.15.136:6789/0 conn(0x7f6fcc1538a0 legacy=0x7f6fcc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:47.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd195f640 1 -- 172.21.15.125:0/2549617314 >> v1:172.21.15.125:6789/0 conn(0x7f6fcc0cf160 legacy=0x7f6fcc164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:47.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.656+0000 7f6fd195f640 1 -- 172.21.15.125:0/2549617314 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fcc0cb050 con 0x7f6fcc15d6f0 2023-10-30T13:00:47.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.660+0000 7f6fd195f640 1 --2- 172.21.15.125:0/2549617314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fcc15d6f0 0x7f6fcc15a090 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0097f0 tx=0x7f6fbc01ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a318be5cce9310b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:47.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.660+0000 7f6fd115e640 1 -- 172.21.15.125:0/2549617314 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0a89b0 con 0x7f6fcc15d6f0 2023-10-30T13:00:47.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.660+0000 7f6fd115e640 1 -- 172.21.15.125:0/2549617314 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc0a8b10 con 0x7f6fcc15d6f0 2023-10-30T13:00:47.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.660+0000 7f6fd115e640 1 -- 172.21.15.125:0/2549617314 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0a8ea0 con 0x7f6fcc15d6f0 2023-10-30T13:00:47.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.660+0000 7f6fd43c4640 1 -- 172.21.15.125:0/2549617314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fcc15d6f0 msgr2=0x7f6fcc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:47.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.660+0000 7f6fd43c4640 1 --2- 172.21.15.125:0/2549617314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fcc15d6f0 0x7f6fcc15a090 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc0097f0 tx=0x7f6fbc01ecb0 comp rx=0 tx=0).stop 2023-10-30T13:00:47.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.660+0000 7f6fd43c4640 1 -- 172.21.15.125:0/2549617314 shutdown_connections 2023-10-30T13:00:47.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.660+0000 7f6fd43c4640 1 --2- 172.21.15.125:0/2549617314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fcc15d6f0 0x7f6fcc15a090 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:47.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.660+0000 7f6fd43c4640 1 -- 172.21.15.125:0/2549617314 >> 172.21.15.125:0/2549617314 conn(0x7f6fcc06bfe0 msgr2=0x7f6fcc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:47.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.660+0000 7f6fd43c4640 1 -- 172.21.15.125:0/2549617314 shutdown_connections 2023-10-30T13:00:47.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.660+0000 7f6fd43c4640 1 -- 172.21.15.125:0/2549617314 wait complete. 2023-10-30T13:00:47.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.664+0000 7f6fd43c4640 1 Processor -- start 2023-10-30T13:00:47.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.664+0000 7f6fd43c4640 1 -- start start 2023-10-30T13:00:47.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.664+0000 7f6fd43c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fcc0cf160 0x7f6fcc158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:47.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.664+0000 7f6fd43c4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fcc1538a0 0x7f6fcc14bf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:47.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.664+0000 7f6fd43c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fcc15d6f0 0x7f6fcc14c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:47.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.664+0000 7f6fd43c4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc15bc20 con 0x7f6fcc1538a0 2023-10-30T13:00:47.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.664+0000 7f6fd43c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc151230 con 0x7f6fcc0cf160 2023-10-30T13:00:47.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.664+0000 7f6fd43c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fcc1e2a10 con 0x7f6fcc15d6f0 2023-10-30T13:00:47.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.664+0000 7f6fd195f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fcc1538a0 0x7f6fcc14bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:47.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd195f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fcc1538a0 0x7f6fcc14bf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34534/0 (socket says 172.21.15.125:34534) 2023-10-30T13:00:47.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd195f640 1 -- 172.21.15.125:0/3666499037 learned_addr learned my addr 172.21.15.125:0/3666499037 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:47.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd2961640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fcc15d6f0 0x7f6fcc14c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:47.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd2160640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fcc0cf160 0x7f6fcc158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:47.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd195f640 1 -- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fcc15d6f0 msgr2=0x7f6fcc14c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:47.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd195f640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fcc15d6f0 0x7f6fcc14c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:47.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd195f640 1 -- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fcc0cf160 msgr2=0x7f6fcc158db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:47.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd195f640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fcc0cf160 0x7f6fcc158db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:47.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd195f640 1 -- 172.21.15.125:0/3666499037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fbc009420 con 0x7f6fcc1538a0 2023-10-30T13:00:47.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd2961640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fcc15d6f0 0x7f6fcc14c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:00:47.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd2160640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fcc0cf160 0x7f6fcc158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:00:47.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd195f640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fcc1538a0 0x7f6fcc14bf80 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc01b8c0 tx=0x7f6fbc01ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:47.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fc2ffd640 1 -- 172.21.15.125:0/3666499037 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0b0d80 con 0x7f6fcc1538a0 2023-10-30T13:00:47.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fc2ffd640 1 -- 172.21.15.125:0/3666499037 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fbc0bb440 con 0x7f6fcc1538a0 2023-10-30T13:00:47.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd43c4640 1 -- 172.21.15.125:0/3666499037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fcc1efa80 con 0x7f6fcc1538a0 2023-10-30T13:00:47.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fc2ffd640 1 -- 172.21.15.125:0/3666499037 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fbc0bb7f0 con 0x7f6fcc1538a0 2023-10-30T13:00:47.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd43c4640 1 -- 172.21.15.125:0/3666499037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fcc15be70 con 0x7f6fcc1538a0 2023-10-30T13:00:47.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fd43c4640 1 -- 172.21.15.125:0/3666499037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fcc04e170 con 0x7f6fcc1538a0 2023-10-30T13:00:47.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fc2ffd640 1 -- 172.21.15.125:0/3666499037 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6fbc0b7040 con 0x7f6fcc1538a0 2023-10-30T13:00:47.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.668+0000 7f6fc2ffd640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fb00664a0 0x7f6fb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:47.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.672+0000 7f6fc2ffd640 1 -- 172.21.15.125:0/3666499037 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7f6fbc122240 con 0x7f6fcc1538a0 2023-10-30T13:00:47.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.672+0000 7f6fd2160640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fb00664a0 0x7f6fb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:47.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.672+0000 7f6fd2160640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fb00664a0 0x7f6fb0068960 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6fc80011f0 tx=0x7f6fc8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:47.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.676+0000 7f6fc2ffd640 1 -- 172.21.15.125:0/3666499037 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6fbc0ef100 con 0x7f6fcc1538a0 2023-10-30T13:00:47.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.888+0000 7f6fd43c4640 1 -- 172.21.15.125:0/3666499037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6fcc0b1dd0 con 0x7f6fcc1538a0 2023-10-30T13:00:47.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.888+0000 7f6fc2ffd640 1 -- 172.21.15.125:0/3666499037 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f6fbc0f2720 con 0x7f6fcc1538a0 2023-10-30T13:00:47.895 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:47.895 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:00:40.486046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:47.895 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 146 2023-10-30T13:00:47.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.892+0000 7f6fd43c4640 1 -- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fb00664a0 msgr2=0x7f6fb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:47.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.892+0000 7f6fd43c4640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fb00664a0 0x7f6fb0068960 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6fc80011f0 tx=0x7f6fc8009040 comp rx=0 tx=0).stop 2023-10-30T13:00:47.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.892+0000 7f6fd43c4640 1 -- 172.21.15.125:0/3666499037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fcc1538a0 msgr2=0x7f6fcc14bf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:47.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.892+0000 7f6fd43c4640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fcc1538a0 0x7f6fcc14bf80 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f6fbc01b8c0 tx=0x7f6fbc01ba50 comp rx=0 tx=0).stop 2023-10-30T13:00:47.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.892+0000 7f6fd43c4640 1 -- 172.21.15.125:0/3666499037 shutdown_connections 2023-10-30T13:00:47.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.892+0000 7f6fd43c4640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fb00664a0 0x7f6fb0068960 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:47.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.892+0000 7f6fd43c4640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6fcc15d6f0 0x7f6fcc14c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:47.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.892+0000 7f6fd43c4640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6fcc1538a0 0x7f6fcc14bf80 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:47.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.892+0000 7f6fd43c4640 1 --2- 172.21.15.125:0/3666499037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6fcc0cf160 0x7f6fcc158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:47.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.892+0000 7f6fd43c4640 1 -- 172.21.15.125:0/3666499037 >> 172.21.15.125:0/3666499037 conn(0x7f6fcc06bfe0 msgr2=0x7f6fcc0c4900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:47.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.892+0000 7f6fd43c4640 1 -- 172.21.15.125:0/3666499037 shutdown_connections 2023-10-30T13:00:47.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:47.892+0000 7f6fd43c4640 1 -- 172.21.15.125:0/3666499037 wait complete. 2023-10-30T13:00:47.915 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:00:49.916 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:50.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.056+0000 7f0eb691d640 1 Processor -- start 2023-10-30T13:00:50.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.060+0000 7f0eb691d640 1 -- start start 2023-10-30T13:00:50.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.060+0000 7f0eb691d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0eb01538a0 0x7f0eb015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:50.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.060+0000 7f0eb691d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0eb00ca0e0 0x7f0eb015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:50.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.060+0000 7f0eb691d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb00cb800 con 0x7f0eb00cf160 2023-10-30T13:00:50.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.060+0000 7f0eb691d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb015dc30 con 0x7f0eb00ca0e0 2023-10-30T13:00:50.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.060+0000 7f0eb691d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb00cb330 con 0x7f0eb01538a0 2023-10-30T13:00:50.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.060+0000 7f0eaf7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0eb00ca0e0 0x7f0eb015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:50.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.060+0000 7f0eaf7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0eb00ca0e0 0x7f0eb015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34542/0 (socket says 172.21.15.125:34542) 2023-10-30T13:00:50.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.060+0000 7f0eaf7fe640 1 -- 172.21.15.125:0/2305945221 learned_addr learned my addr 172.21.15.125:0/2305945221 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:50.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.060+0000 7f0eaffff640 1 --2- 172.21.15.125:0/2305945221 >> v2:172.21.15.136:3300/0 conn(0x7f0eb01538a0 0x7f0eb015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:50.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.060+0000 7f0eaf7fe640 1 -- 172.21.15.125:0/2305945221 >> v2:172.21.15.136:3300/0 conn(0x7f0eb01538a0 msgr2=0x7f0eb015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:50.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.060+0000 7f0eaf7fe640 1 --2- 172.21.15.125:0/2305945221 >> v2:172.21.15.136:3300/0 conn(0x7f0eb01538a0 0x7f0eb015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:50.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.060+0000 7f0eaf7fe640 1 -- 172.21.15.125:0/2305945221 >> v1:172.21.15.136:6789/0 conn(0x7f0eb00cf160 legacy=0x7f0eb00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:50.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.060+0000 7f0eaf7fe640 1 -- 172.21.15.125:0/2305945221 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb00cad70 con 0x7f0eb00ca0e0 2023-10-30T13:00:50.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.064+0000 7f0eaf7fe640 1 --2- 172.21.15.125:0/2305945221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb00ca0e0 0x7f0eb015a090 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00cfd10 tx=0x7f0ea401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9490642855001124 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:50.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.064+0000 7f0eaeffd640 1 -- 172.21.15.125:0/2305945221 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea40a89f0 con 0x7f0eb00ca0e0 2023-10-30T13:00:50.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.064+0000 7f0eaeffd640 1 -- 172.21.15.125:0/2305945221 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea40a8b50 con 0x7f0eb00ca0e0 2023-10-30T13:00:50.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.064+0000 7f0eaeffd640 1 -- 172.21.15.125:0/2305945221 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea40b0440 con 0x7f0eb00ca0e0 2023-10-30T13:00:50.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.064+0000 7f0eb691d640 1 -- 172.21.15.125:0/2305945221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb00ca0e0 msgr2=0x7f0eb015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:50.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.064+0000 7f0eb691d640 1 --2- 172.21.15.125:0/2305945221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb00ca0e0 0x7f0eb015a090 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00cfd10 tx=0x7f0ea401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:00:50.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.068+0000 7f0eb691d640 1 -- 172.21.15.125:0/2305945221 shutdown_connections 2023-10-30T13:00:50.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.068+0000 7f0eb691d640 1 --2- 172.21.15.125:0/2305945221 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb00ca0e0 0x7f0eb015a090 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:50.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.068+0000 7f0eb691d640 1 --2- 172.21.15.125:0/2305945221 >> v2:172.21.15.136:3300/0 conn(0x7f0eb01538a0 0x7f0eb015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:50.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.068+0000 7f0eb691d640 1 -- 172.21.15.125:0/2305945221 >> 172.21.15.125:0/2305945221 conn(0x7f0eb006bfe0 msgr2=0x7f0eb006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:50.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.068+0000 7f0eb691d640 1 -- 172.21.15.125:0/2305945221 shutdown_connections 2023-10-30T13:00:50.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.068+0000 7f0eb691d640 1 -- 172.21.15.125:0/2305945221 wait complete. 2023-10-30T13:00:50.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.068+0000 7f0eb691d640 1 Processor -- start 2023-10-30T13:00:50.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.068+0000 7f0eb691d640 1 -- start start 2023-10-30T13:00:50.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb691d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0eb00ca0e0 0x7f0eb0140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:50.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb691d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0eb00cf160 0x7f0eb00ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:50.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb691d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb01538a0 0x7f0eb00cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:50.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb691d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb01e4380 con 0x7f0eb01538a0 2023-10-30T13:00:50.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb691d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb0158620 con 0x7f0eb00cf160 2023-10-30T13:00:50.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb691d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb0159690 con 0x7f0eb00ca0e0 2023-10-30T13:00:50.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb4eba640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb01538a0 0x7f0eb00cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:50.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eaffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0eb00ca0e0 0x7f0eb0140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:50.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb4eba640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb01538a0 0x7f0eb00cd4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34556/0 (socket says 172.21.15.125:34556) 2023-10-30T13:00:50.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eaf7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0eb00cf160 0x7f0eb00ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:50.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb4eba640 1 -- 172.21.15.125:0/2885621607 learned_addr learned my addr 172.21.15.125:0/2885621607 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:50.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb4eba640 1 -- 172.21.15.125:0/2885621607 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0eb00ca0e0 msgr2=0x7f0eb0140ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:50.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb4eba640 1 --2- 172.21.15.125:0/2885621607 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0eb00ca0e0 0x7f0eb0140ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:50.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb4eba640 1 -- 172.21.15.125:0/2885621607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0eb00cf160 msgr2=0x7f0eb00ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:50.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb4eba640 1 --2- 172.21.15.125:0/2885621607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0eb00cf160 0x7f0eb00ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:50.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb4eba640 1 -- 172.21.15.125:0/2885621607 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea40094d0 con 0x7f0eb01538a0 2023-10-30T13:00:50.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb4eba640 1 --2- 172.21.15.125:0/2885621607 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb01538a0 0x7f0eb00cd4f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f0ea800c6f0 tx=0x7f0ea800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:50.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eacff9640 1 -- 172.21.15.125:0/2885621607 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8003980 con 0x7f0eb01538a0 2023-10-30T13:00:50.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eacff9640 1 -- 172.21.15.125:0/2885621607 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea8003ae0 con 0x7f0eb01538a0 2023-10-30T13:00:50.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb691d640 1 -- 172.21.15.125:0/2885621607 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb01e51e0 con 0x7f0eb01538a0 2023-10-30T13:00:50.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eacff9640 1 -- 172.21.15.125:0/2885621607 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8003e90 con 0x7f0eb01538a0 2023-10-30T13:00:50.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb691d640 1 -- 172.21.15.125:0/2885621607 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb01e5610 con 0x7f0eb01538a0 2023-10-30T13:00:50.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.072+0000 7f0eb691d640 1 -- 172.21.15.125:0/2885621607 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb00b2a50 con 0x7f0eb01538a0 2023-10-30T13:00:50.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.076+0000 7f0eacff9640 1 -- 172.21.15.125:0/2885621607 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0ea800a850 con 0x7f0eb01538a0 2023-10-30T13:00:50.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.076+0000 7f0eacff9640 1 --2- 172.21.15.125:0/2885621607 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e9c0664d0 0x7f0e9c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:50.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.076+0000 7f0eacff9640 1 -- 172.21.15.125:0/2885621607 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7f0ea8083ff0 con 0x7f0eb01538a0 2023-10-30T13:00:50.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.076+0000 7f0eaffff640 1 --2- 172.21.15.125:0/2885621607 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e9c0664d0 0x7f0e9c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:50.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.076+0000 7f0eaffff640 1 --2- 172.21.15.125:0/2885621607 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e9c0664d0 0x7f0e9c068990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00cf070 tx=0x7f0ea0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:50.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.080+0000 7f0eacff9640 1 -- 172.21.15.125:0/2885621607 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0ea8050fe0 con 0x7f0eb01538a0 2023-10-30T13:00:50.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.272+0000 7f0eb691d640 1 -- 172.21.15.125:0/2885621607 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0eb0158790 con 0x7f0eb01538a0 2023-10-30T13:00:50.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.272+0000 7f0eacff9640 1 -- 172.21.15.125:0/2885621607 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f0ea8054600 con 0x7f0eb01538a0 2023-10-30T13:00:50.277 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:50.277 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:00:40.486046+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:50.278 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 146 2023-10-30T13:00:50.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.276+0000 7f0eb691d640 1 -- 172.21.15.125:0/2885621607 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e9c0664d0 msgr2=0x7f0e9c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:50.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.276+0000 7f0eb691d640 1 --2- 172.21.15.125:0/2885621607 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e9c0664d0 0x7f0e9c068990 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00cf070 tx=0x7f0ea0008040 comp rx=0 tx=0).stop 2023-10-30T13:00:50.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.276+0000 7f0eb691d640 1 -- 172.21.15.125:0/2885621607 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb01538a0 msgr2=0x7f0eb00cd4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:50.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.276+0000 7f0eb691d640 1 --2- 172.21.15.125:0/2885621607 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb01538a0 0x7f0eb00cd4f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f0ea800c6f0 tx=0x7f0ea800cbb0 comp rx=0 tx=0).stop 2023-10-30T13:00:50.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.276+0000 7f0eb691d640 1 -- 172.21.15.125:0/2885621607 shutdown_connections 2023-10-30T13:00:50.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.276+0000 7f0eb691d640 1 --2- 172.21.15.125:0/2885621607 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0e9c0664d0 0x7f0e9c068990 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:50.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.276+0000 7f0eb691d640 1 --2- 172.21.15.125:0/2885621607 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eb01538a0 0x7f0eb00cd4f0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:50.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.276+0000 7f0eb691d640 1 --2- 172.21.15.125:0/2885621607 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0eb00cf160 0x7f0eb00ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:50.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.276+0000 7f0eb691d640 1 --2- 172.21.15.125:0/2885621607 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0eb00ca0e0 0x7f0eb0140ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:50.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.276+0000 7f0eb691d640 1 -- 172.21.15.125:0/2885621607 >> 172.21.15.125:0/2885621607 conn(0x7f0eb006bfe0 msgr2=0x7f0eb00c5210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:50.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.276+0000 7f0eb691d640 1 -- 172.21.15.125:0/2885621607 shutdown_connections 2023-10-30T13:00:50.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:50.276+0000 7f0eb691d640 1 -- 172.21.15.125:0/2885621607 wait complete. 2023-10-30T13:00:50.298 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:00:51.440 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:00:51.445 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:00:52.300 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:52.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.440+0000 7f7ff8565640 1 Processor -- start 2023-10-30T13:00:52.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.440+0000 7f7ff8565640 1 -- start start 2023-10-30T13:00:52.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff8565640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7ff00c05d0 0x7f7ff013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:52.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff8565640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7ff00c6f60 0x7f7ff013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:52.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff8565640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff00c0a50 con 0x7f7ff00c6b80 2023-10-30T13:00:52.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff8565640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff006f6e0 con 0x7f7ff00c6f60 2023-10-30T13:00:52.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff8565640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff006f9c0 con 0x7f7ff00c05d0 2023-10-30T13:00:52.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff6301640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7ff00c6b80 0x7f7ff00bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42242/0 (socket says 172.21.15.125:42242) 2023-10-30T13:00:52.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff6301640 1 -- 172.21.15.125:0/547252124 learned_addr learned my addr 172.21.15.125:0/547252124 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:00:52.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff5b00640 1 --2- 172.21.15.125:0/547252124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7ff00c05d0 0x7f7ff013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:52.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff6b02640 1 --2- 172.21.15.125:0/547252124 >> v2:172.21.15.136:3300/0 conn(0x7f7ff00c6f60 0x7f7ff013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:52.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 -- 172.21.15.125:0/547252124 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452119008 0 0) 0x7f7ff00c0a50 con 0x7f7ff00c6b80 2023-10-30T13:00:52.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 -- 172.21.15.125:0/547252124 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fdc003530 con 0x7f7ff00c6b80 2023-10-30T13:00:52.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 -- 172.21.15.125:0/547252124 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1970592724 0 0) 0x7f7fdc003530 con 0x7f7ff00c6b80 2023-10-30T13:00:52.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 -- 172.21.15.125:0/547252124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7ff00c05d0 msgr2=0x7f7ff013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:52.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 --2- 172.21.15.125:0/547252124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7ff00c05d0 0x7f7ff013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:52.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 -- 172.21.15.125:0/547252124 >> v2:172.21.15.136:3300/0 conn(0x7f7ff00c6f60 msgr2=0x7f7ff013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:52.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 --2- 172.21.15.125:0/547252124 >> v2:172.21.15.136:3300/0 conn(0x7f7ff00c6f60 0x7f7ff013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:52.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 -- 172.21.15.125:0/547252124 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff006fca0 con 0x7f7ff00c6b80 2023-10-30T13:00:52.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 -- 172.21.15.125:0/547252124 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f7fec001680 con 0x7f7ff00c6b80 2023-10-30T13:00:52.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 -- 172.21.15.125:0/547252124 >> v1:172.21.15.125:6789/0 conn(0x7f7ff00c6b80 legacy=0x7f7ff00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:52.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 --2- 172.21.15.125:0/547252124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fdc003980 0x7f7fdc003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:52.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 --2- 172.21.15.125:0/547252124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fdc0047d0 0x7f7fdc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:52.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 --2- 172.21.15.125:0/547252124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7ff00c05d0 0x7f7fdc005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:52.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 -- 172.21.15.125:0/547252124 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff006f6e0 con 0x7f7fdc003980 2023-10-30T13:00:52.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 -- 172.21.15.125:0/547252124 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff006f9c0 con 0x7f7fdc0047d0 2023-10-30T13:00:52.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff52ff640 1 -- 172.21.15.125:0/547252124 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc00dba0 con 0x7f7ff00c05d0 2023-10-30T13:00:52.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff6301640 1 --2- 172.21.15.125:0/547252124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fdc003980 0x7f7fdc003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:52.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff6b02640 1 --2- 172.21.15.125:0/547252124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7ff00c05d0 0x7f7fdc005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:52.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff5b00640 1 --2- 172.21.15.125:0/547252124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fdc0047d0 0x7f7fdc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:52.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff6301640 1 -- 172.21.15.125:0/547252124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7ff00c05d0 msgr2=0x7f7fdc005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:52.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff6301640 1 --2- 172.21.15.125:0/547252124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7ff00c05d0 0x7f7fdc005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:52.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff6301640 1 -- 172.21.15.125:0/547252124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fdc0047d0 msgr2=0x7f7fdc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:52.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff6301640 1 --2- 172.21.15.125:0/547252124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7fdc0047d0 0x7f7fdc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:52.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.444+0000 7f7ff6301640 1 -- 172.21.15.125:0/547252124 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc003530 con 0x7f7fdc003980 2023-10-30T13:00:52.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.448+0000 7f7ff6301640 1 --2- 172.21.15.125:0/547252124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fdc003980 0x7f7fdc003da0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00c240 tx=0x7f7fec01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84553fd96052d302 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.448+0000 7f7ff52ff640 1 -- 172.21.15.125:0/547252124 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec0058e0 con 0x7f7fdc003980 2023-10-30T13:00:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.448+0000 7f7ff52ff640 1 -- 172.21.15.125:0/547252124 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec005a40 con 0x7f7fdc003980 2023-10-30T13:00:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.448+0000 7f7ff52ff640 1 -- 172.21.15.125:0/547252124 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec005df0 con 0x7f7fdc003980 2023-10-30T13:00:52.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.448+0000 7f7ff8565640 1 -- 172.21.15.125:0/547252124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fdc003980 msgr2=0x7f7fdc003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.448+0000 7f7ff8565640 1 --2- 172.21.15.125:0/547252124 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fdc003980 0x7f7fdc003da0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00c240 tx=0x7f7fec01cc60 comp rx=0 tx=0).stop 2023-10-30T13:00:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.448+0000 7f7ff6b02640 1 -- 172.21.15.125:0/547252124 reap_dead start 2023-10-30T13:00:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.448+0000 7f7ff8565640 1 -- 172.21.15.125:0/547252124 shutdown_connections 2023-10-30T13:00:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.448+0000 7f7ff8565640 1 -- 172.21.15.125:0/547252124 >> 172.21.15.125:0/547252124 conn(0x7f7ff00c7910 msgr2=0x7f7ff00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.448+0000 7f7ff8565640 1 -- 172.21.15.125:0/547252124 shutdown_connections 2023-10-30T13:00:52.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.448+0000 7f7ff8565640 1 -- 172.21.15.125:0/547252124 wait complete. 2023-10-30T13:00:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff8565640 1 Processor -- start 2023-10-30T13:00:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff8565640 1 -- start start 2023-10-30T13:00:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff8565640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fdc003980 0x7f7ff0154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff8565640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7ff01553f0 0x7f7ff01f5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:52.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff8565640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ff01f26d0 0x7f7ff006b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:52.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff8565640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff01f1cc0 con 0x7f7fdc003980 2023-10-30T13:00:52.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff8565640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff013c5f0 con 0x7f7ff01f26d0 2023-10-30T13:00:52.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff8565640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff01f2240 con 0x7f7ff01553f0 2023-10-30T13:00:52.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff6301640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fdc003980 0x7f7ff0154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:52.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff6301640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fdc003980 0x7f7ff0154db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58290/0 (socket says 172.21.15.125:58290) 2023-10-30T13:00:52.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff6301640 1 -- 172.21.15.125:0/2025730657 learned_addr learned my addr 172.21.15.125:0/2025730657 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:52.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff6b02640 1 --2- 172.21.15.125:0/2025730657 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ff01f26d0 0x7f7ff006b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:52.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff5b00640 1 --2- 172.21.15.125:0/2025730657 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7ff01553f0 0x7f7ff01f5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:52.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff6301640 1 -- 172.21.15.125:0/2025730657 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7ff01553f0 msgr2=0x7f7ff01f5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:52.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff6301640 1 --2- 172.21.15.125:0/2025730657 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7ff01553f0 0x7f7ff01f5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:52.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff6301640 1 -- 172.21.15.125:0/2025730657 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ff01f26d0 msgr2=0x7f7ff006b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:52.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff6301640 1 --2- 172.21.15.125:0/2025730657 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ff01f26d0 0x7f7ff006b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:52.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff6301640 1 -- 172.21.15.125:0/2025730657 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff013c840 con 0x7f7fdc003980 2023-10-30T13:00:52.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff6301640 1 --2- 172.21.15.125:0/2025730657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fdc003980 0x7f7ff0154db0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00c240 tx=0x7f7fdc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:52.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7fe6ffd640 1 -- 172.21.15.125:0/2025730657 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec0047f0 con 0x7f7fdc003980 2023-10-30T13:00:52.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7fe6ffd640 1 -- 172.21.15.125:0/2025730657 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fec004950 con 0x7f7fdc003980 2023-10-30T13:00:52.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff8565640 1 -- 172.21.15.125:0/2025730657 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff013cb20 con 0x7f7fdc003980 2023-10-30T13:00:52.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7fe6ffd640 1 -- 172.21.15.125:0/2025730657 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec004cc0 con 0x7f7fdc003980 2023-10-30T13:00:52.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.452+0000 7f7ff8565640 1 -- 172.21.15.125:0/2025730657 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff0150980 con 0x7f7fdc003980 2023-10-30T13:00:52.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.456+0000 7f7ff8565640 1 -- 172.21.15.125:0/2025730657 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff013cd60 con 0x7f7fdc003980 2023-10-30T13:00:52.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.456+0000 7f7fe6ffd640 1 -- 172.21.15.125:0/2025730657 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7fec0070d0 con 0x7f7fdc003980 2023-10-30T13:00:52.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.456+0000 7f7fe6ffd640 1 --2- 172.21.15.125:0/2025730657 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fd40664d0 0x7f7fd4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:52.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.456+0000 7f7fe6ffd640 1 -- 172.21.15.125:0/2025730657 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7f7fec11d550 con 0x7f7fdc003980 2023-10-30T13:00:52.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.456+0000 7f7ff5b00640 1 --2- 172.21.15.125:0/2025730657 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fd40664d0 0x7f7fd4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:52.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.456+0000 7f7ff5b00640 1 --2- 172.21.15.125:0/2025730657 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fd40664d0 0x7f7fd4068990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f7ff013ea10 tx=0x7f7fe0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:52.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.460+0000 7f7fe6ffd640 1 -- 172.21.15.125:0/2025730657 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7ff013cd60 con 0x7f7fdc003980 2023-10-30T13:00:52.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.640+0000 7f7ff8565640 1 -- 172.21.15.125:0/2025730657 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ff013cd60 con 0x7f7fdc003980 2023-10-30T13:00:52.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.644+0000 7f7fe6ffd640 1 -- 172.21.15.125:0/2025730657 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f7ff013cd60 con 0x7f7fdc003980 2023-10-30T13:00:52.647 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:52.647 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:00:50.412582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:52.648 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 147 2023-10-30T13:00:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.644+0000 7f7ff8565640 1 -- 172.21.15.125:0/2025730657 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fd40664d0 msgr2=0x7f7fd4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.644+0000 7f7ff8565640 1 --2- 172.21.15.125:0/2025730657 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fd40664d0 0x7f7fd4068990 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f7ff013ea10 tx=0x7f7fe0008040 comp rx=0 tx=0).stop 2023-10-30T13:00:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.644+0000 7f7ff8565640 1 -- 172.21.15.125:0/2025730657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fdc003980 msgr2=0x7f7ff0154db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:52.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.644+0000 7f7ff8565640 1 --2- 172.21.15.125:0/2025730657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fdc003980 0x7f7ff0154db0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f7fec00c240 tx=0x7f7fdc005800 comp rx=0 tx=0).stop 2023-10-30T13:00:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.644+0000 7f7ff8565640 1 -- 172.21.15.125:0/2025730657 shutdown_connections 2023-10-30T13:00:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.644+0000 7f7ff8565640 1 --2- 172.21.15.125:0/2025730657 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fd40664d0 0x7f7fd4068990 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.644+0000 7f7ff8565640 1 --2- 172.21.15.125:0/2025730657 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7ff01f26d0 0x7f7ff006b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.644+0000 7f7ff8565640 1 --2- 172.21.15.125:0/2025730657 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7ff01553f0 0x7f7ff01f5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:52.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.644+0000 7f7ff8565640 1 --2- 172.21.15.125:0/2025730657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7fdc003980 0x7f7ff0154db0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:52.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.644+0000 7f7ff8565640 1 -- 172.21.15.125:0/2025730657 >> 172.21.15.125:0/2025730657 conn(0x7f7ff00c7910 msgr2=0x7f7ff013aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:52.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.644+0000 7f7ff8565640 1 -- 172.21.15.125:0/2025730657 shutdown_connections 2023-10-30T13:00:52.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:52.644+0000 7f7ff8565640 1 -- 172.21.15.125:0/2025730657 wait complete. 2023-10-30T13:00:52.667 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:00:54.668 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:54.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.812+0000 7f67b1a6b640 1 Processor -- start 2023-10-30T13:00:54.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.812+0000 7f67b1a6b640 1 -- start start 2023-10-30T13:00:54.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.812+0000 7f67b1a6b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f67ac15d6f0 0x7f67ac15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:54.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.812+0000 7f67b1a6b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67ac0cb800 con 0x7f67ac0cf160 2023-10-30T13:00:54.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.812+0000 7f67b1a6b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67ac0cb330 con 0x7f67ac1538a0 2023-10-30T13:00:54.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.812+0000 7f67b1a6b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67ac0cad70 con 0x7f67ac15d6f0 2023-10-30T13:00:54.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.812+0000 7f67aaffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f67ac15d6f0 0x7f67ac15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:54.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.812+0000 7f67abfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f67ac0cf160 0x7f67ac164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42244/0 (socket says 172.21.15.125:42244) 2023-10-30T13:00:54.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.812+0000 7f67abfff640 1 -- 172.21.15.125:0/1701279871 learned_addr learned my addr 172.21.15.125:0/1701279871 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:00:54.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.812+0000 7f67aaffd640 1 -- 172.21.15.125:0/1701279871 >> v1:172.21.15.136:6789/0 conn(0x7f67ac1538a0 legacy=0x7f67ac0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:54.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.812+0000 7f67aaffd640 1 -- 172.21.15.125:0/1701279871 >> v1:172.21.15.125:6789/0 conn(0x7f67ac0cf160 legacy=0x7f67ac164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:54.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.812+0000 7f67aaffd640 1 -- 172.21.15.125:0/1701279871 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ac0cb050 con 0x7f67ac15d6f0 2023-10-30T13:00:54.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.812+0000 7f67aa7fc640 1 -- 172.21.15.125:0/1701279871 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230901622 0 0) 0x7f67ac0cb800 con 0x7f67ac0cf160 2023-10-30T13:00:54.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.816+0000 7f67aaffd640 1 --2- 172.21.15.125:0/1701279871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67ac15d6f0 0x7f67ac15a090 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f67a00097c0 tx=0x7f67a001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a19dfbf48fc1578 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:54.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.816+0000 7f67aa7fc640 1 -- 172.21.15.125:0/1701279871 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a00a8970 con 0x7f67ac15d6f0 2023-10-30T13:00:54.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.816+0000 7f67aa7fc640 1 -- 172.21.15.125:0/1701279871 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a00a8ad0 con 0x7f67ac15d6f0 2023-10-30T13:00:54.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.816+0000 7f67aa7fc640 1 -- 172.21.15.125:0/1701279871 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a00a8e60 con 0x7f67ac15d6f0 2023-10-30T13:00:54.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.816+0000 7f67b1a6b640 1 -- 172.21.15.125:0/1701279871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67ac15d6f0 msgr2=0x7f67ac15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:54.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.816+0000 7f67b1a6b640 1 --2- 172.21.15.125:0/1701279871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67ac15d6f0 0x7f67ac15a090 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f67a00097c0 tx=0x7f67a001ec70 comp rx=0 tx=0).stop 2023-10-30T13:00:54.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.816+0000 7f67b1a6b640 1 -- 172.21.15.125:0/1701279871 shutdown_connections 2023-10-30T13:00:54.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.816+0000 7f67b1a6b640 1 --2- 172.21.15.125:0/1701279871 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67ac15d6f0 0x7f67ac15a090 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:54.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.816+0000 7f67b1a6b640 1 -- 172.21.15.125:0/1701279871 >> 172.21.15.125:0/1701279871 conn(0x7f67ac06bfe0 msgr2=0x7f67ac06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:54.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.816+0000 7f67b1a6b640 1 -- 172.21.15.125:0/1701279871 shutdown_connections 2023-10-30T13:00:54.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.816+0000 7f67b1a6b640 1 -- 172.21.15.125:0/1701279871 wait complete. 2023-10-30T13:00:54.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67b1a6b640 1 Processor -- start 2023-10-30T13:00:54.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67b1a6b640 1 -- start start 2023-10-30T13:00:54.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67b1a6b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67ac0cf160 0x7f67ac14ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:54.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67b1a6b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67ac1538a0 0x7f67ac0ce710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:54.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67b1a6b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67ac15d6f0 0x7f67ac0c4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:54.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67b1a6b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac1ee880 con 0x7f67ac15d6f0 2023-10-30T13:00:54.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67b1a6b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac14d380 con 0x7f67ac0cf160 2023-10-30T13:00:54.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67b1a6b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac15cfa0 con 0x7f67ac1538a0 2023-10-30T13:00:54.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67abfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67ac15d6f0 0x7f67ac0c4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:54.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67abfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67ac15d6f0 0x7f67ac0c4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58314/0 (socket says 172.21.15.125:58314) 2023-10-30T13:00:54.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67ab7fe640 1 --2- 172.21.15.125:0/3301214155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67ac0cf160 0x7f67ac14ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:54.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67abfff640 1 -- 172.21.15.125:0/3301214155 learned_addr learned my addr 172.21.15.125:0/3301214155 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:54.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67aaffd640 1 --2- 172.21.15.125:0/3301214155 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67ac1538a0 0x7f67ac0ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:54.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67abfff640 1 -- 172.21.15.125:0/3301214155 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67ac1538a0 msgr2=0x7f67ac0ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:54.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67abfff640 1 --2- 172.21.15.125:0/3301214155 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67ac1538a0 0x7f67ac0ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:54.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67abfff640 1 -- 172.21.15.125:0/3301214155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67ac0cf160 msgr2=0x7f67ac14ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:54.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67abfff640 1 --2- 172.21.15.125:0/3301214155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67ac0cf160 0x7f67ac14ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:54.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67abfff640 1 -- 172.21.15.125:0/3301214155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a0009420 con 0x7f67ac15d6f0 2023-10-30T13:00:54.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67abfff640 1 --2- 172.21.15.125:0/3301214155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67ac15d6f0 0x7f67ac0c4070 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f67a400c7b0 tx=0x7f67a400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:54.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f6793fff640 1 -- 172.21.15.125:0/3301214155 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a400ce60 con 0x7f67ac15d6f0 2023-10-30T13:00:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f6793fff640 1 -- 172.21.15.125:0/3301214155 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a400a640 con 0x7f67ac15d6f0 2023-10-30T13:00:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67b1a6b640 1 -- 172.21.15.125:0/3301214155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ac1eea70 con 0x7f67ac15d6f0 2023-10-30T13:00:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f6793fff640 1 -- 172.21.15.125:0/3301214155 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a400a9d0 con 0x7f67ac15d6f0 2023-10-30T13:00:54.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.820+0000 7f67b1a6b640 1 -- 172.21.15.125:0/3301214155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ac1ef030 con 0x7f67ac15d6f0 2023-10-30T13:00:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.824+0000 7f67b1a6b640 1 -- 172.21.15.125:0/3301214155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ac04e170 con 0x7f67ac15d6f0 2023-10-30T13:00:54.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.824+0000 7f6793fff640 1 -- 172.21.15.125:0/3301214155 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f67a400ab30 con 0x7f67ac15d6f0 2023-10-30T13:00:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.824+0000 7f6793fff640 1 --2- 172.21.15.125:0/3301214155 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67980664a0 0x7f6798068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.824+0000 7f6793fff640 1 -- 172.21.15.125:0/3301214155 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7f67a4083740 con 0x7f67ac15d6f0 2023-10-30T13:00:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.824+0000 7f67ab7fe640 1 --2- 172.21.15.125:0/3301214155 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67980664a0 0x7f6798068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:54.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.824+0000 7f67ab7fe640 1 --2- 172.21.15.125:0/3301214155 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67980664a0 0x7f6798068960 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f679c007fd0 tx=0x7f679c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:54.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:54.828+0000 7f6793fff640 1 -- 172.21.15.125:0/3301214155 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f67a40506a0 con 0x7f67ac15d6f0 2023-10-30T13:00:55.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:55.008+0000 7f67b1a6b640 1 -- 172.21.15.125:0/3301214155 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67ac0ca9b0 con 0x7f67ac15d6f0 2023-10-30T13:00:55.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:55.008+0000 7f6793fff640 1 -- 172.21.15.125:0/3301214155 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f67a4053cc0 con 0x7f67ac15d6f0 2023-10-30T13:00:55.014 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:55.015 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":147,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:00:50.412582+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:reconnect","state_seq":16,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:55.015 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 147 2023-10-30T13:00:55.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:55.012+0000 7f67b1a6b640 1 -- 172.21.15.125:0/3301214155 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67980664a0 msgr2=0x7f6798068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:55.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:55.012+0000 7f67b1a6b640 1 --2- 172.21.15.125:0/3301214155 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67980664a0 0x7f6798068960 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f679c007fd0 tx=0x7f679c009040 comp rx=0 tx=0).stop 2023-10-30T13:00:55.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:55.012+0000 7f67b1a6b640 1 -- 172.21.15.125:0/3301214155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67ac15d6f0 msgr2=0x7f67ac0c4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:55.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:55.012+0000 7f67b1a6b640 1 --2- 172.21.15.125:0/3301214155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67ac15d6f0 0x7f67ac0c4070 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f67a400c7b0 tx=0x7f67a400cc70 comp rx=0 tx=0).stop 2023-10-30T13:00:55.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:55.012+0000 7f67b1a6b640 1 -- 172.21.15.125:0/3301214155 shutdown_connections 2023-10-30T13:00:55.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:55.012+0000 7f67b1a6b640 1 --2- 172.21.15.125:0/3301214155 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67980664a0 0x7f6798068960 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:55.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:55.012+0000 7f67b1a6b640 1 --2- 172.21.15.125:0/3301214155 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67ac15d6f0 0x7f67ac0c4070 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:55.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:55.012+0000 7f67b1a6b640 1 --2- 172.21.15.125:0/3301214155 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67ac1538a0 0x7f67ac0ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:55.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:55.012+0000 7f67b1a6b640 1 --2- 172.21.15.125:0/3301214155 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67ac0cf160 0x7f67ac14ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:55.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:55.012+0000 7f67b1a6b640 1 -- 172.21.15.125:0/3301214155 >> 172.21.15.125:0/3301214155 conn(0x7f67ac06bfe0 msgr2=0x7f67ac152a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:55.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:55.012+0000 7f67b1a6b640 1 -- 172.21.15.125:0/3301214155 shutdown_connections 2023-10-30T13:00:55.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:55.012+0000 7f67b1a6b640 1 -- 172.21.15.125:0/3301214155 wait complete. 2023-10-30T13:00:55.035 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 147, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 147, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7707": { "addr": "172.21.15.125:6837/3762807620", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 3762807620, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 3762807620, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7707, "incarnation": 146, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 16 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:00:50.412582+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7707 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:00:57.037 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:57.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe57a05640 1 Processor -- start 2023-10-30T13:00:57.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe57a05640 1 -- start start 2023-10-30T13:00:57.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe57a05640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe500cf160 0x7efe500cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:57.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe57a05640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe500cb800 con 0x7efe5015d6f0 2023-10-30T13:00:57.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe57a05640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe500cb330 con 0x7efe501538a0 2023-10-30T13:00:57.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe57a05640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe500cad70 con 0x7efe500cf160 2023-10-30T13:00:57.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe54fa0640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7efe5015d6f0 0x7efe5015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42246/0 (socket says 172.21.15.125:42246) 2023-10-30T13:00:57.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe54fa0640 1 -- 172.21.15.125:0/1164809463 learned_addr learned my addr 172.21.15.125:0/1164809463 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:00:57.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe55fa2640 1 --2- 172.21.15.125:0/1164809463 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe500cf160 0x7efe500cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:57.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 -- 172.21.15.125:0/1164809463 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339925446 0 0) 0x7efe500cb800 con 0x7efe5015d6f0 2023-10-30T13:00:57.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 -- 172.21.15.125:0/1164809463 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe3c003530 con 0x7efe5015d6f0 2023-10-30T13:00:57.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 -- 172.21.15.125:0/1164809463 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2740084372 0 0) 0x7efe3c003530 con 0x7efe5015d6f0 2023-10-30T13:00:57.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 -- 172.21.15.125:0/1164809463 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe500cf160 msgr2=0x7efe500cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:57.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 --2- 172.21.15.125:0/1164809463 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe500cf160 0x7efe500cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:57.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 -- 172.21.15.125:0/1164809463 >> v1:172.21.15.136:6789/0 conn(0x7efe501538a0 legacy=0x7efe500ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:57.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 -- 172.21.15.125:0/1164809463 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe500cb050 con 0x7efe5015d6f0 2023-10-30T13:00:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 -- 172.21.15.125:0/1164809463 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7efe40001680 con 0x7efe5015d6f0 2023-10-30T13:00:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 -- 172.21.15.125:0/1164809463 >> v1:172.21.15.125:6789/0 conn(0x7efe5015d6f0 legacy=0x7efe5015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 --2- 172.21.15.125:0/1164809463 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe500cf160 0x7efe3c0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 --2- 172.21.15.125:0/1164809463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe3c003f30 0x7efe3c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:57.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 --2- 172.21.15.125:0/1164809463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe3c004d10 0x7efe3c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:57.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 -- 172.21.15.125:0/1164809463 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe500cad70 con 0x7efe3c004d10 2023-10-30T13:00:57.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 -- 172.21.15.125:0/1164809463 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c00bb00 con 0x7efe3c003f30 2023-10-30T13:00:57.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.180+0000 7efe47fff640 1 -- 172.21.15.125:0/1164809463 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe3c00bcc0 con 0x7efe500cf160 2023-10-30T13:00:57.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.184+0000 7efe55fa2640 1 --2- 172.21.15.125:0/1164809463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe3c004d10 0x7efe3c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:57.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.184+0000 7efe557a1640 1 --2- 172.21.15.125:0/1164809463 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe500cf160 0x7efe3c0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:57.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.184+0000 7efe54fa0640 1 --2- 172.21.15.125:0/1164809463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe3c003f30 0x7efe3c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:57.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.184+0000 7efe55fa2640 1 -- 172.21.15.125:0/1164809463 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe500cf160 msgr2=0x7efe3c0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:57.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.184+0000 7efe55fa2640 1 --2- 172.21.15.125:0/1164809463 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe500cf160 0x7efe3c0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:57.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.184+0000 7efe55fa2640 1 -- 172.21.15.125:0/1164809463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe3c003f30 msgr2=0x7efe3c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:00:57.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.184+0000 7efe55fa2640 1 --2- 172.21.15.125:0/1164809463 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe3c003f30 0x7efe3c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:57.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.184+0000 7efe55fa2640 1 -- 172.21.15.125:0/1164809463 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3c003530 con 0x7efe3c004d10 2023-10-30T13:00:57.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.184+0000 7efe55fa2640 1 --2- 172.21.15.125:0/1164809463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe3c004d10 0x7efe3c005190 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7efe48009fc0 tx=0x7efe4801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb58f67daddc87c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:57.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.184+0000 7efe47fff640 1 -- 172.21.15.125:0/1164809463 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe480a5c70 con 0x7efe3c004d10 2023-10-30T13:00:57.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.184+0000 7efe47fff640 1 -- 172.21.15.125:0/1164809463 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4800f040 con 0x7efe3c004d10 2023-10-30T13:00:57.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.184+0000 7efe47fff640 1 -- 172.21.15.125:0/1164809463 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe480ad440 con 0x7efe3c004d10 2023-10-30T13:00:57.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe57a05640 1 -- 172.21.15.125:0/1164809463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe3c004d10 msgr2=0x7efe3c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:57.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe57a05640 1 --2- 172.21.15.125:0/1164809463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe3c004d10 0x7efe3c005190 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7efe48009fc0 tx=0x7efe4801cc60 comp rx=0 tx=0).stop 2023-10-30T13:00:57.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe55fa2640 1 -- 172.21.15.125:0/1164809463 reap_dead start 2023-10-30T13:00:57.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe57a05640 1 -- 172.21.15.125:0/1164809463 shutdown_connections 2023-10-30T13:00:57.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe57a05640 1 -- 172.21.15.125:0/1164809463 >> 172.21.15.125:0/1164809463 conn(0x7efe5006bfe0 msgr2=0x7efe5006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:57.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe57a05640 1 -- 172.21.15.125:0/1164809463 shutdown_connections 2023-10-30T13:00:57.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe57a05640 1 -- 172.21.15.125:0/1164809463 wait complete. 2023-10-30T13:00:57.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe57a05640 1 Processor -- start 2023-10-30T13:00:57.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe57a05640 1 -- start start 2023-10-30T13:00:57.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe57a05640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe3c004d10 0x7efe50155590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:57.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe57a05640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe50159290 0x7efe501f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:57.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe57a05640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe501f2800 0x7efe5014f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:57.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe57a05640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5014ffc0 con 0x7efe501f2800 2023-10-30T13:00:57.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe57a05640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe50150210 con 0x7efe50159290 2023-10-30T13:00:57.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe57a05640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe50155ad0 con 0x7efe3c004d10 2023-10-30T13:00:57.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe55fa2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe501f2800 0x7efe5014f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:57.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe557a1640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe3c004d10 0x7efe50155590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:57.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe55fa2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe501f2800 0x7efe5014f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58322/0 (socket says 172.21.15.125:58322) 2023-10-30T13:00:57.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe55fa2640 1 -- 172.21.15.125:0/1973875065 learned_addr learned my addr 172.21.15.125:0/1973875065 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:57.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe54fa0640 1 --2- 172.21.15.125:0/1973875065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe50159290 0x7efe501f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:57.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe55fa2640 1 -- 172.21.15.125:0/1973875065 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe3c004d10 msgr2=0x7efe50155590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:57.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe55fa2640 1 --2- 172.21.15.125:0/1973875065 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe3c004d10 0x7efe50155590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:57.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe55fa2640 1 -- 172.21.15.125:0/1973875065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe50159290 msgr2=0x7efe501f22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:57.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe55fa2640 1 --2- 172.21.15.125:0/1973875065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe50159290 0x7efe501f22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:57.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe55fa2640 1 -- 172.21.15.125:0/1973875065 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe501f2f70 con 0x7efe501f2800 2023-10-30T13:00:57.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.188+0000 7efe54fa0640 1 --2- 172.21.15.125:0/1973875065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe50159290 0x7efe501f22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:00:57.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.192+0000 7efe55fa2640 1 --2- 172.21.15.125:0/1973875065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe501f2800 0x7efe5014f370 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7efe480b0040 tx=0x7efe48009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:57.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.192+0000 7efe45ffb640 1 -- 172.21.15.125:0/1973875065 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe480039a0 con 0x7efe501f2800 2023-10-30T13:00:57.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.192+0000 7efe45ffb640 1 -- 172.21.15.125:0/1973875065 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe48003b00 con 0x7efe501f2800 2023-10-30T13:00:57.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.192+0000 7efe45ffb640 1 -- 172.21.15.125:0/1973875065 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48003e90 con 0x7efe501f2800 2023-10-30T13:00:57.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.192+0000 7efe57a05640 1 -- 172.21.15.125:0/1973875065 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5014ffc0 con 0x7efe501f2800 2023-10-30T13:00:57.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.192+0000 7efe57a05640 1 -- 172.21.15.125:0/1973875065 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe501f2c90 con 0x7efe501f2800 2023-10-30T13:00:57.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.192+0000 7efe57a05640 1 -- 172.21.15.125:0/1973875065 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe500cef50 con 0x7efe501f2800 2023-10-30T13:00:57.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.192+0000 7efe45ffb640 1 -- 172.21.15.125:0/1973875065 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efe480b4040 con 0x7efe501f2800 2023-10-30T13:00:57.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.192+0000 7efe45ffb640 1 --2- 172.21.15.125:0/1973875065 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe340664a0 0x7efe34068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:57.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.192+0000 7efe45ffb640 1 -- 172.21.15.125:0/1973875065 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7efe4811e1a0 con 0x7efe501f2800 2023-10-30T13:00:57.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.192+0000 7efe557a1640 1 --2- 172.21.15.125:0/1973875065 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe340664a0 0x7efe34068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:57.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.196+0000 7efe557a1640 1 --2- 172.21.15.125:0/1973875065 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe340664a0 0x7efe34068960 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7efe4c000c10 tx=0x7efe4c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:57.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.196+0000 7efe45ffb640 1 -- 172.21.15.125:0/1973875065 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efe500cef50 con 0x7efe501f2800 2023-10-30T13:00:57.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.376+0000 7efe57a05640 1 -- 172.21.15.125:0/1973875065 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe50158fb0 con 0x7efe501f2800 2023-10-30T13:00:57.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.376+0000 7efe45ffb640 1 -- 172.21.15.125:0/1973875065 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+5724 (secure 0 0 0) 0x7efe50158fb0 con 0x7efe501f2800 2023-10-30T13:00:57.382 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:57.383 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:00:55.551688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:57.383 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 148 2023-10-30T13:00:57.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.380+0000 7efe57a05640 1 -- 172.21.15.125:0/1973875065 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe340664a0 msgr2=0x7efe34068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:57.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.380+0000 7efe57a05640 1 --2- 172.21.15.125:0/1973875065 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe340664a0 0x7efe34068960 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7efe4c000c10 tx=0x7efe4c009040 comp rx=0 tx=0).stop 2023-10-30T13:00:57.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.380+0000 7efe57a05640 1 -- 172.21.15.125:0/1973875065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe501f2800 msgr2=0x7efe5014f370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:57.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.380+0000 7efe57a05640 1 --2- 172.21.15.125:0/1973875065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe501f2800 0x7efe5014f370 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7efe480b0040 tx=0x7efe48009750 comp rx=0 tx=0).stop 2023-10-30T13:00:57.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.380+0000 7efe57a05640 1 -- 172.21.15.125:0/1973875065 shutdown_connections 2023-10-30T13:00:57.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.380+0000 7efe57a05640 1 --2- 172.21.15.125:0/1973875065 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe340664a0 0x7efe34068960 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:57.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.380+0000 7efe57a05640 1 --2- 172.21.15.125:0/1973875065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe501f2800 0x7efe5014f370 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:57.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.380+0000 7efe57a05640 1 --2- 172.21.15.125:0/1973875065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe50159290 0x7efe501f22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:57.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.380+0000 7efe57a05640 1 --2- 172.21.15.125:0/1973875065 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe3c004d10 0x7efe50155590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:57.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.380+0000 7efe57a05640 1 -- 172.21.15.125:0/1973875065 >> 172.21.15.125:0/1973875065 conn(0x7efe5006bfe0 msgr2=0x7efe500c4a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:57.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.380+0000 7efe57a05640 1 -- 172.21.15.125:0/1973875065 shutdown_connections 2023-10-30T13:00:57.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:57.380+0000 7efe57a05640 1 -- 172.21.15.125:0/1973875065 wait complete. 2023-10-30T13:00:57.403 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:00:59.405 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:00:59.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.548+0000 7f727de9e640 1 Processor -- start 2023-10-30T13:00:59.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.548+0000 7f727de9e640 1 -- start start 2023-10-30T13:00:59.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.548+0000 7f727de9e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f72780cf160 0x7f72780cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:59.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.548+0000 7f727de9e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72780cb800 con 0x7f72781538a0 2023-10-30T13:00:59.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.548+0000 7f727de9e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72780cb330 con 0x7f727815d6f0 2023-10-30T13:00:59.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.548+0000 7f727de9e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72780cad70 con 0x7f72780cf160 2023-10-30T13:00:59.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.548+0000 7f72777fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f72781538a0 0x7f72780ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42258/0 (socket says 172.21.15.125:42258) 2023-10-30T13:00:59.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.548+0000 7f72777fe640 1 -- 172.21.15.125:0/1589498463 learned_addr learned my addr 172.21.15.125:0/1589498463 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:00:59.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.548+0000 7f7277fff640 1 --2- 172.21.15.125:0/1589498463 >> v2:172.21.15.125:3300/0 conn(0x7f72780cf160 0x7f72780cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:59.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.548+0000 7f72767fc640 1 -- 172.21.15.125:0/1589498463 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2828037278 0 0) 0x7f72780cb800 con 0x7f72781538a0 2023-10-30T13:00:59.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.548+0000 7f72767fc640 1 -- 172.21.15.125:0/1589498463 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7260003530 con 0x7f72781538a0 2023-10-30T13:00:59.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.548+0000 7f7277fff640 1 -- 172.21.15.125:0/1589498463 >> v1:172.21.15.136:6789/0 conn(0x7f727815d6f0 legacy=0x7f727815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:59.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.548+0000 7f7277fff640 1 -- 172.21.15.125:0/1589498463 >> v1:172.21.15.125:6789/0 conn(0x7f72781538a0 legacy=0x7f72780ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:59.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.548+0000 7f7277fff640 1 -- 172.21.15.125:0/1589498463 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72780cb050 con 0x7f72780cf160 2023-10-30T13:00:59.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.552+0000 7f7277fff640 1 --2- 172.21.15.125:0/1589498463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72780cf160 0x7f72780cf590 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f7270009d90 tx=0x7f727001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f9fc946682fa3f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:59.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.556+0000 7f72767fc640 1 -- 172.21.15.125:0/1589498463 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72700af600 con 0x7f72780cf160 2023-10-30T13:00:59.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.556+0000 7f72767fc640 1 -- 172.21.15.125:0/1589498463 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72700ab050 con 0x7f72780cf160 2023-10-30T13:00:59.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.556+0000 7f72767fc640 1 -- 172.21.15.125:0/1589498463 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72700af980 con 0x7f72780cf160 2023-10-30T13:00:59.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.556+0000 7f727de9e640 1 -- 172.21.15.125:0/1589498463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72780cf160 msgr2=0x7f72780cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:59.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.556+0000 7f727de9e640 1 --2- 172.21.15.125:0/1589498463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72780cf160 0x7f72780cf590 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f7270009d90 tx=0x7f727001f130 comp rx=0 tx=0).stop 2023-10-30T13:00:59.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.556+0000 7f727de9e640 1 -- 172.21.15.125:0/1589498463 shutdown_connections 2023-10-30T13:00:59.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.556+0000 7f727de9e640 1 --2- 172.21.15.125:0/1589498463 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72780cf160 0x7f72780cf590 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:59.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.556+0000 7f727de9e640 1 -- 172.21.15.125:0/1589498463 >> 172.21.15.125:0/1589498463 conn(0x7f727806bfe0 msgr2=0x7f727806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:59.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.556+0000 7f727de9e640 1 -- 172.21.15.125:0/1589498463 shutdown_connections 2023-10-30T13:00:59.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.556+0000 7f727de9e640 1 -- 172.21.15.125:0/1589498463 wait complete. 2023-10-30T13:00:59.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.556+0000 7f727de9e640 1 Processor -- start 2023-10-30T13:00:59.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.556+0000 7f727de9e640 1 -- start start 2023-10-30T13:00:59.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f727de9e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72780cf160 0x7f72780b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:59.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f727de9e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72781538a0 0x7f7278148c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:59.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f727de9e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727815d6f0 0x7f72781e5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:59.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f727de9e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72780b4350 con 0x7f727815d6f0 2023-10-30T13:00:59.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f727de9e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7278140b50 con 0x7f72780cf160 2023-10-30T13:00:59.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f727de9e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7278145eb0 con 0x7f72781538a0 2023-10-30T13:00:59.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f7277fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727815d6f0 0x7f72781e5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:59.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f7277fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727815d6f0 0x7f72781e5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:58344/0 (socket says 172.21.15.125:58344) 2023-10-30T13:00:59.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f7277fff640 1 -- 172.21.15.125:0/4191816271 learned_addr learned my addr 172.21.15.125:0/4191816271 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:00:59.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f72777fe640 1 --2- 172.21.15.125:0/4191816271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72780cf160 0x7f72780b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:59.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f7276ffd640 1 --2- 172.21.15.125:0/4191816271 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72781538a0 0x7f7278148c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:59.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f7277fff640 1 -- 172.21.15.125:0/4191816271 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72781538a0 msgr2=0x7f7278148c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:59.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f7277fff640 1 --2- 172.21.15.125:0/4191816271 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72781538a0 0x7f7278148c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f7277fff640 1 -- 172.21.15.125:0/4191816271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72780cf160 msgr2=0x7f72780b3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f7277fff640 1 --2- 172.21.15.125:0/4191816271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72780cf160 0x7f72780b3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f7277fff640 1 -- 172.21.15.125:0/4191816271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7270009a70 con 0x7f727815d6f0 2023-10-30T13:00:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f7277fff640 1 --2- 172.21.15.125:0/4191816271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727815d6f0 0x7f72781e5290 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f72700b2040 tx=0x7f727001bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:59.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f725ffff640 1 -- 172.21.15.125:0/4191816271 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7270009450 con 0x7f727815d6f0 2023-10-30T13:00:59.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f725ffff640 1 -- 172.21.15.125:0/4191816271 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72700b9410 con 0x7f727815d6f0 2023-10-30T13:00:59.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f727de9e640 1 -- 172.21.15.125:0/4191816271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72781494a0 con 0x7f727815d6f0 2023-10-30T13:00:59.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f725ffff640 1 -- 172.21.15.125:0/4191816271 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72700b97c0 con 0x7f727815d6f0 2023-10-30T13:00:59.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f727de9e640 1 -- 172.21.15.125:0/4191816271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72781450f0 con 0x7f727815d6f0 2023-10-30T13:00:59.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.560+0000 7f727de9e640 1 -- 172.21.15.125:0/4191816271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f727804e170 con 0x7f727815d6f0 2023-10-30T13:00:59.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.564+0000 7f725ffff640 1 -- 172.21.15.125:0/4191816271 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f72700b6050 con 0x7f727815d6f0 2023-10-30T13:00:59.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.564+0000 7f725ffff640 1 --2- 172.21.15.125:0/4191816271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72640664a0 0x7f7264068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:00:59.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.564+0000 7f725ffff640 1 -- 172.21.15.125:0/4191816271 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7f72701226f0 con 0x7f727815d6f0 2023-10-30T13:00:59.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.564+0000 7f72777fe640 1 --2- 172.21.15.125:0/4191816271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72640664a0 0x7f7264068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:00:59.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.564+0000 7f72777fe640 1 --2- 172.21.15.125:0/4191816271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72640664a0 0x7f7264068960 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f72680011f0 tx=0x7f7268007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:00:59.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.568+0000 7f725ffff640 1 -- 172.21.15.125:0/4191816271 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f72700ef770 con 0x7f727815d6f0 2023-10-30T13:00:59.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.760+0000 7f727de9e640 1 -- 172.21.15.125:0/4191816271 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72780b1dd0 con 0x7f727815d6f0 2023-10-30T13:00:59.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.760+0000 7f725ffff640 1 -- 172.21.15.125:0/4191816271 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f72700f2d90 con 0x7f727815d6f0 2023-10-30T13:00:59.764 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:00:59.765 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:00:55.551688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:00:59.765 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 148 2023-10-30T13:00:59.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.760+0000 7f727de9e640 1 -- 172.21.15.125:0/4191816271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72640664a0 msgr2=0x7f7264068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:59.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.760+0000 7f727de9e640 1 --2- 172.21.15.125:0/4191816271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72640664a0 0x7f7264068960 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f72680011f0 tx=0x7f7268007d40 comp rx=0 tx=0).stop 2023-10-30T13:00:59.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.760+0000 7f727de9e640 1 -- 172.21.15.125:0/4191816271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727815d6f0 msgr2=0x7f72781e5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:00:59.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.760+0000 7f727de9e640 1 --2- 172.21.15.125:0/4191816271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727815d6f0 0x7f72781e5290 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f72700b2040 tx=0x7f727001bed0 comp rx=0 tx=0).stop 2023-10-30T13:00:59.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.760+0000 7f727de9e640 1 -- 172.21.15.125:0/4191816271 shutdown_connections 2023-10-30T13:00:59.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.760+0000 7f727de9e640 1 --2- 172.21.15.125:0/4191816271 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72640664a0 0x7f7264068960 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.760+0000 7f727de9e640 1 --2- 172.21.15.125:0/4191816271 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f727815d6f0 0x7f72781e5290 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.760+0000 7f727de9e640 1 --2- 172.21.15.125:0/4191816271 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72781538a0 0x7f7278148c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.760+0000 7f727de9e640 1 --2- 172.21.15.125:0/4191816271 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72780cf160 0x7f72780b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:00:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.760+0000 7f727de9e640 1 -- 172.21.15.125:0/4191816271 >> 172.21.15.125:0/4191816271 conn(0x7f727806bfe0 msgr2=0x7f72780c51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:00:59.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.760+0000 7f727de9e640 1 -- 172.21.15.125:0/4191816271 shutdown_connections 2023-10-30T13:00:59.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:00:59.760+0000 7f727de9e640 1 -- 172.21.15.125:0/4191816271 wait complete. 2023-10-30T13:00:59.785 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:01:01.786 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:01:01.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.920+0000 7fb7e333e640 1 Processor -- start 2023-10-30T13:01:01.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.920+0000 7fb7e333e640 1 -- start start 2023-10-30T13:01:01.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.924+0000 7fb7e333e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb7dc0c6b80 0x7fb7dc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:01.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.924+0000 7fb7e333e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7dc0c0a50 con 0x7fb7dc0c6f60 2023-10-30T13:01:01.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.924+0000 7fb7e333e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7dc06f6e0 con 0x7fb7dc13f4a0 2023-10-30T13:01:01.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.924+0000 7fb7e333e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7dc06f9c0 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.924+0000 7fb7e10da640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb7dc0c6b80 0x7fb7dc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:01.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.924+0000 7fb7e08d9640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb7dc0c6f60 0x7fb7dc13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44634/0 (socket says 172.21.15.125:44634) 2023-10-30T13:01:01.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.924+0000 7fb7e08d9640 1 -- 172.21.15.125:0/691620797 learned_addr learned my addr 172.21.15.125:0/691620797 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:01:01.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.924+0000 7fb7e10da640 1 -- 172.21.15.125:0/691620797 >> v1:172.21.15.136:6789/0 conn(0x7fb7dc13f4a0 legacy=0x7fb7dc1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:01.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.924+0000 7fb7e10da640 1 -- 172.21.15.125:0/691620797 >> v1:172.21.15.125:6789/0 conn(0x7fb7dc0c6f60 legacy=0x7fb7dc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:01.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.924+0000 7fb7e10da640 1 -- 172.21.15.125:0/691620797 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc06fca0 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.924+0000 7fb7e10da640 1 --2- 172.21.15.125:0/691620797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7dc0c6b80 0x7fb7dc0bfec0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fb7d80097c0 tx=0x7fb7d801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3af35e8993e5435 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:01:01.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.924+0000 7fb7d3fff640 1 -- 172.21.15.125:0/691620797 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d80a8050 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.924+0000 7fb7d3fff640 1 -- 172.21.15.125:0/691620797 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d80ac5f0 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.924+0000 7fb7d3fff640 1 -- 172.21.15.125:0/691620797 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d80ac980 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 -- 172.21.15.125:0/691620797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7dc0c6b80 msgr2=0x7fb7dc0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:01.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 --2- 172.21.15.125:0/691620797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7dc0c6b80 0x7fb7dc0bfec0 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fb7d80097c0 tx=0x7fb7d801c480 comp rx=0 tx=0).stop 2023-10-30T13:01:01.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 -- 172.21.15.125:0/691620797 shutdown_connections 2023-10-30T13:01:01.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 --2- 172.21.15.125:0/691620797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7dc0c6b80 0x7fb7dc0bfec0 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:01.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 -- 172.21.15.125:0/691620797 >> 172.21.15.125:0/691620797 conn(0x7fb7dc0c7910 msgr2=0x7fb7dc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:01:01.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 -- 172.21.15.125:0/691620797 shutdown_connections 2023-10-30T13:01:01.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 -- 172.21.15.125:0/691620797 wait complete. 2023-10-30T13:01:01.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 Processor -- start 2023-10-30T13:01:01.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 -- start start 2023-10-30T13:01:01.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7dc0c6b80 0x7fb7dc0c97e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:01.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7dc0c6f60 0x7fb7dc1314b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:01.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7dc13f4a0 0x7fb7dc0afb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:01.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc1f6c20 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc157c10 con 0x7fb7dc13f4a0 2023-10-30T13:01:01.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e333e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc0c8d40 con 0x7fb7dc0c6f60 2023-10-30T13:01:01.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e10da640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7dc0c6b80 0x7fb7dc0c97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:01.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e10da640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7dc0c6b80 0x7fb7dc0c97e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59928/0 (socket says 172.21.15.125:59928) 2023-10-30T13:01:01.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e10da640 1 -- 172.21.15.125:0/2954817369 learned_addr learned my addr 172.21.15.125:0/2954817369 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:01:01.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e08d9640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7dc0c6f60 0x7fb7dc1314b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:01.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e18db640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7dc13f4a0 0x7fb7dc0afb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:01.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e10da640 1 -- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7dc0c6f60 msgr2=0x7fb7dc1314b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:01.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e10da640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7dc0c6f60 0x7fb7dc1314b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:01.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e10da640 1 -- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7dc13f4a0 msgr2=0x7fb7dc0afb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:01.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e10da640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7dc13f4a0 0x7fb7dc0afb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:01.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e10da640 1 -- 172.21.15.125:0/2954817369 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d8009420 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e18db640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7dc13f4a0 0x7fb7dc0afb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:01:01.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.928+0000 7fb7e08d9640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7dc0c6f60 0x7fb7dc1314b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:01:01.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.932+0000 7fb7e10da640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7dc0c6b80 0x7fb7dc0c97e0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fb7d80af040 tx=0x7fb7d8019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:01:01.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.932+0000 7fb7d1ffb640 1 -- 172.21.15.125:0/2954817369 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d80b5620 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.932+0000 7fb7d1ffb640 1 -- 172.21.15.125:0/2954817369 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d80b5780 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.932+0000 7fb7d1ffb640 1 -- 172.21.15.125:0/2954817369 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d80b5b30 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.932+0000 7fb7e333e640 1 -- 172.21.15.125:0/2954817369 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7dc157930 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.932+0000 7fb7e333e640 1 -- 172.21.15.125:0/2954817369 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7dc0c92c0 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.932+0000 7fb7e333e640 1 -- 172.21.15.125:0/2954817369 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7dc04e170 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.932+0000 7fb7d1ffb640 1 -- 172.21.15.125:0/2954817369 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb7d80b3040 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.932+0000 7fb7d1ffb640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7c00664a0 0x7fb7c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:01.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.932+0000 7fb7d1ffb640 1 -- 172.21.15.125:0/2954817369 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7fb7d811e220 con 0x7fb7dc0c6b80 2023-10-30T13:01:01.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.932+0000 7fb7e08d9640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7c00664a0 0x7fb7c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:01.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.936+0000 7fb7e08d9640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7c00664a0 0x7fb7c0068960 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc13ae80 tx=0x7fb7cc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:01:01.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:01.936+0000 7fb7d1ffb640 1 -- 172.21.15.125:0/2954817369 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb7d80eb050 con 0x7fb7dc0c6b80 2023-10-30T13:01:02.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:02.116+0000 7fb7e333e640 1 -- 172.21.15.125:0/2954817369 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7dc13aed0 con 0x7fb7dc0c6b80 2023-10-30T13:01:02.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:02.116+0000 7fb7d1ffb640 1 -- 172.21.15.125:0/2954817369 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+5724 (secure 0 0 0) 0x7fb7d80ee670 con 0x7fb7dc0c6b80 2023-10-30T13:01:02.122 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:01:02.122 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:00:55.551688+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:01:02.122 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 148 2023-10-30T13:01:02.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:02.120+0000 7fb7e333e640 1 -- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7c00664a0 msgr2=0x7fb7c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:02.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:02.120+0000 7fb7e333e640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7c00664a0 0x7fb7c0068960 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc13ae80 tx=0x7fb7cc009040 comp rx=0 tx=0).stop 2023-10-30T13:01:02.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:02.120+0000 7fb7e333e640 1 -- 172.21.15.125:0/2954817369 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7dc0c6b80 msgr2=0x7fb7dc0c97e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:02.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:02.120+0000 7fb7e333e640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7dc0c6b80 0x7fb7dc0c97e0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fb7d80af040 tx=0x7fb7d8019220 comp rx=0 tx=0).stop 2023-10-30T13:01:02.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:02.120+0000 7fb7e333e640 1 -- 172.21.15.125:0/2954817369 shutdown_connections 2023-10-30T13:01:02.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:02.120+0000 7fb7e333e640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb7c00664a0 0x7fb7c0068960 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:02.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:02.120+0000 7fb7e333e640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb7dc13f4a0 0x7fb7dc0afb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:02.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:02.120+0000 7fb7e333e640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb7dc0c6f60 0x7fb7dc1314b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:02.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:02.120+0000 7fb7e333e640 1 --2- 172.21.15.125:0/2954817369 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb7dc0c6b80 0x7fb7dc0c97e0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:02.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:02.120+0000 7fb7e333e640 1 -- 172.21.15.125:0/2954817369 >> 172.21.15.125:0/2954817369 conn(0x7fb7dc0c7910 msgr2=0x7fb7dc1339b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:01:02.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:02.120+0000 7fb7e333e640 1 -- 172.21.15.125:0/2954817369 shutdown_connections 2023-10-30T13:01:02.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:02.120+0000 7fb7e333e640 1 -- 172.21.15.125:0/2954817369 wait complete. 2023-10-30T13:01:02.142 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:01:04.144 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:01:04.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f927ea46640 1 Processor -- start 2023-10-30T13:01:04.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f927ea46640 1 -- start start 2023-10-30T13:01:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f927ea46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92781538a0 0x7f927815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f927ea46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f92780cf160 0x7f92780cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:04.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f927ea46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92780cb800 con 0x7f92780ca0e0 2023-10-30T13:01:04.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f927ea46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f927815dc30 con 0x7f92780cf160 2023-10-30T13:01:04.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f927ea46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92780cb330 con 0x7f92781538a0 2023-10-30T13:01:04.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f927cfe3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f92780cf160 0x7f92780cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:04.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f927cfe3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f92780cf160 0x7f92780cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59936/0 (socket says 172.21.15.125:59936) 2023-10-30T13:01:04.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f927cfe3640 1 -- 172.21.15.125:0/1489670591 learned_addr learned my addr 172.21.15.125:0/1489670591 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:01:04.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f9277fff640 1 --2- 172.21.15.125:0/1489670591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92781538a0 0x7f927815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:04.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f927cfe3640 1 -- 172.21.15.125:0/1489670591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92781538a0 msgr2=0x7f927815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:01:04.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f927cfe3640 1 --2- 172.21.15.125:0/1489670591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92781538a0 0x7f927815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:04.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f927cfe3640 1 -- 172.21.15.125:0/1489670591 >> v1:172.21.15.136:6789/0 conn(0x7f92780ca0e0 legacy=0x7f927815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:04.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.288+0000 7f927cfe3640 1 -- 172.21.15.125:0/1489670591 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92780cad70 con 0x7f92780cf160 2023-10-30T13:01:04.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.292+0000 7f927cfe3640 1 --2- 172.21.15.125:0/1489670591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92780cf160 0x7f92780cf590 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f9270009e50 tx=0x7f927001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebcffe169928ba06 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:01:04.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.292+0000 7f9276ffd640 1 -- 172.21.15.125:0/1489670591 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92700af400 con 0x7f92780cf160 2023-10-30T13:01:04.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.292+0000 7f9276ffd640 1 -- 172.21.15.125:0/1489670591 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92700ab050 con 0x7f92780cf160 2023-10-30T13:01:04.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.292+0000 7f9276ffd640 1 -- 172.21.15.125:0/1489670591 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92700af780 con 0x7f92780cf160 2023-10-30T13:01:04.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.292+0000 7f927ea46640 1 -- 172.21.15.125:0/1489670591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92780cf160 msgr2=0x7f92780cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:04.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.292+0000 7f927ea46640 1 --2- 172.21.15.125:0/1489670591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92780cf160 0x7f92780cf590 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f9270009e50 tx=0x7f927001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:01:04.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.292+0000 7f927ea46640 1 -- 172.21.15.125:0/1489670591 shutdown_connections 2023-10-30T13:01:04.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.292+0000 7f927ea46640 1 --2- 172.21.15.125:0/1489670591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92780cf160 0x7f92780cf590 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:04.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.292+0000 7f927ea46640 1 --2- 172.21.15.125:0/1489670591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92781538a0 0x7f927815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:04.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.292+0000 7f927ea46640 1 -- 172.21.15.125:0/1489670591 >> 172.21.15.125:0/1489670591 conn(0x7f927806bfe0 msgr2=0x7f927806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:01:04.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.292+0000 7f927ea46640 1 -- 172.21.15.125:0/1489670591 shutdown_connections 2023-10-30T13:01:04.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.292+0000 7f927ea46640 1 -- 172.21.15.125:0/1489670591 wait complete. 2023-10-30T13:01:04.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f927ea46640 1 Processor -- start 2023-10-30T13:01:04.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f927ea46640 1 -- start start 2023-10-30T13:01:04.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f927ea46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92780ca0e0 0x7f927815a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:04.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f927ea46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92780cf160 0x7f927815b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:04.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f927ea46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92781538a0 0x7f927815b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:04.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f927ea46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92781e3960 con 0x7f92780ca0e0 2023-10-30T13:01:04.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f927ea46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92781e4750 con 0x7f92781538a0 2023-10-30T13:01:04.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f927ea46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92781e3ba0 con 0x7f92780cf160 2023-10-30T13:01:04.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f9277fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92780ca0e0 0x7f927815a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:04.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f9277fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92780ca0e0 0x7f927815a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59952/0 (socket says 172.21.15.125:59952) 2023-10-30T13:01:04.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f9277fff640 1 -- 172.21.15.125:0/3633226175 learned_addr learned my addr 172.21.15.125:0/3633226175 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:01:04.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f92777fe640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92780cf160 0x7f927815b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:04.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f927cfe3640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92781538a0 0x7f927815b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:04.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f9277fff640 1 -- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92780cf160 msgr2=0x7f927815b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:04.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f9277fff640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92780cf160 0x7f927815b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:04.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f9277fff640 1 -- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92781538a0 msgr2=0x7f927815b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:04.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f9277fff640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92781538a0 0x7f927815b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:04.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f9277fff640 1 -- 172.21.15.125:0/3633226175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9270009b30 con 0x7f92780ca0e0 2023-10-30T13:01:04.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f927cfe3640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92781538a0 0x7f927815b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:01:04.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f9277fff640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92780ca0e0 0x7f927815a8b0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f926800d5f0 tx=0x7f926800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:01:04.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f9274ff9640 1 -- 172.21.15.125:0/3633226175 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92680125f0 con 0x7f92780ca0e0 2023-10-30T13:01:04.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f9274ff9640 1 -- 172.21.15.125:0/3633226175 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9268012750 con 0x7f92780ca0e0 2023-10-30T13:01:04.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f9274ff9640 1 -- 172.21.15.125:0/3633226175 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9268012ae0 con 0x7f92780ca0e0 2023-10-30T13:01:04.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f927ea46640 1 -- 172.21.15.125:0/3633226175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92781e3430 con 0x7f92780ca0e0 2023-10-30T13:01:04.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.296+0000 7f927ea46640 1 -- 172.21.15.125:0/3633226175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92781e3db0 con 0x7f92780ca0e0 2023-10-30T13:01:04.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.300+0000 7f927ea46640 1 -- 172.21.15.125:0/3633226175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92781e49c0 con 0x7f92780ca0e0 2023-10-30T13:01:04.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.300+0000 7f9274ff9640 1 -- 172.21.15.125:0/3633226175 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9268012c40 con 0x7f92780ca0e0 2023-10-30T13:01:04.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.300+0000 7f9274ff9640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f92640664a0 0x7f9264068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:04.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.300+0000 7f9274ff9640 1 -- 172.21.15.125:0/3633226175 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7f92680848f0 con 0x7f92780ca0e0 2023-10-30T13:01:04.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.300+0000 7f92777fe640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f92640664a0 0x7f9264068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:04.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.300+0000 7f92777fe640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f92640664a0 0x7f9264068960 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f927815a130 tx=0x7f926c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:01:04.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.304+0000 7f9274ff9640 1 -- 172.21.15.125:0/3633226175 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f92680517a0 con 0x7f92780ca0e0 2023-10-30T13:01:04.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.484+0000 7f927ea46640 1 -- 172.21.15.125:0/3633226175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f92780b2a50 con 0x7f92780ca0e0 2023-10-30T13:01:04.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.484+0000 7f9274ff9640 1 -- 172.21.15.125:0/3633226175 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f9268007380 con 0x7f92780ca0e0 2023-10-30T13:01:04.490 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:01:04.490 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":149,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:01:02.384461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:01:04.490 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 149 2023-10-30T13:01:04.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.488+0000 7f927ea46640 1 -- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f92640664a0 msgr2=0x7f9264068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:04.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.488+0000 7f927ea46640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f92640664a0 0x7f9264068960 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f927815a130 tx=0x7f926c009040 comp rx=0 tx=0).stop 2023-10-30T13:01:04.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.488+0000 7f927ea46640 1 -- 172.21.15.125:0/3633226175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92780ca0e0 msgr2=0x7f927815a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:04.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.488+0000 7f927ea46640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92780ca0e0 0x7f927815a8b0 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f926800d5f0 tx=0x7f926800dab0 comp rx=0 tx=0).stop 2023-10-30T13:01:04.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.488+0000 7f927ea46640 1 -- 172.21.15.125:0/3633226175 shutdown_connections 2023-10-30T13:01:04.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.488+0000 7f927ea46640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f92640664a0 0x7f9264068960 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:04.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.488+0000 7f927ea46640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f92781538a0 0x7f927815b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:04.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.488+0000 7f927ea46640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f92780cf160 0x7f927815b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:04.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.488+0000 7f927ea46640 1 --2- 172.21.15.125:0/3633226175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f92780ca0e0 0x7f927815a8b0 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:04.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.488+0000 7f927ea46640 1 -- 172.21.15.125:0/3633226175 >> 172.21.15.125:0/3633226175 conn(0x7f927806bfe0 msgr2=0x7f92780c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:01:04.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.488+0000 7f927ea46640 1 -- 172.21.15.125:0/3633226175 shutdown_connections 2023-10-30T13:01:04.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:04.488+0000 7f927ea46640 1 -- 172.21.15.125:0/3633226175 wait complete. 2023-10-30T13:01:04.510 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=7554 2023-10-30T13:01:04.510 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.e 2023-10-30T13:01:08.511 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T13:01:08.512 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T13:01:08.512 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T13:01:08.517 INFO:tasks.ceph.mds.e:Started 2023-10-30T13:01:08.517 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:01:08.623 INFO:tasks.ceph.mds.e.smithi125.stdout:starting mds.e at 2023-10-30T13:01:08.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.680+0000 7fa8d5406640 1 Processor -- start 2023-10-30T13:01:08.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.680+0000 7fa8d5406640 1 -- start start 2023-10-30T13:01:08.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8d5406640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa8d0157c80 0x7fa8d0161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:08.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8d5406640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d0161f60 con 0x7fa8d006fef0 2023-10-30T13:01:08.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8d5406640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d01616c0 con 0x7fa8d006ab20 2023-10-30T13:01:08.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8d5406640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d015ef30 con 0x7fa8d0157c80 2023-10-30T13:01:08.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8d4c05640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa8d006ab20 0x7fa8d0070c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:43834/0 (socket says 172.21.15.125:43834) 2023-10-30T13:01:08.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8d4c05640 1 -- 172.21.15.125:0/1068441967 learned_addr learned my addr 172.21.15.125:0/1068441967 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:01:08.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471704935 0 0) 0x7fa8d01616c0 con 0x7fa8d006ab20 2023-10-30T13:01:08.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b8003530 con 0x7fa8d006ab20 2023-10-30T13:01:08.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8cffff640 1 --2- 172.21.15.125:0/1068441967 >> v2:172.21.15.136:3300/0 conn(0x7fa8d0157c80 0x7fa8d0161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:08.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132382475 0 0) 0x7fa8d0161f60 con 0x7fa8d006fef0 2023-10-30T13:01:08.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d01616c0 con 0x7fa8d006fef0 2023-10-30T13:01:08.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2125934039 0 0) 0x7fa8b8003530 con 0x7fa8d006ab20 2023-10-30T13:01:08.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 >> v2:172.21.15.136:3300/0 conn(0x7fa8d0157c80 msgr2=0x7fa8d0161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:01:08.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8ceffd640 1 --2- 172.21.15.125:0/1068441967 >> v2:172.21.15.136:3300/0 conn(0x7fa8d0157c80 0x7fa8d0161030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:08.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 >> v1:172.21.15.125:6789/0 conn(0x7fa8d006fef0 legacy=0x7fa8d006f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:08.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d01619a0 con 0x7fa8d006ab20 2023-10-30T13:01:08.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fa8c8001a00 con 0x7fa8d006ab20 2023-10-30T13:01:08.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 >> v1:172.21.15.136:6789/0 conn(0x7fa8d006ab20 legacy=0x7fa8d0070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:08.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8ceffd640 1 --2- 172.21.15.125:0/1068441967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b8003b10 0x7fa8b8003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:08.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8ceffd640 1 --2- 172.21.15.125:0/1068441967 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8b80047d0 0x7fa8b8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:08.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.684+0000 7fa8ceffd640 1 --2- 172.21.15.125:0/1068441967 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8b8005190 0x7fa8b800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:08.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.688+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b800bf00 con 0x7fa8b80047d0 2023-10-30T13:01:08.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.688+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b800c070 con 0x7fa8b8003b10 2023-10-30T13:01:08.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.688+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b800c230 con 0x7fa8b8005190 2023-10-30T13:01:08.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.688+0000 7fa8d4c05640 1 --2- 172.21.15.125:0/1068441967 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8b8005190 0x7fa8b800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:08.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.688+0000 7fa8cf7fe640 1 --2- 172.21.15.125:0/1068441967 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8b80047d0 0x7fa8b8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:08.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.688+0000 7fa8cffff640 1 --2- 172.21.15.125:0/1068441967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b8003b10 0x7fa8b8003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:08.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.688+0000 7fa8cf7fe640 1 -- 172.21.15.125:0/1068441967 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8b8005190 msgr2=0x7fa8b800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:01:08.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.688+0000 7fa8cf7fe640 1 --2- 172.21.15.125:0/1068441967 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8b8005190 0x7fa8b800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:08.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.688+0000 7fa8cf7fe640 1 -- 172.21.15.125:0/1068441967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b8003b10 msgr2=0x7fa8b8003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:01:08.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.688+0000 7fa8cf7fe640 1 --2- 172.21.15.125:0/1068441967 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8b8003b10 0x7fa8b8003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:08.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.688+0000 7fa8cf7fe640 1 -- 172.21.15.125:0/1068441967 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d01619a0 con 0x7fa8b80047d0 2023-10-30T13:01:08.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.688+0000 7fa8d4c05640 1 -- 172.21.15.125:0/1068441967 reap_dead start 2023-10-30T13:01:08.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.692+0000 7fa8cf7fe640 1 --2- 172.21.15.125:0/1068441967 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8b80047d0 0x7fa8b8004c50 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4007fd0 tx=0x7fa8c401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f860cab553b949e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:01:08.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.692+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c4005270 con 0x7fa8b80047d0 2023-10-30T13:01:08.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.692+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c40a7070 con 0x7fa8b80047d0 2023-10-30T13:01:08.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.692+0000 7fa8ceffd640 1 -- 172.21.15.125:0/1068441967 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c4003600 con 0x7fa8b80047d0 2023-10-30T13:01:08.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.692+0000 7fa8d5406640 1 -- 172.21.15.125:0/1068441967 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8b80047d0 msgr2=0x7fa8b8004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:08.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.692+0000 7fa8d5406640 1 --2- 172.21.15.125:0/1068441967 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8b80047d0 0x7fa8b8004c50 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4007fd0 tx=0x7fa8c401c3a0 comp rx=0 tx=0).stop 2023-10-30T13:01:08.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.692+0000 7fa8d5406640 1 -- 172.21.15.125:0/1068441967 shutdown_connections 2023-10-30T13:01:08.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.692+0000 7fa8d5406640 1 --2- 172.21.15.125:0/1068441967 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8b80047d0 0x7fa8b8004c50 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:08.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.692+0000 7fa8d5406640 1 -- 172.21.15.125:0/1068441967 >> 172.21.15.125:0/1068441967 conn(0x7fa8d00c75d0 msgr2=0x7fa8d00c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:01:08.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.692+0000 7fa8d5406640 1 -- 172.21.15.125:0/1068441967 shutdown_connections 2023-10-30T13:01:08.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.692+0000 7fa8d5406640 1 -- 172.21.15.125:0/1068441967 wait complete. 2023-10-30T13:01:08.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.696+0000 7fa8d5406640 1 Processor -- start 2023-10-30T13:01:08.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.696+0000 7fa8d5406640 1 -- start start 2023-10-30T13:01:08.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d5406640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8b80047d0 0x7fa8d01f6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:08.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d5406640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8d0150b50 0x7fa8d0150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:08.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d5406640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8d015dc80 0x7fa8d015e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:08.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d5406640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d00c6eb0 con 0x7fa8d0150b50 2023-10-30T13:01:08.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d5406640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d00c70a0 con 0x7fa8d015dc80 2023-10-30T13:01:08.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d5406640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d0151510 con 0x7fa8b80047d0 2023-10-30T13:01:08.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8cffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8b80047d0 0x7fa8d01f6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:08.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d4c05640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8d015dc80 0x7fa8d015e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:08.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d4c05640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8d015dc80 0x7fa8d015e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:51618/0 (socket says 172.21.15.125:51618) 2023-10-30T13:01:08.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d4c05640 1 -- 172.21.15.125:0/2979956178 learned_addr learned my addr 172.21.15.125:0/2979956178 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:01:08.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d4c05640 1 -- 172.21.15.125:0/2979956178 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8b80047d0 msgr2=0x7fa8d01f6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:08.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d4c05640 1 --2- 172.21.15.125:0/2979956178 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8b80047d0 0x7fa8d01f6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:08.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d4c05640 1 -- 172.21.15.125:0/2979956178 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8d0150b50 msgr2=0x7fa8d0150fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:08.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d4c05640 1 --2- 172.21.15.125:0/2979956178 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8d0150b50 0x7fa8d0150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:08.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d4c05640 1 -- 172.21.15.125:0/2979956178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d00c95d0 con 0x7fa8d015dc80 2023-10-30T13:01:08.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8cffff640 1 --2- 172.21.15.125:0/2979956178 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8b80047d0 0x7fa8d01f6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:01:08.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d4c05640 1 --2- 172.21.15.125:0/2979956178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8d015dc80 0x7fa8d015e100 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0066ec0 tx=0x7fa8c80115c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:01:08.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8ccff9640 1 -- 172.21.15.125:0/2979956178 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8011e20 con 0x7fa8d015dc80 2023-10-30T13:01:08.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8ccff9640 1 -- 172.21.15.125:0/2979956178 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c8004b70 con 0x7fa8d015dc80 2023-10-30T13:01:08.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d5406640 1 -- 172.21.15.125:0/2979956178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d015e640 con 0x7fa8d015dc80 2023-10-30T13:01:08.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8d5406640 1 -- 172.21.15.125:0/2979956178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d01f3bd0 con 0x7fa8d015dc80 2023-10-30T13:01:08.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.700+0000 7fa8ccff9640 1 -- 172.21.15.125:0/2979956178 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8006070 con 0x7fa8d015dc80 2023-10-30T13:01:08.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.704+0000 7fa8d5406640 1 -- 172.21.15.125:0/2979956178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d01f3a10 con 0x7fa8d015dc80 2023-10-30T13:01:08.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.704+0000 7fa8ccff9640 1 -- 172.21.15.125:0/2979956178 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa8c80061d0 con 0x7fa8d015dc80 2023-10-30T13:01:08.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.704+0000 7fa8ccff9640 1 --2- 172.21.15.125:0/2979956178 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa8bc0664a0 0x7fa8bc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:08.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.704+0000 7fa8ccff9640 1 -- 172.21.15.125:0/2979956178 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7fa8c8087210 con 0x7fa8d015dc80 2023-10-30T13:01:08.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.704+0000 7fa8cffff640 1 --2- 172.21.15.125:0/2979956178 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa8bc0664a0 0x7fa8bc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:08.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.708+0000 7fa8cffff640 1 --2- 172.21.15.125:0/2979956178 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa8bc0664a0 0x7fa8bc068960 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009ae0 tx=0x7fa8c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:01:08.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.720+0000 7fa8ccff9640 1 -- 172.21.15.125:0/2979956178 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa8c8053f30 con 0x7fa8d015dc80 2023-10-30T13:01:08.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.932+0000 7fa8d5406640 1 -- 172.21.15.125:0/2979956178 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8d004e170 con 0x7fa8d015dc80 2023-10-30T13:01:08.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.932+0000 7fa8ccff9640 1 -- 172.21.15.125:0/2979956178 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 149 v149) v1 ==== 95+0+5724 (secure 0 0 0) 0x7fa8c801c090 con 0x7fa8d015dc80 2023-10-30T13:01:08.936 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:01:08.936 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":149,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:01:02.384461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:01:08.937 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 149 2023-10-30T13:01:08.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.932+0000 7fa8d5406640 1 -- 172.21.15.125:0/2979956178 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa8bc0664a0 msgr2=0x7fa8bc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:08.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.932+0000 7fa8d5406640 1 --2- 172.21.15.125:0/2979956178 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa8bc0664a0 0x7fa8bc068960 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fa8c0009ae0 tx=0x7fa8c0008040 comp rx=0 tx=0).stop 2023-10-30T13:01:08.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.932+0000 7fa8d5406640 1 -- 172.21.15.125:0/2979956178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8d015dc80 msgr2=0x7fa8d015e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:08.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.932+0000 7fa8d5406640 1 --2- 172.21.15.125:0/2979956178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8d015dc80 0x7fa8d015e100 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0066ec0 tx=0x7fa8c80115c0 comp rx=0 tx=0).stop 2023-10-30T13:01:08.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.932+0000 7fa8d5406640 1 -- 172.21.15.125:0/2979956178 shutdown_connections 2023-10-30T13:01:08.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.932+0000 7fa8d5406640 1 --2- 172.21.15.125:0/2979956178 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa8bc0664a0 0x7fa8bc068960 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:08.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.932+0000 7fa8d5406640 1 --2- 172.21.15.125:0/2979956178 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8d015dc80 0x7fa8d015e100 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:08.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.932+0000 7fa8d5406640 1 --2- 172.21.15.125:0/2979956178 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8d0150b50 0x7fa8d0150fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:08.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.932+0000 7fa8d5406640 1 --2- 172.21.15.125:0/2979956178 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8b80047d0 0x7fa8d01f6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:08.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.932+0000 7fa8d5406640 1 -- 172.21.15.125:0/2979956178 >> 172.21.15.125:0/2979956178 conn(0x7fa8d00c75d0 msgr2=0x7fa8d00c9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:01:08.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.932+0000 7fa8d5406640 1 -- 172.21.15.125:0/2979956178 shutdown_connections 2023-10-30T13:01:08.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:08.932+0000 7fa8d5406640 1 -- 172.21.15.125:0/2979956178 wait complete. 2023-10-30T13:01:08.957 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T13:01:09.493 INFO:tasks.ceph.mds.c.smithi125.stderr:2023-10-30T13:01:09.492+0000 7fc78e2dd640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:01:10.185 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 9% (53/540) Updating files: 10% (54/540) Updating files: 11% (60/540) Updating files: 12% (65/540) Updating files: 13% (71/540) Updating files: 14% (76/540) Updating files: 15% (81/540) Updating files: 16% (87/540) Updating files: 17% (92/540) Updating files: 18% (98/540) Updating files: 19% (103/540) Updating files: 20% (108/540) Updating files: 21% (114/540) Updating files: 22% (119/540) Updating files: 23% (125/540) Updating files: 24% (130/540) Updating files: 25% (135/540) Updating files: 26% (141/540) Updating files: 27% (146/540) Updating files: 28% (152/540) Updating files: 29% (157/540) Updating files: 30% (162/540) Updating files: 31% (168/540) Updating files: 32% (173/540) Updating files: 33% (179/540) Updating files: 34% (184/540) Updating files: 35% (189/540) Updating files: 36% (195/540) Updating files: 37% (200/540) Updating files: 38% (206/540) Updating files: 39% (211/540) Updating files: 40% (216/540) Updating files: 41% (222/540) Updating files: 41% (224/540) Updating files: 42% (227/540) Updating files: 43% (233/540) Updating files: 44% (238/540) Updating files: 45% (243/540) Updating files: 46% (249/540) Updating files: 47% (254/540) Updating files: 48% (260/540) Updating files: 49% (265/540) Updating files: 50% (270/540) Updating files: 51% (276/540) Updating files: 52% (281/540) Updating files: 53% (287/540) Updating files: 54% (292/540) Updating files: 55% (297/540) Updating files: 56% (303/540) Updating files: 57% (308/540) Updating files: 58% (314/540) Updating files: 59% (319/540) Updating files: 60% (324/540) Updating files: 61% (330/540) Updating files: 62% (335/540) Updating files: 63% (341/540) Updating files: 64% (346/540) Updating files: 65% (351/540) Updating files: 66% (357/540) Updating files: 67% (362/540) Updating files: 68% (368/540) Updating files: 69% (373/540) Updating files: 70% (378/540) Updating files: 71% (384/540) Updating files: 72% (389/540) Updating files: 73% (395/540) Updating files: 74% (400/540) Updating files: 75% (405/540) Updating files: 76% (411/540) Updating files: 77% (416/540) Updating files: 77% (421/540) Updating files: 78% (422/540) Updating files: 79% (427/540) Updating files: 80% (432/540) Updating files: 81% (438/540) Updating files: 82% (443/540) Updating files: 83% (449/540) Updating files: 84% (454/540) Updating files: 84% (457/540) Updating files: 85% (459/540) Updating files: 86% (465/540) Updating files: 87% (470/540) Updating files: 88% (476/540) Updating files: 89% (481/540) Updating files: 90% (486/540) Updating files: 91% (492/540) Updating files: 92% (497/540) Updating files: 93% (503/540) Updating files: 94% (508/540) Updating files: 95% (513/540) Updating files: 96% (519/540) Updating files: 97% (524/540) Updating files: 98% (530/540) Updating files: 99% (535/540) Updating files: 100% (540/540) Updating files: 100% (540/540), done. 2023-10-30T13:01:10.292 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 690f8175606 v0.55 2023-10-30T13:01:10.298 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.55 2023-10-30T13:01:10.322 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 56 -eq 48 ] 2023-10-30T13:01:10.322 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.56 2023-10-30T13:01:10.323 INFO:tasks.workunit.client.0.smithi125.stdout:v0.56 2023-10-30T13:01:10.323 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.56 2023-10-30T13:01:10.323 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.56 2023-10-30T13:01:10.958 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:01:11.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4aba0c8640 1 Processor -- start 2023-10-30T13:01:11.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4aba0c8640 1 -- start start 2023-10-30T13:01:11.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4aba0c8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4ab41538f0 0x7f4ab4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4aba0c8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ab415cd50 0x7f4ab40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4aba0c8640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab415d300 con 0x7f4ab40cefa0 2023-10-30T13:01:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4aba0c8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab415d5e0 con 0x7f4ab41538f0 2023-10-30T13:01:11.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4aba0c8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab415d8c0 con 0x7f4ab415cd50 2023-10-30T13:01:11.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4ab98c7640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4ab40cefa0 0x7f4ab40cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35042/0 (socket says 172.21.15.125:35042) 2023-10-30T13:01:11.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4ab98c7640 1 -- 172.21.15.125:0/1632802560 learned_addr learned my addr 172.21.15.125:0/1632802560 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:01:11.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4ab88c5640 1 --2- 172.21.15.125:0/1632802560 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ab415cd50 0x7f4ab40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:11.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4ab90c6640 1 --2- 172.21.15.125:0/1632802560 >> v2:172.21.15.136:3300/0 conn(0x7f4ab41538f0 0x7f4ab4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:11.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4aabfff640 1 -- 172.21.15.125:0/1632802560 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647465559 0 0) 0x7f4ab415d300 con 0x7f4ab40cefa0 2023-10-30T13:01:11.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4aabfff640 1 -- 172.21.15.125:0/1632802560 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4aa0003530 con 0x7f4ab40cefa0 2023-10-30T13:01:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4ab88c5640 1 -- 172.21.15.125:0/1632802560 >> v2:172.21.15.136:3300/0 conn(0x7f4ab41538f0 msgr2=0x7f4ab4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:01:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4ab88c5640 1 --2- 172.21.15.125:0/1632802560 >> v2:172.21.15.136:3300/0 conn(0x7f4ab41538f0 0x7f4ab4159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4ab88c5640 1 -- 172.21.15.125:0/1632802560 >> v1:172.21.15.125:6789/0 conn(0x7f4ab40cefa0 legacy=0x7f4ab40cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.100+0000 7f4ab88c5640 1 -- 172.21.15.125:0/1632802560 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab415dba0 con 0x7f4ab415cd50 2023-10-30T13:01:11.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.104+0000 7f4ab88c5640 1 --2- 172.21.15.125:0/1632802560 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ab415cd50 0x7f4ab40c9f90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40097c0 tx=0x7f4aa401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=661c985c263ccf94 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:01:11.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.104+0000 7f4aabfff640 1 -- 172.21.15.125:0/1632802560 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa40a8970 con 0x7f4ab415cd50 2023-10-30T13:01:11.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.104+0000 7f4aabfff640 1 -- 172.21.15.125:0/1632802560 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa40a8ad0 con 0x7f4ab415cd50 2023-10-30T13:01:11.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.104+0000 7f4aabfff640 1 -- 172.21.15.125:0/1632802560 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa40a8e40 con 0x7f4ab415cd50 2023-10-30T13:01:11.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.104+0000 7f4aba0c8640 1 -- 172.21.15.125:0/1632802560 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ab415cd50 msgr2=0x7f4ab40c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:11.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.104+0000 7f4aba0c8640 1 --2- 172.21.15.125:0/1632802560 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ab415cd50 0x7f4ab40c9f90 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4aa40097c0 tx=0x7f4aa401ec70 comp rx=0 tx=0).stop 2023-10-30T13:01:11.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.104+0000 7f4aba0c8640 1 -- 172.21.15.125:0/1632802560 shutdown_connections 2023-10-30T13:01:11.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.104+0000 7f4aba0c8640 1 --2- 172.21.15.125:0/1632802560 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ab415cd50 0x7f4ab40c9f90 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:11.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.104+0000 7f4aba0c8640 1 --2- 172.21.15.125:0/1632802560 >> v2:172.21.15.136:3300/0 conn(0x7f4ab41538f0 0x7f4ab4159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:11.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.104+0000 7f4aba0c8640 1 -- 172.21.15.125:0/1632802560 >> 172.21.15.125:0/1632802560 conn(0x7f4ab406a640 msgr2=0x7f4ab4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:01:11.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.104+0000 7f4aba0c8640 1 -- 172.21.15.125:0/1632802560 shutdown_connections 2023-10-30T13:01:11.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.104+0000 7f4aba0c8640 1 -- 172.21.15.125:0/1632802560 wait complete. 2023-10-30T13:01:11.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4aba0c8640 1 Processor -- start 2023-10-30T13:01:11.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4aba0c8640 1 -- start start 2023-10-30T13:01:11.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4aba0c8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ab40cefa0 0x7f4ab4158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:11.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4aba0c8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ab41538f0 0x7f4ab41f0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:11.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4aba0c8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ab415cd50 0x7f4ab415a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:11.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4aba0c8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab4158e90 con 0x7f4ab415cd50 2023-10-30T13:01:11.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4aba0c8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab4159150 con 0x7f4ab41538f0 2023-10-30T13:01:11.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4aba0c8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab4159430 con 0x7f4ab40cefa0 2023-10-30T13:01:11.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4ab98c7640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ab415cd50 0x7f4ab415a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:11.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4ab98c7640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ab415cd50 0x7f4ab415a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:57160/0 (socket says 172.21.15.125:57160) 2023-10-30T13:01:11.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4ab98c7640 1 -- 172.21.15.125:0/2497220917 learned_addr learned my addr 172.21.15.125:0/2497220917 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:01:11.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4ab90c6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ab40cefa0 0x7f4ab4158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:11.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4ab88c5640 1 --2- 172.21.15.125:0/2497220917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ab41538f0 0x7f4ab41f0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:11.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4ab98c7640 1 -- 172.21.15.125:0/2497220917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ab40cefa0 msgr2=0x7f4ab4158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:11.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4ab98c7640 1 --2- 172.21.15.125:0/2497220917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ab40cefa0 0x7f4ab4158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:11.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4ab98c7640 1 -- 172.21.15.125:0/2497220917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ab41538f0 msgr2=0x7f4ab41f0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:11.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4ab98c7640 1 --2- 172.21.15.125:0/2497220917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ab41538f0 0x7f4ab41f0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:11.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4ab98c7640 1 -- 172.21.15.125:0/2497220917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa4009420 con 0x7f4ab415cd50 2023-10-30T13:01:11.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4ab98c7640 1 --2- 172.21.15.125:0/2497220917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ab415cd50 0x7f4ab415a670 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f4aac00caa0 tx=0x7f4aac00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:01:11.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4aa9ffb640 1 -- 172.21.15.125:0/2497220917 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac00ac00 con 0x7f4ab415cd50 2023-10-30T13:01:11.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4aa9ffb640 1 -- 172.21.15.125:0/2497220917 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aac00ad60 con 0x7f4ab415cd50 2023-10-30T13:01:11.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4aba0c8640 1 -- 172.21.15.125:0/2497220917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab41573a0 con 0x7f4ab415cd50 2023-10-30T13:01:11.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4aa9ffb640 1 -- 172.21.15.125:0/2497220917 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac012620 con 0x7f4ab415cd50 2023-10-30T13:01:11.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.108+0000 7f4aba0c8640 1 -- 172.21.15.125:0/2497220917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab4157960 con 0x7f4ab415cd50 2023-10-30T13:01:11.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.112+0000 7f4aba0c8640 1 -- 172.21.15.125:0/2497220917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab4159730 con 0x7f4ab415cd50 2023-10-30T13:01:11.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.112+0000 7f4aa9ffb640 1 -- 172.21.15.125:0/2497220917 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4aac007070 con 0x7f4ab415cd50 2023-10-30T13:01:11.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.112+0000 7f4aa9ffb640 1 --2- 172.21.15.125:0/2497220917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a980664a0 0x7f4a98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:01:11.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.112+0000 7f4aa9ffb640 1 -- 172.21.15.125:0/2497220917 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7f4aac084410 con 0x7f4ab415cd50 2023-10-30T13:01:11.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.112+0000 7f4ab90c6640 1 --2- 172.21.15.125:0/2497220917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a980664a0 0x7f4a98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:01:11.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.112+0000 7f4ab90c6640 1 --2- 172.21.15.125:0/2497220917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a980664a0 0x7f4a98068960 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0009b60 tx=0x7f4ab0009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:01:11.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.116+0000 7f4aa9ffb640 1 -- 172.21.15.125:0/2497220917 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4aac0512f0 con 0x7f4ab415cd50 2023-10-30T13:01:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.300+0000 7f4aba0c8640 1 -- 172.21.15.125:0/2497220917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ab404e170 con 0x7f4ab415cd50 2023-10-30T13:01:11.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.300+0000 7f4aa9ffb640 1 -- 172.21.15.125:0/2497220917 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f4aac054910 con 0x7f4ab415cd50 2023-10-30T13:01:11.300 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:01:11.300 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7866,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:01:02.384461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:01:11.301 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 150 2023-10-30T13:01:11.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.300+0000 7f4aba0c8640 1 -- 172.21.15.125:0/2497220917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a980664a0 msgr2=0x7f4a98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:11.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.300+0000 7f4aba0c8640 1 --2- 172.21.15.125:0/2497220917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a980664a0 0x7f4a98068960 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0009b60 tx=0x7f4ab0009190 comp rx=0 tx=0).stop 2023-10-30T13:01:11.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.300+0000 7f4aba0c8640 1 -- 172.21.15.125:0/2497220917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ab415cd50 msgr2=0x7f4ab415a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:01:11.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.300+0000 7f4aba0c8640 1 --2- 172.21.15.125:0/2497220917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ab415cd50 0x7f4ab415a670 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7f4aac00caa0 tx=0x7f4aac00cf60 comp rx=0 tx=0).stop 2023-10-30T13:01:11.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.300+0000 7f4aba0c8640 1 -- 172.21.15.125:0/2497220917 shutdown_connections 2023-10-30T13:01:11.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.300+0000 7f4aba0c8640 1 --2- 172.21.15.125:0/2497220917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a980664a0 0x7f4a98068960 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:11.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.300+0000 7f4aba0c8640 1 --2- 172.21.15.125:0/2497220917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ab415cd50 0x7f4ab415a670 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:11.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.300+0000 7f4aba0c8640 1 --2- 172.21.15.125:0/2497220917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ab41538f0 0x7f4ab41f0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:11.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.300+0000 7f4aba0c8640 1 --2- 172.21.15.125:0/2497220917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ab40cefa0 0x7f4ab4158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:01:11.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.300+0000 7f4aba0c8640 1 -- 172.21.15.125:0/2497220917 >> 172.21.15.125:0/2497220917 conn(0x7f4ab406a640 msgr2=0x7f4ab40cf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:01:11.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.300+0000 7f4aba0c8640 1 -- 172.21.15.125:0/2497220917 shutdown_connections 2023-10-30T13:01:11.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:01:11.300+0000 7f4aba0c8640 1 -- 172.21.15.125:0/2497220917 wait complete. 2023-10-30T13:01:11.320 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T13:01:11.321 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 69 secs before thrashing 2023-10-30T13:01:19.747 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 28% (69/246) Updating files: 29% (72/246) Updating files: 30% (74/246) Updating files: 31% (77/246) Updating files: 32% (79/246) Updating files: 33% (82/246) Updating files: 34% (84/246) Updating files: 35% (87/246) Updating files: 35% (88/246) Updating files: 36% (89/246) Updating files: 37% (92/246) Updating files: 38% (94/246) Updating files: 39% (96/246) Updating files: 40% (99/246) Updating files: 41% (101/246) Updating files: 41% (103/246) Updating files: 42% (104/246) Updating files: 43% (106/246) Updating files: 44% (109/246) Updating files: 45% (111/246) Updating files: 46% (114/246) Updating files: 47% (116/246) Updating files: 48% (119/246) Updating files: 49% (121/246) Updating files: 50% (123/246) Updating files: 51% (126/246) Updating files: 52% (128/246) Updating files: 53% (131/246) Updating files: 54% (133/246) Updating files: 55% (136/246) Updating files: 56% (138/246) Updating files: 57% (141/246) Updating files: 58% (143/246) Updating files: 59% (146/246) Updating files: 60% (148/246) Updating files: 61% (151/246) Updating files: 62% (153/246) Updating files: 63% (155/246) Updating files: 64% (158/246) Updating files: 65% (160/246) Updating files: 66% (163/246) Updating files: 67% (165/246) Updating files: 68% (168/246) Updating files: 69% (170/246) Updating files: 70% (173/246) Updating files: 71% (175/246) Updating files: 72% (178/246) Updating files: 73% (180/246) Updating files: 74% (183/246) Updating files: 74% (184/246) Updating files: 75% (185/246) Updating files: 76% (187/246) Updating files: 77% (190/246) Updating files: 78% (192/246) Updating files: 79% (195/246) Updating files: 80% (197/246) Updating files: 81% (200/246) Updating files: 82% (202/246) Updating files: 83% (205/246) Updating files: 84% (207/246) Updating files: 85% (210/246) Updating files: 86% (212/246) Updating files: 87% (215/246) Updating files: 88% (217/246) Updating files: 89% (219/246) Updating files: 90% (222/246) Updating files: 91% (224/246) Updating files: 92% (227/246) Updating files: 93% (229/246) Updating files: 94% (232/246) Updating files: 95% (234/246) Updating files: 96% (237/246) Updating files: 97% (239/246) Updating files: 98% (242/246) Updating files: 99% (244/246) Updating files: 100% (246/246) Updating files: 100% (246/246), done. 2023-10-30T13:01:19.776 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 1a32f0a0b42 v0.56 2023-10-30T13:01:19.782 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.56 2023-10-30T13:01:20.202 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 57 -eq 48 ] 2023-10-30T13:01:20.202 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.57 2023-10-30T13:01:20.203 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.57 2023-10-30T13:01:20.203 INFO:tasks.workunit.client.0.smithi125.stdout:v0.57 2023-10-30T13:01:20.203 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.57 2023-10-30T13:01:21.497 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:01:21.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:01:23.391 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 70% (215/304) Updating files: 71% (216/304) Updating files: 72% (219/304) Updating files: 73% (222/304) Updating files: 74% (225/304) Updating files: 75% (228/304) Updating files: 76% (232/304) Updating files: 77% (235/304) Updating files: 78% (238/304) Updating files: 79% (241/304) Updating files: 80% (244/304) Updating files: 81% (247/304) Updating files: 82% (250/304) Updating files: 83% (253/304) Updating files: 84% (256/304) Updating files: 85% (259/304) Updating files: 86% (262/304) Updating files: 87% (265/304) Updating files: 88% (268/304) Updating files: 89% (271/304) Updating files: 90% (274/304) Updating files: 91% (277/304) Updating files: 92% (280/304) Updating files: 93% (283/304) Updating files: 94% (286/304) Updating files: 95% (289/304) Updating files: 96% (292/304) Updating files: 97% (295/304) Updating files: 98% (298/304) Updating files: 99% (301/304) Updating files: 100% (304/304) Updating files: 100% (304/304), done. 2023-10-30T13:01:23.428 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 9a7a9d06c06 v0.57 2023-10-30T13:01:23.435 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.57 2023-10-30T13:01:23.727 INFO:tasks.workunit.client.0.smithi125.stdout:v0.58 2023-10-30T13:01:23.727 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 58 -eq 48 ] 2023-10-30T13:01:23.727 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.58 2023-10-30T13:01:23.728 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.58 2023-10-30T13:01:23.728 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.58 2023-10-30T13:01:28.279 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 8% (22/268) Updating files: 9% (25/268) Updating files: 10% (27/268) Updating files: 11% (30/268) Updating files: 12% (33/268) Updating files: 13% (35/268) Updating files: 14% (38/268) Updating files: 15% (41/268) Updating files: 16% (43/268) Updating files: 17% (46/268) Updating files: 18% (49/268) Updating files: 19% (51/268) Updating files: 20% (54/268) Updating files: 21% (57/268) Updating files: 22% (59/268) Updating files: 23% (62/268) Updating files: 24% (65/268) Updating files: 25% (67/268) Updating files: 26% (70/268) Updating files: 27% (73/268) Updating files: 28% (76/268) Updating files: 29% (78/268) Updating files: 30% (81/268) Updating files: 31% (84/268) Updating files: 32% (86/268) Updating files: 33% (89/268) Updating files: 34% (92/268) Updating files: 35% (94/268) Updating files: 36% (97/268) Updating files: 37% (100/268) Updating files: 38% (102/268) Updating files: 39% (105/268) Updating files: 40% (108/268) Updating files: 41% (110/268) Updating files: 42% (113/268) Updating files: 43% (116/268) Updating files: 44% (118/268) Updating files: 45% (121/268) Updating files: 46% (124/268) Updating files: 47% (126/268) Updating files: 48% (129/268) Updating files: 49% (132/268) Updating files: 50% (134/268) Updating files: 51% (137/268) Updating files: 52% (140/268) Updating files: 53% (143/268) Updating files: 54% (145/268) Updating files: 55% (148/268) Updating files: 56% (151/268) Updating files: 57% (153/268) Updating files: 58% (156/268) Updating files: 59% (159/268) Updating files: 60% (161/268) Updating files: 61% (164/268) Updating files: 62% (167/268) Updating files: 63% (169/268) Updating files: 64% (172/268) Updating files: 64% (173/268) Updating files: 65% (175/268) Updating files: 66% (177/268) Updating files: 67% (180/268) Updating files: 68% (183/268) Updating files: 69% (185/268) Updating files: 70% (188/268) Updating files: 71% (191/268) Updating files: 72% (193/268) Updating files: 73% (196/268) Updating files: 74% (199/268) Updating files: 75% (201/268) Updating files: 76% (204/268) Updating files: 77% (207/268) Updating files: 78% (210/268) Updating files: 79% (212/268) Updating files: 80% (215/268) Updating files: 81% (218/268) Updating files: 82% (220/268) Updating files: 83% (223/268) Updating files: 84% (226/268) Updating files: 85% (228/268) Updating files: 86% (231/268) Updating files: 87% (234/268) Updating files: 88% (236/268) Updating files: 89% (239/268) Updating files: 90% (242/268) Updating files: 91% (244/268) Updating files: 92% (247/268) Updating files: 93% (250/268) Updating files: 94% (252/268) Updating files: 95% (255/268) Updating files: 96% (258/268) Updating files: 97% (260/268) Updating files: 98% (263/268) Updating files: 99% (266/268) Updating files: 100% (268/268) Updating files: 100% (268/268), done. 2023-10-30T13:01:28.319 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at ba3f91e7504 v0.58 2023-10-30T13:01:28.326 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.58 2023-10-30T13:01:28.392 INFO:tasks.workunit.client.0.smithi125.stdout:v0.59 2023-10-30T13:01:28.393 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 59 -eq 48 ] 2023-10-30T13:01:28.393 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.59 2023-10-30T13:01:28.393 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.59 2023-10-30T13:01:28.393 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.59 2023-10-30T13:01:38.292 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 8% (17/211) Updating files: 9% (19/211) Updating files: 10% (22/211) Updating files: 11% (24/211) Updating files: 12% (26/211) Updating files: 13% (28/211) Updating files: 14% (30/211) Updating files: 15% (32/211) Updating files: 16% (34/211) Updating files: 17% (36/211) Updating files: 18% (38/211) Updating files: 19% (41/211) Updating files: 20% (43/211) Updating files: 21% (45/211) Updating files: 22% (47/211) Updating files: 23% (49/211) Updating files: 24% (51/211) Updating files: 25% (53/211) Updating files: 26% (55/211) Updating files: 27% (57/211) Updating files: 28% (60/211) Updating files: 29% (62/211) Updating files: 30% (64/211) Updating files: 31% (66/211) Updating files: 32% (68/211) Updating files: 33% (70/211) Updating files: 34% (72/211) Updating files: 35% (74/211) Updating files: 36% (76/211) Updating files: 37% (79/211) Updating files: 38% (81/211) Updating files: 39% (83/211) Updating files: 40% (85/211) Updating files: 41% (87/211) Updating files: 42% (89/211) Updating files: 43% (91/211) Updating files: 44% (93/211) Updating files: 45% (95/211) Updating files: 46% (98/211) Updating files: 47% (100/211) Updating files: 48% (102/211) Updating files: 49% (104/211) Updating files: 50% (106/211) Updating files: 51% (108/211) Updating files: 52% (110/211) Updating files: 53% (112/211) Updating files: 54% (114/211) Updating files: 55% (117/211) Updating files: 56% (119/211) Updating files: 57% (121/211) Updating files: 58% (123/211) Updating files: 59% (125/211) Updating files: 60% (127/211) Updating files: 61% (129/211) Updating files: 62% (131/211) Updating files: 63% (133/211) Updating files: 64% (136/211) Updating files: 65% (138/211) Updating files: 66% (140/211) Updating files: 67% (142/211) Updating files: 68% (144/211) Updating files: 69% (146/211) Updating files: 70% (148/211) Updating files: 71% (150/211) Updating files: 72% (152/211) Updating files: 73% (155/211) Updating files: 74% (157/211) Updating files: 75% (159/211) Updating files: 76% (161/211) Updating files: 77% (163/211) Updating files: 78% (165/211) Updating files: 79% (167/211) Updating files: 80% (169/211) Updating files: 81% (171/211) Updating files: 82% (174/211) Updating files: 83% (176/211) Updating files: 84% (178/211) Updating files: 85% (180/211) Updating files: 86% (182/211) Updating files: 87% (184/211) Updating files: 88% (186/211) Updating files: 89% (188/211) Updating files: 90% (190/211) Updating files: 91% (193/211) Updating files: 92% (195/211) Updating files: 93% (197/211) Updating files: 94% (199/211) Updating files: 95% (201/211) Updating files: 96% (203/211) Updating files: 97% (205/211) Updating files: 98% (207/211) Updating files: 99% (209/211) Updating files: 100% (211/211) Updating files: 100% (211/211), done. 2023-10-30T13:01:38.745 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at cbae6a435c6 v0.59 2023-10-30T13:01:38.751 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.59 2023-10-30T13:01:39.082 INFO:tasks.workunit.client.0.smithi125.stdout:v0.60 2023-10-30T13:01:39.083 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 60 -eq 48 ] 2023-10-30T13:01:39.083 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.60 2023-10-30T13:01:39.083 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.60 2023-10-30T13:01:39.083 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.60 2023-10-30T13:01:41.955 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 38% (98/256) Updating files: 39% (100/256) Updating files: 40% (103/256) Updating files: 41% (105/256) Updating files: 42% (108/256) Updating files: 43% (111/256) Updating files: 44% (113/256) Updating files: 45% (116/256) Updating files: 46% (118/256) Updating files: 46% (119/256) Updating files: 47% (121/256) Updating files: 48% (123/256) Updating files: 49% (126/256) Updating files: 50% (128/256) Updating files: 51% (131/256) Updating files: 52% (134/256) Updating files: 53% (136/256) Updating files: 54% (139/256) Updating files: 55% (141/256) Updating files: 56% (144/256) Updating files: 57% (146/256) Updating files: 58% (149/256) Updating files: 59% (152/256) Updating files: 60% (154/256) Updating files: 61% (157/256) Updating files: 62% (159/256) Updating files: 63% (162/256) Updating files: 64% (164/256) Updating files: 65% (167/256) Updating files: 66% (169/256) Updating files: 67% (172/256) Updating files: 68% (175/256) Updating files: 69% (177/256) Updating files: 70% (180/256) Updating files: 71% (182/256) Updating files: 72% (185/256) Updating files: 73% (187/256) Updating files: 74% (190/256) Updating files: 75% (192/256) Updating files: 76% (195/256) Updating files: 77% (198/256) Updating files: 78% (200/256) Updating files: 79% (203/256) Updating files: 80% (205/256) Updating files: 81% (208/256) Updating files: 82% (210/256) Updating files: 83% (213/256) Updating files: 84% (216/256) Updating files: 85% (218/256) Updating files: 86% (221/256) Updating files: 87% (223/256) Updating files: 88% (226/256) Updating files: 89% (228/256) Updating files: 90% (231/256) Updating files: 91% (233/256) Updating files: 92% (236/256) Updating files: 93% (239/256) Updating files: 94% (241/256) Updating files: 95% (244/256) Updating files: 96% (246/256) Updating files: 97% (249/256) Updating files: 98% (251/256) Updating files: 99% (254/256) Updating files: 100% (256/256) Updating files: 100% (256/256), done. 2023-10-30T13:01:41.994 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at f26f7a39021 v0.60 2023-10-30T13:01:42.002 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.60 2023-10-30T13:01:42.646 INFO:tasks.workunit.client.0.smithi125.stdout:v0.61 2023-10-30T13:01:42.646 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 61 -eq 48 ] 2023-10-30T13:01:42.646 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.61 2023-10-30T13:01:42.647 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.61 2023-10-30T13:01:42.647 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.61 2023-10-30T13:01:51.580 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:01:51.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:01:59.982 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 1% (7/376) Updating files: 2% (8/376) Updating files: 3% (12/376) Updating files: 3% (14/376) Updating files: 4% (16/376) Updating files: 5% (19/376) Updating files: 6% (23/376) Updating files: 7% (27/376) Updating files: 8% (31/376) Updating files: 9% (34/376) Updating files: 10% (38/376) Updating files: 11% (42/376) Updating files: 12% (46/376) Updating files: 13% (49/376) Updating files: 14% (53/376) Updating files: 15% (57/376) Updating files: 16% (61/376) Updating files: 17% (64/376) Updating files: 18% (68/376) Updating files: 19% (72/376) Updating files: 19% (75/376) Updating files: 20% (76/376) Updating files: 21% (79/376) Updating files: 22% (83/376) Updating files: 23% (87/376) Updating files: 24% (91/376) Updating files: 25% (94/376) Updating files: 26% (98/376) Updating files: 27% (102/376) Updating files: 28% (106/376) Updating files: 29% (110/376) Updating files: 30% (113/376) Updating files: 31% (117/376) Updating files: 32% (121/376) Updating files: 33% (125/376) Updating files: 34% (128/376) Updating files: 35% (132/376) Updating files: 36% (136/376) Updating files: 36% (137/376) Updating files: 37% (140/376) Updating files: 38% (143/376) Updating files: 39% (147/376) Updating files: 40% (151/376) Updating files: 41% (155/376) Updating files: 42% (158/376) Updating files: 43% (162/376) Updating files: 44% (166/376) Updating files: 45% (170/376) Updating files: 46% (173/376) Updating files: 47% (177/376) Updating files: 48% (181/376) Updating files: 49% (185/376) Updating files: 50% (188/376) Updating files: 51% (192/376) Updating files: 52% (196/376) Updating files: 53% (200/376) Updating files: 54% (204/376) Updating files: 54% (206/376) Updating files: 55% (207/376) Updating files: 56% (211/376) Updating files: 56% (212/376) Updating files: 57% (215/376) Updating files: 58% (219/376) Updating files: 59% (222/376) Updating files: 60% (226/376) Updating files: 61% (230/376) Updating files: 62% (234/376) Updating files: 63% (237/376) Updating files: 64% (241/376) Updating files: 65% (245/376) Updating files: 65% (247/376) Updating files: 66% (249/376) Updating files: 67% (252/376) Updating files: 68% (256/376) Updating files: 69% (260/376) Updating files: 70% (264/376) Updating files: 71% (267/376) Updating files: 72% (271/376) Updating files: 73% (275/376) Updating files: 74% (279/376) Updating files: 75% (282/376) Updating files: 76% (286/376) Updating files: 77% (290/376) Updating files: 78% (294/376) Updating files: 79% (298/376) Updating files: 80% (301/376) Updating files: 81% (305/376) Updating files: 82% (309/376) Updating files: 83% (313/376) Updating files: 84% (316/376) Updating files: 85% (320/376) Updating files: 86% (324/376) Updating files: 87% (328/376) Updating files: 88% (331/376) Updating files: 89% (335/376) Updating files: 90% (339/376) Updating files: 91% (343/376) Updating files: 92% (346/376) Updating files: 93% (350/376) Updating files: 94% (354/376) Updating files: 95% (358/376) Updating files: 96% (361/376) Updating files: 97% (365/376) Updating files: 98% (369/376) Updating files: 99% (373/376) Updating files: 100% (376/376) Updating files: 100% (376/376), done. 2023-10-30T13:02:00.352 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 237f3f1e8d8 v0.61 2023-10-30T13:02:00.361 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.61 2023-10-30T13:02:00.643 INFO:tasks.workunit.client.0.smithi125.stdout:v0.62 2023-10-30T13:02:00.643 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 62 -eq 48 ] 2023-10-30T13:02:00.643 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.62 2023-10-30T13:02:00.643 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.62 2023-10-30T13:02:00.644 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.62 2023-10-30T13:02:03.363 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 83% (49/59) Updating files: 84% (50/59) Updating files: 86% (51/59) Updating files: 88% (52/59) Updating files: 89% (53/59) Updating files: 91% (54/59) Updating files: 93% (55/59) Updating files: 94% (56/59) Updating files: 96% (57/59) Updating files: 98% (58/59) Updating files: 100% (59/59) Updating files: 100% (59/59), done. 2023-10-30T13:02:03.419 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 9382379cc6b v0.62 2023-10-30T13:02:03.422 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.62 2023-10-30T13:02:03.478 INFO:tasks.workunit.client.0.smithi125.stdout:v0.63 2023-10-30T13:02:03.479 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 63 -eq 48 ] 2023-10-30T13:02:03.479 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.63 2023-10-30T13:02:03.479 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.63 2023-10-30T13:02:03.479 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.63 2023-10-30T13:02:07.271 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 68% (100/146) Updating files: 69% (101/146) Updating files: 70% (103/146) Updating files: 71% (104/146) Updating files: 72% (106/146) Updating files: 73% (107/146) Updating files: 74% (109/146) Updating files: 75% (110/146) Updating files: 76% (111/146) Updating files: 77% (113/146) Updating files: 78% (114/146) Updating files: 79% (116/146) Updating files: 80% (117/146) Updating files: 81% (119/146) Updating files: 82% (120/146) Updating files: 83% (122/146) Updating files: 84% (123/146) Updating files: 85% (125/146) Updating files: 86% (126/146) Updating files: 87% (128/146) Updating files: 88% (129/146) Updating files: 89% (130/146) Updating files: 90% (132/146) Updating files: 91% (133/146) Updating files: 92% (135/146) Updating files: 93% (136/146) Updating files: 94% (138/146) Updating files: 95% (139/146) Updating files: 96% (141/146) Updating files: 97% (142/146) Updating files: 98% (144/146) Updating files: 99% (145/146) Updating files: 100% (146/146) Updating files: 100% (146/146), done. 2023-10-30T13:02:07.321 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 054e96cf79e v0.63 2023-10-30T13:02:07.328 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.63 2023-10-30T13:02:07.377 INFO:tasks.workunit.client.0.smithi125.stdout:v0.64 2023-10-30T13:02:07.377 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 64 -eq 48 ] 2023-10-30T13:02:07.378 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.64 2023-10-30T13:02:07.378 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.64 2023-10-30T13:02:07.378 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.64 2023-10-30T13:02:10.155 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 11% (17/143) Updating files: 12% (18/143) Updating files: 13% (19/143) Updating files: 14% (21/143) Updating files: 15% (22/143) Updating files: 16% (23/143) Updating files: 17% (25/143) Updating files: 18% (26/143) Updating files: 19% (28/143) Updating files: 20% (29/143) Updating files: 21% (31/143) Updating files: 22% (32/143) Updating files: 23% (33/143) Updating files: 24% (35/143) Updating files: 25% (36/143) Updating files: 26% (38/143) Updating files: 27% (39/143) Updating files: 28% (41/143) Updating files: 29% (42/143) Updating files: 30% (43/143) Updating files: 31% (45/143) Updating files: 32% (46/143) Updating files: 33% (48/143) Updating files: 34% (49/143) Updating files: 35% (51/143) Updating files: 36% (52/143) Updating files: 37% (53/143) Updating files: 38% (55/143) Updating files: 39% (56/143) Updating files: 40% (58/143) Updating files: 41% (59/143) Updating files: 42% (61/143) Updating files: 43% (62/143) Updating files: 44% (63/143) Updating files: 45% (65/143) Updating files: 46% (66/143) Updating files: 47% (68/143) Updating files: 48% (69/143) Updating files: 49% (71/143) Updating files: 50% (72/143) Updating files: 51% (73/143) Updating files: 52% (75/143) Updating files: 53% (76/143) Updating files: 54% (78/143) Updating files: 55% (79/143) Updating files: 56% (81/143) Updating files: 57% (82/143) Updating files: 58% (83/143) Updating files: 59% (85/143) Updating files: 60% (86/143) Updating files: 61% (88/143) Updating files: 62% (89/143) Updating files: 63% (91/143) Updating files: 64% (92/143) Updating files: 65% (93/143) Updating files: 66% (95/143) Updating files: 67% (96/143) Updating files: 68% (98/143) Updating files: 69% (99/143) Updating files: 70% (101/143) Updating files: 71% (102/143) Updating files: 72% (103/143) Updating files: 73% (105/143) Updating files: 74% (106/143) Updating files: 75% (108/143) Updating files: 76% (109/143) Updating files: 77% (111/143) Updating files: 78% (112/143) Updating files: 79% (113/143) Updating files: 80% (115/143) Updating files: 81% (116/143) Updating files: 82% (118/143) Updating files: 83% (119/143) Updating files: 84% (121/143) Updating files: 85% (122/143) Updating files: 86% (123/143) Updating files: 87% (125/143) Updating files: 88% (126/143) Updating files: 89% (128/143) Updating files: 90% (129/143) Updating files: 91% (131/143) Updating files: 92% (132/143) Updating files: 93% (133/143) Updating files: 94% (135/143) Updating files: 95% (136/143) Updating files: 96% (138/143) Updating files: 97% (139/143) Updating files: 98% (141/143) Updating files: 99% (142/143) Updating files: 100% (143/143) Updating files: 100% (143/143), done. 2023-10-30T13:02:10.199 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 42e06c12db6 v0.64 2023-10-30T13:02:10.205 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.64 2023-10-30T13:02:10.293 INFO:tasks.workunit.client.0.smithi125.stdout:v0.65 2023-10-30T13:02:10.294 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 65 -eq 48 ] 2023-10-30T13:02:10.294 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.65 2023-10-30T13:02:10.294 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.65 2023-10-30T13:02:10.294 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.65 2023-10-30T13:02:14.240 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 6% (13/203) Updating files: 7% (15/203) Updating files: 8% (17/203) Updating files: 9% (19/203) Updating files: 10% (21/203) Updating files: 11% (23/203) Updating files: 12% (25/203) Updating files: 13% (27/203) Updating files: 14% (29/203) Updating files: 15% (31/203) Updating files: 16% (33/203) Updating files: 17% (35/203) Updating files: 18% (37/203) Updating files: 19% (39/203) Updating files: 20% (41/203) Updating files: 21% (43/203) Updating files: 22% (45/203) Updating files: 23% (47/203) Updating files: 24% (49/203) Updating files: 25% (51/203) Updating files: 26% (53/203) Updating files: 27% (55/203) Updating files: 28% (57/203) Updating files: 29% (59/203) Updating files: 30% (61/203) Updating files: 31% (63/203) Updating files: 32% (65/203) Updating files: 33% (67/203) Updating files: 34% (70/203) Updating files: 35% (72/203) Updating files: 36% (74/203) Updating files: 37% (76/203) Updating files: 38% (78/203) Updating files: 39% (80/203) Updating files: 40% (82/203) Updating files: 41% (84/203) Updating files: 42% (86/203) Updating files: 43% (88/203) Updating files: 44% (90/203) Updating files: 45% (92/203) Updating files: 46% (94/203) Updating files: 47% (96/203) Updating files: 48% (98/203) Updating files: 49% (100/203) Updating files: 50% (102/203) Updating files: 51% (104/203) Updating files: 52% (106/203) Updating files: 53% (108/203) Updating files: 54% (110/203) Updating files: 55% (112/203) Updating files: 56% (114/203) Updating files: 57% (116/203) Updating files: 58% (118/203) Updating files: 59% (120/203) Updating files: 60% (122/203) Updating files: 61% (124/203) Updating files: 62% (126/203) Updating files: 63% (128/203) Updating files: 64% (130/203) Updating files: 65% (132/203) Updating files: 66% (134/203) Updating files: 67% (137/203) Updating files: 68% (139/203) Updating files: 69% (141/203) Updating files: 70% (143/203) Updating files: 71% (145/203) Updating files: 72% (147/203) Updating files: 73% (149/203) Updating files: 74% (151/203) Updating files: 75% (153/203) Updating files: 76% (155/203) Updating files: 77% (157/203) Updating files: 78% (159/203) Updating files: 79% (161/203) Updating files: 80% (163/203) Updating files: 81% (165/203) Updating files: 82% (167/203) Updating files: 83% (169/203) Updating files: 84% (171/203) Updating files: 85% (173/203) Updating files: 86% (175/203) Updating files: 87% (177/203) Updating files: 88% (179/203) Updating files: 89% (181/203) Updating files: 90% (183/203) Updating files: 91% (185/203) Updating files: 92% (187/203) Updating files: 93% (189/203) Updating files: 94% (191/203) Updating files: 95% (193/203) Updating files: 96% (195/203) Updating files: 97% (197/203) Updating files: 98% (199/203) Updating files: 99% (201/203) Updating files: 100% (203/203) Updating files: 100% (203/203), done. 2023-10-30T13:02:14.272 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at c2d517ef966 v0.65 2023-10-30T13:02:14.277 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.65 2023-10-30T13:02:14.590 INFO:tasks.workunit.client.0.smithi125.stdout:v0.66 2023-10-30T13:02:14.591 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 66 -eq 48 ] 2023-10-30T13:02:14.591 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.66 2023-10-30T13:02:14.591 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.66 2023-10-30T13:02:14.591 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.66 2023-10-30T13:02:20.321 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:20.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.466+0000 7f1f6d46b640 1 Processor -- start 2023-10-30T13:02:20.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.466+0000 7f1f6d46b640 1 -- start start 2023-10-30T13:02:20.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.466+0000 7f1f6d46b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1f680c05d0 0x7f1f6813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:20.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.466+0000 7f1f6d46b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f680c0a50 con 0x7f1f680c6f60 2023-10-30T13:02:20.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.466+0000 7f1f6d46b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f6806f6e0 con 0x7f1f680c6b80 2023-10-30T13:02:20.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.466+0000 7f1f6d46b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f6806f9c0 con 0x7f1f680c05d0 2023-10-30T13:02:20.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f677fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1f680c6f60 0x7f1f681566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60540/0 (socket says 172.21.15.125:60540) 2023-10-30T13:02:20.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f677fe640 1 -- 172.21.15.125:0/3347196557 learned_addr learned my addr 172.21.15.125:0/3347196557 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:02:20.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f5ffff640 1 --2- 172.21.15.125:0/3347196557 >> v2:172.21.15.136:3300/0 conn(0x7f1f680c05d0 0x7f1f6813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:20.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032542730 0 0) 0x7f1f680c0a50 con 0x7f1f680c6f60 2023-10-30T13:02:20.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f54005740 con 0x7f1f680c6f60 2023-10-30T13:02:20.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555046224 0 0) 0x7f1f6806f6e0 con 0x7f1f680c6b80 2023-10-30T13:02:20.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f680c0a50 con 0x7f1f680c6b80 2023-10-30T13:02:20.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 242077188 0 0) 0x7f1f54005740 con 0x7f1f680c6f60 2023-10-30T13:02:20.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 >> v2:172.21.15.136:3300/0 conn(0x7f1f680c05d0 msgr2=0x7f1f6813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:20.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 --2- 172.21.15.125:0/3347196557 >> v2:172.21.15.136:3300/0 conn(0x7f1f680c05d0 0x7f1f6813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:20.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 >> v1:172.21.15.136:6789/0 conn(0x7f1f680c6b80 legacy=0x7f1f680bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:20.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6806fca0 con 0x7f1f680c6f60 2023-10-30T13:02:20.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f1f60001a00 con 0x7f1f680c6f60 2023-10-30T13:02:20.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 >> v1:172.21.15.125:6789/0 conn(0x7f1f680c6f60 legacy=0x7f1f681566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:20.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 --2- 172.21.15.125:0/3347196557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f54005d20 0x7f1f54006120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:20.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 --2- 172.21.15.125:0/3347196557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1f540069e0 0x7f1f54006e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:20.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 --2- 172.21.15.125:0/3347196557 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f540073a0 0x7f1f5400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:20.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6806f9c0 con 0x7f1f54005d20 2023-10-30T13:02:20.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5400e220 con 0x7f1f540073a0 2023-10-30T13:02:20.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f5400e3e0 con 0x7f1f540069e0 2023-10-30T13:02:20.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f66ffd640 1 --2- 172.21.15.125:0/3347196557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f54005d20 0x7f1f54006120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:20.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f5ffff640 1 --2- 172.21.15.125:0/3347196557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1f540069e0 0x7f1f54006e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:20.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f677fe640 1 --2- 172.21.15.125:0/3347196557 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f540073a0 0x7f1f5400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:20.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f66ffd640 1 -- 172.21.15.125:0/3347196557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1f540069e0 msgr2=0x7f1f54006e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:20.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f66ffd640 1 --2- 172.21.15.125:0/3347196557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1f540069e0 0x7f1f54006e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:20.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f66ffd640 1 -- 172.21.15.125:0/3347196557 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f540073a0 msgr2=0x7f1f5400dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:20.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f66ffd640 1 --2- 172.21.15.125:0/3347196557 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f540073a0 0x7f1f5400dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:20.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f66ffd640 1 -- 172.21.15.125:0/3347196557 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f54005740 con 0x7f1f54005d20 2023-10-30T13:02:20.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.470+0000 7f1f677fe640 1 -- 172.21.15.125:0/3347196557 reap_dead start 2023-10-30T13:02:20.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.474+0000 7f1f66ffd640 1 --2- 172.21.15.125:0/3347196557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f54005d20 0x7f1f54006120 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f1f580018d0 tx=0x7f1f5801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd12134e273819f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:20.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.474+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f580052d0 con 0x7f1f54005d20 2023-10-30T13:02:20.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.474+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f580a7050 con 0x7f1f54005d20 2023-10-30T13:02:20.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.474+0000 7f1f667fc640 1 -- 172.21.15.125:0/3347196557 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58003600 con 0x7f1f54005d20 2023-10-30T13:02:20.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.474+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3347196557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f54005d20 msgr2=0x7f1f54006120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:20.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.474+0000 7f1f6d46b640 1 --2- 172.21.15.125:0/3347196557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f54005d20 0x7f1f54006120 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f1f580018d0 tx=0x7f1f5801c540 comp rx=0 tx=0).stop 2023-10-30T13:02:20.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.474+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3347196557 shutdown_connections 2023-10-30T13:02:20.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.474+0000 7f1f6d46b640 1 --2- 172.21.15.125:0/3347196557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f54005d20 0x7f1f54006120 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:20.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.474+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3347196557 >> 172.21.15.125:0/3347196557 conn(0x7f1f680c7910 msgr2=0x7f1f680c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:20.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.474+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3347196557 shutdown_connections 2023-10-30T13:02:20.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.474+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3347196557 wait complete. 2023-10-30T13:02:20.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.474+0000 7f1f6d46b640 1 Processor -- start 2023-10-30T13:02:20.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.474+0000 7f1f6d46b640 1 -- start start 2023-10-30T13:02:20.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f6d46b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f54005d20 0x7f1f681380d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:20.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f6d46b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1f681e8ec0 0x7f1f681e9300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:20.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f6d46b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f6813c620 0x7f1f6813caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:20.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f6d46b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6813e480 con 0x7f1f54005d20 2023-10-30T13:02:20.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f6d46b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f681f65f0 con 0x7f1f6813c620 2023-10-30T13:02:20.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f6d46b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6813bfc0 con 0x7f1f681e8ec0 2023-10-30T13:02:20.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f66ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f54005d20 0x7f1f681380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:20.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f5ffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1f681e8ec0 0x7f1f681e9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:20.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f5ffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1f681e8ec0 0x7f1f681e9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:56102/0 (socket says 172.21.15.125:56102) 2023-10-30T13:02:20.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f5ffff640 1 -- 172.21.15.125:0/3463576839 learned_addr learned my addr 172.21.15.125:0/3463576839 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:20.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f66ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f54005d20 0x7f1f681380d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46510/0 (socket says 172.21.15.125:46510) 2023-10-30T13:02:20.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f677fe640 1 --2- 172.21.15.125:0/3463576839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f6813c620 0x7f1f6813caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:20.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f66ffd640 1 -- 172.21.15.125:0/3463576839 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1f681e8ec0 msgr2=0x7f1f681e9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f66ffd640 1 --2- 172.21.15.125:0/3463576839 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1f681e8ec0 0x7f1f681e9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f66ffd640 1 -- 172.21.15.125:0/3463576839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f6813c620 msgr2=0x7f1f6813caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f66ffd640 1 --2- 172.21.15.125:0/3463576839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f6813c620 0x7f1f6813caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f66ffd640 1 -- 172.21.15.125:0/3463576839 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6813cfe0 con 0x7f1f54005d20 2023-10-30T13:02:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f66ffd640 1 --2- 172.21.15.125:0/3463576839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f54005d20 0x7f1f681380d0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f1f58004000 tx=0x7f1f54007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f5f7fe640 1 -- 172.21.15.125:0/3463576839 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f58003ad0 con 0x7f1f54005d20 2023-10-30T13:02:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f5f7fe640 1 -- 172.21.15.125:0/3463576839 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f58003c30 con 0x7f1f54005d20 2023-10-30T13:02:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f5f7fe640 1 -- 172.21.15.125:0/3463576839 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f580067e0 con 0x7f1f54005d20 2023-10-30T13:02:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3463576839 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f681f5700 con 0x7f1f54005d20 2023-10-30T13:02:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3463576839 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f681f5cc0 con 0x7f1f54005d20 2023-10-30T13:02:20.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.478+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3463576839 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6804e170 con 0x7f1f54005d20 2023-10-30T13:02:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.482+0000 7f1f5f7fe640 1 -- 172.21.15.125:0/3463576839 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1f58006940 con 0x7f1f54005d20 2023-10-30T13:02:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.482+0000 7f1f5f7fe640 1 --2- 172.21.15.125:0/3463576839 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1f480664a0 0x7f1f48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.482+0000 7f1f5f7fe640 1 -- 172.21.15.125:0/3463576839 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7f1f5811d8a0 con 0x7f1f54005d20 2023-10-30T13:02:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.482+0000 7f1f5ffff640 1 --2- 172.21.15.125:0/3463576839 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1f480664a0 0x7f1f48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.482+0000 7f1f5ffff640 1 --2- 172.21.15.125:0/3463576839 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1f480664a0 0x7f1f48068960 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1f680af9f0 tx=0x7f1f50008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.486+0000 7f1f5f7fe640 1 -- 172.21.15.125:0/3463576839 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1f580ea780 con 0x7f1f54005d20 2023-10-30T13:02:20.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.674+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3463576839 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f6806a310 con 0x7f1f54005d20 2023-10-30T13:02:20.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.674+0000 7f1f5f7fe640 1 -- 172.21.15.125:0/3463576839 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f1f580ba090 con 0x7f1f54005d20 2023-10-30T13:02:20.677 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:20.677 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7866,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:01:02.384461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:20.677 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 150 2023-10-30T13:02:20.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.674+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3463576839 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1f480664a0 msgr2=0x7f1f48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:20.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.674+0000 7f1f6d46b640 1 --2- 172.21.15.125:0/3463576839 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1f480664a0 0x7f1f48068960 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f1f680af9f0 tx=0x7f1f50008040 comp rx=0 tx=0).stop 2023-10-30T13:02:20.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.674+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3463576839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f54005d20 msgr2=0x7f1f681380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:20.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.674+0000 7f1f6d46b640 1 --2- 172.21.15.125:0/3463576839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f54005d20 0x7f1f681380d0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f1f58004000 tx=0x7f1f54007a10 comp rx=0 tx=0).stop 2023-10-30T13:02:20.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.674+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3463576839 shutdown_connections 2023-10-30T13:02:20.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.674+0000 7f1f6d46b640 1 --2- 172.21.15.125:0/3463576839 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1f480664a0 0x7f1f48068960 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:20.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.674+0000 7f1f6d46b640 1 --2- 172.21.15.125:0/3463576839 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1f6813c620 0x7f1f6813caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:20.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.674+0000 7f1f6d46b640 1 --2- 172.21.15.125:0/3463576839 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1f681e8ec0 0x7f1f681e9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:20.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.674+0000 7f1f6d46b640 1 --2- 172.21.15.125:0/3463576839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1f54005d20 0x7f1f681380d0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:20.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.674+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3463576839 >> 172.21.15.125:0/3463576839 conn(0x7f1f680c7910 msgr2=0x7f1f6815a470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:20.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.674+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3463576839 shutdown_connections 2023-10-30T13:02:20.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.674+0000 7f1f6d46b640 1 -- 172.21.15.125:0/3463576839 wait complete. 2023-10-30T13:02:20.697 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T13:02:20.697 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:02:20.698 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:02:20.727 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:02:20.728 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:20.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd885479640 1 Processor -- start 2023-10-30T13:02:20.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd885479640 1 -- start start 2023-10-30T13:02:20.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd885479640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd88015d2c0 0x7fd88015d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:20.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd885479640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88015ce30 con 0x7fd8800c90d0 2023-10-30T13:02:20.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd885479640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88015dcb0 con 0x7fd8800c8cf0 2023-10-30T13:02:20.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd885479640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd880159880 con 0x7fd88015d2c0 2023-10-30T13:02:20.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd87e7fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd8800c90d0 0x7fd8800cef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39534/0 (socket says 172.21.15.125:39534) 2023-10-30T13:02:20.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd87e7fc640 1 -- 172.21.15.125:0/2810919412 learned_addr learned my addr 172.21.15.125:0/2810919412 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:02:20.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd87f7fe640 1 --2- 172.21.15.125:0/2810919412 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd88015d2c0 0x7fd88015d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd87dffb640 1 -- 172.21.15.125:0/2810919412 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46399869 0 0) 0x7fd88015ce30 con 0x7fd8800c90d0 2023-10-30T13:02:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd87dffb640 1 -- 172.21.15.125:0/2810919412 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd868003530 con 0x7fd8800c90d0 2023-10-30T13:02:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd87dffb640 1 -- 172.21.15.125:0/2810919412 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2201007270 0 0) 0x7fd868003530 con 0x7fd8800c90d0 2023-10-30T13:02:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd87f7fe640 1 -- 172.21.15.125:0/2810919412 >> v1:172.21.15.136:6789/0 conn(0x7fd8800c8cf0 legacy=0x7fd8800c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:20.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd87f7fe640 1 -- 172.21.15.125:0/2810919412 >> v1:172.21.15.125:6789/0 conn(0x7fd8800c90d0 legacy=0x7fd8800cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.906+0000 7fd87f7fe640 1 -- 172.21.15.125:0/2810919412 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd880159ae0 con 0x7fd88015d2c0 2023-10-30T13:02:20.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.910+0000 7fd87f7fe640 1 --2- 172.21.15.125:0/2810919412 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd88015d2c0 0x7fd88015d770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd878009dc0 tx=0x7fd87801c710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81f7a61bcb98674e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:20.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.910+0000 7fd87dffb640 1 -- 172.21.15.125:0/2810919412 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd874001680 con 0x7fd88015d2c0 2023-10-30T13:02:20.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.910+0000 7fd87dffb640 1 -- 172.21.15.125:0/2810919412 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8780a5550 con 0x7fd88015d2c0 2023-10-30T13:02:20.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.910+0000 7fd87dffb640 1 -- 172.21.15.125:0/2810919412 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8780a58e0 con 0x7fd88015d2c0 2023-10-30T13:02:20.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.910+0000 7fd885479640 1 -- 172.21.15.125:0/2810919412 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd88015d2c0 msgr2=0x7fd88015d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:20.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.910+0000 7fd885479640 1 --2- 172.21.15.125:0/2810919412 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd88015d2c0 0x7fd88015d770 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd878009dc0 tx=0x7fd87801c710 comp rx=0 tx=0).stop 2023-10-30T13:02:20.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.910+0000 7fd885479640 1 -- 172.21.15.125:0/2810919412 shutdown_connections 2023-10-30T13:02:20.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.910+0000 7fd885479640 1 --2- 172.21.15.125:0/2810919412 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd88015d2c0 0x7fd88015d770 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:20.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.910+0000 7fd885479640 1 -- 172.21.15.125:0/2810919412 >> 172.21.15.125:0/2810919412 conn(0x7fd88006be40 msgr2=0x7fd88006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:20.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.910+0000 7fd885479640 1 -- 172.21.15.125:0/2810919412 shutdown_connections 2023-10-30T13:02:20.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.910+0000 7fd885479640 1 -- 172.21.15.125:0/2810919412 wait complete. 2023-10-30T13:02:20.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd885479640 1 Processor -- start 2023-10-30T13:02:20.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd885479640 1 -- start start 2023-10-30T13:02:20.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd885479640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8800c8cf0 0x7fd88014c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:20.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd885479640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8800c90d0 0x7fd8800ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:20.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd885479640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd88015d2c0 0x7fd88015b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:20.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd885479640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8801577a0 con 0x7fd8800c8cf0 2023-10-30T13:02:20.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd885479640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd88014dee0 con 0x7fd8800c90d0 2023-10-30T13:02:20.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd885479640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8800b2f30 con 0x7fd88015d2c0 2023-10-30T13:02:20.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd87effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8800c8cf0 0x7fd88014c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:20.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd87effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8800c8cf0 0x7fd88014c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38550/0 (socket says 172.21.15.125:38550) 2023-10-30T13:02:20.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd87effd640 1 -- 172.21.15.125:0/3070683077 learned_addr learned my addr 172.21.15.125:0/3070683077 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:20.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd87f7fe640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd88015d2c0 0x7fd88015b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:20.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd87e7fc640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8800c90d0 0x7fd8800ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:20.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd87effd640 1 -- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd88015d2c0 msgr2=0x7fd88015b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:20.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd87effd640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd88015d2c0 0x7fd88015b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:20.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd87effd640 1 -- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8800c90d0 msgr2=0x7fd8800ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:20.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd87effd640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8800c90d0 0x7fd8800ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:20.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd87effd640 1 -- 172.21.15.125:0/3070683077 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd878009a70 con 0x7fd8800c8cf0 2023-10-30T13:02:20.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd87f7fe640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd88015d2c0 0x7fd88015b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:02:20.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd87effd640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8800c8cf0 0x7fd88014c2d0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fd87000c530 tx=0x7fd87000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:20.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd8677fe640 1 -- 172.21.15.125:0/3070683077 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd870005480 con 0x7fd8800c8cf0 2023-10-30T13:02:20.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd8677fe640 1 -- 172.21.15.125:0/3070683077 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd870003600 con 0x7fd8800c8cf0 2023-10-30T13:02:20.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd8677fe640 1 -- 172.21.15.125:0/3070683077 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8700039b0 con 0x7fd8800c8cf0 2023-10-30T13:02:20.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd885479640 1 -- 172.21.15.125:0/3070683077 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8800b3210 con 0x7fd8800c8cf0 2023-10-30T13:02:20.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd885479640 1 -- 172.21.15.125:0/3070683077 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8800c2650 con 0x7fd8800c8cf0 2023-10-30T13:02:20.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.914+0000 7fd885479640 1 -- 172.21.15.125:0/3070683077 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8800b1b30 con 0x7fd8800c8cf0 2023-10-30T13:02:20.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.918+0000 7fd8677fe640 1 -- 172.21.15.125:0/3070683077 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd870003b10 con 0x7fd8800c8cf0 2023-10-30T13:02:20.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.918+0000 7fd8677fe640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd86c0664a0 0x7fd86c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:20.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.918+0000 7fd8677fe640 1 -- 172.21.15.125:0/3070683077 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7fd870083b90 con 0x7fd8800c8cf0 2023-10-30T13:02:20.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.918+0000 7fd87e7fc640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd86c0664a0 0x7fd86c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:20.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.918+0000 7fd87e7fc640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd86c0664a0 0x7fd86c068960 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd874001ec0 tx=0x7fd874001c90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:20.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:20.922+0000 7fd8677fe640 1 -- 172.21.15.125:0/3070683077 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd870050930 con 0x7fd8800c8cf0 2023-10-30T13:02:21.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:21.106+0000 7fd885479640 1 -- 172.21.15.125:0/3070683077 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd88004e170 con 0x7fd8800c8cf0 2023-10-30T13:02:21.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:21.106+0000 7fd8677fe640 1 -- 172.21.15.125:0/3070683077 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fd870017020 con 0x7fd8800c8cf0 2023-10-30T13:02:21.109 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:21.109 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7866,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:01:02.384461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:21.110 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 150 2023-10-30T13:02:21.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:21.106+0000 7fd885479640 1 -- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd86c0664a0 msgr2=0x7fd86c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:21.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:21.106+0000 7fd885479640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd86c0664a0 0x7fd86c068960 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd874001ec0 tx=0x7fd874001c90 comp rx=0 tx=0).stop 2023-10-30T13:02:21.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:21.106+0000 7fd885479640 1 -- 172.21.15.125:0/3070683077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8800c8cf0 msgr2=0x7fd88014c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:21.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:21.106+0000 7fd885479640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8800c8cf0 0x7fd88014c2d0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7fd87000c530 tx=0x7fd87000c9f0 comp rx=0 tx=0).stop 2023-10-30T13:02:21.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:21.106+0000 7fd885479640 1 -- 172.21.15.125:0/3070683077 shutdown_connections 2023-10-30T13:02:21.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:21.106+0000 7fd885479640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd86c0664a0 0x7fd86c068960 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:21.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:21.106+0000 7fd885479640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd88015d2c0 0x7fd88015b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:21.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:21.106+0000 7fd885479640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8800c90d0 0x7fd8800ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:21.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:21.106+0000 7fd885479640 1 --2- 172.21.15.125:0/3070683077 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8800c8cf0 0x7fd88014c2d0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:21.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:21.106+0000 7fd885479640 1 -- 172.21.15.125:0/3070683077 >> 172.21.15.125:0/3070683077 conn(0x7fd88006be40 msgr2=0x7fd8800c7d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:21.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:21.106+0000 7fd885479640 1 -- 172.21.15.125:0/3070683077 shutdown_connections 2023-10-30T13:02:21.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:21.106+0000 7fd885479640 1 -- 172.21.15.125:0/3070683077 wait complete. 2023-10-30T13:02:21.130 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:02:21.643 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:02:21.647 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:02:23.130 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:23.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.274+0000 7ff001df5640 1 Processor -- start 2023-10-30T13:02:23.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.274+0000 7ff001df5640 1 -- start start 2023-10-30T13:02:23.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.274+0000 7ff001df5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7feffc0c05d0 0x7feffc13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:23.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7ff001df5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feffc0c0a50 con 0x7feffc0c6f60 2023-10-30T13:02:23.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7ff001df5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feffc06f6e0 con 0x7feffc0c6b80 2023-10-30T13:02:23.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7ff001df5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feffc06f9c0 con 0x7feffc0c05d0 2023-10-30T13:02:23.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7feffaffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7feffc0c05d0 0x7feffc13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:23.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7feffaffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7feffc0c05d0 0x7feffc13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38564/0 (socket says 172.21.15.125:38564) 2023-10-30T13:02:23.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7feffbfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7feffc0c6f60 0x7feffc1566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39538/0 (socket says 172.21.15.125:39538) 2023-10-30T13:02:23.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7feffaffd640 1 -- 172.21.15.125:0/3248090037 learned_addr learned my addr 172.21.15.125:0/3248090037 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:23.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7feffaffd640 1 -- 172.21.15.125:0/3248090037 >> v1:172.21.15.136:6789/0 conn(0x7feffc0c6b80 legacy=0x7feffc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:23.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7feffaffd640 1 -- 172.21.15.125:0/3248090037 >> v1:172.21.15.125:6789/0 conn(0x7feffc0c6f60 legacy=0x7feffc1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:23.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7feffa7fc640 1 -- 172.21.15.125:0/3248090037 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551002061 0 0) 0x7feffc0c0a50 con 0x7feffc0c6f60 2023-10-30T13:02:23.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7feffaffd640 1 -- 172.21.15.125:0/3248090037 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc06fca0 con 0x7feffc0c05d0 2023-10-30T13:02:23.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7feffaffd640 1 --2- 172.21.15.125:0/3248090037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feffc0c05d0 0x7feffc13ed90 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7feff0009870 tx=0x7feff001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30ea24a67470811a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:23.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7feffa7fc640 1 -- 172.21.15.125:0/3248090037 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff00ab650 con 0x7feffc0c05d0 2023-10-30T13:02:23.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7feffa7fc640 1 -- 172.21.15.125:0/3248090037 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff00ab7b0 con 0x7feffc0c05d0 2023-10-30T13:02:23.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.278+0000 7feffa7fc640 1 -- 172.21.15.125:0/3248090037 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff00abb40 con 0x7feffc0c05d0 2023-10-30T13:02:23.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 -- 172.21.15.125:0/3248090037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feffc0c05d0 msgr2=0x7feffc13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:23.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 --2- 172.21.15.125:0/3248090037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feffc0c05d0 0x7feffc13ed90 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7feff0009870 tx=0x7feff001c540 comp rx=0 tx=0).stop 2023-10-30T13:02:23.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 -- 172.21.15.125:0/3248090037 shutdown_connections 2023-10-30T13:02:23.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 --2- 172.21.15.125:0/3248090037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feffc0c05d0 0x7feffc13ed90 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:23.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 -- 172.21.15.125:0/3248090037 >> 172.21.15.125:0/3248090037 conn(0x7feffc0c7910 msgr2=0x7feffc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:23.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 -- 172.21.15.125:0/3248090037 shutdown_connections 2023-10-30T13:02:23.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 -- 172.21.15.125:0/3248090037 wait complete. 2023-10-30T13:02:23.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 Processor -- start 2023-10-30T13:02:23.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 -- start start 2023-10-30T13:02:23.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feffc0c05d0 0x7feffc135f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:23.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feffc0c6b80 0x7feffc153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:23.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feffc0c6f60 0x7feffc153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:23.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc0c1ad0 con 0x7feffc0c6b80 2023-10-30T13:02:23.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc153eb0 con 0x7feffc0c6f60 2023-10-30T13:02:23.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7ff001df5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feffc1351e0 con 0x7feffc0c05d0 2023-10-30T13:02:23.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7feffaffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feffc0c6b80 0x7feffc153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:23.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7feffaffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feffc0c6b80 0x7feffc153430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38580/0 (socket says 172.21.15.125:38580) 2023-10-30T13:02:23.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7feffaffd640 1 -- 172.21.15.125:0/3991309960 learned_addr learned my addr 172.21.15.125:0/3991309960 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:23.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7feffb7fe640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feffc0c05d0 0x7feffc135f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:23.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.282+0000 7feffbfff640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feffc0c6f60 0x7feffc153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:23.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7feffaffd640 1 -- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feffc0c05d0 msgr2=0x7feffc135f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:23.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7feffaffd640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feffc0c05d0 0x7feffc135f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:23.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7feffaffd640 1 -- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feffc0c6f60 msgr2=0x7feffc153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:23.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7feffaffd640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feffc0c6f60 0x7feffc153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:23.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7feffaffd640 1 -- 172.21.15.125:0/3991309960 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff00094d0 con 0x7feffc0c6b80 2023-10-30T13:02:23.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7feffb7fe640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feffc0c05d0 0x7feffc135f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:02:23.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7feffaffd640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feffc0c6b80 0x7feffc153430 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7feff00ad040 tx=0x7feff001ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:23.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7fefe3fff640 1 -- 172.21.15.125:0/3991309960 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff00abe70 con 0x7feffc0c6b80 2023-10-30T13:02:23.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7fefe3fff640 1 -- 172.21.15.125:0/3991309960 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff00b8070 con 0x7feffc0c6b80 2023-10-30T13:02:23.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7fefe3fff640 1 -- 172.21.15.125:0/3991309960 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff0004270 con 0x7feffc0c6b80 2023-10-30T13:02:23.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7ff001df5640 1 -- 172.21.15.125:0/3991309960 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feffc1354c0 con 0x7feffc0c6b80 2023-10-30T13:02:23.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7ff001df5640 1 -- 172.21.15.125:0/3991309960 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc1f5ba0 con 0x7feffc0c6b80 2023-10-30T13:02:23.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7ff001df5640 1 -- 172.21.15.125:0/3991309960 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feffc04e170 con 0x7feffc0c6b80 2023-10-30T13:02:23.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7fefe3fff640 1 -- 172.21.15.125:0/3991309960 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feff00043d0 con 0x7feffc0c6b80 2023-10-30T13:02:23.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7fefe3fff640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefe80664a0 0x7fefe8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:23.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7fefe3fff640 1 -- 172.21.15.125:0/3991309960 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7feff011ed70 con 0x7feffc0c6b80 2023-10-30T13:02:23.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.286+0000 7feffb7fe640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefe80664a0 0x7fefe8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:23.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.290+0000 7feffb7fe640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefe80664a0 0x7fefe8068960 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fefec000f70 tx=0x7fefec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:23.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.290+0000 7fefe3fff640 1 -- 172.21.15.125:0/3991309960 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feff00ebc30 con 0x7feffc0c6b80 2023-10-30T13:02:23.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.470+0000 7ff001df5640 1 -- 172.21.15.125:0/3991309960 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feffc1573c0 con 0x7feffc0c6b80 2023-10-30T13:02:23.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.470+0000 7fefe3fff640 1 -- 172.21.15.125:0/3991309960 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6519 (secure 0 0 0) 0x7feff00ef250 con 0x7feffc0c6b80 2023-10-30T13:02:23.475 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:23.475 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7866,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:01:02.384461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:23.476 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 150 2023-10-30T13:02:23.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.474+0000 7ff001df5640 1 -- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefe80664a0 msgr2=0x7fefe8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:23.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.474+0000 7ff001df5640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefe80664a0 0x7fefe8068960 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fefec000f70 tx=0x7fefec009040 comp rx=0 tx=0).stop 2023-10-30T13:02:23.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.474+0000 7ff001df5640 1 -- 172.21.15.125:0/3991309960 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feffc0c6b80 msgr2=0x7feffc153430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:23.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.474+0000 7ff001df5640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feffc0c6b80 0x7feffc153430 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7feff00ad040 tx=0x7feff001ad80 comp rx=0 tx=0).stop 2023-10-30T13:02:23.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.474+0000 7ff001df5640 1 -- 172.21.15.125:0/3991309960 shutdown_connections 2023-10-30T13:02:23.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.474+0000 7ff001df5640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefe80664a0 0x7fefe8068960 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:23.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.474+0000 7ff001df5640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feffc0c6f60 0x7feffc153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:23.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.474+0000 7ff001df5640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feffc0c6b80 0x7feffc153430 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:23.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.474+0000 7ff001df5640 1 --2- 172.21.15.125:0/3991309960 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feffc0c05d0 0x7feffc135f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:23.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.474+0000 7ff001df5640 1 -- 172.21.15.125:0/3991309960 >> 172.21.15.125:0/3991309960 conn(0x7feffc0c7910 msgr2=0x7feffc0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:23.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.474+0000 7ff001df5640 1 -- 172.21.15.125:0/3991309960 shutdown_connections 2023-10-30T13:02:23.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:23.474+0000 7ff001df5640 1 -- 172.21.15.125:0/3991309960 wait complete. 2023-10-30T13:02:23.495 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:02:25.496 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:25.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.642+0000 7f630ea9c640 1 Processor -- start 2023-10-30T13:02:25.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.642+0000 7f630ea9c640 1 -- start start 2023-10-30T13:02:25.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.642+0000 7f630ea9c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6308070740 0x7f63080c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:25.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.642+0000 7f630ea9c640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63080c5810 con 0x7f630815e700 2023-10-30T13:02:25.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.642+0000 7f630ea9c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f630815eb70 con 0x7f63081580e0 2023-10-30T13:02:25.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.642+0000 7f630ea9c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f630806acb0 con 0x7f6308070740 2023-10-30T13:02:25.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.642+0000 7f62fffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6308070740 0x7f63080c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:25.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.646+0000 7f62fffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6308070740 0x7f63080c6540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38594/0 (socket says 172.21.15.125:38594) 2023-10-30T13:02:25.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.646+0000 7f62fffff640 1 -- 172.21.15.125:0/1676017616 learned_addr learned my addr 172.21.15.125:0/1676017616 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:25.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.646+0000 7f630d039640 1 --1- 172.21.15.125:0/1676017616 >> v1:172.21.15.125:6789/0 conn(0x7f630815e700 0x7f630806a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39542/0 (socket says 172.21.15.125:39542) 2023-10-30T13:02:25.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.646+0000 7f62fffff640 1 -- 172.21.15.125:0/1676017616 >> v1:172.21.15.136:6789/0 conn(0x7f63081580e0 legacy=0x7f6308069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:25.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.646+0000 7f62fffff640 1 -- 172.21.15.125:0/1676017616 >> v1:172.21.15.125:6789/0 conn(0x7f630815e700 legacy=0x7f630806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:25.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.646+0000 7f62ff7fe640 1 -- 172.21.15.125:0/1676017616 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2703097225 0 0) 0x7f63080c5810 con 0x7f630815e700 2023-10-30T13:02:25.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.646+0000 7f62fffff640 1 -- 172.21.15.125:0/1676017616 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63080c6bb0 con 0x7f6308070740 2023-10-30T13:02:25.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.646+0000 7f62fffff640 1 --2- 172.21.15.125:0/1676017616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6308070740 0x7f63080c6540 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f62f8009870 tx=0x7f62f801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe7a5552a909fa29 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:25.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.646+0000 7f62ff7fe640 1 -- 172.21.15.125:0/1676017616 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f80a7070 con 0x7f6308070740 2023-10-30T13:02:25.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.646+0000 7f62ff7fe640 1 -- 172.21.15.125:0/1676017616 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f80ab430 con 0x7f6308070740 2023-10-30T13:02:25.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.646+0000 7f62ff7fe640 1 -- 172.21.15.125:0/1676017616 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f80ab7c0 con 0x7f6308070740 2023-10-30T13:02:25.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 -- 172.21.15.125:0/1676017616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6308070740 msgr2=0x7f63080c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:25.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 --2- 172.21.15.125:0/1676017616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6308070740 0x7f63080c6540 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f62f8009870 tx=0x7f62f801c540 comp rx=0 tx=0).stop 2023-10-30T13:02:25.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 -- 172.21.15.125:0/1676017616 shutdown_connections 2023-10-30T13:02:25.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 --2- 172.21.15.125:0/1676017616 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6308070740 0x7f63080c6540 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:25.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 -- 172.21.15.125:0/1676017616 >> 172.21.15.125:0/1676017616 conn(0x7f63080c7910 msgr2=0x7f63080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:25.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 -- 172.21.15.125:0/1676017616 shutdown_connections 2023-10-30T13:02:25.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 -- 172.21.15.125:0/1676017616 wait complete. 2023-10-30T13:02:25.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 Processor -- start 2023-10-30T13:02:25.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 -- start start 2023-10-30T13:02:25.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6308070740 0x7f63080c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:25.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63081580e0 0x7f63081537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:25.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f630815e700 0x7f6308153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:25.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6308153310 con 0x7f6308070740 2023-10-30T13:02:25.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63081490d0 con 0x7f63081580e0 2023-10-30T13:02:25.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630ea9c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6308143ad0 con 0x7f630815e700 2023-10-30T13:02:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630c838640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6308070740 0x7f63080c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630c838640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6308070740 0x7f63080c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38608/0 (socket says 172.21.15.125:38608) 2023-10-30T13:02:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630c838640 1 -- 172.21.15.125:0/399814759 learned_addr learned my addr 172.21.15.125:0/399814759 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f62fffff640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63081580e0 0x7f63081537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630d039640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f630815e700 0x7f6308153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:25.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630c838640 1 -- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f630815e700 msgr2=0x7f6308153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:25.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630c838640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f630815e700 0x7f6308153ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:25.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630c838640 1 -- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63081580e0 msgr2=0x7f63081537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:25.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630c838640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63081580e0 0x7f63081537a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:25.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630c838640 1 -- 172.21.15.125:0/399814759 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f80094d0 con 0x7f6308070740 2023-10-30T13:02:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.650+0000 7f630d039640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f630815e700 0x7f6308153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:02:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.654+0000 7f630c838640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6308070740 0x7f63080c12a0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f630400c500 tx=0x7f630400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.654+0000 7f62fd7fa640 1 -- 172.21.15.125:0/399814759 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6304005220 con 0x7f6308070740 2023-10-30T13:02:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.654+0000 7f62fd7fa640 1 -- 172.21.15.125:0/399814759 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6304005380 con 0x7f6308070740 2023-10-30T13:02:25.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.654+0000 7f62fd7fa640 1 -- 172.21.15.125:0/399814759 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6304007270 con 0x7f6308070740 2023-10-30T13:02:25.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.654+0000 7f630ea9c640 1 -- 172.21.15.125:0/399814759 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6308149390 con 0x7f6308070740 2023-10-30T13:02:25.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.654+0000 7f630ea9c640 1 -- 172.21.15.125:0/399814759 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6308152fa0 con 0x7f6308070740 2023-10-30T13:02:25.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.654+0000 7f630ea9c640 1 -- 172.21.15.125:0/399814759 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f630804e170 con 0x7f6308070740 2023-10-30T13:02:25.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.654+0000 7f62fd7fa640 1 -- 172.21.15.125:0/399814759 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f63040073d0 con 0x7f6308070740 2023-10-30T13:02:25.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.654+0000 7f62fd7fa640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62f40664a0 0x7f62f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:25.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.654+0000 7f62fd7fa640 1 -- 172.21.15.125:0/399814759 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7f6304083e30 con 0x7f6308070740 2023-10-30T13:02:25.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.654+0000 7f62fffff640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62f40664a0 0x7f62f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:25.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.658+0000 7f62fffff640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62f40664a0 0x7f62f4068960 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f62f80af000 tx=0x7f62f80b6040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:25.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.658+0000 7f62fd7fa640 1 -- 172.21.15.125:0/399814759 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6304050c70 con 0x7f6308070740 2023-10-30T13:02:25.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.838+0000 7f630ea9c640 1 -- 172.21.15.125:0/399814759 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63080c5a30 con 0x7f6308070740 2023-10-30T13:02:25.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.842+0000 7f62fd7fa640 1 -- 172.21.15.125:0/399814759 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f6304054290 con 0x7f6308070740 2023-10-30T13:02:25.843 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:25.844 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7866,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:01:02.384461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:25.844 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 150 2023-10-30T13:02:25.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.842+0000 7f630ea9c640 1 -- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62f40664a0 msgr2=0x7f62f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:25.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.842+0000 7f630ea9c640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62f40664a0 0x7f62f4068960 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f62f80af000 tx=0x7f62f80b6040 comp rx=0 tx=0).stop 2023-10-30T13:02:25.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.842+0000 7f630ea9c640 1 -- 172.21.15.125:0/399814759 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6308070740 msgr2=0x7f63080c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:25.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.842+0000 7f630ea9c640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6308070740 0x7f63080c12a0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f630400c500 tx=0x7f630400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:02:25.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.842+0000 7f630ea9c640 1 -- 172.21.15.125:0/399814759 shutdown_connections 2023-10-30T13:02:25.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.842+0000 7f630ea9c640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62f40664a0 0x7f62f4068960 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:25.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.842+0000 7f630ea9c640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f630815e700 0x7f6308153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:25.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.842+0000 7f630ea9c640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f63081580e0 0x7f63081537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:25.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.842+0000 7f630ea9c640 1 --2- 172.21.15.125:0/399814759 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6308070740 0x7f63080c12a0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:25.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.842+0000 7f630ea9c640 1 -- 172.21.15.125:0/399814759 >> 172.21.15.125:0/399814759 conn(0x7f63080c7910 msgr2=0x7f6308146440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:25.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.842+0000 7f630ea9c640 1 -- 172.21.15.125:0/399814759 shutdown_connections 2023-10-30T13:02:25.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:25.842+0000 7f630ea9c640 1 -- 172.21.15.125:0/399814759 wait complete. 2023-10-30T13:02:25.864 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:02:27.865 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:28.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6de4fc4640 1 Processor -- start 2023-10-30T13:02:28.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6de4fc4640 1 -- start start 2023-10-30T13:02:28.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6de4fc4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6de01538a0 0x7f6de015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:28.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6de4fc4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6de00ca0e0 0x7f6de015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:28.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6de4fc4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6de00cb800 con 0x7f6de00cf160 2023-10-30T13:02:28.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6de4fc4640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6de015dc30 con 0x7f6de01538a0 2023-10-30T13:02:28.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6de4fc4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6de00cb330 con 0x7f6de00ca0e0 2023-10-30T13:02:28.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6dde59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6de01538a0 0x7f6de015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:28.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6dde59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6de01538a0 0x7f6de015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38610/0 (socket says 172.21.15.125:38610) 2023-10-30T13:02:28.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6dde59c640 1 -- 172.21.15.125:0/1272353408 learned_addr learned my addr 172.21.15.125:0/1272353408 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:28.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6dd7fff640 1 --2- 172.21.15.125:0/1272353408 >> v2:172.21.15.136:3300/0 conn(0x7f6de00ca0e0 0x7f6de015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:28.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6dde59c640 1 -- 172.21.15.125:0/1272353408 >> v2:172.21.15.136:3300/0 conn(0x7f6de00ca0e0 msgr2=0x7f6de015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:28.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6dde59c640 1 --2- 172.21.15.125:0/1272353408 >> v2:172.21.15.136:3300/0 conn(0x7f6de00ca0e0 0x7f6de015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:28.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6dde59c640 1 -- 172.21.15.125:0/1272353408 >> v1:172.21.15.136:6789/0 conn(0x7f6de00cf160 legacy=0x7f6de00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:28.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.010+0000 7f6dde59c640 1 -- 172.21.15.125:0/1272353408 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de00cad70 con 0x7f6de01538a0 2023-10-30T13:02:28.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.014+0000 7f6dde59c640 1 --2- 172.21.15.125:0/1272353408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de01538a0 0x7f6de015d6f0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f6de00b30a0 tx=0x7f6dd001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf1e2907adb9c58b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:28.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.014+0000 7f6dddd9b640 1 -- 172.21.15.125:0/1272353408 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd00a89f0 con 0x7f6de01538a0 2023-10-30T13:02:28.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.014+0000 7f6dddd9b640 1 -- 172.21.15.125:0/1272353408 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd00a8b50 con 0x7f6de01538a0 2023-10-30T13:02:28.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.014+0000 7f6dddd9b640 1 -- 172.21.15.125:0/1272353408 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd00b0440 con 0x7f6de01538a0 2023-10-30T13:02:28.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.014+0000 7f6de4fc4640 1 -- 172.21.15.125:0/1272353408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de01538a0 msgr2=0x7f6de015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:28.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.014+0000 7f6de4fc4640 1 --2- 172.21.15.125:0/1272353408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de01538a0 0x7f6de015d6f0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f6de00b30a0 tx=0x7f6dd001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:02:28.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.014+0000 7f6de4fc4640 1 -- 172.21.15.125:0/1272353408 shutdown_connections 2023-10-30T13:02:28.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.014+0000 7f6de4fc4640 1 --2- 172.21.15.125:0/1272353408 >> v2:172.21.15.136:3300/0 conn(0x7f6de00ca0e0 0x7f6de015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:28.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.014+0000 7f6de4fc4640 1 --2- 172.21.15.125:0/1272353408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de01538a0 0x7f6de015d6f0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:28.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.014+0000 7f6de4fc4640 1 -- 172.21.15.125:0/1272353408 >> 172.21.15.125:0/1272353408 conn(0x7f6de006bfe0 msgr2=0x7f6de006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:28.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.014+0000 7f6de4fc4640 1 -- 172.21.15.125:0/1272353408 shutdown_connections 2023-10-30T13:02:28.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.014+0000 7f6de4fc4640 1 -- 172.21.15.125:0/1272353408 wait complete. 2023-10-30T13:02:28.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6de4fc4640 1 Processor -- start 2023-10-30T13:02:28.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6de4fc4640 1 -- start start 2023-10-30T13:02:28.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6de4fc4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6de00ca0e0 0x7f6de0158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:28.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6de4fc4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6de00cf160 0x7f6de00c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:28.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6de4fc4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de01538a0 0x7f6de01ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:28.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6de4fc4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de01e4db0 con 0x7f6de01538a0 2023-10-30T13:02:28.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6de4fc4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6de01ef520 con 0x7f6de00cf160 2023-10-30T13:02:28.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6de4fc4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6de01f2960 con 0x7f6de00ca0e0 2023-10-30T13:02:28.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dded9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de01538a0 0x7f6de01ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:28.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dded9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de01538a0 0x7f6de01ef9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38616/0 (socket says 172.21.15.125:38616) 2023-10-30T13:02:28.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dded9d640 1 -- 172.21.15.125:0/615421408 learned_addr learned my addr 172.21.15.125:0/615421408 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:28.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dde59c640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6de00ca0e0 0x7f6de0158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:28.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dd7fff640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6de00cf160 0x7f6de00c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:28.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dded9d640 1 -- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6de00ca0e0 msgr2=0x7f6de0158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:28.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dded9d640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6de00ca0e0 0x7f6de0158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:28.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dded9d640 1 -- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6de00cf160 msgr2=0x7f6de00c5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:28.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dded9d640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6de00cf160 0x7f6de00c5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:28.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dded9d640 1 -- 172.21.15.125:0/615421408 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd00094d0 con 0x7f6de01538a0 2023-10-30T13:02:28.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dd7fff640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6de00cf160 0x7f6de00c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:02:28.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dded9d640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de01538a0 0x7f6de01ef9b0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800c6c0 tx=0x7f6dd800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:28.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dd6ffd640 1 -- 172.21.15.125:0/615421408 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd8003b80 con 0x7f6de01538a0 2023-10-30T13:02:28.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dd6ffd640 1 -- 172.21.15.125:0/615421408 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dd8003ce0 con 0x7f6de01538a0 2023-10-30T13:02:28.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6dd6ffd640 1 -- 172.21.15.125:0/615421408 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dd80052f0 con 0x7f6de01538a0 2023-10-30T13:02:28.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6de4fc4640 1 -- 172.21.15.125:0/615421408 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6de01f2c20 con 0x7f6de01538a0 2023-10-30T13:02:28.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.018+0000 7f6de4fc4640 1 -- 172.21.15.125:0/615421408 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6de01f31e0 con 0x7f6de01538a0 2023-10-30T13:02:28.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.022+0000 7f6de4fc4640 1 -- 172.21.15.125:0/615421408 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6de00b2a50 con 0x7f6de01538a0 2023-10-30T13:02:28.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.022+0000 7f6dd6ffd640 1 -- 172.21.15.125:0/615421408 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6dd8005450 con 0x7f6de01538a0 2023-10-30T13:02:28.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.022+0000 7f6dd6ffd640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6dc00664d0 0x7f6dc0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:28.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.022+0000 7f6dd6ffd640 1 -- 172.21.15.125:0/615421408 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7f6dd8084180 con 0x7f6de01538a0 2023-10-30T13:02:28.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.022+0000 7f6dde59c640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6dc00664d0 0x7f6dc0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:28.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.022+0000 7f6dde59c640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6dc00664d0 0x7f6dc0068990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00b2000 tx=0x7f6dd0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:28.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.026+0000 7f6dd6ffd640 1 -- 172.21.15.125:0/615421408 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6dd8050fb0 con 0x7f6de01538a0 2023-10-30T13:02:28.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.210+0000 7f6de4fc4640 1 -- 172.21.15.125:0/615421408 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6de0159850 con 0x7f6de01538a0 2023-10-30T13:02:28.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.210+0000 7f6dd6ffd640 1 -- 172.21.15.125:0/615421408 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f6dd80545d0 con 0x7f6de01538a0 2023-10-30T13:02:28.213 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:28.213 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7866,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:01:02.384461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:28.213 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 150 2023-10-30T13:02:28.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.210+0000 7f6de4fc4640 1 -- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6dc00664d0 msgr2=0x7f6dc0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:28.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.210+0000 7f6de4fc4640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6dc00664d0 0x7f6dc0068990 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6dd00b2000 tx=0x7f6dd0009040 comp rx=0 tx=0).stop 2023-10-30T13:02:28.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.210+0000 7f6de4fc4640 1 -- 172.21.15.125:0/615421408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de01538a0 msgr2=0x7f6de01ef9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:28.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.210+0000 7f6de4fc4640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de01538a0 0x7f6de01ef9b0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f6dd800c6c0 tx=0x7f6dd800cb80 comp rx=0 tx=0).stop 2023-10-30T13:02:28.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.210+0000 7f6de4fc4640 1 -- 172.21.15.125:0/615421408 shutdown_connections 2023-10-30T13:02:28.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.210+0000 7f6de4fc4640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6dc00664d0 0x7f6dc0068990 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:28.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.210+0000 7f6de4fc4640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6de01538a0 0x7f6de01ef9b0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:28.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.210+0000 7f6de4fc4640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6de00cf160 0x7f6de00c5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:28.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.210+0000 7f6de4fc4640 1 --2- 172.21.15.125:0/615421408 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6de00ca0e0 0x7f6de0158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:28.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.210+0000 7f6de4fc4640 1 -- 172.21.15.125:0/615421408 >> 172.21.15.125:0/615421408 conn(0x7f6de006bfe0 msgr2=0x7f6de0151f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:28.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.210+0000 7f6de4fc4640 1 -- 172.21.15.125:0/615421408 shutdown_connections 2023-10-30T13:02:28.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:28.210+0000 7f6de4fc4640 1 -- 172.21.15.125:0/615421408 wait complete. 2023-10-30T13:02:28.234 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:02:30.235 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:30.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4d00c9640 1 Processor -- start 2023-10-30T13:02:30.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4d00c9640 1 -- start start 2023-10-30T13:02:30.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4d00c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4c81538a0 0x7fd4c815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:30.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4d00c9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4c80ca0e0 0x7fd4c815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:30.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4d00c9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd4c80cf160 0x7fd4c80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:30.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4d00c9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c80cb800 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4d00c9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4c815dc30 con 0x7fd4c80cf160 2023-10-30T13:02:30.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4d00c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c80cb330 con 0x7fd4c81538a0 2023-10-30T13:02:30.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4cd664640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4c80ca0e0 0x7fd4c815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:30.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4cd664640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4c80ca0e0 0x7fd4c815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38620/0 (socket says 172.21.15.125:38620) 2023-10-30T13:02:30.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4cd664640 1 -- 172.21.15.125:0/4051992919 learned_addr learned my addr 172.21.15.125:0/4051992919 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:30.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4ce666640 1 --2- 172.21.15.125:0/4051992919 >> v2:172.21.15.136:3300/0 conn(0x7fd4c80cf160 0x7fd4c80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:30.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4cde65640 1 --2- 172.21.15.125:0/4051992919 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4c81538a0 0x7fd4c815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:30.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4cd664640 1 -- 172.21.15.125:0/4051992919 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4c81538a0 msgr2=0x7fd4c815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:30.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4cd664640 1 --2- 172.21.15.125:0/4051992919 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4c81538a0 0x7fd4c815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:30.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4cd664640 1 -- 172.21.15.125:0/4051992919 >> v2:172.21.15.136:3300/0 conn(0x7fd4c80cf160 msgr2=0x7fd4c80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:30.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4cd664640 1 --2- 172.21.15.125:0/4051992919 >> v2:172.21.15.136:3300/0 conn(0x7fd4c80cf160 0x7fd4c80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:30.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.382+0000 7fd4cd664640 1 -- 172.21.15.125:0/4051992919 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c80cad70 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.386+0000 7fd4cd664640 1 --2- 172.21.15.125:0/4051992919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4c80ca0e0 0x7fd4c815a090 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fd4c806cb00 tx=0x7fd4b801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c1de900cee5877 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:30.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.386+0000 7fd4cce63640 1 -- 172.21.15.125:0/4051992919 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b80a89b0 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.386+0000 7fd4cce63640 1 -- 172.21.15.125:0/4051992919 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4b80a8b10 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.386+0000 7fd4cce63640 1 -- 172.21.15.125:0/4051992919 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4b80a8ea0 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.386+0000 7fd4d00c9640 1 -- 172.21.15.125:0/4051992919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4c80ca0e0 msgr2=0x7fd4c815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:30.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.386+0000 7fd4d00c9640 1 --2- 172.21.15.125:0/4051992919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4c80ca0e0 0x7fd4c815a090 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fd4c806cb00 tx=0x7fd4b801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:02:30.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.386+0000 7fd4d00c9640 1 -- 172.21.15.125:0/4051992919 shutdown_connections 2023-10-30T13:02:30.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.386+0000 7fd4d00c9640 1 --2- 172.21.15.125:0/4051992919 >> v2:172.21.15.136:3300/0 conn(0x7fd4c80cf160 0x7fd4c80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:30.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.386+0000 7fd4d00c9640 1 --2- 172.21.15.125:0/4051992919 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4c80ca0e0 0x7fd4c815a090 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:30.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.386+0000 7fd4d00c9640 1 --2- 172.21.15.125:0/4051992919 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4c81538a0 0x7fd4c815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:30.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.386+0000 7fd4d00c9640 1 -- 172.21.15.125:0/4051992919 >> 172.21.15.125:0/4051992919 conn(0x7fd4c806bfe0 msgr2=0x7fd4c806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:30.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.386+0000 7fd4d00c9640 1 -- 172.21.15.125:0/4051992919 shutdown_connections 2023-10-30T13:02:30.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.386+0000 7fd4d00c9640 1 -- 172.21.15.125:0/4051992919 wait complete. 2023-10-30T13:02:30.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4d00c9640 1 Processor -- start 2023-10-30T13:02:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4d00c9640 1 -- start start 2023-10-30T13:02:30.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4d00c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4c80ca0e0 0x7fd4c81efac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:30.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4d00c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4c80cf160 0x7fd4c80ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:30.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4d00c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4c81538a0 0x7fd4c81e5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:30.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4d00c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c8142670 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4d00c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c815bf20 con 0x7fd4c81538a0 2023-10-30T13:02:30.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4d00c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4c81f0000 con 0x7fd4c80cf160 2023-10-30T13:02:30.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4cde65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4c80ca0e0 0x7fd4c81efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4cde65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4c80ca0e0 0x7fd4c81efac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38622/0 (socket says 172.21.15.125:38622) 2023-10-30T13:02:30.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4cde65640 1 -- 172.21.15.125:0/3873525610 learned_addr learned my addr 172.21.15.125:0/3873525610 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4cd664640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4c80cf160 0x7fd4c80ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4ce666640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4c81538a0 0x7fd4c81e5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:30.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4cde65640 1 -- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4c80cf160 msgr2=0x7fd4c80ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:30.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4cde65640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4c80cf160 0x7fd4c80ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:30.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4cde65640 1 -- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4c81538a0 msgr2=0x7fd4c81e5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:30.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4cde65640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4c81538a0 0x7fd4c81e5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4cde65640 1 -- 172.21.15.125:0/3873525610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b80094d0 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4cd664640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4c80cf160 0x7fd4c80ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:02:30.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4cde65640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4c80ca0e0 0x7fd4c81efac0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fd4c400d5f0 tx=0x7fd4c400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:30.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4be7fc640 1 -- 172.21.15.125:0/3873525610 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c40125f0 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4be7fc640 1 -- 172.21.15.125:0/3873525610 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c4012750 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4d00c9640 1 -- 172.21.15.125:0/3873525610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4c81e6450 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4be7fc640 1 -- 172.21.15.125:0/3873525610 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4c4012ae0 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4d00c9640 1 -- 172.21.15.125:0/3873525610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4c81e6920 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.390+0000 7fd4d00c9640 1 -- 172.21.15.125:0/3873525610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4c80b2a50 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.394+0000 7fd4be7fc640 1 -- 172.21.15.125:0/3873525610 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd4c4012c40 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.394+0000 7fd4be7fc640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4ac0664a0 0x7fd4ac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:30.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.394+0000 7fd4be7fc640 1 -- 172.21.15.125:0/3873525610 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7fd4c4084a60 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.394+0000 7fd4cd664640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4ac0664a0 0x7fd4ac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:30.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.394+0000 7fd4cd664640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4ac0664a0 0x7fd4ac068960 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80b2000 tx=0x7fd4b8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:30.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.398+0000 7fd4be7fc640 1 -- 172.21.15.125:0/3873525610 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd4c4051800 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.578+0000 7fd4d00c9640 1 -- 172.21.15.125:0/3873525610 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd4c81e6b10 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.582+0000 7fd4be7fc640 1 -- 172.21.15.125:0/3873525610 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fd4c4054e20 con 0x7fd4c80ca0e0 2023-10-30T13:02:30.584 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:30.584 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7866,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:01:02.384461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:30.584 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 150 2023-10-30T13:02:30.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.582+0000 7fd4d00c9640 1 -- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4ac0664a0 msgr2=0x7fd4ac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:30.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.582+0000 7fd4d00c9640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4ac0664a0 0x7fd4ac068960 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fd4b80b2000 tx=0x7fd4b8008040 comp rx=0 tx=0).stop 2023-10-30T13:02:30.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.582+0000 7fd4d00c9640 1 -- 172.21.15.125:0/3873525610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4c80ca0e0 msgr2=0x7fd4c81efac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:30.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.582+0000 7fd4d00c9640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4c80ca0e0 0x7fd4c81efac0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fd4c400d5f0 tx=0x7fd4c400dab0 comp rx=0 tx=0).stop 2023-10-30T13:02:30.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.582+0000 7fd4d00c9640 1 -- 172.21.15.125:0/3873525610 shutdown_connections 2023-10-30T13:02:30.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.582+0000 7fd4d00c9640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd4ac0664a0 0x7fd4ac068960 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:30.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.582+0000 7fd4d00c9640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4c81538a0 0x7fd4c81e5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:30.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.582+0000 7fd4d00c9640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd4c80cf160 0x7fd4c80ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:30.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.582+0000 7fd4d00c9640 1 --2- 172.21.15.125:0/3873525610 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4c80ca0e0 0x7fd4c81efac0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:30.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.582+0000 7fd4d00c9640 1 -- 172.21.15.125:0/3873525610 >> 172.21.15.125:0/3873525610 conn(0x7fd4c806bfe0 msgr2=0x7fd4c81525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:30.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.582+0000 7fd4d00c9640 1 -- 172.21.15.125:0/3873525610 shutdown_connections 2023-10-30T13:02:30.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:30.582+0000 7fd4d00c9640 1 -- 172.21.15.125:0/3873525610 wait complete. 2023-10-30T13:02:30.605 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:02:32.606 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:32.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.754+0000 7f18dce6b640 1 Processor -- start 2023-10-30T13:02:32.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.754+0000 7f18dce6b640 1 -- start start 2023-10-30T13:02:32.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.758+0000 7f18dce6b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f18d80c6bd0 0x7f18d806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:32.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.758+0000 7f18dce6b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f18d8071110 0x7f18d8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:32.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.758+0000 7f18dce6b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18d8066cf0 con 0x7f18d806f7a0 2023-10-30T13:02:32.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.758+0000 7f18dce6b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d80700c0 con 0x7f18d8071110 2023-10-30T13:02:32.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.758+0000 7f18dce6b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18d80c70d0 con 0x7f18d80c6bd0 2023-10-30T13:02:32.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.758+0000 7f18d6d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f18d8071110 0x7f18d8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:32.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.758+0000 7f18d659c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f18d806f7a0 0x7f18d8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58782/0 (socket says 172.21.15.125:58782) 2023-10-30T13:02:32.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.758+0000 7f18d659c640 1 -- 172.21.15.125:0/3837512945 learned_addr learned my addr 172.21.15.125:0/3837512945 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:02:32.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.758+0000 7f18cffff640 1 --2- 172.21.15.125:0/3837512945 >> v2:172.21.15.136:3300/0 conn(0x7f18d80c6bd0 0x7f18d806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:32.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.758+0000 7f18d6d9d640 1 -- 172.21.15.125:0/3837512945 >> v2:172.21.15.136:3300/0 conn(0x7f18d80c6bd0 msgr2=0x7f18d806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:32.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.758+0000 7f18d6d9d640 1 --2- 172.21.15.125:0/3837512945 >> v2:172.21.15.136:3300/0 conn(0x7f18d80c6bd0 0x7f18d806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:32.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.758+0000 7f18d6d9d640 1 -- 172.21.15.125:0/3837512945 >> v1:172.21.15.125:6789/0 conn(0x7f18d806f7a0 legacy=0x7f18d8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:32.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.758+0000 7f18d6d9d640 1 -- 172.21.15.125:0/3837512945 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d80c7390 con 0x7f18d8071110 2023-10-30T13:02:32.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.758+0000 7f18d5d9b640 1 -- 172.21.15.125:0/3837512945 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181807317 0 0) 0x7f18d8066cf0 con 0x7f18d806f7a0 2023-10-30T13:02:32.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.762+0000 7f18d6d9d640 1 --2- 172.21.15.125:0/3837512945 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18d8071110 0x7f18d8158110 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f18d806f1c0 tx=0x7f18d001f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9686ac4aed1a94a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:32.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.762+0000 7f18d5d9b640 1 -- 172.21.15.125:0/3837512945 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d00b0440 con 0x7f18d8071110 2023-10-30T13:02:32.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.762+0000 7f18d5d9b640 1 -- 172.21.15.125:0/3837512945 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d00b05a0 con 0x7f18d8071110 2023-10-30T13:02:32.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.762+0000 7f18d5d9b640 1 -- 172.21.15.125:0/3837512945 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d00b0930 con 0x7f18d8071110 2023-10-30T13:02:32.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.762+0000 7f18dce6b640 1 -- 172.21.15.125:0/3837512945 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18d8071110 msgr2=0x7f18d8158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:32.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.762+0000 7f18dce6b640 1 --2- 172.21.15.125:0/3837512945 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18d8071110 0x7f18d8158110 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f18d806f1c0 tx=0x7f18d001f0f0 comp rx=0 tx=0).stop 2023-10-30T13:02:32.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.762+0000 7f18dce6b640 1 -- 172.21.15.125:0/3837512945 shutdown_connections 2023-10-30T13:02:32.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.762+0000 7f18dce6b640 1 --2- 172.21.15.125:0/3837512945 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18d8071110 0x7f18d8158110 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:32.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.762+0000 7f18dce6b640 1 --2- 172.21.15.125:0/3837512945 >> v2:172.21.15.136:3300/0 conn(0x7f18d80c6bd0 0x7f18d806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:32.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.762+0000 7f18dce6b640 1 -- 172.21.15.125:0/3837512945 >> 172.21.15.125:0/3837512945 conn(0x7f18d80c7910 msgr2=0x7f18d80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:32.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.762+0000 7f18dce6b640 1 -- 172.21.15.125:0/3837512945 shutdown_connections 2023-10-30T13:02:32.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.762+0000 7f18dce6b640 1 -- 172.21.15.125:0/3837512945 wait complete. 2023-10-30T13:02:32.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.762+0000 7f18dce6b640 1 Processor -- start 2023-10-30T13:02:32.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.762+0000 7f18dce6b640 1 -- start start 2023-10-30T13:02:32.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18dce6b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18d806f7a0 0x7f18d8155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:32.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18dce6b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18d8071110 0x7f18d80adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:32.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18dce6b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f18d80c6bd0 0x7f18d8146670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:32.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18dce6b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d8154ff0 con 0x7f18d8071110 2023-10-30T13:02:32.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18dce6b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d8155b50 con 0x7f18d806f7a0 2023-10-30T13:02:32.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18dce6b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18d80aece0 con 0x7f18d80c6bd0 2023-10-30T13:02:32.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18cffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18d8071110 0x7f18d80adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:32.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18cffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18d8071110 0x7f18d80adb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38986/0 (socket says 172.21.15.125:38986) 2023-10-30T13:02:32.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18cffff640 1 -- 172.21.15.125:0/236263436 learned_addr learned my addr 172.21.15.125:0/236263436 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:32.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18d659c640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18d806f7a0 0x7f18d8155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:32.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18d6d9d640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f18d80c6bd0 0x7f18d8146670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:32.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18cffff640 1 -- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f18d80c6bd0 msgr2=0x7f18d8146670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:32.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18cffff640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f18d80c6bd0 0x7f18d8146670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:32.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18cffff640 1 -- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18d806f7a0 msgr2=0x7f18d8155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:32.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18cffff640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18d806f7a0 0x7f18d8155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:32.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18cffff640 1 -- 172.21.15.125:0/236263436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d0009a70 con 0x7f18d8071110 2023-10-30T13:02:32.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18d6d9d640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f18d80c6bd0 0x7f18d8146670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:02:32.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18cffff640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18d8071110 0x7f18d80adb30 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f18c000d7d0 tx=0x7f18c000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:32.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18ceffd640 1 -- 172.21.15.125:0/236263436 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c0018070 con 0x7f18d8071110 2023-10-30T13:02:32.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18ceffd640 1 -- 172.21.15.125:0/236263436 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18c0004020 con 0x7f18d8071110 2023-10-30T13:02:32.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18dce6b640 1 -- 172.21.15.125:0/236263436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d8154d10 con 0x7f18d8071110 2023-10-30T13:02:32.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18ceffd640 1 -- 172.21.15.125:0/236263436 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18c00041d0 con 0x7f18d8071110 2023-10-30T13:02:32.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18dce6b640 1 -- 172.21.15.125:0/236263436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d81f2f50 con 0x7f18d8071110 2023-10-30T13:02:32.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18dce6b640 1 -- 172.21.15.125:0/236263436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d80b02e0 con 0x7f18d8071110 2023-10-30T13:02:32.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.766+0000 7f18ceffd640 1 -- 172.21.15.125:0/236263436 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f18c0004330 con 0x7f18d8071110 2023-10-30T13:02:32.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.770+0000 7f18ceffd640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f18b80664a0 0x7f18b8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:32.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.770+0000 7f18ceffd640 1 -- 172.21.15.125:0/236263436 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7f18c00842a0 con 0x7f18d8071110 2023-10-30T13:02:32.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.770+0000 7f18d659c640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f18b80664a0 0x7f18b8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:32.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.770+0000 7f18d659c640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f18b80664a0 0x7f18b8068960 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f18c8000f70 tx=0x7f18c8007df0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:32.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.774+0000 7f18ceffd640 1 -- 172.21.15.125:0/236263436 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f18d80b02e0 con 0x7f18d8071110 2023-10-30T13:02:32.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.954+0000 7f18dce6b640 1 -- 172.21.15.125:0/236263436 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18d80b02e0 con 0x7f18d8071110 2023-10-30T13:02:32.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.954+0000 7f18ceffd640 1 -- 172.21.15.125:0/236263436 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f18d80b02e0 con 0x7f18d8071110 2023-10-30T13:02:32.957 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:32.957 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7866,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:01:02.384461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:32.957 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 150 2023-10-30T13:02:32.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.954+0000 7f18dce6b640 1 -- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f18b80664a0 msgr2=0x7f18b8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:32.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.954+0000 7f18dce6b640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f18b80664a0 0x7f18b8068960 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f18c8000f70 tx=0x7f18c8007df0 comp rx=0 tx=0).stop 2023-10-30T13:02:32.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.954+0000 7f18dce6b640 1 -- 172.21.15.125:0/236263436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18d8071110 msgr2=0x7f18d80adb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:32.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.954+0000 7f18dce6b640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18d8071110 0x7f18d80adb30 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f18c000d7d0 tx=0x7f18c000dc90 comp rx=0 tx=0).stop 2023-10-30T13:02:32.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.954+0000 7f18dce6b640 1 -- 172.21.15.125:0/236263436 shutdown_connections 2023-10-30T13:02:32.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.954+0000 7f18dce6b640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f18b80664a0 0x7f18b8068960 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:32.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.954+0000 7f18dce6b640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f18d80c6bd0 0x7f18d8146670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:32.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.954+0000 7f18dce6b640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f18d8071110 0x7f18d80adb30 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:32.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.954+0000 7f18dce6b640 1 --2- 172.21.15.125:0/236263436 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f18d806f7a0 0x7f18d8155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:32.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.954+0000 7f18dce6b640 1 -- 172.21.15.125:0/236263436 >> 172.21.15.125:0/236263436 conn(0x7f18d80c7910 msgr2=0x7f18d80c5d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:32.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.954+0000 7f18dce6b640 1 -- 172.21.15.125:0/236263436 shutdown_connections 2023-10-30T13:02:32.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:32.954+0000 7f18dce6b640 1 -- 172.21.15.125:0/236263436 wait complete. 2023-10-30T13:02:32.978 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:02:34.978 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:35.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb2a1e09640 1 Processor -- start 2023-10-30T13:02:35.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb2a1e09640 1 -- start start 2023-10-30T13:02:35.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb2a1e09640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb29c0c05d0 0x7fb29c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:35.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb2a1e09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb29c0c6f60 0x7fb29c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:35.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb2a1e09640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c0c0a50 con 0x7fb29c0c6b80 2023-10-30T13:02:35.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb2a1e09640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb29c06f6e0 con 0x7fb29c0c05d0 2023-10-30T13:02:35.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb2a1e09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c06f9c0 con 0x7fb29c0c6f60 2023-10-30T13:02:35.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb29affd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb29c0c05d0 0x7fb29c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:35.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb29b7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb29c0c6b80 0x7fb29c0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58794/0 (socket says 172.21.15.125:58794) 2023-10-30T13:02:35.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb29b7fe640 1 -- 172.21.15.125:0/3283752700 learned_addr learned my addr 172.21.15.125:0/3283752700 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:02:35.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb29bfff640 1 --2- 172.21.15.125:0/3283752700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb29c0c6f60 0x7fb29c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:35.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb29affd640 1 -- 172.21.15.125:0/3283752700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb29c0c6f60 msgr2=0x7fb29c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:35.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb29affd640 1 --2- 172.21.15.125:0/3283752700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb29c0c6f60 0x7fb29c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:35.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb29affd640 1 -- 172.21.15.125:0/3283752700 >> v1:172.21.15.125:6789/0 conn(0x7fb29c0c6b80 legacy=0x7fb29c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:35.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb29affd640 1 -- 172.21.15.125:0/3283752700 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c06fca0 con 0x7fb29c0c05d0 2023-10-30T13:02:35.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.122+0000 7fb29a7fc640 1 -- 172.21.15.125:0/3283752700 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092340981 0 0) 0x7fb29c0c0a50 con 0x7fb29c0c6b80 2023-10-30T13:02:35.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.126+0000 7fb29affd640 1 --2- 172.21.15.125:0/3283752700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb29c0c05d0 0x7fb29c13ed90 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fb2900097c0 tx=0x7fb29001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f8fb260280d96d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:35.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.126+0000 7fb29a7fc640 1 -- 172.21.15.125:0/3283752700 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2900ad640 con 0x7fb29c0c05d0 2023-10-30T13:02:35.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.126+0000 7fb29a7fc640 1 -- 172.21.15.125:0/3283752700 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2900ad7a0 con 0x7fb29c0c05d0 2023-10-30T13:02:35.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.126+0000 7fb29a7fc640 1 -- 172.21.15.125:0/3283752700 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2900adb30 con 0x7fb29c0c05d0 2023-10-30T13:02:35.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.126+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3283752700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb29c0c05d0 msgr2=0x7fb29c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:35.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.126+0000 7fb2a1e09640 1 --2- 172.21.15.125:0/3283752700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb29c0c05d0 0x7fb29c13ed90 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fb2900097c0 tx=0x7fb29001c480 comp rx=0 tx=0).stop 2023-10-30T13:02:35.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.126+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3283752700 shutdown_connections 2023-10-30T13:02:35.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.126+0000 7fb2a1e09640 1 --2- 172.21.15.125:0/3283752700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb29c0c05d0 0x7fb29c13ed90 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:35.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.126+0000 7fb2a1e09640 1 --2- 172.21.15.125:0/3283752700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb29c0c6f60 0x7fb29c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:35.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.126+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3283752700 >> 172.21.15.125:0/3283752700 conn(0x7fb29c0c7910 msgr2=0x7fb29c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:35.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.126+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3283752700 shutdown_connections 2023-10-30T13:02:35.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.126+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3283752700 wait complete. 2023-10-30T13:02:35.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb2a1e09640 1 Processor -- start 2023-10-30T13:02:35.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb2a1e09640 1 -- start start 2023-10-30T13:02:35.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb2a1e09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb29c0c05d0 0x7fb29c14b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:35.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb2a1e09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb29c0c6b80 0x7fb29c1f02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:35.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb2a1e09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb29c0c6f60 0x7fb29c132450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:35.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb2a1e09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c1efe10 con 0x7fb29c0c6b80 2023-10-30T13:02:35.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb2a1e09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c1f0f70 con 0x7fb29c0c05d0 2023-10-30T13:02:35.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb2a1e09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c1efb00 con 0x7fb29c0c6f60 2023-10-30T13:02:35.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb29affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb29c0c6b80 0x7fb29c1f02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:35.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb29affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb29c0c6b80 0x7fb29c1f02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39002/0 (socket says 172.21.15.125:39002) 2023-10-30T13:02:35.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb29affd640 1 -- 172.21.15.125:0/3919257358 learned_addr learned my addr 172.21.15.125:0/3919257358 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:35.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb29bfff640 1 --2- 172.21.15.125:0/3919257358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb29c0c6f60 0x7fb29c132450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:35.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb29b7fe640 1 --2- 172.21.15.125:0/3919257358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb29c0c05d0 0x7fb29c14b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:35.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb29affd640 1 -- 172.21.15.125:0/3919257358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb29c0c6f60 msgr2=0x7fb29c132450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:35.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb29affd640 1 --2- 172.21.15.125:0/3919257358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb29c0c6f60 0x7fb29c132450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:35.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb29affd640 1 -- 172.21.15.125:0/3919257358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb29c0c05d0 msgr2=0x7fb29c14b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:35.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb29affd640 1 --2- 172.21.15.125:0/3919257358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb29c0c05d0 0x7fb29c14b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:35.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb29affd640 1 -- 172.21.15.125:0/3919257358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb290009420 con 0x7fb29c0c6b80 2023-10-30T13:02:35.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb29affd640 1 --2- 172.21.15.125:0/3919257358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb29c0c6b80 0x7fb29c1f02a0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fb29001a0b0 tx=0x7fb290019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:35.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb283fff640 1 -- 172.21.15.125:0/3919257358 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2900b68e0 con 0x7fb29c0c6b80 2023-10-30T13:02:35.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb283fff640 1 -- 172.21.15.125:0/3919257358 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2900a6040 con 0x7fb29c0c6b80 2023-10-30T13:02:35.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb283fff640 1 -- 172.21.15.125:0/3919257358 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2900b6ca0 con 0x7fb29c0c6b80 2023-10-30T13:02:35.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3919257358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29c1f3ec0 con 0x7fb29c0c6b80 2023-10-30T13:02:35.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3919257358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29c135fb0 con 0x7fb29c0c6b80 2023-10-30T13:02:35.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.130+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3919257358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29c1effc0 con 0x7fb29c0c6b80 2023-10-30T13:02:35.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.134+0000 7fb283fff640 1 -- 172.21.15.125:0/3919257358 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb2900b4040 con 0x7fb29c0c6b80 2023-10-30T13:02:35.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.134+0000 7fb283fff640 1 --2- 172.21.15.125:0/3919257358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2880664a0 0x7fb288068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:35.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.134+0000 7fb283fff640 1 -- 172.21.15.125:0/3919257358 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7200+0+0 (secure 0 0 0) 0x7fb29011e990 con 0x7fb29c0c6b80 2023-10-30T13:02:35.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.134+0000 7fb29b7fe640 1 --2- 172.21.15.125:0/3919257358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2880664a0 0x7fb288068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:35.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.134+0000 7fb29b7fe640 1 --2- 172.21.15.125:0/3919257358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2880664a0 0x7fb288068960 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb28c000f70 tx=0x7fb28c007b60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:35.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.138+0000 7fb283fff640 1 -- 172.21.15.125:0/3919257358 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb2900eb8f0 con 0x7fb29c0c6b80 2023-10-30T13:02:35.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.314+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3919257358 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb29c04e170 con 0x7fb29c0c6b80 2023-10-30T13:02:35.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.314+0000 7fb283fff640 1 -- 172.21.15.125:0/3919257358 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fb2900b2020 con 0x7fb29c0c6b80 2023-10-30T13:02:35.319 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:35.319 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":7866,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":150}],"filesystems":[{"mdsmap":{"epoch":149,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:01:02.384461+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7707},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7707":{"gid":7707,"name":"c","rank":0,"incarnation":146,"state":"up:active","state_seq":20,"addr":"172.21.15.125:6837/3762807620","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":3762807620},{"type":"v1","addr":"172.21.15.125:6837","nonce":3762807620}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:35.320 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 150 2023-10-30T13:02:35.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.318+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3919257358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2880664a0 msgr2=0x7fb288068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:35.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.318+0000 7fb2a1e09640 1 --2- 172.21.15.125:0/3919257358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2880664a0 0x7fb288068960 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fb28c000f70 tx=0x7fb28c007b60 comp rx=0 tx=0).stop 2023-10-30T13:02:35.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.318+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3919257358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb29c0c6b80 msgr2=0x7fb29c1f02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:35.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.318+0000 7fb2a1e09640 1 --2- 172.21.15.125:0/3919257358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb29c0c6b80 0x7fb29c1f02a0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7fb29001a0b0 tx=0x7fb290019220 comp rx=0 tx=0).stop 2023-10-30T13:02:35.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.318+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3919257358 shutdown_connections 2023-10-30T13:02:35.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.318+0000 7fb2a1e09640 1 --2- 172.21.15.125:0/3919257358 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2880664a0 0x7fb288068960 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:35.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.318+0000 7fb2a1e09640 1 --2- 172.21.15.125:0/3919257358 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb29c0c6f60 0x7fb29c132450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:35.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.318+0000 7fb2a1e09640 1 --2- 172.21.15.125:0/3919257358 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb29c0c6b80 0x7fb29c1f02a0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:35.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.318+0000 7fb2a1e09640 1 --2- 172.21.15.125:0/3919257358 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb29c0c05d0 0x7fb29c14b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:35.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.318+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3919257358 >> 172.21.15.125:0/3919257358 conn(0x7fb29c0c7910 msgr2=0x7fb29c136de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:35.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.318+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3919257358 shutdown_connections 2023-10-30T13:02:35.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:35.318+0000 7fb2a1e09640 1 -- 172.21.15.125:0/3919257358 wait complete. 2023-10-30T13:02:35.340 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:02:37.341 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:37.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99b2bd3640 1 Processor -- start 2023-10-30T13:02:37.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99b2bd3640 1 -- start start 2023-10-30T13:02:37.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99b2bd3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f99ac1538a0 0x7f99ac15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:37.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99b2bd3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f99ac0ca0e0 0x7f99ac15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:37.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99b2bd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99ac0cf160 0x7f99ac0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:37.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99b2bd3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99ac0cb800 con 0x7f99ac0ca0e0 2023-10-30T13:02:37.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99b2bd3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99ac15dc30 con 0x7f99ac1538a0 2023-10-30T13:02:37.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99b2bd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac0cb330 con 0x7f99ac0cf160 2023-10-30T13:02:37.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99a3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f99ac0ca0e0 0x7f99ac15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:37.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99a3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f99ac0ca0e0 0x7f99ac15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39012/0 (socket says 172.21.15.125:39012) 2023-10-30T13:02:37.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99a3fff640 1 -- 172.21.15.125:0/495568740 learned_addr learned my addr 172.21.15.125:0/495568740 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:37.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99b1170640 1 --2- 172.21.15.125:0/495568740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99ac0cf160 0x7f99ac0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:37.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99b096f640 1 --2- 172.21.15.125:0/495568740 >> v2:172.21.15.136:3300/0 conn(0x7f99ac1538a0 0x7f99ac15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:37.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99a3fff640 1 -- 172.21.15.125:0/495568740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99ac0cf160 msgr2=0x7f99ac0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:37.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99a3fff640 1 --2- 172.21.15.125:0/495568740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99ac0cf160 0x7f99ac0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99a3fff640 1 -- 172.21.15.125:0/495568740 >> v2:172.21.15.136:3300/0 conn(0x7f99ac1538a0 msgr2=0x7f99ac15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:37.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99a3fff640 1 --2- 172.21.15.125:0/495568740 >> v2:172.21.15.136:3300/0 conn(0x7f99ac1538a0 0x7f99ac15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.486+0000 7f99a3fff640 1 -- 172.21.15.125:0/495568740 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ac0cad70 con 0x7f99ac0ca0e0 2023-10-30T13:02:37.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.490+0000 7f99a3fff640 1 --2- 172.21.15.125:0/495568740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99ac0ca0e0 0x7f99ac15a090 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f99ac0cfd10 tx=0x7f999c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12cc2a2de8d132b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:37.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.490+0000 7f99a37fe640 1 -- 172.21.15.125:0/495568740 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c0a89b0 con 0x7f99ac0ca0e0 2023-10-30T13:02:37.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.490+0000 7f99a37fe640 1 -- 172.21.15.125:0/495568740 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f999c0a8b10 con 0x7f99ac0ca0e0 2023-10-30T13:02:37.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.490+0000 7f99a37fe640 1 -- 172.21.15.125:0/495568740 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f999c0a8ea0 con 0x7f99ac0ca0e0 2023-10-30T13:02:37.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.490+0000 7f99b2bd3640 1 -- 172.21.15.125:0/495568740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99ac0ca0e0 msgr2=0x7f99ac15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:37.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.490+0000 7f99b2bd3640 1 --2- 172.21.15.125:0/495568740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99ac0ca0e0 0x7f99ac15a090 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f99ac0cfd10 tx=0x7f999c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.490+0000 7f99b2bd3640 1 -- 172.21.15.125:0/495568740 shutdown_connections 2023-10-30T13:02:37.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.490+0000 7f99b2bd3640 1 --2- 172.21.15.125:0/495568740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99ac0cf160 0x7f99ac0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.490+0000 7f99b2bd3640 1 --2- 172.21.15.125:0/495568740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99ac0ca0e0 0x7f99ac15a090 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.490+0000 7f99b2bd3640 1 --2- 172.21.15.125:0/495568740 >> v2:172.21.15.136:3300/0 conn(0x7f99ac1538a0 0x7f99ac15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.490+0000 7f99b2bd3640 1 -- 172.21.15.125:0/495568740 >> 172.21.15.125:0/495568740 conn(0x7f99ac06bfe0 msgr2=0x7f99ac06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:37.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.490+0000 7f99b2bd3640 1 -- 172.21.15.125:0/495568740 shutdown_connections 2023-10-30T13:02:37.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.490+0000 7f99b2bd3640 1 -- 172.21.15.125:0/495568740 wait complete. 2023-10-30T13:02:37.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b2bd3640 1 Processor -- start 2023-10-30T13:02:37.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b2bd3640 1 -- start start 2023-10-30T13:02:37.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b2bd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99ac0ca0e0 0x7f99ac1e5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:37.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b2bd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99ac0cf160 0x7f99ac157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:37.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b2bd3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99ac1538a0 0x7f99ac1580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:37.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b2bd3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac15b280 con 0x7f99ac1538a0 2023-10-30T13:02:37.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b2bd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac0ce020 con 0x7f99ac0ca0e0 2023-10-30T13:02:37.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b2bd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ac0cd4a0 con 0x7f99ac0cf160 2023-10-30T13:02:37.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b1170640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99ac1538a0 0x7f99ac1580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:37.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b1170640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99ac1538a0 0x7f99ac1580b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39024/0 (socket says 172.21.15.125:39024) 2023-10-30T13:02:37.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b1170640 1 -- 172.21.15.125:0/2922781382 learned_addr learned my addr 172.21.15.125:0/2922781382 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:37.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b096f640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99ac0ca0e0 0x7f99ac1e5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:37.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99a3fff640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99ac0cf160 0x7f99ac157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:37.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b1170640 1 -- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99ac0cf160 msgr2=0x7f99ac157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:37.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b1170640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99ac0cf160 0x7f99ac157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b1170640 1 -- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99ac0ca0e0 msgr2=0x7f99ac1e5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:37.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b1170640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99ac0ca0e0 0x7f99ac1e5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b1170640 1 -- 172.21.15.125:0/2922781382 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f999c0094d0 con 0x7f99ac1538a0 2023-10-30T13:02:37.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99a3fff640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99ac0cf160 0x7f99ac157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:02:37.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b096f640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99ac0ca0e0 0x7f99ac1e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:02:37.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b1170640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99ac1538a0 0x7f99ac1580b0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f99a4007d10 tx=0x7f99a400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:37.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99a17fa640 1 -- 172.21.15.125:0/2922781382 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4014420 con 0x7f99ac1538a0 2023-10-30T13:02:37.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99a17fa640 1 -- 172.21.15.125:0/2922781382 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a40145a0 con 0x7f99ac1538a0 2023-10-30T13:02:37.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99a17fa640 1 -- 172.21.15.125:0/2922781382 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a4014950 con 0x7f99ac1538a0 2023-10-30T13:02:37.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b2bd3640 1 -- 172.21.15.125:0/2922781382 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99ac0cd780 con 0x7f99ac1538a0 2023-10-30T13:02:37.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.494+0000 7f99b2bd3640 1 -- 172.21.15.125:0/2922781382 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ac0cdd40 con 0x7f99ac1538a0 2023-10-30T13:02:37.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.498+0000 7f99b2bd3640 1 -- 172.21.15.125:0/2922781382 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99ac0b2a50 con 0x7f99ac1538a0 2023-10-30T13:02:37.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.498+0000 7f99a17fa640 1 -- 172.21.15.125:0/2922781382 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f99a4014ab0 con 0x7f99ac1538a0 2023-10-30T13:02:37.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.498+0000 7f99a17fa640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99980664a0 0x7f9998068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:37.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.498+0000 7f99a17fa640 1 -- 172.21.15.125:0/2922781382 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f99a4084e00 con 0x7f99ac1538a0 2023-10-30T13:02:37.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.498+0000 7f99b096f640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99980664a0 0x7f9998068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:37.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.498+0000 7f99b096f640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99980664a0 0x7f9998068960 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f99a800b9f0 tx=0x7f99a8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:37.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.502+0000 7f99a17fa640 1 -- 172.21.15.125:0/2922781382 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f99a4051bb0 con 0x7f99ac1538a0 2023-10-30T13:02:37.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.682+0000 7f99b2bd3640 1 -- 172.21.15.125:0/2922781382 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99ac04e170 con 0x7f99ac1538a0 2023-10-30T13:02:37.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.686+0000 7f99a17fa640 1 -- 172.21.15.125:0/2922781382 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f99a40551d0 con 0x7f99ac1538a0 2023-10-30T13:02:37.687 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:37.687 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:35.533876+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:37.687 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 151 2023-10-30T13:02:37.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.686+0000 7f99b2bd3640 1 -- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99980664a0 msgr2=0x7f9998068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:37.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.686+0000 7f99b2bd3640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99980664a0 0x7f9998068960 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f99a800b9f0 tx=0x7f99a8008040 comp rx=0 tx=0).stop 2023-10-30T13:02:37.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.686+0000 7f99b2bd3640 1 -- 172.21.15.125:0/2922781382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99ac1538a0 msgr2=0x7f99ac1580b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:37.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.686+0000 7f99b2bd3640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99ac1538a0 0x7f99ac1580b0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f99a4007d10 tx=0x7f99a400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.686+0000 7f99b2bd3640 1 -- 172.21.15.125:0/2922781382 shutdown_connections 2023-10-30T13:02:37.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.686+0000 7f99b2bd3640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f99980664a0 0x7f9998068960 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.686+0000 7f99b2bd3640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f99ac1538a0 0x7f99ac1580b0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.686+0000 7f99b2bd3640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f99ac0cf160 0x7f99ac157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.686+0000 7f99b2bd3640 1 --2- 172.21.15.125:0/2922781382 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f99ac0ca0e0 0x7f99ac1e5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.686+0000 7f99b2bd3640 1 -- 172.21.15.125:0/2922781382 >> 172.21.15.125:0/2922781382 conn(0x7f99ac06bfe0 msgr2=0x7f99ac065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:37.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.686+0000 7f99b2bd3640 1 -- 172.21.15.125:0/2922781382 shutdown_connections 2023-10-30T13:02:37.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.686+0000 7f99b2bd3640 1 -- 172.21.15.125:0/2922781382 wait complete. 2023-10-30T13:02:37.708 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:02:37.708 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:02:37.709 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:37.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.842+0000 7f9b2ebab640 1 Processor -- start 2023-10-30T13:02:37.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.842+0000 7f9b2ebab640 1 -- start start 2023-10-30T13:02:37.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.846+0000 7f9b2ebab640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9b280b0dc0 0x7f9b280c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:37.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.846+0000 7f9b2ebab640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9b280c5e70 0x7f9b280c8390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:37.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.846+0000 7f9b2ebab640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b280c74f0 con 0x7f9b28153b10 2023-10-30T13:02:37.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.846+0000 7f9b2ebab640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b280c1f20 con 0x7f9b280b0dc0 2023-10-30T13:02:37.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.846+0000 7f9b2ebab640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b280c77b0 con 0x7f9b280c5e70 2023-10-30T13:02:37.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.846+0000 7f9b1ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9b280b0dc0 0x7f9b280c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:37.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.846+0000 7f9b2c947640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9b28153b10 0x7f9b2815ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58796/0 (socket says 172.21.15.125:58796) 2023-10-30T13:02:37.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.846+0000 7f9b2c947640 1 -- 172.21.15.125:0/3479017976 learned_addr learned my addr 172.21.15.125:0/3479017976 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:02:37.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.846+0000 7f9b2d148640 1 --2- 172.21.15.125:0/3479017976 >> v2:172.21.15.136:3300/0 conn(0x7f9b280c5e70 0x7f9b280c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:37.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.846+0000 7f9b1ffff640 1 -- 172.21.15.125:0/3479017976 >> v2:172.21.15.136:3300/0 conn(0x7f9b280c5e70 msgr2=0x7f9b280c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:37.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.846+0000 7f9b1ffff640 1 --2- 172.21.15.125:0/3479017976 >> v2:172.21.15.136:3300/0 conn(0x7f9b280c5e70 0x7f9b280c8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.846+0000 7f9b1ffff640 1 -- 172.21.15.125:0/3479017976 >> v1:172.21.15.125:6789/0 conn(0x7f9b28153b10 legacy=0x7f9b2815ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:37.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.846+0000 7f9b1ffff640 1 -- 172.21.15.125:0/3479017976 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b28159b80 con 0x7f9b280b0dc0 2023-10-30T13:02:37.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.846+0000 7f9b1f7fe640 1 -- 172.21.15.125:0/3479017976 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3238280696 0 0) 0x7f9b280c74f0 con 0x7f9b28153b10 2023-10-30T13:02:37.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.850+0000 7f9b1ffff640 1 --2- 172.21.15.125:0/3479017976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b280b0dc0 0x7f9b280c7e50 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f9b2815df60 tx=0x7f9b1801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=933624564241270d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:37.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.850+0000 7f9b1f7fe640 1 -- 172.21.15.125:0/3479017976 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b180a8950 con 0x7f9b280b0dc0 2023-10-30T13:02:37.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.850+0000 7f9b1f7fe640 1 -- 172.21.15.125:0/3479017976 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b180a8ab0 con 0x7f9b280b0dc0 2023-10-30T13:02:37.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.850+0000 7f9b1f7fe640 1 -- 172.21.15.125:0/3479017976 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b180a8e40 con 0x7f9b280b0dc0 2023-10-30T13:02:37.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.850+0000 7f9b2ebab640 1 -- 172.21.15.125:0/3479017976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b280b0dc0 msgr2=0x7f9b280c7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:37.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.850+0000 7f9b2ebab640 1 --2- 172.21.15.125:0/3479017976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b280b0dc0 0x7f9b280c7e50 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f9b2815df60 tx=0x7f9b1801ec50 comp rx=0 tx=0).stop 2023-10-30T13:02:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.850+0000 7f9b2ebab640 1 -- 172.21.15.125:0/3479017976 shutdown_connections 2023-10-30T13:02:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.850+0000 7f9b2ebab640 1 --2- 172.21.15.125:0/3479017976 >> v2:172.21.15.136:3300/0 conn(0x7f9b280c5e70 0x7f9b280c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.850+0000 7f9b2ebab640 1 --2- 172.21.15.125:0/3479017976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b280b0dc0 0x7f9b280c7e50 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.850+0000 7f9b2ebab640 1 -- 172.21.15.125:0/3479017976 >> 172.21.15.125:0/3479017976 conn(0x7f9b2806d770 msgr2=0x7f9b2806db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:37.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.850+0000 7f9b2ebab640 1 -- 172.21.15.125:0/3479017976 shutdown_connections 2023-10-30T13:02:37.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.850+0000 7f9b2ebab640 1 -- 172.21.15.125:0/3479017976 wait complete. 2023-10-30T13:02:37.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.850+0000 7f9b2ebab640 1 Processor -- start 2023-10-30T13:02:37.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.850+0000 7f9b2ebab640 1 -- start start 2023-10-30T13:02:37.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b2ebab640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b280b0dc0 0x7f9b28071720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:37.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b2ebab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b280c5e70 0x7f9b281f07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:37.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b2ebab640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b28153b10 0x7f9b281f0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:37.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b2ebab640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2814ca90 con 0x7f9b280c5e70 2023-10-30T13:02:37.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b2ebab640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b2814f800 con 0x7f9b28153b10 2023-10-30T13:02:37.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b2ebab640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b281f0340 con 0x7f9b280b0dc0 2023-10-30T13:02:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b1ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b280c5e70 0x7f9b281f07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b1ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b280c5e70 0x7f9b281f07d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39044/0 (socket says 172.21.15.125:39044) 2023-10-30T13:02:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b1ffff640 1 -- 172.21.15.125:0/2933750076 learned_addr learned my addr 172.21.15.125:0/2933750076 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b2d148640 1 --2- 172.21.15.125:0/2933750076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b28153b10 0x7f9b281f0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:37.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b2c947640 1 --2- 172.21.15.125:0/2933750076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b280b0dc0 0x7f9b28071720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b1ffff640 1 -- 172.21.15.125:0/2933750076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b280b0dc0 msgr2=0x7f9b28071720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b1ffff640 1 --2- 172.21.15.125:0/2933750076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b280b0dc0 0x7f9b28071720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b1ffff640 1 -- 172.21.15.125:0/2933750076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b28153b10 msgr2=0x7f9b281f0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b1ffff640 1 --2- 172.21.15.125:0/2933750076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b28153b10 0x7f9b281f0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:37.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b1ffff640 1 -- 172.21.15.125:0/2933750076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b18009420 con 0x7f9b280c5e70 2023-10-30T13:02:37.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b1ffff640 1 --2- 172.21.15.125:0/2933750076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b280c5e70 0x7f9b281f07d0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f9b1801c880 tx=0x7f9b1801b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:37.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b1d7fa640 1 -- 172.21.15.125:0/2933750076 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b180b9630 con 0x7f9b280c5e70 2023-10-30T13:02:37.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b1d7fa640 1 -- 172.21.15.125:0/2933750076 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b180b97b0 con 0x7f9b280c5e70 2023-10-30T13:02:37.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b2ebab640 1 -- 172.21.15.125:0/2933750076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2814fa30 con 0x7f9b280c5e70 2023-10-30T13:02:37.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b1d7fa640 1 -- 172.21.15.125:0/2933750076 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b180b9b60 con 0x7f9b280c5e70 2023-10-30T13:02:37.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b2ebab640 1 -- 172.21.15.125:0/2933750076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b2814fcf0 con 0x7f9b280c5e70 2023-10-30T13:02:37.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.854+0000 7f9b2ebab640 1 -- 172.21.15.125:0/2933750076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b28074250 con 0x7f9b280c5e70 2023-10-30T13:02:37.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.858+0000 7f9b1d7fa640 1 -- 172.21.15.125:0/2933750076 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9b180b7040 con 0x7f9b280c5e70 2023-10-30T13:02:37.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.858+0000 7f9b1d7fa640 1 --2- 172.21.15.125:0/2933750076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b140664d0 0x7f9b14068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:37.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.858+0000 7f9b1d7fa640 1 -- 172.21.15.125:0/2933750076 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f9b181224c0 con 0x7f9b280c5e70 2023-10-30T13:02:37.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.858+0000 7f9b2c947640 1 --2- 172.21.15.125:0/2933750076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b140664d0 0x7f9b14068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:37.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.858+0000 7f9b2c947640 1 --2- 172.21.15.125:0/2933750076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b140664d0 0x7f9b14068990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9b24000f70 tx=0x7f9b24007c10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:37.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:37.862+0000 7f9b1d7fa640 1 -- 172.21.15.125:0/2933750076 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9b180ef270 con 0x7f9b280c5e70 2023-10-30T13:02:38.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:38.042+0000 7f9b2ebab640 1 -- 172.21.15.125:0/2933750076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b2815a110 con 0x7f9b280c5e70 2023-10-30T13:02:38.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:38.042+0000 7f9b1d7fa640 1 -- 172.21.15.125:0/2933750076 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f9b180f2890 con 0x7f9b280c5e70 2023-10-30T13:02:38.045 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:38.045 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:35.533876+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:38.045 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 151 2023-10-30T13:02:38.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:38.042+0000 7f9b2ebab640 1 -- 172.21.15.125:0/2933750076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b140664d0 msgr2=0x7f9b14068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:38.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:38.042+0000 7f9b2ebab640 1 --2- 172.21.15.125:0/2933750076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b140664d0 0x7f9b14068990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f9b24000f70 tx=0x7f9b24007c10 comp rx=0 tx=0).stop 2023-10-30T13:02:38.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:38.042+0000 7f9b2ebab640 1 -- 172.21.15.125:0/2933750076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b280c5e70 msgr2=0x7f9b281f07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:38.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:38.042+0000 7f9b2ebab640 1 --2- 172.21.15.125:0/2933750076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b280c5e70 0x7f9b281f07d0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f9b1801c880 tx=0x7f9b1801b9f0 comp rx=0 tx=0).stop 2023-10-30T13:02:38.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:38.042+0000 7f9b2ebab640 1 -- 172.21.15.125:0/2933750076 shutdown_connections 2023-10-30T13:02:38.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:38.042+0000 7f9b2ebab640 1 --2- 172.21.15.125:0/2933750076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9b140664d0 0x7f9b14068990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:38.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:38.042+0000 7f9b2ebab640 1 --2- 172.21.15.125:0/2933750076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9b28153b10 0x7f9b281f0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:38.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:38.042+0000 7f9b2ebab640 1 --2- 172.21.15.125:0/2933750076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9b280c5e70 0x7f9b281f07d0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:38.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:38.042+0000 7f9b2ebab640 1 --2- 172.21.15.125:0/2933750076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9b280b0dc0 0x7f9b28071720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:38.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:38.042+0000 7f9b2ebab640 1 -- 172.21.15.125:0/2933750076 >> 172.21.15.125:0/2933750076 conn(0x7f9b2806d770 msgr2=0x7f9b28069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:38.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:38.042+0000 7f9b2ebab640 1 -- 172.21.15.125:0/2933750076 shutdown_connections 2023-10-30T13:02:38.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:38.042+0000 7f9b2ebab640 1 -- 172.21.15.125:0/2933750076 wait complete. 2023-10-30T13:02:38.066 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 151, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 151, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7866": { "addr": "172.21.15.125:6836/1301336591", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1301336591, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1301336591, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7866, "incarnation": 151, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 72, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:02:35.533876+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7866 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:02:40.067 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:40.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.210+0000 7f2e99a17640 1 Processor -- start 2023-10-30T13:02:40.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e99a17640 1 -- start start 2023-10-30T13:02:40.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e99a17640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2e940c6bd0 0x7f2e9406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:40.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e99a17640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2e94071110 0x7f2e94158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:40.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e99a17640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e94066cf0 con 0x7f2e9406f7a0 2023-10-30T13:02:40.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e99a17640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e940700c0 con 0x7f2e940c6bd0 2023-10-30T13:02:40.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e99a17640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e940c70d0 con 0x7f2e94071110 2023-10-30T13:02:40.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e8a7fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2e940c6bd0 0x7f2e9406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:40.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e92ffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2e9406f7a0 0x7f2e94069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58808/0 (socket says 172.21.15.125:58808) 2023-10-30T13:02:40.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e92ffd640 1 -- 172.21.15.125:0/788325377 learned_addr learned my addr 172.21.15.125:0/788325377 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:02:40.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e937fe640 1 --2- 172.21.15.125:0/788325377 >> v2:172.21.15.136:3300/0 conn(0x7f2e94071110 0x7f2e94158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:40.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e8a7fc640 1 -- 172.21.15.125:0/788325377 >> v2:172.21.15.136:3300/0 conn(0x7f2e94071110 msgr2=0x7f2e94158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:40.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e8a7fc640 1 --2- 172.21.15.125:0/788325377 >> v2:172.21.15.136:3300/0 conn(0x7f2e94071110 0x7f2e94158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:40.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e8a7fc640 1 -- 172.21.15.125:0/788325377 >> v1:172.21.15.125:6789/0 conn(0x7f2e9406f7a0 legacy=0x7f2e94069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:40.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e8a7fc640 1 -- 172.21.15.125:0/788325377 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e940c7390 con 0x7f2e940c6bd0 2023-10-30T13:02:40.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.214+0000 7f2e927fc640 1 -- 172.21.15.125:0/788325377 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355781099 0 0) 0x7f2e94066cf0 con 0x7f2e9406f7a0 2023-10-30T13:02:40.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.218+0000 7f2e8a7fc640 1 --2- 172.21.15.125:0/788325377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e940c6bd0 0x7f2e9406fb80 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f2e9406cdd0 tx=0x7f2e7c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c29d2828acfffc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:40.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.218+0000 7f2e927fc640 1 -- 172.21.15.125:0/788325377 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c0a8950 con 0x7f2e940c6bd0 2023-10-30T13:02:40.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.218+0000 7f2e927fc640 1 -- 172.21.15.125:0/788325377 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7c0a8ab0 con 0x7f2e940c6bd0 2023-10-30T13:02:40.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.218+0000 7f2e927fc640 1 -- 172.21.15.125:0/788325377 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c0a8e40 con 0x7f2e940c6bd0 2023-10-30T13:02:40.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.218+0000 7f2e99a17640 1 -- 172.21.15.125:0/788325377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e940c6bd0 msgr2=0x7f2e9406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:40.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.218+0000 7f2e99a17640 1 --2- 172.21.15.125:0/788325377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e940c6bd0 0x7f2e9406fb80 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f2e9406cdd0 tx=0x7f2e7c01ec50 comp rx=0 tx=0).stop 2023-10-30T13:02:40.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.218+0000 7f2e99a17640 1 -- 172.21.15.125:0/788325377 shutdown_connections 2023-10-30T13:02:40.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.218+0000 7f2e99a17640 1 --2- 172.21.15.125:0/788325377 >> v2:172.21.15.136:3300/0 conn(0x7f2e94071110 0x7f2e94158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:40.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.218+0000 7f2e99a17640 1 --2- 172.21.15.125:0/788325377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e940c6bd0 0x7f2e9406fb80 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:40.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.218+0000 7f2e99a17640 1 -- 172.21.15.125:0/788325377 >> 172.21.15.125:0/788325377 conn(0x7f2e940c7910 msgr2=0x7f2e940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:40.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.218+0000 7f2e99a17640 1 -- 172.21.15.125:0/788325377 shutdown_connections 2023-10-30T13:02:40.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.218+0000 7f2e99a17640 1 -- 172.21.15.125:0/788325377 wait complete. 2023-10-30T13:02:40.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.218+0000 7f2e99a17640 1 Processor -- start 2023-10-30T13:02:40.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.218+0000 7f2e99a17640 1 -- start start 2023-10-30T13:02:40.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e99a17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e9406f7a0 0x7f2e94156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:40.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e99a17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e94071110 0x7f2e941f2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:40.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e99a17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e940c6bd0 0x7f2e9414f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:40.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e99a17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e941f1f00 con 0x7f2e94071110 2023-10-30T13:02:40.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e99a17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e940afbf0 con 0x7f2e940c6bd0 2023-10-30T13:02:40.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e99a17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e941eedc0 con 0x7f2e9406f7a0 2023-10-30T13:02:40.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e8a7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e94071110 0x7f2e941f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:40.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e8a7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e94071110 0x7f2e941f2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39058/0 (socket says 172.21.15.125:39058) 2023-10-30T13:02:40.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e8a7fc640 1 -- 172.21.15.125:0/2086055726 learned_addr learned my addr 172.21.15.125:0/2086055726 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:40.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e92ffd640 1 --2- 172.21.15.125:0/2086055726 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e9406f7a0 0x7f2e94156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:40.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e937fe640 1 --2- 172.21.15.125:0/2086055726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e940c6bd0 0x7f2e9414f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:40.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e8a7fc640 1 -- 172.21.15.125:0/2086055726 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e9406f7a0 msgr2=0x7f2e94156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:40.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e8a7fc640 1 --2- 172.21.15.125:0/2086055726 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e9406f7a0 0x7f2e94156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:40.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e8a7fc640 1 -- 172.21.15.125:0/2086055726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e940c6bd0 msgr2=0x7f2e9414f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:40.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e8a7fc640 1 --2- 172.21.15.125:0/2086055726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e940c6bd0 0x7f2e9414f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:40.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e8a7fc640 1 -- 172.21.15.125:0/2086055726 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c009420 con 0x7f2e94071110 2023-10-30T13:02:40.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e8a7fc640 1 --2- 172.21.15.125:0/2086055726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e94071110 0x7f2e941f2670 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c01b860 tx=0x7f2e7c01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:40.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e90ff9640 1 -- 172.21.15.125:0/2086055726 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c0bb640 con 0x7f2e94071110 2023-10-30T13:02:40.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e90ff9640 1 -- 172.21.15.125:0/2086055726 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7c0bb7a0 con 0x7f2e94071110 2023-10-30T13:02:40.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e99a17640 1 -- 172.21.15.125:0/2086055726 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e941ef0a0 con 0x7f2e94071110 2023-10-30T13:02:40.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e90ff9640 1 -- 172.21.15.125:0/2086055726 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c0bbb10 con 0x7f2e94071110 2023-10-30T13:02:40.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e99a17640 1 -- 172.21.15.125:0/2086055726 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e941ef660 con 0x7f2e94071110 2023-10-30T13:02:40.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.222+0000 7f2e99a17640 1 -- 172.21.15.125:0/2086055726 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e9404e170 con 0x7f2e94071110 2023-10-30T13:02:40.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.226+0000 7f2e90ff9640 1 -- 172.21.15.125:0/2086055726 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2e7c0b7040 con 0x7f2e94071110 2023-10-30T13:02:40.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.226+0000 7f2e90ff9640 1 --2- 172.21.15.125:0/2086055726 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e740664d0 0x7f2e74068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:40.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.226+0000 7f2e90ff9640 1 -- 172.21.15.125:0/2086055726 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f2e7c122860 con 0x7f2e94071110 2023-10-30T13:02:40.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.226+0000 7f2e92ffd640 1 --2- 172.21.15.125:0/2086055726 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e740664d0 0x7f2e74068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:40.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.226+0000 7f2e92ffd640 1 --2- 172.21.15.125:0/2086055726 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e740664d0 0x7f2e74068990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2e84000f70 tx=0x7f2e84007c10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:40.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.230+0000 7f2e90ff9640 1 -- 172.21.15.125:0/2086055726 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2e7c0ef6b0 con 0x7f2e94071110 2023-10-30T13:02:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.410+0000 7f2e99a17640 1 -- 172.21.15.125:0/2086055726 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e941510d0 con 0x7f2e94071110 2023-10-30T13:02:40.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.410+0000 7f2e90ff9640 1 -- 172.21.15.125:0/2086055726 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f2e7c0b5090 con 0x7f2e94071110 2023-10-30T13:02:40.413 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:40.414 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:35.533876+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:40.414 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 151 2023-10-30T13:02:40.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.414+0000 7f2e99a17640 1 -- 172.21.15.125:0/2086055726 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e740664d0 msgr2=0x7f2e74068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:40.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.414+0000 7f2e99a17640 1 --2- 172.21.15.125:0/2086055726 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e740664d0 0x7f2e74068990 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f2e84000f70 tx=0x7f2e84007c10 comp rx=0 tx=0).stop 2023-10-30T13:02:40.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.414+0000 7f2e99a17640 1 -- 172.21.15.125:0/2086055726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e94071110 msgr2=0x7f2e941f2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:40.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.414+0000 7f2e99a17640 1 --2- 172.21.15.125:0/2086055726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e94071110 0x7f2e941f2670 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c01b860 tx=0x7f2e7c01b9f0 comp rx=0 tx=0).stop 2023-10-30T13:02:40.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.414+0000 7f2e99a17640 1 -- 172.21.15.125:0/2086055726 shutdown_connections 2023-10-30T13:02:40.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.414+0000 7f2e99a17640 1 --2- 172.21.15.125:0/2086055726 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e740664d0 0x7f2e74068990 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:40.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.414+0000 7f2e99a17640 1 --2- 172.21.15.125:0/2086055726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e940c6bd0 0x7f2e9414f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.414+0000 7f2e99a17640 1 --2- 172.21.15.125:0/2086055726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e94071110 0x7f2e941f2670 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.414+0000 7f2e99a17640 1 --2- 172.21.15.125:0/2086055726 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e9406f7a0 0x7f2e94156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.414+0000 7f2e99a17640 1 -- 172.21.15.125:0/2086055726 >> 172.21.15.125:0/2086055726 conn(0x7f2e940c7910 msgr2=0x7f2e940c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:40.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.414+0000 7f2e99a17640 1 -- 172.21.15.125:0/2086055726 shutdown_connections 2023-10-30T13:02:40.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:40.414+0000 7f2e99a17640 1 -- 172.21.15.125:0/2086055726 wait complete. 2023-10-30T13:02:40.434 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:02:42.435 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:42.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2f209640 1 Processor -- start 2023-10-30T13:02:42.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2f209640 1 -- start start 2023-10-30T13:02:42.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2f209640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9f2806f7a0 0x7f9f2806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:42.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2f209640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9f28071110 0x7f9f28158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:42.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2f209640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f28066cf0 con 0x7f9f280c6bd0 2023-10-30T13:02:42.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2f209640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f280700c0 con 0x7f9f28071110 2023-10-30T13:02:42.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2f209640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f280c70d0 con 0x7f9f2806f7a0 2023-10-30T13:02:42.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2d7a6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9f28071110 0x7f9f28158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:42.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2d7a6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9f28071110 0x7f9f28158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34874/0 (socket says 172.21.15.125:34874) 2023-10-30T13:02:42.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2d7a6640 1 -- 172.21.15.125:0/253546606 learned_addr learned my addr 172.21.15.125:0/253546606 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:42.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2cfa5640 1 --2- 172.21.15.125:0/253546606 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9f2806f7a0 0x7f9f2806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:42.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2d7a6640 1 -- 172.21.15.125:0/253546606 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9f2806f7a0 msgr2=0x7f9f2806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:42.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2d7a6640 1 --2- 172.21.15.125:0/253546606 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9f2806f7a0 0x7f9f2806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:42.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2d7a6640 1 -- 172.21.15.125:0/253546606 >> v1:172.21.15.125:6789/0 conn(0x7f9f280c6bd0 legacy=0x7f9f28069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:42.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.582+0000 7f9f2d7a6640 1 -- 172.21.15.125:0/253546606 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f280c7390 con 0x7f9f28071110 2023-10-30T13:02:42.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.586+0000 7f9f2d7a6640 1 --2- 172.21.15.125:0/253546606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f28071110 0x7f9f28158110 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f9f20009e50 tx=0x7f9f2001f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf2169de58d8470f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:42.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.586+0000 7f9f1f7fe640 1 -- 172.21.15.125:0/253546606 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f200af400 con 0x7f9f28071110 2023-10-30T13:02:42.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.586+0000 7f9f1f7fe640 1 -- 172.21.15.125:0/253546606 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f200ab050 con 0x7f9f28071110 2023-10-30T13:02:42.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.586+0000 7f9f1f7fe640 1 -- 172.21.15.125:0/253546606 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f200af780 con 0x7f9f28071110 2023-10-30T13:02:42.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.586+0000 7f9f2f209640 1 -- 172.21.15.125:0/253546606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f28071110 msgr2=0x7f9f28158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:42.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.586+0000 7f9f2f209640 1 --2- 172.21.15.125:0/253546606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f28071110 0x7f9f28158110 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f9f20009e50 tx=0x7f9f2001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:02:42.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.586+0000 7f9f2f209640 1 -- 172.21.15.125:0/253546606 shutdown_connections 2023-10-30T13:02:42.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.586+0000 7f9f2f209640 1 --2- 172.21.15.125:0/253546606 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f28071110 0x7f9f28158110 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:42.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.586+0000 7f9f2f209640 1 --2- 172.21.15.125:0/253546606 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9f2806f7a0 0x7f9f2806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:42.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.586+0000 7f9f2f209640 1 -- 172.21.15.125:0/253546606 >> 172.21.15.125:0/253546606 conn(0x7f9f280c7910 msgr2=0x7f9f280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:42.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.586+0000 7f9f2f209640 1 -- 172.21.15.125:0/253546606 shutdown_connections 2023-10-30T13:02:42.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.586+0000 7f9f2f209640 1 -- 172.21.15.125:0/253546606 wait complete. 2023-10-30T13:02:42.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2f209640 1 Processor -- start 2023-10-30T13:02:42.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2f209640 1 -- start start 2023-10-30T13:02:42.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2f209640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f2806f7a0 0x7f9f2815a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:42.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2f209640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9f28071110 0x7f9f2814db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:42.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2f209640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f280c6bd0 0x7f9f28159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:42.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2f209640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f28155170 con 0x7f9f280c6bd0 2023-10-30T13:02:42.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2f209640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f281547e0 con 0x7f9f2806f7a0 2023-10-30T13:02:42.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2f209640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f2815a8f0 con 0x7f9f28071110 2023-10-30T13:02:42.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2d7a6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f280c6bd0 0x7f9f28159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:42.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2d7a6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f280c6bd0 0x7f9f28159240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34880/0 (socket says 172.21.15.125:34880) 2023-10-30T13:02:42.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2d7a6640 1 -- 172.21.15.125:0/300777712 learned_addr learned my addr 172.21.15.125:0/300777712 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:42.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f1ffff640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9f28071110 0x7f9f2814db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:42.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2cfa5640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f2806f7a0 0x7f9f2815a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:42.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2d7a6640 1 -- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9f28071110 msgr2=0x7f9f2814db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:42.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2d7a6640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9f28071110 0x7f9f2814db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:42.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2d7a6640 1 -- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f2806f7a0 msgr2=0x7f9f2815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:42.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2d7a6640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f2806f7a0 0x7f9f2815a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:42.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2d7a6640 1 -- 172.21.15.125:0/300777712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f20009b30 con 0x7f9f280c6bd0 2023-10-30T13:02:42.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f1ffff640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9f28071110 0x7f9f2814db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:02:42.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2d7a6640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f280c6bd0 0x7f9f28159240 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f9f2001ce20 tx=0x7f9f2001bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:42.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f1d7fa640 1 -- 172.21.15.125:0/300777712 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f200afde0 con 0x7f9f280c6bd0 2023-10-30T13:02:42.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f1d7fa640 1 -- 172.21.15.125:0/300777712 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f20009510 con 0x7f9f280c6bd0 2023-10-30T13:02:42.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f1d7fa640 1 -- 172.21.15.125:0/300777712 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f200ba660 con 0x7f9f280c6bd0 2023-10-30T13:02:42.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2f209640 1 -- 172.21.15.125:0/300777712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f28154bf0 con 0x7f9f280c6bd0 2023-10-30T13:02:42.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2f209640 1 -- 172.21.15.125:0/300777712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f28158a40 con 0x7f9f280c6bd0 2023-10-30T13:02:42.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.590+0000 7f9f2f209640 1 -- 172.21.15.125:0/300777712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f281588a0 con 0x7f9f280c6bd0 2023-10-30T13:02:42.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.594+0000 7f9f1d7fa640 1 -- 172.21.15.125:0/300777712 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9f200b6040 con 0x7f9f280c6bd0 2023-10-30T13:02:42.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.594+0000 7f9f1d7fa640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9f140664a0 0x7f9f14068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:42.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.594+0000 7f9f1d7fa640 1 -- 172.21.15.125:0/300777712 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f9f201226e0 con 0x7f9f280c6bd0 2023-10-30T13:02:42.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.594+0000 7f9f2cfa5640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9f140664a0 0x7f9f14068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:42.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.594+0000 7f9f2cfa5640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9f140664a0 0x7f9f14068960 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f9f280c8f20 tx=0x7f9f24008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:42.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.598+0000 7f9f1d7fa640 1 -- 172.21.15.125:0/300777712 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9f281588a0 con 0x7f9f280c6bd0 2023-10-30T13:02:42.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.798+0000 7f9f2f209640 1 -- 172.21.15.125:0/300777712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f281588a0 con 0x7f9f280c6bd0 2023-10-30T13:02:42.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.798+0000 7f9f1d7fa640 1 -- 172.21.15.125:0/300777712 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f9f281588a0 con 0x7f9f280c6bd0 2023-10-30T13:02:42.802 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:42.802 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:35.533876+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:42.802 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 151 2023-10-30T13:02:42.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.802+0000 7f9f2f209640 1 -- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9f140664a0 msgr2=0x7f9f14068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:42.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.802+0000 7f9f2f209640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9f140664a0 0x7f9f14068960 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f9f280c8f20 tx=0x7f9f24008040 comp rx=0 tx=0).stop 2023-10-30T13:02:42.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.802+0000 7f9f2f209640 1 -- 172.21.15.125:0/300777712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f280c6bd0 msgr2=0x7f9f28159240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:42.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.802+0000 7f9f2f209640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f280c6bd0 0x7f9f28159240 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f9f2001ce20 tx=0x7f9f2001bf90 comp rx=0 tx=0).stop 2023-10-30T13:02:42.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.802+0000 7f9f2f209640 1 -- 172.21.15.125:0/300777712 shutdown_connections 2023-10-30T13:02:42.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.802+0000 7f9f2f209640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9f140664a0 0x7f9f14068960 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:42.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.802+0000 7f9f2f209640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9f280c6bd0 0x7f9f28159240 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:42.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.802+0000 7f9f2f209640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9f28071110 0x7f9f2814db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:42.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.802+0000 7f9f2f209640 1 --2- 172.21.15.125:0/300777712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9f2806f7a0 0x7f9f2815a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:42.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.802+0000 7f9f2f209640 1 -- 172.21.15.125:0/300777712 >> 172.21.15.125:0/300777712 conn(0x7f9f280c7910 msgr2=0x7f9f280ca670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:42.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.802+0000 7f9f2f209640 1 -- 172.21.15.125:0/300777712 shutdown_connections 2023-10-30T13:02:42.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:42.802+0000 7f9f2f209640 1 -- 172.21.15.125:0/300777712 wait complete. 2023-10-30T13:02:42.826 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:02:44.827 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:44.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.970+0000 7ff0a6d0e640 1 Processor -- start 2023-10-30T13:02:44.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.970+0000 7ff0a6d0e640 1 -- start start 2023-10-30T13:02:44.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.970+0000 7ff0a6d0e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff0a00c05d0 0x7ff0a013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:44.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.970+0000 7ff0a6d0e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0a00c6f60 0x7ff0a013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:44.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.970+0000 7ff0a6d0e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a00c0a50 con 0x7ff0a00c6b80 2023-10-30T13:02:44.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.970+0000 7ff0a6d0e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a006f6e0 con 0x7ff0a00c05d0 2023-10-30T13:02:44.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.970+0000 7ff0a6d0e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a006f9c0 con 0x7ff0a00c6f60 2023-10-30T13:02:44.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.970+0000 7ff09ffff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff0a00c05d0 0x7ff0a013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:44.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.970+0000 7ff0a52ab640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0a00c6f60 0x7ff0a013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:44.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.970+0000 7ff0a4aaa640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff0a00c6b80 0x7ff0a00bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:45084/0 (socket says 172.21.15.125:45084) 2023-10-30T13:02:44.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.970+0000 7ff0a4aaa640 1 -- 172.21.15.125:0/1543177502 learned_addr learned my addr 172.21.15.125:0/1543177502 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:02:44.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.974+0000 7ff09ffff640 1 -- 172.21.15.125:0/1543177502 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0a00c6f60 msgr2=0x7ff0a013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:44.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.974+0000 7ff09ffff640 1 --2- 172.21.15.125:0/1543177502 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0a00c6f60 0x7ff0a013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:44.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.974+0000 7ff09ffff640 1 -- 172.21.15.125:0/1543177502 >> v1:172.21.15.136:6789/0 conn(0x7ff0a00c6b80 legacy=0x7ff0a00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:44.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.974+0000 7ff09ffff640 1 -- 172.21.15.125:0/1543177502 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a006fca0 con 0x7ff0a00c05d0 2023-10-30T13:02:44.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.974+0000 7ff09f7fe640 1 -- 172.21.15.125:0/1543177502 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744674837 0 0) 0x7ff0a00c0a50 con 0x7ff0a00c6b80 2023-10-30T13:02:44.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.974+0000 7ff09ffff640 1 --2- 172.21.15.125:0/1543177502 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0a00c05d0 0x7ff0a013ed90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff08c0097c0 tx=0x7ff08c01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cfe62e5562428af5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:44.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.974+0000 7ff09f7fe640 1 -- 172.21.15.125:0/1543177502 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c0ad640 con 0x7ff0a00c05d0 2023-10-30T13:02:44.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.974+0000 7ff09f7fe640 1 -- 172.21.15.125:0/1543177502 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff08c0ad7a0 con 0x7ff0a00c05d0 2023-10-30T13:02:44.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.974+0000 7ff09f7fe640 1 -- 172.21.15.125:0/1543177502 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff08c0adb30 con 0x7ff0a00c05d0 2023-10-30T13:02:44.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/1543177502 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0a00c05d0 msgr2=0x7ff0a013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:44.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 --2- 172.21.15.125:0/1543177502 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0a00c05d0 0x7ff0a013ed90 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff08c0097c0 tx=0x7ff08c01c480 comp rx=0 tx=0).stop 2023-10-30T13:02:44.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/1543177502 shutdown_connections 2023-10-30T13:02:44.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 --2- 172.21.15.125:0/1543177502 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0a00c6f60 0x7ff0a013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:44.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 --2- 172.21.15.125:0/1543177502 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0a00c05d0 0x7ff0a013ed90 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:44.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/1543177502 >> 172.21.15.125:0/1543177502 conn(0x7ff0a00c7910 msgr2=0x7ff0a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:44.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/1543177502 shutdown_connections 2023-10-30T13:02:44.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/1543177502 wait complete. 2023-10-30T13:02:44.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 Processor -- start 2023-10-30T13:02:44.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 -- start start 2023-10-30T13:02:44.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0a00c05d0 0x7ff0a0158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:44.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0a00c6b80 0x7ff0a014bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:44.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0a00c6f60 0x7ff0a0150450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:44.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a01314e0 con 0x7ff0a00c05d0 2023-10-30T13:02:44.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a0157910 con 0x7ff0a00c6b80 2023-10-30T13:02:44.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a6d0e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a0157bf0 con 0x7ff0a00c6f60 2023-10-30T13:02:44.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a4aaa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0a00c05d0 0x7ff0a0158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:44.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a4aaa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0a00c05d0 0x7ff0a0158ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34896/0 (socket says 172.21.15.125:34896) 2023-10-30T13:02:44.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a4aaa640 1 -- 172.21.15.125:0/2199800693 learned_addr learned my addr 172.21.15.125:0/2199800693 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:44.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff0a52ab640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0a00c6f60 0x7ff0a0150450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:44.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.978+0000 7ff09ffff640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0a00c6b80 0x7ff0a014bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:44.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff0a4aaa640 1 -- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0a00c6f60 msgr2=0x7ff0a0150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:44.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff0a4aaa640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0a00c6f60 0x7ff0a0150450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:44.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff0a4aaa640 1 -- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0a00c6b80 msgr2=0x7ff0a014bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:44.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff0a4aaa640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0a00c6b80 0x7ff0a014bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:44.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff0a4aaa640 1 -- 172.21.15.125:0/2199800693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08c009420 con 0x7ff0a00c05d0 2023-10-30T13:02:44.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff09ffff640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0a00c6b80 0x7ff0a014bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:02:44.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff0a4aaa640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0a00c05d0 0x7ff0a0158ae0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7ff09400c4a0 tx=0x7ff09400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:44.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff09d7fa640 1 -- 172.21.15.125:0/2199800693 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0940074e0 con 0x7ff0a00c05d0 2023-10-30T13:02:44.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff09d7fa640 1 -- 172.21.15.125:0/2199800693 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff094007640 con 0x7ff0a00c05d0 2023-10-30T13:02:44.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/2199800693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a0131780 con 0x7ff0a00c05d0 2023-10-30T13:02:44.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff09d7fa640 1 -- 172.21.15.125:0/2199800693 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff094004a30 con 0x7ff0a00c05d0 2023-10-30T13:02:44.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/2199800693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a0150990 con 0x7ff0a00c05d0 2023-10-30T13:02:44.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/2199800693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a004e170 con 0x7ff0a00c05d0 2023-10-30T13:02:44.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff09d7fa640 1 -- 172.21.15.125:0/2199800693 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff094004b90 con 0x7ff0a00c05d0 2023-10-30T13:02:44.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff09d7fa640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0840664a0 0x7ff084068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:44.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff09d7fa640 1 -- 172.21.15.125:0/2199800693 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7ff094083790 con 0x7ff0a00c05d0 2023-10-30T13:02:44.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.982+0000 7ff09ffff640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0840664a0 0x7ff084068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:44.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.986+0000 7ff09ffff640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0840664a0 0x7ff084068960 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff08c0af000 tx=0x7ff08c0a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:44.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:44.990+0000 7ff09d7fa640 1 -- 172.21.15.125:0/2199800693 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff094050570 con 0x7ff0a00c05d0 2023-10-30T13:02:45.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:45.186+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/2199800693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0a014c150 con 0x7ff0a00c05d0 2023-10-30T13:02:45.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:45.186+0000 7ff09d7fa640 1 -- 172.21.15.125:0/2199800693 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+5723 (secure 0 0 0) 0x7ff094053b90 con 0x7ff0a00c05d0 2023-10-30T13:02:45.190 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:45.191 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:35.533876+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:45.191 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 151 2023-10-30T13:02:45.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:45.190+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0840664a0 msgr2=0x7ff084068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:45.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:45.190+0000 7ff0a6d0e640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0840664a0 0x7ff084068960 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7ff08c0af000 tx=0x7ff08c0a7070 comp rx=0 tx=0).stop 2023-10-30T13:02:45.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:45.190+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/2199800693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0a00c05d0 msgr2=0x7ff0a0158ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:45.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:45.190+0000 7ff0a6d0e640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0a00c05d0 0x7ff0a0158ae0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7ff09400c4a0 tx=0x7ff09400c960 comp rx=0 tx=0).stop 2023-10-30T13:02:45.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:45.190+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/2199800693 shutdown_connections 2023-10-30T13:02:45.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:45.190+0000 7ff0a6d0e640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0840664a0 0x7ff084068960 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:45.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:45.190+0000 7ff0a6d0e640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff0a00c6f60 0x7ff0a0150450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:45.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:45.190+0000 7ff0a6d0e640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0a00c6b80 0x7ff0a014bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:45.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:45.190+0000 7ff0a6d0e640 1 --2- 172.21.15.125:0/2199800693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0a00c05d0 0x7ff0a0158ae0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:45.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:45.190+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/2199800693 >> 172.21.15.125:0/2199800693 conn(0x7ff0a00c7910 msgr2=0x7ff0a006be40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:45.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:45.190+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/2199800693 shutdown_connections 2023-10-30T13:02:45.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:45.190+0000 7ff0a6d0e640 1 -- 172.21.15.125:0/2199800693 wait complete. 2023-10-30T13:02:45.211 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:02:47.212 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:47.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f673d09c640 1 Processor -- start 2023-10-30T13:02:47.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f673d09c640 1 -- start start 2023-10-30T13:02:47.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f673d09c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f67380c6b80 0x7f67380bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:47.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f673d09c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f67380c6f60 0x7f67380c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:47.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f673d09c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67380c0a50 con 0x7f67380c8430 2023-10-30T13:02:47.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f673d09c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673806f6e0 con 0x7f67380c6b80 2023-10-30T13:02:47.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f673d09c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673806f9c0 con 0x7f67380c6f60 2023-10-30T13:02:47.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f6736d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f67380c6b80 0x7f67380bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:47.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f6736d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f67380c6b80 0x7f67380bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34898/0 (socket says 172.21.15.125:34898) 2023-10-30T13:02:47.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f6736d9d640 1 -- 172.21.15.125:0/2022623756 learned_addr learned my addr 172.21.15.125:0/2022623756 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:47.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f672e59c640 1 --2- 172.21.15.125:0/2022623756 >> v2:172.21.15.136:3300/0 conn(0x7f67380c6f60 0x7f67380c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:47.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f6736d9d640 1 -- 172.21.15.125:0/2022623756 >> v2:172.21.15.136:3300/0 conn(0x7f67380c6f60 msgr2=0x7f67380c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:47.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f6736d9d640 1 --2- 172.21.15.125:0/2022623756 >> v2:172.21.15.136:3300/0 conn(0x7f67380c6f60 0x7f67380c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:47.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f6736d9d640 1 -- 172.21.15.125:0/2022623756 >> v1:172.21.15.136:6789/0 conn(0x7f67380c8430 legacy=0x7f673813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:47.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.358+0000 7f6736d9d640 1 -- 172.21.15.125:0/2022623756 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673806fca0 con 0x7f67380c6b80 2023-10-30T13:02:47.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.362+0000 7f6736d9d640 1 --2- 172.21.15.125:0/2022623756 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67380c6b80 0x7f67380bfec0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f6738159b90 tx=0x7f672801eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460dbb8346ed8b52 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:47.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.362+0000 7f673659c640 1 -- 172.21.15.125:0/2022623756 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67280a7b70 con 0x7f67380c6b80 2023-10-30T13:02:47.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.362+0000 7f673659c640 1 -- 172.21.15.125:0/2022623756 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67280a7cd0 con 0x7f67380c6b80 2023-10-30T13:02:47.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.362+0000 7f673659c640 1 -- 172.21.15.125:0/2022623756 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67280af660 con 0x7f67380c6b80 2023-10-30T13:02:47.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.362+0000 7f673d09c640 1 -- 172.21.15.125:0/2022623756 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67380c6b80 msgr2=0x7f67380bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:47.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.362+0000 7f673d09c640 1 --2- 172.21.15.125:0/2022623756 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67380c6b80 0x7f67380bfec0 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f6738159b90 tx=0x7f672801eb90 comp rx=0 tx=0).stop 2023-10-30T13:02:47.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.362+0000 7f673d09c640 1 -- 172.21.15.125:0/2022623756 shutdown_connections 2023-10-30T13:02:47.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.362+0000 7f673d09c640 1 --2- 172.21.15.125:0/2022623756 >> v2:172.21.15.136:3300/0 conn(0x7f67380c6f60 0x7f67380c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:47.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.362+0000 7f673d09c640 1 --2- 172.21.15.125:0/2022623756 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67380c6b80 0x7f67380bfec0 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:47.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.362+0000 7f673d09c640 1 -- 172.21.15.125:0/2022623756 >> 172.21.15.125:0/2022623756 conn(0x7f67380c7910 msgr2=0x7f67380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:47.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.362+0000 7f673d09c640 1 -- 172.21.15.125:0/2022623756 shutdown_connections 2023-10-30T13:02:47.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.362+0000 7f673d09c640 1 -- 172.21.15.125:0/2022623756 wait complete. 2023-10-30T13:02:47.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f673d09c640 1 Processor -- start 2023-10-30T13:02:47.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f673d09c640 1 -- start start 2023-10-30T13:02:47.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f673d09c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67380c6b80 0x7f67380c16c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:47.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f673d09c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67380c6f60 0x7f6738155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:47.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f673d09c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67380c8430 0x7f67381f5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:47.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f673d09c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67380c4a60 con 0x7f67380c6b80 2023-10-30T13:02:47.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f673d09c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673813d4b0 con 0x7f67380c6f60 2023-10-30T13:02:47.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f673d09c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f673813d1d0 con 0x7f67380c8430 2023-10-30T13:02:47.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f6736d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67380c6b80 0x7f67380c16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:47.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f6736d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67380c6b80 0x7f67380c16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34904/0 (socket says 172.21.15.125:34904) 2023-10-30T13:02:47.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f6736d9d640 1 -- 172.21.15.125:0/2691299012 learned_addr learned my addr 172.21.15.125:0/2691299012 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:47.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f672e59c640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67380c6f60 0x7f6738155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:47.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f673759e640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67380c8430 0x7f67381f5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:47.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f6736d9d640 1 -- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67380c8430 msgr2=0x7f67381f5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:47.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f6736d9d640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67380c8430 0x7f67381f5500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:47.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f6736d9d640 1 -- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67380c6f60 msgr2=0x7f6738155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:47.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f6736d9d640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67380c6f60 0x7f6738155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:47.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f6736d9d640 1 -- 172.21.15.125:0/2691299012 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f672800b770 con 0x7f67380c6b80 2023-10-30T13:02:47.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f673759e640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67380c8430 0x7f67381f5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:02:47.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f6736d9d640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67380c6b80 0x7f67380c16c0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f672800b2c0 tx=0x7f672801d3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:47.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f672ffff640 1 -- 172.21.15.125:0/2691299012 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67280ba680 con 0x7f67380c6b80 2023-10-30T13:02:47.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f672ffff640 1 -- 172.21.15.125:0/2691299012 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67280ba7e0 con 0x7f67380c6b80 2023-10-30T13:02:47.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f673d09c640 1 -- 172.21.15.125:0/2691299012 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67380af6c0 con 0x7f67380c6b80 2023-10-30T13:02:47.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f672ffff640 1 -- 172.21.15.125:0/2691299012 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67280bab70 con 0x7f67380c6b80 2023-10-30T13:02:47.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f673d09c640 1 -- 172.21.15.125:0/2691299012 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67380af9a0 con 0x7f67380c6b80 2023-10-30T13:02:47.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.366+0000 7f673d09c640 1 -- 172.21.15.125:0/2691299012 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f673804e170 con 0x7f67380c6b80 2023-10-30T13:02:47.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.370+0000 7f672ffff640 1 -- 172.21.15.125:0/2691299012 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f67280bacd0 con 0x7f67380c6b80 2023-10-30T13:02:47.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.370+0000 7f672ffff640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67180664a0 0x7f6718068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:47.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.370+0000 7f672ffff640 1 -- 172.21.15.125:0/2691299012 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f67280a7e40 con 0x7f67380c6b80 2023-10-30T13:02:47.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.370+0000 7f672e59c640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67180664a0 0x7f6718068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:47.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.370+0000 7f672e59c640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67180664a0 0x7f6718068960 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f67380c1320 tx=0x7f6720008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:47.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.374+0000 7f672ffff640 1 -- 172.21.15.125:0/2691299012 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f67280ecb50 con 0x7f67380c6b80 2023-10-30T13:02:47.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.558+0000 7f673d09c640 1 -- 172.21.15.125:0/2691299012 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f673804e170 con 0x7f67380c6b80 2023-10-30T13:02:47.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.558+0000 7f672ffff640 1 -- 172.21.15.125:0/2691299012 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f67280b4070 con 0x7f67380c6b80 2023-10-30T13:02:47.560 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:47.560 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:45.096054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:47.560 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 152 2023-10-30T13:02:47.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.558+0000 7f673d09c640 1 -- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67180664a0 msgr2=0x7f6718068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:47.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.558+0000 7f673d09c640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67180664a0 0x7f6718068960 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f67380c1320 tx=0x7f6720008040 comp rx=0 tx=0).stop 2023-10-30T13:02:47.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.558+0000 7f673d09c640 1 -- 172.21.15.125:0/2691299012 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67380c6b80 msgr2=0x7f67380c16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:47.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.558+0000 7f673d09c640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67380c6b80 0x7f67380c16c0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f672800b2c0 tx=0x7f672801d3d0 comp rx=0 tx=0).stop 2023-10-30T13:02:47.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.558+0000 7f673d09c640 1 -- 172.21.15.125:0/2691299012 shutdown_connections 2023-10-30T13:02:47.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.558+0000 7f673d09c640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67180664a0 0x7f6718068960 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:47.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.558+0000 7f673d09c640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67380c8430 0x7f67381f5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:47.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.558+0000 7f673d09c640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67380c6f60 0x7f6738155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:47.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.558+0000 7f673d09c640 1 --2- 172.21.15.125:0/2691299012 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67380c6b80 0x7f67380c16c0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:47.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.558+0000 7f673d09c640 1 -- 172.21.15.125:0/2691299012 >> 172.21.15.125:0/2691299012 conn(0x7f67380c7910 msgr2=0x7f67380c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:47.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.558+0000 7f673d09c640 1 -- 172.21.15.125:0/2691299012 shutdown_connections 2023-10-30T13:02:47.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:47.558+0000 7f673d09c640 1 -- 172.21.15.125:0/2691299012 wait complete. 2023-10-30T13:02:47.582 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:02:49.583 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:49.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.726+0000 7f954f6d6640 1 Processor -- start 2023-10-30T13:02:49.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.726+0000 7f954f6d6640 1 -- start start 2023-10-30T13:02:49.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.730+0000 7f954f6d6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f95481538a0 0x7f954815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:49.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.730+0000 7f954f6d6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f95480cf160 0x7f95480cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:49.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.730+0000 7f954f6d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95480cb800 con 0x7f95480ca0e0 2023-10-30T13:02:49.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.730+0000 7f954f6d6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f954815dc30 con 0x7f95481538a0 2023-10-30T13:02:49.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.730+0000 7f954f6d6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95480cb330 con 0x7f95480cf160 2023-10-30T13:02:49.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.730+0000 7f954d472640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f95481538a0 0x7f954815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:49.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.730+0000 7f954cc71640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f95480ca0e0 0x7f954815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47840/0 (socket says 172.21.15.125:47840) 2023-10-30T13:02:49.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.730+0000 7f954d472640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f95481538a0 0x7f954815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34906/0 (socket says 172.21.15.125:34906) 2023-10-30T13:02:49.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.730+0000 7f954cc71640 1 -- 172.21.15.125:0/4203447944 learned_addr learned my addr 172.21.15.125:0/4203447944 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:02:49.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.730+0000 7f954dc73640 1 --2- 172.21.15.125:0/4203447944 >> v2:172.21.15.136:3300/0 conn(0x7f95480cf160 0x7f95480cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:49.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.730+0000 7f954d472640 1 -- 172.21.15.125:0/4203447944 >> v2:172.21.15.136:3300/0 conn(0x7f95480cf160 msgr2=0x7f95480cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:49.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.730+0000 7f954d472640 1 --2- 172.21.15.125:0/4203447944 >> v2:172.21.15.136:3300/0 conn(0x7f95480cf160 0x7f95480cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:49.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.730+0000 7f954d472640 1 -- 172.21.15.125:0/4203447944 >> v1:172.21.15.125:6789/0 conn(0x7f95480ca0e0 legacy=0x7f954815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:49.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.730+0000 7f954d472640 1 -- 172.21.15.125:0/4203447944 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95480cad70 con 0x7f95481538a0 2023-10-30T13:02:49.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954d472640 1 --2- 172.21.15.125:0/4203447944 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95481538a0 0x7f954815d6f0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f95480b5330 tx=0x7f954401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95d6353ded57d9c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:49.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f953ffff640 1 -- 172.21.15.125:0/4203447944 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95440a8950 con 0x7f95481538a0 2023-10-30T13:02:49.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f953ffff640 1 -- 172.21.15.125:0/4203447944 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95440a8ab0 con 0x7f95481538a0 2023-10-30T13:02:49.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f953ffff640 1 -- 172.21.15.125:0/4203447944 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95440a8e40 con 0x7f95481538a0 2023-10-30T13:02:49.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 -- 172.21.15.125:0/4203447944 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95481538a0 msgr2=0x7f954815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:49.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 --2- 172.21.15.125:0/4203447944 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95481538a0 0x7f954815d6f0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f95480b5330 tx=0x7f954401ec50 comp rx=0 tx=0).stop 2023-10-30T13:02:49.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 -- 172.21.15.125:0/4203447944 shutdown_connections 2023-10-30T13:02:49.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 --2- 172.21.15.125:0/4203447944 >> v2:172.21.15.136:3300/0 conn(0x7f95480cf160 0x7f95480cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:49.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 --2- 172.21.15.125:0/4203447944 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95481538a0 0x7f954815d6f0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:49.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 -- 172.21.15.125:0/4203447944 >> 172.21.15.125:0/4203447944 conn(0x7f954806bfe0 msgr2=0x7f954806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:49.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 -- 172.21.15.125:0/4203447944 shutdown_connections 2023-10-30T13:02:49.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 -- 172.21.15.125:0/4203447944 wait complete. 2023-10-30T13:02:49.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 Processor -- start 2023-10-30T13:02:49.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 -- start start 2023-10-30T13:02:49.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95480ca0e0 0x7f9548158810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:49.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95480cf160 0x7f95481e40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:49.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95481538a0 0x7f95481e4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:49.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954815d9f0 con 0x7f95481538a0 2023-10-30T13:02:49.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.734+0000 7f954f6d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954815d710 con 0x7f95480ca0e0 2023-10-30T13:02:49.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954f6d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95481e3600 con 0x7f95480cf160 2023-10-30T13:02:49.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954dc73640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95481538a0 0x7f95481e4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:49.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954dc73640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95481538a0 0x7f95481e4620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34908/0 (socket says 172.21.15.125:34908) 2023-10-30T13:02:49.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954dc73640 1 -- 172.21.15.125:0/4271143379 learned_addr learned my addr 172.21.15.125:0/4271143379 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:49.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954d472640 1 --2- 172.21.15.125:0/4271143379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95480ca0e0 0x7f9548158810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:49.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954cc71640 1 --2- 172.21.15.125:0/4271143379 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95480cf160 0x7f95481e40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:49.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954dc73640 1 -- 172.21.15.125:0/4271143379 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95480cf160 msgr2=0x7f95481e40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:49.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954dc73640 1 --2- 172.21.15.125:0/4271143379 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95480cf160 0x7f95481e40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:49.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954dc73640 1 -- 172.21.15.125:0/4271143379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95480ca0e0 msgr2=0x7f9548158810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:49.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954dc73640 1 --2- 172.21.15.125:0/4271143379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95480ca0e0 0x7f9548158810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:49.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954dc73640 1 -- 172.21.15.125:0/4271143379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9544009420 con 0x7f95481538a0 2023-10-30T13:02:49.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954dc73640 1 --2- 172.21.15.125:0/4271143379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95481538a0 0x7f95481e4620 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f9540007d90 tx=0x7f954000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:49.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f953dffb640 1 -- 172.21.15.125:0/4271143379 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540014420 con 0x7f95481538a0 2023-10-30T13:02:49.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f953dffb640 1 -- 172.21.15.125:0/4271143379 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95400145a0 con 0x7f95481538a0 2023-10-30T13:02:49.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954f6d6640 1 -- 172.21.15.125:0/4271143379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95481e38e0 con 0x7f95481538a0 2023-10-30T13:02:49.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f953dffb640 1 -- 172.21.15.125:0/4271143379 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540014950 con 0x7f95481538a0 2023-10-30T13:02:49.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954f6d6640 1 -- 172.21.15.125:0/4271143379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95481595d0 con 0x7f95481538a0 2023-10-30T13:02:49.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f954f6d6640 1 -- 172.21.15.125:0/4271143379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95480b2a50 con 0x7f95481538a0 2023-10-30T13:02:49.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f953dffb640 1 -- 172.21.15.125:0/4271143379 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9540014ab0 con 0x7f95481538a0 2023-10-30T13:02:49.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f953dffb640 1 --2- 172.21.15.125:0/4271143379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f952c0664a0 0x7f952c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:49.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.738+0000 7f953dffb640 1 -- 172.21.15.125:0/4271143379 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f9540084e00 con 0x7f95481538a0 2023-10-30T13:02:49.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.742+0000 7f954d472640 1 --2- 172.21.15.125:0/4271143379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f952c0664a0 0x7f952c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:49.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.742+0000 7f954d472640 1 --2- 172.21.15.125:0/4271143379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f952c0664a0 0x7f952c068960 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f95440b2000 tx=0x7f9544008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:49.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.746+0000 7f953dffb640 1 -- 172.21.15.125:0/4271143379 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9540051bb0 con 0x7f95481538a0 2023-10-30T13:02:49.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.926+0000 7f954f6d6640 1 -- 172.21.15.125:0/4271143379 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95481e3e00 con 0x7f95481538a0 2023-10-30T13:02:49.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.926+0000 7f953dffb640 1 -- 172.21.15.125:0/4271143379 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f95481e3e00 con 0x7f95481538a0 2023-10-30T13:02:49.930 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:49.930 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":152,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:45.096054+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:49.930 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 152 2023-10-30T13:02:49.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.930+0000 7f954f6d6640 1 -- 172.21.15.125:0/4271143379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f952c0664a0 msgr2=0x7f952c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:49.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.930+0000 7f954f6d6640 1 --2- 172.21.15.125:0/4271143379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f952c0664a0 0x7f952c068960 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f95440b2000 tx=0x7f9544008040 comp rx=0 tx=0).stop 2023-10-30T13:02:49.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.930+0000 7f954f6d6640 1 -- 172.21.15.125:0/4271143379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95481538a0 msgr2=0x7f95481e4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:49.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.930+0000 7f954f6d6640 1 --2- 172.21.15.125:0/4271143379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95481538a0 0x7f95481e4620 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f9540007d90 tx=0x7f954000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:02:49.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.930+0000 7f954f6d6640 1 -- 172.21.15.125:0/4271143379 shutdown_connections 2023-10-30T13:02:49.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.930+0000 7f954f6d6640 1 --2- 172.21.15.125:0/4271143379 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f952c0664a0 0x7f952c068960 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:49.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.930+0000 7f954f6d6640 1 --2- 172.21.15.125:0/4271143379 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95481538a0 0x7f95481e4620 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:49.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.930+0000 7f954f6d6640 1 --2- 172.21.15.125:0/4271143379 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95480cf160 0x7f95481e40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:49.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.930+0000 7f954f6d6640 1 --2- 172.21.15.125:0/4271143379 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95480ca0e0 0x7f9548158810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:49.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.930+0000 7f954f6d6640 1 -- 172.21.15.125:0/4271143379 >> 172.21.15.125:0/4271143379 conn(0x7f954806bfe0 msgr2=0x7f9548065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:49.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.930+0000 7f954f6d6640 1 -- 172.21.15.125:0/4271143379 shutdown_connections 2023-10-30T13:02:49.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:49.930+0000 7f954f6d6640 1 -- 172.21.15.125:0/4271143379 wait complete. 2023-10-30T13:02:49.951 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 152, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 152, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7866": { "addr": "172.21.15.125:6836/1301336591", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 1301336591, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 1301336591, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7866, "incarnation": 151, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:reconnect", "state_seq": 26 } }, "last_failure": 0, "last_failure_osd_epoch": 72, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:02:45.096054+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 7866 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:02:51.699 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:02:51.703 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:02:51.952 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:52.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc45142f640 1 Processor -- start 2023-10-30T13:02:52.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc45142f640 1 -- start start 2023-10-30T13:02:52.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc45142f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc44c1538f0 0x7fc44c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:52.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc45142f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc44c15cd50 0x7fc44c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:52.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc45142f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc44c15d300 con 0x7fc44c0cefa0 2023-10-30T13:02:52.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc45142f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc44c15d5e0 con 0x7fc44c1538f0 2023-10-30T13:02:52.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc45142f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c15d8c0 con 0x7fc44c15cd50 2023-10-30T13:02:52.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc44bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc44c1538f0 0x7fc44c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:52.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc44bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc44c1538f0 0x7fc44c159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44372/0 (socket says 172.21.15.125:44372) 2023-10-30T13:02:52.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc44bfff640 1 -- 172.21.15.125:0/3775302451 learned_addr learned my addr 172.21.15.125:0/3775302451 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:52.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc44b7fe640 1 --2- 172.21.15.125:0/3775302451 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc44c15cd50 0x7fc44c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:52.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc44bfff640 1 -- 172.21.15.125:0/3775302451 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc44c15cd50 msgr2=0x7fc44c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:52.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc44bfff640 1 --2- 172.21.15.125:0/3775302451 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc44c15cd50 0x7fc44c0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:52.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc44bfff640 1 -- 172.21.15.125:0/3775302451 >> v1:172.21.15.136:6789/0 conn(0x7fc44c0cefa0 legacy=0x7fc44c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:52.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.102+0000 7fc44bfff640 1 -- 172.21.15.125:0/3775302451 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c15dba0 con 0x7fc44c1538f0 2023-10-30T13:02:52.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.106+0000 7fc44bfff640 1 --2- 172.21.15.125:0/3775302451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc44c1538f0 0x7fc44c159fb0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009870 tx=0x7fc43c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c3b1be78879ea2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:52.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.106+0000 7fc44affd640 1 -- 172.21.15.125:0/3775302451 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c0a8a30 con 0x7fc44c1538f0 2023-10-30T13:02:52.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.106+0000 7fc44affd640 1 -- 172.21.15.125:0/3775302451 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43c0a8b90 con 0x7fc44c1538f0 2023-10-30T13:02:52.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.106+0000 7fc44affd640 1 -- 172.21.15.125:0/3775302451 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43c0b0440 con 0x7fc44c1538f0 2023-10-30T13:02:52.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.106+0000 7fc45142f640 1 -- 172.21.15.125:0/3775302451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc44c1538f0 msgr2=0x7fc44c159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:52.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.106+0000 7fc45142f640 1 --2- 172.21.15.125:0/3775302451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc44c1538f0 0x7fc44c159fb0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fc43c009870 tx=0x7fc43c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:02:52.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.106+0000 7fc45142f640 1 -- 172.21.15.125:0/3775302451 shutdown_connections 2023-10-30T13:02:52.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.106+0000 7fc45142f640 1 --2- 172.21.15.125:0/3775302451 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc44c15cd50 0x7fc44c0c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:52.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.106+0000 7fc45142f640 1 --2- 172.21.15.125:0/3775302451 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc44c1538f0 0x7fc44c159fb0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:52.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.106+0000 7fc45142f640 1 -- 172.21.15.125:0/3775302451 >> 172.21.15.125:0/3775302451 conn(0x7fc44c06a640 msgr2=0x7fc44c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:52.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.106+0000 7fc45142f640 1 -- 172.21.15.125:0/3775302451 shutdown_connections 2023-10-30T13:02:52.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.106+0000 7fc45142f640 1 -- 172.21.15.125:0/3775302451 wait complete. 2023-10-30T13:02:52.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.106+0000 7fc45142f640 1 Processor -- start 2023-10-30T13:02:52.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.106+0000 7fc45142f640 1 -- start start 2023-10-30T13:02:52.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc45142f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc44c0cefa0 0x7fc44c0b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:52.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc45142f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc44c1538f0 0x7fc44c1e3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:52.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc45142f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc44c15cd50 0x7fc44c1f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:52.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc45142f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c0b48f0 con 0x7fc44c1538f0 2023-10-30T13:02:52.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc45142f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c148180 con 0x7fc44c0cefa0 2023-10-30T13:02:52.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc45142f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c1f2780 con 0x7fc44c15cd50 2023-10-30T13:02:52.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc44b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc44c1538f0 0x7fc44c1e3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:52.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc44b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc44c1538f0 0x7fc44c1e3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44384/0 (socket says 172.21.15.125:44384) 2023-10-30T13:02:52.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc44b7fe640 1 -- 172.21.15.125:0/1202365573 learned_addr learned my addr 172.21.15.125:0/1202365573 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:52.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc450c2e640 1 --2- 172.21.15.125:0/1202365573 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc44c15cd50 0x7fc44c1f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:52.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc44bfff640 1 --2- 172.21.15.125:0/1202365573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc44c0cefa0 0x7fc44c0b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:52.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc44b7fe640 1 -- 172.21.15.125:0/1202365573 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc44c15cd50 msgr2=0x7fc44c1f2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:52.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc44b7fe640 1 --2- 172.21.15.125:0/1202365573 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc44c15cd50 0x7fc44c1f2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:52.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc44b7fe640 1 -- 172.21.15.125:0/1202365573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc44c0cefa0 msgr2=0x7fc44c0b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:52.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc44b7fe640 1 --2- 172.21.15.125:0/1202365573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc44c0cefa0 0x7fc44c0b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:52.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc44b7fe640 1 -- 172.21.15.125:0/1202365573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c0094d0 con 0x7fc44c1538f0 2023-10-30T13:02:52.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc44b7fe640 1 --2- 172.21.15.125:0/1202365573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc44c1538f0 0x7fc44c1e3300 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fc44000d620 tx=0x7fc44000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:52.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc448ff9640 1 -- 172.21.15.125:0/1202365573 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44000b620 con 0x7fc44c1538f0 2023-10-30T13:02:52.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc448ff9640 1 -- 172.21.15.125:0/1202365573 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44000b780 con 0x7fc44c1538f0 2023-10-30T13:02:52.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc448ff9640 1 -- 172.21.15.125:0/1202365573 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44000bb10 con 0x7fc44c1538f0 2023-10-30T13:02:52.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc45142f640 1 -- 172.21.15.125:0/1202365573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44c1483b0 con 0x7fc44c1538f0 2023-10-30T13:02:52.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc45142f640 1 -- 172.21.15.125:0/1202365573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44c1e5510 con 0x7fc44c1538f0 2023-10-30T13:02:52.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc45142f640 1 -- 172.21.15.125:0/1202365573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc44c1f2930 con 0x7fc44c1538f0 2023-10-30T13:02:52.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.110+0000 7fc448ff9640 1 -- 172.21.15.125:0/1202365573 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc44000bc70 con 0x7fc44c1538f0 2023-10-30T13:02:52.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.114+0000 7fc448ff9640 1 --2- 172.21.15.125:0/1202365573 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4380664d0 0x7fc438068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:52.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.114+0000 7fc448ff9640 1 -- 172.21.15.125:0/1202365573 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7fc440084a50 con 0x7fc44c1538f0 2023-10-30T13:02:52.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.114+0000 7fc44bfff640 1 --2- 172.21.15.125:0/1202365573 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4380664d0 0x7fc438068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:52.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.114+0000 7fc44bfff640 1 --2- 172.21.15.125:0/1202365573 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4380664d0 0x7fc438068990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc43c0b2000 tx=0x7fc43c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:52.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.118+0000 7fc448ff9640 1 -- 172.21.15.125:0/1202365573 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc44c1f2930 con 0x7fc44c1538f0 2023-10-30T13:02:52.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.298+0000 7fc45142f640 1 -- 172.21.15.125:0/1202365573 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc44c1f28f0 con 0x7fc44c1538f0 2023-10-30T13:02:52.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.302+0000 7fc448ff9640 1 -- 172.21.15.125:0/1202365573 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+5724 (secure 0 0 0) 0x7fc440051800 con 0x7fc44c1538f0 2023-10-30T13:02:52.303 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:52.303 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:50.858127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:52.304 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 153 2023-10-30T13:02:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.302+0000 7fc45142f640 1 -- 172.21.15.125:0/1202365573 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4380664d0 msgr2=0x7fc438068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.302+0000 7fc45142f640 1 --2- 172.21.15.125:0/1202365573 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4380664d0 0x7fc438068990 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fc43c0b2000 tx=0x7fc43c008040 comp rx=0 tx=0).stop 2023-10-30T13:02:52.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.302+0000 7fc45142f640 1 -- 172.21.15.125:0/1202365573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc44c1538f0 msgr2=0x7fc44c1e3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.302+0000 7fc45142f640 1 --2- 172.21.15.125:0/1202365573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc44c1538f0 0x7fc44c1e3300 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7fc44000d620 tx=0x7fc44000dae0 comp rx=0 tx=0).stop 2023-10-30T13:02:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.302+0000 7fc45142f640 1 -- 172.21.15.125:0/1202365573 shutdown_connections 2023-10-30T13:02:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.302+0000 7fc45142f640 1 --2- 172.21.15.125:0/1202365573 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4380664d0 0x7fc438068990 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:52.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.302+0000 7fc45142f640 1 --2- 172.21.15.125:0/1202365573 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc44c15cd50 0x7fc44c1f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:52.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.302+0000 7fc45142f640 1 --2- 172.21.15.125:0/1202365573 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc44c1538f0 0x7fc44c1e3300 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:52.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.302+0000 7fc45142f640 1 --2- 172.21.15.125:0/1202365573 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc44c0cefa0 0x7fc44c0b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:52.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.302+0000 7fc45142f640 1 -- 172.21.15.125:0/1202365573 >> 172.21.15.125:0/1202365573 conn(0x7fc44c06a640 msgr2=0x7fc44c1489b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:52.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.302+0000 7fc45142f640 1 -- 172.21.15.125:0/1202365573 shutdown_connections 2023-10-30T13:02:52.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:52.302+0000 7fc45142f640 1 -- 172.21.15.125:0/1202365573 wait complete. 2023-10-30T13:02:52.324 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:02:54.325 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:54.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4be6a6b640 1 Processor -- start 2023-10-30T13:02:54.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4be6a6b640 1 -- start start 2023-10-30T13:02:54.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4be6a6b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be00c6fe0 0x7f4be0071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:54.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4be6a6b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4be00c5f60 0x7f4be00cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:54.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4be6a6b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be00700b0 con 0x7f4be0153bf0 2023-10-30T13:02:54.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4be6a6b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4be00c6810 con 0x7f4be00c5f60 2023-10-30T13:02:54.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4be6a6b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be0159900 con 0x7f4be00c6fe0 2023-10-30T13:02:54.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4be5008640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4be00c5f60 0x7f4be00cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:54.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4be5008640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4be00c5f60 0x7f4be00cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44390/0 (socket says 172.21.15.125:44390) 2023-10-30T13:02:54.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4be5008640 1 -- 172.21.15.125:0/190783481 learned_addr learned my addr 172.21.15.125:0/190783481 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:54.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4bdffff640 1 --2- 172.21.15.125:0/190783481 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be00c6fe0 0x7f4be0071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:54.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4be5008640 1 -- 172.21.15.125:0/190783481 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be00c6fe0 msgr2=0x7f4be0071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:54.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4be5008640 1 --2- 172.21.15.125:0/190783481 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be00c6fe0 0x7f4be0071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:54.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4be5008640 1 -- 172.21.15.125:0/190783481 >> v1:172.21.15.136:6789/0 conn(0x7f4be0153bf0 legacy=0x7f4be00c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:54.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.470+0000 7f4be5008640 1 -- 172.21.15.125:0/190783481 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be00c6530 con 0x7f4be00c5f60 2023-10-30T13:02:54.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.474+0000 7f4be5008640 1 --2- 172.21.15.125:0/190783481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be00c5f60 0x7f4be00cb570 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8009e50 tx=0x7f4bd801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a4d430603a2c48c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:54.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.474+0000 7f4bdf7fe640 1 -- 172.21.15.125:0/190783481 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd80af400 con 0x7f4be00c5f60 2023-10-30T13:02:54.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.474+0000 7f4bdf7fe640 1 -- 172.21.15.125:0/190783481 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd80ab050 con 0x7f4be00c5f60 2023-10-30T13:02:54.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.474+0000 7f4bdf7fe640 1 -- 172.21.15.125:0/190783481 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd80af780 con 0x7f4be00c5f60 2023-10-30T13:02:54.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.474+0000 7f4be6a6b640 1 -- 172.21.15.125:0/190783481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be00c5f60 msgr2=0x7f4be00cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:54.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.474+0000 7f4be6a6b640 1 --2- 172.21.15.125:0/190783481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be00c5f60 0x7f4be00cb570 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f4bd8009e50 tx=0x7f4bd801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:02:54.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.474+0000 7f4be6a6b640 1 -- 172.21.15.125:0/190783481 shutdown_connections 2023-10-30T13:02:54.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.474+0000 7f4be6a6b640 1 --2- 172.21.15.125:0/190783481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be00c5f60 0x7f4be00cb570 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:54.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.474+0000 7f4be6a6b640 1 --2- 172.21.15.125:0/190783481 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be00c6fe0 0x7f4be0071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:54.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.474+0000 7f4be6a6b640 1 -- 172.21.15.125:0/190783481 >> 172.21.15.125:0/190783481 conn(0x7f4be006c6e0 msgr2=0x7f4be0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:54.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.474+0000 7f4be6a6b640 1 -- 172.21.15.125:0/190783481 shutdown_connections 2023-10-30T13:02:54.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.474+0000 7f4be6a6b640 1 -- 172.21.15.125:0/190783481 wait complete. 2023-10-30T13:02:54.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4be6a6b640 1 Processor -- start 2023-10-30T13:02:54.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4be6a6b640 1 -- start start 2023-10-30T13:02:54.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4be6a6b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4be00c5f60 0x7f4be00c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:54.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4be6a6b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be00c6fe0 0x7f4be00c5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:54.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4be6a6b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be0153bf0 0x7f4be01ef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:54.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4be6a6b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be0154e10 con 0x7f4be00c6fe0 2023-10-30T13:02:54.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4be6a6b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be01587f0 con 0x7f4be00c5f60 2023-10-30T13:02:54.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4be6a6b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4be01ef150 con 0x7f4be0153bf0 2023-10-30T13:02:54.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4bdffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be00c6fe0 0x7f4be00c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:54.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4bdffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be00c6fe0 0x7f4be00c5360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44406/0 (socket says 172.21.15.125:44406) 2023-10-30T13:02:54.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4bdffff640 1 -- 172.21.15.125:0/3881846886 learned_addr learned my addr 172.21.15.125:0/3881846886 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:54.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4be4807640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4be00c5f60 0x7f4be00c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:54.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4be5008640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be0153bf0 0x7f4be01ef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:54.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4bdffff640 1 -- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be0153bf0 msgr2=0x7f4be01ef5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:54.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4bdffff640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be0153bf0 0x7f4be01ef5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:54.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4bdffff640 1 -- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4be00c5f60 msgr2=0x7f4be00c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:54.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4bdffff640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4be00c5f60 0x7f4be00c7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:54.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4bdffff640 1 -- 172.21.15.125:0/3881846886 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd8009b30 con 0x7f4be00c6fe0 2023-10-30T13:02:54.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4be4807640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4be00c5f60 0x7f4be00c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:02:54.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4bdffff640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be00c6fe0 0x7f4be00c5360 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00d800 tx=0x7f4bcc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:54.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4bdd7fa640 1 -- 172.21.15.125:0/3881846886 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc00b6c0 con 0x7f4be00c6fe0 2023-10-30T13:02:54.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4bdd7fa640 1 -- 172.21.15.125:0/3881846886 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bcc00b860 con 0x7f4be00c6fe0 2023-10-30T13:02:54.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4be6a6b640 1 -- 172.21.15.125:0/3881846886 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be01540b0 con 0x7f4be00c6fe0 2023-10-30T13:02:54.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4bdd7fa640 1 -- 172.21.15.125:0/3881846886 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bcc00bc10 con 0x7f4be00c6fe0 2023-10-30T13:02:54.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.478+0000 7f4be6a6b640 1 -- 172.21.15.125:0/3881846886 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be0154300 con 0x7f4be00c6fe0 2023-10-30T13:02:54.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.482+0000 7f4be6a6b640 1 -- 172.21.15.125:0/3881846886 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be004e170 con 0x7f4be00c6fe0 2023-10-30T13:02:54.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.482+0000 7f4bdd7fa640 1 -- 172.21.15.125:0/3881846886 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4bcc01e460 con 0x7f4be00c6fe0 2023-10-30T13:02:54.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.482+0000 7f4bdd7fa640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bc40664a0 0x7f4bc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:54.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.482+0000 7f4bdd7fa640 1 -- 172.21.15.125:0/3881846886 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f4bcc084a10 con 0x7f4be00c6fe0 2023-10-30T13:02:54.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.482+0000 7f4be4807640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bc40664a0 0x7f4bc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:54.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.482+0000 7f4be4807640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bc40664a0 0x7f4bc4068960 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4000f70 tx=0x7f4bd4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:54.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.486+0000 7f4bdd7fa640 1 -- 172.21.15.125:0/3881846886 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4bcc051850 con 0x7f4be00c6fe0 2023-10-30T13:02:54.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.666+0000 7f4be6a6b640 1 -- 172.21.15.125:0/3881846886 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4be006fe90 con 0x7f4be00c6fe0 2023-10-30T13:02:54.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.670+0000 7f4bdd7fa640 1 -- 172.21.15.125:0/3881846886 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f4bcc054e70 con 0x7f4be00c6fe0 2023-10-30T13:02:54.671 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:54.672 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:50.858127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:54.672 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 153 2023-10-30T13:02:54.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.670+0000 7f4be6a6b640 1 -- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bc40664a0 msgr2=0x7f4bc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:54.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.670+0000 7f4be6a6b640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bc40664a0 0x7f4bc4068960 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4000f70 tx=0x7f4bd4009040 comp rx=0 tx=0).stop 2023-10-30T13:02:54.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.670+0000 7f4be6a6b640 1 -- 172.21.15.125:0/3881846886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be00c6fe0 msgr2=0x7f4be00c5360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:54.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.670+0000 7f4be6a6b640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be00c6fe0 0x7f4be00c5360 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc00d800 tx=0x7f4bcc00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:02:54.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.670+0000 7f4be6a6b640 1 -- 172.21.15.125:0/3881846886 shutdown_connections 2023-10-30T13:02:54.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.670+0000 7f4be6a6b640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bc40664a0 0x7f4bc4068960 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:54.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.670+0000 7f4be6a6b640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4be0153bf0 0x7f4be01ef5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:54.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.670+0000 7f4be6a6b640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4be00c6fe0 0x7f4be00c5360 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:54.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.670+0000 7f4be6a6b640 1 --2- 172.21.15.125:0/3881846886 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4be00c5f60 0x7f4be00c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:54.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.670+0000 7f4be6a6b640 1 -- 172.21.15.125:0/3881846886 >> 172.21.15.125:0/3881846886 conn(0x7f4be006c6e0 msgr2=0x7f4be0157e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:54.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.670+0000 7f4be6a6b640 1 -- 172.21.15.125:0/3881846886 shutdown_connections 2023-10-30T13:02:54.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:54.670+0000 7f4be6a6b640 1 -- 172.21.15.125:0/3881846886 wait complete. 2023-10-30T13:02:54.692 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:02:56.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:56.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19aa93e640 1 Processor -- start 2023-10-30T13:02:56.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19aa93e640 1 -- start start 2023-10-30T13:02:56.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19aa93e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19a40c6b80 0x7f19a40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:56.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19aa93e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f19a413f4a0 0x7f19a40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:56.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19aa93e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19a40c0a50 con 0x7f19a40c6f60 2023-10-30T13:02:56.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19aa93e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19a406f6e0 con 0x7f19a413f4a0 2023-10-30T13:02:56.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19aa93e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a406f9c0 con 0x7f19a40c6b80 2023-10-30T13:02:56.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19a8edb640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f19a413f4a0 0x7f19a40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:56.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19a8edb640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f19a413f4a0 0x7f19a40c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44418/0 (socket says 172.21.15.125:44418) 2023-10-30T13:02:56.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19a8edb640 1 -- 172.21.15.125:0/2023088380 learned_addr learned my addr 172.21.15.125:0/2023088380 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:56.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19a3fff640 1 --2- 172.21.15.125:0/2023088380 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19a40c6b80 0x7f19a40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:56.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19a8edb640 1 -- 172.21.15.125:0/2023088380 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19a40c6b80 msgr2=0x7f19a40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:02:56.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19a8edb640 1 --2- 172.21.15.125:0/2023088380 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19a40c6b80 0x7f19a40bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:56.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19a8edb640 1 -- 172.21.15.125:0/2023088380 >> v1:172.21.15.125:6789/0 conn(0x7f19a40c6f60 legacy=0x7f19a413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:56.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.842+0000 7f19a8edb640 1 -- 172.21.15.125:0/2023088380 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a406fca0 con 0x7f19a413f4a0 2023-10-30T13:02:56.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.838+0000 7f19a2ffd640 1 -- 172.21.15.125:0/2023088380 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456375081 0 0) 0x7f19a40c0a50 con 0x7f19a40c6f60 2023-10-30T13:02:56.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.842+0000 7f19a8edb640 1 --2- 172.21.15.125:0/2023088380 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19a413f4a0 0x7f19a40c0400 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f199c009e50 tx=0x7f199c01ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b119bb27dc38546a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:56.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.842+0000 7f19a2ffd640 1 -- 172.21.15.125:0/2023088380 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c0a57f0 con 0x7f19a413f4a0 2023-10-30T13:02:56.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.842+0000 7f19a2ffd640 1 -- 172.21.15.125:0/2023088380 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199c0a5950 con 0x7f19a413f4a0 2023-10-30T13:02:56.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.842+0000 7f19a2ffd640 1 -- 172.21.15.125:0/2023088380 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c0a5ce0 con 0x7f19a413f4a0 2023-10-30T13:02:56.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.842+0000 7f19aa93e640 1 -- 172.21.15.125:0/2023088380 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19a413f4a0 msgr2=0x7f19a40c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:56.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.842+0000 7f19aa93e640 1 --2- 172.21.15.125:0/2023088380 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19a413f4a0 0x7f19a40c0400 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f199c009e50 tx=0x7f199c01ca10 comp rx=0 tx=0).stop 2023-10-30T13:02:56.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.842+0000 7f19aa93e640 1 -- 172.21.15.125:0/2023088380 shutdown_connections 2023-10-30T13:02:56.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.842+0000 7f19aa93e640 1 --2- 172.21.15.125:0/2023088380 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19a413f4a0 0x7f19a40c0400 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:56.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.842+0000 7f19aa93e640 1 --2- 172.21.15.125:0/2023088380 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19a40c6b80 0x7f19a40bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:56.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.842+0000 7f19aa93e640 1 -- 172.21.15.125:0/2023088380 >> 172.21.15.125:0/2023088380 conn(0x7f19a40c7910 msgr2=0x7f19a40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:56.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.842+0000 7f19aa93e640 1 -- 172.21.15.125:0/2023088380 shutdown_connections 2023-10-30T13:02:56.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.842+0000 7f19aa93e640 1 -- 172.21.15.125:0/2023088380 wait complete. 2023-10-30T13:02:56.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19aa93e640 1 Processor -- start 2023-10-30T13:02:56.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19aa93e640 1 -- start start 2023-10-30T13:02:56.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19aa93e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19a40c6b80 0x7f19a413e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:56.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19aa93e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19a40c6f60 0x7f19a41f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:56.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19aa93e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19a413f4a0 0x7f19a40c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:56.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19aa93e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a4155e40 con 0x7f19a40c6b80 2023-10-30T13:02:56.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19aa93e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a413b280 con 0x7f19a40c6f60 2023-10-30T13:02:56.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19aa93e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19a41f1a40 con 0x7f19a413f4a0 2023-10-30T13:02:56.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19a3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19a40c6b80 0x7f19a413e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:56.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19a3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19a40c6b80 0x7f19a413e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44434/0 (socket says 172.21.15.125:44434) 2023-10-30T13:02:56.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19a3fff640 1 -- 172.21.15.125:0/1428712034 learned_addr learned my addr 172.21.15.125:0/1428712034 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:56.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19a37fe640 1 --2- 172.21.15.125:0/1428712034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19a40c6f60 0x7f19a41f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:56.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19a8edb640 1 --2- 172.21.15.125:0/1428712034 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19a413f4a0 0x7f19a40c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:56.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19a3fff640 1 -- 172.21.15.125:0/1428712034 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19a413f4a0 msgr2=0x7f19a40c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:56.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19a3fff640 1 --2- 172.21.15.125:0/1428712034 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19a413f4a0 0x7f19a40c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:56.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19a3fff640 1 -- 172.21.15.125:0/1428712034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19a40c6f60 msgr2=0x7f19a41f2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:56.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19a3fff640 1 --2- 172.21.15.125:0/1428712034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19a40c6f60 0x7f19a41f2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:56.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19a3fff640 1 -- 172.21.15.125:0/1428712034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c009b30 con 0x7f19a40c6b80 2023-10-30T13:02:56.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19a3fff640 1 --2- 172.21.15.125:0/1428712034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19a40c6b80 0x7f19a413e080 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f199400d800 tx=0x7f199400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:56.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19a0ff9640 1 -- 172.21.15.125:0/1428712034 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199400b6c0 con 0x7f19a40c6b80 2023-10-30T13:02:56.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19a0ff9640 1 -- 172.21.15.125:0/1428712034 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199400b820 con 0x7f19a40c6b80 2023-10-30T13:02:56.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19aa93e640 1 -- 172.21.15.125:0/1428712034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a413ad50 con 0x7f19a40c6b80 2023-10-30T13:02:56.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19a0ff9640 1 -- 172.21.15.125:0/1428712034 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199400bb90 con 0x7f19a40c6b80 2023-10-30T13:02:56.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.846+0000 7f19aa93e640 1 -- 172.21.15.125:0/1428712034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a41f2990 con 0x7f19a40c6b80 2023-10-30T13:02:56.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.850+0000 7f19aa93e640 1 -- 172.21.15.125:0/1428712034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a404e170 con 0x7f19a40c6b80 2023-10-30T13:02:56.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.850+0000 7f19a0ff9640 1 -- 172.21.15.125:0/1428712034 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f199400bcf0 con 0x7f19a40c6b80 2023-10-30T13:02:56.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.850+0000 7f19a0ff9640 1 --2- 172.21.15.125:0/1428712034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19900664d0 0x7f1990068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:56.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.850+0000 7f19a0ff9640 1 -- 172.21.15.125:0/1428712034 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f19940849f0 con 0x7f19a40c6b80 2023-10-30T13:02:56.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.850+0000 7f19a37fe640 1 --2- 172.21.15.125:0/1428712034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19900664d0 0x7f1990068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:56.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.854+0000 7f19a37fe640 1 --2- 172.21.15.125:0/1428712034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19900664d0 0x7f1990068990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f19a413f160 tx=0x7f1998009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:56.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:56.858+0000 7f19a0ff9640 1 -- 172.21.15.125:0/1428712034 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1994051830 con 0x7f19a40c6b80 2023-10-30T13:02:57.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:57.090+0000 7f19aa93e640 1 -- 172.21.15.125:0/1428712034 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19a4138d00 con 0x7f19a40c6b80 2023-10-30T13:02:57.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:57.090+0000 7f19a0ff9640 1 -- 172.21.15.125:0/1428712034 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 153 v153) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f1994054e50 con 0x7f19a40c6b80 2023-10-30T13:02:57.094 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:57.095 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":153,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":153,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:50.858127+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:57.095 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 153 2023-10-30T13:02:57.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:57.094+0000 7f19aa93e640 1 -- 172.21.15.125:0/1428712034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19900664d0 msgr2=0x7f1990068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:57.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:57.094+0000 7f19aa93e640 1 --2- 172.21.15.125:0/1428712034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19900664d0 0x7f1990068990 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f19a413f160 tx=0x7f1998009040 comp rx=0 tx=0).stop 2023-10-30T13:02:57.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:57.094+0000 7f19aa93e640 1 -- 172.21.15.125:0/1428712034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19a40c6b80 msgr2=0x7f19a413e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:57.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:57.094+0000 7f19aa93e640 1 --2- 172.21.15.125:0/1428712034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19a40c6b80 0x7f19a413e080 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f199400d800 tx=0x7f199400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:02:57.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:57.094+0000 7f19aa93e640 1 -- 172.21.15.125:0/1428712034 shutdown_connections 2023-10-30T13:02:57.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:57.094+0000 7f19aa93e640 1 --2- 172.21.15.125:0/1428712034 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19900664d0 0x7f1990068990 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:57.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:57.094+0000 7f19aa93e640 1 --2- 172.21.15.125:0/1428712034 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19a413f4a0 0x7f19a40c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:57.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:57.094+0000 7f19aa93e640 1 --2- 172.21.15.125:0/1428712034 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19a40c6f60 0x7f19a41f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:57.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:57.094+0000 7f19aa93e640 1 --2- 172.21.15.125:0/1428712034 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19a40c6b80 0x7f19a413e080 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:57.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:57.094+0000 7f19aa93e640 1 -- 172.21.15.125:0/1428712034 >> 172.21.15.125:0/1428712034 conn(0x7f19a40c7910 msgr2=0x7f19a40bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:57.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:57.094+0000 7f19aa93e640 1 -- 172.21.15.125:0/1428712034 shutdown_connections 2023-10-30T13:02:57.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:57.094+0000 7f19aa93e640 1 -- 172.21.15.125:0/1428712034 wait complete. 2023-10-30T13:02:57.116 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:02:59.118 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:02:59.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.262+0000 7f0ef2a54640 1 Processor -- start 2023-10-30T13:02:59.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.262+0000 7f0ef2a54640 1 -- start start 2023-10-30T13:02:59.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.262+0000 7f0ef2a54640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0eec0cf160 0x7f0eec0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:59.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.262+0000 7f0ef2a54640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eec0cb800 con 0x7f0eec15d6f0 2023-10-30T13:02:59.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.262+0000 7f0ef2a54640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eec0cb330 con 0x7f0eec1538a0 2023-10-30T13:02:59.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.262+0000 7f0ef2a54640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eec0cad70 con 0x7f0eec0cf160 2023-10-30T13:02:59.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.262+0000 7f0ef0ff1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0eec0cf160 0x7f0eec0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:59.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.262+0000 7f0ef0ff1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0eec0cf160 0x7f0eec0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44450/0 (socket says 172.21.15.125:44450) 2023-10-30T13:02:59.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.262+0000 7f0eeb7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0eec15d6f0 0x7f0eec15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55620/0 (socket says 172.21.15.125:55620) 2023-10-30T13:02:59.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.262+0000 7f0ef0ff1640 1 -- 172.21.15.125:0/3271496843 learned_addr learned my addr 172.21.15.125:0/3271496843 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:59.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.262+0000 7f0ef0ff1640 1 -- 172.21.15.125:0/3271496843 >> v1:172.21.15.136:6789/0 conn(0x7f0eec1538a0 legacy=0x7f0eec0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:59.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.262+0000 7f0ef0ff1640 1 -- 172.21.15.125:0/3271496843 >> v1:172.21.15.125:6789/0 conn(0x7f0eec15d6f0 legacy=0x7f0eec15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:59.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.262+0000 7f0ef0ff1640 1 -- 172.21.15.125:0/3271496843 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec0cb050 con 0x7f0eec0cf160 2023-10-30T13:02:59.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.266+0000 7f0ef0ff1640 1 --2- 172.21.15.125:0/3271496843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eec0cf160 0x7f0eec0cf590 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009e50 tx=0x7f0ee401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e48f38a95b8f516 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:59.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.266+0000 7f0eeaffd640 1 -- 172.21.15.125:0/3271496843 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee40af400 con 0x7f0eec0cf160 2023-10-30T13:02:59.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.266+0000 7f0eeaffd640 1 -- 172.21.15.125:0/3271496843 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee40ab050 con 0x7f0eec0cf160 2023-10-30T13:02:59.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.266+0000 7f0eeaffd640 1 -- 172.21.15.125:0/3271496843 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee40af780 con 0x7f0eec0cf160 2023-10-30T13:02:59.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.266+0000 7f0ef2a54640 1 -- 172.21.15.125:0/3271496843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eec0cf160 msgr2=0x7f0eec0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:59.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.266+0000 7f0ef2a54640 1 --2- 172.21.15.125:0/3271496843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eec0cf160 0x7f0eec0cf590 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009e50 tx=0x7f0ee401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:02:59.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.266+0000 7f0ef2a54640 1 -- 172.21.15.125:0/3271496843 shutdown_connections 2023-10-30T13:02:59.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.266+0000 7f0ef2a54640 1 --2- 172.21.15.125:0/3271496843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eec0cf160 0x7f0eec0cf590 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:59.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.266+0000 7f0ef2a54640 1 -- 172.21.15.125:0/3271496843 >> 172.21.15.125:0/3271496843 conn(0x7f0eec06bfe0 msgr2=0x7f0eec06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:59.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.266+0000 7f0ef2a54640 1 -- 172.21.15.125:0/3271496843 shutdown_connections 2023-10-30T13:02:59.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.266+0000 7f0ef2a54640 1 -- 172.21.15.125:0/3271496843 wait complete. 2023-10-30T13:02:59.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0ef2a54640 1 Processor -- start 2023-10-30T13:02:59.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0ef2a54640 1 -- start start 2023-10-30T13:02:59.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0ef2a54640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0eec0cf160 0x7f0eec154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:59.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0ef2a54640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eec1538a0 0x7f0eec0b4870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:59.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0ef2a54640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0eec15d6f0 0x7f0eec1e2a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:59.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0ef2a54640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec1559b0 con 0x7f0eec1538a0 2023-10-30T13:02:59.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0ef2a54640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec159410 con 0x7f0eec0cf160 2023-10-30T13:02:59.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0ef2a54640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec1e3de0 con 0x7f0eec15d6f0 2023-10-30T13:02:59.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0eeb7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eec1538a0 0x7f0eec0b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:59.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0eeb7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eec1538a0 0x7f0eec0b4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44462/0 (socket says 172.21.15.125:44462) 2023-10-30T13:02:59.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0eeb7fe640 1 -- 172.21.15.125:0/1394664763 learned_addr learned my addr 172.21.15.125:0/1394664763 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:02:59.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0eebfff640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0eec0cf160 0x7f0eec154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:59.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0ef0ff1640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0eec15d6f0 0x7f0eec1e2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:59.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0eeb7fe640 1 -- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0eec15d6f0 msgr2=0x7f0eec1e2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:59.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0eeb7fe640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0eec15d6f0 0x7f0eec1e2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:59.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0eeb7fe640 1 -- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0eec0cf160 msgr2=0x7f0eec154dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:59.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0eeb7fe640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0eec0cf160 0x7f0eec154dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:59.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0eeb7fe640 1 -- 172.21.15.125:0/1394664763 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee4009b30 con 0x7f0eec1538a0 2023-10-30T13:02:59.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0eebfff640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0eec0cf160 0x7f0eec154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:02:59.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0eeb7fe640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eec1538a0 0x7f0eec0b4870 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000c460 tx=0x7f0ee000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:59.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0ee8ff9640 1 -- 172.21.15.125:0/1394664763 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0007480 con 0x7f0eec1538a0 2023-10-30T13:02:59.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0ee8ff9640 1 -- 172.21.15.125:0/1394664763 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee00075e0 con 0x7f0eec1538a0 2023-10-30T13:02:59.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0ee8ff9640 1 -- 172.21.15.125:0/1394664763 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0004a10 con 0x7f0eec1538a0 2023-10-30T13:02:59.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0ef2a54640 1 -- 172.21.15.125:0/1394664763 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eec156ce0 con 0x7f0eec1538a0 2023-10-30T13:02:59.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.270+0000 7f0ef2a54640 1 -- 172.21.15.125:0/1394664763 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eec155300 con 0x7f0eec1538a0 2023-10-30T13:02:59.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.274+0000 7f0ef2a54640 1 -- 172.21.15.125:0/1394664763 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eec04e170 con 0x7f0eec1538a0 2023-10-30T13:02:59.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.274+0000 7f0ee8ff9640 1 -- 172.21.15.125:0/1394664763 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0ee0004b70 con 0x7f0eec1538a0 2023-10-30T13:02:59.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.274+0000 7f0ee8ff9640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ed80664a0 0x7f0ed8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:02:59.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.274+0000 7f0ee8ff9640 1 -- 172.21.15.125:0/1394664763 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f0ee00836e0 con 0x7f0eec1538a0 2023-10-30T13:02:59.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.274+0000 7f0eebfff640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ed80664a0 0x7f0ed8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:02:59.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.274+0000 7f0eebfff640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ed80664a0 0x7f0ed8068960 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f0edc000f70 tx=0x7f0edc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:02:59.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.278+0000 7f0ee8ff9640 1 -- 172.21.15.125:0/1394664763 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0ee0050490 con 0x7f0eec1538a0 2023-10-30T13:02:59.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.458+0000 7f0ef2a54640 1 -- 172.21.15.125:0/1394664763 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0eec0b1dd0 con 0x7f0eec1538a0 2023-10-30T13:02:59.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.458+0000 7f0ee8ff9640 1 -- 172.21.15.125:0/1394664763 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f0ee0053ab0 con 0x7f0eec1538a0 2023-10-30T13:02:59.463 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:02:59.463 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:57.950518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:active","state_seq":30,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:02:59.463 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 154 2023-10-30T13:02:59.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.462+0000 7f0ef2a54640 1 -- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ed80664a0 msgr2=0x7f0ed8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:59.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.462+0000 7f0ef2a54640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ed80664a0 0x7f0ed8068960 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f0edc000f70 tx=0x7f0edc009040 comp rx=0 tx=0).stop 2023-10-30T13:02:59.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.462+0000 7f0ef2a54640 1 -- 172.21.15.125:0/1394664763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eec1538a0 msgr2=0x7f0eec0b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:02:59.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.462+0000 7f0ef2a54640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eec1538a0 0x7f0eec0b4870 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000c460 tx=0x7f0ee000c920 comp rx=0 tx=0).stop 2023-10-30T13:02:59.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.462+0000 7f0ef2a54640 1 -- 172.21.15.125:0/1394664763 shutdown_connections 2023-10-30T13:02:59.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.462+0000 7f0ef2a54640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ed80664a0 0x7f0ed8068960 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:59.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.462+0000 7f0ef2a54640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0eec15d6f0 0x7f0eec1e2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:59.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.462+0000 7f0ef2a54640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0eec1538a0 0x7f0eec0b4870 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:59.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.462+0000 7f0ef2a54640 1 --2- 172.21.15.125:0/1394664763 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0eec0cf160 0x7f0eec154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:02:59.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.462+0000 7f0ef2a54640 1 -- 172.21.15.125:0/1394664763 >> 172.21.15.125:0/1394664763 conn(0x7f0eec06bfe0 msgr2=0x7f0eec0735c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:02:59.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.462+0000 7f0ef2a54640 1 -- 172.21.15.125:0/1394664763 shutdown_connections 2023-10-30T13:02:59.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:02:59.462+0000 7f0ef2a54640 1 -- 172.21.15.125:0/1394664763 wait complete. 2023-10-30T13:02:59.484 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=7707 2023-10-30T13:02:59.484 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.c 2023-10-30T13:03:05.485 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:03:05.485 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:03:05.485 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:03:05.490 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:03:05.490 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:03:05.531 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T13:03:05.530+0000 7ff692164640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:03:05.574 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 4% (7/152) Updating files: 5% (8/152) Updating files: 6% (10/152) Updating files: 7% (11/152) Updating files: 8% (13/152) Updating files: 9% (14/152) Updating files: 10% (16/152) Updating files: 11% (17/152) Updating files: 12% (19/152) Updating files: 13% (20/152) Updating files: 14% (22/152) Updating files: 15% (23/152) Updating files: 16% (25/152) Updating files: 17% (26/152) Updating files: 18% (28/152) Updating files: 19% (29/152) Updating files: 20% (31/152) Updating files: 21% (32/152) Updating files: 22% (34/152) Updating files: 23% (35/152) Updating files: 24% (37/152) Updating files: 25% (38/152) Updating files: 26% (40/152) Updating files: 27% (42/152) Updating files: 28% (43/152) Updating files: 29% (45/152) Updating files: 30% (46/152) Updating files: 31% (48/152) Updating files: 32% (49/152) Updating files: 33% (51/152) Updating files: 34% (52/152) Updating files: 35% (54/152) Updating files: 36% (55/152) Updating files: 37% (57/152) Updating files: 38% (58/152) Updating files: 39% (60/152) Updating files: 40% (61/152) Updating files: 41% (63/152) Updating files: 42% (64/152) Updating files: 43% (66/152) Updating files: 44% (67/152) Updating files: 45% (69/152) Updating files: 46% (70/152) Updating files: 47% (72/152) Updating files: 48% (73/152) Updating files: 49% (75/152) Updating files: 50% (76/152) Updating files: 51% (78/152) Updating files: 52% (80/152) Updating files: 53% (81/152) Updating files: 54% (83/152) Updating files: 55% (84/152) Updating files: 56% (86/152) Updating files: 57% (87/152) Updating files: 58% (89/152) Updating files: 59% (90/152) Updating files: 60% (92/152) Updating files: 61% (93/152) Updating files: 62% (95/152) Updating files: 63% (96/152) Updating files: 64% (98/152) Updating files: 65% (99/152) Updating files: 66% (101/152) Updating files: 67% (102/152) Updating files: 68% (104/152) Updating files: 69% (105/152) Updating files: 70% (107/152) Updating files: 71% (108/152) Updating files: 72% (110/152) Updating files: 73% (111/152) Updating files: 74% (113/152) Updating files: 75% (114/152) Updating files: 76% (116/152) Updating files: 77% (118/152) Updating files: 78% (119/152) Updating files: 79% (121/152) Updating files: 80% (122/152) Updating files: 81% (124/152) Updating files: 82% (125/152) Updating files: 83% (127/152) Updating files: 84% (128/152) Updating files: 85% (130/152) Updating files: 86% (131/152) Updating files: 87% (133/152) Updating files: 88% (134/152) Updating files: 89% (136/152) Updating files: 90% (137/152) Updating files: 91% (139/152) Updating files: 92% (140/152) Updating files: 93% (142/152) Updating files: 94% (143/152) Updating files: 95% (145/152) Updating files: 96% (146/152) Updating files: 97% (148/152) Updating files: 98% (149/152) Updating files: 99% (151/152) Updating files: 100% (152/152) Updating files: 100% (152/152), done. 2023-10-30T13:03:05.584 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2023-10-30T13:03:05.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.642+0000 7f0c420ad640 1 Processor -- start 2023-10-30T13:03:05.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.642+0000 7f0c420ad640 1 -- start start 2023-10-30T13:03:05.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c420ad640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c3c1538f0 0x7f0c3c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:03:05.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c420ad640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0c3c15cd50 0x7f0c3c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:03:05.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c420ad640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c3c15d300 con 0x7f0c3c0cefa0 2023-10-30T13:03:05.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c420ad640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c3c15d5e0 con 0x7f0c3c15cd50 2023-10-30T13:03:05.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c420ad640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c15d8c0 con 0x7f0c3c1538f0 2023-10-30T13:03:05.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c418ac640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0c3c0cefa0 0x7f0c3c0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55904/0 (socket says 172.21.15.125:55904) 2023-10-30T13:03:05.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c418ac640 1 -- 172.21.15.125:0/1133937998 learned_addr learned my addr 172.21.15.125:0/1133937998 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:03:05.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c410ab640 1 --2- 172.21.15.125:0/1133937998 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c3c1538f0 0x7f0c3c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:03:05.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c408aa640 1 --2- 172.21.15.125:0/1133937998 >> v2:172.21.15.136:3300/0 conn(0x7f0c3c15cd50 0x7f0c3c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:03:05.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c33fff640 1 -- 172.21.15.125:0/1133937998 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550505358 0 0) 0x7f0c3c15d300 con 0x7f0c3c0cefa0 2023-10-30T13:03:05.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c33fff640 1 -- 172.21.15.125:0/1133937998 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c28003530 con 0x7f0c3c0cefa0 2023-10-30T13:03:05.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c408aa640 1 -- 172.21.15.125:0/1133937998 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c3c1538f0 msgr2=0x7f0c3c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:03:05.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c408aa640 1 --2- 172.21.15.125:0/1133937998 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c3c1538f0 0x7f0c3c159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:05.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c408aa640 1 -- 172.21.15.125:0/1133937998 >> v1:172.21.15.125:6789/0 conn(0x7f0c3c0cefa0 legacy=0x7f0c3c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:03:05.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c408aa640 1 -- 172.21.15.125:0/1133937998 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3c15dba0 con 0x7f0c3c15cd50 2023-10-30T13:03:05.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.646+0000 7f0c410ab640 1 --2- 172.21.15.125:0/1133937998 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c3c1538f0 0x7f0c3c159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:03:05.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.650+0000 7f0c408aa640 1 --2- 172.21.15.125:0/1133937998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c3c15cd50 0x7f0c3c0c9f90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0097c0 tx=0x7f0c2c01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87cf2c11c5720868 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:03:05.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.650+0000 7f0c33fff640 1 -- 172.21.15.125:0/1133937998 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c0a8970 con 0x7f0c3c15cd50 2023-10-30T13:03:05.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.650+0000 7f0c33fff640 1 -- 172.21.15.125:0/1133937998 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c2c0a8ad0 con 0x7f0c3c15cd50 2023-10-30T13:03:05.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.650+0000 7f0c33fff640 1 -- 172.21.15.125:0/1133937998 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c2c0a8e60 con 0x7f0c3c15cd50 2023-10-30T13:03:05.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.650+0000 7f0c420ad640 1 -- 172.21.15.125:0/1133937998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c3c15cd50 msgr2=0x7f0c3c0c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:03:05.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.650+0000 7f0c420ad640 1 --2- 172.21.15.125:0/1133937998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c3c15cd50 0x7f0c3c0c9f90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f0c2c0097c0 tx=0x7f0c2c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:03:05.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.650+0000 7f0c420ad640 1 -- 172.21.15.125:0/1133937998 shutdown_connections 2023-10-30T13:03:05.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.650+0000 7f0c420ad640 1 --2- 172.21.15.125:0/1133937998 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c3c15cd50 0x7f0c3c0c9f90 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:05.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.650+0000 7f0c420ad640 1 --2- 172.21.15.125:0/1133937998 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c3c1538f0 0x7f0c3c159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:05.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.650+0000 7f0c420ad640 1 -- 172.21.15.125:0/1133937998 >> 172.21.15.125:0/1133937998 conn(0x7f0c3c06a640 msgr2=0x7f0c3c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:03:05.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.650+0000 7f0c420ad640 1 -- 172.21.15.125:0/1133937998 shutdown_connections 2023-10-30T13:03:05.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.650+0000 7f0c420ad640 1 -- 172.21.15.125:0/1133937998 wait complete. 2023-10-30T13:03:05.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.650+0000 7f0c420ad640 1 Processor -- start 2023-10-30T13:03:05.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.650+0000 7f0c420ad640 1 -- start start 2023-10-30T13:03:05.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c420ad640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c3c0cefa0 0x7f0c3c157b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:03:05.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c420ad640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c3c1538f0 0x7f0c3c15b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:03:05.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c420ad640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c3c15cd50 0x7f0c3c15b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:03:05.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c420ad640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c0b30a0 con 0x7f0c3c15cd50 2023-10-30T13:03:05.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c420ad640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c15adf0 con 0x7f0c3c0cefa0 2023-10-30T13:03:05.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c420ad640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c1f3170 con 0x7f0c3c1538f0 2023-10-30T13:03:05.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c418ac640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c3c15cd50 0x7f0c3c15b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:03:05.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c410ab640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c3c0cefa0 0x7f0c3c157b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:03:05.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c418ac640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c3c15cd50 0x7f0c3c15b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34256/0 (socket says 172.21.15.125:34256) 2023-10-30T13:03:05.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c418ac640 1 -- 172.21.15.125:0/1754000866 learned_addr learned my addr 172.21.15.125:0/1754000866 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:03:05.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c408aa640 1 --2- 172.21.15.125:0/1754000866 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c3c1538f0 0x7f0c3c15b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:03:05.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c418ac640 1 -- 172.21.15.125:0/1754000866 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c3c1538f0 msgr2=0x7f0c3c15b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:03:05.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c418ac640 1 --2- 172.21.15.125:0/1754000866 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c3c1538f0 0x7f0c3c15b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:05.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c418ac640 1 -- 172.21.15.125:0/1754000866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c3c0cefa0 msgr2=0x7f0c3c157b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:03:05.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c418ac640 1 --2- 172.21.15.125:0/1754000866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c3c0cefa0 0x7f0c3c157b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:05.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c418ac640 1 -- 172.21.15.125:0/1754000866 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c38009420 con 0x7f0c3c15cd50 2023-10-30T13:03:05.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c418ac640 1 --2- 172.21.15.125:0/1754000866 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c3c15cd50 0x7f0c3c15b7c0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400cb70 tx=0x7f0c3400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:03:05.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c31ffb640 1 -- 172.21.15.125:0/1754000866 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3400ad40 con 0x7f0c3c15cd50 2023-10-30T13:03:05.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c31ffb640 1 -- 172.21.15.125:0/1754000866 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c34012400 con 0x7f0c3c15cd50 2023-10-30T13:03:05.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c420ad640 1 -- 172.21.15.125:0/1754000866 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c2c009420 con 0x7f0c3c15cd50 2023-10-30T13:03:05.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c31ffb640 1 -- 172.21.15.125:0/1754000866 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3400aea0 con 0x7f0c3c15cd50 2023-10-30T13:03:05.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.654+0000 7f0c420ad640 1 -- 172.21.15.125:0/1754000866 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3c1f2e90 con 0x7f0c3c15cd50 2023-10-30T13:03:05.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.658+0000 7f0c420ad640 1 -- 172.21.15.125:0/1754000866 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c3c15afa0 con 0x7f0c3c15cd50 2023-10-30T13:03:05.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.658+0000 7f0c31ffb640 1 -- 172.21.15.125:0/1754000866 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0c34007070 con 0x7f0c3c15cd50 2023-10-30T13:03:05.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.658+0000 7f0c31ffb640 1 --2- 172.21.15.125:0/1754000866 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c200664a0 0x7f0c20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:03:05.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.658+0000 7f0c31ffb640 1 -- 172.21.15.125:0/1754000866 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f0c340844e0 con 0x7f0c3c15cd50 2023-10-30T13:03:05.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.658+0000 7f0c410ab640 1 --2- 172.21.15.125:0/1754000866 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c200664a0 0x7f0c20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:03:05.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.662+0000 7f0c410ab640 1 --2- 172.21.15.125:0/1754000866 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c200664a0 0x7f0c20068960 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0c38009cf0 tx=0x7f0c38009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:03:05.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.662+0000 7f0c31ffb640 1 -- 172.21.15.125:0/1754000866 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0c3c15afa0 con 0x7f0c3c15cd50 2023-10-30T13:03:05.711 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at b6b48dbefad v0.66 2023-10-30T13:03:05.715 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.66 2023-10-30T13:03:05.739 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 67 -eq 48 ] 2023-10-30T13:03:05.739 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.67 2023-10-30T13:03:05.739 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.67 2023-10-30T13:03:05.740 INFO:tasks.workunit.client.0.smithi125.stdout:v0.67 2023-10-30T13:03:05.740 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.67 2023-10-30T13:03:05.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.882+0000 7f0c420ad640 1 -- 172.21.15.125:0/1754000866 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c3c15afa0 con 0x7f0c3c15cd50 2023-10-30T13:03:05.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.882+0000 7f0c31ffb640 1 -- 172.21.15.125:0/1754000866 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f0c3c15afa0 con 0x7f0c3c15cd50 2023-10-30T13:03:05.885 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:03:05.886 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:57.950518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:active","state_seq":30,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:03:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 154 2023-10-30T13:03:05.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.886+0000 7f0c420ad640 1 -- 172.21.15.125:0/1754000866 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c200664a0 msgr2=0x7f0c20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:03:05.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.886+0000 7f0c420ad640 1 --2- 172.21.15.125:0/1754000866 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c200664a0 0x7f0c20068960 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f0c38009cf0 tx=0x7f0c38009190 comp rx=0 tx=0).stop 2023-10-30T13:03:05.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.886+0000 7f0c420ad640 1 -- 172.21.15.125:0/1754000866 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c3c15cd50 msgr2=0x7f0c3c15b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:03:05.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.886+0000 7f0c420ad640 1 --2- 172.21.15.125:0/1754000866 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c3c15cd50 0x7f0c3c15b7c0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400cb70 tx=0x7f0c3400a430 comp rx=0 tx=0).stop 2023-10-30T13:03:05.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.886+0000 7f0c420ad640 1 -- 172.21.15.125:0/1754000866 shutdown_connections 2023-10-30T13:03:05.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.886+0000 7f0c420ad640 1 --2- 172.21.15.125:0/1754000866 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c200664a0 0x7f0c20068960 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:05.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.886+0000 7f0c420ad640 1 --2- 172.21.15.125:0/1754000866 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c3c15cd50 0x7f0c3c15b7c0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:05.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.886+0000 7f0c420ad640 1 --2- 172.21.15.125:0/1754000866 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c3c1538f0 0x7f0c3c15b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:05.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.886+0000 7f0c420ad640 1 --2- 172.21.15.125:0/1754000866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c3c0cefa0 0x7f0c3c157b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:05.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.886+0000 7f0c420ad640 1 -- 172.21.15.125:0/1754000866 >> 172.21.15.125:0/1754000866 conn(0x7f0c3c06a640 msgr2=0x7f0c3c0c4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:03:05.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.886+0000 7f0c420ad640 1 -- 172.21.15.125:0/1754000866 shutdown_connections 2023-10-30T13:03:05.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:05.886+0000 7f0c420ad640 1 -- 172.21.15.125:0/1754000866 wait complete. 2023-10-30T13:03:05.905 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:03:07.906 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:03:08.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bd271d640 1 Processor -- start 2023-10-30T13:03:08.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bd271d640 1 -- start start 2023-10-30T13:03:08.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bd271d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bcc0c6bd0 0x7f1bcc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:03:08.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bd271d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1bcc071110 0x7f1bcc158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:03:08.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bd271d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bcc066cf0 con 0x7f1bcc06f7a0 2023-10-30T13:03:08.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bd271d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bcc0700c0 con 0x7f1bcc071110 2023-10-30T13:03:08.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bd271d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc0c70d0 con 0x7f1bcc0c6bd0 2023-10-30T13:03:08.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bcb7fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bcc0c6bd0 0x7f1bcc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:03:08.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bcb7fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bcc0c6bd0 0x7f1bcc06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:50542/0 (socket says 172.21.15.125:50542) 2023-10-30T13:03:08.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bd0cba640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1bcc071110 0x7f1bcc158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:03:08.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bcb7fe640 1 -- 172.21.15.125:0/1475020679 learned_addr learned my addr 172.21.15.125:0/1475020679 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:03:08.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bcbfff640 1 --1- 172.21.15.125:0/1475020679 >> v1:172.21.15.136:6789/0 conn(0x7f1bcc06f7a0 0x7f1bcc069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:54000/0 (socket says 172.21.15.125:54000) 2023-10-30T13:03:08.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bd0cba640 1 -- 172.21.15.125:0/1475020679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bcc0c6bd0 msgr2=0x7f1bcc06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:03:08.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bd0cba640 1 --2- 172.21.15.125:0/1475020679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bcc0c6bd0 0x7f1bcc06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:08.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bd0cba640 1 -- 172.21.15.125:0/1475020679 >> v1:172.21.15.136:6789/0 conn(0x7f1bcc06f7a0 legacy=0x7f1bcc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:03:08.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bd0cba640 1 -- 172.21.15.125:0/1475020679 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bcc0c7390 con 0x7f1bcc071110 2023-10-30T13:03:08.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.073+0000 7f1bcb7fe640 1 --2- 172.21.15.125:0/1475020679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bcc0c6bd0 0x7f1bcc06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:03:08.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.077+0000 7f1bd0cba640 1 --2- 172.21.15.125:0/1475020679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bcc071110 0x7f1bcc158110 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009d90 tx=0x7f1bc401f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=380eb1efb951935 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:03:08.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.077+0000 7f1bcaffd640 1 -- 172.21.15.125:0/1475020679 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc40af600 con 0x7f1bcc071110 2023-10-30T13:03:08.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.077+0000 7f1bcaffd640 1 -- 172.21.15.125:0/1475020679 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bc40ab050 con 0x7f1bcc071110 2023-10-30T13:03:08.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.077+0000 7f1bcaffd640 1 -- 172.21.15.125:0/1475020679 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bc40af980 con 0x7f1bcc071110 2023-10-30T13:03:08.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.077+0000 7f1bd271d640 1 -- 172.21.15.125:0/1475020679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bcc071110 msgr2=0x7f1bcc158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:03:08.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.077+0000 7f1bd271d640 1 --2- 172.21.15.125:0/1475020679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bcc071110 0x7f1bcc158110 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009d90 tx=0x7f1bc401f130 comp rx=0 tx=0).stop 2023-10-30T13:03:08.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd271d640 1 -- 172.21.15.125:0/1475020679 shutdown_connections 2023-10-30T13:03:08.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd271d640 1 --2- 172.21.15.125:0/1475020679 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bcc071110 0x7f1bcc158110 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:08.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd271d640 1 --2- 172.21.15.125:0/1475020679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bcc0c6bd0 0x7f1bcc06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:08.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd271d640 1 -- 172.21.15.125:0/1475020679 >> 172.21.15.125:0/1475020679 conn(0x7f1bcc0c7910 msgr2=0x7f1bcc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:03:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd271d640 1 -- 172.21.15.125:0/1475020679 shutdown_connections 2023-10-30T13:03:08.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd271d640 1 -- 172.21.15.125:0/1475020679 wait complete. 2023-10-30T13:03:08.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd271d640 1 Processor -- start 2023-10-30T13:03:08.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd271d640 1 -- start start 2023-10-30T13:03:08.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd271d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bcc06f7a0 0x7f1bcc15a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:03:08.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd271d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bcc071110 0x7f1bcc1f2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:03:08.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd271d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bcc0c6bd0 0x7f1bcc1588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:03:08.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd271d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc156f10 con 0x7f1bcc06f7a0 2023-10-30T13:03:08.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd271d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc15b850 con 0x7f1bcc0c6bd0 2023-10-30T13:03:08.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd271d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bcc06aad0 con 0x7f1bcc071110 2023-10-30T13:03:08.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bcbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bcc06f7a0 0x7f1bcc15a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:03:08.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bcbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bcc06f7a0 0x7f1bcc15a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34264/0 (socket says 172.21.15.125:34264) 2023-10-30T13:03:08.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bcbfff640 1 -- 172.21.15.125:0/1764495351 learned_addr learned my addr 172.21.15.125:0/1764495351 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:03:08.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bcb7fe640 1 --2- 172.21.15.125:0/1764495351 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bcc071110 0x7f1bcc1f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:03:08.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bd0cba640 1 --2- 172.21.15.125:0/1764495351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bcc0c6bd0 0x7f1bcc1588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:03:08.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bcbfff640 1 -- 172.21.15.125:0/1764495351 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bcc071110 msgr2=0x7f1bcc1f2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:03:08.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bcbfff640 1 --2- 172.21.15.125:0/1764495351 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bcc071110 0x7f1bcc1f2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:08.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bcbfff640 1 -- 172.21.15.125:0/1764495351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bcc0c6bd0 msgr2=0x7f1bcc1588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:03:08.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bcbfff640 1 --2- 172.21.15.125:0/1764495351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bcc0c6bd0 0x7f1bcc1588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:08.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.081+0000 7f1bcbfff640 1 -- 172.21.15.125:0/1764495351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc00094d0 con 0x7f1bcc06f7a0 2023-10-30T13:03:08.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.085+0000 7f1bcbfff640 1 --2- 172.21.15.125:0/1764495351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bcc06f7a0 0x7f1bcc15a860 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00c370 tx=0x7f1bbc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:03:08.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.085+0000 7f1bc8ff9640 1 -- 172.21.15.125:0/1764495351 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc0074e0 con 0x7f1bcc06f7a0 2023-10-30T13:03:08.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.085+0000 7f1bc8ff9640 1 -- 172.21.15.125:0/1764495351 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc007640 con 0x7f1bcc06f7a0 2023-10-30T13:03:08.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.085+0000 7f1bc8ff9640 1 -- 172.21.15.125:0/1764495351 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc004a30 con 0x7f1bcc06f7a0 2023-10-30T13:03:08.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.085+0000 7f1bd271d640 1 -- 172.21.15.125:0/1764495351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc4009a70 con 0x7f1bcc06f7a0 2023-10-30T13:03:08.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.085+0000 7f1bd271d640 1 -- 172.21.15.125:0/1764495351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bcc1f2fe0 con 0x7f1bcc06f7a0 2023-10-30T13:03:08.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.085+0000 7f1bd271d640 1 -- 172.21.15.125:0/1764495351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bcc1f3200 con 0x7f1bcc06f7a0 2023-10-30T13:03:08.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.085+0000 7f1bc8ff9640 1 -- 172.21.15.125:0/1764495351 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1bbc004b90 con 0x7f1bcc06f7a0 2023-10-30T13:03:08.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.085+0000 7f1bc8ff9640 1 --2- 172.21.15.125:0/1764495351 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bb80664a0 0x7f1bb8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:03:08.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.085+0000 7f1bc8ff9640 1 -- 172.21.15.125:0/1764495351 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f1bbc083da0 con 0x7f1bcc06f7a0 2023-10-30T13:03:08.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.085+0000 7f1bcb7fe640 1 --2- 172.21.15.125:0/1764495351 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bb80664a0 0x7f1bb8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:03:08.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.089+0000 7f1bcb7fe640 1 --2- 172.21.15.125:0/1764495351 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bb80664a0 0x7f1bb8068960 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc14cf80 tx=0x7f1bc0009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:03:08.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.089+0000 7f1bc8ff9640 1 -- 172.21.15.125:0/1764495351 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1bbc050c00 con 0x7f1bcc06f7a0 2023-10-30T13:03:08.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.273+0000 7f1bd271d640 1 -- 172.21.15.125:0/1764495351 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bcc04e170 con 0x7f1bcc06f7a0 2023-10-30T13:03:08.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.273+0000 7f1bc8ff9640 1 -- 172.21.15.125:0/1764495351 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f1bbc054220 con 0x7f1bcc06f7a0 2023-10-30T13:03:08.277 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:03:08.277 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:57.950518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:active","state_seq":30,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:03:08.278 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 155 2023-10-30T13:03:08.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.273+0000 7f1bd271d640 1 -- 172.21.15.125:0/1764495351 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bb80664a0 msgr2=0x7f1bb8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:03:08.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.273+0000 7f1bd271d640 1 --2- 172.21.15.125:0/1764495351 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bb80664a0 0x7f1bb8068960 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc14cf80 tx=0x7f1bc0009240 comp rx=0 tx=0).stop 2023-10-30T13:03:08.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.273+0000 7f1bd271d640 1 -- 172.21.15.125:0/1764495351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bcc06f7a0 msgr2=0x7f1bcc15a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:03:08.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.273+0000 7f1bd271d640 1 --2- 172.21.15.125:0/1764495351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bcc06f7a0 0x7f1bcc15a860 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00c370 tx=0x7f1bbc00c830 comp rx=0 tx=0).stop 2023-10-30T13:03:08.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.273+0000 7f1bd271d640 1 -- 172.21.15.125:0/1764495351 shutdown_connections 2023-10-30T13:03:08.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.273+0000 7f1bd271d640 1 --2- 172.21.15.125:0/1764495351 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bb80664a0 0x7f1bb8068960 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:08.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.273+0000 7f1bd271d640 1 --2- 172.21.15.125:0/1764495351 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bcc0c6bd0 0x7f1bcc1588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:08.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.273+0000 7f1bd271d640 1 --2- 172.21.15.125:0/1764495351 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bcc071110 0x7f1bcc1f2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:08.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.273+0000 7f1bd271d640 1 --2- 172.21.15.125:0/1764495351 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bcc06f7a0 0x7f1bcc15a860 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:03:08.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.273+0000 7f1bd271d640 1 -- 172.21.15.125:0/1764495351 >> 172.21.15.125:0/1764495351 conn(0x7f1bcc0c7910 msgr2=0x7f1bcc0c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:03:08.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.277+0000 7f1bd271d640 1 -- 172.21.15.125:0/1764495351 shutdown_connections 2023-10-30T13:03:08.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:03:08.277+0000 7f1bd271d640 1 -- 172.21.15.125:0/1764495351 wait complete. 2023-10-30T13:03:08.297 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:03:08.298 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 85 secs before thrashing 2023-10-30T13:03:17.952 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 4% (17/355) Updating files: 5% (18/355) Updating files: 6% (22/355) Updating files: 7% (25/355) Updating files: 8% (29/355) Updating files: 9% (32/355) Updating files: 10% (36/355) Updating files: 11% (40/355) Updating files: 12% (43/355) Updating files: 13% (47/355) Updating files: 14% (50/355) Updating files: 15% (54/355) Updating files: 16% (57/355) Updating files: 17% (61/355) Updating files: 18% (64/355) Updating files: 19% (68/355) Updating files: 20% (71/355) Updating files: 21% (75/355) Updating files: 22% (79/355) Updating files: 23% (82/355) Updating files: 24% (86/355) Updating files: 25% (89/355) Updating files: 25% (90/355) Updating files: 26% (93/355) Updating files: 27% (96/355) Updating files: 28% (100/355) Updating files: 29% (103/355) Updating files: 30% (107/355) Updating files: 31% (111/355) Updating files: 32% (114/355) Updating files: 33% (118/355) Updating files: 34% (121/355) Updating files: 35% (125/355) Updating files: 36% (128/355) Updating files: 37% (132/355) Updating files: 38% (135/355) Updating files: 39% (139/355) Updating files: 40% (142/355) Updating files: 41% (146/355) Updating files: 42% (150/355) Updating files: 43% (153/355) Updating files: 44% (157/355) Updating files: 45% (160/355) Updating files: 46% (164/355) Updating files: 47% (167/355) Updating files: 48% (171/355) Updating files: 49% (174/355) Updating files: 50% (178/355) Updating files: 51% (182/355) Updating files: 52% (185/355) Updating files: 53% (189/355) Updating files: 54% (192/355) Updating files: 55% (196/355) Updating files: 56% (199/355) Updating files: 57% (203/355) Updating files: 58% (206/355) Updating files: 59% (210/355) Updating files: 60% (213/355) Updating files: 61% (217/355) Updating files: 62% (221/355) Updating files: 63% (224/355) Updating files: 63% (227/355) Updating files: 64% (228/355) Updating files: 65% (231/355) Updating files: 66% (235/355) Updating files: 67% (238/355) Updating files: 68% (242/355) Updating files: 69% (245/355) Updating files: 70% (249/355) Updating files: 71% (253/355) Updating files: 72% (256/355) Updating files: 73% (260/355) Updating files: 74% (263/355) Updating files: 75% (267/355) Updating files: 75% (268/355) Updating files: 76% (270/355) Updating files: 77% (274/355) Updating files: 78% (277/355) Updating files: 79% (281/355) Updating files: 80% (284/355) Updating files: 81% (288/355) Updating files: 82% (292/355) Updating files: 83% (295/355) Updating files: 84% (299/355) Updating files: 85% (302/355) Updating files: 86% (306/355) Updating files: 87% (309/355) Updating files: 87% (311/355) Updating files: 88% (313/355) Updating files: 89% (316/355) Updating files: 90% (320/355) Updating files: 91% (324/355) Updating files: 92% (327/355) Updating files: 93% (331/355) Updating files: 94% (334/355) Updating files: 95% (338/355) Updating files: 96% (341/355) Updating files: 97% (345/355) Updating files: 98% (348/355) Updating files: 99% (352/355) Updating files: 100% (355/355) Updating files: 100% (355/355), done. 2023-10-30T13:03:18.041 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at e3b7bc5bce8 v0.67 2023-10-30T13:03:18.048 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.67 2023-10-30T13:03:18.090 INFO:tasks.workunit.client.0.smithi125.stdout:v0.68 2023-10-30T13:03:18.090 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 68 -eq 48 ] 2023-10-30T13:03:18.090 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.68 2023-10-30T13:03:18.091 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.68 2023-10-30T13:03:18.091 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.68 2023-10-30T13:03:22.431 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:03:22.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:03:25.801 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 31% (76/241) Updating files: 32% (78/241) Updating files: 33% (80/241) Updating files: 34% (82/241) Updating files: 35% (85/241) Updating files: 36% (87/241) Updating files: 37% (90/241) Updating files: 38% (92/241) Updating files: 39% (94/241) Updating files: 40% (97/241) Updating files: 41% (99/241) Updating files: 42% (102/241) Updating files: 43% (104/241) Updating files: 44% (107/241) Updating files: 45% (109/241) Updating files: 46% (111/241) Updating files: 47% (114/241) Updating files: 48% (116/241) Updating files: 49% (119/241) Updating files: 50% (121/241) Updating files: 51% (123/241) Updating files: 52% (126/241) Updating files: 53% (128/241) Updating files: 54% (131/241) Updating files: 55% (133/241) Updating files: 56% (135/241) Updating files: 57% (138/241) Updating files: 58% (140/241) Updating files: 59% (143/241) Updating files: 60% (145/241) Updating files: 61% (148/241) Updating files: 62% (150/241) Updating files: 63% (152/241) Updating files: 64% (155/241) Updating files: 65% (157/241) Updating files: 66% (160/241) Updating files: 67% (162/241) Updating files: 68% (164/241) Updating files: 69% (167/241) Updating files: 70% (169/241) Updating files: 71% (172/241) Updating files: 72% (174/241) Updating files: 73% (176/241) Updating files: 74% (179/241) Updating files: 74% (180/241) Updating files: 75% (181/241) Updating files: 76% (184/241) Updating files: 77% (186/241) Updating files: 78% (188/241) Updating files: 79% (191/241) Updating files: 80% (193/241) Updating files: 81% (196/241) Updating files: 82% (198/241) Updating files: 83% (201/241) Updating files: 84% (203/241) Updating files: 85% (205/241) Updating files: 86% (208/241) Updating files: 87% (210/241) Updating files: 88% (213/241) Updating files: 89% (215/241) Updating files: 90% (217/241) Updating files: 91% (220/241) Updating files: 92% (222/241) Updating files: 93% (225/241) Updating files: 94% (227/241) Updating files: 95% (229/241) Updating files: 96% (232/241) Updating files: 97% (234/241) Updating files: 98% (237/241) Updating files: 99% (239/241) Updating files: 100% (241/241) Updating files: 100% (241/241), done. 2023-10-30T13:03:25.852 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at b4cf0f2574b v0.68 2023-10-30T13:03:25.858 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.68 2023-10-30T13:03:26.006 INFO:tasks.workunit.client.0.smithi125.stdout:v0.69 2023-10-30T13:03:26.006 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 69 -eq 48 ] 2023-10-30T13:03:26.007 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.69 2023-10-30T13:03:26.007 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.69 2023-10-30T13:03:26.007 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.69 2023-10-30T13:03:30.494 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 58% (90/155) Updating files: 59% (92/155) Updating files: 60% (93/155) Updating files: 61% (95/155) Updating files: 62% (97/155) Updating files: 63% (98/155) Updating files: 64% (100/155) Updating files: 65% (101/155) Updating files: 66% (103/155) Updating files: 67% (104/155) Updating files: 68% (106/155) Updating files: 69% (107/155) Updating files: 70% (109/155) Updating files: 71% (111/155) Updating files: 72% (112/155) Updating files: 73% (114/155) Updating files: 74% (115/155) Updating files: 75% (117/155) Updating files: 76% (118/155) Updating files: 77% (120/155) Updating files: 78% (121/155) Updating files: 79% (123/155) Updating files: 80% (124/155) Updating files: 81% (126/155) Updating files: 82% (128/155) Updating files: 83% (129/155) Updating files: 84% (131/155) Updating files: 85% (132/155) Updating files: 85% (133/155) Updating files: 86% (134/155) Updating files: 87% (135/155) Updating files: 88% (137/155) Updating files: 89% (138/155) Updating files: 90% (140/155) Updating files: 91% (142/155) Updating files: 92% (143/155) Updating files: 93% (145/155) Updating files: 94% (146/155) Updating files: 95% (148/155) Updating files: 96% (149/155) Updating files: 97% (151/155) Updating files: 98% (152/155) Updating files: 99% (154/155) Updating files: 100% (155/155) Updating files: 100% (155/155), done. 2023-10-30T13:03:30.541 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 6ca6f2f9f75 v0.69 2023-10-30T13:03:30.548 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.69 2023-10-30T13:03:31.121 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 70 -eq 48 ] 2023-10-30T13:03:31.121 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.70 2023-10-30T13:03:31.122 INFO:tasks.workunit.client.0.smithi125.stdout:v0.70 2023-10-30T13:03:31.122 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.70 2023-10-30T13:03:31.122 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.70 2023-10-30T13:03:34.414 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 47% (21/44) Updating files: 50% (22/44) Updating files: 52% (23/44) Updating files: 54% (24/44) Updating files: 56% (25/44) Updating files: 59% (26/44) Updating files: 61% (27/44) Updating files: 63% (28/44) Updating files: 65% (29/44) Updating files: 68% (30/44) Updating files: 70% (31/44) Updating files: 72% (32/44) Updating files: 75% (33/44) Updating files: 77% (34/44) Updating files: 79% (35/44) Updating files: 81% (36/44) Updating files: 84% (37/44) Updating files: 86% (38/44) Updating files: 88% (39/44) Updating files: 90% (40/44) Updating files: 93% (41/44) Updating files: 95% (42/44) Updating files: 97% (43/44) Updating files: 100% (44/44) Updating files: 100% (44/44), done. 2023-10-30T13:03:34.455 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at e3bb0656d92 v0.70 2023-10-30T13:03:34.457 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.70 2023-10-30T13:03:34.620 INFO:tasks.workunit.client.0.smithi125.stdout:v0.71 2023-10-30T13:03:34.620 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 71 -eq 48 ] 2023-10-30T13:03:34.620 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.71 2023-10-30T13:03:34.621 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.71 2023-10-30T13:03:34.621 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.71 2023-10-30T13:03:45.210 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 7% (29/405) Updating files: 7% (30/405) Updating files: 8% (33/405) Updating files: 9% (37/405) Updating files: 9% (39/405) Updating files: 10% (41/405) Updating files: 11% (45/405) Updating files: 12% (49/405) Updating files: 13% (53/405) Updating files: 13% (55/405) Updating files: 14% (57/405) Updating files: 15% (61/405) Updating files: 16% (65/405) Updating files: 17% (69/405) Updating files: 18% (73/405) Updating files: 19% (77/405) Updating files: 20% (81/405) Updating files: 21% (86/405) Updating files: 22% (90/405) Updating files: 23% (94/405) Updating files: 24% (98/405) Updating files: 25% (102/405) Updating files: 26% (106/405) Updating files: 27% (110/405) Updating files: 28% (114/405) Updating files: 29% (118/405) Updating files: 30% (122/405) Updating files: 31% (126/405) Updating files: 32% (130/405) Updating files: 32% (132/405) Updating files: 33% (134/405) Updating files: 34% (138/405) Updating files: 35% (142/405) Updating files: 36% (146/405) Updating files: 37% (150/405) Updating files: 38% (154/405) Updating files: 39% (158/405) Updating files: 39% (159/405) Updating files: 40% (162/405) Updating files: 41% (167/405) Updating files: 42% (171/405) Updating files: 43% (175/405) Updating files: 44% (179/405) Updating files: 45% (183/405) Updating files: 46% (187/405) Updating files: 47% (191/405) Updating files: 48% (195/405) Updating files: 49% (199/405) Updating files: 50% (203/405) Updating files: 51% (207/405) Updating files: 52% (211/405) Updating files: 53% (215/405) Updating files: 54% (219/405) Updating files: 55% (223/405) Updating files: 55% (226/405) Updating files: 56% (227/405) Updating files: 57% (231/405) Updating files: 58% (235/405) Updating files: 59% (239/405) Updating files: 60% (243/405) Updating files: 61% (248/405) Updating files: 62% (252/405) Updating files: 63% (256/405) Updating files: 63% (258/405) Updating files: 64% (260/405) Updating files: 65% (264/405) Updating files: 66% (268/405) Updating files: 67% (272/405) Updating files: 68% (276/405) Updating files: 69% (280/405) Updating files: 70% (284/405) Updating files: 71% (288/405) Updating files: 72% (292/405) Updating files: 73% (296/405) Updating files: 74% (300/405) Updating files: 75% (304/405) Updating files: 76% (308/405) Updating files: 77% (312/405) Updating files: 78% (316/405) Updating files: 79% (320/405) Updating files: 80% (324/405) Updating files: 81% (329/405) Updating files: 82% (333/405) Updating files: 83% (337/405) Updating files: 84% (341/405) Updating files: 85% (345/405) Updating files: 86% (349/405) Updating files: 87% (353/405) Updating files: 88% (357/405) Updating files: 89% (361/405) Updating files: 90% (365/405) Updating files: 91% (369/405) Updating files: 92% (373/405) Updating files: 93% (377/405) Updating files: 94% (381/405) Updating files: 95% (385/405) Updating files: 96% (389/405) Updating files: 97% (393/405) Updating files: 98% (397/405) Updating files: 99% (401/405) Updating files: 100% (405/405) Updating files: 100% (405/405), done. 2023-10-30T13:03:45.246 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at e509cb1e69c v0.71 2023-10-30T13:03:45.254 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.71 2023-10-30T13:03:46.014 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 72 -eq 48 ] 2023-10-30T13:03:46.015 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.72 2023-10-30T13:03:46.015 INFO:tasks.workunit.client.0.smithi125.stdout:v0.72 2023-10-30T13:03:46.015 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.72 2023-10-30T13:03:46.016 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.72 2023-10-30T13:03:48.218 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 89% (170/191) Updating files: 90% (172/191) Updating files: 91% (174/191) Updating files: 92% (176/191) Updating files: 93% (178/191) Updating files: 94% (180/191) Updating files: 95% (182/191) Updating files: 96% (184/191) Updating files: 97% (186/191) Updating files: 98% (188/191) Updating files: 99% (190/191) Updating files: 100% (191/191) Updating files: 100% (191/191), done. 2023-10-30T13:03:48.270 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 5832e2603c7 v0.72 2023-10-30T13:03:48.277 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.72 2023-10-30T13:03:48.409 INFO:tasks.workunit.client.0.smithi125.stdout:v0.73 2023-10-30T13:03:48.410 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 73 -eq 48 ] 2023-10-30T13:03:48.410 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.73 2023-10-30T13:03:48.410 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.73 2023-10-30T13:03:48.410 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.73 2023-10-30T13:03:52.487 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:03:52.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:03:59.452 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 64% (95/148) Updating files: 65% (97/148) Updating files: 66% (98/148) Updating files: 67% (100/148) Updating files: 68% (101/148) Updating files: 69% (103/148) Updating files: 70% (104/148) Updating files: 71% (106/148) Updating files: 72% (107/148) Updating files: 73% (109/148) Updating files: 74% (110/148) Updating files: 75% (111/148) Updating files: 76% (113/148) Updating files: 77% (114/148) Updating files: 78% (116/148) Updating files: 79% (117/148) Updating files: 80% (119/148) Updating files: 81% (120/148) Updating files: 82% (122/148) Updating files: 83% (123/148) Updating files: 84% (125/148) Updating files: 85% (126/148) Updating files: 86% (128/148) Updating files: 87% (129/148) Updating files: 88% (131/148) Updating files: 89% (132/148) Updating files: 90% (134/148) Updating files: 91% (135/148) Updating files: 92% (137/148) Updating files: 93% (138/148) Updating files: 94% (140/148) Updating files: 95% (141/148) Updating files: 96% (143/148) Updating files: 97% (144/148) Updating files: 98% (146/148) Updating files: 99% (147/148) Updating files: 100% (148/148) Updating files: 100% (148/148), done. 2023-10-30T13:03:59.482 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at d8ad51ee8ab v0.73 2023-10-30T13:03:59.487 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.73 2023-10-30T13:03:59.775 INFO:tasks.workunit.client.0.smithi125.stdout:v0.74 2023-10-30T13:03:59.775 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 74 -eq 48 ] 2023-10-30T13:03:59.775 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.74 2023-10-30T13:03:59.776 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.74 2023-10-30T13:03:59.776 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.74 2023-10-30T13:04:06.719 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 4% (9/192) Updating files: 5% (10/192) Updating files: 6% (12/192) Updating files: 7% (14/192) Updating files: 8% (16/192) Updating files: 9% (18/192) Updating files: 10% (20/192) Updating files: 11% (22/192) Updating files: 12% (24/192) Updating files: 13% (25/192) Updating files: 14% (27/192) Updating files: 15% (29/192) Updating files: 16% (31/192) Updating files: 17% (33/192) Updating files: 18% (35/192) Updating files: 19% (37/192) Updating files: 20% (39/192) Updating files: 21% (41/192) Updating files: 22% (43/192) Updating files: 23% (45/192) Updating files: 24% (47/192) Updating files: 25% (48/192) Updating files: 26% (50/192) Updating files: 27% (52/192) Updating files: 28% (54/192) Updating files: 29% (56/192) Updating files: 30% (58/192) Updating files: 31% (60/192) Updating files: 32% (62/192) Updating files: 33% (64/192) Updating files: 34% (66/192) Updating files: 35% (68/192) Updating files: 36% (70/192) Updating files: 37% (72/192) Updating files: 38% (73/192) Updating files: 39% (75/192) Updating files: 40% (77/192) Updating files: 41% (79/192) Updating files: 42% (81/192) Updating files: 43% (83/192) Updating files: 44% (85/192) Updating files: 45% (87/192) Updating files: 46% (89/192) Updating files: 47% (91/192) Updating files: 48% (93/192) Updating files: 49% (95/192) Updating files: 50% (96/192) Updating files: 51% (98/192) Updating files: 52% (100/192) Updating files: 53% (102/192) Updating files: 54% (104/192) Updating files: 55% (106/192) Updating files: 56% (108/192) Updating files: 57% (110/192) Updating files: 58% (112/192) Updating files: 59% (114/192) Updating files: 60% (116/192) Updating files: 61% (118/192) Updating files: 62% (120/192) Updating files: 63% (121/192) Updating files: 64% (123/192) Updating files: 65% (125/192) Updating files: 66% (127/192) Updating files: 67% (129/192) Updating files: 68% (131/192) Updating files: 69% (133/192) Updating files: 70% (135/192) Updating files: 71% (137/192) Updating files: 72% (139/192) Updating files: 73% (141/192) Updating files: 74% (143/192) Updating files: 75% (144/192) Updating files: 76% (146/192) Updating files: 77% (148/192) Updating files: 78% (150/192) Updating files: 79% (152/192) Updating files: 80% (154/192) Updating files: 81% (156/192) Updating files: 82% (158/192) Updating files: 83% (160/192) Updating files: 84% (162/192) Updating files: 85% (164/192) Updating files: 86% (166/192) Updating files: 87% (168/192) Updating files: 88% (169/192) Updating files: 89% (171/192) Updating files: 90% (173/192) Updating files: 91% (175/192) Updating files: 92% (177/192) Updating files: 93% (179/192) Updating files: 94% (181/192) Updating files: 95% (183/192) Updating files: 96% (185/192) Updating files: 97% (187/192) Updating files: 98% (189/192) Updating files: 99% (191/192) Updating files: 100% (192/192) Updating files: 100% (192/192), done. 2023-10-30T13:04:06.760 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at c165483bc72 v0.74 2023-10-30T13:04:06.766 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.74 2023-10-30T13:04:06.860 INFO:tasks.workunit.client.0.smithi125.stdout:v0.75 2023-10-30T13:04:06.861 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 75 -eq 48 ] 2023-10-30T13:04:06.861 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.75 2023-10-30T13:04:06.861 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.75 2023-10-30T13:04:06.862 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.75 2023-10-30T13:04:10.436 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 56% (165/291) Updating files: 57% (166/291) Updating files: 58% (169/291) Updating files: 59% (172/291) Updating files: 60% (175/291) Updating files: 61% (178/291) Updating files: 62% (181/291) Updating files: 63% (184/291) Updating files: 64% (187/291) Updating files: 65% (190/291) Updating files: 66% (193/291) Updating files: 67% (195/291) Updating files: 68% (198/291) Updating files: 69% (201/291) Updating files: 70% (204/291) Updating files: 71% (207/291) Updating files: 72% (210/291) Updating files: 73% (213/291) Updating files: 74% (216/291) Updating files: 75% (219/291) Updating files: 76% (222/291) Updating files: 77% (225/291) Updating files: 78% (227/291) Updating files: 79% (230/291) Updating files: 80% (233/291) Updating files: 81% (236/291) Updating files: 82% (239/291) Updating files: 83% (242/291) Updating files: 84% (245/291) Updating files: 85% (248/291) Updating files: 86% (251/291) Updating files: 87% (254/291) Updating files: 88% (257/291) Updating files: 89% (259/291) Updating files: 90% (262/291) Updating files: 91% (265/291) Updating files: 92% (268/291) Updating files: 93% (271/291) Updating files: 94% (274/291) Updating files: 95% (277/291) Updating files: 96% (280/291) Updating files: 97% (283/291) Updating files: 98% (286/291) Updating files: 99% (289/291) Updating files: 100% (291/291) Updating files: 100% (291/291), done. 2023-10-30T13:04:10.744 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 946d6036958 v0.75 2023-10-30T13:04:10.749 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.75 2023-10-30T13:04:10.831 INFO:tasks.workunit.client.0.smithi125.stdout:v0.76 2023-10-30T13:04:10.832 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 76 -eq 48 ] 2023-10-30T13:04:10.832 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.76 2023-10-30T13:04:10.832 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.76 2023-10-30T13:04:10.833 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.76 2023-10-30T13:04:13.335 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 60% (102/168) Updating files: 61% (103/168) Updating files: 62% (105/168) Updating files: 63% (106/168) Updating files: 64% (108/168) Updating files: 65% (110/168) Updating files: 66% (111/168) Updating files: 67% (113/168) Updating files: 68% (115/168) Updating files: 69% (116/168) Updating files: 70% (118/168) Updating files: 71% (120/168) Updating files: 72% (121/168) Updating files: 73% (123/168) Updating files: 74% (125/168) Updating files: 75% (126/168) Updating files: 76% (128/168) Updating files: 77% (130/168) Updating files: 78% (132/168) Updating files: 79% (133/168) Updating files: 80% (135/168) Updating files: 81% (137/168) Updating files: 82% (138/168) Updating files: 83% (140/168) Updating files: 84% (142/168) Updating files: 85% (143/168) Updating files: 86% (145/168) Updating files: 87% (147/168) Updating files: 88% (148/168) Updating files: 89% (150/168) Updating files: 90% (152/168) Updating files: 91% (153/168) Updating files: 92% (155/168) Updating files: 93% (157/168) Updating files: 94% (158/168) Updating files: 95% (160/168) Updating files: 96% (162/168) Updating files: 97% (163/168) Updating files: 98% (165/168) Updating files: 99% (167/168) Updating files: 100% (168/168) Updating files: 100% (168/168), done. 2023-10-30T13:04:13.385 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 3b990136bfa v0.76 2023-10-30T13:04:13.390 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.76 2023-10-30T13:04:13.664 INFO:tasks.workunit.client.0.smithi125.stdout:v0.77 2023-10-30T13:04:13.665 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 77 -eq 48 ] 2023-10-30T13:04:13.665 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.77 2023-10-30T13:04:13.665 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.77 2023-10-30T13:04:13.665 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.77 2023-10-30T13:04:22.543 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:04:22.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:04:30.455 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 39% (117/298) Updating files: 40% (120/298) Updating files: 41% (123/298) Updating files: 42% (126/298) Updating files: 43% (129/298) Updating files: 44% (132/298) Updating files: 45% (135/298) Updating files: 46% (138/298) Updating files: 47% (141/298) Updating files: 48% (144/298) Updating files: 49% (147/298) Updating files: 50% (149/298) Updating files: 51% (152/298) Updating files: 52% (155/298) Updating files: 53% (158/298) Updating files: 54% (161/298) Updating files: 55% (164/298) Updating files: 56% (167/298) Updating files: 57% (170/298) Updating files: 58% (173/298) Updating files: 59% (176/298) Updating files: 60% (179/298) Updating files: 61% (182/298) Updating files: 62% (185/298) Updating files: 63% (188/298) Updating files: 64% (191/298) Updating files: 65% (194/298) Updating files: 66% (197/298) Updating files: 67% (200/298) Updating files: 68% (203/298) Updating files: 69% (206/298) Updating files: 70% (209/298) Updating files: 71% (212/298) Updating files: 72% (215/298) Updating files: 73% (218/298) Updating files: 74% (221/298) Updating files: 75% (224/298) Updating files: 76% (227/298) Updating files: 77% (230/298) Updating files: 78% (233/298) Updating files: 79% (236/298) Updating files: 80% (239/298) Updating files: 81% (242/298) Updating files: 82% (245/298) Updating files: 83% (248/298) Updating files: 84% (251/298) Updating files: 85% (254/298) Updating files: 86% (257/298) Updating files: 86% (259/298) Updating files: 87% (260/298) Updating files: 88% (263/298) Updating files: 89% (266/298) Updating files: 90% (269/298) Updating files: 91% (272/298) Updating files: 92% (275/298) Updating files: 93% (278/298) Updating files: 94% (281/298) Updating files: 95% (284/298) Updating files: 96% (287/298) Updating files: 97% (290/298) Updating files: 98% (293/298) Updating files: 99% (296/298) Updating files: 100% (298/298) Updating files: 100% (298/298), done. 2023-10-30T13:04:30.526 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 1bca9c5c412 v0.77 2023-10-30T13:04:30.537 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.77 2023-10-30T13:04:30.602 INFO:tasks.workunit.client.0.smithi125.stdout:v0.78 2023-10-30T13:04:30.602 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 78 -eq 48 ] 2023-10-30T13:04:30.602 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.78 2023-10-30T13:04:30.603 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.78 2023-10-30T13:04:30.603 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.78 2023-10-30T13:04:33.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:04:33.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.432+0000 7f7552994640 1 Processor -- start 2023-10-30T13:04:33.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.432+0000 7f7552994640 1 -- start start 2023-10-30T13:04:33.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.432+0000 7f7552994640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f754c06f7a0 0x7f754c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:33.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.432+0000 7f7552994640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754c066cf0 con 0x7f754c071110 2023-10-30T13:04:33.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.432+0000 7f7552994640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754c0c70d0 con 0x7f754c0c6bd0 2023-10-30T13:04:33.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.432+0000 7f7552994640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f754c0700c0 con 0x7f754c06f7a0 2023-10-30T13:04:33.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.432+0000 7f754bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f754c06f7a0 0x7f754c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:33.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.432+0000 7f754bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f754c06f7a0 0x7f754c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37824/0 (socket says 172.21.15.125:37824) 2023-10-30T13:04:33.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.432+0000 7f754bfff640 1 -- 172.21.15.125:0/3357617022 learned_addr learned my addr 172.21.15.125:0/3357617022 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:33.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.432+0000 7f754bfff640 1 -- 172.21.15.125:0/3357617022 >> v1:172.21.15.136:6789/0 conn(0x7f754c0c6bd0 legacy=0x7f754c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:04:33.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.432+0000 7f754bfff640 1 -- 172.21.15.125:0/3357617022 >> v1:172.21.15.125:6789/0 conn(0x7f754c071110 legacy=0x7f754c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:33.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.432+0000 7f754bfff640 1 -- 172.21.15.125:0/3357617022 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f754c0c7390 con 0x7f754c06f7a0 2023-10-30T13:04:33.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.436+0000 7f754bfff640 1 --2- 172.21.15.125:0/3357617022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c06f7a0 0x7f754c06fb80 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f753c009870 tx=0x7f753c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e5091b67ddae300 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:33.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.436+0000 7f754affd640 1 -- 172.21.15.125:0/3357617022 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c0a89f0 con 0x7f754c06f7a0 2023-10-30T13:04:33.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.436+0000 7f754affd640 1 -- 172.21.15.125:0/3357617022 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753c0a8b50 con 0x7f754c06f7a0 2023-10-30T13:04:33.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.436+0000 7f754affd640 1 -- 172.21.15.125:0/3357617022 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753c0b0440 con 0x7f754c06f7a0 2023-10-30T13:04:33.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.436+0000 7f7552994640 1 -- 172.21.15.125:0/3357617022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c06f7a0 msgr2=0x7f754c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:33.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.436+0000 7f7552994640 1 --2- 172.21.15.125:0/3357617022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c06f7a0 0x7f754c06fb80 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f753c009870 tx=0x7f753c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:04:33.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f7552994640 1 -- 172.21.15.125:0/3357617022 shutdown_connections 2023-10-30T13:04:33.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f7552994640 1 --2- 172.21.15.125:0/3357617022 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c06f7a0 0x7f754c06fb80 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:33.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f7552994640 1 -- 172.21.15.125:0/3357617022 >> 172.21.15.125:0/3357617022 conn(0x7f754c0c7910 msgr2=0x7f754c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:33.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f7552994640 1 -- 172.21.15.125:0/3357617022 shutdown_connections 2023-10-30T13:04:33.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f7552994640 1 -- 172.21.15.125:0/3357617022 wait complete. 2023-10-30T13:04:33.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f7552994640 1 Processor -- start 2023-10-30T13:04:33.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f7552994640 1 -- start start 2023-10-30T13:04:33.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f7552994640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f754c06f7a0 0x7f754c1531c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:33.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f7552994640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c071110 0x7f754c14a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:33.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f7552994640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f754c0c6bd0 0x7f754c14a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:33.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f7552994640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c152340 con 0x7f754c071110 2023-10-30T13:04:33.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f7552994640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754c14ae10 con 0x7f754c0c6bd0 2023-10-30T13:04:33.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f7552994640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f754c0c3240 con 0x7f754c06f7a0 2023-10-30T13:04:33.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c071110 0x7f754c14a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:33.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f754b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c071110 0x7f754c14a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37826/0 (socket says 172.21.15.125:37826) 2023-10-30T13:04:33.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f754b7fe640 1 -- 172.21.15.125:0/3636584640 learned_addr learned my addr 172.21.15.125:0/3636584640 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:33.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f754bfff640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f754c06f7a0 0x7f754c1531c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:33.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f7550f31640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f754c0c6bd0 0x7f754c14a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:33.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f754b7fe640 1 -- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f754c06f7a0 msgr2=0x7f754c1531c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:33.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f754b7fe640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f754c06f7a0 0x7f754c1531c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:33.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f754b7fe640 1 -- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f754c0c6bd0 msgr2=0x7f754c14a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:33.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f754b7fe640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f754c0c6bd0 0x7f754c14a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:33.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f754b7fe640 1 -- 172.21.15.125:0/3636584640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f753c0094d0 con 0x7f754c071110 2023-10-30T13:04:33.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f754bfff640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f754c06f7a0 0x7f754c1531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:04:33.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.440+0000 7f754b7fe640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c071110 0x7f754c14a390 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f754000c440 tx=0x7f754000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:33.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.444+0000 7f7548ff9640 1 -- 172.21.15.125:0/3636584640 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7540012830 con 0x7f754c071110 2023-10-30T13:04:33.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.444+0000 7f7548ff9640 1 -- 172.21.15.125:0/3636584640 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7540012990 con 0x7f754c071110 2023-10-30T13:04:33.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.444+0000 7f7548ff9640 1 -- 172.21.15.125:0/3636584640 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7540012d40 con 0x7f754c071110 2023-10-30T13:04:33.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.444+0000 7f7552994640 1 -- 172.21.15.125:0/3636584640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f754c0c0a70 con 0x7f754c071110 2023-10-30T13:04:33.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.444+0000 7f7552994640 1 -- 172.21.15.125:0/3636584640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f754c0c1030 con 0x7f754c071110 2023-10-30T13:04:33.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.444+0000 7f7552994640 1 -- 172.21.15.125:0/3636584640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f754c04e170 con 0x7f754c071110 2023-10-30T13:04:33.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.444+0000 7f7548ff9640 1 -- 172.21.15.125:0/3636584640 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f754001b400 con 0x7f754c071110 2023-10-30T13:04:33.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.444+0000 7f7548ff9640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75380664d0 0x7f7538068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:33.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.444+0000 7f7548ff9640 1 -- 172.21.15.125:0/3636584640 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f7540084460 con 0x7f754c071110 2023-10-30T13:04:33.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.444+0000 7f754bfff640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75380664d0 0x7f7538068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:33.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.448+0000 7f754bfff640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75380664d0 0x7f7538068990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f753c0b2000 tx=0x7f753c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:33.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.448+0000 7f7548ff9640 1 -- 172.21.15.125:0/3636584640 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7540051210 con 0x7f754c071110 2023-10-30T13:04:33.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.632+0000 7f7552994640 1 -- 172.21.15.125:0/3636584640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f754c0c47f0 con 0x7f754c071110 2023-10-30T13:04:33.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.632+0000 7f7548ff9640 1 -- 172.21.15.125:0/3636584640 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f7540054830 con 0x7f754c071110 2023-10-30T13:04:33.638 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:04:33.638 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:57.950518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:active","state_seq":30,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:04:33.638 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 155 2023-10-30T13:04:33.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.636+0000 7f7552994640 1 -- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75380664d0 msgr2=0x7f7538068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:33.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.636+0000 7f7552994640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75380664d0 0x7f7538068990 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f753c0b2000 tx=0x7f753c008040 comp rx=0 tx=0).stop 2023-10-30T13:04:33.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.636+0000 7f7552994640 1 -- 172.21.15.125:0/3636584640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c071110 msgr2=0x7f754c14a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:33.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.636+0000 7f7552994640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c071110 0x7f754c14a390 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f754000c440 tx=0x7f754000c900 comp rx=0 tx=0).stop 2023-10-30T13:04:33.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.636+0000 7f7552994640 1 -- 172.21.15.125:0/3636584640 shutdown_connections 2023-10-30T13:04:33.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.636+0000 7f7552994640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75380664d0 0x7f7538068990 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:33.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.636+0000 7f7552994640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f754c0c6bd0 0x7f754c14a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:33.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.636+0000 7f7552994640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f754c071110 0x7f754c14a390 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:33.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.636+0000 7f7552994640 1 --2- 172.21.15.125:0/3636584640 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f754c06f7a0 0x7f754c1531c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:33.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.636+0000 7f7552994640 1 -- 172.21.15.125:0/3636584640 >> 172.21.15.125:0/3636584640 conn(0x7f754c0c7910 msgr2=0x7f754c0ca0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:33.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.636+0000 7f7552994640 1 -- 172.21.15.125:0/3636584640 shutdown_connections 2023-10-30T13:04:33.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.636+0000 7f7552994640 1 -- 172.21.15.125:0/3636584640 wait complete. 2023-10-30T13:04:33.658 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T13:04:33.658 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T13:04:33.659 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:04:33.692 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T13:04:33.692 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:04:33.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.868+0000 7fdf815a1640 1 Processor -- start 2023-10-30T13:04:33.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.868+0000 7fdf815a1640 1 -- start start 2023-10-30T13:04:33.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.868+0000 7fdf815a1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdf7c1538a0 0x7fdf7c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:33.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.868+0000 7fdf815a1640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7c0cb800 con 0x7fdf7c0cf160 2023-10-30T13:04:33.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.868+0000 7fdf815a1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7c15dc30 con 0x7fdf7c0ca0e0 2023-10-30T13:04:33.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.868+0000 7fdf815a1640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7c0cb330 con 0x7fdf7c1538a0 2023-10-30T13:04:33.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.868+0000 7fdf7affd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdf7c1538a0 0x7fdf7c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:33.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.868+0000 7fdf7b7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdf7c0cf160 0x7fdf7c164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46352/0 (socket says 172.21.15.125:46352) 2023-10-30T13:04:33.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.868+0000 7fdf7b7fe640 1 -- 172.21.15.125:0/2929897603 learned_addr learned my addr 172.21.15.125:0/2929897603 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:04:33.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.868+0000 7fdf7affd640 1 -- 172.21.15.125:0/2929897603 >> v1:172.21.15.136:6789/0 conn(0x7fdf7c0ca0e0 legacy=0x7fdf7c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:04:33.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.868+0000 7fdf7a7fc640 1 -- 172.21.15.125:0/2929897603 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604464835 0 0) 0x7fdf7c0cb800 con 0x7fdf7c0cf160 2023-10-30T13:04:33.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.868+0000 7fdf7affd640 1 -- 172.21.15.125:0/2929897603 >> v1:172.21.15.125:6789/0 conn(0x7fdf7c0cf160 legacy=0x7fdf7c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:33.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.868+0000 7fdf7affd640 1 -- 172.21.15.125:0/2929897603 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c0cad70 con 0x7fdf7c1538a0 2023-10-30T13:04:33.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.872+0000 7fdf7affd640 1 --2- 172.21.15.125:0/2929897603 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf7c1538a0 0x7fdf7c15d6f0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0097c0 tx=0x7fdf6c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f303a8491e2fcb82 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:33.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.872+0000 7fdf7a7fc640 1 -- 172.21.15.125:0/2929897603 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c0a8970 con 0x7fdf7c1538a0 2023-10-30T13:04:33.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.872+0000 7fdf7a7fc640 1 -- 172.21.15.125:0/2929897603 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c0a8ad0 con 0x7fdf7c1538a0 2023-10-30T13:04:33.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.872+0000 7fdf7a7fc640 1 -- 172.21.15.125:0/2929897603 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c0a8e60 con 0x7fdf7c1538a0 2023-10-30T13:04:33.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.872+0000 7fdf815a1640 1 -- 172.21.15.125:0/2929897603 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf7c1538a0 msgr2=0x7fdf7c15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:33.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.872+0000 7fdf815a1640 1 --2- 172.21.15.125:0/2929897603 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf7c1538a0 0x7fdf7c15d6f0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0097c0 tx=0x7fdf6c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:04:33.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.872+0000 7fdf815a1640 1 -- 172.21.15.125:0/2929897603 shutdown_connections 2023-10-30T13:04:33.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.872+0000 7fdf815a1640 1 --2- 172.21.15.125:0/2929897603 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf7c1538a0 0x7fdf7c15d6f0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:33.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.872+0000 7fdf815a1640 1 -- 172.21.15.125:0/2929897603 >> 172.21.15.125:0/2929897603 conn(0x7fdf7c06bfe0 msgr2=0x7fdf7c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:33.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.872+0000 7fdf815a1640 1 -- 172.21.15.125:0/2929897603 shutdown_connections 2023-10-30T13:04:33.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.872+0000 7fdf815a1640 1 -- 172.21.15.125:0/2929897603 wait complete. 2023-10-30T13:04:33.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf815a1640 1 Processor -- start 2023-10-30T13:04:33.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf815a1640 1 -- start start 2023-10-30T13:04:33.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf815a1640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdf7c0ca0e0 0x7fdf7c158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:33.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf815a1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf7c0cf160 0x7fdf7c0b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:33.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf815a1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdf7c1538a0 0x7fdf7c1eee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:33.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf815a1640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c0c8ed0 con 0x7fdf7c0cf160 2023-10-30T13:04:33.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf815a1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c1f2400 con 0x7fdf7c1538a0 2023-10-30T13:04:33.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf815a1640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c158e70 con 0x7fdf7c0ca0e0 2023-10-30T13:04:33.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf727fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf7c0cf160 0x7fdf7c0b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf727fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf7c0cf160 0x7fdf7c0b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37852/0 (socket says 172.21.15.125:37852) 2023-10-30T13:04:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf727fc640 1 -- 172.21.15.125:0/576444685 learned_addr learned my addr 172.21.15.125:0/576444685 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf7affd640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdf7c0ca0e0 0x7fdf7c158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf7b7fe640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdf7c1538a0 0x7fdf7c1eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf727fc640 1 -- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdf7c0ca0e0 msgr2=0x7fdf7c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:33.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf727fc640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdf7c0ca0e0 0x7fdf7c158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:33.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf727fc640 1 -- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdf7c1538a0 msgr2=0x7fdf7c1eee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:33.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf727fc640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdf7c1538a0 0x7fdf7c1eee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:33.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf727fc640 1 -- 172.21.15.125:0/576444685 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c009420 con 0x7fdf7c0cf160 2023-10-30T13:04:33.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf7b7fe640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdf7c1538a0 0x7fdf7c1eee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:04:33.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf727fc640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf7c0cf160 0x7fdf7c0b4450 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fdf6400c3c0 tx=0x7fdf6400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:33.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf73fff640 1 -- 172.21.15.125:0/576444685 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf640052d0 con 0x7fdf7c0cf160 2023-10-30T13:04:33.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf73fff640 1 -- 172.21.15.125:0/576444685 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf64005430 con 0x7fdf7c0cf160 2023-10-30T13:04:33.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf815a1640 1 -- 172.21.15.125:0/576444685 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7c1f2610 con 0x7fdf7c0cf160 2023-10-30T13:04:33.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf73fff640 1 -- 172.21.15.125:0/576444685 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf64003840 con 0x7fdf7c0cf160 2023-10-30T13:04:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.876+0000 7fdf815a1640 1 -- 172.21.15.125:0/576444685 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7c1f2bd0 con 0x7fdf7c0cf160 2023-10-30T13:04:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.880+0000 7fdf815a1640 1 -- 172.21.15.125:0/576444685 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7c04e170 con 0x7fdf7c0cf160 2023-10-30T13:04:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.880+0000 7fdf73fff640 1 -- 172.21.15.125:0/576444685 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdf640039a0 con 0x7fdf7c0cf160 2023-10-30T13:04:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.880+0000 7fdf73fff640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdf600664a0 0x7fdf60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.880+0000 7fdf73fff640 1 -- 172.21.15.125:0/576444685 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7fdf640837a0 con 0x7fdf7c0cf160 2023-10-30T13:04:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.880+0000 7fdf7affd640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdf600664a0 0x7fdf60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:33.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.880+0000 7fdf7affd640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdf600664a0 0x7fdf60068960 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0b2000 tx=0x7fdf6c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:33.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:33.884+0000 7fdf73fff640 1 -- 172.21.15.125:0/576444685 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdf64050690 con 0x7fdf7c0cf160 2023-10-30T13:04:34.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:34.064+0000 7fdf815a1640 1 -- 172.21.15.125:0/576444685 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf7c15dc30 con 0x7fdf7c0cf160 2023-10-30T13:04:34.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:34.068+0000 7fdf73fff640 1 -- 172.21.15.125:0/576444685 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fdf7c15dc30 con 0x7fdf7c0cf160 2023-10-30T13:04:34.071 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:04:34.071 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:57.950518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:active","state_seq":30,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:04:34.072 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 155 2023-10-30T13:04:34.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:34.068+0000 7fdf815a1640 1 -- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdf600664a0 msgr2=0x7fdf60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:34.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:34.068+0000 7fdf815a1640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdf600664a0 0x7fdf60068960 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0b2000 tx=0x7fdf6c008040 comp rx=0 tx=0).stop 2023-10-30T13:04:34.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:34.068+0000 7fdf815a1640 1 -- 172.21.15.125:0/576444685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf7c0cf160 msgr2=0x7fdf7c0b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:34.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:34.068+0000 7fdf815a1640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf7c0cf160 0x7fdf7c0b4450 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7fdf6400c3c0 tx=0x7fdf6400c880 comp rx=0 tx=0).stop 2023-10-30T13:04:34.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:34.068+0000 7fdf815a1640 1 -- 172.21.15.125:0/576444685 shutdown_connections 2023-10-30T13:04:34.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:34.068+0000 7fdf815a1640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdf600664a0 0x7fdf60068960 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:34.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:34.068+0000 7fdf815a1640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdf7c1538a0 0x7fdf7c1eee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:34.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:34.068+0000 7fdf815a1640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdf7c0cf160 0x7fdf7c0b4450 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:34.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:34.068+0000 7fdf815a1640 1 --2- 172.21.15.125:0/576444685 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdf7c0ca0e0 0x7fdf7c158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:34.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:34.068+0000 7fdf815a1640 1 -- 172.21.15.125:0/576444685 >> 172.21.15.125:0/576444685 conn(0x7fdf7c06bfe0 msgr2=0x7fdf7c0c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:34.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:34.068+0000 7fdf815a1640 1 -- 172.21.15.125:0/576444685 shutdown_connections 2023-10-30T13:04:34.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:34.068+0000 7fdf815a1640 1 -- 172.21.15.125:0/576444685 wait complete. 2023-10-30T13:04:34.092 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:04:36.093 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:04:36.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.240+0000 7f63925aa640 1 Processor -- start 2023-10-30T13:04:36.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.240+0000 7f63925aa640 1 -- start start 2023-10-30T13:04:36.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.244+0000 7f63925aa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f638c0c68b0 0x7f638c06c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:36.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.244+0000 7f63925aa640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c150940 con 0x7f638c070650 2023-10-30T13:04:36.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.244+0000 7f63925aa640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638c1573f0 con 0x7f638c150560 2023-10-30T13:04:36.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.244+0000 7f63925aa640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f638c156590 con 0x7f638c0c68b0 2023-10-30T13:04:36.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.244+0000 7f6390b47640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f638c070650 0x7f638c160d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46356/0 (socket says 172.21.15.125:46356) 2023-10-30T13:04:36.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.244+0000 7f6390b47640 1 -- 172.21.15.125:0/1921434277 learned_addr learned my addr 172.21.15.125:0/1921434277 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:04:36.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.244+0000 7f638b7fe640 1 --2- 172.21.15.125:0/1921434277 >> v2:172.21.15.125:3300/0 conn(0x7f638c0c68b0 0x7f638c06c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:36.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.244+0000 7f638affd640 1 -- 172.21.15.125:0/1921434277 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855525279 0 0) 0x7f638c150940 con 0x7f638c070650 2023-10-30T13:04:36.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.244+0000 7f638affd640 1 -- 172.21.15.125:0/1921434277 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6374005740 con 0x7f638c070650 2023-10-30T13:04:36.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.244+0000 7f638b7fe640 1 -- 172.21.15.125:0/1921434277 >> v1:172.21.15.136:6789/0 conn(0x7f638c150560 legacy=0x7f638c06afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:36.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.244+0000 7f638b7fe640 1 -- 172.21.15.125:0/1921434277 >> v1:172.21.15.125:6789/0 conn(0x7f638c070650 legacy=0x7f638c160d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:36.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.244+0000 7f638b7fe640 1 -- 172.21.15.125:0/1921434277 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f638c156870 con 0x7f638c0c68b0 2023-10-30T13:04:36.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.248+0000 7f638b7fe640 1 --2- 172.21.15.125:0/1921434277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638c0c68b0 0x7f638c06c8f0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f63800097c0 tx=0x7f638001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5babd7166ebe14a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:36.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.248+0000 7f638affd640 1 -- 172.21.15.125:0/1921434277 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6380004230 con 0x7f638c0c68b0 2023-10-30T13:04:36.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.248+0000 7f638affd640 1 -- 172.21.15.125:0/1921434277 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63800a7050 con 0x7f638c0c68b0 2023-10-30T13:04:36.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.248+0000 7f638affd640 1 -- 172.21.15.125:0/1921434277 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6380003620 con 0x7f638c0c68b0 2023-10-30T13:04:36.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.248+0000 7f63925aa640 1 -- 172.21.15.125:0/1921434277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638c0c68b0 msgr2=0x7f638c06c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:36.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.248+0000 7f63925aa640 1 --2- 172.21.15.125:0/1921434277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638c0c68b0 0x7f638c06c8f0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f63800097c0 tx=0x7f638001c450 comp rx=0 tx=0).stop 2023-10-30T13:04:36.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.248+0000 7f63925aa640 1 -- 172.21.15.125:0/1921434277 shutdown_connections 2023-10-30T13:04:36.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.248+0000 7f63925aa640 1 --2- 172.21.15.125:0/1921434277 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638c0c68b0 0x7f638c06c8f0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:36.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.248+0000 7f63925aa640 1 -- 172.21.15.125:0/1921434277 >> 172.21.15.125:0/1921434277 conn(0x7f638c0cbeb0 msgr2=0x7f638c0cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:36.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.248+0000 7f63925aa640 1 -- 172.21.15.125:0/1921434277 shutdown_connections 2023-10-30T13:04:36.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.248+0000 7f63925aa640 1 -- 172.21.15.125:0/1921434277 wait complete. 2023-10-30T13:04:36.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.248+0000 7f63925aa640 1 Processor -- start 2023-10-30T13:04:36.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.248+0000 7f63925aa640 1 -- start start 2023-10-30T13:04:36.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f63925aa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f638c070650 0x7f638c161f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:36.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f63925aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f638c0c68b0 0x7f638c144cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:36.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f63925aa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638c150560 0x7f638c154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:36.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f63925aa640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638c1543f0 con 0x7f638c150560 2023-10-30T13:04:36.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f63925aa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f638c1546d0 con 0x7f638c070650 2023-10-30T13:04:36.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f63925aa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f638c1539c0 con 0x7f638c0c68b0 2023-10-30T13:04:36.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f6390b47640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638c150560 0x7f638c154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:36.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f6390b47640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638c150560 0x7f638c154b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37876/0 (socket says 172.21.15.125:37876) 2023-10-30T13:04:36.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f6390b47640 1 -- 172.21.15.125:0/892352017 learned_addr learned my addr 172.21.15.125:0/892352017 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:36.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f638b7fe640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f638c0c68b0 0x7f638c144cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:36.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f638bfff640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f638c070650 0x7f638c161f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:36.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f6390b47640 1 -- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f638c0c68b0 msgr2=0x7f638c144cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:36.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f6390b47640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f638c0c68b0 0x7f638c144cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:36.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f6390b47640 1 -- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f638c070650 msgr2=0x7f638c161f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:36.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f6390b47640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f638c070650 0x7f638c161f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:36.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f6390b47640 1 -- 172.21.15.125:0/892352017 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6380009420 con 0x7f638c150560 2023-10-30T13:04:36.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f638bfff640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f638c070650 0x7f638c161f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:04:36.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f6390b47640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638c150560 0x7f638c154b60 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f638400caa0 tx=0x7f638400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:36.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f6388ff9640 1 -- 172.21.15.125:0/892352017 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f638400ac00 con 0x7f638c150560 2023-10-30T13:04:36.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f6388ff9640 1 -- 172.21.15.125:0/892352017 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f638400ad60 con 0x7f638c150560 2023-10-30T13:04:36.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f6388ff9640 1 -- 172.21.15.125:0/892352017 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6384012620 con 0x7f638c150560 2023-10-30T13:04:36.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f63925aa640 1 -- 172.21.15.125:0/892352017 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f638c153ca0 con 0x7f638c150560 2023-10-30T13:04:36.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f63925aa640 1 -- 172.21.15.125:0/892352017 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f638c1f6410 con 0x7f638c150560 2023-10-30T13:04:36.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f63925aa640 1 -- 172.21.15.125:0/892352017 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f638c154840 con 0x7f638c150560 2023-10-30T13:04:36.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.252+0000 7f6388ff9640 1 -- 172.21.15.125:0/892352017 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6384007070 con 0x7f638c150560 2023-10-30T13:04:36.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.256+0000 7f6388ff9640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f63780664a0 0x7f6378068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:36.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.256+0000 7f6388ff9640 1 -- 172.21.15.125:0/892352017 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f63840845d0 con 0x7f638c150560 2023-10-30T13:04:36.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.256+0000 7f638bfff640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f63780664a0 0x7f6378068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:36.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.256+0000 7f638bfff640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f63780664a0 0x7f6378068960 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f637c000f70 tx=0x7f637c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:36.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.260+0000 7f6388ff9640 1 -- 172.21.15.125:0/892352017 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f63840512f0 con 0x7f638c150560 2023-10-30T13:04:36.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.440+0000 7f63925aa640 1 -- 172.21.15.125:0/892352017 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f638c0c6d40 con 0x7f638c150560 2023-10-30T13:04:36.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.440+0000 7f6388ff9640 1 -- 172.21.15.125:0/892352017 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f6384017090 con 0x7f638c150560 2023-10-30T13:04:36.446 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:04:36.446 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:57.950518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:active","state_seq":30,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:04:36.447 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 155 2023-10-30T13:04:36.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.444+0000 7f63925aa640 1 -- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f63780664a0 msgr2=0x7f6378068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:36.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.444+0000 7f63925aa640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f63780664a0 0x7f6378068960 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f637c000f70 tx=0x7f637c009040 comp rx=0 tx=0).stop 2023-10-30T13:04:36.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.444+0000 7f63925aa640 1 -- 172.21.15.125:0/892352017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638c150560 msgr2=0x7f638c154b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:36.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.444+0000 7f63925aa640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638c150560 0x7f638c154b60 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f638400caa0 tx=0x7f638400cf60 comp rx=0 tx=0).stop 2023-10-30T13:04:36.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.444+0000 7f63925aa640 1 -- 172.21.15.125:0/892352017 shutdown_connections 2023-10-30T13:04:36.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.444+0000 7f63925aa640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f63780664a0 0x7f6378068960 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:36.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.444+0000 7f63925aa640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f638c150560 0x7f638c154b60 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:36.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.444+0000 7f63925aa640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f638c0c68b0 0x7f638c144cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:36.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.444+0000 7f63925aa640 1 --2- 172.21.15.125:0/892352017 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f638c070650 0x7f638c161f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:36.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.444+0000 7f63925aa640 1 -- 172.21.15.125:0/892352017 >> 172.21.15.125:0/892352017 conn(0x7f638c0cbeb0 msgr2=0x7f638c14db60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:36.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.444+0000 7f63925aa640 1 -- 172.21.15.125:0/892352017 shutdown_connections 2023-10-30T13:04:36.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:36.444+0000 7f63925aa640 1 -- 172.21.15.125:0/892352017 wait complete. 2023-10-30T13:04:36.467 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:04:38.468 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:04:38.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820e571640 1 Processor -- start 2023-10-30T13:04:38.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820e571640 1 -- start start 2023-10-30T13:04:38.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820e571640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f82081538f0 0x7f8208159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:38.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820e571640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f820815cd50 0x7f82080c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:38.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820e571640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f820815d300 con 0x7f82080cefa0 2023-10-30T13:04:38.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820e571640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f820815d5e0 con 0x7f82081538f0 2023-10-30T13:04:38.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820e571640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f820815d8c0 con 0x7f820815cd50 2023-10-30T13:04:38.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820d56f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f82081538f0 0x7f8208159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:38.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820d56f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f82081538f0 0x7f8208159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37878/0 (socket says 172.21.15.125:37878) 2023-10-30T13:04:38.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820dd70640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f82080cefa0 0x7f82080cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46362/0 (socket says 172.21.15.125:46362) 2023-10-30T13:04:38.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820d56f640 1 -- 172.21.15.125:0/1766288939 learned_addr learned my addr 172.21.15.125:0/1766288939 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:38.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820cd6e640 1 --2- 172.21.15.125:0/1766288939 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f820815cd50 0x7f82080c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:38.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820d56f640 1 -- 172.21.15.125:0/1766288939 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f820815cd50 msgr2=0x7f82080c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:04:38.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820d56f640 1 --2- 172.21.15.125:0/1766288939 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f820815cd50 0x7f82080c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:38.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f81fffff640 1 -- 172.21.15.125:0/1766288939 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885111478 0 0) 0x7f820815d300 con 0x7f82080cefa0 2023-10-30T13:04:38.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820d56f640 1 -- 172.21.15.125:0/1766288939 >> v1:172.21.15.125:6789/0 conn(0x7f82080cefa0 legacy=0x7f82080cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:38.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.612+0000 7f820d56f640 1 -- 172.21.15.125:0/1766288939 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f820815dba0 con 0x7f82081538f0 2023-10-30T13:04:38.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.616+0000 7f820d56f640 1 --2- 172.21.15.125:0/1766288939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82081538f0 0x7f8208159fb0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f8204009870 tx=0x7f820401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25fd7517ae2f898f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:38.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.616+0000 7f81fffff640 1 -- 172.21.15.125:0/1766288939 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82040a8a30 con 0x7f82081538f0 2023-10-30T13:04:38.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.616+0000 7f81fffff640 1 -- 172.21.15.125:0/1766288939 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82040a8b90 con 0x7f82081538f0 2023-10-30T13:04:38.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.616+0000 7f81fffff640 1 -- 172.21.15.125:0/1766288939 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82040b0440 con 0x7f82081538f0 2023-10-30T13:04:38.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.616+0000 7f820e571640 1 -- 172.21.15.125:0/1766288939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82081538f0 msgr2=0x7f8208159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:38.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.616+0000 7f820e571640 1 --2- 172.21.15.125:0/1766288939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82081538f0 0x7f8208159fb0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f8204009870 tx=0x7f820401ed30 comp rx=0 tx=0).stop 2023-10-30T13:04:38.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.616+0000 7f820e571640 1 -- 172.21.15.125:0/1766288939 shutdown_connections 2023-10-30T13:04:38.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.616+0000 7f820e571640 1 --2- 172.21.15.125:0/1766288939 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f820815cd50 0x7f82080c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:38.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.616+0000 7f820e571640 1 --2- 172.21.15.125:0/1766288939 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82081538f0 0x7f8208159fb0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:38.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.616+0000 7f820e571640 1 -- 172.21.15.125:0/1766288939 >> 172.21.15.125:0/1766288939 conn(0x7f820806a640 msgr2=0x7f8208065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:38.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.616+0000 7f820e571640 1 -- 172.21.15.125:0/1766288939 shutdown_connections 2023-10-30T13:04:38.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.616+0000 7f820e571640 1 -- 172.21.15.125:0/1766288939 wait complete. 2023-10-30T13:04:38.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820e571640 1 Processor -- start 2023-10-30T13:04:38.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820e571640 1 -- start start 2023-10-30T13:04:38.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820e571640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82080cefa0 0x7f82080b3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:38.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820e571640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82081538f0 0x7f82081efb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:38.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820e571640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f820815cd50 0x7f8208147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:38.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820e571640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82081f3480 con 0x7f82080cefa0 2023-10-30T13:04:38.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820e571640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82081ef0d0 con 0x7f820815cd50 2023-10-30T13:04:38.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820e571640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82081e6410 con 0x7f82081538f0 2023-10-30T13:04:38.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820d56f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82080cefa0 0x7f82080b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:38.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820d56f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82080cefa0 0x7f82080b3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37882/0 (socket says 172.21.15.125:37882) 2023-10-30T13:04:38.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820d56f640 1 -- 172.21.15.125:0/3437095414 learned_addr learned my addr 172.21.15.125:0/3437095414 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:38.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820cd6e640 1 --2- 172.21.15.125:0/3437095414 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82081538f0 0x7f82081efb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:38.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820dd70640 1 --2- 172.21.15.125:0/3437095414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f820815cd50 0x7f8208147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:38.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820d56f640 1 -- 172.21.15.125:0/3437095414 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82081538f0 msgr2=0x7f82081efb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:38.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820d56f640 1 --2- 172.21.15.125:0/3437095414 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82081538f0 0x7f82081efb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:38.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820d56f640 1 -- 172.21.15.125:0/3437095414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f820815cd50 msgr2=0x7f8208147bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:38.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820d56f640 1 --2- 172.21.15.125:0/3437095414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f820815cd50 0x7f8208147bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:38.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820d56f640 1 -- 172.21.15.125:0/3437095414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82040094d0 con 0x7f82080cefa0 2023-10-30T13:04:38.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820d56f640 1 --2- 172.21.15.125:0/3437095414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82080cefa0 0x7f82080b3050 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f82040b3040 tx=0x7f820401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:38.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f81fdffb640 1 -- 172.21.15.125:0/3437095414 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82040b0dd0 con 0x7f82080cefa0 2023-10-30T13:04:38.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f81fdffb640 1 -- 172.21.15.125:0/3437095414 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82040bb460 con 0x7f82080cefa0 2023-10-30T13:04:38.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f81fdffb640 1 -- 172.21.15.125:0/3437095414 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82040bb810 con 0x7f82080cefa0 2023-10-30T13:04:38.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820e571640 1 -- 172.21.15.125:0/3437095414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82081e66f0 con 0x7f82080cefa0 2023-10-30T13:04:38.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820e571640 1 -- 172.21.15.125:0/3437095414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82081ef390 con 0x7f82080cefa0 2023-10-30T13:04:38.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.620+0000 7f820e571640 1 -- 172.21.15.125:0/3437095414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f820804e170 con 0x7f82080cefa0 2023-10-30T13:04:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.624+0000 7f81fdffb640 1 -- 172.21.15.125:0/3437095414 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82040b83f0 con 0x7f82080cefa0 2023-10-30T13:04:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.624+0000 7f81fdffb640 1 --2- 172.21.15.125:0/3437095414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f81ec0664a0 0x7f81ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.624+0000 7f81fdffb640 1 -- 172.21.15.125:0/3437095414 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f8204121da0 con 0x7f82080cefa0 2023-10-30T13:04:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.624+0000 7f820cd6e640 1 --2- 172.21.15.125:0/3437095414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f81ec0664a0 0x7f81ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:38.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.624+0000 7f820cd6e640 1 --2- 172.21.15.125:0/3437095414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f81ec0664a0 0x7f81ec068960 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8208158e70 tx=0x7f81f8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:38.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.628+0000 7f81fdffb640 1 -- 172.21.15.125:0/3437095414 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82040eebe0 con 0x7f82080cefa0 2023-10-30T13:04:38.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.812+0000 7f820e571640 1 -- 172.21.15.125:0/3437095414 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8208159a60 con 0x7f82080cefa0 2023-10-30T13:04:38.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.812+0000 7f81fdffb640 1 -- 172.21.15.125:0/3437095414 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f82040b5090 con 0x7f82080cefa0 2023-10-30T13:04:38.816 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:04:38.817 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:57.950518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:active","state_seq":30,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:04:38.817 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 155 2023-10-30T13:04:38.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.812+0000 7f820e571640 1 -- 172.21.15.125:0/3437095414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f81ec0664a0 msgr2=0x7f81ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:38.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.812+0000 7f820e571640 1 --2- 172.21.15.125:0/3437095414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f81ec0664a0 0x7f81ec068960 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f8208158e70 tx=0x7f81f8008040 comp rx=0 tx=0).stop 2023-10-30T13:04:38.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.812+0000 7f820e571640 1 -- 172.21.15.125:0/3437095414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82080cefa0 msgr2=0x7f82080b3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:38.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.812+0000 7f820e571640 1 --2- 172.21.15.125:0/3437095414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82080cefa0 0x7f82080b3050 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f82040b3040 tx=0x7f820401bad0 comp rx=0 tx=0).stop 2023-10-30T13:04:38.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.812+0000 7f820e571640 1 -- 172.21.15.125:0/3437095414 shutdown_connections 2023-10-30T13:04:38.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.812+0000 7f820e571640 1 --2- 172.21.15.125:0/3437095414 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f81ec0664a0 0x7f81ec068960 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:38.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.812+0000 7f820e571640 1 --2- 172.21.15.125:0/3437095414 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f820815cd50 0x7f8208147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:38.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.812+0000 7f820e571640 1 --2- 172.21.15.125:0/3437095414 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82081538f0 0x7f82081efb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:38.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.812+0000 7f820e571640 1 --2- 172.21.15.125:0/3437095414 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82080cefa0 0x7f82080b3050 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:38.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.812+0000 7f820e571640 1 -- 172.21.15.125:0/3437095414 >> 172.21.15.125:0/3437095414 conn(0x7f820806a640 msgr2=0x7f82080caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:38.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.812+0000 7f820e571640 1 -- 172.21.15.125:0/3437095414 shutdown_connections 2023-10-30T13:04:38.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:38.812+0000 7f820e571640 1 -- 172.21.15.125:0/3437095414 wait complete. 2023-10-30T13:04:38.837 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:04:40.839 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:04:40.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.980+0000 7ff951679640 1 Processor -- start 2023-10-30T13:04:40.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.980+0000 7ff951679640 1 -- start start 2023-10-30T13:04:40.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff951679640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff94c0c6b80 0x7ff94c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:40.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff951679640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff94c0c6f60 0x7ff94c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:40.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff951679640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff94c0c8430 0x7ff94c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:40.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff951679640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff94c0c0a50 con 0x7ff94c0c8430 2023-10-30T13:04:40.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff951679640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff94c06f6e0 con 0x7ff94c0c6b80 2023-10-30T13:04:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff951679640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c06f9c0 con 0x7ff94c0c6f60 2023-10-30T13:04:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff94b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff94c0c8430 0x7ff94c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff94b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff94c0c8430 0x7ff94c13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59116/0 (socket says 172.21.15.125:59116) 2023-10-30T13:04:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff94b7fe640 1 -- 172.21.15.125:0/774230548 learned_addr learned my addr 172.21.15.125:0/774230548 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:40.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff9427fc640 1 --2- 172.21.15.125:0/774230548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff94c0c6f60 0x7ff94c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff94affd640 1 --2- 172.21.15.125:0/774230548 >> v2:172.21.15.136:3300/0 conn(0x7ff94c0c6b80 0x7ff94c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff94b7fe640 1 -- 172.21.15.125:0/774230548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff94c0c6f60 msgr2=0x7ff94c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:04:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff94b7fe640 1 --2- 172.21.15.125:0/774230548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff94c0c6f60 0x7ff94c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:40.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff94b7fe640 1 -- 172.21.15.125:0/774230548 >> v2:172.21.15.136:3300/0 conn(0x7ff94c0c6b80 msgr2=0x7ff94c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:04:40.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff94b7fe640 1 --2- 172.21.15.125:0/774230548 >> v2:172.21.15.136:3300/0 conn(0x7ff94c0c6b80 0x7ff94c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:40.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff94b7fe640 1 -- 172.21.15.125:0/774230548 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94c06fca0 con 0x7ff94c0c8430 2023-10-30T13:04:40.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff94affd640 1 --2- 172.21.15.125:0/774230548 >> v2:172.21.15.136:3300/0 conn(0x7ff94c0c6b80 0x7ff94c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:04:40.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.984+0000 7ff94b7fe640 1 --2- 172.21.15.125:0/774230548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff94c0c8430 0x7ff94c13ed90 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7ff94c06f1b0 tx=0x7ff94401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be554353261c0110 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:40.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff94a7fc640 1 -- 172.21.15.125:0/774230548 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9440a56d0 con 0x7ff94c0c8430 2023-10-30T13:04:40.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff94a7fc640 1 -- 172.21.15.125:0/774230548 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9440a5830 con 0x7ff94c0c8430 2023-10-30T13:04:40.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff94a7fc640 1 -- 172.21.15.125:0/774230548 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9440a5bc0 con 0x7ff94c0c8430 2023-10-30T13:04:40.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 -- 172.21.15.125:0/774230548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff94c0c8430 msgr2=0x7ff94c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:40.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 --2- 172.21.15.125:0/774230548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff94c0c8430 0x7ff94c13ed90 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7ff94c06f1b0 tx=0x7ff94401c940 comp rx=0 tx=0).stop 2023-10-30T13:04:40.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 -- 172.21.15.125:0/774230548 shutdown_connections 2023-10-30T13:04:40.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 --2- 172.21.15.125:0/774230548 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff94c0c8430 0x7ff94c13ed90 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:40.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 --2- 172.21.15.125:0/774230548 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff94c0c6f60 0x7ff94c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:40.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 --2- 172.21.15.125:0/774230548 >> v2:172.21.15.136:3300/0 conn(0x7ff94c0c6b80 0x7ff94c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:40.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 -- 172.21.15.125:0/774230548 >> 172.21.15.125:0/774230548 conn(0x7ff94c0c7910 msgr2=0x7ff94c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:40.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 -- 172.21.15.125:0/774230548 shutdown_connections 2023-10-30T13:04:40.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 -- 172.21.15.125:0/774230548 wait complete. 2023-10-30T13:04:40.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 Processor -- start 2023-10-30T13:04:40.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 -- start start 2023-10-30T13:04:40.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff94c0c6b80 0x7ff94c1349a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:40.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff94c0c6f60 0x7ff94c135e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:40.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff94c0c8430 0x7ff94c136370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:40.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c133f50 con 0x7ff94c0c8430 2023-10-30T13:04:40.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c134480 con 0x7ff94c0c6b80 2023-10-30T13:04:40.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff951679640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94c06c110 con 0x7ff94c0c6f60 2023-10-30T13:04:40.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff94b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff94c0c8430 0x7ff94c136370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:40.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.988+0000 7ff94b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff94c0c8430 0x7ff94c136370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59120/0 (socket says 172.21.15.125:59120) 2023-10-30T13:04:40.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff94b7fe640 1 -- 172.21.15.125:0/206808327 learned_addr learned my addr 172.21.15.125:0/206808327 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:40.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff9427fc640 1 --2- 172.21.15.125:0/206808327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff94c0c6f60 0x7ff94c135e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:40.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff94affd640 1 --2- 172.21.15.125:0/206808327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff94c0c6b80 0x7ff94c1349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:40.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff94b7fe640 1 -- 172.21.15.125:0/206808327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff94c0c6f60 msgr2=0x7ff94c135e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:41.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff94b7fe640 1 --2- 172.21.15.125:0/206808327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff94c0c6f60 0x7ff94c135e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:41.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff94b7fe640 1 -- 172.21.15.125:0/206808327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff94c0c6b80 msgr2=0x7ff94c1349a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:41.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff94b7fe640 1 --2- 172.21.15.125:0/206808327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff94c0c6b80 0x7ff94c1349a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:41.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff94b7fe640 1 -- 172.21.15.125:0/206808327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff944009b30 con 0x7ff94c0c8430 2023-10-30T13:04:41.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff94b7fe640 1 --2- 172.21.15.125:0/206808327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff94c0c8430 0x7ff94c136370 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7ff94401d1e0 tx=0x7ff9440196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:41.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff943fff640 1 -- 172.21.15.125:0/206808327 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9440b8880 con 0x7ff94c0c8430 2023-10-30T13:04:41.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff943fff640 1 -- 172.21.15.125:0/206808327 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9440b8a00 con 0x7ff94c0c8430 2023-10-30T13:04:41.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff943fff640 1 -- 172.21.15.125:0/206808327 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9440b8db0 con 0x7ff94c0c8430 2023-10-30T13:04:41.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff951679640 1 -- 172.21.15.125:0/206808327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff94c1341a0 con 0x7ff94c0c8430 2023-10-30T13:04:41.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff951679640 1 -- 172.21.15.125:0/206808327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94c135910 con 0x7ff94c0c8430 2023-10-30T13:04:41.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff951679640 1 -- 172.21.15.125:0/206808327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff94c04e170 con 0x7ff94c0c8430 2023-10-30T13:04:41.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff943fff640 1 -- 172.21.15.125:0/206808327 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff9440b6440 con 0x7ff94c0c8430 2023-10-30T13:04:41.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff943fff640 1 --2- 172.21.15.125:0/206808327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff92c0664d0 0x7ff92c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:41.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.992+0000 7ff943fff640 1 -- 172.21.15.125:0/206808327 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7ff94411fcf0 con 0x7ff94c0c8430 2023-10-30T13:04:41.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.996+0000 7ff94affd640 1 --2- 172.21.15.125:0/206808327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff92c0664d0 0x7ff92c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:41.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:40.996+0000 7ff94affd640 1 --2- 172.21.15.125:0/206808327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff92c0664d0 0x7ff92c068990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff93400c880 tx=0x7ff934008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:41.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.000+0000 7ff943fff640 1 -- 172.21.15.125:0/206808327 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff9440ecd00 con 0x7ff94c0c8430 2023-10-30T13:04:41.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.176+0000 7ff951679640 1 -- 172.21.15.125:0/206808327 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff94c06ae60 con 0x7ff94c0c8430 2023-10-30T13:04:41.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.180+0000 7ff943fff640 1 -- 172.21.15.125:0/206808327 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+6519 (secure 0 0 0) 0x7ff944009510 con 0x7ff94c0c8430 2023-10-30T13:04:41.183 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:04:41.183 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:57.950518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:active","state_seq":30,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:04:41.184 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 155 2023-10-30T13:04:41.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.180+0000 7ff951679640 1 -- 172.21.15.125:0/206808327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff92c0664d0 msgr2=0x7ff92c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:41.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.180+0000 7ff951679640 1 --2- 172.21.15.125:0/206808327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff92c0664d0 0x7ff92c068990 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7ff93400c880 tx=0x7ff934008040 comp rx=0 tx=0).stop 2023-10-30T13:04:41.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.180+0000 7ff951679640 1 -- 172.21.15.125:0/206808327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff94c0c8430 msgr2=0x7ff94c136370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:41.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.180+0000 7ff951679640 1 --2- 172.21.15.125:0/206808327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff94c0c8430 0x7ff94c136370 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7ff94401d1e0 tx=0x7ff9440196e0 comp rx=0 tx=0).stop 2023-10-30T13:04:41.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.180+0000 7ff951679640 1 -- 172.21.15.125:0/206808327 shutdown_connections 2023-10-30T13:04:41.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.180+0000 7ff951679640 1 --2- 172.21.15.125:0/206808327 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff92c0664d0 0x7ff92c068990 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:41.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.180+0000 7ff951679640 1 --2- 172.21.15.125:0/206808327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff94c0c8430 0x7ff94c136370 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:41.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.180+0000 7ff951679640 1 --2- 172.21.15.125:0/206808327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff94c0c6f60 0x7ff94c135e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:41.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.180+0000 7ff951679640 1 --2- 172.21.15.125:0/206808327 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff94c0c6b80 0x7ff94c1349a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:41.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.180+0000 7ff951679640 1 -- 172.21.15.125:0/206808327 >> 172.21.15.125:0/206808327 conn(0x7ff94c0c7910 msgr2=0x7ff94c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:41.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.180+0000 7ff951679640 1 -- 172.21.15.125:0/206808327 shutdown_connections 2023-10-30T13:04:41.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:41.180+0000 7ff951679640 1 -- 172.21.15.125:0/206808327 wait complete. 2023-10-30T13:04:41.205 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:04:43.206 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:04:43.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.348+0000 7f4f71c46640 1 Processor -- start 2023-10-30T13:04:43.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.348+0000 7f4f71c46640 1 -- start start 2023-10-30T13:04:43.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.348+0000 7f4f71c46640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4f6c0c6b80 0x7f4f6c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:43.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.348+0000 7f4f71c46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c0c0a50 con 0x7f4f6c13f4a0 2023-10-30T13:04:43.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.348+0000 7f4f71c46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f6c06f6e0 con 0x7f4f6c0c6f60 2023-10-30T13:04:43.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.348+0000 7f4f71c46640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f6c06f9c0 con 0x7f4f6c0c6b80 2023-10-30T13:04:43.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.348+0000 7f4f6bfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4f6c13f4a0 0x7f4f6c1566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59498/0 (socket says 172.21.15.125:59498) 2023-10-30T13:04:43.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.348+0000 7f4f6bfff640 1 -- 172.21.15.125:0/2910567582 learned_addr learned my addr 172.21.15.125:0/2910567582 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:04:43.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.348+0000 7f4f6b7fe640 1 --2- 172.21.15.125:0/2910567582 >> v2:172.21.15.136:3300/0 conn(0x7f4f6c0c6b80 0x7f4f6c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:43.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.352+0000 7f4f6a7fc640 1 -- 172.21.15.125:0/2910567582 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177509284 0 0) 0x7f4f6c0c0a50 con 0x7f4f6c13f4a0 2023-10-30T13:04:43.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.352+0000 7f4f6a7fc640 1 -- 172.21.15.125:0/2910567582 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f54005740 con 0x7f4f6c13f4a0 2023-10-30T13:04:43.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.352+0000 7f4f6b7fe640 1 -- 172.21.15.125:0/2910567582 >> v1:172.21.15.136:6789/0 conn(0x7f4f6c0c6f60 legacy=0x7f4f6c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:43.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.352+0000 7f4f6b7fe640 1 -- 172.21.15.125:0/2910567582 >> v1:172.21.15.125:6789/0 conn(0x7f4f6c13f4a0 legacy=0x7f4f6c1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:43.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.352+0000 7f4f6b7fe640 1 -- 172.21.15.125:0/2910567582 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6c06fca0 con 0x7f4f6c0c6b80 2023-10-30T13:04:43.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.352+0000 7f4f6b7fe640 1 --2- 172.21.15.125:0/2910567582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f6c0c6b80 0x7f4f6c0bfec0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0097c0 tx=0x7f4f5c01c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39945eae9157684d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:43.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f6a7fc640 1 -- 172.21.15.125:0/2910567582 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c0ab600 con 0x7f4f6c0c6b80 2023-10-30T13:04:43.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f6a7fc640 1 -- 172.21.15.125:0/2910567582 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f5c0a7050 con 0x7f4f6c0c6b80 2023-10-30T13:04:43.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f6a7fc640 1 -- 172.21.15.125:0/2910567582 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f5c0ab960 con 0x7f4f6c0c6b80 2023-10-30T13:04:43.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 -- 172.21.15.125:0/2910567582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f6c0c6b80 msgr2=0x7f4f6c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:43.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 --2- 172.21.15.125:0/2910567582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f6c0c6b80 0x7f4f6c0bfec0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0097c0 tx=0x7f4f5c01c480 comp rx=0 tx=0).stop 2023-10-30T13:04:43.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 -- 172.21.15.125:0/2910567582 shutdown_connections 2023-10-30T13:04:43.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 --2- 172.21.15.125:0/2910567582 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f6c0c6b80 0x7f4f6c0bfec0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:43.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 -- 172.21.15.125:0/2910567582 >> 172.21.15.125:0/2910567582 conn(0x7f4f6c0c7910 msgr2=0x7f4f6c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:43.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 -- 172.21.15.125:0/2910567582 shutdown_connections 2023-10-30T13:04:43.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 -- 172.21.15.125:0/2910567582 wait complete. 2023-10-30T13:04:43.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 Processor -- start 2023-10-30T13:04:43.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 -- start start 2023-10-30T13:04:43.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f6c0c6b80 0x7f4f6c158c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:43.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f6c0c6f60 0x7f4f6c13b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:43.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f6c13f4a0 0x7f4f6c1f4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:43.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c151780 con 0x7f4f6c0c6f60 2023-10-30T13:04:43.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c155d70 con 0x7f4f6c13f4a0 2023-10-30T13:04:43.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f71c46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f6c1554d0 con 0x7f4f6c0c6b80 2023-10-30T13:04:43.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.356+0000 7f4f6affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f6c0c6f60 0x7f4f6c13b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:43.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f6affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f6c0c6f60 0x7f4f6c13b7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59128/0 (socket says 172.21.15.125:59128) 2023-10-30T13:04:43.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f6affd640 1 -- 172.21.15.125:0/549477188 learned_addr learned my addr 172.21.15.125:0/549477188 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:43.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f6b7fe640 1 --2- 172.21.15.125:0/549477188 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f6c0c6b80 0x7f4f6c158c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:43.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f6bfff640 1 --2- 172.21.15.125:0/549477188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f6c13f4a0 0x7f4f6c1f4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:43.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f6affd640 1 -- 172.21.15.125:0/549477188 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f6c0c6b80 msgr2=0x7f4f6c158c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:43.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f6affd640 1 --2- 172.21.15.125:0/549477188 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f6c0c6b80 0x7f4f6c158c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:43.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f6affd640 1 -- 172.21.15.125:0/549477188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f6c13f4a0 msgr2=0x7f4f6c1f4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:43.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f6affd640 1 --2- 172.21.15.125:0/549477188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f6c13f4a0 0x7f4f6c1f4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:43.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f6affd640 1 -- 172.21.15.125:0/549477188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5c009420 con 0x7f4f6c0c6f60 2023-10-30T13:04:43.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f6affd640 1 --2- 172.21.15.125:0/549477188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f6c0c6f60 0x7f4f6c13b7b0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000c500 tx=0x7f4f6000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:43.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f53fff640 1 -- 172.21.15.125:0/549477188 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60005220 con 0x7f4f6c0c6f60 2023-10-30T13:04:43.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f53fff640 1 -- 172.21.15.125:0/549477188 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f60005380 con 0x7f4f6c0c6f60 2023-10-30T13:04:43.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f53fff640 1 -- 172.21.15.125:0/549477188 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60007270 con 0x7f4f6c0c6f60 2023-10-30T13:04:43.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f71c46640 1 -- 172.21.15.125:0/549477188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f6c1557b0 con 0x7f4f6c0c6f60 2023-10-30T13:04:43.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f71c46640 1 -- 172.21.15.125:0/549477188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f6c1f3b00 con 0x7f4f6c0c6f60 2023-10-30T13:04:43.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f71c46640 1 -- 172.21.15.125:0/549477188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f6c1f3960 con 0x7f4f6c0c6f60 2023-10-30T13:04:43.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f53fff640 1 -- 172.21.15.125:0/549477188 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4f600073d0 con 0x7f4f6c0c6f60 2023-10-30T13:04:43.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.360+0000 7f4f53fff640 1 --2- 172.21.15.125:0/549477188 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f580664d0 0x7f4f58068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:43.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.364+0000 7f4f53fff640 1 -- 172.21.15.125:0/549477188 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 7286+0+0 (secure 0 0 0) 0x7f4f60083d90 con 0x7f4f6c0c6f60 2023-10-30T13:04:43.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.364+0000 7f4f6b7fe640 1 --2- 172.21.15.125:0/549477188 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f580664d0 0x7f4f58068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:43.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.364+0000 7f4f6b7fe640 1 --2- 172.21.15.125:0/549477188 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f580664d0 0x7f4f58068990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0ad000 tx=0x7f4f5c0a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:43.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.364+0000 7f4f53fff640 1 -- 172.21.15.125:0/549477188 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4f6c1f3960 con 0x7f4f6c0c6f60 2023-10-30T13:04:43.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.552+0000 7f4f71c46640 1 -- 172.21.15.125:0/549477188 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f6c1f3960 con 0x7f4f6c0c6f60 2023-10-30T13:04:43.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.552+0000 7f4f53fff640 1 -- 172.21.15.125:0/549477188 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 155 v155) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f4f6c1f3960 con 0x7f4f6c0c6f60 2023-10-30T13:04:43.557 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:04:43.557 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":155,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8022,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":155}],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:02:57.950518+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7866},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_7866":{"gid":7866,"name":"e","rank":0,"incarnation":151,"state":"up:active","state_seq":30,"addr":"172.21.15.125:6836/1301336591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":1301336591},{"type":"v1","addr":"172.21.15.125:6836","nonce":1301336591}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:04:43.557 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 155 2023-10-30T13:04:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.552+0000 7f4f71c46640 1 -- 172.21.15.125:0/549477188 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f580664d0 msgr2=0x7f4f58068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.552+0000 7f4f71c46640 1 --2- 172.21.15.125:0/549477188 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f580664d0 0x7f4f58068990 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c0ad000 tx=0x7f4f5c0a6000 comp rx=0 tx=0).stop 2023-10-30T13:04:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.552+0000 7f4f71c46640 1 -- 172.21.15.125:0/549477188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f6c0c6f60 msgr2=0x7f4f6c13b7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.552+0000 7f4f71c46640 1 --2- 172.21.15.125:0/549477188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f6c0c6f60 0x7f4f6c13b7b0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f4f6000c500 tx=0x7f4f6000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:04:43.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.552+0000 7f4f71c46640 1 -- 172.21.15.125:0/549477188 shutdown_connections 2023-10-30T13:04:43.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.552+0000 7f4f71c46640 1 --2- 172.21.15.125:0/549477188 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f580664d0 0x7f4f58068990 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:43.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.552+0000 7f4f71c46640 1 --2- 172.21.15.125:0/549477188 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f6c13f4a0 0x7f4f6c1f4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:43.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.552+0000 7f4f71c46640 1 --2- 172.21.15.125:0/549477188 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f6c0c6f60 0x7f4f6c13b7b0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:43.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.552+0000 7f4f71c46640 1 --2- 172.21.15.125:0/549477188 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f6c0c6b80 0x7f4f6c158c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:43.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.552+0000 7f4f71c46640 1 -- 172.21.15.125:0/549477188 >> 172.21.15.125:0/549477188 conn(0x7f4f6c0c7910 msgr2=0x7f4f6c0b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.552+0000 7f4f71c46640 1 -- 172.21.15.125:0/549477188 shutdown_connections 2023-10-30T13:04:43.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:43.552+0000 7f4f71c46640 1 -- 172.21.15.125:0/549477188 wait complete. 2023-10-30T13:04:43.580 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:04:45.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:04:45.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.720+0000 7fb9ee66b640 1 Processor -- start 2023-10-30T13:04:45.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.720+0000 7fb9ee66b640 1 -- start start 2023-10-30T13:04:45.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.724+0000 7fb9ee66b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb9e81538a0 0x7fb9e815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:45.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.724+0000 7fb9ee66b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e80cb800 con 0x7fb9e80cf160 2023-10-30T13:04:45.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.724+0000 7fb9ee66b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9e815dc30 con 0x7fb9e80ca0e0 2023-10-30T13:04:45.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.724+0000 7fb9ee66b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9e80cb330 con 0x7fb9e81538a0 2023-10-30T13:04:45.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.724+0000 7fb9ecc08640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb9e80cf160 0x7fb9e8164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59500/0 (socket says 172.21.15.125:59500) 2023-10-30T13:04:45.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.724+0000 7fb9ecc08640 1 -- 172.21.15.125:0/2355192537 learned_addr learned my addr 172.21.15.125:0/2355192537 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:04:45.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.724+0000 7fb9e7fff640 1 --2- 172.21.15.125:0/2355192537 >> v2:172.21.15.125:3300/0 conn(0x7fb9e81538a0 0x7fb9e815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:45.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.724+0000 7fb9e6ffd640 1 -- 172.21.15.125:0/2355192537 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883012632 0 0) 0x7fb9e80cb800 con 0x7fb9e80cf160 2023-10-30T13:04:45.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.724+0000 7fb9e6ffd640 1 -- 172.21.15.125:0/2355192537 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d0003530 con 0x7fb9e80cf160 2023-10-30T13:04:45.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.724+0000 7fb9e7fff640 1 -- 172.21.15.125:0/2355192537 >> v1:172.21.15.136:6789/0 conn(0x7fb9e80ca0e0 legacy=0x7fb9e815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:45.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.724+0000 7fb9e7fff640 1 -- 172.21.15.125:0/2355192537 >> v1:172.21.15.125:6789/0 conn(0x7fb9e80cf160 legacy=0x7fb9e8164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:45.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.724+0000 7fb9e7fff640 1 -- 172.21.15.125:0/2355192537 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9e80cad70 con 0x7fb9e81538a0 2023-10-30T13:04:45.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9e7fff640 1 --2- 172.21.15.125:0/2355192537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9e81538a0 0x7fb9e815d6f0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fb9d80097c0 tx=0x7fb9d801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e1b124b2e305d45 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:45.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9e6ffd640 1 -- 172.21.15.125:0/2355192537 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d80a8970 con 0x7fb9e81538a0 2023-10-30T13:04:45.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9e6ffd640 1 -- 172.21.15.125:0/2355192537 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9d80a8ad0 con 0x7fb9e81538a0 2023-10-30T13:04:45.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9e6ffd640 1 -- 172.21.15.125:0/2355192537 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9d80a8e60 con 0x7fb9e81538a0 2023-10-30T13:04:45.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9ee66b640 1 -- 172.21.15.125:0/2355192537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9e81538a0 msgr2=0x7fb9e815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:45.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9ee66b640 1 --2- 172.21.15.125:0/2355192537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9e81538a0 0x7fb9e815d6f0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7fb9d80097c0 tx=0x7fb9d801ec70 comp rx=0 tx=0).stop 2023-10-30T13:04:45.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9ee66b640 1 -- 172.21.15.125:0/2355192537 shutdown_connections 2023-10-30T13:04:45.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9ee66b640 1 --2- 172.21.15.125:0/2355192537 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9e81538a0 0x7fb9e815d6f0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:45.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9ee66b640 1 -- 172.21.15.125:0/2355192537 >> 172.21.15.125:0/2355192537 conn(0x7fb9e806bfe0 msgr2=0x7fb9e806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:45.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9ee66b640 1 -- 172.21.15.125:0/2355192537 shutdown_connections 2023-10-30T13:04:45.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9ee66b640 1 -- 172.21.15.125:0/2355192537 wait complete. 2023-10-30T13:04:45.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9ee66b640 1 Processor -- start 2023-10-30T13:04:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9ee66b640 1 -- start start 2023-10-30T13:04:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9ee66b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9e80ca0e0 0x7fb9e80ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.728+0000 7fb9ee66b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9e80cf160 0x7fb9e8148f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9ee66b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9e81538a0 0x7fb9e81ef3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:45.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9ee66b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e8146720 con 0x7fb9e80cf160 2023-10-30T13:04:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9ee66b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e80b31a0 con 0x7fb9e81538a0 2023-10-30T13:04:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9ee66b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9e80b42d0 con 0x7fb9e80ca0e0 2023-10-30T13:04:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9e80cf160 0x7fb9e8148f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9e80cf160 0x7fb9e8148f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59152/0 (socket says 172.21.15.125:59152) 2023-10-30T13:04:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e77fe640 1 -- 172.21.15.125:0/515422007 learned_addr learned my addr 172.21.15.125:0/515422007 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:45.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e7fff640 1 --2- 172.21.15.125:0/515422007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9e80ca0e0 0x7fb9e80ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:45.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9ecc08640 1 --2- 172.21.15.125:0/515422007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9e81538a0 0x7fb9e81ef3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:45.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e77fe640 1 -- 172.21.15.125:0/515422007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9e80ca0e0 msgr2=0x7fb9e80ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:45.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e77fe640 1 --2- 172.21.15.125:0/515422007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9e80ca0e0 0x7fb9e80ceaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:45.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e77fe640 1 -- 172.21.15.125:0/515422007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9e81538a0 msgr2=0x7fb9e81ef3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:45.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e77fe640 1 --2- 172.21.15.125:0/515422007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9e81538a0 0x7fb9e81ef3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:45.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e77fe640 1 -- 172.21.15.125:0/515422007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d8009420 con 0x7fb9e80cf160 2023-10-30T13:04:45.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e77fe640 1 --2- 172.21.15.125:0/515422007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9e80cf160 0x7fb9e8148f60 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc00c3f0 tx=0x7fb9dc00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:45.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e4ff9640 1 -- 172.21.15.125:0/515422007 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc0052d0 con 0x7fb9e80cf160 2023-10-30T13:04:45.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e4ff9640 1 -- 172.21.15.125:0/515422007 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9dc005430 con 0x7fb9e80cf160 2023-10-30T13:04:45.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9ee66b640 1 -- 172.21.15.125:0/515422007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9e81f36b0 con 0x7fb9e80cf160 2023-10-30T13:04:45.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e4ff9640 1 -- 172.21.15.125:0/515422007 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9dc003840 con 0x7fb9e80cf160 2023-10-30T13:04:45.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9ee66b640 1 -- 172.21.15.125:0/515422007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9e80b33f0 con 0x7fb9e80cf160 2023-10-30T13:04:45.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9ee66b640 1 -- 172.21.15.125:0/515422007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9e804e170 con 0x7fb9e80cf160 2023-10-30T13:04:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e4ff9640 1 -- 172.21.15.125:0/515422007 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9dc0039a0 con 0x7fb9e80cf160 2023-10-30T13:04:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.732+0000 7fb9e4ff9640 1 --2- 172.21.15.125:0/515422007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9d40664d0 0x7fb9d4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.736+0000 7fb9e4ff9640 1 -- 172.21.15.125:0/515422007 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7fb9dc083960 con 0x7fb9e80cf160 2023-10-30T13:04:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.736+0000 7fb9e7fff640 1 --2- 172.21.15.125:0/515422007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9d40664d0 0x7fb9d4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.736+0000 7fb9e7fff640 1 --2- 172.21.15.125:0/515422007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9d40664d0 0x7fb9d4068990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fb9d80b2000 tx=0x7fb9d8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:45.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.736+0000 7fb9e4ff9640 1 -- 172.21.15.125:0/515422007 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9dc0506c0 con 0x7fb9e80cf160 2023-10-30T13:04:45.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.924+0000 7fb9ee66b640 1 -- 172.21.15.125:0/515422007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9e815dc30 con 0x7fb9e80cf160 2023-10-30T13:04:45.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.924+0000 7fb9e4ff9640 1 -- 172.21.15.125:0/515422007 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+5723 (secure 0 0 0) 0x7fb9e815dc30 con 0x7fb9e80cf160 2023-10-30T13:04:45.929 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:04:45.929 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:04:45.567546+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:04:45.930 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 156 2023-10-30T13:04:45.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.924+0000 7fb9ee66b640 1 -- 172.21.15.125:0/515422007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9d40664d0 msgr2=0x7fb9d4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:45.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.924+0000 7fb9ee66b640 1 --2- 172.21.15.125:0/515422007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9d40664d0 0x7fb9d4068990 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7fb9d80b2000 tx=0x7fb9d8008040 comp rx=0 tx=0).stop 2023-10-30T13:04:45.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.924+0000 7fb9ee66b640 1 -- 172.21.15.125:0/515422007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9e80cf160 msgr2=0x7fb9e8148f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:45.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.924+0000 7fb9ee66b640 1 --2- 172.21.15.125:0/515422007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9e80cf160 0x7fb9e8148f60 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7fb9dc00c3f0 tx=0x7fb9dc00c8b0 comp rx=0 tx=0).stop 2023-10-30T13:04:45.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.924+0000 7fb9ee66b640 1 -- 172.21.15.125:0/515422007 shutdown_connections 2023-10-30T13:04:45.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.924+0000 7fb9ee66b640 1 --2- 172.21.15.125:0/515422007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9d40664d0 0x7fb9d4068990 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:45.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.924+0000 7fb9ee66b640 1 --2- 172.21.15.125:0/515422007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9e81538a0 0x7fb9e81ef3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:45.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.924+0000 7fb9ee66b640 1 --2- 172.21.15.125:0/515422007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9e80cf160 0x7fb9e8148f60 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:45.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.924+0000 7fb9ee66b640 1 --2- 172.21.15.125:0/515422007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9e80ca0e0 0x7fb9e80ceaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:45.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.924+0000 7fb9ee66b640 1 -- 172.21.15.125:0/515422007 >> 172.21.15.125:0/515422007 conn(0x7fb9e806bfe0 msgr2=0x7fb9e80c5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:45.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.924+0000 7fb9ee66b640 1 -- 172.21.15.125:0/515422007 shutdown_connections 2023-10-30T13:04:45.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:45.924+0000 7fb9ee66b640 1 -- 172.21.15.125:0/515422007 wait complete. 2023-10-30T13:04:45.950 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T13:04:45.951 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:04:45.951 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:04:46.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f841973e640 1 Processor -- start 2023-10-30T13:04:46.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f841973e640 1 -- start start 2023-10-30T13:04:46.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f841973e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84140c6b80 0x7f84140bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:46.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f841973e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f841413f4a0 0x7f84140c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:46.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f841973e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84140c0a50 con 0x7f84140c6f60 2023-10-30T13:04:46.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f841973e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f841406f6e0 con 0x7f841413f4a0 2023-10-30T13:04:46.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f841973e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f841406f9c0 con 0x7f84140c6b80 2023-10-30T13:04:46.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f840bfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f84140c6f60 0x7f841413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59504/0 (socket says 172.21.15.125:59504) 2023-10-30T13:04:46.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f840bfff640 1 -- 172.21.15.125:0/4088036706 learned_addr learned my addr 172.21.15.125:0/4088036706 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:04:46.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f84137fe640 1 --2- 172.21.15.125:0/4088036706 >> v2:172.21.15.125:3300/0 conn(0x7f841413f4a0 0x7f84140c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:46.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f8412ffd640 1 --2- 172.21.15.125:0/4088036706 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84140c6b80 0x7f84140bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:46.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f84127fc640 1 -- 172.21.15.125:0/4088036706 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 618824596 0 0) 0x7f84140c0a50 con 0x7f84140c6f60 2023-10-30T13:04:46.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f84127fc640 1 -- 172.21.15.125:0/4088036706 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8400003530 con 0x7f84140c6f60 2023-10-30T13:04:46.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f84137fe640 1 -- 172.21.15.125:0/4088036706 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84140c6b80 msgr2=0x7f84140bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:04:46.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f84137fe640 1 --2- 172.21.15.125:0/4088036706 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84140c6b80 0x7f84140bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:46.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f84137fe640 1 -- 172.21.15.125:0/4088036706 >> v1:172.21.15.125:6789/0 conn(0x7f84140c6f60 legacy=0x7f841413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:46.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.084+0000 7f84137fe640 1 -- 172.21.15.125:0/4088036706 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841406fca0 con 0x7f841413f4a0 2023-10-30T13:04:46.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.088+0000 7f84137fe640 1 --2- 172.21.15.125:0/4088036706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f841413f4a0 0x7f84140c0400 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f840c009d90 tx=0x7f840c01c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e42362f2113f18 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:46.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.088+0000 7f84127fc640 1 -- 172.21.15.125:0/4088036706 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840c0a5920 con 0x7f841413f4a0 2023-10-30T13:04:46.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.088+0000 7f84127fc640 1 -- 172.21.15.125:0/4088036706 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840c0a5a80 con 0x7f841413f4a0 2023-10-30T13:04:46.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.088+0000 7f84127fc640 1 -- 172.21.15.125:0/4088036706 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840c0a5e10 con 0x7f841413f4a0 2023-10-30T13:04:46.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.088+0000 7f841973e640 1 -- 172.21.15.125:0/4088036706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f841413f4a0 msgr2=0x7f84140c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:46.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.088+0000 7f841973e640 1 --2- 172.21.15.125:0/4088036706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f841413f4a0 0x7f84140c0400 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f840c009d90 tx=0x7f840c01c950 comp rx=0 tx=0).stop 2023-10-30T13:04:46.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.088+0000 7f841973e640 1 -- 172.21.15.125:0/4088036706 shutdown_connections 2023-10-30T13:04:46.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.088+0000 7f841973e640 1 --2- 172.21.15.125:0/4088036706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f841413f4a0 0x7f84140c0400 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:46.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.088+0000 7f841973e640 1 --2- 172.21.15.125:0/4088036706 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84140c6b80 0x7f84140bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:46.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.088+0000 7f841973e640 1 -- 172.21.15.125:0/4088036706 >> 172.21.15.125:0/4088036706 conn(0x7f84140c7910 msgr2=0x7f84140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:46.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.088+0000 7f841973e640 1 -- 172.21.15.125:0/4088036706 shutdown_connections 2023-10-30T13:04:46.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.088+0000 7f841973e640 1 -- 172.21.15.125:0/4088036706 wait complete. 2023-10-30T13:04:46.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f841973e640 1 Processor -- start 2023-10-30T13:04:46.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f841973e640 1 -- start start 2023-10-30T13:04:46.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f841973e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84140c6b80 0x7f841415e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:46.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f841973e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84140c6f60 0x7f841415e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:46.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f841973e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f841413f4a0 0x7f84141380d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:46.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f841973e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8414154e40 con 0x7f84140c6f60 2023-10-30T13:04:46.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f841973e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f841415dff0 con 0x7f841413f4a0 2023-10-30T13:04:46.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f841973e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84140c22c0 con 0x7f84140c6b80 2023-10-30T13:04:46.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f840bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84140c6f60 0x7f841415e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:46.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f840bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84140c6f60 0x7f841415e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59164/0 (socket says 172.21.15.125:59164) 2023-10-30T13:04:46.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f840bfff640 1 -- 172.21.15.125:0/2250933286 learned_addr learned my addr 172.21.15.125:0/2250933286 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:46.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f8412ffd640 1 --2- 172.21.15.125:0/2250933286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84140c6b80 0x7f841415e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:46.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f84137fe640 1 --2- 172.21.15.125:0/2250933286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f841413f4a0 0x7f84141380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:46.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f840bfff640 1 -- 172.21.15.125:0/2250933286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84140c6b80 msgr2=0x7f841415e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:46.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f840bfff640 1 --2- 172.21.15.125:0/2250933286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84140c6b80 0x7f841415e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:46.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f840bfff640 1 -- 172.21.15.125:0/2250933286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f841413f4a0 msgr2=0x7f84141380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:46.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f840bfff640 1 --2- 172.21.15.125:0/2250933286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f841413f4a0 0x7f84141380d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:46.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f840bfff640 1 -- 172.21.15.125:0/2250933286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840c009a70 con 0x7f84140c6f60 2023-10-30T13:04:46.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f840bfff640 1 --2- 172.21.15.125:0/2250933286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84140c6f60 0x7f841415e980 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00c420 tx=0x7f83fc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:46.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f840b7fe640 1 -- 172.21.15.125:0/2250933286 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc005cf0 con 0x7f84140c6f60 2023-10-30T13:04:46.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f840b7fe640 1 -- 172.21.15.125:0/2250933286 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83fc005e70 con 0x7f84140c6f60 2023-10-30T13:04:46.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f840b7fe640 1 -- 172.21.15.125:0/2250933286 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83fc0072f0 con 0x7f84140c6f60 2023-10-30T13:04:46.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f841973e640 1 -- 172.21.15.125:0/2250933286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84141f6060 con 0x7f84140c6f60 2023-10-30T13:04:46.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.092+0000 7f841973e640 1 -- 172.21.15.125:0/2250933286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8414155310 con 0x7f84140c6f60 2023-10-30T13:04:46.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.096+0000 7f841973e640 1 -- 172.21.15.125:0/2250933286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f841415e160 con 0x7f84140c6f60 2023-10-30T13:04:46.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.096+0000 7f840b7fe640 1 -- 172.21.15.125:0/2250933286 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f83fc007450 con 0x7f84140c6f60 2023-10-30T13:04:46.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.096+0000 7f840b7fe640 1 --2- 172.21.15.125:0/2250933286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83f40664a0 0x7f83f4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:46.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.096+0000 7f840b7fe640 1 -- 172.21.15.125:0/2250933286 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f83fc083c30 con 0x7f84140c6f60 2023-10-30T13:04:46.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.096+0000 7f8412ffd640 1 --2- 172.21.15.125:0/2250933286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83f40664a0 0x7f83f4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:46.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.096+0000 7f8412ffd640 1 --2- 172.21.15.125:0/2250933286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83f40664a0 0x7f83f4068960 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8404009af0 tx=0x7f8404008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:46.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.100+0000 7f840b7fe640 1 -- 172.21.15.125:0/2250933286 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f841415e160 con 0x7f84140c6f60 2023-10-30T13:04:46.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.284+0000 7f841973e640 1 -- 172.21.15.125:0/2250933286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f841415e160 con 0x7f84140c6f60 2023-10-30T13:04:46.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.284+0000 7f840b7fe640 1 -- 172.21.15.125:0/2250933286 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f841415e160 con 0x7f84140c6f60 2023-10-30T13:04:46.289 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:04:46.290 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:04:45.567546+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:04:46.290 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 156 2023-10-30T13:04:46.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.284+0000 7f841973e640 1 -- 172.21.15.125:0/2250933286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83f40664a0 msgr2=0x7f83f4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:46.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.284+0000 7f841973e640 1 --2- 172.21.15.125:0/2250933286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83f40664a0 0x7f83f4068960 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8404009af0 tx=0x7f8404008040 comp rx=0 tx=0).stop 2023-10-30T13:04:46.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.284+0000 7f841973e640 1 -- 172.21.15.125:0/2250933286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84140c6f60 msgr2=0x7f841415e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:46.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.284+0000 7f841973e640 1 --2- 172.21.15.125:0/2250933286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84140c6f60 0x7f841415e980 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00c420 tx=0x7f83fc00c8e0 comp rx=0 tx=0).stop 2023-10-30T13:04:46.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.288+0000 7f841973e640 1 -- 172.21.15.125:0/2250933286 shutdown_connections 2023-10-30T13:04:46.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.288+0000 7f841973e640 1 --2- 172.21.15.125:0/2250933286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83f40664a0 0x7f83f4068960 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:46.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.288+0000 7f841973e640 1 --2- 172.21.15.125:0/2250933286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f841413f4a0 0x7f84141380d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:46.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.288+0000 7f841973e640 1 --2- 172.21.15.125:0/2250933286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f84140c6f60 0x7f841415e980 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:46.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.288+0000 7f841973e640 1 --2- 172.21.15.125:0/2250933286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f84140c6b80 0x7f841415e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:46.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.288+0000 7f841973e640 1 -- 172.21.15.125:0/2250933286 >> 172.21.15.125:0/2250933286 conn(0x7f84140c7910 msgr2=0x7f84140c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:46.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.288+0000 7f841973e640 1 -- 172.21.15.125:0/2250933286 shutdown_connections 2023-10-30T13:04:46.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:46.288+0000 7f841973e640 1 -- 172.21.15.125:0/2250933286 wait complete. 2023-10-30T13:04:46.311 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 156, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 156, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8022": { "addr": "172.21.15.125:6837/2196617378", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2196617378, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2196617378, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8022, "incarnation": 156, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:04:45.567546+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 8022 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:04:48.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:04:48.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.452+0000 7fb0c582f640 1 Processor -- start 2023-10-30T13:04:48.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.452+0000 7fb0c582f640 1 -- start start 2023-10-30T13:04:48.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c582f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb0c01538f0 0x7fb0c0159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:48.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c582f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c00ca6a0 con 0x7fb0c00cefa0 2023-10-30T13:04:48.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c582f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0c015d300 con 0x7fb0c015cd50 2023-10-30T13:04:48.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c582f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0c015d5e0 con 0x7fb0c01538f0 2023-10-30T13:04:48.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c502e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb0c00cefa0 0x7fb0c01640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59520/0 (socket says 172.21.15.125:59520) 2023-10-30T13:04:48.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c502e640 1 -- 172.21.15.125:0/3789942035 learned_addr learned my addr 172.21.15.125:0/3789942035 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:04:48.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c482d640 1 --2- 172.21.15.125:0/3789942035 >> v2:172.21.15.136:3300/0 conn(0x7fb0c01538f0 0x7fb0c0159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:48.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 -- 172.21.15.125:0/3789942035 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342770583 0 0) 0x7fb0c00ca6a0 con 0x7fb0c00cefa0 2023-10-30T13:04:48.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 -- 172.21.15.125:0/3789942035 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0b0003530 con 0x7fb0c00cefa0 2023-10-30T13:04:48.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 -- 172.21.15.125:0/3789942035 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3014213480 0 0) 0x7fb0b0003530 con 0x7fb0c00cefa0 2023-10-30T13:04:48.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 -- 172.21.15.125:0/3789942035 >> v2:172.21.15.136:3300/0 conn(0x7fb0c01538f0 msgr2=0x7fb0c0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:04:48.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 --2- 172.21.15.125:0/3789942035 >> v2:172.21.15.136:3300/0 conn(0x7fb0c01538f0 0x7fb0c0159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:48.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 -- 172.21.15.125:0/3789942035 >> v1:172.21.15.136:6789/0 conn(0x7fb0c015cd50 legacy=0x7fb0c00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:48.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 -- 172.21.15.125:0/3789942035 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c015d8c0 con 0x7fb0c00cefa0 2023-10-30T13:04:48.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 -- 172.21.15.125:0/3789942035 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fb0b8001a00 con 0x7fb0c00cefa0 2023-10-30T13:04:48.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 -- 172.21.15.125:0/3789942035 >> v1:172.21.15.125:6789/0 conn(0x7fb0c00cefa0 legacy=0x7fb0c01640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:48.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 --2- 172.21.15.125:0/3789942035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b0003980 0x7fb0b0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:48.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 --2- 172.21.15.125:0/3789942035 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0b00047d0 0x7fb0b0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:48.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 --2- 172.21.15.125:0/3789942035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0b0005190 0x7fb0b000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:48.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 -- 172.21.15.125:0/3789942035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c015d5e0 con 0x7fb0b0003980 2023-10-30T13:04:48.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 -- 172.21.15.125:0/3789942035 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b000c040 con 0x7fb0b0005190 2023-10-30T13:04:48.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bf7fe640 1 -- 172.21.15.125:0/3789942035 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0b000c200 con 0x7fb0b00047d0 2023-10-30T13:04:48.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c482d640 1 --2- 172.21.15.125:0/3789942035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b0003980 0x7fb0b0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:48.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c502e640 1 --2- 172.21.15.125:0/3789942035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0b0005190 0x7fb0b000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:48.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0bffff640 1 --2- 172.21.15.125:0/3789942035 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0b00047d0 0x7fb0b0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:48.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c482d640 1 -- 172.21.15.125:0/3789942035 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0b00047d0 msgr2=0x7fb0b0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:04:48.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c482d640 1 --2- 172.21.15.125:0/3789942035 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0b00047d0 0x7fb0b0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:48.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c482d640 1 -- 172.21.15.125:0/3789942035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0b0005190 msgr2=0x7fb0b000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:04:48.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c482d640 1 --2- 172.21.15.125:0/3789942035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0b0005190 0x7fb0b000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:48.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c482d640 1 -- 172.21.15.125:0/3789942035 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0b0003530 con 0x7fb0b0003980 2023-10-30T13:04:48.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c502e640 1 --2- 172.21.15.125:0/3789942035 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0b0005190 0x7fb0b000b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:04:48.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.456+0000 7fb0c502e640 1 -- 172.21.15.125:0/3789942035 reap_dead start 2023-10-30T13:04:48.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.460+0000 7fb0c482d640 1 --2- 172.21.15.125:0/3789942035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b0003980 0x7fb0b0003da0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009ae0 tx=0x7fb0b401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2bada4be7c61b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:48.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.460+0000 7fb0bf7fe640 1 -- 172.21.15.125:0/3789942035 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40a5640 con 0x7fb0b0003980 2023-10-30T13:04:48.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.460+0000 7fb0bf7fe640 1 -- 172.21.15.125:0/3789942035 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b40a57a0 con 0x7fb0b0003980 2023-10-30T13:04:48.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.460+0000 7fb0bf7fe640 1 -- 172.21.15.125:0/3789942035 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40a5b50 con 0x7fb0b0003980 2023-10-30T13:04:48.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.460+0000 7fb0c582f640 1 -- 172.21.15.125:0/3789942035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b0003980 msgr2=0x7fb0b0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:48.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.460+0000 7fb0c582f640 1 --2- 172.21.15.125:0/3789942035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b0003980 0x7fb0b0003da0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009ae0 tx=0x7fb0b401ca50 comp rx=0 tx=0).stop 2023-10-30T13:04:48.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.460+0000 7fb0c582f640 1 -- 172.21.15.125:0/3789942035 shutdown_connections 2023-10-30T13:04:48.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.460+0000 7fb0c582f640 1 --2- 172.21.15.125:0/3789942035 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b0003980 0x7fb0b0003da0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:48.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.460+0000 7fb0c582f640 1 -- 172.21.15.125:0/3789942035 >> 172.21.15.125:0/3789942035 conn(0x7fb0c006a640 msgr2=0x7fb0c0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:48.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.460+0000 7fb0c582f640 1 -- 172.21.15.125:0/3789942035 shutdown_connections 2023-10-30T13:04:48.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.460+0000 7fb0c582f640 1 -- 172.21.15.125:0/3789942035 wait complete. 2023-10-30T13:04:48.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c582f640 1 Processor -- start 2023-10-30T13:04:48.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c582f640 1 -- start start 2023-10-30T13:04:48.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c582f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b0003980 0x7fb0c01f0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:48.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c582f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0c0152710 0x7fb0c00b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:48.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c582f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0c01f33c0 0x7fb0c015bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:48.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c582f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c0153230 con 0x7fb0b0003980 2023-10-30T13:04:48.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c582f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c0143460 con 0x7fb0c0152710 2023-10-30T13:04:48.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c582f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c0152c70 con 0x7fb0c01f33c0 2023-10-30T13:04:48.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c482d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b0003980 0x7fb0c01f0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:48.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c482d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b0003980 0x7fb0c01f0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59190/0 (socket says 172.21.15.125:59190) 2023-10-30T13:04:48.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c482d640 1 -- 172.21.15.125:0/2438181250 learned_addr learned my addr 172.21.15.125:0/2438181250 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:48.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c502e640 1 --2- 172.21.15.125:0/2438181250 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0c01f33c0 0x7fb0c015bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:48.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0bffff640 1 --2- 172.21.15.125:0/2438181250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0c0152710 0x7fb0c00b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:48.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c482d640 1 -- 172.21.15.125:0/2438181250 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0c01f33c0 msgr2=0x7fb0c015bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:48.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c482d640 1 --2- 172.21.15.125:0/2438181250 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0c01f33c0 0x7fb0c015bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:48.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c482d640 1 -- 172.21.15.125:0/2438181250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0c0152710 msgr2=0x7fb0c00b4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:48.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c482d640 1 --2- 172.21.15.125:0/2438181250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0c0152710 0x7fb0c00b4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:48.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c482d640 1 -- 172.21.15.125:0/2438181250 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c0152f50 con 0x7fb0b0003980 2023-10-30T13:04:48.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c482d640 1 --2- 172.21.15.125:0/2438181250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b0003980 0x7fb0c01f0070 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fb0b401c880 tx=0x7fb0b40197f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:48.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0bd7fa640 1 -- 172.21.15.125:0/2438181250 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40bd070 con 0x7fb0b0003980 2023-10-30T13:04:48.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0bd7fa640 1 -- 172.21.15.125:0/2438181250 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0b4004020 con 0x7fb0b0003980 2023-10-30T13:04:48.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c582f640 1 -- 172.21.15.125:0/2438181250 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c01ef5b0 con 0x7fb0b0003980 2023-10-30T13:04:48.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0bd7fa640 1 -- 172.21.15.125:0/2438181250 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0b40043d0 con 0x7fb0b0003980 2023-10-30T13:04:48.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c582f640 1 -- 172.21.15.125:0/2438181250 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c01efb70 con 0x7fb0b0003980 2023-10-30T13:04:48.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.464+0000 7fb0c582f640 1 -- 172.21.15.125:0/2438181250 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c004e170 con 0x7fb0b0003980 2023-10-30T13:04:48.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.468+0000 7fb0bd7fa640 1 -- 172.21.15.125:0/2438181250 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb0b40b4070 con 0x7fb0b0003980 2023-10-30T13:04:48.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.468+0000 7fb0bd7fa640 1 --2- 172.21.15.125:0/2438181250 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0a40664a0 0x7fb0a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:48.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.468+0000 7fb0bd7fa640 1 -- 172.21.15.125:0/2438181250 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7fb0b411e260 con 0x7fb0b0003980 2023-10-30T13:04:48.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.468+0000 7fb0bffff640 1 --2- 172.21.15.125:0/2438181250 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0a40664a0 0x7fb0a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:48.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.468+0000 7fb0bffff640 1 --2- 172.21.15.125:0/2438181250 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0a40664a0 0x7fb0a4068960 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb0c015c4e0 tx=0x7fb0ac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:48.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.472+0000 7fb0bd7fa640 1 -- 172.21.15.125:0/2438181250 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb0b40eafc0 con 0x7fb0b0003980 2023-10-30T13:04:48.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.652+0000 7fb0c582f640 1 -- 172.21.15.125:0/2438181250 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0c015a6c0 con 0x7fb0b0003980 2023-10-30T13:04:48.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.652+0000 7fb0bd7fa640 1 -- 172.21.15.125:0/2438181250 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+5723 (secure 0 0 0) 0x7fb0b40ee5e0 con 0x7fb0b0003980 2023-10-30T13:04:48.658 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:04:48.658 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:04:45.567546+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:04:48.658 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 156 2023-10-30T13:04:48.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.656+0000 7fb0c582f640 1 -- 172.21.15.125:0/2438181250 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0a40664a0 msgr2=0x7fb0a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:48.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.656+0000 7fb0c582f640 1 --2- 172.21.15.125:0/2438181250 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0a40664a0 0x7fb0a4068960 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fb0c015c4e0 tx=0x7fb0ac009040 comp rx=0 tx=0).stop 2023-10-30T13:04:48.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.656+0000 7fb0c582f640 1 -- 172.21.15.125:0/2438181250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b0003980 msgr2=0x7fb0c01f0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:48.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.656+0000 7fb0c582f640 1 --2- 172.21.15.125:0/2438181250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b0003980 0x7fb0c01f0070 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7fb0b401c880 tx=0x7fb0b40197f0 comp rx=0 tx=0).stop 2023-10-30T13:04:48.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.656+0000 7fb0c582f640 1 -- 172.21.15.125:0/2438181250 shutdown_connections 2023-10-30T13:04:48.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.656+0000 7fb0c582f640 1 --2- 172.21.15.125:0/2438181250 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb0a40664a0 0x7fb0a4068960 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:48.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.656+0000 7fb0c582f640 1 --2- 172.21.15.125:0/2438181250 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb0c01f33c0 0x7fb0c015bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:48.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.656+0000 7fb0c582f640 1 --2- 172.21.15.125:0/2438181250 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb0c0152710 0x7fb0c00b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:48.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.656+0000 7fb0c582f640 1 --2- 172.21.15.125:0/2438181250 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb0b0003980 0x7fb0c01f0070 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:48.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.656+0000 7fb0c582f640 1 -- 172.21.15.125:0/2438181250 >> 172.21.15.125:0/2438181250 conn(0x7fb0c006a640 msgr2=0x7fb0c00c49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:48.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.656+0000 7fb0c582f640 1 -- 172.21.15.125:0/2438181250 shutdown_connections 2023-10-30T13:04:48.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:48.656+0000 7fb0c582f640 1 -- 172.21.15.125:0/2438181250 wait complete. 2023-10-30T13:04:48.679 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:04:50.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:04:50.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f223686b640 1 Processor -- start 2023-10-30T13:04:50.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f223686b640 1 -- start start 2023-10-30T13:04:50.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f223686b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22300c6b80 0x7f22300bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:50.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f223686b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22300c0a50 con 0x7f223013f4a0 2023-10-30T13:04:50.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f223686b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223006f6e0 con 0x7f22300c6f60 2023-10-30T13:04:50.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f223686b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f223006f9c0 con 0x7f22300c6b80 2023-10-30T13:04:50.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f2234e08640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f223013f4a0 0x7f22301566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34326/0 (socket says 172.21.15.125:34326) 2023-10-30T13:04:50.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f2234e08640 1 -- 172.21.15.125:0/2295048118 learned_addr learned my addr 172.21.15.125:0/2295048118 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:04:50.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f222ffff640 1 --2- 172.21.15.125:0/2295048118 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22300c6b80 0x7f22300bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:50.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f222effd640 1 -- 172.21.15.125:0/2295048118 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68861356 0 0) 0x7f22300c0a50 con 0x7f223013f4a0 2023-10-30T13:04:50.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f222effd640 1 -- 172.21.15.125:0/2295048118 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2218003530 con 0x7f223013f4a0 2023-10-30T13:04:50.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f222effd640 1 -- 172.21.15.125:0/2295048118 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380741530 0 0) 0x7f223006f6e0 con 0x7f22300c6f60 2023-10-30T13:04:50.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f222effd640 1 -- 172.21.15.125:0/2295048118 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22300c0a50 con 0x7f22300c6f60 2023-10-30T13:04:50.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f222ffff640 1 -- 172.21.15.125:0/2295048118 >> v1:172.21.15.136:6789/0 conn(0x7f22300c6f60 legacy=0x7f223013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:50.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f222effd640 1 -- 172.21.15.125:0/2295048118 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2026775283 0 0) 0x7f2218003530 con 0x7f223013f4a0 2023-10-30T13:04:50.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f222ffff640 1 -- 172.21.15.125:0/2295048118 >> v1:172.21.15.125:6789/0 conn(0x7f223013f4a0 legacy=0x7f22301566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:50.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.828+0000 7f222ffff640 1 -- 172.21.15.125:0/2295048118 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223006fca0 con 0x7f22300c6b80 2023-10-30T13:04:50.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.832+0000 7f222ffff640 1 --2- 172.21.15.125:0/2295048118 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22300c6b80 0x7f22300bfec0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f22200097f0 tx=0x7f222001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e9e819b9624bc91 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:50.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.832+0000 7f222effd640 1 -- 172.21.15.125:0/2295048118 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2228001a00 con 0x7f22300c6b80 2023-10-30T13:04:50.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.832+0000 7f222effd640 1 -- 172.21.15.125:0/2295048118 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22200a7050 con 0x7f22300c6b80 2023-10-30T13:04:50.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.832+0000 7f222effd640 1 -- 172.21.15.125:0/2295048118 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22200ab600 con 0x7f22300c6b80 2023-10-30T13:04:50.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.832+0000 7f223686b640 1 -- 172.21.15.125:0/2295048118 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22300c6b80 msgr2=0x7f22300bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:50.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.832+0000 7f223686b640 1 --2- 172.21.15.125:0/2295048118 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22300c6b80 0x7f22300bfec0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f22200097f0 tx=0x7f222001c3e0 comp rx=0 tx=0).stop 2023-10-30T13:04:50.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.832+0000 7f223686b640 1 -- 172.21.15.125:0/2295048118 shutdown_connections 2023-10-30T13:04:50.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.832+0000 7f223686b640 1 --2- 172.21.15.125:0/2295048118 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f22300c6b80 0x7f22300bfec0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:50.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.832+0000 7f223686b640 1 -- 172.21.15.125:0/2295048118 >> 172.21.15.125:0/2295048118 conn(0x7f22300c7910 msgr2=0x7f22300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:50.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.832+0000 7f223686b640 1 -- 172.21.15.125:0/2295048118 shutdown_connections 2023-10-30T13:04:50.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.832+0000 7f223686b640 1 -- 172.21.15.125:0/2295048118 wait complete. 2023-10-30T13:04:50.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.836+0000 7f223686b640 1 Processor -- start 2023-10-30T13:04:50.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.836+0000 7f223686b640 1 -- start start 2023-10-30T13:04:50.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f223686b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22300c6b80 0x7f22301545a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:50.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f223686b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22300c6f60 0x7f2230154ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:50.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f223686b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f223013f4a0 0x7f2230157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:50.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f223686b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2230137de0 con 0x7f22300c6b80 2023-10-30T13:04:50.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f223686b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2230159330 con 0x7f22300c6f60 2023-10-30T13:04:50.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f223686b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22301561c0 con 0x7f223013f4a0 2023-10-30T13:04:50.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f222ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22300c6b80 0x7f22301545a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:50.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f222ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22300c6b80 0x7f22301545a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37148/0 (socket says 172.21.15.125:37148) 2023-10-30T13:04:50.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f222ffff640 1 -- 172.21.15.125:0/211401339 learned_addr learned my addr 172.21.15.125:0/211401339 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:50.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f222f7fe640 1 --2- 172.21.15.125:0/211401339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22300c6f60 0x7f2230154ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:50.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f2234e08640 1 --2- 172.21.15.125:0/211401339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f223013f4a0 0x7f2230157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:50.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f222ffff640 1 -- 172.21.15.125:0/211401339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f223013f4a0 msgr2=0x7f2230157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:50.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f222ffff640 1 --2- 172.21.15.125:0/211401339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f223013f4a0 0x7f2230157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:50.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f222ffff640 1 -- 172.21.15.125:0/211401339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22300c6f60 msgr2=0x7f2230154ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:50.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f222ffff640 1 --2- 172.21.15.125:0/211401339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22300c6f60 0x7f2230154ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:50.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f222ffff640 1 -- 172.21.15.125:0/211401339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2220009420 con 0x7f22300c6b80 2023-10-30T13:04:50.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f222ffff640 1 --2- 172.21.15.125:0/211401339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22300c6b80 0x7f22301545a0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f222001a010 tx=0x7f2220019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:50.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f222cff9640 1 -- 172.21.15.125:0/211401339 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2220003ab0 con 0x7f22300c6b80 2023-10-30T13:04:50.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f222cff9640 1 -- 172.21.15.125:0/211401339 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2220003c10 con 0x7f22300c6b80 2023-10-30T13:04:50.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f222cff9640 1 -- 172.21.15.125:0/211401339 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22200ab400 con 0x7f22300c6b80 2023-10-30T13:04:50.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f223686b640 1 -- 172.21.15.125:0/211401339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22301351c0 con 0x7f22300c6b80 2023-10-30T13:04:50.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f223686b640 1 -- 172.21.15.125:0/211401339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2230159540 con 0x7f22300c6b80 2023-10-30T13:04:50.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f223686b640 1 -- 172.21.15.125:0/211401339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2230157790 con 0x7f22300c6b80 2023-10-30T13:04:50.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.840+0000 7f222cff9640 1 -- 172.21.15.125:0/211401339 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f22200b2040 con 0x7f22300c6b80 2023-10-30T13:04:50.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.844+0000 7f222cff9640 1 --2- 172.21.15.125:0/211401339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f221c0664a0 0x7f221c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:50.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.844+0000 7f222cff9640 1 -- 172.21.15.125:0/211401339 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f222011e1f0 con 0x7f22300c6b80 2023-10-30T13:04:50.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.844+0000 7f222f7fe640 1 --2- 172.21.15.125:0/211401339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f221c0664a0 0x7f221c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:50.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.844+0000 7f222f7fe640 1 --2- 172.21.15.125:0/211401339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f221c0664a0 0x7f221c068960 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f22300c6760 tx=0x7f2224009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:50.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:50.852+0000 7f222cff9640 1 -- 172.21.15.125:0/211401339 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f22200eb000 con 0x7f22300c6b80 2023-10-30T13:04:51.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:51.068+0000 7f223686b640 1 -- 172.21.15.125:0/211401339 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f223013a450 con 0x7f22300c6b80 2023-10-30T13:04:51.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:51.068+0000 7f222cff9640 1 -- 172.21.15.125:0/211401339 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f22200ee620 con 0x7f22300c6b80 2023-10-30T13:04:51.072 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:04:51.072 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:04:45.567546+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:04:51.073 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 156 2023-10-30T13:04:51.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:51.068+0000 7f223686b640 1 -- 172.21.15.125:0/211401339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f221c0664a0 msgr2=0x7f221c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:51.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:51.068+0000 7f223686b640 1 --2- 172.21.15.125:0/211401339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f221c0664a0 0x7f221c068960 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f22300c6760 tx=0x7f2224009040 comp rx=0 tx=0).stop 2023-10-30T13:04:51.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:51.068+0000 7f223686b640 1 -- 172.21.15.125:0/211401339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22300c6b80 msgr2=0x7f22301545a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:51.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:51.068+0000 7f223686b640 1 --2- 172.21.15.125:0/211401339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22300c6b80 0x7f22301545a0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f222001a010 tx=0x7f2220019180 comp rx=0 tx=0).stop 2023-10-30T13:04:51.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:51.068+0000 7f223686b640 1 -- 172.21.15.125:0/211401339 shutdown_connections 2023-10-30T13:04:51.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:51.068+0000 7f223686b640 1 --2- 172.21.15.125:0/211401339 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f221c0664a0 0x7f221c068960 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:51.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:51.068+0000 7f223686b640 1 --2- 172.21.15.125:0/211401339 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f223013f4a0 0x7f2230157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:51.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:51.068+0000 7f223686b640 1 --2- 172.21.15.125:0/211401339 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f22300c6f60 0x7f2230154ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:51.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:51.068+0000 7f223686b640 1 --2- 172.21.15.125:0/211401339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f22300c6b80 0x7f22301545a0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:51.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:51.068+0000 7f223686b640 1 -- 172.21.15.125:0/211401339 >> 172.21.15.125:0/211401339 conn(0x7f22300c7910 msgr2=0x7f22300c5810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:51.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:51.068+0000 7f223686b640 1 -- 172.21.15.125:0/211401339 shutdown_connections 2023-10-30T13:04:51.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:51.068+0000 7f223686b640 1 -- 172.21.15.125:0/211401339 wait complete. 2023-10-30T13:04:51.096 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:04:52.615 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:04:52.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:04:53.097 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:04:53.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.240+0000 7f95ea879640 1 Processor -- start 2023-10-30T13:04:53.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.240+0000 7f95ea879640 1 -- start start 2023-10-30T13:04:53.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.240+0000 7f95ea879640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95e415d6f0 0x7f95e415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:53.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.240+0000 7f95ea879640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f95e40cf160 0x7f95e40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:53.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.248+0000 7f95ea879640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95e40cb800 con 0x7f95e41538a0 2023-10-30T13:04:53.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.248+0000 7f95ea879640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95e40cb330 con 0x7f95e40cf160 2023-10-30T13:04:53.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.248+0000 7f95ea879640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e40cad70 con 0x7f95e415d6f0 2023-10-30T13:04:53.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.248+0000 7f95e3fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f95e41538a0 0x7f95e40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:51158/0 (socket says 172.21.15.125:51158) 2023-10-30T13:04:53.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.248+0000 7f95e3fff640 1 -- 172.21.15.125:0/3159162062 learned_addr learned my addr 172.21.15.125:0/3159162062 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:04:53.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.248+0000 7f95e8e16640 1 --2- 172.21.15.125:0/3159162062 >> v2:172.21.15.136:3300/0 conn(0x7f95e40cf160 0x7f95e40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:53.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.248+0000 7f95e37fe640 1 --2- 172.21.15.125:0/3159162062 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95e415d6f0 0x7f95e415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:53.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.248+0000 7f95e2ffd640 1 -- 172.21.15.125:0/3159162062 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089459755 0 0) 0x7f95e40cb800 con 0x7f95e41538a0 2023-10-30T13:04:53.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.248+0000 7f95e2ffd640 1 -- 172.21.15.125:0/3159162062 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95cc003530 con 0x7f95e41538a0 2023-10-30T13:04:53.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.248+0000 7f95e37fe640 1 -- 172.21.15.125:0/3159162062 >> v2:172.21.15.136:3300/0 conn(0x7f95e40cf160 msgr2=0x7f95e40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:04:53.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.248+0000 7f95e37fe640 1 --2- 172.21.15.125:0/3159162062 >> v2:172.21.15.136:3300/0 conn(0x7f95e40cf160 0x7f95e40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:53.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.248+0000 7f95e37fe640 1 -- 172.21.15.125:0/3159162062 >> v1:172.21.15.136:6789/0 conn(0x7f95e41538a0 legacy=0x7f95e40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:53.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.248+0000 7f95e37fe640 1 -- 172.21.15.125:0/3159162062 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e40cb050 con 0x7f95e415d6f0 2023-10-30T13:04:53.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.252+0000 7f95e37fe640 1 --2- 172.21.15.125:0/3159162062 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95e415d6f0 0x7f95e415a090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f95d80097c0 tx=0x7f95d801ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acf9d74e596826d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:53.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.252+0000 7f95e2ffd640 1 -- 172.21.15.125:0/3159162062 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d80a8970 con 0x7f95e415d6f0 2023-10-30T13:04:53.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.252+0000 7f95e2ffd640 1 -- 172.21.15.125:0/3159162062 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d80a8ad0 con 0x7f95e415d6f0 2023-10-30T13:04:53.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.252+0000 7f95e2ffd640 1 -- 172.21.15.125:0/3159162062 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d80a8e60 con 0x7f95e415d6f0 2023-10-30T13:04:53.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.252+0000 7f95ea879640 1 -- 172.21.15.125:0/3159162062 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95e415d6f0 msgr2=0x7f95e415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:53.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.252+0000 7f95ea879640 1 --2- 172.21.15.125:0/3159162062 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95e415d6f0 0x7f95e415a090 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f95d80097c0 tx=0x7f95d801ec70 comp rx=0 tx=0).stop 2023-10-30T13:04:53.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.252+0000 7f95ea879640 1 -- 172.21.15.125:0/3159162062 shutdown_connections 2023-10-30T13:04:53.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.252+0000 7f95ea879640 1 --2- 172.21.15.125:0/3159162062 >> v2:172.21.15.136:3300/0 conn(0x7f95e40cf160 0x7f95e40cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:53.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.252+0000 7f95ea879640 1 --2- 172.21.15.125:0/3159162062 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95e415d6f0 0x7f95e415a090 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:53.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.252+0000 7f95ea879640 1 -- 172.21.15.125:0/3159162062 >> 172.21.15.125:0/3159162062 conn(0x7f95e406bfe0 msgr2=0x7f95e406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:53.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.252+0000 7f95ea879640 1 -- 172.21.15.125:0/3159162062 shutdown_connections 2023-10-30T13:04:53.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.252+0000 7f95ea879640 1 -- 172.21.15.125:0/3159162062 wait complete. 2023-10-30T13:04:53.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95ea879640 1 Processor -- start 2023-10-30T13:04:53.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95ea879640 1 -- start start 2023-10-30T13:04:53.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95ea879640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95e40cf160 0x7f95e414f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:53.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95ea879640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95e41538a0 0x7f95e414e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:53.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95ea879640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95e415d6f0 0x7f95e40b3370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:53.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95ea879640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e40b41f0 con 0x7f95e41538a0 2023-10-30T13:04:53.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95ea879640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e414eb50 con 0x7f95e415d6f0 2023-10-30T13:04:53.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95ea879640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95e4150500 con 0x7f95e40cf160 2023-10-30T13:04:53.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95e37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95e41538a0 0x7f95e414e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:53.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95e37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95e41538a0 0x7f95e414e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37158/0 (socket says 172.21.15.125:37158) 2023-10-30T13:04:53.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95e37fe640 1 -- 172.21.15.125:0/1506275636 learned_addr learned my addr 172.21.15.125:0/1506275636 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:53.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95e3fff640 1 --2- 172.21.15.125:0/1506275636 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95e40cf160 0x7f95e414f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:53.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95e8e16640 1 --2- 172.21.15.125:0/1506275636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95e415d6f0 0x7f95e40b3370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:53.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95e37fe640 1 -- 172.21.15.125:0/1506275636 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95e40cf160 msgr2=0x7f95e414f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:53.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95e37fe640 1 --2- 172.21.15.125:0/1506275636 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95e40cf160 0x7f95e414f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:53.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95e37fe640 1 -- 172.21.15.125:0/1506275636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95e415d6f0 msgr2=0x7f95e40b3370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:53.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95e37fe640 1 --2- 172.21.15.125:0/1506275636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95e415d6f0 0x7f95e40b3370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:53.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95e37fe640 1 -- 172.21.15.125:0/1506275636 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d8009420 con 0x7f95e41538a0 2023-10-30T13:04:53.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.256+0000 7f95e37fe640 1 --2- 172.21.15.125:0/1506275636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95e41538a0 0x7f95e414e560 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f95d801d340 tx=0x7f95d801ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:53.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.260+0000 7f95e0ff9640 1 -- 172.21.15.125:0/1506275636 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d80b0d60 con 0x7f95e41538a0 2023-10-30T13:04:53.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.260+0000 7f95e0ff9640 1 -- 172.21.15.125:0/1506275636 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95d80bb440 con 0x7f95e41538a0 2023-10-30T13:04:53.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.260+0000 7f95e0ff9640 1 -- 172.21.15.125:0/1506275636 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95d80bb7f0 con 0x7f95e41538a0 2023-10-30T13:04:53.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.260+0000 7f95ea879640 1 -- 172.21.15.125:0/1506275636 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e41507e0 con 0x7f95e41538a0 2023-10-30T13:04:53.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.260+0000 7f95ea879640 1 -- 172.21.15.125:0/1506275636 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e40b3e80 con 0x7f95e41538a0 2023-10-30T13:04:53.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.260+0000 7f95ea879640 1 -- 172.21.15.125:0/1506275636 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e40b2a50 con 0x7f95e41538a0 2023-10-30T13:04:53.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.260+0000 7f95e0ff9640 1 -- 172.21.15.125:0/1506275636 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f95d80b7040 con 0x7f95e41538a0 2023-10-30T13:04:53.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.260+0000 7f95e0ff9640 1 --2- 172.21.15.125:0/1506275636 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f95d00664a0 0x7f95d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:53.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.260+0000 7f95e0ff9640 1 -- 172.21.15.125:0/1506275636 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f95d81223e0 con 0x7f95e41538a0 2023-10-30T13:04:53.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.260+0000 7f95e3fff640 1 --2- 172.21.15.125:0/1506275636 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f95d00664a0 0x7f95d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:53.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.264+0000 7f95e3fff640 1 --2- 172.21.15.125:0/1506275636 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f95d00664a0 0x7f95d0068960 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f95d40011f0 tx=0x7f95d4007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:53.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.264+0000 7f95e0ff9640 1 -- 172.21.15.125:0/1506275636 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f95d80ef310 con 0x7f95e41538a0 2023-10-30T13:04:53.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.488+0000 7f95ea879640 1 -- 172.21.15.125:0/1506275636 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95e40b43a0 con 0x7f95e41538a0 2023-10-30T13:04:53.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.488+0000 7f95e0ff9640 1 -- 172.21.15.125:0/1506275636 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f95e40b43a0 con 0x7f95e41538a0 2023-10-30T13:04:53.490 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:04:53.490 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:04:45.567546+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:04:53.491 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 156 2023-10-30T13:04:53.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.492+0000 7f95ea879640 1 -- 172.21.15.125:0/1506275636 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f95d00664a0 msgr2=0x7f95d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:53.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.492+0000 7f95ea879640 1 --2- 172.21.15.125:0/1506275636 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f95d00664a0 0x7f95d0068960 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f95d40011f0 tx=0x7f95d4007d40 comp rx=0 tx=0).stop 2023-10-30T13:04:53.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.492+0000 7f95ea879640 1 -- 172.21.15.125:0/1506275636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95e41538a0 msgr2=0x7f95e414e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:53.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.492+0000 7f95ea879640 1 --2- 172.21.15.125:0/1506275636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95e41538a0 0x7f95e414e560 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f95d801d340 tx=0x7f95d801ba30 comp rx=0 tx=0).stop 2023-10-30T13:04:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.492+0000 7f95ea879640 1 -- 172.21.15.125:0/1506275636 shutdown_connections 2023-10-30T13:04:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.492+0000 7f95ea879640 1 --2- 172.21.15.125:0/1506275636 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f95d00664a0 0x7f95d0068960 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.492+0000 7f95ea879640 1 --2- 172.21.15.125:0/1506275636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f95e415d6f0 0x7f95e40b3370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.492+0000 7f95ea879640 1 --2- 172.21.15.125:0/1506275636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f95e41538a0 0x7f95e414e560 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:53.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.492+0000 7f95ea879640 1 --2- 172.21.15.125:0/1506275636 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f95e40cf160 0x7f95e414f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:53.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.492+0000 7f95ea879640 1 -- 172.21.15.125:0/1506275636 >> 172.21.15.125:0/1506275636 conn(0x7f95e406bfe0 msgr2=0x7f95e40ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:53.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.492+0000 7f95ea879640 1 -- 172.21.15.125:0/1506275636 shutdown_connections 2023-10-30T13:04:53.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:53.492+0000 7f95ea879640 1 -- 172.21.15.125:0/1506275636 wait complete. 2023-10-30T13:04:53.518 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:04:55.519 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:04:55.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.664+0000 7f1c6a817640 1 Processor -- start 2023-10-30T13:04:55.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.664+0000 7f1c6a817640 1 -- start start 2023-10-30T13:04:55.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.668+0000 7f1c6a817640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c640c6bd0 0x7f1c6406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:55.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.668+0000 7f1c6a817640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1c64071110 0x7f1c64158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:55.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.668+0000 7f1c6a817640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c64066cf0 con 0x7f1c6406f7a0 2023-10-30T13:04:55.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.668+0000 7f1c6a817640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c640700c0 con 0x7f1c64071110 2023-10-30T13:04:55.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.668+0000 7f1c6a817640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c640c70d0 con 0x7f1c640c6bd0 2023-10-30T13:04:55.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.668+0000 7f1c637fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c640c6bd0 0x7f1c6406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:55.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.668+0000 7f1c637fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c640c6bd0 0x7f1c6406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:42466/0 (socket says 172.21.15.125:42466) 2023-10-30T13:04:55.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.668+0000 7f1c68db4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1c64071110 0x7f1c64158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:55.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.668+0000 7f1c637fe640 1 -- 172.21.15.125:0/1541606513 learned_addr learned my addr 172.21.15.125:0/1541606513 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:55.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.668+0000 7f1c637fe640 1 -- 172.21.15.125:0/1541606513 >> v2:172.21.15.136:3300/0 conn(0x7f1c64071110 msgr2=0x7f1c64158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:04:55.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.668+0000 7f1c637fe640 1 --2- 172.21.15.125:0/1541606513 >> v2:172.21.15.136:3300/0 conn(0x7f1c64071110 0x7f1c64158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:55.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.668+0000 7f1c637fe640 1 -- 172.21.15.125:0/1541606513 >> v1:172.21.15.136:6789/0 conn(0x7f1c6406f7a0 legacy=0x7f1c64069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:55.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.668+0000 7f1c637fe640 1 -- 172.21.15.125:0/1541606513 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640c7390 con 0x7f1c640c6bd0 2023-10-30T13:04:55.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.668+0000 7f1c68db4640 1 --2- 172.21.15.125:0/1541606513 >> v2:172.21.15.136:3300/0 conn(0x7f1c64071110 0x7f1c64158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:04:55.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.672+0000 7f1c637fe640 1 --2- 172.21.15.125:0/1541606513 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c640c6bd0 0x7f1c6406fb80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1c58009870 tx=0x7f1c5801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41b119dd9d6f1c82 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:55.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.672+0000 7f1c62ffd640 1 -- 172.21.15.125:0/1541606513 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c580a8a30 con 0x7f1c640c6bd0 2023-10-30T13:04:55.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.672+0000 7f1c62ffd640 1 -- 172.21.15.125:0/1541606513 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c580a8b90 con 0x7f1c640c6bd0 2023-10-30T13:04:55.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.672+0000 7f1c62ffd640 1 -- 172.21.15.125:0/1541606513 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c580b0440 con 0x7f1c640c6bd0 2023-10-30T13:04:55.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.672+0000 7f1c6a817640 1 -- 172.21.15.125:0/1541606513 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c640c6bd0 msgr2=0x7f1c6406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:55.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.672+0000 7f1c6a817640 1 --2- 172.21.15.125:0/1541606513 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c640c6bd0 0x7f1c6406fb80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1c58009870 tx=0x7f1c5801ed30 comp rx=0 tx=0).stop 2023-10-30T13:04:55.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.672+0000 7f1c6a817640 1 -- 172.21.15.125:0/1541606513 shutdown_connections 2023-10-30T13:04:55.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.672+0000 7f1c6a817640 1 --2- 172.21.15.125:0/1541606513 >> v2:172.21.15.136:3300/0 conn(0x7f1c64071110 0x7f1c64158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:55.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.672+0000 7f1c6a817640 1 --2- 172.21.15.125:0/1541606513 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c640c6bd0 0x7f1c6406fb80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:55.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.672+0000 7f1c6a817640 1 -- 172.21.15.125:0/1541606513 >> 172.21.15.125:0/1541606513 conn(0x7f1c640c7910 msgr2=0x7f1c640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:55.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.672+0000 7f1c6a817640 1 -- 172.21.15.125:0/1541606513 shutdown_connections 2023-10-30T13:04:55.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.672+0000 7f1c6a817640 1 -- 172.21.15.125:0/1541606513 wait complete. 2023-10-30T13:04:55.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.672+0000 7f1c6a817640 1 Processor -- start 2023-10-30T13:04:55.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c6a817640 1 -- start start 2023-10-30T13:04:55.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c6a817640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c6406f7a0 0x7f1c64155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:55.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c6a817640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c64071110 0x7f1c640c3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:55.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c6a817640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c640c6bd0 0x7f1c641f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:55.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c6a817640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c641565a0 con 0x7f1c640c6bd0 2023-10-30T13:04:55.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c6a817640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6414e1d0 con 0x7f1c64071110 2023-10-30T13:04:55.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c6a817640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6414f7b0 con 0x7f1c6406f7a0 2023-10-30T13:04:55.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c68db4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c640c6bd0 0x7f1c641f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:55.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c68db4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c640c6bd0 0x7f1c641f2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37164/0 (socket says 172.21.15.125:37164) 2023-10-30T13:04:55.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c68db4640 1 -- 172.21.15.125:0/1377971837 learned_addr learned my addr 172.21.15.125:0/1377971837 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:55.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c637fe640 1 --2- 172.21.15.125:0/1377971837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c64071110 0x7f1c640c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:55.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c63fff640 1 --2- 172.21.15.125:0/1377971837 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c6406f7a0 0x7f1c64155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:55.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c68db4640 1 -- 172.21.15.125:0/1377971837 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c6406f7a0 msgr2=0x7f1c64155af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:55.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c68db4640 1 --2- 172.21.15.125:0/1377971837 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c6406f7a0 0x7f1c64155af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:55.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c68db4640 1 -- 172.21.15.125:0/1377971837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c64071110 msgr2=0x7f1c640c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:55.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c68db4640 1 --2- 172.21.15.125:0/1377971837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c64071110 0x7f1c640c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:55.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c68db4640 1 -- 172.21.15.125:0/1377971837 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c009a70 con 0x7f1c640c6bd0 2023-10-30T13:04:55.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c68db4640 1 --2- 172.21.15.125:0/1377971837 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c640c6bd0 0x7f1c641f2470 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c00efc0 tx=0x7f1c5c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:55.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c60ff9640 1 -- 172.21.15.125:0/1377971837 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c009cd0 con 0x7f1c640c6bd0 2023-10-30T13:04:55.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c60ff9640 1 -- 172.21.15.125:0/1377971837 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5c00f040 con 0x7f1c640c6bd0 2023-10-30T13:04:55.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c60ff9640 1 -- 172.21.15.125:0/1377971837 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c0039a0 con 0x7f1c640c6bd0 2023-10-30T13:04:55.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c6a817640 1 -- 172.21.15.125:0/1377971837 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c580094d0 con 0x7f1c640c6bd0 2023-10-30T13:04:55.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c6a817640 1 -- 172.21.15.125:0/1377971837 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c641562c0 con 0x7f1c640c6bd0 2023-10-30T13:04:55.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.676+0000 7f1c6a817640 1 -- 172.21.15.125:0/1377971837 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6414e380 con 0x7f1c640c6bd0 2023-10-30T13:04:55.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.680+0000 7f1c60ff9640 1 -- 172.21.15.125:0/1377971837 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1c5c014480 con 0x7f1c640c6bd0 2023-10-30T13:04:55.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.680+0000 7f1c60ff9640 1 --2- 172.21.15.125:0/1377971837 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1c500664d0 0x7f1c50068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:55.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.680+0000 7f1c60ff9640 1 -- 172.21.15.125:0/1377971837 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f1c5c084fa0 con 0x7f1c640c6bd0 2023-10-30T13:04:55.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.680+0000 7f1c63fff640 1 --2- 172.21.15.125:0/1377971837 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1c500664d0 0x7f1c50068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:55.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.680+0000 7f1c63fff640 1 --2- 172.21.15.125:0/1377971837 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1c500664d0 0x7f1c50068990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1c54001770 tx=0x7f1c54009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:55.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.684+0000 7f1c60ff9640 1 -- 172.21.15.125:0/1377971837 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1c5c0b6090 con 0x7f1c640c6bd0 2023-10-30T13:04:55.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.864+0000 7f1c6a817640 1 -- 172.21.15.125:0/1377971837 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c6404e170 con 0x7f1c640c6bd0 2023-10-30T13:04:55.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.864+0000 7f1c60ff9640 1 -- 172.21.15.125:0/1377971837 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f1c5c018020 con 0x7f1c640c6bd0 2023-10-30T13:04:55.866 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:04:55.866 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:04:54.953323+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:04:55.867 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 157 2023-10-30T13:04:55.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.868+0000 7f1c6a817640 1 -- 172.21.15.125:0/1377971837 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1c500664d0 msgr2=0x7f1c50068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:55.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.868+0000 7f1c6a817640 1 --2- 172.21.15.125:0/1377971837 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1c500664d0 0x7f1c50068990 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f1c54001770 tx=0x7f1c54009040 comp rx=0 tx=0).stop 2023-10-30T13:04:55.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.868+0000 7f1c6a817640 1 -- 172.21.15.125:0/1377971837 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c640c6bd0 msgr2=0x7f1c641f2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:55.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.868+0000 7f1c6a817640 1 --2- 172.21.15.125:0/1377971837 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c640c6bd0 0x7f1c641f2470 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c00efc0 tx=0x7f1c5c00c490 comp rx=0 tx=0).stop 2023-10-30T13:04:55.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.868+0000 7f1c6a817640 1 -- 172.21.15.125:0/1377971837 shutdown_connections 2023-10-30T13:04:55.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.868+0000 7f1c6a817640 1 --2- 172.21.15.125:0/1377971837 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1c500664d0 0x7f1c50068990 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:55.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.868+0000 7f1c6a817640 1 --2- 172.21.15.125:0/1377971837 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1c640c6bd0 0x7f1c641f2470 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:55.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.868+0000 7f1c6a817640 1 --2- 172.21.15.125:0/1377971837 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c64071110 0x7f1c640c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:55.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.868+0000 7f1c6a817640 1 --2- 172.21.15.125:0/1377971837 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c6406f7a0 0x7f1c64155af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:55.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.868+0000 7f1c6a817640 1 -- 172.21.15.125:0/1377971837 >> 172.21.15.125:0/1377971837 conn(0x7f1c640c7910 msgr2=0x7f1c640c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:55.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.868+0000 7f1c6a817640 1 -- 172.21.15.125:0/1377971837 shutdown_connections 2023-10-30T13:04:55.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:55.868+0000 7f1c6a817640 1 -- 172.21.15.125:0/1377971837 wait complete. 2023-10-30T13:04:55.887 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:04:57.888 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:04:58.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.036+0000 7f8f5fe17640 1 Processor -- start 2023-10-30T13:04:58.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5fe17640 1 -- start start 2023-10-30T13:04:58.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5fe17640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8f5815d6f0 0x7f8f5815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:58.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5fe17640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f580cb800 con 0x7f8f580cf160 2023-10-30T13:04:58.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5fe17640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f580cb330 con 0x7f8f581538a0 2023-10-30T13:04:58.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5fe17640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f580cad70 con 0x7f8f5815d6f0 2023-10-30T13:04:58.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5e3b4640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8f580cf160 0x7f8f58164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34330/0 (socket says 172.21.15.125:34330) 2023-10-30T13:04:58.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5e3b4640 1 -- 172.21.15.125:0/3198704194 learned_addr learned my addr 172.21.15.125:0/3198704194 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:04:58.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5d3b2640 1 --2- 172.21.15.125:0/3198704194 >> v2:172.21.15.136:3300/0 conn(0x7f8f5815d6f0 0x7f8f5815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:58.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710334846 0 0) 0x7f8f580cb800 con 0x7f8f580cf160 2023-10-30T13:04:58.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f44003530 con 0x7f8f580cf160 2023-10-30T13:04:58.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395224781 0 0) 0x7f8f580cb330 con 0x7f8f581538a0 2023-10-30T13:04:58.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f580cb800 con 0x7f8f581538a0 2023-10-30T13:04:58.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 226946610 0 0) 0x7f8f44003530 con 0x7f8f580cf160 2023-10-30T13:04:58.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 >> v2:172.21.15.136:3300/0 conn(0x7f8f5815d6f0 msgr2=0x7f8f5815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:04:58.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 --2- 172.21.15.125:0/3198704194 >> v2:172.21.15.136:3300/0 conn(0x7f8f5815d6f0 0x7f8f5815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:58.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 >> v1:172.21.15.136:6789/0 conn(0x7f8f581538a0 legacy=0x7f8f580ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:58.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f580cb050 con 0x7f8f580cf160 2023-10-30T13:04:58.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f8f50001a00 con 0x7f8f580cf160 2023-10-30T13:04:58.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 >> v1:172.21.15.125:6789/0 conn(0x7f8f580cf160 legacy=0x7f8f58164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:58.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 --2- 172.21.15.125:0/3198704194 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f44003b10 0x7f8f44003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:58.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 --2- 172.21.15.125:0/3198704194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440047d0 0x7f8f44004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:58.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 --2- 172.21.15.125:0/3198704194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f44005190 0x7f8f4400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:58.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f580cad70 con 0x7f8f440047d0 2023-10-30T13:04:58.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4400bf80 con 0x7f8f44005190 2023-10-30T13:04:58.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4400c140 con 0x7f8f44003b10 2023-10-30T13:04:58.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5d3b2640 1 --2- 172.21.15.125:0/3198704194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440047d0 0x7f8f44004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:58.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5dbb3640 1 --2- 172.21.15.125:0/3198704194 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f44003b10 0x7f8f44003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:58.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5e3b4640 1 --2- 172.21.15.125:0/3198704194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f44005190 0x7f8f4400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:58.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5d3b2640 1 -- 172.21.15.125:0/3198704194 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f44003b10 msgr2=0x7f8f44003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:04:58.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5d3b2640 1 --2- 172.21.15.125:0/3198704194 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f44003b10 0x7f8f44003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:58.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5d3b2640 1 -- 172.21.15.125:0/3198704194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f44005190 msgr2=0x7f8f4400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:04:58.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5d3b2640 1 --2- 172.21.15.125:0/3198704194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f44005190 0x7f8f4400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:58.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5e3b4640 1 -- 172.21.15.125:0/3198704194 reap_dead start 2023-10-30T13:04:58.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.040+0000 7f8f5d3b2640 1 -- 172.21.15.125:0/3198704194 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f44003530 con 0x7f8f440047d0 2023-10-30T13:04:58.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.044+0000 7f8f5d3b2640 1 --2- 172.21.15.125:0/3198704194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440047d0 0x7f8f44004c50 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f8f48009ab0 tx=0x7f8f4801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9278f5d9bcc800ce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:58.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.044+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f480a5510 con 0x7f8f440047d0 2023-10-30T13:04:58.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.044+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f480a5670 con 0x7f8f440047d0 2023-10-30T13:04:58.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.044+0000 7f8f5cbb1640 1 -- 172.21.15.125:0/3198704194 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f480a5a20 con 0x7f8f440047d0 2023-10-30T13:04:58.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.044+0000 7f8f5fe17640 1 -- 172.21.15.125:0/3198704194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440047d0 msgr2=0x7f8f44004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:58.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.044+0000 7f8f5fe17640 1 --2- 172.21.15.125:0/3198704194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440047d0 0x7f8f44004c50 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7f8f48009ab0 tx=0x7f8f4801ca30 comp rx=0 tx=0).stop 2023-10-30T13:04:58.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.044+0000 7f8f5fe17640 1 -- 172.21.15.125:0/3198704194 shutdown_connections 2023-10-30T13:04:58.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.044+0000 7f8f5fe17640 1 --2- 172.21.15.125:0/3198704194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440047d0 0x7f8f44004c50 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:58.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.044+0000 7f8f5fe17640 1 -- 172.21.15.125:0/3198704194 >> 172.21.15.125:0/3198704194 conn(0x7f8f5806bfe0 msgr2=0x7f8f5806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:58.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.044+0000 7f8f5fe17640 1 -- 172.21.15.125:0/3198704194 shutdown_connections 2023-10-30T13:04:58.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.044+0000 7f8f5fe17640 1 -- 172.21.15.125:0/3198704194 wait complete. 2023-10-30T13:04:58.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5fe17640 1 Processor -- start 2023-10-30T13:04:58.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5fe17640 1 -- start start 2023-10-30T13:04:58.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5fe17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440047d0 0x7f8f58157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:58.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5fe17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f58146550 0x7f8f580b3310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:58.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5fe17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f5815c320 0x7f8f580b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:58.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5fe17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f58158220 con 0x7f8f440047d0 2023-10-30T13:04:58.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5fe17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f580b4b00 con 0x7f8f5815c320 2023-10-30T13:04:58.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5fe17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f58159520 con 0x7f8f58146550 2023-10-30T13:04:58.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5dbb3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440047d0 0x7f8f58157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:58.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5dbb3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440047d0 0x7f8f58157720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37180/0 (socket says 172.21.15.125:37180) 2023-10-30T13:04:58.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5dbb3640 1 -- 172.21.15.125:0/736356131 learned_addr learned my addr 172.21.15.125:0/736356131 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:04:58.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5d3b2640 1 --2- 172.21.15.125:0/736356131 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f58146550 0x7f8f580b3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:58.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5e3b4640 1 --2- 172.21.15.125:0/736356131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f5815c320 0x7f8f580b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:58.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5dbb3640 1 -- 172.21.15.125:0/736356131 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f58146550 msgr2=0x7f8f580b3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:58.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5dbb3640 1 --2- 172.21.15.125:0/736356131 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f58146550 0x7f8f580b3310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:58.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5dbb3640 1 -- 172.21.15.125:0/736356131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f5815c320 msgr2=0x7f8f580b3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:58.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5dbb3640 1 --2- 172.21.15.125:0/736356131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f5815c320 0x7f8f580b3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:58.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5dbb3640 1 -- 172.21.15.125:0/736356131 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f581479a0 con 0x7f8f440047d0 2023-10-30T13:04:58.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5dbb3640 1 --2- 172.21.15.125:0/736356131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440047d0 0x7f8f58157720 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400e5b0 tx=0x7f8f5400ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:58.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f4e7fc640 1 -- 172.21.15.125:0/736356131 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f540052e0 con 0x7f8f440047d0 2023-10-30T13:04:58.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f4e7fc640 1 -- 172.21.15.125:0/736356131 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f54005460 con 0x7f8f440047d0 2023-10-30T13:04:58.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5fe17640 1 -- 172.21.15.125:0/736356131 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f580b4d50 con 0x7f8f440047d0 2023-10-30T13:04:58.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f4e7fc640 1 -- 172.21.15.125:0/736356131 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54003840 con 0x7f8f440047d0 2023-10-30T13:04:58.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.048+0000 7f8f5fe17640 1 -- 172.21.15.125:0/736356131 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f58157c60 con 0x7f8f440047d0 2023-10-30T13:04:58.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.052+0000 7f8f5fe17640 1 -- 172.21.15.125:0/736356131 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f5804e170 con 0x7f8f440047d0 2023-10-30T13:04:58.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.052+0000 7f8f4e7fc640 1 -- 172.21.15.125:0/736356131 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f540039a0 con 0x7f8f440047d0 2023-10-30T13:04:58.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.052+0000 7f8f4e7fc640 1 --2- 172.21.15.125:0/736356131 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f3c0664d0 0x7f8f3c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:04:58.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.052+0000 7f8f4e7fc640 1 -- 172.21.15.125:0/736356131 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f8f540842e0 con 0x7f8f440047d0 2023-10-30T13:04:58.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.052+0000 7f8f5d3b2640 1 --2- 172.21.15.125:0/736356131 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f3c0664d0 0x7f8f3c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:04:58.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.052+0000 7f8f5d3b2640 1 --2- 172.21.15.125:0/736356131 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f3c0664d0 0x7f8f3c068990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8f48009ab0 tx=0x7f8f480a7040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:04:58.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.056+0000 7f8f4e7fc640 1 -- 172.21.15.125:0/736356131 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8f54051050 con 0x7f8f440047d0 2023-10-30T13:04:58.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.236+0000 7f8f5fe17640 1 -- 172.21.15.125:0/736356131 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f58146b60 con 0x7f8f440047d0 2023-10-30T13:04:58.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.236+0000 7f8f4e7fc640 1 -- 172.21.15.125:0/736356131 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f8f54054670 con 0x7f8f440047d0 2023-10-30T13:04:58.238 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:04:58.239 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:04:54.953323+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:04:58.240 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 157 2023-10-30T13:04:58.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.240+0000 7f8f5fe17640 1 -- 172.21.15.125:0/736356131 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f3c0664d0 msgr2=0x7f8f3c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:58.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.240+0000 7f8f5fe17640 1 --2- 172.21.15.125:0/736356131 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f3c0664d0 0x7f8f3c068990 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8f48009ab0 tx=0x7f8f480a7040 comp rx=0 tx=0).stop 2023-10-30T13:04:58.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.240+0000 7f8f5fe17640 1 -- 172.21.15.125:0/736356131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440047d0 msgr2=0x7f8f58157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:04:58.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.240+0000 7f8f5fe17640 1 --2- 172.21.15.125:0/736356131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440047d0 0x7f8f58157720 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400e5b0 tx=0x7f8f5400ea70 comp rx=0 tx=0).stop 2023-10-30T13:04:58.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.240+0000 7f8f5fe17640 1 -- 172.21.15.125:0/736356131 shutdown_connections 2023-10-30T13:04:58.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.240+0000 7f8f5fe17640 1 --2- 172.21.15.125:0/736356131 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f3c0664d0 0x7f8f3c068990 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:58.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.240+0000 7f8f5fe17640 1 --2- 172.21.15.125:0/736356131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f5815c320 0x7f8f580b3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:58.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.240+0000 7f8f5fe17640 1 --2- 172.21.15.125:0/736356131 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f58146550 0x7f8f580b3310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:58.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.240+0000 7f8f5fe17640 1 --2- 172.21.15.125:0/736356131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f440047d0 0x7f8f58157720 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:04:58.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.240+0000 7f8f5fe17640 1 -- 172.21.15.125:0/736356131 >> 172.21.15.125:0/736356131 conn(0x7f8f5806bfe0 msgr2=0x7f8f58152080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:04:58.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.240+0000 7f8f5fe17640 1 -- 172.21.15.125:0/736356131 shutdown_connections 2023-10-30T13:04:58.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:04:58.240+0000 7f8f5fe17640 1 -- 172.21.15.125:0/736356131 wait complete. 2023-10-30T13:04:58.259 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 157, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 157, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8022": { "addr": "172.21.15.125:6837/2196617378", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2196617378, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2196617378, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8022, "incarnation": 156, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:reconnect", "state_seq": 29 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:04:54.953323+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 8022 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:05:00.260 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:00.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfecfc5640 1 Processor -- start 2023-10-30T13:05:00.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfecfc5640 1 -- start start 2023-10-30T13:05:00.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfecfc5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fcfe8153bf0 0x7fcfe8071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:00.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfecfc5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcfe80c6fe0 0x7fcfe80c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:00.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfecfc5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe80700b0 con 0x7fcfe80c79e0 2023-10-30T13:05:00.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfecfc5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe80c6810 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfecfc5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe8159900 con 0x7fcfe8153bf0 2023-10-30T13:05:00.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfddd9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcfe80c6fe0 0x7fcfe80c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:00.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfe6d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcfe80c79e0 0x7fcfe80cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34336/0 (socket says 172.21.15.125:34336) 2023-10-30T13:05:00.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfe6d9d640 1 -- 172.21.15.125:0/1245480222 learned_addr learned my addr 172.21.15.125:0/1245480222 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:05:00.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfe659c640 1 --2- 172.21.15.125:0/1245480222 >> v2:172.21.15.136:3300/0 conn(0x7fcfe8153bf0 0x7fcfe8071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:00.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfddd9b640 1 -- 172.21.15.125:0/1245480222 >> v2:172.21.15.136:3300/0 conn(0x7fcfe8153bf0 msgr2=0x7fcfe8071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:00.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfddd9b640 1 --2- 172.21.15.125:0/1245480222 >> v2:172.21.15.136:3300/0 conn(0x7fcfe8153bf0 0x7fcfe8071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:00.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfddd9b640 1 -- 172.21.15.125:0/1245480222 >> v1:172.21.15.125:6789/0 conn(0x7fcfe80c79e0 legacy=0x7fcfe80cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:00.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfddd9b640 1 -- 172.21.15.125:0/1245480222 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe80c6530 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.400+0000 7fcfe5d9b640 1 -- 172.21.15.125:0/1245480222 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2221802212 0 0) 0x7fcfe80700b0 con 0x7fcfe80c79e0 2023-10-30T13:05:00.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.404+0000 7fcfddd9b640 1 --2- 172.21.15.125:0/1245480222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfe80c6fe0 0x7fcfe80c74a0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80c9150 tx=0x7fcfd001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3829dce8a825d343 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:00.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.404+0000 7fcfe5d9b640 1 -- 172.21.15.125:0/1245480222 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd00a8950 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.404+0000 7fcfe5d9b640 1 -- 172.21.15.125:0/1245480222 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd00a8ab0 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.404+0000 7fcfe5d9b640 1 -- 172.21.15.125:0/1245480222 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd00a8e40 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.404+0000 7fcfecfc5640 1 -- 172.21.15.125:0/1245480222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfe80c6fe0 msgr2=0x7fcfe80c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:00.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.404+0000 7fcfecfc5640 1 --2- 172.21.15.125:0/1245480222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfe80c6fe0 0x7fcfe80c74a0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80c9150 tx=0x7fcfd001ec50 comp rx=0 tx=0).stop 2023-10-30T13:05:00.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.404+0000 7fcfecfc5640 1 -- 172.21.15.125:0/1245480222 shutdown_connections 2023-10-30T13:05:00.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.404+0000 7fcfecfc5640 1 --2- 172.21.15.125:0/1245480222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfe80c6fe0 0x7fcfe80c74a0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:00.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.404+0000 7fcfecfc5640 1 --2- 172.21.15.125:0/1245480222 >> v2:172.21.15.136:3300/0 conn(0x7fcfe8153bf0 0x7fcfe8071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:00.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.404+0000 7fcfecfc5640 1 -- 172.21.15.125:0/1245480222 >> 172.21.15.125:0/1245480222 conn(0x7fcfe806c6e0 msgr2=0x7fcfe8069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:00.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.404+0000 7fcfecfc5640 1 -- 172.21.15.125:0/1245480222 shutdown_connections 2023-10-30T13:05:00.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.404+0000 7fcfecfc5640 1 -- 172.21.15.125:0/1245480222 wait complete. 2023-10-30T13:05:00.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfecfc5640 1 Processor -- start 2023-10-30T13:05:00.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfecfc5640 1 -- start start 2023-10-30T13:05:00.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfecfc5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfe80c6fe0 0x7fcfe81f2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:00.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfecfc5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfe80c79e0 0x7fcfe80ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:00.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfecfc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfe8153bf0 0x7fcfe814a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:00.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfecfc5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe814f200 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfecfc5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe814c8f0 con 0x7fcfe80c79e0 2023-10-30T13:05:00.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfecfc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe814e040 con 0x7fcfe8153bf0 2023-10-30T13:05:00.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfe659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfe80c6fe0 0x7fcfe81f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:00.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfe659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfe80c6fe0 0x7fcfe81f2560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37198/0 (socket says 172.21.15.125:37198) 2023-10-30T13:05:00.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfe659c640 1 -- 172.21.15.125:0/4124544859 learned_addr learned my addr 172.21.15.125:0/4124544859 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:00.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfddd9b640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfe80c79e0 0x7fcfe80ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:00.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfe6d9d640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfe8153bf0 0x7fcfe814a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:00.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfe659c640 1 -- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfe8153bf0 msgr2=0x7fcfe814a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:00.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfe659c640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfe8153bf0 0x7fcfe814a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:00.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfe659c640 1 -- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfe80c79e0 msgr2=0x7fcfe80ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:00.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfe659c640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfe80c79e0 0x7fcfe80ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:00.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfe659c640 1 -- 172.21.15.125:0/4124544859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd0009420 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfe6d9d640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfe8153bf0 0x7fcfe814a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:05:00.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfe659c640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfe80c6fe0 0x7fcfe81f2560 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8007e40 tx=0x7fcfd800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:00.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfdf7fe640 1 -- 172.21.15.125:0/4124544859 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8014620 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfdf7fe640 1 -- 172.21.15.125:0/4124544859 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd800ee90 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfecfc5640 1 -- 172.21.15.125:0/4124544859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe814e320 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfdf7fe640 1 -- 172.21.15.125:0/4124544859 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd80149c0 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.408+0000 7fcfecfc5640 1 -- 172.21.15.125:0/4124544859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe814ce20 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.412+0000 7fcfecfc5640 1 -- 172.21.15.125:0/4124544859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe80c9660 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.412+0000 7fcfdf7fe640 1 -- 172.21.15.125:0/4124544859 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcfd8014b20 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.412+0000 7fcfdf7fe640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcfc80664a0 0x7fcfc8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:00.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.412+0000 7fcfdf7fe640 1 -- 172.21.15.125:0/4124544859 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7fcfd8084da0 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.412+0000 7fcfddd9b640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcfc80664a0 0x7fcfc8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:00.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.412+0000 7fcfddd9b640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcfc80664a0 0x7fcfc8068960 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00b2000 tx=0x7fcfd00aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:00.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.416+0000 7fcfdf7fe640 1 -- 172.21.15.125:0/4124544859 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcfd8051b90 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.596+0000 7fcfecfc5640 1 -- 172.21.15.125:0/4124544859 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfe804e170 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.596+0000 7fcfdf7fe640 1 -- 172.21.15.125:0/4124544859 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+5727 (secure 0 0 0) 0x7fcfd80551b0 con 0x7fcfe80c6fe0 2023-10-30T13:05:00.599 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:00.599 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":157,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:04:54.953323+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:00.599 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 157 2023-10-30T13:05:00.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.600+0000 7fcfecfc5640 1 -- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcfc80664a0 msgr2=0x7fcfc8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:00.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.600+0000 7fcfecfc5640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcfc80664a0 0x7fcfc8068960 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fcfd00b2000 tx=0x7fcfd00aa070 comp rx=0 tx=0).stop 2023-10-30T13:05:00.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.600+0000 7fcfecfc5640 1 -- 172.21.15.125:0/4124544859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfe80c6fe0 msgr2=0x7fcfe81f2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:00.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.600+0000 7fcfecfc5640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfe80c6fe0 0x7fcfe81f2560 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fcfd8007e40 tx=0x7fcfd800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:05:00.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.600+0000 7fcfecfc5640 1 -- 172.21.15.125:0/4124544859 shutdown_connections 2023-10-30T13:05:00.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.600+0000 7fcfecfc5640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcfc80664a0 0x7fcfc8068960 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:00.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.600+0000 7fcfecfc5640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfe8153bf0 0x7fcfe814a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:00.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.600+0000 7fcfecfc5640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfe80c79e0 0x7fcfe80ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:00.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.600+0000 7fcfecfc5640 1 --2- 172.21.15.125:0/4124544859 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfe80c6fe0 0x7fcfe81f2560 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:00.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.600+0000 7fcfecfc5640 1 -- 172.21.15.125:0/4124544859 >> 172.21.15.125:0/4124544859 conn(0x7fcfe806c6e0 msgr2=0x7fcfe8069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:00.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.600+0000 7fcfecfc5640 1 -- 172.21.15.125:0/4124544859 shutdown_connections 2023-10-30T13:05:00.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:00.600+0000 7fcfecfc5640 1 -- 172.21.15.125:0/4124544859 wait complete. 2023-10-30T13:05:00.619 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:05:02.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:02.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f3754fc5640 1 Processor -- start 2023-10-30T13:05:02.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f3754fc5640 1 -- start start 2023-10-30T13:05:02.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f3754fc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3750069e90 0x7f375015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:02.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f3754fc5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37500c6c10 con 0x7f375006a270 2023-10-30T13:05:02.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f3754fc5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37500c6ef0 con 0x7f375006cb80 2023-10-30T13:05:02.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f3754fc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37500c8570 con 0x7f3750069e90 2023-10-30T13:05:02.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374dd9b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f375006a270 0x7f37500c9cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47668/0 (socket says 172.21.15.125:47668) 2023-10-30T13:05:02.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374dd9b640 1 -- 172.21.15.125:0/2593594848 learned_addr learned my addr 172.21.15.125:0/2593594848 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:05:02.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374e59c640 1 --2- 172.21.15.125:0/2593594848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3750069e90 0x7f375015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:02.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342392464 0 0) 0x7f37500c6c10 con 0x7f375006a270 2023-10-30T13:05:02.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3738003530 con 0x7f375006a270 2023-10-30T13:05:02.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700078370 0 0) 0x7f37500c6ef0 con 0x7f375006cb80 2023-10-30T13:05:02.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37500c6c10 con 0x7f375006cb80 2023-10-30T13:05:02.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2659915592 0 0) 0x7f3738003530 con 0x7f375006a270 2023-10-30T13:05:02.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3750069e90 msgr2=0x7f375015a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:02.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 --2- 172.21.15.125:0/2593594848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3750069e90 0x7f375015a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:02.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 >> v1:172.21.15.136:6789/0 conn(0x7f375006cb80 legacy=0x7f375006d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:02.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37500c71d0 con 0x7f375006a270 2023-10-30T13:05:02.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f3744001600 con 0x7f375006a270 2023-10-30T13:05:02.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 >> v1:172.21.15.125:6789/0 conn(0x7f375006a270 legacy=0x7f37500c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:02.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 --2- 172.21.15.125:0/2593594848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3738003b40 0x7f3738003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:02.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 --2- 172.21.15.125:0/2593594848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37380047d0 0x7f3738004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:02.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 --2- 172.21.15.125:0/2593594848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3738005190 0x7f373800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:02.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37500c8570 con 0x7f3738005190 2023-10-30T13:05:02.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373800bfb0 con 0x7f37380047d0 2023-10-30T13:05:02.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373800c170 con 0x7f3738003b40 2023-10-30T13:05:02.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374e59c640 1 --2- 172.21.15.125:0/2593594848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3750069e90 0x7f375015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:05:02.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374ed9d640 1 --2- 172.21.15.125:0/2593594848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3738005190 0x7f373800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:02.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374dd9b640 1 --2- 172.21.15.125:0/2593594848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37380047d0 0x7f3738004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:02.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374e59c640 1 --2- 172.21.15.125:0/2593594848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3738003b40 0x7f3738003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:02.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374ed9d640 1 -- 172.21.15.125:0/2593594848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3738003b40 msgr2=0x7f3738003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:02.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374ed9d640 1 --2- 172.21.15.125:0/2593594848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3738003b40 0x7f3738003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:02.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374ed9d640 1 -- 172.21.15.125:0/2593594848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37380047d0 msgr2=0x7f3738004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:02.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374ed9d640 1 --2- 172.21.15.125:0/2593594848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f37380047d0 0x7f3738004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:02.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.768+0000 7f374ed9d640 1 -- 172.21.15.125:0/2593594848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3738003530 con 0x7f3738005190 2023-10-30T13:05:02.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.772+0000 7f374ed9d640 1 --2- 172.21.15.125:0/2593594848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3738005190 0x7f373800b990 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f3748001fc0 tx=0x7f374801c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1168233e52237f13 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:02.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.772+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3748003b80 con 0x7f3738005190 2023-10-30T13:05:02.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.772+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3748003ce0 con 0x7f3738005190 2023-10-30T13:05:02.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.772+0000 7f374d59a640 1 -- 172.21.15.125:0/2593594848 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37480052f0 con 0x7f3738005190 2023-10-30T13:05:02.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.772+0000 7f3754fc5640 1 -- 172.21.15.125:0/2593594848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3738005190 msgr2=0x7f373800b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:02.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.772+0000 7f3754fc5640 1 --2- 172.21.15.125:0/2593594848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3738005190 0x7f373800b990 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f3748001fc0 tx=0x7f374801c8e0 comp rx=0 tx=0).stop 2023-10-30T13:05:02.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.772+0000 7f374ed9d640 1 -- 172.21.15.125:0/2593594848 reap_dead start 2023-10-30T13:05:02.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.772+0000 7f3754fc5640 1 -- 172.21.15.125:0/2593594848 shutdown_connections 2023-10-30T13:05:02.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.772+0000 7f3754fc5640 1 -- 172.21.15.125:0/2593594848 >> 172.21.15.125:0/2593594848 conn(0x7f375006dfd0 msgr2=0x7f375006e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:02.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.772+0000 7f3754fc5640 1 -- 172.21.15.125:0/2593594848 shutdown_connections 2023-10-30T13:05:02.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.772+0000 7f3754fc5640 1 -- 172.21.15.125:0/2593594848 wait complete. 2023-10-30T13:05:02.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f3754fc5640 1 Processor -- start 2023-10-30T13:05:02.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f3754fc5640 1 -- start start 2023-10-30T13:05:02.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f3754fc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3738005190 0x7f3750158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:02.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f3754fc5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f375014ff00 0x7f375015a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:02.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f3754fc5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375006ad80 0x7f375014f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:02.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f3754fc5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375015daf0 con 0x7f375006ad80 2023-10-30T13:05:02.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f3754fc5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f375015af00 con 0x7f375014ff00 2023-10-30T13:05:02.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f3754fc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3750155140 con 0x7f3738005190 2023-10-30T13:05:02.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f374ed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375006ad80 0x7f375014f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:02.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f374ed9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375006ad80 0x7f375014f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37376/0 (socket says 172.21.15.125:37376) 2023-10-30T13:05:02.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f374ed9d640 1 -- 172.21.15.125:0/3745204926 learned_addr learned my addr 172.21.15.125:0/3745204926 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:02.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f374e59c640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3738005190 0x7f3750158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:02.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f374dd9b640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f375014ff00 0x7f375015a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:02.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f374ed9d640 1 -- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3738005190 msgr2=0x7f3750158d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:02.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f374ed9d640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3738005190 0x7f3750158d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:02.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f374ed9d640 1 -- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f375014ff00 msgr2=0x7f375015a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:02.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f374ed9d640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f375014ff00 0x7f375015a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:02.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f374ed9d640 1 -- 172.21.15.125:0/3745204926 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3750156530 con 0x7f375006ad80 2023-10-30T13:05:02.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f374dd9b640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f375014ff00 0x7f375015a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:05:02.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f374ed9d640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375006ad80 0x7f375014f310 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f3748005040 tx=0x7f3738005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:02.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f373effd640 1 -- 172.21.15.125:0/3745204926 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3748003e50 con 0x7f375006ad80 2023-10-30T13:05:02.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f373effd640 1 -- 172.21.15.125:0/3745204926 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3748005650 con 0x7f375006ad80 2023-10-30T13:05:02.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f3754fc5640 1 -- 172.21.15.125:0/3745204926 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37501ef150 con 0x7f375006ad80 2023-10-30T13:05:02.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.776+0000 7f373effd640 1 -- 172.21.15.125:0/3745204926 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3748006b60 con 0x7f375006ad80 2023-10-30T13:05:02.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.780+0000 7f3754fc5640 1 -- 172.21.15.125:0/3745204926 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3750155350 con 0x7f375006ad80 2023-10-30T13:05:02.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.780+0000 7f3754fc5640 1 -- 172.21.15.125:0/3745204926 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3750155540 con 0x7f375006ad80 2023-10-30T13:05:02.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.780+0000 7f373effd640 1 -- 172.21.15.125:0/3745204926 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3748006cc0 con 0x7f375006ad80 2023-10-30T13:05:02.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.780+0000 7f373effd640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37300664d0 0x7f3730068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:02.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.780+0000 7f373effd640 1 -- 172.21.15.125:0/3745204926 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f374811dad0 con 0x7f375006ad80 2023-10-30T13:05:02.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.780+0000 7f374e59c640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37300664d0 0x7f3730068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:02.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.780+0000 7f374e59c640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37300664d0 0x7f3730068990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f375006ed20 tx=0x7f3740008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:02.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:02.784+0000 7f373effd640 1 -- 172.21.15.125:0/3745204926 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f37480ea7a0 con 0x7f375006ad80 2023-10-30T13:05:03.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:03.032+0000 7f3754fc5640 1 -- 172.21.15.125:0/3745204926 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3750155540 con 0x7f375006ad80 2023-10-30T13:05:03.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:03.032+0000 7f373effd640 1 -- 172.21.15.125:0/3745204926 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f37480ba070 con 0x7f375006ad80 2023-10-30T13:05:03.033 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:03.034 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:00.671291+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:03.034 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 158 2023-10-30T13:05:03.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:03.032+0000 7f3754fc5640 1 -- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37300664d0 msgr2=0x7f3730068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:03.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:03.032+0000 7f3754fc5640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37300664d0 0x7f3730068990 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f375006ed20 tx=0x7f3740008040 comp rx=0 tx=0).stop 2023-10-30T13:05:03.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:03.032+0000 7f3754fc5640 1 -- 172.21.15.125:0/3745204926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375006ad80 msgr2=0x7f375014f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:03.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:03.032+0000 7f3754fc5640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375006ad80 0x7f375014f310 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f3748005040 tx=0x7f3738005800 comp rx=0 tx=0).stop 2023-10-30T13:05:03.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:03.036+0000 7f3754fc5640 1 -- 172.21.15.125:0/3745204926 shutdown_connections 2023-10-30T13:05:03.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:03.036+0000 7f3754fc5640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37300664d0 0x7f3730068990 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:03.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:03.036+0000 7f3754fc5640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f375006ad80 0x7f375014f310 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:03.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:03.036+0000 7f3754fc5640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f375014ff00 0x7f375015a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:03.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:03.036+0000 7f3754fc5640 1 --2- 172.21.15.125:0/3745204926 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3738005190 0x7f3750158d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:03.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:03.036+0000 7f3754fc5640 1 -- 172.21.15.125:0/3745204926 >> 172.21.15.125:0/3745204926 conn(0x7f375006dfd0 msgr2=0x7f3750153920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:03.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:03.036+0000 7f3754fc5640 1 -- 172.21.15.125:0/3745204926 shutdown_connections 2023-10-30T13:05:03.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:03.036+0000 7f3754fc5640 1 -- 172.21.15.125:0/3745204926 wait complete. 2023-10-30T13:05:03.163 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:05:05.165 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:05.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.308+0000 7fccb7ad6640 1 Processor -- start 2023-10-30T13:05:05.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.308+0000 7fccb7ad6640 1 -- start start 2023-10-30T13:05:05.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb7ad6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fccb006a720 0x7fccb006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:05.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb7ad6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb0071290 con 0x7fccb0159e60 2023-10-30T13:05:05.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb7ad6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb0071570 con 0x7fccb00bfec0 2023-10-30T13:05:05.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb7ad6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb0071850 con 0x7fccb006a720 2023-10-30T13:05:05.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb5872640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fccb0159e60 0x7fccb006f540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47684/0 (socket says 172.21.15.125:47684) 2023-10-30T13:05:05.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb5872640 1 -- 172.21.15.125:0/240806790 learned_addr learned my addr 172.21.15.125:0/240806790 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:05:05.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb6073640 1 --2- 172.21.15.125:0/240806790 >> v2:172.21.15.136:3300/0 conn(0x7fccb006a720 0x7fccb006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:05.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130350299 0 0) 0x7fccb0071290 con 0x7fccb0159e60 2023-10-30T13:05:05.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc9c003530 con 0x7fccb0159e60 2023-10-30T13:05:05.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597394281 0 0) 0x7fccb0071570 con 0x7fccb00bfec0 2023-10-30T13:05:05.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccb0071290 con 0x7fccb00bfec0 2023-10-30T13:05:05.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1740923556 0 0) 0x7fcc9c003530 con 0x7fccb0159e60 2023-10-30T13:05:05.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 >> v2:172.21.15.136:3300/0 conn(0x7fccb006a720 msgr2=0x7fccb006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:05.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 --2- 172.21.15.125:0/240806790 >> v2:172.21.15.136:3300/0 conn(0x7fccb006a720 0x7fccb006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:05.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 >> v1:172.21.15.136:6789/0 conn(0x7fccb00bfec0 legacy=0x7fccb00c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:05.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb0071b30 con 0x7fccb0159e60 2023-10-30T13:05:05.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fccac001680 con 0x7fccb0159e60 2023-10-30T13:05:05.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 >> v1:172.21.15.125:6789/0 conn(0x7fccb0159e60 legacy=0x7fccb006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:05.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 --2- 172.21.15.125:0/240806790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc9c003b10 0x7fcc9c003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:05.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 --2- 172.21.15.125:0/240806790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9c0047d0 0x7fcc9c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:05.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 --2- 172.21.15.125:0/240806790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc9c005190 0x7fcc9c00dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:05.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0071850 con 0x7fcc9c0047d0 2023-10-30T13:05:05.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c00e190 con 0x7fcc9c005190 2023-10-30T13:05:05.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9c00e350 con 0x7fcc9c003b10 2023-10-30T13:05:05.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb5071640 1 --2- 172.21.15.125:0/240806790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9c0047d0 0x7fcc9c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:05.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb5872640 1 --2- 172.21.15.125:0/240806790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc9c003b10 0x7fcc9c003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:05.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb6073640 1 --2- 172.21.15.125:0/240806790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc9c005190 0x7fcc9c00dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:05.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb5071640 1 -- 172.21.15.125:0/240806790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc9c003b10 msgr2=0x7fcc9c003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:05.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb5071640 1 --2- 172.21.15.125:0/240806790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc9c003b10 0x7fcc9c003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:05.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb5071640 1 -- 172.21.15.125:0/240806790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc9c005190 msgr2=0x7fcc9c00dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:05.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb5071640 1 --2- 172.21.15.125:0/240806790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc9c005190 0x7fcc9c00dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:05.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb5071640 1 -- 172.21.15.125:0/240806790 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c003530 con 0x7fcc9c0047d0 2023-10-30T13:05:05.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb6073640 1 --2- 172.21.15.125:0/240806790 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc9c005190 0x7fcc9c00dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:05:05.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.312+0000 7fccb6073640 1 -- 172.21.15.125:0/240806790 reap_dead start 2023-10-30T13:05:05.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.316+0000 7fccb5071640 1 --2- 172.21.15.125:0/240806790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9c0047d0 0x7fcc9c004c50 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fcca0007fd0 tx=0x7fcca001c4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0854e873dbd1ad server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:05.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.316+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccac007ea0 con 0x7fcc9c0047d0 2023-10-30T13:05:05.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.316+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca00052f0 con 0x7fcc9c0047d0 2023-10-30T13:05:05.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.316+0000 7fccb4870640 1 -- 172.21.15.125:0/240806790 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca0003600 con 0x7fcc9c0047d0 2023-10-30T13:05:05.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.316+0000 7fccb7ad6640 1 -- 172.21.15.125:0/240806790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9c0047d0 msgr2=0x7fcc9c004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:05.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.316+0000 7fccb7ad6640 1 --2- 172.21.15.125:0/240806790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9c0047d0 0x7fcc9c004c50 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7fcca0007fd0 tx=0x7fcca001c4e0 comp rx=0 tx=0).stop 2023-10-30T13:05:05.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.316+0000 7fccb7ad6640 1 -- 172.21.15.125:0/240806790 shutdown_connections 2023-10-30T13:05:05.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.316+0000 7fccb7ad6640 1 --2- 172.21.15.125:0/240806790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc9c0047d0 0x7fcc9c004c50 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:05.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.316+0000 7fccb7ad6640 1 -- 172.21.15.125:0/240806790 >> 172.21.15.125:0/240806790 conn(0x7fccb00c7910 msgr2=0x7fccb00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:05.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.316+0000 7fccb7ad6640 1 -- 172.21.15.125:0/240806790 shutdown_connections 2023-10-30T13:05:05.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.316+0000 7fccb7ad6640 1 -- 172.21.15.125:0/240806790 wait complete. 2023-10-30T13:05:05.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb7ad6640 1 Processor -- start 2023-10-30T13:05:05.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb7ad6640 1 -- start start 2023-10-30T13:05:05.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb7ad6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc9c0047d0 0x7fccb01e5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:05.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb7ad6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fccb00c6960 0x7fccb014a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:05.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb7ad6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccb00c6dc0 0x7fccb014ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:05.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb7ad6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0156aa0 con 0x7fccb00c6dc0 2023-10-30T13:05:05.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb7ad6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb01eee50 con 0x7fccb00c6960 2023-10-30T13:05:05.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb7ad6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb01df0a0 con 0x7fcc9c0047d0 2023-10-30T13:05:05.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb6073640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccb00c6dc0 0x7fccb014ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:05.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb6073640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccb00c6dc0 0x7fccb014ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37394/0 (socket says 172.21.15.125:37394) 2023-10-30T13:05:05.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb6073640 1 -- 172.21.15.125:0/787428770 learned_addr learned my addr 172.21.15.125:0/787428770 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:05.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb5872640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc9c0047d0 0x7fccb01e5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:05.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb5071640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fccb00c6960 0x7fccb014a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:05.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb6073640 1 -- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc9c0047d0 msgr2=0x7fccb01e5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:05.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb6073640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc9c0047d0 0x7fccb01e5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:05.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb6073640 1 -- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fccb00c6960 msgr2=0x7fccb014a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:05.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb6073640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fccb00c6960 0x7fccb014a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:05.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb6073640 1 -- 172.21.15.125:0/787428770 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb01ef060 con 0x7fccb00c6dc0 2023-10-30T13:05:05.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb5071640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fccb00c6960 0x7fccb014a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:05:05.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb6073640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccb00c6dc0 0x7fccb014ad40 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fccb006f1b0 tx=0x7fcca8010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:05.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fcca67fc640 1 -- 172.21.15.125:0/787428770 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca801b070 con 0x7fccb00c6dc0 2023-10-30T13:05:05.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fcca67fc640 1 -- 172.21.15.125:0/787428770 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca800f040 con 0x7fccb00c6dc0 2023-10-30T13:05:05.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fcca67fc640 1 -- 172.21.15.125:0/787428770 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca800e050 con 0x7fccb00c6dc0 2023-10-30T13:05:05.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb7ad6640 1 -- 172.21.15.125:0/787428770 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb01ef340 con 0x7fccb00c6dc0 2023-10-30T13:05:05.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.320+0000 7fccb7ad6640 1 -- 172.21.15.125:0/787428770 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb01ef900 con 0x7fccb00c6dc0 2023-10-30T13:05:05.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.324+0000 7fccb7ad6640 1 -- 172.21.15.125:0/787428770 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb004e170 con 0x7fccb00c6dc0 2023-10-30T13:05:05.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.324+0000 7fcca67fc640 1 -- 172.21.15.125:0/787428770 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcca8015440 con 0x7fccb00c6dc0 2023-10-30T13:05:05.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.324+0000 7fcca67fc640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc940664d0 0x7fcc94068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:05.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.324+0000 7fcca67fc640 1 -- 172.21.15.125:0/787428770 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7fcca80846c0 con 0x7fccb00c6dc0 2023-10-30T13:05:05.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.324+0000 7fccb5872640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc940664d0 0x7fcc94068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:05.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.324+0000 7fccb5872640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc940664d0 0x7fcc94068990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fccac001060 tx=0x7fccac001e10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:05.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.328+0000 7fcca67fc640 1 -- 172.21.15.125:0/787428770 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcca8051390 con 0x7fccb00c6dc0 2023-10-30T13:05:05.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.507+0000 7fccb7ad6640 1 -- 172.21.15.125:0/787428770 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fccb00c83b0 con 0x7fccb00c6dc0 2023-10-30T13:05:05.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.507+0000 7fcca67fc640 1 -- 172.21.15.125:0/787428770 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+5724 (secure 0 0 0) 0x7fcca80549b0 con 0x7fccb00c6dc0 2023-10-30T13:05:05.508 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:05.508 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:00.671291+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:rejoin","state_seq":31,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:05.508 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 158 2023-10-30T13:05:05.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.507+0000 7fccb7ad6640 1 -- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc940664d0 msgr2=0x7fcc94068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:05.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.507+0000 7fccb7ad6640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc940664d0 0x7fcc94068990 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fccac001060 tx=0x7fccac001e10 comp rx=0 tx=0).stop 2023-10-30T13:05:05.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.507+0000 7fccb7ad6640 1 -- 172.21.15.125:0/787428770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccb00c6dc0 msgr2=0x7fccb014ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:05.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.507+0000 7fccb7ad6640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccb00c6dc0 0x7fccb014ad40 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7fccb006f1b0 tx=0x7fcca8010590 comp rx=0 tx=0).stop 2023-10-30T13:05:05.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.507+0000 7fccb7ad6640 1 -- 172.21.15.125:0/787428770 shutdown_connections 2023-10-30T13:05:05.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.507+0000 7fccb7ad6640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc940664d0 0x7fcc94068990 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:05.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.507+0000 7fccb7ad6640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fccb00c6dc0 0x7fccb014ad40 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:05.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.507+0000 7fccb7ad6640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fccb00c6960 0x7fccb014a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:05.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.507+0000 7fccb7ad6640 1 --2- 172.21.15.125:0/787428770 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc9c0047d0 0x7fccb01e5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:05.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.507+0000 7fccb7ad6640 1 -- 172.21.15.125:0/787428770 >> 172.21.15.125:0/787428770 conn(0x7fccb00c7910 msgr2=0x7fccb00c98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:05.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.507+0000 7fccb7ad6640 1 -- 172.21.15.125:0/787428770 shutdown_connections 2023-10-30T13:05:05.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:05.507+0000 7fccb7ad6640 1 -- 172.21.15.125:0/787428770 wait complete. 2023-10-30T13:05:05.528 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:05:07.530 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:07.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.675+0000 7f72ff491640 1 Processor -- start 2023-10-30T13:05:07.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.675+0000 7f72ff491640 1 -- start start 2023-10-30T13:05:07.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.675+0000 7f72ff491640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f72f81538a0 0x7f72f815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:07.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.675+0000 7f72ff491640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f80cb800 con 0x7f72f80ca0e0 2023-10-30T13:05:07.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.675+0000 7f72ff491640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f815dc30 con 0x7f72f80cf160 2023-10-30T13:05:07.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.675+0000 7f72ff491640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f80cb330 con 0x7f72f81538a0 2023-10-30T13:05:07.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.679+0000 7f72fca2c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f72f80ca0e0 0x7f72f815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47688/0 (socket says 172.21.15.125:47688) 2023-10-30T13:05:07.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.679+0000 7f72fca2c640 1 -- 172.21.15.125:0/1421420619 learned_addr learned my addr 172.21.15.125:0/1421420619 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:05:07.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.679+0000 7f72fd22d640 1 --2- 172.21.15.125:0/1421420619 >> v2:172.21.15.136:3300/0 conn(0x7f72f81538a0 0x7f72f815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:07.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.679+0000 7f72effff640 1 -- 172.21.15.125:0/1421420619 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385011376 0 0) 0x7f72f80cb800 con 0x7f72f80ca0e0 2023-10-30T13:05:07.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.679+0000 7f72effff640 1 -- 172.21.15.125:0/1421420619 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e4003530 con 0x7f72f80ca0e0 2023-10-30T13:05:07.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.679+0000 7f72fd22d640 1 -- 172.21.15.125:0/1421420619 >> v1:172.21.15.136:6789/0 conn(0x7f72f80cf160 legacy=0x7f72f8164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:07.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.679+0000 7f72fd22d640 1 -- 172.21.15.125:0/1421420619 >> v1:172.21.15.125:6789/0 conn(0x7f72f80ca0e0 legacy=0x7f72f815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:07.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.679+0000 7f72fd22d640 1 -- 172.21.15.125:0/1421420619 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f80cad70 con 0x7f72f81538a0 2023-10-30T13:05:07.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.679+0000 7f72fd22d640 1 --2- 172.21.15.125:0/1421420619 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f81538a0 0x7f72f815d6f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f72f40097c0 tx=0x7f72f401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b29bb17c4cd6c7e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:07.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.683+0000 7f72effff640 1 -- 172.21.15.125:0/1421420619 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f40a8970 con 0x7f72f81538a0 2023-10-30T13:05:07.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.683+0000 7f72effff640 1 -- 172.21.15.125:0/1421420619 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f40a8ad0 con 0x7f72f81538a0 2023-10-30T13:05:07.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.683+0000 7f72effff640 1 -- 172.21.15.125:0/1421420619 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f40a8e60 con 0x7f72f81538a0 2023-10-30T13:05:07.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 -- 172.21.15.125:0/1421420619 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f81538a0 msgr2=0x7f72f815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:07.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 --2- 172.21.15.125:0/1421420619 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f81538a0 0x7f72f815d6f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f72f40097c0 tx=0x7f72f401ec70 comp rx=0 tx=0).stop 2023-10-30T13:05:07.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 -- 172.21.15.125:0/1421420619 shutdown_connections 2023-10-30T13:05:07.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 --2- 172.21.15.125:0/1421420619 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f81538a0 0x7f72f815d6f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:07.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 -- 172.21.15.125:0/1421420619 >> 172.21.15.125:0/1421420619 conn(0x7f72f806bfe0 msgr2=0x7f72f806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:07.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 -- 172.21.15.125:0/1421420619 shutdown_connections 2023-10-30T13:05:07.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 -- 172.21.15.125:0/1421420619 wait complete. 2023-10-30T13:05:07.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 Processor -- start 2023-10-30T13:05:07.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 -- start start 2023-10-30T13:05:07.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72f80ca0e0 0x7f72f80b49f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:07.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72f80cf160 0x7f72f8157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:07.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f81538a0 0x7f72f8157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:07.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8157520 con 0x7f72f80cf160 2023-10-30T13:05:07.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f815c0e0 con 0x7f72f81538a0 2023-10-30T13:05:07.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72ff491640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8159510 con 0x7f72f80ca0e0 2023-10-30T13:05:07.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72fca2c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72f80cf160 0x7f72f8157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:07.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72fca2c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72f80cf160 0x7f72f8157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37398/0 (socket says 172.21.15.125:37398) 2023-10-30T13:05:07.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72fca2c640 1 -- 172.21.15.125:0/2610661637 learned_addr learned my addr 172.21.15.125:0/2610661637 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:07.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72fd22d640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72f80ca0e0 0x7f72f80b49f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:07.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.687+0000 7f72fda2e640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f81538a0 0x7f72f8157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:07.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72fca2c640 1 -- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72f80ca0e0 msgr2=0x7f72f80b49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:07.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72fca2c640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72f80ca0e0 0x7f72f80b49f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:07.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72fca2c640 1 -- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f81538a0 msgr2=0x7f72f8157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:07.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72fca2c640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f81538a0 0x7f72f8157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:07.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72fca2c640 1 -- 172.21.15.125:0/2610661637 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f4009420 con 0x7f72f80cf160 2023-10-30T13:05:07.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72fda2e640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f81538a0 0x7f72f8157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:05:07.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72fca2c640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72f80cf160 0x7f72f8157970 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f72e800c450 tx=0x7f72e800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:07.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72edffb640 1 -- 172.21.15.125:0/2610661637 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8005cf0 con 0x7f72f80cf160 2023-10-30T13:05:07.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72edffb640 1 -- 172.21.15.125:0/2610661637 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e8005e50 con 0x7f72f80cf160 2023-10-30T13:05:07.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72edffb640 1 -- 172.21.15.125:0/2610661637 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e8007270 con 0x7f72f80cf160 2023-10-30T13:05:07.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72ff491640 1 -- 172.21.15.125:0/2610661637 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f815c350 con 0x7f72f80cf160 2023-10-30T13:05:07.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72ff491640 1 -- 172.21.15.125:0/2610661637 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f815c910 con 0x7f72f80cf160 2023-10-30T13:05:07.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72ff491640 1 -- 172.21.15.125:0/2610661637 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f8157690 con 0x7f72f80cf160 2023-10-30T13:05:07.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72edffb640 1 -- 172.21.15.125:0/2610661637 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f72e80073d0 con 0x7f72f80cf160 2023-10-30T13:05:07.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72edffb640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72dc0664f0 0x7f72dc0689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:07.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.691+0000 7f72edffb640 1 -- 172.21.15.125:0/2610661637 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f72e8083a80 con 0x7f72f80cf160 2023-10-30T13:05:07.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.695+0000 7f72fd22d640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72dc0664f0 0x7f72dc0689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:07.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.695+0000 7f72fd22d640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72dc0664f0 0x7f72dc0689b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f72f40b2000 tx=0x7f72f40aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:07.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.699+0000 7f72edffb640 1 -- 172.21.15.125:0/2610661637 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f72f8157690 con 0x7f72f80cf160 2023-10-30T13:05:07.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.879+0000 7f72ff491640 1 -- 172.21.15.125:0/2610661637 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72f8157690 con 0x7f72f80cf160 2023-10-30T13:05:07.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.879+0000 7f72edffb640 1 -- 172.21.15.125:0/2610661637 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f72f8157690 con 0x7f72f80cf160 2023-10-30T13:05:07.883 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:07.883 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:07.674509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:07.883 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 159 2023-10-30T13:05:07.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.883+0000 7f72ff491640 1 -- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72dc0664f0 msgr2=0x7f72dc0689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:07.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.883+0000 7f72ff491640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72dc0664f0 0x7f72dc0689b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f72f40b2000 tx=0x7f72f40aa070 comp rx=0 tx=0).stop 2023-10-30T13:05:07.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.883+0000 7f72ff491640 1 -- 172.21.15.125:0/2610661637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72f80cf160 msgr2=0x7f72f8157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:07.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.883+0000 7f72ff491640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72f80cf160 0x7f72f8157970 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f72e800c450 tx=0x7f72e800c910 comp rx=0 tx=0).stop 2023-10-30T13:05:07.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.883+0000 7f72ff491640 1 -- 172.21.15.125:0/2610661637 shutdown_connections 2023-10-30T13:05:07.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.883+0000 7f72ff491640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72dc0664f0 0x7f72dc0689b0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:07.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.883+0000 7f72ff491640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72f81538a0 0x7f72f8157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:07.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.883+0000 7f72ff491640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72f80cf160 0x7f72f8157970 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:07.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.883+0000 7f72ff491640 1 --2- 172.21.15.125:0/2610661637 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72f80ca0e0 0x7f72f80b49f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:07.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.883+0000 7f72ff491640 1 -- 172.21.15.125:0/2610661637 >> 172.21.15.125:0/2610661637 conn(0x7f72f806bfe0 msgr2=0x7f72f80c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:07.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.883+0000 7f72ff491640 1 -- 172.21.15.125:0/2610661637 shutdown_connections 2023-10-30T13:05:07.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:07.883+0000 7f72ff491640 1 -- 172.21.15.125:0/2610661637 wait complete. 2023-10-30T13:05:07.903 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:05:09.904 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:10.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.047+0000 7f0389617640 1 Processor -- start 2023-10-30T13:05:10.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.047+0000 7f0389617640 1 -- start start 2023-10-30T13:05:10.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f0389617640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f038406f7a0 0x7f038406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:10.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f0389617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03840c6bd0 0x7f0384069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:10.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f0389617640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038406a3d0 con 0x7f0384071110 2023-10-30T13:05:10.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f0389617640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03840700c0 con 0x7f038406f7a0 2023-10-30T13:05:10.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f0389617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0384066cf0 con 0x7f03840c6bd0 2023-10-30T13:05:10.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f037a7fc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03840c6bd0 0x7f0384069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:10.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f0382ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f038406f7a0 0x7f038406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:10.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f0382ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f038406f7a0 0x7f038406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:48938/0 (socket says 172.21.15.125:48938) 2023-10-30T13:05:10.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f0382ffd640 1 -- 172.21.15.125:0/3000835924 learned_addr learned my addr 172.21.15.125:0/3000835924 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:10.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f037a7fc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03840c6bd0 0x7f0384069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:37974/0 (socket says 172.21.15.125:37974) 2023-10-30T13:05:10.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f03837fe640 1 --1- 172.21.15.125:0/3000835924 >> v1:172.21.15.136:6789/0 conn(0x7f0384071110 0x7f0384158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:43844/0 (socket says 172.21.15.125:43844) 2023-10-30T13:05:10.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f0382ffd640 1 -- 172.21.15.125:0/3000835924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03840c6bd0 msgr2=0x7f0384069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:10.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f0382ffd640 1 --2- 172.21.15.125:0/3000835924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03840c6bd0 0x7f0384069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:10.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f0382ffd640 1 -- 172.21.15.125:0/3000835924 >> v1:172.21.15.136:6789/0 conn(0x7f0384071110 legacy=0x7f0384158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:10.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f0382ffd640 1 -- 172.21.15.125:0/3000835924 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03840c70d0 con 0x7f038406f7a0 2023-10-30T13:05:10.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.051+0000 7f037a7fc640 1 --2- 172.21.15.125:0/3000835924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03840c6bd0 0x7f0384069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:05:10.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f0382ffd640 1 --2- 172.21.15.125:0/3000835924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f038406f7a0 0x7f038406fb80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0374009870 tx=0x7f037401ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58503ab6953dbd70 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:10.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f03827fc640 1 -- 172.21.15.125:0/3000835924 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03740a8a30 con 0x7f038406f7a0 2023-10-30T13:05:10.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f03827fc640 1 -- 172.21.15.125:0/3000835924 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03740a8b90 con 0x7f038406f7a0 2023-10-30T13:05:10.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f03827fc640 1 -- 172.21.15.125:0/3000835924 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03740b0440 con 0x7f038406f7a0 2023-10-30T13:05:10.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f0389617640 1 -- 172.21.15.125:0/3000835924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f038406f7a0 msgr2=0x7f038406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:10.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f0389617640 1 --2- 172.21.15.125:0/3000835924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f038406f7a0 0x7f038406fb80 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0374009870 tx=0x7f037401ed30 comp rx=0 tx=0).stop 2023-10-30T13:05:10.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f0389617640 1 -- 172.21.15.125:0/3000835924 shutdown_connections 2023-10-30T13:05:10.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f0389617640 1 --2- 172.21.15.125:0/3000835924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03840c6bd0 0x7f0384069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:10.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f0389617640 1 --2- 172.21.15.125:0/3000835924 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f038406f7a0 0x7f038406fb80 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:10.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f0389617640 1 -- 172.21.15.125:0/3000835924 >> 172.21.15.125:0/3000835924 conn(0x7f03840c7910 msgr2=0x7f03840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:10.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f0389617640 1 -- 172.21.15.125:0/3000835924 shutdown_connections 2023-10-30T13:05:10.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f0389617640 1 -- 172.21.15.125:0/3000835924 wait complete. 2023-10-30T13:05:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f0389617640 1 Processor -- start 2023-10-30T13:05:10.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f0389617640 1 -- start start 2023-10-30T13:05:10.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f0389617640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f038406f7a0 0x7f03841f0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:10.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.055+0000 7f0389617640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0384071110 0x7f0384146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:10.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0389617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03840c6bd0 0x7f038414d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:10.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0389617640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038414cef0 con 0x7f038406f7a0 2023-10-30T13:05:10.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0389617640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038414aa90 con 0x7f0384071110 2023-10-30T13:05:10.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0389617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038414c8c0 con 0x7f03840c6bd0 2023-10-30T13:05:10.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0382ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f038406f7a0 0x7f03841f0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:10.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0382ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f038406f7a0 0x7f03841f0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37402/0 (socket says 172.21.15.125:37402) 2023-10-30T13:05:10.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0382ffd640 1 -- 172.21.15.125:0/1780267853 learned_addr learned my addr 172.21.15.125:0/1780267853 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:10.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f037a7fc640 1 --2- 172.21.15.125:0/1780267853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0384071110 0x7f0384146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:10.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f03837fe640 1 --2- 172.21.15.125:0/1780267853 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03840c6bd0 0x7f038414d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:10.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0382ffd640 1 -- 172.21.15.125:0/1780267853 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03840c6bd0 msgr2=0x7f038414d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:10.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0382ffd640 1 --2- 172.21.15.125:0/1780267853 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03840c6bd0 0x7f038414d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:10.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0382ffd640 1 -- 172.21.15.125:0/1780267853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0384071110 msgr2=0x7f0384146a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:10.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0382ffd640 1 --2- 172.21.15.125:0/1780267853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0384071110 0x7f0384146a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:10.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0382ffd640 1 -- 172.21.15.125:0/1780267853 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036c009420 con 0x7f038406f7a0 2023-10-30T13:05:10.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0382ffd640 1 --2- 172.21.15.125:0/1780267853 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f038406f7a0 0x7f03841f0050 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f03740b3040 tx=0x7f037401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:10.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f037bfff640 1 -- 172.21.15.125:0/1780267853 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03740b0dd0 con 0x7f038406f7a0 2023-10-30T13:05:10.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f037bfff640 1 -- 172.21.15.125:0/1780267853 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03740bb460 con 0x7f038406f7a0 2023-10-30T13:05:10.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0389617640 1 -- 172.21.15.125:0/1780267853 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03740094d0 con 0x7f038406f7a0 2023-10-30T13:05:10.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f037bfff640 1 -- 172.21.15.125:0/1780267853 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03740bb810 con 0x7f038406f7a0 2023-10-30T13:05:10.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0389617640 1 -- 172.21.15.125:0/1780267853 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03841f0b50 con 0x7f038406f7a0 2023-10-30T13:05:10.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f0389617640 1 -- 172.21.15.125:0/1780267853 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f038414d030 con 0x7f038406f7a0 2023-10-30T13:05:10.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.059+0000 7f037bfff640 1 -- 172.21.15.125:0/1780267853 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f03740b83f0 con 0x7f038406f7a0 2023-10-30T13:05:10.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.063+0000 7f037bfff640 1 --2- 172.21.15.125:0/1780267853 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f03680664a0 0x7f0368068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:10.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.063+0000 7f037bfff640 1 -- 172.21.15.125:0/1780267853 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f0374121f10 con 0x7f038406f7a0 2023-10-30T13:05:10.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.063+0000 7f037a7fc640 1 --2- 172.21.15.125:0/1780267853 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f03680664a0 0x7f0368068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:10.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.063+0000 7f037a7fc640 1 --2- 172.21.15.125:0/1780267853 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f03680664a0 0x7f0368068960 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f03841e7470 tx=0x7f036c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:10.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.067+0000 7f037bfff640 1 -- 172.21.15.125:0/1780267853 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f038414d030 con 0x7f038406f7a0 2023-10-30T13:05:10.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.247+0000 7f0389617640 1 -- 172.21.15.125:0/1780267853 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f038414d030 con 0x7f038406f7a0 2023-10-30T13:05:10.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.247+0000 7f037bfff640 1 -- 172.21.15.125:0/1780267853 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f038414d030 con 0x7f038406f7a0 2023-10-30T13:05:10.251 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:10.251 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:07.674509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:10.251 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 159 2023-10-30T13:05:10.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.251+0000 7f0389617640 1 -- 172.21.15.125:0/1780267853 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f03680664a0 msgr2=0x7f0368068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:10.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.251+0000 7f0389617640 1 --2- 172.21.15.125:0/1780267853 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f03680664a0 0x7f0368068960 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f03841e7470 tx=0x7f036c008040 comp rx=0 tx=0).stop 2023-10-30T13:05:10.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.251+0000 7f0389617640 1 -- 172.21.15.125:0/1780267853 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f038406f7a0 msgr2=0x7f03841f0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:10.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.251+0000 7f0389617640 1 --2- 172.21.15.125:0/1780267853 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f038406f7a0 0x7f03841f0050 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f03740b3040 tx=0x7f037401bad0 comp rx=0 tx=0).stop 2023-10-30T13:05:10.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.251+0000 7f0389617640 1 -- 172.21.15.125:0/1780267853 shutdown_connections 2023-10-30T13:05:10.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.251+0000 7f0389617640 1 --2- 172.21.15.125:0/1780267853 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f03680664a0 0x7f0368068960 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:10.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.251+0000 7f0389617640 1 --2- 172.21.15.125:0/1780267853 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f03840c6bd0 0x7f038414d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:10.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.251+0000 7f0389617640 1 --2- 172.21.15.125:0/1780267853 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0384071110 0x7f0384146a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:10.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.251+0000 7f0389617640 1 --2- 172.21.15.125:0/1780267853 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f038406f7a0 0x7f03841f0050 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:10.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.251+0000 7f0389617640 1 -- 172.21.15.125:0/1780267853 >> 172.21.15.125:0/1780267853 conn(0x7f03840c7910 msgr2=0x7f03840c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:10.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.251+0000 7f0389617640 1 -- 172.21.15.125:0/1780267853 shutdown_connections 2023-10-30T13:05:10.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:10.251+0000 7f0389617640 1 -- 172.21.15.125:0/1780267853 wait complete. 2023-10-30T13:05:10.271 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 159, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 159, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8022": { "addr": "172.21.15.125:6837/2196617378", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2196617378, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2196617378, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8022, "incarnation": 156, "join_fscid": -1, "name": "c", "rank": 0, "state": "up:clientreplay", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:05:07.674509+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 8022 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:05:12.272 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:12.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.415+0000 7f019759e640 1 Processor -- start 2023-10-30T13:05:12.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f019759e640 1 -- start start 2023-10-30T13:05:12.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f019759e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f019815cd50 0x7f01980c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:12.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f019759e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019815a6c0 con 0x7f01980cefa0 2023-10-30T13:05:12.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f019759e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f019815d300 con 0x7f01981538f0 2023-10-30T13:05:12.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f019759e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f019815d5e0 con 0x7f019815cd50 2023-10-30T13:05:12.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0196d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f01980cefa0 0x7f01981640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37442/0 (socket says 172.21.15.125:37442) 2023-10-30T13:05:12.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0196d9d640 1 -- 172.21.15.125:0/2710361325 learned_addr learned my addr 172.21.15.125:0/2710361325 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:05:12.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f018dd9b640 1 --2- 172.21.15.125:0/2710361325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f019815cd50 0x7f01980c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:12.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 -- 172.21.15.125:0/2710361325 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631699300 0 0) 0x7f019815a6c0 con 0x7f01980cefa0 2023-10-30T13:05:12.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 -- 172.21.15.125:0/2710361325 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0184003530 con 0x7f01980cefa0 2023-10-30T13:05:12.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 -- 172.21.15.125:0/2710361325 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1196219990 0 0) 0x7f0184003530 con 0x7f01980cefa0 2023-10-30T13:05:12.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 -- 172.21.15.125:0/2710361325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f019815cd50 msgr2=0x7f01980c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:12.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 --2- 172.21.15.125:0/2710361325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f019815cd50 0x7f01980c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:12.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 -- 172.21.15.125:0/2710361325 >> v1:172.21.15.136:6789/0 conn(0x7f01981538f0 legacy=0x7f0198159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:12.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 -- 172.21.15.125:0/2710361325 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f019815d8c0 con 0x7f01980cefa0 2023-10-30T13:05:12.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 -- 172.21.15.125:0/2710361325 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f0190001680 con 0x7f01980cefa0 2023-10-30T13:05:12.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 -- 172.21.15.125:0/2710361325 >> v1:172.21.15.125:6789/0 conn(0x7f01980cefa0 legacy=0x7f01981640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:12.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 --2- 172.21.15.125:0/2710361325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01840039b0 0x7f0184003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:12.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 --2- 172.21.15.125:0/2710361325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0184004360 0x7f0184004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:12.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 --2- 172.21.15.125:0/2710361325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f019815cd50 0x7f0184005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:12.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 -- 172.21.15.125:0/2710361325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f019815d5e0 con 0x7f01840039b0 2023-10-30T13:05:12.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 -- 172.21.15.125:0/2710361325 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f018400b9c0 con 0x7f0184004360 2023-10-30T13:05:12.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0195d9b640 1 -- 172.21.15.125:0/2710361325 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f018400bb80 con 0x7f019815cd50 2023-10-30T13:05:12.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f019659c640 1 --2- 172.21.15.125:0/2710361325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01840039b0 0x7f0184003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:12.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f0196d9d640 1 --2- 172.21.15.125:0/2710361325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f019815cd50 0x7f0184005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:12.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f018dd9b640 1 --2- 172.21.15.125:0/2710361325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0184004360 0x7f0184004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:12.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f019659c640 1 -- 172.21.15.125:0/2710361325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f019815cd50 msgr2=0x7f0184005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:12.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f019659c640 1 --2- 172.21.15.125:0/2710361325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f019815cd50 0x7f0184005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:12.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f019659c640 1 -- 172.21.15.125:0/2710361325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0184004360 msgr2=0x7f0184004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:12.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f019659c640 1 --2- 172.21.15.125:0/2710361325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0184004360 0x7f0184004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:12.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f019659c640 1 -- 172.21.15.125:0/2710361325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0184003530 con 0x7f01840039b0 2023-10-30T13:05:12.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.419+0000 7f018dd9b640 1 --2- 172.21.15.125:0/2710361325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0184004360 0x7f0184004830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:05:12.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.423+0000 7f019659c640 1 --2- 172.21.15.125:0/2710361325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01840039b0 0x7f0184003d90 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f0188002010 tx=0x7f018801f050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2226c51db64adbd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:12.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.423+0000 7f0195d9b640 1 -- 172.21.15.125:0/2710361325 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01880a8c70 con 0x7f01840039b0 2023-10-30T13:05:12.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.423+0000 7f0195d9b640 1 -- 172.21.15.125:0/2710361325 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01880a8df0 con 0x7f01840039b0 2023-10-30T13:05:12.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.423+0000 7f0195d9b640 1 -- 172.21.15.125:0/2710361325 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01880b0680 con 0x7f01840039b0 2023-10-30T13:05:12.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.423+0000 7f019759e640 1 -- 172.21.15.125:0/2710361325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01840039b0 msgr2=0x7f0184003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:12.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.423+0000 7f019759e640 1 --2- 172.21.15.125:0/2710361325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01840039b0 0x7f0184003d90 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f0188002010 tx=0x7f018801f050 comp rx=0 tx=0).stop 2023-10-30T13:05:12.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.423+0000 7f0196d9d640 1 -- 172.21.15.125:0/2710361325 reap_dead start 2023-10-30T13:05:12.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.423+0000 7f019759e640 1 -- 172.21.15.125:0/2710361325 shutdown_connections 2023-10-30T13:05:12.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.423+0000 7f019759e640 1 -- 172.21.15.125:0/2710361325 >> 172.21.15.125:0/2710361325 conn(0x7f019806a640 msgr2=0x7f0198065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:12.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.423+0000 7f019759e640 1 -- 172.21.15.125:0/2710361325 shutdown_connections 2023-10-30T13:05:12.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.423+0000 7f019759e640 1 -- 172.21.15.125:0/2710361325 wait complete. 2023-10-30T13:05:12.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019759e640 1 Processor -- start 2023-10-30T13:05:12.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019759e640 1 -- start start 2023-10-30T13:05:12.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019759e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01840039b0 0x7f019815c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:12.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019759e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0198151120 0x7f0198147a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:12.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019759e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0198143b20 0x7f0198143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:12.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019759e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0198143650 con 0x7f01840039b0 2023-10-30T13:05:12.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019759e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01981f2280 con 0x7f0198143b20 2023-10-30T13:05:12.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019759e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01981458a0 con 0x7f0198151120 2023-10-30T13:05:12.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01840039b0 0x7f019815c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:12.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01840039b0 0x7f019815c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50318/0 (socket says 172.21.15.125:50318) 2023-10-30T13:05:12.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019659c640 1 -- 172.21.15.125:0/2093294483 learned_addr learned my addr 172.21.15.125:0/2093294483 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:12.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f018dd9b640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0198151120 0x7f0198147a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:12.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f0196d9d640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0198143b20 0x7f0198143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:12.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019659c640 1 -- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0198151120 msgr2=0x7f0198147a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:12.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019659c640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0198151120 0x7f0198147a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:12.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019659c640 1 -- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0198143b20 msgr2=0x7f0198143fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:12.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019659c640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0198143b20 0x7f0198143fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:12.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019659c640 1 -- 172.21.15.125:0/2093294483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01981f24d0 con 0x7f01840039b0 2023-10-30T13:05:12.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f0196d9d640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0198143b20 0x7f0198143fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:05:12.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019659c640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01840039b0 0x7f019815c360 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f01880b3040 tx=0x7f0184005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:12.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f018f7fe640 1 -- 172.21.15.125:0/2093294483 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01880a8c70 con 0x7f01840039b0 2023-10-30T13:05:12.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019759e640 1 -- 172.21.15.125:0/2093294483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01981f27b0 con 0x7f01840039b0 2023-10-30T13:05:12.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f018f7fe640 1 -- 172.21.15.125:0/2093294483 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01880050c0 con 0x7f01840039b0 2023-10-30T13:05:12.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f018f7fe640 1 -- 172.21.15.125:0/2093294483 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0188005430 con 0x7f01840039b0 2023-10-30T13:05:12.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.427+0000 7f019759e640 1 -- 172.21.15.125:0/2093294483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01981f2d70 con 0x7f01840039b0 2023-10-30T13:05:12.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.431+0000 7f019759e640 1 -- 172.21.15.125:0/2093294483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f019804e170 con 0x7f01840039b0 2023-10-30T13:05:12.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.431+0000 7f018f7fe640 1 -- 172.21.15.125:0/2093294483 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f01880b7040 con 0x7f01840039b0 2023-10-30T13:05:12.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.431+0000 7f018f7fe640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01780664a0 0x7f0178068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:12.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.431+0000 7f018f7fe640 1 -- 172.21.15.125:0/2093294483 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f0188121fd0 con 0x7f01840039b0 2023-10-30T13:05:12.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.431+0000 7f018dd9b640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01780664a0 0x7f0178068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:12.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.431+0000 7f018dd9b640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01780664a0 0x7f0178068960 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f01980ca400 tx=0x7f0180008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:12.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.435+0000 7f018f7fe640 1 -- 172.21.15.125:0/2093294483 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f01880eed40 con 0x7f01840039b0 2023-10-30T13:05:12.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.615+0000 7f019759e640 1 -- 172.21.15.125:0/2093294483 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f01980b2290 con 0x7f01840039b0 2023-10-30T13:05:12.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.615+0000 7f018f7fe640 1 -- 172.21.15.125:0/2093294483 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5730 (secure 0 0 0) 0x7f01880f2360 con 0x7f01840039b0 2023-10-30T13:05:12.616 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:12.617 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:07.674509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:12.618 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 159 2023-10-30T13:05:12.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.615+0000 7f019759e640 1 -- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01780664a0 msgr2=0x7f0178068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:12.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.615+0000 7f019759e640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01780664a0 0x7f0178068960 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f01980ca400 tx=0x7f0180008040 comp rx=0 tx=0).stop 2023-10-30T13:05:12.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.615+0000 7f019759e640 1 -- 172.21.15.125:0/2093294483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01840039b0 msgr2=0x7f019815c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:12.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.615+0000 7f019759e640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01840039b0 0x7f019815c360 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f01880b3040 tx=0x7f0184005800 comp rx=0 tx=0).stop 2023-10-30T13:05:12.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.615+0000 7f019759e640 1 -- 172.21.15.125:0/2093294483 shutdown_connections 2023-10-30T13:05:12.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.615+0000 7f019759e640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f01780664a0 0x7f0178068960 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:12.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.615+0000 7f019759e640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0198143b20 0x7f0198143fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:12.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.615+0000 7f019759e640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0198151120 0x7f0198147a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:12.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.615+0000 7f019759e640 1 --2- 172.21.15.125:0/2093294483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f01840039b0 0x7f019815c360 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:12.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.615+0000 7f019759e640 1 -- 172.21.15.125:0/2093294483 >> 172.21.15.125:0/2093294483 conn(0x7f019806a640 msgr2=0x7f01980caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:12.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.615+0000 7f019759e640 1 -- 172.21.15.125:0/2093294483 shutdown_connections 2023-10-30T13:05:12.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:12.615+0000 7f019759e640 1 -- 172.21.15.125:0/2093294483 wait complete. 2023-10-30T13:05:12.636 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:05:14.638 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:14.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.783+0000 7ff4eed0f640 1 Processor -- start 2023-10-30T13:05:14.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.783+0000 7ff4eed0f640 1 -- start start 2023-10-30T13:05:14.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.783+0000 7ff4eed0f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff4e81538a0 0x7ff4e815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:14.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.783+0000 7ff4eed0f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e80cb800 con 0x7ff4e80cf160 2023-10-30T13:05:14.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.783+0000 7ff4eed0f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4e815dc30 con 0x7ff4e80ca0e0 2023-10-30T13:05:14.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.783+0000 7ff4eed0f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4e80cb330 con 0x7ff4e81538a0 2023-10-30T13:05:14.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.783+0000 7ff4ecaab640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff4e81538a0 0x7ff4e815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:14.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.783+0000 7ff4ed2ac640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff4e80cf160 0x7ff4e8164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37446/0 (socket says 172.21.15.125:37446) 2023-10-30T13:05:14.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.783+0000 7ff4ed2ac640 1 -- 172.21.15.125:0/2533885306 learned_addr learned my addr 172.21.15.125:0/2533885306 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:05:14.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.783+0000 7ff4ecaab640 1 -- 172.21.15.125:0/2533885306 >> v1:172.21.15.136:6789/0 conn(0x7ff4e80ca0e0 legacy=0x7ff4e815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:14.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.783+0000 7ff4ecaab640 1 -- 172.21.15.125:0/2533885306 >> v1:172.21.15.125:6789/0 conn(0x7ff4e80cf160 legacy=0x7ff4e8164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:14.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.783+0000 7ff4ecaab640 1 -- 172.21.15.125:0/2533885306 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4e80cad70 con 0x7ff4e81538a0 2023-10-30T13:05:14.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.783+0000 7ff4e77fe640 1 -- 172.21.15.125:0/2533885306 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453800638 0 0) 0x7ff4e80cb800 con 0x7ff4e80cf160 2023-10-30T13:05:14.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.787+0000 7ff4ecaab640 1 --2- 172.21.15.125:0/2533885306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4e81538a0 0x7ff4e815d6f0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0097c0 tx=0x7ff4dc01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92912cd6a7eb1b63 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:14.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.787+0000 7ff4e77fe640 1 -- 172.21.15.125:0/2533885306 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc0a8970 con 0x7ff4e81538a0 2023-10-30T13:05:14.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.787+0000 7ff4e77fe640 1 -- 172.21.15.125:0/2533885306 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4dc0a8ad0 con 0x7ff4e81538a0 2023-10-30T13:05:14.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.787+0000 7ff4e77fe640 1 -- 172.21.15.125:0/2533885306 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc0a8e60 con 0x7ff4e81538a0 2023-10-30T13:05:14.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.787+0000 7ff4eed0f640 1 -- 172.21.15.125:0/2533885306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4e81538a0 msgr2=0x7ff4e815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:14.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.787+0000 7ff4eed0f640 1 --2- 172.21.15.125:0/2533885306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4e81538a0 0x7ff4e815d6f0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0097c0 tx=0x7ff4dc01ec70 comp rx=0 tx=0).stop 2023-10-30T13:05:14.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.787+0000 7ff4eed0f640 1 -- 172.21.15.125:0/2533885306 shutdown_connections 2023-10-30T13:05:14.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.787+0000 7ff4eed0f640 1 --2- 172.21.15.125:0/2533885306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4e81538a0 0x7ff4e815d6f0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:14.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.787+0000 7ff4eed0f640 1 -- 172.21.15.125:0/2533885306 >> 172.21.15.125:0/2533885306 conn(0x7ff4e806bfe0 msgr2=0x7ff4e806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:14.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.787+0000 7ff4eed0f640 1 -- 172.21.15.125:0/2533885306 shutdown_connections 2023-10-30T13:05:14.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.787+0000 7ff4eed0f640 1 -- 172.21.15.125:0/2533885306 wait complete. 2023-10-30T13:05:14.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.787+0000 7ff4eed0f640 1 Processor -- start 2023-10-30T13:05:14.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4eed0f640 1 -- start start 2023-10-30T13:05:14.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4eed0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4e80ca0e0 0x7ff4e8158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:14.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4eed0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4e80cf160 0x7ff4e80b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:14.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4eed0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4e81538a0 0x7ff4e81eee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:14.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4eed0f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e80c8ed0 con 0x7ff4e80cf160 2023-10-30T13:05:14.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4eed0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e81f2400 con 0x7ff4e81538a0 2023-10-30T13:05:14.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4eed0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4e8158e70 con 0x7ff4e80ca0e0 2023-10-30T13:05:14.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4e7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4e80cf160 0x7ff4e80b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:14.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4e7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4e80cf160 0x7ff4e80b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50324/0 (socket says 172.21.15.125:50324) 2023-10-30T13:05:14.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4e7fff640 1 -- 172.21.15.125:0/3299543015 learned_addr learned my addr 172.21.15.125:0/3299543015 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:14.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4ecaab640 1 --2- 172.21.15.125:0/3299543015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4e80ca0e0 0x7ff4e8158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:14.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4ed2ac640 1 --2- 172.21.15.125:0/3299543015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4e81538a0 0x7ff4e81eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:14.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4e7fff640 1 -- 172.21.15.125:0/3299543015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4e80ca0e0 msgr2=0x7ff4e8158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:14.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4e7fff640 1 --2- 172.21.15.125:0/3299543015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4e80ca0e0 0x7ff4e8158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:14.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4e7fff640 1 -- 172.21.15.125:0/3299543015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4e81538a0 msgr2=0x7ff4e81eee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:14.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4e7fff640 1 --2- 172.21.15.125:0/3299543015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4e81538a0 0x7ff4e81eee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:14.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4e7fff640 1 -- 172.21.15.125:0/3299543015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4dc009420 con 0x7ff4e80cf160 2023-10-30T13:05:14.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4e7fff640 1 --2- 172.21.15.125:0/3299543015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4e80cf160 0x7ff4e80b4450 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7ff4d400c3c0 tx=0x7ff4d400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:14.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4e57fa640 1 -- 172.21.15.125:0/3299543015 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d40052d0 con 0x7ff4e80cf160 2023-10-30T13:05:14.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4e57fa640 1 -- 172.21.15.125:0/3299543015 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d4005430 con 0x7ff4e80cf160 2023-10-30T13:05:14.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4e57fa640 1 -- 172.21.15.125:0/3299543015 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4003840 con 0x7ff4e80cf160 2023-10-30T13:05:14.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4eed0f640 1 -- 172.21.15.125:0/3299543015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4e81f2610 con 0x7ff4e80cf160 2023-10-30T13:05:14.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4eed0f640 1 -- 172.21.15.125:0/3299543015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4e81f2bd0 con 0x7ff4e80cf160 2023-10-30T13:05:14.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.791+0000 7ff4eed0f640 1 -- 172.21.15.125:0/3299543015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4e804e170 con 0x7ff4e80cf160 2023-10-30T13:05:14.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.795+0000 7ff4e57fa640 1 -- 172.21.15.125:0/3299543015 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4d40039a0 con 0x7ff4e80cf160 2023-10-30T13:05:14.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.795+0000 7ff4e57fa640 1 --2- 172.21.15.125:0/3299543015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4cc0664a0 0x7ff4cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:14.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.795+0000 7ff4e57fa640 1 -- 172.21.15.125:0/3299543015 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7ff4d4083780 con 0x7ff4e80cf160 2023-10-30T13:05:14.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.795+0000 7ff4ecaab640 1 --2- 172.21.15.125:0/3299543015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4cc0664a0 0x7ff4cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:14.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.795+0000 7ff4ecaab640 1 --2- 172.21.15.125:0/3299543015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4cc0664a0 0x7ff4cc068960 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0b2000 tx=0x7ff4dc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:14.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.799+0000 7ff4e57fa640 1 -- 172.21.15.125:0/3299543015 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff4d40506c0 con 0x7ff4e80cf160 2023-10-30T13:05:14.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.983+0000 7ff4eed0f640 1 -- 172.21.15.125:0/3299543015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4e815dc30 con 0x7ff4e80cf160 2023-10-30T13:05:14.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.983+0000 7ff4e57fa640 1 -- 172.21.15.125:0/3299543015 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 159 v159) v1 ==== 95+0+5730 (secure 0 0 0) 0x7ff4e815dc30 con 0x7ff4e80cf160 2023-10-30T13:05:14.986 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:14.986 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":159,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":159,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:07.674509+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:clientreplay","state_seq":33,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:14.987 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 159 2023-10-30T13:05:14.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.987+0000 7ff4eed0f640 1 -- 172.21.15.125:0/3299543015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4cc0664a0 msgr2=0x7ff4cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:14.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.987+0000 7ff4eed0f640 1 --2- 172.21.15.125:0/3299543015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4cc0664a0 0x7ff4cc068960 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0b2000 tx=0x7ff4dc008040 comp rx=0 tx=0).stop 2023-10-30T13:05:14.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.987+0000 7ff4eed0f640 1 -- 172.21.15.125:0/3299543015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4e80cf160 msgr2=0x7ff4e80b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:14.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.987+0000 7ff4eed0f640 1 --2- 172.21.15.125:0/3299543015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4e80cf160 0x7ff4e80b4450 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7ff4d400c3c0 tx=0x7ff4d400c880 comp rx=0 tx=0).stop 2023-10-30T13:05:14.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.987+0000 7ff4eed0f640 1 -- 172.21.15.125:0/3299543015 shutdown_connections 2023-10-30T13:05:14.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.987+0000 7ff4eed0f640 1 --2- 172.21.15.125:0/3299543015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4cc0664a0 0x7ff4cc068960 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:14.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.987+0000 7ff4eed0f640 1 --2- 172.21.15.125:0/3299543015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4e81538a0 0x7ff4e81eee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:14.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.987+0000 7ff4eed0f640 1 --2- 172.21.15.125:0/3299543015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4e80cf160 0x7ff4e80b4450 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:14.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.987+0000 7ff4eed0f640 1 --2- 172.21.15.125:0/3299543015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4e80ca0e0 0x7ff4e8158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:14.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.987+0000 7ff4eed0f640 1 -- 172.21.15.125:0/3299543015 >> 172.21.15.125:0/3299543015 conn(0x7ff4e806bfe0 msgr2=0x7ff4e80c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:14.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.987+0000 7ff4eed0f640 1 -- 172.21.15.125:0/3299543015 shutdown_connections 2023-10-30T13:05:14.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:14.987+0000 7ff4eed0f640 1 -- 172.21.15.125:0/3299543015 wait complete. 2023-10-30T13:05:15.006 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:05:17.007 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:17.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.155+0000 7f2f46a09640 1 Processor -- start 2023-10-30T13:05:17.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.155+0000 7f2f46a09640 1 -- start start 2023-10-30T13:05:17.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.155+0000 7f2f46a09640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2f400c05d0 0x7f2f4013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:17.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f46a09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f400c6f60 0x7f2f4013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:17.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f46a09640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f400c0a50 con 0x7f2f400c6b80 2023-10-30T13:05:17.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f46a09640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f4006f6e0 con 0x7f2f400c05d0 2023-10-30T13:05:17.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f46a09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4006f9c0 con 0x7f2f400c6f60 2023-10-30T13:05:17.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3ffff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2f400c6b80 0x7f2f400bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37462/0 (socket says 172.21.15.125:37462) 2023-10-30T13:05:17.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3ffff640 1 -- 172.21.15.125:0/834771891 learned_addr learned my addr 172.21.15.125:0/834771891 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:05:17.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f44fa6640 1 --2- 172.21.15.125:0/834771891 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f400c6f60 0x7f2f4013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:17.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3f7fe640 1 --2- 172.21.15.125:0/834771891 >> v2:172.21.15.136:3300/0 conn(0x7f2f400c05d0 0x7f2f4013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:17.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 -- 172.21.15.125:0/834771891 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705245604 0 0) 0x7f2f400c0a50 con 0x7f2f400c6b80 2023-10-30T13:05:17.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 -- 172.21.15.125:0/834771891 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f28003530 con 0x7f2f400c6b80 2023-10-30T13:05:17.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 -- 172.21.15.125:0/834771891 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1221735228 0 0) 0x7f2f28003530 con 0x7f2f400c6b80 2023-10-30T13:05:17.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 -- 172.21.15.125:0/834771891 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f400c6f60 msgr2=0x7f2f4013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:17.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 --2- 172.21.15.125:0/834771891 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f400c6f60 0x7f2f4013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:17.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 -- 172.21.15.125:0/834771891 >> v2:172.21.15.136:3300/0 conn(0x7f2f400c05d0 msgr2=0x7f2f4013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:17.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 --2- 172.21.15.125:0/834771891 >> v2:172.21.15.136:3300/0 conn(0x7f2f400c05d0 0x7f2f4013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:17.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 -- 172.21.15.125:0/834771891 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f4006fca0 con 0x7f2f400c6b80 2023-10-30T13:05:17.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 -- 172.21.15.125:0/834771891 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f2f30001680 con 0x7f2f400c6b80 2023-10-30T13:05:17.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 -- 172.21.15.125:0/834771891 >> v1:172.21.15.125:6789/0 conn(0x7f2f400c6b80 legacy=0x7f2f400bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:17.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 --2- 172.21.15.125:0/834771891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f28003980 0x7f2f28003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:17.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 --2- 172.21.15.125:0/834771891 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f400c6f60 0x7f2f280047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:17.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 --2- 172.21.15.125:0/834771891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f28004d10 0x7f2f28005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:17.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 -- 172.21.15.125:0/834771891 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4006f6e0 con 0x7f2f28004d10 2023-10-30T13:05:17.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 -- 172.21.15.125:0/834771891 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4006f9c0 con 0x7f2f28003980 2023-10-30T13:05:17.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3effd640 1 -- 172.21.15.125:0/834771891 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f2800b990 con 0x7f2f400c6f60 2023-10-30T13:05:17.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f44fa6640 1 --2- 172.21.15.125:0/834771891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f28004d10 0x7f2f28005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:17.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3f7fe640 1 --2- 172.21.15.125:0/834771891 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f400c6f60 0x7f2f280047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:17.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f3ffff640 1 --2- 172.21.15.125:0/834771891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f28003980 0x7f2f28003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:17.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f44fa6640 1 -- 172.21.15.125:0/834771891 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f400c6f60 msgr2=0x7f2f280047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:17.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f44fa6640 1 --2- 172.21.15.125:0/834771891 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f400c6f60 0x7f2f280047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:17.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f44fa6640 1 -- 172.21.15.125:0/834771891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f28003980 msgr2=0x7f2f28003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:17.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f44fa6640 1 --2- 172.21.15.125:0/834771891 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f28003980 0x7f2f28003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:17.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.159+0000 7f2f44fa6640 1 -- 172.21.15.125:0/834771891 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f28003530 con 0x7f2f28004d10 2023-10-30T13:05:17.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.163+0000 7f2f44fa6640 1 --2- 172.21.15.125:0/834771891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f28004d10 0x7f2f28005190 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f2f38009fc0 tx=0x7f2f3801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244eacc4c12726d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:17.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.163+0000 7f2f3effd640 1 -- 172.21.15.125:0/834771891 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f380a5c70 con 0x7f2f28004d10 2023-10-30T13:05:17.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.163+0000 7f2f3effd640 1 -- 172.21.15.125:0/834771891 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3800f040 con 0x7f2f28004d10 2023-10-30T13:05:17.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.163+0000 7f2f3effd640 1 -- 172.21.15.125:0/834771891 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f380ad440 con 0x7f2f28004d10 2023-10-30T13:05:17.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.163+0000 7f2f46a09640 1 -- 172.21.15.125:0/834771891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f28004d10 msgr2=0x7f2f28005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:17.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.163+0000 7f2f46a09640 1 --2- 172.21.15.125:0/834771891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f28004d10 0x7f2f28005190 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f2f38009fc0 tx=0x7f2f3801cc60 comp rx=0 tx=0).stop 2023-10-30T13:05:17.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.163+0000 7f2f44fa6640 1 -- 172.21.15.125:0/834771891 reap_dead start 2023-10-30T13:05:17.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.163+0000 7f2f46a09640 1 -- 172.21.15.125:0/834771891 shutdown_connections 2023-10-30T13:05:17.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.163+0000 7f2f46a09640 1 -- 172.21.15.125:0/834771891 >> 172.21.15.125:0/834771891 conn(0x7f2f400c7910 msgr2=0x7f2f400c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:17.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.163+0000 7f2f46a09640 1 -- 172.21.15.125:0/834771891 shutdown_connections 2023-10-30T13:05:17.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.163+0000 7f2f46a09640 1 -- 172.21.15.125:0/834771891 wait complete. 2023-10-30T13:05:17.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.163+0000 7f2f46a09640 1 Processor -- start 2023-10-30T13:05:17.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.163+0000 7f2f46a09640 1 -- start start 2023-10-30T13:05:17.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f46a09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f28004d10 0x7f2f4014af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:17.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f46a09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4014de90 0x7f2f401f2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:17.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f46a09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f401f3090 0x7f2f401305b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:17.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f46a09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400c9c80 con 0x7f2f4014de90 2023-10-30T13:05:17.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f46a09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f40132210 con 0x7f2f401f3090 2023-10-30T13:05:17.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f46a09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f401f23e0 con 0x7f2f28004d10 2023-10-30T13:05:17.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f3f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4014de90 0x7f2f401f2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:17.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f3f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4014de90 0x7f2f401f2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50336/0 (socket says 172.21.15.125:50336) 2023-10-30T13:05:17.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f3f7fe640 1 -- 172.21.15.125:0/3989947864 learned_addr learned my addr 172.21.15.125:0/3989947864 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:17.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f3ffff640 1 --2- 172.21.15.125:0/3989947864 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f28004d10 0x7f2f4014af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:17.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f44fa6640 1 --2- 172.21.15.125:0/3989947864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f401f3090 0x7f2f401305b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:17.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f3f7fe640 1 -- 172.21.15.125:0/3989947864 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f28004d10 msgr2=0x7f2f4014af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:17.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f3f7fe640 1 --2- 172.21.15.125:0/3989947864 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f28004d10 0x7f2f4014af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:17.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f3f7fe640 1 -- 172.21.15.125:0/3989947864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f401f3090 msgr2=0x7f2f401305b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:17.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f3f7fe640 1 --2- 172.21.15.125:0/3989947864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f401f3090 0x7f2f401305b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:17.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f3f7fe640 1 -- 172.21.15.125:0/3989947864 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f401f26c0 con 0x7f2f4014de90 2023-10-30T13:05:17.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f3f7fe640 1 --2- 172.21.15.125:0/3989947864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4014de90 0x7f2f401f2b50 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f2f34011620 tx=0x7f2f34011ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:17.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f3cff9640 1 -- 172.21.15.125:0/3989947864 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3400f920 con 0x7f2f4014de90 2023-10-30T13:05:17.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f3cff9640 1 -- 172.21.15.125:0/3989947864 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f3400fa80 con 0x7f2f4014de90 2023-10-30T13:05:17.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f3cff9640 1 -- 172.21.15.125:0/3989947864 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3400fe50 con 0x7f2f4014de90 2023-10-30T13:05:17.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f46a09640 1 -- 172.21.15.125:0/3989947864 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f400ca540 con 0x7f2f4014de90 2023-10-30T13:05:17.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f46a09640 1 -- 172.21.15.125:0/3989947864 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f4014d8c0 con 0x7f2f4014de90 2023-10-30T13:05:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f46a09640 1 -- 172.21.15.125:0/3989947864 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f401f2850 con 0x7f2f4014de90 2023-10-30T13:05:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.167+0000 7f2f3cff9640 1 -- 172.21.15.125:0/3989947864 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2f34015460 con 0x7f2f4014de90 2023-10-30T13:05:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.171+0000 7f2f3cff9640 1 --2- 172.21.15.125:0/3989947864 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f2c0664a0 0x7f2f2c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:17.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.171+0000 7f2f3cff9640 1 -- 172.21.15.125:0/3989947864 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f2f34084ee0 con 0x7f2f4014de90 2023-10-30T13:05:17.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.171+0000 7f2f3ffff640 1 --2- 172.21.15.125:0/3989947864 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f2c0664a0 0x7f2f2c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:17.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.171+0000 7f2f3ffff640 1 --2- 172.21.15.125:0/3989947864 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f2c0664a0 0x7f2f2c068960 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f2f30000c10 tx=0x7f2f30009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:17.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.175+0000 7f2f3cff9640 1 -- 172.21.15.125:0/3989947864 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2f34051c40 con 0x7f2f4014de90 2023-10-30T13:05:17.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.355+0000 7f2f46a09640 1 -- 172.21.15.125:0/3989947864 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f4004e170 con 0x7f2f4014de90 2023-10-30T13:05:17.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.355+0000 7f2f3cff9640 1 -- 172.21.15.125:0/3989947864 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f2f34055260 con 0x7f2f4014de90 2023-10-30T13:05:17.358 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:17.358 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:15.605070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:17.358 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 160 2023-10-30T13:05:17.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.359+0000 7f2f46a09640 1 -- 172.21.15.125:0/3989947864 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f2c0664a0 msgr2=0x7f2f2c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:17.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.359+0000 7f2f46a09640 1 --2- 172.21.15.125:0/3989947864 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f2c0664a0 0x7f2f2c068960 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f2f30000c10 tx=0x7f2f30009040 comp rx=0 tx=0).stop 2023-10-30T13:05:17.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.359+0000 7f2f46a09640 1 -- 172.21.15.125:0/3989947864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4014de90 msgr2=0x7f2f401f2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:17.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.359+0000 7f2f46a09640 1 --2- 172.21.15.125:0/3989947864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4014de90 0x7f2f401f2b50 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7f2f34011620 tx=0x7f2f34011ae0 comp rx=0 tx=0).stop 2023-10-30T13:05:17.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.359+0000 7f2f46a09640 1 -- 172.21.15.125:0/3989947864 shutdown_connections 2023-10-30T13:05:17.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.359+0000 7f2f46a09640 1 --2- 172.21.15.125:0/3989947864 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f2c0664a0 0x7f2f2c068960 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:17.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.359+0000 7f2f46a09640 1 --2- 172.21.15.125:0/3989947864 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f401f3090 0x7f2f401305b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:17.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.359+0000 7f2f46a09640 1 --2- 172.21.15.125:0/3989947864 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f4014de90 0x7f2f401f2b50 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:17.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.359+0000 7f2f46a09640 1 --2- 172.21.15.125:0/3989947864 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f28004d10 0x7f2f4014af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:17.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.359+0000 7f2f46a09640 1 -- 172.21.15.125:0/3989947864 >> 172.21.15.125:0/3989947864 conn(0x7f2f400c7910 msgr2=0x7f2f400c19e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:17.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.359+0000 7f2f46a09640 1 -- 172.21.15.125:0/3989947864 shutdown_connections 2023-10-30T13:05:17.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:17.359+0000 7f2f46a09640 1 -- 172.21.15.125:0/3989947864 wait complete. 2023-10-30T13:05:17.379 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=0, replacing gid=7866 2023-10-30T13:05:17.379 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.e 2023-10-30T13:05:22.587 INFO:tasks.ceph.mds.c.smithi125.stderr:2023-10-30T13:05:22.587+0000 7f67ea472640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:05:22.955 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:05:22.961 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:05:25.632 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 12% (61/496) Updating files: 13% (65/496) Updating files: 14% (70/496) Updating files: 15% (75/496) Updating files: 16% (80/496) Updating files: 17% (85/496) Updating files: 18% (90/496) Updating files: 19% (95/496) Updating files: 20% (100/496) Updating files: 21% (105/496) Updating files: 22% (110/496) Updating files: 23% (115/496) Updating files: 24% (120/496) Updating files: 25% (124/496) Updating files: 26% (129/496) Updating files: 27% (134/496) Updating files: 28% (139/496) Updating files: 29% (144/496) Updating files: 30% (149/496) Updating files: 31% (154/496) Updating files: 32% (159/496) Updating files: 33% (164/496) Updating files: 34% (169/496) Updating files: 35% (174/496) Updating files: 36% (179/496) Updating files: 37% (184/496) Updating files: 38% (189/496) Updating files: 39% (194/496) Updating files: 40% (199/496) Updating files: 41% (204/496) Updating files: 42% (209/496) Updating files: 43% (214/496) Updating files: 44% (219/496) Updating files: 44% (222/496) Updating files: 45% (224/496) Updating files: 46% (229/496) Updating files: 47% (234/496) Updating files: 48% (239/496) Updating files: 49% (244/496) Updating files: 50% (248/496) Updating files: 51% (253/496) Updating files: 52% (258/496) Updating files: 53% (263/496) Updating files: 54% (268/496) Updating files: 55% (273/496) Updating files: 56% (278/496) Updating files: 57% (283/496) Updating files: 58% (288/496) Updating files: 59% (293/496) Updating files: 60% (298/496) Updating files: 61% (303/496) Updating files: 62% (308/496) Updating files: 63% (313/496) Updating files: 64% (318/496) Updating files: 65% (323/496) Updating files: 66% (328/496) Updating files: 67% (333/496) Updating files: 67% (337/496) Updating files: 68% (338/496) Updating files: 69% (343/496) Updating files: 70% (348/496) Updating files: 71% (353/496) Updating files: 72% (358/496) Updating files: 73% (363/496) Updating files: 74% (368/496) Updating files: 75% (372/496) Updating files: 76% (377/496) Updating files: 77% (382/496) Updating files: 78% (387/496) Updating files: 78% (388/496) Updating files: 79% (392/496) Updating files: 80% (397/496) Updating files: 81% (402/496) Updating files: 82% (407/496) Updating files: 83% (412/496) Updating files: 84% (417/496) Updating files: 85% (422/496) Updating files: 86% (427/496) Updating files: 87% (432/496) Updating files: 88% (437/496) Updating files: 89% (442/496) Updating files: 90% (447/496) Updating files: 91% (452/496) Updating files: 92% (457/496) Updating files: 93% (462/496) Updating files: 94% (467/496) Updating files: 95% (472/496) Updating files: 96% (477/496) Updating files: 96% (478/496) Updating files: 97% (482/496) Updating files: 98% (487/496) Updating files: 99% (492/496) Updating files: 100% (496/496) Updating files: 100% (496/496), done. 2023-10-30T13:05:26.380 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T13:05:26.380 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T13:05:26.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T13:05:26.385 INFO:tasks.ceph.mds.e:Started 2023-10-30T13:05:26.385 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:26.470 INFO:tasks.ceph.mds.e.smithi125.stdout:starting mds.e at 2023-10-30T13:05:26.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.523+0000 7f6e9759e640 1 Processor -- start 2023-10-30T13:05:26.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9759e640 1 -- start start 2023-10-30T13:05:26.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9759e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6e98153750 0x7f6e980c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:26.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9759e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e9806afa0 0x7f6e980ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:26.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9759e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6e98159930 0x7f6e98159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:26.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9759e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e98153b30 con 0x7f6e98153750 2023-10-30T13:05:26.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9759e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e980b2480 con 0x7f6e98159930 2023-10-30T13:05:26.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9759e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9806fdd0 con 0x7f6e9806afa0 2023-10-30T13:05:26.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6e98153750 0x7f6e980c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:26.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6e98153750 0x7f6e980c9db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51704/0 (socket says 172.21.15.125:51704) 2023-10-30T13:05:26.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9659c640 1 -- 172.21.15.125:0/256923839 learned_addr learned my addr 172.21.15.125:0/256923839 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:26.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e95d9b640 1 --2- 172.21.15.125:0/256923839 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e9806afa0 0x7f6e980ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:26.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e96d9d640 1 --2- 172.21.15.125:0/256923839 >> v2:172.21.15.136:3300/0 conn(0x7f6e98159930 0x7f6e98159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:26.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9659c640 1 -- 172.21.15.125:0/256923839 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e9806afa0 msgr2=0x7f6e980ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:26.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9659c640 1 --2- 172.21.15.125:0/256923839 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e9806afa0 0x7f6e980ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:26.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9659c640 1 -- 172.21.15.125:0/256923839 >> v2:172.21.15.136:3300/0 conn(0x7f6e98159930 msgr2=0x7f6e98159dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:26.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9659c640 1 --2- 172.21.15.125:0/256923839 >> v2:172.21.15.136:3300/0 conn(0x7f6e98159930 0x7f6e98159dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:26.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.527+0000 7f6e9659c640 1 -- 172.21.15.125:0/256923839 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9815a300 con 0x7f6e98153750 2023-10-30T13:05:26.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9659c640 1 --2- 172.21.15.125:0/256923839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e98153750 0x7f6e980c9db0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f6e980c4200 tx=0x7f6e8801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14286f07848144ff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:26.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9559a640 1 -- 172.21.15.125:0/256923839 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e880ab600 con 0x7f6e98153750 2023-10-30T13:05:26.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9559a640 1 -- 172.21.15.125:0/256923839 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e880a7050 con 0x7f6e98153750 2023-10-30T13:05:26.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9559a640 1 -- 172.21.15.125:0/256923839 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e880ab980 con 0x7f6e98153750 2023-10-30T13:05:26.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9759e640 1 -- 172.21.15.125:0/256923839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e98153750 msgr2=0x7f6e980c9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:26.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9759e640 1 --2- 172.21.15.125:0/256923839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e98153750 0x7f6e980c9db0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f6e980c4200 tx=0x7f6e8801c500 comp rx=0 tx=0).stop 2023-10-30T13:05:26.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9759e640 1 -- 172.21.15.125:0/256923839 shutdown_connections 2023-10-30T13:05:26.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9759e640 1 --2- 172.21.15.125:0/256923839 >> v2:172.21.15.136:3300/0 conn(0x7f6e98159930 0x7f6e98159dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:26.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9759e640 1 --2- 172.21.15.125:0/256923839 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e9806afa0 0x7f6e980ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:26.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9759e640 1 --2- 172.21.15.125:0/256923839 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e98153750 0x7f6e980c9db0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:26.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9759e640 1 -- 172.21.15.125:0/256923839 >> 172.21.15.125:0/256923839 conn(0x7f6e9806d560 msgr2=0x7f6e9806d970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:26.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9759e640 1 -- 172.21.15.125:0/256923839 shutdown_connections 2023-10-30T13:05:26.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9759e640 1 -- 172.21.15.125:0/256923839 wait complete. 2023-10-30T13:05:26.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9759e640 1 Processor -- start 2023-10-30T13:05:26.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.531+0000 7f6e9759e640 1 -- start start 2023-10-30T13:05:26.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9759e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e9806afa0 0x7f6e9814aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:26.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9759e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e98153750 0x7f6e981496b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:26.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9759e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e98159930 0x7f6e981f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:26.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9759e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980b03d0 con 0x7f6e9806afa0 2023-10-30T13:05:26.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9759e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980b19f0 con 0x7f6e98153750 2023-10-30T13:05:26.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9759e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e980b0110 con 0x7f6e98159930 2023-10-30T13:05:26.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e9806afa0 0x7f6e9814aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:26.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e9806afa0 0x7f6e9814aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51714/0 (socket says 172.21.15.125:51714) 2023-10-30T13:05:26.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9659c640 1 -- 172.21.15.125:0/1317584107 learned_addr learned my addr 172.21.15.125:0/1317584107 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:26.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e96d9d640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e98159930 0x7f6e981f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:26.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e95d9b640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e98153750 0x7f6e981496b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:26.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9659c640 1 -- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e98159930 msgr2=0x7f6e981f2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:26.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9659c640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e98159930 0x7f6e981f2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:26.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9659c640 1 -- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e98153750 msgr2=0x7f6e981496b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:26.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9659c640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e98153750 0x7f6e981496b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:26.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9659c640 1 -- 172.21.15.125:0/1317584107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e880094d0 con 0x7f6e9806afa0 2023-10-30T13:05:26.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e96d9d640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e98159930 0x7f6e981f2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:05:26.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9659c640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e9806afa0 0x7f6e9814aa60 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f6e880ae040 tx=0x7f6e880192a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:26.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e86ffd640 1 -- 172.21.15.125:0/1317584107 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e880b4440 con 0x7f6e9806afa0 2023-10-30T13:05:26.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e86ffd640 1 -- 172.21.15.125:0/1317584107 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e880b45a0 con 0x7f6e9806afa0 2023-10-30T13:05:26.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e86ffd640 1 -- 172.21.15.125:0/1317584107 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e880b4930 con 0x7f6e9806afa0 2023-10-30T13:05:26.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9759e640 1 -- 172.21.15.125:0/1317584107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e980b10c0 con 0x7f6e9806afa0 2023-10-30T13:05:26.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9759e640 1 -- 172.21.15.125:0/1317584107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e980b1680 con 0x7f6e9806afa0 2023-10-30T13:05:26.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e9759e640 1 -- 172.21.15.125:0/1317584107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e980b1810 con 0x7f6e9806afa0 2023-10-30T13:05:26.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.535+0000 7f6e86ffd640 1 -- 172.21.15.125:0/1317584107 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6e880b2070 con 0x7f6e9806afa0 2023-10-30T13:05:26.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.539+0000 7f6e86ffd640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e780664d0 0x7f6e78068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:26.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.539+0000 7f6e86ffd640 1 -- 172.21.15.125:0/1317584107 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f6e8811e480 con 0x7f6e9806afa0 2023-10-30T13:05:26.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.539+0000 7f6e95d9b640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e780664d0 0x7f6e78068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:26.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.539+0000 7f6e95d9b640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e780664d0 0x7f6e78068990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6e9814e370 tx=0x7f6e8c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:26.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.543+0000 7f6e86ffd640 1 -- 172.21.15.125:0/1317584107 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6e880eb270 con 0x7f6e9806afa0 2023-10-30T13:05:26.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.775+0000 7f6e9759e640 1 -- 172.21.15.125:0/1317584107 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e9806f400 con 0x7f6e9806afa0 2023-10-30T13:05:26.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.775+0000 7f6e86ffd640 1 -- 172.21.15.125:0/1317584107 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f6e880ee890 con 0x7f6e9806afa0 2023-10-30T13:05:26.778 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:26.778 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8178,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:15.605070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:26.779 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 161 2023-10-30T13:05:26.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.779+0000 7f6e9759e640 1 -- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e780664d0 msgr2=0x7f6e78068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.779+0000 7f6e9759e640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e780664d0 0x7f6e78068990 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f6e9814e370 tx=0x7f6e8c008040 comp rx=0 tx=0).stop 2023-10-30T13:05:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.779+0000 7f6e9759e640 1 -- 172.21.15.125:0/1317584107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e9806afa0 msgr2=0x7f6e9814aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.779+0000 7f6e9759e640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e9806afa0 0x7f6e9814aa60 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f6e880ae040 tx=0x7f6e880192a0 comp rx=0 tx=0).stop 2023-10-30T13:05:26.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.779+0000 7f6e9759e640 1 -- 172.21.15.125:0/1317584107 shutdown_connections 2023-10-30T13:05:26.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.779+0000 7f6e9759e640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e780664d0 0x7f6e78068990 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:26.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.779+0000 7f6e9759e640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e98159930 0x7f6e981f2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:26.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.779+0000 7f6e9759e640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e98153750 0x7f6e981496b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:26.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.779+0000 7f6e9759e640 1 --2- 172.21.15.125:0/1317584107 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e9806afa0 0x7f6e9814aa60 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:26.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.779+0000 7f6e9759e640 1 -- 172.21.15.125:0/1317584107 >> 172.21.15.125:0/1317584107 conn(0x7f6e9806d560 msgr2=0x7f6e980c9d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:26.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.779+0000 7f6e9759e640 1 -- 172.21.15.125:0/1317584107 shutdown_connections 2023-10-30T13:05:26.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:26.779+0000 7f6e9759e640 1 -- 172.21.15.125:0/1317584107 wait complete. 2023-10-30T13:05:26.799 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at f6c746c314d 0.78 2023-10-30T13:05:26.803 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T13:05:26.804 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 12 secs before thrashing 2023-10-30T13:05:26.808 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.78 2023-10-30T13:05:27.721 INFO:tasks.workunit.client.0.smithi125.stdout:v0.79 2023-10-30T13:05:27.722 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 79 -eq 48 ] 2023-10-30T13:05:27.722 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.79 2023-10-30T13:05:27.722 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.79 2023-10-30T13:05:27.722 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.79 2023-10-30T13:05:38.805 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:38.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.943+0000 7f94fc446640 1 Processor -- start 2023-10-30T13:05:38.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.943+0000 7f94fc446640 1 -- start start 2023-10-30T13:05:38.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.943+0000 7f94fc446640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f94f406f7a0 0x7f94f406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:38.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.943+0000 7f94fc446640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f94f4071110 0x7f94f4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:38.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.943+0000 7f94fc446640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f4066cf0 con 0x7f94f40c6bd0 2023-10-30T13:05:38.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.943+0000 7f94fc446640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f40700c0 con 0x7f94f406f7a0 2023-10-30T13:05:38.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.943+0000 7f94fc446640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f40c70d0 con 0x7f94f4071110 2023-10-30T13:05:38.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.943+0000 7f94fa1e2640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f94f406f7a0 0x7f94f406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:38.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.947+0000 7f94fa1e2640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f94f406f7a0 0x7f94f406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55328/0 (socket says 172.21.15.125:55328) 2023-10-30T13:05:38.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.947+0000 7f94f99e1640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f94f40c6bd0 0x7f94f4069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35962/0 (socket says 172.21.15.125:35962) 2023-10-30T13:05:38.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.947+0000 7f94fa1e2640 1 -- 172.21.15.125:0/2816997831 learned_addr learned my addr 172.21.15.125:0/2816997831 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:38.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.947+0000 7f94fa9e3640 1 --2- 172.21.15.125:0/2816997831 >> v2:172.21.15.136:3300/0 conn(0x7f94f4071110 0x7f94f4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:38.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.947+0000 7f94fa1e2640 1 -- 172.21.15.125:0/2816997831 >> v2:172.21.15.136:3300/0 conn(0x7f94f4071110 msgr2=0x7f94f4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:38.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.947+0000 7f94fa1e2640 1 --2- 172.21.15.125:0/2816997831 >> v2:172.21.15.136:3300/0 conn(0x7f94f4071110 0x7f94f4158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:38.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.947+0000 7f94fa1e2640 1 -- 172.21.15.125:0/2816997831 >> v1:172.21.15.125:6789/0 conn(0x7f94f40c6bd0 legacy=0x7f94f4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:38.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.947+0000 7f94fa1e2640 1 -- 172.21.15.125:0/2816997831 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f40c7390 con 0x7f94f406f7a0 2023-10-30T13:05:38.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.947+0000 7f94f91e0640 1 -- 172.21.15.125:0/2816997831 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289820840 0 0) 0x7f94f4066cf0 con 0x7f94f40c6bd0 2023-10-30T13:05:38.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.947+0000 7f94fa1e2640 1 --2- 172.21.15.125:0/2816997831 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94f406f7a0 0x7f94f406fb80 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f94f406cdd0 tx=0x7f94f001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3afa8ba94219f09 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:38.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.947+0000 7f94f91e0640 1 -- 172.21.15.125:0/2816997831 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f00a89f0 con 0x7f94f406f7a0 2023-10-30T13:05:38.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.947+0000 7f94f91e0640 1 -- 172.21.15.125:0/2816997831 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f00a8b50 con 0x7f94f406f7a0 2023-10-30T13:05:38.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.947+0000 7f94f91e0640 1 -- 172.21.15.125:0/2816997831 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f00b0440 con 0x7f94f406f7a0 2023-10-30T13:05:38.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 -- 172.21.15.125:0/2816997831 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94f406f7a0 msgr2=0x7f94f406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:38.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 --2- 172.21.15.125:0/2816997831 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94f406f7a0 0x7f94f406fb80 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f94f406cdd0 tx=0x7f94f001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:05:38.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 -- 172.21.15.125:0/2816997831 shutdown_connections 2023-10-30T13:05:38.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 --2- 172.21.15.125:0/2816997831 >> v2:172.21.15.136:3300/0 conn(0x7f94f4071110 0x7f94f4158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:38.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 --2- 172.21.15.125:0/2816997831 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94f406f7a0 0x7f94f406fb80 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:38.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 -- 172.21.15.125:0/2816997831 >> 172.21.15.125:0/2816997831 conn(0x7f94f40c7910 msgr2=0x7f94f40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:38.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 -- 172.21.15.125:0/2816997831 shutdown_connections 2023-10-30T13:05:38.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 -- 172.21.15.125:0/2816997831 wait complete. 2023-10-30T13:05:38.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 Processor -- start 2023-10-30T13:05:38.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 -- start start 2023-10-30T13:05:38.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94f406f7a0 0x7f94f40c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:38.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94f4071110 0x7f94f4154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:38.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94f40c6bd0 0x7f94f41529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:38.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f41f1e00 con 0x7f94f4071110 2023-10-30T13:05:38.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f41f2940 con 0x7f94f406f7a0 2023-10-30T13:05:38.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fc446640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f41554d0 con 0x7f94f40c6bd0 2023-10-30T13:05:38.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94f99e1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94f4071110 0x7f94f4154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:38.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94f99e1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94f4071110 0x7f94f4154980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55334/0 (socket says 172.21.15.125:55334) 2023-10-30T13:05:38.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94f99e1640 1 -- 172.21.15.125:0/342333684 learned_addr learned my addr 172.21.15.125:0/342333684 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:38.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fa1e2640 1 --2- 172.21.15.125:0/342333684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94f406f7a0 0x7f94f40c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:38.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94fa9e3640 1 --2- 172.21.15.125:0/342333684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94f40c6bd0 0x7f94f41529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:38.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94f99e1640 1 -- 172.21.15.125:0/342333684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94f40c6bd0 msgr2=0x7f94f41529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:38.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94f99e1640 1 --2- 172.21.15.125:0/342333684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94f40c6bd0 0x7f94f41529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:38.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94f99e1640 1 -- 172.21.15.125:0/342333684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94f406f7a0 msgr2=0x7f94f40c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:38.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94f99e1640 1 --2- 172.21.15.125:0/342333684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94f406f7a0 0x7f94f40c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:38.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.951+0000 7f94f99e1640 1 -- 172.21.15.125:0/342333684 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f00094d0 con 0x7f94f4071110 2023-10-30T13:05:38.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.955+0000 7f94f99e1640 1 --2- 172.21.15.125:0/342333684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94f4071110 0x7f94f4154980 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f94e400c4b0 tx=0x7f94e400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:38.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.955+0000 7f94eaffd640 1 -- 172.21.15.125:0/342333684 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e40074e0 con 0x7f94f4071110 2023-10-30T13:05:38.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.955+0000 7f94eaffd640 1 -- 172.21.15.125:0/342333684 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e4007660 con 0x7f94f4071110 2023-10-30T13:05:38.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.955+0000 7f94eaffd640 1 -- 172.21.15.125:0/342333684 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e4004a30 con 0x7f94f4071110 2023-10-30T13:05:38.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.955+0000 7f94fc446640 1 -- 172.21.15.125:0/342333684 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f41f2010 con 0x7f94f4071110 2023-10-30T13:05:38.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.955+0000 7f94fc446640 1 -- 172.21.15.125:0/342333684 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f41f25d0 con 0x7f94f4071110 2023-10-30T13:05:38.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.955+0000 7f94fc446640 1 -- 172.21.15.125:0/342333684 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f404e170 con 0x7f94f4071110 2023-10-30T13:05:38.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.955+0000 7f94eaffd640 1 -- 172.21.15.125:0/342333684 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f94e4004b90 con 0x7f94f4071110 2023-10-30T13:05:38.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.955+0000 7f94eaffd640 1 --2- 172.21.15.125:0/342333684 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94d80664d0 0x7f94d8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:38.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.955+0000 7f94eaffd640 1 -- 172.21.15.125:0/342333684 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f94e40836f0 con 0x7f94f4071110 2023-10-30T13:05:38.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.959+0000 7f94fa1e2640 1 --2- 172.21.15.125:0/342333684 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94d80664d0 0x7f94d8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:38.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.959+0000 7f94fa1e2640 1 --2- 172.21.15.125:0/342333684 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94d80664d0 0x7f94d8068990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f94f00b2000 tx=0x7f94f0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:38.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:38.959+0000 7f94eaffd640 1 -- 172.21.15.125:0/342333684 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f94e4050530 con 0x7f94f4071110 2023-10-30T13:05:39.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.139+0000 7f94fc446640 1 -- 172.21.15.125:0/342333684 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94f41f2a80 con 0x7f94f4071110 2023-10-30T13:05:39.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.143+0000 7f94eaffd640 1 -- 172.21.15.125:0/342333684 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f94e4018090 con 0x7f94f4071110 2023-10-30T13:05:39.144 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:39.144 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8178,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:15.605070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:39.144 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 161 2023-10-30T13:05:39.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.143+0000 7f94fc446640 1 -- 172.21.15.125:0/342333684 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94d80664d0 msgr2=0x7f94d8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:39.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.143+0000 7f94fc446640 1 --2- 172.21.15.125:0/342333684 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94d80664d0 0x7f94d8068990 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f94f00b2000 tx=0x7f94f0009040 comp rx=0 tx=0).stop 2023-10-30T13:05:39.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.143+0000 7f94fc446640 1 -- 172.21.15.125:0/342333684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94f4071110 msgr2=0x7f94f4154980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:39.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.143+0000 7f94fc446640 1 --2- 172.21.15.125:0/342333684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94f4071110 0x7f94f4154980 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f94e400c4b0 tx=0x7f94e400c970 comp rx=0 tx=0).stop 2023-10-30T13:05:39.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.143+0000 7f94fc446640 1 -- 172.21.15.125:0/342333684 shutdown_connections 2023-10-30T13:05:39.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.143+0000 7f94fc446640 1 --2- 172.21.15.125:0/342333684 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f94d80664d0 0x7f94d8068990 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:39.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.143+0000 7f94fc446640 1 --2- 172.21.15.125:0/342333684 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f94f40c6bd0 0x7f94f41529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:39.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.143+0000 7f94fc446640 1 --2- 172.21.15.125:0/342333684 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f94f4071110 0x7f94f4154980 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:39.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.143+0000 7f94fc446640 1 --2- 172.21.15.125:0/342333684 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f94f406f7a0 0x7f94f40c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:39.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.143+0000 7f94fc446640 1 -- 172.21.15.125:0/342333684 >> 172.21.15.125:0/342333684 conn(0x7f94f40c7910 msgr2=0x7f94f406a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:39.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.143+0000 7f94fc446640 1 -- 172.21.15.125:0/342333684 shutdown_connections 2023-10-30T13:05:39.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.143+0000 7f94fc446640 1 -- 172.21.15.125:0/342333684 wait complete. 2023-10-30T13:05:39.164 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=0) 2023-10-30T13:05:39.164 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:05:39.164 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:05:39.196 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:05:39.196 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:39.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc7b1d640 1 Processor -- start 2023-10-30T13:05:39.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc7b1d640 1 -- start start 2023-10-30T13:05:39.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc7b1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bc00c6bd0 0x7f0bc006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:39.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc7b1d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0bc0071110 0x7f0bc0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:39.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc7b1d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc0066cf0 con 0x7f0bc006f7a0 2023-10-30T13:05:39.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc7b1d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc00700c0 con 0x7f0bc0071110 2023-10-30T13:05:39.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc7b1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc00c70d0 con 0x7f0bc00c6bd0 2023-10-30T13:05:39.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc50b8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bc00c6bd0 0x7f0bc006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:39.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc60ba640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0bc0071110 0x7f0bc0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:39.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc50b8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bc00c6bd0 0x7f0bc006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:47760/0 (socket says 172.21.15.125:47760) 2023-10-30T13:05:39.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc60ba640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0bc0071110 0x7f0bc0158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:55170/0 (socket says 172.21.15.125:55170) 2023-10-30T13:05:39.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc50b8640 1 -- 172.21.15.125:0/3109666187 learned_addr learned my addr 172.21.15.125:0/3109666187 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:39.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc60ba640 1 -- 172.21.15.125:0/3109666187 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bc00c6bd0 msgr2=0x7f0bc006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:39.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc60ba640 1 --2- 172.21.15.125:0/3109666187 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bc00c6bd0 0x7f0bc006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:39.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc60ba640 1 -- 172.21.15.125:0/3109666187 >> v1:172.21.15.136:6789/0 conn(0x7f0bc006f7a0 legacy=0x7f0bc0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:39.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.375+0000 7f0bc60ba640 1 -- 172.21.15.125:0/3109666187 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc00c7390 con 0x7f0bc0071110 2023-10-30T13:05:39.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.379+0000 7f0bc60ba640 1 --2- 172.21.15.125:0/3109666187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bc0071110 0x7f0bc0158110 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009d90 tx=0x7f0bb801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22c6258bf27de3ca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:39.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.379+0000 7f0bc48b7640 1 -- 172.21.15.125:0/3109666187 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb80af600 con 0x7f0bc0071110 2023-10-30T13:05:39.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.379+0000 7f0bc48b7640 1 -- 172.21.15.125:0/3109666187 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb80ab050 con 0x7f0bc0071110 2023-10-30T13:05:39.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.379+0000 7f0bc48b7640 1 -- 172.21.15.125:0/3109666187 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb80af980 con 0x7f0bc0071110 2023-10-30T13:05:39.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.379+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/3109666187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bc0071110 msgr2=0x7f0bc0158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:39.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.379+0000 7f0bc7b1d640 1 --2- 172.21.15.125:0/3109666187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bc0071110 0x7f0bc0158110 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f0bb8009d90 tx=0x7f0bb801f130 comp rx=0 tx=0).stop 2023-10-30T13:05:39.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.379+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/3109666187 shutdown_connections 2023-10-30T13:05:39.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.379+0000 7f0bc7b1d640 1 --2- 172.21.15.125:0/3109666187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bc0071110 0x7f0bc0158110 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:39.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.379+0000 7f0bc7b1d640 1 --2- 172.21.15.125:0/3109666187 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bc00c6bd0 0x7f0bc006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:39.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.379+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/3109666187 >> 172.21.15.125:0/3109666187 conn(0x7f0bc00c7910 msgr2=0x7f0bc00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:39.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.379+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/3109666187 shutdown_connections 2023-10-30T13:05:39.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.379+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/3109666187 wait complete. 2023-10-30T13:05:39.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc7b1d640 1 Processor -- start 2023-10-30T13:05:39.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc7b1d640 1 -- start start 2023-10-30T13:05:39.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc7b1d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bc006f7a0 0x7f0bc0154a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:39.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc7b1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bc0071110 0x7f0bc00c3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:39.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc7b1d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bc00c6bd0 0x7f0bc0153cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:39.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc7b1d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc014ee90 con 0x7f0bc006f7a0 2023-10-30T13:05:39.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc7b1d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc014f6a0 con 0x7f0bc00c6bd0 2023-10-30T13:05:39.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc7b1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc0151f20 con 0x7f0bc0071110 2023-10-30T13:05:39.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc58b9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bc006f7a0 0x7f0bc0154a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:39.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc58b9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bc006f7a0 0x7f0bc0154a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55342/0 (socket says 172.21.15.125:55342) 2023-10-30T13:05:39.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc58b9640 1 -- 172.21.15.125:0/1842322345 learned_addr learned my addr 172.21.15.125:0/1842322345 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:39.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc50b8640 1 --2- 172.21.15.125:0/1842322345 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bc0071110 0x7f0bc00c3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:39.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc60ba640 1 --2- 172.21.15.125:0/1842322345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bc00c6bd0 0x7f0bc0153cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:39.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc58b9640 1 -- 172.21.15.125:0/1842322345 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bc0071110 msgr2=0x7f0bc00c3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:39.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc58b9640 1 --2- 172.21.15.125:0/1842322345 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bc0071110 0x7f0bc00c3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:39.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc58b9640 1 -- 172.21.15.125:0/1842322345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bc00c6bd0 msgr2=0x7f0bc0153cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:39.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc58b9640 1 --2- 172.21.15.125:0/1842322345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bc00c6bd0 0x7f0bc0153cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:39.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc58b9640 1 -- 172.21.15.125:0/1842322345 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb8009a70 con 0x7f0bc006f7a0 2023-10-30T13:05:39.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc58b9640 1 --2- 172.21.15.125:0/1842322345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bc006f7a0 0x7f0bc0154a30 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00c3c0 tx=0x7f0bbc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:39.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bb67fc640 1 -- 172.21.15.125:0/1842322345 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc0052d0 con 0x7f0bc006f7a0 2023-10-30T13:05:39.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bb67fc640 1 -- 172.21.15.125:0/1842322345 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc005430 con 0x7f0bc006f7a0 2023-10-30T13:05:39.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bb67fc640 1 -- 172.21.15.125:0/1842322345 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc003840 con 0x7f0bc006f7a0 2023-10-30T13:05:39.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/1842322345 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc0152200 con 0x7f0bc006f7a0 2023-10-30T13:05:39.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.383+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/1842322345 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc014f3c0 con 0x7f0bc006f7a0 2023-10-30T13:05:39.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.387+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/1842322345 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc004e170 con 0x7f0bc006f7a0 2023-10-30T13:05:39.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.387+0000 7f0bb67fc640 1 -- 172.21.15.125:0/1842322345 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0bbc0039a0 con 0x7f0bc006f7a0 2023-10-30T13:05:39.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.387+0000 7f0bb67fc640 1 --2- 172.21.15.125:0/1842322345 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ba40664a0 0x7f0ba4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:39.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.387+0000 7f0bb67fc640 1 -- 172.21.15.125:0/1842322345 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f0bbc083b00 con 0x7f0bc006f7a0 2023-10-30T13:05:39.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.387+0000 7f0bc50b8640 1 --2- 172.21.15.125:0/1842322345 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ba40664a0 0x7f0ba4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:39.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.391+0000 7f0bc50b8640 1 --2- 172.21.15.125:0/1842322345 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ba40664a0 0x7f0ba4068960 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0bc01e7010 tx=0x7f0bb0009240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:39.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.391+0000 7f0bb67fc640 1 -- 172.21.15.125:0/1842322345 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0bbc085e60 con 0x7f0bc006f7a0 2023-10-30T13:05:39.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.575+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/1842322345 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0bc00c6160 con 0x7f0bc006f7a0 2023-10-30T13:05:39.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.575+0000 7f0bb67fc640 1 -- 172.21.15.125:0/1842322345 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f0bbc017090 con 0x7f0bc006f7a0 2023-10-30T13:05:39.578 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:39.578 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8178,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:15.605070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:39.579 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 161 2023-10-30T13:05:39.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.579+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/1842322345 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ba40664a0 msgr2=0x7f0ba4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:39.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.579+0000 7f0bc7b1d640 1 --2- 172.21.15.125:0/1842322345 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ba40664a0 0x7f0ba4068960 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0bc01e7010 tx=0x7f0bb0009240 comp rx=0 tx=0).stop 2023-10-30T13:05:39.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.579+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/1842322345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bc006f7a0 msgr2=0x7f0bc0154a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:39.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.579+0000 7f0bc7b1d640 1 --2- 172.21.15.125:0/1842322345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bc006f7a0 0x7f0bc0154a30 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc00c3c0 tx=0x7f0bbc00c880 comp rx=0 tx=0).stop 2023-10-30T13:05:39.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.579+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/1842322345 shutdown_connections 2023-10-30T13:05:39.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.579+0000 7f0bc7b1d640 1 --2- 172.21.15.125:0/1842322345 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ba40664a0 0x7f0ba4068960 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:39.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.579+0000 7f0bc7b1d640 1 --2- 172.21.15.125:0/1842322345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0bc00c6bd0 0x7f0bc0153cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:39.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.579+0000 7f0bc7b1d640 1 --2- 172.21.15.125:0/1842322345 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0bc0071110 0x7f0bc00c3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:39.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.579+0000 7f0bc7b1d640 1 --2- 172.21.15.125:0/1842322345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0bc006f7a0 0x7f0bc0154a30 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.579+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/1842322345 >> 172.21.15.125:0/1842322345 conn(0x7f0bc00c7910 msgr2=0x7f0bc00c9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.579+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/1842322345 shutdown_connections 2023-10-30T13:05:39.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:39.579+0000 7f0bc7b1d640 1 -- 172.21.15.125:0/1842322345 wait complete. 2023-10-30T13:05:39.599 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:05:41.600 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:41.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.743+0000 7f2d02580640 1 Processor -- start 2023-10-30T13:05:41.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.743+0000 7f2d02580640 1 -- start start 2023-10-30T13:05:41.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.743+0000 7f2d02580640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2cfc06f7a0 0x7f2cfc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:41.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.743+0000 7f2d02580640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2cfc0c6bd0 0x7f2cfc069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:41.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.743+0000 7f2d02580640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cfc071110 0x7f2cfc158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:41.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.743+0000 7f2d02580640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc0700c0 con 0x7f2cfc06f7a0 2023-10-30T13:05:41.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.743+0000 7f2d02580640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cfc06a3d0 con 0x7f2cfc0c6bd0 2023-10-30T13:05:41.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.743+0000 7f2d02580640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc066cf0 con 0x7f2cfc071110 2023-10-30T13:05:41.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.743+0000 7f2cfbfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2cfc06f7a0 0x7f2cfc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:41.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.747+0000 7f2cfbfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2cfc06f7a0 0x7f2cfc06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55812/0 (socket says 172.21.15.125:55812) 2023-10-30T13:05:41.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.747+0000 7f2cfbfff640 1 -- 172.21.15.125:0/142046844 learned_addr learned my addr 172.21.15.125:0/142046844 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:41.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.747+0000 7f2d00b1d640 1 --2- 172.21.15.125:0/142046844 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cfc071110 0x7f2cfc158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:41.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.747+0000 7f2cfb7fe640 1 --2- 172.21.15.125:0/142046844 >> v2:172.21.15.136:3300/0 conn(0x7f2cfc0c6bd0 0x7f2cfc069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:41.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.747+0000 7f2cfbfff640 1 -- 172.21.15.125:0/142046844 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cfc071110 msgr2=0x7f2cfc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:41.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.747+0000 7f2cfbfff640 1 --2- 172.21.15.125:0/142046844 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cfc071110 0x7f2cfc158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:41.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.747+0000 7f2cfbfff640 1 -- 172.21.15.125:0/142046844 >> v2:172.21.15.136:3300/0 conn(0x7f2cfc0c6bd0 msgr2=0x7f2cfc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:41.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.747+0000 7f2cfbfff640 1 --2- 172.21.15.125:0/142046844 >> v2:172.21.15.136:3300/0 conn(0x7f2cfc0c6bd0 0x7f2cfc069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:41.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.747+0000 7f2cfbfff640 1 -- 172.21.15.125:0/142046844 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cfc0c70d0 con 0x7f2cfc06f7a0 2023-10-30T13:05:41.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.747+0000 7f2cfbfff640 1 --2- 172.21.15.125:0/142046844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cfc06f7a0 0x7f2cfc06fb80 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc06f1c0 tx=0x7f2cec01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f7f1f4927c6242 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:41.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.747+0000 7f2cfaffd640 1 -- 172.21.15.125:0/142046844 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec0a89b0 con 0x7f2cfc06f7a0 2023-10-30T13:05:41.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.747+0000 7f2cfaffd640 1 -- 172.21.15.125:0/142046844 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cec0a8b10 con 0x7f2cfc06f7a0 2023-10-30T13:05:41.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.747+0000 7f2cfaffd640 1 -- 172.21.15.125:0/142046844 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cec0a8ea0 con 0x7f2cfc06f7a0 2023-10-30T13:05:41.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 -- 172.21.15.125:0/142046844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cfc06f7a0 msgr2=0x7f2cfc06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:41.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 --2- 172.21.15.125:0/142046844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cfc06f7a0 0x7f2cfc06fb80 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc06f1c0 tx=0x7f2cec01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:05:41.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 -- 172.21.15.125:0/142046844 shutdown_connections 2023-10-30T13:05:41.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 --2- 172.21.15.125:0/142046844 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cfc071110 0x7f2cfc158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:41.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 --2- 172.21.15.125:0/142046844 >> v2:172.21.15.136:3300/0 conn(0x7f2cfc0c6bd0 0x7f2cfc069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:41.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 --2- 172.21.15.125:0/142046844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cfc06f7a0 0x7f2cfc06fb80 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:41.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 -- 172.21.15.125:0/142046844 >> 172.21.15.125:0/142046844 conn(0x7f2cfc0c7910 msgr2=0x7f2cfc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:41.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 -- 172.21.15.125:0/142046844 shutdown_connections 2023-10-30T13:05:41.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 -- 172.21.15.125:0/142046844 wait complete. 2023-10-30T13:05:41.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 Processor -- start 2023-10-30T13:05:41.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 -- start start 2023-10-30T13:05:41.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cfc06f7a0 0x7f2cfc147d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:41.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cfc071110 0x7f2cfc1eee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:41.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cfc0c6bd0 0x7f2cfc150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:41.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc0c3f80 con 0x7f2cfc071110 2023-10-30T13:05:41.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc14c120 con 0x7f2cfc0c6bd0 2023-10-30T13:05:41.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d02580640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc0c9930 con 0x7f2cfc06f7a0 2023-10-30T13:05:41.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2cfb7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cfc071110 0x7f2cfc1eee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:41.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2cfb7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cfc071110 0x7f2cfc1eee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55824/0 (socket says 172.21.15.125:55824) 2023-10-30T13:05:41.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2cfb7fe640 1 -- 172.21.15.125:0/3375419740 learned_addr learned my addr 172.21.15.125:0/3375419740 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:41.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2cfbfff640 1 --2- 172.21.15.125:0/3375419740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cfc06f7a0 0x7f2cfc147d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:41.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2d00b1d640 1 --2- 172.21.15.125:0/3375419740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cfc0c6bd0 0x7f2cfc150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:41.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2cfb7fe640 1 -- 172.21.15.125:0/3375419740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cfc06f7a0 msgr2=0x7f2cfc147d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:41.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2cfb7fe640 1 --2- 172.21.15.125:0/3375419740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cfc06f7a0 0x7f2cfc147d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:41.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2cfb7fe640 1 -- 172.21.15.125:0/3375419740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cfc0c6bd0 msgr2=0x7f2cfc150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:41.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2cfb7fe640 1 --2- 172.21.15.125:0/3375419740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cfc0c6bd0 0x7f2cfc150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:41.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.751+0000 7f2cfb7fe640 1 -- 172.21.15.125:0/3375419740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec0094d0 con 0x7f2cfc071110 2023-10-30T13:05:41.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.755+0000 7f2cfb7fe640 1 --2- 172.21.15.125:0/3375419740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cfc071110 0x7f2cfc1eee60 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f2cf000d5d0 tx=0x7f2cf000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:41.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.755+0000 7f2cf8ff9640 1 -- 172.21.15.125:0/3375419740 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0004240 con 0x7f2cfc071110 2023-10-30T13:05:41.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.755+0000 7f2cf8ff9640 1 -- 172.21.15.125:0/3375419740 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf00043a0 con 0x7f2cfc071110 2023-10-30T13:05:41.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.755+0000 7f2cf8ff9640 1 -- 172.21.15.125:0/3375419740 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf0003840 con 0x7f2cfc071110 2023-10-30T13:05:41.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.755+0000 7f2d02580640 1 -- 172.21.15.125:0/3375419740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc0c4160 con 0x7f2cfc071110 2023-10-30T13:05:41.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.755+0000 7f2d02580640 1 -- 172.21.15.125:0/3375419740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc0c92e0 con 0x7f2cfc071110 2023-10-30T13:05:41.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.755+0000 7f2d02580640 1 -- 172.21.15.125:0/3375419740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cfc0c9ae0 con 0x7f2cfc071110 2023-10-30T13:05:41.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.755+0000 7f2cf8ff9640 1 -- 172.21.15.125:0/3375419740 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2cf00039a0 con 0x7f2cfc071110 2023-10-30T13:05:41.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.755+0000 7f2cf8ff9640 1 --2- 172.21.15.125:0/3375419740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ce80664a0 0x7f2ce8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:41.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.755+0000 7f2cf8ff9640 1 -- 172.21.15.125:0/3375419740 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f2cf0084580 con 0x7f2cfc071110 2023-10-30T13:05:41.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.755+0000 7f2cfbfff640 1 --2- 172.21.15.125:0/3375419740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ce80664a0 0x7f2ce8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:41.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.759+0000 7f2cfbfff640 1 --2- 172.21.15.125:0/3375419740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ce80664a0 0x7f2ce8068960 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0b2000 tx=0x7f2cec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:41.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.759+0000 7f2cf8ff9640 1 -- 172.21.15.125:0/3375419740 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2cfc0c9ae0 con 0x7f2cfc071110 2023-10-30T13:05:41.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.943+0000 7f2d02580640 1 -- 172.21.15.125:0/3375419740 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cfc0c9ae0 con 0x7f2cfc071110 2023-10-30T13:05:41.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.947+0000 7f2cf8ff9640 1 -- 172.21.15.125:0/3375419740 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f2cfc0c9ae0 con 0x7f2cfc071110 2023-10-30T13:05:41.948 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:41.948 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8178,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:15.605070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:41.949 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 161 2023-10-30T13:05:41.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.947+0000 7f2d02580640 1 -- 172.21.15.125:0/3375419740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ce80664a0 msgr2=0x7f2ce8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:41.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.947+0000 7f2d02580640 1 --2- 172.21.15.125:0/3375419740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ce80664a0 0x7f2ce8068960 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f2cec0b2000 tx=0x7f2cec009040 comp rx=0 tx=0).stop 2023-10-30T13:05:41.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.947+0000 7f2d02580640 1 -- 172.21.15.125:0/3375419740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cfc071110 msgr2=0x7f2cfc1eee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:41.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.947+0000 7f2d02580640 1 --2- 172.21.15.125:0/3375419740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cfc071110 0x7f2cfc1eee60 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f2cf000d5d0 tx=0x7f2cf000da90 comp rx=0 tx=0).stop 2023-10-30T13:05:41.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.947+0000 7f2d02580640 1 -- 172.21.15.125:0/3375419740 shutdown_connections 2023-10-30T13:05:41.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.947+0000 7f2d02580640 1 --2- 172.21.15.125:0/3375419740 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2ce80664a0 0x7f2ce8068960 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:41.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.947+0000 7f2d02580640 1 --2- 172.21.15.125:0/3375419740 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2cfc0c6bd0 0x7f2cfc150760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:41.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.947+0000 7f2d02580640 1 --2- 172.21.15.125:0/3375419740 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2cfc071110 0x7f2cfc1eee60 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:41.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.947+0000 7f2d02580640 1 --2- 172.21.15.125:0/3375419740 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2cfc06f7a0 0x7f2cfc147d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:41.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.947+0000 7f2d02580640 1 -- 172.21.15.125:0/3375419740 >> 172.21.15.125:0/3375419740 conn(0x7f2cfc0c7910 msgr2=0x7f2cfc06be40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:41.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.947+0000 7f2d02580640 1 -- 172.21.15.125:0/3375419740 shutdown_connections 2023-10-30T13:05:41.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:41.947+0000 7f2d02580640 1 -- 172.21.15.125:0/3375419740 wait complete. 2023-10-30T13:05:41.969 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:05:43.969 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:44.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.115+0000 7feb132d6640 1 Processor -- start 2023-10-30T13:05:44.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.115+0000 7feb132d6640 1 -- start start 2023-10-30T13:05:44.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.115+0000 7feb132d6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7feb0c069e90 0x7feb0c15a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:44.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.115+0000 7feb132d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0c0c6c10 con 0x7feb0c06cb80 2023-10-30T13:05:44.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.115+0000 7feb132d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0c0c6ef0 con 0x7feb0c06a270 2023-10-30T13:05:44.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.115+0000 7feb132d6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0c0c8570 con 0x7feb0c069e90 2023-10-30T13:05:44.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.115+0000 7feb11072640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7feb0c069e90 0x7feb0c15a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:44.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.115+0000 7feb11072640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7feb0c069e90 0x7feb0c15a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55834/0 (socket says 172.21.15.125:55834) 2023-10-30T13:05:44.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.115+0000 7feb11072640 1 -- 172.21.15.125:0/3222318540 learned_addr learned my addr 172.21.15.125:0/3222318540 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:44.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.115+0000 7feb11873640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7feb0c06cb80 0x7feb0c06cff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33962/0 (socket says 172.21.15.125:33962) 2023-10-30T13:05:44.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.115+0000 7feb11072640 1 -- 172.21.15.125:0/3222318540 >> v1:172.21.15.136:6789/0 conn(0x7feb0c06a270 legacy=0x7feb0c0c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:44.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.115+0000 7feb11072640 1 -- 172.21.15.125:0/3222318540 >> v1:172.21.15.125:6789/0 conn(0x7feb0c06cb80 legacy=0x7feb0c06cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:44.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.115+0000 7feb11072640 1 -- 172.21.15.125:0/3222318540 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c0c71d0 con 0x7feb0c069e90 2023-10-30T13:05:44.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.115+0000 7feb03fff640 1 -- 172.21.15.125:0/3222318540 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432163523 0 0) 0x7feb0c0c6c10 con 0x7feb0c06cb80 2023-10-30T13:05:44.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.119+0000 7feb11072640 1 --2- 172.21.15.125:0/3222318540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0c069e90 0x7feb0c15a410 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7feb08009870 tx=0x7feb0801c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c064c6292ea8515 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:44.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.119+0000 7feb03fff640 1 -- 172.21.15.125:0/3222318540 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb080a7070 con 0x7feb0c069e90 2023-10-30T13:05:44.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.119+0000 7feb03fff640 1 -- 172.21.15.125:0/3222318540 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb080ab430 con 0x7feb0c069e90 2023-10-30T13:05:44.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.119+0000 7feb03fff640 1 -- 172.21.15.125:0/3222318540 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb080ab7c0 con 0x7feb0c069e90 2023-10-30T13:05:44.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.119+0000 7feb132d6640 1 -- 172.21.15.125:0/3222318540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0c069e90 msgr2=0x7feb0c15a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:44.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.119+0000 7feb132d6640 1 --2- 172.21.15.125:0/3222318540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0c069e90 0x7feb0c15a410 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7feb08009870 tx=0x7feb0801c540 comp rx=0 tx=0).stop 2023-10-30T13:05:44.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.119+0000 7feb132d6640 1 -- 172.21.15.125:0/3222318540 shutdown_connections 2023-10-30T13:05:44.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.119+0000 7feb132d6640 1 --2- 172.21.15.125:0/3222318540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0c069e90 0x7feb0c15a410 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:44.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.119+0000 7feb132d6640 1 -- 172.21.15.125:0/3222318540 >> 172.21.15.125:0/3222318540 conn(0x7feb0c06dfd0 msgr2=0x7feb0c06e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:44.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.119+0000 7feb132d6640 1 -- 172.21.15.125:0/3222318540 shutdown_connections 2023-10-30T13:05:44.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.119+0000 7feb132d6640 1 -- 172.21.15.125:0/3222318540 wait complete. 2023-10-30T13:05:44.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb132d6640 1 Processor -- start 2023-10-30T13:05:44.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb132d6640 1 -- start start 2023-10-30T13:05:44.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb132d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0c069e90 0x7feb0c145ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:44.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb132d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feb0c06a270 0x7feb0c146b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:44.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb132d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb0c06cb80 0x7feb0c1470a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:44.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb132d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c0b1e60 con 0x7feb0c069e90 2023-10-30T13:05:44.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb132d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c14e150 con 0x7feb0c06cb80 2023-10-30T13:05:44.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb132d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0c1466d0 con 0x7feb0c06a270 2023-10-30T13:05:44.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb11072640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0c069e90 0x7feb0c145ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:44.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb11072640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0c069e90 0x7feb0c145ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55848/0 (socket says 172.21.15.125:55848) 2023-10-30T13:05:44.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb11072640 1 -- 172.21.15.125:0/1626450192 learned_addr learned my addr 172.21.15.125:0/1626450192 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:44.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb10871640 1 --2- 172.21.15.125:0/1626450192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feb0c06a270 0x7feb0c146b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:44.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb11873640 1 --2- 172.21.15.125:0/1626450192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb0c06cb80 0x7feb0c1470a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:44.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb11072640 1 -- 172.21.15.125:0/1626450192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feb0c06a270 msgr2=0x7feb0c146b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:44.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb11072640 1 --2- 172.21.15.125:0/1626450192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feb0c06a270 0x7feb0c146b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:44.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb11072640 1 -- 172.21.15.125:0/1626450192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb0c06cb80 msgr2=0x7feb0c1470a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:44.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb11072640 1 --2- 172.21.15.125:0/1626450192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb0c06cb80 0x7feb0c1470a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:44.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb11072640 1 -- 172.21.15.125:0/1626450192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb080094d0 con 0x7feb0c069e90 2023-10-30T13:05:44.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb11072640 1 --2- 172.21.15.125:0/1626450192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0c069e90 0x7feb0c145ae0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7feb080ad040 tx=0x7feb0801ad80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:44.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb01ffb640 1 -- 172.21.15.125:0/1626450192 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb080b4800 con 0x7feb0c069e90 2023-10-30T13:05:44.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb132d6640 1 -- 172.21.15.125:0/1626450192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0c154f80 con 0x7feb0c069e90 2023-10-30T13:05:44.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb01ffb640 1 -- 172.21.15.125:0/1626450192 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb080b4980 con 0x7feb0c069e90 2023-10-30T13:05:44.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb01ffb640 1 -- 172.21.15.125:0/1626450192 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb080b4d30 con 0x7feb0c069e90 2023-10-30T13:05:44.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.123+0000 7feb132d6640 1 -- 172.21.15.125:0/1626450192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0c14d5a0 con 0x7feb0c069e90 2023-10-30T13:05:44.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.127+0000 7feb132d6640 1 -- 172.21.15.125:0/1626450192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0c04e170 con 0x7feb0c069e90 2023-10-30T13:05:44.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.127+0000 7feb01ffb640 1 -- 172.21.15.125:0/1626450192 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feb080b3380 con 0x7feb0c069e90 2023-10-30T13:05:44.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.127+0000 7feb01ffb640 1 --2- 172.21.15.125:0/1626450192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feaf00664a0 0x7feaf0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:44.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.127+0000 7feb01ffb640 1 -- 172.21.15.125:0/1626450192 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7feb0811e750 con 0x7feb0c069e90 2023-10-30T13:05:44.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.127+0000 7feb10871640 1 --2- 172.21.15.125:0/1626450192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feaf00664a0 0x7feaf0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:44.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.127+0000 7feb10871640 1 --2- 172.21.15.125:0/1626450192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feaf00664a0 0x7feaf0068960 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7feb0c1463a0 tx=0x7feafc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:44.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.131+0000 7feb01ffb640 1 -- 172.21.15.125:0/1626450192 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feb080eb3a0 con 0x7feb0c069e90 2023-10-30T13:05:44.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.311+0000 7feb132d6640 1 -- 172.21.15.125:0/1626450192 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb0c06ef60 con 0x7feb0c069e90 2023-10-30T13:05:44.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.311+0000 7feb01ffb640 1 -- 172.21.15.125:0/1626450192 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6519 (secure 0 0 0) 0x7feb080ee9c0 con 0x7feb0c069e90 2023-10-30T13:05:44.315 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:44.315 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8178,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:15.605070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:44.316 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 161 2023-10-30T13:05:44.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.315+0000 7feb132d6640 1 -- 172.21.15.125:0/1626450192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feaf00664a0 msgr2=0x7feaf0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:44.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.315+0000 7feb132d6640 1 --2- 172.21.15.125:0/1626450192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feaf00664a0 0x7feaf0068960 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7feb0c1463a0 tx=0x7feafc009040 comp rx=0 tx=0).stop 2023-10-30T13:05:44.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.315+0000 7feb132d6640 1 -- 172.21.15.125:0/1626450192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0c069e90 msgr2=0x7feb0c145ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:44.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.315+0000 7feb132d6640 1 --2- 172.21.15.125:0/1626450192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0c069e90 0x7feb0c145ae0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7feb080ad040 tx=0x7feb0801ad80 comp rx=0 tx=0).stop 2023-10-30T13:05:44.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.315+0000 7feb132d6640 1 -- 172.21.15.125:0/1626450192 shutdown_connections 2023-10-30T13:05:44.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.315+0000 7feb132d6640 1 --2- 172.21.15.125:0/1626450192 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feaf00664a0 0x7feaf0068960 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:44.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.315+0000 7feb132d6640 1 --2- 172.21.15.125:0/1626450192 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb0c06cb80 0x7feb0c1470a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:44.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.315+0000 7feb132d6640 1 --2- 172.21.15.125:0/1626450192 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feb0c06a270 0x7feb0c146b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:44.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.315+0000 7feb132d6640 1 --2- 172.21.15.125:0/1626450192 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0c069e90 0x7feb0c145ae0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:44.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.315+0000 7feb132d6640 1 -- 172.21.15.125:0/1626450192 >> 172.21.15.125:0/1626450192 conn(0x7feb0c06dfd0 msgr2=0x7feb0c0c9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:44.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.315+0000 7feb132d6640 1 -- 172.21.15.125:0/1626450192 shutdown_connections 2023-10-30T13:05:44.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:44.315+0000 7feb132d6640 1 -- 172.21.15.125:0/1626450192 wait complete. 2023-10-30T13:05:44.336 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:05:46.337 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:46.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1be8de4640 1 Processor -- start 2023-10-30T13:05:46.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1be8de4640 1 -- start start 2023-10-30T13:05:46.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1be8de4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1be41538f0 0x7f1be4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:46.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1be8de4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1be415cd50 0x7f1be40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:46.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1be8de4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1be415d300 con 0x7f1be40cefa0 2023-10-30T13:05:46.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1be8de4640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be415d5e0 con 0x7f1be41538f0 2023-10-30T13:05:46.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1be8de4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1be415d8c0 con 0x7f1be415cd50 2023-10-30T13:05:46.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1be37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1be41538f0 0x7f1be4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:46.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1be37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1be41538f0 0x7f1be4159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55854/0 (socket says 172.21.15.125:55854) 2023-10-30T13:05:46.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1be37fe640 1 -- 172.21.15.125:0/2135100752 learned_addr learned my addr 172.21.15.125:0/2135100752 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:46.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1bdbfff640 1 --2- 172.21.15.125:0/2135100752 >> v2:172.21.15.136:3300/0 conn(0x7f1be415cd50 0x7f1be40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:46.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1be37fe640 1 -- 172.21.15.125:0/2135100752 >> v2:172.21.15.136:3300/0 conn(0x7f1be415cd50 msgr2=0x7f1be40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:46.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1be37fe640 1 --2- 172.21.15.125:0/2135100752 >> v2:172.21.15.136:3300/0 conn(0x7f1be415cd50 0x7f1be40c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:46.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1be37fe640 1 -- 172.21.15.125:0/2135100752 >> v1:172.21.15.136:6789/0 conn(0x7f1be40cefa0 legacy=0x7f1be40cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:46.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.483+0000 7f1be37fe640 1 -- 172.21.15.125:0/2135100752 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be415dba0 con 0x7f1be41538f0 2023-10-30T13:05:46.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.487+0000 7f1be37fe640 1 --2- 172.21.15.125:0/2135100752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1be41538f0 0x7f1be4159fb0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f1be4072e90 tx=0x7f1bd401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23698fa3c104d762 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:46.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.487+0000 7f1be2ffd640 1 -- 172.21.15.125:0/2135100752 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd40a89b0 con 0x7f1be41538f0 2023-10-30T13:05:46.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.487+0000 7f1be2ffd640 1 -- 172.21.15.125:0/2135100752 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd40a8b10 con 0x7f1be41538f0 2023-10-30T13:05:46.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.487+0000 7f1be2ffd640 1 -- 172.21.15.125:0/2135100752 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd40a8ea0 con 0x7f1be41538f0 2023-10-30T13:05:46.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.487+0000 7f1be8de4640 1 -- 172.21.15.125:0/2135100752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1be41538f0 msgr2=0x7f1be4159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:46.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.487+0000 7f1be8de4640 1 --2- 172.21.15.125:0/2135100752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1be41538f0 0x7f1be4159fb0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f1be4072e90 tx=0x7f1bd401ecb0 comp rx=0 tx=0).stop 2023-10-30T13:05:46.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.487+0000 7f1be8de4640 1 -- 172.21.15.125:0/2135100752 shutdown_connections 2023-10-30T13:05:46.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.487+0000 7f1be8de4640 1 --2- 172.21.15.125:0/2135100752 >> v2:172.21.15.136:3300/0 conn(0x7f1be415cd50 0x7f1be40c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:46.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.487+0000 7f1be8de4640 1 --2- 172.21.15.125:0/2135100752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1be41538f0 0x7f1be4159fb0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:46.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.487+0000 7f1be8de4640 1 -- 172.21.15.125:0/2135100752 >> 172.21.15.125:0/2135100752 conn(0x7f1be406a640 msgr2=0x7f1be4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:46.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.487+0000 7f1be8de4640 1 -- 172.21.15.125:0/2135100752 shutdown_connections 2023-10-30T13:05:46.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.487+0000 7f1be8de4640 1 -- 172.21.15.125:0/2135100752 wait complete. 2023-10-30T13:05:46.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be8de4640 1 Processor -- start 2023-10-30T13:05:46.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be8de4640 1 -- start start 2023-10-30T13:05:46.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be8de4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1be40cefa0 0x7f1be41efd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:46.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be8de4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1be41538f0 0x7f1be41523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:46.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be8de4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be415cd50 0x7f1be41573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:46.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be8de4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be4146010 con 0x7f1be40cefa0 2023-10-30T13:05:46.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be8de4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1be41f02a0 con 0x7f1be415cd50 2023-10-30T13:05:46.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be8de4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be41594c0 con 0x7f1be41538f0 2023-10-30T13:05:46.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1be40cefa0 0x7f1be41efd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:46.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1be40cefa0 0x7f1be41efd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55868/0 (socket says 172.21.15.125:55868) 2023-10-30T13:05:46.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be37fe640 1 -- 172.21.15.125:0/691831598 learned_addr learned my addr 172.21.15.125:0/691831598 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:46.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1bdbfff640 1 --2- 172.21.15.125:0/691831598 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1be41538f0 0x7f1be41523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:46.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be3fff640 1 --2- 172.21.15.125:0/691831598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be415cd50 0x7f1be41573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:46.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be37fe640 1 -- 172.21.15.125:0/691831598 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1be41538f0 msgr2=0x7f1be41523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:46.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be37fe640 1 --2- 172.21.15.125:0/691831598 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1be41538f0 0x7f1be41523d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:46.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be37fe640 1 -- 172.21.15.125:0/691831598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be415cd50 msgr2=0x7f1be41573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:46.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be37fe640 1 --2- 172.21.15.125:0/691831598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be415cd50 0x7f1be41573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:46.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be37fe640 1 -- 172.21.15.125:0/691831598 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bd40094d0 con 0x7f1be40cefa0 2023-10-30T13:05:46.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be37fe640 1 --2- 172.21.15.125:0/691831598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1be40cefa0 0x7f1be41efd60 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40b3040 tx=0x7f1bd401ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:46.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be0ff9640 1 -- 172.21.15.125:0/691831598 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd40b0e80 con 0x7f1be40cefa0 2023-10-30T13:05:46.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be0ff9640 1 -- 172.21.15.125:0/691831598 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bd40bb480 con 0x7f1be40cefa0 2023-10-30T13:05:46.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be0ff9640 1 -- 172.21.15.125:0/691831598 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bd40bb810 con 0x7f1be40cefa0 2023-10-30T13:05:46.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be8de4640 1 -- 172.21.15.125:0/691831598 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1be415a770 con 0x7f1be40cefa0 2023-10-30T13:05:46.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.491+0000 7f1be8de4640 1 -- 172.21.15.125:0/691831598 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1be415ad30 con 0x7f1be40cefa0 2023-10-30T13:05:46.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.495+0000 7f1be8de4640 1 -- 172.21.15.125:0/691831598 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1be404e170 con 0x7f1be40cefa0 2023-10-30T13:05:46.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.495+0000 7f1be0ff9640 1 -- 172.21.15.125:0/691831598 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1bd40bb970 con 0x7f1be40cefa0 2023-10-30T13:05:46.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.495+0000 7f1be0ff9640 1 --2- 172.21.15.125:0/691831598 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bc40664d0 0x7f1bc4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:46.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.495+0000 7f1be0ff9640 1 -- 172.21.15.125:0/691831598 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f1bd4122c50 con 0x7f1be40cefa0 2023-10-30T13:05:46.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.495+0000 7f1bdbfff640 1 --2- 172.21.15.125:0/691831598 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bc40664d0 0x7f1bc4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:46.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.495+0000 7f1bdbfff640 1 --2- 172.21.15.125:0/691831598 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bc40664d0 0x7f1bc4068990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f1be4148f30 tx=0x7f1bcc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:46.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.499+0000 7f1be0ff9640 1 -- 172.21.15.125:0/691831598 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1bd40ef920 con 0x7f1be40cefa0 2023-10-30T13:05:46.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.679+0000 7f1be8de4640 1 -- 172.21.15.125:0/691831598 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1be404e170 con 0x7f1be40cefa0 2023-10-30T13:05:46.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.679+0000 7f1be0ff9640 1 -- 172.21.15.125:0/691831598 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f1bd40b5020 con 0x7f1be40cefa0 2023-10-30T13:05:46.683 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:46.683 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8178,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:15.605070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:46.683 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 161 2023-10-30T13:05:46.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.683+0000 7f1be8de4640 1 -- 172.21.15.125:0/691831598 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bc40664d0 msgr2=0x7f1bc4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:46.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.683+0000 7f1be8de4640 1 --2- 172.21.15.125:0/691831598 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bc40664d0 0x7f1bc4068990 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f1be4148f30 tx=0x7f1bcc008040 comp rx=0 tx=0).stop 2023-10-30T13:05:46.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.683+0000 7f1be8de4640 1 -- 172.21.15.125:0/691831598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1be40cefa0 msgr2=0x7f1be41efd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:46.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.683+0000 7f1be8de4640 1 --2- 172.21.15.125:0/691831598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1be40cefa0 0x7f1be41efd60 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f1bd40b3040 tx=0x7f1bd401ba50 comp rx=0 tx=0).stop 2023-10-30T13:05:46.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.683+0000 7f1be8de4640 1 -- 172.21.15.125:0/691831598 shutdown_connections 2023-10-30T13:05:46.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.683+0000 7f1be8de4640 1 --2- 172.21.15.125:0/691831598 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bc40664d0 0x7f1bc4068990 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:46.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.683+0000 7f1be8de4640 1 --2- 172.21.15.125:0/691831598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1be415cd50 0x7f1be41573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:46.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.683+0000 7f1be8de4640 1 --2- 172.21.15.125:0/691831598 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1be41538f0 0x7f1be41523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:46.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.683+0000 7f1be8de4640 1 --2- 172.21.15.125:0/691831598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1be40cefa0 0x7f1be41efd60 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:46.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.683+0000 7f1be8de4640 1 -- 172.21.15.125:0/691831598 >> 172.21.15.125:0/691831598 conn(0x7f1be406a640 msgr2=0x7f1be4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:46.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.683+0000 7f1be8de4640 1 -- 172.21.15.125:0/691831598 shutdown_connections 2023-10-30T13:05:46.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:46.683+0000 7f1be8de4640 1 -- 172.21.15.125:0/691831598 wait complete. 2023-10-30T13:05:46.703 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:05:48.705 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:48.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc74f4ea640 1 Processor -- start 2023-10-30T13:05:48.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc74f4ea640 1 -- start start 2023-10-30T13:05:48.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc74f4ea640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc748070740 0x7fc7480c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:48.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc74f4ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc74815e700 0x7fc74806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:48.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc74f4ea640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7480c5810 con 0x7fc7481580e0 2023-10-30T13:05:48.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc74f4ea640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc74815eb90 con 0x7fc748070740 2023-10-30T13:05:48.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc74f4ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7480c6bb0 con 0x7fc74815e700 2023-10-30T13:05:48.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc74d286640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc7481580e0 0x7fc748069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33978/0 (socket says 172.21.15.125:33978) 2023-10-30T13:05:48.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc74d286640 1 -- 172.21.15.125:0/4089404640 learned_addr learned my addr 172.21.15.125:0/4089404640 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:05:48.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc74da87640 1 --2- 172.21.15.125:0/4089404640 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc74815e700 0x7fc74806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:48.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc74ca85640 1 --2- 172.21.15.125:0/4089404640 >> v2:172.21.15.136:3300/0 conn(0x7fc748070740 0x7fc7480c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:48.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 -- 172.21.15.125:0/4089404640 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 553106178 0 0) 0x7fc7480c5810 con 0x7fc7481580e0 2023-10-30T13:05:48.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 -- 172.21.15.125:0/4089404640 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc734003530 con 0x7fc7481580e0 2023-10-30T13:05:48.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 -- 172.21.15.125:0/4089404640 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1337581912 0 0) 0x7fc734003530 con 0x7fc7481580e0 2023-10-30T13:05:48.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 -- 172.21.15.125:0/4089404640 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc74815e700 msgr2=0x7fc74806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:48.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 --2- 172.21.15.125:0/4089404640 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc74815e700 0x7fc74806a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:48.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 -- 172.21.15.125:0/4089404640 >> v2:172.21.15.136:3300/0 conn(0x7fc748070740 msgr2=0x7fc7480c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:48.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 --2- 172.21.15.125:0/4089404640 >> v2:172.21.15.136:3300/0 conn(0x7fc748070740 0x7fc7480c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:48.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 -- 172.21.15.125:0/4089404640 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7480c6e70 con 0x7fc7481580e0 2023-10-30T13:05:48.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 -- 172.21.15.125:0/4089404640 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fc744001680 con 0x7fc7481580e0 2023-10-30T13:05:48.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 -- 172.21.15.125:0/4089404640 >> v1:172.21.15.125:6789/0 conn(0x7fc7481580e0 legacy=0x7fc748069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:48.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 --2- 172.21.15.125:0/4089404640 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc74815e700 0x7fc734003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:48.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 --2- 172.21.15.125:0/4089404640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc734003f00 0x7fc7340047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:48.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 --2- 172.21.15.125:0/4089404640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc734004d10 0x7fc734005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:48.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 -- 172.21.15.125:0/4089404640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74815eb90 con 0x7fc734003f00 2023-10-30T13:05:48.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 -- 172.21.15.125:0/4089404640 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7480c6bb0 con 0x7fc734004d10 2023-10-30T13:05:48.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc73ffff640 1 -- 172.21.15.125:0/4089404640 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc73400b990 con 0x7fc74815e700 2023-10-30T13:05:48.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.851+0000 7fc74ca85640 1 --2- 172.21.15.125:0/4089404640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc734003f00 0x7fc7340047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:48.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.855+0000 7fc74d286640 1 --2- 172.21.15.125:0/4089404640 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc74815e700 0x7fc734003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:48.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.855+0000 7fc74da87640 1 --2- 172.21.15.125:0/4089404640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc734004d10 0x7fc734005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:48.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.855+0000 7fc74ca85640 1 -- 172.21.15.125:0/4089404640 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc74815e700 msgr2=0x7fc734003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:48.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.855+0000 7fc74ca85640 1 --2- 172.21.15.125:0/4089404640 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc74815e700 0x7fc734003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:48.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.855+0000 7fc74ca85640 1 -- 172.21.15.125:0/4089404640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc734004d10 msgr2=0x7fc734005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:48.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.855+0000 7fc74ca85640 1 --2- 172.21.15.125:0/4089404640 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc734004d10 0x7fc734005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:48.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.855+0000 7fc74ca85640 1 -- 172.21.15.125:0/4089404640 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc734003530 con 0x7fc734003f00 2023-10-30T13:05:48.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.855+0000 7fc74ca85640 1 --2- 172.21.15.125:0/4089404640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc734003f00 0x7fc7340047d0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fc738009ae0 tx=0x7fc73801ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f2cb6f8c4d6f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:48.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.855+0000 7fc73ffff640 1 -- 172.21.15.125:0/4089404640 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7380a5680 con 0x7fc734003f00 2023-10-30T13:05:48.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.855+0000 7fc73ffff640 1 -- 172.21.15.125:0/4089404640 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7380a57e0 con 0x7fc734003f00 2023-10-30T13:05:48.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.855+0000 7fc73ffff640 1 -- 172.21.15.125:0/4089404640 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7380a5b90 con 0x7fc734003f00 2023-10-30T13:05:48.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74f4ea640 1 -- 172.21.15.125:0/4089404640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc734003f00 msgr2=0x7fc7340047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:48.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74f4ea640 1 --2- 172.21.15.125:0/4089404640 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc734003f00 0x7fc7340047d0 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fc738009ae0 tx=0x7fc73801ca30 comp rx=0 tx=0).stop 2023-10-30T13:05:48.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74da87640 1 -- 172.21.15.125:0/4089404640 reap_dead start 2023-10-30T13:05:48.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74f4ea640 1 -- 172.21.15.125:0/4089404640 shutdown_connections 2023-10-30T13:05:48.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74f4ea640 1 -- 172.21.15.125:0/4089404640 >> 172.21.15.125:0/4089404640 conn(0x7fc7480c7910 msgr2=0x7fc7480c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:48.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74f4ea640 1 -- 172.21.15.125:0/4089404640 shutdown_connections 2023-10-30T13:05:48.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74f4ea640 1 -- 172.21.15.125:0/4089404640 wait complete. 2023-10-30T13:05:48.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74f4ea640 1 Processor -- start 2023-10-30T13:05:48.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74f4ea640 1 -- start start 2023-10-30T13:05:48.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74f4ea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc734003f00 0x7fc74815f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:48.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74f4ea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7481565d0 0x7fc74815f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:48.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74f4ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7481f4a10 0x7fc7481f4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:48.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74f4ea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc748161be0 con 0x7fc734003f00 2023-10-30T13:05:48.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74f4ea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc74815dc90 con 0x7fc7481565d0 2023-10-30T13:05:48.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74f4ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc74815e260 con 0x7fc7481f4a10 2023-10-30T13:05:48.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74d286640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc734003f00 0x7fc74815f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:48.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74d286640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc734003f00 0x7fc74815f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55882/0 (socket says 172.21.15.125:55882) 2023-10-30T13:05:48.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74d286640 1 -- 172.21.15.125:0/2751796984 learned_addr learned my addr 172.21.15.125:0/2751796984 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:48.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74ca85640 1 --2- 172.21.15.125:0/2751796984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7481565d0 0x7fc74815f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:48.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74da87640 1 --2- 172.21.15.125:0/2751796984 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7481f4a10 0x7fc7481f4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:48.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74d286640 1 -- 172.21.15.125:0/2751796984 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7481f4a10 msgr2=0x7fc7481f4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:48.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74d286640 1 --2- 172.21.15.125:0/2751796984 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7481f4a10 0x7fc7481f4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:48.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74d286640 1 -- 172.21.15.125:0/2751796984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7481565d0 msgr2=0x7fc74815f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:48.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74d286640 1 --2- 172.21.15.125:0/2751796984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7481565d0 0x7fc74815f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:48.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74d286640 1 -- 172.21.15.125:0/2751796984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc748161840 con 0x7fc734003f00 2023-10-30T13:05:48.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.859+0000 7fc74d286640 1 --2- 172.21.15.125:0/2751796984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc734003f00 0x7fc74815f0a0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fc744012570 tx=0x7fc744012a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:48.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.863+0000 7fc73dffb640 1 -- 172.21.15.125:0/2751796984 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc744014810 con 0x7fc734003f00 2023-10-30T13:05:48.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.863+0000 7fc73dffb640 1 -- 172.21.15.125:0/2751796984 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7440149b0 con 0x7fc734003f00 2023-10-30T13:05:48.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.863+0000 7fc73dffb640 1 -- 172.21.15.125:0/2751796984 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc744014d60 con 0x7fc734003f00 2023-10-30T13:05:48.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.863+0000 7fc74f4ea640 1 -- 172.21.15.125:0/2751796984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7481f4580 con 0x7fc734003f00 2023-10-30T13:05:48.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.863+0000 7fc74f4ea640 1 -- 172.21.15.125:0/2751796984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc74815d9b0 con 0x7fc734003f00 2023-10-30T13:05:48.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.863+0000 7fc74f4ea640 1 -- 172.21.15.125:0/2751796984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7481f4710 con 0x7fc734003f00 2023-10-30T13:05:48.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.863+0000 7fc73dffb640 1 -- 172.21.15.125:0/2751796984 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc744016480 con 0x7fc734003f00 2023-10-30T13:05:48.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.863+0000 7fc73dffb640 1 --2- 172.21.15.125:0/2751796984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc72c0664a0 0x7fc72c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:48.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.863+0000 7fc73dffb640 1 -- 172.21.15.125:0/2751796984 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7fc744087a00 con 0x7fc734003f00 2023-10-30T13:05:48.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.867+0000 7fc74ca85640 1 --2- 172.21.15.125:0/2751796984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc72c0664a0 0x7fc72c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:48.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.867+0000 7fc74ca85640 1 --2- 172.21.15.125:0/2751796984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc72c0664a0 0x7fc72c068960 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc7380af000 tx=0x7fc7380046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:48.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:48.871+0000 7fc73dffb640 1 -- 172.21.15.125:0/2751796984 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc7440547f0 con 0x7fc734003f00 2023-10-30T13:05:49.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:49.055+0000 7fc74f4ea640 1 -- 172.21.15.125:0/2751796984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc74804e170 con 0x7fc734003f00 2023-10-30T13:05:49.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:49.055+0000 7fc73dffb640 1 -- 172.21.15.125:0/2751796984 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fc744057e10 con 0x7fc734003f00 2023-10-30T13:05:49.058 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:49.058 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8178,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:15.605070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:49.059 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 161 2023-10-30T13:05:49.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:49.059+0000 7fc74f4ea640 1 -- 172.21.15.125:0/2751796984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc72c0664a0 msgr2=0x7fc72c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:49.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:49.059+0000 7fc74f4ea640 1 --2- 172.21.15.125:0/2751796984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc72c0664a0 0x7fc72c068960 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fc7380af000 tx=0x7fc7380046d0 comp rx=0 tx=0).stop 2023-10-30T13:05:49.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:49.059+0000 7fc74f4ea640 1 -- 172.21.15.125:0/2751796984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc734003f00 msgr2=0x7fc74815f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:49.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:49.059+0000 7fc74f4ea640 1 --2- 172.21.15.125:0/2751796984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc734003f00 0x7fc74815f0a0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fc744012570 tx=0x7fc744012a30 comp rx=0 tx=0).stop 2023-10-30T13:05:49.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:49.059+0000 7fc74f4ea640 1 -- 172.21.15.125:0/2751796984 shutdown_connections 2023-10-30T13:05:49.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:49.059+0000 7fc74f4ea640 1 --2- 172.21.15.125:0/2751796984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc72c0664a0 0x7fc72c068960 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:49.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:49.059+0000 7fc74f4ea640 1 --2- 172.21.15.125:0/2751796984 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc7481f4a10 0x7fc7481f4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:49.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:49.059+0000 7fc74f4ea640 1 --2- 172.21.15.125:0/2751796984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc7481565d0 0x7fc74815f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:49.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:49.059+0000 7fc74f4ea640 1 --2- 172.21.15.125:0/2751796984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc734003f00 0x7fc74815f0a0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:49.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:49.059+0000 7fc74f4ea640 1 -- 172.21.15.125:0/2751796984 >> 172.21.15.125:0/2751796984 conn(0x7fc7480c7910 msgr2=0x7fc748071b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:49.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:49.059+0000 7fc74f4ea640 1 -- 172.21.15.125:0/2751796984 shutdown_connections 2023-10-30T13:05:49.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:49.059+0000 7fc74f4ea640 1 -- 172.21.15.125:0/2751796984 wait complete. 2023-10-30T13:05:49.079 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:05:51.080 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:51.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.223+0000 7f0c2a194640 1 Processor -- start 2023-10-30T13:05:51.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.223+0000 7f0c2a194640 1 -- start start 2023-10-30T13:05:51.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c2a194640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c24153bf0 0x7f0c24071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:51.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c2a194640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0c240c5f60 0x7f0c240cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:51.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c2a194640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c240700b0 con 0x7f0c240c6fe0 2023-10-30T13:05:51.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c2a194640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c240c6810 con 0x7f0c240c5f60 2023-10-30T13:05:51.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c2a194640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c24159900 con 0x7f0c24153bf0 2023-10-30T13:05:51.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c237fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c24153bf0 0x7f0c24071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:51.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c23fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0c240c5f60 0x7f0c240cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:51.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c22ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0c240c6fe0 0x7f0c240c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:54162/0 (socket says 172.21.15.125:54162) 2023-10-30T13:05:51.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c237fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c24153bf0 0x7f0c24071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:56344/0 (socket says 172.21.15.125:56344) 2023-10-30T13:05:51.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c22ffd640 1 -- 172.21.15.125:0/691577008 learned_addr learned my addr 172.21.15.125:0/691577008 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:05:51.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c23fff640 1 -- 172.21.15.125:0/691577008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c24153bf0 msgr2=0x7f0c24071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:51.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c23fff640 1 --2- 172.21.15.125:0/691577008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c24153bf0 0x7f0c24071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:51.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c23fff640 1 -- 172.21.15.125:0/691577008 >> v1:172.21.15.136:6789/0 conn(0x7f0c240c6fe0 legacy=0x7f0c240c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:51.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c23fff640 1 -- 172.21.15.125:0/691577008 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c240c6530 con 0x7f0c240c5f60 2023-10-30T13:05:51.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.227+0000 7f0c237fe640 1 --2- 172.21.15.125:0/691577008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c24153bf0 0x7f0c24071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:05:51.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.231+0000 7f0c23fff640 1 --2- 172.21.15.125:0/691577008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c240c5f60 0x7f0c240cb570 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c009d90 tx=0x7f0c1c01f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6c4ffa803c89d69 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:51.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.231+0000 7f0c227fc640 1 -- 172.21.15.125:0/691577008 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c0af600 con 0x7f0c240c5f60 2023-10-30T13:05:51.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.231+0000 7f0c227fc640 1 -- 172.21.15.125:0/691577008 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1c0ab050 con 0x7f0c240c5f60 2023-10-30T13:05:51.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.231+0000 7f0c227fc640 1 -- 172.21.15.125:0/691577008 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c0af980 con 0x7f0c240c5f60 2023-10-30T13:05:51.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.231+0000 7f0c2a194640 1 -- 172.21.15.125:0/691577008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c240c5f60 msgr2=0x7f0c240cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:51.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.231+0000 7f0c2a194640 1 --2- 172.21.15.125:0/691577008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c240c5f60 0x7f0c240cb570 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c009d90 tx=0x7f0c1c01f130 comp rx=0 tx=0).stop 2023-10-30T13:05:51.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.231+0000 7f0c2a194640 1 -- 172.21.15.125:0/691577008 shutdown_connections 2023-10-30T13:05:51.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.231+0000 7f0c2a194640 1 --2- 172.21.15.125:0/691577008 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c240c5f60 0x7f0c240cb570 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:51.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.231+0000 7f0c2a194640 1 --2- 172.21.15.125:0/691577008 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c24153bf0 0x7f0c24071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:51.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.231+0000 7f0c2a194640 1 -- 172.21.15.125:0/691577008 >> 172.21.15.125:0/691577008 conn(0x7f0c2406c6e0 msgr2=0x7f0c24069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:51.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.231+0000 7f0c2a194640 1 -- 172.21.15.125:0/691577008 shutdown_connections 2023-10-30T13:05:51.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.231+0000 7f0c2a194640 1 -- 172.21.15.125:0/691577008 wait complete. 2023-10-30T13:05:51.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.231+0000 7f0c2a194640 1 Processor -- start 2023-10-30T13:05:51.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c2a194640 1 -- start start 2023-10-30T13:05:51.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c2a194640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c240c5f60 0x7f0c240c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:51.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c2a194640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c240c6fe0 0x7f0c240c5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:51.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c2a194640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c24153bf0 0x7f0c241ef5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:51.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c2a194640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c24154e10 con 0x7f0c24153bf0 2023-10-30T13:05:51.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c2a194640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c241587f0 con 0x7f0c240c5f60 2023-10-30T13:05:51.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c2a194640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c241ef150 con 0x7f0c240c6fe0 2023-10-30T13:05:51.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c23fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c24153bf0 0x7f0c241ef5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:51.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c23fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c24153bf0 0x7f0c241ef5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55426/0 (socket says 172.21.15.125:55426) 2023-10-30T13:05:51.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c23fff640 1 -- 172.21.15.125:0/374715420 learned_addr learned my addr 172.21.15.125:0/374715420 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:51.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c237fe640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c240c5f60 0x7f0c240c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:51.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c22ffd640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c240c6fe0 0x7f0c240c5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:51.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c23fff640 1 -- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c240c6fe0 msgr2=0x7f0c240c5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:51.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c23fff640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c240c6fe0 0x7f0c240c5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:51.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c23fff640 1 -- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c240c5f60 msgr2=0x7f0c240c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:51.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c23fff640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c240c5f60 0x7f0c240c7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:51.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c23fff640 1 -- 172.21.15.125:0/374715420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c14009420 con 0x7f0c24153bf0 2023-10-30T13:05:51.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c237fe640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c240c5f60 0x7f0c240c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:05:51.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c23fff640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c24153bf0 0x7f0c241ef5a0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c01cd60 tx=0x7f0c1c01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:51.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c0bfff640 1 -- 172.21.15.125:0/374715420 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c0afca0 con 0x7f0c24153bf0 2023-10-30T13:05:51.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c0bfff640 1 -- 172.21.15.125:0/374715420 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1c0afe00 con 0x7f0c24153bf0 2023-10-30T13:05:51.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c0bfff640 1 -- 172.21.15.125:0/374715420 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1c0ba660 con 0x7f0c24153bf0 2023-10-30T13:05:51.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c2a194640 1 -- 172.21.15.125:0/374715420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c1c009a70 con 0x7f0c24153bf0 2023-10-30T13:05:51.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c2a194640 1 -- 172.21.15.125:0/374715420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c241540b0 con 0x7f0c24153bf0 2023-10-30T13:05:51.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.235+0000 7f0c2a194640 1 -- 172.21.15.125:0/374715420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c241ef2c0 con 0x7f0c24153bf0 2023-10-30T13:05:51.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.239+0000 7f0c0bfff640 1 -- 172.21.15.125:0/374715420 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0c1c0ba7c0 con 0x7f0c24153bf0 2023-10-30T13:05:51.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.239+0000 7f0c0bfff640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c100664a0 0x7f0c10068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:51.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.239+0000 7f0c0bfff640 1 -- 172.21.15.125:0/374715420 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7f0c1c122f80 con 0x7f0c24153bf0 2023-10-30T13:05:51.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.239+0000 7f0c237fe640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c100664a0 0x7f0c10068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:51.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.239+0000 7f0c237fe640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c100664a0 0x7f0c10068960 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0c14009ca0 tx=0x7f0c14008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:51.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.243+0000 7f0c0bfff640 1 -- 172.21.15.125:0/374715420 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0c241ef2c0 con 0x7f0c24153bf0 2023-10-30T13:05:51.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.423+0000 7f0c2a194640 1 -- 172.21.15.125:0/374715420 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c241ef2c0 con 0x7f0c24153bf0 2023-10-30T13:05:51.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.427+0000 7f0c0bfff640 1 -- 172.21.15.125:0/374715420 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f0c241ef2c0 con 0x7f0c24153bf0 2023-10-30T13:05:51.428 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:51.428 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8178,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:15.605070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 161 2023-10-30T13:05:51.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.427+0000 7f0c2a194640 1 -- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c100664a0 msgr2=0x7f0c10068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:51.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.427+0000 7f0c2a194640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c100664a0 0x7f0c10068960 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f0c14009ca0 tx=0x7f0c14008040 comp rx=0 tx=0).stop 2023-10-30T13:05:51.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.427+0000 7f0c2a194640 1 -- 172.21.15.125:0/374715420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c24153bf0 msgr2=0x7f0c241ef5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:51.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.427+0000 7f0c2a194640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c24153bf0 0x7f0c241ef5a0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f0c1c01cd60 tx=0x7f0c1c01bed0 comp rx=0 tx=0).stop 2023-10-30T13:05:51.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.427+0000 7f0c2a194640 1 -- 172.21.15.125:0/374715420 shutdown_connections 2023-10-30T13:05:51.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.427+0000 7f0c2a194640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c100664a0 0x7f0c10068960 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.427+0000 7f0c2a194640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c24153bf0 0x7f0c241ef5a0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.427+0000 7f0c2a194640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c240c6fe0 0x7f0c240c5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.427+0000 7f0c2a194640 1 --2- 172.21.15.125:0/374715420 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c240c5f60 0x7f0c240c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.427+0000 7f0c2a194640 1 -- 172.21.15.125:0/374715420 >> 172.21.15.125:0/374715420 conn(0x7f0c2406c6e0 msgr2=0x7f0c24157e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.427+0000 7f0c2a194640 1 -- 172.21.15.125:0/374715420 shutdown_connections 2023-10-30T13:05:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:51.427+0000 7f0c2a194640 1 -- 172.21.15.125:0/374715420 wait complete. 2023-10-30T13:05:51.449 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:05:53.019 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:05:53.023 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:05:53.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:53.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.595+0000 7fa2d3532640 1 Processor -- start 2023-10-30T13:05:53.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.595+0000 7fa2d3532640 1 -- start start 2023-10-30T13:05:53.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d3532640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa2cc1538f0 0x7fa2cc159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:53.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d3532640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa2cc15cd50 0x7fa2cc0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:53.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d3532640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2cc15d300 con 0x7fa2cc0cefa0 2023-10-30T13:05:53.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d3532640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2cc15d5e0 con 0x7fa2cc15cd50 2023-10-30T13:05:53.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d3532640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2cc15d8c0 con 0x7fa2cc1538f0 2023-10-30T13:05:53.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d1d2f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa2cc15cd50 0x7fa2cc0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:53.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d2d31640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa2cc0cefa0 0x7fa2cc0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57556/0 (socket says 172.21.15.125:57556) 2023-10-30T13:05:53.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d2d31640 1 -- 172.21.15.125:0/292193889 learned_addr learned my addr 172.21.15.125:0/292193889 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:05:53.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d2530640 1 --2- 172.21.15.125:0/292193889 >> v2:172.21.15.136:3300/0 conn(0x7fa2cc1538f0 0x7fa2cc159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:53.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d1d2f640 1 -- 172.21.15.125:0/292193889 >> v2:172.21.15.136:3300/0 conn(0x7fa2cc1538f0 msgr2=0x7fa2cc159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:53.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d1d2f640 1 --2- 172.21.15.125:0/292193889 >> v2:172.21.15.136:3300/0 conn(0x7fa2cc1538f0 0x7fa2cc159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:53.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d1d2f640 1 -- 172.21.15.125:0/292193889 >> v1:172.21.15.125:6789/0 conn(0x7fa2cc0cefa0 legacy=0x7fa2cc0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:53.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d1d2f640 1 -- 172.21.15.125:0/292193889 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc15dba0 con 0x7fa2cc15cd50 2023-10-30T13:05:53.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d2530640 1 --2- 172.21.15.125:0/292193889 >> v2:172.21.15.136:3300/0 conn(0x7fa2cc1538f0 0x7fa2cc159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:05:53.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.599+0000 7fa2d152e640 1 -- 172.21.15.125:0/292193889 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1835308540 0 0) 0x7fa2cc15d300 con 0x7fa2cc0cefa0 2023-10-30T13:05:53.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d1d2f640 1 --2- 172.21.15.125:0/292193889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2cc15cd50 0x7fa2cc0c9f90 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0c9cb0 tx=0x7fa2bc01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ef4f50d6c62940 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:53.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d152e640 1 -- 172.21.15.125:0/292193889 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc0a8950 con 0x7fa2cc15cd50 2023-10-30T13:05:53.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d152e640 1 -- 172.21.15.125:0/292193889 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2bc0a8ab0 con 0x7fa2cc15cd50 2023-10-30T13:05:53.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d152e640 1 -- 172.21.15.125:0/292193889 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc0a8e40 con 0x7fa2cc15cd50 2023-10-30T13:05:53.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d3532640 1 -- 172.21.15.125:0/292193889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2cc15cd50 msgr2=0x7fa2cc0c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:53.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d3532640 1 --2- 172.21.15.125:0/292193889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2cc15cd50 0x7fa2cc0c9f90 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fa2cc0c9cb0 tx=0x7fa2bc01ec50 comp rx=0 tx=0).stop 2023-10-30T13:05:53.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d3532640 1 -- 172.21.15.125:0/292193889 shutdown_connections 2023-10-30T13:05:53.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d3532640 1 --2- 172.21.15.125:0/292193889 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2cc15cd50 0x7fa2cc0c9f90 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:53.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d3532640 1 --2- 172.21.15.125:0/292193889 >> v2:172.21.15.136:3300/0 conn(0x7fa2cc1538f0 0x7fa2cc159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:53.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d3532640 1 -- 172.21.15.125:0/292193889 >> 172.21.15.125:0/292193889 conn(0x7fa2cc06a640 msgr2=0x7fa2cc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:53.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d3532640 1 -- 172.21.15.125:0/292193889 shutdown_connections 2023-10-30T13:05:53.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d3532640 1 -- 172.21.15.125:0/292193889 wait complete. 2023-10-30T13:05:53.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d3532640 1 Processor -- start 2023-10-30T13:05:53.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d3532640 1 -- start start 2023-10-30T13:05:53.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d3532640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2cc0cefa0 0x7fa2cc145690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:53.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d3532640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2cc1538f0 0x7fa2cc1519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:53.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d3532640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2cc15cd50 0x7fa2cc0b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:53.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.603+0000 7fa2d3532640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc15c5c0 con 0x7fa2cc1538f0 2023-10-30T13:05:53.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d3532640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc151530 con 0x7fa2cc0cefa0 2023-10-30T13:05:53.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d3532640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2cc0b3dd0 con 0x7fa2cc15cd50 2023-10-30T13:05:53.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d1d2f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2cc1538f0 0x7fa2cc1519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:53.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d1d2f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2cc1538f0 0x7fa2cc1519c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55458/0 (socket says 172.21.15.125:55458) 2023-10-30T13:05:53.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d1d2f640 1 -- 172.21.15.125:0/2330474070 learned_addr learned my addr 172.21.15.125:0/2330474070 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:53.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d2d31640 1 --2- 172.21.15.125:0/2330474070 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2cc15cd50 0x7fa2cc0b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:53.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d2530640 1 --2- 172.21.15.125:0/2330474070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2cc0cefa0 0x7fa2cc145690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:53.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d1d2f640 1 -- 172.21.15.125:0/2330474070 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2cc15cd50 msgr2=0x7fa2cc0b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:53.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d1d2f640 1 --2- 172.21.15.125:0/2330474070 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2cc15cd50 0x7fa2cc0b49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:53.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d1d2f640 1 -- 172.21.15.125:0/2330474070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2cc0cefa0 msgr2=0x7fa2cc145690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:53.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d1d2f640 1 --2- 172.21.15.125:0/2330474070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2cc0cefa0 0x7fa2cc145690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:53.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d1d2f640 1 -- 172.21.15.125:0/2330474070 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc009420 con 0x7fa2cc1538f0 2023-10-30T13:05:53.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d1d2f640 1 --2- 172.21.15.125:0/2330474070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2cc1538f0 0x7fa2cc1519c0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc01c880 tx=0x7fa2bc01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:53.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2c2ffd640 1 -- 172.21.15.125:0/2330474070 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc0b9630 con 0x7fa2cc1538f0 2023-10-30T13:05:53.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2c2ffd640 1 -- 172.21.15.125:0/2330474070 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2bc0b97b0 con 0x7fa2cc1538f0 2023-10-30T13:05:53.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2c2ffd640 1 -- 172.21.15.125:0/2330474070 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2bc0b9b60 con 0x7fa2cc1538f0 2023-10-30T13:05:53.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d3532640 1 -- 172.21.15.125:0/2330474070 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2cc15c810 con 0x7fa2cc1538f0 2023-10-30T13:05:53.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d3532640 1 -- 172.21.15.125:0/2330474070 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2cc1511c0 con 0x7fa2cc1538f0 2023-10-30T13:05:53.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2d3532640 1 -- 172.21.15.125:0/2330474070 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2cc04e170 con 0x7fa2cc1538f0 2023-10-30T13:05:53.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2c2ffd640 1 -- 172.21.15.125:0/2330474070 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa2bc0b7040 con 0x7fa2cc1538f0 2023-10-30T13:05:53.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.607+0000 7fa2c2ffd640 1 --2- 172.21.15.125:0/2330474070 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2b00664a0 0x7fa2b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:53.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.611+0000 7fa2c2ffd640 1 -- 172.21.15.125:0/2330474070 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 7372+0+0 (secure 0 0 0) 0x7fa2bc1224b0 con 0x7fa2cc1538f0 2023-10-30T13:05:53.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.611+0000 7fa2d2530640 1 --2- 172.21.15.125:0/2330474070 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2b00664a0 0x7fa2b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:53.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.611+0000 7fa2d2530640 1 --2- 172.21.15.125:0/2330474070 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2b00664a0 0x7fa2b0068960 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fa2c800c900 tx=0x7fa2c8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:53.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.615+0000 7fa2c2ffd640 1 -- 172.21.15.125:0/2330474070 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa2bc0ef2a0 con 0x7fa2cc1538f0 2023-10-30T13:05:53.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.791+0000 7fa2d3532640 1 -- 172.21.15.125:0/2330474070 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2cc0cfca0 con 0x7fa2cc1538f0 2023-10-30T13:05:53.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.795+0000 7fa2c2ffd640 1 -- 172.21.15.125:0/2330474070 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fa2bc0f28c0 con 0x7fa2cc1538f0 2023-10-30T13:05:53.795 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:53.795 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8178,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":161}],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:15.605070+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8022},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8022":{"gid":8022,"name":"c","rank":0,"incarnation":156,"state":"up:active","state_seq":35,"addr":"172.21.15.125:6837/2196617378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2196617378},{"type":"v1","addr":"172.21.15.125:6837","nonce":2196617378}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:53.796 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 161 2023-10-30T13:05:53.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.795+0000 7fa2d3532640 1 -- 172.21.15.125:0/2330474070 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2b00664a0 msgr2=0x7fa2b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:53.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.795+0000 7fa2d3532640 1 --2- 172.21.15.125:0/2330474070 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2b00664a0 0x7fa2b0068960 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7fa2c800c900 tx=0x7fa2c8008040 comp rx=0 tx=0).stop 2023-10-30T13:05:53.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.795+0000 7fa2d3532640 1 -- 172.21.15.125:0/2330474070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2cc1538f0 msgr2=0x7fa2cc1519c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:53.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.795+0000 7fa2d3532640 1 --2- 172.21.15.125:0/2330474070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2cc1538f0 0x7fa2cc1519c0 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fa2bc01c880 tx=0x7fa2bc01b9f0 comp rx=0 tx=0).stop 2023-10-30T13:05:53.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.795+0000 7fa2d3532640 1 -- 172.21.15.125:0/2330474070 shutdown_connections 2023-10-30T13:05:53.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.795+0000 7fa2d3532640 1 --2- 172.21.15.125:0/2330474070 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2b00664a0 0x7fa2b0068960 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:53.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.795+0000 7fa2d3532640 1 --2- 172.21.15.125:0/2330474070 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2cc15cd50 0x7fa2cc0b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:53.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.795+0000 7fa2d3532640 1 --2- 172.21.15.125:0/2330474070 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2cc1538f0 0x7fa2cc1519c0 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:53.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.795+0000 7fa2d3532640 1 --2- 172.21.15.125:0/2330474070 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2cc0cefa0 0x7fa2cc145690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:53.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.795+0000 7fa2d3532640 1 -- 172.21.15.125:0/2330474070 >> 172.21.15.125:0/2330474070 conn(0x7fa2cc06a640 msgr2=0x7fa2cc0caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:53.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.795+0000 7fa2d3532640 1 -- 172.21.15.125:0/2330474070 shutdown_connections 2023-10-30T13:05:53.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:53.795+0000 7fa2d3532640 1 -- 172.21.15.125:0/2330474070 wait complete. 2023-10-30T13:05:53.816 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:05:55.817 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:55.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33fa846640 1 Processor -- start 2023-10-30T13:05:55.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33fa846640 1 -- start start 2023-10-30T13:05:55.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33fa846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33f41538a0 0x7f33f415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:55.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33fa846640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f33f40cf160 0x7f33f40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:55.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33fa846640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33f40cb800 con 0x7f33f40ca0e0 2023-10-30T13:05:55.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33fa846640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33f415dc30 con 0x7f33f40cf160 2023-10-30T13:05:55.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33fa846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f40cb330 con 0x7f33f41538a0 2023-10-30T13:05:55.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33f3fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33f41538a0 0x7f33f415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:55.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33f8de3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f33f40cf160 0x7f33f40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:55.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33f37fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f33f40ca0e0 0x7f33f415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:54164/0 (socket says 172.21.15.125:54164) 2023-10-30T13:05:55.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33f37fe640 1 -- 172.21.15.125:0/2636263255 learned_addr learned my addr 172.21.15.125:0/2636263255 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:05:55.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33f2ffd640 1 -- 172.21.15.125:0/2636263255 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832910877 0 0) 0x7f33f40cb800 con 0x7f33f40ca0e0 2023-10-30T13:05:55.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33f2ffd640 1 -- 172.21.15.125:0/2636263255 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33dc003530 con 0x7f33f40ca0e0 2023-10-30T13:05:55.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33f3fff640 1 -- 172.21.15.125:0/2636263255 >> v2:172.21.15.136:3300/0 conn(0x7f33f40cf160 msgr2=0x7f33f40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:55.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33f3fff640 1 --2- 172.21.15.125:0/2636263255 >> v2:172.21.15.136:3300/0 conn(0x7f33f40cf160 0x7f33f40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:55.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33f3fff640 1 -- 172.21.15.125:0/2636263255 >> v1:172.21.15.136:6789/0 conn(0x7f33f40ca0e0 legacy=0x7f33f415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:55.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.962+0000 7f33f3fff640 1 -- 172.21.15.125:0/2636263255 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f40cad70 con 0x7f33f41538a0 2023-10-30T13:05:55.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.966+0000 7f33f3fff640 1 --2- 172.21.15.125:0/2636263255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33f41538a0 0x7f33f415d6f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f33e40097c0 tx=0x7f33e401ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6079335c685297cb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:55.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.966+0000 7f33f2ffd640 1 -- 172.21.15.125:0/2636263255 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e40a8970 con 0x7f33f41538a0 2023-10-30T13:05:55.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.966+0000 7f33f2ffd640 1 -- 172.21.15.125:0/2636263255 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e40a8ad0 con 0x7f33f41538a0 2023-10-30T13:05:55.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.966+0000 7f33f2ffd640 1 -- 172.21.15.125:0/2636263255 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e40a8e40 con 0x7f33f41538a0 2023-10-30T13:05:55.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.966+0000 7f33fa846640 1 -- 172.21.15.125:0/2636263255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33f41538a0 msgr2=0x7f33f415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:55.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.966+0000 7f33fa846640 1 --2- 172.21.15.125:0/2636263255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33f41538a0 0x7f33f415d6f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f33e40097c0 tx=0x7f33e401ec70 comp rx=0 tx=0).stop 2023-10-30T13:05:55.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.966+0000 7f33fa846640 1 -- 172.21.15.125:0/2636263255 shutdown_connections 2023-10-30T13:05:55.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.966+0000 7f33fa846640 1 --2- 172.21.15.125:0/2636263255 >> v2:172.21.15.136:3300/0 conn(0x7f33f40cf160 0x7f33f40cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:55.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.966+0000 7f33fa846640 1 --2- 172.21.15.125:0/2636263255 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33f41538a0 0x7f33f415d6f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:55.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.966+0000 7f33fa846640 1 -- 172.21.15.125:0/2636263255 >> 172.21.15.125:0/2636263255 conn(0x7f33f406bfe0 msgr2=0x7f33f406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:55.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.966+0000 7f33fa846640 1 -- 172.21.15.125:0/2636263255 shutdown_connections 2023-10-30T13:05:55.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.966+0000 7f33fa846640 1 -- 172.21.15.125:0/2636263255 wait complete. 2023-10-30T13:05:55.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33fa846640 1 Processor -- start 2023-10-30T13:05:55.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33fa846640 1 -- start start 2023-10-30T13:05:55.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33fa846640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33f40ca0e0 0x7f33f40b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:55.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33fa846640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33f40cf160 0x7f33f40b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:55.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33fa846640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f41538a0 0x7f33f415cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:55.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33fa846640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f4148180 con 0x7f33f41538a0 2023-10-30T13:05:55.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33fa846640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f40b3990 con 0x7f33f40ca0e0 2023-10-30T13:05:55.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33fa846640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33f4147bc0 con 0x7f33f40cf160 2023-10-30T13:05:55.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f8de3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f41538a0 0x7f33f415cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:55.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f8de3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f41538a0 0x7f33f415cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55460/0 (socket says 172.21.15.125:55460) 2023-10-30T13:05:55.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f8de3640 1 -- 172.21.15.125:0/3992733239 learned_addr learned my addr 172.21.15.125:0/3992733239 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:55.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f37fe640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33f40cf160 0x7f33f40b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:55.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f3fff640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33f40ca0e0 0x7f33f40b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:55.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f8de3640 1 -- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33f40cf160 msgr2=0x7f33f40b3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:55.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f8de3640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33f40cf160 0x7f33f40b3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:55.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f8de3640 1 -- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33f40ca0e0 msgr2=0x7f33f40b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:55.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f8de3640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33f40ca0e0 0x7f33f40b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:55.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f8de3640 1 -- 172.21.15.125:0/3992733239 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e4009420 con 0x7f33f41538a0 2023-10-30T13:05:55.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f3fff640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33f40ca0e0 0x7f33f40b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:05:55.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f8de3640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f41538a0 0x7f33f415cc70 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f33ec00efc0 tx=0x7f33ec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:55.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f0ff9640 1 -- 172.21.15.125:0/3992733239 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec009b10 con 0x7f33f41538a0 2023-10-30T13:05:55.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f0ff9640 1 -- 172.21.15.125:0/3992733239 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec00f040 con 0x7f33f41538a0 2023-10-30T13:05:55.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33fa846640 1 -- 172.21.15.125:0/3992733239 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f4147ea0 con 0x7f33f41538a0 2023-10-30T13:05:55.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.970+0000 7f33f0ff9640 1 -- 172.21.15.125:0/3992733239 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec0146c0 con 0x7f33f41538a0 2023-10-30T13:05:55.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.974+0000 7f33fa846640 1 -- 172.21.15.125:0/3992733239 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f40b3620 con 0x7f33f41538a0 2023-10-30T13:05:55.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.974+0000 7f33fa846640 1 -- 172.21.15.125:0/3992733239 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33f40b4360 con 0x7f33f41538a0 2023-10-30T13:05:55.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.974+0000 7f33f0ff9640 1 -- 172.21.15.125:0/3992733239 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f33ec014820 con 0x7f33f41538a0 2023-10-30T13:05:55.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.974+0000 7f33f0ff9640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33e00664d0 0x7f33e0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:55.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.974+0000 7f33f0ff9640 1 -- 172.21.15.125:0/3992733239 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f33ec0852c0 con 0x7f33f41538a0 2023-10-30T13:05:55.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.974+0000 7f33f3fff640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33e00664d0 0x7f33e0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:55.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.978+0000 7f33f3fff640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33e00664d0 0x7f33e0068990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f33e40b2000 tx=0x7f33e40aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:55.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:55.978+0000 7f33f0ff9640 1 -- 172.21.15.125:0/3992733239 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f33ec052050 con 0x7f33f41538a0 2023-10-30T13:05:56.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.166+0000 7f33fa846640 1 -- 172.21.15.125:0/3992733239 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33f40b2a50 con 0x7f33f41538a0 2023-10-30T13:05:56.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.166+0000 7f33f0ff9640 1 -- 172.21.15.125:0/3992733239 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f33ec055670 con 0x7f33f41538a0 2023-10-30T13:05:56.168 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:56.169 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:55.585798+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8178},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:56.169 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 162 2023-10-30T13:05:56.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.166+0000 7f33fa846640 1 -- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33e00664d0 msgr2=0x7f33e0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:56.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.166+0000 7f33fa846640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33e00664d0 0x7f33e0068990 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f33e40b2000 tx=0x7f33e40aa070 comp rx=0 tx=0).stop 2023-10-30T13:05:56.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.166+0000 7f33fa846640 1 -- 172.21.15.125:0/3992733239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f41538a0 msgr2=0x7f33f415cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:56.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.166+0000 7f33fa846640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f41538a0 0x7f33f415cc70 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f33ec00efc0 tx=0x7f33ec00c490 comp rx=0 tx=0).stop 2023-10-30T13:05:56.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.166+0000 7f33fa846640 1 -- 172.21.15.125:0/3992733239 shutdown_connections 2023-10-30T13:05:56.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.166+0000 7f33fa846640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33e00664d0 0x7f33e0068990 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.166+0000 7f33fa846640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33f41538a0 0x7f33f415cc70 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.166+0000 7f33fa846640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33f40cf160 0x7f33f40b3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.166+0000 7f33fa846640 1 --2- 172.21.15.125:0/3992733239 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33f40ca0e0 0x7f33f40b4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.166+0000 7f33fa846640 1 -- 172.21.15.125:0/3992733239 >> 172.21.15.125:0/3992733239 conn(0x7f33f406bfe0 msgr2=0x7f33f4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:56.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.166+0000 7f33fa846640 1 -- 172.21.15.125:0/3992733239 shutdown_connections 2023-10-30T13:05:56.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.166+0000 7f33fa846640 1 -- 172.21.15.125:0/3992733239 wait complete. 2023-10-30T13:05:56.190 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:05:56.190 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:05:56.190 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:56.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e16565640 1 Processor -- start 2023-10-30T13:05:56.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e16565640 1 -- start start 2023-10-30T13:05:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e16565640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e10151800 0x7f9e1006e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e16565640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9e100c1ee0 0x7f9e100c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:56.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e16565640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9e100c9230 0x7f9e100cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:56.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e16565640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1006dc70 con 0x7f9e100c9230 2023-10-30T13:05:56.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e16565640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e1006e090 con 0x7f9e100c1ee0 2023-10-30T13:05:56.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e16565640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e100c8a50 con 0x7f9e10151800 2023-10-30T13:05:56.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e14b02640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9e100c9230 0x7f9e100cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:56.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e14b02640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9e100c9230 0x7f9e100cce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55472/0 (socket says 172.21.15.125:55472) 2023-10-30T13:05:56.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e14b02640 1 -- 172.21.15.125:0/621039673 learned_addr learned my addr 172.21.15.125:0/621039673 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:56.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e0ffff640 1 --2- 172.21.15.125:0/621039673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e10151800 0x7f9e1006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:56.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e0f7fe640 1 --2- 172.21.15.125:0/621039673 >> v2:172.21.15.136:3300/0 conn(0x7f9e100c1ee0 0x7f9e100c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:56.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e14b02640 1 -- 172.21.15.125:0/621039673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e10151800 msgr2=0x7f9e1006e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:56.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e14b02640 1 --2- 172.21.15.125:0/621039673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e10151800 0x7f9e1006e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e14b02640 1 -- 172.21.15.125:0/621039673 >> v2:172.21.15.136:3300/0 conn(0x7f9e100c1ee0 msgr2=0x7f9e100c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:56.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e14b02640 1 --2- 172.21.15.125:0/621039673 >> v2:172.21.15.136:3300/0 conn(0x7f9e100c1ee0 0x7f9e100c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.326+0000 7f9e14b02640 1 -- 172.21.15.125:0/621039673 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e100c3b10 con 0x7f9e100c9230 2023-10-30T13:05:56.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.330+0000 7f9e14b02640 1 --2- 172.21.15.125:0/621039673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e100c9230 0x7f9e100cce00 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f9e10070140 tx=0x7f9e0801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e53f62c5b3fa7c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:56.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.330+0000 7f9e0effd640 1 -- 172.21.15.125:0/621039673 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e080a5820 con 0x7f9e100c9230 2023-10-30T13:05:56.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.330+0000 7f9e0effd640 1 -- 172.21.15.125:0/621039673 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e080a5980 con 0x7f9e100c9230 2023-10-30T13:05:56.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.330+0000 7f9e0effd640 1 -- 172.21.15.125:0/621039673 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e080a5d10 con 0x7f9e100c9230 2023-10-30T13:05:56.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.330+0000 7f9e16565640 1 -- 172.21.15.125:0/621039673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e100c9230 msgr2=0x7f9e100cce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:56.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.330+0000 7f9e16565640 1 --2- 172.21.15.125:0/621039673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e100c9230 0x7f9e100cce00 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f9e10070140 tx=0x7f9e0801c9a0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.330+0000 7f9e16565640 1 -- 172.21.15.125:0/621039673 shutdown_connections 2023-10-30T13:05:56.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.330+0000 7f9e16565640 1 --2- 172.21.15.125:0/621039673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e100c9230 0x7f9e100cce00 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.330+0000 7f9e16565640 1 --2- 172.21.15.125:0/621039673 >> v2:172.21.15.136:3300/0 conn(0x7f9e100c1ee0 0x7f9e100c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.330+0000 7f9e16565640 1 --2- 172.21.15.125:0/621039673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e10151800 0x7f9e1006e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.330+0000 7f9e16565640 1 -- 172.21.15.125:0/621039673 >> 172.21.15.125:0/621039673 conn(0x7f9e10071ea0 msgr2=0x7f9e10065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:56.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.330+0000 7f9e16565640 1 -- 172.21.15.125:0/621039673 shutdown_connections 2023-10-30T13:05:56.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.330+0000 7f9e16565640 1 -- 172.21.15.125:0/621039673 wait complete. 2023-10-30T13:05:56.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e16565640 1 Processor -- start 2023-10-30T13:05:56.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e16565640 1 -- start start 2023-10-30T13:05:56.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e16565640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e100c1ee0 0x7f9e101efe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:56.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e16565640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e100c9230 0x7f9e10155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:56.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e16565640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e10151800 0x7f9e1014aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:56.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e16565640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1014aa30 con 0x7f9e10151800 2023-10-30T13:05:56.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e16565640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e101498d0 con 0x7f9e100c9230 2023-10-30T13:05:56.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e16565640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e10149b40 con 0x7f9e100c1ee0 2023-10-30T13:05:56.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e14b02640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e10151800 0x7f9e1014aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:56.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e14b02640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e10151800 0x7f9e1014aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55476/0 (socket says 172.21.15.125:55476) 2023-10-30T13:05:56.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e14b02640 1 -- 172.21.15.125:0/1967220396 learned_addr learned my addr 172.21.15.125:0/1967220396 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:56.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e0ffff640 1 --2- 172.21.15.125:0/1967220396 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e100c1ee0 0x7f9e101efe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:56.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e0f7fe640 1 --2- 172.21.15.125:0/1967220396 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e100c9230 0x7f9e10155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:56.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e14b02640 1 -- 172.21.15.125:0/1967220396 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e100c1ee0 msgr2=0x7f9e101efe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:56.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e14b02640 1 --2- 172.21.15.125:0/1967220396 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e100c1ee0 0x7f9e101efe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e14b02640 1 -- 172.21.15.125:0/1967220396 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e100c9230 msgr2=0x7f9e10155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:56.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e14b02640 1 --2- 172.21.15.125:0/1967220396 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e100c9230 0x7f9e10155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e14b02640 1 -- 172.21.15.125:0/1967220396 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e08009b30 con 0x7f9e10151800 2023-10-30T13:05:56.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e14b02640 1 --2- 172.21.15.125:0/1967220396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e10151800 0x7f9e1014aec0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f9e0801d240 tx=0x7f9e08019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:56.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e0cff9640 1 -- 172.21.15.125:0/1967220396 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e080b8670 con 0x7f9e10151800 2023-10-30T13:05:56.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e0cff9640 1 -- 172.21.15.125:0/1967220396 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e080b8810 con 0x7f9e10151800 2023-10-30T13:05:56.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e0cff9640 1 -- 172.21.15.125:0/1967220396 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e080b8bc0 con 0x7f9e10151800 2023-10-30T13:05:56.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e16565640 1 -- 172.21.15.125:0/1967220396 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e10149e20 con 0x7f9e10151800 2023-10-30T13:05:56.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.334+0000 7f9e16565640 1 -- 172.21.15.125:0/1967220396 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1014a3e0 con 0x7f9e10151800 2023-10-30T13:05:56.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.338+0000 7f9e16565640 1 -- 172.21.15.125:0/1967220396 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1014abe0 con 0x7f9e10151800 2023-10-30T13:05:56.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.338+0000 7f9e0cff9640 1 -- 172.21.15.125:0/1967220396 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9e080b8d20 con 0x7f9e10151800 2023-10-30T13:05:56.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.338+0000 7f9e0cff9640 1 --2- 172.21.15.125:0/1967220396 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9dfc0664d0 0x7f9dfc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:56.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.338+0000 7f9e0cff9640 1 -- 172.21.15.125:0/1967220396 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f9e0811f9f0 con 0x7f9e10151800 2023-10-30T13:05:56.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.338+0000 7f9e0ffff640 1 --2- 172.21.15.125:0/1967220396 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9dfc0664d0 0x7f9dfc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:56.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.338+0000 7f9e0ffff640 1 --2- 172.21.15.125:0/1967220396 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9dfc0664d0 0x7f9dfc068990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009ac0 tx=0x7f9e00009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:56.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.342+0000 7f9e0cff9640 1 -- 172.21.15.125:0/1967220396 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9e1014abe0 con 0x7f9e10151800 2023-10-30T13:05:56.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.522+0000 7f9e16565640 1 -- 172.21.15.125:0/1967220396 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e1014abe0 con 0x7f9e10151800 2023-10-30T13:05:56.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.522+0000 7f9e0cff9640 1 -- 172.21.15.125:0/1967220396 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+5723 (secure 0 0 0) 0x7f9e1014abe0 con 0x7f9e10151800 2023-10-30T13:05:56.526 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:56.526 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:55.585798+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8178},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:56.527 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 162 2023-10-30T13:05:56.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.526+0000 7f9e16565640 1 -- 172.21.15.125:0/1967220396 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9dfc0664d0 msgr2=0x7f9dfc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:56.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.526+0000 7f9e16565640 1 --2- 172.21.15.125:0/1967220396 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9dfc0664d0 0x7f9dfc068990 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009ac0 tx=0x7f9e00009190 comp rx=0 tx=0).stop 2023-10-30T13:05:56.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.526+0000 7f9e16565640 1 -- 172.21.15.125:0/1967220396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e10151800 msgr2=0x7f9e1014aec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:56.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.526+0000 7f9e16565640 1 --2- 172.21.15.125:0/1967220396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e10151800 0x7f9e1014aec0 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7f9e0801d240 tx=0x7f9e08019740 comp rx=0 tx=0).stop 2023-10-30T13:05:56.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.526+0000 7f9e16565640 1 -- 172.21.15.125:0/1967220396 shutdown_connections 2023-10-30T13:05:56.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.526+0000 7f9e16565640 1 --2- 172.21.15.125:0/1967220396 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9dfc0664d0 0x7f9dfc068990 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.526+0000 7f9e16565640 1 --2- 172.21.15.125:0/1967220396 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9e10151800 0x7f9e1014aec0 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.526+0000 7f9e16565640 1 --2- 172.21.15.125:0/1967220396 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9e100c9230 0x7f9e10155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.526+0000 7f9e16565640 1 --2- 172.21.15.125:0/1967220396 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9e100c1ee0 0x7f9e101efe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:56.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.526+0000 7f9e16565640 1 -- 172.21.15.125:0/1967220396 >> 172.21.15.125:0/1967220396 conn(0x7f9e10071ea0 msgr2=0x7f9e101576b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:56.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.526+0000 7f9e16565640 1 -- 172.21.15.125:0/1967220396 shutdown_connections 2023-10-30T13:05:56.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:56.526+0000 7f9e16565640 1 -- 172.21.15.125:0/1967220396 wait complete. 2023-10-30T13:05:56.548 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 162, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 162, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8178": { "addr": "172.21.15.125:6836/3711687505", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3711687505, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3711687505, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8178, "incarnation": 162, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:05:55.585798+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 8178 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:05:58.550 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:05:58.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.694+0000 7fe0eb3d3640 1 Processor -- start 2023-10-30T13:05:58.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.694+0000 7fe0eb3d3640 1 -- start start 2023-10-30T13:05:58.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0eb3d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0e40c6bd0 0x7fe0e406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:58.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0eb3d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe0e4071110 0x7fe0e4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:58.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0eb3d3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e4066cf0 con 0x7fe0e406f7a0 2023-10-30T13:05:58.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0eb3d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e40700c0 con 0x7fe0e4071110 2023-10-30T13:05:58.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0eb3d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e40c70d0 con 0x7fe0e40c6bd0 2023-10-30T13:05:58.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0e916f640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe0e406f7a0 0x7fe0e4069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57568/0 (socket says 172.21.15.125:57568) 2023-10-30T13:05:58.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0e916f640 1 -- 172.21.15.125:0/997232231 learned_addr learned my addr 172.21.15.125:0/997232231 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:05:58.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0e9970640 1 --2- 172.21.15.125:0/997232231 >> v2:172.21.15.125:3300/0 conn(0x7fe0e4071110 0x7fe0e4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:58.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0e896e640 1 --2- 172.21.15.125:0/997232231 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0e40c6bd0 0x7fe0e406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:58.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0dbfff640 1 -- 172.21.15.125:0/997232231 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447778634 0 0) 0x7fe0e4066cf0 con 0x7fe0e406f7a0 2023-10-30T13:05:58.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0dbfff640 1 -- 172.21.15.125:0/997232231 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d0003530 con 0x7fe0e406f7a0 2023-10-30T13:05:58.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0e9970640 1 -- 172.21.15.125:0/997232231 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0e40c6bd0 msgr2=0x7fe0e406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:05:58.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0e9970640 1 --2- 172.21.15.125:0/997232231 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0e40c6bd0 0x7fe0e406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:58.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0e9970640 1 -- 172.21.15.125:0/997232231 >> v1:172.21.15.125:6789/0 conn(0x7fe0e406f7a0 legacy=0x7fe0e4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:58.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0e9970640 1 -- 172.21.15.125:0/997232231 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e40c7390 con 0x7fe0e4071110 2023-10-30T13:05:58.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0e896e640 1 --2- 172.21.15.125:0/997232231 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0e40c6bd0 0x7fe0e406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:05:58.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.698+0000 7fe0e9970640 1 --2- 172.21.15.125:0/997232231 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0e4071110 0x7fe0e4158110 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc009d90 tx=0x7fe0dc01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb26821ecdbb4db7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:58.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0dbfff640 1 -- 172.21.15.125:0/997232231 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc0af600 con 0x7fe0e4071110 2023-10-30T13:05:58.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0dbfff640 1 -- 172.21.15.125:0/997232231 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0dc0ab050 con 0x7fe0e4071110 2023-10-30T13:05:58.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0dbfff640 1 -- 172.21.15.125:0/997232231 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc0af980 con 0x7fe0e4071110 2023-10-30T13:05:58.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/997232231 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0e4071110 msgr2=0x7fe0e4158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:58.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 --2- 172.21.15.125:0/997232231 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0e4071110 0x7fe0e4158110 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc009d90 tx=0x7fe0dc01f130 comp rx=0 tx=0).stop 2023-10-30T13:05:58.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/997232231 shutdown_connections 2023-10-30T13:05:58.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 --2- 172.21.15.125:0/997232231 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0e4071110 0x7fe0e4158110 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:58.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 --2- 172.21.15.125:0/997232231 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0e40c6bd0 0x7fe0e406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:58.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/997232231 >> 172.21.15.125:0/997232231 conn(0x7fe0e40c7910 msgr2=0x7fe0e40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:58.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/997232231 shutdown_connections 2023-10-30T13:05:58.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/997232231 wait complete. 2023-10-30T13:05:58.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 Processor -- start 2023-10-30T13:05:58.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 -- start start 2023-10-30T13:05:58.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0e406f7a0 0x7fe0e41ee790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:58.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0e4071110 0x7fe0e41554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:58.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0e40c6bd0 0x7fe0e40c3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:58.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e41569e0 con 0x7fe0e40c6bd0 2023-10-30T13:05:58.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e4155da0 con 0x7fe0e406f7a0 2023-10-30T13:05:58.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0eb3d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e4156080 con 0x7fe0e4071110 2023-10-30T13:05:58.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0e9970640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0e40c6bd0 0x7fe0e40c3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:58.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0e9970640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0e40c6bd0 0x7fe0e40c3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:55500/0 (socket says 172.21.15.125:55500) 2023-10-30T13:05:58.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0e9970640 1 -- 172.21.15.125:0/923418389 learned_addr learned my addr 172.21.15.125:0/923418389 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:05:58.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0e896e640 1 --2- 172.21.15.125:0/923418389 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0e4071110 0x7fe0e41554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:58.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.702+0000 7fe0e916f640 1 --2- 172.21.15.125:0/923418389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0e406f7a0 0x7fe0e41ee790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:58.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0e9970640 1 -- 172.21.15.125:0/923418389 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0e4071110 msgr2=0x7fe0e41554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:58.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0e9970640 1 --2- 172.21.15.125:0/923418389 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0e4071110 0x7fe0e41554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:58.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0e9970640 1 -- 172.21.15.125:0/923418389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0e406f7a0 msgr2=0x7fe0e41ee790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:58.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0e9970640 1 --2- 172.21.15.125:0/923418389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0e406f7a0 0x7fe0e41ee790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:58.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0e9970640 1 -- 172.21.15.125:0/923418389 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc009a70 con 0x7fe0e40c6bd0 2023-10-30T13:05:58.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0e9970640 1 --2- 172.21.15.125:0/923418389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0e40c6bd0 0x7fe0e40c3340 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0b2040 tx=0x7fe0dc01bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:58.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0d9ffb640 1 -- 172.21.15.125:0/923418389 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc009450 con 0x7fe0e40c6bd0 2023-10-30T13:05:58.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0d9ffb640 1 -- 172.21.15.125:0/923418389 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0dc0b9410 con 0x7fe0e40c6bd0 2023-10-30T13:05:58.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0d9ffb640 1 -- 172.21.15.125:0/923418389 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc0b97c0 con 0x7fe0e40c6bd0 2023-10-30T13:05:58.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/923418389 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e4156360 con 0x7fe0e40c6bd0 2023-10-30T13:05:58.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/923418389 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e4156f10 con 0x7fe0e40c6bd0 2023-10-30T13:05:58.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/923418389 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e4157100 con 0x7fe0e40c6bd0 2023-10-30T13:05:58.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0d9ffb640 1 -- 172.21.15.125:0/923418389 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe0dc0b6050 con 0x7fe0e40c6bd0 2023-10-30T13:05:58.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0d9ffb640 1 --2- 172.21.15.125:0/923418389 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0c80664a0 0x7fe0c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:05:58.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.706+0000 7fe0d9ffb640 1 -- 172.21.15.125:0/923418389 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7fe0dc1229f0 con 0x7fe0e40c6bd0 2023-10-30T13:05:58.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.710+0000 7fe0e916f640 1 --2- 172.21.15.125:0/923418389 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0c80664a0 0x7fe0c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:05:58.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.710+0000 7fe0e916f640 1 --2- 172.21.15.125:0/923418389 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0c80664a0 0x7fe0c8068960 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00011f0 tx=0x7fe0e0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:05:58.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.710+0000 7fe0d9ffb640 1 -- 172.21.15.125:0/923418389 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe0dc0ef790 con 0x7fe0e40c6bd0 2023-10-30T13:05:58.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.894+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/923418389 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0e404e170 con 0x7fe0e40c6bd0 2023-10-30T13:05:58.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.894+0000 7fe0d9ffb640 1 -- 172.21.15.125:0/923418389 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+5723 (secure 0 0 0) 0x7fe0dc0b4090 con 0x7fe0e40c6bd0 2023-10-30T13:05:58.897 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:05:58.897 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:55.585798+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8178},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:05:58.897 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 162 2023-10-30T13:05:58.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.894+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/923418389 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0c80664a0 msgr2=0x7fe0c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:58.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.894+0000 7fe0eb3d3640 1 --2- 172.21.15.125:0/923418389 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0c80664a0 0x7fe0c8068960 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00011f0 tx=0x7fe0e0009040 comp rx=0 tx=0).stop 2023-10-30T13:05:58.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.894+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/923418389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0e40c6bd0 msgr2=0x7fe0e40c3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:05:58.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.894+0000 7fe0eb3d3640 1 --2- 172.21.15.125:0/923418389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0e40c6bd0 0x7fe0e40c3340 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0b2040 tx=0x7fe0dc01bed0 comp rx=0 tx=0).stop 2023-10-30T13:05:58.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.894+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/923418389 shutdown_connections 2023-10-30T13:05:58.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.894+0000 7fe0eb3d3640 1 --2- 172.21.15.125:0/923418389 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0c80664a0 0x7fe0c8068960 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:58.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.894+0000 7fe0eb3d3640 1 --2- 172.21.15.125:0/923418389 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0e40c6bd0 0x7fe0e40c3340 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:58.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.894+0000 7fe0eb3d3640 1 --2- 172.21.15.125:0/923418389 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0e4071110 0x7fe0e41554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:58.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.894+0000 7fe0eb3d3640 1 --2- 172.21.15.125:0/923418389 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0e406f7a0 0x7fe0e41ee790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:05:58.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.894+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/923418389 >> 172.21.15.125:0/923418389 conn(0x7fe0e40c7910 msgr2=0x7fe0e40c8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:05:58.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.894+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/923418389 shutdown_connections 2023-10-30T13:05:58.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:05:58.894+0000 7fe0eb3d3640 1 -- 172.21.15.125:0/923418389 wait complete. 2023-10-30T13:05:58.918 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:06:00.920 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:01.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.066+0000 7fd9c0e09640 1 Processor -- start 2023-10-30T13:06:01.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.066+0000 7fd9c0e09640 1 -- start start 2023-10-30T13:06:01.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.066+0000 7fd9c0e09640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd9bc15d6f0 0x7fd9bc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:01.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.066+0000 7fd9c0e09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9bc0cf160 0x7fd9bc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:01.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.066+0000 7fd9c0e09640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9bc0cb800 con 0x7fd9bc1538a0 2023-10-30T13:06:01.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.066+0000 7fd9c0e09640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9bc0cb330 con 0x7fd9bc15d6f0 2023-10-30T13:06:01.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.066+0000 7fd9c0e09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc0cad70 con 0x7fd9bc0cf160 2023-10-30T13:06:01.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.066+0000 7fd9ba59c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd9bc1538a0 0x7fd9bc0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53246/0 (socket says 172.21.15.125:53246) 2023-10-30T13:06:01.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.066+0000 7fd9ba59c640 1 -- 172.21.15.125:0/736784598 learned_addr learned my addr 172.21.15.125:0/736784598 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:06:01.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.066+0000 7fd9bad9d640 1 --2- 172.21.15.125:0/736784598 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9bc0cf160 0x7fd9bc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:01.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.066+0000 7fd9b9d9b640 1 --2- 172.21.15.125:0/736784598 >> v2:172.21.15.136:3300/0 conn(0x7fd9bc15d6f0 0x7fd9bc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:01.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.066+0000 7fd9b959a640 1 -- 172.21.15.125:0/736784598 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3810615970 0 0) 0x7fd9bc0cb800 con 0x7fd9bc1538a0 2023-10-30T13:06:01.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.066+0000 7fd9b959a640 1 -- 172.21.15.125:0/736784598 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9a4003530 con 0x7fd9bc1538a0 2023-10-30T13:06:01.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b959a640 1 -- 172.21.15.125:0/736784598 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3389518127 0 0) 0x7fd9a4003530 con 0x7fd9bc1538a0 2023-10-30T13:06:01.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b959a640 1 -- 172.21.15.125:0/736784598 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9bc0cf160 msgr2=0x7fd9bc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:01.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b959a640 1 --2- 172.21.15.125:0/736784598 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9bc0cf160 0x7fd9bc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:01.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b959a640 1 -- 172.21.15.125:0/736784598 >> v2:172.21.15.136:3300/0 conn(0x7fd9bc15d6f0 msgr2=0x7fd9bc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:01.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b959a640 1 --2- 172.21.15.125:0/736784598 >> v2:172.21.15.136:3300/0 conn(0x7fd9bc15d6f0 0x7fd9bc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:01.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b959a640 1 -- 172.21.15.125:0/736784598 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9bc0cb050 con 0x7fd9bc1538a0 2023-10-30T13:06:01.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b959a640 1 -- 172.21.15.125:0/736784598 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fd9ac001680 con 0x7fd9bc1538a0 2023-10-30T13:06:01.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b959a640 1 -- 172.21.15.125:0/736784598 >> v1:172.21.15.125:6789/0 conn(0x7fd9bc1538a0 legacy=0x7fd9bc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:01.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b959a640 1 --2- 172.21.15.125:0/736784598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9a4003980 0x7fd9a4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:01.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b959a640 1 --2- 172.21.15.125:0/736784598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9a40047d0 0x7fd9a4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:01.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b959a640 1 --2- 172.21.15.125:0/736784598 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9bc0cf160 0x7fd9a4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:01.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b959a640 1 -- 172.21.15.125:0/736784598 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc0cb330 con 0x7fd9a40047d0 2023-10-30T13:06:01.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b959a640 1 -- 172.21.15.125:0/736784598 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc0cad70 con 0x7fd9a4003980 2023-10-30T13:06:01.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b959a640 1 -- 172.21.15.125:0/736784598 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9a400b990 con 0x7fd9bc0cf160 2023-10-30T13:06:01.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b9d9b640 1 --2- 172.21.15.125:0/736784598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9a40047d0 0x7fd9a4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:01.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9bad9d640 1 --2- 172.21.15.125:0/736784598 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9bc0cf160 0x7fd9a4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:01.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9ba59c640 1 --2- 172.21.15.125:0/736784598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9a4003980 0x7fd9a4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:01.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b9d9b640 1 -- 172.21.15.125:0/736784598 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9bc0cf160 msgr2=0x7fd9a4005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:01.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b9d9b640 1 --2- 172.21.15.125:0/736784598 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9bc0cf160 0x7fd9a4005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:01.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b9d9b640 1 -- 172.21.15.125:0/736784598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9a4003980 msgr2=0x7fd9a4003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:01.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b9d9b640 1 --2- 172.21.15.125:0/736784598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9a4003980 0x7fd9a4003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:01.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9b9d9b640 1 -- 172.21.15.125:0/736784598 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9a4003530 con 0x7fd9a40047d0 2023-10-30T13:06:01.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.070+0000 7fd9ba59c640 1 --2- 172.21.15.125:0/736784598 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9a4003980 0x7fd9a4003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:01.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.074+0000 7fd9b9d9b640 1 --2- 172.21.15.125:0/736784598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9a40047d0 0x7fd9a4004c50 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009ae0 tx=0x7fd9b001ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=112f1e1c14075b5f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:01.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.074+0000 7fd9b959a640 1 -- 172.21.15.125:0/736784598 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b00a55a0 con 0x7fd9a40047d0 2023-10-30T13:06:01.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.074+0000 7fd9b959a640 1 -- 172.21.15.125:0/736784598 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b00a5700 con 0x7fd9a40047d0 2023-10-30T13:06:01.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.074+0000 7fd9b959a640 1 -- 172.21.15.125:0/736784598 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b00a5ab0 con 0x7fd9a40047d0 2023-10-30T13:06:01.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.074+0000 7fd9c0e09640 1 -- 172.21.15.125:0/736784598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9a40047d0 msgr2=0x7fd9a4004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.074+0000 7fd9c0e09640 1 --2- 172.21.15.125:0/736784598 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9a40047d0 0x7fd9a4004c50 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fd9b0009ae0 tx=0x7fd9b001ca90 comp rx=0 tx=0).stop 2023-10-30T13:06:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.074+0000 7fd9bad9d640 1 -- 172.21.15.125:0/736784598 reap_dead start 2023-10-30T13:06:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.074+0000 7fd9c0e09640 1 -- 172.21.15.125:0/736784598 shutdown_connections 2023-10-30T13:06:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.074+0000 7fd9c0e09640 1 -- 172.21.15.125:0/736784598 >> 172.21.15.125:0/736784598 conn(0x7fd9bc06bfe0 msgr2=0x7fd9bc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:01.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.074+0000 7fd9c0e09640 1 -- 172.21.15.125:0/736784598 shutdown_connections 2023-10-30T13:06:01.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.074+0000 7fd9c0e09640 1 -- 172.21.15.125:0/736784598 wait complete. 2023-10-30T13:06:01.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9c0e09640 1 Processor -- start 2023-10-30T13:06:01.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9c0e09640 1 -- start start 2023-10-30T13:06:01.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9c0e09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9a40047d0 0x7fd9bc15bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:01.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9c0e09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9bc15c260 0x7fd9bc158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:01.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9c0e09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9bc159060 0x7fd9bc1f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:01.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9c0e09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc0cd0b0 con 0x7fd9bc159060 2023-10-30T13:06:01.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9c0e09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc15b2c0 con 0x7fd9a40047d0 2023-10-30T13:06:01.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9c0e09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9bc1efd80 con 0x7fd9bc15c260 2023-10-30T13:06:01.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9bad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9bc159060 0x7fd9bc1f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:01.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9bad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9bc159060 0x7fd9bc1f2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45452/0 (socket says 172.21.15.125:45452) 2023-10-30T13:06:01.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9bad9d640 1 -- 172.21.15.125:0/4255568539 learned_addr learned my addr 172.21.15.125:0/4255568539 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:01.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9ba59c640 1 --2- 172.21.15.125:0/4255568539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9a40047d0 0x7fd9bc15bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:01.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9b9d9b640 1 --2- 172.21.15.125:0/4255568539 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9bc15c260 0x7fd9bc158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:01.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9bad9d640 1 -- 172.21.15.125:0/4255568539 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9bc15c260 msgr2=0x7fd9bc158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:01.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9bad9d640 1 --2- 172.21.15.125:0/4255568539 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9bc15c260 0x7fd9bc158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:01.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9bad9d640 1 -- 172.21.15.125:0/4255568539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9a40047d0 msgr2=0x7fd9bc15bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:01.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9bad9d640 1 --2- 172.21.15.125:0/4255568539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9a40047d0 0x7fd9bc15bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:01.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9bad9d640 1 -- 172.21.15.125:0/4255568539 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9bc15a920 con 0x7fd9bc159060 2023-10-30T13:06:01.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9bad9d640 1 --2- 172.21.15.125:0/4255568539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9bc159060 0x7fd9bc1f2380 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc072ce0 tx=0x7fd9b400f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:01.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9aaffd640 1 -- 172.21.15.125:0/4255568539 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b400e040 con 0x7fd9bc159060 2023-10-30T13:06:01.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9aaffd640 1 -- 172.21.15.125:0/4255568539 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9b400f040 con 0x7fd9bc159060 2023-10-30T13:06:01.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9c0e09640 1 -- 172.21.15.125:0/4255568539 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9bc15c6a0 con 0x7fd9bc159060 2023-10-30T13:06:01.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9aaffd640 1 -- 172.21.15.125:0/4255568539 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9b4015430 con 0x7fd9bc159060 2023-10-30T13:06:01.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9c0e09640 1 -- 172.21.15.125:0/4255568539 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9bc15b510 con 0x7fd9bc159060 2023-10-30T13:06:01.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.078+0000 7fd9c0e09640 1 -- 172.21.15.125:0/4255568539 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9bc15b700 con 0x7fd9bc159060 2023-10-30T13:06:01.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.082+0000 7fd9aaffd640 1 -- 172.21.15.125:0/4255568539 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd9b4015590 con 0x7fd9bc159060 2023-10-30T13:06:01.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.082+0000 7fd9aaffd640 1 --2- 172.21.15.125:0/4255568539 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd99c0664d0 0x7fd99c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:01.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.082+0000 7fd9aaffd640 1 -- 172.21.15.125:0/4255568539 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7fd9b4084aa0 con 0x7fd9bc159060 2023-10-30T13:06:01.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.082+0000 7fd9ba59c640 1 --2- 172.21.15.125:0/4255568539 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd99c0664d0 0x7fd99c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:01.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.082+0000 7fd9ba59c640 1 --2- 172.21.15.125:0/4255568539 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd99c0664d0 0x7fd99c068990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac001ec0 tx=0x7fd9ac007040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:01.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.090+0000 7fd9aaffd640 1 -- 172.21.15.125:0/4255568539 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd9b4051850 con 0x7fd9bc159060 2023-10-30T13:06:01.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.306+0000 7fd9c0e09640 1 -- 172.21.15.125:0/4255568539 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9bc04e170 con 0x7fd9bc159060 2023-10-30T13:06:01.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.306+0000 7fd9aaffd640 1 -- 172.21.15.125:0/4255568539 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+5723 (secure 0 0 0) 0x7fd9b4054e70 con 0x7fd9bc159060 2023-10-30T13:06:01.310 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:01.310 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:55.585798+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8178},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:01.310 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 162 2023-10-30T13:06:01.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.310+0000 7fd9c0e09640 1 -- 172.21.15.125:0/4255568539 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd99c0664d0 msgr2=0x7fd99c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:01.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.310+0000 7fd9c0e09640 1 --2- 172.21.15.125:0/4255568539 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd99c0664d0 0x7fd99c068990 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fd9ac001ec0 tx=0x7fd9ac007040 comp rx=0 tx=0).stop 2023-10-30T13:06:01.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.310+0000 7fd9c0e09640 1 -- 172.21.15.125:0/4255568539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9bc159060 msgr2=0x7fd9bc1f2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:01.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.310+0000 7fd9c0e09640 1 --2- 172.21.15.125:0/4255568539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9bc159060 0x7fd9bc1f2380 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7fd9bc072ce0 tx=0x7fd9b400f1b0 comp rx=0 tx=0).stop 2023-10-30T13:06:01.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.310+0000 7fd9c0e09640 1 -- 172.21.15.125:0/4255568539 shutdown_connections 2023-10-30T13:06:01.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.310+0000 7fd9c0e09640 1 --2- 172.21.15.125:0/4255568539 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd99c0664d0 0x7fd99c068990 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:01.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.310+0000 7fd9c0e09640 1 --2- 172.21.15.125:0/4255568539 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9bc159060 0x7fd9bc1f2380 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:01.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.310+0000 7fd9c0e09640 1 --2- 172.21.15.125:0/4255568539 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9bc15c260 0x7fd9bc158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:01.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.310+0000 7fd9c0e09640 1 --2- 172.21.15.125:0/4255568539 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9a40047d0 0x7fd9bc15bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:01.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.310+0000 7fd9c0e09640 1 -- 172.21.15.125:0/4255568539 >> 172.21.15.125:0/4255568539 conn(0x7fd9bc06bfe0 msgr2=0x7fd9bc151870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:01.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.310+0000 7fd9c0e09640 1 -- 172.21.15.125:0/4255568539 shutdown_connections 2023-10-30T13:06:01.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:01.310+0000 7fd9c0e09640 1 -- 172.21.15.125:0/4255568539 wait complete. 2023-10-30T13:06:01.337 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:06:03.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:03.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.486+0000 7fdd39a9e640 1 Processor -- start 2023-10-30T13:06:03.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.486+0000 7fdd39a9e640 1 -- start start 2023-10-30T13:06:03.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.486+0000 7fdd39a9e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdd3406f7a0 0x7fdd3406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:03.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.486+0000 7fdd39a9e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd340c6bd0 0x7fdd34069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:03.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.486+0000 7fdd39a9e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdd34071110 0x7fdd34158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:03.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.486+0000 7fdd39a9e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd340700c0 con 0x7fdd34071110 2023-10-30T13:06:03.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.486+0000 7fdd39a9e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd3406a3d0 con 0x7fdd3406f7a0 2023-10-30T13:06:03.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.486+0000 7fdd39a9e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd34066cf0 con 0x7fdd340c6bd0 2023-10-30T13:06:03.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.486+0000 7fdd33fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdd34071110 0x7fdd34158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:03.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.490+0000 7fdd33fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdd34071110 0x7fdd34158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45456/0 (socket says 172.21.15.125:45456) 2023-10-30T13:06:03.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.490+0000 7fdd33fff640 1 -- 172.21.15.125:0/88862709 learned_addr learned my addr 172.21.15.125:0/88862709 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:03.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.490+0000 7fdd32ffd640 1 --2- 172.21.15.125:0/88862709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd340c6bd0 0x7fdd34069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:03.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.490+0000 7fdd337fe640 1 --2- 172.21.15.125:0/88862709 >> v2:172.21.15.136:3300/0 conn(0x7fdd3406f7a0 0x7fdd3406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:03.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.490+0000 7fdd33fff640 1 -- 172.21.15.125:0/88862709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd340c6bd0 msgr2=0x7fdd34069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:03.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.490+0000 7fdd33fff640 1 --2- 172.21.15.125:0/88862709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd340c6bd0 0x7fdd34069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:03.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.490+0000 7fdd33fff640 1 -- 172.21.15.125:0/88862709 >> v2:172.21.15.136:3300/0 conn(0x7fdd3406f7a0 msgr2=0x7fdd3406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:03.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.490+0000 7fdd33fff640 1 --2- 172.21.15.125:0/88862709 >> v2:172.21.15.136:3300/0 conn(0x7fdd3406f7a0 0x7fdd3406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:03.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.490+0000 7fdd33fff640 1 -- 172.21.15.125:0/88862709 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd340c70d0 con 0x7fdd34071110 2023-10-30T13:06:03.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.490+0000 7fdd337fe640 1 --2- 172.21.15.125:0/88862709 >> v2:172.21.15.136:3300/0 conn(0x7fdd3406f7a0 0x7fdd3406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:03.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.490+0000 7fdd32ffd640 1 --2- 172.21.15.125:0/88862709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd340c6bd0 0x7fdd34069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:03.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd33fff640 1 --2- 172.21.15.125:0/88862709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd34071110 0x7fdd34158110 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fdd3406c830 tx=0x7fdd2c01f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34beedc4f2de4a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:03.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd327fc640 1 -- 172.21.15.125:0/88862709 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c0af400 con 0x7fdd34071110 2023-10-30T13:06:03.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd327fc640 1 -- 172.21.15.125:0/88862709 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2c0ab050 con 0x7fdd34071110 2023-10-30T13:06:03.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd327fc640 1 -- 172.21.15.125:0/88862709 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c0af780 con 0x7fdd34071110 2023-10-30T13:06:03.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd39a9e640 1 -- 172.21.15.125:0/88862709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd34071110 msgr2=0x7fdd34158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:03.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd39a9e640 1 --2- 172.21.15.125:0/88862709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd34071110 0x7fdd34158110 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7fdd3406c830 tx=0x7fdd2c01f1c0 comp rx=0 tx=0).stop 2023-10-30T13:06:03.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd39a9e640 1 -- 172.21.15.125:0/88862709 shutdown_connections 2023-10-30T13:06:03.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd39a9e640 1 --2- 172.21.15.125:0/88862709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd34071110 0x7fdd34158110 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:03.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd39a9e640 1 --2- 172.21.15.125:0/88862709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd340c6bd0 0x7fdd34069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:03.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd39a9e640 1 --2- 172.21.15.125:0/88862709 >> v2:172.21.15.136:3300/0 conn(0x7fdd3406f7a0 0x7fdd3406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:03.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd39a9e640 1 -- 172.21.15.125:0/88862709 >> 172.21.15.125:0/88862709 conn(0x7fdd340c7910 msgr2=0x7fdd340c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:03.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd39a9e640 1 -- 172.21.15.125:0/88862709 shutdown_connections 2023-10-30T13:06:03.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd39a9e640 1 -- 172.21.15.125:0/88862709 wait complete. 2023-10-30T13:06:03.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd39a9e640 1 Processor -- start 2023-10-30T13:06:03.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.494+0000 7fdd39a9e640 1 -- start start 2023-10-30T13:06:03.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd39a9e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd3406f7a0 0x7fdd340ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:03.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd39a9e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd34071110 0x7fdd341f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:03.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd39a9e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd340c6bd0 0x7fdd341496c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:03.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd39a9e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd340c1ab0 con 0x7fdd3406f7a0 2023-10-30T13:06:03.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd39a9e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd340c17c0 con 0x7fdd34071110 2023-10-30T13:06:03.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd39a9e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd340c03f0 con 0x7fdd340c6bd0 2023-10-30T13:06:03.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd337fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd3406f7a0 0x7fdd340ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:03.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd337fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd3406f7a0 0x7fdd340ad8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45462/0 (socket says 172.21.15.125:45462) 2023-10-30T13:06:03.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd337fe640 1 -- 172.21.15.125:0/2238105948 learned_addr learned my addr 172.21.15.125:0/2238105948 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:03.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd32ffd640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd34071110 0x7fdd341f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:03.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd33fff640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd340c6bd0 0x7fdd341496c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:03.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd337fe640 1 -- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd340c6bd0 msgr2=0x7fdd341496c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:03.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd337fe640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd340c6bd0 0x7fdd341496c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:03.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd337fe640 1 -- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd34071110 msgr2=0x7fdd341f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:03.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd337fe640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd34071110 0x7fdd341f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:03.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd337fe640 1 -- 172.21.15.125:0/2238105948 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c009b30 con 0x7fdd3406f7a0 2023-10-30T13:06:03.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd32ffd640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd34071110 0x7fdd341f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:03.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd33fff640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd340c6bd0 0x7fdd341496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:03.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd337fe640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd3406f7a0 0x7fdd340ad8d0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400d800 tx=0x7fdd2400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:03.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd1bfff640 1 -- 172.21.15.125:0/2238105948 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24004030 con 0x7fdd3406f7a0 2023-10-30T13:06:03.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd1bfff640 1 -- 172.21.15.125:0/2238105948 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd24004190 con 0x7fdd3406f7a0 2023-10-30T13:06:03.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd39a9e640 1 -- 172.21.15.125:0/2238105948 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd340bfec0 con 0x7fdd3406f7a0 2023-10-30T13:06:03.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd1bfff640 1 -- 172.21.15.125:0/2238105948 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24004520 con 0x7fdd3406f7a0 2023-10-30T13:06:03.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd39a9e640 1 -- 172.21.15.125:0/2238105948 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd3414d090 con 0x7fdd3406f7a0 2023-10-30T13:06:03.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.498+0000 7fdd39a9e640 1 -- 172.21.15.125:0/2238105948 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3404e170 con 0x7fdd3406f7a0 2023-10-30T13:06:03.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.502+0000 7fdd1bfff640 1 -- 172.21.15.125:0/2238105948 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdd240096a0 con 0x7fdd3406f7a0 2023-10-30T13:06:03.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.502+0000 7fdd1bfff640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdd200664d0 0x7fdd20068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:03.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.502+0000 7fdd1bfff640 1 -- 172.21.15.125:0/2238105948 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7fdd240847f0 con 0x7fdd3406f7a0 2023-10-30T13:06:03.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.502+0000 7fdd32ffd640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdd200664d0 0x7fdd20068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:03.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.502+0000 7fdd32ffd640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdd200664d0 0x7fdd20068990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdd3406c5a0 tx=0x7fdd28008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:03.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.506+0000 7fdd1bfff640 1 -- 172.21.15.125:0/2238105948 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdd24051590 con 0x7fdd3406f7a0 2023-10-30T13:06:03.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.714+0000 7fdd39a9e640 1 -- 172.21.15.125:0/2238105948 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdd3414db30 con 0x7fdd3406f7a0 2023-10-30T13:06:03.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.714+0000 7fdd1bfff640 1 -- 172.21.15.125:0/2238105948 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+5723 (secure 0 0 0) 0x7fdd24018090 con 0x7fdd3406f7a0 2023-10-30T13:06:03.718 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:03.718 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":162,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:05:55.585798+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8178},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:03.719 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 162 2023-10-30T13:06:03.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.718+0000 7fdd39a9e640 1 -- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdd200664d0 msgr2=0x7fdd20068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:03.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.718+0000 7fdd39a9e640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdd200664d0 0x7fdd20068990 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fdd3406c5a0 tx=0x7fdd28008040 comp rx=0 tx=0).stop 2023-10-30T13:06:03.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.718+0000 7fdd39a9e640 1 -- 172.21.15.125:0/2238105948 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd3406f7a0 msgr2=0x7fdd340ad8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:03.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.718+0000 7fdd39a9e640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd3406f7a0 0x7fdd340ad8d0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400d800 tx=0x7fdd2400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:06:03.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.718+0000 7fdd39a9e640 1 -- 172.21.15.125:0/2238105948 shutdown_connections 2023-10-30T13:06:03.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.718+0000 7fdd39a9e640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdd200664d0 0x7fdd20068990 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:03.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.718+0000 7fdd39a9e640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd340c6bd0 0x7fdd341496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:03.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.718+0000 7fdd39a9e640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd34071110 0x7fdd341f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:03.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.718+0000 7fdd39a9e640 1 --2- 172.21.15.125:0/2238105948 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd3406f7a0 0x7fdd340ad8d0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:03.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.718+0000 7fdd39a9e640 1 -- 172.21.15.125:0/2238105948 >> 172.21.15.125:0/2238105948 conn(0x7fdd340c7910 msgr2=0x7fdd340c3940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:03.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.718+0000 7fdd39a9e640 1 -- 172.21.15.125:0/2238105948 shutdown_connections 2023-10-30T13:06:03.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:03.718+0000 7fdd39a9e640 1 -- 172.21.15.125:0/2238105948 wait complete. 2023-10-30T13:06:03.741 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:06:05.742 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:05.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.886+0000 7f0d3ed1d640 1 Processor -- start 2023-10-30T13:06:05.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.886+0000 7f0d3ed1d640 1 -- start start 2023-10-30T13:06:05.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.886+0000 7f0d3ed1d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0d381538a0 0x7f0d3815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:05.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.890+0000 7f0d3ed1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0d380cf160 0x7f0d380cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:05.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.890+0000 7f0d3ed1d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d380cb800 con 0x7f0d380ca0e0 2023-10-30T13:06:05.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.890+0000 7f0d3ed1d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d3815dc30 con 0x7f0d381538a0 2023-10-30T13:06:05.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.890+0000 7f0d3ed1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d380cb330 con 0x7f0d380cf160 2023-10-30T13:06:05.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.890+0000 7f0d3d2ba640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0d380cf160 0x7f0d380cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:05.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.890+0000 7f0d3cab9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0d381538a0 0x7f0d3815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:05.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.890+0000 7f0d3d2ba640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0d380cf160 0x7f0d380cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:34838/0 (socket says 172.21.15.125:34838) 2023-10-30T13:06:05.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.890+0000 7f0d3d2ba640 1 -- 172.21.15.125:0/1062883214 learned_addr learned my addr 172.21.15.125:0/1062883214 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:05.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.890+0000 7f0d3d2ba640 1 -- 172.21.15.125:0/1062883214 >> v2:172.21.15.136:3300/0 conn(0x7f0d381538a0 msgr2=0x7f0d3815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:05.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.890+0000 7f0d3d2ba640 1 --2- 172.21.15.125:0/1062883214 >> v2:172.21.15.136:3300/0 conn(0x7f0d381538a0 0x7f0d3815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:05.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.890+0000 7f0d3d2ba640 1 -- 172.21.15.125:0/1062883214 >> v1:172.21.15.136:6789/0 conn(0x7f0d380ca0e0 legacy=0x7f0d3815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:05.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.890+0000 7f0d3d2ba640 1 -- 172.21.15.125:0/1062883214 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d380cad70 con 0x7f0d380cf160 2023-10-30T13:06:05.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.890+0000 7f0d3cab9640 1 --2- 172.21.15.125:0/1062883214 >> v2:172.21.15.136:3300/0 conn(0x7f0d381538a0 0x7f0d3815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:06:05.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d3d2ba640 1 --2- 172.21.15.125:0/1062883214 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0d380cf160 0x7f0d380cf590 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0d30009e50 tx=0x7f0d3001f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55b9faf6f3304eee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:05.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d2f7fe640 1 -- 172.21.15.125:0/1062883214 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d300af400 con 0x7f0d380cf160 2023-10-30T13:06:05.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d2f7fe640 1 -- 172.21.15.125:0/1062883214 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d300ab050 con 0x7f0d380cf160 2023-10-30T13:06:05.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d2f7fe640 1 -- 172.21.15.125:0/1062883214 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d300af780 con 0x7f0d380cf160 2023-10-30T13:06:05.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1062883214 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0d380cf160 msgr2=0x7f0d380cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:05.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d3ed1d640 1 --2- 172.21.15.125:0/1062883214 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0d380cf160 0x7f0d380cf590 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0d30009e50 tx=0x7f0d3001f1f0 comp rx=0 tx=0).stop 2023-10-30T13:06:05.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1062883214 shutdown_connections 2023-10-30T13:06:05.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d3ed1d640 1 --2- 172.21.15.125:0/1062883214 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0d380cf160 0x7f0d380cf590 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:05.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d3ed1d640 1 --2- 172.21.15.125:0/1062883214 >> v2:172.21.15.136:3300/0 conn(0x7f0d381538a0 0x7f0d3815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:05.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1062883214 >> 172.21.15.125:0/1062883214 conn(0x7f0d3806bfe0 msgr2=0x7f0d3806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:05.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1062883214 shutdown_connections 2023-10-30T13:06:05.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1062883214 wait complete. 2023-10-30T13:06:05.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d3ed1d640 1 Processor -- start 2023-10-30T13:06:05.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d3ed1d640 1 -- start start 2023-10-30T13:06:05.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.894+0000 7f0d3ed1d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d380ca0e0 0x7f0d38157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:05.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3ed1d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d380cf160 0x7f0d380b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:05.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3ed1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0d381538a0 0x7f0d380b3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:05.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3ed1d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d381589f0 con 0x7f0d380ca0e0 2023-10-30T13:06:05.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3ed1d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d380b5050 con 0x7f0d380cf160 2023-10-30T13:06:05.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3ed1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d381ef5c0 con 0x7f0d381538a0 2023-10-30T13:06:05.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3cab9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d380ca0e0 0x7f0d38157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:05.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3cab9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d380ca0e0 0x7f0d38157780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45468/0 (socket says 172.21.15.125:45468) 2023-10-30T13:06:05.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3cab9640 1 -- 172.21.15.125:0/1271602041 learned_addr learned my addr 172.21.15.125:0/1271602041 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:05.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d2ffff640 1 --2- 172.21.15.125:0/1271602041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d380cf160 0x7f0d380b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:05.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3d2ba640 1 --2- 172.21.15.125:0/1271602041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0d381538a0 0x7f0d380b3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:05.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3cab9640 1 -- 172.21.15.125:0/1271602041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0d381538a0 msgr2=0x7f0d380b3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:05.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3cab9640 1 --2- 172.21.15.125:0/1271602041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0d381538a0 0x7f0d380b3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:05.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3cab9640 1 -- 172.21.15.125:0/1271602041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d380cf160 msgr2=0x7f0d380b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:05.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3cab9640 1 --2- 172.21.15.125:0/1271602041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d380cf160 0x7f0d380b31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:05.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3cab9640 1 -- 172.21.15.125:0/1271602041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d34009420 con 0x7f0d380ca0e0 2023-10-30T13:06:05.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3cab9640 1 --2- 172.21.15.125:0/1271602041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d380ca0e0 0x7f0d38157780 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400e980 tx=0x7f0d3400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:05.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d2d7fa640 1 -- 172.21.15.125:0/1271602041 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3400ca00 con 0x7f0d380ca0e0 2023-10-30T13:06:05.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d2d7fa640 1 -- 172.21.15.125:0/1271602041 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d3400cb60 con 0x7f0d380ca0e0 2023-10-30T13:06:05.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d2d7fa640 1 -- 172.21.15.125:0/1271602041 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d34013470 con 0x7f0d380ca0e0 2023-10-30T13:06:05.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1271602041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d30009b30 con 0x7f0d380ca0e0 2023-10-30T13:06:05.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1271602041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d381efb80 con 0x7f0d380ca0e0 2023-10-30T13:06:05.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1271602041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d380b2a50 con 0x7f0d380ca0e0 2023-10-30T13:06:05.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.898+0000 7f0d2d7fa640 1 -- 172.21.15.125:0/1271602041 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0d340135d0 con 0x7f0d380ca0e0 2023-10-30T13:06:05.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.902+0000 7f0d2d7fa640 1 --2- 172.21.15.125:0/1271602041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0d240664a0 0x7f0d24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:05.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.902+0000 7f0d2d7fa640 1 -- 172.21.15.125:0/1271602041 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f0d34084790 con 0x7f0d380ca0e0 2023-10-30T13:06:05.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.902+0000 7f0d2ffff640 1 --2- 172.21.15.125:0/1271602041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0d240664a0 0x7f0d24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:05.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.902+0000 7f0d2ffff640 1 --2- 172.21.15.125:0/1271602041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0d240664a0 0x7f0d24068960 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f0d38149850 tx=0x7f0d28009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:05.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:05.906+0000 7f0d2d7fa640 1 -- 172.21.15.125:0/1271602041 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0d34051540 con 0x7f0d380ca0e0 2023-10-30T13:06:06.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:06.086+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1271602041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d3804e170 con 0x7f0d380ca0e0 2023-10-30T13:06:06.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:06.086+0000 7f0d2d7fa640 1 -- 172.21.15.125:0/1271602041 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+5727 (secure 0 0 0) 0x7f0d34054b60 con 0x7f0d380ca0e0 2023-10-30T13:06:06.089 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:06.089 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:03.848705+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8178},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:06.090 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 163 2023-10-30T13:06:06.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:06.086+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1271602041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0d240664a0 msgr2=0x7f0d24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:06.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:06.086+0000 7f0d3ed1d640 1 --2- 172.21.15.125:0/1271602041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0d240664a0 0x7f0d24068960 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f0d38149850 tx=0x7f0d28009040 comp rx=0 tx=0).stop 2023-10-30T13:06:06.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:06.086+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1271602041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d380ca0e0 msgr2=0x7f0d38157780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:06.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:06.086+0000 7f0d3ed1d640 1 --2- 172.21.15.125:0/1271602041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d380ca0e0 0x7f0d38157780 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400e980 tx=0x7f0d3400ee40 comp rx=0 tx=0).stop 2023-10-30T13:06:06.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:06.086+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1271602041 shutdown_connections 2023-10-30T13:06:06.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:06.086+0000 7f0d3ed1d640 1 --2- 172.21.15.125:0/1271602041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0d240664a0 0x7f0d24068960 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:06.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:06.086+0000 7f0d3ed1d640 1 --2- 172.21.15.125:0/1271602041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0d381538a0 0x7f0d380b3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:06.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:06.086+0000 7f0d3ed1d640 1 --2- 172.21.15.125:0/1271602041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0d380cf160 0x7f0d380b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:06.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:06.086+0000 7f0d3ed1d640 1 --2- 172.21.15.125:0/1271602041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0d380ca0e0 0x7f0d38157780 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:06.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:06.086+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1271602041 >> 172.21.15.125:0/1271602041 conn(0x7f0d3806bfe0 msgr2=0x7f0d381525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:06.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:06.086+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1271602041 shutdown_connections 2023-10-30T13:06:06.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:06.086+0000 7f0d3ed1d640 1 -- 172.21.15.125:0/1271602041 wait complete. 2023-10-30T13:06:06.109 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:06:08.110 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:08.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff02029e640 1 Processor -- start 2023-10-30T13:06:08.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff02029e640 1 -- start start 2023-10-30T13:06:08.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff02029e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff0180cf160 0x7ff0180cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:08.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff02029e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0180cb800 con 0x7ff01815d6f0 2023-10-30T13:06:08.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff02029e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0180cb330 con 0x7ff0181538a0 2023-10-30T13:06:08.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff02029e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0180cad70 con 0x7ff0180cf160 2023-10-30T13:06:08.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01d839640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff01815d6f0 0x7ff01815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:53254/0 (socket says 172.21.15.125:53254) 2023-10-30T13:06:08.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01d839640 1 -- 172.21.15.125:0/2171153040 learned_addr learned my addr 172.21.15.125:0/2171153040 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:06:08.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01e83b640 1 --2- 172.21.15.125:0/2171153040 >> v2:172.21.15.136:3300/0 conn(0x7ff0180cf160 0x7ff0180cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:08.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222186040 0 0) 0x7ff0180cb800 con 0x7ff01815d6f0 2023-10-30T13:06:08.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff004003530 con 0x7ff01815d6f0 2023-10-30T13:06:08.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863326039 0 0) 0x7ff0180cb330 con 0x7ff0181538a0 2023-10-30T13:06:08.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0180cb800 con 0x7ff0181538a0 2023-10-30T13:06:08.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2460252383 0 0) 0x7ff004003530 con 0x7ff01815d6f0 2023-10-30T13:06:08.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 >> v2:172.21.15.136:3300/0 conn(0x7ff0180cf160 msgr2=0x7ff0180cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:08.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01d038640 1 --2- 172.21.15.125:0/2171153040 >> v2:172.21.15.136:3300/0 conn(0x7ff0180cf160 0x7ff0180cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:08.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 >> v1:172.21.15.136:6789/0 conn(0x7ff0181538a0 legacy=0x7ff0180ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:08.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0180cb050 con 0x7ff01815d6f0 2023-10-30T13:06:08.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7ff008001680 con 0x7ff01815d6f0 2023-10-30T13:06:08.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 >> v1:172.21.15.125:6789/0 conn(0x7ff01815d6f0 legacy=0x7ff01815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:08.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.254+0000 7ff01d038640 1 --2- 172.21.15.125:0/2171153040 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff004003b10 0x7ff004003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:08.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d038640 1 --2- 172.21.15.125:0/2171153040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0040047d0 0x7ff004004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:08.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d038640 1 --2- 172.21.15.125:0/2171153040 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff004005190 0x7ff00400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:08.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0180cad70 con 0x7ff0040047d0 2023-10-30T13:06:08.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00400bf80 con 0x7ff004005190 2023-10-30T13:06:08.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00400c140 con 0x7ff004003b10 2023-10-30T13:06:08.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d839640 1 --2- 172.21.15.125:0/2171153040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0040047d0 0x7ff004004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:08.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01e83b640 1 --2- 172.21.15.125:0/2171153040 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff004005190 0x7ff00400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:08.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01e03a640 1 --2- 172.21.15.125:0/2171153040 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff004003b10 0x7ff004003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:08.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d839640 1 -- 172.21.15.125:0/2171153040 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff004003b10 msgr2=0x7ff004003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:08.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d839640 1 --2- 172.21.15.125:0/2171153040 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff004003b10 0x7ff004003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:08.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d839640 1 -- 172.21.15.125:0/2171153040 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff004005190 msgr2=0x7ff00400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:08.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d839640 1 --2- 172.21.15.125:0/2171153040 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff004005190 0x7ff00400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:08.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d839640 1 -- 172.21.15.125:0/2171153040 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff004003530 con 0x7ff0040047d0 2023-10-30T13:06:08.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01e83b640 1 -- 172.21.15.125:0/2171153040 reap_dead start 2023-10-30T13:06:08.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01e03a640 1 --2- 172.21.15.125:0/2171153040 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff004003b10 0x7ff004003f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:08.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d839640 1 --2- 172.21.15.125:0/2171153040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0040047d0 0x7ff004004c50 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7ff008001ec0 tx=0x7ff00801f290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce3f51d60554f17 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:08.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0080a8cb0 con 0x7ff0040047d0 2023-10-30T13:06:08.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0080a8e30 con 0x7ff0040047d0 2023-10-30T13:06:08.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.258+0000 7ff01d038640 1 -- 172.21.15.125:0/2171153040 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff008005b20 con 0x7ff0040047d0 2023-10-30T13:06:08.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 -- 172.21.15.125:0/2171153040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0040047d0 msgr2=0x7ff004004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:08.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 --2- 172.21.15.125:0/2171153040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0040047d0 0x7ff004004c50 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7ff008001ec0 tx=0x7ff00801f290 comp rx=0 tx=0).stop 2023-10-30T13:06:08.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 -- 172.21.15.125:0/2171153040 shutdown_connections 2023-10-30T13:06:08.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 --2- 172.21.15.125:0/2171153040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0040047d0 0x7ff004004c50 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:08.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 -- 172.21.15.125:0/2171153040 >> 172.21.15.125:0/2171153040 conn(0x7ff01806bfe0 msgr2=0x7ff01806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:08.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 -- 172.21.15.125:0/2171153040 shutdown_connections 2023-10-30T13:06:08.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 -- 172.21.15.125:0/2171153040 wait complete. 2023-10-30T13:06:08.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 Processor -- start 2023-10-30T13:06:08.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 -- start start 2023-10-30T13:06:08.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0040047d0 0x7ff0180b4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:08.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0180b3060 0x7ff0180b4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:08.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff018157720 0x7ff018157ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:08.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff018158860 con 0x7ff0040047d0 2023-10-30T13:06:08.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0180b42f0 con 0x7ff0180b3060 2023-10-30T13:06:08.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff02029e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0180b39b0 con 0x7ff018157720 2023-10-30T13:06:08.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff01e03a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0040047d0 0x7ff0180b4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:08.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff01e03a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0040047d0 0x7ff0180b4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45482/0 (socket says 172.21.15.125:45482) 2023-10-30T13:06:08.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff01e03a640 1 -- 172.21.15.125:0/1119979776 learned_addr learned my addr 172.21.15.125:0/1119979776 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:08.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff01e83b640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff018157720 0x7ff018157ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:08.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff01d839640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0180b3060 0x7ff0180b4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:08.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff01e03a640 1 -- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff018157720 msgr2=0x7ff018157ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:08.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.262+0000 7ff01e03a640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff018157720 0x7ff018157ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:08.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff01e03a640 1 -- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0180b3060 msgr2=0x7ff0180b4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:08.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff01e03a640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0180b3060 0x7ff0180b4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:08.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff01e03a640 1 -- 172.21.15.125:0/1119979776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0181ef6f0 con 0x7ff0040047d0 2023-10-30T13:06:08.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff01d839640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0180b3060 0x7ff0180b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:06:08.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff01e03a640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0040047d0 0x7ff0180b4780 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7ff01400e500 tx=0x7ff01400e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:08.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff00effd640 1 -- 172.21.15.125:0/1119979776 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff014017070 con 0x7ff0040047d0 2023-10-30T13:06:08.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff00effd640 1 -- 172.21.15.125:0/1119979776 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0140052e0 con 0x7ff0040047d0 2023-10-30T13:06:08.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff00effd640 1 -- 172.21.15.125:0/1119979776 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff014005450 con 0x7ff0040047d0 2023-10-30T13:06:08.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff02029e640 1 -- 172.21.15.125:0/1119979776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0180b3c50 con 0x7ff0040047d0 2023-10-30T13:06:08.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff02029e640 1 -- 172.21.15.125:0/1119979776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0181ef9d0 con 0x7ff0040047d0 2023-10-30T13:06:08.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff02029e640 1 -- 172.21.15.125:0/1119979776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0180b4100 con 0x7ff0040047d0 2023-10-30T13:06:08.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff00effd640 1 -- 172.21.15.125:0/1119979776 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff014007020 con 0x7ff0040047d0 2023-10-30T13:06:08.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff00effd640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feffc0664d0 0x7feffc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:08.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff00effd640 1 -- 172.21.15.125:0/1119979776 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7ff014083cd0 con 0x7ff0040047d0 2023-10-30T13:06:08.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.266+0000 7ff01d839640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feffc0664d0 0x7feffc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:08.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.270+0000 7ff01d839640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feffc0664d0 0x7feffc068990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff008001ec0 tx=0x7ff008001a00 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:08.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.270+0000 7ff00effd640 1 -- 172.21.15.125:0/1119979776 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff014054470 con 0x7ff0040047d0 2023-10-30T13:06:08.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.454+0000 7ff02029e640 1 -- 172.21.15.125:0/1119979776 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff01804e170 con 0x7ff0040047d0 2023-10-30T13:06:08.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.454+0000 7ff00effd640 1 -- 172.21.15.125:0/1119979776 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+5727 (secure 0 0 0) 0x7ff014072070 con 0x7ff0040047d0 2023-10-30T13:06:08.457 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:08.457 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:03.848705+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8178},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:08.457 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 163 2023-10-30T13:06:08.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.454+0000 7ff02029e640 1 -- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feffc0664d0 msgr2=0x7feffc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:08.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.454+0000 7ff02029e640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feffc0664d0 0x7feffc068990 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7ff008001ec0 tx=0x7ff008001a00 comp rx=0 tx=0).stop 2023-10-30T13:06:08.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.454+0000 7ff02029e640 1 -- 172.21.15.125:0/1119979776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0040047d0 msgr2=0x7ff0180b4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:08.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.454+0000 7ff02029e640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0040047d0 0x7ff0180b4780 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7ff01400e500 tx=0x7ff01400e9c0 comp rx=0 tx=0).stop 2023-10-30T13:06:08.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.454+0000 7ff02029e640 1 -- 172.21.15.125:0/1119979776 shutdown_connections 2023-10-30T13:06:08.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.454+0000 7ff02029e640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feffc0664d0 0x7feffc068990 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:08.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.454+0000 7ff02029e640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff018157720 0x7ff018157ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:08.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.454+0000 7ff02029e640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff0180b3060 0x7ff0180b4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:08.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.454+0000 7ff02029e640 1 --2- 172.21.15.125:0/1119979776 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff0040047d0 0x7ff0180b4780 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:08.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.454+0000 7ff02029e640 1 -- 172.21.15.125:0/1119979776 >> 172.21.15.125:0/1119979776 conn(0x7ff01806bfe0 msgr2=0x7ff0180c49f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:08.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.454+0000 7ff02029e640 1 -- 172.21.15.125:0/1119979776 shutdown_connections 2023-10-30T13:06:08.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:08.454+0000 7ff02029e640 1 -- 172.21.15.125:0/1119979776 wait complete. 2023-10-30T13:06:08.478 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 163, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 163, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8178": { "addr": "172.21.15.125:6836/3711687505", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3711687505, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3711687505, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8178, "incarnation": 162, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:reconnect", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 1, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:06:03.848705+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 1, 2 ], "tableserver": 0, "up": { "mds_0": 8178 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:06:10.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:10.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a733e4640 1 Processor -- start 2023-10-30T13:06:10.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a733e4640 1 -- start start 2023-10-30T13:06:10.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a733e4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8a6c1538f0 0x7f8a6c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:10.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a733e4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8a6c15cd50 0x7f8a6c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:10.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a733e4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8a6c0cefa0 0x7f8a6c0cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:10.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a733e4640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a6c15d300 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a733e4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a6c15d5e0 con 0x7f8a6c1538f0 2023-10-30T13:06:10.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a733e4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c15d8c0 con 0x7f8a6c15cd50 2023-10-30T13:06:10.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a72be3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8a6c0cefa0 0x7f8a6c0cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:10.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a72be3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8a6c0cefa0 0x7f8a6c0cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45492/0 (socket says 172.21.15.125:45492) 2023-10-30T13:06:10.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a72be3640 1 -- 172.21.15.125:0/3592826327 learned_addr learned my addr 172.21.15.125:0/3592826327 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:10.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a71be1640 1 --2- 172.21.15.125:0/3592826327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8a6c15cd50 0x7f8a6c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:10.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a723e2640 1 --2- 172.21.15.125:0/3592826327 >> v2:172.21.15.136:3300/0 conn(0x7f8a6c1538f0 0x7f8a6c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:10.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a72be3640 1 -- 172.21.15.125:0/3592826327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8a6c15cd50 msgr2=0x7f8a6c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:10.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a72be3640 1 --2- 172.21.15.125:0/3592826327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8a6c15cd50 0x7f8a6c0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:10.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a72be3640 1 -- 172.21.15.125:0/3592826327 >> v2:172.21.15.136:3300/0 conn(0x7f8a6c1538f0 msgr2=0x7f8a6c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:10.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a72be3640 1 --2- 172.21.15.125:0/3592826327 >> v2:172.21.15.136:3300/0 conn(0x7f8a6c1538f0 0x7f8a6c159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:10.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a72be3640 1 -- 172.21.15.125:0/3592826327 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a6c15dba0 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.626+0000 7f8a723e2640 1 --2- 172.21.15.125:0/3592826327 >> v2:172.21.15.136:3300/0 conn(0x7f8a6c1538f0 0x7f8a6c159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:10.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.630+0000 7f8a72be3640 1 --2- 172.21.15.125:0/3592826327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a6c0cefa0 0x7f8a6c0cf3f0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0c9cb0 tx=0x7f8a6401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f9909e0324f181 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:10.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.630+0000 7f8a713e0640 1 -- 172.21.15.125:0/3592826327 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a640af400 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.630+0000 7f8a713e0640 1 -- 172.21.15.125:0/3592826327 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a640ab050 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.630+0000 7f8a713e0640 1 -- 172.21.15.125:0/3592826327 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a640af780 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.630+0000 7f8a733e4640 1 -- 172.21.15.125:0/3592826327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a6c0cefa0 msgr2=0x7f8a6c0cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:10.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.630+0000 7f8a733e4640 1 --2- 172.21.15.125:0/3592826327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a6c0cefa0 0x7f8a6c0cf3f0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0c9cb0 tx=0x7f8a6401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:06:10.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.630+0000 7f8a733e4640 1 -- 172.21.15.125:0/3592826327 shutdown_connections 2023-10-30T13:06:10.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.630+0000 7f8a733e4640 1 --2- 172.21.15.125:0/3592826327 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a6c0cefa0 0x7f8a6c0cf3f0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:10.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.630+0000 7f8a733e4640 1 --2- 172.21.15.125:0/3592826327 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8a6c15cd50 0x7f8a6c0c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:10.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.630+0000 7f8a733e4640 1 --2- 172.21.15.125:0/3592826327 >> v2:172.21.15.136:3300/0 conn(0x7f8a6c1538f0 0x7f8a6c159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:10.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.630+0000 7f8a733e4640 1 -- 172.21.15.125:0/3592826327 >> 172.21.15.125:0/3592826327 conn(0x7f8a6c06a640 msgr2=0x7f8a6c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:10.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.630+0000 7f8a733e4640 1 -- 172.21.15.125:0/3592826327 shutdown_connections 2023-10-30T13:06:10.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.630+0000 7f8a733e4640 1 -- 172.21.15.125:0/3592826327 wait complete. 2023-10-30T13:06:10.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a733e4640 1 Processor -- start 2023-10-30T13:06:10.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a733e4640 1 -- start start 2023-10-30T13:06:10.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a733e4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a6c0cefa0 0x7f8a6c1eef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:10.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a733e4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a6c1538f0 0x7f8a6c152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:10.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a733e4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8a6c15cd50 0x7f8a6c15a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:10.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a733e4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c153170 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a733e4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c1527a0 con 0x7f8a6c1538f0 2023-10-30T13:06:10.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a733e4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c1594e0 con 0x7f8a6c15cd50 2023-10-30T13:06:10.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a723e2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a6c0cefa0 0x7f8a6c1eef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:10.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a723e2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a6c0cefa0 0x7f8a6c1eef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45506/0 (socket says 172.21.15.125:45506) 2023-10-30T13:06:10.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a723e2640 1 -- 172.21.15.125:0/3168317980 learned_addr learned my addr 172.21.15.125:0/3168317980 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:10.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a71be1640 1 --2- 172.21.15.125:0/3168317980 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a6c1538f0 0x7f8a6c152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:10.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a72be3640 1 --2- 172.21.15.125:0/3168317980 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8a6c15cd50 0x7f8a6c15a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:10.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a723e2640 1 -- 172.21.15.125:0/3168317980 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8a6c15cd50 msgr2=0x7f8a6c15a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:10.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a723e2640 1 --2- 172.21.15.125:0/3168317980 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8a6c15cd50 0x7f8a6c15a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:10.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a723e2640 1 -- 172.21.15.125:0/3168317980 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a6c1538f0 msgr2=0x7f8a6c152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:10.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a723e2640 1 --2- 172.21.15.125:0/3168317980 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a6c1538f0 0x7f8a6c152c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:10.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a723e2640 1 -- 172.21.15.125:0/3168317980 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a64009b30 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a723e2640 1 --2- 172.21.15.125:0/3168317980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a6c0cefa0 0x7f8a6c1eef60 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800e840 tx=0x7f8a6800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:10.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a62ffd640 1 -- 172.21.15.125:0/3168317980 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68004030 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a62ffd640 1 -- 172.21.15.125:0/3168317980 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a68004190 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a62ffd640 1 -- 172.21.15.125:0/3168317980 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a68004520 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a733e4640 1 -- 172.21.15.125:0/3168317980 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a6c15ae40 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a733e4640 1 -- 172.21.15.125:0/3168317980 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6c144b80 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.634+0000 7f8a733e4640 1 -- 172.21.15.125:0/3168317980 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6c04e170 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.638+0000 7f8a62ffd640 1 -- 172.21.15.125:0/3168317980 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8a6800c6d0 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.638+0000 7f8a62ffd640 1 --2- 172.21.15.125:0/3168317980 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8a500664d0 0x7f8a50068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:10.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.638+0000 7f8a62ffd640 1 -- 172.21.15.125:0/3168317980 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f8a68084500 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.638+0000 7f8a71be1640 1 --2- 172.21.15.125:0/3168317980 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8a500664d0 0x7f8a50068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:10.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.638+0000 7f8a71be1640 1 --2- 172.21.15.125:0/3168317980 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8a500664d0 0x7f8a50068990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c159400 tx=0x7f8a5c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:10.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.642+0000 7f8a62ffd640 1 -- 172.21.15.125:0/3168317980 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8a680511f0 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.822+0000 7f8a733e4640 1 -- 172.21.15.125:0/3168317980 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a6c146220 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.822+0000 7f8a62ffd640 1 -- 172.21.15.125:0/3168317980 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f8a68054810 con 0x7f8a6c0cefa0 2023-10-30T13:06:10.827 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:10.827 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:09.611035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8178},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 164 2023-10-30T13:06:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.826+0000 7f8a733e4640 1 -- 172.21.15.125:0/3168317980 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8a500664d0 msgr2=0x7f8a50068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.826+0000 7f8a733e4640 1 --2- 172.21.15.125:0/3168317980 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8a500664d0 0x7f8a50068990 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c159400 tx=0x7f8a5c008040 comp rx=0 tx=0).stop 2023-10-30T13:06:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.826+0000 7f8a733e4640 1 -- 172.21.15.125:0/3168317980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a6c0cefa0 msgr2=0x7f8a6c1eef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:10.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.826+0000 7f8a733e4640 1 --2- 172.21.15.125:0/3168317980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a6c0cefa0 0x7f8a6c1eef60 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f8a6800e840 tx=0x7f8a6800ed00 comp rx=0 tx=0).stop 2023-10-30T13:06:10.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.826+0000 7f8a733e4640 1 -- 172.21.15.125:0/3168317980 shutdown_connections 2023-10-30T13:06:10.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.826+0000 7f8a733e4640 1 --2- 172.21.15.125:0/3168317980 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8a500664d0 0x7f8a50068990 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:10.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.826+0000 7f8a733e4640 1 --2- 172.21.15.125:0/3168317980 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8a6c15cd50 0x7f8a6c15a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:10.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.826+0000 7f8a733e4640 1 --2- 172.21.15.125:0/3168317980 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8a6c1538f0 0x7f8a6c152c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.826+0000 7f8a733e4640 1 --2- 172.21.15.125:0/3168317980 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8a6c0cefa0 0x7f8a6c1eef60 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.826+0000 7f8a733e4640 1 -- 172.21.15.125:0/3168317980 >> 172.21.15.125:0/3168317980 conn(0x7f8a6c06a640 msgr2=0x7f8a6c0c40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.826+0000 7f8a733e4640 1 -- 172.21.15.125:0/3168317980 shutdown_connections 2023-10-30T13:06:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:10.826+0000 7f8a733e4640 1 -- 172.21.15.125:0/3168317980 wait complete. 2023-10-30T13:06:10.848 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:06:12.849 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:12.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.994+0000 7f833d879640 1 Processor -- start 2023-10-30T13:06:12.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.994+0000 7f833d879640 1 -- start start 2023-10-30T13:06:12.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.994+0000 7f833d879640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83381538a0 0x7f833815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:12.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.994+0000 7f833d879640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f83380cf160 0x7f83380cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:13.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.994+0000 7f833d879640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83380cb800 con 0x7f83380ca0e0 2023-10-30T13:06:13.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.994+0000 7f833d879640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833815dc30 con 0x7f83380cf160 2023-10-30T13:06:13.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.994+0000 7f833d879640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380cb330 con 0x7f83381538a0 2023-10-30T13:06:13.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.994+0000 7f832e7fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f83380ca0e0 0x7f833815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48008/0 (socket says 172.21.15.125:48008) 2023-10-30T13:06:13.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.994+0000 7f832e7fc640 1 -- 172.21.15.125:0/880235093 learned_addr learned my addr 172.21.15.125:0/880235093 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:06:13.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.994+0000 7f83377fe640 1 --2- 172.21.15.125:0/880235093 >> v2:172.21.15.136:3300/0 conn(0x7f83380cf160 0x7f83380cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:13.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.994+0000 7f8336ffd640 1 --2- 172.21.15.125:0/880235093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83381538a0 0x7f833815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:13.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.994+0000 7f83367fc640 1 -- 172.21.15.125:0/880235093 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437515168 0 0) 0x7f83380cb800 con 0x7f83380ca0e0 2023-10-30T13:06:13.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.994+0000 7f83367fc640 1 -- 172.21.15.125:0/880235093 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8324003530 con 0x7f83380ca0e0 2023-10-30T13:06:13.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.998+0000 7f83377fe640 1 -- 172.21.15.125:0/880235093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83381538a0 msgr2=0x7f833815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:13.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.998+0000 7f83377fe640 1 --2- 172.21.15.125:0/880235093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83381538a0 0x7f833815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:13.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.998+0000 7f83377fe640 1 -- 172.21.15.125:0/880235093 >> v1:172.21.15.125:6789/0 conn(0x7f83380ca0e0 legacy=0x7f833815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:13.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.998+0000 7f83377fe640 1 -- 172.21.15.125:0/880235093 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83380cad70 con 0x7f83380cf160 2023-10-30T13:06:13.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.998+0000 7f8336ffd640 1 --2- 172.21.15.125:0/880235093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83381538a0 0x7f833815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:06:13.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.998+0000 7f83377fe640 1 --2- 172.21.15.125:0/880235093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f83380cf160 0x7f83380cf590 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8330009d90 tx=0x7f833001f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9df5b39ce3f1aff4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:13.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.998+0000 7f83367fc640 1 -- 172.21.15.125:0/880235093 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83300af600 con 0x7f83380cf160 2023-10-30T13:06:13.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.998+0000 7f83367fc640 1 -- 172.21.15.125:0/880235093 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83300ab050 con 0x7f83380cf160 2023-10-30T13:06:13.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.998+0000 7f83367fc640 1 -- 172.21.15.125:0/880235093 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83300af960 con 0x7f83380cf160 2023-10-30T13:06:13.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.998+0000 7f833d879640 1 -- 172.21.15.125:0/880235093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f83380cf160 msgr2=0x7f83380cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:13.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:12.998+0000 7f833d879640 1 --2- 172.21.15.125:0/880235093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f83380cf160 0x7f83380cf590 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f8330009d90 tx=0x7f833001f130 comp rx=0 tx=0).stop 2023-10-30T13:06:13.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f833d879640 1 -- 172.21.15.125:0/880235093 shutdown_connections 2023-10-30T13:06:13.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f833d879640 1 --2- 172.21.15.125:0/880235093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f83380cf160 0x7f83380cf590 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:13.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f833d879640 1 --2- 172.21.15.125:0/880235093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83381538a0 0x7f833815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:13.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f833d879640 1 -- 172.21.15.125:0/880235093 >> 172.21.15.125:0/880235093 conn(0x7f833806bfe0 msgr2=0x7f833806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:13.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f833d879640 1 -- 172.21.15.125:0/880235093 shutdown_connections 2023-10-30T13:06:13.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f833d879640 1 -- 172.21.15.125:0/880235093 wait complete. 2023-10-30T13:06:13.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f833d879640 1 Processor -- start 2023-10-30T13:06:13.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f833d879640 1 -- start start 2023-10-30T13:06:13.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f833d879640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83380ca0e0 0x7f83381524d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:13.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f833d879640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83380cf160 0x7f8338147f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:13.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f833d879640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f83381538a0 0x7f83381ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:13.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f833d879640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380c64d0 con 0x7f83380cf160 2023-10-30T13:06:13.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f833d879640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83380c67b0 con 0x7f83381538a0 2023-10-30T13:06:13.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f833d879640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380c6a90 con 0x7f83380ca0e0 2023-10-30T13:06:13.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f832e7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83380cf160 0x7f8338147f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:13.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f832e7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83380cf160 0x7f8338147f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35734/0 (socket says 172.21.15.125:35734) 2023-10-30T13:06:13.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f832e7fc640 1 -- 172.21.15.125:0/1689691838 learned_addr learned my addr 172.21.15.125:0/1689691838 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:13.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f83377fe640 1 --2- 172.21.15.125:0/1689691838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f83381538a0 0x7f83381ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:13.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f8336ffd640 1 --2- 172.21.15.125:0/1689691838 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83380ca0e0 0x7f83381524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:13.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f832e7fc640 1 -- 172.21.15.125:0/1689691838 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83380ca0e0 msgr2=0x7f83381524d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:13.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f832e7fc640 1 --2- 172.21.15.125:0/1689691838 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83380ca0e0 0x7f83381524d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:13.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f832e7fc640 1 -- 172.21.15.125:0/1689691838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f83381538a0 msgr2=0x7f83381ef9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:13.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f832e7fc640 1 --2- 172.21.15.125:0/1689691838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f83381538a0 0x7f83381ef9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:13.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f832e7fc640 1 -- 172.21.15.125:0/1689691838 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8328009420 con 0x7f83380cf160 2023-10-30T13:06:13.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.002+0000 7f832e7fc640 1 --2- 172.21.15.125:0/1689691838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83380cf160 0x7f8338147f60 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f832000c450 tx=0x7f832000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:13.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.006+0000 7f8334ff9640 1 -- 172.21.15.125:0/1689691838 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8320005cf0 con 0x7f83380cf160 2023-10-30T13:06:13.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.006+0000 7f8334ff9640 1 -- 172.21.15.125:0/1689691838 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8320005e50 con 0x7f83380cf160 2023-10-30T13:06:13.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.006+0000 7f833d879640 1 -- 172.21.15.125:0/1689691838 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8330009a70 con 0x7f83380cf160 2023-10-30T13:06:13.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.006+0000 7f8334ff9640 1 -- 172.21.15.125:0/1689691838 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8320007290 con 0x7f83380cf160 2023-10-30T13:06:13.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.006+0000 7f833d879640 1 -- 172.21.15.125:0/1689691838 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83380c7050 con 0x7f83380cf160 2023-10-30T13:06:13.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.006+0000 7f833d879640 1 -- 172.21.15.125:0/1689691838 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83380b2a50 con 0x7f83380cf160 2023-10-30T13:06:13.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.006+0000 7f8334ff9640 1 -- 172.21.15.125:0/1689691838 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f83200073f0 con 0x7f83380cf160 2023-10-30T13:06:13.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.006+0000 7f8334ff9640 1 --2- 172.21.15.125:0/1689691838 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83180664d0 0x7f8318068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:13.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.006+0000 7f8334ff9640 1 -- 172.21.15.125:0/1689691838 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f8320083a80 con 0x7f83380cf160 2023-10-30T13:06:13.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.006+0000 7f8336ffd640 1 --2- 172.21.15.125:0/1689691838 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83180664d0 0x7f8318068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:13.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.010+0000 7f8336ffd640 1 --2- 172.21.15.125:0/1689691838 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83180664d0 0x7f8318068990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f8328009c50 tx=0x7f8328009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:13.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.010+0000 7f8334ff9640 1 -- 172.21.15.125:0/1689691838 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f83200508c0 con 0x7f83380cf160 2023-10-30T13:06:13.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.194+0000 7f833d879640 1 -- 172.21.15.125:0/1689691838 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f833804e170 con 0x7f83380cf160 2023-10-30T13:06:13.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.194+0000 7f8334ff9640 1 -- 172.21.15.125:0/1689691838 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f8320017020 con 0x7f83380cf160 2023-10-30T13:06:13.199 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:13.199 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:09.611035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8178},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:13.200 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 164 2023-10-30T13:06:13.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.198+0000 7f833d879640 1 -- 172.21.15.125:0/1689691838 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83180664d0 msgr2=0x7f8318068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:13.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.198+0000 7f833d879640 1 --2- 172.21.15.125:0/1689691838 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83180664d0 0x7f8318068990 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f8328009c50 tx=0x7f8328009190 comp rx=0 tx=0).stop 2023-10-30T13:06:13.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.198+0000 7f833d879640 1 -- 172.21.15.125:0/1689691838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83380cf160 msgr2=0x7f8338147f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:13.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.198+0000 7f833d879640 1 --2- 172.21.15.125:0/1689691838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83380cf160 0x7f8338147f60 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f832000c450 tx=0x7f832000c910 comp rx=0 tx=0).stop 2023-10-30T13:06:13.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.198+0000 7f833d879640 1 -- 172.21.15.125:0/1689691838 shutdown_connections 2023-10-30T13:06:13.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.198+0000 7f833d879640 1 --2- 172.21.15.125:0/1689691838 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83180664d0 0x7f8318068990 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:13.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.198+0000 7f833d879640 1 --2- 172.21.15.125:0/1689691838 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f83381538a0 0x7f83381ef9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:13.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.198+0000 7f833d879640 1 --2- 172.21.15.125:0/1689691838 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83380cf160 0x7f8338147f60 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:13.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.198+0000 7f833d879640 1 --2- 172.21.15.125:0/1689691838 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83380ca0e0 0x7f83381524d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:13.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.198+0000 7f833d879640 1 -- 172.21.15.125:0/1689691838 >> 172.21.15.125:0/1689691838 conn(0x7f833806bfe0 msgr2=0x7f83380c5520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:13.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.198+0000 7f833d879640 1 -- 172.21.15.125:0/1689691838 shutdown_connections 2023-10-30T13:06:13.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:13.198+0000 7f833d879640 1 -- 172.21.15.125:0/1689691838 wait complete. 2023-10-30T13:06:13.220 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:06:15.221 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:15.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.386+0000 7f605d017640 1 Processor -- start 2023-10-30T13:06:15.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.386+0000 7f605d017640 1 -- start start 2023-10-30T13:06:15.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.386+0000 7f605d017640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f605814fc00 0x7f6058156330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:15.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.386+0000 7f605d017640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6058156da0 0x7f6058157200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:15.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.386+0000 7f605d017640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6058156870 con 0x7f6058069e90 2023-10-30T13:06:15.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.386+0000 7f605d017640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6058072180 con 0x7f6058156da0 2023-10-30T13:06:15.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.386+0000 7f605d017640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6058071cf0 con 0x7f605814fc00 2023-10-30T13:06:15.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.386+0000 7f605659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6058156da0 0x7f6058157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:15.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.390+0000 7f605659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6058156da0 0x7f6058157200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35738/0 (socket says 172.21.15.125:35738) 2023-10-30T13:06:15.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.390+0000 7f605659c640 1 -- 172.21.15.125:0/666144190 learned_addr learned my addr 172.21.15.125:0/666144190 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:15.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.390+0000 7f6056d9d640 1 --2- 172.21.15.125:0/666144190 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f605814fc00 0x7f6058156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:15.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.390+0000 7f6055d9b640 1 -- 172.21.15.125:0/666144190 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065260033 0 0) 0x7f6058156870 con 0x7f6058069e90 2023-10-30T13:06:15.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.390+0000 7f6055d9b640 1 -- 172.21.15.125:0/666144190 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6040003530 con 0x7f6058069e90 2023-10-30T13:06:15.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.390+0000 7f6056d9d640 1 -- 172.21.15.125:0/666144190 >> v2:172.21.15.125:3300/0 conn(0x7f6058156da0 msgr2=0x7f6058157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:15.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.390+0000 7f6056d9d640 1 --2- 172.21.15.125:0/666144190 >> v2:172.21.15.125:3300/0 conn(0x7f6058156da0 0x7f6058157200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:15.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.390+0000 7f6056d9d640 1 -- 172.21.15.125:0/666144190 >> v1:172.21.15.136:6789/0 conn(0x7f6058069e90 legacy=0x7f605806a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:15.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.390+0000 7f6056d9d640 1 -- 172.21.15.125:0/666144190 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6058070b90 con 0x7f605814fc00 2023-10-30T13:06:15.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.398+0000 7f6056d9d640 1 --2- 172.21.15.125:0/666144190 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f605814fc00 0x7f6058156330 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f604c009fd0 tx=0x7f604c01fea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50586c3c78f45d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:15.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.398+0000 7f6055d9b640 1 -- 172.21.15.125:0/666144190 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c0a9cb0 con 0x7f605814fc00 2023-10-30T13:06:15.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.398+0000 7f6055d9b640 1 -- 172.21.15.125:0/666144190 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f604c0a9e10 con 0x7f605814fc00 2023-10-30T13:06:15.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.398+0000 7f6055d9b640 1 -- 172.21.15.125:0/666144190 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f604c0b1660 con 0x7f605814fc00 2023-10-30T13:06:15.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.398+0000 7f605d017640 1 -- 172.21.15.125:0/666144190 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f605814fc00 msgr2=0x7f6058156330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:15.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.398+0000 7f605d017640 1 --2- 172.21.15.125:0/666144190 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f605814fc00 0x7f6058156330 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f604c009fd0 tx=0x7f604c01fea0 comp rx=0 tx=0).stop 2023-10-30T13:06:15.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.398+0000 7f605d017640 1 -- 172.21.15.125:0/666144190 shutdown_connections 2023-10-30T13:06:15.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.398+0000 7f605d017640 1 --2- 172.21.15.125:0/666144190 >> v2:172.21.15.125:3300/0 conn(0x7f6058156da0 0x7f6058157200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:15.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.398+0000 7f605d017640 1 --2- 172.21.15.125:0/666144190 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f605814fc00 0x7f6058156330 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:15.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.398+0000 7f605d017640 1 -- 172.21.15.125:0/666144190 >> 172.21.15.125:0/666144190 conn(0x7f60580c7910 msgr2=0x7f60580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:15.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.402+0000 7f605d017640 1 -- 172.21.15.125:0/666144190 shutdown_connections 2023-10-30T13:06:15.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.402+0000 7f605d017640 1 -- 172.21.15.125:0/666144190 wait complete. 2023-10-30T13:06:15.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.402+0000 7f605d017640 1 Processor -- start 2023-10-30T13:06:15.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.402+0000 7f605d017640 1 -- start start 2023-10-30T13:06:15.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605d017640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6058069e90 0x7f60581408f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:15.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605d017640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f605814fc00 0x7f60580c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:15.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605d017640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6058156da0 0x7f60580c1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:15.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605d017640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60581f7c00 con 0x7f6058156da0 2023-10-30T13:06:15.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605d017640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60581413f0 con 0x7f6058069e90 2023-10-30T13:06:15.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605d017640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60581f6fc0 con 0x7f605814fc00 2023-10-30T13:06:15.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605759e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6058156da0 0x7f60580c1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:15.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605759e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6058156da0 0x7f60580c1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35754/0 (socket says 172.21.15.125:35754) 2023-10-30T13:06:15.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605759e640 1 -- 172.21.15.125:0/3200674580 learned_addr learned my addr 172.21.15.125:0/3200674580 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:15.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f6056d9d640 1 --2- 172.21.15.125:0/3200674580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6058069e90 0x7f60581408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:15.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605659c640 1 --2- 172.21.15.125:0/3200674580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f605814fc00 0x7f60580c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:15.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605759e640 1 -- 172.21.15.125:0/3200674580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f605814fc00 msgr2=0x7f60580c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:15.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605759e640 1 --2- 172.21.15.125:0/3200674580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f605814fc00 0x7f60580c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:15.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605759e640 1 -- 172.21.15.125:0/3200674580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6058069e90 msgr2=0x7f60581408f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:15.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605759e640 1 --2- 172.21.15.125:0/3200674580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6058069e90 0x7f60581408f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:15.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605759e640 1 -- 172.21.15.125:0/3200674580 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c009cb0 con 0x7f6058156da0 2023-10-30T13:06:15.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605759e640 1 --2- 172.21.15.125:0/3200674580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6058156da0 0x7f60580c1580 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f605000c770 tx=0x7f605000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:15.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f603f7fe640 1 -- 172.21.15.125:0/3200674580 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6050003d90 con 0x7f6058156da0 2023-10-30T13:06:15.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f603f7fe640 1 -- 172.21.15.125:0/3200674580 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f605000ce80 con 0x7f6058156da0 2023-10-30T13:06:15.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605d017640 1 -- 172.21.15.125:0/3200674580 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60581f7260 con 0x7f6058156da0 2023-10-30T13:06:15.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f603f7fe640 1 -- 172.21.15.125:0/3200674580 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60500052b0 con 0x7f6058156da0 2023-10-30T13:06:15.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605d017640 1 -- 172.21.15.125:0/3200674580 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60581f7820 con 0x7f6058156da0 2023-10-30T13:06:15.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.406+0000 7f605d017640 1 -- 172.21.15.125:0/3200674580 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60581f7d40 con 0x7f6058156da0 2023-10-30T13:06:15.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.410+0000 7f603f7fe640 1 -- 172.21.15.125:0/3200674580 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6050005410 con 0x7f6058156da0 2023-10-30T13:06:15.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.410+0000 7f603f7fe640 1 --2- 172.21.15.125:0/3200674580 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60440664a0 0x7f6044068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:15.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.410+0000 7f603f7fe640 1 -- 172.21.15.125:0/3200674580 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f6050083ca0 con 0x7f6058156da0 2023-10-30T13:06:15.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.410+0000 7f6056d9d640 1 --2- 172.21.15.125:0/3200674580 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60440664a0 0x7f6044068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:15.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.410+0000 7f6056d9d640 1 --2- 172.21.15.125:0/3200674580 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60440664a0 0x7f6044068960 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f604c0b3000 tx=0x7f604c0ab070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:15.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.414+0000 7f603f7fe640 1 -- 172.21.15.125:0/3200674580 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f60581f7d40 con 0x7f6058156da0 2023-10-30T13:06:15.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.666+0000 7f605d017640 1 -- 172.21.15.125:0/3200674580 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60581f7d40 con 0x7f6058156da0 2023-10-30T13:06:15.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.666+0000 7f603f7fe640 1 -- 172.21.15.125:0/3200674580 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 164 v164) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f60581f7d40 con 0x7f6058156da0 2023-10-30T13:06:15.672 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:15.673 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":164,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":164,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:09.611035+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8178},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:15.673 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 164 2023-10-30T13:06:15.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.670+0000 7f605d017640 1 -- 172.21.15.125:0/3200674580 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60440664a0 msgr2=0x7f6044068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:15.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.670+0000 7f605d017640 1 --2- 172.21.15.125:0/3200674580 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60440664a0 0x7f6044068960 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f604c0b3000 tx=0x7f604c0ab070 comp rx=0 tx=0).stop 2023-10-30T13:06:15.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.670+0000 7f605d017640 1 -- 172.21.15.125:0/3200674580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6058156da0 msgr2=0x7f60580c1580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:15.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.670+0000 7f605d017640 1 --2- 172.21.15.125:0/3200674580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6058156da0 0x7f60580c1580 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f605000c770 tx=0x7f605000cc30 comp rx=0 tx=0).stop 2023-10-30T13:06:15.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.670+0000 7f605d017640 1 -- 172.21.15.125:0/3200674580 shutdown_connections 2023-10-30T13:06:15.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.670+0000 7f605d017640 1 --2- 172.21.15.125:0/3200674580 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60440664a0 0x7f6044068960 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:15.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.670+0000 7f605d017640 1 --2- 172.21.15.125:0/3200674580 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6058156da0 0x7f60580c1580 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:15.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.670+0000 7f605d017640 1 --2- 172.21.15.125:0/3200674580 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f605814fc00 0x7f60580c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:15.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.670+0000 7f605d017640 1 --2- 172.21.15.125:0/3200674580 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6058069e90 0x7f60581408f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:15.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.670+0000 7f605d017640 1 -- 172.21.15.125:0/3200674580 >> 172.21.15.125:0/3200674580 conn(0x7f60580c7910 msgr2=0x7f605814d350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:15.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.670+0000 7f605d017640 1 -- 172.21.15.125:0/3200674580 shutdown_connections 2023-10-30T13:06:15.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:15.670+0000 7f605d017640 1 -- 172.21.15.125:0/3200674580 wait complete. 2023-10-30T13:06:15.695 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:06:17.696 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:17.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.838+0000 7f87978d6640 1 Processor -- start 2023-10-30T13:06:17.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.838+0000 7f87978d6640 1 -- start start 2023-10-30T13:06:17.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f87978d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87900c6f60 0x7f87901566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:17.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f87978d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87900c0a50 con 0x7f87900c6b80 2023-10-30T13:06:17.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f87978d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f879006f6e0 con 0x7f87900c05d0 2023-10-30T13:06:17.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f87978d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f879006f9c0 con 0x7f87900c6f60 2023-10-30T13:06:17.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8795672640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f87900c6b80 0x7f87900bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48014/0 (socket says 172.21.15.125:48014) 2023-10-30T13:06:17.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8795672640 1 -- 172.21.15.125:0/3680685106 learned_addr learned my addr 172.21.15.125:0/3680685106 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:06:17.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8795e73640 1 --2- 172.21.15.125:0/3680685106 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87900c6f60 0x7f87901566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:17.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 -- 172.21.15.125:0/3680685106 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943999721 0 0) 0x7f87900c0a50 con 0x7f87900c6b80 2023-10-30T13:06:17.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 -- 172.21.15.125:0/3680685106 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f877c005740 con 0x7f87900c6b80 2023-10-30T13:06:17.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 -- 172.21.15.125:0/3680685106 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 150946938 0 0) 0x7f877c005740 con 0x7f87900c6b80 2023-10-30T13:06:17.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 -- 172.21.15.125:0/3680685106 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87900c6f60 msgr2=0x7f87901566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:17.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 --2- 172.21.15.125:0/3680685106 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87900c6f60 0x7f87901566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:17.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 -- 172.21.15.125:0/3680685106 >> v1:172.21.15.136:6789/0 conn(0x7f87900c05d0 legacy=0x7f879013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:17.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 -- 172.21.15.125:0/3680685106 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f879006fca0 con 0x7f87900c6b80 2023-10-30T13:06:17.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 -- 172.21.15.125:0/3680685106 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f878c001680 con 0x7f87900c6b80 2023-10-30T13:06:17.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 -- 172.21.15.125:0/3680685106 >> v1:172.21.15.125:6789/0 conn(0x7f87900c6b80 legacy=0x7f87900bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:17.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 --2- 172.21.15.125:0/3680685106 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87900c6f60 0x7f877c005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:17.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 --2- 172.21.15.125:0/3680685106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f877c006140 0x7f877c0069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:17.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 --2- 172.21.15.125:0/3680685106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f877c006f20 0x7f877c0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:17.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 -- 172.21.15.125:0/3680685106 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879006f9c0 con 0x7f877c006140 2023-10-30T13:06:17.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 -- 172.21.15.125:0/3680685106 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f877c00dc80 con 0x7f877c006f20 2023-10-30T13:06:17.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8787fff640 1 -- 172.21.15.125:0/3680685106 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f877c00de40 con 0x7f87900c6f60 2023-10-30T13:06:17.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8794e71640 1 --2- 172.21.15.125:0/3680685106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f877c006140 0x7f877c0069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:17.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8795672640 1 --2- 172.21.15.125:0/3680685106 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87900c6f60 0x7f877c005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:17.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8795e73640 1 --2- 172.21.15.125:0/3680685106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f877c006f20 0x7f877c0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:17.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8794e71640 1 -- 172.21.15.125:0/3680685106 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87900c6f60 msgr2=0x7f877c005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:17.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8794e71640 1 --2- 172.21.15.125:0/3680685106 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87900c6f60 0x7f877c005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:17.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8794e71640 1 -- 172.21.15.125:0/3680685106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f877c006f20 msgr2=0x7f877c0073a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:17.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8794e71640 1 --2- 172.21.15.125:0/3680685106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f877c006f20 0x7f877c0073a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:17.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8794e71640 1 -- 172.21.15.125:0/3680685106 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877c005740 con 0x7f877c006140 2023-10-30T13:06:17.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.842+0000 7f8795e73640 1 --2- 172.21.15.125:0/3680685106 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f877c006f20 0x7f877c0073a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:17.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.846+0000 7f8794e71640 1 --2- 172.21.15.125:0/3680685106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f877c006140 0x7f877c0069e0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f8780001750 tx=0x7f878001c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6312fe895bf3ee7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:17.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.846+0000 7f8787fff640 1 -- 172.21.15.125:0/3680685106 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87800052d0 con 0x7f877c006140 2023-10-30T13:06:17.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.846+0000 7f8787fff640 1 -- 172.21.15.125:0/3680685106 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87800a7050 con 0x7f877c006140 2023-10-30T13:06:17.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.846+0000 7f8787fff640 1 -- 172.21.15.125:0/3680685106 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8780003600 con 0x7f877c006140 2023-10-30T13:06:17.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.846+0000 7f87978d6640 1 -- 172.21.15.125:0/3680685106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f877c006140 msgr2=0x7f877c0069e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:17.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.846+0000 7f87978d6640 1 --2- 172.21.15.125:0/3680685106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f877c006140 0x7f877c0069e0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f8780001750 tx=0x7f878001c3c0 comp rx=0 tx=0).stop 2023-10-30T13:06:17.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.846+0000 7f8795e73640 1 -- 172.21.15.125:0/3680685106 reap_dead start 2023-10-30T13:06:17.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.846+0000 7f87978d6640 1 -- 172.21.15.125:0/3680685106 shutdown_connections 2023-10-30T13:06:17.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.846+0000 7f87978d6640 1 -- 172.21.15.125:0/3680685106 >> 172.21.15.125:0/3680685106 conn(0x7f87900c7910 msgr2=0x7f87900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:17.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.846+0000 7f87978d6640 1 -- 172.21.15.125:0/3680685106 shutdown_connections 2023-10-30T13:06:17.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.846+0000 7f87978d6640 1 -- 172.21.15.125:0/3680685106 wait complete. 2023-10-30T13:06:17.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f87978d6640 1 Processor -- start 2023-10-30T13:06:17.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f87978d6640 1 -- start start 2023-10-30T13:06:17.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f87978d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f877c006140 0x7f8790152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:17.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f87978d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f879013cac0 0x7f8790139240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:17.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f87978d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8790152be0 0x7f8790139780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:17.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f87978d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87901efbc0 con 0x7f8790152be0 2023-10-30T13:06:17.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f87978d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87900cabe0 con 0x7f877c006140 2023-10-30T13:06:17.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f87978d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87900ca900 con 0x7f879013cac0 2023-10-30T13:06:17.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f8795e73640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8790152be0 0x7f8790139780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:17.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f8795e73640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8790152be0 0x7f8790139780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35762/0 (socket says 172.21.15.125:35762) 2023-10-30T13:06:17.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f8795e73640 1 -- 172.21.15.125:0/752319709 learned_addr learned my addr 172.21.15.125:0/752319709 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:17.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f8795672640 1 --2- 172.21.15.125:0/752319709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f877c006140 0x7f8790152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:17.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f8794e71640 1 --2- 172.21.15.125:0/752319709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f879013cac0 0x7f8790139240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:17.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f8795e73640 1 -- 172.21.15.125:0/752319709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f879013cac0 msgr2=0x7f8790139240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:17.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f8795e73640 1 --2- 172.21.15.125:0/752319709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f879013cac0 0x7f8790139240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:17.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f8795e73640 1 -- 172.21.15.125:0/752319709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f877c006140 msgr2=0x7f8790152420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:17.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f8795e73640 1 --2- 172.21.15.125:0/752319709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f877c006140 0x7f8790152420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:17.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f8795e73640 1 -- 172.21.15.125:0/752319709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8790139e00 con 0x7f8790152be0 2023-10-30T13:06:17.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f8795e73640 1 --2- 172.21.15.125:0/752319709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8790152be0 0x7f8790139780 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f87900b1190 tx=0x7f8788010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:17.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f8785ffb640 1 -- 172.21.15.125:0/752319709 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878801b070 con 0x7f8790152be0 2023-10-30T13:06:17.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f8785ffb640 1 -- 172.21.15.125:0/752319709 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878800f040 con 0x7f8790152be0 2023-10-30T13:06:17.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f8785ffb640 1 -- 172.21.15.125:0/752319709 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878800e050 con 0x7f8790152be0 2023-10-30T13:06:17.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f87978d6640 1 -- 172.21.15.125:0/752319709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87901ef7e0 con 0x7f8790152be0 2023-10-30T13:06:17.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f87978d6640 1 -- 172.21.15.125:0/752319709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8790152960 con 0x7f8790152be0 2023-10-30T13:06:17.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.850+0000 7f87978d6640 1 -- 172.21.15.125:0/752319709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f879004e170 con 0x7f8790152be0 2023-10-30T13:06:17.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.854+0000 7f8785ffb640 1 -- 172.21.15.125:0/752319709 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8788003be0 con 0x7f8790152be0 2023-10-30T13:06:17.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.854+0000 7f8785ffb640 1 --2- 172.21.15.125:0/752319709 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87740664a0 0x7f8774068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:17.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.854+0000 7f8785ffb640 1 -- 172.21.15.125:0/752319709 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f8788084af0 con 0x7f8790152be0 2023-10-30T13:06:17.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.854+0000 7f8795672640 1 --2- 172.21.15.125:0/752319709 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87740664a0 0x7f8774068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:17.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.854+0000 7f8795672640 1 --2- 172.21.15.125:0/752319709 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87740664a0 0x7f8774068960 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f878c0015d0 tx=0x7f878c001560 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:17.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:17.858+0000 7f8785ffb640 1 -- 172.21.15.125:0/752319709 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8788051880 con 0x7f8790152be0 2023-10-30T13:06:18.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.038+0000 7f87978d6640 1 -- 172.21.15.125:0/752319709 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8790073ea0 con 0x7f8790152be0 2023-10-30T13:06:18.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.038+0000 7f8785ffb640 1 -- 172.21.15.125:0/752319709 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f8788054ea0 con 0x7f8790152be0 2023-10-30T13:06:18.043 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:18.043 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:16.451066+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8178},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:18.044 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 165 2023-10-30T13:06:18.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.042+0000 7f87978d6640 1 -- 172.21.15.125:0/752319709 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87740664a0 msgr2=0x7f8774068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:18.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.042+0000 7f87978d6640 1 --2- 172.21.15.125:0/752319709 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87740664a0 0x7f8774068960 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f878c0015d0 tx=0x7f878c001560 comp rx=0 tx=0).stop 2023-10-30T13:06:18.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.042+0000 7f87978d6640 1 -- 172.21.15.125:0/752319709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8790152be0 msgr2=0x7f8790139780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:18.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.042+0000 7f87978d6640 1 --2- 172.21.15.125:0/752319709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8790152be0 0x7f8790139780 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f87900b1190 tx=0x7f8788010590 comp rx=0 tx=0).stop 2023-10-30T13:06:18.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.042+0000 7f87978d6640 1 -- 172.21.15.125:0/752319709 shutdown_connections 2023-10-30T13:06:18.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.042+0000 7f87978d6640 1 --2- 172.21.15.125:0/752319709 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87740664a0 0x7f8774068960 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:18.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.042+0000 7f87978d6640 1 --2- 172.21.15.125:0/752319709 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8790152be0 0x7f8790139780 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:18.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.042+0000 7f87978d6640 1 --2- 172.21.15.125:0/752319709 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f879013cac0 0x7f8790139240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:18.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.042+0000 7f87978d6640 1 --2- 172.21.15.125:0/752319709 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f877c006140 0x7f8790152420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:18.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.042+0000 7f87978d6640 1 -- 172.21.15.125:0/752319709 >> 172.21.15.125:0/752319709 conn(0x7f87900c7910 msgr2=0x7f879006b570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:18.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.042+0000 7f87978d6640 1 -- 172.21.15.125:0/752319709 shutdown_connections 2023-10-30T13:06:18.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.042+0000 7f87978d6640 1 -- 172.21.15.125:0/752319709 wait complete. 2023-10-30T13:06:18.064 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=8022 2023-10-30T13:06:18.064 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.c 2023-10-30T13:06:18.064 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:06:18.064 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:06:18.065 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:06:18.068 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:06:18.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:18.155 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2023-10-30T13:06:18.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b73c91640 1 Processor -- start 2023-10-30T13:06:18.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b73c91640 1 -- start start 2023-10-30T13:06:18.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b73c91640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4b6c1538a0 0x7f4b6c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:18.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b73c91640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b6c0cf160 0x7f4b6c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:18.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b73c91640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b6c0cb800 con 0x7f4b6c0ca0e0 2023-10-30T13:06:18.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b73c91640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b6c15dc30 con 0x7f4b6c1538a0 2023-10-30T13:06:18.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b73c91640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c0cb330 con 0x7f4b6c0cf160 2023-10-30T13:06:18.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b7222e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b6c0cf160 0x7f4b6c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:18.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b7222e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b6c0cf160 0x7f4b6c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38408/0 (socket says 172.21.15.125:38408) 2023-10-30T13:06:18.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b7222e640 1 -- 172.21.15.125:0/4006481931 learned_addr learned my addr 172.21.15.125:0/4006481931 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:18.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b71a2d640 1 --2- 172.21.15.125:0/4006481931 >> v2:172.21.15.136:3300/0 conn(0x7f4b6c1538a0 0x7f4b6c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:18.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b7222e640 1 -- 172.21.15.125:0/4006481931 >> v2:172.21.15.136:3300/0 conn(0x7f4b6c1538a0 msgr2=0x7f4b6c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:18.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b7222e640 1 --2- 172.21.15.125:0/4006481931 >> v2:172.21.15.136:3300/0 conn(0x7f4b6c1538a0 0x7f4b6c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:18.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b7222e640 1 -- 172.21.15.125:0/4006481931 >> v1:172.21.15.136:6789/0 conn(0x7f4b6c0ca0e0 legacy=0x7f4b6c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:18.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.206+0000 7f4b7222e640 1 -- 172.21.15.125:0/4006481931 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6c0cad70 con 0x7f4b6c0cf160 2023-10-30T13:06:18.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.210+0000 7f4b7222e640 1 --2- 172.21.15.125:0/4006481931 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b6c0cf160 0x7f4b6c0cf590 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4b64009e50 tx=0x7f4b6401f1f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e4f809571c9130d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:18.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.210+0000 7f4b70a2b640 1 -- 172.21.15.125:0/4006481931 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b640af400 con 0x7f4b6c0cf160 2023-10-30T13:06:18.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.210+0000 7f4b70a2b640 1 -- 172.21.15.125:0/4006481931 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b640ab050 con 0x7f4b6c0cf160 2023-10-30T13:06:18.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.210+0000 7f4b70a2b640 1 -- 172.21.15.125:0/4006481931 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b640af780 con 0x7f4b6c0cf160 2023-10-30T13:06:18.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.210+0000 7f4b73c91640 1 -- 172.21.15.125:0/4006481931 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b6c0cf160 msgr2=0x7f4b6c0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:18.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.210+0000 7f4b73c91640 1 --2- 172.21.15.125:0/4006481931 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b6c0cf160 0x7f4b6c0cf590 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4b64009e50 tx=0x7f4b6401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:06:18.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.210+0000 7f4b73c91640 1 -- 172.21.15.125:0/4006481931 shutdown_connections 2023-10-30T13:06:18.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.210+0000 7f4b73c91640 1 --2- 172.21.15.125:0/4006481931 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b6c0cf160 0x7f4b6c0cf590 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:18.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.210+0000 7f4b73c91640 1 --2- 172.21.15.125:0/4006481931 >> v2:172.21.15.136:3300/0 conn(0x7f4b6c1538a0 0x7f4b6c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:18.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.210+0000 7f4b73c91640 1 -- 172.21.15.125:0/4006481931 >> 172.21.15.125:0/4006481931 conn(0x7f4b6c06bfe0 msgr2=0x7f4b6c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:18.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.210+0000 7f4b73c91640 1 -- 172.21.15.125:0/4006481931 shutdown_connections 2023-10-30T13:06:18.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.210+0000 7f4b73c91640 1 -- 172.21.15.125:0/4006481931 wait complete. 2023-10-30T13:06:18.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b73c91640 1 Processor -- start 2023-10-30T13:06:18.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b73c91640 1 -- start start 2023-10-30T13:06:18.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b73c91640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b6c0ca0e0 0x7f4b6c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:18.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b73c91640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b6c0cf160 0x7f4b6c142550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:18.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b73c91640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b6c1538a0 0x7f4b6c157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:18.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b73c91640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c0ce820 con 0x7f4b6c1538a0 2023-10-30T13:06:18.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b73c91640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c0ceae0 con 0x7f4b6c0cf160 2023-10-30T13:06:18.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b73c91640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c0cedc0 con 0x7f4b6c0ca0e0 2023-10-30T13:06:18.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b7222e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b6c1538a0 0x7f4b6c157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:18.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b7222e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b6c1538a0 0x7f4b6c157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35784/0 (socket says 172.21.15.125:35784) 2023-10-30T13:06:18.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b7222e640 1 -- 172.21.15.125:0/840251352 learned_addr learned my addr 172.21.15.125:0/840251352 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:18.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b71a2d640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b6c0ca0e0 0x7f4b6c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:18.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b7122c640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b6c0cf160 0x7f4b6c142550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:18.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b7222e640 1 -- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b6c0ca0e0 msgr2=0x7f4b6c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:18.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b7222e640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b6c0ca0e0 0x7f4b6c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:18.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b7222e640 1 -- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b6c0cf160 msgr2=0x7f4b6c142550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:18.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b7222e640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b6c0cf160 0x7f4b6c142550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:18.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b7222e640 1 -- 172.21.15.125:0/840251352 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b64009b30 con 0x7f4b6c1538a0 2023-10-30T13:06:18.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b7122c640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b6c0cf160 0x7f4b6c142550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:18.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b7222e640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b6c1538a0 0x7f4b6c157520 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f4b6401ce20 tx=0x7f4b6401bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:18.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b627fc640 1 -- 172.21.15.125:0/840251352 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b640afc80 con 0x7f4b6c1538a0 2023-10-30T13:06:18.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b627fc640 1 -- 172.21.15.125:0/840251352 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b640afde0 con 0x7f4b6c1538a0 2023-10-30T13:06:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b627fc640 1 -- 172.21.15.125:0/840251352 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b640ba660 con 0x7f4b6c1538a0 2023-10-30T13:06:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b73c91640 1 -- 172.21.15.125:0/840251352 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6c1efb80 con 0x7f4b6c1538a0 2023-10-30T13:06:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.214+0000 7f4b73c91640 1 -- 172.21.15.125:0/840251352 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6c1f0140 con 0x7f4b6c1538a0 2023-10-30T13:06:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.218+0000 7f4b73c91640 1 -- 172.21.15.125:0/840251352 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6c0b2a50 con 0x7f4b6c1538a0 2023-10-30T13:06:18.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.218+0000 7f4b627fc640 1 -- 172.21.15.125:0/840251352 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4b640b6040 con 0x7f4b6c1538a0 2023-10-30T13:06:18.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.218+0000 7f4b627fc640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b500664a0 0x7f4b50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:18.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.218+0000 7f4b627fc640 1 -- 172.21.15.125:0/840251352 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f4b641230a0 con 0x7f4b6c1538a0 2023-10-30T13:06:18.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.218+0000 7f4b71a2d640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b500664a0 0x7f4b50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:18.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.218+0000 7f4b71a2d640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b500664a0 0x7f4b50068960 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009ac0 tx=0x7f4b68009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:18.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.222+0000 7f4b627fc640 1 -- 172.21.15.125:0/840251352 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4b640f0000 con 0x7f4b6c1538a0 2023-10-30T13:06:18.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.402+0000 7f4b73c91640 1 -- 172.21.15.125:0/840251352 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b6c1f0330 con 0x7f4b6c1538a0 2023-10-30T13:06:18.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.402+0000 7f4b627fc640 1 -- 172.21.15.125:0/840251352 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 165 v165) v1 ==== 95+0+5724 (secure 0 0 0) 0x7f4b64009510 con 0x7f4b6c1538a0 2023-10-30T13:06:18.405 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:18.406 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":165,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:16.451066+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8178},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:18.407 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 165 2023-10-30T13:06:18.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.402+0000 7f4b73c91640 1 -- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b500664a0 msgr2=0x7f4b50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:18.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.402+0000 7f4b73c91640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b500664a0 0x7f4b50068960 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009ac0 tx=0x7f4b68009190 comp rx=0 tx=0).stop 2023-10-30T13:06:18.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.402+0000 7f4b73c91640 1 -- 172.21.15.125:0/840251352 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b6c1538a0 msgr2=0x7f4b6c157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:18.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.402+0000 7f4b73c91640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b6c1538a0 0x7f4b6c157520 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f4b6401ce20 tx=0x7f4b6401bf90 comp rx=0 tx=0).stop 2023-10-30T13:06:18.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.402+0000 7f4b73c91640 1 -- 172.21.15.125:0/840251352 shutdown_connections 2023-10-30T13:06:18.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.402+0000 7f4b73c91640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b500664a0 0x7f4b50068960 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:18.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.402+0000 7f4b73c91640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b6c1538a0 0x7f4b6c157520 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:18.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.402+0000 7f4b73c91640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b6c0cf160 0x7f4b6c142550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:18.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.402+0000 7f4b73c91640 1 --2- 172.21.15.125:0/840251352 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b6c0ca0e0 0x7f4b6c0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:18.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.402+0000 7f4b73c91640 1 -- 172.21.15.125:0/840251352 >> 172.21.15.125:0/840251352 conn(0x7f4b6c06bfe0 msgr2=0x7f4b6c0c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:18.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.402+0000 7f4b73c91640 1 -- 172.21.15.125:0/840251352 shutdown_connections 2023-10-30T13:06:18.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:18.402+0000 7f4b73c91640 1 -- 172.21.15.125:0/840251352 wait complete. 2023-10-30T13:06:18.426 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:06:20.427 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:20.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.570+0000 7f680013e640 1 Processor -- start 2023-10-30T13:06:20.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.570+0000 7f680013e640 1 -- start start 2023-10-30T13:06:20.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.570+0000 7f680013e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67f8157010 0x7f67f80715c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:20.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.570+0000 7f680013e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f67f8070650 0x7f67f8159810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:20.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.570+0000 7f680013e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f8159d50 con 0x7f67f80c8460 2023-10-30T13:06:20.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.570+0000 7f680013e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f80c21b0 con 0x7f67f8070650 2023-10-30T13:06:20.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.570+0000 7f680013e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f8070ae0 con 0x7f67f8157010 2023-10-30T13:06:20.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.570+0000 7f67fd6d9640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f67f80c8460 0x7f67f8071bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48022/0 (socket says 172.21.15.125:48022) 2023-10-30T13:06:20.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.570+0000 7f67fd6d9640 1 -- 172.21.15.125:0/2687070010 learned_addr learned my addr 172.21.15.125:0/2687070010 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:06:20.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.570+0000 7f67fe6db640 1 --2- 172.21.15.125:0/2687070010 >> v2:172.21.15.136:3300/0 conn(0x7f67f8070650 0x7f67f8159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:20.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.570+0000 7f67fdeda640 1 --2- 172.21.15.125:0/2687070010 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67f8157010 0x7f67f80715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:20.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.574+0000 7f67fced8640 1 -- 172.21.15.125:0/2687070010 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557830506 0 0) 0x7f67f8159d50 con 0x7f67f80c8460 2023-10-30T13:06:20.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.574+0000 7f67fced8640 1 -- 172.21.15.125:0/2687070010 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e4003530 con 0x7f67f80c8460 2023-10-30T13:06:20.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.574+0000 7f67fe6db640 1 -- 172.21.15.125:0/2687070010 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67f8157010 msgr2=0x7f67f80715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:20.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.574+0000 7f67fe6db640 1 --2- 172.21.15.125:0/2687070010 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67f8157010 0x7f67f80715c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:20.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.574+0000 7f67fced8640 1 -- 172.21.15.125:0/2687070010 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1974911274 0 0) 0x7f67e4003530 con 0x7f67f80c8460 2023-10-30T13:06:20.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.574+0000 7f67fe6db640 1 -- 172.21.15.125:0/2687070010 >> v1:172.21.15.125:6789/0 conn(0x7f67f80c8460 legacy=0x7f67f8071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:20.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.574+0000 7f67fe6db640 1 -- 172.21.15.125:0/2687070010 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f80c1530 con 0x7f67f8070650 2023-10-30T13:06:20.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.574+0000 7f67fe6db640 1 --2- 172.21.15.125:0/2687070010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f8070650 0x7f67f8159810 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f67f0009d90 tx=0x7f67f001c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de23d47b0c1a128d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:20.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.574+0000 7f67fced8640 1 -- 172.21.15.125:0/2687070010 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67e8001680 con 0x7f67f8070650 2023-10-30T13:06:20.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.574+0000 7f67fced8640 1 -- 172.21.15.125:0/2687070010 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f00a5640 con 0x7f67f8070650 2023-10-30T13:06:20.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.574+0000 7f67fced8640 1 -- 172.21.15.125:0/2687070010 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f00a59d0 con 0x7f67f8070650 2023-10-30T13:06:20.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 -- 172.21.15.125:0/2687070010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f8070650 msgr2=0x7f67f8159810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:20.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 --2- 172.21.15.125:0/2687070010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f8070650 0x7f67f8159810 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f67f0009d90 tx=0x7f67f001c840 comp rx=0 tx=0).stop 2023-10-30T13:06:20.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 -- 172.21.15.125:0/2687070010 shutdown_connections 2023-10-30T13:06:20.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 --2- 172.21.15.125:0/2687070010 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f8070650 0x7f67f8159810 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:20.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 --2- 172.21.15.125:0/2687070010 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67f8157010 0x7f67f80715c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:20.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 -- 172.21.15.125:0/2687070010 >> 172.21.15.125:0/2687070010 conn(0x7f67f80cbeb0 msgr2=0x7f67f80cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:20.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 -- 172.21.15.125:0/2687070010 shutdown_connections 2023-10-30T13:06:20.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 -- 172.21.15.125:0/2687070010 wait complete. 2023-10-30T13:06:20.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 Processor -- start 2023-10-30T13:06:20.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 -- start start 2023-10-30T13:06:20.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f8070650 0x7f67f81f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:20.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67f80c8460 0x7f67f813c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:20.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67f8157010 0x7f67f814ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:20.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f814bb60 con 0x7f67f8157010 2023-10-30T13:06:20.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f806a9f0 con 0x7f67f8070650 2023-10-30T13:06:20.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f680013e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f81f2d00 con 0x7f67f80c8460 2023-10-30T13:06:20.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f67fe6db640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67f8157010 0x7f67f814ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:20.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f67fe6db640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67f8157010 0x7f67f814ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35796/0 (socket says 172.21.15.125:35796) 2023-10-30T13:06:20.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f67fe6db640 1 -- 172.21.15.125:0/1239046529 learned_addr learned my addr 172.21.15.125:0/1239046529 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:20.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f67fdeda640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f8070650 0x7f67f81f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:20.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f67fd6d9640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67f80c8460 0x7f67f813c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:20.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f67fe6db640 1 -- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67f80c8460 msgr2=0x7f67f813c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:20.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f67fe6db640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67f80c8460 0x7f67f813c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:20.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f67fe6db640 1 -- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f8070650 msgr2=0x7f67f81f27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:20.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.578+0000 7f67fe6db640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f8070650 0x7f67f81f27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:20.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.582+0000 7f67fe6db640 1 -- 172.21.15.125:0/1239046529 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f0009a70 con 0x7f67f8157010 2023-10-30T13:06:20.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.582+0000 7f67fdeda640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f8070650 0x7f67f81f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:20.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.582+0000 7f67fe6db640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67f8157010 0x7f67f814ae20 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f67f00b0040 tx=0x7f67f00195e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:20.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.582+0000 7f67ee7fc640 1 -- 172.21.15.125:0/1239046529 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f00b9860 con 0x7f67f8157010 2023-10-30T13:06:20.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.582+0000 7f67ee7fc640 1 -- 172.21.15.125:0/1239046529 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f00b99c0 con 0x7f67f8157010 2023-10-30T13:06:20.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.582+0000 7f67ee7fc640 1 -- 172.21.15.125:0/1239046529 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f00b9d70 con 0x7f67f8157010 2023-10-30T13:06:20.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.582+0000 7f680013e640 1 -- 172.21.15.125:0/1239046529 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f80c7de0 con 0x7f67f8157010 2023-10-30T13:06:20.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.582+0000 7f680013e640 1 -- 172.21.15.125:0/1239046529 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f806a3a0 con 0x7f67f8157010 2023-10-30T13:06:20.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.582+0000 7f680013e640 1 -- 172.21.15.125:0/1239046529 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f804e170 con 0x7f67f8157010 2023-10-30T13:06:20.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.582+0000 7f67ee7fc640 1 -- 172.21.15.125:0/1239046529 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f67f00ad440 con 0x7f67f8157010 2023-10-30T13:06:20.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.582+0000 7f67ee7fc640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67dc0664d0 0x7f67dc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:20.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.582+0000 7f67ee7fc640 1 -- 172.21.15.125:0/1239046529 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f67f011f840 con 0x7f67f8157010 2023-10-30T13:06:20.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.582+0000 7f67fdeda640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67dc0664d0 0x7f67dc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:20.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.586+0000 7f67fdeda640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67dc0664d0 0x7f67dc068990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f67f4009b10 tx=0x7f67f4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:20.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.586+0000 7f67ee7fc640 1 -- 172.21.15.125:0/1239046529 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f67f00ec5d0 con 0x7f67f8157010 2023-10-30T13:06:20.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.770+0000 7f680013e640 1 -- 172.21.15.125:0/1239046529 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67f806ab30 con 0x7f67f8157010 2023-10-30T13:06:20.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.770+0000 7f67ee7fc640 1 -- 172.21.15.125:0/1239046529 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+6519 (secure 0 0 0) 0x7f67f0009450 con 0x7f67f8157010 2023-10-30T13:06:20.775 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:20.775 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8301,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:16.451066+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8178},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:20.776 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 166 2023-10-30T13:06:20.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.774+0000 7f680013e640 1 -- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67dc0664d0 msgr2=0x7f67dc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:20.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.774+0000 7f680013e640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67dc0664d0 0x7f67dc068990 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f67f4009b10 tx=0x7f67f4008040 comp rx=0 tx=0).stop 2023-10-30T13:06:20.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.774+0000 7f680013e640 1 -- 172.21.15.125:0/1239046529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67f8157010 msgr2=0x7f67f814ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:20.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.774+0000 7f680013e640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67f8157010 0x7f67f814ae20 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f67f00b0040 tx=0x7f67f00195e0 comp rx=0 tx=0).stop 2023-10-30T13:06:20.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.774+0000 7f680013e640 1 -- 172.21.15.125:0/1239046529 shutdown_connections 2023-10-30T13:06:20.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.774+0000 7f680013e640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67dc0664d0 0x7f67dc068990 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:20.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.774+0000 7f680013e640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67f8157010 0x7f67f814ae20 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:20.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.774+0000 7f680013e640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67f80c8460 0x7f67f813c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:20.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.774+0000 7f680013e640 1 --2- 172.21.15.125:0/1239046529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67f8070650 0x7f67f81f27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:20.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.774+0000 7f680013e640 1 -- 172.21.15.125:0/1239046529 >> 172.21.15.125:0/1239046529 conn(0x7f67f80cbeb0 msgr2=0x7f67f8153aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:20.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.774+0000 7f680013e640 1 -- 172.21.15.125:0/1239046529 shutdown_connections 2023-10-30T13:06:20.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:20.774+0000 7f680013e640 1 -- 172.21.15.125:0/1239046529 wait complete. 2023-10-30T13:06:20.796 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:06:20.796 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 10 secs before thrashing 2023-10-30T13:06:22.611 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T13:06:22.606+0000 7f2a86270640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:06:23.075 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:06:23.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:06:30.562 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 15% (50/329) Updating files: 16% (53/329) Updating files: 17% (56/329) Updating files: 18% (60/329) Updating files: 19% (63/329) Updating files: 20% (66/329) Updating files: 21% (70/329) Updating files: 22% (73/329) Updating files: 23% (76/329) Updating files: 24% (79/329) Updating files: 25% (83/329) Updating files: 26% (86/329) Updating files: 27% (89/329) Updating files: 27% (90/329) Updating files: 28% (93/329) Updating files: 29% (96/329) Updating files: 30% (99/329) Updating files: 31% (102/329) Updating files: 32% (106/329) Updating files: 33% (109/329) Updating files: 34% (112/329) Updating files: 35% (116/329) Updating files: 36% (119/329) Updating files: 37% (122/329) Updating files: 38% (126/329) Updating files: 39% (129/329) Updating files: 40% (132/329) Updating files: 41% (135/329) Updating files: 42% (139/329) Updating files: 43% (142/329) Updating files: 44% (145/329) Updating files: 45% (149/329) Updating files: 46% (152/329) Updating files: 47% (155/329) Updating files: 48% (158/329) Updating files: 49% (162/329) Updating files: 50% (165/329) Updating files: 51% (168/329) Updating files: 52% (172/329) Updating files: 53% (175/329) Updating files: 54% (178/329) Updating files: 55% (181/329) Updating files: 55% (184/329) Updating files: 56% (185/329) Updating files: 57% (188/329) Updating files: 58% (191/329) Updating files: 59% (195/329) Updating files: 60% (198/329) Updating files: 61% (201/329) Updating files: 62% (204/329) Updating files: 63% (208/329) Updating files: 64% (211/329) Updating files: 65% (214/329) Updating files: 66% (218/329) Updating files: 67% (221/329) Updating files: 68% (224/329) Updating files: 69% (228/329) Updating files: 70% (231/329) Updating files: 71% (234/329) Updating files: 72% (237/329) Updating files: 73% (241/329) Updating files: 74% (244/329) Updating files: 75% (247/329) Updating files: 76% (251/329) Updating files: 77% (254/329) Updating files: 78% (257/329) Updating files: 79% (260/329) Updating files: 80% (264/329) Updating files: 81% (267/329) Updating files: 82% (270/329) Updating files: 83% (274/329) Updating files: 84% (277/329) Updating files: 85% (280/329) Updating files: 86% (283/329) Updating files: 86% (286/329) Updating files: 87% (287/329) Updating files: 88% (290/329) Updating files: 89% (293/329) Updating files: 90% (297/329) Updating files: 91% (300/329) Updating files: 92% (303/329) Updating files: 93% (306/329) Updating files: 94% (310/329) Updating files: 95% (313/329) Updating files: 96% (316/329) Updating files: 96% (319/329) Updating files: 97% (320/329) Updating files: 98% (323/329) Updating files: 99% (326/329) Updating files: 100% (329/329) Updating files: 100% (329/329), done. 2023-10-30T13:06:30.702 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 4c2d73a5095 0.79 2023-10-30T13:06:30.708 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.79 2023-10-30T13:06:30.762 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 80 -eq 48 ] 2023-10-30T13:06:30.762 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.80 2023-10-30T13:06:30.762 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.80 2023-10-30T13:06:30.763 INFO:tasks.workunit.client.0.smithi125.stdout:v0.80 2023-10-30T13:06:30.763 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.80 2023-10-30T13:06:30.797 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:30.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.934+0000 7fac34c46640 1 Processor -- start 2023-10-30T13:06:30.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.934+0000 7fac34c46640 1 -- start start 2023-10-30T13:06:30.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.938+0000 7fac34c46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac300715c0 0x7fac30070650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:30.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.938+0000 7fac34c46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac300c1530 con 0x7fac30157010 2023-10-30T13:06:30.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.938+0000 7fac34c46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac300c0f70 con 0x7fac300c8460 2023-10-30T13:06:30.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.938+0000 7fac34c46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac300c21b0 con 0x7fac300715c0 2023-10-30T13:06:30.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.938+0000 7fac2ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac300715c0 0x7fac30070650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:30.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.938+0000 7fac2ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fac300715c0 0x7fac30070650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45166/0 (socket says 172.21.15.125:45166) 2023-10-30T13:06:30.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.938+0000 7fac2ed9d640 1 -- 172.21.15.125:0/845522007 learned_addr learned my addr 172.21.15.125:0/845522007 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:30.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.938+0000 7fac2ed9d640 1 -- 172.21.15.125:0/845522007 >> v1:172.21.15.136:6789/0 conn(0x7fac300c8460 legacy=0x7fac301620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:30.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.938+0000 7fac2ed9d640 1 -- 172.21.15.125:0/845522007 >> v1:172.21.15.125:6789/0 conn(0x7fac30157010 legacy=0x7fac30071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:30.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.938+0000 7fac2ed9d640 1 -- 172.21.15.125:0/845522007 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac300c1250 con 0x7fac300715c0 2023-10-30T13:06:30.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac2ed9d640 1 --2- 172.21.15.125:0/845522007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac300715c0 0x7fac30070650 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fac28009e50 tx=0x7fac2801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=113d99c2817d4d0f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:30.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac2d59a640 1 -- 172.21.15.125:0/845522007 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac280a5860 con 0x7fac300715c0 2023-10-30T13:06:30.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac2d59a640 1 -- 172.21.15.125:0/845522007 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac280a59c0 con 0x7fac300715c0 2023-10-30T13:06:30.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac2d59a640 1 -- 172.21.15.125:0/845522007 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac280a5d50 con 0x7fac300715c0 2023-10-30T13:06:30.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 -- 172.21.15.125:0/845522007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac300715c0 msgr2=0x7fac30070650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:30.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 --2- 172.21.15.125:0/845522007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac300715c0 0x7fac30070650 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7fac28009e50 tx=0x7fac2801c9a0 comp rx=0 tx=0).stop 2023-10-30T13:06:30.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 -- 172.21.15.125:0/845522007 shutdown_connections 2023-10-30T13:06:30.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 --2- 172.21.15.125:0/845522007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac300715c0 0x7fac30070650 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:30.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 -- 172.21.15.125:0/845522007 >> 172.21.15.125:0/845522007 conn(0x7fac300cbeb0 msgr2=0x7fac300cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:30.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 -- 172.21.15.125:0/845522007 shutdown_connections 2023-10-30T13:06:30.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 -- 172.21.15.125:0/845522007 wait complete. 2023-10-30T13:06:30.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 Processor -- start 2023-10-30T13:06:30.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 -- start start 2023-10-30T13:06:30.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac300715c0 0x7fac30155160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:30.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac300c8460 0x7fac300ca350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:30.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac30157010 0x7fac300ca890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:30.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac30148290 con 0x7fac300715c0 2023-10-30T13:06:30.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac30154c20 con 0x7fac300c8460 2023-10-30T13:06:30.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.942+0000 7fac34c46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac3006ed60 con 0x7fac30157010 2023-10-30T13:06:30.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac2e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac300715c0 0x7fac30155160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:30.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac2e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac300715c0 0x7fac30155160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45172/0 (socket says 172.21.15.125:45172) 2023-10-30T13:06:30.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac2e59c640 1 -- 172.21.15.125:0/1550006977 learned_addr learned my addr 172.21.15.125:0/1550006977 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:30.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac2ed9d640 1 --2- 172.21.15.125:0/1550006977 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac30157010 0x7fac300ca890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:30.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac2dd9b640 1 --2- 172.21.15.125:0/1550006977 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac300c8460 0x7fac300ca350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:30.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac2e59c640 1 -- 172.21.15.125:0/1550006977 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac30157010 msgr2=0x7fac300ca890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:30.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac2e59c640 1 --2- 172.21.15.125:0/1550006977 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac30157010 0x7fac300ca890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:30.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac2e59c640 1 -- 172.21.15.125:0/1550006977 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac300c8460 msgr2=0x7fac300ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:30.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac2e59c640 1 --2- 172.21.15.125:0/1550006977 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac300c8460 0x7fac300ca350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:30.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac2e59c640 1 -- 172.21.15.125:0/1550006977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac28009b30 con 0x7fac300715c0 2023-10-30T13:06:30.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac2e59c640 1 --2- 172.21.15.125:0/1550006977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac300715c0 0x7fac30155160 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fac2000c420 tx=0x7fac2000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:30.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac1effd640 1 -- 172.21.15.125:0/1550006977 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac200052d0 con 0x7fac300715c0 2023-10-30T13:06:30.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac1effd640 1 -- 172.21.15.125:0/1550006977 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac20005430 con 0x7fac300715c0 2023-10-30T13:06:30.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac1effd640 1 -- 172.21.15.125:0/1550006977 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac20007260 con 0x7fac300715c0 2023-10-30T13:06:30.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac34c46640 1 -- 172.21.15.125:0/1550006977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac3014ba20 con 0x7fac300715c0 2023-10-30T13:06:30.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac34c46640 1 -- 172.21.15.125:0/1550006977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac3014af10 con 0x7fac300715c0 2023-10-30T13:06:30.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac34c46640 1 -- 172.21.15.125:0/1550006977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac3004e170 con 0x7fac300715c0 2023-10-30T13:06:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac1effd640 1 -- 172.21.15.125:0/1550006977 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fac200073c0 con 0x7fac300715c0 2023-10-30T13:06:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.946+0000 7fac1effd640 1 --2- 172.21.15.125:0/1550006977 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac10066470 0x7fac10068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.950+0000 7fac1effd640 1 -- 172.21.15.125:0/1550006977 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7fac20083aa0 con 0x7fac300715c0 2023-10-30T13:06:30.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.950+0000 7fac2dd9b640 1 --2- 172.21.15.125:0/1550006977 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac10066470 0x7fac10068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:30.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.950+0000 7fac2dd9b640 1 --2- 172.21.15.125:0/1550006977 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac10066470 0x7fac10068930 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fac3015dd30 tx=0x7fac24009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:30.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:30.954+0000 7fac1effd640 1 -- 172.21.15.125:0/1550006977 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fac20050710 con 0x7fac300715c0 2023-10-30T13:06:31.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.134+0000 7fac34c46640 1 -- 172.21.15.125:0/1550006977 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac300c8240 con 0x7fac300715c0 2023-10-30T13:06:31.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.138+0000 7fac1effd640 1 -- 172.21.15.125:0/1550006977 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+6519 (secure 0 0 0) 0x7fac20053d30 con 0x7fac300715c0 2023-10-30T13:06:31.140 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:31.140 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111},{"gid":8301,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":166}],"filesystems":[{"mdsmap":{"epoch":165,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:16.451066+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8178},"failed":[],"damaged":[],"stopped":[1,2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:31.140 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 166 2023-10-30T13:06:31.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.138+0000 7fac34c46640 1 -- 172.21.15.125:0/1550006977 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac10066470 msgr2=0x7fac10068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:31.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.138+0000 7fac34c46640 1 --2- 172.21.15.125:0/1550006977 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac10066470 0x7fac10068930 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fac3015dd30 tx=0x7fac24009040 comp rx=0 tx=0).stop 2023-10-30T13:06:31.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.138+0000 7fac34c46640 1 -- 172.21.15.125:0/1550006977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac300715c0 msgr2=0x7fac30155160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:31.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.138+0000 7fac34c46640 1 --2- 172.21.15.125:0/1550006977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac300715c0 0x7fac30155160 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fac2000c420 tx=0x7fac2000c8e0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.138+0000 7fac34c46640 1 -- 172.21.15.125:0/1550006977 shutdown_connections 2023-10-30T13:06:31.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.138+0000 7fac34c46640 1 --2- 172.21.15.125:0/1550006977 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac10066470 0x7fac10068930 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.138+0000 7fac34c46640 1 --2- 172.21.15.125:0/1550006977 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac30157010 0x7fac300ca890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.138+0000 7fac34c46640 1 --2- 172.21.15.125:0/1550006977 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac300c8460 0x7fac300ca350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.138+0000 7fac34c46640 1 --2- 172.21.15.125:0/1550006977 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac300715c0 0x7fac30155160 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.138+0000 7fac34c46640 1 -- 172.21.15.125:0/1550006977 >> 172.21.15.125:0/1550006977 conn(0x7fac300cbeb0 msgr2=0x7fac301515d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:31.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.138+0000 7fac34c46640 1 -- 172.21.15.125:0/1550006977 shutdown_connections 2023-10-30T13:06:31.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.138+0000 7fac34c46640 1 -- 172.21.15.125:0/1550006977 wait complete. 2023-10-30T13:06:31.160 INFO:tasks.mds_thrash.fs.[cephfs]:thrashing max_mds: 1 -> 3 2023-10-30T13:06:31.160 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-10-30T13:06:31.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.298+0000 7fabc4c46640 1 Processor -- start 2023-10-30T13:06:31.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.298+0000 7fabc4c46640 1 -- start start 2023-10-30T13:06:31.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.302+0000 7fabc4c46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fabc0153980 0x7fabc00caca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:31.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.302+0000 7fabc4c46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc015a920 con 0x7fabc0159930 2023-10-30T13:06:31.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.302+0000 7fabc4c46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc00cb930 con 0x7fabc00cb1e0 2023-10-30T13:06:31.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.302+0000 7fabc4c46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc0159dc0 con 0x7fabc0153980 2023-10-30T13:06:31.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.302+0000 7fabbe59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fabc0153980 0x7fabc00caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:31.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.302+0000 7fabbdd9b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fabc0159930 0x7fabc00cf1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46818/0 (socket says 172.21.15.125:46818) 2023-10-30T13:06:31.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.302+0000 7fabbe59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fabc0153980 0x7fabc00caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45178/0 (socket says 172.21.15.125:45178) 2023-10-30T13:06:31.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.302+0000 7fabbdd9b640 1 -- 172.21.15.125:0/3900604471 learned_addr learned my addr 172.21.15.125:0/3900604471 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:06:31.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.302+0000 7fabbe59c640 1 -- 172.21.15.125:0/3900604471 >> v1:172.21.15.136:6789/0 conn(0x7fabc00cb1e0 legacy=0x7fabc015d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:31.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.302+0000 7fabbe59c640 1 -- 172.21.15.125:0/3900604471 >> v1:172.21.15.125:6789/0 conn(0x7fabc0159930 legacy=0x7fabc00cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:31.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.302+0000 7fabbe59c640 1 -- 172.21.15.125:0/3900604471 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc015a080 con 0x7fabc0153980 2023-10-30T13:06:31.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.302+0000 7fabbd59a640 1 -- 172.21.15.125:0/3900604471 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802937713 0 0) 0x7fabc015a920 con 0x7fabc0159930 2023-10-30T13:06:31.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.306+0000 7fabbe59c640 1 --2- 172.21.15.125:0/3900604471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabc0153980 0x7fabc00caca0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fabb00097c0 tx=0x7fabb001c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800852b044ba45bf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:31.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.306+0000 7fabbd59a640 1 -- 172.21.15.125:0/3900604471 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb00ad660 con 0x7fabc0153980 2023-10-30T13:06:31.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.306+0000 7fabbd59a640 1 -- 172.21.15.125:0/3900604471 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb00ad7c0 con 0x7fabc0153980 2023-10-30T13:06:31.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.306+0000 7fabbd59a640 1 -- 172.21.15.125:0/3900604471 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb00adb50 con 0x7fabc0153980 2023-10-30T13:06:31.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.306+0000 7fabc4c46640 1 -- 172.21.15.125:0/3900604471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabc0153980 msgr2=0x7fabc00caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:31.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.306+0000 7fabc4c46640 1 --2- 172.21.15.125:0/3900604471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabc0153980 0x7fabc00caca0 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fabb00097c0 tx=0x7fabb001c450 comp rx=0 tx=0).stop 2023-10-30T13:06:31.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.306+0000 7fabc4c46640 1 -- 172.21.15.125:0/3900604471 shutdown_connections 2023-10-30T13:06:31.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.306+0000 7fabc4c46640 1 --2- 172.21.15.125:0/3900604471 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabc0153980 0x7fabc00caca0 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.306+0000 7fabc4c46640 1 -- 172.21.15.125:0/3900604471 >> 172.21.15.125:0/3900604471 conn(0x7fabc006c0f0 msgr2=0x7fabc006c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:31.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.306+0000 7fabc4c46640 1 -- 172.21.15.125:0/3900604471 shutdown_connections 2023-10-30T13:06:31.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.306+0000 7fabc4c46640 1 -- 172.21.15.125:0/3900604471 wait complete. 2023-10-30T13:06:31.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabc4c46640 1 Processor -- start 2023-10-30T13:06:31.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabc4c46640 1 -- start start 2023-10-30T13:06:31.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabc4c46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fabc00cb1e0 0x7fabc0154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:31.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabc4c46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabc0153980 0x7fabc0156b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:31.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabc4c46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabc00cf400 0x7fabc01f4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:31.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabc4c46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc0156650 con 0x7fabc0153980 2023-10-30T13:06:31.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabc4c46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc01543a0 con 0x7fabc00cb1e0 2023-10-30T13:06:31.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabc4c46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc006c980 con 0x7fabc00cf400 2023-10-30T13:06:31.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabbdd9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabc0153980 0x7fabc0156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:31.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabbdd9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabc0153980 0x7fabc0156b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45184/0 (socket says 172.21.15.125:45184) 2023-10-30T13:06:31.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabbdd9b640 1 -- 172.21.15.125:0/2707369515 learned_addr learned my addr 172.21.15.125:0/2707369515 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:31.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabbed9d640 1 --2- 172.21.15.125:0/2707369515 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabc00cf400 0x7fabc01f4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:31.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabbe59c640 1 --2- 172.21.15.125:0/2707369515 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fabc00cb1e0 0x7fabc0154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:31.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabbdd9b640 1 -- 172.21.15.125:0/2707369515 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabc00cf400 msgr2=0x7fabc01f4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:31.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabbdd9b640 1 --2- 172.21.15.125:0/2707369515 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabc00cf400 0x7fabc01f4fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabbdd9b640 1 -- 172.21.15.125:0/2707369515 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fabc00cb1e0 msgr2=0x7fabc0154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:31.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabbdd9b640 1 --2- 172.21.15.125:0/2707369515 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fabc00cb1e0 0x7fabc0154e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.310+0000 7fabbdd9b640 1 -- 172.21.15.125:0/2707369515 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb0009420 con 0x7fabc0153980 2023-10-30T13:06:31.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.314+0000 7fabbdd9b640 1 --2- 172.21.15.125:0/2707369515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabc0153980 0x7fabc0156b90 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fabb400c4a0 tx=0x7fabb400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:31.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.314+0000 7fabaeffd640 1 -- 172.21.15.125:0/2707369515 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb40074e0 con 0x7fabc0153980 2023-10-30T13:06:31.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.314+0000 7fabaeffd640 1 -- 172.21.15.125:0/2707369515 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb4007640 con 0x7fabc0153980 2023-10-30T13:06:31.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.314+0000 7fabaeffd640 1 -- 172.21.15.125:0/2707369515 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4004a30 con 0x7fabc0153980 2023-10-30T13:06:31.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.314+0000 7fabc4c46640 1 -- 172.21.15.125:0/2707369515 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc01577f0 con 0x7fabc0153980 2023-10-30T13:06:31.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.314+0000 7fabc4c46640 1 -- 172.21.15.125:0/2707369515 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc0154640 con 0x7fabc0153980 2023-10-30T13:06:31.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.314+0000 7fabc4c46640 1 -- 172.21.15.125:0/2707369515 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabc0155380 con 0x7fabc0153980 2023-10-30T13:06:31.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.314+0000 7fabaeffd640 1 -- 172.21.15.125:0/2707369515 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fabb4004b90 con 0x7fabc0153980 2023-10-30T13:06:31.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.314+0000 7fabaeffd640 1 --2- 172.21.15.125:0/2707369515 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faba00664a0 0x7faba0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:31.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.314+0000 7fabaeffd640 1 -- 172.21.15.125:0/2707369515 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7fabb4083750 con 0x7fabc0153980 2023-10-30T13:06:31.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.314+0000 7fabbe59c640 1 --2- 172.21.15.125:0/2707369515 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faba00664a0 0x7faba0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:31.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.318+0000 7fabbe59c640 1 --2- 172.21.15.125:0/2707369515 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faba00664a0 0x7faba0068960 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fabb00af000 tx=0x7fabb00a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:31.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.318+0000 7fabaeffd640 1 -- 172.21.15.125:0/2707369515 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fabb4050590 con 0x7fabc0153980 2023-10-30T13:06:31.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.518+0000 7fabc4c46640 1 -- 172.21.15.125:0/2707369515 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7fabc0154b60 con 0x7fabc0153980 2023-10-30T13:06:31.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.606+0000 7fabaeffd640 1 -- 172.21.15.125:0/2707369515 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v167) v1 ==== 105+0+0 (secure 0 0 0) 0x7fabc0154b60 con 0x7fabc0153980 2023-10-30T13:06:31.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.610+0000 7fabc4c46640 1 -- 172.21.15.125:0/2707369515 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faba00664a0 msgr2=0x7faba0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:31.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.610+0000 7fabc4c46640 1 --2- 172.21.15.125:0/2707369515 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faba00664a0 0x7faba0068960 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fabb00af000 tx=0x7fabb00a7000 comp rx=0 tx=0).stop 2023-10-30T13:06:31.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.610+0000 7fabc4c46640 1 -- 172.21.15.125:0/2707369515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabc0153980 msgr2=0x7fabc0156b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:31.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.610+0000 7fabc4c46640 1 --2- 172.21.15.125:0/2707369515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabc0153980 0x7fabc0156b90 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fabb400c4a0 tx=0x7fabb400c960 comp rx=0 tx=0).stop 2023-10-30T13:06:31.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.610+0000 7fabc4c46640 1 -- 172.21.15.125:0/2707369515 shutdown_connections 2023-10-30T13:06:31.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.610+0000 7fabc4c46640 1 --2- 172.21.15.125:0/2707369515 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faba00664a0 0x7faba0068960 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.610+0000 7fabc4c46640 1 --2- 172.21.15.125:0/2707369515 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fabc00cf400 0x7fabc01f4fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.610+0000 7fabc4c46640 1 --2- 172.21.15.125:0/2707369515 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fabc0153980 0x7fabc0156b90 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.610+0000 7fabc4c46640 1 --2- 172.21.15.125:0/2707369515 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fabc00cb1e0 0x7fabc0154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.610+0000 7fabc4c46640 1 -- 172.21.15.125:0/2707369515 >> 172.21.15.125:0/2707369515 conn(0x7fabc006c0f0 msgr2=0x7fabc01581c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:31.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.610+0000 7fabc4c46640 1 -- 172.21.15.125:0/2707369515 shutdown_connections 2023-10-30T13:06:31.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.610+0000 7fabc4c46640 1 -- 172.21.15.125:0/2707369515 wait complete. 2023-10-30T13:06:31.654 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:06:31.655 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:31.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.842+0000 7fd267d94640 1 Processor -- start 2023-10-30T13:06:31.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.842+0000 7fd267d94640 1 -- start start 2023-10-30T13:06:31.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd267d94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2600c6bd0 0x7fd26006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:31.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd267d94640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd260071110 0x7fd260158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:31.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd267d94640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd260066cf0 con 0x7fd26006f7a0 2023-10-30T13:06:31.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd267d94640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2600700c0 con 0x7fd260071110 2023-10-30T13:06:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd267d94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2600c70d0 con 0x7fd2600c6bd0 2023-10-30T13:06:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd265b30640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd26006f7a0 0x7fd260069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46834/0 (socket says 172.21.15.125:46834) 2023-10-30T13:06:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd265b30640 1 -- 172.21.15.125:0/3954994638 learned_addr learned my addr 172.21.15.125:0/3954994638 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:06:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd266331640 1 --2- 172.21.15.125:0/3954994638 >> v2:172.21.15.136:3300/0 conn(0x7fd260071110 0x7fd260158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:31.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd26532f640 1 --2- 172.21.15.125:0/3954994638 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2600c6bd0 0x7fd26006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:31.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd264b2e640 1 -- 172.21.15.125:0/3954994638 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1409550827 0 0) 0x7fd260066cf0 con 0x7fd26006f7a0 2023-10-30T13:06:31.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd264b2e640 1 -- 172.21.15.125:0/3954994638 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd24c003530 con 0x7fd26006f7a0 2023-10-30T13:06:31.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd266331640 1 -- 172.21.15.125:0/3954994638 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2600c6bd0 msgr2=0x7fd26006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:31.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd266331640 1 --2- 172.21.15.125:0/3954994638 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2600c6bd0 0x7fd26006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd266331640 1 -- 172.21.15.125:0/3954994638 >> v1:172.21.15.125:6789/0 conn(0x7fd26006f7a0 legacy=0x7fd260069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:31.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd264b2e640 1 -- 172.21.15.125:0/3954994638 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1777005773 0 0) 0x7fd24c003530 con 0x7fd26006f7a0 2023-10-30T13:06:31.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd266331640 1 -- 172.21.15.125:0/3954994638 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2600c7390 con 0x7fd260071110 2023-10-30T13:06:31.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.846+0000 7fd26532f640 1 --2- 172.21.15.125:0/3954994638 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2600c6bd0 0x7fd26006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:06:31.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.850+0000 7fd266331640 1 --2- 172.21.15.125:0/3954994638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd260071110 0x7fd260158110 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd258009d90 tx=0x7fd25801f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be22175767cf5603 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:31.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.850+0000 7fd264b2e640 1 -- 172.21.15.125:0/3954994638 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25c001680 con 0x7fd260071110 2023-10-30T13:06:31.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.850+0000 7fd264b2e640 1 -- 172.21.15.125:0/3954994638 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2580a8bb0 con 0x7fd260071110 2023-10-30T13:06:31.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.850+0000 7fd264b2e640 1 -- 172.21.15.125:0/3954994638 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2580b0440 con 0x7fd260071110 2023-10-30T13:06:31.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.850+0000 7fd267d94640 1 -- 172.21.15.125:0/3954994638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd260071110 msgr2=0x7fd260158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:31.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.850+0000 7fd267d94640 1 --2- 172.21.15.125:0/3954994638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd260071110 0x7fd260158110 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd258009d90 tx=0x7fd25801f060 comp rx=0 tx=0).stop 2023-10-30T13:06:31.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.850+0000 7fd267d94640 1 -- 172.21.15.125:0/3954994638 shutdown_connections 2023-10-30T13:06:31.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.850+0000 7fd267d94640 1 --2- 172.21.15.125:0/3954994638 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd260071110 0x7fd260158110 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.850+0000 7fd267d94640 1 --2- 172.21.15.125:0/3954994638 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2600c6bd0 0x7fd26006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.850+0000 7fd267d94640 1 -- 172.21.15.125:0/3954994638 >> 172.21.15.125:0/3954994638 conn(0x7fd2600c7910 msgr2=0x7fd2600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:31.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.850+0000 7fd267d94640 1 -- 172.21.15.125:0/3954994638 shutdown_connections 2023-10-30T13:06:31.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.850+0000 7fd267d94640 1 -- 172.21.15.125:0/3954994638 wait complete. 2023-10-30T13:06:31.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd267d94640 1 Processor -- start 2023-10-30T13:06:31.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd267d94640 1 -- start start 2023-10-30T13:06:31.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd267d94640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd26006f7a0 0x7fd260157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:31.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd267d94640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd260071110 0x7fd26014fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:31.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd267d94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2600c6bd0 0x7fd2601f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:31.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd267d94640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd260155f90 con 0x7fd260071110 2023-10-30T13:06:31.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd267d94640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2601503d0 con 0x7fd26006f7a0 2023-10-30T13:06:31.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd267d94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd260157760 con 0x7fd2600c6bd0 2023-10-30T13:06:31.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd26532f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd260071110 0x7fd26014fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:31.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd26532f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd260071110 0x7fd26014fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45196/0 (socket says 172.21.15.125:45196) 2023-10-30T13:06:31.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd26532f640 1 -- 172.21.15.125:0/1413806476 learned_addr learned my addr 172.21.15.125:0/1413806476 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:31.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd265b30640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd26006f7a0 0x7fd260157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:31.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd266331640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2600c6bd0 0x7fd2601f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:31.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd26532f640 1 -- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2600c6bd0 msgr2=0x7fd2601f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:31.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd26532f640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2600c6bd0 0x7fd2601f2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd26532f640 1 -- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd26006f7a0 msgr2=0x7fd260157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:31.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd26532f640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd26006f7a0 0x7fd260157050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:31.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd26532f640 1 -- 172.21.15.125:0/1413806476 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd250009420 con 0x7fd260071110 2023-10-30T13:06:31.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd266331640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2600c6bd0 0x7fd2601f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:31.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.854+0000 7fd26532f640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd260071110 0x7fd26014fe90 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fd25000ebe0 tx=0x7fd25000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:31.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.858+0000 7fd2567fc640 1 -- 172.21.15.125:0/1413806476 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd25000eee0 con 0x7fd260071110 2023-10-30T13:06:31.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.858+0000 7fd2567fc640 1 -- 172.21.15.125:0/1413806476 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd25000cc30 con 0x7fd260071110 2023-10-30T13:06:31.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.858+0000 7fd267d94640 1 -- 172.21.15.125:0/1413806476 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd258009a70 con 0x7fd260071110 2023-10-30T13:06:31.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.858+0000 7fd2567fc640 1 -- 172.21.15.125:0/1413806476 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd250013470 con 0x7fd260071110 2023-10-30T13:06:31.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.858+0000 7fd267d94640 1 -- 172.21.15.125:0/1413806476 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2601558d0 con 0x7fd260071110 2023-10-30T13:06:31.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.858+0000 7fd267d94640 1 -- 172.21.15.125:0/1413806476 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26004e170 con 0x7fd260071110 2023-10-30T13:06:31.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.858+0000 7fd2567fc640 1 -- 172.21.15.125:0/1413806476 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd2500135d0 con 0x7fd260071110 2023-10-30T13:06:31.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.858+0000 7fd2567fc640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2440664a0 0x7fd244068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:31.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.862+0000 7fd2567fc640 1 -- 172.21.15.125:0/1413806476 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7fd250084840 con 0x7fd260071110 2023-10-30T13:06:31.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.862+0000 7fd265b30640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2440664a0 0x7fd244068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:31.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.862+0000 7fd265b30640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2440664a0 0x7fd244068960 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0011f0 tx=0x7fd25c0016e0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:31.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:31.862+0000 7fd2567fc640 1 -- 172.21.15.125:0/1413806476 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd2500515f0 con 0x7fd260071110 2023-10-30T13:06:32.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:32.046+0000 7fd267d94640 1 -- 172.21.15.125:0/1413806476 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2601f30a0 con 0x7fd260071110 2023-10-30T13:06:32.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:32.046+0000 7fd2567fc640 1 -- 172.21.15.125:0/1413806476 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+6533 (secure 0 0 0) 0x7fd250054c10 con 0x7fd260071110 2023-10-30T13:06:32.049 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:32.050 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":168,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:31.614807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":8178,"mds_1":8301},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:starting","state_seq":1,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:32.051 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 168 2023-10-30T13:06:32.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:32.046+0000 7fd267d94640 1 -- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2440664a0 msgr2=0x7fd244068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:32.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:32.046+0000 7fd267d94640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2440664a0 0x7fd244068960 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fd25c0011f0 tx=0x7fd25c0016e0 comp rx=0 tx=0).stop 2023-10-30T13:06:32.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:32.046+0000 7fd267d94640 1 -- 172.21.15.125:0/1413806476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd260071110 msgr2=0x7fd26014fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:32.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:32.046+0000 7fd267d94640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd260071110 0x7fd26014fe90 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fd25000ebe0 tx=0x7fd25000c520 comp rx=0 tx=0).stop 2023-10-30T13:06:32.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:32.046+0000 7fd267d94640 1 -- 172.21.15.125:0/1413806476 shutdown_connections 2023-10-30T13:06:32.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:32.046+0000 7fd267d94640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2440664a0 0x7fd244068960 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:32.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:32.046+0000 7fd267d94640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2600c6bd0 0x7fd2601f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:32.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:32.046+0000 7fd267d94640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd260071110 0x7fd26014fe90 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:32.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:32.046+0000 7fd267d94640 1 --2- 172.21.15.125:0/1413806476 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd26006f7a0 0x7fd260157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:32.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:32.046+0000 7fd267d94640 1 -- 172.21.15.125:0/1413806476 >> 172.21.15.125:0/1413806476 conn(0x7fd2600c7910 msgr2=0x7fd2600c8210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:32.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:32.046+0000 7fd267d94640 1 -- 172.21.15.125:0/1413806476 shutdown_connections 2023-10-30T13:06:32.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:32.046+0000 7fd267d94640 1 -- 172.21.15.125:0/1413806476 wait complete. 2023-10-30T13:06:32.070 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 168, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 168, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1 ], "info": { "gid_8178": { "addr": "172.21.15.125:6836/3711687505", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3711687505, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3711687505, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8178, "incarnation": 162, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 15 }, "gid_8301": { "addr": "172.21.15.125:6837/2540063352", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2540063352, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2540063352, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8301, "incarnation": 168, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:starting", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:06:31.614807+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [ 2 ], "tableserver": 0, "up": { "mds_0": 8178, "mds_1": 8301 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 60, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 0, "join_fscid": -1, "name": "d", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 111, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 0, "join_fscid": -1, "name": "f", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:06:34.071 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:34.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.206+0000 7fcc8569a640 1 Processor -- start 2023-10-30T13:06:34.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.206+0000 7fcc8569a640 1 -- start start 2023-10-30T13:06:34.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc8569a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc8015d6f0 0x7fcc8015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:34.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc8569a640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc800cb800 con 0x7fcc801538a0 2023-10-30T13:06:34.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc8569a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc800cb330 con 0x7fcc800cf160 2023-10-30T13:06:34.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc8569a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc800cad70 con 0x7fcc8015d6f0 2023-10-30T13:06:34.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7effd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcc801538a0 0x7fcc800ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46838/0 (socket says 172.21.15.125:46838) 2023-10-30T13:06:34.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7effd640 1 -- 172.21.15.125:0/1257008359 learned_addr learned my addr 172.21.15.125:0/1257008359 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:06:34.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc767fc640 1 --2- 172.21.15.125:0/1257008359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc8015d6f0 0x7fcc8015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:34.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 -- 172.21.15.125:0/1257008359 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102530495 0 0) 0x7fcc800cb800 con 0x7fcc801538a0 2023-10-30T13:06:34.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 -- 172.21.15.125:0/1257008359 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc60003530 con 0x7fcc801538a0 2023-10-30T13:06:34.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 -- 172.21.15.125:0/1257008359 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1860985722 0 0) 0x7fcc60003530 con 0x7fcc801538a0 2023-10-30T13:06:34.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 -- 172.21.15.125:0/1257008359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc8015d6f0 msgr2=0x7fcc8015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:34.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 --2- 172.21.15.125:0/1257008359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc8015d6f0 0x7fcc8015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:34.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 -- 172.21.15.125:0/1257008359 >> v1:172.21.15.136:6789/0 conn(0x7fcc800cf160 legacy=0x7fcc80164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:34.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 -- 172.21.15.125:0/1257008359 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc800cb050 con 0x7fcc801538a0 2023-10-30T13:06:34.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 -- 172.21.15.125:0/1257008359 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fcc70001680 con 0x7fcc801538a0 2023-10-30T13:06:34.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 -- 172.21.15.125:0/1257008359 >> v1:172.21.15.125:6789/0 conn(0x7fcc801538a0 legacy=0x7fcc800ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:34.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 --2- 172.21.15.125:0/1257008359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc8015d6f0 0x7fcc600039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:34.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 --2- 172.21.15.125:0/1257008359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc60003f30 0x7fcc600047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:34.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 --2- 172.21.15.125:0/1257008359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc60004d10 0x7fcc60005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:34.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 -- 172.21.15.125:0/1257008359 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc800cad70 con 0x7fcc60003f30 2023-10-30T13:06:34.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 -- 172.21.15.125:0/1257008359 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6000bb00 con 0x7fcc60004d10 2023-10-30T13:06:34.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7e7fc640 1 -- 172.21.15.125:0/1257008359 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6000bcc0 con 0x7fcc8015d6f0 2023-10-30T13:06:34.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc767fc640 1 --2- 172.21.15.125:0/1257008359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc60003f30 0x7fcc600047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:34.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7effd640 1 --2- 172.21.15.125:0/1257008359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc8015d6f0 0x7fcc600039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:34.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7f7fe640 1 --2- 172.21.15.125:0/1257008359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc60004d10 0x7fcc60005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:34.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc767fc640 1 -- 172.21.15.125:0/1257008359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc8015d6f0 msgr2=0x7fcc600039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:34.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc767fc640 1 --2- 172.21.15.125:0/1257008359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc8015d6f0 0x7fcc600039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:34.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc767fc640 1 -- 172.21.15.125:0/1257008359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc60004d10 msgr2=0x7fcc60005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:34.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc767fc640 1 --2- 172.21.15.125:0/1257008359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc60004d10 0x7fcc60005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:34.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc767fc640 1 -- 172.21.15.125:0/1257008359 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc60003530 con 0x7fcc60003f30 2023-10-30T13:06:34.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7f7fe640 1 --2- 172.21.15.125:0/1257008359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc60004d10 0x7fcc60005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:06:34.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.210+0000 7fcc7effd640 1 --2- 172.21.15.125:0/1257008359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc8015d6f0 0x7fcc600039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:34.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.214+0000 7fcc767fc640 1 --2- 172.21.15.125:0/1257008359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc60003f30 0x7fcc600047d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009ae0 tx=0x7fcc6801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9002e1211a0b55cd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:34.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.214+0000 7fcc7e7fc640 1 -- 172.21.15.125:0/1257008359 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc680a5640 con 0x7fcc60003f30 2023-10-30T13:06:34.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.214+0000 7fcc7e7fc640 1 -- 172.21.15.125:0/1257008359 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc680a57a0 con 0x7fcc60003f30 2023-10-30T13:06:34.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.214+0000 7fcc7e7fc640 1 -- 172.21.15.125:0/1257008359 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc680a5b50 con 0x7fcc60003f30 2023-10-30T13:06:34.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.214+0000 7fcc8569a640 1 -- 172.21.15.125:0/1257008359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc60003f30 msgr2=0x7fcc600047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:34.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.214+0000 7fcc8569a640 1 --2- 172.21.15.125:0/1257008359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc60003f30 0x7fcc600047d0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fcc68009ae0 tx=0x7fcc6801ca50 comp rx=0 tx=0).stop 2023-10-30T13:06:34.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.214+0000 7fcc7f7fe640 1 -- 172.21.15.125:0/1257008359 reap_dead start 2023-10-30T13:06:34.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.214+0000 7fcc8569a640 1 -- 172.21.15.125:0/1257008359 shutdown_connections 2023-10-30T13:06:34.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.214+0000 7fcc8569a640 1 -- 172.21.15.125:0/1257008359 >> 172.21.15.125:0/1257008359 conn(0x7fcc8006bfe0 msgr2=0x7fcc8006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:34.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.214+0000 7fcc8569a640 1 -- 172.21.15.125:0/1257008359 shutdown_connections 2023-10-30T13:06:34.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.214+0000 7fcc8569a640 1 -- 172.21.15.125:0/1257008359 wait complete. 2023-10-30T13:06:34.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.214+0000 7fcc8569a640 1 Processor -- start 2023-10-30T13:06:34.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc8569a640 1 -- start start 2023-10-30T13:06:34.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc8569a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc60003f30 0x7fcc801f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:34.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc8569a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc801ef0d0 0x7fcc8014f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:34.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc8569a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc8014d0f0 0x7fcc800b4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:34.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc8569a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc801f26e0 con 0x7fcc60003f30 2023-10-30T13:06:34.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc8569a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc801e2f90 con 0x7fcc8014d0f0 2023-10-30T13:06:34.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc8569a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc801ef530 con 0x7fcc801ef0d0 2023-10-30T13:06:34.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc7effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc60003f30 0x7fcc801f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:34.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc7effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc60003f30 0x7fcc801f3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45218/0 (socket says 172.21.15.125:45218) 2023-10-30T13:06:34.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc7effd640 1 -- 172.21.15.125:0/3504329991 learned_addr learned my addr 172.21.15.125:0/3504329991 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:34.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc767fc640 1 --2- 172.21.15.125:0/3504329991 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc801ef0d0 0x7fcc8014f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:34.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc7f7fe640 1 --2- 172.21.15.125:0/3504329991 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc8014d0f0 0x7fcc800b4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:34.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc7effd640 1 -- 172.21.15.125:0/3504329991 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc801ef0d0 msgr2=0x7fcc8014f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:34.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc7effd640 1 --2- 172.21.15.125:0/3504329991 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc801ef0d0 0x7fcc8014f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:34.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc7effd640 1 -- 172.21.15.125:0/3504329991 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc8014d0f0 msgr2=0x7fcc800b4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:34.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc7effd640 1 --2- 172.21.15.125:0/3504329991 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc8014d0f0 0x7fcc800b4830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:34.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc7effd640 1 -- 172.21.15.125:0/3504329991 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc800c72a0 con 0x7fcc60003f30 2023-10-30T13:06:34.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc7effd640 1 --2- 172.21.15.125:0/3504329991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc60003f30 0x7fcc801f3180 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fcc70001bc0 tx=0x7fcc7000ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:34.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc77fff640 1 -- 172.21.15.125:0/3504329991 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7000c8d0 con 0x7fcc60003f30 2023-10-30T13:06:34.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc77fff640 1 -- 172.21.15.125:0/3504329991 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7000ca30 con 0x7fcc60003f30 2023-10-30T13:06:34.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc77fff640 1 -- 172.21.15.125:0/3504329991 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7000cda0 con 0x7fcc60003f30 2023-10-30T13:06:34.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc8569a640 1 -- 172.21.15.125:0/3504329991 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc800c6ce0 con 0x7fcc60003f30 2023-10-30T13:06:34.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc8569a640 1 -- 172.21.15.125:0/3504329991 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc801f2980 con 0x7fcc60003f30 2023-10-30T13:06:34.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.218+0000 7fcc8569a640 1 -- 172.21.15.125:0/3504329991 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc8004e170 con 0x7fcc60003f30 2023-10-30T13:06:34.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.222+0000 7fcc77fff640 1 -- 172.21.15.125:0/3504329991 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcc7001d460 con 0x7fcc60003f30 2023-10-30T13:06:34.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.222+0000 7fcc77fff640 1 --2- 172.21.15.125:0/3504329991 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc640664a0 0x7fcc64068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:34.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.222+0000 7fcc77fff640 1 -- 172.21.15.125:0/3504329991 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7fcc700846e0 con 0x7fcc60003f30 2023-10-30T13:06:34.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.222+0000 7fcc767fc640 1 --2- 172.21.15.125:0/3504329991 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc640664a0 0x7fcc64068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:34.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.222+0000 7fcc767fc640 1 --2- 172.21.15.125:0/3504329991 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc640664a0 0x7fcc64068960 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fcc680af000 tx=0x7fcc680046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:34.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.226+0000 7fcc77fff640 1 -- 172.21.15.125:0/3504329991 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcc70051350 con 0x7fcc60003f30 2023-10-30T13:06:34.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.490+0000 7fcc8569a640 1 -- 172.21.15.125:0/3504329991 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc800b1dd0 con 0x7fcc60003f30 2023-10-30T13:06:34.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.490+0000 7fcc77fff640 1 -- 172.21.15.125:0/3504329991 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+6533 (secure 0 0 0) 0x7fcc70054970 con 0x7fcc60003f30 2023-10-30T13:06:34.494 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:34.494 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":168,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:31.614807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":8178,"mds_1":8301},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:starting","state_seq":1,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:34.494 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 168 2023-10-30T13:06:34.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.490+0000 7fcc8569a640 1 -- 172.21.15.125:0/3504329991 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc640664a0 msgr2=0x7fcc64068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:34.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.490+0000 7fcc8569a640 1 --2- 172.21.15.125:0/3504329991 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc640664a0 0x7fcc64068960 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fcc680af000 tx=0x7fcc680046d0 comp rx=0 tx=0).stop 2023-10-30T13:06:34.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.490+0000 7fcc8569a640 1 -- 172.21.15.125:0/3504329991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc60003f30 msgr2=0x7fcc801f3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:34.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.490+0000 7fcc8569a640 1 --2- 172.21.15.125:0/3504329991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc60003f30 0x7fcc801f3180 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fcc70001bc0 tx=0x7fcc7000ed50 comp rx=0 tx=0).stop 2023-10-30T13:06:34.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.494+0000 7fcc8569a640 1 -- 172.21.15.125:0/3504329991 shutdown_connections 2023-10-30T13:06:34.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.494+0000 7fcc8569a640 1 --2- 172.21.15.125:0/3504329991 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc640664a0 0x7fcc64068960 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:34.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.494+0000 7fcc8569a640 1 --2- 172.21.15.125:0/3504329991 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc8014d0f0 0x7fcc800b4830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:34.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.494+0000 7fcc8569a640 1 --2- 172.21.15.125:0/3504329991 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc801ef0d0 0x7fcc8014f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:34.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.494+0000 7fcc8569a640 1 --2- 172.21.15.125:0/3504329991 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc60003f30 0x7fcc801f3180 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:34.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.494+0000 7fcc8569a640 1 -- 172.21.15.125:0/3504329991 >> 172.21.15.125:0/3504329991 conn(0x7fcc8006bfe0 msgr2=0x7fcc80065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:34.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.494+0000 7fcc8569a640 1 -- 172.21.15.125:0/3504329991 shutdown_connections 2023-10-30T13:06:34.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:34.494+0000 7fcc8569a640 1 -- 172.21.15.125:0/3504329991 wait complete. 2023-10-30T13:06:34.525 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:06:36.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:36.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.698+0000 7f153da46640 1 Processor -- start 2023-10-30T13:06:36.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.698+0000 7f153da46640 1 -- start start 2023-10-30T13:06:36.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.702+0000 7f153da46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15381538a0 0x7f153815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:36.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.702+0000 7f153da46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15380cf160 0x7f15380cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:36.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.702+0000 7f153da46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15380cb800 con 0x7f15380ca0e0 2023-10-30T13:06:36.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.702+0000 7f153da46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153815dc30 con 0x7f15381538a0 2023-10-30T13:06:36.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.702+0000 7f153da46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15380cb330 con 0x7f15380cf160 2023-10-30T13:06:36.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.702+0000 7f1536ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15381538a0 0x7f153815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:36.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.702+0000 7f1536ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15381538a0 0x7f153815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45228/0 (socket says 172.21.15.125:45228) 2023-10-30T13:06:36.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.702+0000 7f1536ffd640 1 -- 172.21.15.125:0/4262333064 learned_addr learned my addr 172.21.15.125:0/4262333064 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:36.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.702+0000 7f15377fe640 1 --2- 172.21.15.125:0/4262333064 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15380cf160 0x7f15380cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:36.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.702+0000 7f1536ffd640 1 -- 172.21.15.125:0/4262333064 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15380cf160 msgr2=0x7f15380cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:36.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.702+0000 7f1536ffd640 1 --2- 172.21.15.125:0/4262333064 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15380cf160 0x7f15380cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:36.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.702+0000 7f1536ffd640 1 -- 172.21.15.125:0/4262333064 >> v1:172.21.15.136:6789/0 conn(0x7f15380ca0e0 legacy=0x7f153815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:36.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.702+0000 7f1536ffd640 1 -- 172.21.15.125:0/4262333064 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15380cad70 con 0x7f15381538a0 2023-10-30T13:06:36.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.706+0000 7f1536ffd640 1 --2- 172.21.15.125:0/4262333064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15381538a0 0x7f153815d6f0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f1528009870 tx=0x7f152801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fadd1fb9aba373 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:36.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.706+0000 7f1535ffb640 1 -- 172.21.15.125:0/4262333064 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15280a8a30 con 0x7f15381538a0 2023-10-30T13:06:36.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.706+0000 7f1535ffb640 1 -- 172.21.15.125:0/4262333064 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15280a8b90 con 0x7f15381538a0 2023-10-30T13:06:36.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.706+0000 7f1535ffb640 1 -- 172.21.15.125:0/4262333064 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15280b0440 con 0x7f15381538a0 2023-10-30T13:06:36.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.706+0000 7f153da46640 1 -- 172.21.15.125:0/4262333064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15381538a0 msgr2=0x7f153815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:36.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.706+0000 7f153da46640 1 --2- 172.21.15.125:0/4262333064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15381538a0 0x7f153815d6f0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f1528009870 tx=0x7f152801ed30 comp rx=0 tx=0).stop 2023-10-30T13:06:36.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.706+0000 7f153da46640 1 -- 172.21.15.125:0/4262333064 shutdown_connections 2023-10-30T13:06:36.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.706+0000 7f153da46640 1 --2- 172.21.15.125:0/4262333064 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15380cf160 0x7f15380cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:36.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.706+0000 7f153da46640 1 --2- 172.21.15.125:0/4262333064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15381538a0 0x7f153815d6f0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:36.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.706+0000 7f153da46640 1 -- 172.21.15.125:0/4262333064 >> 172.21.15.125:0/4262333064 conn(0x7f153806bfe0 msgr2=0x7f153806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:36.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.706+0000 7f153da46640 1 -- 172.21.15.125:0/4262333064 shutdown_connections 2023-10-30T13:06:36.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.706+0000 7f153da46640 1 -- 172.21.15.125:0/4262333064 wait complete. 2023-10-30T13:06:36.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f153da46640 1 Processor -- start 2023-10-30T13:06:36.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f153da46640 1 -- start start 2023-10-30T13:06:36.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f153da46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15380ca0e0 0x7f15381e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:36.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f153da46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15380cf160 0x7f153815b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:36.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f153da46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15381538a0 0x7f153815bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:36.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f153da46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153814d4f0 con 0x7f15380ca0e0 2023-10-30T13:06:36.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f153da46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15381e4d40 con 0x7f15380cf160 2023-10-30T13:06:36.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f153da46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153815b2a0 con 0x7f15381538a0 2023-10-30T13:06:36.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f1536ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15380ca0e0 0x7f15381e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:36.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f1536ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15380ca0e0 0x7f15381e4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45238/0 (socket says 172.21.15.125:45238) 2023-10-30T13:06:36.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f1536ffd640 1 -- 172.21.15.125:0/2226034362 learned_addr learned my addr 172.21.15.125:0/2226034362 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:36.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f15367fc640 1 --2- 172.21.15.125:0/2226034362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15380cf160 0x7f153815b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:36.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f15377fe640 1 --2- 172.21.15.125:0/2226034362 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15381538a0 0x7f153815bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:36.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f1536ffd640 1 -- 172.21.15.125:0/2226034362 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15381538a0 msgr2=0x7f153815bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:36.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f1536ffd640 1 --2- 172.21.15.125:0/2226034362 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15381538a0 0x7f153815bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:36.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f1536ffd640 1 -- 172.21.15.125:0/2226034362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15380cf160 msgr2=0x7f153815b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:36.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f1536ffd640 1 --2- 172.21.15.125:0/2226034362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15380cf160 0x7f153815b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:36.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f1536ffd640 1 -- 172.21.15.125:0/2226034362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15280094d0 con 0x7f15380ca0e0 2023-10-30T13:06:36.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f1536ffd640 1 --2- 172.21.15.125:0/2226034362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15380ca0e0 0x7f15381e4090 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f15280b3040 tx=0x7f152801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:36.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f151ffff640 1 -- 172.21.15.125:0/2226034362 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15280ba600 con 0x7f15380ca0e0 2023-10-30T13:06:36.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f151ffff640 1 -- 172.21.15.125:0/2226034362 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15280ba780 con 0x7f15380ca0e0 2023-10-30T13:06:36.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f153da46640 1 -- 172.21.15.125:0/2226034362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15380cef10 con 0x7f15380ca0e0 2023-10-30T13:06:36.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f151ffff640 1 -- 172.21.15.125:0/2226034362 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15280bab30 con 0x7f15380ca0e0 2023-10-30T13:06:36.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.710+0000 7f153da46640 1 -- 172.21.15.125:0/2226034362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15380c50b0 con 0x7f15380ca0e0 2023-10-30T13:06:36.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.714+0000 7f153da46640 1 -- 172.21.15.125:0/2226034362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15380b2a50 con 0x7f15380ca0e0 2023-10-30T13:06:36.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.714+0000 7f151ffff640 1 -- 172.21.15.125:0/2226034362 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f15280bac90 con 0x7f15380ca0e0 2023-10-30T13:06:36.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.714+0000 7f151ffff640 1 --2- 172.21.15.125:0/2226034362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15240664a0 0x7f1524068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:36.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.714+0000 7f151ffff640 1 -- 172.21.15.125:0/2226034362 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f15281229d0 con 0x7f15380ca0e0 2023-10-30T13:06:36.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.714+0000 7f15367fc640 1 --2- 172.21.15.125:0/2226034362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15240664a0 0x7f1524068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:36.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.718+0000 7f15367fc640 1 --2- 172.21.15.125:0/2226034362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15240664a0 0x7f1524068960 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f15380c4660 tx=0x7f152c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:36.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.722+0000 7f151ffff640 1 -- 172.21.15.125:0/2226034362 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f15280ef780 con 0x7f15380ca0e0 2023-10-30T13:06:36.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.906+0000 7f153da46640 1 -- 172.21.15.125:0/2226034362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1538151940 con 0x7f15380ca0e0 2023-10-30T13:06:36.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.906+0000 7f151ffff640 1 -- 172.21.15.125:0/2226034362 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+6533 (secure 0 0 0) 0x7f15280b5090 con 0x7f15380ca0e0 2023-10-30T13:06:36.909 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:36.910 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":6017,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":111}],"filesystems":[{"mdsmap":{"epoch":168,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:31.614807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1],"up":{"mds_0":8178,"mds_1":8301},"failed":[],"damaged":[],"stopped":[2],"info":{"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:starting","state_seq":1,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:36.911 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 168 2023-10-30T13:06:36.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.906+0000 7f153da46640 1 -- 172.21.15.125:0/2226034362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15240664a0 msgr2=0x7f1524068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:36.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.906+0000 7f153da46640 1 --2- 172.21.15.125:0/2226034362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15240664a0 0x7f1524068960 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f15380c4660 tx=0x7f152c009040 comp rx=0 tx=0).stop 2023-10-30T13:06:36.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.906+0000 7f153da46640 1 -- 172.21.15.125:0/2226034362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15380ca0e0 msgr2=0x7f15381e4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:36.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.906+0000 7f153da46640 1 --2- 172.21.15.125:0/2226034362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15380ca0e0 0x7f15381e4090 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f15280b3040 tx=0x7f152801bad0 comp rx=0 tx=0).stop 2023-10-30T13:06:36.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.906+0000 7f153da46640 1 -- 172.21.15.125:0/2226034362 shutdown_connections 2023-10-30T13:06:36.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.906+0000 7f153da46640 1 --2- 172.21.15.125:0/2226034362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15240664a0 0x7f1524068960 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:36.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.906+0000 7f153da46640 1 --2- 172.21.15.125:0/2226034362 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15381538a0 0x7f153815bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:36.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.906+0000 7f153da46640 1 --2- 172.21.15.125:0/2226034362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15380cf160 0x7f153815b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:36.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.906+0000 7f153da46640 1 --2- 172.21.15.125:0/2226034362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15380ca0e0 0x7f15381e4090 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:36.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.906+0000 7f153da46640 1 -- 172.21.15.125:0/2226034362 >> 172.21.15.125:0/2226034362 conn(0x7f153806bfe0 msgr2=0x7f1538065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:36.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.906+0000 7f153da46640 1 -- 172.21.15.125:0/2226034362 shutdown_connections 2023-10-30T13:06:36.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:36.906+0000 7f153da46640 1 -- 172.21.15.125:0/2226034362 wait complete. 2023-10-30T13:06:36.929 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:06:38.285 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at b78644e7dee 0.80 2023-10-30T13:06:38.292 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.80 2023-10-30T13:06:38.931 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:39.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f3348780640 1 Processor -- start 2023-10-30T13:06:39.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f3348780640 1 -- start start 2023-10-30T13:06:39.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f3348780640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3340153bf0 0x7f3340071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:39.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f3348780640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f33400c6fe0 0x7f33400c7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:39.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f3348780640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33400c60e0 con 0x7f33400cb550 2023-10-30T13:06:39.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f3348780640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f334015d870 con 0x7f33400c6fe0 2023-10-30T13:06:39.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f3348780640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33400c63a0 con 0x7f3340153bf0 2023-10-30T13:06:39.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f3346d1d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f33400cb550 0x7f3340159900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46842/0 (socket says 172.21.15.125:46842) 2023-10-30T13:06:39.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f3346d1d640 1 -- 172.21.15.125:0/1718023402 learned_addr learned my addr 172.21.15.125:0/1718023402 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:06:39.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334651c640 1 --2- 172.21.15.125:0/1718023402 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3340153bf0 0x7f3340071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:39.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f3345d1b640 1 --2- 172.21.15.125:0/1718023402 >> v2:172.21.15.136:3300/0 conn(0x7f33400c6fe0 0x7f33400c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:39.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 -- 172.21.15.125:0/1718023402 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626867819 0 0) 0x7f33400c60e0 con 0x7f33400cb550 2023-10-30T13:06:39.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 -- 172.21.15.125:0/1718023402 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f332c003530 con 0x7f33400cb550 2023-10-30T13:06:39.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 -- 172.21.15.125:0/1718023402 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1281660806 0 0) 0x7f332c003530 con 0x7f33400cb550 2023-10-30T13:06:39.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 -- 172.21.15.125:0/1718023402 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3340153bf0 msgr2=0x7f3340071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:39.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 --2- 172.21.15.125:0/1718023402 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3340153bf0 0x7f3340071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:39.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 -- 172.21.15.125:0/1718023402 >> v2:172.21.15.136:3300/0 conn(0x7f33400c6fe0 msgr2=0x7f33400c7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:39.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 --2- 172.21.15.125:0/1718023402 >> v2:172.21.15.136:3300/0 conn(0x7f33400c6fe0 0x7f33400c7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:39.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 -- 172.21.15.125:0/1718023402 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33400c6680 con 0x7f33400cb550 2023-10-30T13:06:39.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 -- 172.21.15.125:0/1718023402 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f3338001a00 con 0x7f33400cb550 2023-10-30T13:06:39.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 -- 172.21.15.125:0/1718023402 >> v1:172.21.15.125:6789/0 conn(0x7f33400cb550 legacy=0x7f3340159900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:39.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 --2- 172.21.15.125:0/1718023402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c003980 0x7f332c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:39.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 --2- 172.21.15.125:0/1718023402 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3340153bf0 0x7f332c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:39.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 --2- 172.21.15.125:0/1718023402 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f332c004d10 0x7f332c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:39.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 -- 172.21.15.125:0/1718023402 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334015d870 con 0x7f332c004d10 2023-10-30T13:06:39.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 -- 172.21.15.125:0/1718023402 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33400c63a0 con 0x7f332c003980 2023-10-30T13:06:39.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334551a640 1 -- 172.21.15.125:0/1718023402 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332c00b990 con 0x7f3340153bf0 2023-10-30T13:06:39.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f3346d1d640 1 --2- 172.21.15.125:0/1718023402 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f332c004d10 0x7f332c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:39.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.106+0000 7f334651c640 1 --2- 172.21.15.125:0/1718023402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c003980 0x7f332c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:39.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.110+0000 7f3345d1b640 1 --2- 172.21.15.125:0/1718023402 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3340153bf0 0x7f332c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:39.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.110+0000 7f3346d1d640 1 -- 172.21.15.125:0/1718023402 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3340153bf0 msgr2=0x7f332c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:39.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.110+0000 7f3346d1d640 1 --2- 172.21.15.125:0/1718023402 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3340153bf0 0x7f332c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:39.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.110+0000 7f3346d1d640 1 -- 172.21.15.125:0/1718023402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c003980 msgr2=0x7f332c003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:39.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.110+0000 7f3346d1d640 1 --2- 172.21.15.125:0/1718023402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c003980 0x7f332c003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:39.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.110+0000 7f3346d1d640 1 -- 172.21.15.125:0/1718023402 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332c003530 con 0x7f332c004d10 2023-10-30T13:06:39.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.110+0000 7f334651c640 1 --2- 172.21.15.125:0/1718023402 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f332c003980 0x7f332c003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:39.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.110+0000 7f3346d1d640 1 --2- 172.21.15.125:0/1718023402 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f332c004d10 0x7f332c005190 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f333800a830 tx=0x7f333801d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1585ab549f501ba7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:39.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.110+0000 7f334551a640 1 -- 172.21.15.125:0/1718023402 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33380ad630 con 0x7f332c004d10 2023-10-30T13:06:39.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.110+0000 7f334551a640 1 -- 172.21.15.125:0/1718023402 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33380a8070 con 0x7f332c004d10 2023-10-30T13:06:39.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.110+0000 7f334551a640 1 -- 172.21.15.125:0/1718023402 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33380ad9f0 con 0x7f332c004d10 2023-10-30T13:06:39.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.110+0000 7f3348780640 1 -- 172.21.15.125:0/1718023402 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f332c004d10 msgr2=0x7f332c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:39.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.110+0000 7f3348780640 1 --2- 172.21.15.125:0/1718023402 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f332c004d10 0x7f332c005190 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f333800a830 tx=0x7f333801d360 comp rx=0 tx=0).stop 2023-10-30T13:06:39.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.110+0000 7f3346d1d640 1 -- 172.21.15.125:0/1718023402 reap_dead start 2023-10-30T13:06:39.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3348780640 1 -- 172.21.15.125:0/1718023402 shutdown_connections 2023-10-30T13:06:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3348780640 1 -- 172.21.15.125:0/1718023402 >> 172.21.15.125:0/1718023402 conn(0x7f334006c6e0 msgr2=0x7f3340069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3348780640 1 -- 172.21.15.125:0/1718023402 shutdown_connections 2023-10-30T13:06:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3348780640 1 -- 172.21.15.125:0/1718023402 wait complete. 2023-10-30T13:06:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3348780640 1 Processor -- start 2023-10-30T13:06:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3348780640 1 -- start start 2023-10-30T13:06:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3348780640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f332c004d10 0x7f33401f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3348780640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f334015c710 0x7f33401eef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3348780640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3340147db0 0x7f3340148d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3348780640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3340148b30 con 0x7f334015c710 2023-10-30T13:06:39.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3348780640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33401eeaf0 con 0x7f3340147db0 2023-10-30T13:06:39.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3348780640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3340148570 con 0x7f332c004d10 2023-10-30T13:06:39.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3345d1b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f334015c710 0x7f33401eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:39.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3345d1b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f334015c710 0x7f33401eef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45260/0 (socket says 172.21.15.125:45260) 2023-10-30T13:06:39.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3345d1b640 1 -- 172.21.15.125:0/4117971350 learned_addr learned my addr 172.21.15.125:0/4117971350 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f334651c640 1 --2- 172.21.15.125:0/4117971350 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f332c004d10 0x7f33401f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3346d1d640 1 --2- 172.21.15.125:0/4117971350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3340147db0 0x7f3340148d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3345d1b640 1 -- 172.21.15.125:0/4117971350 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f332c004d10 msgr2=0x7f33401f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3345d1b640 1 --2- 172.21.15.125:0/4117971350 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f332c004d10 0x7f33401f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:39.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3345d1b640 1 -- 172.21.15.125:0/4117971350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3340147db0 msgr2=0x7f3340148d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:39.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3345d1b640 1 --2- 172.21.15.125:0/4117971350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3340147db0 0x7f3340148d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:39.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3345d1b640 1 -- 172.21.15.125:0/4117971350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3340148850 con 0x7f334015c710 2023-10-30T13:06:39.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3345d1b640 1 --2- 172.21.15.125:0/4117971350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f334015c710 0x7f33401eef80 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f3330011600 tx=0x7f3330011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:39.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3336ffd640 1 -- 172.21.15.125:0/4117971350 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3330015870 con 0x7f334015c710 2023-10-30T13:06:39.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.114+0000 7f3336ffd640 1 -- 172.21.15.125:0/4117971350 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33300159d0 con 0x7f334015c710 2023-10-30T13:06:39.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.118+0000 7f3336ffd640 1 -- 172.21.15.125:0/4117971350 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3330015d60 con 0x7f334015c710 2023-10-30T13:06:39.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.118+0000 7f3348780640 1 -- 172.21.15.125:0/4117971350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33401ee780 con 0x7f334015c710 2023-10-30T13:06:39.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.118+0000 7f3348780640 1 -- 172.21.15.125:0/4117971350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33401ef4c0 con 0x7f334015c710 2023-10-30T13:06:39.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.118+0000 7f3348780640 1 -- 172.21.15.125:0/4117971350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33401ee5e0 con 0x7f334015c710 2023-10-30T13:06:39.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.118+0000 7f3336ffd640 1 -- 172.21.15.125:0/4117971350 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f333000f450 con 0x7f334015c710 2023-10-30T13:06:39.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.118+0000 7f3336ffd640 1 --2- 172.21.15.125:0/4117971350 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33240664a0 0x7f3324068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:39.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.118+0000 7f3336ffd640 1 -- 172.21.15.125:0/4117971350 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f3330084e30 con 0x7f334015c710 2023-10-30T13:06:39.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.118+0000 7f334651c640 1 --2- 172.21.15.125:0/4117971350 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33240664a0 0x7f3324068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:39.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.122+0000 7f334651c640 1 --2- 172.21.15.125:0/4117971350 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33240664a0 0x7f3324068960 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f333c0099b0 tx=0x7f333c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:39.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.122+0000 7f3336ffd640 1 -- 172.21.15.125:0/4117971350 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f33401ee5e0 con 0x7f334015c710 2023-10-30T13:06:39.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.306+0000 7f3348780640 1 -- 172.21.15.125:0/4117971350 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33401ee5e0 con 0x7f334015c710 2023-10-30T13:06:39.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.306+0000 7f3336ffd640 1 -- 172.21.15.125:0/4117971350 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+6546 (secure 0 0 0) 0x7f33401ee5e0 con 0x7f334015c710 2023-10-30T13:06:39.309 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:39.309 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:37.658555+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8178,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:starting","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:39.310 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 170 2023-10-30T13:06:39.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.306+0000 7f3348780640 1 -- 172.21.15.125:0/4117971350 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33240664a0 msgr2=0x7f3324068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:39.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.306+0000 7f3348780640 1 --2- 172.21.15.125:0/4117971350 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33240664a0 0x7f3324068960 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f333c0099b0 tx=0x7f333c008040 comp rx=0 tx=0).stop 2023-10-30T13:06:39.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.306+0000 7f3348780640 1 -- 172.21.15.125:0/4117971350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f334015c710 msgr2=0x7f33401eef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:39.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.306+0000 7f3348780640 1 --2- 172.21.15.125:0/4117971350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f334015c710 0x7f33401eef80 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f3330011600 tx=0x7f3330011ac0 comp rx=0 tx=0).stop 2023-10-30T13:06:39.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.306+0000 7f3348780640 1 -- 172.21.15.125:0/4117971350 shutdown_connections 2023-10-30T13:06:39.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.306+0000 7f3348780640 1 --2- 172.21.15.125:0/4117971350 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33240664a0 0x7f3324068960 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:39.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.306+0000 7f3348780640 1 --2- 172.21.15.125:0/4117971350 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3340147db0 0x7f3340148d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:39.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.306+0000 7f3348780640 1 --2- 172.21.15.125:0/4117971350 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f334015c710 0x7f33401eef80 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:39.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.306+0000 7f3348780640 1 --2- 172.21.15.125:0/4117971350 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f332c004d10 0x7f33401f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:39.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.306+0000 7f3348780640 1 -- 172.21.15.125:0/4117971350 >> 172.21.15.125:0/4117971350 conn(0x7f334006c6e0 msgr2=0x7f3340151520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:39.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.306+0000 7f3348780640 1 -- 172.21.15.125:0/4117971350 shutdown_connections 2023-10-30T13:06:39.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:39.306+0000 7f3348780640 1 -- 172.21.15.125:0/4117971350 wait complete. 2023-10-30T13:06:39.330 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:06:41.331 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:41.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.473+0000 7f7321572640 1 Processor -- start 2023-10-30T13:06:41.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.473+0000 7f7321572640 1 -- start start 2023-10-30T13:06:41.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.473+0000 7f7321572640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f731c06f7a0 0x7f731c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:41.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.473+0000 7f7321572640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f731c071110 0x7f731c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:41.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.473+0000 7f7321572640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c066cf0 con 0x7f731c0c6bd0 2023-10-30T13:06:41.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.473+0000 7f7321572640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f731c0700c0 con 0x7f731c06f7a0 2023-10-30T13:06:41.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.473+0000 7f7321572640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731c0c70d0 con 0x7f731c071110 2023-10-30T13:06:41.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.473+0000 7f731affd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f731c06f7a0 0x7f731c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:41.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.473+0000 7f731affd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f731c06f7a0 0x7f731c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54158/0 (socket says 172.21.15.125:54158) 2023-10-30T13:06:41.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.473+0000 7f731affd640 1 -- 172.21.15.125:0/71202156 learned_addr learned my addr 172.21.15.125:0/71202156 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:41.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.473+0000 7f731b7fe640 1 --2- 172.21.15.125:0/71202156 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f731c071110 0x7f731c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:41.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.477+0000 7f731affd640 1 -- 172.21.15.125:0/71202156 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f731c071110 msgr2=0x7f731c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:41.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.477+0000 7f731affd640 1 --2- 172.21.15.125:0/71202156 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f731c071110 0x7f731c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:41.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.477+0000 7f731affd640 1 -- 172.21.15.125:0/71202156 >> v1:172.21.15.136:6789/0 conn(0x7f731c0c6bd0 legacy=0x7f731c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:41.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.477+0000 7f731affd640 1 -- 172.21.15.125:0/71202156 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c0c7390 con 0x7f731c06f7a0 2023-10-30T13:06:41.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.477+0000 7f731affd640 1 --2- 172.21.15.125:0/71202156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f731c06f7a0 0x7f731c06fb80 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f730c009870 tx=0x7f730c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb7199ca508ce817 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:41.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.477+0000 7f731a7fc640 1 -- 172.21.15.125:0/71202156 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c0a8a30 con 0x7f731c06f7a0 2023-10-30T13:06:41.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.477+0000 7f731a7fc640 1 -- 172.21.15.125:0/71202156 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c0a8b90 con 0x7f731c06f7a0 2023-10-30T13:06:41.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.477+0000 7f731a7fc640 1 -- 172.21.15.125:0/71202156 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c0b0440 con 0x7f731c06f7a0 2023-10-30T13:06:41.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.477+0000 7f7321572640 1 -- 172.21.15.125:0/71202156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f731c06f7a0 msgr2=0x7f731c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:41.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.477+0000 7f7321572640 1 --2- 172.21.15.125:0/71202156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f731c06f7a0 0x7f731c06fb80 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f730c009870 tx=0x7f730c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:06:41.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7321572640 1 -- 172.21.15.125:0/71202156 shutdown_connections 2023-10-30T13:06:41.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7321572640 1 --2- 172.21.15.125:0/71202156 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f731c071110 0x7f731c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:41.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7321572640 1 --2- 172.21.15.125:0/71202156 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f731c06f7a0 0x7f731c06fb80 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:41.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7321572640 1 -- 172.21.15.125:0/71202156 >> 172.21.15.125:0/71202156 conn(0x7f731c0c7910 msgr2=0x7f731c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:41.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7321572640 1 -- 172.21.15.125:0/71202156 shutdown_connections 2023-10-30T13:06:41.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7321572640 1 -- 172.21.15.125:0/71202156 wait complete. 2023-10-30T13:06:41.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7321572640 1 Processor -- start 2023-10-30T13:06:41.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7321572640 1 -- start start 2023-10-30T13:06:41.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7321572640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f731c06f7a0 0x7f731c1520f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:41.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7321572640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f731c071110 0x7f731c0c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:41.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7321572640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f731c0c6bd0 0x7f731c1eebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:41.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7321572640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c1f1eb0 con 0x7f731c06f7a0 2023-10-30T13:06:41.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7321572640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731c1f2d10 con 0x7f731c0c6bd0 2023-10-30T13:06:41.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7321572640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f731c152630 con 0x7f731c071110 2023-10-30T13:06:41.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f731affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f731c06f7a0 0x7f731c1520f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:41.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f731affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f731c06f7a0 0x7f731c1520f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54162/0 (socket says 172.21.15.125:54162) 2023-10-30T13:06:41.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f731affd640 1 -- 172.21.15.125:0/2630480906 learned_addr learned my addr 172.21.15.125:0/2630480906 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:41.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f7313fff640 1 --2- 172.21.15.125:0/2630480906 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f731c071110 0x7f731c0c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:41.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f731b7fe640 1 --2- 172.21.15.125:0/2630480906 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f731c0c6bd0 0x7f731c1eebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f731affd640 1 -- 172.21.15.125:0/2630480906 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f731c071110 msgr2=0x7f731c0c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f731affd640 1 --2- 172.21.15.125:0/2630480906 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f731c071110 0x7f731c0c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f731affd640 1 -- 172.21.15.125:0/2630480906 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f731c0c6bd0 msgr2=0x7f731c1eebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f731affd640 1 --2- 172.21.15.125:0/2630480906 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f731c0c6bd0 0x7f731c1eebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f731affd640 1 -- 172.21.15.125:0/2630480906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c0094d0 con 0x7f731c06f7a0 2023-10-30T13:06:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.481+0000 7f731affd640 1 --2- 172.21.15.125:0/2630480906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f731c06f7a0 0x7f731c1520f0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f730c0b3040 tx=0x7f730c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:41.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.485+0000 7f73137fe640 1 -- 172.21.15.125:0/2630480906 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c0be070 con 0x7f731c06f7a0 2023-10-30T13:06:41.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.485+0000 7f73137fe640 1 -- 172.21.15.125:0/2630480906 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c0ba400 con 0x7f731c06f7a0 2023-10-30T13:06:41.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.485+0000 7f73137fe640 1 -- 172.21.15.125:0/2630480906 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c0ba7d0 con 0x7f731c06f7a0 2023-10-30T13:06:41.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.485+0000 7f7321572640 1 -- 172.21.15.125:0/2630480906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731c1f2100 con 0x7f731c06f7a0 2023-10-30T13:06:41.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.485+0000 7f7321572640 1 -- 172.21.15.125:0/2630480906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f731c1f26c0 con 0x7f731c06f7a0 2023-10-30T13:06:41.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.485+0000 7f7321572640 1 -- 172.21.15.125:0/2630480906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731c04e170 con 0x7f731c06f7a0 2023-10-30T13:06:41.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.485+0000 7f73137fe640 1 -- 172.21.15.125:0/2630480906 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f730c0c3410 con 0x7f731c06f7a0 2023-10-30T13:06:41.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.485+0000 7f73137fe640 1 --2- 172.21.15.125:0/2630480906 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72fc0664d0 0x7f72fc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:41.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.485+0000 7f73137fe640 1 -- 172.21.15.125:0/2630480906 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f730c1220b0 con 0x7f731c06f7a0 2023-10-30T13:06:41.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.485+0000 7f7313fff640 1 --2- 172.21.15.125:0/2630480906 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72fc0664d0 0x7f72fc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:41.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.489+0000 7f7313fff640 1 --2- 172.21.15.125:0/2630480906 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72fc0664d0 0x7f72fc068990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f731c0c0f10 tx=0x7f7304009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:41.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.493+0000 7f73137fe640 1 -- 172.21.15.125:0/2630480906 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f730c0eee40 con 0x7f731c06f7a0 2023-10-30T13:06:41.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.697+0000 7f7321572640 1 -- 172.21.15.125:0/2630480906 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f731c0c64a0 con 0x7f731c06f7a0 2023-10-30T13:06:41.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.697+0000 7f73137fe640 1 -- 172.21.15.125:0/2630480906 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+6546 (secure 0 0 0) 0x7f730c0f2460 con 0x7f731c06f7a0 2023-10-30T13:06:41.700 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:41.701 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:37.658555+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8178,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:starting","state_seq":1,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:41.701 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 170 2023-10-30T13:06:41.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.697+0000 7f7321572640 1 -- 172.21.15.125:0/2630480906 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72fc0664d0 msgr2=0x7f72fc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:41.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.697+0000 7f7321572640 1 --2- 172.21.15.125:0/2630480906 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72fc0664d0 0x7f72fc068990 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f731c0c0f10 tx=0x7f7304009040 comp rx=0 tx=0).stop 2023-10-30T13:06:41.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.697+0000 7f7321572640 1 -- 172.21.15.125:0/2630480906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f731c06f7a0 msgr2=0x7f731c1520f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:41.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.697+0000 7f7321572640 1 --2- 172.21.15.125:0/2630480906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f731c06f7a0 0x7f731c1520f0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f730c0b3040 tx=0x7f730c01bad0 comp rx=0 tx=0).stop 2023-10-30T13:06:41.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.697+0000 7f7321572640 1 -- 172.21.15.125:0/2630480906 shutdown_connections 2023-10-30T13:06:41.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.697+0000 7f7321572640 1 --2- 172.21.15.125:0/2630480906 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f72fc0664d0 0x7f72fc068990 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:41.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.697+0000 7f7321572640 1 --2- 172.21.15.125:0/2630480906 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f731c0c6bd0 0x7f731c1eebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:41.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.697+0000 7f7321572640 1 --2- 172.21.15.125:0/2630480906 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f731c071110 0x7f731c0c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:41.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.697+0000 7f7321572640 1 --2- 172.21.15.125:0/2630480906 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f731c06f7a0 0x7f731c1520f0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:41.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.697+0000 7f7321572640 1 -- 172.21.15.125:0/2630480906 >> 172.21.15.125:0/2630480906 conn(0x7f731c0c7910 msgr2=0x7f731c0c4140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:41.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.697+0000 7f7321572640 1 -- 172.21.15.125:0/2630480906 shutdown_connections 2023-10-30T13:06:41.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:41.697+0000 7f7321572640 1 -- 172.21.15.125:0/2630480906 wait complete. 2023-10-30T13:06:41.724 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:06:42.671 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 81 -eq 48 ] 2023-10-30T13:06:42.671 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.81 2023-10-30T13:06:42.672 INFO:tasks.workunit.client.0.smithi125.stdout:v0.81 2023-10-30T13:06:42.672 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.81 2023-10-30T13:06:42.672 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.81 2023-10-30T13:06:43.725 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:43.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.869+0000 7f7d34971640 1 Processor -- start 2023-10-30T13:06:43.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.869+0000 7f7d34971640 1 -- start start 2023-10-30T13:06:43.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.869+0000 7f7d34971640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7d3015cd50 0x7f7d300c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:43.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.869+0000 7f7d34971640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300cefa0 0x7f7d300cf3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:43.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.869+0000 7f7d34971640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3015a6c0 con 0x7f7d301538f0 2023-10-30T13:06:43.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.869+0000 7f7d34971640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3015d300 con 0x7f7d3015cd50 2023-10-30T13:06:43.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.869+0000 7f7d34971640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3015d5e0 con 0x7f7d300cefa0 2023-10-30T13:06:43.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.869+0000 7f7d2effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7d3015cd50 0x7f7d300c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:43.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.869+0000 7f7d2effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7d3015cd50 0x7f7d300c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54164/0 (socket says 172.21.15.125:54164) 2023-10-30T13:06:43.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.869+0000 7f7d2effd640 1 -- 172.21.15.125:0/3148018092 learned_addr learned my addr 172.21.15.125:0/3148018092 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:43.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.869+0000 7f7d2ffff640 1 --2- 172.21.15.125:0/3148018092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300cefa0 0x7f7d300cf3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:43.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.873+0000 7f7d2effd640 1 -- 172.21.15.125:0/3148018092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300cefa0 msgr2=0x7f7d300cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:43.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.873+0000 7f7d2effd640 1 --2- 172.21.15.125:0/3148018092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300cefa0 0x7f7d300cf3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:43.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.873+0000 7f7d2effd640 1 -- 172.21.15.125:0/3148018092 >> v1:172.21.15.136:6789/0 conn(0x7f7d301538f0 legacy=0x7f7d30159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:43.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.873+0000 7f7d2effd640 1 -- 172.21.15.125:0/3148018092 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3015d8c0 con 0x7f7d3015cd50 2023-10-30T13:06:43.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.873+0000 7f7d2effd640 1 --2- 172.21.15.125:0/3148018092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d3015cd50 0x7f7d300c9f90 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009870 tx=0x7f7d2401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9cf4de84ace5116 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:43.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.873+0000 7f7d2e7fc640 1 -- 172.21.15.125:0/3148018092 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d240a8a30 con 0x7f7d3015cd50 2023-10-30T13:06:43.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.873+0000 7f7d2e7fc640 1 -- 172.21.15.125:0/3148018092 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d240a8b90 con 0x7f7d3015cd50 2023-10-30T13:06:43.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.873+0000 7f7d2e7fc640 1 -- 172.21.15.125:0/3148018092 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d240b0440 con 0x7f7d3015cd50 2023-10-30T13:06:43.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.877+0000 7f7d34971640 1 -- 172.21.15.125:0/3148018092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d3015cd50 msgr2=0x7f7d300c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:43.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.877+0000 7f7d34971640 1 --2- 172.21.15.125:0/3148018092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d3015cd50 0x7f7d300c9f90 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009870 tx=0x7f7d2401ed30 comp rx=0 tx=0).stop 2023-10-30T13:06:43.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.877+0000 7f7d34971640 1 -- 172.21.15.125:0/3148018092 shutdown_connections 2023-10-30T13:06:43.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.877+0000 7f7d34971640 1 --2- 172.21.15.125:0/3148018092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300cefa0 0x7f7d300cf3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:43.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.877+0000 7f7d34971640 1 --2- 172.21.15.125:0/3148018092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d3015cd50 0x7f7d300c9f90 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:43.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.877+0000 7f7d34971640 1 -- 172.21.15.125:0/3148018092 >> 172.21.15.125:0/3148018092 conn(0x7f7d3006a640 msgr2=0x7f7d30065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:43.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.877+0000 7f7d34971640 1 -- 172.21.15.125:0/3148018092 shutdown_connections 2023-10-30T13:06:43.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.877+0000 7f7d34971640 1 -- 172.21.15.125:0/3148018092 wait complete. 2023-10-30T13:06:43.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.877+0000 7f7d34971640 1 Processor -- start 2023-10-30T13:06:43.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.877+0000 7f7d34971640 1 -- start start 2023-10-30T13:06:43.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d34971640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d300cefa0 0x7f7d30158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d34971640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d301538f0 0x7f7d300b37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d34971640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d3015cd50 0x7f7d30144700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d34971640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d300b4300 con 0x7f7d301538f0 2023-10-30T13:06:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d34971640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d300b2ff0 con 0x7f7d300cefa0 2023-10-30T13:06:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d34971640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30150f90 con 0x7f7d3015cd50 2023-10-30T13:06:43.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d2effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d301538f0 0x7f7d300b37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:43.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d2effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d301538f0 0x7f7d300b37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54180/0 (socket says 172.21.15.125:54180) 2023-10-30T13:06:43.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d2effd640 1 -- 172.21.15.125:0/2989762173 learned_addr learned my addr 172.21.15.125:0/2989762173 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:43.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d2ffff640 1 --2- 172.21.15.125:0/2989762173 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d3015cd50 0x7f7d30144700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:43.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d2f7fe640 1 --2- 172.21.15.125:0/2989762173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d300cefa0 0x7f7d30158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d2effd640 1 -- 172.21.15.125:0/2989762173 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d3015cd50 msgr2=0x7f7d30144700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d2effd640 1 --2- 172.21.15.125:0/2989762173 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d3015cd50 0x7f7d30144700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d2effd640 1 -- 172.21.15.125:0/2989762173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d300cefa0 msgr2=0x7f7d30158f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:43.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d2effd640 1 --2- 172.21.15.125:0/2989762173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d300cefa0 0x7f7d30158f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:43.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d2effd640 1 -- 172.21.15.125:0/2989762173 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d240094d0 con 0x7f7d301538f0 2023-10-30T13:06:43.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d2effd640 1 --2- 172.21.15.125:0/2989762173 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d301538f0 0x7f7d300b37b0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f7d240b3040 tx=0x7f7d2401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:43.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d17fff640 1 -- 172.21.15.125:0/2989762173 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d240b0dd0 con 0x7f7d301538f0 2023-10-30T13:06:43.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d17fff640 1 -- 172.21.15.125:0/2989762173 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d240bb460 con 0x7f7d301538f0 2023-10-30T13:06:43.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d34971640 1 -- 172.21.15.125:0/2989762173 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d30151270 con 0x7f7d301538f0 2023-10-30T13:06:43.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d17fff640 1 -- 172.21.15.125:0/2989762173 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d240bb810 con 0x7f7d301538f0 2023-10-30T13:06:43.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d34971640 1 -- 172.21.15.125:0/2989762173 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d30151550 con 0x7f7d301538f0 2023-10-30T13:06:43.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.881+0000 7f7d34971640 1 -- 172.21.15.125:0/2989762173 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d300b34d0 con 0x7f7d301538f0 2023-10-30T13:06:43.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.885+0000 7f7d17fff640 1 -- 172.21.15.125:0/2989762173 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7d240b83f0 con 0x7f7d301538f0 2023-10-30T13:06:43.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.885+0000 7f7d17fff640 1 --2- 172.21.15.125:0/2989762173 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d1c0664d0 0x7f7d1c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:43.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.885+0000 7f7d17fff640 1 -- 172.21.15.125:0/2989762173 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f7d24121e90 con 0x7f7d301538f0 2023-10-30T13:06:43.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.885+0000 7f7d2f7fe640 1 --2- 172.21.15.125:0/2989762173 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d1c0664d0 0x7f7d1c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:43.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.885+0000 7f7d2f7fe640 1 --2- 172.21.15.125:0/2989762173 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d1c0664d0 0x7f7d1c068990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f7d20007fd0 tx=0x7f7d20009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:43.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:43.889+0000 7f7d17fff640 1 -- 172.21.15.125:0/2989762173 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7d300b34d0 con 0x7f7d301538f0 2023-10-30T13:06:44.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.073+0000 7f7d34971640 1 -- 172.21.15.125:0/2989762173 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d300b34d0 con 0x7f7d301538f0 2023-10-30T13:06:44.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.073+0000 7f7d17fff640 1 -- 172.21.15.125:0/2989762173 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6546 (secure 0 0 0) 0x7f7d300b34d0 con 0x7f7d301538f0 2023-10-30T13:06:44.079 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:44.080 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:42.679260+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8178,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:44.080 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 171 2023-10-30T13:06:44.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.077+0000 7f7d34971640 1 -- 172.21.15.125:0/2989762173 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d1c0664d0 msgr2=0x7f7d1c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:44.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.077+0000 7f7d34971640 1 --2- 172.21.15.125:0/2989762173 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d1c0664d0 0x7f7d1c068990 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f7d20007fd0 tx=0x7f7d20009040 comp rx=0 tx=0).stop 2023-10-30T13:06:44.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.077+0000 7f7d34971640 1 -- 172.21.15.125:0/2989762173 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d301538f0 msgr2=0x7f7d300b37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:44.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.077+0000 7f7d34971640 1 --2- 172.21.15.125:0/2989762173 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d301538f0 0x7f7d300b37b0 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f7d240b3040 tx=0x7f7d2401bad0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.077+0000 7f7d34971640 1 -- 172.21.15.125:0/2989762173 shutdown_connections 2023-10-30T13:06:44.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.077+0000 7f7d34971640 1 --2- 172.21.15.125:0/2989762173 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d1c0664d0 0x7f7d1c068990 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.077+0000 7f7d34971640 1 --2- 172.21.15.125:0/2989762173 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d3015cd50 0x7f7d30144700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.077+0000 7f7d34971640 1 --2- 172.21.15.125:0/2989762173 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d301538f0 0x7f7d300b37b0 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.077+0000 7f7d34971640 1 --2- 172.21.15.125:0/2989762173 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d300cefa0 0x7f7d30158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.077+0000 7f7d34971640 1 -- 172.21.15.125:0/2989762173 >> 172.21.15.125:0/2989762173 conn(0x7f7d3006a640 msgr2=0x7f7d300cf3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:44.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.077+0000 7f7d34971640 1 -- 172.21.15.125:0/2989762173 shutdown_connections 2023-10-30T13:06:44.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.077+0000 7f7d34971640 1 -- 172.21.15.125:0/2989762173 wait complete. 2023-10-30T13:06:44.100 INFO:tasks.mds_thrash.fs.[cephfs]:mds cluster has 3 alive and active, now stable! 2023-10-30T13:06:44.100 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T13:06:44.100 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T13:06:44.101 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:06:44.130 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T13:06:44.131 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:44.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.301+0000 7f4080a17640 1 Processor -- start 2023-10-30T13:06:44.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f4080a17640 1 -- start start 2023-10-30T13:06:44.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f4080a17640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f407c06f7a0 0x7f407c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:44.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f4080a17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f407c0c6bd0 0x7f407c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:44.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f4080a17640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407c06a3d0 con 0x7f407c071110 2023-10-30T13:06:44.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f4080a17640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407c0700c0 con 0x7f407c06f7a0 2023-10-30T13:06:44.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f4080a17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f407c066cf0 con 0x7f407c0c6bd0 2023-10-30T13:06:44.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f407a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f407c06f7a0 0x7f407c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:44.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f407a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f407c06f7a0 0x7f407c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54188/0 (socket says 172.21.15.125:54188) 2023-10-30T13:06:44.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f407a59c640 1 -- 172.21.15.125:0/279604746 learned_addr learned my addr 172.21.15.125:0/279604746 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:44.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f4071d9b640 1 --2- 172.21.15.125:0/279604746 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f407c0c6bd0 0x7f407c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:44.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f407a59c640 1 -- 172.21.15.125:0/279604746 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f407c0c6bd0 msgr2=0x7f407c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:44.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f407a59c640 1 --2- 172.21.15.125:0/279604746 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f407c0c6bd0 0x7f407c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f407a59c640 1 -- 172.21.15.125:0/279604746 >> v1:172.21.15.136:6789/0 conn(0x7f407c071110 legacy=0x7f407c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:44.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f407a59c640 1 -- 172.21.15.125:0/279604746 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c0c70d0 con 0x7f407c06f7a0 2023-10-30T13:06:44.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.305+0000 7f4071d9b640 1 --2- 172.21.15.125:0/279604746 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f407c0c6bd0 0x7f407c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:44.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.309+0000 7f407a59c640 1 --2- 172.21.15.125:0/279604746 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407c06f7a0 0x7f407c06fb80 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f406c009870 tx=0x7f406c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db824fa522846a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:44.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.309+0000 7f4079d9b640 1 -- 172.21.15.125:0/279604746 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c0a8a30 con 0x7f407c06f7a0 2023-10-30T13:06:44.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.309+0000 7f4079d9b640 1 -- 172.21.15.125:0/279604746 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406c0a8b90 con 0x7f407c06f7a0 2023-10-30T13:06:44.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.309+0000 7f4079d9b640 1 -- 172.21.15.125:0/279604746 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c0b0440 con 0x7f407c06f7a0 2023-10-30T13:06:44.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.309+0000 7f4080a17640 1 -- 172.21.15.125:0/279604746 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407c06f7a0 msgr2=0x7f407c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:44.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.309+0000 7f4080a17640 1 --2- 172.21.15.125:0/279604746 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407c06f7a0 0x7f407c06fb80 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f406c009870 tx=0x7f406c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:06:44.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.309+0000 7f4080a17640 1 -- 172.21.15.125:0/279604746 shutdown_connections 2023-10-30T13:06:44.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.309+0000 7f4080a17640 1 --2- 172.21.15.125:0/279604746 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f407c0c6bd0 0x7f407c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.309+0000 7f4080a17640 1 --2- 172.21.15.125:0/279604746 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407c06f7a0 0x7f407c06fb80 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.309+0000 7f4080a17640 1 -- 172.21.15.125:0/279604746 >> 172.21.15.125:0/279604746 conn(0x7f407c0c7910 msgr2=0x7f407c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:44.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.309+0000 7f4080a17640 1 -- 172.21.15.125:0/279604746 shutdown_connections 2023-10-30T13:06:44.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.309+0000 7f4080a17640 1 -- 172.21.15.125:0/279604746 wait complete. 2023-10-30T13:06:44.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.309+0000 7f4080a17640 1 Processor -- start 2023-10-30T13:06:44.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f4080a17640 1 -- start start 2023-10-30T13:06:44.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f4080a17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f407c06f7a0 0x7f407c159c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:44.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f4080a17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f407c071110 0x7f407c0ae3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:44.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f4080a17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407c0c6bd0 0x7f407c0c0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:44.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f4080a17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c0af980 con 0x7f407c0c6bd0 2023-10-30T13:06:44.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f4080a17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c15a570 con 0x7f407c06f7a0 2023-10-30T13:06:44.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f4080a17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f407c15a290 con 0x7f407c071110 2023-10-30T13:06:44.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f407ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407c0c6bd0 0x7f407c0c0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:44.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f407ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407c0c6bd0 0x7f407c0c0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54194/0 (socket says 172.21.15.125:54194) 2023-10-30T13:06:44.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f407ad9d640 1 -- 172.21.15.125:0/1358363321 learned_addr learned my addr 172.21.15.125:0/1358363321 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:44.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f407a59c640 1 --2- 172.21.15.125:0/1358363321 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f407c06f7a0 0x7f407c159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:44.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f4071d9b640 1 --2- 172.21.15.125:0/1358363321 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f407c071110 0x7f407c0ae3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:44.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f407ad9d640 1 -- 172.21.15.125:0/1358363321 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f407c071110 msgr2=0x7f407c0ae3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:44.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f407ad9d640 1 --2- 172.21.15.125:0/1358363321 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f407c071110 0x7f407c0ae3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f407ad9d640 1 -- 172.21.15.125:0/1358363321 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f407c06f7a0 msgr2=0x7f407c159c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:44.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f407ad9d640 1 --2- 172.21.15.125:0/1358363321 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f407c06f7a0 0x7f407c159c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f407ad9d640 1 -- 172.21.15.125:0/1358363321 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c0094d0 con 0x7f407c0c6bd0 2023-10-30T13:06:44.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f407ad9d640 1 --2- 172.21.15.125:0/1358363321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407c0c6bd0 0x7f407c0c0070 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f407400c730 tx=0x7f407400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:44.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f40737fe640 1 -- 172.21.15.125:0/1358363321 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4074003b80 con 0x7f407c0c6bd0 2023-10-30T13:06:44.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f40737fe640 1 -- 172.21.15.125:0/1358363321 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4074003ce0 con 0x7f407c0c6bd0 2023-10-30T13:06:44.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f40737fe640 1 -- 172.21.15.125:0/1358363321 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f407400aa30 con 0x7f407c0c6bd0 2023-10-30T13:06:44.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f4080a17640 1 -- 172.21.15.125:0/1358363321 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f407c0aefe0 con 0x7f407c0c6bd0 2023-10-30T13:06:44.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f4080a17640 1 -- 172.21.15.125:0/1358363321 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f407c0af5a0 con 0x7f407c0c6bd0 2023-10-30T13:06:44.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.313+0000 7f4080a17640 1 -- 172.21.15.125:0/1358363321 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f407c0afac0 con 0x7f407c0c6bd0 2023-10-30T13:06:44.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.317+0000 7f40737fe640 1 -- 172.21.15.125:0/1358363321 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f407400ab90 con 0x7f407c0c6bd0 2023-10-30T13:06:44.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.317+0000 7f40737fe640 1 --2- 172.21.15.125:0/1358363321 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f405c0664a0 0x7f405c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:44.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.317+0000 7f40737fe640 1 -- 172.21.15.125:0/1358363321 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f4074084300 con 0x7f407c0c6bd0 2023-10-30T13:06:44.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.317+0000 7f407a59c640 1 --2- 172.21.15.125:0/1358363321 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f405c0664a0 0x7f405c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:44.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.317+0000 7f407a59c640 1 --2- 172.21.15.125:0/1358363321 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f405c0664a0 0x7f405c068960 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f406c0b2000 tx=0x7f406c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:44.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.321+0000 7f40737fe640 1 -- 172.21.15.125:0/1358363321 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f40740510a0 con 0x7f407c0c6bd0 2023-10-30T13:06:44.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.501+0000 7f4080a17640 1 -- 172.21.15.125:0/1358363321 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f407c04e170 con 0x7f407c0c6bd0 2023-10-30T13:06:44.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.501+0000 7f40737fe640 1 -- 172.21.15.125:0/1358363321 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6546 (secure 0 0 0) 0x7f40740546c0 con 0x7f407c0c6bd0 2023-10-30T13:06:44.507 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:44.508 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:42.679260+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8178,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:44.508 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 171 2023-10-30T13:06:44.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.505+0000 7f4080a17640 1 -- 172.21.15.125:0/1358363321 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f405c0664a0 msgr2=0x7f405c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:44.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.505+0000 7f4080a17640 1 --2- 172.21.15.125:0/1358363321 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f405c0664a0 0x7f405c068960 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f406c0b2000 tx=0x7f406c009040 comp rx=0 tx=0).stop 2023-10-30T13:06:44.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.505+0000 7f4080a17640 1 -- 172.21.15.125:0/1358363321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407c0c6bd0 msgr2=0x7f407c0c0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:44.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.505+0000 7f4080a17640 1 --2- 172.21.15.125:0/1358363321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407c0c6bd0 0x7f407c0c0070 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f407400c730 tx=0x7f407400cbf0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.505+0000 7f4080a17640 1 -- 172.21.15.125:0/1358363321 shutdown_connections 2023-10-30T13:06:44.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.505+0000 7f4080a17640 1 --2- 172.21.15.125:0/1358363321 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f405c0664a0 0x7f405c068960 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.505+0000 7f4080a17640 1 --2- 172.21.15.125:0/1358363321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f407c0c6bd0 0x7f407c0c0070 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.505+0000 7f4080a17640 1 --2- 172.21.15.125:0/1358363321 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f407c071110 0x7f407c0ae3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.505+0000 7f4080a17640 1 --2- 172.21.15.125:0/1358363321 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f407c06f7a0 0x7f407c159c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:44.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.505+0000 7f4080a17640 1 -- 172.21.15.125:0/1358363321 >> 172.21.15.125:0/1358363321 conn(0x7f407c0c7910 msgr2=0x7f407c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:44.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.505+0000 7f4080a17640 1 -- 172.21.15.125:0/1358363321 shutdown_connections 2023-10-30T13:06:44.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:44.505+0000 7f4080a17640 1 -- 172.21.15.125:0/1358363321 wait complete. 2023-10-30T13:06:44.528 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:06:46.530 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:46.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.673+0000 7fd3ccc46640 1 Processor -- start 2023-10-30T13:06:46.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.673+0000 7fd3ccc46640 1 -- start start 2023-10-30T13:06:46.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3ccc46640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd3c81538a0 0x7fd3c815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:46.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3ccc46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3c80ca0e0 0x7fd3c815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:46.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3ccc46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3c80cb800 con 0x7fd3c80cf160 2023-10-30T13:06:46.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3ccc46640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3c815dc30 con 0x7fd3c81538a0 2023-10-30T13:06:46.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3ccc46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c80cb330 con 0x7fd3c80ca0e0 2023-10-30T13:06:46.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c6d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd3c80cf160 0x7fd3c80cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34870/0 (socket says 172.21.15.125:34870) 2023-10-30T13:06:46.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c6d9d640 1 -- 172.21.15.125:0/3799667976 learned_addr learned my addr 172.21.15.125:0/3799667976 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:06:46.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c5d9b640 1 --2- 172.21.15.125:0/3799667976 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3c80ca0e0 0x7fd3c815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:46.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c659c640 1 --2- 172.21.15.125:0/3799667976 >> v2:172.21.15.136:3300/0 conn(0x7fd3c81538a0 0x7fd3c815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:46.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 -- 172.21.15.125:0/3799667976 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458550605 0 0) 0x7fd3c80cb800 con 0x7fd3c80cf160 2023-10-30T13:06:46.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 -- 172.21.15.125:0/3799667976 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3b0003530 con 0x7fd3c80cf160 2023-10-30T13:06:46.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 -- 172.21.15.125:0/3799667976 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1366885566 0 0) 0x7fd3b0003530 con 0x7fd3c80cf160 2023-10-30T13:06:46.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 -- 172.21.15.125:0/3799667976 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3c80ca0e0 msgr2=0x7fd3c815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:46.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 --2- 172.21.15.125:0/3799667976 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3c80ca0e0 0x7fd3c815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:46.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 -- 172.21.15.125:0/3799667976 >> v2:172.21.15.136:3300/0 conn(0x7fd3c81538a0 msgr2=0x7fd3c815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:46.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 --2- 172.21.15.125:0/3799667976 >> v2:172.21.15.136:3300/0 conn(0x7fd3c81538a0 0x7fd3c815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:46.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 -- 172.21.15.125:0/3799667976 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c80cad70 con 0x7fd3c80cf160 2023-10-30T13:06:46.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 -- 172.21.15.125:0/3799667976 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fd3c0001a00 con 0x7fd3c80cf160 2023-10-30T13:06:46.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 -- 172.21.15.125:0/3799667976 >> v1:172.21.15.125:6789/0 conn(0x7fd3c80cf160 legacy=0x7fd3c80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:46.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 --2- 172.21.15.125:0/3799667976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3b0003980 0x7fd3b0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:46.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 --2- 172.21.15.125:0/3799667976 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3b00047d0 0x7fd3b0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:46.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 --2- 172.21.15.125:0/3799667976 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3c80ca0e0 0x7fd3b0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:46.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 -- 172.21.15.125:0/3799667976 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c815dc30 con 0x7fd3b0003980 2023-10-30T13:06:46.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 -- 172.21.15.125:0/3799667976 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c80cb330 con 0x7fd3b00047d0 2023-10-30T13:06:46.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c559a640 1 -- 172.21.15.125:0/3799667976 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b000b9c0 con 0x7fd3c80ca0e0 2023-10-30T13:06:46.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c659c640 1 --2- 172.21.15.125:0/3799667976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3b0003980 0x7fd3b0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:46.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c5d9b640 1 --2- 172.21.15.125:0/3799667976 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3b00047d0 0x7fd3b0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:46.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c6d9d640 1 --2- 172.21.15.125:0/3799667976 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3c80ca0e0 0x7fd3b0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:46.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c659c640 1 -- 172.21.15.125:0/3799667976 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3c80ca0e0 msgr2=0x7fd3b0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:46.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c659c640 1 --2- 172.21.15.125:0/3799667976 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3c80ca0e0 0x7fd3b0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:46.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c659c640 1 -- 172.21.15.125:0/3799667976 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3b00047d0 msgr2=0x7fd3b0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:46.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c659c640 1 --2- 172.21.15.125:0/3799667976 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3b00047d0 0x7fd3b0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:46.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c659c640 1 -- 172.21.15.125:0/3799667976 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3b0003530 con 0x7fd3b0003980 2023-10-30T13:06:46.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.677+0000 7fd3c5d9b640 1 --2- 172.21.15.125:0/3799667976 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3b00047d0 0x7fd3b0004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:46.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.681+0000 7fd3c659c640 1 --2- 172.21.15.125:0/3799667976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3b0003980 0x7fd3b0003da0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009ae0 tx=0x7fd3b801ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b66810687a7d0000 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:46.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.681+0000 7fd3c559a640 1 -- 172.21.15.125:0/3799667976 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b80a64a0 con 0x7fd3b0003980 2023-10-30T13:06:46.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.681+0000 7fd3c559a640 1 -- 172.21.15.125:0/3799667976 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b80a6600 con 0x7fd3b0003980 2023-10-30T13:06:46.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.681+0000 7fd3c559a640 1 -- 172.21.15.125:0/3799667976 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b80a69b0 con 0x7fd3b0003980 2023-10-30T13:06:46.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.681+0000 7fd3ccc46640 1 -- 172.21.15.125:0/3799667976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3b0003980 msgr2=0x7fd3b0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:46.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.681+0000 7fd3ccc46640 1 --2- 172.21.15.125:0/3799667976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3b0003980 0x7fd3b0003da0 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009ae0 tx=0x7fd3b801ca90 comp rx=0 tx=0).stop 2023-10-30T13:06:46.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.681+0000 7fd3c6d9d640 1 -- 172.21.15.125:0/3799667976 reap_dead start 2023-10-30T13:06:46.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.681+0000 7fd3ccc46640 1 -- 172.21.15.125:0/3799667976 shutdown_connections 2023-10-30T13:06:46.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.681+0000 7fd3ccc46640 1 -- 172.21.15.125:0/3799667976 >> 172.21.15.125:0/3799667976 conn(0x7fd3c806bfe0 msgr2=0x7fd3c806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:46.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.681+0000 7fd3ccc46640 1 -- 172.21.15.125:0/3799667976 shutdown_connections 2023-10-30T13:06:46.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.681+0000 7fd3ccc46640 1 -- 172.21.15.125:0/3799667976 wait complete. 2023-10-30T13:06:46.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3ccc46640 1 Processor -- start 2023-10-30T13:06:46.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3ccc46640 1 -- start start 2023-10-30T13:06:46.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3ccc46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3b0003980 0x7fd3c80ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:46.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3ccc46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3c81f0000 0x7fd3c8157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:46.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3ccc46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3c80cd550 0x7fd3c80b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:46.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3ccc46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c80b4f00 con 0x7fd3b0003980 2023-10-30T13:06:46.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3ccc46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c81efb70 con 0x7fd3c80cd550 2023-10-30T13:06:46.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3ccc46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c80b3650 con 0x7fd3c81f0000 2023-10-30T13:06:46.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3c659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3b0003980 0x7fd3c80ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:46.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3c659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3b0003980 0x7fd3c80ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54198/0 (socket says 172.21.15.125:54198) 2023-10-30T13:06:46.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3c659c640 1 -- 172.21.15.125:0/322003630 learned_addr learned my addr 172.21.15.125:0/322003630 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:46.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3c5d9b640 1 --2- 172.21.15.125:0/322003630 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3c81f0000 0x7fd3c8157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:46.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3c6d9d640 1 --2- 172.21.15.125:0/322003630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3c80cd550 0x7fd3c80b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:46.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3c659c640 1 -- 172.21.15.125:0/322003630 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3c81f0000 msgr2=0x7fd3c8157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:46.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3c659c640 1 --2- 172.21.15.125:0/322003630 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3c81f0000 0x7fd3c8157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:46.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3c659c640 1 -- 172.21.15.125:0/322003630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3c80cd550 msgr2=0x7fd3c80b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:46.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3c659c640 1 --2- 172.21.15.125:0/322003630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3c80cd550 0x7fd3c80b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:46.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3c659c640 1 -- 172.21.15.125:0/322003630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c80b4980 con 0x7fd3b0003980 2023-10-30T13:06:46.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3c659c640 1 --2- 172.21.15.125:0/322003630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3b0003980 0x7fd3c80ccfb0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009ae0 tx=0x7fd3b0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:46.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3b6ffd640 1 -- 172.21.15.125:0/322003630 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b80a7480 con 0x7fd3b0003980 2023-10-30T13:06:46.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3b6ffd640 1 -- 172.21.15.125:0/322003630 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3b80a75e0 con 0x7fd3b0003980 2023-10-30T13:06:46.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3b6ffd640 1 -- 172.21.15.125:0/322003630 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3b80a7970 con 0x7fd3b0003980 2023-10-30T13:06:46.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3ccc46640 1 -- 172.21.15.125:0/322003630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3c80b4f00 con 0x7fd3b0003980 2023-10-30T13:06:46.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3ccc46640 1 -- 172.21.15.125:0/322003630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3c81ef520 con 0x7fd3b0003980 2023-10-30T13:06:46.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.685+0000 7fd3ccc46640 1 -- 172.21.15.125:0/322003630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3c81ef380 con 0x7fd3b0003980 2023-10-30T13:06:46.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.689+0000 7fd3b6ffd640 1 -- 172.21.15.125:0/322003630 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd3b80a7ad0 con 0x7fd3b0003980 2023-10-30T13:06:46.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.689+0000 7fd3b6ffd640 1 --2- 172.21.15.125:0/322003630 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3a8066470 0x7fd3a8068930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:46.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.689+0000 7fd3b6ffd640 1 -- 172.21.15.125:0/322003630 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7fd3b811e460 con 0x7fd3b0003980 2023-10-30T13:06:46.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.689+0000 7fd3c5d9b640 1 --2- 172.21.15.125:0/322003630 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3a8066470 0x7fd3a8068930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:46.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.689+0000 7fd3c5d9b640 1 --2- 172.21.15.125:0/322003630 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3a8066470 0x7fd3a8068930 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd3c81f39b0 tx=0x7fd3bc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:46.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.693+0000 7fd3b6ffd640 1 -- 172.21.15.125:0/322003630 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd3c81ef380 con 0x7fd3b0003980 2023-10-30T13:06:46.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.873+0000 7fd3ccc46640 1 -- 172.21.15.125:0/322003630 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3c81ef380 con 0x7fd3b0003980 2023-10-30T13:06:46.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.873+0000 7fd3b6ffd640 1 -- 172.21.15.125:0/322003630 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6546 (secure 0 0 0) 0x7fd3c81ef380 con 0x7fd3b0003980 2023-10-30T13:06:46.876 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:46.877 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:42.679260+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8178,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:46.877 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 171 2023-10-30T13:06:46.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.873+0000 7fd3ccc46640 1 -- 172.21.15.125:0/322003630 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3a8066470 msgr2=0x7fd3a8068930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:46.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.873+0000 7fd3ccc46640 1 --2- 172.21.15.125:0/322003630 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3a8066470 0x7fd3a8068930 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fd3c81f39b0 tx=0x7fd3bc008040 comp rx=0 tx=0).stop 2023-10-30T13:06:46.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.873+0000 7fd3ccc46640 1 -- 172.21.15.125:0/322003630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3b0003980 msgr2=0x7fd3c80ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:46.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.873+0000 7fd3ccc46640 1 --2- 172.21.15.125:0/322003630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3b0003980 0x7fd3c80ccfb0 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fd3b8009ae0 tx=0x7fd3b0005800 comp rx=0 tx=0).stop 2023-10-30T13:06:46.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.873+0000 7fd3ccc46640 1 -- 172.21.15.125:0/322003630 shutdown_connections 2023-10-30T13:06:46.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.873+0000 7fd3ccc46640 1 --2- 172.21.15.125:0/322003630 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd3a8066470 0x7fd3a8068930 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:46.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.873+0000 7fd3ccc46640 1 --2- 172.21.15.125:0/322003630 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd3c80cd550 0x7fd3c80b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:46.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.873+0000 7fd3ccc46640 1 --2- 172.21.15.125:0/322003630 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd3c81f0000 0x7fd3c8157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:46.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.873+0000 7fd3ccc46640 1 --2- 172.21.15.125:0/322003630 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd3b0003980 0x7fd3c80ccfb0 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:46.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.873+0000 7fd3ccc46640 1 -- 172.21.15.125:0/322003630 >> 172.21.15.125:0/322003630 conn(0x7fd3c806bfe0 msgr2=0x7fd3c80c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:46.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.873+0000 7fd3ccc46640 1 -- 172.21.15.125:0/322003630 shutdown_connections 2023-10-30T13:06:46.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:46.873+0000 7fd3ccc46640 1 -- 172.21.15.125:0/322003630 wait complete. 2023-10-30T13:06:46.897 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:06:48.899 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:49.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.041+0000 7f4ba6e17640 1 Processor -- start 2023-10-30T13:06:49.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.041+0000 7f4ba6e17640 1 -- start start 2023-10-30T13:06:49.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.041+0000 7f4ba6e17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ba0071110 0x7f4ba006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:49.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.041+0000 7f4ba6e17640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba0066cf0 con 0x7f4ba006f7a0 2023-10-30T13:06:49.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.041+0000 7f4ba6e17640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba00c70d0 con 0x7f4ba00c6bd0 2023-10-30T13:06:49.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.041+0000 7f4ba6e17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba00700c0 con 0x7f4ba0071110 2023-10-30T13:06:49.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.041+0000 7f4ba4bb3640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4ba006f7a0 0x7f4ba0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34880/0 (socket says 172.21.15.125:34880) 2023-10-30T13:06:49.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4ba4bb3640 1 -- 172.21.15.125:0/3062725108 learned_addr learned my addr 172.21.15.125:0/3062725108 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:06:49.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4ba53b4640 1 --2- 172.21.15.125:0/3062725108 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ba0071110 0x7f4ba006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:49.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 -- 172.21.15.125:0/3062725108 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065831760 0 0) 0x7f4ba0066cf0 con 0x7f4ba006f7a0 2023-10-30T13:06:49.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 -- 172.21.15.125:0/3062725108 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b88003530 con 0x7f4ba006f7a0 2023-10-30T13:06:49.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 -- 172.21.15.125:0/3062725108 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1839731432 0 0) 0x7f4b88003530 con 0x7f4ba006f7a0 2023-10-30T13:06:49.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 -- 172.21.15.125:0/3062725108 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ba0071110 msgr2=0x7f4ba006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:49.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 --2- 172.21.15.125:0/3062725108 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ba0071110 0x7f4ba006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:49.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 -- 172.21.15.125:0/3062725108 >> v1:172.21.15.136:6789/0 conn(0x7f4ba00c6bd0 legacy=0x7f4ba0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:49.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 -- 172.21.15.125:0/3062725108 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba00c7390 con 0x7f4ba006f7a0 2023-10-30T13:06:49.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 -- 172.21.15.125:0/3062725108 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f4b9c001680 con 0x7f4ba006f7a0 2023-10-30T13:06:49.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 -- 172.21.15.125:0/3062725108 >> v1:172.21.15.125:6789/0 conn(0x7f4ba006f7a0 legacy=0x7f4ba0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:49.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 --2- 172.21.15.125:0/3062725108 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b880039b0 0x7f4b88003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:49.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 --2- 172.21.15.125:0/3062725108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b88004360 0x7f4b88004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:49.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 --2- 172.21.15.125:0/3062725108 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ba0071110 0x7f4b88005090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:49.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 -- 172.21.15.125:0/3062725108 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba00700c0 con 0x7f4b880039b0 2023-10-30T13:06:49.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 -- 172.21.15.125:0/3062725108 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8800b990 con 0x7f4b88004360 2023-10-30T13:06:49.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b977fe640 1 -- 172.21.15.125:0/3062725108 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8800bb50 con 0x7f4ba0071110 2023-10-30T13:06:49.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4ba4bb3640 1 --2- 172.21.15.125:0/3062725108 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b880039b0 0x7f4b88003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:49.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4b97fff640 1 --2- 172.21.15.125:0/3062725108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b88004360 0x7f4b88004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:49.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4ba53b4640 1 --2- 172.21.15.125:0/3062725108 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ba0071110 0x7f4b88005090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:49.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4ba4bb3640 1 -- 172.21.15.125:0/3062725108 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ba0071110 msgr2=0x7f4b88005090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:49.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4ba4bb3640 1 --2- 172.21.15.125:0/3062725108 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4ba0071110 0x7f4b88005090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:49.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4ba4bb3640 1 -- 172.21.15.125:0/3062725108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b88004360 msgr2=0x7f4b88004830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:49.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4ba4bb3640 1 --2- 172.21.15.125:0/3062725108 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4b88004360 0x7f4b88004830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:49.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.045+0000 7f4ba4bb3640 1 -- 172.21.15.125:0/3062725108 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b88003530 con 0x7f4b880039b0 2023-10-30T13:06:49.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba4bb3640 1 --2- 172.21.15.125:0/3062725108 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b880039b0 0x7f4b88003d90 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00c190 tx=0x7f4b9c01cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65add4eed83748fb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:49.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4b977fe640 1 -- 172.21.15.125:0/3062725108 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c005ae0 con 0x7f4b880039b0 2023-10-30T13:06:49.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4b977fe640 1 -- 172.21.15.125:0/3062725108 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9c005c60 con 0x7f4b880039b0 2023-10-30T13:06:49.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4b977fe640 1 -- 172.21.15.125:0/3062725108 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c0a58e0 con 0x7f4b880039b0 2023-10-30T13:06:49.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba6e17640 1 -- 172.21.15.125:0/3062725108 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b880039b0 msgr2=0x7f4b88003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:49.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba6e17640 1 --2- 172.21.15.125:0/3062725108 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4b880039b0 0x7f4b88003d90 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c00c190 tx=0x7f4b9c01cbe0 comp rx=0 tx=0).stop 2023-10-30T13:06:49.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba53b4640 1 -- 172.21.15.125:0/3062725108 reap_dead start 2023-10-30T13:06:49.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba6e17640 1 -- 172.21.15.125:0/3062725108 shutdown_connections 2023-10-30T13:06:49.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba6e17640 1 -- 172.21.15.125:0/3062725108 >> 172.21.15.125:0/3062725108 conn(0x7f4ba00c7910 msgr2=0x7f4ba00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:49.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba6e17640 1 -- 172.21.15.125:0/3062725108 shutdown_connections 2023-10-30T13:06:49.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba6e17640 1 -- 172.21.15.125:0/3062725108 wait complete. 2023-10-30T13:06:49.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba6e17640 1 Processor -- start 2023-10-30T13:06:49.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba6e17640 1 -- start start 2023-10-30T13:06:49.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba6e17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b880039b0 0x7f4ba0153de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:49.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba6e17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ba0157190 0x7f4ba00aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:49.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba6e17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ba01f1e90 0x7f4ba01f2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:49.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba6e17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba01583c0 con 0x7f4ba0157190 2023-10-30T13:06:49.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba6e17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba0154320 con 0x7f4ba01f1e90 2023-10-30T13:06:49.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.049+0000 7f4ba6e17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba01f2850 con 0x7f4b880039b0 2023-10-30T13:06:49.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4b97fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ba0157190 0x7f4ba00aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:49.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4b97fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ba0157190 0x7f4ba00aecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:54206/0 (socket says 172.21.15.125:54206) 2023-10-30T13:06:49.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4b97fff640 1 -- 172.21.15.125:0/1007394720 learned_addr learned my addr 172.21.15.125:0/1007394720 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:49.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4ba53b4640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ba01f1e90 0x7f4ba01f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:49.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4ba4bb3640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b880039b0 0x7f4ba0153de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:49.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4b97fff640 1 -- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b880039b0 msgr2=0x7f4ba0153de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:49.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4b97fff640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b880039b0 0x7f4ba0153de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:49.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4b97fff640 1 -- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ba01f1e90 msgr2=0x7f4ba01f2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:49.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4b97fff640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ba01f1e90 0x7f4ba01f2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:49.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4b97fff640 1 -- 172.21.15.125:0/1007394720 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba0154db0 con 0x7f4ba0157190 2023-10-30T13:06:49.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4ba53b4640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ba01f1e90 0x7f4ba01f2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:49.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4b97fff640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ba0157190 0x7f4ba00aecb0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f4b90010ef0 tx=0x7f4b9000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:49.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4b957fa640 1 -- 172.21.15.125:0/1007394720 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9000ede0 con 0x7f4ba0157190 2023-10-30T13:06:49.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4b957fa640 1 -- 172.21.15.125:0/1007394720 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9000d030 con 0x7f4ba0157190 2023-10-30T13:06:49.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4b957fa640 1 -- 172.21.15.125:0/1007394720 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b900052b0 con 0x7f4ba0157190 2023-10-30T13:06:49.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4ba6e17640 1 -- 172.21.15.125:0/1007394720 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba01580e0 con 0x7f4ba0157190 2023-10-30T13:06:49.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4ba6e17640 1 -- 172.21.15.125:0/1007394720 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba0157c60 con 0x7f4ba0157190 2023-10-30T13:06:49.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4ba6e17640 1 -- 172.21.15.125:0/1007394720 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba0154f40 con 0x7f4ba0157190 2023-10-30T13:06:49.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4b957fa640 1 -- 172.21.15.125:0/1007394720 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4b90003600 con 0x7f4ba0157190 2023-10-30T13:06:49.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.053+0000 7f4b957fa640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b8c0664a0 0x7f4b8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:49.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.057+0000 7f4b957fa640 1 -- 172.21.15.125:0/1007394720 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f4b90087c90 con 0x7f4ba0157190 2023-10-30T13:06:49.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.057+0000 7f4ba4bb3640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b8c0664a0 0x7f4b8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:49.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.057+0000 7f4ba4bb3640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b8c0664a0 0x7f4b8c068960 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c005fd0 tx=0x7f4b9c0a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:49.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.057+0000 7f4b957fa640 1 -- 172.21.15.125:0/1007394720 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4b90054a20 con 0x7f4ba0157190 2023-10-30T13:06:49.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.237+0000 7f4ba6e17640 1 -- 172.21.15.125:0/1007394720 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ba004e170 con 0x7f4ba0157190 2023-10-30T13:06:49.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.241+0000 7f4b957fa640 1 -- 172.21.15.125:0/1007394720 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6546 (secure 0 0 0) 0x7f4b90018090 con 0x7f4ba0157190 2023-10-30T13:06:49.244 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:49.244 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:42.679260+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8178,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:49.245 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 171 2023-10-30T13:06:49.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.241+0000 7f4ba6e17640 1 -- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b8c0664a0 msgr2=0x7f4b8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:49.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.241+0000 7f4ba6e17640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b8c0664a0 0x7f4b8c068960 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f4b9c005fd0 tx=0x7f4b9c0a7070 comp rx=0 tx=0).stop 2023-10-30T13:06:49.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.241+0000 7f4ba6e17640 1 -- 172.21.15.125:0/1007394720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ba0157190 msgr2=0x7f4ba00aecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:49.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.241+0000 7f4ba6e17640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ba0157190 0x7f4ba00aecb0 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f4b90010ef0 tx=0x7f4b9000e530 comp rx=0 tx=0).stop 2023-10-30T13:06:49.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.241+0000 7f4ba6e17640 1 -- 172.21.15.125:0/1007394720 shutdown_connections 2023-10-30T13:06:49.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.241+0000 7f4ba6e17640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4b8c0664a0 0x7f4b8c068960 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:49.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.241+0000 7f4ba6e17640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4ba01f1e90 0x7f4ba01f2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:49.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.241+0000 7f4ba6e17640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4ba0157190 0x7f4ba00aecb0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:49.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.241+0000 7f4ba6e17640 1 --2- 172.21.15.125:0/1007394720 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4b880039b0 0x7f4ba0153de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:49.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.241+0000 7f4ba6e17640 1 -- 172.21.15.125:0/1007394720 >> 172.21.15.125:0/1007394720 conn(0x7f4ba00c7910 msgr2=0x7f4ba00ca540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:49.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.241+0000 7f4ba6e17640 1 -- 172.21.15.125:0/1007394720 shutdown_connections 2023-10-30T13:06:49.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:49.241+0000 7f4ba6e17640 1 -- 172.21.15.125:0/1007394720 wait complete. 2023-10-30T13:06:49.265 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:06:51.266 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:51.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.409+0000 7fcb4050f640 1 Processor -- start 2023-10-30T13:06:51.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.409+0000 7fcb4050f640 1 -- start start 2023-10-30T13:06:51.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.409+0000 7fcb4050f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fcb381580e0 0x7fcb38070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:51.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.409+0000 7fcb4050f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb3815e700 0x7fcb3806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:51.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.409+0000 7fcb4050f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb380c5810 con 0x7fcb380c6540 2023-10-30T13:06:51.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.409+0000 7fcb4050f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb380c6990 con 0x7fcb3815e700 2023-10-30T13:06:51.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.409+0000 7fcb4050f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb3815eb90 con 0x7fcb381580e0 2023-10-30T13:06:51.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.413+0000 7fcb3eaac640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb3815e700 0x7fcb3806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:51.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.413+0000 7fcb3eaac640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb3815e700 0x7fcb3806a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53254/0 (socket says 172.21.15.125:53254) 2023-10-30T13:06:51.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.413+0000 7fcb3eaac640 1 -- 172.21.15.125:0/3885293148 learned_addr learned my addr 172.21.15.125:0/3885293148 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:51.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.413+0000 7fcb3e2ab640 1 --2- 172.21.15.125:0/3885293148 >> v2:172.21.15.136:3300/0 conn(0x7fcb381580e0 0x7fcb38070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:51.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.413+0000 7fcb3eaac640 1 -- 172.21.15.125:0/3885293148 >> v2:172.21.15.136:3300/0 conn(0x7fcb381580e0 msgr2=0x7fcb38070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:51.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.413+0000 7fcb3eaac640 1 --2- 172.21.15.125:0/3885293148 >> v2:172.21.15.136:3300/0 conn(0x7fcb381580e0 0x7fcb38070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:51.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.413+0000 7fcb3eaac640 1 -- 172.21.15.125:0/3885293148 >> v1:172.21.15.136:6789/0 conn(0x7fcb380c6540 legacy=0x7fcb38069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:51.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.413+0000 7fcb3eaac640 1 -- 172.21.15.125:0/3885293148 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb380c6bb0 con 0x7fcb3815e700 2023-10-30T13:06:51.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.413+0000 7fcb3eaac640 1 --2- 172.21.15.125:0/3885293148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb3815e700 0x7fcb3806a5a0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fcb3806eea0 tx=0x7fcb3001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a0e7eee13262fd4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:51.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.413+0000 7fcb3d2a9640 1 -- 172.21.15.125:0/3885293148 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb300a5860 con 0x7fcb3815e700 2023-10-30T13:06:51.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.413+0000 7fcb3d2a9640 1 -- 172.21.15.125:0/3885293148 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb300a59c0 con 0x7fcb3815e700 2023-10-30T13:06:51.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.413+0000 7fcb3d2a9640 1 -- 172.21.15.125:0/3885293148 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb300a5d50 con 0x7fcb3815e700 2023-10-30T13:06:51.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 -- 172.21.15.125:0/3885293148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb3815e700 msgr2=0x7fcb3806a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:51.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 --2- 172.21.15.125:0/3885293148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb3815e700 0x7fcb3806a5a0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fcb3806eea0 tx=0x7fcb3001c9a0 comp rx=0 tx=0).stop 2023-10-30T13:06:51.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 -- 172.21.15.125:0/3885293148 shutdown_connections 2023-10-30T13:06:51.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 --2- 172.21.15.125:0/3885293148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb3815e700 0x7fcb3806a5a0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:51.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 --2- 172.21.15.125:0/3885293148 >> v2:172.21.15.136:3300/0 conn(0x7fcb381580e0 0x7fcb38070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:51.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 -- 172.21.15.125:0/3885293148 >> 172.21.15.125:0/3885293148 conn(0x7fcb380c7910 msgr2=0x7fcb380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:51.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 -- 172.21.15.125:0/3885293148 shutdown_connections 2023-10-30T13:06:51.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 -- 172.21.15.125:0/3885293148 wait complete. 2023-10-30T13:06:51.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 Processor -- start 2023-10-30T13:06:51.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 -- start start 2023-10-30T13:06:51.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb380c6540 0x7fcb380c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:51.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb381580e0 0x7fcb380c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:51.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb3815e700 0x7fcb380c2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:51.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb38148670 con 0x7fcb380c6540 2023-10-30T13:06:51.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb38148400 con 0x7fcb381580e0 2023-10-30T13:06:51.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb4050f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb380c20b0 con 0x7fcb3815e700 2023-10-30T13:06:51.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb3e2ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb380c6540 0x7fcb380c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:51.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb3e2ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb380c6540 0x7fcb380c8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53266/0 (socket says 172.21.15.125:53266) 2023-10-30T13:06:51.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb3e2ab640 1 -- 172.21.15.125:0/4067921654 learned_addr learned my addr 172.21.15.125:0/4067921654 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:51.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb3eaac640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb3815e700 0x7fcb380c2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:51.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb3daaa640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb381580e0 0x7fcb380c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:51.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb3e2ab640 1 -- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb3815e700 msgr2=0x7fcb380c2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:51.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb3e2ab640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb3815e700 0x7fcb380c2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb3e2ab640 1 -- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb381580e0 msgr2=0x7fcb380c25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb3e2ab640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb381580e0 0x7fcb380c25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb3e2ab640 1 -- 172.21.15.125:0/4067921654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb30009b30 con 0x7fcb380c6540 2023-10-30T13:06:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.417+0000 7fcb3eaac640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb3815e700 0x7fcb380c2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:51.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.421+0000 7fcb3e2ab640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb380c6540 0x7fcb380c8f20 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400e930 tx=0x7fcb3400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:51.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.421+0000 7fcb2effd640 1 -- 172.21.15.125:0/4067921654 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb34009690 con 0x7fcb380c6540 2023-10-30T13:06:51.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.421+0000 7fcb2effd640 1 -- 172.21.15.125:0/4067921654 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb3400ca00 con 0x7fcb380c6540 2023-10-30T13:06:51.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.421+0000 7fcb4050f640 1 -- 172.21.15.125:0/4067921654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb381493b0 con 0x7fcb380c6540 2023-10-30T13:06:51.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.421+0000 7fcb2effd640 1 -- 172.21.15.125:0/4067921654 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb3400cd70 con 0x7fcb380c6540 2023-10-30T13:06:51.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.421+0000 7fcb4050f640 1 -- 172.21.15.125:0/4067921654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb38149970 con 0x7fcb380c6540 2023-10-30T13:06:51.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.421+0000 7fcb4050f640 1 -- 172.21.15.125:0/4067921654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3804e170 con 0x7fcb380c6540 2023-10-30T13:06:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.421+0000 7fcb2effd640 1 -- 172.21.15.125:0/4067921654 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcb34013470 con 0x7fcb380c6540 2023-10-30T13:06:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.421+0000 7fcb2effd640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb1c0664a0 0x7fcb1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.421+0000 7fcb2effd640 1 -- 172.21.15.125:0/4067921654 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7fcb340848e0 con 0x7fcb380c6540 2023-10-30T13:06:51.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.421+0000 7fcb3daaa640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb1c0664a0 0x7fcb1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.425+0000 7fcb3daaa640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb1c0664a0 0x7fcb1c068960 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fcb38072210 tx=0x7fcb28009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.425+0000 7fcb2effd640 1 -- 172.21.15.125:0/4067921654 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcb34051610 con 0x7fcb380c6540 2023-10-30T13:06:51.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.609+0000 7fcb4050f640 1 -- 172.21.15.125:0/4067921654 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb381683e0 con 0x7fcb380c6540 2023-10-30T13:06:51.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.613+0000 7fcb2effd640 1 -- 172.21.15.125:0/4067921654 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6546 (secure 0 0 0) 0x7fcb34017090 con 0x7fcb380c6540 2023-10-30T13:06:51.616 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:51.616 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:42.679260+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8178,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:51.616 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 171 2023-10-30T13:06:51.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.613+0000 7fcb4050f640 1 -- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb1c0664a0 msgr2=0x7fcb1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:51.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.613+0000 7fcb4050f640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb1c0664a0 0x7fcb1c068960 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fcb38072210 tx=0x7fcb28009040 comp rx=0 tx=0).stop 2023-10-30T13:06:51.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.613+0000 7fcb4050f640 1 -- 172.21.15.125:0/4067921654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb380c6540 msgr2=0x7fcb380c8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:51.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.613+0000 7fcb4050f640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb380c6540 0x7fcb380c8f20 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fcb3400e930 tx=0x7fcb3400edf0 comp rx=0 tx=0).stop 2023-10-30T13:06:51.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.613+0000 7fcb4050f640 1 -- 172.21.15.125:0/4067921654 shutdown_connections 2023-10-30T13:06:51.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.613+0000 7fcb4050f640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb1c0664a0 0x7fcb1c068960 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:51.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.613+0000 7fcb4050f640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb3815e700 0x7fcb380c2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:51.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.613+0000 7fcb4050f640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb381580e0 0x7fcb380c25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:51.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.613+0000 7fcb4050f640 1 --2- 172.21.15.125:0/4067921654 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb380c6540 0x7fcb380c8f20 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:51.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.613+0000 7fcb4050f640 1 -- 172.21.15.125:0/4067921654 >> 172.21.15.125:0/4067921654 conn(0x7fcb380c7910 msgr2=0x7fcb38156eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:51.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.613+0000 7fcb4050f640 1 -- 172.21.15.125:0/4067921654 shutdown_connections 2023-10-30T13:06:51.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:51.613+0000 7fcb4050f640 1 -- 172.21.15.125:0/4067921654 wait complete. 2023-10-30T13:06:51.637 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:06:53.133 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:06:53.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:06:53.638 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:53.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.781+0000 7f6b5b772640 1 Processor -- start 2023-10-30T13:06:53.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.781+0000 7f6b5b772640 1 -- start start 2023-10-30T13:06:53.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.781+0000 7f6b5b772640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6b541580e0 0x7f6b54070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:53.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.781+0000 7f6b5b772640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b540c5810 con 0x7f6b540c6540 2023-10-30T13:06:53.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.781+0000 7f6b5b772640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b540c6990 con 0x7f6b5415e700 2023-10-30T13:06:53.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.781+0000 7f6b5b772640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b5406acb0 con 0x7f6b541580e0 2023-10-30T13:06:53.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.781+0000 7f6b5950e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6b541580e0 0x7f6b54070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:53.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.781+0000 7f6b58d0d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6b540c6540 0x7f6b54069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54970/0 (socket says 172.21.15.125:54970) 2023-10-30T13:06:53.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.781+0000 7f6b58d0d640 1 -- 172.21.15.125:0/1336732353 learned_addr learned my addr 172.21.15.125:0/1336732353 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:06:53.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.781+0000 7f6b5950e640 1 -- 172.21.15.125:0/1336732353 >> v1:172.21.15.136:6789/0 conn(0x7f6b5415e700 legacy=0x7f6b5406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:53.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.781+0000 7f6b5950e640 1 -- 172.21.15.125:0/1336732353 >> v1:172.21.15.125:6789/0 conn(0x7f6b540c6540 legacy=0x7f6b54069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:53.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.781+0000 7f6b5950e640 1 -- 172.21.15.125:0/1336732353 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5415eb90 con 0x7f6b541580e0 2023-10-30T13:06:53.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.781+0000 7f6b4bfff640 1 -- 172.21.15.125:0/1336732353 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930237523 0 0) 0x7f6b540c5810 con 0x7f6b540c6540 2023-10-30T13:06:53.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.785+0000 7f6b5950e640 1 --2- 172.21.15.125:0/1336732353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b541580e0 0x7f6b54070740 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f6b500097c0 tx=0x7f6b5001c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa5614a169746e1a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:53.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.785+0000 7f6b4bfff640 1 -- 172.21.15.125:0/1336732353 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b500ab640 con 0x7f6b541580e0 2023-10-30T13:06:53.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.785+0000 7f6b4bfff640 1 -- 172.21.15.125:0/1336732353 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b500a7070 con 0x7f6b541580e0 2023-10-30T13:06:53.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.785+0000 7f6b4bfff640 1 -- 172.21.15.125:0/1336732353 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b500ab9c0 con 0x7f6b541580e0 2023-10-30T13:06:53.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.785+0000 7f6b5b772640 1 -- 172.21.15.125:0/1336732353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b541580e0 msgr2=0x7f6b54070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:53.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.785+0000 7f6b5b772640 1 --2- 172.21.15.125:0/1336732353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b541580e0 0x7f6b54070740 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7f6b500097c0 tx=0x7f6b5001c440 comp rx=0 tx=0).stop 2023-10-30T13:06:53.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.785+0000 7f6b5b772640 1 -- 172.21.15.125:0/1336732353 shutdown_connections 2023-10-30T13:06:53.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.785+0000 7f6b5b772640 1 --2- 172.21.15.125:0/1336732353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b541580e0 0x7f6b54070740 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:53.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.785+0000 7f6b5b772640 1 -- 172.21.15.125:0/1336732353 >> 172.21.15.125:0/1336732353 conn(0x7f6b540c7910 msgr2=0x7f6b540c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:53.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.785+0000 7f6b5b772640 1 -- 172.21.15.125:0/1336732353 shutdown_connections 2023-10-30T13:06:53.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.785+0000 7f6b5b772640 1 -- 172.21.15.125:0/1336732353 wait complete. 2023-10-30T13:06:53.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b5b772640 1 Processor -- start 2023-10-30T13:06:53.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b5b772640 1 -- start start 2023-10-30T13:06:53.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b5b772640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b540c6540 0x7f6b541f6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:53.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b5b772640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b541580e0 0x7f6b54160450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:53.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b5b772640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b5415e700 0x7f6b541f9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:53.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b5b772640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b541f64c0 con 0x7f6b5415e700 2023-10-30T13:06:53.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b5b772640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b541fa260 con 0x7f6b541580e0 2023-10-30T13:06:53.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b5b772640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b541543d0 con 0x7f6b540c6540 2023-10-30T13:06:53.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b59d0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b5415e700 0x7f6b541f9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:53.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b59d0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b5415e700 0x7f6b541f9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53288/0 (socket says 172.21.15.125:53288) 2023-10-30T13:06:53.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b59d0f640 1 -- 172.21.15.125:0/3171064530 learned_addr learned my addr 172.21.15.125:0/3171064530 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:53.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b58d0d640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b541580e0 0x7f6b54160450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:53.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b5950e640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b540c6540 0x7f6b541f6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:53.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b59d0f640 1 -- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b540c6540 msgr2=0x7f6b541f6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:53.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b59d0f640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b540c6540 0x7f6b541f6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:53.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b59d0f640 1 -- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b541580e0 msgr2=0x7f6b54160450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:53.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b59d0f640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b541580e0 0x7f6b54160450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:53.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b59d0f640 1 -- 172.21.15.125:0/3171064530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b50009420 con 0x7f6b5415e700 2023-10-30T13:06:53.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b58d0d640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b541580e0 0x7f6b54160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:06:53.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b59d0f640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b5415e700 0x7f6b541f9d20 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00b970 tx=0x7f6b4c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:53.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b49ffb640 1 -- 172.21.15.125:0/3171064530 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c00cae0 con 0x7f6b5415e700 2023-10-30T13:06:53.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b49ffb640 1 -- 172.21.15.125:0/3171064530 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4c00cc40 con 0x7f6b5415e700 2023-10-30T13:06:53.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b5b772640 1 -- 172.21.15.125:0/3171064530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b541f5eb0 con 0x7f6b5415e700 2023-10-30T13:06:53.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b49ffb640 1 -- 172.21.15.125:0/3171064530 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c014680 con 0x7f6b5415e700 2023-10-30T13:06:53.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.789+0000 7f6b5b772640 1 -- 172.21.15.125:0/3171064530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b541f6150 con 0x7f6b5415e700 2023-10-30T13:06:53.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.793+0000 7f6b5b772640 1 -- 172.21.15.125:0/3171064530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b541f6670 con 0x7f6b5415e700 2023-10-30T13:06:53.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.793+0000 7f6b49ffb640 1 -- 172.21.15.125:0/3171064530 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6b4c01d3b0 con 0x7f6b5415e700 2023-10-30T13:06:53.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.793+0000 7f6b49ffb640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6b380664d0 0x7f6b38068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:53.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.793+0000 7f6b49ffb640 1 -- 172.21.15.125:0/3171064530 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 7458+0+0 (secure 0 0 0) 0x7f6b4c084320 con 0x7f6b5415e700 2023-10-30T13:06:53.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.793+0000 7f6b5950e640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6b380664d0 0x7f6b38068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:53.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.793+0000 7f6b5950e640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6b380664d0 0x7f6b38068990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f6b500af000 tx=0x7f6b500a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:53.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.797+0000 7f6b49ffb640 1 -- 172.21.15.125:0/3171064530 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6b541f6670 con 0x7f6b5415e700 2023-10-30T13:06:53.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.981+0000 7f6b5b772640 1 -- 172.21.15.125:0/3171064530 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b541f6670 con 0x7f6b5415e700 2023-10-30T13:06:53.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.981+0000 7f6b49ffb640 1 -- 172.21.15.125:0/3171064530 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+6546 (secure 0 0 0) 0x7f6b541f6670 con 0x7f6b5415e700 2023-10-30T13:06:53.985 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:53.985 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":5234,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:42.679260+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8178,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8178":{"gid":8178,"name":"e","rank":0,"incarnation":162,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3711687505","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3711687505},{"type":"v1","addr":"172.21.15.125:6836","nonce":3711687505}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:53.986 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 171 2023-10-30T13:06:53.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.981+0000 7f6b5b772640 1 -- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6b380664d0 msgr2=0x7f6b38068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:53.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.981+0000 7f6b5b772640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6b380664d0 0x7f6b38068990 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f6b500af000 tx=0x7f6b500a7000 comp rx=0 tx=0).stop 2023-10-30T13:06:53.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.981+0000 7f6b5b772640 1 -- 172.21.15.125:0/3171064530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b5415e700 msgr2=0x7f6b541f9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:53.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.981+0000 7f6b5b772640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b5415e700 0x7f6b541f9d20 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7f6b4c00b970 tx=0x7f6b4c00be30 comp rx=0 tx=0).stop 2023-10-30T13:06:53.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.981+0000 7f6b5b772640 1 -- 172.21.15.125:0/3171064530 shutdown_connections 2023-10-30T13:06:53.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.981+0000 7f6b5b772640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6b380664d0 0x7f6b38068990 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:53.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.981+0000 7f6b5b772640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6b5415e700 0x7f6b541f9d20 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:53.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.981+0000 7f6b5b772640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6b541580e0 0x7f6b54160450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:53.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.981+0000 7f6b5b772640 1 --2- 172.21.15.125:0/3171064530 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6b540c6540 0x7f6b541f6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:53.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.981+0000 7f6b5b772640 1 -- 172.21.15.125:0/3171064530 >> 172.21.15.125:0/3171064530 conn(0x7f6b540c7910 msgr2=0x7f6b54155910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:53.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.981+0000 7f6b5b772640 1 -- 172.21.15.125:0/3171064530 shutdown_connections 2023-10-30T13:06:53.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:53.981+0000 7f6b5b772640 1 -- 172.21.15.125:0/3171064530 wait complete. 2023-10-30T13:06:54.006 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:06:56.007 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:56.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.149+0000 7fc69f3aa640 1 Processor -- start 2023-10-30T13:06:56.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.149+0000 7fc69f3aa640 1 -- start start 2023-10-30T13:06:56.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.149+0000 7fc69f3aa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc698070740 0x7fc6980c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:56.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.149+0000 7fc69f3aa640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6980c5810 con 0x7fc69815e700 2023-10-30T13:06:56.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.149+0000 7fc69f3aa640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69815eb70 con 0x7fc6981580e0 2023-10-30T13:06:56.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.149+0000 7fc69f3aa640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc69806acb0 con 0x7fc698070740 2023-10-30T13:06:56.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.149+0000 7fc69c945640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc698070740 0x7fc6980c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:56.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.149+0000 7fc69d947640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc69815e700 0x7fc69806a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54980/0 (socket says 172.21.15.125:54980) 2023-10-30T13:06:56.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.149+0000 7fc69d947640 1 -- 172.21.15.125:0/3833145200 learned_addr learned my addr 172.21.15.125:0/3833145200 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:06:56.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.149+0000 7fc69c945640 1 -- 172.21.15.125:0/3833145200 >> v1:172.21.15.136:6789/0 conn(0x7fc6981580e0 legacy=0x7fc698069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:56.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.149+0000 7fc69c945640 1 -- 172.21.15.125:0/3833145200 >> v1:172.21.15.125:6789/0 conn(0x7fc69815e700 legacy=0x7fc69806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:56.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.149+0000 7fc68ffff640 1 -- 172.21.15.125:0/3833145200 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035025765 0 0) 0x7fc6980c5810 con 0x7fc69815e700 2023-10-30T13:06:56.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.149+0000 7fc69c945640 1 -- 172.21.15.125:0/3833145200 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6980c6bb0 con 0x7fc698070740 2023-10-30T13:06:56.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.153+0000 7fc69c945640 1 --2- 172.21.15.125:0/3833145200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc698070740 0x7fc6980c6540 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fc6880097c0 tx=0x7fc68801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d905ac8122ed9109 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:56.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.153+0000 7fc68ffff640 1 -- 172.21.15.125:0/3833145200 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6880ab600 con 0x7fc698070740 2023-10-30T13:06:56.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.153+0000 7fc68ffff640 1 -- 172.21.15.125:0/3833145200 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6880a7050 con 0x7fc698070740 2023-10-30T13:06:56.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.153+0000 7fc68ffff640 1 -- 172.21.15.125:0/3833145200 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6880ab980 con 0x7fc698070740 2023-10-30T13:06:56.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.153+0000 7fc69f3aa640 1 -- 172.21.15.125:0/3833145200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc698070740 msgr2=0x7fc6980c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:56.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.153+0000 7fc69f3aa640 1 --2- 172.21.15.125:0/3833145200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc698070740 0x7fc6980c6540 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fc6880097c0 tx=0x7fc68801c480 comp rx=0 tx=0).stop 2023-10-30T13:06:56.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.153+0000 7fc69f3aa640 1 -- 172.21.15.125:0/3833145200 shutdown_connections 2023-10-30T13:06:56.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.153+0000 7fc69f3aa640 1 --2- 172.21.15.125:0/3833145200 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc698070740 0x7fc6980c6540 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.153+0000 7fc69f3aa640 1 -- 172.21.15.125:0/3833145200 >> 172.21.15.125:0/3833145200 conn(0x7fc6980c7910 msgr2=0x7fc6980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:56.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.153+0000 7fc69f3aa640 1 -- 172.21.15.125:0/3833145200 shutdown_connections 2023-10-30T13:06:56.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.153+0000 7fc69f3aa640 1 -- 172.21.15.125:0/3833145200 wait complete. 2023-10-30T13:06:56.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69f3aa640 1 Processor -- start 2023-10-30T13:06:56.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69f3aa640 1 -- start start 2023-10-30T13:06:56.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69f3aa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc698070740 0x7fc6980c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:56.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69f3aa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6981580e0 0x7fc6981537a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:56.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69f3aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc69815e700 0x7fc698153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69f3aa640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc698153310 con 0x7fc698070740 2023-10-30T13:06:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69f3aa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6981490d0 con 0x7fc6981580e0 2023-10-30T13:06:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69f3aa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc698143ad0 con 0x7fc69815e700 2023-10-30T13:06:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69d146640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc698070740 0x7fc6980c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69d146640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc698070740 0x7fc6980c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53304/0 (socket says 172.21.15.125:53304) 2023-10-30T13:06:56.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69d146640 1 -- 172.21.15.125:0/1191180263 learned_addr learned my addr 172.21.15.125:0/1191180263 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:56.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69c945640 1 --2- 172.21.15.125:0/1191180263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6981580e0 0x7fc6981537a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:56.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69d947640 1 --2- 172.21.15.125:0/1191180263 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc69815e700 0x7fc698153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:56.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69d146640 1 -- 172.21.15.125:0/1191180263 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc69815e700 msgr2=0x7fc698153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:56.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69d146640 1 --2- 172.21.15.125:0/1191180263 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc69815e700 0x7fc698153ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69d146640 1 -- 172.21.15.125:0/1191180263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6981580e0 msgr2=0x7fc6981537a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:56.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69d146640 1 --2- 172.21.15.125:0/1191180263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6981580e0 0x7fc6981537a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69d146640 1 -- 172.21.15.125:0/1191180263 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc688009420 con 0x7fc698070740 2023-10-30T13:06:56.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69d146640 1 --2- 172.21.15.125:0/1191180263 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc698070740 0x7fc6980c12a0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fc69400c500 tx=0x7fc69400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:56.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc68dffb640 1 -- 172.21.15.125:0/1191180263 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc694005220 con 0x7fc698070740 2023-10-30T13:06:56.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc68dffb640 1 -- 172.21.15.125:0/1191180263 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc694005380 con 0x7fc698070740 2023-10-30T13:06:56.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc69f3aa640 1 -- 172.21.15.125:0/1191180263 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc698149390 con 0x7fc698070740 2023-10-30T13:06:56.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.157+0000 7fc68dffb640 1 -- 172.21.15.125:0/1191180263 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc694007270 con 0x7fc698070740 2023-10-30T13:06:56.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.161+0000 7fc69f3aa640 1 -- 172.21.15.125:0/1191180263 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc698152fa0 con 0x7fc698070740 2023-10-30T13:06:56.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.161+0000 7fc69f3aa640 1 -- 172.21.15.125:0/1191180263 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc69804e170 con 0x7fc698070740 2023-10-30T13:06:56.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.161+0000 7fc68dffb640 1 -- 172.21.15.125:0/1191180263 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc6940073d0 con 0x7fc698070740 2023-10-30T13:06:56.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.161+0000 7fc68dffb640 1 --2- 172.21.15.125:0/1191180263 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc67c0664a0 0x7fc67c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:56.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.161+0000 7fc68dffb640 1 -- 172.21.15.125:0/1191180263 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7fc694083ef0 con 0x7fc698070740 2023-10-30T13:06:56.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.161+0000 7fc69c945640 1 --2- 172.21.15.125:0/1191180263 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc67c0664a0 0x7fc67c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:56.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.165+0000 7fc69c945640 1 --2- 172.21.15.125:0/1191180263 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc67c0664a0 0x7fc67c068960 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc6880ad000 tx=0x7fc688008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:56.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.165+0000 7fc68dffb640 1 -- 172.21.15.125:0/1191180263 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc694050bb0 con 0x7fc698070740 2023-10-30T13:06:56.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.345+0000 7fc69f3aa640 1 -- 172.21.15.125:0/1191180263 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6980c5a30 con 0x7fc698070740 2023-10-30T13:06:56.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.345+0000 7fc68dffb640 1 -- 172.21.15.125:0/1191180263 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+5751 (secure 0 0 0) 0x7fc6940541d0 con 0x7fc698070740 2023-10-30T13:06:56.351 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:56.351 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:55.602089+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:56.351 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 172 2023-10-30T13:06:56.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.349+0000 7fc69f3aa640 1 -- 172.21.15.125:0/1191180263 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc67c0664a0 msgr2=0x7fc67c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:56.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.349+0000 7fc69f3aa640 1 --2- 172.21.15.125:0/1191180263 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc67c0664a0 0x7fc67c068960 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc6880ad000 tx=0x7fc688008040 comp rx=0 tx=0).stop 2023-10-30T13:06:56.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.349+0000 7fc69f3aa640 1 -- 172.21.15.125:0/1191180263 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc698070740 msgr2=0x7fc6980c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:56.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.349+0000 7fc69f3aa640 1 --2- 172.21.15.125:0/1191180263 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc698070740 0x7fc6980c12a0 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fc69400c500 tx=0x7fc69400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.349+0000 7fc69f3aa640 1 -- 172.21.15.125:0/1191180263 shutdown_connections 2023-10-30T13:06:56.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.349+0000 7fc69f3aa640 1 --2- 172.21.15.125:0/1191180263 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc67c0664a0 0x7fc67c068960 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.349+0000 7fc69f3aa640 1 --2- 172.21.15.125:0/1191180263 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc69815e700 0x7fc698153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.349+0000 7fc69f3aa640 1 --2- 172.21.15.125:0/1191180263 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc6981580e0 0x7fc6981537a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.349+0000 7fc69f3aa640 1 --2- 172.21.15.125:0/1191180263 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc698070740 0x7fc6980c12a0 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.349+0000 7fc69f3aa640 1 -- 172.21.15.125:0/1191180263 >> 172.21.15.125:0/1191180263 conn(0x7fc6980c7910 msgr2=0x7fc698146440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:56.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.349+0000 7fc69f3aa640 1 -- 172.21.15.125:0/1191180263 shutdown_connections 2023-10-30T13:06:56.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.349+0000 7fc69f3aa640 1 -- 172.21.15.125:0/1191180263 wait complete. 2023-10-30T13:06:56.372 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T13:06:56.372 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:06:56.372 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:56.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.505+0000 7fb552e46640 1 Processor -- start 2023-10-30T13:06:56.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.505+0000 7fb552e46640 1 -- start start 2023-10-30T13:06:56.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.505+0000 7fb552e46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb54c071110 0x7fb54c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:56.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.505+0000 7fb552e46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54c066cf0 con 0x7fb54c06f7a0 2023-10-30T13:06:56.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.505+0000 7fb552e46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb54c0c70d0 con 0x7fb54c0c6bd0 2023-10-30T13:06:56.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.505+0000 7fb552e46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb54c0700c0 con 0x7fb54c071110 2023-10-30T13:06:56.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.505+0000 7fb5513e3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb54c071110 0x7fb54c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:56.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.505+0000 7fb5513e3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb54c071110 0x7fb54c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53318/0 (socket says 172.21.15.125:53318) 2023-10-30T13:06:56.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.505+0000 7fb5513e3640 1 -- 172.21.15.125:0/3633757502 learned_addr learned my addr 172.21.15.125:0/3633757502 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:56.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.505+0000 7fb550be2640 1 --1- 172.21.15.125:0/3633757502 >> v1:172.21.15.125:6789/0 conn(0x7fb54c06f7a0 0x7fb54c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54986/0 (socket says 172.21.15.125:54986) 2023-10-30T13:06:56.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.509+0000 7fb5513e3640 1 -- 172.21.15.125:0/3633757502 >> v1:172.21.15.136:6789/0 conn(0x7fb54c0c6bd0 legacy=0x7fb54c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:56.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.509+0000 7fb5513e3640 1 -- 172.21.15.125:0/3633757502 >> v1:172.21.15.125:6789/0 conn(0x7fb54c06f7a0 legacy=0x7fb54c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:56.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.509+0000 7fb5513e3640 1 -- 172.21.15.125:0/3633757502 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c0c7390 con 0x7fb54c071110 2023-10-30T13:06:56.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.509+0000 7fb5513e3640 1 --2- 172.21.15.125:0/3633757502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb54c071110 0x7fb54c06fb80 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fb544009e50 tx=0x7fb54401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5358f3809ca035 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:56.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.509+0000 7fb54b7fe640 1 -- 172.21.15.125:0/3633757502 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5440af400 con 0x7fb54c071110 2023-10-30T13:06:56.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.509+0000 7fb54b7fe640 1 -- 172.21.15.125:0/3633757502 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5440ab050 con 0x7fb54c071110 2023-10-30T13:06:56.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.509+0000 7fb54b7fe640 1 -- 172.21.15.125:0/3633757502 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5440af780 con 0x7fb54c071110 2023-10-30T13:06:56.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.509+0000 7fb552e46640 1 -- 172.21.15.125:0/3633757502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb54c071110 msgr2=0x7fb54c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:56.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.509+0000 7fb552e46640 1 --2- 172.21.15.125:0/3633757502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb54c071110 0x7fb54c06fb80 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fb544009e50 tx=0x7fb54401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb552e46640 1 -- 172.21.15.125:0/3633757502 shutdown_connections 2023-10-30T13:06:56.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb552e46640 1 --2- 172.21.15.125:0/3633757502 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb54c071110 0x7fb54c06fb80 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb552e46640 1 -- 172.21.15.125:0/3633757502 >> 172.21.15.125:0/3633757502 conn(0x7fb54c0c7910 msgr2=0x7fb54c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:56.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb552e46640 1 -- 172.21.15.125:0/3633757502 shutdown_connections 2023-10-30T13:06:56.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb552e46640 1 -- 172.21.15.125:0/3633757502 wait complete. 2023-10-30T13:06:56.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb552e46640 1 Processor -- start 2023-10-30T13:06:56.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb552e46640 1 -- start start 2023-10-30T13:06:56.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb552e46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb54c06f7a0 0x7fb54c1dd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:56.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb552e46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb54c071110 0x7fb54c154690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:56.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb552e46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb54c0c6bd0 0x7fb54c1de500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:56.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb552e46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c155300 con 0x7fb54c06f7a0 2023-10-30T13:06:56.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb552e46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c156bb0 con 0x7fb54c0c6bd0 2023-10-30T13:06:56.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb552e46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c1e42a0 con 0x7fb54c071110 2023-10-30T13:06:56.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb550be2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb54c06f7a0 0x7fb54c1dd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:56.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb550be2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb54c06f7a0 0x7fb54c1dd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53332/0 (socket says 172.21.15.125:53332) 2023-10-30T13:06:56.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb550be2640 1 -- 172.21.15.125:0/360741311 learned_addr learned my addr 172.21.15.125:0/360741311 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:56.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb5513e3640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb54c0c6bd0 0x7fb54c1de500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:56.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb54bfff640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb54c071110 0x7fb54c154690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:56.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb550be2640 1 -- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb54c071110 msgr2=0x7fb54c154690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:56.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb550be2640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb54c071110 0x7fb54c154690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb550be2640 1 -- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb54c0c6bd0 msgr2=0x7fb54c1de500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:56.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb550be2640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb54c0c6bd0 0x7fb54c1de500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb550be2640 1 -- 172.21.15.125:0/360741311 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb544009b30 con 0x7fb54c06f7a0 2023-10-30T13:06:56.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb54bfff640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb54c071110 0x7fb54c154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:06:56.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb550be2640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb54c06f7a0 0x7fb54c1dd7c0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fb54000c430 tx=0x7fb54000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:56.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.513+0000 7fb5497fa640 1 -- 172.21.15.125:0/360741311 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5400074e0 con 0x7fb54c06f7a0 2023-10-30T13:06:56.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.517+0000 7fb5497fa640 1 -- 172.21.15.125:0/360741311 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb540007640 con 0x7fb54c06f7a0 2023-10-30T13:06:56.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.517+0000 7fb552e46640 1 -- 172.21.15.125:0/360741311 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb54c150d70 con 0x7fb54c06f7a0 2023-10-30T13:06:56.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.517+0000 7fb5497fa640 1 -- 172.21.15.125:0/360741311 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb540004a10 con 0x7fb54c06f7a0 2023-10-30T13:06:56.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.517+0000 7fb552e46640 1 -- 172.21.15.125:0/360741311 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb54c155830 con 0x7fb54c06f7a0 2023-10-30T13:06:56.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.517+0000 7fb552e46640 1 -- 172.21.15.125:0/360741311 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb54c155a20 con 0x7fb54c06f7a0 2023-10-30T13:06:56.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.517+0000 7fb5497fa640 1 -- 172.21.15.125:0/360741311 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb540004b70 con 0x7fb54c06f7a0 2023-10-30T13:06:56.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.517+0000 7fb5497fa640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5300664a0 0x7fb530068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:56.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.517+0000 7fb5497fa640 1 -- 172.21.15.125:0/360741311 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7fb540083fe0 con 0x7fb54c06f7a0 2023-10-30T13:06:56.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.517+0000 7fb54bfff640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5300664a0 0x7fb530068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:56.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.521+0000 7fb54bfff640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5300664a0 0x7fb530068960 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0c4b80 tx=0x7fb538009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:56.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.521+0000 7fb5497fa640 1 -- 172.21.15.125:0/360741311 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb540050d20 con 0x7fb54c06f7a0 2023-10-30T13:06:56.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.701+0000 7fb552e46640 1 -- 172.21.15.125:0/360741311 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb54c04e170 con 0x7fb54c06f7a0 2023-10-30T13:06:56.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.701+0000 7fb5497fa640 1 -- 172.21.15.125:0/360741311 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+5751 (secure 0 0 0) 0x7fb540054340 con 0x7fb54c06f7a0 2023-10-30T13:06:56.708 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:56.708 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:55.602089+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:56.708 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 172 2023-10-30T13:06:56.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.705+0000 7fb552e46640 1 -- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5300664a0 msgr2=0x7fb530068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:56.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.705+0000 7fb552e46640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5300664a0 0x7fb530068960 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fb54c0c4b80 tx=0x7fb538009040 comp rx=0 tx=0).stop 2023-10-30T13:06:56.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.705+0000 7fb552e46640 1 -- 172.21.15.125:0/360741311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb54c06f7a0 msgr2=0x7fb54c1dd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:56.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.705+0000 7fb552e46640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb54c06f7a0 0x7fb54c1dd7c0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fb54000c430 tx=0x7fb54000c8f0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.705+0000 7fb552e46640 1 -- 172.21.15.125:0/360741311 shutdown_connections 2023-10-30T13:06:56.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.705+0000 7fb552e46640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5300664a0 0x7fb530068960 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.705+0000 7fb552e46640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb54c0c6bd0 0x7fb54c1de500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.705+0000 7fb552e46640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb54c071110 0x7fb54c154690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.705+0000 7fb552e46640 1 --2- 172.21.15.125:0/360741311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb54c06f7a0 0x7fb54c1dd7c0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:56.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.705+0000 7fb552e46640 1 -- 172.21.15.125:0/360741311 >> 172.21.15.125:0/360741311 conn(0x7fb54c0c7910 msgr2=0x7fb54c0c3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:56.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.705+0000 7fb552e46640 1 -- 172.21.15.125:0/360741311 shutdown_connections 2023-10-30T13:06:56.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:56.705+0000 7fb552e46640 1 -- 172.21.15.125:0/360741311 wait complete. 2023-10-30T13:06:56.729 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 172, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 172, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5234": { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 172, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_6017": { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 170, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 251 }, "gid_8301": { "addr": "172.21.15.125:6837/2540063352", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2540063352, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2540063352, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8301, "incarnation": 168, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 75, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:06:55.602089+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5234, "mds_1": 8301, "mds_2": 6017 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:06:58.730 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:06:58.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.873+0000 7ff2021c4640 1 Processor -- start 2023-10-30T13:06:58.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.873+0000 7ff2021c4640 1 -- start start 2023-10-30T13:06:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.877+0000 7ff2021c4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff1fc0c6b80 0x7ff1fc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.877+0000 7ff2021c4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff1fc13f4a0 0x7ff1fc0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.877+0000 7ff2021c4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc0c0a50 con 0x7ff1fc0c6f60 2023-10-30T13:06:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.877+0000 7ff2021c4640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1fc06f6e0 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.877+0000 7ff2021c4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1fc06f9c0 con 0x7ff1fc0c6b80 2023-10-30T13:06:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.877+0000 7ff1fbfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff1fc13f4a0 0x7ff1fc0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.877+0000 7ff1fbfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff1fc13f4a0 0x7ff1fc0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53336/0 (socket says 172.21.15.125:53336) 2023-10-30T13:06:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.877+0000 7ff1fbfff640 1 -- 172.21.15.125:0/1925080355 learned_addr learned my addr 172.21.15.125:0/1925080355 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:58.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.877+0000 7ff1fb7fe640 1 --2- 172.21.15.125:0/1925080355 >> v2:172.21.15.136:3300/0 conn(0x7ff1fc0c6b80 0x7ff1fc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:58.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.877+0000 7ff1fbfff640 1 -- 172.21.15.125:0/1925080355 >> v2:172.21.15.136:3300/0 conn(0x7ff1fc0c6b80 msgr2=0x7ff1fc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:06:58.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.877+0000 7ff1fbfff640 1 --2- 172.21.15.125:0/1925080355 >> v2:172.21.15.136:3300/0 conn(0x7ff1fc0c6b80 0x7ff1fc0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:58.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.877+0000 7ff1fbfff640 1 -- 172.21.15.125:0/1925080355 >> v1:172.21.15.136:6789/0 conn(0x7ff1fc0c6f60 legacy=0x7ff1fc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:58.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.877+0000 7ff1fbfff640 1 -- 172.21.15.125:0/1925080355 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc06fca0 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.877+0000 7ff1fbfff640 1 --2- 172.21.15.125:0/1925080355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1fc13f4a0 0x7ff1fc0c0400 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc06f1b0 tx=0x7ff1f401c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1dfc950eeb7680f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:58.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff1fa7fc640 1 -- 172.21.15.125:0/1925080355 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f40a57a0 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff1fa7fc640 1 -- 172.21.15.125:0/1925080355 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f40a5900 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff1fa7fc640 1 -- 172.21.15.125:0/1925080355 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f40a5c90 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 -- 172.21.15.125:0/1925080355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1fc13f4a0 msgr2=0x7ff1fc0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:58.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 --2- 172.21.15.125:0/1925080355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1fc13f4a0 0x7ff1fc0c0400 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc06f1b0 tx=0x7ff1f401c9d0 comp rx=0 tx=0).stop 2023-10-30T13:06:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 -- 172.21.15.125:0/1925080355 shutdown_connections 2023-10-30T13:06:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 --2- 172.21.15.125:0/1925080355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1fc13f4a0 0x7ff1fc0c0400 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 --2- 172.21.15.125:0/1925080355 >> v2:172.21.15.136:3300/0 conn(0x7ff1fc0c6b80 0x7ff1fc0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 -- 172.21.15.125:0/1925080355 >> 172.21.15.125:0/1925080355 conn(0x7ff1fc0c7910 msgr2=0x7ff1fc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:58.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 -- 172.21.15.125:0/1925080355 shutdown_connections 2023-10-30T13:06:58.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 -- 172.21.15.125:0/1925080355 wait complete. 2023-10-30T13:06:58.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 Processor -- start 2023-10-30T13:06:58.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 -- start start 2023-10-30T13:06:58.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1fc0c6b80 0x7ff1fc131760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:58.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1fc0c6f60 0x7ff1fc0c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:58.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1fc13f4a0 0x7ff1fc1325e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:58.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc1f3b00 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc131ca0 con 0x7ff1fc0c6f60 2023-10-30T13:06:58.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff2021c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc0ca4f0 con 0x7ff1fc0c6b80 2023-10-30T13:06:58.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.881+0000 7ff1fbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1fc13f4a0 0x7ff1fc1325e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:58.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1fbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1fc13f4a0 0x7ff1fc1325e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53352/0 (socket says 172.21.15.125:53352) 2023-10-30T13:06:58.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1fbfff640 1 -- 172.21.15.125:0/3391833175 learned_addr learned my addr 172.21.15.125:0/3391833175 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:06:58.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1fb7fe640 1 --2- 172.21.15.125:0/3391833175 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1fc0c6b80 0x7ff1fc131760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:58.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1faffd640 1 --2- 172.21.15.125:0/3391833175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1fc0c6f60 0x7ff1fc0c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:58.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1fbfff640 1 -- 172.21.15.125:0/3391833175 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1fc0c6b80 msgr2=0x7ff1fc131760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:58.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1fbfff640 1 --2- 172.21.15.125:0/3391833175 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1fc0c6b80 0x7ff1fc131760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:58.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1fbfff640 1 -- 172.21.15.125:0/3391833175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1fc0c6f60 msgr2=0x7ff1fc0c9860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:58.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1fbfff640 1 --2- 172.21.15.125:0/3391833175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1fc0c6f60 0x7ff1fc0c9860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:58.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1fbfff640 1 -- 172.21.15.125:0/3391833175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f4009b30 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1fbfff640 1 --2- 172.21.15.125:0/3391833175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1fc13f4a0 0x7ff1fc1325e0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40af040 tx=0x7ff1f4019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:58.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1e3fff640 1 -- 172.21.15.125:0/3391833175 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f40b8880 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1e3fff640 1 -- 172.21.15.125:0/3391833175 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f40b8a00 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1e3fff640 1 -- 172.21.15.125:0/3391833175 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f40b8db0 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff2021c4640 1 -- 172.21.15.125:0/3391833175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1fc131fc0 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff2021c4640 1 -- 172.21.15.125:0/3391833175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1fc06a5d0 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff2021c4640 1 -- 172.21.15.125:0/3391833175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1fc132300 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1e3fff640 1 -- 172.21.15.125:0/3391833175 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff1f40c1410 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.885+0000 7ff1e3fff640 1 --2- 172.21.15.125:0/3391833175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1e80664f0 0x7ff1e80689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:06:58.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.889+0000 7ff1e3fff640 1 -- 172.21.15.125:0/3391833175 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7ff1f411ff20 con 0x7ff1fc13f4a0 2023-10-30T13:06:58.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.889+0000 7ff1fb7fe640 1 --2- 172.21.15.125:0/3391833175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1e80664f0 0x7ff1e80689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:06:58.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.889+0000 7ff1fb7fe640 1 --2- 172.21.15.125:0/3391833175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1e80664f0 0x7ff1e80689b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec007a60 tx=0x7ff1ec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:06:58.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:58.893+0000 7ff1e3fff640 1 -- 172.21.15.125:0/3391833175 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff1fc132300 con 0x7ff1fc13f4a0 2023-10-30T13:06:59.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:59.069+0000 7ff2021c4640 1 -- 172.21.15.125:0/3391833175 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1fc1322e0 con 0x7ff1fc13f4a0 2023-10-30T13:06:59.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:59.069+0000 7ff1e3fff640 1 -- 172.21.15.125:0/3391833175 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+5751 (secure 0 0 0) 0x7ff1f40f0a40 con 0x7ff1fc13f4a0 2023-10-30T13:06:59.075 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:06:59.076 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:55.602089+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:06:59.076 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 172 2023-10-30T13:06:59.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:59.073+0000 7ff2021c4640 1 -- 172.21.15.125:0/3391833175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1e80664f0 msgr2=0x7ff1e80689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:59.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:59.073+0000 7ff2021c4640 1 --2- 172.21.15.125:0/3391833175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1e80664f0 0x7ff1e80689b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec007a60 tx=0x7ff1ec008040 comp rx=0 tx=0).stop 2023-10-30T13:06:59.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:59.073+0000 7ff2021c4640 1 -- 172.21.15.125:0/3391833175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1fc13f4a0 msgr2=0x7ff1fc1325e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:06:59.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:59.073+0000 7ff2021c4640 1 --2- 172.21.15.125:0/3391833175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1fc13f4a0 0x7ff1fc1325e0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7ff1f40af040 tx=0x7ff1f4019770 comp rx=0 tx=0).stop 2023-10-30T13:06:59.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:59.073+0000 7ff2021c4640 1 -- 172.21.15.125:0/3391833175 shutdown_connections 2023-10-30T13:06:59.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:59.073+0000 7ff2021c4640 1 --2- 172.21.15.125:0/3391833175 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff1e80664f0 0x7ff1e80689b0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:59.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:59.073+0000 7ff2021c4640 1 --2- 172.21.15.125:0/3391833175 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1fc13f4a0 0x7ff1fc1325e0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:59.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:59.073+0000 7ff2021c4640 1 --2- 172.21.15.125:0/3391833175 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1fc0c6f60 0x7ff1fc0c9860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:59.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:59.073+0000 7ff2021c4640 1 --2- 172.21.15.125:0/3391833175 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1fc0c6b80 0x7ff1fc131760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:06:59.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:59.073+0000 7ff2021c4640 1 -- 172.21.15.125:0/3391833175 >> 172.21.15.125:0/3391833175 conn(0x7ff1fc0c7910 msgr2=0x7ff1fc0c11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:06:59.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:59.073+0000 7ff2021c4640 1 -- 172.21.15.125:0/3391833175 shutdown_connections 2023-10-30T13:06:59.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:06:59.073+0000 7ff2021c4640 1 -- 172.21.15.125:0/3391833175 wait complete. 2023-10-30T13:06:59.097 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:07:01.098 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:01.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.241+0000 7fb428479640 1 Processor -- start 2023-10-30T13:07:01.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.241+0000 7fb428479640 1 -- start start 2023-10-30T13:07:01.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb428479640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb4201538a0 0x7fb42015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:01.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb428479640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4200ca0e0 0x7fb42015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:01.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb428479640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4200cb800 con 0x7fb4200cf160 2023-10-30T13:07:01.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb428479640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb42015dc30 con 0x7fb4201538a0 2023-10-30T13:07:01.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb428479640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4200cb330 con 0x7fb4200ca0e0 2023-10-30T13:07:01.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb426215640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb4201538a0 0x7fb42015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:01.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb426215640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb4201538a0 0x7fb42015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38686/0 (socket says 172.21.15.125:38686) 2023-10-30T13:07:01.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb426215640 1 -- 172.21.15.125:0/600263290 learned_addr learned my addr 172.21.15.125:0/600263290 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:01.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb426a16640 1 --1- 172.21.15.125:0/600263290 >> v1:172.21.15.125:6789/0 conn(0x7fb4200cf160 0x7fb4200cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47898/0 (socket says 172.21.15.125:47898) 2023-10-30T13:07:01.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb425a14640 1 --2- 172.21.15.125:0/600263290 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4200ca0e0 0x7fb42015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:01.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb426215640 1 -- 172.21.15.125:0/600263290 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4200ca0e0 msgr2=0x7fb42015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:01.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb426215640 1 --2- 172.21.15.125:0/600263290 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4200ca0e0 0x7fb42015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:01.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb426215640 1 -- 172.21.15.125:0/600263290 >> v1:172.21.15.125:6789/0 conn(0x7fb4200cf160 legacy=0x7fb4200cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:01.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb426215640 1 -- 172.21.15.125:0/600263290 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4200cad70 con 0x7fb4201538a0 2023-10-30T13:07:01.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.245+0000 7fb425213640 1 -- 172.21.15.125:0/600263290 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320934980 0 0) 0x7fb4200cb800 con 0x7fb4200cf160 2023-10-30T13:07:01.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.249+0000 7fb426215640 1 --2- 172.21.15.125:0/600263290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4201538a0 0x7fb42015d6f0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009870 tx=0x7fb41c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f188311c843056 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:01.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.249+0000 7fb425213640 1 -- 172.21.15.125:0/600263290 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c0a8a30 con 0x7fb4201538a0 2023-10-30T13:07:01.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.249+0000 7fb425213640 1 -- 172.21.15.125:0/600263290 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41c0a8b90 con 0x7fb4201538a0 2023-10-30T13:07:01.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.249+0000 7fb425213640 1 -- 172.21.15.125:0/600263290 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c0b0440 con 0x7fb4201538a0 2023-10-30T13:07:01.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.249+0000 7fb428479640 1 -- 172.21.15.125:0/600263290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4201538a0 msgr2=0x7fb42015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:01.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.249+0000 7fb428479640 1 --2- 172.21.15.125:0/600263290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4201538a0 0x7fb42015d6f0 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fb41c009870 tx=0x7fb41c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:07:01.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.249+0000 7fb428479640 1 -- 172.21.15.125:0/600263290 shutdown_connections 2023-10-30T13:07:01.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.249+0000 7fb428479640 1 --2- 172.21.15.125:0/600263290 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4200ca0e0 0x7fb42015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:01.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.249+0000 7fb428479640 1 --2- 172.21.15.125:0/600263290 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4201538a0 0x7fb42015d6f0 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:01.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.249+0000 7fb428479640 1 -- 172.21.15.125:0/600263290 >> 172.21.15.125:0/600263290 conn(0x7fb42006bfe0 msgr2=0x7fb42006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:01.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.249+0000 7fb428479640 1 -- 172.21.15.125:0/600263290 shutdown_connections 2023-10-30T13:07:01.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.249+0000 7fb428479640 1 -- 172.21.15.125:0/600263290 wait complete. 2023-10-30T13:07:01.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.249+0000 7fb428479640 1 Processor -- start 2023-10-30T13:07:01.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.249+0000 7fb428479640 1 -- start start 2023-10-30T13:07:01.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb428479640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4200ca0e0 0x7fb4201ef240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:01.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb428479640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4200cf160 0x7fb4201f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:01.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb428479640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4201538a0 0x7fb420143eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:01.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb428479640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4200b31a0 con 0x7fb4200cf160 2023-10-30T13:07:01.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb428479640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4201443f0 con 0x7fb4201538a0 2023-10-30T13:07:01.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb428479640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4201f27b0 con 0x7fb4200ca0e0 2023-10-30T13:07:01.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb425a14640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4200cf160 0x7fb4201f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:01.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb425a14640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4200cf160 0x7fb4201f3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38690/0 (socket says 172.21.15.125:38690) 2023-10-30T13:07:01.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb425a14640 1 -- 172.21.15.125:0/3581369884 learned_addr learned my addr 172.21.15.125:0/3581369884 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:01.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb426215640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4200ca0e0 0x7fb4201ef240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:01.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb426a16640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4201538a0 0x7fb420143eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:01.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb425a14640 1 -- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4200ca0e0 msgr2=0x7fb4201ef240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:01.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb425a14640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4200ca0e0 0x7fb4201ef240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:01.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb425a14640 1 -- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4201538a0 msgr2=0x7fb420143eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:01.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb425a14640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4201538a0 0x7fb420143eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:01.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb425a14640 1 -- 172.21.15.125:0/3581369884 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c0094d0 con 0x7fb4200cf160 2023-10-30T13:07:01.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb426a16640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4201538a0 0x7fb420143eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:07:01.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb425a14640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4200cf160 0x7fb4201f3200 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fb41000d620 tx=0x7fb41000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:01.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb416ffd640 1 -- 172.21.15.125:0/3581369884 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41000b620 con 0x7fb4200cf160 2023-10-30T13:07:01.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb416ffd640 1 -- 172.21.15.125:0/3581369884 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41000b780 con 0x7fb4200cf160 2023-10-30T13:07:01.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb416ffd640 1 -- 172.21.15.125:0/3581369884 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41000bb10 con 0x7fb4200cf160 2023-10-30T13:07:01.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb428479640 1 -- 172.21.15.125:0/3581369884 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4201f2d70 con 0x7fb4200cf160 2023-10-30T13:07:01.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb428479640 1 -- 172.21.15.125:0/3581369884 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4200b3690 con 0x7fb4200cf160 2023-10-30T13:07:01.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb428479640 1 -- 172.21.15.125:0/3581369884 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4200b3850 con 0x7fb4200cf160 2023-10-30T13:07:01.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.253+0000 7fb416ffd640 1 -- 172.21.15.125:0/3581369884 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb41000bc70 con 0x7fb4200cf160 2023-10-30T13:07:01.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.257+0000 7fb416ffd640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb4040664a0 0x7fb404068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:01.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.257+0000 7fb416ffd640 1 -- 172.21.15.125:0/3581369884 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7fb410084a30 con 0x7fb4200cf160 2023-10-30T13:07:01.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.257+0000 7fb426215640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb4040664a0 0x7fb404068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:01.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.257+0000 7fb426215640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb4040664a0 0x7fb404068960 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb41c0b2000 tx=0x7fb41c0aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:01.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.261+0000 7fb416ffd640 1 -- 172.21.15.125:0/3581369884 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb4100516f0 con 0x7fb4200cf160 2023-10-30T13:07:01.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.465+0000 7fb428479640 1 -- 172.21.15.125:0/3581369884 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4200b2a50 con 0x7fb4200cf160 2023-10-30T13:07:01.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.465+0000 7fb416ffd640 1 -- 172.21.15.125:0/3581369884 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+5751 (secure 0 0 0) 0x7fb410007380 con 0x7fb4200cf160 2023-10-30T13:07:01.471 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:01.471 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:55.602089+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:01.472 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 172 2023-10-30T13:07:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.469+0000 7fb428479640 1 -- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb4040664a0 msgr2=0x7fb404068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.469+0000 7fb428479640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb4040664a0 0x7fb404068960 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fb41c0b2000 tx=0x7fb41c0aa070 comp rx=0 tx=0).stop 2023-10-30T13:07:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.469+0000 7fb428479640 1 -- 172.21.15.125:0/3581369884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4200cf160 msgr2=0x7fb4201f3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.469+0000 7fb428479640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4200cf160 0x7fb4201f3200 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fb41000d620 tx=0x7fb41000dae0 comp rx=0 tx=0).stop 2023-10-30T13:07:01.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.469+0000 7fb428479640 1 -- 172.21.15.125:0/3581369884 shutdown_connections 2023-10-30T13:07:01.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.469+0000 7fb428479640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb4040664a0 0x7fb404068960 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:01.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.469+0000 7fb428479640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb4201538a0 0x7fb420143eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:01.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.469+0000 7fb428479640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb4200cf160 0x7fb4201f3200 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:01.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.469+0000 7fb428479640 1 --2- 172.21.15.125:0/3581369884 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb4200ca0e0 0x7fb4201ef240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:01.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.469+0000 7fb428479640 1 -- 172.21.15.125:0/3581369884 >> 172.21.15.125:0/3581369884 conn(0x7fb42006bfe0 msgr2=0x7fb420150f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:01.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.469+0000 7fb428479640 1 -- 172.21.15.125:0/3581369884 shutdown_connections 2023-10-30T13:07:01.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:01.469+0000 7fb428479640 1 -- 172.21.15.125:0/3581369884 wait complete. 2023-10-30T13:07:01.491 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:07:03.493 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:03.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.633+0000 7f75b0edc640 1 Processor -- start 2023-10-30T13:07:03.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.633+0000 7f75b0edc640 1 -- start start 2023-10-30T13:07:03.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75b0edc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75ac1538f0 0x7f75ac159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:03.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75b0edc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f75ac15cd50 0x7f75ac0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:03.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75b0edc640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75ac15d300 con 0x7f75ac0cefa0 2023-10-30T13:07:03.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75b0edc640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75ac15d5e0 con 0x7f75ac1538f0 2023-10-30T13:07:03.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75b0edc640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75ac15d8c0 con 0x7f75ac15cd50 2023-10-30T13:07:03.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75ab7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75ac1538f0 0x7f75ac159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:03.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75ab7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75ac1538f0 0x7f75ac159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38702/0 (socket says 172.21.15.125:38702) 2023-10-30T13:07:03.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75ab7fe640 1 -- 172.21.15.125:0/127653111 learned_addr learned my addr 172.21.15.125:0/127653111 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:03.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75abfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f75ac0cefa0 0x7f75ac0cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47900/0 (socket says 172.21.15.125:47900) 2023-10-30T13:07:03.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75a3fff640 1 --2- 172.21.15.125:0/127653111 >> v2:172.21.15.136:3300/0 conn(0x7f75ac15cd50 0x7f75ac0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:03.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75ab7fe640 1 -- 172.21.15.125:0/127653111 >> v2:172.21.15.136:3300/0 conn(0x7f75ac15cd50 msgr2=0x7f75ac0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:03.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75ab7fe640 1 --2- 172.21.15.125:0/127653111 >> v2:172.21.15.136:3300/0 conn(0x7f75ac15cd50 0x7f75ac0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:03.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75ab7fe640 1 -- 172.21.15.125:0/127653111 >> v1:172.21.15.125:6789/0 conn(0x7f75ac0cefa0 legacy=0x7f75ac0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:03.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75ab7fe640 1 -- 172.21.15.125:0/127653111 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75ac15dba0 con 0x7f75ac1538f0 2023-10-30T13:07:03.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.637+0000 7f75ab7fe640 1 --2- 172.21.15.125:0/127653111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac1538f0 0x7f75ac159fb0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f75ac072e90 tx=0x7f759c01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63a224eb45c49b5f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:03.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75aaffd640 1 -- 172.21.15.125:0/127653111 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c0a89f0 con 0x7f75ac1538f0 2023-10-30T13:07:03.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75aaffd640 1 -- 172.21.15.125:0/127653111 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759c0a8b50 con 0x7f75ac1538f0 2023-10-30T13:07:03.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75aaffd640 1 -- 172.21.15.125:0/127653111 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c0b0440 con 0x7f75ac1538f0 2023-10-30T13:07:03.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 -- 172.21.15.125:0/127653111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac1538f0 msgr2=0x7f75ac159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:03.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 --2- 172.21.15.125:0/127653111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac1538f0 0x7f75ac159fb0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f75ac072e90 tx=0x7f759c01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:07:03.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 -- 172.21.15.125:0/127653111 shutdown_connections 2023-10-30T13:07:03.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 --2- 172.21.15.125:0/127653111 >> v2:172.21.15.136:3300/0 conn(0x7f75ac15cd50 0x7f75ac0c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:03.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 --2- 172.21.15.125:0/127653111 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac1538f0 0x7f75ac159fb0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:03.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 -- 172.21.15.125:0/127653111 >> 172.21.15.125:0/127653111 conn(0x7f75ac06a640 msgr2=0x7f75ac065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:03.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 -- 172.21.15.125:0/127653111 shutdown_connections 2023-10-30T13:07:03.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 -- 172.21.15.125:0/127653111 wait complete. 2023-10-30T13:07:03.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 Processor -- start 2023-10-30T13:07:03.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 -- start start 2023-10-30T13:07:03.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac0cefa0 0x7f75ac1573a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:03.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75ac1538f0 0x7f75ac145f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:03.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75ac15cd50 0x7f75ac1f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:03.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac152ad0 con 0x7f75ac0cefa0 2023-10-30T13:07:03.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac146490 con 0x7f75ac1538f0 2023-10-30T13:07:03.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75b0edc640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75ac157ff0 con 0x7f75ac15cd50 2023-10-30T13:07:03.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75ab7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac0cefa0 0x7f75ac1573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:03.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75ab7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac0cefa0 0x7f75ac1573a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38716/0 (socket says 172.21.15.125:38716) 2023-10-30T13:07:03.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75ab7fe640 1 -- 172.21.15.125:0/3339643512 learned_addr learned my addr 172.21.15.125:0/3339643512 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:03.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75abfff640 1 --2- 172.21.15.125:0/3339643512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75ac15cd50 0x7f75ac1f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:03.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.641+0000 7f75a3fff640 1 --2- 172.21.15.125:0/3339643512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75ac1538f0 0x7f75ac145f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:03.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.645+0000 7f75ab7fe640 1 -- 172.21.15.125:0/3339643512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75ac15cd50 msgr2=0x7f75ac1f2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:03.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.645+0000 7f75ab7fe640 1 --2- 172.21.15.125:0/3339643512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75ac15cd50 0x7f75ac1f2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:03.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.645+0000 7f75ab7fe640 1 -- 172.21.15.125:0/3339643512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75ac1538f0 msgr2=0x7f75ac145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:03.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.645+0000 7f75ab7fe640 1 --2- 172.21.15.125:0/3339643512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75ac1538f0 0x7f75ac145f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:03.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.645+0000 7f75ab7fe640 1 -- 172.21.15.125:0/3339643512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c0094d0 con 0x7f75ac0cefa0 2023-10-30T13:07:03.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.645+0000 7f75ab7fe640 1 --2- 172.21.15.125:0/3339643512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac0cefa0 0x7f75ac1573a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f759c01b900 tx=0x7f759c01ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:03.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.645+0000 7f75a8ff9640 1 -- 172.21.15.125:0/3339643512 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c0ba600 con 0x7f75ac0cefa0 2023-10-30T13:07:03.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.645+0000 7f75a8ff9640 1 -- 172.21.15.125:0/3339643512 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759c0ba760 con 0x7f75ac0cefa0 2023-10-30T13:07:03.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.645+0000 7f75a8ff9640 1 -- 172.21.15.125:0/3339643512 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759c0bab10 con 0x7f75ac0cefa0 2023-10-30T13:07:03.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.645+0000 7f75b0edc640 1 -- 172.21.15.125:0/3339643512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75ac157a30 con 0x7f75ac0cefa0 2023-10-30T13:07:03.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.645+0000 7f75b0edc640 1 -- 172.21.15.125:0/3339643512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75ac152480 con 0x7f75ac0cefa0 2023-10-30T13:07:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.645+0000 7f75b0edc640 1 -- 172.21.15.125:0/3339643512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75ac04e170 con 0x7f75ac0cefa0 2023-10-30T13:07:03.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.645+0000 7f75a8ff9640 1 -- 172.21.15.125:0/3339643512 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f759c0bac70 con 0x7f75ac0cefa0 2023-10-30T13:07:03.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.645+0000 7f75a8ff9640 1 --2- 172.21.15.125:0/3339643512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f758c0664d0 0x7f758c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:03.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.649+0000 7f75a8ff9640 1 -- 172.21.15.125:0/3339643512 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f759c122af0 con 0x7f75ac0cefa0 2023-10-30T13:07:03.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.649+0000 7f75a3fff640 1 --2- 172.21.15.125:0/3339643512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f758c0664d0 0x7f758c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:03.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.649+0000 7f75a3fff640 1 --2- 172.21.15.125:0/3339643512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f758c0664d0 0x7f758c068990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f75ac0b1790 tx=0x7f7594008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:03.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.653+0000 7f75a8ff9640 1 -- 172.21.15.125:0/3339643512 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f759c0ef7c0 con 0x7f75ac0cefa0 2023-10-30T13:07:03.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.837+0000 7f75b0edc640 1 -- 172.21.15.125:0/3339643512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75ac150ed0 con 0x7f75ac0cefa0 2023-10-30T13:07:03.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.837+0000 7f75a8ff9640 1 -- 172.21.15.125:0/3339643512 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+5751 (secure 0 0 0) 0x7f759c0b5020 con 0x7f75ac0cefa0 2023-10-30T13:07:03.842 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:03.842 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":172,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:06:55.602089+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:03.842 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 172 2023-10-30T13:07:03.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.837+0000 7f75b0edc640 1 -- 172.21.15.125:0/3339643512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f758c0664d0 msgr2=0x7f758c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:03.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.837+0000 7f75b0edc640 1 --2- 172.21.15.125:0/3339643512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f758c0664d0 0x7f758c068990 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f75ac0b1790 tx=0x7f7594008040 comp rx=0 tx=0).stop 2023-10-30T13:07:03.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.837+0000 7f75b0edc640 1 -- 172.21.15.125:0/3339643512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac0cefa0 msgr2=0x7f75ac1573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:03.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.837+0000 7f75b0edc640 1 --2- 172.21.15.125:0/3339643512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac0cefa0 0x7f75ac1573a0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f759c01b900 tx=0x7f759c01ba90 comp rx=0 tx=0).stop 2023-10-30T13:07:03.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.837+0000 7f75b0edc640 1 -- 172.21.15.125:0/3339643512 shutdown_connections 2023-10-30T13:07:03.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.837+0000 7f75b0edc640 1 --2- 172.21.15.125:0/3339643512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f758c0664d0 0x7f758c068990 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:03.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.837+0000 7f75b0edc640 1 --2- 172.21.15.125:0/3339643512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75ac15cd50 0x7f75ac1f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:03.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.837+0000 7f75b0edc640 1 --2- 172.21.15.125:0/3339643512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75ac1538f0 0x7f75ac145f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:03.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.837+0000 7f75b0edc640 1 --2- 172.21.15.125:0/3339643512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75ac0cefa0 0x7f75ac1573a0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:03.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.837+0000 7f75b0edc640 1 -- 172.21.15.125:0/3339643512 >> 172.21.15.125:0/3339643512 conn(0x7f75ac06a640 msgr2=0x7f75ac065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:03.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.841+0000 7f75b0edc640 1 -- 172.21.15.125:0/3339643512 shutdown_connections 2023-10-30T13:07:03.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:03.841+0000 7f75b0edc640 1 -- 172.21.15.125:0/3339643512 wait complete. 2023-10-30T13:07:03.865 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:07:05.865 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:06.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b06067640 1 Processor -- start 2023-10-30T13:07:06.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b06067640 1 -- start start 2023-10-30T13:07:06.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b06067640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3b001538f0 0x7f3b00159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:06.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b06067640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3b0015cd50 0x7f3b000c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:06.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b06067640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b0015d300 con 0x7f3b000cefa0 2023-10-30T13:07:06.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b06067640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b0015d5e0 con 0x7f3b001538f0 2023-10-30T13:07:06.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b06067640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b0015d8c0 con 0x7f3b0015cd50 2023-10-30T13:07:06.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b05065640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3b001538f0 0x7f3b00159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:06.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b05866640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3b000cefa0 0x7f3b000cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47912/0 (socket says 172.21.15.125:47912) 2023-10-30T13:07:06.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b05866640 1 -- 172.21.15.125:0/1477485524 learned_addr learned my addr 172.21.15.125:0/1477485524 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:07:06.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b04864640 1 --2- 172.21.15.125:0/1477485524 >> v2:172.21.15.136:3300/0 conn(0x7f3b0015cd50 0x7f3b000c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:06.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b05065640 1 -- 172.21.15.125:0/1477485524 >> v2:172.21.15.136:3300/0 conn(0x7f3b0015cd50 msgr2=0x7f3b000c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:06.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b05065640 1 --2- 172.21.15.125:0/1477485524 >> v2:172.21.15.136:3300/0 conn(0x7f3b0015cd50 0x7f3b000c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:06.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b05065640 1 -- 172.21.15.125:0/1477485524 >> v1:172.21.15.125:6789/0 conn(0x7f3b000cefa0 legacy=0x7f3b000cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:06.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b05065640 1 -- 172.21.15.125:0/1477485524 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0015dba0 con 0x7f3b001538f0 2023-10-30T13:07:06.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3b04864640 1 --2- 172.21.15.125:0/1477485524 >> v2:172.21.15.136:3300/0 conn(0x7f3b0015cd50 0x7f3b000c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:07:06.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.009+0000 7f3af7fff640 1 -- 172.21.15.125:0/1477485524 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957140422 0 0) 0x7f3b0015d300 con 0x7f3b000cefa0 2023-10-30T13:07:06.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.013+0000 7f3b05065640 1 --2- 172.21.15.125:0/1477485524 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b001538f0 0x7f3b00159fb0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f3b00072e90 tx=0x7f3afc01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d33f0e2a581ed9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:06.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.013+0000 7f3af7fff640 1 -- 172.21.15.125:0/1477485524 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0a8950 con 0x7f3b001538f0 2023-10-30T13:07:06.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.013+0000 7f3af7fff640 1 -- 172.21.15.125:0/1477485524 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3afc0a8ab0 con 0x7f3b001538f0 2023-10-30T13:07:06.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.013+0000 7f3af7fff640 1 -- 172.21.15.125:0/1477485524 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3afc0a8e40 con 0x7f3b001538f0 2023-10-30T13:07:06.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.013+0000 7f3b06067640 1 -- 172.21.15.125:0/1477485524 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b001538f0 msgr2=0x7f3b00159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:06.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.013+0000 7f3b06067640 1 --2- 172.21.15.125:0/1477485524 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b001538f0 0x7f3b00159fb0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f3b00072e90 tx=0x7f3afc01ec50 comp rx=0 tx=0).stop 2023-10-30T13:07:06.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.013+0000 7f3b06067640 1 -- 172.21.15.125:0/1477485524 shutdown_connections 2023-10-30T13:07:06.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.013+0000 7f3b06067640 1 --2- 172.21.15.125:0/1477485524 >> v2:172.21.15.136:3300/0 conn(0x7f3b0015cd50 0x7f3b000c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:06.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.013+0000 7f3b06067640 1 --2- 172.21.15.125:0/1477485524 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b001538f0 0x7f3b00159fb0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:06.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.013+0000 7f3b06067640 1 -- 172.21.15.125:0/1477485524 >> 172.21.15.125:0/1477485524 conn(0x7f3b0006a640 msgr2=0x7f3b00065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:06.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.013+0000 7f3b06067640 1 -- 172.21.15.125:0/1477485524 shutdown_connections 2023-10-30T13:07:06.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.013+0000 7f3b06067640 1 -- 172.21.15.125:0/1477485524 wait complete. 2023-10-30T13:07:06.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b06067640 1 Processor -- start 2023-10-30T13:07:06.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b06067640 1 -- start start 2023-10-30T13:07:06.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b06067640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b000cefa0 0x7f3b00146820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:06.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b06067640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b001538f0 0x7f3b000b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:06.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b06067640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b0015cd50 0x7f3b000b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:06.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b06067640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0015bea0 con 0x7f3b0015cd50 2023-10-30T13:07:06.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b06067640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b001487e0 con 0x7f3b000cefa0 2023-10-30T13:07:06.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b06067640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b001eece0 con 0x7f3b001538f0 2023-10-30T13:07:06.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b05866640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b0015cd50 0x7f3b000b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:06.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b05866640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b0015cd50 0x7f3b000b45e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38720/0 (socket says 172.21.15.125:38720) 2023-10-30T13:07:06.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b05866640 1 -- 172.21.15.125:0/447923405 learned_addr learned my addr 172.21.15.125:0/447923405 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:06.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b05065640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b000cefa0 0x7f3b00146820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:06.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b04864640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b001538f0 0x7f3b000b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:06.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b05866640 1 -- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b001538f0 msgr2=0x7f3b000b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:06.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b05866640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b001538f0 0x7f3b000b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:06.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b05866640 1 -- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b000cefa0 msgr2=0x7f3b00146820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:06.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b05866640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b000cefa0 0x7f3b00146820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:06.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b05866640 1 -- 172.21.15.125:0/447923405 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3afc009420 con 0x7f3b0015cd50 2023-10-30T13:07:06.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b05065640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b000cefa0 0x7f3b00146820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:07:06.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b04864640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b001538f0 0x7f3b000b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:07:06.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b05866640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b0015cd50 0x7f3b000b45e0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f3af8001570 tx=0x7f3af800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:06.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3af5ffb640 1 -- 172.21.15.125:0/447923405 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af800a640 con 0x7f3b0015cd50 2023-10-30T13:07:06.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3af5ffb640 1 -- 172.21.15.125:0/447923405 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af800a7c0 con 0x7f3b0015cd50 2023-10-30T13:07:06.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3af5ffb640 1 -- 172.21.15.125:0/447923405 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3af800ab70 con 0x7f3b0015cd50 2023-10-30T13:07:06.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b06067640 1 -- 172.21.15.125:0/447923405 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b00148500 con 0x7f3b0015cd50 2023-10-30T13:07:06.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b06067640 1 -- 172.21.15.125:0/447923405 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b001ef260 con 0x7f3b0015cd50 2023-10-30T13:07:06.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.017+0000 7f3b06067640 1 -- 172.21.15.125:0/447923405 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b0004e170 con 0x7f3b0015cd50 2023-10-30T13:07:06.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.021+0000 7f3af5ffb640 1 -- 172.21.15.125:0/447923405 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3af800acd0 con 0x7f3b0015cd50 2023-10-30T13:07:06.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.021+0000 7f3af5ffb640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3ae40664a0 0x7f3ae4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:06.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.021+0000 7f3af5ffb640 1 -- 172.21.15.125:0/447923405 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f3af8083cb0 con 0x7f3b0015cd50 2023-10-30T13:07:06.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.021+0000 7f3b05065640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3ae40664a0 0x7f3ae4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:06.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.021+0000 7f3b05065640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3ae40664a0 0x7f3ae4068960 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0b2000 tx=0x7f3afc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:06.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.025+0000 7f3af5ffb640 1 -- 172.21.15.125:0/447923405 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3af80508d0 con 0x7f3b0015cd50 2023-10-30T13:07:06.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.201+0000 7f3b06067640 1 -- 172.21.15.125:0/447923405 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b000b2290 con 0x7f3b0015cd50 2023-10-30T13:07:06.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.201+0000 7f3af5ffb640 1 -- 172.21.15.125:0/447923405 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+5754 (secure 0 0 0) 0x7f3af8018070 con 0x7f3b0015cd50 2023-10-30T13:07:06.208 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:06.208 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:04.430634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:resolve","state_seq":424,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:06.208 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 173 2023-10-30T13:07:06.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.205+0000 7f3b06067640 1 -- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3ae40664a0 msgr2=0x7f3ae4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:06.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.205+0000 7f3b06067640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3ae40664a0 0x7f3ae4068960 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f3afc0b2000 tx=0x7f3afc008040 comp rx=0 tx=0).stop 2023-10-30T13:07:06.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.205+0000 7f3b06067640 1 -- 172.21.15.125:0/447923405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b0015cd50 msgr2=0x7f3b000b45e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:06.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.205+0000 7f3b06067640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b0015cd50 0x7f3b000b45e0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f3af8001570 tx=0x7f3af800cca0 comp rx=0 tx=0).stop 2023-10-30T13:07:06.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.205+0000 7f3b06067640 1 -- 172.21.15.125:0/447923405 shutdown_connections 2023-10-30T13:07:06.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.205+0000 7f3b06067640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3ae40664a0 0x7f3ae4068960 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:06.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.205+0000 7f3b06067640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b0015cd50 0x7f3b000b45e0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:06.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.205+0000 7f3b06067640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b001538f0 0x7f3b000b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:06.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.205+0000 7f3b06067640 1 --2- 172.21.15.125:0/447923405 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b000cefa0 0x7f3b00146820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:06.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.205+0000 7f3b06067640 1 -- 172.21.15.125:0/447923405 >> 172.21.15.125:0/447923405 conn(0x7f3b0006a640 msgr2=0x7f3b00065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:06.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.205+0000 7f3b06067640 1 -- 172.21.15.125:0/447923405 shutdown_connections 2023-10-30T13:07:06.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:06.205+0000 7f3b06067640 1 -- 172.21.15.125:0/447923405 wait complete. 2023-10-30T13:07:06.228 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:07:08.229 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:08.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.369+0000 7f8e0e965640 1 Processor -- start 2023-10-30T13:07:08.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.369+0000 7f8e0e965640 1 -- start start 2023-10-30T13:07:08.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.373+0000 7f8e0e965640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e080c6bd0 0x7f8e0816a510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:08.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.373+0000 7f8e0e965640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8e080c5810 0x7f8e080c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:08.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.373+0000 7f8e0e965640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e080c61e0 con 0x7f8e080720e0 2023-10-30T13:07:08.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.373+0000 7f8e0e965640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e0806fe90 con 0x7f8e080c5810 2023-10-30T13:07:08.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.373+0000 7f8e0e965640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080c7010 con 0x7f8e080c6bd0 2023-10-30T13:07:08.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.373+0000 7f8e0cf02640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8e080c5810 0x7f8e080c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:08.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.373+0000 7f8e0cf02640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8e080c5810 0x7f8e080c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38732/0 (socket says 172.21.15.125:38732) 2023-10-30T13:07:08.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.373+0000 7f8e0cf02640 1 -- 172.21.15.125:0/3428297262 learned_addr learned my addr 172.21.15.125:0/3428297262 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:08.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.373+0000 7f8e07fff640 1 --1- 172.21.15.125:0/3428297262 >> v1:172.21.15.125:6789/0 conn(0x7f8e080720e0 0x7f8e080bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47928/0 (socket says 172.21.15.125:47928) 2023-10-30T13:07:08.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.373+0000 7f8e077fe640 1 --2- 172.21.15.125:0/3428297262 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e080c6bd0 0x7f8e0816a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:08.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.441+0000 7f8e0cf02640 1 -- 172.21.15.125:0/3428297262 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e080c6bd0 msgr2=0x7f8e0816a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:08.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.441+0000 7f8e0cf02640 1 --2- 172.21.15.125:0/3428297262 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e080c6bd0 0x7f8e0816a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:08.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.441+0000 7f8e06ffd640 1 -- 172.21.15.125:0/3428297262 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472651542 0 0) 0x7f8e080c61e0 con 0x7f8e080720e0 2023-10-30T13:07:08.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.441+0000 7f8e0cf02640 1 -- 172.21.15.125:0/3428297262 >> v1:172.21.15.125:6789/0 conn(0x7f8e080720e0 legacy=0x7f8e080bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:08.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.441+0000 7f8e0cf02640 1 -- 172.21.15.125:0/3428297262 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0816aa50 con 0x7f8e080c5810 2023-10-30T13:07:08.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.445+0000 7f8e0cf02640 1 --2- 172.21.15.125:0/3428297262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e080c5810 0x7f8e080c5ca0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000bef0 tx=0x7f8e0001ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be2b1141024ca09 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:08.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.445+0000 7f8e06ffd640 1 -- 172.21.15.125:0/3428297262 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e000a5a40 con 0x7f8e080c5810 2023-10-30T13:07:08.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.445+0000 7f8e06ffd640 1 -- 172.21.15.125:0/3428297262 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e000a9070 con 0x7f8e080c5810 2023-10-30T13:07:08.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.445+0000 7f8e06ffd640 1 -- 172.21.15.125:0/3428297262 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e000a5dc0 con 0x7f8e080c5810 2023-10-30T13:07:08.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.445+0000 7f8e0e965640 1 -- 172.21.15.125:0/3428297262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e080c5810 msgr2=0x7f8e080c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:08.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.445+0000 7f8e0e965640 1 --2- 172.21.15.125:0/3428297262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e080c5810 0x7f8e080c5ca0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000bef0 tx=0x7f8e0001ca80 comp rx=0 tx=0).stop 2023-10-30T13:07:08.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.445+0000 7f8e0e965640 1 -- 172.21.15.125:0/3428297262 shutdown_connections 2023-10-30T13:07:08.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.445+0000 7f8e0e965640 1 --2- 172.21.15.125:0/3428297262 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e080c5810 0x7f8e080c5ca0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:08.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.445+0000 7f8e0e965640 1 --2- 172.21.15.125:0/3428297262 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e080c6bd0 0x7f8e0816a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:08.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.445+0000 7f8e0e965640 1 -- 172.21.15.125:0/3428297262 >> 172.21.15.125:0/3428297262 conn(0x7f8e080c7910 msgr2=0x7f8e080c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:08.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.445+0000 7f8e0e965640 1 -- 172.21.15.125:0/3428297262 shutdown_connections 2023-10-30T13:07:08.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.445+0000 7f8e0e965640 1 -- 172.21.15.125:0/3428297262 wait complete. 2023-10-30T13:07:08.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.445+0000 7f8e0e965640 1 Processor -- start 2023-10-30T13:07:08.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.445+0000 7f8e0e965640 1 -- start start 2023-10-30T13:07:08.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0e965640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e080720e0 0x7f8e0815b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:08.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0e965640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e080c5810 0x7f8e081fe5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:08.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0e965640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e080c6bd0 0x7f8e0815e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:08.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0e965640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080c0820 con 0x7f8e080c6bd0 2023-10-30T13:07:08.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0e965640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e080c1970 con 0x7f8e080720e0 2023-10-30T13:07:08.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0e965640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0815ffb0 con 0x7f8e080c5810 2023-10-30T13:07:08.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0cf02640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e080c6bd0 0x7f8e0815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:08.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0cf02640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e080c6bd0 0x7f8e0815e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38746/0 (socket says 172.21.15.125:38746) 2023-10-30T13:07:08.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0cf02640 1 -- 172.21.15.125:0/1551878210 learned_addr learned my addr 172.21.15.125:0/1551878210 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:08.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e077fe640 1 --2- 172.21.15.125:0/1551878210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e080c5810 0x7f8e081fe5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:08.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e07fff640 1 --2- 172.21.15.125:0/1551878210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e080720e0 0x7f8e0815b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:08.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0cf02640 1 -- 172.21.15.125:0/1551878210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e080c5810 msgr2=0x7f8e081fe5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:08.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0cf02640 1 --2- 172.21.15.125:0/1551878210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e080c5810 0x7f8e081fe5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:08.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0cf02640 1 -- 172.21.15.125:0/1551878210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e080720e0 msgr2=0x7f8e0815b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:08.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0cf02640 1 --2- 172.21.15.125:0/1551878210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e080720e0 0x7f8e0815b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:08.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0cf02640 1 -- 172.21.15.125:0/1551878210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e0000bb50 con 0x7f8e080c6bd0 2023-10-30T13:07:08.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0cf02640 1 --2- 172.21.15.125:0/1551878210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e080c6bd0 0x7f8e0815e450 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f8e0001b130 tx=0x7f8e0000a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:08.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e04ff9640 1 -- 172.21.15.125:0/1551878210 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e000ba850 con 0x7f8e080c6bd0 2023-10-30T13:07:08.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e04ff9640 1 -- 172.21.15.125:0/1551878210 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e000ba9b0 con 0x7f8e080c6bd0 2023-10-30T13:07:08.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0e965640 1 -- 172.21.15.125:0/1551878210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e080bfff0 con 0x7f8e080c6bd0 2023-10-30T13:07:08.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e04ff9640 1 -- 172.21.15.125:0/1551878210 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e000bad20 con 0x7f8e080c6bd0 2023-10-30T13:07:08.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0e965640 1 -- 172.21.15.125:0/1551878210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e0808b670 con 0x7f8e080c6bd0 2023-10-30T13:07:08.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e0e965640 1 -- 172.21.15.125:0/1551878210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e080c1ae0 con 0x7f8e080c6bd0 2023-10-30T13:07:08.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.449+0000 7f8e04ff9640 1 -- 172.21.15.125:0/1551878210 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e000c5410 con 0x7f8e080c6bd0 2023-10-30T13:07:08.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.453+0000 7f8e04ff9640 1 --2- 172.21.15.125:0/1551878210 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8df40664d0 0x7f8df4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:08.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.453+0000 7f8e04ff9640 1 -- 172.21.15.125:0/1551878210 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f8e0011ee00 con 0x7f8e080c6bd0 2023-10-30T13:07:08.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.453+0000 7f8e07fff640 1 --2- 172.21.15.125:0/1551878210 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8df40664d0 0x7f8df4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:08.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.453+0000 7f8e07fff640 1 --2- 172.21.15.125:0/1551878210 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8df40664d0 0x7f8df4068990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f8df8004860 tx=0x7f8df8006050 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:08.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.457+0000 7f8e04ff9640 1 -- 172.21.15.125:0/1551878210 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e000ebab0 con 0x7f8e080c6bd0 2023-10-30T13:07:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.633+0000 7f8e0e965640 1 -- 172.21.15.125:0/1551878210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8e0816aa50 con 0x7f8e080c6bd0 2023-10-30T13:07:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.637+0000 7f8e04ff9640 1 -- 172.21.15.125:0/1551878210 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+5754 (secure 0 0 0) 0x7f8e0816aa50 con 0x7f8e080c6bd0 2023-10-30T13:07:08.642 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:08.643 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:04.430634+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:resolve","state_seq":424,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 173 2023-10-30T13:07:08.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.641+0000 7f8e0e965640 1 -- 172.21.15.125:0/1551878210 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8df40664d0 msgr2=0x7f8df4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:08.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.641+0000 7f8e0e965640 1 --2- 172.21.15.125:0/1551878210 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8df40664d0 0x7f8df4068990 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f8df8004860 tx=0x7f8df8006050 comp rx=0 tx=0).stop 2023-10-30T13:07:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.641+0000 7f8e0e965640 1 -- 172.21.15.125:0/1551878210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e080c6bd0 msgr2=0x7f8e0815e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.641+0000 7f8e0e965640 1 --2- 172.21.15.125:0/1551878210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e080c6bd0 0x7f8e0815e450 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f8e0001b130 tx=0x7f8e0000a4c0 comp rx=0 tx=0).stop 2023-10-30T13:07:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.641+0000 7f8e0e965640 1 -- 172.21.15.125:0/1551878210 shutdown_connections 2023-10-30T13:07:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.641+0000 7f8e0e965640 1 --2- 172.21.15.125:0/1551878210 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8df40664d0 0x7f8df4068990 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.641+0000 7f8e0e965640 1 --2- 172.21.15.125:0/1551878210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e080c6bd0 0x7f8e0815e450 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.641+0000 7f8e0e965640 1 --2- 172.21.15.125:0/1551878210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e080c5810 0x7f8e081fe5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.641+0000 7f8e0e965640 1 --2- 172.21.15.125:0/1551878210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e080720e0 0x7f8e0815b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.641+0000 7f8e0e965640 1 -- 172.21.15.125:0/1551878210 >> 172.21.15.125:0/1551878210 conn(0x7f8e080c7910 msgr2=0x7f8e08165ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.641+0000 7f8e0e965640 1 -- 172.21.15.125:0/1551878210 shutdown_connections 2023-10-30T13:07:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:08.641+0000 7f8e0e965640 1 -- 172.21.15.125:0/1551878210 wait complete. 2023-10-30T13:07:08.664 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 173, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 173, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5234": { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 172, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:resolve", "state_seq": 424 }, "gid_6017": { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 170, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 251 }, "gid_8301": { "addr": "172.21.15.125:6837/2540063352", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2540063352, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2540063352, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8301, "incarnation": 168, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 75, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:07:04.430634+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5234, "mds_1": 8301, "mds_2": 6017 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:07:10.665 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:10.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.805+0000 7f51fac06640 1 Processor -- start 2023-10-30T13:07:10.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51fac06640 1 -- start start 2023-10-30T13:07:10.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51fac06640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f51f415cd50 0x7f51f40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:10.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51fac06640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f415a6c0 con 0x7f51f40cefa0 2023-10-30T13:07:10.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51fac06640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f415d300 con 0x7f51f41538f0 2023-10-30T13:07:10.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51fac06640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f415d5e0 con 0x7f51f415cd50 2023-10-30T13:07:10.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51fa405640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f51f40cefa0 0x7f51f41640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52706/0 (socket says 172.21.15.125:52706) 2023-10-30T13:07:10.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51fa405640 1 -- 172.21.15.125:0/620146538 learned_addr learned my addr 172.21.15.125:0/620146538 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:07:10.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f9403640 1 --2- 172.21.15.125:0/620146538 >> v2:172.21.15.136:3300/0 conn(0x7f51f415cd50 0x7f51f40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:10.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 -- 172.21.15.125:0/620146538 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945161253 0 0) 0x7f51f415a6c0 con 0x7f51f40cefa0 2023-10-30T13:07:10.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 -- 172.21.15.125:0/620146538 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51e0003530 con 0x7f51f40cefa0 2023-10-30T13:07:10.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 -- 172.21.15.125:0/620146538 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 20923178 0 0) 0x7f51e0003530 con 0x7f51f40cefa0 2023-10-30T13:07:10.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 -- 172.21.15.125:0/620146538 >> v2:172.21.15.136:3300/0 conn(0x7f51f415cd50 msgr2=0x7f51f40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:10.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 --2- 172.21.15.125:0/620146538 >> v2:172.21.15.136:3300/0 conn(0x7f51f415cd50 0x7f51f40c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:10.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 -- 172.21.15.125:0/620146538 >> v1:172.21.15.136:6789/0 conn(0x7f51f41538f0 legacy=0x7f51f4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:10.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 -- 172.21.15.125:0/620146538 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f415d8c0 con 0x7f51f40cefa0 2023-10-30T13:07:10.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 -- 172.21.15.125:0/620146538 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f51ec001a00 con 0x7f51f40cefa0 2023-10-30T13:07:10.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 -- 172.21.15.125:0/620146538 >> v1:172.21.15.125:6789/0 conn(0x7f51f40cefa0 legacy=0x7f51f41640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:10.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 --2- 172.21.15.125:0/620146538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e0003980 0x7f51e0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:10.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 --2- 172.21.15.125:0/620146538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51e00047d0 0x7f51e0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:10.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 --2- 172.21.15.125:0/620146538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51e0005190 0x7f51e000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:10.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 -- 172.21.15.125:0/620146538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f415d5e0 con 0x7f51e0005190 2023-10-30T13:07:10.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 -- 172.21.15.125:0/620146538 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e000c040 con 0x7f51e0003980 2023-10-30T13:07:10.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f8c02640 1 -- 172.21.15.125:0/620146538 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e000c200 con 0x7f51e00047d0 2023-10-30T13:07:10.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51fa405640 1 --2- 172.21.15.125:0/620146538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51e0005190 0x7f51e000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:10.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f9c04640 1 --2- 172.21.15.125:0/620146538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e0003980 0x7f51e0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:10.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51f9403640 1 --2- 172.21.15.125:0/620146538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51e00047d0 0x7f51e0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:10.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51fa405640 1 -- 172.21.15.125:0/620146538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51e00047d0 msgr2=0x7f51e0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:10.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51fa405640 1 --2- 172.21.15.125:0/620146538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51e00047d0 0x7f51e0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:10.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51fa405640 1 -- 172.21.15.125:0/620146538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e0003980 msgr2=0x7f51e0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:10.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51fa405640 1 --2- 172.21.15.125:0/620146538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51e0003980 0x7f51e0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:10.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.809+0000 7f51fa405640 1 -- 172.21.15.125:0/620146538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e0003530 con 0x7f51e0005190 2023-10-30T13:07:10.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.813+0000 7f51fa405640 1 -- 172.21.15.125:0/620146538 reap_dead start 2023-10-30T13:07:10.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.813+0000 7f51fa405640 1 --2- 172.21.15.125:0/620146538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51e0005190 0x7f51e000b990 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00a830 tx=0x7f51ec01d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60a636ecfe500f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:10.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.813+0000 7f51f8c02640 1 -- 172.21.15.125:0/620146538 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0072d0 con 0x7f51e0005190 2023-10-30T13:07:10.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.813+0000 7f51f8c02640 1 -- 172.21.15.125:0/620146538 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51ec0a8070 con 0x7f51e0005190 2023-10-30T13:07:10.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.813+0000 7f51f8c02640 1 -- 172.21.15.125:0/620146538 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0076b0 con 0x7f51e0005190 2023-10-30T13:07:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.813+0000 7f51fac06640 1 -- 172.21.15.125:0/620146538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51e0005190 msgr2=0x7f51e000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.813+0000 7f51fac06640 1 --2- 172.21.15.125:0/620146538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51e0005190 0x7f51e000b990 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f51ec00a830 tx=0x7f51ec01d340 comp rx=0 tx=0).stop 2023-10-30T13:07:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.813+0000 7f51fac06640 1 -- 172.21.15.125:0/620146538 shutdown_connections 2023-10-30T13:07:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.813+0000 7f51fac06640 1 --2- 172.21.15.125:0/620146538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51e0005190 0x7f51e000b990 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:10.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.813+0000 7f51fac06640 1 -- 172.21.15.125:0/620146538 >> 172.21.15.125:0/620146538 conn(0x7f51f406a640 msgr2=0x7f51f4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.813+0000 7f51fac06640 1 -- 172.21.15.125:0/620146538 shutdown_connections 2023-10-30T13:07:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.813+0000 7f51fac06640 1 -- 172.21.15.125:0/620146538 wait complete. 2023-10-30T13:07:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51fac06640 1 Processor -- start 2023-10-30T13:07:10.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51fac06640 1 -- start start 2023-10-30T13:07:10.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51fac06640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51e0005190 0x7f51f4152870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:10.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51fac06640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51f415c870 0x7f51f415a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:10.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51fac06640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51f41f2070 0x7f51f415ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:10.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51fac06640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f40b3d80 con 0x7f51e0005190 2023-10-30T13:07:10.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51fac06640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f40b2f20 con 0x7f51f415c870 2023-10-30T13:07:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51fac06640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f40b3200 con 0x7f51f41f2070 2023-10-30T13:07:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51f9c04640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51e0005190 0x7f51f4152870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51f9c04640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51e0005190 0x7f51f4152870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48858/0 (socket says 172.21.15.125:48858) 2023-10-30T13:07:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51f9c04640 1 -- 172.21.15.125:0/782515479 learned_addr learned my addr 172.21.15.125:0/782515479 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51f9403640 1 --2- 172.21.15.125:0/782515479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51f415c870 0x7f51f415a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51fa405640 1 --2- 172.21.15.125:0/782515479 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51f41f2070 0x7f51f415ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51f9c04640 1 -- 172.21.15.125:0/782515479 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51f41f2070 msgr2=0x7f51f415ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:10.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51f9c04640 1 --2- 172.21.15.125:0/782515479 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51f41f2070 0x7f51f415ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:10.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51f9c04640 1 -- 172.21.15.125:0/782515479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51f415c870 msgr2=0x7f51f415a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:10.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51f9c04640 1 --2- 172.21.15.125:0/782515479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51f415c870 0x7f51f415a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:10.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51f9c04640 1 -- 172.21.15.125:0/782515479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f40b34e0 con 0x7f51e0005190 2023-10-30T13:07:10.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51f9c04640 1 --2- 172.21.15.125:0/782515479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51e0005190 0x7f51f4152870 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f51f000f5f0 tx=0x7f51f000fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51ea7fc640 1 -- 172.21.15.125:0/782515479 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f00052e0 con 0x7f51e0005190 2023-10-30T13:07:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51ea7fc640 1 -- 172.21.15.125:0/782515479 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f0005440 con 0x7f51e0005190 2023-10-30T13:07:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51fac06640 1 -- 172.21.15.125:0/782515479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f40b37c0 con 0x7f51e0005190 2023-10-30T13:07:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51ea7fc640 1 -- 172.21.15.125:0/782515479 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f0003820 con 0x7f51e0005190 2023-10-30T13:07:10.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.817+0000 7f51fac06640 1 -- 172.21.15.125:0/782515479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f40c9260 con 0x7f51e0005190 2023-10-30T13:07:10.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.821+0000 7f51fac06640 1 -- 172.21.15.125:0/782515479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f404e170 con 0x7f51e0005190 2023-10-30T13:07:10.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.821+0000 7f51ea7fc640 1 -- 172.21.15.125:0/782515479 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51f0003980 con 0x7f51e0005190 2023-10-30T13:07:10.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.821+0000 7f51ea7fc640 1 --2- 172.21.15.125:0/782515479 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51d80664d0 0x7f51d8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:10.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.821+0000 7f51ea7fc640 1 -- 172.21.15.125:0/782515479 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f51f00843f0 con 0x7f51e0005190 2023-10-30T13:07:10.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.821+0000 7f51f9403640 1 --2- 172.21.15.125:0/782515479 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51d80664d0 0x7f51d8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:10.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.821+0000 7f51f9403640 1 --2- 172.21.15.125:0/782515479 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51d80664d0 0x7f51d8068990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f51f41530c0 tx=0x7f51e4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:10.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:10.825+0000 7f51ea7fc640 1 -- 172.21.15.125:0/782515479 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51f00511b0 con 0x7f51e0005190 2023-10-30T13:07:11.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:11.009+0000 7f51fac06640 1 -- 172.21.15.125:0/782515479 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51f40b2290 con 0x7f51e0005190 2023-10-30T13:07:11.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:11.009+0000 7f51ea7fc640 1 -- 172.21.15.125:0/782515479 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f51f00547d0 con 0x7f51e0005190 2023-10-30T13:07:11.013 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:11.014 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:09.749853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:reconnect","state_seq":426,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:11.014 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 174 2023-10-30T13:07:11.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:11.009+0000 7f51fac06640 1 -- 172.21.15.125:0/782515479 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51d80664d0 msgr2=0x7f51d8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:11.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:11.009+0000 7f51fac06640 1 --2- 172.21.15.125:0/782515479 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51d80664d0 0x7f51d8068990 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f51f41530c0 tx=0x7f51e4008040 comp rx=0 tx=0).stop 2023-10-30T13:07:11.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:11.009+0000 7f51fac06640 1 -- 172.21.15.125:0/782515479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51e0005190 msgr2=0x7f51f4152870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:11.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:11.009+0000 7f51fac06640 1 --2- 172.21.15.125:0/782515479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51e0005190 0x7f51f4152870 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f51f000f5f0 tx=0x7f51f000fab0 comp rx=0 tx=0).stop 2023-10-30T13:07:11.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:11.009+0000 7f51fac06640 1 -- 172.21.15.125:0/782515479 shutdown_connections 2023-10-30T13:07:11.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:11.009+0000 7f51fac06640 1 --2- 172.21.15.125:0/782515479 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51d80664d0 0x7f51d8068990 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:11.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:11.009+0000 7f51fac06640 1 --2- 172.21.15.125:0/782515479 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f51f41f2070 0x7f51f415ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:11.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:11.009+0000 7f51fac06640 1 --2- 172.21.15.125:0/782515479 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f51f415c870 0x7f51f415a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:11.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:11.009+0000 7f51fac06640 1 --2- 172.21.15.125:0/782515479 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f51e0005190 0x7f51f4152870 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:11.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:11.009+0000 7f51fac06640 1 -- 172.21.15.125:0/782515479 >> 172.21.15.125:0/782515479 conn(0x7f51f406a640 msgr2=0x7f51f40cf6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:11.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:11.009+0000 7f51fac06640 1 -- 172.21.15.125:0/782515479 shutdown_connections 2023-10-30T13:07:11.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:11.009+0000 7f51fac06640 1 -- 172.21.15.125:0/782515479 wait complete. 2023-10-30T13:07:11.034 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:07:13.036 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:13.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.177+0000 7fde7ec6b640 1 Processor -- start 2023-10-30T13:07:13.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde7ec6b640 1 -- start start 2023-10-30T13:07:13.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde7ec6b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fde780c6b80 0x7fde780bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:13.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde7ec6b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fde780c6f60 0x7fde780c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:13.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde7ec6b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde780c0a50 con 0x7fde780c8430 2023-10-30T13:07:13.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde7ec6b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7806f6e0 con 0x7fde780c6b80 2023-10-30T13:07:13.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde7ec6b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde7806f9c0 con 0x7fde780c6f60 2023-10-30T13:07:13.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde7ca07640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fde780c6b80 0x7fde780bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:13.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde7ca07640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fde780c6b80 0x7fde780bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48866/0 (socket says 172.21.15.125:48866) 2023-10-30T13:07:13.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde7ca07640 1 -- 172.21.15.125:0/3928727185 learned_addr learned my addr 172.21.15.125:0/3928727185 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:13.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde7d208640 1 --1- 172.21.15.125:0/3928727185 >> v1:172.21.15.125:6789/0 conn(0x7fde780c8430 0x7fde7813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52712/0 (socket says 172.21.15.125:52712) 2023-10-30T13:07:13.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde6ffff640 1 --2- 172.21.15.125:0/3928727185 >> v2:172.21.15.136:3300/0 conn(0x7fde780c6f60 0x7fde780c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:13.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde7ca07640 1 -- 172.21.15.125:0/3928727185 >> v2:172.21.15.136:3300/0 conn(0x7fde780c6f60 msgr2=0x7fde780c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:13.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde7ca07640 1 --2- 172.21.15.125:0/3928727185 >> v2:172.21.15.136:3300/0 conn(0x7fde780c6f60 0x7fde780c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:13.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde7ca07640 1 -- 172.21.15.125:0/3928727185 >> v1:172.21.15.125:6789/0 conn(0x7fde780c8430 legacy=0x7fde7813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:13.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.181+0000 7fde7ca07640 1 -- 172.21.15.125:0/3928727185 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7806fca0 con 0x7fde780c6b80 2023-10-30T13:07:13.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.185+0000 7fde7ca07640 1 --2- 172.21.15.125:0/3928727185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde780c6b80 0x7fde780bfec0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fde78159b90 tx=0x7fde7401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5cf34f9d604aa12 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:13.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.185+0000 7fde6f7fe640 1 -- 172.21.15.125:0/3928727185 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde740ab610 con 0x7fde780c6b80 2023-10-30T13:07:13.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.185+0000 7fde6f7fe640 1 -- 172.21.15.125:0/3928727185 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde740ab770 con 0x7fde780c6b80 2023-10-30T13:07:13.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.185+0000 7fde6f7fe640 1 -- 172.21.15.125:0/3928727185 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde740abb00 con 0x7fde780c6b80 2023-10-30T13:07:13.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.185+0000 7fde7ec6b640 1 -- 172.21.15.125:0/3928727185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde780c6b80 msgr2=0x7fde780bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:13.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.185+0000 7fde7ec6b640 1 --2- 172.21.15.125:0/3928727185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde780c6b80 0x7fde780bfec0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fde78159b90 tx=0x7fde7401c500 comp rx=0 tx=0).stop 2023-10-30T13:07:13.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.185+0000 7fde7ec6b640 1 -- 172.21.15.125:0/3928727185 shutdown_connections 2023-10-30T13:07:13.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.185+0000 7fde7ec6b640 1 --2- 172.21.15.125:0/3928727185 >> v2:172.21.15.136:3300/0 conn(0x7fde780c6f60 0x7fde780c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:13.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.185+0000 7fde7ec6b640 1 --2- 172.21.15.125:0/3928727185 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde780c6b80 0x7fde780bfec0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:13.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.185+0000 7fde7ec6b640 1 -- 172.21.15.125:0/3928727185 >> 172.21.15.125:0/3928727185 conn(0x7fde780c7910 msgr2=0x7fde780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:13.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.185+0000 7fde7ec6b640 1 -- 172.21.15.125:0/3928727185 shutdown_connections 2023-10-30T13:07:13.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.185+0000 7fde7ec6b640 1 -- 172.21.15.125:0/3928727185 wait complete. 2023-10-30T13:07:13.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.185+0000 7fde7ec6b640 1 Processor -- start 2023-10-30T13:07:13.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.185+0000 7fde7ec6b640 1 -- start start 2023-10-30T13:07:13.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde7ec6b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde780c6b80 0x7fde781edcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:13.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde7ec6b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde780c6f60 0x7fde7813d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:13.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde7ec6b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde780c8430 0x7fde78134ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:13.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde7ec6b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde78150fe0 con 0x7fde780c6f60 2023-10-30T13:07:13.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde7ec6b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde781f1240 con 0x7fde780c8430 2023-10-30T13:07:13.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde7ec6b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde781f1ae0 con 0x7fde780c6b80 2023-10-30T13:07:13.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde6ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde780c6f60 0x7fde7813d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:13.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde6ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde780c6f60 0x7fde7813d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48880/0 (socket says 172.21.15.125:48880) 2023-10-30T13:07:13.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde6ffff640 1 -- 172.21.15.125:0/2896151761 learned_addr learned my addr 172.21.15.125:0/2896151761 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:13.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde7d208640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde780c8430 0x7fde78134ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:13.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde7ca07640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde780c6b80 0x7fde781edcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:13.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde6ffff640 1 -- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde780c6b80 msgr2=0x7fde781edcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:13.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde6ffff640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde780c6b80 0x7fde781edcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:13.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde6ffff640 1 -- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde780c8430 msgr2=0x7fde78134ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:13.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde6ffff640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde780c8430 0x7fde78134ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:13.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde6ffff640 1 -- 172.21.15.125:0/2896151761 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde740094d0 con 0x7fde780c6f60 2023-10-30T13:07:13.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde7ca07640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde780c6b80 0x7fde781edcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:07:13.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde6ffff640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde780c6f60 0x7fde7813d600 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fde6800d830 tx=0x7fde6800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:13.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde6d7fa640 1 -- 172.21.15.125:0/2896151761 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6800b660 con 0x7fde780c6f60 2023-10-30T13:07:13.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde6d7fa640 1 -- 172.21.15.125:0/2896151761 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde6800b7c0 con 0x7fde780c6f60 2023-10-30T13:07:13.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde7ec6b640 1 -- 172.21.15.125:0/2896151761 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde780ca420 con 0x7fde780c6f60 2023-10-30T13:07:13.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde6d7fa640 1 -- 172.21.15.125:0/2896151761 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde6800bb30 con 0x7fde780c6f60 2023-10-30T13:07:13.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde7ec6b640 1 -- 172.21.15.125:0/2896151761 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde780ca9c0 con 0x7fde780c6f60 2023-10-30T13:07:13.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde7ec6b640 1 -- 172.21.15.125:0/2896151761 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7804e170 con 0x7fde780c6f60 2023-10-30T13:07:13.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.189+0000 7fde6d7fa640 1 -- 172.21.15.125:0/2896151761 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fde6801e460 con 0x7fde780c6f60 2023-10-30T13:07:13.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.193+0000 7fde6d7fa640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fde640664a0 0x7fde64068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:13.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.193+0000 7fde6d7fa640 1 -- 172.21.15.125:0/2896151761 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7fde68084e60 con 0x7fde780c6f60 2023-10-30T13:07:13.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.193+0000 7fde7ca07640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fde640664a0 0x7fde64068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:13.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.193+0000 7fde7ca07640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fde640664a0 0x7fde64068960 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fde740af000 tx=0x7fde74008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:13.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.197+0000 7fde6d7fa640 1 -- 172.21.15.125:0/2896151761 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fde68051a80 con 0x7fde780c6f60 2023-10-30T13:07:13.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.377+0000 7fde7ec6b640 1 -- 172.21.15.125:0/2896151761 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde781f1c20 con 0x7fde780c6f60 2023-10-30T13:07:13.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.377+0000 7fde6d7fa640 1 -- 172.21.15.125:0/2896151761 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+5756 (secure 0 0 0) 0x7fde680550a0 con 0x7fde780c6f60 2023-10-30T13:07:13.382 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:13.383 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:09.749853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:reconnect","state_seq":426,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:13.384 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 174 2023-10-30T13:07:13.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.381+0000 7fde7ec6b640 1 -- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fde640664a0 msgr2=0x7fde64068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:13.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.381+0000 7fde7ec6b640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fde640664a0 0x7fde64068960 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7fde740af000 tx=0x7fde74008040 comp rx=0 tx=0).stop 2023-10-30T13:07:13.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.381+0000 7fde7ec6b640 1 -- 172.21.15.125:0/2896151761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde780c6f60 msgr2=0x7fde7813d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:13.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.381+0000 7fde7ec6b640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde780c6f60 0x7fde7813d600 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7fde6800d830 tx=0x7fde6800dcf0 comp rx=0 tx=0).stop 2023-10-30T13:07:13.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.381+0000 7fde7ec6b640 1 -- 172.21.15.125:0/2896151761 shutdown_connections 2023-10-30T13:07:13.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.381+0000 7fde7ec6b640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fde640664a0 0x7fde64068960 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:13.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.381+0000 7fde7ec6b640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fde780c8430 0x7fde78134ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:13.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.381+0000 7fde7ec6b640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fde780c6f60 0x7fde7813d600 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:13.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.381+0000 7fde7ec6b640 1 --2- 172.21.15.125:0/2896151761 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fde780c6b80 0x7fde781edcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:13.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.381+0000 7fde7ec6b640 1 -- 172.21.15.125:0/2896151761 >> 172.21.15.125:0/2896151761 conn(0x7fde780c7910 msgr2=0x7fde780b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:13.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.381+0000 7fde7ec6b640 1 -- 172.21.15.125:0/2896151761 shutdown_connections 2023-10-30T13:07:13.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:13.381+0000 7fde7ec6b640 1 -- 172.21.15.125:0/2896151761 wait complete. 2023-10-30T13:07:13.403 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:07:15.404 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:15.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.545+0000 7fd6e453e640 1 Processor -- start 2023-10-30T13:07:15.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.545+0000 7fd6e453e640 1 -- start start 2023-10-30T13:07:15.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e453e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6dc0715c0 0x7fd6dc070650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:15.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e453e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6dc0c1530 con 0x7fd6dc0c8460 2023-10-30T13:07:15.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e453e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6dc0c0f70 con 0x7fd6dc157010 2023-10-30T13:07:15.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e453e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc0c21b0 con 0x7fd6dc0715c0 2023-10-30T13:07:15.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e1ad9640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd6dc0c8460 0x7fd6dc1620a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52726/0 (socket says 172.21.15.125:52726) 2023-10-30T13:07:15.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e1ad9640 1 -- 172.21.15.125:0/263203325 learned_addr learned my addr 172.21.15.125:0/263203325 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:07:15.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e2adb640 1 --2- 172.21.15.125:0/263203325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6dc0715c0 0x7fd6dc070650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:15.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 -- 172.21.15.125:0/263203325 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080612287 0 0) 0x7fd6dc0c1530 con 0x7fd6dc0c8460 2023-10-30T13:07:15.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 -- 172.21.15.125:0/263203325 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c8003530 con 0x7fd6dc0c8460 2023-10-30T13:07:15.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 -- 172.21.15.125:0/263203325 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1322032645 0 0) 0x7fd6c8003530 con 0x7fd6dc0c8460 2023-10-30T13:07:15.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 -- 172.21.15.125:0/263203325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6dc0715c0 msgr2=0x7fd6dc070650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 --2- 172.21.15.125:0/263203325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6dc0715c0 0x7fd6dc070650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 -- 172.21.15.125:0/263203325 >> v1:172.21.15.136:6789/0 conn(0x7fd6dc157010 legacy=0x7fd6dc071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 -- 172.21.15.125:0/263203325 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc0c1250 con 0x7fd6dc0c8460 2023-10-30T13:07:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 -- 172.21.15.125:0/263203325 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fd6cc001680 con 0x7fd6dc0c8460 2023-10-30T13:07:15.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 -- 172.21.15.125:0/263203325 >> v1:172.21.15.125:6789/0 conn(0x7fd6dc0c8460 legacy=0x7fd6dc1620a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 --2- 172.21.15.125:0/263203325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80039b0 0x7fd6c8003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 --2- 172.21.15.125:0/263203325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6dc0715c0 0x7fd6c80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 --2- 172.21.15.125:0/263203325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6c8004d10 0x7fd6c8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 -- 172.21.15.125:0/263203325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc0c21b0 con 0x7fd6c80039b0 2023-10-30T13:07:15.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 -- 172.21.15.125:0/263203325 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c800ba70 con 0x7fd6c8004d10 2023-10-30T13:07:15.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e12d8640 1 -- 172.21.15.125:0/263203325 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c800bc30 con 0x7fd6dc0715c0 2023-10-30T13:07:15.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e22da640 1 --2- 172.21.15.125:0/263203325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80039b0 0x7fd6c8003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:15.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e1ad9640 1 --2- 172.21.15.125:0/263203325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6dc0715c0 0x7fd6c80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:15.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e2adb640 1 --2- 172.21.15.125:0/263203325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6c8004d10 0x7fd6c8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:15.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e22da640 1 -- 172.21.15.125:0/263203325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6dc0715c0 msgr2=0x7fd6c80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:15.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e22da640 1 --2- 172.21.15.125:0/263203325 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6dc0715c0 0x7fd6c80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:15.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e22da640 1 -- 172.21.15.125:0/263203325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6c8004d10 msgr2=0x7fd6c8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:15.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e22da640 1 --2- 172.21.15.125:0/263203325 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6c8004d10 0x7fd6c8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:15.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.549+0000 7fd6e22da640 1 -- 172.21.15.125:0/263203325 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6c8003530 con 0x7fd6c80039b0 2023-10-30T13:07:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.553+0000 7fd6e22da640 1 --2- 172.21.15.125:0/263203325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80039b0 0x7fd6c8003dd0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8001750 tx=0x7fd6d801c390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb00011b8eaa3f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.553+0000 7fd6e12d8640 1 -- 172.21.15.125:0/263203325 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d80052d0 con 0x7fd6c80039b0 2023-10-30T13:07:15.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.553+0000 7fd6e12d8640 1 -- 172.21.15.125:0/263203325 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d80a7050 con 0x7fd6c80039b0 2023-10-30T13:07:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.553+0000 7fd6e12d8640 1 -- 172.21.15.125:0/263203325 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d8003600 con 0x7fd6c80039b0 2023-10-30T13:07:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.553+0000 7fd6e453e640 1 -- 172.21.15.125:0/263203325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80039b0 msgr2=0x7fd6c8003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.553+0000 7fd6e453e640 1 --2- 172.21.15.125:0/263203325 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6c80039b0 0x7fd6c8003dd0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fd6d8001750 tx=0x7fd6d801c390 comp rx=0 tx=0).stop 2023-10-30T13:07:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.553+0000 7fd6e2adb640 1 -- 172.21.15.125:0/263203325 reap_dead start 2023-10-30T13:07:15.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.553+0000 7fd6e453e640 1 -- 172.21.15.125:0/263203325 shutdown_connections 2023-10-30T13:07:15.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.553+0000 7fd6e453e640 1 -- 172.21.15.125:0/263203325 >> 172.21.15.125:0/263203325 conn(0x7fd6dc0cbeb0 msgr2=0x7fd6dc0cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:15.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.553+0000 7fd6e453e640 1 -- 172.21.15.125:0/263203325 shutdown_connections 2023-10-30T13:07:15.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.553+0000 7fd6e453e640 1 -- 172.21.15.125:0/263203325 wait complete. 2023-10-30T13:07:15.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e453e640 1 Processor -- start 2023-10-30T13:07:15.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e453e640 1 -- start start 2023-10-30T13:07:15.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e453e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6c80039b0 0x7fd6dc1631f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:15.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e453e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6dc1f58a0 0x7fd6dc0c6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:15.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e453e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6dc06ba40 0x7fd6dc14daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:15.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e453e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc0c7550 con 0x7fd6dc1f58a0 2023-10-30T13:07:15.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e453e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc0c6d90 con 0x7fd6dc06ba40 2023-10-30T13:07:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e453e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc0c4a40 con 0x7fd6c80039b0 2023-10-30T13:07:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e1ad9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6dc1f58a0 0x7fd6dc0c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e1ad9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6dc1f58a0 0x7fd6dc0c6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48896/0 (socket says 172.21.15.125:48896) 2023-10-30T13:07:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e1ad9640 1 -- 172.21.15.125:0/3603946581 learned_addr learned my addr 172.21.15.125:0/3603946581 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e22da640 1 --2- 172.21.15.125:0/3603946581 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6c80039b0 0x7fd6dc1631f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:15.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e2adb640 1 --2- 172.21.15.125:0/3603946581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6dc06ba40 0x7fd6dc14daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:15.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e1ad9640 1 -- 172.21.15.125:0/3603946581 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6c80039b0 msgr2=0x7fd6dc1631f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:15.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e1ad9640 1 --2- 172.21.15.125:0/3603946581 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6c80039b0 0x7fd6dc1631f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:15.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e1ad9640 1 -- 172.21.15.125:0/3603946581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6dc06ba40 msgr2=0x7fd6dc14daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:15.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e1ad9640 1 --2- 172.21.15.125:0/3603946581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6dc06ba40 0x7fd6dc14daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:15.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e1ad9640 1 -- 172.21.15.125:0/3603946581 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc0c3270 con 0x7fd6dc1f58a0 2023-10-30T13:07:15.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e1ad9640 1 --2- 172.21.15.125:0/3603946581 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6dc1f58a0 0x7fd6dc0c6850 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00e730 tx=0x7fd6cc00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:15.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6d2ffd640 1 -- 172.21.15.125:0/3603946581 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc00ee40 con 0x7fd6dc1f58a0 2023-10-30T13:07:15.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6d2ffd640 1 -- 172.21.15.125:0/3603946581 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6cc00c8d0 con 0x7fd6dc1f58a0 2023-10-30T13:07:15.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6d2ffd640 1 -- 172.21.15.125:0/3603946581 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc00cc80 con 0x7fd6dc1f58a0 2023-10-30T13:07:15.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e453e640 1 -- 172.21.15.125:0/3603946581 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6dc06b5b0 con 0x7fd6dc1f58a0 2023-10-30T13:07:15.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e453e640 1 -- 172.21.15.125:0/3603946581 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6dc0c34c0 con 0x7fd6dc1f58a0 2023-10-30T13:07:15.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.557+0000 7fd6e453e640 1 -- 172.21.15.125:0/3603946581 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6dc06b740 con 0x7fd6dc1f58a0 2023-10-30T13:07:15.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.561+0000 7fd6d2ffd640 1 -- 172.21.15.125:0/3603946581 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd6cc013420 con 0x7fd6dc1f58a0 2023-10-30T13:07:15.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.561+0000 7fd6d2ffd640 1 --2- 172.21.15.125:0/3603946581 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6c00664a0 0x7fd6c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:15.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.561+0000 7fd6d2ffd640 1 -- 172.21.15.125:0/3603946581 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7fd6cc084190 con 0x7fd6dc1f58a0 2023-10-30T13:07:15.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.561+0000 7fd6e22da640 1 --2- 172.21.15.125:0/3603946581 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6c00664a0 0x7fd6c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:15.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.561+0000 7fd6e22da640 1 --2- 172.21.15.125:0/3603946581 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6c00664a0 0x7fd6c0068960 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80057b0 tx=0x7fd6d80011a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:15.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.565+0000 7fd6d2ffd640 1 -- 172.21.15.125:0/3603946581 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd6cc050f50 con 0x7fd6dc1f58a0 2023-10-30T13:07:15.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.745+0000 7fd6e453e640 1 -- 172.21.15.125:0/3603946581 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6dc04e170 con 0x7fd6dc1f58a0 2023-10-30T13:07:15.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.745+0000 7fd6d2ffd640 1 -- 172.21.15.125:0/3603946581 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 174 v174) v1 ==== 95+0+5756 (secure 0 0 0) 0x7fd6cc054570 con 0x7fd6dc1f58a0 2023-10-30T13:07:15.750 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:15.751 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":174,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":174,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:09.749853+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:reconnect","state_seq":426,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:15.751 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 174 2023-10-30T13:07:15.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.749+0000 7fd6e453e640 1 -- 172.21.15.125:0/3603946581 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6c00664a0 msgr2=0x7fd6c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:15.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.749+0000 7fd6e453e640 1 --2- 172.21.15.125:0/3603946581 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6c00664a0 0x7fd6c0068960 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80057b0 tx=0x7fd6d80011a0 comp rx=0 tx=0).stop 2023-10-30T13:07:15.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.749+0000 7fd6e453e640 1 -- 172.21.15.125:0/3603946581 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6dc1f58a0 msgr2=0x7fd6dc0c6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:15.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.749+0000 7fd6e453e640 1 --2- 172.21.15.125:0/3603946581 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6dc1f58a0 0x7fd6dc0c6850 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00e730 tx=0x7fd6cc00ebf0 comp rx=0 tx=0).stop 2023-10-30T13:07:15.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.749+0000 7fd6e453e640 1 -- 172.21.15.125:0/3603946581 shutdown_connections 2023-10-30T13:07:15.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.749+0000 7fd6e453e640 1 --2- 172.21.15.125:0/3603946581 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd6c00664a0 0x7fd6c0068960 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:15.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.749+0000 7fd6e453e640 1 --2- 172.21.15.125:0/3603946581 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd6dc06ba40 0x7fd6dc14daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:15.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.749+0000 7fd6e453e640 1 --2- 172.21.15.125:0/3603946581 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd6dc1f58a0 0x7fd6dc0c6850 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:15.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.749+0000 7fd6e453e640 1 --2- 172.21.15.125:0/3603946581 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd6c80039b0 0x7fd6dc1631f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:15.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.749+0000 7fd6e453e640 1 -- 172.21.15.125:0/3603946581 >> 172.21.15.125:0/3603946581 conn(0x7fd6dc0cbeb0 msgr2=0x7fd6dc151040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:15.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.749+0000 7fd6e453e640 1 -- 172.21.15.125:0/3603946581 shutdown_connections 2023-10-30T13:07:15.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:15.749+0000 7fd6e453e640 1 -- 172.21.15.125:0/3603946581 wait complete. 2023-10-30T13:07:15.771 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:07:17.773 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:17.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f200c046640 1 Processor -- start 2023-10-30T13:07:17.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f200c046640 1 -- start start 2023-10-30T13:07:17.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f200c046640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f200415d6f0 0x7f200415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:17.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f200c046640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20040cb800 con 0x7f20041538a0 2023-10-30T13:07:17.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f200c046640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20040cb330 con 0x7f20040cf160 2023-10-30T13:07:17.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f200c046640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20040cad70 con 0x7f200415d6f0 2023-10-30T13:07:17.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f2009de2640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f20041538a0 0x7f20040ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52742/0 (socket says 172.21.15.125:52742) 2023-10-30T13:07:17.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f2009de2640 1 -- 172.21.15.125:0/956073690 learned_addr learned my addr 172.21.15.125:0/956073690 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:07:17.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f20095e1640 1 --2- 172.21.15.125:0/956073690 >> v2:172.21.15.136:3300/0 conn(0x7f200415d6f0 0x7f200415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:17.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f2008de0640 1 -- 172.21.15.125:0/956073690 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555029301 0 0) 0x7f20040cb800 con 0x7f20041538a0 2023-10-30T13:07:17.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f2008de0640 1 -- 172.21.15.125:0/956073690 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff0003530 con 0x7f20041538a0 2023-10-30T13:07:17.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f2008de0640 1 -- 172.21.15.125:0/956073690 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2633096390 0 0) 0x7f1ff0003530 con 0x7f20041538a0 2023-10-30T13:07:17.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f2008de0640 1 -- 172.21.15.125:0/956073690 >> v2:172.21.15.136:3300/0 conn(0x7f200415d6f0 msgr2=0x7f200415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:17.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f2008de0640 1 --2- 172.21.15.125:0/956073690 >> v2:172.21.15.136:3300/0 conn(0x7f200415d6f0 0x7f200415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:17.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f2008de0640 1 -- 172.21.15.125:0/956073690 >> v1:172.21.15.136:6789/0 conn(0x7f20040cf160 legacy=0x7f2004164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:17.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f2008de0640 1 -- 172.21.15.125:0/956073690 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20040cb050 con 0x7f20041538a0 2023-10-30T13:07:17.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f2008de0640 1 -- 172.21.15.125:0/956073690 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f2000001680 con 0x7f20041538a0 2023-10-30T13:07:17.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.917+0000 7f2008de0640 1 -- 172.21.15.125:0/956073690 >> v1:172.21.15.125:6789/0 conn(0x7f20041538a0 legacy=0x7f20040ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:17.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f2008de0640 1 --2- 172.21.15.125:0/956073690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ff0003980 0x7f1ff0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:17.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f2008de0640 1 --2- 172.21.15.125:0/956073690 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ff00047d0 0x7f1ff0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:17.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f2008de0640 1 --2- 172.21.15.125:0/956073690 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ff0005190 0x7f1ff000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:17.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f2008de0640 1 -- 172.21.15.125:0/956073690 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20040cad70 con 0x7f1ff00047d0 2023-10-30T13:07:17.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f2008de0640 1 -- 172.21.15.125:0/956073690 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff000c040 con 0x7f1ff0003980 2023-10-30T13:07:17.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f2008de0640 1 -- 172.21.15.125:0/956073690 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff000c200 con 0x7f1ff0005190 2023-10-30T13:07:17.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f20095e1640 1 --2- 172.21.15.125:0/956073690 >> v2:172.21.15.136:3300/0 conn(0x7f200415d6f0 0x7f200415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:07:17.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f20095e1640 1 --2- 172.21.15.125:0/956073690 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ff00047d0 0x7f1ff0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:17.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f2009de2640 1 --2- 172.21.15.125:0/956073690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ff0003980 0x7f1ff0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:17.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f200a5e3640 1 --2- 172.21.15.125:0/956073690 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ff0005190 0x7f1ff000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:17.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f20095e1640 1 -- 172.21.15.125:0/956073690 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ff0005190 msgr2=0x7f1ff000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:17.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f20095e1640 1 --2- 172.21.15.125:0/956073690 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ff0005190 0x7f1ff000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:17.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f20095e1640 1 -- 172.21.15.125:0/956073690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ff0003980 msgr2=0x7f1ff0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:17.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f20095e1640 1 --2- 172.21.15.125:0/956073690 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ff0003980 0x7f1ff0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:17.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f20095e1640 1 -- 172.21.15.125:0/956073690 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff0003530 con 0x7f1ff00047d0 2023-10-30T13:07:17.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f200a5e3640 1 -- 172.21.15.125:0/956073690 reap_dead start 2023-10-30T13:07:17.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f20095e1640 1 --2- 172.21.15.125:0/956073690 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ff00047d0 0x7f1ff0004c50 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009c90 tx=0x7f1ff401cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692c8da6b35ed9c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:17.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f2008de0640 1 -- 172.21.15.125:0/956073690 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4004020 con 0x7f1ff00047d0 2023-10-30T13:07:17.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.921+0000 7f2008de0640 1 -- 172.21.15.125:0/956073690 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff4004180 con 0x7f1ff00047d0 2023-10-30T13:07:17.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f2008de0640 1 -- 172.21.15.125:0/956073690 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff40a58b0 con 0x7f1ff00047d0 2023-10-30T13:07:17.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 -- 172.21.15.125:0/956073690 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ff00047d0 msgr2=0x7f1ff0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:17.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 --2- 172.21.15.125:0/956073690 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ff00047d0 0x7f1ff0004c50 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4009c90 tx=0x7f1ff401cbf0 comp rx=0 tx=0).stop 2023-10-30T13:07:17.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 -- 172.21.15.125:0/956073690 shutdown_connections 2023-10-30T13:07:17.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 --2- 172.21.15.125:0/956073690 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ff00047d0 0x7f1ff0004c50 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:17.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 -- 172.21.15.125:0/956073690 >> 172.21.15.125:0/956073690 conn(0x7f200406bfe0 msgr2=0x7f200406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:17.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 -- 172.21.15.125:0/956073690 shutdown_connections 2023-10-30T13:07:17.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 -- 172.21.15.125:0/956073690 wait complete. 2023-10-30T13:07:17.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 Processor -- start 2023-10-30T13:07:17.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 -- start start 2023-10-30T13:07:17.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ff00047d0 0x7f20041e6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20041e5390 0x7f20041f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20041e4000 0x7f200414cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20041e51e0 con 0x7f20041e5390 2023-10-30T13:07:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20041e4d50 con 0x7f1ff00047d0 2023-10-30T13:07:17.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f200c046640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20041e5840 con 0x7f20041e4000 2023-10-30T13:07:17.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f20095e1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20041e5390 0x7f20041f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:17.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f20095e1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20041e5390 0x7f20041f23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48908/0 (socket says 172.21.15.125:48908) 2023-10-30T13:07:17.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.925+0000 7f20095e1640 1 -- 172.21.15.125:0/3761908454 learned_addr learned my addr 172.21.15.125:0/3761908454 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:17.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f2009de2640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ff00047d0 0x7f20041e6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:17.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f200a5e3640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20041e4000 0x7f200414cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:17.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f20095e1640 1 -- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20041e4000 msgr2=0x7f200414cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:17.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f20095e1640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20041e4000 0x7f200414cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:17.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f20095e1640 1 -- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ff00047d0 msgr2=0x7f20041e6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:17.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f20095e1640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ff00047d0 0x7f20041e6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:17.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f20095e1640 1 -- 172.21.15.125:0/3761908454 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2004150490 con 0x7f20041e5390 2023-10-30T13:07:17.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f2009de2640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ff00047d0 0x7f20041e6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:07:17.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f20095e1640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20041e5390 0x7f20041f23c0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f1ff401ca20 tx=0x7f1ff0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:17.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f1ffa7fc640 1 -- 172.21.15.125:0/3761908454 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff40a5ab0 con 0x7f20041e5390 2023-10-30T13:07:17.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f1ffa7fc640 1 -- 172.21.15.125:0/3761908454 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff4004500 con 0x7f20041e5390 2023-10-30T13:07:17.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f1ffa7fc640 1 -- 172.21.15.125:0/3761908454 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff40a5e70 con 0x7f20041e5390 2023-10-30T13:07:17.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f200c046640 1 -- 172.21.15.125:0/3761908454 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2004150770 con 0x7f20041e5390 2023-10-30T13:07:17.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f200c046640 1 -- 172.21.15.125:0/3761908454 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20041e49e0 con 0x7f20041e5390 2023-10-30T13:07:17.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f200c046640 1 -- 172.21.15.125:0/3761908454 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20041e4ec0 con 0x7f20041e5390 2023-10-30T13:07:17.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f1ffa7fc640 1 -- 172.21.15.125:0/3761908454 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1ff4006040 con 0x7f20041e5390 2023-10-30T13:07:17.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f1ffa7fc640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1fe80664d0 0x7f1fe8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:17.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.929+0000 7f1ffa7fc640 1 -- 172.21.15.125:0/3761908454 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f1ff411f110 con 0x7f20041e5390 2023-10-30T13:07:17.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.933+0000 7f2009de2640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1fe80664d0 0x7f1fe8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:17.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.933+0000 7f2009de2640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1fe80664d0 0x7f1fe8068990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f2000007d50 tx=0x7f2000009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:17.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:17.933+0000 7f1ffa7fc640 1 -- 172.21.15.125:0/3761908454 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1ff40ebee0 con 0x7f20041e5390 2023-10-30T13:07:18.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:18.113+0000 7f200c046640 1 -- 172.21.15.125:0/3761908454 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f200404e170 con 0x7f20041e5390 2023-10-30T13:07:18.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:18.113+0000 7f1ffa7fc640 1 -- 172.21.15.125:0/3761908454 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5753 (secure 0 0 0) 0x7f1ff40ef500 con 0x7f20041e5390 2023-10-30T13:07:18.119 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:18.120 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:15.695661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:rejoin","state_seq":428,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:18.120 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 175 2023-10-30T13:07:18.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:18.117+0000 7f200c046640 1 -- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1fe80664d0 msgr2=0x7f1fe8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:18.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:18.117+0000 7f200c046640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1fe80664d0 0x7f1fe8068990 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f2000007d50 tx=0x7f2000009040 comp rx=0 tx=0).stop 2023-10-30T13:07:18.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:18.117+0000 7f200c046640 1 -- 172.21.15.125:0/3761908454 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20041e5390 msgr2=0x7f20041f23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:18.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:18.117+0000 7f200c046640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20041e5390 0x7f20041f23c0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f1ff401ca20 tx=0x7f1ff0005800 comp rx=0 tx=0).stop 2023-10-30T13:07:18.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:18.117+0000 7f200c046640 1 -- 172.21.15.125:0/3761908454 shutdown_connections 2023-10-30T13:07:18.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:18.117+0000 7f200c046640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1fe80664d0 0x7f1fe8068990 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:18.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:18.117+0000 7f200c046640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20041e4000 0x7f200414cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:18.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:18.117+0000 7f200c046640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20041e5390 0x7f20041f23c0 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:18.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:18.117+0000 7f200c046640 1 --2- 172.21.15.125:0/3761908454 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ff00047d0 0x7f20041e6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:18.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:18.117+0000 7f200c046640 1 -- 172.21.15.125:0/3761908454 >> 172.21.15.125:0/3761908454 conn(0x7f200406bfe0 msgr2=0x7f2004065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:18.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:18.117+0000 7f200c046640 1 -- 172.21.15.125:0/3761908454 shutdown_connections 2023-10-30T13:07:18.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:18.117+0000 7f200c046640 1 -- 172.21.15.125:0/3761908454 wait complete. 2023-10-30T13:07:18.140 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:07:20.142 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:20.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.285+0000 7f5d6fe32640 1 Processor -- start 2023-10-30T13:07:20.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.285+0000 7f5d6fe32640 1 -- start start 2023-10-30T13:07:20.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.285+0000 7f5d6fe32640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5d681538a0 0x7f5d6815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:20.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.285+0000 7f5d6fe32640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d680ca0e0 0x7f5d6815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:20.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.285+0000 7f5d6fe32640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5d680cf160 0x7f5d680cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:20.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.285+0000 7f5d6fe32640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d680cb800 con 0x7f5d680cf160 2023-10-30T13:07:20.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.285+0000 7f5d6fe32640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d6815dc30 con 0x7f5d681538a0 2023-10-30T13:07:20.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.285+0000 7f5d6fe32640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d680cb330 con 0x7f5d680ca0e0 2023-10-30T13:07:20.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.285+0000 7f5d6e3cf640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5d680cf160 0x7f5d680cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:20.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.285+0000 7f5d6e3cf640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5d680cf160 0x7f5d680cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48914/0 (socket says 172.21.15.125:48914) 2023-10-30T13:07:20.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.285+0000 7f5d6e3cf640 1 -- 172.21.15.125:0/46468626 learned_addr learned my addr 172.21.15.125:0/46468626 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:20.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.285+0000 7f5d6d3cd640 1 --2- 172.21.15.125:0/46468626 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d680ca0e0 0x7f5d6815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:20.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.289+0000 7f5d6dbce640 1 --2- 172.21.15.125:0/46468626 >> v2:172.21.15.136:3300/0 conn(0x7f5d681538a0 0x7f5d6815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:20.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.289+0000 7f5d6e3cf640 1 -- 172.21.15.125:0/46468626 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d680ca0e0 msgr2=0x7f5d6815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:20.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.289+0000 7f5d6e3cf640 1 --2- 172.21.15.125:0/46468626 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d680ca0e0 0x7f5d6815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:20.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.289+0000 7f5d6e3cf640 1 -- 172.21.15.125:0/46468626 >> v2:172.21.15.136:3300/0 conn(0x7f5d681538a0 msgr2=0x7f5d6815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:20.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.289+0000 7f5d6e3cf640 1 --2- 172.21.15.125:0/46468626 >> v2:172.21.15.136:3300/0 conn(0x7f5d681538a0 0x7f5d6815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:20.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.289+0000 7f5d6e3cf640 1 -- 172.21.15.125:0/46468626 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d680cad70 con 0x7f5d680cf160 2023-10-30T13:07:20.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.289+0000 7f5d6e3cf640 1 --2- 172.21.15.125:0/46468626 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d680cf160 0x7f5d680cf590 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f5d680cfd10 tx=0x7f5d6001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af898a049f64f40c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:20.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.289+0000 7f5d6cbcc640 1 -- 172.21.15.125:0/46468626 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d600af400 con 0x7f5d680cf160 2023-10-30T13:07:20.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.289+0000 7f5d6cbcc640 1 -- 172.21.15.125:0/46468626 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d600ab050 con 0x7f5d680cf160 2023-10-30T13:07:20.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.289+0000 7f5d6cbcc640 1 -- 172.21.15.125:0/46468626 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d600af780 con 0x7f5d680cf160 2023-10-30T13:07:20.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 -- 172.21.15.125:0/46468626 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d680cf160 msgr2=0x7f5d680cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:20.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 --2- 172.21.15.125:0/46468626 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d680cf160 0x7f5d680cf590 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f5d680cfd10 tx=0x7f5d6001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:07:20.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 -- 172.21.15.125:0/46468626 shutdown_connections 2023-10-30T13:07:20.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 --2- 172.21.15.125:0/46468626 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d680cf160 0x7f5d680cf590 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:20.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 --2- 172.21.15.125:0/46468626 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d680ca0e0 0x7f5d6815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:20.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 --2- 172.21.15.125:0/46468626 >> v2:172.21.15.136:3300/0 conn(0x7f5d681538a0 0x7f5d6815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:20.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 -- 172.21.15.125:0/46468626 >> 172.21.15.125:0/46468626 conn(0x7f5d6806bfe0 msgr2=0x7f5d6806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:20.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 -- 172.21.15.125:0/46468626 shutdown_connections 2023-10-30T13:07:20.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 -- 172.21.15.125:0/46468626 wait complete. 2023-10-30T13:07:20.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 Processor -- start 2023-10-30T13:07:20.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 -- start start 2023-10-30T13:07:20.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d680ca0e0 0x7f5d681f2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:20.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d680cf160 0x7f5d681f2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:20.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d681538a0 0x7f5d680b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:20.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d6815aac0 con 0x7f5d680ca0e0 2023-10-30T13:07:20.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d680cd9c0 con 0x7f5d681538a0 2023-10-30T13:07:20.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6fe32640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d681f35f0 con 0x7f5d680cf160 2023-10-30T13:07:20.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6dbce640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d680ca0e0 0x7f5d681f2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:20.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6dbce640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d680ca0e0 0x7f5d681f2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:48924/0 (socket says 172.21.15.125:48924) 2023-10-30T13:07:20.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6dbce640 1 -- 172.21.15.125:0/3408182321 learned_addr learned my addr 172.21.15.125:0/3408182321 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:20.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6d3cd640 1 --2- 172.21.15.125:0/3408182321 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d680cf160 0x7f5d681f2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:20.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6e3cf640 1 --2- 172.21.15.125:0/3408182321 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d681538a0 0x7f5d680b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:20.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6dbce640 1 -- 172.21.15.125:0/3408182321 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d680cf160 msgr2=0x7f5d681f2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:20.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6dbce640 1 --2- 172.21.15.125:0/3408182321 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d680cf160 0x7f5d681f2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:20.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6dbce640 1 -- 172.21.15.125:0/3408182321 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d681538a0 msgr2=0x7f5d680b3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:20.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6dbce640 1 --2- 172.21.15.125:0/3408182321 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d681538a0 0x7f5d680b3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:20.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.293+0000 7f5d6dbce640 1 -- 172.21.15.125:0/3408182321 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d60009b30 con 0x7f5d680ca0e0 2023-10-30T13:07:20.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.297+0000 7f5d6dbce640 1 --2- 172.21.15.125:0/3408182321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d680ca0e0 0x7f5d681f2a60 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f5d6400e870 tx=0x7f5d6400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:20.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.297+0000 7f5d5e7fc640 1 -- 172.21.15.125:0/3408182321 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d64009510 con 0x7f5d680ca0e0 2023-10-30T13:07:20.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.297+0000 7f5d5e7fc640 1 -- 172.21.15.125:0/3408182321 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d6400eec0 con 0x7f5d680ca0e0 2023-10-30T13:07:20.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.297+0000 7f5d5e7fc640 1 -- 172.21.15.125:0/3408182321 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d6400cb40 con 0x7f5d680ca0e0 2023-10-30T13:07:20.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.297+0000 7f5d6fe32640 1 -- 172.21.15.125:0/3408182321 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d680cdc80 con 0x7f5d680ca0e0 2023-10-30T13:07:20.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.297+0000 7f5d6fe32640 1 -- 172.21.15.125:0/3408182321 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d680ce240 con 0x7f5d680ca0e0 2023-10-30T13:07:20.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.297+0000 7f5d6fe32640 1 -- 172.21.15.125:0/3408182321 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d680ce400 con 0x7f5d680ca0e0 2023-10-30T13:07:20.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.297+0000 7f5d5e7fc640 1 -- 172.21.15.125:0/3408182321 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5d6400cca0 con 0x7f5d680ca0e0 2023-10-30T13:07:20.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.297+0000 7f5d5e7fc640 1 --2- 172.21.15.125:0/3408182321 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5d4c0664d0 0x7f5d4c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:20.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.297+0000 7f5d5e7fc640 1 -- 172.21.15.125:0/3408182321 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f5d64084720 con 0x7f5d680ca0e0 2023-10-30T13:07:20.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.297+0000 7f5d6d3cd640 1 --2- 172.21.15.125:0/3408182321 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5d4c0664d0 0x7f5d4c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:20.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.301+0000 7f5d6d3cd640 1 --2- 172.21.15.125:0/3408182321 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5d4c0664d0 0x7f5d4c068990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f5d680c47a0 tx=0x7f5d58008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:20.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.301+0000 7f5d5e7fc640 1 -- 172.21.15.125:0/3408182321 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5d640513d0 con 0x7f5d680ca0e0 2023-10-30T13:07:20.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.481+0000 7f5d6fe32640 1 -- 172.21.15.125:0/3408182321 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d680b2a50 con 0x7f5d680ca0e0 2023-10-30T13:07:20.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.485+0000 7f5d5e7fc640 1 -- 172.21.15.125:0/3408182321 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5753 (secure 0 0 0) 0x7f5d640549f0 con 0x7f5d680ca0e0 2023-10-30T13:07:20.488 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:20.488 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:15.695661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:rejoin","state_seq":428,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:20.489 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 175 2023-10-30T13:07:20.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.485+0000 7f5d6fe32640 1 -- 172.21.15.125:0/3408182321 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5d4c0664d0 msgr2=0x7f5d4c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:20.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.485+0000 7f5d6fe32640 1 --2- 172.21.15.125:0/3408182321 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5d4c0664d0 0x7f5d4c068990 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f5d680c47a0 tx=0x7f5d58008040 comp rx=0 tx=0).stop 2023-10-30T13:07:20.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.485+0000 7f5d6fe32640 1 -- 172.21.15.125:0/3408182321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d680ca0e0 msgr2=0x7f5d681f2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:20.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.485+0000 7f5d6fe32640 1 --2- 172.21.15.125:0/3408182321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d680ca0e0 0x7f5d681f2a60 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f5d6400e870 tx=0x7f5d6400ec70 comp rx=0 tx=0).stop 2023-10-30T13:07:20.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.485+0000 7f5d6fe32640 1 -- 172.21.15.125:0/3408182321 shutdown_connections 2023-10-30T13:07:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.485+0000 7f5d6fe32640 1 --2- 172.21.15.125:0/3408182321 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5d4c0664d0 0x7f5d4c068990 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.485+0000 7f5d6fe32640 1 --2- 172.21.15.125:0/3408182321 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d681538a0 0x7f5d680b3440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.485+0000 7f5d6fe32640 1 --2- 172.21.15.125:0/3408182321 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d680cf160 0x7f5d681f2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.485+0000 7f5d6fe32640 1 --2- 172.21.15.125:0/3408182321 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d680ca0e0 0x7f5d681f2a60 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:20.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.485+0000 7f5d6fe32640 1 -- 172.21.15.125:0/3408182321 >> 172.21.15.125:0/3408182321 conn(0x7f5d6806bfe0 msgr2=0x7f5d68065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.485+0000 7f5d6fe32640 1 -- 172.21.15.125:0/3408182321 shutdown_connections 2023-10-30T13:07:20.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:20.485+0000 7f5d6fe32640 1 -- 172.21.15.125:0/3408182321 wait complete. 2023-10-30T13:07:20.509 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 175, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 175, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_5234": { "addr": "172.21.15.136:6835/2194892271", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 2194892271, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 2194892271, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 5234, "incarnation": 172, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 428 }, "gid_6017": { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 170, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 251 }, "gid_8301": { "addr": "172.21.15.125:6837/2540063352", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2540063352, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2540063352, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8301, "incarnation": 168, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 } }, "last_failure": 0, "last_failure_osd_epoch": 75, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:07:15.695661+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5234, "mds_1": 8301, "mds_2": 6017 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:07:22.511 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:22.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.653+0000 7fbd729c4640 1 Processor -- start 2023-10-30T13:07:22.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.653+0000 7fbd729c4640 1 -- start start 2023-10-30T13:07:22.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.657+0000 7fbd729c4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbd6c06e470 0x7fbd6c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:22.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.657+0000 7fbd729c4640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6c06afa0 con 0x7fbd6c153d70 2023-10-30T13:07:22.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.657+0000 7fbd729c4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6c154150 con 0x7fbd6c0c8490 2023-10-30T13:07:22.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.657+0000 7fbd729c4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd6c06d780 con 0x7fbd6c06e470 2023-10-30T13:07:22.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.657+0000 7fbd6bfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fbd6c153d70 0x7fbd6c0cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37286/0 (socket says 172.21.15.125:37286) 2023-10-30T13:07:22.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.657+0000 7fbd6bfff640 1 -- 172.21.15.125:0/428875858 learned_addr learned my addr 172.21.15.125:0/428875858 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:07:22.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.657+0000 7fbd6b7fe640 1 --2- 172.21.15.125:0/428875858 >> v2:172.21.15.136:3300/0 conn(0x7fbd6c06e470 0x7fbd6c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:22.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.657+0000 7fbd6affd640 1 -- 172.21.15.125:0/428875858 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319690653 0 0) 0x7fbd6c06afa0 con 0x7fbd6c153d70 2023-10-30T13:07:22.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.657+0000 7fbd6affd640 1 -- 172.21.15.125:0/428875858 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd54003530 con 0x7fbd6c153d70 2023-10-30T13:07:22.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.657+0000 7fbd6b7fe640 1 -- 172.21.15.125:0/428875858 >> v1:172.21.15.136:6789/0 conn(0x7fbd6c0c8490 legacy=0x7fbd6c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:22.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.657+0000 7fbd6b7fe640 1 -- 172.21.15.125:0/428875858 >> v1:172.21.15.125:6789/0 conn(0x7fbd6c153d70 legacy=0x7fbd6c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:22.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.657+0000 7fbd6b7fe640 1 -- 172.21.15.125:0/428875858 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6c06e030 con 0x7fbd6c06e470 2023-10-30T13:07:22.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.661+0000 7fbd6b7fe640 1 --2- 172.21.15.125:0/428875858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd6c06e470 0x7fbd6c06ca80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fbd600097c0 tx=0x7fbd6001c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d478c5acb8141571 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:22.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.661+0000 7fbd6affd640 1 -- 172.21.15.125:0/428875858 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd600ab650 con 0x7fbd6c06e470 2023-10-30T13:07:22.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.661+0000 7fbd6affd640 1 -- 172.21.15.125:0/428875858 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd600ab7b0 con 0x7fbd6c06e470 2023-10-30T13:07:22.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.661+0000 7fbd6affd640 1 -- 172.21.15.125:0/428875858 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd600abb20 con 0x7fbd6c06e470 2023-10-30T13:07:22.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.661+0000 7fbd729c4640 1 -- 172.21.15.125:0/428875858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd6c06e470 msgr2=0x7fbd6c06ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:22.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.661+0000 7fbd729c4640 1 --2- 172.21.15.125:0/428875858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd6c06e470 0x7fbd6c06ca80 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fbd600097c0 tx=0x7fbd6001c450 comp rx=0 tx=0).stop 2023-10-30T13:07:22.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.661+0000 7fbd729c4640 1 -- 172.21.15.125:0/428875858 shutdown_connections 2023-10-30T13:07:22.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.661+0000 7fbd729c4640 1 --2- 172.21.15.125:0/428875858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd6c06e470 0x7fbd6c06ca80 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:22.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.661+0000 7fbd729c4640 1 -- 172.21.15.125:0/428875858 >> 172.21.15.125:0/428875858 conn(0x7fbd6c071ea0 msgr2=0x7fbd6c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:22.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.661+0000 7fbd729c4640 1 -- 172.21.15.125:0/428875858 shutdown_connections 2023-10-30T13:07:22.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.661+0000 7fbd729c4640 1 -- 172.21.15.125:0/428875858 wait complete. 2023-10-30T13:07:22.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd729c4640 1 Processor -- start 2023-10-30T13:07:22.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd729c4640 1 -- start start 2023-10-30T13:07:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd729c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbd6c06e470 0x7fbd6c138600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd729c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd6c0c8490 0x7fbd6c1f5e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd729c4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd6c153d70 0x7fbd6c0cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd729c4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c14efb0 con 0x7fbd6c153d70 2023-10-30T13:07:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd729c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c13d470 con 0x7fbd6c0c8490 2023-10-30T13:07:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd729c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6c138b40 con 0x7fbd6c06e470 2023-10-30T13:07:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd70f61640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd6c153d70 0x7fbd6c0cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd70f61640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd6c153d70 0x7fbd6c0cbdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38696/0 (socket says 172.21.15.125:38696) 2023-10-30T13:07:22.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd70f61640 1 -- 172.21.15.125:0/2812576735 learned_addr learned my addr 172.21.15.125:0/2812576735 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd6b7fe640 1 --2- 172.21.15.125:0/2812576735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd6c0c8490 0x7fbd6c1f5e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd6bfff640 1 --2- 172.21.15.125:0/2812576735 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbd6c06e470 0x7fbd6c138600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd70f61640 1 -- 172.21.15.125:0/2812576735 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbd6c06e470 msgr2=0x7fbd6c138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd70f61640 1 --2- 172.21.15.125:0/2812576735 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbd6c06e470 0x7fbd6c138600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd70f61640 1 -- 172.21.15.125:0/2812576735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd6c0c8490 msgr2=0x7fbd6c1f5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd70f61640 1 --2- 172.21.15.125:0/2812576735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd6c0c8490 0x7fbd6c1f5e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd70f61640 1 -- 172.21.15.125:0/2812576735 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd60009420 con 0x7fbd6c153d70 2023-10-30T13:07:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd70f61640 1 --2- 172.21.15.125:0/2812576735 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd6c153d70 0x7fbd6c0cbdf0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400c7a0 tx=0x7fbd6400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd68ff9640 1 -- 172.21.15.125:0/2812576735 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd64003b80 con 0x7fbd6c153d70 2023-10-30T13:07:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd68ff9640 1 -- 172.21.15.125:0/2812576735 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd64003ce0 con 0x7fbd6c153d70 2023-10-30T13:07:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd68ff9640 1 -- 172.21.15.125:0/2812576735 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6400aa90 con 0x7fbd6c153d70 2023-10-30T13:07:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd729c4640 1 -- 172.21.15.125:0/2812576735 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6c0ccb60 con 0x7fbd6c153d70 2023-10-30T13:07:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.665+0000 7fbd729c4640 1 -- 172.21.15.125:0/2812576735 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6c14ecd0 con 0x7fbd6c153d70 2023-10-30T13:07:22.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.669+0000 7fbd729c4640 1 -- 172.21.15.125:0/2812576735 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6c04e170 con 0x7fbd6c153d70 2023-10-30T13:07:22.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.669+0000 7fbd68ff9640 1 -- 172.21.15.125:0/2812576735 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbd6400abf0 con 0x7fbd6c153d70 2023-10-30T13:07:22.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.669+0000 7fbd68ff9640 1 --2- 172.21.15.125:0/2812576735 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbd580664a0 0x7fbd58068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:22.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.669+0000 7fbd68ff9640 1 -- 172.21.15.125:0/2812576735 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7fbd64084320 con 0x7fbd6c153d70 2023-10-30T13:07:22.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.669+0000 7fbd6bfff640 1 --2- 172.21.15.125:0/2812576735 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbd580664a0 0x7fbd58068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:22.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.669+0000 7fbd6bfff640 1 --2- 172.21.15.125:0/2812576735 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbd580664a0 0x7fbd58068960 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c0011f0 tx=0x7fbd5c007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.673+0000 7fbd68ff9640 1 -- 172.21.15.125:0/2812576735 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbd64051110 con 0x7fbd6c153d70 2023-10-30T13:07:22.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.857+0000 7fbd729c4640 1 -- 172.21.15.125:0/2812576735 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd6c13d5b0 con 0x7fbd6c153d70 2023-10-30T13:07:22.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.857+0000 7fbd68ff9640 1 -- 172.21.15.125:0/2812576735 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 175 v175) v1 ==== 95+0+5753 (secure 0 0 0) 0x7fbd64054730 con 0x7fbd6c153d70 2023-10-30T13:07:22.861 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:22.861 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":175,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":175,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:15.695661+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:rejoin","state_seq":428,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:22.863 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 175 2023-10-30T13:07:22.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.857+0000 7fbd729c4640 1 -- 172.21.15.125:0/2812576735 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbd580664a0 msgr2=0x7fbd58068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:22.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.857+0000 7fbd729c4640 1 --2- 172.21.15.125:0/2812576735 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbd580664a0 0x7fbd58068960 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c0011f0 tx=0x7fbd5c007d40 comp rx=0 tx=0).stop 2023-10-30T13:07:22.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.857+0000 7fbd729c4640 1 -- 172.21.15.125:0/2812576735 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd6c153d70 msgr2=0x7fbd6c0cbdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:22.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.857+0000 7fbd729c4640 1 --2- 172.21.15.125:0/2812576735 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd6c153d70 0x7fbd6c0cbdf0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7fbd6400c7a0 tx=0x7fbd6400cc60 comp rx=0 tx=0).stop 2023-10-30T13:07:22.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.857+0000 7fbd729c4640 1 -- 172.21.15.125:0/2812576735 shutdown_connections 2023-10-30T13:07:22.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.857+0000 7fbd729c4640 1 --2- 172.21.15.125:0/2812576735 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbd580664a0 0x7fbd58068960 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:22.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.857+0000 7fbd729c4640 1 --2- 172.21.15.125:0/2812576735 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbd6c153d70 0x7fbd6c0cbdf0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:22.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.857+0000 7fbd729c4640 1 --2- 172.21.15.125:0/2812576735 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbd6c0c8490 0x7fbd6c1f5e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:22.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.857+0000 7fbd729c4640 1 --2- 172.21.15.125:0/2812576735 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbd6c06e470 0x7fbd6c138600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:22.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.857+0000 7fbd729c4640 1 -- 172.21.15.125:0/2812576735 >> 172.21.15.125:0/2812576735 conn(0x7fbd6c071ea0 msgr2=0x7fbd6c152c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:22.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.857+0000 7fbd729c4640 1 -- 172.21.15.125:0/2812576735 shutdown_connections 2023-10-30T13:07:22.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:22.857+0000 7fbd729c4640 1 -- 172.21.15.125:0/2812576735 wait complete. 2023-10-30T13:07:22.882 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:07:23.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:07:23.213 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:07:24.883 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:25.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc242b65640 1 Processor -- start 2023-10-30T13:07:25.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc242b65640 1 -- start start 2023-10-30T13:07:25.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc242b65640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc23c153d70 0x7fc23c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:25.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc242b65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc23c06e470 0x7fc23c0cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:25.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc242b65640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc23c06afa0 con 0x7fc23c0c8490 2023-10-30T13:07:25.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc242b65640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc23c06d780 con 0x7fc23c153d70 2023-10-30T13:07:25.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc242b65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c154150 con 0x7fc23c06e470 2023-10-30T13:07:25.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc240901640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc23c153d70 0x7fc23c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:25.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc240901640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc23c153d70 0x7fc23c06ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38700/0 (socket says 172.21.15.125:38700) 2023-10-30T13:07:25.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc240901640 1 -- 172.21.15.125:0/2993826711 learned_addr learned my addr 172.21.15.125:0/2993826711 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:25.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc241102640 1 --1- 172.21.15.125:0/2993826711 >> v1:172.21.15.125:6789/0 conn(0x7fc23c0c8490 0x7fc23c0c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37296/0 (socket says 172.21.15.125:37296) 2023-10-30T13:07:25.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc233fff640 1 --2- 172.21.15.125:0/2993826711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc23c06e470 0x7fc23c0cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:25.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc240901640 1 -- 172.21.15.125:0/2993826711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc23c06e470 msgr2=0x7fc23c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:25.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc240901640 1 --2- 172.21.15.125:0/2993826711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc23c06e470 0x7fc23c0cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:25.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc240901640 1 -- 172.21.15.125:0/2993826711 >> v1:172.21.15.125:6789/0 conn(0x7fc23c0c8490 legacy=0x7fc23c0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:25.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc240901640 1 -- 172.21.15.125:0/2993826711 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc23c06e030 con 0x7fc23c153d70 2023-10-30T13:07:25.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.025+0000 7fc233fff640 1 --2- 172.21.15.125:0/2993826711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc23c06e470 0x7fc23c0cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:07:25.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.029+0000 7fc240901640 1 --2- 172.21.15.125:0/2993826711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc23c153d70 0x7fc23c06ca80 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fc238009870 tx=0x7fc23801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8376c116c524a665 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:25.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.029+0000 7fc2337fe640 1 -- 172.21.15.125:0/2993826711 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2380aa670 con 0x7fc23c153d70 2023-10-30T13:07:25.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.029+0000 7fc2337fe640 1 -- 172.21.15.125:0/2993826711 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2380aa7d0 con 0x7fc23c153d70 2023-10-30T13:07:25.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.029+0000 7fc2337fe640 1 -- 172.21.15.125:0/2993826711 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2380aab60 con 0x7fc23c153d70 2023-10-30T13:07:25.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 -- 172.21.15.125:0/2993826711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc23c153d70 msgr2=0x7fc23c06ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:25.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 --2- 172.21.15.125:0/2993826711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc23c153d70 0x7fc23c06ca80 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7fc238009870 tx=0x7fc23801c510 comp rx=0 tx=0).stop 2023-10-30T13:07:25.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 -- 172.21.15.125:0/2993826711 shutdown_connections 2023-10-30T13:07:25.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 --2- 172.21.15.125:0/2993826711 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc23c06e470 0x7fc23c0cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:25.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 --2- 172.21.15.125:0/2993826711 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc23c153d70 0x7fc23c06ca80 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:25.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 -- 172.21.15.125:0/2993826711 >> 172.21.15.125:0/2993826711 conn(0x7fc23c071ea0 msgr2=0x7fc23c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:25.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 -- 172.21.15.125:0/2993826711 shutdown_connections 2023-10-30T13:07:25.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 -- 172.21.15.125:0/2993826711 wait complete. 2023-10-30T13:07:25.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 Processor -- start 2023-10-30T13:07:25.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 -- start start 2023-10-30T13:07:25.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc23c06e470 0x7fc23c14e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:25.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c0c8490 0x7fc23c1556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:25.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc23c153d70 0x7fc23c145700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:25.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c1f3740 con 0x7fc23c06e470 2023-10-30T13:07:25.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c1f6e00 con 0x7fc23c0c8490 2023-10-30T13:07:25.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc242b65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc23c1f70c0 con 0x7fc23c153d70 2023-10-30T13:07:25.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.033+0000 7fc240901640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc23c06e470 0x7fc23c14e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:25.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc240901640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc23c06e470 0x7fc23c14e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38704/0 (socket says 172.21.15.125:38704) 2023-10-30T13:07:25.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc240901640 1 -- 172.21.15.125:0/2206796494 learned_addr learned my addr 172.21.15.125:0/2206796494 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:25.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc241102640 1 --2- 172.21.15.125:0/2206796494 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc23c153d70 0x7fc23c145700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:25.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc233fff640 1 --2- 172.21.15.125:0/2206796494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c0c8490 0x7fc23c1556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:25.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc240901640 1 -- 172.21.15.125:0/2206796494 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc23c153d70 msgr2=0x7fc23c145700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:25.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc240901640 1 --2- 172.21.15.125:0/2206796494 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc23c153d70 0x7fc23c145700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:25.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc240901640 1 -- 172.21.15.125:0/2206796494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c0c8490 msgr2=0x7fc23c1556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:25.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc240901640 1 --2- 172.21.15.125:0/2206796494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c0c8490 0x7fc23c1556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:25.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc240901640 1 -- 172.21.15.125:0/2206796494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2380094d0 con 0x7fc23c06e470 2023-10-30T13:07:25.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc240901640 1 --2- 172.21.15.125:0/2206796494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc23c06e470 0x7fc23c14e8a0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fc2380ac040 tx=0x7fc23801ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:25.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc2317fa640 1 -- 172.21.15.125:0/2206796494 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2380b3800 con 0x7fc23c06e470 2023-10-30T13:07:25.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc2317fa640 1 -- 172.21.15.125:0/2206796494 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2380b3980 con 0x7fc23c06e470 2023-10-30T13:07:25.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc2317fa640 1 -- 172.21.15.125:0/2206796494 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2380b3d30 con 0x7fc23c06e470 2023-10-30T13:07:25.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc242b65640 1 -- 172.21.15.125:0/2206796494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc23c1f2e10 con 0x7fc23c06e470 2023-10-30T13:07:25.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc242b65640 1 -- 172.21.15.125:0/2206796494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc23c1f33d0 con 0x7fc23c06e470 2023-10-30T13:07:25.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc242b65640 1 -- 172.21.15.125:0/2206796494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc23c04e170 con 0x7fc23c06e470 2023-10-30T13:07:25.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.037+0000 7fc2317fa640 1 -- 172.21.15.125:0/2206796494 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc2380c73a0 con 0x7fc23c06e470 2023-10-30T13:07:25.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.041+0000 7fc2317fa640 1 --2- 172.21.15.125:0/2206796494 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2280664d0 0x7fc228068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:25.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.041+0000 7fc2317fa640 1 -- 172.21.15.125:0/2206796494 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7fc23811f240 con 0x7fc23c06e470 2023-10-30T13:07:25.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.041+0000 7fc233fff640 1 --2- 172.21.15.125:0/2206796494 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2280664d0 0x7fc228068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:25.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.041+0000 7fc233fff640 1 --2- 172.21.15.125:0/2206796494 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2280664d0 0x7fc228068990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc23c14f1c0 tx=0x7fc22c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:25.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.045+0000 7fc2317fa640 1 -- 172.21.15.125:0/2206796494 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc2380ec030 con 0x7fc23c06e470 2023-10-30T13:07:25.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.221+0000 7fc242b65640 1 -- 172.21.15.125:0/2206796494 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc23c14de00 con 0x7fc23c06e470 2023-10-30T13:07:25.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.225+0000 7fc2317fa640 1 -- 172.21.15.125:0/2206796494 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+5759 (secure 0 0 0) 0x7fc2380c3090 con 0x7fc23c06e470 2023-10-30T13:07:25.228 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:25.229 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:22.748964+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:clientreplay","state_seq":430,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:25.229 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 176 2023-10-30T13:07:25.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.225+0000 7fc242b65640 1 -- 172.21.15.125:0/2206796494 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2280664d0 msgr2=0x7fc228068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:25.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.225+0000 7fc242b65640 1 --2- 172.21.15.125:0/2206796494 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2280664d0 0x7fc228068990 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc23c14f1c0 tx=0x7fc22c008040 comp rx=0 tx=0).stop 2023-10-30T13:07:25.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.225+0000 7fc242b65640 1 -- 172.21.15.125:0/2206796494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc23c06e470 msgr2=0x7fc23c14e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:25.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.225+0000 7fc242b65640 1 --2- 172.21.15.125:0/2206796494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc23c06e470 0x7fc23c14e8a0 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7fc2380ac040 tx=0x7fc23801ad50 comp rx=0 tx=0).stop 2023-10-30T13:07:25.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.225+0000 7fc242b65640 1 -- 172.21.15.125:0/2206796494 shutdown_connections 2023-10-30T13:07:25.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.225+0000 7fc242b65640 1 --2- 172.21.15.125:0/2206796494 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2280664d0 0x7fc228068990 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:25.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.225+0000 7fc242b65640 1 --2- 172.21.15.125:0/2206796494 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc23c153d70 0x7fc23c145700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:25.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.225+0000 7fc242b65640 1 --2- 172.21.15.125:0/2206796494 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc23c0c8490 0x7fc23c1556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:25.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.225+0000 7fc242b65640 1 --2- 172.21.15.125:0/2206796494 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc23c06e470 0x7fc23c14e8a0 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:25.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.225+0000 7fc242b65640 1 -- 172.21.15.125:0/2206796494 >> 172.21.15.125:0/2206796494 conn(0x7fc23c071ea0 msgr2=0x7fc23c153020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:25.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.225+0000 7fc242b65640 1 -- 172.21.15.125:0/2206796494 shutdown_connections 2023-10-30T13:07:25.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:25.225+0000 7fc242b65640 1 -- 172.21.15.125:0/2206796494 wait complete. 2023-10-30T13:07:25.249 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:07:27.250 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:27.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.389+0000 7effe2e46640 1 Processor -- start 2023-10-30T13:07:27.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe2e46640 1 -- start start 2023-10-30T13:07:27.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe2e46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7effdc0c05d0 0x7effdc13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:27.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe2e46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effdc0c0a50 con 0x7effdc0c6b80 2023-10-30T13:07:27.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe2e46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effdc06f6e0 con 0x7effdc0c6f60 2023-10-30T13:07:27.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe2e46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effdc06f9c0 con 0x7effdc0c05d0 2023-10-30T13:07:27.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe0be2640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7effdc0c6b80 0x7effdc0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37300/0 (socket says 172.21.15.125:37300) 2023-10-30T13:07:27.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe0be2640 1 -- 172.21.15.125:0/1885147219 learned_addr learned my addr 172.21.15.125:0/1885147219 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:07:27.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdbfff640 1 --2- 172.21.15.125:0/1885147219 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7effdc0c05d0 0x7effdc13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:27.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102148278 0 0) 0x7effdc0c0a50 con 0x7effdc0c6b80 2023-10-30T13:07:27.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effcc005740 con 0x7effdc0c6b80 2023-10-30T13:07:27.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855758480 0 0) 0x7effdc06f6e0 con 0x7effdc0c6f60 2023-10-30T13:07:27.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effdc0c0a50 con 0x7effdc0c6f60 2023-10-30T13:07:27.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2467424949 0 0) 0x7effcc005740 con 0x7effdc0c6b80 2023-10-30T13:07:27.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7effdc0c05d0 msgr2=0x7effdc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:27.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 --2- 172.21.15.125:0/1885147219 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7effdc0c05d0 0x7effdc13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:27.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 >> v1:172.21.15.136:6789/0 conn(0x7effdc0c6f60 legacy=0x7effdc1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:27.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc06fca0 con 0x7effdc0c6b80 2023-10-30T13:07:27.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7effd0001680 con 0x7effdc0c6b80 2023-10-30T13:07:27.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 >> v1:172.21.15.125:6789/0 conn(0x7effdc0c6b80 legacy=0x7effdc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:27.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 --2- 172.21.15.125:0/1885147219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effcc005d50 0x7effcc006150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:27.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 --2- 172.21.15.125:0/1885147219 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7effdc0c05d0 0x7effcc0069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:27.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 --2- 172.21.15.125:0/1885147219 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effcc006f20 0x7effcc0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:27.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc06f9c0 con 0x7effcc006f20 2023-10-30T13:07:27.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effcc00dc50 con 0x7effcc005d50 2023-10-30T13:07:27.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effcc00de10 con 0x7effdc0c05d0 2023-10-30T13:07:27.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe13e3640 1 --2- 172.21.15.125:0/1885147219 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effcc006f20 0x7effcc0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:27.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe0be2640 1 --2- 172.21.15.125:0/1885147219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effcc005d50 0x7effcc006150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:27.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effdbfff640 1 --2- 172.21.15.125:0/1885147219 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7effdc0c05d0 0x7effcc0069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:27.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe13e3640 1 -- 172.21.15.125:0/1885147219 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7effdc0c05d0 msgr2=0x7effcc0069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:27.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe13e3640 1 --2- 172.21.15.125:0/1885147219 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7effdc0c05d0 0x7effcc0069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:27.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe13e3640 1 -- 172.21.15.125:0/1885147219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effcc005d50 msgr2=0x7effcc006150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:27.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe13e3640 1 --2- 172.21.15.125:0/1885147219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effcc005d50 0x7effcc006150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:27.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe13e3640 1 -- 172.21.15.125:0/1885147219 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effcc005740 con 0x7effcc006f20 2023-10-30T13:07:27.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.393+0000 7effe0be2640 1 --2- 172.21.15.125:0/1885147219 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effcc005d50 0x7effcc006150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:07:27.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.397+0000 7effe13e3640 1 --2- 172.21.15.125:0/1885147219 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effcc006f20 0x7effcc0073a0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7effd4001fd0 tx=0x7effd401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c52cd0f8ef9c27 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:27.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.397+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4003b80 con 0x7effcc006f20 2023-10-30T13:07:27.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.397+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effd4003d00 con 0x7effcc006f20 2023-10-30T13:07:27.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.397+0000 7effdb7fe640 1 -- 172.21.15.125:0/1885147219 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4005310 con 0x7effcc006f20 2023-10-30T13:07:27.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.397+0000 7effe2e46640 1 -- 172.21.15.125:0/1885147219 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effcc006f20 msgr2=0x7effcc0073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:27.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.397+0000 7effe2e46640 1 --2- 172.21.15.125:0/1885147219 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effcc006f20 0x7effcc0073a0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7effd4001fd0 tx=0x7effd401c8e0 comp rx=0 tx=0).stop 2023-10-30T13:07:27.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.397+0000 7effe13e3640 1 -- 172.21.15.125:0/1885147219 reap_dead start 2023-10-30T13:07:27.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.397+0000 7effe2e46640 1 -- 172.21.15.125:0/1885147219 shutdown_connections 2023-10-30T13:07:27.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.397+0000 7effe2e46640 1 -- 172.21.15.125:0/1885147219 >> 172.21.15.125:0/1885147219 conn(0x7effdc0c7910 msgr2=0x7effdc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:27.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.397+0000 7effe2e46640 1 -- 172.21.15.125:0/1885147219 shutdown_connections 2023-10-30T13:07:27.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.397+0000 7effe2e46640 1 -- 172.21.15.125:0/1885147219 wait complete. 2023-10-30T13:07:27.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effe2e46640 1 Processor -- start 2023-10-30T13:07:27.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effe2e46640 1 -- start start 2023-10-30T13:07:27.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effe2e46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effcc006f20 0x7effdc0ae2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:27.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effe2e46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effdc0b0360 0x7effdc152ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:27.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effe2e46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7effdc0b0780 0x7effdc13d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:27.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effe2e46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc0c0440 con 0x7effdc0b0360 2023-10-30T13:07:27.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effe2e46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc0aff40 con 0x7effcc006f20 2023-10-30T13:07:27.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effe2e46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effdc0afbd0 con 0x7effdc0b0780 2023-10-30T13:07:27.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effdbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effdc0b0360 0x7effdc152ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:27.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effdbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effdc0b0360 0x7effdc152ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38722/0 (socket says 172.21.15.125:38722) 2023-10-30T13:07:27.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effdbfff640 1 -- 172.21.15.125:0/4240628833 learned_addr learned my addr 172.21.15.125:0/4240628833 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:27.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effe0be2640 1 --2- 172.21.15.125:0/4240628833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effcc006f20 0x7effdc0ae2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:27.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effe13e3640 1 --2- 172.21.15.125:0/4240628833 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7effdc0b0780 0x7effdc13d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:27.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effdbfff640 1 -- 172.21.15.125:0/4240628833 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7effdc0b0780 msgr2=0x7effdc13d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:27.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effdbfff640 1 --2- 172.21.15.125:0/4240628833 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7effdc0b0780 0x7effdc13d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:27.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effdbfff640 1 -- 172.21.15.125:0/4240628833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effcc006f20 msgr2=0x7effdc0ae2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:27.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effdbfff640 1 --2- 172.21.15.125:0/4240628833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effcc006f20 0x7effdc0ae2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:27.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effdbfff640 1 -- 172.21.15.125:0/4240628833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc0c0160 con 0x7effdc0b0360 2023-10-30T13:07:27.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effdbfff640 1 --2- 172.21.15.125:0/4240628833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effdc0b0360 0x7effdc152ed0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7effc80115e0 tx=0x7effc8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:27.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effd97fa640 1 -- 172.21.15.125:0/4240628833 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc800f8f0 con 0x7effdc0b0360 2023-10-30T13:07:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effd97fa640 1 -- 172.21.15.125:0/4240628833 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc800fa50 con 0x7effdc0b0360 2023-10-30T13:07:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effe2e46640 1 -- 172.21.15.125:0/4240628833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effdc154640 con 0x7effdc0b0360 2023-10-30T13:07:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effd97fa640 1 -- 172.21.15.125:0/4240628833 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc800fdc0 con 0x7effdc0b0360 2023-10-30T13:07:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.401+0000 7effe2e46640 1 -- 172.21.15.125:0/4240628833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effdc153e00 con 0x7effdc0b0360 2023-10-30T13:07:27.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.405+0000 7effe2e46640 1 -- 172.21.15.125:0/4240628833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effdc0b01b0 con 0x7effdc0b0360 2023-10-30T13:07:27.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.405+0000 7effd97fa640 1 -- 172.21.15.125:0/4240628833 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7effc8015460 con 0x7effdc0b0360 2023-10-30T13:07:27.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.405+0000 7effd97fa640 1 --2- 172.21.15.125:0/4240628833 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7effc00664a0 0x7effc0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:27.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.405+0000 7effd97fa640 1 -- 172.21.15.125:0/4240628833 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7effc8084e80 con 0x7effdc0b0360 2023-10-30T13:07:27.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.405+0000 7effe0be2640 1 --2- 172.21.15.125:0/4240628833 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7effc00664a0 0x7effc0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:27.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.405+0000 7effe0be2640 1 --2- 172.21.15.125:0/4240628833 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7effc00664a0 0x7effc0068960 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7effd0000c10 tx=0x7effd0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:27.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.409+0000 7effd97fa640 1 -- 172.21.15.125:0/4240628833 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7effc8051bc0 con 0x7effdc0b0360 2023-10-30T13:07:27.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.589+0000 7effe2e46640 1 -- 172.21.15.125:0/4240628833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effdc06ff50 con 0x7effdc0b0360 2023-10-30T13:07:27.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.589+0000 7effd97fa640 1 -- 172.21.15.125:0/4240628833 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+5759 (secure 0 0 0) 0x7effc80551e0 con 0x7effdc0b0360 2023-10-30T13:07:27.595 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:27.596 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:22.748964+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:clientreplay","state_seq":430,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:27.596 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 176 2023-10-30T13:07:27.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.593+0000 7effe2e46640 1 -- 172.21.15.125:0/4240628833 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7effc00664a0 msgr2=0x7effc0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:27.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.593+0000 7effe2e46640 1 --2- 172.21.15.125:0/4240628833 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7effc00664a0 0x7effc0068960 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7effd0000c10 tx=0x7effd0009040 comp rx=0 tx=0).stop 2023-10-30T13:07:27.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.593+0000 7effe2e46640 1 -- 172.21.15.125:0/4240628833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effdc0b0360 msgr2=0x7effdc152ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:27.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.593+0000 7effe2e46640 1 --2- 172.21.15.125:0/4240628833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effdc0b0360 0x7effdc152ed0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7effc80115e0 tx=0x7effc8011aa0 comp rx=0 tx=0).stop 2023-10-30T13:07:27.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.593+0000 7effe2e46640 1 -- 172.21.15.125:0/4240628833 shutdown_connections 2023-10-30T13:07:27.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.593+0000 7effe2e46640 1 --2- 172.21.15.125:0/4240628833 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7effc00664a0 0x7effc0068960 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:27.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.593+0000 7effe2e46640 1 --2- 172.21.15.125:0/4240628833 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7effdc0b0780 0x7effdc13d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:27.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.593+0000 7effe2e46640 1 --2- 172.21.15.125:0/4240628833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7effdc0b0360 0x7effdc152ed0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:27.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.593+0000 7effe2e46640 1 --2- 172.21.15.125:0/4240628833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7effcc006f20 0x7effdc0ae2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:27.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.593+0000 7effe2e46640 1 -- 172.21.15.125:0/4240628833 >> 172.21.15.125:0/4240628833 conn(0x7effdc0c7910 msgr2=0x7effdc06bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:27.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.593+0000 7effe2e46640 1 -- 172.21.15.125:0/4240628833 shutdown_connections 2023-10-30T13:07:27.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:27.593+0000 7effe2e46640 1 -- 172.21.15.125:0/4240628833 wait complete. 2023-10-30T13:07:27.616 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:07:29.617 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:29.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.776+0000 7fd4906d6640 1 Processor -- start 2023-10-30T13:07:29.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.776+0000 7fd4906d6640 1 -- start start 2023-10-30T13:07:29.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.776+0000 7fd4906d6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4880c6b80 0x7fd4880bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:29.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.776+0000 7fd4906d6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd48813f4a0 0x7fd4880c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:29.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.776+0000 7fd4906d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4880c0a50 con 0x7fd4880c6f60 2023-10-30T13:07:29.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.776+0000 7fd4906d6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd48806f6e0 con 0x7fd4880c6b80 2023-10-30T13:07:29.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.776+0000 7fd4906d6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd48806f9c0 con 0x7fd48813f4a0 2023-10-30T13:07:29.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.780+0000 7fd48e472640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd4880c6b80 0x7fd4880bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:29.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.780+0000 7fd48dc71640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd4880c6f60 0x7fd48813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:46446/0 (socket says 172.21.15.125:46446) 2023-10-30T13:07:29.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.780+0000 7fd48dc71640 1 -- 172.21.15.125:0/3055795416 learned_addr learned my addr 172.21.15.125:0/3055795416 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:07:29.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.780+0000 7fd48ec73640 1 --2- 172.21.15.125:0/3055795416 >> v2:172.21.15.136:3300/0 conn(0x7fd48813f4a0 0x7fd4880c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:29.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.780+0000 7fd48d470640 1 -- 172.21.15.125:0/3055795416 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840857909 0 0) 0x7fd4880c0a50 con 0x7fd4880c6f60 2023-10-30T13:07:29.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.780+0000 7fd48e472640 1 -- 172.21.15.125:0/3055795416 >> v2:172.21.15.136:3300/0 conn(0x7fd48813f4a0 msgr2=0x7fd4880c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:29.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.780+0000 7fd48e472640 1 --2- 172.21.15.125:0/3055795416 >> v2:172.21.15.136:3300/0 conn(0x7fd48813f4a0 0x7fd4880c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:29.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.780+0000 7fd48e472640 1 -- 172.21.15.125:0/3055795416 >> v1:172.21.15.136:6789/0 conn(0x7fd4880c6f60 legacy=0x7fd48813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:29.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.780+0000 7fd48e472640 1 -- 172.21.15.125:0/3055795416 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48806fca0 con 0x7fd4880c6b80 2023-10-30T13:07:29.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd48e472640 1 --2- 172.21.15.125:0/3055795416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4880c6b80 0x7fd4880bfec0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fd4880c8980 tx=0x7fd48401c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af449b5a4556f8a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:29.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd48d470640 1 -- 172.21.15.125:0/3055795416 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4840ab600 con 0x7fd4880c6b80 2023-10-30T13:07:29.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd48d470640 1 -- 172.21.15.125:0/3055795416 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4840a7050 con 0x7fd4880c6b80 2023-10-30T13:07:29.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd48d470640 1 -- 172.21.15.125:0/3055795416 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4840ab980 con 0x7fd4880c6b80 2023-10-30T13:07:29.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd4906d6640 1 -- 172.21.15.125:0/3055795416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4880c6b80 msgr2=0x7fd4880bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:29.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd4906d6640 1 --2- 172.21.15.125:0/3055795416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4880c6b80 0x7fd4880bfec0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fd4880c8980 tx=0x7fd48401c460 comp rx=0 tx=0).stop 2023-10-30T13:07:29.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd4906d6640 1 -- 172.21.15.125:0/3055795416 shutdown_connections 2023-10-30T13:07:29.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd4906d6640 1 --2- 172.21.15.125:0/3055795416 >> v2:172.21.15.136:3300/0 conn(0x7fd48813f4a0 0x7fd4880c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:29.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd4906d6640 1 --2- 172.21.15.125:0/3055795416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4880c6b80 0x7fd4880bfec0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:29.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd4906d6640 1 -- 172.21.15.125:0/3055795416 >> 172.21.15.125:0/3055795416 conn(0x7fd4880c7910 msgr2=0x7fd4880c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:29.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd4906d6640 1 -- 172.21.15.125:0/3055795416 shutdown_connections 2023-10-30T13:07:29.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd4906d6640 1 -- 172.21.15.125:0/3055795416 wait complete. 2023-10-30T13:07:29.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd4906d6640 1 Processor -- start 2023-10-30T13:07:29.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd4906d6640 1 -- start start 2023-10-30T13:07:29.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.784+0000 7fd4906d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4880c6b80 0x7fd4881f5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:29.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd4906d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4880c6f60 0x7fd4880c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:29.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd4906d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd48813f4a0 0x7fd4880c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:29.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd4906d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd488163390 con 0x7fd4880c6f60 2023-10-30T13:07:29.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd4906d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48813abd0 con 0x7fd4880c6b80 2023-10-30T13:07:29.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd4906d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4881629f0 con 0x7fd48813f4a0 2023-10-30T13:07:29.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd48dc71640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4880c6f60 0x7fd4880c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:29.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd48e472640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4880c6b80 0x7fd4881f5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:29.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd48dc71640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4880c6f60 0x7fd4880c8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38734/0 (socket says 172.21.15.125:38734) 2023-10-30T13:07:29.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd48e472640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4880c6b80 0x7fd4881f5480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:59826/0 (socket says 172.21.15.125:59826) 2023-10-30T13:07:29.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd48dc71640 1 -- 172.21.15.125:0/2770921965 learned_addr learned my addr 172.21.15.125:0/2770921965 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:29.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd48ec73640 1 --2- 172.21.15.125:0/2770921965 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd48813f4a0 0x7fd4880c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:29.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd48dc71640 1 -- 172.21.15.125:0/2770921965 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd48813f4a0 msgr2=0x7fd4880c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:29.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd48dc71640 1 --2- 172.21.15.125:0/2770921965 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd48813f4a0 0x7fd4880c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:29.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd48dc71640 1 -- 172.21.15.125:0/2770921965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4880c6b80 msgr2=0x7fd4881f5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:29.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd48dc71640 1 --2- 172.21.15.125:0/2770921965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4880c6b80 0x7fd4881f5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:29.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd48dc71640 1 -- 172.21.15.125:0/2770921965 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd484009420 con 0x7fd4880c6f60 2023-10-30T13:07:29.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd48dc71640 1 --2- 172.21.15.125:0/2770921965 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4880c6f60 0x7fd4880c8d10 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fd47800c520 tx=0x7fd47800c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:29.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd47effd640 1 -- 172.21.15.125:0/2770921965 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4780074e0 con 0x7fd4880c6f60 2023-10-30T13:07:29.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd47effd640 1 -- 172.21.15.125:0/2770921965 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd478007640 con 0x7fd4880c6f60 2023-10-30T13:07:29.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd47effd640 1 -- 172.21.15.125:0/2770921965 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4780049f0 con 0x7fd4880c6f60 2023-10-30T13:07:29.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd4906d6640 1 -- 172.21.15.125:0/2770921965 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd488162cd0 con 0x7fd4880c6f60 2023-10-30T13:07:29.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd4906d6640 1 -- 172.21.15.125:0/2770921965 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4881f5f90 con 0x7fd4880c6f60 2023-10-30T13:07:29.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd4906d6640 1 -- 172.21.15.125:0/2770921965 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4881634d0 con 0x7fd4880c6f60 2023-10-30T13:07:29.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.788+0000 7fd47effd640 1 -- 172.21.15.125:0/2770921965 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd478004b50 con 0x7fd4880c6f60 2023-10-30T13:07:29.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.792+0000 7fd47effd640 1 --2- 172.21.15.125:0/2770921965 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd46c0664d0 0x7fd46c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:29.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.792+0000 7fd47effd640 1 -- 172.21.15.125:0/2770921965 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7fd478083770 con 0x7fd4880c6f60 2023-10-30T13:07:29.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.792+0000 7fd48e472640 1 --2- 172.21.15.125:0/2770921965 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd46c0664d0 0x7fd46c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:29.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.792+0000 7fd48e472640 1 --2- 172.21.15.125:0/2770921965 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd46c0664d0 0x7fd46c068990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fd4840ad000 tx=0x7fd4840a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:29.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.796+0000 7fd47effd640 1 -- 172.21.15.125:0/2770921965 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd4780504d0 con 0x7fd4880c6f60 2023-10-30T13:07:29.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.984+0000 7fd4906d6640 1 -- 172.21.15.125:0/2770921965 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd48804e170 con 0x7fd4880c6f60 2023-10-30T13:07:29.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.984+0000 7fd47effd640 1 -- 172.21.15.125:0/2770921965 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+5759 (secure 0 0 0) 0x7fd478053af0 con 0x7fd4880c6f60 2023-10-30T13:07:29.993 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:29.993 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:22.748964+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:clientreplay","state_seq":430,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:29.994 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 176 2023-10-30T13:07:29.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.988+0000 7fd4906d6640 1 -- 172.21.15.125:0/2770921965 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd46c0664d0 msgr2=0x7fd46c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:29.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.988+0000 7fd4906d6640 1 --2- 172.21.15.125:0/2770921965 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd46c0664d0 0x7fd46c068990 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fd4840ad000 tx=0x7fd4840a6000 comp rx=0 tx=0).stop 2023-10-30T13:07:29.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.988+0000 7fd4906d6640 1 -- 172.21.15.125:0/2770921965 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4880c6f60 msgr2=0x7fd4880c8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:29.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.988+0000 7fd4906d6640 1 --2- 172.21.15.125:0/2770921965 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4880c6f60 0x7fd4880c8d10 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fd47800c520 tx=0x7fd47800c9e0 comp rx=0 tx=0).stop 2023-10-30T13:07:29.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.988+0000 7fd4906d6640 1 -- 172.21.15.125:0/2770921965 shutdown_connections 2023-10-30T13:07:29.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.988+0000 7fd4906d6640 1 --2- 172.21.15.125:0/2770921965 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd46c0664d0 0x7fd46c068990 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:29.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.988+0000 7fd4906d6640 1 --2- 172.21.15.125:0/2770921965 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd48813f4a0 0x7fd4880c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:29.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.988+0000 7fd4906d6640 1 --2- 172.21.15.125:0/2770921965 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd4880c6f60 0x7fd4880c8d10 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:29.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.988+0000 7fd4906d6640 1 --2- 172.21.15.125:0/2770921965 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd4880c6b80 0x7fd4881f5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:29.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.988+0000 7fd4906d6640 1 -- 172.21.15.125:0/2770921965 >> 172.21.15.125:0/2770921965 conn(0x7fd4880c7910 msgr2=0x7fd4880c15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:29.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.988+0000 7fd4906d6640 1 -- 172.21.15.125:0/2770921965 shutdown_connections 2023-10-30T13:07:29.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:29.988+0000 7fd4906d6640 1 -- 172.21.15.125:0/2770921965 wait complete. 2023-10-30T13:07:30.013 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:07:32.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:32.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.156+0000 7f64d6274640 1 Processor -- start 2023-10-30T13:07:32.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.156+0000 7f64d6274640 1 -- start start 2023-10-30T13:07:32.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.156+0000 7f64d6274640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f64d015cd50 0x7f64d00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:32.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.156+0000 7f64d6274640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d015a6c0 con 0x7f64d01538f0 2023-10-30T13:07:32.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.156+0000 7f64d6274640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64d015d300 con 0x7f64d00cefa0 2023-10-30T13:07:32.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.156+0000 7f64d6274640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64d015d5e0 con 0x7f64d015cd50 2023-10-30T13:07:32.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.156+0000 7f64d5272640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f64d01538f0 0x7f64d0159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55074/0 (socket says 172.21.15.125:55074) 2023-10-30T13:07:32.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.156+0000 7f64d5272640 1 -- 172.21.15.125:0/3852722828 learned_addr learned my addr 172.21.15.125:0/3852722828 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:07:32.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64d4a71640 1 --2- 172.21.15.125:0/3852722828 >> v2:172.21.15.136:3300/0 conn(0x7f64d015cd50 0x7f64d00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:32.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 -- 172.21.15.125:0/3852722828 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375475206 0 0) 0x7f64d015a6c0 con 0x7f64d01538f0 2023-10-30T13:07:32.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 -- 172.21.15.125:0/3852722828 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64bc003530 con 0x7f64d01538f0 2023-10-30T13:07:32.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 -- 172.21.15.125:0/3852722828 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4005773817 0 0) 0x7f64bc003530 con 0x7f64d01538f0 2023-10-30T13:07:32.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 -- 172.21.15.125:0/3852722828 >> v2:172.21.15.136:3300/0 conn(0x7f64d015cd50 msgr2=0x7f64d00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:32.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 --2- 172.21.15.125:0/3852722828 >> v2:172.21.15.136:3300/0 conn(0x7f64d015cd50 0x7f64d00c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:32.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 -- 172.21.15.125:0/3852722828 >> v1:172.21.15.136:6789/0 conn(0x7f64d00cefa0 legacy=0x7f64d01640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:32.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 -- 172.21.15.125:0/3852722828 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d015d8c0 con 0x7f64d01538f0 2023-10-30T13:07:32.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 -- 172.21.15.125:0/3852722828 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f64cc001680 con 0x7f64d01538f0 2023-10-30T13:07:32.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 -- 172.21.15.125:0/3852722828 >> v1:172.21.15.125:6789/0 conn(0x7f64d01538f0 legacy=0x7f64d0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:32.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 --2- 172.21.15.125:0/3852722828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64bc003980 0x7f64bc003d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:32.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 --2- 172.21.15.125:0/3852722828 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f64bc004380 0x7f64bc004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:32.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 --2- 172.21.15.125:0/3852722828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64bc005050 0x7f64bc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:32.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 -- 172.21.15.125:0/3852722828 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d015d5e0 con 0x7f64bc005050 2023-10-30T13:07:32.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 -- 172.21.15.125:0/3852722828 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc00bf00 con 0x7f64bc003980 2023-10-30T13:07:32.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64c7fff640 1 -- 172.21.15.125:0/3852722828 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc00c0c0 con 0x7f64bc004380 2023-10-30T13:07:32.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64d5272640 1 --2- 172.21.15.125:0/3852722828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64bc003980 0x7f64bc003d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:32.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64d5a73640 1 --2- 172.21.15.125:0/3852722828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64bc005050 0x7f64bc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:32.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64d4a71640 1 --2- 172.21.15.125:0/3852722828 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f64bc004380 0x7f64bc004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:32.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64d5a73640 1 -- 172.21.15.125:0/3852722828 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f64bc004380 msgr2=0x7f64bc004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:32.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64d5a73640 1 --2- 172.21.15.125:0/3852722828 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f64bc004380 0x7f64bc004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:32.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64d5a73640 1 -- 172.21.15.125:0/3852722828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64bc003980 msgr2=0x7f64bc003d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:32.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64d5a73640 1 --2- 172.21.15.125:0/3852722828 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64bc003980 0x7f64bc003d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:32.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.160+0000 7f64d5a73640 1 -- 172.21.15.125:0/3852722828 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc003530 con 0x7f64bc005050 2023-10-30T13:07:32.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.164+0000 7f64d5a73640 1 -- 172.21.15.125:0/3852722828 reap_dead start 2023-10-30T13:07:32.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.164+0000 7f64d5a73640 1 --2- 172.21.15.125:0/3852722828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64bc005050 0x7f64bc00b990 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f64c8002010 tx=0x7f64c801c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539b9392477dfc35 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:32.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.164+0000 7f64c7fff640 1 -- 172.21.15.125:0/3852722828 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c8003b80 con 0x7f64bc005050 2023-10-30T13:07:32.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.164+0000 7f64c7fff640 1 -- 172.21.15.125:0/3852722828 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c8003d00 con 0x7f64bc005050 2023-10-30T13:07:32.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.164+0000 7f64c7fff640 1 -- 172.21.15.125:0/3852722828 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c8005310 con 0x7f64bc005050 2023-10-30T13:07:32.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.164+0000 7f64d6274640 1 -- 172.21.15.125:0/3852722828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64bc005050 msgr2=0x7f64bc00b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:32.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.164+0000 7f64d6274640 1 --2- 172.21.15.125:0/3852722828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64bc005050 0x7f64bc00b990 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f64c8002010 tx=0x7f64c801c8c0 comp rx=0 tx=0).stop 2023-10-30T13:07:32.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.164+0000 7f64d6274640 1 -- 172.21.15.125:0/3852722828 shutdown_connections 2023-10-30T13:07:32.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.164+0000 7f64d6274640 1 --2- 172.21.15.125:0/3852722828 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64bc005050 0x7f64bc00b990 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:32.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.164+0000 7f64d6274640 1 -- 172.21.15.125:0/3852722828 >> 172.21.15.125:0/3852722828 conn(0x7f64d006a640 msgr2=0x7f64d0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:32.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.164+0000 7f64d6274640 1 -- 172.21.15.125:0/3852722828 shutdown_connections 2023-10-30T13:07:32.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.164+0000 7f64d6274640 1 -- 172.21.15.125:0/3852722828 wait complete. 2023-10-30T13:07:32.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.164+0000 7f64d6274640 1 Processor -- start 2023-10-30T13:07:32.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.164+0000 7f64d6274640 1 -- start start 2023-10-30T13:07:32.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d6274640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f64bc005050 0x7f64d014eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:32.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d6274640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64d014f780 0x7f64d0145ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:32.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d6274640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64d0148890 0x7f64d0148d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:32.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d6274640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d00b3e40 con 0x7f64d0148890 2023-10-30T13:07:32.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d6274640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d00b3530 con 0x7f64d014f780 2023-10-30T13:07:32.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d6274640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64d014f530 con 0x7f64bc005050 2023-10-30T13:07:32.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d5a73640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64d0148890 0x7f64d0148d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:32.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d5a73640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64d0148890 0x7f64d0148d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:47792/0 (socket says 172.21.15.125:47792) 2023-10-30T13:07:32.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d5a73640 1 -- 172.21.15.125:0/430839075 learned_addr learned my addr 172.21.15.125:0/430839075 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:32.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d5272640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f64bc005050 0x7f64d014eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:32.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d4a71640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64d014f780 0x7f64d0145ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:32.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d5a73640 1 -- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f64bc005050 msgr2=0x7f64d014eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:32.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d5a73640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f64bc005050 0x7f64d014eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:32.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d5a73640 1 -- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64d014f780 msgr2=0x7f64d0145ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:32.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d5a73640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64d014f780 0x7f64d0145ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:32.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d5a73640 1 -- 172.21.15.125:0/430839075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d00b46e0 con 0x7f64d0148890 2023-10-30T13:07:32.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d5272640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f64bc005050 0x7f64d014eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:07:32.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d5a73640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64d0148890 0x7f64d0148d40 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f64c801cda0 tx=0x7f64bc005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:32.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64c5ffb640 1 -- 172.21.15.125:0/430839075 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c8006d60 con 0x7f64d0148890 2023-10-30T13:07:32.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64c5ffb640 1 -- 172.21.15.125:0/430839075 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c8003b90 con 0x7f64d0148890 2023-10-30T13:07:32.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d6274640 1 -- 172.21.15.125:0/430839075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d0149440 con 0x7f64d0148890 2023-10-30T13:07:32.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64c5ffb640 1 -- 172.21.15.125:0/430839075 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64c80a57d0 con 0x7f64d0148890 2023-10-30T13:07:32.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d6274640 1 -- 172.21.15.125:0/430839075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d00b2f90 con 0x7f64d0148890 2023-10-30T13:07:32.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64d6274640 1 -- 172.21.15.125:0/430839075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d0159a30 con 0x7f64d0148890 2023-10-30T13:07:32.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.168+0000 7f64c5ffb640 1 -- 172.21.15.125:0/430839075 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f64c80a5930 con 0x7f64d0148890 2023-10-30T13:07:32.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.172+0000 7f64c5ffb640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64b40664d0 0x7f64b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:32.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.172+0000 7f64c5ffb640 1 -- 172.21.15.125:0/430839075 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f64c811d1e0 con 0x7f64d0148890 2023-10-30T13:07:32.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.172+0000 7f64d5272640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64b40664d0 0x7f64b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:32.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.172+0000 7f64d5272640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64b40664d0 0x7f64b4068990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f64cc000c10 tx=0x7f64cc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:32.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.176+0000 7f64c5ffb640 1 -- 172.21.15.125:0/430839075 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f64c80e9f30 con 0x7f64d0148890 2023-10-30T13:07:32.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.352+0000 7f64d6274640 1 -- 172.21.15.125:0/430839075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64d004e170 con 0x7f64d0148890 2023-10-30T13:07:32.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.356+0000 7f64c5ffb640 1 -- 172.21.15.125:0/430839075 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+5753 (secure 0 0 0) 0x7f64c80b5090 con 0x7f64d0148890 2023-10-30T13:07:32.360 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:32.360 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":177,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:29.953005+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:active","state_seq":432,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:32.360 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 177 2023-10-30T13:07:32.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.356+0000 7f64d6274640 1 -- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64b40664d0 msgr2=0x7f64b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:32.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.356+0000 7f64d6274640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64b40664d0 0x7f64b4068990 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f64cc000c10 tx=0x7f64cc009040 comp rx=0 tx=0).stop 2023-10-30T13:07:32.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.356+0000 7f64d6274640 1 -- 172.21.15.125:0/430839075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64d0148890 msgr2=0x7f64d0148d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:32.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.356+0000 7f64d6274640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64d0148890 0x7f64d0148d40 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f64c801cda0 tx=0x7f64bc005800 comp rx=0 tx=0).stop 2023-10-30T13:07:32.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.356+0000 7f64d6274640 1 -- 172.21.15.125:0/430839075 shutdown_connections 2023-10-30T13:07:32.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.356+0000 7f64d6274640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64b40664d0 0x7f64b4068990 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:32.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.356+0000 7f64d6274640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64d0148890 0x7f64d0148d40 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:32.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.356+0000 7f64d6274640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64d014f780 0x7f64d0145ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:32.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.356+0000 7f64d6274640 1 --2- 172.21.15.125:0/430839075 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f64bc005050 0x7f64d014eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:32.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.356+0000 7f64d6274640 1 -- 172.21.15.125:0/430839075 >> 172.21.15.125:0/430839075 conn(0x7f64d006a640 msgr2=0x7f64d00c50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:32.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.356+0000 7f64d6274640 1 -- 172.21.15.125:0/430839075 shutdown_connections 2023-10-30T13:07:32.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:32.356+0000 7f64d6274640 1 -- 172.21.15.125:0/430839075 wait complete. 2023-10-30T13:07:32.381 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=8178 2023-10-30T13:07:32.381 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.e 2023-10-30T13:07:40.333 INFO:tasks.ceph.mds.d.smithi136.stderr:2023-10-30T13:07:40.327+0000 7f5a56d93640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:07:40.333 INFO:tasks.ceph.mds.d.smithi136.stderr:2023-10-30T13:07:40.327+0000 7f5a56d93640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:07:40.333 INFO:tasks.ceph.mds.d.smithi136.stderr:2023-10-30T13:07:40.327+0000 7f5a56d93640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:07:41.106 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 23% (74/309) Updating files: 24% (75/309) Updating files: 25% (78/309) Updating files: 26% (81/309) Updating files: 27% (84/309) Updating files: 28% (87/309) Updating files: 29% (90/309) Updating files: 30% (93/309) Updating files: 31% (96/309) Updating files: 32% (99/309) Updating files: 33% (102/309) Updating files: 34% (106/309) Updating files: 35% (109/309) Updating files: 36% (112/309) Updating files: 37% (115/309) Updating files: 38% (118/309) Updating files: 39% (121/309) Updating files: 40% (124/309) Updating files: 41% (127/309) Updating files: 42% (130/309) Updating files: 43% (133/309) Updating files: 43% (134/309) Updating files: 44% (136/309) Updating files: 45% (140/309) Updating files: 46% (143/309) Updating files: 47% (146/309) Updating files: 48% (149/309) Updating files: 49% (152/309) Updating files: 50% (155/309) Updating files: 51% (158/309) Updating files: 52% (161/309) Updating files: 53% (164/309) Updating files: 54% (167/309) Updating files: 55% (170/309) Updating files: 56% (174/309) Updating files: 57% (177/309) Updating files: 58% (180/309) Updating files: 59% (183/309) Updating files: 60% (186/309) Updating files: 61% (189/309) Updating files: 62% (192/309) Updating files: 63% (195/309) Updating files: 64% (198/309) Updating files: 65% (201/309) Updating files: 66% (204/309) Updating files: 66% (206/309) Updating files: 67% (208/309) Updating files: 68% (211/309) Updating files: 69% (214/309) Updating files: 70% (217/309) Updating files: 71% (220/309) Updating files: 72% (223/309) Updating files: 73% (226/309) Updating files: 74% (229/309) Updating files: 75% (232/309) Updating files: 76% (235/309) Updating files: 77% (238/309) Updating files: 78% (242/309) Updating files: 78% (244/309) Updating files: 79% (245/309) Updating files: 80% (248/309) Updating files: 81% (251/309) Updating files: 82% (254/309) Updating files: 83% (257/309) Updating files: 84% (260/309) Updating files: 85% (263/309) Updating files: 86% (266/309) Updating files: 87% (269/309) Updating files: 88% (272/309) Updating files: 89% (276/309) Updating files: 90% (279/309) Updating files: 91% (282/309) Updating files: 92% (285/309) Updating files: 93% (288/309) Updating files: 94% (291/309) Updating files: 95% (294/309) Updating files: 96% (297/309) Updating files: 97% (300/309) Updating files: 98% (303/309) Updating files: 99% (306/309) Updating files: 100% (309/309) Updating files: 100% (309/309), done. 2023-10-30T13:07:41.382 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T13:07:41.383 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T13:07:41.383 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T13:07:41.387 INFO:tasks.ceph.mds.e:Started 2023-10-30T13:07:41.387 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:41.472 INFO:tasks.ceph.mds.e.smithi125.stdout:starting mds.e at 2023-10-30T13:07:41.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.520+0000 7f390d565640 1 Processor -- start 2023-10-30T13:07:41.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.520+0000 7f390d565640 1 -- start start 2023-10-30T13:07:41.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f390d565640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39081538a0 0x7f390815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:41.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f390d565640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39080cb800 con 0x7f39080cf160 2023-10-30T13:07:41.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f390d565640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f390815dc30 con 0x7f39080ca0e0 2023-10-30T13:07:41.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f390d565640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39080cb330 con 0x7f39081538a0 2023-10-30T13:07:41.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39077fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f39080cf160 0x7f3908164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41216/0 (socket says 172.21.15.125:41216) 2023-10-30T13:07:41.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39077fe640 1 -- 172.21.15.125:0/1570695093 learned_addr learned my addr 172.21.15.125:0/1570695093 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:07:41.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f3906ffd640 1 --2- 172.21.15.125:0/1570695093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39081538a0 0x7f390815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:41.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 -- 172.21.15.125:0/1570695093 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392963099 0 0) 0x7f39080cb800 con 0x7f39080cf160 2023-10-30T13:07:41.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 -- 172.21.15.125:0/1570695093 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f4003530 con 0x7f39080cf160 2023-10-30T13:07:41.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 -- 172.21.15.125:0/1570695093 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 615471606 0 0) 0x7f38f4003530 con 0x7f39080cf160 2023-10-30T13:07:41.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 -- 172.21.15.125:0/1570695093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39081538a0 msgr2=0x7f390815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:41.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 --2- 172.21.15.125:0/1570695093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39081538a0 0x7f390815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:41.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 -- 172.21.15.125:0/1570695093 >> v1:172.21.15.136:6789/0 conn(0x7f39080ca0e0 legacy=0x7f390815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:41.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 -- 172.21.15.125:0/1570695093 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39080cad70 con 0x7f39080cf160 2023-10-30T13:07:41.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 -- 172.21.15.125:0/1570695093 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f3900001a00 con 0x7f39080cf160 2023-10-30T13:07:41.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 -- 172.21.15.125:0/1570695093 >> v1:172.21.15.125:6789/0 conn(0x7f39080cf160 legacy=0x7f3908164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:41.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 --2- 172.21.15.125:0/1570695093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38f40039b0 0x7f38f4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:41.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 --2- 172.21.15.125:0/1570695093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39081538a0 0x7f38f40047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:41.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 --2- 172.21.15.125:0/1570695093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38f4004d10 0x7f38f4005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:41.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 -- 172.21.15.125:0/1570695093 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39080cb330 con 0x7f38f4004d10 2023-10-30T13:07:41.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 -- 172.21.15.125:0/1570695093 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38f400b990 con 0x7f38f40039b0 2023-10-30T13:07:41.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39067fc640 1 -- 172.21.15.125:0/1570695093 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38f400bb50 con 0x7f39081538a0 2023-10-30T13:07:41.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39077fe640 1 --2- 172.21.15.125:0/1570695093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38f4004d10 0x7f38f4005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:41.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f3906ffd640 1 --2- 172.21.15.125:0/1570695093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38f40039b0 0x7f38f4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:41.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f38fe7fc640 1 --2- 172.21.15.125:0/1570695093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39081538a0 0x7f38f40047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:41.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39077fe640 1 -- 172.21.15.125:0/1570695093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39081538a0 msgr2=0x7f38f40047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:41.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39077fe640 1 --2- 172.21.15.125:0/1570695093 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39081538a0 0x7f38f40047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:41.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39077fe640 1 -- 172.21.15.125:0/1570695093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38f40039b0 msgr2=0x7f38f4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:41.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39077fe640 1 --2- 172.21.15.125:0/1570695093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38f40039b0 0x7f38f4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:41.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f39077fe640 1 -- 172.21.15.125:0/1570695093 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f4003530 con 0x7f38f4004d10 2023-10-30T13:07:41.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.524+0000 7f3906ffd640 1 --2- 172.21.15.125:0/1570695093 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38f40039b0 0x7f38f4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:07:41.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.528+0000 7f39077fe640 1 --2- 172.21.15.125:0/1570695093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38f4004d10 0x7f38f4005190 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f390000a840 tx=0x7f390001d320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44a3b1211af6638b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:41.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.528+0000 7f39067fc640 1 -- 172.21.15.125:0/1570695093 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39000ad630 con 0x7f38f4004d10 2023-10-30T13:07:41.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.528+0000 7f39067fc640 1 -- 172.21.15.125:0/1570695093 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39000a8070 con 0x7f38f4004d10 2023-10-30T13:07:41.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.528+0000 7f39067fc640 1 -- 172.21.15.125:0/1570695093 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39000ad9d0 con 0x7f38f4004d10 2023-10-30T13:07:41.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.528+0000 7f390d565640 1 -- 172.21.15.125:0/1570695093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38f4004d10 msgr2=0x7f38f4005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:41.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.528+0000 7f390d565640 1 --2- 172.21.15.125:0/1570695093 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38f4004d10 0x7f38f4005190 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f390000a840 tx=0x7f390001d320 comp rx=0 tx=0).stop 2023-10-30T13:07:41.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.528+0000 7f39077fe640 1 -- 172.21.15.125:0/1570695093 reap_dead start 2023-10-30T13:07:41.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.528+0000 7f390d565640 1 -- 172.21.15.125:0/1570695093 shutdown_connections 2023-10-30T13:07:41.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.528+0000 7f390d565640 1 -- 172.21.15.125:0/1570695093 >> 172.21.15.125:0/1570695093 conn(0x7f390806bfe0 msgr2=0x7f390806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:41.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.528+0000 7f390d565640 1 -- 172.21.15.125:0/1570695093 shutdown_connections 2023-10-30T13:07:41.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.528+0000 7f390d565640 1 -- 172.21.15.125:0/1570695093 wait complete. 2023-10-30T13:07:41.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f390d565640 1 Processor -- start 2023-10-30T13:07:41.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f390d565640 1 -- start start 2023-10-30T13:07:41.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f390d565640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38f4004d10 0x7f390815d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:41.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f390d565640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39080b1960 0x7f390815baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:41.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f390d565640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f390815c030 0x7f3908142c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:41.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f390d565640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3908144880 con 0x7f390815c030 2023-10-30T13:07:41.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f390d565640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f390815aa50 con 0x7f38f4004d10 2023-10-30T13:07:41.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f390d565640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f390815ad30 con 0x7f39080b1960 2023-10-30T13:07:41.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f39077fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f390815c030 0x7f3908142c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:41.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f39077fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f390815c030 0x7f3908142c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34564/0 (socket says 172.21.15.125:34564) 2023-10-30T13:07:41.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f39077fe640 1 -- 172.21.15.125:0/2658473450 learned_addr learned my addr 172.21.15.125:0/2658473450 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:41.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f3906ffd640 1 --2- 172.21.15.125:0/2658473450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38f4004d10 0x7f390815d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:41.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f38fe7fc640 1 --2- 172.21.15.125:0/2658473450 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39080b1960 0x7f390815baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:41.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f39077fe640 1 -- 172.21.15.125:0/2658473450 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39080b1960 msgr2=0x7f390815baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:41.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f39077fe640 1 --2- 172.21.15.125:0/2658473450 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39080b1960 0x7f390815baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:41.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f39077fe640 1 -- 172.21.15.125:0/2658473450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38f4004d10 msgr2=0x7f390815d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:41.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f39077fe640 1 --2- 172.21.15.125:0/2658473450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38f4004d10 0x7f390815d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:41.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f39077fe640 1 -- 172.21.15.125:0/2658473450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f390815b010 con 0x7f390815c030 2023-10-30T13:07:41.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f39077fe640 1 --2- 172.21.15.125:0/2658473450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f390815c030 0x7f3908142c60 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f390001d150 tx=0x7f38f4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:41.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f38fffff640 1 -- 172.21.15.125:0/2658473450 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3900004d70 con 0x7f390815c030 2023-10-30T13:07:41.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f38fffff640 1 -- 172.21.15.125:0/2658473450 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39000017c0 con 0x7f390815c030 2023-10-30T13:07:41.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f38fffff640 1 -- 172.21.15.125:0/2658473450 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39000ad430 con 0x7f390815c030 2023-10-30T13:07:41.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f390d565640 1 -- 172.21.15.125:0/2658473450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f390815b2f0 con 0x7f390815c030 2023-10-30T13:07:41.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.532+0000 7f390d565640 1 -- 172.21.15.125:0/2658473450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39081431a0 con 0x7f390815c030 2023-10-30T13:07:41.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.536+0000 7f390d565640 1 -- 172.21.15.125:0/2658473450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3908159830 con 0x7f390815c030 2023-10-30T13:07:41.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.536+0000 7f38fffff640 1 -- 172.21.15.125:0/2658473450 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f39000aa030 con 0x7f390815c030 2023-10-30T13:07:41.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.536+0000 7f38fffff640 1 --2- 172.21.15.125:0/2658473450 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38e80664d0 0x7f38e8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:41.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.536+0000 7f38fffff640 1 -- 172.21.15.125:0/2658473450 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f390011eda0 con 0x7f390815c030 2023-10-30T13:07:41.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.536+0000 7f3906ffd640 1 --2- 172.21.15.125:0/2658473450 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38e80664d0 0x7f38e8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:41.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.536+0000 7f3906ffd640 1 --2- 172.21.15.125:0/2658473450 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38e80664d0 0x7f38e8068990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f38f8009b60 tx=0x7f38f8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:41.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.540+0000 7f38fffff640 1 -- 172.21.15.125:0/2658473450 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f39000ebc30 con 0x7f390815c030 2023-10-30T13:07:41.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.724+0000 7f390d565640 1 -- 172.21.15.125:0/2658473450 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3908159830 con 0x7f390815c030 2023-10-30T13:07:41.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.728+0000 7f38fffff640 1 -- 172.21.15.125:0/2658473450 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+6549 (secure 0 0 0) 0x7f39000ef250 con 0x7f390815c030 2023-10-30T13:07:41.733 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:41.733 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:40.986565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:active","state_seq":432,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:41.733 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 178 2023-10-30T13:07:41.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.728+0000 7f390d565640 1 -- 172.21.15.125:0/2658473450 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38e80664d0 msgr2=0x7f38e8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:41.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.728+0000 7f390d565640 1 --2- 172.21.15.125:0/2658473450 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38e80664d0 0x7f38e8068990 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f38f8009b60 tx=0x7f38f8008040 comp rx=0 tx=0).stop 2023-10-30T13:07:41.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.728+0000 7f390d565640 1 -- 172.21.15.125:0/2658473450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f390815c030 msgr2=0x7f3908142c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:41.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.728+0000 7f390d565640 1 --2- 172.21.15.125:0/2658473450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f390815c030 0x7f3908142c60 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f390001d150 tx=0x7f38f4005800 comp rx=0 tx=0).stop 2023-10-30T13:07:41.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.728+0000 7f390d565640 1 -- 172.21.15.125:0/2658473450 shutdown_connections 2023-10-30T13:07:41.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.728+0000 7f390d565640 1 --2- 172.21.15.125:0/2658473450 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38e80664d0 0x7f38e8068990 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:41.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.728+0000 7f390d565640 1 --2- 172.21.15.125:0/2658473450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f390815c030 0x7f3908142c60 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:41.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.728+0000 7f390d565640 1 --2- 172.21.15.125:0/2658473450 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f39080b1960 0x7f390815baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:41.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.728+0000 7f390d565640 1 --2- 172.21.15.125:0/2658473450 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38f4004d10 0x7f390815d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:41.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.728+0000 7f390d565640 1 -- 172.21.15.125:0/2658473450 >> 172.21.15.125:0/2658473450 conn(0x7f390806bfe0 msgr2=0x7f3908159f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:41.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.728+0000 7f390d565640 1 -- 172.21.15.125:0/2658473450 shutdown_connections 2023-10-30T13:07:41.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:41.728+0000 7f390d565640 1 -- 172.21.15.125:0/2658473450 wait complete. 2023-10-30T13:07:41.753 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T13:07:41.754 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before thrashing 2023-10-30T13:07:42.345 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 8de9501df27 0.81 2023-10-30T13:07:42.352 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.81 2023-10-30T13:07:42.663 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 82 -eq 48 ] 2023-10-30T13:07:42.663 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.82 2023-10-30T13:07:42.664 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.82 2023-10-30T13:07:42.664 INFO:tasks.workunit.client.0.smithi125.stdout:v0.82 2023-10-30T13:07:42.664 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.82 2023-10-30T13:07:44.754 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:44.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.896+0000 7f32f6c46640 1 Processor -- start 2023-10-30T13:07:44.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.896+0000 7f32f6c46640 1 -- start start 2023-10-30T13:07:44.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.896+0000 7f32f6c46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f32f00cf160 0x7f32f00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:44.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.896+0000 7f32f6c46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f00cb800 con 0x7f32f015d6f0 2023-10-30T13:07:44.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.896+0000 7f32f6c46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f00cb330 con 0x7f32f01538a0 2023-10-30T13:07:44.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.896+0000 7f32f6c46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f00cad70 con 0x7f32f00cf160 2023-10-30T13:07:44.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.896+0000 7f32f51e3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f32f00cf160 0x7f32f00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:44.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.896+0000 7f32f51e3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f32f00cf160 0x7f32f00cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34566/0 (socket says 172.21.15.125:34566) 2023-10-30T13:07:44.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.896+0000 7f32f51e3640 1 -- 172.21.15.125:0/4278049587 learned_addr learned my addr 172.21.15.125:0/4278049587 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:44.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.896+0000 7f32f51e3640 1 -- 172.21.15.125:0/4278049587 >> v1:172.21.15.136:6789/0 conn(0x7f32f01538a0 legacy=0x7f32f00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:44.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.896+0000 7f32ef7fe640 1 -- 172.21.15.125:0/4278049587 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737438782 0 0) 0x7f32f00cb800 con 0x7f32f015d6f0 2023-10-30T13:07:44.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.896+0000 7f32f51e3640 1 -- 172.21.15.125:0/4278049587 >> v1:172.21.15.125:6789/0 conn(0x7f32f015d6f0 legacy=0x7f32f015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:44.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.896+0000 7f32f51e3640 1 -- 172.21.15.125:0/4278049587 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f00cb050 con 0x7f32f00cf160 2023-10-30T13:07:44.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.900+0000 7f32f51e3640 1 --2- 172.21.15.125:0/4278049587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32f00cf160 0x7f32f00cf590 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009e50 tx=0x7f32e801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671f8491eff4ba3a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:44.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.900+0000 7f32ef7fe640 1 -- 172.21.15.125:0/4278049587 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e80af400 con 0x7f32f00cf160 2023-10-30T13:07:44.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.900+0000 7f32ef7fe640 1 -- 172.21.15.125:0/4278049587 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e80ab050 con 0x7f32f00cf160 2023-10-30T13:07:44.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.900+0000 7f32ef7fe640 1 -- 172.21.15.125:0/4278049587 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e80af780 con 0x7f32f00cf160 2023-10-30T13:07:44.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.900+0000 7f32f6c46640 1 -- 172.21.15.125:0/4278049587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32f00cf160 msgr2=0x7f32f00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:44.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.900+0000 7f32f6c46640 1 --2- 172.21.15.125:0/4278049587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32f00cf160 0x7f32f00cf590 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f32e8009e50 tx=0x7f32e801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:07:44.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.900+0000 7f32f6c46640 1 -- 172.21.15.125:0/4278049587 shutdown_connections 2023-10-30T13:07:44.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.900+0000 7f32f6c46640 1 --2- 172.21.15.125:0/4278049587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32f00cf160 0x7f32f00cf590 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:44.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.900+0000 7f32f6c46640 1 -- 172.21.15.125:0/4278049587 >> 172.21.15.125:0/4278049587 conn(0x7f32f006bfe0 msgr2=0x7f32f006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:44.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.900+0000 7f32f6c46640 1 -- 172.21.15.125:0/4278049587 shutdown_connections 2023-10-30T13:07:44.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.900+0000 7f32f6c46640 1 -- 172.21.15.125:0/4278049587 wait complete. 2023-10-30T13:07:44.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f6c46640 1 Processor -- start 2023-10-30T13:07:44.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f6c46640 1 -- start start 2023-10-30T13:07:44.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f6c46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32f00cf160 0x7f32f014fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:44.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f6c46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32f01538a0 0x7f32f01500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:44.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f6c46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32f015d6f0 0x7f32f00b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:44.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f6c46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f014f630 con 0x7f32f015d6f0 2023-10-30T13:07:44.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f6c46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f01519c0 con 0x7f32f01538a0 2023-10-30T13:07:44.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f6c46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f014dc80 con 0x7f32f00cf160 2023-10-30T13:07:44.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f51e3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32f015d6f0 0x7f32f00b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:44.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f51e3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32f015d6f0 0x7f32f00b45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34572/0 (socket says 172.21.15.125:34572) 2023-10-30T13:07:44.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f51e3640 1 -- 172.21.15.125:0/1039811512 learned_addr learned my addr 172.21.15.125:0/1039811512 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:44.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f49e2640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32f00cf160 0x7f32f014fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:44.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32effff640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32f01538a0 0x7f32f01500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:44.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f51e3640 1 -- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32f00cf160 msgr2=0x7f32f014fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:44.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f51e3640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32f00cf160 0x7f32f014fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:44.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f51e3640 1 -- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32f01538a0 msgr2=0x7f32f01500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:44.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f51e3640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32f01538a0 0x7f32f01500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:44.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f51e3640 1 -- 172.21.15.125:0/1039811512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e8009b30 con 0x7f32f015d6f0 2023-10-30T13:07:44.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f49e2640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32f00cf160 0x7f32f014fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:07:44.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32effff640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32f01538a0 0x7f32f01500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:07:44.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f51e3640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32f015d6f0 0x7f32f00b45c0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f32e801be00 tx=0x7f32e801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:44.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32ed7fa640 1 -- 172.21.15.125:0/1039811512 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e80afd30 con 0x7f32f015d6f0 2023-10-30T13:07:44.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32ed7fa640 1 -- 172.21.15.125:0/1039811512 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e80afe90 con 0x7f32f015d6f0 2023-10-30T13:07:44.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f6c46640 1 -- 172.21.15.125:0/1039811512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f0151700 con 0x7f32f015d6f0 2023-10-30T13:07:44.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32ed7fa640 1 -- 172.21.15.125:0/1039811512 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e80ba660 con 0x7f32f015d6f0 2023-10-30T13:07:44.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.904+0000 7f32f6c46640 1 -- 172.21.15.125:0/1039811512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f014e200 con 0x7f32f015d6f0 2023-10-30T13:07:44.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.908+0000 7f32f6c46640 1 -- 172.21.15.125:0/1039811512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f004e170 con 0x7f32f015d6f0 2023-10-30T13:07:44.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.908+0000 7f32ed7fa640 1 -- 172.21.15.125:0/1039811512 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f32e80b6040 con 0x7f32f015d6f0 2023-10-30T13:07:44.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.908+0000 7f32ed7fa640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32d40664d0 0x7f32d4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:44.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.908+0000 7f32ed7fa640 1 -- 172.21.15.125:0/1039811512 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f32e81227d0 con 0x7f32f015d6f0 2023-10-30T13:07:44.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.908+0000 7f32f49e2640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32d40664d0 0x7f32d4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:44.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.912+0000 7f32f49e2640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32d40664d0 0x7f32d4068990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f32e4000f70 tx=0x7f32e4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:44.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:44.912+0000 7f32ed7fa640 1 -- 172.21.15.125:0/1039811512 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f32e80ef5e0 con 0x7f32f015d6f0 2023-10-30T13:07:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.108+0000 7f32f6c46640 1 -- 172.21.15.125:0/1039811512 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32f00b1750 con 0x7f32f015d6f0 2023-10-30T13:07:45.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.108+0000 7f32ed7fa640 1 -- 172.21.15.125:0/1039811512 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+6549 (secure 0 0 0) 0x7f32e8009510 con 0x7f32f015d6f0 2023-10-30T13:07:45.115 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:45.115 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:40.986565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:active","state_seq":432,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:45.115 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 178 2023-10-30T13:07:45.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.112+0000 7f32f6c46640 1 -- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32d40664d0 msgr2=0x7f32d4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:45.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.112+0000 7f32f6c46640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32d40664d0 0x7f32d4068990 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f32e4000f70 tx=0x7f32e4009040 comp rx=0 tx=0).stop 2023-10-30T13:07:45.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.112+0000 7f32f6c46640 1 -- 172.21.15.125:0/1039811512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32f015d6f0 msgr2=0x7f32f00b45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:45.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.112+0000 7f32f6c46640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32f015d6f0 0x7f32f00b45c0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f32e801be00 tx=0x7f32e801bf90 comp rx=0 tx=0).stop 2023-10-30T13:07:45.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.112+0000 7f32f6c46640 1 -- 172.21.15.125:0/1039811512 shutdown_connections 2023-10-30T13:07:45.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.112+0000 7f32f6c46640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f32d40664d0 0x7f32d4068990 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:45.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.112+0000 7f32f6c46640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f32f015d6f0 0x7f32f00b45c0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:45.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.112+0000 7f32f6c46640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f32f01538a0 0x7f32f01500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:45.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.112+0000 7f32f6c46640 1 --2- 172.21.15.125:0/1039811512 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f32f00cf160 0x7f32f014fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:45.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.112+0000 7f32f6c46640 1 -- 172.21.15.125:0/1039811512 >> 172.21.15.125:0/1039811512 conn(0x7f32f006bfe0 msgr2=0x7f32f00cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:45.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.112+0000 7f32f6c46640 1 -- 172.21.15.125:0/1039811512 shutdown_connections 2023-10-30T13:07:45.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.112+0000 7f32f6c46640 1 -- 172.21.15.125:0/1039811512 wait complete. 2023-10-30T13:07:45.135 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-10-30T13:07:45.136 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:07:45.136 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:07:45.170 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:07:45.171 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:45.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.304+0000 7f4f8c49e640 1 Processor -- start 2023-10-30T13:07:45.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.304+0000 7f4f8c49e640 1 -- start start 2023-10-30T13:07:45.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.304+0000 7f4f8c49e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4f840c05d0 0x7f4f8413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:45.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.304+0000 7f4f8c49e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f840c6f60 0x7f4f8413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:45.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.304+0000 7f4f8c49e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f840c0a50 con 0x7f4f840c6b80 2023-10-30T13:07:45.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.304+0000 7f4f8c49e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f8406f6e0 con 0x7f4f840c05d0 2023-10-30T13:07:45.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.304+0000 7f4f8c49e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f8406f9c0 con 0x7f4f840c6f60 2023-10-30T13:07:45.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.304+0000 7f4f89a39640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4f840c05d0 0x7f4f8413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:45.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.304+0000 7f4f8aa3b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f840c6f60 0x7f4f8413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:45.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.308+0000 7f4f89a39640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4f840c05d0 0x7f4f8413ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:57402/0 (socket says 172.21.15.125:57402) 2023-10-30T13:07:45.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.308+0000 7f4f8a23a640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f4f840c6b80 0x7f4f840bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:59800/0 (socket says 172.21.15.125:59800) 2023-10-30T13:07:45.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.308+0000 7f4f8aa3b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f840c6f60 0x7f4f8413f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:37170/0 (socket says 172.21.15.125:37170) 2023-10-30T13:07:45.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.308+0000 7f4f89a39640 1 -- 172.21.15.125:0/1116891303 learned_addr learned my addr 172.21.15.125:0/1116891303 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:45.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.308+0000 7f4f8aa3b640 1 -- 172.21.15.125:0/1116891303 >> v2:172.21.15.136:3300/0 conn(0x7f4f840c05d0 msgr2=0x7f4f8413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:45.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.308+0000 7f4f8aa3b640 1 --2- 172.21.15.125:0/1116891303 >> v2:172.21.15.136:3300/0 conn(0x7f4f840c05d0 0x7f4f8413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:45.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.308+0000 7f4f8aa3b640 1 -- 172.21.15.125:0/1116891303 >> v1:172.21.15.136:6789/0 conn(0x7f4f840c6b80 legacy=0x7f4f840bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:45.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.308+0000 7f4f8aa3b640 1 -- 172.21.15.125:0/1116891303 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8406fca0 con 0x7f4f840c6f60 2023-10-30T13:07:45.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.308+0000 7f4f89a39640 1 --2- 172.21.15.125:0/1116891303 >> v2:172.21.15.136:3300/0 conn(0x7f4f840c05d0 0x7f4f8413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:07:45.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.308+0000 7f4f8aa3b640 1 --2- 172.21.15.125:0/1116891303 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f840c6f60 0x7f4f8413f2d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c009d90 tx=0x7f4f7c01c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92d00f2367e5ece6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:45.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.308+0000 7f4f89238640 1 -- 172.21.15.125:0/1116891303 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0a5920 con 0x7f4f840c6f60 2023-10-30T13:07:45.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f89238640 1 -- 172.21.15.125:0/1116891303 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7c0a5a80 con 0x7f4f840c6f60 2023-10-30T13:07:45.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f89238640 1 -- 172.21.15.125:0/1116891303 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7c0a5e10 con 0x7f4f840c6f60 2023-10-30T13:07:45.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 -- 172.21.15.125:0/1116891303 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f840c6f60 msgr2=0x7f4f8413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:45.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 --2- 172.21.15.125:0/1116891303 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f840c6f60 0x7f4f8413f2d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c009d90 tx=0x7f4f7c01c950 comp rx=0 tx=0).stop 2023-10-30T13:07:45.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 -- 172.21.15.125:0/1116891303 shutdown_connections 2023-10-30T13:07:45.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 --2- 172.21.15.125:0/1116891303 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f840c6f60 0x7f4f8413f2d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:45.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 --2- 172.21.15.125:0/1116891303 >> v2:172.21.15.136:3300/0 conn(0x7f4f840c05d0 0x7f4f8413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:45.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 -- 172.21.15.125:0/1116891303 >> 172.21.15.125:0/1116891303 conn(0x7f4f840c7910 msgr2=0x7f4f840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:45.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 -- 172.21.15.125:0/1116891303 shutdown_connections 2023-10-30T13:07:45.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 -- 172.21.15.125:0/1116891303 wait complete. 2023-10-30T13:07:45.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 Processor -- start 2023-10-30T13:07:45.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 -- start start 2023-10-30T13:07:45.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f840c05d0 0x7f4f84155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:45.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f840c6b80 0x7f4f841591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:45.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f840c6f60 0x7f4f84152600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:45.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f841596e0 con 0x7f4f840c6b80 2023-10-30T13:07:45.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f841f5aa0 con 0x7f4f840c6f60 2023-10-30T13:07:45.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8c49e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f841f4e90 con 0x7f4f840c05d0 2023-10-30T13:07:45.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f89a39640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f840c6b80 0x7f4f841591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:45.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f89a39640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f840c6b80 0x7f4f841591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34576/0 (socket says 172.21.15.125:34576) 2023-10-30T13:07:45.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f89a39640 1 -- 172.21.15.125:0/224498289 learned_addr learned my addr 172.21.15.125:0/224498289 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:45.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8aa3b640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f840c6f60 0x7f4f84152600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:45.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.312+0000 7f4f8a23a640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f840c05d0 0x7f4f84155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:45.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f89a39640 1 -- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f840c05d0 msgr2=0x7f4f84155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:45.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f89a39640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f840c05d0 0x7f4f84155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:45.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f89a39640 1 -- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f840c6f60 msgr2=0x7f4f84152600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:45.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f89a39640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f840c6f60 0x7f4f84152600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:45.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f89a39640 1 -- 172.21.15.125:0/224498289 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f740094d0 con 0x7f4f840c6b80 2023-10-30T13:07:45.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f8aa3b640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f840c6f60 0x7f4f84152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:07:45.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f89a39640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f840c6b80 0x7f4f841591a0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400e9d0 tx=0x7f4f7400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:45.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f7affd640 1 -- 172.21.15.125:0/224498289 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f7400ca60 con 0x7f4f840c6b80 2023-10-30T13:07:45.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f7affd640 1 -- 172.21.15.125:0/224498289 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f7400cbc0 con 0x7f4f840c6b80 2023-10-30T13:07:45.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f7affd640 1 -- 172.21.15.125:0/224498289 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f74013470 con 0x7f4f840c6b80 2023-10-30T13:07:45.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f8c49e640 1 -- 172.21.15.125:0/224498289 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7c009a70 con 0x7f4f840c6b80 2023-10-30T13:07:45.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f8c49e640 1 -- 172.21.15.125:0/224498289 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f841f5450 con 0x7f4f840c6b80 2023-10-30T13:07:45.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f8c49e640 1 -- 172.21.15.125:0/224498289 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f8404e170 con 0x7f4f840c6b80 2023-10-30T13:07:45.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f7affd640 1 -- 172.21.15.125:0/224498289 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4f740135d0 con 0x7f4f840c6b80 2023-10-30T13:07:45.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f7affd640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f680664a0 0x7f4f68068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:45.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f7affd640 1 -- 172.21.15.125:0/224498289 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f4f74084ba0 con 0x7f4f840c6b80 2023-10-30T13:07:45.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.316+0000 7f4f8a23a640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f680664a0 0x7f4f68068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:45.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.320+0000 7f4f8a23a640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f680664a0 0x7f4f68068960 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f4f80000f70 tx=0x7f4f80007e90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:45.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.320+0000 7f4f7affd640 1 -- 172.21.15.125:0/224498289 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4f74051910 con 0x7f4f840c6b80 2023-10-30T13:07:45.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.504+0000 7f4f8c49e640 1 -- 172.21.15.125:0/224498289 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f840c8070 con 0x7f4f840c6b80 2023-10-30T13:07:45.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.504+0000 7f4f7affd640 1 -- 172.21.15.125:0/224498289 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+6549 (secure 0 0 0) 0x7f4f74017090 con 0x7f4f840c6b80 2023-10-30T13:07:45.509 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:45.509 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:40.986565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:active","state_seq":432,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:45.510 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 178 2023-10-30T13:07:45.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.504+0000 7f4f8c49e640 1 -- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f680664a0 msgr2=0x7f4f68068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:45.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.504+0000 7f4f8c49e640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f680664a0 0x7f4f68068960 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f4f80000f70 tx=0x7f4f80007e90 comp rx=0 tx=0).stop 2023-10-30T13:07:45.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.504+0000 7f4f8c49e640 1 -- 172.21.15.125:0/224498289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f840c6b80 msgr2=0x7f4f841591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:45.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.504+0000 7f4f8c49e640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f840c6b80 0x7f4f841591a0 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f4f7400e9d0 tx=0x7f4f7400ee90 comp rx=0 tx=0).stop 2023-10-30T13:07:45.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.504+0000 7f4f8c49e640 1 -- 172.21.15.125:0/224498289 shutdown_connections 2023-10-30T13:07:45.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.504+0000 7f4f8c49e640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f680664a0 0x7f4f68068960 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:45.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.504+0000 7f4f8c49e640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f840c6f60 0x7f4f84152600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:45.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.504+0000 7f4f8c49e640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f840c6b80 0x7f4f841591a0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:45.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.504+0000 7f4f8c49e640 1 --2- 172.21.15.125:0/224498289 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f840c05d0 0x7f4f84155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:45.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.504+0000 7f4f8c49e640 1 -- 172.21.15.125:0/224498289 >> 172.21.15.125:0/224498289 conn(0x7f4f840c7910 msgr2=0x7f4f840bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:45.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.504+0000 7f4f8c49e640 1 -- 172.21.15.125:0/224498289 shutdown_connections 2023-10-30T13:07:45.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:45.504+0000 7f4f8c49e640 1 -- 172.21.15.125:0/224498289 wait complete. 2023-10-30T13:07:45.529 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:07:47.530 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:47.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.672+0000 7f5caf0dc640 1 Processor -- start 2023-10-30T13:07:47.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.672+0000 7f5caf0dc640 1 -- start start 2023-10-30T13:07:47.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.672+0000 7f5caf0dc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5ca80717b0 0x7f5ca815ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:47.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.672+0000 7f5caf0dc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ca8071b90 0x7f5ca81513c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:47.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.672+0000 7f5caf0dc640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca815b060 con 0x7f5ca80c6290 2023-10-30T13:07:47.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.672+0000 7f5caf0dc640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca8157530 con 0x7f5ca8071b90 2023-10-30T13:07:47.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.672+0000 7f5caf0dc640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca80c4ab0 con 0x7f5ca80717b0 2023-10-30T13:07:47.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.672+0000 7f5cad8d9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5ca8071b90 0x7f5ca81513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.672+0000 7f5cae8db640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5ca80c6290 0x7f5ca8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41236/0 (socket says 172.21.15.125:41236) 2023-10-30T13:07:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.672+0000 7f5cae8db640 1 -- 172.21.15.125:0/925544019 learned_addr learned my addr 172.21.15.125:0/925544019 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:07:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.676+0000 7f5cae0da640 1 --2- 172.21.15.125:0/925544019 >> v2:172.21.15.136:3300/0 conn(0x7f5ca80717b0 0x7f5ca815ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.676+0000 7f5cad8d9640 1 -- 172.21.15.125:0/925544019 >> v2:172.21.15.136:3300/0 conn(0x7f5ca80717b0 msgr2=0x7f5ca815ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:47.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.676+0000 7f5cad8d9640 1 --2- 172.21.15.125:0/925544019 >> v2:172.21.15.136:3300/0 conn(0x7f5ca80717b0 0x7f5ca815ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:47.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.676+0000 7f5cad8d9640 1 -- 172.21.15.125:0/925544019 >> v1:172.21.15.125:6789/0 conn(0x7f5ca80c6290 legacy=0x7f5ca8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:47.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.676+0000 7f5cad8d9640 1 -- 172.21.15.125:0/925544019 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca80c4d90 con 0x7f5ca8071b90 2023-10-30T13:07:47.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.676+0000 7f5cad0d8640 1 -- 172.21.15.125:0/925544019 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404483861 0 0) 0x7f5ca815b060 con 0x7f5ca80c6290 2023-10-30T13:07:47.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.676+0000 7f5cad8d9640 1 --2- 172.21.15.125:0/925544019 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca8071b90 0x7f5ca81513c0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f5ca806b360 tx=0x7f5c9801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a5ed611c6f68ae9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:47.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5cad0d8640 1 -- 172.21.15.125:0/925544019 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c980a8950 con 0x7f5ca8071b90 2023-10-30T13:07:47.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5cad0d8640 1 -- 172.21.15.125:0/925544019 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c980a8ab0 con 0x7f5ca8071b90 2023-10-30T13:07:47.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5cad0d8640 1 -- 172.21.15.125:0/925544019 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c980a8e40 con 0x7f5ca8071b90 2023-10-30T13:07:47.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 -- 172.21.15.125:0/925544019 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca8071b90 msgr2=0x7f5ca81513c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:47.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 --2- 172.21.15.125:0/925544019 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca8071b90 0x7f5ca81513c0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f5ca806b360 tx=0x7f5c9801ec50 comp rx=0 tx=0).stop 2023-10-30T13:07:47.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 -- 172.21.15.125:0/925544019 shutdown_connections 2023-10-30T13:07:47.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 --2- 172.21.15.125:0/925544019 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca8071b90 0x7f5ca81513c0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:47.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 --2- 172.21.15.125:0/925544019 >> v2:172.21.15.136:3300/0 conn(0x7f5ca80717b0 0x7f5ca815ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:47.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 -- 172.21.15.125:0/925544019 >> 172.21.15.125:0/925544019 conn(0x7f5ca80c75d0 msgr2=0x7f5ca80c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:47.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 -- 172.21.15.125:0/925544019 shutdown_connections 2023-10-30T13:07:47.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 -- 172.21.15.125:0/925544019 wait complete. 2023-10-30T13:07:47.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 Processor -- start 2023-10-30T13:07:47.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 -- start start 2023-10-30T13:07:47.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca80717b0 0x7f5ca81ed4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:47.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ca8071b90 0x7f5ca8148460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:47.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ca80c6290 0x7f5ca814b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:47.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca8151880 con 0x7f5ca80717b0 2023-10-30T13:07:47.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca80c8930 con 0x7f5ca80c6290 2023-10-30T13:07:47.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.680+0000 7f5caf0dc640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca80c8b80 con 0x7f5ca8071b90 2023-10-30T13:07:47.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5cae0da640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca80717b0 0x7f5ca81ed4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:47.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5cae0da640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca80717b0 0x7f5ca81ed4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34600/0 (socket says 172.21.15.125:34600) 2023-10-30T13:07:47.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5cae0da640 1 -- 172.21.15.125:0/3818900001 learned_addr learned my addr 172.21.15.125:0/3818900001 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:47.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5cae8db640 1 --2- 172.21.15.125:0/3818900001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ca80c6290 0x7f5ca814b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:47.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5cad8d9640 1 --2- 172.21.15.125:0/3818900001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ca8071b90 0x7f5ca8148460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:47.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5cae0da640 1 -- 172.21.15.125:0/3818900001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ca8071b90 msgr2=0x7f5ca8148460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:47.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5cae0da640 1 --2- 172.21.15.125:0/3818900001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ca8071b90 0x7f5ca8148460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:47.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5cae0da640 1 -- 172.21.15.125:0/3818900001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ca80c6290 msgr2=0x7f5ca814b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:47.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5cae0da640 1 --2- 172.21.15.125:0/3818900001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ca80c6290 0x7f5ca814b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:47.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5cae0da640 1 -- 172.21.15.125:0/3818900001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c98009420 con 0x7f5ca80717b0 2023-10-30T13:07:47.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5cae0da640 1 --2- 172.21.15.125:0/3818900001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca80717b0 0x7f5ca81ed4e0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400d7d0 tx=0x7f5ca400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:47.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5c9effd640 1 -- 172.21.15.125:0/3818900001 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca400b6c0 con 0x7f5ca80717b0 2023-10-30T13:07:47.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5c9effd640 1 -- 172.21.15.125:0/3818900001 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ca400b820 con 0x7f5ca80717b0 2023-10-30T13:07:47.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5caf0dc640 1 -- 172.21.15.125:0/3818900001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca80c8e60 con 0x7f5ca80717b0 2023-10-30T13:07:47.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5c9effd640 1 -- 172.21.15.125:0/3818900001 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ca400bbd0 con 0x7f5ca80717b0 2023-10-30T13:07:47.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5caf0dc640 1 -- 172.21.15.125:0/3818900001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca80c9420 con 0x7f5ca80717b0 2023-10-30T13:07:47.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5caf0dc640 1 -- 172.21.15.125:0/3818900001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca804e170 con 0x7f5ca80717b0 2023-10-30T13:07:47.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.684+0000 7f5c9effd640 1 -- 172.21.15.125:0/3818900001 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5ca400bd30 con 0x7f5ca80717b0 2023-10-30T13:07:47.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.688+0000 7f5c9effd640 1 --2- 172.21.15.125:0/3818900001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5c8c0664a0 0x7f5c8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:47.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.688+0000 7f5c9effd640 1 -- 172.21.15.125:0/3818900001 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f5ca4084a20 con 0x7f5ca80717b0 2023-10-30T13:07:47.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.688+0000 7f5cad8d9640 1 --2- 172.21.15.125:0/3818900001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5c8c0664a0 0x7f5c8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:47.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.688+0000 7f5cad8d9640 1 --2- 172.21.15.125:0/3818900001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5c8c0664a0 0x7f5c8c068960 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5c980b2000 tx=0x7f5c98008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:47.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.692+0000 7f5c9effd640 1 -- 172.21.15.125:0/3818900001 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5ca4051800 con 0x7f5ca80717b0 2023-10-30T13:07:47.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.872+0000 7f5caf0dc640 1 -- 172.21.15.125:0/3818900001 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ca80c3410 con 0x7f5ca80717b0 2023-10-30T13:07:47.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.872+0000 7f5c9effd640 1 -- 172.21.15.125:0/3818900001 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+6549 (secure 0 0 0) 0x7f5ca4054e20 con 0x7f5ca80717b0 2023-10-30T13:07:47.877 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:47.878 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:40.986565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:active","state_seq":432,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:47.878 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 178 2023-10-30T13:07:47.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.872+0000 7f5caf0dc640 1 -- 172.21.15.125:0/3818900001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5c8c0664a0 msgr2=0x7f5c8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:47.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.872+0000 7f5caf0dc640 1 --2- 172.21.15.125:0/3818900001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5c8c0664a0 0x7f5c8c068960 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f5c980b2000 tx=0x7f5c98008040 comp rx=0 tx=0).stop 2023-10-30T13:07:47.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.872+0000 7f5caf0dc640 1 -- 172.21.15.125:0/3818900001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca80717b0 msgr2=0x7f5ca81ed4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:47.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.872+0000 7f5caf0dc640 1 --2- 172.21.15.125:0/3818900001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca80717b0 0x7f5ca81ed4e0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f5ca400d7d0 tx=0x7f5ca400dc90 comp rx=0 tx=0).stop 2023-10-30T13:07:47.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.872+0000 7f5caf0dc640 1 -- 172.21.15.125:0/3818900001 shutdown_connections 2023-10-30T13:07:47.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.872+0000 7f5caf0dc640 1 --2- 172.21.15.125:0/3818900001 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5c8c0664a0 0x7f5c8c068960 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:47.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.872+0000 7f5caf0dc640 1 --2- 172.21.15.125:0/3818900001 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5ca80c6290 0x7f5ca814b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:47.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.872+0000 7f5caf0dc640 1 --2- 172.21.15.125:0/3818900001 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5ca8071b90 0x7f5ca8148460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:47.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.872+0000 7f5caf0dc640 1 --2- 172.21.15.125:0/3818900001 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5ca80717b0 0x7f5ca81ed4e0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:47.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.872+0000 7f5caf0dc640 1 -- 172.21.15.125:0/3818900001 >> 172.21.15.125:0/3818900001 conn(0x7f5ca80c75d0 msgr2=0x7f5ca80c2230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:47.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.872+0000 7f5caf0dc640 1 -- 172.21.15.125:0/3818900001 shutdown_connections 2023-10-30T13:07:47.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:47.872+0000 7f5caf0dc640 1 -- 172.21.15.125:0/3818900001 wait complete. 2023-10-30T13:07:47.898 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:07:49.899 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:50.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.040+0000 7f1937a54640 1 Processor -- start 2023-10-30T13:07:50.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.040+0000 7f1937a54640 1 -- start start 2023-10-30T13:07:50.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.044+0000 7f1937a54640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f19301539a0 0x7f193015a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:50.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.044+0000 7f1937a54640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1930070a00 0x7f1930070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:50.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.044+0000 7f1937a54640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f193015a8c0 con 0x7f1930069e90 2023-10-30T13:07:50.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.044+0000 7f1937a54640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f193006fbb0 con 0x7f19301539a0 2023-10-30T13:07:50.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.044+0000 7f1937a54640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19300713b0 con 0x7f1930070a00 2023-10-30T13:07:50.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.044+0000 7f19357f0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f19301539a0 0x7f193015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:50.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.044+0000 7f19357f0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f19301539a0 0x7f193015a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34606/0 (socket says 172.21.15.125:34606) 2023-10-30T13:07:50.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.044+0000 7f19357f0640 1 -- 172.21.15.125:0/1368284677 learned_addr learned my addr 172.21.15.125:0/1368284677 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:50.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.044+0000 7f1934fef640 1 --2- 172.21.15.125:0/1368284677 >> v2:172.21.15.136:3300/0 conn(0x7f1930070a00 0x7f1930070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:50.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.044+0000 7f19357f0640 1 -- 172.21.15.125:0/1368284677 >> v2:172.21.15.136:3300/0 conn(0x7f1930070a00 msgr2=0x7f1930070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:50.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.044+0000 7f19357f0640 1 --2- 172.21.15.125:0/1368284677 >> v2:172.21.15.136:3300/0 conn(0x7f1930070a00 0x7f1930070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:50.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.044+0000 7f19357f0640 1 -- 172.21.15.125:0/1368284677 >> v1:172.21.15.136:6789/0 conn(0x7f1930069e90 legacy=0x7f193006a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:50.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.044+0000 7f19357f0640 1 -- 172.21.15.125:0/1368284677 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1930153d80 con 0x7f19301539a0 2023-10-30T13:07:50.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.048+0000 7f19357f0640 1 --2- 172.21.15.125:0/1368284677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19301539a0 0x7f193015a380 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f1930072640 tx=0x7f192c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b46735ccf6ce80 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:50.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.048+0000 7f1927fff640 1 -- 172.21.15.125:0/1368284677 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c0ab600 con 0x7f19301539a0 2023-10-30T13:07:50.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.048+0000 7f1927fff640 1 -- 172.21.15.125:0/1368284677 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192c0ab760 con 0x7f19301539a0 2023-10-30T13:07:50.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.048+0000 7f1927fff640 1 -- 172.21.15.125:0/1368284677 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192c0abaf0 con 0x7f19301539a0 2023-10-30T13:07:50.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.048+0000 7f1937a54640 1 -- 172.21.15.125:0/1368284677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19301539a0 msgr2=0x7f193015a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:50.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.048+0000 7f1937a54640 1 --2- 172.21.15.125:0/1368284677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19301539a0 0x7f193015a380 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f1930072640 tx=0x7f192c01c400 comp rx=0 tx=0).stop 2023-10-30T13:07:50.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.048+0000 7f1937a54640 1 -- 172.21.15.125:0/1368284677 shutdown_connections 2023-10-30T13:07:50.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.048+0000 7f1937a54640 1 --2- 172.21.15.125:0/1368284677 >> v2:172.21.15.136:3300/0 conn(0x7f1930070a00 0x7f1930070e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:50.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.048+0000 7f1937a54640 1 --2- 172.21.15.125:0/1368284677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19301539a0 0x7f193015a380 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:50.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.048+0000 7f1937a54640 1 -- 172.21.15.125:0/1368284677 >> 172.21.15.125:0/1368284677 conn(0x7f19300c7910 msgr2=0x7f19300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:50.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.048+0000 7f1937a54640 1 -- 172.21.15.125:0/1368284677 shutdown_connections 2023-10-30T13:07:50.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.048+0000 7f1937a54640 1 -- 172.21.15.125:0/1368284677 wait complete. 2023-10-30T13:07:50.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.048+0000 7f1937a54640 1 Processor -- start 2023-10-30T13:07:50.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.048+0000 7f1937a54640 1 -- start start 2023-10-30T13:07:50.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1937a54640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1930069e90 0x7f1930147b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:50.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1937a54640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1930070a00 0x7f19300736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:50.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1937a54640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19301539a0 0x7f19301f5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:50.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1937a54640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19301582f0 con 0x7f1930070a00 2023-10-30T13:07:50.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1937a54640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1930148080 con 0x7f1930069e90 2023-10-30T13:07:50.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1937a54640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19301482f0 con 0x7f19301539a0 2023-10-30T13:07:50.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1934fef640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1930070a00 0x7f19300736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:50.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1934fef640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1930070a00 0x7f19300736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34612/0 (socket says 172.21.15.125:34612) 2023-10-30T13:07:50.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1934fef640 1 -- 172.21.15.125:0/370043224 learned_addr learned my addr 172.21.15.125:0/370043224 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:50.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f19357f0640 1 --2- 172.21.15.125:0/370043224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1930069e90 0x7f1930147b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:50.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1935ff1640 1 --2- 172.21.15.125:0/370043224 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19301539a0 0x7f19301f5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:50.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1934fef640 1 -- 172.21.15.125:0/370043224 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19301539a0 msgr2=0x7f19301f5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:50.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1934fef640 1 --2- 172.21.15.125:0/370043224 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19301539a0 0x7f19301f5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:50.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1934fef640 1 -- 172.21.15.125:0/370043224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1930069e90 msgr2=0x7f1930147b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:50.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1934fef640 1 --2- 172.21.15.125:0/370043224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1930069e90 0x7f1930147b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:50.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1934fef640 1 -- 172.21.15.125:0/370043224 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192c0094d0 con 0x7f1930070a00 2023-10-30T13:07:50.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1934fef640 1 --2- 172.21.15.125:0/370043224 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1930070a00 0x7f19300736e0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f192000d800 tx=0x7f192000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:50.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1925ffb640 1 -- 172.21.15.125:0/370043224 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192000b6c0 con 0x7f1930070a00 2023-10-30T13:07:50.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1925ffb640 1 -- 172.21.15.125:0/370043224 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f192000b820 con 0x7f1930070a00 2023-10-30T13:07:50.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1925ffb640 1 -- 172.21.15.125:0/370043224 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f192000bb90 con 0x7f1930070a00 2023-10-30T13:07:50.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1937a54640 1 -- 172.21.15.125:0/370043224 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19301595c0 con 0x7f1930070a00 2023-10-30T13:07:50.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1937a54640 1 -- 172.21.15.125:0/370043224 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1930157950 con 0x7f1930070a00 2023-10-30T13:07:50.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1937a54640 1 -- 172.21.15.125:0/370043224 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f193004e170 con 0x7f1930070a00 2023-10-30T13:07:50.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.052+0000 7f1925ffb640 1 -- 172.21.15.125:0/370043224 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f192000bcf0 con 0x7f1930070a00 2023-10-30T13:07:50.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.056+0000 7f1925ffb640 1 --2- 172.21.15.125:0/370043224 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19140664a0 0x7f1914068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:50.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.056+0000 7f1925ffb640 1 -- 172.21.15.125:0/370043224 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f1920084af0 con 0x7f1930070a00 2023-10-30T13:07:50.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.056+0000 7f19357f0640 1 --2- 172.21.15.125:0/370043224 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19140664a0 0x7f1914068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:50.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.056+0000 7f19357f0640 1 --2- 172.21.15.125:0/370043224 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19140664a0 0x7f1914068960 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f192c0af000 tx=0x7f192c0a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:50.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.060+0000 7f1925ffb640 1 -- 172.21.15.125:0/370043224 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1920051830 con 0x7f1930070a00 2023-10-30T13:07:50.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.240+0000 7f1937a54640 1 -- 172.21.15.125:0/370043224 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f193014def0 con 0x7f1930070a00 2023-10-30T13:07:50.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.240+0000 7f1925ffb640 1 -- 172.21.15.125:0/370043224 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+6549 (secure 0 0 0) 0x7f1920054e50 con 0x7f1930070a00 2023-10-30T13:07:50.245 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:50.245 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:40.986565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:active","state_seq":432,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:50.245 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 178 2023-10-30T13:07:50.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.240+0000 7f1937a54640 1 -- 172.21.15.125:0/370043224 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19140664a0 msgr2=0x7f1914068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:50.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.240+0000 7f1937a54640 1 --2- 172.21.15.125:0/370043224 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19140664a0 0x7f1914068960 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f192c0af000 tx=0x7f192c0a7000 comp rx=0 tx=0).stop 2023-10-30T13:07:50.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.240+0000 7f1937a54640 1 -- 172.21.15.125:0/370043224 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1930070a00 msgr2=0x7f19300736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:50.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.240+0000 7f1937a54640 1 --2- 172.21.15.125:0/370043224 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1930070a00 0x7f19300736e0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f192000d800 tx=0x7f192000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:07:50.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.240+0000 7f1937a54640 1 -- 172.21.15.125:0/370043224 shutdown_connections 2023-10-30T13:07:50.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.240+0000 7f1937a54640 1 --2- 172.21.15.125:0/370043224 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19140664a0 0x7f1914068960 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:50.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.240+0000 7f1937a54640 1 --2- 172.21.15.125:0/370043224 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19301539a0 0x7f19301f5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:50.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.240+0000 7f1937a54640 1 --2- 172.21.15.125:0/370043224 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1930070a00 0x7f19300736e0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:50.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.240+0000 7f1937a54640 1 --2- 172.21.15.125:0/370043224 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1930069e90 0x7f1930147b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:50.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.240+0000 7f1937a54640 1 -- 172.21.15.125:0/370043224 >> 172.21.15.125:0/370043224 conn(0x7f19300c7910 msgr2=0x7f193014e110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:50.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.240+0000 7f1937a54640 1 -- 172.21.15.125:0/370043224 shutdown_connections 2023-10-30T13:07:50.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:50.240+0000 7f1937a54640 1 -- 172.21.15.125:0/370043224 wait complete. 2023-10-30T13:07:50.265 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:07:52.266 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:52.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.408+0000 7f3798b0f640 1 Processor -- start 2023-10-30T13:07:52.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.408+0000 7f3798b0f640 1 -- start start 2023-10-30T13:07:52.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.408+0000 7f3798b0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6b80 0x7f37940bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:52.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.408+0000 7f3798b0f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f37940c6f60 0x7f37940c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:52.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.408+0000 7f3798b0f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37940c0a50 con 0x7f37940c8430 2023-10-30T13:07:52.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3798b0f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f379406f6e0 con 0x7f37940c6f60 2023-10-30T13:07:52.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3798b0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379406f9c0 con 0x7f37940c6b80 2023-10-30T13:07:52.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3792d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f37940c8430 0x7f379413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34124/0 (socket says 172.21.15.125:34124) 2023-10-30T13:07:52.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3792d9d640 1 -- 172.21.15.125:0/911467094 learned_addr learned my addr 172.21.15.125:0/911467094 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:07:52.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f379259c640 1 --2- 172.21.15.125:0/911467094 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6b80 0x7f37940bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:52.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3789d9b640 1 --2- 172.21.15.125:0/911467094 >> v2:172.21.15.136:3300/0 conn(0x7f37940c6f60 0x7f37940c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:52.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 -- 172.21.15.125:0/911467094 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690038831 0 0) 0x7f37940c0a50 con 0x7f37940c8430 2023-10-30T13:07:52.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 -- 172.21.15.125:0/911467094 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3780003530 con 0x7f37940c8430 2023-10-30T13:07:52.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 -- 172.21.15.125:0/911467094 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1992417081 0 0) 0x7f3780003530 con 0x7f37940c8430 2023-10-30T13:07:52.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 -- 172.21.15.125:0/911467094 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6b80 msgr2=0x7f37940bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:52.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 --2- 172.21.15.125:0/911467094 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6b80 0x7f37940bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:52.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 -- 172.21.15.125:0/911467094 >> v2:172.21.15.136:3300/0 conn(0x7f37940c6f60 msgr2=0x7f37940c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:52.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 --2- 172.21.15.125:0/911467094 >> v2:172.21.15.136:3300/0 conn(0x7f37940c6f60 0x7f37940c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:52.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 -- 172.21.15.125:0/911467094 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379406fca0 con 0x7f37940c8430 2023-10-30T13:07:52.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 -- 172.21.15.125:0/911467094 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f378c001a00 con 0x7f37940c8430 2023-10-30T13:07:52.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 -- 172.21.15.125:0/911467094 >> v1:172.21.15.125:6789/0 conn(0x7f37940c8430 legacy=0x7f379413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:52.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 --2- 172.21.15.125:0/911467094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3780003980 0x7f3780003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:52.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 --2- 172.21.15.125:0/911467094 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6b80 0x7f37800047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:52.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 --2- 172.21.15.125:0/911467094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3780004d10 0x7f3780005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:52.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 -- 172.21.15.125:0/911467094 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379406f6e0 con 0x7f3780003980 2023-10-30T13:07:52.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 -- 172.21.15.125:0/911467094 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379406f9c0 con 0x7f3780004d10 2023-10-30T13:07:52.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3791d9b640 1 -- 172.21.15.125:0/911467094 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378000b990 con 0x7f37940c6b80 2023-10-30T13:07:52.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f379259c640 1 --2- 172.21.15.125:0/911467094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3780003980 0x7f3780003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:52.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3789d9b640 1 --2- 172.21.15.125:0/911467094 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6b80 0x7f37800047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:52.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f3792d9d640 1 --2- 172.21.15.125:0/911467094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3780004d10 0x7f3780005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:52.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f379259c640 1 -- 172.21.15.125:0/911467094 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6b80 msgr2=0x7f37800047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:52.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f379259c640 1 --2- 172.21.15.125:0/911467094 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f37940c6b80 0x7f37800047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:52.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f379259c640 1 -- 172.21.15.125:0/911467094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3780004d10 msgr2=0x7f3780005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:52.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f379259c640 1 --2- 172.21.15.125:0/911467094 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3780004d10 0x7f3780005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:52.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.412+0000 7f379259c640 1 -- 172.21.15.125:0/911467094 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780003530 con 0x7f3780003980 2023-10-30T13:07:52.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.416+0000 7f379259c640 1 --2- 172.21.15.125:0/911467094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3780003980 0x7f3780003da0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f377c009670 tx=0x7f377c020e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dde68e811f9bd69 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:52.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.416+0000 7f3791d9b640 1 -- 172.21.15.125:0/911467094 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c0a9970 con 0x7f3780003980 2023-10-30T13:07:52.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.416+0000 7f3791d9b640 1 -- 172.21.15.125:0/911467094 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377c0a9ad0 con 0x7f3780003980 2023-10-30T13:07:52.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.416+0000 7f3791d9b640 1 -- 172.21.15.125:0/911467094 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c0a9e80 con 0x7f3780003980 2023-10-30T13:07:52.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.416+0000 7f3798b0f640 1 -- 172.21.15.125:0/911467094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3780003980 msgr2=0x7f3780003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:52.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.416+0000 7f3798b0f640 1 --2- 172.21.15.125:0/911467094 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3780003980 0x7f3780003da0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f377c009670 tx=0x7f377c020e80 comp rx=0 tx=0).stop 2023-10-30T13:07:52.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.416+0000 7f3792d9d640 1 -- 172.21.15.125:0/911467094 reap_dead start 2023-10-30T13:07:52.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.416+0000 7f3798b0f640 1 -- 172.21.15.125:0/911467094 shutdown_connections 2023-10-30T13:07:52.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.416+0000 7f3798b0f640 1 -- 172.21.15.125:0/911467094 >> 172.21.15.125:0/911467094 conn(0x7f37940c7910 msgr2=0x7f37940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:52.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.416+0000 7f3798b0f640 1 -- 172.21.15.125:0/911467094 shutdown_connections 2023-10-30T13:07:52.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.416+0000 7f3798b0f640 1 -- 172.21.15.125:0/911467094 wait complete. 2023-10-30T13:07:52.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.416+0000 7f3798b0f640 1 Processor -- start 2023-10-30T13:07:52.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.416+0000 7f3798b0f640 1 -- start start 2023-10-30T13:07:52.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f3798b0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3780003980 0x7f379413d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:52.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f3798b0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f379413de80 0x7f37940c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:52.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f3798b0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f379413e4c0 0x7f37940ca2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:52.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f3798b0f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37941f1920 con 0x7f3780003980 2023-10-30T13:07:52.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f3798b0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37941551d0 con 0x7f379413e4c0 2023-10-30T13:07:52.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f3798b0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37941f0d30 con 0x7f379413de80 2023-10-30T13:07:52.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f379259c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3780003980 0x7f379413d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:52.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f379259c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3780003980 0x7f379413d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41684/0 (socket says 172.21.15.125:41684) 2023-10-30T13:07:52.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f379259c640 1 -- 172.21.15.125:0/1023137056 learned_addr learned my addr 172.21.15.125:0/1023137056 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:52.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f3789d9b640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f379413de80 0x7f37940c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:52.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f3792d9d640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f379413e4c0 0x7f37940ca2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:52.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f379259c640 1 -- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f379413de80 msgr2=0x7f37940c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:52.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f379259c640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f379413de80 0x7f37940c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:52.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f379259c640 1 -- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f379413e4c0 msgr2=0x7f37940ca2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:52.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f379259c640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f379413e4c0 0x7f37940ca2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:52.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f379259c640 1 -- 172.21.15.125:0/1023137056 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3794155490 con 0x7f3780003980 2023-10-30T13:07:52.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f3789d9b640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f379413de80 0x7f37940c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:07:52.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f3792d9d640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f379413e4c0 0x7f37940ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:07:52.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f379259c640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3780003980 0x7f379413d940 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f377c0b4040 tx=0x7f3780005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:52.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f378b7fe640 1 -- 172.21.15.125:0/1023137056 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c0c2070 con 0x7f3780003980 2023-10-30T13:07:52.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f378b7fe640 1 -- 172.21.15.125:0/1023137056 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377c00f000 con 0x7f3780003980 2023-10-30T13:07:52.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f378b7fe640 1 -- 172.21.15.125:0/1023137056 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377c0b1c00 con 0x7f3780003980 2023-10-30T13:07:52.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f3798b0f640 1 -- 172.21.15.125:0/1023137056 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37941f0ff0 con 0x7f3780003980 2023-10-30T13:07:52.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f3798b0f640 1 -- 172.21.15.125:0/1023137056 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37941f15b0 con 0x7f3780003980 2023-10-30T13:07:52.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.420+0000 7f3798b0f640 1 -- 172.21.15.125:0/1023137056 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37941f1770 con 0x7f3780003980 2023-10-30T13:07:52.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.424+0000 7f378b7fe640 1 -- 172.21.15.125:0/1023137056 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f377c0b8040 con 0x7f3780003980 2023-10-30T13:07:52.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.424+0000 7f378b7fe640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37740664a0 0x7f3774068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:52.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.424+0000 7f378b7fe640 1 -- 172.21.15.125:0/1023137056 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f377c1224b0 con 0x7f3780003980 2023-10-30T13:07:52.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.424+0000 7f3789d9b640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37740664a0 0x7f3774068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:52.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.424+0000 7f3789d9b640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37740664a0 0x7f3774068960 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f37940c54d0 tx=0x7f3784008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:52.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.428+0000 7f378b7fe640 1 -- 172.21.15.125:0/1023137056 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f377c0ef1f0 con 0x7f3780003980 2023-10-30T13:07:52.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.608+0000 7f3798b0f640 1 -- 172.21.15.125:0/1023137056 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f379404e170 con 0x7f3780003980 2023-10-30T13:07:52.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.608+0000 7f378b7fe640 1 -- 172.21.15.125:0/1023137056 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+6549 (secure 0 0 0) 0x7f377c0f2810 con 0x7f3780003980 2023-10-30T13:07:52.614 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:52.614 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:40.986565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:active","state_seq":432,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:52.615 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 178 2023-10-30T13:07:52.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.608+0000 7f3798b0f640 1 -- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37740664a0 msgr2=0x7f3774068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:52.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.608+0000 7f3798b0f640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37740664a0 0x7f3774068960 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f37940c54d0 tx=0x7f3784008040 comp rx=0 tx=0).stop 2023-10-30T13:07:52.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.608+0000 7f3798b0f640 1 -- 172.21.15.125:0/1023137056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3780003980 msgr2=0x7f379413d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:52.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.608+0000 7f3798b0f640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3780003980 0x7f379413d940 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f377c0b4040 tx=0x7f3780005800 comp rx=0 tx=0).stop 2023-10-30T13:07:52.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.608+0000 7f3798b0f640 1 -- 172.21.15.125:0/1023137056 shutdown_connections 2023-10-30T13:07:52.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.608+0000 7f3798b0f640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f37740664a0 0x7f3774068960 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:52.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.608+0000 7f3798b0f640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f379413e4c0 0x7f37940ca2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:52.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.608+0000 7f3798b0f640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f379413de80 0x7f37940c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:52.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.608+0000 7f3798b0f640 1 --2- 172.21.15.125:0/1023137056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3780003980 0x7f379413d940 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:52.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.608+0000 7f3798b0f640 1 -- 172.21.15.125:0/1023137056 >> 172.21.15.125:0/1023137056 conn(0x7f37940c7910 msgr2=0x7f3794159be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:52.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.612+0000 7f3798b0f640 1 -- 172.21.15.125:0/1023137056 shutdown_connections 2023-10-30T13:07:52.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:52.612+0000 7f3798b0f640 1 -- 172.21.15.125:0/1023137056 wait complete. 2023-10-30T13:07:52.635 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:07:53.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:07:53.316 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:07:54.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:54.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f51629c5640 1 Processor -- start 2023-10-30T13:07:54.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f51629c5640 1 -- start start 2023-10-30T13:07:54.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f51629c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f515c06f7a0 0x7f515c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:54.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f51629c5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f515c0c6bd0 0x7f515c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:54.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f51629c5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515c06a3d0 con 0x7f515c071110 2023-10-30T13:07:54.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f51629c5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f515c0700c0 con 0x7f515c0c6bd0 2023-10-30T13:07:54.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f51629c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515c066cf0 con 0x7f515c06f7a0 2023-10-30T13:07:54.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f515b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f515c0c6bd0 0x7f515c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:54.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f515b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f515c0c6bd0 0x7f515c069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41690/0 (socket says 172.21.15.125:41690) 2023-10-30T13:07:54.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f515b7fe640 1 -- 172.21.15.125:0/890488487 learned_addr learned my addr 172.21.15.125:0/890488487 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:54.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f515bfff640 1 --2- 172.21.15.125:0/890488487 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f515c06f7a0 0x7f515c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:54.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f515b7fe640 1 -- 172.21.15.125:0/890488487 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f515c06f7a0 msgr2=0x7f515c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:54.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f515b7fe640 1 --2- 172.21.15.125:0/890488487 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f515c06f7a0 0x7f515c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:54.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f515b7fe640 1 -- 172.21.15.125:0/890488487 >> v1:172.21.15.136:6789/0 conn(0x7f515c071110 legacy=0x7f515c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:54.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.780+0000 7f515b7fe640 1 -- 172.21.15.125:0/890488487 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515c0c70d0 con 0x7f515c0c6bd0 2023-10-30T13:07:54.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.784+0000 7f515b7fe640 1 --2- 172.21.15.125:0/890488487 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f515c0c6bd0 0x7f515c069e90 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f5150009870 tx=0x7f515001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a8ab74427e1642e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:54.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.784+0000 7f515affd640 1 -- 172.21.15.125:0/890488487 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51500a8a30 con 0x7f515c0c6bd0 2023-10-30T13:07:54.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.784+0000 7f515affd640 1 -- 172.21.15.125:0/890488487 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51500a8b90 con 0x7f515c0c6bd0 2023-10-30T13:07:54.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.784+0000 7f515affd640 1 -- 172.21.15.125:0/890488487 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51500b0440 con 0x7f515c0c6bd0 2023-10-30T13:07:54.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.784+0000 7f51629c5640 1 -- 172.21.15.125:0/890488487 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f515c0c6bd0 msgr2=0x7f515c069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:54.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.784+0000 7f51629c5640 1 --2- 172.21.15.125:0/890488487 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f515c0c6bd0 0x7f515c069e90 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f5150009870 tx=0x7f515001ed30 comp rx=0 tx=0).stop 2023-10-30T13:07:54.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.784+0000 7f51629c5640 1 -- 172.21.15.125:0/890488487 shutdown_connections 2023-10-30T13:07:54.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.784+0000 7f51629c5640 1 --2- 172.21.15.125:0/890488487 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f515c0c6bd0 0x7f515c069e90 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:54.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.784+0000 7f51629c5640 1 --2- 172.21.15.125:0/890488487 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f515c06f7a0 0x7f515c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:54.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.784+0000 7f51629c5640 1 -- 172.21.15.125:0/890488487 >> 172.21.15.125:0/890488487 conn(0x7f515c0c7910 msgr2=0x7f515c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:54.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.784+0000 7f51629c5640 1 -- 172.21.15.125:0/890488487 shutdown_connections 2023-10-30T13:07:54.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.784+0000 7f51629c5640 1 -- 172.21.15.125:0/890488487 wait complete. 2023-10-30T13:07:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f51629c5640 1 Processor -- start 2023-10-30T13:07:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f51629c5640 1 -- start start 2023-10-30T13:07:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f51629c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f515c06f7a0 0x7f515c0ae110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f51629c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f515c071110 0x7f515c159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:54.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f51629c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f515c0c6bd0 0x7f515c15ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:54.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f51629c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c15b1d0 con 0x7f515c071110 2023-10-30T13:07:54.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f51629c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515c15b570 con 0x7f515c06f7a0 2023-10-30T13:07:54.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f51629c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515c159a20 con 0x7f515c0c6bd0 2023-10-30T13:07:54.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f515b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f515c071110 0x7f515c159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f515b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f515c071110 0x7f515c159eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41700/0 (socket says 172.21.15.125:41700) 2023-10-30T13:07:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f515b7fe640 1 -- 172.21.15.125:0/660852594 learned_addr learned my addr 172.21.15.125:0/660852594 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f515bfff640 1 --2- 172.21.15.125:0/660852594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f515c06f7a0 0x7f515c0ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f5160f62640 1 --2- 172.21.15.125:0/660852594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f515c0c6bd0 0x7f515c15ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f515b7fe640 1 -- 172.21.15.125:0/660852594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f515c0c6bd0 msgr2=0x7f515c15ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f515b7fe640 1 --2- 172.21.15.125:0/660852594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f515c0c6bd0 0x7f515c15ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f515b7fe640 1 -- 172.21.15.125:0/660852594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f515c06f7a0 msgr2=0x7f515c0ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f515b7fe640 1 --2- 172.21.15.125:0/660852594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f515c06f7a0 0x7f515c0ae110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f515b7fe640 1 -- 172.21.15.125:0/660852594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51500094d0 con 0x7f515c071110 2023-10-30T13:07:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f515b7fe640 1 --2- 172.21.15.125:0/660852594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f515c071110 0x7f515c159eb0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f51500b3040 tx=0x7f515001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f5158ff9640 1 -- 172.21.15.125:0/660852594 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51500be070 con 0x7f515c071110 2023-10-30T13:07:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f5158ff9640 1 -- 172.21.15.125:0/660852594 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51500ba400 con 0x7f515c071110 2023-10-30T13:07:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f5158ff9640 1 -- 172.21.15.125:0/660852594 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51500ba7d0 con 0x7f515c071110 2023-10-30T13:07:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f51629c5640 1 -- 172.21.15.125:0/660852594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515c1f2860 con 0x7f515c071110 2023-10-30T13:07:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f51629c5640 1 -- 172.21.15.125:0/660852594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f515c1593d0 con 0x7f515c071110 2023-10-30T13:07:54.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.788+0000 7f51629c5640 1 -- 172.21.15.125:0/660852594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f515c159210 con 0x7f515c071110 2023-10-30T13:07:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.792+0000 7f5158ff9640 1 -- 172.21.15.125:0/660852594 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f51500c3410 con 0x7f515c071110 2023-10-30T13:07:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.792+0000 7f5158ff9640 1 --2- 172.21.15.125:0/660852594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51480664d0 0x7f5148068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.792+0000 7f5158ff9640 1 -- 172.21.15.125:0/660852594 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f5150121fe0 con 0x7f515c071110 2023-10-30T13:07:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.792+0000 7f515bfff640 1 --2- 172.21.15.125:0/660852594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51480664d0 0x7f5148068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:54.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.792+0000 7f515bfff640 1 --2- 172.21.15.125:0/660852594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51480664d0 0x7f5148068990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f514c0079c0 tx=0x7f514c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:54.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.796+0000 7f5158ff9640 1 -- 172.21.15.125:0/660852594 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f51500eed80 con 0x7f515c071110 2023-10-30T13:07:54.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.976+0000 7f51629c5640 1 -- 172.21.15.125:0/660852594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f515c04e170 con 0x7f515c071110 2023-10-30T13:07:54.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.976+0000 7f5158ff9640 1 -- 172.21.15.125:0/660852594 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+6549 (secure 0 0 0) 0x7f51500f23a0 con 0x7f515c071110 2023-10-30T13:07:54.981 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:54.982 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:40.986565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:active","state_seq":432,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:54.982 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 178 2023-10-30T13:07:54.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.976+0000 7f51629c5640 1 -- 172.21.15.125:0/660852594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51480664d0 msgr2=0x7f5148068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:54.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.976+0000 7f51629c5640 1 --2- 172.21.15.125:0/660852594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51480664d0 0x7f5148068990 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f514c0079c0 tx=0x7f514c008040 comp rx=0 tx=0).stop 2023-10-30T13:07:54.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.976+0000 7f51629c5640 1 -- 172.21.15.125:0/660852594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f515c071110 msgr2=0x7f515c159eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:54.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.976+0000 7f51629c5640 1 --2- 172.21.15.125:0/660852594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f515c071110 0x7f515c159eb0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f51500b3040 tx=0x7f515001bad0 comp rx=0 tx=0).stop 2023-10-30T13:07:54.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.976+0000 7f51629c5640 1 -- 172.21.15.125:0/660852594 shutdown_connections 2023-10-30T13:07:54.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.976+0000 7f51629c5640 1 --2- 172.21.15.125:0/660852594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f51480664d0 0x7f5148068990 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:54.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.976+0000 7f51629c5640 1 --2- 172.21.15.125:0/660852594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f515c0c6bd0 0x7f515c15ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:54.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.976+0000 7f51629c5640 1 --2- 172.21.15.125:0/660852594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f515c071110 0x7f515c159eb0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:54.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.976+0000 7f51629c5640 1 --2- 172.21.15.125:0/660852594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f515c06f7a0 0x7f515c0ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:54.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.976+0000 7f51629c5640 1 -- 172.21.15.125:0/660852594 >> 172.21.15.125:0/660852594 conn(0x7f515c0c7910 msgr2=0x7f515c158010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:54.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.976+0000 7f51629c5640 1 -- 172.21.15.125:0/660852594 shutdown_connections 2023-10-30T13:07:54.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:54.976+0000 7f51629c5640 1 -- 172.21.15.125:0/660852594 wait complete. 2023-10-30T13:07:55.003 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:07:57.003 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:57.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49ac5c5640 1 Processor -- start 2023-10-30T13:07:57.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49ac5c5640 1 -- start start 2023-10-30T13:07:57.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49ac5c5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f49a40c8600 0x7f49a4165230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:57.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49ac5c5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a40c8da0 con 0x7f49a4157080 2023-10-30T13:07:57.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49ac5c5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a406cb60 con 0x7f49a40c7990 2023-10-30T13:07:57.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49ac5c5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a4157460 con 0x7f49a40c8600 2023-10-30T13:07:57.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49aa361640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f49a4157080 0x7f49a4071b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34132/0 (socket says 172.21.15.125:34132) 2023-10-30T13:07:57.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49aa361640 1 -- 172.21.15.125:0/2015060685 learned_addr learned my addr 172.21.15.125:0/2015060685 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:07:57.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49aab62640 1 --2- 172.21.15.125:0/2015060685 >> v2:172.21.15.136:3300/0 conn(0x7f49a40c8600 0x7f49a4165230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:57.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 -- 172.21.15.125:0/2015060685 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1744018526 0 0) 0x7f49a40c8da0 con 0x7f49a4157080 2023-10-30T13:07:57.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 -- 172.21.15.125:0/2015060685 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4990003530 con 0x7f49a4157080 2023-10-30T13:07:57.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 -- 172.21.15.125:0/2015060685 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3077559951 0 0) 0x7f4990003530 con 0x7f49a4157080 2023-10-30T13:07:57.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 -- 172.21.15.125:0/2015060685 >> v2:172.21.15.136:3300/0 conn(0x7f49a40c8600 msgr2=0x7f49a4165230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:57.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 --2- 172.21.15.125:0/2015060685 >> v2:172.21.15.136:3300/0 conn(0x7f49a40c8600 0x7f49a4165230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:57.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 -- 172.21.15.125:0/2015060685 >> v1:172.21.15.136:6789/0 conn(0x7f49a40c7990 legacy=0x7f49a40c7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:57.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 -- 172.21.15.125:0/2015060685 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a406ce20 con 0x7f49a4157080 2023-10-30T13:07:57.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 -- 172.21.15.125:0/2015060685 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f49a0001680 con 0x7f49a4157080 2023-10-30T13:07:57.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 -- 172.21.15.125:0/2015060685 >> v1:172.21.15.125:6789/0 conn(0x7f49a4157080 legacy=0x7f49a4071b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:57.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 --2- 172.21.15.125:0/2015060685 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4990003980 0x7f4990003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:57.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 --2- 172.21.15.125:0/2015060685 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f49900047d0 0x7f4990004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:57.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 --2- 172.21.15.125:0/2015060685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49900052b0 0x7f499000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:57.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 -- 172.21.15.125:0/2015060685 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a4157460 con 0x7f49900052b0 2023-10-30T13:07:57.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 -- 172.21.15.125:0/2015060685 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499000c180 con 0x7f4990003980 2023-10-30T13:07:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a935f640 1 -- 172.21.15.125:0/2015060685 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499000c340 con 0x7f49900047d0 2023-10-30T13:07:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49aab62640 1 --2- 172.21.15.125:0/2015060685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49900052b0 0x7f499000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49aa361640 1 --2- 172.21.15.125:0/2015060685 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4990003980 0x7f4990003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.148+0000 7f49a9b60640 1 --2- 172.21.15.125:0/2015060685 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f49900047d0 0x7f4990004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49aab62640 1 -- 172.21.15.125:0/2015060685 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f49900047d0 msgr2=0x7f4990004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:57.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49aab62640 1 --2- 172.21.15.125:0/2015060685 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f49900047d0 0x7f4990004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:57.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49aab62640 1 -- 172.21.15.125:0/2015060685 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4990003980 msgr2=0x7f4990003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:57.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49aab62640 1 --2- 172.21.15.125:0/2015060685 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4990003980 0x7f4990003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:57.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49aab62640 1 -- 172.21.15.125:0/2015060685 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4990003530 con 0x7f49900052b0 2023-10-30T13:07:57.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49aab62640 1 -- 172.21.15.125:0/2015060685 reap_dead start 2023-10-30T13:07:57.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49aab62640 1 --2- 172.21.15.125:0/2015060685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49900052b0 0x7f499000b990 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f499c009f90 tx=0x7f499c01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd52ad2588841352 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:57.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49a935f640 1 -- 172.21.15.125:0/2015060685 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c003ba0 con 0x7f49900052b0 2023-10-30T13:07:57.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49a935f640 1 -- 172.21.15.125:0/2015060685 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499c00f040 con 0x7f49900052b0 2023-10-30T13:07:57.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49a935f640 1 -- 172.21.15.125:0/2015060685 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c0a5940 con 0x7f49900052b0 2023-10-30T13:07:57.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49ac5c5640 1 -- 172.21.15.125:0/2015060685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49900052b0 msgr2=0x7f499000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:57.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49ac5c5640 1 --2- 172.21.15.125:0/2015060685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49900052b0 0x7f499000b990 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f499c009f90 tx=0x7f499c01cc60 comp rx=0 tx=0).stop 2023-10-30T13:07:57.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49ac5c5640 1 -- 172.21.15.125:0/2015060685 shutdown_connections 2023-10-30T13:07:57.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49ac5c5640 1 --2- 172.21.15.125:0/2015060685 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49900052b0 0x7f499000b990 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:57.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.152+0000 7f49ac5c5640 1 -- 172.21.15.125:0/2015060685 >> 172.21.15.125:0/2015060685 conn(0x7f49a4069e90 msgr2=0x7f49a406a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:57.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49ac5c5640 1 -- 172.21.15.125:0/2015060685 shutdown_connections 2023-10-30T13:07:57.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49ac5c5640 1 -- 172.21.15.125:0/2015060685 wait complete. 2023-10-30T13:07:57.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49ac5c5640 1 Processor -- start 2023-10-30T13:07:57.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49ac5c5640 1 -- start start 2023-10-30T13:07:57.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49ac5c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49900052b0 0x7f49a40c6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:57.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49ac5c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f49a4145810 0x7f49a41f3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:57.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49ac5c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49a4165260 0x7f49a414d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:57.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49ac5c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a41f6140 con 0x7f49900052b0 2023-10-30T13:07:57.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49ac5c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a41f5830 con 0x7f49a4165260 2023-10-30T13:07:57.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49ac5c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a41446f0 con 0x7f49a4145810 2023-10-30T13:07:57.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49aa361640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49900052b0 0x7f49a40c6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:57.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49aa361640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49900052b0 0x7f49a40c6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41708/0 (socket says 172.21.15.125:41708) 2023-10-30T13:07:57.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49aa361640 1 -- 172.21.15.125:0/3909087721 learned_addr learned my addr 172.21.15.125:0/3909087721 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:57.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49a9b60640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f49a4145810 0x7f49a41f3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:57.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49aab62640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49a4165260 0x7f49a414d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:57.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49aa361640 1 -- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f49a4145810 msgr2=0x7f49a41f3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:57.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49aa361640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f49a4145810 0x7f49a41f3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:57.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49aa361640 1 -- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49a4165260 msgr2=0x7f49a414d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:57.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49aa361640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49a4165260 0x7f49a414d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:57.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49aa361640 1 -- 172.21.15.125:0/3909087721 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a40c6870 con 0x7f49900052b0 2023-10-30T13:07:57.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49aab62640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49a4165260 0x7f49a414d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:07:57.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49a9b60640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f49a4145810 0x7f49a41f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:07:57.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49aa361640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49900052b0 0x7f49a40c6210 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f49a00072c0 tx=0x7f49a0007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:57.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f499affd640 1 -- 172.21.15.125:0/3909087721 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a000f8e0 con 0x7f49900052b0 2023-10-30T13:07:57.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f499affd640 1 -- 172.21.15.125:0/3909087721 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a000fa40 con 0x7f49900052b0 2023-10-30T13:07:57.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f499affd640 1 -- 172.21.15.125:0/3909087721 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a000fdf0 con 0x7f49900052b0 2023-10-30T13:07:57.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.156+0000 7f49ac5c5640 1 -- 172.21.15.125:0/3909087721 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a40c6b50 con 0x7f49900052b0 2023-10-30T13:07:57.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.160+0000 7f49ac5c5640 1 -- 172.21.15.125:0/3909087721 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a41f5d60 con 0x7f49900052b0 2023-10-30T13:07:57.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.160+0000 7f49ac5c5640 1 -- 172.21.15.125:0/3909087721 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a41f5f20 con 0x7f49900052b0 2023-10-30T13:07:57.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.160+0000 7f499affd640 1 -- 172.21.15.125:0/3909087721 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f49a001d460 con 0x7f49900052b0 2023-10-30T13:07:57.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.160+0000 7f499affd640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49880664a0 0x7f4988068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:57.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.160+0000 7f499affd640 1 -- 172.21.15.125:0/3909087721 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f49a0084740 con 0x7f49900052b0 2023-10-30T13:07:57.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.160+0000 7f49a9b60640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49880664a0 0x7f4988068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:57.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.164+0000 7f49a9b60640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49880664a0 0x7f4988068960 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f49a41efef0 tx=0x7f4994009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:57.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.164+0000 7f499affd640 1 -- 172.21.15.125:0/3909087721 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f49a00513f0 con 0x7f49900052b0 2023-10-30T13:07:57.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.344+0000 7f49ac5c5640 1 -- 172.21.15.125:0/3909087721 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49a406c560 con 0x7f49900052b0 2023-10-30T13:07:57.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.344+0000 7f499affd640 1 -- 172.21.15.125:0/3909087721 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+6549 (secure 0 0 0) 0x7f49a0054a10 con 0x7f49900052b0 2023-10-30T13:07:57.352 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:57.352 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:40.986565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:active","state_seq":432,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:57.352 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 178 2023-10-30T13:07:57.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.348+0000 7f49ac5c5640 1 -- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49880664a0 msgr2=0x7f4988068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:57.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.348+0000 7f49ac5c5640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49880664a0 0x7f4988068960 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f49a41efef0 tx=0x7f4994009040 comp rx=0 tx=0).stop 2023-10-30T13:07:57.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.348+0000 7f49ac5c5640 1 -- 172.21.15.125:0/3909087721 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49900052b0 msgr2=0x7f49a40c6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:57.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.348+0000 7f49ac5c5640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49900052b0 0x7f49a40c6210 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f49a00072c0 tx=0x7f49a0007780 comp rx=0 tx=0).stop 2023-10-30T13:07:57.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.348+0000 7f49ac5c5640 1 -- 172.21.15.125:0/3909087721 shutdown_connections 2023-10-30T13:07:57.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.348+0000 7f49ac5c5640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f49880664a0 0x7f4988068960 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:57.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.348+0000 7f49ac5c5640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f49a4165260 0x7f49a414d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:57.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.348+0000 7f49ac5c5640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f49a4145810 0x7f49a41f3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:57.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.348+0000 7f49ac5c5640 1 --2- 172.21.15.125:0/3909087721 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f49900052b0 0x7f49a40c6210 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:57.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.348+0000 7f49ac5c5640 1 -- 172.21.15.125:0/3909087721 >> 172.21.15.125:0/3909087721 conn(0x7f49a4069e90 msgr2=0x7f49a4155a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:57.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.348+0000 7f49ac5c5640 1 -- 172.21.15.125:0/3909087721 shutdown_connections 2023-10-30T13:07:57.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:57.348+0000 7f49ac5c5640 1 -- 172.21.15.125:0/3909087721 wait complete. 2023-10-30T13:07:57.372 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:07:59.373 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:07:59.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.512+0000 7f6ab6b3e640 1 Processor -- start 2023-10-30T13:07:59.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.512+0000 7f6ab6b3e640 1 -- start start 2023-10-30T13:07:59.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.512+0000 7f6ab6b3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ab00c6b80 0x7f6ab00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:59.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.512+0000 7f6ab6b3e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6ab00c6f60 0x7f6ab00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:59.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.516+0000 7f6ab6b3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab00c0a50 con 0x7f6ab00c8430 2023-10-30T13:07:59.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.516+0000 7f6ab6b3e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ab006f6e0 con 0x7f6ab00c6f60 2023-10-30T13:07:59.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.516+0000 7f6ab6b3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab006f9c0 con 0x7f6ab00c6b80 2023-10-30T13:07:59.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.516+0000 7f6aaffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6ab00c6f60 0x7f6ab00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:59.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.516+0000 7f6aaffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6ab00c6f60 0x7f6ab00c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41714/0 (socket says 172.21.15.125:41714) 2023-10-30T13:07:59.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.516+0000 7f6aaffff640 1 -- 172.21.15.125:0/2677531572 learned_addr learned my addr 172.21.15.125:0/2677531572 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.516+0000 7f6ab48da640 1 --2- 172.21.15.125:0/2677531572 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ab00c6b80 0x7f6ab00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.516+0000 7f6aaffff640 1 -- 172.21.15.125:0/2677531572 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ab00c6b80 msgr2=0x7f6ab00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:07:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.516+0000 7f6aaffff640 1 --2- 172.21.15.125:0/2677531572 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ab00c6b80 0x7f6ab00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:59.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.516+0000 7f6aaffff640 1 -- 172.21.15.125:0/2677531572 >> v1:172.21.15.136:6789/0 conn(0x7f6ab00c8430 legacy=0x7f6ab013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:59.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.516+0000 7f6aaffff640 1 -- 172.21.15.125:0/2677531572 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab006fca0 con 0x7f6ab00c6f60 2023-10-30T13:07:59.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.516+0000 7f6aaffff640 1 --2- 172.21.15.125:0/2677531572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ab00c6f60 0x7f6ab00c0400 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009870 tx=0x7f6a9c01c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d3804021c40dd09 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:59.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.516+0000 7f6aaf7fe640 1 -- 172.21.15.125:0/2677531572 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c0ab650 con 0x7f6ab00c6f60 2023-10-30T13:07:59.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6aaf7fe640 1 -- 172.21.15.125:0/2677531572 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9c0ab7b0 con 0x7f6ab00c6f60 2023-10-30T13:07:59.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6aaf7fe640 1 -- 172.21.15.125:0/2677531572 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c0abb40 con 0x7f6ab00c6f60 2023-10-30T13:07:59.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/2677531572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ab00c6f60 msgr2=0x7f6ab00c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:59.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 --2- 172.21.15.125:0/2677531572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ab00c6f60 0x7f6ab00c0400 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c009870 tx=0x7f6a9c01c540 comp rx=0 tx=0).stop 2023-10-30T13:07:59.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/2677531572 shutdown_connections 2023-10-30T13:07:59.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 --2- 172.21.15.125:0/2677531572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ab00c6f60 0x7f6ab00c0400 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:59.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 --2- 172.21.15.125:0/2677531572 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ab00c6b80 0x7f6ab00bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:59.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/2677531572 >> 172.21.15.125:0/2677531572 conn(0x7f6ab00c7910 msgr2=0x7f6ab00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:59.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/2677531572 shutdown_connections 2023-10-30T13:07:59.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/2677531572 wait complete. 2023-10-30T13:07:59.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 Processor -- start 2023-10-30T13:07:59.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 -- start start 2023-10-30T13:07:59.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ab00c6b80 0x7f6ab01f3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:59.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ab00c6f60 0x7f6ab013dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:59.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ab00c8430 0x7f6ab014f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:59.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab01f3a80 con 0x7f6ab00c6b80 2023-10-30T13:07:59.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab013b250 con 0x7f6ab00c6f60 2023-10-30T13:07:59.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab6b3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab013ac90 con 0x7f6ab00c8430 2023-10-30T13:07:59.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab48da640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ab00c6b80 0x7f6ab01f3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:59.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab48da640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ab00c6b80 0x7f6ab01f3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41716/0 (socket says 172.21.15.125:41716) 2023-10-30T13:07:59.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab48da640 1 -- 172.21.15.125:0/1320276984 learned_addr learned my addr 172.21.15.125:0/1320276984 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:07:59.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6ab50db640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ab00c8430 0x7f6ab014f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:59.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.520+0000 7f6aaffff640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ab00c6f60 0x7f6ab013dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:59.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6ab48da640 1 -- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ab00c8430 msgr2=0x7f6ab014f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:59.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6ab48da640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ab00c8430 0x7f6ab014f400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:59.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6ab48da640 1 -- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ab00c6f60 msgr2=0x7f6ab013dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:59.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6ab48da640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ab00c6f60 0x7f6ab013dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:59.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6ab48da640 1 -- 172.21.15.125:0/1320276984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9c0094d0 con 0x7f6ab00c6b80 2023-10-30T13:07:59.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6aaffff640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ab00c6f60 0x7f6ab013dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:07:59.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6ab48da640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ab00c6b80 0x7f6ab01f3f10 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400e930 tx=0x7f6aa400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:59.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6aad7fa640 1 -- 172.21.15.125:0/1320276984 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4009690 con 0x7f6ab00c6b80 2023-10-30T13:07:59.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6aad7fa640 1 -- 172.21.15.125:0/1320276984 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa400ca00 con 0x7f6ab00c6b80 2023-10-30T13:07:59.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/1320276984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab013af70 con 0x7f6ab00c6b80 2023-10-30T13:07:59.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6aad7fa640 1 -- 172.21.15.125:0/1320276984 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa400cd70 con 0x7f6ab00c6b80 2023-10-30T13:07:59.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/1320276984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab006a800 con 0x7f6ab00c6b80 2023-10-30T13:07:59.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/1320276984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab004e170 con 0x7f6ab00c6b80 2023-10-30T13:07:59.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6aad7fa640 1 -- 172.21.15.125:0/1320276984 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6aa4013470 con 0x7f6ab00c6b80 2023-10-30T13:07:59.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6aad7fa640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6a940664d0 0x7f6a94068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:07:59.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.524+0000 7f6aad7fa640 1 -- 172.21.15.125:0/1320276984 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 7544+0+0 (secure 0 0 0) 0x7f6aa40847d0 con 0x7f6ab00c6b80 2023-10-30T13:07:59.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.528+0000 7f6aaffff640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6a940664d0 0x7f6a94068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:07:59.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.528+0000 7f6aaffff640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6a940664d0 0x7f6a94068990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c0af000 tx=0x7f6a9c0a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:07:59.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.528+0000 7f6aad7fa640 1 -- 172.21.15.125:0/1320276984 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6aa40515f0 con 0x7f6ab00c6b80 2023-10-30T13:07:59.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.712+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/1320276984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ab006b8e0 con 0x7f6ab00c6b80 2023-10-30T13:07:59.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.716+0000 7f6aad7fa640 1 -- 172.21.15.125:0/1320276984 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+6549 (secure 0 0 0) 0x7f6aa4054c10 con 0x7f6ab00c6b80 2023-10-30T13:07:59.720 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:07:59.720 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8529,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":178}],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:07:40.986565+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":5234,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5234":{"gid":5234,"name":"d","rank":0,"incarnation":172,"state":"up:active","state_seq":432,"addr":"172.21.15.136:6835/2194892271","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2194892271},{"type":"v1","addr":"172.21.15.136:6835","nonce":2194892271}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:07:59.721 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 178 2023-10-30T13:07:59.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.716+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6a940664d0 msgr2=0x7f6a94068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:59.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.716+0000 7f6ab6b3e640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6a940664d0 0x7f6a94068990 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c0af000 tx=0x7f6a9c0a7000 comp rx=0 tx=0).stop 2023-10-30T13:07:59.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.716+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/1320276984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ab00c6b80 msgr2=0x7f6ab01f3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:07:59.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.716+0000 7f6ab6b3e640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ab00c6b80 0x7f6ab01f3f10 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f6aa400e930 tx=0x7f6aa400edf0 comp rx=0 tx=0).stop 2023-10-30T13:07:59.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.716+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/1320276984 shutdown_connections 2023-10-30T13:07:59.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.716+0000 7f6ab6b3e640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6a940664d0 0x7f6a94068990 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:59.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.716+0000 7f6ab6b3e640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ab00c8430 0x7f6ab014f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:59.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.716+0000 7f6ab6b3e640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ab00c6f60 0x7f6ab013dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:59.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.716+0000 7f6ab6b3e640 1 --2- 172.21.15.125:0/1320276984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ab00c6b80 0x7f6ab01f3f10 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:07:59.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.716+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/1320276984 >> 172.21.15.125:0/1320276984 conn(0x7f6ab00c7910 msgr2=0x7f6ab0136570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:07:59.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.716+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/1320276984 shutdown_connections 2023-10-30T13:07:59.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:07:59.716+0000 7f6ab6b3e640 1 -- 172.21.15.125:0/1320276984 wait complete. 2023-10-30T13:07:59.741 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:08:01.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:01.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.884+0000 7f9c4ea38640 1 Processor -- start 2023-10-30T13:08:01.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.884+0000 7f9c4ea38640 1 -- start start 2023-10-30T13:08:01.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.884+0000 7f9c4ea38640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9c481538f0 0x7f9c48159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:01.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.884+0000 7f9c4ea38640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c480ca6a0 con 0x7f9c4815cd50 2023-10-30T13:08:01.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.884+0000 7f9c4ea38640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c4815d300 con 0x7f9c480cefa0 2023-10-30T13:08:01.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.884+0000 7f9c4ea38640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c4815d5e0 con 0x7f9c481538f0 2023-10-30T13:08:01.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.884+0000 7f9c4da36640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9c481538f0 0x7f9c48159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:01.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.884+0000 7f9c4da36640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9c481538f0 0x7f9c48159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49258/0 (socket says 172.21.15.125:49258) 2023-10-30T13:08:01.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.884+0000 7f9c4da36640 1 -- 172.21.15.125:0/3956154780 learned_addr learned my addr 172.21.15.125:0/3956154780 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:01.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.884+0000 7f9c4da36640 1 -- 172.21.15.125:0/3956154780 >> v1:172.21.15.136:6789/0 conn(0x7f9c480cefa0 legacy=0x7f9c481640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:01.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.884+0000 7f9c4da36640 1 -- 172.21.15.125:0/3956154780 >> v1:172.21.15.125:6789/0 conn(0x7f9c4815cd50 legacy=0x7f9c480c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:01.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.884+0000 7f9c4da36640 1 -- 172.21.15.125:0/3956154780 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4815d8c0 con 0x7f9c481538f0 2023-10-30T13:08:01.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.888+0000 7f9c4da36640 1 --2- 172.21.15.125:0/3956154780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c481538f0 0x7f9c48159fb0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009870 tx=0x7f9c4401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7dcc8fde0af8155 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:01.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.888+0000 7f9c4ca34640 1 -- 172.21.15.125:0/3956154780 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c440a89f0 con 0x7f9c481538f0 2023-10-30T13:08:01.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.888+0000 7f9c4ca34640 1 -- 172.21.15.125:0/3956154780 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c440a8b50 con 0x7f9c481538f0 2023-10-30T13:08:01.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.888+0000 7f9c4ca34640 1 -- 172.21.15.125:0/3956154780 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c440b0440 con 0x7f9c481538f0 2023-10-30T13:08:01.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.888+0000 7f9c4ea38640 1 -- 172.21.15.125:0/3956154780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c481538f0 msgr2=0x7f9c48159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:01.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.888+0000 7f9c4ea38640 1 --2- 172.21.15.125:0/3956154780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c481538f0 0x7f9c48159fb0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f9c44009870 tx=0x7f9c4401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:08:01.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.888+0000 7f9c4ea38640 1 -- 172.21.15.125:0/3956154780 shutdown_connections 2023-10-30T13:08:01.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.888+0000 7f9c4ea38640 1 --2- 172.21.15.125:0/3956154780 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c481538f0 0x7f9c48159fb0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:01.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.888+0000 7f9c4ea38640 1 -- 172.21.15.125:0/3956154780 >> 172.21.15.125:0/3956154780 conn(0x7f9c4806a640 msgr2=0x7f9c48065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:01.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.888+0000 7f9c4ea38640 1 -- 172.21.15.125:0/3956154780 shutdown_connections 2023-10-30T13:08:01.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.888+0000 7f9c4ea38640 1 -- 172.21.15.125:0/3956154780 wait complete. 2023-10-30T13:08:01.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.888+0000 7f9c4ea38640 1 Processor -- start 2023-10-30T13:08:01.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.888+0000 7f9c4ea38640 1 -- start start 2023-10-30T13:08:01.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4ea38640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c480cefa0 0x7f9c480c6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:01.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4ea38640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c481538f0 0x7f9c4813e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:01.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4ea38640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c4815cd50 0x7f9c4813ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:01.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4ea38640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c48159800 con 0x7f9c481538f0 2023-10-30T13:08:01.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4ea38640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c48157dc0 con 0x7f9c480cefa0 2023-10-30T13:08:01.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4ea38640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c4813f2b0 con 0x7f9c4815cd50 2023-10-30T13:08:01.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4d235640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c481538f0 0x7f9c4813e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:01.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4d235640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c481538f0 0x7f9c4813e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49272/0 (socket says 172.21.15.125:49272) 2023-10-30T13:08:01.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4d235640 1 -- 172.21.15.125:0/1897650242 learned_addr learned my addr 172.21.15.125:0/1897650242 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:01.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4da36640 1 --2- 172.21.15.125:0/1897650242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c480cefa0 0x7f9c480c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:01.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4e237640 1 --2- 172.21.15.125:0/1897650242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c4815cd50 0x7f9c4813ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:01.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4d235640 1 -- 172.21.15.125:0/1897650242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c4815cd50 msgr2=0x7f9c4813ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:01.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4d235640 1 --2- 172.21.15.125:0/1897650242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c4815cd50 0x7f9c4813ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:01.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4d235640 1 -- 172.21.15.125:0/1897650242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c480cefa0 msgr2=0x7f9c480c6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:01.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4d235640 1 --2- 172.21.15.125:0/1897650242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c480cefa0 0x7f9c480c6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:01.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4d235640 1 -- 172.21.15.125:0/1897650242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c440094d0 con 0x7f9c481538f0 2023-10-30T13:08:01.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4d235640 1 --2- 172.21.15.125:0/1897650242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c481538f0 0x7f9c4813e830 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800c3c0 tx=0x7f9c3800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:01.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c3e7fc640 1 -- 172.21.15.125:0/1897650242 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c380052d0 con 0x7f9c481538f0 2023-10-30T13:08:01.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c3e7fc640 1 -- 172.21.15.125:0/1897650242 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c38005430 con 0x7f9c481538f0 2023-10-30T13:08:01.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c3e7fc640 1 -- 172.21.15.125:0/1897650242 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c38003840 con 0x7f9c481538f0 2023-10-30T13:08:01.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4ea38640 1 -- 172.21.15.125:0/1897650242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c48158020 con 0x7f9c481538f0 2023-10-30T13:08:01.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4ea38640 1 -- 172.21.15.125:0/1897650242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c48159520 con 0x7f9c481538f0 2023-10-30T13:08:01.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c4ea38640 1 -- 172.21.15.125:0/1897650242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c4804e170 con 0x7f9c481538f0 2023-10-30T13:08:01.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.892+0000 7f9c3e7fc640 1 -- 172.21.15.125:0/1897650242 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9c380039a0 con 0x7f9c481538f0 2023-10-30T13:08:01.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.896+0000 7f9c3e7fc640 1 --2- 172.21.15.125:0/1897650242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c2c0664d0 0x7f9c2c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:01.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.896+0000 7f9c3e7fc640 1 -- 172.21.15.125:0/1897650242 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f9c38083bf0 con 0x7f9c481538f0 2023-10-30T13:08:01.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.896+0000 7f9c4da36640 1 --2- 172.21.15.125:0/1897650242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c2c0664d0 0x7f9c2c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:01.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.896+0000 7f9c4da36640 1 --2- 172.21.15.125:0/1897650242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c2c0664d0 0x7f9c2c068990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9c440b2000 tx=0x7f9c44008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:01.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:01.900+0000 7f9c3e7fc640 1 -- 172.21.15.125:0/1897650242 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9c38050850 con 0x7f9c481538f0 2023-10-30T13:08:02.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.076+0000 7f9c4ea38640 1 -- 172.21.15.125:0/1897650242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c480ca880 con 0x7f9c481538f0 2023-10-30T13:08:02.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.080+0000 7f9c3e7fc640 1 -- 172.21.15.125:0/1897650242 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5751 (secure 0 0 0) 0x7f9c38053e70 con 0x7f9c481538f0 2023-10-30T13:08:02.085 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:02.085 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:00.628485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:02.085 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 179 2023-10-30T13:08:02.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.080+0000 7f9c4ea38640 1 -- 172.21.15.125:0/1897650242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c2c0664d0 msgr2=0x7f9c2c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:02.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.080+0000 7f9c4ea38640 1 --2- 172.21.15.125:0/1897650242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c2c0664d0 0x7f9c2c068990 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f9c440b2000 tx=0x7f9c44008040 comp rx=0 tx=0).stop 2023-10-30T13:08:02.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.080+0000 7f9c4ea38640 1 -- 172.21.15.125:0/1897650242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c481538f0 msgr2=0x7f9c4813e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:02.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.080+0000 7f9c4ea38640 1 --2- 172.21.15.125:0/1897650242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c481538f0 0x7f9c4813e830 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f9c3800c3c0 tx=0x7f9c3800c880 comp rx=0 tx=0).stop 2023-10-30T13:08:02.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.080+0000 7f9c4ea38640 1 -- 172.21.15.125:0/1897650242 shutdown_connections 2023-10-30T13:08:02.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.080+0000 7f9c4ea38640 1 --2- 172.21.15.125:0/1897650242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9c2c0664d0 0x7f9c2c068990 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:02.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.080+0000 7f9c4ea38640 1 --2- 172.21.15.125:0/1897650242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9c4815cd50 0x7f9c4813ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:02.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.080+0000 7f9c4ea38640 1 --2- 172.21.15.125:0/1897650242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9c481538f0 0x7f9c4813e830 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:02.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.080+0000 7f9c4ea38640 1 --2- 172.21.15.125:0/1897650242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9c480cefa0 0x7f9c480c6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:02.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.080+0000 7f9c4ea38640 1 -- 172.21.15.125:0/1897650242 >> 172.21.15.125:0/1897650242 conn(0x7f9c4806a640 msgr2=0x7f9c480c4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:02.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.080+0000 7f9c4ea38640 1 -- 172.21.15.125:0/1897650242 shutdown_connections 2023-10-30T13:08:02.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.080+0000 7f9c4ea38640 1 -- 172.21.15.125:0/1897650242 wait complete. 2023-10-30T13:08:02.105 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T13:08:02.105 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:08:02.105 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:02.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.236+0000 7f155852a640 1 Processor -- start 2023-10-30T13:08:02.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.236+0000 7f155852a640 1 -- start start 2023-10-30T13:08:02.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f155852a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15500c6b80 0x7f15500bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:02.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f155852a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f155013f4a0 0x7f15500c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:02.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f155852a640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15500c0a50 con 0x7f15500c6f60 2023-10-30T13:08:02.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f155852a640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155006f6e0 con 0x7f15500c6b80 2023-10-30T13:08:02.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f155852a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155006f9c0 con 0x7f155013f4a0 2023-10-30T13:08:02.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f15562c6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15500c6b80 0x7f15500bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:02.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f1555ac5640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f15500c6f60 0x7f155013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38274/0 (socket says 172.21.15.125:38274) 2023-10-30T13:08:02.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f1555ac5640 1 -- 172.21.15.125:0/1878969876 learned_addr learned my addr 172.21.15.125:0/1878969876 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:02.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f15562c6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15500c6b80 0x7f15500bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49282/0 (socket says 172.21.15.125:49282) 2023-10-30T13:08:02.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f1556ac7640 1 --2- 172.21.15.125:0/1878969876 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f155013f4a0 0x7f15500c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:02.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f15562c6640 1 -- 172.21.15.125:0/1878969876 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f155013f4a0 msgr2=0x7f15500c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:02.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f15562c6640 1 --2- 172.21.15.125:0/1878969876 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f155013f4a0 0x7f15500c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:02.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f15562c6640 1 -- 172.21.15.125:0/1878969876 >> v1:172.21.15.125:6789/0 conn(0x7f15500c6f60 legacy=0x7f155013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:02.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f15562c6640 1 -- 172.21.15.125:0/1878969876 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155006fca0 con 0x7f15500c6b80 2023-10-30T13:08:02.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.240+0000 7f15552c4640 1 -- 172.21.15.125:0/1878969876 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547759141 0 0) 0x7f15500c0a50 con 0x7f15500c6f60 2023-10-30T13:08:02.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.244+0000 7f15562c6640 1 --2- 172.21.15.125:0/1878969876 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15500c6b80 0x7f15500bfec0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f154c0097c0 tx=0x7f154c01c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6844aa7647155b4e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:02.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.244+0000 7f15552c4640 1 -- 172.21.15.125:0/1878969876 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c0ad640 con 0x7f15500c6b80 2023-10-30T13:08:02.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.244+0000 7f15552c4640 1 -- 172.21.15.125:0/1878969876 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154c0ad7a0 con 0x7f15500c6b80 2023-10-30T13:08:02.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.244+0000 7f15552c4640 1 -- 172.21.15.125:0/1878969876 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c0adb30 con 0x7f15500c6b80 2023-10-30T13:08:02.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.244+0000 7f155852a640 1 -- 172.21.15.125:0/1878969876 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15500c6b80 msgr2=0x7f15500bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:02.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.244+0000 7f155852a640 1 --2- 172.21.15.125:0/1878969876 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15500c6b80 0x7f15500bfec0 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f154c0097c0 tx=0x7f154c01c480 comp rx=0 tx=0).stop 2023-10-30T13:08:02.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.244+0000 7f155852a640 1 -- 172.21.15.125:0/1878969876 shutdown_connections 2023-10-30T13:08:02.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.244+0000 7f155852a640 1 --2- 172.21.15.125:0/1878969876 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f155013f4a0 0x7f15500c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:02.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.244+0000 7f155852a640 1 --2- 172.21.15.125:0/1878969876 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15500c6b80 0x7f15500bfec0 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:02.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.244+0000 7f155852a640 1 -- 172.21.15.125:0/1878969876 >> 172.21.15.125:0/1878969876 conn(0x7f15500c7910 msgr2=0x7f15500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:02.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.244+0000 7f155852a640 1 -- 172.21.15.125:0/1878969876 shutdown_connections 2023-10-30T13:08:02.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.244+0000 7f155852a640 1 -- 172.21.15.125:0/1878969876 wait complete. 2023-10-30T13:08:02.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.244+0000 7f155852a640 1 Processor -- start 2023-10-30T13:08:02.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.244+0000 7f155852a640 1 -- start start 2023-10-30T13:08:02.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f155852a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15500c6b80 0x7f155013c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:02.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f155852a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15500c6f60 0x7f15501556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:02.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f155852a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155013f4a0 0x7f155015ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:02.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f155852a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15501f2a20 con 0x7f155013f4a0 2023-10-30T13:08:02.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f155852a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15501f1b90 con 0x7f15500c6f60 2023-10-30T13:08:02.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f155852a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15501f1da0 con 0x7f15500c6b80 2023-10-30T13:08:02.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f1556ac7640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155013f4a0 0x7f155015ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:02.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f1556ac7640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155013f4a0 0x7f155015ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49290/0 (socket says 172.21.15.125:49290) 2023-10-30T13:08:02.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f1556ac7640 1 -- 172.21.15.125:0/2755061899 learned_addr learned my addr 172.21.15.125:0/2755061899 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:02.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f15562c6640 1 --2- 172.21.15.125:0/2755061899 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15500c6b80 0x7f155013c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:02.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f1555ac5640 1 --2- 172.21.15.125:0/2755061899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15500c6f60 0x7f15501556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:02.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f1556ac7640 1 -- 172.21.15.125:0/2755061899 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15500c6b80 msgr2=0x7f155013c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:02.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f1556ac7640 1 --2- 172.21.15.125:0/2755061899 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15500c6b80 0x7f155013c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:02.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f1556ac7640 1 -- 172.21.15.125:0/2755061899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15500c6f60 msgr2=0x7f15501556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:02.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f1556ac7640 1 --2- 172.21.15.125:0/2755061899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15500c6f60 0x7f15501556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:02.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f1556ac7640 1 -- 172.21.15.125:0/2755061899 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c009420 con 0x7f155013f4a0 2023-10-30T13:08:02.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f1556ac7640 1 --2- 172.21.15.125:0/2755061899 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155013f4a0 0x7f155015ea00 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f1548007e60 tx=0x7f154800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:02.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f1546ffd640 1 -- 172.21.15.125:0/2755061899 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1548014420 con 0x7f155013f4a0 2023-10-30T13:08:02.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f1546ffd640 1 -- 172.21.15.125:0/2755061899 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15480145a0 con 0x7f155013f4a0 2023-10-30T13:08:02.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f155852a640 1 -- 172.21.15.125:0/2755061899 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15501f2080 con 0x7f155013f4a0 2023-10-30T13:08:02.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f1546ffd640 1 -- 172.21.15.125:0/2755061899 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1548014950 con 0x7f155013f4a0 2023-10-30T13:08:02.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f155852a640 1 -- 172.21.15.125:0/2755061899 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15501f2640 con 0x7f155013f4a0 2023-10-30T13:08:02.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f155852a640 1 -- 172.21.15.125:0/2755061899 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1550072560 con 0x7f155013f4a0 2023-10-30T13:08:02.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.248+0000 7f1546ffd640 1 -- 172.21.15.125:0/2755061899 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1548014ab0 con 0x7f155013f4a0 2023-10-30T13:08:02.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.252+0000 7f1546ffd640 1 --2- 172.21.15.125:0/2755061899 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15340664a0 0x7f1534068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:02.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.252+0000 7f1546ffd640 1 -- 172.21.15.125:0/2755061899 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f1548085060 con 0x7f155013f4a0 2023-10-30T13:08:02.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.252+0000 7f15562c6640 1 --2- 172.21.15.125:0/2755061899 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15340664a0 0x7f1534068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:02.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.252+0000 7f15562c6640 1 --2- 172.21.15.125:0/2755061899 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15340664a0 0x7f1534068960 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f154c0af000 tx=0x7f154c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:02.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.256+0000 7f1546ffd640 1 -- 172.21.15.125:0/2755061899 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f15480b6090 con 0x7f155013f4a0 2023-10-30T13:08:02.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.436+0000 7f155852a640 1 -- 172.21.15.125:0/2755061899 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f155004e170 con 0x7f155013f4a0 2023-10-30T13:08:02.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.436+0000 7f1546ffd640 1 -- 172.21.15.125:0/2755061899 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5751 (secure 0 0 0) 0x7f1548055750 con 0x7f155013f4a0 2023-10-30T13:08:02.441 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:02.441 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:00.628485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:02.441 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 179 2023-10-30T13:08:02.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.436+0000 7f155852a640 1 -- 172.21.15.125:0/2755061899 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15340664a0 msgr2=0x7f1534068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:02.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.436+0000 7f155852a640 1 --2- 172.21.15.125:0/2755061899 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15340664a0 0x7f1534068960 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f154c0af000 tx=0x7f154c008040 comp rx=0 tx=0).stop 2023-10-30T13:08:02.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.436+0000 7f155852a640 1 -- 172.21.15.125:0/2755061899 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155013f4a0 msgr2=0x7f155015ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:02.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.436+0000 7f155852a640 1 --2- 172.21.15.125:0/2755061899 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155013f4a0 0x7f155015ea00 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f1548007e60 tx=0x7f154800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:08:02.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.436+0000 7f155852a640 1 -- 172.21.15.125:0/2755061899 shutdown_connections 2023-10-30T13:08:02.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.436+0000 7f155852a640 1 --2- 172.21.15.125:0/2755061899 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15340664a0 0x7f1534068960 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:02.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.436+0000 7f155852a640 1 --2- 172.21.15.125:0/2755061899 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f155013f4a0 0x7f155015ea00 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:02.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.436+0000 7f155852a640 1 --2- 172.21.15.125:0/2755061899 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f15500c6f60 0x7f15501556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:02.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.436+0000 7f155852a640 1 --2- 172.21.15.125:0/2755061899 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15500c6b80 0x7f155013c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:02.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.436+0000 7f155852a640 1 -- 172.21.15.125:0/2755061899 >> 172.21.15.125:0/2755061899 conn(0x7f15500c7910 msgr2=0x7f15500c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:02.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.436+0000 7f155852a640 1 -- 172.21.15.125:0/2755061899 shutdown_connections 2023-10-30T13:08:02.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:02.436+0000 7f155852a640 1 -- 172.21.15.125:0/2755061899 wait complete. 2023-10-30T13:08:02.462 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 179, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 179, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6017": { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 170, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 251 }, "gid_8301": { "addr": "172.21.15.125:6837/2540063352", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2540063352, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2540063352, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8301, "incarnation": 168, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_8529": { "addr": "172.21.15.125:6836/3950044434", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3950044434, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3950044434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8529, "incarnation": 179, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 76, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:08:00.628485+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8529, "mds_1": 8301, "mds_2": 6017 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:08:04.463 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:04.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.604+0000 7f0b6333e640 1 Processor -- start 2023-10-30T13:08:04.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.604+0000 7f0b6333e640 1 -- start start 2023-10-30T13:08:04.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.604+0000 7f0b6333e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0b5c070a00 0x7f0b5c070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:04.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.604+0000 7f0b6333e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b5c069e90 0x7f0b5c06a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:04.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.604+0000 7f0b6333e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b5c0713b0 con 0x7f0b5c1539a0 2023-10-30T13:08:04.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.604+0000 7f0b6333e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b5c06fbb0 con 0x7f0b5c070a00 2023-10-30T13:08:04.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.604+0000 7f0b6333e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c153d80 con 0x7f0b5c069e90 2023-10-30T13:08:04.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b610da640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0b5c1539a0 0x7f0b5c15a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38282/0 (socket says 172.21.15.125:38282) 2023-10-30T13:08:04.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b610da640 1 -- 172.21.15.125:0/2878289193 learned_addr learned my addr 172.21.15.125:0/2878289193 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:04.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b618db640 1 --2- 172.21.15.125:0/2878289193 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b5c069e90 0x7f0b5c06a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:04.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b608d9640 1 --2- 172.21.15.125:0/2878289193 >> v2:172.21.15.136:3300/0 conn(0x7f0b5c070a00 0x7f0b5c070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:04.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 -- 172.21.15.125:0/2878289193 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2474127713 0 0) 0x7f0b5c0713b0 con 0x7f0b5c1539a0 2023-10-30T13:08:04.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 -- 172.21.15.125:0/2878289193 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b48003530 con 0x7f0b5c1539a0 2023-10-30T13:08:04.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 -- 172.21.15.125:0/2878289193 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1554540386 0 0) 0x7f0b48003530 con 0x7f0b5c1539a0 2023-10-30T13:08:04.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 -- 172.21.15.125:0/2878289193 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b5c069e90 msgr2=0x7f0b5c06a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:04.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 --2- 172.21.15.125:0/2878289193 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b5c069e90 0x7f0b5c06a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:04.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 -- 172.21.15.125:0/2878289193 >> v2:172.21.15.136:3300/0 conn(0x7f0b5c070a00 msgr2=0x7f0b5c070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:04.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 --2- 172.21.15.125:0/2878289193 >> v2:172.21.15.136:3300/0 conn(0x7f0b5c070a00 0x7f0b5c070e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:04.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 -- 172.21.15.125:0/2878289193 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5c0c5c50 con 0x7f0b5c1539a0 2023-10-30T13:08:04.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 -- 172.21.15.125:0/2878289193 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f0b58001680 con 0x7f0b5c1539a0 2023-10-30T13:08:04.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 -- 172.21.15.125:0/2878289193 >> v1:172.21.15.125:6789/0 conn(0x7f0b5c1539a0 legacy=0x7f0b5c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:04.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 --2- 172.21.15.125:0/2878289193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b48003980 0x7f0b48003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:04.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 --2- 172.21.15.125:0/2878289193 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b5c069e90 0x7f0b480047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:04.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 --2- 172.21.15.125:0/2878289193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b48004d10 0x7f0b48005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:04.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 -- 172.21.15.125:0/2878289193 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c06fbb0 con 0x7f0b48004d10 2023-10-30T13:08:04.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 -- 172.21.15.125:0/2878289193 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c153d80 con 0x7f0b48003980 2023-10-30T13:08:04.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b53fff640 1 -- 172.21.15.125:0/2878289193 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4800b990 con 0x7f0b5c069e90 2023-10-30T13:08:04.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b618db640 1 --2- 172.21.15.125:0/2878289193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b48004d10 0x7f0b48005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:04.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b608d9640 1 --2- 172.21.15.125:0/2878289193 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b5c069e90 0x7f0b480047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:04.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b610da640 1 --2- 172.21.15.125:0/2878289193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b48003980 0x7f0b48003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:04.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b618db640 1 -- 172.21.15.125:0/2878289193 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b5c069e90 msgr2=0x7f0b480047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:04.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b618db640 1 --2- 172.21.15.125:0/2878289193 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b5c069e90 0x7f0b480047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:04.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b618db640 1 -- 172.21.15.125:0/2878289193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b48003980 msgr2=0x7f0b48003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:04.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b618db640 1 --2- 172.21.15.125:0/2878289193 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b48003980 0x7f0b48003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:04.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.608+0000 7f0b618db640 1 -- 172.21.15.125:0/2878289193 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b48003530 con 0x7f0b48004d10 2023-10-30T13:08:04.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.612+0000 7f0b618db640 1 --2- 172.21.15.125:0/2878289193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b48004d10 0x7f0b48005190 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f0b54009fc0 tx=0x7f0b5401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d0b453b29ea4d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:04.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.612+0000 7f0b53fff640 1 -- 172.21.15.125:0/2878289193 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b540a5b60 con 0x7f0b48004d10 2023-10-30T13:08:04.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.612+0000 7f0b53fff640 1 -- 172.21.15.125:0/2878289193 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5400f040 con 0x7f0b48004d10 2023-10-30T13:08:04.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.612+0000 7f0b53fff640 1 -- 172.21.15.125:0/2878289193 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b540ad440 con 0x7f0b48004d10 2023-10-30T13:08:04.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.612+0000 7f0b6333e640 1 -- 172.21.15.125:0/2878289193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b48004d10 msgr2=0x7f0b48005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:04.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.612+0000 7f0b6333e640 1 --2- 172.21.15.125:0/2878289193 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b48004d10 0x7f0b48005190 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f0b54009fc0 tx=0x7f0b5401cc60 comp rx=0 tx=0).stop 2023-10-30T13:08:04.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.612+0000 7f0b618db640 1 -- 172.21.15.125:0/2878289193 reap_dead start 2023-10-30T13:08:04.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.612+0000 7f0b6333e640 1 -- 172.21.15.125:0/2878289193 shutdown_connections 2023-10-30T13:08:04.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.612+0000 7f0b6333e640 1 -- 172.21.15.125:0/2878289193 >> 172.21.15.125:0/2878289193 conn(0x7f0b5c0c7910 msgr2=0x7f0b5c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:04.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.612+0000 7f0b6333e640 1 -- 172.21.15.125:0/2878289193 shutdown_connections 2023-10-30T13:08:04.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.612+0000 7f0b6333e640 1 -- 172.21.15.125:0/2878289193 wait complete. 2023-10-30T13:08:04.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.612+0000 7f0b6333e640 1 Processor -- start 2023-10-30T13:08:04.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.612+0000 7f0b6333e640 1 -- start start 2023-10-30T13:08:04.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b6333e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b48004d10 0x7f0b5c1556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:04.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b6333e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b5c156f80 0x7f0b5c0c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:04.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b6333e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b5c1f29c0 0x7f0b5c1f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:04.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b6333e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1579e0 con 0x7f0b48004d10 2023-10-30T13:08:04.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b6333e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c155e80 con 0x7f0b5c156f80 2023-10-30T13:08:04.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b6333e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c156140 con 0x7f0b5c1f29c0 2023-10-30T13:08:04.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b610da640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b48004d10 0x7f0b5c1556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:04.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b610da640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b48004d10 0x7f0b5c1556f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49310/0 (socket says 172.21.15.125:49310) 2023-10-30T13:08:04.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b610da640 1 -- 172.21.15.125:0/2152647485 learned_addr learned my addr 172.21.15.125:0/2152647485 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:04.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b608d9640 1 --2- 172.21.15.125:0/2152647485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b5c156f80 0x7f0b5c0c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:04.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b618db640 1 --2- 172.21.15.125:0/2152647485 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b5c1f29c0 0x7f0b5c1f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:04.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b610da640 1 -- 172.21.15.125:0/2152647485 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b5c1f29c0 msgr2=0x7f0b5c1f2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:04.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b610da640 1 --2- 172.21.15.125:0/2152647485 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b5c1f29c0 0x7f0b5c1f2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:04.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b610da640 1 -- 172.21.15.125:0/2152647485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b5c156f80 msgr2=0x7f0b5c0c12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:04.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b610da640 1 --2- 172.21.15.125:0/2152647485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b5c156f80 0x7f0b5c0c12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:04.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b610da640 1 -- 172.21.15.125:0/2152647485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5c1563b0 con 0x7f0b48004d10 2023-10-30T13:08:04.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b610da640 1 --2- 172.21.15.125:0/2152647485 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b48004d10 0x7f0b5c1556f0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f0b58011950 tx=0x7f0b58011e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:04.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b51ffb640 1 -- 172.21.15.125:0/2152647485 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5800fb60 con 0x7f0b48004d10 2023-10-30T13:08:04.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b51ffb640 1 -- 172.21.15.125:0/2152647485 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5800fcc0 con 0x7f0b48004d10 2023-10-30T13:08:04.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b51ffb640 1 -- 172.21.15.125:0/2152647485 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b58015630 con 0x7f0b48004d10 2023-10-30T13:08:04.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b6333e640 1 -- 172.21.15.125:0/2152647485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5c156690 con 0x7f0b48004d10 2023-10-30T13:08:04.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b6333e640 1 -- 172.21.15.125:0/2152647485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5c1f24a0 con 0x7f0b48004d10 2023-10-30T13:08:04.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b6333e640 1 -- 172.21.15.125:0/2152647485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5c156bb0 con 0x7f0b48004d10 2023-10-30T13:08:04.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.616+0000 7f0b51ffb640 1 -- 172.21.15.125:0/2152647485 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0b58015790 con 0x7f0b48004d10 2023-10-30T13:08:04.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.620+0000 7f0b51ffb640 1 --2- 172.21.15.125:0/2152647485 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0b400664a0 0x7f0b40068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.620+0000 7f0b51ffb640 1 -- 172.21.15.125:0/2152647485 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f0b58087a00 con 0x7f0b48004d10 2023-10-30T13:08:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.620+0000 7f0b608d9640 1 --2- 172.21.15.125:0/2152647485 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0b400664a0 0x7f0b40068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.620+0000 7f0b608d9640 1 --2- 172.21.15.125:0/2152647485 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0b400664a0 0x7f0b40068960 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c159a30 tx=0x7f0b4c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.620+0000 7f0b51ffb640 1 -- 172.21.15.125:0/2152647485 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0b580546f0 con 0x7f0b48004d10 2023-10-30T13:08:04.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.804+0000 7f0b6333e640 1 -- 172.21.15.125:0/2152647485 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b5c04e170 con 0x7f0b48004d10 2023-10-30T13:08:04.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.804+0000 7f0b51ffb640 1 -- 172.21.15.125:0/2152647485 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5751 (secure 0 0 0) 0x7f0b58057d10 con 0x7f0b48004d10 2023-10-30T13:08:04.810 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:04.811 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:00.628485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:04.811 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 179 2023-10-30T13:08:04.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.808+0000 7f0b6333e640 1 -- 172.21.15.125:0/2152647485 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0b400664a0 msgr2=0x7f0b40068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:04.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.808+0000 7f0b6333e640 1 --2- 172.21.15.125:0/2152647485 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0b400664a0 0x7f0b40068960 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c159a30 tx=0x7f0b4c008040 comp rx=0 tx=0).stop 2023-10-30T13:08:04.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.808+0000 7f0b6333e640 1 -- 172.21.15.125:0/2152647485 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b48004d10 msgr2=0x7f0b5c1556f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:04.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.808+0000 7f0b6333e640 1 --2- 172.21.15.125:0/2152647485 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b48004d10 0x7f0b5c1556f0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f0b58011950 tx=0x7f0b58011e10 comp rx=0 tx=0).stop 2023-10-30T13:08:04.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.808+0000 7f0b6333e640 1 -- 172.21.15.125:0/2152647485 shutdown_connections 2023-10-30T13:08:04.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.808+0000 7f0b6333e640 1 --2- 172.21.15.125:0/2152647485 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0b400664a0 0x7f0b40068960 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:04.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.808+0000 7f0b6333e640 1 --2- 172.21.15.125:0/2152647485 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0b5c1f29c0 0x7f0b5c1f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:04.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.808+0000 7f0b6333e640 1 --2- 172.21.15.125:0/2152647485 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0b5c156f80 0x7f0b5c0c12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:04.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.808+0000 7f0b6333e640 1 --2- 172.21.15.125:0/2152647485 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0b48004d10 0x7f0b5c1556f0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:04.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.808+0000 7f0b6333e640 1 -- 172.21.15.125:0/2152647485 >> 172.21.15.125:0/2152647485 conn(0x7f0b5c0c7910 msgr2=0x7f0b5c073480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:04.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.808+0000 7f0b6333e640 1 -- 172.21.15.125:0/2152647485 shutdown_connections 2023-10-30T13:08:04.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:04.808+0000 7f0b6333e640 1 -- 172.21.15.125:0/2152647485 wait complete. 2023-10-30T13:08:04.831 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:08:06.834 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:06.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.980+0000 7f7176df2640 1 Processor -- start 2023-10-30T13:08:06.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.980+0000 7f7176df2640 1 -- start start 2023-10-30T13:08:06.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7176df2640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71700c6840 0x7f717015a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:06.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7176df2640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f71700c6c20 0x7f7170156590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:06.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7176df2640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71700c3b30 con 0x7f71700c3640 2023-10-30T13:08:06.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7176df2640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71700c2aa0 con 0x7f71700c6c20 2023-10-30T13:08:06.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7176df2640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71700c3df0 con 0x7f71700c6840 2023-10-30T13:08:06.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f71755ef640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f71700c3640 0x7f7170153730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38288/0 (socket says 172.21.15.125:38288) 2023-10-30T13:08:06.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f71755ef640 1 -- 172.21.15.125:0/3546164526 learned_addr learned my addr 172.21.15.125:0/3546164526 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:06.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f71765f1640 1 --2- 172.21.15.125:0/3546164526 >> v2:172.21.15.136:3300/0 conn(0x7f71700c6c20 0x7f7170156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7175df0640 1 --2- 172.21.15.125:0/3546164526 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71700c6840 0x7f717015a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 -- 172.21.15.125:0/3546164526 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228035997 0 0) 0x7f71700c3b30 con 0x7f71700c3640 2023-10-30T13:08:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 -- 172.21.15.125:0/3546164526 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f715c005740 con 0x7f71700c3640 2023-10-30T13:08:06.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 -- 172.21.15.125:0/3546164526 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 260665559 0 0) 0x7f715c005740 con 0x7f71700c3640 2023-10-30T13:08:06.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 -- 172.21.15.125:0/3546164526 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71700c6840 msgr2=0x7f717015a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:06.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 --2- 172.21.15.125:0/3546164526 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71700c6840 0x7f717015a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:06.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 -- 172.21.15.125:0/3546164526 >> v2:172.21.15.136:3300/0 conn(0x7f71700c6c20 msgr2=0x7f7170156590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:06.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 --2- 172.21.15.125:0/3546164526 >> v2:172.21.15.136:3300/0 conn(0x7f71700c6c20 0x7f7170156590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:06.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 -- 172.21.15.125:0/3546164526 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7170159ba0 con 0x7f71700c3640 2023-10-30T13:08:06.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 -- 172.21.15.125:0/3546164526 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f7160001680 con 0x7f71700c3640 2023-10-30T13:08:06.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 -- 172.21.15.125:0/3546164526 >> v1:172.21.15.125:6789/0 conn(0x7f71700c3640 legacy=0x7f7170153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:06.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 --2- 172.21.15.125:0/3546164526 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f715c005b90 0x7f715c005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:06.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 --2- 172.21.15.125:0/3546164526 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71700c6840 0x7f715c0069e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:06.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 --2- 172.21.15.125:0/3546164526 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f715c006f20 0x7f715c0073a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:06.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 -- 172.21.15.125:0/3546164526 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71700c3df0 con 0x7f715c006f20 2023-10-30T13:08:06.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 -- 172.21.15.125:0/3546164526 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715c00dba0 con 0x7f715c005b90 2023-10-30T13:08:06.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7174dee640 1 -- 172.21.15.125:0/3546164526 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f715c00dce0 con 0x7f71700c6840 2023-10-30T13:08:06.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f71765f1640 1 --2- 172.21.15.125:0/3546164526 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f715c006f20 0x7f715c0073a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:06.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f7175df0640 1 --2- 172.21.15.125:0/3546164526 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f715c005b90 0x7f715c005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:06.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.984+0000 7f71755ef640 1 --2- 172.21.15.125:0/3546164526 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71700c6840 0x7f715c0069e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:06.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.988+0000 7f71765f1640 1 -- 172.21.15.125:0/3546164526 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71700c6840 msgr2=0x7f715c0069e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:06.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.988+0000 7f71765f1640 1 --2- 172.21.15.125:0/3546164526 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f71700c6840 0x7f715c0069e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:06.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.988+0000 7f71765f1640 1 -- 172.21.15.125:0/3546164526 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f715c005b90 msgr2=0x7f715c005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:06.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.988+0000 7f71765f1640 1 --2- 172.21.15.125:0/3546164526 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f715c005b90 0x7f715c005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:06.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.988+0000 7f71765f1640 1 -- 172.21.15.125:0/3546164526 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c005740 con 0x7f715c006f20 2023-10-30T13:08:06.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.988+0000 7f71765f1640 1 --2- 172.21.15.125:0/3546164526 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f715c006f20 0x7f715c0073a0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f7168009f90 tx=0x7f716801cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a41697aeff093c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:06.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.988+0000 7f7174dee640 1 -- 172.21.15.125:0/3546164526 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71680a5c70 con 0x7f715c006f20 2023-10-30T13:08:06.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.988+0000 7f7174dee640 1 -- 172.21.15.125:0/3546164526 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f716800f040 con 0x7f715c006f20 2023-10-30T13:08:07.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.988+0000 7f7174dee640 1 -- 172.21.15.125:0/3546164526 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71680ad440 con 0x7f715c006f20 2023-10-30T13:08:07.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.992+0000 7f7176df2640 1 -- 172.21.15.125:0/3546164526 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f715c006f20 msgr2=0x7f715c0073a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:07.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.992+0000 7f7176df2640 1 --2- 172.21.15.125:0/3546164526 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f715c006f20 0x7f715c0073a0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f7168009f90 tx=0x7f716801cc60 comp rx=0 tx=0).stop 2023-10-30T13:08:07.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.992+0000 7f71765f1640 1 -- 172.21.15.125:0/3546164526 reap_dead start 2023-10-30T13:08:07.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.992+0000 7f7176df2640 1 -- 172.21.15.125:0/3546164526 shutdown_connections 2023-10-30T13:08:07.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.992+0000 7f7176df2640 1 -- 172.21.15.125:0/3546164526 >> 172.21.15.125:0/3546164526 conn(0x7f71700c75d0 msgr2=0x7f71700c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:07.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.992+0000 7f7176df2640 1 -- 172.21.15.125:0/3546164526 shutdown_connections 2023-10-30T13:08:07.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.992+0000 7f7176df2640 1 -- 172.21.15.125:0/3546164526 wait complete. 2023-10-30T13:08:07.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.992+0000 7f7176df2640 1 Processor -- start 2023-10-30T13:08:07.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.992+0000 7f7176df2640 1 -- start start 2023-10-30T13:08:07.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.992+0000 7f7176df2640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f715c006f20 0x7f7170150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:07.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f7176df2640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f717014c270 0x7f717006fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:07.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f7176df2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f717014d5f0 0x7f71700701f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:07.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f7176df2640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71700c9bc0 con 0x7f717014d5f0 2023-10-30T13:08:07.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f7176df2640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71700ca4a0 con 0x7f715c006f20 2023-10-30T13:08:07.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f7176df2640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f717006d630 con 0x7f717014c270 2023-10-30T13:08:07.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f71765f1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f717014d5f0 0x7f71700701f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:07.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f71765f1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f717014d5f0 0x7f71700701f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49334/0 (socket says 172.21.15.125:49334) 2023-10-30T13:08:07.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f71765f1640 1 -- 172.21.15.125:0/703843820 learned_addr learned my addr 172.21.15.125:0/703843820 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:07.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f71755ef640 1 --2- 172.21.15.125:0/703843820 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f717014c270 0x7f717006fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:07.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f7175df0640 1 --2- 172.21.15.125:0/703843820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f715c006f20 0x7f7170150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:07.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f71765f1640 1 -- 172.21.15.125:0/703843820 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f717014c270 msgr2=0x7f717006fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:07.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f71765f1640 1 --2- 172.21.15.125:0/703843820 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f717014c270 0x7f717006fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:07.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f71765f1640 1 -- 172.21.15.125:0/703843820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f715c006f20 msgr2=0x7f7170150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:07.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f71765f1640 1 --2- 172.21.15.125:0/703843820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f715c006f20 0x7f7170150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:07.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f71765f1640 1 -- 172.21.15.125:0/703843820 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717014c760 con 0x7f717014d5f0 2023-10-30T13:08:07.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f71765f1640 1 --2- 172.21.15.125:0/703843820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f717014d5f0 0x7f71700701f0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f71680b0040 tx=0x7f7168009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:07.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f71667fc640 1 -- 172.21.15.125:0/703843820 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71680bc070 con 0x7f717014d5f0 2023-10-30T13:08:07.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f71667fc640 1 -- 172.21.15.125:0/703843820 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71680039a0 con 0x7f717014d5f0 2023-10-30T13:08:07.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f71667fc640 1 -- 172.21.15.125:0/703843820 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7168003b30 con 0x7f717014d5f0 2023-10-30T13:08:07.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f7176df2640 1 -- 172.21.15.125:0/703843820 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f717014ca40 con 0x7f717014d5f0 2023-10-30T13:08:07.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f7176df2640 1 -- 172.21.15.125:0/703843820 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f717014d000 con 0x7f717014d5f0 2023-10-30T13:08:07.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f7176df2640 1 -- 172.21.15.125:0/703843820 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f717004e170 con 0x7f717014d5f0 2023-10-30T13:08:07.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:06.996+0000 7f71667fc640 1 -- 172.21.15.125:0/703843820 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f71680b4040 con 0x7f717014d5f0 2023-10-30T13:08:07.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.000+0000 7f71667fc640 1 --2- 172.21.15.125:0/703843820 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71540664d0 0x7f7154068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:07.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.000+0000 7f71667fc640 1 -- 172.21.15.125:0/703843820 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f716811e820 con 0x7f717014d5f0 2023-10-30T13:08:07.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.000+0000 7f7175df0640 1 --2- 172.21.15.125:0/703843820 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71540664d0 0x7f7154068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:07.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.000+0000 7f7175df0640 1 --2- 172.21.15.125:0/703843820 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71540664d0 0x7f7154068990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f716c009bc0 tx=0x7f716c006f90 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:07.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.004+0000 7f71667fc640 1 -- 172.21.15.125:0/703843820 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f71680eb530 con 0x7f717014d5f0 2023-10-30T13:08:07.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.212+0000 7f7176df2640 1 -- 172.21.15.125:0/703843820 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71700c18d0 con 0x7f717014d5f0 2023-10-30T13:08:07.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.212+0000 7f71667fc640 1 -- 172.21.15.125:0/703843820 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5751 (secure 0 0 0) 0x7f71680eeb50 con 0x7f717014d5f0 2023-10-30T13:08:07.217 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:07.218 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:00.628485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:07.218 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 179 2023-10-30T13:08:07.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.212+0000 7f7176df2640 1 -- 172.21.15.125:0/703843820 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71540664d0 msgr2=0x7f7154068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:07.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.212+0000 7f7176df2640 1 --2- 172.21.15.125:0/703843820 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71540664d0 0x7f7154068990 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f716c009bc0 tx=0x7f716c006f90 comp rx=0 tx=0).stop 2023-10-30T13:08:07.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.212+0000 7f7176df2640 1 -- 172.21.15.125:0/703843820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f717014d5f0 msgr2=0x7f71700701f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:07.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.212+0000 7f7176df2640 1 --2- 172.21.15.125:0/703843820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f717014d5f0 0x7f71700701f0 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f71680b0040 tx=0x7f7168009750 comp rx=0 tx=0).stop 2023-10-30T13:08:07.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.212+0000 7f7176df2640 1 -- 172.21.15.125:0/703843820 shutdown_connections 2023-10-30T13:08:07.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.212+0000 7f7176df2640 1 --2- 172.21.15.125:0/703843820 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71540664d0 0x7f7154068990 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:07.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.212+0000 7f7176df2640 1 --2- 172.21.15.125:0/703843820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f717014d5f0 0x7f71700701f0 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:07.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.212+0000 7f7176df2640 1 --2- 172.21.15.125:0/703843820 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f717014c270 0x7f717006fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:07.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.212+0000 7f7176df2640 1 --2- 172.21.15.125:0/703843820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f715c006f20 0x7f7170150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:07.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.212+0000 7f7176df2640 1 -- 172.21.15.125:0/703843820 >> 172.21.15.125:0/703843820 conn(0x7f71700c75d0 msgr2=0x7f7170152280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:07.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.212+0000 7f7176df2640 1 -- 172.21.15.125:0/703843820 shutdown_connections 2023-10-30T13:08:07.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:07.212+0000 7f7176df2640 1 -- 172.21.15.125:0/703843820 wait complete. 2023-10-30T13:08:07.246 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:08:09.247 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:09.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f72dc4640 1 Processor -- start 2023-10-30T13:08:09.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f72dc4640 1 -- start start 2023-10-30T13:08:09.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f72dc4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f6c06d290 0x7f2f6c06a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:09.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f72dc4640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c0c6c10 con 0x7f2f6c15a410 2023-10-30T13:08:09.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f72dc4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f6c0c6ef0 con 0x7f2f6c069e90 2023-10-30T13:08:09.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f72dc4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c0c8570 con 0x7f2f6c06d290 2023-10-30T13:08:09.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f6bfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2f6c15a410 0x7f2f6c06cb80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38302/0 (socket says 172.21.15.125:38302) 2023-10-30T13:08:09.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f6bfff640 1 -- 172.21.15.125:0/4097098563 learned_addr learned my addr 172.21.15.125:0/4097098563 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:09.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f71361640 1 --2- 172.21.15.125:0/4097098563 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f6c06d290 0x7f2f6c06a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:09.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f6b7fe640 1 -- 172.21.15.125:0/4097098563 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505834602 0 0) 0x7f2f6c0c6c10 con 0x7f2f6c15a410 2023-10-30T13:08:09.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f6b7fe640 1 -- 172.21.15.125:0/4097098563 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f5c003530 con 0x7f2f6c15a410 2023-10-30T13:08:09.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f6b7fe640 1 -- 172.21.15.125:0/4097098563 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087613490 0 0) 0x7f2f6c0c6ef0 con 0x7f2f6c069e90 2023-10-30T13:08:09.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f6b7fe640 1 -- 172.21.15.125:0/4097098563 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f6c0c6c10 con 0x7f2f6c069e90 2023-10-30T13:08:09.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f6b7fe640 1 -- 172.21.15.125:0/4097098563 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3452104158 0 0) 0x7f2f5c003530 con 0x7f2f6c15a410 2023-10-30T13:08:09.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f71361640 1 -- 172.21.15.125:0/4097098563 >> v1:172.21.15.136:6789/0 conn(0x7f2f6c069e90 legacy=0x7f2f6c0c9cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:09.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f71361640 1 -- 172.21.15.125:0/4097098563 >> v1:172.21.15.125:6789/0 conn(0x7f2f6c15a410 legacy=0x7f2f6c06cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:09.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.396+0000 7f2f71361640 1 -- 172.21.15.125:0/4097098563 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6c0c71d0 con 0x7f2f6c06d290 2023-10-30T13:08:09.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.400+0000 7f2f71361640 1 --2- 172.21.15.125:0/4097098563 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f6c06d290 0x7f2f6c06a270 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2f64009dc0 tx=0x7f2f6401c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f23831d445c1ef85 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:09.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.400+0000 7f2f6b7fe640 1 -- 172.21.15.125:0/4097098563 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f58001680 con 0x7f2f6c06d290 2023-10-30T13:08:09.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.400+0000 7f2f6b7fe640 1 -- 172.21.15.125:0/4097098563 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f640a64e0 con 0x7f2f6c06d290 2023-10-30T13:08:09.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.400+0000 7f2f6b7fe640 1 -- 172.21.15.125:0/4097098563 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f640a6870 con 0x7f2f6c06d290 2023-10-30T13:08:09.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.400+0000 7f2f72dc4640 1 -- 172.21.15.125:0/4097098563 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f6c06d290 msgr2=0x7f2f6c06a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:09.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.400+0000 7f2f72dc4640 1 --2- 172.21.15.125:0/4097098563 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f6c06d290 0x7f2f6c06a270 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2f64009dc0 tx=0x7f2f6401c890 comp rx=0 tx=0).stop 2023-10-30T13:08:09.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.400+0000 7f2f72dc4640 1 -- 172.21.15.125:0/4097098563 shutdown_connections 2023-10-30T13:08:09.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.400+0000 7f2f72dc4640 1 --2- 172.21.15.125:0/4097098563 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f6c06d290 0x7f2f6c06a270 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:09.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.400+0000 7f2f72dc4640 1 -- 172.21.15.125:0/4097098563 >> 172.21.15.125:0/4097098563 conn(0x7f2f6c06dfd0 msgr2=0x7f2f6c06e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:09.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.400+0000 7f2f72dc4640 1 -- 172.21.15.125:0/4097098563 shutdown_connections 2023-10-30T13:08:09.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.400+0000 7f2f72dc4640 1 -- 172.21.15.125:0/4097098563 wait complete. 2023-10-30T13:08:09.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f72dc4640 1 Processor -- start 2023-10-30T13:08:09.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f72dc4640 1 -- start start 2023-10-30T13:08:09.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f72dc4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f6c069e90 0x7f2f6c0cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:09.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f72dc4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c06d290 0x7f2f6c154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:09.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f72dc4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f6c15a410 0x7f2f6c156330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:09.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f72dc4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c14df10 con 0x7f2f6c06d290 2023-10-30T13:08:09.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f72dc4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c1f5460 con 0x7f2f6c15a410 2023-10-30T13:08:09.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f72dc4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c1f5740 con 0x7f2f6c069e90 2023-10-30T13:08:09.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f6bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c06d290 0x7f2f6c154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:09.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f6bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c06d290 0x7f2f6c154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49346/0 (socket says 172.21.15.125:49346) 2023-10-30T13:08:09.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f6bfff640 1 -- 172.21.15.125:0/3800782589 learned_addr learned my addr 172.21.15.125:0/3800782589 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:09.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f70b60640 1 --2- 172.21.15.125:0/3800782589 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f6c069e90 0x7f2f6c0cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:09.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f71361640 1 --2- 172.21.15.125:0/3800782589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f6c15a410 0x7f2f6c156330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:09.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f6bfff640 1 -- 172.21.15.125:0/3800782589 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f6c069e90 msgr2=0x7f2f6c0cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:09.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f6bfff640 1 --2- 172.21.15.125:0/3800782589 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f6c069e90 0x7f2f6c0cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:09.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f6bfff640 1 -- 172.21.15.125:0/3800782589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f6c15a410 msgr2=0x7f2f6c156330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:09.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f6bfff640 1 --2- 172.21.15.125:0/3800782589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f6c15a410 0x7f2f6c156330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:09.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f6bfff640 1 -- 172.21.15.125:0/3800782589 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f64009a70 con 0x7f2f6c06d290 2023-10-30T13:08:09.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f6bfff640 1 --2- 172.21.15.125:0/3800782589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c06d290 0x7f2f6c154b40 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f2f58001ec0 tx=0x7f2f5800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:09.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f697fa640 1 -- 172.21.15.125:0/3800782589 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f58005ae0 con 0x7f2f6c06d290 2023-10-30T13:08:09.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f697fa640 1 -- 172.21.15.125:0/3800782589 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f58005c40 con 0x7f2f6c06d290 2023-10-30T13:08:09.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f697fa640 1 -- 172.21.15.125:0/3800782589 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f58007070 con 0x7f2f6c06d290 2023-10-30T13:08:09.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.404+0000 7f2f72dc4640 1 -- 172.21.15.125:0/3800782589 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c155850 con 0x7f2f6c06d290 2023-10-30T13:08:09.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.408+0000 7f2f72dc4640 1 -- 172.21.15.125:0/3800782589 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c155e10 con 0x7f2f6c06d290 2023-10-30T13:08:09.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.408+0000 7f2f72dc4640 1 -- 172.21.15.125:0/3800782589 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c1541b0 con 0x7f2f6c06d290 2023-10-30T13:08:09.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.408+0000 7f2f697fa640 1 -- 172.21.15.125:0/3800782589 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2f580071d0 con 0x7f2f6c06d290 2023-10-30T13:08:09.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.408+0000 7f2f697fa640 1 --2- 172.21.15.125:0/3800782589 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f500664a0 0x7f2f50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:09.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.408+0000 7f2f697fa640 1 -- 172.21.15.125:0/3800782589 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f2f58084230 con 0x7f2f6c06d290 2023-10-30T13:08:09.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.408+0000 7f2f70b60640 1 --2- 172.21.15.125:0/3800782589 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f500664a0 0x7f2f50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:09.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.408+0000 7f2f70b60640 1 --2- 172.21.15.125:0/3800782589 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f500664a0 0x7f2f50068960 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f2f600011f0 tx=0x7f2f60009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:09.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.412+0000 7f2f697fa640 1 -- 172.21.15.125:0/3800782589 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2f58050fd0 con 0x7f2f6c06d290 2023-10-30T13:08:09.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.604+0000 7f2f72dc4640 1 -- 172.21.15.125:0/3800782589 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f6c156030 con 0x7f2f6c06d290 2023-10-30T13:08:09.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.604+0000 7f2f697fa640 1 -- 172.21.15.125:0/3800782589 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 179 v179) v1 ==== 95+0+5751 (secure 0 0 0) 0x7f2f580545f0 con 0x7f2f6c06d290 2023-10-30T13:08:09.606 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:09.607 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":179,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":179,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:00.628485+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:09.607 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 179 2023-10-30T13:08:09.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.608+0000 7f2f72dc4640 1 -- 172.21.15.125:0/3800782589 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f500664a0 msgr2=0x7f2f50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:09.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.608+0000 7f2f72dc4640 1 --2- 172.21.15.125:0/3800782589 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f500664a0 0x7f2f50068960 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f2f600011f0 tx=0x7f2f60009040 comp rx=0 tx=0).stop 2023-10-30T13:08:09.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.608+0000 7f2f72dc4640 1 -- 172.21.15.125:0/3800782589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c06d290 msgr2=0x7f2f6c154b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:09.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.608+0000 7f2f72dc4640 1 --2- 172.21.15.125:0/3800782589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c06d290 0x7f2f6c154b40 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f2f58001ec0 tx=0x7f2f5800cbe0 comp rx=0 tx=0).stop 2023-10-30T13:08:09.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.608+0000 7f2f72dc4640 1 -- 172.21.15.125:0/3800782589 shutdown_connections 2023-10-30T13:08:09.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.608+0000 7f2f72dc4640 1 --2- 172.21.15.125:0/3800782589 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2f500664a0 0x7f2f50068960 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.608+0000 7f2f72dc4640 1 --2- 172.21.15.125:0/3800782589 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2f6c15a410 0x7f2f6c156330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.608+0000 7f2f72dc4640 1 --2- 172.21.15.125:0/3800782589 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2f6c06d290 0x7f2f6c154b40 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.608+0000 7f2f72dc4640 1 --2- 172.21.15.125:0/3800782589 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2f6c069e90 0x7f2f6c0cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.608+0000 7f2f72dc4640 1 -- 172.21.15.125:0/3800782589 >> 172.21.15.125:0/3800782589 conn(0x7f2f6c06dfd0 msgr2=0x7f2f6c0717c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:09.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.608+0000 7f2f72dc4640 1 -- 172.21.15.125:0/3800782589 shutdown_connections 2023-10-30T13:08:09.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:09.608+0000 7f2f72dc4640 1 -- 172.21.15.125:0/3800782589 wait complete. 2023-10-30T13:08:09.628 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:08:11.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:11.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.776+0000 7f190930e640 1 Processor -- start 2023-10-30T13:08:11.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.776+0000 7f190930e640 1 -- start start 2023-10-30T13:08:11.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f190930e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1904153d70 0x7f190406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:11.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f190930e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f19040c8490 0x7f19040c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:11.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f190930e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f190406afa0 con 0x7f190406e470 2023-10-30T13:08:11.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f190930e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f190406d780 con 0x7f1904153d70 2023-10-30T13:08:11.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f190930e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1904154150 con 0x7f19040c8490 2023-10-30T13:08:11.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f1902ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1904153d70 0x7f190406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:11.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f18fa7fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f190406e470 0x7f19040cce00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47048/0 (socket says 172.21.15.125:47048) 2023-10-30T13:08:11.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f18fa7fc640 1 -- 172.21.15.125:0/2520473771 learned_addr learned my addr 172.21.15.125:0/2520473771 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:11.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f19037fe640 1 --2- 172.21.15.125:0/2520473771 >> v2:172.21.15.136:3300/0 conn(0x7f19040c8490 0x7f19040c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:11.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f1902ffd640 1 -- 172.21.15.125:0/2520473771 >> v2:172.21.15.136:3300/0 conn(0x7f19040c8490 msgr2=0x7f19040c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:11.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f1902ffd640 1 --2- 172.21.15.125:0/2520473771 >> v2:172.21.15.136:3300/0 conn(0x7f19040c8490 0x7f19040c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:11.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f1902ffd640 1 -- 172.21.15.125:0/2520473771 >> v1:172.21.15.125:6789/0 conn(0x7f190406e470 legacy=0x7f19040cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:11.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f1902ffd640 1 -- 172.21.15.125:0/2520473771 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f190406e030 con 0x7f1904153d70 2023-10-30T13:08:11.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f19027fc640 1 -- 172.21.15.125:0/2520473771 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754329484 0 0) 0x7f190406afa0 con 0x7f190406e470 2023-10-30T13:08:11.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f1902ffd640 1 --2- 172.21.15.125:0/2520473771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1904153d70 0x7f190406ca80 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f19040cddf0 tx=0x7f18f401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76cce2e8af589475 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:11.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f19027fc640 1 -- 172.21.15.125:0/2520473771 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f40a7070 con 0x7f1904153d70 2023-10-30T13:08:11.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f19027fc640 1 -- 172.21.15.125:0/2520473771 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f40ab430 con 0x7f1904153d70 2023-10-30T13:08:11.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.780+0000 7f19027fc640 1 -- 172.21.15.125:0/2520473771 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f40ab7c0 con 0x7f1904153d70 2023-10-30T13:08:11.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 -- 172.21.15.125:0/2520473771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1904153d70 msgr2=0x7f190406ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:11.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 --2- 172.21.15.125:0/2520473771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1904153d70 0x7f190406ca80 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f19040cddf0 tx=0x7f18f401c430 comp rx=0 tx=0).stop 2023-10-30T13:08:11.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 -- 172.21.15.125:0/2520473771 shutdown_connections 2023-10-30T13:08:11.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 --2- 172.21.15.125:0/2520473771 >> v2:172.21.15.136:3300/0 conn(0x7f19040c8490 0x7f19040c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:11.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 --2- 172.21.15.125:0/2520473771 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1904153d70 0x7f190406ca80 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:11.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 -- 172.21.15.125:0/2520473771 >> 172.21.15.125:0/2520473771 conn(0x7f1904071ea0 msgr2=0x7f1904065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:11.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 -- 172.21.15.125:0/2520473771 shutdown_connections 2023-10-30T13:08:11.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 -- 172.21.15.125:0/2520473771 wait complete. 2023-10-30T13:08:11.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 Processor -- start 2023-10-30T13:08:11.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 -- start start 2023-10-30T13:08:11.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f190406e470 0x7f1904147c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:11.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19040c8490 0x7f1904151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:11.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1904153d70 0x7f1904145c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:11.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1904151350 con 0x7f190406e470 2023-10-30T13:08:11.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19041457b0 con 0x7f19040c8490 2023-10-30T13:08:11.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f190930e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19041493b0 con 0x7f1904153d70 2023-10-30T13:08:11.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f1902ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f190406e470 0x7f1904147c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:11.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f1902ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f190406e470 0x7f1904147c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35388/0 (socket says 172.21.15.125:35388) 2023-10-30T13:08:11.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f1902ffd640 1 -- 172.21.15.125:0/1189210808 learned_addr learned my addr 172.21.15.125:0/1189210808 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:11.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f18fa7fc640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19040c8490 0x7f1904151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:11.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f19037fe640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1904153d70 0x7f1904145c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:11.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f1902ffd640 1 -- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1904153d70 msgr2=0x7f1904145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:11.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f1902ffd640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1904153d70 0x7f1904145c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:11.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f1902ffd640 1 -- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19040c8490 msgr2=0x7f1904151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:11.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f1902ffd640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19040c8490 0x7f1904151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:11.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f1902ffd640 1 -- 172.21.15.125:0/1189210808 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f4009420 con 0x7f190406e470 2023-10-30T13:08:11.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.784+0000 7f18fa7fc640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19040c8490 0x7f1904151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:08:11.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.788+0000 7f1902ffd640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f190406e470 0x7f1904147c50 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f18f40ad040 tx=0x7f18f401ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:11.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.788+0000 7f18fbfff640 1 -- 172.21.15.125:0/1189210808 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f40b5810 con 0x7f190406e470 2023-10-30T13:08:11.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.788+0000 7f18fbfff640 1 -- 172.21.15.125:0/1189210808 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18f40b5990 con 0x7f190406e470 2023-10-30T13:08:11.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.788+0000 7f190930e640 1 -- 172.21.15.125:0/1189210808 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1904149670 con 0x7f190406e470 2023-10-30T13:08:11.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.788+0000 7f18fbfff640 1 -- 172.21.15.125:0/1189210808 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f40b5d40 con 0x7f190406e470 2023-10-30T13:08:11.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.788+0000 7f190930e640 1 -- 172.21.15.125:0/1189210808 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1904144e80 con 0x7f190406e470 2023-10-30T13:08:11.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.788+0000 7f190930e640 1 -- 172.21.15.125:0/1189210808 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f190404e170 con 0x7f190406e470 2023-10-30T13:08:11.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.788+0000 7f18fbfff640 1 -- 172.21.15.125:0/1189210808 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f18f40c7390 con 0x7f190406e470 2023-10-30T13:08:11.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.788+0000 7f18fbfff640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f18e80664a0 0x7f18e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:11.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.788+0000 7f18fbfff640 1 -- 172.21.15.125:0/1189210808 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f18f411f190 con 0x7f190406e470 2023-10-30T13:08:11.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.788+0000 7f18fa7fc640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f18e80664a0 0x7f18e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:11.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.792+0000 7f18fa7fc640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f18e80664a0 0x7f18e8068960 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f19040c73a0 tx=0x7f18ec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:11.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.792+0000 7f18fbfff640 1 -- 172.21.15.125:0/1189210808 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f18f40ebdf0 con 0x7f190406e470 2023-10-30T13:08:11.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.972+0000 7f190930e640 1 -- 172.21.15.125:0/1189210808 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1904140d20 con 0x7f190406e470 2023-10-30T13:08:11.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.972+0000 7f18fbfff640 1 -- 172.21.15.125:0/1189210808 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5752 (secure 0 0 0) 0x7f18f4006e20 con 0x7f190406e470 2023-10-30T13:08:11.975 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:11.976 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:10.073697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:resolve","state_seq":9,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:11.976 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 180 2023-10-30T13:08:11.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.976+0000 7f190930e640 1 -- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f18e80664a0 msgr2=0x7f18e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:11.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.976+0000 7f190930e640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f18e80664a0 0x7f18e8068960 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f19040c73a0 tx=0x7f18ec009040 comp rx=0 tx=0).stop 2023-10-30T13:08:11.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.976+0000 7f190930e640 1 -- 172.21.15.125:0/1189210808 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f190406e470 msgr2=0x7f1904147c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:11.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.976+0000 7f190930e640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f190406e470 0x7f1904147c50 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f18f40ad040 tx=0x7f18f401ac70 comp rx=0 tx=0).stop 2023-10-30T13:08:11.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.976+0000 7f190930e640 1 -- 172.21.15.125:0/1189210808 shutdown_connections 2023-10-30T13:08:11.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.976+0000 7f190930e640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f18e80664a0 0x7f18e8068960 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:11.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.976+0000 7f190930e640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1904153d70 0x7f1904145c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:11.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.976+0000 7f190930e640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19040c8490 0x7f1904151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:11.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.976+0000 7f190930e640 1 --2- 172.21.15.125:0/1189210808 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f190406e470 0x7f1904147c50 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:11.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.976+0000 7f190930e640 1 -- 172.21.15.125:0/1189210808 >> 172.21.15.125:0/1189210808 conn(0x7f1904071ea0 msgr2=0x7f1904143d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:11.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.976+0000 7f190930e640 1 -- 172.21.15.125:0/1189210808 shutdown_connections 2023-10-30T13:08:11.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:11.976+0000 7f190930e640 1 -- 172.21.15.125:0/1189210808 wait complete. 2023-10-30T13:08:11.996 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:08:13.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:14.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1ab070f640 1 Processor -- start 2023-10-30T13:08:14.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1ab070f640 1 -- start start 2023-10-30T13:08:14.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1ab070f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1aa81538a0 0x7f1aa815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1ab070f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1aa80ca0e0 0x7f1aa815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1ab070f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa80cb800 con 0x7f1aa80cf160 2023-10-30T13:08:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1ab070f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa815dc30 con 0x7f1aa81538a0 2023-10-30T13:08:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1ab070f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa80cb330 con 0x7f1aa80ca0e0 2023-10-30T13:08:14.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1aae4ab640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1aa81538a0 0x7f1aa815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1aae4ab640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1aa81538a0 0x7f1aa815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35404/0 (socket says 172.21.15.125:35404) 2023-10-30T13:08:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1aae4ab640 1 -- 172.21.15.125:0/118962220 learned_addr learned my addr 172.21.15.125:0/118962220 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1aadcaa640 1 --2- 172.21.15.125:0/118962220 >> v2:172.21.15.136:3300/0 conn(0x7f1aa80ca0e0 0x7f1aa815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1aae4ab640 1 -- 172.21.15.125:0/118962220 >> v2:172.21.15.136:3300/0 conn(0x7f1aa80ca0e0 msgr2=0x7f1aa815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:14.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1aae4ab640 1 --2- 172.21.15.125:0/118962220 >> v2:172.21.15.136:3300/0 conn(0x7f1aa80ca0e0 0x7f1aa815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:14.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1aae4ab640 1 -- 172.21.15.125:0/118962220 >> v1:172.21.15.136:6789/0 conn(0x7f1aa80cf160 legacy=0x7f1aa80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:14.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.144+0000 7f1aae4ab640 1 -- 172.21.15.125:0/118962220 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa80cad70 con 0x7f1aa81538a0 2023-10-30T13:08:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.148+0000 7f1aae4ab640 1 --2- 172.21.15.125:0/118962220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1aa81538a0 0x7f1aa815d6f0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80b30a0 tx=0x7f1aa401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39818fe1ec3e86b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.148+0000 7f1aad4a9640 1 -- 172.21.15.125:0/118962220 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa40a89b0 con 0x7f1aa81538a0 2023-10-30T13:08:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.148+0000 7f1aad4a9640 1 -- 172.21.15.125:0/118962220 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa40a8b10 con 0x7f1aa81538a0 2023-10-30T13:08:14.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.148+0000 7f1aad4a9640 1 -- 172.21.15.125:0/118962220 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa40a8ea0 con 0x7f1aa81538a0 2023-10-30T13:08:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.148+0000 7f1ab070f640 1 -- 172.21.15.125:0/118962220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1aa81538a0 msgr2=0x7f1aa815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:14.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.148+0000 7f1ab070f640 1 --2- 172.21.15.125:0/118962220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1aa81538a0 0x7f1aa815d6f0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f1aa80b30a0 tx=0x7f1aa401ecb0 comp rx=0 tx=0).stop 2023-10-30T13:08:14.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.148+0000 7f1ab070f640 1 -- 172.21.15.125:0/118962220 shutdown_connections 2023-10-30T13:08:14.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.148+0000 7f1ab070f640 1 --2- 172.21.15.125:0/118962220 >> v2:172.21.15.136:3300/0 conn(0x7f1aa80ca0e0 0x7f1aa815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:14.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.148+0000 7f1ab070f640 1 --2- 172.21.15.125:0/118962220 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1aa81538a0 0x7f1aa815d6f0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:14.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.148+0000 7f1ab070f640 1 -- 172.21.15.125:0/118962220 >> 172.21.15.125:0/118962220 conn(0x7f1aa806bfe0 msgr2=0x7f1aa806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:14.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.148+0000 7f1ab070f640 1 -- 172.21.15.125:0/118962220 shutdown_connections 2023-10-30T13:08:14.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.148+0000 7f1ab070f640 1 -- 172.21.15.125:0/118962220 wait complete. 2023-10-30T13:08:14.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1ab070f640 1 Processor -- start 2023-10-30T13:08:14.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1ab070f640 1 -- start start 2023-10-30T13:08:14.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1ab070f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1aa80ca0e0 0x7f1aa80b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1ab070f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1aa80cf160 0x7f1aa81466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1ab070f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1aa81538a0 0x7f1aa81498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1ab070f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa80cda00 con 0x7f1aa80ca0e0 2023-10-30T13:08:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1ab070f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa81e2ae0 con 0x7f1aa80cf160 2023-10-30T13:08:14.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1ab070f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa80b3570 con 0x7f1aa81538a0 2023-10-30T13:08:14.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1aae4ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1aa80ca0e0 0x7f1aa80b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:14.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1aae4ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1aa80ca0e0 0x7f1aa80b4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35418/0 (socket says 172.21.15.125:35418) 2023-10-30T13:08:14.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1aae4ab640 1 -- 172.21.15.125:0/1609039150 learned_addr learned my addr 172.21.15.125:0/1609039150 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:14.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1aadcaa640 1 --2- 172.21.15.125:0/1609039150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1aa80cf160 0x7f1aa81466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:14.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1aaecac640 1 --2- 172.21.15.125:0/1609039150 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1aa81538a0 0x7f1aa81498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:14.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1aae4ab640 1 -- 172.21.15.125:0/1609039150 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1aa81538a0 msgr2=0x7f1aa81498e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:14.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1aae4ab640 1 --2- 172.21.15.125:0/1609039150 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1aa81538a0 0x7f1aa81498e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:14.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1aae4ab640 1 -- 172.21.15.125:0/1609039150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1aa80cf160 msgr2=0x7f1aa81466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:14.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1aae4ab640 1 --2- 172.21.15.125:0/1609039150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1aa80cf160 0x7f1aa81466e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:14.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1aae4ab640 1 -- 172.21.15.125:0/1609039150 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa40094d0 con 0x7f1aa80ca0e0 2023-10-30T13:08:14.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1aae4ab640 1 --2- 172.21.15.125:0/1609039150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1aa80ca0e0 0x7f1aa80b4d60 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40b3040 tx=0x7f1aa401ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:14.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1a9effd640 1 -- 172.21.15.125:0/1609039150 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa40b0e80 con 0x7f1aa80ca0e0 2023-10-30T13:08:14.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1a9effd640 1 -- 172.21.15.125:0/1609039150 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa40bb480 con 0x7f1aa80ca0e0 2023-10-30T13:08:14.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1a9effd640 1 -- 172.21.15.125:0/1609039150 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa40bb810 con 0x7f1aa80ca0e0 2023-10-30T13:08:14.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1ab070f640 1 -- 172.21.15.125:0/1609039150 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa8145ff0 con 0x7f1aa80ca0e0 2023-10-30T13:08:14.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1ab070f640 1 -- 172.21.15.125:0/1609039150 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa80ccfb0 con 0x7f1aa80ca0e0 2023-10-30T13:08:14.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.152+0000 7f1ab070f640 1 -- 172.21.15.125:0/1609039150 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa80b2a50 con 0x7f1aa80ca0e0 2023-10-30T13:08:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.156+0000 7f1a9effd640 1 -- 172.21.15.125:0/1609039150 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1aa40bb970 con 0x7f1aa80ca0e0 2023-10-30T13:08:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.156+0000 7f1a9effd640 1 --2- 172.21.15.125:0/1609039150 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a8c0664a0 0x7f1a8c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:14.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.156+0000 7f1a9effd640 1 -- 172.21.15.125:0/1609039150 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f1aa4122b90 con 0x7f1aa80ca0e0 2023-10-30T13:08:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.156+0000 7f1aadcaa640 1 --2- 172.21.15.125:0/1609039150 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a8c0664a0 0x7f1a8c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.156+0000 7f1aadcaa640 1 --2- 172.21.15.125:0/1609039150 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a8c0664a0 0x7f1a8c068960 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1aa81e5710 tx=0x7f1a98008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:14.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.160+0000 7f1a9effd640 1 -- 172.21.15.125:0/1609039150 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1aa40ef880 con 0x7f1aa80ca0e0 2023-10-30T13:08:14.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.340+0000 7f1ab070f640 1 -- 172.21.15.125:0/1609039150 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1aa80b2a50 con 0x7f1aa80ca0e0 2023-10-30T13:08:14.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.340+0000 7f1a9effd640 1 -- 172.21.15.125:0/1609039150 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+5752 (secure 0 0 0) 0x7f1aa40f2ea0 con 0x7f1aa80ca0e0 2023-10-30T13:08:14.341 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:14.342 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:10.073697+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:resolve","state_seq":9,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:14.342 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 180 2023-10-30T13:08:14.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.340+0000 7f1ab070f640 1 -- 172.21.15.125:0/1609039150 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a8c0664a0 msgr2=0x7f1a8c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:14.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.340+0000 7f1ab070f640 1 --2- 172.21.15.125:0/1609039150 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a8c0664a0 0x7f1a8c068960 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f1aa81e5710 tx=0x7f1a98008040 comp rx=0 tx=0).stop 2023-10-30T13:08:14.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.340+0000 7f1ab070f640 1 -- 172.21.15.125:0/1609039150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1aa80ca0e0 msgr2=0x7f1aa80b4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:14.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.340+0000 7f1ab070f640 1 --2- 172.21.15.125:0/1609039150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1aa80ca0e0 0x7f1aa80b4d60 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f1aa40b3040 tx=0x7f1aa401ba50 comp rx=0 tx=0).stop 2023-10-30T13:08:14.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.340+0000 7f1ab070f640 1 -- 172.21.15.125:0/1609039150 shutdown_connections 2023-10-30T13:08:14.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.340+0000 7f1ab070f640 1 --2- 172.21.15.125:0/1609039150 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a8c0664a0 0x7f1a8c068960 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:14.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.340+0000 7f1ab070f640 1 --2- 172.21.15.125:0/1609039150 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1aa81538a0 0x7f1aa81498e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:14.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.340+0000 7f1ab070f640 1 --2- 172.21.15.125:0/1609039150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1aa80cf160 0x7f1aa81466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:14.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.340+0000 7f1ab070f640 1 --2- 172.21.15.125:0/1609039150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1aa80ca0e0 0x7f1aa80b4d60 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:14.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.340+0000 7f1ab070f640 1 -- 172.21.15.125:0/1609039150 >> 172.21.15.125:0/1609039150 conn(0x7f1aa806bfe0 msgr2=0x7f1aa80c5460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:14.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.340+0000 7f1ab070f640 1 -- 172.21.15.125:0/1609039150 shutdown_connections 2023-10-30T13:08:14.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:14.340+0000 7f1ab070f640 1 -- 172.21.15.125:0/1609039150 wait complete. 2023-10-30T13:08:14.363 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 180, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 180, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6017": { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 170, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 251 }, "gid_8301": { "addr": "172.21.15.125:6837/2540063352", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2540063352, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2540063352, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8301, "incarnation": 168, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_8529": { "addr": "172.21.15.125:6836/3950044434", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3950044434, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3950044434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8529, "incarnation": 179, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:resolve", "state_seq": 9 } }, "last_failure": 0, "last_failure_osd_epoch": 76, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:08:10.073697+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8529, "mds_1": 8301, "mds_2": 6017 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:08:16.364 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:16.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.508+0000 7fd026d0e640 1 Processor -- start 2023-10-30T13:08:16.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.508+0000 7fd026d0e640 1 -- start start 2023-10-30T13:08:16.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.508+0000 7fd026d0e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd0200c05d0 0x7fd02013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:16.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.508+0000 7fd026d0e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd0200c6f60 0x7fd02013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:16.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.508+0000 7fd026d0e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0200c0a50 con 0x7fd0200c6b80 2023-10-30T13:08:16.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.508+0000 7fd026d0e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd02006f6e0 con 0x7fd0200c6f60 2023-10-30T13:08:16.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.508+0000 7fd026d0e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd02006f9c0 con 0x7fd0200c05d0 2023-10-30T13:08:16.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.508+0000 7fd0252ab640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd0200c6f60 0x7fd02013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:16.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.508+0000 7fd0252ab640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd0200c6f60 0x7fd02013f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35434/0 (socket says 172.21.15.125:35434) 2023-10-30T13:08:16.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.508+0000 7fd0252ab640 1 -- 172.21.15.125:0/999878953 learned_addr learned my addr 172.21.15.125:0/999878953 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:16.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.512+0000 7fd01ffff640 1 --2- 172.21.15.125:0/999878953 >> v2:172.21.15.136:3300/0 conn(0x7fd0200c05d0 0x7fd02013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:16.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.512+0000 7fd0252ab640 1 -- 172.21.15.125:0/999878953 >> v2:172.21.15.136:3300/0 conn(0x7fd0200c05d0 msgr2=0x7fd02013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.512+0000 7fd0252ab640 1 --2- 172.21.15.125:0/999878953 >> v2:172.21.15.136:3300/0 conn(0x7fd0200c05d0 0x7fd02013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.512+0000 7fd0252ab640 1 -- 172.21.15.125:0/999878953 >> v1:172.21.15.136:6789/0 conn(0x7fd0200c6b80 legacy=0x7fd0200bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.512+0000 7fd0252ab640 1 -- 172.21.15.125:0/999878953 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02006fca0 con 0x7fd0200c6f60 2023-10-30T13:08:16.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.512+0000 7fd01ffff640 1 --2- 172.21.15.125:0/999878953 >> v2:172.21.15.136:3300/0 conn(0x7fd0200c05d0 0x7fd02013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.512+0000 7fd0252ab640 1 --2- 172.21.15.125:0/999878953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0200c6f60 0x7fd02013f2d0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fd02006c7e0 tx=0x7fd01801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbee82de9fed638c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.512+0000 7fd01f7fe640 1 -- 172.21.15.125:0/999878953 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0180a57a0 con 0x7fd0200c6f60 2023-10-30T13:08:16.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.512+0000 7fd01f7fe640 1 -- 172.21.15.125:0/999878953 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0180a5900 con 0x7fd0200c6f60 2023-10-30T13:08:16.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.512+0000 7fd01f7fe640 1 -- 172.21.15.125:0/999878953 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0180a5c90 con 0x7fd0200c6f60 2023-10-30T13:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 -- 172.21.15.125:0/999878953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0200c6f60 msgr2=0x7fd02013f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 --2- 172.21.15.125:0/999878953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0200c6f60 0x7fd02013f2d0 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7fd02006c7e0 tx=0x7fd01801c9d0 comp rx=0 tx=0).stop 2023-10-30T13:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 -- 172.21.15.125:0/999878953 shutdown_connections 2023-10-30T13:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 --2- 172.21.15.125:0/999878953 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0200c6f60 0x7fd02013f2d0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 --2- 172.21.15.125:0/999878953 >> v2:172.21.15.136:3300/0 conn(0x7fd0200c05d0 0x7fd02013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:16.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 -- 172.21.15.125:0/999878953 >> 172.21.15.125:0/999878953 conn(0x7fd0200c7910 msgr2=0x7fd0200c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:16.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 -- 172.21.15.125:0/999878953 shutdown_connections 2023-10-30T13:08:16.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 -- 172.21.15.125:0/999878953 wait complete. 2023-10-30T13:08:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 Processor -- start 2023-10-30T13:08:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 -- start start 2023-10-30T13:08:16.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0200c05d0 0x7fd0200c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0200c6b80 0x7fd0201f1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0200c6f60 0x7fd0201f1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd020134600 con 0x7fd0200c6f60 2023-10-30T13:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0201f4ae0 con 0x7fd0200c6b80 2023-10-30T13:08:16.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd026d0e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0201f5620 con 0x7fd0200c05d0 2023-10-30T13:08:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd0252ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0200c6f60 0x7fd0201f1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd0252ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0200c6f60 0x7fd0201f1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35448/0 (socket says 172.21.15.125:35448) 2023-10-30T13:08:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd0252ab640 1 -- 172.21.15.125:0/4136884242 learned_addr learned my addr 172.21.15.125:0/4136884242 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd01ffff640 1 --2- 172.21.15.125:0/4136884242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0200c6b80 0x7fd0201f1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:16.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd024aaa640 1 --2- 172.21.15.125:0/4136884242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0200c05d0 0x7fd0200c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:16.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd0252ab640 1 -- 172.21.15.125:0/4136884242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0200c05d0 msgr2=0x7fd0200c5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:16.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd0252ab640 1 --2- 172.21.15.125:0/4136884242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0200c05d0 0x7fd0200c5a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:16.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd0252ab640 1 -- 172.21.15.125:0/4136884242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0200c6b80 msgr2=0x7fd0201f1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:16.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd0252ab640 1 --2- 172.21.15.125:0/4136884242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0200c6b80 0x7fd0201f1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:16.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.516+0000 7fd0252ab640 1 -- 172.21.15.125:0/4136884242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd018009b30 con 0x7fd0200c6f60 2023-10-30T13:08:16.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.520+0000 7fd0252ab640 1 --2- 172.21.15.125:0/4136884242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0200c6f60 0x7fd0201f1ab0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fd0180af040 tx=0x7fd018019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:16.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.520+0000 7fd01d7fa640 1 -- 172.21.15.125:0/4136884242 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0180b8880 con 0x7fd0200c6f60 2023-10-30T13:08:16.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.520+0000 7fd01d7fa640 1 -- 172.21.15.125:0/4136884242 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0180b8a20 con 0x7fd0200c6f60 2023-10-30T13:08:16.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.520+0000 7fd01d7fa640 1 -- 172.21.15.125:0/4136884242 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0180b8dd0 con 0x7fd0200c6f60 2023-10-30T13:08:16.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.520+0000 7fd026d0e640 1 -- 172.21.15.125:0/4136884242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0201347f0 con 0x7fd0200c6f60 2023-10-30T13:08:16.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.520+0000 7fd026d0e640 1 -- 172.21.15.125:0/4136884242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0201f4fd0 con 0x7fd0200c6f60 2023-10-30T13:08:16.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.520+0000 7fd026d0e640 1 -- 172.21.15.125:0/4136884242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0201f57b0 con 0x7fd0200c6f60 2023-10-30T13:08:16.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.520+0000 7fd01d7fa640 1 -- 172.21.15.125:0/4136884242 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd0180c3460 con 0x7fd0200c6f60 2023-10-30T13:08:16.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.520+0000 7fd01d7fa640 1 --2- 172.21.15.125:0/4136884242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0040664d0 0x7fd004068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:16.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.520+0000 7fd01d7fa640 1 -- 172.21.15.125:0/4136884242 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7fd01811fda0 con 0x7fd0200c6f60 2023-10-30T13:08:16.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.520+0000 7fd024aaa640 1 --2- 172.21.15.125:0/4136884242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0040664d0 0x7fd004068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:16.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.524+0000 7fd024aaa640 1 --2- 172.21.15.125:0/4136884242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0040664d0 0x7fd004068990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd014007fd0 tx=0x7fd014009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:16.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.524+0000 7fd01d7fa640 1 -- 172.21.15.125:0/4136884242 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd0180ecc70 con 0x7fd0200c6f60 2023-10-30T13:08:16.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.712+0000 7fd026d0e640 1 -- 172.21.15.125:0/4136884242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd02004e170 con 0x7fd0200c6f60 2023-10-30T13:08:16.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.712+0000 7fd01d7fa640 1 -- 172.21.15.125:0/4136884242 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+5755 (secure 0 0 0) 0x7fd0180f0290 con 0x7fd0200c6f60 2023-10-30T13:08:16.715 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:16.715 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:15.789388+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:16.716 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 181 2023-10-30T13:08:16.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.716+0000 7fd026d0e640 1 -- 172.21.15.125:0/4136884242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0040664d0 msgr2=0x7fd004068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:16.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.716+0000 7fd026d0e640 1 --2- 172.21.15.125:0/4136884242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0040664d0 0x7fd004068990 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7fd014007fd0 tx=0x7fd014009040 comp rx=0 tx=0).stop 2023-10-30T13:08:16.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.716+0000 7fd026d0e640 1 -- 172.21.15.125:0/4136884242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0200c6f60 msgr2=0x7fd0201f1ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:16.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.716+0000 7fd026d0e640 1 --2- 172.21.15.125:0/4136884242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0200c6f60 0x7fd0201f1ab0 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7fd0180af040 tx=0x7fd018019770 comp rx=0 tx=0).stop 2023-10-30T13:08:16.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.716+0000 7fd026d0e640 1 -- 172.21.15.125:0/4136884242 shutdown_connections 2023-10-30T13:08:16.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.716+0000 7fd026d0e640 1 --2- 172.21.15.125:0/4136884242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0040664d0 0x7fd004068990 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:16.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.716+0000 7fd026d0e640 1 --2- 172.21.15.125:0/4136884242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0200c6f60 0x7fd0201f1ab0 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:16.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.716+0000 7fd026d0e640 1 --2- 172.21.15.125:0/4136884242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0200c6b80 0x7fd0201f1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:16.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.716+0000 7fd026d0e640 1 --2- 172.21.15.125:0/4136884242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0200c05d0 0x7fd0200c5a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:16.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.716+0000 7fd026d0e640 1 -- 172.21.15.125:0/4136884242 >> 172.21.15.125:0/4136884242 conn(0x7fd0200c7910 msgr2=0x7fd0200c1320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:16.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.716+0000 7fd026d0e640 1 -- 172.21.15.125:0/4136884242 shutdown_connections 2023-10-30T13:08:16.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:16.716+0000 7fd026d0e640 1 -- 172.21.15.125:0/4136884242 wait complete. 2023-10-30T13:08:16.736 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:08:18.737 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:18.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.884+0000 7ff2f4a9e640 1 Processor -- start 2023-10-30T13:08:18.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.884+0000 7ff2f4a9e640 1 -- start start 2023-10-30T13:08:18.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.884+0000 7ff2f4a9e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff2f006f7a0 0x7ff2f006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:18.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.884+0000 7ff2f4a9e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff2f0071110 0x7ff2f0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:18.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.884+0000 7ff2f4a9e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2f0066cf0 con 0x7ff2f00c6bd0 2023-10-30T13:08:18.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.884+0000 7ff2f4a9e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f00700c0 con 0x7ff2f006f7a0 2023-10-30T13:08:18.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.884+0000 7ff2f4a9e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2f00c70d0 con 0x7ff2f0071110 2023-10-30T13:08:18.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.884+0000 7ff2ee59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff2f006f7a0 0x7ff2f006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:18.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.884+0000 7ff2e7fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff2f00c6bd0 0x7ff2f0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47058/0 (socket says 172.21.15.125:47058) 2023-10-30T13:08:18.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.884+0000 7ff2e7fff640 1 -- 172.21.15.125:0/3786323187 learned_addr learned my addr 172.21.15.125:0/3786323187 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:18.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2eed9d640 1 --2- 172.21.15.125:0/3786323187 >> v2:172.21.15.136:3300/0 conn(0x7ff2f0071110 0x7ff2f0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:18.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2ee59c640 1 -- 172.21.15.125:0/3786323187 >> v2:172.21.15.136:3300/0 conn(0x7ff2f0071110 msgr2=0x7ff2f0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:18.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2ee59c640 1 --2- 172.21.15.125:0/3786323187 >> v2:172.21.15.136:3300/0 conn(0x7ff2f0071110 0x7ff2f0158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:18.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2ee59c640 1 -- 172.21.15.125:0/3786323187 >> v1:172.21.15.125:6789/0 conn(0x7ff2f00c6bd0 legacy=0x7ff2f0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:18.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2ee59c640 1 -- 172.21.15.125:0/3786323187 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f00c7390 con 0x7ff2f006f7a0 2023-10-30T13:08:18.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2edd9b640 1 -- 172.21.15.125:0/3786323187 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861569626 0 0) 0x7ff2f0066cf0 con 0x7ff2f00c6bd0 2023-10-30T13:08:18.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2ee59c640 1 --2- 172.21.15.125:0/3786323187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f006f7a0 0x7ff2f006fb80 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7ff2f006cdd0 tx=0x7ff2e001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f1a3ccb2ab2795a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:18.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2edd9b640 1 -- 172.21.15.125:0/3786323187 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e00a8950 con 0x7ff2f006f7a0 2023-10-30T13:08:18.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2edd9b640 1 -- 172.21.15.125:0/3786323187 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e00a8ab0 con 0x7ff2f006f7a0 2023-10-30T13:08:18.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2edd9b640 1 -- 172.21.15.125:0/3786323187 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e00a8e40 con 0x7ff2f006f7a0 2023-10-30T13:08:18.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/3786323187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f006f7a0 msgr2=0x7ff2f006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:18.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2f4a9e640 1 --2- 172.21.15.125:0/3786323187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f006f7a0 0x7ff2f006fb80 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7ff2f006cdd0 tx=0x7ff2e001ec50 comp rx=0 tx=0).stop 2023-10-30T13:08:18.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/3786323187 shutdown_connections 2023-10-30T13:08:18.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2f4a9e640 1 --2- 172.21.15.125:0/3786323187 >> v2:172.21.15.136:3300/0 conn(0x7ff2f0071110 0x7ff2f0158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:18.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2f4a9e640 1 --2- 172.21.15.125:0/3786323187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f006f7a0 0x7ff2f006fb80 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:18.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/3786323187 >> 172.21.15.125:0/3786323187 conn(0x7ff2f00c7910 msgr2=0x7ff2f00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:18.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.888+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/3786323187 shutdown_connections 2023-10-30T13:08:18.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/3786323187 wait complete. 2023-10-30T13:08:18.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2f4a9e640 1 Processor -- start 2023-10-30T13:08:18.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2f4a9e640 1 -- start start 2023-10-30T13:08:18.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2f4a9e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f006f7a0 0x7ff2f01ef0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:18.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2f4a9e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2f0071110 0x7ff2f00c5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:18.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2f4a9e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff2f00c6bd0 0x7ff2f01e3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:18.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2f4a9e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f00bfec0 con 0x7ff2f006f7a0 2023-10-30T13:08:18.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2f4a9e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f01ef610 con 0x7ff2f0071110 2023-10-30T13:08:18.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2f4a9e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f01e1f90 con 0x7ff2f00c6bd0 2023-10-30T13:08:18.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2ee59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f006f7a0 0x7ff2f01ef0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:18.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2ee59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f006f7a0 0x7ff2f01ef0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35470/0 (socket says 172.21.15.125:35470) 2023-10-30T13:08:18.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2ee59c640 1 -- 172.21.15.125:0/1639291087 learned_addr learned my addr 172.21.15.125:0/1639291087 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:18.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2eed9d640 1 --2- 172.21.15.125:0/1639291087 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff2f00c6bd0 0x7ff2f01e3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:18.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2e7fff640 1 --2- 172.21.15.125:0/1639291087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2f0071110 0x7ff2f00c5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:18.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2ee59c640 1 -- 172.21.15.125:0/1639291087 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff2f00c6bd0 msgr2=0x7ff2f01e3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:18.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2ee59c640 1 --2- 172.21.15.125:0/1639291087 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff2f00c6bd0 0x7ff2f01e3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:18.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2ee59c640 1 -- 172.21.15.125:0/1639291087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2f0071110 msgr2=0x7ff2f00c5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:18.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2ee59c640 1 --2- 172.21.15.125:0/1639291087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2f0071110 0x7ff2f00c5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:18.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2ee59c640 1 -- 172.21.15.125:0/1639291087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e0009420 con 0x7ff2f006f7a0 2023-10-30T13:08:18.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2ee59c640 1 --2- 172.21.15.125:0/1639291087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f006f7a0 0x7ff2f01ef0d0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00b3040 tx=0x7ff2e001b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:18.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2e6ffd640 1 -- 172.21.15.125:0/1639291087 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e00b0ea0 con 0x7ff2f006f7a0 2023-10-30T13:08:18.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.892+0000 7ff2e6ffd640 1 -- 172.21.15.125:0/1639291087 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2e00b9410 con 0x7ff2f006f7a0 2023-10-30T13:08:18.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.896+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/1639291087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f0157d60 con 0x7ff2f006f7a0 2023-10-30T13:08:18.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.896+0000 7ff2e6ffd640 1 -- 172.21.15.125:0/1639291087 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2e00b97c0 con 0x7ff2f006f7a0 2023-10-30T13:08:18.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.896+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/1639291087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f01e24c0 con 0x7ff2f006f7a0 2023-10-30T13:08:18.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.896+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/1639291087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f004e170 con 0x7ff2f006f7a0 2023-10-30T13:08:18.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.896+0000 7ff2e6ffd640 1 -- 172.21.15.125:0/1639291087 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff2e00b7040 con 0x7ff2f006f7a0 2023-10-30T13:08:18.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.896+0000 7ff2e6ffd640 1 --2- 172.21.15.125:0/1639291087 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff2d00664d0 0x7ff2d0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:18.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.896+0000 7ff2e6ffd640 1 -- 172.21.15.125:0/1639291087 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7ff2e0122210 con 0x7ff2f006f7a0 2023-10-30T13:08:18.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.896+0000 7ff2e7fff640 1 --2- 172.21.15.125:0/1639291087 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff2d00664d0 0x7ff2d0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:18.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.900+0000 7ff2e7fff640 1 --2- 172.21.15.125:0/1639291087 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff2d00664d0 0x7ff2d0068990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff2f01ec280 tx=0x7ff2d8007b60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:18.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:18.900+0000 7ff2e6ffd640 1 -- 172.21.15.125:0/1639291087 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff2e00eef00 con 0x7ff2f006f7a0 2023-10-30T13:08:19.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:19.080+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/1639291087 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2f006a830 con 0x7ff2f006f7a0 2023-10-30T13:08:19.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:19.084+0000 7ff2e6ffd640 1 -- 172.21.15.125:0/1639291087 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+5755 (secure 0 0 0) 0x7ff2e00b5020 con 0x7ff2f006f7a0 2023-10-30T13:08:19.084 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:19.085 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:15.789388+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:19.086 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 181 2023-10-30T13:08:19.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:19.084+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/1639291087 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff2d00664d0 msgr2=0x7ff2d0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:19.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:19.084+0000 7ff2f4a9e640 1 --2- 172.21.15.125:0/1639291087 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff2d00664d0 0x7ff2d0068990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7ff2f01ec280 tx=0x7ff2d8007b60 comp rx=0 tx=0).stop 2023-10-30T13:08:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:19.084+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/1639291087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f006f7a0 msgr2=0x7ff2f01ef0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:19.084+0000 7ff2f4a9e640 1 --2- 172.21.15.125:0/1639291087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f006f7a0 0x7ff2f01ef0d0 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7ff2e00b3040 tx=0x7ff2e001b9f0 comp rx=0 tx=0).stop 2023-10-30T13:08:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:19.084+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/1639291087 shutdown_connections 2023-10-30T13:08:19.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:19.084+0000 7ff2f4a9e640 1 --2- 172.21.15.125:0/1639291087 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff2d00664d0 0x7ff2d0068990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:19.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:19.084+0000 7ff2f4a9e640 1 --2- 172.21.15.125:0/1639291087 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff2f00c6bd0 0x7ff2f01e3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:19.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:19.084+0000 7ff2f4a9e640 1 --2- 172.21.15.125:0/1639291087 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff2f0071110 0x7ff2f00c5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:19.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:19.084+0000 7ff2f4a9e640 1 --2- 172.21.15.125:0/1639291087 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff2f006f7a0 0x7ff2f01ef0d0 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:19.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:19.084+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/1639291087 >> 172.21.15.125:0/1639291087 conn(0x7ff2f00c7910 msgr2=0x7ff2f00c8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:19.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:19.084+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/1639291087 shutdown_connections 2023-10-30T13:08:19.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:19.084+0000 7ff2f4a9e640 1 -- 172.21.15.125:0/1639291087 wait complete. 2023-10-30T13:08:19.105 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:08:21.106 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:21.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf851d640 1 Processor -- start 2023-10-30T13:08:21.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf851d640 1 -- start start 2023-10-30T13:08:21.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf851d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3cf00c05d0 0x7f3cf013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:21.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf851d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cf00c6f60 0x7f3cf013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:21.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf851d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cf00c0a50 con 0x7f3cf00c6b80 2023-10-30T13:08:21.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf851d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cf006f6e0 con 0x7f3cf00c05d0 2023-10-30T13:08:21.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf851d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf006f9c0 con 0x7f3cf00c6f60 2023-10-30T13:08:21.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf5ab8640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3cf00c05d0 0x7f3cf013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:21.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf5ab8640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3cf00c05d0 0x7f3cf013ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43934/0 (socket says 172.21.15.125:43934) 2023-10-30T13:08:21.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf5ab8640 1 -- 172.21.15.125:0/1265698657 learned_addr learned my addr 172.21.15.125:0/1265698657 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:21.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf6aba640 1 --2- 172.21.15.125:0/1265698657 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cf00c6f60 0x7f3cf013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:21.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf5ab8640 1 -- 172.21.15.125:0/1265698657 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cf00c6f60 msgr2=0x7f3cf013f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:21.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf5ab8640 1 --2- 172.21.15.125:0/1265698657 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cf00c6f60 0x7f3cf013f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:21.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf5ab8640 1 -- 172.21.15.125:0/1265698657 >> v1:172.21.15.136:6789/0 conn(0x7f3cf00c6b80 legacy=0x7f3cf00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:21.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.251+0000 7f3cf5ab8640 1 -- 172.21.15.125:0/1265698657 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cf006fca0 con 0x7f3cf00c05d0 2023-10-30T13:08:21.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.255+0000 7f3cf5ab8640 1 --2- 172.21.15.125:0/1265698657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cf00c05d0 0x7f3cf013ed90 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009870 tx=0x7f3ce001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf6fefb39e43bca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:21.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.255+0000 7f3cf52b7640 1 -- 172.21.15.125:0/1265698657 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce00ac630 con 0x7f3cf00c05d0 2023-10-30T13:08:21.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.255+0000 7f3cf52b7640 1 -- 172.21.15.125:0/1265698657 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce00ac790 con 0x7f3cf00c05d0 2023-10-30T13:08:21.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.255+0000 7f3cf52b7640 1 -- 172.21.15.125:0/1265698657 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce00acb20 con 0x7f3cf00c05d0 2023-10-30T13:08:21.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.255+0000 7f3cf851d640 1 -- 172.21.15.125:0/1265698657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cf00c05d0 msgr2=0x7f3cf013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:21.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.255+0000 7f3cf851d640 1 --2- 172.21.15.125:0/1265698657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cf00c05d0 0x7f3cf013ed90 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009870 tx=0x7f3ce001c500 comp rx=0 tx=0).stop 2023-10-30T13:08:21.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.255+0000 7f3cf851d640 1 -- 172.21.15.125:0/1265698657 shutdown_connections 2023-10-30T13:08:21.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.255+0000 7f3cf851d640 1 --2- 172.21.15.125:0/1265698657 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cf00c6f60 0x7f3cf013f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:21.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.255+0000 7f3cf851d640 1 --2- 172.21.15.125:0/1265698657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cf00c05d0 0x7f3cf013ed90 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:21.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.255+0000 7f3cf851d640 1 -- 172.21.15.125:0/1265698657 >> 172.21.15.125:0/1265698657 conn(0x7f3cf00c7910 msgr2=0x7f3cf00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:21.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.255+0000 7f3cf851d640 1 -- 172.21.15.125:0/1265698657 shutdown_connections 2023-10-30T13:08:21.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.255+0000 7f3cf851d640 1 -- 172.21.15.125:0/1265698657 wait complete. 2023-10-30T13:08:21.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.255+0000 7f3cf851d640 1 Processor -- start 2023-10-30T13:08:21.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf851d640 1 -- start start 2023-10-30T13:08:21.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf851d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3cf00c05d0 0x7f3cf01f3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:21.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf851d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cf00c6b80 0x7f3cf01f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:21.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf851d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cf00c6f60 0x7f3cf0155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:21.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf851d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf01560a0 con 0x7f3cf00c6f60 2023-10-30T13:08:21.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf851d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf00c1260 con 0x7f3cf00c05d0 2023-10-30T13:08:21.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf851d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf00c1770 con 0x7f3cf00c6b80 2023-10-30T13:08:21.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf6aba640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cf00c6f60 0x7f3cf0155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:21.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf6aba640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cf00c6f60 0x7f3cf0155270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43944/0 (socket says 172.21.15.125:43944) 2023-10-30T13:08:21.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf6aba640 1 -- 172.21.15.125:0/3750106114 learned_addr learned my addr 172.21.15.125:0/3750106114 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:21.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf5ab8640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cf00c6b80 0x7f3cf01f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:21.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf62b9640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3cf00c05d0 0x7f3cf01f3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:21.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf6aba640 1 -- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cf00c6b80 msgr2=0x7f3cf01f3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:21.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf6aba640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cf00c6b80 0x7f3cf01f3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:21.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf6aba640 1 -- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3cf00c05d0 msgr2=0x7f3cf01f3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:21.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf6aba640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3cf00c05d0 0x7f3cf01f3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:21.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf6aba640 1 -- 172.21.15.125:0/3750106114 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce00094d0 con 0x7f3cf00c6f60 2023-10-30T13:08:21.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf62b9640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3cf00c05d0 0x7f3cf01f3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:21.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf6aba640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cf00c6f60 0x7f3cf0155270 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8007e60 tx=0x7f3ce800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:21.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3ce6ffd640 1 -- 172.21.15.125:0/3750106114 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8014420 con 0x7f3cf00c6f60 2023-10-30T13:08:21.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3ce6ffd640 1 -- 172.21.15.125:0/3750106114 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce80145a0 con 0x7f3cf00c6f60 2023-10-30T13:08:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3ce6ffd640 1 -- 172.21.15.125:0/3750106114 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce8014950 con 0x7f3cf00c6f60 2023-10-30T13:08:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf851d640 1 -- 172.21.15.125:0/3750106114 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf00c0d30 con 0x7f3cf00c6f60 2023-10-30T13:08:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf851d640 1 -- 172.21.15.125:0/3750106114 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf01f3d20 con 0x7f3cf00c6f60 2023-10-30T13:08:21.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.259+0000 7f3cf851d640 1 -- 172.21.15.125:0/3750106114 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf004e170 con 0x7f3cf00c6f60 2023-10-30T13:08:21.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.263+0000 7f3ce6ffd640 1 -- 172.21.15.125:0/3750106114 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3ce8014ab0 con 0x7f3cf00c6f60 2023-10-30T13:08:21.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.263+0000 7f3ce6ffd640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cd40664a0 0x7f3cd4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:21.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.263+0000 7f3ce6ffd640 1 -- 172.21.15.125:0/3750106114 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f3ce8084eb0 con 0x7f3cf00c6f60 2023-10-30T13:08:21.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.263+0000 7f3cf62b9640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cd40664a0 0x7f3cd4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:21.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.263+0000 7f3cf62b9640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cd40664a0 0x7f3cd4068960 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3cec003ec0 tx=0x7f3cec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:21.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.267+0000 7f3ce6ffd640 1 -- 172.21.15.125:0/3750106114 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3ce8051ba0 con 0x7f3cf00c6f60 2023-10-30T13:08:21.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.447+0000 7f3cf851d640 1 -- 172.21.15.125:0/3750106114 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3cf00c0210 con 0x7f3cf00c6f60 2023-10-30T13:08:21.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.447+0000 7f3ce6ffd640 1 -- 172.21.15.125:0/3750106114 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f3ce80551c0 con 0x7f3cf00c6f60 2023-10-30T13:08:21.449 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:21.450 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:15.789388+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:21.450 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 181 2023-10-30T13:08:21.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.447+0000 7f3cf851d640 1 -- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cd40664a0 msgr2=0x7f3cd4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:21.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.447+0000 7f3cf851d640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cd40664a0 0x7f3cd4068960 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f3cec003ec0 tx=0x7f3cec009040 comp rx=0 tx=0).stop 2023-10-30T13:08:21.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.447+0000 7f3cf851d640 1 -- 172.21.15.125:0/3750106114 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cf00c6f60 msgr2=0x7f3cf0155270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:21.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.447+0000 7f3cf851d640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cf00c6f60 0x7f3cf0155270 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8007e60 tx=0x7f3ce800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:08:21.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.447+0000 7f3cf851d640 1 -- 172.21.15.125:0/3750106114 shutdown_connections 2023-10-30T13:08:21.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.447+0000 7f3cf851d640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3cd40664a0 0x7f3cd4068960 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:21.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.447+0000 7f3cf851d640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3cf00c6f60 0x7f3cf0155270 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:21.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.447+0000 7f3cf851d640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3cf00c6b80 0x7f3cf01f3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:21.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.447+0000 7f3cf851d640 1 --2- 172.21.15.125:0/3750106114 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3cf00c05d0 0x7f3cf01f3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:21.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.447+0000 7f3cf851d640 1 -- 172.21.15.125:0/3750106114 >> 172.21.15.125:0/3750106114 conn(0x7f3cf00c7910 msgr2=0x7f3cf006a890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:21.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.447+0000 7f3cf851d640 1 -- 172.21.15.125:0/3750106114 shutdown_connections 2023-10-30T13:08:21.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:21.447+0000 7f3cf851d640 1 -- 172.21.15.125:0/3750106114 wait complete. 2023-10-30T13:08:21.470 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:08:23.370 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:08:23.374 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:08:23.471 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c25ed6640 1 Processor -- start 2023-10-30T13:08:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c25ed6640 1 -- start start 2023-10-30T13:08:23.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c25ed6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c2015d6f0 0x7f0c2015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:23.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c25ed6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0c200cf160 0x7f0c200cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:23.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c25ed6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c200cb800 con 0x7f0c201538a0 2023-10-30T13:08:23.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c25ed6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c200cb330 con 0x7f0c200cf160 2023-10-30T13:08:23.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c25ed6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c200cad70 con 0x7f0c2015d6f0 2023-10-30T13:08:23.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1f7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0c201538a0 0x7f0c200ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51986/0 (socket says 172.21.15.125:51986) 2023-10-30T13:08:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1f7fe640 1 -- 172.21.15.125:0/1515728941 learned_addr learned my addr 172.21.15.125:0/1515728941 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1effd640 1 --2- 172.21.15.125:0/1515728941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c2015d6f0 0x7f0c2015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1ffff640 1 --2- 172.21.15.125:0/1515728941 >> v2:172.21.15.136:3300/0 conn(0x7f0c200cf160 0x7f0c200cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 -- 172.21.15.125:0/1515728941 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096266243 0 0) 0x7f0c200cb800 con 0x7f0c201538a0 2023-10-30T13:08:23.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 -- 172.21.15.125:0/1515728941 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c08003530 con 0x7f0c201538a0 2023-10-30T13:08:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 -- 172.21.15.125:0/1515728941 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2030990968 0 0) 0x7f0c08003530 con 0x7f0c201538a0 2023-10-30T13:08:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 -- 172.21.15.125:0/1515728941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c2015d6f0 msgr2=0x7f0c2015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 --2- 172.21.15.125:0/1515728941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c2015d6f0 0x7f0c2015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 -- 172.21.15.125:0/1515728941 >> v2:172.21.15.136:3300/0 conn(0x7f0c200cf160 msgr2=0x7f0c200cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:23.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 --2- 172.21.15.125:0/1515728941 >> v2:172.21.15.136:3300/0 conn(0x7f0c200cf160 0x7f0c200cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 -- 172.21.15.125:0/1515728941 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c200cb050 con 0x7f0c201538a0 2023-10-30T13:08:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 -- 172.21.15.125:0/1515728941 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f0c10001680 con 0x7f0c201538a0 2023-10-30T13:08:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 -- 172.21.15.125:0/1515728941 >> v1:172.21.15.125:6789/0 conn(0x7f0c201538a0 legacy=0x7f0c200ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:23.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 --2- 172.21.15.125:0/1515728941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c08003980 0x7f0c08003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:23.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 --2- 172.21.15.125:0/1515728941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c2015d6f0 0x7f0c080047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:23.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 --2- 172.21.15.125:0/1515728941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c08004d10 0x7f0c08005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:23.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 -- 172.21.15.125:0/1515728941 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c200cb330 con 0x7f0c08004d10 2023-10-30T13:08:23.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 -- 172.21.15.125:0/1515728941 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c200cad70 con 0x7f0c08003980 2023-10-30T13:08:23.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1e7fc640 1 -- 172.21.15.125:0/1515728941 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0800b990 con 0x7f0c2015d6f0 2023-10-30T13:08:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.615+0000 7f0c1ffff640 1 --2- 172.21.15.125:0/1515728941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c08004d10 0x7f0c08005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c1effd640 1 --2- 172.21.15.125:0/1515728941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c2015d6f0 0x7f0c080047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:23.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c1f7fe640 1 --2- 172.21.15.125:0/1515728941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c08003980 0x7f0c08003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c1ffff640 1 -- 172.21.15.125:0/1515728941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c2015d6f0 msgr2=0x7f0c080047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c1ffff640 1 --2- 172.21.15.125:0/1515728941 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c2015d6f0 0x7f0c080047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:23.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c1ffff640 1 -- 172.21.15.125:0/1515728941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c08003980 msgr2=0x7f0c08003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:23.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c1ffff640 1 --2- 172.21.15.125:0/1515728941 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c08003980 0x7f0c08003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:23.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c1ffff640 1 -- 172.21.15.125:0/1515728941 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c08003530 con 0x7f0c08004d10 2023-10-30T13:08:23.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c1ffff640 1 --2- 172.21.15.125:0/1515728941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c08004d10 0x7f0c08005190 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f0c18009fc0 tx=0x7f0c1801cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=752fe529e21e7c97 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:23.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c1e7fc640 1 -- 172.21.15.125:0/1515728941 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c180a5cb0 con 0x7f0c08004d10 2023-10-30T13:08:23.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c1e7fc640 1 -- 172.21.15.125:0/1515728941 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1800f040 con 0x7f0c08004d10 2023-10-30T13:08:23.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c1e7fc640 1 -- 172.21.15.125:0/1515728941 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c180ad680 con 0x7f0c08004d10 2023-10-30T13:08:23.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c25ed6640 1 -- 172.21.15.125:0/1515728941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c08004d10 msgr2=0x7f0c08005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:23.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c25ed6640 1 --2- 172.21.15.125:0/1515728941 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c08004d10 0x7f0c08005190 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f0c18009fc0 tx=0x7f0c1801cca0 comp rx=0 tx=0).stop 2023-10-30T13:08:23.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c1ffff640 1 -- 172.21.15.125:0/1515728941 reap_dead start 2023-10-30T13:08:23.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c25ed6640 1 -- 172.21.15.125:0/1515728941 shutdown_connections 2023-10-30T13:08:23.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c25ed6640 1 -- 172.21.15.125:0/1515728941 >> 172.21.15.125:0/1515728941 conn(0x7f0c2006bfe0 msgr2=0x7f0c2006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:23.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.619+0000 7f0c25ed6640 1 -- 172.21.15.125:0/1515728941 shutdown_connections 2023-10-30T13:08:23.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c25ed6640 1 -- 172.21.15.125:0/1515728941 wait complete. 2023-10-30T13:08:23.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c25ed6640 1 Processor -- start 2023-10-30T13:08:23.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c25ed6640 1 -- start start 2023-10-30T13:08:23.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c25ed6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c08004d10 0x7f0c200c64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:23.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c25ed6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c20156210 0x7f0c201569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:23.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c25ed6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c201ef2f0 0x7f0c201ef770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:23.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c25ed6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c200c6ed0 con 0x7f0c08004d10 2023-10-30T13:08:23.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c25ed6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c200c8030 con 0x7f0c20156210 2023-10-30T13:08:23.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c25ed6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c20155b50 con 0x7f0c201ef2f0 2023-10-30T13:08:23.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c1f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c08004d10 0x7f0c200c64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:23.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c1f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c08004d10 0x7f0c200c64d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43966/0 (socket says 172.21.15.125:43966) 2023-10-30T13:08:23.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c1f7fe640 1 -- 172.21.15.125:0/3654594601 learned_addr learned my addr 172.21.15.125:0/3654594601 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:23.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c1effd640 1 --2- 172.21.15.125:0/3654594601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c20156210 0x7f0c201569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:23.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c1ffff640 1 --2- 172.21.15.125:0/3654594601 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c201ef2f0 0x7f0c201ef770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:23.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c1f7fe640 1 -- 172.21.15.125:0/3654594601 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c201ef2f0 msgr2=0x7f0c201ef770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:23.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c1f7fe640 1 --2- 172.21.15.125:0/3654594601 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c201ef2f0 0x7f0c201ef770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:23.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c1f7fe640 1 -- 172.21.15.125:0/3654594601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c20156210 msgr2=0x7f0c201569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:23.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c1f7fe640 1 --2- 172.21.15.125:0/3654594601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c20156210 0x7f0c201569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:23.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c1f7fe640 1 -- 172.21.15.125:0/3654594601 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c20155e30 con 0x7f0c08004d10 2023-10-30T13:08:23.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c1f7fe640 1 --2- 172.21.15.125:0/3654594601 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c08004d10 0x7f0c200c64d0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f0c1000e7e0 tx=0x7f0c1000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:23.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c07fff640 1 -- 172.21.15.125:0/3654594601 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1000c8d0 con 0x7f0c08004d10 2023-10-30T13:08:23.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c07fff640 1 -- 172.21.15.125:0/3654594601 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c1000ca50 con 0x7f0c08004d10 2023-10-30T13:08:23.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c25ed6640 1 -- 172.21.15.125:0/3654594601 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c200c82a0 con 0x7f0c08004d10 2023-10-30T13:08:23.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.623+0000 7f0c07fff640 1 -- 172.21.15.125:0/3654594601 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c1000ce00 con 0x7f0c08004d10 2023-10-30T13:08:23.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.627+0000 7f0c25ed6640 1 -- 172.21.15.125:0/3654594601 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c200c8860 con 0x7f0c08004d10 2023-10-30T13:08:23.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.627+0000 7f0c25ed6640 1 -- 172.21.15.125:0/3654594601 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c2004e170 con 0x7f0c08004d10 2023-10-30T13:08:23.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.627+0000 7f0c07fff640 1 -- 172.21.15.125:0/3654594601 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0c1001d460 con 0x7f0c08004d10 2023-10-30T13:08:23.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.627+0000 7f0c07fff640 1 --2- 172.21.15.125:0/3654594601 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c0c0664a0 0x7f0c0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:23.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.627+0000 7f0c07fff640 1 -- 172.21.15.125:0/3654594601 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f0c100846f0 con 0x7f0c08004d10 2023-10-30T13:08:23.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.627+0000 7f0c1effd640 1 --2- 172.21.15.125:0/3654594601 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c0c0664a0 0x7f0c0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:23.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.631+0000 7f0c1effd640 1 --2- 172.21.15.125:0/3654594601 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c0c0664a0 0x7f0c0c068960 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f0c20156710 tx=0x7f0c14008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:23.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.631+0000 7f0c07fff640 1 -- 172.21.15.125:0/3654594601 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0c10051350 con 0x7f0c08004d10 2023-10-30T13:08:23.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.811+0000 7f0c25ed6640 1 -- 172.21.15.125:0/3654594601 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c200b1dd0 con 0x7f0c08004d10 2023-10-30T13:08:23.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.815+0000 7f0c07fff640 1 -- 172.21.15.125:0/3654594601 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5752 (secure 0 0 0) 0x7f0c10054970 con 0x7f0c08004d10 2023-10-30T13:08:23.817 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:23.817 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:21.695915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:23.818 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 182 2023-10-30T13:08:23.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.815+0000 7f0c25ed6640 1 -- 172.21.15.125:0/3654594601 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c0c0664a0 msgr2=0x7f0c0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:23.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.815+0000 7f0c25ed6640 1 --2- 172.21.15.125:0/3654594601 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c0c0664a0 0x7f0c0c068960 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f0c20156710 tx=0x7f0c14008040 comp rx=0 tx=0).stop 2023-10-30T13:08:23.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.815+0000 7f0c25ed6640 1 -- 172.21.15.125:0/3654594601 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c08004d10 msgr2=0x7f0c200c64d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:23.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.815+0000 7f0c25ed6640 1 --2- 172.21.15.125:0/3654594601 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c08004d10 0x7f0c200c64d0 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f0c1000e7e0 tx=0x7f0c1000eca0 comp rx=0 tx=0).stop 2023-10-30T13:08:23.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.815+0000 7f0c25ed6640 1 -- 172.21.15.125:0/3654594601 shutdown_connections 2023-10-30T13:08:23.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.815+0000 7f0c25ed6640 1 --2- 172.21.15.125:0/3654594601 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c0c0664a0 0x7f0c0c068960 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:23.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.815+0000 7f0c25ed6640 1 --2- 172.21.15.125:0/3654594601 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c201ef2f0 0x7f0c201ef770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:23.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.815+0000 7f0c25ed6640 1 --2- 172.21.15.125:0/3654594601 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c20156210 0x7f0c201569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:23.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.815+0000 7f0c25ed6640 1 --2- 172.21.15.125:0/3654594601 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c08004d10 0x7f0c200c64d0 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:23.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.815+0000 7f0c25ed6640 1 -- 172.21.15.125:0/3654594601 >> 172.21.15.125:0/3654594601 conn(0x7f0c2006bfe0 msgr2=0x7f0c200c5390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:23.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.815+0000 7f0c25ed6640 1 -- 172.21.15.125:0/3654594601 shutdown_connections 2023-10-30T13:08:23.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:23.815+0000 7f0c25ed6640 1 -- 172.21.15.125:0/3654594601 wait complete. 2023-10-30T13:08:23.837 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:08:25.838 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:25.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb9719d640 1 Processor -- start 2023-10-30T13:08:25.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb9719d640 1 -- start start 2023-10-30T13:08:25.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb9719d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb9006f7a0 0x7fbb9006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:25.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb9719d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb900c6bd0 0x7fbb90069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:25.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb9719d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9006a3d0 con 0x7fbb90071110 2023-10-30T13:08:25.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb9719d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb900700c0 con 0x7fbb9006f7a0 2023-10-30T13:08:25.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb9719d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb90066cf0 con 0x7fbb900c6bd0 2023-10-30T13:08:25.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb94f39640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb9006f7a0 0x7fbb9006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:25.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb94f39640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbb9006f7a0 0x7fbb9006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43976/0 (socket says 172.21.15.125:43976) 2023-10-30T13:08:25.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb94f39640 1 -- 172.21.15.125:0/1404938595 learned_addr learned my addr 172.21.15.125:0/1404938595 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:25.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb87fff640 1 --2- 172.21.15.125:0/1404938595 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb900c6bd0 0x7fbb90069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:25.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb94f39640 1 -- 172.21.15.125:0/1404938595 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb900c6bd0 msgr2=0x7fbb90069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:25.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb94f39640 1 --2- 172.21.15.125:0/1404938595 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb900c6bd0 0x7fbb90069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:25.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb94f39640 1 -- 172.21.15.125:0/1404938595 >> v1:172.21.15.136:6789/0 conn(0x7fbb90071110 legacy=0x7fbb90158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:25.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.983+0000 7fbb94f39640 1 -- 172.21.15.125:0/1404938595 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb900c70d0 con 0x7fbb9006f7a0 2023-10-30T13:08:25.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.987+0000 7fbb94f39640 1 --2- 172.21.15.125:0/1404938595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9006f7a0 0x7fbb9006fb80 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009870 tx=0x7fbb8c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa323f580916458 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:25.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.987+0000 7fbb877fe640 1 -- 172.21.15.125:0/1404938595 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c0a8a30 con 0x7fbb9006f7a0 2023-10-30T13:08:25.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.987+0000 7fbb877fe640 1 -- 172.21.15.125:0/1404938595 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8c0a8b90 con 0x7fbb9006f7a0 2023-10-30T13:08:25.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.987+0000 7fbb877fe640 1 -- 172.21.15.125:0/1404938595 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c0b0440 con 0x7fbb9006f7a0 2023-10-30T13:08:25.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.987+0000 7fbb9719d640 1 -- 172.21.15.125:0/1404938595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9006f7a0 msgr2=0x7fbb9006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:25.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.987+0000 7fbb9719d640 1 --2- 172.21.15.125:0/1404938595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9006f7a0 0x7fbb9006fb80 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c009870 tx=0x7fbb8c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:08:25.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.987+0000 7fbb9719d640 1 -- 172.21.15.125:0/1404938595 shutdown_connections 2023-10-30T13:08:25.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.987+0000 7fbb9719d640 1 --2- 172.21.15.125:0/1404938595 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb900c6bd0 0x7fbb90069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:25.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.987+0000 7fbb9719d640 1 --2- 172.21.15.125:0/1404938595 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb9006f7a0 0x7fbb9006fb80 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:25.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.987+0000 7fbb9719d640 1 -- 172.21.15.125:0/1404938595 >> 172.21.15.125:0/1404938595 conn(0x7fbb900c7910 msgr2=0x7fbb900c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:25.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.987+0000 7fbb9719d640 1 -- 172.21.15.125:0/1404938595 shutdown_connections 2023-10-30T13:08:25.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.987+0000 7fbb9719d640 1 -- 172.21.15.125:0/1404938595 wait complete. 2023-10-30T13:08:25.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb9719d640 1 Processor -- start 2023-10-30T13:08:25.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb9719d640 1 -- start start 2023-10-30T13:08:25.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb9719d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb9006f7a0 0x7fbb901f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:25.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb9719d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb90071110 0x7fbb900adb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:25.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb9719d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb900c6bd0 0x7fbb900aead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:25.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb9719d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb901effa0 con 0x7fbb90071110 2023-10-30T13:08:25.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb9719d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb90158d50 con 0x7fbb9006f7a0 2023-10-30T13:08:25.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb9719d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb901f34a0 con 0x7fbb900c6bd0 2023-10-30T13:08:25.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb87fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb90071110 0x7fbb900adb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:25.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb87fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb90071110 0x7fbb900adb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43978/0 (socket says 172.21.15.125:43978) 2023-10-30T13:08:25.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb87fff640 1 -- 172.21.15.125:0/1967831846 learned_addr learned my addr 172.21.15.125:0/1967831846 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:25.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb94f39640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb9006f7a0 0x7fbb901f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:25.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb9573a640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb900c6bd0 0x7fbb900aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:25.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb87fff640 1 -- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb900c6bd0 msgr2=0x7fbb900aead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:25.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb87fff640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb900c6bd0 0x7fbb900aead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:25.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb87fff640 1 -- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb9006f7a0 msgr2=0x7fbb901f2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:25.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb87fff640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb9006f7a0 0x7fbb901f2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:25.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb87fff640 1 -- 172.21.15.125:0/1967831846 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8c0094d0 con 0x7fbb90071110 2023-10-30T13:08:25.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb94f39640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb9006f7a0 0x7fbb901f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:25.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb87fff640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb90071110 0x7fbb900adb30 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000d5f0 tx=0x7fbb8000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:25.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb857fa640 1 -- 172.21.15.125:0/1967831846 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb800125f0 con 0x7fbb90071110 2023-10-30T13:08:25.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb857fa640 1 -- 172.21.15.125:0/1967831846 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb80012750 con 0x7fbb90071110 2023-10-30T13:08:25.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb857fa640 1 -- 172.21.15.125:0/1967831846 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb80012b00 con 0x7fbb90071110 2023-10-30T13:08:26.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb9719d640 1 -- 172.21.15.125:0/1967831846 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb901ef180 con 0x7fbb90071110 2023-10-30T13:08:26.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.991+0000 7fbb9719d640 1 -- 172.21.15.125:0/1967831846 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb901ef700 con 0x7fbb90071110 2023-10-30T13:08:26.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.995+0000 7fbb9719d640 1 -- 172.21.15.125:0/1967831846 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb9004e170 con 0x7fbb90071110 2023-10-30T13:08:26.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.995+0000 7fbb857fa640 1 -- 172.21.15.125:0/1967831846 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbb80012c60 con 0x7fbb90071110 2023-10-30T13:08:26.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.995+0000 7fbb857fa640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb7c0664a0 0x7fbb7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:26.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.995+0000 7fbb857fa640 1 -- 172.21.15.125:0/1967831846 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7fbb80084b80 con 0x7fbb90071110 2023-10-30T13:08:26.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.995+0000 7fbb94f39640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb7c0664a0 0x7fbb7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:26.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.995+0000 7fbb94f39640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb7c0664a0 0x7fbb7c068960 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0b2000 tx=0x7fbb8c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:26.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:25.999+0000 7fbb857fa640 1 -- 172.21.15.125:0/1967831846 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbb800517f0 con 0x7fbb90071110 2023-10-30T13:08:26.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:26.179+0000 7fbb9719d640 1 -- 172.21.15.125:0/1967831846 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb900c4d90 con 0x7fbb90071110 2023-10-30T13:08:26.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:26.179+0000 7fbb857fa640 1 -- 172.21.15.125:0/1967831846 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5752 (secure 0 0 0) 0x7fbb80054e10 con 0x7fbb90071110 2023-10-30T13:08:26.183 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:26.183 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:21.695915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:26.183 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 182 2023-10-30T13:08:26.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:26.183+0000 7fbb9719d640 1 -- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb7c0664a0 msgr2=0x7fbb7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:26.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:26.183+0000 7fbb9719d640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb7c0664a0 0x7fbb7c068960 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0b2000 tx=0x7fbb8c008040 comp rx=0 tx=0).stop 2023-10-30T13:08:26.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:26.183+0000 7fbb9719d640 1 -- 172.21.15.125:0/1967831846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb90071110 msgr2=0x7fbb900adb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:26.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:26.183+0000 7fbb9719d640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb90071110 0x7fbb900adb30 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000d5f0 tx=0x7fbb8000dab0 comp rx=0 tx=0).stop 2023-10-30T13:08:26.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:26.183+0000 7fbb9719d640 1 -- 172.21.15.125:0/1967831846 shutdown_connections 2023-10-30T13:08:26.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:26.183+0000 7fbb9719d640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbb7c0664a0 0x7fbb7c068960 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:26.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:26.183+0000 7fbb9719d640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbb900c6bd0 0x7fbb900aead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:26.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:26.183+0000 7fbb9719d640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbb90071110 0x7fbb900adb30 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:26.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:26.183+0000 7fbb9719d640 1 --2- 172.21.15.125:0/1967831846 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbb9006f7a0 0x7fbb901f2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:26.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:26.183+0000 7fbb9719d640 1 -- 172.21.15.125:0/1967831846 >> 172.21.15.125:0/1967831846 conn(0x7fbb900c7910 msgr2=0x7fbb900c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:26.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:26.183+0000 7fbb9719d640 1 -- 172.21.15.125:0/1967831846 shutdown_connections 2023-10-30T13:08:26.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:26.183+0000 7fbb9719d640 1 -- 172.21.15.125:0/1967831846 wait complete. 2023-10-30T13:08:26.204 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 182, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 182, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_6017": { "addr": "172.21.15.136:6839/223464319", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 223464319, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 223464319, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 6017, "incarnation": 170, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 251 }, "gid_8301": { "addr": "172.21.15.125:6837/2540063352", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2540063352, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2540063352, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8301, "incarnation": 168, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_8529": { "addr": "172.21.15.125:6836/3950044434", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3950044434, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3950044434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8529, "incarnation": 179, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:rejoin", "state_seq": 13 } }, "last_failure": 0, "last_failure_osd_epoch": 76, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:08:21.695915+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8529, "mds_1": 8301, "mds_2": 6017 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:08:28.205 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:28.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.339+0000 7f4be130e640 1 Processor -- start 2023-10-30T13:08:28.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.343+0000 7f4be130e640 1 -- start start 2023-10-30T13:08:28.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.343+0000 7f4be130e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4bdc15d6f0 0x7f4bdc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:28.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.343+0000 7f4be130e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4bdc0cf160 0x7f4bdc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:28.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.343+0000 7f4be130e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bdc0cb800 con 0x7f4bdc1538a0 2023-10-30T13:08:28.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.343+0000 7f4be130e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bdc0cb330 con 0x7f4bdc0cf160 2023-10-30T13:08:28.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.343+0000 7f4be130e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc0cad70 con 0x7f4bdc15d6f0 2023-10-30T13:08:28.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.343+0000 7f4bdb7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4bdc0cf160 0x7f4bdc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:28.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.343+0000 7f4bdb7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4bdc0cf160 0x7f4bdc0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43980/0 (socket says 172.21.15.125:43980) 2023-10-30T13:08:28.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.343+0000 7f4bdb7fe640 1 -- 172.21.15.125:0/2901606915 learned_addr learned my addr 172.21.15.125:0/2901606915 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:28.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.343+0000 7f4bda7fc640 1 --2- 172.21.15.125:0/2901606915 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4bdc15d6f0 0x7f4bdc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:28.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.343+0000 7f4bdb7fe640 1 -- 172.21.15.125:0/2901606915 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4bdc15d6f0 msgr2=0x7f4bdc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:28.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.343+0000 7f4bdb7fe640 1 --2- 172.21.15.125:0/2901606915 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4bdc15d6f0 0x7f4bdc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:28.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.343+0000 7f4bdb7fe640 1 -- 172.21.15.125:0/2901606915 >> v1:172.21.15.136:6789/0 conn(0x7f4bdc1538a0 legacy=0x7f4bdc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:28.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.343+0000 7f4bdb7fe640 1 -- 172.21.15.125:0/2901606915 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bdc0cb050 con 0x7f4bdc0cf160 2023-10-30T13:08:28.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.347+0000 7f4bdb7fe640 1 --2- 172.21.15.125:0/2901606915 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4bdc0cf160 0x7f4bdc0cf590 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009e50 tx=0x7f4bd401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a47091f52392ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:28.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.347+0000 7f4bd9ffb640 1 -- 172.21.15.125:0/2901606915 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd40af400 con 0x7f4bdc0cf160 2023-10-30T13:08:28.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.347+0000 7f4bd9ffb640 1 -- 172.21.15.125:0/2901606915 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd40ab050 con 0x7f4bdc0cf160 2023-10-30T13:08:28.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.347+0000 7f4bd9ffb640 1 -- 172.21.15.125:0/2901606915 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd40af780 con 0x7f4bdc0cf160 2023-10-30T13:08:28.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.347+0000 7f4be130e640 1 -- 172.21.15.125:0/2901606915 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4bdc0cf160 msgr2=0x7f4bdc0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:28.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.347+0000 7f4be130e640 1 --2- 172.21.15.125:0/2901606915 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4bdc0cf160 0x7f4bdc0cf590 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7f4bd4009e50 tx=0x7f4bd401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:08:28.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.347+0000 7f4be130e640 1 -- 172.21.15.125:0/2901606915 shutdown_connections 2023-10-30T13:08:28.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.347+0000 7f4be130e640 1 --2- 172.21.15.125:0/2901606915 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4bdc0cf160 0x7f4bdc0cf590 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:28.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.347+0000 7f4be130e640 1 --2- 172.21.15.125:0/2901606915 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4bdc15d6f0 0x7f4bdc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:28.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.347+0000 7f4be130e640 1 -- 172.21.15.125:0/2901606915 >> 172.21.15.125:0/2901606915 conn(0x7f4bdc06bfe0 msgr2=0x7f4bdc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:28.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.347+0000 7f4be130e640 1 -- 172.21.15.125:0/2901606915 shutdown_connections 2023-10-30T13:08:28.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.347+0000 7f4be130e640 1 -- 172.21.15.125:0/2901606915 wait complete. 2023-10-30T13:08:28.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.347+0000 7f4be130e640 1 Processor -- start 2023-10-30T13:08:28.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.347+0000 7f4be130e640 1 -- start start 2023-10-30T13:08:28.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4be130e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bdc0cf160 0x7f4bdc15b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:28.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4be130e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4bdc1538a0 0x7f4bdc0c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:28.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4be130e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4bdc15d6f0 0x7f4bdc0c8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:28.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4be130e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc1f24d0 con 0x7f4bdc1538a0 2023-10-30T13:08:28.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4be130e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc0b4ee0 con 0x7f4bdc0cf160 2023-10-30T13:08:28.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4be130e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bdc0c9290 con 0x7f4bdc15d6f0 2023-10-30T13:08:28.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bda7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4bdc1538a0 0x7f4bdc0c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:28.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bda7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4bdc1538a0 0x7f4bdc0c7ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:43992/0 (socket says 172.21.15.125:43992) 2023-10-30T13:08:28.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bda7fc640 1 -- 172.21.15.125:0/1197934151 learned_addr learned my addr 172.21.15.125:0/1197934151 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:28.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bdb7fe640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4bdc15d6f0 0x7f4bdc0c8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:28.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bdaffd640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bdc0cf160 0x7f4bdc15b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:28.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bda7fc640 1 -- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4bdc15d6f0 msgr2=0x7f4bdc0c8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:28.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bda7fc640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4bdc15d6f0 0x7f4bdc0c8400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:28.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bda7fc640 1 -- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bdc0cf160 msgr2=0x7f4bdc15b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:28.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bda7fc640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bdc0cf160 0x7f4bdc15b2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:28.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bda7fc640 1 -- 172.21.15.125:0/1197934151 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd4009b30 con 0x7f4bdc1538a0 2023-10-30T13:08:28.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bdb7fe640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4bdc15d6f0 0x7f4bdc0c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:28.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bda7fc640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4bdc1538a0 0x7f4bdc0c7ec0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000d5f0 tx=0x7f4bd000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:28.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bc37fe640 1 -- 172.21.15.125:0/1197934151 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd00125f0 con 0x7f4bdc1538a0 2023-10-30T13:08:28.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bc37fe640 1 -- 172.21.15.125:0/1197934151 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bd0012750 con 0x7f4bdc1538a0 2023-10-30T13:08:28.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4be130e640 1 -- 172.21.15.125:0/1197934151 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bdc1f26e0 con 0x7f4bdc1538a0 2023-10-30T13:08:28.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bc37fe640 1 -- 172.21.15.125:0/1197934151 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0012ae0 con 0x7f4bdc1538a0 2023-10-30T13:08:28.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4be130e640 1 -- 172.21.15.125:0/1197934151 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bdc1f2ca0 con 0x7f4bdc1538a0 2023-10-30T13:08:28.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4be130e640 1 -- 172.21.15.125:0/1197934151 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bdc0b2a50 con 0x7f4bdc1538a0 2023-10-30T13:08:28.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.351+0000 7f4bc37fe640 1 -- 172.21.15.125:0/1197934151 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4bd0012c40 con 0x7f4bdc1538a0 2023-10-30T13:08:28.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.355+0000 7f4bc37fe640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bc80664a0 0x7f4bc8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:28.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.355+0000 7f4bc37fe640 1 -- 172.21.15.125:0/1197934151 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f4bd0084ab0 con 0x7f4bdc1538a0 2023-10-30T13:08:28.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.355+0000 7f4bdaffd640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bc80664a0 0x7f4bc8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:28.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.355+0000 7f4bdaffd640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bc80664a0 0x7f4bc8068960 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc004e30 tx=0x7f4bcc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:28.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.359+0000 7f4bc37fe640 1 -- 172.21.15.125:0/1197934151 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4bd00517a0 con 0x7f4bdc1538a0 2023-10-30T13:08:28.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.567+0000 7f4be130e640 1 -- 172.21.15.125:0/1197934151 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4bdc04e170 con 0x7f4bdc1538a0 2023-10-30T13:08:28.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.567+0000 7f4bc37fe640 1 -- 172.21.15.125:0/1197934151 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+5752 (secure 0 0 0) 0x7f4bd0007380 con 0x7f4bdc1538a0 2023-10-30T13:08:28.572 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:28.572 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":182,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:21.695915+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:28.572 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 182 2023-10-30T13:08:28.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.571+0000 7f4be130e640 1 -- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bc80664a0 msgr2=0x7f4bc8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:28.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.571+0000 7f4be130e640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bc80664a0 0x7f4bc8068960 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc004e30 tx=0x7f4bcc009040 comp rx=0 tx=0).stop 2023-10-30T13:08:28.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.571+0000 7f4be130e640 1 -- 172.21.15.125:0/1197934151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4bdc1538a0 msgr2=0x7f4bdc0c7ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:28.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.571+0000 7f4be130e640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4bdc1538a0 0x7f4bdc0c7ec0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000d5f0 tx=0x7f4bd000dab0 comp rx=0 tx=0).stop 2023-10-30T13:08:28.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.571+0000 7f4be130e640 1 -- 172.21.15.125:0/1197934151 shutdown_connections 2023-10-30T13:08:28.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.571+0000 7f4be130e640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4bc80664a0 0x7f4bc8068960 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:28.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.571+0000 7f4be130e640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4bdc15d6f0 0x7f4bdc0c8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:28.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.571+0000 7f4be130e640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4bdc1538a0 0x7f4bdc0c7ec0 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:28.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.571+0000 7f4be130e640 1 --2- 172.21.15.125:0/1197934151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4bdc0cf160 0x7f4bdc15b2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:28.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.571+0000 7f4be130e640 1 -- 172.21.15.125:0/1197934151 >> 172.21.15.125:0/1197934151 conn(0x7f4bdc06bfe0 msgr2=0x7f4bdc0ca0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:28.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.571+0000 7f4be130e640 1 -- 172.21.15.125:0/1197934151 shutdown_connections 2023-10-30T13:08:28.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:28.571+0000 7f4be130e640 1 -- 172.21.15.125:0/1197934151 wait complete. 2023-10-30T13:08:28.593 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:08:30.594 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:30.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.739+0000 7f485e691640 1 Processor -- start 2023-10-30T13:08:30.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.739+0000 7f485e691640 1 -- start start 2023-10-30T13:08:30.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.743+0000 7f485e691640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f48580c6b80 0x7f48580bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:30.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.743+0000 7f485e691640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f48580c6f60 0x7f48580c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:30.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.743+0000 7f485e691640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48580c0a50 con 0x7f48580c8430 2023-10-30T13:08:30.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.743+0000 7f485e691640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485806f6e0 con 0x7f48580c6f60 2023-10-30T13:08:30.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.743+0000 7f485e691640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f485806f9c0 con 0x7f48580c6b80 2023-10-30T13:08:30.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.743+0000 7f48577fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f48580c6f60 0x7f48580c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:30.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.743+0000 7f48577fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f48580c6f60 0x7f48580c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44000/0 (socket says 172.21.15.125:44000) 2023-10-30T13:08:30.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.743+0000 7f48577fe640 1 -- 172.21.15.125:0/184424976 learned_addr learned my addr 172.21.15.125:0/184424976 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:30.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.743+0000 7f4857fff640 1 --2- 172.21.15.125:0/184424976 >> v2:172.21.15.136:3300/0 conn(0x7f48580c6b80 0x7f48580bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:30.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.743+0000 7f48577fe640 1 -- 172.21.15.125:0/184424976 >> v2:172.21.15.136:3300/0 conn(0x7f48580c6b80 msgr2=0x7f48580bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:30.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.743+0000 7f48577fe640 1 --2- 172.21.15.125:0/184424976 >> v2:172.21.15.136:3300/0 conn(0x7f48580c6b80 0x7f48580bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:30.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.743+0000 7f48577fe640 1 -- 172.21.15.125:0/184424976 >> v1:172.21.15.136:6789/0 conn(0x7f48580c8430 legacy=0x7f485813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:30.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.743+0000 7f48577fe640 1 -- 172.21.15.125:0/184424976 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485806fca0 con 0x7f48580c6f60 2023-10-30T13:08:30.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.747+0000 7f48577fe640 1 --2- 172.21.15.125:0/184424976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48580c6f60 0x7f48580c0400 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f485806f1b0 tx=0x7f484c01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad3f72d58c3717a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:30.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.747+0000 7f4856ffd640 1 -- 172.21.15.125:0/184424976 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c0ab610 con 0x7f48580c6f60 2023-10-30T13:08:30.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.747+0000 7f4856ffd640 1 -- 172.21.15.125:0/184424976 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484c0ab770 con 0x7f48580c6f60 2023-10-30T13:08:30.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.747+0000 7f4856ffd640 1 -- 172.21.15.125:0/184424976 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484c0abb00 con 0x7f48580c6f60 2023-10-30T13:08:30.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.747+0000 7f485e691640 1 -- 172.21.15.125:0/184424976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48580c6f60 msgr2=0x7f48580c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:30.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.747+0000 7f485e691640 1 --2- 172.21.15.125:0/184424976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48580c6f60 0x7f48580c0400 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f485806f1b0 tx=0x7f484c01c500 comp rx=0 tx=0).stop 2023-10-30T13:08:30.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.747+0000 7f485e691640 1 -- 172.21.15.125:0/184424976 shutdown_connections 2023-10-30T13:08:30.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.747+0000 7f485e691640 1 --2- 172.21.15.125:0/184424976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48580c6f60 0x7f48580c0400 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:30.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.747+0000 7f485e691640 1 --2- 172.21.15.125:0/184424976 >> v2:172.21.15.136:3300/0 conn(0x7f48580c6b80 0x7f48580bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:30.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.747+0000 7f485e691640 1 -- 172.21.15.125:0/184424976 >> 172.21.15.125:0/184424976 conn(0x7f48580c7910 msgr2=0x7f48580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:30.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.747+0000 7f485e691640 1 -- 172.21.15.125:0/184424976 shutdown_connections 2023-10-30T13:08:30.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.747+0000 7f485e691640 1 -- 172.21.15.125:0/184424976 wait complete. 2023-10-30T13:08:30.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.747+0000 7f485e691640 1 Processor -- start 2023-10-30T13:08:30.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.747+0000 7f485e691640 1 -- start start 2023-10-30T13:08:30.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f485e691640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48580c6b80 0x7f48581f1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:30.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f485e691640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48580c6f60 0x7f48581604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:30.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f485e691640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f48580c8430 0x7f48581366a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:30.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f485e691640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f485813dc00 con 0x7f48580c6b80 2023-10-30T13:08:30.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f485e691640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48581f5bf0 con 0x7f48580c6f60 2023-10-30T13:08:30.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f485e691640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48581634e0 con 0x7f48580c8430 2023-10-30T13:08:30.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f4857fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48580c6b80 0x7f48581f1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:30.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f4857fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48580c6b80 0x7f48581f1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45250/0 (socket says 172.21.15.125:45250) 2023-10-30T13:08:30.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f4857fff640 1 -- 172.21.15.125:0/4080392075 learned_addr learned my addr 172.21.15.125:0/4080392075 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:30.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f48577fe640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48580c6f60 0x7f48581604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:30.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f485cc2e640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f48580c8430 0x7f48581366a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:30.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f4857fff640 1 -- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f48580c8430 msgr2=0x7f48581366a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:30.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f4857fff640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f48580c8430 0x7f48581366a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:30.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f4857fff640 1 -- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48580c6f60 msgr2=0x7f48581604f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:30.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f4857fff640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48580c6f60 0x7f48581604f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:30.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f4857fff640 1 -- 172.21.15.125:0/4080392075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c0094d0 con 0x7f48580c6b80 2023-10-30T13:08:30.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f48577fe640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48580c6f60 0x7f48581604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:08:30.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f4857fff640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48580c6b80 0x7f48581f1f00 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f484800e930 tx=0x7f484800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:30.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f4854ff9640 1 -- 172.21.15.125:0/4080392075 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848009690 con 0x7f48580c6b80 2023-10-30T13:08:30.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f4854ff9640 1 -- 172.21.15.125:0/4080392075 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484800ca00 con 0x7f48580c6b80 2023-10-30T13:08:30.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f485e691640 1 -- 172.21.15.125:0/4080392075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4858153f20 con 0x7f48580c6b80 2023-10-30T13:08:30.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f4854ff9640 1 -- 172.21.15.125:0/4080392075 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484800cd70 con 0x7f48580c6b80 2023-10-30T13:08:30.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f485e691640 1 -- 172.21.15.125:0/4080392075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48581f55a0 con 0x7f48580c6b80 2023-10-30T13:08:30.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f485e691640 1 -- 172.21.15.125:0/4080392075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f485804e170 con 0x7f48580c6b80 2023-10-30T13:08:30.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.751+0000 7f4854ff9640 1 -- 172.21.15.125:0/4080392075 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4848013470 con 0x7f48580c6b80 2023-10-30T13:08:30.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.755+0000 7f4854ff9640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f48440664d0 0x7f4844068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:30.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.755+0000 7f4854ff9640 1 -- 172.21.15.125:0/4080392075 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f48480848d0 con 0x7f48580c6b80 2023-10-30T13:08:30.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.755+0000 7f48577fe640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f48440664d0 0x7f4844068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:30.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.755+0000 7f48577fe640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f48440664d0 0x7f4844068990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f484c0af000 tx=0x7f484c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:30.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.759+0000 7f4854ff9640 1 -- 172.21.15.125:0/4080392075 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f48480515f0 con 0x7f48580c6b80 2023-10-30T13:08:30.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.939+0000 7f485e691640 1 -- 172.21.15.125:0/4080392075 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48581f2440 con 0x7f48580c6b80 2023-10-30T13:08:30.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.939+0000 7f4854ff9640 1 -- 172.21.15.125:0/4080392075 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+5752 (secure 0 0 0) 0x7f4848054c10 con 0x7f48580c6b80 2023-10-30T13:08:30.944 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:30.944 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:28.631412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:30.946 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 183 2023-10-30T13:08:30.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.943+0000 7f485e691640 1 -- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f48440664d0 msgr2=0x7f4844068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:30.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.943+0000 7f485e691640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f48440664d0 0x7f4844068990 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f484c0af000 tx=0x7f484c008040 comp rx=0 tx=0).stop 2023-10-30T13:08:30.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.943+0000 7f485e691640 1 -- 172.21.15.125:0/4080392075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48580c6b80 msgr2=0x7f48581f1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:30.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.943+0000 7f485e691640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48580c6b80 0x7f48581f1f00 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7f484800e930 tx=0x7f484800edf0 comp rx=0 tx=0).stop 2023-10-30T13:08:30.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.943+0000 7f485e691640 1 -- 172.21.15.125:0/4080392075 shutdown_connections 2023-10-30T13:08:30.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.943+0000 7f485e691640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f48440664d0 0x7f4844068990 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:30.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.943+0000 7f485e691640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f48580c8430 0x7f48581366a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:30.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.943+0000 7f485e691640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f48580c6f60 0x7f48581604f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:30.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.943+0000 7f485e691640 1 --2- 172.21.15.125:0/4080392075 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f48580c6b80 0x7f48581f1f00 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:30.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.943+0000 7f485e691640 1 -- 172.21.15.125:0/4080392075 >> 172.21.15.125:0/4080392075 conn(0x7f48580c7910 msgr2=0x7f48580c12b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:30.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.943+0000 7f485e691640 1 -- 172.21.15.125:0/4080392075 shutdown_connections 2023-10-30T13:08:30.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:30.943+0000 7f485e691640 1 -- 172.21.15.125:0/4080392075 wait complete. 2023-10-30T13:08:30.964 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=0, replacing gid=5234 2023-10-30T13:08:30.964 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.d 2023-10-30T13:08:32.965 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T13:08:32.965 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:08:32.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:08:32.969 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:08:32.969 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:33.045 INFO:tasks.ceph.mds.d.smithi136.stdout:starting mds.d at 2023-10-30T13:08:33.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd08c90e640 1 Processor -- start 2023-10-30T13:08:33.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd08c90e640 1 -- start start 2023-10-30T13:08:33.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd08c90e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd0881538a0 0x7fd08815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:33.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd08c90e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd0880cf160 0x7fd0880cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:33.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd08c90e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0880cb800 con 0x7fd0880ca0e0 2023-10-30T13:08:33.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd08c90e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd08815dc30 con 0x7fd0880cf160 2023-10-30T13:08:33.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd08c90e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0880cb330 con 0x7fd0881538a0 2023-10-30T13:08:33.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd086d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd0880cf160 0x7fd0880cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:33.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd086d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd0880cf160 0x7fd0880cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45254/0 (socket says 172.21.15.125:45254) 2023-10-30T13:08:33.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd086d9d640 1 -- 172.21.15.125:0/1917321675 learned_addr learned my addr 172.21.15.125:0/1917321675 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:33.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd08659c640 1 --2- 172.21.15.125:0/1917321675 >> v2:172.21.15.136:3300/0 conn(0x7fd0881538a0 0x7fd08815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:33.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd086d9d640 1 -- 172.21.15.125:0/1917321675 >> v2:172.21.15.136:3300/0 conn(0x7fd0881538a0 msgr2=0x7fd08815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:33.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd086d9d640 1 --2- 172.21.15.125:0/1917321675 >> v2:172.21.15.136:3300/0 conn(0x7fd0881538a0 0x7fd08815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:33.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd086d9d640 1 -- 172.21.15.125:0/1917321675 >> v1:172.21.15.125:6789/0 conn(0x7fd0880ca0e0 legacy=0x7fd08815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:33.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd086d9d640 1 -- 172.21.15.125:0/1917321675 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0880cad70 con 0x7fd0880cf160 2023-10-30T13:08:33.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.115+0000 7fd085d9b640 1 -- 172.21.15.125:0/1917321675 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632687019 0 0) 0x7fd0880cb800 con 0x7fd0880ca0e0 2023-10-30T13:08:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.119+0000 7fd086d9d640 1 --2- 172.21.15.125:0/1917321675 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0880cf160 0x7fd0880cf590 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fd0880cfd10 tx=0x7fd08001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=346c60361c902aae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:33.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.119+0000 7fd085d9b640 1 -- 172.21.15.125:0/1917321675 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0800a89f0 con 0x7fd0880cf160 2023-10-30T13:08:33.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.119+0000 7fd085d9b640 1 -- 172.21.15.125:0/1917321675 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0800a8b50 con 0x7fd0880cf160 2023-10-30T13:08:33.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.119+0000 7fd085d9b640 1 -- 172.21.15.125:0/1917321675 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0800b0440 con 0x7fd0880cf160 2023-10-30T13:08:33.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.119+0000 7fd08c90e640 1 -- 172.21.15.125:0/1917321675 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0880cf160 msgr2=0x7fd0880cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:33.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.119+0000 7fd08c90e640 1 --2- 172.21.15.125:0/1917321675 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0880cf160 0x7fd0880cf590 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fd0880cfd10 tx=0x7fd08001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:08:33.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.119+0000 7fd08c90e640 1 -- 172.21.15.125:0/1917321675 shutdown_connections 2023-10-30T13:08:33.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.119+0000 7fd08c90e640 1 --2- 172.21.15.125:0/1917321675 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0880cf160 0x7fd0880cf590 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:33.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.119+0000 7fd08c90e640 1 --2- 172.21.15.125:0/1917321675 >> v2:172.21.15.136:3300/0 conn(0x7fd0881538a0 0x7fd08815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:33.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.119+0000 7fd08c90e640 1 -- 172.21.15.125:0/1917321675 >> 172.21.15.125:0/1917321675 conn(0x7fd08806bfe0 msgr2=0x7fd08806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:33.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.119+0000 7fd08c90e640 1 -- 172.21.15.125:0/1917321675 shutdown_connections 2023-10-30T13:08:33.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.119+0000 7fd08c90e640 1 -- 172.21.15.125:0/1917321675 wait complete. 2023-10-30T13:08:33.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd08c90e640 1 Processor -- start 2023-10-30T13:08:33.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd08c90e640 1 -- start start 2023-10-30T13:08:33.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd08c90e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0880ca0e0 0x7fd0880b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:33.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd08c90e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0880cf160 0x7fd0880ce650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:33.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd08c90e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0881538a0 0x7fd0880b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:33.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd08c90e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0880c5a00 con 0x7fd0880cf160 2023-10-30T13:08:33.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd08c90e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd088158200 con 0x7fd0880ca0e0 2023-10-30T13:08:33.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd08c90e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd088157470 con 0x7fd0881538a0 2023-10-30T13:08:33.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd07dd9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0880cf160 0x7fd0880ce650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:33.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd07dd9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0880cf160 0x7fd0880ce650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45270/0 (socket says 172.21.15.125:45270) 2023-10-30T13:08:33.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd07dd9b640 1 -- 172.21.15.125:0/454985594 learned_addr learned my addr 172.21.15.125:0/454985594 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:33.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd08659c640 1 --2- 172.21.15.125:0/454985594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0880ca0e0 0x7fd0880b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:33.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd086d9d640 1 --2- 172.21.15.125:0/454985594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0881538a0 0x7fd0880b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:33.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd07dd9b640 1 -- 172.21.15.125:0/454985594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0881538a0 msgr2=0x7fd0880b4d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:33.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd07dd9b640 1 --2- 172.21.15.125:0/454985594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0881538a0 0x7fd0880b4d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:33.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd07dd9b640 1 -- 172.21.15.125:0/454985594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0880ca0e0 msgr2=0x7fd0880b1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:33.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd07dd9b640 1 --2- 172.21.15.125:0/454985594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0880ca0e0 0x7fd0880b1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:33.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd07dd9b640 1 -- 172.21.15.125:0/454985594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0800094d0 con 0x7fd0880cf160 2023-10-30T13:08:33.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd07dd9b640 1 --2- 172.21.15.125:0/454985594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0880cf160 0x7fd0880ce650 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fd07000c7a0 tx=0x7fd07000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:33.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd07f7fe640 1 -- 172.21.15.125:0/454985594 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070003db0 con 0x7fd0880cf160 2023-10-30T13:08:33.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd07f7fe640 1 -- 172.21.15.125:0/454985594 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd070005070 con 0x7fd0880cf160 2023-10-30T13:08:33.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd08c90e640 1 -- 172.21.15.125:0/454985594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0881efb50 con 0x7fd0880cf160 2023-10-30T13:08:33.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd07f7fe640 1 -- 172.21.15.125:0/454985594 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0700053e0 con 0x7fd0880cf160 2023-10-30T13:08:33.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.123+0000 7fd08c90e640 1 -- 172.21.15.125:0/454985594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0881eed90 con 0x7fd0880cf160 2023-10-30T13:08:33.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.127+0000 7fd08c90e640 1 -- 172.21.15.125:0/454985594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0880b2a50 con 0x7fd0880cf160 2023-10-30T13:08:33.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.127+0000 7fd07f7fe640 1 -- 172.21.15.125:0/454985594 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd070005540 con 0x7fd0880cf160 2023-10-30T13:08:33.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.127+0000 7fd07f7fe640 1 --2- 172.21.15.125:0/454985594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0680664a0 0x7fd068068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:33.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.127+0000 7fd07f7fe640 1 -- 172.21.15.125:0/454985594 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7fd070083e50 con 0x7fd0880cf160 2023-10-30T13:08:33.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.127+0000 7fd08659c640 1 --2- 172.21.15.125:0/454985594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0680664a0 0x7fd068068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:33.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.127+0000 7fd08659c640 1 --2- 172.21.15.125:0/454985594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0680664a0 0x7fd068068960 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fd07800b9d0 tx=0x7fd078008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:33.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.131+0000 7fd07f7fe640 1 -- 172.21.15.125:0/454985594 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd070050b50 con 0x7fd0880cf160 2023-10-30T13:08:33.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.311+0000 7fd08c90e640 1 -- 172.21.15.125:0/454985594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd08804e170 con 0x7fd0880cf160 2023-10-30T13:08:33.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.311+0000 7fd07f7fe640 1 -- 172.21.15.125:0/454985594 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+5752 (secure 0 0 0) 0x7fd070017070 con 0x7fd0880cf160 2023-10-30T13:08:33.315 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:33.315 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:28.631412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:33.316 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 183 2023-10-30T13:08:33.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.315+0000 7fd08c90e640 1 -- 172.21.15.125:0/454985594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0680664a0 msgr2=0x7fd068068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:33.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.315+0000 7fd08c90e640 1 --2- 172.21.15.125:0/454985594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0680664a0 0x7fd068068960 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fd07800b9d0 tx=0x7fd078008040 comp rx=0 tx=0).stop 2023-10-30T13:08:33.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.315+0000 7fd08c90e640 1 -- 172.21.15.125:0/454985594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0880cf160 msgr2=0x7fd0880ce650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:33.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.315+0000 7fd08c90e640 1 --2- 172.21.15.125:0/454985594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0880cf160 0x7fd0880ce650 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fd07000c7a0 tx=0x7fd07000cc60 comp rx=0 tx=0).stop 2023-10-30T13:08:33.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.315+0000 7fd08c90e640 1 -- 172.21.15.125:0/454985594 shutdown_connections 2023-10-30T13:08:33.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.315+0000 7fd08c90e640 1 --2- 172.21.15.125:0/454985594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd0680664a0 0x7fd068068960 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:33.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.315+0000 7fd08c90e640 1 --2- 172.21.15.125:0/454985594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd0881538a0 0x7fd0880b4d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:33.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.315+0000 7fd08c90e640 1 --2- 172.21.15.125:0/454985594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd0880cf160 0x7fd0880ce650 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:33.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.315+0000 7fd08c90e640 1 --2- 172.21.15.125:0/454985594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd0880ca0e0 0x7fd0880b1870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:33.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.315+0000 7fd08c90e640 1 -- 172.21.15.125:0/454985594 >> 172.21.15.125:0/454985594 conn(0x7fd08806bfe0 msgr2=0x7fd088066310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:33.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.315+0000 7fd08c90e640 1 -- 172.21.15.125:0/454985594 shutdown_connections 2023-10-30T13:08:33.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:33.315+0000 7fd08c90e640 1 -- 172.21.15.125:0/454985594 wait complete. 2023-10-30T13:08:33.335 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T13:08:35.336 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:35.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20b0180640 1 Processor -- start 2023-10-30T13:08:35.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20b0180640 1 -- start start 2023-10-30T13:08:35.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20b0180640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f20a81580e0 0x7f20a8070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:35.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20b0180640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20a815e700 0x7f20a806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:35.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20b0180640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a80c5810 con 0x7f20a80c6540 2023-10-30T13:08:35.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20b0180640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a80c6990 con 0x7f20a81580e0 2023-10-30T13:08:35.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20b0180640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a815eb90 con 0x7f20a815e700 2023-10-30T13:08:35.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20ad71b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f20a80c6540 0x7f20a8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41622/0 (socket says 172.21.15.125:41622) 2023-10-30T13:08:35.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20ad71b640 1 -- 172.21.15.125:0/3339959726 learned_addr learned my addr 172.21.15.125:0/3339959726 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:35.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20ae71d640 1 --2- 172.21.15.125:0/3339959726 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20a815e700 0x7f20a806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:35.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20adf1c640 1 --2- 172.21.15.125:0/3339959726 >> v2:172.21.15.136:3300/0 conn(0x7f20a81580e0 0x7f20a8070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:35.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 -- 172.21.15.125:0/3339959726 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632122807 0 0) 0x7f20a80c5810 con 0x7f20a80c6540 2023-10-30T13:08:35.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 -- 172.21.15.125:0/3339959726 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2094003530 con 0x7f20a80c6540 2023-10-30T13:08:35.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 -- 172.21.15.125:0/3339959726 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3444864209 0 0) 0x7f2094003530 con 0x7f20a80c6540 2023-10-30T13:08:35.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 -- 172.21.15.125:0/3339959726 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20a815e700 msgr2=0x7f20a806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:35.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 --2- 172.21.15.125:0/3339959726 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20a815e700 0x7f20a806a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:35.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 -- 172.21.15.125:0/3339959726 >> v2:172.21.15.136:3300/0 conn(0x7f20a81580e0 msgr2=0x7f20a8070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:35.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 --2- 172.21.15.125:0/3339959726 >> v2:172.21.15.136:3300/0 conn(0x7f20a81580e0 0x7f20a8070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:35.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 -- 172.21.15.125:0/3339959726 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a80c6bb0 con 0x7f20a80c6540 2023-10-30T13:08:35.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 -- 172.21.15.125:0/3339959726 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f2098001680 con 0x7f20a80c6540 2023-10-30T13:08:35.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 -- 172.21.15.125:0/3339959726 >> v1:172.21.15.125:6789/0 conn(0x7f20a80c6540 legacy=0x7f20a8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:35.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 --2- 172.21.15.125:0/3339959726 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20a815e700 0x7f2094003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:35.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 --2- 172.21.15.125:0/3339959726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2094003f00 0x7f20940047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:35.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 --2- 172.21.15.125:0/3339959726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2094004d10 0x7f2094005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:35.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 -- 172.21.15.125:0/3339959726 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a80c6990 con 0x7f2094004d10 2023-10-30T13:08:35.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 -- 172.21.15.125:0/3339959726 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a815eb90 con 0x7f2094003f00 2023-10-30T13:08:35.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20acf1a640 1 -- 172.21.15.125:0/3339959726 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209400b990 con 0x7f20a815e700 2023-10-30T13:08:35.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20ae71d640 1 --2- 172.21.15.125:0/3339959726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2094004d10 0x7f2094005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20ad71b640 1 --2- 172.21.15.125:0/3339959726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2094003f00 0x7f20940047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20adf1c640 1 --2- 172.21.15.125:0/3339959726 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20a815e700 0x7f2094003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20ae71d640 1 -- 172.21.15.125:0/3339959726 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20a815e700 msgr2=0x7f2094003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20ae71d640 1 --2- 172.21.15.125:0/3339959726 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20a815e700 0x7f2094003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20ae71d640 1 -- 172.21.15.125:0/3339959726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2094003f00 msgr2=0x7f20940047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20ae71d640 1 --2- 172.21.15.125:0/3339959726 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2094003f00 0x7f20940047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:35.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.479+0000 7f20ae71d640 1 -- 172.21.15.125:0/3339959726 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2094003530 con 0x7f2094004d10 2023-10-30T13:08:35.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.483+0000 7f20adf1c640 1 --2- 172.21.15.125:0/3339959726 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20a815e700 0x7f2094003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:35.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.483+0000 7f20ae71d640 1 --2- 172.21.15.125:0/3339959726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2094004d10 0x7f2094005190 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009fc0 tx=0x7f20a001cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c17d56ed126a2d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.483+0000 7f20acf1a640 1 -- 172.21.15.125:0/3339959726 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a00a5c70 con 0x7f2094004d10 2023-10-30T13:08:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.483+0000 7f20acf1a640 1 -- 172.21.15.125:0/3339959726 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a000f040 con 0x7f2094004d10 2023-10-30T13:08:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.483+0000 7f20acf1a640 1 -- 172.21.15.125:0/3339959726 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a00ad440 con 0x7f2094004d10 2023-10-30T13:08:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.483+0000 7f20b0180640 1 -- 172.21.15.125:0/3339959726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2094004d10 msgr2=0x7f2094005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:35.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.483+0000 7f20b0180640 1 --2- 172.21.15.125:0/3339959726 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2094004d10 0x7f2094005190 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009fc0 tx=0x7f20a001cc00 comp rx=0 tx=0).stop 2023-10-30T13:08:35.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.483+0000 7f20ae71d640 1 -- 172.21.15.125:0/3339959726 reap_dead start 2023-10-30T13:08:35.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.483+0000 7f20b0180640 1 -- 172.21.15.125:0/3339959726 shutdown_connections 2023-10-30T13:08:35.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.483+0000 7f20b0180640 1 -- 172.21.15.125:0/3339959726 >> 172.21.15.125:0/3339959726 conn(0x7f20a80c7910 msgr2=0x7f20a80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:35.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.483+0000 7f20b0180640 1 -- 172.21.15.125:0/3339959726 shutdown_connections 2023-10-30T13:08:35.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.483+0000 7f20b0180640 1 -- 172.21.15.125:0/3339959726 wait complete. 2023-10-30T13:08:35.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20b0180640 1 Processor -- start 2023-10-30T13:08:35.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20b0180640 1 -- start start 2023-10-30T13:08:35.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20b0180640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20a80c4e90 0x7f20a815ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:35.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20b0180640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20a81f6f80 0x7f20a8153da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:35.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20b0180640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20a80c5270 0x7f20a814c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:35.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20b0180640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a8152a00 con 0x7f20a80c4e90 2023-10-30T13:08:35.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20b0180640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20a81f7470 con 0x7f20a80c5270 2023-10-30T13:08:35.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20b0180640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a81536f0 con 0x7f20a81f6f80 2023-10-30T13:08:35.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20adf1c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20a80c4e90 0x7f20a815ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20adf1c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20a80c4e90 0x7f20a815ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45298/0 (socket says 172.21.15.125:45298) 2023-10-30T13:08:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20adf1c640 1 -- 172.21.15.125:0/3850394677 learned_addr learned my addr 172.21.15.125:0/3850394677 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20ad71b640 1 --2- 172.21.15.125:0/3850394677 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20a81f6f80 0x7f20a8153da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20ae71d640 1 --2- 172.21.15.125:0/3850394677 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20a80c5270 0x7f20a814c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20adf1c640 1 -- 172.21.15.125:0/3850394677 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20a81f6f80 msgr2=0x7f20a8153da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:35.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20adf1c640 1 --2- 172.21.15.125:0/3850394677 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20a81f6f80 0x7f20a8153da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20adf1c640 1 -- 172.21.15.125:0/3850394677 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20a80c5270 msgr2=0x7f20a814c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20adf1c640 1 --2- 172.21.15.125:0/3850394677 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20a80c5270 0x7f20a814c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20adf1c640 1 -- 172.21.15.125:0/3850394677 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a8153990 con 0x7f20a80c4e90 2023-10-30T13:08:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20adf1c640 1 --2- 172.21.15.125:0/3850394677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20a80c4e90 0x7f20a815ee40 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f20a40118d0 tx=0x7f20a4011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:35.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f209e7fc640 1 -- 172.21.15.125:0/3850394677 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a4004220 con 0x7f20a80c4e90 2023-10-30T13:08:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f209e7fc640 1 -- 172.21.15.125:0/3850394677 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20a4004380 con 0x7f20a80c4e90 2023-10-30T13:08:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f209e7fc640 1 -- 172.21.15.125:0/3850394677 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20a400fc10 con 0x7f20a80c4e90 2023-10-30T13:08:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.487+0000 7f20b0180640 1 -- 172.21.15.125:0/3850394677 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a81f4470 con 0x7f20a80c4e90 2023-10-30T13:08:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.491+0000 7f20b0180640 1 -- 172.21.15.125:0/3850394677 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a81f4a30 con 0x7f20a80c4e90 2023-10-30T13:08:35.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.491+0000 7f20b0180640 1 -- 172.21.15.125:0/3850394677 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a81f4bf0 con 0x7f20a80c4e90 2023-10-30T13:08:35.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.491+0000 7f209e7fc640 1 -- 172.21.15.125:0/3850394677 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f20a400fd70 con 0x7f20a80c4e90 2023-10-30T13:08:35.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.491+0000 7f209e7fc640 1 --2- 172.21.15.125:0/3850394677 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f208c0664d0 0x7f208c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:35.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.491+0000 7f209e7fc640 1 -- 172.21.15.125:0/3850394677 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f20a40847f0 con 0x7f20a80c4e90 2023-10-30T13:08:35.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.491+0000 7f20ad71b640 1 --2- 172.21.15.125:0/3850394677 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f208c0664d0 0x7f208c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:35.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.495+0000 7f20ad71b640 1 --2- 172.21.15.125:0/3850394677 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f208c0664d0 0x7f208c068990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f20a814cc60 tx=0x7f2098007900 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:35.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.495+0000 7f209e7fc640 1 -- 172.21.15.125:0/3850394677 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f20a40515a0 con 0x7f20a80c4e90 2023-10-30T13:08:35.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.691+0000 7f20b0180640 1 -- 172.21.15.125:0/3850394677 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20a804e170 con 0x7f20a80c4e90 2023-10-30T13:08:35.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.691+0000 7f209e7fc640 1 -- 172.21.15.125:0/3850394677 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+6547 (secure 0 0 0) 0x7f20a4054bc0 con 0x7f20a80c4e90 2023-10-30T13:08:35.695 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:35.696 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:28.631412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:35.696 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 184 2023-10-30T13:08:35.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.695+0000 7f20b0180640 1 -- 172.21.15.125:0/3850394677 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f208c0664d0 msgr2=0x7f208c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:35.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.695+0000 7f20b0180640 1 --2- 172.21.15.125:0/3850394677 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f208c0664d0 0x7f208c068990 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f20a814cc60 tx=0x7f2098007900 comp rx=0 tx=0).stop 2023-10-30T13:08:35.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.695+0000 7f20b0180640 1 -- 172.21.15.125:0/3850394677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20a80c4e90 msgr2=0x7f20a815ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:35.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.695+0000 7f20b0180640 1 --2- 172.21.15.125:0/3850394677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20a80c4e90 0x7f20a815ee40 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7f20a40118d0 tx=0x7f20a4011d90 comp rx=0 tx=0).stop 2023-10-30T13:08:35.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.695+0000 7f20b0180640 1 -- 172.21.15.125:0/3850394677 shutdown_connections 2023-10-30T13:08:35.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.695+0000 7f20b0180640 1 --2- 172.21.15.125:0/3850394677 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f208c0664d0 0x7f208c068990 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:35.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.695+0000 7f20b0180640 1 --2- 172.21.15.125:0/3850394677 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f20a80c5270 0x7f20a814c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:35.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.695+0000 7f20b0180640 1 --2- 172.21.15.125:0/3850394677 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f20a81f6f80 0x7f20a8153da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:35.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.695+0000 7f20b0180640 1 --2- 172.21.15.125:0/3850394677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f20a80c4e90 0x7f20a815ee40 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:35.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.695+0000 7f20b0180640 1 -- 172.21.15.125:0/3850394677 >> 172.21.15.125:0/3850394677 conn(0x7f20a80c7910 msgr2=0x7f20a806b440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:35.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.695+0000 7f20b0180640 1 -- 172.21.15.125:0/3850394677 shutdown_connections 2023-10-30T13:08:35.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:35.695+0000 7f20b0180640 1 -- 172.21.15.125:0/3850394677 wait complete. 2023-10-30T13:08:35.715 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T13:08:35.716 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before thrashing 2023-10-30T13:08:37.660 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T13:08:37.659+0000 7f510d857640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:08:37.661 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T13:08:37.659+0000 7f510d857640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:08:37.661 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T13:08:37.659+0000 7f510d857640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:08:38.717 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:38.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.863+0000 7fdecf9d3640 1 Processor -- start 2023-10-30T13:08:38.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.863+0000 7fdecf9d3640 1 -- start start 2023-10-30T13:08:38.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.863+0000 7fdecf9d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdec81580e0 0x7fdec8070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:38.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.863+0000 7fdecf9d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdec815e700 0x7fdec806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:38.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.863+0000 7fdecf9d3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdec80c5810 con 0x7fdec80c6540 2023-10-30T13:08:38.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.863+0000 7fdecf9d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec80c6990 con 0x7fdec81580e0 2023-10-30T13:08:38.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.863+0000 7fdecf9d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdec815eb90 con 0x7fdec815e700 2023-10-30T13:08:38.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.867+0000 7fdecd76f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdec81580e0 0x7fdec8070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:38.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.867+0000 7fdeccf6e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdec80c6540 0x7fdec8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41626/0 (socket says 172.21.15.125:41626) 2023-10-30T13:08:38.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.867+0000 7fdeccf6e640 1 -- 172.21.15.125:0/2522057517 learned_addr learned my addr 172.21.15.125:0/2522057517 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:38.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.867+0000 7fdecdf70640 1 --2- 172.21.15.125:0/2522057517 >> v2:172.21.15.136:3300/0 conn(0x7fdec815e700 0x7fdec806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:38.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.867+0000 7fdecd76f640 1 -- 172.21.15.125:0/2522057517 >> v2:172.21.15.136:3300/0 conn(0x7fdec815e700 msgr2=0x7fdec806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:38.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.867+0000 7fdecd76f640 1 --2- 172.21.15.125:0/2522057517 >> v2:172.21.15.136:3300/0 conn(0x7fdec815e700 0x7fdec806a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:38.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.867+0000 7fdecd76f640 1 -- 172.21.15.125:0/2522057517 >> v1:172.21.15.125:6789/0 conn(0x7fdec80c6540 legacy=0x7fdec8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:38.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.867+0000 7fdecd76f640 1 -- 172.21.15.125:0/2522057517 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec80c6bb0 con 0x7fdec81580e0 2023-10-30T13:08:38.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.867+0000 7fdecd76f640 1 --2- 172.21.15.125:0/2522057517 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec81580e0 0x7fdec8070740 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fdec8066ec0 tx=0x7fdec401c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e986df478559fdd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:38.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.867+0000 7fdebffff640 1 -- 172.21.15.125:0/2522057517 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec40a7070 con 0x7fdec81580e0 2023-10-30T13:08:38.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.867+0000 7fdebffff640 1 -- 172.21.15.125:0/2522057517 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec40ab430 con 0x7fdec81580e0 2023-10-30T13:08:38.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.867+0000 7fdebffff640 1 -- 172.21.15.125:0/2522057517 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec40ab7c0 con 0x7fdec81580e0 2023-10-30T13:08:38.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2522057517 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec81580e0 msgr2=0x7fdec8070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:38.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 --2- 172.21.15.125:0/2522057517 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec81580e0 0x7fdec8070740 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7fdec8066ec0 tx=0x7fdec401c430 comp rx=0 tx=0).stop 2023-10-30T13:08:38.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2522057517 shutdown_connections 2023-10-30T13:08:38.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 --2- 172.21.15.125:0/2522057517 >> v2:172.21.15.136:3300/0 conn(0x7fdec815e700 0x7fdec806a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:38.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 --2- 172.21.15.125:0/2522057517 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec81580e0 0x7fdec8070740 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:38.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2522057517 >> 172.21.15.125:0/2522057517 conn(0x7fdec80c7910 msgr2=0x7fdec80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:38.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2522057517 shutdown_connections 2023-10-30T13:08:38.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2522057517 wait complete. 2023-10-30T13:08:38.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 Processor -- start 2023-10-30T13:08:38.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 -- start start 2023-10-30T13:08:38.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec80c6540 0x7fdec8161d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:38.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec81580e0 0x7fdec80ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:38.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdec815e700 0x7fdec815ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:38.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec8161200 con 0x7fdec81580e0 2023-10-30T13:08:38.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec815d8f0 con 0x7fdec80c6540 2023-10-30T13:08:38.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecf9d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec815de20 con 0x7fdec815e700 2023-10-30T13:08:38.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdeccf6e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec81580e0 0x7fdec80ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:38.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdeccf6e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec81580e0 0x7fdec80ca380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45324/0 (socket says 172.21.15.125:45324) 2023-10-30T13:08:38.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdeccf6e640 1 -- 172.21.15.125:0/2991682359 learned_addr learned my addr 172.21.15.125:0/2991682359 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:38.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecdf70640 1 --2- 172.21.15.125:0/2991682359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdec815e700 0x7fdec815ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:38.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdecd76f640 1 --2- 172.21.15.125:0/2991682359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec80c6540 0x7fdec8161d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:38.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdeccf6e640 1 -- 172.21.15.125:0/2991682359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdec815e700 msgr2=0x7fdec815ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:38.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdeccf6e640 1 --2- 172.21.15.125:0/2991682359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdec815e700 0x7fdec815ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:38.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdeccf6e640 1 -- 172.21.15.125:0/2991682359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec80c6540 msgr2=0x7fdec8161d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:38.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdeccf6e640 1 --2- 172.21.15.125:0/2991682359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec80c6540 0x7fdec8161d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:38.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.871+0000 7fdeccf6e640 1 -- 172.21.15.125:0/2991682359 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec4009420 con 0x7fdec81580e0 2023-10-30T13:08:38.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.875+0000 7fdeccf6e640 1 --2- 172.21.15.125:0/2991682359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec81580e0 0x7fdec80ca380 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fdeb800c500 tx=0x7fdeb800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:38.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.875+0000 7fdebdffb640 1 -- 172.21.15.125:0/2991682359 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8005cf0 con 0x7fdec81580e0 2023-10-30T13:08:38.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.875+0000 7fdebdffb640 1 -- 172.21.15.125:0/2991682359 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdeb8005e50 con 0x7fdec81580e0 2023-10-30T13:08:38.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.875+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2991682359 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec815db40 con 0x7fdec81580e0 2023-10-30T13:08:38.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.875+0000 7fdebdffb640 1 -- 172.21.15.125:0/2991682359 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeb80072f0 con 0x7fdec81580e0 2023-10-30T13:08:38.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.875+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2991682359 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec80c91b0 con 0x7fdec81580e0 2023-10-30T13:08:38.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.875+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2991682359 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec815e320 con 0x7fdec81580e0 2023-10-30T13:08:38.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.875+0000 7fdebdffb640 1 -- 172.21.15.125:0/2991682359 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdeb8007450 con 0x7fdec81580e0 2023-10-30T13:08:38.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.875+0000 7fdebdffb640 1 --2- 172.21.15.125:0/2991682359 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeac0664d0 0x7fdeac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:38.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.875+0000 7fdebdffb640 1 -- 172.21.15.125:0/2991682359 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7fdeb8083a60 con 0x7fdec81580e0 2023-10-30T13:08:38.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.875+0000 7fdecd76f640 1 --2- 172.21.15.125:0/2991682359 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeac0664d0 0x7fdeac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:38.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.879+0000 7fdecd76f640 1 --2- 172.21.15.125:0/2991682359 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeac0664d0 0x7fdeac068990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fdec40af000 tx=0x7fdec40a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:38.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:38.879+0000 7fdebdffb640 1 -- 172.21.15.125:0/2991682359 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdeb8050800 con 0x7fdec81580e0 2023-10-30T13:08:39.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.063+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2991682359 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdec804e170 con 0x7fdec81580e0 2023-10-30T13:08:39.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.063+0000 7fdebdffb640 1 -- 172.21.15.125:0/2991682359 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+6547 (secure 0 0 0) 0x7fdeb8017090 con 0x7fdec81580e0 2023-10-30T13:08:39.067 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:39.068 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:28.631412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:39.068 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 184 2023-10-30T13:08:39.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.067+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2991682359 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeac0664d0 msgr2=0x7fdeac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:39.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.067+0000 7fdecf9d3640 1 --2- 172.21.15.125:0/2991682359 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeac0664d0 0x7fdeac068990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fdec40af000 tx=0x7fdec40a7000 comp rx=0 tx=0).stop 2023-10-30T13:08:39.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.067+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2991682359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec81580e0 msgr2=0x7fdec80ca380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:39.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.067+0000 7fdecf9d3640 1 --2- 172.21.15.125:0/2991682359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec81580e0 0x7fdec80ca380 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fdeb800c500 tx=0x7fdeb800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:08:39.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.067+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2991682359 shutdown_connections 2023-10-30T13:08:39.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.067+0000 7fdecf9d3640 1 --2- 172.21.15.125:0/2991682359 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeac0664d0 0x7fdeac068990 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:39.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.067+0000 7fdecf9d3640 1 --2- 172.21.15.125:0/2991682359 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdec815e700 0x7fdec815ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:39.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.067+0000 7fdecf9d3640 1 --2- 172.21.15.125:0/2991682359 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec81580e0 0x7fdec80ca380 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:39.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.067+0000 7fdecf9d3640 1 --2- 172.21.15.125:0/2991682359 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec80c6540 0x7fdec8161d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:39.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.067+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2991682359 >> 172.21.15.125:0/2991682359 conn(0x7fdec80c7910 msgr2=0x7fdec8156fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:39.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.067+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2991682359 shutdown_connections 2023-10-30T13:08:39.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.067+0000 7fdecf9d3640 1 -- 172.21.15.125:0/2991682359 wait complete. 2023-10-30T13:08:39.088 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=2) 2023-10-30T13:08:39.088 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T13:08:39.089 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:08:39.105 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T13:08:39.105 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:39.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.267+0000 7fb995a68640 1 Processor -- start 2023-10-30T13:08:39.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.267+0000 7fb995a68640 1 -- start start 2023-10-30T13:08:39.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.267+0000 7fb995a68640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb9901538a0 0x7fb99015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:39.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.267+0000 7fb995a68640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb9900ca0e0 0x7fb99015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:39.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.267+0000 7fb995a68640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9900cb800 con 0x7fb9900cf160 2023-10-30T13:08:39.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.267+0000 7fb995a68640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb99015dc30 con 0x7fb9901538a0 2023-10-30T13:08:39.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.267+0000 7fb995a68640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9900cb330 con 0x7fb9900ca0e0 2023-10-30T13:08:39.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.267+0000 7fb98ffff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb9900cf160 0x7fb9900cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:41630/0 (socket says 172.21.15.125:41630) 2023-10-30T13:08:39.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.267+0000 7fb98ffff640 1 -- 172.21.15.125:0/2031153971 learned_addr learned my addr 172.21.15.125:0/2031153971 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:39.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.267+0000 7fb98effd640 1 --2- 172.21.15.125:0/2031153971 >> v2:172.21.15.136:3300/0 conn(0x7fb9900ca0e0 0x7fb99015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:39.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.271+0000 7fb98f7fe640 1 --2- 172.21.15.125:0/2031153971 >> v2:172.21.15.125:3300/0 conn(0x7fb9901538a0 0x7fb99015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:39.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.271+0000 7fb98e7fc640 1 -- 172.21.15.125:0/2031153971 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976228765 0 0) 0x7fb9900cb800 con 0x7fb9900cf160 2023-10-30T13:08:39.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.271+0000 7fb98e7fc640 1 -- 172.21.15.125:0/2031153971 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb978003530 con 0x7fb9900cf160 2023-10-30T13:08:39.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.271+0000 7fb98e7fc640 1 -- 172.21.15.125:0/2031153971 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2447887086 0 0) 0x7fb978003530 con 0x7fb9900cf160 2023-10-30T13:08:39.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.271+0000 7fb98effd640 1 -- 172.21.15.125:0/2031153971 >> v2:172.21.15.125:3300/0 conn(0x7fb9901538a0 msgr2=0x7fb99015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:39.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.271+0000 7fb98effd640 1 --2- 172.21.15.125:0/2031153971 >> v2:172.21.15.125:3300/0 conn(0x7fb9901538a0 0x7fb99015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:39.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.271+0000 7fb98effd640 1 -- 172.21.15.125:0/2031153971 >> v1:172.21.15.125:6789/0 conn(0x7fb9900cf160 legacy=0x7fb9900cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:39.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.271+0000 7fb98effd640 1 -- 172.21.15.125:0/2031153971 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9900cad70 con 0x7fb9900ca0e0 2023-10-30T13:08:39.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.275+0000 7fb98effd640 1 --2- 172.21.15.125:0/2031153971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9900ca0e0 0x7fb99015a090 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fb9900b30a0 tx=0x7fb98401eb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=267c761b506a9f94 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:39.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.275+0000 7fb98e7fc640 1 -- 172.21.15.125:0/2031153971 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb988001a00 con 0x7fb9900ca0e0 2023-10-30T13:08:39.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.275+0000 7fb98e7fc640 1 -- 172.21.15.125:0/2031153971 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9840a85f0 con 0x7fb9900ca0e0 2023-10-30T13:08:39.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.275+0000 7fb98e7fc640 1 -- 172.21.15.125:0/2031153971 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9840a8960 con 0x7fb9900ca0e0 2023-10-30T13:08:39.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.275+0000 7fb995a68640 1 -- 172.21.15.125:0/2031153971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9900ca0e0 msgr2=0x7fb99015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:39.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.275+0000 7fb995a68640 1 --2- 172.21.15.125:0/2031153971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9900ca0e0 0x7fb99015a090 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fb9900b30a0 tx=0x7fb98401eb70 comp rx=0 tx=0).stop 2023-10-30T13:08:39.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.275+0000 7fb995a68640 1 -- 172.21.15.125:0/2031153971 shutdown_connections 2023-10-30T13:08:39.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.275+0000 7fb995a68640 1 --2- 172.21.15.125:0/2031153971 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9900ca0e0 0x7fb99015a090 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:39.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.275+0000 7fb995a68640 1 --2- 172.21.15.125:0/2031153971 >> v2:172.21.15.125:3300/0 conn(0x7fb9901538a0 0x7fb99015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:39.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.275+0000 7fb995a68640 1 -- 172.21.15.125:0/2031153971 >> 172.21.15.125:0/2031153971 conn(0x7fb99006bfe0 msgr2=0x7fb99006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:39.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.275+0000 7fb995a68640 1 -- 172.21.15.125:0/2031153971 shutdown_connections 2023-10-30T13:08:39.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.275+0000 7fb995a68640 1 -- 172.21.15.125:0/2031153971 wait complete. 2023-10-30T13:08:39.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb995a68640 1 Processor -- start 2023-10-30T13:08:39.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb995a68640 1 -- start start 2023-10-30T13:08:39.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb995a68640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9900ca0e0 0x7fb9900b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:39.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb995a68640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9900cf160 0x7fb9901502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:39.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb995a68640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9901538a0 0x7fb990150800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:39.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb995a68640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9900b35e0 con 0x7fb9900cf160 2023-10-30T13:08:39.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb995a68640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9900b3000 con 0x7fb9900ca0e0 2023-10-30T13:08:39.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb995a68640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb990158f50 con 0x7fb9901538a0 2023-10-30T13:08:39.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb98effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9900cf160 0x7fb9901502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:39.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb98f7fe640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9900ca0e0 0x7fb9900b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:39.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb98effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9900cf160 0x7fb9901502c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45352/0 (socket says 172.21.15.125:45352) 2023-10-30T13:08:39.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb98effd640 1 -- 172.21.15.125:0/2489055131 learned_addr learned my addr 172.21.15.125:0/2489055131 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:39.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb98ffff640 1 --2- 172.21.15.125:0/2489055131 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9901538a0 0x7fb990150800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:39.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb98effd640 1 -- 172.21.15.125:0/2489055131 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9901538a0 msgr2=0x7fb990150800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:39.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb98effd640 1 --2- 172.21.15.125:0/2489055131 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9901538a0 0x7fb990150800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:39.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb98effd640 1 -- 172.21.15.125:0/2489055131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9900ca0e0 msgr2=0x7fb9900b3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:39.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb98effd640 1 --2- 172.21.15.125:0/2489055131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9900ca0e0 0x7fb9900b3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:39.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb98effd640 1 -- 172.21.15.125:0/2489055131 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb984009420 con 0x7fb9900cf160 2023-10-30T13:08:39.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb98ffff640 1 --2- 172.21.15.125:0/2489055131 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9901538a0 0x7fb990150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:39.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb98effd640 1 --2- 172.21.15.125:0/2489055131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9900cf160 0x7fb9901502c0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fb984000c10 tx=0x7fb98401b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:39.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb977fff640 1 -- 172.21.15.125:0/2489055131 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9840a9c80 con 0x7fb9900cf160 2023-10-30T13:08:39.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb977fff640 1 -- 172.21.15.125:0/2489055131 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9840a9de0 con 0x7fb9900cf160 2023-10-30T13:08:39.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb977fff640 1 -- 172.21.15.125:0/2489055131 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb984003840 con 0x7fb9900cf160 2023-10-30T13:08:39.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb995a68640 1 -- 172.21.15.125:0/2489055131 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb990159230 con 0x7fb9900cf160 2023-10-30T13:08:39.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.279+0000 7fb995a68640 1 -- 172.21.15.125:0/2489055131 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9900cecb0 con 0x7fb9900cf160 2023-10-30T13:08:39.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.283+0000 7fb995a68640 1 -- 172.21.15.125:0/2489055131 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9900b2a50 con 0x7fb9900cf160 2023-10-30T13:08:39.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.283+0000 7fb977fff640 1 -- 172.21.15.125:0/2489055131 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9840b7040 con 0x7fb9900cf160 2023-10-30T13:08:39.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.283+0000 7fb977fff640 1 --2- 172.21.15.125:0/2489055131 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb97c0664a0 0x7fb97c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:39.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.283+0000 7fb977fff640 1 -- 172.21.15.125:0/2489055131 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7fb984121c80 con 0x7fb9900cf160 2023-10-30T13:08:39.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.283+0000 7fb98f7fe640 1 --2- 172.21.15.125:0/2489055131 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb97c0664a0 0x7fb97c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:39.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.283+0000 7fb98f7fe640 1 --2- 172.21.15.125:0/2489055131 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb97c0664a0 0x7fb97c068960 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fb980009ac0 tx=0x7fb980009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:39.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.287+0000 7fb977fff640 1 -- 172.21.15.125:0/2489055131 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb9840ee970 con 0x7fb9900cf160 2023-10-30T13:08:39.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.475+0000 7fb995a68640 1 -- 172.21.15.125:0/2489055131 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb99004e170 con 0x7fb9900cf160 2023-10-30T13:08:39.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.475+0000 7fb977fff640 1 -- 172.21.15.125:0/2489055131 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 184 v184) v1 ==== 95+0+6547 (secure 0 0 0) 0x7fb9840b5020 con 0x7fb9900cf160 2023-10-30T13:08:39.478 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:39.479 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":184,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:28.631412+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:39.479 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 184 2023-10-30T13:08:39.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.479+0000 7fb995a68640 1 -- 172.21.15.125:0/2489055131 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb97c0664a0 msgr2=0x7fb97c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:39.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.479+0000 7fb995a68640 1 --2- 172.21.15.125:0/2489055131 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb97c0664a0 0x7fb97c068960 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fb980009ac0 tx=0x7fb980009190 comp rx=0 tx=0).stop 2023-10-30T13:08:39.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.479+0000 7fb995a68640 1 -- 172.21.15.125:0/2489055131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9900cf160 msgr2=0x7fb9901502c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:39.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.479+0000 7fb995a68640 1 --2- 172.21.15.125:0/2489055131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9900cf160 0x7fb9901502c0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7fb984000c10 tx=0x7fb98401b910 comp rx=0 tx=0).stop 2023-10-30T13:08:39.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.479+0000 7fb995a68640 1 -- 172.21.15.125:0/2489055131 shutdown_connections 2023-10-30T13:08:39.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.479+0000 7fb995a68640 1 --2- 172.21.15.125:0/2489055131 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb97c0664a0 0x7fb97c068960 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:39.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.479+0000 7fb995a68640 1 --2- 172.21.15.125:0/2489055131 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9901538a0 0x7fb990150800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:39.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.479+0000 7fb995a68640 1 --2- 172.21.15.125:0/2489055131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9900cf160 0x7fb9901502c0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:39.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.479+0000 7fb995a68640 1 --2- 172.21.15.125:0/2489055131 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9900ca0e0 0x7fb9900b3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:39.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.479+0000 7fb995a68640 1 -- 172.21.15.125:0/2489055131 >> 172.21.15.125:0/2489055131 conn(0x7fb99006bfe0 msgr2=0x7fb990065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:39.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.479+0000 7fb995a68640 1 -- 172.21.15.125:0/2489055131 shutdown_connections 2023-10-30T13:08:39.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:39.479+0000 7fb995a68640 1 -- 172.21.15.125:0/2489055131 wait complete. 2023-10-30T13:08:39.501 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:08:41.502 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:41.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.647+0000 7f9d4ad80640 1 Processor -- start 2023-10-30T13:08:41.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.647+0000 7f9d4ad80640 1 -- start start 2023-10-30T13:08:41.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.647+0000 7f9d4ad80640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9d44070a00 0x7f9d44070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:41.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.647+0000 7f9d4ad80640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9d44069e90 0x7f9d4406a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:41.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.647+0000 7f9d4ad80640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d440713b0 con 0x7f9d441539a0 2023-10-30T13:08:41.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.647+0000 7f9d4ad80640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4406fbb0 con 0x7f9d44070a00 2023-10-30T13:08:41.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.647+0000 7f9d4ad80640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d44153d80 con 0x7f9d44069e90 2023-10-30T13:08:41.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.647+0000 7f9d48b1c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9d441539a0 0x7f9d4415a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46196/0 (socket says 172.21.15.125:46196) 2023-10-30T13:08:41.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.647+0000 7f9d48b1c640 1 -- 172.21.15.125:0/2736491648 learned_addr learned my addr 172.21.15.125:0/2736491648 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:41.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.647+0000 7f9d3bfff640 1 --2- 172.21.15.125:0/2736491648 >> v2:172.21.15.125:3300/0 conn(0x7f9d44070a00 0x7f9d44070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:41.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.647+0000 7f9d4931d640 1 --2- 172.21.15.125:0/2736491648 >> v2:172.21.15.136:3300/0 conn(0x7f9d44069e90 0x7f9d4406a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:41.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.647+0000 7f9d3b7fe640 1 -- 172.21.15.125:0/2736491648 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092843226 0 0) 0x7f9d440713b0 con 0x7f9d441539a0 2023-10-30T13:08:41.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.647+0000 7f9d3b7fe640 1 -- 172.21.15.125:0/2736491648 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d2c003530 con 0x7f9d441539a0 2023-10-30T13:08:41.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d3bfff640 1 -- 172.21.15.125:0/2736491648 >> v2:172.21.15.136:3300/0 conn(0x7f9d44069e90 msgr2=0x7f9d4406a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:41.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d3bfff640 1 --2- 172.21.15.125:0/2736491648 >> v2:172.21.15.136:3300/0 conn(0x7f9d44069e90 0x7f9d4406a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:41.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d3bfff640 1 -- 172.21.15.125:0/2736491648 >> v1:172.21.15.125:6789/0 conn(0x7f9d441539a0 legacy=0x7f9d4415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:41.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d3bfff640 1 -- 172.21.15.125:0/2736491648 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d440c5c50 con 0x7f9d44070a00 2023-10-30T13:08:41.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d3bfff640 1 --2- 172.21.15.125:0/2736491648 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d44070a00 0x7f9d44070e70 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f9d440b1080 tx=0x7f9d3401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69ef789129ecc1d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:41.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d3b7fe640 1 -- 172.21.15.125:0/2736491648 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d340ab600 con 0x7f9d44070a00 2023-10-30T13:08:41.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d3b7fe640 1 -- 172.21.15.125:0/2736491648 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d340a7050 con 0x7f9d44070a00 2023-10-30T13:08:41.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d3b7fe640 1 -- 172.21.15.125:0/2736491648 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d340ab960 con 0x7f9d44070a00 2023-10-30T13:08:41.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d4ad80640 1 -- 172.21.15.125:0/2736491648 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d44070a00 msgr2=0x7f9d44070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:41.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d4ad80640 1 --2- 172.21.15.125:0/2736491648 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d44070a00 0x7f9d44070e70 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f9d440b1080 tx=0x7f9d3401c3a0 comp rx=0 tx=0).stop 2023-10-30T13:08:41.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d4ad80640 1 -- 172.21.15.125:0/2736491648 shutdown_connections 2023-10-30T13:08:41.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d4ad80640 1 --2- 172.21.15.125:0/2736491648 >> v2:172.21.15.136:3300/0 conn(0x7f9d44069e90 0x7f9d4406a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:41.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d4ad80640 1 --2- 172.21.15.125:0/2736491648 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d44070a00 0x7f9d44070e70 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:41.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d4ad80640 1 -- 172.21.15.125:0/2736491648 >> 172.21.15.125:0/2736491648 conn(0x7f9d440c7910 msgr2=0x7f9d440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:41.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d4ad80640 1 -- 172.21.15.125:0/2736491648 shutdown_connections 2023-10-30T13:08:41.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.651+0000 7f9d4ad80640 1 -- 172.21.15.125:0/2736491648 wait complete. 2023-10-30T13:08:41.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d4ad80640 1 Processor -- start 2023-10-30T13:08:41.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d4ad80640 1 -- start start 2023-10-30T13:08:41.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d4ad80640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9d44069e90 0x7f9d44144620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:41.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d4ad80640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d44070a00 0x7f9d441f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:41.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d4ad80640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9d441539a0 0x7f9d44140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:41.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d4ad80640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44147810 con 0x7f9d44070a00 2023-10-30T13:08:41.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d4ad80640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d440c8e40 con 0x7f9d44069e90 2023-10-30T13:08:41.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d4ad80640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44146740 con 0x7f9d441539a0 2023-10-30T13:08:41.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d3bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d44070a00 0x7f9d441f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:41.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d3bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d44070a00 0x7f9d441f3b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41494/0 (socket says 172.21.15.125:41494) 2023-10-30T13:08:41.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d3bfff640 1 -- 172.21.15.125:0/3816214908 learned_addr learned my addr 172.21.15.125:0/3816214908 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:41.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d48b1c640 1 --2- 172.21.15.125:0/3816214908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9d44069e90 0x7f9d44144620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:41.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d4931d640 1 --2- 172.21.15.125:0/3816214908 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9d441539a0 0x7f9d44140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:41.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d3bfff640 1 -- 172.21.15.125:0/3816214908 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9d441539a0 msgr2=0x7f9d44140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:41.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d3bfff640 1 --2- 172.21.15.125:0/3816214908 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9d441539a0 0x7f9d44140960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:41.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d3bfff640 1 -- 172.21.15.125:0/3816214908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9d44069e90 msgr2=0x7f9d44144620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:41.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d3bfff640 1 --2- 172.21.15.125:0/3816214908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9d44069e90 0x7f9d44144620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:41.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d3bfff640 1 -- 172.21.15.125:0/3816214908 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d34009420 con 0x7f9d44070a00 2023-10-30T13:08:41.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d3bfff640 1 --2- 172.21.15.125:0/3816214908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d44070a00 0x7f9d441f3b40 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f9d34019fd0 tx=0x7f9d34019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:41.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d397fa640 1 -- 172.21.15.125:0/3816214908 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d340b4400 con 0x7f9d44070a00 2023-10-30T13:08:41.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d397fa640 1 -- 172.21.15.125:0/3816214908 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d340b4560 con 0x7f9d44070a00 2023-10-30T13:08:41.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d4ad80640 1 -- 172.21.15.125:0/3816214908 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d441461a0 con 0x7f9d44070a00 2023-10-30T13:08:41.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.655+0000 7f9d397fa640 1 -- 172.21.15.125:0/3816214908 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d340b4910 con 0x7f9d44070a00 2023-10-30T13:08:41.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.659+0000 7f9d4ad80640 1 -- 172.21.15.125:0/3816214908 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d44147250 con 0x7f9d44070a00 2023-10-30T13:08:41.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.659+0000 7f9d4ad80640 1 -- 172.21.15.125:0/3816214908 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d440c8c10 con 0x7f9d44070a00 2023-10-30T13:08:41.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.659+0000 7f9d397fa640 1 -- 172.21.15.125:0/3816214908 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9d340b2040 con 0x7f9d44070a00 2023-10-30T13:08:41.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.659+0000 7f9d397fa640 1 --2- 172.21.15.125:0/3816214908 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9d300664d0 0x7f9d30068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:41.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.659+0000 7f9d397fa640 1 -- 172.21.15.125:0/3816214908 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f9d3411e460 con 0x7f9d44070a00 2023-10-30T13:08:41.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.659+0000 7f9d48b1c640 1 --2- 172.21.15.125:0/3816214908 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9d300664d0 0x7f9d30068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:41.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.663+0000 7f9d48b1c640 1 --2- 172.21.15.125:0/3816214908 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9d300664d0 0x7f9d30068990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f9d40005890 tx=0x7f9d40009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:41.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.663+0000 7f9d397fa640 1 -- 172.21.15.125:0/3816214908 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9d440c8c10 con 0x7f9d44070a00 2023-10-30T13:08:41.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.847+0000 7f9d4ad80640 1 -- 172.21.15.125:0/3816214908 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d440c8ff0 con 0x7f9d44070a00 2023-10-30T13:08:41.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.847+0000 7f9d397fa640 1 -- 172.21.15.125:0/3816214908 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+6548 (secure 0 0 0) 0x7f9d440c8ff0 con 0x7f9d44070a00 2023-10-30T13:08:41.851 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:41.852 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:39.252653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:41.852 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 185 2023-10-30T13:08:41.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.851+0000 7f9d4ad80640 1 -- 172.21.15.125:0/3816214908 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9d300664d0 msgr2=0x7f9d30068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:41.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.851+0000 7f9d4ad80640 1 --2- 172.21.15.125:0/3816214908 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9d300664d0 0x7f9d30068990 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f9d40005890 tx=0x7f9d40009040 comp rx=0 tx=0).stop 2023-10-30T13:08:41.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.851+0000 7f9d4ad80640 1 -- 172.21.15.125:0/3816214908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d44070a00 msgr2=0x7f9d441f3b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:41.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.851+0000 7f9d4ad80640 1 --2- 172.21.15.125:0/3816214908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d44070a00 0x7f9d441f3b40 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f9d34019fd0 tx=0x7f9d34019140 comp rx=0 tx=0).stop 2023-10-30T13:08:41.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.851+0000 7f9d4ad80640 1 -- 172.21.15.125:0/3816214908 shutdown_connections 2023-10-30T13:08:41.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.851+0000 7f9d4ad80640 1 --2- 172.21.15.125:0/3816214908 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9d300664d0 0x7f9d30068990 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:41.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.851+0000 7f9d4ad80640 1 --2- 172.21.15.125:0/3816214908 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9d441539a0 0x7f9d44140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:41.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.851+0000 7f9d4ad80640 1 --2- 172.21.15.125:0/3816214908 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9d44070a00 0x7f9d441f3b40 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:41.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.851+0000 7f9d4ad80640 1 --2- 172.21.15.125:0/3816214908 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9d44069e90 0x7f9d44144620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:41.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.851+0000 7f9d4ad80640 1 -- 172.21.15.125:0/3816214908 >> 172.21.15.125:0/3816214908 conn(0x7f9d440c7910 msgr2=0x7f9d441501c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:41.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.851+0000 7f9d4ad80640 1 -- 172.21.15.125:0/3816214908 shutdown_connections 2023-10-30T13:08:41.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:41.851+0000 7f9d4ad80640 1 -- 172.21.15.125:0/3816214908 wait complete. 2023-10-30T13:08:41.872 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:08:43.873 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:44.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a0ab03640 1 Processor -- start 2023-10-30T13:08:44.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a0ab03640 1 -- start start 2023-10-30T13:08:44.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a0ab03640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1a041538f0 0x7f1a04159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:44.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a0ab03640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1a0415cd50 0x7f1a040c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:44.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a0ab03640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a0415d300 con 0x7f1a040cefa0 2023-10-30T13:08:44.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a0ab03640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a0415d5e0 con 0x7f1a041538f0 2023-10-30T13:08:44.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a0ab03640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a0415d8c0 con 0x7f1a0415cd50 2023-10-30T13:08:44.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a09b01640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1a041538f0 0x7f1a04159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:44.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a0a302640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1a040cefa0 0x7f1a040cba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46208/0 (socket says 172.21.15.125:46208) 2023-10-30T13:08:44.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a0a302640 1 -- 172.21.15.125:0/857748056 learned_addr learned my addr 172.21.15.125:0/857748056 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:44.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a09300640 1 --2- 172.21.15.125:0/857748056 >> v2:172.21.15.136:3300/0 conn(0x7f1a0415cd50 0x7f1a040c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:44.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a09b01640 1 -- 172.21.15.125:0/857748056 >> v2:172.21.15.136:3300/0 conn(0x7f1a0415cd50 msgr2=0x7f1a040c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:44.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a09b01640 1 --2- 172.21.15.125:0/857748056 >> v2:172.21.15.136:3300/0 conn(0x7f1a0415cd50 0x7f1a040c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:44.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a09b01640 1 -- 172.21.15.125:0/857748056 >> v1:172.21.15.125:6789/0 conn(0x7f1a040cefa0 legacy=0x7f1a040cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:44.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a09b01640 1 -- 172.21.15.125:0/857748056 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0415dba0 con 0x7f1a041538f0 2023-10-30T13:08:44.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a08aff640 1 -- 172.21.15.125:0/857748056 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145250845 0 0) 0x7f1a0415d300 con 0x7f1a040cefa0 2023-10-30T13:08:44.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.015+0000 7f1a09300640 1 --2- 172.21.15.125:0/857748056 >> v2:172.21.15.136:3300/0 conn(0x7f1a0415cd50 0x7f1a040c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:44.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.019+0000 7f1a09b01640 1 --2- 172.21.15.125:0/857748056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a041538f0 0x7f1a04159fb0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f1a04072e90 tx=0x7f1a0001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db40ad00793cb508 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:44.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.019+0000 7f1a08aff640 1 -- 172.21.15.125:0/857748056 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a000a8950 con 0x7f1a041538f0 2023-10-30T13:08:44.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.019+0000 7f1a08aff640 1 -- 172.21.15.125:0/857748056 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a000a8ab0 con 0x7f1a041538f0 2023-10-30T13:08:44.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.019+0000 7f1a08aff640 1 -- 172.21.15.125:0/857748056 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a000a8e40 con 0x7f1a041538f0 2023-10-30T13:08:44.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.019+0000 7f1a0ab03640 1 -- 172.21.15.125:0/857748056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a041538f0 msgr2=0x7f1a04159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:44.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.019+0000 7f1a0ab03640 1 --2- 172.21.15.125:0/857748056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a041538f0 0x7f1a04159fb0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f1a04072e90 tx=0x7f1a0001ec50 comp rx=0 tx=0).stop 2023-10-30T13:08:44.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.019+0000 7f1a0ab03640 1 -- 172.21.15.125:0/857748056 shutdown_connections 2023-10-30T13:08:44.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.019+0000 7f1a0ab03640 1 --2- 172.21.15.125:0/857748056 >> v2:172.21.15.136:3300/0 conn(0x7f1a0415cd50 0x7f1a040c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:44.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.019+0000 7f1a0ab03640 1 --2- 172.21.15.125:0/857748056 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a041538f0 0x7f1a04159fb0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:44.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.019+0000 7f1a0ab03640 1 -- 172.21.15.125:0/857748056 >> 172.21.15.125:0/857748056 conn(0x7f1a0406a640 msgr2=0x7f1a04065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:44.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.019+0000 7f1a0ab03640 1 -- 172.21.15.125:0/857748056 shutdown_connections 2023-10-30T13:08:44.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.019+0000 7f1a0ab03640 1 -- 172.21.15.125:0/857748056 wait complete. 2023-10-30T13:08:44.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.019+0000 7f1a0ab03640 1 Processor -- start 2023-10-30T13:08:44.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0ab03640 1 -- start start 2023-10-30T13:08:44.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0ab03640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a040cefa0 0x7f1a041e53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:44.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0ab03640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a041538f0 0x7f1a041453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:44.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0ab03640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a0415cd50 0x7f1a0406bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:44.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0ab03640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a041f3400 con 0x7f1a0415cd50 2023-10-30T13:08:44.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0ab03640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a041f2860 con 0x7f1a040cefa0 2023-10-30T13:08:44.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0ab03640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a041f2ad0 con 0x7f1a041538f0 2023-10-30T13:08:44.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0a302640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a0415cd50 0x7f1a0406bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:44.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0a302640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a0415cd50 0x7f1a0406bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41522/0 (socket says 172.21.15.125:41522) 2023-10-30T13:08:44.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0a302640 1 -- 172.21.15.125:0/1513616830 learned_addr learned my addr 172.21.15.125:0/1513616830 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:44.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a09b01640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a040cefa0 0x7f1a041e53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:44.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a09300640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a041538f0 0x7f1a041453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:44.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0a302640 1 -- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a041538f0 msgr2=0x7f1a041453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:44.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0a302640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a041538f0 0x7f1a041453c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:44.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0a302640 1 -- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a040cefa0 msgr2=0x7f1a041e53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:44.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0a302640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a040cefa0 0x7f1a041e53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:44.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0a302640 1 -- 172.21.15.125:0/1513616830 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a00009420 con 0x7f1a0415cd50 2023-10-30T13:08:44.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a09b01640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a040cefa0 0x7f1a041e53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:44.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0a302640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a0415cd50 0x7f1a0406bd00 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00c810 tx=0x7f19fc00ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:44.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f19fa7fc640 1 -- 172.21.15.125:0/1513616830 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19fc00a640 con 0x7f1a0415cd50 2023-10-30T13:08:44.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f19fa7fc640 1 -- 172.21.15.125:0/1513616830 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19fc00a7c0 con 0x7f1a0415cd50 2023-10-30T13:08:44.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0ab03640 1 -- 172.21.15.125:0/1513616830 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a041f2db0 con 0x7f1a0415cd50 2023-10-30T13:08:44.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f19fa7fc640 1 -- 172.21.15.125:0/1513616830 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19fc00ab70 con 0x7f1a0415cd50 2023-10-30T13:08:44.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0ab03640 1 -- 172.21.15.125:0/1513616830 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a041e5920 con 0x7f1a0415cd50 2023-10-30T13:08:44.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.023+0000 7f1a0ab03640 1 -- 172.21.15.125:0/1513616830 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a041f35b0 con 0x7f1a0415cd50 2023-10-30T13:08:44.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.027+0000 7f19fa7fc640 1 -- 172.21.15.125:0/1513616830 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f19fc00acd0 con 0x7f1a0415cd50 2023-10-30T13:08:44.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.027+0000 7f19fa7fc640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19e80664d0 0x7f19e8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:44.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.027+0000 7f19fa7fc640 1 -- 172.21.15.125:0/1513616830 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f19fc083df0 con 0x7f1a0415cd50 2023-10-30T13:08:44.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.027+0000 7f1a09b01640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19e80664d0 0x7f19e8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:44.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.027+0000 7f1a09b01640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19e80664d0 0x7f19e8068990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f1a000b2000 tx=0x7f1a00008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:44.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.031+0000 7f19fa7fc640 1 -- 172.21.15.125:0/1513616830 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1a041f35b0 con 0x7f1a0415cd50 2023-10-30T13:08:44.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.243+0000 7f1a0ab03640 1 -- 172.21.15.125:0/1513616830 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a0404e170 con 0x7f1a0415cd50 2023-10-30T13:08:44.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.243+0000 7f19fa7fc640 1 -- 172.21.15.125:0/1513616830 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 185 v185) v1 ==== 95+0+6548 (secure 0 0 0) 0x7f19fc018090 con 0x7f1a0415cd50 2023-10-30T13:08:44.247 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:44.247 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":185,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":185,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:39.252653+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:44.248 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 185 2023-10-30T13:08:44.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.247+0000 7f1a0ab03640 1 -- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19e80664d0 msgr2=0x7f19e8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:44.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.247+0000 7f1a0ab03640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19e80664d0 0x7f19e8068990 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f1a000b2000 tx=0x7f1a00008040 comp rx=0 tx=0).stop 2023-10-30T13:08:44.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.247+0000 7f1a0ab03640 1 -- 172.21.15.125:0/1513616830 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a0415cd50 msgr2=0x7f1a0406bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:44.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.247+0000 7f1a0ab03640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a0415cd50 0x7f1a0406bd00 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f19fc00c810 tx=0x7f19fc00ccd0 comp rx=0 tx=0).stop 2023-10-30T13:08:44.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.247+0000 7f1a0ab03640 1 -- 172.21.15.125:0/1513616830 shutdown_connections 2023-10-30T13:08:44.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.247+0000 7f1a0ab03640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19e80664d0 0x7f19e8068990 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:44.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.247+0000 7f1a0ab03640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a0415cd50 0x7f1a0406bd00 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:44.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.247+0000 7f1a0ab03640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a041538f0 0x7f1a041453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:44.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.247+0000 7f1a0ab03640 1 --2- 172.21.15.125:0/1513616830 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a040cefa0 0x7f1a041e53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:44.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.247+0000 7f1a0ab03640 1 -- 172.21.15.125:0/1513616830 >> 172.21.15.125:0/1513616830 conn(0x7f1a0406a640 msgr2=0x7f1a040c4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:44.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.247+0000 7f1a0ab03640 1 -- 172.21.15.125:0/1513616830 shutdown_connections 2023-10-30T13:08:44.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:44.247+0000 7f1a0ab03640 1 -- 172.21.15.125:0/1513616830 wait complete. 2023-10-30T13:08:44.269 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:08:46.270 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:46.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f1333646640 1 Processor -- start 2023-10-30T13:08:46.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f1333646640 1 -- start start 2023-10-30T13:08:46.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f1333646640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f132c1539a0 0x7f132c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:46.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f1333646640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f132c069e90 0x7f132c06a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:46.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f1333646640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132c15a8c0 con 0x7f132c070a00 2023-10-30T13:08:46.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f1333646640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f132c06fbb0 con 0x7f132c1539a0 2023-10-30T13:08:46.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f1333646640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f132c153d80 con 0x7f132c069e90 2023-10-30T13:08:46.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f1330be1640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f132c070a00 0x7f132c070e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46214/0 (socket says 172.21.15.125:46214) 2023-10-30T13:08:46.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f1330be1640 1 -- 172.21.15.125:0/470399229 learned_addr learned my addr 172.21.15.125:0/470399229 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:46.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f13313e2640 1 --2- 172.21.15.125:0/470399229 >> v2:172.21.15.125:3300/0 conn(0x7f132c1539a0 0x7f132c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:46.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f1331be3640 1 --2- 172.21.15.125:0/470399229 >> v2:172.21.15.136:3300/0 conn(0x7f132c069e90 0x7f132c06a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:46.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f1323fff640 1 -- 172.21.15.125:0/470399229 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600475621 0 0) 0x7f132c15a8c0 con 0x7f132c070a00 2023-10-30T13:08:46.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f13313e2640 1 -- 172.21.15.125:0/470399229 >> v2:172.21.15.136:3300/0 conn(0x7f132c069e90 msgr2=0x7f132c06a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:46.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f13313e2640 1 --2- 172.21.15.125:0/470399229 >> v2:172.21.15.136:3300/0 conn(0x7f132c069e90 0x7f132c06a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:46.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f13313e2640 1 -- 172.21.15.125:0/470399229 >> v1:172.21.15.125:6789/0 conn(0x7f132c070a00 legacy=0x7f132c070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:46.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.407+0000 7f13313e2640 1 -- 172.21.15.125:0/470399229 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132c0c5c50 con 0x7f132c1539a0 2023-10-30T13:08:46.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.411+0000 7f13313e2640 1 --2- 172.21.15.125:0/470399229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f132c1539a0 0x7f132c15a380 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f132c0c8900 tx=0x7f132801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f1dcc8e7e7c4ef server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:46.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.411+0000 7f1323fff640 1 -- 172.21.15.125:0/470399229 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13280ab600 con 0x7f132c1539a0 2023-10-30T13:08:46.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.411+0000 7f1323fff640 1 -- 172.21.15.125:0/470399229 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13280a7050 con 0x7f132c1539a0 2023-10-30T13:08:46.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.411+0000 7f1323fff640 1 -- 172.21.15.125:0/470399229 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13280ab980 con 0x7f132c1539a0 2023-10-30T13:08:46.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.411+0000 7f1333646640 1 -- 172.21.15.125:0/470399229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f132c1539a0 msgr2=0x7f132c15a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:46.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.411+0000 7f1333646640 1 --2- 172.21.15.125:0/470399229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f132c1539a0 0x7f132c15a380 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f132c0c8900 tx=0x7f132801c3a0 comp rx=0 tx=0).stop 2023-10-30T13:08:46.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.411+0000 7f1333646640 1 -- 172.21.15.125:0/470399229 shutdown_connections 2023-10-30T13:08:46.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.411+0000 7f1333646640 1 --2- 172.21.15.125:0/470399229 >> v2:172.21.15.136:3300/0 conn(0x7f132c069e90 0x7f132c06a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:46.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.411+0000 7f1333646640 1 --2- 172.21.15.125:0/470399229 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f132c1539a0 0x7f132c15a380 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:46.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.411+0000 7f1333646640 1 -- 172.21.15.125:0/470399229 >> 172.21.15.125:0/470399229 conn(0x7f132c0c7910 msgr2=0x7f132c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:46.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.411+0000 7f1333646640 1 -- 172.21.15.125:0/470399229 shutdown_connections 2023-10-30T13:08:46.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.411+0000 7f1333646640 1 -- 172.21.15.125:0/470399229 wait complete. 2023-10-30T13:08:46.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1333646640 1 Processor -- start 2023-10-30T13:08:46.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1333646640 1 -- start start 2023-10-30T13:08:46.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1333646640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f132c069e90 0x7f132c156230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:46.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1333646640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f132c070a00 0x7f132c143ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:46.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1333646640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f132c1539a0 0x7f132c0c1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:46.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1333646640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132c156770 con 0x7f132c069e90 2023-10-30T13:08:46.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1333646640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132c1541f0 con 0x7f132c1539a0 2023-10-30T13:08:46.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1333646640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f132c144010 con 0x7f132c070a00 2023-10-30T13:08:46.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f13313e2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f132c069e90 0x7f132c156230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:46.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f13313e2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f132c069e90 0x7f132c156230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41538/0 (socket says 172.21.15.125:41538) 2023-10-30T13:08:46.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f13313e2640 1 -- 172.21.15.125:0/881253659 learned_addr learned my addr 172.21.15.125:0/881253659 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:46.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1331be3640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f132c1539a0 0x7f132c0c1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:46.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1330be1640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f132c070a00 0x7f132c143ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:46.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f13313e2640 1 -- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f132c070a00 msgr2=0x7f132c143ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:46.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f13313e2640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f132c070a00 0x7f132c143ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:46.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f13313e2640 1 -- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f132c1539a0 msgr2=0x7f132c0c1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:46.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f13313e2640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f132c1539a0 0x7f132c0c1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:46.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f13313e2640 1 -- 172.21.15.125:0/881253659 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1328009420 con 0x7f132c069e90 2023-10-30T13:08:46.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1331be3640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f132c1539a0 0x7f132c0c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:46.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1330be1640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f132c070a00 0x7f132c143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:46.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f13313e2640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f132c069e90 0x7f132c156230 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f13280ae040 tx=0x7f132801abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:46.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1321ffb640 1 -- 172.21.15.125:0/881253659 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13280b7650 con 0x7f132c069e90 2023-10-30T13:08:46.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1321ffb640 1 -- 172.21.15.125:0/881253659 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13280b77d0 con 0x7f132c069e90 2023-10-30T13:08:46.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1333646640 1 -- 172.21.15.125:0/881253659 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f132c151030 con 0x7f132c069e90 2023-10-30T13:08:46.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.415+0000 7f1321ffb640 1 -- 172.21.15.125:0/881253659 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13280b7b80 con 0x7f132c069e90 2023-10-30T13:08:46.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.419+0000 7f1333646640 1 -- 172.21.15.125:0/881253659 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f132c1500b0 con 0x7f132c069e90 2023-10-30T13:08:46.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.419+0000 7f1333646640 1 -- 172.21.15.125:0/881253659 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f132c04e170 con 0x7f132c069e90 2023-10-30T13:08:46.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.419+0000 7f1321ffb640 1 -- 172.21.15.125:0/881253659 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f13280b7ce0 con 0x7f132c069e90 2023-10-30T13:08:46.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.419+0000 7f1321ffb640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13100664a0 0x7f1310068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:46.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.419+0000 7f1321ffb640 1 -- 172.21.15.125:0/881253659 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f132811ee20 con 0x7f132c069e90 2023-10-30T13:08:46.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.419+0000 7f1330be1640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13100664a0 0x7f1310068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:46.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.423+0000 7f1330be1640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13100664a0 0x7f1310068960 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f132c0712b0 tx=0x7f131c007c40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:46.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.423+0000 7f1321ffb640 1 -- 172.21.15.125:0/881253659 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f13280ebc30 con 0x7f132c069e90 2023-10-30T13:08:46.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.603+0000 7f1333646640 1 -- 172.21.15.125:0/881253659 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f132c0c67a0 con 0x7f132c069e90 2023-10-30T13:08:46.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.603+0000 7f1321ffb640 1 -- 172.21.15.125:0/881253659 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+6550 (secure 0 0 0) 0x7f13280ef250 con 0x7f132c069e90 2023-10-30T13:08:46.607 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:46.608 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:44.194744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:46.608 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 186 2023-10-30T13:08:46.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.607+0000 7f1333646640 1 -- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13100664a0 msgr2=0x7f1310068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:46.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.607+0000 7f1333646640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13100664a0 0x7f1310068960 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f132c0712b0 tx=0x7f131c007c40 comp rx=0 tx=0).stop 2023-10-30T13:08:46.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.607+0000 7f1333646640 1 -- 172.21.15.125:0/881253659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f132c069e90 msgr2=0x7f132c156230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:46.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.607+0000 7f1333646640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f132c069e90 0x7f132c156230 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f13280ae040 tx=0x7f132801abe0 comp rx=0 tx=0).stop 2023-10-30T13:08:46.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.607+0000 7f1333646640 1 -- 172.21.15.125:0/881253659 shutdown_connections 2023-10-30T13:08:46.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.607+0000 7f1333646640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13100664a0 0x7f1310068960 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:46.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.607+0000 7f1333646640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f132c1539a0 0x7f132c0c1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:46.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.607+0000 7f1333646640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f132c070a00 0x7f132c143ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:46.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.607+0000 7f1333646640 1 --2- 172.21.15.125:0/881253659 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f132c069e90 0x7f132c156230 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:46.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.607+0000 7f1333646640 1 -- 172.21.15.125:0/881253659 >> 172.21.15.125:0/881253659 conn(0x7f132c0c7910 msgr2=0x7f132c149be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:46.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.607+0000 7f1333646640 1 -- 172.21.15.125:0/881253659 shutdown_connections 2023-10-30T13:08:46.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:46.607+0000 7f1333646640 1 -- 172.21.15.125:0/881253659 wait complete. 2023-10-30T13:08:46.628 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:08:48.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:48.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f535e3aa640 1 Processor -- start 2023-10-30T13:08:48.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f535e3aa640 1 -- start start 2023-10-30T13:08:48.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f535e3aa640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f53580c6bd0 0x7f535806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:48.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f535e3aa640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5358066cf0 con 0x7f535806f7a0 2023-10-30T13:08:48.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f535e3aa640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53580c70d0 con 0x7f5358071110 2023-10-30T13:08:48.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f535e3aa640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53580700c0 con 0x7f53580c6bd0 2023-10-30T13:08:48.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5357fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f535806f7a0 0x7f5358069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46218/0 (socket says 172.21.15.125:46218) 2023-10-30T13:08:48.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5357fff640 1 -- 172.21.15.125:0/3961394058 learned_addr learned my addr 172.21.15.125:0/3961394058 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:48.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f53577fe640 1 --2- 172.21.15.125:0/3961394058 >> v2:172.21.15.136:3300/0 conn(0x7f53580c6bd0 0x7f535806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:48.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 -- 172.21.15.125:0/3961394058 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 185885294 0 0) 0x7f5358066cf0 con 0x7f535806f7a0 2023-10-30T13:08:48.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 -- 172.21.15.125:0/3961394058 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5340003530 con 0x7f535806f7a0 2023-10-30T13:08:48.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 -- 172.21.15.125:0/3961394058 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2698895969 0 0) 0x7f5340003530 con 0x7f535806f7a0 2023-10-30T13:08:48.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 -- 172.21.15.125:0/3961394058 >> v2:172.21.15.136:3300/0 conn(0x7f53580c6bd0 msgr2=0x7f535806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:48.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 --2- 172.21.15.125:0/3961394058 >> v2:172.21.15.136:3300/0 conn(0x7f53580c6bd0 0x7f535806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:48.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 -- 172.21.15.125:0/3961394058 >> v1:172.21.15.136:6789/0 conn(0x7f5358071110 legacy=0x7f5358158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:48.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 -- 172.21.15.125:0/3961394058 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53580c7390 con 0x7f535806f7a0 2023-10-30T13:08:48.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 -- 172.21.15.125:0/3961394058 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f5348001680 con 0x7f535806f7a0 2023-10-30T13:08:48.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 -- 172.21.15.125:0/3961394058 >> v1:172.21.15.125:6789/0 conn(0x7f535806f7a0 legacy=0x7f5358069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:48.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 --2- 172.21.15.125:0/3961394058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5340003980 0x7f5340003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:48.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 --2- 172.21.15.125:0/3961394058 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53400047d0 0x7f5340004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:48.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 --2- 172.21.15.125:0/3961394058 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340005190 0x7f534000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:48.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 -- 172.21.15.125:0/3961394058 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53580700c0 con 0x7f5340005190 2023-10-30T13:08:48.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 -- 172.21.15.125:0/3961394058 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f534000c040 con 0x7f5340003980 2023-10-30T13:08:48.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5356ffd640 1 -- 172.21.15.125:0/3961394058 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f534000c200 con 0x7f53400047d0 2023-10-30T13:08:48.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f535c947640 1 --2- 172.21.15.125:0/3961394058 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340005190 0x7f534000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:48.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f5357fff640 1 --2- 172.21.15.125:0/3961394058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5340003980 0x7f5340003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:48.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f53577fe640 1 --2- 172.21.15.125:0/3961394058 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53400047d0 0x7f5340004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:48.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f535c947640 1 -- 172.21.15.125:0/3961394058 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53400047d0 msgr2=0x7f5340004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:48.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f535c947640 1 --2- 172.21.15.125:0/3961394058 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53400047d0 0x7f5340004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:48.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f535c947640 1 -- 172.21.15.125:0/3961394058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5340003980 msgr2=0x7f5340003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:48.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f535c947640 1 --2- 172.21.15.125:0/3961394058 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5340003980 0x7f5340003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:48.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.775+0000 7f535c947640 1 -- 172.21.15.125:0/3961394058 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5340003530 con 0x7f5340005190 2023-10-30T13:08:48.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.779+0000 7f535c947640 1 -- 172.21.15.125:0/3961394058 reap_dead start 2023-10-30T13:08:48.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.779+0000 7f535c947640 1 --2- 172.21.15.125:0/3961394058 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340005190 0x7f534000b990 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f5350001e70 tx=0x7f535001c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a766b5cdf2bbb073 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:48.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.779+0000 7f5356ffd640 1 -- 172.21.15.125:0/3961394058 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350003b80 con 0x7f5340005190 2023-10-30T13:08:48.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.779+0000 7f5356ffd640 1 -- 172.21.15.125:0/3961394058 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5350003d20 con 0x7f5340005190 2023-10-30T13:08:48.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.779+0000 7f5356ffd640 1 -- 172.21.15.125:0/3961394058 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350005260 con 0x7f5340005190 2023-10-30T13:08:48.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.779+0000 7f535e3aa640 1 -- 172.21.15.125:0/3961394058 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340005190 msgr2=0x7f534000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:48.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.779+0000 7f535e3aa640 1 --2- 172.21.15.125:0/3961394058 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340005190 0x7f534000b990 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f5350001e70 tx=0x7f535001c760 comp rx=0 tx=0).stop 2023-10-30T13:08:48.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.779+0000 7f535e3aa640 1 -- 172.21.15.125:0/3961394058 shutdown_connections 2023-10-30T13:08:48.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.779+0000 7f535e3aa640 1 --2- 172.21.15.125:0/3961394058 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340005190 0x7f534000b990 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:48.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.779+0000 7f535e3aa640 1 -- 172.21.15.125:0/3961394058 >> 172.21.15.125:0/3961394058 conn(0x7f53580c7910 msgr2=0x7f53580c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:48.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.779+0000 7f535e3aa640 1 -- 172.21.15.125:0/3961394058 shutdown_connections 2023-10-30T13:08:48.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.779+0000 7f535e3aa640 1 -- 172.21.15.125:0/3961394058 wait complete. 2023-10-30T13:08:48.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f535e3aa640 1 Processor -- start 2023-10-30T13:08:48.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f535e3aa640 1 -- start start 2023-10-30T13:08:48.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f535e3aa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340005190 0x7f5358146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:48.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f535e3aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53581ec680 0x7f5358144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:48.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f535e3aa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5358145200 0x7f5358143770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:48.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f535e3aa640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535814a8b0 con 0x7f5340005190 2023-10-30T13:08:48.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f535e3aa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5358144550 con 0x7f5358145200 2023-10-30T13:08:48.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f535e3aa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5358144830 con 0x7f53581ec680 2023-10-30T13:08:48.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f5357fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340005190 0x7f5358146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:48.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f5357fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340005190 0x7f5358146da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41550/0 (socket says 172.21.15.125:41550) 2023-10-30T13:08:48.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f5357fff640 1 -- 172.21.15.125:0/3107183341 learned_addr learned my addr 172.21.15.125:0/3107183341 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:48.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f535c947640 1 --2- 172.21.15.125:0/3107183341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5358145200 0x7f5358143770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:48.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f53577fe640 1 --2- 172.21.15.125:0/3107183341 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53581ec680 0x7f5358144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:48.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f5357fff640 1 -- 172.21.15.125:0/3107183341 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53581ec680 msgr2=0x7f5358144cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:48.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f5357fff640 1 --2- 172.21.15.125:0/3107183341 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53581ec680 0x7f5358144cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:48.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f5357fff640 1 -- 172.21.15.125:0/3107183341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5358145200 msgr2=0x7f5358143770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:48.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f5357fff640 1 --2- 172.21.15.125:0/3107183341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5358145200 0x7f5358143770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:48.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f5357fff640 1 -- 172.21.15.125:0/3107183341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53581ecbe0 con 0x7f5340005190 2023-10-30T13:08:48.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f5357fff640 1 --2- 172.21.15.125:0/3107183341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340005190 0x7f5358146da0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f5348007290 tx=0x7f5348007750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:48.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f5354ff9640 1 -- 172.21.15.125:0/3107183341 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f534800f9c0 con 0x7f5340005190 2023-10-30T13:08:48.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f5354ff9640 1 -- 172.21.15.125:0/3107183341 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f534800fb40 con 0x7f5340005190 2023-10-30T13:08:48.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f535e3aa640 1 -- 172.21.15.125:0/3107183341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53581ecec0 con 0x7f5340005190 2023-10-30T13:08:48.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f5354ff9640 1 -- 172.21.15.125:0/3107183341 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53480133f0 con 0x7f5340005190 2023-10-30T13:08:48.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.783+0000 7f535e3aa640 1 -- 172.21.15.125:0/3107183341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53581ed480 con 0x7f5340005190 2023-10-30T13:08:48.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.787+0000 7f535e3aa640 1 -- 172.21.15.125:0/3107183341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53581ed610 con 0x7f5340005190 2023-10-30T13:08:48.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.787+0000 7f5354ff9640 1 -- 172.21.15.125:0/3107183341 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5348013550 con 0x7f5340005190 2023-10-30T13:08:48.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.787+0000 7f5354ff9640 1 --2- 172.21.15.125:0/3107183341 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f53440664a0 0x7f5344068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:48.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.787+0000 7f5354ff9640 1 -- 172.21.15.125:0/3107183341 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f53480848c0 con 0x7f5340005190 2023-10-30T13:08:48.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.787+0000 7f53577fe640 1 --2- 172.21.15.125:0/3107183341 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f53440664a0 0x7f5344068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:48.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.787+0000 7f53577fe640 1 --2- 172.21.15.125:0/3107183341 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f53440664a0 0x7f5344068960 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f53580c2c30 tx=0x7f534c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:48.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.791+0000 7f5354ff9640 1 -- 172.21.15.125:0/3107183341 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5348051490 con 0x7f5340005190 2023-10-30T13:08:48.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.971+0000 7f535e3aa640 1 -- 172.21.15.125:0/3107183341 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f535804e170 con 0x7f5340005190 2023-10-30T13:08:48.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.971+0000 7f5354ff9640 1 -- 172.21.15.125:0/3107183341 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+6550 (secure 0 0 0) 0x7f5348054ab0 con 0x7f5340005190 2023-10-30T13:08:48.975 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:48.975 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:44.194744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:48.977 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 186 2023-10-30T13:08:48.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.975+0000 7f535e3aa640 1 -- 172.21.15.125:0/3107183341 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f53440664a0 msgr2=0x7f5344068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:48.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.975+0000 7f535e3aa640 1 --2- 172.21.15.125:0/3107183341 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f53440664a0 0x7f5344068960 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f53580c2c30 tx=0x7f534c008040 comp rx=0 tx=0).stop 2023-10-30T13:08:48.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.975+0000 7f535e3aa640 1 -- 172.21.15.125:0/3107183341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340005190 msgr2=0x7f5358146da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:48.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.975+0000 7f535e3aa640 1 --2- 172.21.15.125:0/3107183341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340005190 0x7f5358146da0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f5348007290 tx=0x7f5348007750 comp rx=0 tx=0).stop 2023-10-30T13:08:48.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.975+0000 7f535e3aa640 1 -- 172.21.15.125:0/3107183341 shutdown_connections 2023-10-30T13:08:48.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.975+0000 7f535e3aa640 1 --2- 172.21.15.125:0/3107183341 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f53440664a0 0x7f5344068960 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:48.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.975+0000 7f535e3aa640 1 --2- 172.21.15.125:0/3107183341 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5358145200 0x7f5358143770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:48.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.975+0000 7f535e3aa640 1 --2- 172.21.15.125:0/3107183341 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53581ec680 0x7f5358144cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:48.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.975+0000 7f535e3aa640 1 --2- 172.21.15.125:0/3107183341 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5340005190 0x7f5358146da0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:48.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.975+0000 7f535e3aa640 1 -- 172.21.15.125:0/3107183341 >> 172.21.15.125:0/3107183341 conn(0x7f53580c7910 msgr2=0x7f53580ca5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:48.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.975+0000 7f535e3aa640 1 -- 172.21.15.125:0/3107183341 shutdown_connections 2023-10-30T13:08:48.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:48.975+0000 7f535e3aa640 1 -- 172.21.15.125:0/3107183341 wait complete. 2023-10-30T13:08:48.996 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:08:50.996 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:51.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.139+0000 7f7588565640 1 Processor -- start 2023-10-30T13:08:51.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.139+0000 7f7588565640 1 -- start start 2023-10-30T13:08:51.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f7588565640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f758006f7a0 0x7f758006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:51.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f7588565640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f75800c6bd0 0x7f7580069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:51.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f7588565640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f758006a3d0 con 0x7f7580071110 2023-10-30T13:08:51.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f7588565640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75800700c0 con 0x7f758006f7a0 2023-10-30T13:08:51.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f7588565640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7580066cf0 con 0x7f75800c6bd0 2023-10-30T13:08:51.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f7586301640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f758006f7a0 0x7f758006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:51.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f7586301640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f758006f7a0 0x7f758006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56004/0 (socket says 172.21.15.125:56004) 2023-10-30T13:08:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f7586301640 1 -- 172.21.15.125:0/2575344259 learned_addr learned my addr 172.21.15.125:0/2575344259 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f7585b00640 1 --2- 172.21.15.125:0/2575344259 >> v2:172.21.15.136:3300/0 conn(0x7f75800c6bd0 0x7f7580069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f7586301640 1 -- 172.21.15.125:0/2575344259 >> v2:172.21.15.136:3300/0 conn(0x7f75800c6bd0 msgr2=0x7f7580069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f7586301640 1 --2- 172.21.15.125:0/2575344259 >> v2:172.21.15.136:3300/0 conn(0x7f75800c6bd0 0x7f7580069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:51.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f7586301640 1 -- 172.21.15.125:0/2575344259 >> v1:172.21.15.136:6789/0 conn(0x7f7580071110 legacy=0x7f7580158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:51.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f7586301640 1 -- 172.21.15.125:0/2575344259 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75800c70d0 con 0x7f758006f7a0 2023-10-30T13:08:51.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f7586301640 1 --2- 172.21.15.125:0/2575344259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758006f7a0 0x7f758006fb80 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f758006f1c0 tx=0x7f757c01ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4435bd9f37927986 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f75852ff640 1 -- 172.21.15.125:0/2575344259 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c0a89b0 con 0x7f758006f7a0 2023-10-30T13:08:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.143+0000 7f75852ff640 1 -- 172.21.15.125:0/2575344259 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c0a8b10 con 0x7f758006f7a0 2023-10-30T13:08:51.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f75852ff640 1 -- 172.21.15.125:0/2575344259 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c0a8ea0 con 0x7f758006f7a0 2023-10-30T13:08:51.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 -- 172.21.15.125:0/2575344259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758006f7a0 msgr2=0x7f758006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:51.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 --2- 172.21.15.125:0/2575344259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758006f7a0 0x7f758006fb80 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f758006f1c0 tx=0x7f757c01ecb0 comp rx=0 tx=0).stop 2023-10-30T13:08:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 -- 172.21.15.125:0/2575344259 shutdown_connections 2023-10-30T13:08:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 --2- 172.21.15.125:0/2575344259 >> v2:172.21.15.136:3300/0 conn(0x7f75800c6bd0 0x7f7580069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 --2- 172.21.15.125:0/2575344259 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758006f7a0 0x7f758006fb80 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 -- 172.21.15.125:0/2575344259 >> 172.21.15.125:0/2575344259 conn(0x7f75800c7910 msgr2=0x7f75800c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:51.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 -- 172.21.15.125:0/2575344259 shutdown_connections 2023-10-30T13:08:51.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 -- 172.21.15.125:0/2575344259 wait complete. 2023-10-30T13:08:51.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 Processor -- start 2023-10-30T13:08:51.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 -- start start 2023-10-30T13:08:51.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758006f7a0 0x7f758014af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:51.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7580071110 0x7f758014ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:51.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75800c6bd0 0x7f7580150490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:51.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758014fb30 con 0x7f758006f7a0 2023-10-30T13:08:51.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758014ef40 con 0x7f75800c6bd0 2023-10-30T13:08:51.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7588565640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75801efd00 con 0x7f7580071110 2023-10-30T13:08:51.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7586301640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758006f7a0 0x7f758014af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:51.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7586301640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758006f7a0 0x7f758014af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56020/0 (socket says 172.21.15.125:56020) 2023-10-30T13:08:51.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7586301640 1 -- 172.21.15.125:0/1986524693 learned_addr learned my addr 172.21.15.125:0/1986524693 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:51.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7585b00640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7580071110 0x7f758014ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:51.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7586b02640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75800c6bd0 0x7f7580150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:51.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7586301640 1 -- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7580071110 msgr2=0x7f758014ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:51.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7586301640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7580071110 0x7f758014ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:51.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7586301640 1 -- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75800c6bd0 msgr2=0x7f7580150490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:51.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7586301640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75800c6bd0 0x7f7580150490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:51.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.147+0000 7f7586301640 1 -- 172.21.15.125:0/1986524693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c0094d0 con 0x7f758006f7a0 2023-10-30T13:08:51.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.151+0000 7f7585b00640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7580071110 0x7f758014ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:08:51.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.151+0000 7f7586b02640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75800c6bd0 0x7f7580150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:08:51.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.151+0000 7f7586301640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758006f7a0 0x7f758014af00 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f757c01d360 tx=0x7f757c01ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:51.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.151+0000 7f7576ffd640 1 -- 172.21.15.125:0/1986524693 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c0ba600 con 0x7f758006f7a0 2023-10-30T13:08:51.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.151+0000 7f7576ffd640 1 -- 172.21.15.125:0/1986524693 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757c0ba760 con 0x7f758006f7a0 2023-10-30T13:08:51.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.151+0000 7f7576ffd640 1 -- 172.21.15.125:0/1986524693 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757c0bab10 con 0x7f758006f7a0 2023-10-30T13:08:51.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.151+0000 7f7588565640 1 -- 172.21.15.125:0/1986524693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f758014f190 con 0x7f758006f7a0 2023-10-30T13:08:51.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.151+0000 7f7588565640 1 -- 172.21.15.125:0/1986524693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f758014f750 con 0x7f758006f7a0 2023-10-30T13:08:51.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.151+0000 7f7588565640 1 -- 172.21.15.125:0/1986524693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f758014fda0 con 0x7f758006f7a0 2023-10-30T13:08:51.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.151+0000 7f7576ffd640 1 -- 172.21.15.125:0/1986524693 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f757c0bac70 con 0x7f758006f7a0 2023-10-30T13:08:51.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.151+0000 7f7576ffd640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75640664a0 0x7f7564068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:51.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.151+0000 7f7576ffd640 1 -- 172.21.15.125:0/1986524693 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f757c122b60 con 0x7f758006f7a0 2023-10-30T13:08:51.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.151+0000 7f7585b00640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75640664a0 0x7f7564068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:51.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.155+0000 7f7585b00640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75640664a0 0x7f7564068960 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f75800b0650 tx=0x7f7570008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:51.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.155+0000 7f7576ffd640 1 -- 172.21.15.125:0/1986524693 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f757c0ef970 con 0x7f758006f7a0 2023-10-30T13:08:51.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.339+0000 7f7588565640 1 -- 172.21.15.125:0/1986524693 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f758014fda0 con 0x7f758006f7a0 2023-10-30T13:08:51.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.339+0000 7f7576ffd640 1 -- 172.21.15.125:0/1986524693 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+6550 (secure 0 0 0) 0x7f757c0b5020 con 0x7f758006f7a0 2023-10-30T13:08:51.341 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:51.341 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:44.194744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:51.341 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 186 2023-10-30T13:08:51.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.339+0000 7f7588565640 1 -- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75640664a0 msgr2=0x7f7564068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:51.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.339+0000 7f7588565640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75640664a0 0x7f7564068960 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f75800b0650 tx=0x7f7570008040 comp rx=0 tx=0).stop 2023-10-30T13:08:51.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.339+0000 7f7588565640 1 -- 172.21.15.125:0/1986524693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758006f7a0 msgr2=0x7f758014af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:51.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.339+0000 7f7588565640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758006f7a0 0x7f758014af00 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7f757c01d360 tx=0x7f757c01ba50 comp rx=0 tx=0).stop 2023-10-30T13:08:51.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.339+0000 7f7588565640 1 -- 172.21.15.125:0/1986524693 shutdown_connections 2023-10-30T13:08:51.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.339+0000 7f7588565640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75640664a0 0x7f7564068960 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:51.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.339+0000 7f7588565640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75800c6bd0 0x7f7580150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:51.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.339+0000 7f7588565640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7580071110 0x7f758014ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:51.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.339+0000 7f7588565640 1 --2- 172.21.15.125:0/1986524693 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f758006f7a0 0x7f758014af00 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:51.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.339+0000 7f7588565640 1 -- 172.21.15.125:0/1986524693 >> 172.21.15.125:0/1986524693 conn(0x7f75800c7910 msgr2=0x7f75800c7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:51.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.339+0000 7f7588565640 1 -- 172.21.15.125:0/1986524693 shutdown_connections 2023-10-30T13:08:51.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:51.339+0000 7f7588565640 1 -- 172.21.15.125:0/1986524693 wait complete. 2023-10-30T13:08:51.361 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:08:53.362 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:53.426 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:08:53.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:08:53.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.507+0000 7f13acb65640 1 Processor -- start 2023-10-30T13:08:53.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.507+0000 7f13acb65640 1 -- start start 2023-10-30T13:08:53.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f13acb65640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f13a80c05d0 0x7f13a813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:53.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f13acb65640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f13a80c6f60 0x7f13a813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:53.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f13acb65640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13a80c0a50 con 0x7f13a80c6b80 2023-10-30T13:08:53.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f13acb65640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a806f6e0 con 0x7f13a80c05d0 2023-10-30T13:08:53.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f13acb65640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13a806f9c0 con 0x7f13a80c6f60 2023-10-30T13:08:53.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f13a659c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f13a80c6b80 0x7f13a80bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33616/0 (socket says 172.21.15.125:33616) 2023-10-30T13:08:53.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f13a659c640 1 -- 172.21.15.125:0/363158364 learned_addr learned my addr 172.21.15.125:0/363158364 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:53.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f139dd9b640 1 --2- 172.21.15.125:0/363158364 >> v2:172.21.15.125:3300/0 conn(0x7f13a80c05d0 0x7f13a813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:53.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f13a6d9d640 1 --2- 172.21.15.125:0/363158364 >> v2:172.21.15.136:3300/0 conn(0x7f13a80c6f60 0x7f13a813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:53.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f139dd9b640 1 -- 172.21.15.125:0/363158364 >> v2:172.21.15.136:3300/0 conn(0x7f13a80c6f60 msgr2=0x7f13a813f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:53.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f139dd9b640 1 --2- 172.21.15.125:0/363158364 >> v2:172.21.15.136:3300/0 conn(0x7f13a80c6f60 0x7f13a813f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:53.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f139dd9b640 1 -- 172.21.15.125:0/363158364 >> v1:172.21.15.125:6789/0 conn(0x7f13a80c6b80 legacy=0x7f13a80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:53.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f139dd9b640 1 -- 172.21.15.125:0/363158364 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a806fca0 con 0x7f13a80c05d0 2023-10-30T13:08:53.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f13a6d9d640 1 --2- 172.21.15.125:0/363158364 >> v2:172.21.15.136:3300/0 conn(0x7f13a80c6f60 0x7f13a813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:53.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.511+0000 7f13a5d9b640 1 -- 172.21.15.125:0/363158364 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422759564 0 0) 0x7f13a80c0a50 con 0x7f13a80c6b80 2023-10-30T13:08:53.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.515+0000 7f139dd9b640 1 --2- 172.21.15.125:0/363158364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13a80c05d0 0x7f13a813ed90 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f13a80c89b0 tx=0x7f139801c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b997ef1265445e69 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:53.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.515+0000 7f13a5d9b640 1 -- 172.21.15.125:0/363158364 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13980ab600 con 0x7f13a80c05d0 2023-10-30T13:08:53.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.515+0000 7f13a5d9b640 1 -- 172.21.15.125:0/363158364 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13980a7050 con 0x7f13a80c05d0 2023-10-30T13:08:53.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.515+0000 7f13a5d9b640 1 -- 172.21.15.125:0/363158364 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13980ab980 con 0x7f13a80c05d0 2023-10-30T13:08:53.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.515+0000 7f13acb65640 1 -- 172.21.15.125:0/363158364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13a80c05d0 msgr2=0x7f13a813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:53.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.515+0000 7f13acb65640 1 --2- 172.21.15.125:0/363158364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13a80c05d0 0x7f13a813ed90 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7f13a80c89b0 tx=0x7f139801c460 comp rx=0 tx=0).stop 2023-10-30T13:08:53.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.515+0000 7f13acb65640 1 -- 172.21.15.125:0/363158364 shutdown_connections 2023-10-30T13:08:53.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.515+0000 7f13acb65640 1 --2- 172.21.15.125:0/363158364 >> v2:172.21.15.136:3300/0 conn(0x7f13a80c6f60 0x7f13a813f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:53.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.515+0000 7f13acb65640 1 --2- 172.21.15.125:0/363158364 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13a80c05d0 0x7f13a813ed90 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:53.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.515+0000 7f13acb65640 1 -- 172.21.15.125:0/363158364 >> 172.21.15.125:0/363158364 conn(0x7f13a80c7910 msgr2=0x7f13a80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:53.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.515+0000 7f13acb65640 1 -- 172.21.15.125:0/363158364 shutdown_connections 2023-10-30T13:08:53.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.515+0000 7f13acb65640 1 -- 172.21.15.125:0/363158364 wait complete. 2023-10-30T13:08:53.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.515+0000 7f13acb65640 1 Processor -- start 2023-10-30T13:08:53.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.515+0000 7f13acb65640 1 -- start start 2023-10-30T13:08:53.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f13acb65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f13a80c05d0 0x7f13a81303f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:53.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f13acb65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13a80c6b80 0x7f13a81f21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:53.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f13acb65640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13a80c6f60 0x7f13a81f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:53.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f13acb65640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a81f1d80 con 0x7f13a80c6b80 2023-10-30T13:08:53.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f13acb65640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a8132550 con 0x7f13a80c6f60 2023-10-30T13:08:53.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f13acb65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a80c99e0 con 0x7f13a80c05d0 2023-10-30T13:08:53.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f139dd9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13a80c6b80 0x7f13a81f21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:53.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f139dd9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13a80c6b80 0x7f13a81f21b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56038/0 (socket says 172.21.15.125:56038) 2023-10-30T13:08:53.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f139dd9b640 1 -- 172.21.15.125:0/1798717833 learned_addr learned my addr 172.21.15.125:0/1798717833 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:53.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f13a6d9d640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13a80c6f60 0x7f13a81f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:53.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f13a659c640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f13a80c05d0 0x7f13a81303f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:53.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f139dd9b640 1 -- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f13a80c05d0 msgr2=0x7f13a81303f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:53.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f139dd9b640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f13a80c05d0 0x7f13a81303f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:53.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f139dd9b640 1 -- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13a80c6f60 msgr2=0x7f13a81f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:53.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f139dd9b640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13a80c6f60 0x7f13a81f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:53.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f139dd9b640 1 -- 172.21.15.125:0/1798717833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1398009420 con 0x7f13a80c6b80 2023-10-30T13:08:53.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f13a6d9d640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13a80c6f60 0x7f13a81f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:53.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f139dd9b640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13a80c6b80 0x7f13a81f21b0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f13980ae040 tx=0x7f1398019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:53.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f139f7fe640 1 -- 172.21.15.125:0/1798717833 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13980b4840 con 0x7f13a80c6b80 2023-10-30T13:08:53.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f139f7fe640 1 -- 172.21.15.125:0/1798717833 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13980b49c0 con 0x7f13a80c6b80 2023-10-30T13:08:53.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f139f7fe640 1 -- 172.21.15.125:0/1798717833 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13980b4d70 con 0x7f13a80c6b80 2023-10-30T13:08:53.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f13acb65640 1 -- 172.21.15.125:0/1798717833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13a8159570 con 0x7f13a80c6b80 2023-10-30T13:08:53.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f13acb65640 1 -- 172.21.15.125:0/1798717833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13a8132760 con 0x7f13a80c6b80 2023-10-30T13:08:53.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.519+0000 7f13acb65640 1 -- 172.21.15.125:0/1798717833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13a81f1ed0 con 0x7f13a80c6b80 2023-10-30T13:08:53.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.523+0000 7f139f7fe640 1 -- 172.21.15.125:0/1798717833 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f13980b2040 con 0x7f13a80c6b80 2023-10-30T13:08:53.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.523+0000 7f139f7fe640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13880664a0 0x7f1388068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:53.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.523+0000 7f139f7fe640 1 -- 172.21.15.125:0/1798717833 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 7630+0+0 (secure 0 0 0) 0x7f139811eb40 con 0x7f13a80c6b80 2023-10-30T13:08:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.523+0000 7f13a659c640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13880664a0 0x7f1388068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.523+0000 7f13a659c640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13880664a0 0x7f1388068960 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f1390000f70 tx=0x7f1390007b60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:53.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.527+0000 7f139f7fe640 1 -- 172.21.15.125:0/1798717833 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f13a81f1ed0 con 0x7f13a80c6b80 2023-10-30T13:08:53.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.707+0000 7f13acb65640 1 -- 172.21.15.125:0/1798717833 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13a81f1ed0 con 0x7f13a80c6b80 2023-10-30T13:08:53.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.707+0000 7f139f7fe640 1 -- 172.21.15.125:0/1798717833 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+6550 (secure 0 0 0) 0x7f13a81f1ed0 con 0x7f13a80c6b80 2023-10-30T13:08:53.710 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:53.710 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7340,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":184}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:44.194744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":6017},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6017":{"gid":6017,"name":"f","rank":2,"incarnation":170,"state":"up:active","state_seq":251,"addr":"172.21.15.136:6839/223464319","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":223464319},{"type":"v1","addr":"172.21.15.136:6839","nonce":223464319}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:53.710 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 186 2023-10-30T13:08:53.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.707+0000 7f13acb65640 1 -- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13880664a0 msgr2=0x7f1388068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:53.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.707+0000 7f13acb65640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13880664a0 0x7f1388068960 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f1390000f70 tx=0x7f1390007b60 comp rx=0 tx=0).stop 2023-10-30T13:08:53.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.707+0000 7f13acb65640 1 -- 172.21.15.125:0/1798717833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13a80c6b80 msgr2=0x7f13a81f21b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:53.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.707+0000 7f13acb65640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13a80c6b80 0x7f13a81f21b0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7f13980ae040 tx=0x7f1398019200 comp rx=0 tx=0).stop 2023-10-30T13:08:53.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.707+0000 7f13acb65640 1 -- 172.21.15.125:0/1798717833 shutdown_connections 2023-10-30T13:08:53.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.707+0000 7f13acb65640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f13880664a0 0x7f1388068960 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:53.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.707+0000 7f13acb65640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f13a80c6f60 0x7f13a81f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:53.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.707+0000 7f13acb65640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f13a80c6b80 0x7f13a81f21b0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:53.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.707+0000 7f13acb65640 1 --2- 172.21.15.125:0/1798717833 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f13a80c05d0 0x7f13a81303f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:53.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.707+0000 7f13acb65640 1 -- 172.21.15.125:0/1798717833 >> 172.21.15.125:0/1798717833 conn(0x7f13a80c7910 msgr2=0x7f13a80bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:53.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.707+0000 7f13acb65640 1 -- 172.21.15.125:0/1798717833 shutdown_connections 2023-10-30T13:08:53.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:53.707+0000 7f13acb65640 1 -- 172.21.15.125:0/1798717833 wait complete. 2023-10-30T13:08:53.730 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:08:55.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:55.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.871+0000 7f969153e640 1 Processor -- start 2023-10-30T13:08:55.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.871+0000 7f969153e640 1 -- start start 2023-10-30T13:08:55.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.871+0000 7f969153e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f968c0c05d0 0x7f968c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:55.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.871+0000 7f969153e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f968c0c6f60 0x7f968c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:55.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.871+0000 7f969153e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f968c0c0a50 con 0x7f968c0c6b80 2023-10-30T13:08:55.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.871+0000 7f969153e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f968c06f6e0 con 0x7f968c0c05d0 2023-10-30T13:08:55.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.871+0000 7f969153e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f968c06f9c0 con 0x7f968c0c6f60 2023-10-30T13:08:55.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.871+0000 7f9683fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f968c0c05d0 0x7f968c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:55.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.871+0000 7f9683fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f968c0c05d0 0x7f968c13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56046/0 (socket says 172.21.15.125:56046) 2023-10-30T13:08:55.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.871+0000 7f9683fff640 1 -- 172.21.15.125:0/2016182575 learned_addr learned my addr 172.21.15.125:0/2016182575 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:55.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.871+0000 7f968b7fe640 1 --2- 172.21.15.125:0/2016182575 >> v2:172.21.15.136:3300/0 conn(0x7f968c0c6f60 0x7f968c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:55.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f9683fff640 1 -- 172.21.15.125:0/2016182575 >> v2:172.21.15.136:3300/0 conn(0x7f968c0c6f60 msgr2=0x7f968c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:55.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f9683fff640 1 --2- 172.21.15.125:0/2016182575 >> v2:172.21.15.136:3300/0 conn(0x7f968c0c6f60 0x7f968c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:55.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f9683fff640 1 -- 172.21.15.125:0/2016182575 >> v1:172.21.15.136:6789/0 conn(0x7f968c0c6b80 legacy=0x7f968c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:55.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f9683fff640 1 -- 172.21.15.125:0/2016182575 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c06fca0 con 0x7f968c0c05d0 2023-10-30T13:08:55.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f9683fff640 1 --2- 172.21.15.125:0/2016182575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f968c0c05d0 0x7f968c13ed90 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f968c0c89b0 tx=0x7f967401ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=936b286af2aad2b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:55.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f968a7fc640 1 -- 172.21.15.125:0/2016182575 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96740a7bb0 con 0x7f968c0c05d0 2023-10-30T13:08:55.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f968a7fc640 1 -- 172.21.15.125:0/2016182575 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96740a7d10 con 0x7f968c0c05d0 2023-10-30T13:08:55.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f968a7fc640 1 -- 172.21.15.125:0/2016182575 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96740af660 con 0x7f968c0c05d0 2023-10-30T13:08:55.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f969153e640 1 -- 172.21.15.125:0/2016182575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f968c0c05d0 msgr2=0x7f968c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:55.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f969153e640 1 --2- 172.21.15.125:0/2016182575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f968c0c05d0 0x7f968c13ed90 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f968c0c89b0 tx=0x7f967401ebd0 comp rx=0 tx=0).stop 2023-10-30T13:08:55.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f969153e640 1 -- 172.21.15.125:0/2016182575 shutdown_connections 2023-10-30T13:08:55.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f969153e640 1 --2- 172.21.15.125:0/2016182575 >> v2:172.21.15.136:3300/0 conn(0x7f968c0c6f60 0x7f968c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:55.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f969153e640 1 --2- 172.21.15.125:0/2016182575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f968c0c05d0 0x7f968c13ed90 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:55.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f969153e640 1 -- 172.21.15.125:0/2016182575 >> 172.21.15.125:0/2016182575 conn(0x7f968c0c7910 msgr2=0x7f968c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:55.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f969153e640 1 -- 172.21.15.125:0/2016182575 shutdown_connections 2023-10-30T13:08:55.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.875+0000 7f969153e640 1 -- 172.21.15.125:0/2016182575 wait complete. 2023-10-30T13:08:55.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f969153e640 1 Processor -- start 2023-10-30T13:08:55.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f969153e640 1 -- start start 2023-10-30T13:08:55.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f969153e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f968c0c05d0 0x7f968c0c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:55.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f969153e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f968c0c6b80 0x7f968c13a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:55.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f969153e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f968c0c6f60 0x7f968c14f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:55.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f969153e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c159040 con 0x7f968c0c05d0 2023-10-30T13:08:55.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f969153e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f968c15e2a0 con 0x7f968c0c6b80 2023-10-30T13:08:55.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f969153e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f968c0c9240 con 0x7f968c0c6f60 2023-10-30T13:08:55.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f968affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f968c0c05d0 0x7f968c0c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:55.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f968affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f968c0c05d0 0x7f968c0c9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56056/0 (socket says 172.21.15.125:56056) 2023-10-30T13:08:55.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f968affd640 1 -- 172.21.15.125:0/1402854029 learned_addr learned my addr 172.21.15.125:0/1402854029 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:55.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f9683fff640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f968c0c6b80 0x7f968c13a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:55.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f968b7fe640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f968c0c6f60 0x7f968c14f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:55.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f968affd640 1 -- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f968c0c6f60 msgr2=0x7f968c14f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:55.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f968affd640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f968c0c6f60 0x7f968c14f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:55.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f968affd640 1 -- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f968c0c6b80 msgr2=0x7f968c13a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:55.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f968affd640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f968c0c6b80 0x7f968c13a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:55.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f968affd640 1 -- 172.21.15.125:0/1402854029 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f967400b770 con 0x7f968c0c05d0 2023-10-30T13:08:55.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f9683fff640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f968c0c6b80 0x7f968c13a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:55.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f968affd640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f968c0c05d0 0x7f968c0c9500 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f967c00c500 tx=0x7f967c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:55.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f96837fe640 1 -- 172.21.15.125:0/1402854029 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c005220 con 0x7f968c0c05d0 2023-10-30T13:08:55.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f96837fe640 1 -- 172.21.15.125:0/1402854029 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f967c005380 con 0x7f968c0c05d0 2023-10-30T13:08:55.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f969153e640 1 -- 172.21.15.125:0/1402854029 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f968c1f5c10 con 0x7f968c0c05d0 2023-10-30T13:08:55.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f96837fe640 1 -- 172.21.15.125:0/1402854029 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f967c007270 con 0x7f968c0c05d0 2023-10-30T13:08:55.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.879+0000 7f969153e640 1 -- 172.21.15.125:0/1402854029 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f968c15fbd0 con 0x7f968c0c05d0 2023-10-30T13:08:55.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.883+0000 7f969153e640 1 -- 172.21.15.125:0/1402854029 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f968c04e170 con 0x7f968c0c05d0 2023-10-30T13:08:55.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.883+0000 7f96837fe640 1 -- 172.21.15.125:0/1402854029 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f967c0073d0 con 0x7f968c0c05d0 2023-10-30T13:08:55.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.883+0000 7f96837fe640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f966c0664a0 0x7f966c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:55.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.883+0000 7f96837fe640 1 -- 172.21.15.125:0/1402854029 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7f967c083ce0 con 0x7f968c0c05d0 2023-10-30T13:08:55.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.883+0000 7f9683fff640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f966c0664a0 0x7f966c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:55.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.883+0000 7f9683fff640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f966c0664a0 0x7f966c068960 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f96740b1000 tx=0x7f9674008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:55.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:55.887+0000 7f96837fe640 1 -- 172.21.15.125:0/1402854029 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f967c0508f0 con 0x7f968c0c05d0 2023-10-30T13:08:56.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.071+0000 7f969153e640 1 -- 172.21.15.125:0/1402854029 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f968c160440 con 0x7f968c0c05d0 2023-10-30T13:08:56.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.071+0000 7f96837fe640 1 -- 172.21.15.125:0/1402854029 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f967c053f10 con 0x7f968c0c05d0 2023-10-30T13:08:56.073 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:56.074 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:55.644061+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:56.074 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 187 2023-10-30T13:08:56.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.071+0000 7f969153e640 1 -- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f966c0664a0 msgr2=0x7f966c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:56.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.071+0000 7f969153e640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f966c0664a0 0x7f966c068960 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f96740b1000 tx=0x7f9674008040 comp rx=0 tx=0).stop 2023-10-30T13:08:56.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.071+0000 7f969153e640 1 -- 172.21.15.125:0/1402854029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f968c0c05d0 msgr2=0x7f968c0c9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:56.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.071+0000 7f969153e640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f968c0c05d0 0x7f968c0c9500 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f967c00c500 tx=0x7f967c00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.071+0000 7f969153e640 1 -- 172.21.15.125:0/1402854029 shutdown_connections 2023-10-30T13:08:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.071+0000 7f969153e640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f966c0664a0 0x7f966c068960 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.071+0000 7f969153e640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f968c0c6f60 0x7f968c14f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.071+0000 7f969153e640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f968c0c6b80 0x7f968c13a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.071+0000 7f969153e640 1 --2- 172.21.15.125:0/1402854029 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f968c0c05d0 0x7f968c0c9500 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.071+0000 7f969153e640 1 -- 172.21.15.125:0/1402854029 >> 172.21.15.125:0/1402854029 conn(0x7f968c0c7910 msgr2=0x7f968c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.071+0000 7f969153e640 1 -- 172.21.15.125:0/1402854029 shutdown_connections 2023-10-30T13:08:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.071+0000 7f969153e640 1 -- 172.21.15.125:0/1402854029 wait complete. 2023-10-30T13:08:56.094 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T13:08:56.094 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:08:56.094 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:56.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.227+0000 7fba8ea38640 1 Processor -- start 2023-10-30T13:08:56.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8ea38640 1 -- start start 2023-10-30T13:08:56.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8ea38640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fba8815cd50 0x7fba880c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:56.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8ea38640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fba880cefa0 0x7fba880cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:56.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8ea38640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8815a6c0 con 0x7fba881538f0 2023-10-30T13:08:56.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8ea38640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba880cf8f0 con 0x7fba880cefa0 2023-10-30T13:08:56.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8ea38640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8815d300 con 0x7fba8815cd50 2023-10-30T13:08:56.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8da36640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fba881538f0 0x7fba88159fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:33620/0 (socket says 172.21.15.125:33620) 2023-10-30T13:08:56.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8e237640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fba880cefa0 0x7fba880cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:56.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8da36640 1 -- 172.21.15.125:0/2078469040 learned_addr learned my addr 172.21.15.125:0/2078469040 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:08:56.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8d235640 1 --2- 172.21.15.125:0/2078469040 >> v2:172.21.15.136:3300/0 conn(0x7fba8815cd50 0x7fba880c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:56.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8e237640 1 -- 172.21.15.125:0/2078469040 >> v2:172.21.15.136:3300/0 conn(0x7fba8815cd50 msgr2=0x7fba880c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:56.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8e237640 1 --2- 172.21.15.125:0/2078469040 >> v2:172.21.15.136:3300/0 conn(0x7fba8815cd50 0x7fba880c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8e237640 1 -- 172.21.15.125:0/2078469040 >> v1:172.21.15.125:6789/0 conn(0x7fba881538f0 legacy=0x7fba88159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:56.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8e237640 1 -- 172.21.15.125:0/2078469040 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8815d5e0 con 0x7fba880cefa0 2023-10-30T13:08:56.237 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.231+0000 7fba8ca34640 1 -- 172.21.15.125:0/2078469040 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804953574 0 0) 0x7fba8815a6c0 con 0x7fba881538f0 2023-10-30T13:08:56.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.235+0000 7fba8e237640 1 --2- 172.21.15.125:0/2078469040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba880cefa0 0x7fba880cf3b0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fba88072e90 tx=0x7fba8001f0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a74fe8f84417c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:56.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.235+0000 7fba8ca34640 1 -- 172.21.15.125:0/2078469040 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba800b0440 con 0x7fba880cefa0 2023-10-30T13:08:56.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.235+0000 7fba8ca34640 1 -- 172.21.15.125:0/2078469040 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba800b05a0 con 0x7fba880cefa0 2023-10-30T13:08:56.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.235+0000 7fba8ca34640 1 -- 172.21.15.125:0/2078469040 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba800b0930 con 0x7fba880cefa0 2023-10-30T13:08:56.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.235+0000 7fba8ea38640 1 -- 172.21.15.125:0/2078469040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba880cefa0 msgr2=0x7fba880cf3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:56.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.235+0000 7fba8ea38640 1 --2- 172.21.15.125:0/2078469040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba880cefa0 0x7fba880cf3b0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7fba88072e90 tx=0x7fba8001f0f0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.235+0000 7fba8ea38640 1 -- 172.21.15.125:0/2078469040 shutdown_connections 2023-10-30T13:08:56.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.235+0000 7fba8ea38640 1 --2- 172.21.15.125:0/2078469040 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba880cefa0 0x7fba880cf3b0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.235+0000 7fba8ea38640 1 --2- 172.21.15.125:0/2078469040 >> v2:172.21.15.136:3300/0 conn(0x7fba8815cd50 0x7fba880c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.235+0000 7fba8ea38640 1 -- 172.21.15.125:0/2078469040 >> 172.21.15.125:0/2078469040 conn(0x7fba8806a640 msgr2=0x7fba88065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:56.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.235+0000 7fba8ea38640 1 -- 172.21.15.125:0/2078469040 shutdown_connections 2023-10-30T13:08:56.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.235+0000 7fba8ea38640 1 -- 172.21.15.125:0/2078469040 wait complete. 2023-10-30T13:08:56.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.235+0000 7fba8ea38640 1 Processor -- start 2023-10-30T13:08:56.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.235+0000 7fba8ea38640 1 -- start start 2023-10-30T13:08:56.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8ea38640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba880cefa0 0x7fba881ef6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:56.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8ea38640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba881538f0 0x7fba8813f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:56.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8ea38640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba8815cd50 0x7fba8813fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:56.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8ea38640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba881efc00 con 0x7fba8815cd50 2023-10-30T13:08:56.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8ea38640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba881573a0 con 0x7fba881538f0 2023-10-30T13:08:56.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8ea38640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba88157f20 con 0x7fba880cefa0 2023-10-30T13:08:56.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8e237640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba8815cd50 0x7fba8813fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:56.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8e237640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba8815cd50 0x7fba8813fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56074/0 (socket says 172.21.15.125:56074) 2023-10-30T13:08:56.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8e237640 1 -- 172.21.15.125:0/2331196984 learned_addr learned my addr 172.21.15.125:0/2331196984 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:56.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8da36640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba880cefa0 0x7fba881ef6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:56.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8d235640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba881538f0 0x7fba8813f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:56.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8e237640 1 -- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba880cefa0 msgr2=0x7fba881ef6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:56.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8e237640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba880cefa0 0x7fba881ef6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8e237640 1 -- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba881538f0 msgr2=0x7fba8813f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:56.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8e237640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba881538f0 0x7fba8813f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8e237640 1 -- 172.21.15.125:0/2331196984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba80009a70 con 0x7fba8815cd50 2023-10-30T13:08:56.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8da36640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba880cefa0 0x7fba881ef6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:56.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8d235640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba881538f0 0x7fba8813f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:56.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8e237640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba8815cd50 0x7fba8813fdf0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fba8001bd00 tx=0x7fba8001be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:56.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba7e7fc640 1 -- 172.21.15.125:0/2331196984 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba800c2070 con 0x7fba8815cd50 2023-10-30T13:08:56.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba7e7fc640 1 -- 172.21.15.125:0/2331196984 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba80009450 con 0x7fba8815cd50 2023-10-30T13:08:56.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba7e7fc640 1 -- 172.21.15.125:0/2331196984 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba800b0440 con 0x7fba8815cd50 2023-10-30T13:08:56.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8ea38640 1 -- 172.21.15.125:0/2331196984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba881596d0 con 0x7fba8815cd50 2023-10-30T13:08:56.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8ea38640 1 -- 172.21.15.125:0/2331196984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba881578d0 con 0x7fba8815cd50 2023-10-30T13:08:56.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.239+0000 7fba8ea38640 1 -- 172.21.15.125:0/2331196984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba881580d0 con 0x7fba8815cd50 2023-10-30T13:08:56.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.243+0000 7fba7e7fc640 1 -- 172.21.15.125:0/2331196984 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fba800b7050 con 0x7fba8815cd50 2023-10-30T13:08:56.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.243+0000 7fba7e7fc640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fba6c0664d0 0x7fba6c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:56.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.243+0000 7fba7e7fc640 1 -- 172.21.15.125:0/2331196984 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7fba801226c0 con 0x7fba8815cd50 2023-10-30T13:08:56.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.243+0000 7fba8da36640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fba6c0664d0 0x7fba6c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:56.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.243+0000 7fba8da36640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fba6c0664d0 0x7fba6c068990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fba84007fd0 tx=0x7fba84007c20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:56.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.247+0000 7fba7e7fc640 1 -- 172.21.15.125:0/2331196984 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fba881580d0 con 0x7fba8815cd50 2023-10-30T13:08:56.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.431+0000 7fba8ea38640 1 -- 172.21.15.125:0/2331196984 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba881580d0 con 0x7fba8815cd50 2023-10-30T13:08:56.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.431+0000 7fba7e7fc640 1 -- 172.21.15.125:0/2331196984 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+5756 (secure 0 0 0) 0x7fba881580d0 con 0x7fba8815cd50 2023-10-30T13:08:56.433 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:56.434 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:55.644061+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:56.434 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 187 2023-10-30T13:08:56.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.431+0000 7fba8ea38640 1 -- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fba6c0664d0 msgr2=0x7fba6c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:56.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.431+0000 7fba8ea38640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fba6c0664d0 0x7fba6c068990 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fba84007fd0 tx=0x7fba84007c20 comp rx=0 tx=0).stop 2023-10-30T13:08:56.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.431+0000 7fba8ea38640 1 -- 172.21.15.125:0/2331196984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba8815cd50 msgr2=0x7fba8813fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:56.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.431+0000 7fba8ea38640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba8815cd50 0x7fba8813fdf0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7fba8001bd00 tx=0x7fba8001be90 comp rx=0 tx=0).stop 2023-10-30T13:08:56.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.431+0000 7fba8ea38640 1 -- 172.21.15.125:0/2331196984 shutdown_connections 2023-10-30T13:08:56.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.431+0000 7fba8ea38640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fba6c0664d0 0x7fba6c068990 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.431+0000 7fba8ea38640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fba8815cd50 0x7fba8813fdf0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.431+0000 7fba8ea38640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fba881538f0 0x7fba8813f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.431+0000 7fba8ea38640 1 --2- 172.21.15.125:0/2331196984 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fba880cefa0 0x7fba881ef6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:56.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.431+0000 7fba8ea38640 1 -- 172.21.15.125:0/2331196984 >> 172.21.15.125:0/2331196984 conn(0x7fba8806a640 msgr2=0x7fba880caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:56.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.431+0000 7fba8ea38640 1 -- 172.21.15.125:0/2331196984 shutdown_connections 2023-10-30T13:08:56.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:56.431+0000 7fba8ea38640 1 -- 172.21.15.125:0/2331196984 wait complete. 2023-10-30T13:08:56.455 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 187, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 187, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_7340": { "addr": "172.21.15.136:6835/3526499585", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3526499585, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3526499585, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7340, "incarnation": 187, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_8301": { "addr": "172.21.15.125:6837/2540063352", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2540063352, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2540063352, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8301, "incarnation": 168, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_8529": { "addr": "172.21.15.125:6836/3950044434", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3950044434, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3950044434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 8529, "incarnation": 179, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 77, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:08:55.644061+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8529, "mds_1": 8301, "mds_2": 7340 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:08:58.456 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:08:58.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe88dc640 1 Processor -- start 2023-10-30T13:08:58.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe88dc640 1 -- start start 2023-10-30T13:08:58.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe88dc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbfe41538f0 0x7fbfe4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:58.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe88dc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbfe415cd50 0x7fbfe40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:58.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe88dc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbfe40cefa0 0x7fbfe40cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:58.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe88dc640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe415d300 con 0x7fbfe415cd50 2023-10-30T13:08:58.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe88dc640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe415d5e0 con 0x7fbfe40cefa0 2023-10-30T13:08:58.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe88dc640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe415d8c0 con 0x7fbfe41538f0 2023-10-30T13:08:58.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe2ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbfe415cd50 0x7fbfe40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:58.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe2ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbfe415cd50 0x7fbfe40c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56076/0 (socket says 172.21.15.125:56076) 2023-10-30T13:08:58.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe2ffd640 1 -- 172.21.15.125:0/139852416 learned_addr learned my addr 172.21.15.125:0/139852416 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:58.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe37fe640 1 --2- 172.21.15.125:0/139852416 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbfe41538f0 0x7fbfe4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:58.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe3fff640 1 --2- 172.21.15.125:0/139852416 >> v2:172.21.15.136:3300/0 conn(0x7fbfe40cefa0 0x7fbfe40cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:58.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe2ffd640 1 -- 172.21.15.125:0/139852416 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbfe41538f0 msgr2=0x7fbfe4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:58.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe2ffd640 1 --2- 172.21.15.125:0/139852416 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbfe41538f0 0x7fbfe4159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:58.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe2ffd640 1 -- 172.21.15.125:0/139852416 >> v2:172.21.15.136:3300/0 conn(0x7fbfe40cefa0 msgr2=0x7fbfe40cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:08:58.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe2ffd640 1 --2- 172.21.15.125:0/139852416 >> v2:172.21.15.136:3300/0 conn(0x7fbfe40cefa0 0x7fbfe40cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:58.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe2ffd640 1 -- 172.21.15.125:0/139852416 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe415dba0 con 0x7fbfe415cd50 2023-10-30T13:08:58.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.603+0000 7fbfe3fff640 1 --2- 172.21.15.125:0/139852416 >> v2:172.21.15.136:3300/0 conn(0x7fbfe40cefa0 0x7fbfe40cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:08:58.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe2ffd640 1 --2- 172.21.15.125:0/139852416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfe415cd50 0x7fbfe40c9f90 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40b1ae0 tx=0x7fbfd801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e080531c8fb7d81 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:58.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe27fc640 1 -- 172.21.15.125:0/139852416 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd80a89b0 con 0x7fbfe415cd50 2023-10-30T13:08:58.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe27fc640 1 -- 172.21.15.125:0/139852416 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd80a8b10 con 0x7fbfe415cd50 2023-10-30T13:08:58.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe27fc640 1 -- 172.21.15.125:0/139852416 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd80a8ea0 con 0x7fbfe415cd50 2023-10-30T13:08:58.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe88dc640 1 -- 172.21.15.125:0/139852416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfe415cd50 msgr2=0x7fbfe40c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:58.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe88dc640 1 --2- 172.21.15.125:0/139852416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfe415cd50 0x7fbfe40c9f90 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7fbfe40b1ae0 tx=0x7fbfd801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:08:58.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe88dc640 1 -- 172.21.15.125:0/139852416 shutdown_connections 2023-10-30T13:08:58.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe88dc640 1 --2- 172.21.15.125:0/139852416 >> v2:172.21.15.136:3300/0 conn(0x7fbfe40cefa0 0x7fbfe40cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:58.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe88dc640 1 --2- 172.21.15.125:0/139852416 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfe415cd50 0x7fbfe40c9f90 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:58.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe88dc640 1 --2- 172.21.15.125:0/139852416 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbfe41538f0 0x7fbfe4159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:58.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe88dc640 1 -- 172.21.15.125:0/139852416 >> 172.21.15.125:0/139852416 conn(0x7fbfe406a640 msgr2=0x7fbfe4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:58.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe88dc640 1 -- 172.21.15.125:0/139852416 shutdown_connections 2023-10-30T13:08:58.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe88dc640 1 -- 172.21.15.125:0/139852416 wait complete. 2023-10-30T13:08:58.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe88dc640 1 Processor -- start 2023-10-30T13:08:58.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.607+0000 7fbfe88dc640 1 -- start start 2023-10-30T13:08:58.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe88dc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfe40cefa0 0x7fbfe4148100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:58.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe88dc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfe41538f0 0x7fbfe40cdf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:58.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe88dc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbfe415cd50 0x7fbfe40ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:58.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe88dc640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4147520 con 0x7fbfe41538f0 2023-10-30T13:08:58.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe88dc640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4146950 con 0x7fbfe40cefa0 2023-10-30T13:08:58.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe88dc640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe4146bf0 con 0x7fbfe415cd50 2023-10-30T13:08:58.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe2ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfe41538f0 0x7fbfe40cdf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:58.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe2ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfe41538f0 0x7fbfe40cdf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56080/0 (socket says 172.21.15.125:56080) 2023-10-30T13:08:58.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe2ffd640 1 -- 172.21.15.125:0/2733659306 learned_addr learned my addr 172.21.15.125:0/2733659306 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:08:58.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe3fff640 1 --2- 172.21.15.125:0/2733659306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbfe415cd50 0x7fbfe40ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:58.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe37fe640 1 --2- 172.21.15.125:0/2733659306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfe40cefa0 0x7fbfe4148100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:58.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe2ffd640 1 -- 172.21.15.125:0/2733659306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbfe415cd50 msgr2=0x7fbfe40ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:58.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe2ffd640 1 --2- 172.21.15.125:0/2733659306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbfe415cd50 0x7fbfe40ce4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:58.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe2ffd640 1 -- 172.21.15.125:0/2733659306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfe40cefa0 msgr2=0x7fbfe4148100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:58.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe2ffd640 1 --2- 172.21.15.125:0/2733659306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfe40cefa0 0x7fbfe4148100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:58.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe2ffd640 1 -- 172.21.15.125:0/2733659306 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd80094d0 con 0x7fbfe41538f0 2023-10-30T13:08:58.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe2ffd640 1 --2- 172.21.15.125:0/2733659306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfe41538f0 0x7fbfe40cdf60 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80b3040 tx=0x7fbfd801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:58.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfcbfff640 1 -- 172.21.15.125:0/2733659306 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd80b9620 con 0x7fbfe41538f0 2023-10-30T13:08:58.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfcbfff640 1 -- 172.21.15.125:0/2733659306 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd80b9780 con 0x7fbfe41538f0 2023-10-30T13:08:58.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe88dc640 1 -- 172.21.15.125:0/2733659306 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe4146ed0 con 0x7fbfe41538f0 2023-10-30T13:08:58.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfcbfff640 1 -- 172.21.15.125:0/2733659306 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfd80b9b30 con 0x7fbfe41538f0 2023-10-30T13:08:58.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe88dc640 1 -- 172.21.15.125:0/2733659306 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe415c6b0 con 0x7fbfe41538f0 2023-10-30T13:08:58.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.611+0000 7fbfe88dc640 1 -- 172.21.15.125:0/2733659306 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe404e170 con 0x7fbfe41538f0 2023-10-30T13:08:58.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.615+0000 7fbfcbfff640 1 -- 172.21.15.125:0/2733659306 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbfd80b7070 con 0x7fbfe41538f0 2023-10-30T13:08:58.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.615+0000 7fbfcbfff640 1 --2- 172.21.15.125:0/2733659306 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbfd00664a0 0x7fbfd0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:08:58.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.615+0000 7fbfcbfff640 1 -- 172.21.15.125:0/2733659306 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7fbfd8122570 con 0x7fbfe41538f0 2023-10-30T13:08:58.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.615+0000 7fbfe37fe640 1 --2- 172.21.15.125:0/2733659306 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbfd00664a0 0x7fbfd0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:08:58.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.615+0000 7fbfe37fe640 1 --2- 172.21.15.125:0/2733659306 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbfd00664a0 0x7fbfd0068960 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009af0 tx=0x7fbfd4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:08:58.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.619+0000 7fbfcbfff640 1 -- 172.21.15.125:0/2733659306 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbfd80ef200 con 0x7fbfe41538f0 2023-10-30T13:08:58.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.803+0000 7fbfe88dc640 1 -- 172.21.15.125:0/2733659306 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbfe4140730 con 0x7fbfe41538f0 2023-10-30T13:08:58.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.803+0000 7fbfcbfff640 1 -- 172.21.15.125:0/2733659306 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+5756 (secure 0 0 0) 0x7fbfd80f2820 con 0x7fbfe41538f0 2023-10-30T13:08:58.806 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:08:58.807 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:55.644061+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:08:58.807 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 187 2023-10-30T13:08:58.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.807+0000 7fbfe88dc640 1 -- 172.21.15.125:0/2733659306 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbfd00664a0 msgr2=0x7fbfd0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:58.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.807+0000 7fbfe88dc640 1 --2- 172.21.15.125:0/2733659306 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbfd00664a0 0x7fbfd0068960 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009af0 tx=0x7fbfd4008040 comp rx=0 tx=0).stop 2023-10-30T13:08:58.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.807+0000 7fbfe88dc640 1 -- 172.21.15.125:0/2733659306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfe41538f0 msgr2=0x7fbfe40cdf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:08:58.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.807+0000 7fbfe88dc640 1 --2- 172.21.15.125:0/2733659306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfe41538f0 0x7fbfe40cdf60 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80b3040 tx=0x7fbfd801ba90 comp rx=0 tx=0).stop 2023-10-30T13:08:58.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.807+0000 7fbfe88dc640 1 -- 172.21.15.125:0/2733659306 shutdown_connections 2023-10-30T13:08:58.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.807+0000 7fbfe88dc640 1 --2- 172.21.15.125:0/2733659306 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbfd00664a0 0x7fbfd0068960 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:58.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.807+0000 7fbfe88dc640 1 --2- 172.21.15.125:0/2733659306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbfe415cd50 0x7fbfe40ce4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:58.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.807+0000 7fbfe88dc640 1 --2- 172.21.15.125:0/2733659306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbfe41538f0 0x7fbfe40cdf60 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:58.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.807+0000 7fbfe88dc640 1 --2- 172.21.15.125:0/2733659306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbfe40cefa0 0x7fbfe4148100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:08:58.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.807+0000 7fbfe88dc640 1 -- 172.21.15.125:0/2733659306 >> 172.21.15.125:0/2733659306 conn(0x7fbfe406a640 msgr2=0x7fbfe406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:08:58.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.807+0000 7fbfe88dc640 1 -- 172.21.15.125:0/2733659306 shutdown_connections 2023-10-30T13:08:58.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:08:58.807+0000 7fbfe88dc640 1 -- 172.21.15.125:0/2733659306 wait complete. 2023-10-30T13:08:58.827 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:09:00.828 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:09:00.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.967+0000 7f7d38454640 1 Processor -- start 2023-10-30T13:09:00.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.967+0000 7f7d38454640 1 -- start start 2023-10-30T13:09:00.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d38454640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300c6b80 0x7f7d300bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:00.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d38454640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7d300c6f60 0x7f7d300c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:00.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d38454640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7d300c8430 0x7f7d3013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:00.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d38454640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d300c0a50 con 0x7f7d300c6f60 2023-10-30T13:09:00.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d38454640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d3006f6e0 con 0x7f7d300c8430 2023-10-30T13:09:00.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d38454640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3006f9c0 con 0x7f7d300c6b80 2023-10-30T13:09:00.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d359ef640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7d300c6f60 0x7f7d300c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:00.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d359ef640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7d300c6f60 0x7f7d300c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56250/0 (socket says 172.21.15.125:56250) 2023-10-30T13:09:00.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d359ef640 1 -- 172.21.15.125:0/2162567506 learned_addr learned my addr 172.21.15.125:0/2162567506 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:00.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d361f0640 1 --2- 172.21.15.125:0/2162567506 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300c6b80 0x7f7d300bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:00.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d369f1640 1 --2- 172.21.15.125:0/2162567506 >> v2:172.21.15.136:3300/0 conn(0x7f7d300c8430 0x7f7d3013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:00.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d359ef640 1 -- 172.21.15.125:0/2162567506 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300c6b80 msgr2=0x7f7d300bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:09:00.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d359ef640 1 --2- 172.21.15.125:0/2162567506 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300c6b80 0x7f7d300bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:00.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d359ef640 1 -- 172.21.15.125:0/2162567506 >> v2:172.21.15.136:3300/0 conn(0x7f7d300c8430 msgr2=0x7f7d3013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:09:00.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d359ef640 1 --2- 172.21.15.125:0/2162567506 >> v2:172.21.15.136:3300/0 conn(0x7f7d300c8430 0x7f7d3013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:00.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d359ef640 1 -- 172.21.15.125:0/2162567506 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3006fca0 con 0x7f7d300c6f60 2023-10-30T13:09:00.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.971+0000 7f7d369f1640 1 --2- 172.21.15.125:0/2162567506 >> v2:172.21.15.136:3300/0 conn(0x7f7d300c8430 0x7f7d3013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:09:00.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d359ef640 1 --2- 172.21.15.125:0/2162567506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d300c6f60 0x7f7d300c0400 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f7d300b0860 tx=0x7f7d2001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c98df2dbe9896 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:00.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d351ee640 1 -- 172.21.15.125:0/2162567506 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d200ac630 con 0x7f7d300c6f60 2023-10-30T13:09:00.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d351ee640 1 -- 172.21.15.125:0/2162567506 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d200ac790 con 0x7f7d300c6f60 2023-10-30T13:09:00.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d351ee640 1 -- 172.21.15.125:0/2162567506 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d200acb20 con 0x7f7d300c6f60 2023-10-30T13:09:00.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 -- 172.21.15.125:0/2162567506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d300c6f60 msgr2=0x7f7d300c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:00.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 --2- 172.21.15.125:0/2162567506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d300c6f60 0x7f7d300c0400 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f7d300b0860 tx=0x7f7d2001c500 comp rx=0 tx=0).stop 2023-10-30T13:09:00.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 -- 172.21.15.125:0/2162567506 shutdown_connections 2023-10-30T13:09:00.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 --2- 172.21.15.125:0/2162567506 >> v2:172.21.15.136:3300/0 conn(0x7f7d300c8430 0x7f7d3013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:00.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 --2- 172.21.15.125:0/2162567506 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d300c6f60 0x7f7d300c0400 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:00.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 --2- 172.21.15.125:0/2162567506 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300c6b80 0x7f7d300bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:00.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 -- 172.21.15.125:0/2162567506 >> 172.21.15.125:0/2162567506 conn(0x7f7d300c7910 msgr2=0x7f7d300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:00.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 -- 172.21.15.125:0/2162567506 shutdown_connections 2023-10-30T13:09:00.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 -- 172.21.15.125:0/2162567506 wait complete. 2023-10-30T13:09:00.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 Processor -- start 2023-10-30T13:09:00.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 -- start start 2023-10-30T13:09:00.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300c6b80 0x7f7d30155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:00.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d300c6f60 0x7f7d300c9600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:00.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d300c8430 0x7f7d300c9b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:00.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3015fed0 con 0x7f7d300c8430 2023-10-30T13:09:00.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3013d080 con 0x7f7d300c6f60 2023-10-30T13:09:00.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.975+0000 7f7d38454640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3013d850 con 0x7f7d300c6b80 2023-10-30T13:09:00.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d369f1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d300c8430 0x7f7d300c9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:00.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d369f1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d300c8430 0x7f7d300c9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56256/0 (socket says 172.21.15.125:56256) 2023-10-30T13:09:00.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d369f1640 1 -- 172.21.15.125:0/61214181 learned_addr learned my addr 172.21.15.125:0/61214181 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:00.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d361f0640 1 --2- 172.21.15.125:0/61214181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300c6b80 0x7f7d30155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:00.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d359ef640 1 --2- 172.21.15.125:0/61214181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d300c6f60 0x7f7d300c9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:00.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d369f1640 1 -- 172.21.15.125:0/61214181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300c6b80 msgr2=0x7f7d30155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:00.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d369f1640 1 --2- 172.21.15.125:0/61214181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300c6b80 0x7f7d30155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:00.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d369f1640 1 -- 172.21.15.125:0/61214181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d300c6f60 msgr2=0x7f7d300c9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:00.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d369f1640 1 --2- 172.21.15.125:0/61214181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d300c6f60 0x7f7d300c9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:00.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d369f1640 1 -- 172.21.15.125:0/61214181 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d200094d0 con 0x7f7d300c8430 2023-10-30T13:09:00.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d369f1640 1 --2- 172.21.15.125:0/61214181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d300c8430 0x7f7d300c9b40 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f7d28007df0 tx=0x7f7d2800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:00.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d26ffd640 1 -- 172.21.15.125:0/61214181 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d280039a0 con 0x7f7d300c8430 2023-10-30T13:09:00.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d26ffd640 1 -- 172.21.15.125:0/61214181 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2800f040 con 0x7f7d300c8430 2023-10-30T13:09:00.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d26ffd640 1 -- 172.21.15.125:0/61214181 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d28003d40 con 0x7f7d300c8430 2023-10-30T13:09:00.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d38454640 1 -- 172.21.15.125:0/61214181 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3013db30 con 0x7f7d300c8430 2023-10-30T13:09:00.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d38454640 1 -- 172.21.15.125:0/61214181 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3013e0f0 con 0x7f7d300c8430 2023-10-30T13:09:00.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d38454640 1 -- 172.21.15.125:0/61214181 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3004e170 con 0x7f7d300c8430 2023-10-30T13:09:00.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d26ffd640 1 -- 172.21.15.125:0/61214181 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7d28004030 con 0x7f7d300c8430 2023-10-30T13:09:00.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.979+0000 7f7d26ffd640 1 --2- 172.21.15.125:0/61214181 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d140664d0 0x7f7d14068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:00.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.983+0000 7f7d26ffd640 1 -- 172.21.15.125:0/61214181 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7f7d28084870 con 0x7f7d300c8430 2023-10-30T13:09:00.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.983+0000 7f7d361f0640 1 --2- 172.21.15.125:0/61214181 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d140664d0 0x7f7d14068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:00.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.983+0000 7f7d361f0640 1 --2- 172.21.15.125:0/61214181 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d140664d0 0x7f7d14068990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009ae0 tx=0x7f7d2c009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:00.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:00.987+0000 7f7d26ffd640 1 -- 172.21.15.125:0/61214181 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7d280515c0 con 0x7f7d300c8430 2023-10-30T13:09:01.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:01.171+0000 7f7d38454640 1 -- 172.21.15.125:0/61214181 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d3006c5c0 con 0x7f7d300c8430 2023-10-30T13:09:01.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:01.171+0000 7f7d26ffd640 1 -- 172.21.15.125:0/61214181 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f7d28054be0 con 0x7f7d300c8430 2023-10-30T13:09:01.173 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:09:01.173 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":187,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:08:55.644061+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:09:01.174 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 187 2023-10-30T13:09:01.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:01.171+0000 7f7d38454640 1 -- 172.21.15.125:0/61214181 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d140664d0 msgr2=0x7f7d14068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:01.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:01.171+0000 7f7d38454640 1 --2- 172.21.15.125:0/61214181 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d140664d0 0x7f7d14068990 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c009ae0 tx=0x7f7d2c009190 comp rx=0 tx=0).stop 2023-10-30T13:09:01.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:01.171+0000 7f7d38454640 1 -- 172.21.15.125:0/61214181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d300c8430 msgr2=0x7f7d300c9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:01.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:01.171+0000 7f7d38454640 1 --2- 172.21.15.125:0/61214181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d300c8430 0x7f7d300c9b40 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f7d28007df0 tx=0x7f7d2800d4a0 comp rx=0 tx=0).stop 2023-10-30T13:09:01.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:01.171+0000 7f7d38454640 1 -- 172.21.15.125:0/61214181 shutdown_connections 2023-10-30T13:09:01.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:01.171+0000 7f7d38454640 1 --2- 172.21.15.125:0/61214181 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7d140664d0 0x7f7d14068990 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:01.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:01.171+0000 7f7d38454640 1 --2- 172.21.15.125:0/61214181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7d300c8430 0x7f7d300c9b40 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:01.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:01.171+0000 7f7d38454640 1 --2- 172.21.15.125:0/61214181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7d300c6f60 0x7f7d300c9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:01.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:01.171+0000 7f7d38454640 1 --2- 172.21.15.125:0/61214181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7d300c6b80 0x7f7d30155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:01.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:01.171+0000 7f7d38454640 1 -- 172.21.15.125:0/61214181 >> 172.21.15.125:0/61214181 conn(0x7f7d300c7910 msgr2=0x7f7d3013a200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:01.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:01.171+0000 7f7d38454640 1 -- 172.21.15.125:0/61214181 shutdown_connections 2023-10-30T13:09:01.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:01.171+0000 7f7d38454640 1 -- 172.21.15.125:0/61214181 wait complete. 2023-10-30T13:09:01.194 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:09:03.195 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:09:03.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.339+0000 7fb91abab640 1 Processor -- start 2023-10-30T13:09:03.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.339+0000 7fb91abab640 1 -- start start 2023-10-30T13:09:03.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.339+0000 7fb91abab640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9140c6b80 0x7fb9140bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:03.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.339+0000 7fb91abab640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb9140c6f60 0x7fb9140c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:03.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.339+0000 7fb91abab640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9140c0a50 con 0x7fb9140c8430 2023-10-30T13:09:03.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.339+0000 7fb91abab640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb91406f6e0 con 0x7fb9140c6f60 2023-10-30T13:09:03.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.339+0000 7fb91abab640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb91406f9c0 con 0x7fb9140c6b80 2023-10-30T13:09:03.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.339+0000 7fb918947640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9140c6b80 0x7fb9140bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:03.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.339+0000 7fb918947640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9140c6b80 0x7fb9140bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:53806/0 (socket says 172.21.15.125:53806) 2023-10-30T13:09:03.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.339+0000 7fb918947640 1 -- 172.21.15.125:0/2993830540 learned_addr learned my addr 172.21.15.125:0/2993830540 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:03.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.339+0000 7fb90bfff640 1 --2- 172.21.15.125:0/2993830540 >> v2:172.21.15.136:3300/0 conn(0x7fb9140c6f60 0x7fb9140c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:03.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.343+0000 7fb918947640 1 -- 172.21.15.125:0/2993830540 >> v2:172.21.15.136:3300/0 conn(0x7fb9140c6f60 msgr2=0x7fb9140c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:09:03.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.343+0000 7fb918947640 1 --2- 172.21.15.125:0/2993830540 >> v2:172.21.15.136:3300/0 conn(0x7fb9140c6f60 0x7fb9140c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:03.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.343+0000 7fb918947640 1 -- 172.21.15.125:0/2993830540 >> v1:172.21.15.136:6789/0 conn(0x7fb9140c8430 legacy=0x7fb91413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:03.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.343+0000 7fb918947640 1 -- 172.21.15.125:0/2993830540 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91406fca0 con 0x7fb9140c6b80 2023-10-30T13:09:03.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.343+0000 7fb90bfff640 1 --2- 172.21.15.125:0/2993830540 >> v2:172.21.15.136:3300/0 conn(0x7fb9140c6f60 0x7fb9140c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:09:03.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.343+0000 7fb918947640 1 --2- 172.21.15.125:0/2993830540 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9140c6b80 0x7fb9140bfec0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb910009870 tx=0x7fb91001c540 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b74e1fc1f6fa16f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:03.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.343+0000 7fb90b7fe640 1 -- 172.21.15.125:0/2993830540 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9100ab650 con 0x7fb9140c6b80 2023-10-30T13:09:03.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.343+0000 7fb90b7fe640 1 -- 172.21.15.125:0/2993830540 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9100ab7b0 con 0x7fb9140c6b80 2023-10-30T13:09:03.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.343+0000 7fb90b7fe640 1 -- 172.21.15.125:0/2993830540 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9100abb40 con 0x7fb9140c6b80 2023-10-30T13:09:03.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 -- 172.21.15.125:0/2993830540 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9140c6b80 msgr2=0x7fb9140bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:03.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 --2- 172.21.15.125:0/2993830540 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9140c6b80 0x7fb9140bfec0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb910009870 tx=0x7fb91001c540 comp rx=0 tx=0).stop 2023-10-30T13:09:03.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 -- 172.21.15.125:0/2993830540 shutdown_connections 2023-10-30T13:09:03.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 --2- 172.21.15.125:0/2993830540 >> v2:172.21.15.136:3300/0 conn(0x7fb9140c6f60 0x7fb9140c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:03.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 --2- 172.21.15.125:0/2993830540 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9140c6b80 0x7fb9140bfec0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:03.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 -- 172.21.15.125:0/2993830540 >> 172.21.15.125:0/2993830540 conn(0x7fb9140c7910 msgr2=0x7fb9140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:03.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 -- 172.21.15.125:0/2993830540 shutdown_connections 2023-10-30T13:09:03.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 -- 172.21.15.125:0/2993830540 wait complete. 2023-10-30T13:09:03.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 Processor -- start 2023-10-30T13:09:03.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 -- start start 2023-10-30T13:09:03.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9140c6b80 0x7fb914138a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:03.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9140c6f60 0x7fb91415f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:03.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9140c8430 0x7fb91415fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:03.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9141f2a40 con 0x7fb9140c6f60 2023-10-30T13:09:03.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb914138f80 con 0x7fb9140c8430 2023-10-30T13:09:03.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb91abab640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9141f1b50 con 0x7fb9140c6b80 2023-10-30T13:09:03.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb90bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9140c6f60 0x7fb91415f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:03.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb90bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9140c6f60 0x7fb91415f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56258/0 (socket says 172.21.15.125:56258) 2023-10-30T13:09:03.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb90bfff640 1 -- 172.21.15.125:0/744343867 learned_addr learned my addr 172.21.15.125:0/744343867 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:03.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb918947640 1 --2- 172.21.15.125:0/744343867 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9140c6b80 0x7fb914138a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:03.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.347+0000 7fb919148640 1 --2- 172.21.15.125:0/744343867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9140c8430 0x7fb91415fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:03.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb90bfff640 1 -- 172.21.15.125:0/744343867 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9140c6b80 msgr2=0x7fb914138a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:03.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb90bfff640 1 --2- 172.21.15.125:0/744343867 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9140c6b80 0x7fb914138a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb90bfff640 1 -- 172.21.15.125:0/744343867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9140c8430 msgr2=0x7fb91415fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb90bfff640 1 --2- 172.21.15.125:0/744343867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9140c8430 0x7fb91415fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb90bfff640 1 -- 172.21.15.125:0/744343867 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb904009420 con 0x7fb9140c6f60 2023-10-30T13:09:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb90bfff640 1 --2- 172.21.15.125:0/744343867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9140c6f60 0x7fb91415f680 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fb90400e9d0 tx=0x7fb90400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb9097fa640 1 -- 172.21.15.125:0/744343867 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb90400ca60 con 0x7fb9140c6f60 2023-10-30T13:09:03.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb9097fa640 1 -- 172.21.15.125:0/744343867 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb90400cbc0 con 0x7fb9140c6f60 2023-10-30T13:09:03.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb9097fa640 1 -- 172.21.15.125:0/744343867 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb904013470 con 0x7fb9140c6f60 2023-10-30T13:09:03.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb91abab640 1 -- 172.21.15.125:0/744343867 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9100094d0 con 0x7fb9140c6f60 2023-10-30T13:09:03.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb91abab640 1 -- 172.21.15.125:0/744343867 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9141f2110 con 0x7fb9140c6f60 2023-10-30T13:09:03.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb91abab640 1 -- 172.21.15.125:0/744343867 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb91404e170 con 0x7fb9140c6f60 2023-10-30T13:09:03.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb9097fa640 1 -- 172.21.15.125:0/744343867 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb9040059a0 con 0x7fb9140c6f60 2023-10-30T13:09:03.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb9097fa640 1 --2- 172.21.15.125:0/744343867 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9000664a0 0x7fb900068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:03.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb9097fa640 1 -- 172.21.15.125:0/744343867 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7fb904084640 con 0x7fb9140c6f60 2023-10-30T13:09:03.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.351+0000 7fb918947640 1 --2- 172.21.15.125:0/744343867 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9000664a0 0x7fb900068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:03.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.355+0000 7fb918947640 1 --2- 172.21.15.125:0/744343867 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9000664a0 0x7fb900068960 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb9100af000 tx=0x7fb9100b6040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:03.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.355+0000 7fb9097fa640 1 -- 172.21.15.125:0/744343867 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb904051260 con 0x7fb9140c6f60 2023-10-30T13:09:03.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.535+0000 7fb91abab640 1 -- 172.21.15.125:0/744343867 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb914158ba0 con 0x7fb9140c6f60 2023-10-30T13:09:03.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.535+0000 7fb9097fa640 1 -- 172.21.15.125:0/744343867 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+5757 (secure 0 0 0) 0x7fb904017090 con 0x7fb9140c6f60 2023-10-30T13:09:03.540 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:09:03.540 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:09:00.664980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:resolve","state_seq":8,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:09:03.541 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 188 2023-10-30T13:09:03.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.539+0000 7fb91abab640 1 -- 172.21.15.125:0/744343867 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9000664a0 msgr2=0x7fb900068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:03.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.539+0000 7fb91abab640 1 --2- 172.21.15.125:0/744343867 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9000664a0 0x7fb900068960 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fb9100af000 tx=0x7fb9100b6040 comp rx=0 tx=0).stop 2023-10-30T13:09:03.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.539+0000 7fb91abab640 1 -- 172.21.15.125:0/744343867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9140c6f60 msgr2=0x7fb91415f680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:03.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.539+0000 7fb91abab640 1 --2- 172.21.15.125:0/744343867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9140c6f60 0x7fb91415f680 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7fb90400e9d0 tx=0x7fb90400ee90 comp rx=0 tx=0).stop 2023-10-30T13:09:03.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.539+0000 7fb91abab640 1 -- 172.21.15.125:0/744343867 shutdown_connections 2023-10-30T13:09:03.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.539+0000 7fb91abab640 1 --2- 172.21.15.125:0/744343867 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9000664a0 0x7fb900068960 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:03.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.539+0000 7fb91abab640 1 --2- 172.21.15.125:0/744343867 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9140c8430 0x7fb91415fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:03.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.539+0000 7fb91abab640 1 --2- 172.21.15.125:0/744343867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9140c6f60 0x7fb91415f680 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:03.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.539+0000 7fb91abab640 1 --2- 172.21.15.125:0/744343867 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9140c6b80 0x7fb914138a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:03.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.539+0000 7fb91abab640 1 -- 172.21.15.125:0/744343867 >> 172.21.15.125:0/744343867 conn(0x7fb9140c7910 msgr2=0x7fb91413ed50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:03.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.539+0000 7fb91abab640 1 -- 172.21.15.125:0/744343867 shutdown_connections 2023-10-30T13:09:03.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:03.539+0000 7fb91abab640 1 -- 172.21.15.125:0/744343867 wait complete. 2023-10-30T13:09:03.560 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:09:05.561 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:09:05.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f450c5aa640 1 Processor -- start 2023-10-30T13:09:05.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f450c5aa640 1 -- start start 2023-10-30T13:09:05.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f450c5aa640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f45041538a0 0x7f450415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:05.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f450c5aa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f45040ca0e0 0x7f450415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:05.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f450c5aa640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45040cb800 con 0x7f45040cf160 2023-10-30T13:09:05.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f450c5aa640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f450415dc30 con 0x7f45040ca0e0 2023-10-30T13:09:05.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f450c5aa640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45040cb330 con 0x7f45041538a0 2023-10-30T13:09:05.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f4509b45640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f45040ca0e0 0x7f450415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:05.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f4509b45640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f45040ca0e0 0x7f450415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56268/0 (socket says 172.21.15.125:56268) 2023-10-30T13:09:05.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f4509b45640 1 -- 172.21.15.125:0/1027244246 learned_addr learned my addr 172.21.15.125:0/1027244246 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:05.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f450a346640 1 --2- 172.21.15.125:0/1027244246 >> v2:172.21.15.136:3300/0 conn(0x7f45041538a0 0x7f450415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:05.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f4509b45640 1 -- 172.21.15.125:0/1027244246 >> v2:172.21.15.136:3300/0 conn(0x7f45041538a0 msgr2=0x7f450415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:09:05.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f4509b45640 1 --2- 172.21.15.125:0/1027244246 >> v2:172.21.15.136:3300/0 conn(0x7f45041538a0 0x7f450415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:05.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f4509b45640 1 -- 172.21.15.125:0/1027244246 >> v1:172.21.15.136:6789/0 conn(0x7f45040cf160 legacy=0x7f45040cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:09:05.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.707+0000 7f4509b45640 1 -- 172.21.15.125:0/1027244246 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45040cad70 con 0x7f45040ca0e0 2023-10-30T13:09:05.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.711+0000 7f4509b45640 1 --2- 172.21.15.125:0/1027244246 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45040ca0e0 0x7f450415a090 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f45040cfd10 tx=0x7f44f401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9056f9a43b223bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:05.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.711+0000 7f4509344640 1 -- 172.21.15.125:0/1027244246 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f40a89b0 con 0x7f45040ca0e0 2023-10-30T13:09:05.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.711+0000 7f4509344640 1 -- 172.21.15.125:0/1027244246 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f40a8b10 con 0x7f45040ca0e0 2023-10-30T13:09:05.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.711+0000 7f4509344640 1 -- 172.21.15.125:0/1027244246 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44f40a8ea0 con 0x7f45040ca0e0 2023-10-30T13:09:05.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.711+0000 7f450c5aa640 1 -- 172.21.15.125:0/1027244246 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45040ca0e0 msgr2=0x7f450415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:05.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.711+0000 7f450c5aa640 1 --2- 172.21.15.125:0/1027244246 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45040ca0e0 0x7f450415a090 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f45040cfd10 tx=0x7f44f401ecb0 comp rx=0 tx=0).stop 2023-10-30T13:09:05.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.711+0000 7f450c5aa640 1 -- 172.21.15.125:0/1027244246 shutdown_connections 2023-10-30T13:09:05.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.711+0000 7f450c5aa640 1 --2- 172.21.15.125:0/1027244246 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45040ca0e0 0x7f450415a090 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:05.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.711+0000 7f450c5aa640 1 --2- 172.21.15.125:0/1027244246 >> v2:172.21.15.136:3300/0 conn(0x7f45041538a0 0x7f450415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:05.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.711+0000 7f450c5aa640 1 -- 172.21.15.125:0/1027244246 >> 172.21.15.125:0/1027244246 conn(0x7f450406bfe0 msgr2=0x7f450406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:05.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.711+0000 7f450c5aa640 1 -- 172.21.15.125:0/1027244246 shutdown_connections 2023-10-30T13:09:05.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.711+0000 7f450c5aa640 1 -- 172.21.15.125:0/1027244246 wait complete. 2023-10-30T13:09:05.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450c5aa640 1 Processor -- start 2023-10-30T13:09:05.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450c5aa640 1 -- start start 2023-10-30T13:09:05.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450c5aa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45040ca0e0 0x7f45041f0340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:05.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450c5aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f45040cf160 0x7f45041e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:05.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450c5aa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45041538a0 0x7f4504157ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:05.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450c5aa640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45041e4450 con 0x7f45041538a0 2023-10-30T13:09:05.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450c5aa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4504152eb0 con 0x7f45040ca0e0 2023-10-30T13:09:05.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450c5aa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45041f0880 con 0x7f45040cf160 2023-10-30T13:09:05.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450ab47640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45041538a0 0x7f4504157ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:05.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450ab47640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45041538a0 0x7f4504157ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56278/0 (socket says 172.21.15.125:56278) 2023-10-30T13:09:05.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450ab47640 1 -- 172.21.15.125:0/4114086362 learned_addr learned my addr 172.21.15.125:0/4114086362 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:05.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450a346640 1 --2- 172.21.15.125:0/4114086362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45040ca0e0 0x7f45041f0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:05.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f4509b45640 1 --2- 172.21.15.125:0/4114086362 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f45040cf160 0x7f45041e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:05.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450ab47640 1 -- 172.21.15.125:0/4114086362 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f45040cf160 msgr2=0x7f45041e5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:05.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450ab47640 1 --2- 172.21.15.125:0/4114086362 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f45040cf160 0x7f45041e5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:05.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450ab47640 1 -- 172.21.15.125:0/4114086362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45040ca0e0 msgr2=0x7f45041f0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:05.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450ab47640 1 --2- 172.21.15.125:0/4114086362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45040ca0e0 0x7f45041f0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:05.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450ab47640 1 -- 172.21.15.125:0/4114086362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f40094d0 con 0x7f45041538a0 2023-10-30T13:09:05.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450ab47640 1 --2- 172.21.15.125:0/4114086362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45041538a0 0x7f4504157ee0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f44fc00c7b0 tx=0x7f44fc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:05.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f44faffd640 1 -- 172.21.15.125:0/4114086362 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc004b20 con 0x7f45041538a0 2023-10-30T13:09:05.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f44faffd640 1 -- 172.21.15.125:0/4114086362 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44fc004c80 con 0x7f45041538a0 2023-10-30T13:09:05.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f44faffd640 1 -- 172.21.15.125:0/4114086362 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc00ce60 con 0x7f45041538a0 2023-10-30T13:09:05.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450c5aa640 1 -- 172.21.15.125:0/4114086362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45041e3a40 con 0x7f45041538a0 2023-10-30T13:09:05.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.715+0000 7f450c5aa640 1 -- 172.21.15.125:0/4114086362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45041e2c80 con 0x7f45041538a0 2023-10-30T13:09:05.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.719+0000 7f450c5aa640 1 -- 172.21.15.125:0/4114086362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45040b2a50 con 0x7f45041538a0 2023-10-30T13:09:05.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.719+0000 7f44faffd640 1 -- 172.21.15.125:0/4114086362 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f44fc00a7f0 con 0x7f45041538a0 2023-10-30T13:09:05.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.719+0000 7f44faffd640 1 --2- 172.21.15.125:0/4114086362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f44e80664a0 0x7f44e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:05.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.719+0000 7f44faffd640 1 -- 172.21.15.125:0/4114086362 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7f44fc084570 con 0x7f45041538a0 2023-10-30T13:09:05.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.719+0000 7f450a346640 1 --2- 172.21.15.125:0/4114086362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f44e80664a0 0x7f44e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:05.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.719+0000 7f450a346640 1 --2- 172.21.15.125:0/4114086362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f44e80664a0 0x7f44e8068960 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f45000079c0 tx=0x7f4500008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:05.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.723+0000 7f44faffd640 1 -- 172.21.15.125:0/4114086362 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f44fc0512b0 con 0x7f45041538a0 2023-10-30T13:09:05.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.903+0000 7f450c5aa640 1 -- 172.21.15.125:0/4114086362 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f450404e170 con 0x7f45041538a0 2023-10-30T13:09:05.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.907+0000 7f44faffd640 1 -- 172.21.15.125:0/4114086362 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+5757 (secure 0 0 0) 0x7f44fc0548d0 con 0x7f45041538a0 2023-10-30T13:09:05.909 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:09:05.910 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:09:00.664980+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:resolve","state_seq":8,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:09:05.911 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 188 2023-10-30T13:09:05.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.907+0000 7f450c5aa640 1 -- 172.21.15.125:0/4114086362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f44e80664a0 msgr2=0x7f44e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:05.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.907+0000 7f450c5aa640 1 --2- 172.21.15.125:0/4114086362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f44e80664a0 0x7f44e8068960 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f45000079c0 tx=0x7f4500008040 comp rx=0 tx=0).stop 2023-10-30T13:09:05.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.907+0000 7f450c5aa640 1 -- 172.21.15.125:0/4114086362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45041538a0 msgr2=0x7f4504157ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.907+0000 7f450c5aa640 1 --2- 172.21.15.125:0/4114086362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45041538a0 0x7f4504157ee0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f44fc00c7b0 tx=0x7f44fc00cc70 comp rx=0 tx=0).stop 2023-10-30T13:09:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.907+0000 7f450c5aa640 1 -- 172.21.15.125:0/4114086362 shutdown_connections 2023-10-30T13:09:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.907+0000 7f450c5aa640 1 --2- 172.21.15.125:0/4114086362 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f44e80664a0 0x7f44e8068960 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:05.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.907+0000 7f450c5aa640 1 --2- 172.21.15.125:0/4114086362 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f45041538a0 0x7f4504157ee0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:05.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.907+0000 7f450c5aa640 1 --2- 172.21.15.125:0/4114086362 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f45040cf160 0x7f45041e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:05.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.907+0000 7f450c5aa640 1 --2- 172.21.15.125:0/4114086362 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f45040ca0e0 0x7f45041f0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:05.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.907+0000 7f450c5aa640 1 -- 172.21.15.125:0/4114086362 >> 172.21.15.125:0/4114086362 conn(0x7f450406bfe0 msgr2=0x7f4504073580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:05.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.907+0000 7f450c5aa640 1 -- 172.21.15.125:0/4114086362 shutdown_connections 2023-10-30T13:09:05.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:05.907+0000 7f450c5aa640 1 -- 172.21.15.125:0/4114086362 wait complete. 2023-10-30T13:09:05.930 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:09:07.931 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:09:08.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.067+0000 7f1ccf53e640 1 Processor -- start 2023-10-30T13:09:08.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.067+0000 7f1ccf53e640 1 -- start start 2023-10-30T13:09:08.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccf53e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1cc8159e60 0x7f1cc806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:08.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccf53e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cc80bfec0 0x7f1cc80c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:08.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccf53e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1cc806a720 0x7f1cc806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:08.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccf53e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc806fa80 con 0x7f1cc8159e60 2023-10-30T13:09:08.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccf53e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc8071290 con 0x7f1cc806a720 2023-10-30T13:09:08.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccf53e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc8071570 con 0x7f1cc80bfec0 2023-10-30T13:09:08.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccd2da640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1cc8159e60 0x7f1cc806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:08.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccd2da640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1cc8159e60 0x7f1cc806f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56290/0 (socket says 172.21.15.125:56290) 2023-10-30T13:09:08.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccd2da640 1 -- 172.21.15.125:0/1101561553 learned_addr learned my addr 172.21.15.125:0/1101561553 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:08.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1cccad9640 1 --2- 172.21.15.125:0/1101561553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cc80bfec0 0x7f1cc80c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:08.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccdadb640 1 --2- 172.21.15.125:0/1101561553 >> v2:172.21.15.136:3300/0 conn(0x7f1cc806a720 0x7f1cc806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:08.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccd2da640 1 -- 172.21.15.125:0/1101561553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cc80bfec0 msgr2=0x7f1cc80c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:09:08.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccd2da640 1 --2- 172.21.15.125:0/1101561553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cc80bfec0 0x7f1cc80c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:08.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccd2da640 1 -- 172.21.15.125:0/1101561553 >> v2:172.21.15.136:3300/0 conn(0x7f1cc806a720 msgr2=0x7f1cc806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:09:08.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccd2da640 1 --2- 172.21.15.125:0/1101561553 >> v2:172.21.15.136:3300/0 conn(0x7f1cc806a720 0x7f1cc806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:08.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccd2da640 1 -- 172.21.15.125:0/1101561553 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc8071850 con 0x7f1cc8159e60 2023-10-30T13:09:08.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.071+0000 7f1ccd2da640 1 --2- 172.21.15.125:0/1101561553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc8159e60 0x7f1cc806f540 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f1cc80c8900 tx=0x7f1cc401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d731982795359f24 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:08.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1cbffff640 1 -- 172.21.15.125:0/1101561553 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc40a89b0 con 0x7f1cc8159e60 2023-10-30T13:09:08.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1cbffff640 1 -- 172.21.15.125:0/1101561553 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc40a8b10 con 0x7f1cc8159e60 2023-10-30T13:09:08.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1cbffff640 1 -- 172.21.15.125:0/1101561553 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc40a8ea0 con 0x7f1cc8159e60 2023-10-30T13:09:08.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 -- 172.21.15.125:0/1101561553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc8159e60 msgr2=0x7f1cc806f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:08.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 --2- 172.21.15.125:0/1101561553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc8159e60 0x7f1cc806f540 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f1cc80c8900 tx=0x7f1cc401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:09:08.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 -- 172.21.15.125:0/1101561553 shutdown_connections 2023-10-30T13:09:08.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 --2- 172.21.15.125:0/1101561553 >> v2:172.21.15.136:3300/0 conn(0x7f1cc806a720 0x7f1cc806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:08.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 --2- 172.21.15.125:0/1101561553 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cc80bfec0 0x7f1cc80c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:08.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 --2- 172.21.15.125:0/1101561553 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc8159e60 0x7f1cc806f540 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:08.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 -- 172.21.15.125:0/1101561553 >> 172.21.15.125:0/1101561553 conn(0x7f1cc80c7910 msgr2=0x7f1cc80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:08.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 -- 172.21.15.125:0/1101561553 shutdown_connections 2023-10-30T13:09:08.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 -- 172.21.15.125:0/1101561553 wait complete. 2023-10-30T13:09:08.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 Processor -- start 2023-10-30T13:09:08.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 -- start start 2023-10-30T13:09:08.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cc806a720 0x7f1cc81f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:08.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc80bfec0 0x7f1cc814db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:08.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc8159e60 0x7f1cc80c9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:08.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80c3e90 con 0x7f1cc8159e60 2023-10-30T13:09:08.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80c4980 con 0x7f1cc80bfec0 2023-10-30T13:09:08.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccf53e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80c2c30 con 0x7f1cc806a720 2023-10-30T13:09:08.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.075+0000 7f1ccdadb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc8159e60 0x7f1cc80c9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:08.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1ccdadb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc8159e60 0x7f1cc80c9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56298/0 (socket says 172.21.15.125:56298) 2023-10-30T13:09:08.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1ccdadb640 1 -- 172.21.15.125:0/967384922 learned_addr learned my addr 172.21.15.125:0/967384922 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:08.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1ccd2da640 1 --2- 172.21.15.125:0/967384922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cc806a720 0x7f1cc81f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:08.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1cccad9640 1 --2- 172.21.15.125:0/967384922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc80bfec0 0x7f1cc814db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:08.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1ccdadb640 1 -- 172.21.15.125:0/967384922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cc806a720 msgr2=0x7f1cc81f1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:08.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1ccdadb640 1 --2- 172.21.15.125:0/967384922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cc806a720 0x7f1cc81f1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:08.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1ccdadb640 1 -- 172.21.15.125:0/967384922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc80bfec0 msgr2=0x7f1cc814db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:08.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1ccdadb640 1 --2- 172.21.15.125:0/967384922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc80bfec0 0x7f1cc814db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:08.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1ccdadb640 1 -- 172.21.15.125:0/967384922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc40094d0 con 0x7f1cc8159e60 2023-10-30T13:09:08.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1ccdadb640 1 --2- 172.21.15.125:0/967384922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc8159e60 0x7f1cc80c9620 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0007e60 tx=0x7f1cc000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:08.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1cbdffb640 1 -- 172.21.15.125:0/967384922 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc0014620 con 0x7f1cc8159e60 2023-10-30T13:09:08.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1cbdffb640 1 -- 172.21.15.125:0/967384922 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc000ee90 con 0x7f1cc8159e60 2023-10-30T13:09:08.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1ccf53e640 1 -- 172.21.15.125:0/967384922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc80ca2f0 con 0x7f1cc8159e60 2023-10-30T13:09:08.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1cbdffb640 1 -- 172.21.15.125:0/967384922 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cc00149e0 con 0x7f1cc8159e60 2023-10-30T13:09:08.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1ccf53e640 1 -- 172.21.15.125:0/967384922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc80c43c0 con 0x7f1cc8159e60 2023-10-30T13:09:08.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1ccf53e640 1 -- 172.21.15.125:0/967384922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc804e170 con 0x7f1cc8159e60 2023-10-30T13:09:08.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1cbdffb640 1 -- 172.21.15.125:0/967384922 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1cc0014b40 con 0x7f1cc8159e60 2023-10-30T13:09:08.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.079+0000 7f1cbdffb640 1 --2- 172.21.15.125:0/967384922 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1cac0664a0 0x7f1cac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:08.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.083+0000 7f1cbdffb640 1 -- 172.21.15.125:0/967384922 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7f1cc0084e50 con 0x7f1cc8159e60 2023-10-30T13:09:08.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.083+0000 7f1ccd2da640 1 --2- 172.21.15.125:0/967384922 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1cac0664a0 0x7f1cac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:08.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.083+0000 7f1ccd2da640 1 --2- 172.21.15.125:0/967384922 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1cac0664a0 0x7f1cac068960 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f1cc40b2000 tx=0x7f1cc40aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:08.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.083+0000 7f1cbdffb640 1 -- 172.21.15.125:0/967384922 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1cc0051ba0 con 0x7f1cc8159e60 2023-10-30T13:09:08.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.271+0000 7f1ccf53e640 1 -- 172.21.15.125:0/967384922 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cc80c52e0 con 0x7f1cc8159e60 2023-10-30T13:09:08.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.271+0000 7f1cbdffb640 1 -- 172.21.15.125:0/967384922 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+5760 (secure 0 0 0) 0x7f1cc00551c0 con 0x7f1cc8159e60 2023-10-30T13:09:08.273 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:09:08.273 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:09:06.666587+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:09:08.273 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 189 2023-10-30T13:09:08.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.271+0000 7f1ccf53e640 1 -- 172.21.15.125:0/967384922 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1cac0664a0 msgr2=0x7f1cac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:08.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.271+0000 7f1ccf53e640 1 --2- 172.21.15.125:0/967384922 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1cac0664a0 0x7f1cac068960 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f1cc40b2000 tx=0x7f1cc40aa070 comp rx=0 tx=0).stop 2023-10-30T13:09:08.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.271+0000 7f1ccf53e640 1 -- 172.21.15.125:0/967384922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc8159e60 msgr2=0x7f1cc80c9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:08.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.271+0000 7f1ccf53e640 1 --2- 172.21.15.125:0/967384922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc8159e60 0x7f1cc80c9620 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f1cc0007e60 tx=0x7f1cc000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:09:08.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.271+0000 7f1ccf53e640 1 -- 172.21.15.125:0/967384922 shutdown_connections 2023-10-30T13:09:08.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.271+0000 7f1ccf53e640 1 --2- 172.21.15.125:0/967384922 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1cac0664a0 0x7f1cac068960 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:08.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.271+0000 7f1ccf53e640 1 --2- 172.21.15.125:0/967384922 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1cc8159e60 0x7f1cc80c9620 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:08.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.271+0000 7f1ccf53e640 1 --2- 172.21.15.125:0/967384922 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1cc80bfec0 0x7f1cc814db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:08.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.271+0000 7f1ccf53e640 1 --2- 172.21.15.125:0/967384922 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1cc806a720 0x7f1cc81f1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:08.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.271+0000 7f1ccf53e640 1 -- 172.21.15.125:0/967384922 >> 172.21.15.125:0/967384922 conn(0x7f1cc80c7910 msgr2=0x7f1cc814fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:08.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.271+0000 7f1ccf53e640 1 -- 172.21.15.125:0/967384922 shutdown_connections 2023-10-30T13:09:08.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:08.271+0000 7f1ccf53e640 1 -- 172.21.15.125:0/967384922 wait complete. 2023-10-30T13:09:08.293 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 189, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 189, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_7340": { "addr": "172.21.15.136:6835/3526499585", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3526499585, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3526499585, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7340, "incarnation": 187, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:reconnect", "state_seq": 10 }, "gid_8301": { "addr": "172.21.15.125:6837/2540063352", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2540063352, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2540063352, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8301, "incarnation": 168, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_8529": { "addr": "172.21.15.125:6836/3950044434", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3950044434, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3950044434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 8529, "incarnation": 179, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 77, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:09:06.666587+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8529, "mds_1": 8301, "mds_2": 7340 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:09:10.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:09:10.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb744d6640 1 Processor -- start 2023-10-30T13:09:10.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb744d6640 1 -- start start 2023-10-30T13:09:10.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb744d6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fcb6c0c6b80 0x7fcb6c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:10.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb744d6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb6c0c6f60 0x7fcb6c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:10.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb744d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb6c0c0a50 con 0x7fcb6c0c8430 2023-10-30T13:09:10.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb744d6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb6c06f6e0 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb744d6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb6c06f9c0 con 0x7fcb6c0c6b80 2023-10-30T13:09:10.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb72a73640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcb6c0c8430 0x7fcb6c13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37218/0 (socket says 172.21.15.125:37218) 2023-10-30T13:09:10.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb72a73640 1 -- 172.21.15.125:0/4269430 learned_addr learned my addr 172.21.15.125:0/4269430 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:09:10.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb71a71640 1 --2- 172.21.15.125:0/4269430 >> v2:172.21.15.125:3300/0 conn(0x7fcb6c0c6f60 0x7fcb6c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:10.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb72272640 1 --2- 172.21.15.125:0/4269430 >> v2:172.21.15.136:3300/0 conn(0x7fcb6c0c6b80 0x7fcb6c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:10.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb71a71640 1 -- 172.21.15.125:0/4269430 >> v2:172.21.15.136:3300/0 conn(0x7fcb6c0c6b80 msgr2=0x7fcb6c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:09:10.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb71a71640 1 --2- 172.21.15.125:0/4269430 >> v2:172.21.15.136:3300/0 conn(0x7fcb6c0c6b80 0x7fcb6c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:10.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb71a71640 1 -- 172.21.15.125:0/4269430 >> v1:172.21.15.125:6789/0 conn(0x7fcb6c0c8430 legacy=0x7fcb6c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:10.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb71a71640 1 -- 172.21.15.125:0/4269430 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb6c06fca0 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.439+0000 7fcb71270640 1 -- 172.21.15.125:0/4269430 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400141841 0 0) 0x7fcb6c0c0a50 con 0x7fcb6c0c8430 2023-10-30T13:09:10.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.443+0000 7fcb71a71640 1 --2- 172.21.15.125:0/4269430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb6c0c6f60 0x7fcb6c0c0400 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c06f1b0 tx=0x7fcb5c01c460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ea03d29999eefae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:10.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.443+0000 7fcb71270640 1 -- 172.21.15.125:0/4269430 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0ab600 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.443+0000 7fcb71270640 1 -- 172.21.15.125:0/4269430 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0a7050 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.443+0000 7fcb71270640 1 -- 172.21.15.125:0/4269430 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0ab980 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.443+0000 7fcb744d6640 1 -- 172.21.15.125:0/4269430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb6c0c6f60 msgr2=0x7fcb6c0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:10.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.443+0000 7fcb744d6640 1 --2- 172.21.15.125:0/4269430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb6c0c6f60 0x7fcb6c0c0400 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c06f1b0 tx=0x7fcb5c01c460 comp rx=0 tx=0).stop 2023-10-30T13:09:10.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.443+0000 7fcb744d6640 1 -- 172.21.15.125:0/4269430 shutdown_connections 2023-10-30T13:09:10.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.443+0000 7fcb744d6640 1 --2- 172.21.15.125:0/4269430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb6c0c6f60 0x7fcb6c0c0400 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:10.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.443+0000 7fcb744d6640 1 --2- 172.21.15.125:0/4269430 >> v2:172.21.15.136:3300/0 conn(0x7fcb6c0c6b80 0x7fcb6c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:10.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.443+0000 7fcb744d6640 1 -- 172.21.15.125:0/4269430 >> 172.21.15.125:0/4269430 conn(0x7fcb6c0c7910 msgr2=0x7fcb6c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:10.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.443+0000 7fcb744d6640 1 -- 172.21.15.125:0/4269430 shutdown_connections 2023-10-30T13:09:10.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.443+0000 7fcb744d6640 1 -- 172.21.15.125:0/4269430 wait complete. 2023-10-30T13:09:10.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb744d6640 1 Processor -- start 2023-10-30T13:09:10.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb744d6640 1 -- start start 2023-10-30T13:09:10.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb744d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb6c0c6b80 0x7fcb6c155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:10.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb744d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb6c0c6f60 0x7fcb6c1f5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:10.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb744d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb6c0c8430 0x7fcb6c1367d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:10.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb744d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c159360 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb744d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c1f6180 con 0x7fcb6c0c8430 2023-10-30T13:09:10.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb744d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6c1f5ea0 con 0x7fcb6c0c6b80 2023-10-30T13:09:10.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb71a71640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb6c0c6f60 0x7fcb6c1f5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:10.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb71a71640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb6c0c6f60 0x7fcb6c1f5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56314/0 (socket says 172.21.15.125:56314) 2023-10-30T13:09:10.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb71a71640 1 -- 172.21.15.125:0/2642030328 learned_addr learned my addr 172.21.15.125:0/2642030328 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:10.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb72272640 1 --2- 172.21.15.125:0/2642030328 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb6c0c6b80 0x7fcb6c155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:10.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb72a73640 1 --2- 172.21.15.125:0/2642030328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb6c0c8430 0x7fcb6c1367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:10.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb71a71640 1 -- 172.21.15.125:0/2642030328 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb6c0c6b80 msgr2=0x7fcb6c155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:10.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb71a71640 1 --2- 172.21.15.125:0/2642030328 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb6c0c6b80 0x7fcb6c155370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:10.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb71a71640 1 -- 172.21.15.125:0/2642030328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb6c0c8430 msgr2=0x7fcb6c1367d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:10.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb71a71640 1 --2- 172.21.15.125:0/2642030328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb6c0c8430 0x7fcb6c1367d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:10.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb71a71640 1 -- 172.21.15.125:0/2642030328 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5c009420 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb71a71640 1 --2- 172.21.15.125:0/2642030328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb6c0c6f60 0x7fcb6c1f5490 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0ae040 tx=0x7fcb5c019200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:10.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb62ffd640 1 -- 172.21.15.125:0/2642030328 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0b4840 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb62ffd640 1 -- 172.21.15.125:0/2642030328 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0b49c0 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb62ffd640 1 -- 172.21.15.125:0/2642030328 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c0b4d70 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb744d6640 1 -- 172.21.15.125:0/2642030328 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb6c0ca7a0 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.447+0000 7fcb744d6640 1 -- 172.21.15.125:0/2642030328 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb6c1558b0 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.451+0000 7fcb744d6640 1 -- 172.21.15.125:0/2642030328 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb6c0cac50 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.451+0000 7fcb62ffd640 1 -- 172.21.15.125:0/2642030328 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcb5c0b2040 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.451+0000 7fcb62ffd640 1 --2- 172.21.15.125:0/2642030328 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb500664a0 0x7fcb50068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:10.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.451+0000 7fcb62ffd640 1 -- 172.21.15.125:0/2642030328 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7fcb5c11eb30 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.451+0000 7fcb72272640 1 --2- 172.21.15.125:0/2642030328 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb500664a0 0x7fcb50068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:10.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.451+0000 7fcb72272640 1 --2- 172.21.15.125:0/2642030328 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb500664a0 0x7fcb50068960 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009af0 tx=0x7fcb68008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:10.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.455+0000 7fcb62ffd640 1 -- 172.21.15.125:0/2642030328 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcb5c0eb730 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.631+0000 7fcb744d6640 1 -- 172.21.15.125:0/2642030328 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb6c1594d0 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.635+0000 7fcb62ffd640 1 -- 172.21.15.125:0/2642030328 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 189 v189) v1 ==== 95+0+5760 (secure 0 0 0) 0x7fcb5c0b0090 con 0x7fcb6c0c6f60 2023-10-30T13:09:10.636 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:09:10.637 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":189,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":189,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:09:06.666587+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:reconnect","state_seq":10,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:09:10.637 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 189 2023-10-30T13:09:10.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.635+0000 7fcb744d6640 1 -- 172.21.15.125:0/2642030328 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb500664a0 msgr2=0x7fcb50068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:10.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.635+0000 7fcb744d6640 1 --2- 172.21.15.125:0/2642030328 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb500664a0 0x7fcb50068960 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fcb68009af0 tx=0x7fcb68008040 comp rx=0 tx=0).stop 2023-10-30T13:09:10.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.635+0000 7fcb744d6640 1 -- 172.21.15.125:0/2642030328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb6c0c6f60 msgr2=0x7fcb6c1f5490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:10.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.635+0000 7fcb744d6640 1 --2- 172.21.15.125:0/2642030328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb6c0c6f60 0x7fcb6c1f5490 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fcb5c0ae040 tx=0x7fcb5c019200 comp rx=0 tx=0).stop 2023-10-30T13:09:10.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.635+0000 7fcb744d6640 1 -- 172.21.15.125:0/2642030328 shutdown_connections 2023-10-30T13:09:10.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.635+0000 7fcb744d6640 1 --2- 172.21.15.125:0/2642030328 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb500664a0 0x7fcb50068960 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:10.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.635+0000 7fcb744d6640 1 --2- 172.21.15.125:0/2642030328 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb6c0c8430 0x7fcb6c1367d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:10.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.635+0000 7fcb744d6640 1 --2- 172.21.15.125:0/2642030328 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb6c0c6f60 0x7fcb6c1f5490 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:10.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.635+0000 7fcb744d6640 1 --2- 172.21.15.125:0/2642030328 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb6c0c6b80 0x7fcb6c155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:10.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.635+0000 7fcb744d6640 1 -- 172.21.15.125:0/2642030328 >> 172.21.15.125:0/2642030328 conn(0x7fcb6c0c7910 msgr2=0x7fcb6c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:10.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.635+0000 7fcb744d6640 1 -- 172.21.15.125:0/2642030328 shutdown_connections 2023-10-30T13:09:10.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:10.635+0000 7fcb744d6640 1 -- 172.21.15.125:0/2642030328 wait complete. 2023-10-30T13:09:10.657 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:09:12.659 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:09:12.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.803+0000 7f8126bf8640 1 Processor -- start 2023-10-30T13:09:12.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.803+0000 7f8126bf8640 1 -- start start 2023-10-30T13:09:12.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.803+0000 7f8126bf8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8120153bf0 0x7f8120071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:12.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.803+0000 7f8126bf8640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81200700b0 con 0x7f81200c6fe0 2023-10-30T13:09:12.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.803+0000 7f8126bf8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81200c6530 con 0x7f81200c5f60 2023-10-30T13:09:12.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.803+0000 7f8126bf8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81200c6810 con 0x7f8120153bf0 2023-10-30T13:09:12.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.803+0000 7f8117fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f81200c6fe0 0x7f81200c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57822/0 (socket says 172.21.15.125:57822) 2023-10-30T13:09:12.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.803+0000 7f8117fff640 1 -- 172.21.15.125:0/1570453262 learned_addr learned my addr 172.21.15.125:0/1570453262 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:09:12.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.803+0000 7f8124994640 1 --2- 172.21.15.125:0/1570453262 >> v2:172.21.15.136:3300/0 conn(0x7f8120153bf0 0x7f8120071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:12.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.803+0000 7f81177fe640 1 -- 172.21.15.125:0/1570453262 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907415796 0 0) 0x7f81200700b0 con 0x7f81200c6fe0 2023-10-30T13:09:12.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.803+0000 7f81177fe640 1 -- 172.21.15.125:0/1570453262 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8108003530 con 0x7f81200c6fe0 2023-10-30T13:09:12.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.807+0000 7f8124994640 1 -- 172.21.15.125:0/1570453262 >> v1:172.21.15.136:6789/0 conn(0x7f81200c5f60 legacy=0x7f81200cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:12.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.807+0000 7f8124994640 1 -- 172.21.15.125:0/1570453262 >> v1:172.21.15.125:6789/0 conn(0x7f81200c6fe0 legacy=0x7f81200c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:12.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.807+0000 7f8124994640 1 -- 172.21.15.125:0/1570453262 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8120159b10 con 0x7f8120153bf0 2023-10-30T13:09:12.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.807+0000 7f8124994640 1 --2- 172.21.15.125:0/1570453262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8120153bf0 0x7f8120071260 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f811c0097c0 tx=0x7f811c01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=933585f77108c296 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:12.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.807+0000 7f81177fe640 1 -- 172.21.15.125:0/1570453262 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c0a8970 con 0x7f8120153bf0 2023-10-30T13:09:12.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.807+0000 7f81177fe640 1 -- 172.21.15.125:0/1570453262 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c0a8ad0 con 0x7f8120153bf0 2023-10-30T13:09:12.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.807+0000 7f81177fe640 1 -- 172.21.15.125:0/1570453262 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c0a8e60 con 0x7f8120153bf0 2023-10-30T13:09:12.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 -- 172.21.15.125:0/1570453262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8120153bf0 msgr2=0x7f8120071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:12.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 --2- 172.21.15.125:0/1570453262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8120153bf0 0x7f8120071260 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f811c0097c0 tx=0x7f811c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:09:12.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 -- 172.21.15.125:0/1570453262 shutdown_connections 2023-10-30T13:09:12.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 --2- 172.21.15.125:0/1570453262 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8120153bf0 0x7f8120071260 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:12.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 -- 172.21.15.125:0/1570453262 >> 172.21.15.125:0/1570453262 conn(0x7f812006c6e0 msgr2=0x7f8120069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:12.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 -- 172.21.15.125:0/1570453262 shutdown_connections 2023-10-30T13:09:12.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 -- 172.21.15.125:0/1570453262 wait complete. 2023-10-30T13:09:12.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 Processor -- start 2023-10-30T13:09:12.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 -- start start 2023-10-30T13:09:12.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81200c5f60 0x7f81200c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:12.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81200c6fe0 0x7f81201557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:12.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8120153bf0 0x7f8120155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:12.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8120155350 con 0x7f81200c5f60 2023-10-30T13:09:12.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812014fdb0 con 0x7f8120153bf0 2023-10-30T13:09:12.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8126bf8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81201ef9a0 con 0x7f81200c6fe0 2023-10-30T13:09:12.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8124994640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81200c5f60 0x7f81200c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:12.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8124994640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81200c5f60 0x7f81200c43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60200/0 (socket says 172.21.15.125:60200) 2023-10-30T13:09:12.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8124994640 1 -- 172.21.15.125:0/3156151801 learned_addr learned my addr 172.21.15.125:0/3156151801 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:12.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8117fff640 1 --2- 172.21.15.125:0/3156151801 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81200c6fe0 0x7f81201557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:12.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8125195640 1 --2- 172.21.15.125:0/3156151801 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8120153bf0 0x7f8120155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:12.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8124994640 1 -- 172.21.15.125:0/3156151801 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81200c6fe0 msgr2=0x7f81201557e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:12.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8124994640 1 --2- 172.21.15.125:0/3156151801 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81200c6fe0 0x7f81201557e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:12.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8124994640 1 -- 172.21.15.125:0/3156151801 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8120153bf0 msgr2=0x7f8120155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:12.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8124994640 1 --2- 172.21.15.125:0/3156151801 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8120153bf0 0x7f8120155d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:12.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.811+0000 7f8124994640 1 -- 172.21.15.125:0/3156151801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811c009420 con 0x7f81200c5f60 2023-10-30T13:09:12.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.815+0000 7f8124994640 1 --2- 172.21.15.125:0/3156151801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81200c5f60 0x7f81200c43c0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f811c0b3040 tx=0x7f811c01ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:12.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.815+0000 7f81157fa640 1 -- 172.21.15.125:0/3156151801 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c0b9630 con 0x7f81200c5f60 2023-10-30T13:09:12.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.815+0000 7f81157fa640 1 -- 172.21.15.125:0/3156151801 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f811c0b97b0 con 0x7f81200c5f60 2023-10-30T13:09:12.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.815+0000 7f81157fa640 1 -- 172.21.15.125:0/3156151801 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f811c0b9b60 con 0x7f81200c5f60 2023-10-30T13:09:12.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.815+0000 7f8126bf8640 1 -- 172.21.15.125:0/3156151801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8120150050 con 0x7f81200c5f60 2023-10-30T13:09:12.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.815+0000 7f8126bf8640 1 -- 172.21.15.125:0/3156151801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81201efc80 con 0x7f81200c5f60 2023-10-30T13:09:12.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.815+0000 7f8126bf8640 1 -- 172.21.15.125:0/3156151801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f812004e170 con 0x7f81200c5f60 2023-10-30T13:09:12.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.815+0000 7f81157fa640 1 -- 172.21.15.125:0/3156151801 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f811c0b7040 con 0x7f81200c5f60 2023-10-30T13:09:12.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.815+0000 7f81157fa640 1 --2- 172.21.15.125:0/3156151801 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f810c0664d0 0x7f810c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:12.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.815+0000 7f81157fa640 1 -- 172.21.15.125:0/3156151801 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7f811c122530 con 0x7f81200c5f60 2023-10-30T13:09:12.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.819+0000 7f8117fff640 1 --2- 172.21.15.125:0/3156151801 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f810c0664d0 0x7f810c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:12.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.819+0000 7f8117fff640 1 --2- 172.21.15.125:0/3156151801 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f810c0664d0 0x7f810c068990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8120151c40 tx=0x7f8110009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:12.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:12.823+0000 7f81157fa640 1 -- 172.21.15.125:0/3156151801 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f811c0ef130 con 0x7f81200c5f60 2023-10-30T13:09:13.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:13.002+0000 7f8126bf8640 1 -- 172.21.15.125:0/3156151801 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f812006fe90 con 0x7f81200c5f60 2023-10-30T13:09:13.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:13.002+0000 7f81157fa640 1 -- 172.21.15.125:0/3156151801 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5757 (secure 0 0 0) 0x7f811c0f2750 con 0x7f81200c5f60 2023-10-30T13:09:13.008 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:09:13.008 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:09:12.673410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:09:13.009 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 190 2023-10-30T13:09:13.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:13.006+0000 7f8126bf8640 1 -- 172.21.15.125:0/3156151801 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f810c0664d0 msgr2=0x7f810c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:13.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:13.006+0000 7f8126bf8640 1 --2- 172.21.15.125:0/3156151801 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f810c0664d0 0x7f810c068990 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8120151c40 tx=0x7f8110009040 comp rx=0 tx=0).stop 2023-10-30T13:09:13.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:13.006+0000 7f8126bf8640 1 -- 172.21.15.125:0/3156151801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81200c5f60 msgr2=0x7f81200c43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:13.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:13.006+0000 7f8126bf8640 1 --2- 172.21.15.125:0/3156151801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81200c5f60 0x7f81200c43c0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f811c0b3040 tx=0x7f811c01ba10 comp rx=0 tx=0).stop 2023-10-30T13:09:13.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:13.006+0000 7f8126bf8640 1 -- 172.21.15.125:0/3156151801 shutdown_connections 2023-10-30T13:09:13.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:13.006+0000 7f8126bf8640 1 --2- 172.21.15.125:0/3156151801 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f810c0664d0 0x7f810c068990 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:13.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:13.006+0000 7f8126bf8640 1 --2- 172.21.15.125:0/3156151801 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8120153bf0 0x7f8120155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:13.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:13.006+0000 7f8126bf8640 1 --2- 172.21.15.125:0/3156151801 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81200c6fe0 0x7f81201557e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:13.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:13.006+0000 7f8126bf8640 1 --2- 172.21.15.125:0/3156151801 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81200c5f60 0x7f81200c43c0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:13.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:13.006+0000 7f8126bf8640 1 -- 172.21.15.125:0/3156151801 >> 172.21.15.125:0/3156151801 conn(0x7f812006c6e0 msgr2=0x7f8120152c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:13.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:13.006+0000 7f8126bf8640 1 -- 172.21.15.125:0/3156151801 shutdown_connections 2023-10-30T13:09:13.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:13.006+0000 7f8126bf8640 1 -- 172.21.15.125:0/3156151801 wait complete. 2023-10-30T13:09:13.028 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:09:15.029 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:09:15.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.174+0000 7feda9994640 1 Processor -- start 2023-10-30T13:09:15.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.174+0000 7feda9994640 1 -- start start 2023-10-30T13:09:15.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.178+0000 7feda9994640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feda40c6b80 0x7feda40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:15.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.178+0000 7feda9994640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7feda40c6f60 0x7feda40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:15.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.178+0000 7feda9994640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda40c0a50 con 0x7feda40c8430 2023-10-30T13:09:15.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.178+0000 7feda9994640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feda406f6e0 con 0x7feda40c6f60 2023-10-30T13:09:15.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.178+0000 7feda9994640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda406f9c0 con 0x7feda40c6b80 2023-10-30T13:09:15.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.178+0000 7fed9a7fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7feda40c6f60 0x7feda40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:15.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.178+0000 7feda2ffd640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feda40c6b80 0x7feda40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:15.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.178+0000 7fed9a7fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7feda40c6f60 0x7feda40c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:47268/0 (socket says 172.21.15.125:47268) 2023-10-30T13:09:15.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.178+0000 7fed9a7fc640 1 -- 172.21.15.125:0/2291155027 learned_addr learned my addr 172.21.15.125:0/2291155027 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:15.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.178+0000 7fed9a7fc640 1 -- 172.21.15.125:0/2291155027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feda40c6b80 msgr2=0x7feda40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:09:15.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.178+0000 7fed9a7fc640 1 --2- 172.21.15.125:0/2291155027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feda40c6b80 0x7feda40bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:15.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.178+0000 7fed9a7fc640 1 -- 172.21.15.125:0/2291155027 >> v1:172.21.15.136:6789/0 conn(0x7feda40c8430 legacy=0x7feda413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:15.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.178+0000 7fed9a7fc640 1 -- 172.21.15.125:0/2291155027 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda406fca0 con 0x7feda40c6f60 2023-10-30T13:09:15.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.178+0000 7feda2ffd640 1 --2- 172.21.15.125:0/2291155027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feda40c6b80 0x7feda40bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:09:15.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.182+0000 7fed9a7fc640 1 --2- 172.21.15.125:0/2291155027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda40c6f60 0x7feda40c0400 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fed94009870 tx=0x7fed9401c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7213797df0ce55ae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:15.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.182+0000 7feda27fc640 1 -- 172.21.15.125:0/2291155027 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed940ab650 con 0x7feda40c6f60 2023-10-30T13:09:15.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.182+0000 7feda27fc640 1 -- 172.21.15.125:0/2291155027 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed940ab7b0 con 0x7feda40c6f60 2023-10-30T13:09:15.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.182+0000 7feda27fc640 1 -- 172.21.15.125:0/2291155027 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed940abb40 con 0x7feda40c6f60 2023-10-30T13:09:15.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.182+0000 7feda9994640 1 -- 172.21.15.125:0/2291155027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda40c6f60 msgr2=0x7feda40c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:15.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.182+0000 7feda9994640 1 --2- 172.21.15.125:0/2291155027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda40c6f60 0x7feda40c0400 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fed94009870 tx=0x7fed9401c540 comp rx=0 tx=0).stop 2023-10-30T13:09:15.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.182+0000 7feda9994640 1 -- 172.21.15.125:0/2291155027 shutdown_connections 2023-10-30T13:09:15.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.182+0000 7feda9994640 1 --2- 172.21.15.125:0/2291155027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda40c6f60 0x7feda40c0400 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:15.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.182+0000 7feda9994640 1 --2- 172.21.15.125:0/2291155027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feda40c6b80 0x7feda40bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:15.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.182+0000 7feda9994640 1 -- 172.21.15.125:0/2291155027 >> 172.21.15.125:0/2291155027 conn(0x7feda40c7910 msgr2=0x7feda40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:15.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.182+0000 7feda9994640 1 -- 172.21.15.125:0/2291155027 shutdown_connections 2023-10-30T13:09:15.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.182+0000 7feda9994640 1 -- 172.21.15.125:0/2291155027 wait complete. 2023-10-30T13:09:15.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.182+0000 7feda9994640 1 Processor -- start 2023-10-30T13:09:15.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.182+0000 7feda9994640 1 -- start start 2023-10-30T13:09:15.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda9994640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feda40c6b80 0x7feda413dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:15.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda9994640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feda40c6f60 0x7feda4154290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:15.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda9994640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda40c8430 0x7feda40c4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:15.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda9994640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda4160d90 con 0x7feda40c6b80 2023-10-30T13:09:15.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda9994640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feda41f2720 con 0x7feda40c8430 2023-10-30T13:09:15.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda9994640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feda41f6100 con 0x7feda40c6f60 2023-10-30T13:09:15.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda2ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feda40c6b80 0x7feda413dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:15.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda2ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feda40c6b80 0x7feda413dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60202/0 (socket says 172.21.15.125:60202) 2023-10-30T13:09:15.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda2ffd640 1 -- 172.21.15.125:0/3906384071 learned_addr learned my addr 172.21.15.125:0/3906384071 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:15.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7fed9a7fc640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feda40c6f60 0x7feda4154290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:15.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda37fe640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda40c8430 0x7feda40c4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:15.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda2ffd640 1 -- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feda40c6f60 msgr2=0x7feda4154290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:15.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda2ffd640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feda40c6f60 0x7feda4154290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:15.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda2ffd640 1 -- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda40c8430 msgr2=0x7feda40c4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:15.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda2ffd640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda40c8430 0x7feda40c4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:15.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda2ffd640 1 -- 172.21.15.125:0/3906384071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8c009420 con 0x7feda40c6b80 2023-10-30T13:09:15.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7fed9a7fc640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feda40c6f60 0x7feda4154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:09:15.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda2ffd640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feda40c6b80 0x7feda413dc00 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fed8c00ebe0 tx=0x7fed8c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:15.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda0ff9640 1 -- 172.21.15.125:0/3906384071 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8c00eee0 con 0x7feda40c6b80 2023-10-30T13:09:15.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda0ff9640 1 -- 172.21.15.125:0/3906384071 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8c00cc30 con 0x7feda40c6b80 2023-10-30T13:09:15.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda0ff9640 1 -- 172.21.15.125:0/3906384071 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8c013470 con 0x7feda40c6b80 2023-10-30T13:09:15.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda9994640 1 -- 172.21.15.125:0/3906384071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed940094d0 con 0x7feda40c6b80 2023-10-30T13:09:15.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda9994640 1 -- 172.21.15.125:0/3906384071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feda41f2bb0 con 0x7feda40c6b80 2023-10-30T13:09:15.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.186+0000 7feda9994640 1 -- 172.21.15.125:0/3906384071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feda404e170 con 0x7feda40c6b80 2023-10-30T13:09:15.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.190+0000 7feda0ff9640 1 -- 172.21.15.125:0/3906384071 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fed8c0135d0 con 0x7feda40c6b80 2023-10-30T13:09:15.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.190+0000 7feda0ff9640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed840664d0 0x7fed84068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:15.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.190+0000 7feda0ff9640 1 -- 172.21.15.125:0/3906384071 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7fed8c084e00 con 0x7feda40c6b80 2023-10-30T13:09:15.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.190+0000 7fed9a7fc640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed840664d0 0x7fed84068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:15.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.190+0000 7fed9a7fc640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed840664d0 0x7fed84068990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fed940af000 tx=0x7fed940a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:15.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.194+0000 7feda0ff9640 1 -- 172.21.15.125:0/3906384071 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fed8c051970 con 0x7feda40c6b80 2023-10-30T13:09:15.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.374+0000 7feda9994640 1 -- 172.21.15.125:0/3906384071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feda41f5e20 con 0x7feda40c6b80 2023-10-30T13:09:15.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.374+0000 7feda0ff9640 1 -- 172.21.15.125:0/3906384071 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5757 (secure 0 0 0) 0x7fed8c021090 con 0x7feda40c6b80 2023-10-30T13:09:15.378 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:09:15.379 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:09:12.673410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:09:15.379 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 190 2023-10-30T13:09:15.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.378+0000 7feda9994640 1 -- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed840664d0 msgr2=0x7fed84068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:15.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.378+0000 7feda9994640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed840664d0 0x7fed84068990 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fed940af000 tx=0x7fed940a7000 comp rx=0 tx=0).stop 2023-10-30T13:09:15.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.378+0000 7feda9994640 1 -- 172.21.15.125:0/3906384071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feda40c6b80 msgr2=0x7feda413dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:15.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.378+0000 7feda9994640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feda40c6b80 0x7feda413dc00 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fed8c00ebe0 tx=0x7fed8c00c520 comp rx=0 tx=0).stop 2023-10-30T13:09:15.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.378+0000 7feda9994640 1 -- 172.21.15.125:0/3906384071 shutdown_connections 2023-10-30T13:09:15.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.378+0000 7feda9994640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fed840664d0 0x7fed84068990 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:15.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.378+0000 7feda9994640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feda40c8430 0x7feda40c4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:15.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.378+0000 7feda9994640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feda40c6f60 0x7feda4154290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:15.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.378+0000 7feda9994640 1 --2- 172.21.15.125:0/3906384071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feda40c6b80 0x7feda413dc00 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:15.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.378+0000 7feda9994640 1 -- 172.21.15.125:0/3906384071 >> 172.21.15.125:0/3906384071 conn(0x7feda40c7910 msgr2=0x7feda413ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:15.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.378+0000 7feda9994640 1 -- 172.21.15.125:0/3906384071 shutdown_connections 2023-10-30T13:09:15.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:15.378+0000 7feda9994640 1 -- 172.21.15.125:0/3906384071 wait complete. 2023-10-30T13:09:15.399 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:09:17.400 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:09:17.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.542+0000 7fefdbbb8640 1 Processor -- start 2023-10-30T13:09:17.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.542+0000 7fefdbbb8640 1 -- start start 2023-10-30T13:09:17.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.546+0000 7fefdbbb8640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fefd41580e0 0x7fefd4070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:17.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.546+0000 7fefdbbb8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefd40c6540 0x7fefd415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:17.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.546+0000 7fefdbbb8640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd40c5810 con 0x7fefd4069e90 2023-10-30T13:09:17.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.546+0000 7fefdbbb8640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd40c6960 con 0x7fefd41580e0 2023-10-30T13:09:17.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.546+0000 7fefdbbb8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd40c6bb0 con 0x7fefd40c6540 2023-10-30T13:09:17.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.546+0000 7fefd9954640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fefd41580e0 0x7fefd4070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:17.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.546+0000 7fefda155640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fefd4069e90 0x7fefd406a320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:57824/0 (socket says 172.21.15.125:57824) 2023-10-30T13:09:17.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.546+0000 7fefda155640 1 -- 172.21.15.125:0/105430246 learned_addr learned my addr 172.21.15.125:0/105430246 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:09:17.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.546+0000 7fefd9153640 1 --2- 172.21.15.125:0/105430246 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefd40c6540 0x7fefd415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:17.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.546+0000 7fefd9954640 1 -- 172.21.15.125:0/105430246 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefd40c6540 msgr2=0x7fefd415e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:09:17.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.546+0000 7fefd9954640 1 --2- 172.21.15.125:0/105430246 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefd40c6540 0x7fefd415e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:17.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.546+0000 7fefd9954640 1 -- 172.21.15.125:0/105430246 >> v1:172.21.15.125:6789/0 conn(0x7fefd4069e90 legacy=0x7fefd406a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:17.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.546+0000 7fefd9954640 1 -- 172.21.15.125:0/105430246 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd40c6e70 con 0x7fefd41580e0 2023-10-30T13:09:17.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.546+0000 7fefd8952640 1 -- 172.21.15.125:0/105430246 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666034913 0 0) 0x7fefd40c5810 con 0x7fefd4069e90 2023-10-30T13:09:17.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefd9954640 1 --2- 172.21.15.125:0/105430246 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefd41580e0 0x7fefd4070740 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fefd00097c0 tx=0x7fefd001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=957a54f94edaa17e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:17.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefd8952640 1 -- 172.21.15.125:0/105430246 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd00ab680 con 0x7fefd41580e0 2023-10-30T13:09:17.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefd8952640 1 -- 172.21.15.125:0/105430246 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefd00ab7e0 con 0x7fefd41580e0 2023-10-30T13:09:17.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefd8952640 1 -- 172.21.15.125:0/105430246 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefd00abb70 con 0x7fefd41580e0 2023-10-30T13:09:17.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 -- 172.21.15.125:0/105430246 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefd41580e0 msgr2=0x7fefd4070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:17.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 --2- 172.21.15.125:0/105430246 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefd41580e0 0x7fefd4070740 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7fefd00097c0 tx=0x7fefd001c480 comp rx=0 tx=0).stop 2023-10-30T13:09:17.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 -- 172.21.15.125:0/105430246 shutdown_connections 2023-10-30T13:09:17.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 --2- 172.21.15.125:0/105430246 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefd40c6540 0x7fefd415e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:17.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 --2- 172.21.15.125:0/105430246 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefd41580e0 0x7fefd4070740 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:17.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 -- 172.21.15.125:0/105430246 >> 172.21.15.125:0/105430246 conn(0x7fefd40c7910 msgr2=0x7fefd40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:17.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 -- 172.21.15.125:0/105430246 shutdown_connections 2023-10-30T13:09:17.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 -- 172.21.15.125:0/105430246 wait complete. 2023-10-30T13:09:17.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 Processor -- start 2023-10-30T13:09:17.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 -- start start 2023-10-30T13:09:17.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fefd4069e90 0x7fefd41fa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:17.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefd40c6540 0x7fefd41f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:17.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefd41580e0 0x7fefd4154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:17.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd41f6370 con 0x7fefd40c6540 2023-10-30T13:09:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd40c5010 con 0x7fefd4069e90 2023-10-30T13:09:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.550+0000 7fefdbbb8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd41fa830 con 0x7fefd41580e0 2023-10-30T13:09:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefd9153640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefd40c6540 0x7fefd41f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefd9153640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefd40c6540 0x7fefd41f6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60226/0 (socket says 172.21.15.125:60226) 2023-10-30T13:09:17.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefd9153640 1 -- 172.21.15.125:0/2571576706 learned_addr learned my addr 172.21.15.125:0/2571576706 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:17.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefda155640 1 --2- 172.21.15.125:0/2571576706 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefd41580e0 0x7fefd4154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:17.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefd9954640 1 --2- 172.21.15.125:0/2571576706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fefd4069e90 0x7fefd41fa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:17.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefd9153640 1 -- 172.21.15.125:0/2571576706 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefd41580e0 msgr2=0x7fefd4154e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:17.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefd9153640 1 --2- 172.21.15.125:0/2571576706 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefd41580e0 0x7fefd4154e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:17.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefd9153640 1 -- 172.21.15.125:0/2571576706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fefd4069e90 msgr2=0x7fefd41fa2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:17.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefd9153640 1 --2- 172.21.15.125:0/2571576706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fefd4069e90 0x7fefd41fa2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:17.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefd9153640 1 -- 172.21.15.125:0/2571576706 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd0009420 con 0x7fefd40c6540 2023-10-30T13:09:17.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefd9153640 1 --2- 172.21.15.125:0/2571576706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefd40c6540 0x7fefd41f6d40 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fefc400e960 tx=0x7fefc400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:17.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefca7fc640 1 -- 172.21.15.125:0/2571576706 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc4009690 con 0x7fefd40c6540 2023-10-30T13:09:17.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefca7fc640 1 -- 172.21.15.125:0/2571576706 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefc400ca00 con 0x7fefd40c6540 2023-10-30T13:09:17.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefdbbb8640 1 -- 172.21.15.125:0/2571576706 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd40c47b0 con 0x7fefd40c6540 2023-10-30T13:09:17.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefca7fc640 1 -- 172.21.15.125:0/2571576706 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefc400cd70 con 0x7fefd40c6540 2023-10-30T13:09:17.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefdbbb8640 1 -- 172.21.15.125:0/2571576706 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd41f65d0 con 0x7fefd40c6540 2023-10-30T13:09:17.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefdbbb8640 1 -- 172.21.15.125:0/2571576706 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd404e170 con 0x7fefd40c6540 2023-10-30T13:09:17.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefca7fc640 1 -- 172.21.15.125:0/2571576706 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fefc4013470 con 0x7fefd40c6540 2023-10-30T13:09:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.554+0000 7fefca7fc640 1 --2- 172.21.15.125:0/2571576706 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefb80664d0 0x7fefb8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.558+0000 7fefca7fc640 1 -- 172.21.15.125:0/2571576706 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7fefc4084950 con 0x7fefd40c6540 2023-10-30T13:09:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.558+0000 7fefd9954640 1 --2- 172.21.15.125:0/2571576706 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefb80664d0 0x7fefb8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.558+0000 7fefd9954640 1 --2- 172.21.15.125:0/2571576706 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefb80664d0 0x7fefb8068990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fefd00af000 tx=0x7fefd00a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:17.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.558+0000 7fefca7fc640 1 -- 172.21.15.125:0/2571576706 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fefc4051500 con 0x7fefd40c6540 2023-10-30T13:09:17.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.742+0000 7fefdbbb8640 1 -- 172.21.15.125:0/2571576706 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefd40c4df0 con 0x7fefd40c6540 2023-10-30T13:09:17.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.746+0000 7fefca7fc640 1 -- 172.21.15.125:0/2571576706 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+5757 (secure 0 0 0) 0x7fefc4017090 con 0x7fefd40c6540 2023-10-30T13:09:17.749 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:09:17.749 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:09:12.673410+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:rejoin","state_seq":12,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:09:17.749 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 190 2023-10-30T13:09:17.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.746+0000 7fefdbbb8640 1 -- 172.21.15.125:0/2571576706 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefb80664d0 msgr2=0x7fefb8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:17.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.746+0000 7fefdbbb8640 1 --2- 172.21.15.125:0/2571576706 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefb80664d0 0x7fefb8068990 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fefd00af000 tx=0x7fefd00a7000 comp rx=0 tx=0).stop 2023-10-30T13:09:17.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.746+0000 7fefdbbb8640 1 -- 172.21.15.125:0/2571576706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefd40c6540 msgr2=0x7fefd41f6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:17.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.746+0000 7fefdbbb8640 1 --2- 172.21.15.125:0/2571576706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefd40c6540 0x7fefd41f6d40 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7fefc400e960 tx=0x7fefc400ee20 comp rx=0 tx=0).stop 2023-10-30T13:09:17.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.746+0000 7fefdbbb8640 1 -- 172.21.15.125:0/2571576706 shutdown_connections 2023-10-30T13:09:17.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.746+0000 7fefdbbb8640 1 --2- 172.21.15.125:0/2571576706 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefb80664d0 0x7fefb8068990 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:17.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.746+0000 7fefdbbb8640 1 --2- 172.21.15.125:0/2571576706 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefd41580e0 0x7fefd4154e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:17.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.746+0000 7fefdbbb8640 1 --2- 172.21.15.125:0/2571576706 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefd40c6540 0x7fefd41f6d40 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:17.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.746+0000 7fefdbbb8640 1 --2- 172.21.15.125:0/2571576706 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fefd4069e90 0x7fefd41fa2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:17.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.746+0000 7fefdbbb8640 1 -- 172.21.15.125:0/2571576706 >> 172.21.15.125:0/2571576706 conn(0x7fefd40c7910 msgr2=0x7fefd4152dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:17.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.746+0000 7fefdbbb8640 1 -- 172.21.15.125:0/2571576706 shutdown_connections 2023-10-30T13:09:17.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:17.746+0000 7fefdbbb8640 1 -- 172.21.15.125:0/2571576706 wait complete. 2023-10-30T13:09:17.769 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:09:19.770 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:09:19.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.914+0000 7fbc65c46640 1 Processor -- start 2023-10-30T13:09:19.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.914+0000 7fbc65c46640 1 -- start start 2023-10-30T13:09:19.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.914+0000 7fbc65c46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc601538a0 0x7fbc6015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:19.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.914+0000 7fbc65c46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbc600cf160 0x7fbc600cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:19.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.914+0000 7fbc65c46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc600cb800 con 0x7fbc600ca0e0 2023-10-30T13:09:19.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.914+0000 7fbc65c46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc6015dc30 con 0x7fbc600cf160 2023-10-30T13:09:19.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.914+0000 7fbc65c46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc600cb330 con 0x7fbc601538a0 2023-10-30T13:09:19.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.914+0000 7fbc5ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbc600cf160 0x7fbc600cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:19.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.918+0000 7fbc5ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbc600cf160 0x7fbc600cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60230/0 (socket says 172.21.15.125:60230) 2023-10-30T13:09:19.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.918+0000 7fbc5ffff640 1 -- 172.21.15.125:0/561665660 learned_addr learned my addr 172.21.15.125:0/561665660 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:19.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.918+0000 7fbc5f7fe640 1 --2- 172.21.15.125:0/561665660 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc601538a0 0x7fbc6015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:19.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.918+0000 7fbc5ffff640 1 -- 172.21.15.125:0/561665660 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc601538a0 msgr2=0x7fbc6015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:09:19.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.918+0000 7fbc5ffff640 1 --2- 172.21.15.125:0/561665660 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc601538a0 0x7fbc6015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:19.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.918+0000 7fbc5ffff640 1 -- 172.21.15.125:0/561665660 >> v1:172.21.15.136:6789/0 conn(0x7fbc600ca0e0 legacy=0x7fbc6015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:19.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.918+0000 7fbc5ffff640 1 -- 172.21.15.125:0/561665660 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc600cad70 con 0x7fbc600cf160 2023-10-30T13:09:19.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.922+0000 7fbc5ffff640 1 --2- 172.21.15.125:0/561665660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc600cf160 0x7fbc600cf590 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009e50 tx=0x7fbc5801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeffd3acff487e4c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:19.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.922+0000 7fbc5e7fc640 1 -- 172.21.15.125:0/561665660 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc580af400 con 0x7fbc600cf160 2023-10-30T13:09:19.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.922+0000 7fbc5e7fc640 1 -- 172.21.15.125:0/561665660 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc580ab050 con 0x7fbc600cf160 2023-10-30T13:09:19.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.922+0000 7fbc5e7fc640 1 -- 172.21.15.125:0/561665660 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc580af780 con 0x7fbc600cf160 2023-10-30T13:09:19.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.922+0000 7fbc65c46640 1 -- 172.21.15.125:0/561665660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc600cf160 msgr2=0x7fbc600cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:19.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.922+0000 7fbc65c46640 1 --2- 172.21.15.125:0/561665660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc600cf160 0x7fbc600cf590 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009e50 tx=0x7fbc5801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:09:19.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.922+0000 7fbc65c46640 1 -- 172.21.15.125:0/561665660 shutdown_connections 2023-10-30T13:09:19.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.922+0000 7fbc65c46640 1 --2- 172.21.15.125:0/561665660 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc600cf160 0x7fbc600cf590 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:19.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.922+0000 7fbc65c46640 1 --2- 172.21.15.125:0/561665660 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc601538a0 0x7fbc6015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:19.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.922+0000 7fbc65c46640 1 -- 172.21.15.125:0/561665660 >> 172.21.15.125:0/561665660 conn(0x7fbc6006bfe0 msgr2=0x7fbc6006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:19.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.922+0000 7fbc65c46640 1 -- 172.21.15.125:0/561665660 shutdown_connections 2023-10-30T13:09:19.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.922+0000 7fbc65c46640 1 -- 172.21.15.125:0/561665660 wait complete. 2023-10-30T13:09:19.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.922+0000 7fbc65c46640 1 Processor -- start 2023-10-30T13:09:19.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc65c46640 1 -- start start 2023-10-30T13:09:19.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc65c46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc600ca0e0 0x7fbc6015a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:19.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc65c46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc600cf160 0x7fbc6015b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:19.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc65c46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc601538a0 0x7fbc6015b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:19.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc65c46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc601e3960 con 0x7fbc600ca0e0 2023-10-30T13:09:19.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc65c46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc601e4750 con 0x7fbc600cf160 2023-10-30T13:09:19.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc65c46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc601e3ba0 con 0x7fbc601538a0 2023-10-30T13:09:19.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc5f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc600ca0e0 0x7fbc6015a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:19.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc5f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc600ca0e0 0x7fbc6015a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60232/0 (socket says 172.21.15.125:60232) 2023-10-30T13:09:19.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc5f7fe640 1 -- 172.21.15.125:0/3031627319 learned_addr learned my addr 172.21.15.125:0/3031627319 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:19.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc5effd640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc600cf160 0x7fbc6015b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:19.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc5ffff640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc601538a0 0x7fbc6015b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:19.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc5f7fe640 1 -- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc601538a0 msgr2=0x7fbc6015b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:19.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc5f7fe640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc601538a0 0x7fbc6015b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:19.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc5f7fe640 1 -- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc600cf160 msgr2=0x7fbc6015b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:19.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc5f7fe640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc600cf160 0x7fbc6015b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:19.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc5f7fe640 1 -- 172.21.15.125:0/3031627319 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc58009b30 con 0x7fbc600ca0e0 2023-10-30T13:09:19.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc5ffff640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc601538a0 0x7fbc6015b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:09:19.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc5f7fe640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc600ca0e0 0x7fbc6015a8b0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000e870 tx=0x7fbc5000ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:19.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc47fff640 1 -- 172.21.15.125:0/3031627319 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc50009510 con 0x7fbc600ca0e0 2023-10-30T13:09:19.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc47fff640 1 -- 172.21.15.125:0/3031627319 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc5000eec0 con 0x7fbc600ca0e0 2023-10-30T13:09:19.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc65c46640 1 -- 172.21.15.125:0/3031627319 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc601e3430 con 0x7fbc600ca0e0 2023-10-30T13:09:19.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc47fff640 1 -- 172.21.15.125:0/3031627319 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5000cb40 con 0x7fbc600ca0e0 2023-10-30T13:09:19.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.926+0000 7fbc65c46640 1 -- 172.21.15.125:0/3031627319 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc601e3db0 con 0x7fbc600ca0e0 2023-10-30T13:09:19.935 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.930+0000 7fbc65c46640 1 -- 172.21.15.125:0/3031627319 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc600b2a50 con 0x7fbc600ca0e0 2023-10-30T13:09:19.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.930+0000 7fbc47fff640 1 -- 172.21.15.125:0/3031627319 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbc5000cca0 con 0x7fbc600ca0e0 2023-10-30T13:09:19.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.930+0000 7fbc47fff640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbc4c0664a0 0x7fbc4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:19.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.930+0000 7fbc47fff640 1 -- 172.21.15.125:0/3031627319 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7fbc50084730 con 0x7fbc600ca0e0 2023-10-30T13:09:19.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.930+0000 7fbc5effd640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbc4c0664a0 0x7fbc4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:19.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.930+0000 7fbc5effd640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbc4c0664a0 0x7fbc4c068960 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fbc6015a130 tx=0x7fbc54009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:19.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:19.934+0000 7fbc47fff640 1 -- 172.21.15.125:0/3031627319 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbc50051480 con 0x7fbc600ca0e0 2023-10-30T13:09:20.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.134+0000 7fbc65c46640 1 -- 172.21.15.125:0/3031627319 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc6004e170 con 0x7fbc600ca0e0 2023-10-30T13:09:20.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.134+0000 7fbc47fff640 1 -- 172.21.15.125:0/3031627319 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+5757 (secure 0 0 0) 0x7fbc50054aa0 con 0x7fbc600ca0e0 2023-10-30T13:09:20.140 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:09:20.140 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:09:17.957726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:active","state_seq":14,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:09:20.141 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 191 2023-10-30T13:09:20.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.138+0000 7fbc65c46640 1 -- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbc4c0664a0 msgr2=0x7fbc4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:20.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.138+0000 7fbc65c46640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbc4c0664a0 0x7fbc4c068960 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fbc6015a130 tx=0x7fbc54009040 comp rx=0 tx=0).stop 2023-10-30T13:09:20.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.138+0000 7fbc65c46640 1 -- 172.21.15.125:0/3031627319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc600ca0e0 msgr2=0x7fbc6015a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:20.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.138+0000 7fbc65c46640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc600ca0e0 0x7fbc6015a8b0 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fbc5000e870 tx=0x7fbc5000ec70 comp rx=0 tx=0).stop 2023-10-30T13:09:20.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.138+0000 7fbc65c46640 1 -- 172.21.15.125:0/3031627319 shutdown_connections 2023-10-30T13:09:20.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.138+0000 7fbc65c46640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbc4c0664a0 0x7fbc4c068960 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.138+0000 7fbc65c46640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc601538a0 0x7fbc6015b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.138+0000 7fbc65c46640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc600cf160 0x7fbc6015b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.138+0000 7fbc65c46640 1 --2- 172.21.15.125:0/3031627319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc600ca0e0 0x7fbc6015a8b0 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.138+0000 7fbc65c46640 1 -- 172.21.15.125:0/3031627319 >> 172.21.15.125:0/3031627319 conn(0x7fbc6006bfe0 msgr2=0x7fbc600c4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:20.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.138+0000 7fbc65c46640 1 -- 172.21.15.125:0/3031627319 shutdown_connections 2023-10-30T13:09:20.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.138+0000 7fbc65c46640 1 -- 172.21.15.125:0/3031627319 wait complete. 2023-10-30T13:09:20.160 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=2, replacing gid=6017 2023-10-30T13:09:20.161 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.f 2023-10-30T13:09:20.161 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T13:09:20.161 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T13:09:20.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T13:09:20.164 INFO:tasks.ceph.mds.f:Started 2023-10-30T13:09:20.164 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:09:20.240 INFO:tasks.ceph.mds.f.smithi136.stdout:starting mds.f at 2023-10-30T13:09:20.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.298+0000 7f1595002640 1 Processor -- start 2023-10-30T13:09:20.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.298+0000 7f1595002640 1 -- start start 2023-10-30T13:09:20.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.298+0000 7f1595002640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1590071800 0x7f159015a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:20.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.298+0000 7f1595002640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15900c91b0 0x7f1590071be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:20.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.298+0000 7f1595002640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1590072120 0x7f15900c6550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:20.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.298+0000 7f1595002640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f159006bce0 con 0x7f15900c91b0 2023-10-30T13:09:20.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.298+0000 7f1595002640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f159006ac40 con 0x7f1590072120 2023-10-30T13:09:20.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.298+0000 7f1595002640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f159006a030 con 0x7f1590071800 2023-10-30T13:09:20.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.302+0000 7f158659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15900c91b0 0x7f1590071be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:20.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.302+0000 7f158659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f15900c91b0 0x7f1590071be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60246/0 (socket says 172.21.15.125:60246) 2023-10-30T13:09:20.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.302+0000 7f158659c640 1 -- 172.21.15.125:0/1289843676 learned_addr learned my addr 172.21.15.125:0/1289843676 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:20.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.302+0000 7f158ed9d640 1 --2- 172.21.15.125:0/1289843676 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1590071800 0x7f159015a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:20.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.302+0000 7f158f59e640 1 --2- 172.21.15.125:0/1289843676 >> v2:172.21.15.136:3300/0 conn(0x7f1590072120 0x7f15900c6550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:20.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.302+0000 7f158659c640 1 -- 172.21.15.125:0/1289843676 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1590071800 msgr2=0x7f159015a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:09:20.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.302+0000 7f158659c640 1 --2- 172.21.15.125:0/1289843676 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1590071800 0x7f159015a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.302+0000 7f158659c640 1 -- 172.21.15.125:0/1289843676 >> v2:172.21.15.136:3300/0 conn(0x7f1590072120 msgr2=0x7f15900c6550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:09:20.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.302+0000 7f158659c640 1 --2- 172.21.15.125:0/1289843676 >> v2:172.21.15.136:3300/0 conn(0x7f1590072120 0x7f15900c6550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.302+0000 7f158659c640 1 -- 172.21.15.125:0/1289843676 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f159006a310 con 0x7f15900c91b0 2023-10-30T13:09:20.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.302+0000 7f158659c640 1 --2- 172.21.15.125:0/1289843676 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15900c91b0 0x7f1590071be0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f1590070450 tx=0x7f157801c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14211e494412b40c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:20.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.302+0000 7f158e59c640 1 -- 172.21.15.125:0/1289843676 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15780a5650 con 0x7f15900c91b0 2023-10-30T13:09:20.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.302+0000 7f158e59c640 1 -- 172.21.15.125:0/1289843676 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15780a57b0 con 0x7f15900c91b0 2023-10-30T13:09:20.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.302+0000 7f158e59c640 1 -- 172.21.15.125:0/1289843676 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15780a5b40 con 0x7f15900c91b0 2023-10-30T13:09:20.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 -- 172.21.15.125:0/1289843676 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15900c91b0 msgr2=0x7f1590071be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:20.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 --2- 172.21.15.125:0/1289843676 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15900c91b0 0x7f1590071be0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f1590070450 tx=0x7f157801c9d0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 -- 172.21.15.125:0/1289843676 shutdown_connections 2023-10-30T13:09:20.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 --2- 172.21.15.125:0/1289843676 >> v2:172.21.15.136:3300/0 conn(0x7f1590072120 0x7f15900c6550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 --2- 172.21.15.125:0/1289843676 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f15900c91b0 0x7f1590071be0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 --2- 172.21.15.125:0/1289843676 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1590071800 0x7f159015a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 -- 172.21.15.125:0/1289843676 >> 172.21.15.125:0/1289843676 conn(0x7f15900c8650 msgr2=0x7f15900c8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:20.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 -- 172.21.15.125:0/1289843676 shutdown_connections 2023-10-30T13:09:20.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 -- 172.21.15.125:0/1289843676 wait complete. 2023-10-30T13:09:20.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 Processor -- start 2023-10-30T13:09:20.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 -- start start 2023-10-30T13:09:20.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1590071800 0x7f15900c3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:20.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1590072120 0x7f1590158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:20.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15900c91b0 0x7f1590159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:20.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15900c3a40 con 0x7f1590072120 2023-10-30T13:09:20.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15900c3da0 con 0x7f1590071800 2023-10-30T13:09:20.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f1595002640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1590153f10 con 0x7f15900c91b0 2023-10-30T13:09:20.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f158659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1590072120 0x7f1590158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:20.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.306+0000 7f158659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1590072120 0x7f1590158740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60262/0 (socket says 172.21.15.125:60262) 2023-10-30T13:09:20.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f158659c640 1 -- 172.21.15.125:0/1207609210 learned_addr learned my addr 172.21.15.125:0/1207609210 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:20.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f158ed9d640 1 --2- 172.21.15.125:0/1207609210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1590071800 0x7f15900c3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:20.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f158f59e640 1 --2- 172.21.15.125:0/1207609210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15900c91b0 0x7f1590159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:20.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f158659c640 1 -- 172.21.15.125:0/1207609210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15900c91b0 msgr2=0x7f1590159870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:20.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f158659c640 1 --2- 172.21.15.125:0/1207609210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15900c91b0 0x7f1590159870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f158659c640 1 -- 172.21.15.125:0/1207609210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1590071800 msgr2=0x7f15900c3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:20.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f158659c640 1 --2- 172.21.15.125:0/1207609210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1590071800 0x7f15900c3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f158659c640 1 -- 172.21.15.125:0/1207609210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1578009b30 con 0x7f1590072120 2023-10-30T13:09:20.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f158659c640 1 --2- 172.21.15.125:0/1207609210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1590072120 0x7f1590158740 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f1578009040 tx=0x7f1578019770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:20.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f1587fff640 1 -- 172.21.15.125:0/1207609210 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15780be070 con 0x7f1590072120 2023-10-30T13:09:20.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f1587fff640 1 -- 172.21.15.125:0/1207609210 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1578009510 con 0x7f1590072120 2023-10-30T13:09:20.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f1587fff640 1 -- 172.21.15.125:0/1207609210 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15780b86a0 con 0x7f1590072120 2023-10-30T13:09:20.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f1595002640 1 -- 172.21.15.125:0/1207609210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f159014d9d0 con 0x7f1590072120 2023-10-30T13:09:20.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f1595002640 1 -- 172.21.15.125:0/1207609210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1590157f40 con 0x7f1590072120 2023-10-30T13:09:20.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f1595002640 1 -- 172.21.15.125:0/1207609210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f159006ef10 con 0x7f1590072120 2023-10-30T13:09:20.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f1587fff640 1 -- 172.21.15.125:0/1207609210 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f15780b8800 con 0x7f1590072120 2023-10-30T13:09:20.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f1587fff640 1 --2- 172.21.15.125:0/1207609210 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15700664a0 0x7f1570068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:20.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.310+0000 7f1587fff640 1 -- 172.21.15.125:0/1207609210 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7f157811ecd0 con 0x7f1590072120 2023-10-30T13:09:20.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.314+0000 7f158ed9d640 1 --2- 172.21.15.125:0/1207609210 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15700664a0 0x7f1570068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:20.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.314+0000 7f158ed9d640 1 --2- 172.21.15.125:0/1207609210 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15700664a0 0x7f1570068960 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f1580009a40 tx=0x7f1580009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:20.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.314+0000 7f1587fff640 1 -- 172.21.15.125:0/1207609210 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f15780eb980 con 0x7f1590072120 2023-10-30T13:09:20.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.498+0000 7f1595002640 1 -- 172.21.15.125:0/1207609210 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f159004e170 con 0x7f1590072120 2023-10-30T13:09:20.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.498+0000 7f1587fff640 1 -- 172.21.15.125:0/1207609210 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+5757 (secure 0 0 0) 0x7f15780eefa0 con 0x7f1590072120 2023-10-30T13:09:20.501 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:09:20.502 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:09:17.957726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:active","state_seq":14,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:09:20.502 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 191 2023-10-30T13:09:20.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.498+0000 7f1595002640 1 -- 172.21.15.125:0/1207609210 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15700664a0 msgr2=0x7f1570068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:20.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.498+0000 7f1595002640 1 --2- 172.21.15.125:0/1207609210 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15700664a0 0x7f1570068960 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f1580009a40 tx=0x7f1580009190 comp rx=0 tx=0).stop 2023-10-30T13:09:20.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.498+0000 7f1595002640 1 -- 172.21.15.125:0/1207609210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1590072120 msgr2=0x7f1590158740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:20.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.498+0000 7f1595002640 1 --2- 172.21.15.125:0/1207609210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1590072120 0x7f1590158740 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f1578009040 tx=0x7f1578019770 comp rx=0 tx=0).stop 2023-10-30T13:09:20.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.498+0000 7f1595002640 1 -- 172.21.15.125:0/1207609210 shutdown_connections 2023-10-30T13:09:20.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.498+0000 7f1595002640 1 --2- 172.21.15.125:0/1207609210 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f15700664a0 0x7f1570068960 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.498+0000 7f1595002640 1 --2- 172.21.15.125:0/1207609210 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f15900c91b0 0x7f1590159870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.498+0000 7f1595002640 1 --2- 172.21.15.125:0/1207609210 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1590072120 0x7f1590158740 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.498+0000 7f1595002640 1 --2- 172.21.15.125:0/1207609210 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1590071800 0x7f15900c3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.498+0000 7f1595002640 1 -- 172.21.15.125:0/1207609210 >> 172.21.15.125:0/1207609210 conn(0x7f15900c8650 msgr2=0x7f15900c77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.498+0000 7f1595002640 1 -- 172.21.15.125:0/1207609210 shutdown_connections 2023-10-30T13:09:20.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:20.498+0000 7f1595002640 1 -- 172.21.15.125:0/1207609210 wait complete. 2023-10-30T13:09:20.522 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T13:09:22.523 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:09:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.666+0000 7fafb4491640 1 Processor -- start 2023-10-30T13:09:22.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.666+0000 7fafb4491640 1 -- start start 2023-10-30T13:09:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.670+0000 7fafb4491640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fafac15d6f0 0x7fafac15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.670+0000 7fafb4491640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafac0cb800 con 0x7fafac1538a0 2023-10-30T13:09:22.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.670+0000 7fafb4491640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafac0cb330 con 0x7fafac0cf160 2023-10-30T13:09:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.670+0000 7fafb4491640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafac0cad70 con 0x7fafac15d6f0 2023-10-30T13:09:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.670+0000 7fafb1a2c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fafac15d6f0 0x7fafac15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.670+0000 7fafb222d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fafac1538a0 0x7fafac0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:42602/0 (socket says 172.21.15.125:42602) 2023-10-30T13:09:22.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.670+0000 7fafb222d640 1 -- 172.21.15.125:0/788949999 learned_addr learned my addr 172.21.15.125:0/788949999 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:09:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.670+0000 7fafb1a2c640 1 -- 172.21.15.125:0/788949999 >> v1:172.21.15.136:6789/0 conn(0x7fafac0cf160 legacy=0x7fafac164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.670+0000 7fafb1a2c640 1 -- 172.21.15.125:0/788949999 >> v1:172.21.15.125:6789/0 conn(0x7fafac1538a0 legacy=0x7fafac0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.670+0000 7fafb122b640 1 -- 172.21.15.125:0/788949999 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627757765 0 0) 0x7fafac0cb800 con 0x7fafac1538a0 2023-10-30T13:09:22.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.670+0000 7fafb1a2c640 1 -- 172.21.15.125:0/788949999 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac0cb050 con 0x7fafac15d6f0 2023-10-30T13:09:22.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.674+0000 7fafb1a2c640 1 --2- 172.21.15.125:0/788949999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac15d6f0 0x7fafac15a090 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0097c0 tx=0x7faf9c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=310490210113421e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:22.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.674+0000 7fafb122b640 1 -- 172.21.15.125:0/788949999 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9c0a8970 con 0x7fafac15d6f0 2023-10-30T13:09:22.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.674+0000 7fafb122b640 1 -- 172.21.15.125:0/788949999 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9c0a8ad0 con 0x7fafac15d6f0 2023-10-30T13:09:22.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.674+0000 7fafb122b640 1 -- 172.21.15.125:0/788949999 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9c0a8e60 con 0x7fafac15d6f0 2023-10-30T13:09:22.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.674+0000 7fafb4491640 1 -- 172.21.15.125:0/788949999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac15d6f0 msgr2=0x7fafac15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.674+0000 7fafb4491640 1 --2- 172.21.15.125:0/788949999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac15d6f0 0x7fafac15a090 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7faf9c0097c0 tx=0x7faf9c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:09:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.674+0000 7fafb4491640 1 -- 172.21.15.125:0/788949999 shutdown_connections 2023-10-30T13:09:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.674+0000 7fafb4491640 1 --2- 172.21.15.125:0/788949999 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac15d6f0 0x7fafac15a090 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:22.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.674+0000 7fafb4491640 1 -- 172.21.15.125:0/788949999 >> 172.21.15.125:0/788949999 conn(0x7fafac06bfe0 msgr2=0x7fafac06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:22.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.674+0000 7fafb4491640 1 -- 172.21.15.125:0/788949999 shutdown_connections 2023-10-30T13:09:22.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.674+0000 7fafb4491640 1 -- 172.21.15.125:0/788949999 wait complete. 2023-10-30T13:09:22.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb4491640 1 Processor -- start 2023-10-30T13:09:22.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb4491640 1 -- start start 2023-10-30T13:09:22.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb4491640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafac0cf160 0x7fafac1587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:22.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb4491640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fafac1538a0 0x7fafac15b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:22.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb4491640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac15d6f0 0x7fafac1e3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:22.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb4491640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac15c500 con 0x7fafac15d6f0 2023-10-30T13:09:22.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb4491640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafac14d770 con 0x7fafac0cf160 2023-10-30T13:09:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb4491640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafac1e2f50 con 0x7fafac1538a0 2023-10-30T13:09:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb2a2e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac15d6f0 0x7fafac1e3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb2a2e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac15d6f0 0x7fafac1e3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46594/0 (socket says 172.21.15.125:46594) 2023-10-30T13:09:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb2a2e640 1 -- 172.21.15.125:0/3461081426 learned_addr learned my addr 172.21.15.125:0/3461081426 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:09:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb1a2c640 1 --2- 172.21.15.125:0/3461081426 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fafac1538a0 0x7fafac15b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb222d640 1 --2- 172.21.15.125:0/3461081426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafac0cf160 0x7fafac1587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:22.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb2a2e640 1 -- 172.21.15.125:0/3461081426 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fafac1538a0 msgr2=0x7fafac15b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:22.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb2a2e640 1 --2- 172.21.15.125:0/3461081426 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fafac1538a0 0x7fafac15b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:22.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb2a2e640 1 -- 172.21.15.125:0/3461081426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafac0cf160 msgr2=0x7fafac1587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:22.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb2a2e640 1 --2- 172.21.15.125:0/3461081426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafac0cf160 0x7fafac1587d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:22.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb2a2e640 1 -- 172.21.15.125:0/3461081426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9c009420 con 0x7fafac15d6f0 2023-10-30T13:09:22.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb2a2e640 1 --2- 172.21.15.125:0/3461081426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac15d6f0 0x7fafac1e3990 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fafa400c730 tx=0x7fafa400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:22.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafa2ffd640 1 -- 172.21.15.125:0/3461081426 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4003b80 con 0x7fafac15d6f0 2023-10-30T13:09:22.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafa2ffd640 1 -- 172.21.15.125:0/3461081426 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa4003ce0 con 0x7fafac15d6f0 2023-10-30T13:09:22.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafa2ffd640 1 -- 172.21.15.125:0/3461081426 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa4005310 con 0x7fafac15d6f0 2023-10-30T13:09:22.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb4491640 1 -- 172.21.15.125:0/3461081426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafac15c860 con 0x7fafac15d6f0 2023-10-30T13:09:22.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.678+0000 7fafb4491640 1 -- 172.21.15.125:0/3461081426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafac0c6a70 con 0x7fafac15d6f0 2023-10-30T13:09:22.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.682+0000 7fafb4491640 1 -- 172.21.15.125:0/3461081426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafac153c80 con 0x7fafac15d6f0 2023-10-30T13:09:22.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.686+0000 7fafa2ffd640 1 -- 172.21.15.125:0/3461081426 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fafa4005680 con 0x7fafac15d6f0 2023-10-30T13:09:22.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.686+0000 7fafa2ffd640 1 --2- 172.21.15.125:0/3461081426 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf900664a0 0x7faf90068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:09:22.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.686+0000 7fafa2ffd640 1 -- 172.21.15.125:0/3461081426 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7fafa4085070 con 0x7fafac15d6f0 2023-10-30T13:09:22.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.686+0000 7fafb222d640 1 --2- 172.21.15.125:0/3461081426 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf900664a0 0x7faf90068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:09:22.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.686+0000 7fafb222d640 1 --2- 172.21.15.125:0/3461081426 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf900664a0 0x7faf90068960 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fafa8000f70 tx=0x7fafa8007240 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:09:22.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.690+0000 7fafa2ffd640 1 -- 172.21.15.125:0/3461081426 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fafa4051d10 con 0x7fafac15d6f0 2023-10-30T13:09:22.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.870+0000 7fafb4491640 1 -- 172.21.15.125:0/3461081426 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafac15a580 con 0x7fafac15d6f0 2023-10-30T13:09:22.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.874+0000 7fafa2ffd640 1 -- 172.21.15.125:0/3461081426 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+6552 (secure 0 0 0) 0x7fafa4055330 con 0x7fafac15d6f0 2023-10-30T13:09:22.876 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:09:22.876 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8443,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:09:17.957726+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:active","state_seq":14,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:09:22.877 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 192 2023-10-30T13:09:22.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.874+0000 7fafb4491640 1 -- 172.21.15.125:0/3461081426 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf900664a0 msgr2=0x7faf90068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:22.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.874+0000 7fafb4491640 1 --2- 172.21.15.125:0/3461081426 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf900664a0 0x7faf90068960 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fafa8000f70 tx=0x7fafa8007240 comp rx=0 tx=0).stop 2023-10-30T13:09:22.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.874+0000 7fafb4491640 1 -- 172.21.15.125:0/3461081426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac15d6f0 msgr2=0x7fafac1e3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:09:22.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.874+0000 7fafb4491640 1 --2- 172.21.15.125:0/3461081426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac15d6f0 0x7fafac1e3990 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fafa400c730 tx=0x7fafa400cbf0 comp rx=0 tx=0).stop 2023-10-30T13:09:22.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.874+0000 7fafb4491640 1 -- 172.21.15.125:0/3461081426 shutdown_connections 2023-10-30T13:09:22.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.874+0000 7fafb4491640 1 --2- 172.21.15.125:0/3461081426 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faf900664a0 0x7faf90068960 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:22.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.874+0000 7fafb4491640 1 --2- 172.21.15.125:0/3461081426 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fafac15d6f0 0x7fafac1e3990 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:22.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.874+0000 7fafb4491640 1 --2- 172.21.15.125:0/3461081426 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fafac1538a0 0x7fafac15b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:22.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.874+0000 7fafb4491640 1 --2- 172.21.15.125:0/3461081426 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fafac0cf160 0x7fafac1587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:09:22.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.874+0000 7fafb4491640 1 -- 172.21.15.125:0/3461081426 >> 172.21.15.125:0/3461081426 conn(0x7fafac06bfe0 msgr2=0x7fafac06c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:09:22.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.874+0000 7fafb4491640 1 -- 172.21.15.125:0/3461081426 shutdown_connections 2023-10-30T13:09:22.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:09:22.874+0000 7fafb4491640 1 -- 172.21.15.125:0/3461081426 wait complete. 2023-10-30T13:09:22.896 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T13:09:22.896 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 114 secs before thrashing 2023-10-30T13:09:23.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:09:23.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:09:24.593 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 5% (11/204) Updating files: 6% (13/204) Updating files: 7% (15/204) Updating files: 8% (17/204) Updating files: 9% (19/204) Updating files: 10% (21/204) Updating files: 11% (23/204) Updating files: 12% (25/204) Updating files: 13% (27/204) Updating files: 14% (29/204) Updating files: 15% (31/204) Updating files: 16% (33/204) Updating files: 17% (35/204) Updating files: 18% (37/204) Updating files: 19% (39/204) Updating files: 20% (41/204) Updating files: 21% (43/204) Updating files: 22% (45/204) Updating files: 23% (47/204) Updating files: 24% (49/204) Updating files: 24% (50/204) Updating files: 25% (51/204) Updating files: 26% (54/204) Updating files: 27% (56/204) Updating files: 28% (58/204) Updating files: 29% (60/204) Updating files: 30% (62/204) Updating files: 31% (64/204) Updating files: 32% (66/204) Updating files: 33% (68/204) Updating files: 34% (70/204) Updating files: 35% (72/204) Updating files: 36% (74/204) Updating files: 37% (76/204) Updating files: 38% (78/204) Updating files: 39% (80/204) Updating files: 40% (82/204) Updating files: 41% (84/204) Updating files: 42% (86/204) Updating files: 42% (87/204) Updating files: 43% (88/204) Updating files: 44% (90/204) Updating files: 45% (92/204) Updating files: 46% (94/204) Updating files: 47% (96/204) Updating files: 48% (98/204) Updating files: 49% (100/204) Updating files: 50% (102/204) Updating files: 51% (105/204) Updating files: 52% (107/204) Updating files: 53% (109/204) Updating files: 54% (111/204) Updating files: 55% (113/204) Updating files: 56% (115/204) Updating files: 57% (117/204) Updating files: 58% (119/204) Updating files: 59% (121/204) Updating files: 60% (123/204) Updating files: 61% (125/204) Updating files: 62% (127/204) Updating files: 63% (129/204) Updating files: 64% (131/204) Updating files: 65% (133/204) Updating files: 66% (135/204) Updating files: 67% (137/204) Updating files: 68% (139/204) Updating files: 69% (141/204) Updating files: 70% (143/204) Updating files: 71% (145/204) Updating files: 72% (147/204) Updating files: 73% (149/204) Updating files: 74% (151/204) Updating files: 75% (153/204) Updating files: 76% (156/204) Updating files: 77% (158/204) Updating files: 78% (160/204) Updating files: 79% (162/204) Updating files: 80% (164/204) Updating files: 81% (166/204) Updating files: 82% (168/204) Updating files: 83% (170/204) Updating files: 84% (172/204) Updating files: 84% (173/204) Updating files: 85% (174/204) Updating files: 86% (176/204) Updating files: 87% (178/204) Updating files: 88% (180/204) Updating files: 89% (182/204) Updating files: 90% (184/204) Updating files: 91% (186/204) Updating files: 92% (188/204) Updating files: 93% (190/204) Updating files: 94% (192/204) Updating files: 95% (194/204) Updating files: 96% (196/204) Updating files: 97% (198/204) Updating files: 98% (200/204) Updating files: 99% (202/204) Updating files: 100% (204/204) Updating files: 100% (204/204), done. 2023-10-30T13:09:24.797 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 14085f42ddd 0.82 2023-10-30T13:09:24.886 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.82 2023-10-30T13:09:25.423 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 83 -eq 48 ] 2023-10-30T13:09:25.424 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.83 2023-10-30T13:09:25.424 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.83 2023-10-30T13:09:25.424 INFO:tasks.workunit.client.0.smithi125.stdout:v0.83 2023-10-30T13:09:25.424 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.83 2023-10-30T13:09:43.444 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 39% (86/218) Updating files: 40% (88/218) Updating files: 41% (90/218) Updating files: 42% (92/218) Updating files: 43% (94/218) Updating files: 44% (96/218) Updating files: 45% (99/218) Updating files: 46% (101/218) Updating files: 47% (103/218) Updating files: 48% (105/218) Updating files: 49% (107/218) Updating files: 50% (109/218) Updating files: 51% (112/218) Updating files: 52% (114/218) Updating files: 53% (116/218) Updating files: 54% (118/218) Updating files: 55% (120/218) Updating files: 56% (123/218) Updating files: 57% (125/218) Updating files: 58% (127/218) Updating files: 59% (129/218) Updating files: 60% (131/218) Updating files: 61% (133/218) Updating files: 62% (136/218) Updating files: 63% (138/218) Updating files: 64% (140/218) Updating files: 65% (142/218) Updating files: 66% (144/218) Updating files: 67% (147/218) Updating files: 68% (149/218) Updating files: 69% (151/218) Updating files: 70% (153/218) Updating files: 71% (155/218) Updating files: 72% (157/218) Updating files: 73% (160/218) Updating files: 74% (162/218) Updating files: 75% (164/218) Updating files: 76% (166/218) Updating files: 77% (168/218) Updating files: 78% (171/218) Updating files: 79% (173/218) Updating files: 80% (175/218) Updating files: 81% (177/218) Updating files: 82% (179/218) Updating files: 83% (181/218) Updating files: 84% (184/218) Updating files: 85% (186/218) Updating files: 86% (188/218) Updating files: 87% (190/218) Updating files: 88% (192/218) Updating files: 89% (195/218) Updating files: 90% (197/218) Updating files: 91% (199/218) Updating files: 92% (201/218) Updating files: 93% (203/218) Updating files: 94% (205/218) Updating files: 95% (208/218) Updating files: 96% (210/218) Updating files: 97% (212/218) Updating files: 98% (214/218) Updating files: 99% (216/218) Updating files: 100% (218/218) Updating files: 100% (218/218), done. 2023-10-30T13:09:43.605 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 78ff1f0a5df 0.83 2023-10-30T13:09:43.610 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.83 2023-10-30T13:09:43.685 INFO:tasks.workunit.client.0.smithi125.stdout:v0.84 2023-10-30T13:09:43.685 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 84 -eq 48 ] 2023-10-30T13:09:43.685 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.84 2023-10-30T13:09:43.686 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.84 2023-10-30T13:09:43.686 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.84 2023-10-30T13:09:50.963 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 31% (113/356) Updating files: 32% (114/356) Updating files: 33% (118/356) Updating files: 34% (122/356) Updating files: 35% (125/356) Updating files: 36% (129/356) Updating files: 37% (132/356) Updating files: 38% (136/356) Updating files: 39% (139/356) Updating files: 40% (143/356) Updating files: 41% (146/356) Updating files: 42% (150/356) Updating files: 43% (154/356) Updating files: 44% (157/356) Updating files: 45% (161/356) Updating files: 46% (164/356) Updating files: 47% (168/356) Updating files: 48% (171/356) Updating files: 49% (175/356) Updating files: 50% (178/356) Updating files: 51% (182/356) Updating files: 52% (186/356) Updating files: 53% (189/356) Updating files: 54% (193/356) Updating files: 55% (196/356) Updating files: 56% (200/356) Updating files: 57% (203/356) Updating files: 58% (207/356) Updating files: 59% (211/356) Updating files: 60% (214/356) Updating files: 61% (218/356) Updating files: 62% (221/356) Updating files: 63% (225/356) Updating files: 64% (228/356) Updating files: 65% (232/356) Updating files: 66% (235/356) Updating files: 67% (239/356) Updating files: 68% (243/356) Updating files: 69% (246/356) Updating files: 70% (250/356) Updating files: 71% (253/356) Updating files: 71% (254/356) Updating files: 72% (257/356) Updating files: 73% (260/356) Updating files: 74% (264/356) Updating files: 75% (267/356) Updating files: 76% (271/356) Updating files: 77% (275/356) Updating files: 78% (278/356) Updating files: 79% (282/356) Updating files: 80% (285/356) Updating files: 81% (289/356) Updating files: 82% (292/356) Updating files: 83% (296/356) Updating files: 84% (300/356) Updating files: 85% (303/356) Updating files: 86% (307/356) Updating files: 87% (310/356) Updating files: 88% (314/356) Updating files: 89% (317/356) Updating files: 90% (321/356) Updating files: 91% (324/356) Updating files: 92% (328/356) Updating files: 93% (332/356) Updating files: 94% (335/356) Updating files: 95% (339/356) Updating files: 96% (342/356) Updating files: 97% (346/356) Updating files: 98% (349/356) Updating files: 99% (353/356) Updating files: 100% (356/356) Updating files: 100% (356/356), done. 2023-10-30T13:09:51.067 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 8336f81c5c7 0.84 2023-10-30T13:09:51.077 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.84 2023-10-30T13:09:52.089 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 85 -eq 48 ] 2023-10-30T13:09:52.090 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.85 2023-10-30T13:09:52.090 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.85 2023-10-30T13:09:52.090 INFO:tasks.workunit.client.0.smithi125.stdout:v0.85 2023-10-30T13:09:52.091 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.85 2023-10-30T13:09:53.547 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:09:53.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:09:53.595 INFO:tasks.ceph.mds.a.smithi125.stderr:2023-10-30T13:09:53.590+0000 7fa146ce0640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 64908) UID: 0 2023-10-30T13:09:53.596 INFO:tasks.ceph.mds.c.smithi125.stderr:2023-10-30T13:09:53.594+0000 7f6ed6e04640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 64908) UID: 0 2023-10-30T13:09:53.596 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T13:09:53.594+0000 7f511185f640 -1 received signal: Hangup from killall ceph-mds -1 (PID: 64908) UID: 0 2023-10-30T13:09:58.856 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 50% (123/243) Updating files: 51% (124/243) Updating files: 52% (127/243) Updating files: 53% (129/243) Updating files: 54% (132/243) Updating files: 55% (134/243) Updating files: 56% (137/243) Updating files: 57% (139/243) Updating files: 58% (141/243) Updating files: 59% (144/243) Updating files: 60% (146/243) Updating files: 61% (149/243) Updating files: 62% (151/243) Updating files: 63% (154/243) Updating files: 64% (156/243) Updating files: 65% (158/243) Updating files: 66% (161/243) Updating files: 67% (163/243) Updating files: 68% (166/243) Updating files: 69% (168/243) Updating files: 70% (171/243) Updating files: 71% (173/243) Updating files: 72% (175/243) Updating files: 73% (178/243) Updating files: 74% (180/243) Updating files: 75% (183/243) Updating files: 76% (185/243) Updating files: 77% (188/243) Updating files: 78% (190/243) Updating files: 79% (192/243) Updating files: 79% (193/243) Updating files: 80% (195/243) Updating files: 81% (197/243) Updating files: 82% (200/243) Updating files: 83% (202/243) Updating files: 83% (203/243) Updating files: 84% (205/243) Updating files: 85% (207/243) Updating files: 86% (209/243) Updating files: 87% (212/243) Updating files: 88% (214/243) Updating files: 89% (217/243) Updating files: 90% (219/243) Updating files: 91% (222/243) Updating files: 92% (224/243) Updating files: 93% (226/243) Updating files: 94% (229/243) Updating files: 95% (231/243) Updating files: 96% (234/243) Updating files: 97% (236/243) Updating files: 98% (239/243) Updating files: 99% (241/243) Updating files: 100% (243/243) Updating files: 100% (243/243), done. 2023-10-30T13:09:58.985 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at a0c22842db9 0.85 2023-10-30T13:09:58.993 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.85 2023-10-30T13:09:59.977 INFO:tasks.workunit.client.0.smithi125.stdout:v0.86 2023-10-30T13:09:59.977 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 86 -eq 48 ] 2023-10-30T13:09:59.978 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.86 2023-10-30T13:09:59.978 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.86 2023-10-30T13:09:59.978 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.86 2023-10-30T13:10:06.500 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 18% (84/447) Updating files: 19% (85/447) Updating files: 20% (90/447) Updating files: 21% (94/447) Updating files: 22% (99/447) Updating files: 23% (103/447) Updating files: 24% (108/447) Updating files: 25% (112/447) Updating files: 26% (117/447) Updating files: 27% (121/447) Updating files: 28% (126/447) Updating files: 29% (130/447) Updating files: 30% (135/447) Updating files: 31% (139/447) Updating files: 32% (144/447) Updating files: 33% (148/447) Updating files: 34% (152/447) Updating files: 35% (157/447) Updating files: 36% (161/447) Updating files: 37% (166/447) Updating files: 38% (170/447) Updating files: 39% (175/447) Updating files: 40% (179/447) Updating files: 41% (184/447) Updating files: 42% (188/447) Updating files: 43% (193/447) Updating files: 44% (197/447) Updating files: 45% (202/447) Updating files: 46% (206/447) Updating files: 47% (211/447) Updating files: 48% (215/447) Updating files: 49% (220/447) Updating files: 50% (224/447) Updating files: 51% (228/447) Updating files: 52% (233/447) Updating files: 53% (237/447) Updating files: 54% (242/447) Updating files: 55% (246/447) Updating files: 56% (251/447) Updating files: 57% (255/447) Updating files: 58% (260/447) Updating files: 59% (264/447) Updating files: 60% (269/447) Updating files: 61% (273/447) Updating files: 62% (278/447) Updating files: 63% (282/447) Updating files: 64% (287/447) Updating files: 65% (291/447) Updating files: 66% (296/447) Updating files: 67% (300/447) Updating files: 68% (304/447) Updating files: 69% (309/447) Updating files: 70% (313/447) Updating files: 71% (318/447) Updating files: 72% (322/447) Updating files: 73% (327/447) Updating files: 74% (331/447) Updating files: 75% (336/447) Updating files: 76% (340/447) Updating files: 77% (345/447) Updating files: 78% (349/447) Updating files: 79% (354/447) Updating files: 80% (358/447) Updating files: 81% (363/447) Updating files: 82% (367/447) Updating files: 83% (372/447) Updating files: 84% (376/447) Updating files: 85% (380/447) Updating files: 86% (385/447) Updating files: 86% (386/447) Updating files: 87% (389/447) Updating files: 88% (394/447) Updating files: 89% (398/447) Updating files: 90% (403/447) Updating files: 91% (407/447) Updating files: 92% (412/447) Updating files: 92% (414/447) Updating files: 93% (416/447) Updating files: 94% (421/447) Updating files: 95% (425/447) Updating files: 96% (430/447) Updating files: 97% (434/447) Updating files: 98% (439/447) Updating files: 99% (443/447) Updating files: 100% (447/447) Updating files: 100% (447/447), done. 2023-10-30T13:10:06.603 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 97dcc0539df 0.86 2023-10-30T13:10:06.612 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.86 2023-10-30T13:10:07.087 INFO:tasks.workunit.client.0.smithi125.stdout:v0.87 2023-10-30T13:10:07.087 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 87 -eq 48 ] 2023-10-30T13:10:07.088 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.87 2023-10-30T13:10:07.088 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.87 2023-10-30T13:10:07.088 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.87 2023-10-30T13:10:10.304 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 79% (47/59) Updating files: 81% (48/59) Updating files: 83% (49/59) Updating files: 84% (50/59) Updating files: 86% (51/59) Updating files: 88% (52/59) Updating files: 89% (53/59) Updating files: 91% (54/59) Updating files: 93% (55/59) Updating files: 94% (56/59) Updating files: 96% (57/59) Updating files: 98% (58/59) Updating files: 100% (59/59) Updating files: 100% (59/59), done. 2023-10-30T13:10:10.476 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at c51c8f9d80f 0.87 2023-10-30T13:10:10.479 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.87 2023-10-30T13:10:10.548 INFO:tasks.workunit.client.0.smithi125.stdout:v0.88 2023-10-30T13:10:10.548 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 88 -eq 48 ] 2023-10-30T13:10:10.548 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.88 2023-10-30T13:10:10.549 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.88 2023-10-30T13:10:10.549 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.88 2023-10-30T13:10:18.450 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 12% (41/335) Updating files: 13% (44/335) Updating files: 14% (47/335) Updating files: 15% (51/335) Updating files: 16% (54/335) Updating files: 17% (57/335) Updating files: 18% (61/335) Updating files: 19% (64/335) Updating files: 20% (67/335) Updating files: 21% (71/335) Updating files: 22% (74/335) Updating files: 23% (78/335) Updating files: 24% (81/335) Updating files: 25% (84/335) Updating files: 26% (88/335) Updating files: 27% (91/335) Updating files: 28% (94/335) Updating files: 29% (98/335) Updating files: 30% (101/335) Updating files: 31% (104/335) Updating files: 32% (108/335) Updating files: 33% (111/335) Updating files: 34% (114/335) Updating files: 35% (118/335) Updating files: 36% (121/335) Updating files: 37% (124/335) Updating files: 38% (128/335) Updating files: 39% (131/335) Updating files: 40% (134/335) Updating files: 41% (138/335) Updating files: 42% (141/335) Updating files: 43% (145/335) Updating files: 44% (148/335) Updating files: 45% (151/335) Updating files: 46% (155/335) Updating files: 47% (158/335) Updating files: 47% (159/335) Updating files: 48% (161/335) Updating files: 49% (165/335) Updating files: 50% (168/335) Updating files: 51% (171/335) Updating files: 52% (175/335) Updating files: 53% (178/335) Updating files: 53% (179/335) Updating files: 54% (181/335) Updating files: 55% (185/335) Updating files: 56% (188/335) Updating files: 57% (191/335) Updating files: 58% (195/335) Updating files: 59% (198/335) Updating files: 60% (201/335) Updating files: 61% (205/335) Updating files: 62% (208/335) Updating files: 63% (212/335) Updating files: 64% (215/335) Updating files: 65% (218/335) Updating files: 66% (222/335) Updating files: 67% (225/335) Updating files: 68% (228/335) Updating files: 69% (232/335) Updating files: 70% (235/335) Updating files: 71% (238/335) Updating files: 72% (242/335) Updating files: 73% (245/335) Updating files: 74% (248/335) Updating files: 75% (252/335) Updating files: 76% (255/335) Updating files: 77% (258/335) Updating files: 78% (262/335) Updating files: 79% (265/335) Updating files: 80% (268/335) Updating files: 80% (271/335) Updating files: 81% (272/335) Updating files: 82% (275/335) Updating files: 83% (279/335) Updating files: 84% (282/335) Updating files: 85% (285/335) Updating files: 86% (289/335) Updating files: 87% (292/335) Updating files: 88% (295/335) Updating files: 88% (296/335) Updating files: 89% (299/335) Updating files: 90% (302/335) Updating files: 91% (305/335) Updating files: 92% (309/335) Updating files: 93% (312/335) Updating files: 94% (315/335) Updating files: 95% (319/335) Updating files: 96% (322/335) Updating files: 97% (325/335) Updating files: 98% (329/335) Updating files: 99% (332/335) Updating files: 100% (335/335) Updating files: 100% (335/335), done. 2023-10-30T13:10:18.835 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 4be687bf448 0.88 2023-10-30T13:10:18.843 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.88 2023-10-30T13:10:18.955 INFO:tasks.workunit.client.0.smithi125.stdout:v0.89 2023-10-30T13:10:18.956 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 89 -eq 48 ] 2023-10-30T13:10:18.956 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.89 2023-10-30T13:10:18.956 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.89 2023-10-30T13:10:18.956 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.89 2023-10-30T13:10:24.073 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 5% (11/197) Updating files: 6% (12/197) Updating files: 7% (14/197) Updating files: 8% (16/197) Updating files: 9% (18/197) Updating files: 10% (20/197) Updating files: 11% (22/197) Updating files: 12% (24/197) Updating files: 13% (26/197) Updating files: 14% (28/197) Updating files: 15% (30/197) Updating files: 16% (32/197) Updating files: 17% (34/197) Updating files: 18% (36/197) Updating files: 19% (38/197) Updating files: 20% (40/197) Updating files: 21% (42/197) Updating files: 22% (44/197) Updating files: 23% (46/197) Updating files: 24% (48/197) Updating files: 25% (50/197) Updating files: 26% (52/197) Updating files: 27% (54/197) Updating files: 28% (56/197) Updating files: 29% (58/197) Updating files: 30% (60/197) Updating files: 31% (62/197) Updating files: 32% (64/197) Updating files: 33% (66/197) Updating files: 34% (67/197) Updating files: 35% (69/197) Updating files: 36% (71/197) Updating files: 37% (73/197) Updating files: 38% (75/197) Updating files: 39% (77/197) Updating files: 40% (79/197) Updating files: 41% (81/197) Updating files: 42% (83/197) Updating files: 43% (85/197) Updating files: 44% (87/197) Updating files: 45% (89/197) Updating files: 46% (91/197) Updating files: 47% (93/197) Updating files: 48% (95/197) Updating files: 49% (97/197) Updating files: 50% (99/197) Updating files: 51% (101/197) Updating files: 52% (103/197) Updating files: 53% (105/197) Updating files: 54% (107/197) Updating files: 55% (109/197) Updating files: 56% (111/197) Updating files: 57% (113/197) Updating files: 58% (115/197) Updating files: 59% (117/197) Updating files: 60% (119/197) Updating files: 61% (121/197) Updating files: 62% (123/197) Updating files: 63% (125/197) Updating files: 64% (127/197) Updating files: 65% (129/197) Updating files: 66% (131/197) Updating files: 67% (132/197) Updating files: 68% (134/197) Updating files: 69% (136/197) Updating files: 70% (138/197) Updating files: 71% (140/197) Updating files: 72% (142/197) Updating files: 73% (144/197) Updating files: 74% (146/197) Updating files: 75% (148/197) Updating files: 76% (150/197) Updating files: 77% (152/197) Updating files: 78% (154/197) Updating files: 79% (156/197) Updating files: 80% (158/197) Updating files: 81% (160/197) Updating files: 82% (162/197) Updating files: 83% (164/197) Updating files: 84% (166/197) Updating files: 85% (168/197) Updating files: 86% (170/197) Updating files: 87% (172/197) Updating files: 88% (174/197) Updating files: 89% (176/197) Updating files: 90% (178/197) Updating files: 91% (180/197) Updating files: 92% (182/197) Updating files: 93% (184/197) Updating files: 94% (186/197) Updating files: 95% (188/197) Updating files: 96% (190/197) Updating files: 97% (192/197) Updating files: 98% (194/197) Updating files: 99% (196/197) Updating files: 100% (197/197) Updating files: 100% (197/197), done. 2023-10-30T13:10:24.190 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 68fdc0f68e6 0.89 2023-10-30T13:10:24.202 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.89 2023-10-30T13:10:24.304 INFO:tasks.workunit.client.0.smithi125.stdout:v0.90 2023-10-30T13:10:24.305 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 90 -eq 48 ] 2023-10-30T13:10:24.305 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.90 2023-10-30T13:10:24.305 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo v0.90 2023-10-30T13:10:24.305 INFO:tasks.workunit.client.0.smithi125.stderr:+ git reset --hard v0.90 2023-10-30T13:10:30.191 INFO:tasks.workunit.client.0.smithi125.stderr:Updating files: 31% (83/265) Updating files: 32% (85/265) Updating files: 33% (88/265) Updating files: 34% (91/265) Updating files: 35% (93/265) Updating files: 36% (96/265) Updating files: 37% (99/265) Updating files: 38% (101/265) Updating files: 39% (104/265) Updating files: 40% (106/265) Updating files: 41% (109/265) Updating files: 42% (112/265) Updating files: 43% (114/265) Updating files: 44% (117/265) Updating files: 45% (120/265) Updating files: 46% (122/265) Updating files: 47% (125/265) Updating files: 48% (128/265) Updating files: 49% (130/265) Updating files: 50% (133/265) Updating files: 51% (136/265) Updating files: 52% (138/265) Updating files: 53% (141/265) Updating files: 54% (144/265) Updating files: 55% (146/265) Updating files: 56% (149/265) Updating files: 57% (152/265) Updating files: 58% (154/265) Updating files: 59% (157/265) Updating files: 60% (159/265) Updating files: 61% (162/265) Updating files: 62% (165/265) Updating files: 63% (167/265) Updating files: 64% (170/265) Updating files: 65% (173/265) Updating files: 66% (175/265) Updating files: 67% (178/265) Updating files: 68% (181/265) Updating files: 69% (183/265) Updating files: 70% (186/265) Updating files: 71% (189/265) Updating files: 72% (191/265) Updating files: 73% (194/265) Updating files: 74% (197/265) Updating files: 75% (199/265) Updating files: 76% (202/265) Updating files: 77% (205/265) Updating files: 78% (207/265) Updating files: 79% (210/265) Updating files: 80% (212/265) Updating files: 81% (215/265) Updating files: 82% (218/265) Updating files: 83% (220/265) Updating files: 84% (223/265) Updating files: 85% (226/265) Updating files: 86% (228/265) Updating files: 87% (231/265) Updating files: 88% (234/265) Updating files: 89% (236/265) Updating files: 90% (239/265) Updating files: 91% (242/265) Updating files: 92% (244/265) Updating files: 93% (247/265) Updating files: 94% (250/265) Updating files: 95% (252/265) Updating files: 96% (255/265) Updating files: 97% (258/265) Updating files: 98% (260/265) Updating files: 99% (263/265) Updating files: 100% (265/265) Updating files: 100% (265/265), done. 2023-10-30T13:10:30.313 INFO:tasks.workunit.client.0.smithi125.stdout:HEAD is now at 08bd1e1eee3 0.90 2023-10-30T13:10:30.319 INFO:tasks.workunit.client.0.smithi125.stderr:+ mkdir .snap/v0.90 2023-10-30T13:10:30.598 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.1 2023-10-30T13:10:30.598 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 1 -eq 48 ] 2023-10-30T13:10:30.599 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.1 2023-10-30T13:10:30.599 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.1 2023-10-30T13:10:30.599 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.1 2023-10-30T13:10:30.670 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:10:39.968 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:10:39.969 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 2 -eq 48 ] 2023-10-30T13:10:39.969 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.2 2023-10-30T13:10:39.969 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.2 2023-10-30T13:10:39.969 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.2 2023-10-30T13:10:39.970 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.2 2023-10-30T13:10:39.970 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:10:43.395 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:10:43.396 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 3 -eq 48 ] 2023-10-30T13:10:43.396 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.3 2023-10-30T13:10:43.396 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.3 2023-10-30T13:10:43.397 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.3 2023-10-30T13:10:43.397 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.3 2023-10-30T13:10:43.397 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:10:48.033 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:10:48.034 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 4 -eq 48 ] 2023-10-30T13:10:48.034 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.4 2023-10-30T13:10:48.034 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.4 2023-10-30T13:10:48.035 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.4 2023-10-30T13:10:48.035 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.4 2023-10-30T13:10:48.035 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:10:52.389 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:10:52.389 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 5 -eq 48 ] 2023-10-30T13:10:52.389 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.5 2023-10-30T13:10:52.390 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.5 2023-10-30T13:10:52.390 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.5 2023-10-30T13:10:52.391 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.5 2023-10-30T13:10:52.391 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:10:55.396 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:10:55.397 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.6 2023-10-30T13:10:55.397 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 6 -eq 48 ] 2023-10-30T13:10:55.397 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.6 2023-10-30T13:10:55.398 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.6 2023-10-30T13:10:55.398 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.6 2023-10-30T13:10:55.398 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:11:00.187 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:11:00.187 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 7 -eq 48 ] 2023-10-30T13:11:00.187 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.7 2023-10-30T13:11:00.187 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.7 2023-10-30T13:11:00.188 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.7 2023-10-30T13:11:00.188 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.7 2023-10-30T13:11:00.188 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:11:05.695 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:11:05.696 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.8 2023-10-30T13:11:05.697 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 8 -eq 48 ] 2023-10-30T13:11:05.697 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.8 2023-10-30T13:11:05.697 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.8 2023-10-30T13:11:05.698 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.8 2023-10-30T13:11:05.698 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:11:10.481 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:11:10.481 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 9 -eq 48 ] 2023-10-30T13:11:10.481 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.9 2023-10-30T13:11:10.482 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.9 2023-10-30T13:11:10.482 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.9 2023-10-30T13:11:10.483 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.9 2023-10-30T13:11:10.483 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:11:14.648 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:11:14.649 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 10 -eq 48 ] 2023-10-30T13:11:14.649 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.10 2023-10-30T13:11:14.649 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.10 2023-10-30T13:11:14.649 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.10 2023-10-30T13:11:14.650 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.10 2023-10-30T13:11:14.650 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:11:16.897 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:17.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.036+0000 7f1dfb054640 1 Processor -- start 2023-10-30T13:11:17.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.036+0000 7f1dfb054640 1 -- start start 2023-10-30T13:11:17.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.036+0000 7f1dfb054640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1df4153bf0 0x7f1df4071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:17.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.036+0000 7f1dfb054640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df40700b0 con 0x7f1df40c6fe0 2023-10-30T13:11:17.042 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.036+0000 7f1dfb054640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df40c6530 con 0x7f1df40c5f60 2023-10-30T13:11:17.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.036+0000 7f1dfb054640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df40c6810 con 0x7f1df4153bf0 2023-10-30T13:11:17.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.036+0000 7f1df8df0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1df4153bf0 0x7f1df4071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:17.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.036+0000 7f1df3fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1df40c6fe0 0x7f1df40c74a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46142/0 (socket says 172.21.15.125:46142) 2023-10-30T13:11:17.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.036+0000 7f1df3fff640 1 -- 172.21.15.125:0/527587483 learned_addr learned my addr 172.21.15.125:0/527587483 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:11:17.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.036+0000 7f1df8df0640 1 -- 172.21.15.125:0/527587483 >> v1:172.21.15.136:6789/0 conn(0x7f1df40c5f60 legacy=0x7f1df40cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:17.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.036+0000 7f1df8df0640 1 -- 172.21.15.125:0/527587483 >> v1:172.21.15.125:6789/0 conn(0x7f1df40c6fe0 legacy=0x7f1df40c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:17.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.036+0000 7f1df8df0640 1 -- 172.21.15.125:0/527587483 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df4159b10 con 0x7f1df4153bf0 2023-10-30T13:11:17.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.036+0000 7f1df37fe640 1 -- 172.21.15.125:0/527587483 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937499954 0 0) 0x7f1df40700b0 con 0x7f1df40c6fe0 2023-10-30T13:11:17.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.040+0000 7f1df8df0640 1 --2- 172.21.15.125:0/527587483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1df4153bf0 0x7f1df4071260 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f1de80097c0 tx=0x7f1de801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45e0389fefcb8c86 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:17.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.040+0000 7f1df37fe640 1 -- 172.21.15.125:0/527587483 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de80a8970 con 0x7f1df4153bf0 2023-10-30T13:11:17.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.040+0000 7f1df37fe640 1 -- 172.21.15.125:0/527587483 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de80a8ad0 con 0x7f1df4153bf0 2023-10-30T13:11:17.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.040+0000 7f1df37fe640 1 -- 172.21.15.125:0/527587483 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de80a8e60 con 0x7f1df4153bf0 2023-10-30T13:11:17.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.040+0000 7f1dfb054640 1 -- 172.21.15.125:0/527587483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1df4153bf0 msgr2=0x7f1df4071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:17.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.040+0000 7f1dfb054640 1 --2- 172.21.15.125:0/527587483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1df4153bf0 0x7f1df4071260 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f1de80097c0 tx=0x7f1de801ec70 comp rx=0 tx=0).stop 2023-10-30T13:11:17.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1dfb054640 1 -- 172.21.15.125:0/527587483 shutdown_connections 2023-10-30T13:11:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1dfb054640 1 --2- 172.21.15.125:0/527587483 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1df4153bf0 0x7f1df4071260 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1dfb054640 1 -- 172.21.15.125:0/527587483 >> 172.21.15.125:0/527587483 conn(0x7f1df406c6e0 msgr2=0x7f1df4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1dfb054640 1 -- 172.21.15.125:0/527587483 shutdown_connections 2023-10-30T13:11:17.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1dfb054640 1 -- 172.21.15.125:0/527587483 wait complete. 2023-10-30T13:11:17.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1dfb054640 1 Processor -- start 2023-10-30T13:11:17.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1dfb054640 1 -- start start 2023-10-30T13:11:17.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1dfb054640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1df40c5f60 0x7f1df414ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:17.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1dfb054640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1df40c6fe0 0x7f1df4154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:17.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1dfb054640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1df4153bf0 0x7f1df41547b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:17.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1dfb054640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df41ef400 con 0x7f1df40c5f60 2023-10-30T13:11:17.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1dfb054640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df414e700 con 0x7f1df40c6fe0 2023-10-30T13:11:17.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1dfb054640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df414e9e0 con 0x7f1df4153bf0 2023-10-30T13:11:17.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1df8df0640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1df40c5f60 0x7f1df414ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:17.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1df8df0640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1df40c5f60 0x7f1df414ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38248/0 (socket says 172.21.15.125:38248) 2023-10-30T13:11:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1df8df0640 1 -- 172.21.15.125:0/1319728285 learned_addr learned my addr 172.21.15.125:0/1319728285 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1df3fff640 1 --2- 172.21.15.125:0/1319728285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1df40c6fe0 0x7f1df4154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.044+0000 7f1df95f1640 1 --2- 172.21.15.125:0/1319728285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1df4153bf0 0x7f1df41547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.048+0000 7f1df8df0640 1 -- 172.21.15.125:0/1319728285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1df4153bf0 msgr2=0x7f1df41547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:17.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.048+0000 7f1df8df0640 1 --2- 172.21.15.125:0/1319728285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1df4153bf0 0x7f1df41547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.048+0000 7f1df8df0640 1 -- 172.21.15.125:0/1319728285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1df40c6fe0 msgr2=0x7f1df4154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:17.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.048+0000 7f1df8df0640 1 --2- 172.21.15.125:0/1319728285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1df40c6fe0 0x7f1df4154270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.048+0000 7f1df8df0640 1 -- 172.21.15.125:0/1319728285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de8009420 con 0x7f1df40c5f60 2023-10-30T13:11:17.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.048+0000 7f1df8df0640 1 --2- 172.21.15.125:0/1319728285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1df40c5f60 0x7f1df414ca40 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f1de801b880 tx=0x7f1de801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:17.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.048+0000 7f1df17fa640 1 -- 172.21.15.125:0/1319728285 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de80b0ea0 con 0x7f1df40c5f60 2023-10-30T13:11:17.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.048+0000 7f1df17fa640 1 -- 172.21.15.125:0/1319728285 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de80b9410 con 0x7f1df40c5f60 2023-10-30T13:11:17.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.048+0000 7f1dfb054640 1 -- 172.21.15.125:0/1319728285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df414ecc0 con 0x7f1df40c5f60 2023-10-30T13:11:17.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.048+0000 7f1df17fa640 1 -- 172.21.15.125:0/1319728285 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de80b97a0 con 0x7f1df40c5f60 2023-10-30T13:11:17.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.048+0000 7f1dfb054640 1 -- 172.21.15.125:0/1319728285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df414f280 con 0x7f1df40c5f60 2023-10-30T13:11:17.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.048+0000 7f1dfb054640 1 -- 172.21.15.125:0/1319728285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df414f470 con 0x7f1df40c5f60 2023-10-30T13:11:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.048+0000 7f1df17fa640 1 -- 172.21.15.125:0/1319728285 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1de80b7040 con 0x7f1df40c5f60 2023-10-30T13:11:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.052+0000 7f1df17fa640 1 --2- 172.21.15.125:0/1319728285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1dd80664d0 0x7f1dd8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.052+0000 7f1df17fa640 1 -- 172.21.15.125:0/1319728285 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7f1de81223a0 con 0x7f1df40c5f60 2023-10-30T13:11:17.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.052+0000 7f1df3fff640 1 --2- 172.21.15.125:0/1319728285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1dd80664d0 0x7f1dd8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.052+0000 7f1df3fff640 1 --2- 172.21.15.125:0/1319728285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1dd80664d0 0x7f1dd8068990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1df4150c50 tx=0x7f1de0007ba0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:17.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.056+0000 7f1df17fa640 1 -- 172.21.15.125:0/1319728285 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1de80ef0d0 con 0x7f1df40c5f60 2023-10-30T13:11:17.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.240+0000 7f1dfb054640 1 -- 172.21.15.125:0/1319728285 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1df404e170 con 0x7f1df40c5f60 2023-10-30T13:11:17.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.240+0000 7f1df17fa640 1 -- 172.21.15.125:0/1319728285 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+6555 (secure 0 0 0) 0x7f1de80f26f0 con 0x7f1df40c5f60 2023-10-30T13:11:17.248 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:17.248 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8443,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:09:32.699420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:active","state_seq":14,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:17.248 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 194 2023-10-30T13:11:17.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.244+0000 7f1dfb054640 1 -- 172.21.15.125:0/1319728285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1dd80664d0 msgr2=0x7f1dd8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:17.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.244+0000 7f1dfb054640 1 --2- 172.21.15.125:0/1319728285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1dd80664d0 0x7f1dd8068990 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f1df4150c50 tx=0x7f1de0007ba0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.244+0000 7f1dfb054640 1 -- 172.21.15.125:0/1319728285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1df40c5f60 msgr2=0x7f1df414ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:17.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.244+0000 7f1dfb054640 1 --2- 172.21.15.125:0/1319728285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1df40c5f60 0x7f1df414ca40 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f1de801b880 tx=0x7f1de801ba10 comp rx=0 tx=0).stop 2023-10-30T13:11:17.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.244+0000 7f1dfb054640 1 -- 172.21.15.125:0/1319728285 shutdown_connections 2023-10-30T13:11:17.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.244+0000 7f1dfb054640 1 --2- 172.21.15.125:0/1319728285 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1dd80664d0 0x7f1dd8068990 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.244+0000 7f1dfb054640 1 --2- 172.21.15.125:0/1319728285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1df4153bf0 0x7f1df41547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.244+0000 7f1dfb054640 1 --2- 172.21.15.125:0/1319728285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1df40c6fe0 0x7f1df4154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.244+0000 7f1dfb054640 1 --2- 172.21.15.125:0/1319728285 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1df40c5f60 0x7f1df414ca40 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.244+0000 7f1dfb054640 1 -- 172.21.15.125:0/1319728285 >> 172.21.15.125:0/1319728285 conn(0x7f1df406c6e0 msgr2=0x7f1df4151d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:17.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.244+0000 7f1dfb054640 1 -- 172.21.15.125:0/1319728285 shutdown_connections 2023-10-30T13:11:17.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.244+0000 7f1dfb054640 1 -- 172.21.15.125:0/1319728285 wait complete. 2023-10-30T13:11:17.269 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=2) 2023-10-30T13:11:17.269 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:11:17.269 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:11:17.293 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:11:17.293 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:17.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.428+0000 7f6a1653e640 1 Processor -- start 2023-10-30T13:11:17.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.428+0000 7f6a1653e640 1 -- start start 2023-10-30T13:11:17.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a1653e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6a101538a0 0x7f6a1015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:17.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a1653e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a100ca0e0 0x7f6a1015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:17.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a1653e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a100cb800 con 0x7f6a100cf160 2023-10-30T13:11:17.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a1653e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1015dc30 con 0x7f6a101538a0 2023-10-30T13:11:17.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a1653e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a100cb330 con 0x7f6a100ca0e0 2023-10-30T13:11:17.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a0ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6a101538a0 0x7f6a1015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:17.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a14adb640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6a100cf160 0x7f6a100cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46152/0 (socket says 172.21.15.125:46152) 2023-10-30T13:11:17.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a14adb640 1 -- 172.21.15.125:0/1632593688 learned_addr learned my addr 172.21.15.125:0/1632593688 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:11:17.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a0ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6a101538a0 0x7f6a1015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38252/0 (socket says 172.21.15.125:38252) 2023-10-30T13:11:17.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a0f7fe640 1 --2- 172.21.15.125:0/1632593688 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a100ca0e0 0x7f6a1015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:17.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a0ffff640 1 -- 172.21.15.125:0/1632593688 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a100ca0e0 msgr2=0x7f6a1015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:17.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a0ffff640 1 --2- 172.21.15.125:0/1632593688 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a100ca0e0 0x7f6a1015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a0ffff640 1 -- 172.21.15.125:0/1632593688 >> v1:172.21.15.125:6789/0 conn(0x7f6a100cf160 legacy=0x7f6a100cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:17.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a0ffff640 1 -- 172.21.15.125:0/1632593688 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a100cad70 con 0x7f6a101538a0 2023-10-30T13:11:17.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.432+0000 7f6a0f7fe640 1 --2- 172.21.15.125:0/1632593688 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a100ca0e0 0x7f6a1015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:11:17.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.436+0000 7f6a0ffff640 1 --2- 172.21.15.125:0/1632593688 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a101538a0 0x7f6a1015d6f0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f6a000097c0 tx=0x7f6a0001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50c346bb45d842f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:17.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.436+0000 7f6a0effd640 1 -- 172.21.15.125:0/1632593688 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a000a8970 con 0x7f6a101538a0 2023-10-30T13:11:17.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.436+0000 7f6a0effd640 1 -- 172.21.15.125:0/1632593688 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a000a8ad0 con 0x7f6a101538a0 2023-10-30T13:11:17.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.436+0000 7f6a0effd640 1 -- 172.21.15.125:0/1632593688 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a000a8e60 con 0x7f6a101538a0 2023-10-30T13:11:17.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.436+0000 7f6a1653e640 1 -- 172.21.15.125:0/1632593688 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a101538a0 msgr2=0x7f6a1015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:17.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.436+0000 7f6a1653e640 1 --2- 172.21.15.125:0/1632593688 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a101538a0 0x7f6a1015d6f0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f6a000097c0 tx=0x7f6a0001ec70 comp rx=0 tx=0).stop 2023-10-30T13:11:17.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.436+0000 7f6a1653e640 1 -- 172.21.15.125:0/1632593688 shutdown_connections 2023-10-30T13:11:17.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.436+0000 7f6a1653e640 1 --2- 172.21.15.125:0/1632593688 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a100ca0e0 0x7f6a1015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.436+0000 7f6a1653e640 1 --2- 172.21.15.125:0/1632593688 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a101538a0 0x7f6a1015d6f0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.436+0000 7f6a1653e640 1 -- 172.21.15.125:0/1632593688 >> 172.21.15.125:0/1632593688 conn(0x7f6a1006bfe0 msgr2=0x7f6a1006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:17.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.436+0000 7f6a1653e640 1 -- 172.21.15.125:0/1632593688 shutdown_connections 2023-10-30T13:11:17.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.436+0000 7f6a1653e640 1 -- 172.21.15.125:0/1632593688 wait complete. 2023-10-30T13:11:17.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.436+0000 7f6a1653e640 1 Processor -- start 2023-10-30T13:11:17.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.436+0000 7f6a1653e640 1 -- start start 2023-10-30T13:11:17.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a1653e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100ca0e0 0x7f6a101e5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:17.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a1653e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a100cf160 0x7f6a101e3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:17.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a1653e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538a0 0x7f6a101e3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:17.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a1653e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a101e3590 con 0x7f6a100ca0e0 2023-10-30T13:11:17.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a1653e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a101e5910 con 0x7f6a100cf160 2023-10-30T13:11:17.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a1653e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a100c9320 con 0x7f6a101538a0 2023-10-30T13:11:17.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a0ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100ca0e0 0x7f6a101e5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:17.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a0ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100ca0e0 0x7f6a101e5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38266/0 (socket says 172.21.15.125:38266) 2023-10-30T13:11:17.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a0ffff640 1 -- 172.21.15.125:0/4094758374 learned_addr learned my addr 172.21.15.125:0/4094758374 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:17.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a14adb640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538a0 0x7f6a101e3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:17.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a0f7fe640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a100cf160 0x7f6a101e3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:17.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a0ffff640 1 -- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538a0 msgr2=0x7f6a101e3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:17.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a0ffff640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538a0 0x7f6a101e3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a0ffff640 1 -- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a100cf160 msgr2=0x7f6a101e3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:17.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a0ffff640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a100cf160 0x7f6a101e3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a0ffff640 1 -- 172.21.15.125:0/4094758374 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a00009420 con 0x7f6a100ca0e0 2023-10-30T13:11:17.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a14adb640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538a0 0x7f6a101e3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:11:17.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a0ffff640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100ca0e0 0x7f6a101e5050 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f6a000b3040 tx=0x7f6a0001ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:17.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a0cff9640 1 -- 172.21.15.125:0/4094758374 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a000b9630 con 0x7f6a100ca0e0 2023-10-30T13:11:17.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a0cff9640 1 -- 172.21.15.125:0/4094758374 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a000b97b0 con 0x7f6a100ca0e0 2023-10-30T13:11:17.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a1653e640 1 -- 172.21.15.125:0/4094758374 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a101e2c60 con 0x7f6a100ca0e0 2023-10-30T13:11:17.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a0cff9640 1 -- 172.21.15.125:0/4094758374 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a000b9b60 con 0x7f6a100ca0e0 2023-10-30T13:11:17.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a1653e640 1 -- 172.21.15.125:0/4094758374 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a101e3220 con 0x7f6a100ca0e0 2023-10-30T13:11:17.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a1653e640 1 -- 172.21.15.125:0/4094758374 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a100b2a50 con 0x7f6a100ca0e0 2023-10-30T13:11:17.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.440+0000 7f6a0cff9640 1 -- 172.21.15.125:0/4094758374 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6a000b7040 con 0x7f6a100ca0e0 2023-10-30T13:11:17.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.444+0000 7f6a0cff9640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69fc0664d0 0x7f69fc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:17.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.444+0000 7f6a0cff9640 1 -- 172.21.15.125:0/4094758374 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7f6a001226f0 con 0x7f6a100ca0e0 2023-10-30T13:11:17.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.444+0000 7f6a0f7fe640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69fc0664d0 0x7f69fc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:17.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.444+0000 7f6a0f7fe640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69fc0664d0 0x7f69fc068990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f6a100cf8b0 tx=0x7f6a04008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:17.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.448+0000 7f6a0cff9640 1 -- 172.21.15.125:0/4094758374 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6a000ef380 con 0x7f6a100ca0e0 2023-10-30T13:11:17.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.628+0000 7f6a1653e640 1 -- 172.21.15.125:0/4094758374 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a1004e170 con 0x7f6a100ca0e0 2023-10-30T13:11:17.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.628+0000 7f6a0cff9640 1 -- 172.21.15.125:0/4094758374 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+6555 (secure 0 0 0) 0x7f6a000b5020 con 0x7f6a100ca0e0 2023-10-30T13:11:17.634 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:17.635 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8443,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:09:32.699420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:active","state_seq":14,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:17.636 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 194 2023-10-30T13:11:17.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.628+0000 7f6a1653e640 1 -- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69fc0664d0 msgr2=0x7f69fc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:17.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.628+0000 7f6a1653e640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69fc0664d0 0x7f69fc068990 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f6a100cf8b0 tx=0x7f6a04008040 comp rx=0 tx=0).stop 2023-10-30T13:11:17.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.628+0000 7f6a1653e640 1 -- 172.21.15.125:0/4094758374 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100ca0e0 msgr2=0x7f6a101e5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:17.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.628+0000 7f6a1653e640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100ca0e0 0x7f6a101e5050 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f6a000b3040 tx=0x7f6a0001ba10 comp rx=0 tx=0).stop 2023-10-30T13:11:17.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.628+0000 7f6a1653e640 1 -- 172.21.15.125:0/4094758374 shutdown_connections 2023-10-30T13:11:17.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.628+0000 7f6a1653e640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69fc0664d0 0x7f69fc068990 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.628+0000 7f6a1653e640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538a0 0x7f6a101e3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.628+0000 7f6a1653e640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a100cf160 0x7f6a101e3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.628+0000 7f6a1653e640 1 --2- 172.21.15.125:0/4094758374 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100ca0e0 0x7f6a101e5050 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:17.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.628+0000 7f6a1653e640 1 -- 172.21.15.125:0/4094758374 >> 172.21.15.125:0/4094758374 conn(0x7f6a1006bfe0 msgr2=0x7f6a10151c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:17.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.628+0000 7f6a1653e640 1 -- 172.21.15.125:0/4094758374 shutdown_connections 2023-10-30T13:11:17.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:17.628+0000 7f6a1653e640 1 -- 172.21.15.125:0/4094758374 wait complete. 2023-10-30T13:11:17.655 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:11:19.656 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:19.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa69d12a640 1 Processor -- start 2023-10-30T13:11:19.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa69d12a640 1 -- start start 2023-10-30T13:11:19.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa69d12a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa6980cf160 0x7fa6980cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:19.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa69d12a640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6980cb800 con 0x7fa6981538a0 2023-10-30T13:11:19.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa69d12a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6980cb330 con 0x7fa69815d6f0 2023-10-30T13:11:19.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa69d12a640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6980cad70 con 0x7fa6980cf160 2023-10-30T13:11:19.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa696d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa6981538a0 0x7fa6980ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46162/0 (socket says 172.21.15.125:46162) 2023-10-30T13:11:19.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa696d9d640 1 -- 172.21.15.125:0/2672391136 learned_addr learned my addr 172.21.15.125:0/2672391136 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:11:19.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa69759e640 1 --2- 172.21.15.125:0/2672391136 >> v2:172.21.15.136:3300/0 conn(0x7fa6980cf160 0x7fa6980cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:19.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 -- 172.21.15.125:0/2672391136 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288227559 0 0) 0x7fa6980cb800 con 0x7fa6981538a0 2023-10-30T13:11:19.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 -- 172.21.15.125:0/2672391136 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa680003530 con 0x7fa6981538a0 2023-10-30T13:11:19.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 -- 172.21.15.125:0/2672391136 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1043865633 0 0) 0x7fa680003530 con 0x7fa6981538a0 2023-10-30T13:11:19.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 -- 172.21.15.125:0/2672391136 >> v2:172.21.15.136:3300/0 conn(0x7fa6980cf160 msgr2=0x7fa6980cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:19.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 --2- 172.21.15.125:0/2672391136 >> v2:172.21.15.136:3300/0 conn(0x7fa6980cf160 0x7fa6980cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:19.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 -- 172.21.15.125:0/2672391136 >> v1:172.21.15.136:6789/0 conn(0x7fa69815d6f0 legacy=0x7fa69815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:19.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 -- 172.21.15.125:0/2672391136 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6980cb050 con 0x7fa6981538a0 2023-10-30T13:11:19.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 -- 172.21.15.125:0/2672391136 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fa688001680 con 0x7fa6981538a0 2023-10-30T13:11:19.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa69759e640 1 --2- 172.21.15.125:0/2672391136 >> v2:172.21.15.136:3300/0 conn(0x7fa6980cf160 0x7fa6980cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:11:19.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 -- 172.21.15.125:0/2672391136 >> v1:172.21.15.125:6789/0 conn(0x7fa6981538a0 legacy=0x7fa6980ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:19.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 --2- 172.21.15.125:0/2672391136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa680003980 0x7fa680003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:19.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 --2- 172.21.15.125:0/2672391136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6800047d0 0x7fa680004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:19.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 --2- 172.21.15.125:0/2672391136 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa680005190 0x7fa68000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:19.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 -- 172.21.15.125:0/2672391136 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6980cad70 con 0x7fa680003980 2023-10-30T13:11:19.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 -- 172.21.15.125:0/2672391136 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa68000c040 con 0x7fa6800047d0 2023-10-30T13:11:19.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.792+0000 7fa695d9b640 1 -- 172.21.15.125:0/2672391136 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa68000c200 con 0x7fa680005190 2023-10-30T13:11:19.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.796+0000 7fa696d9d640 1 --2- 172.21.15.125:0/2672391136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa680003980 0x7fa680003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:19.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.796+0000 7fa69659c640 1 --2- 172.21.15.125:0/2672391136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6800047d0 0x7fa680004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:19.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.796+0000 7fa696d9d640 1 -- 172.21.15.125:0/2672391136 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa680005190 msgr2=0x7fa68000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:19.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.796+0000 7fa69759e640 1 --2- 172.21.15.125:0/2672391136 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa680005190 0x7fa68000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:19.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.796+0000 7fa696d9d640 1 --2- 172.21.15.125:0/2672391136 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa680005190 0x7fa68000b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:19.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.796+0000 7fa696d9d640 1 -- 172.21.15.125:0/2672391136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6800047d0 msgr2=0x7fa680004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:19.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.796+0000 7fa696d9d640 1 --2- 172.21.15.125:0/2672391136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6800047d0 0x7fa680004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:19.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.796+0000 7fa696d9d640 1 -- 172.21.15.125:0/2672391136 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa680003530 con 0x7fa680003980 2023-10-30T13:11:19.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.796+0000 7fa69659c640 1 --2- 172.21.15.125:0/2672391136 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6800047d0 0x7fa680004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:11:19.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.796+0000 7fa69759e640 1 -- 172.21.15.125:0/2672391136 reap_dead start 2023-10-30T13:11:19.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.796+0000 7fa696d9d640 1 --2- 172.21.15.125:0/2672391136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa680003980 0x7fa680003da0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fa68800c240 tx=0x7fa68801f2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580089dbb6b4ce76 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:19.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.796+0000 7fa695d9b640 1 -- 172.21.15.125:0/2672391136 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6880058e0 con 0x7fa680003980 2023-10-30T13:11:19.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.796+0000 7fa695d9b640 1 -- 172.21.15.125:0/2672391136 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa688005a40 con 0x7fa680003980 2023-10-30T13:11:19.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.796+0000 7fa695d9b640 1 -- 172.21.15.125:0/2672391136 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa688005e70 con 0x7fa680003980 2023-10-30T13:11:19.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.800+0000 7fa69d12a640 1 -- 172.21.15.125:0/2672391136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa680003980 msgr2=0x7fa680003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:19.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.800+0000 7fa69d12a640 1 --2- 172.21.15.125:0/2672391136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa680003980 0x7fa680003da0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fa68800c240 tx=0x7fa68801f2b0 comp rx=0 tx=0).stop 2023-10-30T13:11:19.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.800+0000 7fa69d12a640 1 -- 172.21.15.125:0/2672391136 shutdown_connections 2023-10-30T13:11:19.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.800+0000 7fa69d12a640 1 --2- 172.21.15.125:0/2672391136 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa680003980 0x7fa680003da0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:19.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.800+0000 7fa69d12a640 1 -- 172.21.15.125:0/2672391136 >> 172.21.15.125:0/2672391136 conn(0x7fa69806bfe0 msgr2=0x7fa69806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:19.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.800+0000 7fa69d12a640 1 -- 172.21.15.125:0/2672391136 shutdown_connections 2023-10-30T13:11:19.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.800+0000 7fa69d12a640 1 -- 172.21.15.125:0/2672391136 wait complete. 2023-10-30T13:11:19.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.800+0000 7fa69d12a640 1 Processor -- start 2023-10-30T13:11:19.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.800+0000 7fa69d12a640 1 -- start start 2023-10-30T13:11:19.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa69d12a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa680003980 0x7fa6981f3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:19.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa69d12a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa698065f90 0x7fa6980ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:19.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa69d12a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6980c9010 0x7fa6981ef4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:19.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa69d12a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6981581f0 con 0x7fa680003980 2023-10-30T13:11:19.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa69d12a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6981e53f0 con 0x7fa698065f90 2023-10-30T13:11:19.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa69d12a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa698150950 con 0x7fa6980c9010 2023-10-30T13:11:19.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa696d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa680003980 0x7fa6981f3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:19.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa696d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa680003980 0x7fa6981f3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38286/0 (socket says 172.21.15.125:38286) 2023-10-30T13:11:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa696d9d640 1 -- 172.21.15.125:0/2178302244 learned_addr learned my addr 172.21.15.125:0/2178302244 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa69659c640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa698065f90 0x7fa6980ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa69759e640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6980c9010 0x7fa6981ef4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa696d9d640 1 -- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6980c9010 msgr2=0x7fa6981ef4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:19.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa696d9d640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6980c9010 0x7fa6981ef4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa696d9d640 1 -- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa698065f90 msgr2=0x7fa6980ce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa696d9d640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa698065f90 0x7fa6980ce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa696d9d640 1 -- 172.21.15.125:0/2178302244 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6981f3870 con 0x7fa680003980 2023-10-30T13:11:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa69759e640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6980c9010 0x7fa6981ef4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:11:19.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa696d9d640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa680003980 0x7fa6981f3330 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fa688025f20 tx=0x7fa680005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa67f7fe640 1 -- 172.21.15.125:0/2178302244 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6880b7070 con 0x7fa680003980 2023-10-30T13:11:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa67f7fe640 1 -- 172.21.15.125:0/2178302244 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa688000ee0 con 0x7fa680003980 2023-10-30T13:11:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa69d12a640 1 -- 172.21.15.125:0/2178302244 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6980c9570 con 0x7fa680003980 2023-10-30T13:11:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa67f7fe640 1 -- 172.21.15.125:0/2178302244 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa688005bb0 con 0x7fa680003980 2023-10-30T13:11:19.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa69d12a640 1 -- 172.21.15.125:0/2178302244 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa698157850 con 0x7fa680003980 2023-10-30T13:11:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.804+0000 7fa69d12a640 1 -- 172.21.15.125:0/2178302244 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69804e170 con 0x7fa680003980 2023-10-30T13:11:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.808+0000 7fa67f7fe640 1 -- 172.21.15.125:0/2178302244 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa688005d10 con 0x7fa680003980 2023-10-30T13:11:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.808+0000 7fa67f7fe640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6840664a0 0x7fa684068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.808+0000 7fa67f7fe640 1 -- 172.21.15.125:0/2178302244 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7fa688120fc0 con 0x7fa680003980 2023-10-30T13:11:19.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.808+0000 7fa69659c640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6840664a0 0x7fa684068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.808+0000 7fa69659c640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6840664a0 0x7fa684068960 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fa698066a00 tx=0x7fa68c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:19.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.812+0000 7fa67f7fe640 1 -- 172.21.15.125:0/2178302244 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6880edbc0 con 0x7fa680003980 2023-10-30T13:11:20.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.996+0000 7fa69d12a640 1 -- 172.21.15.125:0/2178302244 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6980b1dd0 con 0x7fa680003980 2023-10-30T13:11:20.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:19.996+0000 7fa67f7fe640 1 -- 172.21.15.125:0/2178302244 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+6555 (secure 0 0 0) 0x7fa6880f11e0 con 0x7fa680003980 2023-10-30T13:11:20.002 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:20.002 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8443,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:09:32.699420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:active","state_seq":14,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:20.002 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 194 2023-10-30T13:11:20.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:20.000+0000 7fa69d12a640 1 -- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6840664a0 msgr2=0x7fa684068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:20.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:20.000+0000 7fa69d12a640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6840664a0 0x7fa684068960 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fa698066a00 tx=0x7fa68c009040 comp rx=0 tx=0).stop 2023-10-30T13:11:20.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:20.000+0000 7fa69d12a640 1 -- 172.21.15.125:0/2178302244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa680003980 msgr2=0x7fa6981f3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:20.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:20.000+0000 7fa69d12a640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa680003980 0x7fa6981f3330 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fa688025f20 tx=0x7fa680005800 comp rx=0 tx=0).stop 2023-10-30T13:11:20.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:20.000+0000 7fa69d12a640 1 -- 172.21.15.125:0/2178302244 shutdown_connections 2023-10-30T13:11:20.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:20.000+0000 7fa69d12a640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6840664a0 0x7fa684068960 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:20.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:20.000+0000 7fa69d12a640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6980c9010 0x7fa6981ef4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:20.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:20.000+0000 7fa69d12a640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa698065f90 0x7fa6980ce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:20.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:20.000+0000 7fa69d12a640 1 --2- 172.21.15.125:0/2178302244 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa680003980 0x7fa6981f3330 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:20.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:20.000+0000 7fa69d12a640 1 -- 172.21.15.125:0/2178302244 >> 172.21.15.125:0/2178302244 conn(0x7fa69806bfe0 msgr2=0x7fa6981510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:20.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:20.000+0000 7fa69d12a640 1 -- 172.21.15.125:0/2178302244 shutdown_connections 2023-10-30T13:11:20.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:20.000+0000 7fa69d12a640 1 -- 172.21.15.125:0/2178302244 wait complete. 2023-10-30T13:11:20.024 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:11:22.026 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:22.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85f5617640 1 Processor -- start 2023-10-30T13:11:22.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85f5617640 1 -- start start 2023-10-30T13:11:22.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85f5617640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f85f006f7a0 0x7f85f006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:22.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85f5617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85f00c6bd0 0x7f85f0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:22.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85f5617640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f85f0071110 0x7f85f0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:22.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85f5617640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85f00700c0 con 0x7f85f006f7a0 2023-10-30T13:11:22.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85f5617640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85f006a3d0 con 0x7f85f0071110 2023-10-30T13:11:22.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85f5617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f0066cf0 con 0x7f85f00c6bd0 2023-10-30T13:11:22.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85eeffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f85f006f7a0 0x7f85f006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:22.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85eeffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f85f006f7a0 0x7f85f006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38906/0 (socket says 172.21.15.125:38906) 2023-10-30T13:11:22.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85eeffd640 1 -- 172.21.15.125:0/2591276635 learned_addr learned my addr 172.21.15.125:0/2591276635 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:22.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85ef7fe640 1 --2- 172.21.15.125:0/2591276635 >> v2:172.21.15.136:3300/0 conn(0x7f85f0071110 0x7f85f0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:22.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85e7fff640 1 --2- 172.21.15.125:0/2591276635 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85f00c6bd0 0x7f85f0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:22.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85eeffd640 1 -- 172.21.15.125:0/2591276635 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85f00c6bd0 msgr2=0x7f85f0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:22.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85eeffd640 1 --2- 172.21.15.125:0/2591276635 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85f00c6bd0 0x7f85f0069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:22.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85eeffd640 1 -- 172.21.15.125:0/2591276635 >> v2:172.21.15.136:3300/0 conn(0x7f85f0071110 msgr2=0x7f85f0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:22.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85eeffd640 1 --2- 172.21.15.125:0/2591276635 >> v2:172.21.15.136:3300/0 conn(0x7f85f0071110 0x7f85f0158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:22.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.160+0000 7f85eeffd640 1 -- 172.21.15.125:0/2591276635 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f00c70d0 con 0x7f85f006f7a0 2023-10-30T13:11:22.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.164+0000 7f85eeffd640 1 --2- 172.21.15.125:0/2591276635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85f006f7a0 0x7f85f006fb80 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f85f006c7d0 tx=0x7f85e001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0b5d84c2279908 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:22.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.164+0000 7f85ee7fc640 1 -- 172.21.15.125:0/2591276635 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e00a89b0 con 0x7f85f006f7a0 2023-10-30T13:11:22.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.164+0000 7f85ee7fc640 1 -- 172.21.15.125:0/2591276635 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e00a8b10 con 0x7f85f006f7a0 2023-10-30T13:11:22.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.164+0000 7f85ee7fc640 1 -- 172.21.15.125:0/2591276635 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e00a8ea0 con 0x7f85f006f7a0 2023-10-30T13:11:22.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.164+0000 7f85f5617640 1 -- 172.21.15.125:0/2591276635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85f006f7a0 msgr2=0x7f85f006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:22.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.164+0000 7f85f5617640 1 --2- 172.21.15.125:0/2591276635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85f006f7a0 0x7f85f006fb80 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f85f006c7d0 tx=0x7f85e001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:11:22.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.164+0000 7f85f5617640 1 -- 172.21.15.125:0/2591276635 shutdown_connections 2023-10-30T13:11:22.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.164+0000 7f85f5617640 1 --2- 172.21.15.125:0/2591276635 >> v2:172.21.15.136:3300/0 conn(0x7f85f0071110 0x7f85f0158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:22.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.164+0000 7f85f5617640 1 --2- 172.21.15.125:0/2591276635 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85f00c6bd0 0x7f85f0069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:22.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.164+0000 7f85f5617640 1 --2- 172.21.15.125:0/2591276635 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85f006f7a0 0x7f85f006fb80 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:22.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.164+0000 7f85f5617640 1 -- 172.21.15.125:0/2591276635 >> 172.21.15.125:0/2591276635 conn(0x7f85f00c7910 msgr2=0x7f85f00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:22.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.164+0000 7f85f5617640 1 -- 172.21.15.125:0/2591276635 shutdown_connections 2023-10-30T13:11:22.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.164+0000 7f85f5617640 1 -- 172.21.15.125:0/2591276635 wait complete. 2023-10-30T13:11:22.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85f5617640 1 Processor -- start 2023-10-30T13:11:22.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85f5617640 1 -- start start 2023-10-30T13:11:22.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85f5617640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85f006f7a0 0x7f85f00c0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:22.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85f5617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85f0071110 0x7f85f00c05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:22.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85f5617640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85f00c6bd0 0x7f85f01496c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:22.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85f5617640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f014df90 con 0x7f85f00c6bd0 2023-10-30T13:11:22.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85f5617640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f014f880 con 0x7f85f006f7a0 2023-10-30T13:11:22.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85f5617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f014fb60 con 0x7f85f0071110 2023-10-30T13:11:22.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85ef7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85f00c6bd0 0x7f85f01496c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:22.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85ef7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85f00c6bd0 0x7f85f01496c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38910/0 (socket says 172.21.15.125:38910) 2023-10-30T13:11:22.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85ef7fe640 1 -- 172.21.15.125:0/288015133 learned_addr learned my addr 172.21.15.125:0/288015133 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:22.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85e7fff640 1 --2- 172.21.15.125:0/288015133 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85f0071110 0x7f85f00c05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:22.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85eeffd640 1 --2- 172.21.15.125:0/288015133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85f006f7a0 0x7f85f00c0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:22.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85ef7fe640 1 -- 172.21.15.125:0/288015133 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85f0071110 msgr2=0x7f85f00c05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:22.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85ef7fe640 1 --2- 172.21.15.125:0/288015133 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85f0071110 0x7f85f00c05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:22.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85ef7fe640 1 -- 172.21.15.125:0/288015133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85f006f7a0 msgr2=0x7f85f00c0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:22.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85ef7fe640 1 --2- 172.21.15.125:0/288015133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85f006f7a0 0x7f85f00c0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:22.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85ef7fe640 1 -- 172.21.15.125:0/288015133 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e00094d0 con 0x7f85f00c6bd0 2023-10-30T13:11:22.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85ef7fe640 1 --2- 172.21.15.125:0/288015133 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85f00c6bd0 0x7f85f01496c0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f85e8007d20 tx=0x7f85e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:22.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85e77fe640 1 -- 172.21.15.125:0/288015133 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8014420 con 0x7f85f00c6bd0 2023-10-30T13:11:22.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85e77fe640 1 -- 172.21.15.125:0/288015133 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e80145a0 con 0x7f85f00c6bd0 2023-10-30T13:11:22.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85e77fe640 1 -- 172.21.15.125:0/288015133 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8014950 con 0x7f85f00c6bd0 2023-10-30T13:11:22.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85f5617640 1 -- 172.21.15.125:0/288015133 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f014e230 con 0x7f85f00c6bd0 2023-10-30T13:11:22.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.168+0000 7f85f5617640 1 -- 172.21.15.125:0/288015133 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f014fe40 con 0x7f85f00c6bd0 2023-10-30T13:11:22.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.172+0000 7f85f5617640 1 -- 172.21.15.125:0/288015133 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f004e170 con 0x7f85f00c6bd0 2023-10-30T13:11:22.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.172+0000 7f85e77fe640 1 -- 172.21.15.125:0/288015133 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f85e8014ab0 con 0x7f85f00c6bd0 2023-10-30T13:11:22.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.172+0000 7f85e77fe640 1 --2- 172.21.15.125:0/288015133 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85d00664a0 0x7f85d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:22.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.172+0000 7f85e77fe640 1 -- 172.21.15.125:0/288015133 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7f85e8084e60 con 0x7f85f00c6bd0 2023-10-30T13:11:22.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.172+0000 7f85eeffd640 1 --2- 172.21.15.125:0/288015133 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85d00664a0 0x7f85d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:22.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.172+0000 7f85eeffd640 1 --2- 172.21.15.125:0/288015133 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85d00664a0 0x7f85d0068960 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f85e00b2000 tx=0x7f85e0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:22.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.176+0000 7f85e77fe640 1 -- 172.21.15.125:0/288015133 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f85e8051b20 con 0x7f85f00c6bd0 2023-10-30T13:11:22.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.360+0000 7f85f5617640 1 -- 172.21.15.125:0/288015133 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85f006c5c0 con 0x7f85f00c6bd0 2023-10-30T13:11:22.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.360+0000 7f85e77fe640 1 -- 172.21.15.125:0/288015133 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 194 v194) v1 ==== 95+0+6555 (secure 0 0 0) 0x7f85e8055140 con 0x7f85f00c6bd0 2023-10-30T13:11:22.367 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:22.367 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":194,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8443,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":194,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:09:32.699420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:active","state_seq":14,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:22.367 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 194 2023-10-30T13:11:22.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.364+0000 7f85f5617640 1 -- 172.21.15.125:0/288015133 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85d00664a0 msgr2=0x7f85d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:22.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.364+0000 7f85f5617640 1 --2- 172.21.15.125:0/288015133 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85d00664a0 0x7f85d0068960 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f85e00b2000 tx=0x7f85e0009040 comp rx=0 tx=0).stop 2023-10-30T13:11:22.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.364+0000 7f85f5617640 1 -- 172.21.15.125:0/288015133 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85f00c6bd0 msgr2=0x7f85f01496c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:22.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.364+0000 7f85f5617640 1 --2- 172.21.15.125:0/288015133 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85f00c6bd0 0x7f85f01496c0 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f85e8007d20 tx=0x7f85e800e4b0 comp rx=0 tx=0).stop 2023-10-30T13:11:22.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.364+0000 7f85f5617640 1 -- 172.21.15.125:0/288015133 shutdown_connections 2023-10-30T13:11:22.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.364+0000 7f85f5617640 1 --2- 172.21.15.125:0/288015133 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f85d00664a0 0x7f85d0068960 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:22.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.364+0000 7f85f5617640 1 --2- 172.21.15.125:0/288015133 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f85f00c6bd0 0x7f85f01496c0 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:22.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.364+0000 7f85f5617640 1 --2- 172.21.15.125:0/288015133 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f85f0071110 0x7f85f00c05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:22.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.364+0000 7f85f5617640 1 --2- 172.21.15.125:0/288015133 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f85f006f7a0 0x7f85f00c0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:22.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.364+0000 7f85f5617640 1 -- 172.21.15.125:0/288015133 >> 172.21.15.125:0/288015133 conn(0x7f85f00c7910 msgr2=0x7f85f00c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:22.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.364+0000 7f85f5617640 1 -- 172.21.15.125:0/288015133 shutdown_connections 2023-10-30T13:11:22.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:22.364+0000 7f85f5617640 1 -- 172.21.15.125:0/288015133 wait complete. 2023-10-30T13:11:22.388 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:11:24.389 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:24.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f4590c17640 1 Processor -- start 2023-10-30T13:11:24.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f4590c17640 1 -- start start 2023-10-30T13:11:24.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f4590c17640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f458c1539a0 0x7f458c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:24.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f4590c17640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f458c069e90 0x7f458c06a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:24.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f4590c17640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f458c15a8c0 con 0x7f458c070a00 2023-10-30T13:11:24.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f4590c17640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f458c06fbb0 con 0x7f458c1539a0 2023-10-30T13:11:24.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f4590c17640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f458c153d80 con 0x7f458c069e90 2023-10-30T13:11:24.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f458a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f458c1539a0 0x7f458c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:24.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f458a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f458c1539a0 0x7f458c15a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38916/0 (socket says 172.21.15.125:38916) 2023-10-30T13:11:24.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f458a59c640 1 -- 172.21.15.125:0/2425782194 learned_addr learned my addr 172.21.15.125:0/2425782194 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:24.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f458ad9d640 1 --2- 172.21.15.125:0/2425782194 >> v2:172.21.15.136:3300/0 conn(0x7f458c069e90 0x7f458c06a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:24.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f458a59c640 1 -- 172.21.15.125:0/2425782194 >> v2:172.21.15.136:3300/0 conn(0x7f458c069e90 msgr2=0x7f458c06a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:24.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f458a59c640 1 --2- 172.21.15.125:0/2425782194 >> v2:172.21.15.136:3300/0 conn(0x7f458c069e90 0x7f458c06a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:24.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f458a59c640 1 -- 172.21.15.125:0/2425782194 >> v1:172.21.15.136:6789/0 conn(0x7f458c070a00 legacy=0x7f458c070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:24.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.524+0000 7f458a59c640 1 -- 172.21.15.125:0/2425782194 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458c0c5c50 con 0x7f458c1539a0 2023-10-30T13:11:24.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.528+0000 7f458a59c640 1 --2- 172.21.15.125:0/2425782194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458c1539a0 0x7f458c15a380 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f458c0c8900 tx=0x7f457c01c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c797edd770a36404 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:24.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.528+0000 7f458959a640 1 -- 172.21.15.125:0/2425782194 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c0ac640 con 0x7f458c1539a0 2023-10-30T13:11:24.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.528+0000 7f458959a640 1 -- 172.21.15.125:0/2425782194 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c0ac7a0 con 0x7f458c1539a0 2023-10-30T13:11:24.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.528+0000 7f458959a640 1 -- 172.21.15.125:0/2425782194 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c0acb30 con 0x7f458c1539a0 2023-10-30T13:11:24.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.528+0000 7f4590c17640 1 -- 172.21.15.125:0/2425782194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458c1539a0 msgr2=0x7f458c15a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:24.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.528+0000 7f4590c17640 1 --2- 172.21.15.125:0/2425782194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458c1539a0 0x7f458c15a380 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f458c0c8900 tx=0x7f457c01c400 comp rx=0 tx=0).stop 2023-10-30T13:11:24.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.528+0000 7f4590c17640 1 -- 172.21.15.125:0/2425782194 shutdown_connections 2023-10-30T13:11:24.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.528+0000 7f4590c17640 1 --2- 172.21.15.125:0/2425782194 >> v2:172.21.15.136:3300/0 conn(0x7f458c069e90 0x7f458c06a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:24.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.528+0000 7f4590c17640 1 --2- 172.21.15.125:0/2425782194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458c1539a0 0x7f458c15a380 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:24.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.528+0000 7f4590c17640 1 -- 172.21.15.125:0/2425782194 >> 172.21.15.125:0/2425782194 conn(0x7f458c0c7910 msgr2=0x7f458c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:24.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.528+0000 7f4590c17640 1 -- 172.21.15.125:0/2425782194 shutdown_connections 2023-10-30T13:11:24.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.528+0000 7f4590c17640 1 -- 172.21.15.125:0/2425782194 wait complete. 2023-10-30T13:11:24.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f4590c17640 1 Processor -- start 2023-10-30T13:11:24.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f4590c17640 1 -- start start 2023-10-30T13:11:24.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f4590c17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458c069e90 0x7f458c158a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:24.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f4590c17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f458c070a00 0x7f458c1439f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:24.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f4590c17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f458c1539a0 0x7f458c0c1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:24.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f4590c17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c0c9c60 con 0x7f458c069e90 2023-10-30T13:11:24.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f4590c17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c157e90 con 0x7f458c1539a0 2023-10-30T13:11:24.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f4590c17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458c1585d0 con 0x7f458c070a00 2023-10-30T13:11:24.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f458a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458c069e90 0x7f458c158a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:24.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f458a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458c069e90 0x7f458c158a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38926/0 (socket says 172.21.15.125:38926) 2023-10-30T13:11:24.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f458a59c640 1 -- 172.21.15.125:0/2111206956 learned_addr learned my addr 172.21.15.125:0/2111206956 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:24.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f458ad9d640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f458c1539a0 0x7f458c0c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:24.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f4589d9b640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f458c070a00 0x7f458c1439f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:24.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f458a59c640 1 -- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f458c070a00 msgr2=0x7f458c1439f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:24.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f458a59c640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f458c070a00 0x7f458c1439f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:24.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f458a59c640 1 -- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f458c1539a0 msgr2=0x7f458c0c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:24.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f458a59c640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f458c1539a0 0x7f458c0c1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:24.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f458a59c640 1 -- 172.21.15.125:0/2111206956 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c0094d0 con 0x7f458c069e90 2023-10-30T13:11:24.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f4589d9b640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f458c070a00 0x7f458c1439f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:11:24.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f458ad9d640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f458c1539a0 0x7f458c0c1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:11:24.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f458a59c640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458c069e90 0x7f458c158a60 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f457c0af040 tx=0x7f457c0191a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:24.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f457affd640 1 -- 172.21.15.125:0/2111206956 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c0b5640 con 0x7f458c069e90 2023-10-30T13:11:24.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f457affd640 1 -- 172.21.15.125:0/2111206956 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c0b57c0 con 0x7f458c069e90 2023-10-30T13:11:24.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f4590c17640 1 -- 172.21.15.125:0/2111206956 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458c157910 con 0x7f458c069e90 2023-10-30T13:11:24.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f457affd640 1 -- 172.21.15.125:0/2111206956 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c0b5b70 con 0x7f458c069e90 2023-10-30T13:11:24.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.532+0000 7f4590c17640 1 -- 172.21.15.125:0/2111206956 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458c157bb0 con 0x7f458c069e90 2023-10-30T13:11:24.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.536+0000 7f4590c17640 1 -- 172.21.15.125:0/2111206956 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458c04e170 con 0x7f458c069e90 2023-10-30T13:11:24.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.536+0000 7f457affd640 1 -- 172.21.15.125:0/2111206956 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f457c0b3070 con 0x7f458c069e90 2023-10-30T13:11:24.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.536+0000 7f457affd640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f456c0664d0 0x7f456c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:24.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.536+0000 7f457affd640 1 -- 172.21.15.125:0/2111206956 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7f457c11e9c0 con 0x7f458c069e90 2023-10-30T13:11:24.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.536+0000 7f4589d9b640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f456c0664d0 0x7f456c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:24.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.536+0000 7f4589d9b640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f456c0664d0 0x7f456c068990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f458c0c53e0 tx=0x7f4580009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:24.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.540+0000 7f457affd640 1 -- 172.21.15.125:0/2111206956 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f457c0eb670 con 0x7f458c069e90 2023-10-30T13:11:24.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.724+0000 7f4590c17640 1 -- 172.21.15.125:0/2111206956 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f458c14da30 con 0x7f458c069e90 2023-10-30T13:11:24.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.724+0000 7f457affd640 1 -- 172.21.15.125:0/2111206956 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+6553 (secure 0 0 0) 0x7f457c0eec90 con 0x7f458c069e90 2023-10-30T13:11:24.732 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:24.733 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8443,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:22.662866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:active","state_seq":14,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:24.733 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 195 2023-10-30T13:11:24.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.728+0000 7f4590c17640 1 -- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f456c0664d0 msgr2=0x7f456c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:24.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.728+0000 7f4590c17640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f456c0664d0 0x7f456c068990 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f458c0c53e0 tx=0x7f4580009040 comp rx=0 tx=0).stop 2023-10-30T13:11:24.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.728+0000 7f4590c17640 1 -- 172.21.15.125:0/2111206956 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458c069e90 msgr2=0x7f458c158a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:24.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.728+0000 7f4590c17640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458c069e90 0x7f458c158a60 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f457c0af040 tx=0x7f457c0191a0 comp rx=0 tx=0).stop 2023-10-30T13:11:24.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.728+0000 7f4590c17640 1 -- 172.21.15.125:0/2111206956 shutdown_connections 2023-10-30T13:11:24.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.728+0000 7f4590c17640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f456c0664d0 0x7f456c068990 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:24.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.728+0000 7f4590c17640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f458c1539a0 0x7f458c0c1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:24.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.728+0000 7f4590c17640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f458c070a00 0x7f458c1439f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:24.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.728+0000 7f4590c17640 1 --2- 172.21.15.125:0/2111206956 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f458c069e90 0x7f458c158a60 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:24.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.728+0000 7f4590c17640 1 -- 172.21.15.125:0/2111206956 >> 172.21.15.125:0/2111206956 conn(0x7f458c0c7910 msgr2=0x7f458c06b340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:24.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.728+0000 7f4590c17640 1 -- 172.21.15.125:0/2111206956 shutdown_connections 2023-10-30T13:11:24.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:24.728+0000 7f4590c17640 1 -- 172.21.15.125:0/2111206956 wait complete. 2023-10-30T13:11:24.753 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:11:26.754 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:26.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f5732b9d640 1 Processor -- start 2023-10-30T13:11:26.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f5732b9d640 1 -- start start 2023-10-30T13:11:26.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f5732b9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f572c06f7a0 0x7f572c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:26.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f5732b9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f572c071110 0x7f572c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:26.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f5732b9d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572c066cf0 con 0x7f572c0c6bd0 2023-10-30T13:11:26.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f5732b9d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f572c0700c0 con 0x7f572c071110 2023-10-30T13:11:26.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f5732b9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f572c0c70d0 con 0x7f572c06f7a0 2023-10-30T13:11:26.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f573113a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f572c071110 0x7f572c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:26.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f573113a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f572c071110 0x7f572c158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38942/0 (socket says 172.21.15.125:38942) 2023-10-30T13:11:26.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f573113a640 1 -- 172.21.15.125:0/1025880013 learned_addr learned my addr 172.21.15.125:0/1025880013 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:26.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f5730939640 1 --2- 172.21.15.125:0/1025880013 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f572c06f7a0 0x7f572c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:26.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f573113a640 1 -- 172.21.15.125:0/1025880013 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f572c06f7a0 msgr2=0x7f572c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:26.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f573113a640 1 --2- 172.21.15.125:0/1025880013 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f572c06f7a0 0x7f572c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:26.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f573113a640 1 -- 172.21.15.125:0/1025880013 >> v1:172.21.15.136:6789/0 conn(0x7f572c0c6bd0 legacy=0x7f572c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:26.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.888+0000 7f573113a640 1 -- 172.21.15.125:0/1025880013 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572c0c7390 con 0x7f572c071110 2023-10-30T13:11:26.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.892+0000 7f573113a640 1 --2- 172.21.15.125:0/1025880013 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f572c071110 0x7f572c158110 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f5724009e50 tx=0x7f572401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283ab3052a93fb02 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:26.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.892+0000 7f572b7fe640 1 -- 172.21.15.125:0/1025880013 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57240af400 con 0x7f572c071110 2023-10-30T13:11:26.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.892+0000 7f572b7fe640 1 -- 172.21.15.125:0/1025880013 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57240ab050 con 0x7f572c071110 2023-10-30T13:11:26.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.892+0000 7f572b7fe640 1 -- 172.21.15.125:0/1025880013 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57240af780 con 0x7f572c071110 2023-10-30T13:11:26.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.892+0000 7f5732b9d640 1 -- 172.21.15.125:0/1025880013 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f572c071110 msgr2=0x7f572c158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:26.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.892+0000 7f5732b9d640 1 --2- 172.21.15.125:0/1025880013 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f572c071110 0x7f572c158110 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f5724009e50 tx=0x7f572401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:11:26.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.892+0000 7f5732b9d640 1 -- 172.21.15.125:0/1025880013 shutdown_connections 2023-10-30T13:11:26.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.892+0000 7f5732b9d640 1 --2- 172.21.15.125:0/1025880013 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f572c071110 0x7f572c158110 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:26.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.892+0000 7f5732b9d640 1 --2- 172.21.15.125:0/1025880013 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f572c06f7a0 0x7f572c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:26.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.892+0000 7f5732b9d640 1 -- 172.21.15.125:0/1025880013 >> 172.21.15.125:0/1025880013 conn(0x7f572c0c7910 msgr2=0x7f572c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:26.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.892+0000 7f5732b9d640 1 -- 172.21.15.125:0/1025880013 shutdown_connections 2023-10-30T13:11:26.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.892+0000 7f5732b9d640 1 -- 172.21.15.125:0/1025880013 wait complete. 2023-10-30T13:11:26.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.892+0000 7f5732b9d640 1 Processor -- start 2023-10-30T13:11:26.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f5732b9d640 1 -- start start 2023-10-30T13:11:26.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f5732b9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f572c06f7a0 0x7f572c1ed120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:26.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f5732b9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f572c071110 0x7f572c152d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:26.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f5732b9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f572c0c6bd0 0x7f572c15b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:26.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f5732b9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572c1550d0 con 0x7f572c071110 2023-10-30T13:11:26.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f5732b9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572c1ed660 con 0x7f572c0c6bd0 2023-10-30T13:11:26.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f5732b9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f572c0c94e0 con 0x7f572c06f7a0 2023-10-30T13:11:26.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f572bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f572c071110 0x7f572c152d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:26.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f572bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f572c071110 0x7f572c152d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38948/0 (socket says 172.21.15.125:38948) 2023-10-30T13:11:26.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f572bfff640 1 -- 172.21.15.125:0/3054215774 learned_addr learned my addr 172.21.15.125:0/3054215774 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:26.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f573113a640 1 --2- 172.21.15.125:0/3054215774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f572c0c6bd0 0x7f572c15b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:26.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f5730939640 1 --2- 172.21.15.125:0/3054215774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f572c06f7a0 0x7f572c1ed120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:26.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f572bfff640 1 -- 172.21.15.125:0/3054215774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f572c06f7a0 msgr2=0x7f572c1ed120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:26.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f572bfff640 1 --2- 172.21.15.125:0/3054215774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f572c06f7a0 0x7f572c1ed120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:26.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f572bfff640 1 -- 172.21.15.125:0/3054215774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f572c0c6bd0 msgr2=0x7f572c15b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:26.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f572bfff640 1 --2- 172.21.15.125:0/3054215774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f572c0c6bd0 0x7f572c15b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:26.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f572bfff640 1 -- 172.21.15.125:0/3054215774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5724009b30 con 0x7f572c071110 2023-10-30T13:11:26.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f572bfff640 1 --2- 172.21.15.125:0/3054215774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f572c071110 0x7f572c152d50 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f571800c530 tx=0x7f571800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:26.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f57297fa640 1 -- 172.21.15.125:0/3054215774 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5718005220 con 0x7f572c071110 2023-10-30T13:11:26.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f57297fa640 1 -- 172.21.15.125:0/3054215774 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5718005380 con 0x7f572c071110 2023-10-30T13:11:26.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f57297fa640 1 -- 172.21.15.125:0/3054215774 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5718007270 con 0x7f572c071110 2023-10-30T13:11:26.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f5732b9d640 1 -- 172.21.15.125:0/3054215774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f572c154b10 con 0x7f572c071110 2023-10-30T13:11:26.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f5732b9d640 1 -- 172.21.15.125:0/3054215774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f572c152270 con 0x7f572c071110 2023-10-30T13:11:26.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.896+0000 7f5732b9d640 1 -- 172.21.15.125:0/3054215774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f572c0c4590 con 0x7f572c071110 2023-10-30T13:11:26.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.900+0000 7f57297fa640 1 -- 172.21.15.125:0/3054215774 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f57180073d0 con 0x7f572c071110 2023-10-30T13:11:26.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.900+0000 7f57297fa640 1 --2- 172.21.15.125:0/3054215774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57100664a0 0x7f5710068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:26.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.900+0000 7f57297fa640 1 -- 172.21.15.125:0/3054215774 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7f5718083d80 con 0x7f572c071110 2023-10-30T13:11:26.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.900+0000 7f5730939640 1 --2- 172.21.15.125:0/3054215774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57100664a0 0x7f5710068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:26.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.900+0000 7f5730939640 1 --2- 172.21.15.125:0/3054215774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57100664a0 0x7f5710068960 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f57200079c0 tx=0x7f5720008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:26.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:26.904+0000 7f57297fa640 1 -- 172.21.15.125:0/3054215774 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5718050b60 con 0x7f572c071110 2023-10-30T13:11:27.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:27.088+0000 7f5732b9d640 1 -- 172.21.15.125:0/3054215774 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f572c04e170 con 0x7f572c071110 2023-10-30T13:11:27.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:27.088+0000 7f57297fa640 1 -- 172.21.15.125:0/3054215774 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+6553 (secure 0 0 0) 0x7f5718054180 con 0x7f572c071110 2023-10-30T13:11:27.094 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:27.095 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8443,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:22.662866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:active","state_seq":14,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:27.095 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 195 2023-10-30T13:11:27.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:27.088+0000 7f5732b9d640 1 -- 172.21.15.125:0/3054215774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57100664a0 msgr2=0x7f5710068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:27.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:27.088+0000 7f5732b9d640 1 --2- 172.21.15.125:0/3054215774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57100664a0 0x7f5710068960 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f57200079c0 tx=0x7f5720008040 comp rx=0 tx=0).stop 2023-10-30T13:11:27.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:27.088+0000 7f5732b9d640 1 -- 172.21.15.125:0/3054215774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f572c071110 msgr2=0x7f572c152d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:27.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:27.088+0000 7f5732b9d640 1 --2- 172.21.15.125:0/3054215774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f572c071110 0x7f572c152d50 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f571800c530 tx=0x7f571800c9f0 comp rx=0 tx=0).stop 2023-10-30T13:11:27.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:27.088+0000 7f5732b9d640 1 -- 172.21.15.125:0/3054215774 shutdown_connections 2023-10-30T13:11:27.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:27.088+0000 7f5732b9d640 1 --2- 172.21.15.125:0/3054215774 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f57100664a0 0x7f5710068960 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:27.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:27.088+0000 7f5732b9d640 1 --2- 172.21.15.125:0/3054215774 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f572c0c6bd0 0x7f572c15b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:27.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:27.088+0000 7f5732b9d640 1 --2- 172.21.15.125:0/3054215774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f572c071110 0x7f572c152d50 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:27.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:27.088+0000 7f5732b9d640 1 --2- 172.21.15.125:0/3054215774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f572c06f7a0 0x7f572c1ed120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:27.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:27.088+0000 7f5732b9d640 1 -- 172.21.15.125:0/3054215774 >> 172.21.15.125:0/3054215774 conn(0x7f572c0c7910 msgr2=0x7f572c0c3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:27.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:27.088+0000 7f5732b9d640 1 -- 172.21.15.125:0/3054215774 shutdown_connections 2023-10-30T13:11:27.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:27.088+0000 7f5732b9d640 1 -- 172.21.15.125:0/3054215774 wait complete. 2023-10-30T13:11:27.115 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:11:29.117 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:29.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.248+0000 7f7affcea640 1 Processor -- start 2023-10-30T13:11:29.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.248+0000 7f7affcea640 1 -- start start 2023-10-30T13:11:29.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7affcea640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7af8159e60 0x7f7af806f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:29.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7affcea640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7af80bfec0 0x7f7af80c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:29.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7affcea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7af806a720 0x7f7af806abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:29.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7affcea640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af806fa80 con 0x7f7af80bfec0 2023-10-30T13:11:29.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7affcea640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af8071290 con 0x7f7af8159e60 2023-10-30T13:11:29.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7affcea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af8071570 con 0x7f7af806a720 2023-10-30T13:11:29.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7afd285640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7af80bfec0 0x7f7af80c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:29.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7afd285640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7af80bfec0 0x7f7af80c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38954/0 (socket says 172.21.15.125:38954) 2023-10-30T13:11:29.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7afd285640 1 -- 172.21.15.125:0/3009606658 learned_addr learned my addr 172.21.15.125:0/3009606658 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:29.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7afe287640 1 --2- 172.21.15.125:0/3009606658 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7af806a720 0x7f7af806abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:29.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7afda86640 1 --2- 172.21.15.125:0/3009606658 >> v2:172.21.15.136:3300/0 conn(0x7f7af8159e60 0x7f7af806f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:29.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7afd285640 1 -- 172.21.15.125:0/3009606658 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7af806a720 msgr2=0x7f7af806abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:29.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7afd285640 1 --2- 172.21.15.125:0/3009606658 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7af806a720 0x7f7af806abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:29.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7afd285640 1 -- 172.21.15.125:0/3009606658 >> v2:172.21.15.136:3300/0 conn(0x7f7af8159e60 msgr2=0x7f7af806f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:29.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7afd285640 1 --2- 172.21.15.125:0/3009606658 >> v2:172.21.15.136:3300/0 conn(0x7f7af8159e60 0x7f7af806f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:29.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7afd285640 1 -- 172.21.15.125:0/3009606658 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af8071850 con 0x7f7af80bfec0 2023-10-30T13:11:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.252+0000 7f7afd285640 1 --2- 172.21.15.125:0/3009606658 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af80bfec0 0x7f7af80c1a40 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f7af80c8950 tx=0x7f7ae8020f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e396b895eb4711 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:29.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7afca84640 1 -- 172.21.15.125:0/3009606658 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae80aabc0 con 0x7f7af80bfec0 2023-10-30T13:11:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7afca84640 1 -- 172.21.15.125:0/3009606658 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae80aad20 con 0x7f7af80bfec0 2023-10-30T13:11:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7afca84640 1 -- 172.21.15.125:0/3009606658 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae80b2660 con 0x7f7af80bfec0 2023-10-30T13:11:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 -- 172.21.15.125:0/3009606658 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af80bfec0 msgr2=0x7f7af80c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 --2- 172.21.15.125:0/3009606658 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af80bfec0 0x7f7af80c1a40 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f7af80c8950 tx=0x7f7ae8020f00 comp rx=0 tx=0).stop 2023-10-30T13:11:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 -- 172.21.15.125:0/3009606658 shutdown_connections 2023-10-30T13:11:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 --2- 172.21.15.125:0/3009606658 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7af806a720 0x7f7af806abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 --2- 172.21.15.125:0/3009606658 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af80bfec0 0x7f7af80c1a40 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 --2- 172.21.15.125:0/3009606658 >> v2:172.21.15.136:3300/0 conn(0x7f7af8159e60 0x7f7af806f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 -- 172.21.15.125:0/3009606658 >> 172.21.15.125:0/3009606658 conn(0x7f7af80c7910 msgr2=0x7f7af80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:29.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 -- 172.21.15.125:0/3009606658 shutdown_connections 2023-10-30T13:11:29.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 -- 172.21.15.125:0/3009606658 wait complete. 2023-10-30T13:11:29.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 Processor -- start 2023-10-30T13:11:29.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 -- start start 2023-10-30T13:11:29.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7af806a720 0x7f7af8159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:29.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af80bfec0 0x7f7af80c1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:29.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7af8159e60 0x7f7af80c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af81f2d80 con 0x7f7af80bfec0 2023-10-30T13:11:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af81f2140 con 0x7f7af8159e60 2023-10-30T13:11:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7affcea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af81f23e0 con 0x7f7af806a720 2023-10-30T13:11:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7afd285640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af80bfec0 0x7f7af80c1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7afd285640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af80bfec0 0x7f7af80c1f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38964/0 (socket says 172.21.15.125:38964) 2023-10-30T13:11:29.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7afd285640 1 -- 172.21.15.125:0/4171173363 learned_addr learned my addr 172.21.15.125:0/4171173363 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:29.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.256+0000 7f7afda86640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7af806a720 0x7f7af8159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:29.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7afe287640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7af8159e60 0x7f7af80c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:29.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7afd285640 1 -- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7af806a720 msgr2=0x7f7af8159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:29.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7afd285640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7af806a720 0x7f7af8159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:29.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7afd285640 1 -- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7af8159e60 msgr2=0x7f7af80c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:29.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7afd285640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7af8159e60 0x7f7af80c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:29.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7afd285640 1 -- 172.21.15.125:0/4171173363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae80094d0 con 0x7f7af80bfec0 2023-10-30T13:11:29.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7afda86640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7af806a720 0x7f7af8159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:11:29.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7afd285640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af80bfec0 0x7f7af80c1f00 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80b5040 tx=0x7f7ae801dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:29.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7aee7fc640 1 -- 172.21.15.125:0/4171173363 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae80b2e00 con 0x7f7af80bfec0 2023-10-30T13:11:29.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7aee7fc640 1 -- 172.21.15.125:0/4171173363 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae80aae90 con 0x7f7af80bfec0 2023-10-30T13:11:29.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7affcea640 1 -- 172.21.15.125:0/4171173363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af81f26c0 con 0x7f7af80bfec0 2023-10-30T13:11:29.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7aee7fc640 1 -- 172.21.15.125:0/4171173363 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae80bd6a0 con 0x7f7af80bfec0 2023-10-30T13:11:29.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7affcea640 1 -- 172.21.15.125:0/4171173363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af80c25b0 con 0x7f7af80bfec0 2023-10-30T13:11:29.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7affcea640 1 -- 172.21.15.125:0/4171173363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af804e170 con 0x7f7af80bfec0 2023-10-30T13:11:29.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7aee7fc640 1 -- 172.21.15.125:0/4171173363 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7ae80c5360 con 0x7f7af80bfec0 2023-10-30T13:11:29.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7aee7fc640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7adc0664d0 0x7f7adc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:29.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.260+0000 7f7aee7fc640 1 -- 172.21.15.125:0/4171173363 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7716+0+0 (secure 0 0 0) 0x7f7ae8121ef0 con 0x7f7af80bfec0 2023-10-30T13:11:29.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.264+0000 7f7afda86640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7adc0664d0 0x7f7adc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:29.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.264+0000 7f7afda86640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7adc0664d0 0x7f7adc068990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7af400b9f0 tx=0x7f7af4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:29.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.264+0000 7f7aee7fc640 1 -- 172.21.15.125:0/4171173363 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7ae80eea60 con 0x7f7af80bfec0 2023-10-30T13:11:29.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.448+0000 7f7affcea640 1 -- 172.21.15.125:0/4171173363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7af81f2520 con 0x7f7af80bfec0 2023-10-30T13:11:29.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.448+0000 7f7aee7fc640 1 -- 172.21.15.125:0/4171173363 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+6553 (secure 0 0 0) 0x7f7af81f2520 con 0x7f7af80bfec0 2023-10-30T13:11:29.453 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:29.454 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8443,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":192}],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:22.662866+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":7340},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7340":{"gid":7340,"name":"d","rank":2,"incarnation":187,"state":"up:active","state_seq":14,"addr":"172.21.15.136:6835/3526499585","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3526499585},{"type":"v1","addr":"172.21.15.136:6835","nonce":3526499585}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:29.454 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 195 2023-10-30T13:11:29.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.448+0000 7f7affcea640 1 -- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7adc0664d0 msgr2=0x7f7adc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:29.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.448+0000 7f7affcea640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7adc0664d0 0x7f7adc068990 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7af400b9f0 tx=0x7f7af4008040 comp rx=0 tx=0).stop 2023-10-30T13:11:29.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.448+0000 7f7affcea640 1 -- 172.21.15.125:0/4171173363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af80bfec0 msgr2=0x7f7af80c1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:29.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.448+0000 7f7affcea640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af80bfec0 0x7f7af80c1f00 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f7ae80b5040 tx=0x7f7ae801dca0 comp rx=0 tx=0).stop 2023-10-30T13:11:29.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.448+0000 7f7affcea640 1 -- 172.21.15.125:0/4171173363 shutdown_connections 2023-10-30T13:11:29.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.448+0000 7f7affcea640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7adc0664d0 0x7f7adc068990 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:29.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.448+0000 7f7affcea640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7af8159e60 0x7f7af80c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:29.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.448+0000 7f7affcea640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7af80bfec0 0x7f7af80c1f00 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:29.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.448+0000 7f7affcea640 1 --2- 172.21.15.125:0/4171173363 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7af806a720 0x7f7af8159220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:29.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.448+0000 7f7affcea640 1 -- 172.21.15.125:0/4171173363 >> 172.21.15.125:0/4171173363 conn(0x7f7af80c7910 msgr2=0x7f7af80ca530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:29.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.448+0000 7f7affcea640 1 -- 172.21.15.125:0/4171173363 shutdown_connections 2023-10-30T13:11:29.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:29.448+0000 7f7affcea640 1 -- 172.21.15.125:0/4171173363 wait complete. 2023-10-30T13:11:29.474 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:11:31.476 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:31.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.612+0000 7f4dad0a9640 1 Processor -- start 2023-10-30T13:11:31.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.612+0000 7f4dad0a9640 1 -- start start 2023-10-30T13:11:31.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.612+0000 7f4dad0a9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4da81538a0 0x7f4da815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:31.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.612+0000 7f4dad0a9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4da80ca0e0 0x7f4da815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:31.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.612+0000 7f4dad0a9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da80cb800 con 0x7f4da80cf160 2023-10-30T13:11:31.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.612+0000 7f4dad0a9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4da815dc30 con 0x7f4da81538a0 2023-10-30T13:11:31.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.612+0000 7f4dad0a9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4da80cb330 con 0x7f4da80ca0e0 2023-10-30T13:11:31.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.612+0000 7f4da6d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4da81538a0 0x7f4da815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:31.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.612+0000 7f4da6d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4da81538a0 0x7f4da815d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34422/0 (socket says 172.21.15.125:34422) 2023-10-30T13:11:31.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.612+0000 7f4da6d9d640 1 -- 172.21.15.125:0/46491540 learned_addr learned my addr 172.21.15.125:0/46491540 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:31.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.612+0000 7f4da659c640 1 --2- 172.21.15.125:0/46491540 >> v2:172.21.15.136:3300/0 conn(0x7f4da80ca0e0 0x7f4da815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:31.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.616+0000 7f4da6d9d640 1 -- 172.21.15.125:0/46491540 >> v2:172.21.15.136:3300/0 conn(0x7f4da80ca0e0 msgr2=0x7f4da815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:31.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.616+0000 7f4da6d9d640 1 --2- 172.21.15.125:0/46491540 >> v2:172.21.15.136:3300/0 conn(0x7f4da80ca0e0 0x7f4da815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.616+0000 7f4da6d9d640 1 -- 172.21.15.125:0/46491540 >> v1:172.21.15.136:6789/0 conn(0x7f4da80cf160 legacy=0x7f4da80cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:31.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.616+0000 7f4da6d9d640 1 -- 172.21.15.125:0/46491540 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da80cad70 con 0x7f4da81538a0 2023-10-30T13:11:31.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.616+0000 7f4da6d9d640 1 --2- 172.21.15.125:0/46491540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4da81538a0 0x7f4da815d6f0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f4da80b30a0 tx=0x7f4d9801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=779c0067e74f18ff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:31.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.616+0000 7f4da5d9b640 1 -- 172.21.15.125:0/46491540 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d980a89f0 con 0x7f4da81538a0 2023-10-30T13:11:31.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.616+0000 7f4da5d9b640 1 -- 172.21.15.125:0/46491540 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d980a8b50 con 0x7f4da81538a0 2023-10-30T13:11:31.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.616+0000 7f4da5d9b640 1 -- 172.21.15.125:0/46491540 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d980b0440 con 0x7f4da81538a0 2023-10-30T13:11:31.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 -- 172.21.15.125:0/46491540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4da81538a0 msgr2=0x7f4da815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:31.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 --2- 172.21.15.125:0/46491540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4da81538a0 0x7f4da815d6f0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f4da80b30a0 tx=0x7f4d9801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 -- 172.21.15.125:0/46491540 shutdown_connections 2023-10-30T13:11:31.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 --2- 172.21.15.125:0/46491540 >> v2:172.21.15.136:3300/0 conn(0x7f4da80ca0e0 0x7f4da815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 --2- 172.21.15.125:0/46491540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4da81538a0 0x7f4da815d6f0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 -- 172.21.15.125:0/46491540 >> 172.21.15.125:0/46491540 conn(0x7f4da806bfe0 msgr2=0x7f4da806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:31.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 -- 172.21.15.125:0/46491540 shutdown_connections 2023-10-30T13:11:31.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 -- 172.21.15.125:0/46491540 wait complete. 2023-10-30T13:11:31.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 Processor -- start 2023-10-30T13:11:31.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 -- start start 2023-10-30T13:11:31.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4da80ca0e0 0x7f4da8158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:31.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4da80cf160 0x7f4da80c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:31.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da81538a0 0x7f4da81ef9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:31.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da81e4db0 con 0x7f4da80cf160 2023-10-30T13:11:31.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da81ef520 con 0x7f4da81538a0 2023-10-30T13:11:31.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4dad0a9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da81f2960 con 0x7f4da80ca0e0 2023-10-30T13:11:31.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4da659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4da80cf160 0x7f4da80c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:31.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4da659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4da80cf160 0x7f4da80c5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34438/0 (socket says 172.21.15.125:34438) 2023-10-30T13:11:31.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4da659c640 1 -- 172.21.15.125:0/3231941535 learned_addr learned my addr 172.21.15.125:0/3231941535 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:31.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4da6d9d640 1 --2- 172.21.15.125:0/3231941535 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4da80ca0e0 0x7f4da8158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:31.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.620+0000 7f4da759e640 1 --2- 172.21.15.125:0/3231941535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da81538a0 0x7f4da81ef9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:31.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4da659c640 1 -- 172.21.15.125:0/3231941535 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4da80ca0e0 msgr2=0x7f4da8158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:31.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4da659c640 1 --2- 172.21.15.125:0/3231941535 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4da80ca0e0 0x7f4da8158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4da659c640 1 -- 172.21.15.125:0/3231941535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da81538a0 msgr2=0x7f4da81ef9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:31.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4da659c640 1 --2- 172.21.15.125:0/3231941535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da81538a0 0x7f4da81ef9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4da659c640 1 -- 172.21.15.125:0/3231941535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d980094d0 con 0x7f4da80cf160 2023-10-30T13:11:31.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4da659c640 1 --2- 172.21.15.125:0/3231941535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4da80cf160 0x7f4da80c5230 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00d800 tx=0x7f4d9c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:31.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4d8f7fe640 1 -- 172.21.15.125:0/3231941535 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c00b6c0 con 0x7f4da80cf160 2023-10-30T13:11:31.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4d8f7fe640 1 -- 172.21.15.125:0/3231941535 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d9c00b820 con 0x7f4da80cf160 2023-10-30T13:11:31.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4d8f7fe640 1 -- 172.21.15.125:0/3231941535 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c00bb90 con 0x7f4da80cf160 2023-10-30T13:11:31.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4dad0a9640 1 -- 172.21.15.125:0/3231941535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da81f2c20 con 0x7f4da80cf160 2023-10-30T13:11:31.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4dad0a9640 1 -- 172.21.15.125:0/3231941535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da81f31e0 con 0x7f4da80cf160 2023-10-30T13:11:31.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4dad0a9640 1 -- 172.21.15.125:0/3231941535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da80b2a50 con 0x7f4da80cf160 2023-10-30T13:11:31.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4d8f7fe640 1 -- 172.21.15.125:0/3231941535 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4d9c00bcf0 con 0x7f4da80cf160 2023-10-30T13:11:31.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4d8f7fe640 1 --2- 172.21.15.125:0/3231941535 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4d940664d0 0x7f4d94068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:31.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.624+0000 7f4d8f7fe640 1 -- 172.21.15.125:0/3231941535 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7f4d9c084b20 con 0x7f4da80cf160 2023-10-30T13:11:31.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.628+0000 7f4da6d9d640 1 --2- 172.21.15.125:0/3231941535 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4d940664d0 0x7f4d94068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:31.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.628+0000 7f4da6d9d640 1 --2- 172.21.15.125:0/3231941535 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4d940664d0 0x7f4d94068990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f4d980b2000 tx=0x7f4d98009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:31.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.628+0000 7f4d8f7fe640 1 -- 172.21.15.125:0/3231941535 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4d9c051780 con 0x7f4da80cf160 2023-10-30T13:11:31.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.812+0000 7f4dad0a9640 1 -- 172.21.15.125:0/3231941535 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4da81ef660 con 0x7f4da80cf160 2023-10-30T13:11:31.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.812+0000 7f4d8f7fe640 1 -- 172.21.15.125:0/3231941535 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+5757 (secure 0 0 0) 0x7f4d9c054da0 con 0x7f4da80cf160 2023-10-30T13:11:31.815 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:31.815 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:30.689876+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:31.815 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 196 2023-10-30T13:11:31.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.812+0000 7f4dad0a9640 1 -- 172.21.15.125:0/3231941535 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4d940664d0 msgr2=0x7f4d94068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:31.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.812+0000 7f4dad0a9640 1 --2- 172.21.15.125:0/3231941535 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4d940664d0 0x7f4d94068990 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f4d980b2000 tx=0x7f4d98009040 comp rx=0 tx=0).stop 2023-10-30T13:11:31.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.812+0000 7f4dad0a9640 1 -- 172.21.15.125:0/3231941535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4da80cf160 msgr2=0x7f4da80c5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:31.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.812+0000 7f4dad0a9640 1 --2- 172.21.15.125:0/3231941535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4da80cf160 0x7f4da80c5230 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00d800 tx=0x7f4d9c00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.812+0000 7f4dad0a9640 1 -- 172.21.15.125:0/3231941535 shutdown_connections 2023-10-30T13:11:31.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.816+0000 7f4dad0a9640 1 --2- 172.21.15.125:0/3231941535 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4d940664d0 0x7f4d94068990 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.816+0000 7f4dad0a9640 1 --2- 172.21.15.125:0/3231941535 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4da81538a0 0x7f4da81ef9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.816+0000 7f4dad0a9640 1 --2- 172.21.15.125:0/3231941535 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4da80cf160 0x7f4da80c5230 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.816+0000 7f4dad0a9640 1 --2- 172.21.15.125:0/3231941535 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4da80ca0e0 0x7f4da8158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.816+0000 7f4dad0a9640 1 -- 172.21.15.125:0/3231941535 >> 172.21.15.125:0/3231941535 conn(0x7f4da806bfe0 msgr2=0x7f4da8151f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:31.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.816+0000 7f4dad0a9640 1 -- 172.21.15.125:0/3231941535 shutdown_connections 2023-10-30T13:11:31.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.816+0000 7f4dad0a9640 1 -- 172.21.15.125:0/3231941535 wait complete. 2023-10-30T13:11:31.835 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T13:11:31.835 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:11:31.836 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:31.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f06015f8640 1 Processor -- start 2023-10-30T13:11:31.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f06015f8640 1 -- start start 2023-10-30T13:11:31.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f06015f8640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f05fc1538a0 0x7f05fc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:31.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f06015f8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05fc0cf160 0x7f05fc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:31.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f06015f8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc0cb800 con 0x7f05fc0ca0e0 2023-10-30T13:11:31.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f06015f8640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05fc15dc30 con 0x7f05fc1538a0 2023-10-30T13:11:31.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f06015f8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc0cb330 con 0x7f05fc0cf160 2023-10-30T13:11:31.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f05faffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f05fc1538a0 0x7f05fc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:31.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f05faffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f05fc1538a0 0x7f05fc15d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34444/0 (socket says 172.21.15.125:34444) 2023-10-30T13:11:31.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f05faffd640 1 -- 172.21.15.125:0/993863698 learned_addr learned my addr 172.21.15.125:0/993863698 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:31.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f05fb7fe640 1 --2- 172.21.15.125:0/993863698 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05fc0cf160 0x7f05fc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:31.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f05faffd640 1 -- 172.21.15.125:0/993863698 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05fc0cf160 msgr2=0x7f05fc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:31.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f05faffd640 1 --2- 172.21.15.125:0/993863698 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05fc0cf160 0x7f05fc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f05faffd640 1 -- 172.21.15.125:0/993863698 >> v1:172.21.15.136:6789/0 conn(0x7f05fc0ca0e0 legacy=0x7f05fc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:31.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.976+0000 7f05faffd640 1 -- 172.21.15.125:0/993863698 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc0cad70 con 0x7f05fc1538a0 2023-10-30T13:11:31.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.980+0000 7f05faffd640 1 --2- 172.21.15.125:0/993863698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05fc1538a0 0x7f05fc15d6f0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009870 tx=0x7f05ec01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edab54c35fe9cdb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:31.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.980+0000 7f05f9ffb640 1 -- 172.21.15.125:0/993863698 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec0a8a30 con 0x7f05fc1538a0 2023-10-30T13:11:31.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.980+0000 7f05f9ffb640 1 -- 172.21.15.125:0/993863698 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec0a8b90 con 0x7f05fc1538a0 2023-10-30T13:11:31.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.980+0000 7f05f9ffb640 1 -- 172.21.15.125:0/993863698 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec0b0440 con 0x7f05fc1538a0 2023-10-30T13:11:31.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.980+0000 7f06015f8640 1 -- 172.21.15.125:0/993863698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05fc1538a0 msgr2=0x7f05fc15d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:31.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.980+0000 7f06015f8640 1 --2- 172.21.15.125:0/993863698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05fc1538a0 0x7f05fc15d6f0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009870 tx=0x7f05ec01ed30 comp rx=0 tx=0).stop 2023-10-30T13:11:31.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.980+0000 7f06015f8640 1 -- 172.21.15.125:0/993863698 shutdown_connections 2023-10-30T13:11:31.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.980+0000 7f06015f8640 1 --2- 172.21.15.125:0/993863698 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05fc0cf160 0x7f05fc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.980+0000 7f06015f8640 1 --2- 172.21.15.125:0/993863698 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05fc1538a0 0x7f05fc15d6f0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.980+0000 7f06015f8640 1 -- 172.21.15.125:0/993863698 >> 172.21.15.125:0/993863698 conn(0x7f05fc06bfe0 msgr2=0x7f05fc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:31.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.980+0000 7f06015f8640 1 -- 172.21.15.125:0/993863698 shutdown_connections 2023-10-30T13:11:31.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.980+0000 7f06015f8640 1 -- 172.21.15.125:0/993863698 wait complete. 2023-10-30T13:11:31.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f06015f8640 1 Processor -- start 2023-10-30T13:11:31.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f06015f8640 1 -- start start 2023-10-30T13:11:31.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f06015f8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05fc0ca0e0 0x7f05fc1e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:31.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f06015f8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05fc0cf160 0x7f05fc15b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:31.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f06015f8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05fc1538a0 0x7f05fc15bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:31.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f06015f8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc14d4f0 con 0x7f05fc0ca0e0 2023-10-30T13:11:31.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f06015f8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1e4d40 con 0x7f05fc0cf160 2023-10-30T13:11:31.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f06015f8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc15b2a0 con 0x7f05fc1538a0 2023-10-30T13:11:31.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f05faffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05fc0ca0e0 0x7f05fc1e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:31.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f05faffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05fc0ca0e0 0x7f05fc1e4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34448/0 (socket says 172.21.15.125:34448) 2023-10-30T13:11:31.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f05faffd640 1 -- 172.21.15.125:0/1980770228 learned_addr learned my addr 172.21.15.125:0/1980770228 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:31.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f05fb7fe640 1 --2- 172.21.15.125:0/1980770228 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05fc1538a0 0x7f05fc15bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:31.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f05fa7fc640 1 --2- 172.21.15.125:0/1980770228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05fc0cf160 0x7f05fc15b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:31.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f05faffd640 1 -- 172.21.15.125:0/1980770228 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05fc1538a0 msgr2=0x7f05fc15bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:31.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f05faffd640 1 --2- 172.21.15.125:0/1980770228 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05fc1538a0 0x7f05fc15bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f05faffd640 1 -- 172.21.15.125:0/1980770228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05fc0cf160 msgr2=0x7f05fc15b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:31.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f05faffd640 1 --2- 172.21.15.125:0/1980770228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05fc0cf160 0x7f05fc15b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:31.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f05faffd640 1 -- 172.21.15.125:0/1980770228 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ec0094d0 con 0x7f05fc0ca0e0 2023-10-30T13:11:31.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f05faffd640 1 --2- 172.21.15.125:0/1980770228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05fc0ca0e0 0x7f05fc1e4090 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0b3040 tx=0x7f05ec01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:31.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f05e37fe640 1 -- 172.21.15.125:0/1980770228 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec0ba600 con 0x7f05fc0ca0e0 2023-10-30T13:11:31.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f05e37fe640 1 -- 172.21.15.125:0/1980770228 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec0ba780 con 0x7f05fc0ca0e0 2023-10-30T13:11:31.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f06015f8640 1 -- 172.21.15.125:0/1980770228 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc0cef10 con 0x7f05fc0ca0e0 2023-10-30T13:11:31.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f05e37fe640 1 -- 172.21.15.125:0/1980770228 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec0bab30 con 0x7f05fc0ca0e0 2023-10-30T13:11:31.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f06015f8640 1 -- 172.21.15.125:0/1980770228 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc0c50b0 con 0x7f05fc0ca0e0 2023-10-30T13:11:31.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.984+0000 7f06015f8640 1 -- 172.21.15.125:0/1980770228 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05fc0b2a50 con 0x7f05fc0ca0e0 2023-10-30T13:11:31.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.988+0000 7f05e37fe640 1 -- 172.21.15.125:0/1980770228 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f05ec0bac90 con 0x7f05fc0ca0e0 2023-10-30T13:11:31.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.988+0000 7f05e37fe640 1 --2- 172.21.15.125:0/1980770228 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f05e80664a0 0x7f05e8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:31.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.988+0000 7f05e37fe640 1 -- 172.21.15.125:0/1980770228 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7f05ec122c30 con 0x7f05fc0ca0e0 2023-10-30T13:11:31.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.988+0000 7f05fa7fc640 1 --2- 172.21.15.125:0/1980770228 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f05e80664a0 0x7f05e8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:31.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.988+0000 7f05fa7fc640 1 --2- 172.21.15.125:0/1980770228 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f05e80664a0 0x7f05e8068960 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f05fc0c4660 tx=0x7f05f0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:31.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:31.992+0000 7f05e37fe640 1 -- 172.21.15.125:0/1980770228 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f05ec0ef7e0 con 0x7f05fc0ca0e0 2023-10-30T13:11:32.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:32.176+0000 7f06015f8640 1 -- 172.21.15.125:0/1980770228 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05fc151940 con 0x7f05fc0ca0e0 2023-10-30T13:11:32.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:32.176+0000 7f05e37fe640 1 -- 172.21.15.125:0/1980770228 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+5757 (secure 0 0 0) 0x7f05ec0b5020 con 0x7f05fc0ca0e0 2023-10-30T13:11:32.180 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:32.180 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:30.689876+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:32.181 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 196 2023-10-30T13:11:32.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:32.180+0000 7f06015f8640 1 -- 172.21.15.125:0/1980770228 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f05e80664a0 msgr2=0x7f05e8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:32.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:32.180+0000 7f06015f8640 1 --2- 172.21.15.125:0/1980770228 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f05e80664a0 0x7f05e8068960 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f05fc0c4660 tx=0x7f05f0009040 comp rx=0 tx=0).stop 2023-10-30T13:11:32.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:32.180+0000 7f06015f8640 1 -- 172.21.15.125:0/1980770228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05fc0ca0e0 msgr2=0x7f05fc1e4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:32.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:32.180+0000 7f06015f8640 1 --2- 172.21.15.125:0/1980770228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05fc0ca0e0 0x7f05fc1e4090 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0b3040 tx=0x7f05ec01bad0 comp rx=0 tx=0).stop 2023-10-30T13:11:32.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:32.180+0000 7f06015f8640 1 -- 172.21.15.125:0/1980770228 shutdown_connections 2023-10-30T13:11:32.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:32.180+0000 7f06015f8640 1 --2- 172.21.15.125:0/1980770228 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f05e80664a0 0x7f05e8068960 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:32.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:32.180+0000 7f06015f8640 1 --2- 172.21.15.125:0/1980770228 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f05fc1538a0 0x7f05fc15bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:32.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:32.180+0000 7f06015f8640 1 --2- 172.21.15.125:0/1980770228 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f05fc0cf160 0x7f05fc15b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:32.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:32.180+0000 7f06015f8640 1 --2- 172.21.15.125:0/1980770228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05fc0ca0e0 0x7f05fc1e4090 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:32.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:32.180+0000 7f06015f8640 1 -- 172.21.15.125:0/1980770228 >> 172.21.15.125:0/1980770228 conn(0x7f05fc06bfe0 msgr2=0x7f05fc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:32.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:32.180+0000 7f06015f8640 1 -- 172.21.15.125:0/1980770228 shutdown_connections 2023-10-30T13:11:32.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:32.180+0000 7f06015f8640 1 -- 172.21.15.125:0/1980770228 wait complete. 2023-10-30T13:11:32.203 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 196, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 196, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_8301": { "addr": "172.21.15.125:6837/2540063352", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2540063352, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2540063352, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 8301, "incarnation": 168, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_8443": { "addr": "172.21.15.136:6839/3776186190", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3776186190, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3776186190, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8443, "incarnation": 196, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_8529": { "addr": "172.21.15.125:6836/3950044434", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3950044434, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3950044434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 8529, "incarnation": 179, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 78, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:11:30.689876+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8529, "mds_1": 8301, "mds_2": 8443 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:11:34.204 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:34.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a14f71640 1 Processor -- start 2023-10-30T13:11:34.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a14f71640 1 -- start start 2023-10-30T13:11:34.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a14f71640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538f0 0x7f6a10159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:34.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a14f71640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6a100cefa0 0x7f6a100cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:34.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a14f71640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a100ca6a0 con 0x7f6a1015cd50 2023-10-30T13:11:34.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a14f71640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a1015d300 con 0x7f6a100cefa0 2023-10-30T13:11:34.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a14f71640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1015d5e0 con 0x7f6a101538f0 2023-10-30T13:11:34.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a0ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6a100cefa0 0x7f6a100cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:34.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a0ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6a100cefa0 0x7f6a100cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34458/0 (socket says 172.21.15.125:34458) 2023-10-30T13:11:34.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a0ffff640 1 -- 172.21.15.125:0/131373843 learned_addr learned my addr 172.21.15.125:0/131373843 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:34.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a0f7fe640 1 --2- 172.21.15.125:0/131373843 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538f0 0x7f6a10159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:34.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a0ffff640 1 -- 172.21.15.125:0/131373843 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538f0 msgr2=0x7f6a10159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:34.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a0ffff640 1 --2- 172.21.15.125:0/131373843 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538f0 0x7f6a10159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:34.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a0ffff640 1 -- 172.21.15.125:0/131373843 >> v1:172.21.15.136:6789/0 conn(0x7f6a1015cd50 legacy=0x7f6a100c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:34.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.344+0000 7f6a0ffff640 1 -- 172.21.15.125:0/131373843 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a1015d8c0 con 0x7f6a100cefa0 2023-10-30T13:11:34.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.348+0000 7f6a0ffff640 1 --2- 172.21.15.125:0/131373843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100cefa0 0x7f6a100cf3f0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f6a08009e50 tx=0x7f6a0801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3448e51b6a500561 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:34.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.348+0000 7f6a0e7fc640 1 -- 172.21.15.125:0/131373843 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a080af400 con 0x7f6a100cefa0 2023-10-30T13:11:34.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.348+0000 7f6a0e7fc640 1 -- 172.21.15.125:0/131373843 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a080ab050 con 0x7f6a100cefa0 2023-10-30T13:11:34.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.348+0000 7f6a0e7fc640 1 -- 172.21.15.125:0/131373843 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a080af780 con 0x7f6a100cefa0 2023-10-30T13:11:34.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.348+0000 7f6a14f71640 1 -- 172.21.15.125:0/131373843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100cefa0 msgr2=0x7f6a100cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:34.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.348+0000 7f6a14f71640 1 --2- 172.21.15.125:0/131373843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100cefa0 0x7f6a100cf3f0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f6a08009e50 tx=0x7f6a0801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:11:34.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.348+0000 7f6a14f71640 1 -- 172.21.15.125:0/131373843 shutdown_connections 2023-10-30T13:11:34.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.348+0000 7f6a14f71640 1 --2- 172.21.15.125:0/131373843 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100cefa0 0x7f6a100cf3f0 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:34.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.348+0000 7f6a14f71640 1 --2- 172.21.15.125:0/131373843 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538f0 0x7f6a10159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:34.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.348+0000 7f6a14f71640 1 -- 172.21.15.125:0/131373843 >> 172.21.15.125:0/131373843 conn(0x7f6a1006a640 msgr2=0x7f6a10065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:34.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.348+0000 7f6a14f71640 1 -- 172.21.15.125:0/131373843 shutdown_connections 2023-10-30T13:11:34.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.348+0000 7f6a14f71640 1 -- 172.21.15.125:0/131373843 wait complete. 2023-10-30T13:11:34.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a14f71640 1 Processor -- start 2023-10-30T13:11:34.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a14f71640 1 -- start start 2023-10-30T13:11:34.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a14f71640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100cefa0 0x7f6a10145e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:34.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a14f71640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538f0 0x7f6a101528e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:34.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a14f71640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a1015cd50 0x7f6a10151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:34.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a14f71640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1015aa80 con 0x7f6a100cefa0 2023-10-30T13:11:34.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a14f71640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a1015c0e0 con 0x7f6a1015cd50 2023-10-30T13:11:34.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a14f71640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a100b4a80 con 0x7f6a101538f0 2023-10-30T13:11:34.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a0f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100cefa0 0x7f6a10145e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:34.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a0f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100cefa0 0x7f6a10145e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34470/0 (socket says 172.21.15.125:34470) 2023-10-30T13:11:34.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a0f7fe640 1 -- 172.21.15.125:0/2459181289 learned_addr learned my addr 172.21.15.125:0/2459181289 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:34.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a0ffff640 1 --2- 172.21.15.125:0/2459181289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a1015cd50 0x7f6a10151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:34.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a0effd640 1 --2- 172.21.15.125:0/2459181289 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538f0 0x7f6a101528e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:34.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a0f7fe640 1 -- 172.21.15.125:0/2459181289 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538f0 msgr2=0x7f6a101528e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:34.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a0f7fe640 1 --2- 172.21.15.125:0/2459181289 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538f0 0x7f6a101528e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:34.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a0f7fe640 1 -- 172.21.15.125:0/2459181289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a1015cd50 msgr2=0x7f6a10151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:34.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a0f7fe640 1 --2- 172.21.15.125:0/2459181289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a1015cd50 0x7f6a10151080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:34.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a0f7fe640 1 -- 172.21.15.125:0/2459181289 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a08009b30 con 0x7f6a100cefa0 2023-10-30T13:11:34.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a0f7fe640 1 --2- 172.21.15.125:0/2459181289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100cefa0 0x7f6a10145e30 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f6a0000d800 tx=0x7f6a0000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:34.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f69f7fff640 1 -- 172.21.15.125:0/2459181289 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0000b6c0 con 0x7f6a100cefa0 2023-10-30T13:11:34.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f69f7fff640 1 -- 172.21.15.125:0/2459181289 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a0000b820 con 0x7f6a100cefa0 2023-10-30T13:11:34.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a14f71640 1 -- 172.21.15.125:0/2459181289 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a1015c4d0 con 0x7f6a100cefa0 2023-10-30T13:11:34.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f69f7fff640 1 -- 172.21.15.125:0/2459181289 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a0000bb90 con 0x7f6a100cefa0 2023-10-30T13:11:34.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a14f71640 1 -- 172.21.15.125:0/2459181289 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a1015a710 con 0x7f6a100cefa0 2023-10-30T13:11:34.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.352+0000 7f6a14f71640 1 -- 172.21.15.125:0/2459181289 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a1004e170 con 0x7f6a100cefa0 2023-10-30T13:11:34.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.356+0000 7f69f7fff640 1 -- 172.21.15.125:0/2459181289 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6a0000bcf0 con 0x7f6a100cefa0 2023-10-30T13:11:34.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.356+0000 7f69f7fff640 1 --2- 172.21.15.125:0/2459181289 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69fc0664a0 0x7f69fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:34.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.356+0000 7f69f7fff640 1 -- 172.21.15.125:0/2459181289 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7f6a00084c80 con 0x7f6a100cefa0 2023-10-30T13:11:34.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.356+0000 7f6a0effd640 1 --2- 172.21.15.125:0/2459181289 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69fc0664a0 0x7f69fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:34.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.356+0000 7f6a0effd640 1 --2- 172.21.15.125:0/2459181289 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69fc0664a0 0x7f69fc068960 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f6a10065b80 tx=0x7f6a04008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:34.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.360+0000 7f69f7fff640 1 -- 172.21.15.125:0/2459181289 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6a000518c0 con 0x7f6a100cefa0 2023-10-30T13:11:34.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.584+0000 7f6a14f71640 1 -- 172.21.15.125:0/2459181289 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a1014d6a0 con 0x7f6a100cefa0 2023-10-30T13:11:34.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.584+0000 7f69f7fff640 1 -- 172.21.15.125:0/2459181289 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+5757 (secure 0 0 0) 0x7f6a00054ee0 con 0x7f6a100cefa0 2023-10-30T13:11:34.586 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:34.586 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:30.689876+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:34.587 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 196 2023-10-30T13:11:34.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.584+0000 7f6a14f71640 1 -- 172.21.15.125:0/2459181289 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69fc0664a0 msgr2=0x7f69fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:34.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.584+0000 7f6a14f71640 1 --2- 172.21.15.125:0/2459181289 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69fc0664a0 0x7f69fc068960 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f6a10065b80 tx=0x7f6a04008040 comp rx=0 tx=0).stop 2023-10-30T13:11:34.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.584+0000 7f6a14f71640 1 -- 172.21.15.125:0/2459181289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100cefa0 msgr2=0x7f6a10145e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:34.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.584+0000 7f6a14f71640 1 --2- 172.21.15.125:0/2459181289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100cefa0 0x7f6a10145e30 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f6a0000d800 tx=0x7f6a0000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:11:34.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.584+0000 7f6a14f71640 1 -- 172.21.15.125:0/2459181289 shutdown_connections 2023-10-30T13:11:34.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.584+0000 7f6a14f71640 1 --2- 172.21.15.125:0/2459181289 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f69fc0664a0 0x7f69fc068960 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:34.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.584+0000 7f6a14f71640 1 --2- 172.21.15.125:0/2459181289 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6a1015cd50 0x7f6a10151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:34.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.584+0000 7f6a14f71640 1 --2- 172.21.15.125:0/2459181289 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6a101538f0 0x7f6a101528e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:34.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.584+0000 7f6a14f71640 1 --2- 172.21.15.125:0/2459181289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6a100cefa0 0x7f6a10145e30 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:34.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.584+0000 7f6a14f71640 1 -- 172.21.15.125:0/2459181289 >> 172.21.15.125:0/2459181289 conn(0x7f6a1006a640 msgr2=0x7f6a1006c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:34.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.584+0000 7f6a14f71640 1 -- 172.21.15.125:0/2459181289 shutdown_connections 2023-10-30T13:11:34.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:34.584+0000 7f6a14f71640 1 -- 172.21.15.125:0/2459181289 wait complete. 2023-10-30T13:11:34.607 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:11:36.608 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:36.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.748+0000 7fc2a58c9640 1 Processor -- start 2023-10-30T13:11:36.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.748+0000 7fc2a58c9640 1 -- start start 2023-10-30T13:11:36.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc2a58c9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc2a006f7a0 0x7fc2a006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:36.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc2a58c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a00c6bd0 0x7fc2a0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:36.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc2a58c9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a006a3d0 con 0x7fc2a0071110 2023-10-30T13:11:36.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc2a58c9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a00700c0 con 0x7fc2a006f7a0 2023-10-30T13:11:36.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc2a58c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a0066cf0 con 0x7fc2a00c6bd0 2023-10-30T13:11:36.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc29effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc2a006f7a0 0x7fc2a006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:36.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc29effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc2a006f7a0 0x7fc2a006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34476/0 (socket says 172.21.15.125:34476) 2023-10-30T13:11:36.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc29f7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc2a0071110 0x7fc2a0158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46532/0 (socket says 172.21.15.125:46532) 2023-10-30T13:11:36.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc29effd640 1 -- 172.21.15.125:0/1976179673 learned_addr learned my addr 172.21.15.125:0/1976179673 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:36.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc297fff640 1 --2- 172.21.15.125:0/1976179673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a00c6bd0 0x7fc2a0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:36.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc29effd640 1 -- 172.21.15.125:0/1976179673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a00c6bd0 msgr2=0x7fc2a0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:36.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc29effd640 1 --2- 172.21.15.125:0/1976179673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a00c6bd0 0x7fc2a0069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:36.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc29effd640 1 -- 172.21.15.125:0/1976179673 >> v1:172.21.15.125:6789/0 conn(0x7fc2a0071110 legacy=0x7fc2a0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:36.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc29effd640 1 -- 172.21.15.125:0/1976179673 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a00c70d0 con 0x7fc2a006f7a0 2023-10-30T13:11:36.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.752+0000 7fc29e7fc640 1 -- 172.21.15.125:0/1976179673 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070811935 0 0) 0x7fc2a006a3d0 con 0x7fc2a0071110 2023-10-30T13:11:36.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc29effd640 1 --2- 172.21.15.125:0/1976179673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a006f7a0 0x7fc2a006fb80 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fc290009870 tx=0x7fc29001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cea6b072087cbb6a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:36.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc29e7fc640 1 -- 172.21.15.125:0/1976179673 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2900a8a30 con 0x7fc2a006f7a0 2023-10-30T13:11:36.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc29e7fc640 1 -- 172.21.15.125:0/1976179673 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2900a8b90 con 0x7fc2a006f7a0 2023-10-30T13:11:36.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc29e7fc640 1 -- 172.21.15.125:0/1976179673 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2900b0440 con 0x7fc2a006f7a0 2023-10-30T13:11:36.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 -- 172.21.15.125:0/1976179673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a006f7a0 msgr2=0x7fc2a006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:36.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 --2- 172.21.15.125:0/1976179673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a006f7a0 0x7fc2a006fb80 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fc290009870 tx=0x7fc29001ed30 comp rx=0 tx=0).stop 2023-10-30T13:11:36.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 -- 172.21.15.125:0/1976179673 shutdown_connections 2023-10-30T13:11:36.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 --2- 172.21.15.125:0/1976179673 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a00c6bd0 0x7fc2a0069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:36.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 --2- 172.21.15.125:0/1976179673 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a006f7a0 0x7fc2a006fb80 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:36.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 -- 172.21.15.125:0/1976179673 >> 172.21.15.125:0/1976179673 conn(0x7fc2a00c7910 msgr2=0x7fc2a00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:36.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 -- 172.21.15.125:0/1976179673 shutdown_connections 2023-10-30T13:11:36.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 -- 172.21.15.125:0/1976179673 wait complete. 2023-10-30T13:11:36.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 Processor -- start 2023-10-30T13:11:36.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 -- start start 2023-10-30T13:11:36.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a006f7a0 0x7fc2a01ecaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:36.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2a0071110 0x7fc2a015b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:36.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a00c6bd0 0x7fc2a00c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:36.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a01ed180 con 0x7fc2a006f7a0 2023-10-30T13:11:36.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a01ed460 con 0x7fc2a0071110 2023-10-30T13:11:36.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc2a58c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a01ed740 con 0x7fc2a00c6bd0 2023-10-30T13:11:36.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc29effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a006f7a0 0x7fc2a01ecaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:36.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc29effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a006f7a0 0x7fc2a01ecaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34480/0 (socket says 172.21.15.125:34480) 2023-10-30T13:11:36.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.756+0000 7fc29effd640 1 -- 172.21.15.125:0/4292408523 learned_addr learned my addr 172.21.15.125:0/4292408523 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:36.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc297fff640 1 --2- 172.21.15.125:0/4292408523 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2a0071110 0x7fc2a015b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:36.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc29f7fe640 1 --2- 172.21.15.125:0/4292408523 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a00c6bd0 0x7fc2a00c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:36.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc29effd640 1 -- 172.21.15.125:0/4292408523 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a00c6bd0 msgr2=0x7fc2a00c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:36.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc29effd640 1 --2- 172.21.15.125:0/4292408523 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a00c6bd0 0x7fc2a00c9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:36.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc29effd640 1 -- 172.21.15.125:0/4292408523 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2a0071110 msgr2=0x7fc2a015b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:36.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc29effd640 1 --2- 172.21.15.125:0/4292408523 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2a0071110 0x7fc2a015b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:36.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc29effd640 1 -- 172.21.15.125:0/4292408523 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2900094d0 con 0x7fc2a006f7a0 2023-10-30T13:11:36.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc29effd640 1 --2- 172.21.15.125:0/4292408523 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a006f7a0 0x7fc2a01ecaf0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fc2900b3040 tx=0x7fc29001bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:36.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc29cff9640 1 -- 172.21.15.125:0/4292408523 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2900b0dd0 con 0x7fc2a006f7a0 2023-10-30T13:11:36.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc29cff9640 1 -- 172.21.15.125:0/4292408523 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2900bb460 con 0x7fc2a006f7a0 2023-10-30T13:11:36.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc2a58c9640 1 -- 172.21.15.125:0/4292408523 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a01eda20 con 0x7fc2a006f7a0 2023-10-30T13:11:36.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc29cff9640 1 -- 172.21.15.125:0/4292408523 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2900bb810 con 0x7fc2a006f7a0 2023-10-30T13:11:36.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc2a58c9640 1 -- 172.21.15.125:0/4292408523 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a0158c20 con 0x7fc2a006f7a0 2023-10-30T13:11:36.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc2a58c9640 1 -- 172.21.15.125:0/4292408523 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a00c8870 con 0x7fc2a006f7a0 2023-10-30T13:11:36.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc29cff9640 1 -- 172.21.15.125:0/4292408523 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc2900b83f0 con 0x7fc2a006f7a0 2023-10-30T13:11:36.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.760+0000 7fc29cff9640 1 --2- 172.21.15.125:0/4292408523 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2800664a0 0x7fc280068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:36.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.764+0000 7fc29cff9640 1 -- 172.21.15.125:0/4292408523 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7fc290122190 con 0x7fc2a006f7a0 2023-10-30T13:11:36.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.764+0000 7fc297fff640 1 --2- 172.21.15.125:0/4292408523 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2800664a0 0x7fc280068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:36.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.764+0000 7fc297fff640 1 --2- 172.21.15.125:0/4292408523 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2800664a0 0x7fc280068960 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00cada0 tx=0x7fc288008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:36.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.768+0000 7fc29cff9640 1 -- 172.21.15.125:0/4292408523 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc2900eecb0 con 0x7fc2a006f7a0 2023-10-30T13:11:36.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.948+0000 7fc2a58c9640 1 -- 172.21.15.125:0/4292408523 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc2a004e170 con 0x7fc2a006f7a0 2023-10-30T13:11:36.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.948+0000 7fc29cff9640 1 -- 172.21.15.125:0/4292408523 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+5759 (secure 0 0 0) 0x7fc2900b5090 con 0x7fc2a006f7a0 2023-10-30T13:11:36.951 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:36.951 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:36.234036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:resolve","state_seq":35,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:36.952 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 197 2023-10-30T13:11:36.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.952+0000 7fc2a58c9640 1 -- 172.21.15.125:0/4292408523 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2800664a0 msgr2=0x7fc280068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:36.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.952+0000 7fc2a58c9640 1 --2- 172.21.15.125:0/4292408523 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2800664a0 0x7fc280068960 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fc2a00cada0 tx=0x7fc288008040 comp rx=0 tx=0).stop 2023-10-30T13:11:36.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.952+0000 7fc2a58c9640 1 -- 172.21.15.125:0/4292408523 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a006f7a0 msgr2=0x7fc2a01ecaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:36.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.952+0000 7fc2a58c9640 1 --2- 172.21.15.125:0/4292408523 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a006f7a0 0x7fc2a01ecaf0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7fc2900b3040 tx=0x7fc29001bad0 comp rx=0 tx=0).stop 2023-10-30T13:11:36.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.952+0000 7fc2a58c9640 1 -- 172.21.15.125:0/4292408523 shutdown_connections 2023-10-30T13:11:36.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.952+0000 7fc2a58c9640 1 --2- 172.21.15.125:0/4292408523 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc2800664a0 0x7fc280068960 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:36.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.952+0000 7fc2a58c9640 1 --2- 172.21.15.125:0/4292408523 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc2a00c6bd0 0x7fc2a00c9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:36.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.952+0000 7fc2a58c9640 1 --2- 172.21.15.125:0/4292408523 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc2a0071110 0x7fc2a015b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:36.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.952+0000 7fc2a58c9640 1 --2- 172.21.15.125:0/4292408523 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc2a006f7a0 0x7fc2a01ecaf0 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.952+0000 7fc2a58c9640 1 -- 172.21.15.125:0/4292408523 >> 172.21.15.125:0/4292408523 conn(0x7fc2a00c7910 msgr2=0x7fc2a00c3900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.952+0000 7fc2a58c9640 1 -- 172.21.15.125:0/4292408523 shutdown_connections 2023-10-30T13:11:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:36.952+0000 7fc2a58c9640 1 -- 172.21.15.125:0/4292408523 wait complete. 2023-10-30T13:11:36.973 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:11:38.974 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:39.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f337a26b640 1 Processor -- start 2023-10-30T13:11:39.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f337a26b640 1 -- start start 2023-10-30T13:11:39.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f337a26b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33741538a0 0x7f337415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:39.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f337a26b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f33740ca0e0 0x7f337415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:39.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f337a26b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f33740cf160 0x7f33740cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:39.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f337a26b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33740cb800 con 0x7f33740ca0e0 2023-10-30T13:11:39.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f337a26b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f337415dc30 con 0x7f33740cf160 2023-10-30T13:11:39.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f337a26b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33740cb330 con 0x7f33741538a0 2023-10-30T13:11:39.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f33737fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f33740ca0e0 0x7f337415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:39.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f33737fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f33740ca0e0 0x7f337415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34484/0 (socket says 172.21.15.125:34484) 2023-10-30T13:11:39.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f33737fe640 1 -- 172.21.15.125:0/3765597774 learned_addr learned my addr 172.21.15.125:0/3765597774 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:39.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f3378808640 1 --2- 172.21.15.125:0/3765597774 >> v2:172.21.15.136:3300/0 conn(0x7f33740cf160 0x7f33740cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:39.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f3373fff640 1 --2- 172.21.15.125:0/3765597774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33741538a0 0x7f337415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:39.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f33737fe640 1 -- 172.21.15.125:0/3765597774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33741538a0 msgr2=0x7f337415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:39.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f33737fe640 1 --2- 172.21.15.125:0/3765597774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33741538a0 0x7f337415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:39.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f33737fe640 1 -- 172.21.15.125:0/3765597774 >> v2:172.21.15.136:3300/0 conn(0x7f33740cf160 msgr2=0x7f33740cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:39.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f33737fe640 1 --2- 172.21.15.125:0/3765597774 >> v2:172.21.15.136:3300/0 conn(0x7f33740cf160 0x7f33740cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:39.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.112+0000 7f33737fe640 1 -- 172.21.15.125:0/3765597774 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740cad70 con 0x7f33740ca0e0 2023-10-30T13:11:39.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.116+0000 7f33737fe640 1 --2- 172.21.15.125:0/3765597774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33740ca0e0 0x7f337415a090 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f337406cb00 tx=0x7f336801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ecce40b7fed9822 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:39.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.116+0000 7f3372ffd640 1 -- 172.21.15.125:0/3765597774 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33680a89b0 con 0x7f33740ca0e0 2023-10-30T13:11:39.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.116+0000 7f3372ffd640 1 -- 172.21.15.125:0/3765597774 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33680a8b10 con 0x7f33740ca0e0 2023-10-30T13:11:39.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.116+0000 7f3372ffd640 1 -- 172.21.15.125:0/3765597774 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33680a8ea0 con 0x7f33740ca0e0 2023-10-30T13:11:39.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.116+0000 7f337a26b640 1 -- 172.21.15.125:0/3765597774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33740ca0e0 msgr2=0x7f337415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:39.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.116+0000 7f337a26b640 1 --2- 172.21.15.125:0/3765597774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33740ca0e0 0x7f337415a090 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f337406cb00 tx=0x7f336801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:11:39.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.116+0000 7f337a26b640 1 -- 172.21.15.125:0/3765597774 shutdown_connections 2023-10-30T13:11:39.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.116+0000 7f337a26b640 1 --2- 172.21.15.125:0/3765597774 >> v2:172.21.15.136:3300/0 conn(0x7f33740cf160 0x7f33740cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:39.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.116+0000 7f337a26b640 1 --2- 172.21.15.125:0/3765597774 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33740ca0e0 0x7f337415a090 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:39.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.116+0000 7f337a26b640 1 --2- 172.21.15.125:0/3765597774 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33741538a0 0x7f337415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:39.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.116+0000 7f337a26b640 1 -- 172.21.15.125:0/3765597774 >> 172.21.15.125:0/3765597774 conn(0x7f337406bfe0 msgr2=0x7f337406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:39.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.116+0000 7f337a26b640 1 -- 172.21.15.125:0/3765597774 shutdown_connections 2023-10-30T13:11:39.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.116+0000 7f337a26b640 1 -- 172.21.15.125:0/3765597774 wait complete. 2023-10-30T13:11:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f337a26b640 1 Processor -- start 2023-10-30T13:11:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f337a26b640 1 -- start start 2023-10-30T13:11:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f337a26b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33740ca0e0 0x7f33741efac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f337a26b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33740cf160 0x7f33740ce260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f337a26b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33741538a0 0x7f33741e5090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f337a26b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3374142670 con 0x7f33740cf160 2023-10-30T13:11:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f337a26b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337415bf20 con 0x7f33740ca0e0 2023-10-30T13:11:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f337a26b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33741f0000 con 0x7f33741538a0 2023-10-30T13:11:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f33737fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33740cf160 0x7f33740ce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:39.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f33737fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33740cf160 0x7f33740ce260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34488/0 (socket says 172.21.15.125:34488) 2023-10-30T13:11:39.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f33737fe640 1 -- 172.21.15.125:0/505361614 learned_addr learned my addr 172.21.15.125:0/505361614 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:39.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f3378808640 1 --2- 172.21.15.125:0/505361614 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33741538a0 0x7f33741e5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:39.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f3373fff640 1 --2- 172.21.15.125:0/505361614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33740ca0e0 0x7f33741efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:39.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f33737fe640 1 -- 172.21.15.125:0/505361614 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33741538a0 msgr2=0x7f33741e5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f33737fe640 1 --2- 172.21.15.125:0/505361614 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33741538a0 0x7f33741e5090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f33737fe640 1 -- 172.21.15.125:0/505361614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33740ca0e0 msgr2=0x7f33741efac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f33737fe640 1 --2- 172.21.15.125:0/505361614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33740ca0e0 0x7f33741efac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f33737fe640 1 -- 172.21.15.125:0/505361614 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33680094d0 con 0x7f33740cf160 2023-10-30T13:11:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f33737fe640 1 --2- 172.21.15.125:0/505361614 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33740cf160 0x7f33740ce260 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f336801d3a0 tx=0x7f336801ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:39.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f3370ff9640 1 -- 172.21.15.125:0/505361614 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33680ba600 con 0x7f33740cf160 2023-10-30T13:11:39.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f3370ff9640 1 -- 172.21.15.125:0/505361614 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33680ba760 con 0x7f33740cf160 2023-10-30T13:11:39.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f337a26b640 1 -- 172.21.15.125:0/505361614 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33741e6450 con 0x7f33740cf160 2023-10-30T13:11:39.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f3370ff9640 1 -- 172.21.15.125:0/505361614 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33680bab10 con 0x7f33740cf160 2023-10-30T13:11:39.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f337a26b640 1 -- 172.21.15.125:0/505361614 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33741ef070 con 0x7f33740cf160 2023-10-30T13:11:39.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.120+0000 7f337a26b640 1 -- 172.21.15.125:0/505361614 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33740b2a50 con 0x7f33740cf160 2023-10-30T13:11:39.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.124+0000 7f3370ff9640 1 -- 172.21.15.125:0/505361614 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f33680bac70 con 0x7f33740cf160 2023-10-30T13:11:39.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.124+0000 7f3370ff9640 1 --2- 172.21.15.125:0/505361614 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33600664a0 0x7f3360068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:39.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.124+0000 7f3370ff9640 1 -- 172.21.15.125:0/505361614 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7f3368122da0 con 0x7f33740cf160 2023-10-30T13:11:39.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.124+0000 7f3373fff640 1 --2- 172.21.15.125:0/505361614 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33600664a0 0x7f3360068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:39.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.124+0000 7f3373fff640 1 --2- 172.21.15.125:0/505361614 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33600664a0 0x7f3360068960 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f336400b930 tx=0x7f3364008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:39.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.128+0000 7f3370ff9640 1 -- 172.21.15.125:0/505361614 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f33680ef960 con 0x7f33740cf160 2023-10-30T13:11:39.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.312+0000 7f337a26b640 1 -- 172.21.15.125:0/505361614 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33741e3c30 con 0x7f33740cf160 2023-10-30T13:11:39.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.312+0000 7f3370ff9640 1 -- 172.21.15.125:0/505361614 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+5759 (secure 0 0 0) 0x7f33680b5020 con 0x7f33740cf160 2023-10-30T13:11:39.314 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:39.314 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:36.234036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:resolve","state_seq":35,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:39.314 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 197 2023-10-30T13:11:39.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.312+0000 7f337a26b640 1 -- 172.21.15.125:0/505361614 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33600664a0 msgr2=0x7f3360068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:39.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.312+0000 7f337a26b640 1 --2- 172.21.15.125:0/505361614 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33600664a0 0x7f3360068960 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f336400b930 tx=0x7f3364008040 comp rx=0 tx=0).stop 2023-10-30T13:11:39.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.312+0000 7f337a26b640 1 -- 172.21.15.125:0/505361614 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33740cf160 msgr2=0x7f33740ce260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:39.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.312+0000 7f337a26b640 1 --2- 172.21.15.125:0/505361614 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33740cf160 0x7f33740ce260 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f336801d3a0 tx=0x7f336801ba90 comp rx=0 tx=0).stop 2023-10-30T13:11:39.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.312+0000 7f337a26b640 1 -- 172.21.15.125:0/505361614 shutdown_connections 2023-10-30T13:11:39.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.312+0000 7f337a26b640 1 --2- 172.21.15.125:0/505361614 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f33600664a0 0x7f3360068960 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:39.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.312+0000 7f337a26b640 1 --2- 172.21.15.125:0/505361614 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f33741538a0 0x7f33741e5090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:39.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.312+0000 7f337a26b640 1 --2- 172.21.15.125:0/505361614 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f33740cf160 0x7f33740ce260 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:39.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.312+0000 7f337a26b640 1 --2- 172.21.15.125:0/505361614 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f33740ca0e0 0x7f33741efac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:39.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.312+0000 7f337a26b640 1 -- 172.21.15.125:0/505361614 >> 172.21.15.125:0/505361614 conn(0x7f337406bfe0 msgr2=0x7f33741525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:39.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.312+0000 7f337a26b640 1 -- 172.21.15.125:0/505361614 shutdown_connections 2023-10-30T13:11:39.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:39.312+0000 7f337a26b640 1 -- 172.21.15.125:0/505361614 wait complete. 2023-10-30T13:11:39.335 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:11:41.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:41.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe586a9c640 1 Processor -- start 2023-10-30T13:11:41.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe586a9c640 1 -- start start 2023-10-30T13:11:41.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe586a9c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe5801538a0 0x7fe58015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:41.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe586a9c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe5800cf160 0x7fe5800cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:41.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe586a9c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5800cb800 con 0x7fe5800ca0e0 2023-10-30T13:11:41.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe586a9c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58015dc30 con 0x7fe5800cf160 2023-10-30T13:11:41.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe586a9c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5800cb330 con 0x7fe5801538a0 2023-10-30T13:11:41.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe585039640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe5800cf160 0x7fe5800cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:41.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe585039640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe5800cf160 0x7fe5800cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60004/0 (socket says 172.21.15.125:60004) 2023-10-30T13:11:41.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe585039640 1 -- 172.21.15.125:0/637994233 learned_addr learned my addr 172.21.15.125:0/637994233 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:41.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe584838640 1 --2- 172.21.15.125:0/637994233 >> v2:172.21.15.136:3300/0 conn(0x7fe5801538a0 0x7fe58015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:41.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe585039640 1 -- 172.21.15.125:0/637994233 >> v2:172.21.15.136:3300/0 conn(0x7fe5801538a0 msgr2=0x7fe58015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:41.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe585039640 1 --2- 172.21.15.125:0/637994233 >> v2:172.21.15.136:3300/0 conn(0x7fe5801538a0 0x7fe58015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:41.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe585039640 1 -- 172.21.15.125:0/637994233 >> v1:172.21.15.136:6789/0 conn(0x7fe5800ca0e0 legacy=0x7fe58015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:41.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.471+0000 7fe585039640 1 -- 172.21.15.125:0/637994233 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5800cad70 con 0x7fe5800cf160 2023-10-30T13:11:41.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.475+0000 7fe585039640 1 --2- 172.21.15.125:0/637994233 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5800cf160 0x7fe5800cf590 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fe5800cfd10 tx=0x7fe57801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4495db0c826a0410 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:41.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.475+0000 7fe57f7fe640 1 -- 172.21.15.125:0/637994233 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5780af400 con 0x7fe5800cf160 2023-10-30T13:11:41.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.475+0000 7fe57f7fe640 1 -- 172.21.15.125:0/637994233 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5780ab050 con 0x7fe5800cf160 2023-10-30T13:11:41.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.475+0000 7fe57f7fe640 1 -- 172.21.15.125:0/637994233 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5780af780 con 0x7fe5800cf160 2023-10-30T13:11:41.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.475+0000 7fe586a9c640 1 -- 172.21.15.125:0/637994233 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5800cf160 msgr2=0x7fe5800cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:41.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.475+0000 7fe586a9c640 1 --2- 172.21.15.125:0/637994233 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5800cf160 0x7fe5800cf590 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fe5800cfd10 tx=0x7fe57801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:11:41.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.475+0000 7fe586a9c640 1 -- 172.21.15.125:0/637994233 shutdown_connections 2023-10-30T13:11:41.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.475+0000 7fe586a9c640 1 --2- 172.21.15.125:0/637994233 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5800cf160 0x7fe5800cf590 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:41.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.475+0000 7fe586a9c640 1 --2- 172.21.15.125:0/637994233 >> v2:172.21.15.136:3300/0 conn(0x7fe5801538a0 0x7fe58015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:41.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.475+0000 7fe586a9c640 1 -- 172.21.15.125:0/637994233 >> 172.21.15.125:0/637994233 conn(0x7fe58006bfe0 msgr2=0x7fe58006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:41.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.475+0000 7fe586a9c640 1 -- 172.21.15.125:0/637994233 shutdown_connections 2023-10-30T13:11:41.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.475+0000 7fe586a9c640 1 -- 172.21.15.125:0/637994233 wait complete. 2023-10-30T13:11:41.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.479+0000 7fe586a9c640 1 Processor -- start 2023-10-30T13:11:41.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.479+0000 7fe586a9c640 1 -- start start 2023-10-30T13:11:41.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.479+0000 7fe586a9c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5800ca0e0 0x7fe5801480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:41.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.479+0000 7fe586a9c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5800cf160 0x7fe580158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:41.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.479+0000 7fe586a9c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5801538a0 0x7fe5801f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.479+0000 7fe586a9c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5801e5330 con 0x7fe5800cf160 2023-10-30T13:11:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.479+0000 7fe586a9c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5800b4f30 con 0x7fe5801538a0 2023-10-30T13:11:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.479+0000 7fe586a9c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5801e4d70 con 0x7fe5800ca0e0 2023-10-30T13:11:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.479+0000 7fe57ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5800cf160 0x7fe580158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.479+0000 7fe57ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5800cf160 0x7fe580158f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60018/0 (socket says 172.21.15.125:60018) 2023-10-30T13:11:41.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.479+0000 7fe57ffff640 1 -- 172.21.15.125:0/1444997857 learned_addr learned my addr 172.21.15.125:0/1444997857 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:41.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.479+0000 7fe584838640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5800ca0e0 0x7fe5801480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:41.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.479+0000 7fe585039640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5801538a0 0x7fe5801f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:41.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.483+0000 7fe57ffff640 1 -- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5800ca0e0 msgr2=0x7fe5801480f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:41.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.483+0000 7fe57ffff640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5800ca0e0 0x7fe5801480f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:41.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.483+0000 7fe57ffff640 1 -- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5801538a0 msgr2=0x7fe5801f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:41.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.483+0000 7fe57ffff640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5801538a0 0x7fe5801f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:41.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.483+0000 7fe57ffff640 1 -- 172.21.15.125:0/1444997857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe578009b30 con 0x7fe5800cf160 2023-10-30T13:11:41.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.483+0000 7fe585039640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5801538a0 0x7fe5801f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:11:41.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.483+0000 7fe57ffff640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5800cf160 0x7fe580158f80 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00c3c0 tx=0x7fe56c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:41.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.483+0000 7fe57d7fa640 1 -- 172.21.15.125:0/1444997857 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c0052d0 con 0x7fe5800cf160 2023-10-30T13:11:41.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.483+0000 7fe57d7fa640 1 -- 172.21.15.125:0/1444997857 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe56c005430 con 0x7fe5800cf160 2023-10-30T13:11:41.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.483+0000 7fe586a9c640 1 -- 172.21.15.125:0/1444997857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5801e5050 con 0x7fe5800cf160 2023-10-30T13:11:41.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.483+0000 7fe57d7fa640 1 -- 172.21.15.125:0/1444997857 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe56c003840 con 0x7fe5800cf160 2023-10-30T13:11:41.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.483+0000 7fe586a9c640 1 -- 172.21.15.125:0/1444997857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5800b4bc0 con 0x7fe5800cf160 2023-10-30T13:11:41.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.483+0000 7fe586a9c640 1 -- 172.21.15.125:0/1444997857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5800b4d80 con 0x7fe5800cf160 2023-10-30T13:11:41.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.483+0000 7fe57d7fa640 1 -- 172.21.15.125:0/1444997857 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe56c0039a0 con 0x7fe5800cf160 2023-10-30T13:11:41.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.487+0000 7fe57d7fa640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5640664a0 0x7fe564068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:41.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.487+0000 7fe57d7fa640 1 -- 172.21.15.125:0/1444997857 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7fe56c083cd0 con 0x7fe5800cf160 2023-10-30T13:11:41.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.487+0000 7fe584838640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5640664a0 0x7fe564068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:41.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.487+0000 7fe584838640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5640664a0 0x7fe564068960 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fe57400b9d0 tx=0x7fe574008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:41.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.491+0000 7fe57d7fa640 1 -- 172.21.15.125:0/1444997857 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe56c0507f0 con 0x7fe5800cf160 2023-10-30T13:11:41.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.671+0000 7fe586a9c640 1 -- 172.21.15.125:0/1444997857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5800b2a50 con 0x7fe5800cf160 2023-10-30T13:11:41.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.671+0000 7fe57d7fa640 1 -- 172.21.15.125:0/1444997857 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+5759 (secure 0 0 0) 0x7fe56c017070 con 0x7fe5800cf160 2023-10-30T13:11:41.676 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:41.676 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":197,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:36.234036+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:resolve","state_seq":35,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:41.676 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 197 2023-10-30T13:11:41.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.675+0000 7fe586a9c640 1 -- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5640664a0 msgr2=0x7fe564068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:41.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.675+0000 7fe586a9c640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5640664a0 0x7fe564068960 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fe57400b9d0 tx=0x7fe574008040 comp rx=0 tx=0).stop 2023-10-30T13:11:41.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.675+0000 7fe586a9c640 1 -- 172.21.15.125:0/1444997857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5800cf160 msgr2=0x7fe580158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:41.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.675+0000 7fe586a9c640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5800cf160 0x7fe580158f80 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fe56c00c3c0 tx=0x7fe56c00c880 comp rx=0 tx=0).stop 2023-10-30T13:11:41.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.675+0000 7fe586a9c640 1 -- 172.21.15.125:0/1444997857 shutdown_connections 2023-10-30T13:11:41.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.675+0000 7fe586a9c640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5640664a0 0x7fe564068960 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:41.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.675+0000 7fe586a9c640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5801538a0 0x7fe5801f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:41.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.675+0000 7fe586a9c640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5800cf160 0x7fe580158f80 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:41.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.675+0000 7fe586a9c640 1 --2- 172.21.15.125:0/1444997857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5800ca0e0 0x7fe5801480f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:41.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.675+0000 7fe586a9c640 1 -- 172.21.15.125:0/1444997857 >> 172.21.15.125:0/1444997857 conn(0x7fe58006bfe0 msgr2=0x7fe58006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:41.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.675+0000 7fe586a9c640 1 -- 172.21.15.125:0/1444997857 shutdown_connections 2023-10-30T13:11:41.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:41.675+0000 7fe586a9c640 1 -- 172.21.15.125:0/1444997857 wait complete. 2023-10-30T13:11:41.697 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:11:43.698 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:43.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.863+0000 7f9bda765640 1 Processor -- start 2023-10-30T13:11:43.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.863+0000 7f9bda765640 1 -- start start 2023-10-30T13:11:43.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.867+0000 7f9bda765640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9bd41538a0 0x7f9bd415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:43.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.867+0000 7f9bda765640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd40cb800 con 0x7f9bd40cf160 2023-10-30T13:11:43.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.867+0000 7f9bda765640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd415dc30 con 0x7f9bd40ca0e0 2023-10-30T13:11:43.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.867+0000 7f9bda765640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd40cb330 con 0x7f9bd41538a0 2023-10-30T13:11:43.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.867+0000 7f9bd3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9bd41538a0 0x7f9bd415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:43.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.867+0000 7f9bd3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9bd41538a0 0x7f9bd415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60030/0 (socket says 172.21.15.125:60030) 2023-10-30T13:11:43.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.867+0000 7f9bd3fff640 1 -- 172.21.15.125:0/3120600696 learned_addr learned my addr 172.21.15.125:0/3120600696 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:43.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.867+0000 7f9bd3fff640 1 -- 172.21.15.125:0/3120600696 >> v1:172.21.15.136:6789/0 conn(0x7f9bd40ca0e0 legacy=0x7f9bd415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:43.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.867+0000 7f9bd3fff640 1 -- 172.21.15.125:0/3120600696 >> v1:172.21.15.125:6789/0 conn(0x7f9bd40cf160 legacy=0x7f9bd4164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:43.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.867+0000 7f9bd3fff640 1 -- 172.21.15.125:0/3120600696 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd40cad70 con 0x7f9bd41538a0 2023-10-30T13:11:43.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.867+0000 7f9bd2ffd640 1 -- 172.21.15.125:0/3120600696 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095572752 0 0) 0x7f9bd40cb800 con 0x7f9bd40cf160 2023-10-30T13:11:43.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.871+0000 7f9bd3fff640 1 --2- 172.21.15.125:0/3120600696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd41538a0 0x7f9bd415d6f0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009870 tx=0x7f9bc401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d87b931fd03d3e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:43.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.871+0000 7f9bd2ffd640 1 -- 172.21.15.125:0/3120600696 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc40a8a30 con 0x7f9bd41538a0 2023-10-30T13:11:43.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.871+0000 7f9bd2ffd640 1 -- 172.21.15.125:0/3120600696 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc40a8b90 con 0x7f9bd41538a0 2023-10-30T13:11:43.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.871+0000 7f9bd2ffd640 1 -- 172.21.15.125:0/3120600696 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc40b0440 con 0x7f9bd41538a0 2023-10-30T13:11:43.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.871+0000 7f9bda765640 1 -- 172.21.15.125:0/3120600696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd41538a0 msgr2=0x7f9bd415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:43.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.871+0000 7f9bda765640 1 --2- 172.21.15.125:0/3120600696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd41538a0 0x7f9bd415d6f0 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009870 tx=0x7f9bc401ed30 comp rx=0 tx=0).stop 2023-10-30T13:11:43.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.871+0000 7f9bda765640 1 -- 172.21.15.125:0/3120600696 shutdown_connections 2023-10-30T13:11:43.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.871+0000 7f9bda765640 1 --2- 172.21.15.125:0/3120600696 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd41538a0 0x7f9bd415d6f0 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:43.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.871+0000 7f9bda765640 1 -- 172.21.15.125:0/3120600696 >> 172.21.15.125:0/3120600696 conn(0x7f9bd406bfe0 msgr2=0x7f9bd406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:43.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.871+0000 7f9bda765640 1 -- 172.21.15.125:0/3120600696 shutdown_connections 2023-10-30T13:11:43.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.871+0000 7f9bda765640 1 -- 172.21.15.125:0/3120600696 wait complete. 2023-10-30T13:11:43.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.871+0000 7f9bda765640 1 Processor -- start 2023-10-30T13:11:43.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bda765640 1 -- start start 2023-10-30T13:11:43.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bda765640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd40ca0e0 0x7f9bd41efaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:43.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bda765640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd40cf160 0x7f9bd40b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:43.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bda765640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd41538a0 0x7f9bd4158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:43.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bda765640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd4157520 con 0x7f9bd40ca0e0 2023-10-30T13:11:43.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bda765640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd4158380 con 0x7f9bd40cf160 2023-10-30T13:11:43.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bda765640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd4157770 con 0x7f9bd41538a0 2023-10-30T13:11:43.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bd3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd40ca0e0 0x7f9bd41efaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:43.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bd3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd40ca0e0 0x7f9bd41efaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60044/0 (socket says 172.21.15.125:60044) 2023-10-30T13:11:43.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bd3fff640 1 -- 172.21.15.125:0/2836189493 learned_addr learned my addr 172.21.15.125:0/2836189493 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:43.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bd37fe640 1 --2- 172.21.15.125:0/2836189493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd40cf160 0x7f9bd40b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:43.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bd8d02640 1 --2- 172.21.15.125:0/2836189493 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd41538a0 0x7f9bd4158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:43.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bd3fff640 1 -- 172.21.15.125:0/2836189493 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd41538a0 msgr2=0x7f9bd4158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:43.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bd3fff640 1 --2- 172.21.15.125:0/2836189493 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd41538a0 0x7f9bd4158c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:43.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bd3fff640 1 -- 172.21.15.125:0/2836189493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd40cf160 msgr2=0x7f9bd40b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:43.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bd3fff640 1 --2- 172.21.15.125:0/2836189493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd40cf160 0x7f9bd40b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:43.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bd3fff640 1 -- 172.21.15.125:0/2836189493 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc40094d0 con 0x7f9bd40ca0e0 2023-10-30T13:11:43.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bd3fff640 1 --2- 172.21.15.125:0/2836189493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd40ca0e0 0x7f9bd41efaa0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009870 tx=0x7f9bc401d570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:43.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bd0ff9640 1 -- 172.21.15.125:0/2836189493 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc40b1e30 con 0x7f9bd40ca0e0 2023-10-30T13:11:43.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bd0ff9640 1 -- 172.21.15.125:0/2836189493 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc40bb420 con 0x7f9bd40ca0e0 2023-10-30T13:11:43.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bda765640 1 -- 172.21.15.125:0/2836189493 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd4157a50 con 0x7f9bd40ca0e0 2023-10-30T13:11:43.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bd0ff9640 1 -- 172.21.15.125:0/2836189493 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc40bb790 con 0x7f9bd40ca0e0 2023-10-30T13:11:43.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bda765640 1 -- 172.21.15.125:0/2836189493 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd4158010 con 0x7f9bd40ca0e0 2023-10-30T13:11:43.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.875+0000 7f9bda765640 1 -- 172.21.15.125:0/2836189493 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd41581a0 con 0x7f9bd40ca0e0 2023-10-30T13:11:43.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.879+0000 7f9bd0ff9640 1 -- 172.21.15.125:0/2836189493 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9bc40b7040 con 0x7f9bd40ca0e0 2023-10-30T13:11:43.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.879+0000 7f9bd0ff9640 1 --2- 172.21.15.125:0/2836189493 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9bc00664a0 0x7f9bc0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:43.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.879+0000 7f9bd0ff9640 1 -- 172.21.15.125:0/2836189493 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7f9bc4122660 con 0x7f9bd40ca0e0 2023-10-30T13:11:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.879+0000 7f9bd37fe640 1 --2- 172.21.15.125:0/2836189493 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9bc00664a0 0x7f9bc0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:43.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.879+0000 7f9bd37fe640 1 --2- 172.21.15.125:0/2836189493 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9bc00664a0 0x7f9bc0068960 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8007fd0 tx=0x7f9bc8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:43.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:43.883+0000 7f9bd0ff9640 1 -- 172.21.15.125:0/2836189493 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9bc40ef2e0 con 0x7f9bd40ca0e0 2023-10-30T13:11:44.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:44.067+0000 7f9bda765640 1 -- 172.21.15.125:0/2836189493 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bd404e170 con 0x7f9bd40ca0e0 2023-10-30T13:11:44.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:44.067+0000 7f9bd0ff9640 1 -- 172.21.15.125:0/2836189493 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+5761 (secure 0 0 0) 0x7f9bc40f2900 con 0x7f9bd40ca0e0 2023-10-30T13:11:44.072 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:44.073 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:41.766852+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:44.073 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 198 2023-10-30T13:11:44.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:44.071+0000 7f9bda765640 1 -- 172.21.15.125:0/2836189493 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9bc00664a0 msgr2=0x7f9bc0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:44.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:44.071+0000 7f9bda765640 1 --2- 172.21.15.125:0/2836189493 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9bc00664a0 0x7f9bc0068960 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8007fd0 tx=0x7f9bc8009040 comp rx=0 tx=0).stop 2023-10-30T13:11:44.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:44.071+0000 7f9bda765640 1 -- 172.21.15.125:0/2836189493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd40ca0e0 msgr2=0x7f9bd41efaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:44.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:44.071+0000 7f9bda765640 1 --2- 172.21.15.125:0/2836189493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd40ca0e0 0x7f9bd41efaa0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f9bc4009870 tx=0x7f9bc401d570 comp rx=0 tx=0).stop 2023-10-30T13:11:44.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:44.071+0000 7f9bda765640 1 -- 172.21.15.125:0/2836189493 shutdown_connections 2023-10-30T13:11:44.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:44.071+0000 7f9bda765640 1 --2- 172.21.15.125:0/2836189493 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9bc00664a0 0x7f9bc0068960 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:44.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:44.071+0000 7f9bda765640 1 --2- 172.21.15.125:0/2836189493 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd41538a0 0x7f9bd4158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:44.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:44.071+0000 7f9bda765640 1 --2- 172.21.15.125:0/2836189493 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd40cf160 0x7f9bd40b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:44.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:44.071+0000 7f9bda765640 1 --2- 172.21.15.125:0/2836189493 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd40ca0e0 0x7f9bd41efaa0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:44.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:44.071+0000 7f9bda765640 1 -- 172.21.15.125:0/2836189493 >> 172.21.15.125:0/2836189493 conn(0x7f9bd406bfe0 msgr2=0x7f9bd41516c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:44.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:44.071+0000 7f9bda765640 1 -- 172.21.15.125:0/2836189493 shutdown_connections 2023-10-30T13:11:44.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:44.071+0000 7f9bda765640 1 -- 172.21.15.125:0/2836189493 wait complete. 2023-10-30T13:11:44.095 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 198, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 198, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_8301": { "addr": "172.21.15.125:6837/2540063352", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 2540063352, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 2540063352, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 2 ], "features": 4540138322906710015, "flags": 0, "gid": 8301, "incarnation": 168, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 6 }, "gid_8443": { "addr": "172.21.15.136:6839/3776186190", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3776186190, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3776186190, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8443, "incarnation": 196, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:reconnect", "state_seq": 37 }, "gid_8529": { "addr": "172.21.15.125:6836/3950044434", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3950044434, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3950044434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 8529, "incarnation": 179, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 78, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:11:41.766852+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8529, "mds_1": 8301, "mds_2": 8443 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:11:46.097 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:46.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.235+0000 7f129bad6640 1 Processor -- start 2023-10-30T13:11:46.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.235+0000 7f129bad6640 1 -- start start 2023-10-30T13:11:46.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.235+0000 7f129bad6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f12940cf160 0x7f12940cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:46.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.235+0000 7f129bad6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12940cb800 con 0x7f129415d6f0 2023-10-30T13:11:46.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.235+0000 7f129bad6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12940cb330 con 0x7f12941538a0 2023-10-30T13:11:46.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.235+0000 7f129bad6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12940cad70 con 0x7f12940cf160 2023-10-30T13:11:46.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.235+0000 7f1299071640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f129415d6f0 0x7f129415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48922/0 (socket says 172.21.15.125:48922) 2023-10-30T13:11:46.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.235+0000 7f1299071640 1 -- 172.21.15.125:0/1874420804 learned_addr learned my addr 172.21.15.125:0/1874420804 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:11:46.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.235+0000 7f129a073640 1 --2- 172.21.15.125:0/1874420804 >> v2:172.21.15.125:3300/0 conn(0x7f12940cf160 0x7f12940cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:46.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.235+0000 7f129a073640 1 -- 172.21.15.125:0/1874420804 >> v1:172.21.15.136:6789/0 conn(0x7f12941538a0 legacy=0x7f12940ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:46.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.235+0000 7f129a073640 1 -- 172.21.15.125:0/1874420804 >> v1:172.21.15.125:6789/0 conn(0x7f129415d6f0 legacy=0x7f129415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:46.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.235+0000 7f129a073640 1 -- 172.21.15.125:0/1874420804 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12940cb050 con 0x7f12940cf160 2023-10-30T13:11:46.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.239+0000 7f129a073640 1 --2- 172.21.15.125:0/1874420804 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12940cf160 0x7f12940cf590 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f128c009d90 tx=0x7f128c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=911b227dd480d2cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:46.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.239+0000 7f1298870640 1 -- 172.21.15.125:0/1874420804 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c0af600 con 0x7f12940cf160 2023-10-30T13:11:46.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.239+0000 7f1298870640 1 -- 172.21.15.125:0/1874420804 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c0ab050 con 0x7f12940cf160 2023-10-30T13:11:46.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.239+0000 7f1298870640 1 -- 172.21.15.125:0/1874420804 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c0af980 con 0x7f12940cf160 2023-10-30T13:11:46.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.239+0000 7f129bad6640 1 -- 172.21.15.125:0/1874420804 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12940cf160 msgr2=0x7f12940cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:46.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.239+0000 7f129bad6640 1 --2- 172.21.15.125:0/1874420804 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12940cf160 0x7f12940cf590 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7f128c009d90 tx=0x7f128c01f130 comp rx=0 tx=0).stop 2023-10-30T13:11:46.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.239+0000 7f129bad6640 1 -- 172.21.15.125:0/1874420804 shutdown_connections 2023-10-30T13:11:46.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.239+0000 7f129bad6640 1 --2- 172.21.15.125:0/1874420804 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12940cf160 0x7f12940cf590 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:46.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.239+0000 7f129bad6640 1 -- 172.21.15.125:0/1874420804 >> 172.21.15.125:0/1874420804 conn(0x7f129406bfe0 msgr2=0x7f129406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:46.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.239+0000 7f129bad6640 1 -- 172.21.15.125:0/1874420804 shutdown_connections 2023-10-30T13:11:46.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f129bad6640 1 -- 172.21.15.125:0/1874420804 wait complete. 2023-10-30T13:11:46.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f129bad6640 1 Processor -- start 2023-10-30T13:11:46.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f129bad6640 1 -- start start 2023-10-30T13:11:46.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f129bad6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12940cf160 0x7f12940b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:46.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f129bad6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12941538a0 0x7f129414e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:46.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f129bad6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f129415d6f0 0x7f12941f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:46.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f129bad6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1294158e30 con 0x7f12940cf160 2023-10-30T13:11:46.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f129bad6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1294158320 con 0x7f12941538a0 2023-10-30T13:11:46.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f129bad6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1294159710 con 0x7f129415d6f0 2023-10-30T13:11:46.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f1299872640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12940cf160 0x7f12940b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:46.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f1299872640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12940cf160 0x7f12940b4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60060/0 (socket says 172.21.15.125:60060) 2023-10-30T13:11:46.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f1299872640 1 -- 172.21.15.125:0/576980032 learned_addr learned my addr 172.21.15.125:0/576980032 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:46.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f129a073640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f129415d6f0 0x7f12941f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:46.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f1299071640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12941538a0 0x7f129414e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:46.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f1299872640 1 -- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f129415d6f0 msgr2=0x7f12941f2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:46.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f1299872640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f129415d6f0 0x7f12941f2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:46.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f1299872640 1 -- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12941538a0 msgr2=0x7f129414e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:46.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f1299872640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12941538a0 0x7f129414e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:46.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f1299872640 1 -- 172.21.15.125:0/576980032 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c009a70 con 0x7f12940cf160 2023-10-30T13:11:46.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f129a073640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f129415d6f0 0x7f12941f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:11:46.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f1299872640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12940cf160 0x7f12940b4450 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f129000c530 tx=0x7f129000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:46.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f128a7fc640 1 -- 172.21.15.125:0/576980032 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290005220 con 0x7f12940cf160 2023-10-30T13:11:46.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f128a7fc640 1 -- 172.21.15.125:0/576980032 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1290005380 con 0x7f12940cf160 2023-10-30T13:11:46.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f128a7fc640 1 -- 172.21.15.125:0/576980032 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1290007270 con 0x7f12940cf160 2023-10-30T13:11:46.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.243+0000 7f129bad6640 1 -- 172.21.15.125:0/576980032 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129414e110 con 0x7f12940cf160 2023-10-30T13:11:46.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.247+0000 7f129bad6640 1 -- 172.21.15.125:0/576980032 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1294158b50 con 0x7f12940cf160 2023-10-30T13:11:46.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.247+0000 7f129bad6640 1 -- 172.21.15.125:0/576980032 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1294159850 con 0x7f12940cf160 2023-10-30T13:11:46.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.247+0000 7f128a7fc640 1 -- 172.21.15.125:0/576980032 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f12900073d0 con 0x7f12940cf160 2023-10-30T13:11:46.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.247+0000 7f128a7fc640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12780664a0 0x7f1278068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:46.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.247+0000 7f128a7fc640 1 -- 172.21.15.125:0/576980032 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7f1290083d80 con 0x7f12940cf160 2023-10-30T13:11:46.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.247+0000 7f1299071640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12780664a0 0x7f1278068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:46.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.251+0000 7f1299071640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12780664a0 0x7f1278068960 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f12940c5ae0 tx=0x7f1284009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:46.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.251+0000 7f128a7fc640 1 -- 172.21.15.125:0/576980032 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f12900509c0 con 0x7f12940cf160 2023-10-30T13:11:46.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.435+0000 7f129bad6640 1 -- 172.21.15.125:0/576980032 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f129414e110 con 0x7f12940cf160 2023-10-30T13:11:46.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.435+0000 7f128a7fc640 1 -- 172.21.15.125:0/576980032 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+5761 (secure 0 0 0) 0x7f129414e110 con 0x7f12940cf160 2023-10-30T13:11:46.439 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:46.439 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:41.766852+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:46.440 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 198 2023-10-30T13:11:46.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.439+0000 7f129bad6640 1 -- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12780664a0 msgr2=0x7f1278068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:46.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.439+0000 7f129bad6640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12780664a0 0x7f1278068960 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f12940c5ae0 tx=0x7f1284009040 comp rx=0 tx=0).stop 2023-10-30T13:11:46.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.439+0000 7f129bad6640 1 -- 172.21.15.125:0/576980032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12940cf160 msgr2=0x7f12940b4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:46.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.439+0000 7f129bad6640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12940cf160 0x7f12940b4450 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f129000c530 tx=0x7f129000c9f0 comp rx=0 tx=0).stop 2023-10-30T13:11:46.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.439+0000 7f129bad6640 1 -- 172.21.15.125:0/576980032 shutdown_connections 2023-10-30T13:11:46.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.439+0000 7f129bad6640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f12780664a0 0x7f1278068960 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:46.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.439+0000 7f129bad6640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f129415d6f0 0x7f12941f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:46.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.439+0000 7f129bad6640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f12941538a0 0x7f129414e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:46.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.439+0000 7f129bad6640 1 --2- 172.21.15.125:0/576980032 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12940cf160 0x7f12940b4450 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:46.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.439+0000 7f129bad6640 1 -- 172.21.15.125:0/576980032 >> 172.21.15.125:0/576980032 conn(0x7f129406bfe0 msgr2=0x7f12940cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:46.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.439+0000 7f129bad6640 1 -- 172.21.15.125:0/576980032 shutdown_connections 2023-10-30T13:11:46.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:46.439+0000 7f129bad6640 1 -- 172.21.15.125:0/576980032 wait complete. 2023-10-30T13:11:46.461 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:11:48.462 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:48.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.607+0000 7fd0814c9640 1 Processor -- start 2023-10-30T13:11:48.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.607+0000 7fd0814c9640 1 -- start start 2023-10-30T13:11:48.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.611+0000 7fd0814c9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd07c1538a0 0x7fd07c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:48.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.611+0000 7fd0814c9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd07c0ca0e0 0x7fd07c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:48.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.611+0000 7fd0814c9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07c0cb800 con 0x7fd07c0cf160 2023-10-30T13:11:48.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.611+0000 7fd0814c9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07c15dc30 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.611+0000 7fd0814c9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07c0cb330 con 0x7fd07c1538a0 2023-10-30T13:11:48.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.611+0000 7fd0727fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd07c0ca0e0 0x7fd07c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:48.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.611+0000 7fd0727fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd07c0ca0e0 0x7fd07c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60064/0 (socket says 172.21.15.125:60064) 2023-10-30T13:11:48.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.611+0000 7fd07b7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd07c0cf160 0x7fd07c0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:48938/0 (socket says 172.21.15.125:48938) 2023-10-30T13:11:48.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.611+0000 7fd0727fc640 1 -- 172.21.15.125:0/1038172933 learned_addr learned my addr 172.21.15.125:0/1038172933 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:48.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.611+0000 7fd07affd640 1 --2- 172.21.15.125:0/1038172933 >> v2:172.21.15.136:3300/0 conn(0x7fd07c1538a0 0x7fd07c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:48.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.611+0000 7fd0727fc640 1 -- 172.21.15.125:0/1038172933 >> v2:172.21.15.136:3300/0 conn(0x7fd07c1538a0 msgr2=0x7fd07c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:48.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.611+0000 7fd0727fc640 1 --2- 172.21.15.125:0/1038172933 >> v2:172.21.15.136:3300/0 conn(0x7fd07c1538a0 0x7fd07c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:48.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.611+0000 7fd0727fc640 1 -- 172.21.15.125:0/1038172933 >> v1:172.21.15.125:6789/0 conn(0x7fd07c0cf160 legacy=0x7fd07c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:48.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.611+0000 7fd0727fc640 1 -- 172.21.15.125:0/1038172933 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c0cad70 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.615+0000 7fd0727fc640 1 --2- 172.21.15.125:0/1038172933 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd07c0ca0e0 0x7fd07c15a090 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fd07c0cfd10 tx=0x7fd06401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d82f08608f085d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:48.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.615+0000 7fd07a7fc640 1 -- 172.21.15.125:0/1038172933 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0640a89f0 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.615+0000 7fd07a7fc640 1 -- 172.21.15.125:0/1038172933 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0640a8b50 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.615+0000 7fd07a7fc640 1 -- 172.21.15.125:0/1038172933 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0640b0440 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.615+0000 7fd0814c9640 1 -- 172.21.15.125:0/1038172933 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd07c0ca0e0 msgr2=0x7fd07c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:48.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.615+0000 7fd0814c9640 1 --2- 172.21.15.125:0/1038172933 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd07c0ca0e0 0x7fd07c15a090 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fd07c0cfd10 tx=0x7fd06401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:11:48.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.615+0000 7fd0814c9640 1 -- 172.21.15.125:0/1038172933 shutdown_connections 2023-10-30T13:11:48.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.615+0000 7fd0814c9640 1 --2- 172.21.15.125:0/1038172933 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd07c0ca0e0 0x7fd07c15a090 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:48.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.615+0000 7fd0814c9640 1 --2- 172.21.15.125:0/1038172933 >> v2:172.21.15.136:3300/0 conn(0x7fd07c1538a0 0x7fd07c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:48.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.615+0000 7fd0814c9640 1 -- 172.21.15.125:0/1038172933 >> 172.21.15.125:0/1038172933 conn(0x7fd07c06bfe0 msgr2=0x7fd07c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:48.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.615+0000 7fd0814c9640 1 -- 172.21.15.125:0/1038172933 shutdown_connections 2023-10-30T13:11:48.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.615+0000 7fd0814c9640 1 -- 172.21.15.125:0/1038172933 wait complete. 2023-10-30T13:11:48.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.615+0000 7fd0814c9640 1 Processor -- start 2023-10-30T13:11:48.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.615+0000 7fd0814c9640 1 -- start start 2023-10-30T13:11:48.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd0814c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd07c0ca0e0 0x7fd07c140ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:48.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd0814c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd07c0cf160 0x7fd07c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:48.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd0814c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd07c1538a0 0x7fd07c0cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:48.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd0814c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1e4380 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd0814c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c158620 con 0x7fd07c0cf160 2023-10-30T13:11:48.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd0814c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c159690 con 0x7fd07c1538a0 2023-10-30T13:11:48.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd07affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd07c0ca0e0 0x7fd07c140ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:48.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd07affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd07c0ca0e0 0x7fd07c140ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60072/0 (socket says 172.21.15.125:60072) 2023-10-30T13:11:48.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd07affd640 1 -- 172.21.15.125:0/2147474445 learned_addr learned my addr 172.21.15.125:0/2147474445 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:48.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd0727fc640 1 --2- 172.21.15.125:0/2147474445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd07c0cf160 0x7fd07c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:48.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd07b7fe640 1 --2- 172.21.15.125:0/2147474445 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd07c1538a0 0x7fd07c0cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:48.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd07affd640 1 -- 172.21.15.125:0/2147474445 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd07c1538a0 msgr2=0x7fd07c0cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:48.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd07affd640 1 --2- 172.21.15.125:0/2147474445 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd07c1538a0 0x7fd07c0cd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:48.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd07affd640 1 -- 172.21.15.125:0/2147474445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd07c0cf160 msgr2=0x7fd07c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:48.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd07affd640 1 --2- 172.21.15.125:0/2147474445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd07c0cf160 0x7fd07c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:48.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd07affd640 1 -- 172.21.15.125:0/2147474445 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0640094d0 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd07affd640 1 --2- 172.21.15.125:0/2147474445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd07c0ca0e0 0x7fd07c140ea0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00d620 tx=0x7fd06c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:48.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd073fff640 1 -- 172.21.15.125:0/2147474445 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c00b620 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd073fff640 1 -- 172.21.15.125:0/2147474445 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c00b780 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd0814c9640 1 -- 172.21.15.125:0/2147474445 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07c1e51e0 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd073fff640 1 -- 172.21.15.125:0/2147474445 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c00bb10 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd0814c9640 1 -- 172.21.15.125:0/2147474445 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07c1e5610 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd0814c9640 1 -- 172.21.15.125:0/2147474445 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07c0b2a50 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.619+0000 7fd073fff640 1 -- 172.21.15.125:0/2147474445 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd06c00bc70 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.623+0000 7fd073fff640 1 --2- 172.21.15.125:0/2147474445 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd05c0664d0 0x7fd05c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:48.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.623+0000 7fd073fff640 1 -- 172.21.15.125:0/2147474445 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7fd06c084aa0 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.623+0000 7fd0727fc640 1 --2- 172.21.15.125:0/2147474445 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd05c0664d0 0x7fd05c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:48.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.623+0000 7fd0727fc640 1 --2- 172.21.15.125:0/2147474445 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd05c0664d0 0x7fd05c068990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fd0640b2000 tx=0x7fd064008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:48.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.627+0000 7fd073fff640 1 -- 172.21.15.125:0/2147474445 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd06c051780 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.811+0000 7fd0814c9640 1 -- 172.21.15.125:0/2147474445 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd07c04e170 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.811+0000 7fd073fff640 1 -- 172.21.15.125:0/2147474445 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+5758 (secure 0 0 0) 0x7fd06c007380 con 0x7fd07c0ca0e0 2023-10-30T13:11:48.814 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:48.814 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:47.718720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:48.814 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 199 2023-10-30T13:11:48.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.811+0000 7fd0814c9640 1 -- 172.21.15.125:0/2147474445 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd05c0664d0 msgr2=0x7fd05c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:48.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.811+0000 7fd0814c9640 1 --2- 172.21.15.125:0/2147474445 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd05c0664d0 0x7fd05c068990 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fd0640b2000 tx=0x7fd064008040 comp rx=0 tx=0).stop 2023-10-30T13:11:48.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.811+0000 7fd0814c9640 1 -- 172.21.15.125:0/2147474445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd07c0ca0e0 msgr2=0x7fd07c140ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:48.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.811+0000 7fd0814c9640 1 --2- 172.21.15.125:0/2147474445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd07c0ca0e0 0x7fd07c140ea0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fd06c00d620 tx=0x7fd06c00dae0 comp rx=0 tx=0).stop 2023-10-30T13:11:48.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.811+0000 7fd0814c9640 1 -- 172.21.15.125:0/2147474445 shutdown_connections 2023-10-30T13:11:48.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.811+0000 7fd0814c9640 1 --2- 172.21.15.125:0/2147474445 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd05c0664d0 0x7fd05c068990 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:48.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.811+0000 7fd0814c9640 1 --2- 172.21.15.125:0/2147474445 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd07c1538a0 0x7fd07c0cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:48.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.811+0000 7fd0814c9640 1 --2- 172.21.15.125:0/2147474445 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd07c0cf160 0x7fd07c0ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:48.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.811+0000 7fd0814c9640 1 --2- 172.21.15.125:0/2147474445 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd07c0ca0e0 0x7fd07c140ea0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:48.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.811+0000 7fd0814c9640 1 -- 172.21.15.125:0/2147474445 >> 172.21.15.125:0/2147474445 conn(0x7fd07c06bfe0 msgr2=0x7fd07c0c5210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:48.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.811+0000 7fd0814c9640 1 -- 172.21.15.125:0/2147474445 shutdown_connections 2023-10-30T13:11:48.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:48.811+0000 7fd0814c9640 1 -- 172.21.15.125:0/2147474445 wait complete. 2023-10-30T13:11:48.835 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:11:50.836 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:50.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.983+0000 7f8fc8dd3640 1 Processor -- start 2023-10-30T13:11:50.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.983+0000 7f8fc8dd3640 1 -- start start 2023-10-30T13:11:50.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.987+0000 7f8fc8dd3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8fc4153bf0 0x7f8fc4071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:50.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.987+0000 7f8fc8dd3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8fc40c5f60 0x7f8fc40cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:50.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.987+0000 7f8fc8dd3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc40700b0 con 0x7f8fc40c6fe0 2023-10-30T13:11:50.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.987+0000 7f8fc8dd3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc40c6810 con 0x7f8fc4153bf0 2023-10-30T13:11:50.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.987+0000 7f8fc8dd3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc4159900 con 0x7f8fc40c5f60 2023-10-30T13:11:50.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.987+0000 7f8fc259c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8fc4153bf0 0x7f8fc4071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:50.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.987+0000 7f8fc259c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8fc4153bf0 0x7f8fc4071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50902/0 (socket says 172.21.15.125:50902) 2023-10-30T13:11:50.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.987+0000 7f8fc259c640 1 -- 172.21.15.125:0/3109696228 learned_addr learned my addr 172.21.15.125:0/3109696228 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:50.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.987+0000 7f8fc2d9d640 1 --2- 172.21.15.125:0/3109696228 >> v2:172.21.15.136:3300/0 conn(0x7f8fc40c5f60 0x7f8fc40cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:50.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.987+0000 7f8fc259c640 1 -- 172.21.15.125:0/3109696228 >> v2:172.21.15.136:3300/0 conn(0x7f8fc40c5f60 msgr2=0x7f8fc40cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:50.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.987+0000 7f8fc259c640 1 --2- 172.21.15.125:0/3109696228 >> v2:172.21.15.136:3300/0 conn(0x7f8fc40c5f60 0x7f8fc40cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:50.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.987+0000 7f8fc259c640 1 -- 172.21.15.125:0/3109696228 >> v1:172.21.15.136:6789/0 conn(0x7f8fc40c6fe0 legacy=0x7f8fc40c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:50.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.987+0000 7f8fc259c640 1 -- 172.21.15.125:0/3109696228 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc40c6530 con 0x7f8fc4153bf0 2023-10-30T13:11:50.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.991+0000 7f8fc259c640 1 --2- 172.21.15.125:0/3109696228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8fc4153bf0 0x7f8fc4071260 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f8fc406ce90 tx=0x7f8fb401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a9e7e2e2c79a5e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:50.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.991+0000 7f8fc159a640 1 -- 172.21.15.125:0/3109696228 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb40a89f0 con 0x7f8fc4153bf0 2023-10-30T13:11:50.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.991+0000 7f8fc159a640 1 -- 172.21.15.125:0/3109696228 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb40a8b50 con 0x7f8fc4153bf0 2023-10-30T13:11:50.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.991+0000 7f8fc159a640 1 -- 172.21.15.125:0/3109696228 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb40b0440 con 0x7f8fc4153bf0 2023-10-30T13:11:50.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.991+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/3109696228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8fc4153bf0 msgr2=0x7f8fc4071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:50.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.991+0000 7f8fc8dd3640 1 --2- 172.21.15.125:0/3109696228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8fc4153bf0 0x7f8fc4071260 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f8fc406ce90 tx=0x7f8fb401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:11:50.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.991+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/3109696228 shutdown_connections 2023-10-30T13:11:50.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.991+0000 7f8fc8dd3640 1 --2- 172.21.15.125:0/3109696228 >> v2:172.21.15.136:3300/0 conn(0x7f8fc40c5f60 0x7f8fc40cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:50.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.991+0000 7f8fc8dd3640 1 --2- 172.21.15.125:0/3109696228 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8fc4153bf0 0x7f8fc4071260 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:50.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.991+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/3109696228 >> 172.21.15.125:0/3109696228 conn(0x7f8fc406c6e0 msgr2=0x7f8fc4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:50.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.991+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/3109696228 shutdown_connections 2023-10-30T13:11:50.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.991+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/3109696228 wait complete. 2023-10-30T13:11:50.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.991+0000 7f8fc8dd3640 1 Processor -- start 2023-10-30T13:11:50.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.991+0000 7f8fc8dd3640 1 -- start start 2023-10-30T13:11:50.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc8dd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fc40c5f60 0x7f8fc4156ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:50.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc8dd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8fc40c6fe0 0x7f8fc4154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:50.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc8dd3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8fc4153bf0 0x7f8fc40c4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:50.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc8dd3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc41549b0 con 0x7f8fc4153bf0 2023-10-30T13:11:50.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc8dd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc414e690 con 0x7f8fc40c5f60 2023-10-30T13:11:50.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc8dd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc4155310 con 0x7f8fc40c6fe0 2023-10-30T13:11:51.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc2d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8fc4153bf0 0x7f8fc40c4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:51.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc2d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8fc4153bf0 0x7f8fc40c4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50910/0 (socket says 172.21.15.125:50910) 2023-10-30T13:11:51.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc2d9d640 1 -- 172.21.15.125:0/1665177866 learned_addr learned my addr 172.21.15.125:0/1665177866 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:51.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc259c640 1 --2- 172.21.15.125:0/1665177866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fc40c5f60 0x7f8fc4156ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:51.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc1d9b640 1 --2- 172.21.15.125:0/1665177866 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8fc40c6fe0 0x7f8fc4154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:51.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc2d9d640 1 -- 172.21.15.125:0/1665177866 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8fc40c6fe0 msgr2=0x7f8fc4154dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:51.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc2d9d640 1 --2- 172.21.15.125:0/1665177866 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8fc40c6fe0 0x7f8fc4154dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:51.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc2d9d640 1 -- 172.21.15.125:0/1665177866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fc40c5f60 msgr2=0x7f8fc4156ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:51.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc2d9d640 1 --2- 172.21.15.125:0/1665177866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fc40c5f60 0x7f8fc4156ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:51.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc2d9d640 1 -- 172.21.15.125:0/1665177866 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb40094d0 con 0x7f8fc4153bf0 2023-10-30T13:11:51.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc2d9d640 1 --2- 172.21.15.125:0/1665177866 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8fc4153bf0 0x7f8fc40c4400 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc007e70 tx=0x7f8fbc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:51.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fb2ffd640 1 -- 172.21.15.125:0/1665177866 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc014620 con 0x7f8fc4153bf0 2023-10-30T13:11:51.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fb2ffd640 1 -- 172.21.15.125:0/1665177866 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fbc0147a0 con 0x7f8fc4153bf0 2023-10-30T13:11:51.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fb2ffd640 1 -- 172.21.15.125:0/1665177866 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc014b50 con 0x7f8fc4153bf0 2023-10-30T13:11:51.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/1665177866 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc40ca7d0 con 0x7f8fc4153bf0 2023-10-30T13:11:51.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/1665177866 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc40ca4f0 con 0x7f8fc4153bf0 2023-10-30T13:11:51.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/1665177866 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc4154af0 con 0x7f8fc4153bf0 2023-10-30T13:11:51.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.995+0000 7f8fb2ffd640 1 -- 172.21.15.125:0/1665177866 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8fbc014cb0 con 0x7f8fc4153bf0 2023-10-30T13:11:51.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.999+0000 7f8fb2ffd640 1 --2- 172.21.15.125:0/1665177866 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8fa40664a0 0x7f8fa4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:51.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.999+0000 7f8fb2ffd640 1 -- 172.21.15.125:0/1665177866 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7f8fbc084fb0 con 0x7f8fc4153bf0 2023-10-30T13:11:51.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.999+0000 7f8fc259c640 1 --2- 172.21.15.125:0/1665177866 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8fa40664a0 0x7f8fa4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:51.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:50.999+0000 7f8fc259c640 1 --2- 172.21.15.125:0/1665177866 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8fa40664a0 0x7f8fa4068960 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40b2000 tx=0x7f8fb40aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:51.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.003+0000 7f8fb2ffd640 1 -- 172.21.15.125:0/1665177866 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8fbc0b6090 con 0x7f8fc4153bf0 2023-10-30T13:11:51.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.183+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/1665177866 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8fc404e170 con 0x7f8fc4153bf0 2023-10-30T13:11:51.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.183+0000 7f8fb2ffd640 1 -- 172.21.15.125:0/1665177866 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f8fbc055700 con 0x7f8fc4153bf0 2023-10-30T13:11:51.186 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:51.186 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:47.718720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:51.188 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 199 2023-10-30T13:11:51.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.183+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/1665177866 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8fa40664a0 msgr2=0x7f8fa4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:51.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.183+0000 7f8fc8dd3640 1 --2- 172.21.15.125:0/1665177866 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8fa40664a0 0x7f8fa4068960 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f8fb40b2000 tx=0x7f8fb40aa070 comp rx=0 tx=0).stop 2023-10-30T13:11:51.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.183+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/1665177866 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8fc4153bf0 msgr2=0x7f8fc40c4400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:51.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.183+0000 7f8fc8dd3640 1 --2- 172.21.15.125:0/1665177866 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8fc4153bf0 0x7f8fc40c4400 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc007e70 tx=0x7f8fbc00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:11:51.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.183+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/1665177866 shutdown_connections 2023-10-30T13:11:51.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.183+0000 7f8fc8dd3640 1 --2- 172.21.15.125:0/1665177866 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8fa40664a0 0x7f8fa4068960 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:51.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.183+0000 7f8fc8dd3640 1 --2- 172.21.15.125:0/1665177866 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8fc4153bf0 0x7f8fc40c4400 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:51.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.183+0000 7f8fc8dd3640 1 --2- 172.21.15.125:0/1665177866 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8fc40c6fe0 0x7f8fc4154dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:51.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.183+0000 7f8fc8dd3640 1 --2- 172.21.15.125:0/1665177866 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8fc40c5f60 0x7f8fc4156ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:51.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.183+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/1665177866 >> 172.21.15.125:0/1665177866 conn(0x7f8fc406c6e0 msgr2=0x7f8fc41e5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:51.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.183+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/1665177866 shutdown_connections 2023-10-30T13:11:51.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:51.183+0000 7f8fc8dd3640 1 -- 172.21.15.125:0/1665177866 wait complete. 2023-10-30T13:11:51.207 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:11:53.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:53.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.351+0000 7fe24d7c5640 1 Processor -- start 2023-10-30T13:11:53.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.351+0000 7fe24d7c5640 1 -- start start 2023-10-30T13:11:53.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe24d7c5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe2481538a0 0x7fe24815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:53.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe24d7c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2480ca0e0 0x7fe24815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:53.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe24d7c5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2480cb800 con 0x7fe2480cf160 2023-10-30T13:11:53.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe24d7c5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe24815dc30 con 0x7fe2481538a0 2023-10-30T13:11:53.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe24d7c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2480cb330 con 0x7fe2480ca0e0 2023-10-30T13:11:53.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe246ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe2481538a0 0x7fe24815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:53.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe2477fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe2480cf160 0x7fe2480cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35966/0 (socket says 172.21.15.125:35966) 2023-10-30T13:11:53.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe2477fe640 1 -- 172.21.15.125:0/4285128530 learned_addr learned my addr 172.21.15.125:0/4285128530 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:11:53.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe23ffff640 1 --2- 172.21.15.125:0/4285128530 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2480ca0e0 0x7fe24815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:53.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe246ffd640 1 -- 172.21.15.125:0/4285128530 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2480ca0e0 msgr2=0x7fe24815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:53.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe246ffd640 1 --2- 172.21.15.125:0/4285128530 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2480ca0e0 0x7fe24815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:53.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe246ffd640 1 -- 172.21.15.125:0/4285128530 >> v1:172.21.15.125:6789/0 conn(0x7fe2480cf160 legacy=0x7fe2480cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:53.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe246ffd640 1 -- 172.21.15.125:0/4285128530 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2480cad70 con 0x7fe2481538a0 2023-10-30T13:11:53.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe2467fc640 1 -- 172.21.15.125:0/4285128530 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972989936 0 0) 0x7fe2480cb800 con 0x7fe2480cf160 2023-10-30T13:11:53.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe246ffd640 1 --2- 172.21.15.125:0/4285128530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2481538a0 0x7fe24815d6f0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fe2380097c0 tx=0x7fe23801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82c27dfb52d9ecc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:53.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe2467fc640 1 -- 172.21.15.125:0/4285128530 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2380a8970 con 0x7fe2481538a0 2023-10-30T13:11:53.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.355+0000 7fe2467fc640 1 -- 172.21.15.125:0/4285128530 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2380a8ad0 con 0x7fe2481538a0 2023-10-30T13:11:53.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe2467fc640 1 -- 172.21.15.125:0/4285128530 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2380a8e60 con 0x7fe2481538a0 2023-10-30T13:11:53.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 -- 172.21.15.125:0/4285128530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2481538a0 msgr2=0x7fe24815d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:53.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 --2- 172.21.15.125:0/4285128530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2481538a0 0x7fe24815d6f0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fe2380097c0 tx=0x7fe23801ec70 comp rx=0 tx=0).stop 2023-10-30T13:11:53.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 -- 172.21.15.125:0/4285128530 shutdown_connections 2023-10-30T13:11:53.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 --2- 172.21.15.125:0/4285128530 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2480ca0e0 0x7fe24815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:53.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 --2- 172.21.15.125:0/4285128530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2481538a0 0x7fe24815d6f0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:53.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 -- 172.21.15.125:0/4285128530 >> 172.21.15.125:0/4285128530 conn(0x7fe24806bfe0 msgr2=0x7fe24806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:53.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 -- 172.21.15.125:0/4285128530 shutdown_connections 2023-10-30T13:11:53.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 -- 172.21.15.125:0/4285128530 wait complete. 2023-10-30T13:11:53.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 Processor -- start 2023-10-30T13:11:53.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 -- start start 2023-10-30T13:11:53.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2480ca0e0 0x7fe248140530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:53.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2480cf160 0x7fe248146bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:53.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2481538a0 0x7fe248147110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:53.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2480ce8c0 con 0x7fe2480cf160 2023-10-30T13:11:53.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe24814eed0 con 0x7fe2480ca0e0 2023-10-30T13:11:53.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe24d7c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2481f3290 con 0x7fe2481538a0 2023-10-30T13:11:53.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe23ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2480cf160 0x7fe248146bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:53.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe23ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2480cf160 0x7fe248146bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50928/0 (socket says 172.21.15.125:50928) 2023-10-30T13:11:53.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe23ffff640 1 -- 172.21.15.125:0/2676023544 learned_addr learned my addr 172.21.15.125:0/2676023544 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:53.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe246ffd640 1 --2- 172.21.15.125:0/2676023544 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2480ca0e0 0x7fe248140530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:53.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe2477fe640 1 --2- 172.21.15.125:0/2676023544 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2481538a0 0x7fe248147110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:53.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe23ffff640 1 -- 172.21.15.125:0/2676023544 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2481538a0 msgr2=0x7fe248147110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:53.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.359+0000 7fe23ffff640 1 --2- 172.21.15.125:0/2676023544 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2481538a0 0x7fe248147110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:53.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.363+0000 7fe23ffff640 1 -- 172.21.15.125:0/2676023544 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2480ca0e0 msgr2=0x7fe248140530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:53.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.363+0000 7fe23ffff640 1 --2- 172.21.15.125:0/2676023544 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2480ca0e0 0x7fe248140530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:53.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.363+0000 7fe23ffff640 1 -- 172.21.15.125:0/2676023544 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe238009420 con 0x7fe2480cf160 2023-10-30T13:11:53.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.363+0000 7fe23ffff640 1 --2- 172.21.15.125:0/2676023544 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2480cf160 0x7fe248146bd0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fe23000d800 tx=0x7fe23000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:53.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.363+0000 7fe23f7fe640 1 -- 172.21.15.125:0/2676023544 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23000b660 con 0x7fe2480cf160 2023-10-30T13:11:53.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.363+0000 7fe23f7fe640 1 -- 172.21.15.125:0/2676023544 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe23000b7c0 con 0x7fe2480cf160 2023-10-30T13:11:53.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.363+0000 7fe24d7c5640 1 -- 172.21.15.125:0/2676023544 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2481f2430 con 0x7fe2480cf160 2023-10-30T13:11:53.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.363+0000 7fe23f7fe640 1 -- 172.21.15.125:0/2676023544 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe23000bb30 con 0x7fe2480cf160 2023-10-30T13:11:53.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.363+0000 7fe24d7c5640 1 -- 172.21.15.125:0/2676023544 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2481f29f0 con 0x7fe2480cf160 2023-10-30T13:11:53.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.363+0000 7fe24d7c5640 1 -- 172.21.15.125:0/2676023544 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2480b2a50 con 0x7fe2480cf160 2023-10-30T13:11:53.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.363+0000 7fe23f7fe640 1 -- 172.21.15.125:0/2676023544 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe23000bc90 con 0x7fe2480cf160 2023-10-30T13:11:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.363+0000 7fe23f7fe640 1 --2- 172.21.15.125:0/2676023544 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2280664a0 0x7fe228068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.363+0000 7fe23f7fe640 1 -- 172.21.15.125:0/2676023544 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7fe230084d40 con 0x7fe2480cf160 2023-10-30T13:11:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.363+0000 7fe246ffd640 1 --2- 172.21.15.125:0/2676023544 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2280664a0 0x7fe228068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.367+0000 7fe246ffd640 1 --2- 172.21.15.125:0/2676023544 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2280664a0 0x7fe228068960 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe2380b2000 tx=0x7fe238008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:53.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.367+0000 7fe23f7fe640 1 -- 172.21.15.125:0/2676023544 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe230051860 con 0x7fe2480cf160 2023-10-30T13:11:53.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.547+0000 7fe24d7c5640 1 -- 172.21.15.125:0/2676023544 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe24804e170 con 0x7fe2480cf160 2023-10-30T13:11:53.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.547+0000 7fe23f7fe640 1 -- 172.21.15.125:0/2676023544 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 199 v199) v1 ==== 95+0+5758 (secure 0 0 0) 0x7fe230011340 con 0x7fe2480cf160 2023-10-30T13:11:53.552 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:53.552 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":199,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":199,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:47.718720+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:rejoin","state_seq":39,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:53.552 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 199 2023-10-30T13:11:53.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.551+0000 7fe24d7c5640 1 -- 172.21.15.125:0/2676023544 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2280664a0 msgr2=0x7fe228068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:53.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.551+0000 7fe24d7c5640 1 --2- 172.21.15.125:0/2676023544 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2280664a0 0x7fe228068960 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fe2380b2000 tx=0x7fe238008040 comp rx=0 tx=0).stop 2023-10-30T13:11:53.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.551+0000 7fe24d7c5640 1 -- 172.21.15.125:0/2676023544 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2480cf160 msgr2=0x7fe248146bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:53.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.551+0000 7fe24d7c5640 1 --2- 172.21.15.125:0/2676023544 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2480cf160 0x7fe248146bd0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fe23000d800 tx=0x7fe23000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:11:53.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.551+0000 7fe24d7c5640 1 -- 172.21.15.125:0/2676023544 shutdown_connections 2023-10-30T13:11:53.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.551+0000 7fe24d7c5640 1 --2- 172.21.15.125:0/2676023544 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe2280664a0 0x7fe228068960 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:53.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.551+0000 7fe24d7c5640 1 --2- 172.21.15.125:0/2676023544 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe2481538a0 0x7fe248147110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:53.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.551+0000 7fe24d7c5640 1 --2- 172.21.15.125:0/2676023544 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe2480cf160 0x7fe248146bd0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:53.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.551+0000 7fe24d7c5640 1 --2- 172.21.15.125:0/2676023544 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe2480ca0e0 0x7fe248140530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:53.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.551+0000 7fe24d7c5640 1 -- 172.21.15.125:0/2676023544 >> 172.21.15.125:0/2676023544 conn(0x7fe24806bfe0 msgr2=0x7fe2481527a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:53.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.551+0000 7fe24d7c5640 1 -- 172.21.15.125:0/2676023544 shutdown_connections 2023-10-30T13:11:53.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:53.551+0000 7fe24d7c5640 1 -- 172.21.15.125:0/2676023544 wait complete. 2023-10-30T13:11:53.573 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:11:55.574 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:55.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.723+0000 7f43e9817640 1 Processor -- start 2023-10-30T13:11:55.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.723+0000 7f43e9817640 1 -- start start 2023-10-30T13:11:55.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.723+0000 7f43e9817640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f43e406f7a0 0x7f43e406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:55.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.723+0000 7f43e9817640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f43e4071110 0x7f43e4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:55.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.723+0000 7f43e9817640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e4066cf0 con 0x7f43e40c6bd0 2023-10-30T13:11:55.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.723+0000 7f43e9817640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e40700c0 con 0x7f43e4071110 2023-10-30T13:11:55.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.723+0000 7f43e9817640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43e40c70d0 con 0x7f43e406f7a0 2023-10-30T13:11:55.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.727+0000 7f43e37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f43e4071110 0x7f43e4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:55.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.727+0000 7f43e37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f43e4071110 0x7f43e4158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50936/0 (socket says 172.21.15.125:50936) 2023-10-30T13:11:55.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.727+0000 7f43e37fe640 1 -- 172.21.15.125:0/2205993102 learned_addr learned my addr 172.21.15.125:0/2205993102 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:55.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.727+0000 7f43e2ffd640 1 --2- 172.21.15.125:0/2205993102 >> v2:172.21.15.136:3300/0 conn(0x7f43e406f7a0 0x7f43e406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.727+0000 7f43e37fe640 1 -- 172.21.15.125:0/2205993102 >> v2:172.21.15.136:3300/0 conn(0x7f43e406f7a0 msgr2=0x7f43e406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.727+0000 7f43e37fe640 1 --2- 172.21.15.125:0/2205993102 >> v2:172.21.15.136:3300/0 conn(0x7f43e406f7a0 0x7f43e406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.727+0000 7f43e37fe640 1 -- 172.21.15.125:0/2205993102 >> v1:172.21.15.136:6789/0 conn(0x7f43e40c6bd0 legacy=0x7f43e4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:11:55.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.727+0000 7f43e37fe640 1 -- 172.21.15.125:0/2205993102 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e40c7390 con 0x7f43e4071110 2023-10-30T13:11:55.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.727+0000 7f43e37fe640 1 --2- 172.21.15.125:0/2205993102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43e4071110 0x7f43e4158110 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f43e406c830 tx=0x7f43dc01ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50c3924eec681147 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:55.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.727+0000 7f43e27fc640 1 -- 172.21.15.125:0/2205993102 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc0a89b0 con 0x7f43e4071110 2023-10-30T13:11:55.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.727+0000 7f43e27fc640 1 -- 172.21.15.125:0/2205993102 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43dc0a8b10 con 0x7f43e4071110 2023-10-30T13:11:55.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.727+0000 7f43e27fc640 1 -- 172.21.15.125:0/2205993102 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43dc0a8ea0 con 0x7f43e4071110 2023-10-30T13:11:55.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 -- 172.21.15.125:0/2205993102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43e4071110 msgr2=0x7f43e4158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:55.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 --2- 172.21.15.125:0/2205993102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43e4071110 0x7f43e4158110 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7f43e406c830 tx=0x7f43dc01ecb0 comp rx=0 tx=0).stop 2023-10-30T13:11:55.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 -- 172.21.15.125:0/2205993102 shutdown_connections 2023-10-30T13:11:55.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 --2- 172.21.15.125:0/2205993102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43e4071110 0x7f43e4158110 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:55.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 --2- 172.21.15.125:0/2205993102 >> v2:172.21.15.136:3300/0 conn(0x7f43e406f7a0 0x7f43e406fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:55.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 -- 172.21.15.125:0/2205993102 >> 172.21.15.125:0/2205993102 conn(0x7f43e40c7910 msgr2=0x7f43e40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:55.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 -- 172.21.15.125:0/2205993102 shutdown_connections 2023-10-30T13:11:55.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 -- 172.21.15.125:0/2205993102 wait complete. 2023-10-30T13:11:55.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 Processor -- start 2023-10-30T13:11:55.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 -- start start 2023-10-30T13:11:55.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43e406f7a0 0x7f43e4154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:55.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f43e4071110 0x7f43e4157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:55.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43e40c6bd0 0x7f43e41ef280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:55.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e41eedf0 con 0x7f43e406f7a0 2023-10-30T13:11:55.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43e4155550 con 0x7f43e40c6bd0 2023-10-30T13:11:55.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e9817640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43e41eeae0 con 0x7f43e4071110 2023-10-30T13:11:55.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e2ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43e406f7a0 0x7f43e4154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:55.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e2ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43e406f7a0 0x7f43e4154d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50952/0 (socket says 172.21.15.125:50952) 2023-10-30T13:11:55.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e2ffd640 1 -- 172.21.15.125:0/1087138286 learned_addr learned my addr 172.21.15.125:0/1087138286 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:55.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43da7fc640 1 --2- 172.21.15.125:0/1087138286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f43e4071110 0x7f43e4157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:55.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e37fe640 1 --2- 172.21.15.125:0/1087138286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43e40c6bd0 0x7f43e41ef280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:55.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e2ffd640 1 -- 172.21.15.125:0/1087138286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f43e4071110 msgr2=0x7f43e4157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:55.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e2ffd640 1 --2- 172.21.15.125:0/1087138286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f43e4071110 0x7f43e4157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:55.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e2ffd640 1 -- 172.21.15.125:0/1087138286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43e40c6bd0 msgr2=0x7f43e41ef280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:55.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e2ffd640 1 --2- 172.21.15.125:0/1087138286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43e40c6bd0 0x7f43e41ef280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:55.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.731+0000 7f43e2ffd640 1 -- 172.21.15.125:0/1087138286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43dc0094d0 con 0x7f43e406f7a0 2023-10-30T13:11:55.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.735+0000 7f43e2ffd640 1 --2- 172.21.15.125:0/1087138286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43e406f7a0 0x7f43e4154d90 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f43d400d620 tx=0x7f43d400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:55.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.735+0000 7f43e0ff9640 1 -- 172.21.15.125:0/1087138286 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d400b620 con 0x7f43e406f7a0 2023-10-30T13:11:55.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.735+0000 7f43e0ff9640 1 -- 172.21.15.125:0/1087138286 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43d400b780 con 0x7f43e406f7a0 2023-10-30T13:11:55.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.735+0000 7f43e0ff9640 1 -- 172.21.15.125:0/1087138286 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d400bb10 con 0x7f43e406f7a0 2023-10-30T13:11:55.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.735+0000 7f43e9817640 1 -- 172.21.15.125:0/1087138286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e4156900 con 0x7f43e406f7a0 2023-10-30T13:11:55.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.735+0000 7f43e9817640 1 -- 172.21.15.125:0/1087138286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e4156ec0 con 0x7f43e406f7a0 2023-10-30T13:11:55.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.735+0000 7f43e9817640 1 -- 172.21.15.125:0/1087138286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e404e170 con 0x7f43e406f7a0 2023-10-30T13:11:55.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.735+0000 7f43e0ff9640 1 -- 172.21.15.125:0/1087138286 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f43d400bc70 con 0x7f43e406f7a0 2023-10-30T13:11:55.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.739+0000 7f43e0ff9640 1 --2- 172.21.15.125:0/1087138286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f43c40664a0 0x7f43c4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:55.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.739+0000 7f43e0ff9640 1 -- 172.21.15.125:0/1087138286 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7f43d4084c30 con 0x7f43e406f7a0 2023-10-30T13:11:55.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.739+0000 7f43da7fc640 1 --2- 172.21.15.125:0/1087138286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f43c40664a0 0x7f43c4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:55.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.739+0000 7f43da7fc640 1 --2- 172.21.15.125:0/1087138286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f43c40664a0 0x7f43c4068960 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f43e41e6410 tx=0x7f43cc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:55.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.743+0000 7f43e0ff9640 1 -- 172.21.15.125:0/1087138286 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f43d40517f0 con 0x7f43e406f7a0 2023-10-30T13:11:55.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.923+0000 7f43e9817640 1 -- 172.21.15.125:0/1087138286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43e40c7e70 con 0x7f43e406f7a0 2023-10-30T13:11:55.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.927+0000 7f43e0ff9640 1 -- 172.21.15.125:0/1087138286 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f43d4007380 con 0x7f43e406f7a0 2023-10-30T13:11:55.929 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:55.930 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:55.584904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:55.931 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 200 2023-10-30T13:11:55.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.927+0000 7f43e9817640 1 -- 172.21.15.125:0/1087138286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f43c40664a0 msgr2=0x7f43c4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:55.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.927+0000 7f43e9817640 1 --2- 172.21.15.125:0/1087138286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f43c40664a0 0x7f43c4068960 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f43e41e6410 tx=0x7f43cc009040 comp rx=0 tx=0).stop 2023-10-30T13:11:55.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.927+0000 7f43e9817640 1 -- 172.21.15.125:0/1087138286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43e406f7a0 msgr2=0x7f43e4154d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:55.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.927+0000 7f43e9817640 1 --2- 172.21.15.125:0/1087138286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43e406f7a0 0x7f43e4154d90 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7f43d400d620 tx=0x7f43d400dae0 comp rx=0 tx=0).stop 2023-10-30T13:11:55.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.927+0000 7f43e9817640 1 -- 172.21.15.125:0/1087138286 shutdown_connections 2023-10-30T13:11:55.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.927+0000 7f43e9817640 1 --2- 172.21.15.125:0/1087138286 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f43c40664a0 0x7f43c4068960 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:55.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.927+0000 7f43e9817640 1 --2- 172.21.15.125:0/1087138286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43e40c6bd0 0x7f43e41ef280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:55.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.927+0000 7f43e9817640 1 --2- 172.21.15.125:0/1087138286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f43e4071110 0x7f43e4157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:55.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.927+0000 7f43e9817640 1 --2- 172.21.15.125:0/1087138286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f43e406f7a0 0x7f43e4154d90 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:55.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.927+0000 7f43e9817640 1 -- 172.21.15.125:0/1087138286 >> 172.21.15.125:0/1087138286 conn(0x7f43e40c7910 msgr2=0x7f43e40c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:55.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.927+0000 7f43e9817640 1 -- 172.21.15.125:0/1087138286 shutdown_connections 2023-10-30T13:11:55.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:55.927+0000 7f43e9817640 1 -- 172.21.15.125:0/1087138286 wait complete. 2023-10-30T13:11:55.951 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=2, replacing gid=7340 2023-10-30T13:11:55.951 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.d 2023-10-30T13:11:57.671 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T13:11:57.667+0000 7f510d857640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:11:58.951 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T13:11:58.952 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:11:58.952 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:11:58.955 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:11:58.955 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:11:59.032 INFO:tasks.ceph.mds.d.smithi136.stdout:starting mds.d at 2023-10-30T13:11:59.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.099+0000 7f431fcd6640 1 Processor -- start 2023-10-30T13:11:59.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.099+0000 7f431fcd6640 1 -- start start 2023-10-30T13:11:59.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.103+0000 7f431fcd6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4318071110 0x7f431806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:59.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.103+0000 7f431fcd6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4318066cf0 con 0x7f43180c6bd0 2023-10-30T13:11:59.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.103+0000 7f431fcd6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43180c70d0 con 0x7f431806f7a0 2023-10-30T13:11:59.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.103+0000 7f431fcd6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43180700c0 con 0x7f4318071110 2023-10-30T13:11:59.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.103+0000 7f431d271640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f43180c6bd0 0x7f4318158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35972/0 (socket says 172.21.15.125:35972) 2023-10-30T13:11:59.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.103+0000 7f431d271640 1 -- 172.21.15.125:0/4208497162 learned_addr learned my addr 172.21.15.125:0/4208497162 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:11:59.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.103+0000 7f431e273640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4318071110 0x7f431806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:59.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.103+0000 7f431e273640 1 -- 172.21.15.125:0/4208497162 >> v1:172.21.15.136:6789/0 conn(0x7f431806f7a0 legacy=0x7f4318069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:59.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.103+0000 7f431e273640 1 -- 172.21.15.125:0/4208497162 >> v1:172.21.15.125:6789/0 conn(0x7f43180c6bd0 legacy=0x7f4318158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:59.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.103+0000 7f431e273640 1 -- 172.21.15.125:0/4208497162 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43180c7390 con 0x7f4318071110 2023-10-30T13:11:59.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.103+0000 7f431e273640 1 --2- 172.21.15.125:0/4208497162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4318071110 0x7f431806fb80 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f4310009d90 tx=0x7f431001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4fa26d4923161e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:59.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.103+0000 7f431ca70640 1 -- 172.21.15.125:0/4208497162 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43100af600 con 0x7f4318071110 2023-10-30T13:11:59.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.103+0000 7f431ca70640 1 -- 172.21.15.125:0/4208497162 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43100ab050 con 0x7f4318071110 2023-10-30T13:11:59.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.103+0000 7f431ca70640 1 -- 172.21.15.125:0/4208497162 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43100af980 con 0x7f4318071110 2023-10-30T13:11:59.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 -- 172.21.15.125:0/4208497162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4318071110 msgr2=0x7f431806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:59.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 --2- 172.21.15.125:0/4208497162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4318071110 0x7f431806fb80 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f4310009d90 tx=0x7f431001f130 comp rx=0 tx=0).stop 2023-10-30T13:11:59.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 -- 172.21.15.125:0/4208497162 shutdown_connections 2023-10-30T13:11:59.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 --2- 172.21.15.125:0/4208497162 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4318071110 0x7f431806fb80 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:59.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 -- 172.21.15.125:0/4208497162 >> 172.21.15.125:0/4208497162 conn(0x7f43180c7910 msgr2=0x7f43180c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:59.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 -- 172.21.15.125:0/4208497162 shutdown_connections 2023-10-30T13:11:59.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 -- 172.21.15.125:0/4208497162 wait complete. 2023-10-30T13:11:59.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 Processor -- start 2023-10-30T13:11:59.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 -- start start 2023-10-30T13:11:59.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f431806f7a0 0x7f4318154010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:59.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4318071110 0x7f4318154a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:59.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43180c6bd0 0x7f43181579a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:59.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43180c08d0 con 0x7f4318071110 2023-10-30T13:11:59.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4318152840 con 0x7f43180c6bd0 2023-10-30T13:11:59.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431fcd6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4318152b20 con 0x7f431806f7a0 2023-10-30T13:11:59.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431d271640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4318071110 0x7f4318154a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:59.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431d271640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4318071110 0x7f4318154a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50964/0 (socket says 172.21.15.125:50964) 2023-10-30T13:11:59.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431d271640 1 -- 172.21.15.125:0/2206478587 learned_addr learned my addr 172.21.15.125:0/2206478587 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:11:59.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431e273640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43180c6bd0 0x7f43181579a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:59.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.107+0000 7f431da72640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f431806f7a0 0x7f4318154010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:59.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f431d271640 1 -- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f431806f7a0 msgr2=0x7f4318154010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:59.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f431d271640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f431806f7a0 0x7f4318154010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:59.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f431d271640 1 -- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43180c6bd0 msgr2=0x7f43181579a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:59.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f431d271640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43180c6bd0 0x7f43181579a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:59.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f431d271640 1 -- 172.21.15.125:0/2206478587 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4310009a70 con 0x7f4318071110 2023-10-30T13:11:59.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f431da72640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f431806f7a0 0x7f4318154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:11:59.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f431d271640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4318071110 0x7f4318154a00 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f430800c4d0 tx=0x7f430800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:59.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f430e7fc640 1 -- 172.21.15.125:0/2206478587 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308005cf0 con 0x7f4318071110 2023-10-30T13:11:59.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f430e7fc640 1 -- 172.21.15.125:0/2206478587 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4308005e50 con 0x7f4318071110 2023-10-30T13:11:59.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f430e7fc640 1 -- 172.21.15.125:0/2206478587 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4308007270 con 0x7f4318071110 2023-10-30T13:11:59.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f431fcd6640 1 -- 172.21.15.125:0/2206478587 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4318152e00 con 0x7f4318071110 2023-10-30T13:11:59.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f431fcd6640 1 -- 172.21.15.125:0/2206478587 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43181571a0 con 0x7f4318071110 2023-10-30T13:11:59.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f431fcd6640 1 -- 172.21.15.125:0/2206478587 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431804e170 con 0x7f4318071110 2023-10-30T13:11:59.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f430e7fc640 1 -- 172.21.15.125:0/2206478587 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f43080073d0 con 0x7f4318071110 2023-10-30T13:11:59.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f430e7fc640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42fc0664d0 0x7f42fc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:11:59.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f430e7fc640 1 -- 172.21.15.125:0/2206478587 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7f4308083bd0 con 0x7f4318071110 2023-10-30T13:11:59.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.111+0000 7f431da72640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42fc0664d0 0x7f42fc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:11:59.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.115+0000 7f431da72640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42fc0664d0 0x7f42fc068990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f4314000f70 tx=0x7f4314009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:11:59.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.115+0000 7f430e7fc640 1 -- 172.21.15.125:0/2206478587 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f43080507a0 con 0x7f4318071110 2023-10-30T13:11:59.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.303+0000 7f431fcd6640 1 -- 172.21.15.125:0/2206478587 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43180b0c30 con 0x7f4318071110 2023-10-30T13:11:59.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.303+0000 7f430e7fc640 1 -- 172.21.15.125:0/2206478587 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f4308017030 con 0x7f4318071110 2023-10-30T13:11:59.306 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:11:59.307 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:55.584904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:11:59.307 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 200 2023-10-30T13:11:59.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.303+0000 7f431fcd6640 1 -- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42fc0664d0 msgr2=0x7f42fc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:59.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.303+0000 7f431fcd6640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42fc0664d0 0x7f42fc068990 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f4314000f70 tx=0x7f4314009040 comp rx=0 tx=0).stop 2023-10-30T13:11:59.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.303+0000 7f431fcd6640 1 -- 172.21.15.125:0/2206478587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4318071110 msgr2=0x7f4318154a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:11:59.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.303+0000 7f431fcd6640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4318071110 0x7f4318154a00 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f430800c4d0 tx=0x7f430800c990 comp rx=0 tx=0).stop 2023-10-30T13:11:59.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.303+0000 7f431fcd6640 1 -- 172.21.15.125:0/2206478587 shutdown_connections 2023-10-30T13:11:59.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.303+0000 7f431fcd6640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42fc0664d0 0x7f42fc068990 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:59.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.303+0000 7f431fcd6640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f43180c6bd0 0x7f43181579a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:59.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.303+0000 7f431fcd6640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4318071110 0x7f4318154a00 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:59.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.303+0000 7f431fcd6640 1 --2- 172.21.15.125:0/2206478587 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f431806f7a0 0x7f4318154010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:11:59.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.303+0000 7f431fcd6640 1 -- 172.21.15.125:0/2206478587 >> 172.21.15.125:0/2206478587 conn(0x7f43180c7910 msgr2=0x7f43180c7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:11:59.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.303+0000 7f431fcd6640 1 -- 172.21.15.125:0/2206478587 shutdown_connections 2023-10-30T13:11:59.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:11:59.303+0000 7f431fcd6640 1 -- 172.21.15.125:0/2206478587 wait complete. 2023-10-30T13:11:59.327 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T13:12:01.328 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:01.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.471+0000 7f2c31df8640 1 Processor -- start 2023-10-30T13:12:01.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.471+0000 7f2c31df8640 1 -- start start 2023-10-30T13:12:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c31df8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2c2c15d6f0 0x7f2c2c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c31df8640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2c0cb800 con 0x7f2c2c1538a0 2023-10-30T13:12:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c31df8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c2c0cb330 con 0x7f2c2c0cf160 2023-10-30T13:12:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c31df8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c2c0cad70 con 0x7f2c2c15d6f0 2023-10-30T13:12:01.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2b7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2c2c1538a0 0x7f2c2c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35970/0 (socket says 172.21.15.125:35970) 2023-10-30T13:12:01.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2b7fe640 1 -- 172.21.15.125:0/113426274 learned_addr learned my addr 172.21.15.125:0/113426274 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:12:01.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2affd640 1 --2- 172.21.15.125:0/113426274 >> v2:172.21.15.136:3300/0 conn(0x7f2c2c15d6f0 0x7f2c2c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:01.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 -- 172.21.15.125:0/113426274 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988933429 0 0) 0x7f2c2c0cb800 con 0x7f2c2c1538a0 2023-10-30T13:12:01.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 -- 172.21.15.125:0/113426274 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c14003530 con 0x7f2c2c1538a0 2023-10-30T13:12:01.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 -- 172.21.15.125:0/113426274 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1347793622 0 0) 0x7f2c14003530 con 0x7f2c2c1538a0 2023-10-30T13:12:01.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 -- 172.21.15.125:0/113426274 >> v2:172.21.15.136:3300/0 conn(0x7f2c2c15d6f0 msgr2=0x7f2c2c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:01.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 --2- 172.21.15.125:0/113426274 >> v2:172.21.15.136:3300/0 conn(0x7f2c2c15d6f0 0x7f2c2c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:01.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 -- 172.21.15.125:0/113426274 >> v1:172.21.15.136:6789/0 conn(0x7f2c2c0cf160 legacy=0x7f2c2c164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:01.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 -- 172.21.15.125:0/113426274 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2c0cb050 con 0x7f2c2c1538a0 2023-10-30T13:12:01.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 -- 172.21.15.125:0/113426274 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f2c1c001680 con 0x7f2c2c1538a0 2023-10-30T13:12:01.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2affd640 1 --2- 172.21.15.125:0/113426274 >> v2:172.21.15.136:3300/0 conn(0x7f2c2c15d6f0 0x7f2c2c15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:12:01.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 -- 172.21.15.125:0/113426274 >> v1:172.21.15.125:6789/0 conn(0x7f2c2c1538a0 legacy=0x7f2c2c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:01.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 --2- 172.21.15.125:0/113426274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c14003980 0x7f2c14003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:01.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 --2- 172.21.15.125:0/113426274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c140047d0 0x7f2c14004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:01.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 --2- 172.21.15.125:0/113426274 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c14005190 0x7f2c1400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:01.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 -- 172.21.15.125:0/113426274 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c0cad70 con 0x7f2c14003980 2023-10-30T13:12:01.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 -- 172.21.15.125:0/113426274 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1400c040 con 0x7f2c140047d0 2023-10-30T13:12:01.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2a7fc640 1 -- 172.21.15.125:0/113426274 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c1400c200 con 0x7f2c14005190 2023-10-30T13:12:01.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2b7fe640 1 --2- 172.21.15.125:0/113426274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c14003980 0x7f2c14003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:01.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2affd640 1 --2- 172.21.15.125:0/113426274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c140047d0 0x7f2c14004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:01.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2bfff640 1 --2- 172.21.15.125:0/113426274 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c14005190 0x7f2c1400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:01.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2b7fe640 1 -- 172.21.15.125:0/113426274 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c14005190 msgr2=0x7f2c1400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:01.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2b7fe640 1 --2- 172.21.15.125:0/113426274 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c14005190 0x7f2c1400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:01.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2b7fe640 1 -- 172.21.15.125:0/113426274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c140047d0 msgr2=0x7f2c14004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:01.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2b7fe640 1 --2- 172.21.15.125:0/113426274 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c140047d0 0x7f2c14004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:01.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2b7fe640 1 -- 172.21.15.125:0/113426274 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c14003530 con 0x7f2c14003980 2023-10-30T13:12:01.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.475+0000 7f2c2bfff640 1 -- 172.21.15.125:0/113426274 reap_dead start 2023-10-30T13:12:01.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.479+0000 7f2c2b7fe640 1 --2- 172.21.15.125:0/113426274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c14003980 0x7f2c14003da0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00c240 tx=0x7f2c1c01f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f4f2d1991272a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:01.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.479+0000 7f2c2a7fc640 1 -- 172.21.15.125:0/113426274 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c005ae0 con 0x7f2c14003980 2023-10-30T13:12:01.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.479+0000 7f2c2a7fc640 1 -- 172.21.15.125:0/113426274 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c1c0ab070 con 0x7f2c14003980 2023-10-30T13:12:01.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.479+0000 7f2c2a7fc640 1 -- 172.21.15.125:0/113426274 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c1c007070 con 0x7f2c14003980 2023-10-30T13:12:01.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.479+0000 7f2c31df8640 1 -- 172.21.15.125:0/113426274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c14003980 msgr2=0x7f2c14003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:01.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.479+0000 7f2c31df8640 1 --2- 172.21.15.125:0/113426274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c14003980 0x7f2c14003da0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c00c240 tx=0x7f2c1c01f3b0 comp rx=0 tx=0).stop 2023-10-30T13:12:01.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.479+0000 7f2c31df8640 1 -- 172.21.15.125:0/113426274 shutdown_connections 2023-10-30T13:12:01.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.479+0000 7f2c31df8640 1 --2- 172.21.15.125:0/113426274 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c14003980 0x7f2c14003da0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:01.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.479+0000 7f2c31df8640 1 -- 172.21.15.125:0/113426274 >> 172.21.15.125:0/113426274 conn(0x7f2c2c06bfe0 msgr2=0x7f2c2c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:01.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.479+0000 7f2c31df8640 1 -- 172.21.15.125:0/113426274 shutdown_connections 2023-10-30T13:12:01.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.479+0000 7f2c31df8640 1 -- 172.21.15.125:0/113426274 wait complete. 2023-10-30T13:12:01.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.479+0000 7f2c31df8640 1 Processor -- start 2023-10-30T13:12:01.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c31df8640 1 -- start start 2023-10-30T13:12:01.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c31df8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c14003980 0x7f2c2c0b4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:01.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c31df8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c2c1e3e60 0x7f2c2c1ef7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:01.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c31df8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c2c1e2aa0 0x7f2c2c1efd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:01.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c31df8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c1e5790 con 0x7f2c2c1e3e60 2023-10-30T13:12:01.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c31df8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c1f0250 con 0x7f2c2c1e2aa0 2023-10-30T13:12:01.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c31df8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c2c159400 con 0x7f2c14003980 2023-10-30T13:12:01.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c2affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c2c1e3e60 0x7f2c2c1ef7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:01.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c2affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c2c1e3e60 0x7f2c2c1ef7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45824/0 (socket says 172.21.15.125:45824) 2023-10-30T13:12:01.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c2affd640 1 -- 172.21.15.125:0/1621562542 learned_addr learned my addr 172.21.15.125:0/1621562542 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:01.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c2b7fe640 1 --2- 172.21.15.125:0/1621562542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c14003980 0x7f2c2c0b4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:01.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c2bfff640 1 --2- 172.21.15.125:0/1621562542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c2c1e2aa0 0x7f2c2c1efd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:01.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c2affd640 1 -- 172.21.15.125:0/1621562542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c14003980 msgr2=0x7f2c2c0b4380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:01.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c2affd640 1 --2- 172.21.15.125:0/1621562542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c14003980 0x7f2c2c0b4380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:01.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c2affd640 1 -- 172.21.15.125:0/1621562542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c2c1e2aa0 msgr2=0x7f2c2c1efd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:01.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c2affd640 1 --2- 172.21.15.125:0/1621562542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c2c1e2aa0 0x7f2c2c1efd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:01.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c2affd640 1 -- 172.21.15.125:0/1621562542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c2c1e3050 con 0x7f2c2c1e3e60 2023-10-30T13:12:01.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c2affd640 1 --2- 172.21.15.125:0/1621562542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c2c1e3e60 0x7f2c2c1ef7d0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f2c20011830 tx=0x7f2c20011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:01.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c13fff640 1 -- 172.21.15.125:0/1621562542 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2000f9c0 con 0x7f2c2c1e3e60 2023-10-30T13:12:01.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c13fff640 1 -- 172.21.15.125:0/1621562542 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c2000fb20 con 0x7f2c2c1e3e60 2023-10-30T13:12:01.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c31df8640 1 -- 172.21.15.125:0/1621562542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c2c158af0 con 0x7f2c2c1e3e60 2023-10-30T13:12:01.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c13fff640 1 -- 172.21.15.125:0/1621562542 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c2000fe90 con 0x7f2c2c1e3e60 2023-10-30T13:12:01.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c31df8640 1 -- 172.21.15.125:0/1621562542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c2c159090 con 0x7f2c2c1e3e60 2023-10-30T13:12:01.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.483+0000 7f2c31df8640 1 -- 172.21.15.125:0/1621562542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c2c04e170 con 0x7f2c2c1e3e60 2023-10-30T13:12:01.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.487+0000 7f2c13fff640 1 -- 172.21.15.125:0/1621562542 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c20015670 con 0x7f2c2c1e3e60 2023-10-30T13:12:01.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.487+0000 7f2c13fff640 1 --2- 172.21.15.125:0/1621562542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c180664d0 0x7f2c18068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:01.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.487+0000 7f2c13fff640 1 -- 172.21.15.125:0/1621562542 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7f2c200852e0 con 0x7f2c2c1e3e60 2023-10-30T13:12:01.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.487+0000 7f2c2b7fe640 1 --2- 172.21.15.125:0/1621562542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c180664d0 0x7f2c18068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:01.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.487+0000 7f2c2b7fe640 1 --2- 172.21.15.125:0/1621562542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c180664d0 0x7f2c18068990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0af000 tx=0x7f2c1c0aa000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:01.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.491+0000 7f2c13fff640 1 -- 172.21.15.125:0/1621562542 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c20051e00 con 0x7f2c2c1e3e60 2023-10-30T13:12:01.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.683+0000 7f2c31df8640 1 -- 172.21.15.125:0/1621562542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c2c159540 con 0x7f2c2c1e3e60 2023-10-30T13:12:01.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.683+0000 7f2c13fff640 1 -- 172.21.15.125:0/1621562542 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+6550 (secure 0 0 0) 0x7f2c20006c40 con 0x7f2c2c1e3e60 2023-10-30T13:12:01.686 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:12:01.687 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":201,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:11:55.584904+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:01.687 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 201 2023-10-30T13:12:01.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.683+0000 7f2c31df8640 1 -- 172.21.15.125:0/1621562542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c180664d0 msgr2=0x7f2c18068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:01.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.683+0000 7f2c31df8640 1 --2- 172.21.15.125:0/1621562542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c180664d0 0x7f2c18068990 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f2c1c0af000 tx=0x7f2c1c0aa000 comp rx=0 tx=0).stop 2023-10-30T13:12:01.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.683+0000 7f2c31df8640 1 -- 172.21.15.125:0/1621562542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c2c1e3e60 msgr2=0x7f2c2c1ef7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:01.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.683+0000 7f2c31df8640 1 --2- 172.21.15.125:0/1621562542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c2c1e3e60 0x7f2c2c1ef7d0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f2c20011830 tx=0x7f2c20011cf0 comp rx=0 tx=0).stop 2023-10-30T13:12:01.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.683+0000 7f2c31df8640 1 -- 172.21.15.125:0/1621562542 shutdown_connections 2023-10-30T13:12:01.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.683+0000 7f2c31df8640 1 --2- 172.21.15.125:0/1621562542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c180664d0 0x7f2c18068990 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:01.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.683+0000 7f2c31df8640 1 --2- 172.21.15.125:0/1621562542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c2c1e2aa0 0x7f2c2c1efd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:01.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.683+0000 7f2c31df8640 1 --2- 172.21.15.125:0/1621562542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c2c1e3e60 0x7f2c2c1ef7d0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:01.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.683+0000 7f2c31df8640 1 --2- 172.21.15.125:0/1621562542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c14003980 0x7f2c2c0b4380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:01.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.683+0000 7f2c31df8640 1 -- 172.21.15.125:0/1621562542 >> 172.21.15.125:0/1621562542 conn(0x7f2c2c06bfe0 msgr2=0x7f2c2c0c48f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:01.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.683+0000 7f2c31df8640 1 -- 172.21.15.125:0/1621562542 shutdown_connections 2023-10-30T13:12:01.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:01.683+0000 7f2c31df8640 1 -- 172.21.15.125:0/1621562542 wait complete. 2023-10-30T13:12:01.708 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T13:12:01.708 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 32 secs before thrashing 2023-10-30T13:12:03.022 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:12:03.023 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.11 2023-10-30T13:12:03.023 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 11 -eq 48 ] 2023-10-30T13:12:03.024 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.11 2023-10-30T13:12:03.024 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.11 2023-10-30T13:12:03.024 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.11 2023-10-30T13:12:03.024 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:12:12.951 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:12:12.952 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.12 2023-10-30T13:12:12.952 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 12 -eq 48 ] 2023-10-30T13:12:12.952 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.12 2023-10-30T13:12:12.953 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.12 2023-10-30T13:12:12.953 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.12 2023-10-30T13:12:12.953 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:12:16.548 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:12:16.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:12:24.222 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:12:24.223 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.13 2023-10-30T13:12:24.223 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 13 -eq 48 ] 2023-10-30T13:12:24.223 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.13 2023-10-30T13:12:24.224 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.13 2023-10-30T13:12:24.224 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.13 2023-10-30T13:12:24.224 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:12:29.552 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:12:29.552 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 14 -eq 48 ] 2023-10-30T13:12:29.552 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.14 2023-10-30T13:12:29.553 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.14 2023-10-30T13:12:29.553 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.14 2023-10-30T13:12:29.554 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.14 2023-10-30T13:12:29.554 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:12:33.709 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:33.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.854+0000 7f4a41e09640 1 Processor -- start 2023-10-30T13:12:33.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.854+0000 7f4a41e09640 1 -- start start 2023-10-30T13:12:33.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.854+0000 7f4a41e09640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4a3c0c6bd0 0x7f4a3c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:33.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.854+0000 7f4a41e09640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a3c066cf0 con 0x7f4a3c06f7a0 2023-10-30T13:12:33.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.854+0000 7f4a41e09640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a3c0c70d0 con 0x7f4a3c071110 2023-10-30T13:12:33.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.854+0000 7f4a41e09640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a3c0700c0 con 0x7f4a3c0c6bd0 2023-10-30T13:12:33.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.854+0000 7f4a32ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4a3c0c6bd0 0x7f4a3c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:33.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.854+0000 7f4a32ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4a3c0c6bd0 0x7f4a3c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53680/0 (socket says 172.21.15.125:53680) 2023-10-30T13:12:33.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.854+0000 7f4a32ffd640 1 -- 172.21.15.125:0/3418741905 learned_addr learned my addr 172.21.15.125:0/3418741905 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:33.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.854+0000 7f4a32ffd640 1 -- 172.21.15.125:0/3418741905 >> v1:172.21.15.136:6789/0 conn(0x7f4a3c071110 legacy=0x7f4a3c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:33.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.854+0000 7f4a32ffd640 1 -- 172.21.15.125:0/3418741905 >> v1:172.21.15.125:6789/0 conn(0x7f4a3c06f7a0 legacy=0x7f4a3c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:33.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.854+0000 7f4a32ffd640 1 -- 172.21.15.125:0/3418741905 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c0c7390 con 0x7f4a3c0c6bd0 2023-10-30T13:12:33.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.858+0000 7f4a32ffd640 1 --2- 172.21.15.125:0/3418741905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a3c0c6bd0 0x7f4a3c06fb80 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f4a24009870 tx=0x7f4a2401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e5a0ccbfa9cbcf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:33.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.858+0000 7f4a3affd640 1 -- 172.21.15.125:0/3418741905 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a240a8a30 con 0x7f4a3c0c6bd0 2023-10-30T13:12:33.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.858+0000 7f4a3affd640 1 -- 172.21.15.125:0/3418741905 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a240a8b90 con 0x7f4a3c0c6bd0 2023-10-30T13:12:33.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.858+0000 7f4a3affd640 1 -- 172.21.15.125:0/3418741905 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a240b0440 con 0x7f4a3c0c6bd0 2023-10-30T13:12:33.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.858+0000 7f4a41e09640 1 -- 172.21.15.125:0/3418741905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a3c0c6bd0 msgr2=0x7f4a3c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:33.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.858+0000 7f4a41e09640 1 --2- 172.21.15.125:0/3418741905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a3c0c6bd0 0x7f4a3c06fb80 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f4a24009870 tx=0x7f4a2401ed30 comp rx=0 tx=0).stop 2023-10-30T13:12:33.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.858+0000 7f4a41e09640 1 -- 172.21.15.125:0/3418741905 shutdown_connections 2023-10-30T13:12:33.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.858+0000 7f4a41e09640 1 --2- 172.21.15.125:0/3418741905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a3c0c6bd0 0x7f4a3c06fb80 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:33.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.858+0000 7f4a41e09640 1 -- 172.21.15.125:0/3418741905 >> 172.21.15.125:0/3418741905 conn(0x7f4a3c0c7910 msgr2=0x7f4a3c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:33.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.858+0000 7f4a41e09640 1 -- 172.21.15.125:0/3418741905 shutdown_connections 2023-10-30T13:12:33.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.858+0000 7f4a41e09640 1 -- 172.21.15.125:0/3418741905 wait complete. 2023-10-30T13:12:33.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a41e09640 1 Processor -- start 2023-10-30T13:12:33.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a41e09640 1 -- start start 2023-10-30T13:12:33.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a41e09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a3c06f7a0 0x7f4a3c1576e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a41e09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a3c071110 0x7f4a3c1e3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a41e09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a3c0c6bd0 0x7f4a3c1e43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a41e09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c1544d0 con 0x7f4a3c06f7a0 2023-10-30T13:12:33.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a41e09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c0c0070 con 0x7f4a3c071110 2023-10-30T13:12:33.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a41e09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c1f3e50 con 0x7f4a3c0c6bd0 2023-10-30T13:12:33.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a3b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a3c06f7a0 0x7f4a3c1576e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:33.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a3b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a3c06f7a0 0x7f4a3c1576e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53684/0 (socket says 172.21.15.125:53684) 2023-10-30T13:12:33.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a3b7fe640 1 -- 172.21.15.125:0/1158570702 learned_addr learned my addr 172.21.15.125:0/1158570702 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:33.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a3bfff640 1 --2- 172.21.15.125:0/1158570702 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a3c0c6bd0 0x7f4a3c1e43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:33.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a32ffd640 1 --2- 172.21.15.125:0/1158570702 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a3c071110 0x7f4a3c1e3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:33.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a3b7fe640 1 -- 172.21.15.125:0/1158570702 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a3c0c6bd0 msgr2=0x7f4a3c1e43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:33.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a3b7fe640 1 --2- 172.21.15.125:0/1158570702 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a3c0c6bd0 0x7f4a3c1e43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:33.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a3b7fe640 1 -- 172.21.15.125:0/1158570702 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a3c071110 msgr2=0x7f4a3c1e3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:33.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a3b7fe640 1 --2- 172.21.15.125:0/1158570702 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a3c071110 0x7f4a3c1e3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:33.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a3b7fe640 1 -- 172.21.15.125:0/1158570702 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a240094d0 con 0x7f4a3c06f7a0 2023-10-30T13:12:33.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a3b7fe640 1 --2- 172.21.15.125:0/1158570702 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a3c06f7a0 0x7f4a3c1576e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00c7d0 tx=0x7f4a2c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:33.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a38ff9640 1 -- 172.21.15.125:0/1158570702 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c003b80 con 0x7f4a3c06f7a0 2023-10-30T13:12:33.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a38ff9640 1 -- 172.21.15.125:0/1158570702 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c003ce0 con 0x7f4a3c06f7a0 2023-10-30T13:12:33.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a38ff9640 1 -- 172.21.15.125:0/1158570702 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c00aa30 con 0x7f4a3c06f7a0 2023-10-30T13:12:33.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a41e09640 1 -- 172.21.15.125:0/1158570702 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3c1f7260 con 0x7f4a3c06f7a0 2023-10-30T13:12:33.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a41e09640 1 -- 172.21.15.125:0/1158570702 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3c1f3520 con 0x7f4a3c06f7a0 2023-10-30T13:12:33.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.862+0000 7f4a41e09640 1 -- 172.21.15.125:0/1158570702 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3c04e170 con 0x7f4a3c06f7a0 2023-10-30T13:12:33.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.866+0000 7f4a38ff9640 1 -- 172.21.15.125:0/1158570702 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4a2c00ab90 con 0x7f4a3c06f7a0 2023-10-30T13:12:33.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.866+0000 7f4a38ff9640 1 --2- 172.21.15.125:0/1158570702 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a1c0664a0 0x7f4a1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:33.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.866+0000 7f4a38ff9640 1 -- 172.21.15.125:0/1158570702 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7f4a2c084520 con 0x7f4a3c06f7a0 2023-10-30T13:12:33.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.866+0000 7f4a32ffd640 1 --2- 172.21.15.125:0/1158570702 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a1c0664a0 0x7f4a1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:33.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.866+0000 7f4a32ffd640 1 --2- 172.21.15.125:0/1158570702 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a1c0664a0 0x7f4a1c068960 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4a240b2000 tx=0x7f4a240aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:33.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:33.870+0000 7f4a38ff9640 1 -- 172.21.15.125:0/1158570702 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4a2c051160 con 0x7f4a3c06f7a0 2023-10-30T13:12:34.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.058+0000 7f4a41e09640 1 -- 172.21.15.125:0/1158570702 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a3c0c39d0 con 0x7f4a3c06f7a0 2023-10-30T13:12:34.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.058+0000 7f4a38ff9640 1 -- 172.21.15.125:0/1158570702 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+6552 (secure 0 0 0) 0x7f4a2c054780 con 0x7f4a3c06f7a0 2023-10-30T13:12:34.063 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:12:34.063 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:12:12.664330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:34.064 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 204 2023-10-30T13:12:34.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.058+0000 7f4a41e09640 1 -- 172.21.15.125:0/1158570702 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a1c0664a0 msgr2=0x7f4a1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:34.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.058+0000 7f4a41e09640 1 --2- 172.21.15.125:0/1158570702 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a1c0664a0 0x7f4a1c068960 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4a240b2000 tx=0x7f4a240aa070 comp rx=0 tx=0).stop 2023-10-30T13:12:34.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.058+0000 7f4a41e09640 1 -- 172.21.15.125:0/1158570702 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a3c06f7a0 msgr2=0x7f4a3c1576e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:34.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.058+0000 7f4a41e09640 1 --2- 172.21.15.125:0/1158570702 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a3c06f7a0 0x7f4a3c1576e0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00c7d0 tx=0x7f4a2c00cc90 comp rx=0 tx=0).stop 2023-10-30T13:12:34.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.058+0000 7f4a41e09640 1 -- 172.21.15.125:0/1158570702 shutdown_connections 2023-10-30T13:12:34.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.058+0000 7f4a41e09640 1 --2- 172.21.15.125:0/1158570702 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a1c0664a0 0x7f4a1c068960 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:34.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.058+0000 7f4a41e09640 1 --2- 172.21.15.125:0/1158570702 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a3c0c6bd0 0x7f4a3c1e43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:34.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.058+0000 7f4a41e09640 1 --2- 172.21.15.125:0/1158570702 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a3c071110 0x7f4a3c1e3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:34.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.058+0000 7f4a41e09640 1 --2- 172.21.15.125:0/1158570702 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a3c06f7a0 0x7f4a3c1576e0 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:34.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.058+0000 7f4a41e09640 1 -- 172.21.15.125:0/1158570702 >> 172.21.15.125:0/1158570702 conn(0x7f4a3c0c7910 msgr2=0x7f4a3c0c3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:34.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.062+0000 7f4a41e09640 1 -- 172.21.15.125:0/1158570702 shutdown_connections 2023-10-30T13:12:34.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.062+0000 7f4a41e09640 1 -- 172.21.15.125:0/1158570702 wait complete. 2023-10-30T13:12:34.084 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.c (rank=1) 2023-10-30T13:12:34.084 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:12:34.085 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:12:34.112 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:12:34.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:34.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b1fa0640 1 Processor -- start 2023-10-30T13:12:34.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b1fa0640 1 -- start start 2023-10-30T13:12:34.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b1fa0640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5ac1538f0 0x7fb5ac159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:34.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b1fa0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb5ac0cefa0 0x7fb5ac0cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:34.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b1fa0640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5ac0ca6a0 con 0x7fb5ac15cd50 2023-10-30T13:12:34.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b1fa0640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5ac15d300 con 0x7fb5ac0cefa0 2023-10-30T13:12:34.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b1fa0640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac15d5e0 con 0x7fb5ac1538f0 2023-10-30T13:12:34.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b179f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb5ac0cefa0 0x7fb5ac0cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:34.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b179f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb5ac0cefa0 0x7fb5ac0cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53698/0 (socket says 172.21.15.125:53698) 2023-10-30T13:12:34.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b179f640 1 -- 172.21.15.125:0/3165473558 learned_addr learned my addr 172.21.15.125:0/3165473558 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:34.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b0f9e640 1 --2- 172.21.15.125:0/3165473558 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5ac1538f0 0x7fb5ac159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b179f640 1 -- 172.21.15.125:0/3165473558 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5ac1538f0 msgr2=0x7fb5ac159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b179f640 1 --2- 172.21.15.125:0/3165473558 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5ac1538f0 0x7fb5ac159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b179f640 1 -- 172.21.15.125:0/3165473558 >> v1:172.21.15.125:6789/0 conn(0x7fb5ac15cd50 legacy=0x7fb5ac0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b179f640 1 -- 172.21.15.125:0/3165473558 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ac15d8c0 con 0x7fb5ac0cefa0 2023-10-30T13:12:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.290+0000 7fb5b0f9e640 1 --2- 172.21.15.125:0/3165473558 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5ac1538f0 0x7fb5ac159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:34.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.294+0000 7fb5b179f640 1 --2- 172.21.15.125:0/3165473558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5ac0cefa0 0x7fb5ac0cf3f0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009e50 tx=0x7fb5a401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be638493759e9d9e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:34.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.294+0000 7fb5a37fe640 1 -- 172.21.15.125:0/3165473558 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a40af400 con 0x7fb5ac0cefa0 2023-10-30T13:12:34.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.294+0000 7fb5a37fe640 1 -- 172.21.15.125:0/3165473558 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a40ab050 con 0x7fb5ac0cefa0 2023-10-30T13:12:34.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.294+0000 7fb5a37fe640 1 -- 172.21.15.125:0/3165473558 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a40af780 con 0x7fb5ac0cefa0 2023-10-30T13:12:34.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.294+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/3165473558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5ac0cefa0 msgr2=0x7fb5ac0cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:34.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.294+0000 7fb5b1fa0640 1 --2- 172.21.15.125:0/3165473558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5ac0cefa0 0x7fb5ac0cf3f0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009e50 tx=0x7fb5a401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:12:34.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.294+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/3165473558 shutdown_connections 2023-10-30T13:12:34.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.294+0000 7fb5b1fa0640 1 --2- 172.21.15.125:0/3165473558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5ac0cefa0 0x7fb5ac0cf3f0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:34.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.294+0000 7fb5b1fa0640 1 --2- 172.21.15.125:0/3165473558 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5ac1538f0 0x7fb5ac159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:34.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.294+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/3165473558 >> 172.21.15.125:0/3165473558 conn(0x7fb5ac06a640 msgr2=0x7fb5ac065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:34.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.294+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/3165473558 shutdown_connections 2023-10-30T13:12:34.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.294+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/3165473558 wait complete. 2023-10-30T13:12:34.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b1fa0640 1 Processor -- start 2023-10-30T13:12:34.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b1fa0640 1 -- start start 2023-10-30T13:12:34.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b1fa0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5ac0cefa0 0x7fb5ac158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:34.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b1fa0640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5ac1538f0 0x7fb5ac142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:34.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b1fa0640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5ac15cd50 0x7fb5ac147f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:34.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b1fa0640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac15bcf0 con 0x7fb5ac15cd50 2023-10-30T13:12:34.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b1fa0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac1efb30 con 0x7fb5ac0cefa0 2023-10-30T13:12:34.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b1fa0640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ac152210 con 0x7fb5ac1538f0 2023-10-30T13:12:34.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b179f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5ac15cd50 0x7fb5ac147f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:34.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b179f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5ac15cd50 0x7fb5ac147f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53710/0 (socket says 172.21.15.125:53710) 2023-10-30T13:12:34.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b179f640 1 -- 172.21.15.125:0/389204695 learned_addr learned my addr 172.21.15.125:0/389204695 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:34.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5a3fff640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5ac1538f0 0x7fb5ac142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:34.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b0f9e640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5ac0cefa0 0x7fb5ac158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:34.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b179f640 1 -- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5ac1538f0 msgr2=0x7fb5ac142390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:34.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b179f640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5ac1538f0 0x7fb5ac142390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:34.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b179f640 1 -- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5ac0cefa0 msgr2=0x7fb5ac158300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:34.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b179f640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5ac0cefa0 0x7fb5ac158300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:34.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b179f640 1 -- 172.21.15.125:0/389204695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a4009b30 con 0x7fb5ac15cd50 2023-10-30T13:12:34.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5a3fff640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5ac1538f0 0x7fb5ac142390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:34.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b179f640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5ac15cd50 0x7fb5ac147f10 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fb5a401ce20 tx=0x7fb5a401bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:34.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5a17fa640 1 -- 172.21.15.125:0/389204695 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a40afde0 con 0x7fb5ac15cd50 2023-10-30T13:12:34.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5a17fa640 1 -- 172.21.15.125:0/389204695 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a4009510 con 0x7fb5ac15cd50 2023-10-30T13:12:34.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5a17fa640 1 -- 172.21.15.125:0/389204695 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a40ba660 con 0x7fb5ac15cd50 2023-10-30T13:12:34.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/389204695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ac1524f0 con 0x7fb5ac15cd50 2023-10-30T13:12:34.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.298+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/389204695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ac1ef7c0 con 0x7fb5ac15cd50 2023-10-30T13:12:34.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.302+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/389204695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ac1efca0 con 0x7fb5ac15cd50 2023-10-30T13:12:34.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.302+0000 7fb5a17fa640 1 -- 172.21.15.125:0/389204695 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb5a40b6040 con 0x7fb5ac15cd50 2023-10-30T13:12:34.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.302+0000 7fb5a17fa640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5980664d0 0x7fb598068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:34.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.302+0000 7fb5a17fa640 1 -- 172.21.15.125:0/389204695 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7fb5a41228b0 con 0x7fb5ac15cd50 2023-10-30T13:12:34.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.302+0000 7fb5b0f9e640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5980664d0 0x7fb598068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:34.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.302+0000 7fb5b0f9e640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5980664d0 0x7fb598068990 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800c900 tx=0x7fb5a8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:34.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.306+0000 7fb5a17fa640 1 -- 172.21.15.125:0/389204695 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb5a40ef5b0 con 0x7fb5ac15cd50 2023-10-30T13:12:34.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.486+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/389204695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5ac04e170 con 0x7fb5ac15cd50 2023-10-30T13:12:34.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.486+0000 7fb5a17fa640 1 -- 172.21.15.125:0/389204695 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+6552 (secure 0 0 0) 0x7fb5a40f2bd0 con 0x7fb5ac15cd50 2023-10-30T13:12:34.491 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:12:34.491 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:12:12.664330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:34.491 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 204 2023-10-30T13:12:34.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.486+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5980664d0 msgr2=0x7fb598068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:34.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.486+0000 7fb5b1fa0640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5980664d0 0x7fb598068990 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb5a800c900 tx=0x7fb5a8008040 comp rx=0 tx=0).stop 2023-10-30T13:12:34.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.486+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/389204695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5ac15cd50 msgr2=0x7fb5ac147f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:34.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.486+0000 7fb5b1fa0640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5ac15cd50 0x7fb5ac147f10 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7fb5a401ce20 tx=0x7fb5a401bf90 comp rx=0 tx=0).stop 2023-10-30T13:12:34.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.490+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/389204695 shutdown_connections 2023-10-30T13:12:34.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.490+0000 7fb5b1fa0640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb5980664d0 0x7fb598068990 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:34.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.490+0000 7fb5b1fa0640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb5ac15cd50 0x7fb5ac147f10 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:34.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.490+0000 7fb5b1fa0640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb5ac1538f0 0x7fb5ac142390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:34.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.490+0000 7fb5b1fa0640 1 --2- 172.21.15.125:0/389204695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb5ac0cefa0 0x7fb5ac158300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:34.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.490+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/389204695 >> 172.21.15.125:0/389204695 conn(0x7fb5ac06a640 msgr2=0x7fb5ac0cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:34.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.490+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/389204695 shutdown_connections 2023-10-30T13:12:34.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:34.490+0000 7fb5b1fa0640 1 -- 172.21.15.125:0/389204695 wait complete. 2023-10-30T13:12:34.512 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:12:36.513 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:36.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.654+0000 7fd2b4e91640 1 Processor -- start 2023-10-30T13:12:36.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.654+0000 7fd2b4e91640 1 -- start start 2023-10-30T13:12:36.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2b4e91640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd2b006f7a0 0x7fd2b006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:36.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2b4e91640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2b0071110 0x7fd2b0158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:36.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2b4e91640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b0066cf0 con 0x7fd2b00c6bd0 2023-10-30T13:12:36.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2b4e91640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b00700c0 con 0x7fd2b006f7a0 2023-10-30T13:12:36.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2b4e91640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b00c70d0 con 0x7fd2b0071110 2023-10-30T13:12:36.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2add9b640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd2b00c6bd0 0x7fd2b0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44100/0 (socket says 172.21.15.125:44100) 2023-10-30T13:12:36.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2add9b640 1 -- 172.21.15.125:0/256183510 learned_addr learned my addr 172.21.15.125:0/256183510 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:12:36.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ae59c640 1 --2- 172.21.15.125:0/256183510 >> v2:172.21.15.136:3300/0 conn(0x7fd2b006f7a0 0x7fd2b006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:36.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2aed9d640 1 --2- 172.21.15.125:0/256183510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2b0071110 0x7fd2b0158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:36.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 -- 172.21.15.125:0/256183510 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744485210 0 0) 0x7fd2b0066cf0 con 0x7fd2b00c6bd0 2023-10-30T13:12:36.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 -- 172.21.15.125:0/256183510 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd298003530 con 0x7fd2b00c6bd0 2023-10-30T13:12:36.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 -- 172.21.15.125:0/256183510 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1947055024 0 0) 0x7fd298003530 con 0x7fd2b00c6bd0 2023-10-30T13:12:36.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 -- 172.21.15.125:0/256183510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2b0071110 msgr2=0x7fd2b0158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:36.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 --2- 172.21.15.125:0/256183510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2b0071110 0x7fd2b0158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:36.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 -- 172.21.15.125:0/256183510 >> v2:172.21.15.136:3300/0 conn(0x7fd2b006f7a0 msgr2=0x7fd2b006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:36.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 --2- 172.21.15.125:0/256183510 >> v2:172.21.15.136:3300/0 conn(0x7fd2b006f7a0 0x7fd2b006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:36.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 -- 172.21.15.125:0/256183510 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b00c7390 con 0x7fd2b00c6bd0 2023-10-30T13:12:36.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 -- 172.21.15.125:0/256183510 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fd2a4001680 con 0x7fd2b00c6bd0 2023-10-30T13:12:36.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 -- 172.21.15.125:0/256183510 >> v1:172.21.15.125:6789/0 conn(0x7fd2b00c6bd0 legacy=0x7fd2b0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:36.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 --2- 172.21.15.125:0/256183510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2b0071110 0x7fd298003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:36.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 --2- 172.21.15.125:0/256183510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd298003f00 0x7fd2980047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:36.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 --2- 172.21.15.125:0/256183510 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd298004d10 0x7fd298005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:36.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 -- 172.21.15.125:0/256183510 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b00700c0 con 0x7fd298003f00 2023-10-30T13:12:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 -- 172.21.15.125:0/256183510 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b00c70d0 con 0x7fd298004d10 2023-10-30T13:12:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ad59a640 1 -- 172.21.15.125:0/256183510 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29800b9c0 con 0x7fd2b0071110 2023-10-30T13:12:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2add9b640 1 --2- 172.21.15.125:0/256183510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd298003f00 0x7fd2980047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ae59c640 1 --2- 172.21.15.125:0/256183510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2b0071110 0x7fd298003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:36.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2aed9d640 1 --2- 172.21.15.125:0/256183510 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd298004d10 0x7fd298005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:36.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2add9b640 1 -- 172.21.15.125:0/256183510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2b0071110 msgr2=0x7fd298003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:36.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2add9b640 1 --2- 172.21.15.125:0/256183510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2b0071110 0x7fd298003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:36.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2add9b640 1 -- 172.21.15.125:0/256183510 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd298004d10 msgr2=0x7fd298005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:36.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2add9b640 1 --2- 172.21.15.125:0/256183510 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd298004d10 0x7fd298005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:36.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2add9b640 1 -- 172.21.15.125:0/256183510 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd298003530 con 0x7fd298003f00 2023-10-30T13:12:36.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.658+0000 7fd2ae59c640 1 --2- 172.21.15.125:0/256183510 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2b0071110 0x7fd298003980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:36.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.662+0000 7fd2add9b640 1 --2- 172.21.15.125:0/256183510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd298003f00 0x7fd2980047d0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400c210 tx=0x7fd2a401cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3f344648054feeb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:36.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.662+0000 7fd2ad59a640 1 -- 172.21.15.125:0/256183510 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a40a5940 con 0x7fd298003f00 2023-10-30T13:12:36.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.662+0000 7fd2ad59a640 1 -- 172.21.15.125:0/256183510 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a40a5aa0 con 0x7fd298003f00 2023-10-30T13:12:36.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.662+0000 7fd2ad59a640 1 -- 172.21.15.125:0/256183510 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a40058e0 con 0x7fd298003f00 2023-10-30T13:12:36.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.662+0000 7fd2b4e91640 1 -- 172.21.15.125:0/256183510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd298003f00 msgr2=0x7fd2980047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:36.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.662+0000 7fd2b4e91640 1 --2- 172.21.15.125:0/256183510 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd298003f00 0x7fd2980047d0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fd2a400c210 tx=0x7fd2a401cc40 comp rx=0 tx=0).stop 2023-10-30T13:12:36.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.662+0000 7fd2aed9d640 1 -- 172.21.15.125:0/256183510 reap_dead start 2023-10-30T13:12:36.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.662+0000 7fd2b4e91640 1 -- 172.21.15.125:0/256183510 shutdown_connections 2023-10-30T13:12:36.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.662+0000 7fd2b4e91640 1 -- 172.21.15.125:0/256183510 >> 172.21.15.125:0/256183510 conn(0x7fd2b00c7910 msgr2=0x7fd2b00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:36.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.662+0000 7fd2b4e91640 1 -- 172.21.15.125:0/256183510 shutdown_connections 2023-10-30T13:12:36.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.662+0000 7fd2b4e91640 1 -- 172.21.15.125:0/256183510 wait complete. 2023-10-30T13:12:36.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2b4e91640 1 Processor -- start 2023-10-30T13:12:36.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2b4e91640 1 -- start start 2023-10-30T13:12:36.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2b4e91640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd298003f00 0x7fd2b01ecf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:36.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2b4e91640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2b0147610 0x7fd2b0149690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:36.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2b4e91640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2b01dda20 0x7fd2b01469c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:36.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2b4e91640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b00c3de0 con 0x7fd2b0147610 2023-10-30T13:12:36.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2b4e91640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b01eca70 con 0x7fd298003f00 2023-10-30T13:12:36.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2b4e91640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b01ec460 con 0x7fd2b01dda20 2023-10-30T13:12:36.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2add9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2b0147610 0x7fd2b0149690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:36.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2add9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2b0147610 0x7fd2b0149690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53728/0 (socket says 172.21.15.125:53728) 2023-10-30T13:12:36.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2add9b640 1 -- 172.21.15.125:0/211200063 learned_addr learned my addr 172.21.15.125:0/211200063 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:36.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2ae59c640 1 --2- 172.21.15.125:0/211200063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd298003f00 0x7fd2b01ecf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:36.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2aed9d640 1 --2- 172.21.15.125:0/211200063 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2b01dda20 0x7fd2b01469c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:36.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2add9b640 1 -- 172.21.15.125:0/211200063 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2b01dda20 msgr2=0x7fd2b01469c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:36.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2add9b640 1 --2- 172.21.15.125:0/211200063 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2b01dda20 0x7fd2b01469c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:36.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2add9b640 1 -- 172.21.15.125:0/211200063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd298003f00 msgr2=0x7fd2b01ecf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:36.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2add9b640 1 --2- 172.21.15.125:0/211200063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd298003f00 0x7fd2b01ecf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:36.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2add9b640 1 -- 172.21.15.125:0/211200063 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b01f0470 con 0x7fd2b0147610 2023-10-30T13:12:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2add9b640 1 --2- 172.21.15.125:0/211200063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2b0147610 0x7fd2b0149690 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fd2a40ab040 tx=0x7fd298005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd29effd640 1 -- 172.21.15.125:0/211200063 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4001c60 con 0x7fd2b0147610 2023-10-30T13:12:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd29effd640 1 -- 172.21.15.125:0/211200063 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a40b3070 con 0x7fd2b0147610 2023-10-30T13:12:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd29effd640 1 -- 172.21.15.125:0/211200063 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a40072d0 con 0x7fd2b0147610 2023-10-30T13:12:36.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2b4e91640 1 -- 172.21.15.125:0/211200063 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b01ec700 con 0x7fd2b0147610 2023-10-30T13:12:36.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2b4e91640 1 -- 172.21.15.125:0/211200063 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b014d8a0 con 0x7fd2b0147610 2023-10-30T13:12:36.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.666+0000 7fd2b4e91640 1 -- 172.21.15.125:0/211200063 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b01ec8c0 con 0x7fd2b0147610 2023-10-30T13:12:36.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.670+0000 7fd29effd640 1 -- 172.21.15.125:0/211200063 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd2a4007430 con 0x7fd2b0147610 2023-10-30T13:12:36.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.670+0000 7fd29effd640 1 --2- 172.21.15.125:0/211200063 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2900664a0 0x7fd290068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:36.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.670+0000 7fd29effd640 1 -- 172.21.15.125:0/211200063 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7fd2a40b5070 con 0x7fd2b0147610 2023-10-30T13:12:36.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.670+0000 7fd2ae59c640 1 --2- 172.21.15.125:0/211200063 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2900664a0 0x7fd290068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:36.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.670+0000 7fd2ae59c640 1 --2- 172.21.15.125:0/211200063 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2900664a0 0x7fd290068960 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009a30 tx=0x7fd2a0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:36.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.674+0000 7fd29effd640 1 -- 172.21.15.125:0/211200063 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd2a40ebb40 con 0x7fd2b0147610 2023-10-30T13:12:36.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.862+0000 7fd2b4e91640 1 -- 172.21.15.125:0/211200063 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd2b004e170 con 0x7fd2b0147610 2023-10-30T13:12:36.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.862+0000 7fd29effd640 1 -- 172.21.15.125:0/211200063 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+6552 (secure 0 0 0) 0x7fd2a40bf030 con 0x7fd2b0147610 2023-10-30T13:12:36.867 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:12:36.867 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:12:12.664330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:36.868 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 204 2023-10-30T13:12:36.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.862+0000 7fd2b4e91640 1 -- 172.21.15.125:0/211200063 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2900664a0 msgr2=0x7fd290068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:36.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.862+0000 7fd2b4e91640 1 --2- 172.21.15.125:0/211200063 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2900664a0 0x7fd290068960 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fd2a0009a30 tx=0x7fd2a0008040 comp rx=0 tx=0).stop 2023-10-30T13:12:36.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.862+0000 7fd2b4e91640 1 -- 172.21.15.125:0/211200063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2b0147610 msgr2=0x7fd2b0149690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:36.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.862+0000 7fd2b4e91640 1 --2- 172.21.15.125:0/211200063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2b0147610 0x7fd2b0149690 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fd2a40ab040 tx=0x7fd298005800 comp rx=0 tx=0).stop 2023-10-30T13:12:36.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.866+0000 7fd2b4e91640 1 -- 172.21.15.125:0/211200063 shutdown_connections 2023-10-30T13:12:36.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.866+0000 7fd2b4e91640 1 --2- 172.21.15.125:0/211200063 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2900664a0 0x7fd290068960 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:36.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.866+0000 7fd2b4e91640 1 --2- 172.21.15.125:0/211200063 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2b01dda20 0x7fd2b01469c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:36.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.866+0000 7fd2b4e91640 1 --2- 172.21.15.125:0/211200063 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2b0147610 0x7fd2b0149690 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:36.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.866+0000 7fd2b4e91640 1 --2- 172.21.15.125:0/211200063 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd298003f00 0x7fd2b01ecf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:36.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.866+0000 7fd2b4e91640 1 -- 172.21.15.125:0/211200063 >> 172.21.15.125:0/211200063 conn(0x7fd2b00c7910 msgr2=0x7fd2b006a5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:36.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.866+0000 7fd2b4e91640 1 -- 172.21.15.125:0/211200063 shutdown_connections 2023-10-30T13:12:36.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:36.866+0000 7fd2b4e91640 1 -- 172.21.15.125:0/211200063 wait complete. 2023-10-30T13:12:36.888 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:12:38.889 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:39.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.030+0000 7f466bd3e640 1 Processor -- start 2023-10-30T13:12:39.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.030+0000 7f466bd3e640 1 -- start start 2023-10-30T13:12:39.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.030+0000 7f466bd3e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4664070a00 0x7f4664070e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:39.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.030+0000 7f466bd3e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4664069e90 0x7f466406a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:39.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.030+0000 7f466bd3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46640713b0 con 0x7f46641539a0 2023-10-30T13:12:39.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.030+0000 7f466bd3e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f466406fbb0 con 0x7f4664070a00 2023-10-30T13:12:39.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.030+0000 7f466bd3e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4664153d80 con 0x7f4664069e90 2023-10-30T13:12:39.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.030+0000 7f46692d9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4664070a00 0x7f4664070e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:39.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.030+0000 7f46692d9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4664070a00 0x7f4664070e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53734/0 (socket says 172.21.15.125:53734) 2023-10-30T13:12:39.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.030+0000 7f46692d9640 1 -- 172.21.15.125:0/3640901611 learned_addr learned my addr 172.21.15.125:0/3640901611 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:39.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f466a2db640 1 --2- 172.21.15.125:0/3640901611 >> v2:172.21.15.136:3300/0 conn(0x7f4664069e90 0x7f466406a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:39.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f46692d9640 1 -- 172.21.15.125:0/3640901611 >> v2:172.21.15.136:3300/0 conn(0x7f4664069e90 msgr2=0x7f466406a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:39.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f46692d9640 1 --2- 172.21.15.125:0/3640901611 >> v2:172.21.15.136:3300/0 conn(0x7f4664069e90 0x7f466406a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:39.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f46692d9640 1 -- 172.21.15.125:0/3640901611 >> v1:172.21.15.136:6789/0 conn(0x7f46641539a0 legacy=0x7f466415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:39.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f46692d9640 1 -- 172.21.15.125:0/3640901611 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46640c5c50 con 0x7f4664070a00 2023-10-30T13:12:39.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f46692d9640 1 --2- 172.21.15.125:0/3640901611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4664070a00 0x7f4664070e70 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f46640b1080 tx=0x7f465401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5a9d1f6a98b820e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:39.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f4668ad8640 1 -- 172.21.15.125:0/3640901611 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4654004230 con 0x7f4664070a00 2023-10-30T13:12:39.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f4668ad8640 1 -- 172.21.15.125:0/3640901611 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46540a7050 con 0x7f4664070a00 2023-10-30T13:12:39.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f4668ad8640 1 -- 172.21.15.125:0/3640901611 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4654003620 con 0x7f4664070a00 2023-10-30T13:12:39.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f466bd3e640 1 -- 172.21.15.125:0/3640901611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4664070a00 msgr2=0x7f4664070e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:39.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f466bd3e640 1 --2- 172.21.15.125:0/3640901611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4664070a00 0x7f4664070e70 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f46640b1080 tx=0x7f465401c440 comp rx=0 tx=0).stop 2023-10-30T13:12:39.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f466bd3e640 1 -- 172.21.15.125:0/3640901611 shutdown_connections 2023-10-30T13:12:39.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f466bd3e640 1 --2- 172.21.15.125:0/3640901611 >> v2:172.21.15.136:3300/0 conn(0x7f4664069e90 0x7f466406a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:39.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f466bd3e640 1 --2- 172.21.15.125:0/3640901611 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4664070a00 0x7f4664070e70 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:39.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.034+0000 7f466bd3e640 1 -- 172.21.15.125:0/3640901611 >> 172.21.15.125:0/3640901611 conn(0x7f46640c7910 msgr2=0x7f46640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:39.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466bd3e640 1 -- 172.21.15.125:0/3640901611 shutdown_connections 2023-10-30T13:12:39.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466bd3e640 1 -- 172.21.15.125:0/3640901611 wait complete. 2023-10-30T13:12:39.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466bd3e640 1 Processor -- start 2023-10-30T13:12:39.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466bd3e640 1 -- start start 2023-10-30T13:12:39.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466bd3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4664069e90 0x7f4664143de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:39.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466bd3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4664070a00 0x7f4664144320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:39.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466bd3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46641539a0 0x7f46641f6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:39.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466bd3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46640c9550 con 0x7f46641539a0 2023-10-30T13:12:39.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466bd3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46640c8cb0 con 0x7f4664069e90 2023-10-30T13:12:39.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466bd3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46640c8f90 con 0x7f4664070a00 2023-10-30T13:12:39.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466a2db640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46641539a0 0x7f46641f6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:39.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466a2db640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46641539a0 0x7f46641f6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53742/0 (socket says 172.21.15.125:53742) 2023-10-30T13:12:39.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466a2db640 1 -- 172.21.15.125:0/3421488142 learned_addr learned my addr 172.21.15.125:0/3421488142 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:39.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f46692d9640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4664070a00 0x7f4664144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:39.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f4669ada640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4664069e90 0x7f4664143de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:39.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466a2db640 1 -- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4664070a00 msgr2=0x7f4664144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:39.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466a2db640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4664070a00 0x7f4664144320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:39.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466a2db640 1 -- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4664069e90 msgr2=0x7f4664143de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:39.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466a2db640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4664069e90 0x7f4664143de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:39.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466a2db640 1 -- 172.21.15.125:0/3421488142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46540094d0 con 0x7f46641539a0 2023-10-30T13:12:39.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f4669ada640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4664069e90 0x7f4664143de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:12:39.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466a2db640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46641539a0 0x7f46641f6060 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f465c007e40 tx=0x7f465c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:39.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f465a7fc640 1 -- 172.21.15.125:0/3421488142 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c014620 con 0x7f46641539a0 2023-10-30T13:12:39.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f465a7fc640 1 -- 172.21.15.125:0/3421488142 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f465c00ee90 con 0x7f46641539a0 2023-10-30T13:12:39.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466bd3e640 1 -- 172.21.15.125:0/3421488142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46640c9270 con 0x7f46641539a0 2023-10-30T13:12:39.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f465a7fc640 1 -- 172.21.15.125:0/3421488142 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f465c0149e0 con 0x7f46641539a0 2023-10-30T13:12:39.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.038+0000 7f466bd3e640 1 -- 172.21.15.125:0/3421488142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4664143430 con 0x7f46641539a0 2023-10-30T13:12:39.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.042+0000 7f466bd3e640 1 -- 172.21.15.125:0/3421488142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46640c8780 con 0x7f46641539a0 2023-10-30T13:12:39.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.042+0000 7f465a7fc640 1 -- 172.21.15.125:0/3421488142 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f465c014b40 con 0x7f46641539a0 2023-10-30T13:12:39.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.042+0000 7f465a7fc640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46480664a0 0x7f4648068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:39.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.042+0000 7f465a7fc640 1 -- 172.21.15.125:0/3421488142 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7f465c085160 con 0x7f46641539a0 2023-10-30T13:12:39.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.042+0000 7f4669ada640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46480664a0 0x7f4648068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:39.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.046+0000 7f4669ada640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46480664a0 0x7f4648068960 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4660000f70 tx=0x7f4660009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:39.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.046+0000 7f465a7fc640 1 -- 172.21.15.125:0/3421488142 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f465c051d10 con 0x7f46641539a0 2023-10-30T13:12:39.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.226+0000 7f466bd3e640 1 -- 172.21.15.125:0/3421488142 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f466404e170 con 0x7f46641539a0 2023-10-30T13:12:39.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.226+0000 7f465a7fc640 1 -- 172.21.15.125:0/3421488142 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+6552 (secure 0 0 0) 0x7f465c055330 con 0x7f46641539a0 2023-10-30T13:12:39.232 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:12:39.232 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:12:12.664330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:39.233 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 204 2023-10-30T13:12:39.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.230+0000 7f466bd3e640 1 -- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46480664a0 msgr2=0x7f4648068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:39.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.230+0000 7f466bd3e640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46480664a0 0x7f4648068960 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4660000f70 tx=0x7f4660009040 comp rx=0 tx=0).stop 2023-10-30T13:12:39.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.230+0000 7f466bd3e640 1 -- 172.21.15.125:0/3421488142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46641539a0 msgr2=0x7f46641f6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:39.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.230+0000 7f466bd3e640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46641539a0 0x7f46641f6060 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f465c007e40 tx=0x7f465c00e4b0 comp rx=0 tx=0).stop 2023-10-30T13:12:39.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.230+0000 7f466bd3e640 1 -- 172.21.15.125:0/3421488142 shutdown_connections 2023-10-30T13:12:39.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.230+0000 7f466bd3e640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f46480664a0 0x7f4648068960 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:39.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.230+0000 7f466bd3e640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f46641539a0 0x7f46641f6060 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:39.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.230+0000 7f466bd3e640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4664070a00 0x7f4664144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:39.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.230+0000 7f466bd3e640 1 --2- 172.21.15.125:0/3421488142 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4664069e90 0x7f4664143de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:39.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.230+0000 7f466bd3e640 1 -- 172.21.15.125:0/3421488142 >> 172.21.15.125:0/3421488142 conn(0x7f46640c7910 msgr2=0x7f4664151c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:39.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.230+0000 7f466bd3e640 1 -- 172.21.15.125:0/3421488142 shutdown_connections 2023-10-30T13:12:39.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:39.230+0000 7f466bd3e640 1 -- 172.21.15.125:0/3421488142 wait complete. 2023-10-30T13:12:39.253 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:12:41.254 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:41.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bd5d9d640 1 Processor -- start 2023-10-30T13:12:41.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bd5d9d640 1 -- start start 2023-10-30T13:12:41.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bd5d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9bd01538a0 0x7f9bd015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:41.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bd5d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9bd00ca0e0 0x7f9bd015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:41.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bd5d9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd00cf160 0x7f9bd00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:41.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bd5d9d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd00cb800 con 0x7f9bd01538a0 2023-10-30T13:12:41.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bd5d9d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd015dc30 con 0x7f9bd00ca0e0 2023-10-30T13:12:41.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bd5d9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd00cb330 con 0x7f9bd00cf160 2023-10-30T13:12:41.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bcf7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9bd01538a0 0x7f9bd015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:41.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bcf7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f9bd01538a0 0x7f9bd015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50334/0 (socket says 172.21.15.125:50334) 2023-10-30T13:12:41.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bcf7fe640 1 -- 172.21.15.125:0/1574603713 learned_addr learned my addr 172.21.15.125:0/1574603713 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:41.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bcffff640 1 --2- 172.21.15.125:0/1574603713 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd00cf160 0x7f9bd00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:41.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bceffd640 1 --2- 172.21.15.125:0/1574603713 >> v2:172.21.15.136:3300/0 conn(0x7f9bd00ca0e0 0x7f9bd015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:41.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bcf7fe640 1 -- 172.21.15.125:0/1574603713 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd00cf160 msgr2=0x7f9bd00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:41.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bcf7fe640 1 --2- 172.21.15.125:0/1574603713 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd00cf160 0x7f9bd00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:41.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bcf7fe640 1 -- 172.21.15.125:0/1574603713 >> v2:172.21.15.136:3300/0 conn(0x7f9bd00ca0e0 msgr2=0x7f9bd015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:41.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bcf7fe640 1 --2- 172.21.15.125:0/1574603713 >> v2:172.21.15.136:3300/0 conn(0x7f9bd00ca0e0 0x7f9bd015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:41.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.398+0000 7f9bcf7fe640 1 -- 172.21.15.125:0/1574603713 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd00cad70 con 0x7f9bd01538a0 2023-10-30T13:12:41.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.402+0000 7f9bcf7fe640 1 --2- 172.21.15.125:0/1574603713 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd01538a0 0x7f9bd015d6f0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00b30a0 tx=0x7f9bc001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d134eacb8d756868 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:41.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.402+0000 7f9bce7fc640 1 -- 172.21.15.125:0/1574603713 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc00a89b0 con 0x7f9bd01538a0 2023-10-30T13:12:41.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.402+0000 7f9bce7fc640 1 -- 172.21.15.125:0/1574603713 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc00a8b10 con 0x7f9bd01538a0 2023-10-30T13:12:41.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.402+0000 7f9bce7fc640 1 -- 172.21.15.125:0/1574603713 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc00a8ea0 con 0x7f9bd01538a0 2023-10-30T13:12:41.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.402+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/1574603713 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd01538a0 msgr2=0x7f9bd015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:41.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.402+0000 7f9bd5d9d640 1 --2- 172.21.15.125:0/1574603713 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd01538a0 0x7f9bd015d6f0 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00b30a0 tx=0x7f9bc001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:12:41.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.402+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/1574603713 shutdown_connections 2023-10-30T13:12:41.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.402+0000 7f9bd5d9d640 1 --2- 172.21.15.125:0/1574603713 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd00cf160 0x7f9bd00cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:41.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.402+0000 7f9bd5d9d640 1 --2- 172.21.15.125:0/1574603713 >> v2:172.21.15.136:3300/0 conn(0x7f9bd00ca0e0 0x7f9bd015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:41.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.402+0000 7f9bd5d9d640 1 --2- 172.21.15.125:0/1574603713 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd01538a0 0x7f9bd015d6f0 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:41.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.402+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/1574603713 >> 172.21.15.125:0/1574603713 conn(0x7f9bd006bfe0 msgr2=0x7f9bd006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:41.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.402+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/1574603713 shutdown_connections 2023-10-30T13:12:41.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.402+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/1574603713 wait complete. 2023-10-30T13:12:41.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bd5d9d640 1 Processor -- start 2023-10-30T13:12:41.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bd5d9d640 1 -- start start 2023-10-30T13:12:41.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bd5d9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd00ca0e0 0x7f9bd015ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:41.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bd5d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd00cf160 0x7f9bd00b4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:41.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bd5d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd01538a0 0x7f9bd01f69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:41.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bd5d9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd015b1c0 con 0x7f9bd00cf160 2023-10-30T13:12:41.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bd5d9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd015a8b0 con 0x7f9bd01538a0 2023-10-30T13:12:41.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bd5d9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd015d580 con 0x7f9bd00ca0e0 2023-10-30T13:12:41.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bceffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd00cf160 0x7f9bd00b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:41.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bceffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd00cf160 0x7f9bd00b4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50340/0 (socket says 172.21.15.125:50340) 2023-10-30T13:12:41.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bceffd640 1 -- 172.21.15.125:0/2735680765 learned_addr learned my addr 172.21.15.125:0/2735680765 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:41.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bcf7fe640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd00ca0e0 0x7f9bd015ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:41.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bcffff640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd01538a0 0x7f9bd01f69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:41.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bceffd640 1 -- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd00ca0e0 msgr2=0x7f9bd015ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:41.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bceffd640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd00ca0e0 0x7f9bd015ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:41.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bceffd640 1 -- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd01538a0 msgr2=0x7f9bd01f69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:41.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bceffd640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd01538a0 0x7f9bd01f69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:41.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bceffd640 1 -- 172.21.15.125:0/2735680765 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc00094d0 con 0x7f9bd00cf160 2023-10-30T13:12:41.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bcffff640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd01538a0 0x7f9bd01f69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:41.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bceffd640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd00cf160 0x7f9bd00b4980 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400d800 tx=0x7f9bc400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:41.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bb7fff640 1 -- 172.21.15.125:0/2735680765 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc40096a0 con 0x7f9bd00cf160 2023-10-30T13:12:41.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bb7fff640 1 -- 172.21.15.125:0/2735680765 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc4009800 con 0x7f9bd00cf160 2023-10-30T13:12:41.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bb7fff640 1 -- 172.21.15.125:0/2735680765 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc4009b70 con 0x7f9bd00cf160 2023-10-30T13:12:41.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/2735680765 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd015d860 con 0x7f9bd00cf160 2023-10-30T13:12:41.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/2735680765 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd015ade0 con 0x7f9bd00cf160 2023-10-30T13:12:41.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.406+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/2735680765 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd00b2a50 con 0x7f9bd00cf160 2023-10-30T13:12:41.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.410+0000 7f9bb7fff640 1 -- 172.21.15.125:0/2735680765 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9bc4009cd0 con 0x7f9bd00cf160 2023-10-30T13:12:41.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.410+0000 7f9bb7fff640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9bbc0664d0 0x7f9bbc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:41.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.410+0000 7f9bb7fff640 1 -- 172.21.15.125:0/2735680765 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7f9bc4084d40 con 0x7f9bd00cf160 2023-10-30T13:12:41.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.410+0000 7f9bcf7fe640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9bbc0664d0 0x7f9bbc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:41.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.410+0000 7f9bcf7fe640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9bbc0664d0 0x7f9bbc068990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00b2000 tx=0x7f9bc0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:41.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.414+0000 7f9bb7fff640 1 -- 172.21.15.125:0/2735680765 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9bc4051980 con 0x7f9bd00cf160 2023-10-30T13:12:41.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.594+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/2735680765 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bd004e170 con 0x7f9bd00cf160 2023-10-30T13:12:41.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.594+0000 7f9bb7fff640 1 -- 172.21.15.125:0/2735680765 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+6552 (secure 0 0 0) 0x7f9bc4054fa0 con 0x7f9bd00cf160 2023-10-30T13:12:41.601 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:12:41.601 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:12:12.664330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:41.601 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 204 2023-10-30T13:12:41.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.598+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9bbc0664d0 msgr2=0x7f9bbc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:41.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.598+0000 7f9bd5d9d640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9bbc0664d0 0x7f9bbc068990 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00b2000 tx=0x7f9bc0009040 comp rx=0 tx=0).stop 2023-10-30T13:12:41.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.598+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/2735680765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd00cf160 msgr2=0x7f9bd00b4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:41.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.598+0000 7f9bd5d9d640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd00cf160 0x7f9bd00b4980 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f9bc400d800 tx=0x7f9bc400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:12:41.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.598+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/2735680765 shutdown_connections 2023-10-30T13:12:41.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.598+0000 7f9bd5d9d640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9bbc0664d0 0x7f9bbc068990 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:41.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.598+0000 7f9bd5d9d640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9bd01538a0 0x7f9bd01f69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:41.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.598+0000 7f9bd5d9d640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9bd00cf160 0x7f9bd00b4980 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:41.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.598+0000 7f9bd5d9d640 1 --2- 172.21.15.125:0/2735680765 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9bd00ca0e0 0x7f9bd015ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:41.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.598+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/2735680765 >> 172.21.15.125:0/2735680765 conn(0x7f9bd006bfe0 msgr2=0x7f9bd01524c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:41.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.598+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/2735680765 shutdown_connections 2023-10-30T13:12:41.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:41.598+0000 7f9bd5d9d640 1 -- 172.21.15.125:0/2735680765 wait complete. 2023-10-30T13:12:41.622 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:12:43.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:43.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.766+0000 7f9075879640 1 Processor -- start 2023-10-30T13:12:43.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.766+0000 7f9075879640 1 -- start start 2023-10-30T13:12:43.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.766+0000 7f9075879640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f907015d6f0 0x7f907015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:43.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.766+0000 7f9075879640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90700cb800 con 0x7f90701538a0 2023-10-30T13:12:43.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.766+0000 7f9075879640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90700cb330 con 0x7f90700cf160 2023-10-30T13:12:43.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.766+0000 7f9075879640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90700cad70 con 0x7f907015d6f0 2023-10-30T13:12:43.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.766+0000 7f90667fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f907015d6f0 0x7f907015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:43.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.766+0000 7f90667fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f907015d6f0 0x7f907015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50342/0 (socket says 172.21.15.125:50342) 2023-10-30T13:12:43.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.766+0000 7f90667fc640 1 -- 172.21.15.125:0/4212227902 learned_addr learned my addr 172.21.15.125:0/4212227902 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:43.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.766+0000 7f90667fc640 1 -- 172.21.15.125:0/4212227902 >> v1:172.21.15.136:6789/0 conn(0x7f90700cf160 legacy=0x7f9070164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:43.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.766+0000 7f90667fc640 1 -- 172.21.15.125:0/4212227902 >> v1:172.21.15.125:6789/0 conn(0x7f90701538a0 legacy=0x7f90700ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:43.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.766+0000 7f90667fc640 1 -- 172.21.15.125:0/4212227902 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90700cb050 con 0x7f907015d6f0 2023-10-30T13:12:43.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.770+0000 7f90667fc640 1 --2- 172.21.15.125:0/4212227902 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907015d6f0 0x7f907015a090 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f9060009870 tx=0x7f906001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e11849aa80462f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:43.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.770+0000 7f906e7fc640 1 -- 172.21.15.125:0/4212227902 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90600a89f0 con 0x7f907015d6f0 2023-10-30T13:12:43.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.770+0000 7f906e7fc640 1 -- 172.21.15.125:0/4212227902 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90600a8b50 con 0x7f907015d6f0 2023-10-30T13:12:43.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.770+0000 7f906e7fc640 1 -- 172.21.15.125:0/4212227902 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90600b0440 con 0x7f907015d6f0 2023-10-30T13:12:43.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.770+0000 7f9075879640 1 -- 172.21.15.125:0/4212227902 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907015d6f0 msgr2=0x7f907015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:43.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.770+0000 7f9075879640 1 --2- 172.21.15.125:0/4212227902 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907015d6f0 0x7f907015a090 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f9060009870 tx=0x7f906001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:12:43.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.770+0000 7f9075879640 1 -- 172.21.15.125:0/4212227902 shutdown_connections 2023-10-30T13:12:43.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.770+0000 7f9075879640 1 --2- 172.21.15.125:0/4212227902 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907015d6f0 0x7f907015a090 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:43.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.770+0000 7f9075879640 1 -- 172.21.15.125:0/4212227902 >> 172.21.15.125:0/4212227902 conn(0x7f907006bfe0 msgr2=0x7f907006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:43.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.770+0000 7f9075879640 1 -- 172.21.15.125:0/4212227902 shutdown_connections 2023-10-30T13:12:43.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.770+0000 7f9075879640 1 -- 172.21.15.125:0/4212227902 wait complete. 2023-10-30T13:12:43.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f9075879640 1 Processor -- start 2023-10-30T13:12:43.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f9075879640 1 -- start start 2023-10-30T13:12:43.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f9075879640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90700cf160 0x7f907014d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:43.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f9075879640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90701538a0 0x7f90701eef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:43.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f9075879640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907015d6f0 0x7f90701e5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:43.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f9075879640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f907014d7c0 con 0x7f907015d6f0 2023-10-30T13:12:43.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f9075879640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9070158fa0 con 0x7f90700cf160 2023-10-30T13:12:43.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f9075879640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9070158410 con 0x7f90701538a0 2023-10-30T13:12:43.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f906f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907015d6f0 0x7f90701e5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:43.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f906f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907015d6f0 0x7f90701e5cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50358/0 (socket says 172.21.15.125:50358) 2023-10-30T13:12:43.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f906f7fe640 1 -- 172.21.15.125:0/1519320858 learned_addr learned my addr 172.21.15.125:0/1519320858 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:43.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f90667fc640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90701538a0 0x7f90701eef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:43.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f906effd640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90700cf160 0x7f907014d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:43.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f906f7fe640 1 -- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90701538a0 msgr2=0x7f90701eef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:43.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f906f7fe640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90701538a0 0x7f90701eef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:43.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f906f7fe640 1 -- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90700cf160 msgr2=0x7f907014d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:43.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f906f7fe640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90700cf160 0x7f907014d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:43.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f906f7fe640 1 -- 172.21.15.125:0/1519320858 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90600094d0 con 0x7f907015d6f0 2023-10-30T13:12:43.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f906effd640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90700cf160 0x7f907014d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:43.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f906f7fe640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907015d6f0 0x7f90701e5cf0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f906800b9e0 tx=0x7f906800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:43.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f906cff9640 1 -- 172.21.15.125:0/1519320858 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9068003b80 con 0x7f907015d6f0 2023-10-30T13:12:43.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f906cff9640 1 -- 172.21.15.125:0/1519320858 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9068003ce0 con 0x7f907015d6f0 2023-10-30T13:12:43.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f906cff9640 1 -- 172.21.15.125:0/1519320858 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f906800cae0 con 0x7f907015d6f0 2023-10-30T13:12:43.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f9075879640 1 -- 172.21.15.125:0/1519320858 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90701593a0 con 0x7f907015d6f0 2023-10-30T13:12:43.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.774+0000 7f9075879640 1 -- 172.21.15.125:0/1519320858 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9070159640 con 0x7f907015d6f0 2023-10-30T13:12:43.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.778+0000 7f9075879640 1 -- 172.21.15.125:0/1519320858 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907004e170 con 0x7f907015d6f0 2023-10-30T13:12:43.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.778+0000 7f906cff9640 1 -- 172.21.15.125:0/1519320858 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f906800cc40 con 0x7f907015d6f0 2023-10-30T13:12:43.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.778+0000 7f906cff9640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90500664a0 0x7f9050068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:43.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.778+0000 7f906cff9640 1 -- 172.21.15.125:0/1519320858 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7f9068084380 con 0x7f907015d6f0 2023-10-30T13:12:43.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.778+0000 7f906effd640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90500664a0 0x7f9050068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:43.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.782+0000 7f906effd640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90500664a0 0x7f9050068960 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f9058000f70 tx=0x7f9058009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:43.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.782+0000 7f906cff9640 1 -- 172.21.15.125:0/1519320858 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9068050fe0 con 0x7f907015d6f0 2023-10-30T13:12:43.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.966+0000 7f9075879640 1 -- 172.21.15.125:0/1519320858 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9070150b60 con 0x7f907015d6f0 2023-10-30T13:12:43.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.966+0000 7f906cff9640 1 -- 172.21.15.125:0/1519320858 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+6552 (secure 0 0 0) 0x7f9068054600 con 0x7f907015d6f0 2023-10-30T13:12:43.970 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:12:43.970 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:12:12.664330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:43.971 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 204 2023-10-30T13:12:43.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.966+0000 7f9075879640 1 -- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90500664a0 msgr2=0x7f9050068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:43.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.966+0000 7f9075879640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90500664a0 0x7f9050068960 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f9058000f70 tx=0x7f9058009040 comp rx=0 tx=0).stop 2023-10-30T13:12:43.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.966+0000 7f9075879640 1 -- 172.21.15.125:0/1519320858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907015d6f0 msgr2=0x7f90701e5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:43.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.966+0000 7f9075879640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907015d6f0 0x7f90701e5cf0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f906800b9e0 tx=0x7f906800bea0 comp rx=0 tx=0).stop 2023-10-30T13:12:43.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.966+0000 7f9075879640 1 -- 172.21.15.125:0/1519320858 shutdown_connections 2023-10-30T13:12:43.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.966+0000 7f9075879640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90500664a0 0x7f9050068960 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:43.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.966+0000 7f9075879640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f907015d6f0 0x7f90701e5cf0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:43.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.966+0000 7f9075879640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90701538a0 0x7f90701eef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:43.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.966+0000 7f9075879640 1 --2- 172.21.15.125:0/1519320858 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90700cf160 0x7f907014d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:43.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.966+0000 7f9075879640 1 -- 172.21.15.125:0/1519320858 >> 172.21.15.125:0/1519320858 conn(0x7f907006bfe0 msgr2=0x7f90700c5300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:43.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.966+0000 7f9075879640 1 -- 172.21.15.125:0/1519320858 shutdown_connections 2023-10-30T13:12:43.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:43.966+0000 7f9075879640 1 -- 172.21.15.125:0/1519320858 wait complete. 2023-10-30T13:12:43.992 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:12:45.993 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:46.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae75b3e640 1 Processor -- start 2023-10-30T13:12:46.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae75b3e640 1 -- start start 2023-10-30T13:12:46.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae75b3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae700c05d0 0x7fae7013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:46.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae75b3e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae700c0a50 con 0x7fae700c6b80 2023-10-30T13:12:46.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae75b3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7006f6e0 con 0x7fae700c6f60 2023-10-30T13:12:46.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae75b3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae7006f9c0 con 0x7fae700c05d0 2023-10-30T13:12:46.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6f7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fae700c6b80 0x7fae700bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37640/0 (socket says 172.21.15.125:37640) 2023-10-30T13:12:46.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6f7fe640 1 -- 172.21.15.125:0/984462884 learned_addr learned my addr 172.21.15.125:0/984462884 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:12:46.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6effd640 1 --2- 172.21.15.125:0/984462884 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae700c05d0 0x7fae7013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:46.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 -- 172.21.15.125:0/984462884 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825985636 0 0) 0x7fae700c0a50 con 0x7fae700c6b80 2023-10-30T13:12:46.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 -- 172.21.15.125:0/984462884 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae58003530 con 0x7fae700c6b80 2023-10-30T13:12:46.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 -- 172.21.15.125:0/984462884 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1953144034 0 0) 0x7fae58003530 con 0x7fae700c6b80 2023-10-30T13:12:46.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 -- 172.21.15.125:0/984462884 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae700c05d0 msgr2=0x7fae7013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:46.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 --2- 172.21.15.125:0/984462884 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae700c05d0 0x7fae7013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:46.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 -- 172.21.15.125:0/984462884 >> v1:172.21.15.136:6789/0 conn(0x7fae700c6f60 legacy=0x7fae701566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:46.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 -- 172.21.15.125:0/984462884 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7006fca0 con 0x7fae700c6b80 2023-10-30T13:12:46.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 -- 172.21.15.125:0/984462884 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fae60001680 con 0x7fae700c6b80 2023-10-30T13:12:46.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 -- 172.21.15.125:0/984462884 >> v1:172.21.15.125:6789/0 conn(0x7fae700c6b80 legacy=0x7fae700bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:46.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 --2- 172.21.15.125:0/984462884 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae700c05d0 0x7fae580039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:46.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 --2- 172.21.15.125:0/984462884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae58003f30 0x7fae580047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:46.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 --2- 172.21.15.125:0/984462884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae58004d10 0x7fae58005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:46.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 -- 172.21.15.125:0/984462884 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7006f9c0 con 0x7fae58003f30 2023-10-30T13:12:46.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 -- 172.21.15.125:0/984462884 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae5800ba70 con 0x7fae58004d10 2023-10-30T13:12:46.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6e7fc640 1 -- 172.21.15.125:0/984462884 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae5800de40 con 0x7fae700c05d0 2023-10-30T13:12:46.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6effd640 1 --2- 172.21.15.125:0/984462884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae58003f30 0x7fae580047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:46.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6f7fe640 1 --2- 172.21.15.125:0/984462884 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae700c05d0 0x7fae580039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:46.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6ffff640 1 --2- 172.21.15.125:0/984462884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae58004d10 0x7fae58005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:46.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6effd640 1 -- 172.21.15.125:0/984462884 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae700c05d0 msgr2=0x7fae580039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:46.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6effd640 1 --2- 172.21.15.125:0/984462884 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae700c05d0 0x7fae580039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:46.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6effd640 1 -- 172.21.15.125:0/984462884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae58004d10 msgr2=0x7fae58005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:46.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6effd640 1 --2- 172.21.15.125:0/984462884 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae58004d10 0x7fae58005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:46.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.138+0000 7fae6effd640 1 -- 172.21.15.125:0/984462884 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae58003530 con 0x7fae58003f30 2023-10-30T13:12:46.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.142+0000 7fae6effd640 1 --2- 172.21.15.125:0/984462884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae58003f30 0x7fae580047d0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fae64009ae0 tx=0x7fae6401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa75976c127ddc43 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:46.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.142+0000 7fae6e7fc640 1 -- 172.21.15.125:0/984462884 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae640a5540 con 0x7fae58003f30 2023-10-30T13:12:46.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.142+0000 7fae6e7fc640 1 -- 172.21.15.125:0/984462884 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae640a56a0 con 0x7fae58003f30 2023-10-30T13:12:46.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.142+0000 7fae6e7fc640 1 -- 172.21.15.125:0/984462884 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae640a5a50 con 0x7fae58003f30 2023-10-30T13:12:46.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.142+0000 7fae75b3e640 1 -- 172.21.15.125:0/984462884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae58003f30 msgr2=0x7fae580047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:46.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.142+0000 7fae75b3e640 1 --2- 172.21.15.125:0/984462884 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae58003f30 0x7fae580047d0 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7fae64009ae0 tx=0x7fae6401ca50 comp rx=0 tx=0).stop 2023-10-30T13:12:46.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.142+0000 7fae6ffff640 1 -- 172.21.15.125:0/984462884 reap_dead start 2023-10-30T13:12:46.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.142+0000 7fae75b3e640 1 -- 172.21.15.125:0/984462884 shutdown_connections 2023-10-30T13:12:46.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.142+0000 7fae75b3e640 1 -- 172.21.15.125:0/984462884 >> 172.21.15.125:0/984462884 conn(0x7fae700c7910 msgr2=0x7fae700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:46.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.142+0000 7fae75b3e640 1 -- 172.21.15.125:0/984462884 shutdown_connections 2023-10-30T13:12:46.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.142+0000 7fae75b3e640 1 -- 172.21.15.125:0/984462884 wait complete. 2023-10-30T13:12:46.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae75b3e640 1 Processor -- start 2023-10-30T13:12:46.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae75b3e640 1 -- start start 2023-10-30T13:12:46.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae75b3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae58003f30 0x7fae701ddb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:46.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae75b3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae700ca430 0x7fae701dc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:46.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae75b3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae700c38b0 0x7fae701edd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:46.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae75b3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae700c2ec0 con 0x7fae58003f30 2023-10-30T13:12:46.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae75b3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae701de300 con 0x7fae700ca430 2023-10-30T13:12:46.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae75b3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae700cabe0 con 0x7fae700c38b0 2023-10-30T13:12:46.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae6f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae58003f30 0x7fae701ddb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:46.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae6f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae58003f30 0x7fae701ddb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50384/0 (socket says 172.21.15.125:50384) 2023-10-30T13:12:46.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae6f7fe640 1 -- 172.21.15.125:0/2357547074 learned_addr learned my addr 172.21.15.125:0/2357547074 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:46.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae6ffff640 1 --2- 172.21.15.125:0/2357547074 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae700c38b0 0x7fae701edd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:46.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae6effd640 1 --2- 172.21.15.125:0/2357547074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae700ca430 0x7fae701dc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:46.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae6f7fe640 1 -- 172.21.15.125:0/2357547074 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae700c38b0 msgr2=0x7fae701edd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:46.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae6f7fe640 1 --2- 172.21.15.125:0/2357547074 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae700c38b0 0x7fae701edd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:46.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae6f7fe640 1 -- 172.21.15.125:0/2357547074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae700ca430 msgr2=0x7fae701dc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:46.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae6f7fe640 1 --2- 172.21.15.125:0/2357547074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae700ca430 0x7fae701dc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:46.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae6f7fe640 1 -- 172.21.15.125:0/2357547074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7013e280 con 0x7fae58003f30 2023-10-30T13:12:46.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae6f7fe640 1 --2- 172.21.15.125:0/2357547074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae58003f30 0x7fae701ddb70 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fae60012490 tx=0x7fae60012950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:46.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae57fff640 1 -- 172.21.15.125:0/2357547074 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae60014810 con 0x7fae58003f30 2023-10-30T13:12:46.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae57fff640 1 -- 172.21.15.125:0/2357547074 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae60014970 con 0x7fae58003f30 2023-10-30T13:12:46.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae57fff640 1 -- 172.21.15.125:0/2357547074 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae60014d20 con 0x7fae58003f30 2023-10-30T13:12:46.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae75b3e640 1 -- 172.21.15.125:0/2357547074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae700c22b0 con 0x7fae58003f30 2023-10-30T13:12:46.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.146+0000 7fae75b3e640 1 -- 172.21.15.125:0/2357547074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae700c2870 con 0x7fae58003f30 2023-10-30T13:12:46.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.150+0000 7fae75b3e640 1 -- 172.21.15.125:0/2357547074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7004e170 con 0x7fae58003f30 2023-10-30T13:12:46.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.150+0000 7fae57fff640 1 -- 172.21.15.125:0/2357547074 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fae600153e0 con 0x7fae58003f30 2023-10-30T13:12:46.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.150+0000 7fae57fff640 1 --2- 172.21.15.125:0/2357547074 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fae5c0664a0 0x7fae5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:46.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.150+0000 7fae57fff640 1 -- 172.21.15.125:0/2357547074 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7fae60086f60 con 0x7fae58003f30 2023-10-30T13:12:46.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.150+0000 7fae6effd640 1 --2- 172.21.15.125:0/2357547074 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fae5c0664a0 0x7fae5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:46.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.150+0000 7fae6effd640 1 --2- 172.21.15.125:0/2357547074 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fae5c0664a0 0x7fae5c068960 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fae640af000 tx=0x7fae640046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:46.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.154+0000 7fae57fff640 1 -- 172.21.15.125:0/2357547074 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fae60053bf0 con 0x7fae58003f30 2023-10-30T13:12:46.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.334+0000 7fae75b3e640 1 -- 172.21.15.125:0/2357547074 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae700c81a0 con 0x7fae58003f30 2023-10-30T13:12:46.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.334+0000 7fae57fff640 1 -- 172.21.15.125:0/2357547074 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+6552 (secure 0 0 0) 0x7fae6001c030 con 0x7fae58003f30 2023-10-30T13:12:46.338 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:12:46.338 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:12:12.664330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:46.339 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 204 2023-10-30T13:12:46.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.334+0000 7fae75b3e640 1 -- 172.21.15.125:0/2357547074 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fae5c0664a0 msgr2=0x7fae5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:46.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.334+0000 7fae75b3e640 1 --2- 172.21.15.125:0/2357547074 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fae5c0664a0 0x7fae5c068960 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fae640af000 tx=0x7fae640046d0 comp rx=0 tx=0).stop 2023-10-30T13:12:46.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.334+0000 7fae75b3e640 1 -- 172.21.15.125:0/2357547074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae58003f30 msgr2=0x7fae701ddb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:46.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.334+0000 7fae75b3e640 1 --2- 172.21.15.125:0/2357547074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae58003f30 0x7fae701ddb70 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fae60012490 tx=0x7fae60012950 comp rx=0 tx=0).stop 2023-10-30T13:12:46.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.334+0000 7fae75b3e640 1 -- 172.21.15.125:0/2357547074 shutdown_connections 2023-10-30T13:12:46.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.334+0000 7fae75b3e640 1 --2- 172.21.15.125:0/2357547074 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fae5c0664a0 0x7fae5c068960 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:46.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.334+0000 7fae75b3e640 1 --2- 172.21.15.125:0/2357547074 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fae700c38b0 0x7fae701edd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:46.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.334+0000 7fae75b3e640 1 --2- 172.21.15.125:0/2357547074 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fae700ca430 0x7fae701dc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:46.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.334+0000 7fae75b3e640 1 --2- 172.21.15.125:0/2357547074 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fae58003f30 0x7fae701ddb70 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:46.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.334+0000 7fae75b3e640 1 -- 172.21.15.125:0/2357547074 >> 172.21.15.125:0/2357547074 conn(0x7fae700c7910 msgr2=0x7fae700c19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:46.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.334+0000 7fae75b3e640 1 -- 172.21.15.125:0/2357547074 shutdown_connections 2023-10-30T13:12:46.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:46.334+0000 7fae75b3e640 1 -- 172.21.15.125:0/2357547074 wait complete. 2023-10-30T13:12:46.360 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:12:46.600 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:12:46.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:12:48.361 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:48.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa0a67c4640 1 Processor -- start 2023-10-30T13:12:48.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa0a67c4640 1 -- start start 2023-10-30T13:12:48.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa0a67c4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa0a00c8460 0x7fa0a00715c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:48.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa0a67c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a0070650 0x7fa0a0159810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:48.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa0a67c4640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a0159d50 con 0x7fa0a0157010 2023-10-30T13:12:48.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa0a67c4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a00c21b0 con 0x7fa0a00c8460 2023-10-30T13:12:48.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa0a67c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a00c1530 con 0x7fa0a0070650 2023-10-30T13:12:48.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09ffff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa0a0157010 0x7fa0a0071bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37646/0 (socket says 172.21.15.125:37646) 2023-10-30T13:12:48.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09ffff640 1 -- 172.21.15.125:0/247901413 learned_addr learned my addr 172.21.15.125:0/247901413 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:12:48.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa0a4d61640 1 --2- 172.21.15.125:0/247901413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a0070650 0x7fa0a0159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:48.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09f7fe640 1 --2- 172.21.15.125:0/247901413 >> v2:172.21.15.136:3300/0 conn(0x7fa0a00c8460 0x7fa0a00715c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:48.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 -- 172.21.15.125:0/247901413 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261880970 0 0) 0x7fa0a0159d50 con 0x7fa0a0157010 2023-10-30T13:12:48.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 -- 172.21.15.125:0/247901413 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa088003530 con 0x7fa0a0157010 2023-10-30T13:12:48.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 -- 172.21.15.125:0/247901413 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3504100282 0 0) 0x7fa088003530 con 0x7fa0a0157010 2023-10-30T13:12:48.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 -- 172.21.15.125:0/247901413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a0070650 msgr2=0x7fa0a0159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:48.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 --2- 172.21.15.125:0/247901413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a0070650 0x7fa0a0159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:48.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 -- 172.21.15.125:0/247901413 >> v2:172.21.15.136:3300/0 conn(0x7fa0a00c8460 msgr2=0x7fa0a00715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:48.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 --2- 172.21.15.125:0/247901413 >> v2:172.21.15.136:3300/0 conn(0x7fa0a00c8460 0x7fa0a00715c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:48.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 -- 172.21.15.125:0/247901413 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a00c0f70 con 0x7fa0a0157010 2023-10-30T13:12:48.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 -- 172.21.15.125:0/247901413 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fa090001680 con 0x7fa0a0157010 2023-10-30T13:12:48.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 -- 172.21.15.125:0/247901413 >> v1:172.21.15.125:6789/0 conn(0x7fa0a0157010 legacy=0x7fa0a0071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:48.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 --2- 172.21.15.125:0/247901413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa088003980 0x7fa088003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:48.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 --2- 172.21.15.125:0/247901413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a0070650 0x7fa0880047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:48.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 --2- 172.21.15.125:0/247901413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa088004d10 0x7fa088005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:48.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 -- 172.21.15.125:0/247901413 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a00c21b0 con 0x7fa088003980 2023-10-30T13:12:48.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 -- 172.21.15.125:0/247901413 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a00c1530 con 0x7fa088004d10 2023-10-30T13:12:48.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09effd640 1 -- 172.21.15.125:0/247901413 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa08800b990 con 0x7fa0a0070650 2023-10-30T13:12:48.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.506+0000 7fa09ffff640 1 --2- 172.21.15.125:0/247901413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa088003980 0x7fa088003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:48.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa0a4d61640 1 --2- 172.21.15.125:0/247901413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa088004d10 0x7fa088005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:48.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa09f7fe640 1 --2- 172.21.15.125:0/247901413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a0070650 0x7fa0880047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:48.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa09ffff640 1 -- 172.21.15.125:0/247901413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a0070650 msgr2=0x7fa0880047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:48.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa09ffff640 1 --2- 172.21.15.125:0/247901413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa0a0070650 0x7fa0880047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:48.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa09ffff640 1 -- 172.21.15.125:0/247901413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa088004d10 msgr2=0x7fa088005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:48.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa09ffff640 1 --2- 172.21.15.125:0/247901413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa088004d10 0x7fa088005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:48.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa09ffff640 1 -- 172.21.15.125:0/247901413 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa088003530 con 0x7fa088003980 2023-10-30T13:12:48.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa0a4d61640 1 --2- 172.21.15.125:0/247901413 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa088004d10 0x7fa088005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:48.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa09ffff640 1 --2- 172.21.15.125:0/247901413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa088003980 0x7fa088003da0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fa09000c270 tx=0x7fa09001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3774259376ce1351 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:48.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa09effd640 1 -- 172.21.15.125:0/247901413 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0900058e0 con 0x7fa088003980 2023-10-30T13:12:48.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa09effd640 1 -- 172.21.15.125:0/247901413 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa090005a40 con 0x7fa088003980 2023-10-30T13:12:48.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa09effd640 1 -- 172.21.15.125:0/247901413 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090005e10 con 0x7fa088003980 2023-10-30T13:12:48.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa0a67c4640 1 -- 172.21.15.125:0/247901413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa088003980 msgr2=0x7fa088003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:48.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa0a67c4640 1 --2- 172.21.15.125:0/247901413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa088003980 0x7fa088003da0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fa09000c270 tx=0x7fa09001cc60 comp rx=0 tx=0).stop 2023-10-30T13:12:48.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.510+0000 7fa0a4d61640 1 -- 172.21.15.125:0/247901413 reap_dead start 2023-10-30T13:12:48.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a67c4640 1 -- 172.21.15.125:0/247901413 shutdown_connections 2023-10-30T13:12:48.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a67c4640 1 -- 172.21.15.125:0/247901413 >> 172.21.15.125:0/247901413 conn(0x7fa0a00cbeb0 msgr2=0x7fa0a00cc2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:48.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a67c4640 1 -- 172.21.15.125:0/247901413 shutdown_connections 2023-10-30T13:12:48.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a67c4640 1 -- 172.21.15.125:0/247901413 wait complete. 2023-10-30T13:12:48.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a67c4640 1 Processor -- start 2023-10-30T13:12:48.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a67c4640 1 -- start start 2023-10-30T13:12:48.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a67c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa088003980 0x7fa0a01443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:48.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a67c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a01527d0 0x7fa0a0145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:48.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a67c4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a0152c10 0x7fa0a013f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:48.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a67c4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a01f2f20 con 0x7fa0a0152c10 2023-10-30T13:12:48.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a67c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a00717d0 con 0x7fa0a01527d0 2023-10-30T13:12:48.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a67c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a01f2310 con 0x7fa088003980 2023-10-30T13:12:48.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a4d61640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a0152c10 0x7fa0a013f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:48.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a4d61640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a0152c10 0x7fa0a013f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50410/0 (socket says 172.21.15.125:50410) 2023-10-30T13:12:48.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a4d61640 1 -- 172.21.15.125:0/979758446 learned_addr learned my addr 172.21.15.125:0/979758446 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:48.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa09ffff640 1 --2- 172.21.15.125:0/979758446 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa088003980 0x7fa0a01443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:48.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa09f7fe640 1 --2- 172.21.15.125:0/979758446 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a01527d0 0x7fa0a0145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:48.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a4d61640 1 -- 172.21.15.125:0/979758446 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa088003980 msgr2=0x7fa0a01443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:48.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a4d61640 1 --2- 172.21.15.125:0/979758446 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa088003980 0x7fa0a01443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:48.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a4d61640 1 -- 172.21.15.125:0/979758446 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a01527d0 msgr2=0x7fa0a0145160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:48.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a4d61640 1 --2- 172.21.15.125:0/979758446 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a01527d0 0x7fa0a0145160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:48.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a4d61640 1 -- 172.21.15.125:0/979758446 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a01f2580 con 0x7fa0a0152c10 2023-10-30T13:12:48.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.514+0000 7fa0a4d61640 1 --2- 172.21.15.125:0/979758446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a0152c10 0x7fa0a013f6d0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fa0a015de40 tx=0x7fa098010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:48.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.518+0000 7fa09cff9640 1 -- 172.21.15.125:0/979758446 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09801b070 con 0x7fa0a0152c10 2023-10-30T13:12:48.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.518+0000 7fa09cff9640 1 -- 172.21.15.125:0/979758446 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09800f040 con 0x7fa0a0152c10 2023-10-30T13:12:48.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.518+0000 7fa09cff9640 1 -- 172.21.15.125:0/979758446 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09800e050 con 0x7fa0a0152c10 2023-10-30T13:12:48.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.518+0000 7fa0a67c4640 1 -- 172.21.15.125:0/979758446 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a01f2860 con 0x7fa0a0152c10 2023-10-30T13:12:48.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.518+0000 7fa0a67c4640 1 -- 172.21.15.125:0/979758446 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a0144a60 con 0x7fa0a0152c10 2023-10-30T13:12:48.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.518+0000 7fa0a67c4640 1 -- 172.21.15.125:0/979758446 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a01f3060 con 0x7fa0a0152c10 2023-10-30T13:12:48.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.518+0000 7fa09cff9640 1 -- 172.21.15.125:0/979758446 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa098015440 con 0x7fa0a0152c10 2023-10-30T13:12:48.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.518+0000 7fa09cff9640 1 --2- 172.21.15.125:0/979758446 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa08c0664a0 0x7fa08c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:48.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.518+0000 7fa09cff9640 1 -- 172.21.15.125:0/979758446 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7802+0+0 (secure 0 0 0) 0x7fa098084930 con 0x7fa0a0152c10 2023-10-30T13:12:48.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.518+0000 7fa09ffff640 1 --2- 172.21.15.125:0/979758446 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa08c0664a0 0x7fa08c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:48.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.522+0000 7fa09ffff640 1 --2- 172.21.15.125:0/979758446 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa08c0664a0 0x7fa08c068960 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fa0900aa000 tx=0x7fa090001a80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:48.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.522+0000 7fa09cff9640 1 -- 172.21.15.125:0/979758446 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa0a01f3060 con 0x7fa0a0152c10 2023-10-30T13:12:48.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.706+0000 7fa0a67c4640 1 -- 172.21.15.125:0/979758446 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa0a01f3060 con 0x7fa0a0152c10 2023-10-30T13:12:48.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.706+0000 7fa09cff9640 1 -- 172.21.15.125:0/979758446 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 204 v204) v1 ==== 95+0+6552 (secure 0 0 0) 0x7fa0a01f3060 con 0x7fa0a0152c10 2023-10-30T13:12:48.709 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:12:48.710 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":204,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":7511,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":201}],"filesystems":[{"mdsmap":{"epoch":204,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:12:12.664330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":8301,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8301":{"gid":8301,"name":"c","rank":1,"incarnation":168,"state":"up:active","state_seq":6,"addr":"172.21.15.125:6837/2540063352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":2540063352},{"type":"v1","addr":"172.21.15.125:6837","nonce":2540063352}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:48.710 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 204 2023-10-30T13:12:48.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.706+0000 7fa0a67c4640 1 -- 172.21.15.125:0/979758446 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa08c0664a0 msgr2=0x7fa08c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:48.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.706+0000 7fa0a67c4640 1 --2- 172.21.15.125:0/979758446 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa08c0664a0 0x7fa08c068960 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fa0900aa000 tx=0x7fa090001a80 comp rx=0 tx=0).stop 2023-10-30T13:12:48.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.706+0000 7fa0a67c4640 1 -- 172.21.15.125:0/979758446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a0152c10 msgr2=0x7fa0a013f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:48.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.706+0000 7fa0a67c4640 1 --2- 172.21.15.125:0/979758446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a0152c10 0x7fa0a013f6d0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fa0a015de40 tx=0x7fa098010590 comp rx=0 tx=0).stop 2023-10-30T13:12:48.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.706+0000 7fa0a67c4640 1 -- 172.21.15.125:0/979758446 shutdown_connections 2023-10-30T13:12:48.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.706+0000 7fa0a67c4640 1 --2- 172.21.15.125:0/979758446 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa08c0664a0 0x7fa08c068960 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:48.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.706+0000 7fa0a67c4640 1 --2- 172.21.15.125:0/979758446 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa0a0152c10 0x7fa0a013f6d0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:48.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.706+0000 7fa0a67c4640 1 --2- 172.21.15.125:0/979758446 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa0a01527d0 0x7fa0a0145160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:48.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.706+0000 7fa0a67c4640 1 --2- 172.21.15.125:0/979758446 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa088003980 0x7fa0a01443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:48.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.706+0000 7fa0a67c4640 1 -- 172.21.15.125:0/979758446 >> 172.21.15.125:0/979758446 conn(0x7fa0a00cbeb0 msgr2=0x7fa0a00c7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:48.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.706+0000 7fa0a67c4640 1 -- 172.21.15.125:0/979758446 shutdown_connections 2023-10-30T13:12:48.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:48.706+0000 7fa0a67c4640 1 -- 172.21.15.125:0/979758446 wait complete. 2023-10-30T13:12:48.731 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is laggy/crashed, in failed state, or mds.c is removed from mdsmap 2023-10-30T13:12:50.732 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:50.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fffc5640 1 Processor -- start 2023-10-30T13:12:50.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fffc5640 1 -- start start 2023-10-30T13:12:50.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fffc5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f53f814fd50 0x7f53f80c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:50.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fffc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53f80cb550 0x7f53f80cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:50.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fffc5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53f81560a0 con 0x7f53f8071250 2023-10-30T13:12:50.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fffc5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53f8156380 con 0x7f53f814fd50 2023-10-30T13:12:50.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fffc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f8156660 con 0x7f53f80cb550 2023-10-30T13:12:50.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fdd61640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f53f814fd50 0x7f53f80c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:50.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fdd61640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f53f814fd50 0x7f53f80c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37006/0 (socket says 172.21.15.125:37006) 2023-10-30T13:12:50.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fdd61640 1 -- 172.21.15.125:0/3205829682 learned_addr learned my addr 172.21.15.125:0/3205829682 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:50.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fd560640 1 --2- 172.21.15.125:0/3205829682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53f80cb550 0x7f53f80cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:50.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fdd61640 1 -- 172.21.15.125:0/3205829682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53f80cb550 msgr2=0x7f53f80cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:50.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fdd61640 1 --2- 172.21.15.125:0/3205829682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53f80cb550 0x7f53f80cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:50.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fdd61640 1 -- 172.21.15.125:0/3205829682 >> v1:172.21.15.136:6789/0 conn(0x7f53f8071250 legacy=0x7f53f80716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:50.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fdd61640 1 -- 172.21.15.125:0/3205829682 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f8156940 con 0x7f53f814fd50 2023-10-30T13:12:50.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.878+0000 7f53fd560640 1 --2- 172.21.15.125:0/3205829682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53f80cb550 0x7f53f80cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:50.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.882+0000 7f53fdd61640 1 --2- 172.21.15.125:0/3205829682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f814fd50 0x7f53f80c65d0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009870 tx=0x7f53f401c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a59e8f3a0552dd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:50.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.882+0000 7f53fcd5f640 1 -- 172.21.15.125:0/3205829682 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f40ab650 con 0x7f53f814fd50 2023-10-30T13:12:50.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.882+0000 7f53fcd5f640 1 -- 172.21.15.125:0/3205829682 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f40ab7b0 con 0x7f53f814fd50 2023-10-30T13:12:50.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.882+0000 7f53fcd5f640 1 -- 172.21.15.125:0/3205829682 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f40abb40 con 0x7f53f814fd50 2023-10-30T13:12:50.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.882+0000 7f53fffc5640 1 -- 172.21.15.125:0/3205829682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f814fd50 msgr2=0x7f53f80c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:50.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.882+0000 7f53fffc5640 1 --2- 172.21.15.125:0/3205829682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f814fd50 0x7f53f80c65d0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009870 tx=0x7f53f401c510 comp rx=0 tx=0).stop 2023-10-30T13:12:50.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.882+0000 7f53fffc5640 1 -- 172.21.15.125:0/3205829682 shutdown_connections 2023-10-30T13:12:50.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.882+0000 7f53fffc5640 1 --2- 172.21.15.125:0/3205829682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53f80cb550 0x7f53f80cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:50.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.882+0000 7f53fffc5640 1 --2- 172.21.15.125:0/3205829682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f814fd50 0x7f53f80c65d0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:50.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.882+0000 7f53fffc5640 1 -- 172.21.15.125:0/3205829682 >> 172.21.15.125:0/3205829682 conn(0x7f53f806c6e0 msgr2=0x7f53f8069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:50.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.882+0000 7f53fffc5640 1 -- 172.21.15.125:0/3205829682 shutdown_connections 2023-10-30T13:12:50.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.882+0000 7f53fffc5640 1 -- 172.21.15.125:0/3205829682 wait complete. 2023-10-30T13:12:50.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fffc5640 1 Processor -- start 2023-10-30T13:12:50.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fffc5640 1 -- start start 2023-10-30T13:12:50.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fffc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53f8071250 0x7f53f8150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:50.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fffc5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f80cb550 0x7f53f81f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:50.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fffc5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53f814fd50 0x7f53f81531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:50.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fffc5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f8151130 con 0x7f53f80cb550 2023-10-30T13:12:50.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fffc5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f81f2de0 con 0x7f53f814fd50 2023-10-30T13:12:50.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fffc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53f81524b0 con 0x7f53f8071250 2023-10-30T13:12:50.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fd560640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f80cb550 0x7f53f81f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:50.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fd560640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f80cb550 0x7f53f81f28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37010/0 (socket says 172.21.15.125:37010) 2023-10-30T13:12:50.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fd560640 1 -- 172.21.15.125:0/929073041 learned_addr learned my addr 172.21.15.125:0/929073041 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:50.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fe562640 1 --2- 172.21.15.125:0/929073041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53f814fd50 0x7f53f81531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:50.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fdd61640 1 --2- 172.21.15.125:0/929073041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53f8071250 0x7f53f8150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:50.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fd560640 1 -- 172.21.15.125:0/929073041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53f8071250 msgr2=0x7f53f8150bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:50.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fd560640 1 --2- 172.21.15.125:0/929073041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53f8071250 0x7f53f8150bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:50.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fd560640 1 -- 172.21.15.125:0/929073041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53f814fd50 msgr2=0x7f53f81531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:50.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fd560640 1 --2- 172.21.15.125:0/929073041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53f814fd50 0x7f53f81531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:50.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fd560640 1 -- 172.21.15.125:0/929073041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f40094d0 con 0x7f53f80cb550 2023-10-30T13:12:50.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fd560640 1 --2- 172.21.15.125:0/929073041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f80cb550 0x7f53f81f28a0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f53e800d830 tx=0x7f53e800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:50.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53ee7fc640 1 -- 172.21.15.125:0/929073041 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e8009660 con 0x7f53f80cb550 2023-10-30T13:12:50.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53ee7fc640 1 -- 172.21.15.125:0/929073041 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53e80097c0 con 0x7f53f80cb550 2023-10-30T13:12:50.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53ee7fc640 1 -- 172.21.15.125:0/929073041 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e8009b70 con 0x7f53f80cb550 2023-10-30T13:12:50.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fffc5640 1 -- 172.21.15.125:0/929073041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f8153720 con 0x7f53f80cb550 2023-10-30T13:12:50.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.886+0000 7f53fffc5640 1 -- 172.21.15.125:0/929073041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f81515f0 con 0x7f53f80cb550 2023-10-30T13:12:50.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.890+0000 7f53fffc5640 1 -- 172.21.15.125:0/929073041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f804e170 con 0x7f53f80cb550 2023-10-30T13:12:50.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.890+0000 7f53ee7fc640 1 -- 172.21.15.125:0/929073041 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f53e8014410 con 0x7f53f80cb550 2023-10-30T13:12:50.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.890+0000 7f53ee7fc640 1 --2- 172.21.15.125:0/929073041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f53dc0664a0 0x7f53dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:50.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.890+0000 7f53ee7fc640 1 -- 172.21.15.125:0/929073041 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7f53e8085070 con 0x7f53f80cb550 2023-10-30T13:12:50.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.890+0000 7f53fdd61640 1 --2- 172.21.15.125:0/929073041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f53dc0664a0 0x7f53dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:50.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.890+0000 7f53fdd61640 1 --2- 172.21.15.125:0/929073041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f53dc0664a0 0x7f53dc068960 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f53f40af000 tx=0x7f53f40a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:50.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:50.894+0000 7f53ee7fc640 1 -- 172.21.15.125:0/929073041 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f53e80b6090 con 0x7f53f80cb550 2023-10-30T13:12:51.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.074+0000 7f53fffc5640 1 -- 172.21.15.125:0/929073041 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53f80c7180 con 0x7f53f80cb550 2023-10-30T13:12:51.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.078+0000 7f53ee7fc640 1 -- 172.21.15.125:0/929073041 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5757 (secure 0 0 0) 0x7f53e8051bf0 con 0x7f53f80cb550 2023-10-30T13:12:51.081 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:12:51.081 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":205,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:12:50.713293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:51.082 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 205 2023-10-30T13:12:51.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.078+0000 7f53fffc5640 1 -- 172.21.15.125:0/929073041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f53dc0664a0 msgr2=0x7f53dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:51.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.078+0000 7f53fffc5640 1 --2- 172.21.15.125:0/929073041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f53dc0664a0 0x7f53dc068960 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f53f40af000 tx=0x7f53f40a7000 comp rx=0 tx=0).stop 2023-10-30T13:12:51.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.078+0000 7f53fffc5640 1 -- 172.21.15.125:0/929073041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f80cb550 msgr2=0x7f53f81f28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:51.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.078+0000 7f53fffc5640 1 --2- 172.21.15.125:0/929073041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f80cb550 0x7f53f81f28a0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f53e800d830 tx=0x7f53e800dcf0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.078+0000 7f53fffc5640 1 -- 172.21.15.125:0/929073041 shutdown_connections 2023-10-30T13:12:51.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.078+0000 7f53fffc5640 1 --2- 172.21.15.125:0/929073041 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f53dc0664a0 0x7f53dc068960 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.078+0000 7f53fffc5640 1 --2- 172.21.15.125:0/929073041 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f53f814fd50 0x7f53f81531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.078+0000 7f53fffc5640 1 --2- 172.21.15.125:0/929073041 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f53f80cb550 0x7f53f81f28a0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.078+0000 7f53fffc5640 1 --2- 172.21.15.125:0/929073041 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f53f8071250 0x7f53f8150bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.078+0000 7f53fffc5640 1 -- 172.21.15.125:0/929073041 >> 172.21.15.125:0/929073041 conn(0x7f53f806c6e0 msgr2=0x7f53f80ca4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:51.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.078+0000 7f53fffc5640 1 -- 172.21.15.125:0/929073041 shutdown_connections 2023-10-30T13:12:51.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.078+0000 7f53fffc5640 1 -- 172.21.15.125:0/929073041 wait complete. 2023-10-30T13:12:51.102 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c down, removed from mdsmap 2023-10-30T13:12:51.102 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:12:51.103 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:51.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8ec99d640 1 Processor -- start 2023-10-30T13:12:51.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8ec99d640 1 -- start start 2023-10-30T13:12:51.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8ec99d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff8e815d6f0 0x7ff8e815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:51.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8ec99d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8e80cf160 0x7ff8e80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:51.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8ec99d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8e80cb800 con 0x7ff8e81538a0 2023-10-30T13:12:51.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8ec99d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8e80cb330 con 0x7ff8e815d6f0 2023-10-30T13:12:51.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8ec99d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e80cad70 con 0x7ff8e80cf160 2023-10-30T13:12:51.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e659c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff8e81538a0 0x7ff8e80ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39522/0 (socket says 172.21.15.125:39522) 2023-10-30T13:12:51.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e659c640 1 -- 172.21.15.125:0/365553181 learned_addr learned my addr 172.21.15.125:0/365553181 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:12:51.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e6d9d640 1 --2- 172.21.15.125:0/365553181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8e80cf160 0x7ff8e80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:51.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e5d9b640 1 --2- 172.21.15.125:0/365553181 >> v2:172.21.15.136:3300/0 conn(0x7ff8e815d6f0 0x7ff8e815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:51.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 -- 172.21.15.125:0/365553181 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544833677 0 0) 0x7ff8e80cb800 con 0x7ff8e81538a0 2023-10-30T13:12:51.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 -- 172.21.15.125:0/365553181 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8d0003530 con 0x7ff8e81538a0 2023-10-30T13:12:51.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 -- 172.21.15.125:0/365553181 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1088348740 0 0) 0x7ff8d0003530 con 0x7ff8e81538a0 2023-10-30T13:12:51.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 -- 172.21.15.125:0/365553181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8e80cf160 msgr2=0x7ff8e80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:51.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 --2- 172.21.15.125:0/365553181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8e80cf160 0x7ff8e80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 -- 172.21.15.125:0/365553181 >> v2:172.21.15.136:3300/0 conn(0x7ff8e815d6f0 msgr2=0x7ff8e815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:51.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 --2- 172.21.15.125:0/365553181 >> v2:172.21.15.136:3300/0 conn(0x7ff8e815d6f0 0x7ff8e815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 -- 172.21.15.125:0/365553181 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e80cb050 con 0x7ff8e81538a0 2023-10-30T13:12:51.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 -- 172.21.15.125:0/365553181 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7ff8d8001680 con 0x7ff8e81538a0 2023-10-30T13:12:51.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 -- 172.21.15.125:0/365553181 >> v1:172.21.15.125:6789/0 conn(0x7ff8e81538a0 legacy=0x7ff8e80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:51.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 --2- 172.21.15.125:0/365553181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8d0003980 0x7ff8d0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:51.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 --2- 172.21.15.125:0/365553181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8d00047d0 0x7ff8d0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:51.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 --2- 172.21.15.125:0/365553181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8e80cf160 0x7ff8d0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:51.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 -- 172.21.15.125:0/365553181 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e80cb330 con 0x7ff8d00047d0 2023-10-30T13:12:51.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 -- 172.21.15.125:0/365553181 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e80cad70 con 0x7ff8d0003980 2023-10-30T13:12:51.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e559a640 1 -- 172.21.15.125:0/365553181 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d000b990 con 0x7ff8e80cf160 2023-10-30T13:12:51.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e5d9b640 1 --2- 172.21.15.125:0/365553181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8d00047d0 0x7ff8d0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:51.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e659c640 1 --2- 172.21.15.125:0/365553181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8d0003980 0x7ff8d0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:51.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e6d9d640 1 --2- 172.21.15.125:0/365553181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8e80cf160 0x7ff8d0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:51.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e5d9b640 1 -- 172.21.15.125:0/365553181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8e80cf160 msgr2=0x7ff8d0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:51.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e5d9b640 1 --2- 172.21.15.125:0/365553181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8e80cf160 0x7ff8d0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e5d9b640 1 -- 172.21.15.125:0/365553181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8d0003980 msgr2=0x7ff8d0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:51.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e5d9b640 1 --2- 172.21.15.125:0/365553181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8d0003980 0x7ff8d0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e5d9b640 1 -- 172.21.15.125:0/365553181 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d0003530 con 0x7ff8d00047d0 2023-10-30T13:12:51.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.238+0000 7ff8e6d9d640 1 --2- 172.21.15.125:0/365553181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8e80cf160 0x7ff8d0005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:12:51.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.242+0000 7ff8e5d9b640 1 --2- 172.21.15.125:0/365553181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8d00047d0 0x7ff8d0004c50 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009ae0 tx=0x7ff8dc01ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abba91d21c9060b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:51.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.242+0000 7ff8e559a640 1 -- 172.21.15.125:0/365553181 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc0a55a0 con 0x7ff8d00047d0 2023-10-30T13:12:51.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.242+0000 7ff8e559a640 1 -- 172.21.15.125:0/365553181 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8dc0a5700 con 0x7ff8d00047d0 2023-10-30T13:12:51.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.242+0000 7ff8e559a640 1 -- 172.21.15.125:0/365553181 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc0a5ab0 con 0x7ff8d00047d0 2023-10-30T13:12:51.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.242+0000 7ff8ec99d640 1 -- 172.21.15.125:0/365553181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8d00047d0 msgr2=0x7ff8d0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:51.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.242+0000 7ff8ec99d640 1 --2- 172.21.15.125:0/365553181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8d00047d0 0x7ff8d0004c50 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009ae0 tx=0x7ff8dc01ca90 comp rx=0 tx=0).stop 2023-10-30T13:12:51.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.242+0000 7ff8e6d9d640 1 -- 172.21.15.125:0/365553181 reap_dead start 2023-10-30T13:12:51.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.242+0000 7ff8ec99d640 1 -- 172.21.15.125:0/365553181 shutdown_connections 2023-10-30T13:12:51.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.242+0000 7ff8ec99d640 1 -- 172.21.15.125:0/365553181 >> 172.21.15.125:0/365553181 conn(0x7ff8e806bfe0 msgr2=0x7ff8e806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:51.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.242+0000 7ff8ec99d640 1 -- 172.21.15.125:0/365553181 shutdown_connections 2023-10-30T13:12:51.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.242+0000 7ff8ec99d640 1 -- 172.21.15.125:0/365553181 wait complete. 2023-10-30T13:12:51.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8ec99d640 1 Processor -- start 2023-10-30T13:12:51.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8ec99d640 1 -- start start 2023-10-30T13:12:51.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8ec99d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8d00047d0 0x7ff8e81e4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:51.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8ec99d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e80ce390 0x7ff8e8157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:51.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8ec99d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8e80ce7d0 0x7ff8e8158060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:51.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8ec99d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e8157690 con 0x7ff8e80ce390 2023-10-30T13:12:51.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8ec99d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e80c6fd0 con 0x7ff8d00047d0 2023-10-30T13:12:51.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8ec99d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8e8159670 con 0x7ff8e80ce7d0 2023-10-30T13:12:51.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8e5d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e80ce390 0x7ff8e8157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:51.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8e5d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e80ce390 0x7ff8e8157b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37036/0 (socket says 172.21.15.125:37036) 2023-10-30T13:12:51.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8e5d9b640 1 -- 172.21.15.125:0/1235430957 learned_addr learned my addr 172.21.15.125:0/1235430957 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:51.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8e659c640 1 --2- 172.21.15.125:0/1235430957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8d00047d0 0x7ff8e81e4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:51.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8e6d9d640 1 --2- 172.21.15.125:0/1235430957 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8e80ce7d0 0x7ff8e8158060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:51.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8e5d9b640 1 -- 172.21.15.125:0/1235430957 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8e80ce7d0 msgr2=0x7ff8e8158060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:51.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8e5d9b640 1 --2- 172.21.15.125:0/1235430957 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8e80ce7d0 0x7ff8e8158060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8e5d9b640 1 -- 172.21.15.125:0/1235430957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8d00047d0 msgr2=0x7ff8e81e4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:51.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8e5d9b640 1 --2- 172.21.15.125:0/1235430957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8d00047d0 0x7ff8e81e4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8e5d9b640 1 -- 172.21.15.125:0/1235430957 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e8158af0 con 0x7ff8e80ce390 2023-10-30T13:12:51.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8e5d9b640 1 --2- 172.21.15.125:0/1235430957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e80ce390 0x7ff8e8157b20 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009ae0 tx=0x7ff8d0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:51.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8d6ffd640 1 -- 172.21.15.125:0/1235430957 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc0be070 con 0x7ff8e80ce390 2023-10-30T13:12:51.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8d6ffd640 1 -- 172.21.15.125:0/1235430957 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8dc004020 con 0x7ff8e80ce390 2023-10-30T13:12:51.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8d6ffd640 1 -- 172.21.15.125:0/1235430957 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc004390 con 0x7ff8e80ce390 2023-10-30T13:12:51.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.246+0000 7ff8ec99d640 1 -- 172.21.15.125:0/1235430957 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8e8158dd0 con 0x7ff8e80ce390 2023-10-30T13:12:51.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.250+0000 7ff8ec99d640 1 -- 172.21.15.125:0/1235430957 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8e8159390 con 0x7ff8e80ce390 2023-10-30T13:12:51.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.250+0000 7ff8ec99d640 1 -- 172.21.15.125:0/1235430957 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e8157800 con 0x7ff8e80ce390 2023-10-30T13:12:51.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.250+0000 7ff8d6ffd640 1 -- 172.21.15.125:0/1235430957 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff8dc0b4040 con 0x7ff8e80ce390 2023-10-30T13:12:51.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.250+0000 7ff8d6ffd640 1 --2- 172.21.15.125:0/1235430957 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff8c80664a0 0x7ff8c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:51.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.250+0000 7ff8d6ffd640 1 -- 172.21.15.125:0/1235430957 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7ff8dc11e4f0 con 0x7ff8e80ce390 2023-10-30T13:12:51.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.250+0000 7ff8e659c640 1 --2- 172.21.15.125:0/1235430957 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff8c80664a0 0x7ff8c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:51.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.254+0000 7ff8e659c640 1 --2- 172.21.15.125:0/1235430957 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff8c80664a0 0x7ff8c8068960 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80059c0 tx=0x7ff8d8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:51.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.254+0000 7ff8d6ffd640 1 -- 172.21.15.125:0/1235430957 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff8dc0eb070 con 0x7ff8e80ce390 2023-10-30T13:12:51.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.434+0000 7ff8ec99d640 1 -- 172.21.15.125:0/1235430957 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8e804e170 con 0x7ff8e80ce390 2023-10-30T13:12:51.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.434+0000 7ff8d6ffd640 1 -- 172.21.15.125:0/1235430957 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5757 (secure 0 0 0) 0x7ff8dc0044f0 con 0x7ff8e80ce390 2023-10-30T13:12:51.440 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:12:51.440 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":205,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:12:50.713293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:51.441 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 205 2023-10-30T13:12:51.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.438+0000 7ff8ec99d640 1 -- 172.21.15.125:0/1235430957 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff8c80664a0 msgr2=0x7ff8c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:51.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.438+0000 7ff8ec99d640 1 --2- 172.21.15.125:0/1235430957 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff8c80664a0 0x7ff8c8068960 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7ff8d80059c0 tx=0x7ff8d8009040 comp rx=0 tx=0).stop 2023-10-30T13:12:51.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.438+0000 7ff8ec99d640 1 -- 172.21.15.125:0/1235430957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e80ce390 msgr2=0x7ff8e8157b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:51.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.438+0000 7ff8ec99d640 1 --2- 172.21.15.125:0/1235430957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e80ce390 0x7ff8e8157b20 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc009ae0 tx=0x7ff8d0005800 comp rx=0 tx=0).stop 2023-10-30T13:12:51.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.438+0000 7ff8ec99d640 1 -- 172.21.15.125:0/1235430957 shutdown_connections 2023-10-30T13:12:51.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.438+0000 7ff8ec99d640 1 --2- 172.21.15.125:0/1235430957 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff8c80664a0 0x7ff8c8068960 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.438+0000 7ff8ec99d640 1 --2- 172.21.15.125:0/1235430957 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8e80ce7d0 0x7ff8e8158060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.438+0000 7ff8ec99d640 1 --2- 172.21.15.125:0/1235430957 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8e80ce390 0x7ff8e8157b20 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.438+0000 7ff8ec99d640 1 --2- 172.21.15.125:0/1235430957 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8d00047d0 0x7ff8e81e4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:51.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.438+0000 7ff8ec99d640 1 -- 172.21.15.125:0/1235430957 >> 172.21.15.125:0/1235430957 conn(0x7ff8e806bfe0 msgr2=0x7ff8e80c4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:51.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.438+0000 7ff8ec99d640 1 -- 172.21.15.125:0/1235430957 shutdown_connections 2023-10-30T13:12:51.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:51.438+0000 7ff8ec99d640 1 -- 172.21.15.125:0/1235430957 wait complete. 2023-10-30T13:12:51.461 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 205, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 205, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_7511": { "addr": "172.21.15.136:6835/695089547", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 695089547, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 695089547, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7511, "incarnation": 205, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:replay", "state_seq": 1 }, "gid_8443": { "addr": "172.21.15.136:6839/3776186190", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3776186190, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3776186190, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 8443, "incarnation": 196, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 41 }, "gid_8529": { "addr": "172.21.15.125:6836/3950044434", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3950044434, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3950044434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 8529, "incarnation": 179, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 79, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:12:50.713293+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8529, "mds_1": 7511, "mds_2": 8443 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:12:53.463 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:53.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.602+0000 7feb0f232640 1 Processor -- start 2023-10-30T13:12:53.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feb0f232640 1 -- start start 2023-10-30T13:12:53.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feb0f232640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7feb081538a0 0x7feb0815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:53.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feb0f232640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feb080ca0e0 0x7feb0815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:53.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feb0f232640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb080cb800 con 0x7feb080cf160 2023-10-30T13:12:53.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feb0f232640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0815dc30 con 0x7feb081538a0 2023-10-30T13:12:53.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feb0f232640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb080cb330 con 0x7feb080ca0e0 2023-10-30T13:12:53.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feb0d7cf640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7feb080cf160 0x7feb080cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39528/0 (socket says 172.21.15.125:39528) 2023-10-30T13:12:53.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feb0d7cf640 1 -- 172.21.15.125:0/812428080 learned_addr learned my addr 172.21.15.125:0/812428080 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:12:53.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feafffff640 1 --2- 172.21.15.125:0/812428080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feb080ca0e0 0x7feb0815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:53.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feb0cfce640 1 --2- 172.21.15.125:0/812428080 >> v2:172.21.15.136:3300/0 conn(0x7feb081538a0 0x7feb0815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:53.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 -- 172.21.15.125:0/812428080 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426160300 0 0) 0x7feb080cb800 con 0x7feb080cf160 2023-10-30T13:12:53.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 -- 172.21.15.125:0/812428080 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf0003530 con 0x7feb080cf160 2023-10-30T13:12:53.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 -- 172.21.15.125:0/812428080 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3215572081 0 0) 0x7feaf0003530 con 0x7feb080cf160 2023-10-30T13:12:53.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 -- 172.21.15.125:0/812428080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feb080ca0e0 msgr2=0x7feb0815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:53.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 --2- 172.21.15.125:0/812428080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feb080ca0e0 0x7feb0815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:53.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 -- 172.21.15.125:0/812428080 >> v2:172.21.15.136:3300/0 conn(0x7feb081538a0 msgr2=0x7feb0815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:53.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 --2- 172.21.15.125:0/812428080 >> v2:172.21.15.136:3300/0 conn(0x7feb081538a0 0x7feb0815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:53.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 -- 172.21.15.125:0/812428080 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb080cad70 con 0x7feb080cf160 2023-10-30T13:12:53.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 -- 172.21.15.125:0/812428080 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7feb00001a00 con 0x7feb080cf160 2023-10-30T13:12:53.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 -- 172.21.15.125:0/812428080 >> v1:172.21.15.125:6789/0 conn(0x7feb080cf160 legacy=0x7feb080cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:53.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 --2- 172.21.15.125:0/812428080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaf0003980 0x7feaf0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:53.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 --2- 172.21.15.125:0/812428080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feaf00047d0 0x7feaf0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:53.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 --2- 172.21.15.125:0/812428080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feb080ca0e0 0x7feaf0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:53.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 -- 172.21.15.125:0/812428080 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0815dc30 con 0x7feaf00047d0 2023-10-30T13:12:53.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 -- 172.21.15.125:0/812428080 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb080cb330 con 0x7feaf0003980 2023-10-30T13:12:53.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feaff7fe640 1 -- 172.21.15.125:0/812428080 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf000b9c0 con 0x7feb080ca0e0 2023-10-30T13:12:53.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feafffff640 1 --2- 172.21.15.125:0/812428080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feaf00047d0 0x7feaf0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:53.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feb0cfce640 1 --2- 172.21.15.125:0/812428080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaf0003980 0x7feaf0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:53.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feb0d7cf640 1 --2- 172.21.15.125:0/812428080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feb080ca0e0 0x7feaf0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:53.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feafffff640 1 -- 172.21.15.125:0/812428080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feb080ca0e0 msgr2=0x7feaf0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:53.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feafffff640 1 --2- 172.21.15.125:0/812428080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feb080ca0e0 0x7feaf0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:53.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feafffff640 1 -- 172.21.15.125:0/812428080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaf0003980 msgr2=0x7feaf0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:53.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feafffff640 1 --2- 172.21.15.125:0/812428080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feaf0003980 0x7feaf0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:53.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.606+0000 7feafffff640 1 -- 172.21.15.125:0/812428080 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf0003530 con 0x7feaf00047d0 2023-10-30T13:12:53.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.610+0000 7feafffff640 1 --2- 172.21.15.125:0/812428080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feaf00047d0 0x7feaf0004c50 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009ae0 tx=0x7feaf801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbfdfaf18ed79074 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:53.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.610+0000 7feaff7fe640 1 -- 172.21.15.125:0/812428080 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf80a5560 con 0x7feaf00047d0 2023-10-30T13:12:53.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.610+0000 7feaff7fe640 1 -- 172.21.15.125:0/812428080 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf80a56c0 con 0x7feaf00047d0 2023-10-30T13:12:53.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.610+0000 7feaff7fe640 1 -- 172.21.15.125:0/812428080 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf80a5a70 con 0x7feaf00047d0 2023-10-30T13:12:53.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.610+0000 7feb0f232640 1 -- 172.21.15.125:0/812428080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feaf00047d0 msgr2=0x7feaf0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:53.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.610+0000 7feb0f232640 1 --2- 172.21.15.125:0/812428080 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feaf00047d0 0x7feaf0004c50 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009ae0 tx=0x7feaf801ca50 comp rx=0 tx=0).stop 2023-10-30T13:12:53.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.610+0000 7feb0d7cf640 1 -- 172.21.15.125:0/812428080 reap_dead start 2023-10-30T13:12:53.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.610+0000 7feb0f232640 1 -- 172.21.15.125:0/812428080 shutdown_connections 2023-10-30T13:12:53.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.610+0000 7feb0f232640 1 -- 172.21.15.125:0/812428080 >> 172.21.15.125:0/812428080 conn(0x7feb0806bfe0 msgr2=0x7feb0806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:53.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.610+0000 7feb0f232640 1 -- 172.21.15.125:0/812428080 shutdown_connections 2023-10-30T13:12:53.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.610+0000 7feb0f232640 1 -- 172.21.15.125:0/812428080 wait complete. 2023-10-30T13:12:53.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feb0f232640 1 Processor -- start 2023-10-30T13:12:53.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feb0f232640 1 -- start start 2023-10-30T13:12:53.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feb0f232640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feaf00047d0 0x7feb081448c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:53.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feb0f232640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0815d230 0x7feb0815a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:53.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feb0f232640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb0815adf0 0x7feb08142ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:53.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feb0f232640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb080c9710 con 0x7feb0815d230 2023-10-30T13:12:53.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feb0f232640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb080c9c00 con 0x7feb0815adf0 2023-10-30T13:12:53.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feb0f232640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb081589d0 con 0x7feaf00047d0 2023-10-30T13:12:53.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feafffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0815d230 0x7feb0815a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:53.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feafffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0815d230 0x7feb0815a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37044/0 (socket says 172.21.15.125:37044) 2023-10-30T13:12:53.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feafffff640 1 -- 172.21.15.125:0/1224533242 learned_addr learned my addr 172.21.15.125:0/1224533242 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:53.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feb0cfce640 1 --2- 172.21.15.125:0/1224533242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feaf00047d0 0x7feb081448c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:53.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feb0d7cf640 1 --2- 172.21.15.125:0/1224533242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb0815adf0 0x7feb08142ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:53.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feafffff640 1 -- 172.21.15.125:0/1224533242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feaf00047d0 msgr2=0x7feb081448c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:53.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feafffff640 1 --2- 172.21.15.125:0/1224533242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feaf00047d0 0x7feb081448c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:53.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feafffff640 1 -- 172.21.15.125:0/1224533242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb0815adf0 msgr2=0x7feb08142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:53.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feafffff640 1 --2- 172.21.15.125:0/1224533242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb0815adf0 0x7feb08142ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:53.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feafffff640 1 -- 172.21.15.125:0/1224533242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb080c9940 con 0x7feb0815d230 2023-10-30T13:12:53.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feafffff640 1 --2- 172.21.15.125:0/1224533242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0815d230 0x7feb0815a8b0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009ae0 tx=0x7feaf0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:53.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feafd7fa640 1 -- 172.21.15.125:0/1224533242 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf80bc070 con 0x7feb0815d230 2023-10-30T13:12:53.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feafd7fa640 1 -- 172.21.15.125:0/1224533242 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf800f0c0 con 0x7feb0815d230 2023-10-30T13:12:53.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feafd7fa640 1 -- 172.21.15.125:0/1224533242 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf8004220 con 0x7feb0815d230 2023-10-30T13:12:53.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feb0f232640 1 -- 172.21.15.125:0/1224533242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb08158e60 con 0x7feb0815d230 2023-10-30T13:12:53.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.614+0000 7feb0f232640 1 -- 172.21.15.125:0/1224533242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb08158380 con 0x7feb0815d230 2023-10-30T13:12:53.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.618+0000 7feb0f232640 1 -- 172.21.15.125:0/1224533242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0804e170 con 0x7feb0815d230 2023-10-30T13:12:53.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.618+0000 7feafd7fa640 1 -- 172.21.15.125:0/1224533242 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7feaf80b4040 con 0x7feb0815d230 2023-10-30T13:12:53.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.618+0000 7feafd7fa640 1 --2- 172.21.15.125:0/1224533242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feaf40664a0 0x7feaf4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:53.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.618+0000 7feafd7fa640 1 -- 172.21.15.125:0/1224533242 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7feaf811e770 con 0x7feb0815d230 2023-10-30T13:12:53.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.618+0000 7feb0cfce640 1 --2- 172.21.15.125:0/1224533242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feaf40664a0 0x7feaf4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:53.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.618+0000 7feb0cfce640 1 --2- 172.21.15.125:0/1224533242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feaf40664a0 0x7feaf4068960 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7feb04009ab0 tx=0x7feb04008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:53.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.622+0000 7feafd7fa640 1 -- 172.21.15.125:0/1224533242 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7feaf80eb410 con 0x7feb0815d230 2023-10-30T13:12:53.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.806+0000 7feb0f232640 1 -- 172.21.15.125:0/1224533242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb08158b10 con 0x7feb0815d230 2023-10-30T13:12:53.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.806+0000 7feafd7fa640 1 -- 172.21.15.125:0/1224533242 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5757 (secure 0 0 0) 0x7feaf80eea30 con 0x7feb0815d230 2023-10-30T13:12:53.810 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:12:53.810 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":205,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:12:50.713293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:53.810 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 205 2023-10-30T13:12:53.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.806+0000 7feb0f232640 1 -- 172.21.15.125:0/1224533242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feaf40664a0 msgr2=0x7feaf4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:53.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.806+0000 7feb0f232640 1 --2- 172.21.15.125:0/1224533242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feaf40664a0 0x7feaf4068960 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7feb04009ab0 tx=0x7feb04008040 comp rx=0 tx=0).stop 2023-10-30T13:12:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.806+0000 7feb0f232640 1 -- 172.21.15.125:0/1224533242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0815d230 msgr2=0x7feb0815a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.806+0000 7feb0f232640 1 --2- 172.21.15.125:0/1224533242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0815d230 0x7feb0815a8b0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7feaf8009ae0 tx=0x7feaf0005800 comp rx=0 tx=0).stop 2023-10-30T13:12:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.806+0000 7feb0f232640 1 -- 172.21.15.125:0/1224533242 shutdown_connections 2023-10-30T13:12:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.806+0000 7feb0f232640 1 --2- 172.21.15.125:0/1224533242 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7feaf40664a0 0x7feaf4068960 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.806+0000 7feb0f232640 1 --2- 172.21.15.125:0/1224533242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7feb0815adf0 0x7feb08142ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.806+0000 7feb0f232640 1 --2- 172.21.15.125:0/1224533242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7feb0815d230 0x7feb0815a8b0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.806+0000 7feb0f232640 1 --2- 172.21.15.125:0/1224533242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7feaf00047d0 0x7feb081448c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.806+0000 7feb0f232640 1 -- 172.21.15.125:0/1224533242 >> 172.21.15.125:0/1224533242 conn(0x7feb0806bfe0 msgr2=0x7feb081526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.806+0000 7feb0f232640 1 -- 172.21.15.125:0/1224533242 shutdown_connections 2023-10-30T13:12:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:53.806+0000 7feb0f232640 1 -- 172.21.15.125:0/1224533242 wait complete. 2023-10-30T13:12:53.830 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:12:55.831 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:55.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.970+0000 7f86bb93e640 1 Processor -- start 2023-10-30T13:12:55.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.970+0000 7f86bb93e640 1 -- start start 2023-10-30T13:12:55.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.970+0000 7f86bb93e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f86b40c6540 0x7f86b415e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:55.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.970+0000 7f86bb93e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b40c5810 con 0x7f86b4070740 2023-10-30T13:12:55.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.970+0000 7f86bb93e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b40c69b0 con 0x7f86b41580e0 2023-10-30T13:12:55.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.970+0000 7f86bb93e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86b406acb0 con 0x7f86b40c6540 2023-10-30T13:12:55.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.970+0000 7f86b8ed9640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f86b4070740 0x7f86b406a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39532/0 (socket says 172.21.15.125:39532) 2023-10-30T13:12:55.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.970+0000 7f86b8ed9640 1 -- 172.21.15.125:0/3322518011 learned_addr learned my addr 172.21.15.125:0/3322518011 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:12:55.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.970+0000 7f86b9edb640 1 --2- 172.21.15.125:0/3322518011 >> v2:172.21.15.136:3300/0 conn(0x7f86b40c6540 0x7f86b415e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:55.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.970+0000 7f86abfff640 1 -- 172.21.15.125:0/3322518011 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085418289 0 0) 0x7f86b40c5810 con 0x7f86b4070740 2023-10-30T13:12:55.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.970+0000 7f86abfff640 1 -- 172.21.15.125:0/3322518011 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a0003530 con 0x7f86b4070740 2023-10-30T13:12:55.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.970+0000 7f86b9edb640 1 -- 172.21.15.125:0/3322518011 >> v1:172.21.15.136:6789/0 conn(0x7f86b41580e0 legacy=0x7f86b4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:55.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.970+0000 7f86b9edb640 1 -- 172.21.15.125:0/3322518011 >> v1:172.21.15.125:6789/0 conn(0x7f86b4070740 legacy=0x7f86b406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:55.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.970+0000 7f86b9edb640 1 -- 172.21.15.125:0/3322518011 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b40c6bb0 con 0x7f86b40c6540 2023-10-30T13:12:55.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.974+0000 7f86b9edb640 1 --2- 172.21.15.125:0/3322518011 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86b40c6540 0x7f86b415e700 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f86ac009d90 tx=0x7f86ac01c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7ddc446018e4e35 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:55.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.974+0000 7f86abfff640 1 -- 172.21.15.125:0/3322518011 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac0a5920 con 0x7f86b40c6540 2023-10-30T13:12:55.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.974+0000 7f86abfff640 1 -- 172.21.15.125:0/3322518011 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86ac0a5a80 con 0x7f86b40c6540 2023-10-30T13:12:55.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.974+0000 7f86abfff640 1 -- 172.21.15.125:0/3322518011 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86ac0a5e10 con 0x7f86b40c6540 2023-10-30T13:12:55.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.974+0000 7f86bb93e640 1 -- 172.21.15.125:0/3322518011 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86b40c6540 msgr2=0x7f86b415e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:55.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.974+0000 7f86bb93e640 1 --2- 172.21.15.125:0/3322518011 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86b40c6540 0x7f86b415e700 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f86ac009d90 tx=0x7f86ac01c950 comp rx=0 tx=0).stop 2023-10-30T13:12:55.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.974+0000 7f86bb93e640 1 -- 172.21.15.125:0/3322518011 shutdown_connections 2023-10-30T13:12:55.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.974+0000 7f86bb93e640 1 --2- 172.21.15.125:0/3322518011 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86b40c6540 0x7f86b415e700 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:55.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.974+0000 7f86bb93e640 1 -- 172.21.15.125:0/3322518011 >> 172.21.15.125:0/3322518011 conn(0x7f86b40c7910 msgr2=0x7f86b40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:55.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.974+0000 7f86bb93e640 1 -- 172.21.15.125:0/3322518011 shutdown_connections 2023-10-30T13:12:55.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.974+0000 7f86bb93e640 1 -- 172.21.15.125:0/3322518011 wait complete. 2023-10-30T13:12:55.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86bb93e640 1 Processor -- start 2023-10-30T13:12:55.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86bb93e640 1 -- start start 2023-10-30T13:12:55.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86bb93e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86b4070740 0x7f86b413d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:55.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86bb93e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f86b40c6540 0x7f86b4152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:55.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86bb93e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f86b41580e0 0x7f86b4153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:55.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86bb93e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b41faad0 con 0x7f86b40c6540 2023-10-30T13:12:55.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86bb93e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b40c4e00 con 0x7f86b4070740 2023-10-30T13:12:55.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86bb93e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b41f9e50 con 0x7f86b41580e0 2023-10-30T13:12:55.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86b8ed9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f86b40c6540 0x7f86b4152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:55.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86b8ed9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f86b40c6540 0x7f86b4152be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37058/0 (socket says 172.21.15.125:37058) 2023-10-30T13:12:55.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86b8ed9640 1 -- 172.21.15.125:0/2388443462 learned_addr learned my addr 172.21.15.125:0/2388443462 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:55.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86b96da640 1 --2- 172.21.15.125:0/2388443462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86b4070740 0x7f86b413d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:55.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86b9edb640 1 --2- 172.21.15.125:0/2388443462 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f86b41580e0 0x7f86b4153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:55.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86b8ed9640 1 -- 172.21.15.125:0/2388443462 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f86b41580e0 msgr2=0x7f86b4153120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:55.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86b8ed9640 1 --2- 172.21.15.125:0/2388443462 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f86b41580e0 0x7f86b4153120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:55.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86b8ed9640 1 -- 172.21.15.125:0/2388443462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86b4070740 msgr2=0x7f86b413d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:55.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86b8ed9640 1 --2- 172.21.15.125:0/2388443462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86b4070740 0x7f86b413d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:55.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86b8ed9640 1 -- 172.21.15.125:0/2388443462 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac009a70 con 0x7f86b40c6540 2023-10-30T13:12:55.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86b8ed9640 1 --2- 172.21.15.125:0/2388443462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f86b40c6540 0x7f86b4152be0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f86a400c4d0 tx=0x7f86a400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:55.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86a9ffb640 1 -- 172.21.15.125:0/2388443462 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a4005cf0 con 0x7f86b40c6540 2023-10-30T13:12:55.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86a9ffb640 1 -- 172.21.15.125:0/2388443462 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a4005e50 con 0x7f86b40c6540 2023-10-30T13:12:55.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86a9ffb640 1 -- 172.21.15.125:0/2388443462 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86a4007270 con 0x7f86b40c6540 2023-10-30T13:12:55.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.978+0000 7f86bb93e640 1 -- 172.21.15.125:0/2388443462 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b41fa130 con 0x7f86b40c6540 2023-10-30T13:12:55.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.982+0000 7f86bb93e640 1 -- 172.21.15.125:0/2388443462 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b41fa6f0 con 0x7f86b40c6540 2023-10-30T13:12:55.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.982+0000 7f86bb93e640 1 -- 172.21.15.125:0/2388443462 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b41fac10 con 0x7f86b40c6540 2023-10-30T13:12:55.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.982+0000 7f86a9ffb640 1 -- 172.21.15.125:0/2388443462 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f86a40073d0 con 0x7f86b40c6540 2023-10-30T13:12:55.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.982+0000 7f86a9ffb640 1 --2- 172.21.15.125:0/2388443462 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f86980664d0 0x7f8698068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:55.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.982+0000 7f86a9ffb640 1 -- 172.21.15.125:0/2388443462 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7f86a4083bf0 con 0x7f86b40c6540 2023-10-30T13:12:55.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.982+0000 7f86b96da640 1 --2- 172.21.15.125:0/2388443462 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f86980664d0 0x7f8698068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:55.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.986+0000 7f86b96da640 1 --2- 172.21.15.125:0/2388443462 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f86980664d0 0x7f8698068990 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f86b0001770 tx=0x7f86b0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:55.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:55.986+0000 7f86a9ffb640 1 -- 172.21.15.125:0/2388443462 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f86b41fac10 con 0x7f86b40c6540 2023-10-30T13:12:56.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:56.170+0000 7f86bb93e640 1 -- 172.21.15.125:0/2388443462 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86b41fac10 con 0x7f86b40c6540 2023-10-30T13:12:56.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:56.170+0000 7f86a9ffb640 1 -- 172.21.15.125:0/2388443462 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+5757 (secure 0 0 0) 0x7f86b41fac10 con 0x7f86b40c6540 2023-10-30T13:12:56.174 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:12:56.175 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":205,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:12:50.713293+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:56.175 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 205 2023-10-30T13:12:56.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:56.170+0000 7f86bb93e640 1 -- 172.21.15.125:0/2388443462 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f86980664d0 msgr2=0x7f8698068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:56.170+0000 7f86bb93e640 1 --2- 172.21.15.125:0/2388443462 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f86980664d0 0x7f8698068990 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f86b0001770 tx=0x7f86b0009040 comp rx=0 tx=0).stop 2023-10-30T13:12:56.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:56.170+0000 7f86bb93e640 1 -- 172.21.15.125:0/2388443462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f86b40c6540 msgr2=0x7f86b4152be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:56.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:56.170+0000 7f86bb93e640 1 --2- 172.21.15.125:0/2388443462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f86b40c6540 0x7f86b4152be0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f86a400c4d0 tx=0x7f86a400c990 comp rx=0 tx=0).stop 2023-10-30T13:12:56.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:56.170+0000 7f86bb93e640 1 -- 172.21.15.125:0/2388443462 shutdown_connections 2023-10-30T13:12:56.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:56.170+0000 7f86bb93e640 1 --2- 172.21.15.125:0/2388443462 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f86980664d0 0x7f8698068990 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:56.170+0000 7f86bb93e640 1 --2- 172.21.15.125:0/2388443462 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f86b41580e0 0x7f86b4153120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:56.170+0000 7f86bb93e640 1 --2- 172.21.15.125:0/2388443462 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f86b40c6540 0x7f86b4152be0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:56.170+0000 7f86bb93e640 1 --2- 172.21.15.125:0/2388443462 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f86b4070740 0x7f86b413d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:56.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:56.170+0000 7f86bb93e640 1 -- 172.21.15.125:0/2388443462 >> 172.21.15.125:0/2388443462 conn(0x7f86b40c7910 msgr2=0x7f86b415dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:56.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:56.170+0000 7f86bb93e640 1 -- 172.21.15.125:0/2388443462 shutdown_connections 2023-10-30T13:12:56.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:56.170+0000 7f86bb93e640 1 -- 172.21.15.125:0/2388443462 wait complete. 2023-10-30T13:12:56.197 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:12:58.199 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:12:58.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9ab3a1640 1 Processor -- start 2023-10-30T13:12:58.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9ab3a1640 1 -- start start 2023-10-30T13:12:58.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9ab3a1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb9a41538a0 0x7fb9a415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:58.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9ab3a1640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9a40ca0e0 0x7fb9a415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:58.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9ab3a1640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9a40cb800 con 0x7fb9a40cf160 2023-10-30T13:12:58.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9ab3a1640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9a415dc30 con 0x7fb9a41538a0 2023-10-30T13:12:58.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9ab3a1640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a40cb330 con 0x7fb9a40ca0e0 2023-10-30T13:12:58.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9a993e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb9a40cf160 0x7fb9a40cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39542/0 (socket says 172.21.15.125:39542) 2023-10-30T13:12:58.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9a993e640 1 -- 172.21.15.125:0/2659516345 learned_addr learned my addr 172.21.15.125:0/2659516345 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:12:58.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9a913d640 1 --2- 172.21.15.125:0/2659516345 >> v2:172.21.15.136:3300/0 conn(0x7fb9a41538a0 0x7fb9a415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:58.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9a893c640 1 --2- 172.21.15.125:0/2659516345 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9a40ca0e0 0x7fb9a415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:58.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 -- 172.21.15.125:0/2659516345 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681099574 0 0) 0x7fb9a40cb800 con 0x7fb9a40cf160 2023-10-30T13:12:58.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 -- 172.21.15.125:0/2659516345 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb990003530 con 0x7fb9a40cf160 2023-10-30T13:12:58.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 -- 172.21.15.125:0/2659516345 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 582916630 0 0) 0x7fb990003530 con 0x7fb9a40cf160 2023-10-30T13:12:58.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 -- 172.21.15.125:0/2659516345 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9a40ca0e0 msgr2=0x7fb9a415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:58.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 --2- 172.21.15.125:0/2659516345 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9a40ca0e0 0x7fb9a415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:58.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 -- 172.21.15.125:0/2659516345 >> v2:172.21.15.136:3300/0 conn(0x7fb9a41538a0 msgr2=0x7fb9a415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:58.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 --2- 172.21.15.125:0/2659516345 >> v2:172.21.15.136:3300/0 conn(0x7fb9a41538a0 0x7fb9a415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:58.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 -- 172.21.15.125:0/2659516345 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a40cad70 con 0x7fb9a40cf160 2023-10-30T13:12:58.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 -- 172.21.15.125:0/2659516345 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fb99c001a00 con 0x7fb9a40cf160 2023-10-30T13:12:58.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 -- 172.21.15.125:0/2659516345 >> v1:172.21.15.125:6789/0 conn(0x7fb9a40cf160 legacy=0x7fb9a40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:58.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 --2- 172.21.15.125:0/2659516345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb990003980 0x7fb990003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:58.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 --2- 172.21.15.125:0/2659516345 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9a40ca0e0 0x7fb9900047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:58.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 --2- 172.21.15.125:0/2659516345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb990004d10 0x7fb990005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:58.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 -- 172.21.15.125:0/2659516345 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a415dc30 con 0x7fb990004d10 2023-10-30T13:12:58.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 -- 172.21.15.125:0/2659516345 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a40cb330 con 0x7fb990003980 2023-10-30T13:12:58.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb99bfff640 1 -- 172.21.15.125:0/2659516345 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb99000b990 con 0x7fb9a40ca0e0 2023-10-30T13:12:58.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9a993e640 1 --2- 172.21.15.125:0/2659516345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb990004d10 0x7fb990005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:58.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9a913d640 1 --2- 172.21.15.125:0/2659516345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb990003980 0x7fb990003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:58.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9a893c640 1 --2- 172.21.15.125:0/2659516345 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9a40ca0e0 0x7fb9900047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:58.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9a993e640 1 -- 172.21.15.125:0/2659516345 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9a40ca0e0 msgr2=0x7fb9900047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:58.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9a993e640 1 --2- 172.21.15.125:0/2659516345 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb9a40ca0e0 0x7fb9900047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:58.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9a993e640 1 -- 172.21.15.125:0/2659516345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb990003980 msgr2=0x7fb990003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:12:58.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9a993e640 1 --2- 172.21.15.125:0/2659516345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb990003980 0x7fb990003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:58.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9a993e640 1 -- 172.21.15.125:0/2659516345 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb990003530 con 0x7fb990004d10 2023-10-30T13:12:58.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.342+0000 7fb9a913d640 1 --2- 172.21.15.125:0/2659516345 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb990003980 0x7fb990003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:12:58.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.346+0000 7fb9a993e640 1 --2- 172.21.15.125:0/2659516345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb990004d10 0x7fb990005190 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fb99c00a860 tx=0x7fb99c01d360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c577a68f511a49 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:58.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.346+0000 7fb99bfff640 1 -- 172.21.15.125:0/2659516345 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99c0ad630 con 0x7fb990004d10 2023-10-30T13:12:58.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.346+0000 7fb99bfff640 1 -- 172.21.15.125:0/2659516345 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99c0a8070 con 0x7fb990004d10 2023-10-30T13:12:58.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.346+0000 7fb99bfff640 1 -- 172.21.15.125:0/2659516345 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99c0ad9f0 con 0x7fb990004d10 2023-10-30T13:12:58.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.346+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/2659516345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb990004d10 msgr2=0x7fb990005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:58.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.346+0000 7fb9ab3a1640 1 --2- 172.21.15.125:0/2659516345 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb990004d10 0x7fb990005190 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7fb99c00a860 tx=0x7fb99c01d360 comp rx=0 tx=0).stop 2023-10-30T13:12:58.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.346+0000 7fb9a993e640 1 -- 172.21.15.125:0/2659516345 reap_dead start 2023-10-30T13:12:58.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.346+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/2659516345 shutdown_connections 2023-10-30T13:12:58.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.346+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/2659516345 >> 172.21.15.125:0/2659516345 conn(0x7fb9a406bfe0 msgr2=0x7fb9a406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:58.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.346+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/2659516345 shutdown_connections 2023-10-30T13:12:58.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.346+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/2659516345 wait complete. 2023-10-30T13:12:58.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9ab3a1640 1 Processor -- start 2023-10-30T13:12:58.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9ab3a1640 1 -- start start 2023-10-30T13:12:58.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9ab3a1640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb990004d10 0x7fb9a4145ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:58.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9ab3a1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9a41e4b40 0x7fb9a41e34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:58.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9ab3a1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9a41e4fc0 0x7fb9a41e3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:58.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9ab3a1640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a41595e0 con 0x7fb9a41e4fc0 2023-10-30T13:12:58.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9ab3a1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a41f33f0 con 0x7fb9a41e4b40 2023-10-30T13:12:58.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9ab3a1640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a4159300 con 0x7fb990004d10 2023-10-30T13:12:58.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9a993e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9a41e4fc0 0x7fb9a41e3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:58.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9a993e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9a41e4fc0 0x7fb9a41e3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37082/0 (socket says 172.21.15.125:37082) 2023-10-30T13:12:58.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9a993e640 1 -- 172.21.15.125:0/934206076 learned_addr learned my addr 172.21.15.125:0/934206076 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:12:58.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9a913d640 1 --2- 172.21.15.125:0/934206076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb990004d10 0x7fb9a4145ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:58.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9a893c640 1 --2- 172.21.15.125:0/934206076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9a41e4b40 0x7fb9a41e34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:58.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9a993e640 1 -- 172.21.15.125:0/934206076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb990004d10 msgr2=0x7fb9a4145ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:58.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9a993e640 1 --2- 172.21.15.125:0/934206076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb990004d10 0x7fb9a4145ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:58.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9a993e640 1 -- 172.21.15.125:0/934206076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9a41e4b40 msgr2=0x7fb9a41e34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:58.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9a993e640 1 --2- 172.21.15.125:0/934206076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9a41e4b40 0x7fb9a41e34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:58.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9a993e640 1 -- 172.21.15.125:0/934206076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a41f3110 con 0x7fb9a41e4fc0 2023-10-30T13:12:58.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9a993e640 1 --2- 172.21.15.125:0/934206076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9a41e4fc0 0x7fb9a41e3a30 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fb99c01cdd0 tx=0x7fb990005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:58.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb999ffb640 1 -- 172.21.15.125:0/934206076 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99c0017c0 con 0x7fb9a41e4fc0 2023-10-30T13:12:58.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb999ffb640 1 -- 172.21.15.125:0/934206076 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb99c001920 con 0x7fb9a41e4fc0 2023-10-30T13:12:58.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb999ffb640 1 -- 172.21.15.125:0/934206076 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb99c004d70 con 0x7fb9a41e4fc0 2023-10-30T13:12:58.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/934206076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a4157b70 con 0x7fb9a41e4fc0 2023-10-30T13:12:58.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.350+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/934206076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a40c45c0 con 0x7fb9a41e4fc0 2023-10-30T13:12:58.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.354+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/934206076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a404e170 con 0x7fb9a41e4fc0 2023-10-30T13:12:58.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.354+0000 7fb999ffb640 1 -- 172.21.15.125:0/934206076 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb99c0aa030 con 0x7fb9a41e4fc0 2023-10-30T13:12:58.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.354+0000 7fb999ffb640 1 --2- 172.21.15.125:0/934206076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9880664a0 0x7fb988068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:12:58.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.354+0000 7fb999ffb640 1 -- 172.21.15.125:0/934206076 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7fb99c11e4b0 con 0x7fb9a41e4fc0 2023-10-30T13:12:58.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.354+0000 7fb9a913d640 1 --2- 172.21.15.125:0/934206076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9880664a0 0x7fb988068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:12:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.354+0000 7fb9a913d640 1 --2- 172.21.15.125:0/934206076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9880664a0 0x7fb988068960 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0009a30 tx=0x7fb9a0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:12:58.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.358+0000 7fb999ffb640 1 -- 172.21.15.125:0/934206076 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb99c0eb0c0 con 0x7fb9a41e4fc0 2023-10-30T13:12:58.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.538+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/934206076 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9a414d930 con 0x7fb9a41e4fc0 2023-10-30T13:12:58.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.538+0000 7fb999ffb640 1 -- 172.21.15.125:0/934206076 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+5759 (secure 0 0 0) 0x7fb99c0ee6e0 con 0x7fb9a41e4fc0 2023-10-30T13:12:58.543 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:12:58.543 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":206,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:12:56.602354+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:resolve","state_seq":16,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:12:58.544 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 206 2023-10-30T13:12:58.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.542+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/934206076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9880664a0 msgr2=0x7fb988068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:58.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.542+0000 7fb9ab3a1640 1 --2- 172.21.15.125:0/934206076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9880664a0 0x7fb988068960 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0009a30 tx=0x7fb9a0008040 comp rx=0 tx=0).stop 2023-10-30T13:12:58.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.542+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/934206076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9a41e4fc0 msgr2=0x7fb9a41e3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:12:58.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.542+0000 7fb9ab3a1640 1 --2- 172.21.15.125:0/934206076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9a41e4fc0 0x7fb9a41e3a30 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fb99c01cdd0 tx=0x7fb990005800 comp rx=0 tx=0).stop 2023-10-30T13:12:58.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.542+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/934206076 shutdown_connections 2023-10-30T13:12:58.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.542+0000 7fb9ab3a1640 1 --2- 172.21.15.125:0/934206076 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb9880664a0 0x7fb988068960 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:58.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.542+0000 7fb9ab3a1640 1 --2- 172.21.15.125:0/934206076 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb9a41e4fc0 0x7fb9a41e3a30 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:58.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.542+0000 7fb9ab3a1640 1 --2- 172.21.15.125:0/934206076 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb9a41e4b40 0x7fb9a41e34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:58.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.542+0000 7fb9ab3a1640 1 --2- 172.21.15.125:0/934206076 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb990004d10 0x7fb9a4145ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:12:58.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.542+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/934206076 >> 172.21.15.125:0/934206076 conn(0x7fb9a406bfe0 msgr2=0x7fb9a4152410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:12:58.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.542+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/934206076 shutdown_connections 2023-10-30T13:12:58.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:12:58.542+0000 7fb9ab3a1640 1 -- 172.21.15.125:0/934206076 wait complete. 2023-10-30T13:12:58.564 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:13:00.566 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:13:00.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.702+0000 7fefc4794640 1 Processor -- start 2023-10-30T13:13:00.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.702+0000 7fefc4794640 1 -- start start 2023-10-30T13:13:00.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.702+0000 7fefc4794640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fefbc159e60 0x7fefbc06f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.702+0000 7fefc4794640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefbc06a720 0x7fefbc06abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.702+0000 7fefc4794640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefbc06fa80 con 0x7fefbc0bfec0 2023-10-30T13:13:00.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.702+0000 7fefc4794640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefbc071290 con 0x7fefbc159e60 2023-10-30T13:13:00.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.702+0000 7fefc4794640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc071570 con 0x7fefbc06a720 2023-10-30T13:13:00.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.702+0000 7fefc2530640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fefbc159e60 0x7fefbc06f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.702+0000 7fefc2530640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fefbc159e60 0x7fefbc06f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37090/0 (socket says 172.21.15.125:37090) 2023-10-30T13:13:00.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.702+0000 7fefc2530640 1 -- 172.21.15.125:0/4255043091 learned_addr learned my addr 172.21.15.125:0/4255043091 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:00.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.702+0000 7fefc2d31640 1 --2- 172.21.15.125:0/4255043091 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefbc06a720 0x7fefbc06abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.706+0000 7fefc2530640 1 -- 172.21.15.125:0/4255043091 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefbc06a720 msgr2=0x7fefbc06abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:00.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.706+0000 7fefc2530640 1 --2- 172.21.15.125:0/4255043091 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefbc06a720 0x7fefbc06abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.706+0000 7fefc2530640 1 -- 172.21.15.125:0/4255043091 >> v1:172.21.15.125:6789/0 conn(0x7fefbc0bfec0 legacy=0x7fefbc0c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.706+0000 7fefc2530640 1 -- 172.21.15.125:0/4255043091 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefbc071850 con 0x7fefbc159e60 2023-10-30T13:13:00.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.706+0000 7fefc2530640 1 --2- 172.21.15.125:0/4255043091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefbc159e60 0x7fefbc06f540 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009870 tx=0x7fefb801fe40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd6e06150eee98cf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:00.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.706+0000 7fefc152e640 1 -- 172.21.15.125:0/4255043091 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb80a9b40 con 0x7fefbc159e60 2023-10-30T13:13:00.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.706+0000 7fefc152e640 1 -- 172.21.15.125:0/4255043091 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb80a9ca0 con 0x7fefbc159e60 2023-10-30T13:13:00.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.706+0000 7fefc152e640 1 -- 172.21.15.125:0/4255043091 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb80b1660 con 0x7fefbc159e60 2023-10-30T13:13:00.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.706+0000 7fefc4794640 1 -- 172.21.15.125:0/4255043091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefbc159e60 msgr2=0x7fefbc06f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.706+0000 7fefc4794640 1 --2- 172.21.15.125:0/4255043091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefbc159e60 0x7fefbc06f540 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fefb8009870 tx=0x7fefb801fe40 comp rx=0 tx=0).stop 2023-10-30T13:13:00.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.706+0000 7fefc4794640 1 -- 172.21.15.125:0/4255043091 shutdown_connections 2023-10-30T13:13:00.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.706+0000 7fefc4794640 1 --2- 172.21.15.125:0/4255043091 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefbc06a720 0x7fefbc06abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.706+0000 7fefc4794640 1 --2- 172.21.15.125:0/4255043091 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefbc159e60 0x7fefbc06f540 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.706+0000 7fefc4794640 1 -- 172.21.15.125:0/4255043091 >> 172.21.15.125:0/4255043091 conn(0x7fefbc0c7910 msgr2=0x7fefbc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:00.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc4794640 1 -- 172.21.15.125:0/4255043091 shutdown_connections 2023-10-30T13:13:00.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc4794640 1 -- 172.21.15.125:0/4255043091 wait complete. 2023-10-30T13:13:00.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc4794640 1 Processor -- start 2023-10-30T13:13:00.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc4794640 1 -- start start 2023-10-30T13:13:00.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc4794640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fefbc06a720 0x7fefbc073990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc4794640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefbc0bfec0 0x7fefbc0c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc4794640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefbc159e60 0x7fefbc0afd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc4794640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc0c9c70 con 0x7fefbc159e60 2023-10-30T13:13:00.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc4794640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc153b00 con 0x7fefbc06a720 2023-10-30T13:13:00.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc4794640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc153de0 con 0x7fefbc0bfec0 2023-10-30T13:13:00.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc2d31640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefbc159e60 0x7fefbc0afd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc2d31640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefbc159e60 0x7fefbc0afd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37102/0 (socket says 172.21.15.125:37102) 2023-10-30T13:13:00.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc2d31640 1 -- 172.21.15.125:0/3091448391 learned_addr learned my addr 172.21.15.125:0/3091448391 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:00.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc1d2f640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefbc0bfec0 0x7fefbc0c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc2530640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fefbc06a720 0x7fefbc073990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc2d31640 1 -- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefbc0bfec0 msgr2=0x7fefbc0c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc2d31640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefbc0bfec0 0x7fefbc0c37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc2d31640 1 -- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fefbc06a720 msgr2=0x7fefbc073990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc2d31640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fefbc06a720 0x7fefbc073990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc2d31640 1 -- 172.21.15.125:0/3091448391 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb80094d0 con 0x7fefbc159e60 2023-10-30T13:13:00.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc2530640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fefbc06a720 0x7fefbc073990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:13:00.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc1d2f640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefbc0bfec0 0x7fefbc0c37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:13:00.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc2d31640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefbc159e60 0x7fefbc0afd90 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fefb4007e40 tx=0x7fefb400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:00.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefb2ffd640 1 -- 172.21.15.125:0/3091448391 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb4014620 con 0x7fefbc159e60 2023-10-30T13:13:00.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefb2ffd640 1 -- 172.21.15.125:0/3091448391 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb400ee90 con 0x7fefbc159e60 2023-10-30T13:13:00.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefb2ffd640 1 -- 172.21.15.125:0/3091448391 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb40149e0 con 0x7fefbc159e60 2023-10-30T13:13:00.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc4794640 1 -- 172.21.15.125:0/3091448391 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefbc0c32d0 con 0x7fefbc159e60 2023-10-30T13:13:00.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.710+0000 7fefc4794640 1 -- 172.21.15.125:0/3091448391 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefbc1540c0 con 0x7fefbc159e60 2023-10-30T13:13:00.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.714+0000 7fefc4794640 1 -- 172.21.15.125:0/3091448391 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefbc04e170 con 0x7fefbc159e60 2023-10-30T13:13:00.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.714+0000 7fefb2ffd640 1 -- 172.21.15.125:0/3091448391 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fefb4014b40 con 0x7fefbc159e60 2023-10-30T13:13:00.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.714+0000 7fefb2ffd640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefa00664a0 0x7fefa0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:00.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.714+0000 7fefb2ffd640 1 -- 172.21.15.125:0/3091448391 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7fefb4085110 con 0x7fefbc159e60 2023-10-30T13:13:00.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.714+0000 7fefc2530640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefa00664a0 0x7fefa0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:00.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.714+0000 7fefc2530640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefa00664a0 0x7fefa0068960 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fefb80b3000 tx=0x7fefb80ab070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:00.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.718+0000 7fefb2ffd640 1 -- 172.21.15.125:0/3091448391 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fefb4051d10 con 0x7fefbc159e60 2023-10-30T13:13:00.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.902+0000 7fefc4794640 1 -- 172.21.15.125:0/3091448391 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefbc1e1060 con 0x7fefbc159e60 2023-10-30T13:13:00.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.902+0000 7fefb2ffd640 1 -- 172.21.15.125:0/3091448391 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+5759 (secure 0 0 0) 0x7fefb4055330 con 0x7fefbc159e60 2023-10-30T13:13:00.908 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:13:00.908 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":206,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:12:56.602354+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:resolve","state_seq":16,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:13:00.909 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 206 2023-10-30T13:13:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.906+0000 7fefc4794640 1 -- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefa00664a0 msgr2=0x7fefa0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.906+0000 7fefc4794640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefa00664a0 0x7fefa0068960 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fefb80b3000 tx=0x7fefb80ab070 comp rx=0 tx=0).stop 2023-10-30T13:13:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.906+0000 7fefc4794640 1 -- 172.21.15.125:0/3091448391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefbc159e60 msgr2=0x7fefbc0afd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.906+0000 7fefc4794640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefbc159e60 0x7fefbc0afd90 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fefb4007e40 tx=0x7fefb400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.906+0000 7fefc4794640 1 -- 172.21.15.125:0/3091448391 shutdown_connections 2023-10-30T13:13:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.906+0000 7fefc4794640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fefa00664a0 0x7fefa0068960 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.906+0000 7fefc4794640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fefbc159e60 0x7fefbc0afd90 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.906+0000 7fefc4794640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fefbc0bfec0 0x7fefbc0c37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.906+0000 7fefc4794640 1 --2- 172.21.15.125:0/3091448391 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fefbc06a720 0x7fefbc073990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:00.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.906+0000 7fefc4794640 1 -- 172.21.15.125:0/3091448391 >> 172.21.15.125:0/3091448391 conn(0x7fefbc0c7910 msgr2=0x7fefbc0c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:00.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.906+0000 7fefc4794640 1 -- 172.21.15.125:0/3091448391 shutdown_connections 2023-10-30T13:13:00.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:00.906+0000 7fefc4794640 1 -- 172.21.15.125:0/3091448391 wait complete. 2023-10-30T13:13:00.929 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:13:02.931 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:13:03.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b7780640 1 Processor -- start 2023-10-30T13:13:03.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b7780640 1 -- start start 2023-10-30T13:13:03.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b7780640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f67b015d6f0 0x7f67b015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b7780640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f67b00cf160 0x7f67b00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b7780640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b00cb800 con 0x7f67b01538a0 2023-10-30T13:13:03.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b7780640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67b00cb330 con 0x7f67b015d6f0 2023-10-30T13:13:03.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b7780640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67b00cad70 con 0x7f67b00cf160 2023-10-30T13:13:03.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b4d1b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f67b015d6f0 0x7f67b015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b4d1b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f67b015d6f0 0x7f67b015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38568/0 (socket says 172.21.15.125:38568) 2023-10-30T13:13:03.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b4d1b640 1 -- 172.21.15.125:0/2785142456 learned_addr learned my addr 172.21.15.125:0/2785142456 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:03.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b5d1d640 1 --2- 172.21.15.125:0/2785142456 >> v2:172.21.15.136:3300/0 conn(0x7f67b00cf160 0x7f67b00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b4d1b640 1 -- 172.21.15.125:0/2785142456 >> v2:172.21.15.136:3300/0 conn(0x7f67b00cf160 msgr2=0x7f67b00cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:03.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b4d1b640 1 --2- 172.21.15.125:0/2785142456 >> v2:172.21.15.136:3300/0 conn(0x7f67b00cf160 0x7f67b00cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b4d1b640 1 -- 172.21.15.125:0/2785142456 >> v1:172.21.15.136:6789/0 conn(0x7f67b01538a0 legacy=0x7f67b00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:03.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.074+0000 7f67b4d1b640 1 -- 172.21.15.125:0/2785142456 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b00cb050 con 0x7f67b015d6f0 2023-10-30T13:13:03.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.078+0000 7f67b4d1b640 1 --2- 172.21.15.125:0/2785142456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67b015d6f0 0x7f67b015a090 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f67b00b5330 tx=0x7f67a001ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4a03f3996d7ac0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:03.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.078+0000 7f67a7fff640 1 -- 172.21.15.125:0/2785142456 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a00a89b0 con 0x7f67b015d6f0 2023-10-30T13:13:03.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.078+0000 7f67a7fff640 1 -- 172.21.15.125:0/2785142456 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a00a8b10 con 0x7f67b015d6f0 2023-10-30T13:13:03.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.078+0000 7f67a7fff640 1 -- 172.21.15.125:0/2785142456 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a00a8ea0 con 0x7f67b015d6f0 2023-10-30T13:13:03.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.078+0000 7f67b7780640 1 -- 172.21.15.125:0/2785142456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67b015d6f0 msgr2=0x7f67b015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.078+0000 7f67b7780640 1 --2- 172.21.15.125:0/2785142456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67b015d6f0 0x7f67b015a090 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7f67b00b5330 tx=0x7f67a001ecb0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.078+0000 7f67b7780640 1 -- 172.21.15.125:0/2785142456 shutdown_connections 2023-10-30T13:13:03.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.078+0000 7f67b7780640 1 --2- 172.21.15.125:0/2785142456 >> v2:172.21.15.136:3300/0 conn(0x7f67b00cf160 0x7f67b00cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.078+0000 7f67b7780640 1 --2- 172.21.15.125:0/2785142456 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67b015d6f0 0x7f67b015a090 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.078+0000 7f67b7780640 1 -- 172.21.15.125:0/2785142456 >> 172.21.15.125:0/2785142456 conn(0x7f67b006bfe0 msgr2=0x7f67b006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:03.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.078+0000 7f67b7780640 1 -- 172.21.15.125:0/2785142456 shutdown_connections 2023-10-30T13:13:03.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.078+0000 7f67b7780640 1 -- 172.21.15.125:0/2785142456 wait complete. 2023-10-30T13:13:03.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b7780640 1 Processor -- start 2023-10-30T13:13:03.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b7780640 1 -- start start 2023-10-30T13:13:03.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b7780640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67b00cf160 0x7f67b00b4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b7780640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67b01538a0 0x7f67b014e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b7780640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67b015d6f0 0x7f67b014e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b7780640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b00cd980 con 0x7f67b00cf160 2023-10-30T13:13:03.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b7780640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b014ec50 con 0x7f67b015d6f0 2023-10-30T13:13:03.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b7780640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b00cd050 con 0x7f67b01538a0 2023-10-30T13:13:03.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b551c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67b00cf160 0x7f67b00b4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b551c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67b00cf160 0x7f67b00b4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38574/0 (socket says 172.21.15.125:38574) 2023-10-30T13:13:03.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b551c640 1 -- 172.21.15.125:0/3187214390 learned_addr learned my addr 172.21.15.125:0/3187214390 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:03.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b4d1b640 1 --2- 172.21.15.125:0/3187214390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67b01538a0 0x7f67b014e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b5d1d640 1 --2- 172.21.15.125:0/3187214390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67b015d6f0 0x7f67b014e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b551c640 1 -- 172.21.15.125:0/3187214390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67b01538a0 msgr2=0x7f67b014e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b551c640 1 --2- 172.21.15.125:0/3187214390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67b01538a0 0x7f67b014e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b551c640 1 -- 172.21.15.125:0/3187214390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67b015d6f0 msgr2=0x7f67b014e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b551c640 1 --2- 172.21.15.125:0/3187214390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67b015d6f0 0x7f67b014e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b551c640 1 -- 172.21.15.125:0/3187214390 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a00094d0 con 0x7f67b00cf160 2023-10-30T13:13:03.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b551c640 1 --2- 172.21.15.125:0/3187214390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67b00cf160 0x7f67b00b4c60 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f67ac00c450 tx=0x7f67ac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:03.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67a5ffb640 1 -- 172.21.15.125:0/3187214390 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac012830 con 0x7f67b00cf160 2023-10-30T13:13:03.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67a5ffb640 1 -- 172.21.15.125:0/3187214390 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67ac0129b0 con 0x7f67b00cf160 2023-10-30T13:13:03.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b7780640 1 -- 172.21.15.125:0/3187214390 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b00cd330 con 0x7f67b00cf160 2023-10-30T13:13:03.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67a5ffb640 1 -- 172.21.15.125:0/3187214390 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ac012d60 con 0x7f67b00cf160 2023-10-30T13:13:03.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.082+0000 7f67b7780640 1 -- 172.21.15.125:0/3187214390 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b014ef10 con 0x7f67b00cf160 2023-10-30T13:13:03.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.086+0000 7f67b7780640 1 -- 172.21.15.125:0/3187214390 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b00b2a50 con 0x7f67b00cf160 2023-10-30T13:13:03.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.086+0000 7f67a5ffb640 1 -- 172.21.15.125:0/3187214390 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f67ac01c460 con 0x7f67b00cf160 2023-10-30T13:13:03.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.086+0000 7f67a5ffb640 1 --2- 172.21.15.125:0/3187214390 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67940664a0 0x7f6794068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:03.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.086+0000 7f67a5ffb640 1 -- 172.21.15.125:0/3187214390 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7f67ac084570 con 0x7f67b00cf160 2023-10-30T13:13:03.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.086+0000 7f67b4d1b640 1 --2- 172.21.15.125:0/3187214390 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67940664a0 0x7f6794068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:03.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.086+0000 7f67b4d1b640 1 --2- 172.21.15.125:0/3187214390 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67940664a0 0x7f6794068960 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f67a00b2000 tx=0x7f67a0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:03.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.090+0000 7f67a5ffb640 1 -- 172.21.15.125:0/3187214390 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f67ac051170 con 0x7f67b00cf160 2023-10-30T13:13:03.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.270+0000 7f67b7780640 1 -- 172.21.15.125:0/3187214390 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67b004e170 con 0x7f67b00cf160 2023-10-30T13:13:03.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.270+0000 7f67a5ffb640 1 -- 172.21.15.125:0/3187214390 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+5761 (secure 0 0 0) 0x7f67ac054790 con 0x7f67b00cf160 2023-10-30T13:13:03.276 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:13:03.277 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:13:01.841517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:13:03.277 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 207 2023-10-30T13:13:03.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.274+0000 7f67b7780640 1 -- 172.21.15.125:0/3187214390 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67940664a0 msgr2=0x7f6794068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.274+0000 7f67b7780640 1 --2- 172.21.15.125:0/3187214390 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67940664a0 0x7f6794068960 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f67a00b2000 tx=0x7f67a0008040 comp rx=0 tx=0).stop 2023-10-30T13:13:03.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.274+0000 7f67b7780640 1 -- 172.21.15.125:0/3187214390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67b00cf160 msgr2=0x7f67b00b4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:03.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.274+0000 7f67b7780640 1 --2- 172.21.15.125:0/3187214390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67b00cf160 0x7f67b00b4c60 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f67ac00c450 tx=0x7f67ac00c910 comp rx=0 tx=0).stop 2023-10-30T13:13:03.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.274+0000 7f67b7780640 1 -- 172.21.15.125:0/3187214390 shutdown_connections 2023-10-30T13:13:03.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.274+0000 7f67b7780640 1 --2- 172.21.15.125:0/3187214390 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f67940664a0 0x7f6794068960 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.274+0000 7f67b7780640 1 --2- 172.21.15.125:0/3187214390 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f67b015d6f0 0x7f67b014e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.274+0000 7f67b7780640 1 --2- 172.21.15.125:0/3187214390 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f67b01538a0 0x7f67b014e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.274+0000 7f67b7780640 1 --2- 172.21.15.125:0/3187214390 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f67b00cf160 0x7f67b00b4c60 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:03.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.274+0000 7f67b7780640 1 -- 172.21.15.125:0/3187214390 >> 172.21.15.125:0/3187214390 conn(0x7f67b006bfe0 msgr2=0x7f67b006c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:03.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.274+0000 7f67b7780640 1 -- 172.21.15.125:0/3187214390 shutdown_connections 2023-10-30T13:13:03.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:03.274+0000 7f67b7780640 1 -- 172.21.15.125:0/3187214390 wait complete. 2023-10-30T13:13:03.298 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 207, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 207, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_7511": { "addr": "172.21.15.136:6835/695089547", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 695089547, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 695089547, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 7511, "incarnation": 205, "join_fscid": -1, "name": "d", "rank": 1, "state": "up:reconnect", "state_seq": 18 }, "gid_8443": { "addr": "172.21.15.136:6839/3776186190", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3776186190, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3776186190, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 8443, "incarnation": 196, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 41 }, "gid_8529": { "addr": "172.21.15.125:6836/3950044434", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3950044434, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3950044434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 8529, "incarnation": 179, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 79, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:13:01.841517+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8529, "mds_1": 7511, "mds_2": 8443 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:13:05.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:13:05.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8f097c5640 1 Processor -- start 2023-10-30T13:13:05.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8f097c5640 1 -- start start 2023-10-30T13:13:05.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8f097c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f0406e470 0x7f8f0406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:05.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8f097c5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f040c8490 0x7f8f040c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:05.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8f097c5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f0406afa0 con 0x7f8f04153d70 2023-10-30T13:13:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8f097c5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f0406d780 con 0x7f8f040c8490 2023-10-30T13:13:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8f097c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f04154150 con 0x7f8f0406e470 2023-10-30T13:13:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8f037fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f040c8490 0x7f8f040c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8f037fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f040c8490 0x7f8f040c8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38576/0 (socket says 172.21.15.125:38576) 2023-10-30T13:13:05.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8f037fe640 1 -- 172.21.15.125:0/2076850095 learned_addr learned my addr 172.21.15.125:0/2076850095 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:05.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8efa7fc640 1 --2- 172.21.15.125:0/2076850095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f0406e470 0x7f8f0406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:05.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8f037fe640 1 -- 172.21.15.125:0/2076850095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f0406e470 msgr2=0x7f8f0406ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:05.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8f037fe640 1 --2- 172.21.15.125:0/2076850095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f0406e470 0x7f8f0406ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8f037fe640 1 -- 172.21.15.125:0/2076850095 >> v1:172.21.15.136:6789/0 conn(0x7f8f04153d70 legacy=0x7f8f040cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:05.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.442+0000 7f8f037fe640 1 -- 172.21.15.125:0/2076850095 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0406e030 con 0x7f8f040c8490 2023-10-30T13:13:05.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.446+0000 7f8f037fe640 1 --2- 172.21.15.125:0/2076850095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f040c8490 0x7f8f040c8cc0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f8efc009e50 tx=0x7f8efc01c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a500dcf223526a65 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:05.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.446+0000 7f8f027fc640 1 -- 172.21.15.125:0/2076850095 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc0a5820 con 0x7f8f040c8490 2023-10-30T13:13:05.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.446+0000 7f8f027fc640 1 -- 172.21.15.125:0/2076850095 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8efc0a5980 con 0x7f8f040c8490 2023-10-30T13:13:05.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.446+0000 7f8f027fc640 1 -- 172.21.15.125:0/2076850095 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8efc0a5d10 con 0x7f8f040c8490 2023-10-30T13:13:05.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.446+0000 7f8f097c5640 1 -- 172.21.15.125:0/2076850095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f040c8490 msgr2=0x7f8f040c8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:05.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.446+0000 7f8f097c5640 1 --2- 172.21.15.125:0/2076850095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f040c8490 0x7f8f040c8cc0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f8efc009e50 tx=0x7f8efc01c9a0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.446+0000 7f8f097c5640 1 -- 172.21.15.125:0/2076850095 shutdown_connections 2023-10-30T13:13:05.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.446+0000 7f8f097c5640 1 --2- 172.21.15.125:0/2076850095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f040c8490 0x7f8f040c8cc0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.446+0000 7f8f097c5640 1 --2- 172.21.15.125:0/2076850095 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f0406e470 0x7f8f0406ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.446+0000 7f8f097c5640 1 -- 172.21.15.125:0/2076850095 >> 172.21.15.125:0/2076850095 conn(0x7f8f04071ea0 msgr2=0x7f8f04065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:05.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.446+0000 7f8f097c5640 1 -- 172.21.15.125:0/2076850095 shutdown_connections 2023-10-30T13:13:05.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.446+0000 7f8f097c5640 1 -- 172.21.15.125:0/2076850095 wait complete. 2023-10-30T13:13:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8f097c5640 1 Processor -- start 2023-10-30T13:13:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8f097c5640 1 -- start start 2023-10-30T13:13:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8f097c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f0406e470 0x7f8f041509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8f097c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f040c8490 0x7f8f0413f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:05.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8f097c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f04153d70 0x7f8f041544f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:05.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8f097c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f0406cdd0 con 0x7f8f040c8490 2023-10-30T13:13:05.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8f097c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f040cad10 con 0x7f8f0406e470 2023-10-30T13:13:05.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8f097c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f040caff0 con 0x7f8f04153d70 2023-10-30T13:13:05.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8efa7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f040c8490 0x7f8f0413f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:05.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8efa7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f040c8490 0x7f8f0413f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38592/0 (socket says 172.21.15.125:38592) 2023-10-30T13:13:05.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8efa7fc640 1 -- 172.21.15.125:0/1823601315 learned_addr learned my addr 172.21.15.125:0/1823601315 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:05.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8f037fe640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f04153d70 0x7f8f041544f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:05.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8f02ffd640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f0406e470 0x7f8f041509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:05.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8efa7fc640 1 -- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f04153d70 msgr2=0x7f8f041544f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:05.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8efa7fc640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f04153d70 0x7f8f041544f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8efa7fc640 1 -- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f0406e470 msgr2=0x7f8f041509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:05.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8efa7fc640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f0406e470 0x7f8f041509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8efa7fc640 1 -- 172.21.15.125:0/1823601315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc009b30 con 0x7f8f040c8490 2023-10-30T13:13:05.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8f02ffd640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f0406e470 0x7f8f041509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:13:05.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8efa7fc640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f040c8490 0x7f8f0413f6c0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f8ef400d5f0 tx=0x7f8ef400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:05.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8efbfff640 1 -- 172.21.15.125:0/1823601315 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef40125f0 con 0x7f8f040c8490 2023-10-30T13:13:05.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8efbfff640 1 -- 172.21.15.125:0/1823601315 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef4012750 con 0x7f8f040c8490 2023-10-30T13:13:05.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8f097c5640 1 -- 172.21.15.125:0/1823601315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f040cb2d0 con 0x7f8f040c8490 2023-10-30T13:13:05.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8efbfff640 1 -- 172.21.15.125:0/1823601315 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4012ae0 con 0x7f8f040c8490 2023-10-30T13:13:05.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8f097c5640 1 -- 172.21.15.125:0/1823601315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f04140c90 con 0x7f8f040c8490 2023-10-30T13:13:05.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.450+0000 7f8f097c5640 1 -- 172.21.15.125:0/1823601315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f0404e170 con 0x7f8f040c8490 2023-10-30T13:13:05.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.454+0000 7f8efbfff640 1 -- 172.21.15.125:0/1823601315 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8ef4012c40 con 0x7f8f040c8490 2023-10-30T13:13:05.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.454+0000 7f8efbfff640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8ee40664a0 0x7f8ee4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:05.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.454+0000 7f8efbfff640 1 -- 172.21.15.125:0/1823601315 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7f8ef4084c80 con 0x7f8f040c8490 2023-10-30T13:13:05.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.454+0000 7f8f02ffd640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8ee40664a0 0x7f8ee4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:05.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.454+0000 7f8f02ffd640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8ee40664a0 0x7f8ee4068960 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f8eec003e40 tx=0x7f8eec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:05.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.458+0000 7f8efbfff640 1 -- 172.21.15.125:0/1823601315 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8ef4051800 con 0x7f8f040c8490 2023-10-30T13:13:05.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.642+0000 7f8f097c5640 1 -- 172.21.15.125:0/1823601315 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f0406c2a0 con 0x7f8f040c8490 2023-10-30T13:13:05.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.642+0000 7f8efbfff640 1 -- 172.21.15.125:0/1823601315 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+5761 (secure 0 0 0) 0x7f8ef4007380 con 0x7f8f040c8490 2023-10-30T13:13:05.646 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:13:05.646 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":207,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:13:01.841517+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:13:05.646 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 207 2023-10-30T13:13:05.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.642+0000 7f8f097c5640 1 -- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8ee40664a0 msgr2=0x7f8ee4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:05.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.642+0000 7f8f097c5640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8ee40664a0 0x7f8ee4068960 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f8eec003e40 tx=0x7f8eec009040 comp rx=0 tx=0).stop 2023-10-30T13:13:05.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.642+0000 7f8f097c5640 1 -- 172.21.15.125:0/1823601315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f040c8490 msgr2=0x7f8f0413f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:05.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.642+0000 7f8f097c5640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f040c8490 0x7f8f0413f6c0 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f8ef400d5f0 tx=0x7f8ef400dab0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.642+0000 7f8f097c5640 1 -- 172.21.15.125:0/1823601315 shutdown_connections 2023-10-30T13:13:05.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.642+0000 7f8f097c5640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8ee40664a0 0x7f8ee4068960 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.642+0000 7f8f097c5640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f04153d70 0x7f8f041544f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.642+0000 7f8f097c5640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f040c8490 0x7f8f0413f6c0 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.642+0000 7f8f097c5640 1 --2- 172.21.15.125:0/1823601315 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f0406e470 0x7f8f041509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:05.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.642+0000 7f8f097c5640 1 -- 172.21.15.125:0/1823601315 >> 172.21.15.125:0/1823601315 conn(0x7f8f04071ea0 msgr2=0x7f8f04152d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:05.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.642+0000 7f8f097c5640 1 -- 172.21.15.125:0/1823601315 shutdown_connections 2023-10-30T13:13:05.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:05.642+0000 7f8f097c5640 1 -- 172.21.15.125:0/1823601315 wait complete. 2023-10-30T13:13:05.667 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:13:07.668 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:13:07.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be8654640 1 Processor -- start 2023-10-30T13:13:07.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be8654640 1 -- start start 2023-10-30T13:13:07.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be8654640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0be01538a0 0x7f0be015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:07.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be8654640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0be00ca0e0 0x7f0be015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:07.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be8654640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be00cb800 con 0x7f0be00cf160 2023-10-30T13:13:07.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be8654640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be015dc30 con 0x7f0be01538a0 2023-10-30T13:13:07.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be8654640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be00cb330 con 0x7f0be00ca0e0 2023-10-30T13:13:07.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be63f0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0be01538a0 0x7f0be015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:07.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be63f0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0be01538a0 0x7f0be015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38598/0 (socket says 172.21.15.125:38598) 2023-10-30T13:13:07.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be63f0640 1 -- 172.21.15.125:0/3522881924 learned_addr learned my addr 172.21.15.125:0/3522881924 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:07.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be5bef640 1 --2- 172.21.15.125:0/3522881924 >> v2:172.21.15.136:3300/0 conn(0x7f0be00ca0e0 0x7f0be015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:07.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be63f0640 1 -- 172.21.15.125:0/3522881924 >> v2:172.21.15.136:3300/0 conn(0x7f0be00ca0e0 msgr2=0x7f0be015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:07.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be63f0640 1 --2- 172.21.15.125:0/3522881924 >> v2:172.21.15.136:3300/0 conn(0x7f0be00ca0e0 0x7f0be015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:07.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be63f0640 1 -- 172.21.15.125:0/3522881924 >> v1:172.21.15.136:6789/0 conn(0x7f0be00cf160 legacy=0x7f0be00cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:07.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.814+0000 7f0be63f0640 1 -- 172.21.15.125:0/3522881924 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be00cad70 con 0x7f0be01538a0 2023-10-30T13:13:07.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.818+0000 7f0be63f0640 1 --2- 172.21.15.125:0/3522881924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0be01538a0 0x7f0be015d6f0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f0be00b30a0 tx=0x7f0bdc01ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b8c6ea53338c561 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:07.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.818+0000 7f0be53ee640 1 -- 172.21.15.125:0/3522881924 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc0a89b0 con 0x7f0be01538a0 2023-10-30T13:13:07.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.818+0000 7f0be53ee640 1 -- 172.21.15.125:0/3522881924 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bdc0a8b10 con 0x7f0be01538a0 2023-10-30T13:13:07.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.818+0000 7f0be53ee640 1 -- 172.21.15.125:0/3522881924 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc0a8ea0 con 0x7f0be01538a0 2023-10-30T13:13:07.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.818+0000 7f0be8654640 1 -- 172.21.15.125:0/3522881924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0be01538a0 msgr2=0x7f0be015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:07.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.818+0000 7f0be8654640 1 --2- 172.21.15.125:0/3522881924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0be01538a0 0x7f0be015d6f0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f0be00b30a0 tx=0x7f0bdc01ecb0 comp rx=0 tx=0).stop 2023-10-30T13:13:07.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.818+0000 7f0be8654640 1 -- 172.21.15.125:0/3522881924 shutdown_connections 2023-10-30T13:13:07.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.818+0000 7f0be8654640 1 --2- 172.21.15.125:0/3522881924 >> v2:172.21.15.136:3300/0 conn(0x7f0be00ca0e0 0x7f0be015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:07.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.818+0000 7f0be8654640 1 --2- 172.21.15.125:0/3522881924 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0be01538a0 0x7f0be015d6f0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:07.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.818+0000 7f0be8654640 1 -- 172.21.15.125:0/3522881924 >> 172.21.15.125:0/3522881924 conn(0x7f0be006bfe0 msgr2=0x7f0be006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:07.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.818+0000 7f0be8654640 1 -- 172.21.15.125:0/3522881924 shutdown_connections 2023-10-30T13:13:07.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.818+0000 7f0be8654640 1 -- 172.21.15.125:0/3522881924 wait complete. 2023-10-30T13:13:07.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be8654640 1 Processor -- start 2023-10-30T13:13:07.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be8654640 1 -- start start 2023-10-30T13:13:07.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be8654640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0be00ca0e0 0x7f0be00b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:07.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be8654640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0be00cf160 0x7f0be01466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:07.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be8654640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0be01538a0 0x7f0be01498e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:07.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be8654640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be00cda00 con 0x7f0be01538a0 2023-10-30T13:13:07.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be8654640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be01e2ae0 con 0x7f0be00ca0e0 2023-10-30T13:13:07.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be8654640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be00b3570 con 0x7f0be00cf160 2023-10-30T13:13:07.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be6bf1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0be01538a0 0x7f0be01498e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:07.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be6bf1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0be01538a0 0x7f0be01498e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38608/0 (socket says 172.21.15.125:38608) 2023-10-30T13:13:07.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be6bf1640 1 -- 172.21.15.125:0/417118600 learned_addr learned my addr 172.21.15.125:0/417118600 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:07.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be5bef640 1 --2- 172.21.15.125:0/417118600 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0be00cf160 0x7f0be01466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:07.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be63f0640 1 --2- 172.21.15.125:0/417118600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0be00ca0e0 0x7f0be00b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:07.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be6bf1640 1 -- 172.21.15.125:0/417118600 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0be00cf160 msgr2=0x7f0be01466e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:07.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be6bf1640 1 --2- 172.21.15.125:0/417118600 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0be00cf160 0x7f0be01466e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:07.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be6bf1640 1 -- 172.21.15.125:0/417118600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0be00ca0e0 msgr2=0x7f0be00b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:07.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be6bf1640 1 --2- 172.21.15.125:0/417118600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0be00ca0e0 0x7f0be00b4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:07.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be6bf1640 1 -- 172.21.15.125:0/417118600 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc0094d0 con 0x7f0be01538a0 2023-10-30T13:13:07.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be6bf1640 1 --2- 172.21.15.125:0/417118600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0be01538a0 0x7f0be01498e0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800c780 tx=0x7f0bd800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:07.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0bd6ffd640 1 -- 172.21.15.125:0/417118600 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8004b20 con 0x7f0be01538a0 2023-10-30T13:13:07.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0bd6ffd640 1 -- 172.21.15.125:0/417118600 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd8004c80 con 0x7f0be01538a0 2023-10-30T13:13:07.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0bd6ffd640 1 -- 172.21.15.125:0/417118600 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd800aa90 con 0x7f0be01538a0 2023-10-30T13:13:07.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be8654640 1 -- 172.21.15.125:0/417118600 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be0145ff0 con 0x7f0be01538a0 2023-10-30T13:13:07.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.822+0000 7f0be8654640 1 -- 172.21.15.125:0/417118600 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be00ccfb0 con 0x7f0be01538a0 2023-10-30T13:13:07.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.826+0000 7f0be8654640 1 -- 172.21.15.125:0/417118600 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be00b2a50 con 0x7f0be01538a0 2023-10-30T13:13:07.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.826+0000 7f0bd6ffd640 1 -- 172.21.15.125:0/417118600 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0bd800abf0 con 0x7f0be01538a0 2023-10-30T13:13:07.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.826+0000 7f0bd6ffd640 1 --2- 172.21.15.125:0/417118600 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0bc40664a0 0x7f0bc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:07.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.826+0000 7f0bd6ffd640 1 -- 172.21.15.125:0/417118600 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7f0bd80846b0 con 0x7f0be01538a0 2023-10-30T13:13:07.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.826+0000 7f0be63f0640 1 --2- 172.21.15.125:0/417118600 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0bc40664a0 0x7f0bc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:07.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.826+0000 7f0be63f0640 1 --2- 172.21.15.125:0/417118600 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0bc40664a0 0x7f0bc4068960 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc0b2000 tx=0x7f0bdc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:07.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:07.830+0000 7f0bd6ffd640 1 -- 172.21.15.125:0/417118600 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0bd8051370 con 0x7f0be01538a0 2023-10-30T13:13:08.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:08.010+0000 7f0be8654640 1 -- 172.21.15.125:0/417118600 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0be004e170 con 0x7f0be01538a0 2023-10-30T13:13:08.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:08.010+0000 7f0bd6ffd640 1 -- 172.21.15.125:0/417118600 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f0bd8054990 con 0x7f0be01538a0 2023-10-30T13:13:08.015 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:13:08.015 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:13:07.057408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:13:08.016 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 208 2023-10-30T13:13:08.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:08.014+0000 7f0be8654640 1 -- 172.21.15.125:0/417118600 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0bc40664a0 msgr2=0x7f0bc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:08.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:08.014+0000 7f0be8654640 1 --2- 172.21.15.125:0/417118600 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0bc40664a0 0x7f0bc4068960 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc0b2000 tx=0x7f0bdc009040 comp rx=0 tx=0).stop 2023-10-30T13:13:08.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:08.014+0000 7f0be8654640 1 -- 172.21.15.125:0/417118600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0be01538a0 msgr2=0x7f0be01498e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:08.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:08.014+0000 7f0be8654640 1 --2- 172.21.15.125:0/417118600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0be01538a0 0x7f0be01498e0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f0bd800c780 tx=0x7f0bd800cc40 comp rx=0 tx=0).stop 2023-10-30T13:13:08.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:08.014+0000 7f0be8654640 1 -- 172.21.15.125:0/417118600 shutdown_connections 2023-10-30T13:13:08.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:08.014+0000 7f0be8654640 1 --2- 172.21.15.125:0/417118600 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0bc40664a0 0x7f0bc4068960 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:08.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:08.014+0000 7f0be8654640 1 --2- 172.21.15.125:0/417118600 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0be01538a0 0x7f0be01498e0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:08.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:08.014+0000 7f0be8654640 1 --2- 172.21.15.125:0/417118600 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0be00cf160 0x7f0be01466e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:08.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:08.014+0000 7f0be8654640 1 --2- 172.21.15.125:0/417118600 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0be00ca0e0 0x7f0be00b4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:08.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:08.014+0000 7f0be8654640 1 -- 172.21.15.125:0/417118600 >> 172.21.15.125:0/417118600 conn(0x7f0be006bfe0 msgr2=0x7f0be00c5460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:08.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:08.014+0000 7f0be8654640 1 -- 172.21.15.125:0/417118600 shutdown_connections 2023-10-30T13:13:08.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:08.014+0000 7f0be8654640 1 -- 172.21.15.125:0/417118600 wait complete. 2023-10-30T13:13:08.036 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:13:10.038 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:13:10.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9adff94640 1 Processor -- start 2023-10-30T13:13:10.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9adff94640 1 -- start start 2023-10-30T13:13:10.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9adff94640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9ad806f7a0 0x7f9ad806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:10.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9adff94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ad8071110 0x7f9ad8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:10.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9adff94640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ad8066cf0 con 0x7f9ad80c6bd0 2023-10-30T13:13:10.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9adff94640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ad80700c0 con 0x7f9ad806f7a0 2023-10-30T13:13:10.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9adff94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad80c70d0 con 0x7f9ad8071110 2023-10-30T13:13:10.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9ade531640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ad8071110 0x7f9ad8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:10.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9addd30640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9ad806f7a0 0x7f9ad806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:10.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9addd30640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9ad806f7a0 0x7f9ad806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:36426/0 (socket says 172.21.15.125:36426) 2023-10-30T13:13:10.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9addd30640 1 -- 172.21.15.125:0/3388788285 learned_addr learned my addr 172.21.15.125:0/3388788285 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:10.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9addd30640 1 -- 172.21.15.125:0/3388788285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ad8071110 msgr2=0x7f9ad8158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:10.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9addd30640 1 --2- 172.21.15.125:0/3388788285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ad8071110 0x7f9ad8158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:10.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9addd30640 1 -- 172.21.15.125:0/3388788285 >> v1:172.21.15.136:6789/0 conn(0x7f9ad80c6bd0 legacy=0x7f9ad8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:10.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9addd30640 1 -- 172.21.15.125:0/3388788285 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad80c7390 con 0x7f9ad806f7a0 2023-10-30T13:13:10.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.186+0000 7f9ade531640 1 --2- 172.21.15.125:0/3388788285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ad8071110 0x7f9ad8158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:13:10.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.190+0000 7f9addd30640 1 --2- 172.21.15.125:0/3388788285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ad806f7a0 0x7f9ad806fb80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009870 tx=0x7f9ad401ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bab3ae57ea2c476 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:10.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.190+0000 7f9adcd2e640 1 -- 172.21.15.125:0/3388788285 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad40a8a30 con 0x7f9ad806f7a0 2023-10-30T13:13:10.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.190+0000 7f9adcd2e640 1 -- 172.21.15.125:0/3388788285 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ad40a8b90 con 0x7f9ad806f7a0 2023-10-30T13:13:10.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.190+0000 7f9adcd2e640 1 -- 172.21.15.125:0/3388788285 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad40b0440 con 0x7f9ad806f7a0 2023-10-30T13:13:10.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.190+0000 7f9adff94640 1 -- 172.21.15.125:0/3388788285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ad806f7a0 msgr2=0x7f9ad806fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:10.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.190+0000 7f9adff94640 1 --2- 172.21.15.125:0/3388788285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ad806f7a0 0x7f9ad806fb80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9ad4009870 tx=0x7f9ad401ed30 comp rx=0 tx=0).stop 2023-10-30T13:13:10.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.190+0000 7f9adff94640 1 -- 172.21.15.125:0/3388788285 shutdown_connections 2023-10-30T13:13:10.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.190+0000 7f9adff94640 1 --2- 172.21.15.125:0/3388788285 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ad8071110 0x7f9ad8158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:10.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.190+0000 7f9adff94640 1 --2- 172.21.15.125:0/3388788285 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ad806f7a0 0x7f9ad806fb80 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:10.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.190+0000 7f9adff94640 1 -- 172.21.15.125:0/3388788285 >> 172.21.15.125:0/3388788285 conn(0x7f9ad80c7910 msgr2=0x7f9ad80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:10.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.190+0000 7f9adff94640 1 -- 172.21.15.125:0/3388788285 shutdown_connections 2023-10-30T13:13:10.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.190+0000 7f9adff94640 1 -- 172.21.15.125:0/3388788285 wait complete. 2023-10-30T13:13:10.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9adff94640 1 Processor -- start 2023-10-30T13:13:10.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9adff94640 1 -- start start 2023-10-30T13:13:10.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9adff94640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ad806f7a0 0x7f9ad814fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:10.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9adff94640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ad8071110 0x7f9ad8154100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:10.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9adff94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ad80c6bd0 0x7f9ad8154640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:10.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9adff94640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad8156240 con 0x7f9ad8071110 2023-10-30T13:13:10.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9adff94640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad8156520 con 0x7f9ad806f7a0 2023-10-30T13:13:10.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9adff94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ad8156800 con 0x7f9ad80c6bd0 2023-10-30T13:13:10.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9add52f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ad8071110 0x7f9ad8154100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:10.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9addd30640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ad806f7a0 0x7f9ad814fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:10.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9ade531640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ad80c6bd0 0x7f9ad8154640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:10.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9add52f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ad8071110 0x7f9ad8154100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38622/0 (socket says 172.21.15.125:38622) 2023-10-30T13:13:10.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9add52f640 1 -- 172.21.15.125:0/1345442623 learned_addr learned my addr 172.21.15.125:0/1345442623 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:10.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9add52f640 1 -- 172.21.15.125:0/1345442623 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ad80c6bd0 msgr2=0x7f9ad8154640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:10.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9add52f640 1 --2- 172.21.15.125:0/1345442623 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ad80c6bd0 0x7f9ad8154640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:10.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9add52f640 1 -- 172.21.15.125:0/1345442623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ad806f7a0 msgr2=0x7f9ad814fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:10.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9add52f640 1 --2- 172.21.15.125:0/1345442623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ad806f7a0 0x7f9ad814fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:10.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9add52f640 1 -- 172.21.15.125:0/1345442623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ad0009a70 con 0x7f9ad8071110 2023-10-30T13:13:10.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9add52f640 1 --2- 172.21.15.125:0/1345442623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ad8071110 0x7f9ad8154100 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f9ac800c430 tx=0x7f9ac800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:10.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9ace7fc640 1 -- 172.21.15.125:0/1345442623 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac80052d0 con 0x7f9ad8071110 2023-10-30T13:13:10.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9ace7fc640 1 -- 172.21.15.125:0/1345442623 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ac8005430 con 0x7f9ad8071110 2023-10-30T13:13:10.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9adff94640 1 -- 172.21.15.125:0/1345442623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ad40094d0 con 0x7f9ad8071110 2023-10-30T13:13:10.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9ace7fc640 1 -- 172.21.15.125:0/1345442623 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ac8007260 con 0x7f9ad8071110 2023-10-30T13:13:10.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.194+0000 7f9adff94640 1 -- 172.21.15.125:0/1345442623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ad8156dc0 con 0x7f9ad8071110 2023-10-30T13:13:10.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.198+0000 7f9adff94640 1 -- 172.21.15.125:0/1345442623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ad8156f50 con 0x7f9ad8071110 2023-10-30T13:13:10.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.198+0000 7f9ace7fc640 1 -- 172.21.15.125:0/1345442623 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9ac80073c0 con 0x7f9ad8071110 2023-10-30T13:13:10.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.198+0000 7f9ace7fc640 1 --2- 172.21.15.125:0/1345442623 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9abc0664a0 0x7f9abc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:10.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.198+0000 7f9ace7fc640 1 -- 172.21.15.125:0/1345442623 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7f9ac8083920 con 0x7f9ad8071110 2023-10-30T13:13:10.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.198+0000 7f9addd30640 1 --2- 172.21.15.125:0/1345442623 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9abc0664a0 0x7f9abc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:10.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.198+0000 7f9addd30640 1 --2- 172.21.15.125:0/1345442623 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9abc0664a0 0x7f9abc068960 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40b2000 tx=0x7f9ad40aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:10.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.202+0000 7f9ace7fc640 1 -- 172.21.15.125:0/1345442623 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9ad8156f50 con 0x7f9ad8071110 2023-10-30T13:13:10.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.382+0000 7f9adff94640 1 -- 172.21.15.125:0/1345442623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ad8156f50 con 0x7f9ad8071110 2023-10-30T13:13:10.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.382+0000 7f9ace7fc640 1 -- 172.21.15.125:0/1345442623 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f9ac8050660 con 0x7f9ad8071110 2023-10-30T13:13:10.388 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:13:10.389 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:13:07.057408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:13:10.390 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 208 2023-10-30T13:13:10.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.386+0000 7f9adff94640 1 -- 172.21.15.125:0/1345442623 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9abc0664a0 msgr2=0x7f9abc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:10.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.386+0000 7f9adff94640 1 --2- 172.21.15.125:0/1345442623 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9abc0664a0 0x7f9abc068960 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f9ad40b2000 tx=0x7f9ad40aa070 comp rx=0 tx=0).stop 2023-10-30T13:13:10.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.386+0000 7f9adff94640 1 -- 172.21.15.125:0/1345442623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ad8071110 msgr2=0x7f9ad8154100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:10.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.386+0000 7f9adff94640 1 --2- 172.21.15.125:0/1345442623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ad8071110 0x7f9ad8154100 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f9ac800c430 tx=0x7f9ac800c830 comp rx=0 tx=0).stop 2023-10-30T13:13:10.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.386+0000 7f9adff94640 1 -- 172.21.15.125:0/1345442623 shutdown_connections 2023-10-30T13:13:10.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.386+0000 7f9adff94640 1 --2- 172.21.15.125:0/1345442623 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9abc0664a0 0x7f9abc068960 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:10.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.386+0000 7f9adff94640 1 --2- 172.21.15.125:0/1345442623 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ad80c6bd0 0x7f9ad8154640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:10.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.386+0000 7f9adff94640 1 --2- 172.21.15.125:0/1345442623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ad8071110 0x7f9ad8154100 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:10.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.386+0000 7f9adff94640 1 --2- 172.21.15.125:0/1345442623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ad806f7a0 0x7f9ad814fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:10.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.386+0000 7f9adff94640 1 -- 172.21.15.125:0/1345442623 >> 172.21.15.125:0/1345442623 conn(0x7f9ad80c7910 msgr2=0x7f9ad80c3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:10.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.386+0000 7f9adff94640 1 -- 172.21.15.125:0/1345442623 shutdown_connections 2023-10-30T13:13:10.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:10.386+0000 7f9adff94640 1 -- 172.21.15.125:0/1345442623 wait complete. 2023-10-30T13:13:10.409 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:13:12.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:13:12.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f9674640 1 Processor -- start 2023-10-30T13:13:12.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f9674640 1 -- start start 2023-10-30T13:13:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f9674640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe6f415cd50 0x7fe6f40c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f9674640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe6f40cefa0 0x7fe6f40cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f9674640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f415a6c0 con 0x7fe6f41538f0 2023-10-30T13:13:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f9674640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f40cf8f0 con 0x7fe6f415cd50 2023-10-30T13:13:12.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f9674640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f415d300 con 0x7fe6f40cefa0 2023-10-30T13:13:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe6f415cd50 0x7fe6f40c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe6f415cd50 0x7fe6f40c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49232/0 (socket says 172.21.15.125:49232) 2023-10-30T13:13:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f37fe640 1 -- 172.21.15.125:0/1335691338 learned_addr learned my addr 172.21.15.125:0/1335691338 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f8e73640 1 --2- 172.21.15.125:0/1335691338 >> v2:172.21.15.136:3300/0 conn(0x7fe6f40cefa0 0x7fe6f40cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:12.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f37fe640 1 -- 172.21.15.125:0/1335691338 >> v2:172.21.15.136:3300/0 conn(0x7fe6f40cefa0 msgr2=0x7fe6f40cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f37fe640 1 --2- 172.21.15.125:0/1335691338 >> v2:172.21.15.136:3300/0 conn(0x7fe6f40cefa0 0x7fe6f40cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f37fe640 1 -- 172.21.15.125:0/1335691338 >> v1:172.21.15.136:6789/0 conn(0x7fe6f41538f0 legacy=0x7fe6f4159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:12.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.554+0000 7fe6f37fe640 1 -- 172.21.15.125:0/1335691338 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f415d5e0 con 0x7fe6f415cd50 2023-10-30T13:13:12.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.558+0000 7fe6f37fe640 1 --2- 172.21.15.125:0/1335691338 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f415cd50 0x7fe6f40c9f90 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4066a40 tx=0x7fe6e801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b41ed47757a556fe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:12.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.558+0000 7fe6f2ffd640 1 -- 172.21.15.125:0/1335691338 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e80a89f0 con 0x7fe6f415cd50 2023-10-30T13:13:12.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.558+0000 7fe6f2ffd640 1 -- 172.21.15.125:0/1335691338 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e80a8b50 con 0x7fe6f415cd50 2023-10-30T13:13:12.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.558+0000 7fe6f2ffd640 1 -- 172.21.15.125:0/1335691338 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e80b0440 con 0x7fe6f415cd50 2023-10-30T13:13:12.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.558+0000 7fe6f9674640 1 -- 172.21.15.125:0/1335691338 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f415cd50 msgr2=0x7fe6f40c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:12.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.558+0000 7fe6f9674640 1 --2- 172.21.15.125:0/1335691338 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f415cd50 0x7fe6f40c9f90 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fe6f4066a40 tx=0x7fe6e801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:13:12.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.558+0000 7fe6f9674640 1 -- 172.21.15.125:0/1335691338 shutdown_connections 2023-10-30T13:13:12.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.558+0000 7fe6f9674640 1 --2- 172.21.15.125:0/1335691338 >> v2:172.21.15.136:3300/0 conn(0x7fe6f40cefa0 0x7fe6f40cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:12.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.558+0000 7fe6f9674640 1 --2- 172.21.15.125:0/1335691338 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f415cd50 0x7fe6f40c9f90 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:12.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.558+0000 7fe6f9674640 1 -- 172.21.15.125:0/1335691338 >> 172.21.15.125:0/1335691338 conn(0x7fe6f406a640 msgr2=0x7fe6f4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.558+0000 7fe6f9674640 1 -- 172.21.15.125:0/1335691338 shutdown_connections 2023-10-30T13:13:12.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.558+0000 7fe6f9674640 1 -- 172.21.15.125:0/1335691338 wait complete. 2023-10-30T13:13:12.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.558+0000 7fe6f9674640 1 Processor -- start 2023-10-30T13:13:12.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f9674640 1 -- start start 2023-10-30T13:13:12.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f9674640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f40cefa0 0x7fe6f41453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:12.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f9674640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6f41538f0 0x7fe6f415a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:12.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f9674640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6f415cd50 0x7fe6f4158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:12.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f9674640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f41f02f0 con 0x7fe6f40cefa0 2023-10-30T13:13:12.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f9674640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f4157a20 con 0x7fe6f415cd50 2023-10-30T13:13:12.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f9674640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f41ef2e0 con 0x7fe6f41538f0 2023-10-30T13:13:12.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f40cefa0 0x7fe6f41453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:12.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f40cefa0 0x7fe6f41453c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49234/0 (socket says 172.21.15.125:49234) 2023-10-30T13:13:12.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f3fff640 1 -- 172.21.15.125:0/481316623 learned_addr learned my addr 172.21.15.125:0/481316623 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:12.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f37fe640 1 --2- 172.21.15.125:0/481316623 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6f41538f0 0x7fe6f415a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:12.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f8e73640 1 --2- 172.21.15.125:0/481316623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6f415cd50 0x7fe6f4158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:12.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f3fff640 1 -- 172.21.15.125:0/481316623 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6f41538f0 msgr2=0x7fe6f415a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:12.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f3fff640 1 --2- 172.21.15.125:0/481316623 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6f41538f0 0x7fe6f415a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:12.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f3fff640 1 -- 172.21.15.125:0/481316623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6f415cd50 msgr2=0x7fe6f4158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:12.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f3fff640 1 --2- 172.21.15.125:0/481316623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6f415cd50 0x7fe6f4158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:12.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f3fff640 1 -- 172.21.15.125:0/481316623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e80094d0 con 0x7fe6f40cefa0 2023-10-30T13:13:12.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f3fff640 1 --2- 172.21.15.125:0/481316623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f40cefa0 0x7fe6f41453c0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fe6e400c500 tx=0x7fe6e400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:12.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f0ff9640 1 -- 172.21.15.125:0/481316623 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4005220 con 0x7fe6f40cefa0 2023-10-30T13:13:12.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f0ff9640 1 -- 172.21.15.125:0/481316623 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e4005380 con 0x7fe6f40cefa0 2023-10-30T13:13:12.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f9674640 1 -- 172.21.15.125:0/481316623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f41efe60 con 0x7fe6f40cefa0 2023-10-30T13:13:12.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f0ff9640 1 -- 172.21.15.125:0/481316623 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e4007270 con 0x7fe6f40cefa0 2023-10-30T13:13:12.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f9674640 1 -- 172.21.15.125:0/481316623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f41ef530 con 0x7fe6f40cefa0 2023-10-30T13:13:12.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.562+0000 7fe6f9674640 1 -- 172.21.15.125:0/481316623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6f404e170 con 0x7fe6f40cefa0 2023-10-30T13:13:12.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.566+0000 7fe6f0ff9640 1 -- 172.21.15.125:0/481316623 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe6e40073d0 con 0x7fe6f40cefa0 2023-10-30T13:13:12.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.566+0000 7fe6f0ff9640 1 --2- 172.21.15.125:0/481316623 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6e00664d0 0x7fe6e0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:12.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.566+0000 7fe6f0ff9640 1 -- 172.21.15.125:0/481316623 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7fe6e4084040 con 0x7fe6f40cefa0 2023-10-30T13:13:12.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.566+0000 7fe6f37fe640 1 --2- 172.21.15.125:0/481316623 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6e00664d0 0x7fe6e0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:12.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.566+0000 7fe6f37fe640 1 --2- 172.21.15.125:0/481316623 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6e00664d0 0x7fe6e0068990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fe6e80b2000 tx=0x7fe6e8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:12.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.570+0000 7fe6f0ff9640 1 -- 172.21.15.125:0/481316623 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe6e40b5090 con 0x7fe6f40cefa0 2023-10-30T13:13:12.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.754+0000 7fe6f9674640 1 -- 172.21.15.125:0/481316623 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6f415c470 con 0x7fe6f40cefa0 2023-10-30T13:13:12.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.754+0000 7fe6f0ff9640 1 -- 172.21.15.125:0/481316623 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+5758 (secure 0 0 0) 0x7fe6e4050bd0 con 0x7fe6f40cefa0 2023-10-30T13:13:12.759 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:13:12.759 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:13:07.057408+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:13:12.759 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 208 2023-10-30T13:13:12.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.754+0000 7fe6f9674640 1 -- 172.21.15.125:0/481316623 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6e00664d0 msgr2=0x7fe6e0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:12.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.754+0000 7fe6f9674640 1 --2- 172.21.15.125:0/481316623 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6e00664d0 0x7fe6e0068990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fe6e80b2000 tx=0x7fe6e8008040 comp rx=0 tx=0).stop 2023-10-30T13:13:12.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.754+0000 7fe6f9674640 1 -- 172.21.15.125:0/481316623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f40cefa0 msgr2=0x7fe6f41453c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:12.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.754+0000 7fe6f9674640 1 --2- 172.21.15.125:0/481316623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f40cefa0 0x7fe6f41453c0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fe6e400c500 tx=0x7fe6e400c9c0 comp rx=0 tx=0).stop 2023-10-30T13:13:12.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.754+0000 7fe6f9674640 1 -- 172.21.15.125:0/481316623 shutdown_connections 2023-10-30T13:13:12.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.754+0000 7fe6f9674640 1 --2- 172.21.15.125:0/481316623 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6e00664d0 0x7fe6e0068990 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:12.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.754+0000 7fe6f9674640 1 --2- 172.21.15.125:0/481316623 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe6f415cd50 0x7fe6f4158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:12.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.754+0000 7fe6f9674640 1 --2- 172.21.15.125:0/481316623 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6f41538f0 0x7fe6f415a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:12.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.754+0000 7fe6f9674640 1 --2- 172.21.15.125:0/481316623 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe6f40cefa0 0x7fe6f41453c0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:12.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.754+0000 7fe6f9674640 1 -- 172.21.15.125:0/481316623 >> 172.21.15.125:0/481316623 conn(0x7fe6f406a640 msgr2=0x7fe6f40cd710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:12.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.754+0000 7fe6f9674640 1 -- 172.21.15.125:0/481316623 shutdown_connections 2023-10-30T13:13:12.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:12.754+0000 7fe6f9674640 1 -- 172.21.15.125:0/481316623 wait complete. 2023-10-30T13:13:12.780 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:13:14.780 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:13:14.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.922+0000 7fe86106b640 1 Processor -- start 2023-10-30T13:13:14.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.922+0000 7fe86106b640 1 -- start start 2023-10-30T13:13:14.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.922+0000 7fe86106b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe85c0c05d0 0x7fe85c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:14.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.922+0000 7fe86106b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe85c0c6f60 0x7fe85c13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:14.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.922+0000 7fe86106b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85c0c0a50 con 0x7fe85c0c6b80 2023-10-30T13:13:14.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.922+0000 7fe86106b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe85c06f6e0 con 0x7fe85c0c05d0 2023-10-30T13:13:14.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.922+0000 7fe86106b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c06f9c0 con 0x7fe85c0c6f60 2023-10-30T13:13:14.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.922+0000 7fe853fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe85c0c05d0 0x7fe85c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:14.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.922+0000 7fe853fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe85c0c05d0 0x7fe85c13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49242/0 (socket says 172.21.15.125:49242) 2023-10-30T13:13:14.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.922+0000 7fe853fff640 1 -- 172.21.15.125:0/1393905822 learned_addr learned my addr 172.21.15.125:0/1393905822 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:14.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.922+0000 7fe85b59e640 1 --2- 172.21.15.125:0/1393905822 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe85c0c6f60 0x7fe85c13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:14.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.926+0000 7fe853fff640 1 -- 172.21.15.125:0/1393905822 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe85c0c6f60 msgr2=0x7fe85c13f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:14.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.926+0000 7fe853fff640 1 --2- 172.21.15.125:0/1393905822 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe85c0c6f60 0x7fe85c13f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:14.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.926+0000 7fe853fff640 1 -- 172.21.15.125:0/1393905822 >> v1:172.21.15.136:6789/0 conn(0x7fe85c0c6b80 legacy=0x7fe85c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:14.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.926+0000 7fe853fff640 1 -- 172.21.15.125:0/1393905822 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe85c06fca0 con 0x7fe85c0c05d0 2023-10-30T13:13:14.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.926+0000 7fe853fff640 1 --2- 172.21.15.125:0/1393905822 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe85c0c05d0 0x7fe85c13ed90 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fe844009870 tx=0x7fe84401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b64dabc673dbce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:14.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.926+0000 7fe85a59c640 1 -- 172.21.15.125:0/1393905822 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8440ab650 con 0x7fe85c0c05d0 2023-10-30T13:13:14.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.926+0000 7fe85a59c640 1 -- 172.21.15.125:0/1393905822 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8440ab7b0 con 0x7fe85c0c05d0 2023-10-30T13:13:14.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.926+0000 7fe85a59c640 1 -- 172.21.15.125:0/1393905822 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8440abb40 con 0x7fe85c0c05d0 2023-10-30T13:13:14.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.926+0000 7fe86106b640 1 -- 172.21.15.125:0/1393905822 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe85c0c05d0 msgr2=0x7fe85c13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:14.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.926+0000 7fe86106b640 1 --2- 172.21.15.125:0/1393905822 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe85c0c05d0 0x7fe85c13ed90 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fe844009870 tx=0x7fe84401c540 comp rx=0 tx=0).stop 2023-10-30T13:13:14.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.926+0000 7fe86106b640 1 -- 172.21.15.125:0/1393905822 shutdown_connections 2023-10-30T13:13:14.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.926+0000 7fe86106b640 1 --2- 172.21.15.125:0/1393905822 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe85c0c6f60 0x7fe85c13f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:14.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.926+0000 7fe86106b640 1 --2- 172.21.15.125:0/1393905822 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe85c0c05d0 0x7fe85c13ed90 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:14.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.926+0000 7fe86106b640 1 -- 172.21.15.125:0/1393905822 >> 172.21.15.125:0/1393905822 conn(0x7fe85c0c7910 msgr2=0x7fe85c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:14.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe86106b640 1 -- 172.21.15.125:0/1393905822 shutdown_connections 2023-10-30T13:13:14.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe86106b640 1 -- 172.21.15.125:0/1393905822 wait complete. 2023-10-30T13:13:14.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe86106b640 1 Processor -- start 2023-10-30T13:13:14.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe86106b640 1 -- start start 2023-10-30T13:13:14.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe86106b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe85c0c05d0 0x7fe85c13c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:14.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe86106b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe85c0c6b80 0x7fe85c0c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:14.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe86106b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe85c0c6f60 0x7fe85c0c5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:14.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe86106b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c1f6280 con 0x7fe85c0c6f60 2023-10-30T13:13:14.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe86106b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c1f67b0 con 0x7fe85c0c05d0 2023-10-30T13:13:14.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe86106b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe85c1395e0 con 0x7fe85c0c6b80 2023-10-30T13:13:14.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe85b59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe85c0c6f60 0x7fe85c0c5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:14.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe85b59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe85c0c6f60 0x7fe85c0c5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49244/0 (socket says 172.21.15.125:49244) 2023-10-30T13:13:14.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe85b59e640 1 -- 172.21.15.125:0/3119446305 learned_addr learned my addr 172.21.15.125:0/3119446305 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:14.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe853fff640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe85c0c6b80 0x7fe85c0c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:14.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe85ad9d640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe85c0c05d0 0x7fe85c13c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:14.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe85b59e640 1 -- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe85c0c6b80 msgr2=0x7fe85c0c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:14.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe85b59e640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe85c0c6b80 0x7fe85c0c5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:14.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe85b59e640 1 -- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe85c0c05d0 msgr2=0x7fe85c13c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:14.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe85b59e640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe85c0c05d0 0x7fe85c13c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:14.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe85b59e640 1 -- 172.21.15.125:0/3119446305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8440094d0 con 0x7fe85c0c6f60 2023-10-30T13:13:14.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe853fff640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe85c0c6b80 0x7fe85c0c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:13:14.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe85b59e640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe85c0c6f60 0x7fe85c0c5d50 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fe854007e90 tx=0x7fe85400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:14.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.930+0000 7fe8537fe640 1 -- 172.21.15.125:0/3119446305 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854014420 con 0x7fe85c0c6f60 2023-10-30T13:13:14.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.934+0000 7fe8537fe640 1 -- 172.21.15.125:0/3119446305 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe854014580 con 0x7fe85c0c6f60 2023-10-30T13:13:14.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.934+0000 7fe8537fe640 1 -- 172.21.15.125:0/3119446305 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe854014930 con 0x7fe85c0c6f60 2023-10-30T13:13:14.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.934+0000 7fe86106b640 1 -- 172.21.15.125:0/3119446305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe85c154730 con 0x7fe85c0c6f60 2023-10-30T13:13:14.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.934+0000 7fe86106b640 1 -- 172.21.15.125:0/3119446305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe85c135500 con 0x7fe85c0c6f60 2023-10-30T13:13:14.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.934+0000 7fe86106b640 1 -- 172.21.15.125:0/3119446305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe85c04e170 con 0x7fe85c0c6f60 2023-10-30T13:13:14.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.934+0000 7fe8537fe640 1 -- 172.21.15.125:0/3119446305 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe854014a90 con 0x7fe85c0c6f60 2023-10-30T13:13:14.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.934+0000 7fe8537fe640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe83c0664a0 0x7fe83c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:14.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.934+0000 7fe8537fe640 1 -- 172.21.15.125:0/3119446305 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7fe854085050 con 0x7fe85c0c6f60 2023-10-30T13:13:14.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.934+0000 7fe85ad9d640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe83c0664a0 0x7fe83c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:14.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.938+0000 7fe85ad9d640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe83c0664a0 0x7fe83c068960 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe84c000f70 tx=0x7fe84c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:14.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:14.938+0000 7fe8537fe640 1 -- 172.21.15.125:0/3119446305 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe8540b6090 con 0x7fe85c0c6f60 2023-10-30T13:13:15.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:15.122+0000 7fe86106b640 1 -- 172.21.15.125:0/3119446305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe85c0c17e0 con 0x7fe85c0c6f60 2023-10-30T13:13:15.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:15.122+0000 7fe8537fe640 1 -- 172.21.15.125:0/3119446305 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5758 (secure 0 0 0) 0x7fe8540556f0 con 0x7fe85c0c6f60 2023-10-30T13:13:15.126 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:13:15.126 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:13:13.856950+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:active","state_seq":22,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:13:15.126 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 209 2023-10-30T13:13:15.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:15.122+0000 7fe86106b640 1 -- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe83c0664a0 msgr2=0x7fe83c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:15.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:15.122+0000 7fe86106b640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe83c0664a0 0x7fe83c068960 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe84c000f70 tx=0x7fe84c009040 comp rx=0 tx=0).stop 2023-10-30T13:13:15.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:15.122+0000 7fe86106b640 1 -- 172.21.15.125:0/3119446305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe85c0c6f60 msgr2=0x7fe85c0c5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:15.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:15.122+0000 7fe86106b640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe85c0c6f60 0x7fe85c0c5d50 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fe854007e90 tx=0x7fe85400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:13:15.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:15.122+0000 7fe86106b640 1 -- 172.21.15.125:0/3119446305 shutdown_connections 2023-10-30T13:13:15.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:15.122+0000 7fe86106b640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe83c0664a0 0x7fe83c068960 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:15.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:15.122+0000 7fe86106b640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe85c0c6f60 0x7fe85c0c5d50 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:15.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:15.122+0000 7fe86106b640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe85c0c6b80 0x7fe85c0c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:15.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:15.122+0000 7fe86106b640 1 --2- 172.21.15.125:0/3119446305 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe85c0c05d0 0x7fe85c13c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:15.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:15.122+0000 7fe86106b640 1 -- 172.21.15.125:0/3119446305 >> 172.21.15.125:0/3119446305 conn(0x7fe85c0c7910 msgr2=0x7fe85c0c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:15.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:15.122+0000 7fe86106b640 1 -- 172.21.15.125:0/3119446305 shutdown_connections 2023-10-30T13:13:15.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:15.122+0000 7fe86106b640 1 -- 172.21.15.125:0/3119446305 wait complete. 2023-10-30T13:13:15.147 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=1, replacing gid=8301 2023-10-30T13:13:15.147 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.c 2023-10-30T13:13:16.651 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:13:16.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:13:17.675 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T13:13:17.674+0000 7f510d857640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:13:19.984 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:13:19.985 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.15 2023-10-30T13:13:19.986 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 15 -eq 48 ] 2023-10-30T13:13:19.986 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.15 2023-10-30T13:13:19.986 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.15 2023-10-30T13:13:19.986 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.15 2023-10-30T13:13:19.986 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:13:20.148 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.c 2023-10-30T13:13:20.148 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-30T13:13:20.148 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-30T13:13:20.153 INFO:tasks.ceph.mds.c:Started 2023-10-30T13:13:20.153 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:13:20.243 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2023-10-30T13:13:20.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.314+0000 7f87fe079640 1 Processor -- start 2023-10-30T13:13:20.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.314+0000 7f87fe079640 1 -- start start 2023-10-30T13:13:20.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.314+0000 7f87fe079640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f87f815d6f0 0x7f87f815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:20.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.314+0000 7f87fe079640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87f80cf160 0x7f87f80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:20.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.314+0000 7f87fe079640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f80cb800 con 0x7f87f81538a0 2023-10-30T13:13:20.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.314+0000 7f87fe079640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f80cb330 con 0x7f87f815d6f0 2023-10-30T13:13:20.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.314+0000 7f87fe079640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f80cad70 con 0x7f87f80cf160 2023-10-30T13:13:20.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.314+0000 7f87f7fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87f80cf160 0x7f87f80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:20.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.314+0000 7f87f6ffd640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f87f815d6f0 0x7f87f815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:20.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.314+0000 7f87f7fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87f80cf160 0x7f87f80cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:39100/0 (socket says 172.21.15.125:39100) 2023-10-30T13:13:20.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.314+0000 7f87f7fff640 1 -- 172.21.15.125:0/1292218080 learned_addr learned my addr 172.21.15.125:0/1292218080 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:20.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.318+0000 7f87f6ffd640 1 -- 172.21.15.125:0/1292218080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87f80cf160 msgr2=0x7f87f80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:20.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.318+0000 7f87f6ffd640 1 --2- 172.21.15.125:0/1292218080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87f80cf160 0x7f87f80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:20.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.318+0000 7f87f6ffd640 1 -- 172.21.15.125:0/1292218080 >> v1:172.21.15.136:6789/0 conn(0x7f87f81538a0 legacy=0x7f87f80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:20.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.318+0000 7f87f6ffd640 1 -- 172.21.15.125:0/1292218080 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f80cb050 con 0x7f87f815d6f0 2023-10-30T13:13:20.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.318+0000 7f87f7fff640 1 --2- 172.21.15.125:0/1292218080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87f80cf160 0x7f87f80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:13:20.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.318+0000 7f87f6ffd640 1 --2- 172.21.15.125:0/1292218080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87f815d6f0 0x7f87f815a090 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0097c0 tx=0x7f87ec01ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98a113088b4f420f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:20.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.318+0000 7f87f67fc640 1 -- 172.21.15.125:0/1292218080 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0a8970 con 0x7f87f815d6f0 2023-10-30T13:13:20.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.318+0000 7f87f67fc640 1 -- 172.21.15.125:0/1292218080 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87ec0a8ad0 con 0x7f87f815d6f0 2023-10-30T13:13:20.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.318+0000 7f87f67fc640 1 -- 172.21.15.125:0/1292218080 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87ec0a8e60 con 0x7f87f815d6f0 2023-10-30T13:13:20.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 -- 172.21.15.125:0/1292218080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87f815d6f0 msgr2=0x7f87f815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:20.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 --2- 172.21.15.125:0/1292218080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87f815d6f0 0x7f87f815a090 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0097c0 tx=0x7f87ec01ec70 comp rx=0 tx=0).stop 2023-10-30T13:13:20.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 -- 172.21.15.125:0/1292218080 shutdown_connections 2023-10-30T13:13:20.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 --2- 172.21.15.125:0/1292218080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87f80cf160 0x7f87f80cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:20.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 --2- 172.21.15.125:0/1292218080 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87f815d6f0 0x7f87f815a090 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:20.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 -- 172.21.15.125:0/1292218080 >> 172.21.15.125:0/1292218080 conn(0x7f87f806bfe0 msgr2=0x7f87f806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:20.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 -- 172.21.15.125:0/1292218080 shutdown_connections 2023-10-30T13:13:20.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 -- 172.21.15.125:0/1292218080 wait complete. 2023-10-30T13:13:20.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 Processor -- start 2023-10-30T13:13:20.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 -- start start 2023-10-30T13:13:20.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87f80cf160 0x7f87f81e52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:20.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87f81538a0 0x7f87f81e3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:20.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87f815d6f0 0x7f87f8148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:20.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f81e3450 con 0x7f87f80cf160 2023-10-30T13:13:20.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f81e4430 con 0x7f87f81538a0 2023-10-30T13:13:20.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87fe079640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f81e4d40 con 0x7f87f815d6f0 2023-10-30T13:13:20.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87f77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87f80cf160 0x7f87f81e52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:20.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87f77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87f80cf160 0x7f87f81e52e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:49254/0 (socket says 172.21.15.125:49254) 2023-10-30T13:13:20.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87f77fe640 1 -- 172.21.15.125:0/2712698260 learned_addr learned my addr 172.21.15.125:0/2712698260 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:20.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87f6ffd640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87f81538a0 0x7f87f81e3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:20.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87f7fff640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87f815d6f0 0x7f87f8148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:20.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87f77fe640 1 -- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87f815d6f0 msgr2=0x7f87f8148a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:20.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87f77fe640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87f815d6f0 0x7f87f8148a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:20.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87f77fe640 1 -- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87f81538a0 msgr2=0x7f87f81e3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:20.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87f77fe640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87f81538a0 0x7f87f81e3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:20.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87f77fe640 1 -- 172.21.15.125:0/2712698260 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f0009b30 con 0x7f87f80cf160 2023-10-30T13:13:20.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.322+0000 7f87f6ffd640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87f81538a0 0x7f87f81e3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:13:20.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.326+0000 7f87f77fe640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87f80cf160 0x7f87f81e52e0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f87e800c500 tx=0x7f87e800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:20.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.326+0000 7f87dffff640 1 -- 172.21.15.125:0/2712698260 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e8005220 con 0x7f87f80cf160 2023-10-30T13:13:20.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.326+0000 7f87dffff640 1 -- 172.21.15.125:0/2712698260 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e8005380 con 0x7f87f80cf160 2023-10-30T13:13:20.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.326+0000 7f87fe079640 1 -- 172.21.15.125:0/2712698260 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87ec009420 con 0x7f87f80cf160 2023-10-30T13:13:20.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.326+0000 7f87dffff640 1 -- 172.21.15.125:0/2712698260 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e8007270 con 0x7f87f80cf160 2023-10-30T13:13:20.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.326+0000 7f87fe079640 1 -- 172.21.15.125:0/2712698260 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f81e36f0 con 0x7f87f80cf160 2023-10-30T13:13:20.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.326+0000 7f87fe079640 1 -- 172.21.15.125:0/2712698260 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f80b2a50 con 0x7f87f80cf160 2023-10-30T13:13:20.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.326+0000 7f87dffff640 1 -- 172.21.15.125:0/2712698260 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f87e80073d0 con 0x7f87f80cf160 2023-10-30T13:13:20.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.326+0000 7f87dffff640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87e40664a0 0x7f87e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:20.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.326+0000 7f87dffff640 1 -- 172.21.15.125:0/2712698260 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7f87e8084060 con 0x7f87f80cf160 2023-10-30T13:13:20.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.326+0000 7f87f6ffd640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87e40664a0 0x7f87e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:20.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.330+0000 7f87f6ffd640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87e40664a0 0x7f87e4068960 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0b2000 tx=0x7f87ec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:20.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.330+0000 7f87dffff640 1 -- 172.21.15.125:0/2712698260 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f87e80b5090 con 0x7f87f80cf160 2023-10-30T13:13:20.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.514+0000 7f87fe079640 1 -- 172.21.15.125:0/2712698260 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87f804e170 con 0x7f87f80cf160 2023-10-30T13:13:20.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.518+0000 7f87dffff640 1 -- 172.21.15.125:0/2712698260 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f87e8050bf0 con 0x7f87f80cf160 2023-10-30T13:13:20.521 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:13:20.522 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:13:13.856950+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:active","state_seq":22,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:13:20.522 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 209 2023-10-30T13:13:20.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.518+0000 7f87fe079640 1 -- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87e40664a0 msgr2=0x7f87e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:20.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.518+0000 7f87fe079640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87e40664a0 0x7f87e4068960 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f87ec0b2000 tx=0x7f87ec008040 comp rx=0 tx=0).stop 2023-10-30T13:13:20.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.518+0000 7f87fe079640 1 -- 172.21.15.125:0/2712698260 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87f80cf160 msgr2=0x7f87f81e52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:20.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.518+0000 7f87fe079640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87f80cf160 0x7f87f81e52e0 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f87e800c500 tx=0x7f87e800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:13:20.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.518+0000 7f87fe079640 1 -- 172.21.15.125:0/2712698260 shutdown_connections 2023-10-30T13:13:20.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.518+0000 7f87fe079640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f87e40664a0 0x7f87e4068960 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:20.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.518+0000 7f87fe079640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f87f815d6f0 0x7f87f8148a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:20.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.518+0000 7f87fe079640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f87f81538a0 0x7f87f81e3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:20.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.518+0000 7f87fe079640 1 --2- 172.21.15.125:0/2712698260 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f87f80cf160 0x7f87f81e52e0 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:20.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.518+0000 7f87fe079640 1 -- 172.21.15.125:0/2712698260 >> 172.21.15.125:0/2712698260 conn(0x7f87f806bfe0 msgr2=0x7f87f81529c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:20.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.518+0000 7f87fe079640 1 -- 172.21.15.125:0/2712698260 shutdown_connections 2023-10-30T13:13:20.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:20.518+0000 7f87fe079640 1 -- 172.21.15.125:0/2712698260 wait complete. 2023-10-30T13:13:20.542 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.c is in active, standby or standby-replay 2023-10-30T13:13:22.543 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:13:22.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.678+0000 7ff7c7e09640 1 Processor -- start 2023-10-30T13:13:22.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.678+0000 7ff7c7e09640 1 -- start start 2023-10-30T13:13:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.678+0000 7ff7c7e09640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff7c01580e0 0x7ff7c0070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.678+0000 7ff7c7e09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c015e700 0x7ff7c006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.678+0000 7ff7c7e09640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c00c5810 con 0x7ff7c00c6540 2023-10-30T13:13:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.678+0000 7ff7c7e09640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c00c6990 con 0x7ff7c01580e0 2023-10-30T13:13:22.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.678+0000 7ff7c7e09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c015eb90 con 0x7ff7c015e700 2023-10-30T13:13:22.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.678+0000 7ff7c5ba5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff7c01580e0 0x7ff7c0070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:22.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.678+0000 7ff7c63a6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c015e700 0x7ff7c006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:22.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.678+0000 7ff7c5ba5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff7c01580e0 0x7ff7c0070740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:47304/0 (socket says 172.21.15.125:47304) 2023-10-30T13:13:22.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.678+0000 7ff7c53a4640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff7c00c6540 0x7ff7c0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:53812/0 (socket says 172.21.15.125:53812) 2023-10-30T13:13:22.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.678+0000 7ff7c5ba5640 1 -- 172.21.15.125:0/2043150225 learned_addr learned my addr 172.21.15.125:0/2043150225 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:22.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.682+0000 7ff7c5ba5640 1 -- 172.21.15.125:0/2043150225 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c015e700 msgr2=0x7ff7c006a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:13:22.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.682+0000 7ff7c5ba5640 1 --2- 172.21.15.125:0/2043150225 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c015e700 0x7ff7c006a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:22.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.682+0000 7ff7c5ba5640 1 -- 172.21.15.125:0/2043150225 >> v1:172.21.15.136:6789/0 conn(0x7ff7c00c6540 legacy=0x7ff7c0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:22.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.682+0000 7ff7c5ba5640 1 -- 172.21.15.125:0/2043150225 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c00c6bb0 con 0x7ff7c01580e0 2023-10-30T13:13:22.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.682+0000 7ff7c5ba5640 1 --2- 172.21.15.125:0/2043150225 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c01580e0 0x7ff7c0070740 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009870 tx=0x7ff7bc01c510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe9a1329705b9dca server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:22.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.682+0000 7ff7c4ba3640 1 -- 172.21.15.125:0/2043150225 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc0aa670 con 0x7ff7c01580e0 2023-10-30T13:13:22.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.682+0000 7ff7c4ba3640 1 -- 172.21.15.125:0/2043150225 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7bc0aa7d0 con 0x7ff7c01580e0 2023-10-30T13:13:22.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.682+0000 7ff7c4ba3640 1 -- 172.21.15.125:0/2043150225 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc0aab60 con 0x7ff7c01580e0 2023-10-30T13:13:22.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 -- 172.21.15.125:0/2043150225 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c01580e0 msgr2=0x7ff7c0070740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:22.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 --2- 172.21.15.125:0/2043150225 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c01580e0 0x7ff7c0070740 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009870 tx=0x7ff7bc01c510 comp rx=0 tx=0).stop 2023-10-30T13:13:22.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 -- 172.21.15.125:0/2043150225 shutdown_connections 2023-10-30T13:13:22.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 --2- 172.21.15.125:0/2043150225 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c015e700 0x7ff7c006a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:22.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 --2- 172.21.15.125:0/2043150225 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c01580e0 0x7ff7c0070740 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:22.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 -- 172.21.15.125:0/2043150225 >> 172.21.15.125:0/2043150225 conn(0x7ff7c00c7910 msgr2=0x7ff7c00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:22.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 -- 172.21.15.125:0/2043150225 shutdown_connections 2023-10-30T13:13:22.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 -- 172.21.15.125:0/2043150225 wait complete. 2023-10-30T13:13:22.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 Processor -- start 2023-10-30T13:13:22.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 -- start start 2023-10-30T13:13:22.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c00c6540 0x7ff7c01fa110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:22.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7c01580e0 0x7ff7c00c8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:22.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c015e700 0x7ff7c0149240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:22.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c01f9cf0 con 0x7ff7c01580e0 2023-10-30T13:13:22.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c00ca370 con 0x7ff7c015e700 2023-10-30T13:13:22.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c7e09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c0149780 con 0x7ff7c00c6540 2023-10-30T13:13:22.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c53a4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7c01580e0 0x7ff7c00c8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:22.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c53a4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7c01580e0 0x7ff7c00c8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35294/0 (socket says 172.21.15.125:35294) 2023-10-30T13:13:22.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c53a4640 1 -- 172.21.15.125:0/1545973053 learned_addr learned my addr 172.21.15.125:0/1545973053 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:13:22.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c5ba5640 1 --2- 172.21.15.125:0/1545973053 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c00c6540 0x7ff7c01fa110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:22.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c63a6640 1 --2- 172.21.15.125:0/1545973053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c015e700 0x7ff7c0149240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:22.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c53a4640 1 -- 172.21.15.125:0/1545973053 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c00c6540 msgr2=0x7ff7c01fa110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:22.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.686+0000 7ff7c53a4640 1 --2- 172.21.15.125:0/1545973053 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c00c6540 0x7ff7c01fa110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:22.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.690+0000 7ff7c53a4640 1 -- 172.21.15.125:0/1545973053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c015e700 msgr2=0x7ff7c0149240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:22.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.690+0000 7ff7c53a4640 1 --2- 172.21.15.125:0/1545973053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c015e700 0x7ff7c0149240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:22.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.690+0000 7ff7c53a4640 1 -- 172.21.15.125:0/1545973053 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7bc0094d0 con 0x7ff7c01580e0 2023-10-30T13:13:22.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.690+0000 7ff7c53a4640 1 --2- 172.21.15.125:0/1545973053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7c01580e0 0x7ff7c00c8e20 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7ff7b000c490 tx=0x7ff7b000c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:22.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.690+0000 7ff7b67fc640 1 -- 172.21.15.125:0/1545973053 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0007480 con 0x7ff7c01580e0 2023-10-30T13:13:22.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.690+0000 7ff7b67fc640 1 -- 172.21.15.125:0/1545973053 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7b00075e0 con 0x7ff7c01580e0 2023-10-30T13:13:22.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.690+0000 7ff7c7e09640 1 -- 172.21.15.125:0/1545973053 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c01f9670 con 0x7ff7c01580e0 2023-10-30T13:13:22.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.690+0000 7ff7b67fc640 1 -- 172.21.15.125:0/1545973053 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7b0004a10 con 0x7ff7c01580e0 2023-10-30T13:13:22.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.690+0000 7ff7c7e09640 1 -- 172.21.15.125:0/1545973053 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c00ca880 con 0x7ff7c01580e0 2023-10-30T13:13:22.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.690+0000 7ff7c7e09640 1 -- 172.21.15.125:0/1545973053 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c01f9e30 con 0x7ff7c01580e0 2023-10-30T13:13:22.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.690+0000 7ff7b67fc640 1 -- 172.21.15.125:0/1545973053 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff7b0004b70 con 0x7ff7c01580e0 2023-10-30T13:13:22.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.690+0000 7ff7b67fc640 1 --2- 172.21.15.125:0/1545973053 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7a40664a0 0x7ff7a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:13:22.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.690+0000 7ff7b67fc640 1 -- 172.21.15.125:0/1545973053 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7ff7b0083740 con 0x7ff7c01580e0 2023-10-30T13:13:22.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.690+0000 7ff7c5ba5640 1 --2- 172.21.15.125:0/1545973053 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7a40664a0 0x7ff7a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:13:22.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.694+0000 7ff7c5ba5640 1 --2- 172.21.15.125:0/1545973053 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7a40664a0 0x7ff7a4068960 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc0ae000 tx=0x7ff7bc0b5040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:13:22.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.694+0000 7ff7b67fc640 1 -- 172.21.15.125:0/1545973053 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff7b0050400 con 0x7ff7c01580e0 2023-10-30T13:13:22.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.878+0000 7ff7c7e09640 1 -- 172.21.15.125:0/1545973053 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7c00c5d10 con 0x7ff7c01580e0 2023-10-30T13:13:22.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.878+0000 7ff7b67fc640 1 -- 172.21.15.125:0/1545973053 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 210 v210) v1 ==== 95+0+6550 (secure 0 0 0) 0x7ff7b0053a20 con 0x7ff7c01580e0 2023-10-30T13:13:22.884 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:13:22.885 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":210,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9150,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:13:13.856950+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:active","state_seq":22,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:13:22.885 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 210 2023-10-30T13:13:22.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.882+0000 7ff7c7e09640 1 -- 172.21.15.125:0/1545973053 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7a40664a0 msgr2=0x7ff7a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:22.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.882+0000 7ff7c7e09640 1 --2- 172.21.15.125:0/1545973053 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7a40664a0 0x7ff7a4068960 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc0ae000 tx=0x7ff7bc0b5040 comp rx=0 tx=0).stop 2023-10-30T13:13:22.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.882+0000 7ff7c7e09640 1 -- 172.21.15.125:0/1545973053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7c01580e0 msgr2=0x7ff7c00c8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:13:22.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.882+0000 7ff7c7e09640 1 --2- 172.21.15.125:0/1545973053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7c01580e0 0x7ff7c00c8e20 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7ff7b000c490 tx=0x7ff7b000c950 comp rx=0 tx=0).stop 2023-10-30T13:13:22.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.882+0000 7ff7c7e09640 1 -- 172.21.15.125:0/1545973053 shutdown_connections 2023-10-30T13:13:22.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.882+0000 7ff7c7e09640 1 --2- 172.21.15.125:0/1545973053 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7a40664a0 0x7ff7a4068960 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:22.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.882+0000 7ff7c7e09640 1 --2- 172.21.15.125:0/1545973053 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c015e700 0x7ff7c0149240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:22.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.882+0000 7ff7c7e09640 1 --2- 172.21.15.125:0/1545973053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7c01580e0 0x7ff7c00c8e20 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:22.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.882+0000 7ff7c7e09640 1 --2- 172.21.15.125:0/1545973053 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c00c6540 0x7ff7c01fa110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:13:22.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.882+0000 7ff7c7e09640 1 -- 172.21.15.125:0/1545973053 >> 172.21.15.125:0/1545973053 conn(0x7ff7c00c7910 msgr2=0x7ff7c0070b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:13:22.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.882+0000 7ff7c7e09640 1 -- 172.21.15.125:0/1545973053 shutdown_connections 2023-10-30T13:13:22.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:13:22.882+0000 7ff7c7e09640 1 -- 172.21.15.125:0/1545973053 wait complete. 2023-10-30T13:13:22.906 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c reported in up:standby state 2023-10-30T13:13:22.906 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 88 secs before thrashing 2023-10-30T13:13:25.146 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:13:25.146 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.16 2023-10-30T13:13:25.147 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 16 -eq 48 ] 2023-10-30T13:13:25.147 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.16 2023-10-30T13:13:25.147 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.16 2023-10-30T13:13:25.147 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.16 2023-10-30T13:13:25.148 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:13:28.605 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:13:28.605 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 17 -eq 48 ] 2023-10-30T13:13:28.605 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.17 2023-10-30T13:13:28.605 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.17 2023-10-30T13:13:28.606 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.17 2023-10-30T13:13:28.606 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.17 2023-10-30T13:13:28.606 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:13:35.359 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:13:35.359 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.18 2023-10-30T13:13:35.360 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 18 -eq 48 ] 2023-10-30T13:13:35.360 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.18 2023-10-30T13:13:35.360 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.18 2023-10-30T13:13:35.360 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.18 2023-10-30T13:13:35.361 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:13:42.862 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:13:42.862 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.19 2023-10-30T13:13:42.863 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 19 -eq 48 ] 2023-10-30T13:13:42.863 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.19 2023-10-30T13:13:42.863 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.19 2023-10-30T13:13:42.864 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.19 2023-10-30T13:13:42.864 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:13:46.708 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:13:46.713 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:13:48.722 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:13:48.723 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.20 2023-10-30T13:13:48.723 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 20 -eq 48 ] 2023-10-30T13:13:48.724 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.20 2023-10-30T13:13:48.724 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.20 2023-10-30T13:13:48.724 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.20 2023-10-30T13:13:48.724 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:13:54.901 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:13:54.902 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.21 2023-10-30T13:13:54.902 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 21 -eq 48 ] 2023-10-30T13:13:54.903 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.21 2023-10-30T13:13:54.903 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.21 2023-10-30T13:13:54.903 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.21 2023-10-30T13:13:54.904 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:14:03.285 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:14:03.286 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.22 2023-10-30T13:14:03.287 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 22 -eq 48 ] 2023-10-30T13:14:03.287 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.22 2023-10-30T13:14:03.287 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.22 2023-10-30T13:14:03.287 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.22 2023-10-30T13:14:03.288 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:14:09.550 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:14:09.550 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.23 2023-10-30T13:14:09.551 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 23 -eq 48 ] 2023-10-30T13:14:09.551 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.23 2023-10-30T13:14:09.551 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.23 2023-10-30T13:14:09.551 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.23 2023-10-30T13:14:09.552 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:14:15.162 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:14:15.163 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 24 -eq 48 ] 2023-10-30T13:14:15.163 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.24 2023-10-30T13:14:15.164 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.24 2023-10-30T13:14:15.164 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.24 2023-10-30T13:14:15.164 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.24 2023-10-30T13:14:15.164 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:14:16.795 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:14:16.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:14:22.761 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:14:22.762 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.25 2023-10-30T13:14:22.762 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 25 -eq 48 ] 2023-10-30T13:14:22.763 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.25 2023-10-30T13:14:22.763 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.25 2023-10-30T13:14:22.763 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.25 2023-10-30T13:14:22.763 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:14:29.864 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:14:29.864 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.26 2023-10-30T13:14:29.865 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 26 -eq 48 ] 2023-10-30T13:14:29.865 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.26 2023-10-30T13:14:29.865 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.26 2023-10-30T13:14:29.865 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.26 2023-10-30T13:14:29.866 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:14:35.834 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:14:35.834 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.27 2023-10-30T13:14:35.835 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 27 -eq 48 ] 2023-10-30T13:14:35.835 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.27 2023-10-30T13:14:35.835 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.27 2023-10-30T13:14:35.836 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.27 2023-10-30T13:14:35.836 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:14:40.138 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:14:40.139 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 28 -eq 48 ] 2023-10-30T13:14:40.139 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.28 2023-10-30T13:14:40.139 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.28 2023-10-30T13:14:40.140 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.28 2023-10-30T13:14:40.140 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.28 2023-10-30T13:14:40.140 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:14:43.124 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:14:43.125 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.29 2023-10-30T13:14:43.125 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 29 -eq 48 ] 2023-10-30T13:14:43.126 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.29 2023-10-30T13:14:43.126 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.29 2023-10-30T13:14:43.126 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.29 2023-10-30T13:14:43.126 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:14:46.853 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:14:46.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:14:47.593 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:14:47.594 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.30 2023-10-30T13:14:47.594 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 30 -eq 48 ] 2023-10-30T13:14:47.595 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.30 2023-10-30T13:14:47.595 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.30 2023-10-30T13:14:47.595 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.30 2023-10-30T13:14:47.595 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:14:50.907 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:51.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9d09a7640 1 Processor -- start 2023-10-30T13:14:51.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9d09a7640 1 -- start start 2023-10-30T13:14:51.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9d09a7640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc9cc0c6c20 0x7fc9cc15a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:51.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9d09a7640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9cc0c3b30 con 0x7fc9cc0c3640 2023-10-30T13:14:51.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9d09a7640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9cc0c3df0 con 0x7fc9cc0c6840 2023-10-30T13:14:51.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9d09a7640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9cc0c2aa0 con 0x7fc9cc0c6c20 2023-10-30T13:14:51.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9caffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc9cc0c3640 0x7fc9cc15ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39522/0 (socket says 172.21.15.125:39522) 2023-10-30T13:14:51.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9caffd640 1 -- 172.21.15.125:0/4118443865 learned_addr learned my addr 172.21.15.125:0/4118443865 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:14:51.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9cbfff640 1 --2- 172.21.15.125:0/4118443865 >> v2:172.21.15.136:3300/0 conn(0x7fc9cc0c6c20 0x7fc9cc15a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:51.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 -- 172.21.15.125:0/4118443865 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345894850 0 0) 0x7fc9cc0c3b30 con 0x7fc9cc0c3640 2023-10-30T13:14:51.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 -- 172.21.15.125:0/4118443865 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9b4003530 con 0x7fc9cc0c3640 2023-10-30T13:14:51.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 -- 172.21.15.125:0/4118443865 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3281098348 0 0) 0x7fc9b4003530 con 0x7fc9cc0c3640 2023-10-30T13:14:51.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 -- 172.21.15.125:0/4118443865 >> v2:172.21.15.136:3300/0 conn(0x7fc9cc0c6c20 msgr2=0x7fc9cc15a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:51.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 --2- 172.21.15.125:0/4118443865 >> v2:172.21.15.136:3300/0 conn(0x7fc9cc0c6c20 0x7fc9cc15a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 -- 172.21.15.125:0/4118443865 >> v1:172.21.15.136:6789/0 conn(0x7fc9cc0c6840 legacy=0x7fc9cc153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:51.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 -- 172.21.15.125:0/4118443865 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9cc159ba0 con 0x7fc9cc0c3640 2023-10-30T13:14:51.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 -- 172.21.15.125:0/4118443865 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fc9c0001680 con 0x7fc9cc0c3640 2023-10-30T13:14:51.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 -- 172.21.15.125:0/4118443865 >> v1:172.21.15.125:6789/0 conn(0x7fc9cc0c3640 legacy=0x7fc9cc15ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:51.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 --2- 172.21.15.125:0/4118443865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9b4003980 0x7fc9b4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:51.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 --2- 172.21.15.125:0/4118443865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9b40047d0 0x7fc9b4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:51.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 --2- 172.21.15.125:0/4118443865 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9b4005190 0x7fc9b400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:51.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 -- 172.21.15.125:0/4118443865 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b400c000 con 0x7fc9b4003980 2023-10-30T13:14:51.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 -- 172.21.15.125:0/4118443865 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b400c170 con 0x7fc9b40047d0 2023-10-30T13:14:51.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9ca7fc640 1 -- 172.21.15.125:0/4118443865 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b400c330 con 0x7fc9b4005190 2023-10-30T13:14:51.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9cb7fe640 1 --2- 172.21.15.125:0/4118443865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9b4003980 0x7fc9b4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:51.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9caffd640 1 --2- 172.21.15.125:0/4118443865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9b40047d0 0x7fc9b4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:51.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9cbfff640 1 --2- 172.21.15.125:0/4118443865 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9b4005190 0x7fc9b400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:51.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9cb7fe640 1 -- 172.21.15.125:0/4118443865 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9b4005190 msgr2=0x7fc9b400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:51.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9cb7fe640 1 --2- 172.21.15.125:0/4118443865 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9b4005190 0x7fc9b400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9cb7fe640 1 -- 172.21.15.125:0/4118443865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9b40047d0 msgr2=0x7fc9b4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:51.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9cb7fe640 1 --2- 172.21.15.125:0/4118443865 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9b40047d0 0x7fc9b4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.044+0000 7fc9cb7fe640 1 -- 172.21.15.125:0/4118443865 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b4003530 con 0x7fc9b4003980 2023-10-30T13:14:51.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.048+0000 7fc9cbfff640 1 --2- 172.21.15.125:0/4118443865 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9b4005190 0x7fc9b400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:14:51.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.048+0000 7fc9cbfff640 1 -- 172.21.15.125:0/4118443865 reap_dead start 2023-10-30T13:14:51.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.048+0000 7fc9cb7fe640 1 --2- 172.21.15.125:0/4118443865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9b4003980 0x7fc9b4003da0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc001390 tx=0x7fc9bc01e5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62b82171c2586cd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:51.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.048+0000 7fc9ca7fc640 1 -- 172.21.15.125:0/4118443865 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc0052d0 con 0x7fc9b4003980 2023-10-30T13:14:51.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.048+0000 7fc9ca7fc640 1 -- 172.21.15.125:0/4118443865 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc0a9050 con 0x7fc9b4003980 2023-10-30T13:14:51.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.048+0000 7fc9ca7fc640 1 -- 172.21.15.125:0/4118443865 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc003650 con 0x7fc9b4003980 2023-10-30T13:14:51.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.048+0000 7fc9d09a7640 1 -- 172.21.15.125:0/4118443865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9b4003980 msgr2=0x7fc9b4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:51.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.048+0000 7fc9d09a7640 1 --2- 172.21.15.125:0/4118443865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9b4003980 0x7fc9b4003da0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc001390 tx=0x7fc9bc01e5d0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.048+0000 7fc9d09a7640 1 -- 172.21.15.125:0/4118443865 shutdown_connections 2023-10-30T13:14:51.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.048+0000 7fc9d09a7640 1 --2- 172.21.15.125:0/4118443865 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9b4003980 0x7fc9b4003da0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.048+0000 7fc9d09a7640 1 -- 172.21.15.125:0/4118443865 >> 172.21.15.125:0/4118443865 conn(0x7fc9cc0c75d0 msgr2=0x7fc9cc0c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:51.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.048+0000 7fc9d09a7640 1 -- 172.21.15.125:0/4118443865 shutdown_connections 2023-10-30T13:14:51.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.048+0000 7fc9d09a7640 1 -- 172.21.15.125:0/4118443865 wait complete. 2023-10-30T13:14:51.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9d09a7640 1 Processor -- start 2023-10-30T13:14:51.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9d09a7640 1 -- start start 2023-10-30T13:14:51.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9d09a7640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9b4003980 0x7fc9cc1477f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:51.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9d09a7640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9cc1f1fa0 0x7fc9cc14b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:51.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9d09a7640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9cc14c460 0x7fc9cc14e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:51.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9d09a7640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc14d390 con 0x7fc9cc1f1fa0 2023-10-30T13:14:51.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9d09a7640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc14d670 con 0x7fc9b4003980 2023-10-30T13:14:51.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9d09a7640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9cc14d950 con 0x7fc9cc14c460 2023-10-30T13:14:51.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9caffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9cc1f1fa0 0x7fc9cc14b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:51.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9caffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9cc1f1fa0 0x7fc9cc14b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52440/0 (socket says 172.21.15.125:52440) 2023-10-30T13:14:51.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9caffd640 1 -- 172.21.15.125:0/2194408308 learned_addr learned my addr 172.21.15.125:0/2194408308 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:14:51.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9cb7fe640 1 --2- 172.21.15.125:0/2194408308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9b4003980 0x7fc9cc1477f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:51.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9cbfff640 1 --2- 172.21.15.125:0/2194408308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9cc14c460 0x7fc9cc14e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:51.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9caffd640 1 -- 172.21.15.125:0/2194408308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9cc14c460 msgr2=0x7fc9cc14e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:51.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9caffd640 1 --2- 172.21.15.125:0/2194408308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9cc14c460 0x7fc9cc14e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9caffd640 1 -- 172.21.15.125:0/2194408308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9b4003980 msgr2=0x7fc9cc1477f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:51.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9caffd640 1 --2- 172.21.15.125:0/2194408308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9b4003980 0x7fc9cc1477f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9caffd640 1 -- 172.21.15.125:0/2194408308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9cc14dc30 con 0x7fc9cc1f1fa0 2023-10-30T13:14:51.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9caffd640 1 --2- 172.21.15.125:0/2194408308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9cc1f1fa0 0x7fc9cc14b3c0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00072c0 tx=0x7fc9c0007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:51.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9b3fff640 1 -- 172.21.15.125:0/2194408308 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c000f8c0 con 0x7fc9cc1f1fa0 2023-10-30T13:14:51.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9b3fff640 1 -- 172.21.15.125:0/2194408308 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c000fa60 con 0x7fc9cc1f1fa0 2023-10-30T13:14:51.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9d09a7640 1 -- 172.21.15.125:0/2194408308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9cc14df10 con 0x7fc9cc1f1fa0 2023-10-30T13:14:51.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9b3fff640 1 -- 172.21.15.125:0/2194408308 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9c000fe10 con 0x7fc9cc1f1fa0 2023-10-30T13:14:51.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.052+0000 7fc9d09a7640 1 -- 172.21.15.125:0/2194408308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9cc147d30 con 0x7fc9cc1f1fa0 2023-10-30T13:14:51.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.056+0000 7fc9d09a7640 1 -- 172.21.15.125:0/2194408308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9cc14e150 con 0x7fc9cc1f1fa0 2023-10-30T13:14:51.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.056+0000 7fc9b3fff640 1 -- 172.21.15.125:0/2194408308 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc9c001d460 con 0x7fc9cc1f1fa0 2023-10-30T13:14:51.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.056+0000 7fc9b3fff640 1 --2- 172.21.15.125:0/2194408308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9b80664d0 0x7fc9b8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:51.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.056+0000 7fc9b3fff640 1 -- 172.21.15.125:0/2194408308 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7fc9c0084770 con 0x7fc9cc1f1fa0 2023-10-30T13:14:51.070 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.056+0000 7fc9cb7fe640 1 --2- 172.21.15.125:0/2194408308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9b80664d0 0x7fc9b8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:51.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.056+0000 7fc9cb7fe640 1 --2- 172.21.15.125:0/2194408308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9b80664d0 0x7fc9b8068990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0ae000 tx=0x7fc9bc0011f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:51.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.060+0000 7fc9b3fff640 1 -- 172.21.15.125:0/2194408308 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc9cc14e150 con 0x7fc9cc1f1fa0 2023-10-30T13:14:51.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.244+0000 7fc9d09a7640 1 -- 172.21.15.125:0/2194408308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9cc14e150 con 0x7fc9cc1f1fa0 2023-10-30T13:14:51.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.244+0000 7fc9b3fff640 1 -- 172.21.15.125:0/2194408308 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+6551 (secure 0 0 0) 0x7fc9cc14e150 con 0x7fc9cc1f1fa0 2023-10-30T13:14:51.250 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:14:51.250 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9150,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:14:34.076142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:active","state_seq":22,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:51.251 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 213 2023-10-30T13:14:51.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.244+0000 7fc9d09a7640 1 -- 172.21.15.125:0/2194408308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9b80664d0 msgr2=0x7fc9b8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:51.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.244+0000 7fc9d09a7640 1 --2- 172.21.15.125:0/2194408308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9b80664d0 0x7fc9b8068990 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0ae000 tx=0x7fc9bc0011f0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.244+0000 7fc9d09a7640 1 -- 172.21.15.125:0/2194408308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9cc1f1fa0 msgr2=0x7fc9cc14b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:51.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.244+0000 7fc9d09a7640 1 --2- 172.21.15.125:0/2194408308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9cc1f1fa0 0x7fc9cc14b3c0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fc9c00072c0 tx=0x7fc9c0007780 comp rx=0 tx=0).stop 2023-10-30T13:14:51.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.244+0000 7fc9d09a7640 1 -- 172.21.15.125:0/2194408308 shutdown_connections 2023-10-30T13:14:51.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.244+0000 7fc9d09a7640 1 --2- 172.21.15.125:0/2194408308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc9b80664d0 0x7fc9b8068990 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.244+0000 7fc9d09a7640 1 --2- 172.21.15.125:0/2194408308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc9cc14c460 0x7fc9cc14e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.244+0000 7fc9d09a7640 1 --2- 172.21.15.125:0/2194408308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc9cc1f1fa0 0x7fc9cc14b3c0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.244+0000 7fc9d09a7640 1 --2- 172.21.15.125:0/2194408308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc9b4003980 0x7fc9cc1477f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.244+0000 7fc9d09a7640 1 -- 172.21.15.125:0/2194408308 >> 172.21.15.125:0/2194408308 conn(0x7fc9cc0c75d0 msgr2=0x7fc9cc0c8930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:51.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.244+0000 7fc9d09a7640 1 -- 172.21.15.125:0/2194408308 shutdown_connections 2023-10-30T13:14:51.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.244+0000 7fc9d09a7640 1 -- 172.21.15.125:0/2194408308 wait complete. 2023-10-30T13:14:51.271 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=1) 2023-10-30T13:14:51.272 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:14:51.272 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:14:51.297 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:14:51.298 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:51.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.428+0000 7ffa271a0640 1 Processor -- start 2023-10-30T13:14:51.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.428+0000 7ffa271a0640 1 -- start start 2023-10-30T13:14:51.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.432+0000 7ffa271a0640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffa20069e90 0x7ffa2006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:51.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.432+0000 7ffa271a0640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa200c54d0 con 0x7ffa2006ab20 2023-10-30T13:14:51.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.432+0000 7ffa271a0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa200c5a90 con 0x7ffa20153890 2023-10-30T13:14:51.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.432+0000 7ffa271a0640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa200c1cd0 con 0x7ffa20069e90 2023-10-30T13:14:51.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.432+0000 7ffa2699f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffa20069e90 0x7ffa2006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:51.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.432+0000 7ffa2699f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffa20069e90 0x7ffa2006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52442/0 (socket says 172.21.15.125:52442) 2023-10-30T13:14:51.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.432+0000 7ffa2699f640 1 -- 172.21.15.125:0/873003763 learned_addr learned my addr 172.21.15.125:0/873003763 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:14:51.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.432+0000 7ffa2699f640 1 -- 172.21.15.125:0/873003763 >> v1:172.21.15.136:6789/0 conn(0x7ffa20153890 legacy=0x7ffa20159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:51.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.432+0000 7ffa2699f640 1 -- 172.21.15.125:0/873003763 >> v1:172.21.15.125:6789/0 conn(0x7ffa2006ab20 legacy=0x7ffa200c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:51.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.432+0000 7ffa2699f640 1 -- 172.21.15.125:0/873003763 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa200c6630 con 0x7ffa20069e90 2023-10-30T13:14:51.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.436+0000 7ffa2699f640 1 --2- 172.21.15.125:0/873003763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa20069e90 0x7ffa2006a2e0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009e50 tx=0x7ffa1801ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fe95c5563b63caa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:51.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.436+0000 7ffa2519c640 1 -- 172.21.15.125:0/873003763 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa180a58e0 con 0x7ffa20069e90 2023-10-30T13:14:51.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.436+0000 7ffa2519c640 1 -- 172.21.15.125:0/873003763 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa180a5a40 con 0x7ffa20069e90 2023-10-30T13:14:51.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.436+0000 7ffa2519c640 1 -- 172.21.15.125:0/873003763 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa180a5dd0 con 0x7ffa20069e90 2023-10-30T13:14:51.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.436+0000 7ffa271a0640 1 -- 172.21.15.125:0/873003763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa20069e90 msgr2=0x7ffa2006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:51.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.436+0000 7ffa271a0640 1 --2- 172.21.15.125:0/873003763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa20069e90 0x7ffa2006a2e0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009e50 tx=0x7ffa1801ca10 comp rx=0 tx=0).stop 2023-10-30T13:14:51.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.436+0000 7ffa271a0640 1 -- 172.21.15.125:0/873003763 shutdown_connections 2023-10-30T13:14:51.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.436+0000 7ffa271a0640 1 --2- 172.21.15.125:0/873003763 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa20069e90 0x7ffa2006a2e0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.436+0000 7ffa271a0640 1 -- 172.21.15.125:0/873003763 >> 172.21.15.125:0/873003763 conn(0x7ffa200c75d0 msgr2=0x7ffa200c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:51.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.436+0000 7ffa271a0640 1 -- 172.21.15.125:0/873003763 shutdown_connections 2023-10-30T13:14:51.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.436+0000 7ffa271a0640 1 -- 172.21.15.125:0/873003763 wait complete. 2023-10-30T13:14:51.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.436+0000 7ffa271a0640 1 Processor -- start 2023-10-30T13:14:51.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.436+0000 7ffa271a0640 1 -- start start 2023-10-30T13:14:51.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa271a0640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa20069e90 0x7ffa2014b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:51.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa271a0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa2006ab20 0x7ffa2014f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:51.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa271a0640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa20153890 0x7ffa20149800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:51.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa271a0640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2014b680 con 0x7ffa20069e90 2023-10-30T13:14:51.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa271a0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa201f4bc0 con 0x7ffa2006ab20 2023-10-30T13:14:51.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa271a0640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa20152220 con 0x7ffa20153890 2023-10-30T13:14:51.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa2619e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa20069e90 0x7ffa2014b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:51.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa2619e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa20069e90 0x7ffa2014b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52448/0 (socket says 172.21.15.125:52448) 2023-10-30T13:14:51.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa2619e640 1 -- 172.21.15.125:0/231672222 learned_addr learned my addr 172.21.15.125:0/231672222 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:14:51.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa2699f640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa20153890 0x7ffa20149800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:51.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa2599d640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa2006ab20 0x7ffa2014f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:51.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa2619e640 1 -- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa20153890 msgr2=0x7ffa20149800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:51.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa2619e640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa20153890 0x7ffa20149800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa2619e640 1 -- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa2006ab20 msgr2=0x7ffa2014f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:51.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa2619e640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa2006ab20 0x7ffa2014f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa2619e640 1 -- 172.21.15.125:0/231672222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa18009b30 con 0x7ffa20069e90 2023-10-30T13:14:51.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa2699f640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa20153890 0x7ffa20149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:14:51.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa2599d640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa2006ab20 0x7ffa2014f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:14:51.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa2619e640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa20069e90 0x7ffa2014b060 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00c400 tx=0x7ffa1c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:51.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa16ffd640 1 -- 172.21.15.125:0/231672222 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c0052d0 con 0x7ffa20069e90 2023-10-30T13:14:51.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa16ffd640 1 -- 172.21.15.125:0/231672222 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa1c005470 con 0x7ffa20069e90 2023-10-30T13:14:51.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa16ffd640 1 -- 172.21.15.125:0/231672222 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c006a70 con 0x7ffa20069e90 2023-10-30T13:14:51.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa271a0640 1 -- 172.21.15.125:0/231672222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa201f4e80 con 0x7ffa20069e90 2023-10-30T13:14:51.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa271a0640 1 -- 172.21.15.125:0/231672222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa201f5440 con 0x7ffa20069e90 2023-10-30T13:14:51.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa271a0640 1 -- 172.21.15.125:0/231672222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2004e170 con 0x7ffa20069e90 2023-10-30T13:14:51.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.440+0000 7ffa16ffd640 1 -- 172.21.15.125:0/231672222 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffa1c006bd0 con 0x7ffa20069e90 2023-10-30T13:14:51.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.444+0000 7ffa16ffd640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffa040664d0 0x7ffa04068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:51.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.444+0000 7ffa16ffd640 1 -- 172.21.15.125:0/231672222 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7ffa1c083bf0 con 0x7ffa20069e90 2023-10-30T13:14:51.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.444+0000 7ffa2599d640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffa040664d0 0x7ffa04068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:51.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.444+0000 7ffa2599d640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffa040664d0 0x7ffa04068990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7ffa20145a50 tx=0x7ffa10009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:51.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.448+0000 7ffa16ffd640 1 -- 172.21.15.125:0/231672222 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffa1c050720 con 0x7ffa20069e90 2023-10-30T13:14:51.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.628+0000 7ffa271a0640 1 -- 172.21.15.125:0/231672222 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa200c15a0 con 0x7ffa20069e90 2023-10-30T13:14:51.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.632+0000 7ffa16ffd640 1 -- 172.21.15.125:0/231672222 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+6551 (secure 0 0 0) 0x7ffa1c053d40 con 0x7ffa20069e90 2023-10-30T13:14:51.637 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:14:51.637 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9150,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:14:34.076142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:active","state_seq":22,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:51.637 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 213 2023-10-30T13:14:51.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.632+0000 7ffa271a0640 1 -- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffa040664d0 msgr2=0x7ffa04068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:51.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.632+0000 7ffa271a0640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffa040664d0 0x7ffa04068990 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7ffa20145a50 tx=0x7ffa10009040 comp rx=0 tx=0).stop 2023-10-30T13:14:51.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.632+0000 7ffa271a0640 1 -- 172.21.15.125:0/231672222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa20069e90 msgr2=0x7ffa2014b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:51.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.632+0000 7ffa271a0640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa20069e90 0x7ffa2014b060 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c00c400 tx=0x7ffa1c00c8c0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.632+0000 7ffa271a0640 1 -- 172.21.15.125:0/231672222 shutdown_connections 2023-10-30T13:14:51.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.632+0000 7ffa271a0640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffa040664d0 0x7ffa04068990 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.632+0000 7ffa271a0640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa20153890 0x7ffa20149800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.632+0000 7ffa271a0640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa2006ab20 0x7ffa2014f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.632+0000 7ffa271a0640 1 --2- 172.21.15.125:0/231672222 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa20069e90 0x7ffa2014b060 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:51.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.632+0000 7ffa271a0640 1 -- 172.21.15.125:0/231672222 >> 172.21.15.125:0/231672222 conn(0x7ffa200c75d0 msgr2=0x7ffa20157a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:51.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.632+0000 7ffa271a0640 1 -- 172.21.15.125:0/231672222 shutdown_connections 2023-10-30T13:14:51.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:51.632+0000 7ffa271a0640 1 -- 172.21.15.125:0/231672222 wait complete. 2023-10-30T13:14:51.658 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:14:53.659 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:53.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb210825640 1 Processor -- start 2023-10-30T13:14:53.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb210825640 1 -- start start 2023-10-30T13:14:53.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb210825640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb20c1538a0 0x7fb20c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:53.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb210825640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb20c0ca0e0 0x7fb20c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:53.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb210825640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20c0cb800 con 0x7fb20c0cf160 2023-10-30T13:14:53.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb210825640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb20c15dc30 con 0x7fb20c1538a0 2023-10-30T13:14:53.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb210825640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c0cb330 con 0x7fb20c0ca0e0 2023-10-30T13:14:53.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb20ad9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb20c0cf160 0x7fb20c0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39544/0 (socket says 172.21.15.125:39544) 2023-10-30T13:14:53.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb20ad9d640 1 -- 172.21.15.125:0/3520669066 learned_addr learned my addr 172.21.15.125:0/3520669066 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:14:53.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb201d9b640 1 --2- 172.21.15.125:0/3520669066 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb20c0ca0e0 0x7fb20c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:53.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb20a59c640 1 --2- 172.21.15.125:0/3520669066 >> v2:172.21.15.136:3300/0 conn(0x7fb20c1538a0 0x7fb20c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:53.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb209d9b640 1 -- 172.21.15.125:0/3520669066 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1908044424 0 0) 0x7fb20c0cb800 con 0x7fb20c0cf160 2023-10-30T13:14:53.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb209d9b640 1 -- 172.21.15.125:0/3520669066 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1f8003530 con 0x7fb20c0cf160 2023-10-30T13:14:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb209d9b640 1 -- 172.21.15.125:0/3520669066 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1606616842 0 0) 0x7fb1f8003530 con 0x7fb20c0cf160 2023-10-30T13:14:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb209d9b640 1 -- 172.21.15.125:0/3520669066 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb20c0ca0e0 msgr2=0x7fb20c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb209d9b640 1 --2- 172.21.15.125:0/3520669066 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb20c0ca0e0 0x7fb20c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:53.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb209d9b640 1 -- 172.21.15.125:0/3520669066 >> v2:172.21.15.136:3300/0 conn(0x7fb20c1538a0 msgr2=0x7fb20c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb209d9b640 1 --2- 172.21.15.125:0/3520669066 >> v2:172.21.15.136:3300/0 conn(0x7fb20c1538a0 0x7fb20c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb209d9b640 1 -- 172.21.15.125:0/3520669066 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c0cad70 con 0x7fb20c0cf160 2023-10-30T13:14:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.800+0000 7fb209d9b640 1 -- 172.21.15.125:0/3520669066 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fb204001a00 con 0x7fb20c0cf160 2023-10-30T13:14:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb209d9b640 1 -- 172.21.15.125:0/3520669066 >> v1:172.21.15.125:6789/0 conn(0x7fb20c0cf160 legacy=0x7fb20c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb209d9b640 1 --2- 172.21.15.125:0/3520669066 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1f8003980 0x7fb1f8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:53.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb209d9b640 1 --2- 172.21.15.125:0/3520669066 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb20c0ca0e0 0x7fb1f80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:53.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb209d9b640 1 --2- 172.21.15.125:0/3520669066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1f8004d10 0x7fb1f8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:53.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb209d9b640 1 -- 172.21.15.125:0/3520669066 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c15dc30 con 0x7fb1f8003980 2023-10-30T13:14:53.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb209d9b640 1 -- 172.21.15.125:0/3520669066 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c0cb330 con 0x7fb1f8004d10 2023-10-30T13:14:53.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb209d9b640 1 -- 172.21.15.125:0/3520669066 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f800b990 con 0x7fb20c0ca0e0 2023-10-30T13:14:53.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb20a59c640 1 --2- 172.21.15.125:0/3520669066 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1f8003980 0x7fb1f8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:53.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb201d9b640 1 --2- 172.21.15.125:0/3520669066 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb20c0ca0e0 0x7fb1f80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:53.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb20ad9d640 1 --2- 172.21.15.125:0/3520669066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1f8004d10 0x7fb1f8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:53.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb20a59c640 1 -- 172.21.15.125:0/3520669066 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb20c0ca0e0 msgr2=0x7fb1f80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:53.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb20a59c640 1 --2- 172.21.15.125:0/3520669066 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb20c0ca0e0 0x7fb1f80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:53.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb20a59c640 1 -- 172.21.15.125:0/3520669066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1f8004d10 msgr2=0x7fb1f8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:53.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb20a59c640 1 --2- 172.21.15.125:0/3520669066 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1f8004d10 0x7fb1f8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:53.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb20a59c640 1 -- 172.21.15.125:0/3520669066 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f8003530 con 0x7fb1f8003980 2023-10-30T13:14:53.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb20a59c640 1 --2- 172.21.15.125:0/3520669066 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1f8003980 0x7fb1f8003da0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009ae0 tx=0x7fb1f401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=947878de90af2fa7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:53.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb209d9b640 1 -- 172.21.15.125:0/3520669066 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f40a5570 con 0x7fb1f8003980 2023-10-30T13:14:53.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb209d9b640 1 -- 172.21.15.125:0/3520669066 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f40a56d0 con 0x7fb1f8003980 2023-10-30T13:14:53.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.804+0000 7fb209d9b640 1 -- 172.21.15.125:0/3520669066 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f40a5a80 con 0x7fb1f8003980 2023-10-30T13:14:53.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb210825640 1 -- 172.21.15.125:0/3520669066 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1f8003980 msgr2=0x7fb1f8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:53.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb210825640 1 --2- 172.21.15.125:0/3520669066 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1f8003980 0x7fb1f8003da0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009ae0 tx=0x7fb1f401ca90 comp rx=0 tx=0).stop 2023-10-30T13:14:53.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb20ad9d640 1 -- 172.21.15.125:0/3520669066 reap_dead start 2023-10-30T13:14:53.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb210825640 1 -- 172.21.15.125:0/3520669066 shutdown_connections 2023-10-30T13:14:53.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb210825640 1 -- 172.21.15.125:0/3520669066 >> 172.21.15.125:0/3520669066 conn(0x7fb20c06bfe0 msgr2=0x7fb20c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:53.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb210825640 1 -- 172.21.15.125:0/3520669066 shutdown_connections 2023-10-30T13:14:53.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb210825640 1 -- 172.21.15.125:0/3520669066 wait complete. 2023-10-30T13:14:53.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb210825640 1 Processor -- start 2023-10-30T13:14:53.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb210825640 1 -- start start 2023-10-30T13:14:53.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb210825640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1f8003980 0x7fb20c14e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:53.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb210825640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb20c1e3ff0 0x7fb20c1f2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:53.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb210825640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb20c152760 0x7fb20c152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:53.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb210825640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c14ea00 con 0x7fb20c152760 2023-10-30T13:14:53.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb210825640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c1e4fc0 con 0x7fb20c1e3ff0 2023-10-30T13:14:53.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb210825640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20c15af10 con 0x7fb1f8003980 2023-10-30T13:14:53.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb20ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb20c152760 0x7fb20c152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:53.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb20ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb20c152760 0x7fb20c152be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52458/0 (socket says 172.21.15.125:52458) 2023-10-30T13:14:53.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb20ad9d640 1 -- 172.21.15.125:0/3485230400 learned_addr learned my addr 172.21.15.125:0/3485230400 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:14:53.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb20a59c640 1 --2- 172.21.15.125:0/3485230400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1f8003980 0x7fb20c14e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:53.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.808+0000 7fb201d9b640 1 --2- 172.21.15.125:0/3485230400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb20c1e3ff0 0x7fb20c1f2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:53.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb20ad9d640 1 -- 172.21.15.125:0/3485230400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1f8003980 msgr2=0x7fb20c14e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:53.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb20ad9d640 1 --2- 172.21.15.125:0/3485230400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1f8003980 0x7fb20c14e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:53.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb20ad9d640 1 -- 172.21.15.125:0/3485230400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb20c1e3ff0 msgr2=0x7fb20c1f2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:53.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb20ad9d640 1 --2- 172.21.15.125:0/3485230400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb20c1e3ff0 0x7fb20c1f2270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:53.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb20ad9d640 1 -- 172.21.15.125:0/3485230400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c15a950 con 0x7fb20c152760 2023-10-30T13:14:53.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb20ad9d640 1 --2- 172.21.15.125:0/3485230400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb20c152760 0x7fb20c152be0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0ccc90 tx=0x7fb204011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:53.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb2037fe640 1 -- 172.21.15.125:0/3485230400 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20400f920 con 0x7fb20c152760 2023-10-30T13:14:53.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb2037fe640 1 -- 172.21.15.125:0/3485230400 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20400faa0 con 0x7fb20c152760 2023-10-30T13:14:53.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb2037fe640 1 -- 172.21.15.125:0/3485230400 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb20400fe50 con 0x7fb20c152760 2023-10-30T13:14:53.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb210825640 1 -- 172.21.15.125:0/3485230400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb20c15ac30 con 0x7fb20c152760 2023-10-30T13:14:53.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb210825640 1 -- 172.21.15.125:0/3485230400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb20c1e3c40 con 0x7fb20c152760 2023-10-30T13:14:53.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb210825640 1 -- 172.21.15.125:0/3485230400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb20c1e4e20 con 0x7fb20c152760 2023-10-30T13:14:53.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb2037fe640 1 -- 172.21.15.125:0/3485230400 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb20401f460 con 0x7fb20c152760 2023-10-30T13:14:53.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb2037fe640 1 --2- 172.21.15.125:0/3485230400 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1ec0664a0 0x7fb1ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:53.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb2037fe640 1 -- 172.21.15.125:0/3485230400 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7fb204087740 con 0x7fb20c152760 2023-10-30T13:14:53.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.812+0000 7fb20a59c640 1 --2- 172.21.15.125:0/3485230400 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1ec0664a0 0x7fb1ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:53.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.816+0000 7fb20a59c640 1 --2- 172.21.15.125:0/3485230400 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1ec0664a0 0x7fb1ec068960 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb1f40af000 tx=0x7fb1f40046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:53.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:53.816+0000 7fb2037fe640 1 -- 172.21.15.125:0/3485230400 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb20c1e4e20 con 0x7fb20c152760 2023-10-30T13:14:54.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:54.000+0000 7fb210825640 1 -- 172.21.15.125:0/3485230400 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb20c1e4e20 con 0x7fb20c152760 2023-10-30T13:14:54.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:54.000+0000 7fb2037fe640 1 -- 172.21.15.125:0/3485230400 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+6551 (secure 0 0 0) 0x7fb20c1e4e20 con 0x7fb20c152760 2023-10-30T13:14:54.007 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:14:54.008 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9150,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:14:34.076142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:active","state_seq":22,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:54.008 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 213 2023-10-30T13:14:54.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:54.004+0000 7fb210825640 1 -- 172.21.15.125:0/3485230400 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1ec0664a0 msgr2=0x7fb1ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:54.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:54.004+0000 7fb210825640 1 --2- 172.21.15.125:0/3485230400 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1ec0664a0 0x7fb1ec068960 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fb1f40af000 tx=0x7fb1f40046d0 comp rx=0 tx=0).stop 2023-10-30T13:14:54.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:54.004+0000 7fb210825640 1 -- 172.21.15.125:0/3485230400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb20c152760 msgr2=0x7fb20c152be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:54.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:54.004+0000 7fb210825640 1 --2- 172.21.15.125:0/3485230400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb20c152760 0x7fb20c152be0 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fb20c0ccc90 tx=0x7fb204011c20 comp rx=0 tx=0).stop 2023-10-30T13:14:54.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:54.004+0000 7fb210825640 1 -- 172.21.15.125:0/3485230400 shutdown_connections 2023-10-30T13:14:54.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:54.004+0000 7fb210825640 1 --2- 172.21.15.125:0/3485230400 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1ec0664a0 0x7fb1ec068960 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:54.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:54.004+0000 7fb210825640 1 --2- 172.21.15.125:0/3485230400 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb20c152760 0x7fb20c152be0 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:54.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:54.004+0000 7fb210825640 1 --2- 172.21.15.125:0/3485230400 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb20c1e3ff0 0x7fb20c1f2270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:54.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:54.004+0000 7fb210825640 1 --2- 172.21.15.125:0/3485230400 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1f8003980 0x7fb20c14e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:54.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:54.004+0000 7fb210825640 1 -- 172.21.15.125:0/3485230400 >> 172.21.15.125:0/3485230400 conn(0x7fb20c06bfe0 msgr2=0x7fb20c150e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:54.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:54.004+0000 7fb210825640 1 -- 172.21.15.125:0/3485230400 shutdown_connections 2023-10-30T13:14:54.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:54.004+0000 7fb210825640 1 -- 172.21.15.125:0/3485230400 wait complete. 2023-10-30T13:14:54.028 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:14:56.030 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:56.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.172+0000 7ff13fc91640 1 Processor -- start 2023-10-30T13:14:56.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.172+0000 7ff13fc91640 1 -- start start 2023-10-30T13:14:56.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.172+0000 7ff13fc91640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff138070740 0x7ff1380c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:56.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.172+0000 7ff13fc91640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff13815e700 0x7ff13806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:56.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.172+0000 7ff13fc91640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1380c5810 con 0x7ff1381580e0 2023-10-30T13:14:56.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.172+0000 7ff13fc91640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff138070b60 con 0x7ff13815e700 2023-10-30T13:14:56.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.172+0000 7ff13fc91640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff13815eb90 con 0x7ff138070740 2023-10-30T13:14:56.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13da2d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff1381580e0 0x7ff138069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39546/0 (socket says 172.21.15.125:39546) 2023-10-30T13:14:56.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13da2d640 1 -- 172.21.15.125:0/1535978772 learned_addr learned my addr 172.21.15.125:0/1535978772 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:14:56.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13d22c640 1 --2- 172.21.15.125:0/1535978772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff138070740 0x7ff1380c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:56.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13e22e640 1 --2- 172.21.15.125:0/1535978772 >> v2:172.21.15.136:3300/0 conn(0x7ff13815e700 0x7ff13806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:56.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 -- 172.21.15.125:0/1535978772 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2738440801 0 0) 0x7ff1380c5810 con 0x7ff1381580e0 2023-10-30T13:14:56.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 -- 172.21.15.125:0/1535978772 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff124003530 con 0x7ff1381580e0 2023-10-30T13:14:56.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 -- 172.21.15.125:0/1535978772 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2337335191 0 0) 0x7ff124003530 con 0x7ff1381580e0 2023-10-30T13:14:56.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 -- 172.21.15.125:0/1535978772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff138070740 msgr2=0x7ff1380c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:56.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 --2- 172.21.15.125:0/1535978772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff138070740 0x7ff1380c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:56.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 -- 172.21.15.125:0/1535978772 >> v2:172.21.15.136:3300/0 conn(0x7ff13815e700 msgr2=0x7ff13806a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:56.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 --2- 172.21.15.125:0/1535978772 >> v2:172.21.15.136:3300/0 conn(0x7ff13815e700 0x7ff13806a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:56.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 -- 172.21.15.125:0/1535978772 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1380c6bb0 con 0x7ff1381580e0 2023-10-30T13:14:56.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 -- 172.21.15.125:0/1535978772 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7ff134001680 con 0x7ff1381580e0 2023-10-30T13:14:56.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 -- 172.21.15.125:0/1535978772 >> v1:172.21.15.125:6789/0 conn(0x7ff1381580e0 legacy=0x7ff138069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:56.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 --2- 172.21.15.125:0/1535978772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff124003980 0x7ff124003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:56.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 --2- 172.21.15.125:0/1535978772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1240047d0 0x7ff124004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:56.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 --2- 172.21.15.125:0/1535978772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff138070740 0x7ff124005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:56.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 -- 172.21.15.125:0/1535978772 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff138070b60 con 0x7ff1240047d0 2023-10-30T13:14:56.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 -- 172.21.15.125:0/1535978772 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13815eb90 con 0x7ff124003980 2023-10-30T13:14:56.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13ca2b640 1 -- 172.21.15.125:0/1535978772 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff12400b990 con 0x7ff138070740 2023-10-30T13:14:56.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13d22c640 1 --2- 172.21.15.125:0/1535978772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1240047d0 0x7ff124004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:56.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13da2d640 1 --2- 172.21.15.125:0/1535978772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff124003980 0x7ff124003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:56.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13e22e640 1 --2- 172.21.15.125:0/1535978772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff138070740 0x7ff124005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:56.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13d22c640 1 -- 172.21.15.125:0/1535978772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff138070740 msgr2=0x7ff124005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:56.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13d22c640 1 --2- 172.21.15.125:0/1535978772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff138070740 0x7ff124005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:56.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13d22c640 1 -- 172.21.15.125:0/1535978772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff124003980 msgr2=0x7ff124003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:56.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13d22c640 1 --2- 172.21.15.125:0/1535978772 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff124003980 0x7ff124003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:56.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13d22c640 1 -- 172.21.15.125:0/1535978772 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff124003530 con 0x7ff1240047d0 2023-10-30T13:14:56.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.176+0000 7ff13e22e640 1 --2- 172.21.15.125:0/1535978772 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff138070740 0x7ff124005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:14:56.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.180+0000 7ff13d22c640 1 --2- 172.21.15.125:0/1535978772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1240047d0 0x7ff124004c50 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7ff128009ae0 tx=0x7ff12801ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=949d4ae3f2ccb5e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:56.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.180+0000 7ff13ca2b640 1 -- 172.21.15.125:0/1535978772 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1280a5560 con 0x7ff1240047d0 2023-10-30T13:14:56.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.180+0000 7ff13ca2b640 1 -- 172.21.15.125:0/1535978772 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1280a56c0 con 0x7ff1240047d0 2023-10-30T13:14:56.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.180+0000 7ff13ca2b640 1 -- 172.21.15.125:0/1535978772 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1280a5a70 con 0x7ff1240047d0 2023-10-30T13:14:56.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.180+0000 7ff13fc91640 1 -- 172.21.15.125:0/1535978772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1240047d0 msgr2=0x7ff124004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:56.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.180+0000 7ff13fc91640 1 --2- 172.21.15.125:0/1535978772 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1240047d0 0x7ff124004c50 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7ff128009ae0 tx=0x7ff12801ca50 comp rx=0 tx=0).stop 2023-10-30T13:14:56.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.180+0000 7ff13e22e640 1 -- 172.21.15.125:0/1535978772 reap_dead start 2023-10-30T13:14:56.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.180+0000 7ff13fc91640 1 -- 172.21.15.125:0/1535978772 shutdown_connections 2023-10-30T13:14:56.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.180+0000 7ff13fc91640 1 -- 172.21.15.125:0/1535978772 >> 172.21.15.125:0/1535978772 conn(0x7ff1380c7910 msgr2=0x7ff1380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:56.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.180+0000 7ff13fc91640 1 -- 172.21.15.125:0/1535978772 shutdown_connections 2023-10-30T13:14:56.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.180+0000 7ff13fc91640 1 -- 172.21.15.125:0/1535978772 wait complete. 2023-10-30T13:14:56.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.180+0000 7ff13fc91640 1 Processor -- start 2023-10-30T13:14:56.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.180+0000 7ff13fc91640 1 -- start start 2023-10-30T13:14:56.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13fc91640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1240047d0 0x7ff13814c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:56.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13fc91640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff138161bd0 0x7ff138153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:56.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13fc91640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1381539c0 0x7ff138152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:56.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13fc91640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1381544e0 con 0x7ff1381539c0 2023-10-30T13:14:56.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13fc91640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff138153e40 con 0x7ff138161bd0 2023-10-30T13:14:56.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13fc91640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1380c09b0 con 0x7ff1240047d0 2023-10-30T13:14:56.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13e22e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1381539c0 0x7ff138152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:56.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13e22e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1381539c0 0x7ff138152520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52472/0 (socket says 172.21.15.125:52472) 2023-10-30T13:14:56.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13e22e640 1 -- 172.21.15.125:0/476175570 learned_addr learned my addr 172.21.15.125:0/476175570 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:14:56.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13d22c640 1 --2- 172.21.15.125:0/476175570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff138161bd0 0x7ff138153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:56.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13da2d640 1 --2- 172.21.15.125:0/476175570 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1240047d0 0x7ff13814c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:56.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13e22e640 1 -- 172.21.15.125:0/476175570 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1240047d0 msgr2=0x7ff13814c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:56.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13e22e640 1 --2- 172.21.15.125:0/476175570 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1240047d0 0x7ff13814c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:56.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13e22e640 1 -- 172.21.15.125:0/476175570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff138161bd0 msgr2=0x7ff138153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:56.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13e22e640 1 --2- 172.21.15.125:0/476175570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff138161bd0 0x7ff138153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:56.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13e22e640 1 -- 172.21.15.125:0/476175570 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1380c0c70 con 0x7ff1381539c0 2023-10-30T13:14:56.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13e22e640 1 --2- 172.21.15.125:0/476175570 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1381539c0 0x7ff138152520 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7ff1380c54b0 tx=0x7ff130010590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:56.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff12e7fc640 1 -- 172.21.15.125:0/476175570 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13001b070 con 0x7ff1381539c0 2023-10-30T13:14:56.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff12e7fc640 1 -- 172.21.15.125:0/476175570 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff13000f040 con 0x7ff1381539c0 2023-10-30T13:14:56.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff12e7fc640 1 -- 172.21.15.125:0/476175570 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13000e050 con 0x7ff1381539c0 2023-10-30T13:14:56.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13fc91640 1 -- 172.21.15.125:0/476175570 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1381f6780 con 0x7ff1381539c0 2023-10-30T13:14:56.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13fc91640 1 -- 172.21.15.125:0/476175570 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1381f6d40 con 0x7ff1381539c0 2023-10-30T13:14:56.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff13fc91640 1 -- 172.21.15.125:0/476175570 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff13804e170 con 0x7ff1381539c0 2023-10-30T13:14:56.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.184+0000 7ff12e7fc640 1 -- 172.21.15.125:0/476175570 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff1300039a0 con 0x7ff1381539c0 2023-10-30T13:14:56.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.188+0000 7ff12e7fc640 1 --2- 172.21.15.125:0/476175570 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff11c0664a0 0x7ff11c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:56.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.188+0000 7ff12e7fc640 1 -- 172.21.15.125:0/476175570 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7ff130084c20 con 0x7ff1381539c0 2023-10-30T13:14:56.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.188+0000 7ff13da2d640 1 --2- 172.21.15.125:0/476175570 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff11c0664a0 0x7ff11c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:56.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.188+0000 7ff13da2d640 1 --2- 172.21.15.125:0/476175570 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff11c0664a0 0x7ff11c068960 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7ff134001060 tx=0x7ff134007900 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:56.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.192+0000 7ff12e7fc640 1 -- 172.21.15.125:0/476175570 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff130051830 con 0x7ff1381539c0 2023-10-30T13:14:56.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.372+0000 7ff13fc91640 1 -- 172.21.15.125:0/476175570 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff138162110 con 0x7ff1381539c0 2023-10-30T13:14:56.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.372+0000 7ff12e7fc640 1 -- 172.21.15.125:0/476175570 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+6551 (secure 0 0 0) 0x7ff130054e50 con 0x7ff1381539c0 2023-10-30T13:14:56.378 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:14:56.378 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9150,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:14:34.076142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:active","state_seq":22,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:56.379 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 213 2023-10-30T13:14:56.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.372+0000 7ff13fc91640 1 -- 172.21.15.125:0/476175570 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff11c0664a0 msgr2=0x7ff11c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:56.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.372+0000 7ff13fc91640 1 --2- 172.21.15.125:0/476175570 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff11c0664a0 0x7ff11c068960 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7ff134001060 tx=0x7ff134007900 comp rx=0 tx=0).stop 2023-10-30T13:14:56.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.372+0000 7ff13fc91640 1 -- 172.21.15.125:0/476175570 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1381539c0 msgr2=0x7ff138152520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:56.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.372+0000 7ff13fc91640 1 --2- 172.21.15.125:0/476175570 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1381539c0 0x7ff138152520 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7ff1380c54b0 tx=0x7ff130010590 comp rx=0 tx=0).stop 2023-10-30T13:14:56.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.372+0000 7ff13fc91640 1 -- 172.21.15.125:0/476175570 shutdown_connections 2023-10-30T13:14:56.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.372+0000 7ff13fc91640 1 --2- 172.21.15.125:0/476175570 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff11c0664a0 0x7ff11c068960 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:56.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.372+0000 7ff13fc91640 1 --2- 172.21.15.125:0/476175570 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1381539c0 0x7ff138152520 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:56.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.372+0000 7ff13fc91640 1 --2- 172.21.15.125:0/476175570 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff138161bd0 0x7ff138153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:56.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.372+0000 7ff13fc91640 1 --2- 172.21.15.125:0/476175570 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1240047d0 0x7ff13814c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:56.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.372+0000 7ff13fc91640 1 -- 172.21.15.125:0/476175570 >> 172.21.15.125:0/476175570 conn(0x7ff1380c7910 msgr2=0x7ff138071990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:56.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.372+0000 7ff13fc91640 1 -- 172.21.15.125:0/476175570 shutdown_connections 2023-10-30T13:14:56.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:56.372+0000 7ff13fc91640 1 -- 172.21.15.125:0/476175570 wait complete. 2023-10-30T13:14:56.399 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:14:58.400 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:14:58.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.540+0000 7eff30617640 1 Processor -- start 2023-10-30T13:14:58.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.540+0000 7eff30617640 1 -- start start 2023-10-30T13:14:58.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.540+0000 7eff30617640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7eff2806f7a0 0x7eff2806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:58.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.540+0000 7eff30617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff28071110 0x7eff28158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:58.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff30617640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff28066cf0 con 0x7eff280c6bd0 2023-10-30T13:14:58.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff30617640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff280700c0 con 0x7eff2806f7a0 2023-10-30T13:14:58.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff30617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff280c70d0 con 0x7eff28071110 2023-10-30T13:14:58.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2dbb2640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7eff280c6bd0 0x7eff28069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39550/0 (socket says 172.21.15.125:39550) 2023-10-30T13:14:58.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2dbb2640 1 -- 172.21.15.125:0/277228507 learned_addr learned my addr 172.21.15.125:0/277228507 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:14:58.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2e3b3640 1 --2- 172.21.15.125:0/277228507 >> v2:172.21.15.136:3300/0 conn(0x7eff2806f7a0 0x7eff2806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:58.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2ebb4640 1 --2- 172.21.15.125:0/277228507 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff28071110 0x7eff28158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:58.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 -- 172.21.15.125:0/277228507 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015715296 0 0) 0x7eff28066cf0 con 0x7eff280c6bd0 2023-10-30T13:14:58.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 -- 172.21.15.125:0/277228507 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff14003530 con 0x7eff280c6bd0 2023-10-30T13:14:58.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 -- 172.21.15.125:0/277228507 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1838712950 0 0) 0x7eff14003530 con 0x7eff280c6bd0 2023-10-30T13:14:58.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 -- 172.21.15.125:0/277228507 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff28071110 msgr2=0x7eff28158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:58.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 --2- 172.21.15.125:0/277228507 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff28071110 0x7eff28158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:58.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 -- 172.21.15.125:0/277228507 >> v2:172.21.15.136:3300/0 conn(0x7eff2806f7a0 msgr2=0x7eff2806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:58.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 --2- 172.21.15.125:0/277228507 >> v2:172.21.15.136:3300/0 conn(0x7eff2806f7a0 0x7eff2806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:58.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 -- 172.21.15.125:0/277228507 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff280c7390 con 0x7eff280c6bd0 2023-10-30T13:14:58.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 -- 172.21.15.125:0/277228507 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7eff18001680 con 0x7eff280c6bd0 2023-10-30T13:14:58.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 -- 172.21.15.125:0/277228507 >> v1:172.21.15.125:6789/0 conn(0x7eff280c6bd0 legacy=0x7eff28069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:58.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 --2- 172.21.15.125:0/277228507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff14003980 0x7eff14003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:58.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 --2- 172.21.15.125:0/277228507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff140047d0 0x7eff14004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:58.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 --2- 172.21.15.125:0/277228507 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff28071110 0x7eff14005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:58.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 -- 172.21.15.125:0/277228507 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff280700c0 con 0x7eff14003980 2023-10-30T13:14:58.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 -- 172.21.15.125:0/277228507 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff280c70d0 con 0x7eff140047d0 2023-10-30T13:14:58.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2d3b1640 1 -- 172.21.15.125:0/277228507 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff1400b990 con 0x7eff28071110 2023-10-30T13:14:58.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2e3b3640 1 --2- 172.21.15.125:0/277228507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff14003980 0x7eff14003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:58.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2ebb4640 1 --2- 172.21.15.125:0/277228507 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff28071110 0x7eff14005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:58.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2dbb2640 1 --2- 172.21.15.125:0/277228507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff140047d0 0x7eff14004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:58.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2e3b3640 1 -- 172.21.15.125:0/277228507 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff28071110 msgr2=0x7eff14005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:58.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2e3b3640 1 --2- 172.21.15.125:0/277228507 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff28071110 0x7eff14005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:58.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2e3b3640 1 -- 172.21.15.125:0/277228507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff140047d0 msgr2=0x7eff14004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:14:58.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2e3b3640 1 --2- 172.21.15.125:0/277228507 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff140047d0 0x7eff14004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:58.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.544+0000 7eff2e3b3640 1 -- 172.21.15.125:0/277228507 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff14003530 con 0x7eff14003980 2023-10-30T13:14:58.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.548+0000 7eff2e3b3640 1 --2- 172.21.15.125:0/277228507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff14003980 0x7eff14003da0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7eff24009ae0 tx=0x7eff2401ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=829e91aee664b977 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:58.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.548+0000 7eff2d3b1640 1 -- 172.21.15.125:0/277228507 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff240a64a0 con 0x7eff14003980 2023-10-30T13:14:58.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.548+0000 7eff2d3b1640 1 -- 172.21.15.125:0/277228507 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff240a6600 con 0x7eff14003980 2023-10-30T13:14:58.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.548+0000 7eff2d3b1640 1 -- 172.21.15.125:0/277228507 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff240a69b0 con 0x7eff14003980 2023-10-30T13:14:58.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.548+0000 7eff30617640 1 -- 172.21.15.125:0/277228507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff14003980 msgr2=0x7eff14003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:58.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.548+0000 7eff30617640 1 --2- 172.21.15.125:0/277228507 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff14003980 0x7eff14003da0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7eff24009ae0 tx=0x7eff2401ca90 comp rx=0 tx=0).stop 2023-10-30T13:14:58.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.548+0000 7eff2ebb4640 1 -- 172.21.15.125:0/277228507 reap_dead start 2023-10-30T13:14:58.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.548+0000 7eff30617640 1 -- 172.21.15.125:0/277228507 shutdown_connections 2023-10-30T13:14:58.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.548+0000 7eff30617640 1 -- 172.21.15.125:0/277228507 >> 172.21.15.125:0/277228507 conn(0x7eff280c7910 msgr2=0x7eff280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:58.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.548+0000 7eff30617640 1 -- 172.21.15.125:0/277228507 shutdown_connections 2023-10-30T13:14:58.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.548+0000 7eff30617640 1 -- 172.21.15.125:0/277228507 wait complete. 2023-10-30T13:14:58.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.548+0000 7eff30617640 1 Processor -- start 2023-10-30T13:14:58.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.548+0000 7eff30617640 1 -- start start 2023-10-30T13:14:58.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff30617640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff14003980 0x7eff281f1a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:58.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff30617640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff280c1250 0x7eff2814d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:58.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff30617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff280c18f0 0x7eff2814d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:58.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff30617640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff28143780 con 0x7eff14003980 2023-10-30T13:14:58.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff30617640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff281f23e0 con 0x7eff280c1250 2023-10-30T13:14:58.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff30617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff281f2680 con 0x7eff280c18f0 2023-10-30T13:14:58.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff2e3b3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff14003980 0x7eff281f1a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:58.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff2e3b3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff14003980 0x7eff281f1a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52478/0 (socket says 172.21.15.125:52478) 2023-10-30T13:14:58.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff2e3b3640 1 -- 172.21.15.125:0/1590760847 learned_addr learned my addr 172.21.15.125:0/1590760847 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:14:58.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff2dbb2640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff280c1250 0x7eff2814d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:58.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff2ebb4640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff280c18f0 0x7eff2814d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:58.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff2e3b3640 1 -- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff280c18f0 msgr2=0x7eff2814d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:58.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff2e3b3640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff280c18f0 0x7eff2814d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:58.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff2e3b3640 1 -- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff280c1250 msgr2=0x7eff2814d070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:58.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff2e3b3640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff280c1250 0x7eff2814d070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:58.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff2e3b3640 1 -- 172.21.15.125:0/1590760847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff280c00e0 con 0x7eff14003980 2023-10-30T13:14:58.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff2dbb2640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff280c1250 0x7eff2814d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:14:58.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff2e3b3640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff14003980 0x7eff281f1a90 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7eff24009ae0 tx=0x7eff14005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:58.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff1effd640 1 -- 172.21.15.125:0/1590760847 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff240b5070 con 0x7eff14003980 2023-10-30T13:14:58.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff1effd640 1 -- 172.21.15.125:0/1590760847 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2400f000 con 0x7eff14003980 2023-10-30T13:14:58.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff30617640 1 -- 172.21.15.125:0/1590760847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff281f2940 con 0x7eff14003980 2023-10-30T13:14:58.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff1effd640 1 -- 172.21.15.125:0/1590760847 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff240a64a0 con 0x7eff14003980 2023-10-30T13:14:58.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff30617640 1 -- 172.21.15.125:0/1590760847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff281f2f00 con 0x7eff14003980 2023-10-30T13:14:58.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff30617640 1 -- 172.21.15.125:0/1590760847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff2804e170 con 0x7eff14003980 2023-10-30T13:14:58.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.552+0000 7eff1effd640 1 -- 172.21.15.125:0/1590760847 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7eff240b9070 con 0x7eff14003980 2023-10-30T13:14:58.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.556+0000 7eff1effd640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff0c0664a0 0x7eff0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:14:58.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.556+0000 7eff1effd640 1 -- 172.21.15.125:0/1590760847 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7eff2411e640 con 0x7eff14003980 2023-10-30T13:14:58.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.556+0000 7eff2dbb2640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff0c0664a0 0x7eff0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:14:58.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.556+0000 7eff2dbb2640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff0c0664a0 0x7eff0c068960 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7eff18001060 tx=0x7eff18009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:14:58.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.560+0000 7eff1effd640 1 -- 172.21.15.125:0/1590760847 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7eff240eb250 con 0x7eff14003980 2023-10-30T13:14:58.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.740+0000 7eff30617640 1 -- 172.21.15.125:0/1590760847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff28143780 con 0x7eff14003980 2023-10-30T13:14:58.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.740+0000 7eff1effd640 1 -- 172.21.15.125:0/1590760847 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+6551 (secure 0 0 0) 0x7eff240c0070 con 0x7eff14003980 2023-10-30T13:14:58.746 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:14:58.746 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9150,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:14:34.076142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:active","state_seq":22,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:14:58.747 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 213 2023-10-30T13:14:58.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.740+0000 7eff30617640 1 -- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff0c0664a0 msgr2=0x7eff0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:58.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.740+0000 7eff30617640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff0c0664a0 0x7eff0c068960 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7eff18001060 tx=0x7eff18009040 comp rx=0 tx=0).stop 2023-10-30T13:14:58.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.740+0000 7eff30617640 1 -- 172.21.15.125:0/1590760847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff14003980 msgr2=0x7eff281f1a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:14:58.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.740+0000 7eff30617640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff14003980 0x7eff281f1a90 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7eff24009ae0 tx=0x7eff14005800 comp rx=0 tx=0).stop 2023-10-30T13:14:58.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.740+0000 7eff30617640 1 -- 172.21.15.125:0/1590760847 shutdown_connections 2023-10-30T13:14:58.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.740+0000 7eff30617640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff0c0664a0 0x7eff0c068960 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:58.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.740+0000 7eff30617640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff280c18f0 0x7eff2814d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:58.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.740+0000 7eff30617640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff280c1250 0x7eff2814d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:58.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.740+0000 7eff30617640 1 --2- 172.21.15.125:0/1590760847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff14003980 0x7eff281f1a90 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:14:58.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.740+0000 7eff30617640 1 -- 172.21.15.125:0/1590760847 >> 172.21.15.125:0/1590760847 conn(0x7eff280c7910 msgr2=0x7eff280c9540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:14:58.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.740+0000 7eff30617640 1 -- 172.21.15.125:0/1590760847 shutdown_connections 2023-10-30T13:14:58.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:14:58.740+0000 7eff30617640 1 -- 172.21.15.125:0/1590760847 wait complete. 2023-10-30T13:14:58.767 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:15:00.768 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:00.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04b5091640 1 Processor -- start 2023-10-30T13:15:00.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04b5091640 1 -- start start 2023-10-30T13:15:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04b5091640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f04b00c6b80 0x7f04b00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04b5091640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04b013f4a0 0x7f04b00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04b5091640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b00c0a50 con 0x7f04b00c6f60 2023-10-30T13:15:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04b5091640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b006f6e0 con 0x7f04b00c6b80 2023-10-30T13:15:00.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04b5091640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b006f9c0 con 0x7f04b013f4a0 2023-10-30T13:15:00.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04aed9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f04b00c6b80 0x7f04b00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:00.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04af59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04b013f4a0 0x7f04b00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:00.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04aed9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f04b00c6b80 0x7f04b00bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:45900/0 (socket says 172.21.15.125:45900) 2023-10-30T13:15:00.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04aed9d640 1 -- 172.21.15.125:0/174820367 learned_addr learned my addr 172.21.15.125:0/174820367 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:00.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04aed9d640 1 -- 172.21.15.125:0/174820367 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04b013f4a0 msgr2=0x7f04b00c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:00.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04aed9d640 1 --2- 172.21.15.125:0/174820367 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04b013f4a0 0x7f04b00c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:00.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04aed9d640 1 -- 172.21.15.125:0/174820367 >> v1:172.21.15.136:6789/0 conn(0x7f04b00c6f60 legacy=0x7f04b013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:00.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.904+0000 7f04aed9d640 1 -- 172.21.15.125:0/174820367 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b006fca0 con 0x7f04b00c6b80 2023-10-30T13:15:00.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.908+0000 7f04aed9d640 1 --2- 172.21.15.125:0/174820367 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04b00c6b80 0x7f04b00bfec0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009870 tx=0x7f04a001c540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=798840835a67b56b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:00.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.908+0000 7f04add9b640 1 -- 172.21.15.125:0/174820367 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a00ab650 con 0x7f04b00c6b80 2023-10-30T13:15:00.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.908+0000 7f04add9b640 1 -- 172.21.15.125:0/174820367 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a00ab7b0 con 0x7f04b00c6b80 2023-10-30T13:15:00.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.908+0000 7f04add9b640 1 -- 172.21.15.125:0/174820367 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a00abb40 con 0x7f04b00c6b80 2023-10-30T13:15:00.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.908+0000 7f04b5091640 1 -- 172.21.15.125:0/174820367 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04b00c6b80 msgr2=0x7f04b00bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:00.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.908+0000 7f04b5091640 1 --2- 172.21.15.125:0/174820367 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04b00c6b80 0x7f04b00bfec0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009870 tx=0x7f04a001c540 comp rx=0 tx=0).stop 2023-10-30T13:15:00.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.908+0000 7f04b5091640 1 -- 172.21.15.125:0/174820367 shutdown_connections 2023-10-30T13:15:00.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.908+0000 7f04b5091640 1 --2- 172.21.15.125:0/174820367 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04b013f4a0 0x7f04b00c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:00.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.908+0000 7f04b5091640 1 --2- 172.21.15.125:0/174820367 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04b00c6b80 0x7f04b00bfec0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:00.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.908+0000 7f04b5091640 1 -- 172.21.15.125:0/174820367 >> 172.21.15.125:0/174820367 conn(0x7f04b00c7910 msgr2=0x7f04b00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:00.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.908+0000 7f04b5091640 1 -- 172.21.15.125:0/174820367 shutdown_connections 2023-10-30T13:15:00.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.908+0000 7f04b5091640 1 -- 172.21.15.125:0/174820367 wait complete. 2023-10-30T13:15:00.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04b5091640 1 Processor -- start 2023-10-30T13:15:00.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04b5091640 1 -- start start 2023-10-30T13:15:00.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04b5091640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04b00c6b80 0x7f04b01380d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:00.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04b5091640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04b00c6f60 0x7f04b013c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:00.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04b5091640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04b013f4a0 0x7f04b013c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:00.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04b5091640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b01f6750 con 0x7f04b013f4a0 2023-10-30T13:15:00.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04b5091640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b013a9d0 con 0x7f04b00c6b80 2023-10-30T13:15:00.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04b5091640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b013b510 con 0x7f04b00c6f60 2023-10-30T13:15:00.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04af59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04b013f4a0 0x7f04b013c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:00.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04af59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04b013f4a0 0x7f04b013c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42690/0 (socket says 172.21.15.125:42690) 2023-10-30T13:15:00.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04af59e640 1 -- 172.21.15.125:0/3052928065 learned_addr learned my addr 172.21.15.125:0/3052928065 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:00.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04aed9d640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04b00c6b80 0x7f04b01380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:00.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04ae59c640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04b00c6f60 0x7f04b013c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:00.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04af59e640 1 -- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04b00c6f60 msgr2=0x7f04b013c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:00.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04af59e640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04b00c6f60 0x7f04b013c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:00.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04af59e640 1 -- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04b00c6b80 msgr2=0x7f04b01380d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:00.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04af59e640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04b00c6b80 0x7f04b01380d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:00.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04af59e640 1 -- 172.21.15.125:0/3052928065 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a00094d0 con 0x7f04b013f4a0 2023-10-30T13:15:00.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04ae59c640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04b00c6f60 0x7f04b013c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:00.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04af59e640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04b013f4a0 0x7f04b013c850 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f04a800efc0 tx=0x7f04a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:00.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04977fe640 1 -- 172.21.15.125:0/3052928065 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8009ba0 con 0x7f04b013f4a0 2023-10-30T13:15:00.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04977fe640 1 -- 172.21.15.125:0/3052928065 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a800f040 con 0x7f04b013f4a0 2023-10-30T13:15:00.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04b5091640 1 -- 172.21.15.125:0/3052928065 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b013ac70 con 0x7f04b013f4a0 2023-10-30T13:15:00.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04977fe640 1 -- 172.21.15.125:0/3052928065 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a80146c0 con 0x7f04b013f4a0 2023-10-30T13:15:00.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.912+0000 7f04b5091640 1 -- 172.21.15.125:0/3052928065 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b013b230 con 0x7f04b013f4a0 2023-10-30T13:15:00.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.916+0000 7f04b5091640 1 -- 172.21.15.125:0/3052928065 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04b013c030 con 0x7f04b013f4a0 2023-10-30T13:15:00.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.916+0000 7f04977fe640 1 -- 172.21.15.125:0/3052928065 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f04a8014820 con 0x7f04b013f4a0 2023-10-30T13:15:00.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.916+0000 7f04977fe640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f049c0664a0 0x7f049c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:00.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.916+0000 7f04977fe640 1 -- 172.21.15.125:0/3052928065 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7f04a8085520 con 0x7f04b013f4a0 2023-10-30T13:15:00.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.916+0000 7f04aed9d640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f049c0664a0 0x7f049c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:00.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.916+0000 7f04aed9d640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f049c0664a0 0x7f049c068960 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f04a00af000 tx=0x7f04a00b6040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:00.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:00.920+0000 7f04977fe640 1 -- 172.21.15.125:0/3052928065 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f04b013c030 con 0x7f04b013f4a0 2023-10-30T13:15:01.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:01.104+0000 7f04b5091640 1 -- 172.21.15.125:0/3052928065 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04b013c030 con 0x7f04b013f4a0 2023-10-30T13:15:01.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:01.108+0000 7f04977fe640 1 -- 172.21.15.125:0/3052928065 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+6551 (secure 0 0 0) 0x7f04b013c030 con 0x7f04b013f4a0 2023-10-30T13:15:01.113 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:01.114 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9150,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:14:34.076142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:active","state_seq":22,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:01.114 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 213 2023-10-30T13:15:01.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:01.108+0000 7f04b5091640 1 -- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f049c0664a0 msgr2=0x7f049c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:01.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:01.108+0000 7f04b5091640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f049c0664a0 0x7f049c068960 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f04a00af000 tx=0x7f04a00b6040 comp rx=0 tx=0).stop 2023-10-30T13:15:01.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:01.108+0000 7f04b5091640 1 -- 172.21.15.125:0/3052928065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04b013f4a0 msgr2=0x7f04b013c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:01.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:01.108+0000 7f04b5091640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04b013f4a0 0x7f04b013c850 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f04a800efc0 tx=0x7f04a800c490 comp rx=0 tx=0).stop 2023-10-30T13:15:01.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:01.108+0000 7f04b5091640 1 -- 172.21.15.125:0/3052928065 shutdown_connections 2023-10-30T13:15:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:01.108+0000 7f04b5091640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f049c0664a0 0x7f049c068960 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:01.108+0000 7f04b5091640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04b013f4a0 0x7f04b013c850 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:01.108+0000 7f04b5091640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04b00c6f60 0x7f04b013c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:01.108+0000 7f04b5091640 1 --2- 172.21.15.125:0/3052928065 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04b00c6b80 0x7f04b01380d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:01.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:01.108+0000 7f04b5091640 1 -- 172.21.15.125:0/3052928065 >> 172.21.15.125:0/3052928065 conn(0x7f04b00c7910 msgr2=0x7f04b00c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:01.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:01.108+0000 7f04b5091640 1 -- 172.21.15.125:0/3052928065 shutdown_connections 2023-10-30T13:15:01.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:01.108+0000 7f04b5091640 1 -- 172.21.15.125:0/3052928065 wait complete. 2023-10-30T13:15:01.137 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:15:03.138 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:03.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8d53d3640 1 Processor -- start 2023-10-30T13:15:03.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8d53d3640 1 -- start start 2023-10-30T13:15:03.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8d53d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd8d0153ba0 0x7fd8d015a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:03.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8d53d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8d00c7df0 0x7fd8d00c8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:03.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8d53d3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d0153f80 con 0x7fd8d00c74b0 2023-10-30T13:15:03.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8d53d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8d00c1f50 con 0x7fd8d0153ba0 2023-10-30T13:15:03.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8d53d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d006a5a0 con 0x7fd8d00c7df0 2023-10-30T13:15:03.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8ceffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd8d0153ba0 0x7fd8d015a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:03.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8ceffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd8d0153ba0 0x7fd8d015a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42696/0 (socket says 172.21.15.125:42696) 2023-10-30T13:15:03.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8ceffd640 1 -- 172.21.15.125:0/92553891 learned_addr learned my addr 172.21.15.125:0/92553891 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:03.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8cf7fe640 1 --2- 172.21.15.125:0/92553891 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8d00c7df0 0x7fd8d00c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:03.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8ceffd640 1 -- 172.21.15.125:0/92553891 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8d00c7df0 msgr2=0x7fd8d00c8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:03.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8ceffd640 1 --2- 172.21.15.125:0/92553891 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8d00c7df0 0x7fd8d00c8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8ceffd640 1 -- 172.21.15.125:0/92553891 >> v1:172.21.15.125:6789/0 conn(0x7fd8d00c74b0 legacy=0x7fd8d0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:03.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8ce7fc640 1 -- 172.21.15.125:0/92553891 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4084738990 0 0) 0x7fd8d0153f80 con 0x7fd8d00c74b0 2023-10-30T13:15:03.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.280+0000 7fd8ceffd640 1 -- 172.21.15.125:0/92553891 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d00b0e60 con 0x7fd8d0153ba0 2023-10-30T13:15:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.284+0000 7fd8ceffd640 1 --2- 172.21.15.125:0/92553891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8d0153ba0 0x7fd8d015a200 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00098a0 tx=0x7fd8c001c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ec15c0ad611d91 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.284+0000 7fd8ce7fc640 1 -- 172.21.15.125:0/92553891 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c00aa650 con 0x7fd8d0153ba0 2023-10-30T13:15:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.284+0000 7fd8ce7fc640 1 -- 172.21.15.125:0/92553891 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8c00aa7b0 con 0x7fd8d0153ba0 2023-10-30T13:15:03.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.284+0000 7fd8ce7fc640 1 -- 172.21.15.125:0/92553891 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8c00aab40 con 0x7fd8d0153ba0 2023-10-30T13:15:03.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.284+0000 7fd8d53d3640 1 -- 172.21.15.125:0/92553891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8d0153ba0 msgr2=0x7fd8d015a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:03.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.284+0000 7fd8d53d3640 1 --2- 172.21.15.125:0/92553891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8d0153ba0 0x7fd8d015a200 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00098a0 tx=0x7fd8c001c4f0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.284+0000 7fd8d53d3640 1 -- 172.21.15.125:0/92553891 shutdown_connections 2023-10-30T13:15:03.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.284+0000 7fd8d53d3640 1 --2- 172.21.15.125:0/92553891 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8d00c7df0 0x7fd8d00c8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.284+0000 7fd8d53d3640 1 --2- 172.21.15.125:0/92553891 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8d0153ba0 0x7fd8d015a200 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.284+0000 7fd8d53d3640 1 -- 172.21.15.125:0/92553891 >> 172.21.15.125:0/92553891 conn(0x7fd8d006d770 msgr2=0x7fd8d006db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:03.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.284+0000 7fd8d53d3640 1 -- 172.21.15.125:0/92553891 shutdown_connections 2023-10-30T13:15:03.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.284+0000 7fd8d53d3640 1 -- 172.21.15.125:0/92553891 wait complete. 2023-10-30T13:15:03.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8d53d3640 1 Processor -- start 2023-10-30T13:15:03.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8d53d3640 1 -- start start 2023-10-30T13:15:03.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8d53d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8d00c74b0 0x7fd8d01f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:03.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8d53d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8d00c7df0 0x7fd8d00c92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:03.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8d53d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8d0153ba0 0x7fd8d014d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:03.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8d53d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d01f6080 con 0x7fd8d00c7df0 2023-10-30T13:15:03.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8d53d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d01f27b0 con 0x7fd8d0153ba0 2023-10-30T13:15:03.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8d53d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8d01f1bc0 con 0x7fd8d00c74b0 2023-10-30T13:15:03.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8c67fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8d00c7df0 0x7fd8d00c92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:03.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8c67fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8d00c7df0 0x7fd8d00c92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42700/0 (socket says 172.21.15.125:42700) 2023-10-30T13:15:03.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8c67fc640 1 -- 172.21.15.125:0/2256971981 learned_addr learned my addr 172.21.15.125:0/2256971981 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:03.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8ceffd640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8d00c74b0 0x7fd8d01f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:03.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8cf7fe640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8d0153ba0 0x7fd8d014d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:03.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8c67fc640 1 -- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8d00c74b0 msgr2=0x7fd8d01f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:03.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8c67fc640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8d00c74b0 0x7fd8d01f5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8c67fc640 1 -- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8d0153ba0 msgr2=0x7fd8d014d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:03.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8c67fc640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8d0153ba0 0x7fd8d014d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8c67fc640 1 -- 172.21.15.125:0/2256971981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c00094d0 con 0x7fd8d00c7df0 2023-10-30T13:15:03.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8cf7fe640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8d0153ba0 0x7fd8d014d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:15:03.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8c67fc640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8d00c7df0 0x7fd8d00c92a0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8001570 tx=0x7fd8b800cd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:03.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8c7fff640 1 -- 172.21.15.125:0/2256971981 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8003db0 con 0x7fd8d00c7df0 2023-10-30T13:15:03.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8c7fff640 1 -- 172.21.15.125:0/2256971981 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8b8005070 con 0x7fd8d00c7df0 2023-10-30T13:15:03.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8c7fff640 1 -- 172.21.15.125:0/2256971981 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8005420 con 0x7fd8d00c7df0 2023-10-30T13:15:03.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8d53d3640 1 -- 172.21.15.125:0/2256971981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d01548b0 con 0x7fd8d00c7df0 2023-10-30T13:15:03.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.288+0000 7fd8d53d3640 1 -- 172.21.15.125:0/2256971981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d01f2160 con 0x7fd8d00c7df0 2023-10-30T13:15:03.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.292+0000 7fd8d53d3640 1 -- 172.21.15.125:0/2256971981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d0074250 con 0x7fd8d00c7df0 2023-10-30T13:15:03.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.292+0000 7fd8c7fff640 1 -- 172.21.15.125:0/2256971981 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd8b8005580 con 0x7fd8d00c7df0 2023-10-30T13:15:03.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.292+0000 7fd8c7fff640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8b00664d0 0x7fd8b0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:03.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.292+0000 7fd8c7fff640 1 -- 172.21.15.125:0/2256971981 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7fd8b8084000 con 0x7fd8d00c7df0 2023-10-30T13:15:03.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.292+0000 7fd8ceffd640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8b00664d0 0x7fd8b0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:03.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.292+0000 7fd8ceffd640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8b00664d0 0x7fd8b0068990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00ae000 tx=0x7fd8c0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:03.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.296+0000 7fd8c7fff640 1 -- 172.21.15.125:0/2256971981 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd8b8050c00 con 0x7fd8d00c7df0 2023-10-30T13:15:03.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.480+0000 7fd8d53d3640 1 -- 172.21.15.125:0/2256971981 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8d004e170 con 0x7fd8d00c7df0 2023-10-30T13:15:03.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.480+0000 7fd8c7fff640 1 -- 172.21.15.125:0/2256971981 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 213 v213) v1 ==== 95+0+6551 (secure 0 0 0) 0x7fd8b8017070 con 0x7fd8d00c7df0 2023-10-30T13:15:03.486 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:03.486 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":213,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9150,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":210}],"filesystems":[{"mdsmap":{"epoch":213,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:14:34.076142+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":7511,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7511":{"gid":7511,"name":"d","rank":1,"incarnation":205,"state":"up:active","state_seq":22,"addr":"172.21.15.136:6835/695089547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":695089547},{"type":"v1","addr":"172.21.15.136:6835","nonce":695089547}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:03.487 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 213 2023-10-30T13:15:03.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.480+0000 7fd8d53d3640 1 -- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8b00664d0 msgr2=0x7fd8b0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:03.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.480+0000 7fd8d53d3640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8b00664d0 0x7fd8b0068990 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fd8c00ae000 tx=0x7fd8c0008040 comp rx=0 tx=0).stop 2023-10-30T13:15:03.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.480+0000 7fd8d53d3640 1 -- 172.21.15.125:0/2256971981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8d00c7df0 msgr2=0x7fd8d00c92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:03.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.480+0000 7fd8d53d3640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8d00c7df0 0x7fd8d00c92a0 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7fd8b8001570 tx=0x7fd8b800cd00 comp rx=0 tx=0).stop 2023-10-30T13:15:03.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.480+0000 7fd8d53d3640 1 -- 172.21.15.125:0/2256971981 shutdown_connections 2023-10-30T13:15:03.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.480+0000 7fd8d53d3640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8b00664d0 0x7fd8b0068990 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.480+0000 7fd8d53d3640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8d0153ba0 0x7fd8d014d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.480+0000 7fd8d53d3640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8d00c7df0 0x7fd8d00c92a0 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.480+0000 7fd8d53d3640 1 --2- 172.21.15.125:0/2256971981 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8d00c74b0 0x7fd8d01f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:03.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.480+0000 7fd8d53d3640 1 -- 172.21.15.125:0/2256971981 >> 172.21.15.125:0/2256971981 conn(0x7fd8d006d770 msgr2=0x7fd8d014be80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:03.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.480+0000 7fd8d53d3640 1 -- 172.21.15.125:0/2256971981 shutdown_connections 2023-10-30T13:15:03.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:03.480+0000 7fd8d53d3640 1 -- 172.21.15.125:0/2256971981 wait complete. 2023-10-30T13:15:03.508 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:15:05.508 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:05.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.648+0000 7fa765cd6640 1 Processor -- start 2023-10-30T13:15:05.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.648+0000 7fa765cd6640 1 -- start start 2023-10-30T13:15:05.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.648+0000 7fa765cd6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7601538a0 0x7fa76015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:05.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.648+0000 7fa765cd6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7600cb800 con 0x7fa7600ca0e0 2023-10-30T13:15:05.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.648+0000 7fa765cd6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa76015dc30 con 0x7fa7600cf160 2023-10-30T13:15:05.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.648+0000 7fa765cd6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7600cb330 con 0x7fa7601538a0 2023-10-30T13:15:05.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.648+0000 7fa757fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa7600ca0e0 0x7fa76015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55340/0 (socket says 172.21.15.125:55340) 2023-10-30T13:15:05.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.648+0000 7fa757fff640 1 -- 172.21.15.125:0/1945035743 learned_addr learned my addr 172.21.15.125:0/1945035743 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:15:05.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.648+0000 7fa75f7fe640 1 --2- 172.21.15.125:0/1945035743 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7601538a0 0x7fa76015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:05.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.648+0000 7fa75effd640 1 -- 172.21.15.125:0/1945035743 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286855174 0 0) 0x7fa7600cb800 con 0x7fa7600ca0e0 2023-10-30T13:15:05.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.648+0000 7fa75effd640 1 -- 172.21.15.125:0/1945035743 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa74c003530 con 0x7fa7600ca0e0 2023-10-30T13:15:05.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.652+0000 7fa75effd640 1 -- 172.21.15.125:0/1945035743 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 120561610 0 0) 0x7fa74c003530 con 0x7fa7600ca0e0 2023-10-30T13:15:05.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.652+0000 7fa75f7fe640 1 -- 172.21.15.125:0/1945035743 >> v1:172.21.15.136:6789/0 conn(0x7fa7600cf160 legacy=0x7fa760164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:05.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.652+0000 7fa75f7fe640 1 -- 172.21.15.125:0/1945035743 >> v1:172.21.15.125:6789/0 conn(0x7fa7600ca0e0 legacy=0x7fa76015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:05.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.652+0000 7fa75f7fe640 1 -- 172.21.15.125:0/1945035743 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7600cad70 con 0x7fa7601538a0 2023-10-30T13:15:05.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.652+0000 7fa75f7fe640 1 --2- 172.21.15.125:0/1945035743 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7601538a0 0x7fa76015d6f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa7500097f0 tx=0x7fa75001c320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29e00d2e4183078c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:05.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.652+0000 7fa75effd640 1 -- 172.21.15.125:0/1945035743 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa748001680 con 0x7fa7601538a0 2023-10-30T13:15:05.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.652+0000 7fa75effd640 1 -- 172.21.15.125:0/1945035743 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7500a7050 con 0x7fa7601538a0 2023-10-30T13:15:05.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.652+0000 7fa75effd640 1 -- 172.21.15.125:0/1945035743 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7500ab600 con 0x7fa7601538a0 2023-10-30T13:15:05.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 -- 172.21.15.125:0/1945035743 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7601538a0 msgr2=0x7fa76015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:05.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 --2- 172.21.15.125:0/1945035743 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7601538a0 0x7fa76015d6f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa7500097f0 tx=0x7fa75001c320 comp rx=0 tx=0).stop 2023-10-30T13:15:05.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 -- 172.21.15.125:0/1945035743 shutdown_connections 2023-10-30T13:15:05.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 --2- 172.21.15.125:0/1945035743 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7601538a0 0x7fa76015d6f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:05.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 -- 172.21.15.125:0/1945035743 >> 172.21.15.125:0/1945035743 conn(0x7fa76006bfe0 msgr2=0x7fa76006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:05.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 -- 172.21.15.125:0/1945035743 shutdown_connections 2023-10-30T13:15:05.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 -- 172.21.15.125:0/1945035743 wait complete. 2023-10-30T13:15:05.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 Processor -- start 2023-10-30T13:15:05.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 -- start start 2023-10-30T13:15:05.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7600ca0e0 0x7fa76014a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:05.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7600cf160 0x7fa7600b4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:05.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7601538a0 0x7fa7601ef910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:05.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7601ef4f0 con 0x7fa7601538a0 2023-10-30T13:15:05.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa760157560 con 0x7fa7600ca0e0 2023-10-30T13:15:05.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa765cd6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa760159640 con 0x7fa7600cf160 2023-10-30T13:15:05.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa75ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7601538a0 0x7fa7601ef910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:05.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa75ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7601538a0 0x7fa7601ef910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42708/0 (socket says 172.21.15.125:42708) 2023-10-30T13:15:05.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa75ffff640 1 -- 172.21.15.125:0/4158153298 learned_addr learned my addr 172.21.15.125:0/4158153298 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:05.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa757fff640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7600cf160 0x7fa7600b4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:05.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa75f7fe640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7600ca0e0 0x7fa76014a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:05.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa75ffff640 1 -- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7600cf160 msgr2=0x7fa7600b4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:05.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa75ffff640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7600cf160 0x7fa7600b4450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:05.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa75ffff640 1 -- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7600ca0e0 msgr2=0x7fa76014a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:05.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa75ffff640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7600ca0e0 0x7fa76014a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:05.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa75ffff640 1 -- 172.21.15.125:0/4158153298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa750009420 con 0x7fa7601538a0 2023-10-30T13:15:05.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.656+0000 7fa75f7fe640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7600ca0e0 0x7fa76014a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:05.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.660+0000 7fa75ffff640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7601538a0 0x7fa7601ef910 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fa758004700 tx=0x7fa75800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:05.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.660+0000 7fa75cff9640 1 -- 172.21.15.125:0/4158153298 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758009b80 con 0x7fa7601538a0 2023-10-30T13:15:05.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.660+0000 7fa75cff9640 1 -- 172.21.15.125:0/4158153298 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa758009ce0 con 0x7fa7601538a0 2023-10-30T13:15:05.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.660+0000 7fa75cff9640 1 -- 172.21.15.125:0/4158153298 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa758006c20 con 0x7fa7601538a0 2023-10-30T13:15:05.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.660+0000 7fa765cd6640 1 -- 172.21.15.125:0/4158153298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7601ef130 con 0x7fa7601538a0 2023-10-30T13:15:05.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.660+0000 7fa765cd6640 1 -- 172.21.15.125:0/4158153298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7601f3150 con 0x7fa7601538a0 2023-10-30T13:15:05.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.660+0000 7fa765cd6640 1 -- 172.21.15.125:0/4158153298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7601ef630 con 0x7fa7601538a0 2023-10-30T13:15:05.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.660+0000 7fa75cff9640 1 -- 172.21.15.125:0/4158153298 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa758008070 con 0x7fa7601538a0 2023-10-30T13:15:05.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.660+0000 7fa75cff9640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa7400664d0 0x7fa740068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:05.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.660+0000 7fa75cff9640 1 -- 172.21.15.125:0/4158153298 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7888+0+0 (secure 0 0 0) 0x7fa758086d90 con 0x7fa7601538a0 2023-10-30T13:15:05.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.660+0000 7fa75f7fe640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa7400664d0 0x7fa740068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:05.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.664+0000 7fa75f7fe640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa7400664d0 0x7fa740068990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fa7500ad000 tx=0x7fa7500a6000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:05.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.664+0000 7fa75cff9640 1 -- 172.21.15.125:0/4158153298 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa7601ef630 con 0x7fa7601538a0 2023-10-30T13:15:05.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.848+0000 7fa765cd6640 1 -- 172.21.15.125:0/4158153298 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa76015dc30 con 0x7fa7601538a0 2023-10-30T13:15:05.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.848+0000 7fa75cff9640 1 -- 172.21.15.125:0/4158153298 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 214 v214) v1 ==== 95+0+5755 (secure 0 0 0) 0x7fa76015dc30 con 0x7fa7601538a0 2023-10-30T13:15:05.857 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:05.857 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":214,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:05.766744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:05.858 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 214 2023-10-30T13:15:05.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.852+0000 7fa765cd6640 1 -- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa7400664d0 msgr2=0x7fa740068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:05.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.852+0000 7fa765cd6640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa7400664d0 0x7fa740068990 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fa7500ad000 tx=0x7fa7500a6000 comp rx=0 tx=0).stop 2023-10-30T13:15:05.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.852+0000 7fa765cd6640 1 -- 172.21.15.125:0/4158153298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7601538a0 msgr2=0x7fa7601ef910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:05.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.852+0000 7fa765cd6640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7601538a0 0x7fa7601ef910 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fa758004700 tx=0x7fa75800d430 comp rx=0 tx=0).stop 2023-10-30T13:15:05.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.852+0000 7fa765cd6640 1 -- 172.21.15.125:0/4158153298 shutdown_connections 2023-10-30T13:15:05.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.852+0000 7fa765cd6640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa7400664d0 0x7fa740068990 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:05.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.852+0000 7fa765cd6640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa7601538a0 0x7fa7601ef910 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:05.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.852+0000 7fa765cd6640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa7600cf160 0x7fa7600b4450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:05.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.852+0000 7fa765cd6640 1 --2- 172.21.15.125:0/4158153298 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa7600ca0e0 0x7fa76014a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:05.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.852+0000 7fa765cd6640 1 -- 172.21.15.125:0/4158153298 >> 172.21.15.125:0/4158153298 conn(0x7fa76006bfe0 msgr2=0x7fa760152660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:05.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.852+0000 7fa765cd6640 1 -- 172.21.15.125:0/4158153298 shutdown_connections 2023-10-30T13:15:05.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:05.852+0000 7fa765cd6640 1 -- 172.21.15.125:0/4158153298 wait complete. 2023-10-30T13:15:05.878 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T13:15:05.878 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:15:05.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:06.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.012+0000 7f82d173e640 1 Processor -- start 2023-10-30T13:15:06.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.012+0000 7f82d173e640 1 -- start start 2023-10-30T13:15:06.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.012+0000 7f82d173e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82cc1538a0 0x7f82cc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.012+0000 7f82d173e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f82cc0cf160 0x7f82cc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.012+0000 7f82d173e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82cc0cb800 con 0x7f82cc0ca0e0 2023-10-30T13:15:06.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.012+0000 7f82d173e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82cc15dc30 con 0x7f82cc0cf160 2023-10-30T13:15:06.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.012+0000 7f82d173e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc0cb330 con 0x7f82cc1538a0 2023-10-30T13:15:06.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.012+0000 7f82c3fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f82cc0ca0e0 0x7f82cc15a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55352/0 (socket says 172.21.15.125:55352) 2023-10-30T13:15:06.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.012+0000 7f82c3fff640 1 -- 172.21.15.125:0/1523468561 learned_addr learned my addr 172.21.15.125:0/1523468561 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:15:06.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.012+0000 7f82cb7fe640 1 --2- 172.21.15.125:0/1523468561 >> v2:172.21.15.136:3300/0 conn(0x7f82cc0cf160 0x7f82cc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.012+0000 7f82caffd640 1 --2- 172.21.15.125:0/1523468561 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82cc1538a0 0x7f82cc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.012+0000 7f82ca7fc640 1 -- 172.21.15.125:0/1523468561 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877022655 0 0) 0x7f82cc0cb800 con 0x7f82cc0ca0e0 2023-10-30T13:15:06.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.012+0000 7f82ca7fc640 1 -- 172.21.15.125:0/1523468561 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82b8003530 con 0x7f82cc0ca0e0 2023-10-30T13:15:06.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82ca7fc640 1 -- 172.21.15.125:0/1523468561 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 117919010 0 0) 0x7f82b8003530 con 0x7f82cc0ca0e0 2023-10-30T13:15:06.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82ca7fc640 1 -- 172.21.15.125:0/1523468561 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82cc1538a0 msgr2=0x7f82cc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:06.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82ca7fc640 1 --2- 172.21.15.125:0/1523468561 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82cc1538a0 0x7f82cc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82ca7fc640 1 -- 172.21.15.125:0/1523468561 >> v2:172.21.15.136:3300/0 conn(0x7f82cc0cf160 msgr2=0x7f82cc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:06.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82ca7fc640 1 --2- 172.21.15.125:0/1523468561 >> v2:172.21.15.136:3300/0 conn(0x7f82cc0cf160 0x7f82cc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82ca7fc640 1 -- 172.21.15.125:0/1523468561 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82cc0cad70 con 0x7f82cc0ca0e0 2023-10-30T13:15:06.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82ca7fc640 1 -- 172.21.15.125:0/1523468561 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f82b4001680 con 0x7f82cc0ca0e0 2023-10-30T13:15:06.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82ca7fc640 1 -- 172.21.15.125:0/1523468561 >> v1:172.21.15.125:6789/0 conn(0x7f82cc0ca0e0 legacy=0x7f82cc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:06.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82ca7fc640 1 --2- 172.21.15.125:0/1523468561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82b8003980 0x7f82b8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82ca7fc640 1 --2- 172.21.15.125:0/1523468561 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82cc1538a0 0x7f82b80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82ca7fc640 1 --2- 172.21.15.125:0/1523468561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82b8004d10 0x7f82b8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82ca7fc640 1 -- 172.21.15.125:0/1523468561 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc15dc30 con 0x7f82b8004d10 2023-10-30T13:15:06.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82ca7fc640 1 -- 172.21.15.125:0/1523468561 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc0cb330 con 0x7f82b8003980 2023-10-30T13:15:06.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82ca7fc640 1 -- 172.21.15.125:0/1523468561 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b800b990 con 0x7f82cc1538a0 2023-10-30T13:15:06.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82cb7fe640 1 --2- 172.21.15.125:0/1523468561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82b8004d10 0x7f82b8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82caffd640 1 --2- 172.21.15.125:0/1523468561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82b8003980 0x7f82b8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82c3fff640 1 --2- 172.21.15.125:0/1523468561 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82cc1538a0 0x7f82b80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82cb7fe640 1 -- 172.21.15.125:0/1523468561 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82cc1538a0 msgr2=0x7f82b80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:06.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82cb7fe640 1 --2- 172.21.15.125:0/1523468561 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82cc1538a0 0x7f82b80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82cb7fe640 1 -- 172.21.15.125:0/1523468561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82b8003980 msgr2=0x7f82b8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:06.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82cb7fe640 1 --2- 172.21.15.125:0/1523468561 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82b8003980 0x7f82b8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.016+0000 7f82cb7fe640 1 -- 172.21.15.125:0/1523468561 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b8003530 con 0x7f82b8004d10 2023-10-30T13:15:06.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82cb7fe640 1 --2- 172.21.15.125:0/1523468561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82b8004d10 0x7f82b8005190 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f82c4009fc0 tx=0x7f82c401cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ba466299e02bb1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:06.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82ca7fc640 1 -- 172.21.15.125:0/1523468561 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c40a5cb0 con 0x7f82b8004d10 2023-10-30T13:15:06.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82ca7fc640 1 -- 172.21.15.125:0/1523468561 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82c400f040 con 0x7f82b8004d10 2023-10-30T13:15:06.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82ca7fc640 1 -- 172.21.15.125:0/1523468561 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82c40ad680 con 0x7f82b8004d10 2023-10-30T13:15:06.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82d173e640 1 -- 172.21.15.125:0/1523468561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82b8004d10 msgr2=0x7f82b8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:06.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82d173e640 1 --2- 172.21.15.125:0/1523468561 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82b8004d10 0x7f82b8005190 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f82c4009fc0 tx=0x7f82c401cca0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82cb7fe640 1 -- 172.21.15.125:0/1523468561 reap_dead start 2023-10-30T13:15:06.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82d173e640 1 -- 172.21.15.125:0/1523468561 shutdown_connections 2023-10-30T13:15:06.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82d173e640 1 -- 172.21.15.125:0/1523468561 >> 172.21.15.125:0/1523468561 conn(0x7f82cc06bfe0 msgr2=0x7f82cc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:06.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82d173e640 1 -- 172.21.15.125:0/1523468561 shutdown_connections 2023-10-30T13:15:06.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82d173e640 1 -- 172.21.15.125:0/1523468561 wait complete. 2023-10-30T13:15:06.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82d173e640 1 Processor -- start 2023-10-30T13:15:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82d173e640 1 -- start start 2023-10-30T13:15:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82d173e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82b8004d10 0x7f82cc157f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82d173e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82cc0ce750 0x7f82cc148cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82d173e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82cc158e10 0x7f82cc1f0330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82d173e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc0cef90 con 0x7f82b8004d10 2023-10-30T13:15:06.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82d173e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc0cdf10 con 0x7f82cc0ce750 2023-10-30T13:15:06.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.020+0000 7f82d173e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82cc0cecb0 con 0x7f82cc158e10 2023-10-30T13:15:06.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82caffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82b8004d10 0x7f82cc157f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82caffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82b8004d10 0x7f82cc157f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42726/0 (socket says 172.21.15.125:42726) 2023-10-30T13:15:06.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82caffd640 1 -- 172.21.15.125:0/713727813 learned_addr learned my addr 172.21.15.125:0/713727813 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:06.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82c3fff640 1 --2- 172.21.15.125:0/713727813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82cc0ce750 0x7f82cc148cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82cb7fe640 1 --2- 172.21.15.125:0/713727813 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82cc158e10 0x7f82cc1f0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82caffd640 1 -- 172.21.15.125:0/713727813 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82cc158e10 msgr2=0x7f82cc1f0330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:06.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82caffd640 1 --2- 172.21.15.125:0/713727813 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82cc158e10 0x7f82cc1f0330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82caffd640 1 -- 172.21.15.125:0/713727813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82cc0ce750 msgr2=0x7f82cc148cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:06.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82caffd640 1 --2- 172.21.15.125:0/713727813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82cc0ce750 0x7f82cc148cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82caffd640 1 -- 172.21.15.125:0/713727813 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82cc0cd150 con 0x7f82b8004d10 2023-10-30T13:15:06.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82caffd640 1 --2- 172.21.15.125:0/713727813 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82b8004d10 0x7f82cc157f20 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f82bc011960 tx=0x7f82bc011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:06.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82c37fe640 1 -- 172.21.15.125:0/713727813 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc00faf0 con 0x7f82b8004d10 2023-10-30T13:15:06.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82c37fe640 1 -- 172.21.15.125:0/713727813 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82bc00fc50 con 0x7f82b8004d10 2023-10-30T13:15:06.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82c37fe640 1 -- 172.21.15.125:0/713727813 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82bc014440 con 0x7f82b8004d10 2023-10-30T13:15:06.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82d173e640 1 -- 172.21.15.125:0/713727813 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82cc0cd430 con 0x7f82b8004d10 2023-10-30T13:15:06.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82d173e640 1 -- 172.21.15.125:0/713727813 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82cc0cd9f0 con 0x7f82b8004d10 2023-10-30T13:15:06.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82d173e640 1 -- 172.21.15.125:0/713727813 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82cc0cdbb0 con 0x7f82b8004d10 2023-10-30T13:15:06.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82c37fe640 1 -- 172.21.15.125:0/713727813 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f82bc0145a0 con 0x7f82b8004d10 2023-10-30T13:15:06.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82c37fe640 1 --2- 172.21.15.125:0/713727813 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82ac0664d0 0x7f82ac068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:06.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.024+0000 7f82c37fe640 1 -- 172.21.15.125:0/713727813 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7f82bc084fe0 con 0x7f82b8004d10 2023-10-30T13:15:06.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.028+0000 7f82c3fff640 1 --2- 172.21.15.125:0/713727813 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82ac0664d0 0x7f82ac068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:06.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.028+0000 7f82c3fff640 1 --2- 172.21.15.125:0/713727813 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82ac0664d0 0x7f82ac068990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f82b4000c10 tx=0x7f82b4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:06.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.032+0000 7f82c37fe640 1 -- 172.21.15.125:0/713727813 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f82bc051b70 con 0x7f82b8004d10 2023-10-30T13:15:06.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.216+0000 7f82d173e640 1 -- 172.21.15.125:0/713727813 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82cc04e170 con 0x7f82b8004d10 2023-10-30T13:15:06.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.216+0000 7f82c37fe640 1 -- 172.21.15.125:0/713727813 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 214 v214) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f82bc022070 con 0x7f82b8004d10 2023-10-30T13:15:06.222 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:06.222 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":214,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:05.766744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:06.222 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 214 2023-10-30T13:15:06.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.216+0000 7f82d173e640 1 -- 172.21.15.125:0/713727813 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82ac0664d0 msgr2=0x7f82ac068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:06.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.216+0000 7f82d173e640 1 --2- 172.21.15.125:0/713727813 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82ac0664d0 0x7f82ac068990 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f82b4000c10 tx=0x7f82b4009040 comp rx=0 tx=0).stop 2023-10-30T13:15:06.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.216+0000 7f82d173e640 1 -- 172.21.15.125:0/713727813 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82b8004d10 msgr2=0x7f82cc157f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:06.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.216+0000 7f82d173e640 1 --2- 172.21.15.125:0/713727813 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82b8004d10 0x7f82cc157f20 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f82bc011960 tx=0x7f82bc011e20 comp rx=0 tx=0).stop 2023-10-30T13:15:06.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.216+0000 7f82d173e640 1 -- 172.21.15.125:0/713727813 shutdown_connections 2023-10-30T13:15:06.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.216+0000 7f82d173e640 1 --2- 172.21.15.125:0/713727813 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f82ac0664d0 0x7f82ac068990 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.216+0000 7f82d173e640 1 --2- 172.21.15.125:0/713727813 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f82cc158e10 0x7f82cc1f0330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.216+0000 7f82d173e640 1 --2- 172.21.15.125:0/713727813 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f82cc0ce750 0x7f82cc148cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.216+0000 7f82d173e640 1 --2- 172.21.15.125:0/713727813 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f82b8004d10 0x7f82cc157f20 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:06.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.216+0000 7f82d173e640 1 -- 172.21.15.125:0/713727813 >> 172.21.15.125:0/713727813 conn(0x7f82cc06bfe0 msgr2=0x7f82cc065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:06.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.216+0000 7f82d173e640 1 -- 172.21.15.125:0/713727813 shutdown_connections 2023-10-30T13:15:06.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:06.216+0000 7f82d173e640 1 -- 172.21.15.125:0/713727813 wait complete. 2023-10-30T13:15:06.244 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 214, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 214, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_8443": { "addr": "172.21.15.136:6839/3776186190", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3776186190, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3776186190, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 8443, "incarnation": 196, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 41 }, "gid_8529": { "addr": "172.21.15.125:6836/3950044434", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3950044434, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3950044434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 8529, "incarnation": 179, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 15 }, "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 80, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:15:05.766744+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8529, "mds_1": 9150, "mds_2": 8443 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:15:08.245 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:08.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea9409640 1 Processor -- start 2023-10-30T13:15:08.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea9409640 1 -- start start 2023-10-30T13:15:08.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea9409640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faea4159e60 0x7faea406f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:08.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea9409640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7faea40bfec0 0x7faea40c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:08.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea9409640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea406fa80 con 0x7faea406a720 2023-10-30T13:15:08.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea9409640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea4071290 con 0x7faea4159e60 2023-10-30T13:15:08.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea9409640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea4071570 con 0x7faea40bfec0 2023-10-30T13:15:08.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea2ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faea4159e60 0x7faea406f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:08.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea2ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faea4159e60 0x7faea406f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42738/0 (socket says 172.21.15.125:42738) 2023-10-30T13:15:08.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea2ffd640 1 -- 172.21.15.125:0/3366617929 learned_addr learned my addr 172.21.15.125:0/3366617929 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:08.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea27fc640 1 --2- 172.21.15.125:0/3366617929 >> v2:172.21.15.136:3300/0 conn(0x7faea40bfec0 0x7faea40c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:08.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea2ffd640 1 -- 172.21.15.125:0/3366617929 >> v2:172.21.15.136:3300/0 conn(0x7faea40bfec0 msgr2=0x7faea40c1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:08.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea2ffd640 1 --2- 172.21.15.125:0/3366617929 >> v2:172.21.15.136:3300/0 conn(0x7faea40bfec0 0x7faea40c1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea2ffd640 1 -- 172.21.15.125:0/3366617929 >> v1:172.21.15.136:6789/0 conn(0x7faea406a720 legacy=0x7faea4155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:08.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea2ffd640 1 -- 172.21.15.125:0/3366617929 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea4071850 con 0x7faea4159e60 2023-10-30T13:15:08.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea2ffd640 1 --2- 172.21.15.125:0/3366617929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea4159e60 0x7faea406f540 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7faea406f200 tx=0x7fae94020080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79072b10271da8a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:08.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea1ffb640 1 -- 172.21.15.125:0/3366617929 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae940a9e90 con 0x7faea4159e60 2023-10-30T13:15:08.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea1ffb640 1 -- 172.21.15.125:0/3366617929 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae940b1440 con 0x7faea4159e60 2023-10-30T13:15:08.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea1ffb640 1 -- 172.21.15.125:0/3366617929 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae940b17d0 con 0x7faea4159e60 2023-10-30T13:15:08.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea9409640 1 -- 172.21.15.125:0/3366617929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea4159e60 msgr2=0x7faea406f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:08.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea9409640 1 --2- 172.21.15.125:0/3366617929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea4159e60 0x7faea406f540 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7faea406f200 tx=0x7fae94020080 comp rx=0 tx=0).stop 2023-10-30T13:15:08.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea9409640 1 -- 172.21.15.125:0/3366617929 shutdown_connections 2023-10-30T13:15:08.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea9409640 1 --2- 172.21.15.125:0/3366617929 >> v2:172.21.15.136:3300/0 conn(0x7faea40bfec0 0x7faea40c1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea9409640 1 --2- 172.21.15.125:0/3366617929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea4159e60 0x7faea406f540 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea9409640 1 -- 172.21.15.125:0/3366617929 >> 172.21.15.125:0/3366617929 conn(0x7faea40c7910 msgr2=0x7faea40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:08.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.388+0000 7faea9409640 1 -- 172.21.15.125:0/3366617929 shutdown_connections 2023-10-30T13:15:08.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea9409640 1 -- 172.21.15.125:0/3366617929 wait complete. 2023-10-30T13:15:08.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea9409640 1 Processor -- start 2023-10-30T13:15:08.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea9409640 1 -- start start 2023-10-30T13:15:08.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea9409640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faea406a720 0x7faea40c8e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:08.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea9409640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faea40bfec0 0x7faea41eebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:08.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea9409640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea4159e60 0x7faea414d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:08.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea9409640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea40c32d0 con 0x7faea4159e60 2023-10-30T13:15:08.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea9409640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea41e17e0 con 0x7faea40bfec0 2023-10-30T13:15:08.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea9409640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea41e0c30 con 0x7faea406a720 2023-10-30T13:15:08.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea4159e60 0x7faea414d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:08.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea4159e60 0x7faea414d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42748/0 (socket says 172.21.15.125:42748) 2023-10-30T13:15:08.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea37fe640 1 -- 172.21.15.125:0/3994180739 learned_addr learned my addr 172.21.15.125:0/3994180739 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea27fc640 1 --2- 172.21.15.125:0/3994180739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faea40bfec0 0x7faea41eebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea2ffd640 1 --2- 172.21.15.125:0/3994180739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faea406a720 0x7faea40c8e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea37fe640 1 -- 172.21.15.125:0/3994180739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faea406a720 msgr2=0x7faea40c8e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea37fe640 1 --2- 172.21.15.125:0/3994180739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faea406a720 0x7faea40c8e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea37fe640 1 -- 172.21.15.125:0/3994180739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faea40bfec0 msgr2=0x7faea41eebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea37fe640 1 --2- 172.21.15.125:0/3994180739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faea40bfec0 0x7faea41eebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea37fe640 1 -- 172.21.15.125:0/3994180739 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9400a670 con 0x7faea4159e60 2023-10-30T13:15:08.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7faea37fe640 1 --2- 172.21.15.125:0/3994180739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea4159e60 0x7faea414d450 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fae9c00c7b0 tx=0x7fae9c00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:08.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.392+0000 7fae8b7fe640 1 -- 172.21.15.125:0/3994180739 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c004b70 con 0x7faea4159e60 2023-10-30T13:15:08.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.396+0000 7fae8b7fe640 1 -- 172.21.15.125:0/3994180739 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9c004cd0 con 0x7faea4159e60 2023-10-30T13:15:08.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.396+0000 7fae8b7fe640 1 -- 172.21.15.125:0/3994180739 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c00aa90 con 0x7faea4159e60 2023-10-30T13:15:08.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.396+0000 7faea9409640 1 -- 172.21.15.125:0/3994180739 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea40c4570 con 0x7faea4159e60 2023-10-30T13:15:08.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.396+0000 7faea9409640 1 -- 172.21.15.125:0/3994180739 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea41e1500 con 0x7faea4159e60 2023-10-30T13:15:08.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.396+0000 7faea9409640 1 -- 172.21.15.125:0/3994180739 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea404e170 con 0x7faea4159e60 2023-10-30T13:15:08.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.396+0000 7fae8b7fe640 1 -- 172.21.15.125:0/3994180739 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fae9c00abf0 con 0x7faea4159e60 2023-10-30T13:15:08.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.396+0000 7fae8b7fe640 1 --2- 172.21.15.125:0/3994180739 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fae900664a0 0x7fae90068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:08.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.396+0000 7fae8b7fe640 1 -- 172.21.15.125:0/3994180739 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7fae9c0848d0 con 0x7faea4159e60 2023-10-30T13:15:08.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.396+0000 7faea2ffd640 1 --2- 172.21.15.125:0/3994180739 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fae900664a0 0x7fae90068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:08.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.400+0000 7faea2ffd640 1 --2- 172.21.15.125:0/3994180739 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fae900664a0 0x7fae90068960 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fae940b3000 tx=0x7fae94008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:08.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.400+0000 7fae8b7fe640 1 -- 172.21.15.125:0/3994180739 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fae9c0513d0 con 0x7faea4159e60 2023-10-30T13:15:08.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.580+0000 7faea9409640 1 -- 172.21.15.125:0/3994180739 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faea40c4700 con 0x7faea4159e60 2023-10-30T13:15:08.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.580+0000 7fae8b7fe640 1 -- 172.21.15.125:0/3994180739 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 214 v214) v1 ==== 95+0+5755 (secure 0 0 0) 0x7fae9c0549f0 con 0x7faea4159e60 2023-10-30T13:15:08.588 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:08.588 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":214,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:05.766744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:08.589 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 214 2023-10-30T13:15:08.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.584+0000 7faea9409640 1 -- 172.21.15.125:0/3994180739 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fae900664a0 msgr2=0x7fae90068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:08.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.584+0000 7faea9409640 1 --2- 172.21.15.125:0/3994180739 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fae900664a0 0x7fae90068960 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7fae940b3000 tx=0x7fae94008040 comp rx=0 tx=0).stop 2023-10-30T13:15:08.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.584+0000 7faea9409640 1 -- 172.21.15.125:0/3994180739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea4159e60 msgr2=0x7faea414d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:08.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.584+0000 7faea9409640 1 --2- 172.21.15.125:0/3994180739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea4159e60 0x7faea414d450 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7fae9c00c7b0 tx=0x7fae9c00cc70 comp rx=0 tx=0).stop 2023-10-30T13:15:08.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.584+0000 7faea9409640 1 -- 172.21.15.125:0/3994180739 shutdown_connections 2023-10-30T13:15:08.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.584+0000 7faea9409640 1 --2- 172.21.15.125:0/3994180739 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fae900664a0 0x7fae90068960 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.584+0000 7faea9409640 1 --2- 172.21.15.125:0/3994180739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faea4159e60 0x7faea414d450 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.584+0000 7faea9409640 1 --2- 172.21.15.125:0/3994180739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faea40bfec0 0x7faea41eebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.584+0000 7faea9409640 1 --2- 172.21.15.125:0/3994180739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faea406a720 0x7faea40c8e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:08.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.584+0000 7faea9409640 1 -- 172.21.15.125:0/3994180739 >> 172.21.15.125:0/3994180739 conn(0x7faea40c7910 msgr2=0x7faea40c9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:08.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.584+0000 7faea9409640 1 -- 172.21.15.125:0/3994180739 shutdown_connections 2023-10-30T13:15:08.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:08.584+0000 7faea9409640 1 -- 172.21.15.125:0/3994180739 wait complete. 2023-10-30T13:15:08.610 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:10.611 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:10.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.756+0000 7fdb1293e640 1 Processor -- start 2023-10-30T13:15:10.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.756+0000 7fdb1293e640 1 -- start start 2023-10-30T13:15:10.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.756+0000 7fdb1293e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdb0c1539a0 0x7fdb0c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:10.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.756+0000 7fdb1293e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb0c069e90 0x7fdb0c06a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:10.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.756+0000 7fdb1293e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0c15a8c0 con 0x7fdb0c070a00 2023-10-30T13:15:10.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.756+0000 7fdb1293e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb0c06fbb0 con 0x7fdb0c1539a0 2023-10-30T13:15:10.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.756+0000 7fdb1293e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c153d80 con 0x7fdb0c069e90 2023-10-30T13:15:10.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.756+0000 7fdb10edb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb0c069e90 0x7fdb0c06a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:10.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.756+0000 7fdb0bfff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdb0c1539a0 0x7fdb0c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:10.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.756+0000 7fdb10edb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb0c069e90 0x7fdb0c06a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:39864/0 (socket says 172.21.15.125:39864) 2023-10-30T13:15:10.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.756+0000 7fdb10edb640 1 -- 172.21.15.125:0/1770718227 learned_addr learned my addr 172.21.15.125:0/1770718227 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:10.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.760+0000 7fdb0bfff640 1 -- 172.21.15.125:0/1770718227 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb0c069e90 msgr2=0x7fdb0c06a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:10.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.760+0000 7fdb0bfff640 1 --2- 172.21.15.125:0/1770718227 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb0c069e90 0x7fdb0c06a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.760+0000 7fdb0bfff640 1 -- 172.21.15.125:0/1770718227 >> v1:172.21.15.136:6789/0 conn(0x7fdb0c070a00 legacy=0x7fdb0c070e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:10.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.760+0000 7fdb0bfff640 1 -- 172.21.15.125:0/1770718227 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0c0c5c50 con 0x7fdb0c1539a0 2023-10-30T13:15:10.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.760+0000 7fdb10edb640 1 --2- 172.21.15.125:0/1770718227 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb0c069e90 0x7fdb0c06a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:15:10.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.760+0000 7fdb0bfff640 1 --2- 172.21.15.125:0/1770718227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb0c1539a0 0x7fdb0c15a380 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fdafc0097c0 tx=0x7fdafc01c3c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20f7dc27dd4e7f08 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:10.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb0affd640 1 -- 172.21.15.125:0/1770718227 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc0ab600 con 0x7fdb0c1539a0 2023-10-30T13:15:10.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb0affd640 1 -- 172.21.15.125:0/1770718227 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdafc0a7050 con 0x7fdb0c1539a0 2023-10-30T13:15:10.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb0affd640 1 -- 172.21.15.125:0/1770718227 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdafc0ab980 con 0x7fdb0c1539a0 2023-10-30T13:15:10.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 -- 172.21.15.125:0/1770718227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb0c1539a0 msgr2=0x7fdb0c15a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:10.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 --2- 172.21.15.125:0/1770718227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb0c1539a0 0x7fdb0c15a380 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fdafc0097c0 tx=0x7fdafc01c3c0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 -- 172.21.15.125:0/1770718227 shutdown_connections 2023-10-30T13:15:10.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 --2- 172.21.15.125:0/1770718227 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb0c069e90 0x7fdb0c06a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 --2- 172.21.15.125:0/1770718227 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb0c1539a0 0x7fdb0c15a380 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 -- 172.21.15.125:0/1770718227 >> 172.21.15.125:0/1770718227 conn(0x7fdb0c0c7910 msgr2=0x7fdb0c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:10.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 -- 172.21.15.125:0/1770718227 shutdown_connections 2023-10-30T13:15:10.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 -- 172.21.15.125:0/1770718227 wait complete. 2023-10-30T13:15:10.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 Processor -- start 2023-10-30T13:15:10.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 -- start start 2023-10-30T13:15:10.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb0c069e90 0x7fdb0c0c2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:10.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb0c070a00 0x7fdb0c1f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:10.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb0c1539a0 0x7fdb0c147c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:10.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c0c3d80 con 0x7fdb0c1539a0 2023-10-30T13:15:10.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c1f60e0 con 0x7fdb0c069e90 2023-10-30T13:15:10.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb1293e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0c148a50 con 0x7fdb0c070a00 2023-10-30T13:15:10.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.764+0000 7fdb10edb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb0c1539a0 0x7fdb0c147c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:10.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb10edb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb0c1539a0 0x7fdb0c147c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45610/0 (socket says 172.21.15.125:45610) 2023-10-30T13:15:10.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb10edb640 1 -- 172.21.15.125:0/2080116824 learned_addr learned my addr 172.21.15.125:0/2080116824 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:10.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb0bfff640 1 --2- 172.21.15.125:0/2080116824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb0c069e90 0x7fdb0c0c2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:10.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb0b7fe640 1 --2- 172.21.15.125:0/2080116824 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb0c070a00 0x7fdb0c1f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:10.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb10edb640 1 -- 172.21.15.125:0/2080116824 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb0c070a00 msgr2=0x7fdb0c1f6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:10.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb10edb640 1 --2- 172.21.15.125:0/2080116824 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb0c070a00 0x7fdb0c1f6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb10edb640 1 -- 172.21.15.125:0/2080116824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb0c069e90 msgr2=0x7fdb0c0c2070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:10.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb10edb640 1 --2- 172.21.15.125:0/2080116824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb0c069e90 0x7fdb0c0c2070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb10edb640 1 -- 172.21.15.125:0/2080116824 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb04009b30 con 0x7fdb0c1539a0 2023-10-30T13:15:10.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb10edb640 1 --2- 172.21.15.125:0/2080116824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb0c1539a0 0x7fdb0c147c10 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400efc0 tx=0x7fdb0400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:10.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb08ff9640 1 -- 172.21.15.125:0/2080116824 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb04018070 con 0x7fdb0c1539a0 2023-10-30T13:15:10.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb08ff9640 1 -- 172.21.15.125:0/2080116824 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0400f040 con 0x7fdb0c1539a0 2023-10-30T13:15:10.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb1293e640 1 -- 172.21.15.125:0/2080116824 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdafc009420 con 0x7fdb0c1539a0 2023-10-30T13:15:10.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb08ff9640 1 -- 172.21.15.125:0/2080116824 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb0400ce00 con 0x7fdb0c1539a0 2023-10-30T13:15:10.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb1293e640 1 -- 172.21.15.125:0/2080116824 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0c1f5ad0 con 0x7fdb0c1539a0 2023-10-30T13:15:10.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb1293e640 1 -- 172.21.15.125:0/2080116824 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb0c1f6290 con 0x7fdb0c1539a0 2023-10-30T13:15:10.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.768+0000 7fdb08ff9640 1 -- 172.21.15.125:0/2080116824 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdb04014480 con 0x7fdb0c1539a0 2023-10-30T13:15:10.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.772+0000 7fdb08ff9640 1 --2- 172.21.15.125:0/2080116824 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdaf80664a0 0x7fdaf8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:10.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.772+0000 7fdb08ff9640 1 -- 172.21.15.125:0/2080116824 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7fdb040848a0 con 0x7fdb0c1539a0 2023-10-30T13:15:10.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.772+0000 7fdb0bfff640 1 --2- 172.21.15.125:0/2080116824 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdaf80664a0 0x7fdaf8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:10.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.772+0000 7fdb0bfff640 1 --2- 172.21.15.125:0/2080116824 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdaf80664a0 0x7fdaf8068960 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fdafc0ad000 tx=0x7fdafc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:10.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.776+0000 7fdb08ff9640 1 -- 172.21.15.125:0/2080116824 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdb0c1f6290 con 0x7fdb0c1539a0 2023-10-30T13:15:10.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.960+0000 7fdb1293e640 1 -- 172.21.15.125:0/2080116824 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb0c1f6250 con 0x7fdb0c1539a0 2023-10-30T13:15:10.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.960+0000 7fdb08ff9640 1 -- 172.21.15.125:0/2080116824 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 214 v214) v1 ==== 95+0+5755 (secure 0 0 0) 0x7fdb040514e0 con 0x7fdb0c1539a0 2023-10-30T13:15:10.962 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:10.962 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":214,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":214,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:05.766744+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:10.963 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 214 2023-10-30T13:15:10.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.960+0000 7fdb1293e640 1 -- 172.21.15.125:0/2080116824 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdaf80664a0 msgr2=0x7fdaf8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:10.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.960+0000 7fdb1293e640 1 --2- 172.21.15.125:0/2080116824 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdaf80664a0 0x7fdaf8068960 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7fdafc0ad000 tx=0x7fdafc008040 comp rx=0 tx=0).stop 2023-10-30T13:15:10.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.960+0000 7fdb1293e640 1 -- 172.21.15.125:0/2080116824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb0c1539a0 msgr2=0x7fdb0c147c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:10.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.960+0000 7fdb1293e640 1 --2- 172.21.15.125:0/2080116824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb0c1539a0 0x7fdb0c147c10 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7fdb0400efc0 tx=0x7fdb0400c490 comp rx=0 tx=0).stop 2023-10-30T13:15:10.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.960+0000 7fdb1293e640 1 -- 172.21.15.125:0/2080116824 shutdown_connections 2023-10-30T13:15:10.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.960+0000 7fdb1293e640 1 --2- 172.21.15.125:0/2080116824 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdaf80664a0 0x7fdaf8068960 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.960+0000 7fdb1293e640 1 --2- 172.21.15.125:0/2080116824 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb0c1539a0 0x7fdb0c147c10 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.960+0000 7fdb1293e640 1 --2- 172.21.15.125:0/2080116824 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb0c070a00 0x7fdb0c1f6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.960+0000 7fdb1293e640 1 --2- 172.21.15.125:0/2080116824 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb0c069e90 0x7fdb0c0c2070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.960+0000 7fdb1293e640 1 -- 172.21.15.125:0/2080116824 >> 172.21.15.125:0/2080116824 conn(0x7fdb0c0c7910 msgr2=0x7fdb0c14f190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.960+0000 7fdb1293e640 1 -- 172.21.15.125:0/2080116824 shutdown_connections 2023-10-30T13:15:10.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:10.960+0000 7fdb1293e640 1 -- 172.21.15.125:0/2080116824 wait complete. 2023-10-30T13:15:10.984 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:12.985 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:13.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.120+0000 7fc8b0aea640 1 Processor -- start 2023-10-30T13:15:13.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.120+0000 7fc8b0aea640 1 -- start start 2023-10-30T13:15:13.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.124+0000 7fc8b0aea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8ac0c6bd0 0x7fc8ac06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:13.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.124+0000 7fc8b0aea640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc8ac071110 0x7fc8ac158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:13.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.124+0000 7fc8b0aea640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8ac066cf0 con 0x7fc8ac06f7a0 2023-10-30T13:15:13.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.124+0000 7fc8b0aea640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8ac0700c0 con 0x7fc8ac071110 2023-10-30T13:15:13.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.124+0000 7fc8b0aea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac0c70d0 con 0x7fc8ac0c6bd0 2023-10-30T13:15:13.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.124+0000 7fc8aad9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc8ac071110 0x7fc8ac158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:13.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.124+0000 7fc8aad9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc8ac071110 0x7fc8ac158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45618/0 (socket says 172.21.15.125:45618) 2023-10-30T13:15:13.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.124+0000 7fc8aad9d640 1 -- 172.21.15.125:0/3002767473 learned_addr learned my addr 172.21.15.125:0/3002767473 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:13.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.124+0000 7fc8a9d9b640 1 --2- 172.21.15.125:0/3002767473 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8ac0c6bd0 0x7fc8ac06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:13.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.124+0000 7fc8aad9d640 1 -- 172.21.15.125:0/3002767473 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8ac0c6bd0 msgr2=0x7fc8ac06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:13.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.124+0000 7fc8aad9d640 1 --2- 172.21.15.125:0/3002767473 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8ac0c6bd0 0x7fc8ac06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.124+0000 7fc8aad9d640 1 -- 172.21.15.125:0/3002767473 >> v1:172.21.15.136:6789/0 conn(0x7fc8ac06f7a0 legacy=0x7fc8ac069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:13.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.124+0000 7fc8aad9d640 1 -- 172.21.15.125:0/3002767473 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ac0c7390 con 0x7fc8ac071110 2023-10-30T13:15:13.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.128+0000 7fc8aad9d640 1 --2- 172.21.15.125:0/3002767473 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8ac071110 0x7fc8ac158110 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4009e50 tx=0x7fc8a401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399387a3230e9e1f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:13.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.128+0000 7fc8a959a640 1 -- 172.21.15.125:0/3002767473 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a40af400 con 0x7fc8ac071110 2023-10-30T13:15:13.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.128+0000 7fc8a959a640 1 -- 172.21.15.125:0/3002767473 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a40ab050 con 0x7fc8ac071110 2023-10-30T13:15:13.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.128+0000 7fc8a959a640 1 -- 172.21.15.125:0/3002767473 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a40af780 con 0x7fc8ac071110 2023-10-30T13:15:13.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.128+0000 7fc8b0aea640 1 -- 172.21.15.125:0/3002767473 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8ac071110 msgr2=0x7fc8ac158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:13.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.128+0000 7fc8b0aea640 1 --2- 172.21.15.125:0/3002767473 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8ac071110 0x7fc8ac158110 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4009e50 tx=0x7fc8a401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.128+0000 7fc8b0aea640 1 -- 172.21.15.125:0/3002767473 shutdown_connections 2023-10-30T13:15:13.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.128+0000 7fc8b0aea640 1 --2- 172.21.15.125:0/3002767473 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8ac071110 0x7fc8ac158110 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.128+0000 7fc8b0aea640 1 --2- 172.21.15.125:0/3002767473 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8ac0c6bd0 0x7fc8ac06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.128+0000 7fc8b0aea640 1 -- 172.21.15.125:0/3002767473 >> 172.21.15.125:0/3002767473 conn(0x7fc8ac0c7910 msgr2=0x7fc8ac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:13.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.128+0000 7fc8b0aea640 1 -- 172.21.15.125:0/3002767473 shutdown_connections 2023-10-30T13:15:13.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.128+0000 7fc8b0aea640 1 -- 172.21.15.125:0/3002767473 wait complete. 2023-10-30T13:15:13.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.128+0000 7fc8b0aea640 1 Processor -- start 2023-10-30T13:15:13.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.128+0000 7fc8b0aea640 1 -- start start 2023-10-30T13:15:13.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8b0aea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8ac06f7a0 0x7fc8ac0c40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:13.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8b0aea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8ac071110 0x7fc8ac149730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:13.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8b0aea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8ac0c6bd0 0x7fc8ac149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:13.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8b0aea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac0afbe0 con 0x7fc8ac0c6bd0 2023-10-30T13:15:13.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8b0aea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac0aef90 con 0x7fc8ac071110 2023-10-30T13:15:13.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8b0aea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8ac0b04b0 con 0x7fc8ac06f7a0 2023-10-30T13:15:13.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8aad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8ac0c6bd0 0x7fc8ac149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:13.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8aad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8ac0c6bd0 0x7fc8ac149c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45628/0 (socket says 172.21.15.125:45628) 2023-10-30T13:15:13.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8aad9d640 1 -- 172.21.15.125:0/4059838128 learned_addr learned my addr 172.21.15.125:0/4059838128 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:13.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8a9d9b640 1 --2- 172.21.15.125:0/4059838128 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8ac071110 0x7fc8ac149730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:13.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8aa59c640 1 --2- 172.21.15.125:0/4059838128 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8ac06f7a0 0x7fc8ac0c40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:13.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8aad9d640 1 -- 172.21.15.125:0/4059838128 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8ac06f7a0 msgr2=0x7fc8ac0c40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:13.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8aad9d640 1 --2- 172.21.15.125:0/4059838128 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8ac06f7a0 0x7fc8ac0c40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8aad9d640 1 -- 172.21.15.125:0/4059838128 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8ac071110 msgr2=0x7fc8ac149730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:13.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8aad9d640 1 --2- 172.21.15.125:0/4059838128 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8ac071110 0x7fc8ac149730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8aad9d640 1 -- 172.21.15.125:0/4059838128 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a4009b30 con 0x7fc8ac0c6bd0 2023-10-30T13:15:13.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8aad9d640 1 --2- 172.21.15.125:0/4059838128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8ac0c6bd0 0x7fc8ac149c70 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fc8a401ce20 tx=0x7fc8a401bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:13.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc89affd640 1 -- 172.21.15.125:0/4059838128 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a40afde0 con 0x7fc8ac0c6bd0 2023-10-30T13:15:13.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc89affd640 1 -- 172.21.15.125:0/4059838128 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8a4009510 con 0x7fc8ac0c6bd0 2023-10-30T13:15:13.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8b0aea640 1 -- 172.21.15.125:0/4059838128 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8ac0ae380 con 0x7fc8ac0c6bd0 2023-10-30T13:15:13.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc89affd640 1 -- 172.21.15.125:0/4059838128 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8a40ba660 con 0x7fc8ac0c6bd0 2023-10-30T13:15:13.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8b0aea640 1 -- 172.21.15.125:0/4059838128 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8ac0ae5f0 con 0x7fc8ac0c6bd0 2023-10-30T13:15:13.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc8b0aea640 1 -- 172.21.15.125:0/4059838128 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8ac0af0d0 con 0x7fc8ac0c6bd0 2023-10-30T13:15:13.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.132+0000 7fc89affd640 1 -- 172.21.15.125:0/4059838128 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc8a40b6040 con 0x7fc8ac0c6bd0 2023-10-30T13:15:13.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.136+0000 7fc89affd640 1 --2- 172.21.15.125:0/4059838128 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc88c0664a0 0x7fc88c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:13.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.136+0000 7fc89affd640 1 -- 172.21.15.125:0/4059838128 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7fc8a4122900 con 0x7fc8ac0c6bd0 2023-10-30T13:15:13.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.136+0000 7fc8aa59c640 1 --2- 172.21.15.125:0/4059838128 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc88c0664a0 0x7fc88c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:13.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.136+0000 7fc8aa59c640 1 --2- 172.21.15.125:0/4059838128 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc88c0664a0 0x7fc88c068960 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fc89c000f70 tx=0x7fc89c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:13.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.140+0000 7fc89affd640 1 -- 172.21.15.125:0/4059838128 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc8a40ef670 con 0x7fc8ac0c6bd0 2023-10-30T13:15:13.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.320+0000 7fc8b0aea640 1 -- 172.21.15.125:0/4059838128 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8ac04e170 con 0x7fc8ac0c6bd0 2023-10-30T13:15:13.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.320+0000 7fc89affd640 1 -- 172.21.15.125:0/4059838128 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+5757 (secure 0 0 0) 0x7fc8a40f2c90 con 0x7fc8ac0c6bd0 2023-10-30T13:15:13.323 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:13.324 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":215,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:12.426274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:resolve","state_seq":30,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:13.324 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 215 2023-10-30T13:15:13.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.324+0000 7fc8b0aea640 1 -- 172.21.15.125:0/4059838128 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc88c0664a0 msgr2=0x7fc88c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:13.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.324+0000 7fc8b0aea640 1 --2- 172.21.15.125:0/4059838128 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc88c0664a0 0x7fc88c068960 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fc89c000f70 tx=0x7fc89c009040 comp rx=0 tx=0).stop 2023-10-30T13:15:13.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.324+0000 7fc8b0aea640 1 -- 172.21.15.125:0/4059838128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8ac0c6bd0 msgr2=0x7fc8ac149c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:13.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.324+0000 7fc8b0aea640 1 --2- 172.21.15.125:0/4059838128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8ac0c6bd0 0x7fc8ac149c70 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7fc8a401ce20 tx=0x7fc8a401bf90 comp rx=0 tx=0).stop 2023-10-30T13:15:13.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.324+0000 7fc8b0aea640 1 -- 172.21.15.125:0/4059838128 shutdown_connections 2023-10-30T13:15:13.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.324+0000 7fc8b0aea640 1 --2- 172.21.15.125:0/4059838128 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc88c0664a0 0x7fc88c068960 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.324+0000 7fc8b0aea640 1 --2- 172.21.15.125:0/4059838128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc8ac0c6bd0 0x7fc8ac149c70 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.324+0000 7fc8b0aea640 1 --2- 172.21.15.125:0/4059838128 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc8ac071110 0x7fc8ac149730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.324+0000 7fc8b0aea640 1 --2- 172.21.15.125:0/4059838128 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8ac06f7a0 0x7fc8ac0c40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:13.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.324+0000 7fc8b0aea640 1 -- 172.21.15.125:0/4059838128 >> 172.21.15.125:0/4059838128 conn(0x7fc8ac0c7910 msgr2=0x7fc8ac0c94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:13.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.324+0000 7fc8b0aea640 1 -- 172.21.15.125:0/4059838128 shutdown_connections 2023-10-30T13:15:13.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:13.324+0000 7fc8b0aea640 1 -- 172.21.15.125:0/4059838128 wait complete. 2023-10-30T13:15:13.344 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:15.345 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:15.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.492+0000 7f4ab271d640 1 Processor -- start 2023-10-30T13:15:15.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.492+0000 7f4ab271d640 1 -- start start 2023-10-30T13:15:15.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.492+0000 7f4ab271d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4aac0c6bd0 0x7f4aac06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:15.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.492+0000 7f4ab271d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac066cf0 con 0x7f4aac06f7a0 2023-10-30T13:15:15.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.492+0000 7f4ab271d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac0c70d0 con 0x7f4aac071110 2023-10-30T13:15:15.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.492+0000 7f4ab271d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aac0700c0 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.492+0000 7f4aabfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4aac06f7a0 0x7f4aac069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44886/0 (socket says 172.21.15.125:44886) 2023-10-30T13:15:15.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.492+0000 7f4aabfff640 1 -- 172.21.15.125:0/3989446725 learned_addr learned my addr 172.21.15.125:0/3989446725 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:15:15.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.492+0000 7f4aab7fe640 1 --2- 172.21.15.125:0/3989446725 >> v2:172.21.15.136:3300/0 conn(0x7f4aac0c6bd0 0x7f4aac06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:15.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.492+0000 7f4aaaffd640 1 -- 172.21.15.125:0/3989446725 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452455994 0 0) 0x7f4aac066cf0 con 0x7f4aac06f7a0 2023-10-30T13:15:15.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.492+0000 7f4aaaffd640 1 -- 172.21.15.125:0/3989446725 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a94003530 con 0x7f4aac06f7a0 2023-10-30T13:15:15.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.492+0000 7f4aab7fe640 1 -- 172.21.15.125:0/3989446725 >> v1:172.21.15.136:6789/0 conn(0x7f4aac071110 legacy=0x7f4aac158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:15.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.492+0000 7f4aab7fe640 1 -- 172.21.15.125:0/3989446725 >> v1:172.21.15.125:6789/0 conn(0x7f4aac06f7a0 legacy=0x7f4aac069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:15.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.492+0000 7f4aab7fe640 1 -- 172.21.15.125:0/3989446725 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac0c7390 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.496+0000 7f4aab7fe640 1 --2- 172.21.15.125:0/3989446725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4aac0c6bd0 0x7f4aac06fb80 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00097c0 tx=0x7f4aa001ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e432359327f171b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:15.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.496+0000 7f4aaaffd640 1 -- 172.21.15.125:0/3989446725 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00a8970 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.496+0000 7f4aaaffd640 1 -- 172.21.15.125:0/3989446725 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa00a8ad0 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.496+0000 7f4aaaffd640 1 -- 172.21.15.125:0/3989446725 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa00a8e60 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.496+0000 7f4ab271d640 1 -- 172.21.15.125:0/3989446725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4aac0c6bd0 msgr2=0x7f4aac06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:15.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.496+0000 7f4ab271d640 1 --2- 172.21.15.125:0/3989446725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4aac0c6bd0 0x7f4aac06fb80 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f4aa00097c0 tx=0x7f4aa001ec70 comp rx=0 tx=0).stop 2023-10-30T13:15:15.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.496+0000 7f4ab271d640 1 -- 172.21.15.125:0/3989446725 shutdown_connections 2023-10-30T13:15:15.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.496+0000 7f4ab271d640 1 --2- 172.21.15.125:0/3989446725 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4aac0c6bd0 0x7f4aac06fb80 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.496+0000 7f4ab271d640 1 -- 172.21.15.125:0/3989446725 >> 172.21.15.125:0/3989446725 conn(0x7f4aac0c7910 msgr2=0x7f4aac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:15.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.496+0000 7f4ab271d640 1 -- 172.21.15.125:0/3989446725 shutdown_connections 2023-10-30T13:15:15.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.496+0000 7f4ab271d640 1 -- 172.21.15.125:0/3989446725 wait complete. 2023-10-30T13:15:15.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab271d640 1 Processor -- start 2023-10-30T13:15:15.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab271d640 1 -- start start 2023-10-30T13:15:15.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab271d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4aac06f7a0 0x7f4aac146940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:15.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab271d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4aac071110 0x7f4aac14f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:15.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab271d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4aac0c6bd0 0x7f4aac14f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:15.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab271d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac14eb80 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab271d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac14ee60 con 0x7f4aac071110 2023-10-30T13:15:15.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab271d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac0c1ee0 con 0x7f4aac06f7a0 2023-10-30T13:15:15.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab0cba640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4aac0c6bd0 0x7f4aac14f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:15.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab0cba640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4aac0c6bd0 0x7f4aac14f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45632/0 (socket says 172.21.15.125:45632) 2023-10-30T13:15:15.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab0cba640 1 -- 172.21.15.125:0/2005449009 learned_addr learned my addr 172.21.15.125:0/2005449009 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:15.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4aab7fe640 1 --2- 172.21.15.125:0/2005449009 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4aac071110 0x7f4aac14f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:15.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4aabfff640 1 --2- 172.21.15.125:0/2005449009 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4aac06f7a0 0x7f4aac146940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:15.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab0cba640 1 -- 172.21.15.125:0/2005449009 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4aac06f7a0 msgr2=0x7f4aac146940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:15.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab0cba640 1 --2- 172.21.15.125:0/2005449009 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4aac06f7a0 0x7f4aac146940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab0cba640 1 -- 172.21.15.125:0/2005449009 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4aac071110 msgr2=0x7f4aac14f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:15.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab0cba640 1 --2- 172.21.15.125:0/2005449009 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4aac071110 0x7f4aac14f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab0cba640 1 -- 172.21.15.125:0/2005449009 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aa0009420 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab0cba640 1 --2- 172.21.15.125:0/2005449009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4aac0c6bd0 0x7f4aac14f830 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f4aa400c730 tx=0x7f4aa400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:15.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4aa8ff9640 1 -- 172.21.15.125:0/2005449009 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4003b80 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4aa8ff9640 1 -- 172.21.15.125:0/2005449009 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa4003ce0 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4aa8ff9640 1 -- 172.21.15.125:0/2005449009 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa400aa30 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab271d640 1 -- 172.21.15.125:0/2005449009 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aac14dcd0 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab271d640 1 -- 172.21.15.125:0/2005449009 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aac14e290 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.500+0000 7f4ab271d640 1 -- 172.21.15.125:0/2005449009 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aac04e170 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.504+0000 7f4aa8ff9640 1 -- 172.21.15.125:0/2005449009 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4aa400ab90 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.504+0000 7f4aa8ff9640 1 --2- 172.21.15.125:0/2005449009 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a980664a0 0x7f4a98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:15.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.504+0000 7f4aa8ff9640 1 -- 172.21.15.125:0/2005449009 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7f4aa4084510 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.504+0000 7f4aabfff640 1 --2- 172.21.15.125:0/2005449009 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a980664a0 0x7f4a98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:15.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.504+0000 7f4aabfff640 1 --2- 172.21.15.125:0/2005449009 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a980664a0 0x7f4a98068960 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0011f0 tx=0x7f4a9c007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:15.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.508+0000 7f4aa8ff9640 1 -- 172.21.15.125:0/2005449009 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4aa40510b0 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.688+0000 7f4ab271d640 1 -- 172.21.15.125:0/2005449009 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4aac14efa0 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.688+0000 7f4aa8ff9640 1 -- 172.21.15.125:0/2005449009 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 215 v215) v1 ==== 95+0+5757 (secure 0 0 0) 0x7f4aa40546d0 con 0x7f4aac0c6bd0 2023-10-30T13:15:15.690 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:15.691 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":215,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":215,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:12.426274+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:resolve","state_seq":30,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:15.691 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 215 2023-10-30T13:15:15.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.688+0000 7f4ab271d640 1 -- 172.21.15.125:0/2005449009 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a980664a0 msgr2=0x7f4a98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:15.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.688+0000 7f4ab271d640 1 --2- 172.21.15.125:0/2005449009 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a980664a0 0x7f4a98068960 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c0011f0 tx=0x7f4a9c007d40 comp rx=0 tx=0).stop 2023-10-30T13:15:15.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.688+0000 7f4ab271d640 1 -- 172.21.15.125:0/2005449009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4aac0c6bd0 msgr2=0x7f4aac14f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:15.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.688+0000 7f4ab271d640 1 --2- 172.21.15.125:0/2005449009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4aac0c6bd0 0x7f4aac14f830 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f4aa400c730 tx=0x7f4aa400cbf0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.688+0000 7f4ab271d640 1 -- 172.21.15.125:0/2005449009 shutdown_connections 2023-10-30T13:15:15.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.688+0000 7f4ab271d640 1 --2- 172.21.15.125:0/2005449009 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a980664a0 0x7f4a98068960 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.688+0000 7f4ab271d640 1 --2- 172.21.15.125:0/2005449009 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4aac0c6bd0 0x7f4aac14f830 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.688+0000 7f4ab271d640 1 --2- 172.21.15.125:0/2005449009 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4aac071110 0x7f4aac14f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.688+0000 7f4ab271d640 1 --2- 172.21.15.125:0/2005449009 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4aac06f7a0 0x7f4aac146940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:15.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.688+0000 7f4ab271d640 1 -- 172.21.15.125:0/2005449009 >> 172.21.15.125:0/2005449009 conn(0x7f4aac0c7910 msgr2=0x7f4aac0c3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:15.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.688+0000 7f4ab271d640 1 -- 172.21.15.125:0/2005449009 shutdown_connections 2023-10-30T13:15:15.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:15.688+0000 7f4ab271d640 1 -- 172.21.15.125:0/2005449009 wait complete. 2023-10-30T13:15:15.712 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:16.910 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:15:16.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:15:17.713 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:17.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.856+0000 7f2c9a93e640 1 Processor -- start 2023-10-30T13:15:17.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.856+0000 7f2c9a93e640 1 -- start start 2023-10-30T13:15:17.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c9a93e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2c940c6b80 0x7f2c940bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:17.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c9a93e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2c9413f4a0 0x7f2c940c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:17.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c9a93e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c940c0a50 con 0x7f2c940c6f60 2023-10-30T13:15:17.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c9a93e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c9406f6e0 con 0x7f2c9413f4a0 2023-10-30T13:15:17.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c9a93e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c9406f9c0 con 0x7f2c940c6b80 2023-10-30T13:15:17.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c937fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2c940c6f60 0x7f2c9413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44894/0 (socket says 172.21.15.125:44894) 2023-10-30T13:15:17.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c937fe640 1 -- 172.21.15.125:0/2072714217 learned_addr learned my addr 172.21.15.125:0/2072714217 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:15:17.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c98edb640 1 --2- 172.21.15.125:0/2072714217 >> v2:172.21.15.125:3300/0 conn(0x7f2c9413f4a0 0x7f2c940c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:17.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c93fff640 1 --2- 172.21.15.125:0/2072714217 >> v2:172.21.15.136:3300/0 conn(0x7f2c940c6b80 0x7f2c940bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:17.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c92ffd640 1 -- 172.21.15.125:0/2072714217 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985650760 0 0) 0x7f2c940c0a50 con 0x7f2c940c6f60 2023-10-30T13:15:17.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c92ffd640 1 -- 172.21.15.125:0/2072714217 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c7c003530 con 0x7f2c940c6f60 2023-10-30T13:15:17.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c98edb640 1 -- 172.21.15.125:0/2072714217 >> v2:172.21.15.136:3300/0 conn(0x7f2c940c6b80 msgr2=0x7f2c940bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:17.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c98edb640 1 --2- 172.21.15.125:0/2072714217 >> v2:172.21.15.136:3300/0 conn(0x7f2c940c6b80 0x7f2c940bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:17.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c98edb640 1 -- 172.21.15.125:0/2072714217 >> v1:172.21.15.125:6789/0 conn(0x7f2c940c6f60 legacy=0x7f2c9413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:17.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.860+0000 7f2c98edb640 1 -- 172.21.15.125:0/2072714217 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9406fca0 con 0x7f2c9413f4a0 2023-10-30T13:15:17.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c98edb640 1 --2- 172.21.15.125:0/2072714217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c9413f4a0 0x7f2c940c0400 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f2c9406f1b0 tx=0x7f2c8c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c2ac781c89fbf5 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:17.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c92ffd640 1 -- 172.21.15.125:0/2072714217 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c0a58e0 con 0x7f2c9413f4a0 2023-10-30T13:15:17.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c92ffd640 1 -- 172.21.15.125:0/2072714217 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c0a5a40 con 0x7f2c9413f4a0 2023-10-30T13:15:17.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c92ffd640 1 -- 172.21.15.125:0/2072714217 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8c0a5dd0 con 0x7f2c9413f4a0 2023-10-30T13:15:17.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c9a93e640 1 -- 172.21.15.125:0/2072714217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c9413f4a0 msgr2=0x7f2c940c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:17.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c9a93e640 1 --2- 172.21.15.125:0/2072714217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c9413f4a0 0x7f2c940c0400 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f2c9406f1b0 tx=0x7f2c8c01c910 comp rx=0 tx=0).stop 2023-10-30T13:15:17.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c9a93e640 1 -- 172.21.15.125:0/2072714217 shutdown_connections 2023-10-30T13:15:17.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c9a93e640 1 --2- 172.21.15.125:0/2072714217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c9413f4a0 0x7f2c940c0400 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:17.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c9a93e640 1 --2- 172.21.15.125:0/2072714217 >> v2:172.21.15.136:3300/0 conn(0x7f2c940c6b80 0x7f2c940bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:17.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c9a93e640 1 -- 172.21.15.125:0/2072714217 >> 172.21.15.125:0/2072714217 conn(0x7f2c940c7910 msgr2=0x7f2c940c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:17.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c9a93e640 1 -- 172.21.15.125:0/2072714217 shutdown_connections 2023-10-30T13:15:17.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c9a93e640 1 -- 172.21.15.125:0/2072714217 wait complete. 2023-10-30T13:15:17.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c9a93e640 1 Processor -- start 2023-10-30T13:15:17.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c9a93e640 1 -- start start 2023-10-30T13:15:17.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.864+0000 7f2c9a93e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c940c6b80 0x7f2c9413d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:17.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c9a93e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c940c6f60 0x7f2c940c9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:17.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c9a93e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c9413f4a0 0x7f2c940c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:17.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c9a93e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c94155bd0 con 0x7f2c940c6b80 2023-10-30T13:15:17.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c9a93e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c94155e70 con 0x7f2c9413f4a0 2023-10-30T13:15:17.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c9a93e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c940c8cb0 con 0x7f2c940c6f60 2023-10-30T13:15:17.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c93fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c940c6b80 0x7f2c9413d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:17.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c93fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c940c6b80 0x7f2c9413d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45640/0 (socket says 172.21.15.125:45640) 2023-10-30T13:15:17.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c93fff640 1 -- 172.21.15.125:0/999167799 learned_addr learned my addr 172.21.15.125:0/999167799 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:17.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c937fe640 1 --2- 172.21.15.125:0/999167799 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c940c6f60 0x7f2c940c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:17.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c98edb640 1 --2- 172.21.15.125:0/999167799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c9413f4a0 0x7f2c940c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:17.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c93fff640 1 -- 172.21.15.125:0/999167799 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c940c6f60 msgr2=0x7f2c940c9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:17.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c93fff640 1 --2- 172.21.15.125:0/999167799 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c940c6f60 0x7f2c940c9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:17.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c93fff640 1 -- 172.21.15.125:0/999167799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c9413f4a0 msgr2=0x7f2c940c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:17.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c93fff640 1 --2- 172.21.15.125:0/999167799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c9413f4a0 0x7f2c940c9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:17.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c93fff640 1 -- 172.21.15.125:0/999167799 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c009a70 con 0x7f2c940c6b80 2023-10-30T13:15:17.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c93fff640 1 --2- 172.21.15.125:0/999167799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c940c6b80 0x7f2c9413d990 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f2c8400d800 tx=0x7f2c8400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:17.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c90ff9640 1 -- 172.21.15.125:0/999167799 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8400b6c0 con 0x7f2c940c6b80 2023-10-30T13:15:17.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c90ff9640 1 -- 172.21.15.125:0/999167799 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8400b820 con 0x7f2c940c6b80 2023-10-30T13:15:17.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c9a93e640 1 -- 172.21.15.125:0/999167799 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c940c8f90 con 0x7f2c940c6b80 2023-10-30T13:15:17.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c90ff9640 1 -- 172.21.15.125:0/999167799 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c8400bb90 con 0x7f2c940c6b80 2023-10-30T13:15:17.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c9a93e640 1 -- 172.21.15.125:0/999167799 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c94156450 con 0x7f2c940c6b80 2023-10-30T13:15:17.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c9a93e640 1 -- 172.21.15.125:0/999167799 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9404e170 con 0x7f2c940c6b80 2023-10-30T13:15:17.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.868+0000 7f2c90ff9640 1 -- 172.21.15.125:0/999167799 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c8400bcf0 con 0x7f2c940c6b80 2023-10-30T13:15:17.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.872+0000 7f2c90ff9640 1 --2- 172.21.15.125:0/999167799 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c800664a0 0x7f2c80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:17.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.872+0000 7f2c90ff9640 1 -- 172.21.15.125:0/999167799 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7f2c84084e50 con 0x7f2c940c6b80 2023-10-30T13:15:17.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.872+0000 7f2c937fe640 1 --2- 172.21.15.125:0/999167799 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c800664a0 0x7f2c80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:17.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.872+0000 7f2c937fe640 1 --2- 172.21.15.125:0/999167799 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c800664a0 0x7f2c80068960 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2c94073ef0 tx=0x7f2c88009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:17.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:17.876+0000 7f2c90ff9640 1 -- 172.21.15.125:0/999167799 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c840518c0 con 0x7f2c940c6b80 2023-10-30T13:15:18.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:18.052+0000 7f2c9a93e640 1 -- 172.21.15.125:0/999167799 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c94135010 con 0x7f2c940c6b80 2023-10-30T13:15:18.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:18.056+0000 7f2c90ff9640 1 -- 172.21.15.125:0/999167799 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 216 v216) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f2c84054ee0 con 0x7f2c940c6b80 2023-10-30T13:15:18.057 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:18.057 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":216,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":216,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:15.793121+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:resolve","state_seq":30,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:18.058 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 216 2023-10-30T13:15:18.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:18.056+0000 7f2c9a93e640 1 -- 172.21.15.125:0/999167799 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c800664a0 msgr2=0x7f2c80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:18.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:18.056+0000 7f2c9a93e640 1 --2- 172.21.15.125:0/999167799 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c800664a0 0x7f2c80068960 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f2c94073ef0 tx=0x7f2c88009040 comp rx=0 tx=0).stop 2023-10-30T13:15:18.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:18.056+0000 7f2c9a93e640 1 -- 172.21.15.125:0/999167799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c940c6b80 msgr2=0x7f2c9413d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:18.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:18.056+0000 7f2c9a93e640 1 --2- 172.21.15.125:0/999167799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c940c6b80 0x7f2c9413d990 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f2c8400d800 tx=0x7f2c8400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:18.056+0000 7f2c9a93e640 1 -- 172.21.15.125:0/999167799 shutdown_connections 2023-10-30T13:15:18.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:18.056+0000 7f2c9a93e640 1 --2- 172.21.15.125:0/999167799 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c800664a0 0x7f2c80068960 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:18.056+0000 7f2c9a93e640 1 --2- 172.21.15.125:0/999167799 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c9413f4a0 0x7f2c940c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:18.056+0000 7f2c9a93e640 1 --2- 172.21.15.125:0/999167799 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c940c6f60 0x7f2c940c9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:18.056+0000 7f2c9a93e640 1 --2- 172.21.15.125:0/999167799 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c940c6b80 0x7f2c9413d990 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:18.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:18.056+0000 7f2c9a93e640 1 -- 172.21.15.125:0/999167799 >> 172.21.15.125:0/999167799 conn(0x7f2c940c7910 msgr2=0x7f2c94139790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:18.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:18.056+0000 7f2c9a93e640 1 -- 172.21.15.125:0/999167799 shutdown_connections 2023-10-30T13:15:18.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:18.056+0000 7f2c9a93e640 1 -- 172.21.15.125:0/999167799 wait complete. 2023-10-30T13:15:18.078 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 216, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 216, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_8443": { "addr": "172.21.15.136:6839/3776186190", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 3776186190, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 3776186190, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8443, "incarnation": 196, "join_fscid": -1, "name": "f", "rank": 2, "state": "up:active", "state_seq": 41 }, "gid_8529": { "addr": "172.21.15.125:6836/3950044434", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3950044434, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3950044434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 8529, "incarnation": 179, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 15 }, "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:resolve", "state_seq": 30 } }, "last_failure": 0, "last_failure_osd_epoch": 80, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:15:15.793121+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8529, "mds_1": 9150, "mds_2": 8443 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:15:20.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:20.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.224+0000 7fb8fb501640 1 Processor -- start 2023-10-30T13:15:20.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.224+0000 7fb8fb501640 1 -- start start 2023-10-30T13:15:20.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.228+0000 7fb8fb501640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb8f414fd50 0x7fb8f40c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.228+0000 7fb8fb501640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb8f40cb550 0x7fb8f40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.228+0000 7fb8fb501640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8f41560a0 con 0x7fb8f4071250 2023-10-30T13:15:20.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.228+0000 7fb8fb501640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f4156380 con 0x7fb8f40cb550 2023-10-30T13:15:20.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.228+0000 7fb8fb501640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8f4156660 con 0x7fb8f414fd50 2023-10-30T13:15:20.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.228+0000 7fb8f8a9c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb8f40cb550 0x7fb8f40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.228+0000 7fb8f9a9e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb8f4071250 0x7fb8f4071650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44896/0 (socket says 172.21.15.125:44896) 2023-10-30T13:15:20.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.228+0000 7fb8f9a9e640 1 -- 172.21.15.125:0/100677170 learned_addr learned my addr 172.21.15.125:0/100677170 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:15:20.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.228+0000 7fb8f929d640 1 --2- 172.21.15.125:0/100677170 >> v2:172.21.15.136:3300/0 conn(0x7fb8f414fd50 0x7fb8f40c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.231 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.228+0000 7fb8f8a9c640 1 -- 172.21.15.125:0/100677170 >> v2:172.21.15.136:3300/0 conn(0x7fb8f414fd50 msgr2=0x7fb8f40c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:20.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.228+0000 7fb8f8a9c640 1 --2- 172.21.15.125:0/100677170 >> v2:172.21.15.136:3300/0 conn(0x7fb8f414fd50 0x7fb8f40c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.228+0000 7fb8f8a9c640 1 -- 172.21.15.125:0/100677170 >> v1:172.21.15.125:6789/0 conn(0x7fb8f4071250 legacy=0x7fb8f4071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.228+0000 7fb8f8a9c640 1 -- 172.21.15.125:0/100677170 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f4156940 con 0x7fb8f40cb550 2023-10-30T13:15:20.232 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.228+0000 7fb8ebfff640 1 -- 172.21.15.125:0/100677170 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594320847 0 0) 0x7fb8f41560a0 con 0x7fb8f4071250 2023-10-30T13:15:20.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8f8a9c640 1 --2- 172.21.15.125:0/100677170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8f40cb550 0x7fb8f40cb9e0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fb8f40c90f0 tx=0x7fb8e401c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a999f24acda07d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:20.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8ebfff640 1 -- 172.21.15.125:0/100677170 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e40ab600 con 0x7fb8f40cb550 2023-10-30T13:15:20.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8ebfff640 1 -- 172.21.15.125:0/100677170 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e40a7050 con 0x7fb8f40cb550 2023-10-30T13:15:20.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8ebfff640 1 -- 172.21.15.125:0/100677170 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e40ab980 con 0x7fb8f40cb550 2023-10-30T13:15:20.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 -- 172.21.15.125:0/100677170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8f40cb550 msgr2=0x7fb8f40cb9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 --2- 172.21.15.125:0/100677170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8f40cb550 0x7fb8f40cb9e0 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fb8f40c90f0 tx=0x7fb8e401c3a0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 -- 172.21.15.125:0/100677170 shutdown_connections 2023-10-30T13:15:20.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 --2- 172.21.15.125:0/100677170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8f40cb550 0x7fb8f40cb9e0 unknown :-1 s=CLOSED pgs=1849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 --2- 172.21.15.125:0/100677170 >> v2:172.21.15.136:3300/0 conn(0x7fb8f414fd50 0x7fb8f40c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 -- 172.21.15.125:0/100677170 >> 172.21.15.125:0/100677170 conn(0x7fb8f406c6e0 msgr2=0x7fb8f4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:20.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 -- 172.21.15.125:0/100677170 shutdown_connections 2023-10-30T13:15:20.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 -- 172.21.15.125:0/100677170 wait complete. 2023-10-30T13:15:20.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 Processor -- start 2023-10-30T13:15:20.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 -- start start 2023-10-30T13:15:20.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8f4071250 0x7fb8f40c4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8f40cb550 0x7fb8f413dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8f414fd50 0x7fb8f413e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f414e430 con 0x7fb8f40cb550 2023-10-30T13:15:20.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f414f270 con 0x7fb8f4071250 2023-10-30T13:15:20.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8fb501640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8f414e660 con 0x7fb8f414fd50 2023-10-30T13:15:20.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.232+0000 7fb8f8a9c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8f40cb550 0x7fb8f413dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8f8a9c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8f40cb550 0x7fb8f413dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45660/0 (socket says 172.21.15.125:45660) 2023-10-30T13:15:20.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8f8a9c640 1 -- 172.21.15.125:0/2022719575 learned_addr learned my addr 172.21.15.125:0/2022719575 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:20.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8f929d640 1 --2- 172.21.15.125:0/2022719575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8f4071250 0x7fb8f40c4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8f9a9e640 1 --2- 172.21.15.125:0/2022719575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8f414fd50 0x7fb8f413e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8f8a9c640 1 -- 172.21.15.125:0/2022719575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8f414fd50 msgr2=0x7fb8f413e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8f8a9c640 1 --2- 172.21.15.125:0/2022719575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8f414fd50 0x7fb8f413e2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8f8a9c640 1 -- 172.21.15.125:0/2022719575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8f4071250 msgr2=0x7fb8f40c4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8f8a9c640 1 --2- 172.21.15.125:0/2022719575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8f4071250 0x7fb8f40c4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8f8a9c640 1 -- 172.21.15.125:0/2022719575 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e4009420 con 0x7fb8f40cb550 2023-10-30T13:15:20.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8f8a9c640 1 --2- 172.21.15.125:0/2022719575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8f40cb550 0x7fb8f413dd80 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40ae040 tx=0x7fb8e4019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:20.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8e9ffb640 1 -- 172.21.15.125:0/2022719575 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e40b4430 con 0x7fb8f40cb550 2023-10-30T13:15:20.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8e9ffb640 1 -- 172.21.15.125:0/2022719575 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e40b45b0 con 0x7fb8f40cb550 2023-10-30T13:15:20.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8fb501640 1 -- 172.21.15.125:0/2022719575 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8f414e940 con 0x7fb8f40cb550 2023-10-30T13:15:20.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8e9ffb640 1 -- 172.21.15.125:0/2022719575 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8e40b4960 con 0x7fb8f40cb550 2023-10-30T13:15:20.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8fb501640 1 -- 172.21.15.125:0/2022719575 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8f414ef00 con 0x7fb8f40cb550 2023-10-30T13:15:20.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8fb501640 1 -- 172.21.15.125:0/2022719575 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8f414f3e0 con 0x7fb8f40cb550 2023-10-30T13:15:20.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8e9ffb640 1 -- 172.21.15.125:0/2022719575 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb8e40b2040 con 0x7fb8f40cb550 2023-10-30T13:15:20.243 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.236+0000 7fb8e9ffb640 1 --2- 172.21.15.125:0/2022719575 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8d80664a0 0x7fb8d8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:20.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.240+0000 7fb8e9ffb640 1 -- 172.21.15.125:0/2022719575 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7fb8e411e670 con 0x7fb8f40cb550 2023-10-30T13:15:20.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.240+0000 7fb8f929d640 1 --2- 172.21.15.125:0/2022719575 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8d80664a0 0x7fb8d8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:20.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.240+0000 7fb8f929d640 1 --2- 172.21.15.125:0/2022719575 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8d80664a0 0x7fb8d8068960 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fb8f000b9d0 tx=0x7fb8f0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:20.244 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.240+0000 7fb8e9ffb640 1 -- 172.21.15.125:0/2022719575 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb8e40eb180 con 0x7fb8f40cb550 2023-10-30T13:15:20.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.424+0000 7fb8fb501640 1 -- 172.21.15.125:0/2022719575 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8f4156bf0 con 0x7fb8f40cb550 2023-10-30T13:15:20.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.424+0000 7fb8e9ffb640 1 -- 172.21.15.125:0/2022719575 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+5758 (secure 0 0 0) 0x7fb8e40b0020 con 0x7fb8f40cb550 2023-10-30T13:15:20.426 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:20.427 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:18.200341+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:20.427 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 217 2023-10-30T13:15:20.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.424+0000 7fb8fb501640 1 -- 172.21.15.125:0/2022719575 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8d80664a0 msgr2=0x7fb8d8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.424+0000 7fb8fb501640 1 --2- 172.21.15.125:0/2022719575 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8d80664a0 0x7fb8d8068960 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fb8f000b9d0 tx=0x7fb8f0008040 comp rx=0 tx=0).stop 2023-10-30T13:15:20.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.424+0000 7fb8fb501640 1 -- 172.21.15.125:0/2022719575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8f40cb550 msgr2=0x7fb8f413dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:20.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.424+0000 7fb8fb501640 1 --2- 172.21.15.125:0/2022719575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8f40cb550 0x7fb8f413dd80 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7fb8e40ae040 tx=0x7fb8e4019140 comp rx=0 tx=0).stop 2023-10-30T13:15:20.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.424+0000 7fb8fb501640 1 -- 172.21.15.125:0/2022719575 shutdown_connections 2023-10-30T13:15:20.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.424+0000 7fb8fb501640 1 --2- 172.21.15.125:0/2022719575 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8d80664a0 0x7fb8d8068960 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.424+0000 7fb8fb501640 1 --2- 172.21.15.125:0/2022719575 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb8f414fd50 0x7fb8f413e2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.424+0000 7fb8fb501640 1 --2- 172.21.15.125:0/2022719575 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb8f40cb550 0x7fb8f413dd80 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.424+0000 7fb8fb501640 1 --2- 172.21.15.125:0/2022719575 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb8f4071250 0x7fb8f40c4c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:20.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.424+0000 7fb8fb501640 1 -- 172.21.15.125:0/2022719575 >> 172.21.15.125:0/2022719575 conn(0x7fb8f406c6e0 msgr2=0x7fb8f4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:20.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.424+0000 7fb8fb501640 1 -- 172.21.15.125:0/2022719575 shutdown_connections 2023-10-30T13:15:20.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:20.424+0000 7fb8fb501640 1 -- 172.21.15.125:0/2022719575 wait complete. 2023-10-30T13:15:20.449 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:22.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:22.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d94c09640 1 Processor -- start 2023-10-30T13:15:22.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d94c09640 1 -- start start 2023-10-30T13:15:22.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d94c09640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5d9015d6f0 0x7f5d9015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:22.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d94c09640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5d900cf160 0x7f5d900cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:22.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d94c09640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d900cb800 con 0x7f5d901538a0 2023-10-30T13:15:22.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d94c09640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d900cb330 con 0x7f5d9015d6f0 2023-10-30T13:15:22.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d94c09640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d900cad70 con 0x7f5d900cf160 2023-10-30T13:15:22.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d8dd9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5d9015d6f0 0x7f5d9015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:22.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d8dd9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5d9015d6f0 0x7f5d9015a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40750/0 (socket says 172.21.15.125:40750) 2023-10-30T13:15:22.600 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d8dd9b640 1 -- 172.21.15.125:0/3527857811 learned_addr learned my addr 172.21.15.125:0/3527857811 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:22.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d8e59c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5d901538a0 0x7f5d900ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:55752/0 (socket says 172.21.15.125:55752) 2023-10-30T13:15:22.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d8ed9d640 1 --2- 172.21.15.125:0/3527857811 >> v2:172.21.15.136:3300/0 conn(0x7f5d900cf160 0x7f5d900cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:22.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d8dd9b640 1 -- 172.21.15.125:0/3527857811 >> v2:172.21.15.136:3300/0 conn(0x7f5d900cf160 msgr2=0x7f5d900cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:22.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d8dd9b640 1 --2- 172.21.15.125:0/3527857811 >> v2:172.21.15.136:3300/0 conn(0x7f5d900cf160 0x7f5d900cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:22.601 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d8dd9b640 1 -- 172.21.15.125:0/3527857811 >> v1:172.21.15.125:6789/0 conn(0x7f5d901538a0 legacy=0x7f5d900ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:22.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d8d59a640 1 -- 172.21.15.125:0/3527857811 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611770217 0 0) 0x7f5d900cb800 con 0x7f5d901538a0 2023-10-30T13:15:22.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d8dd9b640 1 -- 172.21.15.125:0/3527857811 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d900cb050 con 0x7f5d9015d6f0 2023-10-30T13:15:22.602 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.595+0000 7f5d8ed9d640 1 --2- 172.21.15.125:0/3527857811 >> v2:172.21.15.136:3300/0 conn(0x7f5d900cf160 0x7f5d900cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:15:22.603 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.599+0000 7f5d8dd9b640 1 --2- 172.21.15.125:0/3527857811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d9015d6f0 0x7f5d9015a090 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f5d900b5330 tx=0x7f5d8401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dad4a1348ff7e7a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:22.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.599+0000 7f5d8d59a640 1 -- 172.21.15.125:0/3527857811 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d840a89f0 con 0x7f5d9015d6f0 2023-10-30T13:15:22.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.599+0000 7f5d8d59a640 1 -- 172.21.15.125:0/3527857811 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d840a8b50 con 0x7f5d9015d6f0 2023-10-30T13:15:22.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.599+0000 7f5d8d59a640 1 -- 172.21.15.125:0/3527857811 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d840b0440 con 0x7f5d9015d6f0 2023-10-30T13:15:22.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.599+0000 7f5d94c09640 1 -- 172.21.15.125:0/3527857811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d9015d6f0 msgr2=0x7f5d9015a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:22.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.599+0000 7f5d94c09640 1 --2- 172.21.15.125:0/3527857811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d9015d6f0 0x7f5d9015a090 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f5d900b5330 tx=0x7f5d8401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:15:22.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.599+0000 7f5d94c09640 1 -- 172.21.15.125:0/3527857811 shutdown_connections 2023-10-30T13:15:22.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.599+0000 7f5d94c09640 1 --2- 172.21.15.125:0/3527857811 >> v2:172.21.15.136:3300/0 conn(0x7f5d900cf160 0x7f5d900cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:22.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.599+0000 7f5d94c09640 1 --2- 172.21.15.125:0/3527857811 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d9015d6f0 0x7f5d9015a090 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:22.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.599+0000 7f5d94c09640 1 -- 172.21.15.125:0/3527857811 >> 172.21.15.125:0/3527857811 conn(0x7f5d9006bfe0 msgr2=0x7f5d9006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:22.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.599+0000 7f5d94c09640 1 -- 172.21.15.125:0/3527857811 shutdown_connections 2023-10-30T13:15:22.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.599+0000 7f5d94c09640 1 -- 172.21.15.125:0/3527857811 wait complete. 2023-10-30T13:15:22.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d94c09640 1 Processor -- start 2023-10-30T13:15:22.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d94c09640 1 -- start start 2023-10-30T13:15:22.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d94c09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d900cf160 0x7f5d900cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:22.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d94c09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d901538a0 0x7f5d900b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:22.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d94c09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d9015d6f0 0x7f5d900b3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:22.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d94c09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d90158a00 con 0x7f5d900cf160 2023-10-30T13:15:22.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d94c09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d90157690 con 0x7f5d901538a0 2023-10-30T13:15:22.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d94c09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d90158c70 con 0x7f5d9015d6f0 2023-10-30T13:15:22.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d8e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d900cf160 0x7f5d900cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:22.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d8e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d900cf160 0x7f5d900cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40756/0 (socket says 172.21.15.125:40756) 2023-10-30T13:15:22.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d8e59c640 1 -- 172.21.15.125:0/1443331650 learned_addr learned my addr 172.21.15.125:0/1443331650 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:22.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d8dd9b640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d901538a0 0x7f5d900b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:22.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d8ed9d640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d9015d6f0 0x7f5d900b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:22.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d8e59c640 1 -- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d9015d6f0 msgr2=0x7f5d900b3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:22.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d8e59c640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d9015d6f0 0x7f5d900b3530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:22.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d8e59c640 1 -- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d901538a0 msgr2=0x7f5d900b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:22.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d8e59c640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d901538a0 0x7f5d900b2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:22.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d8e59c640 1 -- 172.21.15.125:0/1443331650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d840094d0 con 0x7f5d900cf160 2023-10-30T13:15:22.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d8ed9d640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d9015d6f0 0x7f5d900b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:22.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d8e59c640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d900cf160 0x7f5d900cd7a0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f5d800011f0 tx=0x7f5d8000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:22.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d7effd640 1 -- 172.21.15.125:0/1443331650 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d800074e0 con 0x7f5d900cf160 2023-10-30T13:15:22.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d7effd640 1 -- 172.21.15.125:0/1443331650 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d80007640 con 0x7f5d900cf160 2023-10-30T13:15:22.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d7effd640 1 -- 172.21.15.125:0/1443331650 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d80004a10 con 0x7f5d900cf160 2023-10-30T13:15:22.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d94c09640 1 -- 172.21.15.125:0/1443331650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d90157930 con 0x7f5d900cf160 2023-10-30T13:15:22.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d94c09640 1 -- 172.21.15.125:0/1443331650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d900ce230 con 0x7f5d900cf160 2023-10-30T13:15:22.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.603+0000 7f5d94c09640 1 -- 172.21.15.125:0/1443331650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d900b2a50 con 0x7f5d900cf160 2023-10-30T13:15:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.607+0000 7f5d7effd640 1 -- 172.21.15.125:0/1443331650 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5d80004b70 con 0x7f5d900cf160 2023-10-30T13:15:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.607+0000 7f5d7effd640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5d700664a0 0x7f5d70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.607+0000 7f5d7effd640 1 -- 172.21.15.125:0/1443331650 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7f5d800839d0 con 0x7f5d900cf160 2023-10-30T13:15:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.607+0000 7f5d8dd9b640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5d700664a0 0x7f5d70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:22.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.607+0000 7f5d8dd9b640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5d700664a0 0x7f5d70068960 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f5d840b2000 tx=0x7f5d84009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:22.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.611+0000 7f5d7effd640 1 -- 172.21.15.125:0/1443331650 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5d80050600 con 0x7f5d900cf160 2023-10-30T13:15:22.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.795+0000 7f5d94c09640 1 -- 172.21.15.125:0/1443331650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d90157b70 con 0x7f5d900cf160 2023-10-30T13:15:22.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.795+0000 7f5d7effd640 1 -- 172.21.15.125:0/1443331650 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 217 v217) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f5d90157b70 con 0x7f5d900cf160 2023-10-30T13:15:22.798 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:22.798 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":217,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":217,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:18.200341+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:22.798 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 217 2023-10-30T13:15:22.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.795+0000 7f5d94c09640 1 -- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5d700664a0 msgr2=0x7f5d70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:22.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.795+0000 7f5d94c09640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5d700664a0 0x7f5d70068960 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f5d840b2000 tx=0x7f5d84009040 comp rx=0 tx=0).stop 2023-10-30T13:15:22.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.795+0000 7f5d94c09640 1 -- 172.21.15.125:0/1443331650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d900cf160 msgr2=0x7f5d900cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:22.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.795+0000 7f5d94c09640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d900cf160 0x7f5d900cd7a0 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f5d800011f0 tx=0x7f5d8000c9b0 comp rx=0 tx=0).stop 2023-10-30T13:15:22.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.795+0000 7f5d94c09640 1 -- 172.21.15.125:0/1443331650 shutdown_connections 2023-10-30T13:15:22.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.795+0000 7f5d94c09640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5d700664a0 0x7f5d70068960 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:22.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.795+0000 7f5d94c09640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5d9015d6f0 0x7f5d900b3530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:22.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.795+0000 7f5d94c09640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5d901538a0 0x7f5d900b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:22.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.795+0000 7f5d94c09640 1 --2- 172.21.15.125:0/1443331650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5d900cf160 0x7f5d900cd7a0 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:22.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.795+0000 7f5d94c09640 1 -- 172.21.15.125:0/1443331650 >> 172.21.15.125:0/1443331650 conn(0x7f5d9006bfe0 msgr2=0x7f5d900c49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:22.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.795+0000 7f5d94c09640 1 -- 172.21.15.125:0/1443331650 shutdown_connections 2023-10-30T13:15:22.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:22.795+0000 7f5d94c09640 1 -- 172.21.15.125:0/1443331650 wait complete. 2023-10-30T13:15:22.818 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:24.820 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:24.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.963+0000 7f5e413d3640 1 Processor -- start 2023-10-30T13:15:24.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.963+0000 7f5e413d3640 1 -- start start 2023-10-30T13:15:24.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.963+0000 7f5e413d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5e3c153d70 0x7f5e3c06ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:24.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.963+0000 7f5e413d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e3c06e470 0x7f5e3c0cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:24.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.963+0000 7f5e413d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5e3c0c8490 0x7f5e3c0cd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:24.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.963+0000 7f5e413d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3c06d780 con 0x7f5e3c0c8490 2023-10-30T13:15:24.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.963+0000 7f5e413d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3c06afa0 con 0x7f5e3c153d70 2023-10-30T13:15:24.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.963+0000 7f5e413d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c154150 con 0x7f5e3c06e470 2023-10-30T13:15:24.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.963+0000 7f5e3b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5e3c0c8490 0x7f5e3c0cd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:24.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.963+0000 7f5e3b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5e3c0c8490 0x7f5e3c0cd340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40762/0 (socket says 172.21.15.125:40762) 2023-10-30T13:15:24.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.963+0000 7f5e3b7fe640 1 -- 172.21.15.125:0/1870652110 learned_addr learned my addr 172.21.15.125:0/1870652110 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:24.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.963+0000 7f5e327fc640 1 --2- 172.21.15.125:0/1870652110 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e3c06e470 0x7f5e3c0cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:24.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.963+0000 7f5e3affd640 1 --2- 172.21.15.125:0/1870652110 >> v2:172.21.15.136:3300/0 conn(0x7f5e3c153d70 0x7f5e3c06ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:24.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e3b7fe640 1 -- 172.21.15.125:0/1870652110 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e3c06e470 msgr2=0x7f5e3c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:24.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e3b7fe640 1 --2- 172.21.15.125:0/1870652110 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e3c06e470 0x7f5e3c0cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:24.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e3b7fe640 1 -- 172.21.15.125:0/1870652110 >> v2:172.21.15.136:3300/0 conn(0x7f5e3c153d70 msgr2=0x7f5e3c06ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:24.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e3b7fe640 1 --2- 172.21.15.125:0/1870652110 >> v2:172.21.15.136:3300/0 conn(0x7f5e3c153d70 0x7f5e3c06ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:24.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e3b7fe640 1 -- 172.21.15.125:0/1870652110 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c06e030 con 0x7f5e3c0c8490 2023-10-30T13:15:24.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e3affd640 1 --2- 172.21.15.125:0/1870652110 >> v2:172.21.15.136:3300/0 conn(0x7f5e3c153d70 0x7f5e3c06ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:24.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e3b7fe640 1 --2- 172.21.15.125:0/1870652110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e3c0c8490 0x7f5e3c0cd340 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c06d180 tx=0x7f5e3401c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=309fff8dee063e39 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:24.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e3a7fc640 1 -- 172.21.15.125:0/1870652110 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e340aa5f0 con 0x7f5e3c0c8490 2023-10-30T13:15:24.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e3a7fc640 1 -- 172.21.15.125:0/1870652110 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e340aa750 con 0x7f5e3c0c8490 2023-10-30T13:15:24.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e3a7fc640 1 -- 172.21.15.125:0/1870652110 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e340aaae0 con 0x7f5e3c0c8490 2023-10-30T13:15:24.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e413d3640 1 -- 172.21.15.125:0/1870652110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e3c0c8490 msgr2=0x7f5e3c0cd340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:24.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e413d3640 1 --2- 172.21.15.125:0/1870652110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e3c0c8490 0x7f5e3c0cd340 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f5e3c06d180 tx=0x7f5e3401c4d0 comp rx=0 tx=0).stop 2023-10-30T13:15:24.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e413d3640 1 -- 172.21.15.125:0/1870652110 shutdown_connections 2023-10-30T13:15:24.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e413d3640 1 --2- 172.21.15.125:0/1870652110 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e3c0c8490 0x7f5e3c0cd340 unknown :-1 s=CLOSED pgs=1854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:24.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e413d3640 1 --2- 172.21.15.125:0/1870652110 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e3c06e470 0x7f5e3c0cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:24.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e413d3640 1 --2- 172.21.15.125:0/1870652110 >> v2:172.21.15.136:3300/0 conn(0x7f5e3c153d70 0x7f5e3c06ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:24.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e413d3640 1 -- 172.21.15.125:0/1870652110 >> 172.21.15.125:0/1870652110 conn(0x7f5e3c071ea0 msgr2=0x7f5e3c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:24.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.967+0000 7f5e413d3640 1 -- 172.21.15.125:0/1870652110 shutdown_connections 2023-10-30T13:15:24.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e413d3640 1 -- 172.21.15.125:0/1870652110 wait complete. 2023-10-30T13:15:24.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e413d3640 1 Processor -- start 2023-10-30T13:15:24.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e413d3640 1 -- start start 2023-10-30T13:15:24.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e413d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e3c06e470 0x7f5e3c1f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:24.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e413d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e3c0c8490 0x7f5e3c0ad8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:24.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e413d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e3c153d70 0x7f5e3c145240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:24.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e413d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c0aeb70 con 0x7f5e3c153d70 2023-10-30T13:15:24.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e413d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c14b240 con 0x7f5e3c06e470 2023-10-30T13:15:24.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e413d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c149930 con 0x7f5e3c0c8490 2023-10-30T13:15:24.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e3b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e3c153d70 0x7f5e3c145240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:24.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e3b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e3c153d70 0x7f5e3c145240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40774/0 (socket says 172.21.15.125:40774) 2023-10-30T13:15:24.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e3b7fe640 1 -- 172.21.15.125:0/4251395278 learned_addr learned my addr 172.21.15.125:0/4251395278 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:24.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e327fc640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e3c0c8490 0x7f5e3c0ad8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:24.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e3affd640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e3c06e470 0x7f5e3c1f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:24.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e3b7fe640 1 -- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e3c0c8490 msgr2=0x7f5e3c0ad8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:24.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e3b7fe640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e3c0c8490 0x7f5e3c0ad8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:24.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e3b7fe640 1 -- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e3c06e470 msgr2=0x7f5e3c1f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:24.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e3b7fe640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e3c06e470 0x7f5e3c1f2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:24.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e3b7fe640 1 -- 172.21.15.125:0/4251395278 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e340094d0 con 0x7f5e3c153d70 2023-10-30T13:15:24.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.971+0000 7f5e3affd640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e3c06e470 0x7f5e3c1f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:24.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.975+0000 7f5e3b7fe640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e3c153d70 0x7f5e3c145240 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f5e3401c300 tx=0x7f5e3401ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:24.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.975+0000 7f5e33fff640 1 -- 172.21.15.125:0/4251395278 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e340bd5f0 con 0x7f5e3c153d70 2023-10-30T13:15:24.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.975+0000 7f5e33fff640 1 -- 172.21.15.125:0/4251395278 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e340bd750 con 0x7f5e3c153d70 2023-10-30T13:15:24.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.975+0000 7f5e33fff640 1 -- 172.21.15.125:0/4251395278 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e340bdac0 con 0x7f5e3c153d70 2023-10-30T13:15:24.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.975+0000 7f5e413d3640 1 -- 172.21.15.125:0/4251395278 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c1f3300 con 0x7f5e3c153d70 2023-10-30T13:15:24.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.975+0000 7f5e413d3640 1 -- 172.21.15.125:0/4251395278 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c0af770 con 0x7f5e3c153d70 2023-10-30T13:15:24.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.975+0000 7f5e413d3640 1 -- 172.21.15.125:0/4251395278 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c15ce50 con 0x7f5e3c153d70 2023-10-30T13:15:24.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.975+0000 7f5e33fff640 1 -- 172.21.15.125:0/4251395278 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5e340b33f0 con 0x7f5e3c153d70 2023-10-30T13:15:24.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.975+0000 7f5e33fff640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e1c0664a0 0x7f5e1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:24.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.975+0000 7f5e33fff640 1 -- 172.21.15.125:0/4251395278 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7f5e340a7070 con 0x7f5e3c153d70 2023-10-30T13:15:24.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.975+0000 7f5e3affd640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e1c0664a0 0x7f5e1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:24.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.979+0000 7f5e3affd640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e1c0664a0 0x7f5e1c068960 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c007950 tx=0x7f5e2c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:24.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:24.979+0000 7f5e33fff640 1 -- 172.21.15.125:0/4251395278 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5e340ebf10 con 0x7f5e3c153d70 2023-10-30T13:15:25.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:25.167+0000 7f5e413d3640 1 -- 172.21.15.125:0/4251395278 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e3c04e170 con 0x7f5e3c153d70 2023-10-30T13:15:25.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:25.167+0000 7f5e33fff640 1 -- 172.21.15.125:0/4251395278 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f5e340c3020 con 0x7f5e3c153d70 2023-10-30T13:15:25.171 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:25.171 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":218,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:24.197330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:25.171 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 218 2023-10-30T13:15:25.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:25.167+0000 7f5e413d3640 1 -- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e1c0664a0 msgr2=0x7f5e1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:25.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:25.167+0000 7f5e413d3640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e1c0664a0 0x7f5e1c068960 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c007950 tx=0x7f5e2c008040 comp rx=0 tx=0).stop 2023-10-30T13:15:25.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:25.167+0000 7f5e413d3640 1 -- 172.21.15.125:0/4251395278 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e3c153d70 msgr2=0x7f5e3c145240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:25.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:25.167+0000 7f5e413d3640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e3c153d70 0x7f5e3c145240 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f5e3401c300 tx=0x7f5e3401ad10 comp rx=0 tx=0).stop 2023-10-30T13:15:25.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:25.167+0000 7f5e413d3640 1 -- 172.21.15.125:0/4251395278 shutdown_connections 2023-10-30T13:15:25.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:25.167+0000 7f5e413d3640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e1c0664a0 0x7f5e1c068960 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:25.167+0000 7f5e413d3640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e3c153d70 0x7f5e3c145240 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:25.167+0000 7f5e413d3640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e3c0c8490 0x7f5e3c0ad8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:25.167+0000 7f5e413d3640 1 --2- 172.21.15.125:0/4251395278 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e3c06e470 0x7f5e3c1f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:25.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:25.167+0000 7f5e413d3640 1 -- 172.21.15.125:0/4251395278 >> 172.21.15.125:0/4251395278 conn(0x7f5e3c071ea0 msgr2=0x7f5e3c14fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:25.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:25.171+0000 7f5e413d3640 1 -- 172.21.15.125:0/4251395278 shutdown_connections 2023-10-30T13:15:25.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:25.171+0000 7f5e413d3640 1 -- 172.21.15.125:0/4251395278 wait complete. 2023-10-30T13:15:25.192 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:27.193 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:27.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.335+0000 7fe05cc46640 1 Processor -- start 2023-10-30T13:15:27.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.335+0000 7fe05cc46640 1 -- start start 2023-10-30T13:15:27.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.339+0000 7fe05cc46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe058153bf0 0x7fe058071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:27.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.339+0000 7fe05cc46640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe0580c5f60 0x7fe0580cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:27.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.339+0000 7fe05cc46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0580700b0 con 0x7fe0580c6fe0 2023-10-30T13:15:27.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.339+0000 7fe05cc46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0580c6810 con 0x7fe058153bf0 2023-10-30T13:15:27.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.339+0000 7fe05cc46640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe058159900 con 0x7fe0580c5f60 2023-10-30T13:15:27.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.339+0000 7fe05659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe058153bf0 0x7fe058071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:27.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.339+0000 7fe05659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fe058153bf0 0x7fe058071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40786/0 (socket says 172.21.15.125:40786) 2023-10-30T13:15:27.343 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.339+0000 7fe05659c640 1 -- 172.21.15.125:0/2991674152 learned_addr learned my addr 172.21.15.125:0/2991674152 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:27.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.339+0000 7fe056d9d640 1 --2- 172.21.15.125:0/2991674152 >> v2:172.21.15.136:3300/0 conn(0x7fe0580c5f60 0x7fe0580cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:27.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.339+0000 7fe05659c640 1 -- 172.21.15.125:0/2991674152 >> v2:172.21.15.136:3300/0 conn(0x7fe0580c5f60 msgr2=0x7fe0580cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:27.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.339+0000 7fe05659c640 1 --2- 172.21.15.125:0/2991674152 >> v2:172.21.15.136:3300/0 conn(0x7fe0580c5f60 0x7fe0580cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.339+0000 7fe05659c640 1 -- 172.21.15.125:0/2991674152 >> v1:172.21.15.125:6789/0 conn(0x7fe0580c6fe0 legacy=0x7fe0580c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:27.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.339+0000 7fe05659c640 1 -- 172.21.15.125:0/2991674152 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0580c6530 con 0x7fe058153bf0 2023-10-30T13:15:27.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.339+0000 7fe055d9b640 1 -- 172.21.15.125:0/2991674152 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881362196 0 0) 0x7fe0580700b0 con 0x7fe0580c6fe0 2023-10-30T13:15:27.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.343+0000 7fe05659c640 1 --2- 172.21.15.125:0/2991674152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe058153bf0 0x7fe058071260 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fe05806ce90 tx=0x7fe04801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f4de2f807aa3e1a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:27.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.343+0000 7fe055d9b640 1 -- 172.21.15.125:0/2991674152 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0480a89f0 con 0x7fe058153bf0 2023-10-30T13:15:27.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.343+0000 7fe055d9b640 1 -- 172.21.15.125:0/2991674152 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0480a8b50 con 0x7fe058153bf0 2023-10-30T13:15:27.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.343+0000 7fe055d9b640 1 -- 172.21.15.125:0/2991674152 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0480b0440 con 0x7fe058153bf0 2023-10-30T13:15:27.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.343+0000 7fe05cc46640 1 -- 172.21.15.125:0/2991674152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe058153bf0 msgr2=0x7fe058071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:27.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.343+0000 7fe05cc46640 1 --2- 172.21.15.125:0/2991674152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe058153bf0 0x7fe058071260 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fe05806ce90 tx=0x7fe04801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.343+0000 7fe05cc46640 1 -- 172.21.15.125:0/2991674152 shutdown_connections 2023-10-30T13:15:27.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.343+0000 7fe05cc46640 1 --2- 172.21.15.125:0/2991674152 >> v2:172.21.15.136:3300/0 conn(0x7fe0580c5f60 0x7fe0580cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.343+0000 7fe05cc46640 1 --2- 172.21.15.125:0/2991674152 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe058153bf0 0x7fe058071260 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.343+0000 7fe05cc46640 1 -- 172.21.15.125:0/2991674152 >> 172.21.15.125:0/2991674152 conn(0x7fe05806c6e0 msgr2=0x7fe058069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:27.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.343+0000 7fe05cc46640 1 -- 172.21.15.125:0/2991674152 shutdown_connections 2023-10-30T13:15:27.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.343+0000 7fe05cc46640 1 -- 172.21.15.125:0/2991674152 wait complete. 2023-10-30T13:15:27.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.343+0000 7fe05cc46640 1 Processor -- start 2023-10-30T13:15:27.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.343+0000 7fe05cc46640 1 -- start start 2023-10-30T13:15:27.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05cc46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0580c5f60 0x7fe058152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:27.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05cc46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0580c6fe0 0x7fe05814f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:27.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05cc46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe058153bf0 0x7fe05814fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:27.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05cc46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe058151470 con 0x7fe0580c5f60 2023-10-30T13:15:27.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05cc46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0580c5240 con 0x7fe0580c6fe0 2023-10-30T13:15:27.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05cc46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0580c5520 con 0x7fe058153bf0 2023-10-30T13:15:27.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0580c5f60 0x7fe058152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:27.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0580c5f60 0x7fe058152570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40796/0 (socket says 172.21.15.125:40796) 2023-10-30T13:15:27.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05659c640 1 -- 172.21.15.125:0/3177567636 learned_addr learned my addr 172.21.15.125:0/3177567636 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:27.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe04dd9b640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0580c6fe0 0x7fe05814f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:27.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe056d9d640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe058153bf0 0x7fe05814fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:27.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05659c640 1 -- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe058153bf0 msgr2=0x7fe05814fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:27.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05659c640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe058153bf0 0x7fe05814fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05659c640 1 -- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0580c6fe0 msgr2=0x7fe05814f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:27.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05659c640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0580c6fe0 0x7fe05814f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05659c640 1 -- 172.21.15.125:0/3177567636 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0480094d0 con 0x7fe0580c5f60 2023-10-30T13:15:27.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe056d9d640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe058153bf0 0x7fe05814fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:15:27.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05659c640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0580c5f60 0x7fe058152570 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fe04801b900 tx=0x7fe04801d530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:27.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe04f7fe640 1 -- 172.21.15.125:0/3177567636 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0480a89f0 con 0x7fe0580c5f60 2023-10-30T13:15:27.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe04f7fe640 1 -- 172.21.15.125:0/3177567636 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe048004030 con 0x7fe0580c5f60 2023-10-30T13:15:27.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe04f7fe640 1 -- 172.21.15.125:0/3177567636 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0480043a0 con 0x7fe0580c5f60 2023-10-30T13:15:27.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05cc46640 1 -- 172.21.15.125:0/3177567636 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0580c5800 con 0x7fe0580c5f60 2023-10-30T13:15:27.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05cc46640 1 -- 172.21.15.125:0/3177567636 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05814f330 con 0x7fe0580c5f60 2023-10-30T13:15:27.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe05cc46640 1 -- 172.21.15.125:0/3177567636 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe05814ee90 con 0x7fe0580c5f60 2023-10-30T13:15:27.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.347+0000 7fe04f7fe640 1 -- 172.21.15.125:0/3177567636 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe0480b7040 con 0x7fe0580c5f60 2023-10-30T13:15:27.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.351+0000 7fe04f7fe640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0380664a0 0x7fe038068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:27.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.351+0000 7fe04f7fe640 1 -- 172.21.15.125:0/3177567636 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7fe048121b10 con 0x7fe0580c5f60 2023-10-30T13:15:27.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.351+0000 7fe04dd9b640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0380664a0 0x7fe038068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:27.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.351+0000 7fe04dd9b640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0380664a0 0x7fe038068960 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fe058151690 tx=0x7fe040009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:27.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.355+0000 7fe04f7fe640 1 -- 172.21.15.125:0/3177567636 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe0480ee580 con 0x7fe0580c5f60 2023-10-30T13:15:27.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.531+0000 7fe05cc46640 1 -- 172.21.15.125:0/3177567636 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe05815a0a0 con 0x7fe0580c5f60 2023-10-30T13:15:27.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.531+0000 7fe04f7fe640 1 -- 172.21.15.125:0/3177567636 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 218 v218) v1 ==== 95+0+5755 (secure 0 0 0) 0x7fe0480c2020 con 0x7fe0580c5f60 2023-10-30T13:15:27.537 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:27.537 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":218,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":218,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:24.197330+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:rejoin","state_seq":34,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:27.537 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 218 2023-10-30T13:15:27.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.535+0000 7fe05cc46640 1 -- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0380664a0 msgr2=0x7fe038068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:27.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.535+0000 7fe05cc46640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0380664a0 0x7fe038068960 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fe058151690 tx=0x7fe040009040 comp rx=0 tx=0).stop 2023-10-30T13:15:27.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.535+0000 7fe05cc46640 1 -- 172.21.15.125:0/3177567636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0580c5f60 msgr2=0x7fe058152570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:27.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.535+0000 7fe05cc46640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0580c5f60 0x7fe058152570 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fe04801b900 tx=0x7fe04801d530 comp rx=0 tx=0).stop 2023-10-30T13:15:27.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.535+0000 7fe05cc46640 1 -- 172.21.15.125:0/3177567636 shutdown_connections 2023-10-30T13:15:27.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.535+0000 7fe05cc46640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe0380664a0 0x7fe038068960 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.535+0000 7fe05cc46640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe058153bf0 0x7fe05814fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.535+0000 7fe05cc46640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0580c6fe0 0x7fe05814f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.535+0000 7fe05cc46640 1 --2- 172.21.15.125:0/3177567636 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe0580c5f60 0x7fe058152570 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:27.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.535+0000 7fe05cc46640 1 -- 172.21.15.125:0/3177567636 >> 172.21.15.125:0/3177567636 conn(0x7fe05806c6e0 msgr2=0x7fe058158a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:27.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.535+0000 7fe05cc46640 1 -- 172.21.15.125:0/3177567636 shutdown_connections 2023-10-30T13:15:27.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:27.535+0000 7fe05cc46640 1 -- 172.21.15.125:0/3177567636 wait complete. 2023-10-30T13:15:27.558 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:15:29.560 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:29.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.703+0000 7fa1c249e640 1 Processor -- start 2023-10-30T13:15:29.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.703+0000 7fa1c249e640 1 -- start start 2023-10-30T13:15:29.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.703+0000 7fa1c249e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa1bc0c6bd0 0x7fa1bc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:29.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.703+0000 7fa1c249e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa1bc071110 0x7fa1bc158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:29.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.703+0000 7fa1c249e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1bc066cf0 con 0x7fa1bc06f7a0 2023-10-30T13:15:29.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.703+0000 7fa1c249e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1bc0700c0 con 0x7fa1bc0c6bd0 2023-10-30T13:15:29.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.703+0000 7fa1c249e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1bc0c70d0 con 0x7fa1bc071110 2023-10-30T13:15:29.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.707+0000 7fa1bb7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa1bc0c6bd0 0x7fa1bc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:29.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.707+0000 7fa1bb7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa1bc0c6bd0 0x7fa1bc06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40808/0 (socket says 172.21.15.125:40808) 2023-10-30T13:15:29.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.707+0000 7fa1bb7fe640 1 -- 172.21.15.125:0/578120245 learned_addr learned my addr 172.21.15.125:0/578120245 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:29.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.707+0000 7fa1c0a3b640 1 --2- 172.21.15.125:0/578120245 >> v2:172.21.15.136:3300/0 conn(0x7fa1bc071110 0x7fa1bc158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:29.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.707+0000 7fa1bb7fe640 1 -- 172.21.15.125:0/578120245 >> v2:172.21.15.136:3300/0 conn(0x7fa1bc071110 msgr2=0x7fa1bc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:29.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.707+0000 7fa1bb7fe640 1 --2- 172.21.15.125:0/578120245 >> v2:172.21.15.136:3300/0 conn(0x7fa1bc071110 0x7fa1bc158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:29.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.707+0000 7fa1bb7fe640 1 -- 172.21.15.125:0/578120245 >> v1:172.21.15.136:6789/0 conn(0x7fa1bc06f7a0 legacy=0x7fa1bc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:29.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.707+0000 7fa1bb7fe640 1 -- 172.21.15.125:0/578120245 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc0c7390 con 0x7fa1bc0c6bd0 2023-10-30T13:15:29.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.707+0000 7fa1bb7fe640 1 --2- 172.21.15.125:0/578120245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1bc0c6bd0 0x7fa1bc06fb80 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc06cdd0 tx=0x7fa1b001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6d45b1f841f1e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:29.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.707+0000 7fa1baffd640 1 -- 172.21.15.125:0/578120245 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b00a89f0 con 0x7fa1bc0c6bd0 2023-10-30T13:15:29.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.707+0000 7fa1baffd640 1 -- 172.21.15.125:0/578120245 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b00a8b50 con 0x7fa1bc0c6bd0 2023-10-30T13:15:29.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.707+0000 7fa1baffd640 1 -- 172.21.15.125:0/578120245 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b00b0440 con 0x7fa1bc0c6bd0 2023-10-30T13:15:29.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 -- 172.21.15.125:0/578120245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1bc0c6bd0 msgr2=0x7fa1bc06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:29.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 --2- 172.21.15.125:0/578120245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1bc0c6bd0 0x7fa1bc06fb80 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc06cdd0 tx=0x7fa1b001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:15:29.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 -- 172.21.15.125:0/578120245 shutdown_connections 2023-10-30T13:15:29.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 --2- 172.21.15.125:0/578120245 >> v2:172.21.15.136:3300/0 conn(0x7fa1bc071110 0x7fa1bc158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:29.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 --2- 172.21.15.125:0/578120245 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1bc0c6bd0 0x7fa1bc06fb80 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:29.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 -- 172.21.15.125:0/578120245 >> 172.21.15.125:0/578120245 conn(0x7fa1bc0c7910 msgr2=0x7fa1bc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:29.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 -- 172.21.15.125:0/578120245 shutdown_connections 2023-10-30T13:15:29.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 -- 172.21.15.125:0/578120245 wait complete. 2023-10-30T13:15:29.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 Processor -- start 2023-10-30T13:15:29.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 -- start start 2023-10-30T13:15:29.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1bc06f7a0 0x7fa1bc0c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:29.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1bc071110 0x7fa1bc1545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:29.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1bc0c6bd0 0x7fa1bc1eeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:29.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc1eea20 con 0x7fa1bc071110 2023-10-30T13:15:29.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc1e2f20 con 0x7fa1bc06f7a0 2023-10-30T13:15:29.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c249e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1bc1f2200 con 0x7fa1bc0c6bd0 2023-10-30T13:15:29.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1bb7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1bc071110 0x7fa1bc1545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:29.719 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1bb7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1bc071110 0x7fa1bc1545e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40818/0 (socket says 172.21.15.125:40818) 2023-10-30T13:15:29.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1bb7fe640 1 -- 172.21.15.125:0/885240092 learned_addr learned my addr 172.21.15.125:0/885240092 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:29.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.711+0000 7fa1c0a3b640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1bc0c6bd0 0x7fa1bc1eeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:29.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1bbfff640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1bc06f7a0 0x7fa1bc0c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:29.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1bb7fe640 1 -- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1bc0c6bd0 msgr2=0x7fa1bc1eeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:29.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1bb7fe640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1bc0c6bd0 0x7fa1bc1eeeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:29.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1bb7fe640 1 -- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1bc06f7a0 msgr2=0x7fa1bc0c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:29.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1bb7fe640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1bc06f7a0 0x7fa1bc0c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:29.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1bb7fe640 1 -- 172.21.15.125:0/885240092 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b00094d0 con 0x7fa1bc071110 2023-10-30T13:15:29.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1bbfff640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1bc06f7a0 0x7fa1bc0c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:29.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1bb7fe640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1bc071110 0x7fa1bc1545e0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fa1b00b3040 tx=0x7fa1b001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:29.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1b8ff9640 1 -- 172.21.15.125:0/885240092 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b00b9620 con 0x7fa1bc071110 2023-10-30T13:15:29.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1b8ff9640 1 -- 172.21.15.125:0/885240092 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b00b9780 con 0x7fa1bc071110 2023-10-30T13:15:29.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1b8ff9640 1 -- 172.21.15.125:0/885240092 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b00b9b30 con 0x7fa1bc071110 2023-10-30T13:15:29.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1c249e640 1 -- 172.21.15.125:0/885240092 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1bc1f24e0 con 0x7fa1bc071110 2023-10-30T13:15:29.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1c249e640 1 -- 172.21.15.125:0/885240092 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1bc1f2aa0 con 0x7fa1bc071110 2023-10-30T13:15:29.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1c249e640 1 -- 172.21.15.125:0/885240092 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1bc1eeb90 con 0x7fa1bc071110 2023-10-30T13:15:29.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1b8ff9640 1 -- 172.21.15.125:0/885240092 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa1b00b7070 con 0x7fa1bc071110 2023-10-30T13:15:29.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.715+0000 7fa1b8ff9640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1a80664a0 0x7fa1a8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:29.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.719+0000 7fa1b8ff9640 1 -- 172.21.15.125:0/885240092 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7fa1b0122740 con 0x7fa1bc071110 2023-10-30T13:15:29.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.719+0000 7fa1bbfff640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1a80664a0 0x7fa1a8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:29.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.719+0000 7fa1bbfff640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1a80664a0 0x7fa1a8068960 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac000f70 tx=0x7fa1ac009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:29.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.719+0000 7fa1b8ff9640 1 -- 172.21.15.125:0/885240092 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa1b00ef1b0 con 0x7fa1bc071110 2023-10-30T13:15:29.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.907+0000 7fa1c249e640 1 -- 172.21.15.125:0/885240092 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1bc04e170 con 0x7fa1bc071110 2023-10-30T13:15:29.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.911+0000 7fa1b8ff9640 1 -- 172.21.15.125:0/885240092 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) v1 ==== 95+0+5755 (secure 0 0 0) 0x7fa1b00b5070 con 0x7fa1bc071110 2023-10-30T13:15:29.914 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:29.914 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:29.286440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:29.915 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 219 2023-10-30T13:15:29.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.911+0000 7fa1c249e640 1 -- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1a80664a0 msgr2=0x7fa1a8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:29.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.911+0000 7fa1c249e640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1a80664a0 0x7fa1a8068960 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac000f70 tx=0x7fa1ac009040 comp rx=0 tx=0).stop 2023-10-30T13:15:29.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.911+0000 7fa1c249e640 1 -- 172.21.15.125:0/885240092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1bc071110 msgr2=0x7fa1bc1545e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:29.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.911+0000 7fa1c249e640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1bc071110 0x7fa1bc1545e0 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7fa1b00b3040 tx=0x7fa1b001ba90 comp rx=0 tx=0).stop 2023-10-30T13:15:29.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.911+0000 7fa1c249e640 1 -- 172.21.15.125:0/885240092 shutdown_connections 2023-10-30T13:15:29.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.911+0000 7fa1c249e640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa1a80664a0 0x7fa1a8068960 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:29.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.911+0000 7fa1c249e640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1bc0c6bd0 0x7fa1bc1eeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:29.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.911+0000 7fa1c249e640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1bc071110 0x7fa1bc1545e0 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:29.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.911+0000 7fa1c249e640 1 --2- 172.21.15.125:0/885240092 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1bc06f7a0 0x7fa1bc0c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:29.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.911+0000 7fa1c249e640 1 -- 172.21.15.125:0/885240092 >> 172.21.15.125:0/885240092 conn(0x7fa1bc0c7910 msgr2=0x7fa1bc0c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:29.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.911+0000 7fa1c249e640 1 -- 172.21.15.125:0/885240092 shutdown_connections 2023-10-30T13:15:29.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:29.911+0000 7fa1c249e640 1 -- 172.21.15.125:0/885240092 wait complete. 2023-10-30T13:15:29.934 INFO:tasks.mds_thrash.fs.[cephfs]:mds.c has gained rank=1, replacing gid=7511 2023-10-30T13:15:29.935 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.d 2023-10-30T13:15:32.684 INFO:tasks.ceph.mds.e.smithi125.stderr:2023-10-30T13:15:32.679+0000 7f510d857640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:15:35.886 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:15:35.887 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.31 2023-10-30T13:15:35.887 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 31 -eq 48 ] 2023-10-30T13:15:35.887 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.31 2023-10-30T13:15:35.887 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.31 2023-10-30T13:15:35.888 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.31 2023-10-30T13:15:35.888 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:15:36.935 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T13:15:36.936 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:15:36.936 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:15:36.939 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:15:36.939 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:37.016 INFO:tasks.ceph.mds.d.smithi136.stdout:starting mds.d at 2023-10-30T13:15:37.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b953e640 1 Processor -- start 2023-10-30T13:15:37.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b953e640 1 -- start start 2023-10-30T13:15:37.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b953e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f38b415d6f0 0x7f38b415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:37.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b953e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38b40cf160 0x7f38b40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:37.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b953e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b40cb800 con 0x7f38b41538a0 2023-10-30T13:15:37.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b953e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38b40cb330 con 0x7f38b415d6f0 2023-10-30T13:15:37.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b953e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b40cad70 con 0x7f38b40cf160 2023-10-30T13:15:37.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b2ffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f38b41538a0 0x7f38b40ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36884/0 (socket says 172.21.15.125:36884) 2023-10-30T13:15:37.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b2ffd640 1 -- 172.21.15.125:0/2449954566 learned_addr learned my addr 172.21.15.125:0/2449954566 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:15:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b27fc640 1 --2- 172.21.15.125:0/2449954566 >> v2:172.21.15.136:3300/0 conn(0x7f38b415d6f0 0x7f38b415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b37fe640 1 --2- 172.21.15.125:0/2449954566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38b40cf160 0x7f38b40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:37.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b1ffb640 1 -- 172.21.15.125:0/2449954566 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3538843656 0 0) 0x7f38b40cb800 con 0x7f38b41538a0 2023-10-30T13:15:37.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b1ffb640 1 -- 172.21.15.125:0/2449954566 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f389c003530 con 0x7f38b41538a0 2023-10-30T13:15:37.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b37fe640 1 -- 172.21.15.125:0/2449954566 >> v2:172.21.15.136:3300/0 conn(0x7f38b415d6f0 msgr2=0x7f38b415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:37.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b37fe640 1 --2- 172.21.15.125:0/2449954566 >> v2:172.21.15.136:3300/0 conn(0x7f38b415d6f0 0x7f38b415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b37fe640 1 -- 172.21.15.125:0/2449954566 >> v1:172.21.15.125:6789/0 conn(0x7f38b41538a0 legacy=0x7f38b40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:37.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.087+0000 7f38b37fe640 1 -- 172.21.15.125:0/2449954566 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b40cb050 con 0x7f38b40cf160 2023-10-30T13:15:37.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.091+0000 7f38b37fe640 1 --2- 172.21.15.125:0/2449954566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38b40cf160 0x7f38b40cf590 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009d90 tx=0x7f38ac01f130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=995908c1e33a36eb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:37.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.091+0000 7f38b1ffb640 1 -- 172.21.15.125:0/2449954566 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac0af600 con 0x7f38b40cf160 2023-10-30T13:15:37.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.091+0000 7f38b1ffb640 1 -- 172.21.15.125:0/2449954566 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38ac0ab050 con 0x7f38b40cf160 2023-10-30T13:15:37.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.091+0000 7f38b1ffb640 1 -- 172.21.15.125:0/2449954566 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38ac0af980 con 0x7f38b40cf160 2023-10-30T13:15:37.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.091+0000 7f38b953e640 1 -- 172.21.15.125:0/2449954566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38b40cf160 msgr2=0x7f38b40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:37.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.091+0000 7f38b953e640 1 --2- 172.21.15.125:0/2449954566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38b40cf160 0x7f38b40cf590 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009d90 tx=0x7f38ac01f130 comp rx=0 tx=0).stop 2023-10-30T13:15:37.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.091+0000 7f38b953e640 1 -- 172.21.15.125:0/2449954566 shutdown_connections 2023-10-30T13:15:37.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.091+0000 7f38b953e640 1 --2- 172.21.15.125:0/2449954566 >> v2:172.21.15.136:3300/0 conn(0x7f38b415d6f0 0x7f38b415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.091+0000 7f38b953e640 1 --2- 172.21.15.125:0/2449954566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38b40cf160 0x7f38b40cf590 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.091+0000 7f38b953e640 1 -- 172.21.15.125:0/2449954566 >> 172.21.15.125:0/2449954566 conn(0x7f38b406bfe0 msgr2=0x7f38b406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:37.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.091+0000 7f38b953e640 1 -- 172.21.15.125:0/2449954566 shutdown_connections 2023-10-30T13:15:37.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.091+0000 7f38b953e640 1 -- 172.21.15.125:0/2449954566 wait complete. 2023-10-30T13:15:37.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b953e640 1 Processor -- start 2023-10-30T13:15:37.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b953e640 1 -- start start 2023-10-30T13:15:37.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b953e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38b40cf160 0x7f38b4152220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:37.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b953e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b41538a0 0x7f38b4157850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:37.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b953e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38b415d6f0 0x7f38b4157d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:37.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b953e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b41efdd0 con 0x7f38b41538a0 2023-10-30T13:15:37.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b953e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b41ef1e0 con 0x7f38b40cf160 2023-10-30T13:15:37.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b953e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b41f34a0 con 0x7f38b415d6f0 2023-10-30T13:15:37.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b27fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b41538a0 0x7f38b4157850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:37.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b27fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b41538a0 0x7f38b4157850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33390/0 (socket says 172.21.15.125:33390) 2023-10-30T13:15:37.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b27fc640 1 -- 172.21.15.125:0/1346790691 learned_addr learned my addr 172.21.15.125:0/1346790691 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:37.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b2ffd640 1 --2- 172.21.15.125:0/1346790691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38b40cf160 0x7f38b4152220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:37.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b37fe640 1 --2- 172.21.15.125:0/1346790691 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38b415d6f0 0x7f38b4157d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:37.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b27fc640 1 -- 172.21.15.125:0/1346790691 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38b415d6f0 msgr2=0x7f38b4157d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:37.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b27fc640 1 --2- 172.21.15.125:0/1346790691 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38b415d6f0 0x7f38b4157d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b27fc640 1 -- 172.21.15.125:0/1346790691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38b40cf160 msgr2=0x7f38b4152220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:37.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b27fc640 1 --2- 172.21.15.125:0/1346790691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38b40cf160 0x7f38b4152220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b27fc640 1 -- 172.21.15.125:0/1346790691 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38ac009a70 con 0x7f38b41538a0 2023-10-30T13:15:37.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b27fc640 1 --2- 172.21.15.125:0/1346790691 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b41538a0 0x7f38b4157850 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e870 tx=0x7f38a800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:37.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f389b7fe640 1 -- 172.21.15.125:0/1346790691 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a8009510 con 0x7f38b41538a0 2023-10-30T13:15:37.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f389b7fe640 1 -- 172.21.15.125:0/1346790691 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a800eec0 con 0x7f38b41538a0 2023-10-30T13:15:37.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b953e640 1 -- 172.21.15.125:0/1346790691 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b41ef4a0 con 0x7f38b41538a0 2023-10-30T13:15:37.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f389b7fe640 1 -- 172.21.15.125:0/1346790691 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a800cb40 con 0x7f38b41538a0 2023-10-30T13:15:37.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.095+0000 7f38b953e640 1 -- 172.21.15.125:0/1346790691 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b41efa60 con 0x7f38b41538a0 2023-10-30T13:15:37.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.099+0000 7f38b953e640 1 -- 172.21.15.125:0/1346790691 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b40b2a50 con 0x7f38b41538a0 2023-10-30T13:15:37.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.099+0000 7f389b7fe640 1 -- 172.21.15.125:0/1346790691 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f38a800cca0 con 0x7f38b41538a0 2023-10-30T13:15:37.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.099+0000 7f389b7fe640 1 --2- 172.21.15.125:0/1346790691 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38a00664a0 0x7f38a0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:37.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.099+0000 7f389b7fe640 1 -- 172.21.15.125:0/1346790691 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7f38a8084880 con 0x7f38b41538a0 2023-10-30T13:15:37.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.099+0000 7f38b2ffd640 1 --2- 172.21.15.125:0/1346790691 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38a00664a0 0x7f38a0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:37.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.103+0000 7f38b2ffd640 1 --2- 172.21.15.125:0/1346790691 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38a00664a0 0x7f38a0068960 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f38a40011f0 tx=0x7f38a4007d40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:37.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.107+0000 7f389b7fe640 1 -- 172.21.15.125:0/1346790691 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f38a8051430 con 0x7f38b41538a0 2023-10-30T13:15:37.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.323+0000 7f38b953e640 1 -- 172.21.15.125:0/1346790691 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38b404e170 con 0x7f38b41538a0 2023-10-30T13:15:37.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.323+0000 7f389b7fe640 1 -- 172.21.15.125:0/1346790691 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 219 v219) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f38a8054a50 con 0x7f38b41538a0 2023-10-30T13:15:37.327 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:37.328 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":219,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:29.286440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:37.328 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 219 2023-10-30T13:15:37.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.327+0000 7f38b953e640 1 -- 172.21.15.125:0/1346790691 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38a00664a0 msgr2=0x7f38a0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:37.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.327+0000 7f38b953e640 1 --2- 172.21.15.125:0/1346790691 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38a00664a0 0x7f38a0068960 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f38a40011f0 tx=0x7f38a4007d40 comp rx=0 tx=0).stop 2023-10-30T13:15:37.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.327+0000 7f38b953e640 1 -- 172.21.15.125:0/1346790691 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b41538a0 msgr2=0x7f38b4157850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:37.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.327+0000 7f38b953e640 1 --2- 172.21.15.125:0/1346790691 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b41538a0 0x7f38b4157850 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f38a800e870 tx=0x7f38a800ec70 comp rx=0 tx=0).stop 2023-10-30T13:15:37.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.327+0000 7f38b953e640 1 -- 172.21.15.125:0/1346790691 shutdown_connections 2023-10-30T13:15:37.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.327+0000 7f38b953e640 1 --2- 172.21.15.125:0/1346790691 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f38a00664a0 0x7f38a0068960 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.327+0000 7f38b953e640 1 --2- 172.21.15.125:0/1346790691 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f38b415d6f0 0x7f38b4157d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.327+0000 7f38b953e640 1 --2- 172.21.15.125:0/1346790691 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f38b41538a0 0x7f38b4157850 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.327+0000 7f38b953e640 1 --2- 172.21.15.125:0/1346790691 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f38b40cf160 0x7f38b4152220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:37.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.327+0000 7f38b953e640 1 -- 172.21.15.125:0/1346790691 >> 172.21.15.125:0/1346790691 conn(0x7f38b406bfe0 msgr2=0x7f38b406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:37.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.327+0000 7f38b953e640 1 -- 172.21.15.125:0/1346790691 shutdown_connections 2023-10-30T13:15:37.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:37.327+0000 7f38b953e640 1 -- 172.21.15.125:0/1346790691 wait complete. 2023-10-30T13:15:37.350 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T13:15:39.351 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:39.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.487+0000 7fe12e3c5640 1 Processor -- start 2023-10-30T13:15:39.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.487+0000 7fe12e3c5640 1 -- start start 2023-10-30T13:15:39.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe12e3c5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe1280c6f60 0x7fe1281566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:39.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe12e3c5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1280c0a50 con 0x7fe1280c05d0 2023-10-30T13:15:39.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe12e3c5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe12806f6e0 con 0x7fe1280c6b80 2023-10-30T13:15:39.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe12e3c5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe12806f9c0 con 0x7fe1280c6f60 2023-10-30T13:15:39.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe1277fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe1280c05d0 0x7fe12813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36900/0 (socket says 172.21.15.125:36900) 2023-10-30T13:15:39.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe1277fe640 1 -- 172.21.15.125:0/705088566 learned_addr learned my addr 172.21.15.125:0/705088566 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:15:39.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe12c962640 1 --2- 172.21.15.125:0/705088566 >> v2:172.21.15.136:3300/0 conn(0x7fe1280c6f60 0x7fe1281566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:39.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 -- 172.21.15.125:0/705088566 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4142749599 0 0) 0x7fe1280c0a50 con 0x7fe1280c05d0 2023-10-30T13:15:39.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 -- 172.21.15.125:0/705088566 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe110003530 con 0x7fe1280c05d0 2023-10-30T13:15:39.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 -- 172.21.15.125:0/705088566 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3195291543 0 0) 0x7fe110003530 con 0x7fe1280c05d0 2023-10-30T13:15:39.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 -- 172.21.15.125:0/705088566 >> v2:172.21.15.136:3300/0 conn(0x7fe1280c6f60 msgr2=0x7fe1281566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:39.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 --2- 172.21.15.125:0/705088566 >> v2:172.21.15.136:3300/0 conn(0x7fe1280c6f60 0x7fe1281566a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 -- 172.21.15.125:0/705088566 >> v1:172.21.15.136:6789/0 conn(0x7fe1280c6b80 legacy=0x7fe1280bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:39.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 -- 172.21.15.125:0/705088566 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe12806fca0 con 0x7fe1280c05d0 2023-10-30T13:15:39.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 -- 172.21.15.125:0/705088566 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fe11c001680 con 0x7fe1280c05d0 2023-10-30T13:15:39.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 -- 172.21.15.125:0/705088566 >> v1:172.21.15.125:6789/0 conn(0x7fe1280c05d0 legacy=0x7fe12813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:39.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 --2- 172.21.15.125:0/705088566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe110003980 0x7fe110003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:39.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 --2- 172.21.15.125:0/705088566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1100047d0 0x7fe110004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:39.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 --2- 172.21.15.125:0/705088566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe110005190 0x7fe11000dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:39.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 -- 172.21.15.125:0/705088566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12806f9c0 con 0x7fe110005190 2023-10-30T13:15:39.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 -- 172.21.15.125:0/705088566 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe11000e240 con 0x7fe1100047d0 2023-10-30T13:15:39.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe126ffd640 1 -- 172.21.15.125:0/705088566 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe11000e400 con 0x7fe110003980 2023-10-30T13:15:39.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe12c962640 1 --2- 172.21.15.125:0/705088566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe110005190 0x7fe11000dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:39.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe127fff640 1 --2- 172.21.15.125:0/705088566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe110003980 0x7fe110003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:39.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe1277fe640 1 --2- 172.21.15.125:0/705088566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1100047d0 0x7fe110004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:39.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe12c962640 1 -- 172.21.15.125:0/705088566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe110003980 msgr2=0x7fe110003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:39.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe12c962640 1 --2- 172.21.15.125:0/705088566 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe110003980 0x7fe110003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe12c962640 1 -- 172.21.15.125:0/705088566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1100047d0 msgr2=0x7fe110004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:39.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe12c962640 1 --2- 172.21.15.125:0/705088566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1100047d0 0x7fe110004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe12c962640 1 -- 172.21.15.125:0/705088566 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe110003530 con 0x7fe110005190 2023-10-30T13:15:39.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.491+0000 7fe1277fe640 1 --2- 172.21.15.125:0/705088566 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe1100047d0 0x7fe110004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:39.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.495+0000 7fe12c962640 1 -- 172.21.15.125:0/705088566 reap_dead start 2023-10-30T13:15:39.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.495+0000 7fe12c962640 1 --2- 172.21.15.125:0/705088566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe110005190 0x7fe11000dba0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fe120009fc0 tx=0x7fe12001cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5bc7f035db5de7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:39.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.495+0000 7fe126ffd640 1 -- 172.21.15.125:0/705088566 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1200a5cb0 con 0x7fe110005190 2023-10-30T13:15:39.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.495+0000 7fe126ffd640 1 -- 172.21.15.125:0/705088566 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe12000f040 con 0x7fe110005190 2023-10-30T13:15:39.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.495+0000 7fe126ffd640 1 -- 172.21.15.125:0/705088566 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe120003be0 con 0x7fe110005190 2023-10-30T13:15:39.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.495+0000 7fe12e3c5640 1 -- 172.21.15.125:0/705088566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe110005190 msgr2=0x7fe11000dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:39.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.495+0000 7fe12e3c5640 1 --2- 172.21.15.125:0/705088566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe110005190 0x7fe11000dba0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7fe120009fc0 tx=0x7fe12001cca0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.495+0000 7fe12e3c5640 1 -- 172.21.15.125:0/705088566 shutdown_connections 2023-10-30T13:15:39.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.495+0000 7fe12e3c5640 1 --2- 172.21.15.125:0/705088566 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe110005190 0x7fe11000dba0 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.495+0000 7fe12e3c5640 1 -- 172.21.15.125:0/705088566 >> 172.21.15.125:0/705088566 conn(0x7fe1280c7910 msgr2=0x7fe1280c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:39.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.495+0000 7fe12e3c5640 1 -- 172.21.15.125:0/705088566 shutdown_connections 2023-10-30T13:15:39.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.495+0000 7fe12e3c5640 1 -- 172.21.15.125:0/705088566 wait complete. 2023-10-30T13:15:39.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12e3c5640 1 Processor -- start 2023-10-30T13:15:39.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12e3c5640 1 -- start start 2023-10-30T13:15:39.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12e3c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe110005190 0x7fe128158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:39.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12e3c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe128159750 0x7fe1281f0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:39.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12e3c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe128156bc0 0x7fe1281f1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:39.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12e3c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1281f0630 con 0x7fe128156bc0 2023-10-30T13:15:39.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12e3c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe128155890 con 0x7fe128159750 2023-10-30T13:15:39.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12e3c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe128155b40 con 0x7fe110005190 2023-10-30T13:15:39.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12c962640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe128156bc0 0x7fe1281f1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:39.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12c962640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe128156bc0 0x7fe1281f1000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33402/0 (socket says 172.21.15.125:33402) 2023-10-30T13:15:39.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12c962640 1 -- 172.21.15.125:0/1239642044 learned_addr learned my addr 172.21.15.125:0/1239642044 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:39.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe127fff640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe110005190 0x7fe128158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:39.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe1277fe640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe128159750 0x7fe1281f0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:39.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12c962640 1 -- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe110005190 msgr2=0x7fe128158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:39.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12c962640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe110005190 0x7fe128158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12c962640 1 -- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe128159750 msgr2=0x7fe1281f0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:39.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12c962640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe128159750 0x7fe1281f0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12c962640 1 -- 172.21.15.125:0/1239642044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe128155e20 con 0x7fe128156bc0 2023-10-30T13:15:39.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe127fff640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe110005190 0x7fe128158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:39.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12c962640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe128156bc0 0x7fe1281f1000 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fe12001c710 tx=0x7fe120009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:39.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe124ff9640 1 -- 172.21.15.125:0/1239642044 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe120003e70 con 0x7fe128156bc0 2023-10-30T13:15:39.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe124ff9640 1 -- 172.21.15.125:0/1239642044 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe120004030 con 0x7fe128156bc0 2023-10-30T13:15:39.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12e3c5640 1 -- 172.21.15.125:0/1239642044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1281effe0 con 0x7fe128156bc0 2023-10-30T13:15:39.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe124ff9640 1 -- 172.21.15.125:0/1239642044 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1200043a0 con 0x7fe128156bc0 2023-10-30T13:15:39.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12e3c5640 1 -- 172.21.15.125:0/1239642044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe128157440 con 0x7fe128156bc0 2023-10-30T13:15:39.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.499+0000 7fe12e3c5640 1 -- 172.21.15.125:0/1239642044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1281f07a0 con 0x7fe128156bc0 2023-10-30T13:15:39.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.503+0000 7fe124ff9640 1 -- 172.21.15.125:0/1239642044 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe120004500 con 0x7fe128156bc0 2023-10-30T13:15:39.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.503+0000 7fe124ff9640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe1140664a0 0x7fe114068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:39.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.503+0000 7fe124ff9640 1 -- 172.21.15.125:0/1239642044 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7fe120120200 con 0x7fe128156bc0 2023-10-30T13:15:39.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.503+0000 7fe127fff640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe1140664a0 0x7fe114068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:39.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.503+0000 7fe127fff640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe1140664a0 0x7fe114068960 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe118000c10 tx=0x7fe118009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:39.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.507+0000 7fe124ff9640 1 -- 172.21.15.125:0/1239642044 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe1200eceb0 con 0x7fe128156bc0 2023-10-30T13:15:39.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.687+0000 7fe12e3c5640 1 -- 172.21.15.125:0/1239642044 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe12804e170 con 0x7fe128156bc0 2023-10-30T13:15:39.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.687+0000 7fe124ff9640 1 -- 172.21.15.125:0/1239642044 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+6550 (secure 0 0 0) 0x7fe1200f04d0 con 0x7fe128156bc0 2023-10-30T13:15:39.691 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:39.691 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8833,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:29.286440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:39.692 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 220 2023-10-30T13:15:39.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.691+0000 7fe12e3c5640 1 -- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe1140664a0 msgr2=0x7fe114068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:39.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.691+0000 7fe12e3c5640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe1140664a0 0x7fe114068960 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fe118000c10 tx=0x7fe118009040 comp rx=0 tx=0).stop 2023-10-30T13:15:39.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.691+0000 7fe12e3c5640 1 -- 172.21.15.125:0/1239642044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe128156bc0 msgr2=0x7fe1281f1000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:39.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.691+0000 7fe12e3c5640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe128156bc0 0x7fe1281f1000 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7fe12001c710 tx=0x7fe120009750 comp rx=0 tx=0).stop 2023-10-30T13:15:39.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.691+0000 7fe12e3c5640 1 -- 172.21.15.125:0/1239642044 shutdown_connections 2023-10-30T13:15:39.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.691+0000 7fe12e3c5640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe1140664a0 0x7fe114068960 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.691+0000 7fe12e3c5640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe128156bc0 0x7fe1281f1000 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.691+0000 7fe12e3c5640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe128159750 0x7fe1281f0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.691+0000 7fe12e3c5640 1 --2- 172.21.15.125:0/1239642044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe110005190 0x7fe128158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:39.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.691+0000 7fe12e3c5640 1 -- 172.21.15.125:0/1239642044 >> 172.21.15.125:0/1239642044 conn(0x7fe1280c7910 msgr2=0x7fe12813b2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:39.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.691+0000 7fe12e3c5640 1 -- 172.21.15.125:0/1239642044 shutdown_connections 2023-10-30T13:15:39.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:39.691+0000 7fe12e3c5640 1 -- 172.21.15.125:0/1239642044 wait complete. 2023-10-30T13:15:39.712 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T13:15:39.713 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before thrashing 2023-10-30T13:15:40.052 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:15:40.052 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 32 -eq 48 ] 2023-10-30T13:15:40.052 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.32 2023-10-30T13:15:40.053 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.32 2023-10-30T13:15:40.053 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.32 2023-10-30T13:15:40.053 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.32 2023-10-30T13:15:40.054 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:15:44.714 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:44.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbabd6ea640 1 Processor -- start 2023-10-30T13:15:44.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbabd6ea640 1 -- start start 2023-10-30T13:15:44.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbabd6ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbab80c8cf0 0x7fbab80c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:44.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbabd6ea640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbab80c90d0 0x7fbab80ca340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:44.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbabd6ea640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbab80cef90 0x7fbab80cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:44.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbabd6ea640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab815ce30 con 0x7fbab80c90d0 2023-10-30T13:15:44.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbabd6ea640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbab8159880 con 0x7fbab80cef90 2023-10-30T13:15:44.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbabd6ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab8159ae0 con 0x7fbab80c8cf0 2023-10-30T13:15:44.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbab67fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbab80c90d0 0x7fbab80ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:44.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbab67fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbab80c90d0 0x7fbab80ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34578/0 (socket says 172.21.15.125:34578) 2023-10-30T13:15:44.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbab67fc640 1 -- 172.21.15.125:0/968274818 learned_addr learned my addr 172.21.15.125:0/968274818 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:44.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbab77fe640 1 --2- 172.21.15.125:0/968274818 >> v2:172.21.15.136:3300/0 conn(0x7fbab80cef90 0x7fbab80cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:44.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbab6ffd640 1 --2- 172.21.15.125:0/968274818 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbab80c8cf0 0x7fbab80c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:44.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbab67fc640 1 -- 172.21.15.125:0/968274818 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbab80c8cf0 msgr2=0x7fbab80c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:44.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbab67fc640 1 --2- 172.21.15.125:0/968274818 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbab80c8cf0 0x7fbab80c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:44.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbab67fc640 1 -- 172.21.15.125:0/968274818 >> v2:172.21.15.136:3300/0 conn(0x7fbab80cef90 msgr2=0x7fbab80cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:44.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbab67fc640 1 --2- 172.21.15.125:0/968274818 >> v2:172.21.15.136:3300/0 conn(0x7fbab80cef90 0x7fbab80cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:44.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.867+0000 7fbab67fc640 1 -- 172.21.15.125:0/968274818 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab8159dc0 con 0x7fbab80c90d0 2023-10-30T13:15:44.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbab67fc640 1 --2- 172.21.15.125:0/968274818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab80c90d0 0x7fbab80ca340 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fbab806c9f0 tx=0x7fbaac01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe17cc8920d763 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:44.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbab5ffb640 1 -- 172.21.15.125:0/968274818 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac0aa5f0 con 0x7fbab80c90d0 2023-10-30T13:15:44.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbab5ffb640 1 -- 172.21.15.125:0/968274818 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaac0aa750 con 0x7fbab80c90d0 2023-10-30T13:15:44.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbab5ffb640 1 -- 172.21.15.125:0/968274818 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaac0aaae0 con 0x7fbab80c90d0 2023-10-30T13:15:44.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbabd6ea640 1 -- 172.21.15.125:0/968274818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab80c90d0 msgr2=0x7fbab80ca340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:44.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbabd6ea640 1 --2- 172.21.15.125:0/968274818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab80c90d0 0x7fbab80ca340 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7fbab806c9f0 tx=0x7fbaac01c4d0 comp rx=0 tx=0).stop 2023-10-30T13:15:44.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbabd6ea640 1 -- 172.21.15.125:0/968274818 shutdown_connections 2023-10-30T13:15:44.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbabd6ea640 1 --2- 172.21.15.125:0/968274818 >> v2:172.21.15.136:3300/0 conn(0x7fbab80cef90 0x7fbab80cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:44.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbabd6ea640 1 --2- 172.21.15.125:0/968274818 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab80c90d0 0x7fbab80ca340 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:44.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbabd6ea640 1 --2- 172.21.15.125:0/968274818 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbab80c8cf0 0x7fbab80c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:44.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbabd6ea640 1 -- 172.21.15.125:0/968274818 >> 172.21.15.125:0/968274818 conn(0x7fbab806be40 msgr2=0x7fbab806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:44.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbabd6ea640 1 -- 172.21.15.125:0/968274818 shutdown_connections 2023-10-30T13:15:44.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbabd6ea640 1 -- 172.21.15.125:0/968274818 wait complete. 2023-10-30T13:15:44.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbabd6ea640 1 Processor -- start 2023-10-30T13:15:44.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.871+0000 7fbabd6ea640 1 -- start start 2023-10-30T13:15:44.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbabd6ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbab80c8cf0 0x7fbab80c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:44.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbabd6ea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbab80c90d0 0x7fbab81f3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:44.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbabd6ea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab80cef90 0x7fbab81f36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:44.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbabd6ea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab80c7bf0 con 0x7fbab80cef90 2023-10-30T13:15:44.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbabd6ea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab80c6d90 con 0x7fbab80c90d0 2023-10-30T13:15:44.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbabd6ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbab80c88c0 con 0x7fbab80c8cf0 2023-10-30T13:15:44.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbab77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab80cef90 0x7fbab81f36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:44.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbab77fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab80cef90 0x7fbab81f36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34592/0 (socket says 172.21.15.125:34592) 2023-10-30T13:15:44.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbab77fe640 1 -- 172.21.15.125:0/107804857 learned_addr learned my addr 172.21.15.125:0/107804857 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:44.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbab6ffd640 1 --2- 172.21.15.125:0/107804857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbab80c8cf0 0x7fbab80c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:44.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbab67fc640 1 --2- 172.21.15.125:0/107804857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbab80c90d0 0x7fbab81f3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:44.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbab77fe640 1 -- 172.21.15.125:0/107804857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbab80c8cf0 msgr2=0x7fbab80c8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:44.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbab77fe640 1 --2- 172.21.15.125:0/107804857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbab80c8cf0 0x7fbab80c8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:44.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbab77fe640 1 -- 172.21.15.125:0/107804857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbab80c90d0 msgr2=0x7fbab81f3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:44.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbab77fe640 1 --2- 172.21.15.125:0/107804857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbab80c90d0 0x7fbab81f3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:44.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbab77fe640 1 -- 172.21.15.125:0/107804857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaac0094d0 con 0x7fbab80cef90 2023-10-30T13:15:44.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbab77fe640 1 --2- 172.21.15.125:0/107804857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab80cef90 0x7fbab81f36b0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fbab0007e40 tx=0x7fbab000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:44.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fba9f7fe640 1 -- 172.21.15.125:0/107804857 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab0014620 con 0x7fbab80cef90 2023-10-30T13:15:44.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fba9f7fe640 1 -- 172.21.15.125:0/107804857 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab000ee90 con 0x7fbab80cef90 2023-10-30T13:15:44.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fba9f7fe640 1 -- 172.21.15.125:0/107804857 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab00149e0 con 0x7fbab80cef90 2023-10-30T13:15:44.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbabd6ea640 1 -- 172.21.15.125:0/107804857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab8158990 con 0x7fbab80cef90 2023-10-30T13:15:44.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbabd6ea640 1 -- 172.21.15.125:0/107804857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab80c72c0 con 0x7fbab80cef90 2023-10-30T13:15:44.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.875+0000 7fbabd6ea640 1 -- 172.21.15.125:0/107804857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab80c7da0 con 0x7fbab80cef90 2023-10-30T13:15:44.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.879+0000 7fba9f7fe640 1 -- 172.21.15.125:0/107804857 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbab0014b40 con 0x7fbab80cef90 2023-10-30T13:15:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.879+0000 7fba9f7fe640 1 --2- 172.21.15.125:0/107804857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbaa40664a0 0x7fbaa4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.879+0000 7fba9f7fe640 1 -- 172.21.15.125:0/107804857 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7fbab0085110 con 0x7fbab80cef90 2023-10-30T13:15:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.879+0000 7fbab6ffd640 1 --2- 172.21.15.125:0/107804857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbaa40664a0 0x7fbaa4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:44.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.879+0000 7fbab6ffd640 1 --2- 172.21.15.125:0/107804857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbaa40664a0 0x7fbaa4068960 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fbaa80045f0 tx=0x7fbaa8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:44.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:44.883+0000 7fba9f7fe640 1 -- 172.21.15.125:0/107804857 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbab80c7da0 con 0x7fbab80cef90 2023-10-30T13:15:45.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.083+0000 7fbabd6ea640 1 -- 172.21.15.125:0/107804857 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbab80c7da0 con 0x7fbab80cef90 2023-10-30T13:15:45.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.083+0000 7fba9f7fe640 1 -- 172.21.15.125:0/107804857 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 220 v220) v1 ==== 95+0+6550 (secure 0 0 0) 0x7fbab80c7da0 con 0x7fbab80cef90 2023-10-30T13:15:45.088 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:45.088 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":220,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8833,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":219,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:29.286440+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:45.089 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 220 2023-10-30T13:15:45.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.087+0000 7fbabd6ea640 1 -- 172.21.15.125:0/107804857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbaa40664a0 msgr2=0x7fbaa4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.087+0000 7fbabd6ea640 1 --2- 172.21.15.125:0/107804857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbaa40664a0 0x7fbaa4068960 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fbaa80045f0 tx=0x7fbaa8008040 comp rx=0 tx=0).stop 2023-10-30T13:15:45.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.087+0000 7fbabd6ea640 1 -- 172.21.15.125:0/107804857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab80cef90 msgr2=0x7fbab81f36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.087+0000 7fbabd6ea640 1 --2- 172.21.15.125:0/107804857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab80cef90 0x7fbab81f36b0 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fbab0007e40 tx=0x7fbab000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.087+0000 7fbabd6ea640 1 -- 172.21.15.125:0/107804857 shutdown_connections 2023-10-30T13:15:45.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.087+0000 7fbabd6ea640 1 --2- 172.21.15.125:0/107804857 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbaa40664a0 0x7fbaa4068960 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.087+0000 7fbabd6ea640 1 --2- 172.21.15.125:0/107804857 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbab80cef90 0x7fbab81f36b0 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.087+0000 7fbabd6ea640 1 --2- 172.21.15.125:0/107804857 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbab80c90d0 0x7fbab81f3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.087+0000 7fbabd6ea640 1 --2- 172.21.15.125:0/107804857 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbab80c8cf0 0x7fbab80c8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.087+0000 7fbabd6ea640 1 -- 172.21.15.125:0/107804857 >> 172.21.15.125:0/107804857 conn(0x7fbab806be40 msgr2=0x7fbab80cdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:45.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.087+0000 7fbabd6ea640 1 -- 172.21.15.125:0/107804857 shutdown_connections 2023-10-30T13:15:45.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.087+0000 7fbabd6ea640 1 -- 172.21.15.125:0/107804857 wait complete. 2023-10-30T13:15:45.109 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=2) 2023-10-30T13:15:45.110 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T13:15:45.110 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:15:45.137 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T13:15:45.137 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:45.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.271+0000 7f00b1565640 1 Processor -- start 2023-10-30T13:15:45.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.271+0000 7f00b1565640 1 -- start start 2023-10-30T13:15:45.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.271+0000 7f00b1565640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00ac0c05d0 0x7f00ac13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:45.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00b1565640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac0c0a50 con 0x7f00ac0c6b80 2023-10-30T13:15:45.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00b1565640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00ac06f6e0 con 0x7f00ac0c6f60 2023-10-30T13:15:45.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00b1565640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00ac06f9c0 con 0x7f00ac0c05d0 2023-10-30T13:15:45.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00a27fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00ac0c05d0 0x7f00ac13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:45.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00a27fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00ac0c05d0 0x7f00ac13ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34604/0 (socket says 172.21.15.125:34604) 2023-10-30T13:15:45.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00aaffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f00ac0c6b80 0x7f00ac0bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38360/0 (socket says 172.21.15.125:38360) 2023-10-30T13:15:45.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00a27fc640 1 -- 172.21.15.125:0/3466427311 learned_addr learned my addr 172.21.15.125:0/3466427311 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:45.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00a27fc640 1 -- 172.21.15.125:0/3466427311 >> v1:172.21.15.136:6789/0 conn(0x7f00ac0c6f60 legacy=0x7f00ac1566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00a27fc640 1 -- 172.21.15.125:0/3466427311 >> v1:172.21.15.125:6789/0 conn(0x7f00ac0c6b80 legacy=0x7f00ac0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00a27fc640 1 -- 172.21.15.125:0/3466427311 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00ac06fca0 con 0x7f00ac0c05d0 2023-10-30T13:15:45.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00aa7fc640 1 -- 172.21.15.125:0/3466427311 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212017010 0 0) 0x7f00ac0c0a50 con 0x7f00ac0c6b80 2023-10-30T13:15:45.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00a27fc640 1 --2- 172.21.15.125:0/3466427311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00ac0c05d0 0x7f00ac13ed90 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f0094009e50 tx=0x7f009401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8b245d9fd8704e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:45.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00aa7fc640 1 -- 172.21.15.125:0/3466427311 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00940a57f0 con 0x7f00ac0c05d0 2023-10-30T13:15:45.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00aa7fc640 1 -- 172.21.15.125:0/3466427311 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00940a5950 con 0x7f00ac0c05d0 2023-10-30T13:15:45.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.275+0000 7f00aa7fc640 1 -- 172.21.15.125:0/3466427311 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00940a5ce0 con 0x7f00ac0c05d0 2023-10-30T13:15:45.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 -- 172.21.15.125:0/3466427311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00ac0c05d0 msgr2=0x7f00ac13ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 --2- 172.21.15.125:0/3466427311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00ac0c05d0 0x7f00ac13ed90 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f0094009e50 tx=0x7f009401ca10 comp rx=0 tx=0).stop 2023-10-30T13:15:45.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 -- 172.21.15.125:0/3466427311 shutdown_connections 2023-10-30T13:15:45.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 --2- 172.21.15.125:0/3466427311 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00ac0c05d0 0x7f00ac13ed90 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 -- 172.21.15.125:0/3466427311 >> 172.21.15.125:0/3466427311 conn(0x7f00ac0c7910 msgr2=0x7f00ac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:45.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 -- 172.21.15.125:0/3466427311 shutdown_connections 2023-10-30T13:15:45.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 -- 172.21.15.125:0/3466427311 wait complete. 2023-10-30T13:15:45.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 Processor -- start 2023-10-30T13:15:45.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 -- start start 2023-10-30T13:15:45.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00ac0c05d0 0x7f00ac158d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:45.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00ac0c6b80 0x7f00ac138600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:45.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00ac0c6f60 0x7f00ac138b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:45.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac1566b0 con 0x7f00ac0c05d0 2023-10-30T13:15:45.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac13dcd0 con 0x7f00ac0c6b80 2023-10-30T13:15:45.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00b1565640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00ac0c1fe0 con 0x7f00ac0c6f60 2023-10-30T13:15:45.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00aaffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00ac0c05d0 0x7f00ac158d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:45.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00aaffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00ac0c05d0 0x7f00ac158d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34610/0 (socket says 172.21.15.125:34610) 2023-10-30T13:15:45.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00aaffd640 1 -- 172.21.15.125:0/2371106486 learned_addr learned my addr 172.21.15.125:0/2371106486 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:45.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00a27fc640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00ac0c6b80 0x7f00ac138600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:45.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00ab7fe640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00ac0c6f60 0x7f00ac138b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:45.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00aaffd640 1 -- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00ac0c6f60 msgr2=0x7f00ac138b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00aaffd640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00ac0c6f60 0x7f00ac138b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00aaffd640 1 -- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00ac0c6b80 msgr2=0x7f00ac138600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00aaffd640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00ac0c6b80 0x7f00ac138600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.279+0000 7f00aaffd640 1 -- 172.21.15.125:0/2371106486 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0094009b30 con 0x7f00ac0c05d0 2023-10-30T13:15:45.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.283+0000 7f00a27fc640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00ac0c6b80 0x7f00ac138600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:45.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.283+0000 7f00aaffd640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00ac0c05d0 0x7f00ac158d80 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f009c00c4a0 tx=0x7f009c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:45.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.283+0000 7f00a3fff640 1 -- 172.21.15.125:0/2371106486 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c0074e0 con 0x7f00ac0c05d0 2023-10-30T13:15:45.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.283+0000 7f00a3fff640 1 -- 172.21.15.125:0/2371106486 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f009c007640 con 0x7f00ac0c05d0 2023-10-30T13:15:45.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.283+0000 7f00a3fff640 1 -- 172.21.15.125:0/2371106486 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f009c004a30 con 0x7f00ac0c05d0 2023-10-30T13:15:45.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.283+0000 7f00b1565640 1 -- 172.21.15.125:0/2371106486 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00ac155b50 con 0x7f00ac0c05d0 2023-10-30T13:15:45.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.283+0000 7f00b1565640 1 -- 172.21.15.125:0/2371106486 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00ac0c1d00 con 0x7f00ac0c05d0 2023-10-30T13:15:45.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.283+0000 7f00b1565640 1 -- 172.21.15.125:0/2371106486 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00ac0c1b60 con 0x7f00ac0c05d0 2023-10-30T13:15:45.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.283+0000 7f00a3fff640 1 -- 172.21.15.125:0/2371106486 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f009c004b90 con 0x7f00ac0c05d0 2023-10-30T13:15:45.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.283+0000 7f00a3fff640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f008c0664d0 0x7f008c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:45.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.283+0000 7f00a3fff640 1 -- 172.21.15.125:0/2371106486 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7f009c083910 con 0x7f00ac0c05d0 2023-10-30T13:15:45.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.283+0000 7f00a27fc640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f008c0664d0 0x7f008c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:45.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.287+0000 7f00a27fc640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f008c0664d0 0x7f008c068990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f00940ae000 tx=0x7f00940a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:45.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.291+0000 7f00a3fff640 1 -- 172.21.15.125:0/2371106486 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f00ac0c1b60 con 0x7f00ac0c05d0 2023-10-30T13:15:45.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.503+0000 7f00b1565640 1 -- 172.21.15.125:0/2371106486 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00ac0c1b60 con 0x7f00ac0c05d0 2023-10-30T13:15:45.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.503+0000 7f00a3fff640 1 -- 172.21.15.125:0/2371106486 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6551 (secure 0 0 0) 0x7f00ac0c1b60 con 0x7f00ac0c05d0 2023-10-30T13:15:45.509 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:45.509 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8833,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:44.822534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:45.509 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 221 2023-10-30T13:15:45.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.507+0000 7f00b1565640 1 -- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f008c0664d0 msgr2=0x7f008c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.507+0000 7f00b1565640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f008c0664d0 0x7f008c068990 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f00940ae000 tx=0x7f00940a7070 comp rx=0 tx=0).stop 2023-10-30T13:15:45.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.507+0000 7f00b1565640 1 -- 172.21.15.125:0/2371106486 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00ac0c05d0 msgr2=0x7f00ac158d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:45.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.507+0000 7f00b1565640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00ac0c05d0 0x7f00ac158d80 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f009c00c4a0 tx=0x7f009c00c960 comp rx=0 tx=0).stop 2023-10-30T13:15:45.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.507+0000 7f00b1565640 1 -- 172.21.15.125:0/2371106486 shutdown_connections 2023-10-30T13:15:45.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.507+0000 7f00b1565640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f008c0664d0 0x7f008c068990 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.507+0000 7f00b1565640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00ac0c6f60 0x7f00ac138b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.507+0000 7f00b1565640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00ac0c6b80 0x7f00ac138600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.507+0000 7f00b1565640 1 --2- 172.21.15.125:0/2371106486 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00ac0c05d0 0x7f00ac158d80 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:45.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.507+0000 7f00b1565640 1 -- 172.21.15.125:0/2371106486 >> 172.21.15.125:0/2371106486 conn(0x7f00ac0c7910 msgr2=0x7f00ac13ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:45.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.507+0000 7f00b1565640 1 -- 172.21.15.125:0/2371106486 shutdown_connections 2023-10-30T13:15:45.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:45.507+0000 7f00b1565640 1 -- 172.21.15.125:0/2371106486 wait complete. 2023-10-30T13:15:45.533 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:15:46.973 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:15:46.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:15:47.534 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:47.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d326b640 1 Processor -- start 2023-10-30T13:15:47.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d326b640 1 -- start start 2023-10-30T13:15:47.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d326b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f00cc1538a0 0x7f00cc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:47.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d326b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00cc0ca0e0 0x7f00cc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:47.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d326b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00cc0cf160 0x7f00cc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:47.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d326b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00cc0cb800 con 0x7f00cc0cf160 2023-10-30T13:15:47.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d326b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00cc15dc30 con 0x7f00cc1538a0 2023-10-30T13:15:47.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d326b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00cc0cb330 con 0x7f00cc0ca0e0 2023-10-30T13:15:47.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d1808640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00cc0cf160 0x7f00cc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:47.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d1808640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f00cc0cf160 0x7f00cc0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34622/0 (socket says 172.21.15.125:34622) 2023-10-30T13:15:47.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d1808640 1 -- 172.21.15.125:0/3673141391 learned_addr learned my addr 172.21.15.125:0/3673141391 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:47.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d0806640 1 --2- 172.21.15.125:0/3673141391 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00cc0ca0e0 0x7f00cc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:47.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d1808640 1 -- 172.21.15.125:0/3673141391 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00cc0ca0e0 msgr2=0x7f00cc15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:47.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d1808640 1 --2- 172.21.15.125:0/3673141391 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00cc0ca0e0 0x7f00cc15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d1808640 1 -- 172.21.15.125:0/3673141391 >> v2:172.21.15.136:3300/0 conn(0x7f00cc1538a0 msgr2=0x7f00cc15d6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-30T13:15:47.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d1808640 1 --2- 172.21.15.125:0/3673141391 >> v2:172.21.15.136:3300/0 conn(0x7f00cc1538a0 0x7f00cc15d6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.679+0000 7f00d1808640 1 -- 172.21.15.125:0/3673141391 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00cc0cad70 con 0x7f00cc0cf160 2023-10-30T13:15:47.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.683+0000 7f00d1808640 1 --2- 172.21.15.125:0/3673141391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00cc0cf160 0x7f00cc0cf590 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f00cc0cfd10 tx=0x7f00c401f1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=282af72298e2c3f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:47.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.683+0000 7f00c3fff640 1 -- 172.21.15.125:0/3673141391 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c40af400 con 0x7f00cc0cf160 2023-10-30T13:15:47.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.683+0000 7f00c3fff640 1 -- 172.21.15.125:0/3673141391 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00c40ab050 con 0x7f00cc0cf160 2023-10-30T13:15:47.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.683+0000 7f00c3fff640 1 -- 172.21.15.125:0/3673141391 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00c40af780 con 0x7f00cc0cf160 2023-10-30T13:15:47.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.683+0000 7f00d326b640 1 -- 172.21.15.125:0/3673141391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00cc0cf160 msgr2=0x7f00cc0cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:47.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.683+0000 7f00d326b640 1 --2- 172.21.15.125:0/3673141391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00cc0cf160 0x7f00cc0cf590 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f00cc0cfd10 tx=0x7f00c401f1e0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.683+0000 7f00d326b640 1 -- 172.21.15.125:0/3673141391 shutdown_connections 2023-10-30T13:15:47.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.683+0000 7f00d326b640 1 --2- 172.21.15.125:0/3673141391 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00cc0cf160 0x7f00cc0cf590 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.683+0000 7f00d326b640 1 --2- 172.21.15.125:0/3673141391 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00cc0ca0e0 0x7f00cc15a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.683+0000 7f00d326b640 1 --2- 172.21.15.125:0/3673141391 >> v2:172.21.15.136:3300/0 conn(0x7f00cc1538a0 0x7f00cc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.683+0000 7f00d326b640 1 -- 172.21.15.125:0/3673141391 >> 172.21.15.125:0/3673141391 conn(0x7f00cc06bfe0 msgr2=0x7f00cc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:47.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.683+0000 7f00d326b640 1 -- 172.21.15.125:0/3673141391 shutdown_connections 2023-10-30T13:15:47.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.683+0000 7f00d326b640 1 -- 172.21.15.125:0/3673141391 wait complete. 2023-10-30T13:15:47.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d326b640 1 Processor -- start 2023-10-30T13:15:47.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d326b640 1 -- start start 2023-10-30T13:15:47.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d326b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00cc0ca0e0 0x7f00cc1429e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:47.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d326b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00cc0cf160 0x7f00cc0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:47.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d326b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00cc1538a0 0x7f00cc1e5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:47.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d326b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00cc1efc50 con 0x7f00cc0cf160 2023-10-30T13:15:47.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d326b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00cc0ce230 con 0x7f00cc1538a0 2023-10-30T13:15:47.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d326b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00cc0ceac0 con 0x7f00cc0ca0e0 2023-10-30T13:15:47.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d0806640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00cc0cf160 0x7f00cc0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:47.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d0806640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00cc0cf160 0x7f00cc0ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34632/0 (socket says 172.21.15.125:34632) 2023-10-30T13:15:47.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d0806640 1 -- 172.21.15.125:0/412416149 learned_addr learned my addr 172.21.15.125:0/412416149 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:47.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d1007640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00cc0ca0e0 0x7f00cc1429e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:47.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d1808640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00cc1538a0 0x7f00cc1e5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:47.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d0806640 1 -- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00cc0ca0e0 msgr2=0x7f00cc1429e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:47.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d0806640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00cc0ca0e0 0x7f00cc1429e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d0806640 1 -- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00cc1538a0 msgr2=0x7f00cc1e5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:47.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d0806640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00cc1538a0 0x7f00cc1e5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d0806640 1 -- 172.21.15.125:0/412416149 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c4009b30 con 0x7f00cc0cf160 2023-10-30T13:15:47.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d1007640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00cc0ca0e0 0x7f00cc1429e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:47.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d0806640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00cc0cf160 0x7f00cc0ccfb0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00d800 tx=0x7f00bc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:47.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00c1ffb640 1 -- 172.21.15.125:0/412416149 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc00b6c0 con 0x7f00cc0cf160 2023-10-30T13:15:47.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00c1ffb640 1 -- 172.21.15.125:0/412416149 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00bc00b820 con 0x7f00cc0cf160 2023-10-30T13:15:47.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d326b640 1 -- 172.21.15.125:0/412416149 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00cc0ceda0 con 0x7f00cc0cf160 2023-10-30T13:15:47.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00c1ffb640 1 -- 172.21.15.125:0/412416149 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc00bb90 con 0x7f00cc0cf160 2023-10-30T13:15:47.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.687+0000 7f00d326b640 1 -- 172.21.15.125:0/412416149 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00cc1ef8e0 con 0x7f00cc0cf160 2023-10-30T13:15:47.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.691+0000 7f00d326b640 1 -- 172.21.15.125:0/412416149 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00cc0b2a50 con 0x7f00cc0cf160 2023-10-30T13:15:47.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.691+0000 7f00c1ffb640 1 -- 172.21.15.125:0/412416149 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f00bc00bcf0 con 0x7f00cc0cf160 2023-10-30T13:15:47.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.691+0000 7f00c1ffb640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00b00664a0 0x7f00b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:47.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.691+0000 7f00c1ffb640 1 -- 172.21.15.125:0/412416149 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7f00bc084d00 con 0x7f00cc0cf160 2023-10-30T13:15:47.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.691+0000 7f00d1007640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00b00664a0 0x7f00b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:47.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.691+0000 7f00d1007640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00b00664a0 0x7f00b0068960 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f00c8009890 tx=0x7f00c8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:47.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.695+0000 7f00c1ffb640 1 -- 172.21.15.125:0/412416149 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f00bc051890 con 0x7f00cc0cf160 2023-10-30T13:15:47.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.875+0000 7f00d326b640 1 -- 172.21.15.125:0/412416149 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f00cc04e170 con 0x7f00cc0cf160 2023-10-30T13:15:47.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.879+0000 7f00c1ffb640 1 -- 172.21.15.125:0/412416149 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6551 (secure 0 0 0) 0x7f00bc011340 con 0x7f00cc0cf160 2023-10-30T13:15:47.881 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:47.882 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8833,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:44.822534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:47.882 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 221 2023-10-30T13:15:47.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.879+0000 7f00d326b640 1 -- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00b00664a0 msgr2=0x7f00b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:47.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.879+0000 7f00d326b640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00b00664a0 0x7f00b0068960 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f00c8009890 tx=0x7f00c8008040 comp rx=0 tx=0).stop 2023-10-30T13:15:47.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.879+0000 7f00d326b640 1 -- 172.21.15.125:0/412416149 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00cc0cf160 msgr2=0x7f00cc0ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:47.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.879+0000 7f00d326b640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00cc0cf160 0x7f00cc0ccfb0 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7f00bc00d800 tx=0x7f00bc00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.879+0000 7f00d326b640 1 -- 172.21.15.125:0/412416149 shutdown_connections 2023-10-30T13:15:47.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.879+0000 7f00d326b640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f00b00664a0 0x7f00b0068960 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.879+0000 7f00d326b640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f00cc1538a0 0x7f00cc1e5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.879+0000 7f00d326b640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f00cc0cf160 0x7f00cc0ccfb0 unknown :-1 s=CLOSED pgs=1874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.879+0000 7f00d326b640 1 --2- 172.21.15.125:0/412416149 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f00cc0ca0e0 0x7f00cc1429e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:47.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.879+0000 7f00d326b640 1 -- 172.21.15.125:0/412416149 >> 172.21.15.125:0/412416149 conn(0x7f00cc06bfe0 msgr2=0x7f00cc066310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:47.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.879+0000 7f00d326b640 1 -- 172.21.15.125:0/412416149 shutdown_connections 2023-10-30T13:15:47.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:47.879+0000 7f00d326b640 1 -- 172.21.15.125:0/412416149 wait complete. 2023-10-30T13:15:47.902 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:15:49.903 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:50.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f787a409640 1 Processor -- start 2023-10-30T13:15:50.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f787a409640 1 -- start start 2023-10-30T13:15:50.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f787a409640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f78741538a0 0x7f787415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:50.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f787a409640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f78740ca0e0 0x7f787415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:50.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f787a409640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78740cb800 con 0x7f78740cf160 2023-10-30T13:15:50.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f787a409640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f787415dc30 con 0x7f78740ca0e0 2023-10-30T13:15:50.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f787a409640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78740cb330 con 0x7f78741538a0 2023-10-30T13:15:50.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f78737fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f78740ca0e0 0x7f787415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:50.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f78789a6640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f78740cf160 0x7f78740cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38368/0 (socket says 172.21.15.125:38368) 2023-10-30T13:15:50.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f78737fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f78740ca0e0 0x7f787415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34644/0 (socket says 172.21.15.125:34644) 2023-10-30T13:15:50.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f78789a6640 1 -- 172.21.15.125:0/3630086928 learned_addr learned my addr 172.21.15.125:0/3630086928 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:15:50.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f7873fff640 1 --2- 172.21.15.125:0/3630086928 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f78741538a0 0x7f787415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:50.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f78737fe640 1 -- 172.21.15.125:0/3630086928 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f78741538a0 msgr2=0x7f787415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:50.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f78737fe640 1 --2- 172.21.15.125:0/3630086928 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f78741538a0 0x7f787415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:50.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f78737fe640 1 -- 172.21.15.125:0/3630086928 >> v1:172.21.15.125:6789/0 conn(0x7f78740cf160 legacy=0x7f78740cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:50.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f78737fe640 1 -- 172.21.15.125:0/3630086928 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78740cad70 con 0x7f78740ca0e0 2023-10-30T13:15:50.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.047+0000 7f7872ffd640 1 -- 172.21.15.125:0/3630086928 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776571882 0 0) 0x7f78740cb800 con 0x7f78740cf160 2023-10-30T13:15:50.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.051+0000 7f78737fe640 1 --2- 172.21.15.125:0/3630086928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78740ca0e0 0x7f787415a090 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f78680097c0 tx=0x7f786801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49fe8c7a25785bc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:50.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.051+0000 7f7872ffd640 1 -- 172.21.15.125:0/3630086928 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78680a8970 con 0x7f78740ca0e0 2023-10-30T13:15:50.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.051+0000 7f7872ffd640 1 -- 172.21.15.125:0/3630086928 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78680a8ad0 con 0x7f78740ca0e0 2023-10-30T13:15:50.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.051+0000 7f7872ffd640 1 -- 172.21.15.125:0/3630086928 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78680a8e60 con 0x7f78740ca0e0 2023-10-30T13:15:50.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.051+0000 7f787a409640 1 -- 172.21.15.125:0/3630086928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78740ca0e0 msgr2=0x7f787415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:50.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.051+0000 7f787a409640 1 --2- 172.21.15.125:0/3630086928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78740ca0e0 0x7f787415a090 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f78680097c0 tx=0x7f786801ec70 comp rx=0 tx=0).stop 2023-10-30T13:15:50.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.051+0000 7f787a409640 1 -- 172.21.15.125:0/3630086928 shutdown_connections 2023-10-30T13:15:50.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.051+0000 7f787a409640 1 --2- 172.21.15.125:0/3630086928 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78740ca0e0 0x7f787415a090 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:50.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.051+0000 7f787a409640 1 --2- 172.21.15.125:0/3630086928 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f78741538a0 0x7f787415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:50.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.051+0000 7f787a409640 1 -- 172.21.15.125:0/3630086928 >> 172.21.15.125:0/3630086928 conn(0x7f787406bfe0 msgr2=0x7f787406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:50.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.051+0000 7f787a409640 1 -- 172.21.15.125:0/3630086928 shutdown_connections 2023-10-30T13:15:50.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.051+0000 7f787a409640 1 -- 172.21.15.125:0/3630086928 wait complete. 2023-10-30T13:15:50.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f787a409640 1 Processor -- start 2023-10-30T13:15:50.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f787a409640 1 -- start start 2023-10-30T13:15:50.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f787a409640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78740ca0e0 0x7f78741ef500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:50.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f787a409640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78740cf160 0x7f78741e3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:50.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f787a409640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f78741538a0 0x7f78741e5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:50.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f787a409640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78741e4450 con 0x7f78740cf160 2023-10-30T13:15:50.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f787a409640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78741e2ff0 con 0x7f78740ca0e0 2023-10-30T13:15:50.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f787a409640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78741efa40 con 0x7f78741538a0 2023-10-30T13:15:50.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f78737fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78740cf160 0x7f78741e3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:50.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f78737fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78740cf160 0x7f78741e3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34650/0 (socket says 172.21.15.125:34650) 2023-10-30T13:15:50.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f78737fe640 1 -- 172.21.15.125:0/3351368187 learned_addr learned my addr 172.21.15.125:0/3351368187 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:50.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f7873fff640 1 --2- 172.21.15.125:0/3351368187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78740ca0e0 0x7f78741ef500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:50.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f78789a6640 1 --2- 172.21.15.125:0/3351368187 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f78741538a0 0x7f78741e5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:50.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f78737fe640 1 -- 172.21.15.125:0/3351368187 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f78741538a0 msgr2=0x7f78741e5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:50.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f78737fe640 1 --2- 172.21.15.125:0/3351368187 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f78741538a0 0x7f78741e5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:50.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f78737fe640 1 -- 172.21.15.125:0/3351368187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78740ca0e0 msgr2=0x7f78741ef500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:50.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f78737fe640 1 --2- 172.21.15.125:0/3351368187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78740ca0e0 0x7f78741ef500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:50.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f78737fe640 1 -- 172.21.15.125:0/3351368187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7868009420 con 0x7f78740cf160 2023-10-30T13:15:50.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f78737fe640 1 --2- 172.21.15.125:0/3351368187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78740cf160 0x7f78741e3480 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f78680b3040 tx=0x7f786801ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:50.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f7870ff9640 1 -- 172.21.15.125:0/3351368187 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78680b9630 con 0x7f78740cf160 2023-10-30T13:15:50.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f7870ff9640 1 -- 172.21.15.125:0/3351368187 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78680b97b0 con 0x7f78740cf160 2023-10-30T13:15:50.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f787a409640 1 -- 172.21.15.125:0/3351368187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78741e39c0 con 0x7f78740cf160 2023-10-30T13:15:50.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f7870ff9640 1 -- 172.21.15.125:0/3351368187 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78680b9b60 con 0x7f78740cf160 2023-10-30T13:15:50.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.055+0000 7f787a409640 1 -- 172.21.15.125:0/3351368187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78741eff90 con 0x7f78740cf160 2023-10-30T13:15:50.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.059+0000 7f787a409640 1 -- 172.21.15.125:0/3351368187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78740b2a50 con 0x7f78740cf160 2023-10-30T13:15:50.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.059+0000 7f7870ff9640 1 -- 172.21.15.125:0/3351368187 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f78680b7040 con 0x7f78740cf160 2023-10-30T13:15:50.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.059+0000 7f7870ff9640 1 --2- 172.21.15.125:0/3351368187 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f78600664a0 0x7f7860068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:50.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.059+0000 7f7870ff9640 1 -- 172.21.15.125:0/3351368187 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7f7868122790 con 0x7f78740cf160 2023-10-30T13:15:50.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.059+0000 7f7873fff640 1 --2- 172.21.15.125:0/3351368187 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f78600664a0 0x7f7860068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:50.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.059+0000 7f7873fff640 1 --2- 172.21.15.125:0/3351368187 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f78600664a0 0x7f7860068960 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f786400b930 tx=0x7f7864008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:50.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.063+0000 7f7870ff9640 1 -- 172.21.15.125:0/3351368187 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f78680ef290 con 0x7f78740cf160 2023-10-30T13:15:50.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.243+0000 7f787a409640 1 -- 172.21.15.125:0/3351368187 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f787404e170 con 0x7f78740cf160 2023-10-30T13:15:50.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.243+0000 7f7870ff9640 1 -- 172.21.15.125:0/3351368187 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6551 (secure 0 0 0) 0x7f78680b5020 con 0x7f78740cf160 2023-10-30T13:15:50.249 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:50.249 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8833,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:44.822534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:50.250 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 221 2023-10-30T13:15:50.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.247+0000 7f787a409640 1 -- 172.21.15.125:0/3351368187 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f78600664a0 msgr2=0x7f7860068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:50.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.247+0000 7f787a409640 1 --2- 172.21.15.125:0/3351368187 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f78600664a0 0x7f7860068960 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f786400b930 tx=0x7f7864008040 comp rx=0 tx=0).stop 2023-10-30T13:15:50.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.247+0000 7f787a409640 1 -- 172.21.15.125:0/3351368187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78740cf160 msgr2=0x7f78741e3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:50.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.247+0000 7f787a409640 1 --2- 172.21.15.125:0/3351368187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78740cf160 0x7f78741e3480 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f78680b3040 tx=0x7f786801ba10 comp rx=0 tx=0).stop 2023-10-30T13:15:50.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.247+0000 7f787a409640 1 -- 172.21.15.125:0/3351368187 shutdown_connections 2023-10-30T13:15:50.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.247+0000 7f787a409640 1 --2- 172.21.15.125:0/3351368187 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f78600664a0 0x7f7860068960 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:50.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.247+0000 7f787a409640 1 --2- 172.21.15.125:0/3351368187 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f78741538a0 0x7f78741e5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:50.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.247+0000 7f787a409640 1 --2- 172.21.15.125:0/3351368187 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f78740cf160 0x7f78741e3480 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:50.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.247+0000 7f787a409640 1 --2- 172.21.15.125:0/3351368187 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f78740ca0e0 0x7f78741ef500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:50.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.247+0000 7f787a409640 1 -- 172.21.15.125:0/3351368187 >> 172.21.15.125:0/3351368187 conn(0x7f787406bfe0 msgr2=0x7f7874150f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:50.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.247+0000 7f787a409640 1 -- 172.21.15.125:0/3351368187 shutdown_connections 2023-10-30T13:15:50.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:50.247+0000 7f787a409640 1 -- 172.21.15.125:0/3351368187 wait complete. 2023-10-30T13:15:50.270 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:15:52.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:52.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ac6a9c640 1 Processor -- start 2023-10-30T13:15:52.418 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ac6a9c640 1 -- start start 2023-10-30T13:15:52.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ac6a9c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1ac00c6b80 0x7f1ac00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:52.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ac6a9c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1ac00c6f60 0x7f1ac00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:52.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ac6a9c640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac00c0a50 con 0x7f1ac00c8430 2023-10-30T13:15:52.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ac6a9c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac006f6e0 con 0x7f1ac00c6f60 2023-10-30T13:15:52.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ac6a9c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac006f9c0 con 0x7f1ac00c6b80 2023-10-30T13:15:52.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ab7fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1ac00c6f60 0x7f1ac00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:52.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ab7fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1ac00c6f60 0x7f1ac00c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45432/0 (socket says 172.21.15.125:45432) 2023-10-30T13:15:52.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ab7fff640 1 -- 172.21.15.125:0/2150790319 learned_addr learned my addr 172.21.15.125:0/2150790319 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:52.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ac4838640 1 --2- 172.21.15.125:0/2150790319 >> v2:172.21.15.136:3300/0 conn(0x7f1ac00c6b80 0x7f1ac00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:52.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ab7fff640 1 -- 172.21.15.125:0/2150790319 >> v2:172.21.15.136:3300/0 conn(0x7f1ac00c6b80 msgr2=0x7f1ac00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:52.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ab7fff640 1 --2- 172.21.15.125:0/2150790319 >> v2:172.21.15.136:3300/0 conn(0x7f1ac00c6b80 0x7f1ac00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:52.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ab7fff640 1 -- 172.21.15.125:0/2150790319 >> v1:172.21.15.125:6789/0 conn(0x7f1ac00c8430 legacy=0x7f1ac013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:52.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.415+0000 7f1ab7fff640 1 -- 172.21.15.125:0/2150790319 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac006fca0 con 0x7f1ac00c6f60 2023-10-30T13:15:52.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.419+0000 7f1ab7fff640 1 --2- 172.21.15.125:0/2150790319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ac00c6f60 0x7f1ac00c0400 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f1ac006f1b0 tx=0x7f1ab001ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b070ebfe496a353e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:52.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.419+0000 7f1ab77fe640 1 -- 172.21.15.125:0/2150790319 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00a7bb0 con 0x7f1ac00c6f60 2023-10-30T13:15:52.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.419+0000 7f1ab77fe640 1 -- 172.21.15.125:0/2150790319 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab00a7d10 con 0x7f1ac00c6f60 2023-10-30T13:15:52.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.419+0000 7f1ab77fe640 1 -- 172.21.15.125:0/2150790319 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00af660 con 0x7f1ac00c6f60 2023-10-30T13:15:52.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.419+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2150790319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ac00c6f60 msgr2=0x7f1ac00c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:52.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.419+0000 7f1ac6a9c640 1 --2- 172.21.15.125:0/2150790319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ac00c6f60 0x7f1ac00c0400 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f1ac006f1b0 tx=0x7f1ab001ebd0 comp rx=0 tx=0).stop 2023-10-30T13:15:52.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.419+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2150790319 shutdown_connections 2023-10-30T13:15:52.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.419+0000 7f1ac6a9c640 1 --2- 172.21.15.125:0/2150790319 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ac00c6f60 0x7f1ac00c0400 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:52.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.419+0000 7f1ac6a9c640 1 --2- 172.21.15.125:0/2150790319 >> v2:172.21.15.136:3300/0 conn(0x7f1ac00c6b80 0x7f1ac00bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:52.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.419+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2150790319 >> 172.21.15.125:0/2150790319 conn(0x7f1ac00c7910 msgr2=0x7f1ac00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:52.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.419+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2150790319 shutdown_connections 2023-10-30T13:15:52.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.419+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2150790319 wait complete. 2023-10-30T13:15:52.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ac6a9c640 1 Processor -- start 2023-10-30T13:15:52.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ac6a9c640 1 -- start start 2023-10-30T13:15:52.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ac6a9c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ac00c6b80 0x7f1ac00c3100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:52.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ac6a9c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ac00c6f60 0x7f1ac00c3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:52.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ac6a9c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ac00c8430 0x7f1ac0138500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:52.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ac6a9c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0154da0 con 0x7f1ac00c6f60 2023-10-30T13:15:52.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ac6a9c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac00c2d30 con 0x7f1ac00c6b80 2023-10-30T13:15:52.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ac6a9c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac0154fd0 con 0x7f1ac00c8430 2023-10-30T13:15:52.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ab7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ac00c6f60 0x7f1ac00c3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:52.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ab7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ac00c6f60 0x7f1ac00c3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45446/0 (socket says 172.21.15.125:45446) 2023-10-30T13:15:52.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ab7fff640 1 -- 172.21.15.125:0/2970998073 learned_addr learned my addr 172.21.15.125:0/2970998073 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:52.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ac4838640 1 --2- 172.21.15.125:0/2970998073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ac00c6b80 0x7f1ac00c3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:52.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ac5039640 1 --2- 172.21.15.125:0/2970998073 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ac00c8430 0x7f1ac0138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:52.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ab7fff640 1 -- 172.21.15.125:0/2970998073 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ac00c8430 msgr2=0x7f1ac0138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:52.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ab7fff640 1 --2- 172.21.15.125:0/2970998073 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ac00c8430 0x7f1ac0138500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:52.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ab7fff640 1 -- 172.21.15.125:0/2970998073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ac00c6b80 msgr2=0x7f1ac00c3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:52.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ab7fff640 1 --2- 172.21.15.125:0/2970998073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ac00c6b80 0x7f1ac00c3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:52.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ab7fff640 1 -- 172.21.15.125:0/2970998073 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab000b770 con 0x7f1ac00c6f60 2023-10-30T13:15:52.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ab7fff640 1 --2- 172.21.15.125:0/2970998073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ac00c6f60 0x7f1ac00c3640 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f1ab001b7e0 tx=0x7f1ab001b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:52.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ab57fa640 1 -- 172.21.15.125:0/2970998073 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00ba660 con 0x7f1ac00c6f60 2023-10-30T13:15:52.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ab57fa640 1 -- 172.21.15.125:0/2970998073 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab00ba7c0 con 0x7f1ac00c6f60 2023-10-30T13:15:52.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2970998073 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac01552b0 con 0x7f1ac00c6f60 2023-10-30T13:15:52.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ab57fa640 1 -- 172.21.15.125:0/2970998073 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00bab70 con 0x7f1ac00c6f60 2023-10-30T13:15:52.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.423+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2970998073 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac0155870 con 0x7f1ac00c6f60 2023-10-30T13:15:52.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.427+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2970998073 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac004e170 con 0x7f1ac00c6f60 2023-10-30T13:15:52.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.427+0000 7f1ab57fa640 1 -- 172.21.15.125:0/2970998073 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1ab00c33b0 con 0x7f1ac00c6f60 2023-10-30T13:15:52.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.427+0000 7f1ab57fa640 1 --2- 172.21.15.125:0/2970998073 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1aac0664a0 0x7f1aac068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:52.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.427+0000 7f1ab57fa640 1 -- 172.21.15.125:0/2970998073 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7f1ab011f8b0 con 0x7f1ac00c6f60 2023-10-30T13:15:52.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.427+0000 7f1ac4838640 1 --2- 172.21.15.125:0/2970998073 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1aac0664a0 0x7f1aac068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:52.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.427+0000 7f1ac4838640 1 --2- 172.21.15.125:0/2970998073 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1aac0664a0 0x7f1aac068960 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00b9d0 tx=0x7f1abc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:52.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.431+0000 7f1ab57fa640 1 -- 172.21.15.125:0/2970998073 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1ab00ec320 con 0x7f1ac00c6f60 2023-10-30T13:15:52.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.611+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2970998073 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ac00c1010 con 0x7f1ac00c6f60 2023-10-30T13:15:52.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.611+0000 7f1ab57fa640 1 -- 172.21.15.125:0/2970998073 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6551 (secure 0 0 0) 0x7f1ab00bf070 con 0x7f1ac00c6f60 2023-10-30T13:15:52.615 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:52.615 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8833,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:44.822534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:52.616 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 221 2023-10-30T13:15:52.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.611+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2970998073 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1aac0664a0 msgr2=0x7f1aac068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:52.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.611+0000 7f1ac6a9c640 1 --2- 172.21.15.125:0/2970998073 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1aac0664a0 0x7f1aac068960 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f1abc00b9d0 tx=0x7f1abc008040 comp rx=0 tx=0).stop 2023-10-30T13:15:52.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.611+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2970998073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ac00c6f60 msgr2=0x7f1ac00c3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:52.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.611+0000 7f1ac6a9c640 1 --2- 172.21.15.125:0/2970998073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ac00c6f60 0x7f1ac00c3640 secure :-1 s=READY pgs=1880 cs=0 l=1 rev1=1 crypto rx=0x7f1ab001b7e0 tx=0x7f1ab001b970 comp rx=0 tx=0).stop 2023-10-30T13:15:52.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.615+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2970998073 shutdown_connections 2023-10-30T13:15:52.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.615+0000 7f1ac6a9c640 1 --2- 172.21.15.125:0/2970998073 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1aac0664a0 0x7f1aac068960 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:52.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.615+0000 7f1ac6a9c640 1 --2- 172.21.15.125:0/2970998073 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1ac00c8430 0x7f1ac0138500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:52.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.615+0000 7f1ac6a9c640 1 --2- 172.21.15.125:0/2970998073 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1ac00c6f60 0x7f1ac00c3640 unknown :-1 s=CLOSED pgs=1880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:52.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.615+0000 7f1ac6a9c640 1 --2- 172.21.15.125:0/2970998073 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1ac00c6b80 0x7f1ac00c3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:52.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.615+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2970998073 >> 172.21.15.125:0/2970998073 conn(0x7f1ac00c7910 msgr2=0x7f1ac0069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:52.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.615+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2970998073 shutdown_connections 2023-10-30T13:15:52.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:52.615+0000 7f1ac6a9c640 1 -- 172.21.15.125:0/2970998073 wait complete. 2023-10-30T13:15:52.637 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:15:54.637 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:54.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.779+0000 7ff564965640 1 Processor -- start 2023-10-30T13:15:54.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.779+0000 7ff564965640 1 -- start start 2023-10-30T13:15:54.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.783+0000 7ff564965640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff5600c05d0 0x7ff56013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:54.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.783+0000 7ff564965640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff5600c6f60 0x7ff56013f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:54.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.783+0000 7ff564965640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5600c0a50 con 0x7ff5600c6b80 2023-10-30T13:15:54.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.783+0000 7ff564965640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56006f6e0 con 0x7ff5600c6f60 2023-10-30T13:15:54.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.783+0000 7ff564965640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff56006f9c0 con 0x7ff5600c05d0 2023-10-30T13:15:54.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.783+0000 7ff55ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff5600c6f60 0x7ff56013f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:54.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.783+0000 7ff55ed9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff5600c6f60 0x7ff56013f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45454/0 (socket says 172.21.15.125:45454) 2023-10-30T13:15:54.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.783+0000 7ff55ed9d640 1 -- 172.21.15.125:0/2018001131 learned_addr learned my addr 172.21.15.125:0/2018001131 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:54.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.783+0000 7ff555d9b640 1 --2- 172.21.15.125:0/2018001131 >> v2:172.21.15.136:3300/0 conn(0x7ff5600c05d0 0x7ff56013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:54.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.783+0000 7ff55ed9d640 1 -- 172.21.15.125:0/2018001131 >> v2:172.21.15.136:3300/0 conn(0x7ff5600c05d0 msgr2=0x7ff56013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:54.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.783+0000 7ff55ed9d640 1 --2- 172.21.15.125:0/2018001131 >> v2:172.21.15.136:3300/0 conn(0x7ff5600c05d0 0x7ff56013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:54.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.783+0000 7ff55ed9d640 1 -- 172.21.15.125:0/2018001131 >> v1:172.21.15.136:6789/0 conn(0x7ff5600c6b80 legacy=0x7ff5600bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:54.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.783+0000 7ff55ed9d640 1 -- 172.21.15.125:0/2018001131 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff56006fca0 con 0x7ff5600c6f60 2023-10-30T13:15:54.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.787+0000 7ff55ed9d640 1 --2- 172.21.15.125:0/2018001131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5600c6f60 0x7ff56013f2d0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7ff56006c7e0 tx=0x7ff55801c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6d8b050fce5fee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:54.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.787+0000 7ff55dd9b640 1 -- 172.21.15.125:0/2018001131 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5580ab610 con 0x7ff5600c6f60 2023-10-30T13:15:54.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.787+0000 7ff55dd9b640 1 -- 172.21.15.125:0/2018001131 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5580ab770 con 0x7ff5600c6f60 2023-10-30T13:15:54.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.787+0000 7ff55dd9b640 1 -- 172.21.15.125:0/2018001131 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5580abb00 con 0x7ff5600c6f60 2023-10-30T13:15:54.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.787+0000 7ff564965640 1 -- 172.21.15.125:0/2018001131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5600c6f60 msgr2=0x7ff56013f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:54.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.787+0000 7ff564965640 1 --2- 172.21.15.125:0/2018001131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5600c6f60 0x7ff56013f2d0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7ff56006c7e0 tx=0x7ff55801c500 comp rx=0 tx=0).stop 2023-10-30T13:15:54.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.787+0000 7ff564965640 1 -- 172.21.15.125:0/2018001131 shutdown_connections 2023-10-30T13:15:54.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.787+0000 7ff564965640 1 --2- 172.21.15.125:0/2018001131 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5600c6f60 0x7ff56013f2d0 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:54.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.787+0000 7ff564965640 1 --2- 172.21.15.125:0/2018001131 >> v2:172.21.15.136:3300/0 conn(0x7ff5600c05d0 0x7ff56013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:54.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.787+0000 7ff564965640 1 -- 172.21.15.125:0/2018001131 >> 172.21.15.125:0/2018001131 conn(0x7ff5600c7910 msgr2=0x7ff5600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:54.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.787+0000 7ff564965640 1 -- 172.21.15.125:0/2018001131 shutdown_connections 2023-10-30T13:15:54.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.787+0000 7ff564965640 1 -- 172.21.15.125:0/2018001131 wait complete. 2023-10-30T13:15:54.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.787+0000 7ff564965640 1 Processor -- start 2023-10-30T13:15:54.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff564965640 1 -- start start 2023-10-30T13:15:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff564965640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5600c05d0 0x7ff5601f2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:54.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff564965640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5600c6b80 0x7ff560150950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:54.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff564965640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5600c6f60 0x7ff5601514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:54.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff564965640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5601f2c50 con 0x7ff5600c05d0 2023-10-30T13:15:54.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff564965640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff56006a250 con 0x7ff5600c6b80 2023-10-30T13:15:54.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff564965640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff56006a4f0 con 0x7ff5600c6f60 2023-10-30T13:15:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff55e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5600c05d0 0x7ff5601f2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff55e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5600c05d0 0x7ff5601f2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45470/0 (socket says 172.21.15.125:45470) 2023-10-30T13:15:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff55e59c640 1 -- 172.21.15.125:0/3103364317 learned_addr learned my addr 172.21.15.125:0/3103364317 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff555d9b640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5600c6b80 0x7ff560150950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:54.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff55ed9d640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5600c6f60 0x7ff5601514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff55e59c640 1 -- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5600c6f60 msgr2=0x7ff5601514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff55e59c640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5600c6f60 0x7ff5601514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff55e59c640 1 -- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5600c6b80 msgr2=0x7ff560150950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff55e59c640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5600c6b80 0x7ff560150950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:54.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff55e59c640 1 -- 172.21.15.125:0/3103364317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5580094d0 con 0x7ff5600c05d0 2023-10-30T13:15:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff55ed9d640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5600c6f60 0x7ff5601514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff55e59c640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5600c05d0 0x7ff5601f2710 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7ff55000c500 tx=0x7ff55000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff5577fe640 1 -- 172.21.15.125:0/3103364317 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550005220 con 0x7ff5600c05d0 2023-10-30T13:15:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff5577fe640 1 -- 172.21.15.125:0/3103364317 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff550005380 con 0x7ff5600c05d0 2023-10-30T13:15:54.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff564965640 1 -- 172.21.15.125:0/3103364317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5600c8db0 con 0x7ff5600c05d0 2023-10-30T13:15:54.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff5577fe640 1 -- 172.21.15.125:0/3103364317 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550007270 con 0x7ff5600c05d0 2023-10-30T13:15:54.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff564965640 1 -- 172.21.15.125:0/3103364317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5600c9400 con 0x7ff5600c05d0 2023-10-30T13:15:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.791+0000 7ff564965640 1 -- 172.21.15.125:0/3103364317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5600c8c10 con 0x7ff5600c05d0 2023-10-30T13:15:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.795+0000 7ff5577fe640 1 -- 172.21.15.125:0/3103364317 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5500073d0 con 0x7ff5600c05d0 2023-10-30T13:15:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.795+0000 7ff5577fe640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5400664a0 0x7ff540068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:54.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.795+0000 7ff5577fe640 1 -- 172.21.15.125:0/3103364317 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7ff550083fe0 con 0x7ff5600c05d0 2023-10-30T13:15:54.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.795+0000 7ff555d9b640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5400664a0 0x7ff540068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:54.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.795+0000 7ff555d9b640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5400664a0 0x7ff540068960 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ff5600c7ec0 tx=0x7ff54800b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:54.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.799+0000 7ff5577fe640 1 -- 172.21.15.125:0/3103364317 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5600c8c10 con 0x7ff5600c05d0 2023-10-30T13:15:54.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.983+0000 7ff564965640 1 -- 172.21.15.125:0/3103364317 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff560073ea0 con 0x7ff5600c05d0 2023-10-30T13:15:54.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.983+0000 7ff5577fe640 1 -- 172.21.15.125:0/3103364317 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6551 (secure 0 0 0) 0x7ff550050b70 con 0x7ff5600c05d0 2023-10-30T13:15:54.988 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:54.988 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8833,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:44.822534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:54.988 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 221 2023-10-30T13:15:54.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.987+0000 7ff564965640 1 -- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5400664a0 msgr2=0x7ff540068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:54.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.987+0000 7ff564965640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5400664a0 0x7ff540068960 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ff5600c7ec0 tx=0x7ff54800b040 comp rx=0 tx=0).stop 2023-10-30T13:15:54.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.987+0000 7ff564965640 1 -- 172.21.15.125:0/3103364317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5600c05d0 msgr2=0x7ff5601f2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:54.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.987+0000 7ff564965640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5600c05d0 0x7ff5601f2710 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7ff55000c500 tx=0x7ff55000c9c0 comp rx=0 tx=0).stop 2023-10-30T13:15:54.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.987+0000 7ff564965640 1 -- 172.21.15.125:0/3103364317 shutdown_connections 2023-10-30T13:15:54.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.987+0000 7ff564965640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff5400664a0 0x7ff540068960 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:54.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.987+0000 7ff564965640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff5600c6f60 0x7ff5601514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:54.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.987+0000 7ff564965640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5600c6b80 0x7ff560150950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:54.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.987+0000 7ff564965640 1 --2- 172.21.15.125:0/3103364317 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5600c05d0 0x7ff5601f2710 unknown :-1 s=CLOSED pgs=1882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:54.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.987+0000 7ff564965640 1 -- 172.21.15.125:0/3103364317 >> 172.21.15.125:0/3103364317 conn(0x7ff5600c7910 msgr2=0x7ff5601392c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:54.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.987+0000 7ff564965640 1 -- 172.21.15.125:0/3103364317 shutdown_connections 2023-10-30T13:15:54.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:54.987+0000 7ff564965640 1 -- 172.21.15.125:0/3103364317 wait complete. 2023-10-30T13:15:55.009 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:15:57.010 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:57.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.155+0000 7f5e30646640 1 Processor -- start 2023-10-30T13:15:57.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.155+0000 7f5e30646640 1 -- start start 2023-10-30T13:15:57.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e30646640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e281538a0 0x7f5e2815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:57.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e30646640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e280cb800 con 0x7f5e280ca0e0 2023-10-30T13:15:57.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e30646640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e2815dc30 con 0x7f5e280cf160 2023-10-30T13:15:57.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e30646640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e280cb330 con 0x7f5e281538a0 2023-10-30T13:15:57.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2dbe1640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f5e280ca0e0 0x7f5e2815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59258/0 (socket says 172.21.15.125:59258) 2023-10-30T13:15:57.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2dbe1640 1 -- 172.21.15.125:0/3761279789 learned_addr learned my addr 172.21.15.125:0/3761279789 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:15:57.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2e3e2640 1 --2- 172.21.15.125:0/3761279789 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e281538a0 0x7f5e2815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:57.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 -- 172.21.15.125:0/3761279789 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461102481 0 0) 0x7f5e280cb800 con 0x7f5e280ca0e0 2023-10-30T13:15:57.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 -- 172.21.15.125:0/3761279789 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e14003530 con 0x7f5e280ca0e0 2023-10-30T13:15:57.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 -- 172.21.15.125:0/3761279789 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2873538890 0 0) 0x7f5e14003530 con 0x7f5e280ca0e0 2023-10-30T13:15:57.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 -- 172.21.15.125:0/3761279789 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e281538a0 msgr2=0x7f5e2815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:57.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 --2- 172.21.15.125:0/3761279789 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e281538a0 0x7f5e2815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:57.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 -- 172.21.15.125:0/3761279789 >> v1:172.21.15.136:6789/0 conn(0x7f5e280cf160 legacy=0x7f5e28164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:57.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 -- 172.21.15.125:0/3761279789 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e280cad70 con 0x7f5e280ca0e0 2023-10-30T13:15:57.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 -- 172.21.15.125:0/3761279789 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f5e18001680 con 0x7f5e280ca0e0 2023-10-30T13:15:57.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 -- 172.21.15.125:0/3761279789 >> v1:172.21.15.125:6789/0 conn(0x7f5e280ca0e0 legacy=0x7f5e2815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:57.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 --2- 172.21.15.125:0/3761279789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e140039b0 0x7f5e14003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:57.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 --2- 172.21.15.125:0/3761279789 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e281538a0 0x7f5e140047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:57.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 --2- 172.21.15.125:0/3761279789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e14004d10 0x7f5e14005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:57.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 -- 172.21.15.125:0/3761279789 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e280cb330 con 0x7f5e14004d10 2023-10-30T13:15:57.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 -- 172.21.15.125:0/3761279789 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1400bad0 con 0x7f5e140039b0 2023-10-30T13:15:57.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2d3e0640 1 -- 172.21.15.125:0/3761279789 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1400bc90 con 0x7f5e281538a0 2023-10-30T13:15:57.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2ebe3640 1 --2- 172.21.15.125:0/3761279789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e14004d10 0x7f5e14005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:57.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2dbe1640 1 --2- 172.21.15.125:0/3761279789 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e281538a0 0x7f5e140047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:57.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2e3e2640 1 --2- 172.21.15.125:0/3761279789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e140039b0 0x7f5e14003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:57.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2ebe3640 1 -- 172.21.15.125:0/3761279789 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e281538a0 msgr2=0x7f5e140047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:57.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2ebe3640 1 --2- 172.21.15.125:0/3761279789 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e281538a0 0x7f5e140047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:57.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2ebe3640 1 -- 172.21.15.125:0/3761279789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e140039b0 msgr2=0x7f5e14003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:15:57.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2ebe3640 1 --2- 172.21.15.125:0/3761279789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e140039b0 0x7f5e14003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:57.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2ebe3640 1 -- 172.21.15.125:0/3761279789 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e14003530 con 0x7f5e14004d10 2023-10-30T13:15:57.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2e3e2640 1 --2- 172.21.15.125:0/3761279789 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e140039b0 0x7f5e14003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:57.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.159+0000 7f5e2dbe1640 1 --2- 172.21.15.125:0/3761279789 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e281538a0 0x7f5e140047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:15:57.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.163+0000 7f5e2ebe3640 1 --2- 172.21.15.125:0/3761279789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e14004d10 0x7f5e14005190 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f5e20001ca0 tx=0x7f5e2001c5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7b1ff7a0f330710 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:57.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.163+0000 7f5e2d3e0640 1 -- 172.21.15.125:0/3761279789 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e20003b80 con 0x7f5e14004d10 2023-10-30T13:15:57.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.163+0000 7f5e2d3e0640 1 -- 172.21.15.125:0/3761279789 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e20003ce0 con 0x7f5e14004d10 2023-10-30T13:15:57.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.163+0000 7f5e2d3e0640 1 -- 172.21.15.125:0/3761279789 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e20005310 con 0x7f5e14004d10 2023-10-30T13:15:57.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.163+0000 7f5e30646640 1 -- 172.21.15.125:0/3761279789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e14004d10 msgr2=0x7f5e14005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:57.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.163+0000 7f5e30646640 1 --2- 172.21.15.125:0/3761279789 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e14004d10 0x7f5e14005190 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f5e20001ca0 tx=0x7f5e2001c5a0 comp rx=0 tx=0).stop 2023-10-30T13:15:57.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.163+0000 7f5e2ebe3640 1 -- 172.21.15.125:0/3761279789 reap_dead start 2023-10-30T13:15:57.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.163+0000 7f5e30646640 1 -- 172.21.15.125:0/3761279789 shutdown_connections 2023-10-30T13:15:57.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.163+0000 7f5e30646640 1 -- 172.21.15.125:0/3761279789 >> 172.21.15.125:0/3761279789 conn(0x7f5e2806bfe0 msgr2=0x7f5e2806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:57.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.163+0000 7f5e30646640 1 -- 172.21.15.125:0/3761279789 shutdown_connections 2023-10-30T13:15:57.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.163+0000 7f5e30646640 1 -- 172.21.15.125:0/3761279789 wait complete. 2023-10-30T13:15:57.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e30646640 1 Processor -- start 2023-10-30T13:15:57.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e30646640 1 -- start start 2023-10-30T13:15:57.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e30646640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e14004d10 0x7f5e280c57b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:57.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e30646640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2814d490 0x7f5e28157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:57.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e30646640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e281f2490 0x7f5e281f2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:57.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e30646640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e280b4c90 con 0x7f5e2814d490 2023-10-30T13:15:57.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e30646640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e28158f10 con 0x7f5e281f2490 2023-10-30T13:15:57.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e30646640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e280b3ec0 con 0x7f5e14004d10 2023-10-30T13:15:57.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e2dbe1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2814d490 0x7f5e28157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:57.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e2dbe1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2814d490 0x7f5e28157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45490/0 (socket says 172.21.15.125:45490) 2023-10-30T13:15:57.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e2dbe1640 1 -- 172.21.15.125:0/3363379503 learned_addr learned my addr 172.21.15.125:0/3363379503 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:57.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e2e3e2640 1 --2- 172.21.15.125:0/3363379503 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e14004d10 0x7f5e280c57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:57.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e2ebe3640 1 --2- 172.21.15.125:0/3363379503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e281f2490 0x7f5e281f2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:57.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e2dbe1640 1 -- 172.21.15.125:0/3363379503 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e14004d10 msgr2=0x7f5e280c57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:57.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e2dbe1640 1 --2- 172.21.15.125:0/3363379503 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e14004d10 0x7f5e280c57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:57.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e2dbe1640 1 -- 172.21.15.125:0/3363379503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e281f2490 msgr2=0x7f5e281f2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:57.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e2dbe1640 1 --2- 172.21.15.125:0/3363379503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e281f2490 0x7f5e281f2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:57.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e2dbe1640 1 -- 172.21.15.125:0/3363379503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e280b43b0 con 0x7f5e2814d490 2023-10-30T13:15:57.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e2dbe1640 1 --2- 172.21.15.125:0/3363379503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2814d490 0x7f5e28157520 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f5e18001b50 tx=0x7f5e1800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:57.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e1effd640 1 -- 172.21.15.125:0/3363379503 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1800c870 con 0x7f5e2814d490 2023-10-30T13:15:57.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e1effd640 1 -- 172.21.15.125:0/3363379503 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e1800c9d0 con 0x7f5e2814d490 2023-10-30T13:15:57.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e1effd640 1 -- 172.21.15.125:0/3363379503 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e1800cd40 con 0x7f5e2814d490 2023-10-30T13:15:57.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e30646640 1 -- 172.21.15.125:0/3363379503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e280b3640 con 0x7f5e2814d490 2023-10-30T13:15:57.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e30646640 1 -- 172.21.15.125:0/3363379503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e280b40d0 con 0x7f5e2814d490 2023-10-30T13:15:57.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.167+0000 7f5e30646640 1 -- 172.21.15.125:0/3363379503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e280b37d0 con 0x7f5e2814d490 2023-10-30T13:15:57.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.171+0000 7f5e1effd640 1 -- 172.21.15.125:0/3363379503 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5e1801d460 con 0x7f5e2814d490 2023-10-30T13:15:57.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.171+0000 7f5e1effd640 1 --2- 172.21.15.125:0/3363379503 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e0c0664a0 0x7f5e0c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:57.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.171+0000 7f5e1effd640 1 -- 172.21.15.125:0/3363379503 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7f5e18084920 con 0x7f5e2814d490 2023-10-30T13:15:57.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.171+0000 7f5e2e3e2640 1 --2- 172.21.15.125:0/3363379503 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e0c0664a0 0x7f5e0c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:57.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.171+0000 7f5e2e3e2640 1 --2- 172.21.15.125:0/3363379503 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e0c0664a0 0x7f5e0c068960 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f5e240099b0 tx=0x7f5e24008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:57.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.175+0000 7f5e1effd640 1 -- 172.21.15.125:0/3363379503 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5e18051420 con 0x7f5e2814d490 2023-10-30T13:15:57.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.359+0000 7f5e30646640 1 -- 172.21.15.125:0/3363379503 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e2804e170 con 0x7f5e2814d490 2023-10-30T13:15:57.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.359+0000 7f5e1effd640 1 -- 172.21.15.125:0/3363379503 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6551 (secure 0 0 0) 0x7f5e18054a40 con 0x7f5e2814d490 2023-10-30T13:15:57.362 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:57.362 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8833,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:44.822534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:57.363 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 221 2023-10-30T13:15:57.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.359+0000 7f5e30646640 1 -- 172.21.15.125:0/3363379503 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e0c0664a0 msgr2=0x7f5e0c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:57.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.359+0000 7f5e30646640 1 --2- 172.21.15.125:0/3363379503 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e0c0664a0 0x7f5e0c068960 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f5e240099b0 tx=0x7f5e24008040 comp rx=0 tx=0).stop 2023-10-30T13:15:57.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.359+0000 7f5e30646640 1 -- 172.21.15.125:0/3363379503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2814d490 msgr2=0x7f5e28157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:57.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.359+0000 7f5e30646640 1 --2- 172.21.15.125:0/3363379503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2814d490 0x7f5e28157520 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7f5e18001b50 tx=0x7f5e1800ece0 comp rx=0 tx=0).stop 2023-10-30T13:15:57.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.359+0000 7f5e30646640 1 -- 172.21.15.125:0/3363379503 shutdown_connections 2023-10-30T13:15:57.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.359+0000 7f5e30646640 1 --2- 172.21.15.125:0/3363379503 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5e0c0664a0 0x7f5e0c068960 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:57.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.359+0000 7f5e30646640 1 --2- 172.21.15.125:0/3363379503 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5e281f2490 0x7f5e281f2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:57.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.359+0000 7f5e30646640 1 --2- 172.21.15.125:0/3363379503 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5e2814d490 0x7f5e28157520 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:57.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.359+0000 7f5e30646640 1 --2- 172.21.15.125:0/3363379503 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5e14004d10 0x7f5e280c57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:57.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.359+0000 7f5e30646640 1 -- 172.21.15.125:0/3363379503 >> 172.21.15.125:0/3363379503 conn(0x7f5e2806bfe0 msgr2=0x7f5e281492b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:57.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.359+0000 7f5e30646640 1 -- 172.21.15.125:0/3363379503 shutdown_connections 2023-10-30T13:15:57.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:57.359+0000 7f5e30646640 1 -- 172.21.15.125:0/3363379503 wait complete. 2023-10-30T13:15:57.383 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:15:59.384 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:15:59.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.527+0000 7f4a561b8640 1 Processor -- start 2023-10-30T13:15:59.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.527+0000 7f4a561b8640 1 -- start start 2023-10-30T13:15:59.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.527+0000 7f4a561b8640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4a500c6bd0 0x7f4a5006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:59.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.527+0000 7f4a561b8640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a50066cf0 con 0x7f4a50071110 2023-10-30T13:15:59.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.527+0000 7f4a561b8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a500c70d0 con 0x7f4a5006f7a0 2023-10-30T13:15:59.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.527+0000 7f4a561b8640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a500700c0 con 0x7f4a500c6bd0 2023-10-30T13:15:59.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.527+0000 7f4a47fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4a500c6bd0 0x7f4a5006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:59.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.527+0000 7f4a47fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f4a500c6bd0 0x7f4a5006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45500/0 (socket says 172.21.15.125:45500) 2023-10-30T13:15:59.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.527+0000 7f4a47fff640 1 -- 172.21.15.125:0/3488171949 learned_addr learned my addr 172.21.15.125:0/3488171949 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:59.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.527+0000 7f4a4ffff640 1 --1- 172.21.15.125:0/3488171949 >> v1:172.21.15.125:6789/0 conn(0x7f4a50071110 0x7f4a50158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59268/0 (socket says 172.21.15.125:59268) 2023-10-30T13:15:59.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.531+0000 7f4a47fff640 1 -- 172.21.15.125:0/3488171949 >> v1:172.21.15.136:6789/0 conn(0x7f4a5006f7a0 legacy=0x7f4a50069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:59.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.531+0000 7f4a47fff640 1 -- 172.21.15.125:0/3488171949 >> v1:172.21.15.125:6789/0 conn(0x7f4a50071110 legacy=0x7f4a50158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:59.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.531+0000 7f4a47fff640 1 -- 172.21.15.125:0/3488171949 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a500c7390 con 0x7f4a500c6bd0 2023-10-30T13:15:59.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.531+0000 7f4a4effd640 1 -- 172.21.15.125:0/3488171949 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098751674 0 0) 0x7f4a50066cf0 con 0x7f4a50071110 2023-10-30T13:15:59.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.531+0000 7f4a47fff640 1 --2- 172.21.15.125:0/3488171949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a500c6bd0 0x7f4a5006fb80 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f4a38009870 tx=0x7f4a3801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=690917a603bc7bf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:59.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.531+0000 7f4a4effd640 1 -- 172.21.15.125:0/3488171949 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a380a8a30 con 0x7f4a500c6bd0 2023-10-30T13:15:59.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.531+0000 7f4a4effd640 1 -- 172.21.15.125:0/3488171949 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a380a8b90 con 0x7f4a500c6bd0 2023-10-30T13:15:59.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.531+0000 7f4a4effd640 1 -- 172.21.15.125:0/3488171949 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a380b0440 con 0x7f4a500c6bd0 2023-10-30T13:15:59.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.531+0000 7f4a561b8640 1 -- 172.21.15.125:0/3488171949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a500c6bd0 msgr2=0x7f4a5006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:59.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.531+0000 7f4a561b8640 1 --2- 172.21.15.125:0/3488171949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a500c6bd0 0x7f4a5006fb80 secure :-1 s=READY pgs=1887 cs=0 l=1 rev1=1 crypto rx=0x7f4a38009870 tx=0x7f4a3801ed30 comp rx=0 tx=0).stop 2023-10-30T13:15:59.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.531+0000 7f4a561b8640 1 -- 172.21.15.125:0/3488171949 shutdown_connections 2023-10-30T13:15:59.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.531+0000 7f4a561b8640 1 --2- 172.21.15.125:0/3488171949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a500c6bd0 0x7f4a5006fb80 unknown :-1 s=CLOSED pgs=1887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:59.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.531+0000 7f4a561b8640 1 -- 172.21.15.125:0/3488171949 >> 172.21.15.125:0/3488171949 conn(0x7f4a500c7910 msgr2=0x7f4a500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:59.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a561b8640 1 -- 172.21.15.125:0/3488171949 shutdown_connections 2023-10-30T13:15:59.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a561b8640 1 -- 172.21.15.125:0/3488171949 wait complete. 2023-10-30T13:15:59.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a561b8640 1 Processor -- start 2023-10-30T13:15:59.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a561b8640 1 -- start start 2023-10-30T13:15:59.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a561b8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a5006f7a0 0x7f4a501f3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:59.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a561b8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a50071110 0x7f4a501f6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:59.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a561b8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a500c6bd0 0x7f4a500c8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:59.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a561b8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a50158180 con 0x7f4a50071110 2023-10-30T13:15:59.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a561b8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5014ded0 con 0x7f4a500c6bd0 2023-10-30T13:15:59.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a561b8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5014dbd0 con 0x7f4a5006f7a0 2023-10-30T13:15:59.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a47fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a50071110 0x7f4a501f6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:59.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a47fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a50071110 0x7f4a501f6a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45516/0 (socket says 172.21.15.125:45516) 2023-10-30T13:15:59.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a47fff640 1 -- 172.21.15.125:0/2949355593 learned_addr learned my addr 172.21.15.125:0/2949355593 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:15:59.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a4ffff640 1 --2- 172.21.15.125:0/2949355593 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a500c6bd0 0x7f4a500c8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:59.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a4f7fe640 1 --2- 172.21.15.125:0/2949355593 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a5006f7a0 0x7f4a501f3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:59.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a47fff640 1 -- 172.21.15.125:0/2949355593 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a5006f7a0 msgr2=0x7f4a501f3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:59.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a47fff640 1 --2- 172.21.15.125:0/2949355593 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a5006f7a0 0x7f4a501f3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:59.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a47fff640 1 -- 172.21.15.125:0/2949355593 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a500c6bd0 msgr2=0x7f4a500c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:59.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a47fff640 1 --2- 172.21.15.125:0/2949355593 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a500c6bd0 0x7f4a500c8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:59.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a47fff640 1 -- 172.21.15.125:0/2949355593 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a380094d0 con 0x7f4a50071110 2023-10-30T13:15:59.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.535+0000 7f4a47fff640 1 --2- 172.21.15.125:0/2949355593 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a50071110 0x7f4a501f6a60 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f4a380b3040 tx=0x7f4a3801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:59.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.539+0000 7f4a4cff9640 1 -- 172.21.15.125:0/2949355593 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a380b9620 con 0x7f4a50071110 2023-10-30T13:15:59.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.539+0000 7f4a4cff9640 1 -- 172.21.15.125:0/2949355593 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a380b97c0 con 0x7f4a50071110 2023-10-30T13:15:59.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.539+0000 7f4a4cff9640 1 -- 172.21.15.125:0/2949355593 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a380b9b70 con 0x7f4a50071110 2023-10-30T13:15:59.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.539+0000 7f4a561b8640 1 -- 172.21.15.125:0/2949355593 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a501f7080 con 0x7f4a50071110 2023-10-30T13:15:59.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.539+0000 7f4a561b8640 1 -- 172.21.15.125:0/2949355593 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a501f7620 con 0x7f4a50071110 2023-10-30T13:15:59.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.539+0000 7f4a561b8640 1 -- 172.21.15.125:0/2949355593 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a500708f0 con 0x7f4a50071110 2023-10-30T13:15:59.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.539+0000 7f4a4cff9640 1 -- 172.21.15.125:0/2949355593 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4a380b7070 con 0x7f4a50071110 2023-10-30T13:15:59.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.539+0000 7f4a4cff9640 1 --2- 172.21.15.125:0/2949355593 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a300664a0 0x7f4a30068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:15:59.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.539+0000 7f4a4cff9640 1 -- 172.21.15.125:0/2949355593 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7974+0+0 (secure 0 0 0) 0x7f4a38122880 con 0x7f4a50071110 2023-10-30T13:15:59.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.539+0000 7f4a4f7fe640 1 --2- 172.21.15.125:0/2949355593 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a300664a0 0x7f4a30068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:15:59.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.543+0000 7f4a4f7fe640 1 --2- 172.21.15.125:0/2949355593 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a300664a0 0x7f4a30068960 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f4a40000f70 tx=0x7f4a40009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:15:59.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.543+0000 7f4a4cff9640 1 -- 172.21.15.125:0/2949355593 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4a500708f0 con 0x7f4a50071110 2023-10-30T13:15:59.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.723+0000 7f4a561b8640 1 -- 172.21.15.125:0/2949355593 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a500708f0 con 0x7f4a50071110 2023-10-30T13:15:59.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.723+0000 7f4a4cff9640 1 -- 172.21.15.125:0/2949355593 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 221 v221) v1 ==== 95+0+6551 (secure 0 0 0) 0x7f4a500708f0 con 0x7f4a50071110 2023-10-30T13:15:59.730 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:15:59.731 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":221,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8833,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":220}],"filesystems":[{"mdsmap":{"epoch":221,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:15:44.822534+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8443},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8443":{"gid":8443,"name":"f","rank":2,"incarnation":196,"state":"up:active","state_seq":41,"addr":"172.21.15.136:6839/3776186190","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":3776186190},{"type":"v1","addr":"172.21.15.136:6839","nonce":3776186190}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:15:59.732 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 221 2023-10-30T13:15:59.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.727+0000 7f4a561b8640 1 -- 172.21.15.125:0/2949355593 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a300664a0 msgr2=0x7f4a30068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:59.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.727+0000 7f4a561b8640 1 --2- 172.21.15.125:0/2949355593 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a300664a0 0x7f4a30068960 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f4a40000f70 tx=0x7f4a40009040 comp rx=0 tx=0).stop 2023-10-30T13:15:59.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.727+0000 7f4a561b8640 1 -- 172.21.15.125:0/2949355593 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a50071110 msgr2=0x7f4a501f6a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:15:59.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.727+0000 7f4a561b8640 1 --2- 172.21.15.125:0/2949355593 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a50071110 0x7f4a501f6a60 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f4a380b3040 tx=0x7f4a3801bad0 comp rx=0 tx=0).stop 2023-10-30T13:15:59.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.727+0000 7f4a561b8640 1 -- 172.21.15.125:0/2949355593 shutdown_connections 2023-10-30T13:15:59.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.727+0000 7f4a561b8640 1 --2- 172.21.15.125:0/2949355593 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4a300664a0 0x7f4a30068960 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:59.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.727+0000 7f4a561b8640 1 --2- 172.21.15.125:0/2949355593 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4a500c6bd0 0x7f4a500c8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:59.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.727+0000 7f4a561b8640 1 --2- 172.21.15.125:0/2949355593 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4a50071110 0x7f4a501f6a60 unknown :-1 s=CLOSED pgs=1888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:59.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.727+0000 7f4a561b8640 1 --2- 172.21.15.125:0/2949355593 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4a5006f7a0 0x7f4a501f3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:15:59.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.727+0000 7f4a561b8640 1 -- 172.21.15.125:0/2949355593 >> 172.21.15.125:0/2949355593 conn(0x7f4a500c7910 msgr2=0x7f4a500c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:15:59.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.727+0000 7f4a561b8640 1 -- 172.21.15.125:0/2949355593 shutdown_connections 2023-10-30T13:15:59.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:15:59.727+0000 7f4a561b8640 1 -- 172.21.15.125:0/2949355593 wait complete. 2023-10-30T13:15:59.750 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:16:01.751 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:01.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a1153e640 1 Processor -- start 2023-10-30T13:16:01.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a1153e640 1 -- start start 2023-10-30T13:16:01.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a1153e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2a0c15d6f0 0x7f2a0c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:01.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a1153e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a0c0cf160 0x7f2a0c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:01.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a1153e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a0c0cb800 con 0x7f2a0c1538a0 2023-10-30T13:16:01.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a1153e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0c0cb330 con 0x7f2a0c15d6f0 2023-10-30T13:16:01.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a1153e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c0cad70 con 0x7f2a0c0cf160 2023-10-30T13:16:01.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a0affd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2a0c1538a0 0x7f2a0c0ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:56854/0 (socket says 172.21.15.125:56854) 2023-10-30T13:16:01.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a0affd640 1 -- 172.21.15.125:0/4135072712 learned_addr learned my addr 172.21.15.125:0/4135072712 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:16:01.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a0b7fe640 1 --2- 172.21.15.125:0/4135072712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a0c0cf160 0x7f2a0c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:01.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a03fff640 1 --2- 172.21.15.125:0/4135072712 >> v2:172.21.15.136:3300/0 conn(0x7f2a0c15d6f0 0x7f2a0c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:01.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a0a7fc640 1 -- 172.21.15.125:0/4135072712 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3274795599 0 0) 0x7f2a0c0cb800 con 0x7f2a0c1538a0 2023-10-30T13:16:01.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a0a7fc640 1 -- 172.21.15.125:0/4135072712 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f8003530 con 0x7f2a0c1538a0 2023-10-30T13:16:01.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a0a7fc640 1 -- 172.21.15.125:0/4135072712 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2155243443 0 0) 0x7f29f8003530 con 0x7f2a0c1538a0 2023-10-30T13:16:01.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a0a7fc640 1 -- 172.21.15.125:0/4135072712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a0c0cf160 msgr2=0x7f2a0c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:01.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a0a7fc640 1 --2- 172.21.15.125:0/4135072712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a0c0cf160 0x7f2a0c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:01.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a0a7fc640 1 -- 172.21.15.125:0/4135072712 >> v2:172.21.15.136:3300/0 conn(0x7f2a0c15d6f0 msgr2=0x7f2a0c15a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:01.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a0a7fc640 1 --2- 172.21.15.125:0/4135072712 >> v2:172.21.15.136:3300/0 conn(0x7f2a0c15d6f0 0x7f2a0c15a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:01.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a0a7fc640 1 -- 172.21.15.125:0/4135072712 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c0cb050 con 0x7f2a0c1538a0 2023-10-30T13:16:01.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.895+0000 7f2a0a7fc640 1 -- 172.21.15.125:0/4135072712 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f29fc001680 con 0x7f2a0c1538a0 2023-10-30T13:16:01.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0a7fc640 1 -- 172.21.15.125:0/4135072712 >> v1:172.21.15.125:6789/0 conn(0x7f2a0c1538a0 legacy=0x7f2a0c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:01.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0a7fc640 1 --2- 172.21.15.125:0/4135072712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29f8003980 0x7f29f8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:01.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0a7fc640 1 --2- 172.21.15.125:0/4135072712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a0c0cf160 0x7f29f80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:01.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0a7fc640 1 --2- 172.21.15.125:0/4135072712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29f8004d10 0x7f29f8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:01.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0a7fc640 1 -- 172.21.15.125:0/4135072712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c0cb330 con 0x7f29f8004d10 2023-10-30T13:16:01.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0a7fc640 1 -- 172.21.15.125:0/4135072712 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c0cad70 con 0x7f29f8003980 2023-10-30T13:16:01.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0a7fc640 1 -- 172.21.15.125:0/4135072712 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f800b990 con 0x7f2a0c0cf160 2023-10-30T13:16:01.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0b7fe640 1 --2- 172.21.15.125:0/4135072712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29f8004d10 0x7f29f8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:01.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0affd640 1 --2- 172.21.15.125:0/4135072712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29f8003980 0x7f29f8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:01.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a03fff640 1 --2- 172.21.15.125:0/4135072712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a0c0cf160 0x7f29f80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:01.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0b7fe640 1 -- 172.21.15.125:0/4135072712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a0c0cf160 msgr2=0x7f29f80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:01.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0b7fe640 1 --2- 172.21.15.125:0/4135072712 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a0c0cf160 0x7f29f80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:01.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0b7fe640 1 -- 172.21.15.125:0/4135072712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29f8003980 msgr2=0x7f29f8003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:01.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0b7fe640 1 --2- 172.21.15.125:0/4135072712 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29f8003980 0x7f29f8003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:01.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0b7fe640 1 -- 172.21.15.125:0/4135072712 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f8003530 con 0x7f29f8004d10 2023-10-30T13:16:01.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.899+0000 7f2a0b7fe640 1 --2- 172.21.15.125:0/4135072712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29f8004d10 0x7f29f8005190 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f2a04009fc0 tx=0x7f2a0401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a13a9ae41836247a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:01.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a0a7fc640 1 -- 172.21.15.125:0/4135072712 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a040a5c70 con 0x7f29f8004d10 2023-10-30T13:16:01.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a0a7fc640 1 -- 172.21.15.125:0/4135072712 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0400f040 con 0x7f29f8004d10 2023-10-30T13:16:01.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a0a7fc640 1 -- 172.21.15.125:0/4135072712 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a040ad440 con 0x7f29f8004d10 2023-10-30T13:16:01.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a1153e640 1 -- 172.21.15.125:0/4135072712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29f8004d10 msgr2=0x7f29f8005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:01.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a1153e640 1 --2- 172.21.15.125:0/4135072712 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29f8004d10 0x7f29f8005190 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f2a04009fc0 tx=0x7f2a0401cc60 comp rx=0 tx=0).stop 2023-10-30T13:16:01.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a0b7fe640 1 -- 172.21.15.125:0/4135072712 reap_dead start 2023-10-30T13:16:01.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a1153e640 1 -- 172.21.15.125:0/4135072712 shutdown_connections 2023-10-30T13:16:01.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a1153e640 1 -- 172.21.15.125:0/4135072712 >> 172.21.15.125:0/4135072712 conn(0x7f2a0c06bfe0 msgr2=0x7f2a0c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:01.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a1153e640 1 -- 172.21.15.125:0/4135072712 shutdown_connections 2023-10-30T13:16:01.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a1153e640 1 -- 172.21.15.125:0/4135072712 wait complete. 2023-10-30T13:16:01.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a1153e640 1 Processor -- start 2023-10-30T13:16:01.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a1153e640 1 -- start start 2023-10-30T13:16:01.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a1153e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29f8004d10 0x7f2a0c1e4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:01.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a1153e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a0c14fd10 0x7f2a0c150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:01.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a1153e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a0c0c6a90 0x7f2a0c0c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:01.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a1153e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c150690 con 0x7f2a0c14fd10 2023-10-30T13:16:01.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a1153e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c0c7450 con 0x7f2a0c0c6a90 2023-10-30T13:16:01.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a1153e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c1e3540 con 0x7f29f8004d10 2023-10-30T13:16:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a03fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a0c14fd10 0x7f2a0c150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a03fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a0c14fd10 0x7f2a0c150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40128/0 (socket says 172.21.15.125:40128) 2023-10-30T13:16:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a03fff640 1 -- 172.21.15.125:0/1634159218 learned_addr learned my addr 172.21.15.125:0/1634159218 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a0affd640 1 --2- 172.21.15.125:0/1634159218 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29f8004d10 0x7f2a0c1e4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a0b7fe640 1 --2- 172.21.15.125:0/1634159218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a0c0c6a90 0x7f2a0c0c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:01.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.903+0000 7f2a03fff640 1 -- 172.21.15.125:0/1634159218 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29f8004d10 msgr2=0x7f2a0c1e4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:01.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a03fff640 1 --2- 172.21.15.125:0/1634159218 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29f8004d10 0x7f2a0c1e4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:01.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a03fff640 1 -- 172.21.15.125:0/1634159218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a0c0c6a90 msgr2=0x7f2a0c0c6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:01.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a03fff640 1 --2- 172.21.15.125:0/1634159218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a0c0c6a90 0x7f2a0c0c6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:01.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a03fff640 1 -- 172.21.15.125:0/1634159218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c1e3820 con 0x7f2a0c14fd10 2023-10-30T13:16:01.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a03fff640 1 --2- 172.21.15.125:0/1634159218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a0c14fd10 0x7f2a0c150150 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009fd0 tx=0x7f29f4011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:01.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a037fe640 1 -- 172.21.15.125:0/1634159218 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f400fbb0 con 0x7f2a0c14fd10 2023-10-30T13:16:01.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a037fe640 1 -- 172.21.15.125:0/1634159218 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f400fd10 con 0x7f2a0c14fd10 2023-10-30T13:16:01.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a1153e640 1 -- 172.21.15.125:0/1634159218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0c1e3b00 con 0x7f2a0c14fd10 2023-10-30T13:16:01.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a037fe640 1 -- 172.21.15.125:0/1634159218 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f4014660 con 0x7f2a0c14fd10 2023-10-30T13:16:01.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a1153e640 1 -- 172.21.15.125:0/1634159218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c1e40c0 con 0x7f2a0c14fd10 2023-10-30T13:16:01.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a1153e640 1 -- 172.21.15.125:0/1634159218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0c04e170 con 0x7f2a0c14fd10 2023-10-30T13:16:01.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a037fe640 1 -- 172.21.15.125:0/1634159218 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29f40147c0 con 0x7f2a0c14fd10 2023-10-30T13:16:01.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a037fe640 1 --2- 172.21.15.125:0/1634159218 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29ec0664a0 0x7f29ec068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:01.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a037fe640 1 -- 172.21.15.125:0/1634159218 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f29f4085210 con 0x7f2a0c14fd10 2023-10-30T13:16:01.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.907+0000 7f2a0affd640 1 --2- 172.21.15.125:0/1634159218 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29ec0664a0 0x7f29ec068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:01.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.911+0000 7f2a0affd640 1 --2- 172.21.15.125:0/1634159218 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29ec0664a0 0x7f29ec068960 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f29fc000c10 tx=0x7f29fc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:01.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:01.911+0000 7f2a037fe640 1 -- 172.21.15.125:0/1634159218 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29f4051d80 con 0x7f2a0c14fd10 2023-10-30T13:16:02.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.091+0000 7f2a1153e640 1 -- 172.21.15.125:0/1634159218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a0c0b1dd0 con 0x7f2a0c14fd10 2023-10-30T13:16:02.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.091+0000 7f2a037fe640 1 -- 172.21.15.125:0/1634159218 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f29f40553a0 con 0x7f2a0c14fd10 2023-10-30T13:16:02.097 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:16:02.097 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:16:00.781655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:02.097 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 222 2023-10-30T13:16:02.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.095+0000 7f2a1153e640 1 -- 172.21.15.125:0/1634159218 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29ec0664a0 msgr2=0x7f29ec068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:02.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.095+0000 7f2a1153e640 1 --2- 172.21.15.125:0/1634159218 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29ec0664a0 0x7f29ec068960 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f29fc000c10 tx=0x7f29fc009040 comp rx=0 tx=0).stop 2023-10-30T13:16:02.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.095+0000 7f2a1153e640 1 -- 172.21.15.125:0/1634159218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a0c14fd10 msgr2=0x7f2a0c150150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:02.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.095+0000 7f2a1153e640 1 --2- 172.21.15.125:0/1634159218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a0c14fd10 0x7f2a0c150150 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009fd0 tx=0x7f29f4011f00 comp rx=0 tx=0).stop 2023-10-30T13:16:02.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.095+0000 7f2a1153e640 1 -- 172.21.15.125:0/1634159218 shutdown_connections 2023-10-30T13:16:02.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.095+0000 7f2a1153e640 1 --2- 172.21.15.125:0/1634159218 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29ec0664a0 0x7f29ec068960 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:02.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.095+0000 7f2a1153e640 1 --2- 172.21.15.125:0/1634159218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a0c0c6a90 0x7f2a0c0c6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:02.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.095+0000 7f2a1153e640 1 --2- 172.21.15.125:0/1634159218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a0c14fd10 0x7f2a0c150150 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:02.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.095+0000 7f2a1153e640 1 --2- 172.21.15.125:0/1634159218 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29f8004d10 0x7f2a0c1e4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:02.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.095+0000 7f2a1153e640 1 -- 172.21.15.125:0/1634159218 >> 172.21.15.125:0/1634159218 conn(0x7f2a0c06bfe0 msgr2=0x7f2a0c0cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:02.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.095+0000 7f2a1153e640 1 -- 172.21.15.125:0/1634159218 shutdown_connections 2023-10-30T13:16:02.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.095+0000 7f2a1153e640 1 -- 172.21.15.125:0/1634159218 wait complete. 2023-10-30T13:16:02.119 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T13:16:02.119 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:16:02.119 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:02.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac61b62640 1 Processor -- start 2023-10-30T13:16:02.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac61b62640 1 -- start start 2023-10-30T13:16:02.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac61b62640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fac5c1538f0 0x7fac5c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:02.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac61b62640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac5c15cd50 0x7fac5c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:02.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac61b62640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5c15d300 con 0x7fac5c0cefa0 2023-10-30T13:16:02.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac61b62640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5c15d5e0 con 0x7fac5c1538f0 2023-10-30T13:16:02.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac61b62640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c15d8c0 con 0x7fac5c15cd50 2023-10-30T13:16:02.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac5bfff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac5c15cd50 0x7fac5c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:02.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac60b60640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fac5c1538f0 0x7fac5c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:02.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac5bfff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac5c15cd50 0x7fac5c0c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:43028/0 (socket says 172.21.15.125:43028) 2023-10-30T13:16:02.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac5bfff640 1 -- 172.21.15.125:0/4206421924 learned_addr learned my addr 172.21.15.125:0/4206421924 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:02.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac5bfff640 1 -- 172.21.15.125:0/4206421924 >> v2:172.21.15.136:3300/0 conn(0x7fac5c1538f0 msgr2=0x7fac5c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:02.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac5bfff640 1 --2- 172.21.15.125:0/4206421924 >> v2:172.21.15.136:3300/0 conn(0x7fac5c1538f0 0x7fac5c159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:02.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac5bfff640 1 -- 172.21.15.125:0/4206421924 >> v1:172.21.15.136:6789/0 conn(0x7fac5c0cefa0 legacy=0x7fac5c0cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:02.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.255+0000 7fac5bfff640 1 -- 172.21.15.125:0/4206421924 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5c15dba0 con 0x7fac5c15cd50 2023-10-30T13:16:02.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.259+0000 7fac5bfff640 1 --2- 172.21.15.125:0/4206421924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac5c15cd50 0x7fac5c0c9f90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fac48009870 tx=0x7fac4801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ebec30700f5f3f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:02.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.259+0000 7fac5b7fe640 1 -- 172.21.15.125:0/4206421924 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac480a8a30 con 0x7fac5c15cd50 2023-10-30T13:16:02.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.259+0000 7fac5b7fe640 1 -- 172.21.15.125:0/4206421924 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac480a8b90 con 0x7fac5c15cd50 2023-10-30T13:16:02.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.259+0000 7fac5b7fe640 1 -- 172.21.15.125:0/4206421924 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac480b0440 con 0x7fac5c15cd50 2023-10-30T13:16:02.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.259+0000 7fac61b62640 1 -- 172.21.15.125:0/4206421924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac5c15cd50 msgr2=0x7fac5c0c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:02.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.259+0000 7fac61b62640 1 --2- 172.21.15.125:0/4206421924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac5c15cd50 0x7fac5c0c9f90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fac48009870 tx=0x7fac4801ed30 comp rx=0 tx=0).stop 2023-10-30T13:16:02.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.259+0000 7fac61b62640 1 -- 172.21.15.125:0/4206421924 shutdown_connections 2023-10-30T13:16:02.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.259+0000 7fac61b62640 1 --2- 172.21.15.125:0/4206421924 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac5c15cd50 0x7fac5c0c9f90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:02.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.259+0000 7fac61b62640 1 --2- 172.21.15.125:0/4206421924 >> v2:172.21.15.136:3300/0 conn(0x7fac5c1538f0 0x7fac5c159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:02.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.259+0000 7fac61b62640 1 -- 172.21.15.125:0/4206421924 >> 172.21.15.125:0/4206421924 conn(0x7fac5c06a640 msgr2=0x7fac5c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:02.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.259+0000 7fac61b62640 1 -- 172.21.15.125:0/4206421924 shutdown_connections 2023-10-30T13:16:02.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.259+0000 7fac61b62640 1 -- 172.21.15.125:0/4206421924 wait complete. 2023-10-30T13:16:02.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61b62640 1 Processor -- start 2023-10-30T13:16:02.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61b62640 1 -- start start 2023-10-30T13:16:02.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61b62640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac5c0cefa0 0x7fac5c15b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:02.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61b62640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac5c1538f0 0x7fac5c147d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:02.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61b62640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c15cd50 0x7fac5c1482d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:02.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61b62640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c1f30c0 con 0x7fac5c15cd50 2023-10-30T13:16:02.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61b62640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c15ae60 con 0x7fac5c0cefa0 2023-10-30T13:16:02.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61b62640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5c1f2b40 con 0x7fac5c1538f0 2023-10-30T13:16:02.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61361640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c15cd50 0x7fac5c1482d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:02.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61361640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c15cd50 0x7fac5c1482d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40132/0 (socket says 172.21.15.125:40132) 2023-10-30T13:16:02.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61361640 1 -- 172.21.15.125:0/2665370642 learned_addr learned my addr 172.21.15.125:0/2665370642 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:02.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac5bfff640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac5c1538f0 0x7fac5c147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:02.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac60b60640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac5c0cefa0 0x7fac5c15b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:02.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61361640 1 -- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac5c1538f0 msgr2=0x7fac5c147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:02.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61361640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac5c1538f0 0x7fac5c147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:02.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61361640 1 -- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac5c0cefa0 msgr2=0x7fac5c15b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:02.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61361640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac5c0cefa0 0x7fac5c15b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:02.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61361640 1 -- 172.21.15.125:0/2665370642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac480094d0 con 0x7fac5c15cd50 2023-10-30T13:16:02.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac60b60640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac5c0cefa0 0x7fac5c15b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:02.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61361640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c15cd50 0x7fac5c1482d0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fac5400c770 tx=0x7fac5400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:02.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac597fa640 1 -- 172.21.15.125:0/2665370642 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac54003b80 con 0x7fac5c15cd50 2023-10-30T13:16:02.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac597fa640 1 -- 172.21.15.125:0/2665370642 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac54003ce0 con 0x7fac5c15cd50 2023-10-30T13:16:02.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac597fa640 1 -- 172.21.15.125:0/2665370642 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5400aa90 con 0x7fac5c15cd50 2023-10-30T13:16:02.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61b62640 1 -- 172.21.15.125:0/2665370642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5c1f2de0 con 0x7fac5c15cd50 2023-10-30T13:16:02.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.263+0000 7fac61b62640 1 -- 172.21.15.125:0/2665370642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5c15aaf0 con 0x7fac5c15cd50 2023-10-30T13:16:02.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.267+0000 7fac61b62640 1 -- 172.21.15.125:0/2665370642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5c15a670 con 0x7fac5c15cd50 2023-10-30T13:16:02.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.267+0000 7fac597fa640 1 -- 172.21.15.125:0/2665370642 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fac5400abf0 con 0x7fac5c15cd50 2023-10-30T13:16:02.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.267+0000 7fac597fa640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac400664a0 0x7fac40068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:02.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.267+0000 7fac597fa640 1 -- 172.21.15.125:0/2665370642 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7fac540846a0 con 0x7fac5c15cd50 2023-10-30T13:16:02.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.267+0000 7fac60b60640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac400664a0 0x7fac40068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:02.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.267+0000 7fac60b60640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac400664a0 0x7fac40068960 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fac50009bb0 tx=0x7fac50008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:02.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.271+0000 7fac597fa640 1 -- 172.21.15.125:0/2665370642 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fac5c15a670 con 0x7fac5c15cd50 2023-10-30T13:16:02.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.451+0000 7fac61b62640 1 -- 172.21.15.125:0/2665370642 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac5c15a670 con 0x7fac5c15cd50 2023-10-30T13:16:02.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.451+0000 7fac597fa640 1 -- 172.21.15.125:0/2665370642 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+5755 (secure 0 0 0) 0x7fac5c15a670 con 0x7fac5c15cd50 2023-10-30T13:16:02.455 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:16:02.456 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:16:00.781655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:02.456 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 222 2023-10-30T13:16:02.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.455+0000 7fac61b62640 1 -- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac400664a0 msgr2=0x7fac40068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:02.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.455+0000 7fac61b62640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac400664a0 0x7fac40068960 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7fac50009bb0 tx=0x7fac50008040 comp rx=0 tx=0).stop 2023-10-30T13:16:02.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.455+0000 7fac61b62640 1 -- 172.21.15.125:0/2665370642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c15cd50 msgr2=0x7fac5c1482d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:02.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.455+0000 7fac61b62640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c15cd50 0x7fac5c1482d0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fac5400c770 tx=0x7fac5400cc30 comp rx=0 tx=0).stop 2023-10-30T13:16:02.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.455+0000 7fac61b62640 1 -- 172.21.15.125:0/2665370642 shutdown_connections 2023-10-30T13:16:02.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.455+0000 7fac61b62640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fac400664a0 0x7fac40068960 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:02.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.455+0000 7fac61b62640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fac5c15cd50 0x7fac5c1482d0 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:02.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.455+0000 7fac61b62640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fac5c1538f0 0x7fac5c147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:02.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.455+0000 7fac61b62640 1 --2- 172.21.15.125:0/2665370642 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fac5c0cefa0 0x7fac5c15b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:02.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.455+0000 7fac61b62640 1 -- 172.21.15.125:0/2665370642 >> 172.21.15.125:0/2665370642 conn(0x7fac5c06a640 msgr2=0x7fac5c0c53d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:02.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.455+0000 7fac61b62640 1 -- 172.21.15.125:0/2665370642 shutdown_connections 2023-10-30T13:16:02.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:02.455+0000 7fac61b62640 1 -- 172.21.15.125:0/2665370642 wait complete. 2023-10-30T13:16:02.477 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 222, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 222, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_8529": { "addr": "172.21.15.125:6836/3950044434", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3950044434, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3950044434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 8529, "incarnation": 179, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 15 }, "gid_8833": { "addr": "172.21.15.136:6835/3864637420", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3864637420, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3864637420, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8833, "incarnation": 222, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:replay", "state_seq": 1 }, "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 81, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:16:00.781655+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8529, "mds_1": 9150, "mds_2": 8833 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:16:04.478 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:04.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f111b609640 1 Processor -- start 2023-10-30T13:16:04.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f111b609640 1 -- start start 2023-10-30T13:16:04.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f111b609640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f11141538a0 0x7f111415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:04.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f111b609640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11140ca0e0 0x7f111415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:04.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f111b609640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11140cb800 con 0x7f11140cf160 2023-10-30T13:16:04.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f111b609640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f111415dc30 con 0x7f11141538a0 2023-10-30T13:16:04.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f111b609640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11140cb330 con 0x7f11140ca0e0 2023-10-30T13:16:04.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f11193a5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f11141538a0 0x7f111415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:04.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f1119ba6640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f11140cf160 0x7f11140cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:56866/0 (socket says 172.21.15.125:56866) 2023-10-30T13:16:04.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f1119ba6640 1 -- 172.21.15.125:0/3814724460 learned_addr learned my addr 172.21.15.125:0/3814724460 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:16:04.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f1118ba4640 1 --2- 172.21.15.125:0/3814724460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11140ca0e0 0x7f111415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:04.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f11193a5640 1 -- 172.21.15.125:0/3814724460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11140ca0e0 msgr2=0x7f111415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:04.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f11193a5640 1 --2- 172.21.15.125:0/3814724460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11140ca0e0 0x7f111415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:04.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f11193a5640 1 -- 172.21.15.125:0/3814724460 >> v1:172.21.15.125:6789/0 conn(0x7f11140cf160 legacy=0x7f11140cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:04.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f11193a5640 1 -- 172.21.15.125:0/3814724460 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11140cad70 con 0x7f11141538a0 2023-10-30T13:16:04.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.623+0000 7f110bfff640 1 -- 172.21.15.125:0/3814724460 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929937554 0 0) 0x7f11140cb800 con 0x7f11140cf160 2023-10-30T13:16:04.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.627+0000 7f11193a5640 1 --2- 172.21.15.125:0/3814724460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11141538a0 0x7f111415d6f0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f11100097c0 tx=0x7f111001ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4c96b6fd470de8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:04.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.627+0000 7f110bfff640 1 -- 172.21.15.125:0/3814724460 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11100a8970 con 0x7f11141538a0 2023-10-30T13:16:04.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.627+0000 7f110bfff640 1 -- 172.21.15.125:0/3814724460 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11100a8ad0 con 0x7f11141538a0 2023-10-30T13:16:04.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.627+0000 7f110bfff640 1 -- 172.21.15.125:0/3814724460 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11100a8e60 con 0x7f11141538a0 2023-10-30T13:16:04.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.627+0000 7f111b609640 1 -- 172.21.15.125:0/3814724460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11141538a0 msgr2=0x7f111415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:04.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.627+0000 7f111b609640 1 --2- 172.21.15.125:0/3814724460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11141538a0 0x7f111415d6f0 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f11100097c0 tx=0x7f111001ec70 comp rx=0 tx=0).stop 2023-10-30T13:16:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.627+0000 7f111b609640 1 -- 172.21.15.125:0/3814724460 shutdown_connections 2023-10-30T13:16:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.627+0000 7f111b609640 1 --2- 172.21.15.125:0/3814724460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11140ca0e0 0x7f111415a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.627+0000 7f111b609640 1 --2- 172.21.15.125:0/3814724460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11141538a0 0x7f111415d6f0 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.627+0000 7f111b609640 1 -- 172.21.15.125:0/3814724460 >> 172.21.15.125:0/3814724460 conn(0x7f111406bfe0 msgr2=0x7f111406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.627+0000 7f111b609640 1 -- 172.21.15.125:0/3814724460 shutdown_connections 2023-10-30T13:16:04.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.627+0000 7f111b609640 1 -- 172.21.15.125:0/3814724460 wait complete. 2023-10-30T13:16:04.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f111b609640 1 Processor -- start 2023-10-30T13:16:04.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f111b609640 1 -- start start 2023-10-30T13:16:04.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f111b609640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11140ca0e0 0x7f111415b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:04.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f111b609640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11140cf160 0x7f1114150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:04.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f111b609640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11141538a0 0x7f111415c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:04.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f111b609640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1114150a90 con 0x7f11140cf160 2023-10-30T13:16:04.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f111b609640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11141596a0 con 0x7f11140ca0e0 2023-10-30T13:16:04.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f111b609640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1114157600 con 0x7f11141538a0 2023-10-30T13:16:04.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f1118ba4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11140cf160 0x7f1114150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:04.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f1118ba4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11140cf160 0x7f1114150550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40148/0 (socket says 172.21.15.125:40148) 2023-10-30T13:16:04.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f1118ba4640 1 -- 172.21.15.125:0/1395053015 learned_addr learned my addr 172.21.15.125:0/1395053015 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:04.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f1119ba6640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11141538a0 0x7f111415c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:04.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f11193a5640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11140ca0e0 0x7f111415b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:04.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f1118ba4640 1 -- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11141538a0 msgr2=0x7f111415c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:04.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f1118ba4640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11141538a0 0x7f111415c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:04.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f1118ba4640 1 -- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11140ca0e0 msgr2=0x7f111415b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:04.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f1118ba4640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11140ca0e0 0x7f111415b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:04.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f1118ba4640 1 -- 172.21.15.125:0/1395053015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1110009420 con 0x7f11140cf160 2023-10-30T13:16:04.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f11193a5640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11140ca0e0 0x7f111415b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:04.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f1118ba4640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11140cf160 0x7f1114150550 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f110400d800 tx=0x7f110400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:04.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f1109ffb640 1 -- 172.21.15.125:0/1395053015 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110400b6c0 con 0x7f11140cf160 2023-10-30T13:16:04.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f1109ffb640 1 -- 172.21.15.125:0/1395053015 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f110400b820 con 0x7f11140cf160 2023-10-30T13:16:04.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f111b609640 1 -- 172.21.15.125:0/1395053015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11141593c0 con 0x7f11140cf160 2023-10-30T13:16:04.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f1109ffb640 1 -- 172.21.15.125:0/1395053015 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110400bb90 con 0x7f11140cf160 2023-10-30T13:16:04.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.631+0000 7f111b609640 1 -- 172.21.15.125:0/1395053015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1114157b80 con 0x7f11140cf160 2023-10-30T13:16:04.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.635+0000 7f111b609640 1 -- 172.21.15.125:0/1395053015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11140b2a50 con 0x7f11140cf160 2023-10-30T13:16:04.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.635+0000 7f1109ffb640 1 -- 172.21.15.125:0/1395053015 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f110400bcf0 con 0x7f11140cf160 2023-10-30T13:16:04.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.635+0000 7f1109ffb640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f10f80664a0 0x7f10f8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:04.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.635+0000 7f1109ffb640 1 -- 172.21.15.125:0/1395053015 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f1104084c60 con 0x7f11140cf160 2023-10-30T13:16:04.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.635+0000 7f11193a5640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f10f80664a0 0x7f10f8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:04.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.635+0000 7f11193a5640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f10f80664a0 0x7f10f8068960 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f11100b2000 tx=0x7f11100aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:04.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.639+0000 7f1109ffb640 1 -- 172.21.15.125:0/1395053015 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f11040517b0 con 0x7f11140cf160 2023-10-30T13:16:04.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.819+0000 7f111b609640 1 -- 172.21.15.125:0/1395053015 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1114159810 con 0x7f11140cf160 2023-10-30T13:16:04.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.819+0000 7f1109ffb640 1 -- 172.21.15.125:0/1395053015 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 222 v222) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f1104054dd0 con 0x7f11140cf160 2023-10-30T13:16:04.824 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:16:04.824 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":222,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":222,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:16:00.781655+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:04.824 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 222 2023-10-30T13:16:04.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.823+0000 7f111b609640 1 -- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f10f80664a0 msgr2=0x7f10f8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:04.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.823+0000 7f111b609640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f10f80664a0 0x7f10f8068960 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f11100b2000 tx=0x7f11100aa070 comp rx=0 tx=0).stop 2023-10-30T13:16:04.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.823+0000 7f111b609640 1 -- 172.21.15.125:0/1395053015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11140cf160 msgr2=0x7f1114150550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:04.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.823+0000 7f111b609640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11140cf160 0x7f1114150550 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f110400d800 tx=0x7f110400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:16:04.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.823+0000 7f111b609640 1 -- 172.21.15.125:0/1395053015 shutdown_connections 2023-10-30T13:16:04.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.823+0000 7f111b609640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f10f80664a0 0x7f10f8068960 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:04.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.823+0000 7f111b609640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11141538a0 0x7f111415c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:04.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.823+0000 7f111b609640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11140cf160 0x7f1114150550 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:04.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.823+0000 7f111b609640 1 --2- 172.21.15.125:0/1395053015 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f11140ca0e0 0x7f111415b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:04.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.823+0000 7f111b609640 1 -- 172.21.15.125:0/1395053015 >> 172.21.15.125:0/1395053015 conn(0x7f111406bfe0 msgr2=0x7f1114152680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:04.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.823+0000 7f111b609640 1 -- 172.21.15.125:0/1395053015 shutdown_connections 2023-10-30T13:16:04.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:04.823+0000 7f111b609640 1 -- 172.21.15.125:0/1395053015 wait complete. 2023-10-30T13:16:04.848 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:16:06.849 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:06.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f83a5772640 1 Processor -- start 2023-10-30T13:16:06.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f83a5772640 1 -- start start 2023-10-30T13:16:06.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f83a5772640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f83a01538a0 0x7f83a015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:06.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f83a5772640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83a00ca0e0 0x7f83a015a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:06.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f83a5772640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f83a00cf160 0x7f83a00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:06.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f83a5772640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a00cb800 con 0x7f83a00cf160 2023-10-30T13:16:06.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f83a5772640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83a015dc30 con 0x7f83a01538a0 2023-10-30T13:16:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f83a5772640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a00cb330 con 0x7f83a00ca0e0 2023-10-30T13:16:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f839f7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f83a00cf160 0x7f83a00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f839f7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f83a00cf160 0x7f83a00cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40156/0 (socket says 172.21.15.125:40156) 2023-10-30T13:16:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f839f7fe640 1 -- 172.21.15.125:0/3006214557 learned_addr learned my addr 172.21.15.125:0/3006214557 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f839effd640 1 --2- 172.21.15.125:0/3006214557 >> v2:172.21.15.136:3300/0 conn(0x7f83a01538a0 0x7f83a015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:06.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f83967fc640 1 --2- 172.21.15.125:0/3006214557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83a00ca0e0 0x7f83a015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:06.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f839f7fe640 1 -- 172.21.15.125:0/3006214557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83a00ca0e0 msgr2=0x7f83a015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:06.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f839f7fe640 1 --2- 172.21.15.125:0/3006214557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83a00ca0e0 0x7f83a015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:06.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f839f7fe640 1 -- 172.21.15.125:0/3006214557 >> v2:172.21.15.136:3300/0 conn(0x7f83a01538a0 msgr2=0x7f83a015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:06.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f839f7fe640 1 --2- 172.21.15.125:0/3006214557 >> v2:172.21.15.136:3300/0 conn(0x7f83a01538a0 0x7f83a015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:06.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f839f7fe640 1 -- 172.21.15.125:0/3006214557 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a00cad70 con 0x7f83a00cf160 2023-10-30T13:16:06.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.987+0000 7f839effd640 1 --2- 172.21.15.125:0/3006214557 >> v2:172.21.15.136:3300/0 conn(0x7f83a01538a0 0x7f83a015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:06.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.991+0000 7f839f7fe640 1 --2- 172.21.15.125:0/3006214557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a00cf160 0x7f83a00cf590 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f83a00cfd10 tx=0x7f839801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9679d34ed65ae7bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:06.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.991+0000 7f839e7fc640 1 -- 172.21.15.125:0/3006214557 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980a89b0 con 0x7f83a00cf160 2023-10-30T13:16:06.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.991+0000 7f839e7fc640 1 -- 172.21.15.125:0/3006214557 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83980a8b10 con 0x7f83a00cf160 2023-10-30T13:16:06.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.991+0000 7f839e7fc640 1 -- 172.21.15.125:0/3006214557 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980a8ea0 con 0x7f83a00cf160 2023-10-30T13:16:06.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.991+0000 7f83a5772640 1 -- 172.21.15.125:0/3006214557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a00cf160 msgr2=0x7f83a00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:06.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.991+0000 7f83a5772640 1 --2- 172.21.15.125:0/3006214557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a00cf160 0x7f83a00cf590 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f83a00cfd10 tx=0x7f839801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:16:06.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.991+0000 7f83a5772640 1 -- 172.21.15.125:0/3006214557 shutdown_connections 2023-10-30T13:16:06.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.991+0000 7f83a5772640 1 --2- 172.21.15.125:0/3006214557 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a00cf160 0x7f83a00cf590 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:06.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.991+0000 7f83a5772640 1 --2- 172.21.15.125:0/3006214557 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83a00ca0e0 0x7f83a015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:06.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.991+0000 7f83a5772640 1 --2- 172.21.15.125:0/3006214557 >> v2:172.21.15.136:3300/0 conn(0x7f83a01538a0 0x7f83a015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:06.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.991+0000 7f83a5772640 1 -- 172.21.15.125:0/3006214557 >> 172.21.15.125:0/3006214557 conn(0x7f83a006bfe0 msgr2=0x7f83a006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:06.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.991+0000 7f83a5772640 1 -- 172.21.15.125:0/3006214557 shutdown_connections 2023-10-30T13:16:06.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.991+0000 7f83a5772640 1 -- 172.21.15.125:0/3006214557 wait complete. 2023-10-30T13:16:06.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f83a5772640 1 Processor -- start 2023-10-30T13:16:07.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f83a5772640 1 -- start start 2023-10-30T13:16:07.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f83a5772640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a00ca0e0 0x7f83a01e2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:07.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f83a5772640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f83a00cf160 0x7f83a00cd610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:07.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f83a5772640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83a01538a0 0x7f83a00b4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:07.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f83a5772640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a01e31e0 con 0x7f83a00ca0e0 2023-10-30T13:16:07.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f83a5772640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83a01e34b0 con 0x7f83a00cf160 2023-10-30T13:16:07.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f83a5772640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83a01ef390 con 0x7f83a01538a0 2023-10-30T13:16:07.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f839effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a00ca0e0 0x7f83a01e2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:07.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f839effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a00ca0e0 0x7f83a01e2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40166/0 (socket says 172.21.15.125:40166) 2023-10-30T13:16:07.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f839effd640 1 -- 172.21.15.125:0/3570337847 learned_addr learned my addr 172.21.15.125:0/3570337847 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:07.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f83967fc640 1 --2- 172.21.15.125:0/3570337847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f83a00cf160 0x7f83a00cd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:07.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f839f7fe640 1 --2- 172.21.15.125:0/3570337847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83a01538a0 0x7f83a00b4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:07.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f839effd640 1 -- 172.21.15.125:0/3570337847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83a01538a0 msgr2=0x7f83a00b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:07.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f839effd640 1 --2- 172.21.15.125:0/3570337847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83a01538a0 0x7f83a00b4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:07.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f839effd640 1 -- 172.21.15.125:0/3570337847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f83a00cf160 msgr2=0x7f83a00cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:07.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f839effd640 1 --2- 172.21.15.125:0/3570337847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f83a00cf160 0x7f83a00cd610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:07.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f839effd640 1 -- 172.21.15.125:0/3570337847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83980094d0 con 0x7f83a00ca0e0 2023-10-30T13:16:07.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f839effd640 1 --2- 172.21.15.125:0/3570337847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a00ca0e0 0x7f83a01e2ca0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f839000d800 tx=0x7f839000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:07.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f8397fff640 1 -- 172.21.15.125:0/3570337847 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83900096c0 con 0x7f83a00ca0e0 2023-10-30T13:16:07.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f8397fff640 1 -- 172.21.15.125:0/3570337847 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8390009820 con 0x7f83a00ca0e0 2023-10-30T13:16:07.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f83a5772640 1 -- 172.21.15.125:0/3570337847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83a00b3240 con 0x7f83a00ca0e0 2023-10-30T13:16:07.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f8397fff640 1 -- 172.21.15.125:0/3570337847 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390009bd0 con 0x7f83a00ca0e0 2023-10-30T13:16:07.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f83a5772640 1 -- 172.21.15.125:0/3570337847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83a00b3800 con 0x7f83a00ca0e0 2023-10-30T13:16:07.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.995+0000 7f83a5772640 1 -- 172.21.15.125:0/3570337847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83a00b2a50 con 0x7f83a00ca0e0 2023-10-30T13:16:07.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.999+0000 7f8397fff640 1 -- 172.21.15.125:0/3570337847 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8390009d30 con 0x7f83a00ca0e0 2023-10-30T13:16:07.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.999+0000 7f8397fff640 1 --2- 172.21.15.125:0/3570337847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83800664a0 0x7f8380068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:07.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.999+0000 7f8397fff640 1 -- 172.21.15.125:0/3570337847 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f8390084ec0 con 0x7f83a00ca0e0 2023-10-30T13:16:07.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.999+0000 7f83967fc640 1 --2- 172.21.15.125:0/3570337847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83800664a0 0x7f8380068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:07.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:06.999+0000 7f83967fc640 1 --2- 172.21.15.125:0/3570337847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83800664a0 0x7f8380068960 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f83a00b4e70 tx=0x7f8388003a20 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:07.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.003+0000 7f8397fff640 1 -- 172.21.15.125:0/3570337847 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8390051a00 con 0x7f83a00ca0e0 2023-10-30T13:16:07.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.183+0000 7f83a5772640 1 -- 172.21.15.125:0/3570337847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83a004e170 con 0x7f83a00ca0e0 2023-10-30T13:16:07.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.183+0000 7f8397fff640 1 -- 172.21.15.125:0/3570337847 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f8390055020 con 0x7f83a00ca0e0 2023-10-30T13:16:07.188 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:16:07.188 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":223,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:16:06.731480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:resolve","state_seq":9,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:07.189 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 223 2023-10-30T13:16:07.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.187+0000 7f83a5772640 1 -- 172.21.15.125:0/3570337847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83800664a0 msgr2=0x7f8380068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:07.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.187+0000 7f83a5772640 1 --2- 172.21.15.125:0/3570337847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83800664a0 0x7f8380068960 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f83a00b4e70 tx=0x7f8388003a20 comp rx=0 tx=0).stop 2023-10-30T13:16:07.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.187+0000 7f83a5772640 1 -- 172.21.15.125:0/3570337847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a00ca0e0 msgr2=0x7f83a01e2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:07.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.187+0000 7f83a5772640 1 --2- 172.21.15.125:0/3570337847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a00ca0e0 0x7f83a01e2ca0 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f839000d800 tx=0x7f839000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:16:07.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.187+0000 7f83a5772640 1 -- 172.21.15.125:0/3570337847 shutdown_connections 2023-10-30T13:16:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.187+0000 7f83a5772640 1 --2- 172.21.15.125:0/3570337847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f83800664a0 0x7f8380068960 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.187+0000 7f83a5772640 1 --2- 172.21.15.125:0/3570337847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f83a01538a0 0x7f83a00b4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.187+0000 7f83a5772640 1 --2- 172.21.15.125:0/3570337847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f83a00cf160 0x7f83a00cd610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.187+0000 7f83a5772640 1 --2- 172.21.15.125:0/3570337847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f83a00ca0e0 0x7f83a01e2ca0 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:07.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.187+0000 7f83a5772640 1 -- 172.21.15.125:0/3570337847 >> 172.21.15.125:0/3570337847 conn(0x7f83a006bfe0 msgr2=0x7f83a0151880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:07.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.187+0000 7f83a5772640 1 -- 172.21.15.125:0/3570337847 shutdown_connections 2023-10-30T13:16:07.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:07.187+0000 7f83a5772640 1 -- 172.21.15.125:0/3570337847 wait complete. 2023-10-30T13:16:07.209 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:16:09.210 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:09.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f0590765640 1 Processor -- start 2023-10-30T13:16:09.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f0590765640 1 -- start start 2023-10-30T13:16:09.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f0590765640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f058806f7a0 0x7f058806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:09.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f0590765640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0588071110 0x7f0588158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f0590765640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0588066cf0 con 0x7f05880c6bd0 2023-10-30T13:16:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f0590765640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05880700c0 con 0x7f0588071110 2023-10-30T13:16:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f0590765640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05880c70d0 con 0x7f058806f7a0 2023-10-30T13:16:09.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f058ed02640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0588071110 0x7f0588158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:09.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f058ed02640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0588071110 0x7f0588158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40176/0 (socket says 172.21.15.125:40176) 2023-10-30T13:16:09.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f058ed02640 1 -- 172.21.15.125:0/1056802207 learned_addr learned my addr 172.21.15.125:0/1056802207 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:09.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f058e501640 1 --2- 172.21.15.125:0/1056802207 >> v2:172.21.15.136:3300/0 conn(0x7f058806f7a0 0x7f058806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:09.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f058ed02640 1 -- 172.21.15.125:0/1056802207 >> v2:172.21.15.136:3300/0 conn(0x7f058806f7a0 msgr2=0x7f058806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:09.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f058ed02640 1 --2- 172.21.15.125:0/1056802207 >> v2:172.21.15.136:3300/0 conn(0x7f058806f7a0 0x7f058806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:09.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f058ed02640 1 -- 172.21.15.125:0/1056802207 >> v1:172.21.15.125:6789/0 conn(0x7f05880c6bd0 legacy=0x7f0588069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:09.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.355+0000 7f058ed02640 1 -- 172.21.15.125:0/1056802207 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05880c7390 con 0x7f0588071110 2023-10-30T13:16:09.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.359+0000 7f058ed02640 1 --2- 172.21.15.125:0/1056802207 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0588071110 0x7f0588158110 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f058806c830 tx=0x7f058001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d5784722413f4e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:09.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.359+0000 7f058d4ff640 1 -- 172.21.15.125:0/1056802207 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05800af400 con 0x7f0588071110 2023-10-30T13:16:09.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.359+0000 7f058d4ff640 1 -- 172.21.15.125:0/1056802207 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05800ab050 con 0x7f0588071110 2023-10-30T13:16:09.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.359+0000 7f058d4ff640 1 -- 172.21.15.125:0/1056802207 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05800af780 con 0x7f0588071110 2023-10-30T13:16:09.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.359+0000 7f0590765640 1 -- 172.21.15.125:0/1056802207 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0588071110 msgr2=0x7f0588158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:09.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.359+0000 7f0590765640 1 --2- 172.21.15.125:0/1056802207 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0588071110 0x7f0588158110 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f058806c830 tx=0x7f058001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:16:09.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.359+0000 7f0590765640 1 -- 172.21.15.125:0/1056802207 shutdown_connections 2023-10-30T13:16:09.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.359+0000 7f0590765640 1 --2- 172.21.15.125:0/1056802207 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0588071110 0x7f0588158110 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:09.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.359+0000 7f0590765640 1 --2- 172.21.15.125:0/1056802207 >> v2:172.21.15.136:3300/0 conn(0x7f058806f7a0 0x7f058806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:09.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.359+0000 7f0590765640 1 -- 172.21.15.125:0/1056802207 >> 172.21.15.125:0/1056802207 conn(0x7f05880c7910 msgr2=0x7f05880c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:09.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.359+0000 7f0590765640 1 -- 172.21.15.125:0/1056802207 shutdown_connections 2023-10-30T13:16:09.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.359+0000 7f0590765640 1 -- 172.21.15.125:0/1056802207 wait complete. 2023-10-30T13:16:09.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f0590765640 1 Processor -- start 2023-10-30T13:16:09.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f0590765640 1 -- start start 2023-10-30T13:16:09.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f0590765640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f058806f7a0 0x7f05881575a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:09.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f0590765640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0588071110 0x7f0588152620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:09.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f0590765640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05880c6bd0 0x7f058814efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:09.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f0590765640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058806c560 con 0x7f05880c6bd0 2023-10-30T13:16:09.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f0590765640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05880c20b0 con 0x7f058806f7a0 2023-10-30T13:16:09.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f0590765640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05880c9340 con 0x7f0588071110 2023-10-30T13:16:09.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f058ed02640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05880c6bd0 0x7f058814efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:09.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f058ed02640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05880c6bd0 0x7f058814efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40190/0 (socket says 172.21.15.125:40190) 2023-10-30T13:16:09.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f058ed02640 1 -- 172.21.15.125:0/3351371531 learned_addr learned my addr 172.21.15.125:0/3351371531 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:09.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f058e501640 1 --2- 172.21.15.125:0/3351371531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f058806f7a0 0x7f05881575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:09.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f058dd00640 1 --2- 172.21.15.125:0/3351371531 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0588071110 0x7f0588152620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:09.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f058ed02640 1 -- 172.21.15.125:0/3351371531 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0588071110 msgr2=0x7f0588152620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:09.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f058ed02640 1 --2- 172.21.15.125:0/3351371531 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0588071110 0x7f0588152620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:09.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f058ed02640 1 -- 172.21.15.125:0/3351371531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f058806f7a0 msgr2=0x7f05881575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:09.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f058ed02640 1 --2- 172.21.15.125:0/3351371531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f058806f7a0 0x7f05881575a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:09.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f058ed02640 1 -- 172.21.15.125:0/3351371531 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0580009b30 con 0x7f05880c6bd0 2023-10-30T13:16:09.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f058ed02640 1 --2- 172.21.15.125:0/3351371531 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05880c6bd0 0x7f058814efd0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f058001cdf0 tx=0x7f058001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f057effd640 1 -- 172.21.15.125:0/3351371531 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05800afd40 con 0x7f05880c6bd0 2023-10-30T13:16:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f057effd640 1 -- 172.21.15.125:0/3351371531 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05800b9410 con 0x7f05880c6bd0 2023-10-30T13:16:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f057effd640 1 -- 172.21.15.125:0/3351371531 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05800afea0 con 0x7f05880c6bd0 2023-10-30T13:16:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f0590765640 1 -- 172.21.15.125:0/3351371531 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0588154b40 con 0x7f05880c6bd0 2023-10-30T13:16:09.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f0590765640 1 -- 172.21.15.125:0/3351371531 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05880c9060 con 0x7f05880c6bd0 2023-10-30T13:16:09.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.363+0000 7f0590765640 1 -- 172.21.15.125:0/3351371531 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0588070330 con 0x7f05880c6bd0 2023-10-30T13:16:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.367+0000 7f057effd640 1 -- 172.21.15.125:0/3351371531 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f05800b6070 con 0x7f05880c6bd0 2023-10-30T13:16:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.367+0000 7f057effd640 1 --2- 172.21.15.125:0/3351371531 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f056c0664a0 0x7f056c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.367+0000 7f057effd640 1 -- 172.21.15.125:0/3351371531 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f05801224f0 con 0x7f05880c6bd0 2023-10-30T13:16:09.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.367+0000 7f058e501640 1 --2- 172.21.15.125:0/3351371531 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f056c0664a0 0x7f056c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:09.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.367+0000 7f058e501640 1 --2- 172.21.15.125:0/3351371531 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f056c0664a0 0x7f056c068960 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f05840079c0 tx=0x7f0584008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:09.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.371+0000 7f057effd640 1 -- 172.21.15.125:0/3351371531 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0588070330 con 0x7f05880c6bd0 2023-10-30T13:16:09.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.551+0000 7f0590765640 1 -- 172.21.15.125:0/3351371531 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0588070330 con 0x7f05880c6bd0 2023-10-30T13:16:09.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.551+0000 7f057effd640 1 -- 172.21.15.125:0/3351371531 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f0588070330 con 0x7f05880c6bd0 2023-10-30T13:16:09.555 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:16:09.555 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":223,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:16:06.731480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:resolve","state_seq":9,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:09.556 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 223 2023-10-30T13:16:09.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.551+0000 7f0590765640 1 -- 172.21.15.125:0/3351371531 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f056c0664a0 msgr2=0x7f056c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:09.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.551+0000 7f0590765640 1 --2- 172.21.15.125:0/3351371531 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f056c0664a0 0x7f056c068960 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f05840079c0 tx=0x7f0584008040 comp rx=0 tx=0).stop 2023-10-30T13:16:09.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.555+0000 7f0590765640 1 -- 172.21.15.125:0/3351371531 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05880c6bd0 msgr2=0x7f058814efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:09.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.555+0000 7f0590765640 1 --2- 172.21.15.125:0/3351371531 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05880c6bd0 0x7f058814efd0 secure :-1 s=READY pgs=1900 cs=0 l=1 rev1=1 crypto rx=0x7f058001cdf0 tx=0x7f058001bf60 comp rx=0 tx=0).stop 2023-10-30T13:16:09.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.555+0000 7f0590765640 1 -- 172.21.15.125:0/3351371531 shutdown_connections 2023-10-30T13:16:09.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.555+0000 7f0590765640 1 --2- 172.21.15.125:0/3351371531 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f056c0664a0 0x7f056c068960 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:09.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.555+0000 7f0590765640 1 --2- 172.21.15.125:0/3351371531 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f05880c6bd0 0x7f058814efd0 unknown :-1 s=CLOSED pgs=1900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:09.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.555+0000 7f0590765640 1 --2- 172.21.15.125:0/3351371531 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0588071110 0x7f0588152620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:09.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.555+0000 7f0590765640 1 --2- 172.21.15.125:0/3351371531 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f058806f7a0 0x7f05881575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:09.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.555+0000 7f0590765640 1 -- 172.21.15.125:0/3351371531 >> 172.21.15.125:0/3351371531 conn(0x7f05880c7910 msgr2=0x7f05880c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:09.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.555+0000 7f0590765640 1 -- 172.21.15.125:0/3351371531 shutdown_connections 2023-10-30T13:16:09.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:09.555+0000 7f0590765640 1 -- 172.21.15.125:0/3351371531 wait complete. 2023-10-30T13:16:09.576 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:16:11.577 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:11.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.719+0000 7f0abd30e640 1 Processor -- start 2023-10-30T13:16:11.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.719+0000 7f0abd30e640 1 -- start start 2023-10-30T13:16:11.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0abd30e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ab81538a0 0x7f0ab815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:11.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0abd30e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0ab80ca0e0 0x7f0ab815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:11.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0abd30e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0ab80cf160 0x7f0ab80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0abd30e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab80cb800 con 0x7f0ab80cf160 2023-10-30T13:16:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0abd30e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ab815dc30 con 0x7f0ab80ca0e0 2023-10-30T13:16:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0abd30e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab80cb330 con 0x7f0ab81538a0 2023-10-30T13:16:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0ab77fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0ab80cf160 0x7f0ab80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0ab77fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0ab80cf160 0x7f0ab80cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51184/0 (socket says 172.21.15.125:51184) 2023-10-30T13:16:11.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0ab77fe640 1 -- 172.21.15.125:0/1866310326 learned_addr learned my addr 172.21.15.125:0/1866310326 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:11.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0ab6ffd640 1 --2- 172.21.15.125:0/1866310326 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ab81538a0 0x7f0ab815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:11.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0aae7fc640 1 --2- 172.21.15.125:0/1866310326 >> v2:172.21.15.136:3300/0 conn(0x7f0ab80ca0e0 0x7f0ab815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:11.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0ab77fe640 1 -- 172.21.15.125:0/1866310326 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ab81538a0 msgr2=0x7f0ab815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:11.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0ab77fe640 1 --2- 172.21.15.125:0/1866310326 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ab81538a0 0x7f0ab815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:11.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0ab77fe640 1 -- 172.21.15.125:0/1866310326 >> v2:172.21.15.136:3300/0 conn(0x7f0ab80ca0e0 msgr2=0x7f0ab815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0ab77fe640 1 --2- 172.21.15.125:0/1866310326 >> v2:172.21.15.136:3300/0 conn(0x7f0ab80ca0e0 0x7f0ab815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:11.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.723+0000 7f0ab77fe640 1 -- 172.21.15.125:0/1866310326 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab80cad70 con 0x7f0ab80cf160 2023-10-30T13:16:11.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0ab77fe640 1 --2- 172.21.15.125:0/1866310326 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab80cf160 0x7f0ab80cf590 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f0ab80ccd10 tx=0x7f0ab001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de04002c394181b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:11.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0ab67fc640 1 -- 172.21.15.125:0/1866310326 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab00a89b0 con 0x7f0ab80cf160 2023-10-30T13:16:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0ab67fc640 1 -- 172.21.15.125:0/1866310326 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab00a8b10 con 0x7f0ab80cf160 2023-10-30T13:16:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0ab67fc640 1 -- 172.21.15.125:0/1866310326 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab00a8ea0 con 0x7f0ab80cf160 2023-10-30T13:16:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0abd30e640 1 -- 172.21.15.125:0/1866310326 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab80cf160 msgr2=0x7f0ab80cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0abd30e640 1 --2- 172.21.15.125:0/1866310326 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab80cf160 0x7f0ab80cf590 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7f0ab80ccd10 tx=0x7f0ab001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:16:11.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0abd30e640 1 -- 172.21.15.125:0/1866310326 shutdown_connections 2023-10-30T13:16:11.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0abd30e640 1 --2- 172.21.15.125:0/1866310326 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab80cf160 0x7f0ab80cf590 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:11.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0abd30e640 1 --2- 172.21.15.125:0/1866310326 >> v2:172.21.15.136:3300/0 conn(0x7f0ab80ca0e0 0x7f0ab815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:11.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0abd30e640 1 --2- 172.21.15.125:0/1866310326 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ab81538a0 0x7f0ab815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:11.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0abd30e640 1 -- 172.21.15.125:0/1866310326 >> 172.21.15.125:0/1866310326 conn(0x7f0ab806bfe0 msgr2=0x7f0ab806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:11.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0abd30e640 1 -- 172.21.15.125:0/1866310326 shutdown_connections 2023-10-30T13:16:11.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0abd30e640 1 -- 172.21.15.125:0/1866310326 wait complete. 2023-10-30T13:16:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0abd30e640 1 Processor -- start 2023-10-30T13:16:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.727+0000 7f0abd30e640 1 -- start start 2023-10-30T13:16:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0abd30e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ab80ca0e0 0x7f0ab81efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:11.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0abd30e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab80cf160 0x7f0ab81e5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:11.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0abd30e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ab81538a0 0x7f0ab81e2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:11.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0abd30e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab815afa0 con 0x7f0ab80cf160 2023-10-30T13:16:11.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0abd30e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab815a9e0 con 0x7f0ab81538a0 2023-10-30T13:16:11.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0abd30e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab815acc0 con 0x7f0ab80ca0e0 2023-10-30T13:16:11.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0aae7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab80cf160 0x7f0ab81e5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:11.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0aae7fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab80cf160 0x7f0ab81e5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51190/0 (socket says 172.21.15.125:51190) 2023-10-30T13:16:11.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0aae7fc640 1 -- 172.21.15.125:0/1989281826 learned_addr learned my addr 172.21.15.125:0/1989281826 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:11.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0ab6ffd640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ab80ca0e0 0x7f0ab81efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:11.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0ab77fe640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ab81538a0 0x7f0ab81e2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:11.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0aae7fc640 1 -- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ab80ca0e0 msgr2=0x7f0ab81efb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:11.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0aae7fc640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ab80ca0e0 0x7f0ab81efb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:11.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0aae7fc640 1 -- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ab81538a0 msgr2=0x7f0ab81e2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:11.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0aae7fc640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ab81538a0 0x7f0ab81e2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:11.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0aae7fc640 1 -- 172.21.15.125:0/1989281826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab00094d0 con 0x7f0ab80cf160 2023-10-30T13:16:11.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0ab77fe640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ab81538a0 0x7f0ab81e2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:11.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0aae7fc640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab80cf160 0x7f0ab81e5620 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0007e60 tx=0x7f0aa000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:11.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0aaffff640 1 -- 172.21.15.125:0/1989281826 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0014620 con 0x7f0ab80cf160 2023-10-30T13:16:11.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0aaffff640 1 -- 172.21.15.125:0/1989281826 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0aa000ee90 con 0x7f0ab80cf160 2023-10-30T13:16:11.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0aaffff640 1 -- 172.21.15.125:0/1989281826 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa00149c0 con 0x7f0ab80cf160 2023-10-30T13:16:11.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0abd30e640 1 -- 172.21.15.125:0/1989281826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab81ef080 con 0x7f0ab80cf160 2023-10-30T13:16:11.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0abd30e640 1 -- 172.21.15.125:0/1989281826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab81ef640 con 0x7f0ab80cf160 2023-10-30T13:16:11.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0abd30e640 1 -- 172.21.15.125:0/1989281826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab81ef800 con 0x7f0ab80cf160 2023-10-30T13:16:11.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.731+0000 7f0aaffff640 1 -- 172.21.15.125:0/1989281826 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0aa0014b20 con 0x7f0ab80cf160 2023-10-30T13:16:11.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.735+0000 7f0aaffff640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a980664a0 0x7f0a98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:11.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.735+0000 7f0aaffff640 1 -- 172.21.15.125:0/1989281826 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f0aa00850b0 con 0x7f0ab80cf160 2023-10-30T13:16:11.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.735+0000 7f0ab6ffd640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a980664a0 0x7f0a98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.735+0000 7f0ab6ffd640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a980664a0 0x7f0a98068960 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0aa80079c0 tx=0x7f0aa8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:11.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.739+0000 7f0aaffff640 1 -- 172.21.15.125:0/1989281826 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0aa0051c10 con 0x7f0ab80cf160 2023-10-30T13:16:11.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.919+0000 7f0abd30e640 1 -- 172.21.15.125:0/1989281826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ab80b2a50 con 0x7f0ab80cf160 2023-10-30T13:16:11.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.919+0000 7f0aaffff640 1 -- 172.21.15.125:0/1989281826 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 223 v223) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f0aa0009a20 con 0x7f0ab80cf160 2023-10-30T13:16:11.923 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:16:11.924 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":223,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":223,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:16:06.731480+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:resolve","state_seq":9,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:11.924 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 223 2023-10-30T13:16:11.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.923+0000 7f0abd30e640 1 -- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a980664a0 msgr2=0x7f0a98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:11.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.923+0000 7f0abd30e640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a980664a0 0x7f0a98068960 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f0aa80079c0 tx=0x7f0aa8008040 comp rx=0 tx=0).stop 2023-10-30T13:16:11.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.923+0000 7f0abd30e640 1 -- 172.21.15.125:0/1989281826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab80cf160 msgr2=0x7f0ab81e5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:11.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.923+0000 7f0abd30e640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab80cf160 0x7f0ab81e5620 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0007e60 tx=0x7f0aa000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:16:11.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.923+0000 7f0abd30e640 1 -- 172.21.15.125:0/1989281826 shutdown_connections 2023-10-30T13:16:11.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.923+0000 7f0abd30e640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a980664a0 0x7f0a98068960 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:11.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.923+0000 7f0abd30e640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ab81538a0 0x7f0ab81e2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:11.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.923+0000 7f0abd30e640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ab80cf160 0x7f0ab81e5620 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:11.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.923+0000 7f0abd30e640 1 --2- 172.21.15.125:0/1989281826 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ab80ca0e0 0x7f0ab81efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:11.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.923+0000 7f0abd30e640 1 -- 172.21.15.125:0/1989281826 >> 172.21.15.125:0/1989281826 conn(0x7f0ab806bfe0 msgr2=0x7f0ab81526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:11.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.923+0000 7f0abd30e640 1 -- 172.21.15.125:0/1989281826 shutdown_connections 2023-10-30T13:16:11.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:11.923+0000 7f0abd30e640 1 -- 172.21.15.125:0/1989281826 wait complete. 2023-10-30T13:16:11.945 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:16:13.946 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:14.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.091+0000 7ffb8073e640 1 Processor -- start 2023-10-30T13:16:14.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.091+0000 7ffb8073e640 1 -- start start 2023-10-30T13:16:14.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.091+0000 7ffb8073e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffb780c6f60 0x7ffb781566a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:14.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.091+0000 7ffb8073e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb780c0a50 con 0x7ffb780c6b80 2023-10-30T13:16:14.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.091+0000 7ffb8073e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7806f6e0 con 0x7ffb780c05d0 2023-10-30T13:16:14.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.091+0000 7ffb8073e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb7806f9c0 con 0x7ffb780c6f60 2023-10-30T13:16:14.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.091+0000 7ffb7e4da640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ffb780c6b80 0x7ffb780bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40488/0 (socket says 172.21.15.125:40488) 2023-10-30T13:16:14.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.091+0000 7ffb7e4da640 1 -- 172.21.15.125:0/871716536 learned_addr learned my addr 172.21.15.125:0/871716536 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:16:14.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.091+0000 7ffb7ecdb640 1 --2- 172.21.15.125:0/871716536 >> v2:172.21.15.125:3300/0 conn(0x7ffb780c6f60 0x7ffb781566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:14.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.091+0000 7ffb7d4d8640 1 -- 172.21.15.125:0/871716536 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256890635 0 0) 0x7ffb780c0a50 con 0x7ffb780c6b80 2023-10-30T13:16:14.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.091+0000 7ffb7d4d8640 1 -- 172.21.15.125:0/871716536 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb64005740 con 0x7ffb780c6b80 2023-10-30T13:16:14.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.091+0000 7ffb7ecdb640 1 -- 172.21.15.125:0/871716536 >> v1:172.21.15.136:6789/0 conn(0x7ffb780c05d0 legacy=0x7ffb7813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:14.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.091+0000 7ffb7ecdb640 1 -- 172.21.15.125:0/871716536 >> v1:172.21.15.125:6789/0 conn(0x7ffb780c6b80 legacy=0x7ffb780bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:14.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.091+0000 7ffb7ecdb640 1 -- 172.21.15.125:0/871716536 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb7806fca0 con 0x7ffb780c6f60 2023-10-30T13:16:14.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.095+0000 7ffb7ecdb640 1 --2- 172.21.15.125:0/871716536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb780c6f60 0x7ffb781566a0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009d90 tx=0x7ffb7001c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c82ac752405ff183 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:14.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.095+0000 7ffb7d4d8640 1 -- 172.21.15.125:0/871716536 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb700a5820 con 0x7ffb780c6f60 2023-10-30T13:16:14.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.095+0000 7ffb7d4d8640 1 -- 172.21.15.125:0/871716536 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb700a5980 con 0x7ffb780c6f60 2023-10-30T13:16:14.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.095+0000 7ffb7d4d8640 1 -- 172.21.15.125:0/871716536 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb700a5d10 con 0x7ffb780c6f60 2023-10-30T13:16:14.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.095+0000 7ffb8073e640 1 -- 172.21.15.125:0/871716536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb780c6f60 msgr2=0x7ffb781566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:14.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.095+0000 7ffb8073e640 1 --2- 172.21.15.125:0/871716536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb780c6f60 0x7ffb781566a0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7ffb70009d90 tx=0x7ffb7001c950 comp rx=0 tx=0).stop 2023-10-30T13:16:14.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.095+0000 7ffb8073e640 1 -- 172.21.15.125:0/871716536 shutdown_connections 2023-10-30T13:16:14.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.095+0000 7ffb8073e640 1 --2- 172.21.15.125:0/871716536 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb780c6f60 0x7ffb781566a0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:14.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.095+0000 7ffb8073e640 1 -- 172.21.15.125:0/871716536 >> 172.21.15.125:0/871716536 conn(0x7ffb780c7910 msgr2=0x7ffb780c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:14.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.095+0000 7ffb8073e640 1 -- 172.21.15.125:0/871716536 shutdown_connections 2023-10-30T13:16:14.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.095+0000 7ffb8073e640 1 -- 172.21.15.125:0/871716536 wait complete. 2023-10-30T13:16:14.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb8073e640 1 Processor -- start 2023-10-30T13:16:14.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb8073e640 1 -- start start 2023-10-30T13:16:14.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb8073e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb780c05d0 0x7ffb7806a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:14.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb8073e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb780c6b80 0x7ffb780afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:14.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb8073e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb780c6f60 0x7ffb781589f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:14.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb8073e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7813c9e0 con 0x7ffb780c6f60 2023-10-30T13:16:14.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb8073e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb780c9080 con 0x7ffb780c05d0 2023-10-30T13:16:14.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb8073e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb78154ea0 con 0x7ffb780c6b80 2023-10-30T13:16:14.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb7ecdb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb780c6f60 0x7ffb781589f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:14.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb7ecdb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb780c6f60 0x7ffb781589f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51206/0 (socket says 172.21.15.125:51206) 2023-10-30T13:16:14.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb7ecdb640 1 -- 172.21.15.125:0/2065568102 learned_addr learned my addr 172.21.15.125:0/2065568102 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:14.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb7e4da640 1 --2- 172.21.15.125:0/2065568102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb780c05d0 0x7ffb7806a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:14.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb7dcd9640 1 --2- 172.21.15.125:0/2065568102 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb780c6b80 0x7ffb780afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:14.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb7ecdb640 1 -- 172.21.15.125:0/2065568102 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb780c6b80 msgr2=0x7ffb780afe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:14.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb7ecdb640 1 --2- 172.21.15.125:0/2065568102 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb780c6b80 0x7ffb780afe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:14.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb7ecdb640 1 -- 172.21.15.125:0/2065568102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb780c05d0 msgr2=0x7ffb7806a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:14.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb7ecdb640 1 --2- 172.21.15.125:0/2065568102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb780c05d0 0x7ffb7806a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:14.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb7ecdb640 1 -- 172.21.15.125:0/2065568102 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb70009a70 con 0x7ffb780c6f60 2023-10-30T13:16:14.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb7ecdb640 1 --2- 172.21.15.125:0/2065568102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb780c6f60 0x7ffb781589f0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7ffb7001a580 tx=0x7ffb700196f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:14.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb6effd640 1 -- 172.21.15.125:0/2065568102 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb700b9650 con 0x7ffb780c6f60 2023-10-30T13:16:14.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb6effd640 1 -- 172.21.15.125:0/2065568102 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb700b97b0 con 0x7ffb780c6f60 2023-10-30T13:16:14.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb8073e640 1 -- 172.21.15.125:0/2065568102 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb78155380 con 0x7ffb780c6f60 2023-10-30T13:16:14.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb6effd640 1 -- 172.21.15.125:0/2065568102 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb700b9b20 con 0x7ffb780c6f60 2023-10-30T13:16:14.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb8073e640 1 -- 172.21.15.125:0/2065568102 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb78155940 con 0x7ffb780c6f60 2023-10-30T13:16:14.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.099+0000 7ffb8073e640 1 -- 172.21.15.125:0/2065568102 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb7804e170 con 0x7ffb780c6f60 2023-10-30T13:16:14.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.103+0000 7ffb6effd640 1 -- 172.21.15.125:0/2065568102 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffb700b9c80 con 0x7ffb780c6f60 2023-10-30T13:16:14.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.103+0000 7ffb6effd640 1 --2- 172.21.15.125:0/2065568102 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffb5c0664a0 0x7ffb5c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:14.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.103+0000 7ffb6effd640 1 -- 172.21.15.125:0/2065568102 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7ffb7011fa00 con 0x7ffb780c6f60 2023-10-30T13:16:14.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.103+0000 7ffb7e4da640 1 --2- 172.21.15.125:0/2065568102 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffb5c0664a0 0x7ffb5c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:14.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.103+0000 7ffb7e4da640 1 --2- 172.21.15.125:0/2065568102 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffb5c0664a0 0x7ffb5c068960 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7ffb740011f0 tx=0x7ffb74009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:14.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.107+0000 7ffb6effd640 1 -- 172.21.15.125:0/2065568102 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffb700ec4b0 con 0x7ffb780c6f60 2023-10-30T13:16:14.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.287+0000 7ffb8073e640 1 -- 172.21.15.125:0/2065568102 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb7813ac30 con 0x7ffb780c6f60 2023-10-30T13:16:14.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.287+0000 7ffb6effd640 1 -- 172.21.15.125:0/2065568102 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+5759 (secure 0 0 0) 0x7ffb700b2090 con 0x7ffb780c6f60 2023-10-30T13:16:14.291 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:16:14.292 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:16:11.889190+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:14.292 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 224 2023-10-30T13:16:14.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.291+0000 7ffb8073e640 1 -- 172.21.15.125:0/2065568102 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffb5c0664a0 msgr2=0x7ffb5c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:14.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.291+0000 7ffb8073e640 1 --2- 172.21.15.125:0/2065568102 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffb5c0664a0 0x7ffb5c068960 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7ffb740011f0 tx=0x7ffb74009040 comp rx=0 tx=0).stop 2023-10-30T13:16:14.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.291+0000 7ffb8073e640 1 -- 172.21.15.125:0/2065568102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb780c6f60 msgr2=0x7ffb781589f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:14.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.291+0000 7ffb8073e640 1 --2- 172.21.15.125:0/2065568102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb780c6f60 0x7ffb781589f0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7ffb7001a580 tx=0x7ffb700196f0 comp rx=0 tx=0).stop 2023-10-30T13:16:14.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.291+0000 7ffb8073e640 1 -- 172.21.15.125:0/2065568102 shutdown_connections 2023-10-30T13:16:14.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.291+0000 7ffb8073e640 1 --2- 172.21.15.125:0/2065568102 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffb5c0664a0 0x7ffb5c068960 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:14.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.291+0000 7ffb8073e640 1 --2- 172.21.15.125:0/2065568102 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffb780c6f60 0x7ffb781589f0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:14.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.291+0000 7ffb8073e640 1 --2- 172.21.15.125:0/2065568102 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffb780c6b80 0x7ffb780afe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:14.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.291+0000 7ffb8073e640 1 --2- 172.21.15.125:0/2065568102 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffb780c05d0 0x7ffb7806a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:14.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.291+0000 7ffb8073e640 1 -- 172.21.15.125:0/2065568102 >> 172.21.15.125:0/2065568102 conn(0x7ffb780c7910 msgr2=0x7ffb7814cfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:14.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.291+0000 7ffb8073e640 1 -- 172.21.15.125:0/2065568102 shutdown_connections 2023-10-30T13:16:14.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:14.291+0000 7ffb8073e640 1 -- 172.21.15.125:0/2065568102 wait complete. 2023-10-30T13:16:14.313 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 224, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 224, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_8529": { "addr": "172.21.15.125:6836/3950044434", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 3950044434, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 3950044434, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 8529, "incarnation": 179, "join_fscid": -1, "name": "e", "rank": 0, "state": "up:active", "state_seq": 15 }, "gid_8833": { "addr": "172.21.15.136:6835/3864637420", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3864637420, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3864637420, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8833, "incarnation": 222, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:reconnect", "state_seq": 11 }, "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 81, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:16:11.889190+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8529, "mds_1": 9150, "mds_2": 8833 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:16:16.314 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:16.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.455+0000 7f2923c6b640 1 Processor -- start 2023-10-30T13:16:16.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.455+0000 7f2923c6b640 1 -- start start 2023-10-30T13:16:16.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.459+0000 7f2923c6b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f291c0c6b80 0x7f291c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:16.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.459+0000 7f2923c6b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f291c0c6f60 0x7f291c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:16.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.459+0000 7f2923c6b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291c0c0a50 con 0x7f291c0c8430 2023-10-30T13:16:16.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.459+0000 7f2923c6b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f291c06f6e0 con 0x7f291c0c6f60 2023-10-30T13:16:16.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.459+0000 7f2923c6b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291c06f9c0 con 0x7f291c0c6b80 2023-10-30T13:16:16.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.459+0000 7f2921206640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f291c0c6f60 0x7f291c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:16.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.459+0000 7f2921a07640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f291c0c6b80 0x7f291c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:16.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.459+0000 7f2921206640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f291c0c6f60 0x7f291c0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:41396/0 (socket says 172.21.15.125:41396) 2023-10-30T13:16:16.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.459+0000 7f2921206640 1 -- 172.21.15.125:0/164089378 learned_addr learned my addr 172.21.15.125:0/164089378 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:16.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.459+0000 7f2921a07640 1 -- 172.21.15.125:0/164089378 >> v2:172.21.15.136:3300/0 conn(0x7f291c0c6f60 msgr2=0x7f291c0c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:16.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.459+0000 7f2921a07640 1 --2- 172.21.15.125:0/164089378 >> v2:172.21.15.136:3300/0 conn(0x7f291c0c6f60 0x7f291c0c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:16.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.459+0000 7f2921a07640 1 -- 172.21.15.125:0/164089378 >> v1:172.21.15.136:6789/0 conn(0x7f291c0c8430 legacy=0x7f291c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:16.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.459+0000 7f2921a07640 1 -- 172.21.15.125:0/164089378 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c06fca0 con 0x7f291c0c6b80 2023-10-30T13:16:16.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.459+0000 7f2921206640 1 --2- 172.21.15.125:0/164089378 >> v2:172.21.15.136:3300/0 conn(0x7f291c0c6f60 0x7f291c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:16:16.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.463+0000 7f2921a07640 1 --2- 172.21.15.125:0/164089378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f291c0c6b80 0x7f291c0bfec0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f29180097c0 tx=0x7f291801c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ff140490fa44f4b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:16.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.463+0000 7f2920a05640 1 -- 172.21.15.125:0/164089378 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29180ad640 con 0x7f291c0c6b80 2023-10-30T13:16:16.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.463+0000 7f2920a05640 1 -- 172.21.15.125:0/164089378 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29180ad7a0 con 0x7f291c0c6b80 2023-10-30T13:16:16.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.463+0000 7f2920a05640 1 -- 172.21.15.125:0/164089378 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29180adb30 con 0x7f291c0c6b80 2023-10-30T13:16:16.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.463+0000 7f2923c6b640 1 -- 172.21.15.125:0/164089378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f291c0c6b80 msgr2=0x7f291c0bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:16.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.463+0000 7f2923c6b640 1 --2- 172.21.15.125:0/164089378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f291c0c6b80 0x7f291c0bfec0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f29180097c0 tx=0x7f291801c480 comp rx=0 tx=0).stop 2023-10-30T13:16:16.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.463+0000 7f2923c6b640 1 -- 172.21.15.125:0/164089378 shutdown_connections 2023-10-30T13:16:16.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.463+0000 7f2923c6b640 1 --2- 172.21.15.125:0/164089378 >> v2:172.21.15.136:3300/0 conn(0x7f291c0c6f60 0x7f291c0c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:16.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.463+0000 7f2923c6b640 1 --2- 172.21.15.125:0/164089378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f291c0c6b80 0x7f291c0bfec0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:16.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.463+0000 7f2923c6b640 1 -- 172.21.15.125:0/164089378 >> 172.21.15.125:0/164089378 conn(0x7f291c0c7910 msgr2=0x7f291c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:16.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.463+0000 7f2923c6b640 1 -- 172.21.15.125:0/164089378 shutdown_connections 2023-10-30T13:16:16.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.463+0000 7f2923c6b640 1 -- 172.21.15.125:0/164089378 wait complete. 2023-10-30T13:16:16.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2923c6b640 1 Processor -- start 2023-10-30T13:16:16.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2923c6b640 1 -- start start 2023-10-30T13:16:16.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2923c6b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f291c0c6b80 0x7f291c1516e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:16.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2923c6b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f291c0c6f60 0x7f291c159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:16.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2923c6b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f291c0c8430 0x7f291c1319b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:16.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2923c6b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c133440 con 0x7f291c0c6b80 2023-10-30T13:16:16.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2923c6b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f291c162100 con 0x7f291c0c6f60 2023-10-30T13:16:16.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2923c6b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f291c163f60 con 0x7f291c0c8430 2023-10-30T13:16:16.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2921a07640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f291c0c6b80 0x7f291c1516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:16.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2921a07640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f291c0c6b80 0x7f291c1516e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51218/0 (socket says 172.21.15.125:51218) 2023-10-30T13:16:16.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2921a07640 1 -- 172.21.15.125:0/3386296700 learned_addr learned my addr 172.21.15.125:0/3386296700 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:16.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2921206640 1 --2- 172.21.15.125:0/3386296700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f291c0c6f60 0x7f291c159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:16.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2922208640 1 --2- 172.21.15.125:0/3386296700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f291c0c8430 0x7f291c1319b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:16.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2921a07640 1 -- 172.21.15.125:0/3386296700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f291c0c8430 msgr2=0x7f291c1319b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:16.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2921a07640 1 --2- 172.21.15.125:0/3386296700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f291c0c8430 0x7f291c1319b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:16.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2921a07640 1 -- 172.21.15.125:0/3386296700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f291c0c6f60 msgr2=0x7f291c159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:16.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2921a07640 1 --2- 172.21.15.125:0/3386296700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f291c0c6f60 0x7f291c159540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:16.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2921a07640 1 -- 172.21.15.125:0/3386296700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290c0094d0 con 0x7f291c0c6b80 2023-10-30T13:16:16.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2921a07640 1 --2- 172.21.15.125:0/3386296700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f291c0c6b80 0x7f291c1516e0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f291801a0b0 tx=0x7f2918019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:16.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f29127fc640 1 -- 172.21.15.125:0/3386296700 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29180b6820 con 0x7f291c0c6b80 2023-10-30T13:16:16.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f29127fc640 1 -- 172.21.15.125:0/3386296700 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29180a6040 con 0x7f291c0c6b80 2023-10-30T13:16:16.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2923c6b640 1 -- 172.21.15.125:0/3386296700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2918009420 con 0x7f291c0c6b80 2023-10-30T13:16:16.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f29127fc640 1 -- 172.21.15.125:0/3386296700 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29180b6be0 con 0x7f291c0c6b80 2023-10-30T13:16:16.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2923c6b640 1 -- 172.21.15.125:0/3386296700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291c1512a0 con 0x7f291c0c6b80 2023-10-30T13:16:16.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.467+0000 7f2923c6b640 1 -- 172.21.15.125:0/3386296700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291c04e170 con 0x7f291c0c6b80 2023-10-30T13:16:16.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.471+0000 7f29127fc640 1 -- 172.21.15.125:0/3386296700 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f29180b4040 con 0x7f291c0c6b80 2023-10-30T13:16:16.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.471+0000 7f29127fc640 1 --2- 172.21.15.125:0/3386296700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29000664a0 0x7f2900068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:16.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.471+0000 7f29127fc640 1 -- 172.21.15.125:0/3386296700 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f291811ed90 con 0x7f291c0c6b80 2023-10-30T13:16:16.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.471+0000 7f2921206640 1 --2- 172.21.15.125:0/3386296700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29000664a0 0x7f2900068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:16.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.471+0000 7f2921206640 1 --2- 172.21.15.125:0/3386296700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29000664a0 0x7f2900068960 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f291c13baa0 tx=0x7f290c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:16.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.475+0000 7f29127fc640 1 -- 172.21.15.125:0/3386296700 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f29180eba20 con 0x7f291c0c6b80 2023-10-30T13:16:16.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.659+0000 7f2923c6b640 1 -- 172.21.15.125:0/3386296700 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f291c0c1950 con 0x7f291c0c6b80 2023-10-30T13:16:16.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.659+0000 7f29127fc640 1 -- 172.21.15.125:0/3386296700 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 224 v224) v1 ==== 95+0+5759 (secure 0 0 0) 0x7f29180b2030 con 0x7f291c0c6b80 2023-10-30T13:16:16.662 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:16:16.662 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":224,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":224,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:16:11.889190+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:16.662 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 224 2023-10-30T13:16:16.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.659+0000 7f2923c6b640 1 -- 172.21.15.125:0/3386296700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29000664a0 msgr2=0x7f2900068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:16.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.659+0000 7f2923c6b640 1 --2- 172.21.15.125:0/3386296700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29000664a0 0x7f2900068960 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f291c13baa0 tx=0x7f290c008040 comp rx=0 tx=0).stop 2023-10-30T13:16:16.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.659+0000 7f2923c6b640 1 -- 172.21.15.125:0/3386296700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f291c0c6b80 msgr2=0x7f291c1516e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:16.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.659+0000 7f2923c6b640 1 --2- 172.21.15.125:0/3386296700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f291c0c6b80 0x7f291c1516e0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f291801a0b0 tx=0x7f2918019220 comp rx=0 tx=0).stop 2023-10-30T13:16:16.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.659+0000 7f2923c6b640 1 -- 172.21.15.125:0/3386296700 shutdown_connections 2023-10-30T13:16:16.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.659+0000 7f2923c6b640 1 --2- 172.21.15.125:0/3386296700 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f29000664a0 0x7f2900068960 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:16.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.659+0000 7f2923c6b640 1 --2- 172.21.15.125:0/3386296700 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f291c0c8430 0x7f291c1319b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:16.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.659+0000 7f2923c6b640 1 --2- 172.21.15.125:0/3386296700 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f291c0c6f60 0x7f291c159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:16.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.659+0000 7f2923c6b640 1 --2- 172.21.15.125:0/3386296700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f291c0c6b80 0x7f291c1516e0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:16.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.659+0000 7f2923c6b640 1 -- 172.21.15.125:0/3386296700 >> 172.21.15.125:0/3386296700 conn(0x7f291c0c7910 msgr2=0x7f291c0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:16.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.659+0000 7f2923c6b640 1 -- 172.21.15.125:0/3386296700 shutdown_connections 2023-10-30T13:16:16.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:16.659+0000 7f2923c6b640 1 -- 172.21.15.125:0/3386296700 wait complete. 2023-10-30T13:16:16.683 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:16:17.044 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:16:17.048 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:16:18.684 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:18.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff11553e640 1 Processor -- start 2023-10-30T13:16:18.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff11553e640 1 -- start start 2023-10-30T13:16:18.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff11553e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff1100c8cf0 0x7ff1100c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:18.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff11553e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff1100ca340 0x7ff11015d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:18.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff11553e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff11015ce30 con 0x7ff1100c90d0 2023-10-30T13:16:18.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff11553e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff110159880 con 0x7ff1100c8cf0 2023-10-30T13:16:18.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff11553e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff110159ae0 con 0x7ff1100ca340 2023-10-30T13:16:18.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff10effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff1100c8cf0 0x7ff1100c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:18.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff10effd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff1100c8cf0 0x7ff1100c9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51226/0 (socket says 172.21.15.125:51226) 2023-10-30T13:16:18.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff10effd640 1 -- 172.21.15.125:0/2327035700 learned_addr learned my addr 172.21.15.125:0/2327035700 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:18.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff10e7fc640 1 --1- 172.21.15.125:0/2327035700 >> v1:172.21.15.125:6789/0 conn(0x7ff1100c90d0 0x7ff1100cef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40490/0 (socket says 172.21.15.125:40490) 2023-10-30T13:16:18.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff10f7fe640 1 --2- 172.21.15.125:0/2327035700 >> v2:172.21.15.136:3300/0 conn(0x7ff1100ca340 0x7ff11015d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:18.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff10effd640 1 -- 172.21.15.125:0/2327035700 >> v2:172.21.15.136:3300/0 conn(0x7ff1100ca340 msgr2=0x7ff11015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:18.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff10effd640 1 --2- 172.21.15.125:0/2327035700 >> v2:172.21.15.136:3300/0 conn(0x7ff1100ca340 0x7ff11015d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:18.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff10effd640 1 -- 172.21.15.125:0/2327035700 >> v1:172.21.15.125:6789/0 conn(0x7ff1100c90d0 legacy=0x7ff1100cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:18.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff10effd640 1 -- 172.21.15.125:0/2327035700 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff110159dc0 con 0x7ff1100c8cf0 2023-10-30T13:16:18.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.831+0000 7ff10dffb640 1 -- 172.21.15.125:0/2327035700 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 650877228 0 0) 0x7ff11015ce30 con 0x7ff1100c90d0 2023-10-30T13:16:18.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.835+0000 7ff10effd640 1 --2- 172.21.15.125:0/2327035700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100c8cf0 0x7ff1100c9e00 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7ff1100c60c0 tx=0x7ff10001c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8291e610671d941 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:18.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.835+0000 7ff10dffb640 1 -- 172.21.15.125:0/2327035700 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1000ac640 con 0x7ff1100c8cf0 2023-10-30T13:16:18.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.835+0000 7ff10dffb640 1 -- 172.21.15.125:0/2327035700 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1000ac7a0 con 0x7ff1100c8cf0 2023-10-30T13:16:18.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.835+0000 7ff10dffb640 1 -- 172.21.15.125:0/2327035700 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1000acb30 con 0x7ff1100c8cf0 2023-10-30T13:16:18.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.835+0000 7ff11553e640 1 -- 172.21.15.125:0/2327035700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100c8cf0 msgr2=0x7ff1100c9e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:18.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.835+0000 7ff11553e640 1 --2- 172.21.15.125:0/2327035700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100c8cf0 0x7ff1100c9e00 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7ff1100c60c0 tx=0x7ff10001c4d0 comp rx=0 tx=0).stop 2023-10-30T13:16:18.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.835+0000 7ff11553e640 1 -- 172.21.15.125:0/2327035700 shutdown_connections 2023-10-30T13:16:18.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.835+0000 7ff11553e640 1 --2- 172.21.15.125:0/2327035700 >> v2:172.21.15.136:3300/0 conn(0x7ff1100ca340 0x7ff11015d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:18.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.835+0000 7ff11553e640 1 --2- 172.21.15.125:0/2327035700 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100c8cf0 0x7ff1100c9e00 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:18.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.835+0000 7ff11553e640 1 -- 172.21.15.125:0/2327035700 >> 172.21.15.125:0/2327035700 conn(0x7ff11006be40 msgr2=0x7ff11006c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:18.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.835+0000 7ff11553e640 1 -- 172.21.15.125:0/2327035700 shutdown_connections 2023-10-30T13:16:18.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.835+0000 7ff11553e640 1 -- 172.21.15.125:0/2327035700 wait complete. 2023-10-30T13:16:18.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.835+0000 7ff11553e640 1 Processor -- start 2023-10-30T13:16:18.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff11553e640 1 -- start start 2023-10-30T13:16:18.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff11553e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1100c8cf0 0x7ff1101f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:18.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff11553e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1100c90d0 0x7ff1101535c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:18.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff11553e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100ca340 0x7ff11015aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:18.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff11553e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1100c32a0 con 0x7ff1100ca340 2023-10-30T13:16:18.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff11553e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1100c5660 con 0x7ff1100c90d0 2023-10-30T13:16:18.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff11553e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1101f5f90 con 0x7ff1100c8cf0 2023-10-30T13:16:18.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff10f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100ca340 0x7ff11015aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:18.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff10f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100ca340 0x7ff11015aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51232/0 (socket says 172.21.15.125:51232) 2023-10-30T13:16:18.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff10f7fe640 1 -- 172.21.15.125:0/1172925805 learned_addr learned my addr 172.21.15.125:0/1172925805 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:18.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff10e7fc640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1100c90d0 0x7ff1101535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:18.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff10effd640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1100c8cf0 0x7ff1101f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:18.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff10f7fe640 1 -- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1100c8cf0 msgr2=0x7ff1101f5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:18.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff10f7fe640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1100c8cf0 0x7ff1101f5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:18.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff10f7fe640 1 -- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1100c90d0 msgr2=0x7ff1101535c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:18.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff10f7fe640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1100c90d0 0x7ff1101535c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:18.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff10f7fe640 1 -- 172.21.15.125:0/1172925805 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1000094d0 con 0x7ff1100ca340 2023-10-30T13:16:18.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff10e7fc640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1100c90d0 0x7ff1101535c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:18.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff10f7fe640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100ca340 0x7ff11015aa00 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7ff10800efc0 tx=0x7ff10800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:18.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff0f77fe640 1 -- 172.21.15.125:0/1172925805 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff10800ced0 con 0x7ff1100ca340 2023-10-30T13:16:18.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff0f77fe640 1 -- 172.21.15.125:0/1172925805 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff10800f040 con 0x7ff1100ca340 2023-10-30T13:16:18.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff0f77fe640 1 -- 172.21.15.125:0/1172925805 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1080146c0 con 0x7ff1100ca340 2023-10-30T13:16:18.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff11553e640 1 -- 172.21.15.125:0/1172925805 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1100c3490 con 0x7ff1100ca340 2023-10-30T13:16:18.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff11553e640 1 -- 172.21.15.125:0/1172925805 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1101f1e20 con 0x7ff1100ca340 2023-10-30T13:16:18.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.839+0000 7ff11553e640 1 -- 172.21.15.125:0/1172925805 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11004e170 con 0x7ff1100ca340 2023-10-30T13:16:18.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.843+0000 7ff0f77fe640 1 -- 172.21.15.125:0/1172925805 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff108014820 con 0x7ff1100ca340 2023-10-30T13:16:18.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.843+0000 7ff0f77fe640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0fc0664a0 0x7ff0fc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:18.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.843+0000 7ff0f77fe640 1 -- 172.21.15.125:0/1172925805 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7ff108085800 con 0x7ff1100ca340 2023-10-30T13:16:18.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.843+0000 7ff10effd640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0fc0664a0 0x7ff0fc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:18.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.843+0000 7ff10effd640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0fc0664a0 0x7ff0fc068960 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7ff1000ae000 tx=0x7ff1000a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:18.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:18.847+0000 7ff0f77fe640 1 -- 172.21.15.125:0/1172925805 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff108052220 con 0x7ff1100ca340 2023-10-30T13:16:19.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:19.027+0000 7ff11553e640 1 -- 172.21.15.125:0/1172925805 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1100ca9d0 con 0x7ff1100ca340 2023-10-30T13:16:19.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:19.031+0000 7ff0f77fe640 1 -- 172.21.15.125:0/1172925805 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 225 v225) v1 ==== 95+0+5756 (secure 0 0 0) 0x7ff108018090 con 0x7ff1100ca340 2023-10-30T13:16:19.033 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:16:19.034 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":225,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":225,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:16:16.945525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:19.034 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 225 2023-10-30T13:16:19.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:19.031+0000 7ff11553e640 1 -- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0fc0664a0 msgr2=0x7ff0fc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:19.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:19.031+0000 7ff11553e640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0fc0664a0 0x7ff0fc068960 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7ff1000ae000 tx=0x7ff1000a7000 comp rx=0 tx=0).stop 2023-10-30T13:16:19.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:19.031+0000 7ff11553e640 1 -- 172.21.15.125:0/1172925805 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100ca340 msgr2=0x7ff11015aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:19.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:19.031+0000 7ff11553e640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100ca340 0x7ff11015aa00 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7ff10800efc0 tx=0x7ff10800c490 comp rx=0 tx=0).stop 2023-10-30T13:16:19.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:19.031+0000 7ff11553e640 1 -- 172.21.15.125:0/1172925805 shutdown_connections 2023-10-30T13:16:19.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:19.031+0000 7ff11553e640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff0fc0664a0 0x7ff0fc068960 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:19.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:19.031+0000 7ff11553e640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff1100ca340 0x7ff11015aa00 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:19.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:19.031+0000 7ff11553e640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff1100c90d0 0x7ff1101535c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:19.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:19.031+0000 7ff11553e640 1 --2- 172.21.15.125:0/1172925805 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff1100c8cf0 0x7ff1101f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:19.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:19.031+0000 7ff11553e640 1 -- 172.21.15.125:0/1172925805 >> 172.21.15.125:0/1172925805 conn(0x7ff11006be40 msgr2=0x7ff1100cba90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:19.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:19.031+0000 7ff11553e640 1 -- 172.21.15.125:0/1172925805 shutdown_connections 2023-10-30T13:16:19.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:19.031+0000 7ff11553e640 1 -- 172.21.15.125:0/1172925805 wait complete. 2023-10-30T13:16:19.055 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:16:21.055 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:21.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.198+0000 7f5849a9e640 1 Processor -- start 2023-10-30T13:16:21.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.198+0000 7f5849a9e640 1 -- start start 2023-10-30T13:16:21.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5849a9e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f58440c6b80 0x7f58440bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:21.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5849a9e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58440c6f60 0x7f58440c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:21.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5849a9e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f58440c8430 0x7f584413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:21.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5849a9e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58440c0a50 con 0x7f58440c8430 2023-10-30T13:16:21.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5849a9e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f584406f6e0 con 0x7f58440c6b80 2023-10-30T13:16:21.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5849a9e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f584406f9c0 con 0x7f58440c6f60 2023-10-30T13:16:21.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5843fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f58440c8430 0x7f584413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:21.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5843fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f58440c8430 0x7f584413ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38744/0 (socket says 172.21.15.125:38744) 2023-10-30T13:16:21.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5843fff640 1 -- 172.21.15.125:0/2969809790 learned_addr learned my addr 172.21.15.125:0/2969809790 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:21.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5842ffd640 1 --2- 172.21.15.125:0/2969809790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58440c6f60 0x7f58440c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:21.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f58437fe640 1 --2- 172.21.15.125:0/2969809790 >> v2:172.21.15.136:3300/0 conn(0x7f58440c6b80 0x7f58440bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:21.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5843fff640 1 -- 172.21.15.125:0/2969809790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58440c6f60 msgr2=0x7f58440c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:21.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5843fff640 1 --2- 172.21.15.125:0/2969809790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58440c6f60 0x7f58440c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:21.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5843fff640 1 -- 172.21.15.125:0/2969809790 >> v2:172.21.15.136:3300/0 conn(0x7f58440c6b80 msgr2=0x7f58440bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:21.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5843fff640 1 --2- 172.21.15.125:0/2969809790 >> v2:172.21.15.136:3300/0 conn(0x7f58440c6b80 0x7f58440bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:21.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.202+0000 7f5843fff640 1 -- 172.21.15.125:0/2969809790 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584406fca0 con 0x7f58440c8430 2023-10-30T13:16:21.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f5843fff640 1 --2- 172.21.15.125:0/2969809790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440c8430 0x7f584413ed90 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f584406f1b0 tx=0x7f583c01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b6ea6288d031eff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:21.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f58427fc640 1 -- 172.21.15.125:0/2969809790 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c0a56d0 con 0x7f58440c8430 2023-10-30T13:16:21.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f58427fc640 1 -- 172.21.15.125:0/2969809790 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583c0a5830 con 0x7f58440c8430 2023-10-30T13:16:21.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f58427fc640 1 -- 172.21.15.125:0/2969809790 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c0a5bc0 con 0x7f58440c8430 2023-10-30T13:16:21.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f5849a9e640 1 -- 172.21.15.125:0/2969809790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440c8430 msgr2=0x7f584413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:21.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f5849a9e640 1 --2- 172.21.15.125:0/2969809790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440c8430 0x7f584413ed90 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f584406f1b0 tx=0x7f583c01c940 comp rx=0 tx=0).stop 2023-10-30T13:16:21.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f5849a9e640 1 -- 172.21.15.125:0/2969809790 shutdown_connections 2023-10-30T13:16:21.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f5849a9e640 1 --2- 172.21.15.125:0/2969809790 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440c8430 0x7f584413ed90 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:21.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f5849a9e640 1 --2- 172.21.15.125:0/2969809790 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58440c6f60 0x7f58440c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:21.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f5849a9e640 1 --2- 172.21.15.125:0/2969809790 >> v2:172.21.15.136:3300/0 conn(0x7f58440c6b80 0x7f58440bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:21.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f5849a9e640 1 -- 172.21.15.125:0/2969809790 >> 172.21.15.125:0/2969809790 conn(0x7f58440c7910 msgr2=0x7f58440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:21.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f5849a9e640 1 -- 172.21.15.125:0/2969809790 shutdown_connections 2023-10-30T13:16:21.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f5849a9e640 1 -- 172.21.15.125:0/2969809790 wait complete. 2023-10-30T13:16:21.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f5849a9e640 1 Processor -- start 2023-10-30T13:16:21.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.206+0000 7f5849a9e640 1 -- start start 2023-10-30T13:16:21.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5849a9e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58440c6b80 0x7f584406a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:21.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5849a9e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58440c6f60 0x7f584413c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:21.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5849a9e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440c8430 0x7f5844153120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:21.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5849a9e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584413d190 con 0x7f58440c8430 2023-10-30T13:16:21.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5849a9e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58441588d0 con 0x7f58440c6f60 2023-10-30T13:16:21.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5849a9e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f584413e2e0 con 0x7f58440c6b80 2023-10-30T13:16:21.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5843fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440c8430 0x7f5844153120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:21.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5843fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440c8430 0x7f5844153120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38752/0 (socket says 172.21.15.125:38752) 2023-10-30T13:16:21.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5843fff640 1 -- 172.21.15.125:0/3712019460 learned_addr learned my addr 172.21.15.125:0/3712019460 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:21.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5842ffd640 1 --2- 172.21.15.125:0/3712019460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58440c6f60 0x7f584413c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:21.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f58437fe640 1 --2- 172.21.15.125:0/3712019460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58440c6b80 0x7f584406a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:21.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5843fff640 1 -- 172.21.15.125:0/3712019460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58440c6b80 msgr2=0x7f584406a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:21.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5843fff640 1 --2- 172.21.15.125:0/3712019460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58440c6b80 0x7f584406a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:21.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5843fff640 1 -- 172.21.15.125:0/3712019460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58440c6f60 msgr2=0x7f584413c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:21.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5843fff640 1 --2- 172.21.15.125:0/3712019460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58440c6f60 0x7f584413c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:21.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5843fff640 1 -- 172.21.15.125:0/3712019460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f583c009b30 con 0x7f58440c8430 2023-10-30T13:16:21.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5843fff640 1 --2- 172.21.15.125:0/3712019460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440c8430 0x7f5844153120 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f583c0af040 tx=0x7f583c0196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:21.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f582bfff640 1 -- 172.21.15.125:0/3712019460 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c0b8880 con 0x7f58440c8430 2023-10-30T13:16:21.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f582bfff640 1 -- 172.21.15.125:0/3712019460 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583c0b8a00 con 0x7f58440c8430 2023-10-30T13:16:21.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5849a9e640 1 -- 172.21.15.125:0/3712019460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58440cab80 con 0x7f58440c8430 2023-10-30T13:16:21.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f582bfff640 1 -- 172.21.15.125:0/3712019460 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583c0b8db0 con 0x7f58440c8430 2023-10-30T13:16:21.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5849a9e640 1 -- 172.21.15.125:0/3712019460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5844159810 con 0x7f58440c8430 2023-10-30T13:16:21.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f5849a9e640 1 -- 172.21.15.125:0/3712019460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58441599a0 con 0x7f58440c8430 2023-10-30T13:16:21.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.210+0000 7f582bfff640 1 -- 172.21.15.125:0/3712019460 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f583c0b6440 con 0x7f58440c8430 2023-10-30T13:16:21.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.214+0000 7f582bfff640 1 --2- 172.21.15.125:0/3712019460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58300664a0 0x7f5830068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:21.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.214+0000 7f582bfff640 1 -- 172.21.15.125:0/3712019460 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f583c11fe60 con 0x7f58440c8430 2023-10-30T13:16:21.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.214+0000 7f58437fe640 1 --2- 172.21.15.125:0/3712019460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58300664a0 0x7f5830068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:21.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.214+0000 7f58437fe640 1 --2- 172.21.15.125:0/3712019460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58300664a0 0x7f5830068960 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f583400ba30 tx=0x7f5834008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:21.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.218+0000 7f582bfff640 1 -- 172.21.15.125:0/3712019460 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f583c0eca40 con 0x7f58440c8430 2023-10-30T13:16:21.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.398+0000 7f5849a9e640 1 -- 172.21.15.125:0/3712019460 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f584404e170 con 0x7f58440c8430 2023-10-30T13:16:21.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.398+0000 7f582bfff640 1 -- 172.21.15.125:0/3712019460 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 225 v225) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f583c0bc020 con 0x7f58440c8430 2023-10-30T13:16:21.402 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:16:21.402 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":225,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":225,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:16:16.945525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:21.402 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 225 2023-10-30T13:16:21.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.398+0000 7f5849a9e640 1 -- 172.21.15.125:0/3712019460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58300664a0 msgr2=0x7f5830068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:21.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.398+0000 7f5849a9e640 1 --2- 172.21.15.125:0/3712019460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58300664a0 0x7f5830068960 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f583400ba30 tx=0x7f5834008040 comp rx=0 tx=0).stop 2023-10-30T13:16:21.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.398+0000 7f5849a9e640 1 -- 172.21.15.125:0/3712019460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440c8430 msgr2=0x7f5844153120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:21.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.398+0000 7f5849a9e640 1 --2- 172.21.15.125:0/3712019460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440c8430 0x7f5844153120 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f583c0af040 tx=0x7f583c0196e0 comp rx=0 tx=0).stop 2023-10-30T13:16:21.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.398+0000 7f5849a9e640 1 -- 172.21.15.125:0/3712019460 shutdown_connections 2023-10-30T13:16:21.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.398+0000 7f5849a9e640 1 --2- 172.21.15.125:0/3712019460 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58300664a0 0x7f5830068960 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:21.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.398+0000 7f5849a9e640 1 --2- 172.21.15.125:0/3712019460 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58440c8430 0x7f5844153120 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:21.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.398+0000 7f5849a9e640 1 --2- 172.21.15.125:0/3712019460 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58440c6f60 0x7f584413c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:21.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.398+0000 7f5849a9e640 1 --2- 172.21.15.125:0/3712019460 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58440c6b80 0x7f584406a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:21.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.398+0000 7f5849a9e640 1 -- 172.21.15.125:0/3712019460 >> 172.21.15.125:0/3712019460 conn(0x7f58440c7910 msgr2=0x7f5844139650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:21.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.398+0000 7f5849a9e640 1 -- 172.21.15.125:0/3712019460 shutdown_connections 2023-10-30T13:16:21.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:21.398+0000 7f5849a9e640 1 -- 172.21.15.125:0/3712019460 wait complete. 2023-10-30T13:16:21.423 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:16:23.425 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:23.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.566+0000 7f3d8f3d3640 1 Processor -- start 2023-10-30T13:16:23.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.566+0000 7f3d8f3d3640 1 -- start start 2023-10-30T13:16:23.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8f3d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d88070a00 0x7f3d88070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:23.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8f3d3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d88071380 con 0x7f3d881539a0 2023-10-30T13:16:23.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8f3d3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d88153d80 con 0x7f3d88069e90 2023-10-30T13:16:23.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8f3d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8806fbb0 con 0x7f3d88070a00 2023-10-30T13:16:23.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8d16f640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f3d881539a0 0x7f3d8815a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52724/0 (socket says 172.21.15.125:52724) 2023-10-30T13:16:23.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8d16f640 1 -- 172.21.15.125:0/997176191 learned_addr learned my addr 172.21.15.125:0/997176191 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:16:23.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8c96e640 1 --2- 172.21.15.125:0/997176191 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d88070a00 0x7f3d88070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:23.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 -- 172.21.15.125:0/997176191 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561247267 0 0) 0x7f3d88071380 con 0x7f3d881539a0 2023-10-30T13:16:23.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 -- 172.21.15.125:0/997176191 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d74003530 con 0x7f3d881539a0 2023-10-30T13:16:23.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 -- 172.21.15.125:0/997176191 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3133677570 0 0) 0x7f3d74003530 con 0x7f3d881539a0 2023-10-30T13:16:23.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 -- 172.21.15.125:0/997176191 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d88070a00 msgr2=0x7f3d88070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:23.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 --2- 172.21.15.125:0/997176191 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d88070a00 0x7f3d88070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:23.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 -- 172.21.15.125:0/997176191 >> v1:172.21.15.136:6789/0 conn(0x7f3d88069e90 legacy=0x7f3d8806a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:23.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 -- 172.21.15.125:0/997176191 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d880c5c50 con 0x7f3d881539a0 2023-10-30T13:16:23.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 -- 172.21.15.125:0/997176191 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f3d84001680 con 0x7f3d881539a0 2023-10-30T13:16:23.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 -- 172.21.15.125:0/997176191 >> v1:172.21.15.125:6789/0 conn(0x7f3d881539a0 legacy=0x7f3d8815a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:23.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 --2- 172.21.15.125:0/997176191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d740039b0 0x7f3d74003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:23.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 --2- 172.21.15.125:0/997176191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d740047d0 0x7f3d74004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:23.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 --2- 172.21.15.125:0/997176191 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d88070a00 0x7f3d740051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:23.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 -- 172.21.15.125:0/997176191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d8806fbb0 con 0x7f3d740047d0 2023-10-30T13:16:23.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 -- 172.21.15.125:0/997176191 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7400bac0 con 0x7f3d740039b0 2023-10-30T13:16:23.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d7ffff640 1 -- 172.21.15.125:0/997176191 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7400bc80 con 0x7f3d88070a00 2023-10-30T13:16:23.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8c96e640 1 --2- 172.21.15.125:0/997176191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d740047d0 0x7f3d74004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:23.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8d16f640 1 --2- 172.21.15.125:0/997176191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d740039b0 0x7f3d74003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:23.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8d970640 1 --2- 172.21.15.125:0/997176191 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d88070a00 0x7f3d740051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:23.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8c96e640 1 -- 172.21.15.125:0/997176191 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d88070a00 msgr2=0x7f3d740051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:23.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8c96e640 1 --2- 172.21.15.125:0/997176191 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d88070a00 0x7f3d740051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:23.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8c96e640 1 -- 172.21.15.125:0/997176191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d740039b0 msgr2=0x7f3d74003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:23.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8c96e640 1 --2- 172.21.15.125:0/997176191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d740039b0 0x7f3d74003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:23.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8c96e640 1 -- 172.21.15.125:0/997176191 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d74003530 con 0x7f3d740047d0 2023-10-30T13:16:23.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.570+0000 7f3d8d16f640 1 --2- 172.21.15.125:0/997176191 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d740039b0 0x7f3d74003dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:23.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.574+0000 7f3d8c96e640 1 --2- 172.21.15.125:0/997176191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d740047d0 0x7f3d74004c50 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009ae0 tx=0x7f3d7801ca20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e15275be580989 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:23.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.574+0000 7f3d7ffff640 1 -- 172.21.15.125:0/997176191 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d780a6510 con 0x7f3d740047d0 2023-10-30T13:16:23.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.574+0000 7f3d7ffff640 1 -- 172.21.15.125:0/997176191 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d780a6670 con 0x7f3d740047d0 2023-10-30T13:16:23.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.574+0000 7f3d7ffff640 1 -- 172.21.15.125:0/997176191 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d780a6a20 con 0x7f3d740047d0 2023-10-30T13:16:23.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.574+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/997176191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d740047d0 msgr2=0x7f3d74004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:23.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.574+0000 7f3d8f3d3640 1 --2- 172.21.15.125:0/997176191 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d740047d0 0x7f3d74004c50 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f3d78009ae0 tx=0x7f3d7801ca20 comp rx=0 tx=0).stop 2023-10-30T13:16:23.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.574+0000 7f3d8d970640 1 -- 172.21.15.125:0/997176191 reap_dead start 2023-10-30T13:16:23.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.574+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/997176191 shutdown_connections 2023-10-30T13:16:23.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.574+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/997176191 >> 172.21.15.125:0/997176191 conn(0x7f3d880c7910 msgr2=0x7f3d880c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:23.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.574+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/997176191 shutdown_connections 2023-10-30T13:16:23.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.574+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/997176191 wait complete. 2023-10-30T13:16:23.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.574+0000 7f3d8f3d3640 1 Processor -- start 2023-10-30T13:16:23.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.574+0000 7f3d8f3d3640 1 -- start start 2023-10-30T13:16:23.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8f3d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d740047d0 0x7f3d88162440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:23.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8f3d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d88152d00 0x7f3d880c41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:23.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8f3d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d880c4710 0x7f3d881491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:23.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8f3d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d880ad8d0 con 0x7f3d880c4710 2023-10-30T13:16:23.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8f3d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d880c3d40 con 0x7f3d88152d00 2023-10-30T13:16:23.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8f3d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d880c5180 con 0x7f3d740047d0 2023-10-30T13:16:23.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8d970640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d880c4710 0x7f3d881491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:23.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8d970640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d880c4710 0x7f3d881491e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38778/0 (socket says 172.21.15.125:38778) 2023-10-30T13:16:23.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8d970640 1 -- 172.21.15.125:0/3381342602 learned_addr learned my addr 172.21.15.125:0/3381342602 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:23.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8c96e640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d88152d00 0x7f3d880c41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:23.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8d16f640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d740047d0 0x7f3d88162440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:23.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8d970640 1 -- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d740047d0 msgr2=0x7f3d88162440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:23.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8d970640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d740047d0 0x7f3d88162440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:23.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8d970640 1 -- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d88152d00 msgr2=0x7f3d880c41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:23.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8d970640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d88152d00 0x7f3d880c41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:23.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8d970640 1 -- 172.21.15.125:0/3381342602 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d880c2d60 con 0x7f3d880c4710 2023-10-30T13:16:23.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8c96e640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d88152d00 0x7f3d880c41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:16:23.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8d970640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d880c4710 0x7f3d881491e0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f3d74004750 tx=0x7f3d8000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:23.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d7dffb640 1 -- 172.21.15.125:0/3381342602 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d8000eed0 con 0x7f3d880c4710 2023-10-30T13:16:23.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d7dffb640 1 -- 172.21.15.125:0/3381342602 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d80003980 con 0x7f3d880c4710 2023-10-30T13:16:23.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d7dffb640 1 -- 172.21.15.125:0/3381342602 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d80003d30 con 0x7f3d880c4710 2023-10-30T13:16:23.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/3381342602 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d880c3190 con 0x7f3d880c4710 2023-10-30T13:16:23.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/3381342602 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d880c36f0 con 0x7f3d880c4710 2023-10-30T13:16:23.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.578+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/3381342602 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d880c3ef0 con 0x7f3d880c4710 2023-10-30T13:16:23.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.582+0000 7f3d7dffb640 1 -- 172.21.15.125:0/3381342602 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3d800050d0 con 0x7f3d880c4710 2023-10-30T13:16:23.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.582+0000 7f3d7dffb640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3d6c0664d0 0x7f3d6c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:23.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.582+0000 7f3d7dffb640 1 -- 172.21.15.125:0/3381342602 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f3d80084440 con 0x7f3d880c4710 2023-10-30T13:16:23.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.582+0000 7f3d8d16f640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3d6c0664d0 0x7f3d6c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:23.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.582+0000 7f3d8d16f640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3d6c0664d0 0x7f3d6c068990 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f3d84000c10 tx=0x7f3d84007040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:23.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.586+0000 7f3d7dffb640 1 -- 172.21.15.125:0/3381342602 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3d880c3ef0 con 0x7f3d880c4710 2023-10-30T13:16:23.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.770+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/3381342602 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d880c3ef0 con 0x7f3d880c4710 2023-10-30T13:16:23.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.770+0000 7f3d7dffb640 1 -- 172.21.15.125:0/3381342602 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 225 v225) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f3d880c3ef0 con 0x7f3d880c4710 2023-10-30T13:16:23.774 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:16:23.774 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":225,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":225,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:16:16.945525+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:23.774 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 225 2023-10-30T13:16:23.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.770+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3d6c0664d0 msgr2=0x7f3d6c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:23.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.770+0000 7f3d8f3d3640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3d6c0664d0 0x7f3d6c068990 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f3d84000c10 tx=0x7f3d84007040 comp rx=0 tx=0).stop 2023-10-30T13:16:23.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.770+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/3381342602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d880c4710 msgr2=0x7f3d881491e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:23.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.770+0000 7f3d8f3d3640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d880c4710 0x7f3d881491e0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f3d74004750 tx=0x7f3d8000eb90 comp rx=0 tx=0).stop 2023-10-30T13:16:23.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.770+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/3381342602 shutdown_connections 2023-10-30T13:16:23.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.770+0000 7f3d8f3d3640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3d6c0664d0 0x7f3d6c068990 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:23.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.770+0000 7f3d8f3d3640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3d880c4710 0x7f3d881491e0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:23.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.770+0000 7f3d8f3d3640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3d88152d00 0x7f3d880c41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:23.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.770+0000 7f3d8f3d3640 1 --2- 172.21.15.125:0/3381342602 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3d740047d0 0x7f3d88162440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:23.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.770+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/3381342602 >> 172.21.15.125:0/3381342602 conn(0x7f3d880c7910 msgr2=0x7f3d8814e950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:23.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.770+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/3381342602 shutdown_connections 2023-10-30T13:16:23.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:23.770+0000 7f3d8f3d3640 1 -- 172.21.15.125:0/3381342602 wait complete. 2023-10-30T13:16:23.795 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:16:25.795 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:25.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.938+0000 7f8330ba1640 1 Processor -- start 2023-10-30T13:16:25.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.938+0000 7f8330ba1640 1 -- start start 2023-10-30T13:16:25.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.938+0000 7f8330ba1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f832c06f7a0 0x7f832c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:25.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.938+0000 7f8330ba1640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832c066cf0 con 0x7f832c071110 2023-10-30T13:16:25.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.942+0000 7f8330ba1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f832c0c70d0 con 0x7f832c0c6bd0 2023-10-30T13:16:25.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.942+0000 7f8330ba1640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f832c0700c0 con 0x7f832c06f7a0 2023-10-30T13:16:25.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.942+0000 7f832a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f832c06f7a0 0x7f832c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:25.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.942+0000 7f832ad9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f832c071110 0x7f832c158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52736/0 (socket says 172.21.15.125:52736) 2023-10-30T13:16:25.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.942+0000 7f832ad9d640 1 -- 172.21.15.125:0/1018244560 learned_addr learned my addr 172.21.15.125:0/1018244560 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:16:25.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.942+0000 7f832a59c640 1 -- 172.21.15.125:0/1018244560 >> v1:172.21.15.136:6789/0 conn(0x7f832c0c6bd0 legacy=0x7f832c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:25.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.942+0000 7f832a59c640 1 -- 172.21.15.125:0/1018244560 >> v1:172.21.15.125:6789/0 conn(0x7f832c071110 legacy=0x7f832c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:25.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.942+0000 7f832a59c640 1 -- 172.21.15.125:0/1018244560 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f832c0c7390 con 0x7f832c06f7a0 2023-10-30T13:16:25.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.942+0000 7f8329d9b640 1 -- 172.21.15.125:0/1018244560 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586358146 0 0) 0x7f832c066cf0 con 0x7f832c071110 2023-10-30T13:16:25.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.942+0000 7f832a59c640 1 --2- 172.21.15.125:0/1018244560 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f832c06f7a0 0x7f832c06fb80 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f831c009d90 tx=0x7f831c01f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14003ae3e6340813 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:25.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.942+0000 7f8329d9b640 1 -- 172.21.15.125:0/1018244560 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c0af600 con 0x7f832c06f7a0 2023-10-30T13:16:25.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.942+0000 7f8329d9b640 1 -- 172.21.15.125:0/1018244560 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f831c0ab050 con 0x7f832c06f7a0 2023-10-30T13:16:25.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.942+0000 7f8329d9b640 1 -- 172.21.15.125:0/1018244560 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f831c0af980 con 0x7f832c06f7a0 2023-10-30T13:16:25.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 -- 172.21.15.125:0/1018244560 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f832c06f7a0 msgr2=0x7f832c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:25.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 --2- 172.21.15.125:0/1018244560 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f832c06f7a0 0x7f832c06fb80 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f831c009d90 tx=0x7f831c01f130 comp rx=0 tx=0).stop 2023-10-30T13:16:25.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 -- 172.21.15.125:0/1018244560 shutdown_connections 2023-10-30T13:16:25.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 --2- 172.21.15.125:0/1018244560 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f832c06f7a0 0x7f832c06fb80 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:25.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 -- 172.21.15.125:0/1018244560 >> 172.21.15.125:0/1018244560 conn(0x7f832c0c7910 msgr2=0x7f832c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:25.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 -- 172.21.15.125:0/1018244560 shutdown_connections 2023-10-30T13:16:25.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 -- 172.21.15.125:0/1018244560 wait complete. 2023-10-30T13:16:25.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 Processor -- start 2023-10-30T13:16:25.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 -- start start 2023-10-30T13:16:25.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f832c06f7a0 0x7f832c1554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:25.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f832c071110 0x7f832c14f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:25.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f832c0c6bd0 0x7f832c153df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:25.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c152a20 con 0x7f832c0c6bd0 2023-10-30T13:16:25.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f832c0c9020 con 0x7f832c071110 2023-10-30T13:16:25.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8330ba1640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f832c0c9300 con 0x7f832c06f7a0 2023-10-30T13:16:25.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f832ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f832c0c6bd0 0x7f832c153df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:25.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f832ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f832c0c6bd0 0x7f832c153df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38806/0 (socket says 172.21.15.125:38806) 2023-10-30T13:16:25.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f832ad9d640 1 -- 172.21.15.125:0/2215541586 learned_addr learned my addr 172.21.15.125:0/2215541586 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:25.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f832a59c640 1 --2- 172.21.15.125:0/2215541586 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f832c06f7a0 0x7f832c1554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:25.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.946+0000 7f8321d9b640 1 --2- 172.21.15.125:0/2215541586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f832c071110 0x7f832c14f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:25.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.950+0000 7f832ad9d640 1 -- 172.21.15.125:0/2215541586 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f832c06f7a0 msgr2=0x7f832c1554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:25.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.950+0000 7f832ad9d640 1 --2- 172.21.15.125:0/2215541586 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f832c06f7a0 0x7f832c1554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:25.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.950+0000 7f832ad9d640 1 -- 172.21.15.125:0/2215541586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f832c071110 msgr2=0x7f832c14f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:25.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.950+0000 7f832ad9d640 1 --2- 172.21.15.125:0/2215541586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f832c071110 0x7f832c14f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:25.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.950+0000 7f832ad9d640 1 -- 172.21.15.125:0/2215541586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831c009a70 con 0x7f832c0c6bd0 2023-10-30T13:16:25.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.950+0000 7f832ad9d640 1 --2- 172.21.15.125:0/2215541586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f832c0c6bd0 0x7f832c153df0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f832400c3c0 tx=0x7f832400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:25.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.950+0000 7f83237fe640 1 -- 172.21.15.125:0/2215541586 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83240074e0 con 0x7f832c0c6bd0 2023-10-30T13:16:25.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.950+0000 7f83237fe640 1 -- 172.21.15.125:0/2215541586 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8324007640 con 0x7f832c0c6bd0 2023-10-30T13:16:25.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.950+0000 7f83237fe640 1 -- 172.21.15.125:0/2215541586 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8324004a30 con 0x7f832c0c6bd0 2023-10-30T13:16:25.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.950+0000 7f8330ba1640 1 -- 172.21.15.125:0/2215541586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f832c152c70 con 0x7f832c0c6bd0 2023-10-30T13:16:25.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.950+0000 7f8330ba1640 1 -- 172.21.15.125:0/2215541586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832c155a30 con 0x7f832c0c6bd0 2023-10-30T13:16:25.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.950+0000 7f8330ba1640 1 -- 172.21.15.125:0/2215541586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f832c04e170 con 0x7f832c0c6bd0 2023-10-30T13:16:25.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.950+0000 7f83237fe640 1 -- 172.21.15.125:0/2215541586 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8324004b90 con 0x7f832c0c6bd0 2023-10-30T13:16:25.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.954+0000 7f83237fe640 1 --2- 172.21.15.125:0/2215541586 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f830c0664a0 0x7f830c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:25.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.954+0000 7f83237fe640 1 -- 172.21.15.125:0/2215541586 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f8324084410 con 0x7f832c0c6bd0 2023-10-30T13:16:25.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.954+0000 7f832a59c640 1 --2- 172.21.15.125:0/2215541586 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f830c0664a0 0x7f830c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:25.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.954+0000 7f832a59c640 1 --2- 172.21.15.125:0/2215541586 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f830c0664a0 0x7f830c068960 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f831c0b1000 tx=0x7f831c0aa000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:25.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:25.958+0000 7f83237fe640 1 -- 172.21.15.125:0/2215541586 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8324051000 con 0x7f832c0c6bd0 2023-10-30T13:16:26.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:26.138+0000 7f8330ba1640 1 -- 172.21.15.125:0/2215541586 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f832c153120 con 0x7f832c0c6bd0 2023-10-30T13:16:26.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:26.138+0000 7f83237fe640 1 -- 172.21.15.125:0/2215541586 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 226 v226) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f8324054620 con 0x7f832c0c6bd0 2023-10-30T13:16:26.143 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:16:26.143 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":226,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:16:22.927360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:26.144 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 226 2023-10-30T13:16:26.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:26.138+0000 7f8330ba1640 1 -- 172.21.15.125:0/2215541586 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f830c0664a0 msgr2=0x7f830c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:26.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:26.138+0000 7f8330ba1640 1 --2- 172.21.15.125:0/2215541586 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f830c0664a0 0x7f830c068960 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f831c0b1000 tx=0x7f831c0aa000 comp rx=0 tx=0).stop 2023-10-30T13:16:26.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:26.138+0000 7f8330ba1640 1 -- 172.21.15.125:0/2215541586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f832c0c6bd0 msgr2=0x7f832c153df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:26.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:26.138+0000 7f8330ba1640 1 --2- 172.21.15.125:0/2215541586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f832c0c6bd0 0x7f832c153df0 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f832400c3c0 tx=0x7f832400c880 comp rx=0 tx=0).stop 2023-10-30T13:16:26.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:26.138+0000 7f8330ba1640 1 -- 172.21.15.125:0/2215541586 shutdown_connections 2023-10-30T13:16:26.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:26.138+0000 7f8330ba1640 1 --2- 172.21.15.125:0/2215541586 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f830c0664a0 0x7f830c068960 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:26.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:26.138+0000 7f8330ba1640 1 --2- 172.21.15.125:0/2215541586 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f832c0c6bd0 0x7f832c153df0 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:26.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:26.138+0000 7f8330ba1640 1 --2- 172.21.15.125:0/2215541586 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f832c071110 0x7f832c14f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:26.138+0000 7f8330ba1640 1 --2- 172.21.15.125:0/2215541586 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f832c06f7a0 0x7f832c1554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:26.138+0000 7f8330ba1640 1 -- 172.21.15.125:0/2215541586 >> 172.21.15.125:0/2215541586 conn(0x7f832c0c7910 msgr2=0x7f832c0c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:26.138+0000 7f8330ba1640 1 -- 172.21.15.125:0/2215541586 shutdown_connections 2023-10-30T13:16:26.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:26.138+0000 7f8330ba1640 1 -- 172.21.15.125:0/2215541586 wait complete. 2023-10-30T13:16:26.164 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=2, replacing gid=8443 2023-10-30T13:16:26.165 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.f 2023-10-30T13:16:28.165 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T13:16:28.166 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T13:16:28.166 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T13:16:28.169 INFO:tasks.ceph.mds.f:Started 2023-10-30T13:16:28.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:28.248 INFO:tasks.ceph.mds.f.smithi136.stdout:starting mds.f at 2023-10-30T13:16:28.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb09bd3640 1 Processor -- start 2023-10-30T13:16:28.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb09bd3640 1 -- start start 2023-10-30T13:16:28.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb09bd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb041538a0 0x7fcb0415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:28.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb09bd3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb040cb800 con 0x7fcb040ca0e0 2023-10-30T13:16:28.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb09bd3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb0415dc30 con 0x7fcb040cf160 2023-10-30T13:16:28.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb09bd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb040cb330 con 0x7fcb041538a0 2023-10-30T13:16:28.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb02ffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcb040ca0e0 0x7fcb0415a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52750/0 (socket says 172.21.15.125:52750) 2023-10-30T13:16:28.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb02ffd640 1 -- 172.21.15.125:0/748302 learned_addr learned my addr 172.21.15.125:0/748302 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:16:28.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb037fe640 1 --2- 172.21.15.125:0/748302 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb041538a0 0x7fcb0415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:28.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 -- 172.21.15.125:0/748302 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395963856 0 0) 0x7fcb040cb800 con 0x7fcb040ca0e0 2023-10-30T13:16:28.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 -- 172.21.15.125:0/748302 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcaec003530 con 0x7fcb040ca0e0 2023-10-30T13:16:28.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 -- 172.21.15.125:0/748302 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 441150835 0 0) 0x7fcaec003530 con 0x7fcb040ca0e0 2023-10-30T13:16:28.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 -- 172.21.15.125:0/748302 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb041538a0 msgr2=0x7fcb0415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:28.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 --2- 172.21.15.125:0/748302 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb041538a0 0x7fcb0415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:28.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 -- 172.21.15.125:0/748302 >> v1:172.21.15.136:6789/0 conn(0x7fcb040cf160 legacy=0x7fcb04164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:28.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 -- 172.21.15.125:0/748302 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb040cad70 con 0x7fcb040ca0e0 2023-10-30T13:16:28.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 -- 172.21.15.125:0/748302 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fcaf8001680 con 0x7fcb040ca0e0 2023-10-30T13:16:28.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb037fe640 1 --2- 172.21.15.125:0/748302 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb041538a0 0x7fcb0415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:16:28.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 -- 172.21.15.125:0/748302 >> v1:172.21.15.125:6789/0 conn(0x7fcb040ca0e0 legacy=0x7fcb0415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:28.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 --2- 172.21.15.125:0/748302 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb041538a0 0x7fcaec0039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:28.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 --2- 172.21.15.125:0/748302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaec003ef0 0x7fcaec0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:28.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 --2- 172.21.15.125:0/748302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaec004d10 0x7fcaec005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:28.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 -- 172.21.15.125:0/748302 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb040cb330 con 0x7fcaec003ef0 2023-10-30T13:16:28.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 -- 172.21.15.125:0/748302 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaec00b9f0 con 0x7fcaec004d10 2023-10-30T13:16:28.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb027fc640 1 -- 172.21.15.125:0/748302 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaec00bbb0 con 0x7fcb041538a0 2023-10-30T13:16:28.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb02ffd640 1 --2- 172.21.15.125:0/748302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaec003ef0 0x7fcaec0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:28.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb037fe640 1 --2- 172.21.15.125:0/748302 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb041538a0 0x7fcaec0039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:28.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb03fff640 1 --2- 172.21.15.125:0/748302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaec004d10 0x7fcaec005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:28.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb02ffd640 1 -- 172.21.15.125:0/748302 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb041538a0 msgr2=0x7fcaec0039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:28.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb02ffd640 1 --2- 172.21.15.125:0/748302 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb041538a0 0x7fcaec0039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:28.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb02ffd640 1 -- 172.21.15.125:0/748302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaec004d10 msgr2=0x7fcaec005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:28.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb02ffd640 1 --2- 172.21.15.125:0/748302 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaec004d10 0x7fcaec005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:28.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.314+0000 7fcb02ffd640 1 -- 172.21.15.125:0/748302 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf4009420 con 0x7fcaec003ef0 2023-10-30T13:16:28.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.318+0000 7fcb02ffd640 1 --2- 172.21.15.125:0/748302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaec003ef0 0x7fcaec0047d0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800c1c0 tx=0x7fcaf801f2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89b6893e373c7ed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:28.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.318+0000 7fcb027fc640 1 -- 172.21.15.125:0/748302 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf80058e0 con 0x7fcaec003ef0 2023-10-30T13:16:28.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.318+0000 7fcb027fc640 1 -- 172.21.15.125:0/748302 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf8005a40 con 0x7fcaec003ef0 2023-10-30T13:16:28.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.318+0000 7fcb027fc640 1 -- 172.21.15.125:0/748302 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8005e30 con 0x7fcaec003ef0 2023-10-30T13:16:28.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.318+0000 7fcb09bd3640 1 -- 172.21.15.125:0/748302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaec003ef0 msgr2=0x7fcaec0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:28.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.318+0000 7fcb09bd3640 1 --2- 172.21.15.125:0/748302 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaec003ef0 0x7fcaec0047d0 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800c1c0 tx=0x7fcaf801f2d0 comp rx=0 tx=0).stop 2023-10-30T13:16:28.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.318+0000 7fcb03fff640 1 -- 172.21.15.125:0/748302 reap_dead start 2023-10-30T13:16:28.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.318+0000 7fcb09bd3640 1 -- 172.21.15.125:0/748302 shutdown_connections 2023-10-30T13:16:28.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.318+0000 7fcb09bd3640 1 -- 172.21.15.125:0/748302 >> 172.21.15.125:0/748302 conn(0x7fcb0406bfe0 msgr2=0x7fcb0406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:28.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.318+0000 7fcb09bd3640 1 -- 172.21.15.125:0/748302 shutdown_connections 2023-10-30T13:16:28.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.318+0000 7fcb09bd3640 1 -- 172.21.15.125:0/748302 wait complete. 2023-10-30T13:16:28.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb09bd3640 1 Processor -- start 2023-10-30T13:16:28.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb09bd3640 1 -- start start 2023-10-30T13:16:28.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb09bd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaec003ef0 0x7fcb04152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:28.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb09bd3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb041483d0 0x7fcb04153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:28.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb09bd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb041489d0 0x7fcb040b4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:28.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb09bd3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb040cee70 con 0x7fcb041483d0 2023-10-30T13:16:28.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb09bd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb040ce560 con 0x7fcaec003ef0 2023-10-30T13:16:28.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb09bd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb04146ff0 con 0x7fcb041489d0 2023-10-30T13:16:28.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb02ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb041483d0 0x7fcb04153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:28.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb02ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb041483d0 0x7fcb04153040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38824/0 (socket says 172.21.15.125:38824) 2023-10-30T13:16:28.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb02ffd640 1 -- 172.21.15.125:0/2519774190 learned_addr learned my addr 172.21.15.125:0/2519774190 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:28.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb03fff640 1 --2- 172.21.15.125:0/2519774190 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb041489d0 0x7fcb040b4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:28.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb037fe640 1 --2- 172.21.15.125:0/2519774190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaec003ef0 0x7fcb04152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:28.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb02ffd640 1 -- 172.21.15.125:0/2519774190 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb041489d0 msgr2=0x7fcb040b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:28.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb02ffd640 1 --2- 172.21.15.125:0/2519774190 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb041489d0 0x7fcb040b4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:28.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb02ffd640 1 -- 172.21.15.125:0/2519774190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaec003ef0 msgr2=0x7fcb04152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:28.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb02ffd640 1 --2- 172.21.15.125:0/2519774190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaec003ef0 0x7fcb04152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:28.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb02ffd640 1 -- 172.21.15.125:0/2519774190 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb041472d0 con 0x7fcb041483d0 2023-10-30T13:16:28.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb02ffd640 1 --2- 172.21.15.125:0/2519774190 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb041483d0 0x7fcb04153040 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800c1c0 tx=0x7fcaec005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:28.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcaebfff640 1 -- 172.21.15.125:0/2519774190 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf80b7070 con 0x7fcb041483d0 2023-10-30T13:16:28.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcaebfff640 1 -- 172.21.15.125:0/2519774190 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf8001440 con 0x7fcb041483d0 2023-10-30T13:16:28.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcaebfff640 1 -- 172.21.15.125:0/2519774190 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8007090 con 0x7fcb041483d0 2023-10-30T13:16:28.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb09bd3640 1 -- 172.21.15.125:0/2519774190 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb040ce820 con 0x7fcb041483d0 2023-10-30T13:16:28.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.322+0000 7fcb09bd3640 1 -- 172.21.15.125:0/2519774190 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb04146c80 con 0x7fcb041483d0 2023-10-30T13:16:28.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.326+0000 7fcb09bd3640 1 -- 172.21.15.125:0/2519774190 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb0404e170 con 0x7fcb041483d0 2023-10-30T13:16:28.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.326+0000 7fcaebfff640 1 -- 172.21.15.125:0/2519774190 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcaf80071f0 con 0x7fcb041483d0 2023-10-30T13:16:28.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.326+0000 7fcaebfff640 1 --2- 172.21.15.125:0/2519774190 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcaf00664a0 0x7fcaf0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:28.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.326+0000 7fcaebfff640 1 -- 172.21.15.125:0/2519774190 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7fcaf81216c0 con 0x7fcb041483d0 2023-10-30T13:16:28.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.326+0000 7fcb037fe640 1 --2- 172.21.15.125:0/2519774190 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcaf00664a0 0x7fcaf0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:28.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.326+0000 7fcb037fe640 1 --2- 172.21.15.125:0/2519774190 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcaf00664a0 0x7fcaf0068960 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009cb0 tx=0x7fcaf4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:28.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.330+0000 7fcaebfff640 1 -- 172.21.15.125:0/2519774190 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcaf80ee200 con 0x7fcb041483d0 2023-10-30T13:16:28.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.510+0000 7fcb09bd3640 1 -- 172.21.15.125:0/2519774190 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb040b1dd0 con 0x7fcb041483d0 2023-10-30T13:16:28.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.510+0000 7fcaebfff640 1 -- 172.21.15.125:0/2519774190 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 226 v226) v1 ==== 95+0+5756 (secure 0 0 0) 0x7fcaf80f1820 con 0x7fcb041483d0 2023-10-30T13:16:28.516 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:16:28.516 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":226,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:16:22.927360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:28.516 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 226 2023-10-30T13:16:28.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.514+0000 7fcb09bd3640 1 -- 172.21.15.125:0/2519774190 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcaf00664a0 msgr2=0x7fcaf0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:28.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.514+0000 7fcb09bd3640 1 --2- 172.21.15.125:0/2519774190 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcaf00664a0 0x7fcaf0068960 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009cb0 tx=0x7fcaf4008040 comp rx=0 tx=0).stop 2023-10-30T13:16:28.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.514+0000 7fcb09bd3640 1 -- 172.21.15.125:0/2519774190 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb041483d0 msgr2=0x7fcb04153040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:28.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.514+0000 7fcb09bd3640 1 --2- 172.21.15.125:0/2519774190 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb041483d0 0x7fcb04153040 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800c1c0 tx=0x7fcaec005800 comp rx=0 tx=0).stop 2023-10-30T13:16:28.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.514+0000 7fcb09bd3640 1 -- 172.21.15.125:0/2519774190 shutdown_connections 2023-10-30T13:16:28.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.514+0000 7fcb09bd3640 1 --2- 172.21.15.125:0/2519774190 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcaf00664a0 0x7fcaf0068960 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:28.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.514+0000 7fcb09bd3640 1 --2- 172.21.15.125:0/2519774190 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb041489d0 0x7fcb040b4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:28.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.514+0000 7fcb09bd3640 1 --2- 172.21.15.125:0/2519774190 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb041483d0 0x7fcb04153040 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:28.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.514+0000 7fcb09bd3640 1 --2- 172.21.15.125:0/2519774190 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaec003ef0 0x7fcb04152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:28.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.514+0000 7fcb09bd3640 1 -- 172.21.15.125:0/2519774190 >> 172.21.15.125:0/2519774190 conn(0x7fcb0406bfe0 msgr2=0x7fcb040c4070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:28.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.514+0000 7fcb09bd3640 1 -- 172.21.15.125:0/2519774190 shutdown_connections 2023-10-30T13:16:28.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:28.514+0000 7fcb09bd3640 1 -- 172.21.15.125:0/2519774190 wait complete. 2023-10-30T13:16:28.537 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T13:16:30.538 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:16:30.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f3dc79640 1 Processor -- start 2023-10-30T13:16:30.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f3dc79640 1 -- start start 2023-10-30T13:16:30.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f3dc79640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f7f381538a0 0x7f7f3815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:30.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f3dc79640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7f380ca0e0 0x7f7f3815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:30.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f3dc79640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f380cb800 con 0x7f7f380cf160 2023-10-30T13:16:30.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f3dc79640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f3815dc30 con 0x7f7f380ca0e0 2023-10-30T13:16:30.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f3dc79640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f380cb330 con 0x7f7f381538a0 2023-10-30T13:16:30.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f36ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7f380ca0e0 0x7f7f3815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:30.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f37fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7f380cf160 0x7f7f380cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:52752/0 (socket says 172.21.15.125:52752) 2023-10-30T13:16:30.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f36ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7f380ca0e0 0x7f7f3815a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38836/0 (socket says 172.21.15.125:38836) 2023-10-30T13:16:30.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f37fff640 1 -- 172.21.15.125:0/3596189505 learned_addr learned my addr 172.21.15.125:0/3596189505 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:16:30.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f377fe640 1 --2- 172.21.15.125:0/3596189505 >> v2:172.21.15.136:3300/0 conn(0x7f7f381538a0 0x7f7f3815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:30.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f36ffd640 1 -- 172.21.15.125:0/3596189505 >> v2:172.21.15.136:3300/0 conn(0x7f7f381538a0 msgr2=0x7f7f3815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:16:30.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f36ffd640 1 --2- 172.21.15.125:0/3596189505 >> v2:172.21.15.136:3300/0 conn(0x7f7f381538a0 0x7f7f3815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:30.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f36ffd640 1 -- 172.21.15.125:0/3596189505 >> v1:172.21.15.125:6789/0 conn(0x7f7f380cf160 legacy=0x7f7f380cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:30.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f36ffd640 1 -- 172.21.15.125:0/3596189505 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f380cad70 con 0x7f7f380ca0e0 2023-10-30T13:16:30.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.682+0000 7f7f367fc640 1 -- 172.21.15.125:0/3596189505 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942904064 0 0) 0x7f7f380cb800 con 0x7f7f380cf160 2023-10-30T13:16:30.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.686+0000 7f7f36ffd640 1 --2- 172.21.15.125:0/3596189505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f380ca0e0 0x7f7f3815a090 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f7f380cfd10 tx=0x7f7f2c01ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e213aa1fe9c105a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:30.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.686+0000 7f7f367fc640 1 -- 172.21.15.125:0/3596189505 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c0a8950 con 0x7f7f380ca0e0 2023-10-30T13:16:30.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.686+0000 7f7f367fc640 1 -- 172.21.15.125:0/3596189505 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2c0a8ab0 con 0x7f7f380ca0e0 2023-10-30T13:16:30.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.686+0000 7f7f367fc640 1 -- 172.21.15.125:0/3596189505 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c0a8e40 con 0x7f7f380ca0e0 2023-10-30T13:16:30.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.686+0000 7f7f3dc79640 1 -- 172.21.15.125:0/3596189505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f380ca0e0 msgr2=0x7f7f3815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:30.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.686+0000 7f7f3dc79640 1 --2- 172.21.15.125:0/3596189505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f380ca0e0 0x7f7f3815a090 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f7f380cfd10 tx=0x7f7f2c01ec50 comp rx=0 tx=0).stop 2023-10-30T13:16:30.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.686+0000 7f7f3dc79640 1 -- 172.21.15.125:0/3596189505 shutdown_connections 2023-10-30T13:16:30.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.686+0000 7f7f3dc79640 1 --2- 172.21.15.125:0/3596189505 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f380ca0e0 0x7f7f3815a090 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:30.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.686+0000 7f7f3dc79640 1 --2- 172.21.15.125:0/3596189505 >> v2:172.21.15.136:3300/0 conn(0x7f7f381538a0 0x7f7f3815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:30.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.686+0000 7f7f3dc79640 1 -- 172.21.15.125:0/3596189505 >> 172.21.15.125:0/3596189505 conn(0x7f7f3806bfe0 msgr2=0x7f7f3806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:30.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.686+0000 7f7f3dc79640 1 -- 172.21.15.125:0/3596189505 shutdown_connections 2023-10-30T13:16:30.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.686+0000 7f7f3dc79640 1 -- 172.21.15.125:0/3596189505 wait complete. 2023-10-30T13:16:30.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f3dc79640 1 Processor -- start 2023-10-30T13:16:30.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f3dc79640 1 -- start start 2023-10-30T13:16:30.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f3dc79640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f380ca0e0 0x7f7f381e5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:30.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f3dc79640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f380cf160 0x7f7f380ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:30.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f3dc79640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f381538a0 0x7f7f38145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:30.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f3dc79640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f38157dc0 con 0x7f7f380cf160 2023-10-30T13:16:30.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f3dc79640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f381e4ea0 con 0x7f7f380ca0e0 2023-10-30T13:16:30.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f3dc79640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f38158c50 con 0x7f7f381538a0 2023-10-30T13:16:30.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f36ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f380cf160 0x7f7f380ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:30.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f36ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f380cf160 0x7f7f380ccfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38848/0 (socket says 172.21.15.125:38848) 2023-10-30T13:16:30.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f36ffd640 1 -- 172.21.15.125:0/715339888 learned_addr learned my addr 172.21.15.125:0/715339888 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:16:30.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f377fe640 1 --2- 172.21.15.125:0/715339888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f380ca0e0 0x7f7f381e5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:30.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f37fff640 1 --2- 172.21.15.125:0/715339888 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f381538a0 0x7f7f38145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:30.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f36ffd640 1 -- 172.21.15.125:0/715339888 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f381538a0 msgr2=0x7f7f38145ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:30.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f36ffd640 1 --2- 172.21.15.125:0/715339888 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f381538a0 0x7f7f38145ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:30.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f36ffd640 1 -- 172.21.15.125:0/715339888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f380ca0e0 msgr2=0x7f7f381e5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:30.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f36ffd640 1 --2- 172.21.15.125:0/715339888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f380ca0e0 0x7f7f381e5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:30.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f36ffd640 1 -- 172.21.15.125:0/715339888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2c009420 con 0x7f7f380cf160 2023-10-30T13:16:30.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f36ffd640 1 --2- 172.21.15.125:0/715339888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f380cf160 0x7f7f380ccfb0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0b3040 tx=0x7f7f2c01b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:30.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f1ffff640 1 -- 172.21.15.125:0/715339888 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c0b9630 con 0x7f7f380cf160 2023-10-30T13:16:30.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f1ffff640 1 -- 172.21.15.125:0/715339888 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2c0b97b0 con 0x7f7f380cf160 2023-10-30T13:16:30.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f1ffff640 1 -- 172.21.15.125:0/715339888 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c0b9b60 con 0x7f7f380cf160 2023-10-30T13:16:30.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f3dc79640 1 -- 172.21.15.125:0/715339888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f381596a0 con 0x7f7f380cf160 2023-10-30T13:16:30.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.690+0000 7f7f3dc79640 1 -- 172.21.15.125:0/715339888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f38157800 con 0x7f7f380cf160 2023-10-30T13:16:30.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.694+0000 7f7f3dc79640 1 -- 172.21.15.125:0/715339888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f380b2a50 con 0x7f7f380cf160 2023-10-30T13:16:30.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.694+0000 7f7f1ffff640 1 -- 172.21.15.125:0/715339888 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7f2c0b7040 con 0x7f7f380cf160 2023-10-30T13:16:30.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.694+0000 7f7f1ffff640 1 --2- 172.21.15.125:0/715339888 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f240664a0 0x7f7f24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:16:30.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.694+0000 7f7f1ffff640 1 -- 172.21.15.125:0/715339888 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f7f2c122820 con 0x7f7f380cf160 2023-10-30T13:16:30.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.694+0000 7f7f377fe640 1 --2- 172.21.15.125:0/715339888 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f240664a0 0x7f7f24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:16:30.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.694+0000 7f7f377fe640 1 --2- 172.21.15.125:0/715339888 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f240664a0 0x7f7f24068960 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f7f280079c0 tx=0x7f7f28008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:16:30.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.698+0000 7f7f1ffff640 1 -- 172.21.15.125:0/715339888 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7f2c0ef240 con 0x7f7f380cf160 2023-10-30T13:16:30.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.882+0000 7f7f3dc79640 1 -- 172.21.15.125:0/715339888 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f38159830 con 0x7f7f380cf160 2023-10-30T13:16:30.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.882+0000 7f7f1ffff640 1 -- 172.21.15.125:0/715339888 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 227 v227) v1 ==== 95+0+6551 (secure 0 0 0) 0x7f7f2c0b5070 con 0x7f7f380cf160 2023-10-30T13:16:30.885 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:16:30.886 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":227,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8971,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":226,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:16:22.927360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:16:30.886 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 227 2023-10-30T13:16:30.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.882+0000 7f7f3dc79640 1 -- 172.21.15.125:0/715339888 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f240664a0 msgr2=0x7f7f24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:30.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.882+0000 7f7f3dc79640 1 --2- 172.21.15.125:0/715339888 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f240664a0 0x7f7f24068960 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f7f280079c0 tx=0x7f7f28008040 comp rx=0 tx=0).stop 2023-10-30T13:16:30.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.882+0000 7f7f3dc79640 1 -- 172.21.15.125:0/715339888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f380cf160 msgr2=0x7f7f380ccfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:16:30.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.882+0000 7f7f3dc79640 1 --2- 172.21.15.125:0/715339888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f380cf160 0x7f7f380ccfb0 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c0b3040 tx=0x7f7f2c01b9f0 comp rx=0 tx=0).stop 2023-10-30T13:16:30.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.882+0000 7f7f3dc79640 1 -- 172.21.15.125:0/715339888 shutdown_connections 2023-10-30T13:16:30.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.882+0000 7f7f3dc79640 1 --2- 172.21.15.125:0/715339888 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7f240664a0 0x7f7f24068960 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:30.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.882+0000 7f7f3dc79640 1 --2- 172.21.15.125:0/715339888 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7f381538a0 0x7f7f38145ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:30.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.882+0000 7f7f3dc79640 1 --2- 172.21.15.125:0/715339888 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7f380cf160 0x7f7f380ccfb0 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:30.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.882+0000 7f7f3dc79640 1 --2- 172.21.15.125:0/715339888 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7f380ca0e0 0x7f7f381e5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:16:30.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.882+0000 7f7f3dc79640 1 -- 172.21.15.125:0/715339888 >> 172.21.15.125:0/715339888 conn(0x7f7f3806bfe0 msgr2=0x7f7f380cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:16:30.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.882+0000 7f7f3dc79640 1 -- 172.21.15.125:0/715339888 shutdown_connections 2023-10-30T13:16:30.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:16:30.882+0000 7f7f3dc79640 1 -- 172.21.15.125:0/715339888 wait complete. 2023-10-30T13:16:30.907 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T13:16:30.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 70 secs before thrashing 2023-10-30T13:16:30.968 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:16:30.968 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.33 2023-10-30T13:16:30.969 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 33 -eq 48 ] 2023-10-30T13:16:30.969 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.33 2023-10-30T13:16:30.969 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.33 2023-10-30T13:16:30.969 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.33 2023-10-30T13:16:30.970 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:16:37.035 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:16:37.036 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.34 2023-10-30T13:16:37.037 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 34 -eq 48 ] 2023-10-30T13:16:37.037 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.34 2023-10-30T13:16:37.037 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.34 2023-10-30T13:16:37.037 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.34 2023-10-30T13:16:37.037 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:16:41.359 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:16:41.359 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 35 -eq 48 ] 2023-10-30T13:16:41.360 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.35 2023-10-30T13:16:41.360 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.35 2023-10-30T13:16:41.360 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.35 2023-10-30T13:16:41.361 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.35 2023-10-30T13:16:41.361 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:16:47.111 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:16:47.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:16:47.539 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:16:47.539 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.36 2023-10-30T13:16:47.540 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 36 -eq 48 ] 2023-10-30T13:16:47.540 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.36 2023-10-30T13:16:47.540 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.36 2023-10-30T13:16:47.540 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.36 2023-10-30T13:16:47.541 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:16:52.916 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:16:52.916 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 37 -eq 48 ] 2023-10-30T13:16:52.916 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.37 2023-10-30T13:16:52.917 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.37 2023-10-30T13:16:52.917 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.37 2023-10-30T13:16:52.918 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.37 2023-10-30T13:16:52.918 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:16:59.102 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:16:59.103 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.38 2023-10-30T13:16:59.103 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 38 -eq 48 ] 2023-10-30T13:16:59.104 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.38 2023-10-30T13:16:59.104 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.38 2023-10-30T13:16:59.104 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.38 2023-10-30T13:16:59.104 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:17:05.304 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:17:05.304 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 39 -eq 48 ] 2023-10-30T13:17:05.305 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.39 2023-10-30T13:17:05.305 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.39 2023-10-30T13:17:05.306 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.39 2023-10-30T13:17:05.306 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.39 2023-10-30T13:17:05.306 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:17:09.600 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:17:09.601 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.40 2023-10-30T13:17:09.601 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 40 -eq 48 ] 2023-10-30T13:17:09.601 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.40 2023-10-30T13:17:09.602 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.40 2023-10-30T13:17:09.602 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.40 2023-10-30T13:17:09.602 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:17:16.271 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:17:16.272 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.41 2023-10-30T13:17:16.272 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 41 -eq 48 ] 2023-10-30T13:17:16.272 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.41 2023-10-30T13:17:16.273 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.41 2023-10-30T13:17:16.273 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.41 2023-10-30T13:17:16.273 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:17:17.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:17:17.173 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:17:23.812 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:17:23.812 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 42 -eq 48 ] 2023-10-30T13:17:23.812 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.42 2023-10-30T13:17:23.813 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.42 2023-10-30T13:17:23.813 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.42 2023-10-30T13:17:23.813 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.42 2023-10-30T13:17:23.813 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:17:27.344 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:17:27.345 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 43 -eq 48 ] 2023-10-30T13:17:27.345 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.43 2023-10-30T13:17:27.345 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.43 2023-10-30T13:17:27.346 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.43 2023-10-30T13:17:27.346 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.43 2023-10-30T13:17:27.346 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:17:33.389 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:17:33.389 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 44 -eq 48 ] 2023-10-30T13:17:33.390 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.44 2023-10-30T13:17:33.390 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.44 2023-10-30T13:17:33.390 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.44 2023-10-30T13:17:33.391 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.44 2023-10-30T13:17:33.391 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:17:39.743 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:17:39.744 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 45 -eq 48 ] 2023-10-30T13:17:39.744 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.45 2023-10-30T13:17:39.745 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.45 2023-10-30T13:17:39.745 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.45 2023-10-30T13:17:39.745 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.45 2023-10-30T13:17:39.745 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:17:40.909 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:41.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.053+0000 7f720a50f640 1 Processor -- start 2023-10-30T13:17:41.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.053+0000 7f720a50f640 1 -- start start 2023-10-30T13:17:41.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.057+0000 7f720a50f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72041538a0 0x7f720415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:41.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.057+0000 7f720a50f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f72040ca0e0 0x7f720415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:41.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.057+0000 7f720a50f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72040cb800 con 0x7f72040cf160 2023-10-30T13:17:41.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.057+0000 7f720a50f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f720415dc30 con 0x7f72040ca0e0 2023-10-30T13:17:41.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.057+0000 7f720a50f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72040cb330 con 0x7f72041538a0 2023-10-30T13:17:41.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.057+0000 7f72037fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f72040ca0e0 0x7f720415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:41.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.057+0000 7f72037fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f72040ca0e0 0x7f720415a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53266/0 (socket says 172.21.15.125:53266) 2023-10-30T13:17:41.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.057+0000 7f72037fe640 1 -- 172.21.15.125:0/1567903545 learned_addr learned my addr 172.21.15.125:0/1567903545 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:41.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.057+0000 7f7203fff640 1 --2- 172.21.15.125:0/1567903545 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72041538a0 0x7f720415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:41.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.057+0000 7f72037fe640 1 -- 172.21.15.125:0/1567903545 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72041538a0 msgr2=0x7f720415d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:41.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.057+0000 7f72037fe640 1 --2- 172.21.15.125:0/1567903545 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72041538a0 0x7f720415d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.057+0000 7f72037fe640 1 -- 172.21.15.125:0/1567903545 >> v1:172.21.15.136:6789/0 conn(0x7f72040cf160 legacy=0x7f72040cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:41.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.057+0000 7f72037fe640 1 -- 172.21.15.125:0/1567903545 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72040cad70 con 0x7f72040ca0e0 2023-10-30T13:17:41.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.061+0000 7f72037fe640 1 --2- 172.21.15.125:0/1567903545 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72040ca0e0 0x7f720415a090 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009870 tx=0x7f71f801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1d847b09537d4ba server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:41.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.061+0000 7f7202ffd640 1 -- 172.21.15.125:0/1567903545 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f80a8a30 con 0x7f72040ca0e0 2023-10-30T13:17:41.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.061+0000 7f7202ffd640 1 -- 172.21.15.125:0/1567903545 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f80a8b90 con 0x7f72040ca0e0 2023-10-30T13:17:41.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.061+0000 7f7202ffd640 1 -- 172.21.15.125:0/1567903545 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f80b0440 con 0x7f72040ca0e0 2023-10-30T13:17:41.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.061+0000 7f720a50f640 1 -- 172.21.15.125:0/1567903545 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72040ca0e0 msgr2=0x7f720415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:41.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.061+0000 7f720a50f640 1 --2- 172.21.15.125:0/1567903545 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72040ca0e0 0x7f720415a090 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009870 tx=0x7f71f801ed30 comp rx=0 tx=0).stop 2023-10-30T13:17:41.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.061+0000 7f720a50f640 1 -- 172.21.15.125:0/1567903545 shutdown_connections 2023-10-30T13:17:41.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.061+0000 7f720a50f640 1 --2- 172.21.15.125:0/1567903545 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72040ca0e0 0x7f720415a090 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.061+0000 7f720a50f640 1 --2- 172.21.15.125:0/1567903545 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72041538a0 0x7f720415d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.061+0000 7f720a50f640 1 -- 172.21.15.125:0/1567903545 >> 172.21.15.125:0/1567903545 conn(0x7f720406bfe0 msgr2=0x7f720406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:41.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.061+0000 7f720a50f640 1 -- 172.21.15.125:0/1567903545 shutdown_connections 2023-10-30T13:17:41.069 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.061+0000 7f720a50f640 1 -- 172.21.15.125:0/1567903545 wait complete. 2023-10-30T13:17:41.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.061+0000 7f720a50f640 1 Processor -- start 2023-10-30T13:17:41.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f720a50f640 1 -- start start 2023-10-30T13:17:41.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f720a50f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72040ca0e0 0x7f7204158800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:41.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f720a50f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72040cf160 0x7f72041e4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:41.071 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f720a50f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72041538a0 0x7f72040b4160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:41.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f720a50f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7204159000 con 0x7f72040cf160 2023-10-30T13:17:41.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f720a50f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7204157a90 con 0x7f72041538a0 2023-10-30T13:17:41.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f720a50f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72041577b0 con 0x7f72040ca0e0 2023-10-30T13:17:41.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f72037fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72040cf160 0x7f72041e4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:41.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f72037fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72040cf160 0x7f72041e4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53268/0 (socket says 172.21.15.125:53268) 2023-10-30T13:17:41.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f72037fe640 1 -- 172.21.15.125:0/4201293948 learned_addr learned my addr 172.21.15.125:0/4201293948 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:41.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f7203fff640 1 --2- 172.21.15.125:0/4201293948 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72040ca0e0 0x7f7204158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:41.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f7208aac640 1 --2- 172.21.15.125:0/4201293948 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72041538a0 0x7f72040b4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:41.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f72037fe640 1 -- 172.21.15.125:0/4201293948 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72040ca0e0 msgr2=0x7f7204158800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:41.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f72037fe640 1 --2- 172.21.15.125:0/4201293948 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72040ca0e0 0x7f7204158800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f72037fe640 1 -- 172.21.15.125:0/4201293948 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72041538a0 msgr2=0x7f72040b4160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:41.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f72037fe640 1 --2- 172.21.15.125:0/4201293948 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72041538a0 0x7f72040b4160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f72037fe640 1 -- 172.21.15.125:0/4201293948 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f80094d0 con 0x7f72040cf160 2023-10-30T13:17:41.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f72037fe640 1 --2- 172.21.15.125:0/4201293948 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72040cf160 0x7f72041e4b20 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f71f80b3040 tx=0x7f71f801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:41.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f7200ff9640 1 -- 172.21.15.125:0/4201293948 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f80be070 con 0x7f72040cf160 2023-10-30T13:17:41.074 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f7200ff9640 1 -- 172.21.15.125:0/4201293948 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f80ba400 con 0x7f72040cf160 2023-10-30T13:17:41.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f7200ff9640 1 -- 172.21.15.125:0/4201293948 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f80ba7d0 con 0x7f72040cf160 2023-10-30T13:17:41.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f720a50f640 1 -- 172.21.15.125:0/4201293948 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72041ef320 con 0x7f72040cf160 2023-10-30T13:17:41.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f720a50f640 1 -- 172.21.15.125:0/4201293948 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72041ef8e0 con 0x7f72040cf160 2023-10-30T13:17:41.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.065+0000 7f720a50f640 1 -- 172.21.15.125:0/4201293948 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72040b2a50 con 0x7f72040cf160 2023-10-30T13:17:41.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.069+0000 7f7200ff9640 1 -- 172.21.15.125:0/4201293948 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f71f80c3410 con 0x7f72040cf160 2023-10-30T13:17:41.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.069+0000 7f7200ff9640 1 --2- 172.21.15.125:0/4201293948 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71f00664d0 0x7f71f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:41.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.069+0000 7f7200ff9640 1 -- 172.21.15.125:0/4201293948 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f71f8122430 con 0x7f72040cf160 2023-10-30T13:17:41.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.069+0000 7f7203fff640 1 --2- 172.21.15.125:0/4201293948 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71f00664d0 0x7f71f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:41.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.069+0000 7f7203fff640 1 --2- 172.21.15.125:0/4201293948 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71f00664d0 0x7f71f0068990 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f71f400b930 tx=0x7f71f4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:41.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.073+0000 7f7200ff9640 1 -- 172.21.15.125:0/4201293948 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f71f80eeee0 con 0x7f72040cf160 2023-10-30T13:17:41.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.265+0000 7f720a50f640 1 -- 172.21.15.125:0/4201293948 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f720404e170 con 0x7f72040cf160 2023-10-30T13:17:41.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.265+0000 7f7200ff9640 1 -- 172.21.15.125:0/4201293948 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6551 (secure 0 0 0) 0x7f71f80b5090 con 0x7f72040cf160 2023-10-30T13:17:41.273 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:17:41.273 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8971,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:17:24.823595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:41.273 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 229 2023-10-30T13:17:41.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.269+0000 7f720a50f640 1 -- 172.21.15.125:0/4201293948 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71f00664d0 msgr2=0x7f71f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:41.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.269+0000 7f720a50f640 1 --2- 172.21.15.125:0/4201293948 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71f00664d0 0x7f71f0068990 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f71f400b930 tx=0x7f71f4008040 comp rx=0 tx=0).stop 2023-10-30T13:17:41.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.269+0000 7f720a50f640 1 -- 172.21.15.125:0/4201293948 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72040cf160 msgr2=0x7f72041e4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:41.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.269+0000 7f720a50f640 1 --2- 172.21.15.125:0/4201293948 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72040cf160 0x7f72041e4b20 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7f71f80b3040 tx=0x7f71f801bad0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.269+0000 7f720a50f640 1 -- 172.21.15.125:0/4201293948 shutdown_connections 2023-10-30T13:17:41.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.269+0000 7f720a50f640 1 --2- 172.21.15.125:0/4201293948 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f71f00664d0 0x7f71f0068990 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.269+0000 7f720a50f640 1 --2- 172.21.15.125:0/4201293948 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f72041538a0 0x7f72040b4160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.269+0000 7f720a50f640 1 --2- 172.21.15.125:0/4201293948 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f72040cf160 0x7f72041e4b20 unknown :-1 s=CLOSED pgs=1927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.269+0000 7f720a50f640 1 --2- 172.21.15.125:0/4201293948 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f72040ca0e0 0x7f7204158800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.269+0000 7f720a50f640 1 -- 172.21.15.125:0/4201293948 >> 172.21.15.125:0/4201293948 conn(0x7f720406bfe0 msgr2=0x7f72041526f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:41.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.269+0000 7f720a50f640 1 -- 172.21.15.125:0/4201293948 shutdown_connections 2023-10-30T13:17:41.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.269+0000 7f720a50f640 1 -- 172.21.15.125:0/4201293948 wait complete. 2023-10-30T13:17:41.294 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.e (rank=0) 2023-10-30T13:17:41.294 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T13:17:41.295 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:17:41.321 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T13:17:41.322 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:41.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.493+0000 7f218c46b640 1 Processor -- start 2023-10-30T13:17:41.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.493+0000 7f218c46b640 1 -- start start 2023-10-30T13:17:41.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218c46b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f21841539a0 0x7f218415a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:41.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218c46b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2184070a00 0x7f2184070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:41.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218c46b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2184069e90 0x7f218406a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218c46b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f218406fbb0 con 0x7f2184069e90 2023-10-30T13:17:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218c46b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f218415a8c0 con 0x7f21841539a0 2023-10-30T13:17:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218c46b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2184071380 con 0x7f2184070a00 2023-10-30T13:17:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218aa08640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2184069e90 0x7f218406a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:41.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218aa08640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2184069e90 0x7f218406a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53282/0 (socket says 172.21.15.125:53282) 2023-10-30T13:17:41.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218aa08640 1 -- 172.21.15.125:0/1084846413 learned_addr learned my addr 172.21.15.125:0/1084846413 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:41.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218a207640 1 --2- 172.21.15.125:0/1084846413 >> v2:172.21.15.136:3300/0 conn(0x7f21841539a0 0x7f218415a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:41.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f2189a06640 1 --2- 172.21.15.125:0/1084846413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2184070a00 0x7f2184070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:41.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218aa08640 1 -- 172.21.15.125:0/1084846413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2184070a00 msgr2=0x7f2184070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:41.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218aa08640 1 --2- 172.21.15.125:0/1084846413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2184070a00 0x7f2184070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218aa08640 1 -- 172.21.15.125:0/1084846413 >> v2:172.21.15.136:3300/0 conn(0x7f21841539a0 msgr2=0x7f218415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:41.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218aa08640 1 --2- 172.21.15.125:0/1084846413 >> v2:172.21.15.136:3300/0 conn(0x7f21841539a0 0x7f218415a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.497+0000 7f218aa08640 1 -- 172.21.15.125:0/1084846413 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2184153d80 con 0x7f2184069e90 2023-10-30T13:17:41.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f218aa08640 1 --2- 172.21.15.125:0/1084846413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2184069e90 0x7f218406a300 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f21840c8950 tx=0x7f217c01c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d75e28d77029755 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:41.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f2189205640 1 -- 172.21.15.125:0/1084846413 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c0a56a0 con 0x7f2184069e90 2023-10-30T13:17:41.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f2189205640 1 -- 172.21.15.125:0/1084846413 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f217c0a5800 con 0x7f2184069e90 2023-10-30T13:17:41.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f2189205640 1 -- 172.21.15.125:0/1084846413 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c0a5b90 con 0x7f2184069e90 2023-10-30T13:17:41.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f218c46b640 1 -- 172.21.15.125:0/1084846413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2184069e90 msgr2=0x7f218406a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:41.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f218c46b640 1 --2- 172.21.15.125:0/1084846413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2184069e90 0x7f218406a300 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f21840c8950 tx=0x7f217c01c910 comp rx=0 tx=0).stop 2023-10-30T13:17:41.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f218c46b640 1 -- 172.21.15.125:0/1084846413 shutdown_connections 2023-10-30T13:17:41.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f218c46b640 1 --2- 172.21.15.125:0/1084846413 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2184069e90 0x7f218406a300 unknown :-1 s=CLOSED pgs=1928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f218c46b640 1 --2- 172.21.15.125:0/1084846413 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2184070a00 0x7f2184070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f218c46b640 1 --2- 172.21.15.125:0/1084846413 >> v2:172.21.15.136:3300/0 conn(0x7f21841539a0 0x7f218415a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f218c46b640 1 -- 172.21.15.125:0/1084846413 >> 172.21.15.125:0/1084846413 conn(0x7f21840c7910 msgr2=0x7f21840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:41.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f218c46b640 1 -- 172.21.15.125:0/1084846413 shutdown_connections 2023-10-30T13:17:41.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f218c46b640 1 -- 172.21.15.125:0/1084846413 wait complete. 2023-10-30T13:17:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f218c46b640 1 Processor -- start 2023-10-30T13:17:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.501+0000 7f218c46b640 1 -- start start 2023-10-30T13:17:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f218c46b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2184069e90 0x7f21841f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f218c46b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2184070a00 0x7f21841f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:41.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f218c46b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f21841539a0 0x7f21841497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:41.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f218c46b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21841f3100 con 0x7f2184070a00 2023-10-30T13:17:41.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f218c46b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f218415bdb0 con 0x7f2184069e90 2023-10-30T13:17:41.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f218c46b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21841f0630 con 0x7f21841539a0 2023-10-30T13:17:41.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f2189a06640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2184070a00 0x7f21841f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:41.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f2189a06640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2184070a00 0x7f21841f00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53296/0 (socket says 172.21.15.125:53296) 2023-10-30T13:17:41.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f2189a06640 1 -- 172.21.15.125:0/4065957827 learned_addr learned my addr 172.21.15.125:0/4065957827 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:41.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f218aa08640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f21841539a0 0x7f21841497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:41.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f218a207640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2184069e90 0x7f21841f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:41.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f2189a06640 1 -- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f21841539a0 msgr2=0x7f21841497e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:41.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f2189a06640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f21841539a0 0x7f21841497e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f2189a06640 1 -- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2184069e90 msgr2=0x7f21841f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:41.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f2189a06640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2184069e90 0x7f21841f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f2189a06640 1 -- 172.21.15.125:0/4065957827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f217c009b30 con 0x7f2184070a00 2023-10-30T13:17:41.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f218a207640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2184069e90 0x7f21841f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:17:41.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f2189a06640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2184070a00 0x7f21841f00f0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f217400d800 tx=0x7f217400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:41.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f217affd640 1 -- 172.21.15.125:0/4065957827 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21740096a0 con 0x7f2184070a00 2023-10-30T13:17:41.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f217affd640 1 -- 172.21.15.125:0/4065957827 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2174009800 con 0x7f2184070a00 2023-10-30T13:17:41.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f217affd640 1 -- 172.21.15.125:0/4065957827 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2174009b70 con 0x7f2184070a00 2023-10-30T13:17:41.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f218c46b640 1 -- 172.21.15.125:0/4065957827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f218415aee0 con 0x7f2184070a00 2023-10-30T13:17:41.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f218c46b640 1 -- 172.21.15.125:0/4065957827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f218415b480 con 0x7f2184070a00 2023-10-30T13:17:41.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f218c46b640 1 -- 172.21.15.125:0/4065957827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f218404e170 con 0x7f2184070a00 2023-10-30T13:17:41.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.505+0000 7f217affd640 1 -- 172.21.15.125:0/4065957827 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2174009cd0 con 0x7f2184070a00 2023-10-30T13:17:41.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.509+0000 7f217affd640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f21680664a0 0x7f2168068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:41.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.509+0000 7f217affd640 1 -- 172.21.15.125:0/4065957827 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f2174084f80 con 0x7f2184070a00 2023-10-30T13:17:41.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.509+0000 7f218a207640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f21680664a0 0x7f2168068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:41.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.509+0000 7f218a207640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f21680664a0 0x7f2168068960 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2180009840 tx=0x7f2180008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:41.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.513+0000 7f217affd640 1 -- 172.21.15.125:0/4065957827 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2174051ac0 con 0x7f2184070a00 2023-10-30T13:17:41.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.693+0000 7f218c46b640 1 -- 172.21.15.125:0/4065957827 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2184073e10 con 0x7f2184070a00 2023-10-30T13:17:41.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.693+0000 7f217affd640 1 -- 172.21.15.125:0/4065957827 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6551 (secure 0 0 0) 0x7f21740550e0 con 0x7f2184070a00 2023-10-30T13:17:41.698 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:17:41.698 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8971,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:17:24.823595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:41.698 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 229 2023-10-30T13:17:41.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.693+0000 7f218c46b640 1 -- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f21680664a0 msgr2=0x7f2168068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:41.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.693+0000 7f218c46b640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f21680664a0 0x7f2168068960 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f2180009840 tx=0x7f2180008040 comp rx=0 tx=0).stop 2023-10-30T13:17:41.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.693+0000 7f218c46b640 1 -- 172.21.15.125:0/4065957827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2184070a00 msgr2=0x7f21841f00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:41.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.693+0000 7f218c46b640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2184070a00 0x7f21841f00f0 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f217400d800 tx=0x7f217400dcc0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.693+0000 7f218c46b640 1 -- 172.21.15.125:0/4065957827 shutdown_connections 2023-10-30T13:17:41.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.693+0000 7f218c46b640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f21680664a0 0x7f2168068960 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.693+0000 7f218c46b640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f21841539a0 0x7f21841497e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.693+0000 7f218c46b640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2184070a00 0x7f21841f00f0 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.693+0000 7f218c46b640 1 --2- 172.21.15.125:0/4065957827 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2184069e90 0x7f21841f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:41.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.693+0000 7f218c46b640 1 -- 172.21.15.125:0/4065957827 >> 172.21.15.125:0/4065957827 conn(0x7f21840c7910 msgr2=0x7f218414dc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:41.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.693+0000 7f218c46b640 1 -- 172.21.15.125:0/4065957827 shutdown_connections 2023-10-30T13:17:41.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:41.693+0000 7f218c46b640 1 -- 172.21.15.125:0/4065957827 wait complete. 2023-10-30T13:17:41.719 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:17:43.720 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:43.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf9b3e640 1 Processor -- start 2023-10-30T13:17:43.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf9b3e640 1 -- start start 2023-10-30T13:17:43.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf9b3e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcaf406f7a0 0x7fcaf406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:43.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf9b3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcaf40c6bd0 0x7fcaf4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:43.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf9b3e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fcaf4071110 0x7fcaf4158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:43.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf9b3e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf40700c0 con 0x7fcaf406f7a0 2023-10-30T13:17:43.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf9b3e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcaf406a3d0 con 0x7fcaf4071110 2023-10-30T13:17:43.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf9b3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf4066cf0 con 0x7fcaf40c6bd0 2023-10-30T13:17:43.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcaf406f7a0 0x7fcaf406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:43.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcaf406f7a0 0x7fcaf406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53312/0 (socket says 172.21.15.125:53312) 2023-10-30T13:17:43.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf37fe640 1 -- 172.21.15.125:0/1529048064 learned_addr learned my addr 172.21.15.125:0/1529048064 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:43.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf2ffd640 1 --2- 172.21.15.125:0/1529048064 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcaf40c6bd0 0x7fcaf4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:43.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf3fff640 1 --2- 172.21.15.125:0/1529048064 >> v2:172.21.15.136:3300/0 conn(0x7fcaf4071110 0x7fcaf4158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:43.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf37fe640 1 -- 172.21.15.125:0/1529048064 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcaf40c6bd0 msgr2=0x7fcaf4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:43.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf37fe640 1 --2- 172.21.15.125:0/1529048064 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcaf40c6bd0 0x7fcaf4069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:43.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf37fe640 1 -- 172.21.15.125:0/1529048064 >> v2:172.21.15.136:3300/0 conn(0x7fcaf4071110 msgr2=0x7fcaf4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:43.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf37fe640 1 --2- 172.21.15.125:0/1529048064 >> v2:172.21.15.136:3300/0 conn(0x7fcaf4071110 0x7fcaf4158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:43.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.861+0000 7fcaf37fe640 1 -- 172.21.15.125:0/1529048064 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf40c70d0 con 0x7fcaf406f7a0 2023-10-30T13:17:43.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.865+0000 7fcaf37fe640 1 --2- 172.21.15.125:0/1529048064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaf406f7a0 0x7fcaf406fb80 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fcaf406c7d0 tx=0x7fcae401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=198e9fca5eff2be7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:43.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.865+0000 7fcaf27fc640 1 -- 172.21.15.125:0/1529048064 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae40a89b0 con 0x7fcaf406f7a0 2023-10-30T13:17:43.872 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.865+0000 7fcaf27fc640 1 -- 172.21.15.125:0/1529048064 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae40a8b10 con 0x7fcaf406f7a0 2023-10-30T13:17:43.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.865+0000 7fcaf27fc640 1 -- 172.21.15.125:0/1529048064 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae40a8ea0 con 0x7fcaf406f7a0 2023-10-30T13:17:43.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.865+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/1529048064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaf406f7a0 msgr2=0x7fcaf406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:43.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.865+0000 7fcaf9b3e640 1 --2- 172.21.15.125:0/1529048064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaf406f7a0 0x7fcaf406fb80 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fcaf406c7d0 tx=0x7fcae401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:17:43.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.865+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/1529048064 shutdown_connections 2023-10-30T13:17:43.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.865+0000 7fcaf9b3e640 1 --2- 172.21.15.125:0/1529048064 >> v2:172.21.15.136:3300/0 conn(0x7fcaf4071110 0x7fcaf4158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:43.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.865+0000 7fcaf9b3e640 1 --2- 172.21.15.125:0/1529048064 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcaf40c6bd0 0x7fcaf4069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:43.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.865+0000 7fcaf9b3e640 1 --2- 172.21.15.125:0/1529048064 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaf406f7a0 0x7fcaf406fb80 unknown :-1 s=CLOSED pgs=1930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:43.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.865+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/1529048064 >> 172.21.15.125:0/1529048064 conn(0x7fcaf40c7910 msgr2=0x7fcaf40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:43.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.865+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/1529048064 shutdown_connections 2023-10-30T13:17:43.875 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.865+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/1529048064 wait complete. 2023-10-30T13:17:43.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf9b3e640 1 Processor -- start 2023-10-30T13:17:43.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf9b3e640 1 -- start start 2023-10-30T13:17:43.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf9b3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaf406f7a0 0x7fcaf415b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:43.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf9b3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaf4071110 0x7fcaf414c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:43.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf9b3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcaf40c6bd0 0x7fcaf41ed4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:43.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf9b3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf414b850 con 0x7fcaf406f7a0 2023-10-30T13:17:43.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf9b3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf41eda10 con 0x7fcaf4071110 2023-10-30T13:17:43.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf9b3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf415add0 con 0x7fcaf40c6bd0 2023-10-30T13:17:43.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaf406f7a0 0x7fcaf415b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:43.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaf406f7a0 0x7fcaf415b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53322/0 (socket says 172.21.15.125:53322) 2023-10-30T13:17:43.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf37fe640 1 -- 172.21.15.125:0/3685794806 learned_addr learned my addr 172.21.15.125:0/3685794806 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:43.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf3fff640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcaf40c6bd0 0x7fcaf41ed4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:43.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf2ffd640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaf4071110 0x7fcaf414c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:43.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf37fe640 1 -- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcaf40c6bd0 msgr2=0x7fcaf41ed4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:43.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf37fe640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcaf40c6bd0 0x7fcaf41ed4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:43.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf37fe640 1 -- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaf4071110 msgr2=0x7fcaf414c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:43.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf37fe640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaf4071110 0x7fcaf414c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:43.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf37fe640 1 -- 172.21.15.125:0/3685794806 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae40094d0 con 0x7fcaf406f7a0 2023-10-30T13:17:43.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf2ffd640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaf4071110 0x7fcaf414c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:17:43.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf3fff640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcaf40c6bd0 0x7fcaf41ed4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:17:43.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf37fe640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaf406f7a0 0x7fcaf415b870 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fcae40b3040 tx=0x7fcae401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:43.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcadbfff640 1 -- 172.21.15.125:0/3685794806 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae40ba600 con 0x7fcaf406f7a0 2023-10-30T13:17:43.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcadbfff640 1 -- 172.21.15.125:0/3685794806 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcae40ba760 con 0x7fcaf406f7a0 2023-10-30T13:17:43.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/3685794806 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf414b2b0 con 0x7fcaf406f7a0 2023-10-30T13:17:43.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcadbfff640 1 -- 172.21.15.125:0/3685794806 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae40bab10 con 0x7fcaf406f7a0 2023-10-30T13:17:43.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.869+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/3685794806 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf415b070 con 0x7fcaf406f7a0 2023-10-30T13:17:43.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.873+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/3685794806 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf415ac30 con 0x7fcaf406f7a0 2023-10-30T13:17:43.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.873+0000 7fcadbfff640 1 -- 172.21.15.125:0/3685794806 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcae40bac70 con 0x7fcaf406f7a0 2023-10-30T13:17:43.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.873+0000 7fcadbfff640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcae00664a0 0x7fcae0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:43.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.873+0000 7fcadbfff640 1 -- 172.21.15.125:0/3685794806 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7fcae4122df0 con 0x7fcaf406f7a0 2023-10-30T13:17:43.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.873+0000 7fcaf2ffd640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcae00664a0 0x7fcae0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:43.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.873+0000 7fcaf2ffd640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcae00664a0 0x7fcae0068960 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fcaf40c4640 tx=0x7fcae8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:43.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:43.877+0000 7fcadbfff640 1 -- 172.21.15.125:0/3685794806 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcaf415ac30 con 0x7fcaf406f7a0 2023-10-30T13:17:44.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:44.057+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/3685794806 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcaf415ac30 con 0x7fcaf406f7a0 2023-10-30T13:17:44.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:44.057+0000 7fcadbfff640 1 -- 172.21.15.125:0/3685794806 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6551 (secure 0 0 0) 0x7fcaf415ac30 con 0x7fcaf406f7a0 2023-10-30T13:17:44.064 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:17:44.064 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8971,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:17:24.823595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:44.064 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 229 2023-10-30T13:17:44.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:44.061+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcae00664a0 msgr2=0x7fcae0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:44.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:44.061+0000 7fcaf9b3e640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcae00664a0 0x7fcae0068960 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fcaf40c4640 tx=0x7fcae8008040 comp rx=0 tx=0).stop 2023-10-30T13:17:44.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:44.061+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/3685794806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaf406f7a0 msgr2=0x7fcaf415b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:44.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:44.061+0000 7fcaf9b3e640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaf406f7a0 0x7fcaf415b870 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7fcae40b3040 tx=0x7fcae401ba90 comp rx=0 tx=0).stop 2023-10-30T13:17:44.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:44.061+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/3685794806 shutdown_connections 2023-10-30T13:17:44.066 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:44.061+0000 7fcaf9b3e640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcae00664a0 0x7fcae0068960 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:44.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:44.061+0000 7fcaf9b3e640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcaf40c6bd0 0x7fcaf41ed4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:44.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:44.061+0000 7fcaf9b3e640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcaf4071110 0x7fcaf414c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:44.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:44.061+0000 7fcaf9b3e640 1 --2- 172.21.15.125:0/3685794806 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcaf406f7a0 0x7fcaf415b870 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:44.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:44.061+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/3685794806 >> 172.21.15.125:0/3685794806 conn(0x7fcaf40c7910 msgr2=0x7fcaf4150180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:44.067 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:44.061+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/3685794806 shutdown_connections 2023-10-30T13:17:44.068 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:44.061+0000 7fcaf9b3e640 1 -- 172.21.15.125:0/3685794806 wait complete. 2023-10-30T13:17:44.085 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:17:46.086 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:46.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d5cdc640 1 Processor -- start 2023-10-30T13:17:46.233 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d5cdc640 1 -- start start 2023-10-30T13:17:46.234 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d5cdc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f56d015cd50 0x7f56d00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:46.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d5cdc640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d015a6c0 con 0x7f56d00cefa0 2023-10-30T13:17:46.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d5cdc640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d015d300 con 0x7f56d01538f0 2023-10-30T13:17:46.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d5cdc640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d015d5e0 con 0x7f56d015cd50 2023-10-30T13:17:46.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d54db640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f56d00cefa0 0x7f56d01640e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36326/0 (socket says 172.21.15.125:36326) 2023-10-30T13:17:46.235 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d54db640 1 -- 172.21.15.125:0/3910643306 learned_addr learned my addr 172.21.15.125:0/3910643306 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:17:46.236 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cffff640 1 --2- 172.21.15.125:0/3910643306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f56d015cd50 0x7f56d00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:46.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 -- 172.21.15.125:0/3910643306 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092609296 0 0) 0x7f56d015a6c0 con 0x7f56d00cefa0 2023-10-30T13:17:46.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 -- 172.21.15.125:0/3910643306 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56c0003530 con 0x7f56d00cefa0 2023-10-30T13:17:46.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 -- 172.21.15.125:0/3910643306 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2369813539 0 0) 0x7f56c0003530 con 0x7f56d00cefa0 2023-10-30T13:17:46.238 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 -- 172.21.15.125:0/3910643306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f56d015cd50 msgr2=0x7f56d00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:46.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 --2- 172.21.15.125:0/3910643306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f56d015cd50 0x7f56d00c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:46.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 -- 172.21.15.125:0/3910643306 >> v1:172.21.15.136:6789/0 conn(0x7f56d01538f0 legacy=0x7f56d0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:46.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 -- 172.21.15.125:0/3910643306 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d015d8c0 con 0x7f56d00cefa0 2023-10-30T13:17:46.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 -- 172.21.15.125:0/3910643306 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f56c8001a00 con 0x7f56d00cefa0 2023-10-30T13:17:46.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 -- 172.21.15.125:0/3910643306 >> v1:172.21.15.125:6789/0 conn(0x7f56d00cefa0 legacy=0x7f56d01640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:46.239 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 --2- 172.21.15.125:0/3910643306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f56d015cd50 0x7f56c00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:46.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 --2- 172.21.15.125:0/3910643306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56c0003ef0 0x7f56c00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:46.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 --2- 172.21.15.125:0/3910643306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c0004d10 0x7f56c0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:46.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 -- 172.21.15.125:0/3910643306 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d015d5e0 con 0x7f56c0004d10 2023-10-30T13:17:46.240 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 -- 172.21.15.125:0/3910643306 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c000b9f0 con 0x7f56c0003ef0 2023-10-30T13:17:46.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cf7fe640 1 -- 172.21.15.125:0/3910643306 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c000bbb0 con 0x7f56d015cd50 2023-10-30T13:17:46.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d54db640 1 --2- 172.21.15.125:0/3910643306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c0004d10 0x7f56c0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:46.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d4cda640 1 --2- 172.21.15.125:0/3910643306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f56d015cd50 0x7f56c00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:46.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56cffff640 1 --2- 172.21.15.125:0/3910643306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56c0003ef0 0x7f56c00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:46.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d54db640 1 -- 172.21.15.125:0/3910643306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f56d015cd50 msgr2=0x7f56c00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:46.241 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d54db640 1 --2- 172.21.15.125:0/3910643306 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f56d015cd50 0x7f56c00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:46.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d54db640 1 -- 172.21.15.125:0/3910643306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56c0003ef0 msgr2=0x7f56c00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:46.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d54db640 1 --2- 172.21.15.125:0/3910643306 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56c0003ef0 0x7f56c00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:46.242 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.229+0000 7f56d54db640 1 -- 172.21.15.125:0/3910643306 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c0003530 con 0x7f56c0004d10 2023-10-30T13:17:46.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.233+0000 7f56d54db640 1 --2- 172.21.15.125:0/3910643306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c0004d10 0x7f56c0005190 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f56c800a7d0 tx=0x7f56c801d2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c18bb9f472a3fac server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:46.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.233+0000 7f56cf7fe640 1 -- 172.21.15.125:0/3910643306 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c80a5ea0 con 0x7f56c0004d10 2023-10-30T13:17:46.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.233+0000 7f56cf7fe640 1 -- 172.21.15.125:0/3910643306 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c80a8070 con 0x7f56c0004d10 2023-10-30T13:17:46.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.233+0000 7f56cf7fe640 1 -- 172.21.15.125:0/3910643306 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c80ad8d0 con 0x7f56c0004d10 2023-10-30T13:17:46.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.233+0000 7f56d5cdc640 1 -- 172.21.15.125:0/3910643306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c0004d10 msgr2=0x7f56c0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:46.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.233+0000 7f56d5cdc640 1 --2- 172.21.15.125:0/3910643306 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c0004d10 0x7f56c0005190 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f56c800a7d0 tx=0x7f56c801d2c0 comp rx=0 tx=0).stop 2023-10-30T13:17:46.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.233+0000 7f56d54db640 1 -- 172.21.15.125:0/3910643306 reap_dead start 2023-10-30T13:17:46.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.233+0000 7f56d5cdc640 1 -- 172.21.15.125:0/3910643306 shutdown_connections 2023-10-30T13:17:46.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.233+0000 7f56d5cdc640 1 -- 172.21.15.125:0/3910643306 >> 172.21.15.125:0/3910643306 conn(0x7f56d006a640 msgr2=0x7f56d0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:46.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.233+0000 7f56d5cdc640 1 -- 172.21.15.125:0/3910643306 shutdown_connections 2023-10-30T13:17:46.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.233+0000 7f56d5cdc640 1 -- 172.21.15.125:0/3910643306 wait complete. 2023-10-30T13:17:46.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d5cdc640 1 Processor -- start 2023-10-30T13:17:46.247 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d5cdc640 1 -- start start 2023-10-30T13:17:46.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d5cdc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c0004d10 0x7f56d0150f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:46.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d5cdc640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f56d0158880 0x7f56d01efa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:46.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d5cdc640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56d01f26e0 0x7f56d01f2b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:46.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d5cdc640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d0151ae0 con 0x7f56c0004d10 2023-10-30T13:17:46.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d5cdc640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d01ef5b0 con 0x7f56d01f26e0 2023-10-30T13:17:46.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d5cdc640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d0151760 con 0x7f56d0158880 2023-10-30T13:17:46.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d4cda640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c0004d10 0x7f56d0150f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:46.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d4cda640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c0004d10 0x7f56d0150f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53352/0 (socket says 172.21.15.125:53352) 2023-10-30T13:17:46.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d4cda640 1 -- 172.21.15.125:0/2046904005 learned_addr learned my addr 172.21.15.125:0/2046904005 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:46.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d54db640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56d01f26e0 0x7f56d01f2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:46.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56cffff640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f56d0158880 0x7f56d01efa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:46.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d4cda640 1 -- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f56d0158880 msgr2=0x7f56d01efa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:46.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d4cda640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f56d0158880 0x7f56d01efa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:46.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d4cda640 1 -- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56d01f26e0 msgr2=0x7f56d01f2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:46.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d4cda640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56d01f26e0 0x7f56d01f2b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:46.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d4cda640 1 -- 172.21.15.125:0/2046904005 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d01ef280 con 0x7f56c0004d10 2023-10-30T13:17:46.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d54db640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56d01f26e0 0x7f56d01f2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:17:46.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d4cda640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c0004d10 0x7f56d0150f40 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f56c400f840 tx=0x7f56c400fd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:46.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56cd7fa640 1 -- 172.21.15.125:0/2046904005 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c40052e0 con 0x7f56c0004d10 2023-10-30T13:17:46.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56cd7fa640 1 -- 172.21.15.125:0/2046904005 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c4005440 con 0x7f56c0004d10 2023-10-30T13:17:46.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56cd7fa640 1 -- 172.21.15.125:0/2046904005 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c400d970 con 0x7f56c0004d10 2023-10-30T13:17:46.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d5cdc640 1 -- 172.21.15.125:0/2046904005 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d0152e30 con 0x7f56c0004d10 2023-10-30T13:17:46.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.237+0000 7f56d5cdc640 1 -- 172.21.15.125:0/2046904005 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d0151480 con 0x7f56c0004d10 2023-10-30T13:17:46.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.241+0000 7f56d5cdc640 1 -- 172.21.15.125:0/2046904005 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d004e170 con 0x7f56c0004d10 2023-10-30T13:17:46.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.241+0000 7f56cd7fa640 1 -- 172.21.15.125:0/2046904005 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f56c400dad0 con 0x7f56c0004d10 2023-10-30T13:17:46.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.241+0000 7f56cd7fa640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f56b40664d0 0x7f56b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:46.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.241+0000 7f56cd7fa640 1 -- 172.21.15.125:0/2046904005 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f56c40847b0 con 0x7f56c0004d10 2023-10-30T13:17:46.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.241+0000 7f56cffff640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f56b40664d0 0x7f56b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:46.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.241+0000 7f56cffff640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f56b40664d0 0x7f56b4068990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f56d014fd60 tx=0x7f56bc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:46.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.245+0000 7f56cd7fa640 1 -- 172.21.15.125:0/2046904005 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f56c40513a0 con 0x7f56c0004d10 2023-10-30T13:17:46.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.429+0000 7f56d5cdc640 1 -- 172.21.15.125:0/2046904005 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56d00cc110 con 0x7f56c0004d10 2023-10-30T13:17:46.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.429+0000 7f56cd7fa640 1 -- 172.21.15.125:0/2046904005 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6551 (secure 0 0 0) 0x7f56c40549c0 con 0x7f56c0004d10 2023-10-30T13:17:46.436 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:17:46.437 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8971,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:17:24.823595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:46.437 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 229 2023-10-30T13:17:46.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.433+0000 7f56d5cdc640 1 -- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f56b40664d0 msgr2=0x7f56b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:46.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.433+0000 7f56d5cdc640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f56b40664d0 0x7f56b4068990 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f56d014fd60 tx=0x7f56bc008040 comp rx=0 tx=0).stop 2023-10-30T13:17:46.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.433+0000 7f56d5cdc640 1 -- 172.21.15.125:0/2046904005 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c0004d10 msgr2=0x7f56d0150f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:46.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.433+0000 7f56d5cdc640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c0004d10 0x7f56d0150f40 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f56c400f840 tx=0x7f56c400fd00 comp rx=0 tx=0).stop 2023-10-30T13:17:46.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.433+0000 7f56d5cdc640 1 -- 172.21.15.125:0/2046904005 shutdown_connections 2023-10-30T13:17:46.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.433+0000 7f56d5cdc640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f56b40664d0 0x7f56b4068990 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:46.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.433+0000 7f56d5cdc640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f56d01f26e0 0x7f56d01f2b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:46.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.433+0000 7f56d5cdc640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f56d0158880 0x7f56d01efa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:46.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.433+0000 7f56d5cdc640 1 --2- 172.21.15.125:0/2046904005 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f56c0004d10 0x7f56d0150f40 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:46.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.433+0000 7f56d5cdc640 1 -- 172.21.15.125:0/2046904005 >> 172.21.15.125:0/2046904005 conn(0x7f56d006a640 msgr2=0x7f56d0148e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:46.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.433+0000 7f56d5cdc640 1 -- 172.21.15.125:0/2046904005 shutdown_connections 2023-10-30T13:17:46.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:46.433+0000 7f56d5cdc640 1 -- 172.21.15.125:0/2046904005 wait complete. 2023-10-30T13:17:46.458 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:17:47.241 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:17:47.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:17:48.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:48.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.601+0000 7f696646b640 1 Processor -- start 2023-10-30T13:17:48.607 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.601+0000 7f696646b640 1 -- start start 2023-10-30T13:17:48.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.601+0000 7f696646b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f696006f7a0 0x7f696006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:48.608 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.601+0000 7f696646b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6960071110 0x7f6960158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:48.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.601+0000 7f696646b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6960066cf0 con 0x7f69600c6bd0 2023-10-30T13:17:48.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.601+0000 7f696646b640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69600700c0 con 0x7f6960071110 2023-10-30T13:17:48.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.601+0000 7f696646b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69600c70d0 con 0x7f696006f7a0 2023-10-30T13:17:48.609 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.601+0000 7f695f7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f69600c6bd0 0x7f6960069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36336/0 (socket says 172.21.15.125:36336) 2023-10-30T13:17:48.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.601+0000 7f695f7fe640 1 -- 172.21.15.125:0/154167666 learned_addr learned my addr 172.21.15.125:0/154167666 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:17:48.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.601+0000 7f6964a08640 1 --2- 172.21.15.125:0/154167666 >> v2:172.21.15.136:3300/0 conn(0x7f6960071110 0x7f6960158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:48.610 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.605+0000 7f695ffff640 1 --2- 172.21.15.125:0/154167666 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f696006f7a0 0x7f696006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:48.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.605+0000 7f695effd640 1 -- 172.21.15.125:0/154167666 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848783937 0 0) 0x7f6960066cf0 con 0x7f69600c6bd0 2023-10-30T13:17:48.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.605+0000 7f695effd640 1 -- 172.21.15.125:0/154167666 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6948003530 con 0x7f69600c6bd0 2023-10-30T13:17:48.611 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.605+0000 7f6964a08640 1 -- 172.21.15.125:0/154167666 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f696006f7a0 msgr2=0x7f696006fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:48.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.605+0000 7f6964a08640 1 --2- 172.21.15.125:0/154167666 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f696006f7a0 0x7f696006fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:48.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.605+0000 7f6964a08640 1 -- 172.21.15.125:0/154167666 >> v1:172.21.15.125:6789/0 conn(0x7f69600c6bd0 legacy=0x7f6960069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:48.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.605+0000 7f6964a08640 1 -- 172.21.15.125:0/154167666 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69600c7390 con 0x7f6960071110 2023-10-30T13:17:48.612 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.605+0000 7f695ffff640 1 --2- 172.21.15.125:0/154167666 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f696006f7a0 0x7f696006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:17:48.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.605+0000 7f6964a08640 1 --2- 172.21.15.125:0/154167666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6960071110 0x7f6960158110 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f6958009d90 tx=0x7f695801f130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb891b745eb15abb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:48.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.605+0000 7f695effd640 1 -- 172.21.15.125:0/154167666 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69580af600 con 0x7f6960071110 2023-10-30T13:17:48.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f695effd640 1 -- 172.21.15.125:0/154167666 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69580ab050 con 0x7f6960071110 2023-10-30T13:17:48.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f695effd640 1 -- 172.21.15.125:0/154167666 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69580af960 con 0x7f6960071110 2023-10-30T13:17:48.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 -- 172.21.15.125:0/154167666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6960071110 msgr2=0x7f6960158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:48.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 --2- 172.21.15.125:0/154167666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6960071110 0x7f6960158110 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f6958009d90 tx=0x7f695801f130 comp rx=0 tx=0).stop 2023-10-30T13:17:48.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 -- 172.21.15.125:0/154167666 shutdown_connections 2023-10-30T13:17:48.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 --2- 172.21.15.125:0/154167666 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6960071110 0x7f6960158110 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:48.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 --2- 172.21.15.125:0/154167666 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f696006f7a0 0x7f696006fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:48.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 -- 172.21.15.125:0/154167666 >> 172.21.15.125:0/154167666 conn(0x7f69600c7910 msgr2=0x7f69600c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:48.615 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 -- 172.21.15.125:0/154167666 shutdown_connections 2023-10-30T13:17:48.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 -- 172.21.15.125:0/154167666 wait complete. 2023-10-30T13:17:48.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 Processor -- start 2023-10-30T13:17:48.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 -- start start 2023-10-30T13:17:48.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f696006f7a0 0x7f696014fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:48.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6960071110 0x7f69600c3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:48.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69600c6bd0 0x7f696014c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:48.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6960150620 con 0x7f696006f7a0 2023-10-30T13:17:48.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696014d6f0 con 0x7f6960071110 2023-10-30T13:17:48.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f696646b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696014d9d0 con 0x7f69600c6bd0 2023-10-30T13:17:48.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f695ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f696006f7a0 0x7f696014fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:48.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f695ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f696006f7a0 0x7f696014fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53368/0 (socket says 172.21.15.125:53368) 2023-10-30T13:17:48.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.609+0000 7f695ffff640 1 -- 172.21.15.125:0/2596752096 learned_addr learned my addr 172.21.15.125:0/2596752096 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:48.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f695f7fe640 1 --2- 172.21.15.125:0/2596752096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6960071110 0x7f69600c3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:48.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f6964a08640 1 --2- 172.21.15.125:0/2596752096 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69600c6bd0 0x7f696014c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:48.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f695ffff640 1 -- 172.21.15.125:0/2596752096 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69600c6bd0 msgr2=0x7f696014c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:48.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f695ffff640 1 --2- 172.21.15.125:0/2596752096 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69600c6bd0 0x7f696014c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:48.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f695ffff640 1 -- 172.21.15.125:0/2596752096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6960071110 msgr2=0x7f69600c3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:48.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f695ffff640 1 --2- 172.21.15.125:0/2596752096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6960071110 0x7f69600c3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:48.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f695ffff640 1 -- 172.21.15.125:0/2596752096 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6950009420 con 0x7f696006f7a0 2023-10-30T13:17:48.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f695ffff640 1 --2- 172.21.15.125:0/2596752096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f696006f7a0 0x7f696014fe00 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f695000e980 tx=0x7f695000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:48.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f695cff9640 1 -- 172.21.15.125:0/2596752096 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695000ca00 con 0x7f696006f7a0 2023-10-30T13:17:48.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f695cff9640 1 -- 172.21.15.125:0/2596752096 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695000cb60 con 0x7f696006f7a0 2023-10-30T13:17:48.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f695cff9640 1 -- 172.21.15.125:0/2596752096 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6950013470 con 0x7f696006f7a0 2023-10-30T13:17:48.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f696646b640 1 -- 172.21.15.125:0/2596752096 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6958009a70 con 0x7f696006f7a0 2023-10-30T13:17:48.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f696646b640 1 -- 172.21.15.125:0/2596752096 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696014df90 con 0x7f696006f7a0 2023-10-30T13:17:48.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f696646b640 1 -- 172.21.15.125:0/2596752096 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696004e170 con 0x7f696006f7a0 2023-10-30T13:17:48.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f695cff9640 1 -- 172.21.15.125:0/2596752096 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f69500135d0 con 0x7f696006f7a0 2023-10-30T13:17:48.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.613+0000 7f695cff9640 1 --2- 172.21.15.125:0/2596752096 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f694c0664a0 0x7f694c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:48.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.617+0000 7f695cff9640 1 -- 172.21.15.125:0/2596752096 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f6950084b60 con 0x7f696006f7a0 2023-10-30T13:17:48.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.617+0000 7f695f7fe640 1 --2- 172.21.15.125:0/2596752096 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f694c0664a0 0x7f694c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:48.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.617+0000 7f695f7fe640 1 --2- 172.21.15.125:0/2596752096 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f694c0664a0 0x7f694c068960 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f696014edd0 tx=0x7f6954009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:48.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.617+0000 7f695cff9640 1 -- 172.21.15.125:0/2596752096 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6950051580 con 0x7f696006f7a0 2023-10-30T13:17:48.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.801+0000 7f696646b640 1 -- 172.21.15.125:0/2596752096 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f696006a700 con 0x7f696006f7a0 2023-10-30T13:17:48.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.801+0000 7f695cff9640 1 -- 172.21.15.125:0/2596752096 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6551 (secure 0 0 0) 0x7f6950021090 con 0x7f696006f7a0 2023-10-30T13:17:48.806 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:17:48.806 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8971,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:17:24.823595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:48.806 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 229 2023-10-30T13:17:48.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.801+0000 7f696646b640 1 -- 172.21.15.125:0/2596752096 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f694c0664a0 msgr2=0x7f694c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:48.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.801+0000 7f696646b640 1 --2- 172.21.15.125:0/2596752096 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f694c0664a0 0x7f694c068960 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f696014edd0 tx=0x7f6954009040 comp rx=0 tx=0).stop 2023-10-30T13:17:48.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.801+0000 7f696646b640 1 -- 172.21.15.125:0/2596752096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f696006f7a0 msgr2=0x7f696014fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:48.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.801+0000 7f696646b640 1 --2- 172.21.15.125:0/2596752096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f696006f7a0 0x7f696014fe00 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f695000e980 tx=0x7f695000ee40 comp rx=0 tx=0).stop 2023-10-30T13:17:48.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.801+0000 7f696646b640 1 -- 172.21.15.125:0/2596752096 shutdown_connections 2023-10-30T13:17:48.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.801+0000 7f696646b640 1 --2- 172.21.15.125:0/2596752096 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f694c0664a0 0x7f694c068960 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:48.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.801+0000 7f696646b640 1 --2- 172.21.15.125:0/2596752096 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f69600c6bd0 0x7f696014c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:48.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.801+0000 7f696646b640 1 --2- 172.21.15.125:0/2596752096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6960071110 0x7f69600c3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:48.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.801+0000 7f696646b640 1 --2- 172.21.15.125:0/2596752096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f696006f7a0 0x7f696014fe00 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:48.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.801+0000 7f696646b640 1 -- 172.21.15.125:0/2596752096 >> 172.21.15.125:0/2596752096 conn(0x7f69600c7910 msgr2=0x7f69600c00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:48.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.801+0000 7f696646b640 1 -- 172.21.15.125:0/2596752096 shutdown_connections 2023-10-30T13:17:48.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:48.801+0000 7f696646b640 1 -- 172.21.15.125:0/2596752096 wait complete. 2023-10-30T13:17:48.827 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:17:50.828 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:50.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f2170fd3640 1 Processor -- start 2023-10-30T13:17:50.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f2170fd3640 1 -- start start 2023-10-30T13:17:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f2170fd3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f216c0c6bd0 0x7f216c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f2170fd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f216c071110 0x7f216c158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f2170fd3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f216c066cf0 con 0x7f216c06f7a0 2023-10-30T13:17:50.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f2170fd3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f216c0700c0 con 0x7f216c0c6bd0 2023-10-30T13:17:50.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f2170fd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0c70d0 con 0x7f216c071110 2023-10-30T13:17:50.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f2169d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f216c0c6bd0 0x7f216c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:50.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f2169d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f216c0c6bd0 0x7f216c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52956/0 (socket says 172.21.15.125:52956) 2023-10-30T13:17:50.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f2169d9b640 1 -- 172.21.15.125:0/3207941707 learned_addr learned my addr 172.21.15.125:0/3207941707 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:50.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f216ad9d640 1 --2- 172.21.15.125:0/3207941707 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f216c071110 0x7f216c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:50.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f2169d9b640 1 -- 172.21.15.125:0/3207941707 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f216c071110 msgr2=0x7f216c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:50.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f2169d9b640 1 --2- 172.21.15.125:0/3207941707 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f216c071110 0x7f216c158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:50.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f2169d9b640 1 -- 172.21.15.125:0/3207941707 >> v1:172.21.15.136:6789/0 conn(0x7f216c06f7a0 legacy=0x7f216c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:50.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.969+0000 7f2169d9b640 1 -- 172.21.15.125:0/3207941707 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f216c0c7390 con 0x7f216c0c6bd0 2023-10-30T13:17:50.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.973+0000 7f2169d9b640 1 --2- 172.21.15.125:0/3207941707 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f216c0c6bd0 0x7f216c06fb80 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f2160009870 tx=0x7f216001ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6bb440305f1ea15 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:50.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.973+0000 7f216959a640 1 -- 172.21.15.125:0/3207941707 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21600a8a30 con 0x7f216c0c6bd0 2023-10-30T13:17:50.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.973+0000 7f216959a640 1 -- 172.21.15.125:0/3207941707 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21600a8b90 con 0x7f216c0c6bd0 2023-10-30T13:17:50.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.973+0000 7f216959a640 1 -- 172.21.15.125:0/3207941707 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21600b0440 con 0x7f216c0c6bd0 2023-10-30T13:17:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.973+0000 7f2170fd3640 1 -- 172.21.15.125:0/3207941707 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f216c0c6bd0 msgr2=0x7f216c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.973+0000 7f2170fd3640 1 --2- 172.21.15.125:0/3207941707 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f216c0c6bd0 0x7f216c06fb80 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f2160009870 tx=0x7f216001ed30 comp rx=0 tx=0).stop 2023-10-30T13:17:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.973+0000 7f2170fd3640 1 -- 172.21.15.125:0/3207941707 shutdown_connections 2023-10-30T13:17:50.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.973+0000 7f2170fd3640 1 --2- 172.21.15.125:0/3207941707 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f216c071110 0x7f216c158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:50.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.973+0000 7f2170fd3640 1 --2- 172.21.15.125:0/3207941707 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f216c0c6bd0 0x7f216c06fb80 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:50.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.973+0000 7f2170fd3640 1 -- 172.21.15.125:0/3207941707 >> 172.21.15.125:0/3207941707 conn(0x7f216c0c7910 msgr2=0x7f216c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:50.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.973+0000 7f2170fd3640 1 -- 172.21.15.125:0/3207941707 shutdown_connections 2023-10-30T13:17:50.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.973+0000 7f2170fd3640 1 -- 172.21.15.125:0/3207941707 wait complete. 2023-10-30T13:17:50.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f2170fd3640 1 Processor -- start 2023-10-30T13:17:50.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f2170fd3640 1 -- start start 2023-10-30T13:17:50.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f2170fd3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f216c06f7a0 0x7f216c0af330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:50.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f2170fd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f216c071110 0x7f216c1effc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:50.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f2170fd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f216c0c6bd0 0x7f216c14f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:50.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f2170fd3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0c1ae0 con 0x7f216c06f7a0 2023-10-30T13:17:50.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f2170fd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0c1f00 con 0x7f216c0c6bd0 2023-10-30T13:17:50.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f2170fd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216c0c4620 con 0x7f216c071110 2023-10-30T13:17:50.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f216a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f216c06f7a0 0x7f216c0af330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:50.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f216a59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f216c06f7a0 0x7f216c0af330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52960/0 (socket says 172.21.15.125:52960) 2023-10-30T13:17:50.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f216a59c640 1 -- 172.21.15.125:0/1694281572 learned_addr learned my addr 172.21.15.125:0/1694281572 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:50.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f2169d9b640 1 --2- 172.21.15.125:0/1694281572 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f216c071110 0x7f216c1effc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:50.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f216ad9d640 1 --2- 172.21.15.125:0/1694281572 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f216c0c6bd0 0x7f216c14f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:50.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f216a59c640 1 -- 172.21.15.125:0/1694281572 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f216c071110 msgr2=0x7f216c1effc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:50.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f216a59c640 1 --2- 172.21.15.125:0/1694281572 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f216c071110 0x7f216c1effc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:50.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f216a59c640 1 -- 172.21.15.125:0/1694281572 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f216c0c6bd0 msgr2=0x7f216c14f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:50.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f216a59c640 1 --2- 172.21.15.125:0/1694281572 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f216c0c6bd0 0x7f216c14f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:50.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f216a59c640 1 -- 172.21.15.125:0/1694281572 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21600094d0 con 0x7f216c06f7a0 2023-10-30T13:17:50.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f216a59c640 1 --2- 172.21.15.125:0/1694281572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f216c06f7a0 0x7f216c0af330 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f215c00c3f0 tx=0x7f215c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:50.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f215affd640 1 -- 172.21.15.125:0/1694281572 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c0052d0 con 0x7f216c06f7a0 2023-10-30T13:17:50.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f215affd640 1 -- 172.21.15.125:0/1694281572 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215c005430 con 0x7f216c06f7a0 2023-10-30T13:17:50.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f215affd640 1 -- 172.21.15.125:0/1694281572 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215c003840 con 0x7f216c06f7a0 2023-10-30T13:17:50.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.977+0000 7f2170fd3640 1 -- 172.21.15.125:0/1694281572 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216c0af870 con 0x7f216c06f7a0 2023-10-30T13:17:50.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.981+0000 7f2170fd3640 1 -- 172.21.15.125:0/1694281572 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216c0c0190 con 0x7f216c06f7a0 2023-10-30T13:17:50.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.981+0000 7f2170fd3640 1 -- 172.21.15.125:0/1694281572 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216c070610 con 0x7f216c06f7a0 2023-10-30T13:17:50.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.981+0000 7f215affd640 1 -- 172.21.15.125:0/1694281572 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f215c0039a0 con 0x7f216c06f7a0 2023-10-30T13:17:50.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.981+0000 7f215affd640 1 --2- 172.21.15.125:0/1694281572 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f214c0664a0 0x7f214c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:50.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.981+0000 7f215affd640 1 -- 172.21.15.125:0/1694281572 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7f215c083cc0 con 0x7f216c06f7a0 2023-10-30T13:17:50.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.981+0000 7f2169d9b640 1 --2- 172.21.15.125:0/1694281572 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f214c0664a0 0x7f214c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:50.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.985+0000 7f2169d9b640 1 --2- 172.21.15.125:0/1694281572 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f214c0664a0 0x7f214c068960 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f21600b2000 tx=0x7f21600aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:50.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:50.985+0000 7f215affd640 1 -- 172.21.15.125:0/1694281572 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f216c070610 con 0x7f216c06f7a0 2023-10-30T13:17:51.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:51.169+0000 7f2170fd3640 1 -- 172.21.15.125:0/1694281572 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f216c070610 con 0x7f216c06f7a0 2023-10-30T13:17:51.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:51.169+0000 7f215affd640 1 -- 172.21.15.125:0/1694281572 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6551 (secure 0 0 0) 0x7f216c070610 con 0x7f216c06f7a0 2023-10-30T13:17:51.175 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:17:51.176 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8971,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:17:24.823595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:51.176 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 229 2023-10-30T13:17:51.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:51.173+0000 7f2170fd3640 1 -- 172.21.15.125:0/1694281572 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f214c0664a0 msgr2=0x7f214c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:51.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:51.173+0000 7f2170fd3640 1 --2- 172.21.15.125:0/1694281572 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f214c0664a0 0x7f214c068960 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f21600b2000 tx=0x7f21600aa070 comp rx=0 tx=0).stop 2023-10-30T13:17:51.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:51.173+0000 7f2170fd3640 1 -- 172.21.15.125:0/1694281572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f216c06f7a0 msgr2=0x7f216c0af330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:51.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:51.173+0000 7f2170fd3640 1 --2- 172.21.15.125:0/1694281572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f216c06f7a0 0x7f216c0af330 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f215c00c3f0 tx=0x7f215c00c8b0 comp rx=0 tx=0).stop 2023-10-30T13:17:51.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:51.173+0000 7f2170fd3640 1 -- 172.21.15.125:0/1694281572 shutdown_connections 2023-10-30T13:17:51.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:51.173+0000 7f2170fd3640 1 --2- 172.21.15.125:0/1694281572 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f214c0664a0 0x7f214c068960 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:51.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:51.173+0000 7f2170fd3640 1 --2- 172.21.15.125:0/1694281572 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f216c0c6bd0 0x7f216c14f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:51.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:51.173+0000 7f2170fd3640 1 --2- 172.21.15.125:0/1694281572 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f216c071110 0x7f216c1effc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:51.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:51.173+0000 7f2170fd3640 1 --2- 172.21.15.125:0/1694281572 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f216c06f7a0 0x7f216c0af330 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:51.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:51.173+0000 7f2170fd3640 1 -- 172.21.15.125:0/1694281572 >> 172.21.15.125:0/1694281572 conn(0x7f216c0c7910 msgr2=0x7f216c0c2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:51.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:51.173+0000 7f2170fd3640 1 -- 172.21.15.125:0/1694281572 shutdown_connections 2023-10-30T13:17:51.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:51.173+0000 7f2170fd3640 1 -- 172.21.15.125:0/1694281572 wait complete. 2023-10-30T13:17:51.196 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:17:53.197 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:53.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.337+0000 7febecbd3640 1 Processor -- start 2023-10-30T13:17:53.344 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.337+0000 7febecbd3640 1 -- start start 2023-10-30T13:17:53.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febecbd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe806f7a0 0x7febe806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:53.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febecbd3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7febe8071110 0x7febe8158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:53.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febecbd3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe8066cf0 con 0x7febe80c6bd0 2023-10-30T13:17:53.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febecbd3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe80700c0 con 0x7febe8071110 2023-10-30T13:17:53.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febecbd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe80c70d0 con 0x7febe806f7a0 2023-10-30T13:17:53.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febe659c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe806f7a0 0x7febe806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:53.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febe6d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7febe8071110 0x7febe8158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:53.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febe6d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7febe8071110 0x7febe8158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:60844/0 (socket says 172.21.15.125:60844) 2023-10-30T13:17:53.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febddd9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7febe80c6bd0 0x7febe8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:46346/0 (socket says 172.21.15.125:46346) 2023-10-30T13:17:53.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febe659c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe806f7a0 0x7febe806fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:51606/0 (socket says 172.21.15.125:51606) 2023-10-30T13:17:53.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febe6d9d640 1 -- 172.21.15.125:0/752880932 learned_addr learned my addr 172.21.15.125:0/752880932 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:53.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febe6d9d640 1 -- 172.21.15.125:0/752880932 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe806f7a0 msgr2=0x7febe806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:53.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febe6d9d640 1 --2- 172.21.15.125:0/752880932 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe806f7a0 0x7febe806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:53.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febe6d9d640 1 -- 172.21.15.125:0/752880932 >> v1:172.21.15.136:6789/0 conn(0x7febe80c6bd0 legacy=0x7febe8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:53.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febe6d9d640 1 -- 172.21.15.125:0/752880932 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe80c7390 con 0x7febe8071110 2023-10-30T13:17:53.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.341+0000 7febe659c640 1 --2- 172.21.15.125:0/752880932 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe806f7a0 0x7febe806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:17:53.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.345+0000 7febe6d9d640 1 --2- 172.21.15.125:0/752880932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe8071110 0x7febe8158110 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7febe0009870 tx=0x7febe001ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aed2a9d8d5d1e43e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:53.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.345+0000 7febe5d9b640 1 -- 172.21.15.125:0/752880932 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe00a8a30 con 0x7febe8071110 2023-10-30T13:17:53.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.345+0000 7febe5d9b640 1 -- 172.21.15.125:0/752880932 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe00a8b90 con 0x7febe8071110 2023-10-30T13:17:53.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.345+0000 7febe5d9b640 1 -- 172.21.15.125:0/752880932 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe00b0440 con 0x7febe8071110 2023-10-30T13:17:53.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.345+0000 7febecbd3640 1 -- 172.21.15.125:0/752880932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe8071110 msgr2=0x7febe8158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:53.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.345+0000 7febecbd3640 1 --2- 172.21.15.125:0/752880932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe8071110 0x7febe8158110 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7febe0009870 tx=0x7febe001ed30 comp rx=0 tx=0).stop 2023-10-30T13:17:53.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.345+0000 7febecbd3640 1 -- 172.21.15.125:0/752880932 shutdown_connections 2023-10-30T13:17:53.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.345+0000 7febecbd3640 1 --2- 172.21.15.125:0/752880932 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe8071110 0x7febe8158110 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:53.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.345+0000 7febecbd3640 1 --2- 172.21.15.125:0/752880932 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe806f7a0 0x7febe806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:53.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.345+0000 7febecbd3640 1 -- 172.21.15.125:0/752880932 >> 172.21.15.125:0/752880932 conn(0x7febe80c7910 msgr2=0x7febe80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:53.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.345+0000 7febecbd3640 1 -- 172.21.15.125:0/752880932 shutdown_connections 2023-10-30T13:17:53.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.345+0000 7febecbd3640 1 -- 172.21.15.125:0/752880932 wait complete. 2023-10-30T13:17:53.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febecbd3640 1 Processor -- start 2023-10-30T13:17:53.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febecbd3640 1 -- start start 2023-10-30T13:17:53.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febecbd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe806f7a0 0x7febe81f0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:53.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febecbd3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febe8071110 0x7febe81561c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:53.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febecbd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe80c6bd0 0x7febe8156700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:53.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febecbd3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe8155cc0 con 0x7febe8071110 2023-10-30T13:17:53.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febecbd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe8155550 con 0x7febe80c6bd0 2023-10-30T13:17:53.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febecbd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe8155830 con 0x7febe806f7a0 2023-10-30T13:17:53.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febddd9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febe8071110 0x7febe81561c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:53.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febddd9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febe8071110 0x7febe81561c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52976/0 (socket says 172.21.15.125:52976) 2023-10-30T13:17:53.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febddd9b640 1 -- 172.21.15.125:0/1331187309 learned_addr learned my addr 172.21.15.125:0/1331187309 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:53.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febe659c640 1 --2- 172.21.15.125:0/1331187309 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe806f7a0 0x7febe81f0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:53.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febe6d9d640 1 --2- 172.21.15.125:0/1331187309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe80c6bd0 0x7febe8156700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:53.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febddd9b640 1 -- 172.21.15.125:0/1331187309 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe806f7a0 msgr2=0x7febe81f0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:53.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febddd9b640 1 --2- 172.21.15.125:0/1331187309 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe806f7a0 0x7febe81f0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:53.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febddd9b640 1 -- 172.21.15.125:0/1331187309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe80c6bd0 msgr2=0x7febe8156700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:53.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febddd9b640 1 --2- 172.21.15.125:0/1331187309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe80c6bd0 0x7febe8156700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:53.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febddd9b640 1 -- 172.21.15.125:0/1331187309 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd8009a70 con 0x7febe8071110 2023-10-30T13:17:53.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febddd9b640 1 --2- 172.21.15.125:0/1331187309 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febe8071110 0x7febe81561c0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7febd000c4c0 tx=0x7febd000c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:53.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febdf7fe640 1 -- 172.21.15.125:0/1331187309 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0007480 con 0x7febe8071110 2023-10-30T13:17:53.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febdf7fe640 1 -- 172.21.15.125:0/1331187309 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd00075e0 con 0x7febe8071110 2023-10-30T13:17:53.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febecbd3640 1 -- 172.21.15.125:0/1331187309 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe00094d0 con 0x7febe8071110 2023-10-30T13:17:53.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febdf7fe640 1 -- 172.21.15.125:0/1331187309 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0004a10 con 0x7febe8071110 2023-10-30T13:17:53.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febecbd3640 1 -- 172.21.15.125:0/1331187309 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe80c8f30 con 0x7febe8071110 2023-10-30T13:17:53.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.349+0000 7febecbd3640 1 -- 172.21.15.125:0/1331187309 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe804e170 con 0x7febe8071110 2023-10-30T13:17:53.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.353+0000 7febdf7fe640 1 -- 172.21.15.125:0/1331187309 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7febd0004b70 con 0x7febe8071110 2023-10-30T13:17:53.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.353+0000 7febdf7fe640 1 --2- 172.21.15.125:0/1331187309 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febc80664a0 0x7febc8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:53.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.353+0000 7febdf7fe640 1 -- 172.21.15.125:0/1331187309 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7febd0083f10 con 0x7febe8071110 2023-10-30T13:17:53.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.353+0000 7febe659c640 1 --2- 172.21.15.125:0/1331187309 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febc80664a0 0x7febc8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:53.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.353+0000 7febe659c640 1 --2- 172.21.15.125:0/1331187309 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febc80664a0 0x7febc8068960 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7febd8007560 tx=0x7febd800b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:53.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.357+0000 7febdf7fe640 1 -- 172.21.15.125:0/1331187309 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7febd00509c0 con 0x7febe8071110 2023-10-30T13:17:53.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.533+0000 7febecbd3640 1 -- 172.21.15.125:0/1331187309 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febe80c7f00 con 0x7febe8071110 2023-10-30T13:17:53.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.533+0000 7febdf7fe640 1 -- 172.21.15.125:0/1331187309 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 229 v229) v1 ==== 95+0+6551 (secure 0 0 0) 0x7febd0018030 con 0x7febe8071110 2023-10-30T13:17:53.540 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:17:53.540 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":229,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8971,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":227}],"filesystems":[{"mdsmap":{"epoch":229,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:17:24.823595+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8529,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8529":{"gid":8529,"name":"e","rank":0,"incarnation":179,"state":"up:active","state_seq":15,"addr":"172.21.15.125:6836/3950044434","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":3950044434},{"type":"v1","addr":"172.21.15.125:6836","nonce":3950044434}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:53.540 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 229 2023-10-30T13:17:53.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.537+0000 7febecbd3640 1 -- 172.21.15.125:0/1331187309 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febc80664a0 msgr2=0x7febc8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:53.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.537+0000 7febecbd3640 1 --2- 172.21.15.125:0/1331187309 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febc80664a0 0x7febc8068960 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7febd8007560 tx=0x7febd800b040 comp rx=0 tx=0).stop 2023-10-30T13:17:53.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.537+0000 7febecbd3640 1 -- 172.21.15.125:0/1331187309 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febe8071110 msgr2=0x7febe81561c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:53.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.537+0000 7febecbd3640 1 --2- 172.21.15.125:0/1331187309 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febe8071110 0x7febe81561c0 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7febd000c4c0 tx=0x7febd000c980 comp rx=0 tx=0).stop 2023-10-30T13:17:53.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.537+0000 7febecbd3640 1 -- 172.21.15.125:0/1331187309 shutdown_connections 2023-10-30T13:17:53.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.537+0000 7febecbd3640 1 --2- 172.21.15.125:0/1331187309 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7febc80664a0 0x7febc8068960 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:53.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.537+0000 7febecbd3640 1 --2- 172.21.15.125:0/1331187309 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7febe80c6bd0 0x7febe8156700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:53.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.537+0000 7febecbd3640 1 --2- 172.21.15.125:0/1331187309 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7febe8071110 0x7febe81561c0 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:53.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.537+0000 7febecbd3640 1 --2- 172.21.15.125:0/1331187309 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7febe806f7a0 0x7febe81f0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:53.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.537+0000 7febecbd3640 1 -- 172.21.15.125:0/1331187309 >> 172.21.15.125:0/1331187309 conn(0x7febe80c7910 msgr2=0x7febe80c9580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:53.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.537+0000 7febecbd3640 1 -- 172.21.15.125:0/1331187309 shutdown_connections 2023-10-30T13:17:53.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:53.537+0000 7febecbd3640 1 -- 172.21.15.125:0/1331187309 wait complete. 2023-10-30T13:17:53.561 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is laggy/crashed, in failed state, or mds.e is removed from mdsmap 2023-10-30T13:17:55.563 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:55.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9acc25640 1 Processor -- start 2023-10-30T13:17:55.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9acc25640 1 -- start start 2023-10-30T13:17:55.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9acc25640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff9a8153d70 0x7ff9a806ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:55.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9acc25640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9a806e470 0x7ff9a80cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:55.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9acc25640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a806afa0 con 0x7ff9a80c8490 2023-10-30T13:17:55.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9acc25640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a806d780 con 0x7ff9a8153d70 2023-10-30T13:17:55.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9acc25640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a8154150 con 0x7ff9a806e470 2023-10-30T13:17:55.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9a659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff9a8153d70 0x7ff9a806ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:55.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9a659c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff9a8153d70 0x7ff9a806ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52982/0 (socket says 172.21.15.125:52982) 2023-10-30T13:17:55.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9a659c640 1 -- 172.21.15.125:0/921081247 learned_addr learned my addr 172.21.15.125:0/921081247 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:55.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9a6d9d640 1 --1- 172.21.15.125:0/921081247 >> v1:172.21.15.125:6789/0 conn(0x7ff9a80c8490 0x7ff9a80c8cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59942/0 (socket says 172.21.15.125:59942) 2023-10-30T13:17:55.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9a5d9b640 1 --2- 172.21.15.125:0/921081247 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9a806e470 0x7ff9a80cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:55.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9a659c640 1 -- 172.21.15.125:0/921081247 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9a806e470 msgr2=0x7ff9a80cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:55.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9a659c640 1 --2- 172.21.15.125:0/921081247 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9a806e470 0x7ff9a80cce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:55.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9a659c640 1 -- 172.21.15.125:0/921081247 >> v1:172.21.15.125:6789/0 conn(0x7ff9a80c8490 legacy=0x7ff9a80c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:55.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9a559a640 1 -- 172.21.15.125:0/921081247 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671464843 0 0) 0x7ff9a806afa0 con 0x7ff9a80c8490 2023-10-30T13:17:55.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9a659c640 1 -- 172.21.15.125:0/921081247 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a806e030 con 0x7ff9a8153d70 2023-10-30T13:17:55.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.705+0000 7ff9a5d9b640 1 --2- 172.21.15.125:0/921081247 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9a806e470 0x7ff9a80cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:17:55.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.709+0000 7ff9a659c640 1 --2- 172.21.15.125:0/921081247 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9a8153d70 0x7ff9a806ca80 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7ff998009870 tx=0x7ff99801c510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76546d96e87db445 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:55.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.709+0000 7ff9a559a640 1 -- 172.21.15.125:0/921081247 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980aa670 con 0x7ff9a8153d70 2023-10-30T13:17:55.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.709+0000 7ff9a559a640 1 -- 172.21.15.125:0/921081247 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9980aa7d0 con 0x7ff9a8153d70 2023-10-30T13:17:55.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.709+0000 7ff9a559a640 1 -- 172.21.15.125:0/921081247 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9980aab60 con 0x7ff9a8153d70 2023-10-30T13:17:55.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.709+0000 7ff9acc25640 1 -- 172.21.15.125:0/921081247 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9a8153d70 msgr2=0x7ff9a806ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:55.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.709+0000 7ff9acc25640 1 --2- 172.21.15.125:0/921081247 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9a8153d70 0x7ff9a806ca80 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7ff998009870 tx=0x7ff99801c510 comp rx=0 tx=0).stop 2023-10-30T13:17:55.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.709+0000 7ff9acc25640 1 -- 172.21.15.125:0/921081247 shutdown_connections 2023-10-30T13:17:55.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.709+0000 7ff9acc25640 1 --2- 172.21.15.125:0/921081247 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9a806e470 0x7ff9a80cce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:55.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.709+0000 7ff9acc25640 1 --2- 172.21.15.125:0/921081247 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9a8153d70 0x7ff9a806ca80 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:55.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.709+0000 7ff9acc25640 1 -- 172.21.15.125:0/921081247 >> 172.21.15.125:0/921081247 conn(0x7ff9a8071ea0 msgr2=0x7ff9a8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:55.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.709+0000 7ff9acc25640 1 -- 172.21.15.125:0/921081247 shutdown_connections 2023-10-30T13:17:55.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.709+0000 7ff9acc25640 1 -- 172.21.15.125:0/921081247 wait complete. 2023-10-30T13:17:55.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9acc25640 1 Processor -- start 2023-10-30T13:17:55.720 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9acc25640 1 -- start start 2023-10-30T13:17:55.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9acc25640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9a806e470 0x7ff9a80cbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:55.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9acc25640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9a80c8490 0x7ff9a80cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:55.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9acc25640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9a8153d70 0x7ff9a8140000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:55.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9acc25640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a8141410 con 0x7ff9a8153d70 2023-10-30T13:17:55.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9acc25640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a8142560 con 0x7ff9a806e470 2023-10-30T13:17:55.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9acc25640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9a81422a0 con 0x7ff9a80c8490 2023-10-30T13:17:55.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9a6d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9a8153d70 0x7ff9a8140000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:55.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9a6d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9a8153d70 0x7ff9a8140000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:52998/0 (socket says 172.21.15.125:52998) 2023-10-30T13:17:55.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9a6d9d640 1 -- 172.21.15.125:0/2449037632 learned_addr learned my addr 172.21.15.125:0/2449037632 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:55.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9a5d9b640 1 --2- 172.21.15.125:0/2449037632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9a80c8490 0x7ff9a80cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:55.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9a659c640 1 --2- 172.21.15.125:0/2449037632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9a806e470 0x7ff9a80cbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:55.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9a6d9d640 1 -- 172.21.15.125:0/2449037632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9a80c8490 msgr2=0x7ff9a80cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:55.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9a6d9d640 1 --2- 172.21.15.125:0/2449037632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9a80c8490 0x7ff9a80cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:55.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9a6d9d640 1 -- 172.21.15.125:0/2449037632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9a806e470 msgr2=0x7ff9a80cbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:55.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9a6d9d640 1 --2- 172.21.15.125:0/2449037632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9a806e470 0x7ff9a80cbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:55.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9a6d9d640 1 -- 172.21.15.125:0/2449037632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9980094d0 con 0x7ff9a8153d70 2023-10-30T13:17:55.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9a6d9d640 1 --2- 172.21.15.125:0/2449037632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9a8153d70 0x7ff9a8140000 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000c810 tx=0x7ff9a000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:55.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff996ffd640 1 -- 172.21.15.125:0/2449037632 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a000a640 con 0x7ff9a8153d70 2023-10-30T13:17:55.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff996ffd640 1 -- 172.21.15.125:0/2449037632 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a000a7c0 con 0x7ff9a8153d70 2023-10-30T13:17:55.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff996ffd640 1 -- 172.21.15.125:0/2449037632 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a000ab70 con 0x7ff9a8153d70 2023-10-30T13:17:55.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9acc25640 1 -- 172.21.15.125:0/2449037632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a813fb70 con 0x7ff9a8153d70 2023-10-30T13:17:55.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.713+0000 7ff9acc25640 1 -- 172.21.15.125:0/2449037632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a8144c00 con 0x7ff9a8153d70 2023-10-30T13:17:55.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.717+0000 7ff9acc25640 1 -- 172.21.15.125:0/2449037632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a813fd00 con 0x7ff9a8153d70 2023-10-30T13:17:55.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.717+0000 7ff996ffd640 1 -- 172.21.15.125:0/2449037632 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff9a000acd0 con 0x7ff9a8153d70 2023-10-30T13:17:55.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.717+0000 7ff996ffd640 1 --2- 172.21.15.125:0/2449037632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9880664d0 0x7ff988068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:55.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.717+0000 7ff996ffd640 1 -- 172.21.15.125:0/2449037632 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8060+0+0 (secure 0 0 0) 0x7ff9a0083c00 con 0x7ff9a8153d70 2023-10-30T13:17:55.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.717+0000 7ff9a659c640 1 --2- 172.21.15.125:0/2449037632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9880664d0 0x7ff988068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:55.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.717+0000 7ff9a659c640 1 --2- 172.21.15.125:0/2449037632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9880664d0 0x7ff988068990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff9980ae000 tx=0x7ff9980b5040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:55.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.721+0000 7ff996ffd640 1 -- 172.21.15.125:0/2449037632 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff9a0050760 con 0x7ff9a8153d70 2023-10-30T13:17:55.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.905+0000 7ff9acc25640 1 -- 172.21.15.125:0/2449037632 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9a8144e10 con 0x7ff9a8153d70 2023-10-30T13:17:55.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.909+0000 7ff996ffd640 1 -- 172.21.15.125:0/2449037632 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5754 (secure 0 0 0) 0x7ff9a0018070 con 0x7ff9a8153d70 2023-10-30T13:17:55.913 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:17:55.913 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:17:55.811977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:55.914 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 230 2023-10-30T13:17:55.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.909+0000 7ff9acc25640 1 -- 172.21.15.125:0/2449037632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9880664d0 msgr2=0x7ff988068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:55.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.909+0000 7ff9acc25640 1 --2- 172.21.15.125:0/2449037632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9880664d0 0x7ff988068990 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7ff9980ae000 tx=0x7ff9980b5040 comp rx=0 tx=0).stop 2023-10-30T13:17:55.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.909+0000 7ff9acc25640 1 -- 172.21.15.125:0/2449037632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9a8153d70 msgr2=0x7ff9a8140000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:55.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.909+0000 7ff9acc25640 1 --2- 172.21.15.125:0/2449037632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9a8153d70 0x7ff9a8140000 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7ff9a000c810 tx=0x7ff9a000ccd0 comp rx=0 tx=0).stop 2023-10-30T13:17:55.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.909+0000 7ff9acc25640 1 -- 172.21.15.125:0/2449037632 shutdown_connections 2023-10-30T13:17:55.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.909+0000 7ff9acc25640 1 --2- 172.21.15.125:0/2449037632 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff9880664d0 0x7ff988068990 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:55.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.909+0000 7ff9acc25640 1 --2- 172.21.15.125:0/2449037632 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff9a8153d70 0x7ff9a8140000 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:55.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.909+0000 7ff9acc25640 1 --2- 172.21.15.125:0/2449037632 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff9a80c8490 0x7ff9a80cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:55.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.909+0000 7ff9acc25640 1 --2- 172.21.15.125:0/2449037632 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff9a806e470 0x7ff9a80cbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:55.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.909+0000 7ff9acc25640 1 -- 172.21.15.125:0/2449037632 >> 172.21.15.125:0/2449037632 conn(0x7ff9a8071ea0 msgr2=0x7ff9a8151f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:55.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.909+0000 7ff9acc25640 1 -- 172.21.15.125:0/2449037632 shutdown_connections 2023-10-30T13:17:55.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:55.909+0000 7ff9acc25640 1 -- 172.21.15.125:0/2449037632 wait complete. 2023-10-30T13:17:55.934 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e down, removed from mdsmap 2023-10-30T13:17:55.935 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:17:55.935 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:56.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6d58bb640 1 Processor -- start 2023-10-30T13:17:56.073 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6d58bb640 1 -- start start 2023-10-30T13:17:56.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6d58bb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6d015cd50 0x7fa6d00c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:56.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6d58bb640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa6d00cefa0 0x7fa6d00cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:56.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6d58bb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d015a6c0 con 0x7fa6d01538f0 2023-10-30T13:17:56.075 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6d58bb640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d015d300 con 0x7fa6d00cefa0 2023-10-30T13:17:56.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6d58bb640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d015d5e0 con 0x7fa6d015cd50 2023-10-30T13:17:56.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6d50ba640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa6d00cefa0 0x7fa6d00cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:56.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6d50ba640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa6d00cefa0 0x7fa6d00cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53014/0 (socket says 172.21.15.125:53014) 2023-10-30T13:17:56.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6d50ba640 1 -- 172.21.15.125:0/3141468867 learned_addr learned my addr 172.21.15.125:0/3141468867 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6cffff640 1 --2- 172.21.15.125:0/3141468867 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6d015cd50 0x7fa6d00c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6d50ba640 1 -- 172.21.15.125:0/3141468867 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6d015cd50 msgr2=0x7fa6d00c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6d50ba640 1 --2- 172.21.15.125:0/3141468867 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6d015cd50 0x7fa6d00c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6d50ba640 1 -- 172.21.15.125:0/3141468867 >> v1:172.21.15.136:6789/0 conn(0x7fa6d01538f0 legacy=0x7fa6d0159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.069+0000 7fa6d50ba640 1 -- 172.21.15.125:0/3141468867 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d015d8c0 con 0x7fa6d00cefa0 2023-10-30T13:17:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.073+0000 7fa6d50ba640 1 --2- 172.21.15.125:0/3141468867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6d00cefa0 0x7fa6d00cf3f0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8009e50 tx=0x7fa6c801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a7c3e625bd82572 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.073+0000 7fa6cf7fe640 1 -- 172.21.15.125:0/3141468867 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c80af400 con 0x7fa6d00cefa0 2023-10-30T13:17:56.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.073+0000 7fa6cf7fe640 1 -- 172.21.15.125:0/3141468867 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c80ab050 con 0x7fa6d00cefa0 2023-10-30T13:17:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.073+0000 7fa6cf7fe640 1 -- 172.21.15.125:0/3141468867 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c80af780 con 0x7fa6d00cefa0 2023-10-30T13:17:56.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.073+0000 7fa6d58bb640 1 -- 172.21.15.125:0/3141468867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6d00cefa0 msgr2=0x7fa6d00cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.073+0000 7fa6d58bb640 1 --2- 172.21.15.125:0/3141468867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6d00cefa0 0x7fa6d00cf3f0 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fa6c8009e50 tx=0x7fa6c801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:17:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.073+0000 7fa6d58bb640 1 -- 172.21.15.125:0/3141468867 shutdown_connections 2023-10-30T13:17:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.073+0000 7fa6d58bb640 1 --2- 172.21.15.125:0/3141468867 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6d00cefa0 0x7fa6d00cf3f0 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:56.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.073+0000 7fa6d58bb640 1 --2- 172.21.15.125:0/3141468867 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6d015cd50 0x7fa6d00c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:56.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.073+0000 7fa6d58bb640 1 -- 172.21.15.125:0/3141468867 >> 172.21.15.125:0/3141468867 conn(0x7fa6d006a640 msgr2=0x7fa6d0065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:56.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.073+0000 7fa6d58bb640 1 -- 172.21.15.125:0/3141468867 shutdown_connections 2023-10-30T13:17:56.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.073+0000 7fa6d58bb640 1 -- 172.21.15.125:0/3141468867 wait complete. 2023-10-30T13:17:56.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6d58bb640 1 Processor -- start 2023-10-30T13:17:56.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6d58bb640 1 -- start start 2023-10-30T13:17:56.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6d58bb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d00cefa0 0x7fa6d0145910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:56.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6d58bb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6d01538f0 0x7fa6d0145e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:56.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6d58bb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6d015cd50 0x7fa6d01516f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:56.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6d58bb640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d0151260 con 0x7fa6d01538f0 2023-10-30T13:17:56.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6d58bb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d014cd20 con 0x7fa6d00cefa0 2023-10-30T13:17:56.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6d58bb640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d01efd70 con 0x7fa6d015cd50 2023-10-30T13:17:56.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6cffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6d01538f0 0x7fa6d0145e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:56.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6cffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6d01538f0 0x7fa6d0145e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53028/0 (socket says 172.21.15.125:53028) 2023-10-30T13:17:56.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6cffff640 1 -- 172.21.15.125:0/699039929 learned_addr learned my addr 172.21.15.125:0/699039929 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:56.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6d50ba640 1 --2- 172.21.15.125:0/699039929 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6d015cd50 0x7fa6d01516f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:56.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6d48b9640 1 --2- 172.21.15.125:0/699039929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d00cefa0 0x7fa6d0145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:56.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6cffff640 1 -- 172.21.15.125:0/699039929 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6d015cd50 msgr2=0x7fa6d01516f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:56.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6cffff640 1 --2- 172.21.15.125:0/699039929 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6d015cd50 0x7fa6d01516f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:56.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6cffff640 1 -- 172.21.15.125:0/699039929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d00cefa0 msgr2=0x7fa6d0145910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:56.086 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6cffff640 1 --2- 172.21.15.125:0/699039929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d00cefa0 0x7fa6d0145910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:56.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6cffff640 1 -- 172.21.15.125:0/699039929 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c8009b30 con 0x7fa6d01538f0 2023-10-30T13:17:56.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6cffff640 1 --2- 172.21.15.125:0/699039929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6d01538f0 0x7fa6d0145e50 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00e870 tx=0x7fa6bc00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:56.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6cd7fa640 1 -- 172.21.15.125:0/699039929 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc009510 con 0x7fa6d01538f0 2023-10-30T13:17:56.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6cd7fa640 1 -- 172.21.15.125:0/699039929 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc00eec0 con 0x7fa6d01538f0 2023-10-30T13:17:56.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6cd7fa640 1 -- 172.21.15.125:0/699039929 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc00cb40 con 0x7fa6d01538f0 2023-10-30T13:17:56.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6d58bb640 1 -- 172.21.15.125:0/699039929 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d0159600 con 0x7fa6d01538f0 2023-10-30T13:17:56.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6d58bb640 1 -- 172.21.15.125:0/699039929 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d015abd0 con 0x7fa6d01538f0 2023-10-30T13:17:56.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.077+0000 7fa6d58bb640 1 -- 172.21.15.125:0/699039929 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d004e170 con 0x7fa6d01538f0 2023-10-30T13:17:56.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.081+0000 7fa6cd7fa640 1 -- 172.21.15.125:0/699039929 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa6bc00cca0 con 0x7fa6d01538f0 2023-10-30T13:17:56.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.081+0000 7fa6cd7fa640 1 --2- 172.21.15.125:0/699039929 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6b40664d0 0x7fa6b4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:56.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.081+0000 7fa6cd7fa640 1 -- 172.21.15.125:0/699039929 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7fa6bc084a60 con 0x7fa6d01538f0 2023-10-30T13:17:56.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.081+0000 7fa6d48b9640 1 --2- 172.21.15.125:0/699039929 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6b40664d0 0x7fa6b4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:56.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.081+0000 7fa6d48b9640 1 --2- 172.21.15.125:0/699039929 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6b40664d0 0x7fa6b4068990 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4000f70 tx=0x7fa6c4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:56.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.085+0000 7fa6cd7fa640 1 -- 172.21.15.125:0/699039929 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa6bc051420 con 0x7fa6d01538f0 2023-10-30T13:17:56.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.269+0000 7fa6d58bb640 1 -- 172.21.15.125:0/699039929 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6d015db70 con 0x7fa6d01538f0 2023-10-30T13:17:56.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.269+0000 7fa6cd7fa640 1 -- 172.21.15.125:0/699039929 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5754 (secure 0 0 0) 0x7fa6bc021020 con 0x7fa6d01538f0 2023-10-30T13:17:56.276 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:17:56.276 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:17:55.811977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:56.277 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 230 2023-10-30T13:17:56.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.273+0000 7fa6d58bb640 1 -- 172.21.15.125:0/699039929 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6b40664d0 msgr2=0x7fa6b4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:56.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.273+0000 7fa6d58bb640 1 --2- 172.21.15.125:0/699039929 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6b40664d0 0x7fa6b4068990 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4000f70 tx=0x7fa6c4009040 comp rx=0 tx=0).stop 2023-10-30T13:17:56.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.273+0000 7fa6d58bb640 1 -- 172.21.15.125:0/699039929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6d01538f0 msgr2=0x7fa6d0145e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:56.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.273+0000 7fa6d58bb640 1 --2- 172.21.15.125:0/699039929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6d01538f0 0x7fa6d0145e50 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc00e870 tx=0x7fa6bc00ec70 comp rx=0 tx=0).stop 2023-10-30T13:17:56.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.273+0000 7fa6d58bb640 1 -- 172.21.15.125:0/699039929 shutdown_connections 2023-10-30T13:17:56.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.273+0000 7fa6d58bb640 1 --2- 172.21.15.125:0/699039929 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa6b40664d0 0x7fa6b4068990 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:56.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.273+0000 7fa6d58bb640 1 --2- 172.21.15.125:0/699039929 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa6d015cd50 0x7fa6d01516f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:56.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.273+0000 7fa6d58bb640 1 --2- 172.21.15.125:0/699039929 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa6d01538f0 0x7fa6d0145e50 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:56.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.273+0000 7fa6d58bb640 1 --2- 172.21.15.125:0/699039929 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa6d00cefa0 0x7fa6d0145910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:56.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.273+0000 7fa6d58bb640 1 -- 172.21.15.125:0/699039929 >> 172.21.15.125:0/699039929 conn(0x7fa6d006a640 msgr2=0x7fa6d00c5240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:56.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.273+0000 7fa6d58bb640 1 -- 172.21.15.125:0/699039929 shutdown_connections 2023-10-30T13:17:56.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:56.273+0000 7fa6d58bb640 1 -- 172.21.15.125:0/699039929 wait complete. 2023-10-30T13:17:56.298 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 230, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 230, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_8833": { "addr": "172.21.15.136:6835/3864637420", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3864637420, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3864637420, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8833, "incarnation": 222, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 15 }, "gid_8971": { "addr": "172.21.15.136:6839/4284345684", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 4284345684, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 4284345684, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8971, "incarnation": 230, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 82, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:17:55.811977+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8971, "mds_1": 9150, "mds_2": 8833 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:17:58.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:17:58.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.433+0000 7f0467206640 1 Processor -- start 2023-10-30T13:17:58.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.437+0000 7f0467206640 1 -- start start 2023-10-30T13:17:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.437+0000 7f0467206640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f046015cd50 0x7f04600c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.437+0000 7f0467206640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f04600cefa0 0x7f04600cf3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.437+0000 7f0467206640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046015a6c0 con 0x7f04601538f0 2023-10-30T13:17:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.437+0000 7f0467206640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04600cf8f0 con 0x7f046015cd50 2023-10-30T13:17:58.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.437+0000 7f0467206640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f046015d300 con 0x7f04600cefa0 2023-10-30T13:17:58.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.437+0000 7f0465a03640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f046015cd50 0x7f04600c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:58.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.437+0000 7f0465a03640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f046015cd50 0x7f04600c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53032/0 (socket says 172.21.15.125:53032) 2023-10-30T13:17:58.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.437+0000 7f0465a03640 1 -- 172.21.15.125:0/2708255876 learned_addr learned my addr 172.21.15.125:0/2708255876 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:58.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.437+0000 7f0466a05640 1 --2- 172.21.15.125:0/2708255876 >> v2:172.21.15.136:3300/0 conn(0x7f04600cefa0 0x7f04600cf3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:58.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.437+0000 7f0465a03640 1 -- 172.21.15.125:0/2708255876 >> v2:172.21.15.136:3300/0 conn(0x7f04600cefa0 msgr2=0x7f04600cf3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:17:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.437+0000 7f0465a03640 1 --2- 172.21.15.125:0/2708255876 >> v2:172.21.15.136:3300/0 conn(0x7f04600cefa0 0x7f04600cf3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.437+0000 7f0465a03640 1 -- 172.21.15.125:0/2708255876 >> v1:172.21.15.136:6789/0 conn(0x7f04601538f0 legacy=0x7f0460159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:58.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.437+0000 7f0465a03640 1 -- 172.21.15.125:0/2708255876 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046015d5e0 con 0x7f046015cd50 2023-10-30T13:17:58.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.441+0000 7f0465a03640 1 --2- 172.21.15.125:0/2708255876 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f046015cd50 0x7f04600c9f90 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f0460066a40 tx=0x7f045001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c4663d2b191963 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:58.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.441+0000 7f0465202640 1 -- 172.21.15.125:0/2708255876 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04500a89f0 con 0x7f046015cd50 2023-10-30T13:17:58.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.441+0000 7f0465202640 1 -- 172.21.15.125:0/2708255876 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04500a8b50 con 0x7f046015cd50 2023-10-30T13:17:58.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.441+0000 7f0465202640 1 -- 172.21.15.125:0/2708255876 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04500b0440 con 0x7f046015cd50 2023-10-30T13:17:58.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.441+0000 7f0467206640 1 -- 172.21.15.125:0/2708255876 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f046015cd50 msgr2=0x7f04600c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:58.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.441+0000 7f0467206640 1 --2- 172.21.15.125:0/2708255876 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f046015cd50 0x7f04600c9f90 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f0460066a40 tx=0x7f045001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:17:58.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.441+0000 7f0467206640 1 -- 172.21.15.125:0/2708255876 shutdown_connections 2023-10-30T13:17:58.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.441+0000 7f0467206640 1 --2- 172.21.15.125:0/2708255876 >> v2:172.21.15.136:3300/0 conn(0x7f04600cefa0 0x7f04600cf3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:58.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.441+0000 7f0467206640 1 --2- 172.21.15.125:0/2708255876 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f046015cd50 0x7f04600c9f90 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:58.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.441+0000 7f0467206640 1 -- 172.21.15.125:0/2708255876 >> 172.21.15.125:0/2708255876 conn(0x7f046006a640 msgr2=0x7f0460065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:58.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.441+0000 7f0467206640 1 -- 172.21.15.125:0/2708255876 shutdown_connections 2023-10-30T13:17:58.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.441+0000 7f0467206640 1 -- 172.21.15.125:0/2708255876 wait complete. 2023-10-30T13:17:58.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.441+0000 7f0467206640 1 Processor -- start 2023-10-30T13:17:58.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0467206640 1 -- start start 2023-10-30T13:17:58.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0467206640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04600cefa0 0x7f04601453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:58.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0467206640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04601538f0 0x7f046015a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:58.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0467206640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f046015cd50 0x7f0460158ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:58.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0467206640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04601f02f0 con 0x7f04601538f0 2023-10-30T13:17:58.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0467206640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0460157a20 con 0x7f046015cd50 2023-10-30T13:17:58.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0467206640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04601ef2e0 con 0x7f04600cefa0 2023-10-30T13:17:58.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0465a03640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04601538f0 0x7f046015a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:58.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0465a03640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04601538f0 0x7f046015a920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53048/0 (socket says 172.21.15.125:53048) 2023-10-30T13:17:58.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0465a03640 1 -- 172.21.15.125:0/3872026847 learned_addr learned my addr 172.21.15.125:0/3872026847 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:17:58.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0466204640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04600cefa0 0x7f04601453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:58.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0466a05640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f046015cd50 0x7f0460158ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:58.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0465a03640 1 -- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04600cefa0 msgr2=0x7f04601453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:58.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0465a03640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04600cefa0 0x7f04601453c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:58.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0465a03640 1 -- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f046015cd50 msgr2=0x7f0460158ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:58.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0465a03640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f046015cd50 0x7f0460158ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:58.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0465a03640 1 -- 172.21.15.125:0/3872026847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04500094d0 con 0x7f04601538f0 2023-10-30T13:17:58.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0466204640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04600cefa0 0x7f04601453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:17:58.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0465a03640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04601538f0 0x7f046015a920 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f045001b900 tx=0x7f045001ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:58.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0456ffd640 1 -- 172.21.15.125:0/3872026847 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04500ba600 con 0x7f04601538f0 2023-10-30T13:17:58.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0456ffd640 1 -- 172.21.15.125:0/3872026847 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04500ba760 con 0x7f04601538f0 2023-10-30T13:17:58.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0467206640 1 -- 172.21.15.125:0/3872026847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04601efe60 con 0x7f04601538f0 2023-10-30T13:17:58.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0456ffd640 1 -- 172.21.15.125:0/3872026847 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04500bab10 con 0x7f04601538f0 2023-10-30T13:17:58.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0467206640 1 -- 172.21.15.125:0/3872026847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04601ef530 con 0x7f04601538f0 2023-10-30T13:17:58.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.445+0000 7f0467206640 1 -- 172.21.15.125:0/3872026847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04601efff0 con 0x7f04601538f0 2023-10-30T13:17:58.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.449+0000 7f0456ffd640 1 -- 172.21.15.125:0/3872026847 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f04500bac70 con 0x7f04601538f0 2023-10-30T13:17:58.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.449+0000 7f0456ffd640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04440664d0 0x7f0444068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:17:58.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.449+0000 7f0456ffd640 1 -- 172.21.15.125:0/3872026847 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f0450122d90 con 0x7f04601538f0 2023-10-30T13:17:58.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.449+0000 7f0466204640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04440664d0 0x7f0444068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:17:58.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.449+0000 7f0466204640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04440664d0 0x7f0444068990 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f045c000f70 tx=0x7f045c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:17:58.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.453+0000 7f0456ffd640 1 -- 172.21.15.125:0/3872026847 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f04500ef900 con 0x7f04601538f0 2023-10-30T13:17:58.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.633+0000 7f0467206640 1 -- 172.21.15.125:0/3872026847 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f046004e170 con 0x7f04601538f0 2023-10-30T13:17:58.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.633+0000 7f0456ffd640 1 -- 172.21.15.125:0/3872026847 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5754 (secure 0 0 0) 0x7f04500b5020 con 0x7f04601538f0 2023-10-30T13:17:58.640 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:17:58.640 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:17:55.811977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:17:58.641 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 230 2023-10-30T13:17:58.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.637+0000 7f0467206640 1 -- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04440664d0 msgr2=0x7f0444068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:58.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.637+0000 7f0467206640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04440664d0 0x7f0444068990 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f045c000f70 tx=0x7f045c009040 comp rx=0 tx=0).stop 2023-10-30T13:17:58.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.637+0000 7f0467206640 1 -- 172.21.15.125:0/3872026847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04601538f0 msgr2=0x7f046015a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:17:58.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.637+0000 7f0467206640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04601538f0 0x7f046015a920 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7f045001b900 tx=0x7f045001ba90 comp rx=0 tx=0).stop 2023-10-30T13:17:58.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.637+0000 7f0467206640 1 -- 172.21.15.125:0/3872026847 shutdown_connections 2023-10-30T13:17:58.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.637+0000 7f0467206640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04440664d0 0x7f0444068990 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:58.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.637+0000 7f0467206640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f046015cd50 0x7f0460158ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:58.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.637+0000 7f0467206640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04601538f0 0x7f046015a920 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:58.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.637+0000 7f0467206640 1 --2- 172.21.15.125:0/3872026847 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04600cefa0 0x7f04601453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:17:58.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.637+0000 7f0467206640 1 -- 172.21.15.125:0/3872026847 >> 172.21.15.125:0/3872026847 conn(0x7f046006a640 msgr2=0x7f04600cd710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:17:58.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.637+0000 7f0467206640 1 -- 172.21.15.125:0/3872026847 shutdown_connections 2023-10-30T13:17:58.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:17:58.637+0000 7f0467206640 1 -- 172.21.15.125:0/3872026847 wait complete. 2023-10-30T13:17:58.661 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:00.661 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:00.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f425869c640 1 Processor -- start 2023-10-30T13:18:00.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f425869c640 1 -- start start 2023-10-30T13:18:00.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f425869c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42501538a0 0x7f425015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:00.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f425869c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42500cf160 0x7f42500cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:00.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f425869c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42500cb800 con 0x7f42500ca0e0 2023-10-30T13:18:00.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f425869c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f425015dc30 con 0x7f42500cf160 2023-10-30T13:18:00.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f425869c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42500cb330 con 0x7f42501538a0 2023-10-30T13:18:00.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f4256c39640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42500cf160 0x7f42500cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:00.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f4256c39640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f42500cf160 0x7f42500cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34186/0 (socket says 172.21.15.125:34186) 2023-10-30T13:18:00.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f4256c39640 1 -- 172.21.15.125:0/737188677 learned_addr learned my addr 172.21.15.125:0/737188677 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:00.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f4256438640 1 --2- 172.21.15.125:0/737188677 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42501538a0 0x7f425015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:00.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f4256c39640 1 -- 172.21.15.125:0/737188677 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42501538a0 msgr2=0x7f425015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:00.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f4256c39640 1 --2- 172.21.15.125:0/737188677 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42501538a0 0x7f425015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:00.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f4256c39640 1 -- 172.21.15.125:0/737188677 >> v1:172.21.15.136:6789/0 conn(0x7f42500ca0e0 legacy=0x7f425015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:00.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.805+0000 7f4256c39640 1 -- 172.21.15.125:0/737188677 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42500cad70 con 0x7f42500cf160 2023-10-30T13:18:00.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.809+0000 7f4256c39640 1 --2- 172.21.15.125:0/737188677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42500cf160 0x7f42500cf590 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f4248009e50 tx=0x7f424801f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dca190527397cb6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:00.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.809+0000 7f4255436640 1 -- 172.21.15.125:0/737188677 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42480af400 con 0x7f42500cf160 2023-10-30T13:18:00.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.809+0000 7f4255436640 1 -- 172.21.15.125:0/737188677 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42480ab050 con 0x7f42500cf160 2023-10-30T13:18:00.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.809+0000 7f4255436640 1 -- 172.21.15.125:0/737188677 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42480af780 con 0x7f42500cf160 2023-10-30T13:18:00.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.809+0000 7f425869c640 1 -- 172.21.15.125:0/737188677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42500cf160 msgr2=0x7f42500cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:00.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.809+0000 7f425869c640 1 --2- 172.21.15.125:0/737188677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42500cf160 0x7f42500cf590 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7f4248009e50 tx=0x7f424801f1c0 comp rx=0 tx=0).stop 2023-10-30T13:18:00.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.809+0000 7f425869c640 1 -- 172.21.15.125:0/737188677 shutdown_connections 2023-10-30T13:18:00.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.809+0000 7f425869c640 1 --2- 172.21.15.125:0/737188677 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42500cf160 0x7f42500cf590 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:00.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.809+0000 7f425869c640 1 --2- 172.21.15.125:0/737188677 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42501538a0 0x7f425015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:00.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.809+0000 7f425869c640 1 -- 172.21.15.125:0/737188677 >> 172.21.15.125:0/737188677 conn(0x7f425006bfe0 msgr2=0x7f425006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:00.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.809+0000 7f425869c640 1 -- 172.21.15.125:0/737188677 shutdown_connections 2023-10-30T13:18:00.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.809+0000 7f425869c640 1 -- 172.21.15.125:0/737188677 wait complete. 2023-10-30T13:18:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f425869c640 1 Processor -- start 2023-10-30T13:18:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f425869c640 1 -- start start 2023-10-30T13:18:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f425869c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42500ca0e0 0x7f425015c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:00.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f425869c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42500cf160 0x7f42501e3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:00.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f425869c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42501538a0 0x7f42501e3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:00.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f425869c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42501e5970 con 0x7f42500cf160 2023-10-30T13:18:00.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f425869c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42501e3bd0 con 0x7f42500ca0e0 2023-10-30T13:18:00.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f425869c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f425015ad80 con 0x7f42501538a0 2023-10-30T13:18:00.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4255c37640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42500cf160 0x7f42501e3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:00.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4255c37640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42500cf160 0x7f42501e3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34188/0 (socket says 172.21.15.125:34188) 2023-10-30T13:18:00.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4255c37640 1 -- 172.21.15.125:0/578711088 learned_addr learned my addr 172.21.15.125:0/578711088 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:00.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4256438640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42500ca0e0 0x7f425015c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:00.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4256c39640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42501538a0 0x7f42501e3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:00.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4255c37640 1 -- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42501538a0 msgr2=0x7f42501e3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:00.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4255c37640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42501538a0 0x7f42501e3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:00.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4255c37640 1 -- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42500ca0e0 msgr2=0x7f425015c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:00.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4255c37640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42500ca0e0 0x7f425015c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:00.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4255c37640 1 -- 172.21.15.125:0/578711088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4248009b30 con 0x7f42500cf160 2023-10-30T13:18:00.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4256438640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42500ca0e0 0x7f425015c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:00.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4255c37640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42500cf160 0x7f42501e3150 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f424000c450 tx=0x7f424000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:00.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4246ffd640 1 -- 172.21.15.125:0/578711088 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4240012830 con 0x7f42500cf160 2023-10-30T13:18:00.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4246ffd640 1 -- 172.21.15.125:0/578711088 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42400129b0 con 0x7f42500cf160 2023-10-30T13:18:00.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f425869c640 1 -- 172.21.15.125:0/578711088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f425015d490 con 0x7f42500cf160 2023-10-30T13:18:00.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.813+0000 7f4246ffd640 1 -- 172.21.15.125:0/578711088 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4240012d60 con 0x7f42500cf160 2023-10-30T13:18:00.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.817+0000 7f425869c640 1 -- 172.21.15.125:0/578711088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42501e5690 con 0x7f42500cf160 2023-10-30T13:18:00.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.817+0000 7f425869c640 1 -- 172.21.15.125:0/578711088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42500b2a50 con 0x7f42500cf160 2023-10-30T13:18:00.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.817+0000 7f4246ffd640 1 -- 172.21.15.125:0/578711088 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f424001c460 con 0x7f42500cf160 2023-10-30T13:18:00.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.817+0000 7f4246ffd640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42340664a0 0x7f4234068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:00.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.817+0000 7f4246ffd640 1 -- 172.21.15.125:0/578711088 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f4240084680 con 0x7f42500cf160 2023-10-30T13:18:00.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.817+0000 7f4256438640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42340664a0 0x7f4234068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:00.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.821+0000 7f4256438640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42340664a0 0x7f4234068960 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f424c00b9f0 tx=0x7f424c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:00.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:00.821+0000 7f4246ffd640 1 -- 172.21.15.125:0/578711088 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4240051150 con 0x7f42500cf160 2023-10-30T13:18:01.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:01.005+0000 7f425869c640 1 -- 172.21.15.125:0/578711088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f425004e170 con 0x7f42500cf160 2023-10-30T13:18:01.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:01.005+0000 7f4246ffd640 1 -- 172.21.15.125:0/578711088 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5754 (secure 0 0 0) 0x7f4240054770 con 0x7f42500cf160 2023-10-30T13:18:01.010 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:18:01.010 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:17:55.811977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:01.011 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 230 2023-10-30T13:18:01.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:01.005+0000 7f425869c640 1 -- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42340664a0 msgr2=0x7f4234068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:01.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:01.005+0000 7f425869c640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42340664a0 0x7f4234068960 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f424c00b9f0 tx=0x7f424c008040 comp rx=0 tx=0).stop 2023-10-30T13:18:01.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:01.005+0000 7f425869c640 1 -- 172.21.15.125:0/578711088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42500cf160 msgr2=0x7f42501e3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:01.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:01.005+0000 7f425869c640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42500cf160 0x7f42501e3150 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7f424000c450 tx=0x7f424000c910 comp rx=0 tx=0).stop 2023-10-30T13:18:01.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:01.005+0000 7f425869c640 1 -- 172.21.15.125:0/578711088 shutdown_connections 2023-10-30T13:18:01.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:01.005+0000 7f425869c640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42340664a0 0x7f4234068960 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:01.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:01.005+0000 7f425869c640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42501538a0 0x7f42501e3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:01.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:01.005+0000 7f425869c640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42500cf160 0x7f42501e3150 unknown :-1 s=CLOSED pgs=1948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:01.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:01.005+0000 7f425869c640 1 --2- 172.21.15.125:0/578711088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42500ca0e0 0x7f425015c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:01.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:01.005+0000 7f425869c640 1 -- 172.21.15.125:0/578711088 >> 172.21.15.125:0/578711088 conn(0x7f425006bfe0 msgr2=0x7f4250159f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:01.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:01.005+0000 7f425869c640 1 -- 172.21.15.125:0/578711088 shutdown_connections 2023-10-30T13:18:01.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:01.005+0000 7f425869c640 1 -- 172.21.15.125:0/578711088 wait complete. 2023-10-30T13:18:01.031 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:03.032 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:03.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed3ac9640 1 Processor -- start 2023-10-30T13:18:03.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed3ac9640 1 -- start start 2023-10-30T13:18:03.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed3ac9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0ecc15d6f0 0x7f0ecc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:03.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed3ac9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0ecc0cf160 0x7f0ecc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:03.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed3ac9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ecc0cb800 con 0x7f0ecc1538a0 2023-10-30T13:18:03.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed3ac9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ecc0cb330 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed3ac9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ecc0cad70 con 0x7f0ecc0cf160 2023-10-30T13:18:03.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed1064640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0ecc15d6f0 0x7f0ecc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:03.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed1064640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0ecc15d6f0 0x7f0ecc15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34194/0 (socket says 172.21.15.125:34194) 2023-10-30T13:18:03.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed1064640 1 -- 172.21.15.125:0/3913123255 learned_addr learned my addr 172.21.15.125:0/3913123255 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:03.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed2066640 1 --2- 172.21.15.125:0/3913123255 >> v2:172.21.15.136:3300/0 conn(0x7f0ecc0cf160 0x7f0ecc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:03.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed1064640 1 -- 172.21.15.125:0/3913123255 >> v2:172.21.15.136:3300/0 conn(0x7f0ecc0cf160 msgr2=0x7f0ecc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:03.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed1064640 1 --2- 172.21.15.125:0/3913123255 >> v2:172.21.15.136:3300/0 conn(0x7f0ecc0cf160 0x7f0ecc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:03.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed1064640 1 -- 172.21.15.125:0/3913123255 >> v1:172.21.15.136:6789/0 conn(0x7f0ecc1538a0 legacy=0x7f0ecc0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:03.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.169+0000 7f0ed1064640 1 -- 172.21.15.125:0/3913123255 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ecc0cb050 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.173+0000 7f0ed1064640 1 --2- 172.21.15.125:0/3913123255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ecc15d6f0 0x7f0ecc15a090 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0b5330 tx=0x7f0ebc01ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9fae67a667abefe server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:03.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.173+0000 7f0ed0863640 1 -- 172.21.15.125:0/3913123255 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc0a89f0 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.173+0000 7f0ed0863640 1 -- 172.21.15.125:0/3913123255 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ebc0a8b50 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.173+0000 7f0ed0863640 1 -- 172.21.15.125:0/3913123255 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc0b0440 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.173+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/3913123255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ecc15d6f0 msgr2=0x7f0ecc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:03.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.173+0000 7f0ed3ac9640 1 --2- 172.21.15.125:0/3913123255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ecc15d6f0 0x7f0ecc15a090 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc0b5330 tx=0x7f0ebc01ecf0 comp rx=0 tx=0).stop 2023-10-30T13:18:03.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.173+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/3913123255 shutdown_connections 2023-10-30T13:18:03.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.173+0000 7f0ed3ac9640 1 --2- 172.21.15.125:0/3913123255 >> v2:172.21.15.136:3300/0 conn(0x7f0ecc0cf160 0x7f0ecc0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:03.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.173+0000 7f0ed3ac9640 1 --2- 172.21.15.125:0/3913123255 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ecc15d6f0 0x7f0ecc15a090 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:03.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.173+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/3913123255 >> 172.21.15.125:0/3913123255 conn(0x7f0ecc06bfe0 msgr2=0x7f0ecc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:03.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.173+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/3913123255 shutdown_connections 2023-10-30T13:18:03.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.173+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/3913123255 wait complete. 2023-10-30T13:18:03.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed3ac9640 1 Processor -- start 2023-10-30T13:18:03.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed3ac9640 1 -- start start 2023-10-30T13:18:03.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed3ac9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ecc0cf160 0x7f0ecc1e3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:03.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed3ac9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ecc1538a0 0x7f0ecc0b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:03.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed3ac9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ecc15d6f0 0x7f0ecc0b4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:03.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed3ac9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc14e2d0 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed3ac9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc0b33b0 con 0x7f0ecc1538a0 2023-10-30T13:18:03.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed3ac9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ecc0cd9e0 con 0x7f0ecc0cf160 2023-10-30T13:18:03.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed2066640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ecc15d6f0 0x7f0ecc0b4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:03.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed2066640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ecc15d6f0 0x7f0ecc0b4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34198/0 (socket says 172.21.15.125:34198) 2023-10-30T13:18:03.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed2066640 1 -- 172.21.15.125:0/1848405487 learned_addr learned my addr 172.21.15.125:0/1848405487 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:03.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed1064640 1 --2- 172.21.15.125:0/1848405487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ecc1538a0 0x7f0ecc0b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:03.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed1865640 1 --2- 172.21.15.125:0/1848405487 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ecc0cf160 0x7f0ecc1e3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:03.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed2066640 1 -- 172.21.15.125:0/1848405487 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ecc0cf160 msgr2=0x7f0ecc1e3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:03.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed2066640 1 --2- 172.21.15.125:0/1848405487 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ecc0cf160 0x7f0ecc1e3e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:03.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed2066640 1 -- 172.21.15.125:0/1848405487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ecc1538a0 msgr2=0x7f0ecc0b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:03.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed2066640 1 --2- 172.21.15.125:0/1848405487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ecc1538a0 0x7f0ecc0b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:03.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed2066640 1 -- 172.21.15.125:0/1848405487 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ebc0094d0 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed2066640 1 --2- 172.21.15.125:0/1848405487 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ecc15d6f0 0x7f0ecc0b4720 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4007dc0 tx=0x7f0ec400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:03.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ec27fc640 1 -- 172.21.15.125:0/1848405487 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4014420 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ec27fc640 1 -- 172.21.15.125:0/1848405487 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ec40145a0 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ec27fc640 1 -- 172.21.15.125:0/1848405487 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ec4014950 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/1848405487 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ecc0ceaa0 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.177+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/1848405487 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ecc0cdd80 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.181+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/1848405487 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ecc0b2a50 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.181+0000 7f0ec27fc640 1 -- 172.21.15.125:0/1848405487 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0ec4014ab0 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.181+0000 7f0ec27fc640 1 --2- 172.21.15.125:0/1848405487 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0eb00664a0 0x7f0eb0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:03.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.181+0000 7f0ec27fc640 1 -- 172.21.15.125:0/1848405487 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f0ec40850a0 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.181+0000 7f0ed1865640 1 --2- 172.21.15.125:0/1848405487 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0eb00664a0 0x7f0eb0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:03.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.181+0000 7f0ed1865640 1 --2- 172.21.15.125:0/1848405487 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0eb00664a0 0x7f0eb0068960 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8007fd0 tx=0x7f0ec8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:03.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.185+0000 7f0ec27fc640 1 -- 172.21.15.125:0/1848405487 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0ec4051b50 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.373+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/1848405487 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ecc0cec30 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.373+0000 7f0ec27fc640 1 -- 172.21.15.125:0/1848405487 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 230 v230) v1 ==== 95+0+5754 (secure 0 0 0) 0x7f0ec4055170 con 0x7f0ecc15d6f0 2023-10-30T13:18:03.378 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:18:03.378 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":230,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":230,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:17:55.811977+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:03.379 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 230 2023-10-30T13:18:03.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.373+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/1848405487 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0eb00664a0 msgr2=0x7f0eb0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:03.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.373+0000 7f0ed3ac9640 1 --2- 172.21.15.125:0/1848405487 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0eb00664a0 0x7f0eb0068960 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8007fd0 tx=0x7f0ec8009040 comp rx=0 tx=0).stop 2023-10-30T13:18:03.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.373+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/1848405487 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ecc15d6f0 msgr2=0x7f0ecc0b4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:03.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.373+0000 7f0ed3ac9640 1 --2- 172.21.15.125:0/1848405487 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ecc15d6f0 0x7f0ecc0b4720 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f0ec4007dc0 tx=0x7f0ec400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:18:03.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.373+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/1848405487 shutdown_connections 2023-10-30T13:18:03.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.373+0000 7f0ed3ac9640 1 --2- 172.21.15.125:0/1848405487 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0eb00664a0 0x7f0eb0068960 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:03.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.373+0000 7f0ed3ac9640 1 --2- 172.21.15.125:0/1848405487 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0ecc15d6f0 0x7f0ecc0b4720 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:03.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.373+0000 7f0ed3ac9640 1 --2- 172.21.15.125:0/1848405487 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0ecc1538a0 0x7f0ecc0b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:03.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.373+0000 7f0ed3ac9640 1 --2- 172.21.15.125:0/1848405487 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0ecc0cf160 0x7f0ecc1e3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:03.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.373+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/1848405487 >> 172.21.15.125:0/1848405487 conn(0x7f0ecc06bfe0 msgr2=0x7f0ecc1e5360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:03.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.373+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/1848405487 shutdown_connections 2023-10-30T13:18:03.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:03.373+0000 7f0ed3ac9640 1 -- 172.21.15.125:0/1848405487 wait complete. 2023-10-30T13:18:03.401 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:05.402 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:05.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ecee09640 1 Processor -- start 2023-10-30T13:18:05.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ecee09640 1 -- start start 2023-10-30T13:18:05.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ecee09640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9ec80cf160 0x7f9ec80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:05.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ecee09640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec80cb800 con 0x7f9ec815d6f0 2023-10-30T13:18:05.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ecee09640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec80cb330 con 0x7f9ec81538a0 2023-10-30T13:18:05.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ecee09640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec80cad70 con 0x7f9ec80cf160 2023-10-30T13:18:05.549 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebffff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9ec815d6f0 0x7f9ec815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39344/0 (socket says 172.21.15.125:39344) 2023-10-30T13:18:05.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebffff640 1 -- 172.21.15.125:0/165003120 learned_addr learned my addr 172.21.15.125:0/165003120 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:18:05.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ecd3a6640 1 --2- 172.21.15.125:0/165003120 >> v2:172.21.15.136:3300/0 conn(0x7f9ec80cf160 0x7f9ec80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:05.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364638282 0 0) 0x7f9ec80cb800 con 0x7f9ec815d6f0 2023-10-30T13:18:05.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eb0003530 con 0x7f9ec815d6f0 2023-10-30T13:18:05.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025369894 0 0) 0x7f9ec80cb330 con 0x7f9ec81538a0 2023-10-30T13:18:05.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ec80cb800 con 0x7f9ec81538a0 2023-10-30T13:18:05.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 643985018 0 0) 0x7f9eb0003530 con 0x7f9ec815d6f0 2023-10-30T13:18:05.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 >> v2:172.21.15.136:3300/0 conn(0x7f9ec80cf160 msgr2=0x7f9ec80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:05.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 --2- 172.21.15.125:0/165003120 >> v2:172.21.15.136:3300/0 conn(0x7f9ec80cf160 0x7f9ec80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:05.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 >> v1:172.21.15.136:6789/0 conn(0x7f9ec81538a0 legacy=0x7f9ec80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:05.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec80cb050 con 0x7f9ec815d6f0 2023-10-30T13:18:05.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f9eb8001680 con 0x7f9ec815d6f0 2023-10-30T13:18:05.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 >> v1:172.21.15.125:6789/0 conn(0x7f9ec815d6f0 legacy=0x7f9ec815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:05.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 --2- 172.21.15.125:0/165003120 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9eb0003b10 0x7f9eb0003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:05.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 --2- 172.21.15.125:0/165003120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9eb00047d0 0x7f9eb0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:05.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 --2- 172.21.15.125:0/165003120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9eb0005190 0x7f9eb000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:05.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec80cad70 con 0x7f9eb0005190 2023-10-30T13:18:05.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb000bf80 con 0x7f9eb00047d0 2023-10-30T13:18:05.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.541+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb000c140 con 0x7f9eb0003b10 2023-10-30T13:18:05.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.545+0000 7f9ecd3a6640 1 --2- 172.21.15.125:0/165003120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9eb0005190 0x7f9eb000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:05.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.545+0000 7f9ebffff640 1 --2- 172.21.15.125:0/165003120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9eb00047d0 0x7f9eb0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:05.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.545+0000 7f9eccba5640 1 --2- 172.21.15.125:0/165003120 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9eb0003b10 0x7f9eb0003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:05.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.545+0000 7f9ecd3a6640 1 -- 172.21.15.125:0/165003120 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9eb0003b10 msgr2=0x7f9eb0003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:05.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.545+0000 7f9ecd3a6640 1 --2- 172.21.15.125:0/165003120 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9eb0003b10 0x7f9eb0003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:05.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.545+0000 7f9ecd3a6640 1 -- 172.21.15.125:0/165003120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9eb00047d0 msgr2=0x7f9eb0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:05.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.545+0000 7f9ecd3a6640 1 --2- 172.21.15.125:0/165003120 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9eb00047d0 0x7f9eb0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:05.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.545+0000 7f9ecd3a6640 1 -- 172.21.15.125:0/165003120 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb0003530 con 0x7f9eb0005190 2023-10-30T13:18:05.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.545+0000 7f9ecd3a6640 1 -- 172.21.15.125:0/165003120 reap_dead start 2023-10-30T13:18:05.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.545+0000 7f9ecd3a6640 1 --2- 172.21.15.125:0/165003120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9eb0005190 0x7f9eb000b990 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0009f90 tx=0x7f9ec001cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1534f8d23ca3fbc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:05.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.545+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec00a5c70 con 0x7f9eb0005190 2023-10-30T13:18:05.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.545+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec000f040 con 0x7f9eb0005190 2023-10-30T13:18:05.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.545+0000 7f9ebf7fe640 1 -- 172.21.15.125:0/165003120 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec00039a0 con 0x7f9eb0005190 2023-10-30T13:18:05.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 -- 172.21.15.125:0/165003120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9eb0005190 msgr2=0x7f9eb000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:05.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 --2- 172.21.15.125:0/165003120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9eb0005190 0x7f9eb000b990 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0009f90 tx=0x7f9ec001cc60 comp rx=0 tx=0).stop 2023-10-30T13:18:05.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 -- 172.21.15.125:0/165003120 shutdown_connections 2023-10-30T13:18:05.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 --2- 172.21.15.125:0/165003120 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9eb0005190 0x7f9eb000b990 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:05.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 -- 172.21.15.125:0/165003120 >> 172.21.15.125:0/165003120 conn(0x7f9ec806bfe0 msgr2=0x7f9ec806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:05.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 -- 172.21.15.125:0/165003120 shutdown_connections 2023-10-30T13:18:05.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 -- 172.21.15.125:0/165003120 wait complete. 2023-10-30T13:18:05.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 Processor -- start 2023-10-30T13:18:05.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 -- start start 2023-10-30T13:18:05.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9eb0005190 0x7f9ec80b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:05.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ec80cd260 0x7f9ec81f2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:05.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ec80cd6a0 0x7f9ec81f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:05.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec815c730 con 0x7f9eb0005190 2023-10-30T13:18:05.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec80b3850 con 0x7f9ec80cd260 2023-10-30T13:18:05.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecee09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec80b3b30 con 0x7f9ec80cd6a0 2023-10-30T13:18:05.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9eccba5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9eb0005190 0x7f9ec80b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:05.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9eccba5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9eb0005190 0x7f9ec80b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34228/0 (socket says 172.21.15.125:34228) 2023-10-30T13:18:05.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9eccba5640 1 -- 172.21.15.125:0/1874235944 learned_addr learned my addr 172.21.15.125:0/1874235944 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:05.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ebffff640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ec80cd260 0x7f9ec81f2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:05.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9ecd3a6640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ec80cd6a0 0x7f9ec81f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:05.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9eccba5640 1 -- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ec80cd6a0 msgr2=0x7f9ec81f26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:05.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9eccba5640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ec80cd6a0 0x7f9ec81f26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:05.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.549+0000 7f9eccba5640 1 -- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ec80cd260 msgr2=0x7f9ec81f2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:05.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9eccba5640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ec80cd260 0x7f9ec81f2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:05.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9eccba5640 1 -- 172.21.15.125:0/1874235944 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec80b3e10 con 0x7f9eb0005190 2023-10-30T13:18:05.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9ecd3a6640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ec80cd6a0 0x7f9ec81f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:05.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9ebffff640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ec80cd260 0x7f9ec81f2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:05.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9eccba5640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9eb0005190 0x7f9ec80b31a0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400e500 tx=0x7f9ec400e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:05.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9ebd7fa640 1 -- 172.21.15.125:0/1874235944 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec40052e0 con 0x7f9eb0005190 2023-10-30T13:18:05.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9ebd7fa640 1 -- 172.21.15.125:0/1874235944 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec4005460 con 0x7f9eb0005190 2023-10-30T13:18:05.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9ebd7fa640 1 -- 172.21.15.125:0/1874235944 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ec4003840 con 0x7f9eb0005190 2023-10-30T13:18:05.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9ecee09640 1 -- 172.21.15.125:0/1874235944 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec80b40f0 con 0x7f9eb0005190 2023-10-30T13:18:05.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9ecee09640 1 -- 172.21.15.125:0/1874235944 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec80b46b0 con 0x7f9eb0005190 2023-10-30T13:18:05.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9ecee09640 1 -- 172.21.15.125:0/1874235944 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec804e170 con 0x7f9eb0005190 2023-10-30T13:18:05.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9ebd7fa640 1 -- 172.21.15.125:0/1874235944 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9ec40039a0 con 0x7f9eb0005190 2023-10-30T13:18:05.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9ebd7fa640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9eb40664d0 0x7f9eb4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:05.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9ebd7fa640 1 -- 172.21.15.125:0/1874235944 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f9ec4084720 con 0x7f9eb0005190 2023-10-30T13:18:05.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.553+0000 7f9ebffff640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9eb40664d0 0x7f9eb4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:05.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.557+0000 7f9ebffff640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9eb40664d0 0x7f9eb4068990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8001060 tx=0x7f9eb8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:05.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.557+0000 7f9ebd7fa640 1 -- 172.21.15.125:0/1874235944 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9ec40510f0 con 0x7f9eb0005190 2023-10-30T13:18:05.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.737+0000 7f9ecee09640 1 -- 172.21.15.125:0/1874235944 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ec80b1dd0 con 0x7f9eb0005190 2023-10-30T13:18:05.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.737+0000 7f9ebd7fa640 1 -- 172.21.15.125:0/1874235944 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f9ec4054710 con 0x7f9eb0005190 2023-10-30T13:18:05.744 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:18:05.744 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:18:04.766172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:resolve","state_seq":26,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:05.745 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 231 2023-10-30T13:18:05.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.741+0000 7f9ecee09640 1 -- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9eb40664d0 msgr2=0x7f9eb4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:05.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.741+0000 7f9ecee09640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9eb40664d0 0x7f9eb4068990 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8001060 tx=0x7f9eb8009040 comp rx=0 tx=0).stop 2023-10-30T13:18:05.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.741+0000 7f9ecee09640 1 -- 172.21.15.125:0/1874235944 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9eb0005190 msgr2=0x7f9ec80b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:05.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.741+0000 7f9ecee09640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9eb0005190 0x7f9ec80b31a0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f9ec400e500 tx=0x7f9ec400e9c0 comp rx=0 tx=0).stop 2023-10-30T13:18:05.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.741+0000 7f9ecee09640 1 -- 172.21.15.125:0/1874235944 shutdown_connections 2023-10-30T13:18:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.741+0000 7f9ecee09640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9eb40664d0 0x7f9eb4068990 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.741+0000 7f9ecee09640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ec80cd6a0 0x7f9ec81f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.741+0000 7f9ecee09640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ec80cd260 0x7f9ec81f2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.741+0000 7f9ecee09640 1 --2- 172.21.15.125:0/1874235944 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9eb0005190 0x7f9ec80b31a0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:05.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.741+0000 7f9ecee09640 1 -- 172.21.15.125:0/1874235944 >> 172.21.15.125:0/1874235944 conn(0x7f9ec806bfe0 msgr2=0x7f9ec8073650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:05.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.741+0000 7f9ecee09640 1 -- 172.21.15.125:0/1874235944 shutdown_connections 2023-10-30T13:18:05.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:05.741+0000 7f9ecee09640 1 -- 172.21.15.125:0/1874235944 wait complete. 2023-10-30T13:18:05.765 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:07.767 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:07.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c0de09640 1 Processor -- start 2023-10-30T13:18:07.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c0de09640 1 -- start start 2023-10-30T13:18:07.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c0de09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c081538a0 0x7f1c0815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:07.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c0de09640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1c080cf160 0x7f1c080cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:07.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c0de09640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c080cb800 con 0x7f1c080ca0e0 2023-10-30T13:18:07.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c0de09640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c0815dc30 con 0x7f1c080cf160 2023-10-30T13:18:07.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c0de09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c080cb330 con 0x7f1c081538a0 2023-10-30T13:18:07.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c06ffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1c080ca0e0 0x7f1c0815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39350/0 (socket says 172.21.15.125:39350) 2023-10-30T13:18:07.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c06ffd640 1 -- 172.21.15.125:0/938763755 learned_addr learned my addr 172.21.15.125:0/938763755 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:18:07.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c07fff640 1 --2- 172.21.15.125:0/938763755 >> v2:172.21.15.136:3300/0 conn(0x7f1c080cf160 0x7f1c080cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:07.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c077fe640 1 --2- 172.21.15.125:0/938763755 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c081538a0 0x7f1c0815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:07.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 -- 172.21.15.125:0/938763755 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2437818781 0 0) 0x7f1c080cb800 con 0x7f1c080ca0e0 2023-10-30T13:18:07.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 -- 172.21.15.125:0/938763755 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf0003530 con 0x7f1c080ca0e0 2023-10-30T13:18:07.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 -- 172.21.15.125:0/938763755 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4046901217 0 0) 0x7f1bf0003530 con 0x7f1c080ca0e0 2023-10-30T13:18:07.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 -- 172.21.15.125:0/938763755 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c081538a0 msgr2=0x7f1c0815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:07.918 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 --2- 172.21.15.125:0/938763755 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c081538a0 0x7f1c0815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:07.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 -- 172.21.15.125:0/938763755 >> v2:172.21.15.136:3300/0 conn(0x7f1c080cf160 msgr2=0x7f1c080cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:07.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 --2- 172.21.15.125:0/938763755 >> v2:172.21.15.136:3300/0 conn(0x7f1c080cf160 0x7f1c080cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:07.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 -- 172.21.15.125:0/938763755 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c080cad70 con 0x7f1c080ca0e0 2023-10-30T13:18:07.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 -- 172.21.15.125:0/938763755 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f1bfc001680 con 0x7f1c080ca0e0 2023-10-30T13:18:07.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 -- 172.21.15.125:0/938763755 >> v1:172.21.15.125:6789/0 conn(0x7f1c080ca0e0 legacy=0x7f1c0815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:07.919 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 --2- 172.21.15.125:0/938763755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bf0003980 0x7f1bf0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 --2- 172.21.15.125:0/938763755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bf00047d0 0x7f1bf0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 --2- 172.21.15.125:0/938763755 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c081538a0 0x7f1bf0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 -- 172.21.15.125:0/938763755 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0815dc30 con 0x7f1bf00047d0 2023-10-30T13:18:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 -- 172.21.15.125:0/938763755 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c080cb330 con 0x7f1bf0003980 2023-10-30T13:18:07.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c067fc640 1 -- 172.21.15.125:0/938763755 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf000b990 con 0x7f1c081538a0 2023-10-30T13:18:07.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c06ffd640 1 --2- 172.21.15.125:0/938763755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bf00047d0 0x7f1bf0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:07.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c077fe640 1 --2- 172.21.15.125:0/938763755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bf0003980 0x7f1bf0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:07.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c07fff640 1 --2- 172.21.15.125:0/938763755 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c081538a0 0x7f1bf0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:07.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c06ffd640 1 -- 172.21.15.125:0/938763755 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c081538a0 msgr2=0x7f1bf0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:07.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c06ffd640 1 --2- 172.21.15.125:0/938763755 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c081538a0 0x7f1bf0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:07.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c06ffd640 1 -- 172.21.15.125:0/938763755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bf0003980 msgr2=0x7f1bf0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:07.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c06ffd640 1 --2- 172.21.15.125:0/938763755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bf0003980 0x7f1bf0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:07.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c06ffd640 1 -- 172.21.15.125:0/938763755 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf0003530 con 0x7f1bf00047d0 2023-10-30T13:18:07.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.909+0000 7f1c077fe640 1 --2- 172.21.15.125:0/938763755 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bf0003980 0x7f1bf0003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:07.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.913+0000 7f1c06ffd640 1 --2- 172.21.15.125:0/938763755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bf00047d0 0x7f1bf0004c50 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc00c240 tx=0x7f1bfc01cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51ef941948347ff server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:07.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.913+0000 7f1c067fc640 1 -- 172.21.15.125:0/938763755 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc0058e0 con 0x7f1bf00047d0 2023-10-30T13:18:07.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.913+0000 7f1c067fc640 1 -- 172.21.15.125:0/938763755 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bfc005a40 con 0x7f1bf00047d0 2023-10-30T13:18:07.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.913+0000 7f1c067fc640 1 -- 172.21.15.125:0/938763755 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bfc005e10 con 0x7f1bf00047d0 2023-10-30T13:18:07.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.913+0000 7f1c0de09640 1 -- 172.21.15.125:0/938763755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bf00047d0 msgr2=0x7f1bf0004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:07.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.913+0000 7f1c0de09640 1 --2- 172.21.15.125:0/938763755 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bf00047d0 0x7f1bf0004c50 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc00c240 tx=0x7f1bfc01cc60 comp rx=0 tx=0).stop 2023-10-30T13:18:07.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.913+0000 7f1c07fff640 1 -- 172.21.15.125:0/938763755 reap_dead start 2023-10-30T13:18:07.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.913+0000 7f1c0de09640 1 -- 172.21.15.125:0/938763755 shutdown_connections 2023-10-30T13:18:07.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.913+0000 7f1c0de09640 1 -- 172.21.15.125:0/938763755 >> 172.21.15.125:0/938763755 conn(0x7f1c0806bfe0 msgr2=0x7f1c0806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:07.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.913+0000 7f1c0de09640 1 -- 172.21.15.125:0/938763755 shutdown_connections 2023-10-30T13:18:07.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.913+0000 7f1c0de09640 1 -- 172.21.15.125:0/938763755 wait complete. 2023-10-30T13:18:07.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c0de09640 1 Processor -- start 2023-10-30T13:18:07.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c0de09640 1 -- start start 2023-10-30T13:18:07.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c0de09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bf00047d0 0x7f1c080cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:07.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c0de09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c080ce4b0 0x7f1c08147dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:07.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c0de09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c081593d0 0x7f1c080b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:07.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c0de09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c080cd840 con 0x7f1bf00047d0 2023-10-30T13:18:07.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c0de09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0815cc50 con 0x7f1c080ce4b0 2023-10-30T13:18:07.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c0de09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c080b1960 con 0x7f1c081593d0 2023-10-30T13:18:07.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c077fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bf00047d0 0x7f1c080cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:07.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c077fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bf00047d0 0x7f1c080cdca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34252/0 (socket says 172.21.15.125:34252) 2023-10-30T13:18:07.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c077fe640 1 -- 172.21.15.125:0/60478088 learned_addr learned my addr 172.21.15.125:0/60478088 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:07.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c06ffd640 1 --2- 172.21.15.125:0/60478088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c080ce4b0 0x7f1c08147dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:07.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c07fff640 1 --2- 172.21.15.125:0/60478088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c081593d0 0x7f1c080b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:07.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c077fe640 1 -- 172.21.15.125:0/60478088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c081593d0 msgr2=0x7f1c080b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:07.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c077fe640 1 --2- 172.21.15.125:0/60478088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c081593d0 0x7f1c080b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:07.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c077fe640 1 -- 172.21.15.125:0/60478088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c080ce4b0 msgr2=0x7f1c08147dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:07.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c077fe640 1 --2- 172.21.15.125:0/60478088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c080ce4b0 0x7f1c08147dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:07.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c077fe640 1 -- 172.21.15.125:0/60478088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c080ce1e0 con 0x7f1bf00047d0 2023-10-30T13:18:07.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c077fe640 1 --2- 172.21.15.125:0/60478088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bf00047d0 0x7f1c080cdca0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009160 tx=0x7f1bf8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1beffff640 1 -- 172.21.15.125:0/60478088 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8004220 con 0x7f1bf00047d0 2023-10-30T13:18:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1beffff640 1 -- 172.21.15.125:0/60478088 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf8004380 con 0x7f1bf00047d0 2023-10-30T13:18:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c0de09640 1 -- 172.21.15.125:0/60478088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c080b1bb0 con 0x7f1bf00047d0 2023-10-30T13:18:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1beffff640 1 -- 172.21.15.125:0/60478088 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8015890 con 0x7f1bf00047d0 2023-10-30T13:18:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.917+0000 7f1c0de09640 1 -- 172.21.15.125:0/60478088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c080ceb70 con 0x7f1bf00047d0 2023-10-30T13:18:07.931 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.921+0000 7f1c0de09640 1 -- 172.21.15.125:0/60478088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c080ce9d0 con 0x7f1bf00047d0 2023-10-30T13:18:07.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.921+0000 7f1beffff640 1 -- 172.21.15.125:0/60478088 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1bf80159f0 con 0x7f1bf00047d0 2023-10-30T13:18:07.932 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.921+0000 7f1beffff640 1 --2- 172.21.15.125:0/60478088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bf40664d0 0x7f1bf4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:07.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.921+0000 7f1beffff640 1 -- 172.21.15.125:0/60478088 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f1bf8084b60 con 0x7f1bf00047d0 2023-10-30T13:18:07.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.921+0000 7f1c06ffd640 1 --2- 172.21.15.125:0/60478088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bf40664d0 0x7f1bf4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:07.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.921+0000 7f1c06ffd640 1 --2- 172.21.15.125:0/60478088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bf40664d0 0x7f1bf4068990 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc0aa000 tx=0x7f1bfc001a80 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:07.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:07.925+0000 7f1beffff640 1 -- 172.21.15.125:0/60478088 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1c080ce9d0 con 0x7f1bf00047d0 2023-10-30T13:18:08.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:08.105+0000 7f1c0de09640 1 -- 172.21.15.125:0/60478088 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c080ce9d0 con 0x7f1bf00047d0 2023-10-30T13:18:08.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:08.105+0000 7f1beffff640 1 -- 172.21.15.125:0/60478088 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f1c080ce9d0 con 0x7f1bf00047d0 2023-10-30T13:18:08.112 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:18:08.113 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:18:04.766172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:resolve","state_seq":26,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:08.113 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 231 2023-10-30T13:18:08.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:08.109+0000 7f1c0de09640 1 -- 172.21.15.125:0/60478088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bf40664d0 msgr2=0x7f1bf4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:08.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:08.109+0000 7f1c0de09640 1 --2- 172.21.15.125:0/60478088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bf40664d0 0x7f1bf4068990 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc0aa000 tx=0x7f1bfc001a80 comp rx=0 tx=0).stop 2023-10-30T13:18:08.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:08.109+0000 7f1c0de09640 1 -- 172.21.15.125:0/60478088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bf00047d0 msgr2=0x7f1c080cdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:08.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:08.109+0000 7f1c0de09640 1 --2- 172.21.15.125:0/60478088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bf00047d0 0x7f1c080cdca0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f1bf8009160 tx=0x7f1bf8011a60 comp rx=0 tx=0).stop 2023-10-30T13:18:08.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:08.109+0000 7f1c0de09640 1 -- 172.21.15.125:0/60478088 shutdown_connections 2023-10-30T13:18:08.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:08.109+0000 7f1c0de09640 1 --2- 172.21.15.125:0/60478088 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1bf40664d0 0x7f1bf4068990 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:08.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:08.109+0000 7f1c0de09640 1 --2- 172.21.15.125:0/60478088 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1c081593d0 0x7f1c080b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:08.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:08.109+0000 7f1c0de09640 1 --2- 172.21.15.125:0/60478088 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1c080ce4b0 0x7f1c08147dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:08.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:08.109+0000 7f1c0de09640 1 --2- 172.21.15.125:0/60478088 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bf00047d0 0x7f1c080cdca0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:08.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:08.109+0000 7f1c0de09640 1 -- 172.21.15.125:0/60478088 >> 172.21.15.125:0/60478088 conn(0x7f1c0806bfe0 msgr2=0x7f1c08152550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:08.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:08.109+0000 7f1c0de09640 1 -- 172.21.15.125:0/60478088 shutdown_connections 2023-10-30T13:18:08.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:08.109+0000 7f1c0de09640 1 -- 172.21.15.125:0/60478088 wait complete. 2023-10-30T13:18:08.135 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 231, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 231, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_8833": { "addr": "172.21.15.136:6835/3864637420", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3864637420, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3864637420, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8833, "incarnation": 222, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 15 }, "gid_8971": { "addr": "172.21.15.136:6839/4284345684", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 4284345684, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 4284345684, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8971, "incarnation": 230, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:resolve", "state_seq": 26 }, "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 82, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:18:04.766172+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8971, "mds_1": 9150, "mds_2": 8833 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:18:10.136 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:10.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.277+0000 7f58b3b3e640 1 Processor -- start 2023-10-30T13:18:10.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.277+0000 7f58b3b3e640 1 -- start start 2023-10-30T13:18:10.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.281+0000 7f58b3b3e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f58ac0c05d0 0x7f58ac13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:10.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.281+0000 7f58b3b3e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f58ac0c6f60 0x7f58ac13f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:10.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.281+0000 7f58b3b3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ac0c0a50 con 0x7f58ac0c6b80 2023-10-30T13:18:10.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.281+0000 7f58b3b3e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ac06f6e0 con 0x7f58ac0c6f60 2023-10-30T13:18:10.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.281+0000 7f58b3b3e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ac06f9c0 con 0x7f58ac0c05d0 2023-10-30T13:18:10.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.281+0000 7f58b20db640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f58ac0c6f60 0x7f58ac13f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:10.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.281+0000 7f58b20db640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f58ac0c6f60 0x7f58ac13f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34264/0 (socket says 172.21.15.125:34264) 2023-10-30T13:18:10.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.281+0000 7f58b20db640 1 -- 172.21.15.125:0/2883818558 learned_addr learned my addr 172.21.15.125:0/2883818558 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:10.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.281+0000 7f58b10d9640 1 --2- 172.21.15.125:0/2883818558 >> v2:172.21.15.136:3300/0 conn(0x7f58ac0c05d0 0x7f58ac13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:10.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.281+0000 7f58b20db640 1 -- 172.21.15.125:0/2883818558 >> v2:172.21.15.136:3300/0 conn(0x7f58ac0c05d0 msgr2=0x7f58ac13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:10.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.281+0000 7f58b20db640 1 --2- 172.21.15.125:0/2883818558 >> v2:172.21.15.136:3300/0 conn(0x7f58ac0c05d0 0x7f58ac13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:10.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.281+0000 7f58b20db640 1 -- 172.21.15.125:0/2883818558 >> v1:172.21.15.136:6789/0 conn(0x7f58ac0c6b80 legacy=0x7f58ac0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:10.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.281+0000 7f58b20db640 1 -- 172.21.15.125:0/2883818558 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ac06fca0 con 0x7f58ac0c6f60 2023-10-30T13:18:10.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.285+0000 7f58b20db640 1 --2- 172.21.15.125:0/2883818558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58ac0c6f60 0x7f58ac13f2d0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f58ac06c7e0 tx=0x7f58a401ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a6995d50a32a17f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:10.289 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.285+0000 7f58b08d8640 1 -- 172.21.15.125:0/2883818558 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a400b2d0 con 0x7f58ac0c6f60 2023-10-30T13:18:10.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.285+0000 7f58b08d8640 1 -- 172.21.15.125:0/2883818558 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a400b430 con 0x7f58ac0c6f60 2023-10-30T13:18:10.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.285+0000 7f58b08d8640 1 -- 172.21.15.125:0/2883818558 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a40a7bd0 con 0x7f58ac0c6f60 2023-10-30T13:18:10.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.285+0000 7f58b3b3e640 1 -- 172.21.15.125:0/2883818558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58ac0c6f60 msgr2=0x7f58ac13f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:10.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.285+0000 7f58b3b3e640 1 --2- 172.21.15.125:0/2883818558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58ac0c6f60 0x7f58ac13f2d0 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f58ac06c7e0 tx=0x7f58a401ebd0 comp rx=0 tx=0).stop 2023-10-30T13:18:10.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.285+0000 7f58b3b3e640 1 -- 172.21.15.125:0/2883818558 shutdown_connections 2023-10-30T13:18:10.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.285+0000 7f58b3b3e640 1 --2- 172.21.15.125:0/2883818558 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58ac0c6f60 0x7f58ac13f2d0 unknown :-1 s=CLOSED pgs=1957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:10.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.285+0000 7f58b3b3e640 1 --2- 172.21.15.125:0/2883818558 >> v2:172.21.15.136:3300/0 conn(0x7f58ac0c05d0 0x7f58ac13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:10.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.285+0000 7f58b3b3e640 1 -- 172.21.15.125:0/2883818558 >> 172.21.15.125:0/2883818558 conn(0x7f58ac0c7910 msgr2=0x7f58ac0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:10.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.285+0000 7f58b3b3e640 1 -- 172.21.15.125:0/2883818558 shutdown_connections 2023-10-30T13:18:10.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.285+0000 7f58b3b3e640 1 -- 172.21.15.125:0/2883818558 wait complete. 2023-10-30T13:18:10.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.285+0000 7f58b3b3e640 1 Processor -- start 2023-10-30T13:18:10.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.285+0000 7f58b3b3e640 1 -- start start 2023-10-30T13:18:10.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b3b3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58ac0c05d0 0x7f58ac14fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:10.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b3b3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58ac0c6b80 0x7f58ac155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:10.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b3b3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58ac0c6f60 0x7f58ac13caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:10.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b3b3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac1500d0 con 0x7f58ac0c6b80 2023-10-30T13:18:10.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b3b3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac13e480 con 0x7f58ac0c05d0 2023-10-30T13:18:10.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b3b3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ac1549e0 con 0x7f58ac0c6f60 2023-10-30T13:18:10.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b10d9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58ac0c6b80 0x7f58ac155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:10.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b10d9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58ac0c6b80 0x7f58ac155dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:34278/0 (socket says 172.21.15.125:34278) 2023-10-30T13:18:10.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b10d9640 1 -- 172.21.15.125:0/195703647 learned_addr learned my addr 172.21.15.125:0/195703647 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:10.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b18da640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58ac0c05d0 0x7f58ac14fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:10.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b20db640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58ac0c6f60 0x7f58ac13caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:10.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b10d9640 1 -- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58ac0c6f60 msgr2=0x7f58ac13caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:10.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b10d9640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58ac0c6f60 0x7f58ac13caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:10.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b10d9640 1 -- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58ac0c05d0 msgr2=0x7f58ac14fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:10.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b10d9640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58ac0c05d0 0x7f58ac14fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:10.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b10d9640 1 -- 172.21.15.125:0/195703647 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a400bd40 con 0x7f58ac0c6b80 2023-10-30T13:18:10.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b20db640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58ac0c6f60 0x7f58ac13caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:10.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b10d9640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58ac0c6b80 0x7f58ac155dc0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f589c00d7d0 tx=0x7f589c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:10.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58a27fc640 1 -- 172.21.15.125:0/195703647 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c00b6c0 con 0x7f58ac0c6b80 2023-10-30T13:18:10.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58a27fc640 1 -- 172.21.15.125:0/195703647 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c00b820 con 0x7f58ac0c6b80 2023-10-30T13:18:10.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58a27fc640 1 -- 172.21.15.125:0/195703647 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c00bbd0 con 0x7f58ac0c6b80 2023-10-30T13:18:10.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b3b3e640 1 -- 172.21.15.125:0/195703647 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ac14bc50 con 0x7f58ac0c6b80 2023-10-30T13:18:10.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b3b3e640 1 -- 172.21.15.125:0/195703647 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ac154720 con 0x7f58ac0c6b80 2023-10-30T13:18:10.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58b3b3e640 1 -- 172.21.15.125:0/195703647 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ac04e170 con 0x7f58ac0c6b80 2023-10-30T13:18:10.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.289+0000 7f58a27fc640 1 -- 172.21.15.125:0/195703647 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f589c00bd30 con 0x7f58ac0c6b80 2023-10-30T13:18:10.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.293+0000 7f58a27fc640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58900664a0 0x7f5890068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:10.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.293+0000 7f58a27fc640 1 -- 172.21.15.125:0/195703647 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f589c084c60 con 0x7f58ac0c6b80 2023-10-30T13:18:10.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.293+0000 7f58b18da640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58900664a0 0x7f5890068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:10.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.293+0000 7f58b18da640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58900664a0 0x7f5890068960 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f58a8007fd0 tx=0x7f58a8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:10.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.297+0000 7f58a27fc640 1 -- 172.21.15.125:0/195703647 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f589c051740 con 0x7f58ac0c6b80 2023-10-30T13:18:10.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.477+0000 7f58b3b3e640 1 -- 172.21.15.125:0/195703647 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58ac0c1280 con 0x7f58ac0c6b80 2023-10-30T13:18:10.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.477+0000 7f58a27fc640 1 -- 172.21.15.125:0/195703647 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 231 v231) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f589c054d60 con 0x7f58ac0c6b80 2023-10-30T13:18:10.483 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:18:10.483 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":231,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":231,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:18:04.766172+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:resolve","state_seq":26,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:10.484 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 231 2023-10-30T13:18:10.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.481+0000 7f58b3b3e640 1 -- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58900664a0 msgr2=0x7f5890068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:10.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.481+0000 7f58b3b3e640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58900664a0 0x7f5890068960 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f58a8007fd0 tx=0x7f58a8009040 comp rx=0 tx=0).stop 2023-10-30T13:18:10.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.481+0000 7f58b3b3e640 1 -- 172.21.15.125:0/195703647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58ac0c6b80 msgr2=0x7f58ac155dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:10.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.481+0000 7f58b3b3e640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58ac0c6b80 0x7f58ac155dc0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7f589c00d7d0 tx=0x7f589c00dc90 comp rx=0 tx=0).stop 2023-10-30T13:18:10.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.481+0000 7f58b3b3e640 1 -- 172.21.15.125:0/195703647 shutdown_connections 2023-10-30T13:18:10.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.481+0000 7f58b3b3e640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58900664a0 0x7f5890068960 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:10.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.481+0000 7f58b3b3e640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f58ac0c6f60 0x7f58ac13caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:10.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.481+0000 7f58b3b3e640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58ac0c6b80 0x7f58ac155dc0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:10.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.481+0000 7f58b3b3e640 1 --2- 172.21.15.125:0/195703647 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f58ac0c05d0 0x7f58ac14fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:10.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.481+0000 7f58b3b3e640 1 -- 172.21.15.125:0/195703647 >> 172.21.15.125:0/195703647 conn(0x7f58ac0c7910 msgr2=0x7f58ac0bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:10.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.481+0000 7f58b3b3e640 1 -- 172.21.15.125:0/195703647 shutdown_connections 2023-10-30T13:18:10.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:10.481+0000 7f58b3b3e640 1 -- 172.21.15.125:0/195703647 wait complete. 2023-10-30T13:18:10.504 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:12.505 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:12.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.645+0000 7fa9befe4640 1 Processor -- start 2023-10-30T13:18:12.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.649+0000 7fa9befe4640 1 -- start start 2023-10-30T13:18:12.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.649+0000 7fa9befe4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9b81538f0 0x7fa9b8159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:12.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.649+0000 7fa9befe4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa9b815cd50 0x7fa9b80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:12.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.649+0000 7fa9befe4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9b815d300 con 0x7fa9b80cefa0 2023-10-30T13:18:12.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.649+0000 7fa9befe4640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9b815d5e0 con 0x7fa9b815cd50 2023-10-30T13:18:12.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.649+0000 7fa9befe4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b815d8c0 con 0x7fa9b81538f0 2023-10-30T13:18:12.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.649+0000 7fa9bd7e1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa9b815cd50 0x7fa9b80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:12.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.649+0000 7fa9bd7e1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa9b815cd50 0x7fa9b80c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35998/0 (socket says 172.21.15.125:35998) 2023-10-30T13:18:12.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.649+0000 7fa9bd7e1640 1 -- 172.21.15.125:0/97931148 learned_addr learned my addr 172.21.15.125:0/97931148 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:12.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.649+0000 7fa9bdfe2640 1 --2- 172.21.15.125:0/97931148 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9b81538f0 0x7fa9b8159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:12.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.649+0000 7fa9bd7e1640 1 -- 172.21.15.125:0/97931148 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9b81538f0 msgr2=0x7fa9b8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:12.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.649+0000 7fa9bd7e1640 1 --2- 172.21.15.125:0/97931148 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9b81538f0 0x7fa9b8159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:12.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.649+0000 7fa9bd7e1640 1 -- 172.21.15.125:0/97931148 >> v1:172.21.15.136:6789/0 conn(0x7fa9b80cefa0 legacy=0x7fa9b80cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:12.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.649+0000 7fa9bd7e1640 1 -- 172.21.15.125:0/97931148 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9b815dba0 con 0x7fa9b815cd50 2023-10-30T13:18:12.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.653+0000 7fa9bd7e1640 1 --2- 172.21.15.125:0/97931148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9b815cd50 0x7fa9b80c9f90 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009870 tx=0x7fa9a801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22210210c1ce429 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:12.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.653+0000 7fa9bcfe0640 1 -- 172.21.15.125:0/97931148 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80a8a30 con 0x7fa9b815cd50 2023-10-30T13:18:12.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.653+0000 7fa9bcfe0640 1 -- 172.21.15.125:0/97931148 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a80a8b90 con 0x7fa9b815cd50 2023-10-30T13:18:12.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.653+0000 7fa9bcfe0640 1 -- 172.21.15.125:0/97931148 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80b0440 con 0x7fa9b815cd50 2023-10-30T13:18:12.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.653+0000 7fa9befe4640 1 -- 172.21.15.125:0/97931148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9b815cd50 msgr2=0x7fa9b80c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:12.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.653+0000 7fa9befe4640 1 --2- 172.21.15.125:0/97931148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9b815cd50 0x7fa9b80c9f90 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fa9a8009870 tx=0x7fa9a801ed30 comp rx=0 tx=0).stop 2023-10-30T13:18:12.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.653+0000 7fa9befe4640 1 -- 172.21.15.125:0/97931148 shutdown_connections 2023-10-30T13:18:12.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.653+0000 7fa9befe4640 1 --2- 172.21.15.125:0/97931148 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9b815cd50 0x7fa9b80c9f90 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:12.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.653+0000 7fa9befe4640 1 --2- 172.21.15.125:0/97931148 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9b81538f0 0x7fa9b8159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:12.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.653+0000 7fa9befe4640 1 -- 172.21.15.125:0/97931148 >> 172.21.15.125:0/97931148 conn(0x7fa9b806a640 msgr2=0x7fa9b8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:12.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.653+0000 7fa9befe4640 1 -- 172.21.15.125:0/97931148 shutdown_connections 2023-10-30T13:18:12.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.653+0000 7fa9befe4640 1 -- 172.21.15.125:0/97931148 wait complete. 2023-10-30T13:18:12.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.653+0000 7fa9befe4640 1 Processor -- start 2023-10-30T13:18:12.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9befe4640 1 -- start start 2023-10-30T13:18:12.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9befe4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9b80cefa0 0x7fa9b8144ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:12.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9befe4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9b81538f0 0x7fa9b81eef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:12.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9befe4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9b815cd50 0x7fa9b813f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:12.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9befe4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b81f2e20 con 0x7fa9b81538f0 2023-10-30T13:18:12.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9befe4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b80b3290 con 0x7fa9b80cefa0 2023-10-30T13:18:12.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9befe4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9b80b3570 con 0x7fa9b815cd50 2023-10-30T13:18:12.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9bd7e1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9b81538f0 0x7fa9b81eef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:12.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9bd7e1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9b81538f0 0x7fa9b81eef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36010/0 (socket says 172.21.15.125:36010) 2023-10-30T13:18:12.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9bd7e1640 1 -- 172.21.15.125:0/174840105 learned_addr learned my addr 172.21.15.125:0/174840105 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:12.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9be7e3640 1 --2- 172.21.15.125:0/174840105 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9b815cd50 0x7fa9b813f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:12.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9bdfe2640 1 --2- 172.21.15.125:0/174840105 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9b80cefa0 0x7fa9b8144ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:12.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9bd7e1640 1 -- 172.21.15.125:0/174840105 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9b815cd50 msgr2=0x7fa9b813f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:12.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9bd7e1640 1 --2- 172.21.15.125:0/174840105 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9b815cd50 0x7fa9b813f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:12.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9bd7e1640 1 -- 172.21.15.125:0/174840105 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9b80cefa0 msgr2=0x7fa9b8144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:12.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9bd7e1640 1 --2- 172.21.15.125:0/174840105 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9b80cefa0 0x7fa9b8144ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:12.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9bd7e1640 1 -- 172.21.15.125:0/174840105 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a80094d0 con 0x7fa9b81538f0 2023-10-30T13:18:12.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9bd7e1640 1 --2- 172.21.15.125:0/174840105 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9b81538f0 0x7fa9b81eef80 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80b3040 tx=0x7fa9a801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:12.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9ae7fc640 1 -- 172.21.15.125:0/174840105 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80b9620 con 0x7fa9b81538f0 2023-10-30T13:18:12.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9ae7fc640 1 -- 172.21.15.125:0/174840105 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9a80b97a0 con 0x7fa9b81538f0 2023-10-30T13:18:12.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9ae7fc640 1 -- 172.21.15.125:0/174840105 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9a80b9b50 con 0x7fa9b81538f0 2023-10-30T13:18:12.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9befe4640 1 -- 172.21.15.125:0/174840105 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9b81f24f0 con 0x7fa9b81538f0 2023-10-30T13:18:12.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9befe4640 1 -- 172.21.15.125:0/174840105 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9b81f2ab0 con 0x7fa9b81538f0 2023-10-30T13:18:12.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.657+0000 7fa9befe4640 1 -- 172.21.15.125:0/174840105 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9b804e170 con 0x7fa9b81538f0 2023-10-30T13:18:12.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.661+0000 7fa9ae7fc640 1 -- 172.21.15.125:0/174840105 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa9a80b7070 con 0x7fa9b81538f0 2023-10-30T13:18:12.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.661+0000 7fa9ae7fc640 1 --2- 172.21.15.125:0/174840105 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa99c0664d0 0x7fa99c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:12.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.661+0000 7fa9ae7fc640 1 -- 172.21.15.125:0/174840105 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7fa9a8122650 con 0x7fa9b81538f0 2023-10-30T13:18:12.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.661+0000 7fa9bdfe2640 1 --2- 172.21.15.125:0/174840105 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa99c0664d0 0x7fa99c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:12.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.661+0000 7fa9bdfe2640 1 --2- 172.21.15.125:0/174840105 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa99c0664d0 0x7fa99c068990 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400b9d0 tx=0x7fa9b4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:12.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.665+0000 7fa9ae7fc640 1 -- 172.21.15.125:0/174840105 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa9a80ef130 con 0x7fa9b81538f0 2023-10-30T13:18:12.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.845+0000 7fa9befe4640 1 -- 172.21.15.125:0/174840105 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9b80c5690 con 0x7fa9b81538f0 2023-10-30T13:18:12.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.845+0000 7fa9ae7fc640 1 -- 172.21.15.125:0/174840105 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+5758 (secure 0 0 0) 0x7fa9a80f2750 con 0x7fa9b81538f0 2023-10-30T13:18:12.851 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:18:12.852 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:18:10.286790+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:reconnect","state_seq":28,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:12.852 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 232 2023-10-30T13:18:12.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.849+0000 7fa9befe4640 1 -- 172.21.15.125:0/174840105 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa99c0664d0 msgr2=0x7fa99c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:12.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.849+0000 7fa9befe4640 1 --2- 172.21.15.125:0/174840105 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa99c0664d0 0x7fa99c068990 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fa9b400b9d0 tx=0x7fa9b4008040 comp rx=0 tx=0).stop 2023-10-30T13:18:12.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.849+0000 7fa9befe4640 1 -- 172.21.15.125:0/174840105 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9b81538f0 msgr2=0x7fa9b81eef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:12.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.849+0000 7fa9befe4640 1 --2- 172.21.15.125:0/174840105 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9b81538f0 0x7fa9b81eef80 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fa9a80b3040 tx=0x7fa9a801bad0 comp rx=0 tx=0).stop 2023-10-30T13:18:12.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.849+0000 7fa9befe4640 1 -- 172.21.15.125:0/174840105 shutdown_connections 2023-10-30T13:18:12.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.849+0000 7fa9befe4640 1 --2- 172.21.15.125:0/174840105 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa99c0664d0 0x7fa99c068990 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:12.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.849+0000 7fa9befe4640 1 --2- 172.21.15.125:0/174840105 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9b815cd50 0x7fa9b813f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:12.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.849+0000 7fa9befe4640 1 --2- 172.21.15.125:0/174840105 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9b81538f0 0x7fa9b81eef80 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:12.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.849+0000 7fa9befe4640 1 --2- 172.21.15.125:0/174840105 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9b80cefa0 0x7fa9b8144ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:12.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.849+0000 7fa9befe4640 1 -- 172.21.15.125:0/174840105 >> 172.21.15.125:0/174840105 conn(0x7fa9b806a640 msgr2=0x7fa9b80cf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:12.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.849+0000 7fa9befe4640 1 -- 172.21.15.125:0/174840105 shutdown_connections 2023-10-30T13:18:12.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:12.849+0000 7fa9befe4640 1 -- 172.21.15.125:0/174840105 wait complete. 2023-10-30T13:18:12.873 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:14.873 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:15.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe06f90f640 1 Processor -- start 2023-10-30T13:18:15.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe06f90f640 1 -- start start 2023-10-30T13:18:15.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe06f90f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0681538a0 0x7fe06815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:15.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe06f90f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe0680cf160 0x7fe0680cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:15.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe06f90f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0680cb800 con 0x7fe0680ca0e0 2023-10-30T13:18:15.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe06f90f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe06815dc30 con 0x7fe0680cf160 2023-10-30T13:18:15.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe06f90f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680cb330 con 0x7fe0681538a0 2023-10-30T13:18:15.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe06ceaa640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe0680ca0e0 0x7fe06815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43874/0 (socket says 172.21.15.125:43874) 2023-10-30T13:18:15.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe06ceaa640 1 -- 172.21.15.125:0/535058764 learned_addr learned my addr 172.21.15.125:0/535058764 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:18:15.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe06d6ab640 1 --2- 172.21.15.125:0/535058764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0681538a0 0x7fe06815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:15.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe06deac640 1 --2- 172.21.15.125:0/535058764 >> v2:172.21.15.136:3300/0 conn(0x7fe0680cf160 0x7fe0680cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:15.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 -- 172.21.15.125:0/535058764 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230114563 0 0) 0x7fe0680cb800 con 0x7fe0680ca0e0 2023-10-30T13:18:15.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 -- 172.21.15.125:0/535058764 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe054003530 con 0x7fe0680ca0e0 2023-10-30T13:18:15.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 -- 172.21.15.125:0/535058764 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3121971681 0 0) 0x7fe054003530 con 0x7fe0680ca0e0 2023-10-30T13:18:15.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 -- 172.21.15.125:0/535058764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0681538a0 msgr2=0x7fe06815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:15.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 --2- 172.21.15.125:0/535058764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0681538a0 0x7fe06815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:15.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 -- 172.21.15.125:0/535058764 >> v2:172.21.15.136:3300/0 conn(0x7fe0680cf160 msgr2=0x7fe0680cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:15.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 --2- 172.21.15.125:0/535058764 >> v2:172.21.15.136:3300/0 conn(0x7fe0680cf160 0x7fe0680cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:15.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 -- 172.21.15.125:0/535058764 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0680cad70 con 0x7fe0680ca0e0 2023-10-30T13:18:15.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 -- 172.21.15.125:0/535058764 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fe058001680 con 0x7fe0680ca0e0 2023-10-30T13:18:15.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 -- 172.21.15.125:0/535058764 >> v1:172.21.15.125:6789/0 conn(0x7fe0680ca0e0 legacy=0x7fe06815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:15.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 --2- 172.21.15.125:0/535058764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe054003980 0x7fe054003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:15.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 --2- 172.21.15.125:0/535058764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0540047d0 0x7fe054004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:15.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 --2- 172.21.15.125:0/535058764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0681538a0 0x7fe054005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:15.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 -- 172.21.15.125:0/535058764 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06815dc30 con 0x7fe054003980 2023-10-30T13:18:15.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.013+0000 7fe05ffff640 1 -- 172.21.15.125:0/535058764 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0680cb330 con 0x7fe0540047d0 2023-10-30T13:18:15.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe05ffff640 1 -- 172.21.15.125:0/535058764 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05400b990 con 0x7fe0681538a0 2023-10-30T13:18:15.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe06d6ab640 1 --2- 172.21.15.125:0/535058764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe054003980 0x7fe054003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:15.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe06ceaa640 1 --2- 172.21.15.125:0/535058764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0540047d0 0x7fe054004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:15.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe06deac640 1 --2- 172.21.15.125:0/535058764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0681538a0 0x7fe054005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:15.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe06d6ab640 1 -- 172.21.15.125:0/535058764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0681538a0 msgr2=0x7fe054005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:15.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe06d6ab640 1 --2- 172.21.15.125:0/535058764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0681538a0 0x7fe054005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:15.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe06d6ab640 1 -- 172.21.15.125:0/535058764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0540047d0 msgr2=0x7fe054004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:15.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe06d6ab640 1 --2- 172.21.15.125:0/535058764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0540047d0 0x7fe054004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:15.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe06d6ab640 1 -- 172.21.15.125:0/535058764 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe054003530 con 0x7fe054003980 2023-10-30T13:18:15.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe06ceaa640 1 --2- 172.21.15.125:0/535058764 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe0540047d0 0x7fe054004c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:15.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe06deac640 1 --2- 172.21.15.125:0/535058764 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0681538a0 0x7fe054005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:15.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe06d6ab640 1 --2- 172.21.15.125:0/535058764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe054003980 0x7fe054003da0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7fe064009ae0 tx=0x7fe06401ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96367ca9cd8d6cc4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:15.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe05ffff640 1 -- 172.21.15.125:0/535058764 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0640a5560 con 0x7fe054003980 2023-10-30T13:18:15.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe05ffff640 1 -- 172.21.15.125:0/535058764 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0640a56c0 con 0x7fe054003980 2023-10-30T13:18:15.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.017+0000 7fe05ffff640 1 -- 172.21.15.125:0/535058764 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0640a5a70 con 0x7fe054003980 2023-10-30T13:18:15.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06f90f640 1 -- 172.21.15.125:0/535058764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe054003980 msgr2=0x7fe054003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:15.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06f90f640 1 --2- 172.21.15.125:0/535058764 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe054003980 0x7fe054003da0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7fe064009ae0 tx=0x7fe06401ca50 comp rx=0 tx=0).stop 2023-10-30T13:18:15.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06deac640 1 -- 172.21.15.125:0/535058764 reap_dead start 2023-10-30T13:18:15.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06f90f640 1 -- 172.21.15.125:0/535058764 shutdown_connections 2023-10-30T13:18:15.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06f90f640 1 -- 172.21.15.125:0/535058764 >> 172.21.15.125:0/535058764 conn(0x7fe06806bfe0 msgr2=0x7fe06806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:15.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06f90f640 1 -- 172.21.15.125:0/535058764 shutdown_connections 2023-10-30T13:18:15.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06f90f640 1 -- 172.21.15.125:0/535058764 wait complete. 2023-10-30T13:18:15.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06f90f640 1 Processor -- start 2023-10-30T13:18:15.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06f90f640 1 -- start start 2023-10-30T13:18:15.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06f90f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe054003980 0x7fe0681e2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:15.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06f90f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe06815d700 0x7fe0680b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:15.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06f90f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0681f3450 0x7fe0681459d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:15.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06f90f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0681e5730 con 0x7fe054003980 2023-10-30T13:18:15.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06f90f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe068158d70 con 0x7fe06815d700 2023-10-30T13:18:15.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06f90f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0681587b0 con 0x7fe0681f3450 2023-10-30T13:18:15.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06d6ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe054003980 0x7fe0681e2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:15.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06d6ab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe054003980 0x7fe0681e2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36026/0 (socket says 172.21.15.125:36026) 2023-10-30T13:18:15.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06d6ab640 1 -- 172.21.15.125:0/4268058546 learned_addr learned my addr 172.21.15.125:0/4268058546 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:15.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06deac640 1 --2- 172.21.15.125:0/4268058546 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0681f3450 0x7fe0681459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:15.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06ceaa640 1 --2- 172.21.15.125:0/4268058546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe06815d700 0x7fe0680b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:15.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06d6ab640 1 -- 172.21.15.125:0/4268058546 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0681f3450 msgr2=0x7fe0681459d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:15.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06d6ab640 1 --2- 172.21.15.125:0/4268058546 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0681f3450 0x7fe0681459d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:15.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06d6ab640 1 -- 172.21.15.125:0/4268058546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe06815d700 msgr2=0x7fe0680b4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:15.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06d6ab640 1 --2- 172.21.15.125:0/4268058546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe06815d700 0x7fe0680b4a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:15.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.021+0000 7fe06d6ab640 1 -- 172.21.15.125:0/4268058546 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe068158a90 con 0x7fe054003980 2023-10-30T13:18:15.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.025+0000 7fe06d6ab640 1 --2- 172.21.15.125:0/4268058546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe054003980 0x7fe0681e2ae0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fe06401c880 tx=0x7fe054005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:15.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.025+0000 7fe05dffb640 1 -- 172.21.15.125:0/4268058546 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0640bc070 con 0x7fe054003980 2023-10-30T13:18:15.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.025+0000 7fe05dffb640 1 -- 172.21.15.125:0/4268058546 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06400f0c0 con 0x7fe054003980 2023-10-30T13:18:15.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.025+0000 7fe06f90f640 1 -- 172.21.15.125:0/4268058546 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0681f2970 con 0x7fe054003980 2023-10-30T13:18:15.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.025+0000 7fe05dffb640 1 -- 172.21.15.125:0/4268058546 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064004220 con 0x7fe054003980 2023-10-30T13:18:15.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.025+0000 7fe06f90f640 1 -- 172.21.15.125:0/4268058546 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0681f2f30 con 0x7fe054003980 2023-10-30T13:18:15.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.025+0000 7fe06f90f640 1 -- 172.21.15.125:0/4268058546 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe06804e170 con 0x7fe054003980 2023-10-30T13:18:15.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.025+0000 7fe05dffb640 1 -- 172.21.15.125:0/4268058546 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe0640b4040 con 0x7fe054003980 2023-10-30T13:18:15.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.025+0000 7fe05dffb640 1 --2- 172.21.15.125:0/4268058546 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe04c0664a0 0x7fe04c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:15.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.025+0000 7fe05dffb640 1 -- 172.21.15.125:0/4268058546 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7fe06411e620 con 0x7fe054003980 2023-10-30T13:18:15.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.025+0000 7fe06ceaa640 1 --2- 172.21.15.125:0/4268058546 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe04c0664a0 0x7fe04c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:15.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.029+0000 7fe06ceaa640 1 --2- 172.21.15.125:0/4268058546 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe04c0664a0 0x7fe04c068960 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe068151d60 tx=0x7fe058009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:15.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.029+0000 7fe05dffb640 1 -- 172.21.15.125:0/4268058546 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe0640eb070 con 0x7fe054003980 2023-10-30T13:18:15.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.213+0000 7fe06f90f640 1 -- 172.21.15.125:0/4268058546 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0680d0100 con 0x7fe054003980 2023-10-30T13:18:15.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.213+0000 7fe05dffb640 1 -- 172.21.15.125:0/4268058546 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 232 v232) v1 ==== 95+0+5758 (secure 0 0 0) 0x7fe064004380 con 0x7fe054003980 2023-10-30T13:18:15.219 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:18:15.219 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":232,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":232,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:18:10.286790+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:reconnect","state_seq":28,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:15.220 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 232 2023-10-30T13:18:15.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.217+0000 7fe06f90f640 1 -- 172.21.15.125:0/4268058546 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe04c0664a0 msgr2=0x7fe04c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:15.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.217+0000 7fe06f90f640 1 --2- 172.21.15.125:0/4268058546 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe04c0664a0 0x7fe04c068960 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe068151d60 tx=0x7fe058009040 comp rx=0 tx=0).stop 2023-10-30T13:18:15.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.217+0000 7fe06f90f640 1 -- 172.21.15.125:0/4268058546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe054003980 msgr2=0x7fe0681e2ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:15.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.217+0000 7fe06f90f640 1 --2- 172.21.15.125:0/4268058546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe054003980 0x7fe0681e2ae0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7fe06401c880 tx=0x7fe054005800 comp rx=0 tx=0).stop 2023-10-30T13:18:15.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.217+0000 7fe06f90f640 1 -- 172.21.15.125:0/4268058546 shutdown_connections 2023-10-30T13:18:15.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.217+0000 7fe06f90f640 1 --2- 172.21.15.125:0/4268058546 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe04c0664a0 0x7fe04c068960 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:15.222 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.217+0000 7fe06f90f640 1 --2- 172.21.15.125:0/4268058546 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe0681f3450 0x7fe0681459d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:15.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.217+0000 7fe06f90f640 1 --2- 172.21.15.125:0/4268058546 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe06815d700 0x7fe0680b4a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:15.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.217+0000 7fe06f90f640 1 --2- 172.21.15.125:0/4268058546 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe054003980 0x7fe0681e2ae0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:15.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.217+0000 7fe06f90f640 1 -- 172.21.15.125:0/4268058546 >> 172.21.15.125:0/4268058546 conn(0x7fe06806bfe0 msgr2=0x7fe068152660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:15.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.217+0000 7fe06f90f640 1 -- 172.21.15.125:0/4268058546 shutdown_connections 2023-10-30T13:18:15.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:15.217+0000 7fe06f90f640 1 -- 172.21.15.125:0/4268058546 wait complete. 2023-10-30T13:18:15.240 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:17.241 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:17.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:18:17.309 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:18:17.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.385+0000 7f8b5aa17640 1 Processor -- start 2023-10-30T13:18:17.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.385+0000 7f8b5aa17640 1 -- start start 2023-10-30T13:18:17.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.385+0000 7f8b5aa17640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f8b5415d6f0 0x7f8b5415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:17.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.385+0000 7f8b5aa17640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b540cb800 con 0x7f8b540cf160 2023-10-30T13:18:17.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.385+0000 7f8b5aa17640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b540cb330 con 0x7f8b541538a0 2023-10-30T13:18:17.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.385+0000 7f8b5aa17640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b540cad70 con 0x7f8b5415d6f0 2023-10-30T13:18:17.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.385+0000 7f8b58fb4640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8b540cf160 0x7f8b54164070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43876/0 (socket says 172.21.15.125:43876) 2023-10-30T13:18:17.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.385+0000 7f8b58fb4640 1 -- 172.21.15.125:0/2196469433 learned_addr learned my addr 172.21.15.125:0/2196469433 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:18:17.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.385+0000 7f8b537fe640 1 --2- 172.21.15.125:0/2196469433 >> v2:172.21.15.136:3300/0 conn(0x7f8b5415d6f0 0x7f8b5415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:17.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.385+0000 7f8b52ffd640 1 -- 172.21.15.125:0/2196469433 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 969335957 0 0) 0x7f8b540cb800 con 0x7f8b540cf160 2023-10-30T13:18:17.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.385+0000 7f8b52ffd640 1 -- 172.21.15.125:0/2196469433 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b3c003530 con 0x7f8b540cf160 2023-10-30T13:18:17.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.385+0000 7f8b537fe640 1 -- 172.21.15.125:0/2196469433 >> v1:172.21.15.136:6789/0 conn(0x7f8b541538a0 legacy=0x7f8b540ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:17.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.385+0000 7f8b537fe640 1 -- 172.21.15.125:0/2196469433 >> v1:172.21.15.125:6789/0 conn(0x7f8b540cf160 legacy=0x7f8b54164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:17.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.385+0000 7f8b537fe640 1 -- 172.21.15.125:0/2196469433 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b540cb050 con 0x7f8b5415d6f0 2023-10-30T13:18:17.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.389+0000 7f8b537fe640 1 --2- 172.21.15.125:0/2196469433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b5415d6f0 0x7f8b5415a090 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8b480097c0 tx=0x7f8b4801ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83e897e726f9709f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:17.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.389+0000 7f8b52ffd640 1 -- 172.21.15.125:0/2196469433 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b480a8970 con 0x7f8b5415d6f0 2023-10-30T13:18:17.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.389+0000 7f8b52ffd640 1 -- 172.21.15.125:0/2196469433 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b480a8ad0 con 0x7f8b5415d6f0 2023-10-30T13:18:17.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.389+0000 7f8b52ffd640 1 -- 172.21.15.125:0/2196469433 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b480a8e40 con 0x7f8b5415d6f0 2023-10-30T13:18:17.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.389+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2196469433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b5415d6f0 msgr2=0x7f8b5415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:17.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.389+0000 7f8b5aa17640 1 --2- 172.21.15.125:0/2196469433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b5415d6f0 0x7f8b5415a090 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f8b480097c0 tx=0x7f8b4801ec70 comp rx=0 tx=0).stop 2023-10-30T13:18:17.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.389+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2196469433 shutdown_connections 2023-10-30T13:18:17.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.389+0000 7f8b5aa17640 1 --2- 172.21.15.125:0/2196469433 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b5415d6f0 0x7f8b5415a090 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:17.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.389+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2196469433 >> 172.21.15.125:0/2196469433 conn(0x7f8b5406bfe0 msgr2=0x7f8b5406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:17.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.389+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2196469433 shutdown_connections 2023-10-30T13:18:17.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.389+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2196469433 wait complete. 2023-10-30T13:18:17.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b5aa17640 1 Processor -- start 2023-10-30T13:18:17.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b5aa17640 1 -- start start 2023-10-30T13:18:17.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b5aa17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b540cf160 0x7f8b541e5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:17.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b5aa17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b541538a0 0x7f8b54157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:17.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b5aa17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b5415d6f0 0x7f8b54157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:17.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b5aa17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b540cdeb0 con 0x7f8b541538a0 2023-10-30T13:18:17.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b5aa17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b54144600 con 0x7f8b540cf160 2023-10-30T13:18:17.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b5aa17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b540b3c30 con 0x7f8b5415d6f0 2023-10-30T13:18:17.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b537fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b541538a0 0x7f8b54157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:17.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b537fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b541538a0 0x7f8b54157520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36042/0 (socket says 172.21.15.125:36042) 2023-10-30T13:18:17.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b537fe640 1 -- 172.21.15.125:0/2747548814 learned_addr learned my addr 172.21.15.125:0/2747548814 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:17.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b53fff640 1 --2- 172.21.15.125:0/2747548814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b540cf160 0x7f8b541e5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:17.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b58fb4640 1 --2- 172.21.15.125:0/2747548814 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b5415d6f0 0x7f8b54157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:17.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b537fe640 1 -- 172.21.15.125:0/2747548814 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b5415d6f0 msgr2=0x7f8b54157a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:17.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b537fe640 1 --2- 172.21.15.125:0/2747548814 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b5415d6f0 0x7f8b54157a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:17.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b537fe640 1 -- 172.21.15.125:0/2747548814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b540cf160 msgr2=0x7f8b541e5140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:17.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b537fe640 1 --2- 172.21.15.125:0/2747548814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b540cf160 0x7f8b541e5140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:17.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b537fe640 1 -- 172.21.15.125:0/2747548814 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b48009420 con 0x7f8b541538a0 2023-10-30T13:18:17.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b537fe640 1 --2- 172.21.15.125:0/2747548814 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b541538a0 0x7f8b54157520 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f8b48000c10 tx=0x7f8b4801d4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:17.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b50ff9640 1 -- 172.21.15.125:0/2747548814 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48004450 con 0x7f8b541538a0 2023-10-30T13:18:17.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b50ff9640 1 -- 172.21.15.125:0/2747548814 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b48003620 con 0x7f8b541538a0 2023-10-30T13:18:17.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2747548814 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b540b3090 con 0x7f8b541538a0 2023-10-30T13:18:17.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b50ff9640 1 -- 172.21.15.125:0/2747548814 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48003990 con 0x7f8b541538a0 2023-10-30T13:18:17.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.393+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2747548814 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b540b4400 con 0x7f8b541538a0 2023-10-30T13:18:17.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.397+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2747548814 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b5404e170 con 0x7f8b541538a0 2023-10-30T13:18:17.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.397+0000 7f8b50ff9640 1 -- 172.21.15.125:0/2747548814 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8b48003af0 con 0x7f8b541538a0 2023-10-30T13:18:17.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.397+0000 7f8b50ff9640 1 --2- 172.21.15.125:0/2747548814 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b400664d0 0x7f8b40068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:17.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.397+0000 7f8b50ff9640 1 -- 172.21.15.125:0/2747548814 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f8b48122620 con 0x7f8b541538a0 2023-10-30T13:18:17.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.397+0000 7f8b53fff640 1 --2- 172.21.15.125:0/2747548814 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b400664d0 0x7f8b40068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:17.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.397+0000 7f8b53fff640 1 --2- 172.21.15.125:0/2747548814 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b400664d0 0x7f8b40068990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f8b44001810 tx=0x7f8b44009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:17.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.401+0000 7f8b50ff9640 1 -- 172.21.15.125:0/2747548814 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8b480ef370 con 0x7f8b541538a0 2023-10-30T13:18:17.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.581+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2747548814 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b540c4760 con 0x7f8b541538a0 2023-10-30T13:18:17.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.585+0000 7f8b50ff9640 1 -- 172.21.15.125:0/2747548814 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f8b480b5090 con 0x7f8b541538a0 2023-10-30T13:18:17.589 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:18:17.589 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":233,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:18:16.257101+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:rejoin","state_seq":30,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:17.590 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 233 2023-10-30T13:18:17.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.585+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2747548814 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b400664d0 msgr2=0x7f8b40068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:17.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.585+0000 7f8b5aa17640 1 --2- 172.21.15.125:0/2747548814 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b400664d0 0x7f8b40068990 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f8b44001810 tx=0x7f8b44009040 comp rx=0 tx=0).stop 2023-10-30T13:18:17.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.585+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2747548814 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b541538a0 msgr2=0x7f8b54157520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:17.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.585+0000 7f8b5aa17640 1 --2- 172.21.15.125:0/2747548814 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b541538a0 0x7f8b54157520 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f8b48000c10 tx=0x7f8b4801d4b0 comp rx=0 tx=0).stop 2023-10-30T13:18:17.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.585+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2747548814 shutdown_connections 2023-10-30T13:18:17.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.585+0000 7f8b5aa17640 1 --2- 172.21.15.125:0/2747548814 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8b400664d0 0x7f8b40068990 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:17.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.585+0000 7f8b5aa17640 1 --2- 172.21.15.125:0/2747548814 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8b5415d6f0 0x7f8b54157a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:17.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.585+0000 7f8b5aa17640 1 --2- 172.21.15.125:0/2747548814 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8b541538a0 0x7f8b54157520 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:17.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.585+0000 7f8b5aa17640 1 --2- 172.21.15.125:0/2747548814 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8b540cf160 0x7f8b541e5140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:17.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.585+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2747548814 >> 172.21.15.125:0/2747548814 conn(0x7f8b5406bfe0 msgr2=0x7f8b540cf590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:17.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.585+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2747548814 shutdown_connections 2023-10-30T13:18:17.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:17.585+0000 7f8b5aa17640 1 -- 172.21.15.125:0/2747548814 wait complete. 2023-10-30T13:18:17.610 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:19.611 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:19.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97c2679640 1 Processor -- start 2023-10-30T13:18:19.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97c2679640 1 -- start start 2023-10-30T13:18:19.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97c2679640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f97bc1538a0 0x7f97bc15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:19.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97c2679640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f97bc0ca0e0 0x7f97bc15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:19.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97c2679640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97bc0cb800 con 0x7f97bc0cf160 2023-10-30T13:18:19.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97c2679640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97bc15dc30 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97c2679640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97bc0cb330 con 0x7f97bc1538a0 2023-10-30T13:18:19.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97bb7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f97bc0ca0e0 0x7f97bc15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:19.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97bb7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f97bc0ca0e0 0x7f97bc15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36046/0 (socket says 172.21.15.125:36046) 2023-10-30T13:18:19.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97bb7fe640 1 -- 172.21.15.125:0/551616971 learned_addr learned my addr 172.21.15.125:0/551616971 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:19.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97bbfff640 1 --2- 172.21.15.125:0/551616971 >> v2:172.21.15.136:3300/0 conn(0x7f97bc1538a0 0x7f97bc15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:19.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97bb7fe640 1 -- 172.21.15.125:0/551616971 >> v2:172.21.15.136:3300/0 conn(0x7f97bc1538a0 msgr2=0x7f97bc15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:19.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97bb7fe640 1 --2- 172.21.15.125:0/551616971 >> v2:172.21.15.136:3300/0 conn(0x7f97bc1538a0 0x7f97bc15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:19.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97bb7fe640 1 -- 172.21.15.125:0/551616971 >> v1:172.21.15.136:6789/0 conn(0x7f97bc0cf160 legacy=0x7f97bc0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:19.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97bb7fe640 1 -- 172.21.15.125:0/551616971 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97bc0cad70 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.753+0000 7f97bbfff640 1 --2- 172.21.15.125:0/551616971 >> v2:172.21.15.136:3300/0 conn(0x7f97bc1538a0 0x7f97bc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:19.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.757+0000 7f97bb7fe640 1 --2- 172.21.15.125:0/551616971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0ca0e0 0x7f97bc15a090 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0cfd10 tx=0x7f97b001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84bd9de5e69e688a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:19.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.757+0000 7f97baffd640 1 -- 172.21.15.125:0/551616971 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b00a89f0 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.757+0000 7f97baffd640 1 -- 172.21.15.125:0/551616971 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97b00a8b50 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.757+0000 7f97baffd640 1 -- 172.21.15.125:0/551616971 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b00b0440 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.757+0000 7f97c2679640 1 -- 172.21.15.125:0/551616971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0ca0e0 msgr2=0x7f97bc15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:19.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.757+0000 7f97c2679640 1 --2- 172.21.15.125:0/551616971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0ca0e0 0x7f97bc15a090 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f97bc0cfd10 tx=0x7f97b001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:18:19.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.757+0000 7f97c2679640 1 -- 172.21.15.125:0/551616971 shutdown_connections 2023-10-30T13:18:19.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.757+0000 7f97c2679640 1 --2- 172.21.15.125:0/551616971 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0ca0e0 0x7f97bc15a090 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:19.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.757+0000 7f97c2679640 1 --2- 172.21.15.125:0/551616971 >> v2:172.21.15.136:3300/0 conn(0x7f97bc1538a0 0x7f97bc15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:19.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.757+0000 7f97c2679640 1 -- 172.21.15.125:0/551616971 >> 172.21.15.125:0/551616971 conn(0x7f97bc06bfe0 msgr2=0x7f97bc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:19.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.757+0000 7f97c2679640 1 -- 172.21.15.125:0/551616971 shutdown_connections 2023-10-30T13:18:19.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.757+0000 7f97c2679640 1 -- 172.21.15.125:0/551616971 wait complete. 2023-10-30T13:18:19.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97c2679640 1 Processor -- start 2023-10-30T13:18:19.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97c2679640 1 -- start start 2023-10-30T13:18:19.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97c2679640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0ca0e0 0x7f97bc1ef570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:19.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97c2679640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97bc0cf160 0x7f97bc0cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:19.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97c2679640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f97bc1538a0 0x7f97bc1585a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:19.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97c2679640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc0b3110 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97c2679640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc1efab0 con 0x7f97bc0cf160 2023-10-30T13:18:19.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97c2679640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97bc0c4420 con 0x7f97bc1538a0 2023-10-30T13:18:19.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97bbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0ca0e0 0x7f97bc1ef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:19.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97bbfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0ca0e0 0x7f97bc1ef570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36056/0 (socket says 172.21.15.125:36056) 2023-10-30T13:18:19.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97bbfff640 1 -- 172.21.15.125:0/1742610071 learned_addr learned my addr 172.21.15.125:0/1742610071 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:19.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97bb7fe640 1 --2- 172.21.15.125:0/1742610071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97bc0cf160 0x7f97bc0cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:19.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97c0c16640 1 --2- 172.21.15.125:0/1742610071 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f97bc1538a0 0x7f97bc1585a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:19.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97bbfff640 1 -- 172.21.15.125:0/1742610071 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f97bc1538a0 msgr2=0x7f97bc1585a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:19.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97bbfff640 1 --2- 172.21.15.125:0/1742610071 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f97bc1538a0 0x7f97bc1585a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:19.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97bbfff640 1 -- 172.21.15.125:0/1742610071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97bc0cf160 msgr2=0x7f97bc0cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:19.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97bbfff640 1 --2- 172.21.15.125:0/1742610071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97bc0cf160 0x7f97bc0cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:19.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97bbfff640 1 -- 172.21.15.125:0/1742610071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b00094d0 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97bbfff640 1 --2- 172.21.15.125:0/1742610071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0ca0e0 0x7f97bc1ef570 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00d800 tx=0x7f97ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:19.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97b8ff9640 1 -- 172.21.15.125:0/1742610071 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac0096c0 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97b8ff9640 1 -- 172.21.15.125:0/1742610071 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ac009820 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97b8ff9640 1 -- 172.21.15.125:0/1742610071 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac009bd0 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97c2679640 1 -- 172.21.15.125:0/1742610071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97bc1efd20 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97c2679640 1 -- 172.21.15.125:0/1742610071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97bc1f02c0 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.761+0000 7f97c2679640 1 -- 172.21.15.125:0/1742610071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97bc0b2a50 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.765+0000 7f97b8ff9640 1 -- 172.21.15.125:0/1742610071 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f97ac009d30 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.765+0000 7f97b8ff9640 1 --2- 172.21.15.125:0/1742610071 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f97a80664d0 0x7f97a8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:19.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.765+0000 7f97b8ff9640 1 -- 172.21.15.125:0/1742610071 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f97ac084ff0 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.765+0000 7f97bb7fe640 1 --2- 172.21.15.125:0/1742610071 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f97a80664d0 0x7f97a8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:19.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.765+0000 7f97bb7fe640 1 --2- 172.21.15.125:0/1742610071 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f97a80664d0 0x7f97a8068990 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f97b00b2000 tx=0x7f97b0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:19.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.769+0000 7f97b8ff9640 1 -- 172.21.15.125:0/1742610071 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f97ac051a40 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.949+0000 7f97c2679640 1 -- 172.21.15.125:0/1742610071 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f97bc159860 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.949+0000 7f97b8ff9640 1 -- 172.21.15.125:0/1742610071 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f97ac055060 con 0x7f97bc0ca0e0 2023-10-30T13:18:19.956 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:18:19.956 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":233,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:18:16.257101+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:rejoin","state_seq":30,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:19.957 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 233 2023-10-30T13:18:19.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.953+0000 7f97c2679640 1 -- 172.21.15.125:0/1742610071 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f97a80664d0 msgr2=0x7f97a8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:19.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.953+0000 7f97c2679640 1 --2- 172.21.15.125:0/1742610071 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f97a80664d0 0x7f97a8068990 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f97b00b2000 tx=0x7f97b0008040 comp rx=0 tx=0).stop 2023-10-30T13:18:19.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.953+0000 7f97c2679640 1 -- 172.21.15.125:0/1742610071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0ca0e0 msgr2=0x7f97bc1ef570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:19.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.953+0000 7f97c2679640 1 --2- 172.21.15.125:0/1742610071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0ca0e0 0x7f97bc1ef570 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00d800 tx=0x7f97ac00dcc0 comp rx=0 tx=0).stop 2023-10-30T13:18:19.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.953+0000 7f97c2679640 1 -- 172.21.15.125:0/1742610071 shutdown_connections 2023-10-30T13:18:19.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.953+0000 7f97c2679640 1 --2- 172.21.15.125:0/1742610071 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f97a80664d0 0x7f97a8068990 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:19.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.953+0000 7f97c2679640 1 --2- 172.21.15.125:0/1742610071 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f97bc1538a0 0x7f97bc1585a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:19.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.953+0000 7f97c2679640 1 --2- 172.21.15.125:0/1742610071 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f97bc0cf160 0x7f97bc0cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:19.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.953+0000 7f97c2679640 1 --2- 172.21.15.125:0/1742610071 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f97bc0ca0e0 0x7f97bc1ef570 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:19.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.953+0000 7f97c2679640 1 -- 172.21.15.125:0/1742610071 >> 172.21.15.125:0/1742610071 conn(0x7f97bc06bfe0 msgr2=0x7f97bc148dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:19.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.953+0000 7f97c2679640 1 -- 172.21.15.125:0/1742610071 shutdown_connections 2023-10-30T13:18:19.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:19.953+0000 7f97c2679640 1 -- 172.21.15.125:0/1742610071 wait complete. 2023-10-30T13:18:19.978 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 233, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 233, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_8833": { "addr": "172.21.15.136:6835/3864637420", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3864637420, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3864637420, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8833, "incarnation": 222, "join_fscid": -1, "name": "d", "rank": 2, "state": "up:active", "state_seq": 15 }, "gid_8971": { "addr": "172.21.15.136:6839/4284345684", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 4284345684, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 4284345684, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8971, "incarnation": 230, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:rejoin", "state_seq": 30 }, "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 36 } }, "last_failure": 0, "last_failure_osd_epoch": 82, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:18:16.257101+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8971, "mds_1": 9150, "mds_2": 8833 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:18:21.981 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:22.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.125+0000 7f2544e67640 1 Processor -- start 2023-10-30T13:18:22.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.125+0000 7f2544e67640 1 -- start start 2023-10-30T13:18:22.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.129+0000 7f2544e67640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25401538f0 0x7f2540159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.129+0000 7f2544e67640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f25400cefa0 0x7f25400cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.129+0000 7f2544e67640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25400ca6a0 con 0x7f254015cd50 2023-10-30T13:18:22.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.129+0000 7f2544e67640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254015d300 con 0x7f25400cefa0 2023-10-30T13:18:22.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.129+0000 7f2544e67640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254015d5e0 con 0x7f25401538f0 2023-10-30T13:18:22.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.129+0000 7f253ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f25400cefa0 0x7f25400cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.129+0000 7f253ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f25400cefa0 0x7f25400cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59788/0 (socket says 172.21.15.125:59788) 2023-10-30T13:18:22.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.129+0000 7f253ffff640 1 -- 172.21.15.125:0/2954825874 learned_addr learned my addr 172.21.15.125:0/2954825874 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:22.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.129+0000 7f253f7fe640 1 --2- 172.21.15.125:0/2954825874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25401538f0 0x7f2540159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.129+0000 7f253ffff640 1 -- 172.21.15.125:0/2954825874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25401538f0 msgr2=0x7f2540159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:22.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.129+0000 7f253ffff640 1 --2- 172.21.15.125:0/2954825874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25401538f0 0x7f2540159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.129+0000 7f253ffff640 1 -- 172.21.15.125:0/2954825874 >> v1:172.21.15.125:6789/0 conn(0x7f254015cd50 legacy=0x7f25400c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.129+0000 7f253ffff640 1 -- 172.21.15.125:0/2954825874 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254015d8c0 con 0x7f25400cefa0 2023-10-30T13:18:22.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.129+0000 7f253ffff640 1 --2- 172.21.15.125:0/2954825874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25400cefa0 0x7f25400cf3f0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f2538009e50 tx=0x7f253801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b431609f5ad4392 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:22.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f253e7fc640 1 -- 172.21.15.125:0/2954825874 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25380af400 con 0x7f25400cefa0 2023-10-30T13:18:22.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f253e7fc640 1 -- 172.21.15.125:0/2954825874 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25380ab050 con 0x7f25400cefa0 2023-10-30T13:18:22.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f253e7fc640 1 -- 172.21.15.125:0/2954825874 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25380af780 con 0x7f25400cefa0 2023-10-30T13:18:22.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 -- 172.21.15.125:0/2954825874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25400cefa0 msgr2=0x7f25400cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 --2- 172.21.15.125:0/2954825874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25400cefa0 0x7f25400cf3f0 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f2538009e50 tx=0x7f253801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 -- 172.21.15.125:0/2954825874 shutdown_connections 2023-10-30T13:18:22.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 --2- 172.21.15.125:0/2954825874 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25400cefa0 0x7f25400cf3f0 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 --2- 172.21.15.125:0/2954825874 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25401538f0 0x7f2540159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 -- 172.21.15.125:0/2954825874 >> 172.21.15.125:0/2954825874 conn(0x7f254006a640 msgr2=0x7f2540065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:22.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 -- 172.21.15.125:0/2954825874 shutdown_connections 2023-10-30T13:18:22.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 -- 172.21.15.125:0/2954825874 wait complete. 2023-10-30T13:18:22.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 Processor -- start 2023-10-30T13:18:22.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 -- start start 2023-10-30T13:18:22.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25400cefa0 0x7f2540158300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25401538f0 0x7f2540142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f254015cd50 0x7f2540147f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254015bcf0 con 0x7f25400cefa0 2023-10-30T13:18:22.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25401efb30 con 0x7f254015cd50 2023-10-30T13:18:22.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.133+0000 7f2544e67640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2540152210 con 0x7f25401538f0 2023-10-30T13:18:22.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f253f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25400cefa0 0x7f2540158300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f253f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25400cefa0 0x7f2540158300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59802/0 (socket says 172.21.15.125:59802) 2023-10-30T13:18:22.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f253f7fe640 1 -- 172.21.15.125:0/4152094905 learned_addr learned my addr 172.21.15.125:0/4152094905 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:22.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f253f7fe640 1 -- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25401538f0 msgr2=0x7f2540142390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f253effd640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25401538f0 0x7f2540142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f253ffff640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f254015cd50 0x7f2540147f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f253f7fe640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25401538f0 0x7f2540142390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f253f7fe640 1 -- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f254015cd50 msgr2=0x7f2540147f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f253f7fe640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f254015cd50 0x7f2540147f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f253f7fe640 1 -- 172.21.15.125:0/4152094905 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2538009b30 con 0x7f25400cefa0 2023-10-30T13:18:22.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f253effd640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25401538f0 0x7f2540142390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:18:22.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f253ffff640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f254015cd50 0x7f2540147f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:18:22.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f253f7fe640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25400cefa0 0x7f2540158300 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f253000d800 tx=0x7f253000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:22.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f2527fff640 1 -- 172.21.15.125:0/4152094905 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253000b6c0 con 0x7f25400cefa0 2023-10-30T13:18:22.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f2527fff640 1 -- 172.21.15.125:0/4152094905 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253000b820 con 0x7f25400cefa0 2023-10-30T13:18:22.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f2527fff640 1 -- 172.21.15.125:0/4152094905 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253000bb90 con 0x7f25400cefa0 2023-10-30T13:18:22.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f2544e67640 1 -- 172.21.15.125:0/4152094905 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25401524f0 con 0x7f25400cefa0 2023-10-30T13:18:22.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f2544e67640 1 -- 172.21.15.125:0/4152094905 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25401ef7c0 con 0x7f25400cefa0 2023-10-30T13:18:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f2544e67640 1 -- 172.21.15.125:0/4152094905 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25401ef980 con 0x7f25400cefa0 2023-10-30T13:18:22.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.137+0000 7f2527fff640 1 -- 172.21.15.125:0/4152094905 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f253000bcf0 con 0x7f25400cefa0 2023-10-30T13:18:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.141+0000 7f2527fff640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f252c0664d0 0x7f252c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.141+0000 7f2527fff640 1 -- 172.21.15.125:0/4152094905 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f2530084ca0 con 0x7f25400cefa0 2023-10-30T13:18:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.141+0000 7f253effd640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f252c0664d0 0x7f252c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:22.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.141+0000 7f253effd640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f252c0664d0 0x7f252c068990 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f254014d690 tx=0x7f2534009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:22.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.145+0000 7f2527fff640 1 -- 172.21.15.125:0/4152094905 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2530051820 con 0x7f25400cefa0 2023-10-30T13:18:22.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.325+0000 7f2544e67640 1 -- 172.21.15.125:0/4152094905 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f254004e170 con 0x7f25400cefa0 2023-10-30T13:18:22.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.329+0000 7f2527fff640 1 -- 172.21.15.125:0/4152094905 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 233 v233) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f2530054e40 con 0x7f25400cefa0 2023-10-30T13:18:22.333 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:18:22.333 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":233,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":233,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:18:16.257101+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:rejoin","state_seq":30,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:22.333 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 233 2023-10-30T13:18:22.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.329+0000 7f2544e67640 1 -- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f252c0664d0 msgr2=0x7f252c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.329+0000 7f2544e67640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f252c0664d0 0x7f252c068990 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f254014d690 tx=0x7f2534009040 comp rx=0 tx=0).stop 2023-10-30T13:18:22.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.329+0000 7f2544e67640 1 -- 172.21.15.125:0/4152094905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25400cefa0 msgr2=0x7f2540158300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:22.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.329+0000 7f2544e67640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25400cefa0 0x7f2540158300 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f253000d800 tx=0x7f253000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.329+0000 7f2544e67640 1 -- 172.21.15.125:0/4152094905 shutdown_connections 2023-10-30T13:18:22.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.329+0000 7f2544e67640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f252c0664d0 0x7f252c068990 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.329+0000 7f2544e67640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f254015cd50 0x7f2540147f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.329+0000 7f2544e67640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f25401538f0 0x7f2540142390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.329+0000 7f2544e67640 1 --2- 172.21.15.125:0/4152094905 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f25400cefa0 0x7f2540158300 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:22.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.329+0000 7f2544e67640 1 -- 172.21.15.125:0/4152094905 >> 172.21.15.125:0/4152094905 conn(0x7f254006a640 msgr2=0x7f25400cf380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:22.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.329+0000 7f2544e67640 1 -- 172.21.15.125:0/4152094905 shutdown_connections 2023-10-30T13:18:22.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:22.329+0000 7f2544e67640 1 -- 172.21.15.125:0/4152094905 wait complete. 2023-10-30T13:18:22.354 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:18:24.355 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:24.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.496+0000 7f42f31a0640 1 Processor -- start 2023-10-30T13:18:24.502 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.496+0000 7f42f31a0640 1 -- start start 2023-10-30T13:18:24.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.496+0000 7f42f31a0640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f42ec0c6290 0x7f42ec15ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:24.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.496+0000 7f42f31a0640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42ec15b060 con 0x7f42ec071b90 2023-10-30T13:18:24.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.496+0000 7f42f31a0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42ec0c4ab0 con 0x7f42ec0717b0 2023-10-30T13:18:24.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.496+0000 7f42f31a0640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42ec157530 con 0x7f42ec0c6290 2023-10-30T13:18:24.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.496+0000 7f42f199d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f42ec071b90 0x7f42ec15fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35050/0 (socket says 172.21.15.125:35050) 2023-10-30T13:18:24.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.496+0000 7f42f199d640 1 -- 172.21.15.125:0/3949562354 learned_addr learned my addr 172.21.15.125:0/3949562354 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:18:24.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.496+0000 7f42f299f640 1 --2- 172.21.15.125:0/3949562354 >> v2:172.21.15.136:3300/0 conn(0x7f42ec0c6290 0x7f42ec15ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:24.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.496+0000 7f42f119c640 1 -- 172.21.15.125:0/3949562354 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764914594 0 0) 0x7f42ec15b060 con 0x7f42ec071b90 2023-10-30T13:18:24.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.496+0000 7f42f119c640 1 -- 172.21.15.125:0/3949562354 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42d8003530 con 0x7f42ec071b90 2023-10-30T13:18:24.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f119c640 1 -- 172.21.15.125:0/3949562354 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3389631056 0 0) 0x7f42d8003530 con 0x7f42ec071b90 2023-10-30T13:18:24.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f119c640 1 -- 172.21.15.125:0/3949562354 >> v2:172.21.15.136:3300/0 conn(0x7f42ec0c6290 msgr2=0x7f42ec15ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:24.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f119c640 1 --2- 172.21.15.125:0/3949562354 >> v2:172.21.15.136:3300/0 conn(0x7f42ec0c6290 0x7f42ec15ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:24.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f119c640 1 -- 172.21.15.125:0/3949562354 >> v1:172.21.15.136:6789/0 conn(0x7f42ec0717b0 legacy=0x7f42ec1513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:24.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f119c640 1 -- 172.21.15.125:0/3949562354 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec0c4d90 con 0x7f42ec071b90 2023-10-30T13:18:24.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f119c640 1 -- 172.21.15.125:0/3949562354 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f42dc001680 con 0x7f42ec071b90 2023-10-30T13:18:24.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f119c640 1 -- 172.21.15.125:0/3949562354 >> v1:172.21.15.125:6789/0 conn(0x7f42ec071b90 legacy=0x7f42ec15fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:24.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f119c640 1 --2- 172.21.15.125:0/3949562354 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d8003980 0x7f42d8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:24.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f119c640 1 --2- 172.21.15.125:0/3949562354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d80047d0 0x7f42d8004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:24.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f119c640 1 --2- 172.21.15.125:0/3949562354 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d8005190 0x7f42d800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:24.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f119c640 1 -- 172.21.15.125:0/3949562354 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d800c010 con 0x7f42d8003980 2023-10-30T13:18:24.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f119c640 1 -- 172.21.15.125:0/3949562354 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d800c180 con 0x7f42d80047d0 2023-10-30T13:18:24.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f119c640 1 -- 172.21.15.125:0/3949562354 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d800c340 con 0x7f42d8005190 2023-10-30T13:18:24.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f299f640 1 --2- 172.21.15.125:0/3949562354 >> v2:172.21.15.136:3300/0 conn(0x7f42ec0c6290 0x7f42ec15ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:18:24.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f219e640 1 --2- 172.21.15.125:0/3949562354 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d8003980 0x7f42d8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:24.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f199d640 1 --2- 172.21.15.125:0/3949562354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d80047d0 0x7f42d8004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:24.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f299f640 1 --2- 172.21.15.125:0/3949562354 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d8005190 0x7f42d800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:24.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f219e640 1 -- 172.21.15.125:0/3949562354 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d8005190 msgr2=0x7f42d800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:24.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f219e640 1 --2- 172.21.15.125:0/3949562354 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42d8005190 0x7f42d800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:24.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f219e640 1 -- 172.21.15.125:0/3949562354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d80047d0 msgr2=0x7f42d8004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:24.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f219e640 1 --2- 172.21.15.125:0/3949562354 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d80047d0 0x7f42d8004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:24.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f219e640 1 -- 172.21.15.125:0/3949562354 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d8003530 con 0x7f42d8003980 2023-10-30T13:18:24.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.500+0000 7f42f299f640 1 -- 172.21.15.125:0/3949562354 reap_dead start 2023-10-30T13:18:24.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f219e640 1 --2- 172.21.15.125:0/3949562354 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d8003980 0x7f42d8003da0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f42e80017b0 tx=0x7f42e801c3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d38c36e1e6a0ecd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:24.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f119c640 1 -- 172.21.15.125:0/3949562354 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e80052d0 con 0x7f42d8003980 2023-10-30T13:18:24.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f119c640 1 -- 172.21.15.125:0/3949562354 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e80a7050 con 0x7f42d8003980 2023-10-30T13:18:24.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f119c640 1 -- 172.21.15.125:0/3949562354 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e8003650 con 0x7f42d8003980 2023-10-30T13:18:24.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 -- 172.21.15.125:0/3949562354 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d8003980 msgr2=0x7f42d8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:24.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 --2- 172.21.15.125:0/3949562354 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d8003980 0x7f42d8003da0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f42e80017b0 tx=0x7f42e801c3c0 comp rx=0 tx=0).stop 2023-10-30T13:18:24.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 -- 172.21.15.125:0/3949562354 shutdown_connections 2023-10-30T13:18:24.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 --2- 172.21.15.125:0/3949562354 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42d8003980 0x7f42d8003da0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:24.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 -- 172.21.15.125:0/3949562354 >> 172.21.15.125:0/3949562354 conn(0x7f42ec0c75d0 msgr2=0x7f42ec0c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:24.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 -- 172.21.15.125:0/3949562354 shutdown_connections 2023-10-30T13:18:24.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 -- 172.21.15.125:0/3949562354 wait complete. 2023-10-30T13:18:24.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 Processor -- start 2023-10-30T13:18:24.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 -- start start 2023-10-30T13:18:24.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d8003980 0x7f42ec1513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:24.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42ec1f2480 0x7f42ec06f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:24.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42ec1f28c0 0x7f42ec13d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:24.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec0ca340 con 0x7f42ec1f2480 2023-10-30T13:18:24.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec15b5d0 con 0x7f42d8003980 2023-10-30T13:18:24.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.504+0000 7f42f31a0640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec15b890 con 0x7f42ec1f28c0 2023-10-30T13:18:24.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42f199d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42ec1f2480 0x7f42ec06f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:24.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42f199d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42ec1f2480 0x7f42ec06f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59822/0 (socket says 172.21.15.125:59822) 2023-10-30T13:18:24.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42f199d640 1 -- 172.21.15.125:0/1480612773 learned_addr learned my addr 172.21.15.125:0/1480612773 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:24.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42f219e640 1 --2- 172.21.15.125:0/1480612773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d8003980 0x7f42ec1513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:24.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42f299f640 1 --2- 172.21.15.125:0/1480612773 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42ec1f28c0 0x7f42ec13d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:24.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42f199d640 1 -- 172.21.15.125:0/1480612773 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42ec1f28c0 msgr2=0x7f42ec13d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:24.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42f199d640 1 --2- 172.21.15.125:0/1480612773 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42ec1f28c0 0x7f42ec13d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:24.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42f199d640 1 -- 172.21.15.125:0/1480612773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d8003980 msgr2=0x7f42ec1513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:24.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42f199d640 1 --2- 172.21.15.125:0/1480612773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d8003980 0x7f42ec1513f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:24.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42f199d640 1 -- 172.21.15.125:0/1480612773 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec0704b0 con 0x7f42ec1f2480 2023-10-30T13:18:24.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42f199d640 1 --2- 172.21.15.125:0/1480612773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42ec1f2480 0x7f42ec06f340 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f42dc004880 tx=0x7f42dc005870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:24.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42e2ffd640 1 -- 172.21.15.125:0/1480612773 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc014810 con 0x7f42ec1f2480 2023-10-30T13:18:24.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42e2ffd640 1 -- 172.21.15.125:0/1480612773 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42dc014990 con 0x7f42ec1f2480 2023-10-30T13:18:24.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42e2ffd640 1 -- 172.21.15.125:0/1480612773 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc014d40 con 0x7f42ec1f2480 2023-10-30T13:18:24.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42f31a0640 1 -- 172.21.15.125:0/1480612773 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42ec070790 con 0x7f42ec1f2480 2023-10-30T13:18:24.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42f31a0640 1 -- 172.21.15.125:0/1480612773 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ec070d50 con 0x7f42ec1f2480 2023-10-30T13:18:24.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42f31a0640 1 -- 172.21.15.125:0/1480612773 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ec04e170 con 0x7f42ec1f2480 2023-10-30T13:18:24.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.508+0000 7f42e2ffd640 1 -- 172.21.15.125:0/1480612773 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f42dc0153e0 con 0x7f42ec1f2480 2023-10-30T13:18:24.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.512+0000 7f42e2ffd640 1 --2- 172.21.15.125:0/1480612773 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42d00664a0 0x7f42d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:24.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.512+0000 7f42e2ffd640 1 -- 172.21.15.125:0/1480612773 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f42dc0872a0 con 0x7f42ec1f2480 2023-10-30T13:18:24.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.512+0000 7f42f219e640 1 --2- 172.21.15.125:0/1480612773 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42d00664a0 0x7f42d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:24.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.512+0000 7f42f219e640 1 --2- 172.21.15.125:0/1480612773 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42d00664a0 0x7f42d0068960 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f42e80ac000 tx=0x7f42e80011f0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:24.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.516+0000 7f42e2ffd640 1 -- 172.21.15.125:0/1480612773 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f42dc053d70 con 0x7f42ec1f2480 2023-10-30T13:18:24.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.696+0000 7f42f31a0640 1 -- 172.21.15.125:0/1480612773 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f42ec06d280 con 0x7f42ec1f2480 2023-10-30T13:18:24.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.696+0000 7f42e2ffd640 1 -- 172.21.15.125:0/1480612773 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f42dc01c070 con 0x7f42ec1f2480 2023-10-30T13:18:24.702 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:18:24.702 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":234,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:18:22.845721+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:24.702 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 234 2023-10-30T13:18:24.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.696+0000 7f42f31a0640 1 -- 172.21.15.125:0/1480612773 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42d00664a0 msgr2=0x7f42d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:24.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.696+0000 7f42f31a0640 1 --2- 172.21.15.125:0/1480612773 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42d00664a0 0x7f42d0068960 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f42e80ac000 tx=0x7f42e80011f0 comp rx=0 tx=0).stop 2023-10-30T13:18:24.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.696+0000 7f42f31a0640 1 -- 172.21.15.125:0/1480612773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42ec1f2480 msgr2=0x7f42ec06f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:24.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.696+0000 7f42f31a0640 1 --2- 172.21.15.125:0/1480612773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42ec1f2480 0x7f42ec06f340 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f42dc004880 tx=0x7f42dc005870 comp rx=0 tx=0).stop 2023-10-30T13:18:24.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.696+0000 7f42f31a0640 1 -- 172.21.15.125:0/1480612773 shutdown_connections 2023-10-30T13:18:24.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.696+0000 7f42f31a0640 1 --2- 172.21.15.125:0/1480612773 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f42d00664a0 0x7f42d0068960 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:24.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.696+0000 7f42f31a0640 1 --2- 172.21.15.125:0/1480612773 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f42ec1f28c0 0x7f42ec13d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:24.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.696+0000 7f42f31a0640 1 --2- 172.21.15.125:0/1480612773 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f42ec1f2480 0x7f42ec06f340 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:24.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.696+0000 7f42f31a0640 1 --2- 172.21.15.125:0/1480612773 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f42d8003980 0x7f42ec1513f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:24.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.696+0000 7f42f31a0640 1 -- 172.21.15.125:0/1480612773 >> 172.21.15.125:0/1480612773 conn(0x7f42ec0c75d0 msgr2=0x7f42ec0c22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:24.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.696+0000 7f42f31a0640 1 -- 172.21.15.125:0/1480612773 shutdown_connections 2023-10-30T13:18:24.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:24.696+0000 7f42f31a0640 1 -- 172.21.15.125:0/1480612773 wait complete. 2023-10-30T13:18:24.722 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=8529 2023-10-30T13:18:24.723 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.e 2023-10-30T13:18:29.151 INFO:tasks.ceph.mds.f.smithi136.stderr:2023-10-30T13:18:29.148+0000 7feca1c5a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:18:29.151 INFO:tasks.ceph.mds.f.smithi136.stderr:2023-10-30T13:18:29.148+0000 7feca1c5a640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:18:29.152 INFO:tasks.ceph.mds.f.smithi136.stderr:2023-10-30T13:18:29.148+0000 7feca1c5a640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:18:33.723 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.e 2023-10-30T13:18:33.724 INFO:tasks.ceph.mds.e:Restarting daemon 2023-10-30T13:18:33.724 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2023-10-30T13:18:33.729 INFO:tasks.ceph.mds.e:Started 2023-10-30T13:18:33.729 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:33.825 INFO:tasks.ceph.mds.e.smithi125.stdout:starting mds.e at 2023-10-30T13:18:33.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.872+0000 7fcc30ec9640 1 Processor -- start 2023-10-30T13:18:33.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.872+0000 7fcc30ec9640 1 -- start start 2023-10-30T13:18:33.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.876+0000 7fcc30ec9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcc2c06f7a0 0x7fcc2c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:33.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.876+0000 7fcc30ec9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc2c0c6bd0 0x7fcc2c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:33.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.876+0000 7fcc30ec9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2c06a3d0 con 0x7fcc2c071110 2023-10-30T13:18:33.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.876+0000 7fcc30ec9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc2c0700c0 con 0x7fcc2c06f7a0 2023-10-30T13:18:33.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.876+0000 7fcc30ec9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c066cf0 con 0x7fcc2c0c6bd0 2023-10-30T13:18:33.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.876+0000 7fcc2a59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcc2c06f7a0 0x7fcc2c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:33.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.876+0000 7fcc2ad9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fcc2c071110 0x7fcc2c158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47410/0 (socket says 172.21.15.125:47410) 2023-10-30T13:18:33.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.876+0000 7fcc2ad9d640 1 -- 172.21.15.125:0/614753729 learned_addr learned my addr 172.21.15.125:0/614753729 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:18:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.876+0000 7fcc21d9b640 1 --2- 172.21.15.125:0/614753729 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc2c0c6bd0 0x7fcc2c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.876+0000 7fcc2a59c640 1 -- 172.21.15.125:0/614753729 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc2c0c6bd0 msgr2=0x7fcc2c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.876+0000 7fcc2a59c640 1 --2- 172.21.15.125:0/614753729 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc2c0c6bd0 0x7fcc2c069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.876+0000 7fcc2a59c640 1 -- 172.21.15.125:0/614753729 >> v1:172.21.15.125:6789/0 conn(0x7fcc2c071110 legacy=0x7fcc2c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:33.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.876+0000 7fcc2a59c640 1 -- 172.21.15.125:0/614753729 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c0c70d0 con 0x7fcc2c06f7a0 2023-10-30T13:18:33.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.876+0000 7fcc29d9b640 1 -- 172.21.15.125:0/614753729 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091533981 0 0) 0x7fcc2c06a3d0 con 0x7fcc2c071110 2023-10-30T13:18:33.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.880+0000 7fcc2a59c640 1 --2- 172.21.15.125:0/614753729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc2c06f7a0 0x7fcc2c06fb80 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0097c0 tx=0x7fcc1c01ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cfaa793243757b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:33.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.880+0000 7fcc29d9b640 1 -- 172.21.15.125:0/614753729 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c0a8970 con 0x7fcc2c06f7a0 2023-10-30T13:18:33.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.880+0000 7fcc29d9b640 1 -- 172.21.15.125:0/614753729 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc1c0a8ad0 con 0x7fcc2c06f7a0 2023-10-30T13:18:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.880+0000 7fcc29d9b640 1 -- 172.21.15.125:0/614753729 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc1c0a8e60 con 0x7fcc2c06f7a0 2023-10-30T13:18:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.880+0000 7fcc30ec9640 1 -- 172.21.15.125:0/614753729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc2c06f7a0 msgr2=0x7fcc2c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.880+0000 7fcc30ec9640 1 --2- 172.21.15.125:0/614753729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc2c06f7a0 0x7fcc2c06fb80 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0097c0 tx=0x7fcc1c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:18:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.880+0000 7fcc30ec9640 1 -- 172.21.15.125:0/614753729 shutdown_connections 2023-10-30T13:18:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.880+0000 7fcc30ec9640 1 --2- 172.21.15.125:0/614753729 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc2c0c6bd0 0x7fcc2c069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:33.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.880+0000 7fcc30ec9640 1 --2- 172.21.15.125:0/614753729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc2c06f7a0 0x7fcc2c06fb80 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.880+0000 7fcc30ec9640 1 -- 172.21.15.125:0/614753729 >> 172.21.15.125:0/614753729 conn(0x7fcc2c0c7910 msgr2=0x7fcc2c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.880+0000 7fcc30ec9640 1 -- 172.21.15.125:0/614753729 shutdown_connections 2023-10-30T13:18:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.880+0000 7fcc30ec9640 1 -- 172.21.15.125:0/614753729 wait complete. 2023-10-30T13:18:33.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.880+0000 7fcc30ec9640 1 Processor -- start 2023-10-30T13:18:33.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.880+0000 7fcc30ec9640 1 -- start start 2023-10-30T13:18:33.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc30ec9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc2c06f7a0 0x7fcc2c1588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:33.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc30ec9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc2c071110 0x7fcc2c1f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:33.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc30ec9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc2c0c6bd0 0x7fcc2c06b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:33.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc30ec9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c0aed30 con 0x7fcc2c0c6bd0 2023-10-30T13:18:33.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc30ec9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c0af220 con 0x7fcc2c071110 2023-10-30T13:18:33.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc30ec9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c1f1de0 con 0x7fcc2c06f7a0 2023-10-30T13:18:33.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc2ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc2c0c6bd0 0x7fcc2c06b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:33.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc2ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc2c0c6bd0 0x7fcc2c06b810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59924/0 (socket says 172.21.15.125:59924) 2023-10-30T13:18:33.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc2ad9d640 1 -- 172.21.15.125:0/3637013807 learned_addr learned my addr 172.21.15.125:0/3637013807 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:33.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc2a59c640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc2c06f7a0 0x7fcc2c1588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:33.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc21d9b640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc2c071110 0x7fcc2c1f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:33.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc2ad9d640 1 -- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc2c06f7a0 msgr2=0x7fcc2c1588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:33.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc2ad9d640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc2c06f7a0 0x7fcc2c1588a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:33.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc2ad9d640 1 -- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc2c071110 msgr2=0x7fcc2c1f25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:33.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc2ad9d640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc2c071110 0x7fcc2c1f25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:33.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc2ad9d640 1 -- 172.21.15.125:0/3637013807 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc1c009420 con 0x7fcc2c0c6bd0 2023-10-30T13:18:33.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc21d9b640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc2c071110 0x7fcc2c1f25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:18:33.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc2ad9d640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc2c0c6bd0 0x7fcc2c06b810 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400c3c0 tx=0x7fcc2400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:33.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc237fe640 1 -- 172.21.15.125:0/3637013807 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc240074e0 con 0x7fcc2c0c6bd0 2023-10-30T13:18:33.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc237fe640 1 -- 172.21.15.125:0/3637013807 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc24007640 con 0x7fcc2c0c6bd0 2023-10-30T13:18:33.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc30ec9640 1 -- 172.21.15.125:0/3637013807 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2c0af860 con 0x7fcc2c0c6bd0 2023-10-30T13:18:33.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc237fe640 1 -- 172.21.15.125:0/3637013807 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24004a30 con 0x7fcc2c0c6bd0 2023-10-30T13:18:33.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc30ec9640 1 -- 172.21.15.125:0/3637013807 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2c1f2080 con 0x7fcc2c0c6bd0 2023-10-30T13:18:33.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.884+0000 7fcc30ec9640 1 -- 172.21.15.125:0/3637013807 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc2c1f22a0 con 0x7fcc2c0c6bd0 2023-10-30T13:18:33.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.888+0000 7fcc237fe640 1 -- 172.21.15.125:0/3637013807 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcc24004b90 con 0x7fcc2c0c6bd0 2023-10-30T13:18:33.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.888+0000 7fcc237fe640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc0c0664d0 0x7fcc0c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:33.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.888+0000 7fcc237fe640 1 -- 172.21.15.125:0/3637013807 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7fcc24084540 con 0x7fcc2c0c6bd0 2023-10-30T13:18:33.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.888+0000 7fcc2a59c640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc0c0664d0 0x7fcc0c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:33.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.888+0000 7fcc2a59c640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc0c0664d0 0x7fcc0c068990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0b2000 tx=0x7fcc1c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:33.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:33.892+0000 7fcc237fe640 1 -- 172.21.15.125:0/3637013807 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcc24051020 con 0x7fcc2c0c6bd0 2023-10-30T13:18:34.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:34.072+0000 7fcc30ec9640 1 -- 172.21.15.125:0/3637013807 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc2c04e170 con 0x7fcc2c0c6bd0 2023-10-30T13:18:34.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:34.076+0000 7fcc237fe640 1 -- 172.21.15.125:0/3637013807 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 234 v234) v1 ==== 95+0+5755 (secure 0 0 0) 0x7fcc24054640 con 0x7fcc2c0c6bd0 2023-10-30T13:18:34.081 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:18:34.081 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":234,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":234,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:18:22.845721+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:34.081 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 234 2023-10-30T13:18:34.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:34.076+0000 7fcc30ec9640 1 -- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc0c0664d0 msgr2=0x7fcc0c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:34.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:34.076+0000 7fcc30ec9640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc0c0664d0 0x7fcc0c068990 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0b2000 tx=0x7fcc1c008040 comp rx=0 tx=0).stop 2023-10-30T13:18:34.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:34.076+0000 7fcc30ec9640 1 -- 172.21.15.125:0/3637013807 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc2c0c6bd0 msgr2=0x7fcc2c06b810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:34.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:34.076+0000 7fcc30ec9640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc2c0c6bd0 0x7fcc2c06b810 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400c3c0 tx=0x7fcc2400c880 comp rx=0 tx=0).stop 2023-10-30T13:18:34.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:34.076+0000 7fcc30ec9640 1 -- 172.21.15.125:0/3637013807 shutdown_connections 2023-10-30T13:18:34.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:34.076+0000 7fcc30ec9640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcc0c0664d0 0x7fcc0c068990 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:34.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:34.076+0000 7fcc30ec9640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcc2c0c6bd0 0x7fcc2c06b810 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:34.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:34.076+0000 7fcc30ec9640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcc2c071110 0x7fcc2c1f25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:34.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:34.076+0000 7fcc30ec9640 1 --2- 172.21.15.125:0/3637013807 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcc2c06f7a0 0x7fcc2c1588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:34.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:34.076+0000 7fcc30ec9640 1 -- 172.21.15.125:0/3637013807 >> 172.21.15.125:0/3637013807 conn(0x7fcc2c0c7910 msgr2=0x7fcc2c0b06c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:34.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:34.076+0000 7fcc30ec9640 1 -- 172.21.15.125:0/3637013807 shutdown_connections 2023-10-30T13:18:34.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:34.076+0000 7fcc30ec9640 1 -- 172.21.15.125:0/3637013807 wait complete. 2023-10-30T13:18:34.102 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.e is in active, standby or standby-replay 2023-10-30T13:18:36.103 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:18:36.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.244+0000 7f0cbad0e640 1 Processor -- start 2023-10-30T13:18:36.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.248+0000 7f0cbad0e640 1 -- start start 2023-10-30T13:18:36.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.248+0000 7f0cbad0e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0cb40c6b80 0x7f0cb40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:36.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.248+0000 7f0cbad0e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0cb40c6f60 0x7f0cb40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:36.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.248+0000 7f0cbad0e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cb40c0a50 con 0x7f0cb40c8430 2023-10-30T13:18:36.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.248+0000 7f0cbad0e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb406f6e0 con 0x7f0cb40c6b80 2023-10-30T13:18:36.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.248+0000 7f0cbad0e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cb406f9c0 con 0x7f0cb40c6f60 2023-10-30T13:18:36.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.248+0000 7f0cb8aaa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0cb40c6b80 0x7f0cb40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:36.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.248+0000 7f0cb8aaa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0cb40c6b80 0x7f0cb40bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59934/0 (socket says 172.21.15.125:59934) 2023-10-30T13:18:36.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.248+0000 7f0cb8aaa640 1 -- 172.21.15.125:0/3871785355 learned_addr learned my addr 172.21.15.125:0/3871785355 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:36.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.248+0000 7f0cabfff640 1 --2- 172.21.15.125:0/3871785355 >> v2:172.21.15.136:3300/0 conn(0x7f0cb40c6f60 0x7f0cb40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:36.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.248+0000 7f0cb8aaa640 1 -- 172.21.15.125:0/3871785355 >> v2:172.21.15.136:3300/0 conn(0x7f0cb40c6f60 msgr2=0x7f0cb40c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:18:36.256 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.248+0000 7f0cb8aaa640 1 --2- 172.21.15.125:0/3871785355 >> v2:172.21.15.136:3300/0 conn(0x7f0cb40c6f60 0x7f0cb40c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:36.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.248+0000 7f0cb8aaa640 1 -- 172.21.15.125:0/3871785355 >> v1:172.21.15.136:6789/0 conn(0x7f0cb40c8430 legacy=0x7f0cb413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:36.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.248+0000 7f0cb8aaa640 1 -- 172.21.15.125:0/3871785355 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb406fca0 con 0x7f0cb40c6b80 2023-10-30T13:18:36.257 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.252+0000 7f0cb8aaa640 1 --2- 172.21.15.125:0/3871785355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6b80 0x7f0cb40bfec0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4159b90 tx=0x7f0cb001ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f375cf2429111845 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:36.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.252+0000 7f0cab7fe640 1 -- 172.21.15.125:0/3871785355 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb00a7bb0 con 0x7f0cb40c6b80 2023-10-30T13:18:36.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.252+0000 7f0cab7fe640 1 -- 172.21.15.125:0/3871785355 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb00a7d10 con 0x7f0cb40c6b80 2023-10-30T13:18:36.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.252+0000 7f0cab7fe640 1 -- 172.21.15.125:0/3871785355 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb00af660 con 0x7f0cb40c6b80 2023-10-30T13:18:36.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.252+0000 7f0cbad0e640 1 -- 172.21.15.125:0/3871785355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6b80 msgr2=0x7f0cb40bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:36.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.252+0000 7f0cbad0e640 1 --2- 172.21.15.125:0/3871785355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6b80 0x7f0cb40bfec0 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4159b90 tx=0x7f0cb001ebd0 comp rx=0 tx=0).stop 2023-10-30T13:18:36.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.252+0000 7f0cbad0e640 1 -- 172.21.15.125:0/3871785355 shutdown_connections 2023-10-30T13:18:36.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.252+0000 7f0cbad0e640 1 --2- 172.21.15.125:0/3871785355 >> v2:172.21.15.136:3300/0 conn(0x7f0cb40c6f60 0x7f0cb40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:36.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.252+0000 7f0cbad0e640 1 --2- 172.21.15.125:0/3871785355 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6b80 0x7f0cb40bfec0 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:36.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.252+0000 7f0cbad0e640 1 -- 172.21.15.125:0/3871785355 >> 172.21.15.125:0/3871785355 conn(0x7f0cb40c7910 msgr2=0x7f0cb40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:36.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.252+0000 7f0cbad0e640 1 -- 172.21.15.125:0/3871785355 shutdown_connections 2023-10-30T13:18:36.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.252+0000 7f0cbad0e640 1 -- 172.21.15.125:0/3871785355 wait complete. 2023-10-30T13:18:36.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.252+0000 7f0cbad0e640 1 Processor -- start 2023-10-30T13:18:36.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.252+0000 7f0cbad0e640 1 -- start start 2023-10-30T13:18:36.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cbad0e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6b80 0x7f0cb4137ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:36.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cbad0e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cb40c6f60 0x7f0cb4154f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:36.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cbad0e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cb40c8430 0x7f0cb41545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:36.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cbad0e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb4135bf0 con 0x7f0cb40c6b80 2023-10-30T13:18:36.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cbad0e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb4160990 con 0x7f0cb40c6f60 2023-10-30T13:18:36.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cbad0e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb40b09e0 con 0x7f0cb40c8430 2023-10-30T13:18:36.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cb8aaa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6b80 0x7f0cb4137ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:36.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cb8aaa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6b80 0x7f0cb4137ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:59944/0 (socket says 172.21.15.125:59944) 2023-10-30T13:18:36.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cb8aaa640 1 -- 172.21.15.125:0/693911150 learned_addr learned my addr 172.21.15.125:0/693911150 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:18:36.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cabfff640 1 --2- 172.21.15.125:0/693911150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cb40c6f60 0x7f0cb4154f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:36.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cb92ab640 1 --2- 172.21.15.125:0/693911150 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cb40c8430 0x7f0cb41545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:36.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cb8aaa640 1 -- 172.21.15.125:0/693911150 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cb40c8430 msgr2=0x7f0cb41545b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:36.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cb8aaa640 1 --2- 172.21.15.125:0/693911150 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cb40c8430 0x7f0cb41545b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:36.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cb8aaa640 1 -- 172.21.15.125:0/693911150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cb40c6f60 msgr2=0x7f0cb4154f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:36.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cb8aaa640 1 --2- 172.21.15.125:0/693911150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cb40c6f60 0x7f0cb4154f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:36.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cb8aaa640 1 -- 172.21.15.125:0/693911150 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb000b770 con 0x7f0cb40c6b80 2023-10-30T13:18:36.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cb8aaa640 1 --2- 172.21.15.125:0/693911150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6b80 0x7f0cb4137ec0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000b2c0 tx=0x7f0cb001b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:36.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0ca97fa640 1 -- 172.21.15.125:0/693911150 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb00ba640 con 0x7f0cb40c6b80 2023-10-30T13:18:36.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0ca97fa640 1 -- 172.21.15.125:0/693911150 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb00ba7a0 con 0x7f0cb40c6b80 2023-10-30T13:18:36.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cbad0e640 1 -- 172.21.15.125:0/693911150 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb41554d0 con 0x7f0cb40c6b80 2023-10-30T13:18:36.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0ca97fa640 1 -- 172.21.15.125:0/693911150 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb00bab30 con 0x7f0cb40c6b80 2023-10-30T13:18:36.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cbad0e640 1 -- 172.21.15.125:0/693911150 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb413de00 con 0x7f0cb40c6b80 2023-10-30T13:18:36.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0cbad0e640 1 -- 172.21.15.125:0/693911150 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb404e170 con 0x7f0cb40c6b80 2023-10-30T13:18:36.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.256+0000 7f0ca97fa640 1 -- 172.21.15.125:0/693911150 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0cb00b93e0 con 0x7f0cb40c6b80 2023-10-30T13:18:36.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.260+0000 7f0ca97fa640 1 --2- 172.21.15.125:0/693911150 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ca00664d0 0x7f0ca0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:18:36.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.260+0000 7f0ca97fa640 1 -- 172.21.15.125:0/693911150 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f0cb011f8b0 con 0x7f0cb40c6b80 2023-10-30T13:18:36.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.260+0000 7f0cabfff640 1 --2- 172.21.15.125:0/693911150 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ca00664d0 0x7f0ca0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:18:36.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.260+0000 7f0cabfff640 1 --2- 172.21.15.125:0/693911150 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ca00664d0 0x7f0ca0068990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4135dd0 tx=0x7f0ca4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:18:36.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.264+0000 7f0ca97fa640 1 -- 172.21.15.125:0/693911150 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0cb00ec390 con 0x7f0cb40c6b80 2023-10-30T13:18:36.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.444+0000 7f0cbad0e640 1 -- 172.21.15.125:0/693911150 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cb40c1480 con 0x7f0cb40c6b80 2023-10-30T13:18:36.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.444+0000 7f0ca97fa640 1 -- 172.21.15.125:0/693911150 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 235 v235) v1 ==== 95+0+6551 (secure 0 0 0) 0x7f0cb00bf090 con 0x7f0cb40c6b80 2023-10-30T13:18:36.451 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:18:36.451 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":235,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9600,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":235,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:18:33.185497+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:18:36.451 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 235 2023-10-30T13:18:36.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.444+0000 7f0cbad0e640 1 -- 172.21.15.125:0/693911150 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ca00664d0 msgr2=0x7f0ca0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:36.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.444+0000 7f0cbad0e640 1 --2- 172.21.15.125:0/693911150 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ca00664d0 0x7f0ca0068990 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4135dd0 tx=0x7f0ca4008040 comp rx=0 tx=0).stop 2023-10-30T13:18:36.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.444+0000 7f0cbad0e640 1 -- 172.21.15.125:0/693911150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6b80 msgr2=0x7f0cb4137ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:18:36.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.444+0000 7f0cbad0e640 1 --2- 172.21.15.125:0/693911150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6b80 0x7f0cb4137ec0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f0cb000b2c0 tx=0x7f0cb001b970 comp rx=0 tx=0).stop 2023-10-30T13:18:36.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.444+0000 7f0cbad0e640 1 -- 172.21.15.125:0/693911150 shutdown_connections 2023-10-30T13:18:36.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.444+0000 7f0cbad0e640 1 --2- 172.21.15.125:0/693911150 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0ca00664d0 0x7f0ca0068990 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:36.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.444+0000 7f0cbad0e640 1 --2- 172.21.15.125:0/693911150 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0cb40c8430 0x7f0cb41545b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:36.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.444+0000 7f0cbad0e640 1 --2- 172.21.15.125:0/693911150 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0cb40c6f60 0x7f0cb4154f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:36.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.444+0000 7f0cbad0e640 1 --2- 172.21.15.125:0/693911150 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0cb40c6b80 0x7f0cb4137ec0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:18:36.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.444+0000 7f0cbad0e640 1 -- 172.21.15.125:0/693911150 >> 172.21.15.125:0/693911150 conn(0x7f0cb40c7910 msgr2=0x7f0cb40c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:18:36.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.448+0000 7f0cbad0e640 1 -- 172.21.15.125:0/693911150 shutdown_connections 2023-10-30T13:18:36.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:18:36.448+0000 7f0cbad0e640 1 -- 172.21.15.125:0/693911150 wait complete. 2023-10-30T13:18:36.473 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e reported in up:standby state 2023-10-30T13:18:36.473 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 113 secs before thrashing 2023-10-30T13:18:39.945 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:18:39.946 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.46 2023-10-30T13:18:39.946 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 46 -eq 48 ] 2023-10-30T13:18:39.946 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.46 2023-10-30T13:18:39.947 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.46 2023-10-30T13:18:39.947 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.46 2023-10-30T13:18:39.947 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:18:47.371 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:18:47.376 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:18:47.939 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:18:47.940 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.47 2023-10-30T13:18:47.940 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 47 -eq 48 ] 2023-10-30T13:18:47.940 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.47 2023-10-30T13:18:47.941 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.47 2023-10-30T13:18:47.941 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.47 2023-10-30T13:18:47.941 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:18:55.348 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:18:55.349 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 48 -eq 48 ] 2023-10-30T13:18:55.349 INFO:tasks.workunit.client.0.smithi125.stderr:+ continue 2023-10-30T13:18:55.349 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 49 -eq 48 ] 2023-10-30T13:18:55.349 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.49 2023-10-30T13:18:55.350 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.49 2023-10-30T13:18:55.350 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.49 2023-10-30T13:18:55.351 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.49 2023-10-30T13:18:55.351 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:19:02.494 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:19:02.495 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.50 2023-10-30T13:19:02.496 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 50 -eq 48 ] 2023-10-30T13:19:02.496 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.50 2023-10-30T13:19:02.496 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.50 2023-10-30T13:19:02.496 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.50 2023-10-30T13:19:02.497 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:19:10.509 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:19:10.510 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 51 -eq 48 ] 2023-10-30T13:19:10.510 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.51 2023-10-30T13:19:10.511 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.51 2023-10-30T13:19:10.511 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.51 2023-10-30T13:19:10.511 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.51 2023-10-30T13:19:10.511 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:19:17.374 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:19:17.374 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 52 -eq 48 ] 2023-10-30T13:19:17.374 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.52 2023-10-30T13:19:17.375 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.52 2023-10-30T13:19:17.375 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.52 2023-10-30T13:19:17.375 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.52 2023-10-30T13:19:17.376 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:19:17.435 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:19:17.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:19:23.495 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:19:23.496 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 53 -eq 48 ] 2023-10-30T13:19:23.496 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.53 2023-10-30T13:19:23.496 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.53 2023-10-30T13:19:23.497 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.53 2023-10-30T13:19:23.497 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.53 2023-10-30T13:19:23.497 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:19:32.467 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:19:32.467 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.54 2023-10-30T13:19:32.468 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 54 -eq 48 ] 2023-10-30T13:19:32.468 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.54 2023-10-30T13:19:32.468 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.54 2023-10-30T13:19:32.469 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.54 2023-10-30T13:19:32.469 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:19:38.193 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:19:38.193 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.55 2023-10-30T13:19:38.194 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 55 -eq 48 ] 2023-10-30T13:19:38.194 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.55 2023-10-30T13:19:38.195 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.55 2023-10-30T13:19:38.195 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.55 2023-10-30T13:19:38.195 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:19:47.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:19:47.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:19:48.497 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:19:48.498 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.56 2023-10-30T13:19:48.498 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 56 -eq 48 ] 2023-10-30T13:19:48.499 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.56 2023-10-30T13:19:48.499 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.56 2023-10-30T13:19:48.499 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.56 2023-10-30T13:19:48.500 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:20:01.495 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:20:01.495 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.57 2023-10-30T13:20:01.496 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 57 -eq 48 ] 2023-10-30T13:20:01.496 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.57 2023-10-30T13:20:01.497 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.57 2023-10-30T13:20:01.497 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.57 2023-10-30T13:20:01.497 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:20:08.617 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:20:08.618 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.58 2023-10-30T13:20:08.618 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 58 -eq 48 ] 2023-10-30T13:20:08.618 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.58 2023-10-30T13:20:08.619 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.58 2023-10-30T13:20:08.619 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.58 2023-10-30T13:20:08.619 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:20:14.644 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:20:14.644 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.59 2023-10-30T13:20:14.645 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 59 -eq 48 ] 2023-10-30T13:20:14.645 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.59 2023-10-30T13:20:14.645 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.59 2023-10-30T13:20:14.645 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.59 2023-10-30T13:20:14.645 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:20:17.546 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:20:17.551 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:20:21.700 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:20:21.700 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.60 2023-10-30T13:20:21.700 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 60 -eq 48 ] 2023-10-30T13:20:21.701 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.60 2023-10-30T13:20:21.701 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.60 2023-10-30T13:20:21.701 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.60 2023-10-30T13:20:21.701 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:20:29.474 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:29.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.618+0000 7f616a2c9640 1 Processor -- start 2023-10-30T13:20:29.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.618+0000 7f616a2c9640 1 -- start start 2023-10-30T13:20:29.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.618+0000 7f616a2c9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f61640c6bd0 0x7f616406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:29.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.618+0000 7f616a2c9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6164066cf0 con 0x7f6164071110 2023-10-30T13:20:29.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.618+0000 7f616a2c9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61640c70d0 con 0x7f616406f7a0 2023-10-30T13:20:29.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.618+0000 7f616a2c9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61640700c0 con 0x7f61640c6bd0 2023-10-30T13:20:29.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.618+0000 7f61637fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f61640c6bd0 0x7f616406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:29.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.618+0000 7f61637fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f61640c6bd0 0x7f616406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35506/0 (socket says 172.21.15.125:35506) 2023-10-30T13:20:29.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.618+0000 7f61637fe640 1 -- 172.21.15.125:0/652191767 learned_addr learned my addr 172.21.15.125:0/652191767 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:29.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.618+0000 7f61637fe640 1 -- 172.21.15.125:0/652191767 >> v1:172.21.15.136:6789/0 conn(0x7f616406f7a0 legacy=0x7f6164069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:29.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.618+0000 7f61637fe640 1 -- 172.21.15.125:0/652191767 >> v1:172.21.15.125:6789/0 conn(0x7f6164071110 legacy=0x7f6164158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:29.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.618+0000 7f61637fe640 1 -- 172.21.15.125:0/652191767 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61640c7390 con 0x7f61640c6bd0 2023-10-30T13:20:29.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.618+0000 7f6162ffd640 1 -- 172.21.15.125:0/652191767 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881621274 0 0) 0x7f6164066cf0 con 0x7f6164071110 2023-10-30T13:20:29.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.622+0000 7f61637fe640 1 --2- 172.21.15.125:0/652191767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61640c6bd0 0x7f616406fb80 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f6158009870 tx=0x7f615801ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a617611ff9782acf server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:29.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.622+0000 7f6162ffd640 1 -- 172.21.15.125:0/652191767 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61580a8a30 con 0x7f61640c6bd0 2023-10-30T13:20:29.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.622+0000 7f6162ffd640 1 -- 172.21.15.125:0/652191767 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61580a8b90 con 0x7f61640c6bd0 2023-10-30T13:20:29.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.622+0000 7f6162ffd640 1 -- 172.21.15.125:0/652191767 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61580b0440 con 0x7f61640c6bd0 2023-10-30T13:20:29.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.622+0000 7f616a2c9640 1 -- 172.21.15.125:0/652191767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61640c6bd0 msgr2=0x7f616406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:29.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.622+0000 7f616a2c9640 1 --2- 172.21.15.125:0/652191767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61640c6bd0 0x7f616406fb80 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7f6158009870 tx=0x7f615801ed30 comp rx=0 tx=0).stop 2023-10-30T13:20:29.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.622+0000 7f616a2c9640 1 -- 172.21.15.125:0/652191767 shutdown_connections 2023-10-30T13:20:29.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.622+0000 7f616a2c9640 1 --2- 172.21.15.125:0/652191767 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f61640c6bd0 0x7f616406fb80 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.622+0000 7f616a2c9640 1 -- 172.21.15.125:0/652191767 >> 172.21.15.125:0/652191767 conn(0x7f61640c7910 msgr2=0x7f61640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:29.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.622+0000 7f616a2c9640 1 -- 172.21.15.125:0/652191767 shutdown_connections 2023-10-30T13:20:29.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.622+0000 7f616a2c9640 1 -- 172.21.15.125:0/652191767 wait complete. 2023-10-30T13:20:29.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f616a2c9640 1 Processor -- start 2023-10-30T13:20:29.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f616a2c9640 1 -- start start 2023-10-30T13:20:29.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f616a2c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f616406f7a0 0x7f6164154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:29.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f616a2c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6164071110 0x7f61640c2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:29.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f616a2c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61640c6bd0 0x7f61640c2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:29.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f616a2c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61641584c0 con 0x7f6164071110 2023-10-30T13:20:29.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f616a2c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61640ad960 con 0x7f61640c6bd0 2023-10-30T13:20:29.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f616a2c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6164157c60 con 0x7f616406f7a0 2023-10-30T13:20:29.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f61637fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6164071110 0x7f61640c2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:29.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f61637fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6164071110 0x7f61640c2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35512/0 (socket says 172.21.15.125:35512) 2023-10-30T13:20:29.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f61637fe640 1 -- 172.21.15.125:0/1937066072 learned_addr learned my addr 172.21.15.125:0/1937066072 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:29.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f6163fff640 1 --2- 172.21.15.125:0/1937066072 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f616406f7a0 0x7f6164154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:29.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f6168866640 1 --2- 172.21.15.125:0/1937066072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61640c6bd0 0x7f61640c2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:29.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f61637fe640 1 -- 172.21.15.125:0/1937066072 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f616406f7a0 msgr2=0x7f6164154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:29.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f61637fe640 1 --2- 172.21.15.125:0/1937066072 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f616406f7a0 0x7f6164154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f61637fe640 1 -- 172.21.15.125:0/1937066072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61640c6bd0 msgr2=0x7f61640c2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:29.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f61637fe640 1 --2- 172.21.15.125:0/1937066072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61640c6bd0 0x7f61640c2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f61637fe640 1 -- 172.21.15.125:0/1937066072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61580094d0 con 0x7f6164071110 2023-10-30T13:20:29.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f61637fe640 1 --2- 172.21.15.125:0/1937066072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6164071110 0x7f61640c2640 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f61580b3040 tx=0x7f615801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:29.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f6160ff9640 1 -- 172.21.15.125:0/1937066072 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61580b9620 con 0x7f6164071110 2023-10-30T13:20:29.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f6160ff9640 1 -- 172.21.15.125:0/1937066072 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61580b97c0 con 0x7f6164071110 2023-10-30T13:20:29.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f616a2c9640 1 -- 172.21.15.125:0/1937066072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61641efd20 con 0x7f6164071110 2023-10-30T13:20:29.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f6160ff9640 1 -- 172.21.15.125:0/1937066072 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61580b9b70 con 0x7f6164071110 2023-10-30T13:20:29.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f616a2c9640 1 -- 172.21.15.125:0/1937066072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61641581e0 con 0x7f6164071110 2023-10-30T13:20:29.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.626+0000 7f616a2c9640 1 -- 172.21.15.125:0/1937066072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616404e170 con 0x7f6164071110 2023-10-30T13:20:29.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.630+0000 7f6160ff9640 1 -- 172.21.15.125:0/1937066072 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f61580b7070 con 0x7f6164071110 2023-10-30T13:20:29.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.630+0000 7f6160ff9640 1 --2- 172.21.15.125:0/1937066072 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f61500664a0 0x7f6150068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:29.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.630+0000 7f6160ff9640 1 -- 172.21.15.125:0/1937066072 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f6158122940 con 0x7f6164071110 2023-10-30T13:20:29.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.630+0000 7f6163fff640 1 --2- 172.21.15.125:0/1937066072 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f61500664a0 0x7f6150068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:29.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.630+0000 7f6163fff640 1 --2- 172.21.15.125:0/1937066072 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f61500664a0 0x7f6150068960 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f6154000f70 tx=0x7f6154009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:29.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.634+0000 7f6160ff9640 1 -- 172.21.15.125:0/1937066072 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f61580ef300 con 0x7f6164071110 2023-10-30T13:20:29.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.814+0000 7f616a2c9640 1 -- 172.21.15.125:0/1937066072 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61640c1c80 con 0x7f6164071110 2023-10-30T13:20:29.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.814+0000 7f6160ff9640 1 -- 172.21.15.125:0/1937066072 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) v1 ==== 95+0+6554 (secure 0 0 0) 0x7f61580b5030 con 0x7f6164071110 2023-10-30T13:20:29.819 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:29.819 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9600,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:19:34.826796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:29.820 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 239 2023-10-30T13:20:29.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.814+0000 7f616a2c9640 1 -- 172.21.15.125:0/1937066072 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f61500664a0 msgr2=0x7f6150068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:29.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.814+0000 7f616a2c9640 1 --2- 172.21.15.125:0/1937066072 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f61500664a0 0x7f6150068960 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f6154000f70 tx=0x7f6154009040 comp rx=0 tx=0).stop 2023-10-30T13:20:29.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.814+0000 7f616a2c9640 1 -- 172.21.15.125:0/1937066072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6164071110 msgr2=0x7f61640c2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:29.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.814+0000 7f616a2c9640 1 --2- 172.21.15.125:0/1937066072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6164071110 0x7f61640c2640 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7f61580b3040 tx=0x7f615801bad0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.814+0000 7f616a2c9640 1 -- 172.21.15.125:0/1937066072 shutdown_connections 2023-10-30T13:20:29.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.814+0000 7f616a2c9640 1 --2- 172.21.15.125:0/1937066072 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f61500664a0 0x7f6150068960 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.814+0000 7f616a2c9640 1 --2- 172.21.15.125:0/1937066072 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f61640c6bd0 0x7f61640c2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.814+0000 7f616a2c9640 1 --2- 172.21.15.125:0/1937066072 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6164071110 0x7f61640c2640 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.814+0000 7f616a2c9640 1 --2- 172.21.15.125:0/1937066072 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f616406f7a0 0x7f6164154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:29.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.814+0000 7f616a2c9640 1 -- 172.21.15.125:0/1937066072 >> 172.21.15.125:0/1937066072 conn(0x7f61640c7910 msgr2=0x7f61640c3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:29.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.814+0000 7f616a2c9640 1 -- 172.21.15.125:0/1937066072 shutdown_connections 2023-10-30T13:20:29.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.814+0000 7f616a2c9640 1 -- 172.21.15.125:0/1937066072 wait complete. 2023-10-30T13:20:29.839 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=2) 2023-10-30T13:20:29.840 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:20:29.840 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:20:29.859 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:20:29.859 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:30.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.994+0000 7ff58efb5640 1 Processor -- start 2023-10-30T13:20:30.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.994+0000 7ff58efb5640 1 -- start start 2023-10-30T13:20:30.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58efb5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff5880c2ba0 0x7ff5880cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:30.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58efb5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58806fe10 con 0x7ff58806f9f0 2023-10-30T13:20:30.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58efb5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff588070050 con 0x7ff5880ccdc0 2023-10-30T13:20:30.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58efb5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5880cd6e0 con 0x7ff5880c2ba0 2023-10-30T13:20:30.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58d7b2640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff58806f9f0 0x7ff5880c2490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60320/0 (socket says 172.21.15.125:60320) 2023-10-30T13:20:30.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58d7b2640 1 -- 172.21.15.125:0/1883047147 learned_addr learned my addr 172.21.15.125:0/1883047147 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:20:30.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58e7b4640 1 --2- 172.21.15.125:0/1883047147 >> v2:172.21.15.136:3300/0 conn(0x7ff5880c2ba0 0x7ff5880cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:30.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 -- 172.21.15.125:0/1883047147 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138618171 0 0) 0x7ff578001120 con 0x7ff58806f9f0 2023-10-30T13:20:30.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 -- 172.21.15.125:0/1883047147 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff574003530 con 0x7ff58806f9f0 2023-10-30T13:20:30.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 -- 172.21.15.125:0/1883047147 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 597417152 0 0) 0x7ff574003530 con 0x7ff58806f9f0 2023-10-30T13:20:30.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 -- 172.21.15.125:0/1883047147 >> v2:172.21.15.136:3300/0 conn(0x7ff5880c2ba0 msgr2=0x7ff5880cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:30.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 --2- 172.21.15.125:0/1883047147 >> v2:172.21.15.136:3300/0 conn(0x7ff5880c2ba0 0x7ff5880cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:30.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 -- 172.21.15.125:0/1883047147 >> v1:172.21.15.136:6789/0 conn(0x7ff5880ccdc0 legacy=0x7ff58815d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:30.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 -- 172.21.15.125:0/1883047147 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff588159860 con 0x7ff58806f9f0 2023-10-30T13:20:30.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 -- 172.21.15.125:0/1883047147 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7ff578001810 con 0x7ff58806f9f0 2023-10-30T13:20:30.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 -- 172.21.15.125:0/1883047147 >> v1:172.21.15.125:6789/0 conn(0x7ff58806f9f0 legacy=0x7ff5880c2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:30.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 --2- 172.21.15.125:0/1883047147 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff574003980 0x7ff574003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:30.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 --2- 172.21.15.125:0/1883047147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5740047d0 0x7ff574004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:30.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 --2- 172.21.15.125:0/1883047147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff574005190 0x7ff57400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:30.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 -- 172.21.15.125:0/1883047147 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5880cd6e0 con 0x7ff574005190 2023-10-30T13:20:30.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 -- 172.21.15.125:0/1883047147 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff57400c040 con 0x7ff5740047d0 2023-10-30T13:20:30.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58cfb1640 1 -- 172.21.15.125:0/1883047147 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff57400c200 con 0x7ff574003980 2023-10-30T13:20:30.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58e7b4640 1 --2- 172.21.15.125:0/1883047147 >> v2:172.21.15.136:3300/0 conn(0x7ff5880c2ba0 0x7ff5880cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:20:30.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58e7b4640 1 --2- 172.21.15.125:0/1883047147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff574005190 0x7ff57400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:30.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58dfb3640 1 --2- 172.21.15.125:0/1883047147 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff574003980 0x7ff574003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:30.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58d7b2640 1 --2- 172.21.15.125:0/1883047147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5740047d0 0x7ff574004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:30.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58e7b4640 1 -- 172.21.15.125:0/1883047147 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff574003980 msgr2=0x7ff574003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:30.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58e7b4640 1 --2- 172.21.15.125:0/1883047147 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff574003980 0x7ff574003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:30.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58e7b4640 1 -- 172.21.15.125:0/1883047147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5740047d0 msgr2=0x7ff574004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:30.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58e7b4640 1 --2- 172.21.15.125:0/1883047147 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5740047d0 0x7ff574004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:30.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:29.998+0000 7ff58e7b4640 1 -- 172.21.15.125:0/1883047147 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff574003530 con 0x7ff574005190 2023-10-30T13:20:30.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.002+0000 7ff58e7b4640 1 -- 172.21.15.125:0/1883047147 reap_dead start 2023-10-30T13:20:30.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.002+0000 7ff58e7b4640 1 --2- 172.21.15.125:0/1883047147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff574005190 0x7ff57400b990 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7ff58000e350 tx=0x7ff58001ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa08096d433d435 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:30.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.002+0000 7ff58cfb1640 1 -- 172.21.15.125:0/1883047147 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5800a5e50 con 0x7ff574005190 2023-10-30T13:20:30.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.002+0000 7ff58cfb1640 1 -- 172.21.15.125:0/1883047147 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58000f040 con 0x7ff574005190 2023-10-30T13:20:30.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.002+0000 7ff58cfb1640 1 -- 172.21.15.125:0/1883047147 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff580003be0 con 0x7ff574005190 2023-10-30T13:20:30.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.002+0000 7ff58efb5640 1 -- 172.21.15.125:0/1883047147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff574005190 msgr2=0x7ff57400b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:30.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.002+0000 7ff58efb5640 1 --2- 172.21.15.125:0/1883047147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff574005190 0x7ff57400b990 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7ff58000e350 tx=0x7ff58001ce40 comp rx=0 tx=0).stop 2023-10-30T13:20:30.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.002+0000 7ff58efb5640 1 -- 172.21.15.125:0/1883047147 shutdown_connections 2023-10-30T13:20:30.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.002+0000 7ff58efb5640 1 --2- 172.21.15.125:0/1883047147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff574005190 0x7ff57400b990 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:30.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.002+0000 7ff58efb5640 1 -- 172.21.15.125:0/1883047147 >> 172.21.15.125:0/1883047147 conn(0x7ff58806dfc0 msgr2=0x7ff588071710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:30.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.002+0000 7ff58efb5640 1 -- 172.21.15.125:0/1883047147 shutdown_connections 2023-10-30T13:20:30.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.002+0000 7ff58efb5640 1 -- 172.21.15.125:0/1883047147 wait complete. 2023-10-30T13:20:30.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58efb5640 1 Processor -- start 2023-10-30T13:20:30.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58efb5640 1 -- start start 2023-10-30T13:20:30.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58efb5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff574005190 0x7ff5881f2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:30.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58efb5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5881490e0 0x7ff5880c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:30.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58efb5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5880c9de0 0x7ff5881451a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:30.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58efb5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff588147f90 con 0x7ff5880c9de0 2023-10-30T13:20:30.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58efb5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5880cc950 con 0x7ff5881490e0 2023-10-30T13:20:30.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58efb5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5880c9410 con 0x7ff574005190 2023-10-30T13:20:30.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58e7b4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5880c9de0 0x7ff5881451a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:30.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58e7b4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5880c9de0 0x7ff5881451a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35536/0 (socket says 172.21.15.125:35536) 2023-10-30T13:20:30.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58e7b4640 1 -- 172.21.15.125:0/488452662 learned_addr learned my addr 172.21.15.125:0/488452662 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:30.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58dfb3640 1 --2- 172.21.15.125:0/488452662 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff574005190 0x7ff5881f2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:30.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58d7b2640 1 --2- 172.21.15.125:0/488452662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5881490e0 0x7ff5880c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:30.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58e7b4640 1 -- 172.21.15.125:0/488452662 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff574005190 msgr2=0x7ff5881f2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:30.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58e7b4640 1 --2- 172.21.15.125:0/488452662 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff574005190 0x7ff5881f2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:30.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58e7b4640 1 -- 172.21.15.125:0/488452662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5881490e0 msgr2=0x7ff5880c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:30.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58e7b4640 1 --2- 172.21.15.125:0/488452662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5881490e0 0x7ff5880c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:30.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58e7b4640 1 -- 172.21.15.125:0/488452662 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff588148e20 con 0x7ff5880c9de0 2023-10-30T13:20:30.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff58e7b4640 1 --2- 172.21.15.125:0/488452662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5880c9de0 0x7ff5881451a0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7ff58001c8b0 tx=0x7ff580009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:30.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.006+0000 7ff57e7fc640 1 -- 172.21.15.125:0/488452662 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5800bb070 con 0x7ff5880c9de0 2023-10-30T13:20:30.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.010+0000 7ff57e7fc640 1 -- 172.21.15.125:0/488452662 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff580003e40 con 0x7ff5880c9de0 2023-10-30T13:20:30.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.010+0000 7ff57e7fc640 1 -- 172.21.15.125:0/488452662 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5800047a0 con 0x7ff5880c9de0 2023-10-30T13:20:30.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.010+0000 7ff58efb5640 1 -- 172.21.15.125:0/488452662 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5881f2be0 con 0x7ff5880c9de0 2023-10-30T13:20:30.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.010+0000 7ff58efb5640 1 -- 172.21.15.125:0/488452662 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5881f2e80 con 0x7ff5880c9de0 2023-10-30T13:20:30.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.010+0000 7ff58efb5640 1 -- 172.21.15.125:0/488452662 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58804e170 con 0x7ff5880c9de0 2023-10-30T13:20:30.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.010+0000 7ff57e7fc640 1 -- 172.21.15.125:0/488452662 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff5800b6480 con 0x7ff5880c9de0 2023-10-30T13:20:30.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.010+0000 7ff57e7fc640 1 --2- 172.21.15.125:0/488452662 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff56c0664d0 0x7ff56c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:30.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.010+0000 7ff57e7fc640 1 -- 172.21.15.125:0/488452662 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7ff58011f1c0 con 0x7ff5880c9de0 2023-10-30T13:20:30.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.010+0000 7ff58dfb3640 1 --2- 172.21.15.125:0/488452662 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff56c0664d0 0x7ff56c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:30.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.014+0000 7ff58dfb3640 1 --2- 172.21.15.125:0/488452662 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff56c0664d0 0x7ff56c068990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7ff584007fd0 tx=0x7ff584009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:30.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.014+0000 7ff57e7fc640 1 -- 172.21.15.125:0/488452662 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff5800ebca0 con 0x7ff5880c9de0 2023-10-30T13:20:30.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.198+0000 7ff58efb5640 1 -- 172.21.15.125:0/488452662 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff588148e20 con 0x7ff5880c9de0 2023-10-30T13:20:30.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.198+0000 7ff57e7fc640 1 -- 172.21.15.125:0/488452662 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 239 v239) v1 ==== 95+0+6554 (secure 0 0 0) 0x7ff588148e20 con 0x7ff5880c9de0 2023-10-30T13:20:30.204 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:30.204 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":239,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9600,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":239,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:19:34.826796+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:30.204 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 239 2023-10-30T13:20:30.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.202+0000 7ff58efb5640 1 -- 172.21.15.125:0/488452662 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff56c0664d0 msgr2=0x7ff56c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:30.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.202+0000 7ff58efb5640 1 --2- 172.21.15.125:0/488452662 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff56c0664d0 0x7ff56c068990 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7ff584007fd0 tx=0x7ff584009040 comp rx=0 tx=0).stop 2023-10-30T13:20:30.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.202+0000 7ff58efb5640 1 -- 172.21.15.125:0/488452662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5880c9de0 msgr2=0x7ff5881451a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:30.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.202+0000 7ff58efb5640 1 --2- 172.21.15.125:0/488452662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5880c9de0 0x7ff5881451a0 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7ff58001c8b0 tx=0x7ff580009750 comp rx=0 tx=0).stop 2023-10-30T13:20:30.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.202+0000 7ff58efb5640 1 -- 172.21.15.125:0/488452662 shutdown_connections 2023-10-30T13:20:30.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.202+0000 7ff58efb5640 1 --2- 172.21.15.125:0/488452662 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff56c0664d0 0x7ff56c068990 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:30.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.202+0000 7ff58efb5640 1 --2- 172.21.15.125:0/488452662 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff5880c9de0 0x7ff5881451a0 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:30.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.202+0000 7ff58efb5640 1 --2- 172.21.15.125:0/488452662 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff5881490e0 0x7ff5880c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:30.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.202+0000 7ff58efb5640 1 --2- 172.21.15.125:0/488452662 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff574005190 0x7ff5881f2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:30.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.202+0000 7ff58efb5640 1 -- 172.21.15.125:0/488452662 >> 172.21.15.125:0/488452662 conn(0x7ff58806dfc0 msgr2=0x7ff588151810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:30.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.202+0000 7ff58efb5640 1 -- 172.21.15.125:0/488452662 shutdown_connections 2023-10-30T13:20:30.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:30.202+0000 7ff58efb5640 1 -- 172.21.15.125:0/488452662 wait complete. 2023-10-30T13:20:30.225 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:20:32.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:32.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.370+0000 7efe990d6640 1 Processor -- start 2023-10-30T13:20:32.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.370+0000 7efe990d6640 1 -- start start 2023-10-30T13:20:32.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.370+0000 7efe990d6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efe94153bf0 0x7efe94071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:32.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.370+0000 7efe990d6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe940700b0 con 0x7efe940c5f60 2023-10-30T13:20:32.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.370+0000 7efe990d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe940c6530 con 0x7efe940c6fe0 2023-10-30T13:20:32.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.370+0000 7efe990d6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe940c6810 con 0x7efe94153bf0 2023-10-30T13:20:32.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.370+0000 7efe92d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7efe94153bf0 0x7efe94071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:32.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.370+0000 7efe9359e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7efe940c5f60 0x7efe940cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43708/0 (socket says 172.21.15.125:43708) 2023-10-30T13:20:32.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.370+0000 7efe9359e640 1 -- 172.21.15.125:0/3883766289 learned_addr learned my addr 172.21.15.125:0/3883766289 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:20:32.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.374+0000 7efe92d9d640 1 -- 172.21.15.125:0/3883766289 >> v1:172.21.15.136:6789/0 conn(0x7efe940c6fe0 legacy=0x7efe940c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:32.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.374+0000 7efe92d9d640 1 -- 172.21.15.125:0/3883766289 >> v1:172.21.15.125:6789/0 conn(0x7efe940c5f60 legacy=0x7efe940cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:32.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.374+0000 7efe92d9d640 1 -- 172.21.15.125:0/3883766289 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe94159b10 con 0x7efe94153bf0 2023-10-30T13:20:32.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.374+0000 7efe9259c640 1 -- 172.21.15.125:0/3883766289 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228494315 0 0) 0x7efe940700b0 con 0x7efe940c5f60 2023-10-30T13:20:32.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.374+0000 7efe92d9d640 1 --2- 172.21.15.125:0/3883766289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94153bf0 0x7efe94071260 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7efe840097c0 tx=0x7efe8401ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6142eb935c1da34 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:32.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.374+0000 7efe9259c640 1 -- 172.21.15.125:0/3883766289 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe840a8930 con 0x7efe94153bf0 2023-10-30T13:20:32.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.374+0000 7efe9259c640 1 -- 172.21.15.125:0/3883766289 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe840a8a90 con 0x7efe94153bf0 2023-10-30T13:20:32.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.374+0000 7efe9259c640 1 -- 172.21.15.125:0/3883766289 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe840a8e20 con 0x7efe94153bf0 2023-10-30T13:20:32.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 -- 172.21.15.125:0/3883766289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94153bf0 msgr2=0x7efe94071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:32.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 --2- 172.21.15.125:0/3883766289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94153bf0 0x7efe94071260 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7efe840097c0 tx=0x7efe8401ec30 comp rx=0 tx=0).stop 2023-10-30T13:20:32.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 -- 172.21.15.125:0/3883766289 shutdown_connections 2023-10-30T13:20:32.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 --2- 172.21.15.125:0/3883766289 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe94153bf0 0x7efe94071260 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:32.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 -- 172.21.15.125:0/3883766289 >> 172.21.15.125:0/3883766289 conn(0x7efe9406c6e0 msgr2=0x7efe94069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:32.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 -- 172.21.15.125:0/3883766289 shutdown_connections 2023-10-30T13:20:32.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 -- 172.21.15.125:0/3883766289 wait complete. 2023-10-30T13:20:32.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 Processor -- start 2023-10-30T13:20:32.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 -- start start 2023-10-30T13:20:32.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe940c5f60 0x7efe941eecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:32.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe940c6fe0 0x7efe94158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:32.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe94153bf0 0x7efe941592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:32.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe941e5330 con 0x7efe940c5f60 2023-10-30T13:20:32.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe941e55d0 con 0x7efe940c6fe0 2023-10-30T13:20:32.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe990d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe940c3c00 con 0x7efe94153bf0 2023-10-30T13:20:32.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe92d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe940c5f60 0x7efe941eecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:32.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe92d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe940c5f60 0x7efe941eecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46672/0 (socket says 172.21.15.125:46672) 2023-10-30T13:20:32.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe92d9d640 1 -- 172.21.15.125:0/86690429 learned_addr learned my addr 172.21.15.125:0/86690429 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:32.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe8bfff640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe940c6fe0 0x7efe94158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:32.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe92d9d640 1 -- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe94153bf0 msgr2=0x7efe941592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:32.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe9359e640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe94153bf0 0x7efe941592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:32.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe92d9d640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe94153bf0 0x7efe941592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:32.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe92d9d640 1 -- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe940c6fe0 msgr2=0x7efe94158db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:32.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe92d9d640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe940c6fe0 0x7efe94158db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:32.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe92d9d640 1 -- 172.21.15.125:0/86690429 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe84009420 con 0x7efe940c5f60 2023-10-30T13:20:32.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe8bfff640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe940c6fe0 0x7efe94158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:20:32.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe9359e640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe94153bf0 0x7efe941592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:20:32.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.378+0000 7efe92d9d640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe940c5f60 0x7efe941eecf0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7efe840b3040 tx=0x7efe8401b9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:32.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.382+0000 7efe8b7fe640 1 -- 172.21.15.125:0/86690429 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe840b0e40 con 0x7efe940c5f60 2023-10-30T13:20:32.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.382+0000 7efe8b7fe640 1 -- 172.21.15.125:0/86690429 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe840b9410 con 0x7efe940c5f60 2023-10-30T13:20:32.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.382+0000 7efe8b7fe640 1 -- 172.21.15.125:0/86690429 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe840b97c0 con 0x7efe940c5f60 2023-10-30T13:20:32.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.382+0000 7efe990d6640 1 -- 172.21.15.125:0/86690429 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe940c3ee0 con 0x7efe940c5f60 2023-10-30T13:20:32.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.382+0000 7efe990d6640 1 -- 172.21.15.125:0/86690429 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe941f2540 con 0x7efe940c5f60 2023-10-30T13:20:32.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.382+0000 7efe990d6640 1 -- 172.21.15.125:0/86690429 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe941537c0 con 0x7efe940c5f60 2023-10-30T13:20:32.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.382+0000 7efe8b7fe640 1 -- 172.21.15.125:0/86690429 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7efe840b7040 con 0x7efe940c5f60 2023-10-30T13:20:32.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.382+0000 7efe8b7fe640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe740664a0 0x7efe74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:32.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.382+0000 7efe8b7fe640 1 -- 172.21.15.125:0/86690429 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7efe84122680 con 0x7efe940c5f60 2023-10-30T13:20:32.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.382+0000 7efe8bfff640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe740664a0 0x7efe74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:32.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.386+0000 7efe8bfff640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe740664a0 0x7efe74068960 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7efe941535b0 tx=0x7efe7c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:32.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.386+0000 7efe8b7fe640 1 -- 172.21.15.125:0/86690429 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7efe840ef040 con 0x7efe940c5f60 2023-10-30T13:20:32.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.570+0000 7efe990d6640 1 -- 172.21.15.125:0/86690429 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe94159dc0 con 0x7efe940c5f60 2023-10-30T13:20:32.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.570+0000 7efe8b7fe640 1 -- 172.21.15.125:0/86690429 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+6556 (secure 0 0 0) 0x7efe840b5030 con 0x7efe940c5f60 2023-10-30T13:20:32.574 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:32.575 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9600,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:20:29.828080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:32.576 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 240 2023-10-30T13:20:32.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.570+0000 7efe990d6640 1 -- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe740664a0 msgr2=0x7efe74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:32.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.570+0000 7efe990d6640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe740664a0 0x7efe74068960 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7efe941535b0 tx=0x7efe7c009040 comp rx=0 tx=0).stop 2023-10-30T13:20:32.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.570+0000 7efe990d6640 1 -- 172.21.15.125:0/86690429 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe940c5f60 msgr2=0x7efe941eecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:32.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.570+0000 7efe990d6640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe940c5f60 0x7efe941eecf0 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7efe840b3040 tx=0x7efe8401b9d0 comp rx=0 tx=0).stop 2023-10-30T13:20:32.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.570+0000 7efe990d6640 1 -- 172.21.15.125:0/86690429 shutdown_connections 2023-10-30T13:20:32.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.570+0000 7efe990d6640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7efe740664a0 0x7efe74068960 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:32.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.570+0000 7efe990d6640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7efe94153bf0 0x7efe941592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:32.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.570+0000 7efe990d6640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7efe940c6fe0 0x7efe94158db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:32.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.570+0000 7efe990d6640 1 --2- 172.21.15.125:0/86690429 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7efe940c5f60 0x7efe941eecf0 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:32.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.570+0000 7efe990d6640 1 -- 172.21.15.125:0/86690429 >> 172.21.15.125:0/86690429 conn(0x7efe9406c6e0 msgr2=0x7efe9406e140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:32.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.570+0000 7efe990d6640 1 -- 172.21.15.125:0/86690429 shutdown_connections 2023-10-30T13:20:32.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:32.570+0000 7efe990d6640 1 -- 172.21.15.125:0/86690429 wait complete. 2023-10-30T13:20:32.595 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:20:34.596 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:34.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.738+0000 7fa190479640 1 Processor -- start 2023-10-30T13:20:34.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.738+0000 7fa190479640 1 -- start start 2023-10-30T13:20:34.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa190479640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa1880c05d0 0x7fa18813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:34.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa190479640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa1880c6f60 0x7fa18813f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:34.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa190479640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1880c0a50 con 0x7fa1880c6b80 2023-10-30T13:20:34.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa190479640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa18806f6e0 con 0x7fa1880c6f60 2023-10-30T13:20:34.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa190479640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa18806f9c0 con 0x7fa1880c05d0 2023-10-30T13:20:34.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa18ea16640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa1880c6f60 0x7fa18813f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:34.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa18ea16640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa1880c6f60 0x7fa18813f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46674/0 (socket says 172.21.15.125:46674) 2023-10-30T13:20:34.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa18ea16640 1 -- 172.21.15.125:0/1494320997 learned_addr learned my addr 172.21.15.125:0/1494320997 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:34.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa18e215640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa1880c6b80 0x7fa1880bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43712/0 (socket says 172.21.15.125:43712) 2023-10-30T13:20:34.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa18da14640 1 --2- 172.21.15.125:0/1494320997 >> v2:172.21.15.136:3300/0 conn(0x7fa1880c05d0 0x7fa18813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:34.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa18ea16640 1 -- 172.21.15.125:0/1494320997 >> v2:172.21.15.136:3300/0 conn(0x7fa1880c05d0 msgr2=0x7fa18813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:34.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa18ea16640 1 --2- 172.21.15.125:0/1494320997 >> v2:172.21.15.136:3300/0 conn(0x7fa1880c05d0 0x7fa18813ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:34.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa18ea16640 1 -- 172.21.15.125:0/1494320997 >> v1:172.21.15.125:6789/0 conn(0x7fa1880c6b80 legacy=0x7fa1880bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:34.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa18ea16640 1 -- 172.21.15.125:0/1494320997 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18806fca0 con 0x7fa1880c6f60 2023-10-30T13:20:34.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.742+0000 7fa18d213640 1 -- 172.21.15.125:0/1494320997 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418093992 0 0) 0x7fa1880c0a50 con 0x7fa1880c6b80 2023-10-30T13:20:34.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa18ea16640 1 --2- 172.21.15.125:0/1494320997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1880c6f60 0x7fa18813f2d0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fa18806c7e0 tx=0x7fa18001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a98c3ce5c6cd0a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:34.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa18d213640 1 -- 172.21.15.125:0/1494320997 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1800a57a0 con 0x7fa1880c6f60 2023-10-30T13:20:34.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa18d213640 1 -- 172.21.15.125:0/1494320997 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1800a5900 con 0x7fa1880c6f60 2023-10-30T13:20:34.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa18d213640 1 -- 172.21.15.125:0/1494320997 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1800a5c90 con 0x7fa1880c6f60 2023-10-30T13:20:34.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa190479640 1 -- 172.21.15.125:0/1494320997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1880c6f60 msgr2=0x7fa18813f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:34.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa190479640 1 --2- 172.21.15.125:0/1494320997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1880c6f60 0x7fa18813f2d0 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7fa18806c7e0 tx=0x7fa18001c9d0 comp rx=0 tx=0).stop 2023-10-30T13:20:34.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa190479640 1 -- 172.21.15.125:0/1494320997 shutdown_connections 2023-10-30T13:20:34.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa190479640 1 --2- 172.21.15.125:0/1494320997 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1880c6f60 0x7fa18813f2d0 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:34.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa190479640 1 --2- 172.21.15.125:0/1494320997 >> v2:172.21.15.136:3300/0 conn(0x7fa1880c05d0 0x7fa18813ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:34.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa190479640 1 -- 172.21.15.125:0/1494320997 >> 172.21.15.125:0/1494320997 conn(0x7fa1880c7910 msgr2=0x7fa1880c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:34.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa190479640 1 -- 172.21.15.125:0/1494320997 shutdown_connections 2023-10-30T13:20:34.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa190479640 1 -- 172.21.15.125:0/1494320997 wait complete. 2023-10-30T13:20:34.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa190479640 1 Processor -- start 2023-10-30T13:20:34.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa190479640 1 -- start start 2023-10-30T13:20:34.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.746+0000 7fa190479640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1880c05d0 0x7fa18814e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:34.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa190479640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1880c6b80 0x7fa18813c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:34.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa190479640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1880c6f60 0x7fa18814d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:34.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa190479640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18814bff0 con 0x7fa1880c05d0 2023-10-30T13:20:34.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa190479640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1881f5980 con 0x7fa1880c6b80 2023-10-30T13:20:34.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa190479640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1880c62c0 con 0x7fa1880c6f60 2023-10-30T13:20:34.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa18e215640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1880c05d0 0x7fa18814e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:34.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa18e215640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1880c05d0 0x7fa18814e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46676/0 (socket says 172.21.15.125:46676) 2023-10-30T13:20:34.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa18e215640 1 -- 172.21.15.125:0/391024542 learned_addr learned my addr 172.21.15.125:0/391024542 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:34.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa18da14640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1880c6b80 0x7fa18813c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:34.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa18ea16640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1880c6f60 0x7fa18814d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:34.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa18e215640 1 -- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1880c6f60 msgr2=0x7fa18814d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:34.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa18e215640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1880c6f60 0x7fa18814d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:34.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa18e215640 1 -- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1880c6b80 msgr2=0x7fa18813c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:34.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa18e215640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1880c6b80 0x7fa18813c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:34.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa18e215640 1 -- 172.21.15.125:0/391024542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa180009b30 con 0x7fa1880c05d0 2023-10-30T13:20:34.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa18da14640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1880c6b80 0x7fa18813c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:20:34.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa18e215640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1880c05d0 0x7fa18814e880 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fa1840011f0 tx=0x7fa18400c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:34.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa17effd640 1 -- 172.21.15.125:0/391024542 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1840074e0 con 0x7fa1880c05d0 2023-10-30T13:20:34.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa17effd640 1 -- 172.21.15.125:0/391024542 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa184007640 con 0x7fa1880c05d0 2023-10-30T13:20:34.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa17effd640 1 -- 172.21.15.125:0/391024542 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa184004a10 con 0x7fa1880c05d0 2023-10-30T13:20:34.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa190479640 1 -- 172.21.15.125:0/391024542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1881f5050 con 0x7fa1880c05d0 2023-10-30T13:20:34.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa190479640 1 -- 172.21.15.125:0/391024542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1881f5610 con 0x7fa1880c05d0 2023-10-30T13:20:34.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa190479640 1 -- 172.21.15.125:0/391024542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa18804e170 con 0x7fa1880c05d0 2023-10-30T13:20:34.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.750+0000 7fa17effd640 1 -- 172.21.15.125:0/391024542 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa184004b70 con 0x7fa1880c05d0 2023-10-30T13:20:34.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.754+0000 7fa17effd640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa16c0664d0 0x7fa16c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:34.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.754+0000 7fa17effd640 1 -- 172.21.15.125:0/391024542 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7fa184083a30 con 0x7fa1880c05d0 2023-10-30T13:20:34.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.754+0000 7fa18da14640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa16c0664d0 0x7fa16c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:34.762 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.754+0000 7fa18da14640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa16c0664d0 0x7fa16c068990 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fa1881ef670 tx=0x7fa178008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:34.763 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.754+0000 7fa17effd640 1 -- 172.21.15.125:0/391024542 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa184050480 con 0x7fa1880c05d0 2023-10-30T13:20:34.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.934+0000 7fa190479640 1 -- 172.21.15.125:0/391024542 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa1880c0060 con 0x7fa1880c05d0 2023-10-30T13:20:34.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.938+0000 7fa17effd640 1 -- 172.21.15.125:0/391024542 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+6556 (secure 0 0 0) 0x7fa184018020 con 0x7fa1880c05d0 2023-10-30T13:20:34.941 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:34.941 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9600,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:20:29.828080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:34.941 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 240 2023-10-30T13:20:34.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.938+0000 7fa190479640 1 -- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa16c0664d0 msgr2=0x7fa16c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:34.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.938+0000 7fa190479640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa16c0664d0 0x7fa16c068990 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fa1881ef670 tx=0x7fa178008040 comp rx=0 tx=0).stop 2023-10-30T13:20:34.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.938+0000 7fa190479640 1 -- 172.21.15.125:0/391024542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1880c05d0 msgr2=0x7fa18814e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:34.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.938+0000 7fa190479640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1880c05d0 0x7fa18814e880 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7fa1840011f0 tx=0x7fa18400c9b0 comp rx=0 tx=0).stop 2023-10-30T13:20:34.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.938+0000 7fa190479640 1 -- 172.21.15.125:0/391024542 shutdown_connections 2023-10-30T13:20:34.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.938+0000 7fa190479640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa16c0664d0 0x7fa16c068990 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:34.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.938+0000 7fa190479640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa1880c6f60 0x7fa18814d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:34.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.938+0000 7fa190479640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa1880c6b80 0x7fa18813c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:34.944 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.938+0000 7fa190479640 1 --2- 172.21.15.125:0/391024542 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa1880c05d0 0x7fa18814e880 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:34.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.938+0000 7fa190479640 1 -- 172.21.15.125:0/391024542 >> 172.21.15.125:0/391024542 conn(0x7fa1880c7910 msgr2=0x7fa1880c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:34.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.938+0000 7fa190479640 1 -- 172.21.15.125:0/391024542 shutdown_connections 2023-10-30T13:20:34.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:34.938+0000 7fa190479640 1 -- 172.21.15.125:0/391024542 wait complete. 2023-10-30T13:20:34.962 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:20:36.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:37.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.106+0000 7f91032bb640 1 Processor -- start 2023-10-30T13:20:37.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.106+0000 7f91032bb640 1 -- start start 2023-10-30T13:20:37.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.110+0000 7f91032bb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc06fc90 0x7f90fc0c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:37.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.110+0000 7f91032bb640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90fc156220 con 0x7f90fc1566b0 2023-10-30T13:20:37.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.110+0000 7f91032bb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90fc156b00 con 0x7f90fc14ff10 2023-10-30T13:20:37.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.110+0000 7f91032bb640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc155bf0 con 0x7f90fc06fc90 2023-10-30T13:20:37.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.110+0000 7f9102aba640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f90fc1566b0 0x7f90fc06a720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43724/0 (socket says 172.21.15.125:43724) 2023-10-30T13:20:37.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.110+0000 7f9102aba640 1 -- 172.21.15.125:0/2114976309 learned_addr learned my addr 172.21.15.125:0/2114976309 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:20:37.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.110+0000 7f9101ab8640 1 --2- 172.21.15.125:0/2114976309 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc06fc90 0x7f90fc0c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:37.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.110+0000 7f91012b7640 1 -- 172.21.15.125:0/2114976309 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751602967 0 0) 0x7f90fc156220 con 0x7f90fc1566b0 2023-10-30T13:20:37.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.110+0000 7f91012b7640 1 -- 172.21.15.125:0/2114976309 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90e8003530 con 0x7f90fc1566b0 2023-10-30T13:20:37.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.110+0000 7f91012b7640 1 -- 172.21.15.125:0/2114976309 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 543863301 0 0) 0x7f90e8003530 con 0x7f90fc1566b0 2023-10-30T13:20:37.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.110+0000 7f9101ab8640 1 -- 172.21.15.125:0/2114976309 >> v1:172.21.15.136:6789/0 conn(0x7f90fc14ff10 legacy=0x7f90fc0c1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:37.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.110+0000 7f9101ab8640 1 -- 172.21.15.125:0/2114976309 >> v1:172.21.15.125:6789/0 conn(0x7f90fc1566b0 legacy=0x7f90fc06a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:37.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.110+0000 7f9101ab8640 1 -- 172.21.15.125:0/2114976309 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90fc070110 con 0x7f90fc06fc90 2023-10-30T13:20:37.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.114+0000 7f9101ab8640 1 --2- 172.21.15.125:0/2114976309 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc06fc90 0x7f90fc0c54d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f90ec0097f0 tx=0x7f90ec01d4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1555bd550ae43d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:37.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.114+0000 7f91012b7640 1 -- 172.21.15.125:0/2114976309 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f4001a00 con 0x7f90fc06fc90 2023-10-30T13:20:37.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.114+0000 7f91012b7640 1 -- 172.21.15.125:0/2114976309 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ec0a8050 con 0x7f90fc06fc90 2023-10-30T13:20:37.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.114+0000 7f91012b7640 1 -- 172.21.15.125:0/2114976309 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec0ac620 con 0x7f90fc06fc90 2023-10-30T13:20:37.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.114+0000 7f91032bb640 1 -- 172.21.15.125:0/2114976309 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc06fc90 msgr2=0x7f90fc0c54d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:37.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.114+0000 7f91032bb640 1 --2- 172.21.15.125:0/2114976309 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc06fc90 0x7f90fc0c54d0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f90ec0097f0 tx=0x7f90ec01d4f0 comp rx=0 tx=0).stop 2023-10-30T13:20:37.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.114+0000 7f91032bb640 1 -- 172.21.15.125:0/2114976309 shutdown_connections 2023-10-30T13:20:37.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.114+0000 7f91032bb640 1 --2- 172.21.15.125:0/2114976309 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc06fc90 0x7f90fc0c54d0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:37.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.114+0000 7f91032bb640 1 -- 172.21.15.125:0/2114976309 >> 172.21.15.125:0/2114976309 conn(0x7f90fc0c75d0 msgr2=0x7f90fc0c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:37.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.114+0000 7f91032bb640 1 -- 172.21.15.125:0/2114976309 shutdown_connections 2023-10-30T13:20:37.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.114+0000 7f91032bb640 1 -- 172.21.15.125:0/2114976309 wait complete. 2023-10-30T13:20:37.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.114+0000 7f91032bb640 1 Processor -- start 2023-10-30T13:20:37.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.114+0000 7f91032bb640 1 -- start start 2023-10-30T13:20:37.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91032bb640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc06fc90 0x7f90fc149920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:37.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91032bb640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc14ff10 0x7f90fc1491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:37.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91032bb640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc1566b0 0x7f90fc14aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:37.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91032bb640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc159a20 con 0x7f90fc06fc90 2023-10-30T13:20:37.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91032bb640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc157750 con 0x7f90fc14ff10 2023-10-30T13:20:37.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91032bb640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc155a60 con 0x7f90fc1566b0 2023-10-30T13:20:37.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91022b9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc06fc90 0x7f90fc149920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:37.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91022b9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc06fc90 0x7f90fc149920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46688/0 (socket says 172.21.15.125:46688) 2023-10-30T13:20:37.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91022b9640 1 -- 172.21.15.125:0/3172623038 learned_addr learned my addr 172.21.15.125:0/3172623038 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:37.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f9102aba640 1 --2- 172.21.15.125:0/3172623038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc1566b0 0x7f90fc14aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:37.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f9101ab8640 1 --2- 172.21.15.125:0/3172623038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc14ff10 0x7f90fc1491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:37.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91022b9640 1 -- 172.21.15.125:0/3172623038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc1566b0 msgr2=0x7f90fc14aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:37.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91022b9640 1 --2- 172.21.15.125:0/3172623038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc1566b0 0x7f90fc14aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:37.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91022b9640 1 -- 172.21.15.125:0/3172623038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc14ff10 msgr2=0x7f90fc1491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:37.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91022b9640 1 --2- 172.21.15.125:0/3172623038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc14ff10 0x7f90fc1491e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:37.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91022b9640 1 -- 172.21.15.125:0/3172623038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90ec009420 con 0x7f90fc06fc90 2023-10-30T13:20:37.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91022b9640 1 --2- 172.21.15.125:0/3172623038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc06fc90 0x7f90fc149920 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f90f800c450 tx=0x7f90f800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:37.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f90f2ffd640 1 -- 172.21.15.125:0/3172623038 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f80052d0 con 0x7f90fc06fc90 2023-10-30T13:20:37.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f90f2ffd640 1 -- 172.21.15.125:0/3172623038 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f8005430 con 0x7f90fc06fc90 2023-10-30T13:20:37.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f90f2ffd640 1 -- 172.21.15.125:0/3172623038 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f8003890 con 0x7f90fc06fc90 2023-10-30T13:20:37.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91032bb640 1 -- 172.21.15.125:0/3172623038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90fc150b20 con 0x7f90fc06fc90 2023-10-30T13:20:37.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91032bb640 1 -- 172.21.15.125:0/3172623038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90fc14d750 con 0x7f90fc06fc90 2023-10-30T13:20:37.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f91032bb640 1 -- 172.21.15.125:0/3172623038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90fc149720 con 0x7f90fc06fc90 2023-10-30T13:20:37.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.118+0000 7f90f2ffd640 1 -- 172.21.15.125:0/3172623038 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f90f80039f0 con 0x7f90fc06fc90 2023-10-30T13:20:37.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.122+0000 7f90f2ffd640 1 --2- 172.21.15.125:0/3172623038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90e00664a0 0x7f90e0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:37.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.122+0000 7f90f2ffd640 1 -- 172.21.15.125:0/3172623038 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f90f8084350 con 0x7f90fc06fc90 2023-10-30T13:20:37.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.122+0000 7f9101ab8640 1 --2- 172.21.15.125:0/3172623038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90e00664a0 0x7f90e0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:37.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.122+0000 7f9101ab8640 1 --2- 172.21.15.125:0/3172623038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90e00664a0 0x7f90e0068960 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f90ec0ae000 tx=0x7f90ec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:37.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.126+0000 7f90f2ffd640 1 -- 172.21.15.125:0/3172623038 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f90f8050d10 con 0x7f90fc06fc90 2023-10-30T13:20:37.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.306+0000 7f91032bb640 1 -- 172.21.15.125:0/3172623038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90fc150b20 con 0x7f90fc06fc90 2023-10-30T13:20:37.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.306+0000 7f90f2ffd640 1 -- 172.21.15.125:0/3172623038 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+6556 (secure 0 0 0) 0x7f90fc150b20 con 0x7f90fc06fc90 2023-10-30T13:20:37.309 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:37.310 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9600,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:20:29.828080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:37.310 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 240 2023-10-30T13:20:37.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.306+0000 7f91032bb640 1 -- 172.21.15.125:0/3172623038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90e00664a0 msgr2=0x7f90e0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:37.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.306+0000 7f91032bb640 1 --2- 172.21.15.125:0/3172623038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90e00664a0 0x7f90e0068960 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f90ec0ae000 tx=0x7f90ec008040 comp rx=0 tx=0).stop 2023-10-30T13:20:37.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.306+0000 7f91032bb640 1 -- 172.21.15.125:0/3172623038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc06fc90 msgr2=0x7f90fc149920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:37.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.306+0000 7f91032bb640 1 --2- 172.21.15.125:0/3172623038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc06fc90 0x7f90fc149920 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f90f800c450 tx=0x7f90f800c910 comp rx=0 tx=0).stop 2023-10-30T13:20:37.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.306+0000 7f91032bb640 1 -- 172.21.15.125:0/3172623038 shutdown_connections 2023-10-30T13:20:37.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.306+0000 7f91032bb640 1 --2- 172.21.15.125:0/3172623038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90e00664a0 0x7f90e0068960 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:37.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.306+0000 7f91032bb640 1 --2- 172.21.15.125:0/3172623038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc1566b0 0x7f90fc14aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:37.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.306+0000 7f91032bb640 1 --2- 172.21.15.125:0/3172623038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc14ff10 0x7f90fc1491e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:37.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.306+0000 7f91032bb640 1 --2- 172.21.15.125:0/3172623038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc06fc90 0x7f90fc149920 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:37.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.306+0000 7f91032bb640 1 -- 172.21.15.125:0/3172623038 >> 172.21.15.125:0/3172623038 conn(0x7f90fc0c75d0 msgr2=0x7f90fc160c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:37.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.306+0000 7f91032bb640 1 -- 172.21.15.125:0/3172623038 shutdown_connections 2023-10-30T13:20:37.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:37.306+0000 7f91032bb640 1 -- 172.21.15.125:0/3172623038 wait complete. 2023-10-30T13:20:37.330 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:20:39.331 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:39.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.474+0000 7f9355980640 1 Processor -- start 2023-10-30T13:20:39.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.474+0000 7f9355980640 1 -- start start 2023-10-30T13:20:39.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.474+0000 7f9355980640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f93500c6bd0 0x7f935006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:39.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.474+0000 7f9355980640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9350071110 0x7f9350158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:39.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.474+0000 7f9355980640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9350066cf0 con 0x7f935006f7a0 2023-10-30T13:20:39.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.474+0000 7f9355980640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93500700c0 con 0x7f93500c6bd0 2023-10-30T13:20:39.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.474+0000 7f9355980640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93500c70d0 con 0x7f9350071110 2023-10-30T13:20:39.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.478+0000 7f9347fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f93500c6bd0 0x7f935006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:39.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.478+0000 7f934effd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f935006f7a0 0x7f9350069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:43740/0 (socket says 172.21.15.125:43740) 2023-10-30T13:20:39.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.478+0000 7f9347fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f93500c6bd0 0x7f935006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46694/0 (socket says 172.21.15.125:46694) 2023-10-30T13:20:39.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.478+0000 7f934effd640 1 -- 172.21.15.125:0/3344307287 learned_addr learned my addr 172.21.15.125:0/3344307287 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:20:39.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.478+0000 7f934f7fe640 1 --2- 172.21.15.125:0/3344307287 >> v2:172.21.15.136:3300/0 conn(0x7f9350071110 0x7f9350158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:39.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.478+0000 7f9347fff640 1 -- 172.21.15.125:0/3344307287 >> v2:172.21.15.136:3300/0 conn(0x7f9350071110 msgr2=0x7f9350158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:39.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.478+0000 7f9347fff640 1 --2- 172.21.15.125:0/3344307287 >> v2:172.21.15.136:3300/0 conn(0x7f9350071110 0x7f9350158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:39.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.478+0000 7f9347fff640 1 -- 172.21.15.125:0/3344307287 >> v1:172.21.15.125:6789/0 conn(0x7f935006f7a0 legacy=0x7f9350069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:39.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.478+0000 7f9347fff640 1 -- 172.21.15.125:0/3344307287 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93500c7390 con 0x7f93500c6bd0 2023-10-30T13:20:39.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.478+0000 7f934e7fc640 1 -- 172.21.15.125:0/3344307287 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827044865 0 0) 0x7f9350066cf0 con 0x7f935006f7a0 2023-10-30T13:20:39.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.478+0000 7f9347fff640 1 --2- 172.21.15.125:0/3344307287 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93500c6bd0 0x7f935006fb80 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f935006cdd0 tx=0x7f933801ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3725c27fc37274de server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:39.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.478+0000 7f934e7fc640 1 -- 172.21.15.125:0/3344307287 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93380a8950 con 0x7f93500c6bd0 2023-10-30T13:20:39.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.478+0000 7f934e7fc640 1 -- 172.21.15.125:0/3344307287 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93380a8ab0 con 0x7f93500c6bd0 2023-10-30T13:20:39.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.478+0000 7f934e7fc640 1 -- 172.21.15.125:0/3344307287 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93380a8e40 con 0x7f93500c6bd0 2023-10-30T13:20:39.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 -- 172.21.15.125:0/3344307287 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93500c6bd0 msgr2=0x7f935006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:39.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 --2- 172.21.15.125:0/3344307287 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93500c6bd0 0x7f935006fb80 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f935006cdd0 tx=0x7f933801ec50 comp rx=0 tx=0).stop 2023-10-30T13:20:39.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 -- 172.21.15.125:0/3344307287 shutdown_connections 2023-10-30T13:20:39.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 --2- 172.21.15.125:0/3344307287 >> v2:172.21.15.136:3300/0 conn(0x7f9350071110 0x7f9350158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:39.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 --2- 172.21.15.125:0/3344307287 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f93500c6bd0 0x7f935006fb80 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:39.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 -- 172.21.15.125:0/3344307287 >> 172.21.15.125:0/3344307287 conn(0x7f93500c7910 msgr2=0x7f93500c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:39.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 -- 172.21.15.125:0/3344307287 shutdown_connections 2023-10-30T13:20:39.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 -- 172.21.15.125:0/3344307287 wait complete. 2023-10-30T13:20:39.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 Processor -- start 2023-10-30T13:20:39.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 -- start start 2023-10-30T13:20:39.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f935006f7a0 0x7f9350156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:39.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9350071110 0x7f93501f2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:39.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93500c6bd0 0x7f935014f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:39.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93501f1f00 con 0x7f9350071110 2023-10-30T13:20:39.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93500afbf0 con 0x7f93500c6bd0 2023-10-30T13:20:39.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9355980640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93501eedc0 con 0x7f935006f7a0 2023-10-30T13:20:39.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9347fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9350071110 0x7f93501f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:39.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9347fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9350071110 0x7f93501f2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46710/0 (socket says 172.21.15.125:46710) 2023-10-30T13:20:39.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9347fff640 1 -- 172.21.15.125:0/1816806579 learned_addr learned my addr 172.21.15.125:0/1816806579 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:39.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f934effd640 1 --2- 172.21.15.125:0/1816806579 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f935006f7a0 0x7f9350156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:39.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f934f7fe640 1 --2- 172.21.15.125:0/1816806579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93500c6bd0 0x7f935014f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:39.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9347fff640 1 -- 172.21.15.125:0/1816806579 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f935006f7a0 msgr2=0x7f9350156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:39.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9347fff640 1 --2- 172.21.15.125:0/1816806579 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f935006f7a0 0x7f9350156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:39.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9347fff640 1 -- 172.21.15.125:0/1816806579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93500c6bd0 msgr2=0x7f935014f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:39.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.482+0000 7f9347fff640 1 --2- 172.21.15.125:0/1816806579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93500c6bd0 0x7f935014f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:39.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.486+0000 7f9347fff640 1 -- 172.21.15.125:0/1816806579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9338009420 con 0x7f9350071110 2023-10-30T13:20:39.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.486+0000 7f9347fff640 1 --2- 172.21.15.125:0/1816806579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9350071110 0x7f93501f2670 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f933801b860 tx=0x7f933801b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:39.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.486+0000 7f934cff9640 1 -- 172.21.15.125:0/1816806579 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93380b0dd0 con 0x7f9350071110 2023-10-30T13:20:39.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.486+0000 7f934cff9640 1 -- 172.21.15.125:0/1816806579 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93380bb440 con 0x7f9350071110 2023-10-30T13:20:39.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.486+0000 7f934cff9640 1 -- 172.21.15.125:0/1816806579 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93380bb7b0 con 0x7f9350071110 2023-10-30T13:20:39.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.486+0000 7f9355980640 1 -- 172.21.15.125:0/1816806579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93501ef0a0 con 0x7f9350071110 2023-10-30T13:20:39.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.486+0000 7f9355980640 1 -- 172.21.15.125:0/1816806579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93501ef660 con 0x7f9350071110 2023-10-30T13:20:39.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.486+0000 7f9355980640 1 -- 172.21.15.125:0/1816806579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f935004e170 con 0x7f9350071110 2023-10-30T13:20:39.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.486+0000 7f934cff9640 1 -- 172.21.15.125:0/1816806579 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f93380b7040 con 0x7f9350071110 2023-10-30T13:20:39.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.486+0000 7f934cff9640 1 --2- 172.21.15.125:0/1816806579 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93300664d0 0x7f9330068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:39.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.486+0000 7f934cff9640 1 -- 172.21.15.125:0/1816806579 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f9338122820 con 0x7f9350071110 2023-10-30T13:20:39.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.490+0000 7f934effd640 1 --2- 172.21.15.125:0/1816806579 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93300664d0 0x7f9330068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:39.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.490+0000 7f934effd640 1 --2- 172.21.15.125:0/1816806579 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93300664d0 0x7f9330068990 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9340000f70 tx=0x7f9340007c10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:39.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.490+0000 7f934cff9640 1 -- 172.21.15.125:0/1816806579 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f93380ef270 con 0x7f9350071110 2023-10-30T13:20:39.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.670+0000 7f9355980640 1 -- 172.21.15.125:0/1816806579 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93501510d0 con 0x7f9350071110 2023-10-30T13:20:39.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.670+0000 7f934cff9640 1 -- 172.21.15.125:0/1816806579 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+6556 (secure 0 0 0) 0x7f93380f2890 con 0x7f9350071110 2023-10-30T13:20:39.674 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:39.675 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9600,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:20:29.828080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:39.675 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 240 2023-10-30T13:20:39.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.670+0000 7f9355980640 1 -- 172.21.15.125:0/1816806579 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93300664d0 msgr2=0x7f9330068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:39.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.670+0000 7f9355980640 1 --2- 172.21.15.125:0/1816806579 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93300664d0 0x7f9330068990 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f9340000f70 tx=0x7f9340007c10 comp rx=0 tx=0).stop 2023-10-30T13:20:39.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.670+0000 7f9355980640 1 -- 172.21.15.125:0/1816806579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9350071110 msgr2=0x7f93501f2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:39.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.670+0000 7f9355980640 1 --2- 172.21.15.125:0/1816806579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9350071110 0x7f93501f2670 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f933801b860 tx=0x7f933801b9f0 comp rx=0 tx=0).stop 2023-10-30T13:20:39.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.670+0000 7f9355980640 1 -- 172.21.15.125:0/1816806579 shutdown_connections 2023-10-30T13:20:39.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.670+0000 7f9355980640 1 --2- 172.21.15.125:0/1816806579 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f93300664d0 0x7f9330068990 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:39.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.670+0000 7f9355980640 1 --2- 172.21.15.125:0/1816806579 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f93500c6bd0 0x7f935014f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:39.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.670+0000 7f9355980640 1 --2- 172.21.15.125:0/1816806579 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9350071110 0x7f93501f2670 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:39.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.670+0000 7f9355980640 1 --2- 172.21.15.125:0/1816806579 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f935006f7a0 0x7f9350156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:39.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.670+0000 7f9355980640 1 -- 172.21.15.125:0/1816806579 >> 172.21.15.125:0/1816806579 conn(0x7f93500c7910 msgr2=0x7f93500c9f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:39.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.670+0000 7f9355980640 1 -- 172.21.15.125:0/1816806579 shutdown_connections 2023-10-30T13:20:39.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:39.670+0000 7f9355980640 1 -- 172.21.15.125:0/1816806579 wait complete. 2023-10-30T13:20:39.696 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:20:41.697 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:41.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.838+0000 7f8ad5491640 1 Processor -- start 2023-10-30T13:20:41.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.838+0000 7f8ad5491640 1 -- start start 2023-10-30T13:20:41.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.842+0000 7f8ad5491640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8ad00cf160 0x7f8ad00cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:41.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.842+0000 7f8ad5491640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad00cb800 con 0x7f8ad01538a0 2023-10-30T13:20:41.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.842+0000 7f8ad5491640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad00cb330 con 0x7f8ad015d6f0 2023-10-30T13:20:41.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.842+0000 7f8ad5491640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ad00cad70 con 0x7f8ad00cf160 2023-10-30T13:20:41.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.842+0000 7f8aceffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8ad01538a0 0x7f8ad00ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58374/0 (socket says 172.21.15.125:58374) 2023-10-30T13:20:41.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.842+0000 7f8aceffd640 1 -- 172.21.15.125:0/204421300 learned_addr learned my addr 172.21.15.125:0/204421300 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:20:41.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.842+0000 7f8acf7fe640 1 --2- 172.21.15.125:0/204421300 >> v2:172.21.15.125:3300/0 conn(0x7f8ad00cf160 0x7f8ad00cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:41.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.842+0000 7f8acf7fe640 1 -- 172.21.15.125:0/204421300 >> v1:172.21.15.136:6789/0 conn(0x7f8ad015d6f0 legacy=0x7f8ad015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:41.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.842+0000 7f8acf7fe640 1 -- 172.21.15.125:0/204421300 >> v1:172.21.15.125:6789/0 conn(0x7f8ad01538a0 legacy=0x7f8ad00ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:41.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.842+0000 7f8acf7fe640 1 -- 172.21.15.125:0/204421300 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad00cb050 con 0x7f8ad00cf160 2023-10-30T13:20:41.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.842+0000 7f8ace7fc640 1 -- 172.21.15.125:0/204421300 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647714324 0 0) 0x7f8ad00cb800 con 0x7f8ad01538a0 2023-10-30T13:20:41.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.842+0000 7f8acf7fe640 1 --2- 172.21.15.125:0/204421300 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad00cf160 0x7f8ad00cf590 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009d90 tx=0x7f8ac801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b5bd55c86bab203 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:41.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.842+0000 7f8ace7fc640 1 -- 172.21.15.125:0/204421300 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac80af600 con 0x7f8ad00cf160 2023-10-30T13:20:41.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ace7fc640 1 -- 172.21.15.125:0/204421300 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac80ab050 con 0x7f8ad00cf160 2023-10-30T13:20:41.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ace7fc640 1 -- 172.21.15.125:0/204421300 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac80af980 con 0x7f8ad00cf160 2023-10-30T13:20:41.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 -- 172.21.15.125:0/204421300 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad00cf160 msgr2=0x7f8ad00cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:41.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 --2- 172.21.15.125:0/204421300 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad00cf160 0x7f8ad00cf590 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009d90 tx=0x7f8ac801f130 comp rx=0 tx=0).stop 2023-10-30T13:20:41.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 -- 172.21.15.125:0/204421300 shutdown_connections 2023-10-30T13:20:41.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 --2- 172.21.15.125:0/204421300 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad00cf160 0x7f8ad00cf590 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:41.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 -- 172.21.15.125:0/204421300 >> 172.21.15.125:0/204421300 conn(0x7f8ad006bfe0 msgr2=0x7f8ad006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:41.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 -- 172.21.15.125:0/204421300 shutdown_connections 2023-10-30T13:20:41.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 -- 172.21.15.125:0/204421300 wait complete. 2023-10-30T13:20:41.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 Processor -- start 2023-10-30T13:20:41.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 -- start start 2023-10-30T13:20:41.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ad00cf160 0x7f8ad014fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:41.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad01538a0 0x7f8ad014ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:41.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ad015d6f0 0x7f8ad014d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:41.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad00b3bb0 con 0x7f8ad01538a0 2023-10-30T13:20:41.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad0151320 con 0x7f8ad015d6f0 2023-10-30T13:20:41.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ad5491640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad01508e0 con 0x7f8ad00cf160 2023-10-30T13:20:41.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ac7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad01538a0 0x7f8ad014ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:41.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ac7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad01538a0 0x7f8ad014ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60346/0 (socket says 172.21.15.125:60346) 2023-10-30T13:20:41.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8ac7fff640 1 -- 172.21.15.125:0/1586597217 learned_addr learned my addr 172.21.15.125:0/1586597217 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:41.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8aceffd640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ad00cf160 0x7f8ad014fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:41.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.846+0000 7f8acf7fe640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ad015d6f0 0x7f8ad014d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:41.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ac7fff640 1 -- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ad00cf160 msgr2=0x7f8ad014fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:41.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ac7fff640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ad00cf160 0x7f8ad014fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:41.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ac7fff640 1 -- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ad015d6f0 msgr2=0x7f8ad014d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:41.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ac7fff640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ad015d6f0 0x7f8ad014d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:41.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ac7fff640 1 -- 172.21.15.125:0/1586597217 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac8009a70 con 0x7f8ad01538a0 2023-10-30T13:20:41.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8aceffd640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ad00cf160 0x7f8ad014fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:41.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ac7fff640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad01538a0 0x7f8ad014ce00 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f8ab800c3c0 tx=0x7f8ab800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:41.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ac77fe640 1 -- 172.21.15.125:0/1586597217 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab80052d0 con 0x7f8ad01538a0 2023-10-30T13:20:41.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ac77fe640 1 -- 172.21.15.125:0/1586597217 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ab8005430 con 0x7f8ad01538a0 2023-10-30T13:20:41.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ad5491640 1 -- 172.21.15.125:0/1586597217 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad00b3dc0 con 0x7f8ad01538a0 2023-10-30T13:20:41.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ac77fe640 1 -- 172.21.15.125:0/1586597217 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8003840 con 0x7f8ad01538a0 2023-10-30T13:20:41.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ad5491640 1 -- 172.21.15.125:0/1586597217 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad015bce0 con 0x7f8ad01538a0 2023-10-30T13:20:41.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ad5491640 1 -- 172.21.15.125:0/1586597217 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad00b4000 con 0x7f8ad01538a0 2023-10-30T13:20:41.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ac77fe640 1 -- 172.21.15.125:0/1586597217 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8ab80039a0 con 0x7f8ad01538a0 2023-10-30T13:20:41.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ac77fe640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8ab00664a0 0x7f8ab0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:41.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.850+0000 7f8ac77fe640 1 -- 172.21.15.125:0/1586597217 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f8ab8083cc0 con 0x7f8ad01538a0 2023-10-30T13:20:41.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.854+0000 7f8aceffd640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8ab00664a0 0x7f8ab0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:41.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.854+0000 7f8aceffd640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8ab00664a0 0x7f8ab0068960 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0000f70 tx=0x7f8ac0007c60 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:41.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:41.854+0000 7f8ac77fe640 1 -- 172.21.15.125:0/1586597217 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8ab808a080 con 0x7f8ad01538a0 2023-10-30T13:20:42.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:42.038+0000 7f8ad5491640 1 -- 172.21.15.125:0/1586597217 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ad0150e80 con 0x7f8ad01538a0 2023-10-30T13:20:42.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:42.038+0000 7f8ac77fe640 1 -- 172.21.15.125:0/1586597217 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+6556 (secure 0 0 0) 0x7f8ab8017020 con 0x7f8ad01538a0 2023-10-30T13:20:42.041 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:42.042 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9600,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:20:29.828080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:42.042 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 240 2023-10-30T13:20:42.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:42.038+0000 7f8ad5491640 1 -- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8ab00664a0 msgr2=0x7f8ab0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:42.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:42.038+0000 7f8ad5491640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8ab00664a0 0x7f8ab0068960 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0000f70 tx=0x7f8ac0007c60 comp rx=0 tx=0).stop 2023-10-30T13:20:42.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:42.038+0000 7f8ad5491640 1 -- 172.21.15.125:0/1586597217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad01538a0 msgr2=0x7f8ad014ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:42.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:42.038+0000 7f8ad5491640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad01538a0 0x7f8ad014ce00 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f8ab800c3c0 tx=0x7f8ab800c880 comp rx=0 tx=0).stop 2023-10-30T13:20:42.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:42.038+0000 7f8ad5491640 1 -- 172.21.15.125:0/1586597217 shutdown_connections 2023-10-30T13:20:42.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:42.038+0000 7f8ad5491640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8ab00664a0 0x7f8ab0068960 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:42.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:42.038+0000 7f8ad5491640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8ad015d6f0 0x7f8ad014d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:42.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:42.038+0000 7f8ad5491640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8ad01538a0 0x7f8ad014ce00 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:42.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:42.038+0000 7f8ad5491640 1 --2- 172.21.15.125:0/1586597217 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8ad00cf160 0x7f8ad014fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:42.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:42.038+0000 7f8ad5491640 1 -- 172.21.15.125:0/1586597217 >> 172.21.15.125:0/1586597217 conn(0x7f8ad006bfe0 msgr2=0x7f8ad015b460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:42.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:42.038+0000 7f8ad5491640 1 -- 172.21.15.125:0/1586597217 shutdown_connections 2023-10-30T13:20:42.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:42.038+0000 7f8ad5491640 1 -- 172.21.15.125:0/1586597217 wait complete. 2023-10-30T13:20:42.062 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:20:44.063 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:44.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.206+0000 7f7cc7372640 1 Processor -- start 2023-10-30T13:20:44.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.206+0000 7f7cc7372640 1 -- start start 2023-10-30T13:20:44.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cc7372640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7cc0070740 0x7f7cc00c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:44.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cc7372640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f7cc015e700 0x7f7cc006a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:44.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cc7372640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc00c5810 con 0x7f7cc01580e0 2023-10-30T13:20:44.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cc7372640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cc0070b60 con 0x7f7cc015e700 2023-10-30T13:20:44.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cc7372640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc015eb90 con 0x7f7cc0070740 2023-10-30T13:20:44.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cc510e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f7cc01580e0 0x7f7cc0069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:58384/0 (socket says 172.21.15.125:58384) 2023-10-30T13:20:44.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cc510e640 1 -- 172.21.15.125:0/2978498398 learned_addr learned my addr 172.21.15.125:0/2978498398 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:20:44.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cc590f640 1 --2- 172.21.15.125:0/2978498398 >> v2:172.21.15.125:3300/0 conn(0x7f7cc015e700 0x7f7cc006a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:44.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cc490d640 1 --2- 172.21.15.125:0/2978498398 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7cc0070740 0x7f7cc00c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:44.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cb7fff640 1 -- 172.21.15.125:0/2978498398 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685780000 0 0) 0x7f7cc00c5810 con 0x7f7cc01580e0 2023-10-30T13:20:44.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cb7fff640 1 -- 172.21.15.125:0/2978498398 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cac003530 con 0x7f7cc01580e0 2023-10-30T13:20:44.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cc590f640 1 -- 172.21.15.125:0/2978498398 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7cc0070740 msgr2=0x7f7cc00c6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:44.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cc590f640 1 --2- 172.21.15.125:0/2978498398 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7cc0070740 0x7f7cc00c6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:44.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cc590f640 1 -- 172.21.15.125:0/2978498398 >> v1:172.21.15.125:6789/0 conn(0x7f7cc01580e0 legacy=0x7f7cc0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:44.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.210+0000 7f7cc590f640 1 -- 172.21.15.125:0/2978498398 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc00c6bb0 con 0x7f7cc015e700 2023-10-30T13:20:44.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.214+0000 7f7cc590f640 1 --2- 172.21.15.125:0/2978498398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cc015e700 0x7f7cc006a5a0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009d90 tx=0x7f7cb801c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71a23bdddd89a4fa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:44.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.214+0000 7f7cb7fff640 1 -- 172.21.15.125:0/2978498398 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb80a59e0 con 0x7f7cc015e700 2023-10-30T13:20:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.214+0000 7f7cb7fff640 1 -- 172.21.15.125:0/2978498398 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb80a5b40 con 0x7f7cc015e700 2023-10-30T13:20:44.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.214+0000 7f7cb7fff640 1 -- 172.21.15.125:0/2978498398 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cb80ad440 con 0x7f7cc015e700 2023-10-30T13:20:44.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.214+0000 7f7cc7372640 1 -- 172.21.15.125:0/2978498398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cc015e700 msgr2=0x7f7cc006a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:44.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.214+0000 7f7cc7372640 1 --2- 172.21.15.125:0/2978498398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cc015e700 0x7f7cc006a5a0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009d90 tx=0x7f7cb801c920 comp rx=0 tx=0).stop 2023-10-30T13:20:44.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.214+0000 7f7cc7372640 1 -- 172.21.15.125:0/2978498398 shutdown_connections 2023-10-30T13:20:44.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.214+0000 7f7cc7372640 1 --2- 172.21.15.125:0/2978498398 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cc015e700 0x7f7cc006a5a0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:44.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.214+0000 7f7cc7372640 1 --2- 172.21.15.125:0/2978498398 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7cc0070740 0x7f7cc00c6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:44.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.214+0000 7f7cc7372640 1 -- 172.21.15.125:0/2978498398 >> 172.21.15.125:0/2978498398 conn(0x7f7cc00c7910 msgr2=0x7f7cc00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:44.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.214+0000 7f7cc7372640 1 -- 172.21.15.125:0/2978498398 shutdown_connections 2023-10-30T13:20:44.221 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.214+0000 7f7cc7372640 1 -- 172.21.15.125:0/2978498398 wait complete. 2023-10-30T13:20:44.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc7372640 1 Processor -- start 2023-10-30T13:20:44.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc7372640 1 -- start start 2023-10-30T13:20:44.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc7372640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cc0070740 0x7f7cc0161110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:44.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc7372640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cc01580e0 0x7f7cc0161650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:44.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc7372640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7cc015e700 0x7f7cc01f4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:44.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc7372640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc01f3c70 con 0x7f7cc0070740 2023-10-30T13:20:44.224 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc7372640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc01f3ef0 con 0x7f7cc01580e0 2023-10-30T13:20:44.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc7372640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cc01f41d0 con 0x7f7cc015e700 2023-10-30T13:20:44.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc510e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cc0070740 0x7f7cc0161110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:44.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc510e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cc0070740 0x7f7cc0161110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60358/0 (socket says 172.21.15.125:60358) 2023-10-30T13:20:44.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc510e640 1 -- 172.21.15.125:0/2934583151 learned_addr learned my addr 172.21.15.125:0/2934583151 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:44.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc490d640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cc01580e0 0x7f7cc0161650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:44.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc590f640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7cc015e700 0x7f7cc01f4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:44.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc510e640 1 -- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7cc015e700 msgr2=0x7f7cc01f4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:44.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc510e640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7cc015e700 0x7f7cc01f4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:44.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc510e640 1 -- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cc01580e0 msgr2=0x7f7cc0161650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:44.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc510e640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cc01580e0 0x7f7cc0161650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:44.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc510e640 1 -- 172.21.15.125:0/2934583151 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cb8009a70 con 0x7f7cc0070740 2023-10-30T13:20:44.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc590f640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7cc015e700 0x7f7cc01f4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:20:44.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc510e640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cc0070740 0x7f7cc0161110 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc00c4d0 tx=0x7f7cbc00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:44.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cb5ffb640 1 -- 172.21.15.125:0/2934583151 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc005cf0 con 0x7f7cc0070740 2023-10-30T13:20:44.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cb5ffb640 1 -- 172.21.15.125:0/2934583151 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cbc005e50 con 0x7f7cc0070740 2023-10-30T13:20:44.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc7372640 1 -- 172.21.15.125:0/2934583151 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc01f44b0 con 0x7f7cc0070740 2023-10-30T13:20:44.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cb5ffb640 1 -- 172.21.15.125:0/2934583151 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cbc007270 con 0x7f7cc0070740 2023-10-30T13:20:44.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.218+0000 7f7cc7372640 1 -- 172.21.15.125:0/2934583151 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc01f4a70 con 0x7f7cc0070740 2023-10-30T13:20:44.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.222+0000 7f7cc7372640 1 -- 172.21.15.125:0/2934583151 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc004e170 con 0x7f7cc0070740 2023-10-30T13:20:44.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.222+0000 7f7cb5ffb640 1 -- 172.21.15.125:0/2934583151 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7cbc0073d0 con 0x7f7cc0070740 2023-10-30T13:20:44.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.222+0000 7f7cb5ffb640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ca40664a0 0x7f7ca4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:44.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.222+0000 7f7cb5ffb640 1 -- 172.21.15.125:0/2934583151 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8146+0+0 (secure 0 0 0) 0x7f7cbc083e90 con 0x7f7cc0070740 2023-10-30T13:20:44.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.222+0000 7f7cc490d640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ca40664a0 0x7f7ca4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:44.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.222+0000 7f7cc490d640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ca40664a0 0x7f7ca4068960 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f7cc00710c0 tx=0x7f7cb0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:44.230 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.226+0000 7f7cb5ffb640 1 -- 172.21.15.125:0/2934583151 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7cbc050900 con 0x7f7cc0070740 2023-10-30T13:20:44.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.406+0000 7f7cc7372640 1 -- 172.21.15.125:0/2934583151 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cc01f4c60 con 0x7f7cc0070740 2023-10-30T13:20:44.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.406+0000 7f7cb5ffb640 1 -- 172.21.15.125:0/2934583151 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 240 v240) v1 ==== 95+0+6556 (secure 0 0 0) 0x7f7cbc085070 con 0x7f7cc0070740 2023-10-30T13:20:44.412 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:44.412 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":240,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9600,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":235}],"filesystems":[{"mdsmap":{"epoch":240,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:20:29.828080+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":8833},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8833":{"gid":8833,"name":"d","rank":2,"incarnation":222,"state":"up:active","state_seq":15,"addr":"172.21.15.136:6835/3864637420","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3864637420},{"type":"v1","addr":"172.21.15.136:6835","nonce":3864637420}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:44.412 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 240 2023-10-30T13:20:44.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.410+0000 7f7cc7372640 1 -- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ca40664a0 msgr2=0x7f7ca4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:44.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.410+0000 7f7cc7372640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ca40664a0 0x7f7ca4068960 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f7cc00710c0 tx=0x7f7cb0008040 comp rx=0 tx=0).stop 2023-10-30T13:20:44.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.410+0000 7f7cc7372640 1 -- 172.21.15.125:0/2934583151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cc0070740 msgr2=0x7f7cc0161110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:44.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.410+0000 7f7cc7372640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cc0070740 0x7f7cc0161110 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f7cbc00c4d0 tx=0x7f7cbc00c990 comp rx=0 tx=0).stop 2023-10-30T13:20:44.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.410+0000 7f7cc7372640 1 -- 172.21.15.125:0/2934583151 shutdown_connections 2023-10-30T13:20:44.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.410+0000 7f7cc7372640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7ca40664a0 0x7f7ca4068960 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:44.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.410+0000 7f7cc7372640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f7cc015e700 0x7f7cc01f4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:44.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.410+0000 7f7cc7372640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f7cc01580e0 0x7f7cc0161650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:44.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.410+0000 7f7cc7372640 1 --2- 172.21.15.125:0/2934583151 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f7cc0070740 0x7f7cc0161110 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:44.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.410+0000 7f7cc7372640 1 -- 172.21.15.125:0/2934583151 >> 172.21.15.125:0/2934583151 conn(0x7f7cc00c7910 msgr2=0x7f7cc0157240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:44.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.410+0000 7f7cc7372640 1 -- 172.21.15.125:0/2934583151 shutdown_connections 2023-10-30T13:20:44.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:44.410+0000 7f7cc7372640 1 -- 172.21.15.125:0/2934583151 wait complete. 2023-10-30T13:20:44.432 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:20:46.433 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:46.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.574+0000 7ff48e3d3640 1 Processor -- start 2023-10-30T13:20:46.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.574+0000 7ff48e3d3640 1 -- start start 2023-10-30T13:20:46.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff48e3d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff488153bf0 0x7ff488071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:46.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff48e3d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ff4880c6fe0 0x7ff4880c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:46.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff48e3d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff4880c79e0 0x7ff4880cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:46.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff48e3d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4880c6810 con 0x7ff4880c79e0 2023-10-30T13:20:46.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff48e3d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff488159900 con 0x7ff4880c6fe0 2023-10-30T13:20:46.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff48e3d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4880700b0 con 0x7ff488153bf0 2023-10-30T13:20:46.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff48c970640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff4880c79e0 0x7ff4880cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:46.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff48c970640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff4880c79e0 0x7ff4880cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60364/0 (socket says 172.21.15.125:60364) 2023-10-30T13:20:46.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff48c970640 1 -- 172.21.15.125:0/3947754679 learned_addr learned my addr 172.21.15.125:0/3947754679 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:46.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff487fff640 1 --2- 172.21.15.125:0/3947754679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff488153bf0 0x7ff488071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:46.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff4877fe640 1 --2- 172.21.15.125:0/3947754679 >> v2:172.21.15.136:3300/0 conn(0x7ff4880c6fe0 0x7ff4880c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:46.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff48c970640 1 -- 172.21.15.125:0/3947754679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff488153bf0 msgr2=0x7ff488071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:46.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff48c970640 1 --2- 172.21.15.125:0/3947754679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff488153bf0 0x7ff488071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff48c970640 1 -- 172.21.15.125:0/3947754679 >> v2:172.21.15.136:3300/0 conn(0x7ff4880c6fe0 msgr2=0x7ff4880c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:46.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff48c970640 1 --2- 172.21.15.125:0/3947754679 >> v2:172.21.15.136:3300/0 conn(0x7ff4880c6fe0 0x7ff4880c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.578+0000 7ff48c970640 1 -- 172.21.15.125:0/3947754679 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4880c6530 con 0x7ff4880c79e0 2023-10-30T13:20:46.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48c970640 1 --2- 172.21.15.125:0/3947754679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4880c79e0 0x7ff4880cb570 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7ff4880c9180 tx=0x7ff48001f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71569daf15bbde16 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:46.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff486ffd640 1 -- 172.21.15.125:0/3947754679 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4800af400 con 0x7ff4880c79e0 2023-10-30T13:20:46.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff486ffd640 1 -- 172.21.15.125:0/3947754679 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4800ab050 con 0x7ff4880c79e0 2023-10-30T13:20:46.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff486ffd640 1 -- 172.21.15.125:0/3947754679 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4800af780 con 0x7ff4880c79e0 2023-10-30T13:20:46.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 -- 172.21.15.125:0/3947754679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4880c79e0 msgr2=0x7ff4880cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:46.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 --2- 172.21.15.125:0/3947754679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4880c79e0 0x7ff4880cb570 secure :-1 s=READY pgs=2003 cs=0 l=1 rev1=1 crypto rx=0x7ff4880c9180 tx=0x7ff48001f1c0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 -- 172.21.15.125:0/3947754679 shutdown_connections 2023-10-30T13:20:46.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 --2- 172.21.15.125:0/3947754679 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff4880c79e0 0x7ff4880cb570 unknown :-1 s=CLOSED pgs=2003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 --2- 172.21.15.125:0/3947754679 >> v2:172.21.15.136:3300/0 conn(0x7ff4880c6fe0 0x7ff4880c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 --2- 172.21.15.125:0/3947754679 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff488153bf0 0x7ff488071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 -- 172.21.15.125:0/3947754679 >> 172.21.15.125:0/3947754679 conn(0x7ff48806c6e0 msgr2=0x7ff488069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:46.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 -- 172.21.15.125:0/3947754679 shutdown_connections 2023-10-30T13:20:46.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 -- 172.21.15.125:0/3947754679 wait complete. 2023-10-30T13:20:46.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 Processor -- start 2023-10-30T13:20:46.592 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 -- start start 2023-10-30T13:20:46.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4880c6fe0 0x7ff488149bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:46.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4880c79e0 0x7ff48814a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:46.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff488153bf0 0x7ff4881f0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:46.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff488155030 con 0x7ff488153bf0 2023-10-30T13:20:46.593 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4881f0260 con 0x7ff4880c79e0 2023-10-30T13:20:46.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.582+0000 7ff48e3d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4881f0b00 con 0x7ff4880c6fe0 2023-10-30T13:20:46.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff48c970640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff488153bf0 0x7ff4881f0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:46.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff48c970640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff488153bf0 0x7ff4881f0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60376/0 (socket says 172.21.15.125:60376) 2023-10-30T13:20:46.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff48c970640 1 -- 172.21.15.125:0/4044802038 learned_addr learned my addr 172.21.15.125:0/4044802038 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:46.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff487fff640 1 --2- 172.21.15.125:0/4044802038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4880c6fe0 0x7ff488149bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:46.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff4877fe640 1 --2- 172.21.15.125:0/4044802038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4880c79e0 0x7ff48814a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:46.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff48c970640 1 -- 172.21.15.125:0/4044802038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4880c6fe0 msgr2=0x7ff488149bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:46.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff48c970640 1 --2- 172.21.15.125:0/4044802038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4880c6fe0 0x7ff488149bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff48c970640 1 -- 172.21.15.125:0/4044802038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4880c79e0 msgr2=0x7ff48814a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:46.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff48c970640 1 --2- 172.21.15.125:0/4044802038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4880c79e0 0x7ff48814a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff48c970640 1 -- 172.21.15.125:0/4044802038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff480009b30 con 0x7ff488153bf0 2023-10-30T13:20:46.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff48c970640 1 --2- 172.21.15.125:0/4044802038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff488153bf0 0x7ff4881f0f90 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7ff4800b2040 tx=0x7ff48001bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:46.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff484ff9640 1 -- 172.21.15.125:0/4044802038 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4800afdc0 con 0x7ff488153bf0 2023-10-30T13:20:46.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff484ff9640 1 -- 172.21.15.125:0/4044802038 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff480009510 con 0x7ff488153bf0 2023-10-30T13:20:46.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff48e3d3640 1 -- 172.21.15.125:0/4044802038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4880c5320 con 0x7ff488153bf0 2023-10-30T13:20:46.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff484ff9640 1 -- 172.21.15.125:0/4044802038 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4800b8640 con 0x7ff488153bf0 2023-10-30T13:20:46.597 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff48e3d3640 1 -- 172.21.15.125:0/4044802038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4881f04b0 con 0x7ff488153bf0 2023-10-30T13:20:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff48e3d3640 1 -- 172.21.15.125:0/4044802038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4881f0cb0 con 0x7ff488153bf0 2023-10-30T13:20:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff484ff9640 1 -- 172.21.15.125:0/4044802038 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff4800b6040 con 0x7ff488153bf0 2023-10-30T13:20:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff484ff9640 1 --2- 172.21.15.125:0/4044802038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4740664a0 0x7ff474068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.586+0000 7ff484ff9640 1 -- 172.21.15.125:0/4044802038 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7ff480122940 con 0x7ff488153bf0 2023-10-30T13:20:46.598 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.590+0000 7ff487fff640 1 --2- 172.21.15.125:0/4044802038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4740664a0 0x7ff474068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:46.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.590+0000 7ff487fff640 1 --2- 172.21.15.125:0/4044802038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4740664a0 0x7ff474068960 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ff47800b930 tx=0x7ff478008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:46.599 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.590+0000 7ff484ff9640 1 -- 172.21.15.125:0/4044802038 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff4881f0cb0 con 0x7ff488153bf0 2023-10-30T13:20:46.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.770+0000 7ff48e3d3640 1 -- 172.21.15.125:0/4044802038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff4881f0cb0 con 0x7ff488153bf0 2023-10-30T13:20:46.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.770+0000 7ff484ff9640 1 -- 172.21.15.125:0/4044802038 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) v1 ==== 95+0+5757 (secure 0 0 0) 0x7ff4881f0cb0 con 0x7ff488153bf0 2023-10-30T13:20:46.776 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:46.776 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":241,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:20:45.886187+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:46.776 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 241 2023-10-30T13:20:46.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.774+0000 7ff48e3d3640 1 -- 172.21.15.125:0/4044802038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4740664a0 msgr2=0x7ff474068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:46.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.774+0000 7ff48e3d3640 1 --2- 172.21.15.125:0/4044802038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4740664a0 0x7ff474068960 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7ff47800b930 tx=0x7ff478008040 comp rx=0 tx=0).stop 2023-10-30T13:20:46.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.774+0000 7ff48e3d3640 1 -- 172.21.15.125:0/4044802038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff488153bf0 msgr2=0x7ff4881f0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:46.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.774+0000 7ff48e3d3640 1 --2- 172.21.15.125:0/4044802038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff488153bf0 0x7ff4881f0f90 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7ff4800b2040 tx=0x7ff48001bf60 comp rx=0 tx=0).stop 2023-10-30T13:20:46.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.774+0000 7ff48e3d3640 1 -- 172.21.15.125:0/4044802038 shutdown_connections 2023-10-30T13:20:46.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.774+0000 7ff48e3d3640 1 --2- 172.21.15.125:0/4044802038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff4740664a0 0x7ff474068960 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.774+0000 7ff48e3d3640 1 --2- 172.21.15.125:0/4044802038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff488153bf0 0x7ff4881f0f90 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.774+0000 7ff48e3d3640 1 --2- 172.21.15.125:0/4044802038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff4880c79e0 0x7ff48814a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.774+0000 7ff48e3d3640 1 --2- 172.21.15.125:0/4044802038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff4880c6fe0 0x7ff488149bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.774+0000 7ff48e3d3640 1 -- 172.21.15.125:0/4044802038 >> 172.21.15.125:0/4044802038 conn(0x7ff48806c6e0 msgr2=0x7ff488151be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:46.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.774+0000 7ff48e3d3640 1 -- 172.21.15.125:0/4044802038 shutdown_connections 2023-10-30T13:20:46.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.774+0000 7ff48e3d3640 1 -- 172.21.15.125:0/4044802038 wait complete. 2023-10-30T13:20:46.797 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T13:20:46.797 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:20:46.798 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:46.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.934+0000 7f0c8a51d640 1 Processor -- start 2023-10-30T13:20:46.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.934+0000 7f0c8a51d640 1 -- start start 2023-10-30T13:20:46.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.934+0000 7f0c8a51d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c840c6b80 0x7f0c840bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:46.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.934+0000 7f0c8a51d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0c840c6f60 0x7f0c840c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:46.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.934+0000 7f0c8a51d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c840c0a50 con 0x7f0c840c8430 2023-10-30T13:20:46.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.934+0000 7f0c8a51d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c8406f6e0 con 0x7f0c840c6f60 2023-10-30T13:20:46.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.934+0000 7f0c8a51d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8406f9c0 con 0x7f0c840c6b80 2023-10-30T13:20:46.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.934+0000 7f0c837fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0c840c6f60 0x7f0c840c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:46.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.934+0000 7f0c83fff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c840c6b80 0x7f0c840bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:46.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.934+0000 7f0c837fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0c840c6f60 0x7f0c840c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:48778/0 (socket says 172.21.15.125:48778) 2023-10-30T13:20:46.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.934+0000 7f0c837fe640 1 -- 172.21.15.125:0/795646746 learned_addr learned my addr 172.21.15.125:0/795646746 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:46.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.938+0000 7f0c837fe640 1 -- 172.21.15.125:0/795646746 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c840c6b80 msgr2=0x7f0c840bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:46.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.938+0000 7f0c837fe640 1 --2- 172.21.15.125:0/795646746 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c840c6b80 0x7f0c840bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.938+0000 7f0c837fe640 1 -- 172.21.15.125:0/795646746 >> v1:172.21.15.136:6789/0 conn(0x7f0c840c8430 legacy=0x7f0c8413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:46.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.938+0000 7f0c837fe640 1 -- 172.21.15.125:0/795646746 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c8406fca0 con 0x7f0c840c6f60 2023-10-30T13:20:46.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.938+0000 7f0c83fff640 1 --2- 172.21.15.125:0/795646746 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c840c6b80 0x7f0c840bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:20:46.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.938+0000 7f0c837fe640 1 --2- 172.21.15.125:0/795646746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c840c6f60 0x7f0c840c0400 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800b920 tx=0x7f0c7801c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20793e02edc7d8d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:46.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.938+0000 7f0c82ffd640 1 -- 172.21.15.125:0/795646746 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c780a59c0 con 0x7f0c840c6f60 2023-10-30T13:20:46.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.938+0000 7f0c82ffd640 1 -- 172.21.15.125:0/795646746 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c780a5b20 con 0x7f0c840c6f60 2023-10-30T13:20:46.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.938+0000 7f0c82ffd640 1 -- 172.21.15.125:0/795646746 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c780ad440 con 0x7f0c840c6f60 2023-10-30T13:20:46.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 -- 172.21.15.125:0/795646746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c840c6f60 msgr2=0x7f0c840c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:46.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 --2- 172.21.15.125:0/795646746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c840c6f60 0x7f0c840c0400 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f0c7800b920 tx=0x7f0c7801c9e0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 -- 172.21.15.125:0/795646746 shutdown_connections 2023-10-30T13:20:46.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 --2- 172.21.15.125:0/795646746 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c840c6f60 0x7f0c840c0400 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 --2- 172.21.15.125:0/795646746 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c840c6b80 0x7f0c840bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 -- 172.21.15.125:0/795646746 >> 172.21.15.125:0/795646746 conn(0x7f0c840c7910 msgr2=0x7f0c840c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:46.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 -- 172.21.15.125:0/795646746 shutdown_connections 2023-10-30T13:20:46.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 -- 172.21.15.125:0/795646746 wait complete. 2023-10-30T13:20:46.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 Processor -- start 2023-10-30T13:20:46.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 -- start start 2023-10-30T13:20:46.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840c6b80 0x7f0c84153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:46.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c840c6f60 0x7f0c841f1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:46.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c840c8430 0x7f0c841f16a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:46.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840c9720 con 0x7f0c840c6b80 2023-10-30T13:20:46.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c84158240 con 0x7f0c840c8430 2023-10-30T13:20:46.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c8a51d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c84134330 con 0x7f0c840c6f60 2023-10-30T13:20:46.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c83fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840c6b80 0x7f0c84153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:46.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c83fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840c6b80 0x7f0c84153310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60382/0 (socket says 172.21.15.125:60382) 2023-10-30T13:20:46.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c83fff640 1 -- 172.21.15.125:0/1139626481 learned_addr learned my addr 172.21.15.125:0/1139626481 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:46.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.942+0000 7f0c837fe640 1 --2- 172.21.15.125:0/1139626481 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c840c6f60 0x7f0c841f1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:46.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.946+0000 7f0c88aba640 1 --2- 172.21.15.125:0/1139626481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c840c8430 0x7f0c841f16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:46.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.946+0000 7f0c83fff640 1 -- 172.21.15.125:0/1139626481 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c840c6f60 msgr2=0x7f0c841f1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:46.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.946+0000 7f0c83fff640 1 --2- 172.21.15.125:0/1139626481 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c840c6f60 0x7f0c841f1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.946+0000 7f0c83fff640 1 -- 172.21.15.125:0/1139626481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c840c8430 msgr2=0x7f0c841f16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:46.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.946+0000 7f0c83fff640 1 --2- 172.21.15.125:0/1139626481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c840c8430 0x7f0c841f16a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:46.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.946+0000 7f0c83fff640 1 -- 172.21.15.125:0/1139626481 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7400d8c0 con 0x7f0c840c6b80 2023-10-30T13:20:46.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.946+0000 7f0c83fff640 1 --2- 172.21.15.125:0/1139626481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840c6b80 0x7f0c84153310 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400ddb0 tx=0x7f0c74010490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:46.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.946+0000 7f0c80ff9640 1 -- 172.21.15.125:0/1139626481 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c74010da0 con 0x7f0c840c6b80 2023-10-30T13:20:46.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.946+0000 7f0c80ff9640 1 -- 172.21.15.125:0/1139626481 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c74013040 con 0x7f0c840c6b80 2023-10-30T13:20:46.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.946+0000 7f0c80ff9640 1 -- 172.21.15.125:0/1139626481 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c740186a0 con 0x7f0c840c6b80 2023-10-30T13:20:46.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.946+0000 7f0c8a51d640 1 -- 172.21.15.125:0/1139626481 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7800b580 con 0x7f0c840c6b80 2023-10-30T13:20:46.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.946+0000 7f0c8a51d640 1 -- 172.21.15.125:0/1139626481 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c84157c80 con 0x7f0c840c6b80 2023-10-30T13:20:46.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.946+0000 7f0c8a51d640 1 -- 172.21.15.125:0/1139626481 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c8404e170 con 0x7f0c840c6b80 2023-10-30T13:20:46.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.946+0000 7f0c80ff9640 1 -- 172.21.15.125:0/1139626481 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0c74018800 con 0x7f0c840c6b80 2023-10-30T13:20:46.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.950+0000 7f0c80ff9640 1 --2- 172.21.15.125:0/1139626481 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c700664a0 0x7f0c70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:46.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.950+0000 7f0c80ff9640 1 -- 172.21.15.125:0/1139626481 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7f0c74088b80 con 0x7f0c840c6b80 2023-10-30T13:20:46.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.950+0000 7f0c837fe640 1 --2- 172.21.15.125:0/1139626481 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c700664a0 0x7f0c70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:46.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.950+0000 7f0c837fe640 1 --2- 172.21.15.125:0/1139626481 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c700664a0 0x7f0c70068960 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f0c780af000 tx=0x7f0c780a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:46.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:46.950+0000 7f0c80ff9640 1 -- 172.21.15.125:0/1139626481 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0c740555d0 con 0x7f0c840c6b80 2023-10-30T13:20:47.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:47.146+0000 7f0c8a51d640 1 -- 172.21.15.125:0/1139626481 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c84157f60 con 0x7f0c840c6b80 2023-10-30T13:20:47.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:47.146+0000 7f0c80ff9640 1 -- 172.21.15.125:0/1139626481 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) v1 ==== 95+0+5757 (secure 0 0 0) 0x7f0c74058bf0 con 0x7f0c840c6b80 2023-10-30T13:20:47.150 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:47.150 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":241,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:20:45.886187+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:47.151 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 241 2023-10-30T13:20:47.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:47.146+0000 7f0c8a51d640 1 -- 172.21.15.125:0/1139626481 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c700664a0 msgr2=0x7f0c70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:47.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:47.146+0000 7f0c8a51d640 1 --2- 172.21.15.125:0/1139626481 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c700664a0 0x7f0c70068960 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f0c780af000 tx=0x7f0c780a7070 comp rx=0 tx=0).stop 2023-10-30T13:20:47.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:47.146+0000 7f0c8a51d640 1 -- 172.21.15.125:0/1139626481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840c6b80 msgr2=0x7f0c84153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:47.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:47.146+0000 7f0c8a51d640 1 --2- 172.21.15.125:0/1139626481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840c6b80 0x7f0c84153310 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400ddb0 tx=0x7f0c74010490 comp rx=0 tx=0).stop 2023-10-30T13:20:47.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:47.146+0000 7f0c8a51d640 1 -- 172.21.15.125:0/1139626481 shutdown_connections 2023-10-30T13:20:47.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:47.146+0000 7f0c8a51d640 1 --2- 172.21.15.125:0/1139626481 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0c700664a0 0x7f0c70068960 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:47.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:47.146+0000 7f0c8a51d640 1 --2- 172.21.15.125:0/1139626481 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0c840c8430 0x7f0c841f16a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:47.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:47.146+0000 7f0c8a51d640 1 --2- 172.21.15.125:0/1139626481 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0c840c6f60 0x7f0c841f1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:47.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:47.146+0000 7f0c8a51d640 1 --2- 172.21.15.125:0/1139626481 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0c840c6b80 0x7f0c84153310 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:47.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:47.146+0000 7f0c8a51d640 1 -- 172.21.15.125:0/1139626481 >> 172.21.15.125:0/1139626481 conn(0x7f0c840c7910 msgr2=0x7f0c84131e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:47.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:47.146+0000 7f0c8a51d640 1 -- 172.21.15.125:0/1139626481 shutdown_connections 2023-10-30T13:20:47.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:47.146+0000 7f0c8a51d640 1 -- 172.21.15.125:0/1139626481 wait complete. 2023-10-30T13:20:47.171 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 241, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 241, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_8971": { "addr": "172.21.15.136:6839/4284345684", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 4284345684, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 4284345684, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 8971, "incarnation": 230, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 32 }, "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 36 }, "gid_9600": { "addr": "172.21.15.125:6836/2077416857", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2077416857, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2077416857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9600, "incarnation": 241, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 83, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:20:45.886187+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8971, "mds_1": 9150, "mds_2": 9600 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:20:47.727 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:20:47.731 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:20:49.172 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:49.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.314+0000 7fd79d165640 1 Processor -- start 2023-10-30T13:20:49.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.314+0000 7fd79d165640 1 -- start start 2023-10-30T13:20:49.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd79d165640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7980c6b80 0x7fd7980bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:49.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd79d165640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd7980c6f60 0x7fd7980c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:49.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd79d165640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd7980c8430 0x7fd79813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:49.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd79d165640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7980c0a50 con 0x7fd7980c8430 2023-10-30T13:20:49.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd79d165640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd79806f6e0 con 0x7fd7980c6f60 2023-10-30T13:20:49.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd79d165640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79806f9c0 con 0x7fd7980c6b80 2023-10-30T13:20:49.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd79759e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd7980c8430 0x7fd79813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:49.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd79759e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd7980c8430 0x7fd79813ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60386/0 (socket says 172.21.15.125:60386) 2023-10-30T13:20:49.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd79759e640 1 -- 172.21.15.125:0/2425797299 learned_addr learned my addr 172.21.15.125:0/2425797299 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:49.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd796d9d640 1 --2- 172.21.15.125:0/2425797299 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7980c6b80 0x7fd7980bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:49.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd78e59c640 1 --2- 172.21.15.125:0/2425797299 >> v2:172.21.15.136:3300/0 conn(0x7fd7980c6f60 0x7fd7980c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:49.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd79759e640 1 -- 172.21.15.125:0/2425797299 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7980c6b80 msgr2=0x7fd7980bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:49.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd79759e640 1 --2- 172.21.15.125:0/2425797299 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7980c6b80 0x7fd7980bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:49.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd79759e640 1 -- 172.21.15.125:0/2425797299 >> v2:172.21.15.136:3300/0 conn(0x7fd7980c6f60 msgr2=0x7fd7980c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:49.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd79759e640 1 --2- 172.21.15.125:0/2425797299 >> v2:172.21.15.136:3300/0 conn(0x7fd7980c6f60 0x7fd7980c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:49.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.318+0000 7fd79759e640 1 -- 172.21.15.125:0/2425797299 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd79806fca0 con 0x7fd7980c8430 2023-10-30T13:20:49.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79759e640 1 --2- 172.21.15.125:0/2425797299 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7980c8430 0x7fd79813ed90 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fd79806c830 tx=0x7fd79001c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d661add074315c29 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:49.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79659c640 1 -- 172.21.15.125:0/2425797299 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7900a5760 con 0x7fd7980c8430 2023-10-30T13:20:49.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79659c640 1 -- 172.21.15.125:0/2425797299 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7900a58c0 con 0x7fd7980c8430 2023-10-30T13:20:49.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79659c640 1 -- 172.21.15.125:0/2425797299 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7900a5c50 con 0x7fd7980c8430 2023-10-30T13:20:49.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79d165640 1 -- 172.21.15.125:0/2425797299 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7980c8430 msgr2=0x7fd79813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:49.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79d165640 1 --2- 172.21.15.125:0/2425797299 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7980c8430 0x7fd79813ed90 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fd79806c830 tx=0x7fd79001c9d0 comp rx=0 tx=0).stop 2023-10-30T13:20:49.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79d165640 1 -- 172.21.15.125:0/2425797299 shutdown_connections 2023-10-30T13:20:49.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79d165640 1 --2- 172.21.15.125:0/2425797299 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7980c8430 0x7fd79813ed90 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:49.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79d165640 1 --2- 172.21.15.125:0/2425797299 >> v2:172.21.15.136:3300/0 conn(0x7fd7980c6f60 0x7fd7980c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:49.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79d165640 1 --2- 172.21.15.125:0/2425797299 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7980c6b80 0x7fd7980bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:49.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79d165640 1 -- 172.21.15.125:0/2425797299 >> 172.21.15.125:0/2425797299 conn(0x7fd7980c7910 msgr2=0x7fd7980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:49.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79d165640 1 -- 172.21.15.125:0/2425797299 shutdown_connections 2023-10-30T13:20:49.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79d165640 1 -- 172.21.15.125:0/2425797299 wait complete. 2023-10-30T13:20:49.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79d165640 1 Processor -- start 2023-10-30T13:20:49.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.322+0000 7fd79d165640 1 -- start start 2023-10-30T13:20:49.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd79d165640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7980c6b80 0x7fd7981388b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:49.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd79d165640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7980c6f60 0x7fd7981f2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:49.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd79d165640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7980c8430 0x7fd79813d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:49.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd79d165640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd798155370 con 0x7fd7980c6f60 2023-10-30T13:20:49.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd79d165640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd798155ef0 con 0x7fd7980c8430 2023-10-30T13:20:49.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd79d165640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd79813d6a0 con 0x7fd7980c6b80 2023-10-30T13:20:49.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd78e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7980c6f60 0x7fd7981f2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:49.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd78e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7980c6f60 0x7fd7981f2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:60402/0 (socket says 172.21.15.125:60402) 2023-10-30T13:20:49.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd78e59c640 1 -- 172.21.15.125:0/385528534 learned_addr learned my addr 172.21.15.125:0/385528534 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:49.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd796d9d640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7980c6b80 0x7fd7981388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:49.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd79759e640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7980c8430 0x7fd79813d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:49.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd78e59c640 1 -- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7980c6b80 msgr2=0x7fd7981388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:49.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd78e59c640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7980c6b80 0x7fd7981388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:49.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd78e59c640 1 -- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7980c8430 msgr2=0x7fd79813d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:49.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd78e59c640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7980c8430 0x7fd79813d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:49.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd78e59c640 1 -- 172.21.15.125:0/385528534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd790009b30 con 0x7fd7980c6f60 2023-10-30T13:20:49.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd796d9d640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7980c6b80 0x7fd7981388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:49.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd78e59c640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7980c6f60 0x7fd7981f2c70 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fd78800d7d0 tx=0x7fd78800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:49.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd78ffff640 1 -- 172.21.15.125:0/385528534 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78800b6c0 con 0x7fd7980c6f60 2023-10-30T13:20:49.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd78ffff640 1 -- 172.21.15.125:0/385528534 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd78800b820 con 0x7fd7980c6f60 2023-10-30T13:20:49.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd78ffff640 1 -- 172.21.15.125:0/385528534 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd78800bbd0 con 0x7fd7980c6f60 2023-10-30T13:20:49.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd79d165640 1 -- 172.21.15.125:0/385528534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7981545b0 con 0x7fd7980c6f60 2023-10-30T13:20:49.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd79d165640 1 -- 172.21.15.125:0/385528534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd798155830 con 0x7fd7980c6f60 2023-10-30T13:20:49.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd79d165640 1 -- 172.21.15.125:0/385528534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd798156030 con 0x7fd7980c6f60 2023-10-30T13:20:49.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.326+0000 7fd78ffff640 1 -- 172.21.15.125:0/385528534 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd78800bd30 con 0x7fd7980c6f60 2023-10-30T13:20:49.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.330+0000 7fd78ffff640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7780664d0 0x7fd778068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:49.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.330+0000 7fd78ffff640 1 -- 172.21.15.125:0/385528534 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7fd788084d50 con 0x7fd7980c6f60 2023-10-30T13:20:49.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.330+0000 7fd796d9d640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7780664d0 0x7fd778068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:49.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.330+0000 7fd796d9d640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7780664d0 0x7fd778068990 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fd780004660 tx=0x7fd780008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:49.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.334+0000 7fd78ffff640 1 -- 172.21.15.125:0/385528534 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd798156030 con 0x7fd7980c6f60 2023-10-30T13:20:49.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.514+0000 7fd79d165640 1 -- 172.21.15.125:0/385528534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd798156030 con 0x7fd7980c6f60 2023-10-30T13:20:49.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.514+0000 7fd78ffff640 1 -- 172.21.15.125:0/385528534 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) v1 ==== 95+0+5757 (secure 0 0 0) 0x7fd798156030 con 0x7fd7980c6f60 2023-10-30T13:20:49.517 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:49.518 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":241,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:20:45.886187+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:49.518 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 241 2023-10-30T13:20:49.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.514+0000 7fd79d165640 1 -- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7780664d0 msgr2=0x7fd778068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:49.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.514+0000 7fd79d165640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7780664d0 0x7fd778068990 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fd780004660 tx=0x7fd780008040 comp rx=0 tx=0).stop 2023-10-30T13:20:49.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.514+0000 7fd79d165640 1 -- 172.21.15.125:0/385528534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7980c6f60 msgr2=0x7fd7981f2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:49.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.514+0000 7fd79d165640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7980c6f60 0x7fd7981f2c70 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fd78800d7d0 tx=0x7fd78800dc90 comp rx=0 tx=0).stop 2023-10-30T13:20:49.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.514+0000 7fd79d165640 1 -- 172.21.15.125:0/385528534 shutdown_connections 2023-10-30T13:20:49.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.514+0000 7fd79d165640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd7780664d0 0x7fd778068990 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:49.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.514+0000 7fd79d165640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd7980c8430 0x7fd79813d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:49.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.514+0000 7fd79d165640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd7980c6f60 0x7fd7981f2c70 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.514+0000 7fd79d165640 1 --2- 172.21.15.125:0/385528534 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd7980c6b80 0x7fd7981388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.514+0000 7fd79d165640 1 -- 172.21.15.125:0/385528534 >> 172.21.15.125:0/385528534 conn(0x7fd7980c7910 msgr2=0x7fd79806a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.514+0000 7fd79d165640 1 -- 172.21.15.125:0/385528534 shutdown_connections 2023-10-30T13:20:49.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:49.514+0000 7fd79d165640 1 -- 172.21.15.125:0/385528534 wait complete. 2023-10-30T13:20:49.539 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:51.540 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:51.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.682+0000 7f911519d640 1 Processor -- start 2023-10-30T13:20:51.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.682+0000 7f911519d640 1 -- start start 2023-10-30T13:20:51.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.682+0000 7f911519d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f91101538a0 0x7f911015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:51.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.682+0000 7f911519d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91100cb800 con 0x7f91100ca0e0 2023-10-30T13:20:51.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.682+0000 7f911519d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f911015dc30 con 0x7f91100cf160 2023-10-30T13:20:51.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.682+0000 7f911519d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91100cb330 con 0x7f91101538a0 2023-10-30T13:20:51.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.682+0000 7f9107fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f91100ca0e0 0x7f911015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47226/0 (socket says 172.21.15.125:47226) 2023-10-30T13:20:51.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.682+0000 7f9107fff640 1 -- 172.21.15.125:0/3095406728 learned_addr learned my addr 172.21.15.125:0/3095406728 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:20:51.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.682+0000 7f910ed9d640 1 --2- 172.21.15.125:0/3095406728 >> v2:172.21.15.136:3300/0 conn(0x7f91101538a0 0x7f911015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:51.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.682+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691079127 0 0) 0x7f91100cb800 con 0x7f91100ca0e0 2023-10-30T13:20:51.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.682+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90fc003530 con 0x7f91100ca0e0 2023-10-30T13:20:51.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444078491 0 0) 0x7f91080014a0 con 0x7f91100cf160 2023-10-30T13:20:51.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91100cb800 con 0x7f91100cf160 2023-10-30T13:20:51.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 867790843 0 0) 0x7f90fc003530 con 0x7f91100ca0e0 2023-10-30T13:20:51.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 >> v2:172.21.15.136:3300/0 conn(0x7f91101538a0 msgr2=0x7f911015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:51.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 --2- 172.21.15.125:0/3095406728 >> v2:172.21.15.136:3300/0 conn(0x7f91101538a0 0x7f911015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:51.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 >> v1:172.21.15.136:6789/0 conn(0x7f91100cf160 legacy=0x7f9110164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:51.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91100cad70 con 0x7f91100ca0e0 2023-10-30T13:20:51.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f90f8001680 con 0x7f91100ca0e0 2023-10-30T13:20:51.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 >> v1:172.21.15.125:6789/0 conn(0x7f91100ca0e0 legacy=0x7f911015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:51.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 --2- 172.21.15.125:0/3095406728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc003b10 0x7f90fc003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:51.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 --2- 172.21.15.125:0/3095406728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc0047d0 0x7f90fc004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:51.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 --2- 172.21.15.125:0/3095406728 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc005190 0x7f90fc00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:51.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91100cb330 con 0x7f90fc003b10 2023-10-30T13:20:51.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc00bf80 con 0x7f90fc0047d0 2023-10-30T13:20:51.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc00c140 con 0x7f90fc005190 2023-10-30T13:20:51.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910ed9d640 1 --2- 172.21.15.125:0/3095406728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc003b10 0x7f90fc003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:51.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f9107fff640 1 --2- 172.21.15.125:0/3095406728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc0047d0 0x7f90fc004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:51.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910f59e640 1 --2- 172.21.15.125:0/3095406728 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc005190 0x7f90fc00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:51.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910ed9d640 1 -- 172.21.15.125:0/3095406728 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc005190 msgr2=0x7f90fc00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:51.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910ed9d640 1 --2- 172.21.15.125:0/3095406728 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc005190 0x7f90fc00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:51.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910ed9d640 1 -- 172.21.15.125:0/3095406728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc0047d0 msgr2=0x7f90fc004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:51.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910ed9d640 1 --2- 172.21.15.125:0/3095406728 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc0047d0 0x7f90fc004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:51.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910ed9d640 1 -- 172.21.15.125:0/3095406728 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90fc003530 con 0x7f90fc003b10 2023-10-30T13:20:51.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.686+0000 7f910f59e640 1 -- 172.21.15.125:0/3095406728 reap_dead start 2023-10-30T13:20:51.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.690+0000 7f910ed9d640 1 --2- 172.21.15.125:0/3095406728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc003b10 0x7f90fc003f10 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f9100009ab0 tx=0x7f910001ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6baa8f9f670229a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:51.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.690+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91000a5620 con 0x7f90fc003b10 2023-10-30T13:20:51.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.690+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9108002100 con 0x7f90fc003b10 2023-10-30T13:20:51.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.690+0000 7f910e59c640 1 -- 172.21.15.125:0/3095406728 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91000a59c0 con 0x7f90fc003b10 2023-10-30T13:20:51.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.690+0000 7f911519d640 1 -- 172.21.15.125:0/3095406728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc003b10 msgr2=0x7f90fc003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:51.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.690+0000 7f911519d640 1 --2- 172.21.15.125:0/3095406728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc003b10 0x7f90fc003f10 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f9100009ab0 tx=0x7f910001ca30 comp rx=0 tx=0).stop 2023-10-30T13:20:51.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.690+0000 7f911519d640 1 -- 172.21.15.125:0/3095406728 shutdown_connections 2023-10-30T13:20:51.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.690+0000 7f911519d640 1 --2- 172.21.15.125:0/3095406728 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc003b10 0x7f90fc003f10 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:51.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.690+0000 7f911519d640 1 -- 172.21.15.125:0/3095406728 >> 172.21.15.125:0/3095406728 conn(0x7f911006bfe0 msgr2=0x7f911006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:51.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.690+0000 7f911519d640 1 -- 172.21.15.125:0/3095406728 shutdown_connections 2023-10-30T13:20:51.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.690+0000 7f911519d640 1 -- 172.21.15.125:0/3095406728 wait complete. 2023-10-30T13:20:51.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.690+0000 7f911519d640 1 Processor -- start 2023-10-30T13:20:51.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.690+0000 7f911519d640 1 -- start start 2023-10-30T13:20:51.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f911519d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc003b10 0x7f91100b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:51.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f911519d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91101e5430 0x7f9110140980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:51.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f911519d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f91101e3c10 0x7f9110140ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:51.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f911519d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f911015b990 con 0x7f91101e5430 2023-10-30T13:20:51.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f911519d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91101e4fa0 con 0x7f90fc003b10 2023-10-30T13:20:51.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f911519d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9110141400 con 0x7f91101e3c10 2023-10-30T13:20:51.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f9107fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91101e5430 0x7f9110140980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:51.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f9107fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91101e5430 0x7f9110140980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39260/0 (socket says 172.21.15.125:39260) 2023-10-30T13:20:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f9107fff640 1 -- 172.21.15.125:0/3128108591 learned_addr learned my addr 172.21.15.125:0/3128108591 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f910ed9d640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc003b10 0x7f91100b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:51.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f910f59e640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f91101e3c10 0x7f9110140ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:51.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f9107fff640 1 -- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f91101e3c10 msgr2=0x7f9110140ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:51.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f9107fff640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f91101e3c10 0x7f9110140ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:51.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f9107fff640 1 -- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc003b10 msgr2=0x7f91100b31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:51.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f9107fff640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc003b10 0x7f91100b31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:51.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f9107fff640 1 -- 172.21.15.125:0/3128108591 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91101509d0 con 0x7f91101e5430 2023-10-30T13:20:51.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f910f59e640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f91101e3c10 0x7f9110140ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:20:51.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f9107fff640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91101e5430 0x7f9110140980 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f90f800e780 tx=0x7f90f800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:51.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f91077fe640 1 -- 172.21.15.125:0/3128108591 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f800c660 con 0x7f91101e5430 2023-10-30T13:20:51.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f91077fe640 1 -- 172.21.15.125:0/3128108591 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f800c7e0 con 0x7f91101e5430 2023-10-30T13:20:51.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f91077fe640 1 -- 172.21.15.125:0/3128108591 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f800cb90 con 0x7f91101e5430 2023-10-30T13:20:51.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f911519d640 1 -- 172.21.15.125:0/3128108591 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9110150c90 con 0x7f91101e5430 2023-10-30T13:20:51.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f911519d640 1 -- 172.21.15.125:0/3128108591 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91101e4950 con 0x7f91101e5430 2023-10-30T13:20:51.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.694+0000 7f911519d640 1 -- 172.21.15.125:0/3128108591 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f911004e170 con 0x7f91101e5430 2023-10-30T13:20:51.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.698+0000 7f91077fe640 1 -- 172.21.15.125:0/3128108591 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f90f800ccf0 con 0x7f91101e5430 2023-10-30T13:20:51.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.698+0000 7f91077fe640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90f00664d0 0x7f90f0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:51.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.698+0000 7f91077fe640 1 -- 172.21.15.125:0/3128108591 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7f90f8084710 con 0x7f91101e5430 2023-10-30T13:20:51.705 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.698+0000 7f910ed9d640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90f00664d0 0x7f90f0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:51.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.698+0000 7f910ed9d640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90f00664d0 0x7f90f0068990 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f9100009ab0 tx=0x7f9100008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:51.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.702+0000 7f91077fe640 1 -- 172.21.15.125:0/3128108591 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f90f8051160 con 0x7f91101e5430 2023-10-30T13:20:51.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.894+0000 7f911519d640 1 -- 172.21.15.125:0/3128108591 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91100b1dd0 con 0x7f91101e5430 2023-10-30T13:20:51.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.894+0000 7f91077fe640 1 -- 172.21.15.125:0/3128108591 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 241 v241) v1 ==== 95+0+5757 (secure 0 0 0) 0x7f90f8054780 con 0x7f91101e5430 2023-10-30T13:20:51.898 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:51.898 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":241,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":241,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:20:45.886187+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:replay","state_seq":1,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:51.898 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 241 2023-10-30T13:20:51.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.894+0000 7f911519d640 1 -- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90f00664d0 msgr2=0x7f90f0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:51.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.894+0000 7f911519d640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90f00664d0 0x7f90f0068990 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f9100009ab0 tx=0x7f9100008040 comp rx=0 tx=0).stop 2023-10-30T13:20:51.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.894+0000 7f911519d640 1 -- 172.21.15.125:0/3128108591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91101e5430 msgr2=0x7f9110140980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:51.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.894+0000 7f911519d640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91101e5430 0x7f9110140980 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f90f800e780 tx=0x7f90f800ec40 comp rx=0 tx=0).stop 2023-10-30T13:20:51.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.894+0000 7f911519d640 1 -- 172.21.15.125:0/3128108591 shutdown_connections 2023-10-30T13:20:51.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.894+0000 7f911519d640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90f00664d0 0x7f90f0068990 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:51.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.894+0000 7f911519d640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f91101e3c10 0x7f9110140ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:51.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.894+0000 7f911519d640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91101e5430 0x7f9110140980 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:51.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.894+0000 7f911519d640 1 --2- 172.21.15.125:0/3128108591 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc003b10 0x7f91100b31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:51.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.894+0000 7f911519d640 1 -- 172.21.15.125:0/3128108591 >> 172.21.15.125:0/3128108591 conn(0x7f911006bfe0 msgr2=0x7f9110142d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:51.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.894+0000 7f911519d640 1 -- 172.21.15.125:0/3128108591 shutdown_connections 2023-10-30T13:20:51.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:51.894+0000 7f911519d640 1 -- 172.21.15.125:0/3128108591 wait complete. 2023-10-30T13:20:51.922 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:53.923 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:54.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.066+0000 7fbc3df3e640 1 Processor -- start 2023-10-30T13:20:54.072 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.066+0000 7fbc3df3e640 1 -- start start 2023-10-30T13:20:54.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc3df3e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbc380c6b80 0x7fbc380bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:54.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc3df3e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc380c0a50 con 0x7fbc3813f4a0 2023-10-30T13:20:54.076 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc3df3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc3806f6e0 con 0x7fbc380c6f60 2023-10-30T13:20:54.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc3df3e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc3806f9c0 con 0x7fbc380c6b80 2023-10-30T13:20:54.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc37fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fbc3813f4a0 0x7fbc381566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47232/0 (socket says 172.21.15.125:47232) 2023-10-30T13:20:54.077 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc37fff640 1 -- 172.21.15.125:0/1277771499 learned_addr learned my addr 172.21.15.125:0/1277771499 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:20:54.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc377fe640 1 --2- 172.21.15.125:0/1277771499 >> v2:172.21.15.136:3300/0 conn(0x7fbc380c6b80 0x7fbc380bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:54.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 -- 172.21.15.125:0/1277771499 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109215999 0 0) 0x7fbc380c0a50 con 0x7fbc3813f4a0 2023-10-30T13:20:54.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 -- 172.21.15.125:0/1277771499 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc20003530 con 0x7fbc3813f4a0 2023-10-30T13:20:54.078 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 -- 172.21.15.125:0/1277771499 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2129426032 0 0) 0x7fbc20003530 con 0x7fbc3813f4a0 2023-10-30T13:20:54.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 -- 172.21.15.125:0/1277771499 >> v2:172.21.15.136:3300/0 conn(0x7fbc380c6b80 msgr2=0x7fbc380bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:54.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 --2- 172.21.15.125:0/1277771499 >> v2:172.21.15.136:3300/0 conn(0x7fbc380c6b80 0x7fbc380bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:54.079 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 -- 172.21.15.125:0/1277771499 >> v1:172.21.15.136:6789/0 conn(0x7fbc380c6f60 legacy=0x7fbc3813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:54.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 -- 172.21.15.125:0/1277771499 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc3806fca0 con 0x7fbc3813f4a0 2023-10-30T13:20:54.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 -- 172.21.15.125:0/1277771499 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fbc30001a00 con 0x7fbc3813f4a0 2023-10-30T13:20:54.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 -- 172.21.15.125:0/1277771499 >> v1:172.21.15.125:6789/0 conn(0x7fbc3813f4a0 legacy=0x7fbc381566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:54.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 --2- 172.21.15.125:0/1277771499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc20003980 0x7fbc20003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:54.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 --2- 172.21.15.125:0/1277771499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc200047d0 0x7fbc20004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:54.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 --2- 172.21.15.125:0/1277771499 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc20005190 0x7fbc2000dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:54.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 -- 172.21.15.125:0/1277771499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3806f9c0 con 0x7fbc200047d0 2023-10-30T13:20:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 -- 172.21.15.125:0/1277771499 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2000e240 con 0x7fbc20003980 2023-10-30T13:20:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc367fc640 1 -- 172.21.15.125:0/1277771499 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2000e400 con 0x7fbc20005190 2023-10-30T13:20:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc36ffd640 1 --2- 172.21.15.125:0/1277771499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc200047d0 0x7fbc20004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:54.082 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc37fff640 1 --2- 172.21.15.125:0/1277771499 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc20005190 0x7fbc2000dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:54.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc377fe640 1 --2- 172.21.15.125:0/1277771499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc20003980 0x7fbc20003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:54.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc36ffd640 1 -- 172.21.15.125:0/1277771499 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc20005190 msgr2=0x7fbc2000dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:54.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc36ffd640 1 --2- 172.21.15.125:0/1277771499 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc20005190 0x7fbc2000dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:54.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc36ffd640 1 -- 172.21.15.125:0/1277771499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc20003980 msgr2=0x7fbc20003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:54.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc36ffd640 1 --2- 172.21.15.125:0/1277771499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc20003980 0x7fbc20003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:54.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc36ffd640 1 -- 172.21.15.125:0/1277771499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc20003530 con 0x7fbc200047d0 2023-10-30T13:20:54.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.070+0000 7fbc37fff640 1 -- 172.21.15.125:0/1277771499 reap_dead start 2023-10-30T13:20:54.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.074+0000 7fbc36ffd640 1 --2- 172.21.15.125:0/1277771499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc200047d0 0x7fbc20004c50 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c001780 tx=0x7fbc2c01c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b45cf477b3fad8e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:54.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.074+0000 7fbc367fc640 1 -- 172.21.15.125:0/1277771499 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c0a4ea0 con 0x7fbc200047d0 2023-10-30T13:20:54.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.074+0000 7fbc367fc640 1 -- 172.21.15.125:0/1277771499 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2c0a7050 con 0x7fbc200047d0 2023-10-30T13:20:54.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.074+0000 7fbc367fc640 1 -- 172.21.15.125:0/1277771499 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c003600 con 0x7fbc200047d0 2023-10-30T13:20:54.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.074+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1277771499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc200047d0 msgr2=0x7fbc20004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:54.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.074+0000 7fbc3df3e640 1 --2- 172.21.15.125:0/1277771499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc200047d0 0x7fbc20004c50 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c001780 tx=0x7fbc2c01c3e0 comp rx=0 tx=0).stop 2023-10-30T13:20:54.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.074+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1277771499 shutdown_connections 2023-10-30T13:20:54.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.074+0000 7fbc3df3e640 1 --2- 172.21.15.125:0/1277771499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc200047d0 0x7fbc20004c50 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:54.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.074+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1277771499 >> 172.21.15.125:0/1277771499 conn(0x7fbc380c7910 msgr2=0x7fbc380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:54.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.074+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1277771499 shutdown_connections 2023-10-30T13:20:54.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.074+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1277771499 wait complete. 2023-10-30T13:20:54.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc3df3e640 1 Processor -- start 2023-10-30T13:20:54.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc3df3e640 1 -- start start 2023-10-30T13:20:54.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc3df3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc200047d0 0x7fbc38138380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:54.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc3df3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc380c0f80 0x7fbc38139e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:54.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc3df3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc381f2940 0x7fbc381378d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:54.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc3df3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc381f21d0 con 0x7fbc381f2940 2023-10-30T13:20:54.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc3df3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc381393f0 con 0x7fbc200047d0 2023-10-30T13:20:54.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc3df3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc38139690 con 0x7fbc380c0f80 2023-10-30T13:20:54.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc37fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc381f2940 0x7fbc381378d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:54.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc37fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc381f2940 0x7fbc381378d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39272/0 (socket says 172.21.15.125:39272) 2023-10-30T13:20:54.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc37fff640 1 -- 172.21.15.125:0/1990093739 learned_addr learned my addr 172.21.15.125:0/1990093739 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:54.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc377fe640 1 --2- 172.21.15.125:0/1990093739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc200047d0 0x7fbc38138380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:54.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc36ffd640 1 --2- 172.21.15.125:0/1990093739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc380c0f80 0x7fbc38139e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:54.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc37fff640 1 -- 172.21.15.125:0/1990093739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc380c0f80 msgr2=0x7fbc38139e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:54.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc37fff640 1 --2- 172.21.15.125:0/1990093739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc380c0f80 0x7fbc38139e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:54.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc37fff640 1 -- 172.21.15.125:0/1990093739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc200047d0 msgr2=0x7fbc38138380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:54.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc37fff640 1 --2- 172.21.15.125:0/1990093739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc200047d0 0x7fbc38138380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:54.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc37fff640 1 -- 172.21.15.125:0/1990093739 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc38157060 con 0x7fbc381f2940 2023-10-30T13:20:54.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc37fff640 1 --2- 172.21.15.125:0/1990093739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc381f2940 0x7fbc381378d0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fbc30001340 tx=0x7fbc30011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:54.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc1ffff640 1 -- 172.21.15.125:0/1990093739 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3000f920 con 0x7fbc381f2940 2023-10-30T13:20:54.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc1ffff640 1 -- 172.21.15.125:0/1990093739 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3000faa0 con 0x7fbc381f2940 2023-10-30T13:20:54.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1990093739 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc38157320 con 0x7fbc381f2940 2023-10-30T13:20:54.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc1ffff640 1 -- 172.21.15.125:0/1990093739 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc3000fe50 con 0x7fbc381f2940 2023-10-30T13:20:54.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1990093739 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc381f2420 con 0x7fbc381f2940 2023-10-30T13:20:54.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.078+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1990093739 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc3804e170 con 0x7fbc381f2940 2023-10-30T13:20:54.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.082+0000 7fbc1ffff640 1 -- 172.21.15.125:0/1990093739 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbc3001f460 con 0x7fbc381f2940 2023-10-30T13:20:54.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.082+0000 7fbc1ffff640 1 --2- 172.21.15.125:0/1990093739 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbc240664a0 0x7fbc24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:54.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.082+0000 7fbc1ffff640 1 -- 172.21.15.125:0/1990093739 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7fbc300876c0 con 0x7fbc381f2940 2023-10-30T13:20:54.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.082+0000 7fbc377fe640 1 --2- 172.21.15.125:0/1990093739 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbc240664a0 0x7fbc24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:54.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.082+0000 7fbc377fe640 1 --2- 172.21.15.125:0/1990093739 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbc240664a0 0x7fbc24068960 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fbc28009ab0 tx=0x7fbc28008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:54.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.086+0000 7fbc1ffff640 1 -- 172.21.15.125:0/1990093739 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbc300541a0 con 0x7fbc381f2940 2023-10-30T13:20:54.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.266+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1990093739 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc380c9d80 con 0x7fbc381f2940 2023-10-30T13:20:54.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.266+0000 7fbc1ffff640 1 -- 172.21.15.125:0/1990093739 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) v1 ==== 95+0+5759 (secure 0 0 0) 0x7fbc300577c0 con 0x7fbc381f2940 2023-10-30T13:20:54.272 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:54.272 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:20:51.725749+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:resolve","state_seq":36,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:54.273 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 242 2023-10-30T13:20:54.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.270+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1990093739 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbc240664a0 msgr2=0x7fbc24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:54.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.270+0000 7fbc3df3e640 1 --2- 172.21.15.125:0/1990093739 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbc240664a0 0x7fbc24068960 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fbc28009ab0 tx=0x7fbc28008040 comp rx=0 tx=0).stop 2023-10-30T13:20:54.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.270+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1990093739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc381f2940 msgr2=0x7fbc381378d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:54.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.270+0000 7fbc3df3e640 1 --2- 172.21.15.125:0/1990093739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc381f2940 0x7fbc381378d0 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fbc30001340 tx=0x7fbc30011c20 comp rx=0 tx=0).stop 2023-10-30T13:20:54.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.270+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1990093739 shutdown_connections 2023-10-30T13:20:54.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.270+0000 7fbc3df3e640 1 --2- 172.21.15.125:0/1990093739 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbc240664a0 0x7fbc24068960 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:54.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.270+0000 7fbc3df3e640 1 --2- 172.21.15.125:0/1990093739 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbc381f2940 0x7fbc381378d0 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:54.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.270+0000 7fbc3df3e640 1 --2- 172.21.15.125:0/1990093739 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbc380c0f80 0x7fbc38139e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:54.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.270+0000 7fbc3df3e640 1 --2- 172.21.15.125:0/1990093739 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbc200047d0 0x7fbc38138380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:54.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.270+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1990093739 >> 172.21.15.125:0/1990093739 conn(0x7fbc380c7910 msgr2=0x7fbc380ca580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:54.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.270+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1990093739 shutdown_connections 2023-10-30T13:20:54.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:54.270+0000 7fbc3df3e640 1 -- 172.21.15.125:0/1990093739 wait complete. 2023-10-30T13:20:54.293 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:56.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:56.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.438+0000 7f668ec17640 1 Processor -- start 2023-10-30T13:20:56.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.438+0000 7f668ec17640 1 -- start start 2023-10-30T13:20:56.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f668ec17640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f66881580e0 0x7f6688070740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:56.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f668ec17640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f668815e700 0x7f668806a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:56.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f668ec17640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66880c5810 con 0x7f66880c6540 2023-10-30T13:20:56.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f668ec17640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66880c6990 con 0x7f668815e700 2023-10-30T13:20:56.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f668ec17640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f668815eb90 con 0x7f66881580e0 2023-10-30T13:20:56.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f668d1b4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f668815e700 0x7f668806a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:56.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f668d1b4640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f668815e700 0x7f668806a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39288/0 (socket says 172.21.15.125:39288) 2023-10-30T13:20:56.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f667ffff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f66880c6540 0x7f6688069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47246/0 (socket says 172.21.15.125:47246) 2023-10-30T13:20:56.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f668d1b4640 1 -- 172.21.15.125:0/53341998 learned_addr learned my addr 172.21.15.125:0/53341998 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:56.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f668c9b3640 1 --2- 172.21.15.125:0/53341998 >> v2:172.21.15.136:3300/0 conn(0x7f66881580e0 0x7f6688070740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:56.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f668d1b4640 1 -- 172.21.15.125:0/53341998 >> v2:172.21.15.136:3300/0 conn(0x7f66881580e0 msgr2=0x7f6688070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:56.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f668d1b4640 1 --2- 172.21.15.125:0/53341998 >> v2:172.21.15.136:3300/0 conn(0x7f66881580e0 0x7f6688070740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:56.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f668d1b4640 1 -- 172.21.15.125:0/53341998 >> v1:172.21.15.125:6789/0 conn(0x7f66880c6540 legacy=0x7f6688069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:56.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f668d1b4640 1 -- 172.21.15.125:0/53341998 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66880c6bb0 con 0x7f668815e700 2023-10-30T13:20:56.448 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.442+0000 7f667f7fe640 1 -- 172.21.15.125:0/53341998 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5172618 0 0) 0x7f66880c5810 con 0x7f66880c6540 2023-10-30T13:20:56.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.446+0000 7f668d1b4640 1 --2- 172.21.15.125:0/53341998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f668815e700 0x7f668806a5a0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f668806eea0 tx=0x7f668001c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c9c826abbfb8777 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:56.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.446+0000 7f667f7fe640 1 -- 172.21.15.125:0/53341998 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66800a5860 con 0x7f668815e700 2023-10-30T13:20:56.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.446+0000 7f667f7fe640 1 -- 172.21.15.125:0/53341998 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66800a59c0 con 0x7f668815e700 2023-10-30T13:20:56.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.446+0000 7f667f7fe640 1 -- 172.21.15.125:0/53341998 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66800a5d50 con 0x7f668815e700 2023-10-30T13:20:56.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.446+0000 7f668ec17640 1 -- 172.21.15.125:0/53341998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f668815e700 msgr2=0x7f668806a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:56.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.446+0000 7f668ec17640 1 --2- 172.21.15.125:0/53341998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f668815e700 0x7f668806a5a0 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f668806eea0 tx=0x7f668001c9a0 comp rx=0 tx=0).stop 2023-10-30T13:20:56.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.446+0000 7f668ec17640 1 -- 172.21.15.125:0/53341998 shutdown_connections 2023-10-30T13:20:56.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.446+0000 7f668ec17640 1 --2- 172.21.15.125:0/53341998 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f668815e700 0x7f668806a5a0 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:56.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.446+0000 7f668ec17640 1 --2- 172.21.15.125:0/53341998 >> v2:172.21.15.136:3300/0 conn(0x7f66881580e0 0x7f6688070740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:56.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.446+0000 7f668ec17640 1 -- 172.21.15.125:0/53341998 >> 172.21.15.125:0/53341998 conn(0x7f66880c7910 msgr2=0x7f66880c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:56.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.446+0000 7f668ec17640 1 -- 172.21.15.125:0/53341998 shutdown_connections 2023-10-30T13:20:56.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.446+0000 7f668ec17640 1 -- 172.21.15.125:0/53341998 wait complete. 2023-10-30T13:20:56.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.446+0000 7f668ec17640 1 Processor -- start 2023-10-30T13:20:56.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.446+0000 7f668ec17640 1 -- start start 2023-10-30T13:20:56.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668ec17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f66880c6540 0x7f668814c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:56.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668ec17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66881580e0 0x7f66881f5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:56.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668ec17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f668815e700 0x7f6688141b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:56.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668ec17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66881f9170 con 0x7f668815e700 2023-10-30T13:20:56.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668ec17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66880b07a0 con 0x7f66881580e0 2023-10-30T13:20:56.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668ec17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f668814e210 con 0x7f66880c6540 2023-10-30T13:20:56.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668d1b4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f668815e700 0x7f6688141b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:56.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668d1b4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f668815e700 0x7f6688141b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39292/0 (socket says 172.21.15.125:39292) 2023-10-30T13:20:56.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668d1b4640 1 -- 172.21.15.125:0/3832976822 learned_addr learned my addr 172.21.15.125:0/3832976822 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:56.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f667ffff640 1 --2- 172.21.15.125:0/3832976822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66881580e0 0x7f66881f5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:56.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668c9b3640 1 --2- 172.21.15.125:0/3832976822 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f66880c6540 0x7f668814c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668d1b4640 1 -- 172.21.15.125:0/3832976822 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f66880c6540 msgr2=0x7f668814c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668d1b4640 1 --2- 172.21.15.125:0/3832976822 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f66880c6540 0x7f668814c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668d1b4640 1 -- 172.21.15.125:0/3832976822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66881580e0 msgr2=0x7f66881f5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668d1b4640 1 --2- 172.21.15.125:0/3832976822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66881580e0 0x7f66881f5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:56.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668d1b4640 1 -- 172.21.15.125:0/3832976822 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6680009b30 con 0x7f668815e700 2023-10-30T13:20:56.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668d1b4640 1 --2- 172.21.15.125:0/3832976822 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f668815e700 0x7f6688141b20 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f66800af040 tx=0x7f6680019740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:56.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f667d7fa640 1 -- 172.21.15.125:0/3832976822 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66800b8680 con 0x7f668815e700 2023-10-30T13:20:56.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f667d7fa640 1 -- 172.21.15.125:0/3832976822 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66800b8800 con 0x7f668815e700 2023-10-30T13:20:56.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f667d7fa640 1 -- 172.21.15.125:0/3832976822 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66800b8bb0 con 0x7f668815e700 2023-10-30T13:20:56.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668ec17640 1 -- 172.21.15.125:0/3832976822 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f668814e4d0 con 0x7f668815e700 2023-10-30T13:20:56.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668ec17640 1 -- 172.21.15.125:0/3832976822 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66881f93c0 con 0x7f668815e700 2023-10-30T13:20:56.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f668ec17640 1 -- 172.21.15.125:0/3832976822 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f668804e170 con 0x7f668815e700 2023-10-30T13:20:56.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.450+0000 7f667d7fa640 1 -- 172.21.15.125:0/3832976822 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f66800b8d10 con 0x7f668815e700 2023-10-30T13:20:56.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.454+0000 7f667d7fa640 1 --2- 172.21.15.125:0/3832976822 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66740664d0 0x7f6674068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:56.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.454+0000 7f667d7fa640 1 -- 172.21.15.125:0/3832976822 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7f668011fef0 con 0x7f668815e700 2023-10-30T13:20:56.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.454+0000 7f668c9b3640 1 --2- 172.21.15.125:0/3832976822 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66740664d0 0x7f6674068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:56.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.454+0000 7f668c9b3640 1 --2- 172.21.15.125:0/3832976822 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66740664d0 0x7f6674068990 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f668400b9d0 tx=0x7f6684008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:56.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.458+0000 7f667d7fa640 1 -- 172.21.15.125:0/3832976822 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f66800ec9a0 con 0x7f668815e700 2023-10-30T13:20:56.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.638+0000 7f668ec17640 1 -- 172.21.15.125:0/3832976822 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66880c5290 con 0x7f668815e700 2023-10-30T13:20:56.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.638+0000 7f667d7fa640 1 -- 172.21.15.125:0/3832976822 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 242 v242) v1 ==== 95+0+5759 (secure 0 0 0) 0x7f66800bc090 con 0x7f668815e700 2023-10-30T13:20:56.644 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:56.644 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":242,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":242,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:20:51.725749+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:resolve","state_seq":36,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:56.644 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 242 2023-10-30T13:20:56.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.642+0000 7f668ec17640 1 -- 172.21.15.125:0/3832976822 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66740664d0 msgr2=0x7f6674068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:56.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.642+0000 7f668ec17640 1 --2- 172.21.15.125:0/3832976822 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66740664d0 0x7f6674068990 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f668400b9d0 tx=0x7f6684008040 comp rx=0 tx=0).stop 2023-10-30T13:20:56.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.642+0000 7f668ec17640 1 -- 172.21.15.125:0/3832976822 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f668815e700 msgr2=0x7f6688141b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:56.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.642+0000 7f668ec17640 1 --2- 172.21.15.125:0/3832976822 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f668815e700 0x7f6688141b20 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f66800af040 tx=0x7f6680019740 comp rx=0 tx=0).stop 2023-10-30T13:20:56.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.642+0000 7f668ec17640 1 -- 172.21.15.125:0/3832976822 shutdown_connections 2023-10-30T13:20:56.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.642+0000 7f668ec17640 1 --2- 172.21.15.125:0/3832976822 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66740664d0 0x7f6674068990 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:56.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.642+0000 7f668ec17640 1 --2- 172.21.15.125:0/3832976822 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f668815e700 0x7f6688141b20 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:56.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.642+0000 7f668ec17640 1 --2- 172.21.15.125:0/3832976822 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f66881580e0 0x7f66881f5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:56.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.642+0000 7f668ec17640 1 --2- 172.21.15.125:0/3832976822 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f66880c6540 0x7f668814c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:56.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.642+0000 7f668ec17640 1 -- 172.21.15.125:0/3832976822 >> 172.21.15.125:0/3832976822 conn(0x7f66880c7910 msgr2=0x7f6688071f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:56.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.642+0000 7f668ec17640 1 -- 172.21.15.125:0/3832976822 shutdown_connections 2023-10-30T13:20:56.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:56.642+0000 7f668ec17640 1 -- 172.21.15.125:0/3832976822 wait complete. 2023-10-30T13:20:56.665 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:20:58.666 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:20:58.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.806+0000 7f121a0c9640 1 Processor -- start 2023-10-30T13:20:58.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.806+0000 7f121a0c9640 1 -- start start 2023-10-30T13:20:58.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.806+0000 7f121a0c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121406f7a0 0x7f121406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:58.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.806+0000 7f121a0c9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1214071110 0x7f1214158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:58.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.806+0000 7f121a0c9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1214066cf0 con 0x7f12140c6bd0 2023-10-30T13:20:58.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.806+0000 7f121a0c9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12140700c0 con 0x7f1214071110 2023-10-30T13:20:58.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.806+0000 7f121a0c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12140c70d0 con 0x7f121406f7a0 2023-10-30T13:20:58.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.810+0000 7f1213fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1214071110 0x7f1214158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:58.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.810+0000 7f12137fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121406f7a0 0x7f121406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:58.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.810+0000 7f12137fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121406f7a0 0x7f121406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:46490/0 (socket says 172.21.15.125:46490) 2023-10-30T13:20:58.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.810+0000 7f12137fe640 1 -- 172.21.15.125:0/754957164 learned_addr learned my addr 172.21.15.125:0/754957164 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:58.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.810+0000 7f120bfff640 1 --1- 172.21.15.125:0/754957164 >> v1:172.21.15.136:6789/0 conn(0x7f12140c6bd0 0x7f1214069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:41892/0 (socket says 172.21.15.125:41892) 2023-10-30T13:20:58.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.810+0000 7f12137fe640 1 -- 172.21.15.125:0/754957164 >> v2:172.21.15.136:3300/0 conn(0x7f1214071110 msgr2=0x7f1214158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:20:58.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.810+0000 7f12137fe640 1 --2- 172.21.15.125:0/754957164 >> v2:172.21.15.136:3300/0 conn(0x7f1214071110 0x7f1214158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:58.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.810+0000 7f12137fe640 1 -- 172.21.15.125:0/754957164 >> v1:172.21.15.136:6789/0 conn(0x7f12140c6bd0 legacy=0x7f1214069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:58.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.810+0000 7f12137fe640 1 -- 172.21.15.125:0/754957164 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12140c7390 con 0x7f121406f7a0 2023-10-30T13:20:58.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.814+0000 7f12137fe640 1 --2- 172.21.15.125:0/754957164 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121406f7a0 0x7f121406fb80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1204009870 tx=0x7f120401ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=699b016f588beba7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:58.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.814+0000 7f1212ffd640 1 -- 172.21.15.125:0/754957164 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12040a8a30 con 0x7f121406f7a0 2023-10-30T13:20:58.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.814+0000 7f1212ffd640 1 -- 172.21.15.125:0/754957164 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12040a8b90 con 0x7f121406f7a0 2023-10-30T13:20:58.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.814+0000 7f1212ffd640 1 -- 172.21.15.125:0/754957164 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12040b0440 con 0x7f121406f7a0 2023-10-30T13:20:58.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.814+0000 7f121a0c9640 1 -- 172.21.15.125:0/754957164 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121406f7a0 msgr2=0x7f121406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:58.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.814+0000 7f121a0c9640 1 --2- 172.21.15.125:0/754957164 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121406f7a0 0x7f121406fb80 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1204009870 tx=0x7f120401ed30 comp rx=0 tx=0).stop 2023-10-30T13:20:58.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.814+0000 7f121a0c9640 1 -- 172.21.15.125:0/754957164 shutdown_connections 2023-10-30T13:20:58.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.814+0000 7f121a0c9640 1 --2- 172.21.15.125:0/754957164 >> v2:172.21.15.136:3300/0 conn(0x7f1214071110 0x7f1214158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:58.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.814+0000 7f121a0c9640 1 --2- 172.21.15.125:0/754957164 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121406f7a0 0x7f121406fb80 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:58.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.814+0000 7f121a0c9640 1 -- 172.21.15.125:0/754957164 >> 172.21.15.125:0/754957164 conn(0x7f12140c7910 msgr2=0x7f12140c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:58.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.814+0000 7f121a0c9640 1 -- 172.21.15.125:0/754957164 shutdown_connections 2023-10-30T13:20:58.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.814+0000 7f121a0c9640 1 -- 172.21.15.125:0/754957164 wait complete. 2023-10-30T13:20:58.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.814+0000 7f121a0c9640 1 Processor -- start 2023-10-30T13:20:58.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.814+0000 7f121a0c9640 1 -- start start 2023-10-30T13:20:58.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f121a0c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121406f7a0 0x7f12140c38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:58.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f121a0c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1214071110 0x7f12141ef260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:58.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f121a0c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12140c6bd0 0x7f121414a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:58.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f121a0c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12141f3020 con 0x7f12140c6bd0 2023-10-30T13:20:58.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f121a0c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12140bfec0 con 0x7f1214071110 2023-10-30T13:20:58.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f121a0c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12141eeb30 con 0x7f121406f7a0 2023-10-30T13:20:58.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f1213fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12140c6bd0 0x7f121414a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:58.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f1213fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12140c6bd0 0x7f121414a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39306/0 (socket says 172.21.15.125:39306) 2023-10-30T13:20:58.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f1213fff640 1 -- 172.21.15.125:0/1891616499 learned_addr learned my addr 172.21.15.125:0/1891616499 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:20:58.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f120bfff640 1 --2- 172.21.15.125:0/1891616499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1214071110 0x7f12141ef260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:58.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f12137fe640 1 --2- 172.21.15.125:0/1891616499 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121406f7a0 0x7f12140c38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:58.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f1213fff640 1 -- 172.21.15.125:0/1891616499 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121406f7a0 msgr2=0x7f12140c38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:58.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f1213fff640 1 --2- 172.21.15.125:0/1891616499 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121406f7a0 0x7f12140c38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:58.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f1213fff640 1 -- 172.21.15.125:0/1891616499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1214071110 msgr2=0x7f12141ef260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:58.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f1213fff640 1 --2- 172.21.15.125:0/1891616499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1214071110 0x7f12141ef260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:58.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f1213fff640 1 -- 172.21.15.125:0/1891616499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12040094d0 con 0x7f12140c6bd0 2023-10-30T13:20:58.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f1213fff640 1 --2- 172.21.15.125:0/1891616499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12140c6bd0 0x7f121414a6b0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f120c009d30 tx=0x7f120c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:58.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f1210ff9640 1 -- 172.21.15.125:0/1891616499 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c009ba0 con 0x7f12140c6bd0 2023-10-30T13:20:58.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f1210ff9640 1 -- 172.21.15.125:0/1891616499 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f120c00f040 con 0x7f12140c6bd0 2023-10-30T13:20:58.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f121a0c9640 1 -- 172.21.15.125:0/1891616499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12141f24c0 con 0x7f12140c6bd0 2023-10-30T13:20:58.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f1210ff9640 1 -- 172.21.15.125:0/1891616499 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f120c0146c0 con 0x7f12140c6bd0 2023-10-30T13:20:58.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f121a0c9640 1 -- 172.21.15.125:0/1891616499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12141f2a60 con 0x7f12140c6bd0 2023-10-30T13:20:58.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f121a0c9640 1 -- 172.21.15.125:0/1891616499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f121404e170 con 0x7f12140c6bd0 2023-10-30T13:20:58.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.818+0000 7f1210ff9640 1 -- 172.21.15.125:0/1891616499 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f120c014820 con 0x7f12140c6bd0 2023-10-30T13:20:58.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.822+0000 7f1210ff9640 1 --2- 172.21.15.125:0/1891616499 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11f40664d0 0x7f11f4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:20:58.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.822+0000 7f1210ff9640 1 -- 172.21.15.125:0/1891616499 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7f120c085260 con 0x7f12140c6bd0 2023-10-30T13:20:58.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.822+0000 7f12137fe640 1 --2- 172.21.15.125:0/1891616499 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11f40664d0 0x7f11f4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:20:58.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.822+0000 7f12137fe640 1 --2- 172.21.15.125:0/1891616499 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11f40664d0 0x7f11f4068990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f12040b2000 tx=0x7f1204009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:20:58.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:58.826+0000 7f1210ff9640 1 -- 172.21.15.125:0/1891616499 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f120c051cb0 con 0x7f12140c6bd0 2023-10-30T13:20:59.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:59.006+0000 7f121a0c9640 1 -- 172.21.15.125:0/1891616499 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12140c3160 con 0x7f12140c6bd0 2023-10-30T13:20:59.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:59.006+0000 7f1210ff9640 1 -- 172.21.15.125:0/1891616499 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+5761 (secure 0 0 0) 0x7f120c018090 con 0x7f12140c6bd0 2023-10-30T13:20:59.011 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:20:59.011 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:20:57.296538+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:20:59.011 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 243 2023-10-30T13:20:59.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:59.010+0000 7f121a0c9640 1 -- 172.21.15.125:0/1891616499 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11f40664d0 msgr2=0x7f11f4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:59.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:59.010+0000 7f121a0c9640 1 --2- 172.21.15.125:0/1891616499 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11f40664d0 0x7f11f4068990 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f12040b2000 tx=0x7f1204009040 comp rx=0 tx=0).stop 2023-10-30T13:20:59.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:59.010+0000 7f121a0c9640 1 -- 172.21.15.125:0/1891616499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12140c6bd0 msgr2=0x7f121414a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:20:59.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:59.010+0000 7f121a0c9640 1 --2- 172.21.15.125:0/1891616499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12140c6bd0 0x7f121414a6b0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f120c009d30 tx=0x7f120c00c490 comp rx=0 tx=0).stop 2023-10-30T13:20:59.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:59.010+0000 7f121a0c9640 1 -- 172.21.15.125:0/1891616499 shutdown_connections 2023-10-30T13:20:59.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:59.010+0000 7f121a0c9640 1 --2- 172.21.15.125:0/1891616499 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11f40664d0 0x7f11f4068990 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:59.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:59.010+0000 7f121a0c9640 1 --2- 172.21.15.125:0/1891616499 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f12140c6bd0 0x7f121414a6b0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:59.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:59.010+0000 7f121a0c9640 1 --2- 172.21.15.125:0/1891616499 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1214071110 0x7f12141ef260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:59.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:59.010+0000 7f121a0c9640 1 --2- 172.21.15.125:0/1891616499 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f121406f7a0 0x7f12140c38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:20:59.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:59.010+0000 7f121a0c9640 1 -- 172.21.15.125:0/1891616499 >> 172.21.15.125:0/1891616499 conn(0x7f12140c7910 msgr2=0x7f1214156f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:20:59.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:59.010+0000 7f121a0c9640 1 -- 172.21.15.125:0/1891616499 shutdown_connections 2023-10-30T13:20:59.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:20:59.010+0000 7f121a0c9640 1 -- 172.21.15.125:0/1891616499 wait complete. 2023-10-30T13:20:59.032 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 243, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 243, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_8971": { "addr": "172.21.15.136:6839/4284345684", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 4284345684, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 4284345684, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 1, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 8971, "incarnation": 230, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:active", "state_seq": 32 }, "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 36 }, "gid_9600": { "addr": "172.21.15.125:6836/2077416857", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2077416857, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2077416857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9600, "incarnation": 241, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:reconnect", "state_seq": 38 } }, "last_failure": 0, "last_failure_osd_epoch": 83, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:20:57.296538+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8971, "mds_1": 9150, "mds_2": 9600 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:21:01.033 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:01.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.170+0000 7f3e97b1d640 1 Processor -- start 2023-10-30T13:21:01.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.170+0000 7f3e97b1d640 1 -- start start 2023-10-30T13:21:01.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.170+0000 7f3e97b1d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e90153bf0 0x7f3e90071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:01.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.170+0000 7f3e97b1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3e900c6fe0 0x7f3e900c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:01.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.170+0000 7f3e97b1d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e900700b0 con 0x7f3e900c79e0 2023-10-30T13:21:01.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.170+0000 7f3e97b1d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e900c6810 con 0x7f3e90153bf0 2023-10-30T13:21:01.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.170+0000 7f3e97b1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e90159900 con 0x7f3e900c6fe0 2023-10-30T13:21:01.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.174+0000 7f3e958b9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e90153bf0 0x7f3e90071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:01.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.174+0000 7f3e958b9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3e90153bf0 0x7f3e90071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42396/0 (socket says 172.21.15.125:42396) 2023-10-30T13:21:01.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.174+0000 7f3e958b9640 1 -- 172.21.15.125:0/726282308 learned_addr learned my addr 172.21.15.125:0/726282308 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:01.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.174+0000 7f3e950b8640 1 --2- 172.21.15.125:0/726282308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3e900c6fe0 0x7f3e900c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:01.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.174+0000 7f3e958b9640 1 -- 172.21.15.125:0/726282308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3e900c6fe0 msgr2=0x7f3e900c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:01.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.174+0000 7f3e958b9640 1 --2- 172.21.15.125:0/726282308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3e900c6fe0 0x7f3e900c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:01.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.174+0000 7f3e958b9640 1 -- 172.21.15.125:0/726282308 >> v1:172.21.15.136:6789/0 conn(0x7f3e900c79e0 legacy=0x7f3e900cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:01.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.174+0000 7f3e958b9640 1 -- 172.21.15.125:0/726282308 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e900c6530 con 0x7f3e90153bf0 2023-10-30T13:21:01.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.174+0000 7f3e958b9640 1 --2- 172.21.15.125:0/726282308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e90153bf0 0x7f3e90071260 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009870 tx=0x7f3e8c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f5f657ce283b58f server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:01.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.174+0000 7f3e948b7640 1 -- 172.21.15.125:0/726282308 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0a8a30 con 0x7f3e90153bf0 2023-10-30T13:21:01.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.174+0000 7f3e948b7640 1 -- 172.21.15.125:0/726282308 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8c0a8b90 con 0x7f3e90153bf0 2023-10-30T13:21:01.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.174+0000 7f3e948b7640 1 -- 172.21.15.125:0/726282308 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0b0440 con 0x7f3e90153bf0 2023-10-30T13:21:01.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 -- 172.21.15.125:0/726282308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e90153bf0 msgr2=0x7f3e90071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:01.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 --2- 172.21.15.125:0/726282308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e90153bf0 0x7f3e90071260 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c009870 tx=0x7f3e8c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:21:01.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 -- 172.21.15.125:0/726282308 shutdown_connections 2023-10-30T13:21:01.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 --2- 172.21.15.125:0/726282308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3e900c6fe0 0x7f3e900c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:01.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 --2- 172.21.15.125:0/726282308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e90153bf0 0x7f3e90071260 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:01.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 -- 172.21.15.125:0/726282308 >> 172.21.15.125:0/726282308 conn(0x7f3e9006c6e0 msgr2=0x7f3e90069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:01.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 -- 172.21.15.125:0/726282308 shutdown_connections 2023-10-30T13:21:01.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 -- 172.21.15.125:0/726282308 wait complete. 2023-10-30T13:21:01.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 Processor -- start 2023-10-30T13:21:01.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 -- start start 2023-10-30T13:21:01.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e900c6fe0 0x7f3e901eedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:01.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3e900c79e0 0x7f3e901ef2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:01.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e90153bf0 0x7f3e901552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:01.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e90154bc0 con 0x7f3e900c6fe0 2023-10-30T13:21:01.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e901ef940 con 0x7f3e90153bf0 2023-10-30T13:21:01.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e97b1d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e90154e60 con 0x7f3e900c79e0 2023-10-30T13:21:01.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e958b9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e900c6fe0 0x7f3e901eedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:01.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e958b9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e900c6fe0 0x7f3e901eedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42412/0 (socket says 172.21.15.125:42412) 2023-10-30T13:21:01.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e958b9640 1 -- 172.21.15.125:0/1256894353 learned_addr learned my addr 172.21.15.125:0/1256894353 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:01.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e950b8640 1 --2- 172.21.15.125:0/1256894353 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3e900c79e0 0x7f3e901ef2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:01.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e960ba640 1 --2- 172.21.15.125:0/1256894353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e90153bf0 0x7f3e901552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:01.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e958b9640 1 -- 172.21.15.125:0/1256894353 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3e900c79e0 msgr2=0x7f3e901ef2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:01.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e958b9640 1 --2- 172.21.15.125:0/1256894353 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3e900c79e0 0x7f3e901ef2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:01.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e958b9640 1 -- 172.21.15.125:0/1256894353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e90153bf0 msgr2=0x7f3e901552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:01.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e958b9640 1 --2- 172.21.15.125:0/1256894353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e90153bf0 0x7f3e901552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:01.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.178+0000 7f3e958b9640 1 -- 172.21.15.125:0/1256894353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e8c0094d0 con 0x7f3e900c6fe0 2023-10-30T13:21:01.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.182+0000 7f3e958b9640 1 --2- 172.21.15.125:0/1256894353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e900c6fe0 0x7f3e901eedb0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0b3040 tx=0x7f3e8c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:01.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.182+0000 7f3e867fc640 1 -- 172.21.15.125:0/1256894353 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0be070 con 0x7f3e900c6fe0 2023-10-30T13:21:01.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.182+0000 7f3e867fc640 1 -- 172.21.15.125:0/1256894353 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e8c0ba400 con 0x7f3e900c6fe0 2023-10-30T13:21:01.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.182+0000 7f3e97b1d640 1 -- 172.21.15.125:0/1256894353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9014d230 con 0x7f3e900c6fe0 2023-10-30T13:21:01.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.182+0000 7f3e867fc640 1 -- 172.21.15.125:0/1256894353 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0ba7d0 con 0x7f3e900c6fe0 2023-10-30T13:21:01.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.182+0000 7f3e97b1d640 1 -- 172.21.15.125:0/1256894353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9014d7f0 con 0x7f3e900c6fe0 2023-10-30T13:21:01.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.182+0000 7f3e97b1d640 1 -- 172.21.15.125:0/1256894353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e9004e170 con 0x7f3e900c6fe0 2023-10-30T13:21:01.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.182+0000 7f3e867fc640 1 -- 172.21.15.125:0/1256894353 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3e8c0c3410 con 0x7f3e900c6fe0 2023-10-30T13:21:01.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.182+0000 7f3e867fc640 1 --2- 172.21.15.125:0/1256894353 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3e740664a0 0x7f3e74068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:01.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.182+0000 7f3e867fc640 1 -- 172.21.15.125:0/1256894353 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7f3e8c122460 con 0x7f3e900c6fe0 2023-10-30T13:21:01.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.182+0000 7f3e950b8640 1 --2- 172.21.15.125:0/1256894353 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3e740664a0 0x7f3e74068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:01.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.186+0000 7f3e950b8640 1 --2- 172.21.15.125:0/1256894353 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3e740664a0 0x7f3e74068960 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3e90157a20 tx=0x7f3e80008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:01.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.186+0000 7f3e867fc640 1 -- 172.21.15.125:0/1256894353 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3e8c0eee50 con 0x7f3e900c6fe0 2023-10-30T13:21:01.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.366+0000 7f3e97b1d640 1 -- 172.21.15.125:0/1256894353 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e9006fe90 con 0x7f3e900c6fe0 2023-10-30T13:21:01.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.366+0000 7f3e867fc640 1 -- 172.21.15.125:0/1256894353 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 243 v243) v1 ==== 95+0+5761 (secure 0 0 0) 0x7f3e8c0f2470 con 0x7f3e900c6fe0 2023-10-30T13:21:01.371 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:01.371 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":243,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":243,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:20:57.296538+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:reconnect","state_seq":38,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:01.371 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 243 2023-10-30T13:21:01.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.366+0000 7f3e97b1d640 1 -- 172.21.15.125:0/1256894353 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3e740664a0 msgr2=0x7f3e74068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:01.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.366+0000 7f3e97b1d640 1 --2- 172.21.15.125:0/1256894353 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3e740664a0 0x7f3e74068960 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f3e90157a20 tx=0x7f3e80008040 comp rx=0 tx=0).stop 2023-10-30T13:21:01.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.366+0000 7f3e97b1d640 1 -- 172.21.15.125:0/1256894353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e900c6fe0 msgr2=0x7f3e901eedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:01.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.366+0000 7f3e97b1d640 1 --2- 172.21.15.125:0/1256894353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e900c6fe0 0x7f3e901eedb0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c0b3040 tx=0x7f3e8c01bad0 comp rx=0 tx=0).stop 2023-10-30T13:21:01.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.370+0000 7f3e97b1d640 1 -- 172.21.15.125:0/1256894353 shutdown_connections 2023-10-30T13:21:01.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.370+0000 7f3e97b1d640 1 --2- 172.21.15.125:0/1256894353 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3e740664a0 0x7f3e74068960 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:01.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.370+0000 7f3e97b1d640 1 --2- 172.21.15.125:0/1256894353 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3e90153bf0 0x7f3e901552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:01.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.370+0000 7f3e97b1d640 1 --2- 172.21.15.125:0/1256894353 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3e900c79e0 0x7f3e901ef2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:01.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.370+0000 7f3e97b1d640 1 --2- 172.21.15.125:0/1256894353 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3e900c6fe0 0x7f3e901eedb0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:01.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.370+0000 7f3e97b1d640 1 -- 172.21.15.125:0/1256894353 >> 172.21.15.125:0/1256894353 conn(0x7f3e9006c6e0 msgr2=0x7f3e9006e370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:01.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.370+0000 7f3e97b1d640 1 -- 172.21.15.125:0/1256894353 shutdown_connections 2023-10-30T13:21:01.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:01.370+0000 7f3e97b1d640 1 -- 172.21.15.125:0/1256894353 wait complete. 2023-10-30T13:21:01.392 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:21:03.393 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:03.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.534+0000 7f88bc254640 1 Processor -- start 2023-10-30T13:21:03.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.534+0000 7f88bc254640 1 -- start start 2023-10-30T13:21:03.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88bc254640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f88b40c05d0 0x7f88b413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:03.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88bc254640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f88b40c6f60 0x7f88b413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:03.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88bc254640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b40c0a50 con 0x7f88b40c6b80 2023-10-30T13:21:03.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88bc254640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88b406f6e0 con 0x7f88b40c6f60 2023-10-30T13:21:03.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88bc254640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b406f9c0 con 0x7f88b40c05d0 2023-10-30T13:21:03.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88ba7f1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f88b40c6f60 0x7f88b413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:03.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88b97ef640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f88b40c05d0 0x7f88b413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:03.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88ba7f1640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f88b40c6f60 0x7f88b413f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:47982/0 (socket says 172.21.15.125:47982) 2023-10-30T13:21:03.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88ba7f1640 1 -- 172.21.15.125:0/2525476528 learned_addr learned my addr 172.21.15.125:0/2525476528 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:03.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88b9ff0640 1 --1- 172.21.15.125:0/2525476528 >> v1:172.21.15.136:6789/0 conn(0x7f88b40c6b80 0x7f88b40bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:35322/0 (socket says 172.21.15.125:35322) 2023-10-30T13:21:03.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88b97ef640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f88b40c05d0 0x7f88b413ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:59218/0 (socket says 172.21.15.125:59218) 2023-10-30T13:21:03.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88b97ef640 1 -- 172.21.15.125:0/2525476528 >> v2:172.21.15.136:3300/0 conn(0x7f88b40c6f60 msgr2=0x7f88b413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:03.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88b97ef640 1 --2- 172.21.15.125:0/2525476528 >> v2:172.21.15.136:3300/0 conn(0x7f88b40c6f60 0x7f88b413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:03.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88b97ef640 1 -- 172.21.15.125:0/2525476528 >> v1:172.21.15.136:6789/0 conn(0x7f88b40c6b80 legacy=0x7f88b40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:03.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.538+0000 7f88b97ef640 1 -- 172.21.15.125:0/2525476528 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b406fca0 con 0x7f88b40c05d0 2023-10-30T13:21:03.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.542+0000 7f88b97ef640 1 --2- 172.21.15.125:0/2525476528 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f88b40c05d0 0x7f88b413ed90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f88a40097c0 tx=0x7f88a401c480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=839a0e70dfee108d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:03.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.542+0000 7f88b8fee640 1 -- 172.21.15.125:0/2525476528 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a40ad640 con 0x7f88b40c05d0 2023-10-30T13:21:03.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.542+0000 7f88b8fee640 1 -- 172.21.15.125:0/2525476528 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a40ad7a0 con 0x7f88b40c05d0 2023-10-30T13:21:03.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.542+0000 7f88b8fee640 1 -- 172.21.15.125:0/2525476528 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a40adb30 con 0x7f88b40c05d0 2023-10-30T13:21:03.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.542+0000 7f88bc254640 1 -- 172.21.15.125:0/2525476528 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f88b40c05d0 msgr2=0x7f88b413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:03.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.542+0000 7f88bc254640 1 --2- 172.21.15.125:0/2525476528 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f88b40c05d0 0x7f88b413ed90 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f88a40097c0 tx=0x7f88a401c480 comp rx=0 tx=0).stop 2023-10-30T13:21:03.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.542+0000 7f88bc254640 1 -- 172.21.15.125:0/2525476528 shutdown_connections 2023-10-30T13:21:03.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.542+0000 7f88bc254640 1 --2- 172.21.15.125:0/2525476528 >> v2:172.21.15.136:3300/0 conn(0x7f88b40c6f60 0x7f88b413f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:03.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.542+0000 7f88bc254640 1 --2- 172.21.15.125:0/2525476528 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f88b40c05d0 0x7f88b413ed90 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:03.547 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.542+0000 7f88bc254640 1 -- 172.21.15.125:0/2525476528 >> 172.21.15.125:0/2525476528 conn(0x7f88b40c7910 msgr2=0x7f88b40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:03.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.542+0000 7f88bc254640 1 -- 172.21.15.125:0/2525476528 shutdown_connections 2023-10-30T13:21:03.548 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.542+0000 7f88bc254640 1 -- 172.21.15.125:0/2525476528 wait complete. 2023-10-30T13:21:03.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.542+0000 7f88bc254640 1 Processor -- start 2023-10-30T13:21:03.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.542+0000 7f88bc254640 1 -- start start 2023-10-30T13:21:03.550 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88bc254640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f88b40c05d0 0x7f88b4154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:03.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88bc254640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88b40c6b80 0x7f88b41539a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:03.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88bc254640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88b40c6f60 0x7f88b41388b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:03.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88bc254640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4155ae0 con 0x7f88b40c6b80 2023-10-30T13:21:03.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88bc254640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88b4154a30 con 0x7f88b40c6f60 2023-10-30T13:21:03.551 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88bc254640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88b41f3c20 con 0x7f88b40c05d0 2023-10-30T13:21:03.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88b97ef640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88b40c6b80 0x7f88b41539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:03.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88b97ef640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88b40c6b80 0x7f88b41539a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42414/0 (socket says 172.21.15.125:42414) 2023-10-30T13:21:03.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88b97ef640 1 -- 172.21.15.125:0/695302184 learned_addr learned my addr 172.21.15.125:0/695302184 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:03.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88b9ff0640 1 --2- 172.21.15.125:0/695302184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f88b40c05d0 0x7f88b4154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:03.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88ba7f1640 1 --2- 172.21.15.125:0/695302184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88b40c6f60 0x7f88b41388b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:03.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88b97ef640 1 -- 172.21.15.125:0/695302184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f88b40c05d0 msgr2=0x7f88b4154ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:03.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88b97ef640 1 --2- 172.21.15.125:0/695302184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f88b40c05d0 0x7f88b4154ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:03.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88b97ef640 1 -- 172.21.15.125:0/695302184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88b40c6f60 msgr2=0x7f88b41388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:03.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88b97ef640 1 --2- 172.21.15.125:0/695302184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88b40c6f60 0x7f88b41388b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:03.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88b97ef640 1 -- 172.21.15.125:0/695302184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88a4009420 con 0x7f88b40c6b80 2023-10-30T13:21:03.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88b97ef640 1 --2- 172.21.15.125:0/695302184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88b40c6b80 0x7f88b41539a0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f88a401a0b0 tx=0x7f88a4019220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:03.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88aa7fc640 1 -- 172.21.15.125:0/695302184 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a40b6900 con 0x7f88b40c6b80 2023-10-30T13:21:03.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88aa7fc640 1 -- 172.21.15.125:0/695302184 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88a40a6040 con 0x7f88b40c6b80 2023-10-30T13:21:03.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88aa7fc640 1 -- 172.21.15.125:0/695302184 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88a40b6cc0 con 0x7f88b40c6b80 2023-10-30T13:21:03.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88bc254640 1 -- 172.21.15.125:0/695302184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b41f3f00 con 0x7f88b40c6b80 2023-10-30T13:21:03.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88bc254640 1 -- 172.21.15.125:0/695302184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b41546c0 con 0x7f88b40c6b80 2023-10-30T13:21:03.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88bc254640 1 -- 172.21.15.125:0/695302184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b404e170 con 0x7f88b40c6b80 2023-10-30T13:21:03.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.546+0000 7f88aa7fc640 1 -- 172.21.15.125:0/695302184 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f88a40b4040 con 0x7f88b40c6b80 2023-10-30T13:21:03.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.550+0000 7f88aa7fc640 1 --2- 172.21.15.125:0/695302184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f88980664d0 0x7f8898068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:03.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.550+0000 7f88aa7fc640 1 -- 172.21.15.125:0/695302184 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7f88a411ed20 con 0x7f88b40c6b80 2023-10-30T13:21:03.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.550+0000 7f88b9ff0640 1 --2- 172.21.15.125:0/695302184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f88980664d0 0x7f8898068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:03.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.550+0000 7f88b9ff0640 1 --2- 172.21.15.125:0/695302184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f88980664d0 0x7f8898068990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f88b0000f70 tx=0x7f88b0007e50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:03.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.554+0000 7f88aa7fc640 1 -- 172.21.15.125:0/695302184 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f88a40eb830 con 0x7f88b40c6b80 2023-10-30T13:21:03.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.734+0000 7f88bc254640 1 -- 172.21.15.125:0/695302184 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88b4154b70 con 0x7f88b40c6b80 2023-10-30T13:21:03.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.734+0000 7f88aa7fc640 1 -- 172.21.15.125:0/695302184 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f88a40b2030 con 0x7f88b40c6b80 2023-10-30T13:21:03.738 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:03.738 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":244,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:02.907092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:03.739 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 244 2023-10-30T13:21:03.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.734+0000 7f88bc254640 1 -- 172.21.15.125:0/695302184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f88980664d0 msgr2=0x7f8898068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:03.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.734+0000 7f88bc254640 1 --2- 172.21.15.125:0/695302184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f88980664d0 0x7f8898068990 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f88b0000f70 tx=0x7f88b0007e50 comp rx=0 tx=0).stop 2023-10-30T13:21:03.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.734+0000 7f88bc254640 1 -- 172.21.15.125:0/695302184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88b40c6b80 msgr2=0x7f88b41539a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:03.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.734+0000 7f88bc254640 1 --2- 172.21.15.125:0/695302184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88b40c6b80 0x7f88b41539a0 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7f88a401a0b0 tx=0x7f88a4019220 comp rx=0 tx=0).stop 2023-10-30T13:21:03.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.734+0000 7f88bc254640 1 -- 172.21.15.125:0/695302184 shutdown_connections 2023-10-30T13:21:03.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.734+0000 7f88bc254640 1 --2- 172.21.15.125:0/695302184 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f88980664d0 0x7f8898068990 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:03.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.734+0000 7f88bc254640 1 --2- 172.21.15.125:0/695302184 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f88b40c6f60 0x7f88b41388b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:03.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.734+0000 7f88bc254640 1 --2- 172.21.15.125:0/695302184 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f88b40c6b80 0x7f88b41539a0 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:03.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.734+0000 7f88bc254640 1 --2- 172.21.15.125:0/695302184 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f88b40c05d0 0x7f88b4154ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:03.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.734+0000 7f88bc254640 1 -- 172.21.15.125:0/695302184 >> 172.21.15.125:0/695302184 conn(0x7f88b40c7910 msgr2=0x7f88b40c0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:03.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.734+0000 7f88bc254640 1 -- 172.21.15.125:0/695302184 shutdown_connections 2023-10-30T13:21:03.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:03.734+0000 7f88bc254640 1 -- 172.21.15.125:0/695302184 wait complete. 2023-10-30T13:21:03.759 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:21:05.761 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:05.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.902+0000 7f1bcba46640 1 Processor -- start 2023-10-30T13:21:05.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.902+0000 7f1bcba46640 1 -- start start 2023-10-30T13:21:05.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bcba46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bc40c05d0 0x7f1bc413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:05.909 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bcba46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc40c0a50 con 0x7f1bc40c6f60 2023-10-30T13:21:05.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bcba46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc406f6e0 con 0x7f1bc40c6b80 2023-10-30T13:21:05.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bcba46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc406f9c0 con 0x7f1bc40c05d0 2023-10-30T13:21:05.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bc9fe3640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1bc40c6f60 0x7f1bc41566a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47228/0 (socket says 172.21.15.125:47228) 2023-10-30T13:21:05.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bc9fe3640 1 -- 172.21.15.125:0/2741213116 learned_addr learned my addr 172.21.15.125:0/2741213116 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:21:05.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bc8fe1640 1 --2- 172.21.15.125:0/2741213116 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bc40c05d0 0x7f1bc413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:05.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 -- 172.21.15.125:0/2741213116 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339572682 0 0) 0x7f1bc40c0a50 con 0x7f1bc40c6f60 2023-10-30T13:21:05.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 -- 172.21.15.125:0/2741213116 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb0005740 con 0x7f1bc40c6f60 2023-10-30T13:21:05.912 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 -- 172.21.15.125:0/2741213116 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2089493797 0 0) 0x7f1bb0005740 con 0x7f1bc40c6f60 2023-10-30T13:21:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 -- 172.21.15.125:0/2741213116 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bc40c05d0 msgr2=0x7f1bc413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 --2- 172.21.15.125:0/2741213116 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bc40c05d0 0x7f1bc413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 -- 172.21.15.125:0/2741213116 >> v1:172.21.15.136:6789/0 conn(0x7f1bc40c6b80 legacy=0x7f1bc40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 -- 172.21.15.125:0/2741213116 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc406fca0 con 0x7f1bc40c6f60 2023-10-30T13:21:05.913 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 -- 172.21.15.125:0/2741213116 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f1bbc001a00 con 0x7f1bc40c6f60 2023-10-30T13:21:05.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 -- 172.21.15.125:0/2741213116 >> v1:172.21.15.125:6789/0 conn(0x7f1bc40c6f60 legacy=0x7f1bc41566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:05.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 --2- 172.21.15.125:0/2741213116 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bb0005bc0 0x7f1bb0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:05.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 --2- 172.21.15.125:0/2741213116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bb0006520 0x7f1bb0006a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:05.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 --2- 172.21.15.125:0/2741213116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bb0007200 0x7f1bb000dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:05.914 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 -- 172.21.15.125:0/2741213116 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc406f9c0 con 0x7f1bb0007200 2023-10-30T13:21:05.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 -- 172.21.15.125:0/2741213116 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb000e110 con 0x7f1bb0006520 2023-10-30T13:21:05.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bbbfff640 1 -- 172.21.15.125:0/2741213116 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bb000e2d0 con 0x7f1bb0005bc0 2023-10-30T13:21:05.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bc8fe1640 1 --2- 172.21.15.125:0/2741213116 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bc40c05d0 0x7f1bc413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:21:05.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bc9fe3640 1 --2- 172.21.15.125:0/2741213116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bb0007200 0x7f1bb000dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:05.915 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bc97e2640 1 --2- 172.21.15.125:0/2741213116 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bb0005bc0 0x7f1bb0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:05.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bc8fe1640 1 --2- 172.21.15.125:0/2741213116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bb0006520 0x7f1bb0006a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:05.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bc9fe3640 1 -- 172.21.15.125:0/2741213116 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bb0005bc0 msgr2=0x7f1bb0005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:05.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bc9fe3640 1 --2- 172.21.15.125:0/2741213116 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bb0005bc0 0x7f1bb0005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:05.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bc9fe3640 1 -- 172.21.15.125:0/2741213116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bb0006520 msgr2=0x7f1bb0006a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:05.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bc9fe3640 1 --2- 172.21.15.125:0/2741213116 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bb0006520 0x7f1bb0006a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:05.917 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.906+0000 7f1bc9fe3640 1 -- 172.21.15.125:0/2741213116 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bb0005740 con 0x7f1bb0007200 2023-10-30T13:21:05.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.910+0000 7f1bc9fe3640 1 --2- 172.21.15.125:0/2741213116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bb0007200 0x7f1bb000dba0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00a830 tx=0x7f1bbc01faa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d23057a07ed3ddfc server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:05.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.910+0000 7f1bbbfff640 1 -- 172.21.15.125:0/2741213116 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc0aa6d0 con 0x7f1bb0007200 2023-10-30T13:21:05.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.910+0000 7f1bbbfff640 1 -- 172.21.15.125:0/2741213116 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc0aa850 con 0x7f1bb0007200 2023-10-30T13:21:05.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.910+0000 7f1bbbfff640 1 -- 172.21.15.125:0/2741213116 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc0aac40 con 0x7f1bb0007200 2023-10-30T13:21:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.910+0000 7f1bcba46640 1 -- 172.21.15.125:0/2741213116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bb0007200 msgr2=0x7f1bb000dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.910+0000 7f1bcba46640 1 --2- 172.21.15.125:0/2741213116 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bb0007200 0x7f1bb000dba0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00a830 tx=0x7f1bbc01faa0 comp rx=0 tx=0).stop 2023-10-30T13:21:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.910+0000 7f1bc9fe3640 1 -- 172.21.15.125:0/2741213116 reap_dead start 2023-10-30T13:21:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.910+0000 7f1bcba46640 1 -- 172.21.15.125:0/2741213116 shutdown_connections 2023-10-30T13:21:05.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.910+0000 7f1bcba46640 1 -- 172.21.15.125:0/2741213116 >> 172.21.15.125:0/2741213116 conn(0x7f1bc40c7910 msgr2=0x7f1bc40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:05.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.910+0000 7f1bcba46640 1 -- 172.21.15.125:0/2741213116 shutdown_connections 2023-10-30T13:21:05.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.910+0000 7f1bcba46640 1 -- 172.21.15.125:0/2741213116 wait complete. 2023-10-30T13:21:05.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bcba46640 1 Processor -- start 2023-10-30T13:21:05.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bcba46640 1 -- start start 2023-10-30T13:21:05.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bcba46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bb0007200 0x7f1bc41e9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:05.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bcba46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bc40c2210 0x7f1bc40c4120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:05.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bcba46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bc41ede60 0x7f1bc41538b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:05.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bcba46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40ab270 con 0x7f1bc41ede60 2023-10-30T13:21:05.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bcba46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40c4660 con 0x7f1bb0007200 2023-10-30T13:21:05.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bcba46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bc40c4860 con 0x7f1bc40c2210 2023-10-30T13:21:05.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bc9fe3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bc41ede60 0x7f1bc41538b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:05.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bc9fe3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bc41ede60 0x7f1bc41538b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42418/0 (socket says 172.21.15.125:42418) 2023-10-30T13:21:05.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bc9fe3640 1 -- 172.21.15.125:0/1800139378 learned_addr learned my addr 172.21.15.125:0/1800139378 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:05.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bc97e2640 1 --2- 172.21.15.125:0/1800139378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bb0007200 0x7f1bc41e9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:05.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bc8fe1640 1 --2- 172.21.15.125:0/1800139378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bc40c2210 0x7f1bc40c4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:05.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bc9fe3640 1 -- 172.21.15.125:0/1800139378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bc40c2210 msgr2=0x7f1bc40c4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:05.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bc9fe3640 1 --2- 172.21.15.125:0/1800139378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bc40c2210 0x7f1bc40c4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:05.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bc9fe3640 1 -- 172.21.15.125:0/1800139378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bb0007200 msgr2=0x7f1bc41e9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:05.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bc9fe3640 1 --2- 172.21.15.125:0/1800139378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bb0007200 0x7f1bc41e9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:05.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bc9fe3640 1 -- 172.21.15.125:0/1800139378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc40c4b40 con 0x7f1bc41ede60 2023-10-30T13:21:05.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bc9fe3640 1 --2- 172.21.15.125:0/1800139378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bc41ede60 0x7f1bc41538b0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00a830 tx=0x7f1bb0007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:05.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bb9ffb640 1 -- 172.21.15.125:0/1800139378 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc001a00 con 0x7f1bc41ede60 2023-10-30T13:21:05.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bb9ffb640 1 -- 172.21.15.125:0/1800139378 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bbc001fd0 con 0x7f1bc41ede60 2023-10-30T13:21:05.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bb9ffb640 1 -- 172.21.15.125:0/1800139378 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bbc0aa6d0 con 0x7f1bc41ede60 2023-10-30T13:21:05.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bcba46640 1 -- 172.21.15.125:0/1800139378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc40c4e20 con 0x7f1bc41ede60 2023-10-30T13:21:05.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bcba46640 1 -- 172.21.15.125:0/1800139378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc41ed940 con 0x7f1bc41ede60 2023-10-30T13:21:05.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.914+0000 7f1bcba46640 1 -- 172.21.15.125:0/1800139378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bc41ed4c0 con 0x7f1bc41ede60 2023-10-30T13:21:05.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.918+0000 7f1bb9ffb640 1 -- 172.21.15.125:0/1800139378 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1bbc0b2400 con 0x7f1bc41ede60 2023-10-30T13:21:05.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.918+0000 7f1bb9ffb640 1 --2- 172.21.15.125:0/1800139378 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ba80664d0 0x7f1ba8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:05.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.918+0000 7f1bb9ffb640 1 -- 172.21.15.125:0/1800139378 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7f1bbc0b9070 con 0x7f1bc41ede60 2023-10-30T13:21:05.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.918+0000 7f1bc97e2640 1 --2- 172.21.15.125:0/1800139378 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ba80664d0 0x7f1ba8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:05.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.918+0000 7f1bc97e2640 1 --2- 172.21.15.125:0/1800139378 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ba80664d0 0x7f1ba8068990 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0007f60 tx=0x7f1bc0007ef0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:05.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:05.922+0000 7f1bb9ffb640 1 -- 172.21.15.125:0/1800139378 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1bc41ed4c0 con 0x7f1bc41ede60 2023-10-30T13:21:06.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:06.102+0000 7f1bcba46640 1 -- 172.21.15.125:0/1800139378 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bc41ed4c0 con 0x7f1bc41ede60 2023-10-30T13:21:06.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:06.102+0000 7f1bb9ffb640 1 -- 172.21.15.125:0/1800139378 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f1bc41ed4c0 con 0x7f1bc41ede60 2023-10-30T13:21:06.107 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:06.108 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":244,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:02.907092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:06.108 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 244 2023-10-30T13:21:06.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:06.106+0000 7f1bcba46640 1 -- 172.21.15.125:0/1800139378 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ba80664d0 msgr2=0x7f1ba8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:06.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:06.106+0000 7f1bcba46640 1 --2- 172.21.15.125:0/1800139378 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ba80664d0 0x7f1ba8068990 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0007f60 tx=0x7f1bc0007ef0 comp rx=0 tx=0).stop 2023-10-30T13:21:06.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:06.106+0000 7f1bcba46640 1 -- 172.21.15.125:0/1800139378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bc41ede60 msgr2=0x7f1bc41538b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:06.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:06.106+0000 7f1bcba46640 1 --2- 172.21.15.125:0/1800139378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bc41ede60 0x7f1bc41538b0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f1bbc00a830 tx=0x7f1bb0007a10 comp rx=0 tx=0).stop 2023-10-30T13:21:06.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:06.106+0000 7f1bcba46640 1 -- 172.21.15.125:0/1800139378 shutdown_connections 2023-10-30T13:21:06.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:06.106+0000 7f1bcba46640 1 --2- 172.21.15.125:0/1800139378 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1ba80664d0 0x7f1ba8068990 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:06.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:06.106+0000 7f1bcba46640 1 --2- 172.21.15.125:0/1800139378 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1bc41ede60 0x7f1bc41538b0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:06.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:06.106+0000 7f1bcba46640 1 --2- 172.21.15.125:0/1800139378 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1bc40c2210 0x7f1bc40c4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:06.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:06.106+0000 7f1bcba46640 1 --2- 172.21.15.125:0/1800139378 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1bb0007200 0x7f1bc41e9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:06.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:06.106+0000 7f1bcba46640 1 -- 172.21.15.125:0/1800139378 >> 172.21.15.125:0/1800139378 conn(0x7f1bc40c7910 msgr2=0x7f1bc4138da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:06.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:06.106+0000 7f1bcba46640 1 -- 172.21.15.125:0/1800139378 shutdown_connections 2023-10-30T13:21:06.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:06.106+0000 7f1bcba46640 1 -- 172.21.15.125:0/1800139378 wait complete. 2023-10-30T13:21:06.129 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:21:08.130 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:08.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.274+0000 7f700170f640 1 Processor -- start 2023-10-30T13:21:08.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.274+0000 7f700170f640 1 -- start start 2023-10-30T13:21:08.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f700170f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6ffc0c6b80 0x7f6ffc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:08.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f700170f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6ffc0c6f60 0x7f6ffc0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:08.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f700170f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ffc0c8430 0x7f6ffc13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:08.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f700170f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ffc0c0a50 con 0x7f6ffc0c6f60 2023-10-30T13:21:08.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f700170f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ffc06f6e0 con 0x7f6ffc0c6b80 2023-10-30T13:21:08.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f700170f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc06f9c0 con 0x7f6ffc0c8430 2023-10-30T13:21:08.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f6ff27fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6ffc0c6f60 0x7f6ffc0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:08.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f6ff27fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6ffc0c6f60 0x7f6ffc0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42434/0 (socket says 172.21.15.125:42434) 2023-10-30T13:21:08.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f6ff27fc640 1 -- 172.21.15.125:0/2224576281 learned_addr learned my addr 172.21.15.125:0/2224576281 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:08.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f6ffaffd640 1 --2- 172.21.15.125:0/2224576281 >> v2:172.21.15.136:3300/0 conn(0x7f6ffc0c6b80 0x7f6ffc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:08.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f6ffb7fe640 1 --2- 172.21.15.125:0/2224576281 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ffc0c8430 0x7f6ffc13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:08.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f6ff27fc640 1 -- 172.21.15.125:0/2224576281 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ffc0c8430 msgr2=0x7f6ffc13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:08.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f6ff27fc640 1 --2- 172.21.15.125:0/2224576281 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ffc0c8430 0x7f6ffc13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:08.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f6ff27fc640 1 -- 172.21.15.125:0/2224576281 >> v2:172.21.15.136:3300/0 conn(0x7f6ffc0c6b80 msgr2=0x7f6ffc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:08.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f6ff27fc640 1 --2- 172.21.15.125:0/2224576281 >> v2:172.21.15.136:3300/0 conn(0x7f6ffc0c6b80 0x7f6ffc0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:08.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.278+0000 7f6ff27fc640 1 -- 172.21.15.125:0/2224576281 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc06fca0 con 0x7f6ffc0c6f60 2023-10-30T13:21:08.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f6ff27fc640 1 --2- 172.21.15.125:0/2224576281 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ffc0c6f60 0x7f6ffc0c0400 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc06f1b0 tx=0x7f6fe401c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d493b19e8698dddb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:08.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f6ffa7fc640 1 -- 172.21.15.125:0/2224576281 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe40ac630 con 0x7f6ffc0c6f60 2023-10-30T13:21:08.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f6ffa7fc640 1 -- 172.21.15.125:0/2224576281 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fe40ac790 con 0x7f6ffc0c6f60 2023-10-30T13:21:08.286 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f6ffa7fc640 1 -- 172.21.15.125:0/2224576281 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe40acb20 con 0x7f6ffc0c6f60 2023-10-30T13:21:08.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f700170f640 1 -- 172.21.15.125:0/2224576281 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ffc0c6f60 msgr2=0x7f6ffc0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:08.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f700170f640 1 --2- 172.21.15.125:0/2224576281 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ffc0c6f60 0x7f6ffc0c0400 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc06f1b0 tx=0x7f6fe401c500 comp rx=0 tx=0).stop 2023-10-30T13:21:08.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f700170f640 1 -- 172.21.15.125:0/2224576281 shutdown_connections 2023-10-30T13:21:08.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f700170f640 1 --2- 172.21.15.125:0/2224576281 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ffc0c8430 0x7f6ffc13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:08.287 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f700170f640 1 --2- 172.21.15.125:0/2224576281 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ffc0c6f60 0x7f6ffc0c0400 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:08.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f700170f640 1 --2- 172.21.15.125:0/2224576281 >> v2:172.21.15.136:3300/0 conn(0x7f6ffc0c6b80 0x7f6ffc0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:08.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f700170f640 1 -- 172.21.15.125:0/2224576281 >> 172.21.15.125:0/2224576281 conn(0x7f6ffc0c7910 msgr2=0x7f6ffc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:08.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f700170f640 1 -- 172.21.15.125:0/2224576281 shutdown_connections 2023-10-30T13:21:08.288 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f700170f640 1 -- 172.21.15.125:0/2224576281 wait complete. 2023-10-30T13:21:08.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f700170f640 1 Processor -- start 2023-10-30T13:21:08.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.282+0000 7f700170f640 1 -- start start 2023-10-30T13:21:08.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f700170f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ffc0c6b80 0x7f6ffc139ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:08.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f700170f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ffc0c6f60 0x7f6ffc138620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:08.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f700170f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ffc0c8430 0x7f6ffc138b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:08.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f700170f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc155f00 con 0x7f6ffc0c8430 2023-10-30T13:21:08.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f700170f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc1f59f0 con 0x7f6ffc0c6b80 2023-10-30T13:21:08.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f700170f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc1f5cd0 con 0x7f6ffc0c6f60 2023-10-30T13:21:08.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ffb7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ffc0c8430 0x7f6ffc138b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:08.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ffb7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ffc0c8430 0x7f6ffc138b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42450/0 (socket says 172.21.15.125:42450) 2023-10-30T13:21:08.292 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ffb7fe640 1 -- 172.21.15.125:0/468911529 learned_addr learned my addr 172.21.15.125:0/468911529 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:08.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ff27fc640 1 --2- 172.21.15.125:0/468911529 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ffc0c6f60 0x7f6ffc138620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:08.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ffaffd640 1 --2- 172.21.15.125:0/468911529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ffc0c6b80 0x7f6ffc139ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:08.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ffb7fe640 1 -- 172.21.15.125:0/468911529 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ffc0c6f60 msgr2=0x7f6ffc138620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:08.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ffb7fe640 1 --2- 172.21.15.125:0/468911529 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ffc0c6f60 0x7f6ffc138620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:08.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ffb7fe640 1 -- 172.21.15.125:0/468911529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ffc0c6b80 msgr2=0x7f6ffc139ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:08.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ffb7fe640 1 --2- 172.21.15.125:0/468911529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ffc0c6b80 0x7f6ffc139ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:08.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ffb7fe640 1 -- 172.21.15.125:0/468911529 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe40094d0 con 0x7f6ffc0c8430 2023-10-30T13:21:08.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ffb7fe640 1 --2- 172.21.15.125:0/468911529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ffc0c8430 0x7f6ffc138b60 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4007e60 tx=0x7f6ff400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:08.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ff3fff640 1 -- 172.21.15.125:0/468911529 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4014420 con 0x7f6ffc0c8430 2023-10-30T13:21:08.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ff3fff640 1 -- 172.21.15.125:0/468911529 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff40145a0 con 0x7f6ffc0c8430 2023-10-30T13:21:08.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f700170f640 1 -- 172.21.15.125:0/468911529 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ffc1f5fb0 con 0x7f6ffc0c8430 2023-10-30T13:21:08.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ff3fff640 1 -- 172.21.15.125:0/468911529 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4014950 con 0x7f6ffc0c8430 2023-10-30T13:21:08.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f700170f640 1 -- 172.21.15.125:0/468911529 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ffc1f6570 con 0x7f6ffc0c8430 2023-10-30T13:21:08.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f700170f640 1 -- 172.21.15.125:0/468911529 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ffc1f6760 con 0x7f6ffc0c8430 2023-10-30T13:21:08.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.286+0000 7f6ff3fff640 1 -- 172.21.15.125:0/468911529 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6ff4014ab0 con 0x7f6ffc0c8430 2023-10-30T13:21:08.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.290+0000 7f6ff3fff640 1 --2- 172.21.15.125:0/468911529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fe00664a0 0x7f6fe0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:08.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.290+0000 7f6ff3fff640 1 -- 172.21.15.125:0/468911529 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7f6ff4085230 con 0x7f6ffc0c8430 2023-10-30T13:21:08.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.290+0000 7f6ffaffd640 1 --2- 172.21.15.125:0/468911529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fe00664a0 0x7f6fe0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:08.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.290+0000 7f6ffaffd640 1 --2- 172.21.15.125:0/468911529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fe00664a0 0x7f6fe0068960 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00ba30 tx=0x7f6fec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:08.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.294+0000 7f6ff3fff640 1 -- 172.21.15.125:0/468911529 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6ff4051c20 con 0x7f6ffc0c8430 2023-10-30T13:21:08.476 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.474+0000 7f700170f640 1 -- 172.21.15.125:0/468911529 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ffc04e170 con 0x7f6ffc0c8430 2023-10-30T13:21:08.477 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.474+0000 7f6ff3fff640 1 -- 172.21.15.125:0/468911529 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 244 v244) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f6ff4055240 con 0x7f6ffc0c8430 2023-10-30T13:21:08.478 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:08.478 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":244,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":244,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:02.907092+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:rejoin","state_seq":40,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:08.478 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 244 2023-10-30T13:21:08.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.474+0000 7f700170f640 1 -- 172.21.15.125:0/468911529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fe00664a0 msgr2=0x7f6fe0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:08.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.474+0000 7f700170f640 1 --2- 172.21.15.125:0/468911529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fe00664a0 0x7f6fe0068960 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f6fec00ba30 tx=0x7f6fec008040 comp rx=0 tx=0).stop 2023-10-30T13:21:08.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.474+0000 7f700170f640 1 -- 172.21.15.125:0/468911529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ffc0c8430 msgr2=0x7f6ffc138b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:08.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.474+0000 7f700170f640 1 --2- 172.21.15.125:0/468911529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ffc0c8430 0x7f6ffc138b60 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f6ff4007e60 tx=0x7f6ff400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:21:08.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.474+0000 7f700170f640 1 -- 172.21.15.125:0/468911529 shutdown_connections 2023-10-30T13:21:08.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.474+0000 7f700170f640 1 --2- 172.21.15.125:0/468911529 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6fe00664a0 0x7f6fe0068960 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:08.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.474+0000 7f700170f640 1 --2- 172.21.15.125:0/468911529 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ffc0c8430 0x7f6ffc138b60 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:08.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.474+0000 7f700170f640 1 --2- 172.21.15.125:0/468911529 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ffc0c6f60 0x7f6ffc138620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:08.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.474+0000 7f700170f640 1 --2- 172.21.15.125:0/468911529 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ffc0c6b80 0x7f6ffc139ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:08.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.474+0000 7f700170f640 1 -- 172.21.15.125:0/468911529 >> 172.21.15.125:0/468911529 conn(0x7f6ffc0c7910 msgr2=0x7f6ffc0c18a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:08.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.474+0000 7f700170f640 1 -- 172.21.15.125:0/468911529 shutdown_connections 2023-10-30T13:21:08.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:08.474+0000 7f700170f640 1 -- 172.21.15.125:0/468911529 wait complete. 2023-10-30T13:21:08.505 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:21:10.506 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:10.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a22901640 1 Processor -- start 2023-10-30T13:21:10.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a22901640 1 -- start start 2023-10-30T13:21:10.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a22901640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2a1c153bf0 0x7f2a1c071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:10.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a22901640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2a1c0c6fe0 0x7f2a1c0c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:10.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a22901640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a1c0700b0 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a22901640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a1c0c6810 con 0x7f2a1c0c6fe0 2023-10-30T13:21:10.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a22901640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a1c159900 con 0x7f2a1c153bf0 2023-10-30T13:21:10.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a1b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2a1c0c6fe0 0x7f2a1c0c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:10.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a20e9e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2a1c0c79e0 0x7f2a1c0cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47234/0 (socket says 172.21.15.125:47234) 2023-10-30T13:21:10.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a20e9e640 1 -- 172.21.15.125:0/4126086106 learned_addr learned my addr 172.21.15.125:0/4126086106 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:21:10.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a1bfff640 1 --2- 172.21.15.125:0/4126086106 >> v2:172.21.15.136:3300/0 conn(0x7f2a1c153bf0 0x7f2a1c071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:10.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a1affd640 1 -- 172.21.15.125:0/4126086106 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305912455 0 0) 0x7f2a1c0700b0 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a1b7fe640 1 -- 172.21.15.125:0/4126086106 >> v2:172.21.15.136:3300/0 conn(0x7f2a1c153bf0 msgr2=0x7f2a1c071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:10.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a1b7fe640 1 --2- 172.21.15.125:0/4126086106 >> v2:172.21.15.136:3300/0 conn(0x7f2a1c153bf0 0x7f2a1c071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:10.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a1b7fe640 1 -- 172.21.15.125:0/4126086106 >> v1:172.21.15.125:6789/0 conn(0x7f2a1c0c79e0 legacy=0x7f2a1c0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:10.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a1b7fe640 1 -- 172.21.15.125:0/4126086106 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a1c0c6530 con 0x7f2a1c0c6fe0 2023-10-30T13:21:10.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.650+0000 7f2a1bfff640 1 --2- 172.21.15.125:0/4126086106 >> v2:172.21.15.136:3300/0 conn(0x7f2a1c153bf0 0x7f2a1c071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:10.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.654+0000 7f2a1b7fe640 1 --2- 172.21.15.125:0/4126086106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0c6fe0 0x7f2a1c0c74a0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c0c9150 tx=0x7f2a1001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=460c864fa85acfee server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:10.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.654+0000 7f2a1affd640 1 -- 172.21.15.125:0/4126086106 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a100a8950 con 0x7f2a1c0c6fe0 2023-10-30T13:21:10.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.654+0000 7f2a1affd640 1 -- 172.21.15.125:0/4126086106 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a100a8ab0 con 0x7f2a1c0c6fe0 2023-10-30T13:21:10.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.654+0000 7f2a1affd640 1 -- 172.21.15.125:0/4126086106 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a100a8e40 con 0x7f2a1c0c6fe0 2023-10-30T13:21:10.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.654+0000 7f2a22901640 1 -- 172.21.15.125:0/4126086106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0c6fe0 msgr2=0x7f2a1c0c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:10.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.654+0000 7f2a22901640 1 --2- 172.21.15.125:0/4126086106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0c6fe0 0x7f2a1c0c74a0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c0c9150 tx=0x7f2a1001ec50 comp rx=0 tx=0).stop 2023-10-30T13:21:10.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.654+0000 7f2a22901640 1 -- 172.21.15.125:0/4126086106 shutdown_connections 2023-10-30T13:21:10.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.654+0000 7f2a22901640 1 --2- 172.21.15.125:0/4126086106 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0c6fe0 0x7f2a1c0c74a0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:10.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.654+0000 7f2a22901640 1 --2- 172.21.15.125:0/4126086106 >> v2:172.21.15.136:3300/0 conn(0x7f2a1c153bf0 0x7f2a1c071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:10.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.654+0000 7f2a22901640 1 -- 172.21.15.125:0/4126086106 >> 172.21.15.125:0/4126086106 conn(0x7f2a1c06c6e0 msgr2=0x7f2a1c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:10.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.654+0000 7f2a22901640 1 -- 172.21.15.125:0/4126086106 shutdown_connections 2023-10-30T13:21:10.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.654+0000 7f2a22901640 1 -- 172.21.15.125:0/4126086106 wait complete. 2023-10-30T13:21:10.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a22901640 1 Processor -- start 2023-10-30T13:21:10.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a22901640 1 -- start start 2023-10-30T13:21:10.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a22901640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a1c0c6fe0 0x7f2a1c154430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:10.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a22901640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0c79e0 0x7f2a1c14e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:10.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a22901640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a1c153bf0 0x7f2a1c14f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:10.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a22901640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c14a080 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a22901640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c154970 con 0x7f2a1c153bf0 2023-10-30T13:21:10.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a22901640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a1c14e3b0 con 0x7f2a1c0c6fe0 2023-10-30T13:21:10.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a1b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0c79e0 0x7f2a1c14e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:10.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a1b7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0c79e0 0x7f2a1c14e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:42456/0 (socket says 172.21.15.125:42456) 2023-10-30T13:21:10.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a1b7fe640 1 -- 172.21.15.125:0/990668900 learned_addr learned my addr 172.21.15.125:0/990668900 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:10.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a1bfff640 1 --2- 172.21.15.125:0/990668900 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a1c0c6fe0 0x7f2a1c154430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:10.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a20e9e640 1 --2- 172.21.15.125:0/990668900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a1c153bf0 0x7f2a1c14f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:10.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a1b7fe640 1 -- 172.21.15.125:0/990668900 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a1c0c6fe0 msgr2=0x7f2a1c154430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:10.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a1b7fe640 1 --2- 172.21.15.125:0/990668900 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a1c0c6fe0 0x7f2a1c154430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:10.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a1b7fe640 1 -- 172.21.15.125:0/990668900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a1c153bf0 msgr2=0x7f2a1c14f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:10.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a1b7fe640 1 --2- 172.21.15.125:0/990668900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a1c153bf0 0x7f2a1c14f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:10.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a1b7fe640 1 -- 172.21.15.125:0/990668900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a10009420 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a1b7fe640 1 --2- 172.21.15.125:0/990668900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0c79e0 0x7f2a1c14e840 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f2a1001b860 tx=0x7f2a1001b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:10.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a18ff9640 1 -- 172.21.15.125:0/990668900 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a100b9630 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a18ff9640 1 -- 172.21.15.125:0/990668900 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a100b97b0 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a18ff9640 1 -- 172.21.15.125:0/990668900 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a100b9b60 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a22901640 1 -- 172.21.15.125:0/990668900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a1c14fec0 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.658+0000 7f2a22901640 1 -- 172.21.15.125:0/990668900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a1c14e040 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.662+0000 7f2a22901640 1 -- 172.21.15.125:0/990668900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a1c1ef140 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.662+0000 7f2a18ff9640 1 -- 172.21.15.125:0/990668900 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2a100b7040 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.662+0000 7f2a18ff9640 1 --2- 172.21.15.125:0/990668900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2a080664a0 0x7f2a08068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:10.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.662+0000 7f2a18ff9640 1 -- 172.21.15.125:0/990668900 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7f2a101227d0 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.662+0000 7f2a1bfff640 1 --2- 172.21.15.125:0/990668900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2a080664a0 0x7f2a08068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:10.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.662+0000 7f2a1bfff640 1 --2- 172.21.15.125:0/990668900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2a080664a0 0x7f2a08068960 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00ba10 tx=0x7f2a0c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:10.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.666+0000 7f2a18ff9640 1 -- 172.21.15.125:0/990668900 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2a1c1ef140 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.846+0000 7f2a22901640 1 -- 172.21.15.125:0/990668900 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a1c14dea0 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.846+0000 7f2a18ff9640 1 -- 172.21.15.125:0/990668900 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f2a1c14dea0 con 0x7f2a1c0c79e0 2023-10-30T13:21:10.851 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:10.852 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:08.535308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:10.852 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 245 2023-10-30T13:21:10.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.850+0000 7f2a22901640 1 -- 172.21.15.125:0/990668900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2a080664a0 msgr2=0x7f2a08068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:10.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.850+0000 7f2a22901640 1 --2- 172.21.15.125:0/990668900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2a080664a0 0x7f2a08068960 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c00ba10 tx=0x7f2a0c008040 comp rx=0 tx=0).stop 2023-10-30T13:21:10.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.850+0000 7f2a22901640 1 -- 172.21.15.125:0/990668900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0c79e0 msgr2=0x7f2a1c14e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:10.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.850+0000 7f2a22901640 1 --2- 172.21.15.125:0/990668900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0c79e0 0x7f2a1c14e840 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f2a1001b860 tx=0x7f2a1001b9f0 comp rx=0 tx=0).stop 2023-10-30T13:21:10.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.850+0000 7f2a22901640 1 -- 172.21.15.125:0/990668900 shutdown_connections 2023-10-30T13:21:10.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.850+0000 7f2a22901640 1 --2- 172.21.15.125:0/990668900 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2a080664a0 0x7f2a08068960 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:10.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.850+0000 7f2a22901640 1 --2- 172.21.15.125:0/990668900 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2a1c153bf0 0x7f2a1c14f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:10.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.850+0000 7f2a22901640 1 --2- 172.21.15.125:0/990668900 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2a1c0c79e0 0x7f2a1c14e840 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:10.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.850+0000 7f2a22901640 1 --2- 172.21.15.125:0/990668900 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2a1c0c6fe0 0x7f2a1c154430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:10.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.850+0000 7f2a22901640 1 -- 172.21.15.125:0/990668900 >> 172.21.15.125:0/990668900 conn(0x7f2a1c06c6e0 msgr2=0x7f2a1c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:10.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.850+0000 7f2a22901640 1 -- 172.21.15.125:0/990668900 shutdown_connections 2023-10-30T13:21:10.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:10.850+0000 7f2a22901640 1 -- 172.21.15.125:0/990668900 wait complete. 2023-10-30T13:21:10.872 INFO:tasks.mds_thrash.fs.[cephfs]:mds.e has gained rank=2, replacing gid=8833 2023-10-30T13:21:10.872 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.d 2023-10-30T13:21:12.873 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T13:21:12.874 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:21:12.874 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:21:12.877 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:21:12.877 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:12.954 INFO:tasks.ceph.mds.d.smithi136.stdout:starting mds.d at 2023-10-30T13:21:13.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c48617640 1 Processor -- start 2023-10-30T13:21:13.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c48617640 1 -- start start 2023-10-30T13:21:13.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c48617640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5c400c6bd0 0x7f5c4006fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:13.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c48617640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5c40071110 0x7f5c40158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:13.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c48617640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c40066cf0 con 0x7f5c4006f7a0 2023-10-30T13:21:13.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c48617640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c400700c0 con 0x7f5c400c6bd0 2023-10-30T13:21:13.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c48617640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c400c70d0 con 0x7f5c40071110 2023-10-30T13:21:13.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c45bb2640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5c400c6bd0 0x7f5c4006fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:13.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c45bb2640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f5c400c6bd0 0x7f5c4006fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36386/0 (socket says 172.21.15.125:36386) 2023-10-30T13:21:13.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c45bb2640 1 -- 172.21.15.125:0/324115003 learned_addr learned my addr 172.21.15.125:0/324115003 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:13.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c46bb4640 1 --2- 172.21.15.125:0/324115003 >> v2:172.21.15.136:3300/0 conn(0x7f5c40071110 0x7f5c40158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:13.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c45bb2640 1 -- 172.21.15.125:0/324115003 >> v2:172.21.15.136:3300/0 conn(0x7f5c40071110 msgr2=0x7f5c40158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:13.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c45bb2640 1 --2- 172.21.15.125:0/324115003 >> v2:172.21.15.136:3300/0 conn(0x7f5c40071110 0x7f5c40158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:13.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c45bb2640 1 -- 172.21.15.125:0/324115003 >> v1:172.21.15.136:6789/0 conn(0x7f5c4006f7a0 legacy=0x7f5c40069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:13.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.022+0000 7f5c45bb2640 1 -- 172.21.15.125:0/324115003 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c400c7390 con 0x7f5c400c6bd0 2023-10-30T13:21:13.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.026+0000 7f5c45bb2640 1 --2- 172.21.15.125:0/324115003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c400c6bd0 0x7f5c4006fb80 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f5c4006cdd0 tx=0x7f5c3001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e23af5f5924ee1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:13.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.026+0000 7f5c453b1640 1 -- 172.21.15.125:0/324115003 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c300a89f0 con 0x7f5c400c6bd0 2023-10-30T13:21:13.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.026+0000 7f5c453b1640 1 -- 172.21.15.125:0/324115003 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c300a8b50 con 0x7f5c400c6bd0 2023-10-30T13:21:13.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.026+0000 7f5c453b1640 1 -- 172.21.15.125:0/324115003 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c300b0440 con 0x7f5c400c6bd0 2023-10-30T13:21:13.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.026+0000 7f5c48617640 1 -- 172.21.15.125:0/324115003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c400c6bd0 msgr2=0x7f5c4006fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:13.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.026+0000 7f5c48617640 1 --2- 172.21.15.125:0/324115003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c400c6bd0 0x7f5c4006fb80 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f5c4006cdd0 tx=0x7f5c3001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:21:13.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.026+0000 7f5c48617640 1 -- 172.21.15.125:0/324115003 shutdown_connections 2023-10-30T13:21:13.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.026+0000 7f5c48617640 1 --2- 172.21.15.125:0/324115003 >> v2:172.21.15.136:3300/0 conn(0x7f5c40071110 0x7f5c40158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:13.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.026+0000 7f5c48617640 1 --2- 172.21.15.125:0/324115003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c400c6bd0 0x7f5c4006fb80 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:13.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.026+0000 7f5c48617640 1 -- 172.21.15.125:0/324115003 >> 172.21.15.125:0/324115003 conn(0x7f5c400c7910 msgr2=0x7f5c400c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:13.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.026+0000 7f5c48617640 1 -- 172.21.15.125:0/324115003 shutdown_connections 2023-10-30T13:21:13.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.026+0000 7f5c48617640 1 -- 172.21.15.125:0/324115003 wait complete. 2023-10-30T13:21:13.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c48617640 1 Processor -- start 2023-10-30T13:21:13.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c48617640 1 -- start start 2023-10-30T13:21:13.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c48617640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c4006f7a0 0x7f5c401ef2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:13.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c48617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c40071110 0x7f5c401f2820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:13.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c48617640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c400c6bd0 0x7f5c400c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:13.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c48617640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c401ef800 con 0x7f5c4006f7a0 2023-10-30T13:21:13.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c48617640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c401f1dd0 con 0x7f5c400c6bd0 2023-10-30T13:21:13.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c48617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c401f2d60 con 0x7f5c40071110 2023-10-30T13:21:13.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c463b3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c4006f7a0 0x7f5c401ef2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:13.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c463b3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c4006f7a0 0x7f5c401ef2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36388/0 (socket says 172.21.15.125:36388) 2023-10-30T13:21:13.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c463b3640 1 -- 172.21.15.125:0/1581465430 learned_addr learned my addr 172.21.15.125:0/1581465430 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:13.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c46bb4640 1 --2- 172.21.15.125:0/1581465430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c400c6bd0 0x7f5c400c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:13.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c45bb2640 1 --2- 172.21.15.125:0/1581465430 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c40071110 0x7f5c401f2820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:13.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c463b3640 1 -- 172.21.15.125:0/1581465430 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c40071110 msgr2=0x7f5c401f2820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:13.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c463b3640 1 --2- 172.21.15.125:0/1581465430 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c40071110 0x7f5c401f2820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:13.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c463b3640 1 -- 172.21.15.125:0/1581465430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c400c6bd0 msgr2=0x7f5c400c5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:13.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c463b3640 1 --2- 172.21.15.125:0/1581465430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c400c6bd0 0x7f5c400c5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:13.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c463b3640 1 -- 172.21.15.125:0/1581465430 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c300094d0 con 0x7f5c4006f7a0 2023-10-30T13:21:13.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c463b3640 1 --2- 172.21.15.125:0/1581465430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c4006f7a0 0x7f5c401ef2c0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00c3c0 tx=0x7f5c3c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:13.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c36ffd640 1 -- 172.21.15.125:0/1581465430 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c0052d0 con 0x7f5c4006f7a0 2023-10-30T13:21:13.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c36ffd640 1 -- 172.21.15.125:0/1581465430 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3c005430 con 0x7f5c4006f7a0 2023-10-30T13:21:13.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c48617640 1 -- 172.21.15.125:0/1581465430 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c40154730 con 0x7f5c4006f7a0 2023-10-30T13:21:13.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c36ffd640 1 -- 172.21.15.125:0/1581465430 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c003840 con 0x7f5c4006f7a0 2023-10-30T13:21:13.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c48617640 1 -- 172.21.15.125:0/1581465430 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c40154cf0 con 0x7f5c4006f7a0 2023-10-30T13:21:13.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.030+0000 7f5c48617640 1 -- 172.21.15.125:0/1581465430 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c4004e170 con 0x7f5c4006f7a0 2023-10-30T13:21:13.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.034+0000 7f5c36ffd640 1 -- 172.21.15.125:0/1581465430 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5c3c0039a0 con 0x7f5c4006f7a0 2023-10-30T13:21:13.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.034+0000 7f5c36ffd640 1 --2- 172.21.15.125:0/1581465430 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5c240664a0 0x7f5c24068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:13.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.034+0000 7f5c36ffd640 1 -- 172.21.15.125:0/1581465430 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7f5c3c083dd0 con 0x7f5c4006f7a0 2023-10-30T13:21:13.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.034+0000 7f5c45bb2640 1 --2- 172.21.15.125:0/1581465430 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5c240664a0 0x7f5c24068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:13.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.034+0000 7f5c45bb2640 1 --2- 172.21.15.125:0/1581465430 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5c240664a0 0x7f5c24068960 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f5c300b2000 tx=0x7f5c300aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:13.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.038+0000 7f5c36ffd640 1 -- 172.21.15.125:0/1581465430 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5c3c050840 con 0x7f5c4006f7a0 2023-10-30T13:21:13.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.218+0000 7f5c48617640 1 -- 172.21.15.125:0/1581465430 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c400b06c0 con 0x7f5c4006f7a0 2023-10-30T13:21:13.223 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.218+0000 7f5c36ffd640 1 -- 172.21.15.125:0/1581465430 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 245 v245) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f5c3c017070 con 0x7f5c4006f7a0 2023-10-30T13:21:13.224 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:13.224 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":245,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:08.535308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:13.225 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 245 2023-10-30T13:21:13.225 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.222+0000 7f5c48617640 1 -- 172.21.15.125:0/1581465430 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5c240664a0 msgr2=0x7f5c24068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:13.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.222+0000 7f5c48617640 1 --2- 172.21.15.125:0/1581465430 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5c240664a0 0x7f5c24068960 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f5c300b2000 tx=0x7f5c300aa070 comp rx=0 tx=0).stop 2023-10-30T13:21:13.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.222+0000 7f5c48617640 1 -- 172.21.15.125:0/1581465430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c4006f7a0 msgr2=0x7f5c401ef2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:13.226 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.222+0000 7f5c48617640 1 --2- 172.21.15.125:0/1581465430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c4006f7a0 0x7f5c401ef2c0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00c3c0 tx=0x7f5c3c00c880 comp rx=0 tx=0).stop 2023-10-30T13:21:13.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.222+0000 7f5c48617640 1 -- 172.21.15.125:0/1581465430 shutdown_connections 2023-10-30T13:21:13.227 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.222+0000 7f5c48617640 1 --2- 172.21.15.125:0/1581465430 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5c240664a0 0x7f5c24068960 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:13.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.222+0000 7f5c48617640 1 --2- 172.21.15.125:0/1581465430 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5c400c6bd0 0x7f5c400c5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:13.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.222+0000 7f5c48617640 1 --2- 172.21.15.125:0/1581465430 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5c40071110 0x7f5c401f2820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:13.228 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.222+0000 7f5c48617640 1 --2- 172.21.15.125:0/1581465430 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5c4006f7a0 0x7f5c401ef2c0 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:13.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.222+0000 7f5c48617640 1 -- 172.21.15.125:0/1581465430 >> 172.21.15.125:0/1581465430 conn(0x7f5c400c7910 msgr2=0x7f5c400c8c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:13.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.222+0000 7f5c48617640 1 -- 172.21.15.125:0/1581465430 shutdown_connections 2023-10-30T13:21:13.229 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:13.222+0000 7f5c48617640 1 -- 172.21.15.125:0/1581465430 wait complete. 2023-10-30T13:21:13.244 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T13:21:15.246 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:15.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.390+0000 7f239e06b640 1 Processor -- start 2023-10-30T13:21:15.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.390+0000 7f239e06b640 1 -- start start 2023-10-30T13:21:15.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.390+0000 7f239e06b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2398153b10 0x7f239815ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:15.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.390+0000 7f239e06b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f23980c5e70 0x7f23981641f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:15.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.390+0000 7f239e06b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23980c74f0 con 0x7f23980b0dc0 2023-10-30T13:21:15.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.390+0000 7f239e06b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23980c1f20 con 0x7f2398153b10 2023-10-30T13:21:15.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.390+0000 7f239e06b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23980c77b0 con 0x7f23980c5e70 2023-10-30T13:21:15.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.390+0000 7f23977fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2398153b10 0x7f239815ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:15.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.390+0000 7f238ffff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f23980b0dc0 0x7f23980c7e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46724/0 (socket says 172.21.15.125:46724) 2023-10-30T13:21:15.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.390+0000 7f238ffff640 1 -- 172.21.15.125:0/520262538 learned_addr learned my addr 172.21.15.125:0/520262538 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:21:15.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.390+0000 7f2397fff640 1 --2- 172.21.15.125:0/520262538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f23980c5e70 0x7f23981641f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:15.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.390+0000 7f23977fe640 1 -- 172.21.15.125:0/520262538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f23980c5e70 msgr2=0x7f23981641f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:15.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.390+0000 7f23977fe640 1 --2- 172.21.15.125:0/520262538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f23980c5e70 0x7f23981641f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:15.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.390+0000 7f23977fe640 1 -- 172.21.15.125:0/520262538 >> v1:172.21.15.125:6789/0 conn(0x7f23980b0dc0 legacy=0x7f23980c7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:15.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.394+0000 7f2396ffd640 1 -- 172.21.15.125:0/520262538 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260154300 0 0) 0x7f23980c74f0 con 0x7f23980b0dc0 2023-10-30T13:21:15.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.394+0000 7f23977fe640 1 -- 172.21.15.125:0/520262538 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2398159b80 con 0x7f2398153b10 2023-10-30T13:21:15.399 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.394+0000 7f23977fe640 1 --2- 172.21.15.125:0/520262538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2398153b10 0x7f239815ce60 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f23880097c0 tx=0x7f238801ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856f4f3b87baae1d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:15.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.394+0000 7f2396ffd640 1 -- 172.21.15.125:0/520262538 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23880a8970 con 0x7f2398153b10 2023-10-30T13:21:15.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.394+0000 7f2396ffd640 1 -- 172.21.15.125:0/520262538 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23880a8ad0 con 0x7f2398153b10 2023-10-30T13:21:15.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.394+0000 7f2396ffd640 1 -- 172.21.15.125:0/520262538 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23880a8e60 con 0x7f2398153b10 2023-10-30T13:21:15.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.394+0000 7f239e06b640 1 -- 172.21.15.125:0/520262538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2398153b10 msgr2=0x7f239815ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:15.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.394+0000 7f239e06b640 1 --2- 172.21.15.125:0/520262538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2398153b10 0x7f239815ce60 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f23880097c0 tx=0x7f238801ec70 comp rx=0 tx=0).stop 2023-10-30T13:21:15.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.394+0000 7f239e06b640 1 -- 172.21.15.125:0/520262538 shutdown_connections 2023-10-30T13:21:15.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.394+0000 7f239e06b640 1 --2- 172.21.15.125:0/520262538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f23980c5e70 0x7f23981641f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:15.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.394+0000 7f239e06b640 1 --2- 172.21.15.125:0/520262538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2398153b10 0x7f239815ce60 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:15.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.394+0000 7f239e06b640 1 -- 172.21.15.125:0/520262538 >> 172.21.15.125:0/520262538 conn(0x7f239806d770 msgr2=0x7f239806db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:15.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.394+0000 7f239e06b640 1 -- 172.21.15.125:0/520262538 shutdown_connections 2023-10-30T13:21:15.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.394+0000 7f239e06b640 1 -- 172.21.15.125:0/520262538 wait complete. 2023-10-30T13:21:15.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f239e06b640 1 Processor -- start 2023-10-30T13:21:15.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f239e06b640 1 -- start start 2023-10-30T13:21:15.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f239e06b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f23980b0dc0 0x7f2398145620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:15.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f239e06b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f23980c5e70 0x7f2398140400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:15.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f239e06b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2398153b10 0x7f23981440b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:15.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f239e06b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2398148810 con 0x7f23980c5e70 2023-10-30T13:21:15.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f239e06b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23981e6960 con 0x7f2398153b10 2023-10-30T13:21:15.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f239e06b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2398148380 con 0x7f23980b0dc0 2023-10-30T13:21:15.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f238ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f23980c5e70 0x7f2398140400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:15.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f238ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f23980c5e70 0x7f2398140400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36420/0 (socket says 172.21.15.125:36420) 2023-10-30T13:21:15.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f238ffff640 1 -- 172.21.15.125:0/1216627826 learned_addr learned my addr 172.21.15.125:0/1216627826 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:15.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f23977fe640 1 --2- 172.21.15.125:0/1216627826 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f23980b0dc0 0x7f2398145620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:15.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f2397fff640 1 --2- 172.21.15.125:0/1216627826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2398153b10 0x7f23981440b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:15.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f238ffff640 1 -- 172.21.15.125:0/1216627826 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f23980b0dc0 msgr2=0x7f2398145620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:15.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f238ffff640 1 --2- 172.21.15.125:0/1216627826 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f23980b0dc0 0x7f2398145620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:15.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f238ffff640 1 -- 172.21.15.125:0/1216627826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2398153b10 msgr2=0x7f23981440b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:15.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f238ffff640 1 --2- 172.21.15.125:0/1216627826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2398153b10 0x7f23981440b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:15.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f238ffff640 1 -- 172.21.15.125:0/1216627826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2388009420 con 0x7f23980c5e70 2023-10-30T13:21:15.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f238ffff640 1 --2- 172.21.15.125:0/1216627826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f23980c5e70 0x7f2398140400 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f23800011f0 tx=0x7f238000c9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:15.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f2394ff9640 1 -- 172.21.15.125:0/1216627826 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23800074e0 con 0x7f23980c5e70 2023-10-30T13:21:15.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f2394ff9640 1 -- 172.21.15.125:0/1216627826 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2380007640 con 0x7f23980c5e70 2023-10-30T13:21:15.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f239e06b640 1 -- 172.21.15.125:0/1216627826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2398146910 con 0x7f23980c5e70 2023-10-30T13:21:15.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.398+0000 7f2394ff9640 1 -- 172.21.15.125:0/1216627826 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2380004a10 con 0x7f23980c5e70 2023-10-30T13:21:15.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.402+0000 7f239e06b640 1 -- 172.21.15.125:0/1216627826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2398146e90 con 0x7f23980c5e70 2023-10-30T13:21:15.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.402+0000 7f239e06b640 1 -- 172.21.15.125:0/1216627826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2398148530 con 0x7f23980c5e70 2023-10-30T13:21:15.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.402+0000 7f2394ff9640 1 -- 172.21.15.125:0/1216627826 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2380004b90 con 0x7f23980c5e70 2023-10-30T13:21:15.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.402+0000 7f2394ff9640 1 --2- 172.21.15.125:0/1216627826 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f23780664d0 0x7f2378068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:15.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.402+0000 7f2394ff9640 1 -- 172.21.15.125:0/1216627826 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7f2380083d70 con 0x7f23980c5e70 2023-10-30T13:21:15.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.402+0000 7f23977fe640 1 --2- 172.21.15.125:0/1216627826 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f23780664d0 0x7f2378068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:15.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.406+0000 7f23977fe640 1 --2- 172.21.15.125:0/1216627826 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f23780664d0 0x7f2378068990 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f23880b2000 tx=0x7f2388008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:15.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.406+0000 7f2394ff9640 1 -- 172.21.15.125:0/1216627826 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2398148530 con 0x7f23980c5e70 2023-10-30T13:21:15.590 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.586+0000 7f239e06b640 1 -- 172.21.15.125:0/1216627826 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f23981484f0 con 0x7f23980c5e70 2023-10-30T13:21:15.591 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.586+0000 7f2394ff9640 1 -- 172.21.15.125:0/1216627826 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 246 v246) v1 ==== 95+0+6553 (secure 0 0 0) 0x7f2380018070 con 0x7f23980c5e70 2023-10-30T13:21:15.592 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:15.592 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":246,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":245,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:08.535308+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:15.592 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 246 2023-10-30T13:21:15.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.590+0000 7f239e06b640 1 -- 172.21.15.125:0/1216627826 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f23780664d0 msgr2=0x7f2378068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:15.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.590+0000 7f239e06b640 1 --2- 172.21.15.125:0/1216627826 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f23780664d0 0x7f2378068990 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f23880b2000 tx=0x7f2388008040 comp rx=0 tx=0).stop 2023-10-30T13:21:15.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.590+0000 7f239e06b640 1 -- 172.21.15.125:0/1216627826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f23980c5e70 msgr2=0x7f2398140400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:15.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.590+0000 7f239e06b640 1 --2- 172.21.15.125:0/1216627826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f23980c5e70 0x7f2398140400 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f23800011f0 tx=0x7f238000c9b0 comp rx=0 tx=0).stop 2023-10-30T13:21:15.594 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.590+0000 7f239e06b640 1 -- 172.21.15.125:0/1216627826 shutdown_connections 2023-10-30T13:21:15.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.590+0000 7f239e06b640 1 --2- 172.21.15.125:0/1216627826 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f23780664d0 0x7f2378068990 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:15.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.590+0000 7f239e06b640 1 --2- 172.21.15.125:0/1216627826 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2398153b10 0x7f23981440b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:15.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.590+0000 7f239e06b640 1 --2- 172.21.15.125:0/1216627826 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f23980c5e70 0x7f2398140400 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:15.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.590+0000 7f239e06b640 1 --2- 172.21.15.125:0/1216627826 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f23980b0dc0 0x7f2398145620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:15.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.590+0000 7f239e06b640 1 -- 172.21.15.125:0/1216627826 >> 172.21.15.125:0/1216627826 conn(0x7f239806d770 msgr2=0x7f2398071cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:15.595 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.590+0000 7f239e06b640 1 -- 172.21.15.125:0/1216627826 shutdown_connections 2023-10-30T13:21:15.596 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:15.590+0000 7f239e06b640 1 -- 172.21.15.125:0/1216627826 wait complete. 2023-10-30T13:21:15.613 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T13:21:15.613 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 22 secs before thrashing 2023-10-30T13:21:16.165 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:21:16.165 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.61 2023-10-30T13:21:16.166 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 61 -eq 48 ] 2023-10-30T13:21:16.166 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.61 2023-10-30T13:21:16.166 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.61 2023-10-30T13:21:16.166 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.61 2023-10-30T13:21:16.166 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:21:17.791 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:21:17.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:21:23.565 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:21:23.565 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 62 -eq 48 ] 2023-10-30T13:21:23.566 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.62 2023-10-30T13:21:23.567 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.62 2023-10-30T13:21:23.567 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.62 2023-10-30T13:21:23.567 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.62 2023-10-30T13:21:23.567 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:21:32.840 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:21:32.840 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 63 -eq 48 ] 2023-10-30T13:21:32.840 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.63 2023-10-30T13:21:32.841 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.63 2023-10-30T13:21:32.841 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.63 2023-10-30T13:21:32.841 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.63 2023-10-30T13:21:32.842 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:21:37.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:37.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.769+0000 7fe63ed0f640 1 Processor -- start 2023-10-30T13:21:37.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.769+0000 7fe63ed0f640 1 -- start start 2023-10-30T13:21:37.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.773+0000 7fe63ed0f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe63806f7a0 0x7fe63806fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:37.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.773+0000 7fe63ed0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6380c6bd0 0x7fe638069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:37.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.773+0000 7fe63ed0f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe63806a3d0 con 0x7fe638071110 2023-10-30T13:21:37.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.773+0000 7fe63ed0f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6380700c0 con 0x7fe63806f7a0 2023-10-30T13:21:37.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.773+0000 7fe63ed0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe638066cf0 con 0x7fe6380c6bd0 2023-10-30T13:21:37.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.773+0000 7fe62ffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6380c6bd0 0x7fe638069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:37.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.773+0000 7fe63caab640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe63806f7a0 0x7fe63806fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:37.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.773+0000 7fe62ffff640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6380c6bd0 0x7fe638069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:38558/0 (socket says 172.21.15.125:38558) 2023-10-30T13:21:37.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.773+0000 7fe62ffff640 1 -- 172.21.15.125:0/978098540 learned_addr learned my addr 172.21.15.125:0/978098540 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:37.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.773+0000 7fe62ffff640 1 -- 172.21.15.125:0/978098540 >> v2:172.21.15.136:3300/0 conn(0x7fe63806f7a0 msgr2=0x7fe63806fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:37.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.773+0000 7fe62ffff640 1 --2- 172.21.15.125:0/978098540 >> v2:172.21.15.136:3300/0 conn(0x7fe63806f7a0 0x7fe63806fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:37.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.773+0000 7fe62ffff640 1 -- 172.21.15.125:0/978098540 >> v1:172.21.15.136:6789/0 conn(0x7fe638071110 legacy=0x7fe638158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:37.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.773+0000 7fe62ffff640 1 -- 172.21.15.125:0/978098540 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6380c70d0 con 0x7fe6380c6bd0 2023-10-30T13:21:37.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.777+0000 7fe62ffff640 1 --2- 172.21.15.125:0/978098540 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6380c6bd0 0x7fe638069e90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe628009870 tx=0x7fe62801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d9e1fc3b9f843c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:37.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.777+0000 7fe62f7fe640 1 -- 172.21.15.125:0/978098540 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6280a8a30 con 0x7fe6380c6bd0 2023-10-30T13:21:37.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.777+0000 7fe62f7fe640 1 -- 172.21.15.125:0/978098540 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6280a8b90 con 0x7fe6380c6bd0 2023-10-30T13:21:37.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.777+0000 7fe62f7fe640 1 -- 172.21.15.125:0/978098540 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6280b0440 con 0x7fe6380c6bd0 2023-10-30T13:21:37.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.777+0000 7fe63ed0f640 1 -- 172.21.15.125:0/978098540 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6380c6bd0 msgr2=0x7fe638069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:37.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.777+0000 7fe63ed0f640 1 --2- 172.21.15.125:0/978098540 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6380c6bd0 0x7fe638069e90 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe628009870 tx=0x7fe62801ed30 comp rx=0 tx=0).stop 2023-10-30T13:21:37.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.777+0000 7fe63ed0f640 1 -- 172.21.15.125:0/978098540 shutdown_connections 2023-10-30T13:21:37.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.777+0000 7fe63ed0f640 1 --2- 172.21.15.125:0/978098540 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6380c6bd0 0x7fe638069e90 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:37.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.777+0000 7fe63ed0f640 1 --2- 172.21.15.125:0/978098540 >> v2:172.21.15.136:3300/0 conn(0x7fe63806f7a0 0x7fe63806fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:37.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.777+0000 7fe63ed0f640 1 -- 172.21.15.125:0/978098540 >> 172.21.15.125:0/978098540 conn(0x7fe6380c7910 msgr2=0x7fe6380c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:37.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.777+0000 7fe63ed0f640 1 -- 172.21.15.125:0/978098540 shutdown_connections 2023-10-30T13:21:37.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.777+0000 7fe63ed0f640 1 -- 172.21.15.125:0/978098540 wait complete. 2023-10-30T13:21:37.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.777+0000 7fe63ed0f640 1 Processor -- start 2023-10-30T13:21:37.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.777+0000 7fe63ed0f640 1 -- start start 2023-10-30T13:21:37.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63ed0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe63806f7a0 0x7fe6380aff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:37.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63ed0f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe638071110 0x7fe638144140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:37.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63ed0f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6380c6bd0 0x7fe6380c99f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:37.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63ed0f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6380c9360 con 0x7fe63806f7a0 2023-10-30T13:21:37.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63ed0f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6380c9f30 con 0x7fe638071110 2023-10-30T13:21:37.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63ed0f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6381f2fa0 con 0x7fe6380c6bd0 2023-10-30T13:21:37.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63caab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe63806f7a0 0x7fe6380aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:37.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63caab640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe63806f7a0 0x7fe6380aff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46498/0 (socket says 172.21.15.125:46498) 2023-10-30T13:21:37.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63caab640 1 -- 172.21.15.125:0/3553535036 learned_addr learned my addr 172.21.15.125:0/3553535036 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:37.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63d2ac640 1 --2- 172.21.15.125:0/3553535036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6380c6bd0 0x7fe6380c99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:37.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe62ffff640 1 --2- 172.21.15.125:0/3553535036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe638071110 0x7fe638144140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:37.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63caab640 1 -- 172.21.15.125:0/3553535036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6380c6bd0 msgr2=0x7fe6380c99f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:37.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63caab640 1 --2- 172.21.15.125:0/3553535036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6380c6bd0 0x7fe6380c99f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:37.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63caab640 1 -- 172.21.15.125:0/3553535036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe638071110 msgr2=0x7fe638144140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:37.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63caab640 1 --2- 172.21.15.125:0/3553535036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe638071110 0x7fe638144140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:37.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63caab640 1 -- 172.21.15.125:0/3553535036 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6280094d0 con 0x7fe63806f7a0 2023-10-30T13:21:37.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63caab640 1 --2- 172.21.15.125:0/3553535036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe63806f7a0 0x7fe6380aff30 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fe63400e8a0 tx=0x7fe63400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:37.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe62d7fa640 1 -- 172.21.15.125:0/3553535036 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe634009510 con 0x7fe63806f7a0 2023-10-30T13:21:37.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe62d7fa640 1 -- 172.21.15.125:0/3553535036 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe63400ee90 con 0x7fe63806f7a0 2023-10-30T13:21:37.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63ed0f640 1 -- 172.21.15.125:0/3553535036 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6380ade50 con 0x7fe63806f7a0 2023-10-30T13:21:37.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe62d7fa640 1 -- 172.21.15.125:0/3553535036 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe63400cae0 con 0x7fe63806f7a0 2023-10-30T13:21:37.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63ed0f640 1 -- 172.21.15.125:0/3553535036 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6380c8fc0 con 0x7fe63806f7a0 2023-10-30T13:21:37.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.781+0000 7fe63ed0f640 1 -- 172.21.15.125:0/3553535036 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe638070330 con 0x7fe63806f7a0 2023-10-30T13:21:37.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.785+0000 7fe62d7fa640 1 -- 172.21.15.125:0/3553535036 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe63400cc40 con 0x7fe63806f7a0 2023-10-30T13:21:37.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.785+0000 7fe62d7fa640 1 --2- 172.21.15.125:0/3553535036 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6240664a0 0x7fe624068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:37.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.785+0000 7fe62d7fa640 1 -- 172.21.15.125:0/3553535036 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7fe6340848c0 con 0x7fe63806f7a0 2023-10-30T13:21:37.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.785+0000 7fe62ffff640 1 --2- 172.21.15.125:0/3553535036 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6240664a0 0x7fe624068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:37.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.785+0000 7fe62ffff640 1 --2- 172.21.15.125:0/3553535036 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6240664a0 0x7fe624068960 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fe6280b2000 tx=0x7fe628009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:37.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.789+0000 7fe62d7fa640 1 -- 172.21.15.125:0/3553535036 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe638070330 con 0x7fe63806f7a0 2023-10-30T13:21:37.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.969+0000 7fe63ed0f640 1 -- 172.21.15.125:0/3553535036 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe638070330 con 0x7fe63806f7a0 2023-10-30T13:21:37.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.969+0000 7fe62d7fa640 1 -- 172.21.15.125:0/3553535036 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+6550 (secure 0 0 0) 0x7fe638070330 con 0x7fe63806f7a0 2023-10-30T13:21:37.975 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:37.975 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:33.189409+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:37.976 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 247 2023-10-30T13:21:37.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.973+0000 7fe63ed0f640 1 -- 172.21.15.125:0/3553535036 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6240664a0 msgr2=0x7fe624068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:37.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.973+0000 7fe63ed0f640 1 --2- 172.21.15.125:0/3553535036 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6240664a0 0x7fe624068960 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7fe6280b2000 tx=0x7fe628009040 comp rx=0 tx=0).stop 2023-10-30T13:21:37.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.973+0000 7fe63ed0f640 1 -- 172.21.15.125:0/3553535036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe63806f7a0 msgr2=0x7fe6380aff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:37.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.973+0000 7fe63ed0f640 1 --2- 172.21.15.125:0/3553535036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe63806f7a0 0x7fe6380aff30 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7fe63400e8a0 tx=0x7fe63400eca0 comp rx=0 tx=0).stop 2023-10-30T13:21:37.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.973+0000 7fe63ed0f640 1 -- 172.21.15.125:0/3553535036 shutdown_connections 2023-10-30T13:21:37.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.973+0000 7fe63ed0f640 1 --2- 172.21.15.125:0/3553535036 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe6240664a0 0x7fe624068960 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:37.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.973+0000 7fe63ed0f640 1 --2- 172.21.15.125:0/3553535036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe6380c6bd0 0x7fe6380c99f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:37.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.973+0000 7fe63ed0f640 1 --2- 172.21.15.125:0/3553535036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe638071110 0x7fe638144140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:37.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.973+0000 7fe63ed0f640 1 --2- 172.21.15.125:0/3553535036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe63806f7a0 0x7fe6380aff30 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:37.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.973+0000 7fe63ed0f640 1 -- 172.21.15.125:0/3553535036 >> 172.21.15.125:0/3553535036 conn(0x7fe6380c7910 msgr2=0x7fe6380c7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:37.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.973+0000 7fe63ed0f640 1 -- 172.21.15.125:0/3553535036 shutdown_connections 2023-10-30T13:21:37.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:37.973+0000 7fe63ed0f640 1 -- 172.21.15.125:0/3553535036 wait complete. 2023-10-30T13:21:37.997 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.f (rank=0) 2023-10-30T13:21:37.997 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T13:21:37.997 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:21:38.019 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T13:21:38.019 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:38.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.153+0000 7fab0b446640 1 Processor -- start 2023-10-30T13:21:38.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.153+0000 7fab0b446640 1 -- start start 2023-10-30T13:21:38.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.153+0000 7fab0b446640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab04070a00 0x7fab04070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:38.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.153+0000 7fab0b446640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fab04069e90 0x7fab0406a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:38.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.153+0000 7fab0b446640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab04071380 con 0x7fab041539a0 2023-10-30T13:21:38.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.153+0000 7fab0b446640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab0406fbb0 con 0x7fab04069e90 2023-10-30T13:21:38.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.153+0000 7fab0b446640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab04153d80 con 0x7fab04070a00 2023-10-30T13:21:38.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.153+0000 7fab091e2640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fab041539a0 0x7fab0415a380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:44378/0 (socket says 172.21.15.125:44378) 2023-10-30T13:21:38.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.153+0000 7fab091e2640 1 -- 172.21.15.125:0/1555479496 learned_addr learned my addr 172.21.15.125:0/1555479496 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:21:38.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.153+0000 7fab089e1640 1 --2- 172.21.15.125:0/1555479496 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab04070a00 0x7fab04070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:38.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.153+0000 7fab099e3640 1 --2- 172.21.15.125:0/1555479496 >> v2:172.21.15.136:3300/0 conn(0x7fab04069e90 0x7fab0406a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:38.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 -- 172.21.15.125:0/1555479496 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481749928 0 0) 0x7fab04071380 con 0x7fab041539a0 2023-10-30T13:21:38.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 -- 172.21.15.125:0/1555479496 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaf0003530 con 0x7fab041539a0 2023-10-30T13:21:38.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 -- 172.21.15.125:0/1555479496 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 879601999 0 0) 0x7faaf0003530 con 0x7fab041539a0 2023-10-30T13:21:38.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 -- 172.21.15.125:0/1555479496 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab04070a00 msgr2=0x7fab04070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:38.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 --2- 172.21.15.125:0/1555479496 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab04070a00 0x7fab04070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:38.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 -- 172.21.15.125:0/1555479496 >> v2:172.21.15.136:3300/0 conn(0x7fab04069e90 msgr2=0x7fab0406a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:38.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 --2- 172.21.15.125:0/1555479496 >> v2:172.21.15.136:3300/0 conn(0x7fab04069e90 0x7fab0406a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:38.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 -- 172.21.15.125:0/1555479496 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab040c5c50 con 0x7fab041539a0 2023-10-30T13:21:38.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 -- 172.21.15.125:0/1555479496 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fab00001680 con 0x7fab041539a0 2023-10-30T13:21:38.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7fab099e3640 1 --2- 172.21.15.125:0/1555479496 >> v2:172.21.15.136:3300/0 conn(0x7fab04069e90 0x7fab0406a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:21:38.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 -- 172.21.15.125:0/1555479496 >> v1:172.21.15.125:6789/0 conn(0x7fab041539a0 legacy=0x7fab0415a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:38.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 --2- 172.21.15.125:0/1555479496 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaf0003980 0x7faaf0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:38.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 --2- 172.21.15.125:0/1555479496 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaf00047d0 0x7faaf0004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:38.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 --2- 172.21.15.125:0/1555479496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf0005190 0x7faaf000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:38.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 -- 172.21.15.125:0/1555479496 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab0406fbb0 con 0x7faaf0005190 2023-10-30T13:21:38.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 -- 172.21.15.125:0/1555479496 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab04153d80 con 0x7faaf0003980 2023-10-30T13:21:38.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7faafbfff640 1 -- 172.21.15.125:0/1555479496 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaf000bf20 con 0x7faaf00047d0 2023-10-30T13:21:38.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7fab089e1640 1 --2- 172.21.15.125:0/1555479496 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fab04070a00 0x7fab04070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:21:38.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7fab099e3640 1 --2- 172.21.15.125:0/1555479496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf0005190 0x7faaf000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:38.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7fab091e2640 1 --2- 172.21.15.125:0/1555479496 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaf0003980 0x7faaf0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:38.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7fab089e1640 1 --2- 172.21.15.125:0/1555479496 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaf00047d0 0x7faaf0004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:38.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7fab099e3640 1 -- 172.21.15.125:0/1555479496 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaf00047d0 msgr2=0x7faaf0004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:38.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7fab099e3640 1 --2- 172.21.15.125:0/1555479496 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaf00047d0 0x7faaf0004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:38.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7fab099e3640 1 -- 172.21.15.125:0/1555479496 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaf0003980 msgr2=0x7faaf0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:38.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7fab099e3640 1 --2- 172.21.15.125:0/1555479496 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faaf0003980 0x7faaf0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:38.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.157+0000 7fab099e3640 1 -- 172.21.15.125:0/1555479496 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaf0003530 con 0x7faaf0005190 2023-10-30T13:21:38.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.161+0000 7fab099e3640 1 --2- 172.21.15.125:0/1555479496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf0005190 0x7faaf000b990 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7faafc00e380 tx=0x7faafc01ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4174a3a99b78553c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:38.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.161+0000 7faafbfff640 1 -- 172.21.15.125:0/1555479496 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc0a5e50 con 0x7faaf0005190 2023-10-30T13:21:38.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.161+0000 7faafbfff640 1 -- 172.21.15.125:0/1555479496 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faafc00f040 con 0x7faaf0005190 2023-10-30T13:21:38.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.161+0000 7faafbfff640 1 -- 172.21.15.125:0/1555479496 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faafc0ad640 con 0x7faaf0005190 2023-10-30T13:21:38.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.161+0000 7fab0b446640 1 -- 172.21.15.125:0/1555479496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf0005190 msgr2=0x7faaf000b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:38.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.161+0000 7fab0b446640 1 --2- 172.21.15.125:0/1555479496 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faaf0005190 0x7faaf000b990 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7faafc00e380 tx=0x7faafc01ce40 comp rx=0 tx=0).stop 2023-10-30T13:21:38.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.161+0000 7fab099e3640 1 -- 172.21.15.125:0/1555479496 reap_dead start 2023-10-30T13:21:38.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.161+0000 7fab0b446640 1 -- 172.21.15.125:0/1555479496 shutdown_connections 2023-10-30T13:21:38.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.161+0000 7fab0b446640 1 -- 172.21.15.125:0/1555479496 >> 172.21.15.125:0/1555479496 conn(0x7fab040c7910 msgr2=0x7fab040c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:38.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.161+0000 7fab0b446640 1 -- 172.21.15.125:0/1555479496 shutdown_connections 2023-10-30T13:21:38.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.161+0000 7fab0b446640 1 -- 172.21.15.125:0/1555479496 wait complete. 2023-10-30T13:21:38.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.161+0000 7fab0b446640 1 Processor -- start 2023-10-30T13:21:38.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.161+0000 7fab0b446640 1 -- start start 2023-10-30T13:21:38.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab0b446640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaf0005190 0x7fab041541f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:38.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab0b446640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab04157770 0x7fab040c1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:38.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab0b446640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab04156e10 0x7fab0415a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:38.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab0b446640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab04154e60 con 0x7fab04157770 2023-10-30T13:21:38.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab0b446640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab04154730 con 0x7fab04156e10 2023-10-30T13:21:38.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab0b446640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab04155920 con 0x7faaf0005190 2023-10-30T13:21:38.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab089e1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab04157770 0x7fab040c1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:38.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab089e1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab04157770 0x7fab040c1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46514/0 (socket says 172.21.15.125:46514) 2023-10-30T13:21:38.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab089e1640 1 -- 172.21.15.125:0/4171272243 learned_addr learned my addr 172.21.15.125:0/4171272243 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:38.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab091e2640 1 --2- 172.21.15.125:0/4171272243 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaf0005190 0x7fab041541f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:38.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab099e3640 1 --2- 172.21.15.125:0/4171272243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab04156e10 0x7fab0415a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:38.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab089e1640 1 -- 172.21.15.125:0/4171272243 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaf0005190 msgr2=0x7fab041541f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:38.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab089e1640 1 --2- 172.21.15.125:0/4171272243 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaf0005190 0x7fab041541f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:38.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab089e1640 1 -- 172.21.15.125:0/4171272243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab04156e10 msgr2=0x7fab0415a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:38.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab089e1640 1 --2- 172.21.15.125:0/4171272243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab04156e10 0x7fab0415a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:38.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab089e1640 1 -- 172.21.15.125:0/4171272243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab040c10d0 con 0x7fab04157770 2023-10-30T13:21:38.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab089e1640 1 --2- 172.21.15.125:0/4171272243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab04157770 0x7fab040c1560 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7faaf4011800 tx=0x7faaf4011cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:38.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7faaf9ffb640 1 -- 172.21.15.125:0/4171272243 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf400f960 con 0x7fab04157770 2023-10-30T13:21:38.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7faaf9ffb640 1 -- 172.21.15.125:0/4171272243 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf400fac0 con 0x7fab04157770 2023-10-30T13:21:38.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab0b446640 1 -- 172.21.15.125:0/4171272243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab04156070 con 0x7fab04157770 2023-10-30T13:21:38.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7faaf9ffb640 1 -- 172.21.15.125:0/4171272243 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaf400fe70 con 0x7fab04157770 2023-10-30T13:21:38.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab0b446640 1 -- 172.21.15.125:0/4171272243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab041565f0 con 0x7fab04157770 2023-10-30T13:21:38.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7fab0b446640 1 -- 172.21.15.125:0/4171272243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab040c1260 con 0x7fab04157770 2023-10-30T13:21:38.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.165+0000 7faaf9ffb640 1 -- 172.21.15.125:0/4171272243 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faaf4015460 con 0x7fab04157770 2023-10-30T13:21:38.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.169+0000 7faaf9ffb640 1 --2- 172.21.15.125:0/4171272243 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faae80664a0 0x7faae8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:38.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.169+0000 7faaf9ffb640 1 -- 172.21.15.125:0/4171272243 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7faaf4085180 con 0x7fab04157770 2023-10-30T13:21:38.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.169+0000 7fab091e2640 1 --2- 172.21.15.125:0/4171272243 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faae80664a0 0x7faae8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:38.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.169+0000 7fab091e2640 1 --2- 172.21.15.125:0/4171272243 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faae80664a0 0x7faae8068960 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fab00000c10 tx=0x7fab00009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:38.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.173+0000 7faaf9ffb640 1 -- 172.21.15.125:0/4171272243 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faaf4051c00 con 0x7fab04157770 2023-10-30T13:21:38.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.353+0000 7fab0b446640 1 -- 172.21.15.125:0/4171272243 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab0404e170 con 0x7fab04157770 2023-10-30T13:21:38.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.353+0000 7faaf9ffb640 1 -- 172.21.15.125:0/4171272243 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+6550 (secure 0 0 0) 0x7faaf4055220 con 0x7fab04157770 2023-10-30T13:21:38.358 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:38.358 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:33.189409+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:38.359 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 247 2023-10-30T13:21:38.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.353+0000 7fab0b446640 1 -- 172.21.15.125:0/4171272243 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faae80664a0 msgr2=0x7faae8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:38.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.353+0000 7fab0b446640 1 --2- 172.21.15.125:0/4171272243 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faae80664a0 0x7faae8068960 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7fab00000c10 tx=0x7fab00009040 comp rx=0 tx=0).stop 2023-10-30T13:21:38.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.353+0000 7fab0b446640 1 -- 172.21.15.125:0/4171272243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab04157770 msgr2=0x7fab040c1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:38.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.353+0000 7fab0b446640 1 --2- 172.21.15.125:0/4171272243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab04157770 0x7fab040c1560 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7faaf4011800 tx=0x7faaf4011cc0 comp rx=0 tx=0).stop 2023-10-30T13:21:38.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.353+0000 7fab0b446640 1 -- 172.21.15.125:0/4171272243 shutdown_connections 2023-10-30T13:21:38.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.353+0000 7fab0b446640 1 --2- 172.21.15.125:0/4171272243 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faae80664a0 0x7faae8068960 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:38.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.353+0000 7fab0b446640 1 --2- 172.21.15.125:0/4171272243 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fab04156e10 0x7fab0415a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:38.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.353+0000 7fab0b446640 1 --2- 172.21.15.125:0/4171272243 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fab04157770 0x7fab040c1560 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:38.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.353+0000 7fab0b446640 1 --2- 172.21.15.125:0/4171272243 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faaf0005190 0x7fab041541f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:38.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.353+0000 7fab0b446640 1 -- 172.21.15.125:0/4171272243 >> 172.21.15.125:0/4171272243 conn(0x7fab040c7910 msgr2=0x7fab0406a270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:38.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.353+0000 7fab0b446640 1 -- 172.21.15.125:0/4171272243 shutdown_connections 2023-10-30T13:21:38.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:38.353+0000 7fab0b446640 1 -- 172.21.15.125:0/4171272243 wait complete. 2023-10-30T13:21:38.379 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:21:40.379 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:40.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.521+0000 7f810711e640 1 Processor -- start 2023-10-30T13:21:40.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.521+0000 7f810711e640 1 -- start start 2023-10-30T13:21:40.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.521+0000 7f810711e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f81001538f0 0x7f8100159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:40.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.521+0000 7f810711e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f810015cd50 0x7f81000c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:40.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.521+0000 7f810711e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f810015d300 con 0x7f81000cefa0 2023-10-30T13:21:40.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.521+0000 7f810711e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810015d5e0 con 0x7f810015cd50 2023-10-30T13:21:40.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.521+0000 7f810711e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810015d8c0 con 0x7f81001538f0 2023-10-30T13:21:40.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.521+0000 7f810591b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f810015cd50 0x7f81000c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:40.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.521+0000 7f810591b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f810015cd50 0x7f81000c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46518/0 (socket says 172.21.15.125:46518) 2023-10-30T13:21:40.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.521+0000 7f810591b640 1 -- 172.21.15.125:0/820014406 learned_addr learned my addr 172.21.15.125:0/820014406 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:40.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.521+0000 7f810611c640 1 --2- 172.21.15.125:0/820014406 >> v2:172.21.15.136:3300/0 conn(0x7f81001538f0 0x7f8100159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:40.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.525+0000 7f810591b640 1 -- 172.21.15.125:0/820014406 >> v2:172.21.15.136:3300/0 conn(0x7f81001538f0 msgr2=0x7f8100159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:40.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.525+0000 7f810591b640 1 --2- 172.21.15.125:0/820014406 >> v2:172.21.15.136:3300/0 conn(0x7f81001538f0 0x7f8100159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:40.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.525+0000 7f810591b640 1 -- 172.21.15.125:0/820014406 >> v1:172.21.15.125:6789/0 conn(0x7f81000cefa0 legacy=0x7f81000cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:40.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.525+0000 7f810591b640 1 -- 172.21.15.125:0/820014406 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810015dba0 con 0x7f810015cd50 2023-10-30T13:21:40.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.525+0000 7f810511a640 1 -- 172.21.15.125:0/820014406 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1594559086 0 0) 0x7f810015d300 con 0x7f81000cefa0 2023-10-30T13:21:40.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.525+0000 7f810591b640 1 --2- 172.21.15.125:0/820014406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810015cd50 0x7f81000c9f90 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f81000c9cb0 tx=0x7f80f001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b433fbd630e8558a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:40.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.525+0000 7f810511a640 1 -- 172.21.15.125:0/820014406 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f00a89f0 con 0x7f810015cd50 2023-10-30T13:21:40.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.525+0000 7f810511a640 1 -- 172.21.15.125:0/820014406 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f00a8b50 con 0x7f810015cd50 2023-10-30T13:21:40.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.525+0000 7f810511a640 1 -- 172.21.15.125:0/820014406 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f00b0440 con 0x7f810015cd50 2023-10-30T13:21:40.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 -- 172.21.15.125:0/820014406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810015cd50 msgr2=0x7f81000c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:40.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 --2- 172.21.15.125:0/820014406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810015cd50 0x7f81000c9f90 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7f81000c9cb0 tx=0x7f80f001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:21:40.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 -- 172.21.15.125:0/820014406 shutdown_connections 2023-10-30T13:21:40.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 --2- 172.21.15.125:0/820014406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810015cd50 0x7f81000c9f90 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:40.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 --2- 172.21.15.125:0/820014406 >> v2:172.21.15.136:3300/0 conn(0x7f81001538f0 0x7f8100159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:40.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 -- 172.21.15.125:0/820014406 >> 172.21.15.125:0/820014406 conn(0x7f810006a640 msgr2=0x7f8100065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:40.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 -- 172.21.15.125:0/820014406 shutdown_connections 2023-10-30T13:21:40.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 -- 172.21.15.125:0/820014406 wait complete. 2023-10-30T13:21:40.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 Processor -- start 2023-10-30T13:21:40.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 -- start start 2023-10-30T13:21:40.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81000cefa0 0x7f8100145690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:40.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81001538f0 0x7f81001519c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:40.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810015cd50 0x7f81000b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:40.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810015c5c0 con 0x7f81000cefa0 2023-10-30T13:21:40.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8100151530 con 0x7f810015cd50 2023-10-30T13:21:40.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810711e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81000b3dd0 con 0x7f81001538f0 2023-10-30T13:21:40.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810611c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81000cefa0 0x7f8100145690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:40.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810611c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81000cefa0 0x7f8100145690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46532/0 (socket says 172.21.15.125:46532) 2023-10-30T13:21:40.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810611c640 1 -- 172.21.15.125:0/3717922037 learned_addr learned my addr 172.21.15.125:0/3717922037 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:40.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810591b640 1 --2- 172.21.15.125:0/3717922037 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81001538f0 0x7f81001519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:40.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810691d640 1 --2- 172.21.15.125:0/3717922037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810015cd50 0x7f81000b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:40.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810611c640 1 -- 172.21.15.125:0/3717922037 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81001538f0 msgr2=0x7f81001519c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:40.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810611c640 1 --2- 172.21.15.125:0/3717922037 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81001538f0 0x7f81001519c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:40.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810611c640 1 -- 172.21.15.125:0/3717922037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810015cd50 msgr2=0x7f81000b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:40.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810611c640 1 --2- 172.21.15.125:0/3717922037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810015cd50 0x7f81000b49b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:40.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810611c640 1 -- 172.21.15.125:0/3717922037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f00094d0 con 0x7f81000cefa0 2023-10-30T13:21:40.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.529+0000 7f810611c640 1 --2- 172.21.15.125:0/3717922037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81000cefa0 0x7f8100145690 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00d830 tx=0x7f80fc00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:40.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.533+0000 7f80f6ffd640 1 -- 172.21.15.125:0/3717922037 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc00b6c0 con 0x7f81000cefa0 2023-10-30T13:21:40.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.533+0000 7f80f6ffd640 1 -- 172.21.15.125:0/3717922037 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc00b820 con 0x7f81000cefa0 2023-10-30T13:21:40.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.533+0000 7f80f6ffd640 1 -- 172.21.15.125:0/3717922037 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80fc00bb90 con 0x7f81000cefa0 2023-10-30T13:21:40.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.533+0000 7f810711e640 1 -- 172.21.15.125:0/3717922037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810015c810 con 0x7f81000cefa0 2023-10-30T13:21:40.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.533+0000 7f810711e640 1 -- 172.21.15.125:0/3717922037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81001511c0 con 0x7f81000cefa0 2023-10-30T13:21:40.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.533+0000 7f810711e640 1 -- 172.21.15.125:0/3717922037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810004e170 con 0x7f81000cefa0 2023-10-30T13:21:40.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.533+0000 7f80f6ffd640 1 -- 172.21.15.125:0/3717922037 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f80fc01e460 con 0x7f81000cefa0 2023-10-30T13:21:40.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.533+0000 7f80f6ffd640 1 --2- 172.21.15.125:0/3717922037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80e40664a0 0x7f80e4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:40.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.533+0000 7f80f6ffd640 1 -- 172.21.15.125:0/3717922037 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7f80fc085060 con 0x7f81000cefa0 2023-10-30T13:21:40.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.533+0000 7f810591b640 1 --2- 172.21.15.125:0/3717922037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80e40664a0 0x7f80e4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:40.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.537+0000 7f810591b640 1 --2- 172.21.15.125:0/3717922037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80e40664a0 0x7f80e4068960 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f80f00b2000 tx=0x7f80f0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:40.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.537+0000 7f80f6ffd640 1 -- 172.21.15.125:0/3717922037 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f80fc0b6090 con 0x7f81000cefa0 2023-10-30T13:21:40.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.717+0000 7f810711e640 1 -- 172.21.15.125:0/3717922037 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81000cfca0 con 0x7f81000cefa0 2023-10-30T13:21:40.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.717+0000 7f80f6ffd640 1 -- 172.21.15.125:0/3717922037 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+6550 (secure 0 0 0) 0x7f80fc051ae0 con 0x7f81000cefa0 2023-10-30T13:21:40.724 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:40.724 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:33.189409+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:40.725 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 247 2023-10-30T13:21:40.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.721+0000 7f810711e640 1 -- 172.21.15.125:0/3717922037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80e40664a0 msgr2=0x7f80e4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:40.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.721+0000 7f810711e640 1 --2- 172.21.15.125:0/3717922037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80e40664a0 0x7f80e4068960 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f80f00b2000 tx=0x7f80f0008040 comp rx=0 tx=0).stop 2023-10-30T13:21:40.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.721+0000 7f810711e640 1 -- 172.21.15.125:0/3717922037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81000cefa0 msgr2=0x7f8100145690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:40.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.721+0000 7f810711e640 1 --2- 172.21.15.125:0/3717922037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81000cefa0 0x7f8100145690 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00d830 tx=0x7f80fc00dcf0 comp rx=0 tx=0).stop 2023-10-30T13:21:40.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.721+0000 7f810711e640 1 -- 172.21.15.125:0/3717922037 shutdown_connections 2023-10-30T13:21:40.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.721+0000 7f810711e640 1 --2- 172.21.15.125:0/3717922037 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80e40664a0 0x7f80e4068960 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:40.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.721+0000 7f810711e640 1 --2- 172.21.15.125:0/3717922037 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810015cd50 0x7f81000b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:40.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.721+0000 7f810711e640 1 --2- 172.21.15.125:0/3717922037 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f81001538f0 0x7f81001519c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:40.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.721+0000 7f810711e640 1 --2- 172.21.15.125:0/3717922037 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f81000cefa0 0x7f8100145690 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:40.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.721+0000 7f810711e640 1 -- 172.21.15.125:0/3717922037 >> 172.21.15.125:0/3717922037 conn(0x7f810006a640 msgr2=0x7f81000caaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:40.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.721+0000 7f810711e640 1 -- 172.21.15.125:0/3717922037 shutdown_connections 2023-10-30T13:21:40.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:40.721+0000 7f810711e640 1 -- 172.21.15.125:0/3717922037 wait complete. 2023-10-30T13:21:40.745 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:21:42.746 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:42.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.889+0000 7f8f72572640 1 Processor -- start 2023-10-30T13:21:42.892 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.889+0000 7f8f72572640 1 -- start start 2023-10-30T13:21:42.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.889+0000 7f8f72572640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f6c0c6bd0 0x7f8f6c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:42.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.889+0000 7f8f72572640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c066cf0 con 0x7f8f6c06f7a0 2023-10-30T13:21:42.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.889+0000 7f8f72572640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c0c70d0 con 0x7f8f6c071110 2023-10-30T13:21:42.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.889+0000 7f8f72572640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f6c0700c0 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.889+0000 7f8f6b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f6c0c6bd0 0x7f8f6c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:42.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.889+0000 7f8f6b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f6c0c6bd0 0x7f8f6c06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33506/0 (socket says 172.21.15.125:33506) 2023-10-30T13:21:42.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.889+0000 7f8f6bfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8f6c06f7a0 0x7f8f6c069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40066/0 (socket says 172.21.15.125:40066) 2023-10-30T13:21:42.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.889+0000 7f8f6b7fe640 1 -- 172.21.15.125:0/217250095 learned_addr learned my addr 172.21.15.125:0/217250095 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:42.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.889+0000 7f8f6b7fe640 1 -- 172.21.15.125:0/217250095 >> v1:172.21.15.136:6789/0 conn(0x7f8f6c071110 legacy=0x7f8f6c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:42.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.889+0000 7f8f6b7fe640 1 -- 172.21.15.125:0/217250095 >> v1:172.21.15.125:6789/0 conn(0x7f8f6c06f7a0 legacy=0x7f8f6c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:42.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.889+0000 7f8f6affd640 1 -- 172.21.15.125:0/217250095 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038091933 0 0) 0x7f8f6c066cf0 con 0x7f8f6c06f7a0 2023-10-30T13:21:42.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.889+0000 7f8f6b7fe640 1 -- 172.21.15.125:0/217250095 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c0c7390 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.893+0000 7f8f6b7fe640 1 --2- 172.21.15.125:0/217250095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c0c6bd0 0x7f8f6c06fb80 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009870 tx=0x7f8f6001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5819a93b9905beaa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:42.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.893+0000 7f8f6affd640 1 -- 172.21.15.125:0/217250095 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f600a89f0 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.893+0000 7f8f6affd640 1 -- 172.21.15.125:0/217250095 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f600a8b50 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.893+0000 7f8f6affd640 1 -- 172.21.15.125:0/217250095 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f600b0440 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.893+0000 7f8f72572640 1 -- 172.21.15.125:0/217250095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c0c6bd0 msgr2=0x7f8f6c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:42.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.893+0000 7f8f72572640 1 --2- 172.21.15.125:0/217250095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c0c6bd0 0x7f8f6c06fb80 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009870 tx=0x7f8f6001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:21:42.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.893+0000 7f8f72572640 1 -- 172.21.15.125:0/217250095 shutdown_connections 2023-10-30T13:21:42.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.893+0000 7f8f72572640 1 --2- 172.21.15.125:0/217250095 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c0c6bd0 0x7f8f6c06fb80 unknown :-1 s=CLOSED pgs=2042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:42.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.893+0000 7f8f72572640 1 -- 172.21.15.125:0/217250095 >> 172.21.15.125:0/217250095 conn(0x7f8f6c0c7910 msgr2=0x7f8f6c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:42.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.893+0000 7f8f72572640 1 -- 172.21.15.125:0/217250095 shutdown_connections 2023-10-30T13:21:42.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.893+0000 7f8f72572640 1 -- 172.21.15.125:0/217250095 wait complete. 2023-10-30T13:21:42.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.893+0000 7f8f72572640 1 Processor -- start 2023-10-30T13:21:42.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f72572640 1 -- start start 2023-10-30T13:21:42.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f72572640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f6c06f7a0 0x7f8f6c1dd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:42.902 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f72572640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f6c071110 0x7f8f6c14c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:42.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f72572640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c0c6bd0 0x7f8f6c14ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:42.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f72572640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c1de2a0 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f72572640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c1ddf50 con 0x7f8f6c06f7a0 2023-10-30T13:21:42.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f72572640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c1f03b0 con 0x7f8f6c071110 2023-10-30T13:21:42.903 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f70b0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c0c6bd0 0x7f8f6c14ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:42.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f70b0f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c0c6bd0 0x7f8f6c14ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33520/0 (socket says 172.21.15.125:33520) 2023-10-30T13:21:42.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f70b0f640 1 -- 172.21.15.125:0/1112752870 learned_addr learned my addr 172.21.15.125:0/1112752870 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:42.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f6bfff640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f6c06f7a0 0x7f8f6c1dd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:42.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f6b7fe640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f6c071110 0x7f8f6c14c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:42.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f70b0f640 1 -- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f6c071110 msgr2=0x7f8f6c14c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:42.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f70b0f640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f6c071110 0x7f8f6c14c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:42.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f70b0f640 1 -- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f6c06f7a0 msgr2=0x7f8f6c1dd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:42.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f70b0f640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f6c06f7a0 0x7f8f6c1dd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:42.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f70b0f640 1 -- 172.21.15.125:0/1112752870 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f600094d0 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f6b7fe640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f6c071110 0x7f8f6c14c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:42.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f70b0f640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c0c6bd0 0x7f8f6c14ff90 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400b9a0 tx=0x7f8f6400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:42.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f68ff9640 1 -- 172.21.15.125:0/1112752870 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f6400cba0 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f68ff9640 1 -- 172.21.15.125:0/1112752870 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6400cd00 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f72572640 1 -- 172.21.15.125:0/1112752870 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6c1ddd00 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f68ff9640 1 -- 172.21.15.125:0/1112752870 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f64014680 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f72572640 1 -- 172.21.15.125:0/1112752870 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6c1f08a0 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.897+0000 7f8f72572640 1 -- 172.21.15.125:0/1112752870 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6c04e170 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.901+0000 7f8f68ff9640 1 -- 172.21.15.125:0/1112752870 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f6401d3b0 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.901+0000 7f8f68ff9640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f580664d0 0x7f8f58068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:42.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.901+0000 7f8f68ff9640 1 -- 172.21.15.125:0/1112752870 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7f8f64084540 con 0x7f8f6c0c6bd0 2023-10-30T13:21:42.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.901+0000 7f8f6bfff640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f580664d0 0x7f8f58068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:42.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.901+0000 7f8f6bfff640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f580664d0 0x7f8f58068990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c000f70 tx=0x7f8f5c007df0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:42.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:42.905+0000 7f8f68ff9640 1 -- 172.21.15.125:0/1112752870 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8f640510d0 con 0x7f8f6c0c6bd0 2023-10-30T13:21:43.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:43.085+0000 7f8f72572640 1 -- 172.21.15.125:0/1112752870 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f6c073e10 con 0x7f8f6c0c6bd0 2023-10-30T13:21:43.090 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:43.085+0000 7f8f68ff9640 1 -- 172.21.15.125:0/1112752870 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+6550 (secure 0 0 0) 0x7f8f640546f0 con 0x7f8f6c0c6bd0 2023-10-30T13:21:43.090 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:43.090 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:33.189409+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:43.091 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 247 2023-10-30T13:21:43.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:43.085+0000 7f8f72572640 1 -- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f580664d0 msgr2=0x7f8f58068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:43.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:43.085+0000 7f8f72572640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f580664d0 0x7f8f58068990 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c000f70 tx=0x7f8f5c007df0 comp rx=0 tx=0).stop 2023-10-30T13:21:43.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:43.085+0000 7f8f72572640 1 -- 172.21.15.125:0/1112752870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c0c6bd0 msgr2=0x7f8f6c14ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:43.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:43.085+0000 7f8f72572640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c0c6bd0 0x7f8f6c14ff90 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f8f6400b9a0 tx=0x7f8f6400be60 comp rx=0 tx=0).stop 2023-10-30T13:21:43.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:43.089+0000 7f8f72572640 1 -- 172.21.15.125:0/1112752870 shutdown_connections 2023-10-30T13:21:43.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:43.089+0000 7f8f72572640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f580664d0 0x7f8f58068990 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:43.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:43.089+0000 7f8f72572640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f6c0c6bd0 0x7f8f6c14ff90 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:43.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:43.089+0000 7f8f72572640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f6c071110 0x7f8f6c14c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:43.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:43.089+0000 7f8f72572640 1 --2- 172.21.15.125:0/1112752870 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f6c06f7a0 0x7f8f6c1dd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:43.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:43.089+0000 7f8f72572640 1 -- 172.21.15.125:0/1112752870 >> 172.21.15.125:0/1112752870 conn(0x7f8f6c0c7910 msgr2=0x7f8f6c0c3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:43.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:43.089+0000 7f8f72572640 1 -- 172.21.15.125:0/1112752870 shutdown_connections 2023-10-30T13:21:43.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:43.089+0000 7f8f72572640 1 -- 172.21.15.125:0/1112752870 wait complete. 2023-10-30T13:21:43.112 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:21:45.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:45.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.253+0000 7fe56c2c9640 1 Processor -- start 2023-10-30T13:21:45.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.253+0000 7fe56c2c9640 1 -- start start 2023-10-30T13:21:45.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe56c2c9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fe56415d6f0 0x7fe56415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:45.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe56c2c9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5640cb800 con 0x7fe5641538a0 2023-10-30T13:21:45.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe56c2c9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5640cb330 con 0x7fe5640cf160 2023-10-30T13:21:45.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe56c2c9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5640cad70 con 0x7fe56415d6f0 2023-10-30T13:21:45.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe56a065640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fe5641538a0 0x7fe5640ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:40070/0 (socket says 172.21.15.125:40070) 2023-10-30T13:21:45.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe56a065640 1 -- 172.21.15.125:0/3981722994 learned_addr learned my addr 172.21.15.125:0/3981722994 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:21:45.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569864640 1 --2- 172.21.15.125:0/3981722994 >> v2:172.21.15.136:3300/0 conn(0x7fe56415d6f0 0x7fe56415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:45.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 -- 172.21.15.125:0/3981722994 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285089599 0 0) 0x7fe5640cb800 con 0x7fe5641538a0 2023-10-30T13:21:45.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 -- 172.21.15.125:0/3981722994 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe550003530 con 0x7fe5641538a0 2023-10-30T13:21:45.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 -- 172.21.15.125:0/3981722994 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4013699016 0 0) 0x7fe550003530 con 0x7fe5641538a0 2023-10-30T13:21:45.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 -- 172.21.15.125:0/3981722994 >> v2:172.21.15.136:3300/0 conn(0x7fe56415d6f0 msgr2=0x7fe56415a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:45.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 --2- 172.21.15.125:0/3981722994 >> v2:172.21.15.136:3300/0 conn(0x7fe56415d6f0 0x7fe56415a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:45.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 -- 172.21.15.125:0/3981722994 >> v1:172.21.15.136:6789/0 conn(0x7fe5640cf160 legacy=0x7fe564164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:45.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 -- 172.21.15.125:0/3981722994 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5640cb050 con 0x7fe5641538a0 2023-10-30T13:21:45.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 -- 172.21.15.125:0/3981722994 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fe560001680 con 0x7fe5641538a0 2023-10-30T13:21:45.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 -- 172.21.15.125:0/3981722994 >> v1:172.21.15.125:6789/0 conn(0x7fe5641538a0 legacy=0x7fe5640ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:45.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 --2- 172.21.15.125:0/3981722994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe550003980 0x7fe550003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:45.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 --2- 172.21.15.125:0/3981722994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5500047d0 0x7fe550004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:45.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 --2- 172.21.15.125:0/3981722994 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe550005190 0x7fe55000b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:45.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 -- 172.21.15.125:0/3981722994 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5640cad70 con 0x7fe5500047d0 2023-10-30T13:21:45.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 -- 172.21.15.125:0/3981722994 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55000c040 con 0x7fe550003980 2023-10-30T13:21:45.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569063640 1 -- 172.21.15.125:0/3981722994 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe55000c200 con 0x7fe550005190 2023-10-30T13:21:45.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569864640 1 --2- 172.21.15.125:0/3981722994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5500047d0 0x7fe550004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:45.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe56a065640 1 --2- 172.21.15.125:0/3981722994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe550003980 0x7fe550003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:45.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe56a866640 1 --2- 172.21.15.125:0/3981722994 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe550005190 0x7fe55000b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:45.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569864640 1 -- 172.21.15.125:0/3981722994 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe550005190 msgr2=0x7fe55000b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:45.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569864640 1 --2- 172.21.15.125:0/3981722994 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe550005190 0x7fe55000b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:45.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569864640 1 -- 172.21.15.125:0/3981722994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe550003980 msgr2=0x7fe550003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:45.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569864640 1 --2- 172.21.15.125:0/3981722994 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe550003980 0x7fe550003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:45.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe569864640 1 -- 172.21.15.125:0/3981722994 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe550003530 con 0x7fe5500047d0 2023-10-30T13:21:45.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.257+0000 7fe56a866640 1 -- 172.21.15.125:0/3981722994 reap_dead start 2023-10-30T13:21:45.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.261+0000 7fe569864640 1 --2- 172.21.15.125:0/3981722994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5500047d0 0x7fe550004c50 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fe554009ae0 tx=0x7fe55401ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b11bc5da6a6e51 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:45.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.261+0000 7fe569063640 1 -- 172.21.15.125:0/3981722994 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5540a5660 con 0x7fe5500047d0 2023-10-30T13:21:45.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.261+0000 7fe569063640 1 -- 172.21.15.125:0/3981722994 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5540a57c0 con 0x7fe5500047d0 2023-10-30T13:21:45.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.261+0000 7fe569063640 1 -- 172.21.15.125:0/3981722994 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5540a5b70 con 0x7fe5500047d0 2023-10-30T13:21:45.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.261+0000 7fe56c2c9640 1 -- 172.21.15.125:0/3981722994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5500047d0 msgr2=0x7fe550004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:45.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.261+0000 7fe56c2c9640 1 --2- 172.21.15.125:0/3981722994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5500047d0 0x7fe550004c50 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7fe554009ae0 tx=0x7fe55401ca70 comp rx=0 tx=0).stop 2023-10-30T13:21:45.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.261+0000 7fe56c2c9640 1 -- 172.21.15.125:0/3981722994 shutdown_connections 2023-10-30T13:21:45.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.261+0000 7fe56c2c9640 1 --2- 172.21.15.125:0/3981722994 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe5500047d0 0x7fe550004c50 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:45.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.261+0000 7fe56c2c9640 1 -- 172.21.15.125:0/3981722994 >> 172.21.15.125:0/3981722994 conn(0x7fe56406bfe0 msgr2=0x7fe56406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:45.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.261+0000 7fe56c2c9640 1 -- 172.21.15.125:0/3981722994 shutdown_connections 2023-10-30T13:21:45.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.261+0000 7fe56c2c9640 1 -- 172.21.15.125:0/3981722994 wait complete. 2023-10-30T13:21:45.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56c2c9640 1 Processor -- start 2023-10-30T13:21:45.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56c2c9640 1 -- start start 2023-10-30T13:21:45.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56c2c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5500047d0 0x7fe564152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:45.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56c2c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5641eef30 0x7fe5641ef370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:45.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56c2c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe56414ce90 0x7fe56414d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:45.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56c2c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5641f3120 con 0x7fe56414ce90 2023-10-30T13:21:45.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56c2c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5641e2e60 con 0x7fe5641eef30 2023-10-30T13:21:45.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56c2c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe56415b410 con 0x7fe5500047d0 2023-10-30T13:21:45.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56a866640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe56414ce90 0x7fe56414d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:45.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56a866640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe56414ce90 0x7fe56414d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33534/0 (socket says 172.21.15.125:33534) 2023-10-30T13:21:45.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56a866640 1 -- 172.21.15.125:0/1611302230 learned_addr learned my addr 172.21.15.125:0/1611302230 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:45.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56a065640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5500047d0 0x7fe564152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:45.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe569864640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5641eef30 0x7fe5641ef370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:45.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56a866640 1 -- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5500047d0 msgr2=0x7fe564152070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:45.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56a866640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5500047d0 0x7fe564152070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:45.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56a866640 1 -- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5641eef30 msgr2=0x7fe5641ef370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:45.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56a866640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5641eef30 0x7fe5641ef370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:45.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56a866640 1 -- 172.21.15.125:0/1611302230 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5641e3100 con 0x7fe56414ce90 2023-10-30T13:21:45.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe569864640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5641eef30 0x7fe5641ef370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:45.276 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56a866640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe56414ce90 0x7fe56414d310 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fe564072ce0 tx=0x7fe55c00fc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:45.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe55affd640 1 -- 172.21.15.125:0/1611302230 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c00fec0 con 0x7fe56414ce90 2023-10-30T13:21:45.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe55affd640 1 -- 172.21.15.125:0/1611302230 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe55c003b90 con 0x7fe56414ce90 2023-10-30T13:21:45.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe55affd640 1 -- 172.21.15.125:0/1611302230 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe55c0050d0 con 0x7fe56414ce90 2023-10-30T13:21:45.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56c2c9640 1 -- 172.21.15.125:0/1611302230 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5641f24a0 con 0x7fe56414ce90 2023-10-30T13:21:45.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56c2c9640 1 -- 172.21.15.125:0/1611302230 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5641f2a60 con 0x7fe56414ce90 2023-10-30T13:21:45.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.265+0000 7fe56c2c9640 1 -- 172.21.15.125:0/1611302230 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe56404e170 con 0x7fe56414ce90 2023-10-30T13:21:45.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.269+0000 7fe55affd640 1 -- 172.21.15.125:0/1611302230 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fe55c005230 con 0x7fe56414ce90 2023-10-30T13:21:45.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.269+0000 7fe55affd640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5480664d0 0x7fe548068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:45.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.269+0000 7fe55affd640 1 -- 172.21.15.125:0/1611302230 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7fe55c084800 con 0x7fe56414ce90 2023-10-30T13:21:45.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.269+0000 7fe56a065640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5480664d0 0x7fe548068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:45.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.269+0000 7fe56a065640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5480664d0 0x7fe548068990 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fe560001060 tx=0x7fe560009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:45.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.273+0000 7fe55affd640 1 -- 172.21.15.125:0/1611302230 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fe55c0511f0 con 0x7fe56414ce90 2023-10-30T13:21:45.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.453+0000 7fe56c2c9640 1 -- 172.21.15.125:0/1611302230 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5640cfe10 con 0x7fe56414ce90 2023-10-30T13:21:45.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.453+0000 7fe55affd640 1 -- 172.21.15.125:0/1611302230 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+6550 (secure 0 0 0) 0x7fe55c054810 con 0x7fe56414ce90 2023-10-30T13:21:45.458 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:45.458 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:33.189409+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:45.459 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 247 2023-10-30T13:21:45.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.453+0000 7fe56c2c9640 1 -- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5480664d0 msgr2=0x7fe548068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:45.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.453+0000 7fe56c2c9640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5480664d0 0x7fe548068990 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7fe560001060 tx=0x7fe560009040 comp rx=0 tx=0).stop 2023-10-30T13:21:45.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.453+0000 7fe56c2c9640 1 -- 172.21.15.125:0/1611302230 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe56414ce90 msgr2=0x7fe56414d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:45.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.453+0000 7fe56c2c9640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe56414ce90 0x7fe56414d310 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7fe564072ce0 tx=0x7fe55c00fc70 comp rx=0 tx=0).stop 2023-10-30T13:21:45.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.453+0000 7fe56c2c9640 1 -- 172.21.15.125:0/1611302230 shutdown_connections 2023-10-30T13:21:45.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.453+0000 7fe56c2c9640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fe5480664d0 0x7fe548068990 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:45.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.453+0000 7fe56c2c9640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fe56414ce90 0x7fe56414d310 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:45.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.453+0000 7fe56c2c9640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fe5641eef30 0x7fe5641ef370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:45.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.453+0000 7fe56c2c9640 1 --2- 172.21.15.125:0/1611302230 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fe5500047d0 0x7fe564152070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:45.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.453+0000 7fe56c2c9640 1 -- 172.21.15.125:0/1611302230 >> 172.21.15.125:0/1611302230 conn(0x7fe56406bfe0 msgr2=0x7fe56406c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:45.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.453+0000 7fe56c2c9640 1 -- 172.21.15.125:0/1611302230 shutdown_connections 2023-10-30T13:21:45.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:45.453+0000 7fe56c2c9640 1 -- 172.21.15.125:0/1611302230 wait complete. 2023-10-30T13:21:45.479 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:21:47.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:47.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.621+0000 7f9144246640 1 Processor -- start 2023-10-30T13:21:47.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.621+0000 7f9144246640 1 -- start start 2023-10-30T13:21:47.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.621+0000 7f9144246640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f913c06f7a0 0x7f913c06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:47.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.621+0000 7f9144246640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f913c0c6bd0 0x7f913c069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:47.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.621+0000 7f9144246640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c06a3d0 con 0x7f913c071110 2023-10-30T13:21:47.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.621+0000 7f9144246640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f913c0700c0 con 0x7f913c0c6bd0 2023-10-30T13:21:47.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.621+0000 7f9144246640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c066cf0 con 0x7f913c06f7a0 2023-10-30T13:21:47.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.621+0000 7f91417e1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f913c0c6bd0 0x7f913c069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:47.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.621+0000 7f91417e1640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f913c0c6bd0 0x7f913c069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33550/0 (socket says 172.21.15.125:33550) 2023-10-30T13:21:47.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.621+0000 7f91417e1640 1 -- 172.21.15.125:0/1705252276 learned_addr learned my addr 172.21.15.125:0/1705252276 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:47.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.625+0000 7f9141fe2640 1 --2- 172.21.15.125:0/1705252276 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f913c06f7a0 0x7f913c06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:47.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.625+0000 7f91417e1640 1 -- 172.21.15.125:0/1705252276 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f913c06f7a0 msgr2=0x7f913c06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:47.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.625+0000 7f91417e1640 1 --2- 172.21.15.125:0/1705252276 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f913c06f7a0 0x7f913c06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:47.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.625+0000 7f91417e1640 1 -- 172.21.15.125:0/1705252276 >> v1:172.21.15.125:6789/0 conn(0x7f913c071110 legacy=0x7f913c158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:47.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.625+0000 7f91417e1640 1 -- 172.21.15.125:0/1705252276 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c0c70d0 con 0x7f913c0c6bd0 2023-10-30T13:21:47.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.625+0000 7f9140fe0640 1 -- 172.21.15.125:0/1705252276 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999560160 0 0) 0x7f913c06a3d0 con 0x7f913c071110 2023-10-30T13:21:47.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.625+0000 7f9141fe2640 1 --2- 172.21.15.125:0/1705252276 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f913c06f7a0 0x7f913c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:47.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.625+0000 7f91417e1640 1 --2- 172.21.15.125:0/1705252276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f913c0c6bd0 0x7f913c069e90 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f912c009870 tx=0x7f912c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719c3507c64c2918 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:47.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.625+0000 7f9140fe0640 1 -- 172.21.15.125:0/1705252276 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c0a8a30 con 0x7f913c0c6bd0 2023-10-30T13:21:47.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.625+0000 7f9140fe0640 1 -- 172.21.15.125:0/1705252276 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912c0a8b90 con 0x7f913c0c6bd0 2023-10-30T13:21:47.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.625+0000 7f9140fe0640 1 -- 172.21.15.125:0/1705252276 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c0b0440 con 0x7f913c0c6bd0 2023-10-30T13:21:47.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 -- 172.21.15.125:0/1705252276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f913c0c6bd0 msgr2=0x7f913c069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:47.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 --2- 172.21.15.125:0/1705252276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f913c0c6bd0 0x7f913c069e90 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f912c009870 tx=0x7f912c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:21:47.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 -- 172.21.15.125:0/1705252276 shutdown_connections 2023-10-30T13:21:47.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 --2- 172.21.15.125:0/1705252276 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f913c0c6bd0 0x7f913c069e90 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:47.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 --2- 172.21.15.125:0/1705252276 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f913c06f7a0 0x7f913c06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:47.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 -- 172.21.15.125:0/1705252276 >> 172.21.15.125:0/1705252276 conn(0x7f913c0c7910 msgr2=0x7f913c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:47.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 -- 172.21.15.125:0/1705252276 shutdown_connections 2023-10-30T13:21:47.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 -- 172.21.15.125:0/1705252276 wait complete. 2023-10-30T13:21:47.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 Processor -- start 2023-10-30T13:21:47.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 -- start start 2023-10-30T13:21:47.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f913c06f7a0 0x7f913c15b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:47.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f913c071110 0x7f913c1f2740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:47.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f913c0c6bd0 0x7f913c159dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:47.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c15bb20 con 0x7f913c0c6bd0 2023-10-30T13:21:47.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c15b150 con 0x7f913c071110 2023-10-30T13:21:47.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9144246640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c1f2c80 con 0x7f913c06f7a0 2023-10-30T13:21:47.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f91427e3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f913c0c6bd0 0x7f913c159dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:47.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f91427e3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f913c0c6bd0 0x7f913c159dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33556/0 (socket says 172.21.15.125:33556) 2023-10-30T13:21:47.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f91427e3640 1 -- 172.21.15.125:0/4249291374 learned_addr learned my addr 172.21.15.125:0/4249291374 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:47.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f9141fe2640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f913c06f7a0 0x7f913c15b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:47.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f91417e1640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f913c071110 0x7f913c1f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:47.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f91427e3640 1 -- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f913c06f7a0 msgr2=0x7f913c15b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:47.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f91427e3640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f913c06f7a0 0x7f913c15b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:47.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f91427e3640 1 -- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f913c071110 msgr2=0x7f913c1f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:47.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f91427e3640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f913c071110 0x7f913c1f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:47.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f91427e3640 1 -- 172.21.15.125:0/4249291374 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912c0094d0 con 0x7f913c0c6bd0 2023-10-30T13:21:47.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f91417e1640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f913c071110 0x7f913c1f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:47.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.629+0000 7f91427e3640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f913c0c6bd0 0x7f913c159dd0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f913400c7b0 tx=0x7f913400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:47.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.633+0000 7f91327fc640 1 -- 172.21.15.125:0/4249291374 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9134003db0 con 0x7f913c0c6bd0 2023-10-30T13:21:47.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.633+0000 7f91327fc640 1 -- 172.21.15.125:0/4249291374 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913400ce60 con 0x7f913c0c6bd0 2023-10-30T13:21:47.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.633+0000 7f9144246640 1 -- 172.21.15.125:0/4249291374 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c0af270 con 0x7f913c0c6bd0 2023-10-30T13:21:47.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.633+0000 7f91327fc640 1 -- 172.21.15.125:0/4249291374 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340052b0 con 0x7f913c0c6bd0 2023-10-30T13:21:47.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.633+0000 7f9144246640 1 -- 172.21.15.125:0/4249291374 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c0af830 con 0x7f913c0c6bd0 2023-10-30T13:21:47.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.633+0000 7f9144246640 1 -- 172.21.15.125:0/4249291374 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c15b2e0 con 0x7f913c0c6bd0 2023-10-30T13:21:47.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.633+0000 7f91327fc640 1 -- 172.21.15.125:0/4249291374 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9134005410 con 0x7f913c0c6bd0 2023-10-30T13:21:47.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.633+0000 7f91327fc640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91200664a0 0x7f9120068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:47.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.633+0000 7f91327fc640 1 -- 172.21.15.125:0/4249291374 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7f91340846a0 con 0x7f913c0c6bd0 2023-10-30T13:21:47.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.633+0000 7f9141fe2640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91200664a0 0x7f9120068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:47.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.637+0000 7f9141fe2640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91200664a0 0x7f9120068960 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f913800ba10 tx=0x7f9138008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:47.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.637+0000 7f91327fc640 1 -- 172.21.15.125:0/4249291374 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9134051090 con 0x7f913c0c6bd0 2023-10-30T13:21:47.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.821+0000 7f9144246640 1 -- 172.21.15.125:0/4249291374 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f913c04e170 con 0x7f913c0c6bd0 2023-10-30T13:21:47.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.825+0000 7f91327fc640 1 -- 172.21.15.125:0/4249291374 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+6550 (secure 0 0 0) 0x7f91340546b0 con 0x7f913c0c6bd0 2023-10-30T13:21:47.829 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:47.829 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:33.189409+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:47.829 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 247 2023-10-30T13:21:47.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.825+0000 7f9144246640 1 -- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91200664a0 msgr2=0x7f9120068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:47.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.825+0000 7f9144246640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91200664a0 0x7f9120068960 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f913800ba10 tx=0x7f9138008040 comp rx=0 tx=0).stop 2023-10-30T13:21:47.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.825+0000 7f9144246640 1 -- 172.21.15.125:0/4249291374 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f913c0c6bd0 msgr2=0x7f913c159dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:47.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.825+0000 7f9144246640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f913c0c6bd0 0x7f913c159dd0 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7f913400c7b0 tx=0x7f913400cc70 comp rx=0 tx=0).stop 2023-10-30T13:21:47.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.825+0000 7f9144246640 1 -- 172.21.15.125:0/4249291374 shutdown_connections 2023-10-30T13:21:47.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.825+0000 7f9144246640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91200664a0 0x7f9120068960 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:47.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.825+0000 7f9144246640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f913c0c6bd0 0x7f913c159dd0 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:47.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.825+0000 7f9144246640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f913c071110 0x7f913c1f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:47.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.825+0000 7f9144246640 1 --2- 172.21.15.125:0/4249291374 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f913c06f7a0 0x7f913c15b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:47.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.825+0000 7f9144246640 1 -- 172.21.15.125:0/4249291374 >> 172.21.15.125:0/4249291374 conn(0x7f913c0c7910 msgr2=0x7f913c0c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:47.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.825+0000 7f9144246640 1 -- 172.21.15.125:0/4249291374 shutdown_connections 2023-10-30T13:21:47.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:47.825+0000 7f9144246640 1 -- 172.21.15.125:0/4249291374 wait complete. 2023-10-30T13:21:47.848 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:21:47.850 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:21:47.899 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:21:49.851 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:49.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4d510f640 1 Processor -- start 2023-10-30T13:21:49.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4d510f640 1 -- start start 2023-10-30T13:21:49.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4d510f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc4d00c6b80 0x7fc4d00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:49.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4d510f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4d013f4a0 0x7fc4d00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:49.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4d510f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d00c0a50 con 0x7fc4d00c6f60 2023-10-30T13:21:50.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4d510f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4d006f6e0 con 0x7fc4d00c6b80 2023-10-30T13:21:50.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4d510f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d006f9c0 con 0x7fc4d013f4a0 2023-10-30T13:21:50.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4ced9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc4d00c6b80 0x7fc4d00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:50.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4cf59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4d013f4a0 0x7fc4d00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:50.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4ced9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc4d00c6b80 0x7fc4d00bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:56652/0 (socket says 172.21.15.125:56652) 2023-10-30T13:21:50.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4ced9d640 1 -- 172.21.15.125:0/3672722124 learned_addr learned my addr 172.21.15.125:0/3672722124 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:50.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4ced9d640 1 -- 172.21.15.125:0/3672722124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4d013f4a0 msgr2=0x7fc4d00c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:50.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4ced9d640 1 --2- 172.21.15.125:0/3672722124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4d013f4a0 0x7fc4d00c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:50.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4ced9d640 1 -- 172.21.15.125:0/3672722124 >> v1:172.21.15.136:6789/0 conn(0x7fc4d00c6f60 legacy=0x7fc4d013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:50.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.993+0000 7fc4ced9d640 1 -- 172.21.15.125:0/3672722124 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d006fca0 con 0x7fc4d00c6b80 2023-10-30T13:21:50.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.997+0000 7fc4ced9d640 1 --2- 172.21.15.125:0/3672722124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4d00c6b80 0x7fc4d00bfec0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000bb10 tx=0x7fc4c001ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=492aaa587b81d8bb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:50.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.997+0000 7fc4ce59c640 1 -- 172.21.15.125:0/3672722124 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c00a7bf0 con 0x7fc4d00c6b80 2023-10-30T13:21:50.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.997+0000 7fc4ce59c640 1 -- 172.21.15.125:0/3672722124 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c00a7d50 con 0x7fc4d00c6b80 2023-10-30T13:21:50.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.997+0000 7fc4ce59c640 1 -- 172.21.15.125:0/3672722124 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c00af660 con 0x7fc4d00c6b80 2023-10-30T13:21:50.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.997+0000 7fc4d510f640 1 -- 172.21.15.125:0/3672722124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4d00c6b80 msgr2=0x7fc4d00bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:50.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.997+0000 7fc4d510f640 1 --2- 172.21.15.125:0/3672722124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4d00c6b80 0x7fc4d00bfec0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fc4c000bb10 tx=0x7fc4c001ec10 comp rx=0 tx=0).stop 2023-10-30T13:21:50.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.997+0000 7fc4d510f640 1 -- 172.21.15.125:0/3672722124 shutdown_connections 2023-10-30T13:21:50.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.997+0000 7fc4d510f640 1 --2- 172.21.15.125:0/3672722124 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4d013f4a0 0x7fc4d00c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:50.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.997+0000 7fc4d510f640 1 --2- 172.21.15.125:0/3672722124 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4d00c6b80 0x7fc4d00bfec0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:50.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:49.997+0000 7fc4d510f640 1 -- 172.21.15.125:0/3672722124 >> 172.21.15.125:0/3672722124 conn(0x7fc4d00c7910 msgr2=0x7fc4d00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:50.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4d510f640 1 -- 172.21.15.125:0/3672722124 shutdown_connections 2023-10-30T13:21:50.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4d510f640 1 -- 172.21.15.125:0/3672722124 wait complete. 2023-10-30T13:21:50.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4d510f640 1 Processor -- start 2023-10-30T13:21:50.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4d510f640 1 -- start start 2023-10-30T13:21:50.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4d510f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4d00c6b80 0x7fc4d01ee930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:50.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4d510f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4d00c6f60 0x7fc4d013c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:50.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4d510f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4d013f4a0 0x7fc4d0130970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:50.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4d510f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d013be30 con 0x7fc4d00c6f60 2023-10-30T13:21:50.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4d510f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d00ca660 con 0x7fc4d00c6b80 2023-10-30T13:21:50.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4d510f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d013c080 con 0x7fc4d013f4a0 2023-10-30T13:21:50.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4c659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4d00c6f60 0x7fc4d013c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:50.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4c659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4d00c6f60 0x7fc4d013c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:33564/0 (socket says 172.21.15.125:33564) 2023-10-30T13:21:50.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4c659c640 1 -- 172.21.15.125:0/1226279336 learned_addr learned my addr 172.21.15.125:0/1226279336 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:50.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4ced9d640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4d00c6b80 0x7fc4d01ee930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:50.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4cf59e640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4d013f4a0 0x7fc4d0130970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:50.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4c659c640 1 -- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4d013f4a0 msgr2=0x7fc4d0130970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:50.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4c659c640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4d013f4a0 0x7fc4d0130970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:50.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4c659c640 1 -- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4d00c6b80 msgr2=0x7fc4d01ee930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:50.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4c659c640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4d00c6b80 0x7fc4d01ee930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:50.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4c659c640 1 -- 172.21.15.125:0/1226279336 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c000b770 con 0x7fc4d00c6f60 2023-10-30T13:21:50.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.001+0000 7fc4cf59e640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4d013f4a0 0x7fc4d0130970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:50.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.005+0000 7fc4c659c640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4d00c6f60 0x7fc4d013c970 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800c420 tx=0x7fc4b800c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:50.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.005+0000 7fc4c7fff640 1 -- 172.21.15.125:0/1226279336 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b80052d0 con 0x7fc4d00c6f60 2023-10-30T13:21:50.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.005+0000 7fc4c7fff640 1 -- 172.21.15.125:0/1226279336 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4b8005430 con 0x7fc4d00c6f60 2023-10-30T13:21:50.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.005+0000 7fc4d510f640 1 -- 172.21.15.125:0/1226279336 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d01ede50 con 0x7fc4d00c6f60 2023-10-30T13:21:50.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.005+0000 7fc4c7fff640 1 -- 172.21.15.125:0/1226279336 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4b8007260 con 0x7fc4d00c6f60 2023-10-30T13:21:50.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.005+0000 7fc4d510f640 1 -- 172.21.15.125:0/1226279336 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d01ee410 con 0x7fc4d00c6f60 2023-10-30T13:21:50.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.005+0000 7fc4d510f640 1 -- 172.21.15.125:0/1226279336 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d01ee600 con 0x7fc4d00c6f60 2023-10-30T13:21:50.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.005+0000 7fc4c7fff640 1 -- 172.21.15.125:0/1226279336 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc4b80073c0 con 0x7fc4d00c6f60 2023-10-30T13:21:50.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.005+0000 7fc4c7fff640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4b00664a0 0x7fc4b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:50.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.005+0000 7fc4c7fff640 1 -- 172.21.15.125:0/1226279336 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8232+0+0 (secure 0 0 0) 0x7fc4b8083c70 con 0x7fc4d00c6f60 2023-10-30T13:21:50.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.005+0000 7fc4ced9d640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4b00664a0 0x7fc4b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:50.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.009+0000 7fc4ced9d640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4b00664a0 0x7fc4b0068960 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fc4c00b1000 tx=0x7fc4c00a9070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:50.017 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.009+0000 7fc4c7fff640 1 -- 172.21.15.125:0/1226279336 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc4b8050710 con 0x7fc4d00c6f60 2023-10-30T13:21:50.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.193+0000 7fc4d510f640 1 -- 172.21.15.125:0/1226279336 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc4d004e170 con 0x7fc4d00c6f60 2023-10-30T13:21:50.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.193+0000 7fc4c7fff640 1 -- 172.21.15.125:0/1226279336 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 247 v247) v1 ==== 95+0+6550 (secure 0 0 0) 0x7fc4b8053d30 con 0x7fc4d00c6f60 2023-10-30T13:21:50.198 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:50.198 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":247,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8033,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":246}],"filesystems":[{"mdsmap":{"epoch":247,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:33.189409+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8971,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8971":{"gid":8971,"name":"f","rank":0,"incarnation":230,"state":"up:active","state_seq":32,"addr":"172.21.15.136:6839/4284345684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":4284345684},{"type":"v1","addr":"172.21.15.136:6839","nonce":4284345684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:50.200 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 247 2023-10-30T13:21:50.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.193+0000 7fc4d510f640 1 -- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4b00664a0 msgr2=0x7fc4b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:50.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.193+0000 7fc4d510f640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4b00664a0 0x7fc4b0068960 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fc4c00b1000 tx=0x7fc4c00a9070 comp rx=0 tx=0).stop 2023-10-30T13:21:50.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.193+0000 7fc4d510f640 1 -- 172.21.15.125:0/1226279336 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4d00c6f60 msgr2=0x7fc4d013c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:50.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.193+0000 7fc4d510f640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4d00c6f60 0x7fc4d013c970 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fc4b800c420 tx=0x7fc4b800c8e0 comp rx=0 tx=0).stop 2023-10-30T13:21:50.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.193+0000 7fc4d510f640 1 -- 172.21.15.125:0/1226279336 shutdown_connections 2023-10-30T13:21:50.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.193+0000 7fc4d510f640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4b00664a0 0x7fc4b0068960 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:50.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.193+0000 7fc4d510f640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4d013f4a0 0x7fc4d0130970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:50.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.193+0000 7fc4d510f640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4d00c6f60 0x7fc4d013c970 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:50.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.193+0000 7fc4d510f640 1 --2- 172.21.15.125:0/1226279336 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4d00c6b80 0x7fc4d01ee930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:50.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.193+0000 7fc4d510f640 1 -- 172.21.15.125:0/1226279336 >> 172.21.15.125:0/1226279336 conn(0x7fc4d00c7910 msgr2=0x7fc4d0131ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:50.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.193+0000 7fc4d510f640 1 -- 172.21.15.125:0/1226279336 shutdown_connections 2023-10-30T13:21:50.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:50.193+0000 7fc4d510f640 1 -- 172.21.15.125:0/1226279336 wait complete. 2023-10-30T13:21:50.219 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is laggy/crashed, in failed state, or mds.f is removed from mdsmap 2023-10-30T13:21:52.220 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:52.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e63617640 1 Processor -- start 2023-10-30T13:21:52.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e63617640 1 -- start start 2023-10-30T13:21:52.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e63617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c065f10 0x7f2e5c071250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e63617640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c1560a0 con 0x7f2e5c065a80 2023-10-30T13:21:52.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e63617640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e5c156380 con 0x7f2e5c14fd50 2023-10-30T13:21:52.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e63617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c156660 con 0x7f2e5c065f10 2023-10-30T13:21:52.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e60bb2640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2e5c065a80 0x7f2e5c0cb550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46624/0 (socket says 172.21.15.125:46624) 2023-10-30T13:21:52.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e60bb2640 1 -- 172.21.15.125:0/916465194 learned_addr learned my addr 172.21.15.125:0/916465194 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:21:52.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e61bb4640 1 --2- 172.21.15.125:0/916465194 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c065f10 0x7f2e5c071250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4017247310 0 0) 0x7f2e5c1560a0 con 0x7f2e5c065a80 2023-10-30T13:21:52.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e48003530 con 0x7f2e5c065a80 2023-10-30T13:21:52.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900014330 0 0) 0x7f2e5c156380 con 0x7f2e5c14fd50 2023-10-30T13:21:52.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e5c1560a0 con 0x7f2e5c14fd50 2023-10-30T13:21:52.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 548639747 0 0) 0x7f2e48003530 con 0x7f2e5c065a80 2023-10-30T13:21:52.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c065f10 msgr2=0x7f2e5c071250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:52.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 --2- 172.21.15.125:0/916465194 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c065f10 0x7f2e5c071250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 >> v1:172.21.15.136:6789/0 conn(0x7f2e5c14fd50 legacy=0x7f2e5c0c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:52.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5c156940 con 0x7f2e5c065a80 2023-10-30T13:21:52.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f2e4c0016a0 con 0x7f2e5c065a80 2023-10-30T13:21:52.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e61bb4640 1 --2- 172.21.15.125:0/916465194 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c065f10 0x7f2e5c071250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:21:52.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 >> v1:172.21.15.125:6789/0 conn(0x7f2e5c065a80 legacy=0x7f2e5c0cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:52.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 --2- 172.21.15.125:0/916465194 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c065f10 0x7f2e48003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 --2- 172.21.15.125:0/916465194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e48004080 0x7f2e480047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 --2- 172.21.15.125:0/916465194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e48004d10 0x7f2e48005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c156660 con 0x7f2e48004080 2023-10-30T13:21:52.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4800dc50 con 0x7f2e48004d10 2023-10-30T13:21:52.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.361+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e4800de10 con 0x7f2e5c065f10 2023-10-30T13:21:52.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.365+0000 7f2e60bb2640 1 --2- 172.21.15.125:0/916465194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e48004080 0x7f2e480047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.365+0000 7f2e613b3640 1 --2- 172.21.15.125:0/916465194 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c065f10 0x7f2e48003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.365+0000 7f2e61bb4640 1 --2- 172.21.15.125:0/916465194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e48004d10 0x7f2e48005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.365+0000 7f2e60bb2640 1 -- 172.21.15.125:0/916465194 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c065f10 msgr2=0x7f2e48003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:52.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.365+0000 7f2e60bb2640 1 --2- 172.21.15.125:0/916465194 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c065f10 0x7f2e48003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.365+0000 7f2e60bb2640 1 -- 172.21.15.125:0/916465194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e48004d10 msgr2=0x7f2e48005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:52.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.365+0000 7f2e60bb2640 1 --2- 172.21.15.125:0/916465194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e48004d10 0x7f2e48005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.365+0000 7f2e60bb2640 1 -- 172.21.15.125:0/916465194 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e54009a70 con 0x7f2e48004080 2023-10-30T13:21:52.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.365+0000 7f2e61bb4640 1 --2- 172.21.15.125:0/916465194 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e48004d10 0x7f2e48005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:52.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.365+0000 7f2e60bb2640 1 --2- 172.21.15.125:0/916465194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e48004080 0x7f2e480047d0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c06abe0 tx=0x7f2e4c01cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e379eb5324d2ecd server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:52.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.365+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c0058e0 con 0x7f2e48004080 2023-10-30T13:21:52.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.365+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4c005a40 con 0x7f2e48004080 2023-10-30T13:21:52.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.365+0000 7f2e53fff640 1 -- 172.21.15.125:0/916465194 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c005df0 con 0x7f2e48004080 2023-10-30T13:21:52.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e63617640 1 -- 172.21.15.125:0/916465194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e48004080 msgr2=0x7f2e480047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:52.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e63617640 1 --2- 172.21.15.125:0/916465194 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e48004080 0x7f2e480047d0 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c06abe0 tx=0x7f2e4c01cb80 comp rx=0 tx=0).stop 2023-10-30T13:21:52.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e61bb4640 1 -- 172.21.15.125:0/916465194 reap_dead start 2023-10-30T13:21:52.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e63617640 1 -- 172.21.15.125:0/916465194 shutdown_connections 2023-10-30T13:21:52.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e63617640 1 -- 172.21.15.125:0/916465194 >> 172.21.15.125:0/916465194 conn(0x7f2e5c06c6e0 msgr2=0x7f2e5c069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:52.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e63617640 1 -- 172.21.15.125:0/916465194 shutdown_connections 2023-10-30T13:21:52.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e63617640 1 -- 172.21.15.125:0/916465194 wait complete. 2023-10-30T13:21:52.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e63617640 1 Processor -- start 2023-10-30T13:21:52.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e63617640 1 -- start start 2023-10-30T13:21:52.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e63617640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e48004080 0x7f2e5c0c9ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e63617640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c0c3f90 0x7f2e5c154320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e63617640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c0ca4b0 0x7f2e5c1f0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e63617640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c150f20 con 0x7f2e5c0ca4b0 2023-10-30T13:21:52.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e63617640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c1ef980 con 0x7f2e48004080 2023-10-30T13:21:52.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e63617640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c0caa80 con 0x7f2e5c0c3f90 2023-10-30T13:21:52.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e61bb4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c0ca4b0 0x7f2e5c1f0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e61bb4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c0ca4b0 0x7f2e5c1f0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50858/0 (socket says 172.21.15.125:50858) 2023-10-30T13:21:52.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e61bb4640 1 -- 172.21.15.125:0/2714667042 learned_addr learned my addr 172.21.15.125:0/2714667042 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:52.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e613b3640 1 --2- 172.21.15.125:0/2714667042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e48004080 0x7f2e5c0c9ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e60bb2640 1 --2- 172.21.15.125:0/2714667042 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c0c3f90 0x7f2e5c154320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e61bb4640 1 -- 172.21.15.125:0/2714667042 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c0c3f90 msgr2=0x7f2e5c154320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:52.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e61bb4640 1 --2- 172.21.15.125:0/2714667042 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c0c3f90 0x7f2e5c154320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e61bb4640 1 -- 172.21.15.125:0/2714667042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e48004080 msgr2=0x7f2e5c0c9ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:52.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e61bb4640 1 --2- 172.21.15.125:0/2714667042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e48004080 0x7f2e5c0c9ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e61bb4640 1 -- 172.21.15.125:0/2714667042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e5c0cad60 con 0x7f2e5c0ca4b0 2023-10-30T13:21:52.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.369+0000 7f2e61bb4640 1 --2- 172.21.15.125:0/2714667042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c0ca4b0 0x7f2e5c1f0ec0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c0c90a0 tx=0x7f2e540115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:52.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.373+0000 7f2e51ffb640 1 -- 172.21.15.125:0/2714667042 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5400e070 con 0x7f2e5c0ca4b0 2023-10-30T13:21:52.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.373+0000 7f2e51ffb640 1 -- 172.21.15.125:0/2714667042 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e54011ed0 con 0x7f2e5c0ca4b0 2023-10-30T13:21:52.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.373+0000 7f2e63617640 1 -- 172.21.15.125:0/2714667042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e5c0cb040 con 0x7f2e5c0ca4b0 2023-10-30T13:21:52.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.373+0000 7f2e51ffb640 1 -- 172.21.15.125:0/2714667042 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e540039a0 con 0x7f2e5c0ca4b0 2023-10-30T13:21:52.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.373+0000 7f2e63617640 1 -- 172.21.15.125:0/2714667042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e5c1efbd0 con 0x7f2e5c0ca4b0 2023-10-30T13:21:52.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.373+0000 7f2e63617640 1 -- 172.21.15.125:0/2714667042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e5c04e170 con 0x7f2e5c0ca4b0 2023-10-30T13:21:52.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.373+0000 7f2e51ffb640 1 -- 172.21.15.125:0/2714667042 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2e54003b00 con 0x7f2e5c0ca4b0 2023-10-30T13:21:52.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.373+0000 7f2e51ffb640 1 --2- 172.21.15.125:0/2714667042 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e400664a0 0x7f2e40068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.373+0000 7f2e51ffb640 1 -- 172.21.15.125:0/2714667042 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f2e54084fa0 con 0x7f2e5c0ca4b0 2023-10-30T13:21:52.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.373+0000 7f2e613b3640 1 --2- 172.21.15.125:0/2714667042 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e400664a0 0x7f2e40068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.377+0000 7f2e613b3640 1 --2- 172.21.15.125:0/2714667042 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e400664a0 0x7f2e40068960 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f2e58005630 tx=0x7f2e58001490 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:52.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.377+0000 7f2e51ffb640 1 -- 172.21.15.125:0/2714667042 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2e54051960 con 0x7f2e5c0ca4b0 2023-10-30T13:21:52.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.557+0000 7f2e63617640 1 -- 172.21.15.125:0/2714667042 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e5c155d60 con 0x7f2e5c0ca4b0 2023-10-30T13:21:52.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.557+0000 7f2e51ffb640 1 -- 172.21.15.125:0/2714667042 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+5754 (secure 0 0 0) 0x7f2e54054f80 con 0x7f2e5c0ca4b0 2023-10-30T13:21:52.563 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:52.564 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":248,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:50.903243+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:52.564 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 248 2023-10-30T13:21:52.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.561+0000 7f2e63617640 1 -- 172.21.15.125:0/2714667042 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e400664a0 msgr2=0x7f2e40068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:52.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.561+0000 7f2e63617640 1 --2- 172.21.15.125:0/2714667042 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e400664a0 0x7f2e40068960 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f2e58005630 tx=0x7f2e58001490 comp rx=0 tx=0).stop 2023-10-30T13:21:52.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.561+0000 7f2e63617640 1 -- 172.21.15.125:0/2714667042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c0ca4b0 msgr2=0x7f2e5c1f0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:52.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.561+0000 7f2e63617640 1 --2- 172.21.15.125:0/2714667042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c0ca4b0 0x7f2e5c1f0ec0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f2e5c0c90a0 tx=0x7f2e540115a0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.561+0000 7f2e63617640 1 -- 172.21.15.125:0/2714667042 shutdown_connections 2023-10-30T13:21:52.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.561+0000 7f2e63617640 1 --2- 172.21.15.125:0/2714667042 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2e400664a0 0x7f2e40068960 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.561+0000 7f2e63617640 1 --2- 172.21.15.125:0/2714667042 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e5c0ca4b0 0x7f2e5c1f0ec0 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.561+0000 7f2e63617640 1 --2- 172.21.15.125:0/2714667042 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e5c0c3f90 0x7f2e5c154320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.561+0000 7f2e63617640 1 --2- 172.21.15.125:0/2714667042 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e48004080 0x7f2e5c0c9ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.561+0000 7f2e63617640 1 -- 172.21.15.125:0/2714667042 >> 172.21.15.125:0/2714667042 conn(0x7f2e5c06c6e0 msgr2=0x7f2e5c0c73a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:52.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.561+0000 7f2e63617640 1 -- 172.21.15.125:0/2714667042 shutdown_connections 2023-10-30T13:21:52.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.561+0000 7f2e63617640 1 -- 172.21.15.125:0/2714667042 wait complete. 2023-10-30T13:21:52.584 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f down, removed from mdsmap 2023-10-30T13:21:52.584 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:21:52.585 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:52.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdedb49e640 1 Processor -- start 2023-10-30T13:21:52.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdedb49e640 1 -- start start 2023-10-30T13:21:52.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdedb49e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fded40c05d0 0x7fded413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdedb49e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fded40c6f60 0x7fded413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdedb49e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded40c0a50 con 0x7fded40c6b80 2023-10-30T13:21:52.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdedb49e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded406f6e0 con 0x7fded40c05d0 2023-10-30T13:21:52.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdedb49e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded406f9c0 con 0x7fded40c6f60 2023-10-30T13:21:52.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fded923a640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fded40c6b80 0x7fded40bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46632/0 (socket says 172.21.15.125:46632) 2023-10-30T13:21:52.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fded923a640 1 -- 172.21.15.125:0/3948791820 learned_addr learned my addr 172.21.15.125:0/3948791820 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:21:52.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fded9a3b640 1 --2- 172.21.15.125:0/3948791820 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fded40c6f60 0x7fded413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fded8a39640 1 --2- 172.21.15.125:0/3948791820 >> v2:172.21.15.136:3300/0 conn(0x7fded40c05d0 0x7fded413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 -- 172.21.15.125:0/3948791820 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340352869 0 0) 0x7fded40c0a50 con 0x7fded40c6b80 2023-10-30T13:21:52.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 -- 172.21.15.125:0/3948791820 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdec0003530 con 0x7fded40c6b80 2023-10-30T13:21:52.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 -- 172.21.15.125:0/3948791820 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3538499082 0 0) 0x7fdec0003530 con 0x7fded40c6b80 2023-10-30T13:21:52.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 -- 172.21.15.125:0/3948791820 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fded40c6f60 msgr2=0x7fded413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:52.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 --2- 172.21.15.125:0/3948791820 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fded40c6f60 0x7fded413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 -- 172.21.15.125:0/3948791820 >> v2:172.21.15.136:3300/0 conn(0x7fded40c05d0 msgr2=0x7fded413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:52.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 --2- 172.21.15.125:0/3948791820 >> v2:172.21.15.136:3300/0 conn(0x7fded40c05d0 0x7fded413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 -- 172.21.15.125:0/3948791820 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded406fca0 con 0x7fded40c6b80 2023-10-30T13:21:52.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 -- 172.21.15.125:0/3948791820 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fded0001680 con 0x7fded40c6b80 2023-10-30T13:21:52.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 -- 172.21.15.125:0/3948791820 >> v1:172.21.15.125:6789/0 conn(0x7fded40c6b80 legacy=0x7fded40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:52.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 --2- 172.21.15.125:0/3948791820 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fded40c6f60 0x7fdec0003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 --2- 172.21.15.125:0/3948791820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec0003f00 0x7fdec00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 --2- 172.21.15.125:0/3948791820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec0004d10 0x7fdec0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 -- 172.21.15.125:0/3948791820 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded406f6e0 con 0x7fdec0003f00 2023-10-30T13:21:52.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 -- 172.21.15.125:0/3948791820 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded406f9c0 con 0x7fdec0004d10 2023-10-30T13:21:52.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.717+0000 7fdecbfff640 1 -- 172.21.15.125:0/3948791820 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec000b990 con 0x7fded40c6f60 2023-10-30T13:21:52.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.721+0000 7fded8a39640 1 --2- 172.21.15.125:0/3948791820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec0003f00 0x7fdec00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.721+0000 7fded923a640 1 --2- 172.21.15.125:0/3948791820 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fded40c6f60 0x7fdec0003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.721+0000 7fded9a3b640 1 --2- 172.21.15.125:0/3948791820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec0004d10 0x7fdec0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.721+0000 7fded8a39640 1 -- 172.21.15.125:0/3948791820 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fded40c6f60 msgr2=0x7fdec0003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:52.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.721+0000 7fded8a39640 1 --2- 172.21.15.125:0/3948791820 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fded40c6f60 0x7fdec0003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.721+0000 7fded8a39640 1 -- 172.21.15.125:0/3948791820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec0004d10 msgr2=0x7fdec0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:52.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.721+0000 7fded8a39640 1 --2- 172.21.15.125:0/3948791820 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdec0004d10 0x7fdec0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.721+0000 7fded8a39640 1 -- 172.21.15.125:0/3948791820 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec0003530 con 0x7fdec0003f00 2023-10-30T13:21:52.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.721+0000 7fded8a39640 1 --2- 172.21.15.125:0/3948791820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec0003f00 0x7fdec00047d0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fdec4009670 tx=0x7fdec401ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b8b716d115a50e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:52.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.721+0000 7fdecbfff640 1 -- 172.21.15.125:0/3948791820 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec40a7770 con 0x7fdec0003f00 2023-10-30T13:21:52.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.721+0000 7fdecbfff640 1 -- 172.21.15.125:0/3948791820 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec40a78d0 con 0x7fdec0003f00 2023-10-30T13:21:52.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.721+0000 7fdecbfff640 1 -- 172.21.15.125:0/3948791820 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec40a7c80 con 0x7fdec0003f00 2023-10-30T13:21:52.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.721+0000 7fdedb49e640 1 -- 172.21.15.125:0/3948791820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec0003f00 msgr2=0x7fdec00047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:52.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.721+0000 7fdedb49e640 1 --2- 172.21.15.125:0/3948791820 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec0003f00 0x7fdec00047d0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fdec4009670 tx=0x7fdec401ec20 comp rx=0 tx=0).stop 2023-10-30T13:21:52.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fded9a3b640 1 -- 172.21.15.125:0/3948791820 reap_dead start 2023-10-30T13:21:52.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fdedb49e640 1 -- 172.21.15.125:0/3948791820 shutdown_connections 2023-10-30T13:21:52.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fdedb49e640 1 -- 172.21.15.125:0/3948791820 >> 172.21.15.125:0/3948791820 conn(0x7fded40c7910 msgr2=0x7fded40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:52.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fdedb49e640 1 -- 172.21.15.125:0/3948791820 shutdown_connections 2023-10-30T13:21:52.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fdedb49e640 1 -- 172.21.15.125:0/3948791820 wait complete. 2023-10-30T13:21:52.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fdedb49e640 1 Processor -- start 2023-10-30T13:21:52.736 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fdedb49e640 1 -- start start 2023-10-30T13:21:52.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fdedb49e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec0003f00 0x7fded40c0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fdedb49e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fded40ca5a0 0x7fded4152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fdedb49e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fded40c0e50 0x7fded41ebdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fdedb49e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded40c9aa0 con 0x7fdec0003f00 2023-10-30T13:21:52.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fdedb49e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded40c8c10 con 0x7fded40c0e50 2023-10-30T13:21:52.737 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fdedb49e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded40c9ec0 con 0x7fded40ca5a0 2023-10-30T13:21:52.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fded923a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec0003f00 0x7fded40c0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fded923a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec0003f00 0x7fded40c0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50862/0 (socket says 172.21.15.125:50862) 2023-10-30T13:21:52.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fded923a640 1 -- 172.21.15.125:0/2288494365 learned_addr learned my addr 172.21.15.125:0/2288494365 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:52.738 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fded9a3b640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fded40c0e50 0x7fded41ebdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fded8a39640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fded40ca5a0 0x7fded4152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fded923a640 1 -- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fded40ca5a0 msgr2=0x7fded4152b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:52.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fded923a640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fded40ca5a0 0x7fded4152b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fded923a640 1 -- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fded40c0e50 msgr2=0x7fded41ebdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:52.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fded923a640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fded40c0e50 0x7fded41ebdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fded923a640 1 -- 172.21.15.125:0/2288494365 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded406c660 con 0x7fdec0003f00 2023-10-30T13:21:52.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.725+0000 7fded9a3b640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fded40c0e50 0x7fded41ebdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:21:52.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.729+0000 7fded923a640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec0003f00 0x7fded40c0070 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fded0011910 tx=0x7fded0011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:52.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.729+0000 7fdec9ffb640 1 -- 172.21.15.125:0/2288494365 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded000fbc0 con 0x7fdec0003f00 2023-10-30T13:21:52.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.729+0000 7fdec9ffb640 1 -- 172.21.15.125:0/2288494365 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded000fd20 con 0x7fdec0003f00 2023-10-30T13:21:52.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.729+0000 7fdec9ffb640 1 -- 172.21.15.125:0/2288494365 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded0015650 con 0x7fdec0003f00 2023-10-30T13:21:52.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.729+0000 7fdedb49e640 1 -- 172.21.15.125:0/2288494365 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded413d6f0 con 0x7fdec0003f00 2023-10-30T13:21:52.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.729+0000 7fdedb49e640 1 -- 172.21.15.125:0/2288494365 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded40c9100 con 0x7fdec0003f00 2023-10-30T13:21:52.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.729+0000 7fdedb49e640 1 -- 172.21.15.125:0/2288494365 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded40c7710 con 0x7fdec0003f00 2023-10-30T13:21:52.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.729+0000 7fdec9ffb640 1 -- 172.21.15.125:0/2288494365 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fded00157b0 con 0x7fdec0003f00 2023-10-30T13:21:52.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.729+0000 7fdec9ffb640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeb80664a0 0x7fdeb8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:52.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.729+0000 7fdec9ffb640 1 -- 172.21.15.125:0/2288494365 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7fded0087b50 con 0x7fdec0003f00 2023-10-30T13:21:52.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.729+0000 7fded8a39640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeb80664a0 0x7fdeb8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:52.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.733+0000 7fded8a39640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeb80664a0 0x7fdeb8068960 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fdec40b1000 tx=0x7fdec40046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:52.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.733+0000 7fdec9ffb640 1 -- 172.21.15.125:0/2288494365 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fded00544c0 con 0x7fdec0003f00 2023-10-30T13:21:52.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.917+0000 7fdedb49e640 1 -- 172.21.15.125:0/2288494365 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fded406c660 con 0x7fdec0003f00 2023-10-30T13:21:52.921 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.917+0000 7fdec9ffb640 1 -- 172.21.15.125:0/2288494365 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+5754 (secure 0 0 0) 0x7fded406c660 con 0x7fdec0003f00 2023-10-30T13:21:52.921 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:52.921 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":248,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:50.903243+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:52.922 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 248 2023-10-30T13:21:52.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.917+0000 7fdedb49e640 1 -- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeb80664a0 msgr2=0x7fdeb8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:52.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.917+0000 7fdedb49e640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeb80664a0 0x7fdeb8068960 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fdec40b1000 tx=0x7fdec40046d0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.923 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.917+0000 7fdedb49e640 1 -- 172.21.15.125:0/2288494365 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec0003f00 msgr2=0x7fded40c0070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:52.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.917+0000 7fdedb49e640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec0003f00 0x7fded40c0070 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fded0011910 tx=0x7fded0011dd0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.917+0000 7fdedb49e640 1 -- 172.21.15.125:0/2288494365 shutdown_connections 2023-10-30T13:21:52.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.917+0000 7fdedb49e640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdeb80664a0 0x7fdeb8068960 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.917+0000 7fdedb49e640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fded40c0e50 0x7fded41ebdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.917+0000 7fdedb49e640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fded40ca5a0 0x7fded4152b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.917+0000 7fdedb49e640 1 --2- 172.21.15.125:0/2288494365 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdec0003f00 0x7fded40c0070 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:52.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.917+0000 7fdedb49e640 1 -- 172.21.15.125:0/2288494365 >> 172.21.15.125:0/2288494365 conn(0x7fded40c7910 msgr2=0x7fded4137360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:52.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.917+0000 7fdedb49e640 1 -- 172.21.15.125:0/2288494365 shutdown_connections 2023-10-30T13:21:52.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:52.917+0000 7fdedb49e640 1 -- 172.21.15.125:0/2288494365 wait complete. 2023-10-30T13:21:52.943 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 248, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 248, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_8033": { "addr": "172.21.15.136:6835/3480968625", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3480968625, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3480968625, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8033, "incarnation": 248, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 36 }, "gid_9600": { "addr": "172.21.15.125:6836/2077416857", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2077416857, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2077416857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9600, "incarnation": 241, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 42 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:21:50.903243+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8033, "mds_1": 9150, "mds_2": 9600 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:21:54.944 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:55.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.085+0000 7fc85f59e640 1 Processor -- start 2023-10-30T13:21:55.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.085+0000 7fc85f59e640 1 -- start start 2023-10-30T13:21:55.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85f59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc86006fef0 0x7fc860161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:55.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85f59e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc860161f60 con 0x7fc86006ab20 2023-10-30T13:21:55.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85f59e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8601616c0 con 0x7fc860157c80 2023-10-30T13:21:55.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85f59e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86015ef30 con 0x7fc86006fef0 2023-10-30T13:21:55.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85ed9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc86006ab20 0x7fc860070c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46648/0 (socket says 172.21.15.125:46648) 2023-10-30T13:21:55.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85ed9d640 1 -- 172.21.15.125:0/2048170286 learned_addr learned my addr 172.21.15.125:0/2048170286 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:21:55.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc857fff640 1 --2- 172.21.15.125:0/2048170286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc86006fef0 0x7fc860161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:55.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 -- 172.21.15.125:0/2048170286 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135247509 0 0) 0x7fc860161f60 con 0x7fc86006ab20 2023-10-30T13:21:55.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 -- 172.21.15.125:0/2048170286 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc84c003530 con 0x7fc86006ab20 2023-10-30T13:21:55.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 -- 172.21.15.125:0/2048170286 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3837145127 0 0) 0x7fc84c003530 con 0x7fc86006ab20 2023-10-30T13:21:55.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 -- 172.21.15.125:0/2048170286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc86006fef0 msgr2=0x7fc860161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:55.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 --2- 172.21.15.125:0/2048170286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc86006fef0 0x7fc860161030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:55.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 -- 172.21.15.125:0/2048170286 >> v1:172.21.15.136:6789/0 conn(0x7fc860157c80 legacy=0x7fc86006f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:55.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 -- 172.21.15.125:0/2048170286 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8601619a0 con 0x7fc86006ab20 2023-10-30T13:21:55.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 -- 172.21.15.125:0/2048170286 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fc858001a00 con 0x7fc86006ab20 2023-10-30T13:21:55.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 -- 172.21.15.125:0/2048170286 >> v1:172.21.15.125:6789/0 conn(0x7fc86006ab20 legacy=0x7fc860070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:55.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 --2- 172.21.15.125:0/2048170286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc84c0039b0 0x7fc84c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:55.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 --2- 172.21.15.125:0/2048170286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc86006fef0 0x7fc84c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:55.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 --2- 172.21.15.125:0/2048170286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc84c004d10 0x7fc84c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:55.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 -- 172.21.15.125:0/2048170286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c00bad0 con 0x7fc84c0039b0 2023-10-30T13:21:55.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 -- 172.21.15.125:0/2048170286 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c00bc10 con 0x7fc84c004d10 2023-10-30T13:21:55.099 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85dd9b640 1 -- 172.21.15.125:0/2048170286 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c00bdd0 con 0x7fc86006fef0 2023-10-30T13:21:55.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85e59c640 1 --2- 172.21.15.125:0/2048170286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc84c0039b0 0x7fc84c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:55.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc857fff640 1 --2- 172.21.15.125:0/2048170286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc86006fef0 0x7fc84c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:55.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85ed9d640 1 --2- 172.21.15.125:0/2048170286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc84c004d10 0x7fc84c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:55.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85e59c640 1 -- 172.21.15.125:0/2048170286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc86006fef0 msgr2=0x7fc84c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:55.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85e59c640 1 --2- 172.21.15.125:0/2048170286 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc86006fef0 0x7fc84c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:55.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85e59c640 1 -- 172.21.15.125:0/2048170286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc84c004d10 msgr2=0x7fc84c005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:55.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85e59c640 1 --2- 172.21.15.125:0/2048170286 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc84c004d10 0x7fc84c005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:55.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.089+0000 7fc85e59c640 1 -- 172.21.15.125:0/2048170286 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c003530 con 0x7fc84c0039b0 2023-10-30T13:21:55.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.093+0000 7fc85e59c640 1 --2- 172.21.15.125:0/2048170286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc84c0039b0 0x7fc84c003dd0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7fc850001780 tx=0x7fc85001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e05c89e2356805c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:55.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.093+0000 7fc85dd9b640 1 -- 172.21.15.125:0/2048170286 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8500052d0 con 0x7fc84c0039b0 2023-10-30T13:21:55.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.093+0000 7fc85dd9b640 1 -- 172.21.15.125:0/2048170286 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8500a7050 con 0x7fc84c0039b0 2023-10-30T13:21:55.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.093+0000 7fc85dd9b640 1 -- 172.21.15.125:0/2048170286 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc850003650 con 0x7fc84c0039b0 2023-10-30T13:21:55.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.093+0000 7fc85f59e640 1 -- 172.21.15.125:0/2048170286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc84c0039b0 msgr2=0x7fc84c003dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:55.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.093+0000 7fc85f59e640 1 --2- 172.21.15.125:0/2048170286 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc84c0039b0 0x7fc84c003dd0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7fc850001780 tx=0x7fc85001c3a0 comp rx=0 tx=0).stop 2023-10-30T13:21:55.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.093+0000 7fc85ed9d640 1 -- 172.21.15.125:0/2048170286 reap_dead start 2023-10-30T13:21:55.105 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.093+0000 7fc85f59e640 1 -- 172.21.15.125:0/2048170286 shutdown_connections 2023-10-30T13:21:55.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.093+0000 7fc85f59e640 1 -- 172.21.15.125:0/2048170286 >> 172.21.15.125:0/2048170286 conn(0x7fc8600c75d0 msgr2=0x7fc8600c79e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:55.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.093+0000 7fc85f59e640 1 -- 172.21.15.125:0/2048170286 shutdown_connections 2023-10-30T13:21:55.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.093+0000 7fc85f59e640 1 -- 172.21.15.125:0/2048170286 wait complete. 2023-10-30T13:21:55.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85f59e640 1 Processor -- start 2023-10-30T13:21:55.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85f59e640 1 -- start start 2023-10-30T13:21:55.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85f59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc84c0039b0 0x7fc860150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:55.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85f59e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8600c2690 0x7fc860154b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:55.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85f59e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc860069e90 0x7fc86006a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:55.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85f59e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc86015c700 con 0x7fc84c0039b0 2023-10-30T13:21:55.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85f59e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc860150690 con 0x7fc860069e90 2023-10-30T13:21:55.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85f59e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86015d860 con 0x7fc8600c2690 2023-10-30T13:21:55.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc84c0039b0 0x7fc860150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:55.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85e59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc84c0039b0 0x7fc860150150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50876/0 (socket says 172.21.15.125:50876) 2023-10-30T13:21:55.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85e59c640 1 -- 172.21.15.125:0/2855490594 learned_addr learned my addr 172.21.15.125:0/2855490594 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:55.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc857fff640 1 --2- 172.21.15.125:0/2855490594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8600c2690 0x7fc860154b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:55.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85ed9d640 1 --2- 172.21.15.125:0/2855490594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc860069e90 0x7fc86006a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:55.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85e59c640 1 -- 172.21.15.125:0/2855490594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8600c2690 msgr2=0x7fc860154b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:55.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85e59c640 1 --2- 172.21.15.125:0/2855490594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8600c2690 0x7fc860154b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:55.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85e59c640 1 -- 172.21.15.125:0/2855490594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc860069e90 msgr2=0x7fc86006a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:55.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85e59c640 1 --2- 172.21.15.125:0/2855490594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc860069e90 0x7fc86006a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:55.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85e59c640 1 -- 172.21.15.125:0/2855490594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86015d5a0 con 0x7fc84c0039b0 2023-10-30T13:21:55.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85e59c640 1 --2- 172.21.15.125:0/2855490594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc84c0039b0 0x7fc860150150 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fc850005040 tx=0x7fc84c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:55.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc856ffd640 1 -- 172.21.15.125:0/2855490594 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc850006e70 con 0x7fc84c0039b0 2023-10-30T13:21:55.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc856ffd640 1 -- 172.21.15.125:0/2855490594 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc850007a90 con 0x7fc84c0039b0 2023-10-30T13:21:55.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc856ffd640 1 -- 172.21.15.125:0/2855490594 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc850007260 con 0x7fc84c0039b0 2023-10-30T13:21:55.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85f59e640 1 -- 172.21.15.125:0/2855490594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86015e3c0 con 0x7fc84c0039b0 2023-10-30T13:21:55.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85f59e640 1 -- 172.21.15.125:0/2855490594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc860155300 con 0x7fc84c0039b0 2023-10-30T13:21:55.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.097+0000 7fc85f59e640 1 -- 172.21.15.125:0/2855490594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86004e170 con 0x7fc84c0039b0 2023-10-30T13:21:55.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.101+0000 7fc856ffd640 1 -- 172.21.15.125:0/2855490594 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc8500073c0 con 0x7fc84c0039b0 2023-10-30T13:21:55.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.101+0000 7fc856ffd640 1 --2- 172.21.15.125:0/2855490594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc8400664a0 0x7fc840068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:55.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.101+0000 7fc856ffd640 1 -- 172.21.15.125:0/2855490594 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7fc85011f0a0 con 0x7fc84c0039b0 2023-10-30T13:21:55.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.101+0000 7fc857fff640 1 --2- 172.21.15.125:0/2855490594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc8400664a0 0x7fc840068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:55.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.101+0000 7fc857fff640 1 --2- 172.21.15.125:0/2855490594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc8400664a0 0x7fc840068960 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fc8601f0890 tx=0x7fc848008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:55.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.105+0000 7fc856ffd640 1 -- 172.21.15.125:0/2855490594 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc8500ebd00 con 0x7fc84c0039b0 2023-10-30T13:21:55.290 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.285+0000 7fc85f59e640 1 -- 172.21.15.125:0/2855490594 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc86015d5a0 con 0x7fc84c0039b0 2023-10-30T13:21:55.291 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.285+0000 7fc856ffd640 1 -- 172.21.15.125:0/2855490594 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+5754 (secure 0 0 0) 0x7fc86015d5a0 con 0x7fc84c0039b0 2023-10-30T13:21:55.291 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:55.291 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":248,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:50.903243+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:55.292 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 248 2023-10-30T13:21:55.293 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.289+0000 7fc85f59e640 1 -- 172.21.15.125:0/2855490594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc8400664a0 msgr2=0x7fc840068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:55.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.289+0000 7fc85f59e640 1 --2- 172.21.15.125:0/2855490594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc8400664a0 0x7fc840068960 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fc8601f0890 tx=0x7fc848008040 comp rx=0 tx=0).stop 2023-10-30T13:21:55.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.289+0000 7fc85f59e640 1 -- 172.21.15.125:0/2855490594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc84c0039b0 msgr2=0x7fc860150150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:55.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.289+0000 7fc85f59e640 1 --2- 172.21.15.125:0/2855490594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc84c0039b0 0x7fc860150150 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fc850005040 tx=0x7fc84c005800 comp rx=0 tx=0).stop 2023-10-30T13:21:55.294 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.289+0000 7fc85f59e640 1 -- 172.21.15.125:0/2855490594 shutdown_connections 2023-10-30T13:21:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.289+0000 7fc85f59e640 1 --2- 172.21.15.125:0/2855490594 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc8400664a0 0x7fc840068960 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.289+0000 7fc85f59e640 1 --2- 172.21.15.125:0/2855490594 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc860069e90 0x7fc86006a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.289+0000 7fc85f59e640 1 --2- 172.21.15.125:0/2855490594 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc8600c2690 0x7fc860154b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:55.295 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.289+0000 7fc85f59e640 1 --2- 172.21.15.125:0/2855490594 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc84c0039b0 0x7fc860150150 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:55.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.289+0000 7fc85f59e640 1 -- 172.21.15.125:0/2855490594 >> 172.21.15.125:0/2855490594 conn(0x7fc8600c75d0 msgr2=0x7fc860157260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:55.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.289+0000 7fc85f59e640 1 -- 172.21.15.125:0/2855490594 shutdown_connections 2023-10-30T13:21:55.296 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:55.289+0000 7fc85f59e640 1 -- 172.21.15.125:0/2855490594 wait complete. 2023-10-30T13:21:55.312 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:21:57.314 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:57.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.449+0000 7fd99b291640 1 Processor -- start 2023-10-30T13:21:57.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.449+0000 7fd99b291640 1 -- start start 2023-10-30T13:21:57.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.453+0000 7fd99b291640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd994153bf0 0x7fd994071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:57.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.453+0000 7fd99b291640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd9940c5f60 0x7fd9940cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:57.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.453+0000 7fd99b291640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9940700b0 con 0x7fd9940c6fe0 2023-10-30T13:21:57.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.453+0000 7fd99b291640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9940c6810 con 0x7fd9940c5f60 2023-10-30T13:21:57.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.453+0000 7fd99b291640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd994159900 con 0x7fd994153bf0 2023-10-30T13:21:57.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.453+0000 7fd99982e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd9940c5f60 0x7fd9940cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:57.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.453+0000 7fd99982e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd9940c5f60 0x7fd9940cb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50886/0 (socket says 172.21.15.125:50886) 2023-10-30T13:21:57.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.453+0000 7fd99982e640 1 -- 172.21.15.125:0/2688397788 learned_addr learned my addr 172.21.15.125:0/2688397788 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:57.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.453+0000 7fd99902d640 1 --2- 172.21.15.125:0/2688397788 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd994153bf0 0x7fd994071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:57.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.453+0000 7fd99982e640 1 -- 172.21.15.125:0/2688397788 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd994153bf0 msgr2=0x7fd994071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:21:57.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.453+0000 7fd99982e640 1 --2- 172.21.15.125:0/2688397788 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd994153bf0 0x7fd994071260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:57.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.453+0000 7fd99982e640 1 -- 172.21.15.125:0/2688397788 >> v1:172.21.15.136:6789/0 conn(0x7fd9940c6fe0 legacy=0x7fd9940c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:57.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.453+0000 7fd99982e640 1 -- 172.21.15.125:0/2688397788 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9940c6530 con 0x7fd9940c5f60 2023-10-30T13:21:57.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.457+0000 7fd99982e640 1 --2- 172.21.15.125:0/2688397788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9940c5f60 0x7fd9940cb570 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009e50 tx=0x7fd98c01f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f291d7b6c0c037 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:57.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.457+0000 7fd98bfff640 1 -- 172.21.15.125:0/2688397788 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c0af400 con 0x7fd9940c5f60 2023-10-30T13:21:57.461 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.457+0000 7fd98bfff640 1 -- 172.21.15.125:0/2688397788 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd98c0ab050 con 0x7fd9940c5f60 2023-10-30T13:21:57.462 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.457+0000 7fd98bfff640 1 -- 172.21.15.125:0/2688397788 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c0af780 con 0x7fd9940c5f60 2023-10-30T13:21:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.457+0000 7fd99b291640 1 -- 172.21.15.125:0/2688397788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9940c5f60 msgr2=0x7fd9940cb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.457+0000 7fd99b291640 1 --2- 172.21.15.125:0/2688397788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9940c5f60 0x7fd9940cb570 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fd98c009e50 tx=0x7fd98c01f1f0 comp rx=0 tx=0).stop 2023-10-30T13:21:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.457+0000 7fd99b291640 1 -- 172.21.15.125:0/2688397788 shutdown_connections 2023-10-30T13:21:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.457+0000 7fd99b291640 1 --2- 172.21.15.125:0/2688397788 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd9940c5f60 0x7fd9940cb570 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:57.463 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.457+0000 7fd99b291640 1 --2- 172.21.15.125:0/2688397788 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd994153bf0 0x7fd994071260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:57.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.457+0000 7fd99b291640 1 -- 172.21.15.125:0/2688397788 >> 172.21.15.125:0/2688397788 conn(0x7fd99406c6e0 msgr2=0x7fd994069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:57.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.457+0000 7fd99b291640 1 -- 172.21.15.125:0/2688397788 shutdown_connections 2023-10-30T13:21:57.464 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.457+0000 7fd99b291640 1 -- 172.21.15.125:0/2688397788 wait complete. 2023-10-30T13:21:57.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.457+0000 7fd99b291640 1 Processor -- start 2023-10-30T13:21:57.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99b291640 1 -- start start 2023-10-30T13:21:57.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99b291640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9940c5f60 0x7fd994149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:57.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99b291640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9940c6fe0 0x7fd99414de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:57.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99b291640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd994153bf0 0x7fd994152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:57.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99b291640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9940c3a80 con 0x7fd994153bf0 2023-10-30T13:21:57.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99b291640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99414fa30 con 0x7fd9940c6fe0 2023-10-30T13:21:57.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99b291640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99414c3b0 con 0x7fd9940c5f60 2023-10-30T13:21:57.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99982e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd994153bf0 0x7fd994152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:57.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99902d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9940c5f60 0x7fd994149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:57.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99982e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd994153bf0 0x7fd994152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50900/0 (socket says 172.21.15.125:50900) 2023-10-30T13:21:57.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99982e640 1 -- 172.21.15.125:0/1027562534 learned_addr learned my addr 172.21.15.125:0/1027562534 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:57.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99882c640 1 --2- 172.21.15.125:0/1027562534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9940c6fe0 0x7fd99414de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:57.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99982e640 1 -- 172.21.15.125:0/1027562534 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9940c5f60 msgr2=0x7fd994149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:57.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99982e640 1 --2- 172.21.15.125:0/1027562534 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9940c5f60 0x7fd994149930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:57.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99982e640 1 -- 172.21.15.125:0/1027562534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9940c6fe0 msgr2=0x7fd99414de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:57.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99982e640 1 --2- 172.21.15.125:0/1027562534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9940c6fe0 0x7fd99414de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:57.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99982e640 1 -- 172.21.15.125:0/1027562534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd98c009b30 con 0x7fd994153bf0 2023-10-30T13:21:57.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99982e640 1 --2- 172.21.15.125:0/1027562534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd994153bf0 0x7fd994152a00 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fd98c01ce20 tx=0x7fd98c01bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:57.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd989ffb640 1 -- 172.21.15.125:0/1027562534 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c0afd30 con 0x7fd994153bf0 2023-10-30T13:21:57.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd989ffb640 1 -- 172.21.15.125:0/1027562534 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd98c0afe90 con 0x7fd994153bf0 2023-10-30T13:21:57.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99b291640 1 -- 172.21.15.125:0/1027562534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd994149e70 con 0x7fd994153bf0 2023-10-30T13:21:57.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd989ffb640 1 -- 172.21.15.125:0/1027562534 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c0ba660 con 0x7fd994153bf0 2023-10-30T13:21:57.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99b291640 1 -- 172.21.15.125:0/1027562534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd99414b7a0 con 0x7fd994153bf0 2023-10-30T13:21:57.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.461+0000 7fd99b291640 1 -- 172.21.15.125:0/1027562534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99404e170 con 0x7fd994153bf0 2023-10-30T13:21:57.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.465+0000 7fd989ffb640 1 -- 172.21.15.125:0/1027562534 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd98c0b6040 con 0x7fd994153bf0 2023-10-30T13:21:57.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.465+0000 7fd989ffb640 1 --2- 172.21.15.125:0/1027562534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9780664a0 0x7fd978068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:57.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.465+0000 7fd989ffb640 1 -- 172.21.15.125:0/1027562534 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7fd98c122a30 con 0x7fd994153bf0 2023-10-30T13:21:57.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.465+0000 7fd99902d640 1 --2- 172.21.15.125:0/1027562534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9780664a0 0x7fd978068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:57.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.465+0000 7fd99902d640 1 --2- 172.21.15.125:0/1027562534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9780664a0 0x7fd978068960 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fd990009ac0 tx=0x7fd990009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:57.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.469+0000 7fd989ffb640 1 -- 172.21.15.125:0/1027562534 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd98c0ef4b0 con 0x7fd994153bf0 2023-10-30T13:21:57.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.653+0000 7fd99b291640 1 -- 172.21.15.125:0/1027562534 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd99414c4f0 con 0x7fd994153bf0 2023-10-30T13:21:57.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.653+0000 7fd989ffb640 1 -- 172.21.15.125:0/1027562534 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+5754 (secure 0 0 0) 0x7fd98c009510 con 0x7fd994153bf0 2023-10-30T13:21:57.658 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:21:57.659 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":248,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:50.903243+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:21:57.659 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 248 2023-10-30T13:21:57.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.653+0000 7fd99b291640 1 -- 172.21.15.125:0/1027562534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9780664a0 msgr2=0x7fd978068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:57.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.653+0000 7fd99b291640 1 --2- 172.21.15.125:0/1027562534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9780664a0 0x7fd978068960 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fd990009ac0 tx=0x7fd990009190 comp rx=0 tx=0).stop 2023-10-30T13:21:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.653+0000 7fd99b291640 1 -- 172.21.15.125:0/1027562534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd994153bf0 msgr2=0x7fd994152a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.653+0000 7fd99b291640 1 --2- 172.21.15.125:0/1027562534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd994153bf0 0x7fd994152a00 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fd98c01ce20 tx=0x7fd98c01bf90 comp rx=0 tx=0).stop 2023-10-30T13:21:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.653+0000 7fd99b291640 1 -- 172.21.15.125:0/1027562534 shutdown_connections 2023-10-30T13:21:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.653+0000 7fd99b291640 1 --2- 172.21.15.125:0/1027562534 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd9780664a0 0x7fd978068960 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:57.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.653+0000 7fd99b291640 1 --2- 172.21.15.125:0/1027562534 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd994153bf0 0x7fd994152a00 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.653+0000 7fd99b291640 1 --2- 172.21.15.125:0/1027562534 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd9940c6fe0 0x7fd99414de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.653+0000 7fd99b291640 1 --2- 172.21.15.125:0/1027562534 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd9940c5f60 0x7fd994149930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.653+0000 7fd99b291640 1 -- 172.21.15.125:0/1027562534 >> 172.21.15.125:0/1027562534 conn(0x7fd99406c6e0 msgr2=0x7fd9940c77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:57.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.657+0000 7fd99b291640 1 -- 172.21.15.125:0/1027562534 shutdown_connections 2023-10-30T13:21:57.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:57.657+0000 7fd99b291640 1 -- 172.21.15.125:0/1027562534 wait complete. 2023-10-30T13:21:57.679 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:21:59.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:21:59.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.817+0000 7f4f7b172640 1 Processor -- start 2023-10-30T13:21:59.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.817+0000 7f4f7b172640 1 -- start start 2023-10-30T13:21:59.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.821+0000 7f4f7b172640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4f74071110 0x7f4f7406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:59.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.821+0000 7f4f7b172640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f74066cf0 con 0x7f4f740c6bd0 2023-10-30T13:21:59.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.821+0000 7f4f7b172640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f740c70d0 con 0x7f4f7406f7a0 2023-10-30T13:21:59.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.821+0000 7f4f7b172640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f740700c0 con 0x7f4f74071110 2023-10-30T13:21:59.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.821+0000 7f4f6bfff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f4f740c6bd0 0x7f4f74158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:46662/0 (socket says 172.21.15.125:46662) 2023-10-30T13:21:59.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.821+0000 7f4f6bfff640 1 -- 172.21.15.125:0/2726848048 learned_addr learned my addr 172.21.15.125:0/2726848048 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:21:59.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.821+0000 7f4f7970f640 1 --2- 172.21.15.125:0/2726848048 >> v2:172.21.15.136:3300/0 conn(0x7f4f74071110 0x7f4f7406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:59.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.821+0000 7f4f6b7fe640 1 -- 172.21.15.125:0/2726848048 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173343373 0 0) 0x7f4f74066cf0 con 0x7f4f740c6bd0 2023-10-30T13:21:59.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.821+0000 7f4f6b7fe640 1 -- 172.21.15.125:0/2726848048 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f5c003530 con 0x7f4f740c6bd0 2023-10-30T13:21:59.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.821+0000 7f4f7970f640 1 -- 172.21.15.125:0/2726848048 >> v1:172.21.15.136:6789/0 conn(0x7f4f7406f7a0 legacy=0x7f4f74069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:59.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.821+0000 7f4f7970f640 1 -- 172.21.15.125:0/2726848048 >> v1:172.21.15.125:6789/0 conn(0x7f4f740c6bd0 legacy=0x7f4f74158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:59.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.821+0000 7f4f6b7fe640 1 -- 172.21.15.125:0/2726848048 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1550819618 0 0) 0x7f4f5c003530 con 0x7f4f740c6bd0 2023-10-30T13:21:59.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.821+0000 7f4f7970f640 1 -- 172.21.15.125:0/2726848048 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f740c7390 con 0x7f4f74071110 2023-10-30T13:21:59.830 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.825+0000 7f4f7970f640 1 --2- 172.21.15.125:0/2726848048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f74071110 0x7f4f7406fb80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c009d90 tx=0x7f4f6c01f060 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bbc183dc4c71a06 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:59.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.825+0000 7f4f6b7fe640 1 -- 172.21.15.125:0/2726848048 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f64001680 con 0x7f4f74071110 2023-10-30T13:21:59.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.825+0000 7f4f6b7fe640 1 -- 172.21.15.125:0/2726848048 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f6c0a8bf0 con 0x7f4f74071110 2023-10-30T13:21:59.831 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.825+0000 7f4f6b7fe640 1 -- 172.21.15.125:0/2726848048 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f6c0b0440 con 0x7f4f74071110 2023-10-30T13:21:59.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.825+0000 7f4f7b172640 1 -- 172.21.15.125:0/2726848048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f74071110 msgr2=0x7f4f7406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:59.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.825+0000 7f4f7b172640 1 --2- 172.21.15.125:0/2726848048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f74071110 0x7f4f7406fb80 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c009d90 tx=0x7f4f6c01f060 comp rx=0 tx=0).stop 2023-10-30T13:21:59.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.825+0000 7f4f7b172640 1 -- 172.21.15.125:0/2726848048 shutdown_connections 2023-10-30T13:21:59.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.825+0000 7f4f7b172640 1 --2- 172.21.15.125:0/2726848048 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f74071110 0x7f4f7406fb80 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:59.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.825+0000 7f4f7b172640 1 -- 172.21.15.125:0/2726848048 >> 172.21.15.125:0/2726848048 conn(0x7f4f740c7910 msgr2=0x7f4f740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:21:59.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.825+0000 7f4f7b172640 1 -- 172.21.15.125:0/2726848048 shutdown_connections 2023-10-30T13:21:59.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.825+0000 7f4f7b172640 1 -- 172.21.15.125:0/2726848048 wait complete. 2023-10-30T13:21:59.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f7b172640 1 Processor -- start 2023-10-30T13:21:59.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f7b172640 1 -- start start 2023-10-30T13:21:59.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f7b172640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f7406f7a0 0x7f4f741541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:59.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f7b172640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f74071110 0x7f4f740c5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:59.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f7b172640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f740c6bd0 0x7f4f741527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:59.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f7b172640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f74154eb0 con 0x7f4f7406f7a0 2023-10-30T13:21:59.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f7b172640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f74154bf0 con 0x7f4f74071110 2023-10-30T13:21:59.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f7b172640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f741569c0 con 0x7f4f740c6bd0 2023-10-30T13:21:59.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f78f0e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f7406f7a0 0x7f4f741541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:59.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f78f0e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f7406f7a0 0x7f4f741541a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50910/0 (socket says 172.21.15.125:50910) 2023-10-30T13:21:59.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f78f0e640 1 -- 172.21.15.125:0/3477036883 learned_addr learned my addr 172.21.15.125:0/3477036883 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:21:59.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f6bfff640 1 --2- 172.21.15.125:0/3477036883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f74071110 0x7f4f740c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:59.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f7970f640 1 --2- 172.21.15.125:0/3477036883 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f740c6bd0 0x7f4f741527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:59.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f78f0e640 1 -- 172.21.15.125:0/3477036883 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f740c6bd0 msgr2=0x7f4f741527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:59.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f78f0e640 1 --2- 172.21.15.125:0/3477036883 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f740c6bd0 0x7f4f741527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:59.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f78f0e640 1 -- 172.21.15.125:0/3477036883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f74071110 msgr2=0x7f4f740c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:21:59.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f78f0e640 1 --2- 172.21.15.125:0/3477036883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f74071110 0x7f4f740c5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:21:59.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f78f0e640 1 -- 172.21.15.125:0/3477036883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f6c009a70 con 0x7f4f7406f7a0 2023-10-30T13:21:59.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f78f0e640 1 --2- 172.21.15.125:0/3477036883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f7406f7a0 0x7f4f741541a0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f4f7000c3c0 tx=0x7f4f7000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:59.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f697fa640 1 -- 172.21.15.125:0/3477036883 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f700052d0 con 0x7f4f7406f7a0 2023-10-30T13:21:59.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f697fa640 1 -- 172.21.15.125:0/3477036883 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f70005430 con 0x7f4f7406f7a0 2023-10-30T13:21:59.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f7b172640 1 -- 172.21.15.125:0/3477036883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f74156ca0 con 0x7f4f7406f7a0 2023-10-30T13:21:59.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f697fa640 1 -- 172.21.15.125:0/3477036883 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f70003840 con 0x7f4f7406f7a0 2023-10-30T13:21:59.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.829+0000 7f4f7b172640 1 -- 172.21.15.125:0/3477036883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f74157260 con 0x7f4f7406f7a0 2023-10-30T13:21:59.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.833+0000 7f4f7b172640 1 -- 172.21.15.125:0/3477036883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f74152610 con 0x7f4f7406f7a0 2023-10-30T13:21:59.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.833+0000 7f4f697fa640 1 -- 172.21.15.125:0/3477036883 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4f700039a0 con 0x7f4f7406f7a0 2023-10-30T13:21:59.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.833+0000 7f4f697fa640 1 --2- 172.21.15.125:0/3477036883 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f600664a0 0x7f4f60068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:21:59.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.833+0000 7f4f697fa640 1 -- 172.21.15.125:0/3477036883 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f4f70083e20 con 0x7f4f7406f7a0 2023-10-30T13:21:59.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.833+0000 7f4f6bfff640 1 --2- 172.21.15.125:0/3477036883 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f600664a0 0x7f4f60068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:21:59.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.833+0000 7f4f6bfff640 1 --2- 172.21.15.125:0/3477036883 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f600664a0 0x7f4f60068960 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f4f740cade0 tx=0x7f4f64001ac0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:21:59.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:21:59.837+0000 7f4f697fa640 1 -- 172.21.15.125:0/3477036883 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4f70050880 con 0x7f4f7406f7a0 2023-10-30T13:22:00.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:00.025+0000 7f4f7b172640 1 -- 172.21.15.125:0/3477036883 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f7404e170 con 0x7f4f7406f7a0 2023-10-30T13:22:00.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:00.025+0000 7f4f697fa640 1 -- 172.21.15.125:0/3477036883 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 248 v248) v1 ==== 95+0+5754 (secure 0 0 0) 0x7f4f70053ea0 con 0x7f4f7406f7a0 2023-10-30T13:22:00.032 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:22:00.032 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":248,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":248,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:21:50.903243+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:00.033 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 248 2023-10-30T13:22:00.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:00.029+0000 7f4f7b172640 1 -- 172.21.15.125:0/3477036883 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f600664a0 msgr2=0x7f4f60068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:00.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:00.029+0000 7f4f7b172640 1 --2- 172.21.15.125:0/3477036883 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f600664a0 0x7f4f60068960 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f4f740cade0 tx=0x7f4f64001ac0 comp rx=0 tx=0).stop 2023-10-30T13:22:00.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:00.029+0000 7f4f7b172640 1 -- 172.21.15.125:0/3477036883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f7406f7a0 msgr2=0x7f4f741541a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:00.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:00.029+0000 7f4f7b172640 1 --2- 172.21.15.125:0/3477036883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f7406f7a0 0x7f4f741541a0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f4f7000c3c0 tx=0x7f4f7000c880 comp rx=0 tx=0).stop 2023-10-30T13:22:00.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:00.029+0000 7f4f7b172640 1 -- 172.21.15.125:0/3477036883 shutdown_connections 2023-10-30T13:22:00.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:00.029+0000 7f4f7b172640 1 --2- 172.21.15.125:0/3477036883 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4f600664a0 0x7f4f60068960 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:00.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:00.029+0000 7f4f7b172640 1 --2- 172.21.15.125:0/3477036883 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4f740c6bd0 0x7f4f741527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:00.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:00.029+0000 7f4f7b172640 1 --2- 172.21.15.125:0/3477036883 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4f74071110 0x7f4f740c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:00.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:00.029+0000 7f4f7b172640 1 --2- 172.21.15.125:0/3477036883 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4f7406f7a0 0x7f4f741541a0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:00.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:00.029+0000 7f4f7b172640 1 -- 172.21.15.125:0/3477036883 >> 172.21.15.125:0/3477036883 conn(0x7f4f740c7910 msgr2=0x7f4f740c94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:00.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:00.029+0000 7f4f7b172640 1 -- 172.21.15.125:0/3477036883 shutdown_connections 2023-10-30T13:22:00.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:00.029+0000 7f4f7b172640 1 -- 172.21.15.125:0/3477036883 wait complete. 2023-10-30T13:22:00.053 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:22:02.054 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:02.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f294609c640 1 Processor -- start 2023-10-30T13:22:02.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f294609c640 1 -- start start 2023-10-30T13:22:02.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f294609c640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f29400c6b80 0x7f29400bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:02.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f294609c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29400c6f60 0x7f29400c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:02.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f294609c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29400c0a50 con 0x7f29400c8430 2023-10-30T13:22:02.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f294609c640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f294006f6e0 con 0x7f29400c6b80 2023-10-30T13:22:02.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f294609c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294006f9c0 con 0x7f29400c6f60 2023-10-30T13:22:02.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f293f7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f29400c6b80 0x7f29400bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:02.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f293effd640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29400c6f60 0x7f29400c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:02.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f293f7fe640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f29400c6b80 0x7f29400bfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:44148/0 (socket says 172.21.15.125:44148) 2023-10-30T13:22:02.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f293f7fe640 1 -- 172.21.15.125:0/578579964 learned_addr learned my addr 172.21.15.125:0/578579964 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:02.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f293f7fe640 1 -- 172.21.15.125:0/578579964 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29400c6f60 msgr2=0x7f29400c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:02.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f293f7fe640 1 --2- 172.21.15.125:0/578579964 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29400c6f60 0x7f29400c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:02.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f293f7fe640 1 -- 172.21.15.125:0/578579964 >> v1:172.21.15.136:6789/0 conn(0x7f29400c8430 legacy=0x7f294013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:02.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.197+0000 7f293f7fe640 1 -- 172.21.15.125:0/578579964 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294006fca0 con 0x7f29400c6b80 2023-10-30T13:22:02.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.201+0000 7f293f7fe640 1 --2- 172.21.15.125:0/578579964 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29400c6b80 0x7f29400bfec0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2930009870 tx=0x7f293001e6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26df7d4b19b843da server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:02.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.201+0000 7f293e7fc640 1 -- 172.21.15.125:0/578579964 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29300b2600 con 0x7f29400c6b80 2023-10-30T13:22:02.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.201+0000 7f293e7fc640 1 -- 172.21.15.125:0/578579964 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29300b2760 con 0x7f29400c6b80 2023-10-30T13:22:02.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.201+0000 7f293e7fc640 1 -- 172.21.15.125:0/578579964 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29300b2af0 con 0x7f29400c6b80 2023-10-30T13:22:02.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.201+0000 7f294609c640 1 -- 172.21.15.125:0/578579964 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29400c6b80 msgr2=0x7f29400bfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:02.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.201+0000 7f294609c640 1 --2- 172.21.15.125:0/578579964 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29400c6b80 0x7f29400bfec0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2930009870 tx=0x7f293001e6c0 comp rx=0 tx=0).stop 2023-10-30T13:22:02.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.201+0000 7f294609c640 1 -- 172.21.15.125:0/578579964 shutdown_connections 2023-10-30T13:22:02.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.201+0000 7f294609c640 1 --2- 172.21.15.125:0/578579964 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29400c6f60 0x7f29400c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:02.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.201+0000 7f294609c640 1 --2- 172.21.15.125:0/578579964 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29400c6b80 0x7f29400bfec0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:02.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.201+0000 7f294609c640 1 -- 172.21.15.125:0/578579964 >> 172.21.15.125:0/578579964 conn(0x7f29400c7910 msgr2=0x7f29400c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:02.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.201+0000 7f294609c640 1 -- 172.21.15.125:0/578579964 shutdown_connections 2023-10-30T13:22:02.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.201+0000 7f294609c640 1 -- 172.21.15.125:0/578579964 wait complete. 2023-10-30T13:22:02.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f294609c640 1 Processor -- start 2023-10-30T13:22:02.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f294609c640 1 -- start start 2023-10-30T13:22:02.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f294609c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29400c6b80 0x7f2940160600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:02.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f294609c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29400c6f60 0x7f294013aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:02.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f294609c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29400c8430 0x7f294013ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:02.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f294609c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29400c9790 con 0x7f29400c6f60 2023-10-30T13:22:02.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f294609c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2940132000 con 0x7f29400c8430 2023-10-30T13:22:02.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f294609c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2940157790 con 0x7f29400c6b80 2023-10-30T13:22:02.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f293effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29400c6f60 0x7f294013aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:02.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f293effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29400c6f60 0x7f294013aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39152/0 (socket says 172.21.15.125:39152) 2023-10-30T13:22:02.213 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f293effd640 1 -- 172.21.15.125:0/1197775125 learned_addr learned my addr 172.21.15.125:0/1197775125 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:02.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f293f7fe640 1 --2- 172.21.15.125:0/1197775125 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29400c6b80 0x7f2940160600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:02.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f293ffff640 1 --2- 172.21.15.125:0/1197775125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29400c8430 0x7f294013ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:02.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f293effd640 1 -- 172.21.15.125:0/1197775125 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29400c6b80 msgr2=0x7f2940160600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:02.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f293effd640 1 --2- 172.21.15.125:0/1197775125 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29400c6b80 0x7f2940160600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:02.214 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f293effd640 1 -- 172.21.15.125:0/1197775125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29400c8430 msgr2=0x7f294013ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:02.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f293effd640 1 --2- 172.21.15.125:0/1197775125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29400c8430 0x7f294013ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:02.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f293effd640 1 -- 172.21.15.125:0/1197775125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29300094d0 con 0x7f29400c6f60 2023-10-30T13:22:02.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.205+0000 7f293effd640 1 --2- 172.21.15.125:0/1197775125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29400c6f60 0x7f294013aa30 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f293400e930 tx=0x7f293400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:02.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.209+0000 7f2927fff640 1 -- 172.21.15.125:0/1197775125 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2934009690 con 0x7f29400c6f60 2023-10-30T13:22:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.209+0000 7f2927fff640 1 -- 172.21.15.125:0/1197775125 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f293400ca00 con 0x7f29400c6f60 2023-10-30T13:22:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.209+0000 7f2927fff640 1 -- 172.21.15.125:0/1197775125 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f293400cd70 con 0x7f29400c6f60 2023-10-30T13:22:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.209+0000 7f294609c640 1 -- 172.21.15.125:0/1197775125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2940158a80 con 0x7f29400c6f60 2023-10-30T13:22:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.209+0000 7f294609c640 1 -- 172.21.15.125:0/1197775125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2940133b90 con 0x7f29400c6f60 2023-10-30T13:22:02.217 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.209+0000 7f294609c640 1 -- 172.21.15.125:0/1197775125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f294004e170 con 0x7f29400c6f60 2023-10-30T13:22:02.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.209+0000 7f2927fff640 1 -- 172.21.15.125:0/1197775125 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2934013470 con 0x7f29400c6f60 2023-10-30T13:22:02.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.209+0000 7f2927fff640 1 --2- 172.21.15.125:0/1197775125 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f292c0664a0 0x7f292c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:02.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.209+0000 7f2927fff640 1 -- 172.21.15.125:0/1197775125 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f2934084fb0 con 0x7f29400c6f60 2023-10-30T13:22:02.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.209+0000 7f293f7fe640 1 --2- 172.21.15.125:0/1197775125 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f292c0664a0 0x7f292c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:02.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.213+0000 7f293f7fe640 1 --2- 172.21.15.125:0/1197775125 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f292c0664a0 0x7f292c068960 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f29300b1000 tx=0x7f2930008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:02.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.213+0000 7f2927fff640 1 -- 172.21.15.125:0/1197775125 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2934051980 con 0x7f29400c6f60 2023-10-30T13:22:02.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.397+0000 7f294609c640 1 -- 172.21.15.125:0/1197775125 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2940135450 con 0x7f29400c6f60 2023-10-30T13:22:02.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.397+0000 7f2927fff640 1 -- 172.21.15.125:0/1197775125 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f2934021020 con 0x7f29400c6f60 2023-10-30T13:22:02.403 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:22:02.403 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":249,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":249,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:22:00.571802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:resolve","state_seq":13,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:02.403 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 249 2023-10-30T13:22:02.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.401+0000 7f294609c640 1 -- 172.21.15.125:0/1197775125 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f292c0664a0 msgr2=0x7f292c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.401+0000 7f294609c640 1 --2- 172.21.15.125:0/1197775125 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f292c0664a0 0x7f292c068960 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f29300b1000 tx=0x7f2930008040 comp rx=0 tx=0).stop 2023-10-30T13:22:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.401+0000 7f294609c640 1 -- 172.21.15.125:0/1197775125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29400c6f60 msgr2=0x7f294013aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.401+0000 7f294609c640 1 --2- 172.21.15.125:0/1197775125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29400c6f60 0x7f294013aa30 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f293400e930 tx=0x7f293400edf0 comp rx=0 tx=0).stop 2023-10-30T13:22:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.401+0000 7f294609c640 1 -- 172.21.15.125:0/1197775125 shutdown_connections 2023-10-30T13:22:02.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.401+0000 7f294609c640 1 --2- 172.21.15.125:0/1197775125 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f292c0664a0 0x7f292c068960 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:02.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.401+0000 7f294609c640 1 --2- 172.21.15.125:0/1197775125 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f29400c8430 0x7f294013ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:02.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.401+0000 7f294609c640 1 --2- 172.21.15.125:0/1197775125 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f29400c6f60 0x7f294013aa30 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:02.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.401+0000 7f294609c640 1 --2- 172.21.15.125:0/1197775125 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f29400c6b80 0x7f2940160600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:02.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.401+0000 7f294609c640 1 -- 172.21.15.125:0/1197775125 >> 172.21.15.125:0/1197775125 conn(0x7f29400c7910 msgr2=0x7f29400bfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:02.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.401+0000 7f294609c640 1 -- 172.21.15.125:0/1197775125 shutdown_connections 2023-10-30T13:22:02.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:02.401+0000 7f294609c640 1 -- 172.21.15.125:0/1197775125 wait complete. 2023-10-30T13:22:02.424 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:22:04.425 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:04.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.565+0000 7fb1e3994640 1 Processor -- start 2023-10-30T13:22:04.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.565+0000 7fb1e3994640 1 -- start start 2023-10-30T13:22:04.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.569+0000 7fb1e3994640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1dc06f7a0 0x7fb1dc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:04.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.569+0000 7fb1e3994640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb1dc071110 0x7fb1dc158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:04.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.569+0000 7fb1e3994640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1dc066cf0 con 0x7fb1dc0c6bd0 2023-10-30T13:22:04.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.569+0000 7fb1e3994640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1dc0700c0 con 0x7fb1dc071110 2023-10-30T13:22:04.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.569+0000 7fb1e3994640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc0c70d0 con 0x7fb1dc06f7a0 2023-10-30T13:22:04.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.569+0000 7fb1e1f31640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb1dc071110 0x7fb1dc158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:04.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.569+0000 7fb1e1f31640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb1dc071110 0x7fb1dc158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39160/0 (socket says 172.21.15.125:39160) 2023-10-30T13:22:04.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.569+0000 7fb1e1f31640 1 -- 172.21.15.125:0/2547776293 learned_addr learned my addr 172.21.15.125:0/2547776293 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:04.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.569+0000 7fb1e1730640 1 --2- 172.21.15.125:0/2547776293 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1dc06f7a0 0x7fb1dc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:04.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.569+0000 7fb1e1f31640 1 -- 172.21.15.125:0/2547776293 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1dc06f7a0 msgr2=0x7fb1dc06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:04.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.569+0000 7fb1e1f31640 1 --2- 172.21.15.125:0/2547776293 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1dc06f7a0 0x7fb1dc06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:04.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.569+0000 7fb1e1f31640 1 -- 172.21.15.125:0/2547776293 >> v1:172.21.15.136:6789/0 conn(0x7fb1dc0c6bd0 legacy=0x7fb1dc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:04.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.569+0000 7fb1e1f31640 1 -- 172.21.15.125:0/2547776293 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1dc0c7390 con 0x7fb1dc071110 2023-10-30T13:22:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.569+0000 7fb1e1f31640 1 --2- 172.21.15.125:0/2547776293 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1dc071110 0x7fb1dc158110 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009e50 tx=0x7fb1d401f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b04f2a3d799735e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1d3fff640 1 -- 172.21.15.125:0/2547776293 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d40af400 con 0x7fb1dc071110 2023-10-30T13:22:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1d3fff640 1 -- 172.21.15.125:0/2547776293 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d40ab050 con 0x7fb1dc071110 2023-10-30T13:22:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1d3fff640 1 -- 172.21.15.125:0/2547776293 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d40af780 con 0x7fb1dc071110 2023-10-30T13:22:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 -- 172.21.15.125:0/2547776293 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1dc071110 msgr2=0x7fb1dc158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 --2- 172.21.15.125:0/2547776293 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1dc071110 0x7fb1dc158110 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4009e50 tx=0x7fb1d401f1f0 comp rx=0 tx=0).stop 2023-10-30T13:22:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 -- 172.21.15.125:0/2547776293 shutdown_connections 2023-10-30T13:22:04.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 --2- 172.21.15.125:0/2547776293 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1dc071110 0x7fb1dc158110 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 --2- 172.21.15.125:0/2547776293 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1dc06f7a0 0x7fb1dc06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 -- 172.21.15.125:0/2547776293 >> 172.21.15.125:0/2547776293 conn(0x7fb1dc0c7910 msgr2=0x7fb1dc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 -- 172.21.15.125:0/2547776293 shutdown_connections 2023-10-30T13:22:04.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 -- 172.21.15.125:0/2547776293 wait complete. 2023-10-30T13:22:04.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 Processor -- start 2023-10-30T13:22:04.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 -- start start 2023-10-30T13:22:04.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1dc06f7a0 0x7fb1dc1f1a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:04.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1dc071110 0x7fb1dc157700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:04.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1dc0c6bd0 0x7fb1dc157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc1588a0 con 0x7fb1dc06f7a0 2023-10-30T13:22:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc06c680 con 0x7fb1dc071110 2023-10-30T13:22:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e3994640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc14fd40 con 0x7fb1dc0c6bd0 2023-10-30T13:22:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e1730640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1dc06f7a0 0x7fb1dc1f1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:04.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e1730640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1dc06f7a0 0x7fb1dc1f1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39166/0 (socket says 172.21.15.125:39166) 2023-10-30T13:22:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.573+0000 7fb1e1730640 1 -- 172.21.15.125:0/471373622 learned_addr learned my addr 172.21.15.125:0/471373622 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1e1f31640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1dc0c6bd0 0x7fb1dc157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1e0f2f640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1dc071110 0x7fb1dc157700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1e1730640 1 -- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1dc0c6bd0 msgr2=0x7fb1dc157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:04.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1e1730640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1dc0c6bd0 0x7fb1dc157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1e1730640 1 -- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1dc071110 msgr2=0x7fb1dc157700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1e1730640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1dc071110 0x7fb1dc157700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1e1730640 1 -- 172.21.15.125:0/471373622 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d4009b30 con 0x7fb1dc06f7a0 2023-10-30T13:22:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1e0f2f640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1dc071110 0x7fb1dc157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:22:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1e1730640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1dc06f7a0 0x7fb1dc1f1a20 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800e870 tx=0x7fb1d800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:04.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1d1ffb640 1 -- 172.21.15.125:0/471373622 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d8009510 con 0x7fb1dc06f7a0 2023-10-30T13:22:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1d1ffb640 1 -- 172.21.15.125:0/471373622 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1d800eec0 con 0x7fb1dc06f7a0 2023-10-30T13:22:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1d1ffb640 1 -- 172.21.15.125:0/471373622 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1d800cb40 con 0x7fb1dc06f7a0 2023-10-30T13:22:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1e3994640 1 -- 172.21.15.125:0/471373622 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc150020 con 0x7fb1dc06f7a0 2023-10-30T13:22:04.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1e3994640 1 -- 172.21.15.125:0/471373622 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc158af0 con 0x7fb1dc06f7a0 2023-10-30T13:22:04.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1e3994640 1 -- 172.21.15.125:0/471373622 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc04e170 con 0x7fb1dc06f7a0 2023-10-30T13:22:04.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1d1ffb640 1 -- 172.21.15.125:0/471373622 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb1d800cca0 con 0x7fb1dc06f7a0 2023-10-30T13:22:04.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1d1ffb640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1c00664a0 0x7fb1c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:04.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.577+0000 7fb1d1ffb640 1 -- 172.21.15.125:0/471373622 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7fb1d8084db0 con 0x7fb1dc06f7a0 2023-10-30T13:22:04.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.581+0000 7fb1e0f2f640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1c00664a0 0x7fb1c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:04.588 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.581+0000 7fb1e0f2f640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1c00664a0 0x7fb1c0068960 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc0c4560 tx=0x7fb1cc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:04.589 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.581+0000 7fb1d1ffb640 1 -- 172.21.15.125:0/471373622 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb1d8089070 con 0x7fb1dc06f7a0 2023-10-30T13:22:04.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.761+0000 7fb1e3994640 1 -- 172.21.15.125:0/471373622 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1dc1510d0 con 0x7fb1dc06f7a0 2023-10-30T13:22:04.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.761+0000 7fb1d1ffb640 1 -- 172.21.15.125:0/471373622 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 249 v249) v1 ==== 95+0+5756 (secure 0 0 0) 0x7fb1d8021090 con 0x7fb1dc06f7a0 2023-10-30T13:22:04.767 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:22:04.768 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":249,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":249,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:22:00.571802+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:resolve","state_seq":13,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:04.768 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 249 2023-10-30T13:22:04.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.765+0000 7fb1e3994640 1 -- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1c00664a0 msgr2=0x7fb1c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:04.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.765+0000 7fb1e3994640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1c00664a0 0x7fb1c0068960 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fb1dc0c4560 tx=0x7fb1cc009040 comp rx=0 tx=0).stop 2023-10-30T13:22:04.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.765+0000 7fb1e3994640 1 -- 172.21.15.125:0/471373622 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1dc06f7a0 msgr2=0x7fb1dc1f1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:04.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.765+0000 7fb1e3994640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1dc06f7a0 0x7fb1dc1f1a20 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7fb1d800e870 tx=0x7fb1d800ec70 comp rx=0 tx=0).stop 2023-10-30T13:22:04.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.765+0000 7fb1e3994640 1 -- 172.21.15.125:0/471373622 shutdown_connections 2023-10-30T13:22:04.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.765+0000 7fb1e3994640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1c00664a0 0x7fb1c0068960 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:04.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.765+0000 7fb1e3994640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1dc0c6bd0 0x7fb1dc157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:04.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.765+0000 7fb1e3994640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1dc071110 0x7fb1dc157700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:04.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.765+0000 7fb1e3994640 1 --2- 172.21.15.125:0/471373622 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb1dc06f7a0 0x7fb1dc1f1a20 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:04.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.765+0000 7fb1e3994640 1 -- 172.21.15.125:0/471373622 >> 172.21.15.125:0/471373622 conn(0x7fb1dc0c7910 msgr2=0x7fb1dc0ca670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:04.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.765+0000 7fb1e3994640 1 -- 172.21.15.125:0/471373622 shutdown_connections 2023-10-30T13:22:04.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:04.765+0000 7fb1e3994640 1 -- 172.21.15.125:0/471373622 wait complete. 2023-10-30T13:22:04.789 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 249, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 249, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_8033": { "addr": "172.21.15.136:6835/3480968625", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3480968625, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3480968625, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8033, "incarnation": 248, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:resolve", "state_seq": 13 }, "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 36 }, "gid_9600": { "addr": "172.21.15.125:6836/2077416857", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2077416857, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2077416857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9600, "incarnation": 241, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 42 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:22:00.571802+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8033, "mds_1": 9150, "mds_2": 9600 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:22:06.790 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:06.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.933+0000 7f9ffd101640 1 Processor -- start 2023-10-30T13:22:06.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.933+0000 7f9ffd101640 1 -- start start 2023-10-30T13:22:06.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ffd101640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f9ff815d6f0 0x7f9ff815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:06.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ffd101640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ff80cf160 0x7f9ff80cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:06.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ffd101640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff80cb800 con 0x7f9ff81538a0 2023-10-30T13:22:06.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ffd101640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff80cb330 con 0x7f9ff815d6f0 2023-10-30T13:22:06.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ffd101640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff80cad70 con 0x7f9ff80cf160 2023-10-30T13:22:06.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff6d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f9ff81538a0 0x7f9ff80ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:47652/0 (socket says 172.21.15.125:47652) 2023-10-30T13:22:06.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff6d9d640 1 -- 172.21.15.125:0/9899096 learned_addr learned my addr 172.21.15.125:0/9899096 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:22:06.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff659c640 1 --2- 172.21.15.125:0/9899096 >> v2:172.21.15.136:3300/0 conn(0x7f9ff815d6f0 0x7f9ff815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:06.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff759e640 1 --2- 172.21.15.125:0/9899096 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ff80cf160 0x7f9ff80cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:06.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 -- 172.21.15.125:0/9899096 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456846011 0 0) 0x7f9ff80cb800 con 0x7f9ff81538a0 2023-10-30T13:22:06.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 -- 172.21.15.125:0/9899096 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe0003530 con 0x7f9ff81538a0 2023-10-30T13:22:06.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 -- 172.21.15.125:0/9899096 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2239866641 0 0) 0x7f9fe0003530 con 0x7f9ff81538a0 2023-10-30T13:22:06.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 -- 172.21.15.125:0/9899096 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ff80cf160 msgr2=0x7f9ff80cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:06.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 --2- 172.21.15.125:0/9899096 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ff80cf160 0x7f9ff80cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:06.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 -- 172.21.15.125:0/9899096 >> v2:172.21.15.136:3300/0 conn(0x7f9ff815d6f0 msgr2=0x7f9ff815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:06.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 --2- 172.21.15.125:0/9899096 >> v2:172.21.15.136:3300/0 conn(0x7f9ff815d6f0 0x7f9ff815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:06.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 -- 172.21.15.125:0/9899096 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff80cb050 con 0x7f9ff81538a0 2023-10-30T13:22:06.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 -- 172.21.15.125:0/9899096 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f9fe8001680 con 0x7f9ff81538a0 2023-10-30T13:22:06.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 -- 172.21.15.125:0/9899096 >> v1:172.21.15.125:6789/0 conn(0x7f9ff81538a0 legacy=0x7f9ff80ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:06.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 --2- 172.21.15.125:0/9899096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fe0003980 0x7f9fe0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:06.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 --2- 172.21.15.125:0/9899096 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ff80cf160 0x7f9fe00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:06.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 --2- 172.21.15.125:0/9899096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fe0004d10 0x7f9fe0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:06.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 -- 172.21.15.125:0/9899096 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff80cb330 con 0x7f9fe0003980 2023-10-30T13:22:06.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 -- 172.21.15.125:0/9899096 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff80cad70 con 0x7f9fe0004d10 2023-10-30T13:22:06.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff5d9b640 1 -- 172.21.15.125:0/9899096 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe000b990 con 0x7f9ff80cf160 2023-10-30T13:22:06.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff6d9d640 1 --2- 172.21.15.125:0/9899096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fe0003980 0x7f9fe0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:06.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff759e640 1 --2- 172.21.15.125:0/9899096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fe0004d10 0x7f9fe0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:06.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff659c640 1 --2- 172.21.15.125:0/9899096 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ff80cf160 0x7f9fe00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:06.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff6d9d640 1 -- 172.21.15.125:0/9899096 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ff80cf160 msgr2=0x7f9fe00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:06.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff6d9d640 1 --2- 172.21.15.125:0/9899096 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9ff80cf160 0x7f9fe00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:06.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff6d9d640 1 -- 172.21.15.125:0/9899096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fe0004d10 msgr2=0x7f9fe0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:06.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff6d9d640 1 --2- 172.21.15.125:0/9899096 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9fe0004d10 0x7f9fe0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:06.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.937+0000 7f9ff6d9d640 1 -- 172.21.15.125:0/9899096 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe0003530 con 0x7f9fe0003980 2023-10-30T13:22:06.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.941+0000 7f9ff6d9d640 1 --2- 172.21.15.125:0/9899096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fe0003980 0x7f9fe0003da0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f9fe800c210 tx=0x7f9fe801f3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a9591e6391d647 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:06.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.941+0000 7f9ff5d9b640 1 -- 172.21.15.125:0/9899096 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe8005ae0 con 0x7f9fe0003980 2023-10-30T13:22:06.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.941+0000 7f9ff5d9b640 1 -- 172.21.15.125:0/9899096 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe80ab070 con 0x7f9fe0003980 2023-10-30T13:22:06.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.941+0000 7f9ff5d9b640 1 -- 172.21.15.125:0/9899096 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe8005ea0 con 0x7f9fe0003980 2023-10-30T13:22:06.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.941+0000 7f9ffd101640 1 -- 172.21.15.125:0/9899096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fe0003980 msgr2=0x7f9fe0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:06.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.941+0000 7f9ffd101640 1 --2- 172.21.15.125:0/9899096 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9fe0003980 0x7f9fe0003da0 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7f9fe800c210 tx=0x7f9fe801f3b0 comp rx=0 tx=0).stop 2023-10-30T13:22:06.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.941+0000 7f9ff759e640 1 -- 172.21.15.125:0/9899096 reap_dead start 2023-10-30T13:22:06.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.941+0000 7f9ffd101640 1 -- 172.21.15.125:0/9899096 shutdown_connections 2023-10-30T13:22:06.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.941+0000 7f9ffd101640 1 -- 172.21.15.125:0/9899096 >> 172.21.15.125:0/9899096 conn(0x7f9ff806bfe0 msgr2=0x7f9ff806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:06.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.941+0000 7f9ffd101640 1 -- 172.21.15.125:0/9899096 shutdown_connections 2023-10-30T13:22:06.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.941+0000 7f9ffd101640 1 -- 172.21.15.125:0/9899096 wait complete. 2023-10-30T13:22:06.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ffd101640 1 Processor -- start 2023-10-30T13:22:06.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ffd101640 1 -- start start 2023-10-30T13:22:06.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ffd101640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fe0003980 0x7f9ff8143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:06.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ffd101640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ff80cd9c0 0x7f9ff81482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:06.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ffd101640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ff81e6300 0x7f9ff81e67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:06.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ffd101640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff80b51d0 con 0x7f9ff81e6300 2023-10-30T13:22:06.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ffd101640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff8157170 con 0x7f9ff80cd9c0 2023-10-30T13:22:06.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ffd101640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff8148be0 con 0x7f9fe0003980 2023-10-30T13:22:06.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ff759e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ff81e6300 0x7f9ff81e67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:06.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ff759e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ff81e6300 0x7f9ff81e67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39170/0 (socket says 172.21.15.125:39170) 2023-10-30T13:22:06.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ff759e640 1 -- 172.21.15.125:0/3864719761 learned_addr learned my addr 172.21.15.125:0/3864719761 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:06.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ff659c640 1 --2- 172.21.15.125:0/3864719761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ff80cd9c0 0x7f9ff81482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:06.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ff6d9d640 1 --2- 172.21.15.125:0/3864719761 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fe0003980 0x7f9ff8143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:06.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ff759e640 1 -- 172.21.15.125:0/3864719761 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fe0003980 msgr2=0x7f9ff8143510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:06.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ff759e640 1 --2- 172.21.15.125:0/3864719761 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fe0003980 0x7f9ff8143510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:06.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ff759e640 1 -- 172.21.15.125:0/3864719761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ff80cd9c0 msgr2=0x7f9ff81482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:06.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ff759e640 1 --2- 172.21.15.125:0/3864719761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ff80cd9c0 0x7f9ff81482e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:06.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ff759e640 1 -- 172.21.15.125:0/3864719761 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff80b4550 con 0x7f9ff81e6300 2023-10-30T13:22:06.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ff759e640 1 --2- 172.21.15.125:0/3864719761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ff81e6300 0x7f9ff81e67b0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f9ff806cb00 tx=0x7f9ff00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:06.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9fdf7fe640 1 -- 172.21.15.125:0/3864719761 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff000e070 con 0x7f9ff81e6300 2023-10-30T13:22:06.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9fdf7fe640 1 -- 172.21.15.125:0/3864719761 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff0011e10 con 0x7f9ff81e6300 2023-10-30T13:22:06.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9fdf7fe640 1 -- 172.21.15.125:0/3864719761 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff00039a0 con 0x7f9ff81e6300 2023-10-30T13:22:06.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ffd101640 1 -- 172.21.15.125:0/3864719761 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff80b4830 con 0x7f9ff81e6300 2023-10-30T13:22:06.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.945+0000 7f9ffd101640 1 -- 172.21.15.125:0/3864719761 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff80b4df0 con 0x7f9ff81e6300 2023-10-30T13:22:06.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.949+0000 7f9ffd101640 1 -- 172.21.15.125:0/3864719761 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff81f2310 con 0x7f9ff81e6300 2023-10-30T13:22:06.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.949+0000 7f9fdf7fe640 1 -- 172.21.15.125:0/3864719761 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9ff0003b00 con 0x7f9ff81e6300 2023-10-30T13:22:06.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.949+0000 7f9fdf7fe640 1 --2- 172.21.15.125:0/3864719761 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fe40664d0 0x7f9fe4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:06.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.949+0000 7f9fdf7fe640 1 -- 172.21.15.125:0/3864719761 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f9ff0084e40 con 0x7f9ff81e6300 2023-10-30T13:22:06.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.949+0000 7f9ff6d9d640 1 --2- 172.21.15.125:0/3864719761 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fe40664d0 0x7f9fe4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:06.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.949+0000 7f9ff6d9d640 1 --2- 172.21.15.125:0/3864719761 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fe40664d0 0x7f9fe4068990 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80ae000 tx=0x7f9fe80b4040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:06.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:06.953+0000 7f9fdf7fe640 1 -- 172.21.15.125:0/3864719761 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9ff81f2310 con 0x7f9ff81e6300 2023-10-30T13:22:07.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:07.133+0000 7f9ffd101640 1 -- 172.21.15.125:0/3864719761 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ff804e170 con 0x7f9ff81e6300 2023-10-30T13:22:07.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:07.137+0000 7f9fdf7fe640 1 -- 172.21.15.125:0/3864719761 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 250 v250) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f9ff0051800 con 0x7f9ff81e6300 2023-10-30T13:22:07.140 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:22:07.141 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":250,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":250,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:22:06.404246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:07.141 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 250 2023-10-30T13:22:07.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:07.137+0000 7f9ffd101640 1 -- 172.21.15.125:0/3864719761 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fe40664d0 msgr2=0x7f9fe4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:07.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:07.137+0000 7f9ffd101640 1 --2- 172.21.15.125:0/3864719761 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fe40664d0 0x7f9fe4068990 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f9fe80ae000 tx=0x7f9fe80b4040 comp rx=0 tx=0).stop 2023-10-30T13:22:07.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:07.137+0000 7f9ffd101640 1 -- 172.21.15.125:0/3864719761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ff81e6300 msgr2=0x7f9ff81e67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:07.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:07.137+0000 7f9ffd101640 1 --2- 172.21.15.125:0/3864719761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ff81e6300 0x7f9ff81e67b0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f9ff806cb00 tx=0x7f9ff00115a0 comp rx=0 tx=0).stop 2023-10-30T13:22:07.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:07.137+0000 7f9ffd101640 1 -- 172.21.15.125:0/3864719761 shutdown_connections 2023-10-30T13:22:07.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:07.137+0000 7f9ffd101640 1 --2- 172.21.15.125:0/3864719761 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f9fe40664d0 0x7f9fe4068990 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:07.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:07.137+0000 7f9ffd101640 1 --2- 172.21.15.125:0/3864719761 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9ff81e6300 0x7f9ff81e67b0 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:07.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:07.137+0000 7f9ffd101640 1 --2- 172.21.15.125:0/3864719761 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f9ff80cd9c0 0x7f9ff81482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:07.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:07.137+0000 7f9ffd101640 1 --2- 172.21.15.125:0/3864719761 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9fe0003980 0x7f9ff8143510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:07.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:07.137+0000 7f9ffd101640 1 -- 172.21.15.125:0/3864719761 >> 172.21.15.125:0/3864719761 conn(0x7f9ff806bfe0 msgr2=0x7f9ff81510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:07.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:07.137+0000 7f9ffd101640 1 -- 172.21.15.125:0/3864719761 shutdown_connections 2023-10-30T13:22:07.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:07.137+0000 7f9ffd101640 1 -- 172.21.15.125:0/3864719761 wait complete. 2023-10-30T13:22:07.161 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:22:09.162 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:09.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f81108ad640 1 Processor -- start 2023-10-30T13:22:09.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f81108ad640 1 -- start start 2023-10-30T13:22:09.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f81108ad640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f810c1538f0 0x7f810c159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:09.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f81108ad640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f810c15cd50 0x7f810c0c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:09.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f81108ad640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f810c0cefa0 0x7f810c0cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:09.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f81108ad640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810c15d300 con 0x7f810c0cefa0 2023-10-30T13:22:09.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f81108ad640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f810c15d5e0 con 0x7f810c1538f0 2023-10-30T13:22:09.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f81108ad640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810c15d8c0 con 0x7f810c15cd50 2023-10-30T13:22:09.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f810bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f810c0cefa0 0x7f810c0cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:09.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f810bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f810c0cefa0 0x7f810c0cf3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39184/0 (socket says 172.21.15.125:39184) 2023-10-30T13:22:09.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f810bfff640 1 -- 172.21.15.125:0/3685615234 learned_addr learned my addr 172.21.15.125:0/3685615234 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:09.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f810affd640 1 --2- 172.21.15.125:0/3685615234 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f810c15cd50 0x7f810c0c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:09.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f810b7fe640 1 --2- 172.21.15.125:0/3685615234 >> v2:172.21.15.136:3300/0 conn(0x7f810c1538f0 0x7f810c159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:09.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f810bfff640 1 -- 172.21.15.125:0/3685615234 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f810c15cd50 msgr2=0x7f810c0c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:09.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f810bfff640 1 --2- 172.21.15.125:0/3685615234 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f810c15cd50 0x7f810c0c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:09.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f810bfff640 1 -- 172.21.15.125:0/3685615234 >> v2:172.21.15.136:3300/0 conn(0x7f810c1538f0 msgr2=0x7f810c159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:09.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f810bfff640 1 --2- 172.21.15.125:0/3685615234 >> v2:172.21.15.136:3300/0 conn(0x7f810c1538f0 0x7f810c159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:09.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f810bfff640 1 -- 172.21.15.125:0/3685615234 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810c15dba0 con 0x7f810c0cefa0 2023-10-30T13:22:09.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.305+0000 7f810b7fe640 1 --2- 172.21.15.125:0/3685615234 >> v2:172.21.15.136:3300/0 conn(0x7f810c1538f0 0x7f810c159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:09.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.309+0000 7f810bfff640 1 --2- 172.21.15.125:0/3685615234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810c0cefa0 0x7f810c0cf3f0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f810c0c9cb0 tx=0x7f810401f1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0a55093a82ec3c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:09.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.309+0000 7f810a7fc640 1 -- 172.21.15.125:0/3685615234 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81040af400 con 0x7f810c0cefa0 2023-10-30T13:22:09.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.309+0000 7f810a7fc640 1 -- 172.21.15.125:0/3685615234 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81040ab050 con 0x7f810c0cefa0 2023-10-30T13:22:09.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.309+0000 7f810a7fc640 1 -- 172.21.15.125:0/3685615234 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81040af780 con 0x7f810c0cefa0 2023-10-30T13:22:09.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.309+0000 7f81108ad640 1 -- 172.21.15.125:0/3685615234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810c0cefa0 msgr2=0x7f810c0cf3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:09.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.309+0000 7f81108ad640 1 --2- 172.21.15.125:0/3685615234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810c0cefa0 0x7f810c0cf3f0 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f810c0c9cb0 tx=0x7f810401f1c0 comp rx=0 tx=0).stop 2023-10-30T13:22:09.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.309+0000 7f81108ad640 1 -- 172.21.15.125:0/3685615234 shutdown_connections 2023-10-30T13:22:09.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.309+0000 7f81108ad640 1 --2- 172.21.15.125:0/3685615234 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810c0cefa0 0x7f810c0cf3f0 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:09.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.309+0000 7f81108ad640 1 --2- 172.21.15.125:0/3685615234 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f810c15cd50 0x7f810c0c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:09.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.309+0000 7f81108ad640 1 --2- 172.21.15.125:0/3685615234 >> v2:172.21.15.136:3300/0 conn(0x7f810c1538f0 0x7f810c159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:09.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.309+0000 7f81108ad640 1 -- 172.21.15.125:0/3685615234 >> 172.21.15.125:0/3685615234 conn(0x7f810c06a640 msgr2=0x7f810c065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:09.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.309+0000 7f81108ad640 1 -- 172.21.15.125:0/3685615234 shutdown_connections 2023-10-30T13:22:09.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.309+0000 7f81108ad640 1 -- 172.21.15.125:0/3685615234 wait complete. 2023-10-30T13:22:09.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.309+0000 7f81108ad640 1 Processor -- start 2023-10-30T13:22:09.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f81108ad640 1 -- start start 2023-10-30T13:22:09.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f81108ad640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c0cefa0 0x7f810c1eef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:09.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f81108ad640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810c1538f0 0x7f810c152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:09.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f81108ad640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f810c15cd50 0x7f810c15a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:09.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f81108ad640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c153170 con 0x7f810c1538f0 2023-10-30T13:22:09.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f81108ad640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f810c1527a0 con 0x7f810c0cefa0 2023-10-30T13:22:09.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f81108ad640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f810c1594e0 con 0x7f810c15cd50 2023-10-30T13:22:09.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f810affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810c1538f0 0x7f810c152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:09.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f810affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810c1538f0 0x7f810c152c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:39194/0 (socket says 172.21.15.125:39194) 2023-10-30T13:22:09.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f810affd640 1 -- 172.21.15.125:0/132788027 learned_addr learned my addr 172.21.15.125:0/132788027 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:09.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f810bfff640 1 --2- 172.21.15.125:0/132788027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f810c15cd50 0x7f810c15a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:09.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f810b7fe640 1 --2- 172.21.15.125:0/132788027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c0cefa0 0x7f810c1eef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:09.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f810affd640 1 -- 172.21.15.125:0/132788027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f810c15cd50 msgr2=0x7f810c15a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:09.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f810affd640 1 --2- 172.21.15.125:0/132788027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f810c15cd50 0x7f810c15a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:09.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f810affd640 1 -- 172.21.15.125:0/132788027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c0cefa0 msgr2=0x7f810c1eef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:09.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f810affd640 1 --2- 172.21.15.125:0/132788027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c0cefa0 0x7f810c1eef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:09.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f810affd640 1 -- 172.21.15.125:0/132788027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8104009b30 con 0x7f810c1538f0 2023-10-30T13:22:09.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f810affd640 1 --2- 172.21.15.125:0/132788027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810c1538f0 0x7f810c152c30 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f810000d620 tx=0x7f810000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:09.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f80f3fff640 1 -- 172.21.15.125:0/132788027 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810000b620 con 0x7f810c1538f0 2023-10-30T13:22:09.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f80f3fff640 1 -- 172.21.15.125:0/132788027 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810000b780 con 0x7f810c1538f0 2023-10-30T13:22:09.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f80f3fff640 1 -- 172.21.15.125:0/132788027 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810000bb10 con 0x7f810c1538f0 2023-10-30T13:22:09.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f81108ad640 1 -- 172.21.15.125:0/132788027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f810c15ae40 con 0x7f810c1538f0 2023-10-30T13:22:09.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f81108ad640 1 -- 172.21.15.125:0/132788027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f810c144b80 con 0x7f810c1538f0 2023-10-30T13:22:09.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.313+0000 7f81108ad640 1 -- 172.21.15.125:0/132788027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f810c152950 con 0x7f810c1538f0 2023-10-30T13:22:09.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.317+0000 7f80f3fff640 1 -- 172.21.15.125:0/132788027 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f810000bc70 con 0x7f810c1538f0 2023-10-30T13:22:09.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.317+0000 7f80f3fff640 1 --2- 172.21.15.125:0/132788027 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80f80664d0 0x7f80f8068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:09.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.317+0000 7f80f3fff640 1 -- 172.21.15.125:0/132788027 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f8100084e70 con 0x7f810c1538f0 2023-10-30T13:22:09.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.317+0000 7f810b7fe640 1 --2- 172.21.15.125:0/132788027 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80f80664d0 0x7f80f8068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:09.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.317+0000 7f810b7fe640 1 --2- 172.21.15.125:0/132788027 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80f80664d0 0x7f80f8068990 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00c880 tx=0x7f80fc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:09.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.321+0000 7f80f3fff640 1 -- 172.21.15.125:0/132788027 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f810c152950 con 0x7f810c1538f0 2023-10-30T13:22:09.503 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.497+0000 7f81108ad640 1 -- 172.21.15.125:0/132788027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f810c152950 con 0x7f810c1538f0 2023-10-30T13:22:09.504 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.497+0000 7f80f3fff640 1 -- 172.21.15.125:0/132788027 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 250 v250) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f810c152950 con 0x7f810c1538f0 2023-10-30T13:22:09.504 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:22:09.504 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":250,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":250,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:22:06.404246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:09.505 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 250 2023-10-30T13:22:09.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.501+0000 7f81108ad640 1 -- 172.21.15.125:0/132788027 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80f80664d0 msgr2=0x7f80f8068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:09.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.501+0000 7f81108ad640 1 --2- 172.21.15.125:0/132788027 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80f80664d0 0x7f80f8068990 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00c880 tx=0x7f80fc008040 comp rx=0 tx=0).stop 2023-10-30T13:22:09.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.501+0000 7f81108ad640 1 -- 172.21.15.125:0/132788027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810c1538f0 msgr2=0x7f810c152c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:09.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.501+0000 7f81108ad640 1 --2- 172.21.15.125:0/132788027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810c1538f0 0x7f810c152c30 secure :-1 s=READY pgs=2071 cs=0 l=1 rev1=1 crypto rx=0x7f810000d620 tx=0x7f810000dae0 comp rx=0 tx=0).stop 2023-10-30T13:22:09.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.501+0000 7f81108ad640 1 -- 172.21.15.125:0/132788027 shutdown_connections 2023-10-30T13:22:09.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.501+0000 7f81108ad640 1 --2- 172.21.15.125:0/132788027 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f80f80664d0 0x7f80f8068990 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:09.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.501+0000 7f81108ad640 1 --2- 172.21.15.125:0/132788027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f810c15cd50 0x7f810c15a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:09.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.501+0000 7f81108ad640 1 --2- 172.21.15.125:0/132788027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f810c1538f0 0x7f810c152c30 unknown :-1 s=CLOSED pgs=2071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:09.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.501+0000 7f81108ad640 1 --2- 172.21.15.125:0/132788027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f810c0cefa0 0x7f810c1eef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:09.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.501+0000 7f81108ad640 1 -- 172.21.15.125:0/132788027 >> 172.21.15.125:0/132788027 conn(0x7f810c06a640 msgr2=0x7f810c0c40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:09.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.501+0000 7f81108ad640 1 -- 172.21.15.125:0/132788027 shutdown_connections 2023-10-30T13:22:09.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:09.501+0000 7f81108ad640 1 -- 172.21.15.125:0/132788027 wait complete. 2023-10-30T13:22:09.525 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:22:11.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:11.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f253173e640 1 Processor -- start 2023-10-30T13:22:11.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f253173e640 1 -- start start 2023-10-30T13:22:11.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f253173e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f252c15d6f0 0x7f252c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:11.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f253173e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f252c0cf160 0x7f252c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:11.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f253173e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252c0cb800 con 0x7f252c1538a0 2023-10-30T13:22:11.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f253173e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f252c0cb330 con 0x7f252c15d6f0 2023-10-30T13:22:11.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f253173e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252c0cad70 con 0x7f252c0cf160 2023-10-30T13:22:11.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f252b7fe640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f252c0cf160 0x7f252c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:11.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f2523fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f252c15d6f0 0x7f252c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:11.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f2523fff640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f252c15d6f0 0x7f252c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:44110/0 (socket says 172.21.15.125:44110) 2023-10-30T13:22:11.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f2523fff640 1 -- 172.21.15.125:0/4082065542 learned_addr learned my addr 172.21.15.125:0/4082065542 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:11.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f252b7fe640 1 --2- 172.21.15.125:0/4082065542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f252c0cf160 0x7f252c0cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:52000/0 (socket says 172.21.15.125:52000) 2023-10-30T13:22:11.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f2523fff640 1 -- 172.21.15.125:0/4082065542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f252c0cf160 msgr2=0x7f252c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:11.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f2523fff640 1 --2- 172.21.15.125:0/4082065542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f252c0cf160 0x7f252c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:11.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f2523fff640 1 -- 172.21.15.125:0/4082065542 >> v1:172.21.15.136:6789/0 conn(0x7f252c1538a0 legacy=0x7f252c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:11.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f2523fff640 1 -- 172.21.15.125:0/4082065542 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252c0cb050 con 0x7f252c15d6f0 2023-10-30T13:22:11.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.669+0000 7f252b7fe640 1 --2- 172.21.15.125:0/4082065542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f252c0cf160 0x7f252c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:22:11.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.673+0000 7f2523fff640 1 --2- 172.21.15.125:0/4082065542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f252c15d6f0 0x7f252c15a090 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2514009870 tx=0x7f251401ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfe390e3aaf0040d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:11.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.673+0000 7f252a7fc640 1 -- 172.21.15.125:0/4082065542 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25140a8a30 con 0x7f252c15d6f0 2023-10-30T13:22:11.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.673+0000 7f252a7fc640 1 -- 172.21.15.125:0/4082065542 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25140a8b90 con 0x7f252c15d6f0 2023-10-30T13:22:11.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.673+0000 7f252a7fc640 1 -- 172.21.15.125:0/4082065542 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25140b0440 con 0x7f252c15d6f0 2023-10-30T13:22:11.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.673+0000 7f253173e640 1 -- 172.21.15.125:0/4082065542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f252c15d6f0 msgr2=0x7f252c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:11.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.673+0000 7f253173e640 1 --2- 172.21.15.125:0/4082065542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f252c15d6f0 0x7f252c15a090 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f2514009870 tx=0x7f251401ed30 comp rx=0 tx=0).stop 2023-10-30T13:22:11.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.673+0000 7f253173e640 1 -- 172.21.15.125:0/4082065542 shutdown_connections 2023-10-30T13:22:11.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.673+0000 7f253173e640 1 --2- 172.21.15.125:0/4082065542 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f252c0cf160 0x7f252c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:11.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.673+0000 7f253173e640 1 --2- 172.21.15.125:0/4082065542 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f252c15d6f0 0x7f252c15a090 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:11.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.673+0000 7f253173e640 1 -- 172.21.15.125:0/4082065542 >> 172.21.15.125:0/4082065542 conn(0x7f252c06bfe0 msgr2=0x7f252c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:11.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.673+0000 7f253173e640 1 -- 172.21.15.125:0/4082065542 shutdown_connections 2023-10-30T13:22:11.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f253173e640 1 -- 172.21.15.125:0/4082065542 wait complete. 2023-10-30T13:22:11.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f253173e640 1 Processor -- start 2023-10-30T13:22:11.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f253173e640 1 -- start start 2023-10-30T13:22:11.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f253173e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f252c0cf160 0x7f252c0c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:11.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f253173e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f252c1538a0 0x7f252c1e4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:11.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f253173e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f252c15d6f0 0x7f252c1f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:11.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f253173e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1e3960 con 0x7f252c1538a0 2023-10-30T13:22:11.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f253173e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252c159120 con 0x7f252c15d6f0 2023-10-30T13:22:11.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f253173e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252c1f3360 con 0x7f252c0cf160 2023-10-30T13:22:11.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f2523fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f252c1538a0 0x7f252c1e4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:11.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f2523fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f252c1538a0 0x7f252c1e4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37020/0 (socket says 172.21.15.125:37020) 2023-10-30T13:22:11.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f2523fff640 1 -- 172.21.15.125:0/2135603308 learned_addr learned my addr 172.21.15.125:0/2135603308 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:11.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f252affd640 1 --2- 172.21.15.125:0/2135603308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f252c0cf160 0x7f252c0c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:11.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f252b7fe640 1 --2- 172.21.15.125:0/2135603308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f252c15d6f0 0x7f252c1f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:11.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f2523fff640 1 -- 172.21.15.125:0/2135603308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f252c0cf160 msgr2=0x7f252c0c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:11.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f2523fff640 1 --2- 172.21.15.125:0/2135603308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f252c0cf160 0x7f252c0c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:11.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f2523fff640 1 -- 172.21.15.125:0/2135603308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f252c15d6f0 msgr2=0x7f252c1f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:11.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f2523fff640 1 --2- 172.21.15.125:0/2135603308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f252c15d6f0 0x7f252c1f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:11.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f2523fff640 1 -- 172.21.15.125:0/2135603308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2524009a70 con 0x7f252c1538a0 2023-10-30T13:22:11.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f2523fff640 1 --2- 172.21.15.125:0/2135603308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f252c1538a0 0x7f252c1e4c70 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f2514000c10 tx=0x7f251401bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:11.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f25237fe640 1 -- 172.21.15.125:0/2135603308 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25140bf070 con 0x7f252c1538a0 2023-10-30T13:22:11.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f25237fe640 1 -- 172.21.15.125:0/2135603308 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25140b1be0 con 0x7f252c1538a0 2023-10-30T13:22:11.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f25237fe640 1 -- 172.21.15.125:0/2135603308 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25140b9410 con 0x7f252c1538a0 2023-10-30T13:22:11.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.677+0000 7f253173e640 1 -- 172.21.15.125:0/2135603308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25140094d0 con 0x7f252c1538a0 2023-10-30T13:22:11.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.681+0000 7f253173e640 1 -- 172.21.15.125:0/2135603308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252c1e3680 con 0x7f252c1538a0 2023-10-30T13:22:11.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.681+0000 7f253173e640 1 -- 172.21.15.125:0/2135603308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f252c0b2a50 con 0x7f252c1538a0 2023-10-30T13:22:11.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.681+0000 7f25237fe640 1 -- 172.21.15.125:0/2135603308 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f25140b7040 con 0x7f252c1538a0 2023-10-30T13:22:11.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.681+0000 7f25237fe640 1 --2- 172.21.15.125:0/2135603308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f250c0664a0 0x7f250c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:11.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.681+0000 7f25237fe640 1 -- 172.21.15.125:0/2135603308 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f2514121cc0 con 0x7f252c1538a0 2023-10-30T13:22:11.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.681+0000 7f252affd640 1 --2- 172.21.15.125:0/2135603308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f250c0664a0 0x7f250c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:11.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.685+0000 7f252affd640 1 --2- 172.21.15.125:0/2135603308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f250c0664a0 0x7f250c068960 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f251c001650 tx=0x7f251c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:11.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.685+0000 7f25237fe640 1 -- 172.21.15.125:0/2135603308 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2514128080 con 0x7f252c1538a0 2023-10-30T13:22:11.873 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.869+0000 7f253173e640 1 -- 172.21.15.125:0/2135603308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f252c04e170 con 0x7f252c1538a0 2023-10-30T13:22:11.874 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.869+0000 7f25237fe640 1 -- 172.21.15.125:0/2135603308 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 250 v250) v1 ==== 95+0+5758 (secure 0 0 0) 0x7f25140b5030 con 0x7f252c1538a0 2023-10-30T13:22:11.874 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:22:11.875 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":250,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":250,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:22:06.404246+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:11.875 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 250 2023-10-30T13:22:11.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.869+0000 7f253173e640 1 -- 172.21.15.125:0/2135603308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f250c0664a0 msgr2=0x7f250c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:11.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.869+0000 7f253173e640 1 --2- 172.21.15.125:0/2135603308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f250c0664a0 0x7f250c068960 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f251c001650 tx=0x7f251c009040 comp rx=0 tx=0).stop 2023-10-30T13:22:11.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.869+0000 7f253173e640 1 -- 172.21.15.125:0/2135603308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f252c1538a0 msgr2=0x7f252c1e4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:11.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.869+0000 7f253173e640 1 --2- 172.21.15.125:0/2135603308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f252c1538a0 0x7f252c1e4c70 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f2514000c10 tx=0x7f251401bad0 comp rx=0 tx=0).stop 2023-10-30T13:22:11.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.869+0000 7f253173e640 1 -- 172.21.15.125:0/2135603308 shutdown_connections 2023-10-30T13:22:11.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.869+0000 7f253173e640 1 --2- 172.21.15.125:0/2135603308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f250c0664a0 0x7f250c068960 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:11.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.869+0000 7f253173e640 1 --2- 172.21.15.125:0/2135603308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f252c15d6f0 0x7f252c1f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:11.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.869+0000 7f253173e640 1 --2- 172.21.15.125:0/2135603308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f252c1538a0 0x7f252c1e4c70 unknown :-1 s=CLOSED pgs=2072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:11.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.869+0000 7f253173e640 1 --2- 172.21.15.125:0/2135603308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f252c0cf160 0x7f252c0c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:11.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.869+0000 7f253173e640 1 -- 172.21.15.125:0/2135603308 >> 172.21.15.125:0/2135603308 conn(0x7f252c06bfe0 msgr2=0x7f252c1529c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:11.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.869+0000 7f253173e640 1 -- 172.21.15.125:0/2135603308 shutdown_connections 2023-10-30T13:22:11.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:11.869+0000 7f253173e640 1 -- 172.21.15.125:0/2135603308 wait complete. 2023-10-30T13:22:11.896 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:22:13.897 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:14.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b43f80640 1 Processor -- start 2023-10-30T13:22:14.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b43f80640 1 -- start start 2023-10-30T13:22:14.047 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b43f80640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b3c1539a0 0x7f3b3c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:14.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b43f80640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f3b3c070a00 0x7f3b3c070e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:14.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b43f80640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3b3c069e90 0x7f3b3c06a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:14.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b43f80640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b3c06fbb0 con 0x7f3b3c069e90 2023-10-30T13:22:14.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b43f80640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b3c15a8c0 con 0x7f3b3c070a00 2023-10-30T13:22:14.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b43f80640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c071380 con 0x7f3b3c1539a0 2023-10-30T13:22:14.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b4251d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3b3c069e90 0x7f3b3c06a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:14.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b4251d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3b3c069e90 0x7f3b3c06a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37024/0 (socket says 172.21.15.125:37024) 2023-10-30T13:22:14.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b4251d640 1 -- 172.21.15.125:0/3669297652 learned_addr learned my addr 172.21.15.125:0/3669297652 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:14.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b41d1c640 1 --2- 172.21.15.125:0/3669297652 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b3c1539a0 0x7f3b3c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:14.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b4151b640 1 --2- 172.21.15.125:0/3669297652 >> v2:172.21.15.136:3300/0 conn(0x7f3b3c070a00 0x7f3b3c070e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:14.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b4251d640 1 -- 172.21.15.125:0/3669297652 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b3c1539a0 msgr2=0x7f3b3c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:14.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b4251d640 1 --2- 172.21.15.125:0/3669297652 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b3c1539a0 0x7f3b3c15a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:14.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b4251d640 1 -- 172.21.15.125:0/3669297652 >> v2:172.21.15.136:3300/0 conn(0x7f3b3c070a00 msgr2=0x7f3b3c070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:14.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b4251d640 1 --2- 172.21.15.125:0/3669297652 >> v2:172.21.15.136:3300/0 conn(0x7f3b3c070a00 0x7f3b3c070e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:14.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.041+0000 7f3b4251d640 1 -- 172.21.15.125:0/3669297652 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3c153d80 con 0x7f3b3c069e90 2023-10-30T13:22:14.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.045+0000 7f3b4251d640 1 --2- 172.21.15.125:0/3669297652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3c069e90 0x7f3b3c06a300 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c06f230 tx=0x7f3b3401c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd9ee1e1bda38a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:14.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.045+0000 7f3b40d1a640 1 -- 172.21.15.125:0/3669297652 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b340a56a0 con 0x7f3b3c069e90 2023-10-30T13:22:14.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.045+0000 7f3b40d1a640 1 -- 172.21.15.125:0/3669297652 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b340a5800 con 0x7f3b3c069e90 2023-10-30T13:22:14.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.045+0000 7f3b40d1a640 1 -- 172.21.15.125:0/3669297652 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b340a5b90 con 0x7f3b3c069e90 2023-10-30T13:22:14.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.045+0000 7f3b43f80640 1 -- 172.21.15.125:0/3669297652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3c069e90 msgr2=0x7f3b3c06a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:14.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.045+0000 7f3b43f80640 1 --2- 172.21.15.125:0/3669297652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3c069e90 0x7f3b3c06a300 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c06f230 tx=0x7f3b3401c910 comp rx=0 tx=0).stop 2023-10-30T13:22:14.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.045+0000 7f3b43f80640 1 -- 172.21.15.125:0/3669297652 shutdown_connections 2023-10-30T13:22:14.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.045+0000 7f3b43f80640 1 --2- 172.21.15.125:0/3669297652 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3c069e90 0x7f3b3c06a300 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:14.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.045+0000 7f3b43f80640 1 --2- 172.21.15.125:0/3669297652 >> v2:172.21.15.136:3300/0 conn(0x7f3b3c070a00 0x7f3b3c070e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:14.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.045+0000 7f3b43f80640 1 --2- 172.21.15.125:0/3669297652 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b3c1539a0 0x7f3b3c15a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:14.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.045+0000 7f3b43f80640 1 -- 172.21.15.125:0/3669297652 >> 172.21.15.125:0/3669297652 conn(0x7f3b3c0c7910 msgr2=0x7f3b3c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:14.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.045+0000 7f3b43f80640 1 -- 172.21.15.125:0/3669297652 shutdown_connections 2023-10-30T13:22:14.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.045+0000 7f3b43f80640 1 -- 172.21.15.125:0/3669297652 wait complete. 2023-10-30T13:22:14.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b43f80640 1 Processor -- start 2023-10-30T13:22:14.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b43f80640 1 -- start start 2023-10-30T13:22:14.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b43f80640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3c069e90 0x7f3b3c148310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:14.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b43f80640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b3c070a00 0x7f3b3c15af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:14.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b43f80640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b3c1539a0 0x7f3b3c0c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:14.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b43f80640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c0c93a0 con 0x7f3b3c069e90 2023-10-30T13:22:14.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b43f80640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c15bb20 con 0x7f3b3c1539a0 2023-10-30T13:22:14.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b43f80640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b3c1f5e90 con 0x7f3b3c070a00 2023-10-30T13:22:14.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b41d1c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3c069e90 0x7f3b3c148310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:14.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b41d1c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3c069e90 0x7f3b3c148310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37032/0 (socket says 172.21.15.125:37032) 2023-10-30T13:22:14.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b41d1c640 1 -- 172.21.15.125:0/2651839682 learned_addr learned my addr 172.21.15.125:0/2651839682 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:14.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b4251d640 1 --2- 172.21.15.125:0/2651839682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b3c1539a0 0x7f3b3c0c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:14.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b4151b640 1 --2- 172.21.15.125:0/2651839682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b3c070a00 0x7f3b3c15af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:14.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b41d1c640 1 -- 172.21.15.125:0/2651839682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b3c070a00 msgr2=0x7f3b3c15af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:14.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b41d1c640 1 --2- 172.21.15.125:0/2651839682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b3c070a00 0x7f3b3c15af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:14.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b41d1c640 1 -- 172.21.15.125:0/2651839682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b3c1539a0 msgr2=0x7f3b3c0c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:14.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b41d1c640 1 --2- 172.21.15.125:0/2651839682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b3c1539a0 0x7f3b3c0c9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:14.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b41d1c640 1 -- 172.21.15.125:0/2651839682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b34009b30 con 0x7f3b3c069e90 2023-10-30T13:22:14.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b41d1c640 1 --2- 172.21.15.125:0/2651839682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3c069e90 0x7f3b3c148310 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f3b3800d800 tx=0x7f3b3800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:14.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b327fc640 1 -- 172.21.15.125:0/2651839682 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3800b660 con 0x7f3b3c069e90 2023-10-30T13:22:14.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b327fc640 1 -- 172.21.15.125:0/2651839682 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3800b7c0 con 0x7f3b3c069e90 2023-10-30T13:22:14.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b327fc640 1 -- 172.21.15.125:0/2651839682 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3800bb30 con 0x7f3b3c069e90 2023-10-30T13:22:14.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b43f80640 1 -- 172.21.15.125:0/2651839682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3c0c95d0 con 0x7f3b3c069e90 2023-10-30T13:22:14.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.049+0000 7f3b43f80640 1 -- 172.21.15.125:0/2651839682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3c1f5560 con 0x7f3b3c069e90 2023-10-30T13:22:14.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.053+0000 7f3b43f80640 1 -- 172.21.15.125:0/2651839682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3c04e170 con 0x7f3b3c069e90 2023-10-30T13:22:14.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.053+0000 7f3b327fc640 1 -- 172.21.15.125:0/2651839682 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3b3800bc90 con 0x7f3b3c069e90 2023-10-30T13:22:14.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.053+0000 7f3b327fc640 1 --2- 172.21.15.125:0/2651839682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b200664a0 0x7f3b20068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:14.062 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.053+0000 7f3b327fc640 1 -- 172.21.15.125:0/2651839682 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f3b38084cf0 con 0x7f3b3c069e90 2023-10-30T13:22:14.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.053+0000 7f3b4151b640 1 --2- 172.21.15.125:0/2651839682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b200664a0 0x7f3b20068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:14.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.053+0000 7f3b4151b640 1 --2- 172.21.15.125:0/2651839682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b200664a0 0x7f3b20068960 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c0c9a40 tx=0x7f3b2c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:14.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.057+0000 7f3b327fc640 1 -- 172.21.15.125:0/2651839682 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3b38051720 con 0x7f3b3c069e90 2023-10-30T13:22:14.245 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.241+0000 7f3b43f80640 1 -- 172.21.15.125:0/2651839682 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b3c0c72f0 con 0x7f3b3c069e90 2023-10-30T13:22:14.246 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.241+0000 7f3b327fc640 1 -- 172.21.15.125:0/2651839682 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 251 v251) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f3b38054d40 con 0x7f3b3c069e90 2023-10-30T13:22:14.246 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:22:14.247 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":251,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":251,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:22:11.924958+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:14.247 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 251 2023-10-30T13:22:14.248 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.245+0000 7f3b43f80640 1 -- 172.21.15.125:0/2651839682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b200664a0 msgr2=0x7f3b20068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:14.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.245+0000 7f3b43f80640 1 --2- 172.21.15.125:0/2651839682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b200664a0 0x7f3b20068960 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c0c9a40 tx=0x7f3b2c008040 comp rx=0 tx=0).stop 2023-10-30T13:22:14.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.245+0000 7f3b43f80640 1 -- 172.21.15.125:0/2651839682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3c069e90 msgr2=0x7f3b3c148310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:14.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.245+0000 7f3b43f80640 1 --2- 172.21.15.125:0/2651839682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3c069e90 0x7f3b3c148310 secure :-1 s=READY pgs=2074 cs=0 l=1 rev1=1 crypto rx=0x7f3b3800d800 tx=0x7f3b3800dcc0 comp rx=0 tx=0).stop 2023-10-30T13:22:14.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.245+0000 7f3b43f80640 1 -- 172.21.15.125:0/2651839682 shutdown_connections 2023-10-30T13:22:14.249 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.245+0000 7f3b43f80640 1 --2- 172.21.15.125:0/2651839682 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3b200664a0 0x7f3b20068960 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:14.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.245+0000 7f3b43f80640 1 --2- 172.21.15.125:0/2651839682 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3b3c1539a0 0x7f3b3c0c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:14.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.245+0000 7f3b43f80640 1 --2- 172.21.15.125:0/2651839682 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3b3c070a00 0x7f3b3c15af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:14.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.245+0000 7f3b43f80640 1 --2- 172.21.15.125:0/2651839682 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3b3c069e90 0x7f3b3c148310 unknown :-1 s=CLOSED pgs=2074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:14.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.245+0000 7f3b43f80640 1 -- 172.21.15.125:0/2651839682 >> 172.21.15.125:0/2651839682 conn(0x7f3b3c0c7910 msgr2=0x7f3b3c151be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:14.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.245+0000 7f3b43f80640 1 -- 172.21.15.125:0/2651839682 shutdown_connections 2023-10-30T13:22:14.251 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:14.245+0000 7f3b43f80640 1 -- 172.21.15.125:0/2651839682 wait complete. 2023-10-30T13:22:14.267 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:22:16.268 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:16.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.409+0000 7f6eeb479640 1 Processor -- start 2023-10-30T13:22:16.415 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.409+0000 7f6eeb479640 1 -- start start 2023-10-30T13:22:16.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6eeb479640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6ee40cb550 0x7f6ee40cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:16.416 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6eeb479640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee41560a0 con 0x7f6ee414fd50 2023-10-30T13:22:16.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6eeb479640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ee4156380 con 0x7f6ee4071250 2023-10-30T13:22:16.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6eeb479640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ee4156660 con 0x7f6ee40cb550 2023-10-30T13:22:16.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6ee9215640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f6ee414fd50 0x7f6ee40c65d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:39702/0 (socket says 172.21.15.125:39702) 2023-10-30T13:22:16.417 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6ee9215640 1 -- 172.21.15.125:0/309069161 learned_addr learned my addr 172.21.15.125:0/309069161 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:22:16.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6ee8a14640 1 --2- 172.21.15.125:0/309069161 >> v2:172.21.15.136:3300/0 conn(0x7f6ee40cb550 0x7f6ee40cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:16.419 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 -- 172.21.15.125:0/309069161 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553546035 0 0) 0x7f6ee41560a0 con 0x7f6ee414fd50 2023-10-30T13:22:16.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 -- 172.21.15.125:0/309069161 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ed0003450 con 0x7f6ee414fd50 2023-10-30T13:22:16.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 -- 172.21.15.125:0/309069161 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2211706512 0 0) 0x7f6ed0003450 con 0x7f6ee414fd50 2023-10-30T13:22:16.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 -- 172.21.15.125:0/309069161 >> v2:172.21.15.136:3300/0 conn(0x7f6ee40cb550 msgr2=0x7f6ee40cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:16.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 --2- 172.21.15.125:0/309069161 >> v2:172.21.15.136:3300/0 conn(0x7f6ee40cb550 0x7f6ee40cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:16.420 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 -- 172.21.15.125:0/309069161 >> v1:172.21.15.136:6789/0 conn(0x7f6ee4071250 legacy=0x7f6ee4071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:16.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 -- 172.21.15.125:0/309069161 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee4156940 con 0x7f6ee414fd50 2023-10-30T13:22:16.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 -- 172.21.15.125:0/309069161 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f6ee0001680 con 0x7f6ee414fd50 2023-10-30T13:22:16.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 -- 172.21.15.125:0/309069161 >> v1:172.21.15.125:6789/0 conn(0x7f6ee414fd50 legacy=0x7f6ee40c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:16.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 --2- 172.21.15.125:0/309069161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ed0003a70 0x7f6ed0003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:16.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 --2- 172.21.15.125:0/309069161 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ed0008860 0x7f6ed000ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:16.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 --2- 172.21.15.125:0/309069161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ed000b530 0x7f6ed000d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:16.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 -- 172.21.15.125:0/309069161 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee4156660 con 0x7f6ed0003a70 2023-10-30T13:22:16.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 -- 172.21.15.125:0/309069161 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed000e2d0 con 0x7f6ed000b530 2023-10-30T13:22:16.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6edbfff640 1 -- 172.21.15.125:0/309069161 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed000e490 con 0x7f6ed0008860 2023-10-30T13:22:16.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6ee9215640 1 --2- 172.21.15.125:0/309069161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ed0003a70 0x7f6ed0003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:16.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6ee8a14640 1 --2- 172.21.15.125:0/309069161 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ed0008860 0x7f6ed000ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:16.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6ee9a16640 1 --2- 172.21.15.125:0/309069161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ed000b530 0x7f6ed000d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:16.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6ee9215640 1 -- 172.21.15.125:0/309069161 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ed0008860 msgr2=0x7f6ed000ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:16.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6ee9215640 1 --2- 172.21.15.125:0/309069161 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ed0008860 0x7f6ed000ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:16.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6ee9215640 1 -- 172.21.15.125:0/309069161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ed000b530 msgr2=0x7f6ed000d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:16.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6ee9215640 1 --2- 172.21.15.125:0/309069161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ed000b530 0x7f6ed000d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:16.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6ee9215640 1 -- 172.21.15.125:0/309069161 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed0003450 con 0x7f6ed0003a70 2023-10-30T13:22:16.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6ee9a16640 1 --2- 172.21.15.125:0/309069161 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ed000b530 0x7f6ed000d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:16.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.413+0000 7f6ee9a16640 1 -- 172.21.15.125:0/309069161 reap_dead start 2023-10-30T13:22:16.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.417+0000 7f6ee9215640 1 --2- 172.21.15.125:0/309069161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ed0003a70 0x7f6ed0003e90 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000c240 tx=0x7f6ee001cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eca9f5ff6208293 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:16.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.417+0000 7f6edbfff640 1 -- 172.21.15.125:0/309069161 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee0005ae0 con 0x7f6ed0003a70 2023-10-30T13:22:16.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.417+0000 7f6edbfff640 1 -- 172.21.15.125:0/309069161 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee0005c40 con 0x7f6ed0003a70 2023-10-30T13:22:16.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.417+0000 7f6edbfff640 1 -- 172.21.15.125:0/309069161 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00a5880 con 0x7f6ed0003a70 2023-10-30T13:22:16.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.417+0000 7f6eeb479640 1 -- 172.21.15.125:0/309069161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ed0003a70 msgr2=0x7f6ed0003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:16.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.417+0000 7f6eeb479640 1 --2- 172.21.15.125:0/309069161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ed0003a70 0x7f6ed0003e90 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f6ee000c240 tx=0x7f6ee001cbe0 comp rx=0 tx=0).stop 2023-10-30T13:22:16.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.417+0000 7f6eeb479640 1 -- 172.21.15.125:0/309069161 shutdown_connections 2023-10-30T13:22:16.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.417+0000 7f6eeb479640 1 --2- 172.21.15.125:0/309069161 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ed0003a70 0x7f6ed0003e90 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:16.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.417+0000 7f6eeb479640 1 -- 172.21.15.125:0/309069161 >> 172.21.15.125:0/309069161 conn(0x7f6ee406c6e0 msgr2=0x7f6ee4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:16.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.417+0000 7f6eeb479640 1 -- 172.21.15.125:0/309069161 shutdown_connections 2023-10-30T13:22:16.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.417+0000 7f6eeb479640 1 -- 172.21.15.125:0/309069161 wait complete. 2023-10-30T13:22:16.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.417+0000 7f6eeb479640 1 Processor -- start 2023-10-30T13:22:16.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.417+0000 7f6eeb479640 1 -- start start 2023-10-30T13:22:16.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6eeb479640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ed0003a70 0x7f6ee4154310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:16.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6eeb479640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ee4156f50 0x7f6ee40c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:16.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6eeb479640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ee4146450 0x7f6ee41468d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:16.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6eeb479640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee4145690 con 0x7f6ed0003a70 2023-10-30T13:22:16.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6eeb479640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee4154850 con 0x7f6ee4146450 2023-10-30T13:22:16.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6eeb479640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ee4146e10 con 0x7f6ee4156f50 2023-10-30T13:22:16.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ee9215640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ed0003a70 0x7f6ee4154310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:16.431 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ee9215640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ed0003a70 0x7f6ee4154310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37044/0 (socket says 172.21.15.125:37044) 2023-10-30T13:22:16.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ee9215640 1 -- 172.21.15.125:0/906024168 learned_addr learned my addr 172.21.15.125:0/906024168 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:16.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ee8a14640 1 --2- 172.21.15.125:0/906024168 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ee4156f50 0x7f6ee40c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:16.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ee9a16640 1 --2- 172.21.15.125:0/906024168 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ee4146450 0x7f6ee41468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:16.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ee9215640 1 -- 172.21.15.125:0/906024168 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ee4156f50 msgr2=0x7f6ee40c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:16.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ee9215640 1 --2- 172.21.15.125:0/906024168 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ee4156f50 0x7f6ee40c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:16.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ee9215640 1 -- 172.21.15.125:0/906024168 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ee4146450 msgr2=0x7f6ee41468d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:16.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ee9215640 1 --2- 172.21.15.125:0/906024168 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ee4146450 0x7f6ee41468d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:16.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ee9215640 1 -- 172.21.15.125:0/906024168 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ee41554d0 con 0x7f6ed0003a70 2023-10-30T13:22:16.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ee9215640 1 --2- 172.21.15.125:0/906024168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ed0003a70 0x7f6ee4154310 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f6ee001c650 tx=0x7f6ed0003700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:16.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ed9ffb640 1 -- 172.21.15.125:0/906024168 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00b4070 con 0x7f6ed0003a70 2023-10-30T13:22:16.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ed9ffb640 1 -- 172.21.15.125:0/906024168 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee00015c0 con 0x7f6ed0003a70 2023-10-30T13:22:16.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ed9ffb640 1 -- 172.21.15.125:0/906024168 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ee00a5a80 con 0x7f6ed0003a70 2023-10-30T13:22:16.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6eeb479640 1 -- 172.21.15.125:0/906024168 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ee41557b0 con 0x7f6ed0003a70 2023-10-30T13:22:16.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6eeb479640 1 -- 172.21.15.125:0/906024168 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ee4145c50 con 0x7f6ed0003a70 2023-10-30T13:22:16.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6eeb479640 1 -- 172.21.15.125:0/906024168 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ee404e170 con 0x7f6ed0003a70 2023-10-30T13:22:16.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.421+0000 7f6ed9ffb640 1 -- 172.21.15.125:0/906024168 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6ee00a5be0 con 0x7f6ed0003a70 2023-10-30T13:22:16.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.425+0000 7f6ed9ffb640 1 --2- 172.21.15.125:0/906024168 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ec80664a0 0x7f6ec8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:16.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.425+0000 7f6ed9ffb640 1 -- 172.21.15.125:0/906024168 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f6ee011e7f0 con 0x7f6ed0003a70 2023-10-30T13:22:16.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.425+0000 7f6ee8a14640 1 --2- 172.21.15.125:0/906024168 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ec80664a0 0x7f6ec8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:16.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.425+0000 7f6ee8a14640 1 --2- 172.21.15.125:0/906024168 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ec80664a0 0x7f6ec8068960 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f6ee41efe50 tx=0x7f6ed4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:16.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.429+0000 7f6ed9ffb640 1 -- 172.21.15.125:0/906024168 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6ee00eb190 con 0x7f6ed0003a70 2023-10-30T13:22:16.613 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.609+0000 7f6eeb479640 1 -- 172.21.15.125:0/906024168 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ee40c7180 con 0x7f6ed0003a70 2023-10-30T13:22:16.614 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.609+0000 7f6ed9ffb640 1 -- 172.21.15.125:0/906024168 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 251 v251) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f6ee00ee7b0 con 0x7f6ed0003a70 2023-10-30T13:22:16.615 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:22:16.615 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":251,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":251,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:22:11.924958+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:16.616 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 251 2023-10-30T13:22:16.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.609+0000 7f6eeb479640 1 -- 172.21.15.125:0/906024168 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ec80664a0 msgr2=0x7f6ec8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:16.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.609+0000 7f6eeb479640 1 --2- 172.21.15.125:0/906024168 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ec80664a0 0x7f6ec8068960 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f6ee41efe50 tx=0x7f6ed4008040 comp rx=0 tx=0).stop 2023-10-30T13:22:16.616 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.609+0000 7f6eeb479640 1 -- 172.21.15.125:0/906024168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ed0003a70 msgr2=0x7f6ee4154310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:16.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.609+0000 7f6eeb479640 1 --2- 172.21.15.125:0/906024168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ed0003a70 0x7f6ee4154310 secure :-1 s=READY pgs=2077 cs=0 l=1 rev1=1 crypto rx=0x7f6ee001c650 tx=0x7f6ed0003700 comp rx=0 tx=0).stop 2023-10-30T13:22:16.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.609+0000 7f6eeb479640 1 -- 172.21.15.125:0/906024168 shutdown_connections 2023-10-30T13:22:16.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.609+0000 7f6eeb479640 1 --2- 172.21.15.125:0/906024168 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6ec80664a0 0x7f6ec8068960 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:16.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.609+0000 7f6eeb479640 1 --2- 172.21.15.125:0/906024168 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6ee4146450 0x7f6ee41468d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:16.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.609+0000 7f6eeb479640 1 --2- 172.21.15.125:0/906024168 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6ee4156f50 0x7f6ee40c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:16.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.609+0000 7f6eeb479640 1 --2- 172.21.15.125:0/906024168 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6ed0003a70 0x7f6ee4154310 unknown :-1 s=CLOSED pgs=2077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:16.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.609+0000 7f6eeb479640 1 -- 172.21.15.125:0/906024168 >> 172.21.15.125:0/906024168 conn(0x7f6ee406c6e0 msgr2=0x7f6ee40c52d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:16.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.609+0000 7f6eeb479640 1 -- 172.21.15.125:0/906024168 shutdown_connections 2023-10-30T13:22:16.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:16.609+0000 7f6eeb479640 1 -- 172.21.15.125:0/906024168 wait complete. 2023-10-30T13:22:16.635 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 251, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 251, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_8033": { "addr": "172.21.15.136:6835/3480968625", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6834", "nonce": 3480968625, "type": "v2" }, { "addr": "172.21.15.136:6835", "nonce": 3480968625, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 8033, "incarnation": 248, "join_fscid": -1, "name": "d", "rank": 0, "state": "up:rejoin", "state_seq": 17 }, "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 2 ], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 36 }, "gid_9600": { "addr": "172.21.15.125:6836/2077416857", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2077416857, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2077416857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9600, "incarnation": 241, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 42 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:22:11.924958+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8033, "mds_1": 9150, "mds_2": 9600 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:22:17.949 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:22:17.953 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:22:18.635 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.777+0000 7f26e5b65640 1 Processor -- start 2023-10-30T13:22:18.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.777+0000 7f26e5b65640 1 -- start start 2023-10-30T13:22:18.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26e5b65640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f26e0159e60 0x7f26e006f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:18.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26e5b65640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f26e00bfec0 0x7f26e00c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:18.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26e5b65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26e006a720 0x7f26e006abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:18.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26e5b65640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e006fa80 con 0x7f26e00bfec0 2023-10-30T13:22:18.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26e5b65640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e0071290 con 0x7f26e0159e60 2023-10-30T13:22:18.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26e5b65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e0071570 con 0x7f26e006a720 2023-10-30T13:22:18.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26d7fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f26e00bfec0 0x7f26e00c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:18.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26d7fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f26e00bfec0 0x7f26e00c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37056/0 (socket says 172.21.15.125:37056) 2023-10-30T13:22:18.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26d7fff640 1 -- 172.21.15.125:0/485179165 learned_addr learned my addr 172.21.15.125:0/485179165 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:18.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26dffff640 1 --2- 172.21.15.125:0/485179165 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26e006a720 0x7f26e006abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:18.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26df7fe640 1 --2- 172.21.15.125:0/485179165 >> v2:172.21.15.136:3300/0 conn(0x7f26e0159e60 0x7f26e006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:18.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26d7fff640 1 -- 172.21.15.125:0/485179165 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26e006a720 msgr2=0x7f26e006abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:18.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26d7fff640 1 --2- 172.21.15.125:0/485179165 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26e006a720 0x7f26e006abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:18.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26d7fff640 1 -- 172.21.15.125:0/485179165 >> v2:172.21.15.136:3300/0 conn(0x7f26e0159e60 msgr2=0x7f26e006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:18.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26d7fff640 1 --2- 172.21.15.125:0/485179165 >> v2:172.21.15.136:3300/0 conn(0x7f26e0159e60 0x7f26e006f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:18.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26d7fff640 1 -- 172.21.15.125:0/485179165 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e0071850 con 0x7f26e00bfec0 2023-10-30T13:22:18.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.781+0000 7f26d7fff640 1 --2- 172.21.15.125:0/485179165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26e00bfec0 0x7f26e00c1a40 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f26e00c8950 tx=0x7f26c80211a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=699ded1e57fe9940 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:18.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26deffd640 1 -- 172.21.15.125:0/485179165 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8004030 con 0x7f26e00bfec0 2023-10-30T13:22:18.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26deffd640 1 -- 172.21.15.125:0/485179165 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c80ad030 con 0x7f26e00bfec0 2023-10-30T13:22:18.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26deffd640 1 -- 172.21.15.125:0/485179165 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8004410 con 0x7f26e00bfec0 2023-10-30T13:22:18.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 -- 172.21.15.125:0/485179165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26e00bfec0 msgr2=0x7f26e00c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:18.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 --2- 172.21.15.125:0/485179165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26e00bfec0 0x7f26e00c1a40 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f26e00c8950 tx=0x7f26c80211a0 comp rx=0 tx=0).stop 2023-10-30T13:22:18.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 -- 172.21.15.125:0/485179165 shutdown_connections 2023-10-30T13:22:18.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 --2- 172.21.15.125:0/485179165 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26e006a720 0x7f26e006abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:18.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 --2- 172.21.15.125:0/485179165 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26e00bfec0 0x7f26e00c1a40 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:18.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 --2- 172.21.15.125:0/485179165 >> v2:172.21.15.136:3300/0 conn(0x7f26e0159e60 0x7f26e006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:18.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 -- 172.21.15.125:0/485179165 >> 172.21.15.125:0/485179165 conn(0x7f26e00c7910 msgr2=0x7f26e00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:18.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 -- 172.21.15.125:0/485179165 shutdown_connections 2023-10-30T13:22:18.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 -- 172.21.15.125:0/485179165 wait complete. 2023-10-30T13:22:18.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 Processor -- start 2023-10-30T13:22:18.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 -- start start 2023-10-30T13:22:18.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26e006a720 0x7f26e01ef2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:18.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26e00bfec0 0x7f26e00ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:18.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26e0159e60 0x7f26e0157940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:18.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e0158930 con 0x7f26e00bfec0 2023-10-30T13:22:18.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e0157440 con 0x7f26e006a720 2023-10-30T13:22:18.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26e5b65640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e00c48c0 con 0x7f26e0159e60 2023-10-30T13:22:18.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26d7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26e00bfec0 0x7f26e00ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:18.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26d7fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26e00bfec0 0x7f26e00ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37072/0 (socket says 172.21.15.125:37072) 2023-10-30T13:22:18.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26d7fff640 1 -- 172.21.15.125:0/3680073320 learned_addr learned my addr 172.21.15.125:0/3680073320 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:18.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26df7fe640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26e006a720 0x7f26e01ef2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:18.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.785+0000 7f26dffff640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26e0159e60 0x7f26e0157940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:18.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26d7fff640 1 -- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26e0159e60 msgr2=0x7f26e0157940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:18.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26d7fff640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26e0159e60 0x7f26e0157940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:18.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26d7fff640 1 -- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26e006a720 msgr2=0x7f26e01ef2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:18.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26d7fff640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26e006a720 0x7f26e01ef2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:18.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26d7fff640 1 -- 172.21.15.125:0/3680073320 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26c800b780 con 0x7f26e00bfec0 2023-10-30T13:22:18.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26dffff640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26e0159e60 0x7f26e0157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:18.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26df7fe640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26e006a720 0x7f26e01ef2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:18.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26d7fff640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26e00bfec0 0x7f26e00ca0d0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f26c80b5040 tx=0x7f26c801df40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:18.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26dcff9640 1 -- 172.21.15.125:0/3680073320 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8003820 con 0x7f26e00bfec0 2023-10-30T13:22:18.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26dcff9640 1 -- 172.21.15.125:0/3680073320 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26c80039a0 con 0x7f26e00bfec0 2023-10-30T13:22:18.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26e5b65640 1 -- 172.21.15.125:0/3680073320 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e0158b80 con 0x7f26e00bfec0 2023-10-30T13:22:18.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26dcff9640 1 -- 172.21.15.125:0/3680073320 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26c8003d50 con 0x7f26e00bfec0 2023-10-30T13:22:18.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26e5b65640 1 -- 172.21.15.125:0/3680073320 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e0159140 con 0x7f26e00bfec0 2023-10-30T13:22:18.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26e5b65640 1 -- 172.21.15.125:0/3680073320 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e004e170 con 0x7f26e00bfec0 2023-10-30T13:22:18.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26dcff9640 1 -- 172.21.15.125:0/3680073320 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f26c80be470 con 0x7f26e00bfec0 2023-10-30T13:22:18.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26dcff9640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26c00664a0 0x7f26c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:18.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26dcff9640 1 -- 172.21.15.125:0/3680073320 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f26c81227e0 con 0x7f26e00bfec0 2023-10-30T13:22:18.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.789+0000 7f26df7fe640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26c00664a0 0x7f26c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:18.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.793+0000 7f26df7fe640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26c00664a0 0x7f26c0068960 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f26d000b9f0 tx=0x7f26d0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:18.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.793+0000 7f26dcff9640 1 -- 172.21.15.125:0/3680073320 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f26c80ef390 con 0x7f26e00bfec0 2023-10-30T13:22:18.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.977+0000 7f26e5b65640 1 -- 172.21.15.125:0/3680073320 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26e0157660 con 0x7f26e00bfec0 2023-10-30T13:22:18.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.977+0000 7f26dcff9640 1 -- 172.21.15.125:0/3680073320 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f26c80f29b0 con 0x7f26e00bfec0 2023-10-30T13:22:18.982 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:22:18.983 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":252,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:22:17.050102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:active","state_seq":19,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:18.983 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 252 2023-10-30T13:22:18.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.977+0000 7f26e5b65640 1 -- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26c00664a0 msgr2=0x7f26c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:18.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.977+0000 7f26e5b65640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26c00664a0 0x7f26c0068960 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f26d000b9f0 tx=0x7f26d0008040 comp rx=0 tx=0).stop 2023-10-30T13:22:18.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.977+0000 7f26e5b65640 1 -- 172.21.15.125:0/3680073320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26e00bfec0 msgr2=0x7f26e00ca0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:18.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.977+0000 7f26e5b65640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26e00bfec0 0x7f26e00ca0d0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f26c80b5040 tx=0x7f26c801df40 comp rx=0 tx=0).stop 2023-10-30T13:22:18.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.977+0000 7f26e5b65640 1 -- 172.21.15.125:0/3680073320 shutdown_connections 2023-10-30T13:22:18.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.977+0000 7f26e5b65640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f26c00664a0 0x7f26c0068960 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:18.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.977+0000 7f26e5b65640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f26e0159e60 0x7f26e0157940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:18.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.977+0000 7f26e5b65640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f26e00bfec0 0x7f26e00ca0d0 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:18.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.977+0000 7f26e5b65640 1 --2- 172.21.15.125:0/3680073320 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f26e006a720 0x7f26e01ef2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:18.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.977+0000 7f26e5b65640 1 -- 172.21.15.125:0/3680073320 >> 172.21.15.125:0/3680073320 conn(0x7f26e00c7910 msgr2=0x7f26e00c95f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:18.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.981+0000 7f26e5b65640 1 -- 172.21.15.125:0/3680073320 shutdown_connections 2023-10-30T13:22:18.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:18.981+0000 7f26e5b65640 1 -- 172.21.15.125:0/3680073320 wait complete. 2023-10-30T13:22:19.003 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d has gained rank=0, replacing gid=8971 2023-10-30T13:22:19.004 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.f 2023-10-30T13:22:19.004 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.f 2023-10-30T13:22:19.004 INFO:tasks.ceph.mds.f:Restarting daemon 2023-10-30T13:22:19.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2023-10-30T13:22:19.007 INFO:tasks.ceph.mds.f:Started 2023-10-30T13:22:19.007 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:19.085 INFO:tasks.ceph.mds.f.smithi136.stdout:starting mds.f at 2023-10-30T13:22:19.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.141+0000 7f60d9e9a640 1 Processor -- start 2023-10-30T13:22:19.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.141+0000 7f60d9e9a640 1 -- start start 2023-10-30T13:22:19.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.145+0000 7f60d9e9a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f60d4153bf0 0x7f60d4071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:19.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.145+0000 7f60d9e9a640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f60d40c6fe0 0x7f60d40c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:19.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.145+0000 7f60d9e9a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60d40700b0 con 0x7f60d40c79e0 2023-10-30T13:22:19.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.145+0000 7f60d9e9a640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d40c6810 con 0x7f60d4153bf0 2023-10-30T13:22:19.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.145+0000 7f60d9e9a640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60d4159900 con 0x7f60d40c6fe0 2023-10-30T13:22:19.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.145+0000 7f60d37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f60d4153bf0 0x7f60d4071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:19.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.145+0000 7f60d37fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f60d4153bf0 0x7f60d4071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37086/0 (socket says 172.21.15.125:37086) 2023-10-30T13:22:19.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.145+0000 7f60d37fe640 1 -- 172.21.15.125:0/3635273339 learned_addr learned my addr 172.21.15.125:0/3635273339 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:19.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.145+0000 7f60d2ffd640 1 --2- 172.21.15.125:0/3635273339 >> v2:172.21.15.136:3300/0 conn(0x7f60d40c6fe0 0x7f60d40c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:19.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.145+0000 7f60d37fe640 1 -- 172.21.15.125:0/3635273339 >> v2:172.21.15.136:3300/0 conn(0x7f60d40c6fe0 msgr2=0x7f60d40c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:19.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.145+0000 7f60d37fe640 1 --2- 172.21.15.125:0/3635273339 >> v2:172.21.15.136:3300/0 conn(0x7f60d40c6fe0 0x7f60d40c74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:19.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.145+0000 7f60d37fe640 1 -- 172.21.15.125:0/3635273339 >> v1:172.21.15.136:6789/0 conn(0x7f60d40c79e0 legacy=0x7f60d40cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:19.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.145+0000 7f60d37fe640 1 -- 172.21.15.125:0/3635273339 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d40c6530 con 0x7f60d4153bf0 2023-10-30T13:22:19.152 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d37fe640 1 --2- 172.21.15.125:0/3635273339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60d4153bf0 0x7f60d4071260 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f60d40723d0 tx=0x7f60c401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37cd9885d67b6546 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:19.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d27fc640 1 -- 172.21.15.125:0/3635273339 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c40a89f0 con 0x7f60d4153bf0 2023-10-30T13:22:19.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d27fc640 1 -- 172.21.15.125:0/3635273339 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c40a8b50 con 0x7f60d4153bf0 2023-10-30T13:22:19.153 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d27fc640 1 -- 172.21.15.125:0/3635273339 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c40b0440 con 0x7f60d4153bf0 2023-10-30T13:22:19.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 -- 172.21.15.125:0/3635273339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60d4153bf0 msgr2=0x7f60d4071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:19.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 --2- 172.21.15.125:0/3635273339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60d4153bf0 0x7f60d4071260 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7f60d40723d0 tx=0x7f60c401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:22:19.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 -- 172.21.15.125:0/3635273339 shutdown_connections 2023-10-30T13:22:19.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 --2- 172.21.15.125:0/3635273339 >> v2:172.21.15.136:3300/0 conn(0x7f60d40c6fe0 0x7f60d40c74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:19.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 --2- 172.21.15.125:0/3635273339 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60d4153bf0 0x7f60d4071260 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:19.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 -- 172.21.15.125:0/3635273339 >> 172.21.15.125:0/3635273339 conn(0x7f60d406c6e0 msgr2=0x7f60d4069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:19.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 -- 172.21.15.125:0/3635273339 shutdown_connections 2023-10-30T13:22:19.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 -- 172.21.15.125:0/3635273339 wait complete. 2023-10-30T13:22:19.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 Processor -- start 2023-10-30T13:22:19.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 -- start start 2023-10-30T13:22:19.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60d40c6fe0 0x7f60d41499c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:19.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60d40c79e0 0x7f60d414c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:19.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60d4153bf0 0x7f60d414c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:19.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d4156370 con 0x7f60d40c6fe0 2023-10-30T13:22:19.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d414b4d0 con 0x7f60d40c79e0 2023-10-30T13:22:19.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.149+0000 7f60d9e9a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d4149f00 con 0x7f60d4153bf0 2023-10-30T13:22:19.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60d40c6fe0 0x7f60d41499c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:19.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d37fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60d40c6fe0 0x7f60d41499c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:37094/0 (socket says 172.21.15.125:37094) 2023-10-30T13:22:19.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d37fe640 1 -- 172.21.15.125:0/677087181 learned_addr learned my addr 172.21.15.125:0/677087181 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:19.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d2ffd640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60d40c79e0 0x7f60d414c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:19.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d3fff640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60d4153bf0 0x7f60d414c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:19.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d37fe640 1 -- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60d4153bf0 msgr2=0x7f60d414c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:19.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d37fe640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60d4153bf0 0x7f60d414c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:19.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d37fe640 1 -- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60d40c79e0 msgr2=0x7f60d414c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:19.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d37fe640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60d40c79e0 0x7f60d414c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:19.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d37fe640 1 -- 172.21.15.125:0/677087181 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c40094d0 con 0x7f60d40c6fe0 2023-10-30T13:22:19.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d2ffd640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60d40c79e0 0x7f60d414c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:22:19.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d37fe640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60d40c6fe0 0x7f60d41499c0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f60c40097c0 tx=0x7f60c401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:19.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60bbfff640 1 -- 172.21.15.125:0/677087181 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c40a89f0 con 0x7f60d40c6fe0 2023-10-30T13:22:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60bbfff640 1 -- 172.21.15.125:0/677087181 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c40b9410 con 0x7f60d40c6fe0 2023-10-30T13:22:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60bbfff640 1 -- 172.21.15.125:0/677087181 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c40b97a0 con 0x7f60d40c6fe0 2023-10-30T13:22:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d9e9a640 1 -- 172.21.15.125:0/677087181 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d41efb70 con 0x7f60d40c6fe0 2023-10-30T13:22:19.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d9e9a640 1 -- 172.21.15.125:0/677087181 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d414b720 con 0x7f60d40c6fe0 2023-10-30T13:22:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60d9e9a640 1 -- 172.21.15.125:0/677087181 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d404e170 con 0x7f60d40c6fe0 2023-10-30T13:22:19.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.153+0000 7f60bbfff640 1 -- 172.21.15.125:0/677087181 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f60c40b7040 con 0x7f60d40c6fe0 2023-10-30T13:22:19.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.157+0000 7f60bbfff640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60c00664a0 0x7f60c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:19.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.157+0000 7f60bbfff640 1 -- 172.21.15.125:0/677087181 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f60c41223e0 con 0x7f60d40c6fe0 2023-10-30T13:22:19.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.157+0000 7f60d2ffd640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60c00664a0 0x7f60c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:19.164 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.157+0000 7f60d2ffd640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60c00664a0 0x7f60c0068960 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f60d414d140 tx=0x7f60c8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:19.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.157+0000 7f60bbfff640 1 -- 172.21.15.125:0/677087181 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f60c40eede0 con 0x7f60d40c6fe0 2023-10-30T13:22:19.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.341+0000 7f60d9e9a640 1 -- 172.21.15.125:0/677087181 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60d406fe90 con 0x7f60d40c6fe0 2023-10-30T13:22:19.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.341+0000 7f60bbfff640 1 -- 172.21.15.125:0/677087181 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 252 v252) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f60c40b5020 con 0x7f60d40c6fe0 2023-10-30T13:22:19.348 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:22:19.349 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":252,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:22:17.050102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:active","state_seq":19,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:19.349 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 252 2023-10-30T13:22:19.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.345+0000 7f60d9e9a640 1 -- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60c00664a0 msgr2=0x7f60c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:19.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.345+0000 7f60d9e9a640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60c00664a0 0x7f60c0068960 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f60d414d140 tx=0x7f60c8008040 comp rx=0 tx=0).stop 2023-10-30T13:22:19.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.345+0000 7f60d9e9a640 1 -- 172.21.15.125:0/677087181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60d40c6fe0 msgr2=0x7f60d41499c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:19.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.345+0000 7f60d9e9a640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60d40c6fe0 0x7f60d41499c0 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f60c40097c0 tx=0x7f60c401ba90 comp rx=0 tx=0).stop 2023-10-30T13:22:19.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.345+0000 7f60d9e9a640 1 -- 172.21.15.125:0/677087181 shutdown_connections 2023-10-30T13:22:19.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.345+0000 7f60d9e9a640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f60c00664a0 0x7f60c0068960 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:19.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.345+0000 7f60d9e9a640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f60d4153bf0 0x7f60d414c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:19.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.345+0000 7f60d9e9a640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f60d40c79e0 0x7f60d414c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:19.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.345+0000 7f60d9e9a640 1 --2- 172.21.15.125:0/677087181 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f60d40c6fe0 0x7f60d41499c0 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:19.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.345+0000 7f60d9e9a640 1 -- 172.21.15.125:0/677087181 >> 172.21.15.125:0/677087181 conn(0x7f60d406c6e0 msgr2=0x7f60d4151a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:19.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.345+0000 7f60d9e9a640 1 -- 172.21.15.125:0/677087181 shutdown_connections 2023-10-30T13:22:19.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:19.345+0000 7f60d9e9a640 1 -- 172.21.15.125:0/677087181 wait complete. 2023-10-30T13:22:19.369 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.f is in active, standby or standby-replay 2023-10-30T13:22:21.370 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:22:21.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f3160d80640 1 Processor -- start 2023-10-30T13:22:21.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f3160d80640 1 -- start start 2023-10-30T13:22:21.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f3160d80640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f315c15d6f0 0x7f315c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:21.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f3160d80640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f315c0cf160 0x7f315c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:21.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f3160d80640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f315c0cb800 con 0x7f315c1538a0 2023-10-30T13:22:21.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f3160d80640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f315c0cb330 con 0x7f315c15d6f0 2023-10-30T13:22:21.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f3160d80640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315c0cad70 con 0x7f315c0cf160 2023-10-30T13:22:21.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f3151d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f315c15d6f0 0x7f315c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:21.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f3151d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f315c15d6f0 0x7f315c15a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38482/0 (socket says 172.21.15.125:38482) 2023-10-30T13:22:21.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f3151d9b640 1 -- 172.21.15.125:0/65453783 learned_addr learned my addr 172.21.15.125:0/65453783 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:21.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f315ad9d640 1 --2- 172.21.15.125:0/65453783 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f315c0cf160 0x7f315c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:21.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f3151d9b640 1 -- 172.21.15.125:0/65453783 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f315c0cf160 msgr2=0x7f315c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:21.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f3151d9b640 1 --2- 172.21.15.125:0/65453783 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f315c0cf160 0x7f315c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:21.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f3151d9b640 1 -- 172.21.15.125:0/65453783 >> v1:172.21.15.136:6789/0 conn(0x7f315c1538a0 legacy=0x7f315c0ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:22:21.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.513+0000 7f3151d9b640 1 -- 172.21.15.125:0/65453783 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c0cb050 con 0x7f315c15d6f0 2023-10-30T13:22:21.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.517+0000 7f3151d9b640 1 --2- 172.21.15.125:0/65453783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f315c15d6f0 0x7f315c15a090 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f314c009870 tx=0x7f314c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82690a2af2930118 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:21.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.517+0000 7f3159d9b640 1 -- 172.21.15.125:0/65453783 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c0a8a30 con 0x7f315c15d6f0 2023-10-30T13:22:21.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.517+0000 7f3159d9b640 1 -- 172.21.15.125:0/65453783 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314c0a8b90 con 0x7f315c15d6f0 2023-10-30T13:22:21.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.517+0000 7f3159d9b640 1 -- 172.21.15.125:0/65453783 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314c0b0440 con 0x7f315c15d6f0 2023-10-30T13:22:21.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.517+0000 7f3160d80640 1 -- 172.21.15.125:0/65453783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f315c15d6f0 msgr2=0x7f315c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:21.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.517+0000 7f3160d80640 1 --2- 172.21.15.125:0/65453783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f315c15d6f0 0x7f315c15a090 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f314c009870 tx=0x7f314c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:22:21.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.517+0000 7f3160d80640 1 -- 172.21.15.125:0/65453783 shutdown_connections 2023-10-30T13:22:21.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.517+0000 7f3160d80640 1 --2- 172.21.15.125:0/65453783 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f315c0cf160 0x7f315c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:21.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.517+0000 7f3160d80640 1 --2- 172.21.15.125:0/65453783 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f315c15d6f0 0x7f315c15a090 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:21.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.517+0000 7f3160d80640 1 -- 172.21.15.125:0/65453783 >> 172.21.15.125:0/65453783 conn(0x7f315c06bfe0 msgr2=0x7f315c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:21.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.517+0000 7f3160d80640 1 -- 172.21.15.125:0/65453783 shutdown_connections 2023-10-30T13:22:21.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.517+0000 7f3160d80640 1 -- 172.21.15.125:0/65453783 wait complete. 2023-10-30T13:22:21.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f3160d80640 1 Processor -- start 2023-10-30T13:22:21.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f3160d80640 1 -- start start 2023-10-30T13:22:21.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f3160d80640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f315c0cf160 0x7f315c159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:21.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f3160d80640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f315c1538a0 0x7f315c0c52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:21.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f3160d80640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f315c15d6f0 0x7f315c1e2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:21.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f3160d80640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315c1e3bf0 con 0x7f315c15d6f0 2023-10-30T13:22:21.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f3160d80640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f315c1580f0 con 0x7f315c1538a0 2023-10-30T13:22:21.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f3160d80640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f315c0c59a0 con 0x7f315c0cf160 2023-10-30T13:22:21.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f315ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f315c15d6f0 0x7f315c1e2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:21.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f315ad9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f315c15d6f0 0x7f315c1e2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38490/0 (socket says 172.21.15.125:38490) 2023-10-30T13:22:21.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f315ad9d640 1 -- 172.21.15.125:0/2178777170 learned_addr learned my addr 172.21.15.125:0/2178777170 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:22:21.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f315a59c640 1 --2- 172.21.15.125:0/2178777170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f315c0cf160 0x7f315c159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:21.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f3151d9b640 1 --2- 172.21.15.125:0/2178777170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f315c1538a0 0x7f315c0c52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:21.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f315ad9d640 1 -- 172.21.15.125:0/2178777170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f315c0cf160 msgr2=0x7f315c159300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:21.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f315ad9d640 1 --2- 172.21.15.125:0/2178777170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f315c0cf160 0x7f315c159300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:21.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f315ad9d640 1 -- 172.21.15.125:0/2178777170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f315c1538a0 msgr2=0x7f315c0c52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:21.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f315ad9d640 1 --2- 172.21.15.125:0/2178777170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f315c1538a0 0x7f315c0c52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:21.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f315ad9d640 1 -- 172.21.15.125:0/2178777170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c0094d0 con 0x7f315c15d6f0 2023-10-30T13:22:21.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.521+0000 7f315ad9d640 1 --2- 172.21.15.125:0/2178777170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f315c15d6f0 0x7f315c1e2a40 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f3154007d90 tx=0x7f315400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:21.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.525+0000 7f31537fe640 1 -- 172.21.15.125:0/2178777170 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3154014420 con 0x7f315c15d6f0 2023-10-30T13:22:21.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.525+0000 7f31537fe640 1 -- 172.21.15.125:0/2178777170 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31540145a0 con 0x7f315c15d6f0 2023-10-30T13:22:21.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.525+0000 7f31537fe640 1 -- 172.21.15.125:0/2178777170 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3154014950 con 0x7f315c15d6f0 2023-10-30T13:22:21.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.525+0000 7f3160d80640 1 -- 172.21.15.125:0/2178777170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f315c1584d0 con 0x7f315c15d6f0 2023-10-30T13:22:21.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.525+0000 7f3160d80640 1 -- 172.21.15.125:0/2178777170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f315c0c4f60 con 0x7f315c15d6f0 2023-10-30T13:22:21.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.525+0000 7f3160d80640 1 -- 172.21.15.125:0/2178777170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f315c0b2a50 con 0x7f315c15d6f0 2023-10-30T13:22:21.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.525+0000 7f31537fe640 1 -- 172.21.15.125:0/2178777170 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3154014ab0 con 0x7f315c15d6f0 2023-10-30T13:22:21.544 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.525+0000 7f31537fe640 1 --2- 172.21.15.125:0/2178777170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f313c0664d0 0x7f313c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:22:21.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.525+0000 7f31537fe640 1 -- 172.21.15.125:0/2178777170 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f3154085190 con 0x7f315c15d6f0 2023-10-30T13:22:21.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.525+0000 7f315a59c640 1 --2- 172.21.15.125:0/2178777170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f313c0664d0 0x7f313c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:22:21.545 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.529+0000 7f315a59c640 1 --2- 172.21.15.125:0/2178777170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f313c0664d0 0x7f313c068990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f3144007fd0 tx=0x7f3144009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:22:21.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.529+0000 7f31537fe640 1 -- 172.21.15.125:0/2178777170 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3154051b50 con 0x7f315c15d6f0 2023-10-30T13:22:21.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.709+0000 7f3160d80640 1 -- 172.21.15.125:0/2178777170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f315c04e170 con 0x7f315c15d6f0 2023-10-30T13:22:21.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.709+0000 7f31537fe640 1 -- 172.21.15.125:0/2178777170 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 253 v253) v1 ==== 95+0+6550 (secure 0 0 0) 0x7f3154055170 con 0x7f315c15d6f0 2023-10-30T13:22:21.714 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:22:21.715 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":253,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9409,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":253}],"filesystems":[{"mdsmap":{"epoch":252,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:22:17.050102+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:active","state_seq":19,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:22:21.715 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 253 2023-10-30T13:22:21.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.709+0000 7f3160d80640 1 -- 172.21.15.125:0/2178777170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f313c0664d0 msgr2=0x7f313c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:21.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.709+0000 7f3160d80640 1 --2- 172.21.15.125:0/2178777170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f313c0664d0 0x7f313c068990 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f3144007fd0 tx=0x7f3144009040 comp rx=0 tx=0).stop 2023-10-30T13:22:21.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.709+0000 7f3160d80640 1 -- 172.21.15.125:0/2178777170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f315c15d6f0 msgr2=0x7f315c1e2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:22:21.716 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.709+0000 7f3160d80640 1 --2- 172.21.15.125:0/2178777170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f315c15d6f0 0x7f315c1e2a40 secure :-1 s=READY pgs=2084 cs=0 l=1 rev1=1 crypto rx=0x7f3154007d90 tx=0x7f315400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:22:21.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.709+0000 7f3160d80640 1 -- 172.21.15.125:0/2178777170 shutdown_connections 2023-10-30T13:22:21.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.709+0000 7f3160d80640 1 --2- 172.21.15.125:0/2178777170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f313c0664d0 0x7f313c068990 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:21.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.709+0000 7f3160d80640 1 --2- 172.21.15.125:0/2178777170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f315c15d6f0 0x7f315c1e2a40 unknown :-1 s=CLOSED pgs=2084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:21.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.709+0000 7f3160d80640 1 --2- 172.21.15.125:0/2178777170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f315c1538a0 0x7f315c0c52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:21.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.709+0000 7f3160d80640 1 --2- 172.21.15.125:0/2178777170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f315c0cf160 0x7f315c159300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:22:21.717 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.709+0000 7f3160d80640 1 -- 172.21.15.125:0/2178777170 >> 172.21.15.125:0/2178777170 conn(0x7f315c06bfe0 msgr2=0x7f315c14ffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:22:21.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.709+0000 7f3160d80640 1 -- 172.21.15.125:0/2178777170 shutdown_connections 2023-10-30T13:22:21.718 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:22:21.709+0000 7f3160d80640 1 -- 172.21.15.125:0/2178777170 wait complete. 2023-10-30T13:22:21.736 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f reported in up:standby state 2023-10-30T13:22:21.736 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 97 secs before thrashing 2023-10-30T13:22:23.278 INFO:tasks.ceph.mds.d.smithi136.stderr:2023-10-30T13:22:23.275+0000 7f49a63c6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:22:23.278 INFO:tasks.ceph.mds.d.smithi136.stderr:2023-10-30T13:22:23.275+0000 7f49a63c6640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:22:23.278 INFO:tasks.ceph.mds.d.smithi136.stderr:2023-10-30T13:22:23.275+0000 7f49a63c6640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:22:24.794 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:22:24.795 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 64 -eq 48 ] 2023-10-30T13:22:24.795 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.64 2023-10-30T13:22:24.795 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.64 2023-10-30T13:22:24.796 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.64 2023-10-30T13:22:24.796 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.64 2023-10-30T13:22:24.796 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:22:33.189 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:22:33.190 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.65 2023-10-30T13:22:33.190 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 65 -eq 48 ] 2023-10-30T13:22:33.191 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.65 2023-10-30T13:22:33.191 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.65 2023-10-30T13:22:33.191 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.65 2023-10-30T13:22:33.191 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:22:40.056 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:22:40.056 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.66 2023-10-30T13:22:40.057 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 66 -eq 48 ] 2023-10-30T13:22:40.057 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.66 2023-10-30T13:22:40.057 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.66 2023-10-30T13:22:40.057 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.66 2023-10-30T13:22:40.058 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:22:46.325 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:22:46.325 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 67 -eq 48 ] 2023-10-30T13:22:46.326 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.67 2023-10-30T13:22:46.326 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.67 2023-10-30T13:22:46.327 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.67 2023-10-30T13:22:46.327 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.67 2023-10-30T13:22:46.327 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:22:48.062 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:22:48.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:22:51.825 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:22:51.825 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.68 2023-10-30T13:22:51.825 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 68 -eq 48 ] 2023-10-30T13:22:51.826 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.68 2023-10-30T13:22:51.826 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.68 2023-10-30T13:22:51.826 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.68 2023-10-30T13:22:51.827 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:23:01.738 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:23:01.738 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 69 -eq 48 ] 2023-10-30T13:23:01.739 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.69 2023-10-30T13:23:01.739 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.69 2023-10-30T13:23:01.739 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.69 2023-10-30T13:23:01.740 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.69 2023-10-30T13:23:01.740 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:23:08.360 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:23:08.360 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 70 -eq 48 ] 2023-10-30T13:23:08.360 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.70 2023-10-30T13:23:08.361 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.70 2023-10-30T13:23:08.361 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.70 2023-10-30T13:23:08.362 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.70 2023-10-30T13:23:08.362 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:23:14.681 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:23:14.682 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 71 -eq 48 ] 2023-10-30T13:23:14.682 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.71 2023-10-30T13:23:14.683 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.71 2023-10-30T13:23:14.683 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.71 2023-10-30T13:23:14.683 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.71 2023-10-30T13:23:14.684 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:23:18.109 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:23:18.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:23:22.124 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:23:22.124 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 72 -eq 48 ] 2023-10-30T13:23:22.124 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.72 2023-10-30T13:23:22.125 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.72 2023-10-30T13:23:22.125 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.72 2023-10-30T13:23:22.126 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.72 2023-10-30T13:23:22.126 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:23:28.717 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:23:28.718 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 73 -eq 48 ] 2023-10-30T13:23:28.718 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.73 2023-10-30T13:23:28.718 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.73 2023-10-30T13:23:28.719 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.73 2023-10-30T13:23:28.719 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.73 2023-10-30T13:23:28.719 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:23:36.604 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:23:36.604 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 74 -eq 48 ] 2023-10-30T13:23:36.604 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.74 2023-10-30T13:23:36.605 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.74 2023-10-30T13:23:36.605 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.74 2023-10-30T13:23:36.606 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.74 2023-10-30T13:23:36.606 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:23:44.281 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:23:44.282 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.75 2023-10-30T13:23:44.282 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 75 -eq 48 ] 2023-10-30T13:23:44.283 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.75 2023-10-30T13:23:44.283 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.75 2023-10-30T13:23:44.283 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.75 2023-10-30T13:23:44.283 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:23:48.182 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:23:48.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:23:51.398 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:23:51.399 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.76 2023-10-30T13:23:51.399 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 76 -eq 48 ] 2023-10-30T13:23:51.400 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.76 2023-10-30T13:23:51.400 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.76 2023-10-30T13:23:51.400 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.76 2023-10-30T13:23:51.400 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:23:58.737 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:58.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.879+0000 7fb15d70f640 1 Processor -- start 2023-10-30T13:23:58.883 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.879+0000 7fb15d70f640 1 -- start start 2023-10-30T13:23:58.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.883+0000 7fb15d70f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1580cf160 0x7fb1580cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:58.884 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.883+0000 7fb15d70f640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1580cb800 con 0x7fb1581538a0 2023-10-30T13:23:58.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.883+0000 7fb15d70f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1580cb330 con 0x7fb15815d6f0 2023-10-30T13:23:58.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.883+0000 7fb15d70f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1580cad70 con 0x7fb1580cf160 2023-10-30T13:23:58.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.883+0000 7fb156ffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb1581538a0 0x7fb1580ca0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38394/0 (socket says 172.21.15.125:38394) 2023-10-30T13:23:58.885 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.883+0000 7fb156ffd640 1 -- 172.21.15.125:0/851628945 learned_addr learned my addr 172.21.15.125:0/851628945 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:23:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.883+0000 7fb1577fe640 1 --2- 172.21.15.125:0/851628945 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1580cf160 0x7fb1580cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:58.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.883+0000 7fb155ffb640 1 -- 172.21.15.125:0/851628945 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604614123 0 0) 0x7fb1580cb800 con 0x7fb1581538a0 2023-10-30T13:23:58.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.883+0000 7fb155ffb640 1 -- 172.21.15.125:0/851628945 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb140003530 con 0x7fb1581538a0 2023-10-30T13:23:58.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.883+0000 7fb155ffb640 1 -- 172.21.15.125:0/851628945 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3315603995 0 0) 0x7fb1580cb330 con 0x7fb15815d6f0 2023-10-30T13:23:58.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.883+0000 7fb1577fe640 1 -- 172.21.15.125:0/851628945 >> v1:172.21.15.136:6789/0 conn(0x7fb15815d6f0 legacy=0x7fb15815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:58.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.883+0000 7fb1577fe640 1 -- 172.21.15.125:0/851628945 >> v1:172.21.15.125:6789/0 conn(0x7fb1581538a0 legacy=0x7fb1580ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:58.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.883+0000 7fb1577fe640 1 -- 172.21.15.125:0/851628945 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1580cb050 con 0x7fb1580cf160 2023-10-30T13:23:58.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.887+0000 7fb1577fe640 1 --2- 172.21.15.125:0/851628945 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1580cf160 0x7fb1580cf590 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb150009dc0 tx=0x7fb15001f150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7c730f0ada2946e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:58.888 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.887+0000 7fb155ffb640 1 -- 172.21.15.125:0/851628945 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1500af600 con 0x7fb1580cf160 2023-10-30T13:23:58.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.887+0000 7fb155ffb640 1 -- 172.21.15.125:0/851628945 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1500ab050 con 0x7fb1580cf160 2023-10-30T13:23:58.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.887+0000 7fb155ffb640 1 -- 172.21.15.125:0/851628945 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1500af960 con 0x7fb1580cf160 2023-10-30T13:23:58.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.887+0000 7fb15d70f640 1 -- 172.21.15.125:0/851628945 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1580cf160 msgr2=0x7fb1580cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:58.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.887+0000 7fb15d70f640 1 --2- 172.21.15.125:0/851628945 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1580cf160 0x7fb1580cf590 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb150009dc0 tx=0x7fb15001f150 comp rx=0 tx=0).stop 2023-10-30T13:23:58.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.887+0000 7fb15d70f640 1 -- 172.21.15.125:0/851628945 shutdown_connections 2023-10-30T13:23:58.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.887+0000 7fb15d70f640 1 --2- 172.21.15.125:0/851628945 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1580cf160 0x7fb1580cf590 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:58.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.887+0000 7fb15d70f640 1 -- 172.21.15.125:0/851628945 >> 172.21.15.125:0/851628945 conn(0x7fb15806bfe0 msgr2=0x7fb15806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:58.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.887+0000 7fb15d70f640 1 -- 172.21.15.125:0/851628945 shutdown_connections 2023-10-30T13:23:58.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.887+0000 7fb15d70f640 1 -- 172.21.15.125:0/851628945 wait complete. 2023-10-30T13:23:58.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.887+0000 7fb15d70f640 1 Processor -- start 2023-10-30T13:23:58.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.887+0000 7fb15d70f640 1 -- start start 2023-10-30T13:23:58.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb15d70f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1580cf160 0x7fb15814ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:58.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb15d70f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1581538a0 0x7fb15815ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:58.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb15d70f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb15815d6f0 0x7fb1581507a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:58.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb15d70f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1581572b0 con 0x7fb15815d6f0 2023-10-30T13:23:58.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb15d70f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb15814e1b0 con 0x7fb1580cf160 2023-10-30T13:23:58.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb15d70f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1581551c0 con 0x7fb1581538a0 2023-10-30T13:23:58.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb1577fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb15815d6f0 0x7fb1581507a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:58.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb1577fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb15815d6f0 0x7fb1581507a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44084/0 (socket says 172.21.15.125:44084) 2023-10-30T13:23:58.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb1577fe640 1 -- 172.21.15.125:0/3338203307 learned_addr learned my addr 172.21.15.125:0/3338203307 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:23:58.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb156ffd640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1580cf160 0x7fb15814ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:58.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb1567fc640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1581538a0 0x7fb15815ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:58.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb1577fe640 1 -- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1581538a0 msgr2=0x7fb15815ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:58.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb1577fe640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1581538a0 0x7fb15815ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:58.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb1577fe640 1 -- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1580cf160 msgr2=0x7fb15814ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:58.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb1577fe640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1580cf160 0x7fb15814ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:58.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb1577fe640 1 -- 172.21.15.125:0/3338203307 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb150009a70 con 0x7fb15815d6f0 2023-10-30T13:23:58.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb156ffd640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1580cf160 0x7fb15814ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:58.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb1577fe640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb15815d6f0 0x7fb1581507a0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fb15001cd80 tx=0x7fb15001bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:58.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb13f7fe640 1 -- 172.21.15.125:0/3338203307 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1500bf070 con 0x7fb15815d6f0 2023-10-30T13:23:58.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb13f7fe640 1 -- 172.21.15.125:0/3338203307 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb150009450 con 0x7fb15815d6f0 2023-10-30T13:23:58.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb15d70f640 1 -- 172.21.15.125:0/3338203307 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb15814dbf0 con 0x7fb15815d6f0 2023-10-30T13:23:58.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb13f7fe640 1 -- 172.21.15.125:0/3338203307 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb150004450 con 0x7fb15815d6f0 2023-10-30T13:23:58.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb15d70f640 1 -- 172.21.15.125:0/3338203307 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15815a9e0 con 0x7fb15815d6f0 2023-10-30T13:23:58.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb15d70f640 1 -- 172.21.15.125:0/3338203307 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb15804e170 con 0x7fb15815d6f0 2023-10-30T13:23:58.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.891+0000 7fb13f7fe640 1 -- 172.21.15.125:0/3338203307 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb1500045b0 con 0x7fb15815d6f0 2023-10-30T13:23:58.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.895+0000 7fb13f7fe640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1440664d0 0x7fb144068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:58.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.895+0000 7fb13f7fe640 1 -- 172.21.15.125:0/3338203307 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7fb150122a50 con 0x7fb15815d6f0 2023-10-30T13:23:58.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.895+0000 7fb156ffd640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1440664d0 0x7fb144068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:58.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.895+0000 7fb156ffd640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1440664d0 0x7fb144068990 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fb148005890 tx=0x7fb148009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:58.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:58.899+0000 7fb13f7fe640 1 -- 172.21.15.125:0/3338203307 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb1500f2ee0 con 0x7fb15815d6f0 2023-10-30T13:23:59.080 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.079+0000 7fb15d70f640 1 -- 172.21.15.125:0/3338203307 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1580b1dd0 con 0x7fb15815d6f0 2023-10-30T13:23:59.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.079+0000 7fb13f7fe640 1 -- 172.21.15.125:0/3338203307 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+6554 (secure 0 0 0) 0x7fb1500b4070 con 0x7fb15815d6f0 2023-10-30T13:23:59.081 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:23:59.082 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9409,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":253}],"filesystems":[{"mdsmap":{"epoch":258,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:23:34.832807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:active","state_seq":19,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 258 2023-10-30T13:23:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.079+0000 7fb15d70f640 1 -- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1440664d0 msgr2=0x7fb144068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.079+0000 7fb15d70f640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1440664d0 0x7fb144068990 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fb148005890 tx=0x7fb148009040 comp rx=0 tx=0).stop 2023-10-30T13:23:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.079+0000 7fb15d70f640 1 -- 172.21.15.125:0/3338203307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb15815d6f0 msgr2=0x7fb1581507a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:59.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.079+0000 7fb15d70f640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb15815d6f0 0x7fb1581507a0 secure :-1 s=READY pgs=2086 cs=0 l=1 rev1=1 crypto rx=0x7fb15001cd80 tx=0x7fb15001bef0 comp rx=0 tx=0).stop 2023-10-30T13:23:59.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.079+0000 7fb15d70f640 1 -- 172.21.15.125:0/3338203307 shutdown_connections 2023-10-30T13:23:59.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.079+0000 7fb15d70f640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb1440664d0 0x7fb144068990 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:59.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.079+0000 7fb15d70f640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb15815d6f0 0x7fb1581507a0 unknown :-1 s=CLOSED pgs=2086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:59.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.079+0000 7fb15d70f640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb1581538a0 0x7fb15815ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:59.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.079+0000 7fb15d70f640 1 --2- 172.21.15.125:0/3338203307 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb1580cf160 0x7fb15814ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:59.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.079+0000 7fb15d70f640 1 -- 172.21.15.125:0/3338203307 >> 172.21.15.125:0/3338203307 conn(0x7fb15806bfe0 msgr2=0x7fb1580c5640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:59.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.079+0000 7fb15d70f640 1 -- 172.21.15.125:0/3338203307 shutdown_connections 2023-10-30T13:23:59.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.079+0000 7fb15d70f640 1 -- 172.21.15.125:0/3338203307 wait complete. 2023-10-30T13:23:59.102 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.d (rank=0) 2023-10-30T13:23:59.102 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:23:59.103 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:23:59.120 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:23:59.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:23:59.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.255+0000 7f31e4c09640 1 Processor -- start 2023-10-30T13:23:59.258 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.255+0000 7f31e4c09640 1 -- start start 2023-10-30T13:23:59.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.255+0000 7f31e4c09640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31e00c05d0 0x7f31e013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:59.259 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.255+0000 7f31e4c09640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e00c0a50 con 0x7f31e00c6b80 2023-10-30T13:23:59.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.255+0000 7f31e4c09640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e006f6e0 con 0x7f31e00c6f60 2023-10-30T13:23:59.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.255+0000 7f31e4c09640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31e006f9c0 con 0x7f31e00c05d0 2023-10-30T13:23:59.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.255+0000 7f31de59c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f31e00c6b80 0x7f31e00bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38410/0 (socket says 172.21.15.125:38410) 2023-10-30T13:23:59.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.255+0000 7f31d5d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f31e00c05d0 0x7f31e013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:59.260 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.255+0000 7f31de59c640 1 -- 172.21.15.125:0/1234090147 learned_addr learned my addr 172.21.15.125:0/1234090147 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:23:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.255+0000 7f31d5d9b640 1 -- 172.21.15.125:0/1234090147 >> v1:172.21.15.136:6789/0 conn(0x7f31e00c6f60 legacy=0x7f31e01566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:23:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.255+0000 7f31ddd9b640 1 -- 172.21.15.125:0/1234090147 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455032068 0 0) 0x7f31e00c0a50 con 0x7f31e00c6b80 2023-10-30T13:23:59.261 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.259+0000 7f31d5d9b640 1 -- 172.21.15.125:0/1234090147 >> v1:172.21.15.125:6789/0 conn(0x7f31e00c6b80 legacy=0x7f31e00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:59.262 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.259+0000 7f31d5d9b640 1 -- 172.21.15.125:0/1234090147 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e006fca0 con 0x7f31e00c05d0 2023-10-30T13:23:59.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.259+0000 7f31d5d9b640 1 --2- 172.21.15.125:0/1234090147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31e00c05d0 0x7f31e013ed90 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f31c80097c0 tx=0x7f31c801c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdfaac25b0910ba4 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:59.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.259+0000 7f31ddd9b640 1 -- 172.21.15.125:0/1234090147 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c80a8050 con 0x7f31e00c05d0 2023-10-30T13:23:59.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.259+0000 7f31ddd9b640 1 -- 172.21.15.125:0/1234090147 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c80ac5f0 con 0x7f31e00c05d0 2023-10-30T13:23:59.263 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.259+0000 7f31ddd9b640 1 -- 172.21.15.125:0/1234090147 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31c80ac980 con 0x7f31e00c05d0 2023-10-30T13:23:59.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.259+0000 7f31e4c09640 1 -- 172.21.15.125:0/1234090147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31e00c05d0 msgr2=0x7f31e013ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:59.264 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.259+0000 7f31e4c09640 1 --2- 172.21.15.125:0/1234090147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31e00c05d0 0x7f31e013ed90 secure :-1 s=READY pgs=2088 cs=0 l=1 rev1=1 crypto rx=0x7f31c80097c0 tx=0x7f31c801c480 comp rx=0 tx=0).stop 2023-10-30T13:23:59.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.259+0000 7f31e4c09640 1 -- 172.21.15.125:0/1234090147 shutdown_connections 2023-10-30T13:23:59.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.259+0000 7f31e4c09640 1 --2- 172.21.15.125:0/1234090147 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31e00c05d0 0x7f31e013ed90 unknown :-1 s=CLOSED pgs=2088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:59.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.259+0000 7f31e4c09640 1 -- 172.21.15.125:0/1234090147 >> 172.21.15.125:0/1234090147 conn(0x7f31e00c7910 msgr2=0x7f31e00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:59.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.259+0000 7f31e4c09640 1 -- 172.21.15.125:0/1234090147 shutdown_connections 2023-10-30T13:23:59.265 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.259+0000 7f31e4c09640 1 -- 172.21.15.125:0/1234090147 wait complete. 2023-10-30T13:23:59.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31e4c09640 1 Processor -- start 2023-10-30T13:23:59.267 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31e4c09640 1 -- start start 2023-10-30T13:23:59.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31e4c09640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31e00c05d0 0x7f31e00c5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:59.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31e4c09640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31e00c6b80 0x7f31e01efdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:59.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31e4c09640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31e00c6f60 0x7f31e01f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:59.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31e4c09640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e01549f0 con 0x7f31e00c05d0 2023-10-30T13:23:59.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31e4c09640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e00ca230 con 0x7f31e00c6f60 2023-10-30T13:23:59.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31e4c09640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e0156330 con 0x7f31e00c6b80 2023-10-30T13:23:59.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31de59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31e00c05d0 0x7f31e00c5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:59.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31de59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31e00c05d0 0x7f31e00c5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:44092/0 (socket says 172.21.15.125:44092) 2023-10-30T13:23:59.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31de59c640 1 -- 172.21.15.125:0/1300080489 learned_addr learned my addr 172.21.15.125:0/1300080489 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:23:59.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31ded9d640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31e00c6f60 0x7f31e01f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:59.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31d5d9b640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31e00c6b80 0x7f31e01efdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:59.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31de59c640 1 -- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31e00c6b80 msgr2=0x7f31e01efdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:59.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31de59c640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31e00c6b80 0x7f31e01efdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:59.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31de59c640 1 -- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31e00c6f60 msgr2=0x7f31e01f0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:59.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31de59c640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31e00c6f60 0x7f31e01f0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:59.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31de59c640 1 -- 172.21.15.125:0/1300080489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31c8009420 con 0x7f31e00c05d0 2023-10-30T13:23:59.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31ded9d640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31e00c6f60 0x7f31e01f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:23:59.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31de59c640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31e00c05d0 0x7f31e00c5a70 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f31d000c4a0 tx=0x7f31d000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:59.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31d77fe640 1 -- 172.21.15.125:0/1300080489 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d00074e0 con 0x7f31e00c05d0 2023-10-30T13:23:59.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31d77fe640 1 -- 172.21.15.125:0/1300080489 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d0007640 con 0x7f31e00c05d0 2023-10-30T13:23:59.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31d77fe640 1 -- 172.21.15.125:0/1300080489 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d0004a30 con 0x7f31e00c05d0 2023-10-30T13:23:59.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.263+0000 7f31e4c09640 1 -- 172.21.15.125:0/1300080489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e01f0840 con 0x7f31e00c05d0 2023-10-30T13:23:59.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.267+0000 7f31e4c09640 1 -- 172.21.15.125:0/1300080489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e01f3d40 con 0x7f31e00c05d0 2023-10-30T13:23:59.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.267+0000 7f31e4c09640 1 -- 172.21.15.125:0/1300080489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e004e170 con 0x7f31e00c05d0 2023-10-30T13:23:59.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.267+0000 7f31d77fe640 1 -- 172.21.15.125:0/1300080489 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f31d0004b90 con 0x7f31e00c05d0 2023-10-30T13:23:59.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.267+0000 7f31d77fe640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31c00664a0 0x7f31c0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:23:59.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.267+0000 7f31d77fe640 1 -- 172.21.15.125:0/1300080489 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f31d0083b60 con 0x7f31e00c05d0 2023-10-30T13:23:59.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.267+0000 7f31d5d9b640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31c00664a0 0x7f31c0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:23:59.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.271+0000 7f31d5d9b640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31c00664a0 0x7f31c0068960 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f31c80ae000 tx=0x7f31c8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:23:59.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.271+0000 7f31d77fe640 1 -- 172.21.15.125:0/1300080489 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f31d0050590 con 0x7f31e00c05d0 2023-10-30T13:23:59.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.451+0000 7f31e4c09640 1 -- 172.21.15.125:0/1300080489 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31e01f4020 con 0x7f31e00c05d0 2023-10-30T13:23:59.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.451+0000 7f31d77fe640 1 -- 172.21.15.125:0/1300080489 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+6554 (secure 0 0 0) 0x7f31d0018030 con 0x7f31e00c05d0 2023-10-30T13:23:59.456 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:23:59.456 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9409,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":253}],"filesystems":[{"mdsmap":{"epoch":258,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:23:34.832807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:active","state_seq":19,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:23:59.456 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 258 2023-10-30T13:23:59.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.455+0000 7f31e4c09640 1 -- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31c00664a0 msgr2=0x7f31c0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:59.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.455+0000 7f31e4c09640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31c00664a0 0x7f31c0068960 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f31c80ae000 tx=0x7f31c8008040 comp rx=0 tx=0).stop 2023-10-30T13:23:59.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.455+0000 7f31e4c09640 1 -- 172.21.15.125:0/1300080489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31e00c05d0 msgr2=0x7f31e00c5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:23:59.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.455+0000 7f31e4c09640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31e00c05d0 0x7f31e00c5a70 secure :-1 s=READY pgs=2089 cs=0 l=1 rev1=1 crypto rx=0x7f31d000c4a0 tx=0x7f31d000c960 comp rx=0 tx=0).stop 2023-10-30T13:23:59.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.455+0000 7f31e4c09640 1 -- 172.21.15.125:0/1300080489 shutdown_connections 2023-10-30T13:23:59.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.455+0000 7f31e4c09640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f31c00664a0 0x7f31c0068960 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:59.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.455+0000 7f31e4c09640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f31e00c6f60 0x7f31e01f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:59.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.455+0000 7f31e4c09640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f31e00c6b80 0x7f31e01efdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:59.459 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.455+0000 7f31e4c09640 1 --2- 172.21.15.125:0/1300080489 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f31e00c05d0 0x7f31e00c5a70 unknown :-1 s=CLOSED pgs=2089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:23:59.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.455+0000 7f31e4c09640 1 -- 172.21.15.125:0/1300080489 >> 172.21.15.125:0/1300080489 conn(0x7f31e00c7910 msgr2=0x7f31e00c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:23:59.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.455+0000 7f31e4c09640 1 -- 172.21.15.125:0/1300080489 shutdown_connections 2023-10-30T13:23:59.460 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:23:59.455+0000 7f31e4c09640 1 -- 172.21.15.125:0/1300080489 wait complete. 2023-10-30T13:23:59.476 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:24:01.477 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:01.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd23851d640 1 Processor -- start 2023-10-30T13:24:01.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd23851d640 1 -- start start 2023-10-30T13:24:01.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd23851d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd2301538a0 0x7fd23015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:01.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd23851d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2300cf160 0x7fd2300cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:01.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd23851d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2300cb800 con 0x7fd2300ca0e0 2023-10-30T13:24:01.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd23851d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd23015dc30 con 0x7fd2301538a0 2023-10-30T13:24:01.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd23851d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2300cb330 con 0x7fd2300cf160 2023-10-30T13:24:01.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd2362b9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd2301538a0 0x7fd23015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:01.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd2362b9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fd2301538a0 0x7fd23015d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45988/0 (socket says 172.21.15.125:45988) 2023-10-30T13:24:01.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd2362b9640 1 -- 172.21.15.125:0/347268025 learned_addr learned my addr 172.21.15.125:0/347268025 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:01.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd236aba640 1 --2- 172.21.15.125:0/347268025 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2300cf160 0x7fd2300cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:01.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd2362b9640 1 -- 172.21.15.125:0/347268025 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2300cf160 msgr2=0x7fd2300cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:01.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd2362b9640 1 --2- 172.21.15.125:0/347268025 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2300cf160 0x7fd2300cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:01.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd2362b9640 1 -- 172.21.15.125:0/347268025 >> v1:172.21.15.125:6789/0 conn(0x7fd2300ca0e0 legacy=0x7fd23015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:01.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.619+0000 7fd2362b9640 1 -- 172.21.15.125:0/347268025 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2300cad70 con 0x7fd2301538a0 2023-10-30T13:24:01.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.623+0000 7fd2362b9640 1 --2- 172.21.15.125:0/347268025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2301538a0 0x7fd23015d6f0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7fd22c009870 tx=0x7fd22c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e615406f6233605 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.623+0000 7fd2352b7640 1 -- 172.21.15.125:0/347268025 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c0a8a30 con 0x7fd2301538a0 2023-10-30T13:24:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.623+0000 7fd2352b7640 1 -- 172.21.15.125:0/347268025 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22c0a8b90 con 0x7fd2301538a0 2023-10-30T13:24:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.623+0000 7fd2352b7640 1 -- 172.21.15.125:0/347268025 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c0b0440 con 0x7fd2301538a0 2023-10-30T13:24:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.623+0000 7fd23851d640 1 -- 172.21.15.125:0/347268025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2301538a0 msgr2=0x7fd23015d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:01.626 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.623+0000 7fd23851d640 1 --2- 172.21.15.125:0/347268025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2301538a0 0x7fd23015d6f0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7fd22c009870 tx=0x7fd22c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:24:01.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.623+0000 7fd23851d640 1 -- 172.21.15.125:0/347268025 shutdown_connections 2023-10-30T13:24:01.627 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.623+0000 7fd23851d640 1 --2- 172.21.15.125:0/347268025 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2300cf160 0x7fd2300cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:01.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.623+0000 7fd23851d640 1 --2- 172.21.15.125:0/347268025 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2301538a0 0x7fd23015d6f0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:01.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.623+0000 7fd23851d640 1 -- 172.21.15.125:0/347268025 >> 172.21.15.125:0/347268025 conn(0x7fd23006bfe0 msgr2=0x7fd23006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:01.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.623+0000 7fd23851d640 1 -- 172.21.15.125:0/347268025 shutdown_connections 2023-10-30T13:24:01.628 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.623+0000 7fd23851d640 1 -- 172.21.15.125:0/347268025 wait complete. 2023-10-30T13:24:01.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.623+0000 7fd23851d640 1 Processor -- start 2023-10-30T13:24:01.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd23851d640 1 -- start start 2023-10-30T13:24:01.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd23851d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2300ca0e0 0x7fd2301e4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:01.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd23851d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2300cf160 0x7fd23015b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:01.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd23851d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2301538a0 0x7fd23015bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:01.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd23851d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23014d4f0 con 0x7fd2300ca0e0 2023-10-30T13:24:01.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd23851d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2301e4d40 con 0x7fd2301538a0 2023-10-30T13:24:01.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd23851d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd23015b2a0 con 0x7fd2300cf160 2023-10-30T13:24:01.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd2362b9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2300ca0e0 0x7fd2301e4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:01.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd2362b9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2300ca0e0 0x7fd2301e4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:45996/0 (socket says 172.21.15.125:45996) 2023-10-30T13:24:01.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd2362b9640 1 -- 172.21.15.125:0/1555775657 learned_addr learned my addr 172.21.15.125:0/1555775657 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:01.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd236aba640 1 --2- 172.21.15.125:0/1555775657 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2301538a0 0x7fd23015bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:01.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd235ab8640 1 --2- 172.21.15.125:0/1555775657 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2300cf160 0x7fd23015b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:01.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd2362b9640 1 -- 172.21.15.125:0/1555775657 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2300cf160 msgr2=0x7fd23015b630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:01.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd2362b9640 1 --2- 172.21.15.125:0/1555775657 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2300cf160 0x7fd23015b630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:01.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd2362b9640 1 -- 172.21.15.125:0/1555775657 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2301538a0 msgr2=0x7fd23015bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:01.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd2362b9640 1 --2- 172.21.15.125:0/1555775657 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2301538a0 0x7fd23015bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:01.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd2362b9640 1 -- 172.21.15.125:0/1555775657 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22c0094d0 con 0x7fd2300ca0e0 2023-10-30T13:24:01.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd2362b9640 1 --2- 172.21.15.125:0/1555775657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2300ca0e0 0x7fd2301e4090 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0b3040 tx=0x7fd22c01bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:01.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd226ffd640 1 -- 172.21.15.125:0/1555775657 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c0ba600 con 0x7fd2300ca0e0 2023-10-30T13:24:01.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd226ffd640 1 -- 172.21.15.125:0/1555775657 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22c0ba780 con 0x7fd2300ca0e0 2023-10-30T13:24:01.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd226ffd640 1 -- 172.21.15.125:0/1555775657 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c0bab30 con 0x7fd2300ca0e0 2023-10-30T13:24:01.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd23851d640 1 -- 172.21.15.125:0/1555775657 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2300cef10 con 0x7fd2300ca0e0 2023-10-30T13:24:01.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd23851d640 1 -- 172.21.15.125:0/1555775657 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2300c50b0 con 0x7fd2300ca0e0 2023-10-30T13:24:01.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.627+0000 7fd23851d640 1 -- 172.21.15.125:0/1555775657 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2300b2a50 con 0x7fd2300ca0e0 2023-10-30T13:24:01.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.631+0000 7fd226ffd640 1 -- 172.21.15.125:0/1555775657 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd22c0bac90 con 0x7fd2300ca0e0 2023-10-30T13:24:01.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.631+0000 7fd226ffd640 1 --2- 172.21.15.125:0/1555775657 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2140664a0 0x7fd214068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:01.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.631+0000 7fd226ffd640 1 -- 172.21.15.125:0/1555775657 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7fd22c122e40 con 0x7fd2300ca0e0 2023-10-30T13:24:01.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.631+0000 7fd235ab8640 1 --2- 172.21.15.125:0/1555775657 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2140664a0 0x7fd214068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:01.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.631+0000 7fd235ab8640 1 --2- 172.21.15.125:0/1555775657 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2140664a0 0x7fd214068960 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd2300c4660 tx=0x7fd220009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:01.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.635+0000 7fd226ffd640 1 -- 172.21.15.125:0/1555775657 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd22c0ef7e0 con 0x7fd2300ca0e0 2023-10-30T13:24:01.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.811+0000 7fd23851d640 1 -- 172.21.15.125:0/1555775657 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd230151940 con 0x7fd2300ca0e0 2023-10-30T13:24:01.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.815+0000 7fd226ffd640 1 -- 172.21.15.125:0/1555775657 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+6554 (secure 0 0 0) 0x7fd22c0b5020 con 0x7fd2300ca0e0 2023-10-30T13:24:01.817 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:01.817 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9409,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":253}],"filesystems":[{"mdsmap":{"epoch":258,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:23:34.832807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:active","state_seq":19,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:01.817 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 258 2023-10-30T13:24:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.815+0000 7fd23851d640 1 -- 172.21.15.125:0/1555775657 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2140664a0 msgr2=0x7fd214068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:01.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.815+0000 7fd23851d640 1 --2- 172.21.15.125:0/1555775657 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2140664a0 0x7fd214068960 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fd2300c4660 tx=0x7fd220009040 comp rx=0 tx=0).stop 2023-10-30T13:24:01.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.815+0000 7fd23851d640 1 -- 172.21.15.125:0/1555775657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2300ca0e0 msgr2=0x7fd2301e4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:01.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.815+0000 7fd23851d640 1 --2- 172.21.15.125:0/1555775657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2300ca0e0 0x7fd2301e4090 secure :-1 s=READY pgs=2092 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0b3040 tx=0x7fd22c01bad0 comp rx=0 tx=0).stop 2023-10-30T13:24:01.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.815+0000 7fd23851d640 1 -- 172.21.15.125:0/1555775657 shutdown_connections 2023-10-30T13:24:01.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.815+0000 7fd23851d640 1 --2- 172.21.15.125:0/1555775657 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd2140664a0 0x7fd214068960 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:01.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.815+0000 7fd23851d640 1 --2- 172.21.15.125:0/1555775657 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd2301538a0 0x7fd23015bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:01.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.815+0000 7fd23851d640 1 --2- 172.21.15.125:0/1555775657 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd2300cf160 0x7fd23015b630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:01.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.815+0000 7fd23851d640 1 --2- 172.21.15.125:0/1555775657 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd2300ca0e0 0x7fd2301e4090 unknown :-1 s=CLOSED pgs=2092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:01.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.815+0000 7fd23851d640 1 -- 172.21.15.125:0/1555775657 >> 172.21.15.125:0/1555775657 conn(0x7fd23006bfe0 msgr2=0x7fd230065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:01.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.815+0000 7fd23851d640 1 -- 172.21.15.125:0/1555775657 shutdown_connections 2023-10-30T13:24:01.820 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:01.815+0000 7fd23851d640 1 -- 172.21.15.125:0/1555775657 wait complete. 2023-10-30T13:24:01.837 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:24:03.838 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:03.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e7cb3e640 1 Processor -- start 2023-10-30T13:24:03.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e7cb3e640 1 -- start start 2023-10-30T13:24:03.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e7cb3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e7815d6f0 0x7f6e7815a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:03.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e7cb3e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6e780cf160 0x7f6e780cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:03.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e7cb3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e780cb800 con 0x7f6e781538a0 2023-10-30T13:24:03.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e7cb3e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e780cb330 con 0x7f6e780cf160 2023-10-30T13:24:03.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e7cb3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780cad70 con 0x7f6e7815d6f0 2023-10-30T13:24:03.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e75d9b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e7815d6f0 0x7f6e7815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:03.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e76d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6e780cf160 0x7f6e780cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:03.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e76d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6e780cf160 0x7f6e780cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:49242/0 (socket says 172.21.15.125:49242) 2023-10-30T13:24:03.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e76d9d640 1 -- 172.21.15.125:0/2306510080 learned_addr learned my addr 172.21.15.125:0/2306510080 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:03.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e75d9b640 1 -- 172.21.15.125:0/2306510080 >> v2:172.21.15.136:3300/0 conn(0x7f6e780cf160 msgr2=0x7f6e780cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:03.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e75d9b640 1 --2- 172.21.15.125:0/2306510080 >> v2:172.21.15.136:3300/0 conn(0x7f6e780cf160 0x7f6e780cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:03.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e75d9b640 1 -- 172.21.15.125:0/2306510080 >> v1:172.21.15.136:6789/0 conn(0x7f6e781538a0 legacy=0x7f6e780ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:03.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e75d9b640 1 -- 172.21.15.125:0/2306510080 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e780cb050 con 0x7f6e7815d6f0 2023-10-30T13:24:03.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.983+0000 7f6e76d9d640 1 --2- 172.21.15.125:0/2306510080 >> v2:172.21.15.136:3300/0 conn(0x7f6e780cf160 0x7f6e780cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:24:03.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.987+0000 7f6e75d9b640 1 --2- 172.21.15.125:0/2306510080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e7815d6f0 0x7f6e7815a090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0097c0 tx=0x7f6e6c01ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0748ff4475ece6e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:03.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.987+0000 7f6e7559a640 1 -- 172.21.15.125:0/2306510080 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c0a8970 con 0x7f6e7815d6f0 2023-10-30T13:24:03.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.987+0000 7f6e7559a640 1 -- 172.21.15.125:0/2306510080 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e6c0a8ad0 con 0x7f6e7815d6f0 2023-10-30T13:24:03.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.987+0000 7f6e7559a640 1 -- 172.21.15.125:0/2306510080 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c0a8e60 con 0x7f6e7815d6f0 2023-10-30T13:24:03.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.987+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/2306510080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e7815d6f0 msgr2=0x7f6e7815a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:03.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.987+0000 7f6e7cb3e640 1 --2- 172.21.15.125:0/2306510080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e7815d6f0 0x7f6e7815a090 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0097c0 tx=0x7f6e6c01ec70 comp rx=0 tx=0).stop 2023-10-30T13:24:03.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.987+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/2306510080 shutdown_connections 2023-10-30T13:24:03.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.987+0000 7f6e7cb3e640 1 --2- 172.21.15.125:0/2306510080 >> v2:172.21.15.136:3300/0 conn(0x7f6e780cf160 0x7f6e780cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:03.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.987+0000 7f6e7cb3e640 1 --2- 172.21.15.125:0/2306510080 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e7815d6f0 0x7f6e7815a090 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:03.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.987+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/2306510080 >> 172.21.15.125:0/2306510080 conn(0x7f6e7806bfe0 msgr2=0x7f6e7806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:03.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/2306510080 shutdown_connections 2023-10-30T13:24:03.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/2306510080 wait complete. 2023-10-30T13:24:03.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7cb3e640 1 Processor -- start 2023-10-30T13:24:03.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7cb3e640 1 -- start start 2023-10-30T13:24:03.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7cb3e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e780cf160 0x7f6e781f2460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:03.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7cb3e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e781538a0 0x7f6e781e3340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:03.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7cb3e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e7815d6f0 0x7f6e781e4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:03.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7cb3e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e781e4150 con 0x7f6e780cf160 2023-10-30T13:24:03.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7cb3e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e781e4710 con 0x7f6e7815d6f0 2023-10-30T13:24:03.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7cb3e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e780c6b20 con 0x7f6e781538a0 2023-10-30T13:24:03.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e780cf160 0x7f6e781f2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:03.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e780cf160 0x7f6e781f2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46008/0 (socket says 172.21.15.125:46008) 2023-10-30T13:24:03.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7659c640 1 -- 172.21.15.125:0/3984128938 learned_addr learned my addr 172.21.15.125:0/3984128938 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:03.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e76d9d640 1 --2- 172.21.15.125:0/3984128938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e7815d6f0 0x7f6e781e4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:03.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e75d9b640 1 --2- 172.21.15.125:0/3984128938 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e781538a0 0x7f6e781e3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:03.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7659c640 1 -- 172.21.15.125:0/3984128938 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e781538a0 msgr2=0x7f6e781e3340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:04.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7659c640 1 --2- 172.21.15.125:0/3984128938 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e781538a0 0x7f6e781e3340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:04.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7659c640 1 -- 172.21.15.125:0/3984128938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e7815d6f0 msgr2=0x7f6e781e4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:04.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7659c640 1 --2- 172.21.15.125:0/3984128938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e7815d6f0 0x7f6e781e4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:04.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7659c640 1 -- 172.21.15.125:0/3984128938 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e70009b30 con 0x7f6e780cf160 2023-10-30T13:24:04.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.991+0000 7f6e7659c640 1 --2- 172.21.15.125:0/3984128938 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e780cf160 0x7f6e781f2460 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800c530 tx=0x7f6e6800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:04.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.995+0000 7f6e66ffd640 1 -- 172.21.15.125:0/3984128938 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68005220 con 0x7f6e780cf160 2023-10-30T13:24:04.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.995+0000 7f6e66ffd640 1 -- 172.21.15.125:0/3984128938 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e68005380 con 0x7f6e780cf160 2023-10-30T13:24:04.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.995+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/3984128938 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e6c009420 con 0x7f6e780cf160 2023-10-30T13:24:04.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.995+0000 7f6e66ffd640 1 -- 172.21.15.125:0/3984128938 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e68007270 con 0x7f6e780cf160 2023-10-30T13:24:04.001 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.995+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/3984128938 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e781f2c10 con 0x7f6e780cf160 2023-10-30T13:24:04.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.995+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/3984128938 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e780b2a50 con 0x7f6e780cf160 2023-10-30T13:24:04.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.995+0000 7f6e66ffd640 1 -- 172.21.15.125:0/3984128938 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6e680073d0 con 0x7f6e780cf160 2023-10-30T13:24:04.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.995+0000 7f6e66ffd640 1 --2- 172.21.15.125:0/3984128938 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e580664a0 0x7f6e58068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:04.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.995+0000 7f6e66ffd640 1 -- 172.21.15.125:0/3984128938 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f6e680840a0 con 0x7f6e780cf160 2023-10-30T13:24:04.002 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.995+0000 7f6e75d9b640 1 --2- 172.21.15.125:0/3984128938 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e580664a0 0x7f6e58068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:04.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.999+0000 7f6e75d9b640 1 --2- 172.21.15.125:0/3984128938 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e580664a0 0x7f6e58068960 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0b2000 tx=0x7f6e6c0aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:04.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:03.999+0000 7f6e66ffd640 1 -- 172.21.15.125:0/3984128938 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6e68050b80 con 0x7f6e780cf160 2023-10-30T13:24:04.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:04.183+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/3984128938 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e7804e170 con 0x7f6e780cf160 2023-10-30T13:24:04.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:04.183+0000 7f6e66ffd640 1 -- 172.21.15.125:0/3984128938 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+6554 (secure 0 0 0) 0x7f6e680541a0 con 0x7f6e780cf160 2023-10-30T13:24:04.186 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:04.186 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9409,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":253}],"filesystems":[{"mdsmap":{"epoch":258,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:23:34.832807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:active","state_seq":19,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:04.186 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 258 2023-10-30T13:24:04.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:04.183+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/3984128938 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e580664a0 msgr2=0x7f6e58068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:04.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:04.183+0000 7f6e7cb3e640 1 --2- 172.21.15.125:0/3984128938 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e580664a0 0x7f6e58068960 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c0b2000 tx=0x7f6e6c0aa070 comp rx=0 tx=0).stop 2023-10-30T13:24:04.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:04.183+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/3984128938 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e780cf160 msgr2=0x7f6e781f2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:04.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:04.183+0000 7f6e7cb3e640 1 --2- 172.21.15.125:0/3984128938 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e780cf160 0x7f6e781f2460 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f6e6800c530 tx=0x7f6e6800c9f0 comp rx=0 tx=0).stop 2023-10-30T13:24:04.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:04.183+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/3984128938 shutdown_connections 2023-10-30T13:24:04.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:04.183+0000 7f6e7cb3e640 1 --2- 172.21.15.125:0/3984128938 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6e580664a0 0x7f6e58068960 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:04.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:04.183+0000 7f6e7cb3e640 1 --2- 172.21.15.125:0/3984128938 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6e7815d6f0 0x7f6e781e4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:04.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:04.183+0000 7f6e7cb3e640 1 --2- 172.21.15.125:0/3984128938 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6e781538a0 0x7f6e781e3340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:04.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:04.183+0000 7f6e7cb3e640 1 --2- 172.21.15.125:0/3984128938 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6e780cf160 0x7f6e781f2460 unknown :-1 s=CLOSED pgs=2093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:04.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:04.183+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/3984128938 >> 172.21.15.125:0/3984128938 conn(0x7f6e7806bfe0 msgr2=0x7f6e781510d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:04.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:04.183+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/3984128938 shutdown_connections 2023-10-30T13:24:04.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:04.183+0000 7f6e7cb3e640 1 -- 172.21.15.125:0/3984128938 wait complete. 2023-10-30T13:24:04.207 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:24:06.208 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:06.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.351+0000 7f11777c4640 1 Processor -- start 2023-10-30T13:24:06.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.351+0000 7f11777c4640 1 -- start start 2023-10-30T13:24:06.356 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f11777c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11701539a0 0x7f117015a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:06.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f11777c4640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1170069e90 0x7f117006a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:06.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f11777c4640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f117015a8c0 con 0x7f1170070a00 2023-10-30T13:24:06.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f11777c4640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f117006fbb0 con 0x7f1170069e90 2023-10-30T13:24:06.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f11777c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1170153d80 con 0x7f11701539a0 2023-10-30T13:24:06.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1174d5f640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1170070a00 0x7f1170070e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38440/0 (socket says 172.21.15.125:38440) 2023-10-30T13:24:06.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1174d5f640 1 -- 172.21.15.125:0/1841883023 learned_addr learned my addr 172.21.15.125:0/1841883023 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:24:06.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1175560640 1 --2- 172.21.15.125:0/1841883023 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11701539a0 0x7f117015a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:06.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1175d61640 1 --2- 172.21.15.125:0/1841883023 >> v2:172.21.15.136:3300/0 conn(0x7f1170069e90 0x7f117006a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:06.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 -- 172.21.15.125:0/1841883023 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2254632660 0 0) 0x7f117015a8c0 con 0x7f1170070a00 2023-10-30T13:24:06.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 -- 172.21.15.125:0/1841883023 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f115c003530 con 0x7f1170070a00 2023-10-30T13:24:06.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 -- 172.21.15.125:0/1841883023 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3109994027 0 0) 0x7f115c003530 con 0x7f1170070a00 2023-10-30T13:24:06.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 -- 172.21.15.125:0/1841883023 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11701539a0 msgr2=0x7f117015a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:06.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 --2- 172.21.15.125:0/1841883023 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11701539a0 0x7f117015a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:06.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 -- 172.21.15.125:0/1841883023 >> v2:172.21.15.136:3300/0 conn(0x7f1170069e90 msgr2=0x7f117006a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:06.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 --2- 172.21.15.125:0/1841883023 >> v2:172.21.15.136:3300/0 conn(0x7f1170069e90 0x7f117006a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:06.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 -- 172.21.15.125:0/1841883023 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11700c5c50 con 0x7f1170070a00 2023-10-30T13:24:06.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 -- 172.21.15.125:0/1841883023 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f1160001680 con 0x7f1170070a00 2023-10-30T13:24:06.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1175d61640 1 --2- 172.21.15.125:0/1841883023 >> v2:172.21.15.136:3300/0 conn(0x7f1170069e90 0x7f117006a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:24:06.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 -- 172.21.15.125:0/1841883023 >> v1:172.21.15.125:6789/0 conn(0x7f1170070a00 legacy=0x7f1170070e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:06.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 --2- 172.21.15.125:0/1841883023 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11701539a0 0x7f115c003980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:06.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 --2- 172.21.15.125:0/1841883023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f115c003f00 0x7f115c0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:06.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 --2- 172.21.15.125:0/1841883023 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f115c004d10 0x7f115c005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:06.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 -- 172.21.15.125:0/1841883023 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117006fbb0 con 0x7f115c004d10 2023-10-30T13:24:06.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 -- 172.21.15.125:0/1841883023 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1170153d80 con 0x7f115c003f00 2023-10-30T13:24:06.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1167fff640 1 -- 172.21.15.125:0/1841883023 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f115c00b9e0 con 0x7f11701539a0 2023-10-30T13:24:06.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1175d61640 1 --2- 172.21.15.125:0/1841883023 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f115c004d10 0x7f115c005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:06.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1174d5f640 1 --2- 172.21.15.125:0/1841883023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f115c003f00 0x7f115c0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:06.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1175560640 1 --2- 172.21.15.125:0/1841883023 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11701539a0 0x7f115c003980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:06.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1175d61640 1 -- 172.21.15.125:0/1841883023 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11701539a0 msgr2=0x7f115c003980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:06.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1175d61640 1 --2- 172.21.15.125:0/1841883023 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11701539a0 0x7f115c003980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:06.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1175d61640 1 -- 172.21.15.125:0/1841883023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f115c003f00 msgr2=0x7f115c0047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:06.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1175d61640 1 --2- 172.21.15.125:0/1841883023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f115c003f00 0x7f115c0047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:06.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1175d61640 1 -- 172.21.15.125:0/1841883023 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115c003530 con 0x7f115c004d10 2023-10-30T13:24:06.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.355+0000 7f1174d5f640 1 --2- 172.21.15.125:0/1841883023 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f115c003f00 0x7f115c0047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:06.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.359+0000 7f1175d61640 1 --2- 172.21.15.125:0/1841883023 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f115c004d10 0x7f115c005190 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f116800e380 tx=0x7f116801ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e629828445cd0461 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:06.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.359+0000 7f1167fff640 1 -- 172.21.15.125:0/1841883023 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11680a5e50 con 0x7f115c004d10 2023-10-30T13:24:06.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.359+0000 7f1167fff640 1 -- 172.21.15.125:0/1841883023 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116800f040 con 0x7f115c004d10 2023-10-30T13:24:06.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.359+0000 7f1167fff640 1 -- 172.21.15.125:0/1841883023 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11680ad640 con 0x7f115c004d10 2023-10-30T13:24:06.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.359+0000 7f11777c4640 1 -- 172.21.15.125:0/1841883023 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f115c004d10 msgr2=0x7f115c005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:06.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.359+0000 7f11777c4640 1 --2- 172.21.15.125:0/1841883023 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f115c004d10 0x7f115c005190 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f116800e380 tx=0x7f116801ce40 comp rx=0 tx=0).stop 2023-10-30T13:24:06.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.359+0000 7f1175d61640 1 -- 172.21.15.125:0/1841883023 reap_dead start 2023-10-30T13:24:06.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.359+0000 7f11777c4640 1 -- 172.21.15.125:0/1841883023 shutdown_connections 2023-10-30T13:24:06.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.359+0000 7f11777c4640 1 -- 172.21.15.125:0/1841883023 >> 172.21.15.125:0/1841883023 conn(0x7f11700c7910 msgr2=0x7f11700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:06.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.359+0000 7f11777c4640 1 -- 172.21.15.125:0/1841883023 shutdown_connections 2023-10-30T13:24:06.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.359+0000 7f11777c4640 1 -- 172.21.15.125:0/1841883023 wait complete. 2023-10-30T13:24:06.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.359+0000 7f11777c4640 1 Processor -- start 2023-10-30T13:24:06.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f11777c4640 1 -- start start 2023-10-30T13:24:06.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f11777c4640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f115c004d10 0x7f11701f2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:06.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f11777c4640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11700c3660 0x7f11700c12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:06.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f11777c4640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11700c4240 0x7f11700c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:06.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f11777c4640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11701f3270 con 0x7f11700c3660 2023-10-30T13:24:06.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f11777c4640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11700c4c30 con 0x7f115c004d10 2023-10-30T13:24:06.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f11777c4640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1170156ba0 con 0x7f11700c4240 2023-10-30T13:24:06.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f1174d5f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11700c3660 0x7f11700c12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:06.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f1174d5f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11700c3660 0x7f11700c12a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:46020/0 (socket says 172.21.15.125:46020) 2023-10-30T13:24:06.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f1174d5f640 1 -- 172.21.15.125:0/3185627875 learned_addr learned my addr 172.21.15.125:0/3185627875 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:06.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f1175560640 1 --2- 172.21.15.125:0/3185627875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f115c004d10 0x7f11701f2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:06.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f1175d61640 1 --2- 172.21.15.125:0/3185627875 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11700c4240 0x7f11700c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:06.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f1174d5f640 1 -- 172.21.15.125:0/3185627875 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11700c4240 msgr2=0x7f11700c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:06.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f1174d5f640 1 --2- 172.21.15.125:0/3185627875 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11700c4240 0x7f11700c46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:06.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f1174d5f640 1 -- 172.21.15.125:0/3185627875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f115c004d10 msgr2=0x7f11701f2d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:06.373 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f1174d5f640 1 --2- 172.21.15.125:0/3185627875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f115c004d10 0x7f11701f2d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:06.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f1174d5f640 1 -- 172.21.15.125:0/3185627875 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1170156e00 con 0x7f11700c3660 2023-10-30T13:24:06.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f1174d5f640 1 --2- 172.21.15.125:0/3185627875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11700c3660 0x7f11700c12a0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f11600117f0 tx=0x7f1160011cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:06.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f1165ffb640 1 -- 172.21.15.125:0/3185627875 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f116000fb00 con 0x7f11700c3660 2023-10-30T13:24:06.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f1165ffb640 1 -- 172.21.15.125:0/3185627875 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f116000fc60 con 0x7f11700c3660 2023-10-30T13:24:06.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f11777c4640 1 -- 172.21.15.125:0/3185627875 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11701570e0 con 0x7f11700c3660 2023-10-30T13:24:06.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f1165ffb640 1 -- 172.21.15.125:0/3185627875 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1160015410 con 0x7f11700c3660 2023-10-30T13:24:06.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f11777c4640 1 -- 172.21.15.125:0/3185627875 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11700c2590 con 0x7f11700c3660 2023-10-30T13:24:06.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.363+0000 7f11777c4640 1 -- 172.21.15.125:0/3185627875 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11700c3f60 con 0x7f11700c3660 2023-10-30T13:24:06.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.367+0000 7f1165ffb640 1 -- 172.21.15.125:0/3185627875 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1160015590 con 0x7f11700c3660 2023-10-30T13:24:06.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.367+0000 7f1165ffb640 1 --2- 172.21.15.125:0/3185627875 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11540664f0 0x7f11540689b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:06.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.367+0000 7f1165ffb640 1 -- 172.21.15.125:0/3185627875 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f1160087a30 con 0x7f11700c3660 2023-10-30T13:24:06.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.367+0000 7f1175560640 1 --2- 172.21.15.125:0/3185627875 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11540664f0 0x7f11540689b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:06.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.367+0000 7f1175560640 1 --2- 172.21.15.125:0/3185627875 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11540664f0 0x7f11540689b0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f116c009ae0 tx=0x7f116c008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:06.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.371+0000 7f1165ffb640 1 -- 172.21.15.125:0/3185627875 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f11700c3f60 con 0x7f11700c3660 2023-10-30T13:24:06.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.551+0000 7f11777c4640 1 -- 172.21.15.125:0/3185627875 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11700c3f60 con 0x7f11700c3660 2023-10-30T13:24:06.556 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.555+0000 7f1165ffb640 1 -- 172.21.15.125:0/3185627875 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+6554 (secure 0 0 0) 0x7f11700c3f60 con 0x7f11700c3660 2023-10-30T13:24:06.557 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:06.557 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9409,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":253}],"filesystems":[{"mdsmap":{"epoch":258,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:23:34.832807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:active","state_seq":19,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:06.557 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 258 2023-10-30T13:24:06.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.555+0000 7f11777c4640 1 -- 172.21.15.125:0/3185627875 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11540664f0 msgr2=0x7f11540689b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:06.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.555+0000 7f11777c4640 1 --2- 172.21.15.125:0/3185627875 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11540664f0 0x7f11540689b0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f116c009ae0 tx=0x7f116c008040 comp rx=0 tx=0).stop 2023-10-30T13:24:06.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.555+0000 7f11777c4640 1 -- 172.21.15.125:0/3185627875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11700c3660 msgr2=0x7f11700c12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:06.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.555+0000 7f11777c4640 1 --2- 172.21.15.125:0/3185627875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11700c3660 0x7f11700c12a0 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f11600117f0 tx=0x7f1160011cb0 comp rx=0 tx=0).stop 2023-10-30T13:24:06.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.555+0000 7f11777c4640 1 -- 172.21.15.125:0/3185627875 shutdown_connections 2023-10-30T13:24:06.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.555+0000 7f11777c4640 1 --2- 172.21.15.125:0/3185627875 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f11540664f0 0x7f11540689b0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:06.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.555+0000 7f11777c4640 1 --2- 172.21.15.125:0/3185627875 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f11700c4240 0x7f11700c46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:06.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.555+0000 7f11777c4640 1 --2- 172.21.15.125:0/3185627875 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f11700c3660 0x7f11700c12a0 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:06.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.555+0000 7f11777c4640 1 --2- 172.21.15.125:0/3185627875 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f115c004d10 0x7f11701f2d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:06.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.555+0000 7f11777c4640 1 -- 172.21.15.125:0/3185627875 >> 172.21.15.125:0/3185627875 conn(0x7f11700c7910 msgr2=0x7f117014e780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:06.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.555+0000 7f11777c4640 1 -- 172.21.15.125:0/3185627875 shutdown_connections 2023-10-30T13:24:06.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:06.555+0000 7f11777c4640 1 -- 172.21.15.125:0/3185627875 wait complete. 2023-10-30T13:24:06.578 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:24:08.579 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:08.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.723+0000 7ff7cedc5640 1 Processor -- start 2023-10-30T13:24:08.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.723+0000 7ff7cedc5640 1 -- start start 2023-10-30T13:24:08.727 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.723+0000 7ff7cedc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c80c6b80 0x7ff7c80bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:08.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.723+0000 7ff7cedc5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c80c0a50 con 0x7ff7c80c6f60 2023-10-30T13:24:08.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.723+0000 7ff7cedc5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c806f6e0 con 0x7ff7c813f4a0 2023-10-30T13:24:08.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.723+0000 7ff7cedc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c806f9c0 con 0x7ff7c80c6b80 2023-10-30T13:24:08.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.723+0000 7ff7bffff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff7c80c6f60 0x7ff7c813ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:38446/0 (socket says 172.21.15.125:38446) 2023-10-30T13:24:08.728 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.723+0000 7ff7bffff640 1 -- 172.21.15.125:0/2004067492 learned_addr learned my addr 172.21.15.125:0/2004067492 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:24:08.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.723+0000 7ff7ccb61640 1 --2- 172.21.15.125:0/2004067492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c80c6b80 0x7ff7c80bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:08.730 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 -- 172.21.15.125:0/2004067492 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714616235 0 0) 0x7ff7c80c0a50 con 0x7ff7c80c6f60 2023-10-30T13:24:08.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 -- 172.21.15.125:0/2004067492 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7b0003530 con 0x7ff7c80c6f60 2023-10-30T13:24:08.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 -- 172.21.15.125:0/2004067492 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2135908788 0 0) 0x7ff7b0003530 con 0x7ff7c80c6f60 2023-10-30T13:24:08.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 -- 172.21.15.125:0/2004067492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c80c6b80 msgr2=0x7ff7c80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:08.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 --2- 172.21.15.125:0/2004067492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c80c6b80 0x7ff7c80bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:08.731 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 -- 172.21.15.125:0/2004067492 >> v1:172.21.15.136:6789/0 conn(0x7ff7c813f4a0 legacy=0x7ff7c81566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:08.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 -- 172.21.15.125:0/2004067492 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c806fca0 con 0x7ff7c80c6f60 2023-10-30T13:24:08.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 -- 172.21.15.125:0/2004067492 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7ff7b8001680 con 0x7ff7c80c6f60 2023-10-30T13:24:08.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 -- 172.21.15.125:0/2004067492 >> v1:172.21.15.125:6789/0 conn(0x7ff7c80c6f60 legacy=0x7ff7c813ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:08.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 --2- 172.21.15.125:0/2004067492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7b00039b0 0x7ff7b0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:08.732 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 --2- 172.21.15.125:0/2004067492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c80c6b80 0x7ff7b00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:08.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 --2- 172.21.15.125:0/2004067492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7b0004d10 0x7ff7b0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:08.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 -- 172.21.15.125:0/2004067492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c806f9c0 con 0x7ff7b0004d10 2023-10-30T13:24:08.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 -- 172.21.15.125:0/2004067492 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b000b990 con 0x7ff7b00039b0 2023-10-30T13:24:08.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bf7fe640 1 -- 172.21.15.125:0/2004067492 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7b000dd60 con 0x7ff7c80c6b80 2023-10-30T13:24:08.733 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7cd362640 1 --2- 172.21.15.125:0/2004067492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7b0004d10 0x7ff7b0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:08.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7ccb61640 1 --2- 172.21.15.125:0/2004067492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7b00039b0 0x7ff7b0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:08.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bffff640 1 --2- 172.21.15.125:0/2004067492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c80c6b80 0x7ff7b00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:08.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7cd362640 1 -- 172.21.15.125:0/2004067492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c80c6b80 msgr2=0x7ff7b00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:08.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7cd362640 1 --2- 172.21.15.125:0/2004067492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c80c6b80 0x7ff7b00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:08.734 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7cd362640 1 -- 172.21.15.125:0/2004067492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7b00039b0 msgr2=0x7ff7b0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:08.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7cd362640 1 --2- 172.21.15.125:0/2004067492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7b00039b0 0x7ff7b0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:08.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7cd362640 1 -- 172.21.15.125:0/2004067492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b0003530 con 0x7ff7b0004d10 2023-10-30T13:24:08.735 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.727+0000 7ff7bffff640 1 --2- 172.21.15.125:0/2004067492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7c80c6b80 0x7ff7b00047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:08.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cd362640 1 --2- 172.21.15.125:0/2004067492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7b0004d10 0x7ff7b0005190 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7ff7c813f8d0 tx=0x7ff7c001c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60be289445bd2242 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:08.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7bf7fe640 1 -- 172.21.15.125:0/2004067492 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c00a6700 con 0x7ff7b0004d10 2023-10-30T13:24:08.739 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7bf7fe640 1 -- 172.21.15.125:0/2004067492 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c00a6860 con 0x7ff7b0004d10 2023-10-30T13:24:08.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7bf7fe640 1 -- 172.21.15.125:0/2004067492 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c00a6c10 con 0x7ff7b0004d10 2023-10-30T13:24:08.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cedc5640 1 -- 172.21.15.125:0/2004067492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7b0004d10 msgr2=0x7ff7b0005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:08.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cedc5640 1 --2- 172.21.15.125:0/2004067492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7b0004d10 0x7ff7b0005190 secure :-1 s=READY pgs=2098 cs=0 l=1 rev1=1 crypto rx=0x7ff7c813f8d0 tx=0x7ff7c001c800 comp rx=0 tx=0).stop 2023-10-30T13:24:08.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cd362640 1 -- 172.21.15.125:0/2004067492 reap_dead start 2023-10-30T13:24:08.740 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cedc5640 1 -- 172.21.15.125:0/2004067492 shutdown_connections 2023-10-30T13:24:08.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cedc5640 1 -- 172.21.15.125:0/2004067492 >> 172.21.15.125:0/2004067492 conn(0x7ff7c80c7910 msgr2=0x7ff7c80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:08.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cedc5640 1 -- 172.21.15.125:0/2004067492 shutdown_connections 2023-10-30T13:24:08.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cedc5640 1 -- 172.21.15.125:0/2004067492 wait complete. 2023-10-30T13:24:08.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cedc5640 1 Processor -- start 2023-10-30T13:24:08.741 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cedc5640 1 -- start start 2023-10-30T13:24:08.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cedc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7b0004d10 0x7ff7c81f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:08.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cedc5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c813e6c0 0x7ff7c81533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:08.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cedc5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7c8155750 0x7ff7c81385b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:08.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cedc5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8154020 con 0x7ff7c8155750 2023-10-30T13:24:08.742 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cedc5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c81f2a30 con 0x7ff7c813e6c0 2023-10-30T13:24:08.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.731+0000 7ff7cedc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c8152050 con 0x7ff7b0004d10 2023-10-30T13:24:08.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7bffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c813e6c0 0x7ff7c81533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:08.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7ccb61640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7b0004d10 0x7ff7c81f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:08.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7cd362640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7c8155750 0x7ff7c81385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:08.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7bffff640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c813e6c0 0x7ff7c81533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:49298/0 (socket says 172.21.15.125:49298) 2023-10-30T13:24:08.743 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7bffff640 1 -- 172.21.15.125:0/1807393576 learned_addr learned my addr 172.21.15.125:0/1807393576 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:08.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7cd362640 1 -- 172.21.15.125:0/1807393576 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7b0004d10 msgr2=0x7ff7c81f24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:08.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7cd362640 1 --2- 172.21.15.125:0/1807393576 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7b0004d10 0x7ff7c81f24f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:08.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7cd362640 1 -- 172.21.15.125:0/1807393576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c813e6c0 msgr2=0x7ff7c81533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:08.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7cd362640 1 --2- 172.21.15.125:0/1807393576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c813e6c0 0x7ff7c81533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:08.744 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7cd362640 1 -- 172.21.15.125:0/1807393576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c8152330 con 0x7ff7c8155750 2023-10-30T13:24:08.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7cd362640 1 --2- 172.21.15.125:0/1807393576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7c8155750 0x7ff7c81385b0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00ac000 tx=0x7ff7b0005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:08.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7bd7fa640 1 -- 172.21.15.125:0/1807393576 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c00a6de0 con 0x7ff7c8155750 2023-10-30T13:24:08.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7bd7fa640 1 -- 172.21.15.125:0/1807393576 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7c00011d0 con 0x7ff7c8155750 2023-10-30T13:24:08.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7bd7fa640 1 -- 172.21.15.125:0/1807393576 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7c0003b80 con 0x7ff7c8155750 2023-10-30T13:24:08.745 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7cedc5640 1 -- 172.21.15.125:0/1807393576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c8152610 con 0x7ff7c8155750 2023-10-30T13:24:08.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7cedc5640 1 -- 172.21.15.125:0/1807393576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c8152bd0 con 0x7ff7c8155750 2023-10-30T13:24:08.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7cedc5640 1 -- 172.21.15.125:0/1807393576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c804e170 con 0x7ff7c8155750 2023-10-30T13:24:08.746 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7bd7fa640 1 -- 172.21.15.125:0/1807393576 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff7c0003ce0 con 0x7ff7c8155750 2023-10-30T13:24:08.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.735+0000 7ff7bd7fa640 1 --2- 172.21.15.125:0/1807393576 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7b40664a0 0x7ff7b4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:08.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.739+0000 7ff7bd7fa640 1 -- 172.21.15.125:0/1807393576 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7ff7c00aa070 con 0x7ff7c8155750 2023-10-30T13:24:08.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.739+0000 7ff7ccb61640 1 --2- 172.21.15.125:0/1807393576 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7b40664a0 0x7ff7b4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:08.747 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.739+0000 7ff7ccb61640 1 --2- 172.21.15.125:0/1807393576 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7b40664a0 0x7ff7b4068960 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4009b60 tx=0x7ff7c4009190 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:08.748 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.739+0000 7ff7bd7fa640 1 -- 172.21.15.125:0/1807393576 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff7c00ebd10 con 0x7ff7c8155750 2023-10-30T13:24:08.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.923+0000 7ff7cedc5640 1 -- 172.21.15.125:0/1807393576 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff7c813f230 con 0x7ff7c8155750 2023-10-30T13:24:08.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.923+0000 7ff7bd7fa640 1 -- 172.21.15.125:0/1807393576 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+6554 (secure 0 0 0) 0x7ff7c00ef330 con 0x7ff7c8155750 2023-10-30T13:24:08.926 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:08.926 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9409,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":253}],"filesystems":[{"mdsmap":{"epoch":258,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:23:34.832807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:active","state_seq":19,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:08.927 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 258 2023-10-30T13:24:08.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.923+0000 7ff7cedc5640 1 -- 172.21.15.125:0/1807393576 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7b40664a0 msgr2=0x7ff7b4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:08.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.923+0000 7ff7cedc5640 1 --2- 172.21.15.125:0/1807393576 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7b40664a0 0x7ff7b4068960 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4009b60 tx=0x7ff7c4009190 comp rx=0 tx=0).stop 2023-10-30T13:24:08.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.923+0000 7ff7cedc5640 1 -- 172.21.15.125:0/1807393576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7c8155750 msgr2=0x7ff7c81385b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:08.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.923+0000 7ff7cedc5640 1 --2- 172.21.15.125:0/1807393576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7c8155750 0x7ff7c81385b0 secure :-1 s=READY pgs=2099 cs=0 l=1 rev1=1 crypto rx=0x7ff7c00ac000 tx=0x7ff7b0005800 comp rx=0 tx=0).stop 2023-10-30T13:24:08.928 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.923+0000 7ff7cedc5640 1 -- 172.21.15.125:0/1807393576 shutdown_connections 2023-10-30T13:24:08.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.923+0000 7ff7cedc5640 1 --2- 172.21.15.125:0/1807393576 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff7b40664a0 0x7ff7b4068960 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:08.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.923+0000 7ff7cedc5640 1 --2- 172.21.15.125:0/1807393576 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff7c8155750 0x7ff7c81385b0 unknown :-1 s=CLOSED pgs=2099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:08.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.923+0000 7ff7cedc5640 1 --2- 172.21.15.125:0/1807393576 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff7c813e6c0 0x7ff7c81533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:08.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.923+0000 7ff7cedc5640 1 --2- 172.21.15.125:0/1807393576 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff7b0004d10 0x7ff7c81f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:08.929 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.927+0000 7ff7cedc5640 1 -- 172.21.15.125:0/1807393576 >> 172.21.15.125:0/1807393576 conn(0x7ff7c80c7910 msgr2=0x7ff7c80b08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:08.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.927+0000 7ff7cedc5640 1 -- 172.21.15.125:0/1807393576 shutdown_connections 2023-10-30T13:24:08.930 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:08.927+0000 7ff7cedc5640 1 -- 172.21.15.125:0/1807393576 wait complete. 2023-10-30T13:24:08.947 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:24:10.949 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:11.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.095+0000 7ffa6b0ea640 1 Processor -- start 2023-10-30T13:24:11.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.095+0000 7ffa6b0ea640 1 -- start start 2023-10-30T13:24:11.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa6b0ea640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7ffa640c6b80 0x7ffa640bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:11.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa6b0ea640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffa640c6f60 0x7ffa640c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:11.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa6b0ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa640c8430 0x7ffa6413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:11.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa6b0ea640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa640c0a50 con 0x7ffa640c6f60 2023-10-30T13:24:11.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa6b0ea640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa6406f6e0 con 0x7ffa640c6b80 2023-10-30T13:24:11.102 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa6b0ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6406f9c0 con 0x7ffa640c8430 2023-10-30T13:24:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa63fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffa640c6f60 0x7ffa640c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa63fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ffa640c6f60 0x7ffa640c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51030/0 (socket says 172.21.15.125:51030) 2023-10-30T13:24:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa63fff640 1 -- 172.21.15.125:0/3653676079 learned_addr learned my addr 172.21.15.125:0/3653676079 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa68e86640 1 --2- 172.21.15.125:0/3653676079 >> v2:172.21.15.136:3300/0 conn(0x7ffa640c6b80 0x7ffa640bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:11.103 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa69687640 1 --2- 172.21.15.125:0/3653676079 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa640c8430 0x7ffa6413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:11.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa63fff640 1 -- 172.21.15.125:0/3653676079 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa640c8430 msgr2=0x7ffa6413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:11.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa63fff640 1 --2- 172.21.15.125:0/3653676079 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa640c8430 0x7ffa6413ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:11.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa63fff640 1 -- 172.21.15.125:0/3653676079 >> v2:172.21.15.136:3300/0 conn(0x7ffa640c6b80 msgr2=0x7ffa640bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:11.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa63fff640 1 --2- 172.21.15.125:0/3653676079 >> v2:172.21.15.136:3300/0 conn(0x7ffa640c6b80 0x7ffa640bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:11.104 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.099+0000 7ffa63fff640 1 -- 172.21.15.125:0/3653676079 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6406fca0 con 0x7ffa640c6f60 2023-10-30T13:24:11.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa63fff640 1 --2- 172.21.15.125:0/3653676079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa640c6f60 0x7ffa640c0400 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7ffa6406f1b0 tx=0x7ffa5001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d974e0b42ac8c139 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:11.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa637fe640 1 -- 172.21.15.125:0/3653676079 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa500ac630 con 0x7ffa640c6f60 2023-10-30T13:24:11.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa637fe640 1 -- 172.21.15.125:0/3653676079 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa500ac790 con 0x7ffa640c6f60 2023-10-30T13:24:11.106 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa637fe640 1 -- 172.21.15.125:0/3653676079 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa500acb20 con 0x7ffa640c6f60 2023-10-30T13:24:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/3653676079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa640c6f60 msgr2=0x7ffa640c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa6b0ea640 1 --2- 172.21.15.125:0/3653676079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa640c6f60 0x7ffa640c0400 secure :-1 s=READY pgs=2100 cs=0 l=1 rev1=1 crypto rx=0x7ffa6406f1b0 tx=0x7ffa5001c500 comp rx=0 tx=0).stop 2023-10-30T13:24:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/3653676079 shutdown_connections 2023-10-30T13:24:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa6b0ea640 1 --2- 172.21.15.125:0/3653676079 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa640c8430 0x7ffa6413ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:11.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa6b0ea640 1 --2- 172.21.15.125:0/3653676079 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa640c6f60 0x7ffa640c0400 unknown :-1 s=CLOSED pgs=2100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:11.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa6b0ea640 1 --2- 172.21.15.125:0/3653676079 >> v2:172.21.15.136:3300/0 conn(0x7ffa640c6b80 0x7ffa640bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:11.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/3653676079 >> 172.21.15.125:0/3653676079 conn(0x7ffa640c7910 msgr2=0x7ffa640c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:11.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/3653676079 shutdown_connections 2023-10-30T13:24:11.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/3653676079 wait complete. 2023-10-30T13:24:11.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa6b0ea640 1 Processor -- start 2023-10-30T13:24:11.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.103+0000 7ffa6b0ea640 1 -- start start 2023-10-30T13:24:11.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa6b0ea640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa640c6b80 0x7ffa640ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:11.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa6b0ea640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa640c6f60 0x7ffa64163580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:11.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa6b0ea640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa640c8430 0x7ffa64163ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:11.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa6b0ea640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6413eee0 con 0x7ffa640c6b80 2023-10-30T13:24:11.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa6b0ea640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6415a280 con 0x7ffa640c6f60 2023-10-30T13:24:11.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa6b0ea640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa64138980 con 0x7ffa640c8430 2023-10-30T13:24:11.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa68e86640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa640c6b80 0x7ffa640ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:11.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa68e86640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa640c6b80 0x7ffa640ca590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51040/0 (socket says 172.21.15.125:51040) 2023-10-30T13:24:11.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa68e86640 1 -- 172.21.15.125:0/1673739134 learned_addr learned my addr 172.21.15.125:0/1673739134 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:11.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa63fff640 1 --2- 172.21.15.125:0/1673739134 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa640c6f60 0x7ffa64163580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:11.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa69687640 1 --2- 172.21.15.125:0/1673739134 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa640c8430 0x7ffa64163ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:11.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa68e86640 1 -- 172.21.15.125:0/1673739134 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa640c8430 msgr2=0x7ffa64163ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:11.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa68e86640 1 --2- 172.21.15.125:0/1673739134 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa640c8430 0x7ffa64163ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:11.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa68e86640 1 -- 172.21.15.125:0/1673739134 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa640c6f60 msgr2=0x7ffa64163580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:11.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa68e86640 1 --2- 172.21.15.125:0/1673739134 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa640c6f60 0x7ffa64163580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:11.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa68e86640 1 -- 172.21.15.125:0/1673739134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa500094d0 con 0x7ffa640c6b80 2023-10-30T13:24:11.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa68e86640 1 --2- 172.21.15.125:0/1673739134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa640c6b80 0x7ffa640ca590 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800d830 tx=0x7ffa5800dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:11.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa617fa640 1 -- 172.21.15.125:0/1673739134 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa58009640 con 0x7ffa640c6b80 2023-10-30T13:24:11.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa617fa640 1 -- 172.21.15.125:0/1673739134 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa580097a0 con 0x7ffa640c6b80 2023-10-30T13:24:11.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa617fa640 1 -- 172.21.15.125:0/1673739134 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa58009b10 con 0x7ffa640c6b80 2023-10-30T13:24:11.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/1673739134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa64137b20 con 0x7ffa640c6b80 2023-10-30T13:24:11.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/1673739134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa641380e0 con 0x7ffa640c6b80 2023-10-30T13:24:11.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/1673739134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6404e170 con 0x7ffa640c6b80 2023-10-30T13:24:11.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.107+0000 7ffa617fa640 1 -- 172.21.15.125:0/1673739134 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ffa58014410 con 0x7ffa640c6b80 2023-10-30T13:24:11.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.111+0000 7ffa617fa640 1 --2- 172.21.15.125:0/1673739134 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffa480664a0 0x7ffa48068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:11.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.111+0000 7ffa617fa640 1 -- 172.21.15.125:0/1673739134 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7ffa58085270 con 0x7ffa640c6b80 2023-10-30T13:24:11.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.111+0000 7ffa63fff640 1 --2- 172.21.15.125:0/1673739134 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffa480664a0 0x7ffa48068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:11.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.111+0000 7ffa63fff640 1 --2- 172.21.15.125:0/1673739134 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffa480664a0 0x7ffa48068960 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ffa500ae000 tx=0x7ffa50008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:11.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.115+0000 7ffa617fa640 1 -- 172.21.15.125:0/1673739134 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ffa58051c70 con 0x7ffa640c6b80 2023-10-30T13:24:11.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.295+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/1673739134 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa640c0dd0 con 0x7ffa640c6b80 2023-10-30T13:24:11.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.295+0000 7ffa617fa640 1 -- 172.21.15.125:0/1673739134 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+6554 (secure 0 0 0) 0x7ffa58055290 con 0x7ffa640c6b80 2023-10-30T13:24:11.298 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:11.298 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9409,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":253}],"filesystems":[{"mdsmap":{"epoch":258,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:23:34.832807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:active","state_seq":19,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:11.298 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 258 2023-10-30T13:24:11.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.295+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/1673739134 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffa480664a0 msgr2=0x7ffa48068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:11.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.295+0000 7ffa6b0ea640 1 --2- 172.21.15.125:0/1673739134 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffa480664a0 0x7ffa48068960 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ffa500ae000 tx=0x7ffa50008040 comp rx=0 tx=0).stop 2023-10-30T13:24:11.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.295+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/1673739134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa640c6b80 msgr2=0x7ffa640ca590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:11.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.295+0000 7ffa6b0ea640 1 --2- 172.21.15.125:0/1673739134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa640c6b80 0x7ffa640ca590 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7ffa5800d830 tx=0x7ffa5800dcf0 comp rx=0 tx=0).stop 2023-10-30T13:24:11.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.295+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/1673739134 shutdown_connections 2023-10-30T13:24:11.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.295+0000 7ffa6b0ea640 1 --2- 172.21.15.125:0/1673739134 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ffa480664a0 0x7ffa48068960 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:11.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.295+0000 7ffa6b0ea640 1 --2- 172.21.15.125:0/1673739134 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ffa640c8430 0x7ffa64163ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:11.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.295+0000 7ffa6b0ea640 1 --2- 172.21.15.125:0/1673739134 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ffa640c6f60 0x7ffa64163580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:11.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.295+0000 7ffa6b0ea640 1 --2- 172.21.15.125:0/1673739134 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ffa640c6b80 0x7ffa640ca590 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:11.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.295+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/1673739134 >> 172.21.15.125:0/1673739134 conn(0x7ffa640c7910 msgr2=0x7ffa6406a5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:11.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.295+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/1673739134 shutdown_connections 2023-10-30T13:24:11.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:11.295+0000 7ffa6b0ea640 1 -- 172.21.15.125:0/1673739134 wait complete. 2023-10-30T13:24:11.319 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:24:13.320 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:13.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b8cc9640 1 Processor -- start 2023-10-30T13:24:13.465 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b8cc9640 1 -- start start 2023-10-30T13:24:13.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b8cc9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8b406f7a0 0x7fd8b406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:13.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b8cc9640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fd8b40c6bd0 0x7fd8b4069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:13.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b8cc9640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b406a3d0 con 0x7fd8b4071110 2023-10-30T13:24:13.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b8cc9640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b40700c0 con 0x7fd8b40c6bd0 2023-10-30T13:24:13.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b8cc9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b4066cf0 con 0x7fd8b406f7a0 2023-10-30T13:24:13.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b2d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fd8b4071110 0x7fd8b4158110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36510/0 (socket says 172.21.15.125:36510) 2023-10-30T13:24:13.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b2d9d640 1 -- 172.21.15.125:0/4114247845 learned_addr learned my addr 172.21.15.125:0/4114247845 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:24:13.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b259c640 1 --2- 172.21.15.125:0/4114247845 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8b406f7a0 0x7fd8b406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:13.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8a9d9b640 1 --2- 172.21.15.125:0/4114247845 >> v2:172.21.15.136:3300/0 conn(0x7fd8b40c6bd0 0x7fd8b4069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:13.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 -- 172.21.15.125:0/4114247845 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 323606342 0 0) 0x7fd8b406a3d0 con 0x7fd8b4071110 2023-10-30T13:24:13.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 -- 172.21.15.125:0/4114247845 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a0003530 con 0x7fd8b4071110 2023-10-30T13:24:13.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 -- 172.21.15.125:0/4114247845 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1627046959 0 0) 0x7fd8a0003530 con 0x7fd8b4071110 2023-10-30T13:24:13.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 -- 172.21.15.125:0/4114247845 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8b406f7a0 msgr2=0x7fd8b406fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:13.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 --2- 172.21.15.125:0/4114247845 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8b406f7a0 0x7fd8b406fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:13.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 -- 172.21.15.125:0/4114247845 >> v2:172.21.15.136:3300/0 conn(0x7fd8b40c6bd0 msgr2=0x7fd8b4069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:13.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 --2- 172.21.15.125:0/4114247845 >> v2:172.21.15.136:3300/0 conn(0x7fd8b40c6bd0 0x7fd8b4069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:13.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 -- 172.21.15.125:0/4114247845 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b40c70d0 con 0x7fd8b4071110 2023-10-30T13:24:13.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 -- 172.21.15.125:0/4114247845 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fd8ac001680 con 0x7fd8b4071110 2023-10-30T13:24:13.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 -- 172.21.15.125:0/4114247845 >> v1:172.21.15.125:6789/0 conn(0x7fd8b4071110 legacy=0x7fd8b4158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:13.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 --2- 172.21.15.125:0/4114247845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8a0003980 0x7fd8a0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:13.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 --2- 172.21.15.125:0/4114247845 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8b406f7a0 0x7fd8a00047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:13.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 --2- 172.21.15.125:0/4114247845 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8a0004d10 0x7fd8a0005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:13.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 -- 172.21.15.125:0/4114247845 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b40700c0 con 0x7fd8a0003980 2023-10-30T13:24:13.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 -- 172.21.15.125:0/4114247845 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b4066cf0 con 0x7fd8a0004d10 2023-10-30T13:24:13.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b1d9b640 1 -- 172.21.15.125:0/4114247845 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a000b990 con 0x7fd8b406f7a0 2023-10-30T13:24:13.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b259c640 1 --2- 172.21.15.125:0/4114247845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8a0003980 0x7fd8a0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:13.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8a9d9b640 1 --2- 172.21.15.125:0/4114247845 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8b406f7a0 0x7fd8a00047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:13.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b2d9d640 1 --2- 172.21.15.125:0/4114247845 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8a0004d10 0x7fd8a0005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:13.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b259c640 1 -- 172.21.15.125:0/4114247845 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8b406f7a0 msgr2=0x7fd8a00047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:13.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b259c640 1 --2- 172.21.15.125:0/4114247845 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8b406f7a0 0x7fd8a00047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:13.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b259c640 1 -- 172.21.15.125:0/4114247845 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8a0004d10 msgr2=0x7fd8a0005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:13.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b259c640 1 --2- 172.21.15.125:0/4114247845 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8a0004d10 0x7fd8a0005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:13.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.463+0000 7fd8b259c640 1 -- 172.21.15.125:0/4114247845 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a0003530 con 0x7fd8a0003980 2023-10-30T13:24:13.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.467+0000 7fd8b259c640 1 --2- 172.21.15.125:0/4114247845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8a0003980 0x7fd8a0003da0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009fc0 tx=0x7fd8a401cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5304ab9b632b8934 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:13.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.467+0000 7fd8b1d9b640 1 -- 172.21.15.125:0/4114247845 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a40a5b60 con 0x7fd8a0003980 2023-10-30T13:24:13.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.467+0000 7fd8b1d9b640 1 -- 172.21.15.125:0/4114247845 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a400f040 con 0x7fd8a0003980 2023-10-30T13:24:13.475 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.467+0000 7fd8b1d9b640 1 -- 172.21.15.125:0/4114247845 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a40ad440 con 0x7fd8a0003980 2023-10-30T13:24:13.478 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.467+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/4114247845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8a0003980 msgr2=0x7fd8a0003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:13.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.467+0000 7fd8b8cc9640 1 --2- 172.21.15.125:0/4114247845 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8a0003980 0x7fd8a0003da0 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009fc0 tx=0x7fd8a401cc60 comp rx=0 tx=0).stop 2023-10-30T13:24:13.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.467+0000 7fd8b2d9d640 1 -- 172.21.15.125:0/4114247845 reap_dead start 2023-10-30T13:24:13.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.467+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/4114247845 shutdown_connections 2023-10-30T13:24:13.479 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.467+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/4114247845 >> 172.21.15.125:0/4114247845 conn(0x7fd8b40c7910 msgr2=0x7fd8b40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:13.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.467+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/4114247845 shutdown_connections 2023-10-30T13:24:13.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.467+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/4114247845 wait complete. 2023-10-30T13:24:13.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8b8cc9640 1 Processor -- start 2023-10-30T13:24:13.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8b8cc9640 1 -- start start 2023-10-30T13:24:13.480 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8b8cc9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8a0003980 0x7fd8b414fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:13.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8b8cc9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b414edb0 0x7fd8b41554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:13.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8b8cc9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8b4157c20 0x7fd8b41580a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:13.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8b8cc9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b4150200 con 0x7fd8b414edb0 2023-10-30T13:24:13.481 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8b8cc9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b415b800 con 0x7fd8b4157c20 2023-10-30T13:24:13.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8b8cc9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b414e920 con 0x7fd8a0003980 2023-10-30T13:24:13.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8a9d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b414edb0 0x7fd8b41554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:13.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8a9d9b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b414edb0 0x7fd8b41554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51044/0 (socket says 172.21.15.125:51044) 2023-10-30T13:24:13.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8a9d9b640 1 -- 172.21.15.125:0/1050172844 learned_addr learned my addr 172.21.15.125:0/1050172844 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:13.482 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8b259c640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8a0003980 0x7fd8b414fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:13.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8b2d9d640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8b4157c20 0x7fd8b41580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:13.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8a9d9b640 1 -- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8a0003980 msgr2=0x7fd8b414fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:13.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8a9d9b640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8a0003980 0x7fd8b414fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:13.483 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8a9d9b640 1 -- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8b4157c20 msgr2=0x7fd8b41580a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:13.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8a9d9b640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8b4157c20 0x7fd8b41580a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:13.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8a9d9b640 1 -- 172.21.15.125:0/1050172844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b40c4810 con 0x7fd8b414edb0 2023-10-30T13:24:13.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8b2d9d640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8b4157c20 0x7fd8b41580a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:13.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8a9d9b640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b414edb0 0x7fd8b41554e0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7fd89c011660 tx=0x7fd89c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:13.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8ab7fe640 1 -- 172.21.15.125:0/1050172844 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c00f960 con 0x7fd8b414edb0 2023-10-30T13:24:13.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8ab7fe640 1 -- 172.21.15.125:0/1050172844 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd89c00fac0 con 0x7fd8b414edb0 2023-10-30T13:24:13.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/1050172844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b415ba30 con 0x7fd8b414edb0 2023-10-30T13:24:13.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8ab7fe640 1 -- 172.21.15.125:0/1050172844 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd89c00fe30 con 0x7fd8b414edb0 2023-10-30T13:24:13.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.471+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/1050172844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b414e2d0 con 0x7fd8b414edb0 2023-10-30T13:24:13.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.475+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/1050172844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b404e170 con 0x7fd8b414edb0 2023-10-30T13:24:13.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.475+0000 7fd8ab7fe640 1 -- 172.21.15.125:0/1050172844 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fd89c015460 con 0x7fd8b414edb0 2023-10-30T13:24:13.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.475+0000 7fd8ab7fe640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8940664d0 0x7fd894068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:13.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.475+0000 7fd8ab7fe640 1 -- 172.21.15.125:0/1050172844 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7fd89c0850b0 con 0x7fd8b414edb0 2023-10-30T13:24:13.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.475+0000 7fd8b259c640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8940664d0 0x7fd894068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:13.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.475+0000 7fd8b259c640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8940664d0 0x7fd894068990 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fd8a40af000 tx=0x7fd8a4004ac0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:13.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.479+0000 7fd8ab7fe640 1 -- 172.21.15.125:0/1050172844 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fd89c051ab0 con 0x7fd8b414edb0 2023-10-30T13:24:13.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.659+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/1050172844 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd8b414ea60 con 0x7fd8b414edb0 2023-10-30T13:24:13.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.659+0000 7fd8ab7fe640 1 -- 172.21.15.125:0/1050172844 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 258 v258) v1 ==== 95+0+6554 (secure 0 0 0) 0x7fd89c0550d0 con 0x7fd8b414edb0 2023-10-30T13:24:13.663 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:13.663 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":258,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":9409,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":253}],"filesystems":[{"mdsmap":{"epoch":258,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:23:34.832807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":8033,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8033":{"gid":8033,"name":"d","rank":0,"incarnation":248,"state":"up:active","state_seq":19,"addr":"172.21.15.136:6835/3480968625","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":3480968625},{"type":"v1","addr":"172.21.15.136:6835","nonce":3480968625}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:13.664 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 258 2023-10-30T13:24:13.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.663+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8940664d0 msgr2=0x7fd894068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:13.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.663+0000 7fd8b8cc9640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8940664d0 0x7fd894068990 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fd8a40af000 tx=0x7fd8a4004ac0 comp rx=0 tx=0).stop 2023-10-30T13:24:13.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.663+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/1050172844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b414edb0 msgr2=0x7fd8b41554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:13.665 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.663+0000 7fd8b8cc9640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b414edb0 0x7fd8b41554e0 secure :-1 s=READY pgs=2104 cs=0 l=1 rev1=1 crypto rx=0x7fd89c011660 tx=0x7fd89c011b20 comp rx=0 tx=0).stop 2023-10-30T13:24:13.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.663+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/1050172844 shutdown_connections 2023-10-30T13:24:13.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.663+0000 7fd8b8cc9640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fd8940664d0 0x7fd894068990 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:13.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.663+0000 7fd8b8cc9640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fd8b4157c20 0x7fd8b41580a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:13.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.663+0000 7fd8b8cc9640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fd8b414edb0 0x7fd8b41554e0 unknown :-1 s=CLOSED pgs=2104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:13.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.663+0000 7fd8b8cc9640 1 --2- 172.21.15.125:0/1050172844 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fd8a0003980 0x7fd8b414fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:13.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.663+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/1050172844 >> 172.21.15.125:0/1050172844 conn(0x7fd8b40c7910 msgr2=0x7fd8b40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:13.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.663+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/1050172844 shutdown_connections 2023-10-30T13:24:13.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:13.663+0000 7fd8b8cc9640 1 -- 172.21.15.125:0/1050172844 wait complete. 2023-10-30T13:24:13.684 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is laggy/crashed, in failed state, or mds.d is removed from mdsmap 2023-10-30T13:24:15.685 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:15.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f648c93e640 1 Processor -- start 2023-10-30T13:24:15.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f648c93e640 1 -- start start 2023-10-30T13:24:15.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f648c93e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f64881538a0 0x7f648815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:15.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f648c93e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64880cb800 con 0x7f64880ca0e0 2023-10-30T13:24:15.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f648c93e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f648815dc30 con 0x7f64880cf160 2023-10-30T13:24:15.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f648c93e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64880cb330 con 0x7f64881538a0 2023-10-30T13:24:15.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f647ffff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f64880ca0e0 0x7f648815a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36524/0 (socket says 172.21.15.125:36524) 2023-10-30T13:24:15.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f647ffff640 1 -- 172.21.15.125:0/2395788587 learned_addr learned my addr 172.21.15.125:0/2395788587 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:24:15.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f648659c640 1 --2- 172.21.15.125:0/2395788587 >> v2:172.21.15.136:3300/0 conn(0x7f64881538a0 0x7f648815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:15.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 -- 172.21.15.125:0/2395788587 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763913086 0 0) 0x7f64880cb800 con 0x7f64880ca0e0 2023-10-30T13:24:15.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 -- 172.21.15.125:0/2395788587 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6474003530 con 0x7f64880ca0e0 2023-10-30T13:24:15.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 -- 172.21.15.125:0/2395788587 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1983605723 0 0) 0x7f6474003530 con 0x7f64880ca0e0 2023-10-30T13:24:15.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 -- 172.21.15.125:0/2395788587 >> v2:172.21.15.136:3300/0 conn(0x7f64881538a0 msgr2=0x7f648815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:15.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 --2- 172.21.15.125:0/2395788587 >> v2:172.21.15.136:3300/0 conn(0x7f64881538a0 0x7f648815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:15.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 -- 172.21.15.125:0/2395788587 >> v1:172.21.15.136:6789/0 conn(0x7f64880cf160 legacy=0x7f6488164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:15.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 -- 172.21.15.125:0/2395788587 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64880cad70 con 0x7f64880ca0e0 2023-10-30T13:24:15.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 -- 172.21.15.125:0/2395788587 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f6470001680 con 0x7f64880ca0e0 2023-10-30T13:24:15.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 -- 172.21.15.125:0/2395788587 >> v1:172.21.15.125:6789/0 conn(0x7f64880ca0e0 legacy=0x7f648815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:15.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 --2- 172.21.15.125:0/2395788587 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6474003980 0x7f6474003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:15.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 --2- 172.21.15.125:0/2395788587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64740047d0 0x7f6474004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:15.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 --2- 172.21.15.125:0/2395788587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6474005190 0x7f647400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:15.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 -- 172.21.15.125:0/2395788587 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64880cb330 con 0x7f64740047d0 2023-10-30T13:24:15.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 -- 172.21.15.125:0/2395788587 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647400c040 con 0x7f6474005190 2023-10-30T13:24:15.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f6485d9b640 1 -- 172.21.15.125:0/2395788587 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647400c200 con 0x7f6474003980 2023-10-30T13:24:15.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.831+0000 7f647ffff640 1 --2- 172.21.15.125:0/2395788587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64740047d0 0x7f6474004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:15.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f6486d9d640 1 --2- 172.21.15.125:0/2395788587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6474005190 0x7f647400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:15.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f648659c640 1 --2- 172.21.15.125:0/2395788587 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6474003980 0x7f6474003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:15.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f647ffff640 1 -- 172.21.15.125:0/2395788587 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6474003980 msgr2=0x7f6474003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:15.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f647ffff640 1 --2- 172.21.15.125:0/2395788587 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6474003980 0x7f6474003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:15.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f647ffff640 1 -- 172.21.15.125:0/2395788587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6474005190 msgr2=0x7f647400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:15.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f647ffff640 1 --2- 172.21.15.125:0/2395788587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6474005190 0x7f647400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:15.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f647ffff640 1 -- 172.21.15.125:0/2395788587 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6474003530 con 0x7f64740047d0 2023-10-30T13:24:15.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f6486d9d640 1 --2- 172.21.15.125:0/2395788587 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6474005190 0x7f647400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:15.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f6486d9d640 1 -- 172.21.15.125:0/2395788587 reap_dead start 2023-10-30T13:24:15.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f647ffff640 1 --2- 172.21.15.125:0/2395788587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64740047d0 0x7f6474004c50 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f647000c1c0 tx=0x7f647001cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4a99e98654bfae server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:15.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f6485d9b640 1 -- 172.21.15.125:0/2395788587 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64700a5b20 con 0x7f64740047d0 2023-10-30T13:24:15.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f6485d9b640 1 -- 172.21.15.125:0/2395788587 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64700a5c80 con 0x7f64740047d0 2023-10-30T13:24:15.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f6485d9b640 1 -- 172.21.15.125:0/2395788587 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470005b20 con 0x7f64740047d0 2023-10-30T13:24:15.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f648c93e640 1 -- 172.21.15.125:0/2395788587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64740047d0 msgr2=0x7f6474004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:15.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.835+0000 7f648c93e640 1 --2- 172.21.15.125:0/2395788587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64740047d0 0x7f6474004c50 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f647000c1c0 tx=0x7f647001cc20 comp rx=0 tx=0).stop 2023-10-30T13:24:15.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f648c93e640 1 -- 172.21.15.125:0/2395788587 shutdown_connections 2023-10-30T13:24:15.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f648c93e640 1 --2- 172.21.15.125:0/2395788587 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64740047d0 0x7f6474004c50 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:15.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f648c93e640 1 -- 172.21.15.125:0/2395788587 >> 172.21.15.125:0/2395788587 conn(0x7f648806bfe0 msgr2=0x7f648806c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:15.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f648c93e640 1 -- 172.21.15.125:0/2395788587 shutdown_connections 2023-10-30T13:24:15.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f648c93e640 1 -- 172.21.15.125:0/2395788587 wait complete. 2023-10-30T13:24:15.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f648c93e640 1 Processor -- start 2023-10-30T13:24:15.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f648c93e640 1 -- start start 2023-10-30T13:24:15.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f648c93e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64740047d0 0x7f64880b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:15.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f648c93e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6488158300 0x7f64880b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:15.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f648c93e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64880c90d0 0x7f64880c48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:15.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f648c93e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64880b5000 con 0x7f64880c90d0 2023-10-30T13:24:15.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f648c93e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648815d920 con 0x7f64740047d0 2023-10-30T13:24:15.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f648c93e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64881f2530 con 0x7f6488158300 2023-10-30T13:24:15.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f6486d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64880c90d0 0x7f64880c48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:15.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f6486d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64880c90d0 0x7f64880c48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51070/0 (socket says 172.21.15.125:51070) 2023-10-30T13:24:15.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f6486d9d640 1 -- 172.21.15.125:0/1840905794 learned_addr learned my addr 172.21.15.125:0/1840905794 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:15.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f647ffff640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6488158300 0x7f64880b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:15.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f648659c640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64740047d0 0x7f64880b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:15.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f6486d9d640 1 -- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6488158300 msgr2=0x7f64880b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:15.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f6486d9d640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6488158300 0x7f64880b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:15.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f6486d9d640 1 -- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64740047d0 msgr2=0x7f64880b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:15.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f6486d9d640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64740047d0 0x7f64880b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:15.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f6486d9d640 1 -- 172.21.15.125:0/1840905794 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64881f2810 con 0x7f64880c90d0 2023-10-30T13:24:15.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f647ffff640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6488158300 0x7f64880b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:15.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.839+0000 7f6486d9d640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64880c90d0 0x7f64880c48e0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f6488072ce0 tx=0x7f648000e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:15.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.843+0000 7f647effd640 1 -- 172.21.15.125:0/1840905794 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6480003b90 con 0x7f64880c90d0 2023-10-30T13:24:15.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.843+0000 7f647effd640 1 -- 172.21.15.125:0/1840905794 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6480003cf0 con 0x7f64880c90d0 2023-10-30T13:24:15.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.843+0000 7f647effd640 1 -- 172.21.15.125:0/1840905794 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64800052f0 con 0x7f64880c90d0 2023-10-30T13:24:15.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.843+0000 7f648c93e640 1 -- 172.21.15.125:0/1840905794 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64881f2af0 con 0x7f64880c90d0 2023-10-30T13:24:15.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.843+0000 7f648c93e640 1 -- 172.21.15.125:0/1840905794 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64881f30b0 con 0x7f64880c90d0 2023-10-30T13:24:15.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.843+0000 7f648c93e640 1 -- 172.21.15.125:0/1840905794 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f648804e170 con 0x7f64880c90d0 2023-10-30T13:24:15.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.843+0000 7f647effd640 1 -- 172.21.15.125:0/1840905794 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6480005450 con 0x7f64880c90d0 2023-10-30T13:24:15.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.843+0000 7f647effd640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64680664d0 0x7f6468068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:15.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.843+0000 7f647effd640 1 -- 172.21.15.125:0/1840905794 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8318+0+0 (secure 0 0 0) 0x7f6480083c30 con 0x7f64880c90d0 2023-10-30T13:24:15.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.843+0000 7f648659c640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64680664d0 0x7f6468068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:15.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.847+0000 7f648659c640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64680664d0 0x7f6468068990 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f6488069010 tx=0x7f6478008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:15.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:15.847+0000 7f647effd640 1 -- 172.21.15.125:0/1840905794 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6480050660 con 0x7f64880c90d0 2023-10-30T13:24:16.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.031+0000 7f648c93e640 1 -- 172.21.15.125:0/1840905794 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f648815dc30 con 0x7f64880c90d0 2023-10-30T13:24:16.035 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.031+0000 7f647effd640 1 -- 172.21.15.125:0/1840905794 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f648815dc30 con 0x7f64880c90d0 2023-10-30T13:24:16.035 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:16.035 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":259,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:15.942055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:16.036 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 259 2023-10-30T13:24:16.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.035+0000 7f648c93e640 1 -- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64680664d0 msgr2=0x7f6468068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:16.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.035+0000 7f648c93e640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64680664d0 0x7f6468068990 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f6488069010 tx=0x7f6478008040 comp rx=0 tx=0).stop 2023-10-30T13:24:16.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.035+0000 7f648c93e640 1 -- 172.21.15.125:0/1840905794 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64880c90d0 msgr2=0x7f64880c48e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:16.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.035+0000 7f648c93e640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64880c90d0 0x7f64880c48e0 secure :-1 s=READY pgs=2107 cs=0 l=1 rev1=1 crypto rx=0x7f6488072ce0 tx=0x7f648000e960 comp rx=0 tx=0).stop 2023-10-30T13:24:16.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.035+0000 7f648c93e640 1 -- 172.21.15.125:0/1840905794 shutdown_connections 2023-10-30T13:24:16.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.035+0000 7f648c93e640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f64680664d0 0x7f6468068990 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.035+0000 7f648c93e640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f64880c90d0 0x7f64880c48e0 unknown :-1 s=CLOSED pgs=2107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.035+0000 7f648c93e640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6488158300 0x7f64880b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.035+0000 7f648c93e640 1 --2- 172.21.15.125:0/1840905794 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f64740047d0 0x7f64880b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.035+0000 7f648c93e640 1 -- 172.21.15.125:0/1840905794 >> 172.21.15.125:0/1840905794 conn(0x7f648806bfe0 msgr2=0x7f6488065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:16.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.035+0000 7f648c93e640 1 -- 172.21.15.125:0/1840905794 shutdown_connections 2023-10-30T13:24:16.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.035+0000 7f648c93e640 1 -- 172.21.15.125:0/1840905794 wait complete. 2023-10-30T13:24:16.056 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d down, removed from mdsmap 2023-10-30T13:24:16.056 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2023-10-30T13:24:16.056 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:16.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.191+0000 7f4dbc180640 1 Processor -- start 2023-10-30T13:24:16.194 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.191+0000 7f4dbc180640 1 -- start start 2023-10-30T13:24:16.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.191+0000 7f4dbc180640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4db415d6f0 0x7f4db415a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:16.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.191+0000 7f4dbc180640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4db40cf160 0x7f4db40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:16.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.191+0000 7f4dbc180640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db40cb800 con 0x7f4db41538a0 2023-10-30T13:24:16.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.191+0000 7f4dbc180640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db40cb330 con 0x7f4db415d6f0 2023-10-30T13:24:16.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.191+0000 7f4dbc180640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db40cad70 con 0x7f4db40cf160 2023-10-30T13:24:16.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.191+0000 7f4dba71d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4db40cf160 0x7f4db40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:16.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.191+0000 7f4db971b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f4db415d6f0 0x7f4db415a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:16.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.191+0000 7f4dba71d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4db40cf160 0x7f4db40cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:46612/0 (socket says 172.21.15.125:46612) 2023-10-30T13:24:16.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.191+0000 7f4dba71d640 1 -- 172.21.15.125:0/2855361163 learned_addr learned my addr 172.21.15.125:0/2855361163 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:16.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.195+0000 7f4db971b640 1 -- 172.21.15.125:0/2855361163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4db40cf160 msgr2=0x7f4db40cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:16.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.195+0000 7f4db971b640 1 --2- 172.21.15.125:0/2855361163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4db40cf160 0x7f4db40cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.195+0000 7f4db971b640 1 -- 172.21.15.125:0/2855361163 >> v1:172.21.15.136:6789/0 conn(0x7f4db41538a0 legacy=0x7f4db40ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:16.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.195+0000 7f4db971b640 1 -- 172.21.15.125:0/2855361163 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db40cb050 con 0x7f4db415d6f0 2023-10-30T13:24:16.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.195+0000 7f4dba71d640 1 --2- 172.21.15.125:0/2855361163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4db40cf160 0x7f4db40cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:24:16.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.195+0000 7f4db971b640 1 --2- 172.21.15.125:0/2855361163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db415d6f0 0x7f4db415a090 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4da40097c0 tx=0x7f4da401ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=157742abbb97a164 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:16.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.195+0000 7f4db8f1a640 1 -- 172.21.15.125:0/2855361163 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da40a8970 con 0x7f4db415d6f0 2023-10-30T13:24:16.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.195+0000 7f4db8f1a640 1 -- 172.21.15.125:0/2855361163 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da40a8ad0 con 0x7f4db415d6f0 2023-10-30T13:24:16.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.195+0000 7f4db8f1a640 1 -- 172.21.15.125:0/2855361163 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da40a8e60 con 0x7f4db415d6f0 2023-10-30T13:24:16.200 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.195+0000 7f4dbc180640 1 -- 172.21.15.125:0/2855361163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db415d6f0 msgr2=0x7f4db415a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:16.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.195+0000 7f4dbc180640 1 --2- 172.21.15.125:0/2855361163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db415d6f0 0x7f4db415a090 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4da40097c0 tx=0x7f4da401ec70 comp rx=0 tx=0).stop 2023-10-30T13:24:16.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 -- 172.21.15.125:0/2855361163 shutdown_connections 2023-10-30T13:24:16.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 --2- 172.21.15.125:0/2855361163 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4db40cf160 0x7f4db40cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 --2- 172.21.15.125:0/2855361163 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db415d6f0 0x7f4db415a090 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.201 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 -- 172.21.15.125:0/2855361163 >> 172.21.15.125:0/2855361163 conn(0x7f4db406bfe0 msgr2=0x7f4db406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:16.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 -- 172.21.15.125:0/2855361163 shutdown_connections 2023-10-30T13:24:16.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 -- 172.21.15.125:0/2855361163 wait complete. 2023-10-30T13:24:16.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 Processor -- start 2023-10-30T13:24:16.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 -- start start 2023-10-30T13:24:16.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db40cf160 0x7f4db40c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:16.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db41538a0 0x7f4db41e4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:16.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4db415d6f0 0x7f4db41f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:16.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db41e3960 con 0x7f4db41538a0 2023-10-30T13:24:16.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db4159120 con 0x7f4db40cf160 2023-10-30T13:24:16.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db41f3360 con 0x7f4db415d6f0 2023-10-30T13:24:16.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4db971b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db41538a0 0x7f4db41e4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:16.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4db971b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db41538a0 0x7f4db41e4c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51082/0 (socket says 172.21.15.125:51082) 2023-10-30T13:24:16.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4db971b640 1 -- 172.21.15.125:0/2090200128 learned_addr learned my addr 172.21.15.125:0/2090200128 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:16.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4db9f1c640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db40cf160 0x7f4db40c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:16.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dba71d640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4db415d6f0 0x7f4db41f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:16.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4db971b640 1 -- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4db415d6f0 msgr2=0x7f4db41f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:16.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4db971b640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4db415d6f0 0x7f4db41f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4db971b640 1 -- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db40cf160 msgr2=0x7f4db40c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:16.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4db971b640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db40cf160 0x7f4db40c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4db971b640 1 -- 172.21.15.125:0/2090200128 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dac009b30 con 0x7f4db41538a0 2023-10-30T13:24:16.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4db9f1c640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db40cf160 0x7f4db40c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:16.207 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4db971b640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db41538a0 0x7f4db41e4c70 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f4da401d320 tx=0x7f4da401ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:16.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4daa7fc640 1 -- 172.21.15.125:0/2090200128 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da40bf070 con 0x7f4db41538a0 2023-10-30T13:24:16.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4daa7fc640 1 -- 172.21.15.125:0/2090200128 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da40b1c00 con 0x7f4db41538a0 2023-10-30T13:24:16.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4dbc180640 1 -- 172.21.15.125:0/2090200128 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da4009420 con 0x7f4db41538a0 2023-10-30T13:24:16.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.199+0000 7f4daa7fc640 1 -- 172.21.15.125:0/2090200128 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da40b9410 con 0x7f4db41538a0 2023-10-30T13:24:16.208 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.203+0000 7f4dbc180640 1 -- 172.21.15.125:0/2090200128 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db41e3680 con 0x7f4db41538a0 2023-10-30T13:24:16.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.203+0000 7f4dbc180640 1 -- 172.21.15.125:0/2090200128 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db40b2a50 con 0x7f4db41538a0 2023-10-30T13:24:16.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.203+0000 7f4daa7fc640 1 -- 172.21.15.125:0/2090200128 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f4da40b7040 con 0x7f4db41538a0 2023-10-30T13:24:16.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.203+0000 7f4daa7fc640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4d980664a0 0x7f4d98068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:16.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.203+0000 7f4daa7fc640 1 -- 172.21.15.125:0/2090200128 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f4da4121be0 con 0x7f4db41538a0 2023-10-30T13:24:16.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.203+0000 7f4db9f1c640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4d980664a0 0x7f4d98068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:16.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.203+0000 7f4db9f1c640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4d980664a0 0x7f4d98068960 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f4db0001710 tx=0x7f4db0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:16.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.207+0000 7f4daa7fc640 1 -- 172.21.15.125:0/2090200128 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f4da40ee5b0 con 0x7f4db41538a0 2023-10-30T13:24:16.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.395+0000 7f4dbc180640 1 -- 172.21.15.125:0/2090200128 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4db404e170 con 0x7f4db41538a0 2023-10-30T13:24:16.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.395+0000 7f4daa7fc640 1 -- 172.21.15.125:0/2090200128 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f4da40b5020 con 0x7f4db41538a0 2023-10-30T13:24:16.399 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:16.399 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":259,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:15.942055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:16.399 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 259 2023-10-30T13:24:16.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.399+0000 7f4dbc180640 1 -- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4d980664a0 msgr2=0x7f4d98068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:16.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.399+0000 7f4dbc180640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4d980664a0 0x7f4d98068960 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f4db0001710 tx=0x7f4db0009040 comp rx=0 tx=0).stop 2023-10-30T13:24:16.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.399+0000 7f4dbc180640 1 -- 172.21.15.125:0/2090200128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db41538a0 msgr2=0x7f4db41e4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:16.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.399+0000 7f4dbc180640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db41538a0 0x7f4db41e4c70 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f4da401d320 tx=0x7f4da401ba10 comp rx=0 tx=0).stop 2023-10-30T13:24:16.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.399+0000 7f4dbc180640 1 -- 172.21.15.125:0/2090200128 shutdown_connections 2023-10-30T13:24:16.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.399+0000 7f4dbc180640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f4d980664a0 0x7f4d98068960 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.399+0000 7f4dbc180640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f4db415d6f0 0x7f4db41f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.399+0000 7f4dbc180640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f4db41538a0 0x7f4db41e4c70 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.399+0000 7f4dbc180640 1 --2- 172.21.15.125:0/2090200128 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f4db40cf160 0x7f4db40c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:16.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.399+0000 7f4dbc180640 1 -- 172.21.15.125:0/2090200128 >> 172.21.15.125:0/2090200128 conn(0x7f4db406bfe0 msgr2=0x7f4db41529c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:16.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.399+0000 7f4dbc180640 1 -- 172.21.15.125:0/2090200128 shutdown_connections 2023-10-30T13:24:16.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:16.399+0000 7f4dbc180640 1 -- 172.21.15.125:0/2090200128 wait complete. 2023-10-30T13:24:16.419 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 259, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 259, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 36 }, "gid_9409": { "addr": "172.21.15.136:6839/2056603754", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 2056603754, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 2056603754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9409, "incarnation": 259, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:replay", "state_seq": 1 }, "gid_9600": { "addr": "172.21.15.125:6836/2077416857", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2077416857, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2077416857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 9600, "incarnation": 241, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 42 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:24:15.942055+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9409, "mds_1": 9150, "mds_2": 9600 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:24:18.246 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:24:18.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:24:18.421 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:18.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.563+0000 7fb29fed6640 1 Processor -- start 2023-10-30T13:24:18.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.563+0000 7fb29fed6640 1 -- start start 2023-10-30T13:24:18.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.567+0000 7fb29fed6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb2980c5f60 0x7fb298071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:18.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.567+0000 7fb29fed6640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2980700b0 con 0x7fb2980c6fe0 2023-10-30T13:24:18.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.567+0000 7fb29fed6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2980c6530 con 0x7fb298153bf0 2023-10-30T13:24:18.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.567+0000 7fb29fed6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2980c6810 con 0x7fb2980c5f60 2023-10-30T13:24:18.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.567+0000 7fb29d471640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb2980c6fe0 0x7fb2980cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:36540/0 (socket says 172.21.15.125:36540) 2023-10-30T13:24:18.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.567+0000 7fb29d471640 1 -- 172.21.15.125:0/2942623647 learned_addr learned my addr 172.21.15.125:0/2942623647 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:24:18.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.567+0000 7fb29e473640 1 --2- 172.21.15.125:0/2942623647 >> v2:172.21.15.125:3300/0 conn(0x7fb2980c5f60 0x7fb298071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:18.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.567+0000 7fb29cc70640 1 -- 172.21.15.125:0/2942623647 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918306205 0 0) 0x7fb2980700b0 con 0x7fb2980c6fe0 2023-10-30T13:24:18.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.567+0000 7fb29cc70640 1 -- 172.21.15.125:0/2942623647 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb284003530 con 0x7fb2980c6fe0 2023-10-30T13:24:18.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.567+0000 7fb29e473640 1 -- 172.21.15.125:0/2942623647 >> v1:172.21.15.136:6789/0 conn(0x7fb298153bf0 legacy=0x7fb2980c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:18.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.567+0000 7fb29e473640 1 -- 172.21.15.125:0/2942623647 >> v1:172.21.15.125:6789/0 conn(0x7fb2980c6fe0 legacy=0x7fb2980cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:18.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.567+0000 7fb29e473640 1 -- 172.21.15.125:0/2942623647 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb298159b10 con 0x7fb2980c5f60 2023-10-30T13:24:18.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.571+0000 7fb29e473640 1 --2- 172.21.15.125:0/2942623647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2980c5f60 0x7fb298071260 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fb290009d90 tx=0x7fb29001f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c433e0e7468cc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:18.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.571+0000 7fb29cc70640 1 -- 172.21.15.125:0/2942623647 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2900af600 con 0x7fb2980c5f60 2023-10-30T13:24:18.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.571+0000 7fb29cc70640 1 -- 172.21.15.125:0/2942623647 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2900ab050 con 0x7fb2980c5f60 2023-10-30T13:24:18.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.571+0000 7fb29cc70640 1 -- 172.21.15.125:0/2942623647 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2900af980 con 0x7fb2980c5f60 2023-10-30T13:24:18.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.571+0000 7fb29fed6640 1 -- 172.21.15.125:0/2942623647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2980c5f60 msgr2=0x7fb298071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.571+0000 7fb29fed6640 1 --2- 172.21.15.125:0/2942623647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2980c5f60 0x7fb298071260 secure :-1 s=READY pgs=2110 cs=0 l=1 rev1=1 crypto rx=0x7fb290009d90 tx=0x7fb29001f130 comp rx=0 tx=0).stop 2023-10-30T13:24:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.571+0000 7fb29fed6640 1 -- 172.21.15.125:0/2942623647 shutdown_connections 2023-10-30T13:24:18.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.571+0000 7fb29fed6640 1 --2- 172.21.15.125:0/2942623647 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2980c5f60 0x7fb298071260 unknown :-1 s=CLOSED pgs=2110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:18.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.571+0000 7fb29fed6640 1 -- 172.21.15.125:0/2942623647 >> 172.21.15.125:0/2942623647 conn(0x7fb29806c6e0 msgr2=0x7fb298069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:18.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.571+0000 7fb29fed6640 1 -- 172.21.15.125:0/2942623647 shutdown_connections 2023-10-30T13:24:18.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.571+0000 7fb29fed6640 1 -- 172.21.15.125:0/2942623647 wait complete. 2023-10-30T13:24:18.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.571+0000 7fb29fed6640 1 Processor -- start 2023-10-30T13:24:18.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.571+0000 7fb29fed6640 1 -- start start 2023-10-30T13:24:18.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29fed6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2980c5f60 0x7fb2981f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:18.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29fed6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2980c6fe0 0x7fb2981583f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:18.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29fed6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb298153bf0 0x7fb29814f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:18.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29fed6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29814b880 con 0x7fb298153bf0 2023-10-30T13:24:18.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29fed6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2981f3050 con 0x7fb2980c6fe0 2023-10-30T13:24:18.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29fed6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb29814b070 con 0x7fb2980c5f60 2023-10-30T13:24:18.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29e473640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb298153bf0 0x7fb29814f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:18.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29e473640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb298153bf0 0x7fb29814f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:51094/0 (socket says 172.21.15.125:51094) 2023-10-30T13:24:18.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29e473640 1 -- 172.21.15.125:0/669336754 learned_addr learned my addr 172.21.15.125:0/669336754 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:18.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29d471640 1 --2- 172.21.15.125:0/669336754 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2980c6fe0 0x7fb2981583f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:18.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29dc72640 1 --2- 172.21.15.125:0/669336754 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2980c5f60 0x7fb2981f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:18.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29e473640 1 -- 172.21.15.125:0/669336754 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2980c5f60 msgr2=0x7fb2981f2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:18.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29e473640 1 --2- 172.21.15.125:0/669336754 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2980c5f60 0x7fb2981f2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:18.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29e473640 1 -- 172.21.15.125:0/669336754 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2980c6fe0 msgr2=0x7fb2981583f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:18.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29e473640 1 --2- 172.21.15.125:0/669336754 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2980c6fe0 0x7fb2981583f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:18.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29e473640 1 -- 172.21.15.125:0/669336754 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb290009a70 con 0x7fb298153bf0 2023-10-30T13:24:18.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29e473640 1 --2- 172.21.15.125:0/669336754 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb298153bf0 0x7fb29814f530 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fb2900b2040 tx=0x7fb29001bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:18.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb28e7fc640 1 -- 172.21.15.125:0/669336754 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb290009450 con 0x7fb298153bf0 2023-10-30T13:24:18.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb28e7fc640 1 -- 172.21.15.125:0/669336754 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2900b9410 con 0x7fb298153bf0 2023-10-30T13:24:18.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb28e7fc640 1 -- 172.21.15.125:0/669336754 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2900b97c0 con 0x7fb298153bf0 2023-10-30T13:24:18.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29fed6640 1 -- 172.21.15.125:0/669336754 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb298157f60 con 0x7fb298153bf0 2023-10-30T13:24:18.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29fed6640 1 -- 172.21.15.125:0/669336754 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29814b5a0 con 0x7fb298153bf0 2023-10-30T13:24:18.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb29fed6640 1 -- 172.21.15.125:0/669336754 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2981580f0 con 0x7fb298153bf0 2023-10-30T13:24:18.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.575+0000 7fb28e7fc640 1 -- 172.21.15.125:0/669336754 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb2900b6050 con 0x7fb298153bf0 2023-10-30T13:24:18.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.579+0000 7fb28e7fc640 1 --2- 172.21.15.125:0/669336754 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb27c0664a0 0x7fb27c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:18.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.579+0000 7fb28e7fc640 1 -- 172.21.15.125:0/669336754 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fb290122d80 con 0x7fb298153bf0 2023-10-30T13:24:18.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.579+0000 7fb29dc72640 1 --2- 172.21.15.125:0/669336754 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb27c0664a0 0x7fb27c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:18.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.579+0000 7fb29dc72640 1 --2- 172.21.15.125:0/669336754 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb27c0664a0 0x7fb27c068960 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fb294000f70 tx=0x7fb294009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:18.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.583+0000 7fb28e7fc640 1 -- 172.21.15.125:0/669336754 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb2900ef6c0 con 0x7fb298153bf0 2023-10-30T13:24:18.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.759+0000 7fb29fed6640 1 -- 172.21.15.125:0/669336754 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb29804e170 con 0x7fb298153bf0 2023-10-30T13:24:18.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.763+0000 7fb28e7fc640 1 -- 172.21.15.125:0/669336754 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+5755 (secure 0 0 0) 0x7fb2900b4070 con 0x7fb298153bf0 2023-10-30T13:24:18.765 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:18.765 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":259,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:15.942055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:18.765 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 259 2023-10-30T13:24:18.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.763+0000 7fb29fed6640 1 -- 172.21.15.125:0/669336754 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb27c0664a0 msgr2=0x7fb27c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:18.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.763+0000 7fb29fed6640 1 --2- 172.21.15.125:0/669336754 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb27c0664a0 0x7fb27c068960 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7fb294000f70 tx=0x7fb294009040 comp rx=0 tx=0).stop 2023-10-30T13:24:18.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.763+0000 7fb29fed6640 1 -- 172.21.15.125:0/669336754 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb298153bf0 msgr2=0x7fb29814f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:18.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.763+0000 7fb29fed6640 1 --2- 172.21.15.125:0/669336754 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb298153bf0 0x7fb29814f530 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7fb2900b2040 tx=0x7fb29001bed0 comp rx=0 tx=0).stop 2023-10-30T13:24:18.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.763+0000 7fb29fed6640 1 -- 172.21.15.125:0/669336754 shutdown_connections 2023-10-30T13:24:18.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.763+0000 7fb29fed6640 1 --2- 172.21.15.125:0/669336754 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb27c0664a0 0x7fb27c068960 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:18.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.763+0000 7fb29fed6640 1 --2- 172.21.15.125:0/669336754 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb298153bf0 0x7fb29814f530 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:18.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.763+0000 7fb29fed6640 1 --2- 172.21.15.125:0/669336754 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2980c6fe0 0x7fb2981583f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:18.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.763+0000 7fb29fed6640 1 --2- 172.21.15.125:0/669336754 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb2980c5f60 0x7fb2981f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:18.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.763+0000 7fb29fed6640 1 -- 172.21.15.125:0/669336754 >> 172.21.15.125:0/669336754 conn(0x7fb29806c6e0 msgr2=0x7fb2981e5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:18.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.763+0000 7fb29fed6640 1 -- 172.21.15.125:0/669336754 shutdown_connections 2023-10-30T13:24:18.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:18.763+0000 7fb29fed6640 1 -- 172.21.15.125:0/669336754 wait complete. 2023-10-30T13:24:18.785 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:20.786 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:20.933 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f982a93e640 1 Processor -- start 2023-10-30T13:24:20.934 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f982a93e640 1 -- start start 2023-10-30T13:24:20.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f982a93e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98240c6b80 0x7f98240bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:20.936 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f982a93e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f98240c6f60 0x7f98240c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:20.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f982a93e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98240c0a50 con 0x7f98240c8430 2023-10-30T13:24:20.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f982a93e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f982406f6e0 con 0x7f98240c6f60 2023-10-30T13:24:20.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f982a93e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982406f9c0 con 0x7f98240c6b80 2023-10-30T13:24:20.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9828edb640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f98240c8430 0x7f982413ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34906/0 (socket says 172.21.15.125:34906) 2023-10-30T13:24:20.937 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9828edb640 1 -- 172.21.15.125:0/2985321695 learned_addr learned my addr 172.21.15.125:0/2985321695 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:24:20.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9823fff640 1 --2- 172.21.15.125:0/2985321695 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98240c6b80 0x7f98240bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:20.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f98237fe640 1 --2- 172.21.15.125:0/2985321695 >> v2:172.21.15.136:3300/0 conn(0x7f98240c6f60 0x7f98240c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:20.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 -- 172.21.15.125:0/2985321695 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589527408 0 0) 0x7f98240c0a50 con 0x7f98240c8430 2023-10-30T13:24:20.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 -- 172.21.15.125:0/2985321695 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f980c005740 con 0x7f98240c8430 2023-10-30T13:24:20.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 -- 172.21.15.125:0/2985321695 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3611044973 0 0) 0x7f980c005740 con 0x7f98240c8430 2023-10-30T13:24:20.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 -- 172.21.15.125:0/2985321695 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98240c6b80 msgr2=0x7f98240bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:20.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 --2- 172.21.15.125:0/2985321695 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f98240c6b80 0x7f98240bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:20.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 -- 172.21.15.125:0/2985321695 >> v2:172.21.15.136:3300/0 conn(0x7f98240c6f60 msgr2=0x7f98240c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:20.939 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 --2- 172.21.15.125:0/2985321695 >> v2:172.21.15.136:3300/0 conn(0x7f98240c6f60 0x7f98240c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:20.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 -- 172.21.15.125:0/2985321695 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982406fca0 con 0x7f98240c8430 2023-10-30T13:24:20.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 -- 172.21.15.125:0/2985321695 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f981c001a00 con 0x7f98240c8430 2023-10-30T13:24:20.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 -- 172.21.15.125:0/2985321695 >> v1:172.21.15.125:6789/0 conn(0x7f98240c8430 legacy=0x7f982413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:20.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 --2- 172.21.15.125:0/2985321695 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f980c005b90 0x7f980c005fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:20.940 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 --2- 172.21.15.125:0/2985321695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980c006580 0x7f980c006a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:20.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 --2- 172.21.15.125:0/2985321695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f980c0070d0 0x7f980c00dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:20.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 -- 172.21.15.125:0/2985321695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982406f6e0 con 0x7f980c006580 2023-10-30T13:24:20.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 -- 172.21.15.125:0/2985321695 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982406f9c0 con 0x7f980c0070d0 2023-10-30T13:24:20.941 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.931+0000 7f9822ffd640 1 -- 172.21.15.125:0/2985321695 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c00e0e0 con 0x7f980c005b90 2023-10-30T13:24:20.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f98237fe640 1 --2- 172.21.15.125:0/2985321695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980c006580 0x7f980c006a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:20.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f9828edb640 1 --2- 172.21.15.125:0/2985321695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f980c0070d0 0x7f980c00dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:20.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f9823fff640 1 --2- 172.21.15.125:0/2985321695 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f980c005b90 0x7f980c005fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:20.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f98237fe640 1 -- 172.21.15.125:0/2985321695 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f980c005b90 msgr2=0x7f980c005fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:20.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f98237fe640 1 --2- 172.21.15.125:0/2985321695 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f980c005b90 0x7f980c005fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:20.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f98237fe640 1 -- 172.21.15.125:0/2985321695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f980c0070d0 msgr2=0x7f980c00dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:20.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f98237fe640 1 --2- 172.21.15.125:0/2985321695 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f980c0070d0 0x7f980c00dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:20.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f98237fe640 1 -- 172.21.15.125:0/2985321695 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c005740 con 0x7f980c006580 2023-10-30T13:24:20.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f98237fe640 1 --2- 172.21.15.125:0/2985321695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980c006580 0x7f980c006a60 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f9818009ab0 tx=0x7f981801ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c451b996944252 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:20.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f9822ffd640 1 -- 172.21.15.125:0/2985321695 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98180a5550 con 0x7f980c006580 2023-10-30T13:24:20.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f9822ffd640 1 -- 172.21.15.125:0/2985321695 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98180a56b0 con 0x7f980c006580 2023-10-30T13:24:20.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f9822ffd640 1 -- 172.21.15.125:0/2985321695 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98180a5a60 con 0x7f980c006580 2023-10-30T13:24:20.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f982a93e640 1 -- 172.21.15.125:0/2985321695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980c006580 msgr2=0x7f980c006a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:20.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f982a93e640 1 --2- 172.21.15.125:0/2985321695 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f980c006580 0x7f980c006a60 secure :-1 s=READY pgs=2113 cs=0 l=1 rev1=1 crypto rx=0x7f9818009ab0 tx=0x7f981801ca70 comp rx=0 tx=0).stop 2023-10-30T13:24:20.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f9828edb640 1 -- 172.21.15.125:0/2985321695 reap_dead start 2023-10-30T13:24:20.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f982a93e640 1 -- 172.21.15.125:0/2985321695 shutdown_connections 2023-10-30T13:24:20.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.935+0000 7f982a93e640 1 -- 172.21.15.125:0/2985321695 >> 172.21.15.125:0/2985321695 conn(0x7f98240c7910 msgr2=0x7f98240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:20.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f982a93e640 1 -- 172.21.15.125:0/2985321695 shutdown_connections 2023-10-30T13:24:20.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f982a93e640 1 -- 172.21.15.125:0/2985321695 wait complete. 2023-10-30T13:24:20.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f982a93e640 1 Processor -- start 2023-10-30T13:24:20.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f982a93e640 1 -- start start 2023-10-30T13:24:20.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f982a93e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f980c006580 0x7f98240c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:20.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f982a93e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9824157500 0x7f982413a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:20.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f982a93e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f982413a720 0x7f9824152de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:20.949 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f982a93e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9824139c50 con 0x7f9824157500 2023-10-30T13:24:20.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f982a93e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98241396e0 con 0x7f982413a720 2023-10-30T13:24:20.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f982a93e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982413aba0 con 0x7f980c006580 2023-10-30T13:24:20.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f98237fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9824157500 0x7f982413a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:20.950 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f98237fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9824157500 0x7f982413a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35410/0 (socket says 172.21.15.125:35410) 2023-10-30T13:24:20.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f98237fe640 1 -- 172.21.15.125:0/1921222995 learned_addr learned my addr 172.21.15.125:0/1921222995 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:20.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f9828edb640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f982413a720 0x7f9824152de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:20.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f9823fff640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f980c006580 0x7f98240c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:20.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f98237fe640 1 -- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f980c006580 msgr2=0x7f98240c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:20.951 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f98237fe640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f980c006580 0x7f98240c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:20.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f98237fe640 1 -- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f982413a720 msgr2=0x7f9824152de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:20.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f98237fe640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f982413a720 0x7f9824152de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:20.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f98237fe640 1 -- 172.21.15.125:0/1921222995 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9824139400 con 0x7f9824157500 2023-10-30T13:24:20.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f9823fff640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f980c006580 0x7f98240c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:20.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.939+0000 7f98237fe640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9824157500 0x7f982413a1e0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f9818009ab0 tx=0x7f980c007a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:20.952 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.943+0000 7f9820ff9640 1 -- 172.21.15.125:0/1921222995 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818004220 con 0x7f9824157500 2023-10-30T13:24:20.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.943+0000 7f9820ff9640 1 -- 172.21.15.125:0/1921222995 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9818004380 con 0x7f9824157500 2023-10-30T13:24:20.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.943+0000 7f9820ff9640 1 -- 172.21.15.125:0/1921222995 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818003820 con 0x7f9824157500 2023-10-30T13:24:20.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.943+0000 7f982a93e640 1 -- 172.21.15.125:0/1921222995 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98241f3a20 con 0x7f9824157500 2023-10-30T13:24:20.953 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.943+0000 7f982a93e640 1 -- 172.21.15.125:0/1921222995 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9824134a30 con 0x7f9824157500 2023-10-30T13:24:20.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.943+0000 7f982a93e640 1 -- 172.21.15.125:0/1921222995 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982404e170 con 0x7f9824157500 2023-10-30T13:24:20.954 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.943+0000 7f9820ff9640 1 -- 172.21.15.125:0/1921222995 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f98180b4040 con 0x7f9824157500 2023-10-30T13:24:20.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.943+0000 7f9820ff9640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98100664d0 0x7f9810068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:20.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.947+0000 7f9820ff9640 1 -- 172.21.15.125:0/1921222995 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f981811edb0 con 0x7f9824157500 2023-10-30T13:24:20.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.947+0000 7f9823fff640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98100664d0 0x7f9810068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:20.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.947+0000 7f9823fff640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98100664d0 0x7f9810068990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f982406f1b0 tx=0x7f9814008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:20.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:20.951+0000 7f9820ff9640 1 -- 172.21.15.125:0/1921222995 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f98180eb920 con 0x7f9824157500 2023-10-30T13:24:21.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:21.135+0000 7f982a93e640 1 -- 172.21.15.125:0/1921222995 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9824138f60 con 0x7f9824157500 2023-10-30T13:24:21.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:21.135+0000 7f9820ff9640 1 -- 172.21.15.125:0/1921222995 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f98180b2090 con 0x7f9824157500 2023-10-30T13:24:21.138 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:21.139 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":259,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:15.942055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:21.139 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 259 2023-10-30T13:24:21.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:21.139+0000 7f982a93e640 1 -- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98100664d0 msgr2=0x7f9810068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:21.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:21.139+0000 7f982a93e640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98100664d0 0x7f9810068990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f982406f1b0 tx=0x7f9814008040 comp rx=0 tx=0).stop 2023-10-30T13:24:21.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:21.139+0000 7f982a93e640 1 -- 172.21.15.125:0/1921222995 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9824157500 msgr2=0x7f982413a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:21.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:21.139+0000 7f982a93e640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9824157500 0x7f982413a1e0 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7f9818009ab0 tx=0x7f980c007a10 comp rx=0 tx=0).stop 2023-10-30T13:24:21.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:21.139+0000 7f982a93e640 1 -- 172.21.15.125:0/1921222995 shutdown_connections 2023-10-30T13:24:21.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:21.139+0000 7f982a93e640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f98100664d0 0x7f9810068990 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:21.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:21.139+0000 7f982a93e640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f982413a720 0x7f9824152de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:21.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:21.139+0000 7f982a93e640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f9824157500 0x7f982413a1e0 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:21.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:21.139+0000 7f982a93e640 1 --2- 172.21.15.125:0/1921222995 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f980c006580 0x7f98240c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:21.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:21.139+0000 7f982a93e640 1 -- 172.21.15.125:0/1921222995 >> 172.21.15.125:0/1921222995 conn(0x7f98240c7910 msgr2=0x7f98240c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:21.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:21.139+0000 7f982a93e640 1 -- 172.21.15.125:0/1921222995 shutdown_connections 2023-10-30T13:24:21.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:21.139+0000 7f982a93e640 1 -- 172.21.15.125:0/1921222995 wait complete. 2023-10-30T13:24:21.160 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:23.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:23.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.299+0000 7f52bc89c640 1 Processor -- start 2023-10-30T13:24:23.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.299+0000 7f52bc89c640 1 -- start start 2023-10-30T13:24:23.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.299+0000 7f52bc89c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f52b80c05d0 0x7f52b813ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:23.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.299+0000 7f52bc89c640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b80c0a50 con 0x7f52b80c6b80 2023-10-30T13:24:23.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.299+0000 7f52bc89c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b806f6e0 con 0x7f52b80c6f60 2023-10-30T13:24:23.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.299+0000 7f52bc89c640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b806f9c0 con 0x7f52b80c05d0 2023-10-30T13:24:23.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.299+0000 7f52b659c640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f52b80c6b80 0x7f52b80bfec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34908/0 (socket says 172.21.15.125:34908) 2023-10-30T13:24:23.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.303+0000 7f52b659c640 1 -- 172.21.15.125:0/822844798 learned_addr learned my addr 172.21.15.125:0/822844798 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:24:23.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.303+0000 7f52add9b640 1 --2- 172.21.15.125:0/822844798 >> v2:172.21.15.125:3300/0 conn(0x7f52b80c05d0 0x7f52b813ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:23.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.303+0000 7f52b5d9b640 1 -- 172.21.15.125:0/822844798 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442092251 0 0) 0x7f52b80c0a50 con 0x7f52b80c6b80 2023-10-30T13:24:23.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.303+0000 7f52b5d9b640 1 -- 172.21.15.125:0/822844798 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52a4005740 con 0x7f52b80c6b80 2023-10-30T13:24:23.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.303+0000 7f52add9b640 1 -- 172.21.15.125:0/822844798 >> v1:172.21.15.136:6789/0 conn(0x7f52b80c6f60 legacy=0x7f52b81566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:23.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.303+0000 7f52add9b640 1 -- 172.21.15.125:0/822844798 >> v1:172.21.15.125:6789/0 conn(0x7f52b80c6b80 legacy=0x7f52b80bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:23.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.303+0000 7f52add9b640 1 -- 172.21.15.125:0/822844798 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b806fca0 con 0x7f52b80c05d0 2023-10-30T13:24:23.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.303+0000 7f52add9b640 1 --2- 172.21.15.125:0/822844798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b80c05d0 0x7f52b813ed90 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f52a00097c0 tx=0x7f52a001c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79107ec8efa74587 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:23.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.303+0000 7f52b5d9b640 1 -- 172.21.15.125:0/822844798 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a00ab600 con 0x7f52b80c05d0 2023-10-30T13:24:23.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.303+0000 7f52b5d9b640 1 -- 172.21.15.125:0/822844798 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a00a7050 con 0x7f52b80c05d0 2023-10-30T13:24:23.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.303+0000 7f52b5d9b640 1 -- 172.21.15.125:0/822844798 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a00ab980 con 0x7f52b80c05d0 2023-10-30T13:24:23.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 -- 172.21.15.125:0/822844798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b80c05d0 msgr2=0x7f52b813ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:23.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 --2- 172.21.15.125:0/822844798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b80c05d0 0x7f52b813ed90 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7f52a00097c0 tx=0x7f52a001c480 comp rx=0 tx=0).stop 2023-10-30T13:24:23.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 -- 172.21.15.125:0/822844798 shutdown_connections 2023-10-30T13:24:23.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 --2- 172.21.15.125:0/822844798 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b80c05d0 0x7f52b813ed90 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:23.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 -- 172.21.15.125:0/822844798 >> 172.21.15.125:0/822844798 conn(0x7f52b80c7910 msgr2=0x7f52b80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:23.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 -- 172.21.15.125:0/822844798 shutdown_connections 2023-10-30T13:24:23.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 -- 172.21.15.125:0/822844798 wait complete. 2023-10-30T13:24:23.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 Processor -- start 2023-10-30T13:24:23.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 -- start start 2023-10-30T13:24:23.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b80c05d0 0x7f52b8147470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:23.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b80c6b80 0x7f52b8155730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:23.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b80c6f60 0x7f52b8155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:23.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b813dfa0 con 0x7f52b80c6f60 2023-10-30T13:24:23.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b81586b0 con 0x7f52b80c6b80 2023-10-30T13:24:23.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52bc89c640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b8155080 con 0x7f52b80c05d0 2023-10-30T13:24:23.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52b6d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b80c6f60 0x7f52b8155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:23.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52b6d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b80c6f60 0x7f52b8155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35430/0 (socket says 172.21.15.125:35430) 2023-10-30T13:24:23.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52b6d9d640 1 -- 172.21.15.125:0/2428422880 learned_addr learned my addr 172.21.15.125:0/2428422880 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:23.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52b659c640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b80c05d0 0x7f52b8147470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:23.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52add9b640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b80c6b80 0x7f52b8155730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:23.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52b6d9d640 1 -- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b80c05d0 msgr2=0x7f52b8147470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:23.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52b6d9d640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b80c05d0 0x7f52b8147470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:23.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52b6d9d640 1 -- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b80c6b80 msgr2=0x7f52b8155730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:23.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52b6d9d640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b80c6b80 0x7f52b8155730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:23.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52b6d9d640 1 -- 172.21.15.125:0/2428422880 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a0009420 con 0x7f52b80c6f60 2023-10-30T13:24:23.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52b659c640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b80c05d0 0x7f52b8147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:23.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.307+0000 7f52b6d9d640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b80c6f60 0x7f52b8155c70 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f52b000c340 tx=0x7f52b000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:23.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.311+0000 7f52af7fe640 1 -- 172.21.15.125:0/2428422880 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0005220 con 0x7f52b80c6f60 2023-10-30T13:24:23.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.311+0000 7f52af7fe640 1 -- 172.21.15.125:0/2428422880 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52b0005380 con 0x7f52b80c6f60 2023-10-30T13:24:23.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.311+0000 7f52af7fe640 1 -- 172.21.15.125:0/2428422880 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52b0007270 con 0x7f52b80c6f60 2023-10-30T13:24:23.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.311+0000 7f52bc89c640 1 -- 172.21.15.125:0/2428422880 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b81552e0 con 0x7f52b80c6f60 2023-10-30T13:24:23.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.311+0000 7f52bc89c640 1 -- 172.21.15.125:0/2428422880 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b81508f0 con 0x7f52b80c6f60 2023-10-30T13:24:23.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.311+0000 7f52bc89c640 1 -- 172.21.15.125:0/2428422880 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b813e0e0 con 0x7f52b80c6f60 2023-10-30T13:24:23.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.311+0000 7f52af7fe640 1 -- 172.21.15.125:0/2428422880 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f52b00073d0 con 0x7f52b80c6f60 2023-10-30T13:24:23.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.311+0000 7f52af7fe640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f52980664a0 0x7f5298068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:23.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.311+0000 7f52af7fe640 1 -- 172.21.15.125:0/2428422880 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f52b0083f30 con 0x7f52b80c6f60 2023-10-30T13:24:23.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.311+0000 7f52b659c640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f52980664a0 0x7f5298068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:23.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.315+0000 7f52b659c640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f52980664a0 0x7f5298068960 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f52a8001570 tx=0x7f52a8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:23.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.315+0000 7f52af7fe640 1 -- 172.21.15.125:0/2428422880 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f52b813e0e0 con 0x7f52b80c6f60 2023-10-30T13:24:23.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.511+0000 7f52bc89c640 1 -- 172.21.15.125:0/2428422880 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52b813e0e0 con 0x7f52b80c6f60 2023-10-30T13:24:23.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.511+0000 7f52af7fe640 1 -- 172.21.15.125:0/2428422880 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 259 v259) v1 ==== 95+0+5755 (secure 0 0 0) 0x7f52b813e0e0 con 0x7f52b80c6f60 2023-10-30T13:24:23.514 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:23.515 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":259,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":259,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:15.942055+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:replay","state_seq":1,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:23.515 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 259 2023-10-30T13:24:23.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.511+0000 7f52bc89c640 1 -- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f52980664a0 msgr2=0x7f5298068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:23.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.511+0000 7f52bc89c640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f52980664a0 0x7f5298068960 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f52a8001570 tx=0x7f52a8009040 comp rx=0 tx=0).stop 2023-10-30T13:24:23.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.511+0000 7f52bc89c640 1 -- 172.21.15.125:0/2428422880 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b80c6f60 msgr2=0x7f52b8155c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:23.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.511+0000 7f52bc89c640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b80c6f60 0x7f52b8155c70 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7f52b000c340 tx=0x7f52b000c800 comp rx=0 tx=0).stop 2023-10-30T13:24:23.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.511+0000 7f52bc89c640 1 -- 172.21.15.125:0/2428422880 shutdown_connections 2023-10-30T13:24:23.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.511+0000 7f52bc89c640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f52980664a0 0x7f5298068960 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:23.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.511+0000 7f52bc89c640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f52b80c6f60 0x7f52b8155c70 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:23.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.511+0000 7f52bc89c640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f52b80c6b80 0x7f52b8155730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:23.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.511+0000 7f52bc89c640 1 --2- 172.21.15.125:0/2428422880 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f52b80c05d0 0x7f52b8147470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:23.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.515+0000 7f52bc89c640 1 -- 172.21.15.125:0/2428422880 >> 172.21.15.125:0/2428422880 conn(0x7f52b80c7910 msgr2=0x7f52b813ab90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:23.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.515+0000 7f52bc89c640 1 -- 172.21.15.125:0/2428422880 shutdown_connections 2023-10-30T13:24:23.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:23.515+0000 7f52bc89c640 1 -- 172.21.15.125:0/2428422880 wait complete. 2023-10-30T13:24:23.535 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:25.537 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:25.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7effa2772640 1 Processor -- start 2023-10-30T13:24:25.682 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7effa2772640 1 -- start start 2023-10-30T13:24:25.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7effa2772640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff9c0c6b80 0x7eff9c0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:25.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7effa2772640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7eff9c0c6f60 0x7eff9c0c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:25.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7effa2772640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7eff9c0c8430 0x7eff9c13ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:25.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7effa2772640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff9c0c0a50 con 0x7eff9c0c6f60 2023-10-30T13:24:25.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7effa2772640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff9c06f6e0 con 0x7eff9c0c8430 2023-10-30T13:24:25.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7effa2772640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c06f9c0 con 0x7eff9c0c6b80 2023-10-30T13:24:25.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7eff9b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7eff9c0c6f60 0x7eff9c0c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:25.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7eff9b7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7eff9c0c6f60 0x7eff9c0c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35438/0 (socket says 172.21.15.125:35438) 2023-10-30T13:24:25.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7eff9b7fe640 1 -- 172.21.15.125:0/2886424406 learned_addr learned my addr 172.21.15.125:0/2886424406 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:25.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7effa0d0f640 1 --2- 172.21.15.125:0/2886424406 >> v2:172.21.15.136:3300/0 conn(0x7eff9c0c8430 0x7eff9c13ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:25.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7eff9bfff640 1 --2- 172.21.15.125:0/2886424406 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff9c0c6b80 0x7eff9c0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:25.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7eff9b7fe640 1 -- 172.21.15.125:0/2886424406 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff9c0c6b80 msgr2=0x7eff9c0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:25.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7eff9b7fe640 1 --2- 172.21.15.125:0/2886424406 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff9c0c6b80 0x7eff9c0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:25.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7eff9b7fe640 1 -- 172.21.15.125:0/2886424406 >> v2:172.21.15.136:3300/0 conn(0x7eff9c0c8430 msgr2=0x7eff9c13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:25.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7eff9b7fe640 1 --2- 172.21.15.125:0/2886424406 >> v2:172.21.15.136:3300/0 conn(0x7eff9c0c8430 0x7eff9c13ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:25.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.679+0000 7eff9b7fe640 1 -- 172.21.15.125:0/2886424406 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff9c06fca0 con 0x7eff9c0c6f60 2023-10-30T13:24:25.687 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.683+0000 7eff9b7fe640 1 --2- 172.21.15.125:0/2886424406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff9c0c6f60 0x7eff9c0c0400 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0b0860 tx=0x7eff9001c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448a18127a3db9c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:25.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.683+0000 7eff9affd640 1 -- 172.21.15.125:0/2886424406 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff900ac630 con 0x7eff9c0c6f60 2023-10-30T13:24:25.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.683+0000 7eff9affd640 1 -- 172.21.15.125:0/2886424406 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff900ac790 con 0x7eff9c0c6f60 2023-10-30T13:24:25.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.683+0000 7eff9affd640 1 -- 172.21.15.125:0/2886424406 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff900acb20 con 0x7eff9c0c6f60 2023-10-30T13:24:25.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.683+0000 7effa2772640 1 -- 172.21.15.125:0/2886424406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff9c0c6f60 msgr2=0x7eff9c0c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:25.688 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.683+0000 7effa2772640 1 --2- 172.21.15.125:0/2886424406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff9c0c6f60 0x7eff9c0c0400 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0b0860 tx=0x7eff9001c500 comp rx=0 tx=0).stop 2023-10-30T13:24:25.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.683+0000 7effa2772640 1 -- 172.21.15.125:0/2886424406 shutdown_connections 2023-10-30T13:24:25.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.683+0000 7effa2772640 1 --2- 172.21.15.125:0/2886424406 >> v2:172.21.15.136:3300/0 conn(0x7eff9c0c8430 0x7eff9c13ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:25.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.683+0000 7effa2772640 1 --2- 172.21.15.125:0/2886424406 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff9c0c6f60 0x7eff9c0c0400 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:25.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.683+0000 7effa2772640 1 --2- 172.21.15.125:0/2886424406 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff9c0c6b80 0x7eff9c0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:25.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.683+0000 7effa2772640 1 -- 172.21.15.125:0/2886424406 >> 172.21.15.125:0/2886424406 conn(0x7eff9c0c7910 msgr2=0x7eff9c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:25.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.683+0000 7effa2772640 1 -- 172.21.15.125:0/2886424406 shutdown_connections 2023-10-30T13:24:25.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.683+0000 7effa2772640 1 -- 172.21.15.125:0/2886424406 wait complete. 2023-10-30T13:24:25.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7effa2772640 1 Processor -- start 2023-10-30T13:24:25.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7effa2772640 1 -- start start 2023-10-30T13:24:25.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7effa2772640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff9c0c6b80 0x7eff9c155290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:25.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7effa2772640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff9c0c6f60 0x7eff9c158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:25.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7effa2772640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff9c0c8430 0x7eff9c158b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:25.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7effa2772640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c160870 con 0x7eff9c0c6b80 2023-10-30T13:24:25.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7effa2772640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c155fd0 con 0x7eff9c0c8430 2023-10-30T13:24:25.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7effa2772640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff9c14f990 con 0x7eff9c0c6f60 2023-10-30T13:24:25.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7eff9bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff9c0c6b80 0x7eff9c155290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:25.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7eff9bfff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff9c0c6b80 0x7eff9c155290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35442/0 (socket says 172.21.15.125:35442) 2023-10-30T13:24:25.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7eff9bfff640 1 -- 172.21.15.125:0/43854697 learned_addr learned my addr 172.21.15.125:0/43854697 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:25.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7eff9b7fe640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff9c0c6f60 0x7eff9c158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:25.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7effa0d0f640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff9c0c8430 0x7eff9c158b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:25.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7eff9bfff640 1 -- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff9c0c6f60 msgr2=0x7eff9c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:25.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7eff9bfff640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff9c0c6f60 0x7eff9c158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:25.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7eff9bfff640 1 -- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff9c0c8430 msgr2=0x7eff9c158b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:25.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7eff9bfff640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff9c0c8430 0x7eff9c158b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:25.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7eff9bfff640 1 -- 172.21.15.125:0/43854697 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff900094d0 con 0x7eff9c0c6b80 2023-10-30T13:24:25.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7eff9b7fe640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff9c0c6f60 0x7eff9c158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:25.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7eff9bfff640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff9c0c6b80 0x7eff9c155290 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7eff8c00d830 tx=0x7eff8c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:25.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7eff98ff9640 1 -- 172.21.15.125:0/43854697 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c0096a0 con 0x7eff9c0c6b80 2023-10-30T13:24:25.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7eff98ff9640 1 -- 172.21.15.125:0/43854697 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff8c009800 con 0x7eff9c0c6b80 2023-10-30T13:24:25.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7eff98ff9640 1 -- 172.21.15.125:0/43854697 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff8c009b70 con 0x7eff9c0c6b80 2023-10-30T13:24:25.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7effa2772640 1 -- 172.21.15.125:0/43854697 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff9c073cc0 con 0x7eff9c0c6b80 2023-10-30T13:24:25.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.687+0000 7effa2772640 1 -- 172.21.15.125:0/43854697 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff9c156220 con 0x7eff9c0c6b80 2023-10-30T13:24:25.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.691+0000 7effa2772640 1 -- 172.21.15.125:0/43854697 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff9c04e170 con 0x7eff9c0c6b80 2023-10-30T13:24:25.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.691+0000 7eff98ff9640 1 -- 172.21.15.125:0/43854697 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7eff8c014410 con 0x7eff9c0c6b80 2023-10-30T13:24:25.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.691+0000 7eff98ff9640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff880664d0 0x7eff88068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:25.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.691+0000 7eff98ff9640 1 -- 172.21.15.125:0/43854697 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7eff8c085330 con 0x7eff9c0c6b80 2023-10-30T13:24:25.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.691+0000 7eff9b7fe640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff880664d0 0x7eff88068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:25.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.695+0000 7eff9b7fe640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff880664d0 0x7eff88068990 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7eff900ae000 tx=0x7eff90008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:25.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.695+0000 7eff98ff9640 1 -- 172.21.15.125:0/43854697 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7eff8c051c70 con 0x7eff9c0c6b80 2023-10-30T13:24:25.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.875+0000 7effa2772640 1 -- 172.21.15.125:0/43854697 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff9c154eb0 con 0x7eff9c0c6b80 2023-10-30T13:24:25.878 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.875+0000 7eff98ff9640 1 -- 172.21.15.125:0/43854697 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 260 v260) v1 ==== 95+0+5757 (secure 0 0 0) 0x7eff8c055290 con 0x7eff9c0c6b80 2023-10-30T13:24:25.878 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:25.879 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":260,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":260,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:24.849814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:resolve","state_seq":33,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:25.879 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 260 2023-10-30T13:24:25.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.875+0000 7effa2772640 1 -- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff880664d0 msgr2=0x7eff88068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:25.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.875+0000 7effa2772640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff880664d0 0x7eff88068990 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7eff900ae000 tx=0x7eff90008040 comp rx=0 tx=0).stop 2023-10-30T13:24:25.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.875+0000 7effa2772640 1 -- 172.21.15.125:0/43854697 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff9c0c6b80 msgr2=0x7eff9c155290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:25.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.875+0000 7effa2772640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff9c0c6b80 0x7eff9c155290 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7eff8c00d830 tx=0x7eff8c00dcf0 comp rx=0 tx=0).stop 2023-10-30T13:24:25.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.879+0000 7effa2772640 1 -- 172.21.15.125:0/43854697 shutdown_connections 2023-10-30T13:24:25.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.879+0000 7effa2772640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7eff880664d0 0x7eff88068990 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:25.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.879+0000 7effa2772640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7eff9c0c8430 0x7eff9c158b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:25.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.879+0000 7effa2772640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7eff9c0c6f60 0x7eff9c158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:25.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.879+0000 7effa2772640 1 --2- 172.21.15.125:0/43854697 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7eff9c0c6b80 0x7eff9c155290 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:25.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.879+0000 7effa2772640 1 -- 172.21.15.125:0/43854697 >> 172.21.15.125:0/43854697 conn(0x7eff9c0c7910 msgr2=0x7eff9c06bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:25.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.879+0000 7effa2772640 1 -- 172.21.15.125:0/43854697 shutdown_connections 2023-10-30T13:24:25.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:25.879+0000 7effa2772640 1 -- 172.21.15.125:0/43854697 wait complete. 2023-10-30T13:24:25.899 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:27.900 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:28.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.043+0000 7f3a46d03640 1 Processor -- start 2023-10-30T13:24:28.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.047+0000 7f3a46d03640 1 -- start start 2023-10-30T13:24:28.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.047+0000 7f3a46d03640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3a401538f0 0x7f3a40159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:28.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.047+0000 7f3a46d03640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a4015cd50 0x7f3a400c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:28.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.047+0000 7f3a46d03640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4015d300 con 0x7f3a400cefa0 2023-10-30T13:24:28.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.047+0000 7f3a46d03640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a4015d5e0 con 0x7f3a401538f0 2023-10-30T13:24:28.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.047+0000 7f3a46d03640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4015d8c0 con 0x7f3a4015cd50 2023-10-30T13:24:28.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.047+0000 7f3a45d01640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3a401538f0 0x7f3a40159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:28.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.047+0000 7f3a45d01640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f3a401538f0 0x7f3a40159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35458/0 (socket says 172.21.15.125:35458) 2023-10-30T13:24:28.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.047+0000 7f3a45d01640 1 -- 172.21.15.125:0/1172171704 learned_addr learned my addr 172.21.15.125:0/1172171704 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:28.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.047+0000 7f3a45500640 1 --2- 172.21.15.125:0/1172171704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a4015cd50 0x7f3a400c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:28.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.047+0000 7f3a45d01640 1 -- 172.21.15.125:0/1172171704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a4015cd50 msgr2=0x7f3a400c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:28.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.047+0000 7f3a45d01640 1 --2- 172.21.15.125:0/1172171704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a4015cd50 0x7f3a400c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.047+0000 7f3a45d01640 1 -- 172.21.15.125:0/1172171704 >> v1:172.21.15.136:6789/0 conn(0x7f3a400cefa0 legacy=0x7f3a400cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:28.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.047+0000 7f3a45d01640 1 -- 172.21.15.125:0/1172171704 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4015dba0 con 0x7f3a401538f0 2023-10-30T13:24:28.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.051+0000 7f3a45d01640 1 --2- 172.21.15.125:0/1172171704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401538f0 0x7f3a40159fb0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009870 tx=0x7f3a3c01ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c4fb76719e56986 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:28.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.051+0000 7f3a44cff640 1 -- 172.21.15.125:0/1172171704 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0a8a30 con 0x7f3a401538f0 2023-10-30T13:24:28.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.051+0000 7f3a44cff640 1 -- 172.21.15.125:0/1172171704 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3c0a8b90 con 0x7f3a401538f0 2023-10-30T13:24:28.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.051+0000 7f3a44cff640 1 -- 172.21.15.125:0/1172171704 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c0b0440 con 0x7f3a401538f0 2023-10-30T13:24:28.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.051+0000 7f3a46d03640 1 -- 172.21.15.125:0/1172171704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401538f0 msgr2=0x7f3a40159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:28.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.051+0000 7f3a46d03640 1 --2- 172.21.15.125:0/1172171704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401538f0 0x7f3a40159fb0 secure :-1 s=READY pgs=2120 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c009870 tx=0x7f3a3c01ed30 comp rx=0 tx=0).stop 2023-10-30T13:24:28.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.051+0000 7f3a46d03640 1 -- 172.21.15.125:0/1172171704 shutdown_connections 2023-10-30T13:24:28.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.051+0000 7f3a46d03640 1 --2- 172.21.15.125:0/1172171704 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a4015cd50 0x7f3a400c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.051+0000 7f3a46d03640 1 --2- 172.21.15.125:0/1172171704 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401538f0 0x7f3a40159fb0 unknown :-1 s=CLOSED pgs=2120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.051+0000 7f3a46d03640 1 -- 172.21.15.125:0/1172171704 >> 172.21.15.125:0/1172171704 conn(0x7f3a4006a640 msgr2=0x7f3a40065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:28.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.051+0000 7f3a46d03640 1 -- 172.21.15.125:0/1172171704 shutdown_connections 2023-10-30T13:24:28.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.051+0000 7f3a46d03640 1 -- 172.21.15.125:0/1172171704 wait complete. 2023-10-30T13:24:28.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.051+0000 7f3a46d03640 1 Processor -- start 2023-10-30T13:24:28.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.051+0000 7f3a46d03640 1 -- start start 2023-10-30T13:24:28.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a46d03640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a400cefa0 0x7f3a400b43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:28.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a46d03640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401538f0 0x7f3a401e3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:28.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a46d03640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a4015cd50 0x7f3a401f2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:28.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a46d03640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400b48f0 con 0x7f3a401538f0 2023-10-30T13:24:28.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a46d03640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a40148180 con 0x7f3a400cefa0 2023-10-30T13:24:28.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a46d03640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a401f2780 con 0x7f3a4015cd50 2023-10-30T13:24:28.059 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a45500640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401538f0 0x7f3a401e3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:28.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a45500640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401538f0 0x7f3a401e3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35462/0 (socket says 172.21.15.125:35462) 2023-10-30T13:24:28.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a45500640 1 -- 172.21.15.125:0/528177752 learned_addr learned my addr 172.21.15.125:0/528177752 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:28.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a45d01640 1 --2- 172.21.15.125:0/528177752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a400cefa0 0x7f3a400b43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:28.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a46502640 1 --2- 172.21.15.125:0/528177752 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a4015cd50 0x7f3a401f2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:28.060 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a45500640 1 -- 172.21.15.125:0/528177752 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a4015cd50 msgr2=0x7f3a401f2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:28.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a45500640 1 --2- 172.21.15.125:0/528177752 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a4015cd50 0x7f3a401f2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a45500640 1 -- 172.21.15.125:0/528177752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a400cefa0 msgr2=0x7f3a400b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:28.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a45500640 1 --2- 172.21.15.125:0/528177752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a400cefa0 0x7f3a400b43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a45500640 1 -- 172.21.15.125:0/528177752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a3c0094d0 con 0x7f3a401538f0 2023-10-30T13:24:28.061 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a45500640 1 --2- 172.21.15.125:0/528177752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401538f0 0x7f3a401e3300 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000e8a0 tx=0x7f3a3000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:28.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a367fc640 1 -- 172.21.15.125:0/528177752 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30009510 con 0x7f3a401538f0 2023-10-30T13:24:28.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a367fc640 1 -- 172.21.15.125:0/528177752 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3000c940 con 0x7f3a401538f0 2023-10-30T13:24:28.063 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a46d03640 1 -- 172.21.15.125:0/528177752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a401483b0 con 0x7f3a401538f0 2023-10-30T13:24:28.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a367fc640 1 -- 172.21.15.125:0/528177752 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3000ccb0 con 0x7f3a401538f0 2023-10-30T13:24:28.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a46d03640 1 -- 172.21.15.125:0/528177752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a401e5510 con 0x7f3a401538f0 2023-10-30T13:24:28.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.055+0000 7f3a46d03640 1 -- 172.21.15.125:0/528177752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a401f2930 con 0x7f3a401538f0 2023-10-30T13:24:28.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.059+0000 7f3a367fc640 1 -- 172.21.15.125:0/528177752 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f3a30013470 con 0x7f3a401538f0 2023-10-30T13:24:28.064 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.059+0000 7f3a367fc640 1 --2- 172.21.15.125:0/528177752 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3a240664d0 0x7f3a24068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:28.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.059+0000 7f3a367fc640 1 -- 172.21.15.125:0/528177752 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f3a30084d20 con 0x7f3a401538f0 2023-10-30T13:24:28.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.059+0000 7f3a45d01640 1 --2- 172.21.15.125:0/528177752 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3a240664d0 0x7f3a24068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:28.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.059+0000 7f3a45d01640 1 --2- 172.21.15.125:0/528177752 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3a240664d0 0x7f3a24068990 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c0b2000 tx=0x7f3a3c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:28.065 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.063+0000 7f3a367fc640 1 -- 172.21.15.125:0/528177752 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f3a401f2930 con 0x7f3a401538f0 2023-10-30T13:24:28.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.247+0000 7f3a46d03640 1 -- 172.21.15.125:0/528177752 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a401f28f0 con 0x7f3a401538f0 2023-10-30T13:24:28.250 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.247+0000 7f3a367fc640 1 -- 172.21.15.125:0/528177752 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 260 v260) v1 ==== 95+0+5757 (secure 0 0 0) 0x7f3a30021090 con 0x7f3a401538f0 2023-10-30T13:24:28.251 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:28.251 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":260,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":260,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:24.849814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:resolve","state_seq":33,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:28.252 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 260 2023-10-30T13:24:28.252 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.251+0000 7f3a46d03640 1 -- 172.21.15.125:0/528177752 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3a240664d0 msgr2=0x7f3a24068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:28.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.251+0000 7f3a46d03640 1 --2- 172.21.15.125:0/528177752 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3a240664d0 0x7f3a24068990 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c0b2000 tx=0x7f3a3c009040 comp rx=0 tx=0).stop 2023-10-30T13:24:28.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.251+0000 7f3a46d03640 1 -- 172.21.15.125:0/528177752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401538f0 msgr2=0x7f3a401e3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:28.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.251+0000 7f3a46d03640 1 --2- 172.21.15.125:0/528177752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401538f0 0x7f3a401e3300 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000e8a0 tx=0x7f3a3000eca0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.251+0000 7f3a46d03640 1 -- 172.21.15.125:0/528177752 shutdown_connections 2023-10-30T13:24:28.253 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.251+0000 7f3a46d03640 1 --2- 172.21.15.125:0/528177752 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f3a240664d0 0x7f3a24068990 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.251+0000 7f3a46d03640 1 --2- 172.21.15.125:0/528177752 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f3a4015cd50 0x7f3a401f2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.251+0000 7f3a46d03640 1 --2- 172.21.15.125:0/528177752 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f3a401538f0 0x7f3a401e3300 unknown :-1 s=CLOSED pgs=2121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.251+0000 7f3a46d03640 1 --2- 172.21.15.125:0/528177752 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f3a400cefa0 0x7f3a400b43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:28.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.251+0000 7f3a46d03640 1 -- 172.21.15.125:0/528177752 >> 172.21.15.125:0/528177752 conn(0x7f3a4006a640 msgr2=0x7f3a401489b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:28.254 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.251+0000 7f3a46d03640 1 -- 172.21.15.125:0/528177752 shutdown_connections 2023-10-30T13:24:28.255 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:28.251+0000 7f3a46d03640 1 -- 172.21.15.125:0/528177752 wait complete. 2023-10-30T13:24:28.272 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 260, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 260, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 36 }, "gid_9409": { "addr": "172.21.15.136:6839/2056603754", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 2056603754, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 2056603754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9409, "incarnation": 259, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:resolve", "state_seq": 33 }, "gid_9600": { "addr": "172.21.15.125:6836/2077416857", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2077416857, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2077416857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 9600, "incarnation": 241, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 42 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:24:24.849814+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9409, "mds_1": 9150, "mds_2": 9600 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:24:30.273 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:30.421 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb8715c5640 1 Processor -- start 2023-10-30T13:24:30.422 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb8715c5640 1 -- start start 2023-10-30T13:24:30.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb8715c5640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fb86c1538a0 0x7fb86c15d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:30.423 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb8715c5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb86c0ca0e0 0x7fb86c15a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:30.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb8715c5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c0cb800 con 0x7fb86c0cf160 2023-10-30T13:24:30.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb8715c5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86c15dc30 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb8715c5640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86c0cb330 con 0x7fb86c1538a0 2023-10-30T13:24:30.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb86a7fc640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb86c0ca0e0 0x7fb86c15a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:30.424 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb86b7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fb86c0cf160 0x7fb86c0cbbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:34918/0 (socket says 172.21.15.125:34918) 2023-10-30T13:24:30.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb86b7fe640 1 -- 172.21.15.125:0/3973354177 learned_addr learned my addr 172.21.15.125:0/3973354177 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:24:30.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb86affd640 1 --2- 172.21.15.125:0/3973354177 >> v2:172.21.15.136:3300/0 conn(0x7fb86c1538a0 0x7fb86c15d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:30.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb86a7fc640 1 -- 172.21.15.125:0/3973354177 >> v2:172.21.15.136:3300/0 conn(0x7fb86c1538a0 msgr2=0x7fb86c15d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:30.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb86a7fc640 1 --2- 172.21.15.125:0/3973354177 >> v2:172.21.15.136:3300/0 conn(0x7fb86c1538a0 0x7fb86c15d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:30.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb86a7fc640 1 -- 172.21.15.125:0/3973354177 >> v1:172.21.15.125:6789/0 conn(0x7fb86c0cf160 legacy=0x7fb86c0cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:30.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb86a7fc640 1 -- 172.21.15.125:0/3973354177 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c0cad70 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.426 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.419+0000 7fb869ffb640 1 -- 172.21.15.125:0/3973354177 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751171214 0 0) 0x7fb86c0cb800 con 0x7fb86c0cf160 2023-10-30T13:24:30.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.423+0000 7fb86a7fc640 1 --2- 172.21.15.125:0/3973354177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb86c0ca0e0 0x7fb86c15a090 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0cfd10 tx=0x7fb86001ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b39b31a6556bfed server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:30.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.423+0000 7fb869ffb640 1 -- 172.21.15.125:0/3973354177 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8600a8950 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.423+0000 7fb869ffb640 1 -- 172.21.15.125:0/3973354177 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8600a8ab0 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.423+0000 7fb869ffb640 1 -- 172.21.15.125:0/3973354177 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8600a8e40 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.423+0000 7fb8715c5640 1 -- 172.21.15.125:0/3973354177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb86c0ca0e0 msgr2=0x7fb86c15a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:30.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.423+0000 7fb8715c5640 1 --2- 172.21.15.125:0/3973354177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb86c0ca0e0 0x7fb86c15a090 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7fb86c0cfd10 tx=0x7fb86001ec50 comp rx=0 tx=0).stop 2023-10-30T13:24:30.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.423+0000 7fb8715c5640 1 -- 172.21.15.125:0/3973354177 shutdown_connections 2023-10-30T13:24:30.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.423+0000 7fb8715c5640 1 --2- 172.21.15.125:0/3973354177 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb86c0ca0e0 0x7fb86c15a090 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:30.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.423+0000 7fb8715c5640 1 --2- 172.21.15.125:0/3973354177 >> v2:172.21.15.136:3300/0 conn(0x7fb86c1538a0 0x7fb86c15d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:30.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.423+0000 7fb8715c5640 1 -- 172.21.15.125:0/3973354177 >> 172.21.15.125:0/3973354177 conn(0x7fb86c06bfe0 msgr2=0x7fb86c06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:30.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.423+0000 7fb8715c5640 1 -- 172.21.15.125:0/3973354177 shutdown_connections 2023-10-30T13:24:30.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.423+0000 7fb8715c5640 1 -- 172.21.15.125:0/3973354177 wait complete. 2023-10-30T13:24:30.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb8715c5640 1 Processor -- start 2023-10-30T13:24:30.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb8715c5640 1 -- start start 2023-10-30T13:24:30.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb8715c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb86c0ca0e0 0x7fb86c0b4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:30.432 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb8715c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb86c0cf160 0x7fb86c158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:30.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb8715c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb86c1538a0 0x7fb86c158870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:30.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb8715c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c0b3b10 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb8715c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c1e35b0 con 0x7fb86c0cf160 2023-10-30T13:24:30.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb8715c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c0ce8f0 con 0x7fb86c1538a0 2023-10-30T13:24:30.433 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb86affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb86c0ca0e0 0x7fb86c0b4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:30.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb86affd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb86c0ca0e0 0x7fb86c0b4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35476/0 (socket says 172.21.15.125:35476) 2023-10-30T13:24:30.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb86affd640 1 -- 172.21.15.125:0/37325258 learned_addr learned my addr 172.21.15.125:0/37325258 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:30.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb86b7fe640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb86c1538a0 0x7fb86c158870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:30.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb86a7fc640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb86c0cf160 0x7fb86c158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:30.434 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb86affd640 1 -- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb86c1538a0 msgr2=0x7fb86c158870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:30.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb86affd640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb86c1538a0 0x7fb86c158870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:30.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb86affd640 1 -- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb86c0cf160 msgr2=0x7fb86c158330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:30.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb86affd640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb86c0cf160 0x7fb86c158330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:30.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb86affd640 1 -- 172.21.15.125:0/37325258 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb860009420 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.435 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb86b7fe640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb86c1538a0 0x7fb86c158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:30.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb86affd640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb86c0ca0e0 0x7fb86c0b4170 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00d5f0 tx=0x7fb85c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:30.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb8537fe640 1 -- 172.21.15.125:0/37325258 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c0125f0 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb8537fe640 1 -- 172.21.15.125:0/37325258 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85c012750 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb8537fe640 1 -- 172.21.15.125:0/37325258 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85c012ae0 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.436 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb8715c5640 1 -- 172.21.15.125:0/37325258 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c0b4a40 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.427+0000 7fb8715c5640 1 -- 172.21.15.125:0/37325258 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86c0b4ce0 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.437 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.431+0000 7fb8715c5640 1 -- 172.21.15.125:0/37325258 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86c0b2a50 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.431+0000 7fb8537fe640 1 -- 172.21.15.125:0/37325258 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb85c012c40 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.431+0000 7fb8537fe640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8580664a0 0x7fb858068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:30.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.431+0000 7fb8537fe640 1 -- 172.21.15.125:0/37325258 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fb85c084e30 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.431+0000 7fb86a7fc640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8580664a0 0x7fb858068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:30.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.431+0000 7fb86a7fc640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8580664a0 0x7fb858068960 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fb8600b2000 tx=0x7fb860008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:30.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.435+0000 7fb8537fe640 1 -- 172.21.15.125:0/37325258 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb85c051770 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.615+0000 7fb8715c5640 1 -- 172.21.15.125:0/37325258 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb86c04e170 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.618 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.615+0000 7fb8537fe640 1 -- 172.21.15.125:0/37325258 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 260 v260) v1 ==== 95+0+5757 (secure 0 0 0) 0x7fb85c054d90 con 0x7fb86c0ca0e0 2023-10-30T13:24:30.618 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:30.619 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":260,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":260,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:24.849814+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:resolve","state_seq":33,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:30.619 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 260 2023-10-30T13:24:30.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.615+0000 7fb8715c5640 1 -- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8580664a0 msgr2=0x7fb858068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:30.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.615+0000 7fb8715c5640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8580664a0 0x7fb858068960 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fb8600b2000 tx=0x7fb860008040 comp rx=0 tx=0).stop 2023-10-30T13:24:30.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.615+0000 7fb8715c5640 1 -- 172.21.15.125:0/37325258 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb86c0ca0e0 msgr2=0x7fb86c0b4170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:30.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.615+0000 7fb8715c5640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb86c0ca0e0 0x7fb86c0b4170 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7fb85c00d5f0 tx=0x7fb85c00dab0 comp rx=0 tx=0).stop 2023-10-30T13:24:30.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.619+0000 7fb8715c5640 1 -- 172.21.15.125:0/37325258 shutdown_connections 2023-10-30T13:24:30.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.619+0000 7fb8715c5640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb8580664a0 0x7fb858068960 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:30.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.619+0000 7fb8715c5640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb86c1538a0 0x7fb86c158870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:30.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.619+0000 7fb8715c5640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb86c0cf160 0x7fb86c158330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:30.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.619+0000 7fb8715c5640 1 --2- 172.21.15.125:0/37325258 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb86c0ca0e0 0x7fb86c0b4170 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:30.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.619+0000 7fb8715c5640 1 -- 172.21.15.125:0/37325258 >> 172.21.15.125:0/37325258 conn(0x7fb86c06bfe0 msgr2=0x7fb86c1e5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:30.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.619+0000 7fb8715c5640 1 -- 172.21.15.125:0/37325258 shutdown_connections 2023-10-30T13:24:30.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:30.619+0000 7fb8715c5640 1 -- 172.21.15.125:0/37325258 wait complete. 2023-10-30T13:24:30.639 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:32.641 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:32.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.783+0000 7ff866079640 1 Processor -- start 2023-10-30T13:24:32.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.783+0000 7ff866079640 1 -- start start 2023-10-30T13:24:32.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff866079640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8601538a0 0x7ff86015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:32.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff866079640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8600cb800 con 0x7ff8600ca0e0 2023-10-30T13:24:32.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff866079640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff86015dc30 con 0x7ff8600cf160 2023-10-30T13:24:32.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff866079640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8600cb330 con 0x7ff8601538a0 2023-10-30T13:24:32.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85effd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7ff8600ca0e0 0x7ff86015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51670/0 (socket says 172.21.15.125:51670) 2023-10-30T13:24:32.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85effd640 1 -- 172.21.15.125:0/1535690333 learned_addr learned my addr 172.21.15.125:0/1535690333 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:24:32.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85f7fe640 1 --2- 172.21.15.125:0/1535690333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8601538a0 0x7ff86015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:32.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877746933 0 0) 0x7ff8600cb800 con 0x7ff8600ca0e0 2023-10-30T13:24:32.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff848003530 con 0x7ff8600ca0e0 2023-10-30T13:24:32.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715646023 0 0) 0x7ff8580014a0 con 0x7ff8600cf160 2023-10-30T13:24:32.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8600cb800 con 0x7ff8600cf160 2023-10-30T13:24:32.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1650624752 0 0) 0x7ff848003530 con 0x7ff8600ca0e0 2023-10-30T13:24:32.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8601538a0 msgr2=0x7ff86015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:32.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 --2- 172.21.15.125:0/1535690333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8601538a0 0x7ff86015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:32.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 >> v1:172.21.15.136:6789/0 conn(0x7ff8600cf160 legacy=0x7ff860164070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:32.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8600cad70 con 0x7ff8600ca0e0 2023-10-30T13:24:32.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7ff854001680 con 0x7ff8600ca0e0 2023-10-30T13:24:32.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 >> v1:172.21.15.125:6789/0 conn(0x7ff8600ca0e0 legacy=0x7ff86015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:32.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 --2- 172.21.15.125:0/1535690333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff848003b40 0x7ff848003f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:32.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 --2- 172.21.15.125:0/1535690333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8480047d0 0x7ff848004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:32.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 --2- 172.21.15.125:0/1535690333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8601538a0 0x7ff848005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:32.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8600cb330 con 0x7ff848003b40 2023-10-30T13:24:32.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84800b990 con 0x7ff8480047d0 2023-10-30T13:24:32.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84800bb50 con 0x7ff8601538a0 2023-10-30T13:24:32.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85f7fe640 1 --2- 172.21.15.125:0/1535690333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff848003b40 0x7ff848003f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:32.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85ffff640 1 --2- 172.21.15.125:0/1535690333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8601538a0 0x7ff848005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:32.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85effd640 1 --2- 172.21.15.125:0/1535690333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8480047d0 0x7ff848004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:32.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85f7fe640 1 -- 172.21.15.125:0/1535690333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8601538a0 msgr2=0x7ff848005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:32.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85f7fe640 1 --2- 172.21.15.125:0/1535690333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff8601538a0 0x7ff848005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:32.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85f7fe640 1 -- 172.21.15.125:0/1535690333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8480047d0 msgr2=0x7ff848004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:32.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85f7fe640 1 --2- 172.21.15.125:0/1535690333 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff8480047d0 0x7ff848004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:32.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.787+0000 7ff85f7fe640 1 -- 172.21.15.125:0/1535690333 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff848003530 con 0x7ff848003b40 2023-10-30T13:24:32.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.791+0000 7ff85f7fe640 1 --2- 172.21.15.125:0/1535690333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff848003b40 0x7ff848003f20 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7ff850009ae0 tx=0x7ff85001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f44197d0aba96073 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:32.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.791+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8500a5640 con 0x7ff848003b40 2023-10-30T13:24:32.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.791+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff858002100 con 0x7ff848003b40 2023-10-30T13:24:32.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.791+0000 7ff85e7fc640 1 -- 172.21.15.125:0/1535690333 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8500a59e0 con 0x7ff848003b40 2023-10-30T13:24:32.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.791+0000 7ff866079640 1 -- 172.21.15.125:0/1535690333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff848003b40 msgr2=0x7ff848003f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:32.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.791+0000 7ff866079640 1 --2- 172.21.15.125:0/1535690333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff848003b40 0x7ff848003f20 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7ff850009ae0 tx=0x7ff85001ca50 comp rx=0 tx=0).stop 2023-10-30T13:24:32.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.791+0000 7ff85ffff640 1 -- 172.21.15.125:0/1535690333 reap_dead start 2023-10-30T13:24:32.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.791+0000 7ff866079640 1 -- 172.21.15.125:0/1535690333 shutdown_connections 2023-10-30T13:24:32.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.791+0000 7ff866079640 1 -- 172.21.15.125:0/1535690333 >> 172.21.15.125:0/1535690333 conn(0x7ff86006bfe0 msgr2=0x7ff86006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:32.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.791+0000 7ff866079640 1 -- 172.21.15.125:0/1535690333 shutdown_connections 2023-10-30T13:24:32.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.791+0000 7ff866079640 1 -- 172.21.15.125:0/1535690333 wait complete. 2023-10-30T13:24:32.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff866079640 1 Processor -- start 2023-10-30T13:24:32.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff866079640 1 -- start start 2023-10-30T13:24:32.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff866079640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff848003b40 0x7ff8601e31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:32.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff866079640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff86015a8b0 0x7ff8601e36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:32.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff866079640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8601479d0 0x7ff860147e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:32.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff866079640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8601571e0 con 0x7ff8601479d0 2023-10-30T13:24:32.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff866079640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8601e53a0 con 0x7ff848003b40 2023-10-30T13:24:32.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff866079640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8601e2d10 con 0x7ff86015a8b0 2023-10-30T13:24:32.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff85ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8601479d0 0x7ff860147e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:32.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff85ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8601479d0 0x7ff860147e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40872/0 (socket says 172.21.15.125:40872) 2023-10-30T13:24:32.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff85ffff640 1 -- 172.21.15.125:0/1173823848 learned_addr learned my addr 172.21.15.125:0/1173823848 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:32.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff85f7fe640 1 --2- 172.21.15.125:0/1173823848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff848003b40 0x7ff8601e31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:32.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff85effd640 1 --2- 172.21.15.125:0/1173823848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff86015a8b0 0x7ff8601e36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:32.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff85ffff640 1 -- 172.21.15.125:0/1173823848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff86015a8b0 msgr2=0x7ff8601e36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:32.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff85ffff640 1 --2- 172.21.15.125:0/1173823848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff86015a8b0 0x7ff8601e36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:32.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff85ffff640 1 -- 172.21.15.125:0/1173823848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff848003b40 msgr2=0x7ff8601e31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:32.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff85ffff640 1 --2- 172.21.15.125:0/1173823848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff848003b40 0x7ff8601e31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:32.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff85ffff640 1 -- 172.21.15.125:0/1173823848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8601e4b00 con 0x7ff8601479d0 2023-10-30T13:24:32.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff85ffff640 1 --2- 172.21.15.125:0/1173823848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8601479d0 0x7ff860147e80 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7ff8600cfd10 tx=0x7ff858011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:32.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff847fff640 1 -- 172.21.15.125:0/1173823848 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff85800fb30 con 0x7ff8601479d0 2023-10-30T13:24:32.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff847fff640 1 -- 172.21.15.125:0/1173823848 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff85800fc90 con 0x7ff8601479d0 2023-10-30T13:24:32.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff847fff640 1 -- 172.21.15.125:0/1173823848 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff858015410 con 0x7ff8601479d0 2023-10-30T13:24:32.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff866079640 1 -- 172.21.15.125:0/1173823848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8601e4de0 con 0x7ff8601479d0 2023-10-30T13:24:32.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff866079640 1 -- 172.21.15.125:0/1173823848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff860151450 con 0x7ff8601479d0 2023-10-30T13:24:32.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.795+0000 7ff866079640 1 -- 172.21.15.125:0/1173823848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8601e2ec0 con 0x7ff8601479d0 2023-10-30T13:24:32.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.799+0000 7ff847fff640 1 -- 172.21.15.125:0/1173823848 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff858015570 con 0x7ff8601479d0 2023-10-30T13:24:32.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.799+0000 7ff847fff640 1 --2- 172.21.15.125:0/1173823848 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff84c0664d0 0x7ff84c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:32.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.799+0000 7ff847fff640 1 -- 172.21.15.125:0/1173823848 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7ff858087a50 con 0x7ff8601479d0 2023-10-30T13:24:32.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.799+0000 7ff85f7fe640 1 --2- 172.21.15.125:0/1173823848 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff84c0664d0 0x7ff84c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:32.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.799+0000 7ff85f7fe640 1 --2- 172.21.15.125:0/1173823848 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff84c0664d0 0x7ff84c068990 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7ff850009ae0 tx=0x7ff8500046d0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:32.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.803+0000 7ff847fff640 1 -- 172.21.15.125:0/1173823848 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff8601e2ec0 con 0x7ff8601479d0 2023-10-30T13:24:32.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.983+0000 7ff866079640 1 -- 172.21.15.125:0/1173823848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff86015dc30 con 0x7ff8601479d0 2023-10-30T13:24:32.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.983+0000 7ff847fff640 1 -- 172.21.15.125:0/1173823848 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 261 v261) v1 ==== 95+0+5759 (secure 0 0 0) 0x7ff86015dc30 con 0x7ff8601479d0 2023-10-30T13:24:32.986 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:32.986 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":261,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":261,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:30.642173+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:32.987 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 261 2023-10-30T13:24:32.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.983+0000 7ff866079640 1 -- 172.21.15.125:0/1173823848 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff84c0664d0 msgr2=0x7ff84c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:32.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.983+0000 7ff866079640 1 --2- 172.21.15.125:0/1173823848 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff84c0664d0 0x7ff84c068990 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7ff850009ae0 tx=0x7ff8500046d0 comp rx=0 tx=0).stop 2023-10-30T13:24:32.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.983+0000 7ff866079640 1 -- 172.21.15.125:0/1173823848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8601479d0 msgr2=0x7ff860147e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:32.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.983+0000 7ff866079640 1 --2- 172.21.15.125:0/1173823848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8601479d0 0x7ff860147e80 secure :-1 s=READY pgs=2127 cs=0 l=1 rev1=1 crypto rx=0x7ff8600cfd10 tx=0x7ff858011cf0 comp rx=0 tx=0).stop 2023-10-30T13:24:32.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.983+0000 7ff866079640 1 -- 172.21.15.125:0/1173823848 shutdown_connections 2023-10-30T13:24:32.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.983+0000 7ff866079640 1 --2- 172.21.15.125:0/1173823848 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff84c0664d0 0x7ff84c068990 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:32.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.983+0000 7ff866079640 1 --2- 172.21.15.125:0/1173823848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff8601479d0 0x7ff860147e80 unknown :-1 s=CLOSED pgs=2127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:32.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.983+0000 7ff866079640 1 --2- 172.21.15.125:0/1173823848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff86015a8b0 0x7ff8601e36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:32.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.983+0000 7ff866079640 1 --2- 172.21.15.125:0/1173823848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff848003b40 0x7ff8601e31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:32.989 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.983+0000 7ff866079640 1 -- 172.21.15.125:0/1173823848 >> 172.21.15.125:0/1173823848 conn(0x7ff86006bfe0 msgr2=0x7ff860065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:32.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.983+0000 7ff866079640 1 -- 172.21.15.125:0/1173823848 shutdown_connections 2023-10-30T13:24:32.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:32.983+0000 7ff866079640 1 -- 172.21.15.125:0/1173823848 wait complete. 2023-10-30T13:24:33.007 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:35.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:35.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.151+0000 7f8825dc5640 1 Processor -- start 2023-10-30T13:24:35.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.151+0000 7f8825dc5640 1 -- start start 2023-10-30T13:24:35.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f8825dc5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8820159b80 0x7f8820159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:35.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f8825dc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882006b440 0x7f882006b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:35.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f8825dc5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88201566e0 con 0x7f882013f9c0 2023-10-30T13:24:35.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f8825dc5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8820157360 con 0x7f8820159b80 2023-10-30T13:24:35.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f8825dc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f882013fda0 con 0x7f882006b440 2023-10-30T13:24:35.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f881f7fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f882013f9c0 0x7f88200c5a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51672/0 (socket says 172.21.15.125:51672) 2023-10-30T13:24:35.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f881f7fe640 1 -- 172.21.15.125:0/3838452053 learned_addr learned my addr 172.21.15.125:0/3838452053 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:24:35.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f881effd640 1 --2- 172.21.15.125:0/3838452053 >> v2:172.21.15.125:3300/0 conn(0x7f8820159b80 0x7f8820159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:35.158 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f881ffff640 1 --2- 172.21.15.125:0/3838452053 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882006b440 0x7f882006b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:35.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f881e7fc640 1 -- 172.21.15.125:0/3838452053 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491070877 0 0) 0x7f88201566e0 con 0x7f882013f9c0 2023-10-30T13:24:35.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f881e7fc640 1 -- 172.21.15.125:0/3838452053 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8808003530 con 0x7f882013f9c0 2023-10-30T13:24:35.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f881effd640 1 -- 172.21.15.125:0/3838452053 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882006b440 msgr2=0x7f882006b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:35.159 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f881effd640 1 --2- 172.21.15.125:0/3838452053 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882006b440 0x7f882006b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:35.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f881effd640 1 -- 172.21.15.125:0/3838452053 >> v1:172.21.15.125:6789/0 conn(0x7f882013f9c0 legacy=0x7f88200c5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:35.160 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.155+0000 7f881effd640 1 -- 172.21.15.125:0/3838452053 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88201569c0 con 0x7f8820159b80 2023-10-30T13:24:35.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f881effd640 1 --2- 172.21.15.125:0/3838452053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8820159b80 0x7f8820159fa0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f88140097c0 tx=0x7f881401c480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d93544130b5a87d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:35.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f881e7fc640 1 -- 172.21.15.125:0/3838452053 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88140ab600 con 0x7f8820159b80 2023-10-30T13:24:35.161 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f881e7fc640 1 -- 172.21.15.125:0/3838452053 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88140a7050 con 0x7f8820159b80 2023-10-30T13:24:35.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f881e7fc640 1 -- 172.21.15.125:0/3838452053 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88140ab980 con 0x7f8820159b80 2023-10-30T13:24:35.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 -- 172.21.15.125:0/3838452053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8820159b80 msgr2=0x7f8820159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:35.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 --2- 172.21.15.125:0/3838452053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8820159b80 0x7f8820159fa0 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f88140097c0 tx=0x7f881401c480 comp rx=0 tx=0).stop 2023-10-30T13:24:35.162 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 -- 172.21.15.125:0/3838452053 shutdown_connections 2023-10-30T13:24:35.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 --2- 172.21.15.125:0/3838452053 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8820159b80 0x7f8820159fa0 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:35.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 --2- 172.21.15.125:0/3838452053 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882006b440 0x7f882006b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:35.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 -- 172.21.15.125:0/3838452053 >> 172.21.15.125:0/3838452053 conn(0x7f88200cace0 msgr2=0x7f88200cb0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:35.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 -- 172.21.15.125:0/3838452053 shutdown_connections 2023-10-30T13:24:35.163 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 -- 172.21.15.125:0/3838452053 wait complete. 2023-10-30T13:24:35.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 Processor -- start 2023-10-30T13:24:35.165 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 -- start start 2023-10-30T13:24:35.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882006b440 0x7f88201f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:35.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882013f9c0 0x7f88200ca2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:35.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8820159b80 0x7f8820136430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:35.166 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88200c73e0 con 0x7f882006b440 2023-10-30T13:24:35.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88201f2030 con 0x7f8820159b80 2023-10-30T13:24:35.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.159+0000 7f8825dc5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88200c6b40 con 0x7f882013f9c0 2023-10-30T13:24:35.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f881f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882006b440 0x7f88201f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:35.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f881f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882006b440 0x7f88201f1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40888/0 (socket says 172.21.15.125:40888) 2023-10-30T13:24:35.167 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f881f7fe640 1 -- 172.21.15.125:0/2608828571 learned_addr learned my addr 172.21.15.125:0/2608828571 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:35.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f881effd640 1 --2- 172.21.15.125:0/2608828571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882013f9c0 0x7f88200ca2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:35.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f881ffff640 1 --2- 172.21.15.125:0/2608828571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8820159b80 0x7f8820136430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:35.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f881f7fe640 1 -- 172.21.15.125:0/2608828571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882013f9c0 msgr2=0x7f88200ca2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:35.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f881f7fe640 1 --2- 172.21.15.125:0/2608828571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882013f9c0 0x7f88200ca2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:35.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f881f7fe640 1 -- 172.21.15.125:0/2608828571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8820159b80 msgr2=0x7f8820136430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:35.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f881f7fe640 1 --2- 172.21.15.125:0/2608828571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8820159b80 0x7f8820136430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:35.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f881f7fe640 1 -- 172.21.15.125:0/2608828571 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8814009420 con 0x7f882006b440 2023-10-30T13:24:35.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f881f7fe640 1 --2- 172.21.15.125:0/2608828571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882006b440 0x7f88201f1af0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f881000c420 tx=0x7f881000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:35.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f8807fff640 1 -- 172.21.15.125:0/2608828571 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8810005cf0 con 0x7f882006b440 2023-10-30T13:24:35.169 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f8807fff640 1 -- 172.21.15.125:0/2608828571 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8810005e50 con 0x7f882006b440 2023-10-30T13:24:35.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f8807fff640 1 -- 172.21.15.125:0/2608828571 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88100072d0 con 0x7f882006b440 2023-10-30T13:24:35.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f8825dc5640 1 -- 172.21.15.125:0/2608828571 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88200c6e20 con 0x7f882006b440 2023-10-30T13:24:35.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f8825dc5640 1 -- 172.21.15.125:0/2608828571 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88201f12f0 con 0x7f882006b440 2023-10-30T13:24:35.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f8825dc5640 1 -- 172.21.15.125:0/2608828571 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88201f1150 con 0x7f882006b440 2023-10-30T13:24:35.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.163+0000 7f8807fff640 1 -- 172.21.15.125:0/2608828571 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8810007430 con 0x7f882006b440 2023-10-30T13:24:35.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.167+0000 7f8807fff640 1 --2- 172.21.15.125:0/2608828571 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f880c0664a0 0x7f880c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:35.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.167+0000 7f8807fff640 1 -- 172.21.15.125:0/2608828571 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f8810083ed0 con 0x7f882006b440 2023-10-30T13:24:35.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.167+0000 7f881effd640 1 --2- 172.21.15.125:0/2608828571 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f880c0664a0 0x7f880c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:35.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.167+0000 7f881effd640 1 --2- 172.21.15.125:0/2608828571 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f880c0664a0 0x7f880c068960 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f88140ad000 tx=0x7f8814008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:35.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.171+0000 7f8807fff640 1 -- 172.21.15.125:0/2608828571 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f88201f1150 con 0x7f882006b440 2023-10-30T13:24:35.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.351+0000 7f8825dc5640 1 -- 172.21.15.125:0/2608828571 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88201f1150 con 0x7f882006b440 2023-10-30T13:24:35.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.351+0000 7f8807fff640 1 -- 172.21.15.125:0/2608828571 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 261 v261) v1 ==== 95+0+5759 (secure 0 0 0) 0x7f88201f1150 con 0x7f882006b440 2023-10-30T13:24:35.356 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:35.356 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":261,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":261,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:30.642173+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:reconnect","state_seq":35,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:35.358 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 261 2023-10-30T13:24:35.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.355+0000 7f8825dc5640 1 -- 172.21.15.125:0/2608828571 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f880c0664a0 msgr2=0x7f880c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:35.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.355+0000 7f8825dc5640 1 --2- 172.21.15.125:0/2608828571 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f880c0664a0 0x7f880c068960 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f88140ad000 tx=0x7f8814008040 comp rx=0 tx=0).stop 2023-10-30T13:24:35.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.355+0000 7f8825dc5640 1 -- 172.21.15.125:0/2608828571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882006b440 msgr2=0x7f88201f1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:35.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.355+0000 7f8825dc5640 1 --2- 172.21.15.125:0/2608828571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882006b440 0x7f88201f1af0 secure :-1 s=READY pgs=2130 cs=0 l=1 rev1=1 crypto rx=0x7f881000c420 tx=0x7f881000c8e0 comp rx=0 tx=0).stop 2023-10-30T13:24:35.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.355+0000 7f8825dc5640 1 -- 172.21.15.125:0/2608828571 shutdown_connections 2023-10-30T13:24:35.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.355+0000 7f8825dc5640 1 --2- 172.21.15.125:0/2608828571 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f880c0664a0 0x7f880c068960 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:35.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.355+0000 7f8825dc5640 1 --2- 172.21.15.125:0/2608828571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8820159b80 0x7f8820136430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:35.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.355+0000 7f8825dc5640 1 --2- 172.21.15.125:0/2608828571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f882013f9c0 0x7f88200ca2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:35.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.355+0000 7f8825dc5640 1 --2- 172.21.15.125:0/2608828571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f882006b440 0x7f88201f1af0 unknown :-1 s=CLOSED pgs=2130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:35.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.355+0000 7f8825dc5640 1 -- 172.21.15.125:0/2608828571 >> 172.21.15.125:0/2608828571 conn(0x7f88200cace0 msgr2=0x7f88200cb0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:35.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.355+0000 7f8825dc5640 1 -- 172.21.15.125:0/2608828571 shutdown_connections 2023-10-30T13:24:35.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:35.355+0000 7f8825dc5640 1 -- 172.21.15.125:0/2608828571 wait complete. 2023-10-30T13:24:35.377 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:37.377 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:37.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a64b32640 1 Processor -- start 2023-10-30T13:24:37.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a64b32640 1 -- start start 2023-10-30T13:24:37.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a64b32640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1a601538f0 0x7f1a60159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:37.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a64b32640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a6015cd50 0x7f1a600c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:37.526 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a64b32640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6015d300 con 0x7f1a600cefa0 2023-10-30T13:24:37.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a64b32640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a6015d5e0 con 0x7f1a601538f0 2023-10-30T13:24:37.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a64b32640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6015d8c0 con 0x7f1a6015cd50 2023-10-30T13:24:37.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a5f7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1a601538f0 0x7f1a60159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:37.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a5f7fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1a601538f0 0x7f1a60159fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40900/0 (socket says 172.21.15.125:40900) 2023-10-30T13:24:37.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a5f7fe640 1 -- 172.21.15.125:0/1126965786 learned_addr learned my addr 172.21.15.125:0/1126965786 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:37.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a5effd640 1 --2- 172.21.15.125:0/1126965786 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a6015cd50 0x7f1a600c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:37.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a5f7fe640 1 -- 172.21.15.125:0/1126965786 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a6015cd50 msgr2=0x7f1a600c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:37.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a5f7fe640 1 --2- 172.21.15.125:0/1126965786 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a6015cd50 0x7f1a600c9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:37.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a5f7fe640 1 -- 172.21.15.125:0/1126965786 >> v1:172.21.15.136:6789/0 conn(0x7f1a600cefa0 legacy=0x7f1a600cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:37.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.523+0000 7f1a5f7fe640 1 -- 172.21.15.125:0/1126965786 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6015dba0 con 0x7f1a601538f0 2023-10-30T13:24:37.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.527+0000 7f1a5f7fe640 1 --2- 172.21.15.125:0/1126965786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a601538f0 0x7f1a60159fb0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f1a50009870 tx=0x7f1a5001ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f17197eaeb96f4ba server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:37.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.527+0000 7f1a5e7fc640 1 -- 172.21.15.125:0/1126965786 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a500a89b0 con 0x7f1a601538f0 2023-10-30T13:24:37.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.527+0000 7f1a5e7fc640 1 -- 172.21.15.125:0/1126965786 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a500a8b10 con 0x7f1a601538f0 2023-10-30T13:24:37.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.527+0000 7f1a5e7fc640 1 -- 172.21.15.125:0/1126965786 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a500a8ea0 con 0x7f1a601538f0 2023-10-30T13:24:37.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.527+0000 7f1a64b32640 1 -- 172.21.15.125:0/1126965786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a601538f0 msgr2=0x7f1a60159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:37.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.527+0000 7f1a64b32640 1 --2- 172.21.15.125:0/1126965786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a601538f0 0x7f1a60159fb0 secure :-1 s=READY pgs=2131 cs=0 l=1 rev1=1 crypto rx=0x7f1a50009870 tx=0x7f1a5001ecf0 comp rx=0 tx=0).stop 2023-10-30T13:24:37.531 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.527+0000 7f1a64b32640 1 -- 172.21.15.125:0/1126965786 shutdown_connections 2023-10-30T13:24:37.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.527+0000 7f1a64b32640 1 --2- 172.21.15.125:0/1126965786 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a6015cd50 0x7f1a600c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:37.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.527+0000 7f1a64b32640 1 --2- 172.21.15.125:0/1126965786 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a601538f0 0x7f1a60159fb0 unknown :-1 s=CLOSED pgs=2131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:37.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.527+0000 7f1a64b32640 1 -- 172.21.15.125:0/1126965786 >> 172.21.15.125:0/1126965786 conn(0x7f1a6006a640 msgr2=0x7f1a60065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:37.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.527+0000 7f1a64b32640 1 -- 172.21.15.125:0/1126965786 shutdown_connections 2023-10-30T13:24:37.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.527+0000 7f1a64b32640 1 -- 172.21.15.125:0/1126965786 wait complete. 2023-10-30T13:24:37.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.527+0000 7f1a64b32640 1 Processor -- start 2023-10-30T13:24:37.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.527+0000 7f1a64b32640 1 -- start start 2023-10-30T13:24:37.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a64b32640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a600cefa0 0x7f1a600b3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:37.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a64b32640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a601538f0 0x7f1a601f2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:37.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a64b32640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a6015cd50 0x7f1a600c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:37.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a64b32640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a601f32a0 con 0x7f1a6015cd50 2023-10-30T13:24:37.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a64b32640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a601ef590 con 0x7f1a600cefa0 2023-10-30T13:24:37.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a64b32640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a601489a0 con 0x7f1a601538f0 2023-10-30T13:24:37.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a5ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a6015cd50 0x7f1a600c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:37.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a5ffff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a6015cd50 0x7f1a600c9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40908/0 (socket says 172.21.15.125:40908) 2023-10-30T13:24:37.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a5ffff640 1 -- 172.21.15.125:0/2724406138 learned_addr learned my addr 172.21.15.125:0/2724406138 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:37.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a5f7fe640 1 --2- 172.21.15.125:0/2724406138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a600cefa0 0x7f1a600b3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:37.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a5effd640 1 --2- 172.21.15.125:0/2724406138 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a601538f0 0x7f1a601f2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:37.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a5ffff640 1 -- 172.21.15.125:0/2724406138 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a601538f0 msgr2=0x7f1a601f2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:37.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a5ffff640 1 --2- 172.21.15.125:0/2724406138 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a601538f0 0x7f1a601f2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:37.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a5ffff640 1 -- 172.21.15.125:0/2724406138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a600cefa0 msgr2=0x7f1a600b3e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:37.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a5ffff640 1 --2- 172.21.15.125:0/2724406138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a600cefa0 0x7f1a600b3e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:37.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a5ffff640 1 -- 172.21.15.125:0/2724406138 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a500094d0 con 0x7f1a6015cd50 2023-10-30T13:24:37.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a5ffff640 1 --2- 172.21.15.125:0/2724406138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a6015cd50 0x7f1a600c9c90 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f1a5800c780 tx=0x7f1a5800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:37.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a47fff640 1 -- 172.21.15.125:0/2724406138 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a58004b70 con 0x7f1a6015cd50 2023-10-30T13:24:37.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a47fff640 1 -- 172.21.15.125:0/2724406138 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5800ce90 con 0x7f1a6015cd50 2023-10-30T13:24:37.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a47fff640 1 -- 172.21.15.125:0/2724406138 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5800a850 con 0x7f1a6015cd50 2023-10-30T13:24:37.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a64b32640 1 -- 172.21.15.125:0/2724406138 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a601eef80 con 0x7f1a6015cd50 2023-10-30T13:24:37.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a64b32640 1 -- 172.21.15.125:0/2724406138 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a601486c0 con 0x7f1a6015cd50 2023-10-30T13:24:37.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.531+0000 7f1a64b32640 1 -- 172.21.15.125:0/2724406138 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a601eede0 con 0x7f1a6015cd50 2023-10-30T13:24:37.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.535+0000 7f1a47fff640 1 -- 172.21.15.125:0/2724406138 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1a5800a9b0 con 0x7f1a6015cd50 2023-10-30T13:24:37.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.535+0000 7f1a47fff640 1 --2- 172.21.15.125:0/2724406138 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a4c0664a0 0x7f1a4c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:37.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.535+0000 7f1a47fff640 1 -- 172.21.15.125:0/2724406138 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f1a58084b40 con 0x7f1a6015cd50 2023-10-30T13:24:37.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.535+0000 7f1a5f7fe640 1 --2- 172.21.15.125:0/2724406138 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a4c0664a0 0x7f1a4c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:37.542 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.535+0000 7f1a5f7fe640 1 --2- 172.21.15.125:0/2724406138 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a4c0664a0 0x7f1a4c068960 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f1a500b2000 tx=0x7f1a50009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:37.543 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.539+0000 7f1a47fff640 1 -- 172.21.15.125:0/2724406138 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1a601eede0 con 0x7f1a6015cd50 2023-10-30T13:24:37.721 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.719+0000 7f1a64b32640 1 -- 172.21.15.125:0/2724406138 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a601eede0 con 0x7f1a6015cd50 2023-10-30T13:24:37.722 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.719+0000 7f1a47fff640 1 -- 172.21.15.125:0/2724406138 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 262 v262) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f1a601eede0 con 0x7f1a6015cd50 2023-10-30T13:24:37.722 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:37.722 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":262,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:36.411296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:37.724 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 262 2023-10-30T13:24:37.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.719+0000 7f1a64b32640 1 -- 172.21.15.125:0/2724406138 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a4c0664a0 msgr2=0x7f1a4c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:37.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.719+0000 7f1a64b32640 1 --2- 172.21.15.125:0/2724406138 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a4c0664a0 0x7f1a4c068960 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f1a500b2000 tx=0x7f1a50009040 comp rx=0 tx=0).stop 2023-10-30T13:24:37.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.719+0000 7f1a64b32640 1 -- 172.21.15.125:0/2724406138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a6015cd50 msgr2=0x7f1a600c9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:37.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.719+0000 7f1a64b32640 1 --2- 172.21.15.125:0/2724406138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a6015cd50 0x7f1a600c9c90 secure :-1 s=READY pgs=2132 cs=0 l=1 rev1=1 crypto rx=0x7f1a5800c780 tx=0x7f1a5800cc40 comp rx=0 tx=0).stop 2023-10-30T13:24:37.724 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.719+0000 7f1a64b32640 1 -- 172.21.15.125:0/2724406138 shutdown_connections 2023-10-30T13:24:37.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.719+0000 7f1a64b32640 1 --2- 172.21.15.125:0/2724406138 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1a4c0664a0 0x7f1a4c068960 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:37.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.719+0000 7f1a64b32640 1 --2- 172.21.15.125:0/2724406138 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1a6015cd50 0x7f1a600c9c90 unknown :-1 s=CLOSED pgs=2132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:37.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.719+0000 7f1a64b32640 1 --2- 172.21.15.125:0/2724406138 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1a601538f0 0x7f1a601f2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:37.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.719+0000 7f1a64b32640 1 --2- 172.21.15.125:0/2724406138 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1a600cefa0 0x7f1a600b3e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:37.725 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.719+0000 7f1a64b32640 1 -- 172.21.15.125:0/2724406138 >> 172.21.15.125:0/2724406138 conn(0x7f1a6006a640 msgr2=0x7f1a60157bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:37.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.719+0000 7f1a64b32640 1 -- 172.21.15.125:0/2724406138 shutdown_connections 2023-10-30T13:24:37.726 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:37.719+0000 7f1a64b32640 1 -- 172.21.15.125:0/2724406138 wait complete. 2023-10-30T13:24:37.743 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2023-10-30T13:24:39.744 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:39.893 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1d4e17640 1 Processor -- start 2023-10-30T13:24:39.894 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1d4e17640 1 -- start start 2023-10-30T13:24:39.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1d4e17640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc1d00c74b0 0x7fc1d0069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:39.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1d4e17640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc1d00c7df0 0x7fc1d00c8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:39.895 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1d4e17640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1d0153f80 con 0x7fc1d0153ba0 2023-10-30T13:24:39.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1d4e17640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d00c1f50 con 0x7fc1d00c7df0 2023-10-30T13:24:39.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1d4e17640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1d00b0e60 con 0x7fc1d00c74b0 2023-10-30T13:24:39.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1ced9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc1d00c7df0 0x7fc1d00c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:39.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1ced9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc1d00c7df0 0x7fc1d00c8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40910/0 (socket says 172.21.15.125:40910) 2023-10-30T13:24:39.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1ced9d640 1 -- 172.21.15.125:0/2660001003 learned_addr learned my addr 172.21.15.125:0/2660001003 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:39.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1c7fff640 1 --2- 172.21.15.125:0/2660001003 >> v2:172.21.15.136:3300/0 conn(0x7fc1d00c74b0 0x7fc1d0069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:39.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1ced9d640 1 -- 172.21.15.125:0/2660001003 >> v2:172.21.15.136:3300/0 conn(0x7fc1d00c74b0 msgr2=0x7fc1d0069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:39.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1ced9d640 1 --2- 172.21.15.125:0/2660001003 >> v2:172.21.15.136:3300/0 conn(0x7fc1d00c74b0 0x7fc1d0069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:39.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1ced9d640 1 -- 172.21.15.125:0/2660001003 >> v1:172.21.15.136:6789/0 conn(0x7fc1d0153ba0 legacy=0x7fc1d015a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:39.897 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1ced9d640 1 -- 172.21.15.125:0/2660001003 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d00b1140 con 0x7fc1d00c7df0 2023-10-30T13:24:39.898 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.891+0000 7fc1c7fff640 1 --2- 172.21.15.125:0/2660001003 >> v2:172.21.15.136:3300/0 conn(0x7fc1d00c74b0 0x7fc1d0069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:39.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.895+0000 7fc1ced9d640 1 --2- 172.21.15.125:0/2660001003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1d00c7df0 0x7fc1d00c8240 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fc1d006f090 tx=0x7fc1c801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39cdf0559783f7ce server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:39.899 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.895+0000 7fc1cdd9b640 1 -- 172.21.15.125:0/2660001003 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c80a57d0 con 0x7fc1d00c7df0 2023-10-30T13:24:39.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.895+0000 7fc1cdd9b640 1 -- 172.21.15.125:0/2660001003 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c80a5930 con 0x7fc1d00c7df0 2023-10-30T13:24:39.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.895+0000 7fc1cdd9b640 1 -- 172.21.15.125:0/2660001003 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c80a5cc0 con 0x7fc1d00c7df0 2023-10-30T13:24:39.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.895+0000 7fc1d4e17640 1 -- 172.21.15.125:0/2660001003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1d00c7df0 msgr2=0x7fc1d00c8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:39.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.895+0000 7fc1d4e17640 1 --2- 172.21.15.125:0/2660001003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1d00c7df0 0x7fc1d00c8240 secure :-1 s=READY pgs=2133 cs=0 l=1 rev1=1 crypto rx=0x7fc1d006f090 tx=0x7fc1c801c910 comp rx=0 tx=0).stop 2023-10-30T13:24:39.900 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.895+0000 7fc1d4e17640 1 -- 172.21.15.125:0/2660001003 shutdown_connections 2023-10-30T13:24:39.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.895+0000 7fc1d4e17640 1 --2- 172.21.15.125:0/2660001003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1d00c7df0 0x7fc1d00c8240 unknown :-1 s=CLOSED pgs=2133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:39.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.895+0000 7fc1d4e17640 1 --2- 172.21.15.125:0/2660001003 >> v2:172.21.15.136:3300/0 conn(0x7fc1d00c74b0 0x7fc1d0069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:39.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.895+0000 7fc1d4e17640 1 -- 172.21.15.125:0/2660001003 >> 172.21.15.125:0/2660001003 conn(0x7fc1d006d770 msgr2=0x7fc1d006db80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:39.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.895+0000 7fc1d4e17640 1 -- 172.21.15.125:0/2660001003 shutdown_connections 2023-10-30T13:24:39.901 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.895+0000 7fc1d4e17640 1 -- 172.21.15.125:0/2660001003 wait complete. 2023-10-30T13:24:39.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1d4e17640 1 Processor -- start 2023-10-30T13:24:39.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1d4e17640 1 -- start start 2023-10-30T13:24:39.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1d4e17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1d00c74b0 0x7fc1d0148dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:39.904 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1d4e17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1d00c7df0 0x7fc1d014d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:39.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1d4e17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1d0153ba0 0x7fc1d0143860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:39.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1d4e17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d01f23f0 con 0x7fc1d0153ba0 2023-10-30T13:24:39.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1d4e17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d01f2920 con 0x7fc1d00c74b0 2023-10-30T13:24:39.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1d4e17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1d014d090 con 0x7fc1d00c7df0 2023-10-30T13:24:39.905 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1ced9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1d0153ba0 0x7fc1d0143860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:39.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1ced9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1d0153ba0 0x7fc1d0143860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:40914/0 (socket says 172.21.15.125:40914) 2023-10-30T13:24:39.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1ced9d640 1 -- 172.21.15.125:0/858383363 learned_addr learned my addr 172.21.15.125:0/858383363 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:39.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1c7fff640 1 --2- 172.21.15.125:0/858383363 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1d00c7df0 0x7fc1d014d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:39.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1ce59c640 1 --2- 172.21.15.125:0/858383363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1d00c74b0 0x7fc1d0148dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:39.906 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1ced9d640 1 -- 172.21.15.125:0/858383363 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1d00c7df0 msgr2=0x7fc1d014d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:39.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1ced9d640 1 --2- 172.21.15.125:0/858383363 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1d00c7df0 0x7fc1d014d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:39.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1ced9d640 1 -- 172.21.15.125:0/858383363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1d00c74b0 msgr2=0x7fc1d0148dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:39.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1ced9d640 1 --2- 172.21.15.125:0/858383363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1d00c74b0 0x7fc1d0148dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:39.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1ced9d640 1 -- 172.21.15.125:0/858383363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c8009b30 con 0x7fc1d0153ba0 2023-10-30T13:24:39.907 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1ced9d640 1 --2- 172.21.15.125:0/858383363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1d0153ba0 0x7fc1d0143860 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fc1c801a540 tx=0x7fc1c80196b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:39.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1c6ffd640 1 -- 172.21.15.125:0/858383363 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c80b8670 con 0x7fc1d0153ba0 2023-10-30T13:24:39.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1c6ffd640 1 -- 172.21.15.125:0/858383363 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c80b87d0 con 0x7fc1d0153ba0 2023-10-30T13:24:39.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1d4e17640 1 -- 172.21.15.125:0/858383363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1d0147500 con 0x7fc1d0153ba0 2023-10-30T13:24:39.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1c6ffd640 1 -- 172.21.15.125:0/858383363 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c80b8b60 con 0x7fc1d0153ba0 2023-10-30T13:24:39.908 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.899+0000 7fc1d4e17640 1 -- 172.21.15.125:0/858383363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1d01471d0 con 0x7fc1d0153ba0 2023-10-30T13:24:39.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.903+0000 7fc1d4e17640 1 -- 172.21.15.125:0/858383363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1d0074250 con 0x7fc1d0153ba0 2023-10-30T13:24:39.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.903+0000 7fc1c6ffd640 1 -- 172.21.15.125:0/858383363 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc1c80b8cc0 con 0x7fc1d0153ba0 2023-10-30T13:24:39.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.903+0000 7fc1c6ffd640 1 --2- 172.21.15.125:0/858383363 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1b00664a0 0x7fc1b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:39.910 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.903+0000 7fc1c6ffd640 1 -- 172.21.15.125:0/858383363 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fc1c811ff60 con 0x7fc1d0153ba0 2023-10-30T13:24:39.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.903+0000 7fc1ce59c640 1 --2- 172.21.15.125:0/858383363 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1b00664a0 0x7fc1b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:39.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.903+0000 7fc1ce59c640 1 --2- 172.21.15.125:0/858383363 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1b00664a0 0x7fc1b0068960 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0000f70 tx=0x7fc1c0009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:39.911 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:39.907+0000 7fc1c6ffd640 1 -- 172.21.15.125:0/858383363 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc1c80ecb10 con 0x7fc1d0153ba0 2023-10-30T13:24:40.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:40.087+0000 7fc1d4e17640 1 -- 172.21.15.125:0/858383363 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc1d004e170 con 0x7fc1d0153ba0 2023-10-30T13:24:40.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:40.087+0000 7fc1c6ffd640 1 -- 172.21.15.125:0/858383363 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 262 v262) v1 ==== 95+0+5756 (secure 0 0 0) 0x7fc1c80bc090 con 0x7fc1d0153ba0 2023-10-30T13:24:40.092 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:40.092 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":262,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":262,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:36.411296+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:rejoin","state_seq":37,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:40.093 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 262 2023-10-30T13:24:40.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:40.091+0000 7fc1d4e17640 1 -- 172.21.15.125:0/858383363 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1b00664a0 msgr2=0x7fc1b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:40.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:40.091+0000 7fc1d4e17640 1 --2- 172.21.15.125:0/858383363 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1b00664a0 0x7fc1b0068960 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0000f70 tx=0x7fc1c0009040 comp rx=0 tx=0).stop 2023-10-30T13:24:40.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:40.091+0000 7fc1d4e17640 1 -- 172.21.15.125:0/858383363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1d0153ba0 msgr2=0x7fc1d0143860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:40.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:40.091+0000 7fc1d4e17640 1 --2- 172.21.15.125:0/858383363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1d0153ba0 0x7fc1d0143860 secure :-1 s=READY pgs=2134 cs=0 l=1 rev1=1 crypto rx=0x7fc1c801a540 tx=0x7fc1c80196b0 comp rx=0 tx=0).stop 2023-10-30T13:24:40.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:40.091+0000 7fc1d4e17640 1 -- 172.21.15.125:0/858383363 shutdown_connections 2023-10-30T13:24:40.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:40.091+0000 7fc1d4e17640 1 --2- 172.21.15.125:0/858383363 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc1b00664a0 0x7fc1b0068960 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:40.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:40.091+0000 7fc1d4e17640 1 --2- 172.21.15.125:0/858383363 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc1d0153ba0 0x7fc1d0143860 unknown :-1 s=CLOSED pgs=2134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:40.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:40.091+0000 7fc1d4e17640 1 --2- 172.21.15.125:0/858383363 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc1d00c7df0 0x7fc1d014d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:40.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:40.091+0000 7fc1d4e17640 1 --2- 172.21.15.125:0/858383363 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc1d00c74b0 0x7fc1d0148dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:40.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:40.091+0000 7fc1d4e17640 1 -- 172.21.15.125:0/858383363 >> 172.21.15.125:0/858383363 conn(0x7fc1d006d770 msgr2=0x7fc1d014f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:40.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:40.091+0000 7fc1d4e17640 1 -- 172.21.15.125:0/858383363 shutdown_connections 2023-10-30T13:24:40.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:40.091+0000 7fc1d4e17640 1 -- 172.21.15.125:0/858383363 wait complete. 2023-10-30T13:24:40.113 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 262, "feature_flags": { "enable_multiple": true, "ever_enabled_multiple": true }, "filesystems": [ { "id": 1, "mdsmap": { "bal_rank_mask": "-1", "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2023-10-30T12:28:44.410746+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 262, "ever_allowed_features": 0, "explicitly_allowed_features": 0, "failed": [], "flags": 18, "flags_state": { "allow_multimds_snaps": true, "allow_snaps": true, "allow_standby_replay": false, "joinable": true, "refuse_client_session": false, "refuse_standby_for_another_fs": false }, "fs_name": "cephfs", "in": [ 0, 1, 2 ], "info": { "gid_9150": { "addr": "172.21.15.125:6837/796512544", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6834", "nonce": 796512544, "type": "v2" }, { "addr": "172.21.15.125:6837", "nonce": 796512544, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0 ], "features": 4540138322906710015, "flags": 0, "gid": 9150, "incarnation": 214, "join_fscid": -1, "name": "c", "rank": 1, "state": "up:active", "state_seq": 36 }, "gid_9409": { "addr": "172.21.15.136:6839/2056603754", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6838", "nonce": 2056603754, "type": "v2" }, { "addr": "172.21.15.136:6839", "nonce": 2056603754, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 9409, "incarnation": 259, "join_fscid": -1, "name": "f", "rank": 0, "state": "up:rejoin", "state_seq": 37 }, "gid_9600": { "addr": "172.21.15.125:6836/2077416857", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6833", "nonce": 2077416857, "type": "v2" }, { "addr": "172.21.15.125:6836", "nonce": 2077416857, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "export_targets": [ 0, 1 ], "features": 4540138322906710015, "flags": 0, "gid": 9600, "incarnation": 241, "join_fscid": -1, "name": "e", "rank": 2, "state": "up:active", "state_seq": 42 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 3, "max_xattr_size": 65536, "metadata_pool": 2, "modified": "2023-10-30T13:24:36.411296+0000", "required_client_features": {}, "root": 0, "session_autoclose": 300, "session_timeout": 300, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 9409, "mds_1": 9150, "mds_2": 9600 } } } ], "standbys": [ { "addr": "172.21.15.136:6837/2534930361", "addrs": { "addrvec": [ { "addr": "172.21.15.136:6836", "nonce": 2534930361, "type": "v2" }, { "addr": "172.21.15.136:6837", "nonce": 2534930361, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4256, "incarnation": 0, "join_fscid": -1, "name": "b", "rank": -1, "state": "up:standby", "state_seq": 1 }, { "addr": "172.21.15.125:6835/2903453732", "addrs": { "addrvec": [ { "addr": "172.21.15.125:6832", "nonce": 2903453732, "type": "v2" }, { "addr": "172.21.15.125:6835", "nonce": 2903453732, "type": "v1" } ] }, "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_11": "minor log segments", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_7": "mds uses inline data", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "epoch": 3, "export_targets": [], "features": 4540138322906710015, "flags": 0, "gid": 4262, "incarnation": 0, "join_fscid": -1, "name": "a", "rank": -1, "state": "up:standby", "state_seq": 1 } ] } 2023-10-30T13:24:42.114 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:42.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.263+0000 7fcfb4bd3640 1 Processor -- start 2023-10-30T13:24:42.266 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.263+0000 7fcfb4bd3640 1 -- start start 2023-10-30T13:24:42.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.263+0000 7fcfb4bd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfb00c6b80 0x7fcfb00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:42.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.263+0000 7fcfb4bd3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcfb00c6f60 0x7fcfb00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:42.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.263+0000 7fcfb4bd3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fcfb00c8430 0x7fcfb013ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:42.268 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.263+0000 7fcfb4bd3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb00c0a50 con 0x7fcfb00c6f60 2023-10-30T13:24:42.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.263+0000 7fcfb4bd3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb006f6e0 con 0x7fcfb00c8430 2023-10-30T13:24:42.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.263+0000 7fcfb4bd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb006f9c0 con 0x7fcfb00c6b80 2023-10-30T13:24:42.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.263+0000 7fcfa5d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcfb00c6f60 0x7fcfb00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:42.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.263+0000 7fcfa5d9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcfb00c6f60 0x7fcfb00c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35960/0 (socket says 172.21.15.125:35960) 2023-10-30T13:24:42.269 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.263+0000 7fcfa5d9b640 1 -- 172.21.15.125:0/417382729 learned_addr learned my addr 172.21.15.125:0/417382729 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:42.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.263+0000 7fcfaed9d640 1 --2- 172.21.15.125:0/417382729 >> v2:172.21.15.136:3300/0 conn(0x7fcfb00c8430 0x7fcfb013ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:42.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.263+0000 7fcfae59c640 1 --2- 172.21.15.125:0/417382729 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfb00c6b80 0x7fcfb00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:42.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfa5d9b640 1 -- 172.21.15.125:0/417382729 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfb00c6b80 msgr2=0x7fcfb00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:42.270 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfa5d9b640 1 --2- 172.21.15.125:0/417382729 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfb00c6b80 0x7fcfb00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:42.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfa5d9b640 1 -- 172.21.15.125:0/417382729 >> v2:172.21.15.136:3300/0 conn(0x7fcfb00c8430 msgr2=0x7fcfb013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:42.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfa5d9b640 1 --2- 172.21.15.125:0/417382729 >> v2:172.21.15.136:3300/0 conn(0x7fcfb00c8430 0x7fcfb013ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:42.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfa5d9b640 1 -- 172.21.15.125:0/417382729 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb006fca0 con 0x7fcfb00c6f60 2023-10-30T13:24:42.271 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfaed9d640 1 --2- 172.21.15.125:0/417382729 >> v2:172.21.15.136:3300/0 conn(0x7fcfb00c8430 0x7fcfb013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:42.272 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfa5d9b640 1 --2- 172.21.15.125:0/417382729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfb00c6f60 0x7fcfb00c0400 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00b0860 tx=0x7fcf9801e710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab14908cfa0cfc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:42.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfadd9b640 1 -- 172.21.15.125:0/417382729 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf980aa610 con 0x7fcfb00c6f60 2023-10-30T13:24:42.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfadd9b640 1 -- 172.21.15.125:0/417382729 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf980aa770 con 0x7fcfb00c6f60 2023-10-30T13:24:42.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfadd9b640 1 -- 172.21.15.125:0/417382729 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf980aab00 con 0x7fcfb00c6f60 2023-10-30T13:24:42.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/417382729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfb00c6f60 msgr2=0x7fcfb00c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:42.273 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfb4bd3640 1 --2- 172.21.15.125:0/417382729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfb00c6f60 0x7fcfb00c0400 secure :-1 s=READY pgs=2135 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00b0860 tx=0x7fcf9801e710 comp rx=0 tx=0).stop 2023-10-30T13:24:42.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/417382729 shutdown_connections 2023-10-30T13:24:42.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfb4bd3640 1 --2- 172.21.15.125:0/417382729 >> v2:172.21.15.136:3300/0 conn(0x7fcfb00c8430 0x7fcfb013ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:42.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfb4bd3640 1 --2- 172.21.15.125:0/417382729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfb00c6f60 0x7fcfb00c0400 unknown :-1 s=CLOSED pgs=2135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:42.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfb4bd3640 1 --2- 172.21.15.125:0/417382729 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfb00c6b80 0x7fcfb00bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:42.274 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.267+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/417382729 >> 172.21.15.125:0/417382729 conn(0x7fcfb00c7910 msgr2=0x7fcfb00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:42.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/417382729 shutdown_connections 2023-10-30T13:24:42.275 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/417382729 wait complete. 2023-10-30T13:24:42.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfb4bd3640 1 Processor -- start 2023-10-30T13:24:42.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfb4bd3640 1 -- start start 2023-10-30T13:24:42.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfb4bd3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfb00c6b80 0x7fcfb01586b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:42.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfb4bd3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfb00c6f60 0x7fcfb00c1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:42.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfb4bd3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfb00c8430 0x7fcfb01f2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:42.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfb4bd3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb01f24a0 con 0x7fcfb00c6b80 2023-10-30T13:24:42.279 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfb4bd3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb0158bf0 con 0x7fcfb00c8430 2023-10-30T13:24:42.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfb4bd3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb01f2150 con 0x7fcfb00c6f60 2023-10-30T13:24:42.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfae59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfb00c6b80 0x7fcfb01586b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:42.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfae59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfb00c6b80 0x7fcfb01586b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35974/0 (socket says 172.21.15.125:35974) 2023-10-30T13:24:42.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfae59c640 1 -- 172.21.15.125:0/3964711650 learned_addr learned my addr 172.21.15.125:0/3964711650 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:42.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfa5d9b640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfb00c6f60 0x7fcfb00c1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:42.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfaed9d640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfb00c8430 0x7fcfb01f2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:42.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfae59c640 1 -- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfb00c6f60 msgr2=0x7fcfb00c1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:42.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfae59c640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfb00c6f60 0x7fcfb00c1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:42.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfae59c640 1 -- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfb00c8430 msgr2=0x7fcfb01f2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:42.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfae59c640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfb00c8430 0x7fcfb01f2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:42.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfae59c640 1 -- 172.21.15.125:0/3964711650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980094d0 con 0x7fcfb00c6b80 2023-10-30T13:24:42.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfaed9d640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfb00c8430 0x7fcfb01f2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:42.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.271+0000 7fcfa5d9b640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfb00c6f60 0x7fcfb00c1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:42.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.275+0000 7fcfae59c640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfb00c6b80 0x7fcfb01586b0 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0007e60 tx=0x7fcfa000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:42.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.275+0000 7fcfa77fe640 1 -- 172.21.15.125:0/3964711650 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0014620 con 0x7fcfb00c6b80 2023-10-30T13:24:42.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.275+0000 7fcfa77fe640 1 -- 172.21.15.125:0/3964711650 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa000ee90 con 0x7fcfb00c6b80 2023-10-30T13:24:42.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.275+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/3964711650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb00c9420 con 0x7fcfb00c6b80 2023-10-30T13:24:42.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.275+0000 7fcfa77fe640 1 -- 172.21.15.125:0/3964711650 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa00149c0 con 0x7fcfb00c6b80 2023-10-30T13:24:42.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.275+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/3964711650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb01f5b90 con 0x7fcfb00c6b80 2023-10-30T13:24:42.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.275+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/3964711650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb004e170 con 0x7fcfb00c6b80 2023-10-30T13:24:42.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.275+0000 7fcfa77fe640 1 -- 172.21.15.125:0/3964711650 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcfa0014b20 con 0x7fcfb00c6b80 2023-10-30T13:24:42.284 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.275+0000 7fcfa77fe640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcf900664a0 0x7fcf90068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:42.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.275+0000 7fcfa77fe640 1 -- 172.21.15.125:0/3964711650 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fcfa00853c0 con 0x7fcfb00c6b80 2023-10-30T13:24:42.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.275+0000 7fcfa5d9b640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcf900664a0 0x7fcf90068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:42.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.279+0000 7fcfa5d9b640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcf900664a0 0x7fcf90068960 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fcf980b1000 tx=0x7fcf98008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:42.285 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.279+0000 7fcfa77fe640 1 -- 172.21.15.125:0/3964711650 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcfa0051c70 con 0x7fcfb00c6b80 2023-10-30T13:24:42.468 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.467+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/3964711650 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcfb0160110 con 0x7fcfb00c6b80 2023-10-30T13:24:42.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.467+0000 7fcfa77fe640 1 -- 172.21.15.125:0/3964711650 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+5756 (secure 0 0 0) 0x7fcfa0009a20 con 0x7fcfb00c6b80 2023-10-30T13:24:42.469 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:42.470 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:42.059339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:42.470 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 263 2023-10-30T13:24:42.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.467+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcf900664a0 msgr2=0x7fcf90068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:42.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.467+0000 7fcfb4bd3640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcf900664a0 0x7fcf90068960 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fcf980b1000 tx=0x7fcf98008040 comp rx=0 tx=0).stop 2023-10-30T13:24:42.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.467+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/3964711650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfb00c6b80 msgr2=0x7fcfb01586b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:42.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.467+0000 7fcfb4bd3640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfb00c6b80 0x7fcfb01586b0 secure :-1 s=READY pgs=2136 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0007e60 tx=0x7fcfa000e4b0 comp rx=0 tx=0).stop 2023-10-30T13:24:42.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.467+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/3964711650 shutdown_connections 2023-10-30T13:24:42.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.467+0000 7fcfb4bd3640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcf900664a0 0x7fcf90068960 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:42.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.467+0000 7fcfb4bd3640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcfb00c8430 0x7fcfb01f2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:42.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.467+0000 7fcfb4bd3640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcfb00c6f60 0x7fcfb00c1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:42.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.467+0000 7fcfb4bd3640 1 --2- 172.21.15.125:0/3964711650 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcfb00c6b80 0x7fcfb01586b0 unknown :-1 s=CLOSED pgs=2136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:42.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.467+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/3964711650 >> 172.21.15.125:0/3964711650 conn(0x7fcfb00c7910 msgr2=0x7fcfb01380b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:42.473 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.467+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/3964711650 shutdown_connections 2023-10-30T13:24:42.474 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:42.467+0000 7fcfb4bd3640 1 -- 172.21.15.125:0/3964711650 wait complete. 2023-10-30T13:24:42.491 INFO:tasks.mds_thrash.fs.[cephfs]:mds.f has gained rank=0, replacing gid=8033 2023-10-30T13:24:42.491 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.d 2023-10-30T13:24:46.492 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.d 2023-10-30T13:24:46.493 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-30T13:24:46.493 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-30T13:24:46.496 INFO:tasks.ceph.mds.d:Started 2023-10-30T13:24:46.497 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:46.573 INFO:tasks.ceph.mds.d.smithi136.stdout:starting mds.d at 2023-10-30T13:24:46.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.643+0000 7f7792e17640 1 Processor -- start 2023-10-30T13:24:46.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.643+0000 7f7792e17640 1 -- start start 2023-10-30T13:24:46.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.643+0000 7f7792e17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f778c070740 0x7f778c0c6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:46.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.643+0000 7f7792e17640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778c0c5810 con 0x7f778c15e700 2023-10-30T13:24:46.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.647+0000 7f7792e17640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f778c15eb90 con 0x7f778c1580e0 2023-10-30T13:24:46.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.647+0000 7f7792e17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778c06acb0 con 0x7f778c070740 2023-10-30T13:24:46.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.647+0000 7f77913b4640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f778c15e700 0x7f778c06a5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:35446/0 (socket says 172.21.15.125:35446) 2023-10-30T13:24:46.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.647+0000 7f77913b4640 1 -- 172.21.15.125:0/3924245478 learned_addr learned my addr 172.21.15.125:0/3924245478 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:24:46.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.647+0000 7f7783fff640 1 --2- 172.21.15.125:0/3924245478 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f778c070740 0x7f778c0c6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:46.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.647+0000 7f77837fe640 1 -- 172.21.15.125:0/3924245478 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854373503 0 0) 0x7f778c0c5810 con 0x7f778c15e700 2023-10-30T13:24:46.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.647+0000 7f77837fe640 1 -- 172.21.15.125:0/3924245478 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7774003530 con 0x7f778c15e700 2023-10-30T13:24:46.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.647+0000 7f77837fe640 1 -- 172.21.15.125:0/3924245478 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911751674 0 0) 0x7f778c15eb90 con 0x7f778c1580e0 2023-10-30T13:24:46.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.647+0000 7f77837fe640 1 -- 172.21.15.125:0/3924245478 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f778c0c5810 con 0x7f778c1580e0 2023-10-30T13:24:46.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.647+0000 7f7783fff640 1 -- 172.21.15.125:0/3924245478 >> v1:172.21.15.136:6789/0 conn(0x7f778c1580e0 legacy=0x7f778c069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:46.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.647+0000 7f7783fff640 1 -- 172.21.15.125:0/3924245478 >> v1:172.21.15.125:6789/0 conn(0x7f778c15e700 legacy=0x7f778c06a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:46.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.647+0000 7f77837fe640 1 -- 172.21.15.125:0/3924245478 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2368560751 0 0) 0x7f7774003530 con 0x7f778c15e700 2023-10-30T13:24:46.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.647+0000 7f7783fff640 1 -- 172.21.15.125:0/3924245478 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f778c0c6bb0 con 0x7f778c070740 2023-10-30T13:24:46.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.647+0000 7f7783fff640 1 --2- 172.21.15.125:0/3924245478 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f778c070740 0x7f778c0c6540 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f777c0097f0 tx=0x7f777c01c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=494af6ce0bf81353 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:46.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f77837fe640 1 -- 172.21.15.125:0/3924245478 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7784001a00 con 0x7f778c070740 2023-10-30T13:24:46.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f77837fe640 1 -- 172.21.15.125:0/3924245478 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777c0a7050 con 0x7f778c070740 2023-10-30T13:24:46.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f77837fe640 1 -- 172.21.15.125:0/3924245478 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c0ab600 con 0x7f778c070740 2023-10-30T13:24:46.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 -- 172.21.15.125:0/3924245478 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f778c070740 msgr2=0x7f778c0c6540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:46.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 --2- 172.21.15.125:0/3924245478 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f778c070740 0x7f778c0c6540 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f777c0097f0 tx=0x7f777c01c3e0 comp rx=0 tx=0).stop 2023-10-30T13:24:46.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 -- 172.21.15.125:0/3924245478 shutdown_connections 2023-10-30T13:24:46.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 --2- 172.21.15.125:0/3924245478 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f778c070740 0x7f778c0c6540 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:46.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 -- 172.21.15.125:0/3924245478 >> 172.21.15.125:0/3924245478 conn(0x7f778c0c7910 msgr2=0x7f778c0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:46.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 -- 172.21.15.125:0/3924245478 shutdown_connections 2023-10-30T13:24:46.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 -- 172.21.15.125:0/3924245478 wait complete. 2023-10-30T13:24:46.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 Processor -- start 2023-10-30T13:24:46.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 -- start start 2023-10-30T13:24:46.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f778c070740 0x7f778c149d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:46.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f778c1580e0 0x7f778c1f7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:46.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f778c15e700 0x7f778c161400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:46.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c0c5290 con 0x7f778c1580e0 2023-10-30T13:24:46.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f778c1fa1e0 con 0x7f778c15e700 2023-10-30T13:24:46.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7792e17640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f778c1f49c0 con 0x7f778c070740 2023-10-30T13:24:46.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7783fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f778c1580e0 0x7f778c1f7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:46.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7783fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f778c1580e0 0x7f778c1f7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35990/0 (socket says 172.21.15.125:35990) 2023-10-30T13:24:46.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.651+0000 7f7783fff640 1 -- 172.21.15.125:0/4203131538 learned_addr learned my addr 172.21.15.125:0/4203131538 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:46.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f7790bb3640 1 --2- 172.21.15.125:0/4203131538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f778c070740 0x7f778c149d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:46.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f77913b4640 1 --2- 172.21.15.125:0/4203131538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f778c15e700 0x7f778c161400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:46.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f7783fff640 1 -- 172.21.15.125:0/4203131538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f778c070740 msgr2=0x7f778c149d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:46.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f7783fff640 1 --2- 172.21.15.125:0/4203131538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f778c070740 0x7f778c149d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:46.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f7783fff640 1 -- 172.21.15.125:0/4203131538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f778c15e700 msgr2=0x7f778c161400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:46.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f7783fff640 1 --2- 172.21.15.125:0/4203131538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f778c15e700 0x7f778c161400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:46.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f7783fff640 1 -- 172.21.15.125:0/4203131538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c009420 con 0x7f778c1580e0 2023-10-30T13:24:46.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f7783fff640 1 --2- 172.21.15.125:0/4203131538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f778c1580e0 0x7f778c1f7310 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f777c000c10 tx=0x7f777c019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:46.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f77817fa640 1 -- 172.21.15.125:0/4203131538 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c0b4400 con 0x7f778c1580e0 2023-10-30T13:24:46.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f77817fa640 1 -- 172.21.15.125:0/4203131538 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777c0b4560 con 0x7f778c1580e0 2023-10-30T13:24:46.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f77817fa640 1 -- 172.21.15.125:0/4203131538 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777c0b48d0 con 0x7f778c1580e0 2023-10-30T13:24:46.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f7792e17640 1 -- 172.21.15.125:0/4203131538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f778c1f4c60 con 0x7f778c1580e0 2023-10-30T13:24:46.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f7792e17640 1 -- 172.21.15.125:0/4203131538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f778c1f5220 con 0x7f778c1580e0 2023-10-30T13:24:46.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f7792e17640 1 -- 172.21.15.125:0/4203131538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778c04e170 con 0x7f778c1580e0 2023-10-30T13:24:46.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f77817fa640 1 -- 172.21.15.125:0/4203131538 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f777c0b2040 con 0x7f778c1580e0 2023-10-30T13:24:46.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.655+0000 7f77817fa640 1 --2- 172.21.15.125:0/4203131538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f77780664d0 0x7f7778068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:46.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.659+0000 7f77817fa640 1 -- 172.21.15.125:0/4203131538 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f777c11e5c0 con 0x7f778c1580e0 2023-10-30T13:24:46.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.659+0000 7f7790bb3640 1 --2- 172.21.15.125:0/4203131538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f77780664d0 0x7f7778068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:46.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.659+0000 7f7790bb3640 1 --2- 172.21.15.125:0/4203131538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f77780664d0 0x7f7778068990 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f77880011f0 tx=0x7f7788009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:46.664 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.659+0000 7f77817fa640 1 -- 172.21.15.125:0/4203131538 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f777c0eb0a0 con 0x7f778c1580e0 2023-10-30T13:24:46.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.843+0000 7f7792e17640 1 -- 172.21.15.125:0/4203131538 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f778c1f7850 con 0x7f778c1580e0 2023-10-30T13:24:46.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.843+0000 7f77817fa640 1 -- 172.21.15.125:0/4203131538 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 263 v263) v1 ==== 95+0+5756 (secure 0 0 0) 0x7f777c0b9090 con 0x7f778c1580e0 2023-10-30T13:24:46.845 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:46.845 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":263,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:42.059339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:46.846 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 263 2023-10-30T13:24:46.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.843+0000 7f7792e17640 1 -- 172.21.15.125:0/4203131538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f77780664d0 msgr2=0x7f7778068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:46.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.843+0000 7f7792e17640 1 --2- 172.21.15.125:0/4203131538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f77780664d0 0x7f7778068990 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f77880011f0 tx=0x7f7788009040 comp rx=0 tx=0).stop 2023-10-30T13:24:46.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.843+0000 7f7792e17640 1 -- 172.21.15.125:0/4203131538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f778c1580e0 msgr2=0x7f778c1f7310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:46.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.843+0000 7f7792e17640 1 --2- 172.21.15.125:0/4203131538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f778c1580e0 0x7f778c1f7310 secure :-1 s=READY pgs=2139 cs=0 l=1 rev1=1 crypto rx=0x7f777c000c10 tx=0x7f777c019180 comp rx=0 tx=0).stop 2023-10-30T13:24:46.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.843+0000 7f7792e17640 1 -- 172.21.15.125:0/4203131538 shutdown_connections 2023-10-30T13:24:46.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.843+0000 7f7792e17640 1 --2- 172.21.15.125:0/4203131538 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f77780664d0 0x7f7778068990 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:46.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.843+0000 7f7792e17640 1 --2- 172.21.15.125:0/4203131538 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f778c15e700 0x7f778c161400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:46.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.843+0000 7f7792e17640 1 --2- 172.21.15.125:0/4203131538 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f778c1580e0 0x7f778c1f7310 unknown :-1 s=CLOSED pgs=2139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:46.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.843+0000 7f7792e17640 1 --2- 172.21.15.125:0/4203131538 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f778c070740 0x7f778c149d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:46.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.843+0000 7f7792e17640 1 -- 172.21.15.125:0/4203131538 >> 172.21.15.125:0/4203131538 conn(0x7f778c0c7910 msgr2=0x7f778c1557c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:46.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.843+0000 7f7792e17640 1 -- 172.21.15.125:0/4203131538 shutdown_connections 2023-10-30T13:24:46.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:46.843+0000 7f7792e17640 1 -- 172.21.15.125:0/4203131538 wait complete. 2023-10-30T13:24:46.866 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.d is in active, standby or standby-replay 2023-10-30T13:24:48.310 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:24:48.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:24:48.364 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:24:48.364 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.77 2023-10-30T13:24:48.365 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 77 -eq 48 ] 2023-10-30T13:24:48.365 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.77 2023-10-30T13:24:48.365 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.77 2023-10-30T13:24:48.366 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.77 2023-10-30T13:24:48.366 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:24:48.868 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:24:49.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.002+0000 7faee31d3640 1 Processor -- start 2023-10-30T13:24:49.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.002+0000 7faee31d3640 1 -- start start 2023-10-30T13:24:49.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faee31d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7faedc06f7a0 0x7faedc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:49.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faee31d3640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faedc0c6bd0 0x7faedc069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:49.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faee31d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faedc071110 0x7faedc158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:49.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faee31d3640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faedc0700c0 con 0x7faedc0c6bd0 2023-10-30T13:24:49.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faee31d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faedc06a3d0 con 0x7faedc06f7a0 2023-10-30T13:24:49.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faee31d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faedc066cf0 con 0x7faedc071110 2023-10-30T13:24:49.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faedbfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faedc0c6bd0 0x7faedc069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:49.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faedbfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7faedc0c6bd0 0x7faedc069e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:35992/0 (socket says 172.21.15.125:35992) 2023-10-30T13:24:49.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faedbfff640 1 -- 172.21.15.125:0/4068342044 learned_addr learned my addr 172.21.15.125:0/4068342044 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:49.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faee1770640 1 --2- 172.21.15.125:0/4068342044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faedc071110 0x7faedc158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:49.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faee0f6f640 1 --2- 172.21.15.125:0/4068342044 >> v2:172.21.15.136:3300/0 conn(0x7faedc06f7a0 0x7faedc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:49.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faedbfff640 1 -- 172.21.15.125:0/4068342044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faedc071110 msgr2=0x7faedc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:49.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faedbfff640 1 --2- 172.21.15.125:0/4068342044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faedc071110 0x7faedc158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:49.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faedbfff640 1 -- 172.21.15.125:0/4068342044 >> v2:172.21.15.136:3300/0 conn(0x7faedc06f7a0 msgr2=0x7faedc06fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:24:49.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faedbfff640 1 --2- 172.21.15.125:0/4068342044 >> v2:172.21.15.136:3300/0 conn(0x7faedc06f7a0 0x7faedc06fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:49.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.006+0000 7faedbfff640 1 -- 172.21.15.125:0/4068342044 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faedc0c70d0 con 0x7faedc0c6bd0 2023-10-30T13:24:49.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faedbfff640 1 --2- 172.21.15.125:0/4068342044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faedc0c6bd0 0x7faedc069e90 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7faedc06c830 tx=0x7faec801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83fbe6300d9fbbb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:49.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faedb7fe640 1 -- 172.21.15.125:0/4068342044 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec80a89b0 con 0x7faedc0c6bd0 2023-10-30T13:24:49.014 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faedb7fe640 1 -- 172.21.15.125:0/4068342044 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec80a8b10 con 0x7faedc0c6bd0 2023-10-30T13:24:49.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faedb7fe640 1 -- 172.21.15.125:0/4068342044 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec80a8ea0 con 0x7faedc0c6bd0 2023-10-30T13:24:49.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faee31d3640 1 -- 172.21.15.125:0/4068342044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faedc0c6bd0 msgr2=0x7faedc069e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:49.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faee31d3640 1 --2- 172.21.15.125:0/4068342044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faedc0c6bd0 0x7faedc069e90 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7faedc06c830 tx=0x7faec801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:24:49.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faee31d3640 1 -- 172.21.15.125:0/4068342044 shutdown_connections 2023-10-30T13:24:49.015 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faee31d3640 1 --2- 172.21.15.125:0/4068342044 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faedc071110 0x7faedc158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:49.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faee31d3640 1 --2- 172.21.15.125:0/4068342044 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faedc0c6bd0 0x7faedc069e90 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:49.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faee31d3640 1 --2- 172.21.15.125:0/4068342044 >> v2:172.21.15.136:3300/0 conn(0x7faedc06f7a0 0x7faedc06fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:49.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faee31d3640 1 -- 172.21.15.125:0/4068342044 >> 172.21.15.125:0/4068342044 conn(0x7faedc0c7910 msgr2=0x7faedc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:49.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faee31d3640 1 -- 172.21.15.125:0/4068342044 shutdown_connections 2023-10-30T13:24:49.016 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faee31d3640 1 -- 172.21.15.125:0/4068342044 wait complete. 2023-10-30T13:24:49.018 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faee31d3640 1 Processor -- start 2023-10-30T13:24:49.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.010+0000 7faee31d3640 1 -- start start 2023-10-30T13:24:49.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee31d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faedc06f7a0 0x7faedc0bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:49.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee31d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faedc071110 0x7faedc0c8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:49.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee31d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faedc0c6bd0 0x7faedc0adaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:49.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee31d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc1575a0 con 0x7faedc0c6bd0 2023-10-30T13:24:49.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee31d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0c19f0 con 0x7faedc071110 2023-10-30T13:24:49.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee31d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faedc0c1e80 con 0x7faedc06f7a0 2023-10-30T13:24:49.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee1770640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faedc0c6bd0 0x7faedc0adaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:49.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee1770640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faedc0c6bd0 0x7faedc0adaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:36000/0 (socket says 172.21.15.125:36000) 2023-10-30T13:24:49.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee1770640 1 -- 172.21.15.125:0/1860869514 learned_addr learned my addr 172.21.15.125:0/1860869514 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:24:49.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee0f6f640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faedc06f7a0 0x7faedc0bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:49.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faedbfff640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faedc071110 0x7faedc0c8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:49.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee1770640 1 -- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faedc06f7a0 msgr2=0x7faedc0bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:49.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee1770640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faedc06f7a0 0x7faedc0bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:49.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee1770640 1 -- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faedc071110 msgr2=0x7faedc0c8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:49.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee1770640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faedc071110 0x7faedc0c8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:49.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee1770640 1 -- 172.21.15.125:0/1860869514 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec80094d0 con 0x7faedc0c6bd0 2023-10-30T13:24:49.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faedbfff640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faedc071110 0x7faedc0c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:24:49.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee1770640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faedc0c6bd0 0x7faedc0adaa0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7faed4007e60 tx=0x7faed400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:49.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faed97fa640 1 -- 172.21.15.125:0/1860869514 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4014420 con 0x7faedc0c6bd0 2023-10-30T13:24:49.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faed97fa640 1 -- 172.21.15.125:0/1860869514 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed40145a0 con 0x7faedc0c6bd0 2023-10-30T13:24:49.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee31d3640 1 -- 172.21.15.125:0/1860869514 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faedc156fe0 con 0x7faedc0c6bd0 2023-10-30T13:24:49.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faed97fa640 1 -- 172.21.15.125:0/1860869514 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faed4014950 con 0x7faedc0c6bd0 2023-10-30T13:24:49.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee31d3640 1 -- 172.21.15.125:0/1860869514 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faedc0c13a0 con 0x7faedc0c6bd0 2023-10-30T13:24:49.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faee31d3640 1 -- 172.21.15.125:0/1860869514 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faedc0c1ba0 con 0x7faedc0c6bd0 2023-10-30T13:24:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.014+0000 7faed97fa640 1 -- 172.21.15.125:0/1860869514 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7faed4014ab0 con 0x7faedc0c6bd0 2023-10-30T13:24:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.018+0000 7faed97fa640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faec00664a0 0x7faec0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:24:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.018+0000 7faed97fa640 1 -- 172.21.15.125:0/1860869514 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7faed4085150 con 0x7faedc0c6bd0 2023-10-30T13:24:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.018+0000 7faee0f6f640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faec00664a0 0x7faec0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:24:49.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.018+0000 7faee0f6f640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faec00664a0 0x7faec0068960 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7faed00045f0 tx=0x7faed0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:24:49.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.022+0000 7faed97fa640 1 -- 172.21.15.125:0/1860869514 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7faedc0c1ba0 con 0x7faedc0c6bd0 2023-10-30T13:24:49.215 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.210+0000 7faee31d3640 1 -- 172.21.15.125:0/1860869514 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faedc0c1ba0 con 0x7faedc0c6bd0 2023-10-30T13:24:49.216 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.210+0000 7faed97fa640 1 -- 172.21.15.125:0/1860869514 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 264 v264) v1 ==== 95+0+6551 (secure 0 0 0) 0x7faedc0c1ba0 con 0x7faedc0c6bd0 2023-10-30T13:24:49.217 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:24:49.217 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":264,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2118690741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2118690741},{"type":"v1","addr":"172.21.15.136:6835","nonce":2118690741}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":264}],"filesystems":[{"mdsmap":{"epoch":263,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:24:42.059339+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0,1],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:24:49.217 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 264 2023-10-30T13:24:49.218 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.214+0000 7faee31d3640 1 -- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faec00664a0 msgr2=0x7faec0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:49.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.214+0000 7faee31d3640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faec00664a0 0x7faec0068960 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7faed00045f0 tx=0x7faed0008040 comp rx=0 tx=0).stop 2023-10-30T13:24:49.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.214+0000 7faee31d3640 1 -- 172.21.15.125:0/1860869514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faedc0c6bd0 msgr2=0x7faedc0adaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:24:49.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.214+0000 7faee31d3640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faedc0c6bd0 0x7faedc0adaa0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7faed4007e60 tx=0x7faed400e4b0 comp rx=0 tx=0).stop 2023-10-30T13:24:49.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.214+0000 7faee31d3640 1 -- 172.21.15.125:0/1860869514 shutdown_connections 2023-10-30T13:24:49.219 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.214+0000 7faee31d3640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7faec00664a0 0x7faec0068960 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:49.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.214+0000 7faee31d3640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7faedc0c6bd0 0x7faedc0adaa0 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:49.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.214+0000 7faee31d3640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7faedc071110 0x7faedc0c8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:49.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.214+0000 7faee31d3640 1 --2- 172.21.15.125:0/1860869514 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7faedc06f7a0 0x7faedc0bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:24:49.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.214+0000 7faee31d3640 1 -- 172.21.15.125:0/1860869514 >> 172.21.15.125:0/1860869514 conn(0x7faedc0c7910 msgr2=0x7faedc0c2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:24:49.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.214+0000 7faee31d3640 1 -- 172.21.15.125:0/1860869514 shutdown_connections 2023-10-30T13:24:49.220 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:24:49.214+0000 7faee31d3640 1 -- 172.21.15.125:0/1860869514 wait complete. 2023-10-30T13:24:49.237 INFO:tasks.mds_thrash.fs.[cephfs]:mds.d reported in up:standby state 2023-10-30T13:24:49.238 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 113 secs before thrashing 2023-10-30T13:24:49.967 INFO:tasks.ceph.mds.f.smithi136.stderr:2023-10-30T13:24:49.963+0000 7fcfcbdb6640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-30T13:24:49.967 INFO:tasks.ceph.mds.f.smithi136.stderr:2023-10-30T13:24:49.963+0000 7fcfcbdb6640 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-10-30T13:24:49.968 INFO:tasks.ceph.mds.f.smithi136.stderr:2023-10-30T13:24:49.963+0000 7fcfcbdb6640 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2023-10-30T13:24:56.177 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:24:56.177 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.78 2023-10-30T13:24:56.178 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 78 -eq 48 ] 2023-10-30T13:24:56.178 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.78 2023-10-30T13:24:56.178 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.78 2023-10-30T13:24:56.178 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.78 2023-10-30T13:24:56.179 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:25:03.891 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:25:03.892 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.79 2023-10-30T13:25:03.893 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 79 -eq 48 ] 2023-10-30T13:25:03.893 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.79 2023-10-30T13:25:03.893 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.79 2023-10-30T13:25:03.893 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.79 2023-10-30T13:25:03.893 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:25:10.445 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:25:10.446 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.80 2023-10-30T13:25:10.446 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 80 -eq 48 ] 2023-10-30T13:25:10.447 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.80 2023-10-30T13:25:10.447 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.80 2023-10-30T13:25:10.447 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.80 2023-10-30T13:25:10.447 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:25:16.233 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:25:16.233 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 81 -eq 48 ] 2023-10-30T13:25:16.234 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.81 2023-10-30T13:25:16.234 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.81 2023-10-30T13:25:16.234 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.81 2023-10-30T13:25:16.235 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.81 2023-10-30T13:25:16.235 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:25:18.693 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:25:18.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:25:26.200 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd ../.. 2023-10-30T13:25:26.201 INFO:tasks.workunit.client.0.smithi125.stderr:+ [ 82 -eq 48 ] 2023-10-30T13:25:26.201 INFO:tasks.workunit.client.0.smithi125.stdout:checking v0.82 2023-10-30T13:25:26.201 INFO:tasks.workunit.client.0.smithi125.stderr:+ ver=v0.82 2023-10-30T13:25:26.202 INFO:tasks.workunit.client.0.smithi125.stderr:+ echo checking v0.82 2023-10-30T13:25:26.202 INFO:tasks.workunit.client.0.smithi125.stderr:+ cd .snap/v0.82 2023-10-30T13:25:26.202 INFO:tasks.workunit.client.0.smithi125.stderr:+ git diff --exit-code 2023-10-30T13:25:32.383 INFO:tasks.workunit.client.0.smithi125.stdout:diff --git a/.mailmap b/.mailmap 2023-10-30T13:25:32.384 INFO:tasks.workunit.client.0.smithi125.stdout:index 3f0cb6ee549..e69de29bb2d 100644 2023-10-30T13:25:32.384 INFO:tasks.workunit.client.0.smithi125.stdout:--- a/.mailmap 2023-10-30T13:25:32.384 INFO:tasks.workunit.client.0.smithi125.stdout:+++ b/.mailmap 2023-10-30T13:25:32.384 INFO:tasks.workunit.client.0.smithi125.stdout:@@ -1,133 +0,0 @@ 2023-10-30T13:25:32.385 INFO:tasks.workunit.client.0.smithi125.stdout:-# 2023-10-30T13:25:32.385 INFO:tasks.workunit.client.0.smithi125.stdout:-# See .organizationmap for organization affiliation 2023-10-30T13:25:32.385 INFO:tasks.workunit.client.0.smithi125.stdout:-# 2023-10-30T13:25:32.385 INFO:tasks.workunit.client.0.smithi125.stdout:-Sage Weil 2023-10-30T13:25:32.385 INFO:tasks.workunit.client.0.smithi125.stdout:-Sage Weil 2023-10-30T13:25:32.386 INFO:tasks.workunit.client.0.smithi125.stdout:-Sage Weil 2023-10-30T13:25:32.386 INFO:tasks.workunit.client.0.smithi125.stdout:-Sage Weil 2023-10-30T13:25:32.386 INFO:tasks.workunit.client.0.smithi125.stdout:-Sage Weil 2023-10-30T13:25:32.386 INFO:tasks.workunit.client.0.smithi125.stdout:-Sage Weil 2023-10-30T13:25:32.386 INFO:tasks.workunit.client.0.smithi125.stdout:-Sage Weil 2023-10-30T13:25:32.387 INFO:tasks.workunit.client.0.smithi125.stdout:-Sage Weil 2023-10-30T13:25:32.387 INFO:tasks.workunit.client.0.smithi125.stdout:-Sage Weil 2023-10-30T13:25:32.387 INFO:tasks.workunit.client.0.smithi125.stdout:-Sage Weil 2023-10-30T13:25:32.387 INFO:tasks.workunit.client.0.smithi125.stdout:-Sage Weil 2023-10-30T13:25:32.387 INFO:tasks.workunit.client.0.smithi125.stdout:-Yehuda Sadeh 2023-10-30T13:25:32.388 INFO:tasks.workunit.client.0.smithi125.stdout:-Yehuda Sadeh 2023-10-30T13:25:32.388 INFO:tasks.workunit.client.0.smithi125.stdout:-Yehuda Sadeh 2023-10-30T13:25:32.388 INFO:tasks.workunit.client.0.smithi125.stdout:-Yehuda Sadeh 2023-10-30T13:25:32.388 INFO:tasks.workunit.client.0.smithi125.stdout:-Yehuda Sadeh 2023-10-30T13:25:32.388 INFO:tasks.workunit.client.0.smithi125.stdout:-Yehuda Sadeh 2023-10-30T13:25:32.389 INFO:tasks.workunit.client.0.smithi125.stdout:-Yehuda Sadeh 2023-10-30T13:25:32.389 INFO:tasks.workunit.client.0.smithi125.stdout:-Yehuda Sadeh 2023-10-30T13:25:32.389 INFO:tasks.workunit.client.0.smithi125.stdout:-Yehuda Sadeh 2023-10-30T13:25:32.389 INFO:tasks.workunit.client.0.smithi125.stdout:-Colin P. McCabe 2023-10-30T13:25:32.389 INFO:tasks.workunit.client.0.smithi125.stdout:-Colin P. McCabe 2023-10-30T13:25:32.390 INFO:tasks.workunit.client.0.smithi125.stdout:-Greg Farnum 2023-10-30T13:25:32.390 INFO:tasks.workunit.client.0.smithi125.stdout:-Greg Farnum 2023-10-30T13:25:32.390 INFO:tasks.workunit.client.0.smithi125.stdout:-Greg Farnum Gregory Farnum 2023-10-30T13:25:32.390 INFO:tasks.workunit.client.0.smithi125.stdout:-Greg Farnum 2023-10-30T13:25:32.390 INFO:tasks.workunit.client.0.smithi125.stdout:-Greg Farnum 2023-10-30T13:25:32.391 INFO:tasks.workunit.client.0.smithi125.stdout:-Greg Farnum 2023-10-30T13:25:32.391 INFO:tasks.workunit.client.0.smithi125.stdout:-Greg Farnum Greg Farnu 2023-10-30T13:25:32.391 INFO:tasks.workunit.client.0.smithi125.stdout:-Greg Farnum 2023-10-30T13:25:32.391 INFO:tasks.workunit.client.0.smithi125.stdout:-Samuel Just 2023-10-30T13:25:32.391 INFO:tasks.workunit.client.0.smithi125.stdout:-Samuel Just 2023-10-30T13:25:32.391 INFO:tasks.workunit.client.0.smithi125.stdout:-Samuel Just 2023-10-30T13:25:32.392 INFO:tasks.workunit.client.0.smithi125.stdout:-Samuel Just 2023-10-30T13:25:32.392 INFO:tasks.workunit.client.0.smithi125.stdout:-Samuel Just 2023-10-30T13:25:32.392 INFO:tasks.workunit.client.0.smithi125.stdout:-Samuel Just Sam Just 2023-10-30T13:25:32.392 INFO:tasks.workunit.client.0.smithi125.stdout:-Samuel Just athanatos 2023-10-30T13:25:32.392 INFO:tasks.workunit.client.0.smithi125.stdout:-Samuel Just 2023-10-30T13:25:32.393 INFO:tasks.workunit.client.0.smithi125.stdout:-John Wilkins 2023-10-30T13:25:32.394 INFO:tasks.workunit.client.0.smithi125.stdout:-John Wilkins 2023-10-30T13:25:32.394 INFO:tasks.workunit.client.0.smithi125.stdout:-John Wilkins 2023-10-30T13:25:32.394 INFO:tasks.workunit.client.0.smithi125.stdout:-Josh Durgin 2023-10-30T13:25:32.394 INFO:tasks.workunit.client.0.smithi125.stdout:-Josh Durgin 2023-10-30T13:25:32.394 INFO:tasks.workunit.client.0.smithi125.stdout:-Dan Mick 2023-10-30T13:25:32.395 INFO:tasks.workunit.client.0.smithi125.stdout:-Dan Mick 2023-10-30T13:25:32.395 INFO:tasks.workunit.client.0.smithi125.stdout:-Dan Mick 2023-10-30T13:25:32.395 INFO:tasks.workunit.client.0.smithi125.stdout:-Tommi Virtanen 2023-10-30T13:25:32.395 INFO:tasks.workunit.client.0.smithi125.stdout:-Tommi Virtanen 2023-10-30T13:25:32.395 INFO:tasks.workunit.client.0.smithi125.stdout:-Tommi Virtanen 2023-10-30T13:25:32.396 INFO:tasks.workunit.client.0.smithi125.stdout:-João Eduardo Luís 2023-10-30T13:25:32.396 INFO:tasks.workunit.client.0.smithi125.stdout:-João Eduardo Luís Joao Eduardo Luis 2023-10-30T13:25:32.396 INFO:tasks.workunit.client.0.smithi125.stdout:-João Eduardo Luís Joao Luis 2023-10-30T13:25:32.396 INFO:tasks.workunit.client.0.smithi125.stdout:-João Eduardo Luís 2023-10-30T13:25:32.396 INFO:tasks.workunit.client.0.smithi125.stdout:-Sandon Van Ness SandonV 2023-10-30T13:25:32.397 INFO:tasks.workunit.client.0.smithi125.stdout:-Sam Lang 2023-10-30T13:25:32.397 INFO:tasks.workunit.client.0.smithi125.stdout:-Noah Watkins 2023-10-30T13:25:32.397 INFO:tasks.workunit.client.0.smithi125.stdout:-Noah Watkins 2023-10-30T13:25:32.397 INFO:tasks.workunit.client.0.smithi125.stdout:-Gary Lowell 2023-10-30T13:25:32.397 INFO:tasks.workunit.client.0.smithi125.stdout:-Gary Lowell 2023-10-30T13:25:32.398 INFO:tasks.workunit.client.0.smithi125.stdout:-Gary Lowell Gary Lowelll 2023-10-30T13:25:32.398 INFO:tasks.workunit.client.0.smithi125.stdout:-Patience Warnick 2023-10-30T13:25:32.398 INFO:tasks.workunit.client.0.smithi125.stdout:-Wido den Hollander 2023-10-30T13:25:32.398 INFO:tasks.workunit.client.0.smithi125.stdout:-Michael Rodriguez 2023-10-30T13:25:32.398 INFO:tasks.workunit.client.0.smithi125.stdout:-Michael Rodriguez 2023-10-30T13:25:32.399 INFO:tasks.workunit.client.0.smithi125.stdout:-Caleb Miles caleb miles 2023-10-30T13:25:32.399 INFO:tasks.workunit.client.0.smithi125.stdout:-Caleb Miles caleb miles 2023-10-30T13:25:32.399 INFO:tasks.workunit.client.0.smithi125.stdout:-Caleb Miles Caleb Miles 2023-10-30T13:25:32.399 INFO:tasks.workunit.client.0.smithi125.stdout:-Joe Buck 2023-10-30T13:25:32.399 INFO:tasks.workunit.client.0.smithi125.stdout:-Laszlo Boszormenyi Laszlo Boszormenyi (GCS) 2023-10-30T13:25:32.400 INFO:tasks.workunit.client.0.smithi125.stdout:-Roald J. van Loon 2023-10-30T13:25:32.400 INFO:tasks.workunit.client.0.smithi125.stdout:-Alex Elder 2023-10-30T13:25:32.400 INFO:tasks.workunit.client.0.smithi125.stdout:-Alex Elder 2023-10-30T13:25:32.400 INFO:tasks.workunit.client.0.smithi125.stdout:-Alex Elder 2023-10-30T13:25:32.400 INFO:tasks.workunit.client.0.smithi125.stdout:-Alexandre Marangone 2023-10-30T13:25:32.401 INFO:tasks.workunit.client.0.smithi125.stdout:-Alexandre Marangone Alexandre Maragone 2023-10-30T13:25:32.401 INFO:tasks.workunit.client.0.smithi125.stdout:-Alexandre Oliva 2023-10-30T13:25:32.401 INFO:tasks.workunit.client.0.smithi125.stdout:-Alexandre Oliva 2023-10-30T13:25:32.401 INFO:tasks.workunit.client.0.smithi125.stdout:-Ross Turk 2023-10-30T13:25:32.401 INFO:tasks.workunit.client.0.smithi125.stdout:-Ross Turk 2023-10-30T13:25:32.401 INFO:tasks.workunit.client.0.smithi125.stdout:-Patrick McGarry 2023-10-30T13:25:32.402 INFO:tasks.workunit.client.0.smithi125.stdout:-Patrick McGarry scuttlemonkey 2023-10-30T13:25:32.402 INFO:tasks.workunit.client.0.smithi125.stdout:-Mark Nelson 2023-10-30T13:25:32.402 INFO:tasks.workunit.client.0.smithi125.stdout:-Tamil Muthamizhan 2023-10-30T13:25:32.402 INFO:tasks.workunit.client.0.smithi125.stdout:-Tamil Muthamizhan tamil 2023-10-30T13:25:32.402 INFO:tasks.workunit.client.0.smithi125.stdout:-Tamil Muthamizhan 2023-10-30T13:25:32.403 INFO:tasks.workunit.client.0.smithi125.stdout:-Thomas Bechtold 2023-10-30T13:25:32.403 INFO:tasks.workunit.client.0.smithi125.stdout:-Christian Brunner 2023-10-30T13:25:32.403 INFO:tasks.workunit.client.0.smithi125.stdout:-Henry C Chang 2023-10-30T13:25:32.403 INFO:tasks.workunit.client.0.smithi125.stdout:-Alfredo Deza 2023-10-30T13:25:32.403 INFO:tasks.workunit.client.0.smithi125.stdout:-Sylvain Munaut 2023-10-30T13:25:32.404 INFO:tasks.workunit.client.0.smithi125.stdout:-Erwin, Brock A 2023-10-30T13:25:32.404 INFO:tasks.workunit.client.0.smithi125.stdout:-Kacper Kowalik Kacper Kowalik (Xarthisius) 2023-10-30T13:25:32.404 INFO:tasks.workunit.client.0.smithi125.stdout:-Neil Levine 2023-10-30T13:25:32.404 INFO:tasks.workunit.client.0.smithi125.stdout:-Guilhem Lettron 2023-10-30T13:25:32.404 INFO:tasks.workunit.client.0.smithi125.stdout:-Holger Macht 2023-10-30T13:25:32.405 INFO:tasks.workunit.client.0.smithi125.stdout:-Volker Assmann 2023-10-30T13:25:32.405 INFO:tasks.workunit.client.0.smithi125.stdout:-Volker Assmann 2023-10-30T13:25:32.405 INFO:tasks.workunit.client.0.smithi125.stdout:-Sebastien Han 2023-10-30T13:25:32.405 INFO:tasks.workunit.client.0.smithi125.stdout:-Matthew Roy 2023-10-30T13:25:32.405 INFO:tasks.workunit.client.0.smithi125.stdout:-Matthew Roy 2023-10-30T13:25:32.406 INFO:tasks.workunit.client.0.smithi125.stdout:-Matthew Wodrich 2023-10-30T13:25:32.406 INFO:tasks.workunit.client.0.smithi125.stdout:-Christophe Courtaut Kri5 2023-10-30T13:25:32.406 INFO:tasks.workunit.client.0.smithi125.stdout:-Loic Dachary 2023-10-30T13:25:32.406 INFO:tasks.workunit.client.0.smithi125.stdout:-Loic Dachary Loïc Dachary 2023-10-30T13:25:32.406 INFO:tasks.workunit.client.0.smithi125.stdout:-Yan, Zheng Zheng, Yan 2023-10-30T13:25:32.407 INFO:tasks.workunit.client.0.smithi125.stdout:-Yan, Zheng Zheng Yan 2023-10-30T13:25:32.407 INFO:tasks.workunit.client.0.smithi125.stdout:-Yan, Zheng 2023-10-30T13:25:32.407 INFO:tasks.workunit.client.0.smithi125.stdout:-Wang, Yaguang ywang19 2023-10-30T13:25:32.407 INFO:tasks.workunit.client.0.smithi125.stdout:-David Moreau Simard 2023-10-30T13:25:32.407 INFO:tasks.workunit.client.0.smithi125.stdout:-Andrew Leung anwleung 2023-10-30T13:25:32.407 INFO:tasks.workunit.client.0.smithi125.stdout:-Carlos Maltzahn carlosm 2023-10-30T13:25:32.408 INFO:tasks.workunit.client.0.smithi125.stdout:-Esteban Molina-Estolano eestolan 2023-10-30T13:25:32.408 INFO:tasks.workunit.client.0.smithi125.stdout:-Pascal de Bruijn Pascal de Bruijn | Unilogic Networks B.V 2023-10-30T13:25:32.408 INFO:tasks.workunit.client.0.smithi125.stdout:-Huang Jun huangjun 2023-10-30T13:25:32.408 INFO:tasks.workunit.client.0.smithi125.stdout:-Ron Allred rallred 2023-10-30T13:25:32.408 INFO:tasks.workunit.client.0.smithi125.stdout:-Dan Chai danchai 2023-10-30T13:25:32.409 INFO:tasks.workunit.client.0.smithi125.stdout:-Christian Marie 2023-10-30T13:25:32.409 INFO:tasks.workunit.client.0.smithi125.stdout:-Ma Jianpeng majianpeng 2023-10-30T13:25:32.409 INFO:tasks.workunit.client.0.smithi125.stdout:-Eric Mourgaya 2023-10-30T13:25:32.409 INFO:tasks.workunit.client.0.smithi125.stdout:-Danny Al-Gaaf 2023-10-30T13:25:32.409 INFO:tasks.workunit.client.0.smithi125.stdout:-Guang Yang 2023-10-30T13:25:32.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:25:32.411 INFO:tasks.workunit.client.0.smithi125.stdout:-Guang Yang Guang Yang 2023-10-30T13:25:32.411 INFO:tasks.workunit.client.0.smithi125.stdout:-Tyler Brekke Concubidated 2023-10-30T13:25:32.411 INFO:tasks.workunit.client.0.smithi125.stdout:-Huang Jun huang jun 2023-10-30T13:25:32.411 INFO:tasks.workunit.client.0.smithi125.stdout:-Warren Usui wusui 2023-10-30T13:25:32.411 INFO:tasks.workunit.client.0.smithi125.stdout:-Florent Bautista FlorentCoppint 2023-10-30T13:25:32.412 INFO:tasks.workunit.client.0.smithi125.stdout:-Florent Flament 2023-10-30T13:25:32.412 INFO:tasks.workunit.client.0.smithi125.stdout:-Michael Riederer dynamike67 2023-10-30T13:25:32.412 INFO:tasks.workunit.client.0.smithi125.stdout:-Ilya Dryomov 2023-10-30T13:25:32.412 INFO:tasks.workunit.client.0.smithi125.stdout:-Aristoteles Neto 2023-10-30T13:25:32.412 INFO:tasks.workunit.client.0.smithi125.stdout:diff --git a/PendingReleaseNotes b/PendingReleaseNotes 2023-10-30T13:25:32.413 INFO:tasks.workunit.client.0.smithi125.stdout:index aa5db4c1bb4..e69de29bb2d 100644 2023-10-30T13:25:32.413 INFO:tasks.workunit.client.0.smithi125.stdout:--- a/PendingReleaseNotes 2023-10-30T13:25:32.413 INFO:tasks.workunit.client.0.smithi125.stdout:+++ b/PendingReleaseNotes 2023-10-30T13:25:32.413 INFO:tasks.workunit.client.0.smithi125.stdout:@@ -1,4 +0,0 @@ 2023-10-30T13:25:32.413 INFO:tasks.workunit.client.0.smithi125.stdout:- 2023-10-30T13:25:32.414 INFO:tasks.workunit.client.0.smithi125.stdout:-v0.82 2023-10-30T13:25:32.414 INFO:tasks.workunit.client.0.smithi125.stdout:------ 2023-10-30T13:25:32.414 INFO:tasks.workunit.client.0.smithi125.stdout:- 2023-10-30T13:25:32.414 INFO:tasks.workunit:Stopping ['fs/snaps'] on client.0... 2023-10-30T13:25:32.415 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-30T13:25:33.279 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi125 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-10-30T13:25:33.348 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=16ad8708e9a84843ae2d25784cf9a1ab Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_00c1c13cea21a686b046584596fbeac5315a7d71/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi125 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh' 2023-10-30T13:25:33.351 DEBUG:teuthology.run_tasks:Unwinding manager mds_thrash 2023-10-30T13:25:33.361 INFO:tasks.mds_thrash:joining mds_thrasher 2023-10-30T13:25:33.364 INFO:tasks.mds_thrash.fs.[cephfs]:stat['max_mds'] = 3 2023-10-30T13:25:33.364 INFO:tasks.mds_thrash.fs.[cephfs]:stat['deactivate'] = 0 2023-10-30T13:25:33.365 INFO:tasks.mds_thrash.fs.[cephfs]:stat['kill'] = 33 2023-10-30T13:25:33.365 INFO:tasks.mds_thrash:done joining 2023-10-30T13:25:33.366 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-30T13:25:33.375 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-30T13:25:33.375 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T13:25:33.376 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T13:25:33.401 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2023-10-30T13:25:33.401 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-30T13:25:33.452 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:25:33.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2f01365640 1 Processor -- start 2023-10-30T13:25:33.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2f01365640 1 -- start start 2023-10-30T13:25:33.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2f01365640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2efc0c8cf0 0x7f2efc0c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2f01365640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0ca340 0x7f2efc15d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2f01365640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2efc15ce30 con 0x7f2efc0c90d0 2023-10-30T13:25:33.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2f01365640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2efc159880 con 0x7f2efc0c8cf0 2023-10-30T13:25:33.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2f01365640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc159ae0 con 0x7f2efc0ca340 2023-10-30T13:25:33.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2ef27fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2efc0c90d0 0x7f2efc0cef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59584/0 (socket says 172.21.15.125:59584) 2023-10-30T13:25:33.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2ef27fc640 1 -- 172.21.15.125:0/3566496292 learned_addr learned my addr 172.21.15.125:0/3566496292 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:25:33.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efb7fe640 1 --2- 172.21.15.125:0/3566496292 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0ca340 0x7f2efc15d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efaffd640 1 --2- 172.21.15.125:0/3566496292 >> v2:172.21.15.136:3300/0 conn(0x7f2efc0c8cf0 0x7f2efc0c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 -- 172.21.15.125:0/3566496292 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043683514 0 0) 0x7f2efc15ce30 con 0x7f2efc0c90d0 2023-10-30T13:25:33.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 -- 172.21.15.125:0/3566496292 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ee8003530 con 0x7f2efc0c90d0 2023-10-30T13:25:33.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 -- 172.21.15.125:0/3566496292 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2440399165 0 0) 0x7f2ee8003530 con 0x7f2efc0c90d0 2023-10-30T13:25:33.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 -- 172.21.15.125:0/3566496292 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0ca340 msgr2=0x7f2efc15d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:33.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 --2- 172.21.15.125:0/3566496292 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0ca340 0x7f2efc15d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 -- 172.21.15.125:0/3566496292 >> v2:172.21.15.136:3300/0 conn(0x7f2efc0c8cf0 msgr2=0x7f2efc0c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:33.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 --2- 172.21.15.125:0/3566496292 >> v2:172.21.15.136:3300/0 conn(0x7f2efc0c8cf0 0x7f2efc0c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 -- 172.21.15.125:0/3566496292 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc159dc0 con 0x7f2efc0c90d0 2023-10-30T13:25:33.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 -- 172.21.15.125:0/3566496292 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f2ee4001680 con 0x7f2efc0c90d0 2023-10-30T13:25:33.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 -- 172.21.15.125:0/3566496292 >> v1:172.21.15.125:6789/0 conn(0x7f2efc0c90d0 legacy=0x7f2efc0cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:33.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 --2- 172.21.15.125:0/3566496292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee8003980 0x7f2ee8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 --2- 172.21.15.125:0/3566496292 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0ca340 0x7f2ee80047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 --2- 172.21.15.125:0/3566496292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ee8004d10 0x7f2ee8005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 -- 172.21.15.125:0/3566496292 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc159880 con 0x7f2ee8003980 2023-10-30T13:25:33.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 -- 172.21.15.125:0/3566496292 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc159ae0 con 0x7f2ee8004d10 2023-10-30T13:25:33.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efa7fc640 1 -- 172.21.15.125:0/3566496292 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee800b990 con 0x7f2efc0ca340 2023-10-30T13:25:33.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efaffd640 1 --2- 172.21.15.125:0/3566496292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee8003980 0x7f2ee8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efb7fe640 1 --2- 172.21.15.125:0/3566496292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ee8004d10 0x7f2ee8005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2ef27fc640 1 --2- 172.21.15.125:0/3566496292 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0ca340 0x7f2ee80047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efaffd640 1 -- 172.21.15.125:0/3566496292 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0ca340 msgr2=0x7f2ee80047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:33.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efaffd640 1 --2- 172.21.15.125:0/3566496292 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc0ca340 0x7f2ee80047d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efaffd640 1 -- 172.21.15.125:0/3566496292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ee8004d10 msgr2=0x7f2ee8005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:33.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efaffd640 1 --2- 172.21.15.125:0/3566496292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ee8004d10 0x7f2ee8005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efaffd640 1 -- 172.21.15.125:0/3566496292 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee8003530 con 0x7f2ee8003980 2023-10-30T13:25:33.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.630+0000 7f2efb7fe640 1 --2- 172.21.15.125:0/3566496292 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2ee8004d10 0x7f2ee8005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:33.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.634+0000 7f2efaffd640 1 --2- 172.21.15.125:0/3566496292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee8003980 0x7f2ee8003da0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009ae0 tx=0x7f2eec01ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48187b42de27f945 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:33.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.634+0000 7f2efa7fc640 1 -- 172.21.15.125:0/3566496292 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0a5540 con 0x7f2ee8003980 2023-10-30T13:25:33.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.634+0000 7f2efa7fc640 1 -- 172.21.15.125:0/3566496292 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec0a56a0 con 0x7f2ee8003980 2023-10-30T13:25:33.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.634+0000 7f2efa7fc640 1 -- 172.21.15.125:0/3566496292 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0a5a50 con 0x7f2ee8003980 2023-10-30T13:25:33.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.634+0000 7f2f01365640 1 -- 172.21.15.125:0/3566496292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee8003980 msgr2=0x7f2ee8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:33.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.634+0000 7f2f01365640 1 --2- 172.21.15.125:0/3566496292 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee8003980 0x7f2ee8003da0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009ae0 tx=0x7f2eec01ca60 comp rx=0 tx=0).stop 2023-10-30T13:25:33.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.634+0000 7f2efb7fe640 1 -- 172.21.15.125:0/3566496292 reap_dead start 2023-10-30T13:25:33.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.634+0000 7f2f01365640 1 -- 172.21.15.125:0/3566496292 shutdown_connections 2023-10-30T13:25:33.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.634+0000 7f2f01365640 1 -- 172.21.15.125:0/3566496292 >> 172.21.15.125:0/3566496292 conn(0x7f2efc06be40 msgr2=0x7f2efc06c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:33.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.634+0000 7f2f01365640 1 -- 172.21.15.125:0/3566496292 shutdown_connections 2023-10-30T13:25:33.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.634+0000 7f2f01365640 1 -- 172.21.15.125:0/3566496292 wait complete. 2023-10-30T13:25:33.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2f01365640 1 Processor -- start 2023-10-30T13:25:33.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2f01365640 1 -- start start 2023-10-30T13:25:33.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2f01365640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee8003980 0x7f2efc15b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2f01365640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc0ccf80 0x7f2efc1f5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2f01365640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc15bd30 0x7f2efc0c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2f01365640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc15b920 con 0x7f2ee8003980 2023-10-30T13:25:33.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2f01365640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0c8670 con 0x7f2efc0ccf80 2023-10-30T13:25:33.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2f01365640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc1f5bd0 con 0x7f2efc15bd30 2023-10-30T13:25:33.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2efaffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee8003980 0x7f2efc15b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2efaffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee8003980 0x7f2efc15b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38444/0 (socket says 172.21.15.125:38444) 2023-10-30T13:25:33.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2efaffd640 1 -- 172.21.15.125:0/575521511 learned_addr learned my addr 172.21.15.125:0/575521511 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:33.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2ef27fc640 1 --2- 172.21.15.125:0/575521511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc0ccf80 0x7f2efc1f5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2efb7fe640 1 --2- 172.21.15.125:0/575521511 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc15bd30 0x7f2efc0c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2efaffd640 1 -- 172.21.15.125:0/575521511 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc15bd30 msgr2=0x7f2efc0c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:33.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2efaffd640 1 --2- 172.21.15.125:0/575521511 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc15bd30 0x7f2efc0c7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2efaffd640 1 -- 172.21.15.125:0/575521511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc0ccf80 msgr2=0x7f2efc1f5690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:33.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2efaffd640 1 --2- 172.21.15.125:0/575521511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc0ccf80 0x7f2efc1f5690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2efaffd640 1 -- 172.21.15.125:0/575521511 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc15c250 con 0x7f2ee8003980 2023-10-30T13:25:33.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2efaffd640 1 --2- 172.21.15.125:0/575521511 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee8003980 0x7f2efc15b3e0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009ae0 tx=0x7f2ee8005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:33.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2ef3fff640 1 -- 172.21.15.125:0/575521511 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec0a5540 con 0x7f2ee8003980 2023-10-30T13:25:33.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2ef3fff640 1 -- 172.21.15.125:0/575521511 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2eec0a5e90 con 0x7f2ee8003980 2023-10-30T13:25:33.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2f01365640 1 -- 172.21.15.125:0/575521511 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2efc15c530 con 0x7f2ee8003980 2023-10-30T13:25:33.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2ef3fff640 1 -- 172.21.15.125:0/575521511 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2eec004220 con 0x7f2ee8003980 2023-10-30T13:25:33.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.638+0000 7f2f01365640 1 -- 172.21.15.125:0/575521511 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2efc1f4e90 con 0x7f2ee8003980 2023-10-30T13:25:33.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.642+0000 7f2f01365640 1 -- 172.21.15.125:0/575521511 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2efc15ca50 con 0x7f2ee8003980 2023-10-30T13:25:33.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.642+0000 7f2ef3fff640 1 -- 172.21.15.125:0/575521511 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2eec0b4040 con 0x7f2ee8003980 2023-10-30T13:25:33.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.642+0000 7f2ef3fff640 1 --2- 172.21.15.125:0/575521511 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2edc0664a0 0x7f2edc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.642+0000 7f2ef3fff640 1 -- 172.21.15.125:0/575521511 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f2eec11e950 con 0x7f2ee8003980 2023-10-30T13:25:33.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.642+0000 7f2ef27fc640 1 --2- 172.21.15.125:0/575521511 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2edc0664a0 0x7f2edc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.642+0000 7f2ef27fc640 1 --2- 172.21.15.125:0/575521511 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2edc0664a0 0x7f2edc068960 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4001830 tx=0x7f2ee4007040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:33.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.646+0000 7f2ef3fff640 1 -- 172.21.15.125:0/575521511 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2efc15ca50 con 0x7f2ee8003980 2023-10-30T13:25:33.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.798+0000 7f2f01365640 1 -- 172.21.15.125:0/575521511 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2efc15ca50 con 0x7f2ee8003980 2023-10-30T13:25:33.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.798+0000 7f2ef3fff640 1 -- 172.21.15.125:0/575521511 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+20714 (secure 0 0 0) 0x7f2efc15ca50 con 0x7f2ee8003980 2023-10-30T13:25:33.802 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:25:33.803 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T13:24:15.942081+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":55,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:6835/3480968625":"2023-10-31T13:24:15.942040+0000","172.21.15.136:6839/4284345684":"2023-10-31T13:21:50.903213+0000","172.21.15.125:6833/739564283":"2023-10-31T12:57:50.441726+0000","172.21.15.136:6839/1454326577":"2023-10-31T12:40:50.129310+0000","172.21.15.125:6837/2408701650":"2023-10-31T12:52:30.335181+0000","172.21.15.125:6833/1163622496":"2023-10-31T12:55:15.400773+0000","172.21.15.125:6834/1624923396":"2023-10-31T12:42:15.151597+0000","172.21.15.125:6834/2408701650":"2023-10-31T12:52:30.335181+0000","172.21.15.125:6837/3906359913":"2023-10-31T12:59:15.462699+0000","172.21.15.125:6836/1721514617":"2023-10-31T12:51:15.314344+0000","172.21.15.125:6833/2595438444":"2023-10-31T12:46:30.228008+0000","172.21.15.125:6833/1721514617":"2023-10-31T12:51:15.314344+0000","172.21.15.136:6838/1510290691":"2023-10-31T12:47:25.242471+0000","172.21.15.125:6836/2595438444":"2023-10-31T12:46:30.228008+0000","172.21.15.125:6833/2017382874":"2023-10-31T12:36:30.049259+0000","172.21.15.125:6834/2540063352":"2023-10-31T13:12:50.713238+0000","172.21.15.125:6836/3950044434":"2023-10-31T13:17:55.811938+0000","172.21.15.125:6834/2291877604":"2023-10-31T12:32:34.980679+0000","172.21.15.136:6839/3922559697":"2023-10-31T12:44:00.178262+0000","172.21.15.125:6836/1163622496":"2023-10-31T12:55:15.400773+0000","172.21.15.136:6834/3480968625":"2023-10-31T13:24:15.942040+0000","172.21.15.136:6835/3346501886":"2023-10-31T12:30:24.948059+0000","172.21.15.125:6834/3906359913":"2023-10-31T12:59:15.462699+0000","172.21.15.125:6834/2196617378":"2023-10-31T13:05:55.585769+0000","172.21.15.125:6833/3711687505":"2023-10-31T13:06:55.602060+0000","172.21.15.136:6839/2250748765":"2023-10-31T12:35:15.030179+0000","172.21.15.125:6837/2291877604":"2023-10-31T12:32:34.980679+0000","172.21.15.136:6835/3526499585":"2023-10-31T13:11:30.689863+0000","172.21.15.136:6838/1454326577":"2023-10-31T12:40:50.129310+0000","172.21.15.136:6835/695089547":"2023-10-31T13:15:05.766729+0000","172.21.15.136:6839/223464319":"2023-10-31T13:08:55.644039+0000","172.21.15.125:6837/1624923396":"2023-10-31T12:42:15.151597+0000","172.21.15.125:6834/1348687922":"2023-10-31T12:56:35.422955+0000","172.21.15.125:6836/2017382874":"2023-10-31T12:36:30.049259+0000","172.21.15.136:6838/3922559697":"2023-10-31T12:44:00.178262+0000","172.21.15.125:6834/3156415951":"2023-10-31T12:48:25.270273+0000","172.21.15.125:6833/1944683138":"2023-10-31T13:00:40.486003+0000","172.21.15.125:6836/1944683138":"2023-10-31T13:00:40.486003+0000","172.21.15.125:6834/3762807620":"2023-10-31T13:02:35.533843+0000","172.21.15.125:6837/1348687922":"2023-10-31T12:56:35.422955+0000","172.21.15.125:6837/3762807620":"2023-10-31T13:02:35.533843+0000","172.21.15.125:6833/1301336591":"2023-10-31T13:04:45.567511+0000","172.21.15.125:6836/1301336591":"2023-10-31T13:04:45.567511+0000","172.21.15.136:6835/2194892271":"2023-10-31T13:08:00.628452+0000","172.21.15.125:6837/2196617378":"2023-10-31T13:05:55.585769+0000","172.21.15.125:6836/3711687505":"2023-10-31T13:06:55.602060+0000","172.21.15.136:6838/3776186190":"2023-10-31T13:16:00.781626+0000","172.21.15.136:6834/2194892271":"2023-10-31T13:08:00.628452+0000","172.21.15.136:6834/3526499585":"2023-10-31T13:11:30.689863+0000","172.21.15.136:6838/4284345684":"2023-10-31T13:21:50.903213+0000","172.21.15.136:6838/2250748765":"2023-10-31T12:35:15.030179+0000","172.21.15.136:6838/223464319":"2023-10-31T13:08:55.644039+0000","172.21.15.125:6837/3156415951":"2023-10-31T12:48:25.270273+0000","172.21.15.136:6834/695089547":"2023-10-31T13:15:05.766729+0000","172.21.15.125:6837/2540063352":"2023-10-31T13:12:50.713238+0000","172.21.15.136:6834/3346501886":"2023-10-31T12:30:24.948059+0000","172.21.15.136:6839/3776186190":"2023-10-31T13:16:00.781626+0000","172.21.15.125:6833/3950044434":"2023-10-31T13:17:55.811938+0000","172.21.15.136:6839/1510290691":"2023-10-31T12:47:25.242471+0000","172.21.15.136:6834/3864637420":"2023-10-31T13:20:45.886166+0000","172.21.15.125:6836/739564283":"2023-10-31T12:57:50.441726+0000","172.21.15.136:6835/3864637420":"2023-10-31T13:20:45.886166+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:25:33.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.802+0000 7f2f01365640 1 -- 172.21.15.125:0/575521511 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2edc0664a0 msgr2=0x7f2edc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:33.805 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.802+0000 7f2f01365640 1 --2- 172.21.15.125:0/575521511 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2edc0664a0 0x7f2edc068960 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f2ee4001830 tx=0x7f2ee4007040 comp rx=0 tx=0).stop 2023-10-30T13:25:33.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.802+0000 7f2f01365640 1 -- 172.21.15.125:0/575521511 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee8003980 msgr2=0x7f2efc15b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:33.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.802+0000 7f2f01365640 1 --2- 172.21.15.125:0/575521511 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee8003980 0x7f2efc15b3e0 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f2eec009ae0 tx=0x7f2ee8005800 comp rx=0 tx=0).stop 2023-10-30T13:25:33.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.802+0000 7f2f01365640 1 -- 172.21.15.125:0/575521511 shutdown_connections 2023-10-30T13:25:33.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.802+0000 7f2f01365640 1 --2- 172.21.15.125:0/575521511 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2edc0664a0 0x7f2edc068960 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.802+0000 7f2f01365640 1 --2- 172.21.15.125:0/575521511 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2efc15bd30 0x7f2efc0c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.802+0000 7f2f01365640 1 --2- 172.21.15.125:0/575521511 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2efc0ccf80 0x7f2efc1f5690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.802+0000 7f2f01365640 1 --2- 172.21.15.125:0/575521511 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2ee8003980 0x7f2efc15b3e0 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.802+0000 7f2f01365640 1 -- 172.21.15.125:0/575521511 >> 172.21.15.125:0/575521511 conn(0x7f2efc06be40 msgr2=0x7f2efc1590f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:33.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.802+0000 7f2f01365640 1 -- 172.21.15.125:0/575521511 shutdown_connections 2023-10-30T13:25:33.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.802+0000 7f2f01365640 1 -- 172.21.15.125:0/575521511 wait complete. 2023-10-30T13:25:33.825 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-30T12:28:31.063403+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-30T12:28:43.007849+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-30T12:28:43.401231+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '55', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 505, 'snap_epoch': 55, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-30T13:25:33.826 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-30T13:25:33.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.962+0000 7fa9eaf3b640 1 Processor -- start 2023-10-30T13:25:33.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.962+0000 7fa9eaf3b640 1 -- start start 2023-10-30T13:25:33.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.966+0000 7fa9eaf3b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9e4153840 0x7fa9e415a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.966+0000 7fa9eaf3b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa9e4153c20 0x7fa9e40c3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.966+0000 7fa9eaf3b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e415a6b0 con 0x7fa9e40cb4c0 2023-10-30T13:25:33.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.966+0000 7fa9eaf3b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e415dbc0 con 0x7fa9e4153c20 2023-10-30T13:25:33.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.966+0000 7fa9eaf3b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e415d730 con 0x7fa9e4153840 2023-10-30T13:25:33.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.966+0000 7fa9e9738640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa9e4153c20 0x7fa9e40c3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.966+0000 7fa9e9738640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa9e4153c20 0x7fa9e40c3eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38460/0 (socket says 172.21.15.125:38460) 2023-10-30T13:25:33.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.966+0000 7fa9e9738640 1 -- 172.21.15.125:0/4151383522 learned_addr learned my addr 172.21.15.125:0/4151383522 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:33.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.966+0000 7fa9e9f39640 1 --2- 172.21.15.125:0/4151383522 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9e4153840 0x7fa9e415a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.966+0000 7fa9e9738640 1 -- 172.21.15.125:0/4151383522 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9e4153840 msgr2=0x7fa9e415a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:33.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.966+0000 7fa9e9738640 1 --2- 172.21.15.125:0/4151383522 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9e4153840 0x7fa9e415a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.966+0000 7fa9e9738640 1 -- 172.21.15.125:0/4151383522 >> v1:172.21.15.136:6789/0 conn(0x7fa9e40cb4c0 legacy=0x7fa9e40cef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:33.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.966+0000 7fa9e9738640 1 -- 172.21.15.125:0/4151383522 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e40ca8f0 con 0x7fa9e4153c20 2023-10-30T13:25:33.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.966+0000 7fa9e9738640 1 --2- 172.21.15.125:0/4151383522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e4153c20 0x7fa9e40c3eb0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009870 tx=0x7fa9d401ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b78439dfb5dc380 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:33.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9e8f37640 1 -- 172.21.15.125:0/4151383522 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d40a8a30 con 0x7fa9e4153c20 2023-10-30T13:25:33.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9e8f37640 1 -- 172.21.15.125:0/4151383522 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d40a8b90 con 0x7fa9e4153c20 2023-10-30T13:25:33.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9e8f37640 1 -- 172.21.15.125:0/4151383522 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d40b0440 con 0x7fa9e4153c20 2023-10-30T13:25:33.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/4151383522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e4153c20 msgr2=0x7fa9e40c3eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:33.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 --2- 172.21.15.125:0/4151383522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e4153c20 0x7fa9e40c3eb0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009870 tx=0x7fa9d401ed30 comp rx=0 tx=0).stop 2023-10-30T13:25:33.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/4151383522 shutdown_connections 2023-10-30T13:25:33.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 --2- 172.21.15.125:0/4151383522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e4153c20 0x7fa9e40c3eb0 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 --2- 172.21.15.125:0/4151383522 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9e4153840 0x7fa9e415a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/4151383522 >> 172.21.15.125:0/4151383522 conn(0x7fa9e406be40 msgr2=0x7fa9e406c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:33.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/4151383522 shutdown_connections 2023-10-30T13:25:33.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/4151383522 wait complete. 2023-10-30T13:25:33.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 Processor -- start 2023-10-30T13:25:33.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 -- start start 2023-10-30T13:25:33.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e40cb4c0 0x7fa9e40b4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9e4153840 0x7fa9e415bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9e4153c20 0x7fa9e40b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e4154780 con 0x7fa9e40cb4c0 2023-10-30T13:25:33.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e4155070 con 0x7fa9e4153c20 2023-10-30T13:25:33.980 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9eaf3b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e4157c80 con 0x7fa9e4153840 2023-10-30T13:25:33.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.970+0000 7fa9e9f39640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e40cb4c0 0x7fa9e40b4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9e9f39640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e40cb4c0 0x7fa9e40b4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38472/0 (socket says 172.21.15.125:38472) 2023-10-30T13:25:33.981 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9e9f39640 1 -- 172.21.15.125:0/3593900913 learned_addr learned my addr 172.21.15.125:0/3593900913 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:33.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9ea73a640 1 --2- 172.21.15.125:0/3593900913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9e4153c20 0x7fa9e40b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9e9738640 1 --2- 172.21.15.125:0/3593900913 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9e4153840 0x7fa9e415bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.982 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9e9f39640 1 -- 172.21.15.125:0/3593900913 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9e4153840 msgr2=0x7fa9e415bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:33.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9e9f39640 1 --2- 172.21.15.125:0/3593900913 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9e4153840 0x7fa9e415bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9e9f39640 1 -- 172.21.15.125:0/3593900913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9e4153c20 msgr2=0x7fa9e40b30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:33.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9e9f39640 1 --2- 172.21.15.125:0/3593900913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9e4153c20 0x7fa9e40b30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:33.983 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9e9f39640 1 -- 172.21.15.125:0/3593900913 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d40094d0 con 0x7fa9e40cb4c0 2023-10-30T13:25:33.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9e9f39640 1 --2- 172.21.15.125:0/3593900913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e40cb4c0 0x7fa9e40b4bc0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000d800 tx=0x7fa9e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:33.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9da7fc640 1 -- 172.21.15.125:0/3593900913 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e000b6c0 con 0x7fa9e40cb4c0 2023-10-30T13:25:33.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9da7fc640 1 -- 172.21.15.125:0/3593900913 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e000b860 con 0x7fa9e40cb4c0 2023-10-30T13:25:33.984 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/3593900913 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e40b5100 con 0x7fa9e40cb4c0 2023-10-30T13:25:33.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9da7fc640 1 -- 172.21.15.125:0/3593900913 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e000bc10 con 0x7fa9e40cb4c0 2023-10-30T13:25:33.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/3593900913 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e4157e90 con 0x7fa9e40cb4c0 2023-10-30T13:25:33.985 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/3593900913 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e40cfdd0 con 0x7fa9e40cb4c0 2023-10-30T13:25:33.986 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9da7fc640 1 -- 172.21.15.125:0/3593900913 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa9e001e460 con 0x7fa9e40cb4c0 2023-10-30T13:25:33.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9da7fc640 1 --2- 172.21.15.125:0/3593900913 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa9c80664a0 0x7fa9c8068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:33.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.974+0000 7fa9da7fc640 1 -- 172.21.15.125:0/3593900913 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fa9e0084ea0 con 0x7fa9e40cb4c0 2023-10-30T13:25:33.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.978+0000 7fa9e9738640 1 --2- 172.21.15.125:0/3593900913 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa9c80664a0 0x7fa9c8068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:33.987 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.978+0000 7fa9e9738640 1 --2- 172.21.15.125:0/3593900913 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa9c80664a0 0x7fa9c8068960 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40b2000 tx=0x7fa9d4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:33.988 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:33.978+0000 7fa9da7fc640 1 -- 172.21.15.125:0/3593900913 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa9e0051750 con 0x7fa9e40cb4c0 2023-10-30T13:25:34.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.130+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/3593900913 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa9e40b5100 con 0x7fa9e40cb4c0 2023-10-30T13:25:34.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.130+0000 7fa9da7fc640 1 -- 172.21.15.125:0/3593900913 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa9e40b5100 con 0x7fa9e40cb4c0 2023-10-30T13:25:34.136 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 1 2023-10-30T13:25:34.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.134+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/3593900913 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa9c80664a0 msgr2=0x7fa9c8068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.134+0000 7fa9eaf3b640 1 --2- 172.21.15.125:0/3593900913 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa9c80664a0 0x7fa9c8068960 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40b2000 tx=0x7fa9d4008040 comp rx=0 tx=0).stop 2023-10-30T13:25:34.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.134+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/3593900913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e40cb4c0 msgr2=0x7fa9e40b4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.134+0000 7fa9eaf3b640 1 --2- 172.21.15.125:0/3593900913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e40cb4c0 0x7fa9e40b4bc0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7fa9e000d800 tx=0x7fa9e000dcc0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.134+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/3593900913 shutdown_connections 2023-10-30T13:25:34.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.134+0000 7fa9eaf3b640 1 --2- 172.21.15.125:0/3593900913 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa9c80664a0 0x7fa9c8068960 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.134+0000 7fa9eaf3b640 1 --2- 172.21.15.125:0/3593900913 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa9e4153c20 0x7fa9e40b30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.134+0000 7fa9eaf3b640 1 --2- 172.21.15.125:0/3593900913 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa9e4153840 0x7fa9e415bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.134+0000 7fa9eaf3b640 1 --2- 172.21.15.125:0/3593900913 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa9e40cb4c0 0x7fa9e40b4bc0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.134+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/3593900913 >> 172.21.15.125:0/3593900913 conn(0x7fa9e406be40 msgr2=0x7fa9e4152790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:34.140 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.134+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/3593900913 shutdown_connections 2023-10-30T13:25:34.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.134+0000 7fa9eaf3b640 1 -- 172.21.15.125:0/3593900913 wait complete. 2023-10-30T13:25:34.157 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-30T13:25:34.298 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.294+0000 7f800813e640 1 Processor -- start 2023-10-30T13:25:34.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.294+0000 7f800813e640 1 -- start start 2023-10-30T13:25:34.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.294+0000 7f800813e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f80000b0cb0 0x7f800006a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.294+0000 7f800813e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f800006f5b0 0x7f800006ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.294+0000 7f800813e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80001592c0 con 0x7f8000150170 2023-10-30T13:25:34.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.294+0000 7f800813e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f800015a040 con 0x7f800006f5b0 2023-10-30T13:25:34.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.294+0000 7f800813e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8000072170 con 0x7f80000b0cb0 2023-10-30T13:25:34.301 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.298+0000 7f80066db640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f800006f5b0 0x7f800006ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.298+0000 7f80066db640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f800006f5b0 0x7f800006ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38488/0 (socket says 172.21.15.125:38488) 2023-10-30T13:25:34.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.298+0000 7f80066db640 1 -- 172.21.15.125:0/3512145370 learned_addr learned my addr 172.21.15.125:0/3512145370 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:34.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.298+0000 7f80056d9640 1 --2- 172.21.15.125:0/3512145370 >> v2:172.21.15.136:3300/0 conn(0x7f80000b0cb0 0x7f800006a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.298+0000 7f80066db640 1 -- 172.21.15.125:0/3512145370 >> v2:172.21.15.136:3300/0 conn(0x7f80000b0cb0 msgr2=0x7f800006a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:34.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.298+0000 7f80066db640 1 --2- 172.21.15.125:0/3512145370 >> v2:172.21.15.136:3300/0 conn(0x7f80000b0cb0 0x7f800006a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.298+0000 7f80066db640 1 -- 172.21.15.125:0/3512145370 >> v1:172.21.15.136:6789/0 conn(0x7f8000150170 legacy=0x7f8000155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.298+0000 7f80066db640 1 -- 172.21.15.125:0/3512145370 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8000159560 con 0x7f800006f5b0 2023-10-30T13:25:34.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.298+0000 7f80056d9640 1 --2- 172.21.15.125:0/3512145370 >> v2:172.21.15.136:3300/0 conn(0x7f80000b0cb0 0x7f800006a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:34.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.298+0000 7f80066db640 1 --2- 172.21.15.125:0/3512145370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800006f5b0 0x7f800006ac60 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f80000c71d0 tx=0x7f7ff801c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689da07866afbfe3 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:34.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.298+0000 7f8004ed8640 1 -- 172.21.15.125:0/3512145370 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff80a57e0 con 0x7f800006f5b0 2023-10-30T13:25:34.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.298+0000 7f8004ed8640 1 -- 172.21.15.125:0/3512145370 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff80a5940 con 0x7f800006f5b0 2023-10-30T13:25:34.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.298+0000 7f8004ed8640 1 -- 172.21.15.125:0/3512145370 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff80a5cd0 con 0x7f800006f5b0 2023-10-30T13:25:34.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 -- 172.21.15.125:0/3512145370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800006f5b0 msgr2=0x7f800006ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 --2- 172.21.15.125:0/3512145370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800006f5b0 0x7f800006ac60 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f80000c71d0 tx=0x7f7ff801c9a0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 -- 172.21.15.125:0/3512145370 shutdown_connections 2023-10-30T13:25:34.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 --2- 172.21.15.125:0/3512145370 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800006f5b0 0x7f800006ac60 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 --2- 172.21.15.125:0/3512145370 >> v2:172.21.15.136:3300/0 conn(0x7f80000b0cb0 0x7f800006a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.307 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 -- 172.21.15.125:0/3512145370 >> 172.21.15.125:0/3512145370 conn(0x7f80000c7910 msgr2=0x7f80000c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:34.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 -- 172.21.15.125:0/3512145370 shutdown_connections 2023-10-30T13:25:34.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 -- 172.21.15.125:0/3512145370 wait complete. 2023-10-30T13:25:34.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 Processor -- start 2023-10-30T13:25:34.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 -- start start 2023-10-30T13:25:34.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800006f5b0 0x7f800013f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f80000b0cb0 0x7f800014d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8000150170 0x7f800014dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f800013ef50 con 0x7f800006f5b0 2023-10-30T13:25:34.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80001efa00 con 0x7f8000150170 2023-10-30T13:25:34.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f800813e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80001f2cc0 con 0x7f80000b0cb0 2023-10-30T13:25:34.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f8005eda640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800006f5b0 0x7f800013f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f8005eda640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800006f5b0 0x7f800013f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38498/0 (socket says 172.21.15.125:38498) 2023-10-30T13:25:34.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f8005eda640 1 -- 172.21.15.125:0/3495734492 learned_addr learned my addr 172.21.15.125:0/3495734492 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:34.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f80056d9640 1 --2- 172.21.15.125:0/3495734492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f80000b0cb0 0x7f800014d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f80066db640 1 --2- 172.21.15.125:0/3495734492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8000150170 0x7f800014dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f8005eda640 1 -- 172.21.15.125:0/3495734492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f80000b0cb0 msgr2=0x7f800014d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f8005eda640 1 --2- 172.21.15.125:0/3495734492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f80000b0cb0 0x7f800014d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f8005eda640 1 -- 172.21.15.125:0/3495734492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8000150170 msgr2=0x7f800014dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f8005eda640 1 --2- 172.21.15.125:0/3495734492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8000150170 0x7f800014dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f8005eda640 1 -- 172.21.15.125:0/3495734492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff8009b30 con 0x7f800006f5b0 2023-10-30T13:25:34.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f8005eda640 1 --2- 172.21.15.125:0/3495734492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800006f5b0 0x7f800013f3d0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00c450 tx=0x7f7ffc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:34.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.302+0000 7f7ff67fc640 1 -- 172.21.15.125:0/3495734492 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0052d0 con 0x7f800006f5b0 2023-10-30T13:25:34.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.306+0000 7f7ff67fc640 1 -- 172.21.15.125:0/3495734492 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ffc005430 con 0x7f800006f5b0 2023-10-30T13:25:34.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.306+0000 7f7ff67fc640 1 -- 172.21.15.125:0/3495734492 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc003890 con 0x7f800006f5b0 2023-10-30T13:25:34.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.306+0000 7f800813e640 1 -- 172.21.15.125:0/3495734492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80001f2fa0 con 0x7f800006f5b0 2023-10-30T13:25:34.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.306+0000 7f800813e640 1 -- 172.21.15.125:0/3495734492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80001f3560 con 0x7f800006f5b0 2023-10-30T13:25:34.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.306+0000 7f800813e640 1 -- 172.21.15.125:0/3495734492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f800004e170 con 0x7f800006f5b0 2023-10-30T13:25:34.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.306+0000 7f7ff67fc640 1 -- 172.21.15.125:0/3495734492 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f7ffc0039f0 con 0x7f800006f5b0 2023-10-30T13:25:34.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.306+0000 7f7ff67fc640 1 --2- 172.21.15.125:0/3495734492 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fe40664a0 0x7f7fe4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.306+0000 7f7ff67fc640 1 -- 172.21.15.125:0/3495734492 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f7ffc0845f0 con 0x7f800006f5b0 2023-10-30T13:25:34.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.306+0000 7f80056d9640 1 --2- 172.21.15.125:0/3495734492 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fe40664a0 0x7f7fe4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.310+0000 7f80056d9640 1 --2- 172.21.15.125:0/3495734492 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fe40664a0 0x7f7fe4068960 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f80000c7ed0 tx=0x7f7ff0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:34.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.310+0000 7f7ff67fc640 1 -- 172.21.15.125:0/3495734492 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f7ffc050ea0 con 0x7f800006f5b0 2023-10-30T13:25:34.466 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.462+0000 7f800813e640 1 -- 172.21.15.125:0/3495734492 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f800013f1b0 con 0x7f800006f5b0 2023-10-30T13:25:34.467 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.462+0000 7f7ff67fc640 1 -- 172.21.15.125:0/3495734492 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7ffc0544c0 con 0x7f800006f5b0 2023-10-30T13:25:34.467 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 32 2023-10-30T13:25:34.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.466+0000 7f800813e640 1 -- 172.21.15.125:0/3495734492 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fe40664a0 msgr2=0x7f7fe4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.466+0000 7f800813e640 1 --2- 172.21.15.125:0/3495734492 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fe40664a0 0x7f7fe4068960 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f80000c7ed0 tx=0x7f7ff0008040 comp rx=0 tx=0).stop 2023-10-30T13:25:34.469 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.466+0000 7f800813e640 1 -- 172.21.15.125:0/3495734492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800006f5b0 msgr2=0x7f800013f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.466+0000 7f800813e640 1 --2- 172.21.15.125:0/3495734492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800006f5b0 0x7f800013f3d0 secure :-1 s=READY pgs=2148 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00c450 tx=0x7f7ffc00c910 comp rx=0 tx=0).stop 2023-10-30T13:25:34.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.466+0000 7f800813e640 1 -- 172.21.15.125:0/3495734492 shutdown_connections 2023-10-30T13:25:34.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.466+0000 7f800813e640 1 --2- 172.21.15.125:0/3495734492 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f7fe40664a0 0x7f7fe4068960 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.470 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.466+0000 7f800813e640 1 --2- 172.21.15.125:0/3495734492 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8000150170 0x7f800014dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.466+0000 7f800813e640 1 --2- 172.21.15.125:0/3495734492 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f80000b0cb0 0x7f800014d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.466+0000 7f800813e640 1 --2- 172.21.15.125:0/3495734492 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f800006f5b0 0x7f800013f3d0 unknown :-1 s=CLOSED pgs=2148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.471 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.466+0000 7f800813e640 1 -- 172.21.15.125:0/3495734492 >> 172.21.15.125:0/3495734492 conn(0x7f80000c7910 msgr2=0x7f800006d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:34.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.466+0000 7f800813e640 1 -- 172.21.15.125:0/3495734492 shutdown_connections 2023-10-30T13:25:34.472 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.466+0000 7f800813e640 1 -- 172.21.15.125:0/3495734492 wait complete. 2023-10-30T13:25:34.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-30T13:25:34.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f8194640 1 Processor -- start 2023-10-30T13:25:34.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f8194640 1 -- start start 2023-10-30T13:25:34.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f8194640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75f014fe80 0x7f75f006c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f8194640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f75f006b2a0 0x7f75f006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f8194640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f0156640 con 0x7f75f00c7980 2023-10-30T13:25:34.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f8194640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f0156bc0 con 0x7f75f014fe80 2023-10-30T13:25:34.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f8194640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f00c5a50 con 0x7f75f006b2a0 2023-10-30T13:25:34.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f572f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75f014fe80 0x7f75f006c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f572f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f75f014fe80 0x7f75f006c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38510/0 (socket says 172.21.15.125:38510) 2023-10-30T13:25:34.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f572f640 1 -- 172.21.15.125:0/264607522 learned_addr learned my addr 172.21.15.125:0/264607522 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:34.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f6731640 1 --2- 172.21.15.125:0/264607522 >> v2:172.21.15.136:3300/0 conn(0x7f75f006b2a0 0x7f75f006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f572f640 1 -- 172.21.15.125:0/264607522 >> v2:172.21.15.136:3300/0 conn(0x7f75f006b2a0 msgr2=0x7f75f006b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:34.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f572f640 1 --2- 172.21.15.125:0/264607522 >> v2:172.21.15.136:3300/0 conn(0x7f75f006b2a0 0x7f75f006b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f572f640 1 -- 172.21.15.125:0/264607522 >> v1:172.21.15.136:6789/0 conn(0x7f75f00c7980 legacy=0x7f75f0065a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.634 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.626+0000 7f75f572f640 1 -- 172.21.15.125:0/264607522 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f01502a0 con 0x7f75f014fe80 2023-10-30T13:25:34.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.630+0000 7f75f572f640 1 --2- 172.21.15.125:0/264607522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f014fe80 0x7f75f006c280 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f75f00723f0 tx=0x7f75e001c470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76949a1751d0532c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:34.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.630+0000 7f75f4f2e640 1 -- 172.21.15.125:0/264607522 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e00ab600 con 0x7f75f014fe80 2023-10-30T13:25:34.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.630+0000 7f75f4f2e640 1 -- 172.21.15.125:0/264607522 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e00a7050 con 0x7f75f014fe80 2023-10-30T13:25:34.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.630+0000 7f75f4f2e640 1 -- 172.21.15.125:0/264607522 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e00ab980 con 0x7f75f014fe80 2023-10-30T13:25:34.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.630+0000 7f75f8194640 1 -- 172.21.15.125:0/264607522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f014fe80 msgr2=0x7f75f006c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.630+0000 7f75f8194640 1 --2- 172.21.15.125:0/264607522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f014fe80 0x7f75f006c280 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7f75f00723f0 tx=0x7f75e001c470 comp rx=0 tx=0).stop 2023-10-30T13:25:34.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.630+0000 7f75f8194640 1 -- 172.21.15.125:0/264607522 shutdown_connections 2023-10-30T13:25:34.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.630+0000 7f75f8194640 1 --2- 172.21.15.125:0/264607522 >> v2:172.21.15.136:3300/0 conn(0x7f75f006b2a0 0x7f75f006b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.630+0000 7f75f8194640 1 --2- 172.21.15.125:0/264607522 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f014fe80 0x7f75f006c280 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.630+0000 7f75f8194640 1 -- 172.21.15.125:0/264607522 >> 172.21.15.125:0/264607522 conn(0x7f75f00c7fe0 msgr2=0x7f75f00c83f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:34.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.630+0000 7f75f8194640 1 -- 172.21.15.125:0/264607522 shutdown_connections 2023-10-30T13:25:34.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.630+0000 7f75f8194640 1 -- 172.21.15.125:0/264607522 wait complete. 2023-10-30T13:25:34.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f8194640 1 Processor -- start 2023-10-30T13:25:34.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f8194640 1 -- start start 2023-10-30T13:25:34.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f8194640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f006b2a0 0x7f75f0152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f8194640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75f00c7980 0x7f75f0154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f8194640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f014fe80 0x7f75f00c0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f8194640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f0152df0 con 0x7f75f014fe80 2023-10-30T13:25:34.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f8194640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f00bf4b0 con 0x7f75f006b2a0 2023-10-30T13:25:34.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f8194640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f00bf790 con 0x7f75f00c7980 2023-10-30T13:25:34.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f6731640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f014fe80 0x7f75f00c0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f6731640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f014fe80 0x7f75f00c0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38524/0 (socket says 172.21.15.125:38524) 2023-10-30T13:25:34.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f6731640 1 -- 172.21.15.125:0/869363016 learned_addr learned my addr 172.21.15.125:0/869363016 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:34.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f572f640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75f00c7980 0x7f75f0154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f5f30640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f006b2a0 0x7f75f0152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f6731640 1 -- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75f00c7980 msgr2=0x7f75f0154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f6731640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75f00c7980 0x7f75f0154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f6731640 1 -- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f006b2a0 msgr2=0x7f75f0152660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f6731640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f006b2a0 0x7f75f0152660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f6731640 1 -- 172.21.15.125:0/869363016 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75e00094d0 con 0x7f75f014fe80 2023-10-30T13:25:34.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f572f640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75f00c7980 0x7f75f0154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:34.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f6731640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f014fe80 0x7f75f00c0b10 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f75e800efc0 tx=0x7f75e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:34.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75e67fc640 1 -- 172.21.15.125:0/869363016 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e8009ba0 con 0x7f75f014fe80 2023-10-30T13:25:34.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75e67fc640 1 -- 172.21.15.125:0/869363016 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e800f040 con 0x7f75f014fe80 2023-10-30T13:25:34.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75e67fc640 1 -- 172.21.15.125:0/869363016 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75e80146c0 con 0x7f75f014fe80 2023-10-30T13:25:34.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f8194640 1 -- 172.21.15.125:0/869363016 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f00bfa70 con 0x7f75f014fe80 2023-10-30T13:25:34.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.634+0000 7f75f8194640 1 -- 172.21.15.125:0/869363016 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f00c0030 con 0x7f75f014fe80 2023-10-30T13:25:34.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.638+0000 7f75f8194640 1 -- 172.21.15.125:0/869363016 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f004e170 con 0x7f75f014fe80 2023-10-30T13:25:34.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.638+0000 7f75e67fc640 1 -- 172.21.15.125:0/869363016 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f75e8014820 con 0x7f75f014fe80 2023-10-30T13:25:34.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.638+0000 7f75e67fc640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75d40664a0 0x7f75d4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.638+0000 7f75e67fc640 1 -- 172.21.15.125:0/869363016 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f75e80852c0 con 0x7f75f014fe80 2023-10-30T13:25:34.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.638+0000 7f75f5f30640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75d40664a0 0x7f75d4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.642+0000 7f75f5f30640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75d40664a0 0x7f75d4068960 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f75ec007fd0 tx=0x7f75ec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:34.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.642+0000 7f75e67fc640 1 -- 172.21.15.125:0/869363016 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f75e8051cf0 con 0x7f75f014fe80 2023-10-30T13:25:34.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.790+0000 7f75f8194640 1 -- 172.21.15.125:0/869363016 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f75f006c8d0 con 0x7f75f014fe80 2023-10-30T13:25:34.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.790+0000 7f75e67fc640 1 -- 172.21.15.125:0/869363016 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7f75e80180a0 con 0x7f75f014fe80 2023-10-30T13:25:34.796 INFO:teuthology.orchestra.run.smithi125.stdout:pg_num: 64 2023-10-30T13:25:34.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.794+0000 7f75f8194640 1 -- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75d40664a0 msgr2=0x7f75d4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.794+0000 7f75f8194640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75d40664a0 0x7f75d4068960 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f75ec007fd0 tx=0x7f75ec009040 comp rx=0 tx=0).stop 2023-10-30T13:25:34.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.794+0000 7f75f8194640 1 -- 172.21.15.125:0/869363016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f014fe80 msgr2=0x7f75f00c0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.794+0000 7f75f8194640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f014fe80 0x7f75f00c0b10 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7f75e800efc0 tx=0x7f75e800c490 comp rx=0 tx=0).stop 2023-10-30T13:25:34.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.794+0000 7f75f8194640 1 -- 172.21.15.125:0/869363016 shutdown_connections 2023-10-30T13:25:34.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.794+0000 7f75f8194640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f75d40664a0 0x7f75d4068960 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.794+0000 7f75f8194640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f75f014fe80 0x7f75f00c0b10 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.794+0000 7f75f8194640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f75f00c7980 0x7f75f0154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.794+0000 7f75f8194640 1 --2- 172.21.15.125:0/869363016 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f75f006b2a0 0x7f75f0152660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.794+0000 7f75f8194640 1 -- 172.21.15.125:0/869363016 >> 172.21.15.125:0/869363016 conn(0x7f75f00c7fe0 msgr2=0x7f75f014f3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:34.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.794+0000 7f75f8194640 1 -- 172.21.15.125:0/869363016 shutdown_connections 2023-10-30T13:25:34.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.794+0000 7f75f8194640 1 -- 172.21.15.125:0/869363016 wait complete. 2023-10-30T13:25:34.817 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-30T13:25:34.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5e00a54640 1 Processor -- start 2023-10-30T13:25:34.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5e00a54640 1 -- start start 2023-10-30T13:25:34.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5e00a54640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dfc153d10 0x7f5dfc0c5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5e00a54640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5dfc0c64a0 0x7f5dfc15a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5e00a54640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dfc15ce10 con 0x7f5dfc15a540 2023-10-30T13:25:34.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5e00a54640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dfc159820 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5e00a54640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc0c8430 con 0x7f5dfc153d10 2023-10-30T13:25:34.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5dfa59c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dfc153d10 0x7f5dfc0c5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5df1d9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5dfc0c64a0 0x7f5dfc15a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5df1d9b640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5dfc0c64a0 0x7f5dfc15a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:51724/0 (socket says 172.21.15.125:51724) 2023-10-30T13:25:34.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5dfa59c640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dfc153d10 0x7f5dfc0c5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:48096/0 (socket says 172.21.15.125:48096) 2023-10-30T13:25:34.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5dfad9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5dfc15a540 0x7f5dfc0cb570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:44596/0 (socket says 172.21.15.125:44596) 2023-10-30T13:25:34.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5dfa59c640 1 -- 172.21.15.125:0/2338472973 learned_addr learned my addr 172.21.15.125:0/2338472973 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:34.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5df9d9b640 1 -- 172.21.15.125:0/2338472973 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012702551 0 0) 0x7f5dfc15ce10 con 0x7f5dfc15a540 2023-10-30T13:25:34.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5df9d9b640 1 -- 172.21.15.125:0/2338472973 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5de8003530 con 0x7f5dfc15a540 2023-10-30T13:25:34.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5df1d9b640 1 -- 172.21.15.125:0/2338472973 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dfc153d10 msgr2=0x7f5dfc0c5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:34.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5df1d9b640 1 --2- 172.21.15.125:0/2338472973 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dfc153d10 0x7f5dfc0c5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5df1d9b640 1 -- 172.21.15.125:0/2338472973 >> v1:172.21.15.136:6789/0 conn(0x7f5dfc15a540 legacy=0x7f5dfc0cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5df1d9b640 1 -- 172.21.15.125:0/2338472973 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dfc0c8710 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.954+0000 7f5dfa59c640 1 --2- 172.21.15.125:0/2338472973 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dfc153d10 0x7f5dfc0c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:25:34.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.958+0000 7f5df1d9b640 1 --2- 172.21.15.125:0/2338472973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dfc0c64a0 0x7f5dfc15a000 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5de40097c0 tx=0x7f5de401c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4eb2f44452bb00a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:34.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.958+0000 7f5df9d9b640 1 -- 172.21.15.125:0/2338472973 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de40ab650 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.958+0000 7f5df9d9b640 1 -- 172.21.15.125:0/2338472973 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5de40ab7b0 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.958+0000 7f5df9d9b640 1 -- 172.21.15.125:0/2338472973 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de40abb40 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.958+0000 7f5e00a54640 1 -- 172.21.15.125:0/2338472973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dfc0c64a0 msgr2=0x7f5dfc15a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.958+0000 7f5e00a54640 1 --2- 172.21.15.125:0/2338472973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dfc0c64a0 0x7f5dfc15a000 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f5de40097c0 tx=0x7f5de401c450 comp rx=0 tx=0).stop 2023-10-30T13:25:34.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.958+0000 7f5e00a54640 1 -- 172.21.15.125:0/2338472973 shutdown_connections 2023-10-30T13:25:34.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.958+0000 7f5e00a54640 1 --2- 172.21.15.125:0/2338472973 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dfc0c64a0 0x7f5dfc15a000 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.958+0000 7f5e00a54640 1 --2- 172.21.15.125:0/2338472973 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dfc153d10 0x7f5dfc0c5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.958+0000 7f5e00a54640 1 -- 172.21.15.125:0/2338472973 >> 172.21.15.125:0/2338472973 conn(0x7f5dfc06c6b0 msgr2=0x7f5dfc069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:34.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.958+0000 7f5e00a54640 1 -- 172.21.15.125:0/2338472973 shutdown_connections 2023-10-30T13:25:34.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.958+0000 7f5e00a54640 1 -- 172.21.15.125:0/2338472973 wait complete. 2023-10-30T13:25:34.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5e00a54640 1 Processor -- start 2023-10-30T13:25:34.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5e00a54640 1 -- start start 2023-10-30T13:25:34.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5e00a54640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5dfc0c64a0 0x7f5dfc1f53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5e00a54640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dfc153d10 0x7f5dfc14f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5e00a54640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dfc15a540 0x7f5dfc14fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5e00a54640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc14f370 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5e00a54640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc158ad0 con 0x7f5dfc153d10 2023-10-30T13:25:34.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5e00a54640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dfc1f6010 con 0x7f5dfc15a540 2023-10-30T13:25:34.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5dfa59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5dfc0c64a0 0x7f5dfc1f53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5dfa59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5dfc0c64a0 0x7f5dfc1f53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38540/0 (socket says 172.21.15.125:38540) 2023-10-30T13:25:34.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5dfa59c640 1 -- 172.21.15.125:0/4024303628 learned_addr learned my addr 172.21.15.125:0/4024303628 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:34.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5df1d9b640 1 --2- 172.21.15.125:0/4024303628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dfc153d10 0x7f5dfc14f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5dfad9d640 1 --2- 172.21.15.125:0/4024303628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dfc15a540 0x7f5dfc14fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5dfa59c640 1 -- 172.21.15.125:0/4024303628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dfc15a540 msgr2=0x7f5dfc14fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5dfa59c640 1 --2- 172.21.15.125:0/4024303628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dfc15a540 0x7f5dfc14fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.974 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5dfa59c640 1 -- 172.21.15.125:0/4024303628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dfc153d10 msgr2=0x7f5dfc14f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:34.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5dfa59c640 1 --2- 172.21.15.125:0/4024303628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dfc153d10 0x7f5dfc14f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:34.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5dfa59c640 1 -- 172.21.15.125:0/4024303628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dec009b30 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5dfa59c640 1 --2- 172.21.15.125:0/4024303628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5dfc0c64a0 0x7f5dfc1f53d0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00efc0 tx=0x7f5dec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:34.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5df37fe640 1 -- 172.21.15.125:0/4024303628 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec00ce60 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5df37fe640 1 -- 172.21.15.125:0/4024303628 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dec00f040 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5df37fe640 1 -- 172.21.15.125:0/4024303628 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dec014680 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5e00a54640 1 -- 172.21.15.125:0/4024303628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5de4009420 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.962+0000 7f5e00a54640 1 -- 172.21.15.125:0/4024303628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dfc14c8c0 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.966+0000 7f5e00a54640 1 -- 172.21.15.125:0/4024303628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dfc04e170 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.966+0000 7f5df37fe640 1 -- 172.21.15.125:0/4024303628 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5dec0147e0 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.966+0000 7f5df37fe640 1 --2- 172.21.15.125:0/4024303628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5ddc0664a0 0x7f5ddc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:34.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.966+0000 7f5df37fe640 1 -- 172.21.15.125:0/4024303628 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f5dec085200 con 0x7f5dfc0c64a0 2023-10-30T13:25:34.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.966+0000 7f5df1d9b640 1 --2- 172.21.15.125:0/4024303628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5ddc0664a0 0x7f5ddc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:34.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.970+0000 7f5df1d9b640 1 --2- 172.21.15.125:0/4024303628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5ddc0664a0 0x7f5ddc068960 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f5de40af000 tx=0x7f5de40a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:34.979 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:34.970+0000 7f5df37fe640 1 -- 172.21.15.125:0/4024303628 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5dec051b10 con 0x7f5dfc0c64a0 2023-10-30T13:25:35.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.130+0000 7f5e00a54640 1 -- 172.21.15.125:0/4024303628 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5dfc0660f0 con 0x7f5dfc0c64a0 2023-10-30T13:25:35.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.130+0000 7f5df37fe640 1 -- 172.21.15.125:0/4024303628 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 62 entries v85) v1 ==== 81+0+3838 (secure 0 0 0) 0x7f5dec055130 con 0x7f5dfc0c64a0 2023-10-30T13:25:35.136 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6835/3480968625 2023-10-31T13:24:15.942040+0000 2023-10-30T13:25:35.136 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6839/4284345684 2023-10-31T13:21:50.903213+0000 2023-10-30T13:25:35.136 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6833/739564283 2023-10-31T12:57:50.441726+0000 2023-10-30T13:25:35.137 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6839/1454326577 2023-10-31T12:40:50.129310+0000 2023-10-30T13:25:35.137 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/2408701650 2023-10-31T12:52:30.335181+0000 2023-10-30T13:25:35.137 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6833/1163622496 2023-10-31T12:55:15.400773+0000 2023-10-30T13:25:35.138 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6834/1624923396 2023-10-31T12:42:15.151597+0000 2023-10-30T13:25:35.138 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6834/2408701650 2023-10-31T12:52:30.335181+0000 2023-10-30T13:25:35.138 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/3906359913 2023-10-31T12:59:15.462699+0000 2023-10-30T13:25:35.138 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/1721514617 2023-10-31T12:51:15.314344+0000 2023-10-30T13:25:35.139 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6833/2595438444 2023-10-31T12:46:30.228008+0000 2023-10-30T13:25:35.139 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6833/1721514617 2023-10-31T12:51:15.314344+0000 2023-10-30T13:25:35.139 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6838/1510290691 2023-10-31T12:47:25.242471+0000 2023-10-30T13:25:35.139 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/2595438444 2023-10-31T12:46:30.228008+0000 2023-10-30T13:25:35.140 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6833/2017382874 2023-10-31T12:36:30.049259+0000 2023-10-30T13:25:35.140 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6834/2540063352 2023-10-31T13:12:50.713238+0000 2023-10-30T13:25:35.140 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/3950044434 2023-10-31T13:17:55.811938+0000 2023-10-30T13:25:35.141 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6834/2291877604 2023-10-31T12:32:34.980679+0000 2023-10-30T13:25:35.141 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6839/3922559697 2023-10-31T12:44:00.178262+0000 2023-10-30T13:25:35.141 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/1163622496 2023-10-31T12:55:15.400773+0000 2023-10-30T13:25:35.141 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6834/3480968625 2023-10-31T13:24:15.942040+0000 2023-10-30T13:25:35.142 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6835/3346501886 2023-10-31T12:30:24.948059+0000 2023-10-30T13:25:35.142 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6834/3906359913 2023-10-31T12:59:15.462699+0000 2023-10-30T13:25:35.143 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6834/2196617378 2023-10-31T13:05:55.585769+0000 2023-10-30T13:25:35.143 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6833/3711687505 2023-10-31T13:06:55.602060+0000 2023-10-30T13:25:35.143 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6839/2250748765 2023-10-31T12:35:15.030179+0000 2023-10-30T13:25:35.143 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/2291877604 2023-10-31T12:32:34.980679+0000 2023-10-30T13:25:35.144 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6835/3526499585 2023-10-31T13:11:30.689863+0000 2023-10-30T13:25:35.144 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6838/1454326577 2023-10-31T12:40:50.129310+0000 2023-10-30T13:25:35.144 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6835/695089547 2023-10-31T13:15:05.766729+0000 2023-10-30T13:25:35.145 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6839/223464319 2023-10-31T13:08:55.644039+0000 2023-10-30T13:25:35.145 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/1624923396 2023-10-31T12:42:15.151597+0000 2023-10-30T13:25:35.145 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6834/1348687922 2023-10-31T12:56:35.422955+0000 2023-10-30T13:25:35.145 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/2017382874 2023-10-31T12:36:30.049259+0000 2023-10-30T13:25:35.146 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6838/3922559697 2023-10-31T12:44:00.178262+0000 2023-10-30T13:25:35.146 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6834/3156415951 2023-10-31T12:48:25.270273+0000 2023-10-30T13:25:35.146 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6833/1944683138 2023-10-31T13:00:40.486003+0000 2023-10-30T13:25:35.146 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/1944683138 2023-10-31T13:00:40.486003+0000 2023-10-30T13:25:35.147 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6834/3762807620 2023-10-31T13:02:35.533843+0000 2023-10-30T13:25:35.147 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/1348687922 2023-10-31T12:56:35.422955+0000 2023-10-30T13:25:35.147 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/3762807620 2023-10-31T13:02:35.533843+0000 2023-10-30T13:25:35.147 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6833/1301336591 2023-10-31T13:04:45.567511+0000 2023-10-30T13:25:35.148 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/1301336591 2023-10-31T13:04:45.567511+0000 2023-10-30T13:25:35.148 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6835/2194892271 2023-10-31T13:08:00.628452+0000 2023-10-30T13:25:35.148 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/2196617378 2023-10-31T13:05:55.585769+0000 2023-10-30T13:25:35.148 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/3711687505 2023-10-31T13:06:55.602060+0000 2023-10-30T13:25:35.149 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6838/3776186190 2023-10-31T13:16:00.781626+0000 2023-10-30T13:25:35.149 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6834/2194892271 2023-10-31T13:08:00.628452+0000 2023-10-30T13:25:35.149 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6834/3526499585 2023-10-31T13:11:30.689863+0000 2023-10-30T13:25:35.149 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6838/4284345684 2023-10-31T13:21:50.903213+0000 2023-10-30T13:25:35.150 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6838/2250748765 2023-10-31T12:35:15.030179+0000 2023-10-30T13:25:35.150 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6838/223464319 2023-10-31T13:08:55.644039+0000 2023-10-30T13:25:35.150 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/3156415951 2023-10-31T12:48:25.270273+0000 2023-10-30T13:25:35.151 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6834/695089547 2023-10-31T13:15:05.766729+0000 2023-10-30T13:25:35.151 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6837/2540063352 2023-10-31T13:12:50.713238+0000 2023-10-30T13:25:35.151 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6834/3346501886 2023-10-31T12:30:24.948059+0000 2023-10-30T13:25:35.151 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6839/3776186190 2023-10-31T13:16:00.781626+0000 2023-10-30T13:25:35.152 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6833/3950044434 2023-10-31T13:17:55.811938+0000 2023-10-30T13:25:35.152 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6839/1510290691 2023-10-31T12:47:25.242471+0000 2023-10-30T13:25:35.152 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6834/3864637420 2023-10-31T13:20:45.886166+0000 2023-10-30T13:25:35.152 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.125:6836/739564283 2023-10-31T12:57:50.441726+0000 2023-10-30T13:25:35.153 INFO:teuthology.orchestra.run.smithi125.stdout:172.21.15.136:6835/3864637420 2023-10-31T13:20:45.886166+0000 2023-10-30T13:25:35.154 INFO:teuthology.orchestra.run.smithi125.stderr:listed 62 entries 2023-10-30T13:25:35.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.134+0000 7f5e00a54640 1 -- 172.21.15.125:0/4024303628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5ddc0664a0 msgr2=0x7f5ddc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.134+0000 7f5e00a54640 1 --2- 172.21.15.125:0/4024303628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5ddc0664a0 0x7f5ddc068960 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f5de40af000 tx=0x7f5de40a7000 comp rx=0 tx=0).stop 2023-10-30T13:25:35.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.134+0000 7f5e00a54640 1 -- 172.21.15.125:0/4024303628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5dfc0c64a0 msgr2=0x7f5dfc1f53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.134+0000 7f5e00a54640 1 --2- 172.21.15.125:0/4024303628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5dfc0c64a0 0x7f5dfc1f53d0 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00efc0 tx=0x7f5dec00c490 comp rx=0 tx=0).stop 2023-10-30T13:25:35.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.134+0000 7f5e00a54640 1 -- 172.21.15.125:0/4024303628 shutdown_connections 2023-10-30T13:25:35.155 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.134+0000 7f5e00a54640 1 --2- 172.21.15.125:0/4024303628 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5ddc0664a0 0x7f5ddc068960 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.134+0000 7f5e00a54640 1 --2- 172.21.15.125:0/4024303628 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5dfc15a540 0x7f5dfc14fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.134+0000 7f5e00a54640 1 --2- 172.21.15.125:0/4024303628 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5dfc153d10 0x7f5dfc14f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.156 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.134+0000 7f5e00a54640 1 --2- 172.21.15.125:0/4024303628 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5dfc0c64a0 0x7f5dfc1f53d0 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.134+0000 7f5e00a54640 1 -- 172.21.15.125:0/4024303628 >> 172.21.15.125:0/4024303628 conn(0x7f5dfc06c6b0 msgr2=0x7f5dfc15c470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:35.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.134+0000 7f5e00a54640 1 -- 172.21.15.125:0/4024303628 shutdown_connections 2023-10-30T13:25:35.157 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.134+0000 7f5e00a54640 1 -- 172.21.15.125:0/4024303628 wait complete. 2023-10-30T13:25:35.158 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-30T13:25:35.159 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:25:35.159 DEBUG:teuthology.orchestra.run.smithi125:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-30T13:25:35.275 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi125.front.sepia.ceph.com 2023-10-30T13:25:35.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:25:35.276 DEBUG:teuthology.orchestra.run.smithi125:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-30T13:25:35.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:25:35.286 DEBUG:teuthology.orchestra.run.smithi125:> ip netns list 2023-10-30T13:25:35.340 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-30T13:25:35.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:25:35.342 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-30T13:25:35.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-30T13:25:35.409 DEBUG:teuthology.orchestra.run.smithi125:> sudo ip link delete ceph-brx 2023-10-30T13:25:35.622 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-30T13:25:35.632 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-30T13:25:35.633 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:25:35.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0903d94640 1 Processor -- start 2023-10-30T13:25:35.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0903d94640 1 -- start start 2023-10-30T13:25:35.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0903d94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08fc06f7a0 0x7f08fc06fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:35.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0903d94640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f08fc071110 0x7f08fc158110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:35.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0903d94640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08fc066cf0 con 0x7f08fc0c6bd0 2023-10-30T13:25:35.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0903d94640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08fc0700c0 con 0x7f08fc071110 2023-10-30T13:25:35.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0903d94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc0c70d0 con 0x7f08fc06f7a0 2023-10-30T13:25:35.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0901b30640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08fc06f7a0 0x7f08fc06fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:35.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0902331640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f08fc071110 0x7f08fc158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:35.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0901b30640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08fc06f7a0 0x7f08fc06fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:48114/0 (socket says 172.21.15.125:48114) 2023-10-30T13:25:35.776 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0902331640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f08fc071110 0x7f08fc158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:51730/0 (socket says 172.21.15.125:51730) 2023-10-30T13:25:35.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0901b30640 1 -- 172.21.15.125:0/195666581 learned_addr learned my addr 172.21.15.125:0/195666581 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:35.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0901b30640 1 -- 172.21.15.125:0/195666581 >> v2:172.21.15.136:3300/0 conn(0x7f08fc071110 msgr2=0x7f08fc158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:35.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0901b30640 1 --2- 172.21.15.125:0/195666581 >> v2:172.21.15.136:3300/0 conn(0x7f08fc071110 0x7f08fc158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0901b30640 1 -- 172.21.15.125:0/195666581 >> v1:172.21.15.136:6789/0 conn(0x7f08fc0c6bd0 legacy=0x7f08fc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0901b30640 1 -- 172.21.15.125:0/195666581 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08fc0c7390 con 0x7f08fc06f7a0 2023-10-30T13:25:35.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.770+0000 7f0902331640 1 --2- 172.21.15.125:0/195666581 >> v2:172.21.15.136:3300/0 conn(0x7f08fc071110 0x7f08fc158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:25:35.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.774+0000 7f0901b30640 1 --2- 172.21.15.125:0/195666581 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08fc06f7a0 0x7f08fc06fb80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f08f8009870 tx=0x7f08f801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d2e54060465b818 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:35.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.774+0000 7f0900b2e640 1 -- 172.21.15.125:0/195666581 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f80a8a30 con 0x7f08fc06f7a0 2023-10-30T13:25:35.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.774+0000 7f0900b2e640 1 -- 172.21.15.125:0/195666581 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f80a8b90 con 0x7f08fc06f7a0 2023-10-30T13:25:35.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.774+0000 7f0900b2e640 1 -- 172.21.15.125:0/195666581 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f80b0440 con 0x7f08fc06f7a0 2023-10-30T13:25:35.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.774+0000 7f0903d94640 1 -- 172.21.15.125:0/195666581 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08fc06f7a0 msgr2=0x7f08fc06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.774+0000 7f0903d94640 1 --2- 172.21.15.125:0/195666581 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08fc06f7a0 0x7f08fc06fb80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f08f8009870 tx=0x7f08f801ed30 comp rx=0 tx=0).stop 2023-10-30T13:25:35.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.774+0000 7f0903d94640 1 -- 172.21.15.125:0/195666581 shutdown_connections 2023-10-30T13:25:35.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.774+0000 7f0903d94640 1 --2- 172.21.15.125:0/195666581 >> v2:172.21.15.136:3300/0 conn(0x7f08fc071110 0x7f08fc158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.774+0000 7f0903d94640 1 --2- 172.21.15.125:0/195666581 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08fc06f7a0 0x7f08fc06fb80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.774+0000 7f0903d94640 1 -- 172.21.15.125:0/195666581 >> 172.21.15.125:0/195666581 conn(0x7f08fc0c7910 msgr2=0x7f08fc0c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:35.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.774+0000 7f0903d94640 1 -- 172.21.15.125:0/195666581 shutdown_connections 2023-10-30T13:25:35.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.774+0000 7f0903d94640 1 -- 172.21.15.125:0/195666581 wait complete. 2023-10-30T13:25:35.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0903d94640 1 Processor -- start 2023-10-30T13:25:35.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0903d94640 1 -- start start 2023-10-30T13:25:35.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0903d94640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08fc06f7a0 0x7f08fc0c4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:35.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0903d94640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08fc071110 0x7f08fc1596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:35.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0903d94640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08fc0c6bd0 0x7f08fc1f03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:35.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0903d94640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc158c30 con 0x7f08fc06f7a0 2023-10-30T13:25:35.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0903d94640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc1eff30 con 0x7f08fc0c6bd0 2023-10-30T13:25:35.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0903d94640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08fc1589b0 con 0x7f08fc071110 2023-10-30T13:25:35.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0901b30640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08fc06f7a0 0x7f08fc0c4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:35.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0901b30640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08fc06f7a0 0x7f08fc0c4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38554/0 (socket says 172.21.15.125:38554) 2023-10-30T13:25:35.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0901b30640 1 -- 172.21.15.125:0/2627876267 learned_addr learned my addr 172.21.15.125:0/2627876267 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:35.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0902331640 1 --2- 172.21.15.125:0/2627876267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08fc0c6bd0 0x7f08fc1f03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:35.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f090132f640 1 --2- 172.21.15.125:0/2627876267 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08fc071110 0x7f08fc1596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:35.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0901b30640 1 -- 172.21.15.125:0/2627876267 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08fc071110 msgr2=0x7f08fc1596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0901b30640 1 --2- 172.21.15.125:0/2627876267 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08fc071110 0x7f08fc1596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.788 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0901b30640 1 -- 172.21.15.125:0/2627876267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08fc0c6bd0 msgr2=0x7f08fc1f03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0901b30640 1 --2- 172.21.15.125:0/2627876267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08fc0c6bd0 0x7f08fc1f03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0901b30640 1 -- 172.21.15.125:0/2627876267 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f4009a70 con 0x7f08fc06f7a0 2023-10-30T13:25:35.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0901b30640 1 --2- 172.21.15.125:0/2627876267 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08fc06f7a0 0x7f08fc0c4150 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f08f80b3040 tx=0x7f08f801bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:35.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f08f27fc640 1 -- 172.21.15.125:0/2627876267 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f80b0dd0 con 0x7f08fc06f7a0 2023-10-30T13:25:35.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f08f27fc640 1 -- 172.21.15.125:0/2627876267 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08f80bb460 con 0x7f08fc06f7a0 2023-10-30T13:25:35.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0903d94640 1 -- 172.21.15.125:0/2627876267 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f80094d0 con 0x7f08fc06f7a0 2023-10-30T13:25:35.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f08f27fc640 1 -- 172.21.15.125:0/2627876267 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08f80bb810 con 0x7f08fc06f7a0 2023-10-30T13:25:35.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.778+0000 7f0903d94640 1 -- 172.21.15.125:0/2627876267 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08fc14a3f0 con 0x7f08fc06f7a0 2023-10-30T13:25:35.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.782+0000 7f0903d94640 1 -- 172.21.15.125:0/2627876267 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08fc04e170 con 0x7f08fc06f7a0 2023-10-30T13:25:35.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.782+0000 7f08f27fc640 1 -- 172.21.15.125:0/2627876267 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f08f80b83f0 con 0x7f08fc06f7a0 2023-10-30T13:25:35.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.782+0000 7f08f27fc640 1 --2- 172.21.15.125:0/2627876267 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f08e00664d0 0x7f08e0068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:35.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.782+0000 7f08f27fc640 1 -- 172.21.15.125:0/2627876267 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f08f8122200 con 0x7f08fc06f7a0 2023-10-30T13:25:35.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.782+0000 7f090132f640 1 --2- 172.21.15.125:0/2627876267 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f08e00664d0 0x7f08e0068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:35.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.782+0000 7f090132f640 1 --2- 172.21.15.125:0/2627876267 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f08e00664d0 0x7f08e0068990 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f08fc156aa0 tx=0x7f08ec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:35.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.786+0000 7f08f27fc640 1 -- 172.21.15.125:0/2627876267 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f08f80eeb10 con 0x7f08fc06f7a0 2023-10-30T13:25:35.938 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.934+0000 7f0903d94640 1 -- 172.21.15.125:0/2627876267 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f08fc070350 con 0x7f08e00664d0 2023-10-30T13:25:35.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.958+0000 7f08f27fc640 1 -- 172.21.15.125:0/2627876267 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293405 (secure 0 0 0) 0x7f08ec001c80 con 0x7f08e00664d0 2023-10-30T13:25:35.965 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:25:35.967 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T13:25:35.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.962+0000 7f0903d94640 1 -- 172.21.15.125:0/2627876267 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f08e00664d0 msgr2=0x7f08e0068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.962+0000 7f0903d94640 1 --2- 172.21.15.125:0/2627876267 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f08e00664d0 0x7f08e0068990 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f08fc156aa0 tx=0x7f08ec009040 comp rx=0 tx=0).stop 2023-10-30T13:25:35.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.962+0000 7f0903d94640 1 -- 172.21.15.125:0/2627876267 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08fc06f7a0 msgr2=0x7f08fc0c4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:35.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.962+0000 7f0903d94640 1 --2- 172.21.15.125:0/2627876267 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08fc06f7a0 0x7f08fc0c4150 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7f08f80b3040 tx=0x7f08f801bad0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.962+0000 7f0903d94640 1 -- 172.21.15.125:0/2627876267 shutdown_connections 2023-10-30T13:25:35.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.962+0000 7f0903d94640 1 --2- 172.21.15.125:0/2627876267 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f08e00664d0 0x7f08e0068990 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.962+0000 7f0903d94640 1 --2- 172.21.15.125:0/2627876267 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f08fc0c6bd0 0x7f08fc1f03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.962+0000 7f0903d94640 1 --2- 172.21.15.125:0/2627876267 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f08fc071110 0x7f08fc1596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.962+0000 7f0903d94640 1 --2- 172.21.15.125:0/2627876267 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f08fc06f7a0 0x7f08fc0c4150 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:35.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.962+0000 7f0903d94640 1 -- 172.21.15.125:0/2627876267 >> 172.21.15.125:0/2627876267 conn(0x7f08fc0c7910 msgr2=0x7f08fc0c29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:35.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.962+0000 7f0903d94640 1 -- 172.21.15.125:0/2627876267 shutdown_connections 2023-10-30T13:25:35.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:35.962+0000 7f0903d94640 1 -- 172.21.15.125:0/2627876267 wait complete. 2023-10-30T13:25:35.986 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":1789,"stamp":"2023-10-30T13:25:34.269499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2068930286,"num_objects":40960,"num_object_clones":18317,"num_object_copies":81920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40960,"num_whiteouts":14249,"num_read":30914,"num_read_kb":10816677,"num_write":128189,"num_write_kb":3672589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134387,"ondisk_log_size":134387,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5379072,"kb_used_data":4140352,"kb_used_omap":12,"kb_used_meta":1238643,"kb_avail":749595648,"statfs":{"total":773094113280,"available":767585943552,"internally_reserved":0,"allocated":4239720448,"data_stored":4144117292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1268371001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1464234,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":38,"num_read_kb":30453,"num_write":11,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004539"},"pg_stats":[{"pgid":"3.3f","version":"77'1475","reported_seq":3523,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.196345+0000","last_change":"2023-10-30T12:37:07.182969+0000","last_active":"2023-10-30T13:25:16.196345+0000","last_peered":"2023-10-30T13:25:16.196345+0000","last_clean":"2023-10-30T13:25:16.196345+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T13:25:16.196345+0000","last_undegraded":"2023-10-30T13:25:16.196345+0000","last_fullsized":"2023-10-30T13:25:16.196345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1475,"log_dups_size":0,"ondisk_log_size":1475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:33.616938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086120880000000004,"stat_sum":{"num_bytes":27261067,"num_objects":552,"num_object_clones":249,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":197,"num_read":145,"num_read_kb":29501,"num_write":1298,"num_write_kb":29008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"77'1476","reported_seq":3487,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:01.374954+0000","last_change":"2023-10-30T12:37:07.190309+0000","last_active":"2023-10-30T13:25:01.374954+0000","last_peered":"2023-10-30T13:25:01.374954+0000","last_clean":"2023-10-30T13:25:01.374954+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T13:25:01.374954+0000","last_undegraded":"2023-10-30T13:25:01.374954+0000","last_fullsized":"2023-10-30T13:25:01.374954+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:26:31.884771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015306986,"stat_sum":{"num_bytes":13736175,"num_objects":574,"num_object_clones":269,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":223,"num_read":130,"num_read_kb":17219,"num_write":1260,"num_write_kb":15409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"77'1563","reported_seq":3574,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288448+0000","last_change":"2023-10-30T12:37:07.187803+0000","last_active":"2023-10-30T13:24:16.288448+0000","last_peered":"2023-10-30T13:24:16.288448+0000","last_clean":"2023-10-30T13:24:16.288448+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T13:24:16.288448+0000","last_undegraded":"2023-10-30T13:24:16.288448+0000","last_fullsized":"2023-10-30T13:24:16.288448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1563,"log_dups_size":0,"ondisk_log_size":1563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:32.502949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013634598,"stat_sum":{"num_bytes":28390436,"num_objects":628,"num_object_clones":291,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":232,"num_read":152,"num_read_kb":47076,"num_write":1356,"num_write_kb":37143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"77'1577","reported_seq":3585,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.277341+0000","last_change":"2023-10-30T12:37:07.177860+0000","last_active":"2023-10-30T13:25:32.277341+0000","last_peered":"2023-10-30T13:25:32.277341+0000","last_clean":"2023-10-30T13:25:32.277341+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T13:25:32.277341+0000","last_undegraded":"2023-10-30T13:25:32.277341+0000","last_fullsized":"2023-10-30T13:25:32.277341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1577,"log_dups_size":0,"ondisk_log_size":1577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:27.076595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056762749999999997,"stat_sum":{"num_bytes":22921274,"num_objects":629,"num_object_clones":290,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":224,"num_read":145,"num_read_kb":39229,"num_write":1388,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"77'1591","reported_seq":3596,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.167292+0000","last_change":"2023-10-30T12:37:07.181803+0000","last_active":"2023-10-30T13:24:56.167292+0000","last_peered":"2023-10-30T13:24:56.167292+0000","last_clean":"2023-10-30T13:24:56.167292+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T13:24:56.167292+0000","last_undegraded":"2023-10-30T13:24:56.167292+0000","last_fullsized":"2023-10-30T13:24:56.167292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1591,"log_dups_size":0,"ondisk_log_size":1591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:23:58.362682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081298299999999993,"stat_sum":{"num_bytes":26798060,"num_objects":623,"num_object_clones":286,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":214,"num_read":144,"num_read_kb":26713,"num_write":1434,"num_write_kb":32534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"85'1466","reported_seq":3496,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.286314+0000","last_change":"2023-10-30T12:37:07.176046+0000","last_active":"2023-10-30T13:24:16.286314+0000","last_peered":"2023-10-30T13:24:16.286314+0000","last_clean":"2023-10-30T13:24:16.286314+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T13:24:16.286314+0000","last_undegraded":"2023-10-30T13:24:16.286314+0000","last_fullsized":"2023-10-30T13:24:16.286314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:30:51.393008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048510920000000004,"stat_sum":{"num_bytes":13539463,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":205,"num_read":144,"num_read_kb":15818,"num_write":1313,"num_write_kb":14957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"84'1562","reported_seq":3551,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-30T13:23:51.391894+0000","last_change":"2023-10-30T12:37:07.173349+0000","last_active":"2023-10-30T13:23:51.391894+0000","last_peered":"2023-10-30T13:23:51.391894+0000","last_clean":"2023-10-30T13:23:51.391894+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T13:23:51.391894+0000","last_undegraded":"2023-10-30T13:23:51.391894+0000","last_fullsized":"2023-10-30T13:23:51.391894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1562,"log_dups_size":0,"ondisk_log_size":1562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:38:16.606065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082671399999999999,"stat_sum":{"num_bytes":7202428,"num_objects":606,"num_object_clones":284,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":219,"num_read":126,"num_read_kb":1677,"num_write":1361,"num_write_kb":10203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"82'1518","reported_seq":3541,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288347+0000","last_change":"2023-10-30T12:37:07.188855+0000","last_active":"2023-10-30T13:24:16.288347+0000","last_peered":"2023-10-30T13:24:16.288347+0000","last_clean":"2023-10-30T13:24:16.288347+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T13:24:16.288347+0000","last_undegraded":"2023-10-30T13:24:16.288347+0000","last_fullsized":"2023-10-30T13:24:16.288347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:54.679005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014437559000000001,"stat_sum":{"num_bytes":23150826,"num_objects":587,"num_object_clones":269,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":199,"num_read":147,"num_read_kb":38801,"num_write":1362,"num_write_kb":24887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"77'1549","reported_seq":3579,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.247414+0000","last_change":"2023-10-30T12:37:07.102887+0000","last_active":"2023-10-30T13:25:32.247414+0000","last_peered":"2023-10-30T13:25:32.247414+0000","last_clean":"2023-10-30T13:25:32.247414+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T13:25:32.247414+0000","last_undegraded":"2023-10-30T13:25:32.247414+0000","last_fullsized":"2023-10-30T13:25:32.247414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1549,"log_dups_size":0,"ondisk_log_size":1549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:09.601324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014185980000000001,"stat_sum":{"num_bytes":36575095,"num_objects":618,"num_object_clones":282,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":216,"num_read":165,"num_read_kb":36418,"num_write":1354,"num_write_kb":45869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"77'1459","reported_seq":3491,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.224692+0000","last_change":"2023-10-30T12:37:07.183547+0000","last_active":"2023-10-30T13:25:32.224692+0000","last_peered":"2023-10-30T13:25:32.224692+0000","last_clean":"2023-10-30T13:25:32.224692+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T13:25:32.224692+0000","last_undegraded":"2023-10-30T13:25:32.224692+0000","last_fullsized":"2023-10-30T13:25:32.224692+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:43.076339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091404980000000004,"stat_sum":{"num_bytes":11176343,"num_objects":586,"num_object_clones":270,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":213,"num_read":154,"num_read_kb":10770,"num_write":1260,"num_write_kb":13564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"82'1511","reported_seq":3517,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.187191+0000","last_change":"2023-10-30T12:37:07.181463+0000","last_active":"2023-10-30T13:25:16.187191+0000","last_peered":"2023-10-30T13:25:16.187191+0000","last_clean":"2023-10-30T13:25:16.187191+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T13:25:16.187191+0000","last_undegraded":"2023-10-30T13:25:16.187191+0000","last_fullsized":"2023-10-30T13:25:16.187191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1511,"log_dups_size":0,"ondisk_log_size":1511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:06.796994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075712790000000002,"stat_sum":{"num_bytes":23568118,"num_objects":625,"num_object_clones":283,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":230,"num_read":141,"num_read_kb":34199,"num_write":1299,"num_write_kb":24762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"84'1725","reported_seq":3645,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.138485+0000","last_change":"2023-10-30T12:37:07.181114+0000","last_active":"2023-10-30T13:25:16.138485+0000","last_peered":"2023-10-30T13:25:16.138485+0000","last_clean":"2023-10-30T13:25:16.138485+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T13:25:16.138485+0000","last_undegraded":"2023-10-30T13:25:16.138485+0000","last_fullsized":"2023-10-30T13:25:16.138485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1725,"log_dups_size":0,"ondisk_log_size":1725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:57:46.353792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007366493,"stat_sum":{"num_bytes":24142540,"num_objects":721,"num_object_clones":385,"num_object_copies":1442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":231,"num_read":157,"num_read_kb":24833,"num_write":1452,"num_write_kb":33857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"85'1503","reported_seq":3540,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.201400+0000","last_change":"2023-10-30T12:37:07.114858+0000","last_active":"2023-10-30T13:25:16.201400+0000","last_peered":"2023-10-30T13:25:16.201400+0000","last_clean":"2023-10-30T13:25:16.201400+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T13:25:16.201400+0000","last_undegraded":"2023-10-30T13:25:16.201400+0000","last_fullsized":"2023-10-30T13:25:16.201400+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1503,"log_dups_size":0,"ondisk_log_size":1503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:18:13.789863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006759035,"stat_sum":{"num_bytes":14254037,"num_objects":596,"num_object_clones":276,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":205,"num_read":165,"num_read_kb":21722,"num_write":1326,"num_write_kb":24306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"77'1568","reported_seq":3567,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.278066+0000","last_change":"2023-10-30T12:37:07.112609+0000","last_active":"2023-10-30T13:24:57.278066+0000","last_peered":"2023-10-30T13:24:57.278066+0000","last_clean":"2023-10-30T13:24:57.278066+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T13:24:57.278066+0000","last_undegraded":"2023-10-30T13:24:57.278066+0000","last_fullsized":"2023-10-30T13:24:57.278066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1568,"log_dups_size":0,"ondisk_log_size":1568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:43.640916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057386779999999997,"stat_sum":{"num_bytes":10370740,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":213,"num_read":131,"num_read_kb":5845,"num_write":1381,"num_write_kb":12595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"77'1624","reported_seq":3604,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:46.371670+0000","last_change":"2023-10-30T12:37:07.187257+0000","last_active":"2023-10-30T13:24:46.371670+0000","last_peered":"2023-10-30T13:24:46.371670+0000","last_clean":"2023-10-30T13:24:46.371670+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T13:24:46.371670+0000","last_undegraded":"2023-10-30T13:24:46.371670+0000","last_fullsized":"2023-10-30T13:24:46.371670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:22.374872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013549705,"stat_sum":{"num_bytes":19248519,"num_objects":637,"num_object_clones":295,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":221,"num_read":128,"num_read_kb":21688,"num_write":1447,"num_write_kb":21442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"82'1574","reported_seq":3576,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.208932+0000","last_change":"2023-10-30T12:37:07.178284+0000","last_active":"2023-10-30T13:25:32.208932+0000","last_peered":"2023-10-30T13:25:32.208932+0000","last_clean":"2023-10-30T13:25:32.208932+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T13:25:32.208932+0000","last_undegraded":"2023-10-30T13:25:32.208932+0000","last_fullsized":"2023-10-30T13:25:32.208932+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1574,"log_dups_size":0,"ondisk_log_size":1574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:39:06.662733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075608309999999996,"stat_sum":{"num_bytes":14269019,"num_objects":619,"num_object_clones":290,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":233,"num_read":143,"num_read_kb":25965,"num_write":1377,"num_write_kb":20442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"85'1605","reported_seq":3590,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.146156+0000","last_change":"2023-10-30T12:37:07.182349+0000","last_active":"2023-10-30T13:25:16.146156+0000","last_peered":"2023-10-30T13:25:16.146156+0000","last_clean":"2023-10-30T13:25:16.146156+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T13:25:16.146156+0000","last_undegraded":"2023-10-30T13:25:16.146156+0000","last_fullsized":"2023-10-30T13:25:16.146156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1605,"log_dups_size":0,"ondisk_log_size":1605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:47:59.298834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0099357109999999998,"stat_sum":{"num_bytes":14793776,"num_objects":642,"num_object_clones":304,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":242,"num_read":140,"num_read_kb":30098,"num_write":1386,"num_write_kb":20714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"82'1444","reported_seq":3467,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.136464+0000","last_change":"2023-10-30T12:37:07.182572+0000","last_active":"2023-10-30T13:25:16.136464+0000","last_peered":"2023-10-30T13:25:16.136464+0000","last_clean":"2023-10-30T13:25:16.136464+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T13:25:16.136464+0000","last_undegraded":"2023-10-30T13:25:16.136464+0000","last_fullsized":"2023-10-30T13:25:16.136464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1444,"log_dups_size":0,"ondisk_log_size":1444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:41.547635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065893760000000001,"stat_sum":{"num_bytes":15508852,"num_objects":562,"num_object_clones":263,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":216,"num_read":137,"num_read_kb":10064,"num_write":1244,"num_write_kb":17198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"82'1576","reported_seq":3678,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:26.207237+0000","last_change":"2023-10-30T12:37:07.182668+0000","last_active":"2023-10-30T13:25:26.207237+0000","last_peered":"2023-10-30T13:25:26.207237+0000","last_clean":"2023-10-30T13:25:26.207237+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T13:25:26.207237+0000","last_undegraded":"2023-10-30T13:25:26.207237+0000","last_fullsized":"2023-10-30T13:25:26.207237+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1576,"log_dups_size":0,"ondisk_log_size":1576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:03.035306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010472766,"stat_sum":{"num_bytes":15695810,"num_objects":614,"num_object_clones":281,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":221,"num_read":233,"num_read_kb":17180,"num_write":1382,"num_write_kb":23481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"77'1569","reported_seq":3573,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.219508+0000","last_change":"2023-10-30T12:37:07.181518+0000","last_active":"2023-10-30T13:25:16.219508+0000","last_peered":"2023-10-30T13:25:16.219508+0000","last_clean":"2023-10-30T13:25:16.219508+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T13:25:16.219508+0000","last_undegraded":"2023-10-30T13:25:16.219508+0000","last_fullsized":"2023-10-30T13:25:16.219508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:12:17.666786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075455599999999998,"stat_sum":{"num_bytes":5586810,"num_objects":614,"num_object_clones":286,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":230,"num_read":140,"num_read_kb":5846,"num_write":1359,"num_write_kb":15885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"85'1523","reported_seq":3641,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.245123+0000","last_change":"2023-10-30T12:37:07.113494+0000","last_active":"2023-10-30T13:25:16.245123+0000","last_peered":"2023-10-30T13:25:16.245123+0000","last_clean":"2023-10-30T13:25:16.245123+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T13:25:16.245123+0000","last_undegraded":"2023-10-30T13:25:16.245123+0000","last_fullsized":"2023-10-30T13:25:16.245123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1523,"log_dups_size":0,"ondisk_log_size":1523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:16.038384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076434509999999999,"stat_sum":{"num_bytes":23023646,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":222,"num_read":249,"num_read_kb":26208,"num_write":1319,"num_write_kb":28338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"84'1671","reported_seq":3654,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.169370+0000","last_change":"2023-10-30T12:37:07.102791+0000","last_active":"2023-10-30T13:24:56.169370+0000","last_peered":"2023-10-30T13:24:56.169370+0000","last_clean":"2023-10-30T13:24:56.169370+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T13:24:56.169370+0000","last_undegraded":"2023-10-30T13:24:56.169370+0000","last_fullsized":"2023-10-30T13:24:56.169370+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1671,"log_dups_size":0,"ondisk_log_size":1671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:38:00.347865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016105379999999999,"stat_sum":{"num_bytes":11166034,"num_objects":653,"num_object_clones":306,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":233,"num_read":139,"num_read_kb":9627,"num_write":1472,"num_write_kb":18195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"77'1520","reported_seq":3539,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288413+0000","last_change":"2023-10-30T12:37:07.188206+0000","last_active":"2023-10-30T13:24:16.288413+0000","last_peered":"2023-10-30T13:24:16.288413+0000","last_clean":"2023-10-30T13:24:16.288413+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T13:24:16.288413+0000","last_undegraded":"2023-10-30T13:24:16.288413+0000","last_fullsized":"2023-10-30T13:24:16.288413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:21.306614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014064241,"stat_sum":{"num_bytes":10716893,"num_objects":582,"num_object_clones":264,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":203,"num_read":135,"num_read_kb":13340,"num_write":1353,"num_write_kb":16108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"85'1551","reported_seq":3554,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:23.080116+0000","last_change":"2023-10-30T12:37:07.179785+0000","last_active":"2023-10-30T13:25:23.080116+0000","last_peered":"2023-10-30T13:25:23.080116+0000","last_clean":"2023-10-30T13:25:23.080116+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T13:25:23.080116+0000","last_undegraded":"2023-10-30T13:25:23.080116+0000","last_fullsized":"2023-10-30T13:25:23.080116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:49:28.717141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088476170000000003,"stat_sum":{"num_bytes":10233195,"num_objects":597,"num_object_clones":275,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":203,"num_read":127,"num_read_kb":1819,"num_write":1383,"num_write_kb":16400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"82'1465","reported_seq":3479,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.190427+0000","last_change":"2023-10-30T12:37:07.188606+0000","last_active":"2023-10-30T13:25:16.190427+0000","last_peered":"2023-10-30T13:25:16.190427+0000","last_clean":"2023-10-30T13:25:16.190427+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T13:25:16.190427+0000","last_undegraded":"2023-10-30T13:25:16.190427+0000","last_fullsized":"2023-10-30T13:25:16.190427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:30.044315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014279337,"stat_sum":{"num_bytes":22085088,"num_objects":582,"num_object_clones":270,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":216,"num_read":135,"num_read_kb":28666,"num_write":1258,"num_write_kb":25992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"77'1609","reported_seq":3602,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.171560+0000","last_change":"2023-10-30T12:37:07.177071+0000","last_active":"2023-10-30T13:25:16.171560+0000","last_peered":"2023-10-30T13:25:16.171560+0000","last_clean":"2023-10-30T13:25:16.171560+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T13:25:16.171560+0000","last_undegraded":"2023-10-30T13:25:16.171560+0000","last_fullsized":"2023-10-30T13:25:16.171560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1609,"log_dups_size":0,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:19.563113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004006122,"stat_sum":{"num_bytes":15408826,"num_objects":598,"num_object_clones":277,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":213,"num_read":119,"num_read_kb":9406,"num_write":1408,"num_write_kb":18485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"82'1549","reported_seq":3552,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.237486+0000","last_change":"2023-10-30T12:37:07.115775+0000","last_active":"2023-10-30T13:25:32.237486+0000","last_peered":"2023-10-30T13:25:32.237486+0000","last_clean":"2023-10-30T13:25:32.237486+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T13:25:32.237486+0000","last_undegraded":"2023-10-30T13:25:32.237486+0000","last_fullsized":"2023-10-30T13:25:32.237486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1549,"log_dups_size":0,"ondisk_log_size":1549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:59.732034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059072409999999997,"stat_sum":{"num_bytes":25755688,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":240,"num_read":143,"num_read_kb":23034,"num_write":1332,"num_write_kb":28963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"85'1485","reported_seq":3466,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:07.280157+0000","last_change":"2023-10-30T12:37:07.113316+0000","last_active":"2023-10-30T13:25:07.280157+0000","last_peered":"2023-10-30T13:25:07.280157+0000","last_clean":"2023-10-30T13:25:07.280157+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T13:25:07.280157+0000","last_undegraded":"2023-10-30T13:25:07.280157+0000","last_fullsized":"2023-10-30T13:25:07.280157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1485,"log_dups_size":0,"ondisk_log_size":1485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:19:54.028619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062401690000000003,"stat_sum":{"num_bytes":11021781,"num_objects":617,"num_object_clones":286,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":226,"num_read":116,"num_read_kb":10954,"num_write":1301,"num_write_kb":16114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"77'1620","reported_seq":3589,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.156447+0000","last_change":"2023-10-30T12:37:07.189557+0000","last_active":"2023-10-30T13:25:16.156447+0000","last_peered":"2023-10-30T13:25:16.156447+0000","last_clean":"2023-10-30T13:25:16.156447+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T13:25:16.156447+0000","last_undegraded":"2023-10-30T13:25:16.156447+0000","last_fullsized":"2023-10-30T13:25:16.156447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:17:47.861374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014933905000000001,"stat_sum":{"num_bytes":11161783,"num_objects":653,"num_object_clones":309,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":240,"num_read":128,"num_read_kb":9667,"num_write":1416,"num_write_kb":13402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"82'1476","reported_seq":3519,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.195574+0000","last_change":"2023-10-30T12:37:07.180055+0000","last_active":"2023-10-30T13:25:16.195574+0000","last_peered":"2023-10-30T13:25:16.195574+0000","last_clean":"2023-10-30T13:25:16.195574+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T13:25:16.195574+0000","last_undegraded":"2023-10-30T13:25:16.195574+0000","last_fullsized":"2023-10-30T13:25:16.195574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:02.149498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0090148399999999997,"stat_sum":{"num_bytes":11081748,"num_objects":549,"num_object_clones":251,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":201,"num_read":142,"num_read_kb":9474,"num_write":1288,"num_write_kb":13652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"84'1713","reported_seq":3702,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.192435+0000","last_change":"2023-10-30T12:37:07.181770+0000","last_active":"2023-10-30T13:25:16.192435+0000","last_peered":"2023-10-30T13:25:16.192435+0000","last_clean":"2023-10-30T13:25:16.192435+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T13:25:16.192435+0000","last_undegraded":"2023-10-30T13:25:16.192435+0000","last_fullsized":"2023-10-30T13:25:16.192435+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1713,"log_dups_size":0,"ondisk_log_size":1713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:33.319141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066657749999999997,"stat_sum":{"num_bytes":20125251,"num_objects":693,"num_object_clones":330,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":267,"num_read":167,"num_read_kb":23834,"num_write":1457,"num_write_kb":26871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"82'1548","reported_seq":3554,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.216318+0000","last_change":"2023-10-30T12:37:07.176070+0000","last_active":"2023-10-30T13:25:16.216318+0000","last_peered":"2023-10-30T13:25:16.216318+0000","last_clean":"2023-10-30T13:25:16.216318+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T13:25:16.216318+0000","last_undegraded":"2023-10-30T13:25:16.216318+0000","last_fullsized":"2023-10-30T13:25:16.216318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:28:06.978130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005419129,"stat_sum":{"num_bytes":18031744,"num_objects":604,"num_object_clones":282,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":228,"num_read":136,"num_read_kb":25964,"num_write":1344,"num_write_kb":19625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"85'1548","reported_seq":3561,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.144906+0000","last_change":"2023-10-30T12:37:07.102583+0000","last_active":"2023-10-30T13:24:56.144906+0000","last_peered":"2023-10-30T13:24:56.144906+0000","last_clean":"2023-10-30T13:24:56.144906+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T13:24:56.144906+0000","last_undegraded":"2023-10-30T13:24:56.144906+0000","last_fullsized":"2023-10-30T13:24:56.144906+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:32.507814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013742470000000001,"stat_sum":{"num_bytes":22381695,"num_objects":607,"num_object_clones":275,"num_object_copies":1214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":227,"num_read":141,"num_read_kb":33421,"num_write":1348,"num_write_kb":23422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"85'991","reported_seq":1403,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:53.837590+0000","last_change":"2023-10-30T12:28:44.074842+0000","last_active":"2023-10-30T13:24:53.837590+0000","last_peered":"2023-10-30T13:24:53.837590+0000","last_clean":"2023-10-30T13:24:53.837590+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T13:24:53.837590+0000","last_undegraded":"2023-10-30T13:24:53.837590+0000","last_fullsized":"2023-10-30T13:24:53.837590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":991,"log_dups_size":0,"ondisk_log_size":991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:01.428785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":570,"num_read_kb":319827,"num_write":1218,"num_write_kb":80521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"77'1561","reported_seq":3569,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.084148+0000","last_change":"2023-10-30T12:37:07.114520+0000","last_active":"2023-10-30T13:24:56.084148+0000","last_peered":"2023-10-30T13:24:56.084148+0000","last_clean":"2023-10-30T13:24:56.084148+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T13:24:56.084148+0000","last_undegraded":"2023-10-30T13:24:56.084148+0000","last_fullsized":"2023-10-30T13:24:56.084148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1561,"log_dups_size":0,"ondisk_log_size":1561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:06:30.845566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086018069999999995,"stat_sum":{"num_bytes":19940185,"num_objects":616,"num_object_clones":283,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":213,"num_read":144,"num_read_kb":22076,"num_write":1398,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1795","reported_seq":2375,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:52.266854+0000","last_change":"2023-10-30T12:28:44.060621+0000","last_active":"2023-10-30T13:24:52.266854+0000","last_peered":"2023-10-30T13:24:52.266854+0000","last_clean":"2023-10-30T13:24:52.266854+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T13:24:52.266854+0000","last_undegraded":"2023-10-30T13:24:52.266854+0000","last_fullsized":"2023-10-30T13:24:52.266854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1795,"log_dups_size":0,"ondisk_log_size":1795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:53.741238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17028825,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":761,"num_read_kb":396040,"num_write":1931,"num_write_kb":84639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"85'1569","reported_seq":2387,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:27.320748+0000","last_change":"2023-10-30T12:28:44.060171+0000","last_active":"2023-10-30T13:25:27.320748+0000","last_peered":"2023-10-30T13:25:27.320748+0000","last_clean":"2023-10-30T13:25:27.320748+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T13:25:27.320748+0000","last_undegraded":"2023-10-30T13:25:27.320748+0000","last_fullsized":"2023-10-30T13:25:27.320748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:18.181418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182760,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":1117,"num_read_kb":268002,"num_write":1799,"num_write_kb":63121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"77'1801","reported_seq":3808,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.165978+0000","last_change":"2023-10-30T12:37:07.191924+0000","last_active":"2023-10-30T13:25:16.165978+0000","last_peered":"2023-10-30T13:25:16.165978+0000","last_clean":"2023-10-30T13:25:16.165978+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T13:25:16.165978+0000","last_undegraded":"2023-10-30T13:25:16.165978+0000","last_fullsized":"2023-10-30T13:25:16.165978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1801,"log_dups_size":0,"ondisk_log_size":1801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:27.781535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016143469000000001,"stat_sum":{"num_bytes":14540052,"num_objects":709,"num_object_clones":329,"num_object_copies":1418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":254,"num_read":190,"num_read_kb":6718,"num_write":1583,"num_write_kb":26382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"85'930","reported_seq":1332,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:52.371111+0000","last_change":"2023-10-30T12:28:44.058798+0000","last_active":"2023-10-30T13:24:52.371111+0000","last_peered":"2023-10-30T13:24:52.371111+0000","last_clean":"2023-10-30T13:24:52.371111+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T13:24:52.371111+0000","last_undegraded":"2023-10-30T13:24:52.371111+0000","last_fullsized":"2023-10-30T13:24:52.371111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":930,"log_dups_size":0,"ondisk_log_size":930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:08.643080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":542,"num_read_kb":359029,"num_write":1093,"num_write_kb":88246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"77'1524","reported_seq":3543,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.097691+0000","last_change":"2023-10-30T12:37:07.114493+0000","last_active":"2023-10-30T13:24:56.097691+0000","last_peered":"2023-10-30T13:24:56.097691+0000","last_clean":"2023-10-30T13:24:56.097691+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T13:24:56.097691+0000","last_undegraded":"2023-10-30T13:24:56.097691+0000","last_fullsized":"2023-10-30T13:24:56.097691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:55:36.189245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083101389999999994,"stat_sum":{"num_bytes":18904879,"num_objects":589,"num_object_clones":276,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":211,"num_read":147,"num_read_kb":20021,"num_write":1340,"num_write_kb":25244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"85'736","reported_seq":1136,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:58.054623+0000","last_change":"2023-10-30T12:28:44.063439+0000","last_active":"2023-10-30T13:24:58.054623+0000","last_peered":"2023-10-30T13:24:58.054623+0000","last_clean":"2023-10-30T13:24:58.054623+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T13:24:58.054623+0000","last_undegraded":"2023-10-30T13:24:58.054623+0000","last_fullsized":"2023-10-30T13:24:58.054623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":736,"log_dups_size":0,"ondisk_log_size":736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:08.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":616,"num_read_kb":251423,"num_write":889,"num_write_kb":55053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"77'1518","reported_seq":3533,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.444813+0000","last_change":"2023-10-30T12:37:07.177520+0000","last_active":"2023-10-30T13:25:10.444813+0000","last_peered":"2023-10-30T13:25:10.444813+0000","last_clean":"2023-10-30T13:25:10.444813+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T13:25:10.444813+0000","last_undegraded":"2023-10-30T13:25:10.444813+0000","last_fullsized":"2023-10-30T13:25:10.444813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:46:22.808814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043045619999999996,"stat_sum":{"num_bytes":10134301,"num_objects":580,"num_object_clones":271,"num_object_copies":1160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":580,"num_whiteouts":219,"num_read":138,"num_read_kb":8986,"num_write":1318,"num_write_kb":11276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"85'1247","reported_seq":1685,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:24.345143+0000","last_change":"2023-10-30T12:28:44.052641+0000","last_active":"2023-10-30T13:24:24.345143+0000","last_peered":"2023-10-30T13:24:24.345143+0000","last_clean":"2023-10-30T13:24:24.345143+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T13:24:24.345143+0000","last_undegraded":"2023-10-30T13:24:24.345143+0000","last_fullsized":"2023-10-30T13:24:24.345143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1247,"log_dups_size":0,"ondisk_log_size":1247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":659,"num_read_kb":339384,"num_write":1492,"num_write_kb":80693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"82'1603","reported_seq":3592,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.115007+0000","last_change":"2023-10-30T12:37:07.191144+0000","last_active":"2023-10-30T13:24:56.115007+0000","last_peered":"2023-10-30T13:24:56.115007+0000","last_clean":"2023-10-30T13:24:56.115007+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T13:24:56.115007+0000","last_undegraded":"2023-10-30T13:24:56.115007+0000","last_fullsized":"2023-10-30T13:24:56.115007+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1603,"log_dups_size":0,"ondisk_log_size":1603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:46.268962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015702779,"stat_sum":{"num_bytes":15773172,"num_objects":620,"num_object_clones":292,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":204,"num_read":135,"num_read_kb":17351,"num_write":1431,"num_write_kb":17077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"85'1221","reported_seq":1714,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:27.300445+0000","last_change":"2023-10-30T12:28:44.075783+0000","last_active":"2023-10-30T13:25:27.300445+0000","last_peered":"2023-10-30T13:25:27.300445+0000","last_clean":"2023-10-30T13:25:27.300445+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T13:25:27.300445+0000","last_undegraded":"2023-10-30T13:25:27.300445+0000","last_fullsized":"2023-10-30T13:25:27.300445+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1221,"log_dups_size":0,"ondisk_log_size":1221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:01.094929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19773122,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":862,"num_read_kb":221068,"num_write":1451,"num_write_kb":51532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"82'1515","reported_seq":3524,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.077418+0000","last_change":"2023-10-30T12:37:07.178872+0000","last_active":"2023-10-30T13:25:03.077418+0000","last_peered":"2023-10-30T13:25:03.077418+0000","last_clean":"2023-10-30T13:25:03.077418+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T13:25:03.077418+0000","last_undegraded":"2023-10-30T13:25:03.077418+0000","last_fullsized":"2023-10-30T13:25:03.077418+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:03.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080610860000000003,"stat_sum":{"num_bytes":22077043,"num_objects":612,"num_object_clones":281,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":230,"num_read":141,"num_read_kb":18856,"num_write":1294,"num_write_kb":28337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"85'1331","reported_seq":1877,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:12.181476+0000","last_change":"2023-10-30T12:28:44.063367+0000","last_active":"2023-10-30T13:25:12.181476+0000","last_peered":"2023-10-30T13:25:12.181476+0000","last_clean":"2023-10-30T13:25:12.181476+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T13:25:12.181476+0000","last_undegraded":"2023-10-30T13:25:12.181476+0000","last_fullsized":"2023-10-30T13:25:12.181476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1331,"log_dups_size":0,"ondisk_log_size":1331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:45.585121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":865,"num_read_kb":420163,"num_write":1566,"num_write_kb":89397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"82'1535","reported_seq":3533,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.355487+0000","last_change":"2023-10-30T12:37:07.182869+0000","last_active":"2023-10-30T13:24:48.355487+0000","last_peered":"2023-10-30T13:24:48.355487+0000","last_clean":"2023-10-30T13:24:48.355487+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T13:24:48.355487+0000","last_undegraded":"2023-10-30T13:24:48.355487+0000","last_fullsized":"2023-10-30T13:24:48.355487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1535,"log_dups_size":0,"ondisk_log_size":1535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:35.734555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.008640567,"stat_sum":{"num_bytes":23022369,"num_objects":606,"num_object_clones":279,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":220,"num_read":129,"num_read_kb":18941,"num_write":1340,"num_write_kb":24878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"85'1098","reported_seq":1558,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.290468+0000","last_change":"2023-10-30T12:28:44.065545+0000","last_active":"2023-10-30T13:24:47.290468+0000","last_peered":"2023-10-30T13:24:47.290468+0000","last_clean":"2023-10-30T13:24:47.290468+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T13:24:47.290468+0000","last_undegraded":"2023-10-30T13:24:47.290468+0000","last_fullsized":"2023-10-30T13:24:47.290468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1098,"log_dups_size":0,"ondisk_log_size":1098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:09.900189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554466,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":701,"num_read_kb":318483,"num_write":1303,"num_write_kb":68223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"84'1584","reported_seq":3584,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.315736+0000","last_change":"2023-10-30T12:37:07.115595+0000","last_active":"2023-10-30T13:24:48.315736+0000","last_peered":"2023-10-30T13:24:48.315736+0000","last_clean":"2023-10-30T13:24:48.315736+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T13:24:48.315736+0000","last_undegraded":"2023-10-30T13:24:48.315736+0000","last_fullsized":"2023-10-30T13:24:48.315736+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:20:44.527701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073009930000000004,"stat_sum":{"num_bytes":10084838,"num_objects":613,"num_object_clones":288,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":217,"num_read":139,"num_read_kb":5686,"num_write":1383,"num_write_kb":20147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'676","reported_seq":1068,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.649328+0000","last_change":"2023-10-30T12:28:44.061922+0000","last_active":"2023-10-30T13:24:57.649328+0000","last_peered":"2023-10-30T13:24:57.649328+0000","last_clean":"2023-10-30T13:24:57.649328+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T13:24:57.649328+0000","last_undegraded":"2023-10-30T13:24:57.649328+0000","last_fullsized":"2023-10-30T13:24:57.649328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:34.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":600,"num_read_kb":208389,"num_write":899,"num_write_kb":51587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"82'1642","reported_seq":3652,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.151917+0000","last_change":"2023-10-30T12:37:07.180453+0000","last_active":"2023-10-30T13:25:16.151917+0000","last_peered":"2023-10-30T13:25:16.151917+0000","last_clean":"2023-10-30T13:25:16.151917+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T13:25:16.151917+0000","last_undegraded":"2023-10-30T13:25:16.151917+0000","last_fullsized":"2023-10-30T13:25:16.151917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:20.518544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091258150000000007,"stat_sum":{"num_bytes":19392609,"num_objects":655,"num_object_clones":299,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":233,"num_read":161,"num_read_kb":7475,"num_write":1440,"num_write_kb":20643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"85'1371","reported_seq":1755,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.074769+0000","last_change":"2023-10-30T12:28:44.059784+0000","last_active":"2023-10-30T13:24:48.074769+0000","last_peered":"2023-10-30T13:24:48.074769+0000","last_clean":"2023-10-30T13:24:48.074769+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T13:24:48.074769+0000","last_undegraded":"2023-10-30T13:24:48.074769+0000","last_fullsized":"2023-10-30T13:24:48.074769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1371,"log_dups_size":0,"ondisk_log_size":1371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:11.468221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":539,"num_read_kb":266844,"num_write":1512,"num_write_kb":63647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"77'1481","reported_seq":3489,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.310426+0000","last_change":"2023-10-30T12:37:07.102525+0000","last_active":"2023-10-30T13:24:48.310426+0000","last_peered":"2023-10-30T13:24:48.310426+0000","last_clean":"2023-10-30T13:24:48.310426+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T13:24:48.310426+0000","last_undegraded":"2023-10-30T13:24:48.310426+0000","last_fullsized":"2023-10-30T13:24:48.310426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:22.814663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001646376,"stat_sum":{"num_bytes":13778361,"num_objects":590,"num_object_clones":278,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":227,"num_read":137,"num_read_kb":5723,"num_write":1258,"num_write_kb":14976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"85'970","reported_seq":1455,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.407319+0000","last_change":"2023-10-30T12:28:44.065885+0000","last_active":"2023-10-30T13:24:47.407319+0000","last_peered":"2023-10-30T13:24:47.407319+0000","last_clean":"2023-10-30T13:24:47.407319+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T13:24:47.407319+0000","last_undegraded":"2023-10-30T13:24:47.407319+0000","last_fullsized":"2023-10-30T13:24:47.407319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:26.313595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554544,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":863,"num_read_kb":272228,"num_write":1253,"num_write_kb":56723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"82'1616","reported_seq":3636,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.437909+0000","last_change":"2023-10-30T12:37:07.181542+0000","last_active":"2023-10-30T13:25:10.437909+0000","last_peered":"2023-10-30T13:25:10.437909+0000","last_clean":"2023-10-30T13:25:10.437909+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T13:25:10.437909+0000","last_undegraded":"2023-10-30T13:25:10.437909+0000","last_fullsized":"2023-10-30T13:25:10.437909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1616,"log_dups_size":0,"ondisk_log_size":1616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:16:47.953145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098326820000000006,"stat_sum":{"num_bytes":10901701,"num_objects":622,"num_object_clones":292,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":232,"num_read":163,"num_read_kb":14238,"num_write":1406,"num_write_kb":21652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"85'655","reported_seq":1014,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:04.285953+0000","last_change":"2023-10-30T12:28:44.080011+0000","last_active":"2023-10-30T13:25:04.285953+0000","last_peered":"2023-10-30T13:25:04.285953+0000","last_clean":"2023-10-30T13:25:04.285953+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T13:25:04.285953+0000","last_undegraded":"2023-10-30T13:25:04.285953+0000","last_fullsized":"2023-10-30T13:25:04.285953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":443,"num_read_kb":261670,"num_write":858,"num_write_kb":58986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"77'1573","reported_seq":3538,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.065502+0000","last_change":"2023-10-30T12:37:07.177898+0000","last_active":"2023-10-30T13:25:03.065502+0000","last_peered":"2023-10-30T13:25:03.065502+0000","last_clean":"2023-10-30T13:25:03.065502+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T13:25:03.065502+0000","last_undegraded":"2023-10-30T13:25:03.065502+0000","last_fullsized":"2023-10-30T13:25:03.065502+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:52.251046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004602437,"stat_sum":{"num_bytes":20428330,"num_objects":647,"num_object_clones":304,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":239,"num_read":119,"num_read_kb":13097,"num_write":1360,"num_write_kb":26395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"85'717","reported_seq":1166,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.413162+0000","last_change":"2023-10-30T12:28:44.076352+0000","last_active":"2023-10-30T13:24:48.413162+0000","last_peered":"2023-10-30T13:24:48.413162+0000","last_clean":"2023-10-30T13:24:48.413162+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T13:24:48.413162+0000","last_undegraded":"2023-10-30T13:24:48.413162+0000","last_fullsized":"2023-10-30T13:24:48.413162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":635,"num_read_kb":434171,"num_write":906,"num_write_kb":88121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"77'1575","reported_seq":3558,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:15.950548+0000","last_change":"2023-10-30T12:37:07.181502+0000","last_active":"2023-10-30T13:24:15.950548+0000","last_peered":"2023-10-30T13:24:15.950548+0000","last_clean":"2023-10-30T13:24:15.950548+0000","last_became_active":"2023-10-30T12:29:27.340504+0000","last_became_peered":"2023-10-30T12:29:27.340504+0000","last_unstale":"2023-10-30T13:24:15.950548+0000","last_undegraded":"2023-10-30T13:24:15.950548+0000","last_fullsized":"2023-10-30T13:24:15.950548+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1575,"log_dups_size":0,"ondisk_log_size":1575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:48:15.263466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00653923,"stat_sum":{"num_bytes":23737065,"num_objects":641,"num_object_clones":299,"num_object_copies":1282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":641,"num_whiteouts":238,"num_read":136,"num_read_kb":28242,"num_write":1351,"num_write_kb":33550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'716","reported_seq":1095,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.276257+0000","last_change":"2023-10-30T12:28:44.075684+0000","last_active":"2023-10-30T13:24:47.276257+0000","last_peered":"2023-10-30T13:24:47.276257+0000","last_clean":"2023-10-30T13:24:47.276257+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T13:24:47.276257+0000","last_undegraded":"2023-10-30T13:24:47.276257+0000","last_fullsized":"2023-10-30T13:24:47.276257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:07.375795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364369,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":493,"num_read_kb":320778,"num_write":891,"num_write_kb":71265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"77'1599","reported_seq":3612,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.080865+0000","last_change":"2023-10-30T12:37:07.181841+0000","last_active":"2023-10-30T13:25:03.080865+0000","last_peered":"2023-10-30T13:25:03.080865+0000","last_clean":"2023-10-30T13:25:03.080865+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T13:25:03.080865+0000","last_undegraded":"2023-10-30T13:25:03.080865+0000","last_fullsized":"2023-10-30T13:25:03.080865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1599,"log_dups_size":0,"ondisk_log_size":1599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:35:15.227222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076673150000000001,"stat_sum":{"num_bytes":18256447,"num_objects":611,"num_object_clones":286,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":229,"num_read":149,"num_read_kb":24381,"num_write":1383,"num_write_kb":20195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"85'607","reported_seq":959,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.278226+0000","last_change":"2023-10-30T12:28:44.060686+0000","last_active":"2023-10-30T13:24:47.278226+0000","last_peered":"2023-10-30T13:24:47.278226+0000","last_clean":"2023-10-30T13:24:47.278226+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T13:24:47.278226+0000","last_undegraded":"2023-10-30T13:24:47.278226+0000","last_fullsized":"2023-10-30T13:24:47.278226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:23.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360710,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":462,"num_read_kb":342669,"num_write":689,"num_write_kb":66904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"82'1520","reported_seq":3502,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.077446+0000","last_change":"2023-10-30T12:37:07.182282+0000","last_active":"2023-10-30T13:25:03.077446+0000","last_peered":"2023-10-30T13:25:03.077446+0000","last_clean":"2023-10-30T13:25:03.077446+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T13:25:03.077446+0000","last_undegraded":"2023-10-30T13:25:03.077446+0000","last_fullsized":"2023-10-30T13:25:03.077446+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:14.913475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010286791999999999,"stat_sum":{"num_bytes":26944901,"num_objects":612,"num_object_clones":287,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":236,"num_read":119,"num_read_kb":34683,"num_write":1299,"num_write_kb":33129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":256,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288773+0000","last_change":"2023-10-30T12:28:33.433824+0000","last_active":"2023-10-30T13:24:16.288773+0000","last_peered":"2023-10-30T13:24:16.288773+0000","last_clean":"2023-10-30T13:24:16.288773+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T13:24:16.288773+0000","last_undegraded":"2023-10-30T13:24:16.288773+0000","last_fullsized":"2023-10-30T13:24:16.288773+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_clean_scrub_stamp":"2023-10-30T12:28:31.408255+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:17.624179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'1318","reported_seq":1818,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.269121+0000","last_change":"2023-10-30T12:28:44.059051+0000","last_active":"2023-10-30T13:24:47.269121+0000","last_peered":"2023-10-30T13:24:47.269121+0000","last_clean":"2023-10-30T13:24:47.269121+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T13:24:47.269121+0000","last_undegraded":"2023-10-30T13:24:47.269121+0000","last_fullsized":"2023-10-30T13:24:47.269121+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1318,"log_dups_size":0,"ondisk_log_size":1318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:54.810636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165890,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":877,"num_read_kb":231219,"num_write":1489,"num_write_kb":63137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"85'1574","reported_seq":3582,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.122820+0000","last_change":"2023-10-30T12:37:07.116603+0000","last_active":"2023-10-30T13:25:02.122820+0000","last_peered":"2023-10-30T13:25:02.122820+0000","last_clean":"2023-10-30T13:25:02.122820+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T13:25:02.122820+0000","last_undegraded":"2023-10-30T13:25:02.122820+0000","last_fullsized":"2023-10-30T13:25:02.122820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1574,"log_dups_size":0,"ondisk_log_size":1574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:01:30.768741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068470270000000003,"stat_sum":{"num_bytes":8097312,"num_objects":626,"num_object_clones":290,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":225,"num_read":146,"num_read_kb":2079,"num_write":1425,"num_write_kb":13192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"85'2192","reported_seq":2563,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.650111+0000","last_change":"2023-10-30T12:28:44.080378+0000","last_active":"2023-10-30T13:24:57.650111+0000","last_peered":"2023-10-30T13:24:57.650111+0000","last_clean":"2023-10-30T13:24:57.650111+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T13:24:57.650111+0000","last_undegraded":"2023-10-30T13:24:57.650111+0000","last_fullsized":"2023-10-30T13:24:57.650111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2192,"log_dups_size":0,"ondisk_log_size":2192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:22.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":516,"num_read_kb":283293,"num_write":2369,"num_write_kb":72030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"77'1465","reported_seq":3487,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:15.952966+0000","last_change":"2023-10-30T12:37:07.112121+0000","last_active":"2023-10-30T13:24:15.952966+0000","last_peered":"2023-10-30T13:24:15.952966+0000","last_clean":"2023-10-30T13:24:15.952966+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T13:24:15.952966+0000","last_undegraded":"2023-10-30T13:24:15.952966+0000","last_fullsized":"2023-10-30T13:24:15.952966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:12:48.778193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057755250000000001,"stat_sum":{"num_bytes":14269585,"num_objects":570,"num_object_clones":262,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":207,"num_read":132,"num_read_kb":18022,"num_write":1264,"num_write_kb":17917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'1838","reported_seq":3123,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:09.242564+0000","last_change":"2023-10-30T12:28:44.082480+0000","last_active":"2023-10-30T13:25:09.242564+0000","last_peered":"2023-10-30T13:25:09.242564+0000","last_clean":"2023-10-30T13:25:09.242564+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T13:25:09.242564+0000","last_undegraded":"2023-10-30T13:25:09.242564+0000","last_fullsized":"2023-10-30T13:25:09.242564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1838,"log_dups_size":0,"ondisk_log_size":1838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:20.126782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49235352,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":1514,"num_read_kb":360508,"num_write":2035,"num_write_kb":92558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"77'1546","reported_seq":3555,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.373540+0000","last_change":"2023-10-30T12:37:07.190318+0000","last_active":"2023-10-30T13:24:56.373540+0000","last_peered":"2023-10-30T13:24:56.373540+0000","last_clean":"2023-10-30T13:24:56.373540+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T13:24:56.373540+0000","last_undegraded":"2023-10-30T13:24:56.373540+0000","last_fullsized":"2023-10-30T13:24:56.373540+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:19.397149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015331312,"stat_sum":{"num_bytes":28178512,"num_objects":629,"num_object_clones":288,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":231,"num_read":146,"num_read_kb":37712,"num_write":1347,"num_write_kb":29520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'1196","reported_seq":1605,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.648443+0000","last_change":"2023-10-30T12:28:44.059806+0000","last_active":"2023-10-30T13:24:57.648443+0000","last_peered":"2023-10-30T13:24:57.648443+0000","last_clean":"2023-10-30T13:24:57.648443+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T13:24:57.648443+0000","last_undegraded":"2023-10-30T13:24:57.648443+0000","last_fullsized":"2023-10-30T13:24:57.648443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1196,"log_dups_size":0,"ondisk_log_size":1196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:50.939946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":573,"num_read_kb":334248,"num_write":1338,"num_write_kb":75895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"85'1581","reported_seq":3581,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.228379+0000","last_change":"2023-10-30T12:37:07.178785+0000","last_active":"2023-10-30T13:25:16.228379+0000","last_peered":"2023-10-30T13:25:16.228379+0000","last_clean":"2023-10-30T13:25:16.228379+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T13:25:16.228379+0000","last_undegraded":"2023-10-30T13:25:16.228379+0000","last_fullsized":"2023-10-30T13:25:16.228379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1581,"log_dups_size":0,"ondisk_log_size":1581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:27.865544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051193749999999998,"stat_sum":{"num_bytes":23443395,"num_objects":627,"num_object_clones":293,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":236,"num_read":141,"num_read_kb":26943,"num_write":1376,"num_write_kb":26009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"85'1467","reported_seq":1927,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:01.374791+0000","last_change":"2023-10-30T12:28:44.074970+0000","last_active":"2023-10-30T13:25:01.374791+0000","last_peered":"2023-10-30T13:25:01.374791+0000","last_clean":"2023-10-30T13:25:01.374791+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T13:25:01.374791+0000","last_undegraded":"2023-10-30T13:25:01.374791+0000","last_fullsized":"2023-10-30T13:25:01.374791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1467,"log_dups_size":0,"ondisk_log_size":1467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:33.562703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":699,"num_read_kb":404466,"num_write":1620,"num_write_kb":92575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"82'1612","reported_seq":3615,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:18.067135+0000","last_change":"2023-10-30T12:37:07.176969+0000","last_active":"2023-10-30T13:25:18.067135+0000","last_peered":"2023-10-30T13:25:18.067135+0000","last_clean":"2023-10-30T13:25:18.067135+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T13:25:18.067135+0000","last_undegraded":"2023-10-30T13:25:18.067135+0000","last_fullsized":"2023-10-30T13:25:18.067135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1612,"log_dups_size":0,"ondisk_log_size":1612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:33:15.916870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039865179999999997,"stat_sum":{"num_bytes":21812744,"num_objects":637,"num_object_clones":298,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":235,"num_read":149,"num_read_kb":24572,"num_write":1399,"num_write_kb":31699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"85'1013","reported_seq":1422,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.646473+0000","last_change":"2023-10-30T12:28:44.063256+0000","last_active":"2023-10-30T13:24:57.646473+0000","last_peered":"2023-10-30T13:24:57.646473+0000","last_clean":"2023-10-30T13:24:57.646473+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T13:24:57.646473+0000","last_undegraded":"2023-10-30T13:24:57.646473+0000","last_fullsized":"2023-10-30T13:24:57.646473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1013,"log_dups_size":0,"ondisk_log_size":1013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:01:36.144966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":534,"num_read_kb":318720,"num_write":1164,"num_write_kb":75805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"77'1542","reported_seq":3565,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.430277+0000","last_change":"2023-10-30T12:37:07.114718+0000","last_active":"2023-10-30T13:25:10.430277+0000","last_peered":"2023-10-30T13:25:10.430277+0000","last_clean":"2023-10-30T13:25:10.430277+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T13:25:10.430277+0000","last_undegraded":"2023-10-30T13:25:10.430277+0000","last_fullsized":"2023-10-30T13:25:10.430277+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1542,"log_dups_size":0,"ondisk_log_size":1542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:05.944397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070372949999999998,"stat_sum":{"num_bytes":38977850,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":222,"num_read":154,"num_read_kb":43697,"num_write":1351,"num_write_kb":41194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1137","reported_seq":1571,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.180289+0000","last_change":"2023-10-30T12:28:44.055374+0000","last_active":"2023-10-30T13:25:02.180289+0000","last_peered":"2023-10-30T13:25:02.180289+0000","last_clean":"2023-10-30T13:25:02.180289+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T13:25:02.180289+0000","last_undegraded":"2023-10-30T13:25:02.180289+0000","last_fullsized":"2023-10-30T13:25:02.180289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1137,"log_dups_size":0,"ondisk_log_size":1137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":644,"num_read_kb":249397,"num_write":1858,"num_write_kb":63729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"77'1449","reported_seq":3446,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:15.952034+0000","last_change":"2023-10-30T12:37:07.115578+0000","last_active":"2023-10-30T13:24:15.952034+0000","last_peered":"2023-10-30T13:24:15.952034+0000","last_clean":"2023-10-30T13:24:15.952034+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T13:24:15.952034+0000","last_undegraded":"2023-10-30T13:24:15.952034+0000","last_fullsized":"2023-10-30T13:24:15.952034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1449,"log_dups_size":0,"ondisk_log_size":1449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:52:43.980959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074996079999999996,"stat_sum":{"num_bytes":14238343,"num_objects":603,"num_object_clones":280,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":229,"num_read":126,"num_read_kb":9383,"num_write":1239,"num_write_kb":19514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"85'489","reported_seq":838,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:08.078199+0000","last_change":"2023-10-30T12:28:44.058663+0000","last_active":"2023-10-30T13:25:08.078199+0000","last_peered":"2023-10-30T13:25:08.078199+0000","last_clean":"2023-10-30T13:25:08.078199+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T13:25:08.078199+0000","last_undegraded":"2023-10-30T13:25:08.078199+0000","last_fullsized":"2023-10-30T13:25:08.078199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:42:22.279653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21146410,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":514,"num_read_kb":197148,"num_write":650,"num_write_kb":42825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"77'1649","reported_seq":3622,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.197589+0000","last_change":"2023-10-30T12:37:07.115477+0000","last_active":"2023-10-30T13:25:16.197589+0000","last_peered":"2023-10-30T13:25:16.197589+0000","last_clean":"2023-10-30T13:25:16.197589+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T13:25:16.197589+0000","last_undegraded":"2023-10-30T13:25:16.197589+0000","last_fullsized":"2023-10-30T13:25:16.197589+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1649,"log_dups_size":0,"ondisk_log_size":1649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:44.248368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085509929999999998,"stat_sum":{"num_bytes":14746405,"num_objects":647,"num_object_clones":344,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":203,"num_read":166,"num_read_kb":4091,"num_write":1369,"num_write_kb":17013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'870","reported_seq":1184,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.657578+0000","last_change":"2023-10-30T12:28:44.054338+0000","last_active":"2023-10-30T13:25:02.657578+0000","last_peered":"2023-10-30T13:25:02.657578+0000","last_clean":"2023-10-30T13:25:02.657578+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T13:25:02.657578+0000","last_undegraded":"2023-10-30T13:25:02.657578+0000","last_fullsized":"2023-10-30T13:25:02.657578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":870,"log_dups_size":0,"ondisk_log_size":870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:55:35.370652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":472,"num_read_kb":149352,"num_write":1030,"num_write_kb":38491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"77'1485","reported_seq":3500,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.168854+0000","last_change":"2023-10-30T12:37:07.181506+0000","last_active":"2023-10-30T13:25:16.168854+0000","last_peered":"2023-10-30T13:25:16.168854+0000","last_clean":"2023-10-30T13:25:16.168854+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T13:25:16.168854+0000","last_undegraded":"2023-10-30T13:25:16.168854+0000","last_fullsized":"2023-10-30T13:25:16.168854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1485,"log_dups_size":0,"ondisk_log_size":1485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:53:57.876949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096369869999999996,"stat_sum":{"num_bytes":18113290,"num_objects":584,"num_object_clones":270,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":219,"num_read":134,"num_read_kb":20299,"num_write":1266,"num_write_kb":19768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"85'777","reported_seq":1140,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:04.284846+0000","last_change":"2023-10-30T12:28:44.064606+0000","last_active":"2023-10-30T13:25:04.284846+0000","last_peered":"2023-10-30T13:25:04.284846+0000","last_clean":"2023-10-30T13:25:04.284846+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T13:25:04.284846+0000","last_undegraded":"2023-10-30T13:25:04.284846+0000","last_fullsized":"2023-10-30T13:25:04.284846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":777,"log_dups_size":0,"ondisk_log_size":777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777306,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":495,"num_read_kb":170685,"num_write":965,"num_write_kb":43434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"82'1567","reported_seq":3571,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.146212+0000","last_change":"2023-10-30T12:37:07.112752+0000","last_active":"2023-10-30T13:24:56.146212+0000","last_peered":"2023-10-30T13:24:56.146212+0000","last_clean":"2023-10-30T13:24:56.146212+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T13:24:56.146212+0000","last_undegraded":"2023-10-30T13:24:56.146212+0000","last_fullsized":"2023-10-30T13:24:56.146212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1567,"log_dups_size":0,"ondisk_log_size":1567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:41:59.413302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007081922,"stat_sum":{"num_bytes":14751960,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":228,"num_read":136,"num_read_kb":2513,"num_write":1376,"num_write_kb":21368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1125","reported_seq":1613,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:17.281839+0000","last_change":"2023-10-30T12:28:44.072754+0000","last_active":"2023-10-30T13:25:17.281839+0000","last_peered":"2023-10-30T13:25:17.281839+0000","last_clean":"2023-10-30T13:25:17.281839+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T13:25:17.281839+0000","last_undegraded":"2023-10-30T13:25:17.281839+0000","last_fullsized":"2023-10-30T13:25:17.281839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1125,"log_dups_size":0,"ondisk_log_size":1125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:51:55.865069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":763,"num_read_kb":345628,"num_write":1535,"num_write_kb":76701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"77'1460","reported_seq":3474,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.160346+0000","last_change":"2023-10-30T12:37:07.182563+0000","last_active":"2023-10-30T13:24:56.160346+0000","last_peered":"2023-10-30T13:24:56.160346+0000","last_clean":"2023-10-30T13:24:56.160346+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T13:24:56.160346+0000","last_undegraded":"2023-10-30T13:24:56.160346+0000","last_fullsized":"2023-10-30T13:24:56.160346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1460,"log_dups_size":0,"ondisk_log_size":1460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:29:22.948042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060337460000000004,"stat_sum":{"num_bytes":23618154,"num_objects":570,"num_object_clones":263,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":209,"num_read":127,"num_read_kb":29654,"num_write":1265,"num_write_kb":27771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"85'1182","reported_seq":1585,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:58.045831+0000","last_change":"2023-10-30T12:28:44.058801+0000","last_active":"2023-10-30T13:24:58.045831+0000","last_peered":"2023-10-30T13:24:58.045831+0000","last_clean":"2023-10-30T13:24:58.045831+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T13:24:58.045831+0000","last_undegraded":"2023-10-30T13:24:58.045831+0000","last_fullsized":"2023-10-30T13:24:58.045831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1182,"log_dups_size":0,"ondisk_log_size":1182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:11:24.706813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":538,"num_read_kb":321476,"num_write":1330,"num_write_kb":79987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"85'1546","reported_seq":3551,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.277281+0000","last_change":"2023-10-30T12:37:07.180393+0000","last_active":"2023-10-30T13:24:47.277281+0000","last_peered":"2023-10-30T13:24:47.277281+0000","last_clean":"2023-10-30T13:24:47.277281+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T13:24:47.277281+0000","last_undegraded":"2023-10-30T13:24:47.277281+0000","last_fullsized":"2023-10-30T13:24:47.277281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:28:37.347215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091395910000000007,"stat_sum":{"num_bytes":24394000,"num_objects":612,"num_object_clones":279,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":219,"num_read":133,"num_read_kb":28881,"num_write":1364,"num_write_kb":25892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"77'1527","reported_seq":3550,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.236448+0000","last_change":"2023-10-30T12:37:07.182718+0000","last_active":"2023-10-30T13:25:32.236448+0000","last_peered":"2023-10-30T13:25:32.236448+0000","last_clean":"2023-10-30T13:25:32.236448+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T13:25:32.236448+0000","last_undegraded":"2023-10-30T13:25:32.236448+0000","last_fullsized":"2023-10-30T13:25:32.236448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1527,"log_dups_size":0,"ondisk_log_size":1527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:12:33.103610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010120377999999999,"stat_sum":{"num_bytes":19538275,"num_objects":582,"num_object_clones":275,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":200,"num_read":151,"num_read_kb":26132,"num_write":1354,"num_write_kb":27822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"85'880","reported_seq":1327,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.246911+0000","last_change":"2023-10-30T12:28:44.054652+0000","last_active":"2023-10-30T13:24:47.246911+0000","last_peered":"2023-10-30T13:24:47.246911+0000","last_clean":"2023-10-30T13:24:47.246911+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T13:24:47.246911+0000","last_undegraded":"2023-10-30T13:24:47.246911+0000","last_fullsized":"2023-10-30T13:24:47.246911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:05.023374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":739,"num_read_kb":197443,"num_write":1077,"num_write_kb":51357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"82'1642","reported_seq":3637,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.166307+0000","last_change":"2023-10-30T12:37:07.102323+0000","last_active":"2023-10-30T13:24:56.166307+0000","last_peered":"2023-10-30T13:24:56.166307+0000","last_clean":"2023-10-30T13:24:56.166307+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T13:24:56.166307+0000","last_undegraded":"2023-10-30T13:24:56.166307+0000","last_fullsized":"2023-10-30T13:24:56.166307+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:12.249464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011628750000000001,"stat_sum":{"num_bytes":22561655,"num_objects":686,"num_object_clones":314,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":247,"num_read":162,"num_read_kb":29557,"num_write":1441,"num_write_kb":28808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"85'1048","reported_seq":1588,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.267547+0000","last_change":"2023-10-30T12:28:44.052633+0000","last_active":"2023-10-30T13:24:47.267547+0000","last_peered":"2023-10-30T13:24:47.267547+0000","last_clean":"2023-10-30T13:24:47.267547+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T13:24:47.267547+0000","last_undegraded":"2023-10-30T13:24:47.267547+0000","last_fullsized":"2023-10-30T13:24:47.267547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1048,"log_dups_size":0,"ondisk_log_size":1048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":939,"num_read_kb":433966,"num_write":1214,"num_write_kb":88022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"82'1531","reported_seq":3512,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.140708+0000","last_change":"2023-10-30T12:37:07.189494+0000","last_active":"2023-10-30T13:25:16.140708+0000","last_peered":"2023-10-30T13:25:16.140708+0000","last_clean":"2023-10-30T13:25:16.140708+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T13:25:16.140708+0000","last_undegraded":"2023-10-30T13:25:16.140708+0000","last_fullsized":"2023-10-30T13:25:16.140708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1531,"log_dups_size":0,"ondisk_log_size":1531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:34.313998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014632805,"stat_sum":{"num_bytes":14611265,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":226,"num_read":122,"num_read_kb":17905,"num_write":1326,"num_write_kb":16683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"85'753","reported_seq":1130,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.373735+0000","last_change":"2023-10-30T12:28:44.075493+0000","last_active":"2023-10-30T13:24:56.373735+0000","last_peered":"2023-10-30T13:24:56.373735+0000","last_clean":"2023-10-30T13:24:56.373735+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T13:24:56.373735+0000","last_undegraded":"2023-10-30T13:24:56.373735+0000","last_fullsized":"2023-10-30T13:24:56.373735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:00:11.718669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":488,"num_read_kb":379559,"num_write":896,"num_write_kb":83880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"77'1481","reported_seq":3496,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.138978+0000","last_change":"2023-10-30T12:37:07.191451+0000","last_active":"2023-10-30T13:24:56.138978+0000","last_peered":"2023-10-30T13:24:56.138978+0000","last_clean":"2023-10-30T13:24:56.138978+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T13:24:56.138978+0000","last_undegraded":"2023-10-30T13:24:56.138978+0000","last_fullsized":"2023-10-30T13:24:56.138978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:28.248158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015871881000000001,"stat_sum":{"num_bytes":15563160,"num_objects":581,"num_object_clones":268,"num_object_copies":1162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":208,"num_read":132,"num_read_kb":9741,"num_write":1268,"num_write_kb":20904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"85'838","reported_seq":1196,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.123097+0000","last_change":"2023-10-30T12:28:44.065843+0000","last_active":"2023-10-30T13:25:02.123097+0000","last_peered":"2023-10-30T13:25:02.123097+0000","last_clean":"2023-10-30T13:25:02.123097+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T13:25:02.123097+0000","last_undegraded":"2023-10-30T13:25:02.123097+0000","last_fullsized":"2023-10-30T13:25:02.123097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:32.405199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33555014,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":517,"num_read_kb":175570,"num_write":978,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165214918,"num_objects":39301,"num_object_clones":18317,"num_object_copies":78602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39301,"num_whiteouts":14249,"num_read":9248,"num_read_kb":1263731,"num_write":86732,"num_write_kb":1459764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424127488,"data_stored":2332445068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99517,"ondisk_log_size":99517,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":903125000,"num_objects":1657,"num_object_clones":0,"num_object_copies":3314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":21550,"num_read_kb":9552846,"num_write":41298,"num_write_kb":2210409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1806409728,"data_stored":1806250000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34753,"ondisk_log_size":34753,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477425,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":837308,"kb_used_data":639604,"kb_used_omap":1,"kb_used_meta":197694,"kb_avail":93534532,"statfs":{"total":96636764160,"available":95779360768,"internally_reserved":0,"allocated":654954496,"data_stored":640799419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":202439112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42699999999999999,"15min":0.434},"min":{"1min":0.253,"5min":0.253,"15min":0.23200000000000001},"max":{"1min":0.52300000000000002,"5min":0.66900000000000004,"15min":0.94599999999999995},"last":0.377},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.47099999999999997,"15min":0.46999999999999997},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.27700000000000002},"max":{"1min":0.68300000000000005,"5min":0.75800000000000001,"15min":0.81899999999999995},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.503,"15min":0.495},"min":{"1min":0.39000000000000001,"5min":0.33300000000000002,"15min":0.307},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":1.0209999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.76600000000000001,"5min":0.78400000000000003,"15min":0.99299999999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51500000000000001,"15min":0.502},"min":{"1min":0.373,"5min":0.34100000000000003,"15min":0.28599999999999998},"max":{"1min":0.747,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.434},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.42999999999999999,"5min":0.38600000000000001,"15min":0.33200000000000002},"max":{"1min":0.79700000000000004,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77900000000000003,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.94299999999999995,"5min":1.05,"15min":1.548},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.63900000000000001,"5min":0.497,"15min":0.497},"max":{"1min":1.127,"5min":1.149,"15min":1.6950000000000001},"last":0.73299999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.78900000000000003,"15min":0.81399999999999995},"min":{"1min":0.57299999999999995,"5min":0.52800000000000002,"15min":0.496},"max":{"1min":0.96399999999999997,"5min":1.1160000000000001,"15min":1.194},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.80000000000000004,"15min":0.80700000000000005},"min":{"1min":0.66100000000000003,"5min":0.63200000000000001,"15min":0.52100000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.603},"last":0.623}]},{"osd":6,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.64000000000000001,"5min":0.56999999999999995,"15min":0.436},"max":{"1min":1.113,"5min":1.198,"15min":1.2270000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.93100000000000005,"15min":0.92300000000000004},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.55600000000000005},"max":{"1min":1.145,"5min":1.202,"15min":1.3759999999999999},"last":0.75900000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.84799999999999998,"15min":0.83599999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.46400000000000002},"max":{"1min":1.133,"5min":1.133,"15min":1.1919999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.91400000000000003,"15min":0.877},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.47699999999999998},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.2050000000000001},"last":0.83299999999999996}]}]},{"osd":2,"up_from":16,"seq":68719477426,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534076,"kb_used_data":412724,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":93837764,"statfs":{"total":96636764160,"available":96089870336,"internally_reserved":0,"allocated":422629376,"data_stored":409295067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124254662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42999999999999999,"15min":0.43099999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57799999999999996,"5min":0.81299999999999994,"15min":0.91200000000000003},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48799999999999999,"15min":0.47899999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.625,"5min":1.554,"15min":2.8519999999999999},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47799999999999998,"15min":0.497},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.57199999999999995,"5min":1.071,"15min":4.9660000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52700000000000002,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.32100000000000001,"15min":0.28399999999999997},"max":{"1min":0.66600000000000004,"5min":1.198,"15min":6.0759999999999996},"last":0.442}]},{"osd":3,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.505,"15min":0.52800000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.314},"max":{"1min":0.65500000000000003,"5min":0.96799999999999997,"15min":4.6120000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56499999999999995,"15min":0.58699999999999997},"min":{"1min":0.42699999999999999,"5min":0.40899999999999997,"15min":0.379},"max":{"1min":0.68200000000000005,"5min":1.4430000000000001,"15min":4.641},"last":0.433}]},{"osd":4,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79100000000000004,"15min":0.80400000000000005},"min":{"1min":0.58399999999999996,"5min":0.48299999999999998,"15min":0.47999999999999998},"max":{"1min":0.95999999999999996,"5min":1.1719999999999999,"15min":3.5499999999999998},"last":1.083},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77000000000000002,"15min":0.81000000000000005},"min":{"1min":0.53200000000000003,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0389999999999999,"5min":1.262,"15min":5.032},"last":0.76100000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.88800000000000001,"15min":0.87},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.177,"5min":1.494,"15min":6.1289999999999996},"last":1.014},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.84699999999999998,"15min":0.877},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.159,"5min":1.3440000000000001,"15min":3.1459999999999999},"last":0.96899999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.83299999999999996,"15min":0.875},"min":{"1min":0.67600000000000005,"5min":0.54000000000000004,"15min":0.53100000000000003},"max":{"1min":1.0469999999999999,"5min":1.1739999999999999,"15min":5.8159999999999998},"last":1.139},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.82799999999999996,"15min":0.83299999999999996},"min":{"1min":0.66600000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.99299999999999999,"5min":1.6299999999999999,"15min":3.415},"last":1.0620000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.84199999999999997,"15min":0.872},"min":{"1min":0.73999999999999999,"5min":0.60699999999999998,"15min":0.51100000000000001},"max":{"1min":1.121,"5min":1.9139999999999999,"15min":4.883},"last":1.107},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.88400000000000001,"15min":0.90300000000000002},"min":{"1min":0.79300000000000004,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0640000000000001,"5min":1.3460000000000001,"15min":4.9950000000000001},"last":1.1619999999999999}]}]},{"osd":1,"up_from":16,"seq":68719477425,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534824,"kb_used_data":431520,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":93837016,"statfs":{"total":96636764160,"available":96089104384,"internally_reserved":0,"allocated":441876480,"data_stored":431229668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105773510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46899999999999997,"15min":0.52500000000000002},"min":{"1min":0.32800000000000001,"5min":0.27700000000000002,"15min":0.22600000000000001},"max":{"1min":0.66700000000000004,"5min":0.70699999999999996,"15min":20.196000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55100000000000005,"15min":0.54800000000000004},"min":{"1min":0.39800000000000002,"5min":0.31,"15min":0.253},"max":{"1min":0.77200000000000002,"5min":0.91100000000000003,"15min":5.1070000000000002},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48599999999999999,"15min":0.67200000000000004},"min":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.312},"max":{"1min":0.73099999999999998,"5min":0.83999999999999997,"15min":45.866999999999997},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.57099999999999995,"15min":0.64100000000000001},"min":{"1min":0.433,"5min":0.374,"15min":0.34699999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":20.789999999999999},"last":0.46400000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.47899999999999998,"15min":0.57899999999999996},"min":{"1min":0.36899999999999999,"5min":0.318,"15min":0.30499999999999999},"max":{"1min":0.624,"5min":0.80900000000000005,"15min":21.178999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.34999999999999998},"max":{"1min":0.71099999999999997,"5min":0.91800000000000004,"15min":2.2290000000000001},"last":0.50800000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.40400000000000003},"max":{"1min":1.133,"5min":1.133,"15min":5.9770000000000003},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.90400000000000003,"15min":0.86899999999999999},"min":{"1min":0.68400000000000005,"5min":0.64300000000000002,"15min":0.44400000000000001},"max":{"1min":1.093,"5min":1.1619999999999999,"15min":5.5110000000000001},"last":0.875}]},{"osd":5,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82399999999999995,"15min":0.872},"min":{"1min":0.58299999999999996,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0149999999999999,"5min":1.212,"15min":7.5430000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.90100000000000002,"15min":0.94499999999999995},"min":{"1min":0.76700000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0680000000000001,"5min":1.2350000000000001,"15min":7.5750000000000002},"last":0.84199999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.79900000000000004,"15min":0.88300000000000001},"min":{"1min":0.59699999999999998,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0820000000000001,"5min":1.177,"15min":20.827000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.82099999999999995,"15min":0.95399999999999996},"min":{"1min":0.68400000000000005,"5min":0.51800000000000002,"15min":0.51600000000000001},"max":{"1min":1.0469999999999999,"5min":1.159,"15min":20.931000000000001},"last":0.72499999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.83999999999999997,"15min":0.95799999999999996},"min":{"1min":0.73299999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":20.859999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90600000000000003,"15min":0.88500000000000001},"min":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.501},"max":{"1min":1.1870000000000001,"5min":1.194,"15min":4.367},"last":0.91600000000000004}]}]},{"osd":0,"up_from":16,"seq":68719477426,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937928,"kb_used_data":695808,"kb_used_omap":1,"kb_used_meta":242110,"kb_avail":93433912,"statfs":{"total":96636764160,"available":95676325888,"internally_reserved":0,"allocated":712507392,"data_stored":696810257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":247921094},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40600000000000003,"15min":0.41099999999999998},"min":{"1min":0.254,"5min":0.23899999999999999,"15min":0.22700000000000001},"max":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":3.996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51700000000000002,"15min":0.52600000000000002},"min":{"1min":0.41099999999999998,"5min":0.36199999999999999,"15min":0.309},"max":{"1min":0.67600000000000005,"5min":0.75600000000000001,"15min":4.3230000000000004},"last":0.44700000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.27700000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":4.9690000000000003},"last":0.372},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52100000000000002,"15min":0.53200000000000003},"min":{"1min":0.372,"5min":0.34100000000000003,"15min":0.30199999999999999},"max":{"1min":0.621,"5min":0.878,"15min":5.0049999999999999},"last":0.42499999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.48799999999999999},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.28299999999999997},"max":{"1min":0.68100000000000005,"5min":0.82699999999999996,"15min":5.4909999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57499999999999996,"15min":0.58099999999999996},"min":{"1min":0.443,"5min":0.42799999999999999,"15min":0.38600000000000001},"max":{"1min":0.749,"5min":0.90400000000000003,"15min":5.6970000000000001},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.58699999999999997,"5min":0.46700000000000003,"15min":0.377},"max":{"1min":1.111,"5min":1.111,"15min":4.444},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.80100000000000005},"min":{"1min":0.58499999999999996,"5min":0.55000000000000004,"15min":0.48099999999999998},"max":{"1min":0.90500000000000003,"5min":1.117,"15min":5.7750000000000004},"last":0.622}]},{"osd":5,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.83299999999999996,"15min":0.83499999999999996},"min":{"1min":0.66800000000000004,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.246,"5min":1.3340000000000001,"15min":4.5430000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.871,"15min":0.86399999999999999},"min":{"1min":0.59199999999999997,"5min":0.56999999999999995,"15min":0.47099999999999997},"max":{"1min":1.204,"5min":1.204,"15min":4.4790000000000001},"last":0.76200000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.82599999999999996,"15min":0.84199999999999997},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":1.397,"15min":4.5129999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.88500000000000001,"15min":0.89900000000000002},"min":{"1min":0.626,"5min":0.626,"15min":0.56699999999999995},"max":{"1min":1.1699999999999999,"5min":1.3120000000000001,"15min":4.5750000000000002},"last":0.60399999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.875,"15min":0.86799999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.55700000000000005},"max":{"1min":1.0669999999999999,"5min":1.206,"15min":4.3979999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.88900000000000001,"15min":0.91400000000000003},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.53700000000000003},"max":{"1min":1.0760000000000001,"5min":1.2210000000000001,"15min":5.375},"last":0.79700000000000004}]}]},{"osd":6,"up_from":15,"seq":64424510128,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526444,"kb_used_data":409252,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":93845396,"statfs":{"total":96636764160,"available":96097685504,"internally_reserved":0,"allocated":419074048,"data_stored":408471134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":119994823},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.874,"15min":0.86299999999999999},"min":{"1min":0.61099999999999999,"5min":0.49399999999999999,"15min":0.34200000000000003},"max":{"1min":1.167,"5min":1.4470000000000001,"15min":3.1150000000000002},"last":1.103},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.89900000000000002,"15min":0.96999999999999997},"min":{"1min":0.63500000000000001,"5min":0.54500000000000004,"15min":0.51700000000000002},"max":{"1min":1.121,"5min":1.415,"15min":3.3639999999999999},"last":0.89500000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.79800000000000004,"15min":0.79700000000000004},"min":{"1min":0.48799999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.087,"5min":1.202,"15min":1.589},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85899999999999999,"15min":0.86199999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.47699999999999998},"max":{"1min":1.145,"5min":1.2330000000000001,"15min":1.544},"last":0.83699999999999997}]},{"osd":2,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.98199999999999998,"15min":0.96299999999999997},"min":{"1min":0.68300000000000005,"5min":0.57299999999999995,"15min":0.47199999999999998},"max":{"1min":1.2969999999999999,"5min":1.496,"15min":3.4969999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.91600000000000004,"15min":0.94999999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.2090000000000001,"5min":1.391,"15min":1.512},"last":1.018}]},{"osd":3,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.86299999999999999,"15min":0.89100000000000001},"min":{"1min":0.58799999999999997,"5min":0.57099999999999995,"15min":0.52800000000000002},"max":{"1min":1.131,"5min":1.262,"15min":3.6070000000000002},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.91300000000000003,"15min":0.88300000000000001},"min":{"1min":0.69499999999999995,"5min":0.54000000000000004,"15min":0.51600000000000001},"max":{"1min":1.1839999999999999,"5min":1.272,"15min":3.7679999999999998},"last":0.77600000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97299999999999998,"15min":0.93799999999999994},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.2849999999999999,"5min":1.3220000000000001,"15min":1.4019999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.093,"15min":1.046},"min":{"1min":0.80000000000000004,"5min":0.73899999999999999,"15min":0.54700000000000004},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.2509999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.163,"15min":1.125},"min":{"1min":0.90100000000000002,"5min":0.82999999999999996,"15min":0.73399999999999999},"max":{"1min":1.417,"5min":1.4730000000000001,"15min":1.583},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.069,"15min":1.095},"min":{"1min":0.625,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2829999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.137}]},{"osd":7,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.081,"15min":1.0800000000000001},"min":{"1min":0.86099999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.4970000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.22,"15min":1.2170000000000001},"min":{"1min":1.034,"5min":0.76500000000000001,"15min":0.64200000000000002},"max":{"1min":1.804,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.1699999999999999}]}]},{"osd":4,"up_from":15,"seq":64424510130,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645792,"kb_used_data":493912,"kb_used_omap":1,"kb_used_meta":151870,"kb_avail":93726048,"statfs":{"total":96636764160,"available":95975473152,"internally_reserved":0,"allocated":505765888,"data_stored":496226659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":155515335},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.86299999999999999,"15min":0.83899999999999997},"min":{"1min":0.54800000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.1930000000000001,"5min":1.218,"15min":1.292},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.86899999999999999,"15min":0.84699999999999998},"min":{"1min":0.57399999999999995,"5min":0.52700000000000002,"15min":0.52600000000000002},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.369},"last":1.4570000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.79400000000000004,"15min":0.78600000000000003},"min":{"1min":0.60799999999999998,"5min":0.47899999999999998,"15min":0.47499999999999998},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1839999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.879,"5min":0.83699999999999997,"15min":0.83199999999999996},"min":{"1min":0.60899999999999999,"5min":0.53000000000000003,"15min":0.47299999999999998},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.3100000000000001},"last":0.996}]},{"osd":2,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.89500000000000002},"min":{"1min":0.56599999999999995,"5min":0.55500000000000005,"15min":0.55300000000000005},"max":{"1min":1.1899999999999999,"5min":1.2270000000000001,"15min":1.292},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.998,"5min":0.94599999999999995,"15min":0.94699999999999995},"min":{"1min":0.74099999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":1.3540000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.97399999999999998,"15min":0.95299999999999996},"min":{"1min":0.68500000000000005,"5min":0.65100000000000002,"15min":0.52200000000000002},"max":{"1min":1.286,"5min":1.286,"15min":1.514},"last":1.234},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.94699999999999995,"15min":0.96299999999999997},"min":{"1min":0.76600000000000001,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.94999999999999996}]},{"osd":5,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.98299999999999998,"15min":1.0129999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60299999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":2.4430000000000001},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.171,"5min":1.1140000000000001,"15min":1.0660000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.8460000000000001},"last":1.304}]},{"osd":6,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.145,"15min":1.1559999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.419,"5min":1.425,"15min":1.677},"last":1.411},{"interface":"front","average":{"1min":1.306,"5min":1.28,"15min":1.1830000000000001},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.64400000000000002},"max":{"1min":1.528,"5min":1.5409999999999999,"15min":1.9610000000000001},"last":1.5009999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.0529999999999999,"15min":1.1359999999999999},"min":{"1min":0.89500000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.8700000000000001},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.202,"15min":1.266},"min":{"1min":1.016,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.538,"5min":1.591,"15min":1.9450000000000001},"last":1.2689999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542831,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560428,"kb_used_data":416932,"kb_used_omap":1,"kb_used_meta":143486,"kb_avail":93811412,"statfs":{"total":96636764160,"available":96062885888,"internally_reserved":0,"allocated":426938368,"data_stored":418859807,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":146930123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.84399999999999997,"15min":0.82999999999999996},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.41499999999999998},"max":{"1min":1.2130000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.88,"15min":0.872},"min":{"1min":0.56999999999999995,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.333,"5min":1.3380000000000001,"15min":1.3919999999999999},"last":0.877}]},{"osd":1,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.76900000000000002,"15min":0.86799999999999999},"min":{"1min":0.504,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.095,"5min":1.244,"15min":1.4790000000000001},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.91700000000000004,"15min":0.82299999999999995},"min":{"1min":0.51500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.5,"5min":1.524,"15min":1.524},"last":1.006}]},{"osd":2,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0589999999999999,"15min":1.024},"min":{"1min":0.66500000000000004,"5min":0.54700000000000004,"15min":0.52700000000000002},"max":{"1min":1.3109999999999999,"5min":1.833,"15min":1.833},"last":1.147},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1830000000000001,"15min":1.153},"min":{"1min":0.91100000000000003,"5min":0.71499999999999997,"15min":0.69399999999999995},"max":{"1min":1.4510000000000001,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.3140000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0760000000000001,"15min":1.083},"min":{"1min":0.66400000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.2929999999999999,"5min":1.673,"15min":1.673},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.204,"5min":1.244,"15min":1.2230000000000001},"min":{"1min":0.84299999999999997,"5min":0.83499999999999996,"15min":0.81399999999999995},"max":{"1min":1.4490000000000001,"5min":1.8109999999999999,"15min":1.8480000000000001},"last":1.2709999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.86199999999999999,"15min":0.84799999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.171,"5min":1.226,"15min":1.4970000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.95799999999999996,"15min":0.95599999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.242,"5min":1.4970000000000001,"15min":1.77},"last":1.04}]},{"osd":5,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.94299999999999995,"15min":0.97499999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.2490000000000001,"5min":1.3799999999999999,"15min":2},"last":1.091},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.105,"15min":1.0349999999999999},"min":{"1min":0.69599999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.508,"5min":1.7150000000000001,"15min":2.056},"last":1.2450000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.042,"15min":1.008},"min":{"1min":0.68100000000000005,"5min":0.60799999999999998,"15min":0.501},"max":{"1min":1.377,"5min":1.536,"15min":1.571},"last":1.294},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1299999999999999,"15min":1.105},"min":{"1min":0.751,"5min":0.72599999999999998,"15min":0.71199999999999997},"max":{"1min":1.5549999999999999,"5min":1.665,"15min":1.665},"last":1.194}]}]},{"osd":5,"up_from":14,"seq":60129542830,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":802272,"kb_used_data":640600,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93569568,"statfs":{"total":96636764160,"available":95815237632,"internally_reserved":0,"allocated":655974400,"data_stored":642425281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165542342},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.378},"max":{"1min":1.173,"5min":1.276,"15min":1.579},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.85299999999999998,"15min":0.86099999999999999},"min":{"1min":0.499,"5min":0.46200000000000002,"15min":0.34599999999999997},"max":{"1min":1.262,"5min":1.627,"15min":1.627},"last":0.86299999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.81200000000000006,"15min":0.83099999999999996},"min":{"1min":0.55300000000000005,"5min":0.47499999999999998,"15min":0.39500000000000002},"max":{"1min":1.22,"5min":1.226,"15min":1.3220000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.88600000000000001,"15min":0.91900000000000004},"min":{"1min":0.64800000000000002,"5min":0.52600000000000002,"15min":0.376},"max":{"1min":1.1539999999999999,"5min":1.3100000000000001,"15min":1.6619999999999999},"last":0.72299999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.85799999999999998,"15min":0.84799999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.2330000000000001,"5min":1.3220000000000001,"15min":1.486},"last":0.5},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.89300000000000002,"15min":0.88},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2749999999999999,"5min":1.3149999999999999,"15min":1.879},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.96899999999999997,"15min":0.88900000000000001},"min":{"1min":0.65500000000000003,"5min":0.55900000000000005,"15min":0.42799999999999999},"max":{"1min":1.361,"5min":1.361,"15min":2.7330000000000001},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.097,"5min":1.0509999999999999,"15min":0.999},"min":{"1min":0.68899999999999995,"5min":0.51300000000000001,"15min":0.47099999999999997},"max":{"1min":1.359,"5min":1.5169999999999999,"15min":2.79},"last":1.1479999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.97299999999999998,"15min":0.98099999999999998},"min":{"1min":0.68500000000000005,"5min":0.57599999999999996,"15min":0.45300000000000001},"max":{"1min":1.119,"5min":1.4710000000000001,"15min":1.6719999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.034,"15min":1.0940000000000001},"min":{"1min":0.87,"5min":0.626,"15min":0.58799999999999997},"max":{"1min":1.3120000000000001,"5min":1.3540000000000001,"15min":1.7629999999999999},"last":0.77900000000000003}]},{"osd":6,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.1000000000000001,"15min":1.0920000000000001},"min":{"1min":0.69999999999999996,"5min":0.66500000000000004,"15min":0.54600000000000004},"max":{"1min":1.3959999999999999,"5min":1.675,"15min":2.8450000000000002},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.123,"15min":1.111},"min":{"1min":0.73699999999999999,"5min":0.59999999999999998,"15min":0.57999999999999996},"max":{"1min":1.577,"5min":1.625,"15min":1.821},"last":0.98699999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.153,"15min":1.127},"min":{"1min":0.70099999999999996,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7490000000000001,"5min":1.7749999999999999,"15min":3.3069999999999999},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.115,"5min":1.1579999999999999,"15min":1.113},"min":{"1min":0.88500000000000001,"5min":0.59999999999999998,"15min":0.53200000000000003},"max":{"1min":1.3129999999999999,"5min":1.754,"15min":2.036},"last":0.92000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309846016,"data_stored":309822659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142618624,"data_stored":142610667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117452800,"data_stored":117441250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294043648,"data_stored":294028449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":275668992,"data_stored":275627234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":300916736,"data_stored":300897937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":154218496,"data_stored":154206848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":211644416,"data_stored":211614956,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401661952,"data_stored":386457412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297664512,"data_stored":287498447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304177152,"data_stored":291323631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359911424,"data_stored":346240784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":229097472,"data_stored":220069239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":353464320,"data_stored":340406790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":263856128,"data_stored":253734100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":214294528,"data_stored":206714665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:25:35.993 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:25:36.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f6659891640 1 Processor -- start 2023-10-30T13:25:36.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f6659891640 1 -- start start 2023-10-30T13:25:36.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f6659891640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f66540c6b80 0x7f66540bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f6659891640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f665413f4a0 0x7f66540c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f6659891640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66540c0a50 con 0x7f66540c6f60 2023-10-30T13:25:36.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f6659891640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665406f6e0 con 0x7f665413f4a0 2023-10-30T13:25:36.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f6659891640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665406f9c0 con 0x7f66540c6b80 2023-10-30T13:25:36.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f66537fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f665413f4a0 0x7f66540c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f66537fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f665413f4a0 0x7f66540c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38560/0 (socket says 172.21.15.125:38560) 2023-10-30T13:25:36.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f66537fe640 1 -- 172.21.15.125:0/3736926976 learned_addr learned my addr 172.21.15.125:0/3736926976 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:36.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f6652ffd640 1 --2- 172.21.15.125:0/3736926976 >> v2:172.21.15.136:3300/0 conn(0x7f66540c6b80 0x7f66540bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f66537fe640 1 -- 172.21.15.125:0/3736926976 >> v2:172.21.15.136:3300/0 conn(0x7f66540c6b80 msgr2=0x7f66540bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:36.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f66537fe640 1 --2- 172.21.15.125:0/3736926976 >> v2:172.21.15.136:3300/0 conn(0x7f66540c6b80 0x7f66540bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f66537fe640 1 -- 172.21.15.125:0/3736926976 >> v1:172.21.15.136:6789/0 conn(0x7f66540c6f60 legacy=0x7f665413ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.170+0000 7f66537fe640 1 -- 172.21.15.125:0/3736926976 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665406fca0 con 0x7f665413f4a0 2023-10-30T13:25:36.178 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.174+0000 7f66537fe640 1 --2- 172.21.15.125:0/3736926976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f665413f4a0 0x7f66540c0400 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f665406f1b0 tx=0x7f664c01c9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cea15d504d8aaa8 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:36.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.174+0000 7f6651ffb640 1 -- 172.21.15.125:0/3736926976 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c0a57a0 con 0x7f665413f4a0 2023-10-30T13:25:36.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.174+0000 7f6651ffb640 1 -- 172.21.15.125:0/3736926976 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f664c0a5900 con 0x7f665413f4a0 2023-10-30T13:25:36.179 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.174+0000 7f6651ffb640 1 -- 172.21.15.125:0/3736926976 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f664c0a5c90 con 0x7f665413f4a0 2023-10-30T13:25:36.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.174+0000 7f6659891640 1 -- 172.21.15.125:0/3736926976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f665413f4a0 msgr2=0x7f66540c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.174+0000 7f6659891640 1 --2- 172.21.15.125:0/3736926976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f665413f4a0 0x7f66540c0400 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f665406f1b0 tx=0x7f664c01c9d0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.174+0000 7f6659891640 1 -- 172.21.15.125:0/3736926976 shutdown_connections 2023-10-30T13:25:36.180 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.174+0000 7f6659891640 1 --2- 172.21.15.125:0/3736926976 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f665413f4a0 0x7f66540c0400 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.174+0000 7f6659891640 1 --2- 172.21.15.125:0/3736926976 >> v2:172.21.15.136:3300/0 conn(0x7f66540c6b80 0x7f66540bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.174+0000 7f6659891640 1 -- 172.21.15.125:0/3736926976 >> 172.21.15.125:0/3736926976 conn(0x7f66540c7910 msgr2=0x7f66540c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:36.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.174+0000 7f6659891640 1 -- 172.21.15.125:0/3736926976 shutdown_connections 2023-10-30T13:25:36.181 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.174+0000 7f6659891640 1 -- 172.21.15.125:0/3736926976 wait complete. 2023-10-30T13:25:36.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.174+0000 7f6659891640 1 Processor -- start 2023-10-30T13:25:36.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.174+0000 7f6659891640 1 -- start start 2023-10-30T13:25:36.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f6659891640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f66540c6b80 0x7f66541322e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f6659891640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66540c6f60 0x7f665414ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f6659891640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f665413f4a0 0x7f6654131620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f6659891640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6654150310 con 0x7f66540c6f60 2023-10-30T13:25:36.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f6659891640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6654150520 con 0x7f665413f4a0 2023-10-30T13:25:36.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f6659891640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665414f860 con 0x7f66540c6b80 2023-10-30T13:25:36.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f66527fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66540c6f60 0x7f665414ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f66527fc640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66540c6f60 0x7f665414ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38570/0 (socket says 172.21.15.125:38570) 2023-10-30T13:25:36.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f66527fc640 1 -- 172.21.15.125:0/45794540 learned_addr learned my addr 172.21.15.125:0/45794540 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:36.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f6652ffd640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f66540c6b80 0x7f66541322e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f66537fe640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f665413f4a0 0x7f6654131620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f66527fc640 1 -- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f66540c6b80 msgr2=0x7f66541322e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f66527fc640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f66540c6b80 0x7f66541322e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f66527fc640 1 -- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f665413f4a0 msgr2=0x7f6654131620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f66527fc640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f665413f4a0 0x7f6654131620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f66527fc640 1 -- 172.21.15.125:0/45794540 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c009b30 con 0x7f66540c6f60 2023-10-30T13:25:36.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f6652ffd640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f66540c6b80 0x7f66541322e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:36.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f66527fc640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66540c6f60 0x7f665414ed70 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f664800c500 tx=0x7f664800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:36.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f663bfff640 1 -- 172.21.15.125:0/45794540 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648005220 con 0x7f66540c6f60 2023-10-30T13:25:36.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f663bfff640 1 -- 172.21.15.125:0/45794540 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6648005380 con 0x7f66540c6f60 2023-10-30T13:25:36.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f6659891640 1 -- 172.21.15.125:0/45794540 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66541f6080 con 0x7f66540c6f60 2023-10-30T13:25:36.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f663bfff640 1 -- 172.21.15.125:0/45794540 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6648007270 con 0x7f66540c6f60 2023-10-30T13:25:36.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f6659891640 1 -- 172.21.15.125:0/45794540 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66541f5d40 con 0x7f66540c6f60 2023-10-30T13:25:36.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f6659891640 1 -- 172.21.15.125:0/45794540 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665404e170 con 0x7f66540c6f60 2023-10-30T13:25:36.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.178+0000 7f663bfff640 1 -- 172.21.15.125:0/45794540 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f66480073d0 con 0x7f66540c6f60 2023-10-30T13:25:36.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.182+0000 7f663bfff640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66400664d0 0x7f6640068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.182+0000 7f663bfff640 1 -- 172.21.15.125:0/45794540 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f6648084250 con 0x7f66540c6f60 2023-10-30T13:25:36.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.182+0000 7f6652ffd640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66400664d0 0x7f6640068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.182+0000 7f6652ffd640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66400664d0 0x7f6640068990 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f6644007a70 tx=0x7f6644008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:36.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.186+0000 7f663bfff640 1 -- 172.21.15.125:0/45794540 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6648050ba0 con 0x7f66540c6f60 2023-10-30T13:25:36.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.334+0000 7f6659891640 1 -- 172.21.15.125:0/45794540 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6654154070 con 0x7f66400664d0 2023-10-30T13:25:36.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.358+0000 7f663bfff640 1 -- 172.21.15.125:0/45794540 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293478 (secure 0 0 0) 0x7f6654154070 con 0x7f66400664d0 2023-10-30T13:25:36.364 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:25:36.365 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T13:25:36.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.362+0000 7f6659891640 1 -- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66400664d0 msgr2=0x7f6640068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.362+0000 7f6659891640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66400664d0 0x7f6640068990 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f6644007a70 tx=0x7f6644008040 comp rx=0 tx=0).stop 2023-10-30T13:25:36.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.362+0000 7f6659891640 1 -- 172.21.15.125:0/45794540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66540c6f60 msgr2=0x7f665414ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.362+0000 7f6659891640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66540c6f60 0x7f665414ed70 secure :-1 s=READY pgs=2154 cs=0 l=1 rev1=1 crypto rx=0x7f664800c500 tx=0x7f664800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.362+0000 7f6659891640 1 -- 172.21.15.125:0/45794540 shutdown_connections 2023-10-30T13:25:36.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.362+0000 7f6659891640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f66400664d0 0x7f6640068990 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.362+0000 7f6659891640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f665413f4a0 0x7f6654131620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.362+0000 7f6659891640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f66540c6f60 0x7f665414ed70 unknown :-1 s=CLOSED pgs=2154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.362+0000 7f6659891640 1 --2- 172.21.15.125:0/45794540 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f66540c6b80 0x7f66541322e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.362+0000 7f6659891640 1 -- 172.21.15.125:0/45794540 >> 172.21.15.125:0/45794540 conn(0x7f66540c7910 msgr2=0x7f66540c11c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:36.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.362+0000 7f6659891640 1 -- 172.21.15.125:0/45794540 shutdown_connections 2023-10-30T13:25:36.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.362+0000 7f6659891640 1 -- 172.21.15.125:0/45794540 wait complete. 2023-10-30T13:25:36.385 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":1790,"stamp":"2023-10-30T13:25:36.270456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2068931683,"num_objects":40960,"num_object_clones":18317,"num_object_copies":81920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40960,"num_whiteouts":14249,"num_read":30917,"num_read_kb":10818110,"num_write":128192,"num_write_kb":3672593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134390,"ondisk_log_size":134390,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5379072,"kb_used_data":4140352,"kb_used_omap":12,"kb_used_meta":1238643,"kb_avail":749595648,"statfs":{"total":773094113280,"available":767585943552,"internally_reserved":0,"allocated":4239720448,"data_stored":4144118689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1268371001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1465631,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31886,"num_write":14,"num_write_kb":1441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004753"},"pg_stats":[{"pgid":"3.3f","version":"77'1475","reported_seq":3523,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.196345+0000","last_change":"2023-10-30T12:37:07.182969+0000","last_active":"2023-10-30T13:25:16.196345+0000","last_peered":"2023-10-30T13:25:16.196345+0000","last_clean":"2023-10-30T13:25:16.196345+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T13:25:16.196345+0000","last_undegraded":"2023-10-30T13:25:16.196345+0000","last_fullsized":"2023-10-30T13:25:16.196345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1475,"log_dups_size":0,"ondisk_log_size":1475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:33.616938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086120880000000004,"stat_sum":{"num_bytes":27261067,"num_objects":552,"num_object_clones":249,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":197,"num_read":145,"num_read_kb":29501,"num_write":1298,"num_write_kb":29008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"77'1476","reported_seq":3487,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:01.374954+0000","last_change":"2023-10-30T12:37:07.190309+0000","last_active":"2023-10-30T13:25:01.374954+0000","last_peered":"2023-10-30T13:25:01.374954+0000","last_clean":"2023-10-30T13:25:01.374954+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T13:25:01.374954+0000","last_undegraded":"2023-10-30T13:25:01.374954+0000","last_fullsized":"2023-10-30T13:25:01.374954+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:26:31.884771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015306986,"stat_sum":{"num_bytes":13736175,"num_objects":574,"num_object_clones":269,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":223,"num_read":130,"num_read_kb":17219,"num_write":1260,"num_write_kb":15409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"77'1563","reported_seq":3574,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288448+0000","last_change":"2023-10-30T12:37:07.187803+0000","last_active":"2023-10-30T13:24:16.288448+0000","last_peered":"2023-10-30T13:24:16.288448+0000","last_clean":"2023-10-30T13:24:16.288448+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T13:24:16.288448+0000","last_undegraded":"2023-10-30T13:24:16.288448+0000","last_fullsized":"2023-10-30T13:24:16.288448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1563,"log_dups_size":0,"ondisk_log_size":1563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:32.502949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013634598,"stat_sum":{"num_bytes":28390436,"num_objects":628,"num_object_clones":291,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":232,"num_read":152,"num_read_kb":47076,"num_write":1356,"num_write_kb":37143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"77'1577","reported_seq":3585,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.277341+0000","last_change":"2023-10-30T12:37:07.177860+0000","last_active":"2023-10-30T13:25:32.277341+0000","last_peered":"2023-10-30T13:25:32.277341+0000","last_clean":"2023-10-30T13:25:32.277341+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T13:25:32.277341+0000","last_undegraded":"2023-10-30T13:25:32.277341+0000","last_fullsized":"2023-10-30T13:25:32.277341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1577,"log_dups_size":0,"ondisk_log_size":1577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:27.076595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056762749999999997,"stat_sum":{"num_bytes":22921274,"num_objects":629,"num_object_clones":290,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":224,"num_read":145,"num_read_kb":39229,"num_write":1388,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"77'1591","reported_seq":3596,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.167292+0000","last_change":"2023-10-30T12:37:07.181803+0000","last_active":"2023-10-30T13:24:56.167292+0000","last_peered":"2023-10-30T13:24:56.167292+0000","last_clean":"2023-10-30T13:24:56.167292+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T13:24:56.167292+0000","last_undegraded":"2023-10-30T13:24:56.167292+0000","last_fullsized":"2023-10-30T13:24:56.167292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1591,"log_dups_size":0,"ondisk_log_size":1591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:23:58.362682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081298299999999993,"stat_sum":{"num_bytes":26798060,"num_objects":623,"num_object_clones":286,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":214,"num_read":144,"num_read_kb":26713,"num_write":1434,"num_write_kb":32534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"85'1466","reported_seq":3496,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.286314+0000","last_change":"2023-10-30T12:37:07.176046+0000","last_active":"2023-10-30T13:24:16.286314+0000","last_peered":"2023-10-30T13:24:16.286314+0000","last_clean":"2023-10-30T13:24:16.286314+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T13:24:16.286314+0000","last_undegraded":"2023-10-30T13:24:16.286314+0000","last_fullsized":"2023-10-30T13:24:16.286314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:30:51.393008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048510920000000004,"stat_sum":{"num_bytes":13539463,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":205,"num_read":144,"num_read_kb":15818,"num_write":1313,"num_write_kb":14957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"84'1562","reported_seq":3551,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-30T13:23:51.391894+0000","last_change":"2023-10-30T12:37:07.173349+0000","last_active":"2023-10-30T13:23:51.391894+0000","last_peered":"2023-10-30T13:23:51.391894+0000","last_clean":"2023-10-30T13:23:51.391894+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T13:23:51.391894+0000","last_undegraded":"2023-10-30T13:23:51.391894+0000","last_fullsized":"2023-10-30T13:23:51.391894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1562,"log_dups_size":0,"ondisk_log_size":1562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:38:16.606065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082671399999999999,"stat_sum":{"num_bytes":7202428,"num_objects":606,"num_object_clones":284,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":219,"num_read":126,"num_read_kb":1677,"num_write":1361,"num_write_kb":10203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"82'1518","reported_seq":3541,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288347+0000","last_change":"2023-10-30T12:37:07.188855+0000","last_active":"2023-10-30T13:24:16.288347+0000","last_peered":"2023-10-30T13:24:16.288347+0000","last_clean":"2023-10-30T13:24:16.288347+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T13:24:16.288347+0000","last_undegraded":"2023-10-30T13:24:16.288347+0000","last_fullsized":"2023-10-30T13:24:16.288347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:54.679005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014437559000000001,"stat_sum":{"num_bytes":23150826,"num_objects":587,"num_object_clones":269,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":199,"num_read":147,"num_read_kb":38801,"num_write":1362,"num_write_kb":24887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"77'1549","reported_seq":3579,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.247414+0000","last_change":"2023-10-30T12:37:07.102887+0000","last_active":"2023-10-30T13:25:32.247414+0000","last_peered":"2023-10-30T13:25:32.247414+0000","last_clean":"2023-10-30T13:25:32.247414+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T13:25:32.247414+0000","last_undegraded":"2023-10-30T13:25:32.247414+0000","last_fullsized":"2023-10-30T13:25:32.247414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1549,"log_dups_size":0,"ondisk_log_size":1549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:09.601324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014185980000000001,"stat_sum":{"num_bytes":36575095,"num_objects":618,"num_object_clones":282,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":216,"num_read":165,"num_read_kb":36418,"num_write":1354,"num_write_kb":45869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"77'1459","reported_seq":3491,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.224692+0000","last_change":"2023-10-30T12:37:07.183547+0000","last_active":"2023-10-30T13:25:32.224692+0000","last_peered":"2023-10-30T13:25:32.224692+0000","last_clean":"2023-10-30T13:25:32.224692+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T13:25:32.224692+0000","last_undegraded":"2023-10-30T13:25:32.224692+0000","last_fullsized":"2023-10-30T13:25:32.224692+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:43.076339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091404980000000004,"stat_sum":{"num_bytes":11176343,"num_objects":586,"num_object_clones":270,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":213,"num_read":154,"num_read_kb":10770,"num_write":1260,"num_write_kb":13564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"82'1511","reported_seq":3517,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.187191+0000","last_change":"2023-10-30T12:37:07.181463+0000","last_active":"2023-10-30T13:25:16.187191+0000","last_peered":"2023-10-30T13:25:16.187191+0000","last_clean":"2023-10-30T13:25:16.187191+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T13:25:16.187191+0000","last_undegraded":"2023-10-30T13:25:16.187191+0000","last_fullsized":"2023-10-30T13:25:16.187191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1511,"log_dups_size":0,"ondisk_log_size":1511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:06.796994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075712790000000002,"stat_sum":{"num_bytes":23568118,"num_objects":625,"num_object_clones":283,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":230,"num_read":141,"num_read_kb":34199,"num_write":1299,"num_write_kb":24762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"84'1725","reported_seq":3645,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.138485+0000","last_change":"2023-10-30T12:37:07.181114+0000","last_active":"2023-10-30T13:25:16.138485+0000","last_peered":"2023-10-30T13:25:16.138485+0000","last_clean":"2023-10-30T13:25:16.138485+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T13:25:16.138485+0000","last_undegraded":"2023-10-30T13:25:16.138485+0000","last_fullsized":"2023-10-30T13:25:16.138485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1725,"log_dups_size":0,"ondisk_log_size":1725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:57:46.353792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007366493,"stat_sum":{"num_bytes":24142540,"num_objects":721,"num_object_clones":385,"num_object_copies":1442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":231,"num_read":157,"num_read_kb":24833,"num_write":1452,"num_write_kb":33857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"85'1503","reported_seq":3540,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.201400+0000","last_change":"2023-10-30T12:37:07.114858+0000","last_active":"2023-10-30T13:25:16.201400+0000","last_peered":"2023-10-30T13:25:16.201400+0000","last_clean":"2023-10-30T13:25:16.201400+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T13:25:16.201400+0000","last_undegraded":"2023-10-30T13:25:16.201400+0000","last_fullsized":"2023-10-30T13:25:16.201400+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1503,"log_dups_size":0,"ondisk_log_size":1503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:18:13.789863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006759035,"stat_sum":{"num_bytes":14254037,"num_objects":596,"num_object_clones":276,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":205,"num_read":165,"num_read_kb":21722,"num_write":1326,"num_write_kb":24306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"77'1568","reported_seq":3567,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.278066+0000","last_change":"2023-10-30T12:37:07.112609+0000","last_active":"2023-10-30T13:24:57.278066+0000","last_peered":"2023-10-30T13:24:57.278066+0000","last_clean":"2023-10-30T13:24:57.278066+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T13:24:57.278066+0000","last_undegraded":"2023-10-30T13:24:57.278066+0000","last_fullsized":"2023-10-30T13:24:57.278066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1568,"log_dups_size":0,"ondisk_log_size":1568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:43.640916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057386779999999997,"stat_sum":{"num_bytes":10370740,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":213,"num_read":131,"num_read_kb":5845,"num_write":1381,"num_write_kb":12595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"77'1624","reported_seq":3604,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:46.371670+0000","last_change":"2023-10-30T12:37:07.187257+0000","last_active":"2023-10-30T13:24:46.371670+0000","last_peered":"2023-10-30T13:24:46.371670+0000","last_clean":"2023-10-30T13:24:46.371670+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T13:24:46.371670+0000","last_undegraded":"2023-10-30T13:24:46.371670+0000","last_fullsized":"2023-10-30T13:24:46.371670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:22.374872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013549705,"stat_sum":{"num_bytes":19248519,"num_objects":637,"num_object_clones":295,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":221,"num_read":128,"num_read_kb":21688,"num_write":1447,"num_write_kb":21442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"82'1574","reported_seq":3576,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.208932+0000","last_change":"2023-10-30T12:37:07.178284+0000","last_active":"2023-10-30T13:25:32.208932+0000","last_peered":"2023-10-30T13:25:32.208932+0000","last_clean":"2023-10-30T13:25:32.208932+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T13:25:32.208932+0000","last_undegraded":"2023-10-30T13:25:32.208932+0000","last_fullsized":"2023-10-30T13:25:32.208932+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1574,"log_dups_size":0,"ondisk_log_size":1574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:39:06.662733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075608309999999996,"stat_sum":{"num_bytes":14269019,"num_objects":619,"num_object_clones":290,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":233,"num_read":143,"num_read_kb":25965,"num_write":1377,"num_write_kb":20442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"85'1605","reported_seq":3590,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.146156+0000","last_change":"2023-10-30T12:37:07.182349+0000","last_active":"2023-10-30T13:25:16.146156+0000","last_peered":"2023-10-30T13:25:16.146156+0000","last_clean":"2023-10-30T13:25:16.146156+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T13:25:16.146156+0000","last_undegraded":"2023-10-30T13:25:16.146156+0000","last_fullsized":"2023-10-30T13:25:16.146156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1605,"log_dups_size":0,"ondisk_log_size":1605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:47:59.298834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0099357109999999998,"stat_sum":{"num_bytes":14793776,"num_objects":642,"num_object_clones":304,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":242,"num_read":140,"num_read_kb":30098,"num_write":1386,"num_write_kb":20714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"82'1444","reported_seq":3467,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.136464+0000","last_change":"2023-10-30T12:37:07.182572+0000","last_active":"2023-10-30T13:25:16.136464+0000","last_peered":"2023-10-30T13:25:16.136464+0000","last_clean":"2023-10-30T13:25:16.136464+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T13:25:16.136464+0000","last_undegraded":"2023-10-30T13:25:16.136464+0000","last_fullsized":"2023-10-30T13:25:16.136464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1444,"log_dups_size":0,"ondisk_log_size":1444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:41.547635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065893760000000001,"stat_sum":{"num_bytes":15508852,"num_objects":562,"num_object_clones":263,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":216,"num_read":137,"num_read_kb":10064,"num_write":1244,"num_write_kb":17198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"82'1576","reported_seq":3678,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:26.207237+0000","last_change":"2023-10-30T12:37:07.182668+0000","last_active":"2023-10-30T13:25:26.207237+0000","last_peered":"2023-10-30T13:25:26.207237+0000","last_clean":"2023-10-30T13:25:26.207237+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T13:25:26.207237+0000","last_undegraded":"2023-10-30T13:25:26.207237+0000","last_fullsized":"2023-10-30T13:25:26.207237+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1576,"log_dups_size":0,"ondisk_log_size":1576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:03.035306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010472766,"stat_sum":{"num_bytes":15695810,"num_objects":614,"num_object_clones":281,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":221,"num_read":233,"num_read_kb":17180,"num_write":1382,"num_write_kb":23481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"77'1569","reported_seq":3573,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.219508+0000","last_change":"2023-10-30T12:37:07.181518+0000","last_active":"2023-10-30T13:25:16.219508+0000","last_peered":"2023-10-30T13:25:16.219508+0000","last_clean":"2023-10-30T13:25:16.219508+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T13:25:16.219508+0000","last_undegraded":"2023-10-30T13:25:16.219508+0000","last_fullsized":"2023-10-30T13:25:16.219508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:12:17.666786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075455599999999998,"stat_sum":{"num_bytes":5586810,"num_objects":614,"num_object_clones":286,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":230,"num_read":140,"num_read_kb":5846,"num_write":1359,"num_write_kb":15885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"85'1523","reported_seq":3641,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.245123+0000","last_change":"2023-10-30T12:37:07.113494+0000","last_active":"2023-10-30T13:25:16.245123+0000","last_peered":"2023-10-30T13:25:16.245123+0000","last_clean":"2023-10-30T13:25:16.245123+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T13:25:16.245123+0000","last_undegraded":"2023-10-30T13:25:16.245123+0000","last_fullsized":"2023-10-30T13:25:16.245123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1523,"log_dups_size":0,"ondisk_log_size":1523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:16.038384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076434509999999999,"stat_sum":{"num_bytes":23023646,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":222,"num_read":249,"num_read_kb":26208,"num_write":1319,"num_write_kb":28338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"84'1671","reported_seq":3654,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.169370+0000","last_change":"2023-10-30T12:37:07.102791+0000","last_active":"2023-10-30T13:24:56.169370+0000","last_peered":"2023-10-30T13:24:56.169370+0000","last_clean":"2023-10-30T13:24:56.169370+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T13:24:56.169370+0000","last_undegraded":"2023-10-30T13:24:56.169370+0000","last_fullsized":"2023-10-30T13:24:56.169370+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1671,"log_dups_size":0,"ondisk_log_size":1671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:38:00.347865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016105379999999999,"stat_sum":{"num_bytes":11166034,"num_objects":653,"num_object_clones":306,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":233,"num_read":139,"num_read_kb":9627,"num_write":1472,"num_write_kb":18195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"77'1520","reported_seq":3539,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288413+0000","last_change":"2023-10-30T12:37:07.188206+0000","last_active":"2023-10-30T13:24:16.288413+0000","last_peered":"2023-10-30T13:24:16.288413+0000","last_clean":"2023-10-30T13:24:16.288413+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T13:24:16.288413+0000","last_undegraded":"2023-10-30T13:24:16.288413+0000","last_fullsized":"2023-10-30T13:24:16.288413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:21.306614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014064241,"stat_sum":{"num_bytes":10716893,"num_objects":582,"num_object_clones":264,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":203,"num_read":135,"num_read_kb":13340,"num_write":1353,"num_write_kb":16108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"85'1551","reported_seq":3554,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:23.080116+0000","last_change":"2023-10-30T12:37:07.179785+0000","last_active":"2023-10-30T13:25:23.080116+0000","last_peered":"2023-10-30T13:25:23.080116+0000","last_clean":"2023-10-30T13:25:23.080116+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T13:25:23.080116+0000","last_undegraded":"2023-10-30T13:25:23.080116+0000","last_fullsized":"2023-10-30T13:25:23.080116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:49:28.717141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088476170000000003,"stat_sum":{"num_bytes":10233195,"num_objects":597,"num_object_clones":275,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":203,"num_read":127,"num_read_kb":1819,"num_write":1383,"num_write_kb":16400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"82'1465","reported_seq":3479,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.190427+0000","last_change":"2023-10-30T12:37:07.188606+0000","last_active":"2023-10-30T13:25:16.190427+0000","last_peered":"2023-10-30T13:25:16.190427+0000","last_clean":"2023-10-30T13:25:16.190427+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T13:25:16.190427+0000","last_undegraded":"2023-10-30T13:25:16.190427+0000","last_fullsized":"2023-10-30T13:25:16.190427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:30.044315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014279337,"stat_sum":{"num_bytes":22085088,"num_objects":582,"num_object_clones":270,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":216,"num_read":135,"num_read_kb":28666,"num_write":1258,"num_write_kb":25992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"77'1609","reported_seq":3602,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.171560+0000","last_change":"2023-10-30T12:37:07.177071+0000","last_active":"2023-10-30T13:25:16.171560+0000","last_peered":"2023-10-30T13:25:16.171560+0000","last_clean":"2023-10-30T13:25:16.171560+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T13:25:16.171560+0000","last_undegraded":"2023-10-30T13:25:16.171560+0000","last_fullsized":"2023-10-30T13:25:16.171560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1609,"log_dups_size":0,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:19.563113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004006122,"stat_sum":{"num_bytes":15408826,"num_objects":598,"num_object_clones":277,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":213,"num_read":119,"num_read_kb":9406,"num_write":1408,"num_write_kb":18485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"82'1549","reported_seq":3552,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.237486+0000","last_change":"2023-10-30T12:37:07.115775+0000","last_active":"2023-10-30T13:25:32.237486+0000","last_peered":"2023-10-30T13:25:32.237486+0000","last_clean":"2023-10-30T13:25:32.237486+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T13:25:32.237486+0000","last_undegraded":"2023-10-30T13:25:32.237486+0000","last_fullsized":"2023-10-30T13:25:32.237486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1549,"log_dups_size":0,"ondisk_log_size":1549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:59.732034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059072409999999997,"stat_sum":{"num_bytes":25755688,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":240,"num_read":143,"num_read_kb":23034,"num_write":1332,"num_write_kb":28963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"85'1485","reported_seq":3466,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:07.280157+0000","last_change":"2023-10-30T12:37:07.113316+0000","last_active":"2023-10-30T13:25:07.280157+0000","last_peered":"2023-10-30T13:25:07.280157+0000","last_clean":"2023-10-30T13:25:07.280157+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T13:25:07.280157+0000","last_undegraded":"2023-10-30T13:25:07.280157+0000","last_fullsized":"2023-10-30T13:25:07.280157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1485,"log_dups_size":0,"ondisk_log_size":1485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:19:54.028619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062401690000000003,"stat_sum":{"num_bytes":11021781,"num_objects":617,"num_object_clones":286,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":226,"num_read":116,"num_read_kb":10954,"num_write":1301,"num_write_kb":16114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"77'1620","reported_seq":3589,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.156447+0000","last_change":"2023-10-30T12:37:07.189557+0000","last_active":"2023-10-30T13:25:16.156447+0000","last_peered":"2023-10-30T13:25:16.156447+0000","last_clean":"2023-10-30T13:25:16.156447+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T13:25:16.156447+0000","last_undegraded":"2023-10-30T13:25:16.156447+0000","last_fullsized":"2023-10-30T13:25:16.156447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:17:47.861374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014933905000000001,"stat_sum":{"num_bytes":11161783,"num_objects":653,"num_object_clones":309,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":240,"num_read":128,"num_read_kb":9667,"num_write":1416,"num_write_kb":13402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"82'1476","reported_seq":3519,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.195574+0000","last_change":"2023-10-30T12:37:07.180055+0000","last_active":"2023-10-30T13:25:16.195574+0000","last_peered":"2023-10-30T13:25:16.195574+0000","last_clean":"2023-10-30T13:25:16.195574+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T13:25:16.195574+0000","last_undegraded":"2023-10-30T13:25:16.195574+0000","last_fullsized":"2023-10-30T13:25:16.195574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:02.149498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0090148399999999997,"stat_sum":{"num_bytes":11081748,"num_objects":549,"num_object_clones":251,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":201,"num_read":142,"num_read_kb":9474,"num_write":1288,"num_write_kb":13652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"84'1713","reported_seq":3703,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.230251+0000","last_change":"2023-10-30T12:37:07.181770+0000","last_active":"2023-10-30T13:25:16.230251+0000","last_peered":"2023-10-30T13:25:16.230251+0000","last_clean":"2023-10-30T13:25:16.230251+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T13:25:16.230251+0000","last_undegraded":"2023-10-30T13:25:16.230251+0000","last_fullsized":"2023-10-30T13:25:16.230251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1713,"log_dups_size":0,"ondisk_log_size":1713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:33.319141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066657749999999997,"stat_sum":{"num_bytes":20125251,"num_objects":693,"num_object_clones":330,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":267,"num_read":168,"num_read_kb":25254,"num_write":1457,"num_write_kb":26871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"82'1548","reported_seq":3554,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.216318+0000","last_change":"2023-10-30T12:37:07.176070+0000","last_active":"2023-10-30T13:25:16.216318+0000","last_peered":"2023-10-30T13:25:16.216318+0000","last_clean":"2023-10-30T13:25:16.216318+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T13:25:16.216318+0000","last_undegraded":"2023-10-30T13:25:16.216318+0000","last_fullsized":"2023-10-30T13:25:16.216318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:28:06.978130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005419129,"stat_sum":{"num_bytes":18031744,"num_objects":604,"num_object_clones":282,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":228,"num_read":136,"num_read_kb":25964,"num_write":1344,"num_write_kb":19625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"85'1548","reported_seq":3561,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.144906+0000","last_change":"2023-10-30T12:37:07.102583+0000","last_active":"2023-10-30T13:24:56.144906+0000","last_peered":"2023-10-30T13:24:56.144906+0000","last_clean":"2023-10-30T13:24:56.144906+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T13:24:56.144906+0000","last_undegraded":"2023-10-30T13:24:56.144906+0000","last_fullsized":"2023-10-30T13:24:56.144906+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:32.507814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013742470000000001,"stat_sum":{"num_bytes":22381695,"num_objects":607,"num_object_clones":275,"num_object_copies":1214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":227,"num_read":141,"num_read_kb":33421,"num_write":1348,"num_write_kb":23422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"85'991","reported_seq":1403,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:53.837590+0000","last_change":"2023-10-30T12:28:44.074842+0000","last_active":"2023-10-30T13:24:53.837590+0000","last_peered":"2023-10-30T13:24:53.837590+0000","last_clean":"2023-10-30T13:24:53.837590+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T13:24:53.837590+0000","last_undegraded":"2023-10-30T13:24:53.837590+0000","last_fullsized":"2023-10-30T13:24:53.837590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":991,"log_dups_size":0,"ondisk_log_size":991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:01.428785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":570,"num_read_kb":319827,"num_write":1218,"num_write_kb":80521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"77'1561","reported_seq":3569,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.084148+0000","last_change":"2023-10-30T12:37:07.114520+0000","last_active":"2023-10-30T13:24:56.084148+0000","last_peered":"2023-10-30T13:24:56.084148+0000","last_clean":"2023-10-30T13:24:56.084148+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T13:24:56.084148+0000","last_undegraded":"2023-10-30T13:24:56.084148+0000","last_fullsized":"2023-10-30T13:24:56.084148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1561,"log_dups_size":0,"ondisk_log_size":1561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:06:30.845566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086018069999999995,"stat_sum":{"num_bytes":19940185,"num_objects":616,"num_object_clones":283,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":213,"num_read":144,"num_read_kb":22076,"num_write":1398,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1795","reported_seq":2375,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:52.266854+0000","last_change":"2023-10-30T12:28:44.060621+0000","last_active":"2023-10-30T13:24:52.266854+0000","last_peered":"2023-10-30T13:24:52.266854+0000","last_clean":"2023-10-30T13:24:52.266854+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T13:24:52.266854+0000","last_undegraded":"2023-10-30T13:24:52.266854+0000","last_fullsized":"2023-10-30T13:24:52.266854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1795,"log_dups_size":0,"ondisk_log_size":1795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:53.741238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17028825,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":761,"num_read_kb":396040,"num_write":1931,"num_write_kb":84639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"85'1569","reported_seq":2387,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:27.320748+0000","last_change":"2023-10-30T12:28:44.060171+0000","last_active":"2023-10-30T13:25:27.320748+0000","last_peered":"2023-10-30T13:25:27.320748+0000","last_clean":"2023-10-30T13:25:27.320748+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T13:25:27.320748+0000","last_undegraded":"2023-10-30T13:25:27.320748+0000","last_fullsized":"2023-10-30T13:25:27.320748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:18.181418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182760,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":1117,"num_read_kb":268002,"num_write":1799,"num_write_kb":63121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"77'1801","reported_seq":3808,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.165978+0000","last_change":"2023-10-30T12:37:07.191924+0000","last_active":"2023-10-30T13:25:16.165978+0000","last_peered":"2023-10-30T13:25:16.165978+0000","last_clean":"2023-10-30T13:25:16.165978+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T13:25:16.165978+0000","last_undegraded":"2023-10-30T13:25:16.165978+0000","last_fullsized":"2023-10-30T13:25:16.165978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1801,"log_dups_size":0,"ondisk_log_size":1801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:27.781535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016143469000000001,"stat_sum":{"num_bytes":14540052,"num_objects":709,"num_object_clones":329,"num_object_copies":1418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":254,"num_read":190,"num_read_kb":6718,"num_write":1583,"num_write_kb":26382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"85'930","reported_seq":1332,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:52.371111+0000","last_change":"2023-10-30T12:28:44.058798+0000","last_active":"2023-10-30T13:24:52.371111+0000","last_peered":"2023-10-30T13:24:52.371111+0000","last_clean":"2023-10-30T13:24:52.371111+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T13:24:52.371111+0000","last_undegraded":"2023-10-30T13:24:52.371111+0000","last_fullsized":"2023-10-30T13:24:52.371111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":930,"log_dups_size":0,"ondisk_log_size":930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:08.643080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":542,"num_read_kb":359029,"num_write":1093,"num_write_kb":88246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"77'1524","reported_seq":3543,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.097691+0000","last_change":"2023-10-30T12:37:07.114493+0000","last_active":"2023-10-30T13:24:56.097691+0000","last_peered":"2023-10-30T13:24:56.097691+0000","last_clean":"2023-10-30T13:24:56.097691+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T13:24:56.097691+0000","last_undegraded":"2023-10-30T13:24:56.097691+0000","last_fullsized":"2023-10-30T13:24:56.097691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:55:36.189245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083101389999999994,"stat_sum":{"num_bytes":18904879,"num_objects":589,"num_object_clones":276,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":211,"num_read":147,"num_read_kb":20021,"num_write":1340,"num_write_kb":25244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"85'736","reported_seq":1136,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:58.054623+0000","last_change":"2023-10-30T12:28:44.063439+0000","last_active":"2023-10-30T13:24:58.054623+0000","last_peered":"2023-10-30T13:24:58.054623+0000","last_clean":"2023-10-30T13:24:58.054623+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T13:24:58.054623+0000","last_undegraded":"2023-10-30T13:24:58.054623+0000","last_fullsized":"2023-10-30T13:24:58.054623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":736,"log_dups_size":0,"ondisk_log_size":736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:08.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":616,"num_read_kb":251423,"num_write":889,"num_write_kb":55053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"77'1518","reported_seq":3533,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.444813+0000","last_change":"2023-10-30T12:37:07.177520+0000","last_active":"2023-10-30T13:25:10.444813+0000","last_peered":"2023-10-30T13:25:10.444813+0000","last_clean":"2023-10-30T13:25:10.444813+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T13:25:10.444813+0000","last_undegraded":"2023-10-30T13:25:10.444813+0000","last_fullsized":"2023-10-30T13:25:10.444813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:46:22.808814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043045619999999996,"stat_sum":{"num_bytes":10134301,"num_objects":580,"num_object_clones":271,"num_object_copies":1160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":580,"num_whiteouts":219,"num_read":138,"num_read_kb":8986,"num_write":1318,"num_write_kb":11276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"85'1247","reported_seq":1685,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:24.345143+0000","last_change":"2023-10-30T12:28:44.052641+0000","last_active":"2023-10-30T13:24:24.345143+0000","last_peered":"2023-10-30T13:24:24.345143+0000","last_clean":"2023-10-30T13:24:24.345143+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T13:24:24.345143+0000","last_undegraded":"2023-10-30T13:24:24.345143+0000","last_fullsized":"2023-10-30T13:24:24.345143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1247,"log_dups_size":0,"ondisk_log_size":1247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":659,"num_read_kb":339384,"num_write":1492,"num_write_kb":80693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"82'1603","reported_seq":3592,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.115007+0000","last_change":"2023-10-30T12:37:07.191144+0000","last_active":"2023-10-30T13:24:56.115007+0000","last_peered":"2023-10-30T13:24:56.115007+0000","last_clean":"2023-10-30T13:24:56.115007+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T13:24:56.115007+0000","last_undegraded":"2023-10-30T13:24:56.115007+0000","last_fullsized":"2023-10-30T13:24:56.115007+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1603,"log_dups_size":0,"ondisk_log_size":1603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:46.268962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015702779,"stat_sum":{"num_bytes":15773172,"num_objects":620,"num_object_clones":292,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":204,"num_read":135,"num_read_kb":17351,"num_write":1431,"num_write_kb":17077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"85'1224","reported_seq":1717,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:35.202116+0000","last_change":"2023-10-30T12:28:44.075783+0000","last_active":"2023-10-30T13:25:35.202116+0000","last_peered":"2023-10-30T13:25:35.202116+0000","last_clean":"2023-10-30T13:25:35.202116+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T13:25:35.202116+0000","last_undegraded":"2023-10-30T13:25:35.202116+0000","last_fullsized":"2023-10-30T13:25:35.202116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1224,"log_dups_size":0,"ondisk_log_size":1224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:01.094929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19774519,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":862,"num_read_kb":221068,"num_write":1454,"num_write_kb":51536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"82'1515","reported_seq":3524,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.077418+0000","last_change":"2023-10-30T12:37:07.178872+0000","last_active":"2023-10-30T13:25:03.077418+0000","last_peered":"2023-10-30T13:25:03.077418+0000","last_clean":"2023-10-30T13:25:03.077418+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T13:25:03.077418+0000","last_undegraded":"2023-10-30T13:25:03.077418+0000","last_fullsized":"2023-10-30T13:25:03.077418+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:03.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080610860000000003,"stat_sum":{"num_bytes":22077043,"num_objects":612,"num_object_clones":281,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":230,"num_read":141,"num_read_kb":18856,"num_write":1294,"num_write_kb":28337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"85'1331","reported_seq":1877,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:12.181476+0000","last_change":"2023-10-30T12:28:44.063367+0000","last_active":"2023-10-30T13:25:12.181476+0000","last_peered":"2023-10-30T13:25:12.181476+0000","last_clean":"2023-10-30T13:25:12.181476+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T13:25:12.181476+0000","last_undegraded":"2023-10-30T13:25:12.181476+0000","last_fullsized":"2023-10-30T13:25:12.181476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1331,"log_dups_size":0,"ondisk_log_size":1331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:45.585121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":865,"num_read_kb":420163,"num_write":1566,"num_write_kb":89397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"82'1535","reported_seq":3533,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.355487+0000","last_change":"2023-10-30T12:37:07.182869+0000","last_active":"2023-10-30T13:24:48.355487+0000","last_peered":"2023-10-30T13:24:48.355487+0000","last_clean":"2023-10-30T13:24:48.355487+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T13:24:48.355487+0000","last_undegraded":"2023-10-30T13:24:48.355487+0000","last_fullsized":"2023-10-30T13:24:48.355487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1535,"log_dups_size":0,"ondisk_log_size":1535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:35.734555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.008640567,"stat_sum":{"num_bytes":23022369,"num_objects":606,"num_object_clones":279,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":220,"num_read":129,"num_read_kb":18941,"num_write":1340,"num_write_kb":24878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"85'1098","reported_seq":1558,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.290468+0000","last_change":"2023-10-30T12:28:44.065545+0000","last_active":"2023-10-30T13:24:47.290468+0000","last_peered":"2023-10-30T13:24:47.290468+0000","last_clean":"2023-10-30T13:24:47.290468+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T13:24:47.290468+0000","last_undegraded":"2023-10-30T13:24:47.290468+0000","last_fullsized":"2023-10-30T13:24:47.290468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1098,"log_dups_size":0,"ondisk_log_size":1098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:09.900189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554466,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":701,"num_read_kb":318483,"num_write":1303,"num_write_kb":68223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"84'1584","reported_seq":3584,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.315736+0000","last_change":"2023-10-30T12:37:07.115595+0000","last_active":"2023-10-30T13:24:48.315736+0000","last_peered":"2023-10-30T13:24:48.315736+0000","last_clean":"2023-10-30T13:24:48.315736+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T13:24:48.315736+0000","last_undegraded":"2023-10-30T13:24:48.315736+0000","last_fullsized":"2023-10-30T13:24:48.315736+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:20:44.527701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073009930000000004,"stat_sum":{"num_bytes":10084838,"num_objects":613,"num_object_clones":288,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":217,"num_read":139,"num_read_kb":5686,"num_write":1383,"num_write_kb":20147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'676","reported_seq":1068,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.649328+0000","last_change":"2023-10-30T12:28:44.061922+0000","last_active":"2023-10-30T13:24:57.649328+0000","last_peered":"2023-10-30T13:24:57.649328+0000","last_clean":"2023-10-30T13:24:57.649328+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T13:24:57.649328+0000","last_undegraded":"2023-10-30T13:24:57.649328+0000","last_fullsized":"2023-10-30T13:24:57.649328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:34.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":600,"num_read_kb":208389,"num_write":899,"num_write_kb":51587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"82'1642","reported_seq":3652,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.151917+0000","last_change":"2023-10-30T12:37:07.180453+0000","last_active":"2023-10-30T13:25:16.151917+0000","last_peered":"2023-10-30T13:25:16.151917+0000","last_clean":"2023-10-30T13:25:16.151917+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T13:25:16.151917+0000","last_undegraded":"2023-10-30T13:25:16.151917+0000","last_fullsized":"2023-10-30T13:25:16.151917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:20.518544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091258150000000007,"stat_sum":{"num_bytes":19392609,"num_objects":655,"num_object_clones":299,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":233,"num_read":161,"num_read_kb":7475,"num_write":1440,"num_write_kb":20643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"85'1371","reported_seq":1755,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.074769+0000","last_change":"2023-10-30T12:28:44.059784+0000","last_active":"2023-10-30T13:24:48.074769+0000","last_peered":"2023-10-30T13:24:48.074769+0000","last_clean":"2023-10-30T13:24:48.074769+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T13:24:48.074769+0000","last_undegraded":"2023-10-30T13:24:48.074769+0000","last_fullsized":"2023-10-30T13:24:48.074769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1371,"log_dups_size":0,"ondisk_log_size":1371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:11.468221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":539,"num_read_kb":266844,"num_write":1512,"num_write_kb":63647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"77'1481","reported_seq":3489,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.310426+0000","last_change":"2023-10-30T12:37:07.102525+0000","last_active":"2023-10-30T13:24:48.310426+0000","last_peered":"2023-10-30T13:24:48.310426+0000","last_clean":"2023-10-30T13:24:48.310426+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T13:24:48.310426+0000","last_undegraded":"2023-10-30T13:24:48.310426+0000","last_fullsized":"2023-10-30T13:24:48.310426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:22.814663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001646376,"stat_sum":{"num_bytes":13778361,"num_objects":590,"num_object_clones":278,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":227,"num_read":137,"num_read_kb":5723,"num_write":1258,"num_write_kb":14976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"85'970","reported_seq":1455,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.407319+0000","last_change":"2023-10-30T12:28:44.065885+0000","last_active":"2023-10-30T13:24:47.407319+0000","last_peered":"2023-10-30T13:24:47.407319+0000","last_clean":"2023-10-30T13:24:47.407319+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T13:24:47.407319+0000","last_undegraded":"2023-10-30T13:24:47.407319+0000","last_fullsized":"2023-10-30T13:24:47.407319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:26.313595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554544,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":863,"num_read_kb":272228,"num_write":1253,"num_write_kb":56723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"82'1616","reported_seq":3636,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.437909+0000","last_change":"2023-10-30T12:37:07.181542+0000","last_active":"2023-10-30T13:25:10.437909+0000","last_peered":"2023-10-30T13:25:10.437909+0000","last_clean":"2023-10-30T13:25:10.437909+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T13:25:10.437909+0000","last_undegraded":"2023-10-30T13:25:10.437909+0000","last_fullsized":"2023-10-30T13:25:10.437909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1616,"log_dups_size":0,"ondisk_log_size":1616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:16:47.953145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098326820000000006,"stat_sum":{"num_bytes":10901701,"num_objects":622,"num_object_clones":292,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":232,"num_read":163,"num_read_kb":14238,"num_write":1406,"num_write_kb":21652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"85'655","reported_seq":1014,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:04.285953+0000","last_change":"2023-10-30T12:28:44.080011+0000","last_active":"2023-10-30T13:25:04.285953+0000","last_peered":"2023-10-30T13:25:04.285953+0000","last_clean":"2023-10-30T13:25:04.285953+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T13:25:04.285953+0000","last_undegraded":"2023-10-30T13:25:04.285953+0000","last_fullsized":"2023-10-30T13:25:04.285953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":443,"num_read_kb":261670,"num_write":858,"num_write_kb":58986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"77'1573","reported_seq":3538,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.065502+0000","last_change":"2023-10-30T12:37:07.177898+0000","last_active":"2023-10-30T13:25:03.065502+0000","last_peered":"2023-10-30T13:25:03.065502+0000","last_clean":"2023-10-30T13:25:03.065502+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T13:25:03.065502+0000","last_undegraded":"2023-10-30T13:25:03.065502+0000","last_fullsized":"2023-10-30T13:25:03.065502+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:52.251046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004602437,"stat_sum":{"num_bytes":20428330,"num_objects":647,"num_object_clones":304,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":239,"num_read":119,"num_read_kb":13097,"num_write":1360,"num_write_kb":26395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"85'717","reported_seq":1166,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.413162+0000","last_change":"2023-10-30T12:28:44.076352+0000","last_active":"2023-10-30T13:24:48.413162+0000","last_peered":"2023-10-30T13:24:48.413162+0000","last_clean":"2023-10-30T13:24:48.413162+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T13:24:48.413162+0000","last_undegraded":"2023-10-30T13:24:48.413162+0000","last_fullsized":"2023-10-30T13:24:48.413162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":635,"num_read_kb":434171,"num_write":906,"num_write_kb":88121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"77'1575","reported_seq":3560,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.221074+0000","last_change":"2023-10-30T12:37:07.181502+0000","last_active":"2023-10-30T13:25:32.221074+0000","last_peered":"2023-10-30T13:25:32.221074+0000","last_clean":"2023-10-30T13:25:32.221074+0000","last_became_active":"2023-10-30T12:29:27.340504+0000","last_became_peered":"2023-10-30T12:29:27.340504+0000","last_unstale":"2023-10-30T13:25:32.221074+0000","last_undegraded":"2023-10-30T13:25:32.221074+0000","last_fullsized":"2023-10-30T13:25:32.221074+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1575,"log_dups_size":0,"ondisk_log_size":1575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:48:15.263466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00653923,"stat_sum":{"num_bytes":23737065,"num_objects":641,"num_object_clones":299,"num_object_copies":1282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":641,"num_whiteouts":238,"num_read":138,"num_read_kb":28255,"num_write":1351,"num_write_kb":33550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'716","reported_seq":1095,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.276257+0000","last_change":"2023-10-30T12:28:44.075684+0000","last_active":"2023-10-30T13:24:47.276257+0000","last_peered":"2023-10-30T13:24:47.276257+0000","last_clean":"2023-10-30T13:24:47.276257+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T13:24:47.276257+0000","last_undegraded":"2023-10-30T13:24:47.276257+0000","last_fullsized":"2023-10-30T13:24:47.276257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:07.375795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364369,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":493,"num_read_kb":320778,"num_write":891,"num_write_kb":71265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"77'1599","reported_seq":3612,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.080865+0000","last_change":"2023-10-30T12:37:07.181841+0000","last_active":"2023-10-30T13:25:03.080865+0000","last_peered":"2023-10-30T13:25:03.080865+0000","last_clean":"2023-10-30T13:25:03.080865+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T13:25:03.080865+0000","last_undegraded":"2023-10-30T13:25:03.080865+0000","last_fullsized":"2023-10-30T13:25:03.080865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1599,"log_dups_size":0,"ondisk_log_size":1599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:35:15.227222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076673150000000001,"stat_sum":{"num_bytes":18256447,"num_objects":611,"num_object_clones":286,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":229,"num_read":149,"num_read_kb":24381,"num_write":1383,"num_write_kb":20195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"85'607","reported_seq":959,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.278226+0000","last_change":"2023-10-30T12:28:44.060686+0000","last_active":"2023-10-30T13:24:47.278226+0000","last_peered":"2023-10-30T13:24:47.278226+0000","last_clean":"2023-10-30T13:24:47.278226+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T13:24:47.278226+0000","last_undegraded":"2023-10-30T13:24:47.278226+0000","last_fullsized":"2023-10-30T13:24:47.278226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:23.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360710,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":462,"num_read_kb":342669,"num_write":689,"num_write_kb":66904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"82'1520","reported_seq":3502,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.077446+0000","last_change":"2023-10-30T12:37:07.182282+0000","last_active":"2023-10-30T13:25:03.077446+0000","last_peered":"2023-10-30T13:25:03.077446+0000","last_clean":"2023-10-30T13:25:03.077446+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T13:25:03.077446+0000","last_undegraded":"2023-10-30T13:25:03.077446+0000","last_fullsized":"2023-10-30T13:25:03.077446+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:14.913475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010286791999999999,"stat_sum":{"num_bytes":26944901,"num_objects":612,"num_object_clones":287,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":236,"num_read":119,"num_read_kb":34683,"num_write":1299,"num_write_kb":33129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":256,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288773+0000","last_change":"2023-10-30T12:28:33.433824+0000","last_active":"2023-10-30T13:24:16.288773+0000","last_peered":"2023-10-30T13:24:16.288773+0000","last_clean":"2023-10-30T13:24:16.288773+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T13:24:16.288773+0000","last_undegraded":"2023-10-30T13:24:16.288773+0000","last_fullsized":"2023-10-30T13:24:16.288773+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_clean_scrub_stamp":"2023-10-30T12:28:31.408255+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:17.624179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'1318","reported_seq":1818,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.269121+0000","last_change":"2023-10-30T12:28:44.059051+0000","last_active":"2023-10-30T13:24:47.269121+0000","last_peered":"2023-10-30T13:24:47.269121+0000","last_clean":"2023-10-30T13:24:47.269121+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T13:24:47.269121+0000","last_undegraded":"2023-10-30T13:24:47.269121+0000","last_fullsized":"2023-10-30T13:24:47.269121+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1318,"log_dups_size":0,"ondisk_log_size":1318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:54.810636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165890,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":877,"num_read_kb":231219,"num_write":1489,"num_write_kb":63137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"85'1574","reported_seq":3582,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.122820+0000","last_change":"2023-10-30T12:37:07.116603+0000","last_active":"2023-10-30T13:25:02.122820+0000","last_peered":"2023-10-30T13:25:02.122820+0000","last_clean":"2023-10-30T13:25:02.122820+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T13:25:02.122820+0000","last_undegraded":"2023-10-30T13:25:02.122820+0000","last_fullsized":"2023-10-30T13:25:02.122820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1574,"log_dups_size":0,"ondisk_log_size":1574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:01:30.768741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068470270000000003,"stat_sum":{"num_bytes":8097312,"num_objects":626,"num_object_clones":290,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":225,"num_read":146,"num_read_kb":2079,"num_write":1425,"num_write_kb":13192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"85'2192","reported_seq":2563,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.650111+0000","last_change":"2023-10-30T12:28:44.080378+0000","last_active":"2023-10-30T13:24:57.650111+0000","last_peered":"2023-10-30T13:24:57.650111+0000","last_clean":"2023-10-30T13:24:57.650111+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T13:24:57.650111+0000","last_undegraded":"2023-10-30T13:24:57.650111+0000","last_fullsized":"2023-10-30T13:24:57.650111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2192,"log_dups_size":0,"ondisk_log_size":2192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:22.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":516,"num_read_kb":283293,"num_write":2369,"num_write_kb":72030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"77'1465","reported_seq":3487,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:15.952966+0000","last_change":"2023-10-30T12:37:07.112121+0000","last_active":"2023-10-30T13:24:15.952966+0000","last_peered":"2023-10-30T13:24:15.952966+0000","last_clean":"2023-10-30T13:24:15.952966+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T13:24:15.952966+0000","last_undegraded":"2023-10-30T13:24:15.952966+0000","last_fullsized":"2023-10-30T13:24:15.952966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:12:48.778193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057755250000000001,"stat_sum":{"num_bytes":14269585,"num_objects":570,"num_object_clones":262,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":207,"num_read":132,"num_read_kb":18022,"num_write":1264,"num_write_kb":17917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'1838","reported_seq":3123,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:09.242564+0000","last_change":"2023-10-30T12:28:44.082480+0000","last_active":"2023-10-30T13:25:09.242564+0000","last_peered":"2023-10-30T13:25:09.242564+0000","last_clean":"2023-10-30T13:25:09.242564+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T13:25:09.242564+0000","last_undegraded":"2023-10-30T13:25:09.242564+0000","last_fullsized":"2023-10-30T13:25:09.242564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1838,"log_dups_size":0,"ondisk_log_size":1838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:20.126782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49235352,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":1514,"num_read_kb":360508,"num_write":2035,"num_write_kb":92558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"77'1546","reported_seq":3555,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.373540+0000","last_change":"2023-10-30T12:37:07.190318+0000","last_active":"2023-10-30T13:24:56.373540+0000","last_peered":"2023-10-30T13:24:56.373540+0000","last_clean":"2023-10-30T13:24:56.373540+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T13:24:56.373540+0000","last_undegraded":"2023-10-30T13:24:56.373540+0000","last_fullsized":"2023-10-30T13:24:56.373540+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:19.397149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015331312,"stat_sum":{"num_bytes":28178512,"num_objects":629,"num_object_clones":288,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":231,"num_read":146,"num_read_kb":37712,"num_write":1347,"num_write_kb":29520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'1196","reported_seq":1605,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.648443+0000","last_change":"2023-10-30T12:28:44.059806+0000","last_active":"2023-10-30T13:24:57.648443+0000","last_peered":"2023-10-30T13:24:57.648443+0000","last_clean":"2023-10-30T13:24:57.648443+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T13:24:57.648443+0000","last_undegraded":"2023-10-30T13:24:57.648443+0000","last_fullsized":"2023-10-30T13:24:57.648443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1196,"log_dups_size":0,"ondisk_log_size":1196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:50.939946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":573,"num_read_kb":334248,"num_write":1338,"num_write_kb":75895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"85'1581","reported_seq":3581,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.228379+0000","last_change":"2023-10-30T12:37:07.178785+0000","last_active":"2023-10-30T13:25:16.228379+0000","last_peered":"2023-10-30T13:25:16.228379+0000","last_clean":"2023-10-30T13:25:16.228379+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T13:25:16.228379+0000","last_undegraded":"2023-10-30T13:25:16.228379+0000","last_fullsized":"2023-10-30T13:25:16.228379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1581,"log_dups_size":0,"ondisk_log_size":1581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:27.865544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051193749999999998,"stat_sum":{"num_bytes":23443395,"num_objects":627,"num_object_clones":293,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":236,"num_read":141,"num_read_kb":26943,"num_write":1376,"num_write_kb":26009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"85'1467","reported_seq":1927,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:01.374791+0000","last_change":"2023-10-30T12:28:44.074970+0000","last_active":"2023-10-30T13:25:01.374791+0000","last_peered":"2023-10-30T13:25:01.374791+0000","last_clean":"2023-10-30T13:25:01.374791+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T13:25:01.374791+0000","last_undegraded":"2023-10-30T13:25:01.374791+0000","last_fullsized":"2023-10-30T13:25:01.374791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1467,"log_dups_size":0,"ondisk_log_size":1467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:33.562703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":699,"num_read_kb":404466,"num_write":1620,"num_write_kb":92575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"82'1612","reported_seq":3615,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:18.067135+0000","last_change":"2023-10-30T12:37:07.176969+0000","last_active":"2023-10-30T13:25:18.067135+0000","last_peered":"2023-10-30T13:25:18.067135+0000","last_clean":"2023-10-30T13:25:18.067135+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T13:25:18.067135+0000","last_undegraded":"2023-10-30T13:25:18.067135+0000","last_fullsized":"2023-10-30T13:25:18.067135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1612,"log_dups_size":0,"ondisk_log_size":1612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:33:15.916870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039865179999999997,"stat_sum":{"num_bytes":21812744,"num_objects":637,"num_object_clones":298,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":235,"num_read":149,"num_read_kb":24572,"num_write":1399,"num_write_kb":31699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"85'1013","reported_seq":1422,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.646473+0000","last_change":"2023-10-30T12:28:44.063256+0000","last_active":"2023-10-30T13:24:57.646473+0000","last_peered":"2023-10-30T13:24:57.646473+0000","last_clean":"2023-10-30T13:24:57.646473+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T13:24:57.646473+0000","last_undegraded":"2023-10-30T13:24:57.646473+0000","last_fullsized":"2023-10-30T13:24:57.646473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1013,"log_dups_size":0,"ondisk_log_size":1013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:01:36.144966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":534,"num_read_kb":318720,"num_write":1164,"num_write_kb":75805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"77'1542","reported_seq":3565,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.430277+0000","last_change":"2023-10-30T12:37:07.114718+0000","last_active":"2023-10-30T13:25:10.430277+0000","last_peered":"2023-10-30T13:25:10.430277+0000","last_clean":"2023-10-30T13:25:10.430277+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T13:25:10.430277+0000","last_undegraded":"2023-10-30T13:25:10.430277+0000","last_fullsized":"2023-10-30T13:25:10.430277+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1542,"log_dups_size":0,"ondisk_log_size":1542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:05.944397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070372949999999998,"stat_sum":{"num_bytes":38977850,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":222,"num_read":154,"num_read_kb":43697,"num_write":1351,"num_write_kb":41194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1137","reported_seq":1571,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.180289+0000","last_change":"2023-10-30T12:28:44.055374+0000","last_active":"2023-10-30T13:25:02.180289+0000","last_peered":"2023-10-30T13:25:02.180289+0000","last_clean":"2023-10-30T13:25:02.180289+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T13:25:02.180289+0000","last_undegraded":"2023-10-30T13:25:02.180289+0000","last_fullsized":"2023-10-30T13:25:02.180289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1137,"log_dups_size":0,"ondisk_log_size":1137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":644,"num_read_kb":249397,"num_write":1858,"num_write_kb":63729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"77'1449","reported_seq":3446,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:15.952034+0000","last_change":"2023-10-30T12:37:07.115578+0000","last_active":"2023-10-30T13:24:15.952034+0000","last_peered":"2023-10-30T13:24:15.952034+0000","last_clean":"2023-10-30T13:24:15.952034+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T13:24:15.952034+0000","last_undegraded":"2023-10-30T13:24:15.952034+0000","last_fullsized":"2023-10-30T13:24:15.952034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1449,"log_dups_size":0,"ondisk_log_size":1449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:52:43.980959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074996079999999996,"stat_sum":{"num_bytes":14238343,"num_objects":603,"num_object_clones":280,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":229,"num_read":126,"num_read_kb":9383,"num_write":1239,"num_write_kb":19514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"85'489","reported_seq":838,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:08.078199+0000","last_change":"2023-10-30T12:28:44.058663+0000","last_active":"2023-10-30T13:25:08.078199+0000","last_peered":"2023-10-30T13:25:08.078199+0000","last_clean":"2023-10-30T13:25:08.078199+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T13:25:08.078199+0000","last_undegraded":"2023-10-30T13:25:08.078199+0000","last_fullsized":"2023-10-30T13:25:08.078199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:42:22.279653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21146410,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":514,"num_read_kb":197148,"num_write":650,"num_write_kb":42825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"77'1649","reported_seq":3622,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.197589+0000","last_change":"2023-10-30T12:37:07.115477+0000","last_active":"2023-10-30T13:25:16.197589+0000","last_peered":"2023-10-30T13:25:16.197589+0000","last_clean":"2023-10-30T13:25:16.197589+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T13:25:16.197589+0000","last_undegraded":"2023-10-30T13:25:16.197589+0000","last_fullsized":"2023-10-30T13:25:16.197589+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1649,"log_dups_size":0,"ondisk_log_size":1649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:44.248368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085509929999999998,"stat_sum":{"num_bytes":14746405,"num_objects":647,"num_object_clones":344,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":203,"num_read":166,"num_read_kb":4091,"num_write":1369,"num_write_kb":17013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'870","reported_seq":1184,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.657578+0000","last_change":"2023-10-30T12:28:44.054338+0000","last_active":"2023-10-30T13:25:02.657578+0000","last_peered":"2023-10-30T13:25:02.657578+0000","last_clean":"2023-10-30T13:25:02.657578+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T13:25:02.657578+0000","last_undegraded":"2023-10-30T13:25:02.657578+0000","last_fullsized":"2023-10-30T13:25:02.657578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":870,"log_dups_size":0,"ondisk_log_size":870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:55:35.370652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":472,"num_read_kb":149352,"num_write":1030,"num_write_kb":38491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"77'1485","reported_seq":3500,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.168854+0000","last_change":"2023-10-30T12:37:07.181506+0000","last_active":"2023-10-30T13:25:16.168854+0000","last_peered":"2023-10-30T13:25:16.168854+0000","last_clean":"2023-10-30T13:25:16.168854+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T13:25:16.168854+0000","last_undegraded":"2023-10-30T13:25:16.168854+0000","last_fullsized":"2023-10-30T13:25:16.168854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1485,"log_dups_size":0,"ondisk_log_size":1485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:53:57.876949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096369869999999996,"stat_sum":{"num_bytes":18113290,"num_objects":584,"num_object_clones":270,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":219,"num_read":134,"num_read_kb":20299,"num_write":1266,"num_write_kb":19768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"85'777","reported_seq":1140,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:04.284846+0000","last_change":"2023-10-30T12:28:44.064606+0000","last_active":"2023-10-30T13:25:04.284846+0000","last_peered":"2023-10-30T13:25:04.284846+0000","last_clean":"2023-10-30T13:25:04.284846+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T13:25:04.284846+0000","last_undegraded":"2023-10-30T13:25:04.284846+0000","last_fullsized":"2023-10-30T13:25:04.284846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":777,"log_dups_size":0,"ondisk_log_size":777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777306,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":495,"num_read_kb":170685,"num_write":965,"num_write_kb":43434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"82'1567","reported_seq":3571,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.146212+0000","last_change":"2023-10-30T12:37:07.112752+0000","last_active":"2023-10-30T13:24:56.146212+0000","last_peered":"2023-10-30T13:24:56.146212+0000","last_clean":"2023-10-30T13:24:56.146212+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T13:24:56.146212+0000","last_undegraded":"2023-10-30T13:24:56.146212+0000","last_fullsized":"2023-10-30T13:24:56.146212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1567,"log_dups_size":0,"ondisk_log_size":1567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:41:59.413302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007081922,"stat_sum":{"num_bytes":14751960,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":228,"num_read":136,"num_read_kb":2513,"num_write":1376,"num_write_kb":21368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1125","reported_seq":1613,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:17.281839+0000","last_change":"2023-10-30T12:28:44.072754+0000","last_active":"2023-10-30T13:25:17.281839+0000","last_peered":"2023-10-30T13:25:17.281839+0000","last_clean":"2023-10-30T13:25:17.281839+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T13:25:17.281839+0000","last_undegraded":"2023-10-30T13:25:17.281839+0000","last_fullsized":"2023-10-30T13:25:17.281839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1125,"log_dups_size":0,"ondisk_log_size":1125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:51:55.865069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":763,"num_read_kb":345628,"num_write":1535,"num_write_kb":76701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"77'1460","reported_seq":3474,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.160346+0000","last_change":"2023-10-30T12:37:07.182563+0000","last_active":"2023-10-30T13:24:56.160346+0000","last_peered":"2023-10-30T13:24:56.160346+0000","last_clean":"2023-10-30T13:24:56.160346+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T13:24:56.160346+0000","last_undegraded":"2023-10-30T13:24:56.160346+0000","last_fullsized":"2023-10-30T13:24:56.160346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1460,"log_dups_size":0,"ondisk_log_size":1460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:29:22.948042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060337460000000004,"stat_sum":{"num_bytes":23618154,"num_objects":570,"num_object_clones":263,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":209,"num_read":127,"num_read_kb":29654,"num_write":1265,"num_write_kb":27771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"85'1182","reported_seq":1585,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:58.045831+0000","last_change":"2023-10-30T12:28:44.058801+0000","last_active":"2023-10-30T13:24:58.045831+0000","last_peered":"2023-10-30T13:24:58.045831+0000","last_clean":"2023-10-30T13:24:58.045831+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T13:24:58.045831+0000","last_undegraded":"2023-10-30T13:24:58.045831+0000","last_fullsized":"2023-10-30T13:24:58.045831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1182,"log_dups_size":0,"ondisk_log_size":1182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:11:24.706813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":538,"num_read_kb":321476,"num_write":1330,"num_write_kb":79987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"85'1546","reported_seq":3551,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.277281+0000","last_change":"2023-10-30T12:37:07.180393+0000","last_active":"2023-10-30T13:24:47.277281+0000","last_peered":"2023-10-30T13:24:47.277281+0000","last_clean":"2023-10-30T13:24:47.277281+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T13:24:47.277281+0000","last_undegraded":"2023-10-30T13:24:47.277281+0000","last_fullsized":"2023-10-30T13:24:47.277281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:28:37.347215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091395910000000007,"stat_sum":{"num_bytes":24394000,"num_objects":612,"num_object_clones":279,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":219,"num_read":133,"num_read_kb":28881,"num_write":1364,"num_write_kb":25892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"77'1527","reported_seq":3550,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.236448+0000","last_change":"2023-10-30T12:37:07.182718+0000","last_active":"2023-10-30T13:25:32.236448+0000","last_peered":"2023-10-30T13:25:32.236448+0000","last_clean":"2023-10-30T13:25:32.236448+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T13:25:32.236448+0000","last_undegraded":"2023-10-30T13:25:32.236448+0000","last_fullsized":"2023-10-30T13:25:32.236448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1527,"log_dups_size":0,"ondisk_log_size":1527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:12:33.103610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010120377999999999,"stat_sum":{"num_bytes":19538275,"num_objects":582,"num_object_clones":275,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":200,"num_read":151,"num_read_kb":26132,"num_write":1354,"num_write_kb":27822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"85'880","reported_seq":1327,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.246911+0000","last_change":"2023-10-30T12:28:44.054652+0000","last_active":"2023-10-30T13:24:47.246911+0000","last_peered":"2023-10-30T13:24:47.246911+0000","last_clean":"2023-10-30T13:24:47.246911+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T13:24:47.246911+0000","last_undegraded":"2023-10-30T13:24:47.246911+0000","last_fullsized":"2023-10-30T13:24:47.246911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:05.023374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":739,"num_read_kb":197443,"num_write":1077,"num_write_kb":51357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"82'1642","reported_seq":3637,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.166307+0000","last_change":"2023-10-30T12:37:07.102323+0000","last_active":"2023-10-30T13:24:56.166307+0000","last_peered":"2023-10-30T13:24:56.166307+0000","last_clean":"2023-10-30T13:24:56.166307+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T13:24:56.166307+0000","last_undegraded":"2023-10-30T13:24:56.166307+0000","last_fullsized":"2023-10-30T13:24:56.166307+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:12.249464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011628750000000001,"stat_sum":{"num_bytes":22561655,"num_objects":686,"num_object_clones":314,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":247,"num_read":162,"num_read_kb":29557,"num_write":1441,"num_write_kb":28808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"85'1048","reported_seq":1588,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.267547+0000","last_change":"2023-10-30T12:28:44.052633+0000","last_active":"2023-10-30T13:24:47.267547+0000","last_peered":"2023-10-30T13:24:47.267547+0000","last_clean":"2023-10-30T13:24:47.267547+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T13:24:47.267547+0000","last_undegraded":"2023-10-30T13:24:47.267547+0000","last_fullsized":"2023-10-30T13:24:47.267547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1048,"log_dups_size":0,"ondisk_log_size":1048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":939,"num_read_kb":433966,"num_write":1214,"num_write_kb":88022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"82'1531","reported_seq":3512,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.140708+0000","last_change":"2023-10-30T12:37:07.189494+0000","last_active":"2023-10-30T13:25:16.140708+0000","last_peered":"2023-10-30T13:25:16.140708+0000","last_clean":"2023-10-30T13:25:16.140708+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T13:25:16.140708+0000","last_undegraded":"2023-10-30T13:25:16.140708+0000","last_fullsized":"2023-10-30T13:25:16.140708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1531,"log_dups_size":0,"ondisk_log_size":1531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:34.313998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014632805,"stat_sum":{"num_bytes":14611265,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":226,"num_read":122,"num_read_kb":17905,"num_write":1326,"num_write_kb":16683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"85'753","reported_seq":1130,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.373735+0000","last_change":"2023-10-30T12:28:44.075493+0000","last_active":"2023-10-30T13:24:56.373735+0000","last_peered":"2023-10-30T13:24:56.373735+0000","last_clean":"2023-10-30T13:24:56.373735+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T13:24:56.373735+0000","last_undegraded":"2023-10-30T13:24:56.373735+0000","last_fullsized":"2023-10-30T13:24:56.373735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:00:11.718669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":488,"num_read_kb":379559,"num_write":896,"num_write_kb":83880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"77'1481","reported_seq":3496,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.138978+0000","last_change":"2023-10-30T12:37:07.191451+0000","last_active":"2023-10-30T13:24:56.138978+0000","last_peered":"2023-10-30T13:24:56.138978+0000","last_clean":"2023-10-30T13:24:56.138978+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T13:24:56.138978+0000","last_undegraded":"2023-10-30T13:24:56.138978+0000","last_fullsized":"2023-10-30T13:24:56.138978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:28.248158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015871881000000001,"stat_sum":{"num_bytes":15563160,"num_objects":581,"num_object_clones":268,"num_object_copies":1162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":208,"num_read":132,"num_read_kb":9741,"num_write":1268,"num_write_kb":20904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"85'838","reported_seq":1196,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.123097+0000","last_change":"2023-10-30T12:28:44.065843+0000","last_active":"2023-10-30T13:25:02.123097+0000","last_peered":"2023-10-30T13:25:02.123097+0000","last_clean":"2023-10-30T13:25:02.123097+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T13:25:02.123097+0000","last_undegraded":"2023-10-30T13:25:02.123097+0000","last_fullsized":"2023-10-30T13:25:02.123097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:32.405199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33555014,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":517,"num_read_kb":175570,"num_write":978,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165214918,"num_objects":39301,"num_object_clones":18317,"num_object_copies":78602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39301,"num_whiteouts":14249,"num_read":9251,"num_read_kb":1265164,"num_write":86732,"num_write_kb":1459764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424127488,"data_stored":2332445068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99517,"ondisk_log_size":99517,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":903126397,"num_objects":1657,"num_object_clones":0,"num_object_copies":3314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":21550,"num_read_kb":9552846,"num_write":41301,"num_write_kb":2210413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1806409728,"data_stored":1806251397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34756,"ondisk_log_size":34756,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477425,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":837308,"kb_used_data":639604,"kb_used_omap":1,"kb_used_meta":197694,"kb_avail":93534532,"statfs":{"total":96636764160,"available":95779360768,"internally_reserved":0,"allocated":654954496,"data_stored":640799419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":202439112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42699999999999999,"15min":0.434},"min":{"1min":0.253,"5min":0.253,"15min":0.23200000000000001},"max":{"1min":0.52300000000000002,"5min":0.66900000000000004,"15min":0.94599999999999995},"last":0.377},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.47099999999999997,"15min":0.46999999999999997},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.27700000000000002},"max":{"1min":0.68300000000000005,"5min":0.75800000000000001,"15min":0.81899999999999995},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.503,"15min":0.495},"min":{"1min":0.39000000000000001,"5min":0.33300000000000002,"15min":0.307},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":1.0209999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.76600000000000001,"5min":0.78400000000000003,"15min":0.99299999999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51500000000000001,"15min":0.502},"min":{"1min":0.373,"5min":0.34100000000000003,"15min":0.28599999999999998},"max":{"1min":0.747,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.434},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.42999999999999999,"5min":0.38600000000000001,"15min":0.33200000000000002},"max":{"1min":0.79700000000000004,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77900000000000003,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.94299999999999995,"5min":1.05,"15min":1.548},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.63900000000000001,"5min":0.497,"15min":0.497},"max":{"1min":1.127,"5min":1.149,"15min":1.6950000000000001},"last":0.73299999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.78900000000000003,"15min":0.81399999999999995},"min":{"1min":0.57299999999999995,"5min":0.52800000000000002,"15min":0.496},"max":{"1min":0.96399999999999997,"5min":1.1160000000000001,"15min":1.194},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.80000000000000004,"15min":0.80700000000000005},"min":{"1min":0.66100000000000003,"5min":0.63200000000000001,"15min":0.52100000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.603},"last":0.623}]},{"osd":6,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.64000000000000001,"5min":0.56999999999999995,"15min":0.436},"max":{"1min":1.113,"5min":1.198,"15min":1.2270000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.93100000000000005,"15min":0.92300000000000004},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.55600000000000005},"max":{"1min":1.145,"5min":1.202,"15min":1.3759999999999999},"last":0.75900000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.84799999999999998,"15min":0.83599999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.46400000000000002},"max":{"1min":1.133,"5min":1.133,"15min":1.1919999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.91400000000000003,"15min":0.877},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.47699999999999998},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.2050000000000001},"last":0.83299999999999996}]}]},{"osd":2,"up_from":16,"seq":68719477426,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534076,"kb_used_data":412724,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":93837764,"statfs":{"total":96636764160,"available":96089870336,"internally_reserved":0,"allocated":422629376,"data_stored":409295067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124254662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42999999999999999,"15min":0.43099999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57799999999999996,"5min":0.81299999999999994,"15min":0.91200000000000003},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48799999999999999,"15min":0.47899999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.625,"5min":1.554,"15min":2.8519999999999999},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47799999999999998,"15min":0.497},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.57199999999999995,"5min":1.071,"15min":4.9660000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52700000000000002,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.32100000000000001,"15min":0.28399999999999997},"max":{"1min":0.66600000000000004,"5min":1.198,"15min":6.0759999999999996},"last":0.442}]},{"osd":3,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.505,"15min":0.52800000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.314},"max":{"1min":0.65500000000000003,"5min":0.96799999999999997,"15min":4.6120000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56499999999999995,"15min":0.58699999999999997},"min":{"1min":0.42699999999999999,"5min":0.40899999999999997,"15min":0.379},"max":{"1min":0.68200000000000005,"5min":1.4430000000000001,"15min":4.641},"last":0.433}]},{"osd":4,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79100000000000004,"15min":0.80400000000000005},"min":{"1min":0.58399999999999996,"5min":0.48299999999999998,"15min":0.47999999999999998},"max":{"1min":0.95999999999999996,"5min":1.1719999999999999,"15min":3.5499999999999998},"last":1.083},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77000000000000002,"15min":0.81000000000000005},"min":{"1min":0.53200000000000003,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0389999999999999,"5min":1.262,"15min":5.032},"last":0.76100000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.88800000000000001,"15min":0.87},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.177,"5min":1.494,"15min":6.1289999999999996},"last":1.014},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.84699999999999998,"15min":0.877},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.159,"5min":1.3440000000000001,"15min":3.1459999999999999},"last":0.96899999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.83299999999999996,"15min":0.875},"min":{"1min":0.67600000000000005,"5min":0.54000000000000004,"15min":0.53100000000000003},"max":{"1min":1.0469999999999999,"5min":1.1739999999999999,"15min":5.8159999999999998},"last":1.139},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.82799999999999996,"15min":0.83299999999999996},"min":{"1min":0.66600000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.99299999999999999,"5min":1.6299999999999999,"15min":3.415},"last":1.0620000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.84199999999999997,"15min":0.872},"min":{"1min":0.73999999999999999,"5min":0.60699999999999998,"15min":0.51100000000000001},"max":{"1min":1.121,"5min":1.9139999999999999,"15min":4.883},"last":1.107},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.88400000000000001,"15min":0.90300000000000002},"min":{"1min":0.79300000000000004,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0640000000000001,"5min":1.3460000000000001,"15min":4.9950000000000001},"last":1.1619999999999999}]}]},{"osd":1,"up_from":16,"seq":68719477425,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534824,"kb_used_data":431520,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":93837016,"statfs":{"total":96636764160,"available":96089104384,"internally_reserved":0,"allocated":441876480,"data_stored":431229668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105773510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46899999999999997,"15min":0.52500000000000002},"min":{"1min":0.32800000000000001,"5min":0.27700000000000002,"15min":0.22600000000000001},"max":{"1min":0.66700000000000004,"5min":0.70699999999999996,"15min":20.196000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55100000000000005,"15min":0.54800000000000004},"min":{"1min":0.39800000000000002,"5min":0.31,"15min":0.253},"max":{"1min":0.77200000000000002,"5min":0.91100000000000003,"15min":5.1070000000000002},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48599999999999999,"15min":0.67200000000000004},"min":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.312},"max":{"1min":0.73099999999999998,"5min":0.83999999999999997,"15min":45.866999999999997},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.57099999999999995,"15min":0.64100000000000001},"min":{"1min":0.433,"5min":0.374,"15min":0.34699999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":20.789999999999999},"last":0.46400000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.47899999999999998,"15min":0.57899999999999996},"min":{"1min":0.36899999999999999,"5min":0.318,"15min":0.30499999999999999},"max":{"1min":0.624,"5min":0.80900000000000005,"15min":21.178999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.34999999999999998},"max":{"1min":0.71099999999999997,"5min":0.91800000000000004,"15min":2.2290000000000001},"last":0.50800000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.40400000000000003},"max":{"1min":1.133,"5min":1.133,"15min":5.9770000000000003},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.90400000000000003,"15min":0.86899999999999999},"min":{"1min":0.68400000000000005,"5min":0.64300000000000002,"15min":0.44400000000000001},"max":{"1min":1.093,"5min":1.1619999999999999,"15min":5.5110000000000001},"last":0.875}]},{"osd":5,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82399999999999995,"15min":0.872},"min":{"1min":0.58299999999999996,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0149999999999999,"5min":1.212,"15min":7.5430000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.90100000000000002,"15min":0.94499999999999995},"min":{"1min":0.76700000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0680000000000001,"5min":1.2350000000000001,"15min":7.5750000000000002},"last":0.84199999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.79900000000000004,"15min":0.88300000000000001},"min":{"1min":0.59699999999999998,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0820000000000001,"5min":1.177,"15min":20.827000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.82099999999999995,"15min":0.95399999999999996},"min":{"1min":0.68400000000000005,"5min":0.51800000000000002,"15min":0.51600000000000001},"max":{"1min":1.0469999999999999,"5min":1.159,"15min":20.931000000000001},"last":0.72499999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.83999999999999997,"15min":0.95799999999999996},"min":{"1min":0.73299999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":20.859999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90600000000000003,"15min":0.88500000000000001},"min":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.501},"max":{"1min":1.1870000000000001,"5min":1.194,"15min":4.367},"last":0.91600000000000004}]}]},{"osd":0,"up_from":16,"seq":68719477426,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937928,"kb_used_data":695808,"kb_used_omap":1,"kb_used_meta":242110,"kb_avail":93433912,"statfs":{"total":96636764160,"available":95676325888,"internally_reserved":0,"allocated":712507392,"data_stored":696810257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":247921094},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40600000000000003,"15min":0.41099999999999998},"min":{"1min":0.254,"5min":0.23899999999999999,"15min":0.22700000000000001},"max":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":3.996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51700000000000002,"15min":0.52600000000000002},"min":{"1min":0.41099999999999998,"5min":0.36199999999999999,"15min":0.309},"max":{"1min":0.67600000000000005,"5min":0.75600000000000001,"15min":4.3230000000000004},"last":0.44700000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.27700000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":4.9690000000000003},"last":0.372},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52100000000000002,"15min":0.53200000000000003},"min":{"1min":0.372,"5min":0.34100000000000003,"15min":0.30199999999999999},"max":{"1min":0.621,"5min":0.878,"15min":5.0049999999999999},"last":0.42499999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.48799999999999999},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.28299999999999997},"max":{"1min":0.68100000000000005,"5min":0.82699999999999996,"15min":5.4909999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57499999999999996,"15min":0.58099999999999996},"min":{"1min":0.443,"5min":0.42799999999999999,"15min":0.38600000000000001},"max":{"1min":0.749,"5min":0.90400000000000003,"15min":5.6970000000000001},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.58699999999999997,"5min":0.46700000000000003,"15min":0.377},"max":{"1min":1.111,"5min":1.111,"15min":4.444},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.80100000000000005},"min":{"1min":0.58499999999999996,"5min":0.55000000000000004,"15min":0.48099999999999998},"max":{"1min":0.90500000000000003,"5min":1.117,"15min":5.7750000000000004},"last":0.622}]},{"osd":5,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.83299999999999996,"15min":0.83499999999999996},"min":{"1min":0.66800000000000004,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.246,"5min":1.3340000000000001,"15min":4.5430000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.871,"15min":0.86399999999999999},"min":{"1min":0.59199999999999997,"5min":0.56999999999999995,"15min":0.47099999999999997},"max":{"1min":1.204,"5min":1.204,"15min":4.4790000000000001},"last":0.76200000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.82599999999999996,"15min":0.84199999999999997},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":1.397,"15min":4.5129999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.88500000000000001,"15min":0.89900000000000002},"min":{"1min":0.626,"5min":0.626,"15min":0.56699999999999995},"max":{"1min":1.1699999999999999,"5min":1.3120000000000001,"15min":4.5750000000000002},"last":0.60399999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.875,"15min":0.86799999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.55700000000000005},"max":{"1min":1.0669999999999999,"5min":1.206,"15min":4.3979999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.88900000000000001,"15min":0.91400000000000003},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.53700000000000003},"max":{"1min":1.0760000000000001,"5min":1.2210000000000001,"15min":5.375},"last":0.79700000000000004}]}]},{"osd":6,"up_from":15,"seq":64424510128,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526444,"kb_used_data":409252,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":93845396,"statfs":{"total":96636764160,"available":96097685504,"internally_reserved":0,"allocated":419074048,"data_stored":408471134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":119994823},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.874,"15min":0.86299999999999999},"min":{"1min":0.61099999999999999,"5min":0.49399999999999999,"15min":0.34200000000000003},"max":{"1min":1.167,"5min":1.4470000000000001,"15min":3.1150000000000002},"last":1.103},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.89900000000000002,"15min":0.96999999999999997},"min":{"1min":0.63500000000000001,"5min":0.54500000000000004,"15min":0.51700000000000002},"max":{"1min":1.121,"5min":1.415,"15min":3.3639999999999999},"last":0.89500000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.79800000000000004,"15min":0.79700000000000004},"min":{"1min":0.48799999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.087,"5min":1.202,"15min":1.589},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85899999999999999,"15min":0.86199999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.47699999999999998},"max":{"1min":1.145,"5min":1.2330000000000001,"15min":1.544},"last":0.83699999999999997}]},{"osd":2,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.98199999999999998,"15min":0.96299999999999997},"min":{"1min":0.68300000000000005,"5min":0.57299999999999995,"15min":0.47199999999999998},"max":{"1min":1.2969999999999999,"5min":1.496,"15min":3.4969999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.91600000000000004,"15min":0.94999999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.2090000000000001,"5min":1.391,"15min":1.512},"last":1.018}]},{"osd":3,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.86299999999999999,"15min":0.89100000000000001},"min":{"1min":0.58799999999999997,"5min":0.57099999999999995,"15min":0.52800000000000002},"max":{"1min":1.131,"5min":1.262,"15min":3.6070000000000002},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.91300000000000003,"15min":0.88300000000000001},"min":{"1min":0.69499999999999995,"5min":0.54000000000000004,"15min":0.51600000000000001},"max":{"1min":1.1839999999999999,"5min":1.272,"15min":3.7679999999999998},"last":0.77600000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97299999999999998,"15min":0.93799999999999994},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.2849999999999999,"5min":1.3220000000000001,"15min":1.4019999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.093,"15min":1.046},"min":{"1min":0.80000000000000004,"5min":0.73899999999999999,"15min":0.54700000000000004},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.2509999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.163,"15min":1.125},"min":{"1min":0.90100000000000002,"5min":0.82999999999999996,"15min":0.73399999999999999},"max":{"1min":1.417,"5min":1.4730000000000001,"15min":1.583},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.069,"15min":1.095},"min":{"1min":0.625,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2829999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.137}]},{"osd":7,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.081,"15min":1.0800000000000001},"min":{"1min":0.86099999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.4970000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.22,"15min":1.2170000000000001},"min":{"1min":1.034,"5min":0.76500000000000001,"15min":0.64200000000000002},"max":{"1min":1.804,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.1699999999999999}]}]},{"osd":4,"up_from":15,"seq":64424510130,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645792,"kb_used_data":493912,"kb_used_omap":1,"kb_used_meta":151870,"kb_avail":93726048,"statfs":{"total":96636764160,"available":95975473152,"internally_reserved":0,"allocated":505765888,"data_stored":496226659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":155515335},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.86299999999999999,"15min":0.83899999999999997},"min":{"1min":0.54800000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.1930000000000001,"5min":1.218,"15min":1.292},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.86899999999999999,"15min":0.84699999999999998},"min":{"1min":0.57399999999999995,"5min":0.52700000000000002,"15min":0.52600000000000002},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.369},"last":1.4570000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.79400000000000004,"15min":0.78600000000000003},"min":{"1min":0.60799999999999998,"5min":0.47899999999999998,"15min":0.47499999999999998},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1839999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.879,"5min":0.83699999999999997,"15min":0.83199999999999996},"min":{"1min":0.60899999999999999,"5min":0.53000000000000003,"15min":0.47299999999999998},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.3100000000000001},"last":0.996}]},{"osd":2,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.89500000000000002},"min":{"1min":0.56599999999999995,"5min":0.55500000000000005,"15min":0.55300000000000005},"max":{"1min":1.1899999999999999,"5min":1.2270000000000001,"15min":1.292},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.998,"5min":0.94599999999999995,"15min":0.94699999999999995},"min":{"1min":0.74099999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":1.3540000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.97399999999999998,"15min":0.95299999999999996},"min":{"1min":0.68500000000000005,"5min":0.65100000000000002,"15min":0.52200000000000002},"max":{"1min":1.286,"5min":1.286,"15min":1.514},"last":1.234},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.94699999999999995,"15min":0.96299999999999997},"min":{"1min":0.76600000000000001,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.94999999999999996}]},{"osd":5,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.98299999999999998,"15min":1.0129999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60299999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":2.4430000000000001},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.171,"5min":1.1140000000000001,"15min":1.0660000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.8460000000000001},"last":1.304}]},{"osd":6,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.145,"15min":1.1559999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.419,"5min":1.425,"15min":1.677},"last":1.411},{"interface":"front","average":{"1min":1.306,"5min":1.28,"15min":1.1830000000000001},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.64400000000000002},"max":{"1min":1.528,"5min":1.5409999999999999,"15min":1.9610000000000001},"last":1.5009999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.0529999999999999,"15min":1.1359999999999999},"min":{"1min":0.89500000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.8700000000000001},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.202,"15min":1.266},"min":{"1min":1.016,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.538,"5min":1.591,"15min":1.9450000000000001},"last":1.2689999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542832,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560428,"kb_used_data":416932,"kb_used_omap":1,"kb_used_meta":143486,"kb_avail":93811412,"statfs":{"total":96636764160,"available":96062885888,"internally_reserved":0,"allocated":426938368,"data_stored":418861204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":146930123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.84399999999999997,"15min":0.82999999999999996},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.41499999999999998},"max":{"1min":1.2130000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.88,"15min":0.872},"min":{"1min":0.56999999999999995,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.333,"5min":1.3380000000000001,"15min":1.3919999999999999},"last":1.1120000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.76900000000000002,"15min":0.86799999999999999},"min":{"1min":0.504,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.095,"5min":1.244,"15min":1.4790000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.91700000000000004,"15min":0.82299999999999995},"min":{"1min":0.51500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.5,"5min":1.524,"15min":1.524},"last":0.80400000000000005}]},{"osd":2,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0589999999999999,"15min":1.024},"min":{"1min":0.66500000000000004,"5min":0.54700000000000004,"15min":0.52700000000000002},"max":{"1min":1.3109999999999999,"5min":1.833,"15min":1.833},"last":1.022},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1830000000000001,"15min":1.153},"min":{"1min":0.91100000000000003,"5min":0.71499999999999997,"15min":0.69399999999999995},"max":{"1min":1.4510000000000001,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.0660000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0760000000000001,"15min":1.083},"min":{"1min":0.66400000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.2929999999999999,"5min":1.673,"15min":1.673},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.204,"5min":1.244,"15min":1.2230000000000001},"min":{"1min":0.84299999999999997,"5min":0.83499999999999996,"15min":0.81399999999999995},"max":{"1min":1.4490000000000001,"5min":1.8109999999999999,"15min":1.8480000000000001},"last":1.155}]},{"osd":4,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.86199999999999999,"15min":0.84799999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.171,"5min":1.226,"15min":1.4970000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.95799999999999996,"15min":0.95599999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.242,"5min":1.4970000000000001,"15min":1.77},"last":0.72499999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.94299999999999995,"15min":0.97499999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.2490000000000001,"5min":1.3799999999999999,"15min":2},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.105,"15min":1.0349999999999999},"min":{"1min":0.69599999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.508,"5min":1.7150000000000001,"15min":2.056},"last":0.92900000000000005}]},{"osd":6,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.042,"15min":1.008},"min":{"1min":0.68100000000000005,"5min":0.60799999999999998,"15min":0.501},"max":{"1min":1.377,"5min":1.536,"15min":1.571},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1299999999999999,"15min":1.105},"min":{"1min":0.751,"5min":0.72599999999999998,"15min":0.71199999999999997},"max":{"1min":1.5549999999999999,"5min":1.665,"15min":1.665},"last":0.84899999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542830,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":802272,"kb_used_data":640600,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93569568,"statfs":{"total":96636764160,"available":95815237632,"internally_reserved":0,"allocated":655974400,"data_stored":642425281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165542342},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.378},"max":{"1min":1.173,"5min":1.276,"15min":1.579},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.85299999999999998,"15min":0.86099999999999999},"min":{"1min":0.499,"5min":0.46200000000000002,"15min":0.34599999999999997},"max":{"1min":1.262,"5min":1.627,"15min":1.627},"last":0.86299999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.81200000000000006,"15min":0.83099999999999996},"min":{"1min":0.55300000000000005,"5min":0.47499999999999998,"15min":0.39500000000000002},"max":{"1min":1.22,"5min":1.226,"15min":1.3220000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.88600000000000001,"15min":0.91900000000000004},"min":{"1min":0.64800000000000002,"5min":0.52600000000000002,"15min":0.376},"max":{"1min":1.1539999999999999,"5min":1.3100000000000001,"15min":1.6619999999999999},"last":0.72299999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.85799999999999998,"15min":0.84799999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.2330000000000001,"5min":1.3220000000000001,"15min":1.486},"last":0.5},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.89300000000000002,"15min":0.88},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2749999999999999,"5min":1.3149999999999999,"15min":1.879},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.96899999999999997,"15min":0.88900000000000001},"min":{"1min":0.65500000000000003,"5min":0.55900000000000005,"15min":0.42799999999999999},"max":{"1min":1.361,"5min":1.361,"15min":2.7330000000000001},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.097,"5min":1.0509999999999999,"15min":0.999},"min":{"1min":0.68899999999999995,"5min":0.51300000000000001,"15min":0.47099999999999997},"max":{"1min":1.359,"5min":1.5169999999999999,"15min":2.79},"last":1.1479999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.97299999999999998,"15min":0.98099999999999998},"min":{"1min":0.68500000000000005,"5min":0.57599999999999996,"15min":0.45300000000000001},"max":{"1min":1.119,"5min":1.4710000000000001,"15min":1.6719999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.034,"15min":1.0940000000000001},"min":{"1min":0.87,"5min":0.626,"15min":0.58799999999999997},"max":{"1min":1.3120000000000001,"5min":1.3540000000000001,"15min":1.7629999999999999},"last":0.77900000000000003}]},{"osd":6,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.1000000000000001,"15min":1.0920000000000001},"min":{"1min":0.69999999999999996,"5min":0.66500000000000004,"15min":0.54600000000000004},"max":{"1min":1.3959999999999999,"5min":1.675,"15min":2.8450000000000002},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.123,"15min":1.111},"min":{"1min":0.73699999999999999,"5min":0.59999999999999998,"15min":0.57999999999999996},"max":{"1min":1.577,"5min":1.625,"15min":1.821},"last":0.98699999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.153,"15min":1.127},"min":{"1min":0.70099999999999996,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7490000000000001,"5min":1.7749999999999999,"15min":3.3069999999999999},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.115,"5min":1.1579999999999999,"15min":1.113},"min":{"1min":0.88500000000000001,"5min":0.59999999999999998,"15min":0.53200000000000003},"max":{"1min":1.3129999999999999,"5min":1.754,"15min":2.036},"last":0.92000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309846016,"data_stored":309822659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142618624,"data_stored":142610667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117452800,"data_stored":117441250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294043648,"data_stored":294028449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":275668992,"data_stored":275627234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":300916736,"data_stored":300897937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":154218496,"data_stored":154206848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":211644416,"data_stored":211616353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401661952,"data_stored":386457412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297664512,"data_stored":287498447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304177152,"data_stored":291323631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359911424,"data_stored":346240784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":229097472,"data_stored":220069239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":353464320,"data_stored":340406790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":263856128,"data_stored":253734100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":214294528,"data_stored":206714665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:25:36.391 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-30T13:25:36.392 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:25:36.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.562+0000 7f0aa5891640 1 Processor -- start 2023-10-30T13:25:36.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.562+0000 7f0aa5891640 1 -- start start 2023-10-30T13:25:36.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.562+0000 7f0aa5891640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f0aa00c6b80 0x7f0aa00bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.562+0000 7f0aa5891640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f0aa013f4a0 0x7f0aa00c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.562+0000 7f0aa5891640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa00c0a50 con 0x7f0aa00c6f60 2023-10-30T13:25:36.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.562+0000 7f0aa5891640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa006f6e0 con 0x7f0aa013f4a0 2023-10-30T13:25:36.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.562+0000 7f0aa5891640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa006f9c0 con 0x7f0aa00c6b80 2023-10-30T13:25:36.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.562+0000 7f0a97fff640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f0aa00c6f60 0x7f0aa013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59592/0 (socket says 172.21.15.125:59592) 2023-10-30T13:25:36.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.562+0000 7f0a97fff640 1 -- 172.21.15.125:0/249565848 learned_addr learned my addr 172.21.15.125:0/249565848 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:25:36.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.562+0000 7f0a9f7fe640 1 --2- 172.21.15.125:0/249565848 >> v2:172.21.15.125:3300/0 conn(0x7f0aa013f4a0 0x7f0aa00c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.562+0000 7f0a9effd640 1 --2- 172.21.15.125:0/249565848 >> v2:172.21.15.136:3300/0 conn(0x7f0aa00c6b80 0x7f0aa00bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0a9e7fc640 1 -- 172.21.15.125:0/249565848 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628567271 0 0) 0x7f0aa00c0a50 con 0x7f0aa00c6f60 2023-10-30T13:25:36.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0a9e7fc640 1 -- 172.21.15.125:0/249565848 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a8c005740 con 0x7f0aa00c6f60 2023-10-30T13:25:36.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0a9f7fe640 1 -- 172.21.15.125:0/249565848 >> v2:172.21.15.136:3300/0 conn(0x7f0aa00c6b80 msgr2=0x7f0aa00bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:36.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0a9f7fe640 1 --2- 172.21.15.125:0/249565848 >> v2:172.21.15.136:3300/0 conn(0x7f0aa00c6b80 0x7f0aa00bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0a9f7fe640 1 -- 172.21.15.125:0/249565848 >> v1:172.21.15.125:6789/0 conn(0x7f0aa00c6f60 legacy=0x7f0aa013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0a9f7fe640 1 -- 172.21.15.125:0/249565848 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa006fca0 con 0x7f0aa013f4a0 2023-10-30T13:25:36.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0a9f7fe640 1 --2- 172.21.15.125:0/249565848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0aa013f4a0 0x7f0aa00c0400 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f0aa006f1b0 tx=0x7f0a9801c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f23e9a200f0227b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:36.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0a9e7fc640 1 -- 172.21.15.125:0/249565848 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a980a58e0 con 0x7f0aa013f4a0 2023-10-30T13:25:36.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0a9e7fc640 1 -- 172.21.15.125:0/249565848 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a980a5a40 con 0x7f0aa013f4a0 2023-10-30T13:25:36.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0a9e7fc640 1 -- 172.21.15.125:0/249565848 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a980a5dd0 con 0x7f0aa013f4a0 2023-10-30T13:25:36.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0aa5891640 1 -- 172.21.15.125:0/249565848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0aa013f4a0 msgr2=0x7f0aa00c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0aa5891640 1 --2- 172.21.15.125:0/249565848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0aa013f4a0 0x7f0aa00c0400 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f0aa006f1b0 tx=0x7f0a9801c910 comp rx=0 tx=0).stop 2023-10-30T13:25:36.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0aa5891640 1 -- 172.21.15.125:0/249565848 shutdown_connections 2023-10-30T13:25:36.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0aa5891640 1 --2- 172.21.15.125:0/249565848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0aa013f4a0 0x7f0aa00c0400 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0aa5891640 1 --2- 172.21.15.125:0/249565848 >> v2:172.21.15.136:3300/0 conn(0x7f0aa00c6b80 0x7f0aa00bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.566+0000 7f0aa5891640 1 -- 172.21.15.125:0/249565848 >> 172.21.15.125:0/249565848 conn(0x7f0aa00c7910 msgr2=0x7f0aa00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:36.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0aa5891640 1 -- 172.21.15.125:0/249565848 shutdown_connections 2023-10-30T13:25:36.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0aa5891640 1 -- 172.21.15.125:0/249565848 wait complete. 2023-10-30T13:25:36.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0aa5891640 1 Processor -- start 2023-10-30T13:25:36.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0aa5891640 1 -- start start 2023-10-30T13:25:36.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0aa5891640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0aa00c6b80 0x7f0aa0152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0aa5891640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0aa00c6f60 0x7f0aa00c9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0aa5891640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0aa013f4a0 0x7f0aa00ca130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0aa5891640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa0159360 con 0x7f0aa00c6b80 2023-10-30T13:25:36.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0aa5891640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa0153790 con 0x7f0aa013f4a0 2023-10-30T13:25:36.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0aa5891640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa00c8fe0 con 0x7f0aa00c6f60 2023-10-30T13:25:36.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a9effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0aa00c6b80 0x7f0aa0152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a9effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0aa00c6b80 0x7f0aa0152090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38596/0 (socket says 172.21.15.125:38596) 2023-10-30T13:25:36.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a9effd640 1 -- 172.21.15.125:0/448588036 learned_addr learned my addr 172.21.15.125:0/448588036 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:36.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a97fff640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0aa00c6f60 0x7f0aa00c9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a9f7fe640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0aa013f4a0 0x7f0aa00ca130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a9effd640 1 -- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0aa00c6f60 msgr2=0x7f0aa00c9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a9effd640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0aa00c6f60 0x7f0aa00c9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a9effd640 1 -- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0aa013f4a0 msgr2=0x7f0aa00ca130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a9effd640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0aa013f4a0 0x7f0aa00ca130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a9effd640 1 -- 172.21.15.125:0/448588036 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a98009a70 con 0x7f0aa00c6b80 2023-10-30T13:25:36.583 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a9f7fe640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0aa013f4a0 0x7f0aa00ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:25:36.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a9effd640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0aa00c6b80 0x7f0aa0152090 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000e930 tx=0x7f0a9000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:36.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a9cff9640 1 -- 172.21.15.125:0/448588036 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a90009690 con 0x7f0aa00c6b80 2023-10-30T13:25:36.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a9cff9640 1 -- 172.21.15.125:0/448588036 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9000ca00 con 0x7f0aa00c6b80 2023-10-30T13:25:36.584 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0aa5891640 1 -- 172.21.15.125:0/448588036 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa0159570 con 0x7f0aa00c6b80 2023-10-30T13:25:36.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0a9cff9640 1 -- 172.21.15.125:0/448588036 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a9000cd70 con 0x7f0aa00c6b80 2023-10-30T13:25:36.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.570+0000 7f0aa5891640 1 -- 172.21.15.125:0/448588036 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa00c93f0 con 0x7f0aa00c6b80 2023-10-30T13:25:36.585 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.574+0000 7f0aa5891640 1 -- 172.21.15.125:0/448588036 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa004e170 con 0x7f0aa00c6b80 2023-10-30T13:25:36.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.574+0000 7f0a9cff9640 1 -- 172.21.15.125:0/448588036 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f0a90013470 con 0x7f0aa00c6b80 2023-10-30T13:25:36.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.574+0000 7f0a9cff9640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a800664a0 0x7f0a80068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.574+0000 7f0a9cff9640 1 -- 172.21.15.125:0/448588036 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f0a90084cd0 con 0x7f0aa00c6b80 2023-10-30T13:25:36.586 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.574+0000 7f0a97fff640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a800664a0 0x7f0a80068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.574+0000 7f0a97fff640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a800664a0 0x7f0a80068960 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f0aa00cade0 tx=0x7f0a88009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:36.587 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.578+0000 7f0a9cff9640 1 -- 172.21.15.125:0/448588036 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f0a90051640 con 0x7f0aa00c6b80 2023-10-30T13:25:36.729 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.726+0000 7f0aa5891640 1 -- 172.21.15.125:0/448588036 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0aa0159020 con 0x7f0a800664a0 2023-10-30T13:25:36.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.750+0000 7f0a9cff9640 1 -- 172.21.15.125:0/448588036 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293478 (secure 0 0 0) 0x7f0aa0159020 con 0x7f0a800664a0 2023-10-30T13:25:36.755 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:25:36.756 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T13:25:36.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.750+0000 7f0aa5891640 1 -- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a800664a0 msgr2=0x7f0a80068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.750+0000 7f0aa5891640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a800664a0 0x7f0a80068960 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f0aa00cade0 tx=0x7f0a88009040 comp rx=0 tx=0).stop 2023-10-30T13:25:36.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.750+0000 7f0aa5891640 1 -- 172.21.15.125:0/448588036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0aa00c6b80 msgr2=0x7f0aa0152090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.750+0000 7f0aa5891640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0aa00c6b80 0x7f0aa0152090 secure :-1 s=READY pgs=2157 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000e930 tx=0x7f0a9000edf0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.750+0000 7f0aa5891640 1 -- 172.21.15.125:0/448588036 shutdown_connections 2023-10-30T13:25:36.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.750+0000 7f0aa5891640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f0a800664a0 0x7f0a80068960 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.750+0000 7f0aa5891640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f0aa013f4a0 0x7f0aa00ca130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.750+0000 7f0aa5891640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f0aa00c6f60 0x7f0aa00c9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.750+0000 7f0aa5891640 1 --2- 172.21.15.125:0/448588036 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f0aa00c6b80 0x7f0aa0152090 unknown :-1 s=CLOSED pgs=2157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.750+0000 7f0aa5891640 1 -- 172.21.15.125:0/448588036 >> 172.21.15.125:0/448588036 conn(0x7f0aa00c7910 msgr2=0x7f0aa00c11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:36.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.750+0000 7f0aa5891640 1 -- 172.21.15.125:0/448588036 shutdown_connections 2023-10-30T13:25:36.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.754+0000 7f0aa5891640 1 -- 172.21.15.125:0/448588036 wait complete. 2023-10-30T13:25:36.775 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":1790,"stamp":"2023-10-30T13:25:36.270456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2068931683,"num_objects":40960,"num_object_clones":18317,"num_object_copies":81920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40960,"num_whiteouts":14249,"num_read":30917,"num_read_kb":10818110,"num_write":128192,"num_write_kb":3672593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134390,"ondisk_log_size":134390,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5379072,"kb_used_data":4140352,"kb_used_omap":12,"kb_used_meta":1238643,"kb_avail":749595648,"statfs":{"total":773094113280,"available":767585943552,"internally_reserved":0,"allocated":4239720448,"data_stored":4144118689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1268371001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1465631,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":41,"num_read_kb":31886,"num_write":14,"num_write_kb":1441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004753"},"pg_stats":[{"pgid":"3.3f","version":"77'1475","reported_seq":3523,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.196345+0000","last_change":"2023-10-30T12:37:07.182969+0000","last_active":"2023-10-30T13:25:16.196345+0000","last_peered":"2023-10-30T13:25:16.196345+0000","last_clean":"2023-10-30T13:25:16.196345+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T13:25:16.196345+0000","last_undegraded":"2023-10-30T13:25:16.196345+0000","last_fullsized":"2023-10-30T13:25:16.196345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1475,"log_dups_size":0,"ondisk_log_size":1475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:33.616938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086120880000000004,"stat_sum":{"num_bytes":27261067,"num_objects":552,"num_object_clones":249,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":197,"num_read":145,"num_read_kb":29501,"num_write":1298,"num_write_kb":29008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"77'1476","reported_seq":3487,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:01.374954+0000","last_change":"2023-10-30T12:37:07.190309+0000","last_active":"2023-10-30T13:25:01.374954+0000","last_peered":"2023-10-30T13:25:01.374954+0000","last_clean":"2023-10-30T13:25:01.374954+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T13:25:01.374954+0000","last_undegraded":"2023-10-30T13:25:01.374954+0000","last_fullsized":"2023-10-30T13:25:01.374954+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:26:31.884771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015306986,"stat_sum":{"num_bytes":13736175,"num_objects":574,"num_object_clones":269,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":223,"num_read":130,"num_read_kb":17219,"num_write":1260,"num_write_kb":15409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"77'1563","reported_seq":3574,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288448+0000","last_change":"2023-10-30T12:37:07.187803+0000","last_active":"2023-10-30T13:24:16.288448+0000","last_peered":"2023-10-30T13:24:16.288448+0000","last_clean":"2023-10-30T13:24:16.288448+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T13:24:16.288448+0000","last_undegraded":"2023-10-30T13:24:16.288448+0000","last_fullsized":"2023-10-30T13:24:16.288448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1563,"log_dups_size":0,"ondisk_log_size":1563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:32.502949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013634598,"stat_sum":{"num_bytes":28390436,"num_objects":628,"num_object_clones":291,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":232,"num_read":152,"num_read_kb":47076,"num_write":1356,"num_write_kb":37143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"77'1577","reported_seq":3585,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.277341+0000","last_change":"2023-10-30T12:37:07.177860+0000","last_active":"2023-10-30T13:25:32.277341+0000","last_peered":"2023-10-30T13:25:32.277341+0000","last_clean":"2023-10-30T13:25:32.277341+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T13:25:32.277341+0000","last_undegraded":"2023-10-30T13:25:32.277341+0000","last_fullsized":"2023-10-30T13:25:32.277341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1577,"log_dups_size":0,"ondisk_log_size":1577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:27.076595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056762749999999997,"stat_sum":{"num_bytes":22921274,"num_objects":629,"num_object_clones":290,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":224,"num_read":145,"num_read_kb":39229,"num_write":1388,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"77'1591","reported_seq":3596,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.167292+0000","last_change":"2023-10-30T12:37:07.181803+0000","last_active":"2023-10-30T13:24:56.167292+0000","last_peered":"2023-10-30T13:24:56.167292+0000","last_clean":"2023-10-30T13:24:56.167292+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T13:24:56.167292+0000","last_undegraded":"2023-10-30T13:24:56.167292+0000","last_fullsized":"2023-10-30T13:24:56.167292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1591,"log_dups_size":0,"ondisk_log_size":1591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:23:58.362682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081298299999999993,"stat_sum":{"num_bytes":26798060,"num_objects":623,"num_object_clones":286,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":214,"num_read":144,"num_read_kb":26713,"num_write":1434,"num_write_kb":32534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"85'1466","reported_seq":3496,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.286314+0000","last_change":"2023-10-30T12:37:07.176046+0000","last_active":"2023-10-30T13:24:16.286314+0000","last_peered":"2023-10-30T13:24:16.286314+0000","last_clean":"2023-10-30T13:24:16.286314+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T13:24:16.286314+0000","last_undegraded":"2023-10-30T13:24:16.286314+0000","last_fullsized":"2023-10-30T13:24:16.286314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:30:51.393008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048510920000000004,"stat_sum":{"num_bytes":13539463,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":205,"num_read":144,"num_read_kb":15818,"num_write":1313,"num_write_kb":14957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"84'1562","reported_seq":3551,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-30T13:23:51.391894+0000","last_change":"2023-10-30T12:37:07.173349+0000","last_active":"2023-10-30T13:23:51.391894+0000","last_peered":"2023-10-30T13:23:51.391894+0000","last_clean":"2023-10-30T13:23:51.391894+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T13:23:51.391894+0000","last_undegraded":"2023-10-30T13:23:51.391894+0000","last_fullsized":"2023-10-30T13:23:51.391894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1562,"log_dups_size":0,"ondisk_log_size":1562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:38:16.606065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082671399999999999,"stat_sum":{"num_bytes":7202428,"num_objects":606,"num_object_clones":284,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":219,"num_read":126,"num_read_kb":1677,"num_write":1361,"num_write_kb":10203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"82'1518","reported_seq":3541,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288347+0000","last_change":"2023-10-30T12:37:07.188855+0000","last_active":"2023-10-30T13:24:16.288347+0000","last_peered":"2023-10-30T13:24:16.288347+0000","last_clean":"2023-10-30T13:24:16.288347+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T13:24:16.288347+0000","last_undegraded":"2023-10-30T13:24:16.288347+0000","last_fullsized":"2023-10-30T13:24:16.288347+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:54.679005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014437559000000001,"stat_sum":{"num_bytes":23150826,"num_objects":587,"num_object_clones":269,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":199,"num_read":147,"num_read_kb":38801,"num_write":1362,"num_write_kb":24887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"77'1549","reported_seq":3579,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.247414+0000","last_change":"2023-10-30T12:37:07.102887+0000","last_active":"2023-10-30T13:25:32.247414+0000","last_peered":"2023-10-30T13:25:32.247414+0000","last_clean":"2023-10-30T13:25:32.247414+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T13:25:32.247414+0000","last_undegraded":"2023-10-30T13:25:32.247414+0000","last_fullsized":"2023-10-30T13:25:32.247414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1549,"log_dups_size":0,"ondisk_log_size":1549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:09.601324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014185980000000001,"stat_sum":{"num_bytes":36575095,"num_objects":618,"num_object_clones":282,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":216,"num_read":165,"num_read_kb":36418,"num_write":1354,"num_write_kb":45869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"77'1459","reported_seq":3491,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.224692+0000","last_change":"2023-10-30T12:37:07.183547+0000","last_active":"2023-10-30T13:25:32.224692+0000","last_peered":"2023-10-30T13:25:32.224692+0000","last_clean":"2023-10-30T13:25:32.224692+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T13:25:32.224692+0000","last_undegraded":"2023-10-30T13:25:32.224692+0000","last_fullsized":"2023-10-30T13:25:32.224692+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:43.076339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091404980000000004,"stat_sum":{"num_bytes":11176343,"num_objects":586,"num_object_clones":270,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":213,"num_read":154,"num_read_kb":10770,"num_write":1260,"num_write_kb":13564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"82'1511","reported_seq":3517,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.187191+0000","last_change":"2023-10-30T12:37:07.181463+0000","last_active":"2023-10-30T13:25:16.187191+0000","last_peered":"2023-10-30T13:25:16.187191+0000","last_clean":"2023-10-30T13:25:16.187191+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T13:25:16.187191+0000","last_undegraded":"2023-10-30T13:25:16.187191+0000","last_fullsized":"2023-10-30T13:25:16.187191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1511,"log_dups_size":0,"ondisk_log_size":1511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:06.796994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075712790000000002,"stat_sum":{"num_bytes":23568118,"num_objects":625,"num_object_clones":283,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":230,"num_read":141,"num_read_kb":34199,"num_write":1299,"num_write_kb":24762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"84'1725","reported_seq":3645,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.138485+0000","last_change":"2023-10-30T12:37:07.181114+0000","last_active":"2023-10-30T13:25:16.138485+0000","last_peered":"2023-10-30T13:25:16.138485+0000","last_clean":"2023-10-30T13:25:16.138485+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T13:25:16.138485+0000","last_undegraded":"2023-10-30T13:25:16.138485+0000","last_fullsized":"2023-10-30T13:25:16.138485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1725,"log_dups_size":0,"ondisk_log_size":1725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:57:46.353792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007366493,"stat_sum":{"num_bytes":24142540,"num_objects":721,"num_object_clones":385,"num_object_copies":1442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":231,"num_read":157,"num_read_kb":24833,"num_write":1452,"num_write_kb":33857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"85'1503","reported_seq":3540,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.201400+0000","last_change":"2023-10-30T12:37:07.114858+0000","last_active":"2023-10-30T13:25:16.201400+0000","last_peered":"2023-10-30T13:25:16.201400+0000","last_clean":"2023-10-30T13:25:16.201400+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T13:25:16.201400+0000","last_undegraded":"2023-10-30T13:25:16.201400+0000","last_fullsized":"2023-10-30T13:25:16.201400+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1503,"log_dups_size":0,"ondisk_log_size":1503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:18:13.789863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006759035,"stat_sum":{"num_bytes":14254037,"num_objects":596,"num_object_clones":276,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":205,"num_read":165,"num_read_kb":21722,"num_write":1326,"num_write_kb":24306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"77'1568","reported_seq":3567,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.278066+0000","last_change":"2023-10-30T12:37:07.112609+0000","last_active":"2023-10-30T13:24:57.278066+0000","last_peered":"2023-10-30T13:24:57.278066+0000","last_clean":"2023-10-30T13:24:57.278066+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T13:24:57.278066+0000","last_undegraded":"2023-10-30T13:24:57.278066+0000","last_fullsized":"2023-10-30T13:24:57.278066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1568,"log_dups_size":0,"ondisk_log_size":1568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:43.640916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057386779999999997,"stat_sum":{"num_bytes":10370740,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":213,"num_read":131,"num_read_kb":5845,"num_write":1381,"num_write_kb":12595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"77'1624","reported_seq":3604,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:46.371670+0000","last_change":"2023-10-30T12:37:07.187257+0000","last_active":"2023-10-30T13:24:46.371670+0000","last_peered":"2023-10-30T13:24:46.371670+0000","last_clean":"2023-10-30T13:24:46.371670+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T13:24:46.371670+0000","last_undegraded":"2023-10-30T13:24:46.371670+0000","last_fullsized":"2023-10-30T13:24:46.371670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:22.374872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013549705,"stat_sum":{"num_bytes":19248519,"num_objects":637,"num_object_clones":295,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":221,"num_read":128,"num_read_kb":21688,"num_write":1447,"num_write_kb":21442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"82'1574","reported_seq":3576,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.208932+0000","last_change":"2023-10-30T12:37:07.178284+0000","last_active":"2023-10-30T13:25:32.208932+0000","last_peered":"2023-10-30T13:25:32.208932+0000","last_clean":"2023-10-30T13:25:32.208932+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T13:25:32.208932+0000","last_undegraded":"2023-10-30T13:25:32.208932+0000","last_fullsized":"2023-10-30T13:25:32.208932+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1574,"log_dups_size":0,"ondisk_log_size":1574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:39:06.662733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075608309999999996,"stat_sum":{"num_bytes":14269019,"num_objects":619,"num_object_clones":290,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":233,"num_read":143,"num_read_kb":25965,"num_write":1377,"num_write_kb":20442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"85'1605","reported_seq":3590,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.146156+0000","last_change":"2023-10-30T12:37:07.182349+0000","last_active":"2023-10-30T13:25:16.146156+0000","last_peered":"2023-10-30T13:25:16.146156+0000","last_clean":"2023-10-30T13:25:16.146156+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T13:25:16.146156+0000","last_undegraded":"2023-10-30T13:25:16.146156+0000","last_fullsized":"2023-10-30T13:25:16.146156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1605,"log_dups_size":0,"ondisk_log_size":1605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:47:59.298834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0099357109999999998,"stat_sum":{"num_bytes":14793776,"num_objects":642,"num_object_clones":304,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":242,"num_read":140,"num_read_kb":30098,"num_write":1386,"num_write_kb":20714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"82'1444","reported_seq":3467,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.136464+0000","last_change":"2023-10-30T12:37:07.182572+0000","last_active":"2023-10-30T13:25:16.136464+0000","last_peered":"2023-10-30T13:25:16.136464+0000","last_clean":"2023-10-30T13:25:16.136464+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T13:25:16.136464+0000","last_undegraded":"2023-10-30T13:25:16.136464+0000","last_fullsized":"2023-10-30T13:25:16.136464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1444,"log_dups_size":0,"ondisk_log_size":1444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:41.547635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065893760000000001,"stat_sum":{"num_bytes":15508852,"num_objects":562,"num_object_clones":263,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":216,"num_read":137,"num_read_kb":10064,"num_write":1244,"num_write_kb":17198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"82'1576","reported_seq":3678,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:26.207237+0000","last_change":"2023-10-30T12:37:07.182668+0000","last_active":"2023-10-30T13:25:26.207237+0000","last_peered":"2023-10-30T13:25:26.207237+0000","last_clean":"2023-10-30T13:25:26.207237+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T13:25:26.207237+0000","last_undegraded":"2023-10-30T13:25:26.207237+0000","last_fullsized":"2023-10-30T13:25:26.207237+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1576,"log_dups_size":0,"ondisk_log_size":1576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:03.035306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010472766,"stat_sum":{"num_bytes":15695810,"num_objects":614,"num_object_clones":281,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":221,"num_read":233,"num_read_kb":17180,"num_write":1382,"num_write_kb":23481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"77'1569","reported_seq":3573,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.219508+0000","last_change":"2023-10-30T12:37:07.181518+0000","last_active":"2023-10-30T13:25:16.219508+0000","last_peered":"2023-10-30T13:25:16.219508+0000","last_clean":"2023-10-30T13:25:16.219508+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T13:25:16.219508+0000","last_undegraded":"2023-10-30T13:25:16.219508+0000","last_fullsized":"2023-10-30T13:25:16.219508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:12:17.666786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075455599999999998,"stat_sum":{"num_bytes":5586810,"num_objects":614,"num_object_clones":286,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":230,"num_read":140,"num_read_kb":5846,"num_write":1359,"num_write_kb":15885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"85'1523","reported_seq":3641,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.245123+0000","last_change":"2023-10-30T12:37:07.113494+0000","last_active":"2023-10-30T13:25:16.245123+0000","last_peered":"2023-10-30T13:25:16.245123+0000","last_clean":"2023-10-30T13:25:16.245123+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T13:25:16.245123+0000","last_undegraded":"2023-10-30T13:25:16.245123+0000","last_fullsized":"2023-10-30T13:25:16.245123+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1523,"log_dups_size":0,"ondisk_log_size":1523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:16.038384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076434509999999999,"stat_sum":{"num_bytes":23023646,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":222,"num_read":249,"num_read_kb":26208,"num_write":1319,"num_write_kb":28338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"84'1671","reported_seq":3654,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.169370+0000","last_change":"2023-10-30T12:37:07.102791+0000","last_active":"2023-10-30T13:24:56.169370+0000","last_peered":"2023-10-30T13:24:56.169370+0000","last_clean":"2023-10-30T13:24:56.169370+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T13:24:56.169370+0000","last_undegraded":"2023-10-30T13:24:56.169370+0000","last_fullsized":"2023-10-30T13:24:56.169370+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1671,"log_dups_size":0,"ondisk_log_size":1671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:38:00.347865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016105379999999999,"stat_sum":{"num_bytes":11166034,"num_objects":653,"num_object_clones":306,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":233,"num_read":139,"num_read_kb":9627,"num_write":1472,"num_write_kb":18195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"77'1520","reported_seq":3539,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288413+0000","last_change":"2023-10-30T12:37:07.188206+0000","last_active":"2023-10-30T13:24:16.288413+0000","last_peered":"2023-10-30T13:24:16.288413+0000","last_clean":"2023-10-30T13:24:16.288413+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T13:24:16.288413+0000","last_undegraded":"2023-10-30T13:24:16.288413+0000","last_fullsized":"2023-10-30T13:24:16.288413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:21.306614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014064241,"stat_sum":{"num_bytes":10716893,"num_objects":582,"num_object_clones":264,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":203,"num_read":135,"num_read_kb":13340,"num_write":1353,"num_write_kb":16108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"85'1551","reported_seq":3554,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:23.080116+0000","last_change":"2023-10-30T12:37:07.179785+0000","last_active":"2023-10-30T13:25:23.080116+0000","last_peered":"2023-10-30T13:25:23.080116+0000","last_clean":"2023-10-30T13:25:23.080116+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T13:25:23.080116+0000","last_undegraded":"2023-10-30T13:25:23.080116+0000","last_fullsized":"2023-10-30T13:25:23.080116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:49:28.717141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088476170000000003,"stat_sum":{"num_bytes":10233195,"num_objects":597,"num_object_clones":275,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":203,"num_read":127,"num_read_kb":1819,"num_write":1383,"num_write_kb":16400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"82'1465","reported_seq":3479,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.190427+0000","last_change":"2023-10-30T12:37:07.188606+0000","last_active":"2023-10-30T13:25:16.190427+0000","last_peered":"2023-10-30T13:25:16.190427+0000","last_clean":"2023-10-30T13:25:16.190427+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T13:25:16.190427+0000","last_undegraded":"2023-10-30T13:25:16.190427+0000","last_fullsized":"2023-10-30T13:25:16.190427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:30.044315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014279337,"stat_sum":{"num_bytes":22085088,"num_objects":582,"num_object_clones":270,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":216,"num_read":135,"num_read_kb":28666,"num_write":1258,"num_write_kb":25992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"77'1609","reported_seq":3602,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.171560+0000","last_change":"2023-10-30T12:37:07.177071+0000","last_active":"2023-10-30T13:25:16.171560+0000","last_peered":"2023-10-30T13:25:16.171560+0000","last_clean":"2023-10-30T13:25:16.171560+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T13:25:16.171560+0000","last_undegraded":"2023-10-30T13:25:16.171560+0000","last_fullsized":"2023-10-30T13:25:16.171560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1609,"log_dups_size":0,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:19.563113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004006122,"stat_sum":{"num_bytes":15408826,"num_objects":598,"num_object_clones":277,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":213,"num_read":119,"num_read_kb":9406,"num_write":1408,"num_write_kb":18485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"82'1549","reported_seq":3552,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.237486+0000","last_change":"2023-10-30T12:37:07.115775+0000","last_active":"2023-10-30T13:25:32.237486+0000","last_peered":"2023-10-30T13:25:32.237486+0000","last_clean":"2023-10-30T13:25:32.237486+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T13:25:32.237486+0000","last_undegraded":"2023-10-30T13:25:32.237486+0000","last_fullsized":"2023-10-30T13:25:32.237486+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1549,"log_dups_size":0,"ondisk_log_size":1549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:59.732034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059072409999999997,"stat_sum":{"num_bytes":25755688,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":240,"num_read":143,"num_read_kb":23034,"num_write":1332,"num_write_kb":28963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"85'1485","reported_seq":3466,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:07.280157+0000","last_change":"2023-10-30T12:37:07.113316+0000","last_active":"2023-10-30T13:25:07.280157+0000","last_peered":"2023-10-30T13:25:07.280157+0000","last_clean":"2023-10-30T13:25:07.280157+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T13:25:07.280157+0000","last_undegraded":"2023-10-30T13:25:07.280157+0000","last_fullsized":"2023-10-30T13:25:07.280157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1485,"log_dups_size":0,"ondisk_log_size":1485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:19:54.028619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062401690000000003,"stat_sum":{"num_bytes":11021781,"num_objects":617,"num_object_clones":286,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":226,"num_read":116,"num_read_kb":10954,"num_write":1301,"num_write_kb":16114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"77'1620","reported_seq":3589,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.156447+0000","last_change":"2023-10-30T12:37:07.189557+0000","last_active":"2023-10-30T13:25:16.156447+0000","last_peered":"2023-10-30T13:25:16.156447+0000","last_clean":"2023-10-30T13:25:16.156447+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T13:25:16.156447+0000","last_undegraded":"2023-10-30T13:25:16.156447+0000","last_fullsized":"2023-10-30T13:25:16.156447+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:17:47.861374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014933905000000001,"stat_sum":{"num_bytes":11161783,"num_objects":653,"num_object_clones":309,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":240,"num_read":128,"num_read_kb":9667,"num_write":1416,"num_write_kb":13402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"82'1476","reported_seq":3519,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.195574+0000","last_change":"2023-10-30T12:37:07.180055+0000","last_active":"2023-10-30T13:25:16.195574+0000","last_peered":"2023-10-30T13:25:16.195574+0000","last_clean":"2023-10-30T13:25:16.195574+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T13:25:16.195574+0000","last_undegraded":"2023-10-30T13:25:16.195574+0000","last_fullsized":"2023-10-30T13:25:16.195574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:02.149498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0090148399999999997,"stat_sum":{"num_bytes":11081748,"num_objects":549,"num_object_clones":251,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":201,"num_read":142,"num_read_kb":9474,"num_write":1288,"num_write_kb":13652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"84'1713","reported_seq":3703,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.230251+0000","last_change":"2023-10-30T12:37:07.181770+0000","last_active":"2023-10-30T13:25:16.230251+0000","last_peered":"2023-10-30T13:25:16.230251+0000","last_clean":"2023-10-30T13:25:16.230251+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T13:25:16.230251+0000","last_undegraded":"2023-10-30T13:25:16.230251+0000","last_fullsized":"2023-10-30T13:25:16.230251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1713,"log_dups_size":0,"ondisk_log_size":1713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:33.319141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066657749999999997,"stat_sum":{"num_bytes":20125251,"num_objects":693,"num_object_clones":330,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":267,"num_read":168,"num_read_kb":25254,"num_write":1457,"num_write_kb":26871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"82'1548","reported_seq":3554,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.216318+0000","last_change":"2023-10-30T12:37:07.176070+0000","last_active":"2023-10-30T13:25:16.216318+0000","last_peered":"2023-10-30T13:25:16.216318+0000","last_clean":"2023-10-30T13:25:16.216318+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T13:25:16.216318+0000","last_undegraded":"2023-10-30T13:25:16.216318+0000","last_fullsized":"2023-10-30T13:25:16.216318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:28:06.978130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005419129,"stat_sum":{"num_bytes":18031744,"num_objects":604,"num_object_clones":282,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":228,"num_read":136,"num_read_kb":25964,"num_write":1344,"num_write_kb":19625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"85'1548","reported_seq":3561,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.144906+0000","last_change":"2023-10-30T12:37:07.102583+0000","last_active":"2023-10-30T13:24:56.144906+0000","last_peered":"2023-10-30T13:24:56.144906+0000","last_clean":"2023-10-30T13:24:56.144906+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T13:24:56.144906+0000","last_undegraded":"2023-10-30T13:24:56.144906+0000","last_fullsized":"2023-10-30T13:24:56.144906+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:32.507814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013742470000000001,"stat_sum":{"num_bytes":22381695,"num_objects":607,"num_object_clones":275,"num_object_copies":1214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":227,"num_read":141,"num_read_kb":33421,"num_write":1348,"num_write_kb":23422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"85'991","reported_seq":1403,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:53.837590+0000","last_change":"2023-10-30T12:28:44.074842+0000","last_active":"2023-10-30T13:24:53.837590+0000","last_peered":"2023-10-30T13:24:53.837590+0000","last_clean":"2023-10-30T13:24:53.837590+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T13:24:53.837590+0000","last_undegraded":"2023-10-30T13:24:53.837590+0000","last_fullsized":"2023-10-30T13:24:53.837590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":991,"log_dups_size":0,"ondisk_log_size":991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:01.428785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":570,"num_read_kb":319827,"num_write":1218,"num_write_kb":80521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"77'1561","reported_seq":3569,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.084148+0000","last_change":"2023-10-30T12:37:07.114520+0000","last_active":"2023-10-30T13:24:56.084148+0000","last_peered":"2023-10-30T13:24:56.084148+0000","last_clean":"2023-10-30T13:24:56.084148+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T13:24:56.084148+0000","last_undegraded":"2023-10-30T13:24:56.084148+0000","last_fullsized":"2023-10-30T13:24:56.084148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1561,"log_dups_size":0,"ondisk_log_size":1561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:06:30.845566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086018069999999995,"stat_sum":{"num_bytes":19940185,"num_objects":616,"num_object_clones":283,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":213,"num_read":144,"num_read_kb":22076,"num_write":1398,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1795","reported_seq":2375,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:52.266854+0000","last_change":"2023-10-30T12:28:44.060621+0000","last_active":"2023-10-30T13:24:52.266854+0000","last_peered":"2023-10-30T13:24:52.266854+0000","last_clean":"2023-10-30T13:24:52.266854+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T13:24:52.266854+0000","last_undegraded":"2023-10-30T13:24:52.266854+0000","last_fullsized":"2023-10-30T13:24:52.266854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1795,"log_dups_size":0,"ondisk_log_size":1795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:53.741238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17028825,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":761,"num_read_kb":396040,"num_write":1931,"num_write_kb":84639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"85'1569","reported_seq":2387,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:27.320748+0000","last_change":"2023-10-30T12:28:44.060171+0000","last_active":"2023-10-30T13:25:27.320748+0000","last_peered":"2023-10-30T13:25:27.320748+0000","last_clean":"2023-10-30T13:25:27.320748+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T13:25:27.320748+0000","last_undegraded":"2023-10-30T13:25:27.320748+0000","last_fullsized":"2023-10-30T13:25:27.320748+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:18.181418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182760,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":1117,"num_read_kb":268002,"num_write":1799,"num_write_kb":63121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"77'1801","reported_seq":3808,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.165978+0000","last_change":"2023-10-30T12:37:07.191924+0000","last_active":"2023-10-30T13:25:16.165978+0000","last_peered":"2023-10-30T13:25:16.165978+0000","last_clean":"2023-10-30T13:25:16.165978+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T13:25:16.165978+0000","last_undegraded":"2023-10-30T13:25:16.165978+0000","last_fullsized":"2023-10-30T13:25:16.165978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1801,"log_dups_size":0,"ondisk_log_size":1801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:27.781535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016143469000000001,"stat_sum":{"num_bytes":14540052,"num_objects":709,"num_object_clones":329,"num_object_copies":1418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":254,"num_read":190,"num_read_kb":6718,"num_write":1583,"num_write_kb":26382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"85'930","reported_seq":1332,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:52.371111+0000","last_change":"2023-10-30T12:28:44.058798+0000","last_active":"2023-10-30T13:24:52.371111+0000","last_peered":"2023-10-30T13:24:52.371111+0000","last_clean":"2023-10-30T13:24:52.371111+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T13:24:52.371111+0000","last_undegraded":"2023-10-30T13:24:52.371111+0000","last_fullsized":"2023-10-30T13:24:52.371111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":930,"log_dups_size":0,"ondisk_log_size":930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:08.643080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":542,"num_read_kb":359029,"num_write":1093,"num_write_kb":88246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"77'1524","reported_seq":3543,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.097691+0000","last_change":"2023-10-30T12:37:07.114493+0000","last_active":"2023-10-30T13:24:56.097691+0000","last_peered":"2023-10-30T13:24:56.097691+0000","last_clean":"2023-10-30T13:24:56.097691+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T13:24:56.097691+0000","last_undegraded":"2023-10-30T13:24:56.097691+0000","last_fullsized":"2023-10-30T13:24:56.097691+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:55:36.189245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083101389999999994,"stat_sum":{"num_bytes":18904879,"num_objects":589,"num_object_clones":276,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":211,"num_read":147,"num_read_kb":20021,"num_write":1340,"num_write_kb":25244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"85'736","reported_seq":1136,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:58.054623+0000","last_change":"2023-10-30T12:28:44.063439+0000","last_active":"2023-10-30T13:24:58.054623+0000","last_peered":"2023-10-30T13:24:58.054623+0000","last_clean":"2023-10-30T13:24:58.054623+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T13:24:58.054623+0000","last_undegraded":"2023-10-30T13:24:58.054623+0000","last_fullsized":"2023-10-30T13:24:58.054623+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":736,"log_dups_size":0,"ondisk_log_size":736,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:08.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":616,"num_read_kb":251423,"num_write":889,"num_write_kb":55053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"77'1518","reported_seq":3533,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.444813+0000","last_change":"2023-10-30T12:37:07.177520+0000","last_active":"2023-10-30T13:25:10.444813+0000","last_peered":"2023-10-30T13:25:10.444813+0000","last_clean":"2023-10-30T13:25:10.444813+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T13:25:10.444813+0000","last_undegraded":"2023-10-30T13:25:10.444813+0000","last_fullsized":"2023-10-30T13:25:10.444813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:46:22.808814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043045619999999996,"stat_sum":{"num_bytes":10134301,"num_objects":580,"num_object_clones":271,"num_object_copies":1160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":580,"num_whiteouts":219,"num_read":138,"num_read_kb":8986,"num_write":1318,"num_write_kb":11276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"85'1247","reported_seq":1685,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:24.345143+0000","last_change":"2023-10-30T12:28:44.052641+0000","last_active":"2023-10-30T13:24:24.345143+0000","last_peered":"2023-10-30T13:24:24.345143+0000","last_clean":"2023-10-30T13:24:24.345143+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T13:24:24.345143+0000","last_undegraded":"2023-10-30T13:24:24.345143+0000","last_fullsized":"2023-10-30T13:24:24.345143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1247,"log_dups_size":0,"ondisk_log_size":1247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":659,"num_read_kb":339384,"num_write":1492,"num_write_kb":80693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"82'1603","reported_seq":3592,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.115007+0000","last_change":"2023-10-30T12:37:07.191144+0000","last_active":"2023-10-30T13:24:56.115007+0000","last_peered":"2023-10-30T13:24:56.115007+0000","last_clean":"2023-10-30T13:24:56.115007+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T13:24:56.115007+0000","last_undegraded":"2023-10-30T13:24:56.115007+0000","last_fullsized":"2023-10-30T13:24:56.115007+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1603,"log_dups_size":0,"ondisk_log_size":1603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:46.268962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015702779,"stat_sum":{"num_bytes":15773172,"num_objects":620,"num_object_clones":292,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":204,"num_read":135,"num_read_kb":17351,"num_write":1431,"num_write_kb":17077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"85'1224","reported_seq":1717,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:35.202116+0000","last_change":"2023-10-30T12:28:44.075783+0000","last_active":"2023-10-30T13:25:35.202116+0000","last_peered":"2023-10-30T13:25:35.202116+0000","last_clean":"2023-10-30T13:25:35.202116+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T13:25:35.202116+0000","last_undegraded":"2023-10-30T13:25:35.202116+0000","last_fullsized":"2023-10-30T13:25:35.202116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1224,"log_dups_size":0,"ondisk_log_size":1224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:01.094929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19774519,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":862,"num_read_kb":221068,"num_write":1454,"num_write_kb":51536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"82'1515","reported_seq":3524,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.077418+0000","last_change":"2023-10-30T12:37:07.178872+0000","last_active":"2023-10-30T13:25:03.077418+0000","last_peered":"2023-10-30T13:25:03.077418+0000","last_clean":"2023-10-30T13:25:03.077418+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T13:25:03.077418+0000","last_undegraded":"2023-10-30T13:25:03.077418+0000","last_fullsized":"2023-10-30T13:25:03.077418+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:03.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080610860000000003,"stat_sum":{"num_bytes":22077043,"num_objects":612,"num_object_clones":281,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":230,"num_read":141,"num_read_kb":18856,"num_write":1294,"num_write_kb":28337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"85'1331","reported_seq":1877,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:12.181476+0000","last_change":"2023-10-30T12:28:44.063367+0000","last_active":"2023-10-30T13:25:12.181476+0000","last_peered":"2023-10-30T13:25:12.181476+0000","last_clean":"2023-10-30T13:25:12.181476+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T13:25:12.181476+0000","last_undegraded":"2023-10-30T13:25:12.181476+0000","last_fullsized":"2023-10-30T13:25:12.181476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1331,"log_dups_size":0,"ondisk_log_size":1331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:45.585121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":865,"num_read_kb":420163,"num_write":1566,"num_write_kb":89397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"82'1535","reported_seq":3533,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.355487+0000","last_change":"2023-10-30T12:37:07.182869+0000","last_active":"2023-10-30T13:24:48.355487+0000","last_peered":"2023-10-30T13:24:48.355487+0000","last_clean":"2023-10-30T13:24:48.355487+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T13:24:48.355487+0000","last_undegraded":"2023-10-30T13:24:48.355487+0000","last_fullsized":"2023-10-30T13:24:48.355487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1535,"log_dups_size":0,"ondisk_log_size":1535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:35.734555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.008640567,"stat_sum":{"num_bytes":23022369,"num_objects":606,"num_object_clones":279,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":220,"num_read":129,"num_read_kb":18941,"num_write":1340,"num_write_kb":24878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"85'1098","reported_seq":1558,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.290468+0000","last_change":"2023-10-30T12:28:44.065545+0000","last_active":"2023-10-30T13:24:47.290468+0000","last_peered":"2023-10-30T13:24:47.290468+0000","last_clean":"2023-10-30T13:24:47.290468+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T13:24:47.290468+0000","last_undegraded":"2023-10-30T13:24:47.290468+0000","last_fullsized":"2023-10-30T13:24:47.290468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1098,"log_dups_size":0,"ondisk_log_size":1098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:09.900189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554466,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":701,"num_read_kb":318483,"num_write":1303,"num_write_kb":68223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"84'1584","reported_seq":3584,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.315736+0000","last_change":"2023-10-30T12:37:07.115595+0000","last_active":"2023-10-30T13:24:48.315736+0000","last_peered":"2023-10-30T13:24:48.315736+0000","last_clean":"2023-10-30T13:24:48.315736+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T13:24:48.315736+0000","last_undegraded":"2023-10-30T13:24:48.315736+0000","last_fullsized":"2023-10-30T13:24:48.315736+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:20:44.527701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073009930000000004,"stat_sum":{"num_bytes":10084838,"num_objects":613,"num_object_clones":288,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":217,"num_read":139,"num_read_kb":5686,"num_write":1383,"num_write_kb":20147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'676","reported_seq":1068,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.649328+0000","last_change":"2023-10-30T12:28:44.061922+0000","last_active":"2023-10-30T13:24:57.649328+0000","last_peered":"2023-10-30T13:24:57.649328+0000","last_clean":"2023-10-30T13:24:57.649328+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T13:24:57.649328+0000","last_undegraded":"2023-10-30T13:24:57.649328+0000","last_fullsized":"2023-10-30T13:24:57.649328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:34.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":600,"num_read_kb":208389,"num_write":899,"num_write_kb":51587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"82'1642","reported_seq":3652,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.151917+0000","last_change":"2023-10-30T12:37:07.180453+0000","last_active":"2023-10-30T13:25:16.151917+0000","last_peered":"2023-10-30T13:25:16.151917+0000","last_clean":"2023-10-30T13:25:16.151917+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T13:25:16.151917+0000","last_undegraded":"2023-10-30T13:25:16.151917+0000","last_fullsized":"2023-10-30T13:25:16.151917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:20.518544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091258150000000007,"stat_sum":{"num_bytes":19392609,"num_objects":655,"num_object_clones":299,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":233,"num_read":161,"num_read_kb":7475,"num_write":1440,"num_write_kb":20643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"85'1371","reported_seq":1755,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.074769+0000","last_change":"2023-10-30T12:28:44.059784+0000","last_active":"2023-10-30T13:24:48.074769+0000","last_peered":"2023-10-30T13:24:48.074769+0000","last_clean":"2023-10-30T13:24:48.074769+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T13:24:48.074769+0000","last_undegraded":"2023-10-30T13:24:48.074769+0000","last_fullsized":"2023-10-30T13:24:48.074769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1371,"log_dups_size":0,"ondisk_log_size":1371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:11.468221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":539,"num_read_kb":266844,"num_write":1512,"num_write_kb":63647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"77'1481","reported_seq":3489,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.310426+0000","last_change":"2023-10-30T12:37:07.102525+0000","last_active":"2023-10-30T13:24:48.310426+0000","last_peered":"2023-10-30T13:24:48.310426+0000","last_clean":"2023-10-30T13:24:48.310426+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T13:24:48.310426+0000","last_undegraded":"2023-10-30T13:24:48.310426+0000","last_fullsized":"2023-10-30T13:24:48.310426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:22.814663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001646376,"stat_sum":{"num_bytes":13778361,"num_objects":590,"num_object_clones":278,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":227,"num_read":137,"num_read_kb":5723,"num_write":1258,"num_write_kb":14976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"85'970","reported_seq":1455,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.407319+0000","last_change":"2023-10-30T12:28:44.065885+0000","last_active":"2023-10-30T13:24:47.407319+0000","last_peered":"2023-10-30T13:24:47.407319+0000","last_clean":"2023-10-30T13:24:47.407319+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T13:24:47.407319+0000","last_undegraded":"2023-10-30T13:24:47.407319+0000","last_fullsized":"2023-10-30T13:24:47.407319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:24:26.313595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554544,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":863,"num_read_kb":272228,"num_write":1253,"num_write_kb":56723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"82'1616","reported_seq":3636,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.437909+0000","last_change":"2023-10-30T12:37:07.181542+0000","last_active":"2023-10-30T13:25:10.437909+0000","last_peered":"2023-10-30T13:25:10.437909+0000","last_clean":"2023-10-30T13:25:10.437909+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T13:25:10.437909+0000","last_undegraded":"2023-10-30T13:25:10.437909+0000","last_fullsized":"2023-10-30T13:25:10.437909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1616,"log_dups_size":0,"ondisk_log_size":1616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:16:47.953145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098326820000000006,"stat_sum":{"num_bytes":10901701,"num_objects":622,"num_object_clones":292,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":232,"num_read":163,"num_read_kb":14238,"num_write":1406,"num_write_kb":21652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"85'655","reported_seq":1014,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:04.285953+0000","last_change":"2023-10-30T12:28:44.080011+0000","last_active":"2023-10-30T13:25:04.285953+0000","last_peered":"2023-10-30T13:25:04.285953+0000","last_clean":"2023-10-30T13:25:04.285953+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T13:25:04.285953+0000","last_undegraded":"2023-10-30T13:25:04.285953+0000","last_fullsized":"2023-10-30T13:25:04.285953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":443,"num_read_kb":261670,"num_write":858,"num_write_kb":58986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"77'1573","reported_seq":3538,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.065502+0000","last_change":"2023-10-30T12:37:07.177898+0000","last_active":"2023-10-30T13:25:03.065502+0000","last_peered":"2023-10-30T13:25:03.065502+0000","last_clean":"2023-10-30T13:25:03.065502+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T13:25:03.065502+0000","last_undegraded":"2023-10-30T13:25:03.065502+0000","last_fullsized":"2023-10-30T13:25:03.065502+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:52.251046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004602437,"stat_sum":{"num_bytes":20428330,"num_objects":647,"num_object_clones":304,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":239,"num_read":119,"num_read_kb":13097,"num_write":1360,"num_write_kb":26395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"85'717","reported_seq":1166,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.413162+0000","last_change":"2023-10-30T12:28:44.076352+0000","last_active":"2023-10-30T13:24:48.413162+0000","last_peered":"2023-10-30T13:24:48.413162+0000","last_clean":"2023-10-30T13:24:48.413162+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T13:24:48.413162+0000","last_undegraded":"2023-10-30T13:24:48.413162+0000","last_fullsized":"2023-10-30T13:24:48.413162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":635,"num_read_kb":434171,"num_write":906,"num_write_kb":88121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"77'1575","reported_seq":3560,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.221074+0000","last_change":"2023-10-30T12:37:07.181502+0000","last_active":"2023-10-30T13:25:32.221074+0000","last_peered":"2023-10-30T13:25:32.221074+0000","last_clean":"2023-10-30T13:25:32.221074+0000","last_became_active":"2023-10-30T12:29:27.340504+0000","last_became_peered":"2023-10-30T12:29:27.340504+0000","last_unstale":"2023-10-30T13:25:32.221074+0000","last_undegraded":"2023-10-30T13:25:32.221074+0000","last_fullsized":"2023-10-30T13:25:32.221074+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1575,"log_dups_size":0,"ondisk_log_size":1575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:48:15.263466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00653923,"stat_sum":{"num_bytes":23737065,"num_objects":641,"num_object_clones":299,"num_object_copies":1282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":641,"num_whiteouts":238,"num_read":138,"num_read_kb":28255,"num_write":1351,"num_write_kb":33550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'716","reported_seq":1095,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.276257+0000","last_change":"2023-10-30T12:28:44.075684+0000","last_active":"2023-10-30T13:24:47.276257+0000","last_peered":"2023-10-30T13:24:47.276257+0000","last_clean":"2023-10-30T13:24:47.276257+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T13:24:47.276257+0000","last_undegraded":"2023-10-30T13:24:47.276257+0000","last_fullsized":"2023-10-30T13:24:47.276257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:07.375795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364369,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":493,"num_read_kb":320778,"num_write":891,"num_write_kb":71265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"77'1599","reported_seq":3612,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.080865+0000","last_change":"2023-10-30T12:37:07.181841+0000","last_active":"2023-10-30T13:25:03.080865+0000","last_peered":"2023-10-30T13:25:03.080865+0000","last_clean":"2023-10-30T13:25:03.080865+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T13:25:03.080865+0000","last_undegraded":"2023-10-30T13:25:03.080865+0000","last_fullsized":"2023-10-30T13:25:03.080865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1599,"log_dups_size":0,"ondisk_log_size":1599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:35:15.227222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076673150000000001,"stat_sum":{"num_bytes":18256447,"num_objects":611,"num_object_clones":286,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":229,"num_read":149,"num_read_kb":24381,"num_write":1383,"num_write_kb":20195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"85'607","reported_seq":959,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.278226+0000","last_change":"2023-10-30T12:28:44.060686+0000","last_active":"2023-10-30T13:24:47.278226+0000","last_peered":"2023-10-30T13:24:47.278226+0000","last_clean":"2023-10-30T13:24:47.278226+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T13:24:47.278226+0000","last_undegraded":"2023-10-30T13:24:47.278226+0000","last_fullsized":"2023-10-30T13:24:47.278226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:23.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360710,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":462,"num_read_kb":342669,"num_write":689,"num_write_kb":66904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"82'1520","reported_seq":3502,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.077446+0000","last_change":"2023-10-30T12:37:07.182282+0000","last_active":"2023-10-30T13:25:03.077446+0000","last_peered":"2023-10-30T13:25:03.077446+0000","last_clean":"2023-10-30T13:25:03.077446+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T13:25:03.077446+0000","last_undegraded":"2023-10-30T13:25:03.077446+0000","last_fullsized":"2023-10-30T13:25:03.077446+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:14.913475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010286791999999999,"stat_sum":{"num_bytes":26944901,"num_objects":612,"num_object_clones":287,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":236,"num_read":119,"num_read_kb":34683,"num_write":1299,"num_write_kb":33129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":256,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288773+0000","last_change":"2023-10-30T12:28:33.433824+0000","last_active":"2023-10-30T13:24:16.288773+0000","last_peered":"2023-10-30T13:24:16.288773+0000","last_clean":"2023-10-30T13:24:16.288773+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T13:24:16.288773+0000","last_undegraded":"2023-10-30T13:24:16.288773+0000","last_fullsized":"2023-10-30T13:24:16.288773+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_clean_scrub_stamp":"2023-10-30T12:28:31.408255+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:17.624179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'1318","reported_seq":1818,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.269121+0000","last_change":"2023-10-30T12:28:44.059051+0000","last_active":"2023-10-30T13:24:47.269121+0000","last_peered":"2023-10-30T13:24:47.269121+0000","last_clean":"2023-10-30T13:24:47.269121+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T13:24:47.269121+0000","last_undegraded":"2023-10-30T13:24:47.269121+0000","last_fullsized":"2023-10-30T13:24:47.269121+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1318,"log_dups_size":0,"ondisk_log_size":1318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:54.810636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165890,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":877,"num_read_kb":231219,"num_write":1489,"num_write_kb":63137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"85'1574","reported_seq":3582,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.122820+0000","last_change":"2023-10-30T12:37:07.116603+0000","last_active":"2023-10-30T13:25:02.122820+0000","last_peered":"2023-10-30T13:25:02.122820+0000","last_clean":"2023-10-30T13:25:02.122820+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T13:25:02.122820+0000","last_undegraded":"2023-10-30T13:25:02.122820+0000","last_fullsized":"2023-10-30T13:25:02.122820+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1574,"log_dups_size":0,"ondisk_log_size":1574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:01:30.768741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068470270000000003,"stat_sum":{"num_bytes":8097312,"num_objects":626,"num_object_clones":290,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":225,"num_read":146,"num_read_kb":2079,"num_write":1425,"num_write_kb":13192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"85'2192","reported_seq":2563,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.650111+0000","last_change":"2023-10-30T12:28:44.080378+0000","last_active":"2023-10-30T13:24:57.650111+0000","last_peered":"2023-10-30T13:24:57.650111+0000","last_clean":"2023-10-30T13:24:57.650111+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T13:24:57.650111+0000","last_undegraded":"2023-10-30T13:24:57.650111+0000","last_fullsized":"2023-10-30T13:24:57.650111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2192,"log_dups_size":0,"ondisk_log_size":2192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:22.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":516,"num_read_kb":283293,"num_write":2369,"num_write_kb":72030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"77'1465","reported_seq":3487,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:15.952966+0000","last_change":"2023-10-30T12:37:07.112121+0000","last_active":"2023-10-30T13:24:15.952966+0000","last_peered":"2023-10-30T13:24:15.952966+0000","last_clean":"2023-10-30T13:24:15.952966+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T13:24:15.952966+0000","last_undegraded":"2023-10-30T13:24:15.952966+0000","last_fullsized":"2023-10-30T13:24:15.952966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:12:48.778193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057755250000000001,"stat_sum":{"num_bytes":14269585,"num_objects":570,"num_object_clones":262,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":207,"num_read":132,"num_read_kb":18022,"num_write":1264,"num_write_kb":17917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'1838","reported_seq":3123,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:09.242564+0000","last_change":"2023-10-30T12:28:44.082480+0000","last_active":"2023-10-30T13:25:09.242564+0000","last_peered":"2023-10-30T13:25:09.242564+0000","last_clean":"2023-10-30T13:25:09.242564+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T13:25:09.242564+0000","last_undegraded":"2023-10-30T13:25:09.242564+0000","last_fullsized":"2023-10-30T13:25:09.242564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1838,"log_dups_size":0,"ondisk_log_size":1838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:20.126782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49235352,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":1514,"num_read_kb":360508,"num_write":2035,"num_write_kb":92558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"77'1546","reported_seq":3555,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.373540+0000","last_change":"2023-10-30T12:37:07.190318+0000","last_active":"2023-10-30T13:24:56.373540+0000","last_peered":"2023-10-30T13:24:56.373540+0000","last_clean":"2023-10-30T13:24:56.373540+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T13:24:56.373540+0000","last_undegraded":"2023-10-30T13:24:56.373540+0000","last_fullsized":"2023-10-30T13:24:56.373540+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:19.397149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015331312,"stat_sum":{"num_bytes":28178512,"num_objects":629,"num_object_clones":288,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":231,"num_read":146,"num_read_kb":37712,"num_write":1347,"num_write_kb":29520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'1196","reported_seq":1605,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.648443+0000","last_change":"2023-10-30T12:28:44.059806+0000","last_active":"2023-10-30T13:24:57.648443+0000","last_peered":"2023-10-30T13:24:57.648443+0000","last_clean":"2023-10-30T13:24:57.648443+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T13:24:57.648443+0000","last_undegraded":"2023-10-30T13:24:57.648443+0000","last_fullsized":"2023-10-30T13:24:57.648443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1196,"log_dups_size":0,"ondisk_log_size":1196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:50.939946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":573,"num_read_kb":334248,"num_write":1338,"num_write_kb":75895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"85'1581","reported_seq":3581,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.228379+0000","last_change":"2023-10-30T12:37:07.178785+0000","last_active":"2023-10-30T13:25:16.228379+0000","last_peered":"2023-10-30T13:25:16.228379+0000","last_clean":"2023-10-30T13:25:16.228379+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T13:25:16.228379+0000","last_undegraded":"2023-10-30T13:25:16.228379+0000","last_fullsized":"2023-10-30T13:25:16.228379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1581,"log_dups_size":0,"ondisk_log_size":1581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:27.865544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051193749999999998,"stat_sum":{"num_bytes":23443395,"num_objects":627,"num_object_clones":293,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":236,"num_read":141,"num_read_kb":26943,"num_write":1376,"num_write_kb":26009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"85'1467","reported_seq":1927,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:01.374791+0000","last_change":"2023-10-30T12:28:44.074970+0000","last_active":"2023-10-30T13:25:01.374791+0000","last_peered":"2023-10-30T13:25:01.374791+0000","last_clean":"2023-10-30T13:25:01.374791+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T13:25:01.374791+0000","last_undegraded":"2023-10-30T13:25:01.374791+0000","last_fullsized":"2023-10-30T13:25:01.374791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1467,"log_dups_size":0,"ondisk_log_size":1467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:33.562703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":699,"num_read_kb":404466,"num_write":1620,"num_write_kb":92575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"82'1612","reported_seq":3615,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:18.067135+0000","last_change":"2023-10-30T12:37:07.176969+0000","last_active":"2023-10-30T13:25:18.067135+0000","last_peered":"2023-10-30T13:25:18.067135+0000","last_clean":"2023-10-30T13:25:18.067135+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T13:25:18.067135+0000","last_undegraded":"2023-10-30T13:25:18.067135+0000","last_fullsized":"2023-10-30T13:25:18.067135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1612,"log_dups_size":0,"ondisk_log_size":1612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:33:15.916870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039865179999999997,"stat_sum":{"num_bytes":21812744,"num_objects":637,"num_object_clones":298,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":235,"num_read":149,"num_read_kb":24572,"num_write":1399,"num_write_kb":31699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"85'1013","reported_seq":1422,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.646473+0000","last_change":"2023-10-30T12:28:44.063256+0000","last_active":"2023-10-30T13:24:57.646473+0000","last_peered":"2023-10-30T13:24:57.646473+0000","last_clean":"2023-10-30T13:24:57.646473+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T13:24:57.646473+0000","last_undegraded":"2023-10-30T13:24:57.646473+0000","last_fullsized":"2023-10-30T13:24:57.646473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1013,"log_dups_size":0,"ondisk_log_size":1013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:01:36.144966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":534,"num_read_kb":318720,"num_write":1164,"num_write_kb":75805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"77'1542","reported_seq":3565,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.430277+0000","last_change":"2023-10-30T12:37:07.114718+0000","last_active":"2023-10-30T13:25:10.430277+0000","last_peered":"2023-10-30T13:25:10.430277+0000","last_clean":"2023-10-30T13:25:10.430277+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T13:25:10.430277+0000","last_undegraded":"2023-10-30T13:25:10.430277+0000","last_fullsized":"2023-10-30T13:25:10.430277+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1542,"log_dups_size":0,"ondisk_log_size":1542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:05.944397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070372949999999998,"stat_sum":{"num_bytes":38977850,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":222,"num_read":154,"num_read_kb":43697,"num_write":1351,"num_write_kb":41194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1137","reported_seq":1571,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.180289+0000","last_change":"2023-10-30T12:28:44.055374+0000","last_active":"2023-10-30T13:25:02.180289+0000","last_peered":"2023-10-30T13:25:02.180289+0000","last_clean":"2023-10-30T13:25:02.180289+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T13:25:02.180289+0000","last_undegraded":"2023-10-30T13:25:02.180289+0000","last_fullsized":"2023-10-30T13:25:02.180289+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1137,"log_dups_size":0,"ondisk_log_size":1137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":644,"num_read_kb":249397,"num_write":1858,"num_write_kb":63729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"77'1449","reported_seq":3446,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:15.952034+0000","last_change":"2023-10-30T12:37:07.115578+0000","last_active":"2023-10-30T13:24:15.952034+0000","last_peered":"2023-10-30T13:24:15.952034+0000","last_clean":"2023-10-30T13:24:15.952034+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T13:24:15.952034+0000","last_undegraded":"2023-10-30T13:24:15.952034+0000","last_fullsized":"2023-10-30T13:24:15.952034+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1449,"log_dups_size":0,"ondisk_log_size":1449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:52:43.980959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074996079999999996,"stat_sum":{"num_bytes":14238343,"num_objects":603,"num_object_clones":280,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":229,"num_read":126,"num_read_kb":9383,"num_write":1239,"num_write_kb":19514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"85'489","reported_seq":838,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:08.078199+0000","last_change":"2023-10-30T12:28:44.058663+0000","last_active":"2023-10-30T13:25:08.078199+0000","last_peered":"2023-10-30T13:25:08.078199+0000","last_clean":"2023-10-30T13:25:08.078199+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T13:25:08.078199+0000","last_undegraded":"2023-10-30T13:25:08.078199+0000","last_fullsized":"2023-10-30T13:25:08.078199+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":489,"log_dups_size":0,"ondisk_log_size":489,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:42:22.279653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21146410,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":514,"num_read_kb":197148,"num_write":650,"num_write_kb":42825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"77'1649","reported_seq":3622,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.197589+0000","last_change":"2023-10-30T12:37:07.115477+0000","last_active":"2023-10-30T13:25:16.197589+0000","last_peered":"2023-10-30T13:25:16.197589+0000","last_clean":"2023-10-30T13:25:16.197589+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T13:25:16.197589+0000","last_undegraded":"2023-10-30T13:25:16.197589+0000","last_fullsized":"2023-10-30T13:25:16.197589+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1649,"log_dups_size":0,"ondisk_log_size":1649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:37:44.248368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085509929999999998,"stat_sum":{"num_bytes":14746405,"num_objects":647,"num_object_clones":344,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":203,"num_read":166,"num_read_kb":4091,"num_write":1369,"num_write_kb":17013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'870","reported_seq":1184,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.657578+0000","last_change":"2023-10-30T12:28:44.054338+0000","last_active":"2023-10-30T13:25:02.657578+0000","last_peered":"2023-10-30T13:25:02.657578+0000","last_clean":"2023-10-30T13:25:02.657578+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T13:25:02.657578+0000","last_undegraded":"2023-10-30T13:25:02.657578+0000","last_fullsized":"2023-10-30T13:25:02.657578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":870,"log_dups_size":0,"ondisk_log_size":870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:55:35.370652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":472,"num_read_kb":149352,"num_write":1030,"num_write_kb":38491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"77'1485","reported_seq":3500,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.168854+0000","last_change":"2023-10-30T12:37:07.181506+0000","last_active":"2023-10-30T13:25:16.168854+0000","last_peered":"2023-10-30T13:25:16.168854+0000","last_clean":"2023-10-30T13:25:16.168854+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T13:25:16.168854+0000","last_undegraded":"2023-10-30T13:25:16.168854+0000","last_fullsized":"2023-10-30T13:25:16.168854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1485,"log_dups_size":0,"ondisk_log_size":1485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:53:57.876949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096369869999999996,"stat_sum":{"num_bytes":18113290,"num_objects":584,"num_object_clones":270,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":219,"num_read":134,"num_read_kb":20299,"num_write":1266,"num_write_kb":19768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"85'777","reported_seq":1140,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:04.284846+0000","last_change":"2023-10-30T12:28:44.064606+0000","last_active":"2023-10-30T13:25:04.284846+0000","last_peered":"2023-10-30T13:25:04.284846+0000","last_clean":"2023-10-30T13:25:04.284846+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T13:25:04.284846+0000","last_undegraded":"2023-10-30T13:25:04.284846+0000","last_fullsized":"2023-10-30T13:25:04.284846+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":777,"log_dups_size":0,"ondisk_log_size":777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777306,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":495,"num_read_kb":170685,"num_write":965,"num_write_kb":43434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"82'1567","reported_seq":3571,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.146212+0000","last_change":"2023-10-30T12:37:07.112752+0000","last_active":"2023-10-30T13:24:56.146212+0000","last_peered":"2023-10-30T13:24:56.146212+0000","last_clean":"2023-10-30T13:24:56.146212+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T13:24:56.146212+0000","last_undegraded":"2023-10-30T13:24:56.146212+0000","last_fullsized":"2023-10-30T13:24:56.146212+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1567,"log_dups_size":0,"ondisk_log_size":1567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:41:59.413302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007081922,"stat_sum":{"num_bytes":14751960,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":228,"num_read":136,"num_read_kb":2513,"num_write":1376,"num_write_kb":21368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1125","reported_seq":1613,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:17.281839+0000","last_change":"2023-10-30T12:28:44.072754+0000","last_active":"2023-10-30T13:25:17.281839+0000","last_peered":"2023-10-30T13:25:17.281839+0000","last_clean":"2023-10-30T13:25:17.281839+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T13:25:17.281839+0000","last_undegraded":"2023-10-30T13:25:17.281839+0000","last_fullsized":"2023-10-30T13:25:17.281839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1125,"log_dups_size":0,"ondisk_log_size":1125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:51:55.865069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":763,"num_read_kb":345628,"num_write":1535,"num_write_kb":76701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"77'1460","reported_seq":3474,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.160346+0000","last_change":"2023-10-30T12:37:07.182563+0000","last_active":"2023-10-30T13:24:56.160346+0000","last_peered":"2023-10-30T13:24:56.160346+0000","last_clean":"2023-10-30T13:24:56.160346+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T13:24:56.160346+0000","last_undegraded":"2023-10-30T13:24:56.160346+0000","last_fullsized":"2023-10-30T13:24:56.160346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1460,"log_dups_size":0,"ondisk_log_size":1460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:29:22.948042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060337460000000004,"stat_sum":{"num_bytes":23618154,"num_objects":570,"num_object_clones":263,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":209,"num_read":127,"num_read_kb":29654,"num_write":1265,"num_write_kb":27771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"85'1182","reported_seq":1585,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:58.045831+0000","last_change":"2023-10-30T12:28:44.058801+0000","last_active":"2023-10-30T13:24:58.045831+0000","last_peered":"2023-10-30T13:24:58.045831+0000","last_clean":"2023-10-30T13:24:58.045831+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T13:24:58.045831+0000","last_undegraded":"2023-10-30T13:24:58.045831+0000","last_fullsized":"2023-10-30T13:24:58.045831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1182,"log_dups_size":0,"ondisk_log_size":1182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:11:24.706813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":538,"num_read_kb":321476,"num_write":1330,"num_write_kb":79987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"85'1546","reported_seq":3551,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.277281+0000","last_change":"2023-10-30T12:37:07.180393+0000","last_active":"2023-10-30T13:24:47.277281+0000","last_peered":"2023-10-30T13:24:47.277281+0000","last_clean":"2023-10-30T13:24:47.277281+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T13:24:47.277281+0000","last_undegraded":"2023-10-30T13:24:47.277281+0000","last_fullsized":"2023-10-30T13:24:47.277281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:28:37.347215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091395910000000007,"stat_sum":{"num_bytes":24394000,"num_objects":612,"num_object_clones":279,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":219,"num_read":133,"num_read_kb":28881,"num_write":1364,"num_write_kb":25892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"77'1527","reported_seq":3550,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.236448+0000","last_change":"2023-10-30T12:37:07.182718+0000","last_active":"2023-10-30T13:25:32.236448+0000","last_peered":"2023-10-30T13:25:32.236448+0000","last_clean":"2023-10-30T13:25:32.236448+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T13:25:32.236448+0000","last_undegraded":"2023-10-30T13:25:32.236448+0000","last_fullsized":"2023-10-30T13:25:32.236448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1527,"log_dups_size":0,"ondisk_log_size":1527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:12:33.103610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010120377999999999,"stat_sum":{"num_bytes":19538275,"num_objects":582,"num_object_clones":275,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":200,"num_read":151,"num_read_kb":26132,"num_write":1354,"num_write_kb":27822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"85'880","reported_seq":1327,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.246911+0000","last_change":"2023-10-30T12:28:44.054652+0000","last_active":"2023-10-30T13:24:47.246911+0000","last_peered":"2023-10-30T13:24:47.246911+0000","last_clean":"2023-10-30T13:24:47.246911+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T13:24:47.246911+0000","last_undegraded":"2023-10-30T13:24:47.246911+0000","last_fullsized":"2023-10-30T13:24:47.246911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:05.023374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":739,"num_read_kb":197443,"num_write":1077,"num_write_kb":51357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"82'1642","reported_seq":3637,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.166307+0000","last_change":"2023-10-30T12:37:07.102323+0000","last_active":"2023-10-30T13:24:56.166307+0000","last_peered":"2023-10-30T13:24:56.166307+0000","last_clean":"2023-10-30T13:24:56.166307+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T13:24:56.166307+0000","last_undegraded":"2023-10-30T13:24:56.166307+0000","last_fullsized":"2023-10-30T13:24:56.166307+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:12.249464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011628750000000001,"stat_sum":{"num_bytes":22561655,"num_objects":686,"num_object_clones":314,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":247,"num_read":162,"num_read_kb":29557,"num_write":1441,"num_write_kb":28808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"85'1048","reported_seq":1588,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.267547+0000","last_change":"2023-10-30T12:28:44.052633+0000","last_active":"2023-10-30T13:24:47.267547+0000","last_peered":"2023-10-30T13:24:47.267547+0000","last_clean":"2023-10-30T13:24:47.267547+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T13:24:47.267547+0000","last_undegraded":"2023-10-30T13:24:47.267547+0000","last_fullsized":"2023-10-30T13:24:47.267547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1048,"log_dups_size":0,"ondisk_log_size":1048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":939,"num_read_kb":433966,"num_write":1214,"num_write_kb":88022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"82'1531","reported_seq":3512,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.140708+0000","last_change":"2023-10-30T12:37:07.189494+0000","last_active":"2023-10-30T13:25:16.140708+0000","last_peered":"2023-10-30T13:25:16.140708+0000","last_clean":"2023-10-30T13:25:16.140708+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T13:25:16.140708+0000","last_undegraded":"2023-10-30T13:25:16.140708+0000","last_fullsized":"2023-10-30T13:25:16.140708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1531,"log_dups_size":0,"ondisk_log_size":1531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:34.313998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014632805,"stat_sum":{"num_bytes":14611265,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":226,"num_read":122,"num_read_kb":17905,"num_write":1326,"num_write_kb":16683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"85'753","reported_seq":1130,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.373735+0000","last_change":"2023-10-30T12:28:44.075493+0000","last_active":"2023-10-30T13:24:56.373735+0000","last_peered":"2023-10-30T13:24:56.373735+0000","last_clean":"2023-10-30T13:24:56.373735+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T13:24:56.373735+0000","last_undegraded":"2023-10-30T13:24:56.373735+0000","last_fullsized":"2023-10-30T13:24:56.373735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:00:11.718669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":488,"num_read_kb":379559,"num_write":896,"num_write_kb":83880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"77'1481","reported_seq":3496,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.138978+0000","last_change":"2023-10-30T12:37:07.191451+0000","last_active":"2023-10-30T13:24:56.138978+0000","last_peered":"2023-10-30T13:24:56.138978+0000","last_clean":"2023-10-30T13:24:56.138978+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T13:24:56.138978+0000","last_undegraded":"2023-10-30T13:24:56.138978+0000","last_fullsized":"2023-10-30T13:24:56.138978+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:28.248158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015871881000000001,"stat_sum":{"num_bytes":15563160,"num_objects":581,"num_object_clones":268,"num_object_copies":1162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":208,"num_read":132,"num_read_kb":9741,"num_write":1268,"num_write_kb":20904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"85'838","reported_seq":1196,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.123097+0000","last_change":"2023-10-30T12:28:44.065843+0000","last_active":"2023-10-30T13:25:02.123097+0000","last_peered":"2023-10-30T13:25:02.123097+0000","last_clean":"2023-10-30T13:25:02.123097+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T13:25:02.123097+0000","last_undegraded":"2023-10-30T13:25:02.123097+0000","last_fullsized":"2023-10-30T13:25:02.123097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:32.405199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33555014,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":517,"num_read_kb":175570,"num_write":978,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165214918,"num_objects":39301,"num_object_clones":18317,"num_object_copies":78602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39301,"num_whiteouts":14249,"num_read":9251,"num_read_kb":1265164,"num_write":86732,"num_write_kb":1459764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424127488,"data_stored":2332445068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99517,"ondisk_log_size":99517,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":903126397,"num_objects":1657,"num_object_clones":0,"num_object_copies":3314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":21550,"num_read_kb":9552846,"num_write":41301,"num_write_kb":2210413,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1806409728,"data_stored":1806251397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34756,"ondisk_log_size":34756,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477425,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":837308,"kb_used_data":639604,"kb_used_omap":1,"kb_used_meta":197694,"kb_avail":93534532,"statfs":{"total":96636764160,"available":95779360768,"internally_reserved":0,"allocated":654954496,"data_stored":640799419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":202439112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42699999999999999,"15min":0.434},"min":{"1min":0.253,"5min":0.253,"15min":0.23200000000000001},"max":{"1min":0.52300000000000002,"5min":0.66900000000000004,"15min":0.94599999999999995},"last":0.377},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.47099999999999997,"15min":0.46999999999999997},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.27700000000000002},"max":{"1min":0.68300000000000005,"5min":0.75800000000000001,"15min":0.81899999999999995},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.503,"15min":0.495},"min":{"1min":0.39000000000000001,"5min":0.33300000000000002,"15min":0.307},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":1.0209999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.76600000000000001,"5min":0.78400000000000003,"15min":0.99299999999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51500000000000001,"15min":0.502},"min":{"1min":0.373,"5min":0.34100000000000003,"15min":0.28599999999999998},"max":{"1min":0.747,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.434},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.42999999999999999,"5min":0.38600000000000001,"15min":0.33200000000000002},"max":{"1min":0.79700000000000004,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77900000000000003,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.94299999999999995,"5min":1.05,"15min":1.548},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.63900000000000001,"5min":0.497,"15min":0.497},"max":{"1min":1.127,"5min":1.149,"15min":1.6950000000000001},"last":0.73299999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.78900000000000003,"15min":0.81399999999999995},"min":{"1min":0.57299999999999995,"5min":0.52800000000000002,"15min":0.496},"max":{"1min":0.96399999999999997,"5min":1.1160000000000001,"15min":1.194},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.80000000000000004,"15min":0.80700000000000005},"min":{"1min":0.66100000000000003,"5min":0.63200000000000001,"15min":0.52100000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.603},"last":0.623}]},{"osd":6,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.64000000000000001,"5min":0.56999999999999995,"15min":0.436},"max":{"1min":1.113,"5min":1.198,"15min":1.2270000000000001},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.93100000000000005,"15min":0.92300000000000004},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.55600000000000005},"max":{"1min":1.145,"5min":1.202,"15min":1.3759999999999999},"last":0.75900000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.84799999999999998,"15min":0.83599999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.46400000000000002},"max":{"1min":1.133,"5min":1.133,"15min":1.1919999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.91400000000000003,"15min":0.877},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.47699999999999998},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.2050000000000001},"last":0.83299999999999996}]}]},{"osd":2,"up_from":16,"seq":68719477426,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534076,"kb_used_data":412724,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":93837764,"statfs":{"total":96636764160,"available":96089870336,"internally_reserved":0,"allocated":422629376,"data_stored":409295067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124254662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42999999999999999,"15min":0.43099999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57799999999999996,"5min":0.81299999999999994,"15min":0.91200000000000003},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48799999999999999,"15min":0.47899999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.625,"5min":1.554,"15min":2.8519999999999999},"last":0.36399999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47799999999999998,"15min":0.497},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.57199999999999995,"5min":1.071,"15min":4.9660000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52700000000000002,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.32100000000000001,"15min":0.28399999999999997},"max":{"1min":0.66600000000000004,"5min":1.198,"15min":6.0759999999999996},"last":0.442}]},{"osd":3,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.505,"15min":0.52800000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.314},"max":{"1min":0.65500000000000003,"5min":0.96799999999999997,"15min":4.6120000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56499999999999995,"15min":0.58699999999999997},"min":{"1min":0.42699999999999999,"5min":0.40899999999999997,"15min":0.379},"max":{"1min":0.68200000000000005,"5min":1.4430000000000001,"15min":4.641},"last":0.433}]},{"osd":4,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79100000000000004,"15min":0.80400000000000005},"min":{"1min":0.58399999999999996,"5min":0.48299999999999998,"15min":0.47999999999999998},"max":{"1min":0.95999999999999996,"5min":1.1719999999999999,"15min":3.5499999999999998},"last":1.083},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77000000000000002,"15min":0.81000000000000005},"min":{"1min":0.53200000000000003,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0389999999999999,"5min":1.262,"15min":5.032},"last":0.76100000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.88800000000000001,"15min":0.87},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.177,"5min":1.494,"15min":6.1289999999999996},"last":1.014},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.84699999999999998,"15min":0.877},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.159,"5min":1.3440000000000001,"15min":3.1459999999999999},"last":0.96899999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.83299999999999996,"15min":0.875},"min":{"1min":0.67600000000000005,"5min":0.54000000000000004,"15min":0.53100000000000003},"max":{"1min":1.0469999999999999,"5min":1.1739999999999999,"15min":5.8159999999999998},"last":1.139},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.82799999999999996,"15min":0.83299999999999996},"min":{"1min":0.66600000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.99299999999999999,"5min":1.6299999999999999,"15min":3.415},"last":1.0620000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.84199999999999997,"15min":0.872},"min":{"1min":0.73999999999999999,"5min":0.60699999999999998,"15min":0.51100000000000001},"max":{"1min":1.121,"5min":1.9139999999999999,"15min":4.883},"last":1.107},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.88400000000000001,"15min":0.90300000000000002},"min":{"1min":0.79300000000000004,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0640000000000001,"5min":1.3460000000000001,"15min":4.9950000000000001},"last":1.1619999999999999}]}]},{"osd":1,"up_from":16,"seq":68719477425,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534824,"kb_used_data":431520,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":93837016,"statfs":{"total":96636764160,"available":96089104384,"internally_reserved":0,"allocated":441876480,"data_stored":431229668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105773510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46899999999999997,"15min":0.52500000000000002},"min":{"1min":0.32800000000000001,"5min":0.27700000000000002,"15min":0.22600000000000001},"max":{"1min":0.66700000000000004,"5min":0.70699999999999996,"15min":20.196000000000002},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55100000000000005,"15min":0.54800000000000004},"min":{"1min":0.39800000000000002,"5min":0.31,"15min":0.253},"max":{"1min":0.77200000000000002,"5min":0.91100000000000003,"15min":5.1070000000000002},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48599999999999999,"15min":0.67200000000000004},"min":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.312},"max":{"1min":0.73099999999999998,"5min":0.83999999999999997,"15min":45.866999999999997},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.57099999999999995,"15min":0.64100000000000001},"min":{"1min":0.433,"5min":0.374,"15min":0.34699999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":20.789999999999999},"last":0.46400000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.47899999999999998,"15min":0.57899999999999996},"min":{"1min":0.36899999999999999,"5min":0.318,"15min":0.30499999999999999},"max":{"1min":0.624,"5min":0.80900000000000005,"15min":21.178999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.34999999999999998},"max":{"1min":0.71099999999999997,"5min":0.91800000000000004,"15min":2.2290000000000001},"last":0.50800000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.40400000000000003},"max":{"1min":1.133,"5min":1.133,"15min":5.9770000000000003},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.90400000000000003,"15min":0.86899999999999999},"min":{"1min":0.68400000000000005,"5min":0.64300000000000002,"15min":0.44400000000000001},"max":{"1min":1.093,"5min":1.1619999999999999,"15min":5.5110000000000001},"last":0.875}]},{"osd":5,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82399999999999995,"15min":0.872},"min":{"1min":0.58299999999999996,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0149999999999999,"5min":1.212,"15min":7.5430000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.90100000000000002,"15min":0.94499999999999995},"min":{"1min":0.76700000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0680000000000001,"5min":1.2350000000000001,"15min":7.5750000000000002},"last":0.84199999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.79900000000000004,"15min":0.88300000000000001},"min":{"1min":0.59699999999999998,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0820000000000001,"5min":1.177,"15min":20.827000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.82099999999999995,"15min":0.95399999999999996},"min":{"1min":0.68400000000000005,"5min":0.51800000000000002,"15min":0.51600000000000001},"max":{"1min":1.0469999999999999,"5min":1.159,"15min":20.931000000000001},"last":0.72499999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.83999999999999997,"15min":0.95799999999999996},"min":{"1min":0.73299999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":20.859999999999999},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90600000000000003,"15min":0.88500000000000001},"min":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.501},"max":{"1min":1.1870000000000001,"5min":1.194,"15min":4.367},"last":0.91600000000000004}]}]},{"osd":0,"up_from":16,"seq":68719477426,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937928,"kb_used_data":695808,"kb_used_omap":1,"kb_used_meta":242110,"kb_avail":93433912,"statfs":{"total":96636764160,"available":95676325888,"internally_reserved":0,"allocated":712507392,"data_stored":696810257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":247921094},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40600000000000003,"15min":0.41099999999999998},"min":{"1min":0.254,"5min":0.23899999999999999,"15min":0.22700000000000001},"max":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":3.996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51700000000000002,"15min":0.52600000000000002},"min":{"1min":0.41099999999999998,"5min":0.36199999999999999,"15min":0.309},"max":{"1min":0.67600000000000005,"5min":0.75600000000000001,"15min":4.3230000000000004},"last":0.44700000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.27700000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":4.9690000000000003},"last":0.372},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52100000000000002,"15min":0.53200000000000003},"min":{"1min":0.372,"5min":0.34100000000000003,"15min":0.30199999999999999},"max":{"1min":0.621,"5min":0.878,"15min":5.0049999999999999},"last":0.42499999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.48799999999999999},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.28299999999999997},"max":{"1min":0.68100000000000005,"5min":0.82699999999999996,"15min":5.4909999999999997},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57499999999999996,"15min":0.58099999999999996},"min":{"1min":0.443,"5min":0.42799999999999999,"15min":0.38600000000000001},"max":{"1min":0.749,"5min":0.90400000000000003,"15min":5.6970000000000001},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.58699999999999997,"5min":0.46700000000000003,"15min":0.377},"max":{"1min":1.111,"5min":1.111,"15min":4.444},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.80100000000000005},"min":{"1min":0.58499999999999996,"5min":0.55000000000000004,"15min":0.48099999999999998},"max":{"1min":0.90500000000000003,"5min":1.117,"15min":5.7750000000000004},"last":0.622}]},{"osd":5,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.83299999999999996,"15min":0.83499999999999996},"min":{"1min":0.66800000000000004,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.246,"5min":1.3340000000000001,"15min":4.5430000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.871,"15min":0.86399999999999999},"min":{"1min":0.59199999999999997,"5min":0.56999999999999995,"15min":0.47099999999999997},"max":{"1min":1.204,"5min":1.204,"15min":4.4790000000000001},"last":0.76200000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.82599999999999996,"15min":0.84199999999999997},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":1.397,"15min":4.5129999999999999},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.88500000000000001,"15min":0.89900000000000002},"min":{"1min":0.626,"5min":0.626,"15min":0.56699999999999995},"max":{"1min":1.1699999999999999,"5min":1.3120000000000001,"15min":4.5750000000000002},"last":0.60399999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.875,"15min":0.86799999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.55700000000000005},"max":{"1min":1.0669999999999999,"5min":1.206,"15min":4.3979999999999997},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.88900000000000001,"15min":0.91400000000000003},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.53700000000000003},"max":{"1min":1.0760000000000001,"5min":1.2210000000000001,"15min":5.375},"last":0.79700000000000004}]}]},{"osd":6,"up_from":15,"seq":64424510128,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526444,"kb_used_data":409252,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":93845396,"statfs":{"total":96636764160,"available":96097685504,"internally_reserved":0,"allocated":419074048,"data_stored":408471134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":119994823},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.874,"15min":0.86299999999999999},"min":{"1min":0.61099999999999999,"5min":0.49399999999999999,"15min":0.34200000000000003},"max":{"1min":1.167,"5min":1.4470000000000001,"15min":3.1150000000000002},"last":1.103},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.89900000000000002,"15min":0.96999999999999997},"min":{"1min":0.63500000000000001,"5min":0.54500000000000004,"15min":0.51700000000000002},"max":{"1min":1.121,"5min":1.415,"15min":3.3639999999999999},"last":0.89500000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.79800000000000004,"15min":0.79700000000000004},"min":{"1min":0.48799999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.087,"5min":1.202,"15min":1.589},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85899999999999999,"15min":0.86199999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.47699999999999998},"max":{"1min":1.145,"5min":1.2330000000000001,"15min":1.544},"last":0.83699999999999997}]},{"osd":2,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.98199999999999998,"15min":0.96299999999999997},"min":{"1min":0.68300000000000005,"5min":0.57299999999999995,"15min":0.47199999999999998},"max":{"1min":1.2969999999999999,"5min":1.496,"15min":3.4969999999999999},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.91600000000000004,"15min":0.94999999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.2090000000000001,"5min":1.391,"15min":1.512},"last":1.018}]},{"osd":3,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.86299999999999999,"15min":0.89100000000000001},"min":{"1min":0.58799999999999997,"5min":0.57099999999999995,"15min":0.52800000000000002},"max":{"1min":1.131,"5min":1.262,"15min":3.6070000000000002},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.91300000000000003,"15min":0.88300000000000001},"min":{"1min":0.69499999999999995,"5min":0.54000000000000004,"15min":0.51600000000000001},"max":{"1min":1.1839999999999999,"5min":1.272,"15min":3.7679999999999998},"last":0.77600000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97299999999999998,"15min":0.93799999999999994},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.2849999999999999,"5min":1.3220000000000001,"15min":1.4019999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.093,"15min":1.046},"min":{"1min":0.80000000000000004,"5min":0.73899999999999999,"15min":0.54700000000000004},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.2509999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.163,"15min":1.125},"min":{"1min":0.90100000000000002,"5min":0.82999999999999996,"15min":0.73399999999999999},"max":{"1min":1.417,"5min":1.4730000000000001,"15min":1.583},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.069,"15min":1.095},"min":{"1min":0.625,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2829999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.137}]},{"osd":7,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.081,"15min":1.0800000000000001},"min":{"1min":0.86099999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.4970000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.22,"15min":1.2170000000000001},"min":{"1min":1.034,"5min":0.76500000000000001,"15min":0.64200000000000002},"max":{"1min":1.804,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.1699999999999999}]}]},{"osd":4,"up_from":15,"seq":64424510130,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645792,"kb_used_data":493912,"kb_used_omap":1,"kb_used_meta":151870,"kb_avail":93726048,"statfs":{"total":96636764160,"available":95975473152,"internally_reserved":0,"allocated":505765888,"data_stored":496226659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":155515335},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.86299999999999999,"15min":0.83899999999999997},"min":{"1min":0.54800000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.1930000000000001,"5min":1.218,"15min":1.292},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.86899999999999999,"15min":0.84699999999999998},"min":{"1min":0.57399999999999995,"5min":0.52700000000000002,"15min":0.52600000000000002},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.369},"last":1.4570000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.79400000000000004,"15min":0.78600000000000003},"min":{"1min":0.60799999999999998,"5min":0.47899999999999998,"15min":0.47499999999999998},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1839999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.879,"5min":0.83699999999999997,"15min":0.83199999999999996},"min":{"1min":0.60899999999999999,"5min":0.53000000000000003,"15min":0.47299999999999998},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.3100000000000001},"last":0.996}]},{"osd":2,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.89500000000000002},"min":{"1min":0.56599999999999995,"5min":0.55500000000000005,"15min":0.55300000000000005},"max":{"1min":1.1899999999999999,"5min":1.2270000000000001,"15min":1.292},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.998,"5min":0.94599999999999995,"15min":0.94699999999999995},"min":{"1min":0.74099999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":1.3540000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.97399999999999998,"15min":0.95299999999999996},"min":{"1min":0.68500000000000005,"5min":0.65100000000000002,"15min":0.52200000000000002},"max":{"1min":1.286,"5min":1.286,"15min":1.514},"last":1.234},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.94699999999999995,"15min":0.96299999999999997},"min":{"1min":0.76600000000000001,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.94999999999999996}]},{"osd":5,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.98299999999999998,"15min":1.0129999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60299999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":2.4430000000000001},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.171,"5min":1.1140000000000001,"15min":1.0660000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.8460000000000001},"last":1.304}]},{"osd":6,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.145,"15min":1.1559999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.419,"5min":1.425,"15min":1.677},"last":1.411},{"interface":"front","average":{"1min":1.306,"5min":1.28,"15min":1.1830000000000001},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.64400000000000002},"max":{"1min":1.528,"5min":1.5409999999999999,"15min":1.9610000000000001},"last":1.5009999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.0529999999999999,"15min":1.1359999999999999},"min":{"1min":0.89500000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.8700000000000001},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.202,"15min":1.266},"min":{"1min":1.016,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.538,"5min":1.591,"15min":1.9450000000000001},"last":1.2689999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542832,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560428,"kb_used_data":416932,"kb_used_omap":1,"kb_used_meta":143486,"kb_avail":93811412,"statfs":{"total":96636764160,"available":96062885888,"internally_reserved":0,"allocated":426938368,"data_stored":418861204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":146930123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.84399999999999997,"15min":0.82999999999999996},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.41499999999999998},"max":{"1min":1.2130000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.88,"15min":0.872},"min":{"1min":0.56999999999999995,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.333,"5min":1.3380000000000001,"15min":1.3919999999999999},"last":1.1120000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.76900000000000002,"15min":0.86799999999999999},"min":{"1min":0.504,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.095,"5min":1.244,"15min":1.4790000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.91700000000000004,"15min":0.82299999999999995},"min":{"1min":0.51500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.5,"5min":1.524,"15min":1.524},"last":0.80400000000000005}]},{"osd":2,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0589999999999999,"15min":1.024},"min":{"1min":0.66500000000000004,"5min":0.54700000000000004,"15min":0.52700000000000002},"max":{"1min":1.3109999999999999,"5min":1.833,"15min":1.833},"last":1.022},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1830000000000001,"15min":1.153},"min":{"1min":0.91100000000000003,"5min":0.71499999999999997,"15min":0.69399999999999995},"max":{"1min":1.4510000000000001,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.0660000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0760000000000001,"15min":1.083},"min":{"1min":0.66400000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.2929999999999999,"5min":1.673,"15min":1.673},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.204,"5min":1.244,"15min":1.2230000000000001},"min":{"1min":0.84299999999999997,"5min":0.83499999999999996,"15min":0.81399999999999995},"max":{"1min":1.4490000000000001,"5min":1.8109999999999999,"15min":1.8480000000000001},"last":1.155}]},{"osd":4,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.86199999999999999,"15min":0.84799999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.171,"5min":1.226,"15min":1.4970000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.95799999999999996,"15min":0.95599999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.242,"5min":1.4970000000000001,"15min":1.77},"last":0.72499999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.94299999999999995,"15min":0.97499999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.2490000000000001,"5min":1.3799999999999999,"15min":2},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.105,"15min":1.0349999999999999},"min":{"1min":0.69599999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.508,"5min":1.7150000000000001,"15min":2.056},"last":0.92900000000000005}]},{"osd":6,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.042,"15min":1.008},"min":{"1min":0.68100000000000005,"5min":0.60799999999999998,"15min":0.501},"max":{"1min":1.377,"5min":1.536,"15min":1.571},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1299999999999999,"15min":1.105},"min":{"1min":0.751,"5min":0.72599999999999998,"15min":0.71199999999999997},"max":{"1min":1.5549999999999999,"5min":1.665,"15min":1.665},"last":0.84899999999999998}]}]},{"osd":5,"up_from":14,"seq":60129542830,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":802272,"kb_used_data":640600,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93569568,"statfs":{"total":96636764160,"available":95815237632,"internally_reserved":0,"allocated":655974400,"data_stored":642425281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165542342},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.378},"max":{"1min":1.173,"5min":1.276,"15min":1.579},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.85299999999999998,"15min":0.86099999999999999},"min":{"1min":0.499,"5min":0.46200000000000002,"15min":0.34599999999999997},"max":{"1min":1.262,"5min":1.627,"15min":1.627},"last":0.86299999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.81200000000000006,"15min":0.83099999999999996},"min":{"1min":0.55300000000000005,"5min":0.47499999999999998,"15min":0.39500000000000002},"max":{"1min":1.22,"5min":1.226,"15min":1.3220000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.88600000000000001,"15min":0.91900000000000004},"min":{"1min":0.64800000000000002,"5min":0.52600000000000002,"15min":0.376},"max":{"1min":1.1539999999999999,"5min":1.3100000000000001,"15min":1.6619999999999999},"last":0.72299999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.85799999999999998,"15min":0.84799999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.2330000000000001,"5min":1.3220000000000001,"15min":1.486},"last":0.5},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.89300000000000002,"15min":0.88},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2749999999999999,"5min":1.3149999999999999,"15min":1.879},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.96899999999999997,"15min":0.88900000000000001},"min":{"1min":0.65500000000000003,"5min":0.55900000000000005,"15min":0.42799999999999999},"max":{"1min":1.361,"5min":1.361,"15min":2.7330000000000001},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.097,"5min":1.0509999999999999,"15min":0.999},"min":{"1min":0.68899999999999995,"5min":0.51300000000000001,"15min":0.47099999999999997},"max":{"1min":1.359,"5min":1.5169999999999999,"15min":2.79},"last":1.1479999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.97299999999999998,"15min":0.98099999999999998},"min":{"1min":0.68500000000000005,"5min":0.57599999999999996,"15min":0.45300000000000001},"max":{"1min":1.119,"5min":1.4710000000000001,"15min":1.6719999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.034,"15min":1.0940000000000001},"min":{"1min":0.87,"5min":0.626,"15min":0.58799999999999997},"max":{"1min":1.3120000000000001,"5min":1.3540000000000001,"15min":1.7629999999999999},"last":0.77900000000000003}]},{"osd":6,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.1000000000000001,"15min":1.0920000000000001},"min":{"1min":0.69999999999999996,"5min":0.66500000000000004,"15min":0.54600000000000004},"max":{"1min":1.3959999999999999,"5min":1.675,"15min":2.8450000000000002},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.149,"5min":1.123,"15min":1.111},"min":{"1min":0.73699999999999999,"5min":0.59999999999999998,"15min":0.57999999999999996},"max":{"1min":1.577,"5min":1.625,"15min":1.821},"last":0.98699999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.153,"15min":1.127},"min":{"1min":0.70099999999999996,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7490000000000001,"5min":1.7749999999999999,"15min":3.3069999999999999},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.115,"5min":1.1579999999999999,"15min":1.113},"min":{"1min":0.88500000000000001,"5min":0.59999999999999998,"15min":0.53200000000000003},"max":{"1min":1.3129999999999999,"5min":1.754,"15min":2.036},"last":0.92000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309846016,"data_stored":309822659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142618624,"data_stored":142610667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117452800,"data_stored":117441250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294043648,"data_stored":294028449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":275668992,"data_stored":275627234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":300916736,"data_stored":300897937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":154218496,"data_stored":154206848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":211644416,"data_stored":211616353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401661952,"data_stored":386457412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297664512,"data_stored":287498447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304177152,"data_stored":291323631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359911424,"data_stored":346240784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":229097472,"data_stored":220069239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":353464320,"data_stored":340406790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":263856128,"data_stored":253734100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":214294528,"data_stored":206714665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:25:36.781 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:25:36.955 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.950+0000 7fc4e9daa640 1 Processor -- start 2023-10-30T13:25:36.956 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.950+0000 7fc4e9daa640 1 -- start start 2023-10-30T13:25:36.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e9daa640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fc4e40c6b80 0x7fc4e40bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.957 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e9daa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4e40c6f60 0x7fc4e40c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e9daa640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4e40c8430 0x7fc4e413ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e9daa640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e40c0a50 con 0x7fc4e40c8430 2023-10-30T13:25:36.958 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e9daa640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e406f6e0 con 0x7fc4e40c6b80 2023-10-30T13:25:36.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e9daa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e406f9c0 con 0x7fc4e40c6f60 2023-10-30T13:25:36.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4e40c8430 0x7fc4e413ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.959 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e3fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4e40c8430 0x7fc4e413ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38602/0 (socket says 172.21.15.125:38602) 2023-10-30T13:25:36.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e3fff640 1 -- 172.21.15.125:0/3675450461 learned_addr learned my addr 172.21.15.125:0/3675450461 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:36.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e37fe640 1 --2- 172.21.15.125:0/3675450461 >> v2:172.21.15.136:3300/0 conn(0x7fc4e40c6b80 0x7fc4e40bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e2ffd640 1 --2- 172.21.15.125:0/3675450461 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4e40c6f60 0x7fc4e40c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e3fff640 1 -- 172.21.15.125:0/3675450461 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4e40c6f60 msgr2=0x7fc4e40c0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:36.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e3fff640 1 --2- 172.21.15.125:0/3675450461 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4e40c6f60 0x7fc4e40c0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e3fff640 1 -- 172.21.15.125:0/3675450461 >> v2:172.21.15.136:3300/0 conn(0x7fc4e40c6b80 msgr2=0x7fc4e40bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:36.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e3fff640 1 --2- 172.21.15.125:0/3675450461 >> v2:172.21.15.136:3300/0 conn(0x7fc4e40c6b80 0x7fc4e40bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e3fff640 1 -- 172.21.15.125:0/3675450461 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e406fca0 con 0x7fc4e40c8430 2023-10-30T13:25:36.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e2ffd640 1 --2- 172.21.15.125:0/3675450461 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4e40c6f60 0x7fc4e40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:36.962 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.954+0000 7fc4e37fe640 1 --2- 172.21.15.125:0/3675450461 >> v2:172.21.15.136:3300/0 conn(0x7fc4e40c6b80 0x7fc4e40bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:36.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e3fff640 1 --2- 172.21.15.125:0/3675450461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4e40c8430 0x7fc4e413ed90 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fc4e406f1b0 tx=0x7fc4dc01c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c001265716f6ef96 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:36.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e27fc640 1 -- 172.21.15.125:0/3675450461 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0a56d0 con 0x7fc4e40c8430 2023-10-30T13:25:36.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e27fc640 1 -- 172.21.15.125:0/3675450461 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4dc0a5830 con 0x7fc4e40c8430 2023-10-30T13:25:36.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e27fc640 1 -- 172.21.15.125:0/3675450461 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0a5bc0 con 0x7fc4e40c8430 2023-10-30T13:25:36.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e9daa640 1 -- 172.21.15.125:0/3675450461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4e40c8430 msgr2=0x7fc4e413ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e9daa640 1 --2- 172.21.15.125:0/3675450461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4e40c8430 0x7fc4e413ed90 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7fc4e406f1b0 tx=0x7fc4dc01c940 comp rx=0 tx=0).stop 2023-10-30T13:25:36.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e9daa640 1 -- 172.21.15.125:0/3675450461 shutdown_connections 2023-10-30T13:25:36.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e9daa640 1 --2- 172.21.15.125:0/3675450461 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4e40c8430 0x7fc4e413ed90 unknown :-1 s=CLOSED pgs=2158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e9daa640 1 --2- 172.21.15.125:0/3675450461 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4e40c6f60 0x7fc4e40c0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e9daa640 1 --2- 172.21.15.125:0/3675450461 >> v2:172.21.15.136:3300/0 conn(0x7fc4e40c6b80 0x7fc4e40bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e9daa640 1 -- 172.21.15.125:0/3675450461 >> 172.21.15.125:0/3675450461 conn(0x7fc4e40c7910 msgr2=0x7fc4e40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:36.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e9daa640 1 -- 172.21.15.125:0/3675450461 shutdown_connections 2023-10-30T13:25:36.968 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e9daa640 1 -- 172.21.15.125:0/3675450461 wait complete. 2023-10-30T13:25:36.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e9daa640 1 Processor -- start 2023-10-30T13:25:36.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.958+0000 7fc4e9daa640 1 -- start start 2023-10-30T13:25:36.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e9daa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4e40c6b80 0x7fc4e41587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.969 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e9daa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4e40c6f60 0x7fc4e4132680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e9daa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4e40c8430 0x7fc4e41f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e9daa640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e4150650 con 0x7fc4e40c8430 2023-10-30T13:25:36.970 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e9daa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e41591e0 con 0x7fc4e40c6f60 2023-10-30T13:25:36.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e9daa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e4150060 con 0x7fc4e40c6b80 2023-10-30T13:25:36.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4e40c8430 0x7fc4e41f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e3fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4e40c8430 0x7fc4e41f51a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38612/0 (socket says 172.21.15.125:38612) 2023-10-30T13:25:36.971 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e3fff640 1 -- 172.21.15.125:0/991511618 learned_addr learned my addr 172.21.15.125:0/991511618 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:36.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e2ffd640 1 --2- 172.21.15.125:0/991511618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4e40c6f60 0x7fc4e4132680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e37fe640 1 --2- 172.21.15.125:0/991511618 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4e40c6b80 0x7fc4e41587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e3fff640 1 -- 172.21.15.125:0/991511618 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4e40c6b80 msgr2=0x7fc4e41587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.972 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e3fff640 1 --2- 172.21.15.125:0/991511618 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4e40c6b80 0x7fc4e41587f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e3fff640 1 -- 172.21.15.125:0/991511618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4e40c6f60 msgr2=0x7fc4e4132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:36.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e3fff640 1 --2- 172.21.15.125:0/991511618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4e40c6f60 0x7fc4e4132680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:36.973 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e3fff640 1 -- 172.21.15.125:0/991511618 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc009b30 con 0x7fc4e40c8430 2023-10-30T13:25:36.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e3fff640 1 --2- 172.21.15.125:0/991511618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4e40c8430 0x7fc4e41f51a0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc01d1e0 tx=0x7fc4dc0196e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:36.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4cbfff640 1 -- 172.21.15.125:0/991511618 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0b8880 con 0x7fc4e40c8430 2023-10-30T13:25:36.975 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e9daa640 1 -- 172.21.15.125:0/991511618 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e414efc0 con 0x7fc4e40c8430 2023-10-30T13:25:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e9daa640 1 -- 172.21.15.125:0/991511618 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e414f580 con 0x7fc4e40c8430 2023-10-30T13:25:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4cbfff640 1 -- 172.21.15.125:0/991511618 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4dc009510 con 0x7fc4e40c8430 2023-10-30T13:25:36.976 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4cbfff640 1 -- 172.21.15.125:0/991511618 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0b8be0 con 0x7fc4e40c8430 2023-10-30T13:25:36.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.962+0000 7fc4e9daa640 1 -- 172.21.15.125:0/991511618 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c4005260 con 0x7fc4e40c8430 2023-10-30T13:25:36.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.966+0000 7fc4cbfff640 1 -- 172.21.15.125:0/991511618 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc4dc0b8d40 con 0x7fc4e40c8430 2023-10-30T13:25:36.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.966+0000 7fc4cbfff640 1 --2- 172.21.15.125:0/991511618 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4d00664a0 0x7fc4d0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:36.977 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.966+0000 7fc4cbfff640 1 -- 172.21.15.125:0/991511618 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fc4dc1200f0 con 0x7fc4e40c8430 2023-10-30T13:25:36.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.966+0000 7fc4e37fe640 1 --2- 172.21.15.125:0/991511618 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4d00664a0 0x7fc4d0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:36.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.966+0000 7fc4e37fe640 1 --2- 172.21.15.125:0/991511618 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4d00664a0 0x7fc4d0068960 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fc4d400c880 tx=0x7fc4d4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:36.978 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:36.970+0000 7fc4cbfff640 1 -- 172.21.15.125:0/991511618 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc4dc0ec9b0 con 0x7fc4e40c8430 2023-10-30T13:25:37.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:37.122+0000 7fc4e9daa640 1 -- 172.21.15.125:0/991511618 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4c4005510 con 0x7fc4e40c8430 2023-10-30T13:25:37.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:37.122+0000 7fc4cbfff640 1 -- 172.21.15.125:0/991511618 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+20714 (secure 0 0 0) 0x7fc4dc0effd0 con 0x7fc4e40c8430 2023-10-30T13:25:37.127 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:25:37.128 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T13:24:15.942081+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":55,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:6835/3480968625":"2023-10-31T13:24:15.942040+0000","172.21.15.136:6839/4284345684":"2023-10-31T13:21:50.903213+0000","172.21.15.125:6833/739564283":"2023-10-31T12:57:50.441726+0000","172.21.15.136:6839/1454326577":"2023-10-31T12:40:50.129310+0000","172.21.15.125:6837/2408701650":"2023-10-31T12:52:30.335181+0000","172.21.15.125:6833/1163622496":"2023-10-31T12:55:15.400773+0000","172.21.15.125:6834/1624923396":"2023-10-31T12:42:15.151597+0000","172.21.15.125:6834/2408701650":"2023-10-31T12:52:30.335181+0000","172.21.15.125:6837/3906359913":"2023-10-31T12:59:15.462699+0000","172.21.15.125:6836/1721514617":"2023-10-31T12:51:15.314344+0000","172.21.15.125:6833/2595438444":"2023-10-31T12:46:30.228008+0000","172.21.15.125:6833/1721514617":"2023-10-31T12:51:15.314344+0000","172.21.15.136:6838/1510290691":"2023-10-31T12:47:25.242471+0000","172.21.15.125:6836/2595438444":"2023-10-31T12:46:30.228008+0000","172.21.15.125:6833/2017382874":"2023-10-31T12:36:30.049259+0000","172.21.15.125:6834/2540063352":"2023-10-31T13:12:50.713238+0000","172.21.15.125:6836/3950044434":"2023-10-31T13:17:55.811938+0000","172.21.15.125:6834/2291877604":"2023-10-31T12:32:34.980679+0000","172.21.15.136:6839/3922559697":"2023-10-31T12:44:00.178262+0000","172.21.15.125:6836/1163622496":"2023-10-31T12:55:15.400773+0000","172.21.15.136:6834/3480968625":"2023-10-31T13:24:15.942040+0000","172.21.15.136:6835/3346501886":"2023-10-31T12:30:24.948059+0000","172.21.15.125:6834/3906359913":"2023-10-31T12:59:15.462699+0000","172.21.15.125:6834/2196617378":"2023-10-31T13:05:55.585769+0000","172.21.15.125:6833/3711687505":"2023-10-31T13:06:55.602060+0000","172.21.15.136:6839/2250748765":"2023-10-31T12:35:15.030179+0000","172.21.15.125:6837/2291877604":"2023-10-31T12:32:34.980679+0000","172.21.15.136:6835/3526499585":"2023-10-31T13:11:30.689863+0000","172.21.15.136:6838/1454326577":"2023-10-31T12:40:50.129310+0000","172.21.15.136:6835/695089547":"2023-10-31T13:15:05.766729+0000","172.21.15.136:6839/223464319":"2023-10-31T13:08:55.644039+0000","172.21.15.125:6837/1624923396":"2023-10-31T12:42:15.151597+0000","172.21.15.125:6834/1348687922":"2023-10-31T12:56:35.422955+0000","172.21.15.125:6836/2017382874":"2023-10-31T12:36:30.049259+0000","172.21.15.136:6838/3922559697":"2023-10-31T12:44:00.178262+0000","172.21.15.125:6834/3156415951":"2023-10-31T12:48:25.270273+0000","172.21.15.125:6833/1944683138":"2023-10-31T13:00:40.486003+0000","172.21.15.125:6836/1944683138":"2023-10-31T13:00:40.486003+0000","172.21.15.125:6834/3762807620":"2023-10-31T13:02:35.533843+0000","172.21.15.125:6837/1348687922":"2023-10-31T12:56:35.422955+0000","172.21.15.125:6837/3762807620":"2023-10-31T13:02:35.533843+0000","172.21.15.125:6833/1301336591":"2023-10-31T13:04:45.567511+0000","172.21.15.125:6836/1301336591":"2023-10-31T13:04:45.567511+0000","172.21.15.136:6835/2194892271":"2023-10-31T13:08:00.628452+0000","172.21.15.125:6837/2196617378":"2023-10-31T13:05:55.585769+0000","172.21.15.125:6836/3711687505":"2023-10-31T13:06:55.602060+0000","172.21.15.136:6838/3776186190":"2023-10-31T13:16:00.781626+0000","172.21.15.136:6834/2194892271":"2023-10-31T13:08:00.628452+0000","172.21.15.136:6834/3526499585":"2023-10-31T13:11:30.689863+0000","172.21.15.136:6838/4284345684":"2023-10-31T13:21:50.903213+0000","172.21.15.136:6838/2250748765":"2023-10-31T12:35:15.030179+0000","172.21.15.136:6838/223464319":"2023-10-31T13:08:55.644039+0000","172.21.15.125:6837/3156415951":"2023-10-31T12:48:25.270273+0000","172.21.15.136:6834/695089547":"2023-10-31T13:15:05.766729+0000","172.21.15.125:6837/2540063352":"2023-10-31T13:12:50.713238+0000","172.21.15.136:6834/3346501886":"2023-10-31T12:30:24.948059+0000","172.21.15.136:6839/3776186190":"2023-10-31T13:16:00.781626+0000","172.21.15.125:6833/3950044434":"2023-10-31T13:17:55.811938+0000","172.21.15.136:6839/1510290691":"2023-10-31T12:47:25.242471+0000","172.21.15.136:6834/3864637420":"2023-10-31T13:20:45.886166+0000","172.21.15.125:6836/739564283":"2023-10-31T12:57:50.441726+0000","172.21.15.136:6835/3864637420":"2023-10-31T13:20:45.886166+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:25:37.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:37.126+0000 7fc4e9daa640 1 -- 172.21.15.125:0/991511618 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4d00664a0 msgr2=0x7fc4d0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:37.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:37.126+0000 7fc4e9daa640 1 --2- 172.21.15.125:0/991511618 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4d00664a0 0x7fc4d0068960 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fc4d400c880 tx=0x7fc4d4008040 comp rx=0 tx=0).stop 2023-10-30T13:25:37.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:37.126+0000 7fc4e9daa640 1 -- 172.21.15.125:0/991511618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4e40c8430 msgr2=0x7fc4e41f51a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:37.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:37.126+0000 7fc4e9daa640 1 --2- 172.21.15.125:0/991511618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4e40c8430 0x7fc4e41f51a0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc01d1e0 tx=0x7fc4dc0196e0 comp rx=0 tx=0).stop 2023-10-30T13:25:37.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:37.126+0000 7fc4e9daa640 1 -- 172.21.15.125:0/991511618 shutdown_connections 2023-10-30T13:25:37.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:37.126+0000 7fc4e9daa640 1 --2- 172.21.15.125:0/991511618 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc4d00664a0 0x7fc4d0068960 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:37.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:37.126+0000 7fc4e9daa640 1 --2- 172.21.15.125:0/991511618 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4e40c8430 0x7fc4e41f51a0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:37.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:37.126+0000 7fc4e9daa640 1 --2- 172.21.15.125:0/991511618 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4e40c6f60 0x7fc4e4132680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:37.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:37.126+0000 7fc4e9daa640 1 --2- 172.21.15.125:0/991511618 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc4e40c6b80 0x7fc4e41587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:37.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:37.126+0000 7fc4e9daa640 1 -- 172.21.15.125:0/991511618 >> 172.21.15.125:0/991511618 conn(0x7fc4e40c7910 msgr2=0x7fc4e40c1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:37.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:37.126+0000 7fc4e9daa640 1 -- 172.21.15.125:0/991511618 shutdown_connections 2023-10-30T13:25:37.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:37.126+0000 7fc4e9daa640 1 -- 172.21.15.125:0/991511618 wait complete. 2023-10-30T13:25:38.150 INFO:tasks.ceph:Scrubbing osd.0 2023-10-30T13:25:38.151 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:25:38.299 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.294+0000 7f8f9549a640 1 Processor -- start 2023-10-30T13:25:38.300 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.294+0000 7f8f9549a640 1 -- start start 2023-10-30T13:25:38.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f9549a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f90153960 0x7f8f9006e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f9549a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8f90069e90 0x7f8f9006a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f9549a640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9015ccf0 con 0x7f8f900be700 2023-10-30T13:25:38.302 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f9549a640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f90159740 con 0x7f8f90069e90 2023-10-30T13:25:38.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f9549a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f90159a20 con 0x7f8f90153960 2023-10-30T13:25:38.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f867fc640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f8f900be700 0x7f8f90159eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59594/0 (socket says 172.21.15.125:59594) 2023-10-30T13:25:38.303 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f867fc640 1 -- 172.21.15.125:0/2553609819 learned_addr learned my addr 172.21.15.125:0/2553609819 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:25:38.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f8f7fe640 1 --2- 172.21.15.125:0/2553609819 >> v2:172.21.15.125:3300/0 conn(0x7f8f90069e90 0x7f8f9006a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f8effd640 1 --2- 172.21.15.125:0/2553609819 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f90153960 0x7f8f9006e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f8e7fc640 1 -- 172.21.15.125:0/2553609819 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199081357 0 0) 0x7f8f9015ccf0 con 0x7f8f900be700 2023-10-30T13:25:38.304 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f8e7fc640 1 -- 172.21.15.125:0/2553609819 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f7c003530 con 0x7f8f900be700 2023-10-30T13:25:38.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f8f7fe640 1 -- 172.21.15.125:0/2553609819 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f90153960 msgr2=0x7f8f9006e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:38.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f8f7fe640 1 --2- 172.21.15.125:0/2553609819 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f90153960 0x7f8f9006e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.305 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f8f7fe640 1 -- 172.21.15.125:0/2553609819 >> v1:172.21.15.125:6789/0 conn(0x7f8f900be700 legacy=0x7f8f90159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.306 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.298+0000 7f8f8f7fe640 1 -- 172.21.15.125:0/2553609819 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f900c72c0 con 0x7f8f90069e90 2023-10-30T13:25:38.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.302+0000 7f8f8f7fe640 1 --2- 172.21.15.125:0/2553609819 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f90069e90 0x7f8f9006a2c0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f8f880097c0 tx=0x7f8f8801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e5056399f5a6494 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:38.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.302+0000 7f8f8e7fc640 1 -- 172.21.15.125:0/2553609819 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f880ab600 con 0x7f8f90069e90 2023-10-30T13:25:38.308 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.302+0000 7f8f8e7fc640 1 -- 172.21.15.125:0/2553609819 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f880a7050 con 0x7f8f90069e90 2023-10-30T13:25:38.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.302+0000 7f8f8e7fc640 1 -- 172.21.15.125:0/2553609819 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f880ab980 con 0x7f8f90069e90 2023-10-30T13:25:38.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.302+0000 7f8f9549a640 1 -- 172.21.15.125:0/2553609819 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f90069e90 msgr2=0x7f8f9006a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.309 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.302+0000 7f8f9549a640 1 --2- 172.21.15.125:0/2553609819 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f90069e90 0x7f8f9006a2c0 secure :-1 s=READY pgs=2161 cs=0 l=1 rev1=1 crypto rx=0x7f8f880097c0 tx=0x7f8f8801c360 comp rx=0 tx=0).stop 2023-10-30T13:25:38.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.302+0000 7f8f9549a640 1 -- 172.21.15.125:0/2553609819 shutdown_connections 2023-10-30T13:25:38.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.302+0000 7f8f9549a640 1 --2- 172.21.15.125:0/2553609819 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f90069e90 0x7f8f9006a2c0 unknown :-1 s=CLOSED pgs=2161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.302+0000 7f8f9549a640 1 --2- 172.21.15.125:0/2553609819 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f90153960 0x7f8f9006e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.302+0000 7f8f9549a640 1 -- 172.21.15.125:0/2553609819 >> 172.21.15.125:0/2553609819 conn(0x7f8f900cb490 msgr2=0x7f8f900cb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:38.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.302+0000 7f8f9549a640 1 -- 172.21.15.125:0/2553609819 shutdown_connections 2023-10-30T13:25:38.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.302+0000 7f8f9549a640 1 -- 172.21.15.125:0/2553609819 wait complete. 2023-10-30T13:25:38.311 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.302+0000 7f8f9549a640 1 Processor -- start 2023-10-30T13:25:38.312 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f9549a640 1 -- start start 2023-10-30T13:25:38.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f9549a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f90069e90 0x7f8f900c4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f9549a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f900be700 0x7f8f900c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f9549a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f90153960 0x7f8f900c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f9549a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f900c3830 con 0x7f8f90153960 2023-10-30T13:25:38.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f9549a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9015a8c0 con 0x7f8f900be700 2023-10-30T13:25:38.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f9549a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9015b880 con 0x7f8f90069e90 2023-10-30T13:25:38.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f8f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f90153960 0x7f8f900c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f8f7fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f90153960 0x7f8f900c4f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38632/0 (socket says 172.21.15.125:38632) 2023-10-30T13:25:38.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f8f7fe640 1 -- 172.21.15.125:0/3820544170 learned_addr learned my addr 172.21.15.125:0/3820544170 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:38.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f867fc640 1 --2- 172.21.15.125:0/3820544170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f900be700 0x7f8f900c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f8effd640 1 --2- 172.21.15.125:0/3820544170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f90069e90 0x7f8f900c4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f8f7fe640 1 -- 172.21.15.125:0/3820544170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f90069e90 msgr2=0x7f8f900c4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f8f7fe640 1 --2- 172.21.15.125:0/3820544170 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8f90069e90 0x7f8f900c4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f8f7fe640 1 -- 172.21.15.125:0/3820544170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f900be700 msgr2=0x7f8f900c31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f8f7fe640 1 --2- 172.21.15.125:0/3820544170 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8f900be700 0x7f8f900c31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f8f7fe640 1 -- 172.21.15.125:0/3820544170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f88009420 con 0x7f8f90153960 2023-10-30T13:25:38.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f8f7fe640 1 --2- 172.21.15.125:0/3820544170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f90153960 0x7f8f900c4f80 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f8f880ae040 tx=0x7f8f88019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:38.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f87fff640 1 -- 172.21.15.125:0/3820544170 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f880abe90 con 0x7f8f90153960 2023-10-30T13:25:38.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f87fff640 1 -- 172.21.15.125:0/3820544170 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f880b4410 con 0x7f8f90153960 2023-10-30T13:25:38.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f87fff640 1 -- 172.21.15.125:0/3820544170 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f880b47a0 con 0x7f8f90153960 2023-10-30T13:25:38.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f9549a640 1 -- 172.21.15.125:0/3820544170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f9015aad0 con 0x7f8f90153960 2023-10-30T13:25:38.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f9549a640 1 -- 172.21.15.125:0/3820544170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f9015b090 con 0x7f8f90153960 2023-10-30T13:25:38.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.306+0000 7f8f9549a640 1 -- 172.21.15.125:0/3820544170 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8f9015a5c0 con 0x7f8f90153960 2023-10-30T13:25:38.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.310+0000 7f8f87fff640 1 -- 172.21.15.125:0/3820544170 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8f880b2040 con 0x7f8f90153960 2023-10-30T13:25:38.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.310+0000 7f8f87fff640 1 --2- 172.21.15.125:0/3820544170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f700664a0 0x7f8f70068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.324 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.310+0000 7f8f87fff640 1 -- 172.21.15.125:0/3820544170 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f8f8811edf0 con 0x7f8f90153960 2023-10-30T13:25:38.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.310+0000 7f8f87fff640 1 --2- 172.21.15.125:0/3820544170 >> [v2:172.21.15.125:6800/1302652123,v1:172.21.15.125:6801/1302652123] conn(0x7f8f7006ffd0 0x7f8f700723f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.310+0000 7f8f87fff640 1 -- 172.21.15.125:0/3820544170 --> [v2:172.21.15.125:6800/1302652123,v1:172.21.15.125:6801/1302652123] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8f70072ac0 con 0x7f8f7006ffd0 2023-10-30T13:25:38.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.310+0000 7f8f87fff640 1 -- 172.21.15.125:0/3820544170 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f8811f160 con 0x7f8f90153960 2023-10-30T13:25:38.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.310+0000 7f8f8effd640 1 --2- 172.21.15.125:0/3820544170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f700664a0 0x7f8f70068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.310+0000 7f8f867fc640 1 --2- 172.21.15.125:0/3820544170 >> [v2:172.21.15.125:6800/1302652123,v1:172.21.15.125:6801/1302652123] conn(0x7f8f7006ffd0 0x7f8f700723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.310+0000 7f8f867fc640 1 --2- 172.21.15.125:0/3820544170 >> [v2:172.21.15.125:6800/1302652123,v1:172.21.15.125:6801/1302652123] conn(0x7f8f7006ffd0 0x7f8f700723f0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:38.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.310+0000 7f8f8effd640 1 --2- 172.21.15.125:0/3820544170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f700664a0 0x7f8f70068960 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f8f800073f0 tx=0x7f8f8000b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:38.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.310+0000 7f8f87fff640 1 -- 172.21.15.125:0/3820544170 <== osd.0 v2:172.21.15.125:6800/1302652123 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f8f70072ac0 con 0x7f8f7006ffd0 2023-10-30T13:25:38.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.322+0000 7f8f9549a640 1 -- 172.21.15.125:0/3820544170 --> [v2:172.21.15.125:6800/1302652123,v1:172.21.15.125:6801/1302652123] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8f900c7b20 con 0x7f8f7006ffd0 2023-10-30T13:25:38.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.322+0000 7f8f87fff640 1 -- 172.21.15.125:0/3820544170 <== osd.0 v2:172.21.15.125:6800/1302652123 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f8f78001890 con 0x7f8f7006ffd0 2023-10-30T13:25:38.331 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2023-10-30T13:25:38.331 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:25:38.332 INFO:teuthology.orchestra.run.smithi125.stdout:} 2023-10-30T13:25:38.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.322+0000 7f8f9549a640 1 -- 172.21.15.125:0/3820544170 >> [v2:172.21.15.125:6800/1302652123,v1:172.21.15.125:6801/1302652123] conn(0x7f8f7006ffd0 msgr2=0x7f8f700723f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.322+0000 7f8f9549a640 1 --2- 172.21.15.125:0/3820544170 >> [v2:172.21.15.125:6800/1302652123,v1:172.21.15.125:6801/1302652123] conn(0x7f8f7006ffd0 0x7f8f700723f0 crc :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.322+0000 7f8f9549a640 1 -- 172.21.15.125:0/3820544170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f700664a0 msgr2=0x7f8f70068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.322+0000 7f8f9549a640 1 --2- 172.21.15.125:0/3820544170 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8f700664a0 0x7f8f70068960 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f8f800073f0 tx=0x7f8f8000b040 comp rx=0 tx=0).stop 2023-10-30T13:25:38.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.322+0000 7f8f9549a640 1 -- 172.21.15.125:0/3820544170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f90153960 msgr2=0x7f8f900c4f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.322+0000 7f8f9549a640 1 --2- 172.21.15.125:0/3820544170 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8f90153960 0x7f8f900c4f80 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7f8f880ae040 tx=0x7f8f88019100 comp rx=0 tx=0).stop 2023-10-30T13:25:38.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.322+0000 7f8f8f7fe640 1 -- 172.21.15.125:0/3820544170 reap_dead start 2023-10-30T13:25:38.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.326+0000 7f8f9549a640 1 -- 172.21.15.125:0/3820544170 shutdown_connections 2023-10-30T13:25:38.338 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.326+0000 7f8f9549a640 1 -- 172.21.15.125:0/3820544170 >> 172.21.15.125:0/3820544170 conn(0x7f8f900cb490 msgr2=0x7f8f90159eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:38.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.326+0000 7f8f9549a640 1 -- 172.21.15.125:0/3820544170 shutdown_connections 2023-10-30T13:25:38.339 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.326+0000 7f8f9549a640 1 -- 172.21.15.125:0/3820544170 wait complete. 2023-10-30T13:25:38.346 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-30T13:25:38.484 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04c771d640 1 Processor -- start 2023-10-30T13:25:38.485 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04c771d640 1 -- start start 2023-10-30T13:25:38.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04c771d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c0070840 0x7f04c0070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04c771d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c00c6880 con 0x7f04c0069e90 2023-10-30T13:25:38.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04c771d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04c00711c0 con 0x7f04c0153810 2023-10-30T13:25:38.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04c771d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c00c5990 con 0x7f04c0070840 2023-10-30T13:25:38.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04c5cba640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f04c0069e90 0x7f04c006a300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59602/0 (socket says 172.21.15.125:59602) 2023-10-30T13:25:38.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04c5cba640 1 -- 172.21.15.125:0/2676503653 learned_addr learned my addr 172.21.15.125:0/2676503653 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:25:38.488 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04c4cb8640 1 --2- 172.21.15.125:0/2676503653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c0070840 0x7f04c0070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624389653 0 0) 0x7f04c00c6880 con 0x7f04c0069e90 2023-10-30T13:25:38.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ac003530 con 0x7f04c0069e90 2023-10-30T13:25:38.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856012355 0 0) 0x7f04c00711c0 con 0x7f04c0153810 2023-10-30T13:25:38.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04c00c6880 con 0x7f04c0153810 2023-10-30T13:25:38.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4113944917 0 0) 0x7f04ac003530 con 0x7f04c0069e90 2023-10-30T13:25:38.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c0070840 msgr2=0x7f04c0070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:38.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 --2- 172.21.15.125:0/2676503653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c0070840 0x7f04c0070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.493 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 >> v1:172.21.15.136:6789/0 conn(0x7f04c0153810 legacy=0x7f04c015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c00c5c70 con 0x7f04c0069e90 2023-10-30T13:25:38.494 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f04b8001a00 con 0x7f04c0069e90 2023-10-30T13:25:38.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 >> v1:172.21.15.125:6789/0 conn(0x7f04c0069e90 legacy=0x7f04c006a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.495 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 --2- 172.21.15.125:0/2676503653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c0070840 0x7f04ac003b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 --2- 172.21.15.125:0/2676503653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04ac0040a0 0x7f04ac0047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.496 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 --2- 172.21.15.125:0/2676503653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04ac004d10 0x7f04ac005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c00c5990 con 0x7f04ac0040a0 2023-10-30T13:25:38.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac00ba70 con 0x7f04ac004d10 2023-10-30T13:25:38.497 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac00bc30 con 0x7f04c0070840 2023-10-30T13:25:38.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04c4cb8640 1 --2- 172.21.15.125:0/2676503653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04ac0040a0 0x7f04ac0047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.498 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04c5cba640 1 --2- 172.21.15.125:0/2676503653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04ac004d10 0x7f04ac005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.482+0000 7f04c54b9640 1 --2- 172.21.15.125:0/2676503653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c0070840 0x7f04ac003b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.486+0000 7f04c4cb8640 1 -- 172.21.15.125:0/2676503653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c0070840 msgr2=0x7f04ac003b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:38.499 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.486+0000 7f04c4cb8640 1 --2- 172.21.15.125:0/2676503653 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c0070840 0x7f04ac003b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.486+0000 7f04c4cb8640 1 -- 172.21.15.125:0/2676503653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04ac004d10 msgr2=0x7f04ac005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:38.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.486+0000 7f04c4cb8640 1 --2- 172.21.15.125:0/2676503653 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04ac004d10 0x7f04ac005190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.486+0000 7f04c4cb8640 1 -- 172.21.15.125:0/2676503653 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac003530 con 0x7f04ac0040a0 2023-10-30T13:25:38.500 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.486+0000 7f04c4cb8640 1 --2- 172.21.15.125:0/2676503653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04ac0040a0 0x7f04ac0047d0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f04b0009ae0 tx=0x7f04b001ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1108a45458c9d78 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:38.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.486+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b00a5530 con 0x7f04ac0040a0 2023-10-30T13:25:38.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.486+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b00a5690 con 0x7f04ac0040a0 2023-10-30T13:25:38.501 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.486+0000 7f04b7fff640 1 -- 172.21.15.125:0/2676503653 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b00a5a40 con 0x7f04ac0040a0 2023-10-30T13:25:38.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.486+0000 7f04c771d640 1 -- 172.21.15.125:0/2676503653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04ac0040a0 msgr2=0x7f04ac0047d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.486+0000 7f04c771d640 1 --2- 172.21.15.125:0/2676503653 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04ac0040a0 0x7f04ac0047d0 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7f04b0009ae0 tx=0x7f04b001ca50 comp rx=0 tx=0).stop 2023-10-30T13:25:38.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.486+0000 7f04c5cba640 1 -- 172.21.15.125:0/2676503653 reap_dead start 2023-10-30T13:25:38.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.486+0000 7f04c771d640 1 -- 172.21.15.125:0/2676503653 shutdown_connections 2023-10-30T13:25:38.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.486+0000 7f04c771d640 1 -- 172.21.15.125:0/2676503653 >> 172.21.15.125:0/2676503653 conn(0x7f04c00c7910 msgr2=0x7f04c00c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:38.506 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c771d640 1 -- 172.21.15.125:0/2676503653 shutdown_connections 2023-10-30T13:25:38.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c771d640 1 -- 172.21.15.125:0/2676503653 wait complete. 2023-10-30T13:25:38.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c771d640 1 Processor -- start 2023-10-30T13:25:38.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c771d640 1 -- start start 2023-10-30T13:25:38.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c771d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04ac0040a0 0x7f04c01efc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c771d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c01578e0 0x7f04c0158220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c771d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c0159010 0x7f04c0159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c771d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c01f0140 con 0x7f04c01578e0 2023-10-30T13:25:38.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c771d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c00c5270 con 0x7f04ac0040a0 2023-10-30T13:25:38.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c771d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c01599d0 con 0x7f04c0159010 2023-10-30T13:25:38.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c4cb8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c01578e0 0x7f04c0158220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c4cb8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c01578e0 0x7f04c0158220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38658/0 (socket says 172.21.15.125:38658) 2023-10-30T13:25:38.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c4cb8640 1 -- 172.21.15.125:0/3752015958 learned_addr learned my addr 172.21.15.125:0/3752015958 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:38.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c54b9640 1 --2- 172.21.15.125:0/3752015958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04ac0040a0 0x7f04c01efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c5cba640 1 --2- 172.21.15.125:0/3752015958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c0159010 0x7f04c0159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c4cb8640 1 -- 172.21.15.125:0/3752015958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c0159010 msgr2=0x7f04c0159490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c4cb8640 1 --2- 172.21.15.125:0/3752015958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c0159010 0x7f04c0159490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c4cb8640 1 -- 172.21.15.125:0/3752015958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04ac0040a0 msgr2=0x7f04c01efc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c4cb8640 1 --2- 172.21.15.125:0/3752015958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04ac0040a0 0x7f04c01efc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c4cb8640 1 -- 172.21.15.125:0/3752015958 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c00c43c0 con 0x7f04c01578e0 2023-10-30T13:25:38.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04c4cb8640 1 --2- 172.21.15.125:0/3752015958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c01578e0 0x7f04c0158220 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f04b0009ae0 tx=0x7f04ac005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:38.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.490+0000 7f04b5ffb640 1 -- 172.21.15.125:0/3752015958 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b00be070 con 0x7f04c01578e0 2023-10-30T13:25:38.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.494+0000 7f04b5ffb640 1 -- 172.21.15.125:0/3752015958 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04b000f0c0 con 0x7f04c01578e0 2023-10-30T13:25:38.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.494+0000 7f04b5ffb640 1 -- 172.21.15.125:0/3752015958 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b0004220 con 0x7f04c01578e0 2023-10-30T13:25:38.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.494+0000 7f04c771d640 1 -- 172.21.15.125:0/3752015958 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c0157480 con 0x7f04c01578e0 2023-10-30T13:25:38.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.494+0000 7f04c771d640 1 -- 172.21.15.125:0/3752015958 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c00c4940 con 0x7f04c01578e0 2023-10-30T13:25:38.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.494+0000 7f04c771d640 1 -- 172.21.15.125:0/3752015958 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04c004e170 con 0x7f04c01578e0 2023-10-30T13:25:38.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.494+0000 7f04b5ffb640 1 -- 172.21.15.125:0/3752015958 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f04b00b4040 con 0x7f04c01578e0 2023-10-30T13:25:38.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.494+0000 7f04b5ffb640 1 --2- 172.21.15.125:0/3752015958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04a40664a0 0x7f04a4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.494+0000 7f04b5ffb640 1 -- 172.21.15.125:0/3752015958 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f04b011e870 con 0x7f04c01578e0 2023-10-30T13:25:38.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.494+0000 7f04c54b9640 1 --2- 172.21.15.125:0/3752015958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04a40664a0 0x7f04a4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.498+0000 7f04c54b9640 1 --2- 172.21.15.125:0/3752015958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04a40664a0 0x7f04a4068960 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f04bc000c10 tx=0x7f04bc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:38.515 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.498+0000 7f04b5ffb640 1 -- 172.21.15.125:0/3752015958 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f04b00eb2d0 con 0x7f04c01578e0 2023-10-30T13:25:38.651 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.646+0000 7f04c771d640 1 -- 172.21.15.125:0/3752015958 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f04c00c6230 con 0x7f04a40664a0 2023-10-30T13:25:38.652 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.650+0000 7f04b5ffb640 1 -- 172.21.15.125:0/3752015958 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f04bc001560 con 0x7f04a40664a0 2023-10-30T13:25:38.652 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 0 to deep-scrub 2023-10-30T13:25:38.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.650+0000 7f04c771d640 1 -- 172.21.15.125:0/3752015958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04a40664a0 msgr2=0x7f04a4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.650+0000 7f04c771d640 1 --2- 172.21.15.125:0/3752015958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04a40664a0 0x7f04a4068960 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f04bc000c10 tx=0x7f04bc009040 comp rx=0 tx=0).stop 2023-10-30T13:25:38.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.650+0000 7f04c771d640 1 -- 172.21.15.125:0/3752015958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c01578e0 msgr2=0x7f04c0158220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.650+0000 7f04c771d640 1 --2- 172.21.15.125:0/3752015958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c01578e0 0x7f04c0158220 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7f04b0009ae0 tx=0x7f04ac005800 comp rx=0 tx=0).stop 2023-10-30T13:25:38.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.650+0000 7f04c771d640 1 -- 172.21.15.125:0/3752015958 shutdown_connections 2023-10-30T13:25:38.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.650+0000 7f04c771d640 1 --2- 172.21.15.125:0/3752015958 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f04a40664a0 0x7f04a4068960 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.650+0000 7f04c771d640 1 --2- 172.21.15.125:0/3752015958 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f04c0159010 0x7f04c0159490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.650+0000 7f04c771d640 1 --2- 172.21.15.125:0/3752015958 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f04c01578e0 0x7f04c0158220 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.650+0000 7f04c771d640 1 --2- 172.21.15.125:0/3752015958 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f04ac0040a0 0x7f04c01efc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.650+0000 7f04c771d640 1 -- 172.21.15.125:0/3752015958 >> 172.21.15.125:0/3752015958 conn(0x7f04c00c7910 msgr2=0x7f04c0151190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:38.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.650+0000 7f04c771d640 1 -- 172.21.15.125:0/3752015958 shutdown_connections 2023-10-30T13:25:38.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.650+0000 7f04c771d640 1 -- 172.21.15.125:0/3752015958 wait complete. 2023-10-30T13:25:38.674 INFO:tasks.ceph:Scrubbing osd.1 2023-10-30T13:25:38.674 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:25:38.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.810+0000 7fa2ef9d3640 1 Processor -- start 2023-10-30T13:25:38.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.810+0000 7fa2ef9d3640 1 -- start start 2023-10-30T13:25:38.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2ef9d3640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fa2e80c8430 0x7fa2e80c6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.817 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2ef9d3640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e815a270 con 0x7fa2e814cf40 2023-10-30T13:25:38.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2ef9d3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e806c7c0 con 0x7fa2e815a700 2023-10-30T13:25:38.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2ef9d3640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2e8156ed0 con 0x7fa2e80c8430 2023-10-30T13:25:38.818 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2ed76f640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa2e814cf40 0x7fa2e8069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59608/0 (socket says 172.21.15.125:59608) 2023-10-30T13:25:38.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2ed76f640 1 -- 172.21.15.125:0/1992566982 learned_addr learned my addr 172.21.15.125:0/1992566982 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:25:38.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2edf70640 1 --2- 172.21.15.125:0/1992566982 >> v2:172.21.15.136:3300/0 conn(0x7fa2e80c8430 0x7fa2e80c6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 -- 172.21.15.125:0/1992566982 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744893775 0 0) 0x7fa2e815a270 con 0x7fa2e814cf40 2023-10-30T13:25:38.819 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 -- 172.21.15.125:0/1992566982 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2d4003530 con 0x7fa2e814cf40 2023-10-30T13:25:38.821 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 -- 172.21.15.125:0/1992566982 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 110611924 0 0) 0x7fa2d4003530 con 0x7fa2e814cf40 2023-10-30T13:25:38.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 -- 172.21.15.125:0/1992566982 >> v2:172.21.15.136:3300/0 conn(0x7fa2e80c8430 msgr2=0x7fa2e80c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:38.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 --2- 172.21.15.125:0/1992566982 >> v2:172.21.15.136:3300/0 conn(0x7fa2e80c8430 0x7fa2e80c6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.822 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 -- 172.21.15.125:0/1992566982 >> v1:172.21.15.136:6789/0 conn(0x7fa2e815a700 legacy=0x7fa2e8156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 -- 172.21.15.125:0/1992566982 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e81571b0 con 0x7fa2e814cf40 2023-10-30T13:25:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 -- 172.21.15.125:0/1992566982 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fa2e4001680 con 0x7fa2e814cf40 2023-10-30T13:25:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 -- 172.21.15.125:0/1992566982 >> v1:172.21.15.125:6789/0 conn(0x7fa2e814cf40 legacy=0x7fa2e8069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 --2- 172.21.15.125:0/1992566982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4003980 0x7fa2d4003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 --2- 172.21.15.125:0/1992566982 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2d40047d0 0x7fa2d4004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 --2- 172.21.15.125:0/1992566982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2d4005190 0x7fa2d400dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.824 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 -- 172.21.15.125:0/1992566982 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8156ed0 con 0x7fa2d4003980 2023-10-30T13:25:38.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 -- 172.21.15.125:0/1992566982 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d400e240 con 0x7fa2d4005190 2023-10-30T13:25:38.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2dffff640 1 -- 172.21.15.125:0/1992566982 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2d400e400 con 0x7fa2d40047d0 2023-10-30T13:25:38.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2ed76f640 1 --2- 172.21.15.125:0/1992566982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4003980 0x7fa2d4003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.825 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2ecf6e640 1 --2- 172.21.15.125:0/1992566982 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2d40047d0 0x7fa2d4004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2edf70640 1 --2- 172.21.15.125:0/1992566982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2d4005190 0x7fa2d400dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2ed76f640 1 -- 172.21.15.125:0/1992566982 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2d40047d0 msgr2=0x7fa2d4004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2ed76f640 1 --2- 172.21.15.125:0/1992566982 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2d40047d0 0x7fa2d4004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2ed76f640 1 -- 172.21.15.125:0/1992566982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2d4005190 msgr2=0x7fa2d400dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2ed76f640 1 --2- 172.21.15.125:0/1992566982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2d4005190 0x7fa2d400dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2ed76f640 1 -- 172.21.15.125:0/1992566982 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d4003530 con 0x7fa2d4003980 2023-10-30T13:25:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2edf70640 1 --2- 172.21.15.125:0/1992566982 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2d4005190 0x7fa2d400dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:25:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.814+0000 7fa2edf70640 1 -- 172.21.15.125:0/1992566982 reap_dead start 2023-10-30T13:25:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.818+0000 7fa2ed76f640 1 --2- 172.21.15.125:0/1992566982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4003980 0x7fa2d4003da0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fa2e400c240 tx=0x7fa2e401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cb14428c11f7068 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.818+0000 7fa2dffff640 1 -- 172.21.15.125:0/1992566982 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4005ae0 con 0x7fa2d4003980 2023-10-30T13:25:38.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.818+0000 7fa2dffff640 1 -- 172.21.15.125:0/1992566982 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e4005c40 con 0x7fa2d4003980 2023-10-30T13:25:38.829 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.818+0000 7fa2dffff640 1 -- 172.21.15.125:0/1992566982 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e40a59d0 con 0x7fa2d4003980 2023-10-30T13:25:38.832 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.818+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/1992566982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4003980 msgr2=0x7fa2d4003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.818+0000 7fa2ef9d3640 1 --2- 172.21.15.125:0/1992566982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4003980 0x7fa2d4003da0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7fa2e400c240 tx=0x7fa2e401cc00 comp rx=0 tx=0).stop 2023-10-30T13:25:38.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.818+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/1992566982 shutdown_connections 2023-10-30T13:25:38.833 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.818+0000 7fa2ef9d3640 1 --2- 172.21.15.125:0/1992566982 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4003980 0x7fa2d4003da0 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.818+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/1992566982 >> 172.21.15.125:0/1992566982 conn(0x7fa2e80c7910 msgr2=0x7fa2e80c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:38.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.818+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/1992566982 shutdown_connections 2023-10-30T13:25:38.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.818+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/1992566982 wait complete. 2023-10-30T13:25:38.834 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ef9d3640 1 Processor -- start 2023-10-30T13:25:38.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ef9d3640 1 -- start start 2023-10-30T13:25:38.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ef9d3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4003980 0x7fa2e80c67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.835 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ef9d3640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2e8158670 0x7fa2e81f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ef9d3640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e81f2070 0x7fa2e81ee370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ef9d3640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e8143b50 con 0x7fa2d4003980 2023-10-30T13:25:38.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ef9d3640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e81edf40 con 0x7fa2e81f2070 2023-10-30T13:25:38.836 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ef9d3640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e81f1630 con 0x7fa2e8158670 2023-10-30T13:25:38.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ed76f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4003980 0x7fa2e80c67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ed76f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4003980 0x7fa2e80c67e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38682/0 (socket says 172.21.15.125:38682) 2023-10-30T13:25:38.837 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ed76f640 1 -- 172.21.15.125:0/4022756038 learned_addr learned my addr 172.21.15.125:0/4022756038 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:38.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ecf6e640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2e8158670 0x7fa2e81f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ed76f640 1 -- 172.21.15.125:0/4022756038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2e8158670 msgr2=0x7fa2e81f1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ed76f640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa2e8158670 0x7fa2e81f1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ed76f640 1 -- 172.21.15.125:0/4022756038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e81f2070 msgr2=0x7fa2e81ee370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2edf70640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e81f2070 0x7fa2e81ee370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.839 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ed76f640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e81f2070 0x7fa2e81ee370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ed76f640 1 -- 172.21.15.125:0/4022756038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e8158f80 con 0x7fa2d4003980 2023-10-30T13:25:38.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2edf70640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa2e81f2070 0x7fa2e81ee370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:25:38.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ed76f640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4003980 0x7fa2e80c67e0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fa2e400c240 tx=0x7fa2d4005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:38.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ddffb640 1 -- 172.21.15.125:0/4022756038 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e40b4070 con 0x7fa2d4003980 2023-10-30T13:25:38.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ddffb640 1 -- 172.21.15.125:0/4022756038 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e4001480 con 0x7fa2d4003980 2023-10-30T13:25:38.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/4022756038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e8159260 con 0x7fa2d4003980 2023-10-30T13:25:38.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ddffb640 1 -- 172.21.15.125:0/4022756038 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e40a59d0 con 0x7fa2d4003980 2023-10-30T13:25:38.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/4022756038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e8159820 con 0x7fa2d4003980 2023-10-30T13:25:38.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.822+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/4022756038 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa2e81f14b0 con 0x7fa2d4003980 2023-10-30T13:25:38.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.826+0000 7fa2ddffb640 1 -- 172.21.15.125:0/4022756038 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa2e40a5b30 con 0x7fa2d4003980 2023-10-30T13:25:38.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.826+0000 7fa2ddffb640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2cc0664a0 0x7fa2cc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.826+0000 7fa2ddffb640 1 -- 172.21.15.125:0/4022756038 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fa2e411d870 con 0x7fa2d4003980 2023-10-30T13:25:38.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.826+0000 7fa2ddffb640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.125:6816/3694357726,v1:172.21.15.125:6817/3694357726] conn(0x7fa2cc070140 0x7fa2cc072560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:38.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.826+0000 7fa2ecf6e640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2cc0664a0 0x7fa2cc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.826+0000 7fa2ddffb640 1 -- 172.21.15.125:0/4022756038 --> [v2:172.21.15.125:6816/3694357726,v1:172.21.15.125:6817/3694357726] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2cc072c30 con 0x7fa2cc070140 2023-10-30T13:25:38.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.826+0000 7fa2ddffb640 1 -- 172.21.15.125:0/4022756038 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2e411dbe0 con 0x7fa2d4003980 2023-10-30T13:25:38.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.826+0000 7fa2edf70640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.125:6816/3694357726,v1:172.21.15.125:6817/3694357726] conn(0x7fa2cc070140 0x7fa2cc072560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:38.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.826+0000 7fa2edf70640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.125:6816/3694357726,v1:172.21.15.125:6817/3694357726] conn(0x7fa2cc070140 0x7fa2cc072560 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:38.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.826+0000 7fa2ecf6e640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2cc0664a0 0x7fa2cc068960 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8158600 tx=0x7fa2d8009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:38.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.826+0000 7fa2ddffb640 1 -- 172.21.15.125:0/4022756038 <== osd.1 v2:172.21.15.125:6816/3694357726 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fa2cc072c30 con 0x7fa2cc070140 2023-10-30T13:25:38.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.842+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/4022756038 --> [v2:172.21.15.125:6816/3694357726,v1:172.21.15.125:6817/3694357726] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa2e814d320 con 0x7fa2cc070140 2023-10-30T13:25:38.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.842+0000 7fa2ddffb640 1 -- 172.21.15.125:0/4022756038 <== osd.1 v2:172.21.15.125:6816/3694357726 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa2e0009e20 con 0x7fa2cc070140 2023-10-30T13:25:38.847 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2023-10-30T13:25:38.848 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:25:38.848 INFO:teuthology.orchestra.run.smithi125.stdout:} 2023-10-30T13:25:38.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.842+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/4022756038 >> [v2:172.21.15.125:6816/3694357726,v1:172.21.15.125:6817/3694357726] conn(0x7fa2cc070140 msgr2=0x7fa2cc072560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.842+0000 7fa2ef9d3640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.125:6816/3694357726,v1:172.21.15.125:6817/3694357726] conn(0x7fa2cc070140 0x7fa2cc072560 crc :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:38.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.842+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/4022756038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2cc0664a0 msgr2=0x7fa2cc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.842+0000 7fa2ef9d3640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa2cc0664a0 0x7fa2cc068960 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8158600 tx=0x7fa2d8009040 comp rx=0 tx=0).stop 2023-10-30T13:25:38.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.842+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/4022756038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4003980 msgr2=0x7fa2e80c67e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:38.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.842+0000 7fa2ef9d3640 1 --2- 172.21.15.125:0/4022756038 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa2d4003980 0x7fa2e80c67e0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7fa2e400c240 tx=0x7fa2d4005800 comp rx=0 tx=0).stop 2023-10-30T13:25:38.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.842+0000 7fa2edf70640 1 -- 172.21.15.125:0/4022756038 reap_dead start 2023-10-30T13:25:38.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.842+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/4022756038 shutdown_connections 2023-10-30T13:25:38.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.842+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/4022756038 >> 172.21.15.125:0/4022756038 conn(0x7fa2e80c7910 msgr2=0x7fa2e80c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:38.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.842+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/4022756038 shutdown_connections 2023-10-30T13:25:38.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.842+0000 7fa2ef9d3640 1 -- 172.21.15.125:0/4022756038 wait complete. 2023-10-30T13:25:38.863 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-30T13:25:39.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.998+0000 7f194759e640 1 Processor -- start 2023-10-30T13:25:39.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.998+0000 7f194759e640 1 -- start start 2023-10-30T13:25:39.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.998+0000 7f194759e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f19480cb050 0x7f19480cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.998+0000 7f194759e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19480cba30 con 0x7f19480cab00 2023-10-30T13:25:39.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.998+0000 7f194759e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19480ca290 con 0x7f19480c5550 2023-10-30T13:25:39.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:38.998+0000 7f194759e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19480c9cf0 con 0x7f19480cb050 2023-10-30T13:25:39.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.002+0000 7f193dd9b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f19480cb050 0x7f19480cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.002+0000 7f1946d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f19480cab00 0x7f1948163ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59616/0 (socket says 172.21.15.125:59616) 2023-10-30T13:25:39.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.002+0000 7f1946d9d640 1 -- 172.21.15.125:0/3317547411 learned_addr learned my addr 172.21.15.125:0/3317547411 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:25:39.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.002+0000 7f193dd9b640 1 -- 172.21.15.125:0/3317547411 >> v1:172.21.15.136:6789/0 conn(0x7f19480c5550 legacy=0x7f19480cef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.002+0000 7f193dd9b640 1 -- 172.21.15.125:0/3317547411 >> v1:172.21.15.125:6789/0 conn(0x7f19480cab00 legacy=0x7f1948163ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.002+0000 7f193dd9b640 1 -- 172.21.15.125:0/3317547411 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194815d990 con 0x7f19480cb050 2023-10-30T13:25:39.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.002+0000 7f1945d9b640 1 -- 172.21.15.125:0/3317547411 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1023673264 0 0) 0x7f19480cba30 con 0x7f19480cab00 2023-10-30T13:25:39.019 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.002+0000 7f193dd9b640 1 --2- 172.21.15.125:0/3317547411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19480cb050 0x7f19480cbec0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f1930009de0 tx=0x7f193001c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=287ec930959aa65b server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:39.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.002+0000 7f1945d9b640 1 -- 172.21.15.125:0/3317547411 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19300a5850 con 0x7f19480cb050 2023-10-30T13:25:39.020 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.002+0000 7f1945d9b640 1 -- 172.21.15.125:0/3317547411 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19300a59b0 con 0x7f19480cb050 2023-10-30T13:25:39.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.002+0000 7f1945d9b640 1 -- 172.21.15.125:0/3317547411 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19300a5d40 con 0x7f19480cb050 2023-10-30T13:25:39.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 -- 172.21.15.125:0/3317547411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19480cb050 msgr2=0x7f19480cbec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.021 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 --2- 172.21.15.125:0/3317547411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19480cb050 0x7f19480cbec0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7f1930009de0 tx=0x7f193001c990 comp rx=0 tx=0).stop 2023-10-30T13:25:39.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 -- 172.21.15.125:0/3317547411 shutdown_connections 2023-10-30T13:25:39.022 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 --2- 172.21.15.125:0/3317547411 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19480cb050 0x7f19480cbec0 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 -- 172.21.15.125:0/3317547411 >> 172.21.15.125:0/3317547411 conn(0x7f194806bed0 msgr2=0x7f194806c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:39.023 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 -- 172.21.15.125:0/3317547411 shutdown_connections 2023-10-30T13:25:39.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 -- 172.21.15.125:0/3317547411 wait complete. 2023-10-30T13:25:39.024 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 Processor -- start 2023-10-30T13:25:39.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 -- start start 2023-10-30T13:25:39.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19480c5550 0x7f19481f5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.025 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19480cab00 0x7f19481f3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19480cb050 0x7f19481f4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19480cfad0 con 0x7f19480c5550 2023-10-30T13:25:39.026 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19481f2f00 con 0x7f19480cb050 2023-10-30T13:25:39.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194759e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1948157b30 con 0x7f19480cab00 2023-10-30T13:25:39.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19480c5550 0x7f19481f5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194659c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19480c5550 0x7f19481f5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38704/0 (socket says 172.21.15.125:38704) 2023-10-30T13:25:39.027 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194659c640 1 -- 172.21.15.125:0/111397474 learned_addr learned my addr 172.21.15.125:0/111397474 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:39.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f1946d9d640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19480cb050 0x7f19481f4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f193dd9b640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19480cab00 0x7f19481f3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.028 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194659c640 1 -- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19480cab00 msgr2=0x7f19481f3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194659c640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19480cab00 0x7f19481f3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194659c640 1 -- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19480cb050 msgr2=0x7f19481f4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.029 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194659c640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19480cb050 0x7f19481f4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f194659c640 1 -- 172.21.15.125:0/111397474 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1930009a70 con 0x7f19480c5550 2023-10-30T13:25:39.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.006+0000 7f1946d9d640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19480cb050 0x7f19481f4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:39.030 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.010+0000 7f194659c640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19480c5550 0x7f19481f5260 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f193800c3c0 tx=0x7f193800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:39.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.010+0000 7f193f7fe640 1 -- 172.21.15.125:0/111397474 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19380052d0 con 0x7f19480c5550 2023-10-30T13:25:39.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.010+0000 7f193f7fe640 1 -- 172.21.15.125:0/111397474 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1938005430 con 0x7f19480c5550 2023-10-30T13:25:39.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.010+0000 7f193f7fe640 1 -- 172.21.15.125:0/111397474 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1938003840 con 0x7f19480c5550 2023-10-30T13:25:39.031 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.010+0000 7f194759e640 1 -- 172.21.15.125:0/111397474 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19481f5c30 con 0x7f19480c5550 2023-10-30T13:25:39.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.010+0000 7f194759e640 1 -- 172.21.15.125:0/111397474 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19481f3430 con 0x7f19480c5550 2023-10-30T13:25:39.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.010+0000 7f194759e640 1 -- 172.21.15.125:0/111397474 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f194804e170 con 0x7f19480c5550 2023-10-30T13:25:39.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.010+0000 7f193f7fe640 1 -- 172.21.15.125:0/111397474 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f19380039a0 con 0x7f19480c5550 2023-10-30T13:25:39.032 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.010+0000 7f193f7fe640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19280664d0 0x7f1928068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.010+0000 7f193f7fe640 1 -- 172.21.15.125:0/111397474 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f1938083d70 con 0x7f19480c5550 2023-10-30T13:25:39.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.010+0000 7f193dd9b640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19280664d0 0x7f1928068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.033 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.014+0000 7f193dd9b640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19280664d0 0x7f1928068990 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f19300af000 tx=0x7f19300a7070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:39.034 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.014+0000 7f193f7fe640 1 -- 172.21.15.125:0/111397474 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1938050850 con 0x7f19480c5550 2023-10-30T13:25:39.168 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.166+0000 7f194759e640 1 -- 172.21.15.125:0/111397474 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f194815d570 con 0x7f19280664d0 2023-10-30T13:25:39.170 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.166+0000 7f193f7fe640 1 -- 172.21.15.125:0/111397474 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f19300a5770 con 0x7f19280664d0 2023-10-30T13:25:39.170 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 1 to deep-scrub 2023-10-30T13:25:39.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.166+0000 7f194759e640 1 -- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19280664d0 msgr2=0x7f1928068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.166+0000 7f194759e640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19280664d0 0x7f1928068990 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f19300af000 tx=0x7f19300a7070 comp rx=0 tx=0).stop 2023-10-30T13:25:39.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.166+0000 7f194759e640 1 -- 172.21.15.125:0/111397474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19480c5550 msgr2=0x7f19481f5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.166+0000 7f194759e640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19480c5550 0x7f19481f5260 secure :-1 s=READY pgs=2171 cs=0 l=1 rev1=1 crypto rx=0x7f193800c3c0 tx=0x7f193800c880 comp rx=0 tx=0).stop 2023-10-30T13:25:39.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.166+0000 7f194759e640 1 -- 172.21.15.125:0/111397474 shutdown_connections 2023-10-30T13:25:39.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.166+0000 7f194759e640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f19280664d0 0x7f1928068990 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.166+0000 7f194759e640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f19480cb050 0x7f19481f4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.166+0000 7f194759e640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f19480cab00 0x7f19481f3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.166+0000 7f194759e640 1 --2- 172.21.15.125:0/111397474 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f19480c5550 0x7f19481f5260 unknown :-1 s=CLOSED pgs=2171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.166+0000 7f194759e640 1 -- 172.21.15.125:0/111397474 >> 172.21.15.125:0/111397474 conn(0x7f194806bed0 msgr2=0x7f194815c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:39.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.166+0000 7f194759e640 1 -- 172.21.15.125:0/111397474 shutdown_connections 2023-10-30T13:25:39.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.166+0000 7f194759e640 1 -- 172.21.15.125:0/111397474 wait complete. 2023-10-30T13:25:39.191 INFO:tasks.ceph:Scrubbing osd.2 2023-10-30T13:25:39.193 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:25:39.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.330+0000 7f9100c9e640 1 Processor -- start 2023-10-30T13:25:39.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.330+0000 7f9100c9e640 1 -- start start 2023-10-30T13:25:39.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.330+0000 7f9100c9e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc06e620 0x7f90fc0c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.330+0000 7f9100c9e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f90fc15d9f0 0x7f90fc0be770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.330+0000 7f9100c9e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90fc1596f0 con 0x7f90fc15d610 2023-10-30T13:25:39.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.330+0000 7f9100c9e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90fc0c64c0 con 0x7f90fc15d9f0 2023-10-30T13:25:39.336 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.330+0000 7f9100c9e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc0c67a0 con 0x7f90fc06e620 2023-10-30T13:25:39.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.334+0000 7f90f9d9b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc06e620 0x7f90fc0c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.337 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.334+0000 7f90fad9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f90fc15d9f0 0x7f90fc0be770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.334+0000 7f90fad9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f90fc15d9f0 0x7f90fc0be770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:51842/0 (socket says 172.21.15.125:51842) 2023-10-30T13:25:39.340 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.334+0000 7f90fad9d640 1 -- 172.21.15.125:0/917138967 learned_addr learned my addr 172.21.15.125:0/917138967 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:39.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.334+0000 7f90f9d9b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc06e620 0x7f90fc0c78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:48254/0 (socket says 172.21.15.125:48254) 2023-10-30T13:25:39.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.334+0000 7f90f9d9b640 1 -- 172.21.15.125:0/917138967 >> v2:172.21.15.136:3300/0 conn(0x7f90fc15d9f0 msgr2=0x7f90fc0be770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:39.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.334+0000 7f90f9d9b640 1 --2- 172.21.15.125:0/917138967 >> v2:172.21.15.136:3300/0 conn(0x7f90fc15d9f0 0x7f90fc0be770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.341 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.334+0000 7f90f9d9b640 1 -- 172.21.15.125:0/917138967 >> v1:172.21.15.136:6789/0 conn(0x7f90fc15d610 legacy=0x7f90fc069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.334+0000 7f90f9d9b640 1 -- 172.21.15.125:0/917138967 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90fc159970 con 0x7f90fc06e620 2023-10-30T13:25:39.342 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.334+0000 7f90fad9d640 1 --2- 172.21.15.125:0/917138967 >> v2:172.21.15.136:3300/0 conn(0x7f90fc15d9f0 0x7f90fc0be770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:25:39.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.334+0000 7f90f9d9b640 1 --2- 172.21.15.125:0/917138967 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc06e620 0x7f90fc0c78e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f90f00097c0 tx=0x7f90f001c450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd9c7e0ce3c0ed09 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:39.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f90f959a640 1 -- 172.21.15.125:0/917138967 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f00ab670 con 0x7f90fc06e620 2023-10-30T13:25:39.345 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f90f959a640 1 -- 172.21.15.125:0/917138967 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90fc06a7b0 con 0x7f90fc06e620 2023-10-30T13:25:39.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f90f959a640 1 -- 172.21.15.125:0/917138967 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f00ab9f0 con 0x7f90fc06e620 2023-10-30T13:25:39.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 -- 172.21.15.125:0/917138967 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc06e620 msgr2=0x7f90fc0c78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.346 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 --2- 172.21.15.125:0/917138967 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc06e620 0x7f90fc0c78e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f90f00097c0 tx=0x7f90f001c450 comp rx=0 tx=0).stop 2023-10-30T13:25:39.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 -- 172.21.15.125:0/917138967 shutdown_connections 2023-10-30T13:25:39.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 --2- 172.21.15.125:0/917138967 >> v2:172.21.15.136:3300/0 conn(0x7f90fc15d9f0 0x7f90fc0be770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 --2- 172.21.15.125:0/917138967 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc06e620 0x7f90fc0c78e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.347 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 -- 172.21.15.125:0/917138967 >> 172.21.15.125:0/917138967 conn(0x7f90fc0cb490 msgr2=0x7f90fc0cb8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:39.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 -- 172.21.15.125:0/917138967 shutdown_connections 2023-10-30T13:25:39.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 -- 172.21.15.125:0/917138967 wait complete. 2023-10-30T13:25:39.348 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 Processor -- start 2023-10-30T13:25:39.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 -- start start 2023-10-30T13:25:39.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc06e620 0x7f90fc0ca560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.349 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc15d610 0x7f90fc0caaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc15d9f0 0x7f90fc0c29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc1f64e0 con 0x7f90fc06e620 2023-10-30T13:25:39.350 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc0bd030 con 0x7f90fc15d610 2023-10-30T13:25:39.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.338+0000 7f9100c9e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc14f2b0 con 0x7f90fc15d9f0 2023-10-30T13:25:39.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90fa59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc06e620 0x7f90fc0ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90fa59c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc06e620 0x7f90fc0ca560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38710/0 (socket says 172.21.15.125:38710) 2023-10-30T13:25:39.351 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90fa59c640 1 -- 172.21.15.125:0/2819429797 learned_addr learned my addr 172.21.15.125:0/2819429797 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:39.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90f9d9b640 1 --2- 172.21.15.125:0/2819429797 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc15d610 0x7f90fc0caaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90fad9d640 1 --2- 172.21.15.125:0/2819429797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc15d9f0 0x7f90fc0c29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.352 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90fa59c640 1 -- 172.21.15.125:0/2819429797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc15d9f0 msgr2=0x7f90fc0c29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90fa59c640 1 --2- 172.21.15.125:0/2819429797 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f90fc15d9f0 0x7f90fc0c29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90fa59c640 1 -- 172.21.15.125:0/2819429797 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc15d610 msgr2=0x7f90fc0caaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90fa59c640 1 --2- 172.21.15.125:0/2819429797 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f90fc15d610 0x7f90fc0caaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.353 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90fa59c640 1 -- 172.21.15.125:0/2819429797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90f4009b30 con 0x7f90fc06e620 2023-10-30T13:25:39.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90fa59c640 1 --2- 172.21.15.125:0/2819429797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc06e620 0x7f90fc0ca560 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00c450 tx=0x7f90ec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:39.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90eaffd640 1 -- 172.21.15.125:0/2819429797 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec0052d0 con 0x7f90fc06e620 2023-10-30T13:25:39.354 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90eaffd640 1 -- 172.21.15.125:0/2819429797 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90ec005430 con 0x7f90fc06e620 2023-10-30T13:25:39.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f9100c9e640 1 -- 172.21.15.125:0/2819429797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f0009420 con 0x7f90fc06e620 2023-10-30T13:25:39.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90eaffd640 1 -- 172.21.15.125:0/2819429797 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90ec007260 con 0x7f90fc06e620 2023-10-30T13:25:39.355 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f9100c9e640 1 -- 172.21.15.125:0/2819429797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90fc1f66d0 con 0x7f90fc06e620 2023-10-30T13:25:39.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f9100c9e640 1 -- 172.21.15.125:0/2819429797 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f90fc15a580 con 0x7f90fc06e620 2023-10-30T13:25:39.357 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90eaffd640 1 -- 172.21.15.125:0/2819429797 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f90ec0073c0 con 0x7f90fc06e620 2023-10-30T13:25:39.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.342+0000 7f90eaffd640 1 --2- 172.21.15.125:0/2819429797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90dc0664a0 0x7f90dc068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.346+0000 7f90eaffd640 1 -- 172.21.15.125:0/2819429797 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f90ec084290 con 0x7f90fc06e620 2023-10-30T13:25:39.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.346+0000 7f90eaffd640 1 --2- 172.21.15.125:0/2819429797 >> [v2:172.21.15.125:6808/456280330,v1:172.21.15.125:6809/456280330] conn(0x7f90dc070140 0x7f90dc072560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.346+0000 7f90eaffd640 1 -- 172.21.15.125:0/2819429797 --> [v2:172.21.15.125:6808/456280330,v1:172.21.15.125:6809/456280330] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f90dc072c30 con 0x7f90dc070140 2023-10-30T13:25:39.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.346+0000 7f90eaffd640 1 -- 172.21.15.125:0/2819429797 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90ec086870 con 0x7f90fc06e620 2023-10-30T13:25:39.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.346+0000 7f90f9d9b640 1 --2- 172.21.15.125:0/2819429797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90dc0664a0 0x7f90dc068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.346+0000 7f90fad9d640 1 --2- 172.21.15.125:0/2819429797 >> [v2:172.21.15.125:6808/456280330,v1:172.21.15.125:6809/456280330] conn(0x7f90dc070140 0x7f90dc072560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.346+0000 7f90fad9d640 1 --2- 172.21.15.125:0/2819429797 >> [v2:172.21.15.125:6808/456280330,v1:172.21.15.125:6809/456280330] conn(0x7f90dc070140 0x7f90dc072560 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:39.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.346+0000 7f90f9d9b640 1 --2- 172.21.15.125:0/2819429797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90dc0664a0 0x7f90dc068960 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f90f00af000 tx=0x7f90f00a7000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:39.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.346+0000 7f90eaffd640 1 -- 172.21.15.125:0/2819429797 <== osd.2 v2:172.21.15.125:6808/456280330 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f90dc072c30 con 0x7f90dc070140 2023-10-30T13:25:39.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.358+0000 7f9100c9e640 1 -- 172.21.15.125:0/2819429797 --> [v2:172.21.15.125:6808/456280330,v1:172.21.15.125:6809/456280330] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f90fc14f470 con 0x7f90dc070140 2023-10-30T13:25:39.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.358+0000 7f90eaffd640 1 -- 172.21.15.125:0/2819429797 <== osd.2 v2:172.21.15.125:6808/456280330 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f90fc14f470 con 0x7f90dc070140 2023-10-30T13:25:39.363 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2023-10-30T13:25:39.363 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:25:39.363 INFO:teuthology.orchestra.run.smithi125.stdout:} 2023-10-30T13:25:39.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.358+0000 7f9100c9e640 1 -- 172.21.15.125:0/2819429797 >> [v2:172.21.15.125:6808/456280330,v1:172.21.15.125:6809/456280330] conn(0x7f90dc070140 msgr2=0x7f90dc072560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.358+0000 7f9100c9e640 1 --2- 172.21.15.125:0/2819429797 >> [v2:172.21.15.125:6808/456280330,v1:172.21.15.125:6809/456280330] conn(0x7f90dc070140 0x7f90dc072560 crc :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.358+0000 7f9100c9e640 1 -- 172.21.15.125:0/2819429797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90dc0664a0 msgr2=0x7f90dc068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.358+0000 7f9100c9e640 1 --2- 172.21.15.125:0/2819429797 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f90dc0664a0 0x7f90dc068960 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f90f00af000 tx=0x7f90f00a7000 comp rx=0 tx=0).stop 2023-10-30T13:25:39.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.358+0000 7f9100c9e640 1 -- 172.21.15.125:0/2819429797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc06e620 msgr2=0x7f90fc0ca560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.358+0000 7f9100c9e640 1 --2- 172.21.15.125:0/2819429797 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f90fc06e620 0x7f90fc0ca560 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7f90ec00c450 tx=0x7f90ec00c910 comp rx=0 tx=0).stop 2023-10-30T13:25:39.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.358+0000 7f90fad9d640 1 -- 172.21.15.125:0/2819429797 reap_dead start 2023-10-30T13:25:39.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.358+0000 7f9100c9e640 1 -- 172.21.15.125:0/2819429797 shutdown_connections 2023-10-30T13:25:39.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.358+0000 7f9100c9e640 1 -- 172.21.15.125:0/2819429797 >> 172.21.15.125:0/2819429797 conn(0x7f90fc0cb490 msgr2=0x7f90fc0cb870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:39.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.358+0000 7f9100c9e640 1 -- 172.21.15.125:0/2819429797 shutdown_connections 2023-10-30T13:25:39.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.358+0000 7f9100c9e640 1 -- 172.21.15.125:0/2819429797 wait complete. 2023-10-30T13:25:39.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-30T13:25:39.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.514+0000 7fcd790d6640 1 Processor -- start 2023-10-30T13:25:39.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.514+0000 7fcd790d6640 1 -- start start 2023-10-30T13:25:39.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.518+0000 7fcd790d6640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fcd74151910 0x7fcd74158120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.518+0000 7fcd790d6640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcd7406a970 0x7fcd740c5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.518+0000 7fcd790d6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7406a4e0 con 0x7fcd7406ade0 2023-10-30T13:25:39.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.518+0000 7fcd790d6640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd74158660 con 0x7fcd7406a970 2023-10-30T13:25:39.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.518+0000 7fcd790d6640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd740c6c50 con 0x7fcd74151910 2023-10-30T13:25:39.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.518+0000 7fcd7259c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcd7406a970 0x7fcd740c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.518+0000 7fcd7259c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcd7406a970 0x7fcd740c5810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38726/0 (socket says 172.21.15.125:38726) 2023-10-30T13:25:39.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.518+0000 7fcd7259c640 1 -- 172.21.15.125:0/2678987449 learned_addr learned my addr 172.21.15.125:0/2678987449 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:39.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.518+0000 7fcd72d9d640 1 --2- 172.21.15.125:0/2678987449 >> v2:172.21.15.136:3300/0 conn(0x7fcd74151910 0x7fcd74158120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.518+0000 7fcd7259c640 1 -- 172.21.15.125:0/2678987449 >> v2:172.21.15.136:3300/0 conn(0x7fcd74151910 msgr2=0x7fcd74158120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:39.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.518+0000 7fcd7259c640 1 --2- 172.21.15.125:0/2678987449 >> v2:172.21.15.136:3300/0 conn(0x7fcd74151910 0x7fcd74158120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.518+0000 7fcd7259c640 1 -- 172.21.15.125:0/2678987449 >> v1:172.21.15.136:6789/0 conn(0x7fcd7406ade0 legacy=0x7fcd740c5d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.518+0000 7fcd7259c640 1 -- 172.21.15.125:0/2678987449 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd740c6f30 con 0x7fcd7406a970 2023-10-30T13:25:39.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.518+0000 7fcd72d9d640 1 --2- 172.21.15.125:0/2678987449 >> v2:172.21.15.136:3300/0 conn(0x7fcd74151910 0x7fcd74158120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:39.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.522+0000 7fcd7259c640 1 --2- 172.21.15.125:0/2678987449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd7406a970 0x7fcd740c5810 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fcd740b1020 tx=0x7fcd6801ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f800e8e827878d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:39.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.522+0000 7fcd71d9b640 1 -- 172.21.15.125:0/2678987449 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd680a89f0 con 0x7fcd7406a970 2023-10-30T13:25:39.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.522+0000 7fcd71d9b640 1 -- 172.21.15.125:0/2678987449 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd680a8b50 con 0x7fcd7406a970 2023-10-30T13:25:39.527 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.522+0000 7fcd71d9b640 1 -- 172.21.15.125:0/2678987449 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd680b0440 con 0x7fcd7406a970 2023-10-30T13:25:39.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.522+0000 7fcd790d6640 1 -- 172.21.15.125:0/2678987449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd7406a970 msgr2=0x7fcd740c5810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.522+0000 7fcd790d6640 1 --2- 172.21.15.125:0/2678987449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd7406a970 0x7fcd740c5810 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fcd740b1020 tx=0x7fcd6801ecf0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.528 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.522+0000 7fcd790d6640 1 -- 172.21.15.125:0/2678987449 shutdown_connections 2023-10-30T13:25:39.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.522+0000 7fcd790d6640 1 --2- 172.21.15.125:0/2678987449 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd7406a970 0x7fcd740c5810 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.522+0000 7fcd790d6640 1 --2- 172.21.15.125:0/2678987449 >> v2:172.21.15.136:3300/0 conn(0x7fcd74151910 0x7fcd74158120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.522+0000 7fcd790d6640 1 -- 172.21.15.125:0/2678987449 >> 172.21.15.125:0/2678987449 conn(0x7fcd740c7910 msgr2=0x7fcd740c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:39.529 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.522+0000 7fcd790d6640 1 -- 172.21.15.125:0/2678987449 shutdown_connections 2023-10-30T13:25:39.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.522+0000 7fcd790d6640 1 -- 172.21.15.125:0/2678987449 wait complete. 2023-10-30T13:25:39.530 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.522+0000 7fcd790d6640 1 Processor -- start 2023-10-30T13:25:39.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.522+0000 7fcd790d6640 1 -- start start 2023-10-30T13:25:39.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd790d6640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd7406a970 0x7fcd741ed470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd790d6640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcd74151910 0x7fcd741e3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd790d6640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcd7414f280 0x7fcd74157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd790d6640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd74155d60 con 0x7fcd7406a970 2023-10-30T13:25:39.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd790d6640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd74146360 con 0x7fcd74151910 2023-10-30T13:25:39.534 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd790d6640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd741ed9b0 con 0x7fcd7414f280 2023-10-30T13:25:39.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd72d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd7406a970 0x7fcd741ed470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd72d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd7406a970 0x7fcd741ed470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38738/0 (socket says 172.21.15.125:38738) 2023-10-30T13:25:39.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd72d9d640 1 -- 172.21.15.125:0/2103719305 learned_addr learned my addr 172.21.15.125:0/2103719305 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:39.535 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd7259c640 1 --2- 172.21.15.125:0/2103719305 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcd74151910 0x7fcd741e3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd7359e640 1 --2- 172.21.15.125:0/2103719305 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcd7414f280 0x7fcd74157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd72d9d640 1 -- 172.21.15.125:0/2103719305 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcd7414f280 msgr2=0x7fcd74157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd72d9d640 1 --2- 172.21.15.125:0/2103719305 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcd7414f280 0x7fcd74157b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd72d9d640 1 -- 172.21.15.125:0/2103719305 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcd74151910 msgr2=0x7fcd741e3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd72d9d640 1 --2- 172.21.15.125:0/2103719305 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcd74151910 0x7fcd741e3ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd72d9d640 1 -- 172.21.15.125:0/2103719305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd680094d0 con 0x7fcd7406a970 2023-10-30T13:25:39.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd72d9d640 1 --2- 172.21.15.125:0/2103719305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd7406a970 0x7fcd741ed470 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fcd6400e840 tx=0x7fcd6400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:39.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd5b7fe640 1 -- 172.21.15.125:0/2103719305 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6400c870 con 0x7fcd7406a970 2023-10-30T13:25:39.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd5b7fe640 1 -- 172.21.15.125:0/2103719305 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6400c9d0 con 0x7fcd7406a970 2023-10-30T13:25:39.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd5b7fe640 1 -- 172.21.15.125:0/2103719305 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd6400cd60 con 0x7fcd7406a970 2023-10-30T13:25:39.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd790d6640 1 -- 172.21.15.125:0/2103719305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd740ca2d0 con 0x7fcd7406a970 2023-10-30T13:25:39.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd790d6640 1 -- 172.21.15.125:0/2103719305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7414d7a0 con 0x7fcd7406a970 2023-10-30T13:25:39.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd790d6640 1 -- 172.21.15.125:0/2103719305 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7404e170 con 0x7fcd7406a970 2023-10-30T13:25:39.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.526+0000 7fcd5b7fe640 1 -- 172.21.15.125:0/2103719305 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcd64013410 con 0x7fcd7406a970 2023-10-30T13:25:39.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.530+0000 7fcd5b7fe640 1 --2- 172.21.15.125:0/2103719305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcd600664d0 0x7fcd60068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.530+0000 7fcd5b7fe640 1 -- 172.21.15.125:0/2103719305 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fcd640845f0 con 0x7fcd7406a970 2023-10-30T13:25:39.540 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.530+0000 7fcd7259c640 1 --2- 172.21.15.125:0/2103719305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcd600664d0 0x7fcd60068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.530+0000 7fcd7259c640 1 --2- 172.21.15.125:0/2103719305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcd600664d0 0x7fcd60068990 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fcd680b2000 tx=0x7fcd68009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:39.541 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.534+0000 7fcd5b7fe640 1 -- 172.21.15.125:0/2103719305 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fcd64051040 con 0x7fcd7406a970 2023-10-30T13:25:39.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.682+0000 7fcd790d6640 1 -- 172.21.15.125:0/2103719305 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd74069e90 con 0x7fcd600664d0 2023-10-30T13:25:39.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.682+0000 7fcd5b7fe640 1 -- 172.21.15.125:0/2103719305 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcd680b04c0 con 0x7fcd600664d0 2023-10-30T13:25:39.687 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 2 to deep-scrub 2023-10-30T13:25:39.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.686+0000 7fcd790d6640 1 -- 172.21.15.125:0/2103719305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcd600664d0 msgr2=0x7fcd60068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.686+0000 7fcd790d6640 1 --2- 172.21.15.125:0/2103719305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcd600664d0 0x7fcd60068990 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fcd680b2000 tx=0x7fcd68009040 comp rx=0 tx=0).stop 2023-10-30T13:25:39.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.686+0000 7fcd790d6640 1 -- 172.21.15.125:0/2103719305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd7406a970 msgr2=0x7fcd741ed470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.689 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.686+0000 7fcd790d6640 1 --2- 172.21.15.125:0/2103719305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd7406a970 0x7fcd741ed470 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7fcd6400e840 tx=0x7fcd6400ed00 comp rx=0 tx=0).stop 2023-10-30T13:25:39.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.686+0000 7fcd790d6640 1 -- 172.21.15.125:0/2103719305 shutdown_connections 2023-10-30T13:25:39.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.686+0000 7fcd790d6640 1 --2- 172.21.15.125:0/2103719305 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcd600664d0 0x7fcd60068990 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.690 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.686+0000 7fcd790d6640 1 --2- 172.21.15.125:0/2103719305 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcd7414f280 0x7fcd74157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.686+0000 7fcd790d6640 1 --2- 172.21.15.125:0/2103719305 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcd74151910 0x7fcd741e3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.686+0000 7fcd790d6640 1 --2- 172.21.15.125:0/2103719305 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcd7406a970 0x7fcd741ed470 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.686+0000 7fcd790d6640 1 -- 172.21.15.125:0/2103719305 >> 172.21.15.125:0/2103719305 conn(0x7fcd740c7910 msgr2=0x7fcd740c8cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:39.691 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.686+0000 7fcd790d6640 1 -- 172.21.15.125:0/2103719305 shutdown_connections 2023-10-30T13:25:39.692 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.686+0000 7fcd790d6640 1 -- 172.21.15.125:0/2103719305 wait complete. 2023-10-30T13:25:39.708 INFO:tasks.ceph:Scrubbing osd.3 2023-10-30T13:25:39.709 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:25:39.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.846+0000 7fcb36c54640 1 Processor -- start 2023-10-30T13:25:39.850 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.846+0000 7fcb36c54640 1 -- start start 2023-10-30T13:25:39.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.846+0000 7fcb36c54640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb3015a700 0x7fcb30156490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.846+0000 7fcb36c54640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3015a270 con 0x7fcb301569d0 2023-10-30T13:25:39.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.846+0000 7fcb36c54640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb3006c7c0 con 0x7fcb3014cf40 2023-10-30T13:25:39.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.846+0000 7fcb36c54640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb30156ed0 con 0x7fcb3015a700 2023-10-30T13:25:39.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.846+0000 7fcb27fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb3015a700 0x7fcb30156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.846+0000 7fcb27fff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fcb3015a700 0x7fcb30156490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38748/0 (socket says 172.21.15.125:38748) 2023-10-30T13:25:39.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.846+0000 7fcb27fff640 1 -- 172.21.15.125:0/4251496172 learned_addr learned my addr 172.21.15.125:0/4251496172 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:39.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.846+0000 7fcb351f1640 1 --1- 172.21.15.125:0/4251496172 >> v1:172.21.15.125:6789/0 conn(0x7fcb301569d0 0x7fcb300c6920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59628/0 (socket says 172.21.15.125:59628) 2023-10-30T13:25:39.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.846+0000 7fcb27fff640 1 -- 172.21.15.125:0/4251496172 >> v1:172.21.15.136:6789/0 conn(0x7fcb3014cf40 legacy=0x7fcb30069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.846+0000 7fcb27fff640 1 -- 172.21.15.125:0/4251496172 >> v1:172.21.15.125:6789/0 conn(0x7fcb301569d0 legacy=0x7fcb300c6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.846+0000 7fcb27fff640 1 -- 172.21.15.125:0/4251496172 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb301571b0 con 0x7fcb3015a700 2023-10-30T13:25:39.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.846+0000 7fcb277fe640 1 -- 172.21.15.125:0/4251496172 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317801393 0 0) 0x7fcb3015a270 con 0x7fcb301569d0 2023-10-30T13:25:39.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.850+0000 7fcb27fff640 1 --2- 172.21.15.125:0/4251496172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb3015a700 0x7fcb30156490 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fcb20009870 tx=0x7fcb2001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d942ded4055dd16 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:39.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.850+0000 7fcb277fe640 1 -- 172.21.15.125:0/4251496172 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb200a7070 con 0x7fcb3015a700 2023-10-30T13:25:39.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.850+0000 7fcb277fe640 1 -- 172.21.15.125:0/4251496172 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb200ab430 con 0x7fcb3015a700 2023-10-30T13:25:39.856 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.850+0000 7fcb277fe640 1 -- 172.21.15.125:0/4251496172 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb200ab7c0 con 0x7fcb3015a700 2023-10-30T13:25:39.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.850+0000 7fcb36c54640 1 -- 172.21.15.125:0/4251496172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb3015a700 msgr2=0x7fcb30156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.850+0000 7fcb36c54640 1 --2- 172.21.15.125:0/4251496172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb3015a700 0x7fcb30156490 secure :-1 s=READY pgs=2176 cs=0 l=1 rev1=1 crypto rx=0x7fcb20009870 tx=0x7fcb2001c540 comp rx=0 tx=0).stop 2023-10-30T13:25:39.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.850+0000 7fcb36c54640 1 -- 172.21.15.125:0/4251496172 shutdown_connections 2023-10-30T13:25:39.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.850+0000 7fcb36c54640 1 --2- 172.21.15.125:0/4251496172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb3015a700 0x7fcb30156490 unknown :-1 s=CLOSED pgs=2176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.850+0000 7fcb36c54640 1 -- 172.21.15.125:0/4251496172 >> 172.21.15.125:0/4251496172 conn(0x7fcb300c7910 msgr2=0x7fcb300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:39.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.850+0000 7fcb36c54640 1 -- 172.21.15.125:0/4251496172 shutdown_connections 2023-10-30T13:25:39.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.850+0000 7fcb36c54640 1 -- 172.21.15.125:0/4251496172 wait complete. 2023-10-30T13:25:39.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb36c54640 1 Processor -- start 2023-10-30T13:25:39.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb36c54640 1 -- start start 2023-10-30T13:25:39.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb36c54640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb3014cf40 0x7fcb300ca390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb36c54640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb301569d0 0x7fcb3013c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb36c54640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb3015a700 0x7fcb3013ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb36c54640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb300c9da0 con 0x7fcb3015a700 2023-10-30T13:25:39.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb36c54640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3013e420 con 0x7fcb301569d0 2023-10-30T13:25:39.862 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb36c54640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb300c6d50 con 0x7fcb3014cf40 2023-10-30T13:25:39.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb351f1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb3015a700 0x7fcb3013ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb351f1640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb3015a700 0x7fcb3013ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38762/0 (socket says 172.21.15.125:38762) 2023-10-30T13:25:39.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb351f1640 1 -- 172.21.15.125:0/3839156455 learned_addr learned my addr 172.21.15.125:0/3839156455 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:39.863 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb349f0640 1 --2- 172.21.15.125:0/3839156455 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb3014cf40 0x7fcb300ca390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb27fff640 1 --2- 172.21.15.125:0/3839156455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb301569d0 0x7fcb3013c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb351f1640 1 -- 172.21.15.125:0/3839156455 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb3014cf40 msgr2=0x7fcb300ca390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb351f1640 1 --2- 172.21.15.125:0/3839156455 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fcb3014cf40 0x7fcb300ca390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.864 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb351f1640 1 -- 172.21.15.125:0/3839156455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb301569d0 msgr2=0x7fcb3013c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb351f1640 1 --2- 172.21.15.125:0/3839156455 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fcb301569d0 0x7fcb3013c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb351f1640 1 -- 172.21.15.125:0/3839156455 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb200094d0 con 0x7fcb3015a700 2023-10-30T13:25:39.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb351f1640 1 --2- 172.21.15.125:0/3839156455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb3015a700 0x7fcb3013ccf0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fcb2800c7b0 tx=0x7fcb2800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:39.865 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb257fa640 1 -- 172.21.15.125:0/3839156455 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2800a640 con 0x7fcb3015a700 2023-10-30T13:25:39.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb257fa640 1 -- 172.21.15.125:0/3839156455 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2800a7a0 con 0x7fcb3015a700 2023-10-30T13:25:39.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb36c54640 1 -- 172.21.15.125:0/3839156455 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3013d8a0 con 0x7fcb3015a700 2023-10-30T13:25:39.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb257fa640 1 -- 172.21.15.125:0/3839156455 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2800ab50 con 0x7fcb3015a700 2023-10-30T13:25:39.866 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.854+0000 7fcb36c54640 1 -- 172.21.15.125:0/3839156455 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3013de60 con 0x7fcb3015a700 2023-10-30T13:25:39.867 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.858+0000 7fcb36c54640 1 -- 172.21.15.125:0/3839156455 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcb3014d320 con 0x7fcb3015a700 2023-10-30T13:25:39.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.858+0000 7fcb257fa640 1 -- 172.21.15.125:0/3839156455 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fcb2800acb0 con 0x7fcb3015a700 2023-10-30T13:25:39.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.858+0000 7fcb257fa640 1 --2- 172.21.15.125:0/3839156455 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb1c0664d0 0x7fcb1c068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.868 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.858+0000 7fcb257fa640 1 -- 172.21.15.125:0/3839156455 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fcb28084130 con 0x7fcb3015a700 2023-10-30T13:25:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.858+0000 7fcb257fa640 1 --2- 172.21.15.125:0/3839156455 >> [v2:172.21.15.125:6824/3730905694,v1:172.21.15.125:6825/3730905694] conn(0x7fcb1c070170 0x7fcb1c072590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.858+0000 7fcb257fa640 1 -- 172.21.15.125:0/3839156455 --> [v2:172.21.15.125:6824/3730905694,v1:172.21.15.125:6825/3730905694] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb1c072c60 con 0x7fcb1c070170 2023-10-30T13:25:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.858+0000 7fcb257fa640 1 -- 172.21.15.125:0/3839156455 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb28086790 con 0x7fcb3015a700 2023-10-30T13:25:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.858+0000 7fcb349f0640 1 --2- 172.21.15.125:0/3839156455 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb1c0664d0 0x7fcb1c068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.858+0000 7fcb27fff640 1 --2- 172.21.15.125:0/3839156455 >> [v2:172.21.15.125:6824/3730905694,v1:172.21.15.125:6825/3730905694] conn(0x7fcb1c070170 0x7fcb1c072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:39.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.858+0000 7fcb27fff640 1 --2- 172.21.15.125:0/3839156455 >> [v2:172.21.15.125:6824/3730905694,v1:172.21.15.125:6825/3730905694] conn(0x7fcb1c070170 0x7fcb1c072590 crc :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:39.870 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.858+0000 7fcb349f0640 1 --2- 172.21.15.125:0/3839156455 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb1c0664d0 0x7fcb1c068990 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c000f70 tx=0x7fcb2c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:39.871 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.862+0000 7fcb257fa640 1 -- 172.21.15.125:0/3839156455 <== osd.3 v2:172.21.15.125:6824/3730905694 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fcb1c072c60 con 0x7fcb1c070170 2023-10-30T13:25:39.876 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.874+0000 7fcb36c54640 1 -- 172.21.15.125:0/3839156455 --> [v2:172.21.15.125:6824/3730905694,v1:172.21.15.125:6825/3730905694] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fcb3004e170 con 0x7fcb1c070170 2023-10-30T13:25:39.877 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.874+0000 7fcb257fa640 1 -- 172.21.15.125:0/3839156455 <== osd.3 v2:172.21.15.125:6824/3730905694 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fcb3004e170 con 0x7fcb1c070170 2023-10-30T13:25:39.877 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2023-10-30T13:25:39.878 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:25:39.878 INFO:teuthology.orchestra.run.smithi125.stdout:} 2023-10-30T13:25:39.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.874+0000 7fcb36c54640 1 -- 172.21.15.125:0/3839156455 >> [v2:172.21.15.125:6824/3730905694,v1:172.21.15.125:6825/3730905694] conn(0x7fcb1c070170 msgr2=0x7fcb1c072590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.879 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.874+0000 7fcb36c54640 1 --2- 172.21.15.125:0/3839156455 >> [v2:172.21.15.125:6824/3730905694,v1:172.21.15.125:6825/3730905694] conn(0x7fcb1c070170 0x7fcb1c072590 crc :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:39.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.874+0000 7fcb36c54640 1 -- 172.21.15.125:0/3839156455 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb1c0664d0 msgr2=0x7fcb1c068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.874+0000 7fcb36c54640 1 --2- 172.21.15.125:0/3839156455 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fcb1c0664d0 0x7fcb1c068990 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c000f70 tx=0x7fcb2c009040 comp rx=0 tx=0).stop 2023-10-30T13:25:39.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.874+0000 7fcb36c54640 1 -- 172.21.15.125:0/3839156455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb3015a700 msgr2=0x7fcb3013ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:39.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.874+0000 7fcb36c54640 1 --2- 172.21.15.125:0/3839156455 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fcb3015a700 0x7fcb3013ccf0 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7fcb2800c7b0 tx=0x7fcb2800cc70 comp rx=0 tx=0).stop 2023-10-30T13:25:39.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.874+0000 7fcb351f1640 1 -- 172.21.15.125:0/3839156455 reap_dead start 2023-10-30T13:25:39.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.874+0000 7fcb36c54640 1 -- 172.21.15.125:0/3839156455 shutdown_connections 2023-10-30T13:25:39.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.874+0000 7fcb36c54640 1 -- 172.21.15.125:0/3839156455 >> 172.21.15.125:0/3839156455 conn(0x7fcb300c7910 msgr2=0x7fcb301434b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:39.881 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.874+0000 7fcb36c54640 1 -- 172.21.15.125:0/3839156455 shutdown_connections 2023-10-30T13:25:39.882 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:39.874+0000 7fcb36c54640 1 -- 172.21.15.125:0/3839156455 wait complete. 2023-10-30T13:25:39.896 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-30T13:25:40.036 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e174c9640 1 Processor -- start 2023-10-30T13:25:40.037 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e174c9640 1 -- start start 2023-10-30T13:25:40.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e174c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e10070840 0x7f8e10070c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.038 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e174c9640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8e10069e90 0x7f8e1006a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e174c9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e100c6880 con 0x7f8e10153810 2023-10-30T13:25:40.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e174c9640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e100711c0 con 0x7f8e10069e90 2023-10-30T13:25:40.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e174c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e100c5990 con 0x7f8e10070840 2023-10-30T13:25:40.039 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e15a66640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8e10069e90 0x7f8e1006a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e15a66640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f8e10069e90 0x7f8e1006a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38766/0 (socket says 172.21.15.125:38766) 2023-10-30T13:25:40.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e15a66640 1 -- 172.21.15.125:0/3523473377 learned_addr learned my addr 172.21.15.125:0/3523473377 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:40.040 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e14a64640 1 --2- 172.21.15.125:0/3523473377 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e10070840 0x7f8e10070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e15a66640 1 -- 172.21.15.125:0/3523473377 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e10070840 msgr2=0x7f8e10070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:40.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e15a66640 1 --2- 172.21.15.125:0/3523473377 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e10070840 0x7f8e10070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e15a66640 1 -- 172.21.15.125:0/3523473377 >> v1:172.21.15.136:6789/0 conn(0x7f8e10153810 legacy=0x7f8e1015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.041 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.034+0000 7f8e15a66640 1 -- 172.21.15.125:0/3523473377 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e100c5c70 con 0x7f8e10069e90 2023-10-30T13:25:40.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.038+0000 7f8e15a66640 1 --2- 172.21.15.125:0/3523473377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e10069e90 0x7f8e1006a300 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009e50 tx=0x7f8e0801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efdc98e4b804e7b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:40.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.038+0000 7f8e07fff640 1 -- 172.21.15.125:0/3523473377 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e080af400 con 0x7f8e10069e90 2023-10-30T13:25:40.043 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.038+0000 7f8e07fff640 1 -- 172.21.15.125:0/3523473377 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e080ab050 con 0x7f8e10069e90 2023-10-30T13:25:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.038+0000 7f8e07fff640 1 -- 172.21.15.125:0/3523473377 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e080af780 con 0x7f8e10069e90 2023-10-30T13:25:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.038+0000 7f8e174c9640 1 -- 172.21.15.125:0/3523473377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e10069e90 msgr2=0x7f8e1006a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.038+0000 7f8e174c9640 1 --2- 172.21.15.125:0/3523473377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e10069e90 0x7f8e1006a300 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009e50 tx=0x7f8e0801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.044 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.038+0000 7f8e174c9640 1 -- 172.21.15.125:0/3523473377 shutdown_connections 2023-10-30T13:25:40.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.038+0000 7f8e174c9640 1 --2- 172.21.15.125:0/3523473377 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e10069e90 0x7f8e1006a300 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.038+0000 7f8e174c9640 1 --2- 172.21.15.125:0/3523473377 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e10070840 0x7f8e10070c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.045 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.038+0000 7f8e174c9640 1 -- 172.21.15.125:0/3523473377 >> 172.21.15.125:0/3523473377 conn(0x7f8e100c7910 msgr2=0x7f8e100c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:40.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.038+0000 7f8e174c9640 1 -- 172.21.15.125:0/3523473377 shutdown_connections 2023-10-30T13:25:40.046 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.038+0000 7f8e174c9640 1 -- 172.21.15.125:0/3523473377 wait complete. 2023-10-30T13:25:40.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.038+0000 7f8e174c9640 1 Processor -- start 2023-10-30T13:25:40.048 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.038+0000 7f8e174c9640 1 -- start start 2023-10-30T13:25:40.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e174c9640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e10069e90 0x7f8e101584c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e174c9640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e10070840 0x7f8e10158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.049 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e174c9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e10153810 0x7f8e101f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e174c9640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e1015a170 con 0x7f8e10153810 2023-10-30T13:25:40.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e174c9640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e100c1810 con 0x7f8e10070840 2023-10-30T13:25:40.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e174c9640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e10159300 con 0x7f8e10069e90 2023-10-30T13:25:40.050 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e15a66640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e10153810 0x7f8e101f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e15a66640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e10153810 0x7f8e101f2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38774/0 (socket says 172.21.15.125:38774) 2023-10-30T13:25:40.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e15a66640 1 -- 172.21.15.125:0/14797917 learned_addr learned my addr 172.21.15.125:0/14797917 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:40.051 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e15265640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e10069e90 0x7f8e101584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e14a64640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e10070840 0x7f8e10158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e15a66640 1 -- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e10069e90 msgr2=0x7f8e101584c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.052 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e15a66640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e10069e90 0x7f8e101584c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e15a66640 1 -- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e10070840 msgr2=0x7f8e10158a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e15a66640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e10070840 0x7f8e10158a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e15a66640 1 -- 172.21.15.125:0/14797917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e08009b30 con 0x7f8e10153810 2023-10-30T13:25:40.053 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e14a64640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e10070840 0x7f8e10158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:40.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e15a66640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e10153810 0x7f8e101f2690 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f8e0801ce20 tx=0x7f8e0801bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:40.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e05ffb640 1 -- 172.21.15.125:0/14797917 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e080afde0 con 0x7f8e10153810 2023-10-30T13:25:40.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e05ffb640 1 -- 172.21.15.125:0/14797917 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e08009510 con 0x7f8e10153810 2023-10-30T13:25:40.054 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e174c9640 1 -- 172.21.15.125:0/14797917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e10159550 con 0x7f8e10153810 2023-10-30T13:25:40.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e05ffb640 1 -- 172.21.15.125:0/14797917 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e080ba680 con 0x7f8e10153810 2023-10-30T13:25:40.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e174c9640 1 -- 172.21.15.125:0/14797917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e10159bb0 con 0x7f8e10153810 2023-10-30T13:25:40.055 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.042+0000 7f8e174c9640 1 -- 172.21.15.125:0/14797917 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e1004e170 con 0x7f8e10153810 2023-10-30T13:25:40.056 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.046+0000 7f8e05ffb640 1 -- 172.21.15.125:0/14797917 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f8e080b6040 con 0x7f8e10153810 2023-10-30T13:25:40.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.046+0000 7f8e05ffb640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8df40664d0 0x7f8df4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.046+0000 7f8e05ffb640 1 -- 172.21.15.125:0/14797917 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f8e081229f0 con 0x7f8e10153810 2023-10-30T13:25:40.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.046+0000 7f8e15265640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8df40664d0 0x7f8df4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.057 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.046+0000 7f8e15265640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8df40664d0 0x7f8df4068990 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c007fd0 tx=0x7f8e0c009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:40.058 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.050+0000 7f8e05ffb640 1 -- 172.21.15.125:0/14797917 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f8e080ef510 con 0x7f8e10153810 2023-10-30T13:25:40.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.202+0000 7f8e174c9640 1 -- 172.21.15.125:0/14797917 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e100c6230 con 0x7f8df40664d0 2023-10-30T13:25:40.206 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.202+0000 7f8e05ffb640 1 -- 172.21.15.125:0/14797917 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8e0c0014f0 con 0x7f8df40664d0 2023-10-30T13:25:40.206 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 3 to deep-scrub 2023-10-30T13:25:40.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.206+0000 7f8e174c9640 1 -- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8df40664d0 msgr2=0x7f8df4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.206+0000 7f8e174c9640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8df40664d0 0x7f8df4068990 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c007fd0 tx=0x7f8e0c009040 comp rx=0 tx=0).stop 2023-10-30T13:25:40.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.206+0000 7f8e174c9640 1 -- 172.21.15.125:0/14797917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e10153810 msgr2=0x7f8e101f2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.209 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.206+0000 7f8e174c9640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e10153810 0x7f8e101f2690 secure :-1 s=READY pgs=2179 cs=0 l=1 rev1=1 crypto rx=0x7f8e0801ce20 tx=0x7f8e0801bf90 comp rx=0 tx=0).stop 2023-10-30T13:25:40.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.206+0000 7f8e174c9640 1 -- 172.21.15.125:0/14797917 shutdown_connections 2023-10-30T13:25:40.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.206+0000 7f8e174c9640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f8df40664d0 0x7f8df4068990 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.210 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.206+0000 7f8e174c9640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f8e10153810 0x7f8e101f2690 unknown :-1 s=CLOSED pgs=2179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.206+0000 7f8e174c9640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f8e10070840 0x7f8e10158a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.206+0000 7f8e174c9640 1 --2- 172.21.15.125:0/14797917 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f8e10069e90 0x7f8e101584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.206+0000 7f8e174c9640 1 -- 172.21.15.125:0/14797917 >> 172.21.15.125:0/14797917 conn(0x7f8e100c7910 msgr2=0x7f8e101519a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:40.211 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.206+0000 7f8e174c9640 1 -- 172.21.15.125:0/14797917 shutdown_connections 2023-10-30T13:25:40.212 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.206+0000 7f8e174c9640 1 -- 172.21.15.125:0/14797917 wait complete. 2023-10-30T13:25:40.228 INFO:tasks.ceph:Scrubbing osd.4 2023-10-30T13:25:40.229 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:25:40.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.382+0000 7f5bdf103640 1 Processor -- start 2023-10-30T13:25:40.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.382+0000 7f5bdf103640 1 -- start start 2023-10-30T13:25:40.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.386+0000 7f5bdf103640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5bd815cd50 0x7f5bd80c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.386+0000 7f5bdf103640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f5bd80cefa0 0x7f5bd80cf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.386+0000 7f5bdf103640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd815a6c0 con 0x7f5bd81538f0 2023-10-30T13:25:40.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.386+0000 7f5bdf103640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd815d300 con 0x7f5bd80cefa0 2023-10-30T13:25:40.390 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.386+0000 7f5bdf103640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd815d5e0 con 0x7f5bd815cd50 2023-10-30T13:25:40.391 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.386+0000 7f5bdd900640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5bd815cd50 0x7f5bd80c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.386+0000 7f5bdd900640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5bd815cd50 0x7f5bd80c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3301/0 says I am v2:172.21.15.125:48322/0 (socket says 172.21.15.125:48322) 2023-10-30T13:25:40.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.386+0000 7f5bdd900640 1 -- 172.21.15.125:0/1740487507 learned_addr learned my addr 172.21.15.125:0/1740487507 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:40.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.386+0000 7f5bde902640 1 --2- 172.21.15.125:0/1740487507 >> v2:172.21.15.136:3300/0 conn(0x7f5bd80cefa0 0x7f5bd80cf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.392 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.386+0000 7f5bdd900640 1 -- 172.21.15.125:0/1740487507 >> v2:172.21.15.136:3300/0 conn(0x7f5bd80cefa0 msgr2=0x7f5bd80cf3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:40.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.386+0000 7f5bdd900640 1 --2- 172.21.15.125:0/1740487507 >> v2:172.21.15.136:3300/0 conn(0x7f5bd80cefa0 0x7f5bd80cf3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.386+0000 7f5bdd900640 1 -- 172.21.15.125:0/1740487507 >> v1:172.21.15.136:6789/0 conn(0x7f5bd81538f0 legacy=0x7f5bd8159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.393 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.386+0000 7f5bdd900640 1 -- 172.21.15.125:0/1740487507 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd815d8c0 con 0x7f5bd815cd50 2023-10-30T13:25:40.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.390+0000 7f5bdd900640 1 --2- 172.21.15.125:0/1740487507 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5bd815cd50 0x7f5bd80c9f90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009870 tx=0x7f5bc801ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a09a0dd4c19e6eda server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:40.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.390+0000 7f5bdd0ff640 1 -- 172.21.15.125:0/1740487507 <== mon.2 v2:172.21.15.136:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc80a8a30 con 0x7f5bd815cd50 2023-10-30T13:25:40.394 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.390+0000 7f5bdd0ff640 1 -- 172.21.15.125:0/1740487507 <== mon.2 v2:172.21.15.136:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc80a8b90 con 0x7f5bd815cd50 2023-10-30T13:25:40.395 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.390+0000 7f5bdd0ff640 1 -- 172.21.15.125:0/1740487507 <== mon.2 v2:172.21.15.136:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc80b0440 con 0x7f5bd815cd50 2023-10-30T13:25:40.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.390+0000 7f5bdf103640 1 -- 172.21.15.125:0/1740487507 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5bd815cd50 msgr2=0x7f5bd80c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.390+0000 7f5bdf103640 1 --2- 172.21.15.125:0/1740487507 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5bd815cd50 0x7f5bd80c9f90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5bc8009870 tx=0x7f5bc801ed30 comp rx=0 tx=0).stop 2023-10-30T13:25:40.396 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.390+0000 7f5bdf103640 1 -- 172.21.15.125:0/1740487507 shutdown_connections 2023-10-30T13:25:40.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.390+0000 7f5bdf103640 1 --2- 172.21.15.125:0/1740487507 >> v2:172.21.15.136:3300/0 conn(0x7f5bd80cefa0 0x7f5bd80cf3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.390+0000 7f5bdf103640 1 --2- 172.21.15.125:0/1740487507 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5bd815cd50 0x7f5bd80c9f90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.390+0000 7f5bdf103640 1 -- 172.21.15.125:0/1740487507 >> 172.21.15.125:0/1740487507 conn(0x7f5bd806a640 msgr2=0x7f5bd8065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:40.397 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.390+0000 7f5bdf103640 1 -- 172.21.15.125:0/1740487507 shutdown_connections 2023-10-30T13:25:40.398 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.390+0000 7f5bdf103640 1 -- 172.21.15.125:0/1740487507 wait complete. 2023-10-30T13:25:40.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bdf103640 1 Processor -- start 2023-10-30T13:25:40.400 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bdf103640 1 -- start start 2023-10-30T13:25:40.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bdf103640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bd80cefa0 0x7f5bd8151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bdf103640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5bd81538f0 0x7f5bd81420a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bdf103640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bd815cd50 0x7f5bd8147e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.401 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bdf103640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd80b4470 con 0x7f5bd80cefa0 2023-10-30T13:25:40.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bdf103640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd80b2f50 con 0x7f5bd815cd50 2023-10-30T13:25:40.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bdf103640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd80b3d30 con 0x7f5bd81538f0 2023-10-30T13:25:40.402 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bde101640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bd80cefa0 0x7f5bd8151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bde101640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bd80cefa0 0x7f5bd8151b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:38784/0 (socket says 172.21.15.125:38784) 2023-10-30T13:25:40.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bde101640 1 -- 172.21.15.125:0/4181391027 learned_addr learned my addr 172.21.15.125:0/4181391027 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:40.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bdd900640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5bd81538f0 0x7f5bd81420a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.403 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bde902640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bd815cd50 0x7f5bd8147e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bde101640 1 -- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5bd81538f0 msgr2=0x7f5bd81420a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bde101640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5bd81538f0 0x7f5bd81420a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bde101640 1 -- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bd815cd50 msgr2=0x7f5bd8147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.404 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bde101640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bd815cd50 0x7f5bd8147e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bde101640 1 -- 172.21.15.125:0/4181391027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc80094d0 con 0x7f5bd80cefa0 2023-10-30T13:25:40.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bde902640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5bd815cd50 0x7f5bd8147e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:40.405 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bde101640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bd80cefa0 0x7f5bd8151b00 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f5bd000c530 tx=0x7f5bd000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:40.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bceffd640 1 -- 172.21.15.125:0/4181391027 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0005220 con 0x7f5bd80cefa0 2023-10-30T13:25:40.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bceffd640 1 -- 172.21.15.125:0/4181391027 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd0005380 con 0x7f5bd80cefa0 2023-10-30T13:25:40.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bdf103640 1 -- 172.21.15.125:0/4181391027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd80b3180 con 0x7f5bd80cefa0 2023-10-30T13:25:40.406 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bceffd640 1 -- 172.21.15.125:0/4181391027 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0007270 con 0x7f5bd80cefa0 2023-10-30T13:25:40.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.394+0000 7f5bdf103640 1 -- 172.21.15.125:0/4181391027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd8142dd0 con 0x7f5bd80cefa0 2023-10-30T13:25:40.407 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.398+0000 7f5bdf103640 1 -- 172.21.15.125:0/4181391027 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5bd8159a90 con 0x7f5bd80cefa0 2023-10-30T13:25:40.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.398+0000 7f5bceffd640 1 -- 172.21.15.125:0/4181391027 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5bd00073d0 con 0x7f5bd80cefa0 2023-10-30T13:25:40.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.398+0000 7f5bceffd640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5bbc0664d0 0x7f5bbc068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.398+0000 7f5bceffd640 1 -- 172.21.15.125:0/4181391027 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f5bd0084150 con 0x7f5bd80cefa0 2023-10-30T13:25:40.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.398+0000 7f5bceffd640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7f5bbc070170 0x7f5bbc072590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.409 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.398+0000 7f5bceffd640 1 -- 172.21.15.125:0/4181391027 --> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5bbc072c60 con 0x7f5bbc070170 2023-10-30T13:25:40.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.398+0000 7f5bceffd640 1 -- 172.21.15.125:0/4181391027 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5bd00867b0 con 0x7f5bd80cefa0 2023-10-30T13:25:40.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.398+0000 7f5bdd900640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5bbc0664d0 0x7f5bbc068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.410 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.398+0000 7f5bde902640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7f5bbc070170 0x7f5bbc072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.398+0000 7f5bde902640 1 -- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7f5bbc070170 msgr2=0x7f5bbc072590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-10-30T13:25:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.398+0000 7f5bde902640 1 -- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7f5bbc070170 msgr2=0x7f5bbc072590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-10-30T13:25:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.398+0000 7f5bde902640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7f5bbc070170 0x7f5bbc072590 crc :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-30T13:25:40.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.398+0000 7f5bde902640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7f5bbc070170 0x7f5bbc072590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-30T13:25:40.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.398+0000 7f5bdd900640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5bbc0664d0 0x7f5bbc068990 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80b2000 tx=0x7f5bc80aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:40.604 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.602+0000 7f5bde902640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7f5bbc070170 0x7f5bbc072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.605 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.602+0000 7f5bde902640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7f5bbc070170 0x7f5bbc072590 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:40.606 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.602+0000 7f5bceffd640 1 -- 172.21.15.125:0/4181391027 <== osd.4 v2:172.21.15.136:6817/4208991530 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f5bbc072c60 con 0x7f5bbc070170 2023-10-30T13:25:40.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.614+0000 7f5bdf103640 1 -- 172.21.15.125:0/4181391027 --> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f5bd815dba0 con 0x7f5bbc070170 2023-10-30T13:25:40.619 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.618+0000 7f5bceffd640 1 -- 172.21.15.125:0/4181391027 <== osd.4 v2:172.21.15.136:6817/4208991530 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f5bd400f040 con 0x7f5bbc070170 2023-10-30T13:25:40.620 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2023-10-30T13:25:40.620 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:25:40.621 INFO:teuthology.orchestra.run.smithi125.stdout:} 2023-10-30T13:25:40.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.618+0000 7f5bdf103640 1 -- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7f5bbc070170 msgr2=0x7f5bbc072590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.618+0000 7f5bdf103640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6817/4208991530,v1:172.21.15.136:6819/4208991530] conn(0x7f5bbc070170 0x7f5bbc072590 crc :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.618+0000 7f5bdf103640 1 -- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5bbc0664d0 msgr2=0x7f5bbc068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.618+0000 7f5bdf103640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f5bbc0664d0 0x7f5bbc068990 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f5bc80b2000 tx=0x7f5bc80aa070 comp rx=0 tx=0).stop 2023-10-30T13:25:40.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.618+0000 7f5bdf103640 1 -- 172.21.15.125:0/4181391027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bd80cefa0 msgr2=0x7f5bd8151b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.618+0000 7f5bdf103640 1 --2- 172.21.15.125:0/4181391027 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f5bd80cefa0 0x7f5bd8151b00 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f5bd000c530 tx=0x7f5bd000c9f0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.618+0000 7f5bde902640 1 -- 172.21.15.125:0/4181391027 reap_dead start 2023-10-30T13:25:40.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.618+0000 7f5bdf103640 1 -- 172.21.15.125:0/4181391027 shutdown_connections 2023-10-30T13:25:40.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.618+0000 7f5bdf103640 1 -- 172.21.15.125:0/4181391027 >> 172.21.15.125:0/4181391027 conn(0x7f5bd806a640 msgr2=0x7f5bd80c4c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:40.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.618+0000 7f5bdf103640 1 -- 172.21.15.125:0/4181391027 shutdown_connections 2023-10-30T13:25:40.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.618+0000 7f5bdf103640 1 -- 172.21.15.125:0/4181391027 wait complete. 2023-10-30T13:25:40.638 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-30T13:25:40.777 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd9d9d640 1 Processor -- start 2023-10-30T13:25:40.778 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd9d9d640 1 -- start start 2023-10-30T13:25:40.779 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd9d9d640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6cd40c4430 0x7f6cd40ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd9d9d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f6cd4159830 0x7f6cd4159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd9d9d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd40cb780 con 0x7f6cd4153c60 2023-10-30T13:25:40.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd9d9d640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd4154040 con 0x7f6cd40c4430 2023-10-30T13:25:40.780 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd9d9d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd415a2e0 con 0x7f6cd4159830 2023-10-30T13:25:40.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd2ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6cd40c4430 0x7f6cd40ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd2ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f6cd40c4430 0x7f6cd40ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56570/0 (socket says 172.21.15.125:56570) 2023-10-30T13:25:40.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd2ffd640 1 -- 172.21.15.125:0/3786874975 learned_addr learned my addr 172.21.15.125:0/3786874975 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:40.781 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd3fff640 1 --2- 172.21.15.125:0/3786874975 >> v2:172.21.15.136:3300/0 conn(0x7f6cd4159830 0x7f6cd4159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd2ffd640 1 -- 172.21.15.125:0/3786874975 >> v2:172.21.15.136:3300/0 conn(0x7f6cd4159830 msgr2=0x7f6cd4159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:40.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd2ffd640 1 --2- 172.21.15.125:0/3786874975 >> v2:172.21.15.136:3300/0 conn(0x7f6cd4159830 0x7f6cd4159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.782 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd2ffd640 1 -- 172.21.15.125:0/3786874975 >> v1:172.21.15.136:6789/0 conn(0x7f6cd4153c60 legacy=0x7f6cd40cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:40.783 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.774+0000 7f6cd2ffd640 1 -- 172.21.15.125:0/3786874975 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd415a5c0 con 0x7f6cd40c4430 2023-10-30T13:25:40.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.778+0000 7f6cd2ffd640 1 --2- 172.21.15.125:0/3786874975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cd40c4430 0x7f6cd40ca360 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4073da0 tx=0x7f6cc801c4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f0c410dfed6c29c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:40.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.778+0000 7f6cd27fc640 1 -- 172.21.15.125:0/3786874975 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc80ac410 con 0x7f6cd40c4430 2023-10-30T13:25:40.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.778+0000 7f6cd27fc640 1 -- 172.21.15.125:0/3786874975 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc80ac570 con 0x7f6cd40c4430 2023-10-30T13:25:40.784 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.778+0000 7f6cd27fc640 1 -- 172.21.15.125:0/3786874975 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc80ac900 con 0x7f6cd40c4430 2023-10-30T13:25:40.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.778+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/3786874975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cd40c4430 msgr2=0x7f6cd40ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.778+0000 7f6cd9d9d640 1 --2- 172.21.15.125:0/3786874975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cd40c4430 0x7f6cd40ca360 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f6cd4073da0 tx=0x7f6cc801c4c0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.778+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/3786874975 shutdown_connections 2023-10-30T13:25:40.785 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.778+0000 7f6cd9d9d640 1 --2- 172.21.15.125:0/3786874975 >> v2:172.21.15.136:3300/0 conn(0x7f6cd4159830 0x7f6cd4159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.778+0000 7f6cd9d9d640 1 --2- 172.21.15.125:0/3786874975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cd40c4430 0x7f6cd40ca360 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.778+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/3786874975 >> 172.21.15.125:0/3786874975 conn(0x7f6cd406bfe0 msgr2=0x7f6cd406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:40.786 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.778+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/3786874975 shutdown_connections 2023-10-30T13:25:40.787 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.778+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/3786874975 wait complete. 2023-10-30T13:25:40.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd9d9d640 1 Processor -- start 2023-10-30T13:25:40.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd9d9d640 1 -- start start 2023-10-30T13:25:40.789 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd9d9d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cd40c4430 0x7f6cd40cf230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd9d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cd4153c60 0x7f6cd40cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd9d9d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cd4159830 0x7f6cd41f5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.790 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd9d9d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd40cd7d0 con 0x7f6cd4153c60 2023-10-30T13:25:40.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd9d9d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd41f5580 con 0x7f6cd40c4430 2023-10-30T13:25:40.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd9d9d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd40cd4f0 con 0x7f6cd4159830 2023-10-30T13:25:40.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd2ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cd4153c60 0x7f6cd40cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.791 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd2ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cd4153c60 0x7f6cd40cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56586/0 (socket says 172.21.15.125:56586) 2023-10-30T13:25:40.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd2ffd640 1 -- 172.21.15.125:0/2856942304 learned_addr learned my addr 172.21.15.125:0/2856942304 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:40.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd37fe640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cd40c4430 0x7f6cd40cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.792 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd3fff640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cd4159830 0x7f6cd41f5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd2ffd640 1 -- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cd4159830 msgr2=0x7f6cd41f5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd2ffd640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cd4159830 0x7f6cd41f5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd2ffd640 1 -- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cd40c4430 msgr2=0x7f6cd40cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd2ffd640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cd40c4430 0x7f6cd40cf230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd2ffd640 1 -- 172.21.15.125:0/2856942304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc80094d0 con 0x7f6cd4153c60 2023-10-30T13:25:40.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd37fe640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cd40c4430 0x7f6cd40cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:40.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd2ffd640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cd4153c60 0x7f6cd40cdd20 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f6cc80af040 tx=0x7f6cc8019260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:40.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cbbfff640 1 -- 172.21.15.125:0/2856942304 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc80acea0 con 0x7f6cd4153c60 2023-10-30T13:25:40.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cbbfff640 1 -- 172.21.15.125:0/2856942304 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc80b5630 con 0x7f6cd4153c60 2023-10-30T13:25:40.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cbbfff640 1 -- 172.21.15.125:0/2856942304 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc80ac410 con 0x7f6cd4153c60 2023-10-30T13:25:40.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/2856942304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd4146100 con 0x7f6cd4153c60 2023-10-30T13:25:40.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.782+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/2856942304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd41466c0 con 0x7f6cd4153c60 2023-10-30T13:25:40.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.786+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/2856942304 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd404e170 con 0x7f6cd4153c60 2023-10-30T13:25:40.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.786+0000 7f6cbbfff640 1 -- 172.21.15.125:0/2856942304 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f6cc80b3070 con 0x7f6cd4153c60 2023-10-30T13:25:40.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.786+0000 7f6cbbfff640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cc00664a0 0x7f6cc0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:40.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.786+0000 7f6cbbfff640 1 -- 172.21.15.125:0/2856942304 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f6cc811f0a0 con 0x7f6cd4153c60 2023-10-30T13:25:40.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.786+0000 7f6cd37fe640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cc00664a0 0x7f6cc0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:40.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.786+0000 7f6cd37fe640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cc00664a0 0x7f6cc0068960 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4003ec0 tx=0x7f6cc4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:40.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.790+0000 7f6cbbfff640 1 -- 172.21.15.125:0/2856942304 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f6cc80eb9e0 con 0x7f6cd4153c60 2023-10-30T13:25:40.942 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.938+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/2856942304 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cd40cbf90 con 0x7f6cc00664a0 2023-10-30T13:25:40.943 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.938+0000 7f6cbbfff640 1 -- 172.21.15.125:0/2856942304 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6cc4003ab0 con 0x7f6cc00664a0 2023-10-30T13:25:40.943 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 4 to deep-scrub 2023-10-30T13:25:40.945 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.942+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cc00664a0 msgr2=0x7f6cc0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.942+0000 7f6cd9d9d640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cc00664a0 0x7f6cc0068960 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f6cc4003ec0 tx=0x7f6cc4009040 comp rx=0 tx=0).stop 2023-10-30T13:25:40.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.942+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/2856942304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cd4153c60 msgr2=0x7f6cd40cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:40.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.942+0000 7f6cd9d9d640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cd4153c60 0x7f6cd40cdd20 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f6cc80af040 tx=0x7f6cc8019260 comp rx=0 tx=0).stop 2023-10-30T13:25:40.946 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.942+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/2856942304 shutdown_connections 2023-10-30T13:25:40.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.942+0000 7f6cd9d9d640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f6cc00664a0 0x7f6cc0068960 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.942+0000 7f6cd9d9d640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f6cd4159830 0x7f6cd41f5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.947 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.942+0000 7f6cd9d9d640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f6cd4153c60 0x7f6cd40cdd20 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.942+0000 7f6cd9d9d640 1 --2- 172.21.15.125:0/2856942304 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f6cd40c4430 0x7f6cd40cf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:40.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.942+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/2856942304 >> 172.21.15.125:0/2856942304 conn(0x7f6cd406bfe0 msgr2=0x7f6cd40c4810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:40.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.942+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/2856942304 shutdown_connections 2023-10-30T13:25:40.948 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:40.942+0000 7f6cd9d9d640 1 -- 172.21.15.125:0/2856942304 wait complete. 2023-10-30T13:25:40.965 INFO:tasks.ceph:Scrubbing osd.5 2023-10-30T13:25:40.965 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:25:41.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.106+0000 7fdb3cbb8640 1 Processor -- start 2023-10-30T13:25:41.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.106+0000 7fdb3cbb8640 1 -- start start 2023-10-30T13:25:41.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb3cbb8640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdb38156ed0 0x7fdb380c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb3cbb8640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb380c6a50 con 0x7fdb3806f010 2023-10-30T13:25:41.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb3cbb8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb380c4900 con 0x7fdb380c6eb0 2023-10-30T13:25:41.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb3cbb8640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb380c6630 con 0x7fdb38156ed0 2023-10-30T13:25:41.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb36d9d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdb3806f010 0x7fdb38164120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54392/0 (socket says 172.21.15.125:54392) 2023-10-30T13:25:41.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb36d9d640 1 -- 172.21.15.125:0/1047275242 learned_addr learned my addr 172.21.15.125:0/1047275242 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:25:41.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb3659c640 1 --2- 172.21.15.125:0/1047275242 >> v2:172.21.15.136:3300/0 conn(0x7fdb38156ed0 0x7fdb380c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 -- 172.21.15.125:0/1047275242 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398736362 0 0) 0x7fdb380c6a50 con 0x7fdb3806f010 2023-10-30T13:25:41.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 -- 172.21.15.125:0/1047275242 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb24003530 con 0x7fdb3806f010 2023-10-30T13:25:41.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 -- 172.21.15.125:0/1047275242 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 41029518 0 0) 0x7fdb24003530 con 0x7fdb3806f010 2023-10-30T13:25:41.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 -- 172.21.15.125:0/1047275242 >> v2:172.21.15.136:3300/0 conn(0x7fdb38156ed0 msgr2=0x7fdb380c43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:41.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 --2- 172.21.15.125:0/1047275242 >> v2:172.21.15.136:3300/0 conn(0x7fdb38156ed0 0x7fdb380c43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 -- 172.21.15.125:0/1047275242 >> v1:172.21.15.136:6789/0 conn(0x7fdb380c6eb0 legacy=0x7fdb380c2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 -- 172.21.15.125:0/1047275242 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb380c3750 con 0x7fdb3806f010 2023-10-30T13:25:41.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 -- 172.21.15.125:0/1047275242 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fdb30001a00 con 0x7fdb3806f010 2023-10-30T13:25:41.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 -- 172.21.15.125:0/1047275242 >> v1:172.21.15.125:6789/0 conn(0x7fdb3806f010 legacy=0x7fdb38164120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.118 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 --2- 172.21.15.125:0/1047275242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb24003980 0x7fdb24003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 --2- 172.21.15.125:0/1047275242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb240047d0 0x7fdb24004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 --2- 172.21.15.125:0/1047275242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb24005190 0x7fdb2400b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 -- 172.21.15.125:0/1047275242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb380c6630 con 0x7fdb240047d0 2023-10-30T13:25:41.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 -- 172.21.15.125:0/1047275242 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2400c030 con 0x7fdb24005190 2023-10-30T13:25:41.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb35d9b640 1 -- 172.21.15.125:0/1047275242 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2400c1f0 con 0x7fdb24003980 2023-10-30T13:25:41.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb2ffff640 1 --2- 172.21.15.125:0/1047275242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb240047d0 0x7fdb24004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb36d9d640 1 --2- 172.21.15.125:0/1047275242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb24005190 0x7fdb2400b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb3659c640 1 --2- 172.21.15.125:0/1047275242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb24003980 0x7fdb24003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb2ffff640 1 -- 172.21.15.125:0/1047275242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb24003980 msgr2=0x7fdb24003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:41.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb2ffff640 1 --2- 172.21.15.125:0/1047275242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb24003980 0x7fdb24003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb2ffff640 1 -- 172.21.15.125:0/1047275242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb24005190 msgr2=0x7fdb2400b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:41.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb2ffff640 1 --2- 172.21.15.125:0/1047275242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb24005190 0x7fdb2400b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb2ffff640 1 -- 172.21.15.125:0/1047275242 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb24003530 con 0x7fdb240047d0 2023-10-30T13:25:41.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb3659c640 1 --2- 172.21.15.125:0/1047275242 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb24003980 0x7fdb24003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:25:41.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb36d9d640 1 --2- 172.21.15.125:0/1047275242 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb24005190 0x7fdb2400b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:41.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.110+0000 7fdb36d9d640 1 -- 172.21.15.125:0/1047275242 reap_dead start 2023-10-30T13:25:41.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.114+0000 7fdb2ffff640 1 --2- 172.21.15.125:0/1047275242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb240047d0 0x7fdb24004c50 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fdb20001750 tx=0x7fdb2001c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c57a5642b2171d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:41.126 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.114+0000 7fdb35d9b640 1 -- 172.21.15.125:0/1047275242 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb200052d0 con 0x7fdb240047d0 2023-10-30T13:25:41.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.114+0000 7fdb35d9b640 1 -- 172.21.15.125:0/1047275242 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb200a7050 con 0x7fdb240047d0 2023-10-30T13:25:41.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.114+0000 7fdb35d9b640 1 -- 172.21.15.125:0/1047275242 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb20003600 con 0x7fdb240047d0 2023-10-30T13:25:41.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.114+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/1047275242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb240047d0 msgr2=0x7fdb24004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.114+0000 7fdb3cbb8640 1 --2- 172.21.15.125:0/1047275242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb240047d0 0x7fdb24004c50 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7fdb20001750 tx=0x7fdb2001c3a0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.114+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/1047275242 shutdown_connections 2023-10-30T13:25:41.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.114+0000 7fdb3cbb8640 1 --2- 172.21.15.125:0/1047275242 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb240047d0 0x7fdb24004c50 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.114+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/1047275242 >> 172.21.15.125:0/1047275242 conn(0x7fdb380ccf80 msgr2=0x7fdb380cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:41.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.114+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/1047275242 shutdown_connections 2023-10-30T13:25:41.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.114+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/1047275242 wait complete. 2023-10-30T13:25:41.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb3cbb8640 1 Processor -- start 2023-10-30T13:25:41.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb3cbb8640 1 -- start start 2023-10-30T13:25:41.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb3cbb8640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb240047d0 0x7fdb38159020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb3cbb8640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb3815d8f0 0x7fdb38158490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb3cbb8640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb3815c9e0 0x7fdb38154fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb3cbb8640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb38159560 con 0x7fdb3815c9e0 2023-10-30T13:25:41.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb3cbb8640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb381589d0 con 0x7fdb240047d0 2023-10-30T13:25:41.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb3cbb8640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb381576f0 con 0x7fdb3815d8f0 2023-10-30T13:25:41.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb36d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb3815c9e0 0x7fdb38154fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb36d9d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb3815c9e0 0x7fdb38154fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56606/0 (socket says 172.21.15.125:56606) 2023-10-30T13:25:41.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb36d9d640 1 -- 172.21.15.125:0/2988170729 learned_addr learned my addr 172.21.15.125:0/2988170729 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:41.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb3659c640 1 --2- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb240047d0 0x7fdb38159020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb2ffff640 1 --2- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb3815d8f0 0x7fdb38158490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb36d9d640 1 -- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb3815d8f0 msgr2=0x7fdb38158490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb36d9d640 1 --2- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb3815d8f0 0x7fdb38158490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb36d9d640 1 -- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb240047d0 msgr2=0x7fdb38159020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb36d9d640 1 --2- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb240047d0 0x7fdb38159020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb36d9d640 1 -- 172.21.15.125:0/2988170729 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb3815bc40 con 0x7fdb3815c9e0 2023-10-30T13:25:41.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb36d9d640 1 --2- 172.21.15.125:0/2988170729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb3815c9e0 0x7fdb38154fc0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fdb30001f40 tx=0x7fdb30011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:41.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb2effd640 1 -- 172.21.15.125:0/2988170729 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3000f8a0 con 0x7fdb3815c9e0 2023-10-30T13:25:41.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb2effd640 1 -- 172.21.15.125:0/2988170729 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb3000fa00 con 0x7fdb3815c9e0 2023-10-30T13:25:41.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb2effd640 1 -- 172.21.15.125:0/2988170729 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb3000fd70 con 0x7fdb3815c9e0 2023-10-30T13:25:41.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/2988170729 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb38157900 con 0x7fdb3815c9e0 2023-10-30T13:25:41.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/2988170729 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb3815c1e0 con 0x7fdb3815c9e0 2023-10-30T13:25:41.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.118+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/2988170729 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdb380c3a10 con 0x7fdb3815c9e0 2023-10-30T13:25:41.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.122+0000 7fdb2effd640 1 -- 172.21.15.125:0/2988170729 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdb3001f460 con 0x7fdb3815c9e0 2023-10-30T13:25:41.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.122+0000 7fdb2effd640 1 --2- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb180664d0 0x7fdb18068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.122+0000 7fdb2effd640 1 -- 172.21.15.125:0/2988170729 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fdb30088280 con 0x7fdb3815c9e0 2023-10-30T13:25:41.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.122+0000 7fdb2effd640 1 --2- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:6800/1849732357,v1:172.21.15.136:6801/1849732357] conn(0x7fdb18070170 0x7fdb18072590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.122+0000 7fdb2effd640 1 -- 172.21.15.125:0/2988170729 --> [v2:172.21.15.136:6800/1849732357,v1:172.21.15.136:6801/1849732357] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdb18072c60 con 0x7fdb18070170 2023-10-30T13:25:41.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.122+0000 7fdb2effd640 1 -- 172.21.15.125:0/2988170729 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdb300885f0 con 0x7fdb3815c9e0 2023-10-30T13:25:41.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.122+0000 7fdb3659c640 1 --2- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb180664d0 0x7fdb18068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.122+0000 7fdb2ffff640 1 --2- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:6800/1849732357,v1:172.21.15.136:6801/1849732357] conn(0x7fdb18070170 0x7fdb18072590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.138 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.122+0000 7fdb2ffff640 1 --2- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:6800/1849732357,v1:172.21.15.136:6801/1849732357] conn(0x7fdb18070170 0x7fdb18072590 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:41.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.122+0000 7fdb3659c640 1 --2- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb180664d0 0x7fdb18068990 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fdb38069010 tx=0x7fdb28008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:41.139 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.126+0000 7fdb2effd640 1 -- 172.21.15.125:0/2988170729 <== osd.5 v2:172.21.15.136:6800/1849732357 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fdb18072c60 con 0x7fdb18070170 2023-10-30T13:25:41.141 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.138+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/2988170729 --> [v2:172.21.15.136:6800/1849732357,v1:172.21.15.136:6801/1849732357] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fdb381572b0 con 0x7fdb18070170 2023-10-30T13:25:41.142 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.138+0000 7fdb2effd640 1 -- 172.21.15.125:0/2988170729 <== osd.5 v2:172.21.15.136:6800/1849732357 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fdb20005150 con 0x7fdb18070170 2023-10-30T13:25:41.142 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2023-10-30T13:25:41.143 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:25:41.143 INFO:teuthology.orchestra.run.smithi125.stdout:} 2023-10-30T13:25:41.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.138+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:6800/1849732357,v1:172.21.15.136:6801/1849732357] conn(0x7fdb18070170 msgr2=0x7fdb18072590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.138+0000 7fdb3cbb8640 1 --2- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:6800/1849732357,v1:172.21.15.136:6801/1849732357] conn(0x7fdb18070170 0x7fdb18072590 crc :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.138+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb180664d0 msgr2=0x7fdb18068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.138+0000 7fdb3cbb8640 1 --2- 172.21.15.125:0/2988170729 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb180664d0 0x7fdb18068990 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fdb38069010 tx=0x7fdb28008040 comp rx=0 tx=0).stop 2023-10-30T13:25:41.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.138+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/2988170729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb3815c9e0 msgr2=0x7fdb38154fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.138+0000 7fdb3cbb8640 1 --2- 172.21.15.125:0/2988170729 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb3815c9e0 0x7fdb38154fc0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7fdb30001f40 tx=0x7fdb30011c20 comp rx=0 tx=0).stop 2023-10-30T13:25:41.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.138+0000 7fdb36d9d640 1 -- 172.21.15.125:0/2988170729 reap_dead start 2023-10-30T13:25:41.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.138+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/2988170729 shutdown_connections 2023-10-30T13:25:41.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.138+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/2988170729 >> 172.21.15.125:0/2988170729 conn(0x7fdb380ccf80 msgr2=0x7fdb38154370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:41.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.138+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/2988170729 shutdown_connections 2023-10-30T13:25:41.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.138+0000 7fdb3cbb8640 1 -- 172.21.15.125:0/2988170729 wait complete. 2023-10-30T13:25:41.163 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-30T13:25:41.313 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff63b69e640 1 Processor -- start 2023-10-30T13:25:41.314 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff63b69e640 1 -- start start 2023-10-30T13:25:41.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff63b69e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff634153c60 0x7ff6340ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.315 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff63b69e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634159830 0x7ff634159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff63b69e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6340cb780 con 0x7ff6340c4430 2023-10-30T13:25:41.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff63b69e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff634154040 con 0x7ff634153c60 2023-10-30T13:25:41.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff63b69e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff63415a2e0 con 0x7ff634159830 2023-10-30T13:25:41.316 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff63943a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff634153c60 0x7ff6340ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff63943a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7ff634153c60 0x7ff6340ca360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56618/0 (socket says 172.21.15.125:56618) 2023-10-30T13:25:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff63943a640 1 -- 172.21.15.125:0/1073631992 learned_addr learned my addr 172.21.15.125:0/1073631992 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:41.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff639c3b640 1 --2- 172.21.15.125:0/1073631992 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634159830 0x7ff634159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff63943a640 1 -- 172.21.15.125:0/1073631992 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634159830 msgr2=0x7ff634159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff63943a640 1 --2- 172.21.15.125:0/1073631992 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634159830 0x7ff634159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff63943a640 1 -- 172.21.15.125:0/1073631992 >> v1:172.21.15.125:6789/0 conn(0x7ff6340c4430 legacy=0x7ff6340cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.310+0000 7ff63943a640 1 -- 172.21.15.125:0/1073631992 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff63415a5c0 con 0x7ff634153c60 2023-10-30T13:25:41.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.314+0000 7ff63943a640 1 --2- 172.21.15.125:0/1073631992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff634153c60 0x7ff6340ca360 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7ff630009870 tx=0x7ff63001c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f807a8149d3ce3fa server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:41.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.314+0000 7ff62bfff640 1 -- 172.21.15.125:0/1073631992 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6300a7070 con 0x7ff634153c60 2023-10-30T13:25:41.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.314+0000 7ff62bfff640 1 -- 172.21.15.125:0/1073631992 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6300ab430 con 0x7ff634153c60 2023-10-30T13:25:41.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.314+0000 7ff62bfff640 1 -- 172.21.15.125:0/1073631992 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6300ab7c0 con 0x7ff634153c60 2023-10-30T13:25:41.321 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.314+0000 7ff63b69e640 1 -- 172.21.15.125:0/1073631992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff634153c60 msgr2=0x7ff6340ca360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.314+0000 7ff63b69e640 1 --2- 172.21.15.125:0/1073631992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff634153c60 0x7ff6340ca360 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7ff630009870 tx=0x7ff63001c540 comp rx=0 tx=0).stop 2023-10-30T13:25:41.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.314+0000 7ff63b69e640 1 -- 172.21.15.125:0/1073631992 shutdown_connections 2023-10-30T13:25:41.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.314+0000 7ff63b69e640 1 --2- 172.21.15.125:0/1073631992 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634159830 0x7ff634159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.322 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.314+0000 7ff63b69e640 1 --2- 172.21.15.125:0/1073631992 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff634153c60 0x7ff6340ca360 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.314+0000 7ff63b69e640 1 -- 172.21.15.125:0/1073631992 >> 172.21.15.125:0/1073631992 conn(0x7ff63406bfe0 msgr2=0x7ff63406c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:41.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63b69e640 1 -- 172.21.15.125:0/1073631992 shutdown_connections 2023-10-30T13:25:41.323 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63b69e640 1 -- 172.21.15.125:0/1073631992 wait complete. 2023-10-30T13:25:41.325 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63b69e640 1 Processor -- start 2023-10-30T13:25:41.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63b69e640 1 -- start start 2023-10-30T13:25:41.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63b69e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6340c4430 0x7ff634142760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63b69e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff634153c60 0x7ff6340cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63b69e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634159830 0x7ff6340b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63b69e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6341f6030 con 0x7ff6340c4430 2023-10-30T13:25:41.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63b69e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff634142d80 con 0x7ff634153c60 2023-10-30T13:25:41.327 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63b69e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6341f53b0 con 0x7ff634159830 2023-10-30T13:25:41.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63943a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6340c4430 0x7ff634142760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63943a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6340c4430 0x7ff634142760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56632/0 (socket says 172.21.15.125:56632) 2023-10-30T13:25:41.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63943a640 1 -- 172.21.15.125:0/3699780805 learned_addr learned my addr 172.21.15.125:0/3699780805 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:41.328 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff639c3b640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634159830 0x7ff6340b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff638c39640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff634153c60 0x7ff6340cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63943a640 1 -- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634159830 msgr2=0x7ff6340b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63943a640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634159830 0x7ff6340b38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63943a640 1 -- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff634153c60 msgr2=0x7ff6340cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63943a640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff634153c60 0x7ff6340cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63943a640 1 -- 172.21.15.125:0/3699780805 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6300094d0 con 0x7ff6340c4430 2023-10-30T13:25:41.330 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff638c39640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff634153c60 0x7ff6340cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:25:41.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63943a640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6340c4430 0x7ff634142760 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7ff6300ad040 tx=0x7ff6300192e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:41.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff629ffb640 1 -- 172.21.15.125:0/3699780805 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6300bc070 con 0x7ff6340c4430 2023-10-30T13:25:41.331 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff629ffb640 1 -- 172.21.15.125:0/3699780805 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6300abe10 con 0x7ff6340c4430 2023-10-30T13:25:41.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff629ffb640 1 -- 172.21.15.125:0/3699780805 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6300b4890 con 0x7ff6340c4430 2023-10-30T13:25:41.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.318+0000 7ff63b69e640 1 -- 172.21.15.125:0/3699780805 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6341f5690 con 0x7ff6340c4430 2023-10-30T13:25:41.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.322+0000 7ff63b69e640 1 -- 172.21.15.125:0/3699780805 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6341f5c50 con 0x7ff6340c4430 2023-10-30T13:25:41.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.322+0000 7ff63b69e640 1 -- 172.21.15.125:0/3699780805 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff63404e170 con 0x7ff6340c4430 2023-10-30T13:25:41.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.322+0000 7ff629ffb640 1 -- 172.21.15.125:0/3699780805 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7ff6300b49f0 con 0x7ff6340c4430 2023-10-30T13:25:41.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.322+0000 7ff629ffb640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6180664a0 0x7ff618068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.322+0000 7ff629ffb640 1 -- 172.21.15.125:0/3699780805 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7ff63011e3b0 con 0x7ff6340c4430 2023-10-30T13:25:41.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.322+0000 7ff638c39640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6180664a0 0x7ff618068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.326+0000 7ff638c39640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6180664a0 0x7ff618068960 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7ff634163d50 tx=0x7ff624009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:41.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.326+0000 7ff629ffb640 1 -- 172.21.15.125:0/3699780805 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7ff6300ead50 con 0x7ff6340c4430 2023-10-30T13:25:41.486 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.482+0000 7ff63b69e640 1 -- 172.21.15.125:0/3699780805 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6340cbf90 con 0x7ff6180664a0 2023-10-30T13:25:41.487 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.482+0000 7ff629ffb640 1 -- 172.21.15.125:0/3699780805 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff624001cc0 con 0x7ff6180664a0 2023-10-30T13:25:41.487 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 5 to deep-scrub 2023-10-30T13:25:41.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.486+0000 7ff63b69e640 1 -- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6180664a0 msgr2=0x7ff618068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.489 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.486+0000 7ff63b69e640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6180664a0 0x7ff618068960 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7ff634163d50 tx=0x7ff624009040 comp rx=0 tx=0).stop 2023-10-30T13:25:41.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.486+0000 7ff63b69e640 1 -- 172.21.15.125:0/3699780805 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6340c4430 msgr2=0x7ff634142760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.486+0000 7ff63b69e640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6340c4430 0x7ff634142760 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7ff6300ad040 tx=0x7ff6300192e0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.490 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.486+0000 7ff63b69e640 1 -- 172.21.15.125:0/3699780805 shutdown_connections 2023-10-30T13:25:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.486+0000 7ff63b69e640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7ff6180664a0 0x7ff618068960 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.486+0000 7ff63b69e640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7ff634159830 0x7ff6340b38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.486+0000 7ff63b69e640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7ff634153c60 0x7ff6340cdd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.491 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.486+0000 7ff63b69e640 1 --2- 172.21.15.125:0/3699780805 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7ff6340c4430 0x7ff634142760 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.486+0000 7ff63b69e640 1 -- 172.21.15.125:0/3699780805 >> 172.21.15.125:0/3699780805 conn(0x7ff63406bfe0 msgr2=0x7ff634158f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:41.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.486+0000 7ff63b69e640 1 -- 172.21.15.125:0/3699780805 shutdown_connections 2023-10-30T13:25:41.492 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.486+0000 7ff63b69e640 1 -- 172.21.15.125:0/3699780805 wait complete. 2023-10-30T13:25:41.508 INFO:tasks.ceph:Scrubbing osd.6 2023-10-30T13:25:41.508 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:25:41.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.646+0000 7f162951d640 1 Processor -- start 2023-10-30T13:25:41.650 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.646+0000 7f162951d640 1 -- start start 2023-10-30T13:25:41.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.646+0000 7f162951d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f16241536d0 0x7f16240cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.646+0000 7f162951d640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f162415d9d0 0x7f16240c4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.653 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.646+0000 7f162951d640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16240c4540 con 0x7f1624159760 2023-10-30T13:25:41.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.646+0000 7f162951d640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f162415a210 con 0x7f162415d9d0 2023-10-30T13:25:41.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.646+0000 7f162951d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f162415a4f0 con 0x7f16241536d0 2023-10-30T13:25:41.654 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.646+0000 7f16237fe640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f1624159760 0x7f16240cf090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54424/0 (socket says 172.21.15.125:54424) 2023-10-30T13:25:41.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.646+0000 7f16237fe640 1 -- 172.21.15.125:0/1973673448 learned_addr learned my addr 172.21.15.125:0/1973673448 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:25:41.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.646+0000 7f1622ffd640 1 --2- 172.21.15.125:0/1973673448 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f16241536d0 0x7f16240cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.646+0000 7f16227fc640 1 --2- 172.21.15.125:0/1973673448 >> v2:172.21.15.136:3300/0 conn(0x7f162415d9d0 0x7f16240c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.655 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.646+0000 7f1621ffb640 1 -- 172.21.15.125:0/1973673448 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248009835 0 0) 0x7f16240c4540 con 0x7f1624159760 2023-10-30T13:25:41.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 -- 172.21.15.125:0/1973673448 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f160c003530 con 0x7f1624159760 2023-10-30T13:25:41.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 -- 172.21.15.125:0/1973673448 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2805067622 0 0) 0x7f160c003530 con 0x7f1624159760 2023-10-30T13:25:41.656 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 -- 172.21.15.125:0/1973673448 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f16241536d0 msgr2=0x7f16240cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:41.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 --2- 172.21.15.125:0/1973673448 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f16241536d0 0x7f16240cb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 -- 172.21.15.125:0/1973673448 >> v2:172.21.15.136:3300/0 conn(0x7f162415d9d0 msgr2=0x7f16240c4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:41.657 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 --2- 172.21.15.125:0/1973673448 >> v2:172.21.15.136:3300/0 conn(0x7f162415d9d0 0x7f16240c4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 -- 172.21.15.125:0/1973673448 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f162415cef0 con 0x7f1624159760 2023-10-30T13:25:41.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 -- 172.21.15.125:0/1973673448 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f161c001a00 con 0x7f1624159760 2023-10-30T13:25:41.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 -- 172.21.15.125:0/1973673448 >> v1:172.21.15.125:6789/0 conn(0x7f1624159760 legacy=0x7f16240cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.658 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 --2- 172.21.15.125:0/1973673448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f160c003980 0x7f160c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 --2- 172.21.15.125:0/1973673448 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f160c0047d0 0x7f160c004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 --2- 172.21.15.125:0/1973673448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f160c005190 0x7f160c00b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.659 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 -- 172.21.15.125:0/1973673448 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162415a210 con 0x7f160c003980 2023-10-30T13:25:41.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 -- 172.21.15.125:0/1973673448 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162415a4f0 con 0x7f160c005190 2023-10-30T13:25:41.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1621ffb640 1 -- 172.21.15.125:0/1973673448 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160c00bed0 con 0x7f160c0047d0 2023-10-30T13:25:41.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1622ffd640 1 --2- 172.21.15.125:0/1973673448 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f16241536d0 0x7f16240cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:25:41.660 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f16227fc640 1 --2- 172.21.15.125:0/1973673448 >> v2:172.21.15.136:3300/0 conn(0x7f162415d9d0 0x7f16240c4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:25:41.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1622ffd640 1 --2- 172.21.15.125:0/1973673448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f160c003980 0x7f160c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f16237fe640 1 --2- 172.21.15.125:0/1973673448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f160c005190 0x7f160c00b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f16227fc640 1 --2- 172.21.15.125:0/1973673448 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f160c0047d0 0x7f160c004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.661 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1622ffd640 1 -- 172.21.15.125:0/1973673448 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f160c0047d0 msgr2=0x7f160c004c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:41.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1622ffd640 1 --2- 172.21.15.125:0/1973673448 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f160c0047d0 0x7f160c004c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1622ffd640 1 -- 172.21.15.125:0/1973673448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f160c005190 msgr2=0x7f160c00b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:41.662 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1622ffd640 1 --2- 172.21.15.125:0/1973673448 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f160c005190 0x7f160c00b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.663 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1622ffd640 1 -- 172.21.15.125:0/1973673448 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c003530 con 0x7f160c003980 2023-10-30T13:25:41.666 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.650+0000 7f1622ffd640 1 --2- 172.21.15.125:0/1973673448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f160c003980 0x7f160c003da0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f1614009b20 tx=0x7f161401ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba197da4d6ac8cc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:41.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f1621ffb640 1 -- 172.21.15.125:0/1973673448 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1614004030 con 0x7f160c003980 2023-10-30T13:25:41.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f1621ffb640 1 -- 172.21.15.125:0/1973673448 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1614004190 con 0x7f160c003980 2023-10-30T13:25:41.667 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f1621ffb640 1 -- 172.21.15.125:0/1973673448 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1614004520 con 0x7f160c003980 2023-10-30T13:25:41.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f162951d640 1 -- 172.21.15.125:0/1973673448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f160c003980 msgr2=0x7f160c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f162951d640 1 --2- 172.21.15.125:0/1973673448 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f160c003980 0x7f160c003da0 secure :-1 s=READY pgs=2190 cs=0 l=1 rev1=1 crypto rx=0x7f1614009b20 tx=0x7f161401ca60 comp rx=0 tx=0).stop 2023-10-30T13:25:41.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f16237fe640 1 -- 172.21.15.125:0/1973673448 reap_dead start 2023-10-30T13:25:41.668 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f162951d640 1 -- 172.21.15.125:0/1973673448 shutdown_connections 2023-10-30T13:25:41.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f162951d640 1 -- 172.21.15.125:0/1973673448 >> 172.21.15.125:0/1973673448 conn(0x7f162406bf70 msgr2=0x7f162406c380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:41.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f162951d640 1 -- 172.21.15.125:0/1973673448 shutdown_connections 2023-10-30T13:25:41.669 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f162951d640 1 -- 172.21.15.125:0/1973673448 wait complete. 2023-10-30T13:25:41.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f162951d640 1 Processor -- start 2023-10-30T13:25:41.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f162951d640 1 -- start start 2023-10-30T13:25:41.670 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f162951d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f160c003980 0x7f16240ce530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f162951d640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f16240c5960 0x7f16241f1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f162951d640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16240c8c20 0x7f16240cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f162951d640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1624162a30 con 0x7f160c003980 2023-10-30T13:25:41.671 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f162951d640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16241f2ee0 con 0x7f16240c8c20 2023-10-30T13:25:41.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f162951d640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16241f2640 con 0x7f16240c5960 2023-10-30T13:25:41.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f1622ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f160c003980 0x7f16240ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f1622ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f160c003980 0x7f16240ce530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56648/0 (socket says 172.21.15.125:56648) 2023-10-30T13:25:41.672 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f1622ffd640 1 -- 172.21.15.125:0/2780099923 learned_addr learned my addr 172.21.15.125:0/2780099923 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:41.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.654+0000 7f16237fe640 1 --2- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16240c8c20 0x7f16240cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f16227fc640 1 --2- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f16240c5960 0x7f16241f1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.673 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f1622ffd640 1 -- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f16240c5960 msgr2=0x7f16241f1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f1622ffd640 1 --2- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f16240c5960 0x7f16241f1fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f1622ffd640 1 -- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16240c8c20 msgr2=0x7f16240cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f1622ffd640 1 --2- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f16240c8c20 0x7f16240cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.674 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f1622ffd640 1 -- 172.21.15.125:0/2780099923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16241f2920 con 0x7f160c003980 2023-10-30T13:25:41.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f1622ffd640 1 --2- 172.21.15.125:0/2780099923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f160c003980 0x7f16240ce530 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f161401caa0 tx=0x7f160c005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:41.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f160b7fe640 1 -- 172.21.15.125:0/2780099923 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161400f0c0 con 0x7f160c003980 2023-10-30T13:25:41.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f160b7fe640 1 -- 172.21.15.125:0/2780099923 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1614004300 con 0x7f160c003980 2023-10-30T13:25:41.675 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f160b7fe640 1 -- 172.21.15.125:0/2780099923 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16140a5540 con 0x7f160c003980 2023-10-30T13:25:41.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f162951d640 1 -- 172.21.15.125:0/2780099923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16241f2c00 con 0x7f160c003980 2023-10-30T13:25:41.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f162951d640 1 -- 172.21.15.125:0/2780099923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16240cd260 con 0x7f160c003980 2023-10-30T13:25:41.676 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f162951d640 1 -- 172.21.15.125:0/2780099923 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f16240cb7d0 con 0x7f160c003980 2023-10-30T13:25:41.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f160b7fe640 1 -- 172.21.15.125:0/2780099923 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f16140a56a0 con 0x7f160c003980 2023-10-30T13:25:41.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f160b7fe640 1 --2- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f16100664a0 0x7f1610068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f160b7fe640 1 -- 172.21.15.125:0/2780099923 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f161411e3a0 con 0x7f160c003980 2023-10-30T13:25:41.677 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f160b7fe640 1 --2- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:6816/3758986733,v1:172.21.15.136:6818/3758986733] conn(0x7f1610070140 0x7f1610072560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f160b7fe640 1 -- 172.21.15.125:0/2780099923 --> [v2:172.21.15.136:6816/3758986733,v1:172.21.15.136:6818/3758986733] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1610072c30 con 0x7f1610070140 2023-10-30T13:25:41.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.658+0000 7f160b7fe640 1 -- 172.21.15.125:0/2780099923 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1614120980 con 0x7f160c003980 2023-10-30T13:25:41.678 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.662+0000 7f16227fc640 1 --2- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f16100664a0 0x7f1610068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.662+0000 7f16237fe640 1 --2- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:6816/3758986733,v1:172.21.15.136:6818/3758986733] conn(0x7f1610070140 0x7f1610072560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.662+0000 7f16237fe640 1 --2- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:6816/3758986733,v1:172.21.15.136:6818/3758986733] conn(0x7f1610070140 0x7f1610072560 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:41.679 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.662+0000 7f16227fc640 1 --2- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f16100664a0 0x7f1610068960 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f16240b18f0 tx=0x7f1618008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:41.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.662+0000 7f160b7fe640 1 -- 172.21.15.125:0/2780099923 <== osd.6 v2:172.21.15.136:6816/3758986733 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7f1610072c30 con 0x7f1610070140 2023-10-30T13:25:41.680 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.674+0000 7f162951d640 1 -- 172.21.15.125:0/2780099923 --> [v2:172.21.15.136:6816/3758986733,v1:172.21.15.136:6818/3758986733] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f16240cd3f0 con 0x7f1610070140 2023-10-30T13:25:41.681 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.674+0000 7f160b7fe640 1 -- 172.21.15.125:0/2780099923 <== osd.6 v2:172.21.15.136:6816/3758986733 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f16240cd3f0 con 0x7f1610070140 2023-10-30T13:25:41.681 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2023-10-30T13:25:41.681 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:25:41.682 INFO:teuthology.orchestra.run.smithi125.stdout:} 2023-10-30T13:25:41.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.678+0000 7f162951d640 1 -- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:6816/3758986733,v1:172.21.15.136:6818/3758986733] conn(0x7f1610070140 msgr2=0x7f1610072560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.678+0000 7f162951d640 1 --2- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:6816/3758986733,v1:172.21.15.136:6818/3758986733] conn(0x7f1610070140 0x7f1610072560 crc :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.683 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.678+0000 7f162951d640 1 -- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f16100664a0 msgr2=0x7f1610068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.678+0000 7f162951d640 1 --2- 172.21.15.125:0/2780099923 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f16100664a0 0x7f1610068960 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f16240b18f0 tx=0x7f1618008040 comp rx=0 tx=0).stop 2023-10-30T13:25:41.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.678+0000 7f162951d640 1 -- 172.21.15.125:0/2780099923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f160c003980 msgr2=0x7f16240ce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.678+0000 7f162951d640 1 --2- 172.21.15.125:0/2780099923 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f160c003980 0x7f16240ce530 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f161401caa0 tx=0x7f160c005800 comp rx=0 tx=0).stop 2023-10-30T13:25:41.684 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.678+0000 7f16237fe640 1 -- 172.21.15.125:0/2780099923 reap_dead start 2023-10-30T13:25:41.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.678+0000 7f162951d640 1 -- 172.21.15.125:0/2780099923 shutdown_connections 2023-10-30T13:25:41.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.678+0000 7f162951d640 1 -- 172.21.15.125:0/2780099923 >> 172.21.15.125:0/2780099923 conn(0x7f162406bf70 msgr2=0x7f16241524d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:41.685 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.678+0000 7f162951d640 1 -- 172.21.15.125:0/2780099923 shutdown_connections 2023-10-30T13:25:41.686 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.678+0000 7f162951d640 1 -- 172.21.15.125:0/2780099923 wait complete. 2023-10-30T13:25:41.699 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-30T13:25:41.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.834+0000 7f5556e46640 1 Processor -- start 2023-10-30T13:25:41.838 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.834+0000 7f5556e46640 1 -- start start 2023-10-30T13:25:41.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.834+0000 7f5556e46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f55501518a0 0x7f5550158a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.834+0000 7f5556e46640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f55501581b0 0x7f555006fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.840 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.834+0000 7f5556e46640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5550151c80 con 0x7f55500c1f50 2023-10-30T13:25:41.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.834+0000 7f5556e46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55500c30c0 con 0x7f55501518a0 2023-10-30T13:25:41.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.834+0000 7f5556e46640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55500c4120 con 0x7f55501581b0 2023-10-30T13:25:41.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.834+0000 7f5554be2640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f55501518a0 0x7f5550158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.841 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.834+0000 7f55553e3640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f55500c1f50 0x7f55500ccfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54436/0 (socket says 172.21.15.125:54436) 2023-10-30T13:25:41.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.834+0000 7f55553e3640 1 -- 172.21.15.125:0/1121685172 learned_addr learned my addr 172.21.15.125:0/1121685172 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:25:41.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.834+0000 7f554ffff640 1 --2- 172.21.15.125:0/1121685172 >> v2:172.21.15.136:3300/0 conn(0x7f55501581b0 0x7f555006fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.842 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.834+0000 7f554f7fe640 1 -- 172.21.15.125:0/1121685172 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242841444 0 0) 0x7f5550151c80 con 0x7f55500c1f50 2023-10-30T13:25:41.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f554f7fe640 1 -- 172.21.15.125:0/1121685172 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5540003530 con 0x7f55500c1f50 2023-10-30T13:25:41.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f5554be2640 1 -- 172.21.15.125:0/1121685172 >> v2:172.21.15.136:3300/0 conn(0x7f55501581b0 msgr2=0x7f555006fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:41.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f5554be2640 1 --2- 172.21.15.125:0/1121685172 >> v2:172.21.15.136:3300/0 conn(0x7f55501581b0 0x7f555006fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.843 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f5554be2640 1 -- 172.21.15.125:0/1121685172 >> v1:172.21.15.125:6789/0 conn(0x7f55500c1f50 legacy=0x7f55500ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.844 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f5554be2640 1 -- 172.21.15.125:0/1121685172 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55500c3b60 con 0x7f55501518a0 2023-10-30T13:25:41.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f5554be2640 1 --2- 172.21.15.125:0/1121685172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55501518a0 0x7f5550158a20 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f555006e370 tx=0x7f554401ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=518b6094e4b78450 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:41.845 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f554f7fe640 1 -- 172.21.15.125:0/1121685172 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55440a8950 con 0x7f55501518a0 2023-10-30T13:25:41.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f554f7fe640 1 -- 172.21.15.125:0/1121685172 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55440a8ab0 con 0x7f55501518a0 2023-10-30T13:25:41.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f554f7fe640 1 -- 172.21.15.125:0/1121685172 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55440a8e40 con 0x7f55501518a0 2023-10-30T13:25:41.846 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f5556e46640 1 -- 172.21.15.125:0/1121685172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55501518a0 msgr2=0x7f5550158a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f5556e46640 1 --2- 172.21.15.125:0/1121685172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55501518a0 0x7f5550158a20 secure :-1 s=READY pgs=2193 cs=0 l=1 rev1=1 crypto rx=0x7f555006e370 tx=0x7f554401ec50 comp rx=0 tx=0).stop 2023-10-30T13:25:41.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f5556e46640 1 -- 172.21.15.125:0/1121685172 shutdown_connections 2023-10-30T13:25:41.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f5556e46640 1 --2- 172.21.15.125:0/1121685172 >> v2:172.21.15.136:3300/0 conn(0x7f55501581b0 0x7f555006fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.847 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f5556e46640 1 --2- 172.21.15.125:0/1121685172 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55501518a0 0x7f5550158a20 unknown :-1 s=CLOSED pgs=2193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f5556e46640 1 -- 172.21.15.125:0/1121685172 >> 172.21.15.125:0/1121685172 conn(0x7f5550069e90 msgr2=0x7f555006a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:41.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.838+0000 7f5556e46640 1 -- 172.21.15.125:0/1121685172 shutdown_connections 2023-10-30T13:25:41.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f5556e46640 1 -- 172.21.15.125:0/1121685172 wait complete. 2023-10-30T13:25:41.848 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f5556e46640 1 Processor -- start 2023-10-30T13:25:41.849 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f5556e46640 1 -- start start 2023-10-30T13:25:41.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f5556e46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f55500c1f50 0x7f555014afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.851 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f5556e46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55501518a0 0x7f55501ed3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f5556e46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55501581b0 0x7f5550148dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f5556e46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5550149bd0 con 0x7f55501581b0 2023-10-30T13:25:41.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f5556e46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55501ecf10 con 0x7f55501518a0 2023-10-30T13:25:41.852 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f5556e46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55501ecc10 con 0x7f55500c1f50 2023-10-30T13:25:41.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f55553e3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55501581b0 0x7f5550148dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f55553e3640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55501581b0 0x7f5550148dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56660/0 (socket says 172.21.15.125:56660) 2023-10-30T13:25:41.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f55553e3640 1 -- 172.21.15.125:0/1576966308 learned_addr learned my addr 172.21.15.125:0/1576966308 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:41.853 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f5554be2640 1 --2- 172.21.15.125:0/1576966308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f55500c1f50 0x7f555014afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f554ffff640 1 --2- 172.21.15.125:0/1576966308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55501518a0 0x7f55501ed3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f55553e3640 1 -- 172.21.15.125:0/1576966308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f55500c1f50 msgr2=0x7f555014afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.854 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f55553e3640 1 --2- 172.21.15.125:0/1576966308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f55500c1f50 0x7f555014afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.855 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f55553e3640 1 -- 172.21.15.125:0/1576966308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55501518a0 msgr2=0x7f55501ed3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:41.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f55553e3640 1 --2- 172.21.15.125:0/1576966308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55501518a0 0x7f55501ed3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:41.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f55553e3640 1 -- 172.21.15.125:0/1576966308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5544009420 con 0x7f55501581b0 2023-10-30T13:25:41.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f55553e3640 1 --2- 172.21.15.125:0/1576966308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55501581b0 0x7f5550148dc0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f554800cb30 tx=0x7f554800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:41.857 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f554d7fa640 1 -- 172.21.15.125:0/1576966308 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554800aca0 con 0x7f55501581b0 2023-10-30T13:25:41.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f554d7fa640 1 -- 172.21.15.125:0/1576966308 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554800ae00 con 0x7f55501581b0 2023-10-30T13:25:41.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f5556e46640 1 -- 172.21.15.125:0/1576966308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f555014a650 con 0x7f55501581b0 2023-10-30T13:25:41.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.842+0000 7f554d7fa640 1 -- 172.21.15.125:0/1576966308 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5548012600 con 0x7f55501581b0 2023-10-30T13:25:41.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.846+0000 7f5556e46640 1 -- 172.21.15.125:0/1576966308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55500c6600 con 0x7f55501581b0 2023-10-30T13:25:41.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.846+0000 7f5556e46640 1 -- 172.21.15.125:0/1576966308 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55501ed080 con 0x7f55501581b0 2023-10-30T13:25:41.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.846+0000 7f554d7fa640 1 -- 172.21.15.125:0/1576966308 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f5548012820 con 0x7f55501581b0 2023-10-30T13:25:41.859 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.846+0000 7f554d7fa640 1 --2- 172.21.15.125:0/1576966308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f55340664a0 0x7f5534068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:41.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.846+0000 7f554d7fa640 1 -- 172.21.15.125:0/1576966308 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f55480848c0 con 0x7f55501581b0 2023-10-30T13:25:41.860 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.846+0000 7f5554be2640 1 --2- 172.21.15.125:0/1576966308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f55340664a0 0x7f5534068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:41.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.850+0000 7f5554be2640 1 --2- 172.21.15.125:0/1576966308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f55340664a0 0x7f5534068960 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f55440b2000 tx=0x7f5544008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:41.861 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:41.850+0000 7f554d7fa640 1 -- 172.21.15.125:0/1576966308 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5548051220 con 0x7f55501581b0 2023-10-30T13:25:42.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.002+0000 7f5556e46640 1 -- 172.21.15.125:0/1576966308 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f55501576d0 con 0x7f55340664a0 2023-10-30T13:25:42.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.002+0000 7f554d7fa640 1 -- 172.21.15.125:0/1576966308 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f55440b0440 con 0x7f55340664a0 2023-10-30T13:25:42.008 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 6 to deep-scrub 2023-10-30T13:25:42.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.006+0000 7f5556e46640 1 -- 172.21.15.125:0/1576966308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f55340664a0 msgr2=0x7f5534068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.006+0000 7f5556e46640 1 --2- 172.21.15.125:0/1576966308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f55340664a0 0x7f5534068960 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f55440b2000 tx=0x7f5544008040 comp rx=0 tx=0).stop 2023-10-30T13:25:42.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.006+0000 7f5556e46640 1 -- 172.21.15.125:0/1576966308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55501581b0 msgr2=0x7f5550148dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.006+0000 7f5556e46640 1 --2- 172.21.15.125:0/1576966308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55501581b0 0x7f5550148dc0 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f554800cb30 tx=0x7f554800a060 comp rx=0 tx=0).stop 2023-10-30T13:25:42.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.006+0000 7f5556e46640 1 -- 172.21.15.125:0/1576966308 shutdown_connections 2023-10-30T13:25:42.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.006+0000 7f5556e46640 1 --2- 172.21.15.125:0/1576966308 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f55340664a0 0x7f5534068960 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.006+0000 7f5556e46640 1 --2- 172.21.15.125:0/1576966308 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f55501581b0 0x7f5550148dc0 unknown :-1 s=CLOSED pgs=2194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.006+0000 7f5556e46640 1 --2- 172.21.15.125:0/1576966308 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f55501518a0 0x7f55501ed3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.006+0000 7f5556e46640 1 --2- 172.21.15.125:0/1576966308 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f55500c1f50 0x7f555014afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.006+0000 7f5556e46640 1 -- 172.21.15.125:0/1576966308 >> 172.21.15.125:0/1576966308 conn(0x7f5550069e90 msgr2=0x7f555014cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:42.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.006+0000 7f5556e46640 1 -- 172.21.15.125:0/1576966308 shutdown_connections 2023-10-30T13:25:42.013 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.006+0000 7f5556e46640 1 -- 172.21.15.125:0/1576966308 wait complete. 2023-10-30T13:25:42.029 INFO:tasks.ceph:Scrubbing osd.7 2023-10-30T13:25:42.029 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-30T13:25:42.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.166+0000 7fda9f9c5640 1 Processor -- start 2023-10-30T13:25:42.171 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.166+0000 7fda9f9c5640 1 -- start start 2023-10-30T13:25:42.172 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9f9c5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fda98158070 0x7fda980c08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9f9c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda9806a960 0x7fda9815dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9f9c5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda98158450 con 0x7fda980c17f0 2023-10-30T13:25:42.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9f9c5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda980c13d0 con 0x7fda98158070 2023-10-30T13:25:42.173 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9f9c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9815e1d0 con 0x7fda9806a960 2023-10-30T13:25:42.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9d761640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fda98158070 0x7fda980c08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9d761640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fda98158070 0x7fda980c08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56672/0 (socket says 172.21.15.125:56672) 2023-10-30T13:25:42.174 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9d761640 1 -- 172.21.15.125:0/3094179967 learned_addr learned my addr 172.21.15.125:0/3094179967 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:42.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9df62640 1 --1- 172.21.15.125:0/3094179967 >> v1:172.21.15.125:6789/0 conn(0x7fda980c17f0 0x7fda98069e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54446/0 (socket says 172.21.15.125:54446) 2023-10-30T13:25:42.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9cf60640 1 --2- 172.21.15.125:0/3094179967 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda9806a960 0x7fda9815dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9d761640 1 -- 172.21.15.125:0/3094179967 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda9806a960 msgr2=0x7fda9815dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:42.175 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9d761640 1 --2- 172.21.15.125:0/3094179967 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda9806a960 0x7fda9815dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9d761640 1 -- 172.21.15.125:0/3094179967 >> v1:172.21.15.125:6789/0 conn(0x7fda980c17f0 legacy=0x7fda98069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.176 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9d761640 1 -- 172.21.15.125:0/3094179967 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9815e460 con 0x7fda98158070 2023-10-30T13:25:42.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda8ffff640 1 -- 172.21.15.125:0/3094179967 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202154263 0 0) 0x7fda98158450 con 0x7fda980c17f0 2023-10-30T13:25:42.182 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.170+0000 7fda9d761640 1 --2- 172.21.15.125:0/3094179967 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda98158070 0x7fda980c08f0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fda94009870 tx=0x7fda9401c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6e115f32b4f0dbb server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:42.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda8ffff640 1 -- 172.21.15.125:0/3094179967 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda940ac620 con 0x7fda98158070 2023-10-30T13:25:42.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda8ffff640 1 -- 172.21.15.125:0/3094179967 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda940ac780 con 0x7fda98158070 2023-10-30T13:25:42.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda8ffff640 1 -- 172.21.15.125:0/3094179967 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda940acb10 con 0x7fda98158070 2023-10-30T13:25:42.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 -- 172.21.15.125:0/3094179967 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda98158070 msgr2=0x7fda980c08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 --2- 172.21.15.125:0/3094179967 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda98158070 0x7fda980c08f0 secure :-1 s=READY pgs=2196 cs=0 l=1 rev1=1 crypto rx=0x7fda94009870 tx=0x7fda9401c4b0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 -- 172.21.15.125:0/3094179967 shutdown_connections 2023-10-30T13:25:42.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 --2- 172.21.15.125:0/3094179967 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda9806a960 0x7fda9815dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 --2- 172.21.15.125:0/3094179967 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda98158070 0x7fda980c08f0 unknown :-1 s=CLOSED pgs=2196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 -- 172.21.15.125:0/3094179967 >> 172.21.15.125:0/3094179967 conn(0x7fda980c7910 msgr2=0x7fda980c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:42.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 -- 172.21.15.125:0/3094179967 shutdown_connections 2023-10-30T13:25:42.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 -- 172.21.15.125:0/3094179967 wait complete. 2023-10-30T13:25:42.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 Processor -- start 2023-10-30T13:25:42.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 -- start start 2023-10-30T13:25:42.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda9806a960 0x7fda9815d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda980c17f0 0x7fda98152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda98158070 0x7fda981531b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda981f99a0 con 0x7fda98158070 2023-10-30T13:25:42.187 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda981fa790 con 0x7fda9806a960 2023-10-30T13:25:42.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9f9c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9815d380 con 0x7fda980c17f0 2023-10-30T13:25:42.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.174+0000 7fda9df62640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda98158070 0x7fda981531b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.188 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda9df62640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda98158070 0x7fda981531b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56674/0 (socket says 172.21.15.125:56674) 2023-10-30T13:25:42.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda9df62640 1 -- 172.21.15.125:0/1824522950 learned_addr learned my addr 172.21.15.125:0/1824522950 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:42.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda9d761640 1 --2- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda9806a960 0x7fda9815d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.189 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda9cf60640 1 --2- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda980c17f0 0x7fda98152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda9df62640 1 -- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda980c17f0 msgr2=0x7fda98152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda9df62640 1 --2- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fda980c17f0 0x7fda98152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.190 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda9df62640 1 -- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda9806a960 msgr2=0x7fda9815d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda9df62640 1 --2- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fda9806a960 0x7fda9815d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda9df62640 1 -- 172.21.15.125:0/1824522950 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda940094d0 con 0x7fda98158070 2023-10-30T13:25:42.191 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda9df62640 1 --2- 172.21.15.125:0/1824522950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda98158070 0x7fda981531b0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fda9000c780 tx=0x7fda9000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:42.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda8dffb640 1 -- 172.21.15.125:0/1824522950 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9000ce90 con 0x7fda98158070 2023-10-30T13:25:42.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda8dffb640 1 -- 172.21.15.125:0/1824522950 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9000a640 con 0x7fda98158070 2023-10-30T13:25:42.192 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda9f9c5640 1 -- 172.21.15.125:0/1824522950 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda981f9b80 con 0x7fda98158070 2023-10-30T13:25:42.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda8dffb640 1 -- 172.21.15.125:0/1824522950 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9000a9d0 con 0x7fda98158070 2023-10-30T13:25:42.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda9f9c5640 1 -- 172.21.15.125:0/1824522950 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda981fa140 con 0x7fda98158070 2023-10-30T13:25:42.193 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda9f9c5640 1 -- 172.21.15.125:0/1824522950 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fda98070850 con 0x7fda98158070 2023-10-30T13:25:42.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda8dffb640 1 -- 172.21.15.125:0/1824522950 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fda9000ab30 con 0x7fda98158070 2023-10-30T13:25:42.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda8dffb640 1 --2- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda7c0664a0 0x7fda7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.195 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda8dffb640 1 -- 172.21.15.125:0/1824522950 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fda90083e70 con 0x7fda98158070 2023-10-30T13:25:42.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda8dffb640 1 --2- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:6808/3315858521,v1:172.21.15.136:6809/3315858521] conn(0x7fda7c06ffd0 0x7fda7c0723f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda8dffb640 1 -- 172.21.15.125:0/1824522950 --> [v2:172.21.15.136:6808/3315858521,v1:172.21.15.136:6809/3315858521] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda7c072ac0 con 0x7fda7c06ffd0 2023-10-30T13:25:42.196 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.178+0000 7fda8dffb640 1 -- 172.21.15.125:0/1824522950 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda90089500 con 0x7fda98158070 2023-10-30T13:25:42.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.182+0000 7fda9d761640 1 --2- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda7c0664a0 0x7fda7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.182+0000 7fda9cf60640 1 --2- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:6808/3315858521,v1:172.21.15.136:6809/3315858521] conn(0x7fda7c06ffd0 0x7fda7c0723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.197 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.182+0000 7fda9cf60640 1 --2- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:6808/3315858521,v1:172.21.15.136:6809/3315858521] conn(0x7fda7c06ffd0 0x7fda7c0723f0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:42.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.182+0000 7fda9d761640 1 --2- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda7c0664a0 0x7fda7c068960 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fda940ae000 tx=0x7fda94008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:42.198 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.182+0000 7fda8dffb640 1 -- 172.21.15.125:0/1824522950 <== osd.7 v2:172.21.15.136:6808/3315858521 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25755 (crc 0 0 0) 0x7fda7c072ac0 con 0x7fda7c06ffd0 2023-10-30T13:25:42.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.194+0000 7fda9f9c5640 1 -- 172.21.15.125:0/1824522950 --> [v2:172.21.15.136:6808/3315858521,v1:172.21.15.136:6809/3315858521] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fda980c0ff0 con 0x7fda7c06ffd0 2023-10-30T13:25:42.199 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.194+0000 7fda8dffb640 1 -- 172.21.15.125:0/1824522950 <== osd.7 v2:172.21.15.136:6808/3315858521 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fda88007af0 con 0x7fda7c06ffd0 2023-10-30T13:25:42.200 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2023-10-30T13:25:42.201 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-30T13:25:42.201 INFO:teuthology.orchestra.run.smithi125.stdout:} 2023-10-30T13:25:42.202 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.198+0000 7fda9f9c5640 1 -- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:6808/3315858521,v1:172.21.15.136:6809/3315858521] conn(0x7fda7c06ffd0 msgr2=0x7fda7c0723f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.198+0000 7fda9f9c5640 1 --2- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:6808/3315858521,v1:172.21.15.136:6809/3315858521] conn(0x7fda7c06ffd0 0x7fda7c0723f0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.198+0000 7fda9f9c5640 1 -- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda7c0664a0 msgr2=0x7fda7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.198+0000 7fda9f9c5640 1 --2- 172.21.15.125:0/1824522950 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fda7c0664a0 0x7fda7c068960 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fda940ae000 tx=0x7fda94008040 comp rx=0 tx=0).stop 2023-10-30T13:25:42.203 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.198+0000 7fda9f9c5640 1 -- 172.21.15.125:0/1824522950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda98158070 msgr2=0x7fda981531b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.198+0000 7fda9f9c5640 1 --2- 172.21.15.125:0/1824522950 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fda98158070 0x7fda981531b0 secure :-1 s=READY pgs=2197 cs=0 l=1 rev1=1 crypto rx=0x7fda9000c780 tx=0x7fda9000cc40 comp rx=0 tx=0).stop 2023-10-30T13:25:42.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.198+0000 7fda9df62640 1 -- 172.21.15.125:0/1824522950 reap_dead start 2023-10-30T13:25:42.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.198+0000 7fda9f9c5640 1 -- 172.21.15.125:0/1824522950 shutdown_connections 2023-10-30T13:25:42.204 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.198+0000 7fda9f9c5640 1 -- 172.21.15.125:0/1824522950 >> 172.21.15.125:0/1824522950 conn(0x7fda980c7910 msgr2=0x7fda98153960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:42.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.198+0000 7fda9f9c5640 1 -- 172.21.15.125:0/1824522950 shutdown_connections 2023-10-30T13:25:42.205 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.198+0000 7fda9f9c5640 1 -- 172.21.15.125:0/1824522950 wait complete. 2023-10-30T13:25:42.218 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-30T13:25:42.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d386b640 1 Processor -- start 2023-10-30T13:25:42.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d386b640 1 -- start start 2023-10-30T13:25:42.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d386b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62cc153c60 0x7f62cc0ca360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d386b640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f62cc159830 0x7f62cc159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.363 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d386b640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc0cb780 con 0x7f62cc0c4430 2023-10-30T13:25:42.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d386b640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62cc154040 con 0x7f62cc159830 2023-10-30T13:25:42.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d386b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc15a2e0 con 0x7f62cc153c60 2023-10-30T13:25:42.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d1e08640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f62cc159830 0x7f62cc159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d1e08640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f62cc159830 0x7f62cc159ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56676/0 (socket says 172.21.15.125:56676) 2023-10-30T13:25:42.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d1e08640 1 -- 172.21.15.125:0/3752839130 learned_addr learned my addr 172.21.15.125:0/3752839130 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:42.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d0e06640 1 --1- 172.21.15.125:0/3752839130 >> v1:172.21.15.125:6789/0 conn(0x7f62cc0c4430 0x7f62cc0cc4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54456/0 (socket says 172.21.15.125:54456) 2023-10-30T13:25:42.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d1607640 1 --2- 172.21.15.125:0/3752839130 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62cc153c60 0x7f62cc0ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d1e08640 1 -- 172.21.15.125:0/3752839130 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62cc153c60 msgr2=0x7f62cc0ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:25:42.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d1e08640 1 --2- 172.21.15.125:0/3752839130 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62cc153c60 0x7f62cc0ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d1e08640 1 -- 172.21.15.125:0/3752839130 >> v1:172.21.15.125:6789/0 conn(0x7f62cc0c4430 legacy=0x7f62cc0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d1e08640 1 -- 172.21.15.125:0/3752839130 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc15a5c0 con 0x7f62cc159830 2023-10-30T13:25:42.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.358+0000 7f62d1607640 1 --2- 172.21.15.125:0/3752839130 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62cc153c60 0x7f62cc0ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:25:42.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.362+0000 7f62d1e08640 1 --2- 172.21.15.125:0/3752839130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62cc159830 0x7f62cc159ce0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009e50 tx=0x7f62c401ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b3f35e678f21af server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:42.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.362+0000 7f62c3fff640 1 -- 172.21.15.125:0/3752839130 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c40a57d0 con 0x7f62cc159830 2023-10-30T13:25:42.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.362+0000 7f62c3fff640 1 -- 172.21.15.125:0/3752839130 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c40a5930 con 0x7f62cc159830 2023-10-30T13:25:42.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.362+0000 7f62c3fff640 1 -- 172.21.15.125:0/3752839130 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c40a5cc0 con 0x7f62cc159830 2023-10-30T13:25:42.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.362+0000 7f62d386b640 1 -- 172.21.15.125:0/3752839130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62cc159830 msgr2=0x7f62cc159ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.362+0000 7f62d386b640 1 --2- 172.21.15.125:0/3752839130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62cc159830 0x7f62cc159ce0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f62c4009e50 tx=0x7f62c401ca10 comp rx=0 tx=0).stop 2023-10-30T13:25:42.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.362+0000 7f62d386b640 1 -- 172.21.15.125:0/3752839130 shutdown_connections 2023-10-30T13:25:42.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.362+0000 7f62d386b640 1 --2- 172.21.15.125:0/3752839130 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62cc159830 0x7f62cc159ce0 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.362+0000 7f62d386b640 1 --2- 172.21.15.125:0/3752839130 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62cc153c60 0x7f62cc0ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.362+0000 7f62d386b640 1 -- 172.21.15.125:0/3752839130 >> 172.21.15.125:0/3752839130 conn(0x7f62cc06bfe0 msgr2=0x7f62cc06c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:42.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.362+0000 7f62d386b640 1 -- 172.21.15.125:0/3752839130 shutdown_connections 2023-10-30T13:25:42.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.362+0000 7f62d386b640 1 -- 172.21.15.125:0/3752839130 wait complete. 2023-10-30T13:25:42.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d386b640 1 Processor -- start 2023-10-30T13:25:42.372 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d386b640 1 -- start start 2023-10-30T13:25:42.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d386b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62cc0c4430 0x7f62cc0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d386b640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62cc153c60 0x7f62cc0cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d386b640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62cc159830 0x7f62cc145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d386b640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc1f5640 con 0x7f62cc0c4430 2023-10-30T13:25:42.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d386b640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc1f4d70 con 0x7f62cc153c60 2023-10-30T13:25:42.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d386b640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc0cf100 con 0x7f62cc159830 2023-10-30T13:25:42.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d1607640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62cc0c4430 0x7f62cc0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d1607640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62cc0c4430 0x7f62cc0b4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56692/0 (socket says 172.21.15.125:56692) 2023-10-30T13:25:42.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d1607640 1 -- 172.21.15.125:0/4133767758 learned_addr learned my addr 172.21.15.125:0/4133767758 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:42.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d0e06640 1 --2- 172.21.15.125:0/4133767758 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62cc153c60 0x7f62cc0cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d1e08640 1 --2- 172.21.15.125:0/4133767758 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62cc159830 0x7f62cc145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d1607640 1 -- 172.21.15.125:0/4133767758 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62cc159830 msgr2=0x7f62cc145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d1607640 1 --2- 172.21.15.125:0/4133767758 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62cc159830 0x7f62cc145c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d1607640 1 -- 172.21.15.125:0/4133767758 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62cc153c60 msgr2=0x7f62cc0cdf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d1607640 1 --2- 172.21.15.125:0/4133767758 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62cc153c60 0x7f62cc0cdf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d1607640 1 -- 172.21.15.125:0/4133767758 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c4009b30 con 0x7f62cc0c4430 2023-10-30T13:25:42.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d1607640 1 --2- 172.21.15.125:0/4133767758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62cc0c4430 0x7f62cc0b4ed0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f62c800e810 tx=0x7f62c800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:42.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62c1ffb640 1 -- 172.21.15.125:0/4133767758 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c800c870 con 0x7f62cc0c4430 2023-10-30T13:25:42.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62c1ffb640 1 -- 172.21.15.125:0/4133767758 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c800c9d0 con 0x7f62cc0c4430 2023-10-30T13:25:42.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d386b640 1 -- 172.21.15.125:0/4133767758 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62cc14ef00 con 0x7f62cc0c4430 2023-10-30T13:25:42.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62c1ffb640 1 -- 172.21.15.125:0/4133767758 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c800cd60 con 0x7f62cc0c4430 2023-10-30T13:25:42.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.366+0000 7f62d386b640 1 -- 172.21.15.125:0/4133767758 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62cc0cd590 con 0x7f62cc0c4430 2023-10-30T13:25:42.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.370+0000 7f62d386b640 1 -- 172.21.15.125:0/4133767758 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62cc1f58b0 con 0x7f62cc0c4430 2023-10-30T13:25:42.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.370+0000 7f62c1ffb640 1 -- 172.21.15.125:0/4133767758 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f62c8013410 con 0x7f62cc0c4430 2023-10-30T13:25:42.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.370+0000 7f62c1ffb640 1 --2- 172.21.15.125:0/4133767758 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62b00664a0 0x7f62b0068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.370+0000 7f62c1ffb640 1 -- 172.21.15.125:0/4133767758 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f62c8084730 con 0x7f62cc0c4430 2023-10-30T13:25:42.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.370+0000 7f62d0e06640 1 --2- 172.21.15.125:0/4133767758 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62b00664a0 0x7f62b0068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.374+0000 7f62d0e06640 1 --2- 172.21.15.125:0/4133767758 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62b00664a0 0x7f62b0068960 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f62cc14dc90 tx=0x7f62bc007cb0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:42.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.374+0000 7f62c1ffb640 1 -- 172.21.15.125:0/4133767758 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f62c8051070 con 0x7f62cc0c4430 2023-10-30T13:25:42.532 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.530+0000 7f62d386b640 1 -- 172.21.15.125:0/4133767758 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f62cc0cbf90 con 0x7f62b00664a0 2023-10-30T13:25:42.533 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.530+0000 7f62c1ffb640 1 -- 172.21.15.125:0/4133767758 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f62bc005a80 con 0x7f62b00664a0 2023-10-30T13:25:42.533 INFO:teuthology.orchestra.run.smithi125.stderr:instructed osd(s) 7 to deep-scrub 2023-10-30T13:25:42.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.530+0000 7f62d386b640 1 -- 172.21.15.125:0/4133767758 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62b00664a0 msgr2=0x7f62b0068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.530+0000 7f62d386b640 1 --2- 172.21.15.125:0/4133767758 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62b00664a0 0x7f62b0068960 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f62cc14dc90 tx=0x7f62bc007cb0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.530+0000 7f62d386b640 1 -- 172.21.15.125:0/4133767758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62cc0c4430 msgr2=0x7f62cc0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.536 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.530+0000 7f62d386b640 1 --2- 172.21.15.125:0/4133767758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62cc0c4430 0x7f62cc0b4ed0 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7f62c800e810 tx=0x7f62c800ecd0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.530+0000 7f62d386b640 1 -- 172.21.15.125:0/4133767758 shutdown_connections 2023-10-30T13:25:42.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.530+0000 7f62d386b640 1 --2- 172.21.15.125:0/4133767758 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f62b00664a0 0x7f62b0068960 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.530+0000 7f62d386b640 1 --2- 172.21.15.125:0/4133767758 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f62cc159830 0x7f62cc145c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.537 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.530+0000 7f62d386b640 1 --2- 172.21.15.125:0/4133767758 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f62cc153c60 0x7f62cc0cdf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.530+0000 7f62d386b640 1 --2- 172.21.15.125:0/4133767758 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f62cc0c4430 0x7f62cc0b4ed0 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.530+0000 7f62d386b640 1 -- 172.21.15.125:0/4133767758 >> 172.21.15.125:0/4133767758 conn(0x7f62cc06bfe0 msgr2=0x7f62cc158a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:42.538 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.530+0000 7f62d386b640 1 -- 172.21.15.125:0/4133767758 shutdown_connections 2023-10-30T13:25:42.539 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.530+0000 7f62d386b640 1 -- 172.21.15.125:0/4133767758 wait complete. 2023-10-30T13:25:42.555 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:25:42.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.690+0000 7fc4079c5640 1 Processor -- start 2023-10-30T13:25:42.693 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.690+0000 7fc4079c5640 1 -- start start 2023-10-30T13:25:42.694 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.690+0000 7fc4079c5640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4000cf160 0x7fc4000cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.690+0000 7fc4079c5640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4000cb800 con 0x7fc40015d6f0 2023-10-30T13:25:42.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.690+0000 7fc4079c5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4000cb330 con 0x7fc4001538a0 2023-10-30T13:25:42.695 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.690+0000 7fc4079c5640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4000cad70 con 0x7fc4000cf160 2023-10-30T13:25:42.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.690+0000 7fc405f62640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fc4000cf160 0x7fc4000cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.690+0000 7fc404f60640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fc40015d6f0 0x7fc40015a090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:54472/0 (socket says 172.21.15.125:54472) 2023-10-30T13:25:42.696 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.690+0000 7fc404f60640 1 -- 172.21.15.125:0/74878909 learned_addr learned my addr 172.21.15.125:0/74878909 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:25:42.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.690+0000 7fc405f62640 1 -- 172.21.15.125:0/74878909 >> v1:172.21.15.136:6789/0 conn(0x7fc4001538a0 legacy=0x7fc4000ca0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.690+0000 7fc405f62640 1 -- 172.21.15.125:0/74878909 >> v1:172.21.15.125:6789/0 conn(0x7fc40015d6f0 legacy=0x7fc40015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.697 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.690+0000 7fc405f62640 1 -- 172.21.15.125:0/74878909 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4000cb050 con 0x7fc4000cf160 2023-10-30T13:25:42.698 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.690+0000 7fc3f7fff640 1 -- 172.21.15.125:0/74878909 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907376098 0 0) 0x7fc4000cb800 con 0x7fc40015d6f0 2023-10-30T13:25:42.699 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.694+0000 7fc405f62640 1 --2- 172.21.15.125:0/74878909 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4000cf160 0x7fc4000cf590 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8009d90 tx=0x7fc3f801f130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b584cfbd6ecfb9c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:42.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.694+0000 7fc3f7fff640 1 -- 172.21.15.125:0/74878909 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f80af600 con 0x7fc4000cf160 2023-10-30T13:25:42.700 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.694+0000 7fc3f7fff640 1 -- 172.21.15.125:0/74878909 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f80ab050 con 0x7fc4000cf160 2023-10-30T13:25:42.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.694+0000 7fc3f7fff640 1 -- 172.21.15.125:0/74878909 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f80af980 con 0x7fc4000cf160 2023-10-30T13:25:42.701 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.694+0000 7fc4079c5640 1 -- 172.21.15.125:0/74878909 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4000cf160 msgr2=0x7fc4000cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.694+0000 7fc4079c5640 1 --2- 172.21.15.125:0/74878909 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4000cf160 0x7fc4000cf590 secure :-1 s=READY pgs=2202 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8009d90 tx=0x7fc3f801f130 comp rx=0 tx=0).stop 2023-10-30T13:25:42.702 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.694+0000 7fc4079c5640 1 -- 172.21.15.125:0/74878909 shutdown_connections 2023-10-30T13:25:42.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.694+0000 7fc4079c5640 1 --2- 172.21.15.125:0/74878909 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4000cf160 0x7fc4000cf590 unknown :-1 s=CLOSED pgs=2202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.694+0000 7fc4079c5640 1 -- 172.21.15.125:0/74878909 >> 172.21.15.125:0/74878909 conn(0x7fc40006bfe0 msgr2=0x7fc40006c3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:42.703 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.694+0000 7fc4079c5640 1 -- 172.21.15.125:0/74878909 shutdown_connections 2023-10-30T13:25:42.704 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.694+0000 7fc4079c5640 1 -- 172.21.15.125:0/74878909 wait complete. 2023-10-30T13:25:42.707 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc4079c5640 1 Processor -- start 2023-10-30T13:25:42.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc4079c5640 1 -- start start 2023-10-30T13:25:42.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc4079c5640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4000cf160 0x7fc40014fb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc4079c5640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4001538a0 0x7fc4001500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.708 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc4079c5640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc40015d6f0 0x7fc4000b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc4079c5640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40014f630 con 0x7fc4000cf160 2023-10-30T13:25:42.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc4079c5640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4001519c0 con 0x7fc4001538a0 2023-10-30T13:25:42.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc4079c5640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc40014dc80 con 0x7fc40015d6f0 2023-10-30T13:25:42.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc405761640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4000cf160 0x7fc40014fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.709 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc405761640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4000cf160 0x7fc40014fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56708/0 (socket says 172.21.15.125:56708) 2023-10-30T13:25:42.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc405761640 1 -- 172.21.15.125:0/3493796218 learned_addr learned my addr 172.21.15.125:0/3493796218 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:25:42.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc404f60640 1 --2- 172.21.15.125:0/3493796218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4001538a0 0x7fc4001500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.710 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc405f62640 1 --2- 172.21.15.125:0/3493796218 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc40015d6f0 0x7fc4000b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc405761640 1 -- 172.21.15.125:0/3493796218 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc40015d6f0 msgr2=0x7fc4000b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc405761640 1 --2- 172.21.15.125:0/3493796218 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc40015d6f0 0x7fc4000b45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc405761640 1 -- 172.21.15.125:0/3493796218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4001538a0 msgr2=0x7fc4001500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.711 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc405761640 1 --2- 172.21.15.125:0/3493796218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4001538a0 0x7fc4001500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc405761640 1 -- 172.21.15.125:0/3493796218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f8009a70 con 0x7fc4000cf160 2023-10-30T13:25:42.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc405761640 1 --2- 172.21.15.125:0/3493796218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4000cf160 0x7fc40014fb70 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc00c500 tx=0x7fc3fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:42.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc3f5ffb640 1 -- 172.21.15.125:0/3493796218 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc005220 con 0x7fc4000cf160 2023-10-30T13:25:42.712 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc3f5ffb640 1 -- 172.21.15.125:0/3493796218 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3fc005380 con 0x7fc4000cf160 2023-10-30T13:25:42.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc4079c5640 1 -- 172.21.15.125:0/3493796218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc400151700 con 0x7fc4000cf160 2023-10-30T13:25:42.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc3f5ffb640 1 -- 172.21.15.125:0/3493796218 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3fc007270 con 0x7fc4000cf160 2023-10-30T13:25:42.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc4079c5640 1 -- 172.21.15.125:0/3493796218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc40014e200 con 0x7fc4000cf160 2023-10-30T13:25:42.713 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.698+0000 7fc4079c5640 1 -- 172.21.15.125:0/3493796218 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc40004e170 con 0x7fc4000cf160 2023-10-30T13:25:42.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.702+0000 7fc3f5ffb640 1 -- 172.21.15.125:0/3493796218 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fc3fc0073d0 con 0x7fc4000cf160 2023-10-30T13:25:42.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.702+0000 7fc3f5ffb640 1 --2- 172.21.15.125:0/3493796218 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc3e40664d0 0x7fc3e4068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:25:42.714 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.702+0000 7fc3f5ffb640 1 -- 172.21.15.125:0/3493796218 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fc3fc0842e0 con 0x7fc4000cf160 2023-10-30T13:25:42.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.702+0000 7fc404f60640 1 --2- 172.21.15.125:0/3493796218 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc3e40664d0 0x7fc3e4068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:25:42.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.702+0000 7fc404f60640 1 --2- 172.21.15.125:0/3493796218 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc3e40664d0 0x7fc3e4068990 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fc40015ac90 tx=0x7fc3f0007d10 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:25:42.715 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.706+0000 7fc3f5ffb640 1 -- 172.21.15.125:0/3493796218 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fc3fc050c30 con 0x7fc4000cf160 2023-10-30T13:25:42.858 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.854+0000 7fc4079c5640 1 -- 172.21.15.125:0/3493796218 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc400159b70 con 0x7fc3e40664d0 2023-10-30T13:25:42.880 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.878+0000 7fc3f5ffb640 1 -- 172.21.15.125:0/3493796218 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293026 (secure 0 0 0) 0x7fc3f0005b10 con 0x7fc3e40664d0 2023-10-30T13:25:42.883 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:25:42.885 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T13:25:42.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.882+0000 7fc4079c5640 1 -- 172.21.15.125:0/3493796218 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc3e40664d0 msgr2=0x7fc3e4068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.886 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.882+0000 7fc4079c5640 1 --2- 172.21.15.125:0/3493796218 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc3e40664d0 0x7fc3e4068990 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fc40015ac90 tx=0x7fc3f0007d10 comp rx=0 tx=0).stop 2023-10-30T13:25:42.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.882+0000 7fc4079c5640 1 -- 172.21.15.125:0/3493796218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4000cf160 msgr2=0x7fc40014fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:25:42.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.882+0000 7fc4079c5640 1 --2- 172.21.15.125:0/3493796218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4000cf160 0x7fc40014fb70 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc00c500 tx=0x7fc3fc00c9c0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.887 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.882+0000 7fc4079c5640 1 -- 172.21.15.125:0/3493796218 shutdown_connections 2023-10-30T13:25:42.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.882+0000 7fc4079c5640 1 --2- 172.21.15.125:0/3493796218 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fc3e40664d0 0x7fc3e4068990 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.889 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.882+0000 7fc4079c5640 1 --2- 172.21.15.125:0/3493796218 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fc40015d6f0 0x7fc4000b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.882+0000 7fc4079c5640 1 --2- 172.21.15.125:0/3493796218 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fc4001538a0 0x7fc4001500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.882+0000 7fc4079c5640 1 --2- 172.21.15.125:0/3493796218 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fc4000cf160 0x7fc40014fb70 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:25:42.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.882+0000 7fc4079c5640 1 -- 172.21.15.125:0/3493796218 >> 172.21.15.125:0/3493796218 conn(0x7fc40006bfe0 msgr2=0x7fc4000cbbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:25:42.890 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.882+0000 7fc4079c5640 1 -- 172.21.15.125:0/3493796218 shutdown_connections 2023-10-30T13:25:42.891 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:25:42.882+0000 7fc4079c5640 1 -- 172.21.15.125:0/3493796218 wait complete. 2023-10-30T13:25:42.904 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":1793,"stamp":"2023-10-30T13:25:42.272793+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2068932093,"num_objects":40960,"num_object_clones":18317,"num_object_copies":81920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40960,"num_whiteouts":14249,"num_read":30933,"num_read_kb":10838515,"num_write":128197,"num_write_kb":3672596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134394,"ondisk_log_size":134394,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5379072,"kb_used_data":4140352,"kb_used_omap":12,"kb_used_meta":1238643,"kb_avail":749595648,"statfs":{"total":773094113280,"available":767585943552,"internally_reserved":0,"allocated":4239720448,"data_stored":4144120906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1268371001},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1078521,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":52065,"num_write":16,"num_write_kb":1063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004574"},"pg_stats":[{"pgid":"3.3f","version":"77'1475","reported_seq":3523,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.196345+0000","last_change":"2023-10-30T12:37:07.182969+0000","last_active":"2023-10-30T13:25:16.196345+0000","last_peered":"2023-10-30T13:25:16.196345+0000","last_clean":"2023-10-30T13:25:16.196345+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T13:25:16.196345+0000","last_undegraded":"2023-10-30T13:25:16.196345+0000","last_fullsized":"2023-10-30T13:25:16.196345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1475,"log_dups_size":0,"ondisk_log_size":1475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:33.616938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086120880000000004,"stat_sum":{"num_bytes":27261067,"num_objects":552,"num_object_clones":249,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":197,"num_read":145,"num_read_kb":29501,"num_write":1298,"num_write_kb":29008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"77'1476","reported_seq":3488,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.441892+0000","last_change":"2023-10-30T12:37:07.190309+0000","last_active":"2023-10-30T13:25:10.441892+0000","last_peered":"2023-10-30T13:25:10.441892+0000","last_clean":"2023-10-30T13:25:10.441892+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T13:25:10.441892+0000","last_undegraded":"2023-10-30T13:25:10.441892+0000","last_fullsized":"2023-10-30T13:25:10.441892+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:26:31.884771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015306986,"stat_sum":{"num_bytes":13736175,"num_objects":574,"num_object_clones":269,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":223,"num_read":131,"num_read_kb":21315,"num_write":1260,"num_write_kb":15409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"77'1563","reported_seq":3574,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288448+0000","last_change":"2023-10-30T12:37:07.187803+0000","last_active":"2023-10-30T13:24:16.288448+0000","last_peered":"2023-10-30T13:24:16.288448+0000","last_clean":"2023-10-30T13:24:16.288448+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T13:24:16.288448+0000","last_undegraded":"2023-10-30T13:24:16.288448+0000","last_fullsized":"2023-10-30T13:24:16.288448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1563,"log_dups_size":0,"ondisk_log_size":1563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:32.502949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013634598,"stat_sum":{"num_bytes":28390436,"num_objects":628,"num_object_clones":291,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":232,"num_read":152,"num_read_kb":47076,"num_write":1356,"num_write_kb":37143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"77'1577","reported_seq":3585,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.277341+0000","last_change":"2023-10-30T12:37:07.177860+0000","last_active":"2023-10-30T13:25:32.277341+0000","last_peered":"2023-10-30T13:25:32.277341+0000","last_clean":"2023-10-30T13:25:32.277341+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T13:25:32.277341+0000","last_undegraded":"2023-10-30T13:25:32.277341+0000","last_fullsized":"2023-10-30T13:25:32.277341+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1577,"log_dups_size":0,"ondisk_log_size":1577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:08:27.076595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056762749999999997,"stat_sum":{"num_bytes":22921274,"num_objects":629,"num_object_clones":290,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":224,"num_read":145,"num_read_kb":39229,"num_write":1388,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"77'1591","reported_seq":3596,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.167292+0000","last_change":"2023-10-30T12:37:07.181803+0000","last_active":"2023-10-30T13:24:56.167292+0000","last_peered":"2023-10-30T13:24:56.167292+0000","last_clean":"2023-10-30T13:24:56.167292+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T13:24:56.167292+0000","last_undegraded":"2023-10-30T13:24:56.167292+0000","last_fullsized":"2023-10-30T13:24:56.167292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1591,"log_dups_size":0,"ondisk_log_size":1591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:23:58.362682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081298299999999993,"stat_sum":{"num_bytes":26798060,"num_objects":623,"num_object_clones":286,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":214,"num_read":144,"num_read_kb":26713,"num_write":1434,"num_write_kb":32534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"85'1466","reported_seq":3496,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.286314+0000","last_change":"2023-10-30T12:37:07.176046+0000","last_active":"2023-10-30T13:24:16.286314+0000","last_peered":"2023-10-30T13:24:16.286314+0000","last_clean":"2023-10-30T13:24:16.286314+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T13:24:16.286314+0000","last_undegraded":"2023-10-30T13:24:16.286314+0000","last_fullsized":"2023-10-30T13:24:16.286314+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1466,"log_dups_size":0,"ondisk_log_size":1466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:30:51.393008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0048510920000000004,"stat_sum":{"num_bytes":13539463,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":205,"num_read":144,"num_read_kb":15818,"num_write":1313,"num_write_kb":14957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"84'1562","reported_seq":3551,"reported_epoch":84,"state":"active+clean","last_fresh":"2023-10-30T13:23:51.391894+0000","last_change":"2023-10-30T12:37:07.173349+0000","last_active":"2023-10-30T13:23:51.391894+0000","last_peered":"2023-10-30T13:23:51.391894+0000","last_clean":"2023-10-30T13:23:51.391894+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T13:23:51.391894+0000","last_undegraded":"2023-10-30T13:23:51.391894+0000","last_fullsized":"2023-10-30T13:23:51.391894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1562,"log_dups_size":0,"ondisk_log_size":1562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:38:16.606065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082671399999999999,"stat_sum":{"num_bytes":7202428,"num_objects":606,"num_object_clones":284,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":219,"num_read":126,"num_read_kb":1677,"num_write":1361,"num_write_kb":10203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"82'1518","reported_seq":3543,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.219577+0000","last_change":"2023-10-30T12:37:07.188855+0000","last_active":"2023-10-30T13:25:32.219577+0000","last_peered":"2023-10-30T13:25:32.219577+0000","last_clean":"2023-10-30T13:25:32.219577+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T13:25:32.219577+0000","last_undegraded":"2023-10-30T13:25:32.219577+0000","last_fullsized":"2023-10-30T13:25:32.219577+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:54.679005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014437559000000001,"stat_sum":{"num_bytes":23150826,"num_objects":587,"num_object_clones":269,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":199,"num_read":149,"num_read_kb":38846,"num_write":1362,"num_write_kb":24887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"77'1549","reported_seq":3579,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.247414+0000","last_change":"2023-10-30T12:37:07.102887+0000","last_active":"2023-10-30T13:25:32.247414+0000","last_peered":"2023-10-30T13:25:32.247414+0000","last_clean":"2023-10-30T13:25:32.247414+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T13:25:32.247414+0000","last_undegraded":"2023-10-30T13:25:32.247414+0000","last_fullsized":"2023-10-30T13:25:32.247414+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1549,"log_dups_size":0,"ondisk_log_size":1549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:09.601324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014185980000000001,"stat_sum":{"num_bytes":36575095,"num_objects":618,"num_object_clones":282,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":216,"num_read":165,"num_read_kb":36418,"num_write":1354,"num_write_kb":45869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"77'1459","reported_seq":3491,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.224692+0000","last_change":"2023-10-30T12:37:07.183547+0000","last_active":"2023-10-30T13:25:32.224692+0000","last_peered":"2023-10-30T13:25:32.224692+0000","last_clean":"2023-10-30T13:25:32.224692+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T13:25:32.224692+0000","last_undegraded":"2023-10-30T13:25:32.224692+0000","last_fullsized":"2023-10-30T13:25:32.224692+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:33:43.076339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091404980000000004,"stat_sum":{"num_bytes":11176343,"num_objects":586,"num_object_clones":270,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":213,"num_read":154,"num_read_kb":10770,"num_write":1260,"num_write_kb":13564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"82'1511","reported_seq":3517,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.187191+0000","last_change":"2023-10-30T12:37:07.181463+0000","last_active":"2023-10-30T13:25:16.187191+0000","last_peered":"2023-10-30T13:25:16.187191+0000","last_clean":"2023-10-30T13:25:16.187191+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T13:25:16.187191+0000","last_undegraded":"2023-10-30T13:25:16.187191+0000","last_fullsized":"2023-10-30T13:25:16.187191+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1511,"log_dups_size":0,"ondisk_log_size":1511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:57:06.796994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075712790000000002,"stat_sum":{"num_bytes":23568118,"num_objects":625,"num_object_clones":283,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":230,"num_read":141,"num_read_kb":34199,"num_write":1299,"num_write_kb":24762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"84'1725","reported_seq":3645,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.138485+0000","last_change":"2023-10-30T12:37:07.181114+0000","last_active":"2023-10-30T13:25:16.138485+0000","last_peered":"2023-10-30T13:25:16.138485+0000","last_clean":"2023-10-30T13:25:16.138485+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T13:25:16.138485+0000","last_undegraded":"2023-10-30T13:25:16.138485+0000","last_fullsized":"2023-10-30T13:25:16.138485+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1725,"log_dups_size":0,"ondisk_log_size":1725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:57:46.353792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007366493,"stat_sum":{"num_bytes":24142540,"num_objects":721,"num_object_clones":385,"num_object_copies":1442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":231,"num_read":157,"num_read_kb":24833,"num_write":1452,"num_write_kb":33857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"85'1503","reported_seq":3540,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.201400+0000","last_change":"2023-10-30T12:37:07.114858+0000","last_active":"2023-10-30T13:25:16.201400+0000","last_peered":"2023-10-30T13:25:16.201400+0000","last_clean":"2023-10-30T13:25:16.201400+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T13:25:16.201400+0000","last_undegraded":"2023-10-30T13:25:16.201400+0000","last_fullsized":"2023-10-30T13:25:16.201400+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1503,"log_dups_size":0,"ondisk_log_size":1503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:18:13.789863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.006759035,"stat_sum":{"num_bytes":14254037,"num_objects":596,"num_object_clones":276,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":205,"num_read":165,"num_read_kb":21722,"num_write":1326,"num_write_kb":24306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"77'1568","reported_seq":3567,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.278066+0000","last_change":"2023-10-30T12:37:07.112609+0000","last_active":"2023-10-30T13:24:57.278066+0000","last_peered":"2023-10-30T13:24:57.278066+0000","last_clean":"2023-10-30T13:24:57.278066+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T13:24:57.278066+0000","last_undegraded":"2023-10-30T13:24:57.278066+0000","last_fullsized":"2023-10-30T13:24:57.278066+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1568,"log_dups_size":0,"ondisk_log_size":1568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:02:43.640916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057386779999999997,"stat_sum":{"num_bytes":10370740,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":213,"num_read":131,"num_read_kb":5845,"num_write":1381,"num_write_kb":12595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"77'1624","reported_seq":3604,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:46.371670+0000","last_change":"2023-10-30T12:37:07.187257+0000","last_active":"2023-10-30T13:24:46.371670+0000","last_peered":"2023-10-30T13:24:46.371670+0000","last_clean":"2023-10-30T13:24:46.371670+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T13:24:46.371670+0000","last_undegraded":"2023-10-30T13:24:46.371670+0000","last_fullsized":"2023-10-30T13:24:46.371670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:22.374872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013549705,"stat_sum":{"num_bytes":19248519,"num_objects":637,"num_object_clones":295,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":221,"num_read":128,"num_read_kb":21688,"num_write":1447,"num_write_kb":21442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"82'1574","reported_seq":3576,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.208932+0000","last_change":"2023-10-30T12:37:07.178284+0000","last_active":"2023-10-30T13:25:32.208932+0000","last_peered":"2023-10-30T13:25:32.208932+0000","last_clean":"2023-10-30T13:25:32.208932+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T13:25:32.208932+0000","last_undegraded":"2023-10-30T13:25:32.208932+0000","last_fullsized":"2023-10-30T13:25:32.208932+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1574,"log_dups_size":0,"ondisk_log_size":1574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:39:06.662733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075608309999999996,"stat_sum":{"num_bytes":14269019,"num_objects":619,"num_object_clones":290,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":233,"num_read":143,"num_read_kb":25965,"num_write":1377,"num_write_kb":20442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"85'1605","reported_seq":3590,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.146156+0000","last_change":"2023-10-30T12:37:07.182349+0000","last_active":"2023-10-30T13:25:16.146156+0000","last_peered":"2023-10-30T13:25:16.146156+0000","last_clean":"2023-10-30T13:25:16.146156+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T13:25:16.146156+0000","last_undegraded":"2023-10-30T13:25:16.146156+0000","last_fullsized":"2023-10-30T13:25:16.146156+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1605,"log_dups_size":0,"ondisk_log_size":1605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:47:59.298834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0099357109999999998,"stat_sum":{"num_bytes":14793776,"num_objects":642,"num_object_clones":304,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":242,"num_read":140,"num_read_kb":30098,"num_write":1386,"num_write_kb":20714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"82'1444","reported_seq":3467,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.136464+0000","last_change":"2023-10-30T12:37:07.182572+0000","last_active":"2023-10-30T13:25:16.136464+0000","last_peered":"2023-10-30T13:25:16.136464+0000","last_clean":"2023-10-30T13:25:16.136464+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T13:25:16.136464+0000","last_undegraded":"2023-10-30T13:25:16.136464+0000","last_fullsized":"2023-10-30T13:25:16.136464+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1444,"log_dups_size":0,"ondisk_log_size":1444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:03:41.547635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065893760000000001,"stat_sum":{"num_bytes":15508852,"num_objects":562,"num_object_clones":263,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":216,"num_read":137,"num_read_kb":10064,"num_write":1244,"num_write_kb":17198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"82'1576","reported_seq":3678,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:26.207237+0000","last_change":"2023-10-30T12:37:07.182668+0000","last_active":"2023-10-30T13:25:26.207237+0000","last_peered":"2023-10-30T13:25:26.207237+0000","last_clean":"2023-10-30T13:25:26.207237+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T13:25:26.207237+0000","last_undegraded":"2023-10-30T13:25:26.207237+0000","last_fullsized":"2023-10-30T13:25:26.207237+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1576,"log_dups_size":0,"ondisk_log_size":1576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:04:03.035306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010472766,"stat_sum":{"num_bytes":15695810,"num_objects":614,"num_object_clones":281,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":221,"num_read":233,"num_read_kb":17180,"num_write":1382,"num_write_kb":23481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"77'1569","reported_seq":3573,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.219508+0000","last_change":"2023-10-30T12:37:07.181518+0000","last_active":"2023-10-30T13:25:16.219508+0000","last_peered":"2023-10-30T13:25:16.219508+0000","last_clean":"2023-10-30T13:25:16.219508+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T13:25:16.219508+0000","last_undegraded":"2023-10-30T13:25:16.219508+0000","last_fullsized":"2023-10-30T13:25:16.219508+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:12:17.666786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075455599999999998,"stat_sum":{"num_bytes":5586810,"num_objects":614,"num_object_clones":286,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":230,"num_read":140,"num_read_kb":5846,"num_write":1359,"num_write_kb":15885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"85'1523","reported_seq":3642,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:26.213704+0000","last_change":"2023-10-30T12:37:07.113494+0000","last_active":"2023-10-30T13:25:26.213704+0000","last_peered":"2023-10-30T13:25:26.213704+0000","last_clean":"2023-10-30T13:25:26.213704+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T13:25:26.213704+0000","last_undegraded":"2023-10-30T13:25:26.213704+0000","last_fullsized":"2023-10-30T13:25:26.213704+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1523,"log_dups_size":0,"ondisk_log_size":1523,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076434509999999999,"stat_sum":{"num_bytes":23023646,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":222,"num_read":249,"num_read_kb":26208,"num_write":1319,"num_write_kb":28338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"84'1671","reported_seq":3654,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.169370+0000","last_change":"2023-10-30T12:37:07.102791+0000","last_active":"2023-10-30T13:24:56.169370+0000","last_peered":"2023-10-30T13:24:56.169370+0000","last_clean":"2023-10-30T13:24:56.169370+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T13:24:56.169370+0000","last_undegraded":"2023-10-30T13:24:56.169370+0000","last_fullsized":"2023-10-30T13:24:56.169370+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1671,"log_dups_size":0,"ondisk_log_size":1671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:38:00.347865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016105379999999999,"stat_sum":{"num_bytes":11166034,"num_objects":653,"num_object_clones":306,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":233,"num_read":139,"num_read_kb":9627,"num_write":1472,"num_write_kb":18195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"77'1520","reported_seq":3539,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288413+0000","last_change":"2023-10-30T12:37:07.188206+0000","last_active":"2023-10-30T13:24:16.288413+0000","last_peered":"2023-10-30T13:24:16.288413+0000","last_clean":"2023-10-30T13:24:16.288413+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T13:24:16.288413+0000","last_undegraded":"2023-10-30T13:24:16.288413+0000","last_fullsized":"2023-10-30T13:24:16.288413+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:21.306614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014064241,"stat_sum":{"num_bytes":10716893,"num_objects":582,"num_object_clones":264,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":203,"num_read":135,"num_read_kb":13340,"num_write":1353,"num_write_kb":16108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"85'1551","reported_seq":3554,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:23.080116+0000","last_change":"2023-10-30T12:37:07.179785+0000","last_active":"2023-10-30T13:25:23.080116+0000","last_peered":"2023-10-30T13:25:23.080116+0000","last_clean":"2023-10-30T13:25:23.080116+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T13:25:23.080116+0000","last_undegraded":"2023-10-30T13:25:23.080116+0000","last_fullsized":"2023-10-30T13:25:23.080116+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1551,"log_dups_size":0,"ondisk_log_size":1551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:49:28.717141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0088476170000000003,"stat_sum":{"num_bytes":10233195,"num_objects":597,"num_object_clones":275,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":203,"num_read":127,"num_read_kb":1819,"num_write":1383,"num_write_kb":16400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"82'1465","reported_seq":3479,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.190427+0000","last_change":"2023-10-30T12:37:07.188606+0000","last_active":"2023-10-30T13:25:16.190427+0000","last_peered":"2023-10-30T13:25:16.190427+0000","last_clean":"2023-10-30T13:25:16.190427+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T13:25:16.190427+0000","last_undegraded":"2023-10-30T13:25:16.190427+0000","last_fullsized":"2023-10-30T13:25:16.190427+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:06:30.044315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014279337,"stat_sum":{"num_bytes":22085088,"num_objects":582,"num_object_clones":270,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":216,"num_read":135,"num_read_kb":28666,"num_write":1258,"num_write_kb":25992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"77'1609","reported_seq":3602,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.171560+0000","last_change":"2023-10-30T12:37:07.177071+0000","last_active":"2023-10-30T13:25:16.171560+0000","last_peered":"2023-10-30T13:25:16.171560+0000","last_clean":"2023-10-30T13:25:16.171560+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T13:25:16.171560+0000","last_undegraded":"2023-10-30T13:25:16.171560+0000","last_fullsized":"2023-10-30T13:25:16.171560+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1609,"log_dups_size":0,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:19.563113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004006122,"stat_sum":{"num_bytes":15408826,"num_objects":598,"num_object_clones":277,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":213,"num_read":119,"num_read_kb":9406,"num_write":1408,"num_write_kb":18485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"82'1549","reported_seq":3553,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.240701+0000","last_change":"2023-10-30T12:37:07.115775+0000","last_active":"2023-10-30T13:25:32.240701+0000","last_peered":"2023-10-30T13:25:32.240701+0000","last_clean":"2023-10-30T13:25:32.240701+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T13:25:32.240701+0000","last_undegraded":"2023-10-30T13:25:32.240701+0000","last_fullsized":"2023-10-30T13:25:32.240701+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1549,"log_dups_size":0,"ondisk_log_size":1549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:46:59.732034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0059072409999999997,"stat_sum":{"num_bytes":25755688,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":240,"num_read":144,"num_read_kb":24326,"num_write":1332,"num_write_kb":28963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"85'1485","reported_seq":3466,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:07.280157+0000","last_change":"2023-10-30T12:37:07.113316+0000","last_active":"2023-10-30T13:25:07.280157+0000","last_peered":"2023-10-30T13:25:07.280157+0000","last_clean":"2023-10-30T13:25:07.280157+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T13:25:07.280157+0000","last_undegraded":"2023-10-30T13:25:07.280157+0000","last_fullsized":"2023-10-30T13:25:07.280157+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1485,"log_dups_size":0,"ondisk_log_size":1485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:19:54.028619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062401690000000003,"stat_sum":{"num_bytes":11021781,"num_objects":617,"num_object_clones":286,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":226,"num_read":116,"num_read_kb":10954,"num_write":1301,"num_write_kb":16114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"77'1620","reported_seq":3591,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.230454+0000","last_change":"2023-10-30T12:37:07.189557+0000","last_active":"2023-10-30T13:25:32.230454+0000","last_peered":"2023-10-30T13:25:32.230454+0000","last_clean":"2023-10-30T13:25:32.230454+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T13:25:32.230454+0000","last_undegraded":"2023-10-30T13:25:32.230454+0000","last_fullsized":"2023-10-30T13:25:32.230454+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1620,"log_dups_size":0,"ondisk_log_size":1620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:17:47.861374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014933905000000001,"stat_sum":{"num_bytes":11161783,"num_objects":653,"num_object_clones":309,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":240,"num_read":130,"num_read_kb":12352,"num_write":1416,"num_write_kb":13402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"82'1476","reported_seq":3519,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.195574+0000","last_change":"2023-10-30T12:37:07.180055+0000","last_active":"2023-10-30T13:25:16.195574+0000","last_peered":"2023-10-30T13:25:16.195574+0000","last_clean":"2023-10-30T13:25:16.195574+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T13:25:16.195574+0000","last_undegraded":"2023-10-30T13:25:16.195574+0000","last_fullsized":"2023-10-30T13:25:16.195574+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:39:02.149498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0090148399999999997,"stat_sum":{"num_bytes":11081748,"num_objects":549,"num_object_clones":251,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":201,"num_read":142,"num_read_kb":9474,"num_write":1288,"num_write_kb":13652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"84'1713","reported_seq":3703,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.230251+0000","last_change":"2023-10-30T12:37:07.181770+0000","last_active":"2023-10-30T13:25:16.230251+0000","last_peered":"2023-10-30T13:25:16.230251+0000","last_clean":"2023-10-30T13:25:16.230251+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T13:25:16.230251+0000","last_undegraded":"2023-10-30T13:25:16.230251+0000","last_fullsized":"2023-10-30T13:25:16.230251+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1713,"log_dups_size":0,"ondisk_log_size":1713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:44:33.319141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066657749999999997,"stat_sum":{"num_bytes":20125251,"num_objects":693,"num_object_clones":330,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":267,"num_read":168,"num_read_kb":25254,"num_write":1457,"num_write_kb":26871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"82'1548","reported_seq":3554,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.216318+0000","last_change":"2023-10-30T12:37:07.176070+0000","last_active":"2023-10-30T13:25:16.216318+0000","last_peered":"2023-10-30T13:25:16.216318+0000","last_clean":"2023-10-30T13:25:16.216318+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T13:25:16.216318+0000","last_undegraded":"2023-10-30T13:25:16.216318+0000","last_fullsized":"2023-10-30T13:25:16.216318+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:28:06.978130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005419129,"stat_sum":{"num_bytes":18031744,"num_objects":604,"num_object_clones":282,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":228,"num_read":136,"num_read_kb":25964,"num_write":1344,"num_write_kb":19625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"85'1548","reported_seq":3561,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.144906+0000","last_change":"2023-10-30T12:37:07.102583+0000","last_active":"2023-10-30T13:24:56.144906+0000","last_peered":"2023-10-30T13:24:56.144906+0000","last_clean":"2023-10-30T13:24:56.144906+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T13:24:56.144906+0000","last_undegraded":"2023-10-30T13:24:56.144906+0000","last_fullsized":"2023-10-30T13:24:56.144906+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:46:32.507814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013742470000000001,"stat_sum":{"num_bytes":22381695,"num_objects":607,"num_object_clones":275,"num_object_copies":1214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":227,"num_read":141,"num_read_kb":33421,"num_write":1348,"num_write_kb":23422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"85'991","reported_seq":1403,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:53.837590+0000","last_change":"2023-10-30T12:28:44.074842+0000","last_active":"2023-10-30T13:24:53.837590+0000","last_peered":"2023-10-30T13:24:53.837590+0000","last_clean":"2023-10-30T13:24:53.837590+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T13:24:53.837590+0000","last_undegraded":"2023-10-30T13:24:53.837590+0000","last_fullsized":"2023-10-30T13:24:53.837590+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":991,"log_dups_size":0,"ondisk_log_size":991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:18:01.428785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":570,"num_read_kb":319827,"num_write":1218,"num_write_kb":80521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"77'1561","reported_seq":3571,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.278739+0000","last_change":"2023-10-30T12:37:07.114520+0000","last_active":"2023-10-30T13:25:32.278739+0000","last_peered":"2023-10-30T13:25:32.278739+0000","last_clean":"2023-10-30T13:25:32.278739+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T13:25:32.278739+0000","last_undegraded":"2023-10-30T13:25:32.278739+0000","last_fullsized":"2023-10-30T13:25:32.278739+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1561,"log_dups_size":0,"ondisk_log_size":1561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086018069999999995,"stat_sum":{"num_bytes":19940185,"num_objects":616,"num_object_clones":283,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":213,"num_read":145,"num_read_kb":23669,"num_write":1398,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1795","reported_seq":2375,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:52.266854+0000","last_change":"2023-10-30T12:28:44.060621+0000","last_active":"2023-10-30T13:24:52.266854+0000","last_peered":"2023-10-30T13:24:52.266854+0000","last_clean":"2023-10-30T13:24:52.266854+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T13:24:52.266854+0000","last_undegraded":"2023-10-30T13:24:52.266854+0000","last_fullsized":"2023-10-30T13:24:52.266854+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1795,"log_dups_size":0,"ondisk_log_size":1795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:36:53.741238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17028825,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":761,"num_read_kb":396040,"num_write":1931,"num_write_kb":84639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"85'1570","reported_seq":2388,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:27.329932+0000","last_change":"2023-10-30T12:28:44.060171+0000","last_active":"2023-10-30T13:25:27.329932+0000","last_peered":"2023-10-30T13:25:27.329932+0000","last_clean":"2023-10-30T13:25:27.329932+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T13:25:27.329932+0000","last_undegraded":"2023-10-30T13:25:27.329932+0000","last_fullsized":"2023-10-30T13:25:27.329932+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1570,"log_dups_size":0,"ondisk_log_size":1570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:43:18.181418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182949,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":1117,"num_read_kb":268002,"num_write":1800,"num_write_kb":63122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"77'1801","reported_seq":3809,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:25.581602+0000","last_change":"2023-10-30T12:37:07.191924+0000","last_active":"2023-10-30T13:25:25.581602+0000","last_peered":"2023-10-30T13:25:25.581602+0000","last_clean":"2023-10-30T13:25:25.581602+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T13:25:25.581602+0000","last_undegraded":"2023-10-30T13:25:25.581602+0000","last_fullsized":"2023-10-30T13:25:25.581602+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1801,"log_dups_size":0,"ondisk_log_size":1801,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:27.781535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.016143469000000001,"stat_sum":{"num_bytes":14540052,"num_objects":709,"num_object_clones":329,"num_object_copies":1418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":254,"num_read":191,"num_read_kb":6719,"num_write":1583,"num_write_kb":26382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"85'930","reported_seq":1332,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:52.371111+0000","last_change":"2023-10-30T12:28:44.058798+0000","last_active":"2023-10-30T13:24:52.371111+0000","last_peered":"2023-10-30T13:24:52.371111+0000","last_clean":"2023-10-30T13:24:52.371111+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T13:24:52.371111+0000","last_undegraded":"2023-10-30T13:24:52.371111+0000","last_fullsized":"2023-10-30T13:24:52.371111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":930,"log_dups_size":0,"ondisk_log_size":930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:08.643080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":542,"num_read_kb":359029,"num_write":1093,"num_write_kb":88246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"77'1524","reported_seq":3544,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.194605+0000","last_change":"2023-10-30T12:37:07.114493+0000","last_active":"2023-10-30T13:25:16.194605+0000","last_peered":"2023-10-30T13:25:16.194605+0000","last_clean":"2023-10-30T13:25:16.194605+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T13:25:16.194605+0000","last_undegraded":"2023-10-30T13:25:16.194605+0000","last_fullsized":"2023-10-30T13:25:16.194605+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1524,"log_dups_size":0,"ondisk_log_size":1524,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0083101389999999994,"stat_sum":{"num_bytes":18904879,"num_objects":589,"num_object_clones":276,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":211,"num_read":147,"num_read_kb":20021,"num_write":1340,"num_write_kb":25244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"85'737","reported_seq":1137,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:04.838635+0000","last_change":"2023-10-30T12:28:44.063439+0000","last_active":"2023-10-30T13:25:04.838635+0000","last_peered":"2023-10-30T13:25:04.838635+0000","last_clean":"2023-10-30T13:25:04.838635+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T13:25:04.838635+0000","last_undegraded":"2023-10-30T13:25:04.838635+0000","last_fullsized":"2023-10-30T13:25:04.838635+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":737,"log_dups_size":0,"ondisk_log_size":737,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:08.725544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":616,"num_read_kb":251423,"num_write":890,"num_write_kb":55054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"77'1518","reported_seq":3533,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.444813+0000","last_change":"2023-10-30T12:37:07.177520+0000","last_active":"2023-10-30T13:25:10.444813+0000","last_peered":"2023-10-30T13:25:10.444813+0000","last_clean":"2023-10-30T13:25:10.444813+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T13:25:10.444813+0000","last_undegraded":"2023-10-30T13:25:10.444813+0000","last_fullsized":"2023-10-30T13:25:10.444813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:46:22.808814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0043045619999999996,"stat_sum":{"num_bytes":10134301,"num_objects":580,"num_object_clones":271,"num_object_copies":1160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":580,"num_whiteouts":219,"num_read":138,"num_read_kb":8986,"num_write":1318,"num_write_kb":11276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"85'1247","reported_seq":1685,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:24.345143+0000","last_change":"2023-10-30T12:28:44.052641+0000","last_active":"2023-10-30T13:24:24.345143+0000","last_peered":"2023-10-30T13:24:24.345143+0000","last_clean":"2023-10-30T13:24:24.345143+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T13:24:24.345143+0000","last_undegraded":"2023-10-30T13:24:24.345143+0000","last_fullsized":"2023-10-30T13:24:24.345143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1247,"log_dups_size":0,"ondisk_log_size":1247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:17:42.523766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":659,"num_read_kb":339384,"num_write":1492,"num_write_kb":80693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"82'1603","reported_seq":3592,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.115007+0000","last_change":"2023-10-30T12:37:07.191144+0000","last_active":"2023-10-30T13:24:56.115007+0000","last_peered":"2023-10-30T13:24:56.115007+0000","last_clean":"2023-10-30T13:24:56.115007+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T13:24:56.115007+0000","last_undegraded":"2023-10-30T13:24:56.115007+0000","last_fullsized":"2023-10-30T13:24:56.115007+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1603,"log_dups_size":0,"ondisk_log_size":1603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:35:46.268962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015702779,"stat_sum":{"num_bytes":15773172,"num_objects":620,"num_object_clones":292,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":204,"num_read":135,"num_read_kb":17351,"num_write":1431,"num_write_kb":17077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"85'1224","reported_seq":1717,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:35.202116+0000","last_change":"2023-10-30T12:28:44.075783+0000","last_active":"2023-10-30T13:25:35.202116+0000","last_peered":"2023-10-30T13:25:35.202116+0000","last_clean":"2023-10-30T13:25:35.202116+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T13:25:35.202116+0000","last_undegraded":"2023-10-30T13:25:35.202116+0000","last_fullsized":"2023-10-30T13:25:35.202116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1224,"log_dups_size":0,"ondisk_log_size":1224,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:52:01.094929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19774519,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":862,"num_read_kb":221068,"num_write":1454,"num_write_kb":51536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"82'1515","reported_seq":3524,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.077418+0000","last_change":"2023-10-30T12:37:07.178872+0000","last_active":"2023-10-30T13:25:03.077418+0000","last_peered":"2023-10-30T13:25:03.077418+0000","last_clean":"2023-10-30T13:25:03.077418+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T13:25:03.077418+0000","last_undegraded":"2023-10-30T13:25:03.077418+0000","last_fullsized":"2023-10-30T13:25:03.077418+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:27:03.308323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080610860000000003,"stat_sum":{"num_bytes":22077043,"num_objects":612,"num_object_clones":281,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":230,"num_read":141,"num_read_kb":18856,"num_write":1294,"num_write_kb":28337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"85'1331","reported_seq":1877,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:12.181476+0000","last_change":"2023-10-30T12:28:44.063367+0000","last_active":"2023-10-30T13:25:12.181476+0000","last_peered":"2023-10-30T13:25:12.181476+0000","last_clean":"2023-10-30T13:25:12.181476+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T13:25:12.181476+0000","last_undegraded":"2023-10-30T13:25:12.181476+0000","last_fullsized":"2023-10-30T13:25:12.181476+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1331,"log_dups_size":0,"ondisk_log_size":1331,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:56:45.585121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":865,"num_read_kb":420163,"num_write":1566,"num_write_kb":89397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"82'1535","reported_seq":3533,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.355487+0000","last_change":"2023-10-30T12:37:07.182869+0000","last_active":"2023-10-30T13:24:48.355487+0000","last_peered":"2023-10-30T13:24:48.355487+0000","last_clean":"2023-10-30T13:24:48.355487+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T13:24:48.355487+0000","last_undegraded":"2023-10-30T13:24:48.355487+0000","last_fullsized":"2023-10-30T13:24:48.355487+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1535,"log_dups_size":0,"ondisk_log_size":1535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:22:35.734555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.008640567,"stat_sum":{"num_bytes":23022369,"num_objects":606,"num_object_clones":279,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":220,"num_read":129,"num_read_kb":18941,"num_write":1340,"num_write_kb":24878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"85'1098","reported_seq":1558,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.290468+0000","last_change":"2023-10-30T12:28:44.065545+0000","last_active":"2023-10-30T13:24:47.290468+0000","last_peered":"2023-10-30T13:24:47.290468+0000","last_clean":"2023-10-30T13:24:47.290468+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T13:24:47.290468+0000","last_undegraded":"2023-10-30T13:24:47.290468+0000","last_fullsized":"2023-10-30T13:24:47.290468+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1098,"log_dups_size":0,"ondisk_log_size":1098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:38:09.900189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554466,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":701,"num_read_kb":318483,"num_write":1303,"num_write_kb":68223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"84'1584","reported_seq":3584,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.315736+0000","last_change":"2023-10-30T12:37:07.115595+0000","last_active":"2023-10-30T13:24:48.315736+0000","last_peered":"2023-10-30T13:24:48.315736+0000","last_clean":"2023-10-30T13:24:48.315736+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T13:24:48.315736+0000","last_undegraded":"2023-10-30T13:24:48.315736+0000","last_fullsized":"2023-10-30T13:24:48.315736+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1584,"log_dups_size":0,"ondisk_log_size":1584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:20:44.527701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0073009930000000004,"stat_sum":{"num_bytes":10084838,"num_objects":613,"num_object_clones":288,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":217,"num_read":139,"num_read_kb":5686,"num_write":1383,"num_write_kb":20147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'676","reported_seq":1068,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.649328+0000","last_change":"2023-10-30T12:28:44.061922+0000","last_active":"2023-10-30T13:24:57.649328+0000","last_peered":"2023-10-30T13:24:57.649328+0000","last_clean":"2023-10-30T13:24:57.649328+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T13:24:57.649328+0000","last_undegraded":"2023-10-30T13:24:57.649328+0000","last_fullsized":"2023-10-30T13:24:57.649328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:29:34.207258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":600,"num_read_kb":208389,"num_write":899,"num_write_kb":51587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"82'1642","reported_seq":3652,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.151917+0000","last_change":"2023-10-30T12:37:07.180453+0000","last_active":"2023-10-30T13:25:16.151917+0000","last_peered":"2023-10-30T13:25:16.151917+0000","last_clean":"2023-10-30T13:25:16.151917+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T13:25:16.151917+0000","last_undegraded":"2023-10-30T13:25:16.151917+0000","last_fullsized":"2023-10-30T13:25:16.151917+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:46:20.518544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091258150000000007,"stat_sum":{"num_bytes":19392609,"num_objects":655,"num_object_clones":299,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":233,"num_read":161,"num_read_kb":7475,"num_write":1440,"num_write_kb":20643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"85'1371","reported_seq":1755,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.074769+0000","last_change":"2023-10-30T12:28:44.059784+0000","last_active":"2023-10-30T13:24:48.074769+0000","last_peered":"2023-10-30T13:24:48.074769+0000","last_clean":"2023-10-30T13:24:48.074769+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T13:24:48.074769+0000","last_undegraded":"2023-10-30T13:24:48.074769+0000","last_fullsized":"2023-10-30T13:24:48.074769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1371,"log_dups_size":0,"ondisk_log_size":1371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:11.468221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":539,"num_read_kb":266844,"num_write":1512,"num_write_kb":63647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"77'1481","reported_seq":3489,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:48.310426+0000","last_change":"2023-10-30T12:37:07.102525+0000","last_active":"2023-10-30T13:24:48.310426+0000","last_peered":"2023-10-30T13:24:48.310426+0000","last_clean":"2023-10-30T13:24:48.310426+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T13:24:48.310426+0000","last_undegraded":"2023-10-30T13:24:48.310426+0000","last_fullsized":"2023-10-30T13:24:48.310426+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:22.814663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001646376,"stat_sum":{"num_bytes":13778361,"num_objects":590,"num_object_clones":278,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":227,"num_read":137,"num_read_kb":5723,"num_write":1258,"num_write_kb":14976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"85'970","reported_seq":1456,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:52.631806+0000","last_change":"2023-10-30T12:28:44.065885+0000","last_active":"2023-10-30T13:24:52.631806+0000","last_peered":"2023-10-30T13:24:52.631806+0000","last_clean":"2023-10-30T13:24:52.631806+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T13:24:52.631806+0000","last_undegraded":"2023-10-30T13:24:52.631806+0000","last_fullsized":"2023-10-30T13:24:52.631806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":970,"log_dups_size":0,"ondisk_log_size":970,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554544,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":863,"num_read_kb":272228,"num_write":1253,"num_write_kb":56723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"82'1616","reported_seq":3636,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.437909+0000","last_change":"2023-10-30T12:37:07.181542+0000","last_active":"2023-10-30T13:25:10.437909+0000","last_peered":"2023-10-30T13:25:10.437909+0000","last_clean":"2023-10-30T13:25:10.437909+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T13:25:10.437909+0000","last_undegraded":"2023-10-30T13:25:10.437909+0000","last_fullsized":"2023-10-30T13:25:10.437909+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1616,"log_dups_size":0,"ondisk_log_size":1616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:16:47.953145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098326820000000006,"stat_sum":{"num_bytes":10901701,"num_objects":622,"num_object_clones":292,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":232,"num_read":163,"num_read_kb":14238,"num_write":1406,"num_write_kb":21652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"85'655","reported_seq":1014,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:04.285953+0000","last_change":"2023-10-30T12:28:44.080011+0000","last_active":"2023-10-30T13:25:04.285953+0000","last_peered":"2023-10-30T13:25:04.285953+0000","last_clean":"2023-10-30T13:25:04.285953+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T13:25:04.285953+0000","last_undegraded":"2023-10-30T13:25:04.285953+0000","last_fullsized":"2023-10-30T13:25:04.285953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:07:17.215885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":443,"num_read_kb":261670,"num_write":858,"num_write_kb":58986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"77'1573","reported_seq":3538,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.065502+0000","last_change":"2023-10-30T12:37:07.177898+0000","last_active":"2023-10-30T13:25:03.065502+0000","last_peered":"2023-10-30T13:25:03.065502+0000","last_clean":"2023-10-30T13:25:03.065502+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T13:25:03.065502+0000","last_undegraded":"2023-10-30T13:25:03.065502+0000","last_fullsized":"2023-10-30T13:25:03.065502+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1573,"log_dups_size":0,"ondisk_log_size":1573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:30:52.251046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004602437,"stat_sum":{"num_bytes":20428330,"num_objects":647,"num_object_clones":304,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":239,"num_read":119,"num_read_kb":13097,"num_write":1360,"num_write_kb":26395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"85'717","reported_seq":1167,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.122991+0000","last_change":"2023-10-30T12:28:44.076352+0000","last_active":"2023-10-30T13:25:02.122991+0000","last_peered":"2023-10-30T13:25:02.122991+0000","last_clean":"2023-10-30T13:25:02.122991+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T13:25:02.122991+0000","last_undegraded":"2023-10-30T13:25:02.122991+0000","last_fullsized":"2023-10-30T13:25:02.122991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":717,"log_dups_size":0,"ondisk_log_size":717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":635,"num_read_kb":434171,"num_write":906,"num_write_kb":88121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"77'1575","reported_seq":3560,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.221074+0000","last_change":"2023-10-30T12:37:07.181502+0000","last_active":"2023-10-30T13:25:32.221074+0000","last_peered":"2023-10-30T13:25:32.221074+0000","last_clean":"2023-10-30T13:25:32.221074+0000","last_became_active":"2023-10-30T12:29:27.340504+0000","last_became_peered":"2023-10-30T12:29:27.340504+0000","last_unstale":"2023-10-30T13:25:32.221074+0000","last_undegraded":"2023-10-30T13:25:32.221074+0000","last_fullsized":"2023-10-30T13:25:32.221074+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1575,"log_dups_size":0,"ondisk_log_size":1575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:48:15.263466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00653923,"stat_sum":{"num_bytes":23737065,"num_objects":641,"num_object_clones":299,"num_object_copies":1282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":641,"num_whiteouts":238,"num_read":138,"num_read_kb":28255,"num_write":1351,"num_write_kb":33550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'716","reported_seq":1095,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.276257+0000","last_change":"2023-10-30T12:28:44.075684+0000","last_active":"2023-10-30T13:24:47.276257+0000","last_peered":"2023-10-30T13:24:47.276257+0000","last_clean":"2023-10-30T13:24:47.276257+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T13:24:47.276257+0000","last_undegraded":"2023-10-30T13:24:47.276257+0000","last_fullsized":"2023-10-30T13:24:47.276257+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":716,"log_dups_size":0,"ondisk_log_size":716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:31:07.375795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364369,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":493,"num_read_kb":320778,"num_write":891,"num_write_kb":71265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"77'1599","reported_seq":3612,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.080865+0000","last_change":"2023-10-30T12:37:07.181841+0000","last_active":"2023-10-30T13:25:03.080865+0000","last_peered":"2023-10-30T13:25:03.080865+0000","last_clean":"2023-10-30T13:25:03.080865+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T13:25:03.080865+0000","last_undegraded":"2023-10-30T13:25:03.080865+0000","last_fullsized":"2023-10-30T13:25:03.080865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1599,"log_dups_size":0,"ondisk_log_size":1599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:35:15.227222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076673150000000001,"stat_sum":{"num_bytes":18256447,"num_objects":611,"num_object_clones":286,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":229,"num_read":149,"num_read_kb":24381,"num_write":1383,"num_write_kb":20195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"85'607","reported_seq":959,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.278226+0000","last_change":"2023-10-30T12:28:44.060686+0000","last_active":"2023-10-30T13:24:47.278226+0000","last_peered":"2023-10-30T13:24:47.278226+0000","last_clean":"2023-10-30T13:24:47.278226+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T13:24:47.278226+0000","last_undegraded":"2023-10-30T13:24:47.278226+0000","last_fullsized":"2023-10-30T13:24:47.278226+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":607,"log_dups_size":0,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:13:23.155370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360710,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":462,"num_read_kb":342669,"num_write":689,"num_write_kb":66904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"82'1520","reported_seq":3502,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.077446+0000","last_change":"2023-10-30T12:37:07.182282+0000","last_active":"2023-10-30T13:25:03.077446+0000","last_peered":"2023-10-30T13:25:03.077446+0000","last_clean":"2023-10-30T13:25:03.077446+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T13:25:03.077446+0000","last_undegraded":"2023-10-30T13:25:03.077446+0000","last_fullsized":"2023-10-30T13:25:03.077446+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:40:14.913475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010286791999999999,"stat_sum":{"num_bytes":26944901,"num_objects":612,"num_object_clones":287,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":236,"num_read":119,"num_read_kb":34683,"num_write":1299,"num_write_kb":33129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":256,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:16.288773+0000","last_change":"2023-10-30T12:28:33.433824+0000","last_active":"2023-10-30T13:24:16.288773+0000","last_peered":"2023-10-30T13:24:16.288773+0000","last_clean":"2023-10-30T13:24:16.288773+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T13:24:16.288773+0000","last_undegraded":"2023-10-30T13:24:16.288773+0000","last_fullsized":"2023-10-30T13:24:16.288773+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:31.408255+0000","last_clean_scrub_stamp":"2023-10-30T12:28:31.408255+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:57:17.624179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'1318","reported_seq":1818,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.269121+0000","last_change":"2023-10-30T12:28:44.059051+0000","last_active":"2023-10-30T13:24:47.269121+0000","last_peered":"2023-10-30T13:24:47.269121+0000","last_clean":"2023-10-30T13:24:47.269121+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T13:24:47.269121+0000","last_undegraded":"2023-10-30T13:24:47.269121+0000","last_fullsized":"2023-10-30T13:24:47.269121+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1318,"log_dups_size":0,"ondisk_log_size":1318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:37:54.810636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165890,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":877,"num_read_kb":231219,"num_write":1489,"num_write_kb":63137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"85'1574","reported_seq":3583,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.205997+0000","last_change":"2023-10-30T12:37:07.116603+0000","last_active":"2023-10-30T13:25:16.205997+0000","last_peered":"2023-10-30T13:25:16.205997+0000","last_clean":"2023-10-30T13:25:16.205997+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T13:25:16.205997+0000","last_undegraded":"2023-10-30T13:25:16.205997+0000","last_fullsized":"2023-10-30T13:25:16.205997+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1574,"log_dups_size":0,"ondisk_log_size":1574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0068470270000000003,"stat_sum":{"num_bytes":8097312,"num_objects":626,"num_object_clones":290,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":225,"num_read":146,"num_read_kb":2079,"num_write":1425,"num_write_kb":13192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"85'2192","reported_seq":2563,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.650111+0000","last_change":"2023-10-30T12:28:44.080378+0000","last_active":"2023-10-30T13:24:57.650111+0000","last_peered":"2023-10-30T13:24:57.650111+0000","last_clean":"2023-10-30T13:24:57.650111+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T13:24:57.650111+0000","last_undegraded":"2023-10-30T13:24:57.650111+0000","last_fullsized":"2023-10-30T13:24:57.650111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":2192,"log_dups_size":0,"ondisk_log_size":2192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:22.964995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":516,"num_read_kb":283293,"num_write":2369,"num_write_kb":72030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"77'1465","reported_seq":3487,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:15.952966+0000","last_change":"2023-10-30T12:37:07.112121+0000","last_active":"2023-10-30T13:24:15.952966+0000","last_peered":"2023-10-30T13:24:15.952966+0000","last_clean":"2023-10-30T13:24:15.952966+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T13:24:15.952966+0000","last_undegraded":"2023-10-30T13:24:15.952966+0000","last_fullsized":"2023-10-30T13:24:15.952966+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:12:48.778193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057755250000000001,"stat_sum":{"num_bytes":14269585,"num_objects":570,"num_object_clones":262,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":207,"num_read":132,"num_read_kb":18022,"num_write":1264,"num_write_kb":17917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'1838","reported_seq":3123,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:09.242564+0000","last_change":"2023-10-30T12:28:44.082480+0000","last_active":"2023-10-30T13:25:09.242564+0000","last_peered":"2023-10-30T13:25:09.242564+0000","last_clean":"2023-10-30T13:25:09.242564+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T13:25:09.242564+0000","last_undegraded":"2023-10-30T13:25:09.242564+0000","last_fullsized":"2023-10-30T13:25:09.242564+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1838,"log_dups_size":0,"ondisk_log_size":1838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:20.126782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49235352,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":1514,"num_read_kb":360508,"num_write":2035,"num_write_kb":92558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"77'1546","reported_seq":3557,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.217232+0000","last_change":"2023-10-30T12:37:07.190318+0000","last_active":"2023-10-30T13:25:32.217232+0000","last_peered":"2023-10-30T13:25:32.217232+0000","last_clean":"2023-10-30T13:25:32.217232+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T13:25:32.217232+0000","last_undegraded":"2023-10-30T13:25:32.217232+0000","last_fullsized":"2023-10-30T13:25:32.217232+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:19.397149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015331312,"stat_sum":{"num_bytes":28178512,"num_objects":629,"num_object_clones":288,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":231,"num_read":148,"num_read_kb":41809,"num_write":1347,"num_write_kb":29520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'1196","reported_seq":1605,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.648443+0000","last_change":"2023-10-30T12:28:44.059806+0000","last_active":"2023-10-30T13:24:57.648443+0000","last_peered":"2023-10-30T13:24:57.648443+0000","last_clean":"2023-10-30T13:24:57.648443+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T13:24:57.648443+0000","last_undegraded":"2023-10-30T13:24:57.648443+0000","last_fullsized":"2023-10-30T13:24:57.648443+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1196,"log_dups_size":0,"ondisk_log_size":1196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:45:50.939946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":573,"num_read_kb":334248,"num_write":1338,"num_write_kb":75895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"85'1581","reported_seq":3581,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.228379+0000","last_change":"2023-10-30T12:37:07.178785+0000","last_active":"2023-10-30T13:25:16.228379+0000","last_peered":"2023-10-30T13:25:16.228379+0000","last_clean":"2023-10-30T13:25:16.228379+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T13:25:16.228379+0000","last_undegraded":"2023-10-30T13:25:16.228379+0000","last_fullsized":"2023-10-30T13:25:16.228379+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1581,"log_dups_size":0,"ondisk_log_size":1581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:01:27.865544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051193749999999998,"stat_sum":{"num_bytes":23443395,"num_objects":627,"num_object_clones":293,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":236,"num_read":141,"num_read_kb":26943,"num_write":1376,"num_write_kb":26009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"85'1467","reported_seq":1927,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:01.374791+0000","last_change":"2023-10-30T12:28:44.074970+0000","last_active":"2023-10-30T13:25:01.374791+0000","last_peered":"2023-10-30T13:25:01.374791+0000","last_clean":"2023-10-30T13:25:01.374791+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T13:25:01.374791+0000","last_undegraded":"2023-10-30T13:25:01.374791+0000","last_fullsized":"2023-10-30T13:25:01.374791+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1467,"log_dups_size":0,"ondisk_log_size":1467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:33.562703+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":699,"num_read_kb":404466,"num_write":1620,"num_write_kb":92575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"82'1612","reported_seq":3615,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:18.067135+0000","last_change":"2023-10-30T12:37:07.176969+0000","last_active":"2023-10-30T13:25:18.067135+0000","last_peered":"2023-10-30T13:25:18.067135+0000","last_clean":"2023-10-30T13:25:18.067135+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T13:25:18.067135+0000","last_undegraded":"2023-10-30T13:25:18.067135+0000","last_fullsized":"2023-10-30T13:25:18.067135+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1612,"log_dups_size":0,"ondisk_log_size":1612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:33:15.916870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039865179999999997,"stat_sum":{"num_bytes":21812744,"num_objects":637,"num_object_clones":298,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":235,"num_read":149,"num_read_kb":24572,"num_write":1399,"num_write_kb":31699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"85'1013","reported_seq":1422,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.646473+0000","last_change":"2023-10-30T12:28:44.063256+0000","last_active":"2023-10-30T13:24:57.646473+0000","last_peered":"2023-10-30T13:24:57.646473+0000","last_clean":"2023-10-30T13:24:57.646473+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T13:24:57.646473+0000","last_undegraded":"2023-10-30T13:24:57.646473+0000","last_fullsized":"2023-10-30T13:24:57.646473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1013,"log_dups_size":0,"ondisk_log_size":1013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:01:36.144966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":534,"num_read_kb":318720,"num_write":1164,"num_write_kb":75805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"77'1542","reported_seq":3567,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.343364+0000","last_change":"2023-10-30T12:37:07.114718+0000","last_active":"2023-10-30T13:25:32.343364+0000","last_peered":"2023-10-30T13:25:32.343364+0000","last_clean":"2023-10-30T13:25:32.343364+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T13:25:32.343364+0000","last_undegraded":"2023-10-30T13:25:32.343364+0000","last_fullsized":"2023-10-30T13:25:32.343364+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1542,"log_dups_size":0,"ondisk_log_size":1542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:09:05.944397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070372949999999998,"stat_sum":{"num_bytes":38977850,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":222,"num_read":156,"num_read_kb":47793,"num_write":1351,"num_write_kb":41194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1138","reported_seq":1573,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:37.181339+0000","last_change":"2023-10-30T12:28:44.055374+0000","last_active":"2023-10-30T13:25:37.181339+0000","last_peered":"2023-10-30T13:25:37.181339+0000","last_clean":"2023-10-30T13:25:37.181339+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T13:25:37.181339+0000","last_undegraded":"2023-10-30T13:25:37.181339+0000","last_fullsized":"2023-10-30T13:25:37.181339+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1138,"log_dups_size":0,"ondisk_log_size":1138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":644,"num_read_kb":249397,"num_write":1860,"num_write_kb":63729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"77'1449","reported_seq":3447,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:07.124077+0000","last_change":"2023-10-30T12:37:07.115578+0000","last_active":"2023-10-30T13:25:07.124077+0000","last_peered":"2023-10-30T13:25:07.124077+0000","last_clean":"2023-10-30T13:25:07.124077+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T13:25:07.124077+0000","last_undegraded":"2023-10-30T13:25:07.124077+0000","last_fullsized":"2023-10-30T13:25:07.124077+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1449,"log_dups_size":0,"ondisk_log_size":1449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074996079999999996,"stat_sum":{"num_bytes":14238343,"num_objects":603,"num_object_clones":280,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":229,"num_read":126,"num_read_kb":9383,"num_write":1239,"num_write_kb":19514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"85'490","reported_seq":839,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:09.837522+0000","last_change":"2023-10-30T12:28:44.058663+0000","last_active":"2023-10-30T13:25:09.837522+0000","last_peered":"2023-10-30T13:25:09.837522+0000","last_clean":"2023-10-30T13:25:09.837522+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T13:25:09.837522+0000","last_undegraded":"2023-10-30T13:25:09.837522+0000","last_fullsized":"2023-10-30T13:25:09.837522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T12:42:22.279653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21146631,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":514,"num_read_kb":197148,"num_write":651,"num_write_kb":42826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"77'1649","reported_seq":3624,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.355689+0000","last_change":"2023-10-30T12:37:07.115477+0000","last_active":"2023-10-30T13:25:32.355689+0000","last_peered":"2023-10-30T13:25:32.355689+0000","last_clean":"2023-10-30T13:25:32.355689+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T13:25:32.355689+0000","last_undegraded":"2023-10-30T13:25:32.355689+0000","last_fullsized":"2023-10-30T13:25:32.355689+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1649,"log_dups_size":0,"ondisk_log_size":1649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0085509929999999998,"stat_sum":{"num_bytes":14746405,"num_objects":647,"num_object_clones":344,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":203,"num_read":167,"num_read_kb":6583,"num_write":1369,"num_write_kb":17013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'870","reported_seq":1184,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.657578+0000","last_change":"2023-10-30T12:28:44.054338+0000","last_active":"2023-10-30T13:25:02.657578+0000","last_peered":"2023-10-30T13:25:02.657578+0000","last_clean":"2023-10-30T13:25:02.657578+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T13:25:02.657578+0000","last_undegraded":"2023-10-30T13:25:02.657578+0000","last_fullsized":"2023-10-30T13:25:02.657578+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":870,"log_dups_size":0,"ondisk_log_size":870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:55:35.370652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":472,"num_read_kb":149352,"num_write":1030,"num_write_kb":38491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"77'1485","reported_seq":3500,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.168854+0000","last_change":"2023-10-30T12:37:07.181506+0000","last_active":"2023-10-30T13:25:16.168854+0000","last_peered":"2023-10-30T13:25:16.168854+0000","last_clean":"2023-10-30T13:25:16.168854+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T13:25:16.168854+0000","last_undegraded":"2023-10-30T13:25:16.168854+0000","last_fullsized":"2023-10-30T13:25:16.168854+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1485,"log_dups_size":0,"ondisk_log_size":1485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:53:57.876949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096369869999999996,"stat_sum":{"num_bytes":18113290,"num_objects":584,"num_object_clones":270,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":219,"num_read":134,"num_read_kb":20299,"num_write":1266,"num_write_kb":19768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"85'777","reported_seq":1141,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:07.123925+0000","last_change":"2023-10-30T12:28:44.064606+0000","last_active":"2023-10-30T13:25:07.123925+0000","last_peered":"2023-10-30T13:25:07.123925+0000","last_clean":"2023-10-30T13:25:07.123925+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T13:25:07.123925+0000","last_undegraded":"2023-10-30T13:25:07.123925+0000","last_fullsized":"2023-10-30T13:25:07.123925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":777,"log_dups_size":0,"ondisk_log_size":777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777306,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":495,"num_read_kb":170685,"num_write":965,"num_write_kb":43434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"82'1567","reported_seq":3573,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.233660+0000","last_change":"2023-10-30T12:37:07.112752+0000","last_active":"2023-10-30T13:25:32.233660+0000","last_peered":"2023-10-30T13:25:32.233660+0000","last_clean":"2023-10-30T13:25:32.233660+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T13:25:32.233660+0000","last_undegraded":"2023-10-30T13:25:32.233660+0000","last_fullsized":"2023-10-30T13:25:32.233660+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1567,"log_dups_size":0,"ondisk_log_size":1567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007081922,"stat_sum":{"num_bytes":14751960,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":228,"num_read":137,"num_read_kb":2516,"num_write":1376,"num_write_kb":21368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1125","reported_seq":1613,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:17.281839+0000","last_change":"2023-10-30T12:28:44.072754+0000","last_active":"2023-10-30T13:25:17.281839+0000","last_peered":"2023-10-30T13:25:17.281839+0000","last_clean":"2023-10-30T13:25:17.281839+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T13:25:17.281839+0000","last_undegraded":"2023-10-30T13:25:17.281839+0000","last_fullsized":"2023-10-30T13:25:17.281839+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1125,"log_dups_size":0,"ondisk_log_size":1125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:51:55.865069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":763,"num_read_kb":345628,"num_write":1535,"num_write_kb":76701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"77'1460","reported_seq":3474,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.160346+0000","last_change":"2023-10-30T12:37:07.182563+0000","last_active":"2023-10-30T13:24:56.160346+0000","last_peered":"2023-10-30T13:24:56.160346+0000","last_clean":"2023-10-30T13:24:56.160346+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T13:24:56.160346+0000","last_undegraded":"2023-10-30T13:24:56.160346+0000","last_fullsized":"2023-10-30T13:24:56.160346+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1460,"log_dups_size":0,"ondisk_log_size":1460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:29:22.948042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060337460000000004,"stat_sum":{"num_bytes":23618154,"num_objects":570,"num_object_clones":263,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":209,"num_read":127,"num_read_kb":29654,"num_write":1265,"num_write_kb":27771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"85'1182","reported_seq":1585,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:58.045831+0000","last_change":"2023-10-30T12:28:44.058801+0000","last_active":"2023-10-30T13:24:58.045831+0000","last_peered":"2023-10-30T13:24:58.045831+0000","last_clean":"2023-10-30T13:24:58.045831+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T13:24:58.045831+0000","last_undegraded":"2023-10-30T13:24:58.045831+0000","last_fullsized":"2023-10-30T13:24:58.045831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1182,"log_dups_size":0,"ondisk_log_size":1182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:11:24.706813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":538,"num_read_kb":321476,"num_write":1330,"num_write_kb":79987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"85'1546","reported_seq":3551,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.277281+0000","last_change":"2023-10-30T12:37:07.180393+0000","last_active":"2023-10-30T13:24:47.277281+0000","last_peered":"2023-10-30T13:24:47.277281+0000","last_clean":"2023-10-30T13:24:47.277281+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T13:24:47.277281+0000","last_undegraded":"2023-10-30T13:24:47.277281+0000","last_fullsized":"2023-10-30T13:24:47.277281+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:28:37.347215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091395910000000007,"stat_sum":{"num_bytes":24394000,"num_objects":612,"num_object_clones":279,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":219,"num_read":133,"num_read_kb":28881,"num_write":1364,"num_write_kb":25892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"77'1527","reported_seq":3550,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.236448+0000","last_change":"2023-10-30T12:37:07.182718+0000","last_active":"2023-10-30T13:25:32.236448+0000","last_peered":"2023-10-30T13:25:32.236448+0000","last_clean":"2023-10-30T13:25:32.236448+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T13:25:32.236448+0000","last_undegraded":"2023-10-30T13:25:32.236448+0000","last_fullsized":"2023-10-30T13:25:32.236448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1527,"log_dups_size":0,"ondisk_log_size":1527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:12:33.103610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010120377999999999,"stat_sum":{"num_bytes":19538275,"num_objects":582,"num_object_clones":275,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":200,"num_read":151,"num_read_kb":26132,"num_write":1354,"num_write_kb":27822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"85'880","reported_seq":1327,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.246911+0000","last_change":"2023-10-30T12:28:44.054652+0000","last_active":"2023-10-30T13:24:47.246911+0000","last_peered":"2023-10-30T13:24:47.246911+0000","last_clean":"2023-10-30T13:24:47.246911+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T13:24:47.246911+0000","last_undegraded":"2023-10-30T13:24:47.246911+0000","last_fullsized":"2023-10-30T13:24:47.246911+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":880,"log_dups_size":0,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:36:05.023374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":739,"num_read_kb":197443,"num_write":1077,"num_write_kb":51357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"82'1642","reported_seq":3637,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.166307+0000","last_change":"2023-10-30T12:37:07.102323+0000","last_active":"2023-10-30T13:24:56.166307+0000","last_peered":"2023-10-30T13:24:56.166307+0000","last_clean":"2023-10-30T13:24:56.166307+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T13:24:56.166307+0000","last_undegraded":"2023-10-30T13:24:56.166307+0000","last_fullsized":"2023-10-30T13:24:56.166307+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:26:12.249464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011628750000000001,"stat_sum":{"num_bytes":22561655,"num_objects":686,"num_object_clones":314,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":247,"num_read":162,"num_read_kb":29557,"num_write":1441,"num_write_kb":28808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"85'1048","reported_seq":1588,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:47.267547+0000","last_change":"2023-10-30T12:28:44.052633+0000","last_active":"2023-10-30T13:24:47.267547+0000","last_peered":"2023-10-30T13:24:47.267547+0000","last_clean":"2023-10-30T13:24:47.267547+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T13:24:47.267547+0000","last_undegraded":"2023-10-30T13:24:47.267547+0000","last_fullsized":"2023-10-30T13:24:47.267547+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1048,"log_dups_size":0,"ondisk_log_size":1048,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:22.574562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":939,"num_read_kb":433966,"num_write":1214,"num_write_kb":88022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"82'1531","reported_seq":3513,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.214302+0000","last_change":"2023-10-30T12:37:07.189494+0000","last_active":"2023-10-30T13:25:16.214302+0000","last_peered":"2023-10-30T13:25:16.214302+0000","last_clean":"2023-10-30T13:25:16.214302+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T13:25:16.214302+0000","last_undegraded":"2023-10-30T13:25:16.214302+0000","last_fullsized":"2023-10-30T13:25:16.214302+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1531,"log_dups_size":0,"ondisk_log_size":1531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:56:34.313998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014632805,"stat_sum":{"num_bytes":14611265,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":226,"num_read":123,"num_read_kb":17907,"num_write":1326,"num_write_kb":16683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"85'753","reported_seq":1130,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.373735+0000","last_change":"2023-10-30T12:28:44.075493+0000","last_active":"2023-10-30T13:24:56.373735+0000","last_peered":"2023-10-30T13:24:56.373735+0000","last_clean":"2023-10-30T13:24:56.373735+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T13:24:56.373735+0000","last_undegraded":"2023-10-30T13:24:56.373735+0000","last_fullsized":"2023-10-30T13:24:56.373735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":753,"log_dups_size":0,"ondisk_log_size":753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:00:11.718669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":488,"num_read_kb":379559,"num_write":896,"num_write_kb":83880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"77'1481","reported_seq":3497,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:56.373701+0000","last_change":"2023-10-30T12:37:07.191451+0000","last_active":"2023-10-30T13:24:56.373701+0000","last_peered":"2023-10-30T13:24:56.373701+0000","last_clean":"2023-10-30T13:24:56.373701+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T13:24:56.373701+0000","last_undegraded":"2023-10-30T13:24:56.373701+0000","last_fullsized":"2023-10-30T13:24:56.373701+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:28.248158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015871881000000001,"stat_sum":{"num_bytes":15563160,"num_objects":581,"num_object_clones":268,"num_object_copies":1162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":208,"num_read":133,"num_read_kb":9744,"num_write":1268,"num_write_kb":20904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"85'838","reported_seq":1197,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.657659+0000","last_change":"2023-10-30T12:28:44.065843+0000","last_active":"2023-10-30T13:25:02.657659+0000","last_peered":"2023-10-30T13:25:02.657659+0000","last_clean":"2023-10-30T13:25:02.657659+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T13:25:02.657659+0000","last_undegraded":"2023-10-30T13:25:02.657659+0000","last_fullsized":"2023-10-30T13:25:02.657659+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":838,"log_dups_size":0,"ondisk_log_size":838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33555014,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":517,"num_read_kb":175570,"num_write":978,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165214918,"num_objects":39301,"num_object_clones":18317,"num_object_copies":78602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39301,"num_whiteouts":14249,"num_read":9267,"num_read_kb":1285569,"num_write":86732,"num_write_kb":1459764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424127488,"data_stored":2332445068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99517,"ondisk_log_size":99517,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":903126807,"num_objects":1657,"num_object_clones":0,"num_object_copies":3314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":21550,"num_read_kb":9552846,"num_write":41306,"num_write_kb":2210416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1806409728,"data_stored":1806253614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34760,"ondisk_log_size":34760,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477426,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":837308,"kb_used_data":639604,"kb_used_omap":1,"kb_used_meta":197694,"kb_avail":93534532,"statfs":{"total":96636764160,"available":95779360768,"internally_reserved":0,"allocated":654954496,"data_stored":640799640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":202439112},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42699999999999999,"15min":0.434},"min":{"1min":0.253,"5min":0.253,"15min":0.23200000000000001},"max":{"1min":0.52300000000000002,"5min":0.66900000000000004,"15min":0.94599999999999995},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.47099999999999997,"15min":0.46999999999999997},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.27700000000000002},"max":{"1min":0.68300000000000005,"5min":0.75800000000000001,"15min":0.81899999999999995},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.503,"15min":0.495},"min":{"1min":0.39000000000000001,"5min":0.33300000000000002,"15min":0.307},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":1.0209999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.76600000000000001,"5min":0.78400000000000003,"15min":0.99299999999999999},"last":0.503}]},{"osd":2,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51500000000000001,"15min":0.502},"min":{"1min":0.373,"5min":0.34100000000000003,"15min":0.28599999999999998},"max":{"1min":0.747,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.42999999999999999,"5min":0.38600000000000001,"15min":0.33200000000000002},"max":{"1min":0.79700000000000004,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.47899999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77900000000000003,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.94299999999999995,"5min":1.05,"15min":1.548},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.63900000000000001,"5min":0.497,"15min":0.497},"max":{"1min":1.127,"5min":1.149,"15min":1.6950000000000001},"last":0.81799999999999995}]},{"osd":5,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.78900000000000003,"15min":0.81399999999999995},"min":{"1min":0.57299999999999995,"5min":0.52800000000000002,"15min":0.496},"max":{"1min":0.96399999999999997,"5min":1.1160000000000001,"15min":1.194},"last":0.872},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.80000000000000004,"15min":0.80700000000000005},"min":{"1min":0.66100000000000003,"5min":0.63200000000000001,"15min":0.52100000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.603},"last":0.69199999999999995}]},{"osd":6,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.64000000000000001,"5min":0.56999999999999995,"15min":0.436},"max":{"1min":1.113,"5min":1.198,"15min":1.2270000000000001},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.93100000000000005,"15min":0.92300000000000004},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.55600000000000005},"max":{"1min":1.145,"5min":1.202,"15min":1.3759999999999999},"last":0.79300000000000004}]},{"osd":7,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.84799999999999998,"15min":0.83599999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.46400000000000002},"max":{"1min":1.133,"5min":1.133,"15min":1.1919999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.91400000000000003,"15min":0.877},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.47699999999999998},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.2050000000000001},"last":0.76800000000000002}]}]},{"osd":2,"up_from":16,"seq":68719477427,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534076,"kb_used_data":412724,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":93837764,"statfs":{"total":96636764160,"available":96089870336,"internally_reserved":0,"allocated":422629376,"data_stored":409295067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":124254662},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42999999999999999,"15min":0.43099999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57799999999999996,"5min":0.81299999999999994,"15min":0.91200000000000003},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48799999999999999,"15min":0.47899999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.625,"5min":1.554,"15min":2.8519999999999999},"last":0.436}]},{"osd":1,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47799999999999998,"15min":0.497},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.57199999999999995,"5min":1.071,"15min":4.9660000000000002},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52700000000000002,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.32100000000000001,"15min":0.28399999999999997},"max":{"1min":0.66600000000000004,"5min":1.198,"15min":6.0759999999999996},"last":0.41299999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.505,"15min":0.52800000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.314},"max":{"1min":0.65500000000000003,"5min":0.96799999999999997,"15min":4.6120000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56499999999999995,"15min":0.58699999999999997},"min":{"1min":0.42699999999999999,"5min":0.40899999999999997,"15min":0.379},"max":{"1min":0.68200000000000005,"5min":1.4430000000000001,"15min":4.641},"last":0.58599999999999997}]},{"osd":4,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79100000000000004,"15min":0.80400000000000005},"min":{"1min":0.58399999999999996,"5min":0.48299999999999998,"15min":0.47999999999999998},"max":{"1min":0.95999999999999996,"5min":1.1719999999999999,"15min":3.5499999999999998},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77000000000000002,"15min":0.81000000000000005},"min":{"1min":0.53200000000000003,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0389999999999999,"5min":1.262,"15min":5.032},"last":0.84999999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.88800000000000001,"15min":0.87},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.177,"5min":1.494,"15min":6.1289999999999996},"last":0.89800000000000002},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.84699999999999998,"15min":0.877},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.159,"5min":1.3440000000000001,"15min":3.1459999999999999},"last":0.86899999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.83299999999999996,"15min":0.875},"min":{"1min":0.67600000000000005,"5min":0.54000000000000004,"15min":0.53100000000000003},"max":{"1min":1.0469999999999999,"5min":1.1739999999999999,"15min":5.8159999999999998},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.82799999999999996,"15min":0.83299999999999996},"min":{"1min":0.66600000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.99299999999999999,"5min":1.6299999999999999,"15min":3.415},"last":0.74099999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.84199999999999997,"15min":0.872},"min":{"1min":0.73999999999999999,"5min":0.60699999999999998,"15min":0.51100000000000001},"max":{"1min":1.121,"5min":1.9139999999999999,"15min":4.883},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.88400000000000001,"15min":0.90300000000000002},"min":{"1min":0.79300000000000004,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0640000000000001,"5min":1.3460000000000001,"15min":4.9950000000000001},"last":0.93899999999999995}]}]},{"osd":1,"up_from":16,"seq":68719477426,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":534824,"kb_used_data":431520,"kb_used_omap":1,"kb_used_meta":103294,"kb_avail":93837016,"statfs":{"total":96636764160,"available":96089104384,"internally_reserved":0,"allocated":441876480,"data_stored":431229668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":105773510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46899999999999997,"15min":0.52500000000000002},"min":{"1min":0.32800000000000001,"5min":0.27700000000000002,"15min":0.22600000000000001},"max":{"1min":0.66700000000000004,"5min":0.70699999999999996,"15min":20.196000000000002},"last":0.44},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55100000000000005,"15min":0.54800000000000004},"min":{"1min":0.39800000000000002,"5min":0.31,"15min":0.253},"max":{"1min":0.77200000000000002,"5min":0.91100000000000003,"15min":5.1070000000000002},"last":0.46100000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.48599999999999999,"15min":0.67200000000000004},"min":{"1min":0.34599999999999997,"5min":0.32200000000000001,"15min":0.312},"max":{"1min":0.73099999999999998,"5min":0.83999999999999997,"15min":45.866999999999997},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.624,"5min":0.57099999999999995,"15min":0.64100000000000001},"min":{"1min":0.433,"5min":0.374,"15min":0.34699999999999998},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":20.789999999999999},"last":0.498}]},{"osd":3,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.47899999999999998,"15min":0.57899999999999996},"min":{"1min":0.36899999999999999,"5min":0.318,"15min":0.30499999999999999},"max":{"1min":0.624,"5min":0.80900000000000005,"15min":21.178999999999998},"last":0.37},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.56100000000000005,"15min":0.55700000000000005},"min":{"1min":0.45100000000000001,"5min":0.39600000000000002,"15min":0.34999999999999998},"max":{"1min":0.71099999999999997,"5min":0.91800000000000004,"15min":2.2290000000000001},"last":0.48399999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.40400000000000003},"max":{"1min":1.133,"5min":1.133,"15min":5.9770000000000003},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.90400000000000003,"15min":0.86899999999999999},"min":{"1min":0.68400000000000005,"5min":0.64300000000000002,"15min":0.44400000000000001},"max":{"1min":1.093,"5min":1.1619999999999999,"15min":5.5110000000000001},"last":0.82599999999999996}]},{"osd":5,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.82399999999999995,"15min":0.872},"min":{"1min":0.58299999999999996,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0149999999999999,"5min":1.212,"15min":7.5430000000000001},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.90100000000000002,"15min":0.94499999999999995},"min":{"1min":0.76700000000000002,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.0680000000000001,"5min":1.2350000000000001,"15min":7.5750000000000002},"last":0.879}]},{"osd":6,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.79900000000000004,"15min":0.88300000000000001},"min":{"1min":0.59699999999999998,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0820000000000001,"5min":1.177,"15min":20.827000000000002},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.82099999999999995,"15min":0.95399999999999996},"min":{"1min":0.68400000000000005,"5min":0.51800000000000002,"15min":0.51600000000000001},"max":{"1min":1.0469999999999999,"5min":1.159,"15min":20.931000000000001},"last":0.67600000000000005}]},{"osd":7,"last update":"Mon Oct 30 13:24:45 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.83999999999999997,"15min":0.95799999999999996},"min":{"1min":0.73299999999999998,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.2050000000000001,"5min":1.2050000000000001,"15min":20.859999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.90600000000000003,"15min":0.88500000000000001},"min":{"1min":0.56699999999999995,"5min":0.55800000000000005,"15min":0.501},"max":{"1min":1.1870000000000001,"5min":1.194,"15min":4.367},"last":0.89200000000000002}]}]},{"osd":0,"up_from":16,"seq":68719477427,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":937928,"kb_used_data":695808,"kb_used_omap":1,"kb_used_meta":242110,"kb_avail":93433912,"statfs":{"total":96636764160,"available":95676325888,"internally_reserved":0,"allocated":712507392,"data_stored":696810667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":247921094},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40600000000000003,"15min":0.41099999999999998},"min":{"1min":0.254,"5min":0.23899999999999999,"15min":0.22700000000000001},"max":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":3.996},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51700000000000002,"15min":0.52600000000000002},"min":{"1min":0.41099999999999998,"5min":0.36199999999999999,"15min":0.309},"max":{"1min":0.67600000000000005,"5min":0.75600000000000001,"15min":4.3230000000000004},"last":0.57199999999999995}]},{"osd":2,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.27700000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":4.9690000000000003},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52100000000000002,"15min":0.53200000000000003},"min":{"1min":0.372,"5min":0.34100000000000003,"15min":0.30199999999999999},"max":{"1min":0.621,"5min":0.878,"15min":5.0049999999999999},"last":0.48499999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.48799999999999999},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.28299999999999997},"max":{"1min":0.68100000000000005,"5min":0.82699999999999996,"15min":5.4909999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57499999999999996,"15min":0.58099999999999996},"min":{"1min":0.443,"5min":0.42799999999999999,"15min":0.38600000000000001},"max":{"1min":0.749,"5min":0.90400000000000003,"15min":5.6970000000000001},"last":0.54300000000000004}]},{"osd":4,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.58699999999999997,"5min":0.46700000000000003,"15min":0.377},"max":{"1min":1.111,"5min":1.111,"15min":4.444},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.80100000000000005},"min":{"1min":0.58499999999999996,"5min":0.55000000000000004,"15min":0.48099999999999998},"max":{"1min":0.90500000000000003,"5min":1.117,"15min":5.7750000000000004},"last":0.82199999999999995}]},{"osd":5,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.83299999999999996,"15min":0.83499999999999996},"min":{"1min":0.66800000000000004,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.246,"5min":1.3340000000000001,"15min":4.5430000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.871,"15min":0.86399999999999999},"min":{"1min":0.59199999999999997,"5min":0.56999999999999995,"15min":0.47099999999999997},"max":{"1min":1.204,"5min":1.204,"15min":4.4790000000000001},"last":0.88200000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.82599999999999996,"15min":0.84199999999999997},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":1.397,"15min":4.5129999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.88500000000000001,"15min":0.89900000000000002},"min":{"1min":0.626,"5min":0.626,"15min":0.56699999999999995},"max":{"1min":1.1699999999999999,"5min":1.3120000000000001,"15min":4.5750000000000002},"last":0.70599999999999996}]},{"osd":7,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.875,"15min":0.86799999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.55700000000000005},"max":{"1min":1.0669999999999999,"5min":1.206,"15min":4.3979999999999997},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.88900000000000001,"15min":0.91400000000000003},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.53700000000000003},"max":{"1min":1.0760000000000001,"5min":1.2210000000000001,"15min":5.375},"last":0.93600000000000005}]}]},{"osd":6,"up_from":15,"seq":64424510129,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":526444,"kb_used_data":409252,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":93845396,"statfs":{"total":96636764160,"available":96097685504,"internally_reserved":0,"allocated":419074048,"data_stored":408471323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":119994823},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":0.874,"15min":0.86299999999999999},"min":{"1min":0.61099999999999999,"5min":0.49399999999999999,"15min":0.34200000000000003},"max":{"1min":1.167,"5min":1.4470000000000001,"15min":3.1150000000000002},"last":1.103},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.89900000000000002,"15min":0.96999999999999997},"min":{"1min":0.63500000000000001,"5min":0.54500000000000004,"15min":0.51700000000000002},"max":{"1min":1.121,"5min":1.415,"15min":3.3639999999999999},"last":0.89500000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.79800000000000004,"15min":0.79700000000000004},"min":{"1min":0.48799999999999999,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.087,"5min":1.202,"15min":1.589},"last":1.262},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85899999999999999,"15min":0.86199999999999999},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.47699999999999998},"max":{"1min":1.145,"5min":1.2330000000000001,"15min":1.544},"last":1.0640000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.98199999999999998,"15min":0.96299999999999997},"min":{"1min":0.68300000000000005,"5min":0.57299999999999995,"15min":0.47199999999999998},"max":{"1min":1.2969999999999999,"5min":1.496,"15min":3.4969999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.91600000000000004,"15min":0.94999999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.2090000000000001,"5min":1.391,"15min":1.512},"last":1.1839999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:24:46 2023","interfaces":[{"interface":"back","average":{"1min":0.875,"5min":0.86299999999999999,"15min":0.89100000000000001},"min":{"1min":0.58799999999999997,"5min":0.57099999999999995,"15min":0.52800000000000002},"max":{"1min":1.131,"5min":1.262,"15min":3.6070000000000002},"last":1.1200000000000001},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.91300000000000003,"15min":0.88300000000000001},"min":{"1min":0.69499999999999995,"5min":0.54000000000000004,"15min":0.51600000000000001},"max":{"1min":1.1839999999999999,"5min":1.272,"15min":3.7679999999999998},"last":0.98899999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97299999999999998,"15min":0.93799999999999994},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.2849999999999999,"5min":1.3220000000000001,"15min":1.4019999999999999},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.093,"15min":1.046},"min":{"1min":0.80000000000000004,"5min":0.73899999999999999,"15min":0.54700000000000004},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.2330000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.163,"15min":1.125},"min":{"1min":0.90100000000000002,"5min":0.82999999999999996,"15min":0.73399999999999999},"max":{"1min":1.417,"5min":1.4730000000000001,"15min":1.583},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.069,"15min":1.095},"min":{"1min":0.625,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2829999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.3939999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.081,"15min":1.0800000000000001},"min":{"1min":0.86099999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.4970000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.472},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.22,"15min":1.2170000000000001},"min":{"1min":1.034,"5min":0.76500000000000001,"15min":0.64200000000000002},"max":{"1min":1.804,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.611}]}]},{"osd":4,"up_from":15,"seq":64424510132,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":645792,"kb_used_data":493912,"kb_used_omap":1,"kb_used_meta":151870,"kb_avail":93726048,"statfs":{"total":96636764160,"available":95975473152,"internally_reserved":0,"allocated":505765888,"data_stored":496228056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":155515335},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.86299999999999999,"15min":0.83899999999999997},"min":{"1min":0.54800000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.1930000000000001,"5min":1.218,"15min":1.292},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.86899999999999999,"15min":0.84699999999999998},"min":{"1min":0.57399999999999995,"5min":0.52700000000000002,"15min":0.52600000000000002},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.369},"last":0.67800000000000005}]},{"osd":1,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.79400000000000004,"15min":0.78600000000000003},"min":{"1min":0.60799999999999998,"5min":0.47899999999999998,"15min":0.47499999999999998},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1839999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.879,"5min":0.83699999999999997,"15min":0.83199999999999996},"min":{"1min":0.60899999999999999,"5min":0.53000000000000003,"15min":0.47299999999999998},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.3100000000000001},"last":0.55400000000000005}]},{"osd":2,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.89500000000000002},"min":{"1min":0.56599999999999995,"5min":0.55500000000000005,"15min":0.55300000000000005},"max":{"1min":1.1899999999999999,"5min":1.2270000000000001,"15min":1.292},"last":0.626},{"interface":"front","average":{"1min":0.998,"5min":0.94599999999999995,"15min":0.94699999999999995},"min":{"1min":0.74099999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":1.0569999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.97399999999999998,"15min":0.95299999999999996},"min":{"1min":0.68500000000000005,"5min":0.65100000000000002,"15min":0.52200000000000002},"max":{"1min":1.286,"5min":1.286,"15min":1.514},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.94699999999999995,"15min":0.96299999999999997},"min":{"1min":0.76600000000000001,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.86299999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.98299999999999998,"15min":1.0129999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60299999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":2.4430000000000001},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.171,"5min":1.1140000000000001,"15min":1.0660000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.8460000000000001},"last":1.2230000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.145,"15min":1.1559999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.419,"5min":1.425,"15min":1.677},"last":1.179},{"interface":"front","average":{"1min":1.306,"5min":1.28,"15min":1.1830000000000001},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.64400000000000002},"max":{"1min":1.528,"5min":1.5409999999999999,"15min":1.9610000000000001},"last":1.272}]},{"osd":7,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.0529999999999999,"15min":1.1359999999999999},"min":{"1min":0.89500000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.8700000000000001},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.202,"15min":1.266},"min":{"1min":1.016,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.538,"5min":1.591,"15min":1.9450000000000001},"last":1.0289999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542833,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":560428,"kb_used_data":416932,"kb_used_omap":1,"kb_used_meta":143486,"kb_avail":93811412,"statfs":{"total":96636764160,"available":96062885888,"internally_reserved":0,"allocated":426938368,"data_stored":418861204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":146930123},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.84399999999999997,"15min":0.82999999999999996},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.41499999999999998},"max":{"1min":1.2130000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":0.623},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.88,"15min":0.872},"min":{"1min":0.56999999999999995,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.333,"5min":1.3380000000000001,"15min":1.3919999999999999},"last":1.0589999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.76900000000000002,"15min":0.86799999999999999},"min":{"1min":0.504,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.095,"5min":1.244,"15min":1.4790000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.91700000000000004,"15min":0.82299999999999995},"min":{"1min":0.51500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.5,"5min":1.524,"15min":1.524},"last":0.96899999999999997}]},{"osd":2,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0589999999999999,"15min":1.024},"min":{"1min":0.66500000000000004,"5min":0.54700000000000004,"15min":0.52700000000000002},"max":{"1min":1.3109999999999999,"5min":1.833,"15min":1.833},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1830000000000001,"15min":1.153},"min":{"1min":0.91100000000000003,"5min":0.71499999999999997,"15min":0.69399999999999995},"max":{"1min":1.4510000000000001,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.214}]},{"osd":3,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0760000000000001,"15min":1.083},"min":{"1min":0.66400000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.2929999999999999,"5min":1.673,"15min":1.673},"last":0.89500000000000002},{"interface":"front","average":{"1min":1.204,"5min":1.244,"15min":1.2230000000000001},"min":{"1min":0.84299999999999997,"5min":0.83499999999999996,"15min":0.81399999999999995},"max":{"1min":1.4490000000000001,"5min":1.8109999999999999,"15min":1.8480000000000001},"last":1.135}]},{"osd":4,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.86199999999999999,"15min":0.84799999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.171,"5min":1.226,"15min":1.4970000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.95799999999999996,"15min":0.95599999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.242,"5min":1.4970000000000001,"15min":1.77},"last":0.85199999999999998}]},{"osd":5,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.94299999999999995,"15min":0.97499999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.2490000000000001,"5min":1.3799999999999999,"15min":2},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.105,"15min":1.0349999999999999},"min":{"1min":0.69599999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.508,"5min":1.7150000000000001,"15min":2.056},"last":1.0880000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.042,"15min":1.008},"min":{"1min":0.68100000000000005,"5min":0.60799999999999998,"15min":0.501},"max":{"1min":1.377,"5min":1.536,"15min":1.571},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1299999999999999,"15min":1.105},"min":{"1min":0.751,"5min":0.72599999999999998,"15min":0.71199999999999997},"max":{"1min":1.5549999999999999,"5min":1.665,"15min":1.665},"last":1.327}]}]},{"osd":5,"up_from":14,"seq":60129542832,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":802272,"kb_used_data":640600,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93569568,"statfs":{"total":96636764160,"available":95815237632,"internally_reserved":0,"allocated":655974400,"data_stored":642425281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":165542342},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.378},"max":{"1min":1.173,"5min":1.276,"15min":1.579},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.85299999999999998,"15min":0.86099999999999999},"min":{"1min":0.499,"5min":0.46200000000000002,"15min":0.34599999999999997},"max":{"1min":1.262,"5min":1.627,"15min":1.627},"last":0.65800000000000003}]},{"osd":1,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.81200000000000006,"15min":0.83099999999999996},"min":{"1min":0.55300000000000005,"5min":0.47499999999999998,"15min":0.39500000000000002},"max":{"1min":1.22,"5min":1.226,"15min":1.3220000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.88600000000000001,"15min":0.91900000000000004},"min":{"1min":0.64800000000000002,"5min":0.52600000000000002,"15min":0.376},"max":{"1min":1.1539999999999999,"5min":1.3100000000000001,"15min":1.6619999999999999},"last":0.89800000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.85799999999999998,"15min":0.84799999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.2330000000000001,"5min":1.3220000000000001,"15min":1.486},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.89300000000000002,"15min":0.88},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2749999999999999,"5min":1.3149999999999999,"15min":1.879},"last":0.70799999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.96899999999999997,"15min":0.88900000000000001},"min":{"1min":0.65500000000000003,"5min":0.55900000000000005,"15min":0.42799999999999999},"max":{"1min":1.361,"5min":1.361,"15min":2.7330000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.097,"5min":1.0509999999999999,"15min":0.999},"min":{"1min":0.68899999999999995,"5min":0.51300000000000001,"15min":0.47099999999999997},"max":{"1min":1.359,"5min":1.5169999999999999,"15min":2.79},"last":1.022}]},{"osd":4,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.97299999999999998,"15min":0.98099999999999998},"min":{"1min":0.68500000000000005,"5min":0.57599999999999996,"15min":0.45300000000000001},"max":{"1min":1.119,"5min":1.4710000000000001,"15min":1.6719999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.034,"15min":1.0940000000000001},"min":{"1min":0.87,"5min":0.626,"15min":0.58799999999999997},"max":{"1min":1.3120000000000001,"5min":1.3540000000000001,"15min":1.7629999999999999},"last":1.0660000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.1000000000000001,"15min":1.0920000000000001},"min":{"1min":0.69999999999999996,"5min":0.66500000000000004,"15min":0.54600000000000004},"max":{"1min":1.3959999999999999,"5min":1.675,"15min":2.8450000000000002},"last":1.175},{"interface":"front","average":{"1min":1.149,"5min":1.123,"15min":1.111},"min":{"1min":0.73699999999999999,"5min":0.59999999999999998,"15min":0.57999999999999996},"max":{"1min":1.577,"5min":1.625,"15min":1.821},"last":1.1990000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.153,"15min":1.127},"min":{"1min":0.70099999999999996,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7490000000000001,"5min":1.7749999999999999,"15min":3.3069999999999999},"last":1.119},{"interface":"front","average":{"1min":1.115,"5min":1.1579999999999999,"15min":1.113},"min":{"1min":0.88500000000000001,"5min":0.59999999999999998,"15min":0.53200000000000003},"max":{"1min":1.3129999999999999,"5min":1.754,"15min":2.036},"last":1.2709999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309846016,"data_stored":309823069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142618624,"data_stored":142610667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117452800,"data_stored":117441250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294043648,"data_stored":294028670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":275668992,"data_stored":275628631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":300916736,"data_stored":300897937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":154218496,"data_stored":154207037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":211644416,"data_stored":211616353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401661952,"data_stored":386457412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297664512,"data_stored":287498447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304177152,"data_stored":291323631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359911424,"data_stored":346240784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":229097472,"data_stored":220069239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":353464320,"data_stored":340406790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":263856128,"data_stored":253734100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":214294528,"data_stored":206714665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:25:42.913 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.913 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.914 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.914 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.914 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.915 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.915 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.915 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.916 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.916 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.917 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.917 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.918 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.918 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.919 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.919 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.919 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.920 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.920 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.920 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.921 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.921 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.922 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.922 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.922 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.923 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.923 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.924 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.924 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.924 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.925 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.925 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.925 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.926 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.926 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.927 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.927 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.927 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.928 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.928 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.928 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.929 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.929 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.929 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.930 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.930 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.931 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.931 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.931 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.932 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.932 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.933 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.933 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.933 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.934 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.934 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.934 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.935 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.935 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.936 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.936 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.936 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.937 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.937 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.938 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-30T12:28:31.408255+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=31, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.938 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.938 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.939 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.939 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.940 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.940 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.940 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.941 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.941 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.942 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.942 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.942 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.943 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.943 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.943 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.944 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.944 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.944 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.945 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.945 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.946 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.946 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.946 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.947 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.947 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.948 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.949 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.949 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.949 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.950 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.950 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.951 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:25:42.951 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T13:25:48.756 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:25:48.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:26:02.952 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:26:03.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.101+0000 7fbe5d646640 1 Processor -- start 2023-10-30T13:26:03.107 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.101+0000 7fbe5d646640 1 -- start start 2023-10-30T13:26:03.108 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.105+0000 7fbe5d646640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbe580c90d0 0x7fbe580cef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:03.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.105+0000 7fbe5d646640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbe580cf4d0 0x7fbe5815d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:03.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.105+0000 7fbe5d646640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5815ce30 con 0x7fbe580c8cf0 2023-10-30T13:26:03.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.105+0000 7fbe5d646640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe58159880 con 0x7fbe580cf4d0 2023-10-30T13:26:03.109 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.105+0000 7fbe5d646640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe58159ae0 con 0x7fbe580c90d0 2023-10-30T13:26:03.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.105+0000 7fbe577fe640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fbe580cf4d0 0x7fbe5815d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:03.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.105+0000 7fbe56ffd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fbe580c8cf0 0x7fbe580c9e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:51228/0 (socket says 172.21.15.125:51228) 2023-10-30T13:26:03.110 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.105+0000 7fbe56ffd640 1 -- 172.21.15.125:0/75951530 learned_addr learned my addr 172.21.15.125:0/75951530 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:26:03.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.105+0000 7fbe567fc640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fbe580c90d0 0x7fbe580cef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:03.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.105+0000 7fbe577fe640 1 -- 172.21.15.125:0/75951530 >> v2:172.21.15.136:3300/0 conn(0x7fbe580c90d0 msgr2=0x7fbe580cef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:03.111 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.105+0000 7fbe577fe640 1 --2- 172.21.15.125:0/75951530 >> v2:172.21.15.136:3300/0 conn(0x7fbe580c90d0 0x7fbe580cef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:03.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.105+0000 7fbe577fe640 1 -- 172.21.15.125:0/75951530 >> v1:172.21.15.125:6789/0 conn(0x7fbe580c8cf0 legacy=0x7fbe580c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:03.112 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.105+0000 7fbe577fe640 1 -- 172.21.15.125:0/75951530 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe58159dc0 con 0x7fbe580cf4d0 2023-10-30T13:26:03.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.109+0000 7fbe577fe640 1 --2- 172.21.15.125:0/75951530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe580cf4d0 0x7fbe5815d2c0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fbe5806cb60 tx=0x7fbe5001c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71dce95df5484c36 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:03.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.109+0000 7fbe55ffb640 1 -- 172.21.15.125:0/75951530 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe500a59a0 con 0x7fbe580cf4d0 2023-10-30T13:26:03.113 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.109+0000 7fbe55ffb640 1 -- 172.21.15.125:0/75951530 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe500a5b00 con 0x7fbe580cf4d0 2023-10-30T13:26:03.114 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.109+0000 7fbe55ffb640 1 -- 172.21.15.125:0/75951530 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe500a5e90 con 0x7fbe580cf4d0 2023-10-30T13:26:03.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.109+0000 7fbe5d646640 1 -- 172.21.15.125:0/75951530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe580cf4d0 msgr2=0x7fbe5815d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:03.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.109+0000 7fbe5d646640 1 --2- 172.21.15.125:0/75951530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe580cf4d0 0x7fbe5815d2c0 secure :-1 s=READY pgs=2205 cs=0 l=1 rev1=1 crypto rx=0x7fbe5806cb60 tx=0x7fbe5001c8e0 comp rx=0 tx=0).stop 2023-10-30T13:26:03.115 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.109+0000 7fbe5d646640 1 -- 172.21.15.125:0/75951530 shutdown_connections 2023-10-30T13:26:03.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.109+0000 7fbe5d646640 1 --2- 172.21.15.125:0/75951530 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe580cf4d0 0x7fbe5815d2c0 unknown :-1 s=CLOSED pgs=2205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:03.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.109+0000 7fbe5d646640 1 --2- 172.21.15.125:0/75951530 >> v2:172.21.15.136:3300/0 conn(0x7fbe580c90d0 0x7fbe580cef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:03.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.109+0000 7fbe5d646640 1 -- 172.21.15.125:0/75951530 >> 172.21.15.125:0/75951530 conn(0x7fbe5806be40 msgr2=0x7fbe5806c250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:03.116 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.109+0000 7fbe5d646640 1 -- 172.21.15.125:0/75951530 shutdown_connections 2023-10-30T13:26:03.117 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.109+0000 7fbe5d646640 1 -- 172.21.15.125:0/75951530 wait complete. 2023-10-30T13:26:03.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe5d646640 1 Processor -- start 2023-10-30T13:26:03.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe5d646640 1 -- start start 2023-10-30T13:26:03.119 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe5d646640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe580c8cf0 0x7fbe581513c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:03.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe5d646640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe580c90d0 0x7fbe581568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:03.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe5d646640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe580cf4d0 0x7fbe58158850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:03.120 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe5d646640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe5814ff90 con 0x7fbe580cf4d0 2023-10-30T13:26:03.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe5d646640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe581f5580 con 0x7fbe580c8cf0 2023-10-30T13:26:03.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe5d646640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe581f6100 con 0x7fbe580c90d0 2023-10-30T13:26:03.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe577fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe580cf4d0 0x7fbe58158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:03.121 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe56ffd640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe580c8cf0 0x7fbe581513c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:03.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe577fe640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe580cf4d0 0x7fbe58158850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:41104/0 (socket says 172.21.15.125:41104) 2023-10-30T13:26:03.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe577fe640 1 -- 172.21.15.125:0/3476183477 learned_addr learned my addr 172.21.15.125:0/3476183477 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:26:03.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe567fc640 1 --2- 172.21.15.125:0/3476183477 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe580c90d0 0x7fbe581568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:03.122 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe577fe640 1 -- 172.21.15.125:0/3476183477 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe580c90d0 msgr2=0x7fbe581568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:03.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe577fe640 1 --2- 172.21.15.125:0/3476183477 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe580c90d0 0x7fbe581568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:03.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe577fe640 1 -- 172.21.15.125:0/3476183477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe580c8cf0 msgr2=0x7fbe581513c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:03.123 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe577fe640 1 --2- 172.21.15.125:0/3476183477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe580c8cf0 0x7fbe581513c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:03.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe577fe640 1 -- 172.21.15.125:0/3476183477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe50009a70 con 0x7fbe580cf4d0 2023-10-30T13:26:03.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe567fc640 1 --2- 172.21.15.125:0/3476183477 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe580c90d0 0x7fbe581568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:03.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe577fe640 1 --2- 172.21.15.125:0/3476183477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe580cf4d0 0x7fbe58158850 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fbe500b0040 tx=0x7fbe50019680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:03.124 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe3f7fe640 1 -- 172.21.15.125:0/3476183477 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe500ade20 con 0x7fbe580cf4d0 2023-10-30T13:26:03.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe3f7fe640 1 -- 172.21.15.125:0/3476183477 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe50009450 con 0x7fbe580cf4d0 2023-10-30T13:26:03.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe3f7fe640 1 -- 172.21.15.125:0/3476183477 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe500b9690 con 0x7fbe580cf4d0 2023-10-30T13:26:03.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe5d646640 1 -- 172.21.15.125:0/3476183477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe58151db0 con 0x7fbe580cf4d0 2023-10-30T13:26:03.125 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.113+0000 7fbe5d646640 1 -- 172.21.15.125:0/3476183477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe581f5ab0 con 0x7fbe580cf4d0 2023-10-30T13:26:03.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.117+0000 7fbe5d646640 1 -- 172.21.15.125:0/3476183477 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe5804e170 con 0x7fbe580cf4d0 2023-10-30T13:26:03.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.117+0000 7fbe3f7fe640 1 -- 172.21.15.125:0/3476183477 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fbe500b97f0 con 0x7fbe580cf4d0 2023-10-30T13:26:03.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.117+0000 7fbe3f7fe640 1 --2- 172.21.15.125:0/3476183477 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbe440664d0 0x7fbe44068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:03.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.117+0000 7fbe3f7fe640 1 -- 172.21.15.125:0/3476183477 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fbe501202c0 con 0x7fbe580cf4d0 2023-10-30T13:26:03.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.117+0000 7fbe56ffd640 1 --2- 172.21.15.125:0/3476183477 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbe440664d0 0x7fbe44068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:03.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.117+0000 7fbe56ffd640 1 --2- 172.21.15.125:0/3476183477 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbe440664d0 0x7fbe44068990 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fbe48005890 tx=0x7fbe48009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:03.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.121+0000 7fbe3f7fe640 1 -- 172.21.15.125:0/3476183477 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fbe500ecd20 con 0x7fbe580cf4d0 2023-10-30T13:26:03.297 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.293+0000 7fbe5d646640 1 -- 172.21.15.125:0/3476183477 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbe5815dc10 con 0x7fbe440664d0 2023-10-30T13:26:03.310 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.305+0000 7fbe3f7fe640 1 -- 172.21.15.125:0/3476183477 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+293413 (secure 0 0 0) 0x7fbe48001470 con 0x7fbe440664d0 2023-10-30T13:26:03.314 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:26:03.315 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T13:26:03.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.309+0000 7fbe5d646640 1 -- 172.21.15.125:0/3476183477 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbe440664d0 msgr2=0x7fbe44068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:03.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.309+0000 7fbe5d646640 1 --2- 172.21.15.125:0/3476183477 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbe440664d0 0x7fbe44068990 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fbe48005890 tx=0x7fbe48009040 comp rx=0 tx=0).stop 2023-10-30T13:26:03.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.309+0000 7fbe5d646640 1 -- 172.21.15.125:0/3476183477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe580cf4d0 msgr2=0x7fbe58158850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:03.317 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.309+0000 7fbe5d646640 1 --2- 172.21.15.125:0/3476183477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe580cf4d0 0x7fbe58158850 secure :-1 s=READY pgs=2206 cs=0 l=1 rev1=1 crypto rx=0x7fbe500b0040 tx=0x7fbe50019680 comp rx=0 tx=0).stop 2023-10-30T13:26:03.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.309+0000 7fbe5d646640 1 -- 172.21.15.125:0/3476183477 shutdown_connections 2023-10-30T13:26:03.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.309+0000 7fbe5d646640 1 --2- 172.21.15.125:0/3476183477 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fbe440664d0 0x7fbe44068990 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:03.318 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.309+0000 7fbe5d646640 1 --2- 172.21.15.125:0/3476183477 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fbe580cf4d0 0x7fbe58158850 unknown :-1 s=CLOSED pgs=2206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:03.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.309+0000 7fbe5d646640 1 --2- 172.21.15.125:0/3476183477 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fbe580c90d0 0x7fbe581568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:03.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.309+0000 7fbe5d646640 1 --2- 172.21.15.125:0/3476183477 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fbe580c8cf0 0x7fbe581513c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:03.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.309+0000 7fbe5d646640 1 -- 172.21.15.125:0/3476183477 >> 172.21.15.125:0/3476183477 conn(0x7fbe5806be40 msgr2=0x7fbe580c9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:03.319 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.309+0000 7fbe5d646640 1 -- 172.21.15.125:0/3476183477 shutdown_connections 2023-10-30T13:26:03.320 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:03.309+0000 7fbe5d646640 1 -- 172.21.15.125:0/3476183477 wait complete. 2023-10-30T13:26:03.343 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":1803,"stamp":"2023-10-30T13:26:02.278775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2068932093,"num_objects":40960,"num_object_clones":18317,"num_object_copies":81920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40960,"num_whiteouts":14249,"num_read":30933,"num_read_kb":10838515,"num_write":128197,"num_write_kb":3672596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15733835,"num_omap_keys":26734,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":152657,"ondisk_log_size":152657,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5456768,"kb_used_data":4140352,"kb_used_omap":12,"kb_used_meta":1316339,"kb_avail":749517952,"statfs":{"total":773094113280,"available":767506382848,"internally_reserved":0,"allocated":4239720448,"data_stored":4144120906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1347931705},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,1,4,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":9039828,"num_omap_keys":15473,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003625"},"pg_stats":[{"pgid":"3.3f","version":"77'1475","reported_seq":3524,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.357125+0000","last_change":"2023-10-30T12:37:07.182969+0000","last_active":"2023-10-30T13:25:32.357125+0000","last_peered":"2023-10-30T13:25:32.357125+0000","last_clean":"2023-10-30T13:25:32.357125+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T13:25:32.357125+0000","last_undegraded":"2023-10-30T13:25:32.357125+0000","last_fullsized":"2023-10-30T13:25:32.357125+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1475,"log_dups_size":0,"ondisk_log_size":1475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086120880000000004,"stat_sum":{"num_bytes":27261067,"num_objects":552,"num_object_clones":249,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":197,"num_read":145,"num_read_kb":29501,"num_write":1298,"num_write_kb":29008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"77'1476","reported_seq":3489,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.322617+0000","last_change":"2023-10-30T12:37:07.190309+0000","last_active":"2023-10-30T13:25:32.322617+0000","last_peered":"2023-10-30T13:25:32.322617+0000","last_clean":"2023-10-30T13:25:32.322617+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T13:25:32.322617+0000","last_undegraded":"2023-10-30T13:25:32.322617+0000","last_fullsized":"2023-10-30T13:25:32.322617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015306986,"stat_sum":{"num_bytes":13736175,"num_objects":574,"num_object_clones":269,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":223,"num_read":131,"num_read_kb":21315,"num_write":1260,"num_write_kb":15409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"77'1563","reported_seq":3575,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:01.374847+0000","last_change":"2023-10-30T12:37:07.187803+0000","last_active":"2023-10-30T13:25:01.374847+0000","last_peered":"2023-10-30T13:25:01.374847+0000","last_clean":"2023-10-30T13:25:01.374847+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T13:25:01.374847+0000","last_undegraded":"2023-10-30T13:25:01.374847+0000","last_fullsized":"2023-10-30T13:25:01.374847+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1563,"log_dups_size":0,"ondisk_log_size":1563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013634598,"stat_sum":{"num_bytes":28390436,"num_objects":628,"num_object_clones":291,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":232,"num_read":152,"num_read_kb":47076,"num_write":1356,"num_write_kb":37143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"85'2206","reported_seq":4249,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.404657+0000","last_change":"2023-10-30T13:25:48.404708+0000","last_active":"2023-10-30T13:25:48.404657+0000","last_peered":"2023-10-30T13:25:48.404657+0000","last_clean":"2023-10-30T13:25:48.404657+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T13:25:48.404657+0000","last_undegraded":"2023-10-30T13:25:48.404657+0000","last_fullsized":"2023-10-30T13:25:48.404657+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2206","last_scrub_stamp":"2023-10-30T13:25:48.404576+0000","last_deep_scrub":"85'2206","last_deep_scrub_stamp":"2023-10-30T13:25:48.404576+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.404576+0000","objects_scrubbed":629,"log_size":2206,"log_dups_size":0,"ondisk_log_size":2206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:01.872332+0000","scrub_duration":1.22769368,"objects_trimmed":0,"snaptrim_duration":0.0056762749999999997,"stat_sum":{"num_bytes":22921274,"num_objects":629,"num_object_clones":290,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":224,"num_read":145,"num_read_kb":39229,"num_write":1388,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"77'1591","reported_seq":3597,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.231312+0000","last_change":"2023-10-30T12:37:07.181803+0000","last_active":"2023-10-30T13:25:32.231312+0000","last_peered":"2023-10-30T13:25:32.231312+0000","last_clean":"2023-10-30T13:25:32.231312+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T13:25:32.231312+0000","last_undegraded":"2023-10-30T13:25:32.231312+0000","last_fullsized":"2023-10-30T13:25:32.231312+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1591,"log_dups_size":0,"ondisk_log_size":1591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081298299999999993,"stat_sum":{"num_bytes":26798060,"num_objects":623,"num_object_clones":286,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":214,"num_read":144,"num_read_kb":26713,"num_write":1434,"num_write_kb":32534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"85'2029","reported_seq":4090,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.107705+0000","last_change":"2023-10-30T13:25:44.107819+0000","last_active":"2023-10-30T13:25:44.107705+0000","last_peered":"2023-10-30T13:25:44.107705+0000","last_clean":"2023-10-30T13:25:44.107705+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T13:25:44.107705+0000","last_undegraded":"2023-10-30T13:25:44.107705+0000","last_fullsized":"2023-10-30T13:25:44.107705+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2029","last_scrub_stamp":"2023-10-30T13:25:44.107605+0000","last_deep_scrub":"85'2029","last_deep_scrub_stamp":"2023-10-30T13:25:44.107605+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.107605+0000","objects_scrubbed":563,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:35.807432+0000","scrub_duration":0.97669900899999995,"objects_trimmed":0,"snaptrim_duration":0.0048510920000000004,"stat_sum":{"num_bytes":13539463,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":205,"num_read":144,"num_read_kb":15818,"num_write":1313,"num_write_kb":14957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"85'2047","reported_seq":4039,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-30T13:25:58.103269+0000","last_change":"2023-10-30T13:25:57.121453+0000","last_active":"2023-10-30T13:25:58.103269+0000","last_peered":"2023-10-30T13:25:58.103269+0000","last_clean":"2023-10-30T13:25:58.103269+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T13:25:58.103269+0000","last_undegraded":"2023-10-30T13:25:58.103269+0000","last_fullsized":"2023-10-30T13:25:58.103269+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":485,"log_size":2047,"log_dups_size":0,"ondisk_log_size":2047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082671399999999999,"stat_sum":{"num_bytes":7202428,"num_objects":606,"num_object_clones":284,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":219,"num_read":126,"num_read_kb":1677,"num_write":1361,"num_write_kb":10203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"82'1518","reported_seq":3544,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.336206+0000","last_change":"2023-10-30T12:37:07.188855+0000","last_active":"2023-10-30T13:25:32.336206+0000","last_peered":"2023-10-30T13:25:32.336206+0000","last_clean":"2023-10-30T13:25:32.336206+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T13:25:32.336206+0000","last_undegraded":"2023-10-30T13:25:32.336206+0000","last_fullsized":"2023-10-30T13:25:32.336206+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1518,"log_dups_size":0,"ondisk_log_size":1518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014437559000000001,"stat_sum":{"num_bytes":23150826,"num_objects":587,"num_object_clones":269,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":199,"num_read":149,"num_read_kb":38846,"num_write":1362,"num_write_kb":24887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"77'1549","reported_seq":3580,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.362262+0000","last_change":"2023-10-30T12:37:07.102887+0000","last_active":"2023-10-30T13:25:32.362262+0000","last_peered":"2023-10-30T13:25:32.362262+0000","last_clean":"2023-10-30T13:25:32.362262+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T13:25:32.362262+0000","last_undegraded":"2023-10-30T13:25:32.362262+0000","last_fullsized":"2023-10-30T13:25:32.362262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1549,"log_dups_size":0,"ondisk_log_size":1549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014185980000000001,"stat_sum":{"num_bytes":36575095,"num_objects":618,"num_object_clones":282,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":216,"num_read":165,"num_read_kb":36418,"num_write":1354,"num_write_kb":45869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"77'1459","reported_seq":3492,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.241570+0000","last_change":"2023-10-30T12:37:07.183547+0000","last_active":"2023-10-30T13:25:32.241570+0000","last_peered":"2023-10-30T13:25:32.241570+0000","last_clean":"2023-10-30T13:25:32.241570+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T13:25:32.241570+0000","last_undegraded":"2023-10-30T13:25:32.241570+0000","last_fullsized":"2023-10-30T13:25:32.241570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1459,"log_dups_size":0,"ondisk_log_size":1459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091404980000000004,"stat_sum":{"num_bytes":11176343,"num_objects":586,"num_object_clones":270,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":213,"num_read":154,"num_read_kb":10770,"num_write":1260,"num_write_kb":13564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"82'1511","reported_seq":3518,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.222943+0000","last_change":"2023-10-30T12:37:07.181463+0000","last_active":"2023-10-30T13:25:32.222943+0000","last_peered":"2023-10-30T13:25:32.222943+0000","last_clean":"2023-10-30T13:25:32.222943+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T13:25:32.222943+0000","last_undegraded":"2023-10-30T13:25:32.222943+0000","last_fullsized":"2023-10-30T13:25:32.222943+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1511,"log_dups_size":0,"ondisk_log_size":1511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075712790000000002,"stat_sum":{"num_bytes":23568118,"num_objects":625,"num_object_clones":283,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":230,"num_read":141,"num_read_kb":34199,"num_write":1299,"num_write_kb":24762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"84'1725","reported_seq":3646,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.189229+0000","last_change":"2023-10-30T12:37:07.181114+0000","last_active":"2023-10-30T13:25:16.189229+0000","last_peered":"2023-10-30T13:25:16.189229+0000","last_clean":"2023-10-30T13:25:16.189229+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T13:25:16.189229+0000","last_undegraded":"2023-10-30T13:25:16.189229+0000","last_fullsized":"2023-10-30T13:25:16.189229+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1725,"log_dups_size":0,"ondisk_log_size":1725,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.007366493,"stat_sum":{"num_bytes":24142540,"num_objects":721,"num_object_clones":385,"num_object_copies":1442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":231,"num_read":157,"num_read_kb":24833,"num_write":1452,"num_write_kb":33857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"85'2099","reported_seq":4169,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.041415+0000","last_change":"2023-10-30T13:25:54.041494+0000","last_active":"2023-10-30T13:25:54.041415+0000","last_peered":"2023-10-30T13:25:54.041415+0000","last_clean":"2023-10-30T13:25:54.041415+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T13:25:54.041415+0000","last_undegraded":"2023-10-30T13:25:54.041415+0000","last_fullsized":"2023-10-30T13:25:54.041415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2099","last_scrub_stamp":"2023-10-30T13:25:54.041318+0000","last_deep_scrub":"85'2099","last_deep_scrub_stamp":"2023-10-30T13:25:54.041318+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.041318+0000","objects_scrubbed":596,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:14.898677+0000","scrub_duration":1.085239426,"objects_trimmed":0,"snaptrim_duration":0.006759035,"stat_sum":{"num_bytes":14254037,"num_objects":596,"num_object_clones":276,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":205,"num_read":165,"num_read_kb":21722,"num_write":1326,"num_write_kb":24306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"77'1568","reported_seq":3568,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.225757+0000","last_change":"2023-10-30T12:37:07.112609+0000","last_active":"2023-10-30T13:25:16.225757+0000","last_peered":"2023-10-30T13:25:16.225757+0000","last_clean":"2023-10-30T13:25:16.225757+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T13:25:16.225757+0000","last_undegraded":"2023-10-30T13:25:16.225757+0000","last_fullsized":"2023-10-30T13:25:16.225757+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1568,"log_dups_size":0,"ondisk_log_size":1568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057386779999999997,"stat_sum":{"num_bytes":10370740,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":213,"num_read":131,"num_read_kb":5845,"num_write":1381,"num_write_kb":12595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"77'1624","reported_seq":3605,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.250865+0000","last_change":"2023-10-30T12:37:07.187257+0000","last_active":"2023-10-30T13:25:16.250865+0000","last_peered":"2023-10-30T13:25:16.250865+0000","last_clean":"2023-10-30T13:25:16.250865+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T13:25:16.250865+0000","last_undegraded":"2023-10-30T13:25:16.250865+0000","last_fullsized":"2023-10-30T13:25:16.250865+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1624,"log_dups_size":0,"ondisk_log_size":1624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013549705,"stat_sum":{"num_bytes":19248519,"num_objects":637,"num_object_clones":295,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":221,"num_read":128,"num_read_kb":21688,"num_write":1447,"num_write_kb":21442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"85'2193","reported_seq":4229,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:46.252816+0000","last_change":"2023-10-30T13:25:46.252862+0000","last_active":"2023-10-30T13:25:46.252816+0000","last_peered":"2023-10-30T13:25:46.252816+0000","last_clean":"2023-10-30T13:25:46.252816+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T13:25:46.252816+0000","last_undegraded":"2023-10-30T13:25:46.252816+0000","last_fullsized":"2023-10-30T13:25:46.252816+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2193","last_scrub_stamp":"2023-10-30T13:25:46.252731+0000","last_deep_scrub":"85'2193","last_deep_scrub_stamp":"2023-10-30T13:25:46.252731+0000","last_clean_scrub_stamp":"2023-10-30T13:25:46.252731+0000","objects_scrubbed":619,"log_size":2193,"log_dups_size":0,"ondisk_log_size":2193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:51.513414+0000","scrub_duration":0.82934357400000003,"objects_trimmed":0,"snaptrim_duration":0.0075608309999999996,"stat_sum":{"num_bytes":14269019,"num_objects":619,"num_object_clones":290,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":233,"num_read":143,"num_read_kb":25965,"num_write":1377,"num_write_kb":20442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"85'2247","reported_seq":4267,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:45.441798+0000","last_change":"2023-10-30T13:25:45.441869+0000","last_active":"2023-10-30T13:25:45.441798+0000","last_peered":"2023-10-30T13:25:45.441798+0000","last_clean":"2023-10-30T13:25:45.441798+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T13:25:45.441798+0000","last_undegraded":"2023-10-30T13:25:45.441798+0000","last_fullsized":"2023-10-30T13:25:45.441798+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2247","last_scrub_stamp":"2023-10-30T13:25:45.441689+0000","last_deep_scrub":"85'2247","last_deep_scrub_stamp":"2023-10-30T13:25:45.441689+0000","last_clean_scrub_stamp":"2023-10-30T13:25:45.441689+0000","objects_scrubbed":642,"log_size":2247,"log_dups_size":0,"ondisk_log_size":2247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:54:27.498757+0000","scrub_duration":0.97553288699999996,"objects_trimmed":0,"snaptrim_duration":0.0099357109999999998,"stat_sum":{"num_bytes":14793776,"num_objects":642,"num_object_clones":304,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":242,"num_read":140,"num_read_kb":30098,"num_write":1386,"num_write_kb":20714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"85'2006","reported_seq":4061,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.398039+0000","last_change":"2023-10-30T13:25:57.398144+0000","last_active":"2023-10-30T13:25:57.398039+0000","last_peered":"2023-10-30T13:25:57.398039+0000","last_clean":"2023-10-30T13:25:57.398039+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T13:25:57.398039+0000","last_undegraded":"2023-10-30T13:25:57.398039+0000","last_fullsized":"2023-10-30T13:25:57.398039+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2006","last_scrub_stamp":"2023-10-30T13:25:57.397929+0000","last_deep_scrub":"85'2006","last_deep_scrub_stamp":"2023-10-30T13:25:57.397929+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.397929+0000","objects_scrubbed":562,"log_size":2006,"log_dups_size":0,"ondisk_log_size":2006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:28.919675+0000","scrub_duration":1.1960647630000001,"objects_trimmed":0,"snaptrim_duration":0.0065893760000000001,"stat_sum":{"num_bytes":15508852,"num_objects":562,"num_object_clones":263,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":216,"num_read":137,"num_read_kb":10064,"num_write":1244,"num_write_kb":17198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"82'1576","reported_seq":3679,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.228028+0000","last_change":"2023-10-30T12:37:07.182668+0000","last_active":"2023-10-30T13:25:32.228028+0000","last_peered":"2023-10-30T13:25:32.228028+0000","last_clean":"2023-10-30T13:25:32.228028+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T13:25:32.228028+0000","last_undegraded":"2023-10-30T13:25:32.228028+0000","last_fullsized":"2023-10-30T13:25:32.228028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1576,"log_dups_size":0,"ondisk_log_size":1576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010472766,"stat_sum":{"num_bytes":15695810,"num_objects":614,"num_object_clones":281,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":221,"num_read":233,"num_read_kb":17180,"num_write":1382,"num_write_kb":23481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"77'1569","reported_seq":3574,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.238326+0000","last_change":"2023-10-30T12:37:07.181518+0000","last_active":"2023-10-30T13:25:32.238326+0000","last_peered":"2023-10-30T13:25:32.238326+0000","last_clean":"2023-10-30T13:25:32.238326+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T13:25:32.238326+0000","last_undegraded":"2023-10-30T13:25:32.238326+0000","last_fullsized":"2023-10-30T13:25:32.238326+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1569,"log_dups_size":0,"ondisk_log_size":1569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0075455599999999998,"stat_sum":{"num_bytes":5586810,"num_objects":614,"num_object_clones":286,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":230,"num_read":140,"num_read_kb":5846,"num_write":1359,"num_write_kb":15885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"85'2129","reported_seq":4280,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.179234+0000","last_change":"2023-10-30T13:25:54.179340+0000","last_active":"2023-10-30T13:25:54.179234+0000","last_peered":"2023-10-30T13:25:54.179234+0000","last_clean":"2023-10-30T13:25:54.179234+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T13:25:54.179234+0000","last_undegraded":"2023-10-30T13:25:54.179234+0000","last_fullsized":"2023-10-30T13:25:54.179234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2129","last_scrub_stamp":"2023-10-30T13:25:54.178827+0000","last_deep_scrub":"85'2129","last_deep_scrub_stamp":"2023-10-30T13:25:54.178827+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.178827+0000","objects_scrubbed":606,"log_size":2129,"log_dups_size":0,"ondisk_log_size":2129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:18.992880+0000","scrub_duration":1.0904528849999999,"objects_trimmed":0,"snaptrim_duration":0.0076434509999999999,"stat_sum":{"num_bytes":23023646,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":222,"num_read":249,"num_read_kb":26208,"num_write":1319,"num_write_kb":28338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"84'1671","reported_seq":3655,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.088803+0000","last_change":"2023-10-30T12:37:07.102791+0000","last_active":"2023-10-30T13:25:03.088803+0000","last_peered":"2023-10-30T13:25:03.088803+0000","last_clean":"2023-10-30T13:25:03.088803+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T13:25:03.088803+0000","last_undegraded":"2023-10-30T13:25:03.088803+0000","last_fullsized":"2023-10-30T13:25:03.088803+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1671,"log_dups_size":0,"ondisk_log_size":1671,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016105379999999999,"stat_sum":{"num_bytes":11166034,"num_objects":653,"num_object_clones":306,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":233,"num_read":139,"num_read_kb":9627,"num_write":1472,"num_write_kb":18195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"77'1520","reported_seq":3540,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:01.374670+0000","last_change":"2023-10-30T12:37:07.188206+0000","last_active":"2023-10-30T13:25:01.374670+0000","last_peered":"2023-10-30T13:25:01.374670+0000","last_clean":"2023-10-30T13:25:01.374670+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T13:25:01.374670+0000","last_undegraded":"2023-10-30T13:25:01.374670+0000","last_fullsized":"2023-10-30T13:25:01.374670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014064241,"stat_sum":{"num_bytes":10716893,"num_objects":582,"num_object_clones":264,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":203,"num_read":135,"num_read_kb":13340,"num_write":1353,"num_write_kb":16108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"85'2148","reported_seq":4184,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.477345+0000","last_change":"2023-10-30T13:25:44.477389+0000","last_active":"2023-10-30T13:25:44.477345+0000","last_peered":"2023-10-30T13:25:44.477345+0000","last_clean":"2023-10-30T13:25:44.477345+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T13:25:44.477345+0000","last_undegraded":"2023-10-30T13:25:44.477345+0000","last_fullsized":"2023-10-30T13:25:44.477345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2148","last_scrub_stamp":"2023-10-30T13:25:44.477273+0000","last_deep_scrub":"85'2148","last_deep_scrub_stamp":"2023-10-30T13:25:44.477273+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.477273+0000","objects_scrubbed":597,"log_size":2148,"log_dups_size":0,"ondisk_log_size":2148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:41.538304+0000","scrub_duration":0.99138869699999999,"objects_trimmed":0,"snaptrim_duration":0.0088476170000000003,"stat_sum":{"num_bytes":10233195,"num_objects":597,"num_object_clones":275,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":203,"num_read":127,"num_read_kb":1819,"num_write":1383,"num_write_kb":16400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"82'1465","reported_seq":3480,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.203328+0000","last_change":"2023-10-30T12:37:07.188606+0000","last_active":"2023-10-30T13:25:16.203328+0000","last_peered":"2023-10-30T13:25:16.203328+0000","last_clean":"2023-10-30T13:25:16.203328+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T13:25:16.203328+0000","last_undegraded":"2023-10-30T13:25:16.203328+0000","last_fullsized":"2023-10-30T13:25:16.203328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014279337,"stat_sum":{"num_bytes":22085088,"num_objects":582,"num_object_clones":270,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":216,"num_read":135,"num_read_kb":28666,"num_write":1258,"num_write_kb":25992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"77'1609","reported_seq":3603,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.211359+0000","last_change":"2023-10-30T12:37:07.177071+0000","last_active":"2023-10-30T13:25:16.211359+0000","last_peered":"2023-10-30T13:25:16.211359+0000","last_clean":"2023-10-30T13:25:16.211359+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T13:25:16.211359+0000","last_undegraded":"2023-10-30T13:25:16.211359+0000","last_fullsized":"2023-10-30T13:25:16.211359+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1609,"log_dups_size":0,"ondisk_log_size":1609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.004006122,"stat_sum":{"num_bytes":15408826,"num_objects":598,"num_object_clones":277,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":213,"num_read":119,"num_read_kb":9406,"num_write":1408,"num_write_kb":18485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"85'2174","reported_seq":4213,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.084913+0000","last_change":"2023-10-30T13:25:53.084979+0000","last_active":"2023-10-30T13:25:53.084913+0000","last_peered":"2023-10-30T13:25:53.084913+0000","last_clean":"2023-10-30T13:25:53.084913+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T13:25:53.084913+0000","last_undegraded":"2023-10-30T13:25:53.084913+0000","last_fullsized":"2023-10-30T13:25:53.084913+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2174","last_scrub_stamp":"2023-10-30T13:25:53.084846+0000","last_deep_scrub":"85'2174","last_deep_scrub_stamp":"2023-10-30T13:25:53.084846+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.084846+0000","objects_scrubbed":625,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:02.345358+0000","scrub_duration":1.1204613670000001,"objects_trimmed":0,"snaptrim_duration":0.0059072409999999997,"stat_sum":{"num_bytes":25755688,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":240,"num_read":144,"num_read_kb":24326,"num_write":1332,"num_write_kb":28963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"85'1485","reported_seq":3467,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.239575+0000","last_change":"2023-10-30T12:37:07.113316+0000","last_active":"2023-10-30T13:25:32.239575+0000","last_peered":"2023-10-30T13:25:32.239575+0000","last_clean":"2023-10-30T13:25:32.239575+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T13:25:32.239575+0000","last_undegraded":"2023-10-30T13:25:32.239575+0000","last_fullsized":"2023-10-30T13:25:32.239575+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1485,"log_dups_size":0,"ondisk_log_size":1485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062401690000000003,"stat_sum":{"num_bytes":11021781,"num_objects":617,"num_object_clones":286,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":226,"num_read":116,"num_read_kb":10954,"num_write":1301,"num_write_kb":16114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"85'2273","reported_seq":4279,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:55.521292+0000","last_change":"2023-10-30T13:25:55.521358+0000","last_active":"2023-10-30T13:25:55.521292+0000","last_peered":"2023-10-30T13:25:55.521292+0000","last_clean":"2023-10-30T13:25:55.521292+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T13:25:55.521292+0000","last_undegraded":"2023-10-30T13:25:55.521292+0000","last_fullsized":"2023-10-30T13:25:55.521292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2273","last_scrub_stamp":"2023-10-30T13:25:55.521205+0000","last_deep_scrub":"85'2273","last_deep_scrub_stamp":"2023-10-30T13:25:55.521205+0000","last_clean_scrub_stamp":"2023-10-30T13:25:55.521205+0000","objects_scrubbed":653,"log_size":2273,"log_dups_size":0,"ondisk_log_size":2273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:44:08.018678+0000","scrub_duration":1.1272668750000001,"objects_trimmed":0,"snaptrim_duration":0.014933905000000001,"stat_sum":{"num_bytes":11161783,"num_objects":653,"num_object_clones":309,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":240,"num_read":130,"num_read_kb":12352,"num_write":1416,"num_write_kb":13402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"82'1476","reported_seq":3520,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.210549+0000","last_change":"2023-10-30T12:37:07.180055+0000","last_active":"2023-10-30T13:25:16.210549+0000","last_peered":"2023-10-30T13:25:16.210549+0000","last_clean":"2023-10-30T13:25:16.210549+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T13:25:16.210549+0000","last_undegraded":"2023-10-30T13:25:16.210549+0000","last_fullsized":"2023-10-30T13:25:16.210549+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0090148399999999997,"stat_sum":{"num_bytes":11081748,"num_objects":549,"num_object_clones":251,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":201,"num_read":142,"num_read_kb":9474,"num_write":1288,"num_write_kb":13652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"85'2406","reported_seq":4433,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.442339+0000","last_change":"2023-10-30T13:25:54.442416+0000","last_active":"2023-10-30T13:25:54.442339+0000","last_peered":"2023-10-30T13:25:54.442339+0000","last_clean":"2023-10-30T13:25:54.442339+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T13:25:54.442339+0000","last_undegraded":"2023-10-30T13:25:54.442339+0000","last_fullsized":"2023-10-30T13:25:54.442339+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2406","last_scrub_stamp":"2023-10-30T13:25:54.442247+0000","last_deep_scrub":"85'2406","last_deep_scrub_stamp":"2023-10-30T13:25:54.442247+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.442247+0000","objects_scrubbed":693,"log_size":2406,"log_dups_size":0,"ondisk_log_size":2406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:12.648322+0000","scrub_duration":1.2677238399999999,"objects_trimmed":0,"snaptrim_duration":0.0066657749999999997,"stat_sum":{"num_bytes":20125251,"num_objects":693,"num_object_clones":330,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":267,"num_read":168,"num_read_kb":25254,"num_write":1457,"num_write_kb":26871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"85'2152","reported_seq":4191,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:45.218457+0000","last_change":"2023-10-30T13:25:45.218569+0000","last_active":"2023-10-30T13:25:45.218457+0000","last_peered":"2023-10-30T13:25:45.218457+0000","last_clean":"2023-10-30T13:25:45.218457+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T13:25:45.218457+0000","last_undegraded":"2023-10-30T13:25:45.218457+0000","last_fullsized":"2023-10-30T13:25:45.218457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2152","last_scrub_stamp":"2023-10-30T13:25:45.218360+0000","last_deep_scrub":"85'2152","last_deep_scrub_stamp":"2023-10-30T13:25:45.218360+0000","last_clean_scrub_stamp":"2023-10-30T13:25:45.218360+0000","objects_scrubbed":604,"log_size":2152,"log_dups_size":0,"ondisk_log_size":2152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:53:35.163970+0000","scrub_duration":1.1092469380000001,"objects_trimmed":0,"snaptrim_duration":0.005419129,"stat_sum":{"num_bytes":18031744,"num_objects":604,"num_object_clones":282,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":228,"num_read":136,"num_read_kb":25964,"num_write":1344,"num_write_kb":19625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"85'1548","reported_seq":3562,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.161751+0000","last_change":"2023-10-30T12:37:07.102583+0000","last_active":"2023-10-30T13:25:16.161751+0000","last_peered":"2023-10-30T13:25:16.161751+0000","last_clean":"2023-10-30T13:25:16.161751+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T13:25:16.161751+0000","last_undegraded":"2023-10-30T13:25:16.161751+0000","last_fullsized":"2023-10-30T13:25:16.161751+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1548,"log_dups_size":0,"ondisk_log_size":1548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013742470000000001,"stat_sum":{"num_bytes":22381695,"num_objects":607,"num_object_clones":275,"num_object_copies":1214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":227,"num_read":141,"num_read_kb":33421,"num_write":1348,"num_write_kb":23422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1042","reported_seq":1465,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:42.402347+0000","last_change":"2023-10-30T13:25:42.402455+0000","last_active":"2023-10-30T13:25:42.402347+0000","last_peered":"2023-10-30T13:25:42.402347+0000","last_clean":"2023-10-30T13:25:42.402347+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T13:25:42.402347+0000","last_undegraded":"2023-10-30T13:25:42.402347+0000","last_fullsized":"2023-10-30T13:25:42.402347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1042","last_scrub_stamp":"2023-10-30T13:25:42.402195+0000","last_deep_scrub":"85'1042","last_deep_scrub_stamp":"2023-10-30T13:25:42.402195+0000","last_clean_scrub_stamp":"2023-10-30T13:25:42.402195+0000","objects_scrubbed":51,"log_size":1042,"log_dups_size":0,"ondisk_log_size":1042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:20.707501+0000","scrub_duration":0.156286863,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":570,"num_read_kb":319827,"num_write":1218,"num_write_kb":80521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":794018,"num_omap_keys":1312,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"85'2177","reported_seq":4220,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:01.438476+0000","last_change":"2023-10-30T13:26:01.438545+0000","last_active":"2023-10-30T13:26:01.438476+0000","last_peered":"2023-10-30T13:26:01.438476+0000","last_clean":"2023-10-30T13:26:01.438476+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T13:26:01.438476+0000","last_undegraded":"2023-10-30T13:26:01.438476+0000","last_fullsized":"2023-10-30T13:26:01.438476+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2177","last_scrub_stamp":"2023-10-30T13:26:01.438339+0000","last_deep_scrub":"85'2177","last_deep_scrub_stamp":"2023-10-30T13:26:01.438339+0000","last_clean_scrub_stamp":"2023-10-30T13:26:01.438339+0000","objects_scrubbed":616,"log_size":2177,"log_dups_size":0,"ondisk_log_size":2177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:25.473052+0000","scrub_duration":1.2567298680000001,"objects_trimmed":0,"snaptrim_duration":0.0086018069999999995,"stat_sum":{"num_bytes":19940185,"num_objects":616,"num_object_clones":283,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":213,"num_read":145,"num_read_kb":23669,"num_write":1398,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1795","reported_seq":2376,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:57.646617+0000","last_change":"2023-10-30T12:28:44.060621+0000","last_active":"2023-10-30T13:24:57.646617+0000","last_peered":"2023-10-30T13:24:57.646617+0000","last_clean":"2023-10-30T13:24:57.646617+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T13:24:57.646617+0000","last_undegraded":"2023-10-30T13:24:57.646617+0000","last_fullsized":"2023-10-30T13:24:57.646617+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1795,"log_dups_size":0,"ondisk_log_size":1795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17028825,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":761,"num_read_kb":396040,"num_write":1931,"num_write_kb":84639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"85'1626","reported_seq":2455,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:46.170208+0000","last_change":"2023-10-30T13:25:46.170305+0000","last_active":"2023-10-30T13:25:46.170208+0000","last_peered":"2023-10-30T13:25:46.170208+0000","last_clean":"2023-10-30T13:25:46.170208+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T13:25:46.170208+0000","last_undegraded":"2023-10-30T13:25:46.170208+0000","last_fullsized":"2023-10-30T13:25:46.170208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1626","last_scrub_stamp":"2023-10-30T13:25:46.170040+0000","last_deep_scrub":"85'1626","last_deep_scrub_stamp":"2023-10-30T13:25:46.170040+0000","last_clean_scrub_stamp":"2023-10-30T13:25:46.170040+0000","objects_scrubbed":57,"log_size":1626,"log_dups_size":0,"ondisk_log_size":1626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:30:38.684197+0000","scrub_duration":0.213161079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182949,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":1117,"num_read_kb":268002,"num_write":1800,"num_write_kb":63122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":717548,"num_omap_keys":1302,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"85'2510","reported_seq":4555,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.637463+0000","last_change":"2023-10-30T13:25:54.637570+0000","last_active":"2023-10-30T13:25:54.637463+0000","last_peered":"2023-10-30T13:25:54.637463+0000","last_clean":"2023-10-30T13:25:54.637463+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T13:25:54.637463+0000","last_undegraded":"2023-10-30T13:25:54.637463+0000","last_fullsized":"2023-10-30T13:25:54.637463+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2510","last_scrub_stamp":"2023-10-30T13:25:54.637344+0000","last_deep_scrub":"85'2510","last_deep_scrub_stamp":"2023-10-30T13:25:54.637344+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.637344+0000","objects_scrubbed":709,"log_size":2510,"log_dups_size":0,"ondisk_log_size":2510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:29:25.404778+0000","scrub_duration":1.3090190449999999,"objects_trimmed":0,"snaptrim_duration":0.016143469000000001,"stat_sum":{"num_bytes":14540052,"num_objects":709,"num_object_clones":329,"num_object_copies":1418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":254,"num_read":191,"num_read_kb":6719,"num_write":1583,"num_write_kb":26382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"85'930","reported_seq":1333,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:24:52.371140+0000","last_change":"2023-10-30T12:28:44.058798+0000","last_active":"2023-10-30T13:24:52.371140+0000","last_peered":"2023-10-30T13:24:52.371140+0000","last_clean":"2023-10-30T13:24:52.371140+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T13:24:52.371140+0000","last_undegraded":"2023-10-30T13:24:52.371140+0000","last_fullsized":"2023-10-30T13:24:52.371140+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":930,"log_dups_size":0,"ondisk_log_size":930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":542,"num_read_kb":359029,"num_write":1093,"num_write_kb":88246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"85'2113","reported_seq":4164,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:55.071692+0000","last_change":"2023-10-30T13:25:55.071766+0000","last_active":"2023-10-30T13:25:55.071692+0000","last_peered":"2023-10-30T13:25:55.071692+0000","last_clean":"2023-10-30T13:25:55.071692+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T13:25:55.071692+0000","last_undegraded":"2023-10-30T13:25:55.071692+0000","last_fullsized":"2023-10-30T13:25:55.071692+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2113","last_scrub_stamp":"2023-10-30T13:25:55.071614+0000","last_deep_scrub":"85'2113","last_deep_scrub_stamp":"2023-10-30T13:25:55.071614+0000","last_clean_scrub_stamp":"2023-10-30T13:25:55.071614+0000","objects_scrubbed":589,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:07.445157+0000","scrub_duration":0.93122268500000005,"objects_trimmed":0,"snaptrim_duration":0.0083101389999999994,"stat_sum":{"num_bytes":18904879,"num_objects":589,"num_object_clones":276,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":211,"num_read":147,"num_read_kb":20021,"num_write":1340,"num_write_kb":25244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"85'789","reported_seq":1200,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:49.688428+0000","last_change":"2023-10-30T13:25:49.688470+0000","last_active":"2023-10-30T13:25:49.688428+0000","last_peered":"2023-10-30T13:25:49.688428+0000","last_clean":"2023-10-30T13:25:49.688428+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T13:25:49.688428+0000","last_undegraded":"2023-10-30T13:25:49.688428+0000","last_fullsized":"2023-10-30T13:25:49.688428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'789","last_scrub_stamp":"2023-10-30T13:25:49.688344+0000","last_deep_scrub":"85'789","last_deep_scrub_stamp":"2023-10-30T13:25:49.688344+0000","last_clean_scrub_stamp":"2023-10-30T13:25:49.688344+0000","objects_scrubbed":53,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:56.666560+0000","scrub_duration":0.15913772100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":616,"num_read_kb":251423,"num_write":890,"num_write_kb":55054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":627193,"num_omap_keys":1090,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"85'2098","reported_seq":4145,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.184775+0000","last_change":"2023-10-30T13:25:57.184868+0000","last_active":"2023-10-30T13:25:57.184775+0000","last_peered":"2023-10-30T13:25:57.184775+0000","last_clean":"2023-10-30T13:25:57.184775+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T13:25:57.184775+0000","last_undegraded":"2023-10-30T13:25:57.184775+0000","last_fullsized":"2023-10-30T13:25:57.184775+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2098","last_scrub_stamp":"2023-10-30T13:25:57.184676+0000","last_deep_scrub":"85'2098","last_deep_scrub_stamp":"2023-10-30T13:25:57.184676+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.184676+0000","objects_scrubbed":580,"log_size":2098,"log_dups_size":0,"ondisk_log_size":2098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:36.904344+0000","scrub_duration":1.084383391,"objects_trimmed":0,"snaptrim_duration":0.0043045619999999996,"stat_sum":{"num_bytes":10134301,"num_objects":580,"num_object_clones":271,"num_object_copies":1160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":580,"num_whiteouts":219,"num_read":138,"num_read_kb":8986,"num_write":1318,"num_write_kb":11276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"85'1309","reported_seq":1758,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.351882+0000","last_change":"2023-10-30T13:25:54.351972+0000","last_active":"2023-10-30T13:25:54.351882+0000","last_peered":"2023-10-30T13:25:54.351882+0000","last_clean":"2023-10-30T13:25:54.351882+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T13:25:54.351882+0000","last_undegraded":"2023-10-30T13:25:54.351882+0000","last_fullsized":"2023-10-30T13:25:54.351882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1309","last_scrub_stamp":"2023-10-30T13:25:54.351769+0000","last_deep_scrub":"85'1309","last_deep_scrub_stamp":"2023-10-30T13:25:54.351769+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.351769+0000","objects_scrubbed":62,"log_size":1309,"log_dups_size":0,"ondisk_log_size":1309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:15.955314+0000","scrub_duration":0.18331900800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":659,"num_read_kb":339384,"num_write":1492,"num_write_kb":80693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":875213,"num_omap_keys":1399,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"85'2223","reported_seq":4246,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:59.747975+0000","last_change":"2023-10-30T13:25:59.748025+0000","last_active":"2023-10-30T13:25:59.747975+0000","last_peered":"2023-10-30T13:25:59.747975+0000","last_clean":"2023-10-30T13:25:59.747975+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T13:25:59.747975+0000","last_undegraded":"2023-10-30T13:25:59.747975+0000","last_fullsized":"2023-10-30T13:25:59.747975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2223","last_scrub_stamp":"2023-10-30T13:25:59.747898+0000","last_deep_scrub":"85'2223","last_deep_scrub_stamp":"2023-10-30T13:25:59.747898+0000","last_clean_scrub_stamp":"2023-10-30T13:25:59.747898+0000","objects_scrubbed":620,"log_size":2223,"log_dups_size":0,"ondisk_log_size":2223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:22.443096+0000","scrub_duration":1.320524545,"objects_trimmed":0,"snaptrim_duration":0.015702779,"stat_sum":{"num_bytes":15773172,"num_objects":620,"num_object_clones":292,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":204,"num_read":135,"num_read_kb":17351,"num_write":1431,"num_write_kb":17077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"85'1282","reported_seq":1786,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:43.374996+0000","last_change":"2023-10-30T13:25:43.375075+0000","last_active":"2023-10-30T13:25:43.374996+0000","last_peered":"2023-10-30T13:25:43.374996+0000","last_clean":"2023-10-30T13:25:43.374996+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T13:25:43.374996+0000","last_undegraded":"2023-10-30T13:25:43.374996+0000","last_fullsized":"2023-10-30T13:25:43.374996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1282","last_scrub_stamp":"2023-10-30T13:25:43.374894+0000","last_deep_scrub":"85'1282","last_deep_scrub_stamp":"2023-10-30T13:25:43.374894+0000","last_clean_scrub_stamp":"2023-10-30T13:25:43.374894+0000","objects_scrubbed":59,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:32.582441+0000","scrub_duration":0.20001206899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19774519,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":862,"num_read_kb":221068,"num_write":1454,"num_write_kb":51536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":878727,"num_omap_keys":1229,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"82'1515","reported_seq":3525,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.435008+0000","last_change":"2023-10-30T12:37:07.178872+0000","last_active":"2023-10-30T13:25:10.435008+0000","last_peered":"2023-10-30T13:25:10.435008+0000","last_clean":"2023-10-30T13:25:10.435008+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T13:25:10.435008+0000","last_undegraded":"2023-10-30T13:25:10.435008+0000","last_fullsized":"2023-10-30T13:25:10.435008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080610860000000003,"stat_sum":{"num_bytes":22077043,"num_objects":612,"num_object_clones":281,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":230,"num_read":141,"num_read_kb":18856,"num_write":1294,"num_write_kb":28337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"85'1397","reported_seq":1954,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:41.415232+0000","last_change":"2023-10-30T13:25:41.415318+0000","last_active":"2023-10-30T13:25:41.415232+0000","last_peered":"2023-10-30T13:25:41.415232+0000","last_clean":"2023-10-30T13:25:41.415232+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T13:25:41.415232+0000","last_undegraded":"2023-10-30T13:25:41.415232+0000","last_fullsized":"2023-10-30T13:25:41.415232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1397","last_scrub_stamp":"2023-10-30T13:25:41.415138+0000","last_deep_scrub":"85'1397","last_deep_scrub_stamp":"2023-10-30T13:25:41.415138+0000","last_clean_scrub_stamp":"2023-10-30T13:25:41.415138+0000","objects_scrubbed":67,"log_size":1397,"log_dups_size":0,"ondisk_log_size":1397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:14.524619+0000","scrub_duration":0.28929933099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":865,"num_read_kb":420163,"num_write":1566,"num_write_kb":89397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":814706,"num_omap_keys":1307,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"82'1535","reported_seq":3534,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.301105+0000","last_change":"2023-10-30T12:37:07.182869+0000","last_active":"2023-10-30T13:25:32.301105+0000","last_peered":"2023-10-30T13:25:32.301105+0000","last_clean":"2023-10-30T13:25:32.301105+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T13:25:32.301105+0000","last_undegraded":"2023-10-30T13:25:32.301105+0000","last_fullsized":"2023-10-30T13:25:32.301105+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1535,"log_dups_size":0,"ondisk_log_size":1535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.008640567,"stat_sum":{"num_bytes":23022369,"num_objects":606,"num_object_clones":279,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":220,"num_read":129,"num_read_kb":18941,"num_write":1340,"num_write_kb":24878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"85'1164","reported_seq":1635,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.276943+0000","last_change":"2023-10-30T13:25:48.277044+0000","last_active":"2023-10-30T13:25:48.276943+0000","last_peered":"2023-10-30T13:25:48.276943+0000","last_clean":"2023-10-30T13:25:48.276943+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T13:25:48.276943+0000","last_undegraded":"2023-10-30T13:25:48.276943+0000","last_fullsized":"2023-10-30T13:25:48.276943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1164","last_scrub_stamp":"2023-10-30T13:25:48.276852+0000","last_deep_scrub":"85'1164","last_deep_scrub_stamp":"2023-10-30T13:25:48.276852+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.276852+0000","objects_scrubbed":66,"log_size":1164,"log_dups_size":0,"ondisk_log_size":1164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:06.856285+0000","scrub_duration":0.19954839099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554466,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":701,"num_read_kb":318483,"num_write":1303,"num_write_kb":68223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":676667,"num_omap_keys":1245,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"85'2197","reported_seq":4231,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:52.017102+0000","last_change":"2023-10-30T13:25:52.017193+0000","last_active":"2023-10-30T13:25:52.017102+0000","last_peered":"2023-10-30T13:25:52.017102+0000","last_clean":"2023-10-30T13:25:52.017102+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T13:25:52.017102+0000","last_undegraded":"2023-10-30T13:25:52.017102+0000","last_fullsized":"2023-10-30T13:25:52.017102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2197","last_scrub_stamp":"2023-10-30T13:25:52.016976+0000","last_deep_scrub":"85'2197","last_deep_scrub_stamp":"2023-10-30T13:25:52.016976+0000","last_clean_scrub_stamp":"2023-10-30T13:25:52.016976+0000","objects_scrubbed":613,"log_size":2197,"log_dups_size":0,"ondisk_log_size":2197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:09:13.519760+0000","scrub_duration":1.011428851,"objects_trimmed":0,"snaptrim_duration":0.0073009930000000004,"stat_sum":{"num_bytes":10084838,"num_objects":613,"num_object_clones":288,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":217,"num_read":139,"num_read_kb":5686,"num_write":1383,"num_write_kb":20147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'729","reported_seq":1132,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.578886+0000","last_change":"2023-10-30T13:25:54.578930+0000","last_active":"2023-10-30T13:25:54.578886+0000","last_peered":"2023-10-30T13:25:54.578886+0000","last_clean":"2023-10-30T13:25:54.578886+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T13:25:54.578886+0000","last_undegraded":"2023-10-30T13:25:54.578886+0000","last_fullsized":"2023-10-30T13:25:54.578886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'729","last_scrub_stamp":"2023-10-30T13:25:54.578800+0000","last_deep_scrub":"85'729","last_deep_scrub_stamp":"2023-10-30T13:25:54.578800+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.578800+0000","objects_scrubbed":53,"log_size":729,"log_dups_size":0,"ondisk_log_size":729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:06:15.394870+0000","scrub_duration":0.12999461300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":600,"num_read_kb":208389,"num_write":899,"num_write_kb":51587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":569345,"num_omap_keys":982,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"82'1642","reported_seq":3653,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.178174+0000","last_change":"2023-10-30T12:37:07.180453+0000","last_active":"2023-10-30T13:25:16.178174+0000","last_peered":"2023-10-30T13:25:16.178174+0000","last_clean":"2023-10-30T13:25:16.178174+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T13:25:16.178174+0000","last_undegraded":"2023-10-30T13:25:16.178174+0000","last_fullsized":"2023-10-30T13:25:16.178174+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091258150000000007,"stat_sum":{"num_bytes":19392609,"num_objects":655,"num_object_clones":299,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":233,"num_read":161,"num_read_kb":7475,"num_write":1440,"num_write_kb":20643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"85'1416","reported_seq":1810,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:40.394192+0000","last_change":"2023-10-30T13:25:40.394262+0000","last_active":"2023-10-30T13:25:40.394192+0000","last_peered":"2023-10-30T13:25:40.394192+0000","last_clean":"2023-10-30T13:25:40.394192+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T13:25:40.394192+0000","last_undegraded":"2023-10-30T13:25:40.394192+0000","last_fullsized":"2023-10-30T13:25:40.394192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1416","last_scrub_stamp":"2023-10-30T13:25:40.394118+0000","last_deep_scrub":"85'1416","last_deep_scrub_stamp":"2023-10-30T13:25:40.394118+0000","last_clean_scrub_stamp":"2023-10-30T13:25:40.394118+0000","objects_scrubbed":45,"log_size":1416,"log_dups_size":0,"ondisk_log_size":1416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:50:35.732748+0000","scrub_duration":0.10710792099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":539,"num_read_kb":266844,"num_write":1512,"num_write_kb":63647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":566794,"num_omap_keys":1066,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"85'1872","reported_seq":3879,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-30T13:25:58.109988+0000","last_change":"2023-10-30T13:25:57.438014+0000","last_active":"2023-10-30T13:25:58.109988+0000","last_peered":"2023-10-30T13:25:58.109988+0000","last_clean":"2023-10-30T13:25:58.109988+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T13:25:58.109988+0000","last_undegraded":"2023-10-30T13:25:58.109988+0000","last_fullsized":"2023-10-30T13:25:58.109988+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":391,"log_size":1872,"log_dups_size":0,"ondisk_log_size":1872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001646376,"stat_sum":{"num_bytes":13778361,"num_objects":590,"num_object_clones":278,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":227,"num_read":137,"num_read_kb":5723,"num_write":1258,"num_write_kb":14976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"85'1020","reported_seq":1516,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:45.278920+0000","last_change":"2023-10-30T13:25:45.278968+0000","last_active":"2023-10-30T13:25:45.278920+0000","last_peered":"2023-10-30T13:25:45.278920+0000","last_clean":"2023-10-30T13:25:45.278920+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T13:25:45.278920+0000","last_undegraded":"2023-10-30T13:25:45.278920+0000","last_fullsized":"2023-10-30T13:25:45.278920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1020","last_scrub_stamp":"2023-10-30T13:25:45.278835+0000","last_deep_scrub":"85'1020","last_deep_scrub_stamp":"2023-10-30T13:25:45.278835+0000","last_clean_scrub_stamp":"2023-10-30T13:25:45.278835+0000","objects_scrubbed":50,"log_size":1020,"log_dups_size":0,"ondisk_log_size":1020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:18:17.671466+0000","scrub_duration":0.27418579599999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554544,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":863,"num_read_kb":272228,"num_write":1253,"num_write_kb":56723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":534875,"num_omap_keys":741,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"82'1616","reported_seq":3637,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.146925+0000","last_change":"2023-10-30T12:37:07.181542+0000","last_active":"2023-10-30T13:25:16.146925+0000","last_peered":"2023-10-30T13:25:16.146925+0000","last_clean":"2023-10-30T13:25:16.146925+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T13:25:16.146925+0000","last_undegraded":"2023-10-30T13:25:16.146925+0000","last_fullsized":"2023-10-30T13:25:16.146925+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1616,"log_dups_size":0,"ondisk_log_size":1616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098326820000000006,"stat_sum":{"num_bytes":10901701,"num_objects":622,"num_object_clones":292,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":232,"num_read":163,"num_read_kb":14238,"num_write":1406,"num_write_kb":21652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"85'696","reported_seq":1065,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:42.423508+0000","last_change":"2023-10-30T13:25:42.423609+0000","last_active":"2023-10-30T13:25:42.423508+0000","last_peered":"2023-10-30T13:25:42.423508+0000","last_clean":"2023-10-30T13:25:42.423508+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T13:25:42.423508+0000","last_undegraded":"2023-10-30T13:25:42.423508+0000","last_fullsized":"2023-10-30T13:25:42.423508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'696","last_scrub_stamp":"2023-10-30T13:25:42.423407+0000","last_deep_scrub":"85'696","last_deep_scrub_stamp":"2023-10-30T13:25:42.423407+0000","last_clean_scrub_stamp":"2023-10-30T13:25:42.423407+0000","objects_scrubbed":41,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:12:00.329817+0000","scrub_duration":0.18306038499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":443,"num_read_kb":261670,"num_write":858,"num_write_kb":58986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":305625,"num_omap_keys":524,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"85'2220","reported_seq":4221,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.104935+0000","last_change":"2023-10-30T13:25:53.105023+0000","last_active":"2023-10-30T13:25:53.104935+0000","last_peered":"2023-10-30T13:25:53.104935+0000","last_clean":"2023-10-30T13:25:53.104935+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T13:25:53.104935+0000","last_undegraded":"2023-10-30T13:25:53.104935+0000","last_fullsized":"2023-10-30T13:25:53.104935+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2220","last_scrub_stamp":"2023-10-30T13:25:53.104840+0000","last_deep_scrub":"85'2220","last_deep_scrub_stamp":"2023-10-30T13:25:53.104840+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.104840+0000","objects_scrubbed":647,"log_size":2220,"log_dups_size":0,"ondisk_log_size":2220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:47:46.541296+0000","scrub_duration":1.013459932,"objects_trimmed":0,"snaptrim_duration":0.004602437,"stat_sum":{"num_bytes":20428330,"num_objects":647,"num_object_clones":304,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":239,"num_read":119,"num_read_kb":13097,"num_write":1360,"num_write_kb":26395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"85'779","reported_seq":1239,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.328717+0000","last_change":"2023-10-30T13:25:48.328798+0000","last_active":"2023-10-30T13:25:48.328717+0000","last_peered":"2023-10-30T13:25:48.328717+0000","last_clean":"2023-10-30T13:25:48.328717+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T13:25:48.328717+0000","last_undegraded":"2023-10-30T13:25:48.328717+0000","last_fullsized":"2023-10-30T13:25:48.328717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'779","last_scrub_stamp":"2023-10-30T13:25:48.328628+0000","last_deep_scrub":"85'779","last_deep_scrub_stamp":"2023-10-30T13:25:48.328628+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.328628+0000","objects_scrubbed":62,"log_size":779,"log_dups_size":0,"ondisk_log_size":779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:51.114462+0000","scrub_duration":0.22182811399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":635,"num_read_kb":434171,"num_write":906,"num_write_kb":88121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":675117,"num_omap_keys":1207,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"85'2216","reported_seq":4237,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:59.936763+0000","last_change":"2023-10-30T13:25:59.936829+0000","last_active":"2023-10-30T13:25:59.936763+0000","last_peered":"2023-10-30T13:25:59.936763+0000","last_clean":"2023-10-30T13:25:59.936763+0000","last_became_active":"2023-10-30T12:29:27.340504+0000","last_became_peered":"2023-10-30T12:29:27.340504+0000","last_unstale":"2023-10-30T13:25:59.936763+0000","last_undegraded":"2023-10-30T13:25:59.936763+0000","last_fullsized":"2023-10-30T13:25:59.936763+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2216","last_scrub_stamp":"2023-10-30T13:25:59.936683+0000","last_deep_scrub":"85'2216","last_deep_scrub_stamp":"2023-10-30T13:25:59.936683+0000","last_clean_scrub_stamp":"2023-10-30T13:25:59.936683+0000","objects_scrubbed":641,"log_size":2216,"log_dups_size":0,"ondisk_log_size":2216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:57:07.443321+0000","scrub_duration":1.7189143169999999,"objects_trimmed":0,"snaptrim_duration":0.00653923,"stat_sum":{"num_bytes":23737065,"num_objects":641,"num_object_clones":299,"num_object_copies":1282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":641,"num_whiteouts":238,"num_read":138,"num_read_kb":28255,"num_write":1351,"num_write_kb":33550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'766","reported_seq":1157,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:51.439924+0000","last_change":"2023-10-30T13:25:51.439991+0000","last_active":"2023-10-30T13:25:51.439924+0000","last_peered":"2023-10-30T13:25:51.439924+0000","last_clean":"2023-10-30T13:25:51.439924+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T13:25:51.439924+0000","last_undegraded":"2023-10-30T13:25:51.439924+0000","last_fullsized":"2023-10-30T13:25:51.439924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'766","last_scrub_stamp":"2023-10-30T13:25:51.439810+0000","last_deep_scrub":"85'766","last_deep_scrub_stamp":"2023-10-30T13:25:51.439810+0000","last_clean_scrub_stamp":"2023-10-30T13:25:51.439810+0000","objects_scrubbed":50,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:56.709508+0000","scrub_duration":0.13294180999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364369,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":493,"num_read_kb":320778,"num_write":891,"num_write_kb":71265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":522473,"num_omap_keys":1010,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"77'1599","reported_seq":3613,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.438900+0000","last_change":"2023-10-30T12:37:07.181841+0000","last_active":"2023-10-30T13:25:10.438900+0000","last_peered":"2023-10-30T13:25:10.438900+0000","last_clean":"2023-10-30T13:25:10.438900+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T13:25:10.438900+0000","last_undegraded":"2023-10-30T13:25:10.438900+0000","last_fullsized":"2023-10-30T13:25:10.438900+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1599,"log_dups_size":0,"ondisk_log_size":1599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0076673150000000001,"stat_sum":{"num_bytes":18256447,"num_objects":611,"num_object_clones":286,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":229,"num_read":149,"num_read_kb":24381,"num_write":1383,"num_write_kb":20195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"85'645","reported_seq":1007,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.161404+0000","last_change":"2023-10-30T13:25:48.161534+0000","last_active":"2023-10-30T13:25:48.161404+0000","last_peered":"2023-10-30T13:25:48.161404+0000","last_clean":"2023-10-30T13:25:48.161404+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T13:25:48.161404+0000","last_undegraded":"2023-10-30T13:25:48.161404+0000","last_fullsized":"2023-10-30T13:25:48.161404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'645","last_scrub_stamp":"2023-10-30T13:25:48.161279+0000","last_deep_scrub":"85'645","last_deep_scrub_stamp":"2023-10-30T13:25:48.161279+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.161279+0000","objects_scrubbed":38,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:29.280017+0000","scrub_duration":0.15056981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360710,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":462,"num_read_kb":342669,"num_write":689,"num_write_kb":66904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":355436,"num_omap_keys":678,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"82'1520","reported_seq":3503,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.232741+0000","last_change":"2023-10-30T12:37:07.182282+0000","last_active":"2023-10-30T13:25:16.232741+0000","last_peered":"2023-10-30T13:25:16.232741+0000","last_clean":"2023-10-30T13:25:16.232741+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T13:25:16.232741+0000","last_undegraded":"2023-10-30T13:25:16.232741+0000","last_fullsized":"2023-10-30T13:25:16.232741+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010286791999999999,"stat_sum":{"num_bytes":26944901,"num_objects":612,"num_object_clones":287,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":236,"num_read":119,"num_read_kb":34683,"num_write":1299,"num_write_kb":33129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"85'119","reported_seq":267,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:46.486940+0000","last_change":"2023-10-30T13:25:46.486986+0000","last_active":"2023-10-30T13:25:46.486940+0000","last_peered":"2023-10-30T13:25:46.486940+0000","last_clean":"2023-10-30T13:25:46.486940+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T13:25:46.486940+0000","last_undegraded":"2023-10-30T13:25:46.486940+0000","last_fullsized":"2023-10-30T13:25:46.486940+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'119","last_scrub_stamp":"2023-10-30T13:25:46.486855+0000","last_deep_scrub":"85'119","last_deep_scrub_stamp":"2023-10-30T13:25:46.486855+0000","last_clean_scrub_stamp":"2023-10-30T13:25:46.486855+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:13.395892+0000","scrub_duration":0.0083870119999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'1318","reported_seq":1819,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.077367+0000","last_change":"2023-10-30T12:28:44.059051+0000","last_active":"2023-10-30T13:25:03.077367+0000","last_peered":"2023-10-30T13:25:03.077367+0000","last_clean":"2023-10-30T13:25:03.077367+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T13:25:03.077367+0000","last_undegraded":"2023-10-30T13:25:03.077367+0000","last_fullsized":"2023-10-30T13:25:03.077367+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1318,"log_dups_size":0,"ondisk_log_size":1318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165890,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":877,"num_read_kb":231219,"num_write":1489,"num_write_kb":63137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"85'2200","reported_seq":4242,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.149503+0000","last_change":"2023-10-30T13:25:57.149588+0000","last_active":"2023-10-30T13:25:57.149503+0000","last_peered":"2023-10-30T13:25:57.149503+0000","last_clean":"2023-10-30T13:25:57.149503+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T13:25:57.149503+0000","last_undegraded":"2023-10-30T13:25:57.149503+0000","last_fullsized":"2023-10-30T13:25:57.149503+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2200","last_scrub_stamp":"2023-10-30T13:25:57.149395+0000","last_deep_scrub":"85'2200","last_deep_scrub_stamp":"2023-10-30T13:25:57.149395+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.149395+0000","objects_scrubbed":626,"log_size":2200,"log_dups_size":0,"ondisk_log_size":2200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:04.364636+0000","scrub_duration":1.0072585730000001,"objects_trimmed":0,"snaptrim_duration":0.0068470270000000003,"stat_sum":{"num_bytes":8097312,"num_objects":626,"num_object_clones":290,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":225,"num_read":146,"num_read_kb":2079,"num_write":1425,"num_write_kb":13192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"85'2234","reported_seq":2615,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.726414+0000","last_change":"2023-10-30T13:25:57.726488+0000","last_active":"2023-10-30T13:25:57.726414+0000","last_peered":"2023-10-30T13:25:57.726414+0000","last_clean":"2023-10-30T13:25:57.726414+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T13:25:57.726414+0000","last_undegraded":"2023-10-30T13:25:57.726414+0000","last_fullsized":"2023-10-30T13:25:57.726414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2234","last_scrub_stamp":"2023-10-30T13:25:57.726328+0000","last_deep_scrub":"85'2234","last_deep_scrub_stamp":"2023-10-30T13:25:57.726328+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.726328+0000","objects_scrubbed":42,"log_size":2234,"log_dups_size":0,"ondisk_log_size":2234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:51.655678+0000","scrub_duration":0.194142807,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":516,"num_read_kb":283293,"num_write":2369,"num_write_kb":72030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":392036,"num_omap_keys":649,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"77'1465","reported_seq":3488,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:02.278931+0000","last_change":"2023-10-30T12:37:07.112121+0000","last_active":"2023-10-30T13:25:02.278931+0000","last_peered":"2023-10-30T13:25:02.278931+0000","last_clean":"2023-10-30T13:25:02.278931+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T13:25:02.278931+0000","last_undegraded":"2023-10-30T13:25:02.278931+0000","last_fullsized":"2023-10-30T13:25:02.278931+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057755250000000001,"stat_sum":{"num_bytes":14269585,"num_objects":570,"num_object_clones":262,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":207,"num_read":132,"num_read_kb":18022,"num_write":1264,"num_write_kb":17917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'1898","reported_seq":3194,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:52.380541+0000","last_change":"2023-10-30T13:25:52.380604+0000","last_active":"2023-10-30T13:25:52.380541+0000","last_peered":"2023-10-30T13:25:52.380541+0000","last_clean":"2023-10-30T13:25:52.380541+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T13:25:52.380541+0000","last_undegraded":"2023-10-30T13:25:52.380541+0000","last_fullsized":"2023-10-30T13:25:52.380541+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1898","last_scrub_stamp":"2023-10-30T13:25:52.380459+0000","last_deep_scrub":"85'1898","last_deep_scrub_stamp":"2023-10-30T13:25:52.380459+0000","last_clean_scrub_stamp":"2023-10-30T13:25:52.380459+0000","objects_scrubbed":61,"log_size":1898,"log_dups_size":0,"ondisk_log_size":1898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:04.920583+0000","scrub_duration":0.247546554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49235352,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":1514,"num_read_kb":360508,"num_write":2035,"num_write_kb":92558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":705593,"num_omap_keys":1067,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"77'1546","reported_seq":3558,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.283401+0000","last_change":"2023-10-30T12:37:07.190318+0000","last_active":"2023-10-30T13:25:32.283401+0000","last_peered":"2023-10-30T13:25:32.283401+0000","last_clean":"2023-10-30T13:25:32.283401+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T13:25:32.283401+0000","last_undegraded":"2023-10-30T13:25:32.283401+0000","last_fullsized":"2023-10-30T13:25:32.283401+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015331312,"stat_sum":{"num_bytes":28178512,"num_objects":629,"num_object_clones":288,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":231,"num_read":148,"num_read_kb":41809,"num_write":1347,"num_write_kb":29520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'1196","reported_seq":1606,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.077319+0000","last_change":"2023-10-30T12:28:44.059806+0000","last_active":"2023-10-30T13:25:03.077319+0000","last_peered":"2023-10-30T13:25:03.077319+0000","last_clean":"2023-10-30T13:25:03.077319+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T13:25:03.077319+0000","last_undegraded":"2023-10-30T13:25:03.077319+0000","last_fullsized":"2023-10-30T13:25:03.077319+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1196,"log_dups_size":0,"ondisk_log_size":1196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":573,"num_read_kb":334248,"num_write":1338,"num_write_kb":75895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"85'1581","reported_seq":3582,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.209981+0000","last_change":"2023-10-30T12:37:07.178785+0000","last_active":"2023-10-30T13:25:32.209981+0000","last_peered":"2023-10-30T13:25:32.209981+0000","last_clean":"2023-10-30T13:25:32.209981+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T13:25:32.209981+0000","last_undegraded":"2023-10-30T13:25:32.209981+0000","last_fullsized":"2023-10-30T13:25:32.209981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1581,"log_dups_size":0,"ondisk_log_size":1581,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051193749999999998,"stat_sum":{"num_bytes":23443395,"num_objects":627,"num_object_clones":293,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":236,"num_read":141,"num_read_kb":26943,"num_write":1376,"num_write_kb":26009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"85'1517","reported_seq":1988,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.476829+0000","last_change":"2023-10-30T13:25:44.476908+0000","last_active":"2023-10-30T13:25:44.476829+0000","last_peered":"2023-10-30T13:25:44.476829+0000","last_clean":"2023-10-30T13:25:44.476829+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T13:25:44.476829+0000","last_undegraded":"2023-10-30T13:25:44.476829+0000","last_fullsized":"2023-10-30T13:25:44.476829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1517","last_scrub_stamp":"2023-10-30T13:25:44.476706+0000","last_deep_scrub":"85'1517","last_deep_scrub_stamp":"2023-10-30T13:25:44.476706+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.476706+0000","objects_scrubbed":50,"log_size":1517,"log_dups_size":0,"ondisk_log_size":1517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:43:12.189799+0000","scrub_duration":0.194542509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":699,"num_read_kb":404466,"num_write":1620,"num_write_kb":92575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":728614,"num_omap_keys":1399,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"85'2249","reported_seq":4286,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:50.476286+0000","last_change":"2023-10-30T13:25:50.476388+0000","last_active":"2023-10-30T13:25:50.476286+0000","last_peered":"2023-10-30T13:25:50.476286+0000","last_clean":"2023-10-30T13:25:50.476286+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T13:25:50.476286+0000","last_undegraded":"2023-10-30T13:25:50.476286+0000","last_fullsized":"2023-10-30T13:25:50.476286+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2249","last_scrub_stamp":"2023-10-30T13:25:50.476194+0000","last_deep_scrub":"85'2249","last_deep_scrub_stamp":"2023-10-30T13:25:50.476194+0000","last_clean_scrub_stamp":"2023-10-30T13:25:50.476194+0000","objects_scrubbed":637,"log_size":2249,"log_dups_size":0,"ondisk_log_size":2249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:30:58.348787+0000","scrub_duration":1.2919135900000001,"objects_trimmed":0,"snaptrim_duration":0.0039865179999999997,"stat_sum":{"num_bytes":21812744,"num_objects":637,"num_object_clones":298,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":235,"num_read":149,"num_read_kb":24572,"num_write":1399,"num_write_kb":31699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"85'1013","reported_seq":1423,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:18.079607+0000","last_change":"2023-10-30T12:28:44.063256+0000","last_active":"2023-10-30T13:25:18.079607+0000","last_peered":"2023-10-30T13:25:18.079607+0000","last_clean":"2023-10-30T13:25:18.079607+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T13:25:18.079607+0000","last_undegraded":"2023-10-30T13:25:18.079607+0000","last_fullsized":"2023-10-30T13:25:18.079607+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1013,"log_dups_size":0,"ondisk_log_size":1013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":534,"num_read_kb":318720,"num_write":1164,"num_write_kb":75805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"77'1542","reported_seq":3568,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.372387+0000","last_change":"2023-10-30T12:37:07.114718+0000","last_active":"2023-10-30T13:25:32.372387+0000","last_peered":"2023-10-30T13:25:32.372387+0000","last_clean":"2023-10-30T13:25:32.372387+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T13:25:32.372387+0000","last_undegraded":"2023-10-30T13:25:32.372387+0000","last_fullsized":"2023-10-30T13:25:32.372387+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1542,"log_dups_size":0,"ondisk_log_size":1542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0070372949999999998,"stat_sum":{"num_bytes":38977850,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":222,"num_read":156,"num_read_kb":47793,"num_write":1351,"num_write_kb":41194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1184","reported_seq":1628,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:50.553950+0000","last_change":"2023-10-30T13:25:50.554036+0000","last_active":"2023-10-30T13:25:50.553950+0000","last_peered":"2023-10-30T13:25:50.553950+0000","last_clean":"2023-10-30T13:25:50.553950+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T13:25:50.553950+0000","last_undegraded":"2023-10-30T13:25:50.553950+0000","last_fullsized":"2023-10-30T13:25:50.553950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1184","last_scrub_stamp":"2023-10-30T13:25:50.553862+0000","last_deep_scrub":"85'1184","last_deep_scrub_stamp":"2023-10-30T13:25:50.553862+0000","last_clean_scrub_stamp":"2023-10-30T13:25:50.553862+0000","objects_scrubbed":47,"log_size":1184,"log_dups_size":0,"ondisk_log_size":1184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:34.636484+0000","scrub_duration":0.480914603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":644,"num_read_kb":249397,"num_write":1860,"num_write_kb":63729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":781147,"num_omap_keys":1113,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"77'1449","reported_seq":3447,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:07.124077+0000","last_change":"2023-10-30T12:37:07.115578+0000","last_active":"2023-10-30T13:25:07.124077+0000","last_peered":"2023-10-30T13:25:07.124077+0000","last_clean":"2023-10-30T13:25:07.124077+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T13:25:07.124077+0000","last_undegraded":"2023-10-30T13:25:07.124077+0000","last_fullsized":"2023-10-30T13:25:07.124077+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1449,"log_dups_size":0,"ondisk_log_size":1449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0074996079999999996,"stat_sum":{"num_bytes":14238343,"num_objects":603,"num_object_clones":280,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":229,"num_read":126,"num_read_kb":9383,"num_write":1239,"num_write_kb":19514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"85'490","reported_seq":840,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:35.265904+0000","last_change":"2023-10-30T12:28:44.058663+0000","last_active":"2023-10-30T13:25:35.265904+0000","last_peered":"2023-10-30T13:25:35.265904+0000","last_clean":"2023-10-30T13:25:35.265904+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T13:25:35.265904+0000","last_undegraded":"2023-10-30T13:25:35.265904+0000","last_fullsized":"2023-10-30T13:25:35.265904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21146631,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":514,"num_read_kb":197148,"num_write":651,"num_write_kb":42826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"85'2296","reported_seq":4306,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:58.407563+0000","last_change":"2023-10-30T13:25:58.407621+0000","last_active":"2023-10-30T13:25:58.407563+0000","last_peered":"2023-10-30T13:25:58.407563+0000","last_clean":"2023-10-30T13:25:58.407563+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T13:25:58.407563+0000","last_undegraded":"2023-10-30T13:25:58.407563+0000","last_fullsized":"2023-10-30T13:25:58.407563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2296","last_scrub_stamp":"2023-10-30T13:25:58.407437+0000","last_deep_scrub":"85'2296","last_deep_scrub_stamp":"2023-10-30T13:25:58.407437+0000","last_clean_scrub_stamp":"2023-10-30T13:25:58.407437+0000","objects_scrubbed":647,"log_size":2296,"log_dups_size":0,"ondisk_log_size":2296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:12.972219+0000","scrub_duration":1.296777326,"objects_trimmed":0,"snaptrim_duration":0.0085509929999999998,"stat_sum":{"num_bytes":14746405,"num_objects":647,"num_object_clones":344,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":203,"num_read":167,"num_read_kb":6583,"num_write":1369,"num_write_kb":17013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'916","reported_seq":1240,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:51.295504+0000","last_change":"2023-10-30T13:25:51.295571+0000","last_active":"2023-10-30T13:25:51.295504+0000","last_peered":"2023-10-30T13:25:51.295504+0000","last_clean":"2023-10-30T13:25:51.295504+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T13:25:51.295504+0000","last_undegraded":"2023-10-30T13:25:51.295504+0000","last_fullsized":"2023-10-30T13:25:51.295504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'916","last_scrub_stamp":"2023-10-30T13:25:51.295393+0000","last_deep_scrub":"85'916","last_deep_scrub_stamp":"2023-10-30T13:25:51.295393+0000","last_clean_scrub_stamp":"2023-10-30T13:25:51.295393+0000","objects_scrubbed":46,"log_size":916,"log_dups_size":0,"ondisk_log_size":916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:13:46.299260+0000","scrub_duration":0.1008542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":472,"num_read_kb":149352,"num_write":1030,"num_write_kb":38491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":548946,"num_omap_keys":1000,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"85'2069","reported_seq":4117,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.461766+0000","last_change":"2023-10-30T13:25:53.461883+0000","last_active":"2023-10-30T13:25:53.461766+0000","last_peered":"2023-10-30T13:25:53.461766+0000","last_clean":"2023-10-30T13:25:53.461766+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T13:25:53.461766+0000","last_undegraded":"2023-10-30T13:25:53.461766+0000","last_fullsized":"2023-10-30T13:25:53.461766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2069","last_scrub_stamp":"2023-10-30T13:25:53.461661+0000","last_deep_scrub":"85'2069","last_deep_scrub_stamp":"2023-10-30T13:25:53.461661+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.461661+0000","objects_scrubbed":584,"log_size":2069,"log_dups_size":0,"ondisk_log_size":2069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:51.529626+0000","scrub_duration":1.0309153659999999,"objects_trimmed":0,"snaptrim_duration":0.0096369869999999996,"stat_sum":{"num_bytes":18113290,"num_objects":584,"num_object_clones":270,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":219,"num_read":134,"num_read_kb":20299,"num_write":1266,"num_write_kb":19768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"85'826","reported_seq":1199,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:49.233063+0000","last_change":"2023-10-30T13:25:49.233269+0000","last_active":"2023-10-30T13:25:49.233063+0000","last_peered":"2023-10-30T13:25:49.233063+0000","last_clean":"2023-10-30T13:25:49.233063+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T13:25:49.233063+0000","last_undegraded":"2023-10-30T13:25:49.233063+0000","last_fullsized":"2023-10-30T13:25:49.233063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'826","last_scrub_stamp":"2023-10-30T13:25:49.232843+0000","last_deep_scrub":"85'826","last_deep_scrub_stamp":"2023-10-30T13:25:49.232843+0000","last_clean_scrub_stamp":"2023-10-30T13:25:49.232843+0000","objects_scrubbed":49,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:29:29.495169+0000","scrub_duration":0.14346297999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777306,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":495,"num_read_kb":170685,"num_write":965,"num_write_kb":43434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":641331,"num_omap_keys":1102,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"85'2182","reported_seq":4222,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.248973+0000","last_change":"2023-10-30T13:25:53.249016+0000","last_active":"2023-10-30T13:25:53.248973+0000","last_peered":"2023-10-30T13:25:53.248973+0000","last_clean":"2023-10-30T13:25:53.248973+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T13:25:53.248973+0000","last_undegraded":"2023-10-30T13:25:53.248973+0000","last_fullsized":"2023-10-30T13:25:53.248973+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2182","last_scrub_stamp":"2023-10-30T13:25:53.248885+0000","last_deep_scrub":"85'2182","last_deep_scrub_stamp":"2023-10-30T13:25:53.248885+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.248885+0000","objects_scrubbed":615,"log_size":2182,"log_dups_size":0,"ondisk_log_size":2182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:04:59.062499+0000","scrub_duration":1.1234317469999999,"objects_trimmed":0,"snaptrim_duration":0.007081922,"stat_sum":{"num_bytes":14751960,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":228,"num_read":137,"num_read_kb":2516,"num_write":1376,"num_write_kb":21368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1177","reported_seq":1676,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:47.247326+0000","last_change":"2023-10-30T13:25:47.247437+0000","last_active":"2023-10-30T13:25:47.247326+0000","last_peered":"2023-10-30T13:25:47.247326+0000","last_clean":"2023-10-30T13:25:47.247326+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T13:25:47.247326+0000","last_undegraded":"2023-10-30T13:25:47.247326+0000","last_fullsized":"2023-10-30T13:25:47.247326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1177","last_scrub_stamp":"2023-10-30T13:25:47.247175+0000","last_deep_scrub":"85'1177","last_deep_scrub_stamp":"2023-10-30T13:25:47.247175+0000","last_clean_scrub_stamp":"2023-10-30T13:25:47.247175+0000","objects_scrubbed":52,"log_size":1177,"log_dups_size":0,"ondisk_log_size":1177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:36.256311+0000","scrub_duration":0.26537599099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":763,"num_read_kb":345628,"num_write":1535,"num_write_kb":76701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":665341,"num_omap_keys":1170,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"85'2030","reported_seq":4076,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:56.194710+0000","last_change":"2023-10-30T13:25:56.194771+0000","last_active":"2023-10-30T13:25:56.194710+0000","last_peered":"2023-10-30T13:25:56.194710+0000","last_clean":"2023-10-30T13:25:56.194710+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T13:25:56.194710+0000","last_undegraded":"2023-10-30T13:25:56.194710+0000","last_fullsized":"2023-10-30T13:25:56.194710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2030","last_scrub_stamp":"2023-10-30T13:25:56.194609+0000","last_deep_scrub":"85'2030","last_deep_scrub_stamp":"2023-10-30T13:25:56.194609+0000","last_clean_scrub_stamp":"2023-10-30T13:25:56.194609+0000","objects_scrubbed":570,"log_size":2030,"log_dups_size":0,"ondisk_log_size":2030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:38.396542+0000","scrub_duration":0.95094084300000004,"objects_trimmed":0,"snaptrim_duration":0.0060337460000000004,"stat_sum":{"num_bytes":23618154,"num_objects":570,"num_object_clones":263,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":209,"num_read":127,"num_read_kb":29654,"num_write":1265,"num_write_kb":27771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"85'1182","reported_seq":1586,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:03.077348+0000","last_change":"2023-10-30T12:28:44.058801+0000","last_active":"2023-10-30T13:25:03.077348+0000","last_peered":"2023-10-30T13:25:03.077348+0000","last_clean":"2023-10-30T13:25:03.077348+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T13:25:03.077348+0000","last_undegraded":"2023-10-30T13:25:03.077348+0000","last_fullsized":"2023-10-30T13:25:03.077348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:43.012501+0000","last_clean_scrub_stamp":"2023-10-30T12:28:43.012501+0000","objects_scrubbed":0,"log_size":1182,"log_dups_size":0,"ondisk_log_size":1182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":538,"num_read_kb":321476,"num_write":1330,"num_write_kb":79987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"85'1546","reported_seq":3552,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:23.080184+0000","last_change":"2023-10-30T12:37:07.180393+0000","last_active":"2023-10-30T13:25:23.080184+0000","last_peered":"2023-10-30T13:25:23.080184+0000","last_clean":"2023-10-30T13:25:23.080184+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T13:25:23.080184+0000","last_undegraded":"2023-10-30T13:25:23.080184+0000","last_fullsized":"2023-10-30T13:25:23.080184+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091395910000000007,"stat_sum":{"num_bytes":24394000,"num_objects":612,"num_object_clones":279,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":219,"num_read":133,"num_read_kb":28881,"num_write":1364,"num_write_kb":25892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"77'1527","reported_seq":3551,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.265683+0000","last_change":"2023-10-30T12:37:07.182718+0000","last_active":"2023-10-30T13:25:32.265683+0000","last_peered":"2023-10-30T13:25:32.265683+0000","last_clean":"2023-10-30T13:25:32.265683+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T13:25:32.265683+0000","last_undegraded":"2023-10-30T13:25:32.265683+0000","last_fullsized":"2023-10-30T13:25:32.265683+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1527,"log_dups_size":0,"ondisk_log_size":1527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010120377999999999,"stat_sum":{"num_bytes":19538275,"num_objects":582,"num_object_clones":275,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":200,"num_read":151,"num_read_kb":26132,"num_write":1354,"num_write_kb":27822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"85'931","reported_seq":1389,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.318497+0000","last_change":"2023-10-30T13:25:44.318568+0000","last_active":"2023-10-30T13:25:44.318497+0000","last_peered":"2023-10-30T13:25:44.318497+0000","last_clean":"2023-10-30T13:25:44.318497+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T13:25:44.318497+0000","last_undegraded":"2023-10-30T13:25:44.318497+0000","last_fullsized":"2023-10-30T13:25:44.318497+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'931","last_scrub_stamp":"2023-10-30T13:25:44.318392+0000","last_deep_scrub":"85'931","last_deep_scrub_stamp":"2023-10-30T13:25:44.318392+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.318392+0000","objects_scrubbed":51,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:06.786185+0000","scrub_duration":0.105541015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":739,"num_read_kb":197443,"num_write":1077,"num_write_kb":51357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":687759,"num_omap_keys":1211,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"82'1642","reported_seq":3638,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.234615+0000","last_change":"2023-10-30T12:37:07.102323+0000","last_active":"2023-10-30T13:25:32.234615+0000","last_peered":"2023-10-30T13:25:32.234615+0000","last_clean":"2023-10-30T13:25:32.234615+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T13:25:32.234615+0000","last_undegraded":"2023-10-30T13:25:32.234615+0000","last_fullsized":"2023-10-30T13:25:32.234615+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011628750000000001,"stat_sum":{"num_bytes":22561655,"num_objects":686,"num_object_clones":314,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":247,"num_read":162,"num_read_kb":29557,"num_write":1441,"num_write_kb":28808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"85'1103","reported_seq":1654,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:55.242759+0000","last_change":"2023-10-30T13:25:55.242833+0000","last_active":"2023-10-30T13:25:55.242759+0000","last_peered":"2023-10-30T13:25:55.242759+0000","last_clean":"2023-10-30T13:25:55.242759+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T13:25:55.242759+0000","last_undegraded":"2023-10-30T13:25:55.242759+0000","last_fullsized":"2023-10-30T13:25:55.242759+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1103","last_scrub_stamp":"2023-10-30T13:25:55.242662+0000","last_deep_scrub":"85'1103","last_deep_scrub_stamp":"2023-10-30T13:25:55.242662+0000","last_clean_scrub_stamp":"2023-10-30T13:25:55.242662+0000","objects_scrubbed":55,"log_size":1103,"log_dups_size":0,"ondisk_log_size":1103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:05:46.724096+0000","scrub_duration":0.11943021299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":939,"num_read_kb":433966,"num_write":1214,"num_write_kb":88022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":602554,"num_omap_keys":1058,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"82'1531","reported_seq":3514,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.227278+0000","last_change":"2023-10-30T12:37:07.189494+0000","last_active":"2023-10-30T13:25:32.227278+0000","last_peered":"2023-10-30T13:25:32.227278+0000","last_clean":"2023-10-30T13:25:32.227278+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T13:25:32.227278+0000","last_undegraded":"2023-10-30T13:25:32.227278+0000","last_fullsized":"2023-10-30T13:25:32.227278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1531,"log_dups_size":0,"ondisk_log_size":1531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014632805,"stat_sum":{"num_bytes":14611265,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":226,"num_read":123,"num_read_kb":17907,"num_write":1326,"num_write_kb":16683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"85'807","reported_seq":1195,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:50.519115+0000","last_change":"2023-10-30T13:25:50.519195+0000","last_active":"2023-10-30T13:25:50.519115+0000","last_peered":"2023-10-30T13:25:50.519115+0000","last_clean":"2023-10-30T13:25:50.519115+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T13:25:50.519115+0000","last_undegraded":"2023-10-30T13:25:50.519115+0000","last_fullsized":"2023-10-30T13:25:50.519115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'807","last_scrub_stamp":"2023-10-30T13:25:50.519025+0000","last_deep_scrub":"85'807","last_deep_scrub_stamp":"2023-10-30T13:25:50.519025+0000","last_clean_scrub_stamp":"2023-10-30T13:25:50.519025+0000","objects_scrubbed":54,"log_size":807,"log_dups_size":0,"ondisk_log_size":807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:52:13.369063+0000","scrub_duration":0.17624699299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":488,"num_read_kb":379559,"num_write":896,"num_write_kb":83880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":513377,"num_omap_keys":969,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"77'1481","reported_seq":3498,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.207068+0000","last_change":"2023-10-30T12:37:07.191451+0000","last_active":"2023-10-30T13:25:32.207068+0000","last_peered":"2023-10-30T13:25:32.207068+0000","last_clean":"2023-10-30T13:25:32.207068+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T13:25:32.207068+0000","last_undegraded":"2023-10-30T13:25:32.207068+0000","last_fullsized":"2023-10-30T13:25:32.207068+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1481,"log_dups_size":0,"ondisk_log_size":1481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015871881000000001,"stat_sum":{"num_bytes":15563160,"num_objects":581,"num_object_clones":268,"num_object_copies":1162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":208,"num_read":133,"num_read_kb":9744,"num_write":1268,"num_write_kb":20904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"85'885","reported_seq":1253,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:51.286953+0000","last_change":"2023-10-30T13:25:51.287081+0000","last_active":"2023-10-30T13:25:51.286953+0000","last_peered":"2023-10-30T13:25:51.286953+0000","last_clean":"2023-10-30T13:25:51.286953+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T13:25:51.286953+0000","last_undegraded":"2023-10-30T13:25:51.286953+0000","last_fullsized":"2023-10-30T13:25:51.286953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'885","last_scrub_stamp":"2023-10-30T13:25:51.286817+0000","last_deep_scrub":"85'885","last_deep_scrub_stamp":"2023-10-30T13:25:51.286817+0000","last_clean_scrub_stamp":"2023-10-30T13:25:51.286817+0000","objects_scrubbed":47,"log_size":885,"log_dups_size":0,"ondisk_log_size":885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:21.615461+0000","scrub_duration":0.19579813099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33555014,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":517,"num_read_kb":175570,"num_write":978,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":553400,"num_omap_keys":904,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165214918,"num_objects":39301,"num_object_clones":18317,"num_object_copies":78602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39301,"num_whiteouts":14249,"num_read":9267,"num_read_kb":1285569,"num_write":86732,"num_write_kb":1459764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424127488,"data_stored":2332445068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116476,"ondisk_log_size":116476,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":903126807,"num_objects":1657,"num_object_clones":0,"num_object_copies":3314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":21550,"num_read_kb":9552846,"num_write":41306,"num_write_kb":2210416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":15733835,"num_omap_keys":26734,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1806409728,"data_stored":1806253614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36062,"ondisk_log_size":36062,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477430,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":833148,"kb_used_data":639604,"kb_used_omap":1,"kb_used_meta":193534,"kb_avail":93538692,"statfs":{"total":96636764160,"available":95783620608,"internally_reserved":0,"allocated":654954496,"data_stored":640799640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":198179272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42699999999999999,"15min":0.434},"min":{"1min":0.253,"5min":0.253,"15min":0.23200000000000001},"max":{"1min":0.52300000000000002,"5min":0.66900000000000004,"15min":0.94599999999999995},"last":0.247},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.47099999999999997,"15min":0.46999999999999997},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.27700000000000002},"max":{"1min":0.68300000000000005,"5min":0.75800000000000001,"15min":0.81899999999999995},"last":0.63900000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.503,"15min":0.495},"min":{"1min":0.39000000000000001,"5min":0.33300000000000002,"15min":0.307},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":1.0209999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.52700000000000002},"min":{"1min":0.36899999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.76600000000000001,"5min":0.78400000000000003,"15min":0.99299999999999999},"last":0.42099999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.51500000000000001,"15min":0.502},"min":{"1min":0.373,"5min":0.34100000000000003,"15min":0.28599999999999998},"max":{"1min":0.747,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.56000000000000005,"15min":0.55200000000000005},"min":{"1min":0.42999999999999999,"5min":0.38600000000000001,"15min":0.33200000000000002},"max":{"1min":0.79700000000000004,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.61399999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.77900000000000003,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.94299999999999995,"5min":1.05,"15min":1.548},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.63900000000000001,"5min":0.497,"15min":0.497},"max":{"1min":1.127,"5min":1.149,"15min":1.6950000000000001},"last":0.66900000000000004}]},{"osd":5,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.78900000000000003,"15min":0.81399999999999995},"min":{"1min":0.57299999999999995,"5min":0.52800000000000002,"15min":0.496},"max":{"1min":0.96399999999999997,"5min":1.1160000000000001,"15min":1.194},"last":0.5},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.80000000000000004,"15min":0.80700000000000005},"min":{"1min":0.66100000000000003,"5min":0.63200000000000001,"15min":0.52100000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.603},"last":0.71399999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.64000000000000001,"5min":0.56999999999999995,"15min":0.436},"max":{"1min":1.113,"5min":1.198,"15min":1.2270000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.93100000000000005,"15min":0.92300000000000004},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.55600000000000005},"max":{"1min":1.145,"5min":1.202,"15min":1.3759999999999999},"last":0.53900000000000003}]},{"osd":7,"last update":"Mon Oct 30 13:25:13 2023","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.84799999999999998,"15min":0.83599999999999997},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.46400000000000002},"max":{"1min":1.133,"5min":1.133,"15min":1.1919999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.91400000000000003,"15min":0.877},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.47699999999999998},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.2050000000000001},"last":0.69499999999999995}]}]},{"osd":2,"up_from":16,"seq":68719477431,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556284,"kb_used_data":412724,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93815556,"statfs":{"total":96636764160,"available":96067129344,"internally_reserved":0,"allocated":422629376,"data_stored":409295067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146995654},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42999999999999999,"15min":0.43099999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57799999999999996,"5min":0.81299999999999994,"15min":0.91200000000000003},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48799999999999999,"15min":0.47899999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.625,"5min":1.554,"15min":2.8519999999999999},"last":0.66900000000000004}]},{"osd":1,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47799999999999998,"15min":0.497},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.57199999999999995,"5min":1.071,"15min":4.9660000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52700000000000002,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.32100000000000001,"15min":0.28399999999999997},"max":{"1min":0.66600000000000004,"5min":1.198,"15min":6.0759999999999996},"last":0.74099999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.505,"15min":0.52800000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.314},"max":{"1min":0.65500000000000003,"5min":0.96799999999999997,"15min":4.6120000000000001},"last":0.63},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56499999999999995,"15min":0.58699999999999997},"min":{"1min":0.42699999999999999,"5min":0.40899999999999997,"15min":0.379},"max":{"1min":0.68200000000000005,"5min":1.4430000000000001,"15min":4.641},"last":0.72399999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79100000000000004,"15min":0.80400000000000005},"min":{"1min":0.58399999999999996,"5min":0.48299999999999998,"15min":0.47999999999999998},"max":{"1min":0.95999999999999996,"5min":1.1719999999999999,"15min":3.5499999999999998},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77000000000000002,"15min":0.81000000000000005},"min":{"1min":0.53200000000000003,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0389999999999999,"5min":1.262,"15min":5.032},"last":0.68400000000000005}]},{"osd":5,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.88800000000000001,"15min":0.87},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.177,"5min":1.494,"15min":6.1289999999999996},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.84699999999999998,"15min":0.877},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.159,"5min":1.3440000000000001,"15min":3.1459999999999999},"last":0.70299999999999996}]},{"osd":6,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.83299999999999996,"15min":0.875},"min":{"1min":0.67600000000000005,"5min":0.54000000000000004,"15min":0.53100000000000003},"max":{"1min":1.0469999999999999,"5min":1.1739999999999999,"15min":5.8159999999999998},"last":0.621},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.82799999999999996,"15min":0.83299999999999996},"min":{"1min":0.66600000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.99299999999999999,"5min":1.6299999999999999,"15min":3.415},"last":0.65200000000000002}]},{"osd":7,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.84199999999999997,"15min":0.872},"min":{"1min":0.73999999999999999,"5min":0.60699999999999998,"15min":0.51100000000000001},"max":{"1min":1.121,"5min":1.9139999999999999,"15min":4.883},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.88400000000000001,"15min":0.90300000000000002},"min":{"1min":0.79300000000000004,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0640000000000001,"5min":1.3460000000000001,"15min":4.9950000000000001},"last":0.75800000000000001}]}]},{"osd":1,"up_from":16,"seq":68719477430,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538984,"kb_used_data":431520,"kb_used_omap":1,"kb_used_meta":107454,"kb_avail":93832856,"statfs":{"total":96636764160,"available":96084844544,"internally_reserved":0,"allocated":441876480,"data_stored":431229668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110033350},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.46500000000000002,"15min":0.44800000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.22600000000000001},"max":{"1min":0.63400000000000001,"5min":0.70699999999999996,"15min":0.80400000000000005},"last":0.438},{"interface":"front","average":{"1min":0.505,"5min":0.54200000000000004,"15min":0.52800000000000002},"min":{"1min":0.40899999999999997,"5min":0.31,"15min":0.253},"max":{"1min":0.72499999999999998,"5min":0.91100000000000003,"15min":1.1140000000000001},"last":0.27500000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47999999999999998,"15min":0.48999999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68200000000000005,"5min":0.83999999999999997,"15min":5.6120000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56499999999999995,"15min":0.55700000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.34699999999999998},"max":{"1min":0.78200000000000003,"5min":0.82199999999999995,"15min":2.831},"last":0.40799999999999997}]},{"osd":3,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47499999999999998,"15min":0.496},"min":{"1min":0.32600000000000001,"5min":0.318,"15min":0.30499999999999999},"max":{"1min":0.60799999999999998,"5min":0.80900000000000005,"15min":7.6900000000000004},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.39600000000000002,"15min":0.34999999999999998},"max":{"1min":0.66700000000000004,"5min":0.91800000000000004,"15min":1.5309999999999999},"last":0.47999999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84599999999999997,"15min":0.81999999999999995},"min":{"1min":0.58099999999999996,"5min":0.51600000000000001,"15min":0.40400000000000003},"max":{"1min":1.0740000000000001,"5min":1.133,"15min":5.4790000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.90000000000000002,"15min":0.85999999999999999},"min":{"1min":0.72799999999999998,"5min":0.64300000000000002,"15min":0.44400000000000001},"max":{"1min":1.113,"5min":1.1619999999999999,"15min":5.5110000000000001},"last":0.53500000000000003}]},{"osd":5,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.82899999999999996,"15min":0.84099999999999997},"min":{"1min":0.56299999999999994,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1160000000000001,"5min":1.212,"15min":7.5430000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.89600000000000002,"15min":0.91500000000000004},"min":{"1min":0.73599999999999999,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.141,"5min":1.2350000000000001,"15min":7.5750000000000002},"last":0.65900000000000003}]},{"osd":6,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79100000000000004,"15min":0.79800000000000004},"min":{"1min":0.59399999999999997,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.875,"5min":1.177,"15min":5.6639999999999997},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.80700000000000005,"15min":0.86399999999999999},"min":{"1min":0.64000000000000001,"5min":0.51800000000000002,"15min":0.51600000000000001},"max":{"1min":1.0449999999999999,"5min":1.1140000000000001,"15min":13.036},"last":0.97699999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.84899999999999998,"15min":0.87},"min":{"1min":0.58399999999999996,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.0960000000000001,"5min":1.2050000000000001,"15min":3.7690000000000001},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.88900000000000001,"15min":0.874},"min":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.501},"max":{"1min":1.0109999999999999,"5min":1.1870000000000001,"15min":4.0659999999999998},"last":0.60999999999999999}]}]},{"osd":0,"up_from":16,"seq":68719477431,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":947656,"kb_used_data":695808,"kb_used_omap":1,"kb_used_meta":251838,"kb_avail":93424184,"statfs":{"total":96636764160,"available":95666364416,"internally_reserved":0,"allocated":712507392,"data_stored":696810667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":257882566},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40600000000000003,"15min":0.41099999999999998},"min":{"1min":0.254,"5min":0.23899999999999999,"15min":0.22700000000000001},"max":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":3.996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51700000000000002,"15min":0.52600000000000002},"min":{"1min":0.41099999999999998,"5min":0.36199999999999999,"15min":0.309},"max":{"1min":0.67600000000000005,"5min":0.75600000000000001,"15min":4.3230000000000004},"last":0.496}]},{"osd":2,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.27700000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":4.9690000000000003},"last":0.31},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52100000000000002,"15min":0.53200000000000003},"min":{"1min":0.372,"5min":0.34100000000000003,"15min":0.30199999999999999},"max":{"1min":0.621,"5min":0.878,"15min":5.0049999999999999},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.48799999999999999},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.28299999999999997},"max":{"1min":0.68100000000000005,"5min":0.82699999999999996,"15min":5.4909999999999997},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57499999999999996,"15min":0.58099999999999996},"min":{"1min":0.443,"5min":0.42799999999999999,"15min":0.38600000000000001},"max":{"1min":0.749,"5min":0.90400000000000003,"15min":5.6970000000000001},"last":0.58999999999999997}]},{"osd":4,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.58699999999999997,"5min":0.46700000000000003,"15min":0.377},"max":{"1min":1.111,"5min":1.111,"15min":4.444},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.80100000000000005},"min":{"1min":0.58499999999999996,"5min":0.55000000000000004,"15min":0.48099999999999998},"max":{"1min":0.90500000000000003,"5min":1.117,"15min":5.7750000000000004},"last":0.93500000000000005}]},{"osd":5,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.83299999999999996,"15min":0.83499999999999996},"min":{"1min":0.66800000000000004,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.246,"5min":1.3340000000000001,"15min":4.5430000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.871,"15min":0.86399999999999999},"min":{"1min":0.59199999999999997,"5min":0.56999999999999995,"15min":0.47099999999999997},"max":{"1min":1.204,"5min":1.204,"15min":4.4790000000000001},"last":0.54900000000000004}]},{"osd":6,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.82599999999999996,"15min":0.84199999999999997},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":1.397,"15min":4.5129999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.88500000000000001,"15min":0.89900000000000002},"min":{"1min":0.626,"5min":0.626,"15min":0.56699999999999995},"max":{"1min":1.1699999999999999,"5min":1.3120000000000001,"15min":4.5750000000000002},"last":0.76200000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.875,"15min":0.86799999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.55700000000000005},"max":{"1min":1.0669999999999999,"5min":1.206,"15min":4.3979999999999997},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.88900000000000001,"15min":0.91400000000000003},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.53700000000000003},"max":{"1min":1.0760000000000001,"5min":1.2210000000000001,"15min":5.375},"last":0.51600000000000001}]}]},{"osd":6,"up_from":15,"seq":64424510133,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":538924,"kb_used_data":409252,"kb_used_omap":1,"kb_used_meta":129662,"kb_avail":93832916,"statfs":{"total":96636764160,"available":96084905984,"internally_reserved":0,"allocated":419074048,"data_stored":408471323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":132774343},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.85199999999999998,"15min":0.85599999999999998},"min":{"1min":0.62,"5min":0.49399999999999999,"15min":0.34200000000000003},"max":{"1min":1.103,"5min":1.2470000000000001,"15min":1.4730000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.85099999999999998,"15min":0.95099999999999996},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2130000000000001,"5min":1.3149999999999999,"15min":1.415},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.79300000000000004,"15min":0.79700000000000004},"min":{"1min":0.55700000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.262,"5min":1.262,"15min":1.399},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.86699999999999999,"15min":0.86299999999999999},"min":{"1min":0.62,"5min":0.52100000000000002,"15min":0.47699999999999998},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.49299999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.995,"15min":0.96699999999999997},"min":{"1min":0.80300000000000005,"5min":0.68300000000000005,"15min":0.47199999999999998},"max":{"1min":1.375,"5min":1.496,"15min":1.496},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.877,"5min":0.90100000000000002,"15min":0.94199999999999995},"min":{"1min":0.60299999999999998,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1839999999999999,"5min":1.3839999999999999,"15min":1.431},"last":0.32900000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.88600000000000001,"15min":0.88900000000000001},"min":{"1min":0.65500000000000003,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.2589999999999999,"5min":1.262,"15min":1.4530000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":0.88900000000000001},"min":{"1min":0.77600000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.347,"5min":1.347,"15min":1.478},"last":0.59899999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97299999999999998,"15min":0.93799999999999994},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.2849999999999999,"5min":1.3220000000000001,"15min":1.4019999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.093,"15min":1.046},"min":{"1min":0.80000000000000004,"5min":0.73899999999999999,"15min":0.54700000000000004},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.73699999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.163,"15min":1.125},"min":{"1min":0.90100000000000002,"5min":0.82999999999999996,"15min":0.73399999999999999},"max":{"1min":1.417,"5min":1.4730000000000001,"15min":1.583},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.069,"15min":1.095},"min":{"1min":0.625,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2829999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":0.66600000000000004}]},{"osd":7,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.081,"15min":1.0800000000000001},"min":{"1min":0.86099999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.4970000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.71499999999999997},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.22,"15min":1.2170000000000001},"min":{"1min":1.034,"5min":0.76500000000000001,"15min":0.64200000000000002},"max":{"1min":1.804,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":0.76400000000000001}]}]},{"osd":4,"up_from":15,"seq":64424510136,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":665248,"kb_used_data":493912,"kb_used_omap":1,"kb_used_meta":171326,"kb_avail":93706592,"statfs":{"total":96636764160,"available":95955550208,"internally_reserved":0,"allocated":505765888,"data_stored":496228056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":175438279},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,4,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.86299999999999999,"15min":0.83899999999999997},"min":{"1min":0.54800000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.1930000000000001,"5min":1.218,"15min":1.292},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.86899999999999999,"15min":0.84699999999999998},"min":{"1min":0.57399999999999995,"5min":0.52700000000000002,"15min":0.52600000000000002},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.369},"last":0.504}]},{"osd":1,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.79400000000000004,"15min":0.78600000000000003},"min":{"1min":0.60799999999999998,"5min":0.47899999999999998,"15min":0.47499999999999998},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1839999999999999},"last":1.732},{"interface":"front","average":{"1min":0.879,"5min":0.83699999999999997,"15min":0.83199999999999996},"min":{"1min":0.60899999999999999,"5min":0.53000000000000003,"15min":0.47299999999999998},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.3100000000000001},"last":1.8220000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.89500000000000002},"min":{"1min":0.56599999999999995,"5min":0.55500000000000005,"15min":0.55300000000000005},"max":{"1min":1.1899999999999999,"5min":1.2270000000000001,"15min":1.292},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.998,"5min":0.94599999999999995,"15min":0.94699999999999995},"min":{"1min":0.74099999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.71699999999999997}]},{"osd":3,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.97399999999999998,"15min":0.95299999999999996},"min":{"1min":0.68500000000000005,"5min":0.65100000000000002,"15min":0.52200000000000002},"max":{"1min":1.286,"5min":1.286,"15min":1.514},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.94699999999999995,"15min":0.96299999999999997},"min":{"1min":0.76600000000000001,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.82899999999999996}]},{"osd":5,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.98299999999999998,"15min":1.0129999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60299999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":2.4430000000000001},"last":0.88},{"interface":"front","average":{"1min":1.171,"5min":1.1140000000000001,"15min":1.0660000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.8460000000000001},"last":1.151}]},{"osd":6,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.145,"15min":1.1559999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.419,"5min":1.425,"15min":1.677},"last":1.546},{"interface":"front","average":{"1min":1.306,"5min":1.28,"15min":1.1830000000000001},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.64400000000000002},"max":{"1min":1.528,"5min":1.5409999999999999,"15min":1.9610000000000001},"last":1.629}]},{"osd":7,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.0529999999999999,"15min":1.1359999999999999},"min":{"1min":0.89500000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.8700000000000001},"last":1.524},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.202,"15min":1.266},"min":{"1min":1.016,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.538,"5min":1.591,"15min":1.9450000000000001},"last":1.579}]}]},{"osd":7,"up_from":14,"seq":60129542837,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556268,"kb_used_data":416932,"kb_used_omap":1,"kb_used_meta":139326,"kb_avail":93815572,"statfs":{"total":96636764160,"available":96067145728,"internally_reserved":0,"allocated":426938368,"data_stored":418861204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":142670283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.84399999999999997,"15min":0.82999999999999996},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.41499999999999998},"max":{"1min":1.2130000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.88,"15min":0.872},"min":{"1min":0.56999999999999995,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.333,"5min":1.3380000000000001,"15min":1.3919999999999999},"last":0.33000000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.76900000000000002,"15min":0.86799999999999999},"min":{"1min":0.504,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.095,"5min":1.244,"15min":1.4790000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.91700000000000004,"15min":0.82299999999999995},"min":{"1min":0.51500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.5,"5min":1.524,"15min":1.524},"last":0.5}]},{"osd":2,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0589999999999999,"15min":1.024},"min":{"1min":0.66500000000000004,"5min":0.54700000000000004,"15min":0.52700000000000002},"max":{"1min":1.3109999999999999,"5min":1.833,"15min":1.833},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1830000000000001,"15min":1.153},"min":{"1min":0.91100000000000003,"5min":0.71499999999999997,"15min":0.69399999999999995},"max":{"1min":1.4510000000000001,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":0.55800000000000005}]},{"osd":3,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0760000000000001,"15min":1.083},"min":{"1min":0.66400000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.2929999999999999,"5min":1.673,"15min":1.673},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.204,"5min":1.244,"15min":1.2230000000000001},"min":{"1min":0.84299999999999997,"5min":0.83499999999999996,"15min":0.81399999999999995},"max":{"1min":1.4490000000000001,"5min":1.8109999999999999,"15min":1.8480000000000001},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.86199999999999999,"15min":0.84799999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.171,"5min":1.226,"15min":1.4970000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.95799999999999996,"15min":0.95599999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.242,"5min":1.4970000000000001,"15min":1.77},"last":0.624}]},{"osd":5,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.94299999999999995,"15min":0.97499999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.2490000000000001,"5min":1.3799999999999999,"15min":2},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.105,"15min":1.0349999999999999},"min":{"1min":0.69599999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.508,"5min":1.7150000000000001,"15min":2.056},"last":0.54100000000000004}]},{"osd":6,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.042,"15min":1.008},"min":{"1min":0.68100000000000005,"5min":0.60799999999999998,"15min":0.501},"max":{"1min":1.377,"5min":1.536,"15min":1.571},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1299999999999999,"15min":1.105},"min":{"1min":0.751,"5min":0.72599999999999998,"15min":0.71199999999999997},"max":{"1min":1.5549999999999999,"5min":1.665,"15min":1.665},"last":0.59499999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542836,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":820256,"kb_used_data":640600,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93551584,"statfs":{"total":96636764160,"available":95796822016,"internally_reserved":0,"allocated":655974400,"data_stored":642425281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":183957958},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.378},"max":{"1min":1.173,"5min":1.276,"15min":1.579},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.85299999999999998,"15min":0.86099999999999999},"min":{"1min":0.499,"5min":0.46200000000000002,"15min":0.34599999999999997},"max":{"1min":1.262,"5min":1.627,"15min":1.627},"last":0.27300000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.81200000000000006,"15min":0.83099999999999996},"min":{"1min":0.55300000000000005,"5min":0.47499999999999998,"15min":0.39500000000000002},"max":{"1min":1.22,"5min":1.226,"15min":1.3220000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.88600000000000001,"15min":0.91900000000000004},"min":{"1min":0.64800000000000002,"5min":0.52600000000000002,"15min":0.376},"max":{"1min":1.1539999999999999,"5min":1.3100000000000001,"15min":1.6619999999999999},"last":0.56899999999999995}]},{"osd":2,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.85799999999999998,"15min":0.84799999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.2330000000000001,"5min":1.3220000000000001,"15min":1.486},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.89300000000000002,"15min":0.88},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2749999999999999,"5min":1.3149999999999999,"15min":1.879},"last":1.0920000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.96899999999999997,"15min":0.88900000000000001},"min":{"1min":0.65500000000000003,"5min":0.55900000000000005,"15min":0.42799999999999999},"max":{"1min":1.361,"5min":1.361,"15min":2.7330000000000001},"last":1.071},{"interface":"front","average":{"1min":1.097,"5min":1.0509999999999999,"15min":0.999},"min":{"1min":0.68899999999999995,"5min":0.51300000000000001,"15min":0.47099999999999997},"max":{"1min":1.359,"5min":1.5169999999999999,"15min":2.79},"last":0.61499999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.97299999999999998,"15min":0.98099999999999998},"min":{"1min":0.68500000000000005,"5min":0.57599999999999996,"15min":0.45300000000000001},"max":{"1min":1.119,"5min":1.4710000000000001,"15min":1.6719999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.034,"15min":1.0940000000000001},"min":{"1min":0.87,"5min":0.626,"15min":0.58799999999999997},"max":{"1min":1.3120000000000001,"5min":1.3540000000000001,"15min":1.7629999999999999},"last":0.55100000000000005}]},{"osd":6,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.1000000000000001,"15min":1.0920000000000001},"min":{"1min":0.69999999999999996,"5min":0.66500000000000004,"15min":0.54600000000000004},"max":{"1min":1.3959999999999999,"5min":1.675,"15min":2.8450000000000002},"last":1.012},{"interface":"front","average":{"1min":1.149,"5min":1.123,"15min":1.111},"min":{"1min":0.73699999999999999,"5min":0.59999999999999998,"15min":0.57999999999999996},"max":{"1min":1.577,"5min":1.625,"15min":1.821},"last":0.43099999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.153,"15min":1.127},"min":{"1min":0.70099999999999996,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7490000000000001,"5min":1.7749999999999999,"15min":3.3069999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.115,"5min":1.1579999999999999,"15min":1.113},"min":{"1min":0.88500000000000001,"5min":0.59999999999999998,"15min":0.53200000000000003},"max":{"1min":1.3129999999999999,"5min":1.754,"15min":2.036},"last":0.49099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309846016,"data_stored":309823069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142618624,"data_stored":142610667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117452800,"data_stored":117441250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294043648,"data_stored":294028670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":275668992,"data_stored":275628631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":300916736,"data_stored":300897937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":154218496,"data_stored":154207037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":211644416,"data_stored":211616353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401661952,"data_stored":386457412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297664512,"data_stored":287498447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304177152,"data_stored":291323631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359911424,"data_stored":346240784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":229097472,"data_stored":220069239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":353464320,"data_stored":340406790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":263856128,"data_stored":253734100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":214294528,"data_stored":206714665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:26:03.390 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.390 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.391 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.391 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.391 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.392 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.392 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.393 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.393 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.393 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.394 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.394 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.395 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.395 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.395 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.396 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.396 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.397 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.397 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.397 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.398 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.398 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.399 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.399 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.400 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.400 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.400 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.401 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.401 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.402 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.402 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.403 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.403 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.403 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.404 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.404 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.405 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.405 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.405 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.406 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2023-10-30T12:28:43.012501+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=43, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.406 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.407 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.407 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.407 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.408 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:03.408 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T13:26:18.860 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:26:18.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:26:23.410 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:26:23.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70bab10640 1 Processor -- start 2023-10-30T13:26:23.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70bab10640 1 -- start start 2023-10-30T13:26:23.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70bab10640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f70b41538f0 0x7f70b4159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:23.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70bab10640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b40ca6a0 con 0x7f70b415cd50 2023-10-30T13:26:23.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70bab10640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b415d300 con 0x7f70b40cefa0 2023-10-30T13:26:23.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70bab10640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70b415d5e0 con 0x7f70b41538f0 2023-10-30T13:26:23.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70b930d640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f70b415cd50 0x7f70b40c9f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:60760/0 (socket says 172.21.15.125:60760) 2023-10-30T13:26:23.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70b930d640 1 -- 172.21.15.125:0/15719504 learned_addr learned my addr 172.21.15.125:0/15719504 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:26:23.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70b9b0e640 1 --2- 172.21.15.125:0/15719504 >> v2:172.21.15.136:3300/0 conn(0x7f70b41538f0 0x7f70b4159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:23.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70b8b0c640 1 -- 172.21.15.125:0/15719504 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532481776 0 0) 0x7f70b40ca6a0 con 0x7f70b415cd50 2023-10-30T13:26:23.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70b8b0c640 1 -- 172.21.15.125:0/15719504 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70a0003530 con 0x7f70b415cd50 2023-10-30T13:26:23.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70b8b0c640 1 -- 172.21.15.125:0/15719504 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2417259752 0 0) 0x7f70a0003530 con 0x7f70b415cd50 2023-10-30T13:26:23.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70b9b0e640 1 -- 172.21.15.125:0/15719504 >> v1:172.21.15.136:6789/0 conn(0x7f70b40cefa0 legacy=0x7f70b41640e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:23.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70b9b0e640 1 -- 172.21.15.125:0/15719504 >> v1:172.21.15.125:6789/0 conn(0x7f70b415cd50 legacy=0x7f70b40c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:23.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.557+0000 7f70b9b0e640 1 -- 172.21.15.125:0/15719504 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b415d8c0 con 0x7f70b41538f0 2023-10-30T13:26:23.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.561+0000 7f70b9b0e640 1 --2- 172.21.15.125:0/15719504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70b41538f0 0x7f70b4159fb0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f70b00097c0 tx=0x7f70b001eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3f2613b6bd40169 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:23.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.561+0000 7f70b8b0c640 1 -- 172.21.15.125:0/15719504 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a4001680 con 0x7f70b41538f0 2023-10-30T13:26:23.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.561+0000 7f70b8b0c640 1 -- 172.21.15.125:0/15719504 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70b00a8610 con 0x7f70b41538f0 2023-10-30T13:26:23.568 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.561+0000 7f70b8b0c640 1 -- 172.21.15.125:0/15719504 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70b00a8980 con 0x7f70b41538f0 2023-10-30T13:26:23.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.565+0000 7f70bab10640 1 -- 172.21.15.125:0/15719504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70b41538f0 msgr2=0x7f70b4159fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:23.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.565+0000 7f70bab10640 1 --2- 172.21.15.125:0/15719504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70b41538f0 0x7f70b4159fb0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f70b00097c0 tx=0x7f70b001eb90 comp rx=0 tx=0).stop 2023-10-30T13:26:23.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.565+0000 7f70bab10640 1 -- 172.21.15.125:0/15719504 shutdown_connections 2023-10-30T13:26:23.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.565+0000 7f70bab10640 1 --2- 172.21.15.125:0/15719504 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70b41538f0 0x7f70b4159fb0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:23.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.565+0000 7f70bab10640 1 -- 172.21.15.125:0/15719504 >> 172.21.15.125:0/15719504 conn(0x7f70b406a640 msgr2=0x7f70b4065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:23.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.565+0000 7f70bab10640 1 -- 172.21.15.125:0/15719504 shutdown_connections 2023-10-30T13:26:23.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.565+0000 7f70bab10640 1 -- 172.21.15.125:0/15719504 wait complete. 2023-10-30T13:26:23.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.565+0000 7f70bab10640 1 Processor -- start 2023-10-30T13:26:23.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.565+0000 7f70bab10640 1 -- start start 2023-10-30T13:26:23.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70bab10640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70b40cefa0 0x7f70b413e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:23.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70bab10640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70b41538f0 0x7f70b40b4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:23.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70bab10640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70b415cd50 0x7f70b40b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:23.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70bab10640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b41f2560 con 0x7f70b41538f0 2023-10-30T13:26:23.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70bab10640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70b41f30e0 con 0x7f70b415cd50 2023-10-30T13:26:23.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70bab10640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70b415cb50 con 0x7f70b40cefa0 2023-10-30T13:26:23.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70b930d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70b41538f0 0x7f70b40b4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:23.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70b930d640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70b41538f0 0x7f70b40b4370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:50696/0 (socket says 172.21.15.125:50696) 2023-10-30T13:26:23.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70b930d640 1 -- 172.21.15.125:0/2015784596 learned_addr learned my addr 172.21.15.125:0/2015784596 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:26:23.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70b9b0e640 1 --2- 172.21.15.125:0/2015784596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70b40cefa0 0x7f70b413e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:23.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70ba30f640 1 --2- 172.21.15.125:0/2015784596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70b415cd50 0x7f70b40b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:23.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70b930d640 1 -- 172.21.15.125:0/2015784596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70b40cefa0 msgr2=0x7f70b413e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:23.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70b930d640 1 --2- 172.21.15.125:0/2015784596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70b40cefa0 0x7f70b413e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:23.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70b930d640 1 -- 172.21.15.125:0/2015784596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70b415cd50 msgr2=0x7f70b40b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:23.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70b930d640 1 --2- 172.21.15.125:0/2015784596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70b415cd50 0x7f70b40b48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:23.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70b930d640 1 -- 172.21.15.125:0/2015784596 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b0009420 con 0x7f70b41538f0 2023-10-30T13:26:23.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70b930d640 1 --2- 172.21.15.125:0/2015784596 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70b41538f0 0x7f70b40b4370 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f70a400c700 tx=0x7f70a400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:23.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70aa7fc640 1 -- 172.21.15.125:0/2015784596 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a4005ae0 con 0x7f70b41538f0 2023-10-30T13:26:23.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70aa7fc640 1 -- 172.21.15.125:0/2015784596 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a4005c60 con 0x7f70b41538f0 2023-10-30T13:26:23.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70bab10640 1 -- 172.21.15.125:0/2015784596 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b415ac50 con 0x7f70b41538f0 2023-10-30T13:26:23.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70aa7fc640 1 -- 172.21.15.125:0/2015784596 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a40072b0 con 0x7f70b41538f0 2023-10-30T13:26:23.579 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70bab10640 1 -- 172.21.15.125:0/2015784596 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b41f2a90 con 0x7f70b41538f0 2023-10-30T13:26:23.580 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70bab10640 1 -- 172.21.15.125:0/2015784596 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b41f3290 con 0x7f70b41538f0 2023-10-30T13:26:23.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.569+0000 7f70aa7fc640 1 -- 172.21.15.125:0/2015784596 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f70a4007410 con 0x7f70b41538f0 2023-10-30T13:26:23.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.573+0000 7f70aa7fc640 1 --2- 172.21.15.125:0/2015784596 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70980664d0 0x7f7098068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:23.581 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.573+0000 7f70aa7fc640 1 -- 172.21.15.125:0/2015784596 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f70a4083fe0 con 0x7f70b41538f0 2023-10-30T13:26:23.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.573+0000 7f70b9b0e640 1 --2- 172.21.15.125:0/2015784596 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70980664d0 0x7f7098068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:23.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.573+0000 7f70b9b0e640 1 --2- 172.21.15.125:0/2015784596 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70980664d0 0x7f7098068990 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f70b00b2000 tx=0x7f70b00aa070 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:23.582 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.577+0000 7f70aa7fc640 1 -- 172.21.15.125:0/2015784596 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f70b41f3290 con 0x7f70b41538f0 2023-10-30T13:26:23.793 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.789+0000 7f70bab10640 1 -- 172.21.15.125:0/2015784596 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f70b4159a90 con 0x7f70980664d0 2023-10-30T13:26:23.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.801+0000 7f70aa7fc640 1 -- 172.21.15.125:0/2015784596 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+295431 (secure 0 0 0) 0x7f70b00a8b00 con 0x7f70980664d0 2023-10-30T13:26:23.809 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:26:23.811 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T13:26:23.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.805+0000 7f70bab10640 1 -- 172.21.15.125:0/2015784596 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70980664d0 msgr2=0x7f7098068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:23.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.805+0000 7f70bab10640 1 --2- 172.21.15.125:0/2015784596 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70980664d0 0x7f7098068990 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f70b00b2000 tx=0x7f70b00aa070 comp rx=0 tx=0).stop 2023-10-30T13:26:23.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.805+0000 7f70bab10640 1 -- 172.21.15.125:0/2015784596 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70b41538f0 msgr2=0x7f70b40b4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:23.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.805+0000 7f70bab10640 1 --2- 172.21.15.125:0/2015784596 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70b41538f0 0x7f70b40b4370 secure :-1 s=READY pgs=2208 cs=0 l=1 rev1=1 crypto rx=0x7f70a400c700 tx=0x7f70a400cbc0 comp rx=0 tx=0).stop 2023-10-30T13:26:23.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.805+0000 7f70bab10640 1 -- 172.21.15.125:0/2015784596 shutdown_connections 2023-10-30T13:26:23.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.805+0000 7f70bab10640 1 --2- 172.21.15.125:0/2015784596 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f70980664d0 0x7f7098068990 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:23.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.805+0000 7f70bab10640 1 --2- 172.21.15.125:0/2015784596 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f70b415cd50 0x7f70b40b48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:23.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.805+0000 7f70bab10640 1 --2- 172.21.15.125:0/2015784596 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f70b41538f0 0x7f70b40b4370 unknown :-1 s=CLOSED pgs=2208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:23.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.805+0000 7f70bab10640 1 --2- 172.21.15.125:0/2015784596 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f70b40cefa0 0x7f70b413e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:23.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.805+0000 7f70bab10640 1 -- 172.21.15.125:0/2015784596 >> 172.21.15.125:0/2015784596 conn(0x7f70b406a640 msgr2=0x7f70b40c3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:23.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.805+0000 7f70bab10640 1 -- 172.21.15.125:0/2015784596 shutdown_connections 2023-10-30T13:26:23.816 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:23.805+0000 7f70bab10640 1 -- 172.21.15.125:0/2015784596 wait complete. 2023-10-30T13:26:23.829 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":1813,"stamp":"2023-10-30T13:26:22.285595+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2068932093,"num_objects":40960,"num_object_clones":18317,"num_object_copies":81920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40960,"num_whiteouts":14249,"num_read":30933,"num_read_kb":10838515,"num_write":128197,"num_write_kb":3672596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":19724250,"num_omap_keys":33398,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":168448,"ondisk_log_size":168448,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5528960,"kb_used_data":4140352,"kb_used_omap":12,"kb_used_meta":1388531,"kb_avail":749445760,"statfs":{"total":773094113280,"available":767432458240,"internally_reserved":0,"allocated":4239720448,"data_stored":4144120906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1421856313},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":2270722,"num_omap_keys":3884,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003880"},"pg_stats":[{"pgid":"3.3f","version":"85'2027","reported_seq":4106,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:18.022996+0000","last_change":"2023-10-30T13:26:18.023085+0000","last_active":"2023-10-30T13:26:18.022996+0000","last_peered":"2023-10-30T13:26:18.022996+0000","last_clean":"2023-10-30T13:26:18.022996+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T13:26:18.022996+0000","last_undegraded":"2023-10-30T13:26:18.022996+0000","last_fullsized":"2023-10-30T13:26:18.022996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2027","last_scrub_stamp":"2023-10-30T13:26:18.022913+0000","last_deep_scrub":"85'2027","last_deep_scrub_stamp":"2023-10-30T13:26:18.022913+0000","last_clean_scrub_stamp":"2023-10-30T13:26:18.022913+0000","objects_scrubbed":552,"log_size":2027,"log_dups_size":0,"ondisk_log_size":2027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:29.468884+0000","scrub_duration":0.915286406,"objects_trimmed":0,"snaptrim_duration":0.0086120880000000004,"stat_sum":{"num_bytes":27261067,"num_objects":552,"num_object_clones":249,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":197,"num_read":145,"num_read_kb":29501,"num_write":1298,"num_write_kb":29008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"77'1476","reported_seq":3489,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.322617+0000","last_change":"2023-10-30T12:37:07.190309+0000","last_active":"2023-10-30T13:25:32.322617+0000","last_peered":"2023-10-30T13:25:32.322617+0000","last_clean":"2023-10-30T13:25:32.322617+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T13:25:32.322617+0000","last_undegraded":"2023-10-30T13:25:32.322617+0000","last_fullsized":"2023-10-30T13:25:32.322617+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015306986,"stat_sum":{"num_bytes":13736175,"num_objects":574,"num_object_clones":269,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":223,"num_read":131,"num_read_kb":21315,"num_write":1260,"num_write_kb":15409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"77'1563","reported_seq":3575,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:01.374847+0000","last_change":"2023-10-30T12:37:07.187803+0000","last_active":"2023-10-30T13:25:01.374847+0000","last_peered":"2023-10-30T13:25:01.374847+0000","last_clean":"2023-10-30T13:25:01.374847+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T13:25:01.374847+0000","last_undegraded":"2023-10-30T13:25:01.374847+0000","last_fullsized":"2023-10-30T13:25:01.374847+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1563,"log_dups_size":0,"ondisk_log_size":1563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013634598,"stat_sum":{"num_bytes":28390436,"num_objects":628,"num_object_clones":291,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":232,"num_read":152,"num_read_kb":47076,"num_write":1356,"num_write_kb":37143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"85'2206","reported_seq":4249,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.404657+0000","last_change":"2023-10-30T13:25:48.404708+0000","last_active":"2023-10-30T13:25:48.404657+0000","last_peered":"2023-10-30T13:25:48.404657+0000","last_clean":"2023-10-30T13:25:48.404657+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T13:25:48.404657+0000","last_undegraded":"2023-10-30T13:25:48.404657+0000","last_fullsized":"2023-10-30T13:25:48.404657+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2206","last_scrub_stamp":"2023-10-30T13:25:48.404576+0000","last_deep_scrub":"85'2206","last_deep_scrub_stamp":"2023-10-30T13:25:48.404576+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.404576+0000","objects_scrubbed":629,"log_size":2206,"log_dups_size":0,"ondisk_log_size":2206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:01.872332+0000","scrub_duration":1.22769368,"objects_trimmed":0,"snaptrim_duration":0.0056762749999999997,"stat_sum":{"num_bytes":22921274,"num_objects":629,"num_object_clones":290,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":224,"num_read":145,"num_read_kb":39229,"num_write":1388,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"85'2214","reported_seq":4254,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:00.132866+0000","last_change":"2023-10-30T13:26:00.132866+0000","last_active":"2023-10-30T13:26:00.132866+0000","last_peered":"2023-10-30T13:26:00.132866+0000","last_clean":"2023-10-30T13:26:00.132866+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T13:26:00.132866+0000","last_undegraded":"2023-10-30T13:26:00.132866+0000","last_fullsized":"2023-10-30T13:26:00.132866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2214","last_scrub_stamp":"2023-10-30T13:26:00.132634+0000","last_deep_scrub":"85'2214","last_deep_scrub_stamp":"2023-10-30T13:26:00.132634+0000","last_clean_scrub_stamp":"2023-10-30T13:26:00.132634+0000","objects_scrubbed":623,"log_size":2214,"log_dups_size":0,"ondisk_log_size":2214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:56:19.866906+0000","scrub_duration":1.0674117089999999,"objects_trimmed":0,"snaptrim_duration":0.0081298299999999993,"stat_sum":{"num_bytes":26798060,"num_objects":623,"num_object_clones":286,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":214,"num_read":144,"num_read_kb":26713,"num_write":1434,"num_write_kb":32534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"85'2029","reported_seq":4091,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.107819+0000","last_change":"2023-10-30T13:25:44.107819+0000","last_active":"2023-10-30T13:25:44.107819+0000","last_peered":"2023-10-30T13:25:44.107819+0000","last_clean":"2023-10-30T13:25:44.107819+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T13:25:44.107819+0000","last_undegraded":"2023-10-30T13:25:44.107819+0000","last_fullsized":"2023-10-30T13:25:44.107819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2029","last_scrub_stamp":"2023-10-30T13:25:44.107605+0000","last_deep_scrub":"85'2029","last_deep_scrub_stamp":"2023-10-30T13:25:44.107605+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.107605+0000","objects_scrubbed":563,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:35.807432+0000","scrub_duration":0.97669900899999995,"objects_trimmed":0,"snaptrim_duration":0.0048510920000000004,"stat_sum":{"num_bytes":13539463,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":205,"num_read":144,"num_read_kb":15818,"num_write":1313,"num_write_kb":14957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"85'2168","reported_seq":4191,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:58.341167+0000","last_change":"2023-10-30T13:25:58.341234+0000","last_active":"2023-10-30T13:25:58.341167+0000","last_peered":"2023-10-30T13:25:58.341167+0000","last_clean":"2023-10-30T13:25:58.341167+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T13:25:58.341167+0000","last_undegraded":"2023-10-30T13:25:58.341167+0000","last_fullsized":"2023-10-30T13:25:58.341167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2168","last_scrub_stamp":"2023-10-30T13:25:58.341077+0000","last_deep_scrub":"85'2168","last_deep_scrub_stamp":"2023-10-30T13:25:58.341077+0000","last_clean_scrub_stamp":"2023-10-30T13:25:58.341077+0000","objects_scrubbed":606,"log_size":2168,"log_dups_size":0,"ondisk_log_size":2168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:06.931072+0000","scrub_duration":1.219517739,"objects_trimmed":0,"snaptrim_duration":0.00082671399999999999,"stat_sum":{"num_bytes":7202428,"num_objects":606,"num_object_clones":284,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":219,"num_read":126,"num_read_kb":1677,"num_write":1361,"num_write_kb":10203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"85'2105","reported_seq":4163,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:14.120371+0000","last_change":"2023-10-30T13:26:14.120371+0000","last_active":"2023-10-30T13:26:14.120371+0000","last_peered":"2023-10-30T13:26:14.120371+0000","last_clean":"2023-10-30T13:26:14.120371+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T13:26:14.120371+0000","last_undegraded":"2023-10-30T13:26:14.120371+0000","last_fullsized":"2023-10-30T13:26:14.120371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2105","last_scrub_stamp":"2023-10-30T13:26:14.120044+0000","last_deep_scrub":"85'2105","last_deep_scrub_stamp":"2023-10-30T13:26:14.120044+0000","last_clean_scrub_stamp":"2023-10-30T13:26:14.120044+0000","objects_scrubbed":587,"log_size":2105,"log_dups_size":0,"ondisk_log_size":2105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:11:07.710223+0000","scrub_duration":1.494910695,"objects_trimmed":0,"snaptrim_duration":0.014437559000000001,"stat_sum":{"num_bytes":23150826,"num_objects":587,"num_object_clones":269,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":199,"num_read":149,"num_read_kb":38846,"num_write":1362,"num_write_kb":24887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"85'2167","reported_seq":4231,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:00.039174+0000","last_change":"2023-10-30T13:26:00.039258+0000","last_active":"2023-10-30T13:26:00.039174+0000","last_peered":"2023-10-30T13:26:00.039174+0000","last_clean":"2023-10-30T13:26:00.039174+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T13:26:00.039174+0000","last_undegraded":"2023-10-30T13:26:00.039174+0000","last_fullsized":"2023-10-30T13:26:00.039174+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2167","last_scrub_stamp":"2023-10-30T13:26:00.039070+0000","last_deep_scrub":"85'2167","last_deep_scrub_stamp":"2023-10-30T13:26:00.039070+0000","last_clean_scrub_stamp":"2023-10-30T13:26:00.039070+0000","objects_scrubbed":618,"log_size":2167,"log_dups_size":0,"ondisk_log_size":2167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:06:11.823857+0000","scrub_duration":1.5619247110000001,"objects_trimmed":0,"snaptrim_duration":0.0014185980000000001,"stat_sum":{"num_bytes":36575095,"num_objects":618,"num_object_clones":282,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":216,"num_read":165,"num_read_kb":36418,"num_write":1354,"num_write_kb":45869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"85'2045","reported_seq":4109,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:15.050332+0000","last_change":"2023-10-30T13:26:15.050397+0000","last_active":"2023-10-30T13:26:15.050332+0000","last_peered":"2023-10-30T13:26:15.050332+0000","last_clean":"2023-10-30T13:26:15.050332+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T13:26:15.050332+0000","last_undegraded":"2023-10-30T13:26:15.050332+0000","last_fullsized":"2023-10-30T13:26:15.050332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2045","last_scrub_stamp":"2023-10-30T13:26:15.050260+0000","last_deep_scrub":"85'2045","last_deep_scrub_stamp":"2023-10-30T13:26:15.050260+0000","last_clean_scrub_stamp":"2023-10-30T13:26:15.050260+0000","objects_scrubbed":586,"log_size":2045,"log_dups_size":0,"ondisk_log_size":2045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:47.785761+0000","scrub_duration":0.93888254999999998,"objects_trimmed":0,"snaptrim_duration":0.0091404980000000004,"stat_sum":{"num_bytes":11176343,"num_objects":586,"num_object_clones":270,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":213,"num_read":154,"num_read_kb":10770,"num_write":1260,"num_write_kb":13564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"85'2136","reported_seq":4176,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:06.941519+0000","last_change":"2023-10-30T13:26:06.941585+0000","last_active":"2023-10-30T13:26:06.941519+0000","last_peered":"2023-10-30T13:26:06.941519+0000","last_clean":"2023-10-30T13:26:06.941519+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T13:26:06.941519+0000","last_undegraded":"2023-10-30T13:26:06.941519+0000","last_fullsized":"2023-10-30T13:26:06.941519+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2136","last_scrub_stamp":"2023-10-30T13:26:06.941431+0000","last_deep_scrub":"85'2136","last_deep_scrub_stamp":"2023-10-30T13:26:06.941431+0000","last_clean_scrub_stamp":"2023-10-30T13:26:06.941431+0000","objects_scrubbed":625,"log_size":2136,"log_dups_size":0,"ondisk_log_size":2136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:47.801796+0000","scrub_duration":0.82203926400000005,"objects_trimmed":0,"snaptrim_duration":0.0075712790000000002,"stat_sum":{"num_bytes":23568118,"num_objects":625,"num_object_clones":283,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":230,"num_read":141,"num_read_kb":34199,"num_write":1299,"num_write_kb":24762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"85'2446","reported_seq":4405,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:14.612141+0000","last_change":"2023-10-30T13:26:14.612219+0000","last_active":"2023-10-30T13:26:14.612141+0000","last_peered":"2023-10-30T13:26:14.612141+0000","last_clean":"2023-10-30T13:26:14.612141+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T13:26:14.612141+0000","last_undegraded":"2023-10-30T13:26:14.612141+0000","last_fullsized":"2023-10-30T13:26:14.612141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2446","last_scrub_stamp":"2023-10-30T13:26:14.612071+0000","last_deep_scrub":"85'2446","last_deep_scrub_stamp":"2023-10-30T13:26:14.612071+0000","last_clean_scrub_stamp":"2023-10-30T13:26:14.612071+0000","objects_scrubbed":721,"log_size":2446,"log_dups_size":0,"ondisk_log_size":2446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:13:35.491978+0000","scrub_duration":1.5476285650000001,"objects_trimmed":0,"snaptrim_duration":0.007366493,"stat_sum":{"num_bytes":24142540,"num_objects":721,"num_object_clones":385,"num_object_copies":1442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":231,"num_read":157,"num_read_kb":24833,"num_write":1452,"num_write_kb":33857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"85'2099","reported_seq":4169,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.041415+0000","last_change":"2023-10-30T13:25:54.041494+0000","last_active":"2023-10-30T13:25:54.041415+0000","last_peered":"2023-10-30T13:25:54.041415+0000","last_clean":"2023-10-30T13:25:54.041415+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T13:25:54.041415+0000","last_undegraded":"2023-10-30T13:25:54.041415+0000","last_fullsized":"2023-10-30T13:25:54.041415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2099","last_scrub_stamp":"2023-10-30T13:25:54.041318+0000","last_deep_scrub":"85'2099","last_deep_scrub_stamp":"2023-10-30T13:25:54.041318+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.041318+0000","objects_scrubbed":596,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:14.898677+0000","scrub_duration":1.085239426,"objects_trimmed":0,"snaptrim_duration":0.006759035,"stat_sum":{"num_bytes":14254037,"num_objects":596,"num_object_clones":276,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":205,"num_read":165,"num_read_kb":21722,"num_write":1326,"num_write_kb":24306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"85'2174","reported_seq":4206,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:01.905598+0000","last_change":"2023-10-30T13:26:01.905712+0000","last_active":"2023-10-30T13:26:01.905598+0000","last_peered":"2023-10-30T13:26:01.905598+0000","last_clean":"2023-10-30T13:26:01.905598+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T13:26:01.905598+0000","last_undegraded":"2023-10-30T13:26:01.905598+0000","last_fullsized":"2023-10-30T13:26:01.905598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2174","last_scrub_stamp":"2023-10-30T13:26:01.905493+0000","last_deep_scrub":"85'2174","last_deep_scrub_stamp":"2023-10-30T13:26:01.905493+0000","last_clean_scrub_stamp":"2023-10-30T13:26:01.905493+0000","objects_scrubbed":606,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:14:46.370347+0000","scrub_duration":0.97684040100000002,"objects_trimmed":0,"snaptrim_duration":0.0057386779999999997,"stat_sum":{"num_bytes":10370740,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":213,"num_read":131,"num_read_kb":5845,"num_write":1381,"num_write_kb":12595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"85'2261","reported_seq":4276,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:03.616856+0000","last_change":"2023-10-30T13:26:03.616927+0000","last_active":"2023-10-30T13:26:03.616856+0000","last_peered":"2023-10-30T13:26:03.616856+0000","last_clean":"2023-10-30T13:26:03.616856+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T13:26:03.616856+0000","last_undegraded":"2023-10-30T13:26:03.616856+0000","last_fullsized":"2023-10-30T13:26:03.616856+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2261","last_scrub_stamp":"2023-10-30T13:26:03.616747+0000","last_deep_scrub":"85'2261","last_deep_scrub_stamp":"2023-10-30T13:26:03.616747+0000","last_clean_scrub_stamp":"2023-10-30T13:26:03.616747+0000","objects_scrubbed":637,"log_size":2261,"log_dups_size":0,"ondisk_log_size":2261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:45.471660+0000","scrub_duration":1.159355685,"objects_trimmed":0,"snaptrim_duration":0.013549705,"stat_sum":{"num_bytes":19248519,"num_objects":637,"num_object_clones":295,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":221,"num_read":128,"num_read_kb":21688,"num_write":1447,"num_write_kb":21442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"85'2193","reported_seq":4229,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:46.252816+0000","last_change":"2023-10-30T13:25:46.252862+0000","last_active":"2023-10-30T13:25:46.252816+0000","last_peered":"2023-10-30T13:25:46.252816+0000","last_clean":"2023-10-30T13:25:46.252816+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T13:25:46.252816+0000","last_undegraded":"2023-10-30T13:25:46.252816+0000","last_fullsized":"2023-10-30T13:25:46.252816+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2193","last_scrub_stamp":"2023-10-30T13:25:46.252731+0000","last_deep_scrub":"85'2193","last_deep_scrub_stamp":"2023-10-30T13:25:46.252731+0000","last_clean_scrub_stamp":"2023-10-30T13:25:46.252731+0000","objects_scrubbed":619,"log_size":2193,"log_dups_size":0,"ondisk_log_size":2193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:51.513414+0000","scrub_duration":0.82934357400000003,"objects_trimmed":0,"snaptrim_duration":0.0075608309999999996,"stat_sum":{"num_bytes":14269019,"num_objects":619,"num_object_clones":290,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":233,"num_read":143,"num_read_kb":25965,"num_write":1377,"num_write_kb":20442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"85'2247","reported_seq":4267,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:45.441798+0000","last_change":"2023-10-30T13:25:45.441869+0000","last_active":"2023-10-30T13:25:45.441798+0000","last_peered":"2023-10-30T13:25:45.441798+0000","last_clean":"2023-10-30T13:25:45.441798+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T13:25:45.441798+0000","last_undegraded":"2023-10-30T13:25:45.441798+0000","last_fullsized":"2023-10-30T13:25:45.441798+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2247","last_scrub_stamp":"2023-10-30T13:25:45.441689+0000","last_deep_scrub":"85'2247","last_deep_scrub_stamp":"2023-10-30T13:25:45.441689+0000","last_clean_scrub_stamp":"2023-10-30T13:25:45.441689+0000","objects_scrubbed":642,"log_size":2247,"log_dups_size":0,"ondisk_log_size":2247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:54:27.498757+0000","scrub_duration":0.97553288699999996,"objects_trimmed":0,"snaptrim_duration":0.0099357109999999998,"stat_sum":{"num_bytes":14793776,"num_objects":642,"num_object_clones":304,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":242,"num_read":140,"num_read_kb":30098,"num_write":1386,"num_write_kb":20714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"85'2006","reported_seq":4061,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.398039+0000","last_change":"2023-10-30T13:25:57.398144+0000","last_active":"2023-10-30T13:25:57.398039+0000","last_peered":"2023-10-30T13:25:57.398039+0000","last_clean":"2023-10-30T13:25:57.398039+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T13:25:57.398039+0000","last_undegraded":"2023-10-30T13:25:57.398039+0000","last_fullsized":"2023-10-30T13:25:57.398039+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2006","last_scrub_stamp":"2023-10-30T13:25:57.397929+0000","last_deep_scrub":"85'2006","last_deep_scrub_stamp":"2023-10-30T13:25:57.397929+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.397929+0000","objects_scrubbed":562,"log_size":2006,"log_dups_size":0,"ondisk_log_size":2006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:28.919675+0000","scrub_duration":1.1960647630000001,"objects_trimmed":0,"snaptrim_duration":0.0065893760000000001,"stat_sum":{"num_bytes":15508852,"num_objects":562,"num_object_clones":263,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":216,"num_read":137,"num_read_kb":10064,"num_write":1244,"num_write_kb":17198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"82'1576","reported_seq":3679,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.228028+0000","last_change":"2023-10-30T12:37:07.182668+0000","last_active":"2023-10-30T13:25:32.228028+0000","last_peered":"2023-10-30T13:25:32.228028+0000","last_clean":"2023-10-30T13:25:32.228028+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T13:25:32.228028+0000","last_undegraded":"2023-10-30T13:25:32.228028+0000","last_fullsized":"2023-10-30T13:25:32.228028+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1576,"log_dups_size":0,"ondisk_log_size":1576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010472766,"stat_sum":{"num_bytes":15695810,"num_objects":614,"num_object_clones":281,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":221,"num_read":233,"num_read_kb":17180,"num_write":1382,"num_write_kb":23481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"85'2183","reported_seq":4220,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:10.971723+0000","last_change":"2023-10-30T13:26:10.971791+0000","last_active":"2023-10-30T13:26:10.971723+0000","last_peered":"2023-10-30T13:26:10.971723+0000","last_clean":"2023-10-30T13:26:10.971723+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T13:26:10.971723+0000","last_undegraded":"2023-10-30T13:26:10.971723+0000","last_fullsized":"2023-10-30T13:26:10.971723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2183","last_scrub_stamp":"2023-10-30T13:26:10.971644+0000","last_deep_scrub":"85'2183","last_deep_scrub_stamp":"2023-10-30T13:26:10.971644+0000","last_clean_scrub_stamp":"2023-10-30T13:26:10.971644+0000","objects_scrubbed":614,"log_size":2183,"log_dups_size":0,"ondisk_log_size":2183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:19.149519+0000","scrub_duration":0.86719633299999999,"objects_trimmed":0,"snaptrim_duration":0.0075455599999999998,"stat_sum":{"num_bytes":5586810,"num_objects":614,"num_object_clones":286,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":230,"num_read":140,"num_read_kb":5846,"num_write":1359,"num_write_kb":15885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"85'2129","reported_seq":4280,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.179234+0000","last_change":"2023-10-30T13:25:54.179340+0000","last_active":"2023-10-30T13:25:54.179234+0000","last_peered":"2023-10-30T13:25:54.179234+0000","last_clean":"2023-10-30T13:25:54.179234+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T13:25:54.179234+0000","last_undegraded":"2023-10-30T13:25:54.179234+0000","last_fullsized":"2023-10-30T13:25:54.179234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2129","last_scrub_stamp":"2023-10-30T13:25:54.178827+0000","last_deep_scrub":"85'2129","last_deep_scrub_stamp":"2023-10-30T13:25:54.178827+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.178827+0000","objects_scrubbed":606,"log_size":2129,"log_dups_size":0,"ondisk_log_size":2129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:18.992880+0000","scrub_duration":1.0904528849999999,"objects_trimmed":0,"snaptrim_duration":0.0076434509999999999,"stat_sum":{"num_bytes":23023646,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":222,"num_read":249,"num_read_kb":26208,"num_write":1319,"num_write_kb":28338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"85'2324","reported_seq":4342,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:04.661649+0000","last_change":"2023-10-30T13:26:04.661717+0000","last_active":"2023-10-30T13:26:04.661649+0000","last_peered":"2023-10-30T13:26:04.661649+0000","last_clean":"2023-10-30T13:26:04.661649+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T13:26:04.661649+0000","last_undegraded":"2023-10-30T13:26:04.661649+0000","last_fullsized":"2023-10-30T13:26:04.661649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2324","last_scrub_stamp":"2023-10-30T13:26:04.661572+0000","last_deep_scrub":"85'2324","last_deep_scrub_stamp":"2023-10-30T13:26:04.661572+0000","last_clean_scrub_stamp":"2023-10-30T13:26:04.661572+0000","objects_scrubbed":653,"log_size":2324,"log_dups_size":0,"ondisk_log_size":2324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:26:17.690671+0000","scrub_duration":1.171930887,"objects_trimmed":0,"snaptrim_duration":0.0016105379999999999,"stat_sum":{"num_bytes":11166034,"num_objects":653,"num_object_clones":306,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":233,"num_read":139,"num_read_kb":9627,"num_write":1472,"num_write_kb":18195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"85'2102","reported_seq":4154,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:11.723496+0000","last_change":"2023-10-30T13:26:11.723576+0000","last_active":"2023-10-30T13:26:11.723496+0000","last_peered":"2023-10-30T13:26:11.723496+0000","last_clean":"2023-10-30T13:26:11.723496+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T13:26:11.723496+0000","last_undegraded":"2023-10-30T13:26:11.723496+0000","last_fullsized":"2023-10-30T13:26:11.723496+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2102","last_scrub_stamp":"2023-10-30T13:26:11.723400+0000","last_deep_scrub":"85'2102","last_deep_scrub_stamp":"2023-10-30T13:26:11.723400+0000","last_clean_scrub_stamp":"2023-10-30T13:26:11.723400+0000","objects_scrubbed":582,"log_size":2102,"log_dups_size":0,"ondisk_log_size":2102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:07.437907+0000","scrub_duration":1.1504174899999999,"objects_trimmed":0,"snaptrim_duration":0.014064241,"stat_sum":{"num_bytes":10716893,"num_objects":582,"num_object_clones":264,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":203,"num_read":135,"num_read_kb":13340,"num_write":1353,"num_write_kb":16108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"85'2148","reported_seq":4184,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.477345+0000","last_change":"2023-10-30T13:25:44.477389+0000","last_active":"2023-10-30T13:25:44.477345+0000","last_peered":"2023-10-30T13:25:44.477345+0000","last_clean":"2023-10-30T13:25:44.477345+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T13:25:44.477345+0000","last_undegraded":"2023-10-30T13:25:44.477345+0000","last_fullsized":"2023-10-30T13:25:44.477345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2148","last_scrub_stamp":"2023-10-30T13:25:44.477273+0000","last_deep_scrub":"85'2148","last_deep_scrub_stamp":"2023-10-30T13:25:44.477273+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.477273+0000","objects_scrubbed":597,"log_size":2148,"log_dups_size":0,"ondisk_log_size":2148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:41.538304+0000","scrub_duration":0.99138869699999999,"objects_trimmed":0,"snaptrim_duration":0.0088476170000000003,"stat_sum":{"num_bytes":10233195,"num_objects":597,"num_object_clones":275,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":203,"num_read":127,"num_read_kb":1819,"num_write":1383,"num_write_kb":16400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"82'1465","reported_seq":3480,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.203328+0000","last_change":"2023-10-30T12:37:07.188606+0000","last_active":"2023-10-30T13:25:16.203328+0000","last_peered":"2023-10-30T13:25:16.203328+0000","last_clean":"2023-10-30T13:25:16.203328+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T13:25:16.203328+0000","last_undegraded":"2023-10-30T13:25:16.203328+0000","last_fullsized":"2023-10-30T13:25:16.203328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1465,"log_dups_size":0,"ondisk_log_size":1465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014279337,"stat_sum":{"num_bytes":22085088,"num_objects":582,"num_object_clones":270,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":216,"num_read":135,"num_read_kb":28666,"num_write":1258,"num_write_kb":25992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"85'2207","reported_seq":4233,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:02.278563+0000","last_change":"2023-10-30T13:26:02.278655+0000","last_active":"2023-10-30T13:26:02.278563+0000","last_peered":"2023-10-30T13:26:02.278563+0000","last_clean":"2023-10-30T13:26:02.278563+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T13:26:02.278563+0000","last_undegraded":"2023-10-30T13:26:02.278563+0000","last_fullsized":"2023-10-30T13:26:02.278563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2207","last_scrub_stamp":"2023-10-30T13:26:02.278473+0000","last_deep_scrub":"85'2207","last_deep_scrub_stamp":"2023-10-30T13:26:02.278473+0000","last_clean_scrub_stamp":"2023-10-30T13:26:02.278473+0000","objects_scrubbed":598,"log_size":2207,"log_dups_size":0,"ondisk_log_size":2207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:21:31.276370+0000","scrub_duration":1.192745612,"objects_trimmed":0,"snaptrim_duration":0.004006122,"stat_sum":{"num_bytes":15408826,"num_objects":598,"num_object_clones":277,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":213,"num_read":119,"num_read_kb":9406,"num_write":1408,"num_write_kb":18485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"85'2174","reported_seq":4213,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.084913+0000","last_change":"2023-10-30T13:25:53.084979+0000","last_active":"2023-10-30T13:25:53.084913+0000","last_peered":"2023-10-30T13:25:53.084913+0000","last_clean":"2023-10-30T13:25:53.084913+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T13:25:53.084913+0000","last_undegraded":"2023-10-30T13:25:53.084913+0000","last_fullsized":"2023-10-30T13:25:53.084913+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2174","last_scrub_stamp":"2023-10-30T13:25:53.084846+0000","last_deep_scrub":"85'2174","last_deep_scrub_stamp":"2023-10-30T13:25:53.084846+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.084846+0000","objects_scrubbed":625,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:02.345358+0000","scrub_duration":1.1204613670000001,"objects_trimmed":0,"snaptrim_duration":0.0059072409999999997,"stat_sum":{"num_bytes":25755688,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":240,"num_read":144,"num_read_kb":24326,"num_write":1332,"num_write_kb":28963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"85'2102","reported_seq":4117,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:06.032785+0000","last_change":"2023-10-30T13:26:06.032856+0000","last_active":"2023-10-30T13:26:06.032785+0000","last_peered":"2023-10-30T13:26:06.032785+0000","last_clean":"2023-10-30T13:26:06.032785+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T13:26:06.032785+0000","last_undegraded":"2023-10-30T13:26:06.032785+0000","last_fullsized":"2023-10-30T13:26:06.032785+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2102","last_scrub_stamp":"2023-10-30T13:26:06.032657+0000","last_deep_scrub":"85'2102","last_deep_scrub_stamp":"2023-10-30T13:26:06.032657+0000","last_clean_scrub_stamp":"2023-10-30T13:26:06.032657+0000","objects_scrubbed":617,"log_size":2102,"log_dups_size":0,"ondisk_log_size":2102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:44.800470+0000","scrub_duration":1.0262600609999999,"objects_trimmed":0,"snaptrim_duration":0.0062401690000000003,"stat_sum":{"num_bytes":11021781,"num_objects":617,"num_object_clones":286,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":226,"num_read":116,"num_read_kb":10954,"num_write":1301,"num_write_kb":16114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"85'2273","reported_seq":4279,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:55.521292+0000","last_change":"2023-10-30T13:25:55.521358+0000","last_active":"2023-10-30T13:25:55.521292+0000","last_peered":"2023-10-30T13:25:55.521292+0000","last_clean":"2023-10-30T13:25:55.521292+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T13:25:55.521292+0000","last_undegraded":"2023-10-30T13:25:55.521292+0000","last_fullsized":"2023-10-30T13:25:55.521292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2273","last_scrub_stamp":"2023-10-30T13:25:55.521205+0000","last_deep_scrub":"85'2273","last_deep_scrub_stamp":"2023-10-30T13:25:55.521205+0000","last_clean_scrub_stamp":"2023-10-30T13:25:55.521205+0000","objects_scrubbed":653,"log_size":2273,"log_dups_size":0,"ondisk_log_size":2273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:44:08.018678+0000","scrub_duration":1.1272668750000001,"objects_trimmed":0,"snaptrim_duration":0.014933905000000001,"stat_sum":{"num_bytes":11161783,"num_objects":653,"num_object_clones":309,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":240,"num_read":130,"num_read_kb":12352,"num_write":1416,"num_write_kb":13402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"82'1476","reported_seq":3520,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.210549+0000","last_change":"2023-10-30T12:37:07.180055+0000","last_active":"2023-10-30T13:25:16.210549+0000","last_peered":"2023-10-30T13:25:16.210549+0000","last_clean":"2023-10-30T13:25:16.210549+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T13:25:16.210549+0000","last_undegraded":"2023-10-30T13:25:16.210549+0000","last_fullsized":"2023-10-30T13:25:16.210549+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1476,"log_dups_size":0,"ondisk_log_size":1476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0090148399999999997,"stat_sum":{"num_bytes":11081748,"num_objects":549,"num_object_clones":251,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":201,"num_read":142,"num_read_kb":9474,"num_write":1288,"num_write_kb":13652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"85'2406","reported_seq":4433,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.442339+0000","last_change":"2023-10-30T13:25:54.442416+0000","last_active":"2023-10-30T13:25:54.442339+0000","last_peered":"2023-10-30T13:25:54.442339+0000","last_clean":"2023-10-30T13:25:54.442339+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T13:25:54.442339+0000","last_undegraded":"2023-10-30T13:25:54.442339+0000","last_fullsized":"2023-10-30T13:25:54.442339+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2406","last_scrub_stamp":"2023-10-30T13:25:54.442247+0000","last_deep_scrub":"85'2406","last_deep_scrub_stamp":"2023-10-30T13:25:54.442247+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.442247+0000","objects_scrubbed":693,"log_size":2406,"log_dups_size":0,"ondisk_log_size":2406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:12.648322+0000","scrub_duration":1.2677238399999999,"objects_trimmed":0,"snaptrim_duration":0.0066657749999999997,"stat_sum":{"num_bytes":20125251,"num_objects":693,"num_object_clones":330,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":267,"num_read":168,"num_read_kb":25254,"num_write":1457,"num_write_kb":26871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"85'2152","reported_seq":4191,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:45.218457+0000","last_change":"2023-10-30T13:25:45.218569+0000","last_active":"2023-10-30T13:25:45.218457+0000","last_peered":"2023-10-30T13:25:45.218457+0000","last_clean":"2023-10-30T13:25:45.218457+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T13:25:45.218457+0000","last_undegraded":"2023-10-30T13:25:45.218457+0000","last_fullsized":"2023-10-30T13:25:45.218457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2152","last_scrub_stamp":"2023-10-30T13:25:45.218360+0000","last_deep_scrub":"85'2152","last_deep_scrub_stamp":"2023-10-30T13:25:45.218360+0000","last_clean_scrub_stamp":"2023-10-30T13:25:45.218360+0000","objects_scrubbed":604,"log_size":2152,"log_dups_size":0,"ondisk_log_size":2152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:53:35.163970+0000","scrub_duration":1.1092469380000001,"objects_trimmed":0,"snaptrim_duration":0.005419129,"stat_sum":{"num_bytes":18031744,"num_objects":604,"num_object_clones":282,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":228,"num_read":136,"num_read_kb":25964,"num_write":1344,"num_write_kb":19625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"85'2155","reported_seq":4202,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:00.594900+0000","last_change":"2023-10-30T13:26:00.594900+0000","last_active":"2023-10-30T13:26:00.594900+0000","last_peered":"2023-10-30T13:26:00.594900+0000","last_clean":"2023-10-30T13:26:00.594900+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T13:26:00.594900+0000","last_undegraded":"2023-10-30T13:26:00.594900+0000","last_fullsized":"2023-10-30T13:26:00.594900+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2155","last_scrub_stamp":"2023-10-30T13:26:00.594690+0000","last_deep_scrub":"85'2155","last_deep_scrub_stamp":"2023-10-30T13:26:00.594690+0000","last_clean_scrub_stamp":"2023-10-30T13:26:00.594690+0000","objects_scrubbed":607,"log_size":2155,"log_dups_size":0,"ondisk_log_size":2155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:43.537640+0000","scrub_duration":1.1209885449999999,"objects_trimmed":0,"snaptrim_duration":0.0013742470000000001,"stat_sum":{"num_bytes":22381695,"num_objects":607,"num_object_clones":275,"num_object_copies":1214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":227,"num_read":141,"num_read_kb":33421,"num_write":1348,"num_write_kb":23422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1042","reported_seq":1466,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:42.402455+0000","last_change":"2023-10-30T13:25:42.402455+0000","last_active":"2023-10-30T13:25:42.402455+0000","last_peered":"2023-10-30T13:25:42.402455+0000","last_clean":"2023-10-30T13:25:42.402455+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T13:25:42.402455+0000","last_undegraded":"2023-10-30T13:25:42.402455+0000","last_fullsized":"2023-10-30T13:25:42.402455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1042","last_scrub_stamp":"2023-10-30T13:25:42.402195+0000","last_deep_scrub":"85'1042","last_deep_scrub_stamp":"2023-10-30T13:25:42.402195+0000","last_clean_scrub_stamp":"2023-10-30T13:25:42.402195+0000","objects_scrubbed":51,"log_size":1042,"log_dups_size":0,"ondisk_log_size":1042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:20.707501+0000","scrub_duration":0.156286863,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":570,"num_read_kb":319827,"num_write":1218,"num_write_kb":80521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":794018,"num_omap_keys":1312,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"85'2177","reported_seq":4221,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:01.438545+0000","last_change":"2023-10-30T13:26:01.438545+0000","last_active":"2023-10-30T13:26:01.438545+0000","last_peered":"2023-10-30T13:26:01.438545+0000","last_clean":"2023-10-30T13:26:01.438545+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T13:26:01.438545+0000","last_undegraded":"2023-10-30T13:26:01.438545+0000","last_fullsized":"2023-10-30T13:26:01.438545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2177","last_scrub_stamp":"2023-10-30T13:26:01.438339+0000","last_deep_scrub":"85'2177","last_deep_scrub_stamp":"2023-10-30T13:26:01.438339+0000","last_clean_scrub_stamp":"2023-10-30T13:26:01.438339+0000","objects_scrubbed":616,"log_size":2177,"log_dups_size":0,"ondisk_log_size":2177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:25.473052+0000","scrub_duration":1.2567298680000001,"objects_trimmed":0,"snaptrim_duration":0.0086018069999999995,"stat_sum":{"num_bytes":19940185,"num_objects":616,"num_object_clones":283,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":213,"num_read":145,"num_read_kb":23669,"num_write":1398,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1837","reported_seq":2428,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:15.510377+0000","last_change":"2023-10-30T13:26:15.510377+0000","last_active":"2023-10-30T13:26:15.510377+0000","last_peered":"2023-10-30T13:26:15.510377+0000","last_clean":"2023-10-30T13:26:15.510377+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T13:26:15.510377+0000","last_undegraded":"2023-10-30T13:26:15.510377+0000","last_fullsized":"2023-10-30T13:26:15.510377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1837","last_scrub_stamp":"2023-10-30T13:26:15.510196+0000","last_deep_scrub":"85'1837","last_deep_scrub_stamp":"2023-10-30T13:26:15.510196+0000","last_clean_scrub_stamp":"2023-10-30T13:26:15.510196+0000","objects_scrubbed":42,"log_size":1837,"log_dups_size":0,"ondisk_log_size":1837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:16:27.071396+0000","scrub_duration":0.084593108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17028825,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":761,"num_read_kb":396040,"num_write":1931,"num_write_kb":84639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":554115,"num_omap_keys":1012,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"85'1626","reported_seq":2455,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:46.170208+0000","last_change":"2023-10-30T13:25:46.170305+0000","last_active":"2023-10-30T13:25:46.170208+0000","last_peered":"2023-10-30T13:25:46.170208+0000","last_clean":"2023-10-30T13:25:46.170208+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T13:25:46.170208+0000","last_undegraded":"2023-10-30T13:25:46.170208+0000","last_fullsized":"2023-10-30T13:25:46.170208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1626","last_scrub_stamp":"2023-10-30T13:25:46.170040+0000","last_deep_scrub":"85'1626","last_deep_scrub_stamp":"2023-10-30T13:25:46.170040+0000","last_clean_scrub_stamp":"2023-10-30T13:25:46.170040+0000","objects_scrubbed":57,"log_size":1626,"log_dups_size":0,"ondisk_log_size":1626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:30:38.684197+0000","scrub_duration":0.213161079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182949,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":1117,"num_read_kb":268002,"num_write":1800,"num_write_kb":63122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":717548,"num_omap_keys":1302,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"85'2510","reported_seq":4555,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.637463+0000","last_change":"2023-10-30T13:25:54.637570+0000","last_active":"2023-10-30T13:25:54.637463+0000","last_peered":"2023-10-30T13:25:54.637463+0000","last_clean":"2023-10-30T13:25:54.637463+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T13:25:54.637463+0000","last_undegraded":"2023-10-30T13:25:54.637463+0000","last_fullsized":"2023-10-30T13:25:54.637463+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2510","last_scrub_stamp":"2023-10-30T13:25:54.637344+0000","last_deep_scrub":"85'2510","last_deep_scrub_stamp":"2023-10-30T13:25:54.637344+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.637344+0000","objects_scrubbed":709,"log_size":2510,"log_dups_size":0,"ondisk_log_size":2510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:29:25.404778+0000","scrub_duration":1.3090190449999999,"objects_trimmed":0,"snaptrim_duration":0.016143469000000001,"stat_sum":{"num_bytes":14540052,"num_objects":709,"num_object_clones":329,"num_object_copies":1418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":254,"num_read":191,"num_read_kb":6719,"num_write":1583,"num_write_kb":26382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"85'978","reported_seq":1390,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:00.613642+0000","last_change":"2023-10-30T13:26:00.613714+0000","last_active":"2023-10-30T13:26:00.613642+0000","last_peered":"2023-10-30T13:26:00.613642+0000","last_clean":"2023-10-30T13:26:00.613642+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T13:26:00.613642+0000","last_undegraded":"2023-10-30T13:26:00.613642+0000","last_fullsized":"2023-10-30T13:26:00.613642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'978","last_scrub_stamp":"2023-10-30T13:26:00.613566+0000","last_deep_scrub":"85'978","last_deep_scrub_stamp":"2023-10-30T13:26:00.613566+0000","last_clean_scrub_stamp":"2023-10-30T13:26:00.613566+0000","objects_scrubbed":48,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:00:35.001259+0000","scrub_duration":0.17839571300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":542,"num_read_kb":359029,"num_write":1093,"num_write_kb":88246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":622199,"num_omap_keys":1035,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"85'2113","reported_seq":4164,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:55.071692+0000","last_change":"2023-10-30T13:25:55.071766+0000","last_active":"2023-10-30T13:25:55.071692+0000","last_peered":"2023-10-30T13:25:55.071692+0000","last_clean":"2023-10-30T13:25:55.071692+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T13:25:55.071692+0000","last_undegraded":"2023-10-30T13:25:55.071692+0000","last_fullsized":"2023-10-30T13:25:55.071692+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2113","last_scrub_stamp":"2023-10-30T13:25:55.071614+0000","last_deep_scrub":"85'2113","last_deep_scrub_stamp":"2023-10-30T13:25:55.071614+0000","last_clean_scrub_stamp":"2023-10-30T13:25:55.071614+0000","objects_scrubbed":589,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:07.445157+0000","scrub_duration":0.93122268500000005,"objects_trimmed":0,"snaptrim_duration":0.0083101389999999994,"stat_sum":{"num_bytes":18904879,"num_objects":589,"num_object_clones":276,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":211,"num_read":147,"num_read_kb":20021,"num_write":1340,"num_write_kb":25244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"85'789","reported_seq":1200,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:49.688428+0000","last_change":"2023-10-30T13:25:49.688470+0000","last_active":"2023-10-30T13:25:49.688428+0000","last_peered":"2023-10-30T13:25:49.688428+0000","last_clean":"2023-10-30T13:25:49.688428+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T13:25:49.688428+0000","last_undegraded":"2023-10-30T13:25:49.688428+0000","last_fullsized":"2023-10-30T13:25:49.688428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'789","last_scrub_stamp":"2023-10-30T13:25:49.688344+0000","last_deep_scrub":"85'789","last_deep_scrub_stamp":"2023-10-30T13:25:49.688344+0000","last_clean_scrub_stamp":"2023-10-30T13:25:49.688344+0000","objects_scrubbed":53,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:56.666560+0000","scrub_duration":0.15913772100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":616,"num_read_kb":251423,"num_write":890,"num_write_kb":55054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":627193,"num_omap_keys":1090,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"85'2098","reported_seq":4145,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.184775+0000","last_change":"2023-10-30T13:25:57.184868+0000","last_active":"2023-10-30T13:25:57.184775+0000","last_peered":"2023-10-30T13:25:57.184775+0000","last_clean":"2023-10-30T13:25:57.184775+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T13:25:57.184775+0000","last_undegraded":"2023-10-30T13:25:57.184775+0000","last_fullsized":"2023-10-30T13:25:57.184775+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2098","last_scrub_stamp":"2023-10-30T13:25:57.184676+0000","last_deep_scrub":"85'2098","last_deep_scrub_stamp":"2023-10-30T13:25:57.184676+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.184676+0000","objects_scrubbed":580,"log_size":2098,"log_dups_size":0,"ondisk_log_size":2098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:36.904344+0000","scrub_duration":1.084383391,"objects_trimmed":0,"snaptrim_duration":0.0043045619999999996,"stat_sum":{"num_bytes":10134301,"num_objects":580,"num_object_clones":271,"num_object_copies":1160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":580,"num_whiteouts":219,"num_read":138,"num_read_kb":8986,"num_write":1318,"num_write_kb":11276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"85'1309","reported_seq":1758,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.351882+0000","last_change":"2023-10-30T13:25:54.351972+0000","last_active":"2023-10-30T13:25:54.351882+0000","last_peered":"2023-10-30T13:25:54.351882+0000","last_clean":"2023-10-30T13:25:54.351882+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T13:25:54.351882+0000","last_undegraded":"2023-10-30T13:25:54.351882+0000","last_fullsized":"2023-10-30T13:25:54.351882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1309","last_scrub_stamp":"2023-10-30T13:25:54.351769+0000","last_deep_scrub":"85'1309","last_deep_scrub_stamp":"2023-10-30T13:25:54.351769+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.351769+0000","objects_scrubbed":62,"log_size":1309,"log_dups_size":0,"ondisk_log_size":1309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:15.955314+0000","scrub_duration":0.18331900800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":659,"num_read_kb":339384,"num_write":1492,"num_write_kb":80693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":875213,"num_omap_keys":1399,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"85'2223","reported_seq":4246,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:59.747975+0000","last_change":"2023-10-30T13:25:59.748025+0000","last_active":"2023-10-30T13:25:59.747975+0000","last_peered":"2023-10-30T13:25:59.747975+0000","last_clean":"2023-10-30T13:25:59.747975+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T13:25:59.747975+0000","last_undegraded":"2023-10-30T13:25:59.747975+0000","last_fullsized":"2023-10-30T13:25:59.747975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2223","last_scrub_stamp":"2023-10-30T13:25:59.747898+0000","last_deep_scrub":"85'2223","last_deep_scrub_stamp":"2023-10-30T13:25:59.747898+0000","last_clean_scrub_stamp":"2023-10-30T13:25:59.747898+0000","objects_scrubbed":620,"log_size":2223,"log_dups_size":0,"ondisk_log_size":2223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:22.443096+0000","scrub_duration":1.320524545,"objects_trimmed":0,"snaptrim_duration":0.015702779,"stat_sum":{"num_bytes":15773172,"num_objects":620,"num_object_clones":292,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":204,"num_read":135,"num_read_kb":17351,"num_write":1431,"num_write_kb":17077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"85'1282","reported_seq":1786,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:43.374996+0000","last_change":"2023-10-30T13:25:43.375075+0000","last_active":"2023-10-30T13:25:43.374996+0000","last_peered":"2023-10-30T13:25:43.374996+0000","last_clean":"2023-10-30T13:25:43.374996+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T13:25:43.374996+0000","last_undegraded":"2023-10-30T13:25:43.374996+0000","last_fullsized":"2023-10-30T13:25:43.374996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1282","last_scrub_stamp":"2023-10-30T13:25:43.374894+0000","last_deep_scrub":"85'1282","last_deep_scrub_stamp":"2023-10-30T13:25:43.374894+0000","last_clean_scrub_stamp":"2023-10-30T13:25:43.374894+0000","objects_scrubbed":59,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:32.582441+0000","scrub_duration":0.20001206899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19774519,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":862,"num_read_kb":221068,"num_write":1454,"num_write_kb":51536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":878727,"num_omap_keys":1229,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"82'1515","reported_seq":3525,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:10.435008+0000","last_change":"2023-10-30T12:37:07.178872+0000","last_active":"2023-10-30T13:25:10.435008+0000","last_peered":"2023-10-30T13:25:10.435008+0000","last_clean":"2023-10-30T13:25:10.435008+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T13:25:10.435008+0000","last_undegraded":"2023-10-30T13:25:10.435008+0000","last_fullsized":"2023-10-30T13:25:10.435008+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1515,"log_dups_size":0,"ondisk_log_size":1515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080610860000000003,"stat_sum":{"num_bytes":22077043,"num_objects":612,"num_object_clones":281,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":230,"num_read":141,"num_read_kb":18856,"num_write":1294,"num_write_kb":28337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"85'1397","reported_seq":1954,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:41.415232+0000","last_change":"2023-10-30T13:25:41.415318+0000","last_active":"2023-10-30T13:25:41.415232+0000","last_peered":"2023-10-30T13:25:41.415232+0000","last_clean":"2023-10-30T13:25:41.415232+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T13:25:41.415232+0000","last_undegraded":"2023-10-30T13:25:41.415232+0000","last_fullsized":"2023-10-30T13:25:41.415232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1397","last_scrub_stamp":"2023-10-30T13:25:41.415138+0000","last_deep_scrub":"85'1397","last_deep_scrub_stamp":"2023-10-30T13:25:41.415138+0000","last_clean_scrub_stamp":"2023-10-30T13:25:41.415138+0000","objects_scrubbed":67,"log_size":1397,"log_dups_size":0,"ondisk_log_size":1397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:14.524619+0000","scrub_duration":0.28929933099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":865,"num_read_kb":420163,"num_write":1566,"num_write_kb":89397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":814706,"num_omap_keys":1307,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"85'2141","reported_seq":4172,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:01.199914+0000","last_change":"2023-10-30T13:26:01.200009+0000","last_active":"2023-10-30T13:26:01.199914+0000","last_peered":"2023-10-30T13:26:01.199914+0000","last_clean":"2023-10-30T13:26:01.199914+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T13:26:01.199914+0000","last_undegraded":"2023-10-30T13:26:01.199914+0000","last_fullsized":"2023-10-30T13:26:01.199914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2141","last_scrub_stamp":"2023-10-30T13:26:01.199816+0000","last_deep_scrub":"85'2141","last_deep_scrub_stamp":"2023-10-30T13:26:01.199816+0000","last_clean_scrub_stamp":"2023-10-30T13:26:01.199816+0000","objects_scrubbed":606,"log_size":2141,"log_dups_size":0,"ondisk_log_size":2141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:21:03.156283+0000","scrub_duration":1.132667525,"objects_trimmed":0,"snaptrim_duration":0.008640567,"stat_sum":{"num_bytes":23022369,"num_objects":606,"num_object_clones":279,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":220,"num_read":129,"num_read_kb":18941,"num_write":1340,"num_write_kb":24878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"85'1164","reported_seq":1635,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.276943+0000","last_change":"2023-10-30T13:25:48.277044+0000","last_active":"2023-10-30T13:25:48.276943+0000","last_peered":"2023-10-30T13:25:48.276943+0000","last_clean":"2023-10-30T13:25:48.276943+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T13:25:48.276943+0000","last_undegraded":"2023-10-30T13:25:48.276943+0000","last_fullsized":"2023-10-30T13:25:48.276943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1164","last_scrub_stamp":"2023-10-30T13:25:48.276852+0000","last_deep_scrub":"85'1164","last_deep_scrub_stamp":"2023-10-30T13:25:48.276852+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.276852+0000","objects_scrubbed":66,"log_size":1164,"log_dups_size":0,"ondisk_log_size":1164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:06.856285+0000","scrub_duration":0.19954839099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554466,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":701,"num_read_kb":318483,"num_write":1303,"num_write_kb":68223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":676667,"num_omap_keys":1245,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"85'2197","reported_seq":4231,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:52.017102+0000","last_change":"2023-10-30T13:25:52.017193+0000","last_active":"2023-10-30T13:25:52.017102+0000","last_peered":"2023-10-30T13:25:52.017102+0000","last_clean":"2023-10-30T13:25:52.017102+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T13:25:52.017102+0000","last_undegraded":"2023-10-30T13:25:52.017102+0000","last_fullsized":"2023-10-30T13:25:52.017102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2197","last_scrub_stamp":"2023-10-30T13:25:52.016976+0000","last_deep_scrub":"85'2197","last_deep_scrub_stamp":"2023-10-30T13:25:52.016976+0000","last_clean_scrub_stamp":"2023-10-30T13:25:52.016976+0000","objects_scrubbed":613,"log_size":2197,"log_dups_size":0,"ondisk_log_size":2197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:09:13.519760+0000","scrub_duration":1.011428851,"objects_trimmed":0,"snaptrim_duration":0.0073009930000000004,"stat_sum":{"num_bytes":10084838,"num_objects":613,"num_object_clones":288,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":217,"num_read":139,"num_read_kb":5686,"num_write":1383,"num_write_kb":20147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'729","reported_seq":1132,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.578886+0000","last_change":"2023-10-30T13:25:54.578930+0000","last_active":"2023-10-30T13:25:54.578886+0000","last_peered":"2023-10-30T13:25:54.578886+0000","last_clean":"2023-10-30T13:25:54.578886+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T13:25:54.578886+0000","last_undegraded":"2023-10-30T13:25:54.578886+0000","last_fullsized":"2023-10-30T13:25:54.578886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'729","last_scrub_stamp":"2023-10-30T13:25:54.578800+0000","last_deep_scrub":"85'729","last_deep_scrub_stamp":"2023-10-30T13:25:54.578800+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.578800+0000","objects_scrubbed":53,"log_size":729,"log_dups_size":0,"ondisk_log_size":729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:06:15.394870+0000","scrub_duration":0.12999461300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":600,"num_read_kb":208389,"num_write":899,"num_write_kb":51587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":569345,"num_omap_keys":982,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"82'1642","reported_seq":3653,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.178174+0000","last_change":"2023-10-30T12:37:07.180453+0000","last_active":"2023-10-30T13:25:16.178174+0000","last_peered":"2023-10-30T13:25:16.178174+0000","last_clean":"2023-10-30T13:25:16.178174+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T13:25:16.178174+0000","last_undegraded":"2023-10-30T13:25:16.178174+0000","last_fullsized":"2023-10-30T13:25:16.178174+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1642,"log_dups_size":0,"ondisk_log_size":1642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091258150000000007,"stat_sum":{"num_bytes":19392609,"num_objects":655,"num_object_clones":299,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":233,"num_read":161,"num_read_kb":7475,"num_write":1440,"num_write_kb":20643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"85'1416","reported_seq":1810,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:40.394192+0000","last_change":"2023-10-30T13:25:40.394262+0000","last_active":"2023-10-30T13:25:40.394192+0000","last_peered":"2023-10-30T13:25:40.394192+0000","last_clean":"2023-10-30T13:25:40.394192+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T13:25:40.394192+0000","last_undegraded":"2023-10-30T13:25:40.394192+0000","last_fullsized":"2023-10-30T13:25:40.394192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1416","last_scrub_stamp":"2023-10-30T13:25:40.394118+0000","last_deep_scrub":"85'1416","last_deep_scrub_stamp":"2023-10-30T13:25:40.394118+0000","last_clean_scrub_stamp":"2023-10-30T13:25:40.394118+0000","objects_scrubbed":45,"log_size":1416,"log_dups_size":0,"ondisk_log_size":1416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:50:35.732748+0000","scrub_duration":0.10710792099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":539,"num_read_kb":266844,"num_write":1512,"num_write_kb":63647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":566794,"num_omap_keys":1066,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"85'2071","reported_seq":4113,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:58.521640+0000","last_change":"2023-10-30T13:25:58.521719+0000","last_active":"2023-10-30T13:25:58.521640+0000","last_peered":"2023-10-30T13:25:58.521640+0000","last_clean":"2023-10-30T13:25:58.521640+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T13:25:58.521640+0000","last_undegraded":"2023-10-30T13:25:58.521640+0000","last_fullsized":"2023-10-30T13:25:58.521640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2071","last_scrub_stamp":"2023-10-30T13:25:58.521516+0000","last_deep_scrub":"85'2071","last_deep_scrub_stamp":"2023-10-30T13:25:58.521516+0000","last_clean_scrub_stamp":"2023-10-30T13:25:58.521516+0000","objects_scrubbed":590,"log_size":2071,"log_dups_size":0,"ondisk_log_size":2071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:24:10.505945+0000","scrub_duration":1.0833768969999999,"objects_trimmed":0,"snaptrim_duration":0.001646376,"stat_sum":{"num_bytes":13778361,"num_objects":590,"num_object_clones":278,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":227,"num_read":137,"num_read_kb":5723,"num_write":1258,"num_write_kb":14976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"85'1020","reported_seq":1516,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:45.278920+0000","last_change":"2023-10-30T13:25:45.278968+0000","last_active":"2023-10-30T13:25:45.278920+0000","last_peered":"2023-10-30T13:25:45.278920+0000","last_clean":"2023-10-30T13:25:45.278920+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T13:25:45.278920+0000","last_undegraded":"2023-10-30T13:25:45.278920+0000","last_fullsized":"2023-10-30T13:25:45.278920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1020","last_scrub_stamp":"2023-10-30T13:25:45.278835+0000","last_deep_scrub":"85'1020","last_deep_scrub_stamp":"2023-10-30T13:25:45.278835+0000","last_clean_scrub_stamp":"2023-10-30T13:25:45.278835+0000","objects_scrubbed":50,"log_size":1020,"log_dups_size":0,"ondisk_log_size":1020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:18:17.671466+0000","scrub_duration":0.27418579599999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554544,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":863,"num_read_kb":272228,"num_write":1253,"num_write_kb":56723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":534875,"num_omap_keys":741,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"85'1984","reported_seq":4002,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-30T13:26:18.098887+0000","last_change":"2023-10-30T13:26:17.471949+0000","last_active":"2023-10-30T13:26:18.098887+0000","last_peered":"2023-10-30T13:26:18.098887+0000","last_clean":"2023-10-30T13:26:18.098887+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T13:26:18.098887+0000","last_undegraded":"2023-10-30T13:26:18.098887+0000","last_fullsized":"2023-10-30T13:26:18.098887+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":368,"log_size":1984,"log_dups_size":0,"ondisk_log_size":1984,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0098326820000000006,"stat_sum":{"num_bytes":10901701,"num_objects":622,"num_object_clones":292,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":232,"num_read":163,"num_read_kb":14238,"num_write":1406,"num_write_kb":21652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"85'696","reported_seq":1065,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:42.423508+0000","last_change":"2023-10-30T13:25:42.423609+0000","last_active":"2023-10-30T13:25:42.423508+0000","last_peered":"2023-10-30T13:25:42.423508+0000","last_clean":"2023-10-30T13:25:42.423508+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T13:25:42.423508+0000","last_undegraded":"2023-10-30T13:25:42.423508+0000","last_fullsized":"2023-10-30T13:25:42.423508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'696","last_scrub_stamp":"2023-10-30T13:25:42.423407+0000","last_deep_scrub":"85'696","last_deep_scrub_stamp":"2023-10-30T13:25:42.423407+0000","last_clean_scrub_stamp":"2023-10-30T13:25:42.423407+0000","objects_scrubbed":41,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:12:00.329817+0000","scrub_duration":0.18306038499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":443,"num_read_kb":261670,"num_write":858,"num_write_kb":58986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":305625,"num_omap_keys":524,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"85'2220","reported_seq":4221,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.104935+0000","last_change":"2023-10-30T13:25:53.105023+0000","last_active":"2023-10-30T13:25:53.104935+0000","last_peered":"2023-10-30T13:25:53.104935+0000","last_clean":"2023-10-30T13:25:53.104935+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T13:25:53.104935+0000","last_undegraded":"2023-10-30T13:25:53.104935+0000","last_fullsized":"2023-10-30T13:25:53.104935+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2220","last_scrub_stamp":"2023-10-30T13:25:53.104840+0000","last_deep_scrub":"85'2220","last_deep_scrub_stamp":"2023-10-30T13:25:53.104840+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.104840+0000","objects_scrubbed":647,"log_size":2220,"log_dups_size":0,"ondisk_log_size":2220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:47:46.541296+0000","scrub_duration":1.013459932,"objects_trimmed":0,"snaptrim_duration":0.004602437,"stat_sum":{"num_bytes":20428330,"num_objects":647,"num_object_clones":304,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":239,"num_read":119,"num_read_kb":13097,"num_write":1360,"num_write_kb":26395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"85'779","reported_seq":1239,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.328717+0000","last_change":"2023-10-30T13:25:48.328798+0000","last_active":"2023-10-30T13:25:48.328717+0000","last_peered":"2023-10-30T13:25:48.328717+0000","last_clean":"2023-10-30T13:25:48.328717+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T13:25:48.328717+0000","last_undegraded":"2023-10-30T13:25:48.328717+0000","last_fullsized":"2023-10-30T13:25:48.328717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'779","last_scrub_stamp":"2023-10-30T13:25:48.328628+0000","last_deep_scrub":"85'779","last_deep_scrub_stamp":"2023-10-30T13:25:48.328628+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.328628+0000","objects_scrubbed":62,"log_size":779,"log_dups_size":0,"ondisk_log_size":779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:51.114462+0000","scrub_duration":0.22182811399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":635,"num_read_kb":434171,"num_write":906,"num_write_kb":88121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":675117,"num_omap_keys":1207,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"85'2216","reported_seq":4237,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:59.936763+0000","last_change":"2023-10-30T13:25:59.936829+0000","last_active":"2023-10-30T13:25:59.936763+0000","last_peered":"2023-10-30T13:25:59.936763+0000","last_clean":"2023-10-30T13:25:59.936763+0000","last_became_active":"2023-10-30T12:29:27.340504+0000","last_became_peered":"2023-10-30T12:29:27.340504+0000","last_unstale":"2023-10-30T13:25:59.936763+0000","last_undegraded":"2023-10-30T13:25:59.936763+0000","last_fullsized":"2023-10-30T13:25:59.936763+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2216","last_scrub_stamp":"2023-10-30T13:25:59.936683+0000","last_deep_scrub":"85'2216","last_deep_scrub_stamp":"2023-10-30T13:25:59.936683+0000","last_clean_scrub_stamp":"2023-10-30T13:25:59.936683+0000","objects_scrubbed":641,"log_size":2216,"log_dups_size":0,"ondisk_log_size":2216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:57:07.443321+0000","scrub_duration":1.7189143169999999,"objects_trimmed":0,"snaptrim_duration":0.00653923,"stat_sum":{"num_bytes":23737065,"num_objects":641,"num_object_clones":299,"num_object_copies":1282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":641,"num_whiteouts":238,"num_read":138,"num_read_kb":28255,"num_write":1351,"num_write_kb":33550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'766","reported_seq":1157,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:51.439924+0000","last_change":"2023-10-30T13:25:51.439991+0000","last_active":"2023-10-30T13:25:51.439924+0000","last_peered":"2023-10-30T13:25:51.439924+0000","last_clean":"2023-10-30T13:25:51.439924+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T13:25:51.439924+0000","last_undegraded":"2023-10-30T13:25:51.439924+0000","last_fullsized":"2023-10-30T13:25:51.439924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'766","last_scrub_stamp":"2023-10-30T13:25:51.439810+0000","last_deep_scrub":"85'766","last_deep_scrub_stamp":"2023-10-30T13:25:51.439810+0000","last_clean_scrub_stamp":"2023-10-30T13:25:51.439810+0000","objects_scrubbed":50,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:56.709508+0000","scrub_duration":0.13294180999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364369,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":493,"num_read_kb":320778,"num_write":891,"num_write_kb":71265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":522473,"num_omap_keys":1010,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'2210","reported_seq":4257,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:08.440439+0000","last_change":"2023-10-30T13:26:08.440557+0000","last_active":"2023-10-30T13:26:08.440439+0000","last_peered":"2023-10-30T13:26:08.440439+0000","last_clean":"2023-10-30T13:26:08.440439+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T13:26:08.440439+0000","last_undegraded":"2023-10-30T13:26:08.440439+0000","last_fullsized":"2023-10-30T13:26:08.440439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2210","last_scrub_stamp":"2023-10-30T13:26:08.440312+0000","last_deep_scrub":"85'2210","last_deep_scrub_stamp":"2023-10-30T13:26:08.440312+0000","last_clean_scrub_stamp":"2023-10-30T13:26:08.440312+0000","objects_scrubbed":611,"log_size":2210,"log_dups_size":0,"ondisk_log_size":2210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:49.728570+0000","scrub_duration":1.28748733,"objects_trimmed":0,"snaptrim_duration":0.0076673150000000001,"stat_sum":{"num_bytes":18256447,"num_objects":611,"num_object_clones":286,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":229,"num_read":149,"num_read_kb":24381,"num_write":1383,"num_write_kb":20195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"85'645","reported_seq":1007,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.161404+0000","last_change":"2023-10-30T13:25:48.161534+0000","last_active":"2023-10-30T13:25:48.161404+0000","last_peered":"2023-10-30T13:25:48.161404+0000","last_clean":"2023-10-30T13:25:48.161404+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T13:25:48.161404+0000","last_undegraded":"2023-10-30T13:25:48.161404+0000","last_fullsized":"2023-10-30T13:25:48.161404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'645","last_scrub_stamp":"2023-10-30T13:25:48.161279+0000","last_deep_scrub":"85'645","last_deep_scrub_stamp":"2023-10-30T13:25:48.161279+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.161279+0000","objects_scrubbed":38,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:29.280017+0000","scrub_duration":0.15056981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360710,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":462,"num_read_kb":342669,"num_write":689,"num_write_kb":66904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":355436,"num_omap_keys":678,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"82'1520","reported_seq":3503,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:16.232741+0000","last_change":"2023-10-30T12:37:07.182282+0000","last_active":"2023-10-30T13:25:16.232741+0000","last_peered":"2023-10-30T13:25:16.232741+0000","last_clean":"2023-10-30T13:25:16.232741+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T13:25:16.232741+0000","last_undegraded":"2023-10-30T13:25:16.232741+0000","last_fullsized":"2023-10-30T13:25:16.232741+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1520,"log_dups_size":0,"ondisk_log_size":1520,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010286791999999999,"stat_sum":{"num_bytes":26944901,"num_objects":612,"num_object_clones":287,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":236,"num_read":119,"num_read_kb":34683,"num_write":1299,"num_write_kb":33129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"85'119","reported_seq":267,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:46.486940+0000","last_change":"2023-10-30T13:25:46.486986+0000","last_active":"2023-10-30T13:25:46.486940+0000","last_peered":"2023-10-30T13:25:46.486940+0000","last_clean":"2023-10-30T13:25:46.486940+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T13:25:46.486940+0000","last_undegraded":"2023-10-30T13:25:46.486940+0000","last_fullsized":"2023-10-30T13:25:46.486940+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'119","last_scrub_stamp":"2023-10-30T13:25:46.486855+0000","last_deep_scrub":"85'119","last_deep_scrub_stamp":"2023-10-30T13:25:46.486855+0000","last_clean_scrub_stamp":"2023-10-30T13:25:46.486855+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:13.395892+0000","scrub_duration":0.0083870119999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'1371","reported_seq":1882,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:10.600505+0000","last_change":"2023-10-30T13:26:10.600590+0000","last_active":"2023-10-30T13:26:10.600505+0000","last_peered":"2023-10-30T13:26:10.600505+0000","last_clean":"2023-10-30T13:26:10.600505+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T13:26:10.600505+0000","last_undegraded":"2023-10-30T13:26:10.600505+0000","last_fullsized":"2023-10-30T13:26:10.600505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1371","last_scrub_stamp":"2023-10-30T13:26:10.600409+0000","last_deep_scrub":"85'1371","last_deep_scrub_stamp":"2023-10-30T13:26:10.600409+0000","last_clean_scrub_stamp":"2023-10-30T13:26:10.600409+0000","objects_scrubbed":53,"log_size":1371,"log_dups_size":0,"ondisk_log_size":1371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:15.701205+0000","scrub_duration":0.14720427899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165890,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":877,"num_read_kb":231219,"num_write":1489,"num_write_kb":63137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":475860,"num_omap_keys":919,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"85'2200","reported_seq":4242,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.149503+0000","last_change":"2023-10-30T13:25:57.149588+0000","last_active":"2023-10-30T13:25:57.149503+0000","last_peered":"2023-10-30T13:25:57.149503+0000","last_clean":"2023-10-30T13:25:57.149503+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T13:25:57.149503+0000","last_undegraded":"2023-10-30T13:25:57.149503+0000","last_fullsized":"2023-10-30T13:25:57.149503+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2200","last_scrub_stamp":"2023-10-30T13:25:57.149395+0000","last_deep_scrub":"85'2200","last_deep_scrub_stamp":"2023-10-30T13:25:57.149395+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.149395+0000","objects_scrubbed":626,"log_size":2200,"log_dups_size":0,"ondisk_log_size":2200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:04.364636+0000","scrub_duration":1.0072585730000001,"objects_trimmed":0,"snaptrim_duration":0.0068470270000000003,"stat_sum":{"num_bytes":8097312,"num_objects":626,"num_object_clones":290,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":225,"num_read":146,"num_read_kb":2079,"num_write":1425,"num_write_kb":13192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"85'2234","reported_seq":2615,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.726414+0000","last_change":"2023-10-30T13:25:57.726488+0000","last_active":"2023-10-30T13:25:57.726414+0000","last_peered":"2023-10-30T13:25:57.726414+0000","last_clean":"2023-10-30T13:25:57.726414+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T13:25:57.726414+0000","last_undegraded":"2023-10-30T13:25:57.726414+0000","last_fullsized":"2023-10-30T13:25:57.726414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2234","last_scrub_stamp":"2023-10-30T13:25:57.726328+0000","last_deep_scrub":"85'2234","last_deep_scrub_stamp":"2023-10-30T13:25:57.726328+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.726328+0000","objects_scrubbed":42,"log_size":2234,"log_dups_size":0,"ondisk_log_size":2234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:51.655678+0000","scrub_duration":0.194142807,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":516,"num_read_kb":283293,"num_write":2369,"num_write_kb":72030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":392036,"num_omap_keys":649,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"85'2035","reported_seq":4089,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:05.024953+0000","last_change":"2023-10-30T13:26:05.025106+0000","last_active":"2023-10-30T13:26:05.024953+0000","last_peered":"2023-10-30T13:26:05.024953+0000","last_clean":"2023-10-30T13:26:05.024953+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T13:26:05.024953+0000","last_undegraded":"2023-10-30T13:26:05.024953+0000","last_fullsized":"2023-10-30T13:26:05.024953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2035","last_scrub_stamp":"2023-10-30T13:26:05.024740+0000","last_deep_scrub":"85'2035","last_deep_scrub_stamp":"2023-10-30T13:26:05.024740+0000","last_clean_scrub_stamp":"2023-10-30T13:26:05.024740+0000","objects_scrubbed":570,"log_size":2035,"log_dups_size":0,"ondisk_log_size":2035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:48:20.536984+0000","scrub_duration":1.046857548,"objects_trimmed":0,"snaptrim_duration":0.0057755250000000001,"stat_sum":{"num_bytes":14269585,"num_objects":570,"num_object_clones":262,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":207,"num_read":132,"num_read_kb":18022,"num_write":1264,"num_write_kb":17917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'1898","reported_seq":3194,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:52.380541+0000","last_change":"2023-10-30T13:25:52.380604+0000","last_active":"2023-10-30T13:25:52.380541+0000","last_peered":"2023-10-30T13:25:52.380541+0000","last_clean":"2023-10-30T13:25:52.380541+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T13:25:52.380541+0000","last_undegraded":"2023-10-30T13:25:52.380541+0000","last_fullsized":"2023-10-30T13:25:52.380541+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1898","last_scrub_stamp":"2023-10-30T13:25:52.380459+0000","last_deep_scrub":"85'1898","last_deep_scrub_stamp":"2023-10-30T13:25:52.380459+0000","last_clean_scrub_stamp":"2023-10-30T13:25:52.380459+0000","objects_scrubbed":61,"log_size":1898,"log_dups_size":0,"ondisk_log_size":1898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:04.920583+0000","scrub_duration":0.247546554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49235352,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":1514,"num_read_kb":360508,"num_write":2035,"num_write_kb":92558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":705593,"num_omap_keys":1067,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'2175","reported_seq":4220,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:20.787292+0000","last_change":"2023-10-30T13:26:20.787360+0000","last_active":"2023-10-30T13:26:20.787292+0000","last_peered":"2023-10-30T13:26:20.787292+0000","last_clean":"2023-10-30T13:26:20.787292+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T13:26:20.787292+0000","last_undegraded":"2023-10-30T13:26:20.787292+0000","last_fullsized":"2023-10-30T13:26:20.787292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2175","last_scrub_stamp":"2023-10-30T13:26:20.787197+0000","last_deep_scrub":"85'2175","last_deep_scrub_stamp":"2023-10-30T13:26:20.787197+0000","last_clean_scrub_stamp":"2023-10-30T13:26:20.787197+0000","objects_scrubbed":629,"log_size":2175,"log_dups_size":0,"ondisk_log_size":2175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:46.322449+0000","scrub_duration":1.0662295260000001,"objects_trimmed":0,"snaptrim_duration":0.015331312,"stat_sum":{"num_bytes":28178512,"num_objects":629,"num_object_clones":288,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":231,"num_read":148,"num_read_kb":41809,"num_write":1347,"num_write_kb":29520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'1249","reported_seq":1669,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:09.605267+0000","last_change":"2023-10-30T13:26:09.605318+0000","last_active":"2023-10-30T13:26:09.605267+0000","last_peered":"2023-10-30T13:26:09.605267+0000","last_clean":"2023-10-30T13:26:09.605267+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T13:26:09.605267+0000","last_undegraded":"2023-10-30T13:26:09.605267+0000","last_fullsized":"2023-10-30T13:26:09.605267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1249","last_scrub_stamp":"2023-10-30T13:26:09.605188+0000","last_deep_scrub":"85'1249","last_deep_scrub_stamp":"2023-10-30T13:26:09.605188+0000","last_clean_scrub_stamp":"2023-10-30T13:26:09.605188+0000","objects_scrubbed":53,"log_size":1249,"log_dups_size":0,"ondisk_log_size":1249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:52.759600+0000","scrub_duration":0.149468351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":573,"num_read_kb":334248,"num_write":1338,"num_write_kb":75895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":744856,"num_omap_keys":1335,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"85'2208","reported_seq":4242,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:02.512036+0000","last_change":"2023-10-30T13:26:02.512109+0000","last_active":"2023-10-30T13:26:02.512036+0000","last_peered":"2023-10-30T13:26:02.512036+0000","last_clean":"2023-10-30T13:26:02.512036+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T13:26:02.512036+0000","last_undegraded":"2023-10-30T13:26:02.512036+0000","last_fullsized":"2023-10-30T13:26:02.512036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2208","last_scrub_stamp":"2023-10-30T13:26:02.511965+0000","last_deep_scrub":"85'2208","last_deep_scrub_stamp":"2023-10-30T13:26:02.511965+0000","last_clean_scrub_stamp":"2023-10-30T13:26:02.511965+0000","objects_scrubbed":627,"log_size":2208,"log_dups_size":0,"ondisk_log_size":2208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:27.984589+0000","scrub_duration":1.373745343,"objects_trimmed":0,"snaptrim_duration":0.0051193749999999998,"stat_sum":{"num_bytes":23443395,"num_objects":627,"num_object_clones":293,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":236,"num_read":141,"num_read_kb":26943,"num_write":1376,"num_write_kb":26009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"85'1517","reported_seq":1988,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.476829+0000","last_change":"2023-10-30T13:25:44.476908+0000","last_active":"2023-10-30T13:25:44.476829+0000","last_peered":"2023-10-30T13:25:44.476829+0000","last_clean":"2023-10-30T13:25:44.476829+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T13:25:44.476829+0000","last_undegraded":"2023-10-30T13:25:44.476829+0000","last_fullsized":"2023-10-30T13:25:44.476829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1517","last_scrub_stamp":"2023-10-30T13:25:44.476706+0000","last_deep_scrub":"85'1517","last_deep_scrub_stamp":"2023-10-30T13:25:44.476706+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.476706+0000","objects_scrubbed":50,"log_size":1517,"log_dups_size":0,"ondisk_log_size":1517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:43:12.189799+0000","scrub_duration":0.194542509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":699,"num_read_kb":404466,"num_write":1620,"num_write_kb":92575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":728614,"num_omap_keys":1399,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"85'2249","reported_seq":4286,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:50.476286+0000","last_change":"2023-10-30T13:25:50.476388+0000","last_active":"2023-10-30T13:25:50.476286+0000","last_peered":"2023-10-30T13:25:50.476286+0000","last_clean":"2023-10-30T13:25:50.476286+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T13:25:50.476286+0000","last_undegraded":"2023-10-30T13:25:50.476286+0000","last_fullsized":"2023-10-30T13:25:50.476286+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2249","last_scrub_stamp":"2023-10-30T13:25:50.476194+0000","last_deep_scrub":"85'2249","last_deep_scrub_stamp":"2023-10-30T13:25:50.476194+0000","last_clean_scrub_stamp":"2023-10-30T13:25:50.476194+0000","objects_scrubbed":637,"log_size":2249,"log_dups_size":0,"ondisk_log_size":2249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:30:58.348787+0000","scrub_duration":1.2919135900000001,"objects_trimmed":0,"snaptrim_duration":0.0039865179999999997,"stat_sum":{"num_bytes":21812744,"num_objects":637,"num_object_clones":298,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":235,"num_read":149,"num_read_kb":24572,"num_write":1399,"num_write_kb":31699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"85'1056","reported_seq":1475,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:05.605392+0000","last_change":"2023-10-30T13:26:05.605477+0000","last_active":"2023-10-30T13:26:05.605392+0000","last_peered":"2023-10-30T13:26:05.605392+0000","last_clean":"2023-10-30T13:26:05.605392+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T13:26:05.605392+0000","last_undegraded":"2023-10-30T13:26:05.605392+0000","last_fullsized":"2023-10-30T13:26:05.605392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1056","last_scrub_stamp":"2023-10-30T13:26:05.605294+0000","last_deep_scrub":"85'1056","last_deep_scrub_stamp":"2023-10-30T13:26:05.605294+0000","last_clean_scrub_stamp":"2023-10-30T13:26:05.605294+0000","objects_scrubbed":43,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:08:51.834930+0000","scrub_duration":0.16079274700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":534,"num_read_kb":318720,"num_write":1164,"num_write_kb":75805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":590943,"num_omap_keys":951,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"85'2157","reported_seq":4215,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:04.198525+0000","last_change":"2023-10-30T13:26:04.198592+0000","last_active":"2023-10-30T13:26:04.198525+0000","last_peered":"2023-10-30T13:26:04.198525+0000","last_clean":"2023-10-30T13:26:04.198525+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T13:26:04.198525+0000","last_undegraded":"2023-10-30T13:26:04.198525+0000","last_fullsized":"2023-10-30T13:26:04.198525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2157","last_scrub_stamp":"2023-10-30T13:26:04.198444+0000","last_deep_scrub":"85'2157","last_deep_scrub_stamp":"2023-10-30T13:26:04.198444+0000","last_clean_scrub_stamp":"2023-10-30T13:26:04.198444+0000","objects_scrubbed":615,"log_size":2157,"log_dups_size":0,"ondisk_log_size":2157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:17:59.418030+0000","scrub_duration":1.2099730950000001,"objects_trimmed":0,"snaptrim_duration":0.0070372949999999998,"stat_sum":{"num_bytes":38977850,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":222,"num_read":156,"num_read_kb":47793,"num_write":1351,"num_write_kb":41194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1184","reported_seq":1628,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:50.553950+0000","last_change":"2023-10-30T13:25:50.554036+0000","last_active":"2023-10-30T13:25:50.553950+0000","last_peered":"2023-10-30T13:25:50.553950+0000","last_clean":"2023-10-30T13:25:50.553950+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T13:25:50.553950+0000","last_undegraded":"2023-10-30T13:25:50.553950+0000","last_fullsized":"2023-10-30T13:25:50.553950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1184","last_scrub_stamp":"2023-10-30T13:25:50.553862+0000","last_deep_scrub":"85'1184","last_deep_scrub_stamp":"2023-10-30T13:25:50.553862+0000","last_clean_scrub_stamp":"2023-10-30T13:25:50.553862+0000","objects_scrubbed":47,"log_size":1184,"log_dups_size":0,"ondisk_log_size":1184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:34.636484+0000","scrub_duration":0.480914603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":644,"num_read_kb":249397,"num_write":1860,"num_write_kb":63729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":781147,"num_omap_keys":1113,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"85'2052","reported_seq":4082,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:03.318813+0000","last_change":"2023-10-30T13:26:03.318913+0000","last_active":"2023-10-30T13:26:03.318813+0000","last_peered":"2023-10-30T13:26:03.318813+0000","last_clean":"2023-10-30T13:26:03.318813+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T13:26:03.318813+0000","last_undegraded":"2023-10-30T13:26:03.318813+0000","last_fullsized":"2023-10-30T13:26:03.318813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2052","last_scrub_stamp":"2023-10-30T13:26:03.318707+0000","last_deep_scrub":"85'2052","last_deep_scrub_stamp":"2023-10-30T13:26:03.318707+0000","last_clean_scrub_stamp":"2023-10-30T13:26:03.318707+0000","objects_scrubbed":603,"log_size":2052,"log_dups_size":0,"ondisk_log_size":2052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:58.304469+0000","scrub_duration":1.115999714,"objects_trimmed":0,"snaptrim_duration":0.0074996079999999996,"stat_sum":{"num_bytes":14238343,"num_objects":603,"num_object_clones":280,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":229,"num_read":126,"num_read_kb":9383,"num_write":1239,"num_write_kb":19514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"85'541","reported_seq":901,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:08.512407+0000","last_change":"2023-10-30T13:26:08.512451+0000","last_active":"2023-10-30T13:26:08.512407+0000","last_peered":"2023-10-30T13:26:08.512407+0000","last_clean":"2023-10-30T13:26:08.512407+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T13:26:08.512407+0000","last_undegraded":"2023-10-30T13:26:08.512407+0000","last_fullsized":"2023-10-30T13:26:08.512407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'541","last_scrub_stamp":"2023-10-30T13:26:08.512333+0000","last_deep_scrub":"85'541","last_deep_scrub_stamp":"2023-10-30T13:26:08.512333+0000","last_clean_scrub_stamp":"2023-10-30T13:26:08.512333+0000","objects_scrubbed":51,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:40:06.032449+0000","scrub_duration":0.092969488000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21146631,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":514,"num_read_kb":197148,"num_write":651,"num_write_kb":42826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":495891,"num_omap_keys":618,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"85'2296","reported_seq":4306,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:58.407563+0000","last_change":"2023-10-30T13:25:58.407621+0000","last_active":"2023-10-30T13:25:58.407563+0000","last_peered":"2023-10-30T13:25:58.407563+0000","last_clean":"2023-10-30T13:25:58.407563+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T13:25:58.407563+0000","last_undegraded":"2023-10-30T13:25:58.407563+0000","last_fullsized":"2023-10-30T13:25:58.407563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2296","last_scrub_stamp":"2023-10-30T13:25:58.407437+0000","last_deep_scrub":"85'2296","last_deep_scrub_stamp":"2023-10-30T13:25:58.407437+0000","last_clean_scrub_stamp":"2023-10-30T13:25:58.407437+0000","objects_scrubbed":647,"log_size":2296,"log_dups_size":0,"ondisk_log_size":2296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:12.972219+0000","scrub_duration":1.296777326,"objects_trimmed":0,"snaptrim_duration":0.0085509929999999998,"stat_sum":{"num_bytes":14746405,"num_objects":647,"num_object_clones":344,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":203,"num_read":167,"num_read_kb":6583,"num_write":1369,"num_write_kb":17013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'916","reported_seq":1240,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:51.295504+0000","last_change":"2023-10-30T13:25:51.295571+0000","last_active":"2023-10-30T13:25:51.295504+0000","last_peered":"2023-10-30T13:25:51.295504+0000","last_clean":"2023-10-30T13:25:51.295504+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T13:25:51.295504+0000","last_undegraded":"2023-10-30T13:25:51.295504+0000","last_fullsized":"2023-10-30T13:25:51.295504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'916","last_scrub_stamp":"2023-10-30T13:25:51.295393+0000","last_deep_scrub":"85'916","last_deep_scrub_stamp":"2023-10-30T13:25:51.295393+0000","last_clean_scrub_stamp":"2023-10-30T13:25:51.295393+0000","objects_scrubbed":46,"log_size":916,"log_dups_size":0,"ondisk_log_size":916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:13:46.299260+0000","scrub_duration":0.1008542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":472,"num_read_kb":149352,"num_write":1030,"num_write_kb":38491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":548946,"num_omap_keys":1000,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"85'2069","reported_seq":4117,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.461766+0000","last_change":"2023-10-30T13:25:53.461883+0000","last_active":"2023-10-30T13:25:53.461766+0000","last_peered":"2023-10-30T13:25:53.461766+0000","last_clean":"2023-10-30T13:25:53.461766+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T13:25:53.461766+0000","last_undegraded":"2023-10-30T13:25:53.461766+0000","last_fullsized":"2023-10-30T13:25:53.461766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2069","last_scrub_stamp":"2023-10-30T13:25:53.461661+0000","last_deep_scrub":"85'2069","last_deep_scrub_stamp":"2023-10-30T13:25:53.461661+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.461661+0000","objects_scrubbed":584,"log_size":2069,"log_dups_size":0,"ondisk_log_size":2069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:51.529626+0000","scrub_duration":1.0309153659999999,"objects_trimmed":0,"snaptrim_duration":0.0096369869999999996,"stat_sum":{"num_bytes":18113290,"num_objects":584,"num_object_clones":270,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":219,"num_read":134,"num_read_kb":20299,"num_write":1266,"num_write_kb":19768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"85'826","reported_seq":1199,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:49.233063+0000","last_change":"2023-10-30T13:25:49.233269+0000","last_active":"2023-10-30T13:25:49.233063+0000","last_peered":"2023-10-30T13:25:49.233063+0000","last_clean":"2023-10-30T13:25:49.233063+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T13:25:49.233063+0000","last_undegraded":"2023-10-30T13:25:49.233063+0000","last_fullsized":"2023-10-30T13:25:49.233063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'826","last_scrub_stamp":"2023-10-30T13:25:49.232843+0000","last_deep_scrub":"85'826","last_deep_scrub_stamp":"2023-10-30T13:25:49.232843+0000","last_clean_scrub_stamp":"2023-10-30T13:25:49.232843+0000","objects_scrubbed":49,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:29:29.495169+0000","scrub_duration":0.14346297999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777306,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":495,"num_read_kb":170685,"num_write":965,"num_write_kb":43434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":641331,"num_omap_keys":1102,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"85'2182","reported_seq":4222,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.248973+0000","last_change":"2023-10-30T13:25:53.249016+0000","last_active":"2023-10-30T13:25:53.248973+0000","last_peered":"2023-10-30T13:25:53.248973+0000","last_clean":"2023-10-30T13:25:53.248973+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T13:25:53.248973+0000","last_undegraded":"2023-10-30T13:25:53.248973+0000","last_fullsized":"2023-10-30T13:25:53.248973+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2182","last_scrub_stamp":"2023-10-30T13:25:53.248885+0000","last_deep_scrub":"85'2182","last_deep_scrub_stamp":"2023-10-30T13:25:53.248885+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.248885+0000","objects_scrubbed":615,"log_size":2182,"log_dups_size":0,"ondisk_log_size":2182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:04:59.062499+0000","scrub_duration":1.1234317469999999,"objects_trimmed":0,"snaptrim_duration":0.007081922,"stat_sum":{"num_bytes":14751960,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":228,"num_read":137,"num_read_kb":2516,"num_write":1376,"num_write_kb":21368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1177","reported_seq":1676,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:47.247326+0000","last_change":"2023-10-30T13:25:47.247437+0000","last_active":"2023-10-30T13:25:47.247326+0000","last_peered":"2023-10-30T13:25:47.247326+0000","last_clean":"2023-10-30T13:25:47.247326+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T13:25:47.247326+0000","last_undegraded":"2023-10-30T13:25:47.247326+0000","last_fullsized":"2023-10-30T13:25:47.247326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1177","last_scrub_stamp":"2023-10-30T13:25:47.247175+0000","last_deep_scrub":"85'1177","last_deep_scrub_stamp":"2023-10-30T13:25:47.247175+0000","last_clean_scrub_stamp":"2023-10-30T13:25:47.247175+0000","objects_scrubbed":52,"log_size":1177,"log_dups_size":0,"ondisk_log_size":1177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:36.256311+0000","scrub_duration":0.26537599099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":763,"num_read_kb":345628,"num_write":1535,"num_write_kb":76701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":665341,"num_omap_keys":1170,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"85'2030","reported_seq":4076,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:56.194710+0000","last_change":"2023-10-30T13:25:56.194771+0000","last_active":"2023-10-30T13:25:56.194710+0000","last_peered":"2023-10-30T13:25:56.194710+0000","last_clean":"2023-10-30T13:25:56.194710+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T13:25:56.194710+0000","last_undegraded":"2023-10-30T13:25:56.194710+0000","last_fullsized":"2023-10-30T13:25:56.194710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2030","last_scrub_stamp":"2023-10-30T13:25:56.194609+0000","last_deep_scrub":"85'2030","last_deep_scrub_stamp":"2023-10-30T13:25:56.194609+0000","last_clean_scrub_stamp":"2023-10-30T13:25:56.194609+0000","objects_scrubbed":570,"log_size":2030,"log_dups_size":0,"ondisk_log_size":2030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:38.396542+0000","scrub_duration":0.95094084300000004,"objects_trimmed":0,"snaptrim_duration":0.0060337460000000004,"stat_sum":{"num_bytes":23618154,"num_objects":570,"num_object_clones":263,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":209,"num_read":127,"num_read_kb":29654,"num_write":1265,"num_write_kb":27771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"85'1241","reported_seq":1655,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:07.560122+0000","last_change":"2023-10-30T13:26:07.560178+0000","last_active":"2023-10-30T13:26:07.560122+0000","last_peered":"2023-10-30T13:26:07.560122+0000","last_clean":"2023-10-30T13:26:07.560122+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T13:26:07.560122+0000","last_undegraded":"2023-10-30T13:26:07.560122+0000","last_fullsized":"2023-10-30T13:26:07.560122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1241","last_scrub_stamp":"2023-10-30T13:26:07.560043+0000","last_deep_scrub":"85'1241","last_deep_scrub_stamp":"2023-10-30T13:26:07.560043+0000","last_clean_scrub_stamp":"2023-10-30T13:26:07.560043+0000","objects_scrubbed":59,"log_size":1241,"log_dups_size":0,"ondisk_log_size":1241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:34.609715+0000","scrub_duration":0.163669707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":538,"num_read_kb":321476,"num_write":1330,"num_write_kb":79987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":506551,"num_omap_keys":794,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"85'1546","reported_seq":3552,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:23.080184+0000","last_change":"2023-10-30T12:37:07.180393+0000","last_active":"2023-10-30T13:25:23.080184+0000","last_peered":"2023-10-30T13:25:23.080184+0000","last_clean":"2023-10-30T13:25:23.080184+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T13:25:23.080184+0000","last_undegraded":"2023-10-30T13:25:23.080184+0000","last_fullsized":"2023-10-30T13:25:23.080184+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1546,"log_dups_size":0,"ondisk_log_size":1546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091395910000000007,"stat_sum":{"num_bytes":24394000,"num_objects":612,"num_object_clones":279,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":219,"num_read":133,"num_read_kb":28881,"num_write":1364,"num_write_kb":25892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"77'1527","reported_seq":3552,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:40.207075+0000","last_change":"2023-10-30T12:37:07.182718+0000","last_active":"2023-10-30T13:25:40.207075+0000","last_peered":"2023-10-30T13:25:40.207075+0000","last_clean":"2023-10-30T13:25:40.207075+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T13:25:40.207075+0000","last_undegraded":"2023-10-30T13:25:40.207075+0000","last_fullsized":"2023-10-30T13:25:40.207075+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1527,"log_dups_size":0,"ondisk_log_size":1527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010120377999999999,"stat_sum":{"num_bytes":19538275,"num_objects":582,"num_object_clones":275,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":200,"num_read":151,"num_read_kb":26132,"num_write":1354,"num_write_kb":27822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"85'931","reported_seq":1390,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.318568+0000","last_change":"2023-10-30T13:25:44.318568+0000","last_active":"2023-10-30T13:25:44.318568+0000","last_peered":"2023-10-30T13:25:44.318568+0000","last_clean":"2023-10-30T13:25:44.318568+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T13:25:44.318568+0000","last_undegraded":"2023-10-30T13:25:44.318568+0000","last_fullsized":"2023-10-30T13:25:44.318568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'931","last_scrub_stamp":"2023-10-30T13:25:44.318392+0000","last_deep_scrub":"85'931","last_deep_scrub_stamp":"2023-10-30T13:25:44.318392+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.318392+0000","objects_scrubbed":51,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:06.786185+0000","scrub_duration":0.105541015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":739,"num_read_kb":197443,"num_write":1077,"num_write_kb":51357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":687759,"num_omap_keys":1211,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"85'2328","reported_seq":4360,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:02.827670+0000","last_change":"2023-10-30T13:26:02.827670+0000","last_active":"2023-10-30T13:26:02.827670+0000","last_peered":"2023-10-30T13:26:02.827670+0000","last_clean":"2023-10-30T13:26:02.827670+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T13:26:02.827670+0000","last_undegraded":"2023-10-30T13:26:02.827670+0000","last_fullsized":"2023-10-30T13:26:02.827670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2328","last_scrub_stamp":"2023-10-30T13:26:02.827523+0000","last_deep_scrub":"85'2328","last_deep_scrub_stamp":"2023-10-30T13:26:02.827523+0000","last_clean_scrub_stamp":"2023-10-30T13:26:02.827523+0000","objects_scrubbed":686,"log_size":2328,"log_dups_size":0,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:54:56.339329+0000","scrub_duration":1.3553331850000001,"objects_trimmed":0,"snaptrim_duration":0.0011628750000000001,"stat_sum":{"num_bytes":22561655,"num_objects":686,"num_object_clones":314,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":247,"num_read":162,"num_read_kb":29557,"num_write":1441,"num_write_kb":28808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"85'1103","reported_seq":1655,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:55.242833+0000","last_change":"2023-10-30T13:25:55.242833+0000","last_active":"2023-10-30T13:25:55.242833+0000","last_peered":"2023-10-30T13:25:55.242833+0000","last_clean":"2023-10-30T13:25:55.242833+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T13:25:55.242833+0000","last_undegraded":"2023-10-30T13:25:55.242833+0000","last_fullsized":"2023-10-30T13:25:55.242833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1103","last_scrub_stamp":"2023-10-30T13:25:55.242662+0000","last_deep_scrub":"85'1103","last_deep_scrub_stamp":"2023-10-30T13:25:55.242662+0000","last_clean_scrub_stamp":"2023-10-30T13:25:55.242662+0000","objects_scrubbed":55,"log_size":1103,"log_dups_size":0,"ondisk_log_size":1103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:05:46.724096+0000","scrub_duration":0.11943021299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":939,"num_read_kb":433966,"num_write":1214,"num_write_kb":88022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":602554,"num_omap_keys":1058,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"82'1531","reported_seq":3514,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:32.227278+0000","last_change":"2023-10-30T12:37:07.189494+0000","last_active":"2023-10-30T13:25:32.227278+0000","last_peered":"2023-10-30T13:25:32.227278+0000","last_clean":"2023-10-30T13:25:32.227278+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T13:25:32.227278+0000","last_undegraded":"2023-10-30T13:25:32.227278+0000","last_fullsized":"2023-10-30T13:25:32.227278+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":0,"log_size":1531,"log_dups_size":0,"ondisk_log_size":1531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014632805,"stat_sum":{"num_bytes":14611265,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":226,"num_read":123,"num_read_kb":17907,"num_write":1326,"num_write_kb":16683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"85'807","reported_seq":1195,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:50.519115+0000","last_change":"2023-10-30T13:25:50.519195+0000","last_active":"2023-10-30T13:25:50.519115+0000","last_peered":"2023-10-30T13:25:50.519115+0000","last_clean":"2023-10-30T13:25:50.519115+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T13:25:50.519115+0000","last_undegraded":"2023-10-30T13:25:50.519115+0000","last_fullsized":"2023-10-30T13:25:50.519115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'807","last_scrub_stamp":"2023-10-30T13:25:50.519025+0000","last_deep_scrub":"85'807","last_deep_scrub_stamp":"2023-10-30T13:25:50.519025+0000","last_clean_scrub_stamp":"2023-10-30T13:25:50.519025+0000","objects_scrubbed":54,"log_size":807,"log_dups_size":0,"ondisk_log_size":807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:52:13.369063+0000","scrub_duration":0.17624699299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":488,"num_read_kb":379559,"num_write":896,"num_write_kb":83880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":513377,"num_omap_keys":969,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"85'2062","reported_seq":4111,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:16.979624+0000","last_change":"2023-10-30T13:26:16.979757+0000","last_active":"2023-10-30T13:26:16.979624+0000","last_peered":"2023-10-30T13:26:16.979624+0000","last_clean":"2023-10-30T13:26:16.979624+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T13:26:16.979624+0000","last_undegraded":"2023-10-30T13:26:16.979624+0000","last_fullsized":"2023-10-30T13:26:16.979624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2062","last_scrub_stamp":"2023-10-30T13:26:16.979419+0000","last_deep_scrub":"85'2062","last_deep_scrub_stamp":"2023-10-30T13:26:16.979419+0000","last_clean_scrub_stamp":"2023-10-30T13:26:16.979419+0000","objects_scrubbed":581,"log_size":2062,"log_dups_size":0,"ondisk_log_size":2062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:26:59.407523+0000","scrub_duration":1.2670243729999999,"objects_trimmed":0,"snaptrim_duration":0.015871881000000001,"stat_sum":{"num_bytes":15563160,"num_objects":581,"num_object_clones":268,"num_object_copies":1162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":208,"num_read":133,"num_read_kb":9744,"num_write":1268,"num_write_kb":20904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"85'885","reported_seq":1253,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:51.286953+0000","last_change":"2023-10-30T13:25:51.287081+0000","last_active":"2023-10-30T13:25:51.286953+0000","last_peered":"2023-10-30T13:25:51.286953+0000","last_clean":"2023-10-30T13:25:51.286953+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T13:25:51.286953+0000","last_undegraded":"2023-10-30T13:25:51.286953+0000","last_fullsized":"2023-10-30T13:25:51.286953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'885","last_scrub_stamp":"2023-10-30T13:25:51.286817+0000","last_deep_scrub":"85'885","last_deep_scrub_stamp":"2023-10-30T13:25:51.286817+0000","last_clean_scrub_stamp":"2023-10-30T13:25:51.286817+0000","objects_scrubbed":47,"log_size":885,"log_dups_size":0,"ondisk_log_size":885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:21.615461+0000","scrub_duration":0.19579813099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33555014,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":517,"num_read_kb":175570,"num_write":978,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":553400,"num_omap_keys":904,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165214918,"num_objects":39301,"num_object_clones":18317,"num_object_copies":78602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39301,"num_whiteouts":14249,"num_read":9267,"num_read_kb":1285569,"num_write":86732,"num_write_kb":1459764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424127488,"data_stored":2332445068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131918,"ondisk_log_size":131918,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":903126807,"num_objects":1657,"num_object_clones":0,"num_object_copies":3314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":21550,"num_read_kb":9552846,"num_write":41306,"num_write_kb":2210416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":19724250,"num_omap_keys":33398,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1806409728,"data_stored":1806253614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36411,"ondisk_log_size":36411,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477434,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":859516,"kb_used_data":639604,"kb_used_omap":1,"kb_used_meta":219902,"kb_avail":93512324,"statfs":{"total":96636764160,"available":95756619776,"internally_reserved":0,"allocated":654954496,"data_stored":640799640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":225180104},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.41799999999999998,"15min":0.436},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.63200000000000001,"5min":0.66900000000000004,"15min":0.94599999999999995},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.46500000000000002,"15min":0.47399999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":0.63900000000000001,"5min":0.75800000000000001,"15min":0.81899999999999995},"last":0.45600000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.58299999999999996,"5min":0.86099999999999999,"15min":1.0209999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.50600000000000001,"15min":0.52700000000000002},"min":{"1min":0.33000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.60599999999999998,"5min":0.76600000000000001,"15min":0.99299999999999999},"last":0.33000000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.501,"15min":0.504},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.28599999999999998},"max":{"1min":0.56399999999999995,"5min":0.85099999999999998,"15min":0.90800000000000003},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.74399999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.48399999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.75700000000000001,"15min":0.78600000000000003},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.008,"5min":1.05,"15min":1.548},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.80400000000000005,"15min":0.81100000000000005},"min":{"1min":0.56699999999999995,"5min":0.497,"15min":0.497},"max":{"1min":1.1200000000000001,"5min":1.139,"15min":1.6950000000000001},"last":0.76600000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.76700000000000002,"15min":0.80700000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.95899999999999996,"5min":1.1160000000000001,"15min":1.194},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.157,"5min":1.1659999999999999,"15min":1.603},"last":0.63500000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85599999999999998,"15min":0.86399999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.436},"max":{"1min":1.081,"5min":1.198,"15min":1.2270000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.91800000000000004,"15min":0.92300000000000004},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0329999999999999,"5min":1.1639999999999999,"15min":1.3759999999999999},"last":0.97999999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.84099999999999997,"15min":0.84199999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55300000000000005},"max":{"1min":1.137,"5min":1.137,"15min":1.1919999999999999},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.89400000000000002,"15min":0.88},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.47699999999999998},"max":{"1min":0.96599999999999997,"5min":1.1990000000000001,"15min":1.2050000000000001},"last":0.79200000000000004}]}]},{"osd":2,"up_from":16,"seq":68719477435,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":547964,"kb_used_data":412724,"kb_used_omap":1,"kb_used_meta":135230,"kb_avail":93823876,"statfs":{"total":96636764160,"available":96075649024,"internally_reserved":0,"allocated":422629376,"data_stored":409295067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":138475974},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.42999999999999999,"15min":0.43099999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.57799999999999996,"5min":0.81299999999999994,"15min":0.91200000000000003},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.48799999999999999,"15min":0.47899999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.625,"5min":1.554,"15min":2.8519999999999999},"last":0.32400000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.47799999999999998,"15min":0.497},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.57199999999999995,"5min":1.071,"15min":4.9660000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52700000000000002,"15min":0.54600000000000004},"min":{"1min":0.376,"5min":0.32100000000000001,"15min":0.28399999999999997},"max":{"1min":0.66600000000000004,"5min":1.198,"15min":6.0759999999999996},"last":0.44700000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.505,"15min":0.52800000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.314},"max":{"1min":0.65500000000000003,"5min":0.96799999999999997,"15min":4.6120000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.56499999999999995,"15min":0.58699999999999997},"min":{"1min":0.42699999999999999,"5min":0.40899999999999997,"15min":0.379},"max":{"1min":0.68200000000000005,"5min":1.4430000000000001,"15min":4.641},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79100000000000004,"15min":0.80400000000000005},"min":{"1min":0.58399999999999996,"5min":0.48299999999999998,"15min":0.47999999999999998},"max":{"1min":0.95999999999999996,"5min":1.1719999999999999,"15min":3.5499999999999998},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.77000000000000002,"15min":0.81000000000000005},"min":{"1min":0.53200000000000003,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0389999999999999,"5min":1.262,"15min":5.032},"last":0.66900000000000004}]},{"osd":5,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.88800000000000001,"15min":0.87},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.177,"5min":1.494,"15min":6.1289999999999996},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.84699999999999998,"15min":0.877},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.159,"5min":1.3440000000000001,"15min":3.1459999999999999},"last":1.0189999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.83299999999999996,"15min":0.875},"min":{"1min":0.67600000000000005,"5min":0.54000000000000004,"15min":0.53100000000000003},"max":{"1min":1.0469999999999999,"5min":1.1739999999999999,"15min":5.8159999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.82799999999999996,"15min":0.83299999999999996},"min":{"1min":0.66600000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":0.99299999999999999,"5min":1.6299999999999999,"15min":3.415},"last":0.64500000000000002}]},{"osd":7,"last update":"Mon Oct 30 13:25:27 2023","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.84199999999999997,"15min":0.872},"min":{"1min":0.73999999999999999,"5min":0.60699999999999998,"15min":0.51100000000000001},"max":{"1min":1.121,"5min":1.9139999999999999,"15min":4.883},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.88400000000000001,"15min":0.90300000000000002},"min":{"1min":0.79300000000000004,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.0640000000000001,"5min":1.3460000000000001,"15min":4.9950000000000001},"last":0.80800000000000005}]}]},{"osd":1,"up_from":16,"seq":68719477434,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":575080,"kb_used_data":431520,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93796760,"statfs":{"total":96636764160,"available":96047882240,"internally_reserved":0,"allocated":441876480,"data_stored":431229668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":146995654},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.46500000000000002,"15min":0.44800000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.22600000000000001},"max":{"1min":0.63400000000000001,"5min":0.70699999999999996,"15min":0.80400000000000005},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.54200000000000004,"15min":0.52800000000000002},"min":{"1min":0.40899999999999997,"5min":0.31,"15min":0.253},"max":{"1min":0.72499999999999998,"5min":0.91100000000000003,"15min":1.1140000000000001},"last":0.505}]},{"osd":2,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47999999999999998,"15min":0.48999999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68200000000000005,"5min":0.83999999999999997,"15min":5.6120000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56499999999999995,"15min":0.55700000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.34699999999999998},"max":{"1min":0.78200000000000003,"5min":0.82199999999999995,"15min":2.831},"last":0.42499999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47499999999999998,"15min":0.496},"min":{"1min":0.32600000000000001,"5min":0.318,"15min":0.30499999999999999},"max":{"1min":0.60799999999999998,"5min":0.80900000000000005,"15min":7.6900000000000004},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.39600000000000002,"15min":0.34999999999999998},"max":{"1min":0.66700000000000004,"5min":0.91800000000000004,"15min":1.5309999999999999},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84599999999999997,"15min":0.81999999999999995},"min":{"1min":0.58099999999999996,"5min":0.51600000000000001,"15min":0.40400000000000003},"max":{"1min":1.0740000000000001,"5min":1.133,"15min":5.4790000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.90000000000000002,"15min":0.85999999999999999},"min":{"1min":0.72799999999999998,"5min":0.64300000000000002,"15min":0.44400000000000001},"max":{"1min":1.113,"5min":1.1619999999999999,"15min":5.5110000000000001},"last":0.59299999999999997}]},{"osd":5,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.82899999999999996,"15min":0.84099999999999997},"min":{"1min":0.56299999999999994,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1160000000000001,"5min":1.212,"15min":7.5430000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.89600000000000002,"15min":0.91500000000000004},"min":{"1min":0.73599999999999999,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.141,"5min":1.2350000000000001,"15min":7.5750000000000002},"last":0.90200000000000002}]},{"osd":6,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79100000000000004,"15min":0.79800000000000004},"min":{"1min":0.59399999999999997,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.875,"5min":1.177,"15min":5.6639999999999997},"last":0.752},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.80700000000000005,"15min":0.86399999999999999},"min":{"1min":0.64000000000000001,"5min":0.51800000000000002,"15min":0.51600000000000001},"max":{"1min":1.0449999999999999,"5min":1.1140000000000001,"15min":13.036},"last":0.91400000000000003}]},{"osd":7,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.84899999999999998,"15min":0.87},"min":{"1min":0.58399999999999996,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.0960000000000001,"5min":1.2050000000000001,"15min":3.7690000000000001},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.88900000000000001,"15min":0.874},"min":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.501},"max":{"1min":1.0109999999999999,"5min":1.1870000000000001,"15min":4.0659999999999998},"last":0.74399999999999999}]}]},{"osd":0,"up_from":16,"seq":68719477435,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":943496,"kb_used_data":695808,"kb_used_omap":1,"kb_used_meta":247678,"kb_avail":93428344,"statfs":{"total":96636764160,"available":95670624256,"internally_reserved":0,"allocated":712507392,"data_stored":696810667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":253622726},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.40600000000000003,"15min":0.41099999999999998},"min":{"1min":0.254,"5min":0.23899999999999999,"15min":0.22700000000000001},"max":{"1min":0.58099999999999996,"5min":0.61399999999999999,"15min":3.996},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.51700000000000002,"15min":0.52600000000000002},"min":{"1min":0.41099999999999998,"5min":0.36199999999999999,"15min":0.309},"max":{"1min":0.67600000000000005,"5min":0.75600000000000001,"15min":4.3230000000000004},"last":0.41199999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.438,"15min":0.44400000000000001},"min":{"1min":0.308,"5min":0.30399999999999999,"15min":0.27700000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":4.9690000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52100000000000002,"15min":0.53200000000000003},"min":{"1min":0.372,"5min":0.34100000000000003,"15min":0.30199999999999999},"max":{"1min":0.621,"5min":0.878,"15min":5.0049999999999999},"last":0.46200000000000002}]},{"osd":3,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47799999999999998,"15min":0.48799999999999999},"min":{"1min":0.33500000000000002,"5min":0.29299999999999998,"15min":0.28299999999999997},"max":{"1min":0.68100000000000005,"5min":0.82699999999999996,"15min":5.4909999999999997},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57499999999999996,"15min":0.58099999999999996},"min":{"1min":0.443,"5min":0.42799999999999999,"15min":0.38600000000000001},"max":{"1min":0.749,"5min":0.90400000000000003,"15min":5.6970000000000001},"last":0.63900000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82799999999999996,"5min":0.79500000000000004,"15min":0.80900000000000005},"min":{"1min":0.58699999999999997,"5min":0.46700000000000003,"15min":0.377},"max":{"1min":1.111,"5min":1.111,"15min":4.444},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.78900000000000003,"15min":0.80100000000000005},"min":{"1min":0.58499999999999996,"5min":0.55000000000000004,"15min":0.48099999999999998},"max":{"1min":0.90500000000000003,"5min":1.117,"15min":5.7750000000000004},"last":0.745}]},{"osd":5,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.83299999999999996,"15min":0.83499999999999996},"min":{"1min":0.66800000000000004,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.246,"5min":1.3340000000000001,"15min":4.5430000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.871,"15min":0.86399999999999999},"min":{"1min":0.59199999999999997,"5min":0.56999999999999995,"15min":0.47099999999999997},"max":{"1min":1.204,"5min":1.204,"15min":4.4790000000000001},"last":0.72599999999999998}]},{"osd":6,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.82599999999999996,"15min":0.84199999999999997},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.495},"max":{"1min":0.98299999999999998,"5min":1.397,"15min":4.5129999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.88500000000000001,"15min":0.89900000000000002},"min":{"1min":0.626,"5min":0.626,"15min":0.56699999999999995},"max":{"1min":1.1699999999999999,"5min":1.3120000000000001,"15min":4.5750000000000002},"last":0.80000000000000004}]},{"osd":7,"last update":"Mon Oct 30 13:25:29 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.875,"15min":0.86799999999999999},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.55700000000000005},"max":{"1min":1.0669999999999999,"5min":1.206,"15min":4.3979999999999997},"last":0.623},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.88900000000000001,"15min":0.91400000000000003},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.53700000000000003},"max":{"1min":1.0760000000000001,"5min":1.2210000000000001,"15min":5.375},"last":0.82599999999999996}]}]},{"osd":6,"up_from":15,"seq":64424510137,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552812,"kb_used_data":409252,"kb_used_omap":1,"kb_used_meta":143550,"kb_avail":93819028,"statfs":{"total":96636764160,"available":96070684672,"internally_reserved":0,"allocated":419074048,"data_stored":408471323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":146995655},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.85199999999999998,"15min":0.85599999999999998},"min":{"1min":0.62,"5min":0.49399999999999999,"15min":0.34200000000000003},"max":{"1min":1.103,"5min":1.2470000000000001,"15min":1.4730000000000001},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.85099999999999998,"15min":0.95099999999999996},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2130000000000001,"5min":1.3149999999999999,"15min":1.415},"last":0.70699999999999996}]},{"osd":1,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.79300000000000004,"15min":0.79700000000000004},"min":{"1min":0.55700000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.262,"5min":1.262,"15min":1.399},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.86699999999999999,"15min":0.86299999999999999},"min":{"1min":0.62,"5min":0.52100000000000002,"15min":0.47699999999999998},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.996}]},{"osd":2,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.995,"15min":0.96699999999999997},"min":{"1min":0.80300000000000005,"5min":0.68300000000000005,"15min":0.47199999999999998},"max":{"1min":1.375,"5min":1.496,"15min":1.496},"last":1.165},{"interface":"front","average":{"1min":0.877,"5min":0.90100000000000002,"15min":0.94199999999999995},"min":{"1min":0.60299999999999998,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1839999999999999,"5min":1.3839999999999999,"15min":1.431},"last":0.91800000000000004}]},{"osd":3,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.88600000000000001,"15min":0.88900000000000001},"min":{"1min":0.65500000000000003,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.2589999999999999,"5min":1.262,"15min":1.4530000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":0.88900000000000001},"min":{"1min":0.77600000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.347,"5min":1.347,"15min":1.478},"last":1.044}]},{"osd":4,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.97299999999999998,"15min":0.93799999999999994},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.2849999999999999,"5min":1.3220000000000001,"15min":1.4019999999999999},"last":1.089},{"interface":"front","average":{"1min":1.163,"5min":1.093,"15min":1.046},"min":{"1min":0.80000000000000004,"5min":0.73899999999999999,"15min":0.54700000000000004},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.282}]},{"osd":5,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.163,"15min":1.125},"min":{"1min":0.90100000000000002,"5min":0.82999999999999996,"15min":0.73399999999999999},"max":{"1min":1.417,"5min":1.4730000000000001,"15min":1.583},"last":1.4910000000000001},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.069,"15min":1.095},"min":{"1min":0.625,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2829999999999999,"5min":1.6699999999999999,"15min":1.6699999999999999},"last":1.3260000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:25:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.081,"15min":1.0800000000000001},"min":{"1min":0.86099999999999999,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.4970000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.2589999999999999,"5min":1.22,"15min":1.2170000000000001},"min":{"1min":1.034,"5min":0.76500000000000001,"15min":0.64200000000000002},"max":{"1min":1.804,"5min":1.9059999999999999,"15min":1.9059999999999999},"last":1.429}]}]},{"osd":4,"up_from":15,"seq":64424510140,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":669408,"kb_used_data":493912,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":93702432,"statfs":{"total":96636764160,"available":95951290368,"internally_reserved":0,"allocated":505765888,"data_stored":496228056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":179698119},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.86299999999999999,"15min":0.83899999999999997},"min":{"1min":0.54800000000000004,"5min":0.433,"15min":0.433},"max":{"1min":1.1930000000000001,"5min":1.218,"15min":1.292},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.86899999999999999,"15min":0.84699999999999998},"min":{"1min":0.57399999999999995,"5min":0.52700000000000002,"15min":0.52600000000000002},"max":{"1min":1.3480000000000001,"5min":1.3480000000000001,"15min":1.369},"last":0.497}]},{"osd":1,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.79400000000000004,"15min":0.78600000000000003},"min":{"1min":0.60799999999999998,"5min":0.47899999999999998,"15min":0.47499999999999998},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1839999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.879,"5min":0.83699999999999997,"15min":0.83199999999999996},"min":{"1min":0.60899999999999999,"5min":0.53000000000000003,"15min":0.47299999999999998},"max":{"1min":1.2909999999999999,"5min":1.2909999999999999,"15min":1.3100000000000001},"last":0.56100000000000005}]},{"osd":2,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.89500000000000002},"min":{"1min":0.56599999999999995,"5min":0.55500000000000005,"15min":0.55300000000000005},"max":{"1min":1.1899999999999999,"5min":1.2270000000000001,"15min":1.292},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.998,"5min":0.94599999999999995,"15min":0.94699999999999995},"min":{"1min":0.74099999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.70999999999999996}]},{"osd":3,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":0.97399999999999998,"15min":0.95299999999999996},"min":{"1min":0.68500000000000005,"5min":0.65100000000000002,"15min":0.52200000000000002},"max":{"1min":1.286,"5min":1.286,"15min":1.514},"last":1.1160000000000001},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.94699999999999995,"15min":0.96299999999999997},"min":{"1min":0.76600000000000001,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":1.0129999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0409999999999999,"5min":0.98299999999999998,"15min":1.0129999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60299999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":2.4430000000000001},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.171,"5min":1.1140000000000001,"15min":1.0660000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.8460000000000001},"last":1.0609999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.145,"15min":1.1559999999999999},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.419,"5min":1.425,"15min":1.677},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.306,"5min":1.28,"15min":1.1830000000000001},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.64400000000000002},"max":{"1min":1.528,"5min":1.5409999999999999,"15min":1.9610000000000001},"last":1.234}]},{"osd":7,"last update":"Mon Oct 30 13:25:21 2023","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.0529999999999999,"15min":1.1359999999999999},"min":{"1min":0.89500000000000002,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.415,"5min":1.415,"15min":1.8700000000000001},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.202,"15min":1.266},"min":{"1min":1.016,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.538,"5min":1.591,"15min":1.9450000000000001},"last":1.2050000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542841,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564588,"kb_used_data":416932,"kb_used_omap":1,"kb_used_meta":147646,"kb_avail":93807252,"statfs":{"total":96636764160,"available":96058626048,"internally_reserved":0,"allocated":426938368,"data_stored":418861204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":151189963},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.82199999999999995,"5min":0.84399999999999997,"15min":0.82999999999999996},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.41499999999999998},"max":{"1min":1.2130000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.88,"15min":0.872},"min":{"1min":0.56999999999999995,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.333,"5min":1.3380000000000001,"15min":1.3919999999999999},"last":0.83399999999999996}]},{"osd":1,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.76900000000000002,"15min":0.86799999999999999},"min":{"1min":0.504,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.095,"5min":1.244,"15min":1.4790000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.91700000000000004,"15min":0.82299999999999995},"min":{"1min":0.51500000000000001,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.5,"5min":1.524,"15min":1.524},"last":1.002}]},{"osd":2,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.0589999999999999,"15min":1.024},"min":{"1min":0.66500000000000004,"5min":0.54700000000000004,"15min":0.52700000000000002},"max":{"1min":1.3109999999999999,"5min":1.833,"15min":1.833},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1830000000000001,"15min":1.153},"min":{"1min":0.91100000000000003,"5min":0.71499999999999997,"15min":0.69399999999999995},"max":{"1min":1.4510000000000001,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.323}]},{"osd":3,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0760000000000001,"15min":1.083},"min":{"1min":0.66400000000000003,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.2929999999999999,"5min":1.673,"15min":1.673},"last":1.446},{"interface":"front","average":{"1min":1.204,"5min":1.244,"15min":1.2230000000000001},"min":{"1min":0.84299999999999997,"5min":0.83499999999999996,"15min":0.81399999999999995},"max":{"1min":1.4490000000000001,"5min":1.8109999999999999,"15min":1.8480000000000001},"last":1.526}]},{"osd":4,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.86199999999999999,"15min":0.84799999999999998},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.171,"5min":1.226,"15min":1.4970000000000001},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.95799999999999996,"15min":0.95599999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.242,"5min":1.4970000000000001,"15min":1.77},"last":1.073}]},{"osd":5,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":0.94299999999999995,"15min":0.97499999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.2490000000000001,"5min":1.3799999999999999,"15min":2},"last":1.141},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.105,"15min":1.0349999999999999},"min":{"1min":0.69599999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.508,"5min":1.7150000000000001,"15min":2.056},"last":1.286}]},{"osd":6,"last update":"Mon Oct 30 13:25:20 2023","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.042,"15min":1.008},"min":{"1min":0.68100000000000005,"5min":0.60799999999999998,"15min":0.501},"max":{"1min":1.377,"5min":1.536,"15min":1.571},"last":1.411},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1299999999999999,"15min":1.105},"min":{"1min":0.751,"5min":0.72599999999999998,"15min":0.71199999999999997},"max":{"1min":1.5549999999999999,"5min":1.665,"15min":1.665},"last":1.2250000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542840,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":816096,"kb_used_data":640600,"kb_used_omap":1,"kb_used_meta":175486,"kb_avail":93555744,"statfs":{"total":96636764160,"available":95801081856,"internally_reserved":0,"allocated":655974400,"data_stored":642425281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":179698118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.81699999999999995,"15min":0.81499999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.378},"max":{"1min":1.173,"5min":1.276,"15min":1.579},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.85299999999999998,"15min":0.86099999999999999},"min":{"1min":0.499,"5min":0.46200000000000002,"15min":0.34599999999999997},"max":{"1min":1.262,"5min":1.627,"15min":1.627},"last":0.81499999999999995}]},{"osd":1,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.81200000000000006,"15min":0.83099999999999996},"min":{"1min":0.55300000000000005,"5min":0.47499999999999998,"15min":0.39500000000000002},"max":{"1min":1.22,"5min":1.226,"15min":1.3220000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.88600000000000001,"15min":0.91900000000000004},"min":{"1min":0.64800000000000002,"5min":0.52600000000000002,"15min":0.376},"max":{"1min":1.1539999999999999,"5min":1.3100000000000001,"15min":1.6619999999999999},"last":0.86699999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.85799999999999998,"15min":0.84799999999999998},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.2330000000000001,"5min":1.3220000000000001,"15min":1.486},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.89300000000000002,"15min":0.88},"min":{"1min":0.60799999999999998,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.2749999999999999,"5min":1.3149999999999999,"15min":1.879},"last":0.90500000000000003}]},{"osd":3,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.96899999999999997,"15min":0.88900000000000001},"min":{"1min":0.65500000000000003,"5min":0.55900000000000005,"15min":0.42799999999999999},"max":{"1min":1.361,"5min":1.361,"15min":2.7330000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.097,"5min":1.0509999999999999,"15min":0.999},"min":{"1min":0.68899999999999995,"5min":0.51300000000000001,"15min":0.47099999999999997},"max":{"1min":1.359,"5min":1.5169999999999999,"15min":2.79},"last":0.77100000000000002}]},{"osd":4,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.97299999999999998,"15min":0.98099999999999998},"min":{"1min":0.68500000000000005,"5min":0.57599999999999996,"15min":0.45300000000000001},"max":{"1min":1.119,"5min":1.4710000000000001,"15min":1.6719999999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.034,"15min":1.0940000000000001},"min":{"1min":0.87,"5min":0.626,"15min":0.58799999999999997},"max":{"1min":1.3120000000000001,"5min":1.3540000000000001,"15min":1.7629999999999999},"last":1.0920000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.1000000000000001,"15min":1.0920000000000001},"min":{"1min":0.69999999999999996,"5min":0.66500000000000004,"15min":0.54600000000000004},"max":{"1min":1.3959999999999999,"5min":1.675,"15min":2.8450000000000002},"last":1.016},{"interface":"front","average":{"1min":1.149,"5min":1.123,"15min":1.111},"min":{"1min":0.73699999999999999,"5min":0.59999999999999998,"15min":0.57999999999999996},"max":{"1min":1.577,"5min":1.625,"15min":1.821},"last":1.149}]},{"osd":7,"last update":"Mon Oct 30 13:25:23 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.153,"15min":1.127},"min":{"1min":0.70099999999999996,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.7490000000000001,"5min":1.7749999999999999,"15min":3.3069999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.115,"5min":1.1579999999999999,"15min":1.113},"min":{"1min":0.88500000000000001,"5min":0.59999999999999998,"15min":0.53200000000000003},"max":{"1min":1.3129999999999999,"5min":1.754,"15min":2.036},"last":1.232}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309846016,"data_stored":309823069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142618624,"data_stored":142610667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117452800,"data_stored":117441250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294043648,"data_stored":294028670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":275668992,"data_stored":275628631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":300916736,"data_stored":300897937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":154218496,"data_stored":154207037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":211644416,"data_stored":211616353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401661952,"data_stored":386457412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297664512,"data_stored":287498447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304177152,"data_stored":291323631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359911424,"data_stored":346240784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":229097472,"data_stored":220069239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":353464320,"data_stored":340406790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":263856128,"data_stored":253734100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":214294528,"data_stored":206714665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:26:23.836 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:23.837 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:23.837 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:23.838 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:23.838 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:23.839 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:23.839 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:23.839 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:23.840 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:23.841 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:23.841 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:23.841 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:23.842 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T13:26:43.843 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:26:43.990 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.985+0000 7fa835017640 1 Processor -- start 2023-10-30T13:26:43.991 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.985+0000 7fa835017640 1 -- start start 2023-10-30T13:26:43.992 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.985+0000 7fa835017640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8300c6b80 0x7fa8300bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:43.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.985+0000 7fa835017640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa8300c6f60 0x7fa8300c0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:43.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.985+0000 7fa835017640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8300c0a50 con 0x7fa8300c8430 2023-10-30T13:26:43.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.985+0000 7fa835017640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa83006f6e0 con 0x7fa8300c6f60 2023-10-30T13:26:43.993 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.985+0000 7fa835017640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa83006f9c0 con 0x7fa8300c6b80 2023-10-30T13:26:43.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.989+0000 7fa82e59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa8300c6f60 0x7fa8300c0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:43.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.989+0000 7fa82e59c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa8300c6f60 0x7fa8300c0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56260/0 (socket says 172.21.15.125:56260) 2023-10-30T13:26:43.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.989+0000 7fa82f59e640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fa8300c8430 0x7fa83013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:59372/0 (socket says 172.21.15.125:59372) 2023-10-30T13:26:43.994 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.989+0000 7fa82e59c640 1 -- 172.21.15.125:0/1902628333 learned_addr learned my addr 172.21.15.125:0/1902628333 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:26:43.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.989+0000 7fa82ed9d640 1 --2- 172.21.15.125:0/1902628333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8300c6b80 0x7fa8300bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:43.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.989+0000 7fa82e59c640 1 -- 172.21.15.125:0/1902628333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8300c6b80 msgr2=0x7fa8300bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:26:43.995 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.989+0000 7fa82e59c640 1 --2- 172.21.15.125:0/1902628333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8300c6b80 0x7fa8300bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:43.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.989+0000 7fa82e59c640 1 -- 172.21.15.125:0/1902628333 >> v1:172.21.15.125:6789/0 conn(0x7fa8300c8430 legacy=0x7fa83013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:43.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.989+0000 7fa82e59c640 1 -- 172.21.15.125:0/1902628333 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83006fca0 con 0x7fa8300c6f60 2023-10-30T13:26:43.996 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.989+0000 7fa82dd9b640 1 -- 172.21.15.125:0/1902628333 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170716032 0 0) 0x7fa8300c0a50 con 0x7fa8300c8430 2023-10-30T13:26:43.997 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.989+0000 7fa82e59c640 1 --2- 172.21.15.125:0/1902628333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8300c6f60 0x7fa8300c0400 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fa824009870 tx=0x7fa82401c540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d7c0c66e06b7b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:43.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.989+0000 7fa82dd9b640 1 -- 172.21.15.125:0/1902628333 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8240ab650 con 0x7fa8300c6f60 2023-10-30T13:26:43.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.989+0000 7fa82dd9b640 1 -- 172.21.15.125:0/1902628333 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8240ab7b0 con 0x7fa8300c6f60 2023-10-30T13:26:43.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.989+0000 7fa82dd9b640 1 -- 172.21.15.125:0/1902628333 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8240abb40 con 0x7fa8300c6f60 2023-10-30T13:26:43.998 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 -- 172.21.15.125:0/1902628333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8300c6f60 msgr2=0x7fa8300c0400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:43.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 --2- 172.21.15.125:0/1902628333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8300c6f60 0x7fa8300c0400 secure :-1 s=READY pgs=2211 cs=0 l=1 rev1=1 crypto rx=0x7fa824009870 tx=0x7fa82401c540 comp rx=0 tx=0).stop 2023-10-30T13:26:43.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 -- 172.21.15.125:0/1902628333 shutdown_connections 2023-10-30T13:26:43.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 --2- 172.21.15.125:0/1902628333 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8300c6f60 0x7fa8300c0400 unknown :-1 s=CLOSED pgs=2211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:43.999 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 --2- 172.21.15.125:0/1902628333 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8300c6b80 0x7fa8300bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 -- 172.21.15.125:0/1902628333 >> 172.21.15.125:0/1902628333 conn(0x7fa8300c7910 msgr2=0x7fa8300c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:44.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 -- 172.21.15.125:0/1902628333 shutdown_connections 2023-10-30T13:26:44.000 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 -- 172.21.15.125:0/1902628333 wait complete. 2023-10-30T13:26:44.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 Processor -- start 2023-10-30T13:26:44.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 -- start start 2023-10-30T13:26:44.003 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8300c6b80 0x7fa830157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:44.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8300c6f60 0x7fa830155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:44.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8300c8430 0x7fa830158030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:44.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8301f23f0 con 0x7fa8300c8430 2023-10-30T13:26:44.004 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8301f2c30 con 0x7fa8300c6b80 2023-10-30T13:26:44.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa835017640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa830157aa0 con 0x7fa8300c6f60 2023-10-30T13:26:44.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa82f59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8300c8430 0x7fa830158030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:44.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa82f59e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8300c8430 0x7fa830158030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:56262/0 (socket says 172.21.15.125:56262) 2023-10-30T13:26:44.005 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa82f59e640 1 -- 172.21.15.125:0/2710565521 learned_addr learned my addr 172.21.15.125:0/2710565521 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:26:44.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa82e59c640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8300c6f60 0x7fa830155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:44.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.993+0000 7fa82ed9d640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8300c6b80 0x7fa830157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:44.006 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa82f59e640 1 -- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8300c6f60 msgr2=0x7fa830155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:44.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa82f59e640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8300c6f60 0x7fa830155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa82f59e640 1 -- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8300c6b80 msgr2=0x7fa830157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:44.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa82f59e640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8300c6b80 0x7fa830157560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.007 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa82f59e640 1 -- 172.21.15.125:0/2710565521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8240094d0 con 0x7fa8300c8430 2023-10-30T13:26:44.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa82e59c640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8300c6f60 0x7fa830155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:26:44.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa82f59e640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8300c8430 0x7fa830158030 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fa82800c700 tx=0x7fa82800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:44.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa8177fe640 1 -- 172.21.15.125:0/2710565521 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82800ce10 con 0x7fa8300c8430 2023-10-30T13:26:44.008 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa8177fe640 1 -- 172.21.15.125:0/2710565521 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82800a640 con 0x7fa8300c8430 2023-10-30T13:26:44.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa8177fe640 1 -- 172.21.15.125:0/2710565521 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82800aa70 con 0x7fa8300c8430 2023-10-30T13:26:44.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa835017640 1 -- 172.21.15.125:0/2710565521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa830158fe0 con 0x7fa8300c8430 2023-10-30T13:26:44.009 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa835017640 1 -- 172.21.15.125:0/2710565521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8301f25e0 con 0x7fa8300c8430 2023-10-30T13:26:44.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa835017640 1 -- 172.21.15.125:0/2710565521 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8301f2da0 con 0x7fa8300c8430 2023-10-30T13:26:44.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa8177fe640 1 -- 172.21.15.125:0/2710565521 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa82800abd0 con 0x7fa8300c8430 2023-10-30T13:26:44.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa8177fe640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa81c0664a0 0x7fa81c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:26:44.010 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa8177fe640 1 -- 172.21.15.125:0/2710565521 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fa828083f40 con 0x7fa8300c8430 2023-10-30T13:26:44.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:43.997+0000 7fa82ed9d640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa81c0664a0 0x7fa81c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:26:44.011 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.001+0000 7fa82ed9d640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa81c0664a0 0x7fa81c068960 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa82000b9d0 tx=0x7fa820008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:26:44.012 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.001+0000 7fa8177fe640 1 -- 172.21.15.125:0/2710565521 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa8280507f0 con 0x7fa8300c8430 2023-10-30T13:26:44.154 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.149+0000 7fa835017640 1 -- 172.21.15.125:0/2710565521 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa83004e170 con 0x7fa81c0664a0 2023-10-30T13:26:44.177 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.173+0000 7fa8177fe640 1 -- 172.21.15.125:0/2710565521 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+296230 (secure 0 0 0) 0x7fa83004e170 con 0x7fa81c0664a0 2023-10-30T13:26:44.180 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:26:44.182 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T13:26:44.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.177+0000 7fa835017640 1 -- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa81c0664a0 msgr2=0x7fa81c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:44.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.177+0000 7fa835017640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa81c0664a0 0x7fa81c068960 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa82000b9d0 tx=0x7fa820008040 comp rx=0 tx=0).stop 2023-10-30T13:26:44.183 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.177+0000 7fa835017640 1 -- 172.21.15.125:0/2710565521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8300c8430 msgr2=0x7fa830158030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:26:44.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.177+0000 7fa835017640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8300c8430 0x7fa830158030 secure :-1 s=READY pgs=2212 cs=0 l=1 rev1=1 crypto rx=0x7fa82800c700 tx=0x7fa82800cbc0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.177+0000 7fa835017640 1 -- 172.21.15.125:0/2710565521 shutdown_connections 2023-10-30T13:26:44.184 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.177+0000 7fa835017640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa81c0664a0 0x7fa81c068960 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.177+0000 7fa835017640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa8300c8430 0x7fa830158030 unknown :-1 s=CLOSED pgs=2212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.177+0000 7fa835017640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa8300c6f60 0x7fa830155cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.177+0000 7fa835017640 1 --2- 172.21.15.125:0/2710565521 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa8300c6b80 0x7fa830157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:26:44.185 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.177+0000 7fa835017640 1 -- 172.21.15.125:0/2710565521 >> 172.21.15.125:0/2710565521 conn(0x7fa8300c7910 msgr2=0x7fa8300b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:26:44.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.177+0000 7fa835017640 1 -- 172.21.15.125:0/2710565521 shutdown_connections 2023-10-30T13:26:44.186 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:26:44.177+0000 7fa835017640 1 -- 172.21.15.125:0/2710565521 wait complete. 2023-10-30T13:26:44.201 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":1823,"stamp":"2023-10-30T13:26:42.292619+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2068932093,"num_objects":40960,"num_object_clones":18317,"num_object_copies":81920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40960,"num_whiteouts":14249,"num_read":30933,"num_read_kb":10838515,"num_write":128197,"num_write_kb":3672596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":19724250,"num_omap_keys":33398,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175185,"ondisk_log_size":175185,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5580288,"kb_used_data":4140352,"kb_used_omap":12,"kb_used_meta":1439859,"kb_avail":749394432,"statfs":{"total":773094113280,"available":767379898368,"internally_reserved":0,"allocated":4239720448,"data_stored":4144120906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1474416185},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,5,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004334"},"pg_stats":[{"pgid":"3.3f","version":"85'2027","reported_seq":4106,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:18.022996+0000","last_change":"2023-10-30T13:26:18.023085+0000","last_active":"2023-10-30T13:26:18.022996+0000","last_peered":"2023-10-30T13:26:18.022996+0000","last_clean":"2023-10-30T13:26:18.022996+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T13:26:18.022996+0000","last_undegraded":"2023-10-30T13:26:18.022996+0000","last_fullsized":"2023-10-30T13:26:18.022996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2027","last_scrub_stamp":"2023-10-30T13:26:18.022913+0000","last_deep_scrub":"85'2027","last_deep_scrub_stamp":"2023-10-30T13:26:18.022913+0000","last_clean_scrub_stamp":"2023-10-30T13:26:18.022913+0000","objects_scrubbed":552,"log_size":2027,"log_dups_size":0,"ondisk_log_size":2027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:29.468884+0000","scrub_duration":0.915286406,"objects_trimmed":0,"snaptrim_duration":0.0086120880000000004,"stat_sum":{"num_bytes":27261067,"num_objects":552,"num_object_clones":249,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":197,"num_read":145,"num_read_kb":29501,"num_write":1298,"num_write_kb":29008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"85'2050","reported_seq":4094,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:22.709036+0000","last_change":"2023-10-30T13:26:22.709116+0000","last_active":"2023-10-30T13:26:22.709036+0000","last_peered":"2023-10-30T13:26:22.709036+0000","last_clean":"2023-10-30T13:26:22.709036+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T13:26:22.709036+0000","last_undegraded":"2023-10-30T13:26:22.709036+0000","last_fullsized":"2023-10-30T13:26:22.709036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2050","last_scrub_stamp":"2023-10-30T13:26:22.708912+0000","last_deep_scrub":"85'2050","last_deep_scrub_stamp":"2023-10-30T13:26:22.708912+0000","last_clean_scrub_stamp":"2023-10-30T13:26:22.708912+0000","objects_scrubbed":574,"log_size":2050,"log_dups_size":0,"ondisk_log_size":2050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:34:46.206144+0000","scrub_duration":0.99652591300000004,"objects_trimmed":0,"snaptrim_duration":0.015306986,"stat_sum":{"num_bytes":13736175,"num_objects":574,"num_object_clones":269,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":223,"num_read":131,"num_read_kb":21315,"num_write":1260,"num_write_kb":15409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"85'2191","reported_seq":4236,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:29.063137+0000","last_change":"2023-10-30T13:26:29.063266+0000","last_active":"2023-10-30T13:26:29.063137+0000","last_peered":"2023-10-30T13:26:29.063137+0000","last_clean":"2023-10-30T13:26:29.063137+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T13:26:29.063137+0000","last_undegraded":"2023-10-30T13:26:29.063137+0000","last_fullsized":"2023-10-30T13:26:29.063137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2191","last_scrub_stamp":"2023-10-30T13:26:29.062929+0000","last_deep_scrub":"85'2191","last_deep_scrub_stamp":"2023-10-30T13:26:29.062929+0000","last_clean_scrub_stamp":"2023-10-30T13:26:29.062929+0000","objects_scrubbed":628,"log_size":2191,"log_dups_size":0,"ondisk_log_size":2191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:12:09.798552+0000","scrub_duration":1.415347921,"objects_trimmed":0,"snaptrim_duration":0.013634598,"stat_sum":{"num_bytes":28390436,"num_objects":628,"num_object_clones":291,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":232,"num_read":152,"num_read_kb":47076,"num_write":1356,"num_write_kb":37143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"85'2206","reported_seq":4249,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.404657+0000","last_change":"2023-10-30T13:25:48.404708+0000","last_active":"2023-10-30T13:25:48.404657+0000","last_peered":"2023-10-30T13:25:48.404657+0000","last_clean":"2023-10-30T13:25:48.404657+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T13:25:48.404657+0000","last_undegraded":"2023-10-30T13:25:48.404657+0000","last_fullsized":"2023-10-30T13:25:48.404657+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2206","last_scrub_stamp":"2023-10-30T13:25:48.404576+0000","last_deep_scrub":"85'2206","last_deep_scrub_stamp":"2023-10-30T13:25:48.404576+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.404576+0000","objects_scrubbed":629,"log_size":2206,"log_dups_size":0,"ondisk_log_size":2206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:01.872332+0000","scrub_duration":1.22769368,"objects_trimmed":0,"snaptrim_duration":0.0056762749999999997,"stat_sum":{"num_bytes":22921274,"num_objects":629,"num_object_clones":290,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":224,"num_read":145,"num_read_kb":39229,"num_write":1388,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"85'2214","reported_seq":4254,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:00.132866+0000","last_change":"2023-10-30T13:26:00.132866+0000","last_active":"2023-10-30T13:26:00.132866+0000","last_peered":"2023-10-30T13:26:00.132866+0000","last_clean":"2023-10-30T13:26:00.132866+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T13:26:00.132866+0000","last_undegraded":"2023-10-30T13:26:00.132866+0000","last_fullsized":"2023-10-30T13:26:00.132866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2214","last_scrub_stamp":"2023-10-30T13:26:00.132634+0000","last_deep_scrub":"85'2214","last_deep_scrub_stamp":"2023-10-30T13:26:00.132634+0000","last_clean_scrub_stamp":"2023-10-30T13:26:00.132634+0000","objects_scrubbed":623,"log_size":2214,"log_dups_size":0,"ondisk_log_size":2214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:56:19.866906+0000","scrub_duration":1.0674117089999999,"objects_trimmed":0,"snaptrim_duration":0.0081298299999999993,"stat_sum":{"num_bytes":26798060,"num_objects":623,"num_object_clones":286,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":214,"num_read":144,"num_read_kb":26713,"num_write":1434,"num_write_kb":32534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"85'2029","reported_seq":4091,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.107819+0000","last_change":"2023-10-30T13:25:44.107819+0000","last_active":"2023-10-30T13:25:44.107819+0000","last_peered":"2023-10-30T13:25:44.107819+0000","last_clean":"2023-10-30T13:25:44.107819+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T13:25:44.107819+0000","last_undegraded":"2023-10-30T13:25:44.107819+0000","last_fullsized":"2023-10-30T13:25:44.107819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2029","last_scrub_stamp":"2023-10-30T13:25:44.107605+0000","last_deep_scrub":"85'2029","last_deep_scrub_stamp":"2023-10-30T13:25:44.107605+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.107605+0000","objects_scrubbed":563,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:35.807432+0000","scrub_duration":0.97669900899999995,"objects_trimmed":0,"snaptrim_duration":0.0048510920000000004,"stat_sum":{"num_bytes":13539463,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":205,"num_read":144,"num_read_kb":15818,"num_write":1313,"num_write_kb":14957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"85'2168","reported_seq":4191,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:58.341167+0000","last_change":"2023-10-30T13:25:58.341234+0000","last_active":"2023-10-30T13:25:58.341167+0000","last_peered":"2023-10-30T13:25:58.341167+0000","last_clean":"2023-10-30T13:25:58.341167+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T13:25:58.341167+0000","last_undegraded":"2023-10-30T13:25:58.341167+0000","last_fullsized":"2023-10-30T13:25:58.341167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2168","last_scrub_stamp":"2023-10-30T13:25:58.341077+0000","last_deep_scrub":"85'2168","last_deep_scrub_stamp":"2023-10-30T13:25:58.341077+0000","last_clean_scrub_stamp":"2023-10-30T13:25:58.341077+0000","objects_scrubbed":606,"log_size":2168,"log_dups_size":0,"ondisk_log_size":2168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:06.931072+0000","scrub_duration":1.219517739,"objects_trimmed":0,"snaptrim_duration":0.00082671399999999999,"stat_sum":{"num_bytes":7202428,"num_objects":606,"num_object_clones":284,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":219,"num_read":126,"num_read_kb":1677,"num_write":1361,"num_write_kb":10203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"85'2105","reported_seq":4163,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:14.120371+0000","last_change":"2023-10-30T13:26:14.120371+0000","last_active":"2023-10-30T13:26:14.120371+0000","last_peered":"2023-10-30T13:26:14.120371+0000","last_clean":"2023-10-30T13:26:14.120371+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T13:26:14.120371+0000","last_undegraded":"2023-10-30T13:26:14.120371+0000","last_fullsized":"2023-10-30T13:26:14.120371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2105","last_scrub_stamp":"2023-10-30T13:26:14.120044+0000","last_deep_scrub":"85'2105","last_deep_scrub_stamp":"2023-10-30T13:26:14.120044+0000","last_clean_scrub_stamp":"2023-10-30T13:26:14.120044+0000","objects_scrubbed":587,"log_size":2105,"log_dups_size":0,"ondisk_log_size":2105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:11:07.710223+0000","scrub_duration":1.494910695,"objects_trimmed":0,"snaptrim_duration":0.014437559000000001,"stat_sum":{"num_bytes":23150826,"num_objects":587,"num_object_clones":269,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":199,"num_read":149,"num_read_kb":38846,"num_write":1362,"num_write_kb":24887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"85'2167","reported_seq":4231,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:00.039174+0000","last_change":"2023-10-30T13:26:00.039258+0000","last_active":"2023-10-30T13:26:00.039174+0000","last_peered":"2023-10-30T13:26:00.039174+0000","last_clean":"2023-10-30T13:26:00.039174+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T13:26:00.039174+0000","last_undegraded":"2023-10-30T13:26:00.039174+0000","last_fullsized":"2023-10-30T13:26:00.039174+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2167","last_scrub_stamp":"2023-10-30T13:26:00.039070+0000","last_deep_scrub":"85'2167","last_deep_scrub_stamp":"2023-10-30T13:26:00.039070+0000","last_clean_scrub_stamp":"2023-10-30T13:26:00.039070+0000","objects_scrubbed":618,"log_size":2167,"log_dups_size":0,"ondisk_log_size":2167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:06:11.823857+0000","scrub_duration":1.5619247110000001,"objects_trimmed":0,"snaptrim_duration":0.0014185980000000001,"stat_sum":{"num_bytes":36575095,"num_objects":618,"num_object_clones":282,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":216,"num_read":165,"num_read_kb":36418,"num_write":1354,"num_write_kb":45869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"85'2045","reported_seq":4109,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:15.050332+0000","last_change":"2023-10-30T13:26:15.050397+0000","last_active":"2023-10-30T13:26:15.050332+0000","last_peered":"2023-10-30T13:26:15.050332+0000","last_clean":"2023-10-30T13:26:15.050332+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T13:26:15.050332+0000","last_undegraded":"2023-10-30T13:26:15.050332+0000","last_fullsized":"2023-10-30T13:26:15.050332+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2045","last_scrub_stamp":"2023-10-30T13:26:15.050260+0000","last_deep_scrub":"85'2045","last_deep_scrub_stamp":"2023-10-30T13:26:15.050260+0000","last_clean_scrub_stamp":"2023-10-30T13:26:15.050260+0000","objects_scrubbed":586,"log_size":2045,"log_dups_size":0,"ondisk_log_size":2045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:47.785761+0000","scrub_duration":0.93888254999999998,"objects_trimmed":0,"snaptrim_duration":0.0091404980000000004,"stat_sum":{"num_bytes":11176343,"num_objects":586,"num_object_clones":270,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":213,"num_read":154,"num_read_kb":10770,"num_write":1260,"num_write_kb":13564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"85'2136","reported_seq":4176,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:06.941519+0000","last_change":"2023-10-30T13:26:06.941585+0000","last_active":"2023-10-30T13:26:06.941519+0000","last_peered":"2023-10-30T13:26:06.941519+0000","last_clean":"2023-10-30T13:26:06.941519+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T13:26:06.941519+0000","last_undegraded":"2023-10-30T13:26:06.941519+0000","last_fullsized":"2023-10-30T13:26:06.941519+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2136","last_scrub_stamp":"2023-10-30T13:26:06.941431+0000","last_deep_scrub":"85'2136","last_deep_scrub_stamp":"2023-10-30T13:26:06.941431+0000","last_clean_scrub_stamp":"2023-10-30T13:26:06.941431+0000","objects_scrubbed":625,"log_size":2136,"log_dups_size":0,"ondisk_log_size":2136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:47.801796+0000","scrub_duration":0.82203926400000005,"objects_trimmed":0,"snaptrim_duration":0.0075712790000000002,"stat_sum":{"num_bytes":23568118,"num_objects":625,"num_object_clones":283,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":230,"num_read":141,"num_read_kb":34199,"num_write":1299,"num_write_kb":24762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"85'2446","reported_seq":4405,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:14.612141+0000","last_change":"2023-10-30T13:26:14.612219+0000","last_active":"2023-10-30T13:26:14.612141+0000","last_peered":"2023-10-30T13:26:14.612141+0000","last_clean":"2023-10-30T13:26:14.612141+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T13:26:14.612141+0000","last_undegraded":"2023-10-30T13:26:14.612141+0000","last_fullsized":"2023-10-30T13:26:14.612141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2446","last_scrub_stamp":"2023-10-30T13:26:14.612071+0000","last_deep_scrub":"85'2446","last_deep_scrub_stamp":"2023-10-30T13:26:14.612071+0000","last_clean_scrub_stamp":"2023-10-30T13:26:14.612071+0000","objects_scrubbed":721,"log_size":2446,"log_dups_size":0,"ondisk_log_size":2446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:13:35.491978+0000","scrub_duration":1.5476285650000001,"objects_trimmed":0,"snaptrim_duration":0.007366493,"stat_sum":{"num_bytes":24142540,"num_objects":721,"num_object_clones":385,"num_object_copies":1442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":231,"num_read":157,"num_read_kb":24833,"num_write":1452,"num_write_kb":33857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"85'2099","reported_seq":4169,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.041415+0000","last_change":"2023-10-30T13:25:54.041494+0000","last_active":"2023-10-30T13:25:54.041415+0000","last_peered":"2023-10-30T13:25:54.041415+0000","last_clean":"2023-10-30T13:25:54.041415+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T13:25:54.041415+0000","last_undegraded":"2023-10-30T13:25:54.041415+0000","last_fullsized":"2023-10-30T13:25:54.041415+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2099","last_scrub_stamp":"2023-10-30T13:25:54.041318+0000","last_deep_scrub":"85'2099","last_deep_scrub_stamp":"2023-10-30T13:25:54.041318+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.041318+0000","objects_scrubbed":596,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:14.898677+0000","scrub_duration":1.085239426,"objects_trimmed":0,"snaptrim_duration":0.006759035,"stat_sum":{"num_bytes":14254037,"num_objects":596,"num_object_clones":276,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":205,"num_read":165,"num_read_kb":21722,"num_write":1326,"num_write_kb":24306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"85'2174","reported_seq":4206,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:01.905598+0000","last_change":"2023-10-30T13:26:01.905712+0000","last_active":"2023-10-30T13:26:01.905598+0000","last_peered":"2023-10-30T13:26:01.905598+0000","last_clean":"2023-10-30T13:26:01.905598+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T13:26:01.905598+0000","last_undegraded":"2023-10-30T13:26:01.905598+0000","last_fullsized":"2023-10-30T13:26:01.905598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2174","last_scrub_stamp":"2023-10-30T13:26:01.905493+0000","last_deep_scrub":"85'2174","last_deep_scrub_stamp":"2023-10-30T13:26:01.905493+0000","last_clean_scrub_stamp":"2023-10-30T13:26:01.905493+0000","objects_scrubbed":606,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:14:46.370347+0000","scrub_duration":0.97684040100000002,"objects_trimmed":0,"snaptrim_duration":0.0057386779999999997,"stat_sum":{"num_bytes":10370740,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":213,"num_read":131,"num_read_kb":5845,"num_write":1381,"num_write_kb":12595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"85'2261","reported_seq":4276,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:03.616856+0000","last_change":"2023-10-30T13:26:03.616927+0000","last_active":"2023-10-30T13:26:03.616856+0000","last_peered":"2023-10-30T13:26:03.616856+0000","last_clean":"2023-10-30T13:26:03.616856+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T13:26:03.616856+0000","last_undegraded":"2023-10-30T13:26:03.616856+0000","last_fullsized":"2023-10-30T13:26:03.616856+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2261","last_scrub_stamp":"2023-10-30T13:26:03.616747+0000","last_deep_scrub":"85'2261","last_deep_scrub_stamp":"2023-10-30T13:26:03.616747+0000","last_clean_scrub_stamp":"2023-10-30T13:26:03.616747+0000","objects_scrubbed":637,"log_size":2261,"log_dups_size":0,"ondisk_log_size":2261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:45.471660+0000","scrub_duration":1.159355685,"objects_trimmed":0,"snaptrim_duration":0.013549705,"stat_sum":{"num_bytes":19248519,"num_objects":637,"num_object_clones":295,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":221,"num_read":128,"num_read_kb":21688,"num_write":1447,"num_write_kb":21442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"85'2193","reported_seq":4229,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:46.252816+0000","last_change":"2023-10-30T13:25:46.252862+0000","last_active":"2023-10-30T13:25:46.252816+0000","last_peered":"2023-10-30T13:25:46.252816+0000","last_clean":"2023-10-30T13:25:46.252816+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T13:25:46.252816+0000","last_undegraded":"2023-10-30T13:25:46.252816+0000","last_fullsized":"2023-10-30T13:25:46.252816+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2193","last_scrub_stamp":"2023-10-30T13:25:46.252731+0000","last_deep_scrub":"85'2193","last_deep_scrub_stamp":"2023-10-30T13:25:46.252731+0000","last_clean_scrub_stamp":"2023-10-30T13:25:46.252731+0000","objects_scrubbed":619,"log_size":2193,"log_dups_size":0,"ondisk_log_size":2193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:51.513414+0000","scrub_duration":0.82934357400000003,"objects_trimmed":0,"snaptrim_duration":0.0075608309999999996,"stat_sum":{"num_bytes":14269019,"num_objects":619,"num_object_clones":290,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":233,"num_read":143,"num_read_kb":25965,"num_write":1377,"num_write_kb":20442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"85'2247","reported_seq":4267,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:45.441798+0000","last_change":"2023-10-30T13:25:45.441869+0000","last_active":"2023-10-30T13:25:45.441798+0000","last_peered":"2023-10-30T13:25:45.441798+0000","last_clean":"2023-10-30T13:25:45.441798+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T13:25:45.441798+0000","last_undegraded":"2023-10-30T13:25:45.441798+0000","last_fullsized":"2023-10-30T13:25:45.441798+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2247","last_scrub_stamp":"2023-10-30T13:25:45.441689+0000","last_deep_scrub":"85'2247","last_deep_scrub_stamp":"2023-10-30T13:25:45.441689+0000","last_clean_scrub_stamp":"2023-10-30T13:25:45.441689+0000","objects_scrubbed":642,"log_size":2247,"log_dups_size":0,"ondisk_log_size":2247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:54:27.498757+0000","scrub_duration":0.97553288699999996,"objects_trimmed":0,"snaptrim_duration":0.0099357109999999998,"stat_sum":{"num_bytes":14793776,"num_objects":642,"num_object_clones":304,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":242,"num_read":140,"num_read_kb":30098,"num_write":1386,"num_write_kb":20714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"85'2006","reported_seq":4061,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.398039+0000","last_change":"2023-10-30T13:25:57.398144+0000","last_active":"2023-10-30T13:25:57.398039+0000","last_peered":"2023-10-30T13:25:57.398039+0000","last_clean":"2023-10-30T13:25:57.398039+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T13:25:57.398039+0000","last_undegraded":"2023-10-30T13:25:57.398039+0000","last_fullsized":"2023-10-30T13:25:57.398039+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2006","last_scrub_stamp":"2023-10-30T13:25:57.397929+0000","last_deep_scrub":"85'2006","last_deep_scrub_stamp":"2023-10-30T13:25:57.397929+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.397929+0000","objects_scrubbed":562,"log_size":2006,"log_dups_size":0,"ondisk_log_size":2006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:28.919675+0000","scrub_duration":1.1960647630000001,"objects_trimmed":0,"snaptrim_duration":0.0065893760000000001,"stat_sum":{"num_bytes":15508852,"num_objects":562,"num_object_clones":263,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":216,"num_read":137,"num_read_kb":10064,"num_write":1244,"num_write_kb":17198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"85'2190","reported_seq":4325,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:24.796847+0000","last_change":"2023-10-30T13:26:24.796903+0000","last_active":"2023-10-30T13:26:24.796847+0000","last_peered":"2023-10-30T13:26:24.796847+0000","last_clean":"2023-10-30T13:26:24.796847+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T13:26:24.796847+0000","last_undegraded":"2023-10-30T13:26:24.796847+0000","last_fullsized":"2023-10-30T13:26:24.796847+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2190","last_scrub_stamp":"2023-10-30T13:26:24.796760+0000","last_deep_scrub":"85'2190","last_deep_scrub_stamp":"2023-10-30T13:26:24.796760+0000","last_clean_scrub_stamp":"2023-10-30T13:26:24.796760+0000","objects_scrubbed":614,"log_size":2190,"log_dups_size":0,"ondisk_log_size":2190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:13.965904+0000","scrub_duration":1.233998626,"objects_trimmed":0,"snaptrim_duration":0.010472766,"stat_sum":{"num_bytes":15695810,"num_objects":614,"num_object_clones":281,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":221,"num_read":233,"num_read_kb":17180,"num_write":1382,"num_write_kb":23481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"85'2183","reported_seq":4220,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:10.971723+0000","last_change":"2023-10-30T13:26:10.971791+0000","last_active":"2023-10-30T13:26:10.971723+0000","last_peered":"2023-10-30T13:26:10.971723+0000","last_clean":"2023-10-30T13:26:10.971723+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T13:26:10.971723+0000","last_undegraded":"2023-10-30T13:26:10.971723+0000","last_fullsized":"2023-10-30T13:26:10.971723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2183","last_scrub_stamp":"2023-10-30T13:26:10.971644+0000","last_deep_scrub":"85'2183","last_deep_scrub_stamp":"2023-10-30T13:26:10.971644+0000","last_clean_scrub_stamp":"2023-10-30T13:26:10.971644+0000","objects_scrubbed":614,"log_size":2183,"log_dups_size":0,"ondisk_log_size":2183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:19.149519+0000","scrub_duration":0.86719633299999999,"objects_trimmed":0,"snaptrim_duration":0.0075455599999999998,"stat_sum":{"num_bytes":5586810,"num_objects":614,"num_object_clones":286,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":230,"num_read":140,"num_read_kb":5846,"num_write":1359,"num_write_kb":15885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"85'2129","reported_seq":4280,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.179234+0000","last_change":"2023-10-30T13:25:54.179340+0000","last_active":"2023-10-30T13:25:54.179234+0000","last_peered":"2023-10-30T13:25:54.179234+0000","last_clean":"2023-10-30T13:25:54.179234+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T13:25:54.179234+0000","last_undegraded":"2023-10-30T13:25:54.179234+0000","last_fullsized":"2023-10-30T13:25:54.179234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2129","last_scrub_stamp":"2023-10-30T13:25:54.178827+0000","last_deep_scrub":"85'2129","last_deep_scrub_stamp":"2023-10-30T13:25:54.178827+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.178827+0000","objects_scrubbed":606,"log_size":2129,"log_dups_size":0,"ondisk_log_size":2129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:18.992880+0000","scrub_duration":1.0904528849999999,"objects_trimmed":0,"snaptrim_duration":0.0076434509999999999,"stat_sum":{"num_bytes":23023646,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":222,"num_read":249,"num_read_kb":26208,"num_write":1319,"num_write_kb":28338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"85'2324","reported_seq":4342,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:04.661649+0000","last_change":"2023-10-30T13:26:04.661717+0000","last_active":"2023-10-30T13:26:04.661649+0000","last_peered":"2023-10-30T13:26:04.661649+0000","last_clean":"2023-10-30T13:26:04.661649+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T13:26:04.661649+0000","last_undegraded":"2023-10-30T13:26:04.661649+0000","last_fullsized":"2023-10-30T13:26:04.661649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2324","last_scrub_stamp":"2023-10-30T13:26:04.661572+0000","last_deep_scrub":"85'2324","last_deep_scrub_stamp":"2023-10-30T13:26:04.661572+0000","last_clean_scrub_stamp":"2023-10-30T13:26:04.661572+0000","objects_scrubbed":653,"log_size":2324,"log_dups_size":0,"ondisk_log_size":2324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:26:17.690671+0000","scrub_duration":1.171930887,"objects_trimmed":0,"snaptrim_duration":0.0016105379999999999,"stat_sum":{"num_bytes":11166034,"num_objects":653,"num_object_clones":306,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":233,"num_read":139,"num_read_kb":9627,"num_write":1472,"num_write_kb":18195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"85'2102","reported_seq":4154,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:11.723496+0000","last_change":"2023-10-30T13:26:11.723576+0000","last_active":"2023-10-30T13:26:11.723496+0000","last_peered":"2023-10-30T13:26:11.723496+0000","last_clean":"2023-10-30T13:26:11.723496+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T13:26:11.723496+0000","last_undegraded":"2023-10-30T13:26:11.723496+0000","last_fullsized":"2023-10-30T13:26:11.723496+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2102","last_scrub_stamp":"2023-10-30T13:26:11.723400+0000","last_deep_scrub":"85'2102","last_deep_scrub_stamp":"2023-10-30T13:26:11.723400+0000","last_clean_scrub_stamp":"2023-10-30T13:26:11.723400+0000","objects_scrubbed":582,"log_size":2102,"log_dups_size":0,"ondisk_log_size":2102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:07.437907+0000","scrub_duration":1.1504174899999999,"objects_trimmed":0,"snaptrim_duration":0.014064241,"stat_sum":{"num_bytes":10716893,"num_objects":582,"num_object_clones":264,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":203,"num_read":135,"num_read_kb":13340,"num_write":1353,"num_write_kb":16108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"85'2148","reported_seq":4184,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.477345+0000","last_change":"2023-10-30T13:25:44.477389+0000","last_active":"2023-10-30T13:25:44.477345+0000","last_peered":"2023-10-30T13:25:44.477345+0000","last_clean":"2023-10-30T13:25:44.477345+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T13:25:44.477345+0000","last_undegraded":"2023-10-30T13:25:44.477345+0000","last_fullsized":"2023-10-30T13:25:44.477345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2148","last_scrub_stamp":"2023-10-30T13:25:44.477273+0000","last_deep_scrub":"85'2148","last_deep_scrub_stamp":"2023-10-30T13:25:44.477273+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.477273+0000","objects_scrubbed":597,"log_size":2148,"log_dups_size":0,"ondisk_log_size":2148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:41.538304+0000","scrub_duration":0.99138869699999999,"objects_trimmed":0,"snaptrim_duration":0.0088476170000000003,"stat_sum":{"num_bytes":10233195,"num_objects":597,"num_object_clones":275,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":203,"num_read":127,"num_read_kb":1819,"num_write":1383,"num_write_kb":16400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"85'2047","reported_seq":4094,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:31.570381+0000","last_change":"2023-10-30T13:26:31.570447+0000","last_active":"2023-10-30T13:26:31.570381+0000","last_peered":"2023-10-30T13:26:31.570381+0000","last_clean":"2023-10-30T13:26:31.570381+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T13:26:31.570381+0000","last_undegraded":"2023-10-30T13:26:31.570381+0000","last_fullsized":"2023-10-30T13:26:31.570381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2047","last_scrub_stamp":"2023-10-30T13:26:31.570302+0000","last_deep_scrub":"85'2047","last_deep_scrub_stamp":"2023-10-30T13:26:31.570302+0000","last_clean_scrub_stamp":"2023-10-30T13:26:31.570302+0000","objects_scrubbed":582,"log_size":2047,"log_dups_size":0,"ondisk_log_size":2047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:09:52.048922+0000","scrub_duration":0.97138433800000001,"objects_trimmed":0,"snaptrim_duration":0.014279337,"stat_sum":{"num_bytes":22085088,"num_objects":582,"num_object_clones":270,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":216,"num_read":135,"num_read_kb":28666,"num_write":1258,"num_write_kb":25992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"85'2207","reported_seq":4233,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:02.278563+0000","last_change":"2023-10-30T13:26:02.278655+0000","last_active":"2023-10-30T13:26:02.278563+0000","last_peered":"2023-10-30T13:26:02.278563+0000","last_clean":"2023-10-30T13:26:02.278563+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T13:26:02.278563+0000","last_undegraded":"2023-10-30T13:26:02.278563+0000","last_fullsized":"2023-10-30T13:26:02.278563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2207","last_scrub_stamp":"2023-10-30T13:26:02.278473+0000","last_deep_scrub":"85'2207","last_deep_scrub_stamp":"2023-10-30T13:26:02.278473+0000","last_clean_scrub_stamp":"2023-10-30T13:26:02.278473+0000","objects_scrubbed":598,"log_size":2207,"log_dups_size":0,"ondisk_log_size":2207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:21:31.276370+0000","scrub_duration":1.192745612,"objects_trimmed":0,"snaptrim_duration":0.004006122,"stat_sum":{"num_bytes":15408826,"num_objects":598,"num_object_clones":277,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":213,"num_read":119,"num_read_kb":9406,"num_write":1408,"num_write_kb":18485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"85'2174","reported_seq":4213,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.084913+0000","last_change":"2023-10-30T13:25:53.084979+0000","last_active":"2023-10-30T13:25:53.084913+0000","last_peered":"2023-10-30T13:25:53.084913+0000","last_clean":"2023-10-30T13:25:53.084913+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T13:25:53.084913+0000","last_undegraded":"2023-10-30T13:25:53.084913+0000","last_fullsized":"2023-10-30T13:25:53.084913+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2174","last_scrub_stamp":"2023-10-30T13:25:53.084846+0000","last_deep_scrub":"85'2174","last_deep_scrub_stamp":"2023-10-30T13:25:53.084846+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.084846+0000","objects_scrubbed":625,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:02.345358+0000","scrub_duration":1.1204613670000001,"objects_trimmed":0,"snaptrim_duration":0.0059072409999999997,"stat_sum":{"num_bytes":25755688,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":240,"num_read":144,"num_read_kb":24326,"num_write":1332,"num_write_kb":28963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"85'2102","reported_seq":4117,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:06.032785+0000","last_change":"2023-10-30T13:26:06.032856+0000","last_active":"2023-10-30T13:26:06.032785+0000","last_peered":"2023-10-30T13:26:06.032785+0000","last_clean":"2023-10-30T13:26:06.032785+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T13:26:06.032785+0000","last_undegraded":"2023-10-30T13:26:06.032785+0000","last_fullsized":"2023-10-30T13:26:06.032785+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2102","last_scrub_stamp":"2023-10-30T13:26:06.032657+0000","last_deep_scrub":"85'2102","last_deep_scrub_stamp":"2023-10-30T13:26:06.032657+0000","last_clean_scrub_stamp":"2023-10-30T13:26:06.032657+0000","objects_scrubbed":617,"log_size":2102,"log_dups_size":0,"ondisk_log_size":2102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:44.800470+0000","scrub_duration":1.0262600609999999,"objects_trimmed":0,"snaptrim_duration":0.0062401690000000003,"stat_sum":{"num_bytes":11021781,"num_objects":617,"num_object_clones":286,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":226,"num_read":116,"num_read_kb":10954,"num_write":1301,"num_write_kb":16114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"85'2273","reported_seq":4279,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:55.521292+0000","last_change":"2023-10-30T13:25:55.521358+0000","last_active":"2023-10-30T13:25:55.521292+0000","last_peered":"2023-10-30T13:25:55.521292+0000","last_clean":"2023-10-30T13:25:55.521292+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T13:25:55.521292+0000","last_undegraded":"2023-10-30T13:25:55.521292+0000","last_fullsized":"2023-10-30T13:25:55.521292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2273","last_scrub_stamp":"2023-10-30T13:25:55.521205+0000","last_deep_scrub":"85'2273","last_deep_scrub_stamp":"2023-10-30T13:25:55.521205+0000","last_clean_scrub_stamp":"2023-10-30T13:25:55.521205+0000","objects_scrubbed":653,"log_size":2273,"log_dups_size":0,"ondisk_log_size":2273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:44:08.018678+0000","scrub_duration":1.1272668750000001,"objects_trimmed":0,"snaptrim_duration":0.014933905000000001,"stat_sum":{"num_bytes":11161783,"num_objects":653,"num_object_clones":309,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":240,"num_read":130,"num_read_kb":12352,"num_write":1416,"num_write_kb":13402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"85'2025","reported_seq":4099,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:31.763855+0000","last_change":"2023-10-30T13:26:31.763906+0000","last_active":"2023-10-30T13:26:31.763855+0000","last_peered":"2023-10-30T13:26:31.763855+0000","last_clean":"2023-10-30T13:26:31.763855+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T13:26:31.763855+0000","last_undegraded":"2023-10-30T13:26:31.763855+0000","last_fullsized":"2023-10-30T13:26:31.763855+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2025","last_scrub_stamp":"2023-10-30T13:26:31.763774+0000","last_deep_scrub":"85'2025","last_deep_scrub_stamp":"2023-10-30T13:26:31.763774+0000","last_clean_scrub_stamp":"2023-10-30T13:26:31.763774+0000","objects_scrubbed":549,"log_size":2025,"log_dups_size":0,"ondisk_log_size":2025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:58.078390+0000","scrub_duration":1.0885005139999999,"objects_trimmed":0,"snaptrim_duration":0.0090148399999999997,"stat_sum":{"num_bytes":11081748,"num_objects":549,"num_object_clones":251,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":201,"num_read":142,"num_read_kb":9474,"num_write":1288,"num_write_kb":13652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"85'2406","reported_seq":4433,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.442339+0000","last_change":"2023-10-30T13:25:54.442416+0000","last_active":"2023-10-30T13:25:54.442339+0000","last_peered":"2023-10-30T13:25:54.442339+0000","last_clean":"2023-10-30T13:25:54.442339+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T13:25:54.442339+0000","last_undegraded":"2023-10-30T13:25:54.442339+0000","last_fullsized":"2023-10-30T13:25:54.442339+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2406","last_scrub_stamp":"2023-10-30T13:25:54.442247+0000","last_deep_scrub":"85'2406","last_deep_scrub_stamp":"2023-10-30T13:25:54.442247+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.442247+0000","objects_scrubbed":693,"log_size":2406,"log_dups_size":0,"ondisk_log_size":2406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:12.648322+0000","scrub_duration":1.2677238399999999,"objects_trimmed":0,"snaptrim_duration":0.0066657749999999997,"stat_sum":{"num_bytes":20125251,"num_objects":693,"num_object_clones":330,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":267,"num_read":168,"num_read_kb":25254,"num_write":1457,"num_write_kb":26871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"85'2152","reported_seq":4191,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:45.218457+0000","last_change":"2023-10-30T13:25:45.218569+0000","last_active":"2023-10-30T13:25:45.218457+0000","last_peered":"2023-10-30T13:25:45.218457+0000","last_clean":"2023-10-30T13:25:45.218457+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T13:25:45.218457+0000","last_undegraded":"2023-10-30T13:25:45.218457+0000","last_fullsized":"2023-10-30T13:25:45.218457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2152","last_scrub_stamp":"2023-10-30T13:25:45.218360+0000","last_deep_scrub":"85'2152","last_deep_scrub_stamp":"2023-10-30T13:25:45.218360+0000","last_clean_scrub_stamp":"2023-10-30T13:25:45.218360+0000","objects_scrubbed":604,"log_size":2152,"log_dups_size":0,"ondisk_log_size":2152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:53:35.163970+0000","scrub_duration":1.1092469380000001,"objects_trimmed":0,"snaptrim_duration":0.005419129,"stat_sum":{"num_bytes":18031744,"num_objects":604,"num_object_clones":282,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":228,"num_read":136,"num_read_kb":25964,"num_write":1344,"num_write_kb":19625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"85'2155","reported_seq":4202,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:00.594900+0000","last_change":"2023-10-30T13:26:00.594900+0000","last_active":"2023-10-30T13:26:00.594900+0000","last_peered":"2023-10-30T13:26:00.594900+0000","last_clean":"2023-10-30T13:26:00.594900+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T13:26:00.594900+0000","last_undegraded":"2023-10-30T13:26:00.594900+0000","last_fullsized":"2023-10-30T13:26:00.594900+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2155","last_scrub_stamp":"2023-10-30T13:26:00.594690+0000","last_deep_scrub":"85'2155","last_deep_scrub_stamp":"2023-10-30T13:26:00.594690+0000","last_clean_scrub_stamp":"2023-10-30T13:26:00.594690+0000","objects_scrubbed":607,"log_size":2155,"log_dups_size":0,"ondisk_log_size":2155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:43.537640+0000","scrub_duration":1.1209885449999999,"objects_trimmed":0,"snaptrim_duration":0.0013742470000000001,"stat_sum":{"num_bytes":22381695,"num_objects":607,"num_object_clones":275,"num_object_copies":1214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":227,"num_read":141,"num_read_kb":33421,"num_write":1348,"num_write_kb":23422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1042","reported_seq":1466,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:42.402455+0000","last_change":"2023-10-30T13:25:42.402455+0000","last_active":"2023-10-30T13:25:42.402455+0000","last_peered":"2023-10-30T13:25:42.402455+0000","last_clean":"2023-10-30T13:25:42.402455+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T13:25:42.402455+0000","last_undegraded":"2023-10-30T13:25:42.402455+0000","last_fullsized":"2023-10-30T13:25:42.402455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1042","last_scrub_stamp":"2023-10-30T13:25:42.402195+0000","last_deep_scrub":"85'1042","last_deep_scrub_stamp":"2023-10-30T13:25:42.402195+0000","last_clean_scrub_stamp":"2023-10-30T13:25:42.402195+0000","objects_scrubbed":51,"log_size":1042,"log_dups_size":0,"ondisk_log_size":1042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:20.707501+0000","scrub_duration":0.156286863,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":570,"num_read_kb":319827,"num_write":1218,"num_write_kb":80521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":794018,"num_omap_keys":1312,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"85'2177","reported_seq":4221,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:01.438545+0000","last_change":"2023-10-30T13:26:01.438545+0000","last_active":"2023-10-30T13:26:01.438545+0000","last_peered":"2023-10-30T13:26:01.438545+0000","last_clean":"2023-10-30T13:26:01.438545+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T13:26:01.438545+0000","last_undegraded":"2023-10-30T13:26:01.438545+0000","last_fullsized":"2023-10-30T13:26:01.438545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2177","last_scrub_stamp":"2023-10-30T13:26:01.438339+0000","last_deep_scrub":"85'2177","last_deep_scrub_stamp":"2023-10-30T13:26:01.438339+0000","last_clean_scrub_stamp":"2023-10-30T13:26:01.438339+0000","objects_scrubbed":616,"log_size":2177,"log_dups_size":0,"ondisk_log_size":2177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:25.473052+0000","scrub_duration":1.2567298680000001,"objects_trimmed":0,"snaptrim_duration":0.0086018069999999995,"stat_sum":{"num_bytes":19940185,"num_objects":616,"num_object_clones":283,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":213,"num_read":145,"num_read_kb":23669,"num_write":1398,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1837","reported_seq":2428,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:15.510377+0000","last_change":"2023-10-30T13:26:15.510377+0000","last_active":"2023-10-30T13:26:15.510377+0000","last_peered":"2023-10-30T13:26:15.510377+0000","last_clean":"2023-10-30T13:26:15.510377+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T13:26:15.510377+0000","last_undegraded":"2023-10-30T13:26:15.510377+0000","last_fullsized":"2023-10-30T13:26:15.510377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1837","last_scrub_stamp":"2023-10-30T13:26:15.510196+0000","last_deep_scrub":"85'1837","last_deep_scrub_stamp":"2023-10-30T13:26:15.510196+0000","last_clean_scrub_stamp":"2023-10-30T13:26:15.510196+0000","objects_scrubbed":42,"log_size":1837,"log_dups_size":0,"ondisk_log_size":1837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:16:27.071396+0000","scrub_duration":0.084593108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17028825,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":761,"num_read_kb":396040,"num_write":1931,"num_write_kb":84639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":554115,"num_omap_keys":1012,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"85'1626","reported_seq":2455,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:46.170208+0000","last_change":"2023-10-30T13:25:46.170305+0000","last_active":"2023-10-30T13:25:46.170208+0000","last_peered":"2023-10-30T13:25:46.170208+0000","last_clean":"2023-10-30T13:25:46.170208+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T13:25:46.170208+0000","last_undegraded":"2023-10-30T13:25:46.170208+0000","last_fullsized":"2023-10-30T13:25:46.170208+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1626","last_scrub_stamp":"2023-10-30T13:25:46.170040+0000","last_deep_scrub":"85'1626","last_deep_scrub_stamp":"2023-10-30T13:25:46.170040+0000","last_clean_scrub_stamp":"2023-10-30T13:25:46.170040+0000","objects_scrubbed":57,"log_size":1626,"log_dups_size":0,"ondisk_log_size":1626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:30:38.684197+0000","scrub_duration":0.213161079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182949,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":1117,"num_read_kb":268002,"num_write":1800,"num_write_kb":63122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":717548,"num_omap_keys":1302,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"85'2510","reported_seq":4555,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.637463+0000","last_change":"2023-10-30T13:25:54.637570+0000","last_active":"2023-10-30T13:25:54.637463+0000","last_peered":"2023-10-30T13:25:54.637463+0000","last_clean":"2023-10-30T13:25:54.637463+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T13:25:54.637463+0000","last_undegraded":"2023-10-30T13:25:54.637463+0000","last_fullsized":"2023-10-30T13:25:54.637463+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2510","last_scrub_stamp":"2023-10-30T13:25:54.637344+0000","last_deep_scrub":"85'2510","last_deep_scrub_stamp":"2023-10-30T13:25:54.637344+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.637344+0000","objects_scrubbed":709,"log_size":2510,"log_dups_size":0,"ondisk_log_size":2510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:29:25.404778+0000","scrub_duration":1.3090190449999999,"objects_trimmed":0,"snaptrim_duration":0.016143469000000001,"stat_sum":{"num_bytes":14540052,"num_objects":709,"num_object_clones":329,"num_object_copies":1418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":254,"num_read":191,"num_read_kb":6719,"num_write":1583,"num_write_kb":26382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"85'978","reported_seq":1390,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:00.613642+0000","last_change":"2023-10-30T13:26:00.613714+0000","last_active":"2023-10-30T13:26:00.613642+0000","last_peered":"2023-10-30T13:26:00.613642+0000","last_clean":"2023-10-30T13:26:00.613642+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T13:26:00.613642+0000","last_undegraded":"2023-10-30T13:26:00.613642+0000","last_fullsized":"2023-10-30T13:26:00.613642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'978","last_scrub_stamp":"2023-10-30T13:26:00.613566+0000","last_deep_scrub":"85'978","last_deep_scrub_stamp":"2023-10-30T13:26:00.613566+0000","last_clean_scrub_stamp":"2023-10-30T13:26:00.613566+0000","objects_scrubbed":48,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:00:35.001259+0000","scrub_duration":0.17839571300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":542,"num_read_kb":359029,"num_write":1093,"num_write_kb":88246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":622199,"num_omap_keys":1035,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"85'2113","reported_seq":4164,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:55.071692+0000","last_change":"2023-10-30T13:25:55.071766+0000","last_active":"2023-10-30T13:25:55.071692+0000","last_peered":"2023-10-30T13:25:55.071692+0000","last_clean":"2023-10-30T13:25:55.071692+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T13:25:55.071692+0000","last_undegraded":"2023-10-30T13:25:55.071692+0000","last_fullsized":"2023-10-30T13:25:55.071692+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2113","last_scrub_stamp":"2023-10-30T13:25:55.071614+0000","last_deep_scrub":"85'2113","last_deep_scrub_stamp":"2023-10-30T13:25:55.071614+0000","last_clean_scrub_stamp":"2023-10-30T13:25:55.071614+0000","objects_scrubbed":589,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:07.445157+0000","scrub_duration":0.93122268500000005,"objects_trimmed":0,"snaptrim_duration":0.0083101389999999994,"stat_sum":{"num_bytes":18904879,"num_objects":589,"num_object_clones":276,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":211,"num_read":147,"num_read_kb":20021,"num_write":1340,"num_write_kb":25244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"85'789","reported_seq":1200,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:49.688428+0000","last_change":"2023-10-30T13:25:49.688470+0000","last_active":"2023-10-30T13:25:49.688428+0000","last_peered":"2023-10-30T13:25:49.688428+0000","last_clean":"2023-10-30T13:25:49.688428+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T13:25:49.688428+0000","last_undegraded":"2023-10-30T13:25:49.688428+0000","last_fullsized":"2023-10-30T13:25:49.688428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'789","last_scrub_stamp":"2023-10-30T13:25:49.688344+0000","last_deep_scrub":"85'789","last_deep_scrub_stamp":"2023-10-30T13:25:49.688344+0000","last_clean_scrub_stamp":"2023-10-30T13:25:49.688344+0000","objects_scrubbed":53,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:56.666560+0000","scrub_duration":0.15913772100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":616,"num_read_kb":251423,"num_write":890,"num_write_kb":55054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":627193,"num_omap_keys":1090,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"85'2098","reported_seq":4145,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.184775+0000","last_change":"2023-10-30T13:25:57.184868+0000","last_active":"2023-10-30T13:25:57.184775+0000","last_peered":"2023-10-30T13:25:57.184775+0000","last_clean":"2023-10-30T13:25:57.184775+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T13:25:57.184775+0000","last_undegraded":"2023-10-30T13:25:57.184775+0000","last_fullsized":"2023-10-30T13:25:57.184775+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2098","last_scrub_stamp":"2023-10-30T13:25:57.184676+0000","last_deep_scrub":"85'2098","last_deep_scrub_stamp":"2023-10-30T13:25:57.184676+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.184676+0000","objects_scrubbed":580,"log_size":2098,"log_dups_size":0,"ondisk_log_size":2098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:36.904344+0000","scrub_duration":1.084383391,"objects_trimmed":0,"snaptrim_duration":0.0043045619999999996,"stat_sum":{"num_bytes":10134301,"num_objects":580,"num_object_clones":271,"num_object_copies":1160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":580,"num_whiteouts":219,"num_read":138,"num_read_kb":8986,"num_write":1318,"num_write_kb":11276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"85'1309","reported_seq":1758,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.351882+0000","last_change":"2023-10-30T13:25:54.351972+0000","last_active":"2023-10-30T13:25:54.351882+0000","last_peered":"2023-10-30T13:25:54.351882+0000","last_clean":"2023-10-30T13:25:54.351882+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T13:25:54.351882+0000","last_undegraded":"2023-10-30T13:25:54.351882+0000","last_fullsized":"2023-10-30T13:25:54.351882+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1309","last_scrub_stamp":"2023-10-30T13:25:54.351769+0000","last_deep_scrub":"85'1309","last_deep_scrub_stamp":"2023-10-30T13:25:54.351769+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.351769+0000","objects_scrubbed":62,"log_size":1309,"log_dups_size":0,"ondisk_log_size":1309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:15.955314+0000","scrub_duration":0.18331900800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":659,"num_read_kb":339384,"num_write":1492,"num_write_kb":80693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":875213,"num_omap_keys":1399,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"85'2223","reported_seq":4246,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:59.747975+0000","last_change":"2023-10-30T13:25:59.748025+0000","last_active":"2023-10-30T13:25:59.747975+0000","last_peered":"2023-10-30T13:25:59.747975+0000","last_clean":"2023-10-30T13:25:59.747975+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T13:25:59.747975+0000","last_undegraded":"2023-10-30T13:25:59.747975+0000","last_fullsized":"2023-10-30T13:25:59.747975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2223","last_scrub_stamp":"2023-10-30T13:25:59.747898+0000","last_deep_scrub":"85'2223","last_deep_scrub_stamp":"2023-10-30T13:25:59.747898+0000","last_clean_scrub_stamp":"2023-10-30T13:25:59.747898+0000","objects_scrubbed":620,"log_size":2223,"log_dups_size":0,"ondisk_log_size":2223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:22.443096+0000","scrub_duration":1.320524545,"objects_trimmed":0,"snaptrim_duration":0.015702779,"stat_sum":{"num_bytes":15773172,"num_objects":620,"num_object_clones":292,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":204,"num_read":135,"num_read_kb":17351,"num_write":1431,"num_write_kb":17077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"85'1282","reported_seq":1786,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:43.374996+0000","last_change":"2023-10-30T13:25:43.375075+0000","last_active":"2023-10-30T13:25:43.374996+0000","last_peered":"2023-10-30T13:25:43.374996+0000","last_clean":"2023-10-30T13:25:43.374996+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T13:25:43.374996+0000","last_undegraded":"2023-10-30T13:25:43.374996+0000","last_fullsized":"2023-10-30T13:25:43.374996+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1282","last_scrub_stamp":"2023-10-30T13:25:43.374894+0000","last_deep_scrub":"85'1282","last_deep_scrub_stamp":"2023-10-30T13:25:43.374894+0000","last_clean_scrub_stamp":"2023-10-30T13:25:43.374894+0000","objects_scrubbed":59,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:32.582441+0000","scrub_duration":0.20001206899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19774519,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":862,"num_read_kb":221068,"num_write":1454,"num_write_kb":51536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":878727,"num_omap_keys":1229,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"85'2127","reported_seq":4170,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:30.367224+0000","last_change":"2023-10-30T13:26:30.367270+0000","last_active":"2023-10-30T13:26:30.367224+0000","last_peered":"2023-10-30T13:26:30.367224+0000","last_clean":"2023-10-30T13:26:30.367224+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T13:26:30.367224+0000","last_undegraded":"2023-10-30T13:26:30.367224+0000","last_fullsized":"2023-10-30T13:26:30.367224+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2127","last_scrub_stamp":"2023-10-30T13:26:30.367149+0000","last_deep_scrub":"85'2127","last_deep_scrub_stamp":"2023-10-30T13:26:30.367149+0000","last_clean_scrub_stamp":"2023-10-30T13:26:30.367149+0000","objects_scrubbed":612,"log_size":2127,"log_dups_size":0,"ondisk_log_size":2127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:03:49.227794+0000","scrub_duration":0.73352292799999996,"objects_trimmed":0,"snaptrim_duration":0.0080610860000000003,"stat_sum":{"num_bytes":22077043,"num_objects":612,"num_object_clones":281,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":230,"num_read":141,"num_read_kb":18856,"num_write":1294,"num_write_kb":28337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"85'1397","reported_seq":1954,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:41.415232+0000","last_change":"2023-10-30T13:25:41.415318+0000","last_active":"2023-10-30T13:25:41.415232+0000","last_peered":"2023-10-30T13:25:41.415232+0000","last_clean":"2023-10-30T13:25:41.415232+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T13:25:41.415232+0000","last_undegraded":"2023-10-30T13:25:41.415232+0000","last_fullsized":"2023-10-30T13:25:41.415232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1397","last_scrub_stamp":"2023-10-30T13:25:41.415138+0000","last_deep_scrub":"85'1397","last_deep_scrub_stamp":"2023-10-30T13:25:41.415138+0000","last_clean_scrub_stamp":"2023-10-30T13:25:41.415138+0000","objects_scrubbed":67,"log_size":1397,"log_dups_size":0,"ondisk_log_size":1397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:14.524619+0000","scrub_duration":0.28929933099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":865,"num_read_kb":420163,"num_write":1566,"num_write_kb":89397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":814706,"num_omap_keys":1307,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"85'2141","reported_seq":4172,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:01.199914+0000","last_change":"2023-10-30T13:26:01.200009+0000","last_active":"2023-10-30T13:26:01.199914+0000","last_peered":"2023-10-30T13:26:01.199914+0000","last_clean":"2023-10-30T13:26:01.199914+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T13:26:01.199914+0000","last_undegraded":"2023-10-30T13:26:01.199914+0000","last_fullsized":"2023-10-30T13:26:01.199914+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2141","last_scrub_stamp":"2023-10-30T13:26:01.199816+0000","last_deep_scrub":"85'2141","last_deep_scrub_stamp":"2023-10-30T13:26:01.199816+0000","last_clean_scrub_stamp":"2023-10-30T13:26:01.199816+0000","objects_scrubbed":606,"log_size":2141,"log_dups_size":0,"ondisk_log_size":2141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:21:03.156283+0000","scrub_duration":1.132667525,"objects_trimmed":0,"snaptrim_duration":0.008640567,"stat_sum":{"num_bytes":23022369,"num_objects":606,"num_object_clones":279,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":220,"num_read":129,"num_read_kb":18941,"num_write":1340,"num_write_kb":24878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"85'1164","reported_seq":1635,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.276943+0000","last_change":"2023-10-30T13:25:48.277044+0000","last_active":"2023-10-30T13:25:48.276943+0000","last_peered":"2023-10-30T13:25:48.276943+0000","last_clean":"2023-10-30T13:25:48.276943+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T13:25:48.276943+0000","last_undegraded":"2023-10-30T13:25:48.276943+0000","last_fullsized":"2023-10-30T13:25:48.276943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1164","last_scrub_stamp":"2023-10-30T13:25:48.276852+0000","last_deep_scrub":"85'1164","last_deep_scrub_stamp":"2023-10-30T13:25:48.276852+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.276852+0000","objects_scrubbed":66,"log_size":1164,"log_dups_size":0,"ondisk_log_size":1164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:06.856285+0000","scrub_duration":0.19954839099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554466,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":701,"num_read_kb":318483,"num_write":1303,"num_write_kb":68223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":676667,"num_omap_keys":1245,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"85'2197","reported_seq":4231,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:52.017102+0000","last_change":"2023-10-30T13:25:52.017193+0000","last_active":"2023-10-30T13:25:52.017102+0000","last_peered":"2023-10-30T13:25:52.017102+0000","last_clean":"2023-10-30T13:25:52.017102+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T13:25:52.017102+0000","last_undegraded":"2023-10-30T13:25:52.017102+0000","last_fullsized":"2023-10-30T13:25:52.017102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2197","last_scrub_stamp":"2023-10-30T13:25:52.016976+0000","last_deep_scrub":"85'2197","last_deep_scrub_stamp":"2023-10-30T13:25:52.016976+0000","last_clean_scrub_stamp":"2023-10-30T13:25:52.016976+0000","objects_scrubbed":613,"log_size":2197,"log_dups_size":0,"ondisk_log_size":2197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:09:13.519760+0000","scrub_duration":1.011428851,"objects_trimmed":0,"snaptrim_duration":0.0073009930000000004,"stat_sum":{"num_bytes":10084838,"num_objects":613,"num_object_clones":288,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":217,"num_read":139,"num_read_kb":5686,"num_write":1383,"num_write_kb":20147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'729","reported_seq":1132,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.578886+0000","last_change":"2023-10-30T13:25:54.578930+0000","last_active":"2023-10-30T13:25:54.578886+0000","last_peered":"2023-10-30T13:25:54.578886+0000","last_clean":"2023-10-30T13:25:54.578886+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T13:25:54.578886+0000","last_undegraded":"2023-10-30T13:25:54.578886+0000","last_fullsized":"2023-10-30T13:25:54.578886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'729","last_scrub_stamp":"2023-10-30T13:25:54.578800+0000","last_deep_scrub":"85'729","last_deep_scrub_stamp":"2023-10-30T13:25:54.578800+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.578800+0000","objects_scrubbed":53,"log_size":729,"log_dups_size":0,"ondisk_log_size":729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:06:15.394870+0000","scrub_duration":0.12999461300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":600,"num_read_kb":208389,"num_write":899,"num_write_kb":51587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":569345,"num_omap_keys":982,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"85'2297","reported_seq":4342,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:22.388451+0000","last_change":"2023-10-30T13:26:22.388507+0000","last_active":"2023-10-30T13:26:22.388451+0000","last_peered":"2023-10-30T13:26:22.388451+0000","last_clean":"2023-10-30T13:26:22.388451+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T13:26:22.388451+0000","last_undegraded":"2023-10-30T13:26:22.388451+0000","last_fullsized":"2023-10-30T13:26:22.388451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2297","last_scrub_stamp":"2023-10-30T13:26:22.388371+0000","last_deep_scrub":"85'2297","last_deep_scrub_stamp":"2023-10-30T13:26:22.388371+0000","last_clean_scrub_stamp":"2023-10-30T13:26:22.388371+0000","objects_scrubbed":655,"log_size":2297,"log_dups_size":0,"ondisk_log_size":2297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:56.894212+0000","scrub_duration":0.80626637899999998,"objects_trimmed":0,"snaptrim_duration":0.0091258150000000007,"stat_sum":{"num_bytes":19392609,"num_objects":655,"num_object_clones":299,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":233,"num_read":161,"num_read_kb":7475,"num_write":1440,"num_write_kb":20643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"85'1416","reported_seq":1810,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:40.394192+0000","last_change":"2023-10-30T13:25:40.394262+0000","last_active":"2023-10-30T13:25:40.394192+0000","last_peered":"2023-10-30T13:25:40.394192+0000","last_clean":"2023-10-30T13:25:40.394192+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T13:25:40.394192+0000","last_undegraded":"2023-10-30T13:25:40.394192+0000","last_fullsized":"2023-10-30T13:25:40.394192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1416","last_scrub_stamp":"2023-10-30T13:25:40.394118+0000","last_deep_scrub":"85'1416","last_deep_scrub_stamp":"2023-10-30T13:25:40.394118+0000","last_clean_scrub_stamp":"2023-10-30T13:25:40.394118+0000","objects_scrubbed":45,"log_size":1416,"log_dups_size":0,"ondisk_log_size":1416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:50:35.732748+0000","scrub_duration":0.10710792099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":539,"num_read_kb":266844,"num_write":1512,"num_write_kb":63647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":566794,"num_omap_keys":1066,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"85'2071","reported_seq":4113,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:58.521640+0000","last_change":"2023-10-30T13:25:58.521719+0000","last_active":"2023-10-30T13:25:58.521640+0000","last_peered":"2023-10-30T13:25:58.521640+0000","last_clean":"2023-10-30T13:25:58.521640+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T13:25:58.521640+0000","last_undegraded":"2023-10-30T13:25:58.521640+0000","last_fullsized":"2023-10-30T13:25:58.521640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2071","last_scrub_stamp":"2023-10-30T13:25:58.521516+0000","last_deep_scrub":"85'2071","last_deep_scrub_stamp":"2023-10-30T13:25:58.521516+0000","last_clean_scrub_stamp":"2023-10-30T13:25:58.521516+0000","objects_scrubbed":590,"log_size":2071,"log_dups_size":0,"ondisk_log_size":2071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:24:10.505945+0000","scrub_duration":1.0833768969999999,"objects_trimmed":0,"snaptrim_duration":0.001646376,"stat_sum":{"num_bytes":13778361,"num_objects":590,"num_object_clones":278,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":227,"num_read":137,"num_read_kb":5723,"num_write":1258,"num_write_kb":14976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"85'1020","reported_seq":1516,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:45.278920+0000","last_change":"2023-10-30T13:25:45.278968+0000","last_active":"2023-10-30T13:25:45.278920+0000","last_peered":"2023-10-30T13:25:45.278920+0000","last_clean":"2023-10-30T13:25:45.278920+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T13:25:45.278920+0000","last_undegraded":"2023-10-30T13:25:45.278920+0000","last_fullsized":"2023-10-30T13:25:45.278920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1020","last_scrub_stamp":"2023-10-30T13:25:45.278835+0000","last_deep_scrub":"85'1020","last_deep_scrub_stamp":"2023-10-30T13:25:45.278835+0000","last_clean_scrub_stamp":"2023-10-30T13:25:45.278835+0000","objects_scrubbed":50,"log_size":1020,"log_dups_size":0,"ondisk_log_size":1020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:18:17.671466+0000","scrub_duration":0.27418579599999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554544,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":863,"num_read_kb":272228,"num_write":1253,"num_write_kb":56723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":534875,"num_omap_keys":741,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"85'2238","reported_seq":4293,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:18.465655+0000","last_change":"2023-10-30T13:26:18.465733+0000","last_active":"2023-10-30T13:26:18.465655+0000","last_peered":"2023-10-30T13:26:18.465655+0000","last_clean":"2023-10-30T13:26:18.465655+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T13:26:18.465655+0000","last_undegraded":"2023-10-30T13:26:18.465655+0000","last_fullsized":"2023-10-30T13:26:18.465655+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2238","last_scrub_stamp":"2023-10-30T13:26:18.465564+0000","last_deep_scrub":"85'2238","last_deep_scrub_stamp":"2023-10-30T13:26:18.465564+0000","last_clean_scrub_stamp":"2023-10-30T13:26:18.465564+0000","objects_scrubbed":622,"log_size":2238,"log_dups_size":0,"ondisk_log_size":2238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:04.508589+0000","scrub_duration":0.99352104299999999,"objects_trimmed":0,"snaptrim_duration":0.0098326820000000006,"stat_sum":{"num_bytes":10901701,"num_objects":622,"num_object_clones":292,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":232,"num_read":163,"num_read_kb":14238,"num_write":1406,"num_write_kb":21652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"85'696","reported_seq":1065,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:42.423508+0000","last_change":"2023-10-30T13:25:42.423609+0000","last_active":"2023-10-30T13:25:42.423508+0000","last_peered":"2023-10-30T13:25:42.423508+0000","last_clean":"2023-10-30T13:25:42.423508+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T13:25:42.423508+0000","last_undegraded":"2023-10-30T13:25:42.423508+0000","last_fullsized":"2023-10-30T13:25:42.423508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'696","last_scrub_stamp":"2023-10-30T13:25:42.423407+0000","last_deep_scrub":"85'696","last_deep_scrub_stamp":"2023-10-30T13:25:42.423407+0000","last_clean_scrub_stamp":"2023-10-30T13:25:42.423407+0000","objects_scrubbed":41,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:12:00.329817+0000","scrub_duration":0.18306038499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":443,"num_read_kb":261670,"num_write":858,"num_write_kb":58986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":305625,"num_omap_keys":524,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"85'2220","reported_seq":4221,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.104935+0000","last_change":"2023-10-30T13:25:53.105023+0000","last_active":"2023-10-30T13:25:53.104935+0000","last_peered":"2023-10-30T13:25:53.104935+0000","last_clean":"2023-10-30T13:25:53.104935+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T13:25:53.104935+0000","last_undegraded":"2023-10-30T13:25:53.104935+0000","last_fullsized":"2023-10-30T13:25:53.104935+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2220","last_scrub_stamp":"2023-10-30T13:25:53.104840+0000","last_deep_scrub":"85'2220","last_deep_scrub_stamp":"2023-10-30T13:25:53.104840+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.104840+0000","objects_scrubbed":647,"log_size":2220,"log_dups_size":0,"ondisk_log_size":2220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:47:46.541296+0000","scrub_duration":1.013459932,"objects_trimmed":0,"snaptrim_duration":0.004602437,"stat_sum":{"num_bytes":20428330,"num_objects":647,"num_object_clones":304,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":239,"num_read":119,"num_read_kb":13097,"num_write":1360,"num_write_kb":26395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"85'779","reported_seq":1239,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.328717+0000","last_change":"2023-10-30T13:25:48.328798+0000","last_active":"2023-10-30T13:25:48.328717+0000","last_peered":"2023-10-30T13:25:48.328717+0000","last_clean":"2023-10-30T13:25:48.328717+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T13:25:48.328717+0000","last_undegraded":"2023-10-30T13:25:48.328717+0000","last_fullsized":"2023-10-30T13:25:48.328717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'779","last_scrub_stamp":"2023-10-30T13:25:48.328628+0000","last_deep_scrub":"85'779","last_deep_scrub_stamp":"2023-10-30T13:25:48.328628+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.328628+0000","objects_scrubbed":62,"log_size":779,"log_dups_size":0,"ondisk_log_size":779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:51.114462+0000","scrub_duration":0.22182811399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":635,"num_read_kb":434171,"num_write":906,"num_write_kb":88121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":675117,"num_omap_keys":1207,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"85'2216","reported_seq":4237,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:59.936763+0000","last_change":"2023-10-30T13:25:59.936829+0000","last_active":"2023-10-30T13:25:59.936763+0000","last_peered":"2023-10-30T13:25:59.936763+0000","last_clean":"2023-10-30T13:25:59.936763+0000","last_became_active":"2023-10-30T12:29:27.340504+0000","last_became_peered":"2023-10-30T12:29:27.340504+0000","last_unstale":"2023-10-30T13:25:59.936763+0000","last_undegraded":"2023-10-30T13:25:59.936763+0000","last_fullsized":"2023-10-30T13:25:59.936763+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2216","last_scrub_stamp":"2023-10-30T13:25:59.936683+0000","last_deep_scrub":"85'2216","last_deep_scrub_stamp":"2023-10-30T13:25:59.936683+0000","last_clean_scrub_stamp":"2023-10-30T13:25:59.936683+0000","objects_scrubbed":641,"log_size":2216,"log_dups_size":0,"ondisk_log_size":2216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:57:07.443321+0000","scrub_duration":1.7189143169999999,"objects_trimmed":0,"snaptrim_duration":0.00653923,"stat_sum":{"num_bytes":23737065,"num_objects":641,"num_object_clones":299,"num_object_copies":1282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":641,"num_whiteouts":238,"num_read":138,"num_read_kb":28255,"num_write":1351,"num_write_kb":33550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'766","reported_seq":1157,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:51.439924+0000","last_change":"2023-10-30T13:25:51.439991+0000","last_active":"2023-10-30T13:25:51.439924+0000","last_peered":"2023-10-30T13:25:51.439924+0000","last_clean":"2023-10-30T13:25:51.439924+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T13:25:51.439924+0000","last_undegraded":"2023-10-30T13:25:51.439924+0000","last_fullsized":"2023-10-30T13:25:51.439924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'766","last_scrub_stamp":"2023-10-30T13:25:51.439810+0000","last_deep_scrub":"85'766","last_deep_scrub_stamp":"2023-10-30T13:25:51.439810+0000","last_clean_scrub_stamp":"2023-10-30T13:25:51.439810+0000","objects_scrubbed":50,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:56.709508+0000","scrub_duration":0.13294180999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364369,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":493,"num_read_kb":320778,"num_write":891,"num_write_kb":71265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":522473,"num_omap_keys":1010,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'2210","reported_seq":4257,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:08.440439+0000","last_change":"2023-10-30T13:26:08.440557+0000","last_active":"2023-10-30T13:26:08.440439+0000","last_peered":"2023-10-30T13:26:08.440439+0000","last_clean":"2023-10-30T13:26:08.440439+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T13:26:08.440439+0000","last_undegraded":"2023-10-30T13:26:08.440439+0000","last_fullsized":"2023-10-30T13:26:08.440439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2210","last_scrub_stamp":"2023-10-30T13:26:08.440312+0000","last_deep_scrub":"85'2210","last_deep_scrub_stamp":"2023-10-30T13:26:08.440312+0000","last_clean_scrub_stamp":"2023-10-30T13:26:08.440312+0000","objects_scrubbed":611,"log_size":2210,"log_dups_size":0,"ondisk_log_size":2210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:49.728570+0000","scrub_duration":1.28748733,"objects_trimmed":0,"snaptrim_duration":0.0076673150000000001,"stat_sum":{"num_bytes":18256447,"num_objects":611,"num_object_clones":286,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":229,"num_read":149,"num_read_kb":24381,"num_write":1383,"num_write_kb":20195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"85'645","reported_seq":1007,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.161404+0000","last_change":"2023-10-30T13:25:48.161534+0000","last_active":"2023-10-30T13:25:48.161404+0000","last_peered":"2023-10-30T13:25:48.161404+0000","last_clean":"2023-10-30T13:25:48.161404+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T13:25:48.161404+0000","last_undegraded":"2023-10-30T13:25:48.161404+0000","last_fullsized":"2023-10-30T13:25:48.161404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'645","last_scrub_stamp":"2023-10-30T13:25:48.161279+0000","last_deep_scrub":"85'645","last_deep_scrub_stamp":"2023-10-30T13:25:48.161279+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.161279+0000","objects_scrubbed":38,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:29.280017+0000","scrub_duration":0.15056981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360710,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":462,"num_read_kb":342669,"num_write":689,"num_write_kb":66904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":355436,"num_omap_keys":678,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"85'2132","reported_seq":4147,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:36.536448+0000","last_change":"2023-10-30T13:26:36.536557+0000","last_active":"2023-10-30T13:26:36.536448+0000","last_peered":"2023-10-30T13:26:36.536448+0000","last_clean":"2023-10-30T13:26:36.536448+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T13:26:36.536448+0000","last_undegraded":"2023-10-30T13:26:36.536448+0000","last_fullsized":"2023-10-30T13:26:36.536448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2132","last_scrub_stamp":"2023-10-30T13:26:36.536359+0000","last_deep_scrub":"85'2132","last_deep_scrub_stamp":"2023-10-30T13:26:36.536359+0000","last_clean_scrub_stamp":"2023-10-30T13:26:36.536359+0000","objects_scrubbed":612,"log_size":2132,"log_dups_size":0,"ondisk_log_size":2132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:41:52.662853+0000","scrub_duration":0.86036190300000004,"objects_trimmed":0,"snaptrim_duration":0.010286791999999999,"stat_sum":{"num_bytes":26944901,"num_objects":612,"num_object_clones":287,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":236,"num_read":119,"num_read_kb":34683,"num_write":1299,"num_write_kb":33129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"85'119","reported_seq":267,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:46.486940+0000","last_change":"2023-10-30T13:25:46.486986+0000","last_active":"2023-10-30T13:25:46.486940+0000","last_peered":"2023-10-30T13:25:46.486940+0000","last_clean":"2023-10-30T13:25:46.486940+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T13:25:46.486940+0000","last_undegraded":"2023-10-30T13:25:46.486940+0000","last_fullsized":"2023-10-30T13:25:46.486940+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'119","last_scrub_stamp":"2023-10-30T13:25:46.486855+0000","last_deep_scrub":"85'119","last_deep_scrub_stamp":"2023-10-30T13:25:46.486855+0000","last_clean_scrub_stamp":"2023-10-30T13:25:46.486855+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:13.395892+0000","scrub_duration":0.0083870119999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'1371","reported_seq":1882,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:10.600505+0000","last_change":"2023-10-30T13:26:10.600590+0000","last_active":"2023-10-30T13:26:10.600505+0000","last_peered":"2023-10-30T13:26:10.600505+0000","last_clean":"2023-10-30T13:26:10.600505+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T13:26:10.600505+0000","last_undegraded":"2023-10-30T13:26:10.600505+0000","last_fullsized":"2023-10-30T13:26:10.600505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1371","last_scrub_stamp":"2023-10-30T13:26:10.600409+0000","last_deep_scrub":"85'1371","last_deep_scrub_stamp":"2023-10-30T13:26:10.600409+0000","last_clean_scrub_stamp":"2023-10-30T13:26:10.600409+0000","objects_scrubbed":53,"log_size":1371,"log_dups_size":0,"ondisk_log_size":1371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:15.701205+0000","scrub_duration":0.14720427899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165890,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":877,"num_read_kb":231219,"num_write":1489,"num_write_kb":63137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":475860,"num_omap_keys":919,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"85'2200","reported_seq":4242,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.149503+0000","last_change":"2023-10-30T13:25:57.149588+0000","last_active":"2023-10-30T13:25:57.149503+0000","last_peered":"2023-10-30T13:25:57.149503+0000","last_clean":"2023-10-30T13:25:57.149503+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T13:25:57.149503+0000","last_undegraded":"2023-10-30T13:25:57.149503+0000","last_fullsized":"2023-10-30T13:25:57.149503+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2200","last_scrub_stamp":"2023-10-30T13:25:57.149395+0000","last_deep_scrub":"85'2200","last_deep_scrub_stamp":"2023-10-30T13:25:57.149395+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.149395+0000","objects_scrubbed":626,"log_size":2200,"log_dups_size":0,"ondisk_log_size":2200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:04.364636+0000","scrub_duration":1.0072585730000001,"objects_trimmed":0,"snaptrim_duration":0.0068470270000000003,"stat_sum":{"num_bytes":8097312,"num_objects":626,"num_object_clones":290,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":225,"num_read":146,"num_read_kb":2079,"num_write":1425,"num_write_kb":13192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"85'2234","reported_seq":2615,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.726414+0000","last_change":"2023-10-30T13:25:57.726488+0000","last_active":"2023-10-30T13:25:57.726414+0000","last_peered":"2023-10-30T13:25:57.726414+0000","last_clean":"2023-10-30T13:25:57.726414+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T13:25:57.726414+0000","last_undegraded":"2023-10-30T13:25:57.726414+0000","last_fullsized":"2023-10-30T13:25:57.726414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2234","last_scrub_stamp":"2023-10-30T13:25:57.726328+0000","last_deep_scrub":"85'2234","last_deep_scrub_stamp":"2023-10-30T13:25:57.726328+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.726328+0000","objects_scrubbed":42,"log_size":2234,"log_dups_size":0,"ondisk_log_size":2234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:51.655678+0000","scrub_duration":0.194142807,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":516,"num_read_kb":283293,"num_write":2369,"num_write_kb":72030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":392036,"num_omap_keys":649,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"85'2035","reported_seq":4089,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:05.024953+0000","last_change":"2023-10-30T13:26:05.025106+0000","last_active":"2023-10-30T13:26:05.024953+0000","last_peered":"2023-10-30T13:26:05.024953+0000","last_clean":"2023-10-30T13:26:05.024953+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T13:26:05.024953+0000","last_undegraded":"2023-10-30T13:26:05.024953+0000","last_fullsized":"2023-10-30T13:26:05.024953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2035","last_scrub_stamp":"2023-10-30T13:26:05.024740+0000","last_deep_scrub":"85'2035","last_deep_scrub_stamp":"2023-10-30T13:26:05.024740+0000","last_clean_scrub_stamp":"2023-10-30T13:26:05.024740+0000","objects_scrubbed":570,"log_size":2035,"log_dups_size":0,"ondisk_log_size":2035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:48:20.536984+0000","scrub_duration":1.046857548,"objects_trimmed":0,"snaptrim_duration":0.0057755250000000001,"stat_sum":{"num_bytes":14269585,"num_objects":570,"num_object_clones":262,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":207,"num_read":132,"num_read_kb":18022,"num_write":1264,"num_write_kb":17917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'1898","reported_seq":3194,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:52.380541+0000","last_change":"2023-10-30T13:25:52.380604+0000","last_active":"2023-10-30T13:25:52.380541+0000","last_peered":"2023-10-30T13:25:52.380541+0000","last_clean":"2023-10-30T13:25:52.380541+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T13:25:52.380541+0000","last_undegraded":"2023-10-30T13:25:52.380541+0000","last_fullsized":"2023-10-30T13:25:52.380541+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1898","last_scrub_stamp":"2023-10-30T13:25:52.380459+0000","last_deep_scrub":"85'1898","last_deep_scrub_stamp":"2023-10-30T13:25:52.380459+0000","last_clean_scrub_stamp":"2023-10-30T13:25:52.380459+0000","objects_scrubbed":61,"log_size":1898,"log_dups_size":0,"ondisk_log_size":1898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:04.920583+0000","scrub_duration":0.247546554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49235352,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":1514,"num_read_kb":360508,"num_write":2035,"num_write_kb":92558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":705593,"num_omap_keys":1067,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'2175","reported_seq":4220,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:20.787292+0000","last_change":"2023-10-30T13:26:20.787360+0000","last_active":"2023-10-30T13:26:20.787292+0000","last_peered":"2023-10-30T13:26:20.787292+0000","last_clean":"2023-10-30T13:26:20.787292+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T13:26:20.787292+0000","last_undegraded":"2023-10-30T13:26:20.787292+0000","last_fullsized":"2023-10-30T13:26:20.787292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2175","last_scrub_stamp":"2023-10-30T13:26:20.787197+0000","last_deep_scrub":"85'2175","last_deep_scrub_stamp":"2023-10-30T13:26:20.787197+0000","last_clean_scrub_stamp":"2023-10-30T13:26:20.787197+0000","objects_scrubbed":629,"log_size":2175,"log_dups_size":0,"ondisk_log_size":2175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:46.322449+0000","scrub_duration":1.0662295260000001,"objects_trimmed":0,"snaptrim_duration":0.015331312,"stat_sum":{"num_bytes":28178512,"num_objects":629,"num_object_clones":288,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":231,"num_read":148,"num_read_kb":41809,"num_write":1347,"num_write_kb":29520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'1249","reported_seq":1669,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:09.605267+0000","last_change":"2023-10-30T13:26:09.605318+0000","last_active":"2023-10-30T13:26:09.605267+0000","last_peered":"2023-10-30T13:26:09.605267+0000","last_clean":"2023-10-30T13:26:09.605267+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T13:26:09.605267+0000","last_undegraded":"2023-10-30T13:26:09.605267+0000","last_fullsized":"2023-10-30T13:26:09.605267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1249","last_scrub_stamp":"2023-10-30T13:26:09.605188+0000","last_deep_scrub":"85'1249","last_deep_scrub_stamp":"2023-10-30T13:26:09.605188+0000","last_clean_scrub_stamp":"2023-10-30T13:26:09.605188+0000","objects_scrubbed":53,"log_size":1249,"log_dups_size":0,"ondisk_log_size":1249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:52.759600+0000","scrub_duration":0.149468351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":573,"num_read_kb":334248,"num_write":1338,"num_write_kb":75895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":744856,"num_omap_keys":1335,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"85'2208","reported_seq":4242,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:02.512036+0000","last_change":"2023-10-30T13:26:02.512109+0000","last_active":"2023-10-30T13:26:02.512036+0000","last_peered":"2023-10-30T13:26:02.512036+0000","last_clean":"2023-10-30T13:26:02.512036+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T13:26:02.512036+0000","last_undegraded":"2023-10-30T13:26:02.512036+0000","last_fullsized":"2023-10-30T13:26:02.512036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2208","last_scrub_stamp":"2023-10-30T13:26:02.511965+0000","last_deep_scrub":"85'2208","last_deep_scrub_stamp":"2023-10-30T13:26:02.511965+0000","last_clean_scrub_stamp":"2023-10-30T13:26:02.511965+0000","objects_scrubbed":627,"log_size":2208,"log_dups_size":0,"ondisk_log_size":2208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:27.984589+0000","scrub_duration":1.373745343,"objects_trimmed":0,"snaptrim_duration":0.0051193749999999998,"stat_sum":{"num_bytes":23443395,"num_objects":627,"num_object_clones":293,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":236,"num_read":141,"num_read_kb":26943,"num_write":1376,"num_write_kb":26009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"85'1517","reported_seq":1988,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.476829+0000","last_change":"2023-10-30T13:25:44.476908+0000","last_active":"2023-10-30T13:25:44.476829+0000","last_peered":"2023-10-30T13:25:44.476829+0000","last_clean":"2023-10-30T13:25:44.476829+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T13:25:44.476829+0000","last_undegraded":"2023-10-30T13:25:44.476829+0000","last_fullsized":"2023-10-30T13:25:44.476829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1517","last_scrub_stamp":"2023-10-30T13:25:44.476706+0000","last_deep_scrub":"85'1517","last_deep_scrub_stamp":"2023-10-30T13:25:44.476706+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.476706+0000","objects_scrubbed":50,"log_size":1517,"log_dups_size":0,"ondisk_log_size":1517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:43:12.189799+0000","scrub_duration":0.194542509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":699,"num_read_kb":404466,"num_write":1620,"num_write_kb":92575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":728614,"num_omap_keys":1399,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"85'2249","reported_seq":4286,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:50.476286+0000","last_change":"2023-10-30T13:25:50.476388+0000","last_active":"2023-10-30T13:25:50.476286+0000","last_peered":"2023-10-30T13:25:50.476286+0000","last_clean":"2023-10-30T13:25:50.476286+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T13:25:50.476286+0000","last_undegraded":"2023-10-30T13:25:50.476286+0000","last_fullsized":"2023-10-30T13:25:50.476286+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2249","last_scrub_stamp":"2023-10-30T13:25:50.476194+0000","last_deep_scrub":"85'2249","last_deep_scrub_stamp":"2023-10-30T13:25:50.476194+0000","last_clean_scrub_stamp":"2023-10-30T13:25:50.476194+0000","objects_scrubbed":637,"log_size":2249,"log_dups_size":0,"ondisk_log_size":2249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:30:58.348787+0000","scrub_duration":1.2919135900000001,"objects_trimmed":0,"snaptrim_duration":0.0039865179999999997,"stat_sum":{"num_bytes":21812744,"num_objects":637,"num_object_clones":298,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":235,"num_read":149,"num_read_kb":24572,"num_write":1399,"num_write_kb":31699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"85'1056","reported_seq":1475,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:05.605392+0000","last_change":"2023-10-30T13:26:05.605477+0000","last_active":"2023-10-30T13:26:05.605392+0000","last_peered":"2023-10-30T13:26:05.605392+0000","last_clean":"2023-10-30T13:26:05.605392+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T13:26:05.605392+0000","last_undegraded":"2023-10-30T13:26:05.605392+0000","last_fullsized":"2023-10-30T13:26:05.605392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1056","last_scrub_stamp":"2023-10-30T13:26:05.605294+0000","last_deep_scrub":"85'1056","last_deep_scrub_stamp":"2023-10-30T13:26:05.605294+0000","last_clean_scrub_stamp":"2023-10-30T13:26:05.605294+0000","objects_scrubbed":43,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:08:51.834930+0000","scrub_duration":0.16079274700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":534,"num_read_kb":318720,"num_write":1164,"num_write_kb":75805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":590943,"num_omap_keys":951,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"85'2157","reported_seq":4215,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:04.198525+0000","last_change":"2023-10-30T13:26:04.198592+0000","last_active":"2023-10-30T13:26:04.198525+0000","last_peered":"2023-10-30T13:26:04.198525+0000","last_clean":"2023-10-30T13:26:04.198525+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T13:26:04.198525+0000","last_undegraded":"2023-10-30T13:26:04.198525+0000","last_fullsized":"2023-10-30T13:26:04.198525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2157","last_scrub_stamp":"2023-10-30T13:26:04.198444+0000","last_deep_scrub":"85'2157","last_deep_scrub_stamp":"2023-10-30T13:26:04.198444+0000","last_clean_scrub_stamp":"2023-10-30T13:26:04.198444+0000","objects_scrubbed":615,"log_size":2157,"log_dups_size":0,"ondisk_log_size":2157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:17:59.418030+0000","scrub_duration":1.2099730950000001,"objects_trimmed":0,"snaptrim_duration":0.0070372949999999998,"stat_sum":{"num_bytes":38977850,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":222,"num_read":156,"num_read_kb":47793,"num_write":1351,"num_write_kb":41194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1184","reported_seq":1628,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:50.553950+0000","last_change":"2023-10-30T13:25:50.554036+0000","last_active":"2023-10-30T13:25:50.553950+0000","last_peered":"2023-10-30T13:25:50.553950+0000","last_clean":"2023-10-30T13:25:50.553950+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T13:25:50.553950+0000","last_undegraded":"2023-10-30T13:25:50.553950+0000","last_fullsized":"2023-10-30T13:25:50.553950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1184","last_scrub_stamp":"2023-10-30T13:25:50.553862+0000","last_deep_scrub":"85'1184","last_deep_scrub_stamp":"2023-10-30T13:25:50.553862+0000","last_clean_scrub_stamp":"2023-10-30T13:25:50.553862+0000","objects_scrubbed":47,"log_size":1184,"log_dups_size":0,"ondisk_log_size":1184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:34.636484+0000","scrub_duration":0.480914603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":644,"num_read_kb":249397,"num_write":1860,"num_write_kb":63729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":781147,"num_omap_keys":1113,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"85'2052","reported_seq":4082,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:03.318813+0000","last_change":"2023-10-30T13:26:03.318913+0000","last_active":"2023-10-30T13:26:03.318813+0000","last_peered":"2023-10-30T13:26:03.318813+0000","last_clean":"2023-10-30T13:26:03.318813+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T13:26:03.318813+0000","last_undegraded":"2023-10-30T13:26:03.318813+0000","last_fullsized":"2023-10-30T13:26:03.318813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2052","last_scrub_stamp":"2023-10-30T13:26:03.318707+0000","last_deep_scrub":"85'2052","last_deep_scrub_stamp":"2023-10-30T13:26:03.318707+0000","last_clean_scrub_stamp":"2023-10-30T13:26:03.318707+0000","objects_scrubbed":603,"log_size":2052,"log_dups_size":0,"ondisk_log_size":2052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:58.304469+0000","scrub_duration":1.115999714,"objects_trimmed":0,"snaptrim_duration":0.0074996079999999996,"stat_sum":{"num_bytes":14238343,"num_objects":603,"num_object_clones":280,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":229,"num_read":126,"num_read_kb":9383,"num_write":1239,"num_write_kb":19514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"85'541","reported_seq":901,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:08.512407+0000","last_change":"2023-10-30T13:26:08.512451+0000","last_active":"2023-10-30T13:26:08.512407+0000","last_peered":"2023-10-30T13:26:08.512407+0000","last_clean":"2023-10-30T13:26:08.512407+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T13:26:08.512407+0000","last_undegraded":"2023-10-30T13:26:08.512407+0000","last_fullsized":"2023-10-30T13:26:08.512407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'541","last_scrub_stamp":"2023-10-30T13:26:08.512333+0000","last_deep_scrub":"85'541","last_deep_scrub_stamp":"2023-10-30T13:26:08.512333+0000","last_clean_scrub_stamp":"2023-10-30T13:26:08.512333+0000","objects_scrubbed":51,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:40:06.032449+0000","scrub_duration":0.092969488000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21146631,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":514,"num_read_kb":197148,"num_write":651,"num_write_kb":42826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":495891,"num_omap_keys":618,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"85'2296","reported_seq":4306,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:58.407563+0000","last_change":"2023-10-30T13:25:58.407621+0000","last_active":"2023-10-30T13:25:58.407563+0000","last_peered":"2023-10-30T13:25:58.407563+0000","last_clean":"2023-10-30T13:25:58.407563+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T13:25:58.407563+0000","last_undegraded":"2023-10-30T13:25:58.407563+0000","last_fullsized":"2023-10-30T13:25:58.407563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2296","last_scrub_stamp":"2023-10-30T13:25:58.407437+0000","last_deep_scrub":"85'2296","last_deep_scrub_stamp":"2023-10-30T13:25:58.407437+0000","last_clean_scrub_stamp":"2023-10-30T13:25:58.407437+0000","objects_scrubbed":647,"log_size":2296,"log_dups_size":0,"ondisk_log_size":2296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:12.972219+0000","scrub_duration":1.296777326,"objects_trimmed":0,"snaptrim_duration":0.0085509929999999998,"stat_sum":{"num_bytes":14746405,"num_objects":647,"num_object_clones":344,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":203,"num_read":167,"num_read_kb":6583,"num_write":1369,"num_write_kb":17013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'916","reported_seq":1240,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:51.295504+0000","last_change":"2023-10-30T13:25:51.295571+0000","last_active":"2023-10-30T13:25:51.295504+0000","last_peered":"2023-10-30T13:25:51.295504+0000","last_clean":"2023-10-30T13:25:51.295504+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T13:25:51.295504+0000","last_undegraded":"2023-10-30T13:25:51.295504+0000","last_fullsized":"2023-10-30T13:25:51.295504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'916","last_scrub_stamp":"2023-10-30T13:25:51.295393+0000","last_deep_scrub":"85'916","last_deep_scrub_stamp":"2023-10-30T13:25:51.295393+0000","last_clean_scrub_stamp":"2023-10-30T13:25:51.295393+0000","objects_scrubbed":46,"log_size":916,"log_dups_size":0,"ondisk_log_size":916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:13:46.299260+0000","scrub_duration":0.1008542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":472,"num_read_kb":149352,"num_write":1030,"num_write_kb":38491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":548946,"num_omap_keys":1000,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"85'2069","reported_seq":4117,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.461766+0000","last_change":"2023-10-30T13:25:53.461883+0000","last_active":"2023-10-30T13:25:53.461766+0000","last_peered":"2023-10-30T13:25:53.461766+0000","last_clean":"2023-10-30T13:25:53.461766+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T13:25:53.461766+0000","last_undegraded":"2023-10-30T13:25:53.461766+0000","last_fullsized":"2023-10-30T13:25:53.461766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2069","last_scrub_stamp":"2023-10-30T13:25:53.461661+0000","last_deep_scrub":"85'2069","last_deep_scrub_stamp":"2023-10-30T13:25:53.461661+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.461661+0000","objects_scrubbed":584,"log_size":2069,"log_dups_size":0,"ondisk_log_size":2069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:51.529626+0000","scrub_duration":1.0309153659999999,"objects_trimmed":0,"snaptrim_duration":0.0096369869999999996,"stat_sum":{"num_bytes":18113290,"num_objects":584,"num_object_clones":270,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":219,"num_read":134,"num_read_kb":20299,"num_write":1266,"num_write_kb":19768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"85'826","reported_seq":1199,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:49.233063+0000","last_change":"2023-10-30T13:25:49.233269+0000","last_active":"2023-10-30T13:25:49.233063+0000","last_peered":"2023-10-30T13:25:49.233063+0000","last_clean":"2023-10-30T13:25:49.233063+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T13:25:49.233063+0000","last_undegraded":"2023-10-30T13:25:49.233063+0000","last_fullsized":"2023-10-30T13:25:49.233063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'826","last_scrub_stamp":"2023-10-30T13:25:49.232843+0000","last_deep_scrub":"85'826","last_deep_scrub_stamp":"2023-10-30T13:25:49.232843+0000","last_clean_scrub_stamp":"2023-10-30T13:25:49.232843+0000","objects_scrubbed":49,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:29:29.495169+0000","scrub_duration":0.14346297999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777306,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":495,"num_read_kb":170685,"num_write":965,"num_write_kb":43434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":641331,"num_omap_keys":1102,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"85'2182","reported_seq":4222,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.248973+0000","last_change":"2023-10-30T13:25:53.249016+0000","last_active":"2023-10-30T13:25:53.248973+0000","last_peered":"2023-10-30T13:25:53.248973+0000","last_clean":"2023-10-30T13:25:53.248973+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T13:25:53.248973+0000","last_undegraded":"2023-10-30T13:25:53.248973+0000","last_fullsized":"2023-10-30T13:25:53.248973+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2182","last_scrub_stamp":"2023-10-30T13:25:53.248885+0000","last_deep_scrub":"85'2182","last_deep_scrub_stamp":"2023-10-30T13:25:53.248885+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.248885+0000","objects_scrubbed":615,"log_size":2182,"log_dups_size":0,"ondisk_log_size":2182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:04:59.062499+0000","scrub_duration":1.1234317469999999,"objects_trimmed":0,"snaptrim_duration":0.007081922,"stat_sum":{"num_bytes":14751960,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":228,"num_read":137,"num_read_kb":2516,"num_write":1376,"num_write_kb":21368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1177","reported_seq":1676,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:47.247326+0000","last_change":"2023-10-30T13:25:47.247437+0000","last_active":"2023-10-30T13:25:47.247326+0000","last_peered":"2023-10-30T13:25:47.247326+0000","last_clean":"2023-10-30T13:25:47.247326+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T13:25:47.247326+0000","last_undegraded":"2023-10-30T13:25:47.247326+0000","last_fullsized":"2023-10-30T13:25:47.247326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1177","last_scrub_stamp":"2023-10-30T13:25:47.247175+0000","last_deep_scrub":"85'1177","last_deep_scrub_stamp":"2023-10-30T13:25:47.247175+0000","last_clean_scrub_stamp":"2023-10-30T13:25:47.247175+0000","objects_scrubbed":52,"log_size":1177,"log_dups_size":0,"ondisk_log_size":1177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:36.256311+0000","scrub_duration":0.26537599099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":763,"num_read_kb":345628,"num_write":1535,"num_write_kb":76701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":665341,"num_omap_keys":1170,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"85'2030","reported_seq":4076,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:56.194710+0000","last_change":"2023-10-30T13:25:56.194771+0000","last_active":"2023-10-30T13:25:56.194710+0000","last_peered":"2023-10-30T13:25:56.194710+0000","last_clean":"2023-10-30T13:25:56.194710+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T13:25:56.194710+0000","last_undegraded":"2023-10-30T13:25:56.194710+0000","last_fullsized":"2023-10-30T13:25:56.194710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2030","last_scrub_stamp":"2023-10-30T13:25:56.194609+0000","last_deep_scrub":"85'2030","last_deep_scrub_stamp":"2023-10-30T13:25:56.194609+0000","last_clean_scrub_stamp":"2023-10-30T13:25:56.194609+0000","objects_scrubbed":570,"log_size":2030,"log_dups_size":0,"ondisk_log_size":2030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:38.396542+0000","scrub_duration":0.95094084300000004,"objects_trimmed":0,"snaptrim_duration":0.0060337460000000004,"stat_sum":{"num_bytes":23618154,"num_objects":570,"num_object_clones":263,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":209,"num_read":127,"num_read_kb":29654,"num_write":1265,"num_write_kb":27771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"85'1241","reported_seq":1655,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:07.560122+0000","last_change":"2023-10-30T13:26:07.560178+0000","last_active":"2023-10-30T13:26:07.560122+0000","last_peered":"2023-10-30T13:26:07.560122+0000","last_clean":"2023-10-30T13:26:07.560122+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T13:26:07.560122+0000","last_undegraded":"2023-10-30T13:26:07.560122+0000","last_fullsized":"2023-10-30T13:26:07.560122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1241","last_scrub_stamp":"2023-10-30T13:26:07.560043+0000","last_deep_scrub":"85'1241","last_deep_scrub_stamp":"2023-10-30T13:26:07.560043+0000","last_clean_scrub_stamp":"2023-10-30T13:26:07.560043+0000","objects_scrubbed":59,"log_size":1241,"log_dups_size":0,"ondisk_log_size":1241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:34.609715+0000","scrub_duration":0.163669707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":538,"num_read_kb":321476,"num_write":1330,"num_write_kb":79987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":506551,"num_omap_keys":794,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"85'2158","reported_seq":4196,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:19.353150+0000","last_change":"2023-10-30T13:26:19.353226+0000","last_active":"2023-10-30T13:26:19.353150+0000","last_peered":"2023-10-30T13:26:19.353150+0000","last_clean":"2023-10-30T13:26:19.353150+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T13:26:19.353150+0000","last_undegraded":"2023-10-30T13:26:19.353150+0000","last_fullsized":"2023-10-30T13:26:19.353150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2158","last_scrub_stamp":"2023-10-30T13:26:19.353076+0000","last_deep_scrub":"85'2158","last_deep_scrub_stamp":"2023-10-30T13:26:19.353076+0000","last_clean_scrub_stamp":"2023-10-30T13:26:19.353076+0000","objects_scrubbed":612,"log_size":2158,"log_dups_size":0,"ondisk_log_size":2158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:55:44.600734+0000","scrub_duration":0.83190967000000005,"objects_trimmed":0,"snaptrim_duration":0.0091395910000000007,"stat_sum":{"num_bytes":24394000,"num_objects":612,"num_object_clones":279,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":219,"num_read":133,"num_read_kb":28881,"num_write":1364,"num_write_kb":25892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"85'1946","reported_seq":3970,"reported_epoch":85,"state":"active+clean+scrubbing+deep","last_fresh":"2023-10-30T13:26:38.108400+0000","last_change":"2023-10-30T13:26:37.607497+0000","last_active":"2023-10-30T13:26:38.108400+0000","last_peered":"2023-10-30T13:26:38.108400+0000","last_clean":"2023-10-30T13:26:38.108400+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T13:26:38.108400+0000","last_undegraded":"2023-10-30T13:26:38.108400+0000","last_fullsized":"2023-10-30T13:26:38.108400+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-30T12:28:44.020687+0000","last_clean_scrub_stamp":"2023-10-30T12:28:44.020687+0000","objects_scrubbed":419,"log_size":1946,"log_dups_size":0,"ondisk_log_size":1946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010120377999999999,"stat_sum":{"num_bytes":19538275,"num_objects":582,"num_object_clones":275,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":200,"num_read":151,"num_read_kb":26132,"num_write":1354,"num_write_kb":27822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"85'931","reported_seq":1390,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.318568+0000","last_change":"2023-10-30T13:25:44.318568+0000","last_active":"2023-10-30T13:25:44.318568+0000","last_peered":"2023-10-30T13:25:44.318568+0000","last_clean":"2023-10-30T13:25:44.318568+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T13:25:44.318568+0000","last_undegraded":"2023-10-30T13:25:44.318568+0000","last_fullsized":"2023-10-30T13:25:44.318568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'931","last_scrub_stamp":"2023-10-30T13:25:44.318392+0000","last_deep_scrub":"85'931","last_deep_scrub_stamp":"2023-10-30T13:25:44.318392+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.318392+0000","objects_scrubbed":51,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:06.786185+0000","scrub_duration":0.105541015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":739,"num_read_kb":197443,"num_write":1077,"num_write_kb":51357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":687759,"num_omap_keys":1211,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"85'2328","reported_seq":4360,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:02.827670+0000","last_change":"2023-10-30T13:26:02.827670+0000","last_active":"2023-10-30T13:26:02.827670+0000","last_peered":"2023-10-30T13:26:02.827670+0000","last_clean":"2023-10-30T13:26:02.827670+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T13:26:02.827670+0000","last_undegraded":"2023-10-30T13:26:02.827670+0000","last_fullsized":"2023-10-30T13:26:02.827670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2328","last_scrub_stamp":"2023-10-30T13:26:02.827523+0000","last_deep_scrub":"85'2328","last_deep_scrub_stamp":"2023-10-30T13:26:02.827523+0000","last_clean_scrub_stamp":"2023-10-30T13:26:02.827523+0000","objects_scrubbed":686,"log_size":2328,"log_dups_size":0,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:54:56.339329+0000","scrub_duration":1.3553331850000001,"objects_trimmed":0,"snaptrim_duration":0.0011628750000000001,"stat_sum":{"num_bytes":22561655,"num_objects":686,"num_object_clones":314,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":247,"num_read":162,"num_read_kb":29557,"num_write":1441,"num_write_kb":28808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"85'1103","reported_seq":1655,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:55.242833+0000","last_change":"2023-10-30T13:25:55.242833+0000","last_active":"2023-10-30T13:25:55.242833+0000","last_peered":"2023-10-30T13:25:55.242833+0000","last_clean":"2023-10-30T13:25:55.242833+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T13:25:55.242833+0000","last_undegraded":"2023-10-30T13:25:55.242833+0000","last_fullsized":"2023-10-30T13:25:55.242833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1103","last_scrub_stamp":"2023-10-30T13:25:55.242662+0000","last_deep_scrub":"85'1103","last_deep_scrub_stamp":"2023-10-30T13:25:55.242662+0000","last_clean_scrub_stamp":"2023-10-30T13:25:55.242662+0000","objects_scrubbed":55,"log_size":1103,"log_dups_size":0,"ondisk_log_size":1103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:05:46.724096+0000","scrub_duration":0.11943021299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":939,"num_read_kb":433966,"num_write":1214,"num_write_kb":88022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":602554,"num_omap_keys":1058,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"85'2157","reported_seq":4173,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:23.727155+0000","last_change":"2023-10-30T13:26:23.727243+0000","last_active":"2023-10-30T13:26:23.727155+0000","last_peered":"2023-10-30T13:26:23.727155+0000","last_clean":"2023-10-30T13:26:23.727155+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T13:26:23.727155+0000","last_undegraded":"2023-10-30T13:26:23.727155+0000","last_fullsized":"2023-10-30T13:26:23.727155+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2157","last_scrub_stamp":"2023-10-30T13:26:23.727045+0000","last_deep_scrub":"85'2157","last_deep_scrub_stamp":"2023-10-30T13:26:23.727045+0000","last_clean_scrub_stamp":"2023-10-30T13:26:23.727045+0000","objects_scrubbed":626,"log_size":2157,"log_dups_size":0,"ondisk_log_size":2157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:22:59.201600+0000","scrub_duration":1.0286007049999999,"objects_trimmed":0,"snaptrim_duration":0.014632805,"stat_sum":{"num_bytes":14611265,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":226,"num_read":123,"num_read_kb":17907,"num_write":1326,"num_write_kb":16683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"85'807","reported_seq":1195,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:50.519115+0000","last_change":"2023-10-30T13:25:50.519195+0000","last_active":"2023-10-30T13:25:50.519115+0000","last_peered":"2023-10-30T13:25:50.519115+0000","last_clean":"2023-10-30T13:25:50.519115+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T13:25:50.519115+0000","last_undegraded":"2023-10-30T13:25:50.519115+0000","last_fullsized":"2023-10-30T13:25:50.519115+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'807","last_scrub_stamp":"2023-10-30T13:25:50.519025+0000","last_deep_scrub":"85'807","last_deep_scrub_stamp":"2023-10-30T13:25:50.519025+0000","last_clean_scrub_stamp":"2023-10-30T13:25:50.519025+0000","objects_scrubbed":54,"log_size":807,"log_dups_size":0,"ondisk_log_size":807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:52:13.369063+0000","scrub_duration":0.17624699299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":488,"num_read_kb":379559,"num_write":896,"num_write_kb":83880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":513377,"num_omap_keys":969,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"85'2062","reported_seq":4111,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:16.979624+0000","last_change":"2023-10-30T13:26:16.979757+0000","last_active":"2023-10-30T13:26:16.979624+0000","last_peered":"2023-10-30T13:26:16.979624+0000","last_clean":"2023-10-30T13:26:16.979624+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T13:26:16.979624+0000","last_undegraded":"2023-10-30T13:26:16.979624+0000","last_fullsized":"2023-10-30T13:26:16.979624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2062","last_scrub_stamp":"2023-10-30T13:26:16.979419+0000","last_deep_scrub":"85'2062","last_deep_scrub_stamp":"2023-10-30T13:26:16.979419+0000","last_clean_scrub_stamp":"2023-10-30T13:26:16.979419+0000","objects_scrubbed":581,"log_size":2062,"log_dups_size":0,"ondisk_log_size":2062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:26:59.407523+0000","scrub_duration":1.2670243729999999,"objects_trimmed":0,"snaptrim_duration":0.015871881000000001,"stat_sum":{"num_bytes":15563160,"num_objects":581,"num_object_clones":268,"num_object_copies":1162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":208,"num_read":133,"num_read_kb":9744,"num_write":1268,"num_write_kb":20904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"85'885","reported_seq":1253,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:51.286953+0000","last_change":"2023-10-30T13:25:51.287081+0000","last_active":"2023-10-30T13:25:51.286953+0000","last_peered":"2023-10-30T13:25:51.286953+0000","last_clean":"2023-10-30T13:25:51.286953+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T13:25:51.286953+0000","last_undegraded":"2023-10-30T13:25:51.286953+0000","last_fullsized":"2023-10-30T13:25:51.286953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'885","last_scrub_stamp":"2023-10-30T13:25:51.286817+0000","last_deep_scrub":"85'885","last_deep_scrub_stamp":"2023-10-30T13:25:51.286817+0000","last_clean_scrub_stamp":"2023-10-30T13:25:51.286817+0000","objects_scrubbed":47,"log_size":885,"log_dups_size":0,"ondisk_log_size":885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:21.615461+0000","scrub_duration":0.19579813099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33555014,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":517,"num_read_kb":175570,"num_write":978,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":553400,"num_omap_keys":904,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165214918,"num_objects":39301,"num_object_clones":18317,"num_object_copies":78602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39301,"num_whiteouts":14249,"num_read":9267,"num_read_kb":1285569,"num_write":86732,"num_write_kb":1459764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424127488,"data_stored":2332445068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138655,"ondisk_log_size":138655,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":903126807,"num_objects":1657,"num_object_clones":0,"num_object_copies":3314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":21550,"num_read_kb":9552846,"num_write":41306,"num_write_kb":2210416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":19724250,"num_omap_keys":33398,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1806409728,"data_stored":1806253614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36411,"ondisk_log_size":36411,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477438,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877564,"kb_used_data":639604,"kb_used_omap":1,"kb_used_meta":237950,"kb_avail":93494276,"statfs":{"total":96636764160,"available":95738138624,"internally_reserved":0,"allocated":654954496,"data_stored":640799640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":243661256},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,5,7],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.41799999999999998,"15min":0.436},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.63200000000000001,"5min":0.66900000000000004,"15min":0.94599999999999995},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.441,"5min":0.46500000000000002,"15min":0.47399999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":0.63900000000000001,"5min":0.75800000000000001,"15min":0.81899999999999995},"last":0.36299999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.58299999999999996,"5min":0.86099999999999999,"15min":1.0209999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.50600000000000001,"15min":0.52700000000000002},"min":{"1min":0.33000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.60599999999999998,"5min":0.76600000000000001,"15min":0.99299999999999999},"last":0.497}]},{"osd":2,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.501,"15min":0.504},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.28599999999999998},"max":{"1min":0.56399999999999995,"5min":0.85099999999999998,"15min":0.90800000000000003},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.74399999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.86699999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.75700000000000001,"15min":0.78600000000000003},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.008,"5min":1.05,"15min":1.548},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.80400000000000005,"15min":0.81100000000000005},"min":{"1min":0.56699999999999995,"5min":0.497,"15min":0.497},"max":{"1min":1.1200000000000001,"5min":1.139,"15min":1.6950000000000001},"last":0.88800000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.76700000000000002,"15min":0.80700000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.95899999999999996,"5min":1.1160000000000001,"15min":1.194},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.157,"5min":1.1659999999999999,"15min":1.603},"last":0.77100000000000002}]},{"osd":6,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85599999999999998,"15min":0.86399999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.436},"max":{"1min":1.081,"5min":1.198,"15min":1.2270000000000001},"last":0.96799999999999997},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.91800000000000004,"15min":0.92300000000000004},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0329999999999999,"5min":1.1639999999999999,"15min":1.3759999999999999},"last":1.0129999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.84099999999999997,"15min":0.84199999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55300000000000005},"max":{"1min":1.137,"5min":1.137,"15min":1.1919999999999999},"last":1.032},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.89400000000000002,"15min":0.88},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.47699999999999998},"max":{"1min":0.96599999999999997,"5min":1.1990000000000001,"15min":1.2050000000000001},"last":0.99299999999999999}]}]},{"osd":2,"up_from":16,"seq":68719477439,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552124,"kb_used_data":412724,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":93819716,"statfs":{"total":96636764160,"available":96071389184,"internally_reserved":0,"allocated":422629376,"data_stored":409295067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142735814},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:26:29 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.437,"15min":0.436},"min":{"1min":0.254,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.47799999999999998,"15min":0.47199999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.66900000000000004,"5min":1.554,"15min":1.554},"last":0.45500000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:26:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47599999999999998,"15min":0.48199999999999998},"min":{"1min":0.311,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.65500000000000003,"5min":1.071,"15min":1.071},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.52600000000000002,"15min":0.52900000000000003},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.28399999999999997},"max":{"1min":0.74099999999999999,"5min":1.198,"15min":1.198},"last":0.48299999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:26:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.314},"max":{"1min":0.80300000000000005,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.501},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.57199999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.379},"max":{"1min":0.83699999999999997,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.53200000000000003}]},{"osd":4,"last update":"Mon Oct 30 13:26:29 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.78600000000000003,"15min":0.79100000000000004},"min":{"1min":0.59199999999999997,"5min":0.48299999999999998,"15min":0.47999999999999998},"max":{"1min":1.083,"5min":1.1719999999999999,"15min":1.224},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.752,"15min":0.78500000000000003},"min":{"1min":0.54900000000000004,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0009999999999999,"5min":1.262,"15min":1.262},"last":0.66400000000000003}]},{"osd":5,"last update":"Mon Oct 30 13:26:29 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.86699999999999999,"15min":0.84699999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.014,"5min":1.494,"15min":1.494},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.83499999999999996,"15min":0.86699999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0720000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.64200000000000002}]},{"osd":6,"last update":"Mon Oct 30 13:26:29 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.81999999999999995,"15min":0.84299999999999997},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.238,"5min":1.238,"15min":1.294},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.82699999999999996,"15min":0.81699999999999995},"min":{"1min":0.54800000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0620000000000001,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.84899999999999998}]},{"osd":7,"last update":"Mon Oct 30 13:26:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84299999999999997,"15min":0.84399999999999997},"min":{"1min":0.63300000000000001,"5min":0.60699999999999998,"15min":0.51100000000000001},"max":{"1min":1.107,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.69099999999999995,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.212,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.72299999999999998}]}]},{"osd":1,"up_from":16,"seq":68719477438,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583400,"kb_used_data":431520,"kb_used_omap":1,"kb_used_meta":151870,"kb_avail":93788440,"statfs":{"total":96636764160,"available":96039362560,"internally_reserved":0,"allocated":441876480,"data_stored":431229668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":155515334},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.46500000000000002,"15min":0.44800000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.22600000000000001},"max":{"1min":0.63400000000000001,"5min":0.70699999999999996,"15min":0.80400000000000005},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.505,"5min":0.54200000000000004,"15min":0.52800000000000002},"min":{"1min":0.40899999999999997,"5min":0.31,"15min":0.253},"max":{"1min":0.72499999999999998,"5min":0.91100000000000003,"15min":1.1140000000000001},"last":0.61899999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47999999999999998,"15min":0.48999999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.68200000000000005,"5min":0.83999999999999997,"15min":5.6120000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.56499999999999995,"15min":0.55700000000000005},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.34699999999999998},"max":{"1min":0.78200000000000003,"5min":0.82199999999999995,"15min":2.831},"last":0.56499999999999995}]},{"osd":3,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47499999999999998,"15min":0.496},"min":{"1min":0.32600000000000001,"5min":0.318,"15min":0.30499999999999999},"max":{"1min":0.60799999999999998,"5min":0.80900000000000005,"15min":7.6900000000000004},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.55800000000000005,"15min":0.54800000000000004},"min":{"1min":0.39700000000000002,"5min":0.39600000000000002,"15min":0.34999999999999998},"max":{"1min":0.66700000000000004,"5min":0.91800000000000004,"15min":1.5309999999999999},"last":0.54200000000000004}]},{"osd":4,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.84599999999999997,"15min":0.81999999999999995},"min":{"1min":0.58099999999999996,"5min":0.51600000000000001,"15min":0.40400000000000003},"max":{"1min":1.0740000000000001,"5min":1.133,"15min":5.4790000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.90000000000000002,"15min":0.85999999999999999},"min":{"1min":0.72799999999999998,"5min":0.64300000000000002,"15min":0.44400000000000001},"max":{"1min":1.113,"5min":1.1619999999999999,"15min":5.5110000000000001},"last":0.878}]},{"osd":5,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.82899999999999996,"15min":0.84099999999999997},"min":{"1min":0.56299999999999994,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.1160000000000001,"5min":1.212,"15min":7.5430000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.89600000000000002,"15min":0.91500000000000004},"min":{"1min":0.73599999999999999,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.141,"5min":1.2350000000000001,"15min":7.5750000000000002},"last":0.86399999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.79100000000000004,"15min":0.79800000000000004},"min":{"1min":0.59399999999999997,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.875,"5min":1.177,"15min":5.6639999999999997},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.80700000000000005,"15min":0.86399999999999999},"min":{"1min":0.64000000000000001,"5min":0.51800000000000002,"15min":0.51600000000000001},"max":{"1min":1.0449999999999999,"5min":1.1140000000000001,"15min":13.036},"last":0.75}]},{"osd":7,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.84899999999999998,"15min":0.87},"min":{"1min":0.58399999999999996,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.0960000000000001,"5min":1.2050000000000001,"15min":3.7690000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.83899999999999997,"5min":0.88900000000000001,"15min":0.874},"min":{"1min":0.58499999999999996,"5min":0.55800000000000005,"15min":0.501},"max":{"1min":1.0109999999999999,"5min":1.1870000000000001,"15min":4.0659999999999998},"last":0.68200000000000005}]}]},{"osd":0,"up_from":16,"seq":68719477439,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":947656,"kb_used_data":695808,"kb_used_omap":1,"kb_used_meta":251838,"kb_avail":93424184,"statfs":{"total":96636764160,"available":95666364416,"internally_reserved":0,"allocated":712507392,"data_stored":696810667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":257882566},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:26:30 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39900000000000002,"15min":0.40300000000000002},"min":{"1min":0.29199999999999998,"5min":0.23899999999999999,"15min":0.22700000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.50700000000000001,"15min":0.51600000000000001},"min":{"1min":0.40600000000000003,"5min":0.36199999999999999,"15min":0.309},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.45700000000000002}]},{"osd":2,"last update":"Mon Oct 30 13:26:30 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.42699999999999999,"15min":0.433},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":0.58899999999999997,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.51700000000000002,"15min":0.51400000000000001},"min":{"1min":0.371,"5min":0.36099999999999999,"15min":0.30199999999999999},"max":{"1min":0.82799999999999996,"5min":0.878,"15min":0.878},"last":0.499}]},{"osd":3,"last update":"Mon Oct 30 13:26:30 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.46899999999999997,"15min":0.47099999999999997},"min":{"1min":0.309,"5min":0.29299999999999998,"15min":0.28299999999999997},"max":{"1min":0.59899999999999998,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.38600000000000001},"max":{"1min":0.84099999999999997,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.48199999999999998}]},{"osd":4,"last update":"Mon Oct 30 13:26:30 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.78000000000000003,"15min":0.78800000000000003},"min":{"1min":0.48699999999999999,"5min":0.46700000000000003,"15min":0.377},"max":{"1min":1.139,"5min":1.139,"15min":1.2609999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.77600000000000002,"15min":0.77800000000000002},"min":{"1min":0.55200000000000005,"5min":0.55000000000000004,"15min":0.48099999999999998},"max":{"1min":1.153,"5min":1.153,"15min":1.8420000000000001},"last":1.05}]},{"osd":5,"last update":"Mon Oct 30 13:26:30 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.80600000000000005,"15min":0.81200000000000006},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.91100000000000003,"5min":1.246,"15min":1.3340000000000001},"last":1.163},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.85499999999999998,"15min":0.84499999999999997},"min":{"1min":0.504,"5min":0.504,"15min":0.47099999999999997},"max":{"1min":1.0529999999999999,"5min":1.204,"15min":1.762},"last":0.93700000000000006}]},{"osd":6,"last update":"Mon Oct 30 13:26:30 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.80600000000000005,"15min":0.82099999999999995},"min":{"1min":0.59899999999999998,"5min":0.57599999999999996,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":1.1319999999999999,"15min":1.446},"last":0.997},{"interface":"front","average":{"1min":0.746,"5min":0.85099999999999998,"15min":0.875},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.56699999999999995},"max":{"1min":1.1060000000000001,"5min":1.1699999999999999,"15min":1.8169999999999999},"last":1.075}]},{"osd":7,"last update":"Mon Oct 30 13:26:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.85299999999999998,"15min":0.84999999999999998},"min":{"1min":0.60999999999999999,"5min":0.56299999999999994,"15min":0.55700000000000005},"max":{"1min":1.012,"5min":1.206,"15min":1.7949999999999999},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.85899999999999999,"15min":0.88900000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.121,"5min":1.121,"15min":1.274},"last":1.014}]}]},{"osd":6,"up_from":15,"seq":64424510141,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556972,"kb_used_data":409252,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93814868,"statfs":{"total":96636764160,"available":96066424832,"internally_reserved":0,"allocated":419074048,"data_stored":408471323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151255495},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.85199999999999998,"15min":0.85599999999999998},"min":{"1min":0.62,"5min":0.49399999999999999,"15min":0.34200000000000003},"max":{"1min":1.103,"5min":1.2470000000000001,"15min":1.4730000000000001},"last":0.625},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.85099999999999998,"15min":0.95099999999999996},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.2130000000000001,"5min":1.3149999999999999,"15min":1.415},"last":0.66300000000000003}]},{"osd":1,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.79300000000000004,"15min":0.79700000000000004},"min":{"1min":0.55700000000000005,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":1.262,"5min":1.262,"15min":1.399},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.86699999999999999,"15min":0.86299999999999999},"min":{"1min":0.62,"5min":0.52100000000000002,"15min":0.47699999999999998},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.80000000000000004}]},{"osd":2,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.995,"15min":0.96699999999999997},"min":{"1min":0.80300000000000005,"5min":0.68300000000000005,"15min":0.47199999999999998},"max":{"1min":1.375,"5min":1.496,"15min":1.496},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.877,"5min":0.90100000000000002,"15min":0.94199999999999995},"min":{"1min":0.60299999999999998,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.1839999999999999,"5min":1.3839999999999999,"15min":1.431},"last":0.88400000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:25:46 2023","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.88600000000000001,"15min":0.88900000000000001},"min":{"1min":0.65500000000000003,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.2589999999999999,"5min":1.262,"15min":1.4530000000000001},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.98099999999999998,"5min":0.92100000000000004,"15min":0.88900000000000001},"min":{"1min":0.77600000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.347,"5min":1.347,"15min":1.478},"last":1.0249999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:26:19 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.97699999999999998,"15min":0.93999999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0860000000000001,"15min":1.0509999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.484,"5min":1.546,"15min":1.546},"last":1.073}]},{"osd":5,"last update":"Mon Oct 30 13:26:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1499999999999999,"15min":1.123},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.583},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0580000000000001,"15min":1.091},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.149}]},{"osd":7,"last update":"Mon Oct 30 13:26:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.095,"15min":1.077},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.212,"5min":1.2210000000000001,"15min":1.2110000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"last":1.248}]}]},{"osd":4,"up_from":15,"seq":64424510144,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673568,"kb_used_data":493912,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93698272,"statfs":{"total":96636764160,"available":95947030528,"internally_reserved":0,"allocated":505765888,"data_stored":496228056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":183957959},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:26:22 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.82999999999999996,"15min":0.83699999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.83199999999999996,"15min":0.84599999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":1.121}]},{"osd":1,"last update":"Mon Oct 30 13:26:22 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.78100000000000003,"15min":0.78600000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.732,"5min":1.732,"15min":1.732},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.82599999999999996,"15min":0.83199999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":0.76000000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:26:22 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.85299999999999998,"15min":0.89400000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.97499999999999998},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.91200000000000003,"15min":0.93200000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3540000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":1.0529999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:26:22 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.97299999999999998,"15min":0.95599999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":1.1779999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.94199999999999995,"15min":0.95199999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":1.024}]},{"osd":5,"last update":"Mon Oct 30 13:26:22 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.98199999999999998,"15min":0.99199999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.766},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.998,"5min":1.089,"15min":1.0580000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.534,"5min":1.5529999999999999,"15min":1.6100000000000001},"last":1.002}]},{"osd":6,"last update":"Mon Oct 30 13:26:22 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.127,"15min":1.143},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.2609999999999999,"15min":1.1859999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.9610000000000001},"last":1.226}]},{"osd":7,"last update":"Mon Oct 30 13:26:22 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.044,"15min":1.1220000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":0.69999999999999996},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.196,"15min":1.256},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.6000000000000001,"5min":2.6000000000000001,"15min":2.6000000000000001},"last":1.085}]}]},{"osd":7,"up_from":14,"seq":60129542845,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564588,"kb_used_data":416932,"kb_used_omap":1,"kb_used_meta":147646,"kb_avail":93807252,"statfs":{"total":96636764160,"available":96058626048,"internally_reserved":0,"allocated":426938368,"data_stored":418861204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":151189963},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.1060000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":0.745},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.85099999999999998,"15min":0.86899999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.1120000000000001,"5min":1.333,"15min":1.3919999999999999},"last":0.85499999999999998}]},{"osd":1,"last update":"Mon Oct 30 13:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.76900000000000002,"15min":0.85799999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0229999999999999,"5min":1.244,"15min":1.4790000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.90700000000000003,"15min":0.83299999999999996},"min":{"1min":0.5,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0329999999999999,"5min":1.524,"15min":1.524},"last":0.93100000000000005}]},{"osd":2,"last update":"Mon Oct 30 13:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.032,"15min":1.0269999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.333,"5min":1.833,"15min":1.833},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1699999999999999,"15min":1.153},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.4079999999999999,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":1.2250000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0680000000000001,"15min":1.077},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.446,"5min":1.673,"15min":1.673},"last":1.075},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.2250000000000001,"15min":1.2150000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.526,"5min":1.8109999999999999,"15min":1.8480000000000001},"last":1.0049999999999999}]},{"osd":4,"last update":"Mon Oct 30 13:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.84499999999999997,"15min":0.84299999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.97899999999999998,"5min":1.226,"15min":1.4970000000000001},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.93999999999999995,"15min":0.94799999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.073,"5min":1.4970000000000001,"15min":1.77},"last":0.89900000000000002}]},{"osd":5,"last update":"Mon Oct 30 13:26:25 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93200000000000005,"15min":0.96799999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3560000000000001,"5min":1.3799999999999999,"15min":2},"last":0.68999999999999995},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.117,"15min":1.042},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.4850000000000001,"5min":1.7150000000000001,"15min":2.056},"last":0.96499999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.0549999999999999,"15min":1.01},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.4490000000000001,"5min":1.536,"15min":1.571},"last":1.125},{"interface":"front","average":{"1min":1.153,"5min":1.1499999999999999,"15min":1.1060000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.0469999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542844,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":824416,"kb_used_data":640600,"kb_used_omap":1,"kb_used_meta":183806,"kb_avail":93547424,"statfs":{"total":96636764160,"available":95792562176,"internally_reserved":0,"allocated":655974400,"data_stored":642425281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":188217798},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.76500000000000001,"15min":0.80500000000000005},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.94599999999999995,"5min":1.276,"15min":1.343},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.77300000000000002,"15min":0.85999999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.0840000000000001,"5min":1.262,"15min":1.627},"last":0.71599999999999997}]},{"osd":1,"last update":"Mon Oct 30 13:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.77300000000000002,"15min":0.81999999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78800000000000003,"5min":1.22,"15min":1.3220000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.85499999999999998,"15min":0.91000000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1719999999999999,"5min":1.274,"15min":1.389},"last":1.159}]},{"osd":2,"last update":"Mon Oct 30 13:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.82699999999999996,"15min":0.84699999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.0620000000000001,"5min":1.3220000000000001,"15min":1.486},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.90200000000000002,"15min":0.877},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.879},"last":0.94799999999999995}]},{"osd":3,"last update":"Mon Oct 30 13:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.97299999999999998,"15min":0.89200000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3480000000000001,"5min":1.361,"15min":2.097},"last":1.036},{"interface":"front","average":{"1min":0.83199999999999996,"5min":1.02,"15min":0.98899999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1479999999999999,"5min":1.5169999999999999,"15min":1.9650000000000001},"last":0.65900000000000003}]},{"osd":4,"last update":"Mon Oct 30 13:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.96199999999999997,"15min":0.97799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6719999999999999},"last":1.353},{"interface":"front","average":{"1min":0.95799999999999996,"5min":1.016,"15min":1.081},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.3220000000000001}]},{"osd":6,"last update":"Mon Oct 30 13:26:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.117,"15min":1.091},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.944,"5min":1.944,"15min":2.1509999999999998},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4850000000000001,"5min":1.625,"15min":1.821},"last":1.4470000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:26:24 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.177,"15min":1.129},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.875,"5min":1.875,"15min":3.3069999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.052,"5min":1.137,"15min":1.113},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.5569999999999999,"5min":1.754,"15min":2.036},"last":1.3899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309846016,"data_stored":309823069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142618624,"data_stored":142610667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117452800,"data_stored":117441250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294043648,"data_stored":294028670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":275668992,"data_stored":275628631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":300916736,"data_stored":300897937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":154218496,"data_stored":154207037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":211644416,"data_stored":211616353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401661952,"data_stored":386457412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297664512,"data_stored":287498447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304177152,"data_stored":291323631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359911424,"data_stored":346240784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":229097472,"data_stored":220069239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":353464320,"data_stored":340406790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":263856128,"data_stored":253734100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":214294528,"data_stored":206714665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:26:44.210 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2023-10-30T12:28:44.020687+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=12, tm_min=28, tm_sec=44, tm_wday=0, tm_yday=303, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=30, tm_hour=13, tm_min=25, tm_sec=37, tm_wday=0, tm_yday=303, tm_isdst=0) 2023-10-30T13:26:44.210 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-30T13:26:48.940 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:26:48.944 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:04.211 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-30T13:27:04.358 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.352+0000 7f2c761aa640 1 Processor -- start 2023-10-30T13:27:04.359 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.352+0000 7f2c761aa640 1 -- start start 2023-10-30T13:27:04.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.352+0000 7f2c761aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c700c6b80 0x7f2c700bfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.352+0000 7f2c761aa640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c700c0a50 con 0x7f2c700c6f60 2023-10-30T13:27:04.360 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.352+0000 7f2c761aa640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c7006f6e0 con 0x7f2c7013f4a0 2023-10-30T13:27:04.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.352+0000 7f2c761aa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7006f9c0 con 0x7f2c700c6b80 2023-10-30T13:27:04.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6effd640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7f2c700c6f60 0x7f2c7013ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37532/0 (socket says 172.21.15.125:37532) 2023-10-30T13:27:04.361 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6effd640 1 -- 172.21.15.125:0/4020694762 learned_addr learned my addr 172.21.15.125:0/4020694762 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:27:04.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6f7fe640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c700c6b80 0x7f2c700bfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 -- 172.21.15.125:0/4020694762 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586479278 0 0) 0x7f2c700c0a50 con 0x7f2c700c6f60 2023-10-30T13:27:04.362 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 -- 172.21.15.125:0/4020694762 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c58003530 con 0x7f2c700c6f60 2023-10-30T13:27:04.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 -- 172.21.15.125:0/4020694762 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3367146218 0 0) 0x7f2c58003530 con 0x7f2c700c6f60 2023-10-30T13:27:04.364 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 -- 172.21.15.125:0/4020694762 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c700c6b80 msgr2=0x7f2c700bfec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:04.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c700c6b80 0x7f2c700bfec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 -- 172.21.15.125:0/4020694762 >> v1:172.21.15.136:6789/0 conn(0x7f2c7013f4a0 legacy=0x7f2c701566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 -- 172.21.15.125:0/4020694762 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c7006fca0 con 0x7f2c700c6f60 2023-10-30T13:27:04.365 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 -- 172.21.15.125:0/4020694762 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7f2c64001680 con 0x7f2c700c6f60 2023-10-30T13:27:04.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 -- 172.21.15.125:0/4020694762 >> v1:172.21.15.125:6789/0 conn(0x7f2c700c6f60 legacy=0x7f2c7013ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c580039b0 0x7f2c58003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.366 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c58004360 0x7f2c58004830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c58005030 0x7f2c5800b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 -- 172.21.15.125:0/4020694762 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7006f9c0 con 0x7f2c58004360 2023-10-30T13:27:04.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 -- 172.21.15.125:0/4020694762 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5800bf00 con 0x7f2c580039b0 2023-10-30T13:27:04.367 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6e7fc640 1 -- 172.21.15.125:0/4020694762 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c5800c0c0 con 0x7f2c58005030 2023-10-30T13:27:04.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6f7fe640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c700c6b80 0x7f2c700bfec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-30T13:27:04.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6effd640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c58004360 0x7f2c58004830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6f7fe640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c580039b0 0x7f2c58003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.368 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6effd640 1 -- 172.21.15.125:0/4020694762 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c58005030 msgr2=0x7f2c5800b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:04.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6ffff640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c58005030 0x7f2c5800b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6effd640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c58005030 0x7f2c5800b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.369 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6effd640 1 -- 172.21.15.125:0/4020694762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c580039b0 msgr2=0x7f2c58003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6effd640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c580039b0 0x7f2c58003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6effd640 1 -- 172.21.15.125:0/4020694762 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c58003530 con 0x7f2c58004360 2023-10-30T13:27:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6f7fe640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c580039b0 0x7f2c58003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:27:04.370 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.356+0000 7f2c6ffff640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c58005030 0x7f2c5800b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:27:04.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.360+0000 7f2c6effd640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c58004360 0x7f2c58004830 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f2c700c7380 tx=0x7f2c6401cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd8d69a9a7f65f5c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:04.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.360+0000 7f2c6e7fc640 1 -- 172.21.15.125:0/4020694762 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c64005ae0 con 0x7f2c58004360 2023-10-30T13:27:04.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.360+0000 7f2c6e7fc640 1 -- 172.21.15.125:0/4020694762 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c64005c60 con 0x7f2c58004360 2023-10-30T13:27:04.371 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.360+0000 7f2c6e7fc640 1 -- 172.21.15.125:0/4020694762 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c640a57f0 con 0x7f2c58004360 2023-10-30T13:27:04.374 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.360+0000 7f2c761aa640 1 -- 172.21.15.125:0/4020694762 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c58004360 msgr2=0x7f2c58004830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.360+0000 7f2c761aa640 1 --2- 172.21.15.125:0/4020694762 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c58004360 0x7f2c58004830 secure :-1 s=READY pgs=2214 cs=0 l=1 rev1=1 crypto rx=0x7f2c700c7380 tx=0x7f2c6401cc00 comp rx=0 tx=0).stop 2023-10-30T13:27:04.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.360+0000 7f2c6ffff640 1 -- 172.21.15.125:0/4020694762 reap_dead start 2023-10-30T13:27:04.375 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.360+0000 7f2c761aa640 1 -- 172.21.15.125:0/4020694762 shutdown_connections 2023-10-30T13:27:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.360+0000 7f2c761aa640 1 -- 172.21.15.125:0/4020694762 >> 172.21.15.125:0/4020694762 conn(0x7f2c700c7910 msgr2=0x7f2c700c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.360+0000 7f2c761aa640 1 -- 172.21.15.125:0/4020694762 shutdown_connections 2023-10-30T13:27:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.360+0000 7f2c761aa640 1 -- 172.21.15.125:0/4020694762 wait complete. 2023-10-30T13:27:04.376 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.360+0000 7f2c761aa640 1 Processor -- start 2023-10-30T13:27:04.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.360+0000 7f2c761aa640 1 -- start start 2023-10-30T13:27:04.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c761aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c58004360 0x7f2c70138a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.377 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c761aa640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c7013eb00 0x7f2c701f6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c761aa640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c7013e5d0 0x7f2c70155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c761aa640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c700aefc0 con 0x7f2c7013eb00 2023-10-30T13:27:04.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c761aa640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c700b0330 con 0x7f2c7013e5d0 2023-10-30T13:27:04.378 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c761aa640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c700afe40 con 0x7f2c58004360 2023-10-30T13:27:04.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c6effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c7013eb00 0x7f2c701f6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c6effd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c7013eb00 0x7f2c701f6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53760/0 (socket says 172.21.15.125:53760) 2023-10-30T13:27:04.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c6effd640 1 -- 172.21.15.125:0/1669097086 learned_addr learned my addr 172.21.15.125:0/1669097086 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:04.379 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c6ffff640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c7013e5d0 0x7f2c70155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c6f7fe640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c58004360 0x7f2c70138a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c6effd640 1 -- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c58004360 msgr2=0x7f2c70138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.380 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c6effd640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c58004360 0x7f2c70138a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c6effd640 1 -- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c7013e5d0 msgr2=0x7f2c70155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c6effd640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c7013e5d0 0x7f2c70155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c6effd640 1 -- 172.21.15.125:0/1669097086 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c701f5c90 con 0x7f2c7013eb00 2023-10-30T13:27:04.381 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c6ffff640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c7013e5d0 0x7f2c70155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:27:04.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c6f7fe640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c58004360 0x7f2c70138a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:27:04.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c6effd640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c7013eb00 0x7f2c701f6120 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f2c64098510 tx=0x7f2c58005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:04.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c57fff640 1 -- 172.21.15.125:0/1669097086 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c640b4070 con 0x7f2c7013eb00 2023-10-30T13:27:04.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c57fff640 1 -- 172.21.15.125:0/1669097086 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c64001440 con 0x7f2c7013eb00 2023-10-30T13:27:04.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c57fff640 1 -- 172.21.15.125:0/1669097086 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c640a57f0 con 0x7f2c7013eb00 2023-10-30T13:27:04.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c761aa640 1 -- 172.21.15.125:0/1669097086 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c70138f60 con 0x7f2c7013eb00 2023-10-30T13:27:04.383 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c761aa640 1 -- 172.21.15.125:0/1669097086 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c7013d0e0 con 0x7f2c7013eb00 2023-10-30T13:27:04.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c761aa640 1 -- 172.21.15.125:0/1669097086 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c7013dd50 con 0x7f2c7013eb00 2023-10-30T13:27:04.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.364+0000 7f2c57fff640 1 -- 172.21.15.125:0/1669097086 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2c640a5970 con 0x7f2c7013eb00 2023-10-30T13:27:04.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.368+0000 7f2c57fff640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c5c066490 0x7f2c5c068950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.384 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.368+0000 7f2c57fff640 1 -- 172.21.15.125:0/1669097086 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f2c6411e080 con 0x7f2c7013eb00 2023-10-30T13:27:04.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.368+0000 7f2c6f7fe640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c5c066490 0x7f2c5c068950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.385 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.368+0000 7f2c6f7fe640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c5c066490 0x7f2c5c068950 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f2c7006f1b0 tx=0x7f2c60008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:04.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.372+0000 7f2c57fff640 1 -- 172.21.15.125:0/1669097086 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2c640eaab0 con 0x7f2c7013eb00 2023-10-30T13:27:04.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.516+0000 7f2c761aa640 1 -- 172.21.15.125:0/1669097086 --> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2c7004e170 con 0x7f2c5c066490 2023-10-30T13:27:04.546 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.540+0000 7f2c57fff640 1 -- 172.21.15.125:0/1669097086 <== mgr.4108 v2:172.21.15.136:6832/999226929 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+296239 (secure 0 0 0) 0x7f2c7004e170 con 0x7f2c5c066490 2023-10-30T13:27:04.549 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:27:04.551 INFO:teuthology.orchestra.run.smithi125.stderr:dumped all 2023-10-30T13:27:04.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.544+0000 7f2c761aa640 1 -- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c5c066490 msgr2=0x7f2c5c068950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.544+0000 7f2c761aa640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c5c066490 0x7f2c5c068950 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f2c7006f1b0 tx=0x7f2c60008040 comp rx=0 tx=0).stop 2023-10-30T13:27:04.552 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.544+0000 7f2c761aa640 1 -- 172.21.15.125:0/1669097086 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c7013eb00 msgr2=0x7f2c701f6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.544+0000 7f2c761aa640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c7013eb00 0x7f2c701f6120 secure :-1 s=READY pgs=2215 cs=0 l=1 rev1=1 crypto rx=0x7f2c64098510 tx=0x7f2c58005800 comp rx=0 tx=0).stop 2023-10-30T13:27:04.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.544+0000 7f2c761aa640 1 -- 172.21.15.125:0/1669097086 shutdown_connections 2023-10-30T13:27:04.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.544+0000 7f2c761aa640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2c5c066490 0x7f2c5c068950 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.553 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.544+0000 7f2c761aa640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2c7013e5d0 0x7f2c70155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.544+0000 7f2c761aa640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2c7013eb00 0x7f2c701f6120 unknown :-1 s=CLOSED pgs=2215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.544+0000 7f2c761aa640 1 --2- 172.21.15.125:0/1669097086 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2c58004360 0x7f2c70138a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.544+0000 7f2c761aa640 1 -- 172.21.15.125:0/1669097086 >> 172.21.15.125:0/1669097086 conn(0x7f2c700c7910 msgr2=0x7f2c701396f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:04.554 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.544+0000 7f2c761aa640 1 -- 172.21.15.125:0/1669097086 shutdown_connections 2023-10-30T13:27:04.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.544+0000 7f2c761aa640 1 -- 172.21.15.125:0/1669097086 wait complete. 2023-10-30T13:27:04.569 INFO:teuthology.orchestra.run.smithi125.stdout:{"pg_ready":true,"pg_map":{"version":1834,"stamp":"2023-10-30T13:27:04.300474+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2068932093,"num_objects":40960,"num_object_clones":18317,"num_object_copies":81920,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40960,"num_whiteouts":14249,"num_read":30933,"num_read_kb":10838515,"num_write":128197,"num_write_kb":3672596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":19724250,"num_omap_keys":33398,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":175348,"ondisk_log_size":175348,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":5567808,"kb_used_data":4140352,"kb_used_omap":12,"kb_used_meta":1427379,"kb_avail":749406912,"statfs":{"total":773094113280,"available":767392677888,"internally_reserved":0,"allocated":4239720448,"data_stored":4144120906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":1461636665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004273"},"pg_stats":[{"pgid":"3.3f","version":"85'2027","reported_seq":4106,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:18.022996+0000","last_change":"2023-10-30T13:26:18.023085+0000","last_active":"2023-10-30T13:26:18.022996+0000","last_peered":"2023-10-30T13:26:18.022996+0000","last_clean":"2023-10-30T13:26:18.022996+0000","last_became_active":"2023-10-30T12:28:44.957194+0000","last_became_peered":"2023-10-30T12:28:44.957194+0000","last_unstale":"2023-10-30T13:26:18.022996+0000","last_undegraded":"2023-10-30T13:26:18.022996+0000","last_fullsized":"2023-10-30T13:26:18.022996+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2027","last_scrub_stamp":"2023-10-30T13:26:18.022913+0000","last_deep_scrub":"85'2027","last_deep_scrub_stamp":"2023-10-30T13:26:18.022913+0000","last_clean_scrub_stamp":"2023-10-30T13:26:18.022913+0000","objects_scrubbed":552,"log_size":2027,"log_dups_size":0,"ondisk_log_size":2027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:12:29.468884+0000","scrub_duration":0.915286406,"objects_trimmed":0,"snaptrim_duration":0.0086120880000000004,"stat_sum":{"num_bytes":27261067,"num_objects":552,"num_object_clones":249,"num_object_copies":1104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":552,"num_whiteouts":197,"num_read":145,"num_read_kb":29501,"num_write":1298,"num_write_kb":29008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"85'2050","reported_seq":4094,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:22.709036+0000","last_change":"2023-10-30T13:26:22.709116+0000","last_active":"2023-10-30T13:26:22.709036+0000","last_peered":"2023-10-30T13:26:22.709036+0000","last_clean":"2023-10-30T13:26:22.709036+0000","last_became_active":"2023-10-30T12:28:44.978121+0000","last_became_peered":"2023-10-30T12:28:44.978121+0000","last_unstale":"2023-10-30T13:26:22.709036+0000","last_undegraded":"2023-10-30T13:26:22.709036+0000","last_fullsized":"2023-10-30T13:26:22.709036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2050","last_scrub_stamp":"2023-10-30T13:26:22.708912+0000","last_deep_scrub":"85'2050","last_deep_scrub_stamp":"2023-10-30T13:26:22.708912+0000","last_clean_scrub_stamp":"2023-10-30T13:26:22.708912+0000","objects_scrubbed":574,"log_size":2050,"log_dups_size":0,"ondisk_log_size":2050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:34:46.206144+0000","scrub_duration":0.99652591300000004,"objects_trimmed":0,"snaptrim_duration":0.015306986,"stat_sum":{"num_bytes":13736175,"num_objects":574,"num_object_clones":269,"num_object_copies":1148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":574,"num_whiteouts":223,"num_read":131,"num_read_kb":21315,"num_write":1260,"num_write_kb":15409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"85'2191","reported_seq":4236,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:29.063137+0000","last_change":"2023-10-30T13:26:29.063266+0000","last_active":"2023-10-30T13:26:29.063137+0000","last_peered":"2023-10-30T13:26:29.063137+0000","last_clean":"2023-10-30T13:26:29.063137+0000","last_became_active":"2023-10-30T12:28:44.979960+0000","last_became_peered":"2023-10-30T12:28:44.979960+0000","last_unstale":"2023-10-30T13:26:29.063137+0000","last_undegraded":"2023-10-30T13:26:29.063137+0000","last_fullsized":"2023-10-30T13:26:29.063137+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2191","last_scrub_stamp":"2023-10-30T13:26:29.062929+0000","last_deep_scrub":"85'2191","last_deep_scrub_stamp":"2023-10-30T13:26:29.062929+0000","last_clean_scrub_stamp":"2023-10-30T13:26:29.062929+0000","objects_scrubbed":628,"log_size":2191,"log_dups_size":0,"ondisk_log_size":2191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:12:09.798552+0000","scrub_duration":1.415347921,"objects_trimmed":0,"snaptrim_duration":0.013634598,"stat_sum":{"num_bytes":28390436,"num_objects":628,"num_object_clones":291,"num_object_copies":1256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":232,"num_read":152,"num_read_kb":47076,"num_write":1356,"num_write_kb":37143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3c","version":"85'2206","reported_seq":4249,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.404657+0000","last_change":"2023-10-30T13:25:48.404708+0000","last_active":"2023-10-30T13:25:48.404657+0000","last_peered":"2023-10-30T13:25:48.404657+0000","last_clean":"2023-10-30T13:25:48.404657+0000","last_became_active":"2023-10-30T12:28:44.964618+0000","last_became_peered":"2023-10-30T12:28:44.964618+0000","last_unstale":"2023-10-30T13:25:48.404657+0000","last_undegraded":"2023-10-30T13:25:48.404657+0000","last_fullsized":"2023-10-30T13:25:48.404657+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2206","last_scrub_stamp":"2023-10-30T13:25:48.404576+0000","last_deep_scrub":"85'2206","last_deep_scrub_stamp":"2023-10-30T13:25:48.404576+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.404576+0000","objects_scrubbed":629,"log_size":2206,"log_dups_size":0,"ondisk_log_size":2206,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:33:01.872332+0000","scrub_duration":1.22769368,"objects_trimmed":0,"snaptrim_duration":0.0056762749999999997,"stat_sum":{"num_bytes":22921274,"num_objects":629,"num_object_clones":290,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":224,"num_read":145,"num_read_kb":39229,"num_write":1388,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3b","version":"85'2214","reported_seq":4254,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:00.132866+0000","last_change":"2023-10-30T13:26:00.132866+0000","last_active":"2023-10-30T13:26:00.132866+0000","last_peered":"2023-10-30T13:26:00.132866+0000","last_clean":"2023-10-30T13:26:00.132866+0000","last_became_active":"2023-10-30T12:28:44.976692+0000","last_became_peered":"2023-10-30T12:28:44.976692+0000","last_unstale":"2023-10-30T13:26:00.132866+0000","last_undegraded":"2023-10-30T13:26:00.132866+0000","last_fullsized":"2023-10-30T13:26:00.132866+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2214","last_scrub_stamp":"2023-10-30T13:26:00.132634+0000","last_deep_scrub":"85'2214","last_deep_scrub_stamp":"2023-10-30T13:26:00.132634+0000","last_clean_scrub_stamp":"2023-10-30T13:26:00.132634+0000","objects_scrubbed":623,"log_size":2214,"log_dups_size":0,"ondisk_log_size":2214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:56:19.866906+0000","scrub_duration":1.0674117089999999,"objects_trimmed":0,"snaptrim_duration":0.0081298299999999993,"stat_sum":{"num_bytes":26798060,"num_objects":623,"num_object_clones":286,"num_object_copies":1246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":623,"num_whiteouts":214,"num_read":144,"num_read_kb":26713,"num_write":1434,"num_write_kb":32534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3a","version":"85'2029","reported_seq":4091,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.107819+0000","last_change":"2023-10-30T13:25:44.107819+0000","last_active":"2023-10-30T13:25:44.107819+0000","last_peered":"2023-10-30T13:25:44.107819+0000","last_clean":"2023-10-30T13:25:44.107819+0000","last_became_active":"2023-10-30T12:28:44.964748+0000","last_became_peered":"2023-10-30T12:28:44.964748+0000","last_unstale":"2023-10-30T13:25:44.107819+0000","last_undegraded":"2023-10-30T13:25:44.107819+0000","last_fullsized":"2023-10-30T13:25:44.107819+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2029","last_scrub_stamp":"2023-10-30T13:25:44.107605+0000","last_deep_scrub":"85'2029","last_deep_scrub_stamp":"2023-10-30T13:25:44.107605+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.107605+0000","objects_scrubbed":563,"log_size":2029,"log_dups_size":0,"ondisk_log_size":2029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:05:35.807432+0000","scrub_duration":0.97669900899999995,"objects_trimmed":0,"snaptrim_duration":0.0048510920000000004,"stat_sum":{"num_bytes":13539463,"num_objects":563,"num_object_clones":261,"num_object_copies":1126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":563,"num_whiteouts":205,"num_read":144,"num_read_kb":15818,"num_write":1313,"num_write_kb":14957,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.39","version":"85'2168","reported_seq":4191,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:58.341167+0000","last_change":"2023-10-30T13:25:58.341234+0000","last_active":"2023-10-30T13:25:58.341167+0000","last_peered":"2023-10-30T13:25:58.341167+0000","last_clean":"2023-10-30T13:25:58.341167+0000","last_became_active":"2023-10-30T12:28:44.968171+0000","last_became_peered":"2023-10-30T12:28:44.968171+0000","last_unstale":"2023-10-30T13:25:58.341167+0000","last_undegraded":"2023-10-30T13:25:58.341167+0000","last_fullsized":"2023-10-30T13:25:58.341167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2168","last_scrub_stamp":"2023-10-30T13:25:58.341077+0000","last_deep_scrub":"85'2168","last_deep_scrub_stamp":"2023-10-30T13:25:58.341077+0000","last_clean_scrub_stamp":"2023-10-30T13:25:58.341077+0000","objects_scrubbed":606,"log_size":2168,"log_dups_size":0,"ondisk_log_size":2168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:54:06.931072+0000","scrub_duration":1.219517739,"objects_trimmed":0,"snaptrim_duration":0.00082671399999999999,"stat_sum":{"num_bytes":7202428,"num_objects":606,"num_object_clones":284,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":219,"num_read":126,"num_read_kb":1677,"num_write":1361,"num_write_kb":10203,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.38","version":"85'2105","reported_seq":4163,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:14.120371+0000","last_change":"2023-10-30T13:26:14.120371+0000","last_active":"2023-10-30T13:26:14.120371+0000","last_peered":"2023-10-30T13:26:14.120371+0000","last_clean":"2023-10-30T13:26:14.120371+0000","last_became_active":"2023-10-30T12:28:44.972038+0000","last_became_peered":"2023-10-30T12:28:44.972038+0000","last_unstale":"2023-10-30T13:26:14.120371+0000","last_undegraded":"2023-10-30T13:26:14.120371+0000","last_fullsized":"2023-10-30T13:26:14.120371+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2105","last_scrub_stamp":"2023-10-30T13:26:14.120044+0000","last_deep_scrub":"85'2105","last_deep_scrub_stamp":"2023-10-30T13:26:14.120044+0000","last_clean_scrub_stamp":"2023-10-30T13:26:14.120044+0000","objects_scrubbed":587,"log_size":2105,"log_dups_size":0,"ondisk_log_size":2105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:11:07.710223+0000","scrub_duration":1.494910695,"objects_trimmed":0,"snaptrim_duration":0.014437559000000001,"stat_sum":{"num_bytes":23150826,"num_objects":587,"num_object_clones":269,"num_object_copies":1174,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":587,"num_whiteouts":199,"num_read":149,"num_read_kb":38846,"num_write":1362,"num_write_kb":24887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.37","version":"85'2167","reported_seq":4232,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:00.039258+0000","last_change":"2023-10-30T13:26:00.039258+0000","last_active":"2023-10-30T13:26:00.039258+0000","last_peered":"2023-10-30T13:26:00.039258+0000","last_clean":"2023-10-30T13:26:00.039258+0000","last_became_active":"2023-10-30T12:28:44.945852+0000","last_became_peered":"2023-10-30T12:28:44.945852+0000","last_unstale":"2023-10-30T13:26:00.039258+0000","last_undegraded":"2023-10-30T13:26:00.039258+0000","last_fullsized":"2023-10-30T13:26:00.039258+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2167","last_scrub_stamp":"2023-10-30T13:26:00.039070+0000","last_deep_scrub":"85'2167","last_deep_scrub_stamp":"2023-10-30T13:26:00.039070+0000","last_clean_scrub_stamp":"2023-10-30T13:26:00.039070+0000","objects_scrubbed":618,"log_size":2167,"log_dups_size":0,"ondisk_log_size":2167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:06:11.823857+0000","scrub_duration":1.5619247110000001,"objects_trimmed":0,"snaptrim_duration":0.0014185980000000001,"stat_sum":{"num_bytes":36575095,"num_objects":618,"num_object_clones":282,"num_object_copies":1236,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":618,"num_whiteouts":216,"num_read":165,"num_read_kb":36418,"num_write":1354,"num_write_kb":45869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"85'2045","reported_seq":4110,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:15.050397+0000","last_change":"2023-10-30T13:26:15.050397+0000","last_active":"2023-10-30T13:26:15.050397+0000","last_peered":"2023-10-30T13:26:15.050397+0000","last_clean":"2023-10-30T13:26:15.050397+0000","last_became_active":"2023-10-30T12:28:44.956466+0000","last_became_peered":"2023-10-30T12:28:44.956466+0000","last_unstale":"2023-10-30T13:26:15.050397+0000","last_undegraded":"2023-10-30T13:26:15.050397+0000","last_fullsized":"2023-10-30T13:26:15.050397+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2045","last_scrub_stamp":"2023-10-30T13:26:15.050260+0000","last_deep_scrub":"85'2045","last_deep_scrub_stamp":"2023-10-30T13:26:15.050260+0000","last_clean_scrub_stamp":"2023-10-30T13:26:15.050260+0000","objects_scrubbed":586,"log_size":2045,"log_dups_size":0,"ondisk_log_size":2045,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:12:47.785761+0000","scrub_duration":0.93888254999999998,"objects_trimmed":0,"snaptrim_duration":0.0091404980000000004,"stat_sum":{"num_bytes":11176343,"num_objects":586,"num_object_clones":270,"num_object_copies":1172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":586,"num_whiteouts":213,"num_read":154,"num_read_kb":10770,"num_write":1260,"num_write_kb":13564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.35","version":"85'2136","reported_seq":4177,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:06.941585+0000","last_change":"2023-10-30T13:26:06.941585+0000","last_active":"2023-10-30T13:26:06.941585+0000","last_peered":"2023-10-30T13:26:06.941585+0000","last_clean":"2023-10-30T13:26:06.941585+0000","last_became_active":"2023-10-30T12:28:44.955681+0000","last_became_peered":"2023-10-30T12:28:44.955681+0000","last_unstale":"2023-10-30T13:26:06.941585+0000","last_undegraded":"2023-10-30T13:26:06.941585+0000","last_fullsized":"2023-10-30T13:26:06.941585+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2136","last_scrub_stamp":"2023-10-30T13:26:06.941431+0000","last_deep_scrub":"85'2136","last_deep_scrub_stamp":"2023-10-30T13:26:06.941431+0000","last_clean_scrub_stamp":"2023-10-30T13:26:06.941431+0000","objects_scrubbed":625,"log_size":2136,"log_dups_size":0,"ondisk_log_size":2136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:04:47.801796+0000","scrub_duration":0.82203926400000005,"objects_trimmed":0,"snaptrim_duration":0.0075712790000000002,"stat_sum":{"num_bytes":23568118,"num_objects":625,"num_object_clones":283,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":230,"num_read":141,"num_read_kb":34199,"num_write":1299,"num_write_kb":24762,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.34","version":"85'2446","reported_seq":4405,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:14.612141+0000","last_change":"2023-10-30T13:26:14.612219+0000","last_active":"2023-10-30T13:26:14.612141+0000","last_peered":"2023-10-30T13:26:14.612141+0000","last_clean":"2023-10-30T13:26:14.612141+0000","last_became_active":"2023-10-30T12:28:44.956484+0000","last_became_peered":"2023-10-30T12:28:44.956484+0000","last_unstale":"2023-10-30T13:26:14.612141+0000","last_undegraded":"2023-10-30T13:26:14.612141+0000","last_fullsized":"2023-10-30T13:26:14.612141+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2446","last_scrub_stamp":"2023-10-30T13:26:14.612071+0000","last_deep_scrub":"85'2446","last_deep_scrub_stamp":"2023-10-30T13:26:14.612071+0000","last_clean_scrub_stamp":"2023-10-30T13:26:14.612071+0000","objects_scrubbed":721,"log_size":2446,"log_dups_size":0,"ondisk_log_size":2446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:13:35.491978+0000","scrub_duration":1.5476285650000001,"objects_trimmed":0,"snaptrim_duration":0.007366493,"stat_sum":{"num_bytes":24142540,"num_objects":721,"num_object_clones":385,"num_object_copies":1442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":231,"num_read":157,"num_read_kb":24833,"num_write":1452,"num_write_kb":33857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"85'2099","reported_seq":4170,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.041494+0000","last_change":"2023-10-30T13:25:54.041494+0000","last_active":"2023-10-30T13:25:54.041494+0000","last_peered":"2023-10-30T13:25:54.041494+0000","last_clean":"2023-10-30T13:25:54.041494+0000","last_became_active":"2023-10-30T12:28:44.962043+0000","last_became_peered":"2023-10-30T12:28:44.962043+0000","last_unstale":"2023-10-30T13:25:54.041494+0000","last_undegraded":"2023-10-30T13:25:54.041494+0000","last_fullsized":"2023-10-30T13:25:54.041494+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2099","last_scrub_stamp":"2023-10-30T13:25:54.041318+0000","last_deep_scrub":"85'2099","last_deep_scrub_stamp":"2023-10-30T13:25:54.041318+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.041318+0000","objects_scrubbed":596,"log_size":2099,"log_dups_size":0,"ondisk_log_size":2099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:54:14.898677+0000","scrub_duration":1.085239426,"objects_trimmed":0,"snaptrim_duration":0.006759035,"stat_sum":{"num_bytes":14254037,"num_objects":596,"num_object_clones":276,"num_object_copies":1192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":596,"num_whiteouts":205,"num_read":165,"num_read_kb":21722,"num_write":1326,"num_write_kb":24306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"85'2174","reported_seq":4206,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:01.905598+0000","last_change":"2023-10-30T13:26:01.905712+0000","last_active":"2023-10-30T13:26:01.905598+0000","last_peered":"2023-10-30T13:26:01.905598+0000","last_clean":"2023-10-30T13:26:01.905598+0000","last_became_active":"2023-10-30T12:28:44.967040+0000","last_became_peered":"2023-10-30T12:28:44.967040+0000","last_unstale":"2023-10-30T13:26:01.905598+0000","last_undegraded":"2023-10-30T13:26:01.905598+0000","last_fullsized":"2023-10-30T13:26:01.905598+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2174","last_scrub_stamp":"2023-10-30T13:26:01.905493+0000","last_deep_scrub":"85'2174","last_deep_scrub_stamp":"2023-10-30T13:26:01.905493+0000","last_clean_scrub_stamp":"2023-10-30T13:26:01.905493+0000","objects_scrubbed":606,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:14:46.370347+0000","scrub_duration":0.97684040100000002,"objects_trimmed":0,"snaptrim_duration":0.0057386779999999997,"stat_sum":{"num_bytes":10370740,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":213,"num_read":131,"num_read_kb":5845,"num_write":1381,"num_write_kb":12595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.31","version":"85'2261","reported_seq":4276,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:03.616856+0000","last_change":"2023-10-30T13:26:03.616927+0000","last_active":"2023-10-30T13:26:03.616856+0000","last_peered":"2023-10-30T13:26:03.616856+0000","last_clean":"2023-10-30T13:26:03.616856+0000","last_became_active":"2023-10-30T12:28:44.976315+0000","last_became_peered":"2023-10-30T12:28:44.976315+0000","last_unstale":"2023-10-30T13:26:03.616856+0000","last_undegraded":"2023-10-30T13:26:03.616856+0000","last_fullsized":"2023-10-30T13:26:03.616856+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2261","last_scrub_stamp":"2023-10-30T13:26:03.616747+0000","last_deep_scrub":"85'2261","last_deep_scrub_stamp":"2023-10-30T13:26:03.616747+0000","last_clean_scrub_stamp":"2023-10-30T13:26:03.616747+0000","objects_scrubbed":637,"log_size":2261,"log_dups_size":0,"ondisk_log_size":2261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:14:45.471660+0000","scrub_duration":1.159355685,"objects_trimmed":0,"snaptrim_duration":0.013549705,"stat_sum":{"num_bytes":19248519,"num_objects":637,"num_object_clones":295,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":221,"num_read":128,"num_read_kb":21688,"num_write":1447,"num_write_kb":21442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"85'2193","reported_seq":4229,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:46.252816+0000","last_change":"2023-10-30T13:25:46.252862+0000","last_active":"2023-10-30T13:25:46.252816+0000","last_peered":"2023-10-30T13:25:46.252816+0000","last_clean":"2023-10-30T13:25:46.252816+0000","last_became_active":"2023-10-30T12:28:44.948620+0000","last_became_peered":"2023-10-30T12:28:44.948620+0000","last_unstale":"2023-10-30T13:25:46.252816+0000","last_undegraded":"2023-10-30T13:25:46.252816+0000","last_fullsized":"2023-10-30T13:25:46.252816+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2193","last_scrub_stamp":"2023-10-30T13:25:46.252731+0000","last_deep_scrub":"85'2193","last_deep_scrub_stamp":"2023-10-30T13:25:46.252731+0000","last_clean_scrub_stamp":"2023-10-30T13:25:46.252731+0000","objects_scrubbed":619,"log_size":2193,"log_dups_size":0,"ondisk_log_size":2193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:23:51.513414+0000","scrub_duration":0.82934357400000003,"objects_trimmed":0,"snaptrim_duration":0.0075608309999999996,"stat_sum":{"num_bytes":14269019,"num_objects":619,"num_object_clones":290,"num_object_copies":1238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":233,"num_read":143,"num_read_kb":25965,"num_write":1377,"num_write_kb":20442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2f","version":"85'2247","reported_seq":4267,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:45.441798+0000","last_change":"2023-10-30T13:25:45.441869+0000","last_active":"2023-10-30T13:25:45.441798+0000","last_peered":"2023-10-30T13:25:45.441798+0000","last_clean":"2023-10-30T13:25:45.441798+0000","last_became_active":"2023-10-30T12:28:44.960055+0000","last_became_peered":"2023-10-30T12:28:44.960055+0000","last_unstale":"2023-10-30T13:25:45.441798+0000","last_undegraded":"2023-10-30T13:25:45.441798+0000","last_fullsized":"2023-10-30T13:25:45.441798+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2247","last_scrub_stamp":"2023-10-30T13:25:45.441689+0000","last_deep_scrub":"85'2247","last_deep_scrub_stamp":"2023-10-30T13:25:45.441689+0000","last_clean_scrub_stamp":"2023-10-30T13:25:45.441689+0000","objects_scrubbed":642,"log_size":2247,"log_dups_size":0,"ondisk_log_size":2247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:54:27.498757+0000","scrub_duration":0.97553288699999996,"objects_trimmed":0,"snaptrim_duration":0.0099357109999999998,"stat_sum":{"num_bytes":14793776,"num_objects":642,"num_object_clones":304,"num_object_copies":1284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":642,"num_whiteouts":242,"num_read":140,"num_read_kb":30098,"num_write":1386,"num_write_kb":20714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2e","version":"85'2006","reported_seq":4061,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.398039+0000","last_change":"2023-10-30T13:25:57.398144+0000","last_active":"2023-10-30T13:25:57.398039+0000","last_peered":"2023-10-30T13:25:57.398039+0000","last_clean":"2023-10-30T13:25:57.398039+0000","last_became_active":"2023-10-30T12:28:44.968052+0000","last_became_peered":"2023-10-30T12:28:44.968052+0000","last_unstale":"2023-10-30T13:25:57.398039+0000","last_undegraded":"2023-10-30T13:25:57.398039+0000","last_fullsized":"2023-10-30T13:25:57.398039+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2006","last_scrub_stamp":"2023-10-30T13:25:57.397929+0000","last_deep_scrub":"85'2006","last_deep_scrub_stamp":"2023-10-30T13:25:57.397929+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.397929+0000","objects_scrubbed":562,"log_size":2006,"log_dups_size":0,"ondisk_log_size":2006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:28.919675+0000","scrub_duration":1.1960647630000001,"objects_trimmed":0,"snaptrim_duration":0.0065893760000000001,"stat_sum":{"num_bytes":15508852,"num_objects":562,"num_object_clones":263,"num_object_copies":1124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":216,"num_read":137,"num_read_kb":10064,"num_write":1244,"num_write_kb":17198,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2d","version":"85'2190","reported_seq":4325,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:24.796847+0000","last_change":"2023-10-30T13:26:24.796903+0000","last_active":"2023-10-30T13:26:24.796847+0000","last_peered":"2023-10-30T13:26:24.796847+0000","last_clean":"2023-10-30T13:26:24.796847+0000","last_became_active":"2023-10-30T12:28:44.966518+0000","last_became_peered":"2023-10-30T12:28:44.966518+0000","last_unstale":"2023-10-30T13:26:24.796847+0000","last_undegraded":"2023-10-30T13:26:24.796847+0000","last_fullsized":"2023-10-30T13:26:24.796847+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2190","last_scrub_stamp":"2023-10-30T13:26:24.796760+0000","last_deep_scrub":"85'2190","last_deep_scrub_stamp":"2023-10-30T13:26:24.796760+0000","last_clean_scrub_stamp":"2023-10-30T13:26:24.796760+0000","objects_scrubbed":614,"log_size":2190,"log_dups_size":0,"ondisk_log_size":2190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:48:13.965904+0000","scrub_duration":1.233998626,"objects_trimmed":0,"snaptrim_duration":0.010472766,"stat_sum":{"num_bytes":15695810,"num_objects":614,"num_object_clones":281,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":221,"num_read":233,"num_read_kb":17180,"num_write":1382,"num_write_kb":23481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2c","version":"85'2183","reported_seq":4220,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:10.971723+0000","last_change":"2023-10-30T13:26:10.971791+0000","last_active":"2023-10-30T13:26:10.971723+0000","last_peered":"2023-10-30T13:26:10.971723+0000","last_clean":"2023-10-30T13:26:10.971723+0000","last_became_active":"2023-10-30T12:28:44.954449+0000","last_became_peered":"2023-10-30T12:28:44.954449+0000","last_unstale":"2023-10-30T13:26:10.971723+0000","last_undegraded":"2023-10-30T13:26:10.971723+0000","last_fullsized":"2023-10-30T13:26:10.971723+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2183","last_scrub_stamp":"2023-10-30T13:26:10.971644+0000","last_deep_scrub":"85'2183","last_deep_scrub_stamp":"2023-10-30T13:26:10.971644+0000","last_clean_scrub_stamp":"2023-10-30T13:26:10.971644+0000","objects_scrubbed":614,"log_size":2183,"log_dups_size":0,"ondisk_log_size":2183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:45:19.149519+0000","scrub_duration":0.86719633299999999,"objects_trimmed":0,"snaptrim_duration":0.0075455599999999998,"stat_sum":{"num_bytes":5586810,"num_objects":614,"num_object_clones":286,"num_object_copies":1228,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":614,"num_whiteouts":230,"num_read":140,"num_read_kb":5846,"num_write":1359,"num_write_kb":15885,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2b","version":"85'2129","reported_seq":4280,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.179234+0000","last_change":"2023-10-30T13:25:54.179340+0000","last_active":"2023-10-30T13:25:54.179234+0000","last_peered":"2023-10-30T13:25:54.179234+0000","last_clean":"2023-10-30T13:25:54.179234+0000","last_became_active":"2023-10-30T12:28:44.945219+0000","last_became_peered":"2023-10-30T12:28:44.945219+0000","last_unstale":"2023-10-30T13:25:54.179234+0000","last_undegraded":"2023-10-30T13:25:54.179234+0000","last_fullsized":"2023-10-30T13:25:54.179234+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2129","last_scrub_stamp":"2023-10-30T13:25:54.178827+0000","last_deep_scrub":"85'2129","last_deep_scrub_stamp":"2023-10-30T13:25:54.178827+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.178827+0000","objects_scrubbed":606,"log_size":2129,"log_dups_size":0,"ondisk_log_size":2129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:53:18.992880+0000","scrub_duration":1.0904528849999999,"objects_trimmed":0,"snaptrim_duration":0.0076434509999999999,"stat_sum":{"num_bytes":23023646,"num_objects":606,"num_object_clones":281,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":222,"num_read":249,"num_read_kb":26208,"num_write":1319,"num_write_kb":28338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2a","version":"85'2324","reported_seq":4342,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:04.661649+0000","last_change":"2023-10-30T13:26:04.661717+0000","last_active":"2023-10-30T13:26:04.661649+0000","last_peered":"2023-10-30T13:26:04.661649+0000","last_clean":"2023-10-30T13:26:04.661649+0000","last_became_active":"2023-10-30T12:28:44.953808+0000","last_became_peered":"2023-10-30T12:28:44.953808+0000","last_unstale":"2023-10-30T13:26:04.661649+0000","last_undegraded":"2023-10-30T13:26:04.661649+0000","last_fullsized":"2023-10-30T13:26:04.661649+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2324","last_scrub_stamp":"2023-10-30T13:26:04.661572+0000","last_deep_scrub":"85'2324","last_deep_scrub_stamp":"2023-10-30T13:26:04.661572+0000","last_clean_scrub_stamp":"2023-10-30T13:26:04.661572+0000","objects_scrubbed":653,"log_size":2324,"log_dups_size":0,"ondisk_log_size":2324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:26:17.690671+0000","scrub_duration":1.171930887,"objects_trimmed":0,"snaptrim_duration":0.0016105379999999999,"stat_sum":{"num_bytes":11166034,"num_objects":653,"num_object_clones":306,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":233,"num_read":139,"num_read_kb":9627,"num_write":1472,"num_write_kb":18195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.29","version":"85'2102","reported_seq":4154,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:11.723496+0000","last_change":"2023-10-30T13:26:11.723576+0000","last_active":"2023-10-30T13:26:11.723496+0000","last_peered":"2023-10-30T13:26:11.723496+0000","last_clean":"2023-10-30T13:26:11.723496+0000","last_became_active":"2023-10-30T12:28:44.981911+0000","last_became_peered":"2023-10-30T12:28:44.981911+0000","last_unstale":"2023-10-30T13:26:11.723496+0000","last_undegraded":"2023-10-30T13:26:11.723496+0000","last_fullsized":"2023-10-30T13:26:11.723496+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2102","last_scrub_stamp":"2023-10-30T13:26:11.723400+0000","last_deep_scrub":"85'2102","last_deep_scrub_stamp":"2023-10-30T13:26:11.723400+0000","last_clean_scrub_stamp":"2023-10-30T13:26:11.723400+0000","objects_scrubbed":582,"log_size":2102,"log_dups_size":0,"ondisk_log_size":2102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:52:07.437907+0000","scrub_duration":1.1504174899999999,"objects_trimmed":0,"snaptrim_duration":0.014064241,"stat_sum":{"num_bytes":10716893,"num_objects":582,"num_object_clones":264,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":203,"num_read":135,"num_read_kb":13340,"num_write":1353,"num_write_kb":16108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.28","version":"85'2148","reported_seq":4184,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.477345+0000","last_change":"2023-10-30T13:25:44.477389+0000","last_active":"2023-10-30T13:25:44.477345+0000","last_peered":"2023-10-30T13:25:44.477345+0000","last_clean":"2023-10-30T13:25:44.477345+0000","last_became_active":"2023-10-30T12:28:44.980727+0000","last_became_peered":"2023-10-30T12:28:44.980727+0000","last_unstale":"2023-10-30T13:25:44.477345+0000","last_undegraded":"2023-10-30T13:25:44.477345+0000","last_fullsized":"2023-10-30T13:25:44.477345+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2148","last_scrub_stamp":"2023-10-30T13:25:44.477273+0000","last_deep_scrub":"85'2148","last_deep_scrub_stamp":"2023-10-30T13:25:44.477273+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.477273+0000","objects_scrubbed":597,"log_size":2148,"log_dups_size":0,"ondisk_log_size":2148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:36:41.538304+0000","scrub_duration":0.99138869699999999,"objects_trimmed":0,"snaptrim_duration":0.0088476170000000003,"stat_sum":{"num_bytes":10233195,"num_objects":597,"num_object_clones":275,"num_object_copies":1194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":597,"num_whiteouts":203,"num_read":127,"num_read_kb":1819,"num_write":1383,"num_write_kb":16400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.27","version":"85'2047","reported_seq":4094,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:31.570381+0000","last_change":"2023-10-30T13:26:31.570447+0000","last_active":"2023-10-30T13:26:31.570381+0000","last_peered":"2023-10-30T13:26:31.570381+0000","last_clean":"2023-10-30T13:26:31.570381+0000","last_became_active":"2023-10-30T12:28:44.973780+0000","last_became_peered":"2023-10-30T12:28:44.973780+0000","last_unstale":"2023-10-30T13:26:31.570381+0000","last_undegraded":"2023-10-30T13:26:31.570381+0000","last_fullsized":"2023-10-30T13:26:31.570381+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2047","last_scrub_stamp":"2023-10-30T13:26:31.570302+0000","last_deep_scrub":"85'2047","last_deep_scrub_stamp":"2023-10-30T13:26:31.570302+0000","last_clean_scrub_stamp":"2023-10-30T13:26:31.570302+0000","objects_scrubbed":582,"log_size":2047,"log_dups_size":0,"ondisk_log_size":2047,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:09:52.048922+0000","scrub_duration":0.97138433800000001,"objects_trimmed":0,"snaptrim_duration":0.014279337,"stat_sum":{"num_bytes":22085088,"num_objects":582,"num_object_clones":270,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":216,"num_read":135,"num_read_kb":28666,"num_write":1258,"num_write_kb":25992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.26","version":"85'2207","reported_seq":4233,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:02.278563+0000","last_change":"2023-10-30T13:26:02.278655+0000","last_active":"2023-10-30T13:26:02.278563+0000","last_peered":"2023-10-30T13:26:02.278563+0000","last_clean":"2023-10-30T13:26:02.278563+0000","last_became_active":"2023-10-30T12:28:44.949859+0000","last_became_peered":"2023-10-30T12:28:44.949859+0000","last_unstale":"2023-10-30T13:26:02.278563+0000","last_undegraded":"2023-10-30T13:26:02.278563+0000","last_fullsized":"2023-10-30T13:26:02.278563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2207","last_scrub_stamp":"2023-10-30T13:26:02.278473+0000","last_deep_scrub":"85'2207","last_deep_scrub_stamp":"2023-10-30T13:26:02.278473+0000","last_clean_scrub_stamp":"2023-10-30T13:26:02.278473+0000","objects_scrubbed":598,"log_size":2207,"log_dups_size":0,"ondisk_log_size":2207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:21:31.276370+0000","scrub_duration":1.192745612,"objects_trimmed":0,"snaptrim_duration":0.004006122,"stat_sum":{"num_bytes":15408826,"num_objects":598,"num_object_clones":277,"num_object_copies":1196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":598,"num_whiteouts":213,"num_read":119,"num_read_kb":9406,"num_write":1408,"num_write_kb":18485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.25","version":"85'2174","reported_seq":4213,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.084913+0000","last_change":"2023-10-30T13:25:53.084979+0000","last_active":"2023-10-30T13:25:53.084913+0000","last_peered":"2023-10-30T13:25:53.084913+0000","last_clean":"2023-10-30T13:25:53.084913+0000","last_became_active":"2023-10-30T12:28:44.959178+0000","last_became_peered":"2023-10-30T12:28:44.959178+0000","last_unstale":"2023-10-30T13:25:53.084913+0000","last_undegraded":"2023-10-30T13:25:53.084913+0000","last_fullsized":"2023-10-30T13:25:53.084913+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2174","last_scrub_stamp":"2023-10-30T13:25:53.084846+0000","last_deep_scrub":"85'2174","last_deep_scrub_stamp":"2023-10-30T13:25:53.084846+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.084846+0000","objects_scrubbed":625,"log_size":2174,"log_dups_size":0,"ondisk_log_size":2174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:19:02.345358+0000","scrub_duration":1.1204613670000001,"objects_trimmed":0,"snaptrim_duration":0.0059072409999999997,"stat_sum":{"num_bytes":25755688,"num_objects":625,"num_object_clones":291,"num_object_copies":1250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":625,"num_whiteouts":240,"num_read":144,"num_read_kb":24326,"num_write":1332,"num_write_kb":28963,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.24","version":"85'2102","reported_seq":4117,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:06.032785+0000","last_change":"2023-10-30T13:26:06.032856+0000","last_active":"2023-10-30T13:26:06.032785+0000","last_peered":"2023-10-30T13:26:06.032785+0000","last_clean":"2023-10-30T13:26:06.032785+0000","last_became_active":"2023-10-30T12:28:44.971988+0000","last_became_peered":"2023-10-30T12:28:44.971988+0000","last_unstale":"2023-10-30T13:26:06.032785+0000","last_undegraded":"2023-10-30T13:26:06.032785+0000","last_fullsized":"2023-10-30T13:26:06.032785+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2102","last_scrub_stamp":"2023-10-30T13:26:06.032657+0000","last_deep_scrub":"85'2102","last_deep_scrub_stamp":"2023-10-30T13:26:06.032657+0000","last_clean_scrub_stamp":"2023-10-30T13:26:06.032657+0000","objects_scrubbed":617,"log_size":2102,"log_dups_size":0,"ondisk_log_size":2102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:39:44.800470+0000","scrub_duration":1.0262600609999999,"objects_trimmed":0,"snaptrim_duration":0.0062401690000000003,"stat_sum":{"num_bytes":11021781,"num_objects":617,"num_object_clones":286,"num_object_copies":1234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":617,"num_whiteouts":226,"num_read":116,"num_read_kb":10954,"num_write":1301,"num_write_kb":16114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.23","version":"85'2273","reported_seq":4279,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:55.521292+0000","last_change":"2023-10-30T13:25:55.521358+0000","last_active":"2023-10-30T13:25:55.521292+0000","last_peered":"2023-10-30T13:25:55.521292+0000","last_clean":"2023-10-30T13:25:55.521292+0000","last_became_active":"2023-10-30T12:28:44.971963+0000","last_became_peered":"2023-10-30T12:28:44.971963+0000","last_unstale":"2023-10-30T13:25:55.521292+0000","last_undegraded":"2023-10-30T13:25:55.521292+0000","last_fullsized":"2023-10-30T13:25:55.521292+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2273","last_scrub_stamp":"2023-10-30T13:25:55.521205+0000","last_deep_scrub":"85'2273","last_deep_scrub_stamp":"2023-10-30T13:25:55.521205+0000","last_clean_scrub_stamp":"2023-10-30T13:25:55.521205+0000","objects_scrubbed":653,"log_size":2273,"log_dups_size":0,"ondisk_log_size":2273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:44:08.018678+0000","scrub_duration":1.1272668750000001,"objects_trimmed":0,"snaptrim_duration":0.014933905000000001,"stat_sum":{"num_bytes":11161783,"num_objects":653,"num_object_clones":309,"num_object_copies":1306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":653,"num_whiteouts":240,"num_read":130,"num_read_kb":12352,"num_write":1416,"num_write_kb":13402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.22","version":"85'2025","reported_seq":4099,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:31.763855+0000","last_change":"2023-10-30T13:26:31.763906+0000","last_active":"2023-10-30T13:26:31.763855+0000","last_peered":"2023-10-30T13:26:31.763855+0000","last_clean":"2023-10-30T13:26:31.763855+0000","last_became_active":"2023-10-30T12:28:44.980545+0000","last_became_peered":"2023-10-30T12:28:44.980545+0000","last_unstale":"2023-10-30T13:26:31.763855+0000","last_undegraded":"2023-10-30T13:26:31.763855+0000","last_fullsized":"2023-10-30T13:26:31.763855+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2025","last_scrub_stamp":"2023-10-30T13:26:31.763774+0000","last_deep_scrub":"85'2025","last_deep_scrub_stamp":"2023-10-30T13:26:31.763774+0000","last_clean_scrub_stamp":"2023-10-30T13:26:31.763774+0000","objects_scrubbed":549,"log_size":2025,"log_dups_size":0,"ondisk_log_size":2025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:58.078390+0000","scrub_duration":1.0885005139999999,"objects_trimmed":0,"snaptrim_duration":0.0090148399999999997,"stat_sum":{"num_bytes":11081748,"num_objects":549,"num_object_clones":251,"num_object_copies":1098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":549,"num_whiteouts":201,"num_read":142,"num_read_kb":9474,"num_write":1288,"num_write_kb":13652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.21","version":"85'2406","reported_seq":4433,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.442339+0000","last_change":"2023-10-30T13:25:54.442416+0000","last_active":"2023-10-30T13:25:54.442339+0000","last_peered":"2023-10-30T13:25:54.442339+0000","last_clean":"2023-10-30T13:25:54.442339+0000","last_became_active":"2023-10-30T12:28:44.964222+0000","last_became_peered":"2023-10-30T12:28:44.964222+0000","last_unstale":"2023-10-30T13:25:54.442339+0000","last_undegraded":"2023-10-30T13:25:54.442339+0000","last_fullsized":"2023-10-30T13:25:54.442339+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2406","last_scrub_stamp":"2023-10-30T13:25:54.442247+0000","last_deep_scrub":"85'2406","last_deep_scrub_stamp":"2023-10-30T13:25:54.442247+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.442247+0000","objects_scrubbed":693,"log_size":2406,"log_dups_size":0,"ondisk_log_size":2406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:12.648322+0000","scrub_duration":1.2677238399999999,"objects_trimmed":0,"snaptrim_duration":0.0066657749999999997,"stat_sum":{"num_bytes":20125251,"num_objects":693,"num_object_clones":330,"num_object_copies":1386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":693,"num_whiteouts":267,"num_read":168,"num_read_kb":25254,"num_write":1457,"num_write_kb":26871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"85'2152","reported_seq":4191,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:45.218457+0000","last_change":"2023-10-30T13:25:45.218569+0000","last_active":"2023-10-30T13:25:45.218457+0000","last_peered":"2023-10-30T13:25:45.218457+0000","last_clean":"2023-10-30T13:25:45.218457+0000","last_became_active":"2023-10-30T12:28:44.973105+0000","last_became_peered":"2023-10-30T12:28:44.973105+0000","last_unstale":"2023-10-30T13:25:45.218457+0000","last_undegraded":"2023-10-30T13:25:45.218457+0000","last_fullsized":"2023-10-30T13:25:45.218457+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2152","last_scrub_stamp":"2023-10-30T13:25:45.218360+0000","last_deep_scrub":"85'2152","last_deep_scrub_stamp":"2023-10-30T13:25:45.218360+0000","last_clean_scrub_stamp":"2023-10-30T13:25:45.218360+0000","objects_scrubbed":604,"log_size":2152,"log_dups_size":0,"ondisk_log_size":2152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:53:35.163970+0000","scrub_duration":1.1092469380000001,"objects_trimmed":0,"snaptrim_duration":0.005419129,"stat_sum":{"num_bytes":18031744,"num_objects":604,"num_object_clones":282,"num_object_copies":1208,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":228,"num_read":136,"num_read_kb":25964,"num_write":1344,"num_write_kb":19625,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1a","version":"85'2155","reported_seq":4202,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:00.594900+0000","last_change":"2023-10-30T13:26:00.594900+0000","last_active":"2023-10-30T13:26:00.594900+0000","last_peered":"2023-10-30T13:26:00.594900+0000","last_clean":"2023-10-30T13:26:00.594900+0000","last_became_active":"2023-10-30T12:28:44.971463+0000","last_became_peered":"2023-10-30T12:28:44.971463+0000","last_unstale":"2023-10-30T13:26:00.594900+0000","last_undegraded":"2023-10-30T13:26:00.594900+0000","last_fullsized":"2023-10-30T13:26:00.594900+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2155","last_scrub_stamp":"2023-10-30T13:26:00.594690+0000","last_deep_scrub":"85'2155","last_deep_scrub_stamp":"2023-10-30T13:26:00.594690+0000","last_clean_scrub_stamp":"2023-10-30T13:26:00.594690+0000","objects_scrubbed":607,"log_size":2155,"log_dups_size":0,"ondisk_log_size":2155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:43.537640+0000","scrub_duration":1.1209885449999999,"objects_trimmed":0,"snaptrim_duration":0.0013742470000000001,"stat_sum":{"num_bytes":22381695,"num_objects":607,"num_object_clones":275,"num_object_copies":1214,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":607,"num_whiteouts":227,"num_read":141,"num_read_kb":33421,"num_write":1348,"num_write_kb":23422,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1b","version":"85'1042","reported_seq":1466,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:42.402455+0000","last_change":"2023-10-30T13:25:42.402455+0000","last_active":"2023-10-30T13:25:42.402455+0000","last_peered":"2023-10-30T13:25:42.402455+0000","last_clean":"2023-10-30T13:25:42.402455+0000","last_became_active":"2023-10-30T12:28:44.070861+0000","last_became_peered":"2023-10-30T12:28:44.070861+0000","last_unstale":"2023-10-30T13:25:42.402455+0000","last_undegraded":"2023-10-30T13:25:42.402455+0000","last_fullsized":"2023-10-30T13:25:42.402455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1042","last_scrub_stamp":"2023-10-30T13:25:42.402195+0000","last_deep_scrub":"85'1042","last_deep_scrub_stamp":"2023-10-30T13:25:42.402195+0000","last_clean_scrub_stamp":"2023-10-30T13:25:42.402195+0000","objects_scrubbed":51,"log_size":1042,"log_dups_size":0,"ondisk_log_size":1042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:42:20.707501+0000","scrub_duration":0.156286863,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388690,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":570,"num_read_kb":319827,"num_write":1218,"num_write_kb":80521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":794018,"num_omap_keys":1312,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"85'2177","reported_seq":4221,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:01.438545+0000","last_change":"2023-10-30T13:26:01.438545+0000","last_active":"2023-10-30T13:26:01.438545+0000","last_peered":"2023-10-30T13:26:01.438545+0000","last_clean":"2023-10-30T13:26:01.438545+0000","last_became_active":"2023-10-30T12:28:44.957788+0000","last_became_peered":"2023-10-30T12:28:44.957788+0000","last_unstale":"2023-10-30T13:26:01.438545+0000","last_undegraded":"2023-10-30T13:26:01.438545+0000","last_fullsized":"2023-10-30T13:26:01.438545+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2177","last_scrub_stamp":"2023-10-30T13:26:01.438339+0000","last_deep_scrub":"85'2177","last_deep_scrub_stamp":"2023-10-30T13:26:01.438339+0000","last_clean_scrub_stamp":"2023-10-30T13:26:01.438339+0000","objects_scrubbed":616,"log_size":2177,"log_dups_size":0,"ondisk_log_size":2177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:10:25.473052+0000","scrub_duration":1.2567298680000001,"objects_trimmed":0,"snaptrim_duration":0.0086018069999999995,"stat_sum":{"num_bytes":19940185,"num_objects":616,"num_object_clones":283,"num_object_copies":1232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":616,"num_whiteouts":213,"num_read":145,"num_read_kb":23669,"num_write":1398,"num_write_kb":21967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"85'1837","reported_seq":2428,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:15.510377+0000","last_change":"2023-10-30T13:26:15.510377+0000","last_active":"2023-10-30T13:26:15.510377+0000","last_peered":"2023-10-30T13:26:15.510377+0000","last_clean":"2023-10-30T13:26:15.510377+0000","last_became_active":"2023-10-30T12:28:44.060107+0000","last_became_peered":"2023-10-30T12:28:44.060107+0000","last_unstale":"2023-10-30T13:26:15.510377+0000","last_undegraded":"2023-10-30T13:26:15.510377+0000","last_fullsized":"2023-10-30T13:26:15.510377+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1837","last_scrub_stamp":"2023-10-30T13:26:15.510196+0000","last_deep_scrub":"85'1837","last_deep_scrub_stamp":"2023-10-30T13:26:15.510196+0000","last_clean_scrub_stamp":"2023-10-30T13:26:15.510196+0000","objects_scrubbed":42,"log_size":1837,"log_dups_size":0,"ondisk_log_size":1837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:16:27.071396+0000","scrub_duration":0.084593108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":17028825,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":761,"num_read_kb":396040,"num_write":1931,"num_write_kb":84639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":554115,"num_omap_keys":1012,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"85'1626","reported_seq":2456,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:46.170305+0000","last_change":"2023-10-30T13:25:46.170305+0000","last_active":"2023-10-30T13:25:46.170305+0000","last_peered":"2023-10-30T13:25:46.170305+0000","last_clean":"2023-10-30T13:25:46.170305+0000","last_became_active":"2023-10-30T12:28:44.059366+0000","last_became_peered":"2023-10-30T12:28:44.059366+0000","last_unstale":"2023-10-30T13:25:46.170305+0000","last_undegraded":"2023-10-30T13:25:46.170305+0000","last_fullsized":"2023-10-30T13:25:46.170305+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1626","last_scrub_stamp":"2023-10-30T13:25:46.170040+0000","last_deep_scrub":"85'1626","last_deep_scrub_stamp":"2023-10-30T13:25:46.170040+0000","last_clean_scrub_stamp":"2023-10-30T13:25:46.170040+0000","objects_scrubbed":57,"log_size":1626,"log_dups_size":0,"ondisk_log_size":1626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:30:38.684197+0000","scrub_duration":0.213161079,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24182949,"num_objects":57,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":0,"num_read":1117,"num_read_kb":268002,"num_write":1800,"num_write_kb":63122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":717548,"num_omap_keys":1302,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"85'2510","reported_seq":4556,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.637570+0000","last_change":"2023-10-30T13:25:54.637570+0000","last_active":"2023-10-30T13:25:54.637570+0000","last_peered":"2023-10-30T13:25:54.637570+0000","last_clean":"2023-10-30T13:25:54.637570+0000","last_became_active":"2023-10-30T12:28:44.977839+0000","last_became_peered":"2023-10-30T12:28:44.977839+0000","last_unstale":"2023-10-30T13:25:54.637570+0000","last_undegraded":"2023-10-30T13:25:54.637570+0000","last_fullsized":"2023-10-30T13:25:54.637570+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2510","last_scrub_stamp":"2023-10-30T13:25:54.637344+0000","last_deep_scrub":"85'2510","last_deep_scrub_stamp":"2023-10-30T13:25:54.637344+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.637344+0000","objects_scrubbed":709,"log_size":2510,"log_dups_size":0,"ondisk_log_size":2510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:29:25.404778+0000","scrub_duration":1.3090190449999999,"objects_trimmed":0,"snaptrim_duration":0.016143469000000001,"stat_sum":{"num_bytes":14540052,"num_objects":709,"num_object_clones":329,"num_object_copies":1418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":709,"num_whiteouts":254,"num_read":191,"num_read_kb":6719,"num_write":1583,"num_write_kb":26382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.18","version":"85'978","reported_seq":1391,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:00.613714+0000","last_change":"2023-10-30T13:26:00.613714+0000","last_active":"2023-10-30T13:26:00.613714+0000","last_peered":"2023-10-30T13:26:00.613714+0000","last_clean":"2023-10-30T13:26:00.613714+0000","last_became_active":"2023-10-30T12:28:44.058465+0000","last_became_peered":"2023-10-30T12:28:44.058465+0000","last_unstale":"2023-10-30T13:26:00.613714+0000","last_undegraded":"2023-10-30T13:26:00.613714+0000","last_fullsized":"2023-10-30T13:26:00.613714+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'978","last_scrub_stamp":"2023-10-30T13:26:00.613566+0000","last_deep_scrub":"85'978","last_deep_scrub_stamp":"2023-10-30T13:26:00.613566+0000","last_clean_scrub_stamp":"2023-10-30T13:26:00.613566+0000","objects_scrubbed":48,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:00:35.001259+0000","scrub_duration":0.17839571300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":48,"num_object_clones":0,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":542,"num_read_kb":359029,"num_write":1093,"num_write_kb":88246,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":622199,"num_omap_keys":1035,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.19","version":"85'2113","reported_seq":4165,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:55.071766+0000","last_change":"2023-10-30T13:25:55.071766+0000","last_active":"2023-10-30T13:25:55.071766+0000","last_peered":"2023-10-30T13:25:55.071766+0000","last_clean":"2023-10-30T13:25:55.071766+0000","last_became_active":"2023-10-30T12:28:44.948152+0000","last_became_peered":"2023-10-30T12:28:44.948152+0000","last_unstale":"2023-10-30T13:25:55.071766+0000","last_undegraded":"2023-10-30T13:25:55.071766+0000","last_fullsized":"2023-10-30T13:25:55.071766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2113","last_scrub_stamp":"2023-10-30T13:25:55.071614+0000","last_deep_scrub":"85'2113","last_deep_scrub_stamp":"2023-10-30T13:25:55.071614+0000","last_clean_scrub_stamp":"2023-10-30T13:25:55.071614+0000","objects_scrubbed":589,"log_size":2113,"log_dups_size":0,"ondisk_log_size":2113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:23:07.445157+0000","scrub_duration":0.93122268500000005,"objects_trimmed":0,"snaptrim_duration":0.0083101389999999994,"stat_sum":{"num_bytes":18904879,"num_objects":589,"num_object_clones":276,"num_object_copies":1178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":589,"num_whiteouts":211,"num_read":147,"num_read_kb":20021,"num_write":1340,"num_write_kb":25244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"85'789","reported_seq":1201,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:49.688470+0000","last_change":"2023-10-30T13:25:49.688470+0000","last_active":"2023-10-30T13:25:49.688470+0000","last_peered":"2023-10-30T13:25:49.688470+0000","last_clean":"2023-10-30T13:25:49.688470+0000","last_became_active":"2023-10-30T12:28:44.063162+0000","last_became_peered":"2023-10-30T12:28:44.063162+0000","last_unstale":"2023-10-30T13:25:49.688470+0000","last_undegraded":"2023-10-30T13:25:49.688470+0000","last_fullsized":"2023-10-30T13:25:49.688470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'789","last_scrub_stamp":"2023-10-30T13:25:49.688344+0000","last_deep_scrub":"85'789","last_deep_scrub_stamp":"2023-10-30T13:25:49.688344+0000","last_clean_scrub_stamp":"2023-10-30T13:25:49.688344+0000","objects_scrubbed":53,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:56.666560+0000","scrub_duration":0.15913772100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20971610,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":616,"num_read_kb":251423,"num_write":890,"num_write_kb":55054,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":627193,"num_omap_keys":1090,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.16","version":"85'2098","reported_seq":4146,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.184868+0000","last_change":"2023-10-30T13:25:57.184868+0000","last_active":"2023-10-30T13:25:57.184868+0000","last_peered":"2023-10-30T13:25:57.184868+0000","last_clean":"2023-10-30T13:25:57.184868+0000","last_became_active":"2023-10-30T12:28:44.950776+0000","last_became_peered":"2023-10-30T12:28:44.950776+0000","last_unstale":"2023-10-30T13:25:57.184868+0000","last_undegraded":"2023-10-30T13:25:57.184868+0000","last_fullsized":"2023-10-30T13:25:57.184868+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2098","last_scrub_stamp":"2023-10-30T13:25:57.184676+0000","last_deep_scrub":"85'2098","last_deep_scrub_stamp":"2023-10-30T13:25:57.184676+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.184676+0000","objects_scrubbed":580,"log_size":2098,"log_dups_size":0,"ondisk_log_size":2098,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:56:36.904344+0000","scrub_duration":1.084383391,"objects_trimmed":0,"snaptrim_duration":0.0043045619999999996,"stat_sum":{"num_bytes":10134301,"num_objects":580,"num_object_clones":271,"num_object_copies":1160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":580,"num_whiteouts":219,"num_read":138,"num_read_kb":8986,"num_write":1318,"num_write_kb":11276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.16","version":"85'1309","reported_seq":1759,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.351972+0000","last_change":"2023-10-30T13:25:54.351972+0000","last_active":"2023-10-30T13:25:54.351972+0000","last_peered":"2023-10-30T13:25:54.351972+0000","last_clean":"2023-10-30T13:25:54.351972+0000","last_became_active":"2023-10-30T12:28:44.052286+0000","last_became_peered":"2023-10-30T12:28:44.052286+0000","last_unstale":"2023-10-30T13:25:54.351972+0000","last_undegraded":"2023-10-30T13:25:54.351972+0000","last_fullsized":"2023-10-30T13:25:54.351972+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1309","last_scrub_stamp":"2023-10-30T13:25:54.351769+0000","last_deep_scrub":"85'1309","last_deep_scrub_stamp":"2023-10-30T13:25:54.351769+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.351769+0000","objects_scrubbed":62,"log_size":1309,"log_dups_size":0,"ondisk_log_size":1309,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:12:15.955314+0000","scrub_duration":0.18331900800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":659,"num_read_kb":339384,"num_write":1492,"num_write_kb":80693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":875213,"num_omap_keys":1399,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"85'2223","reported_seq":4247,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:59.748025+0000","last_change":"2023-10-30T13:25:59.748025+0000","last_active":"2023-10-30T13:25:59.748025+0000","last_peered":"2023-10-30T13:25:59.748025+0000","last_clean":"2023-10-30T13:25:59.748025+0000","last_became_active":"2023-10-30T12:28:44.980964+0000","last_became_peered":"2023-10-30T12:28:44.980964+0000","last_unstale":"2023-10-30T13:25:59.748025+0000","last_undegraded":"2023-10-30T13:25:59.748025+0000","last_fullsized":"2023-10-30T13:25:59.748025+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2223","last_scrub_stamp":"2023-10-30T13:25:59.747898+0000","last_deep_scrub":"85'2223","last_deep_scrub_stamp":"2023-10-30T13:25:59.747898+0000","last_clean_scrub_stamp":"2023-10-30T13:25:59.747898+0000","objects_scrubbed":620,"log_size":2223,"log_dups_size":0,"ondisk_log_size":2223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:59:22.443096+0000","scrub_duration":1.320524545,"objects_trimmed":0,"snaptrim_duration":0.015702779,"stat_sum":{"num_bytes":15773172,"num_objects":620,"num_object_clones":292,"num_object_copies":1240,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":620,"num_whiteouts":204,"num_read":135,"num_read_kb":17351,"num_write":1431,"num_write_kb":17077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.15","version":"85'1282","reported_seq":1787,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:43.375075+0000","last_change":"2023-10-30T13:25:43.375075+0000","last_active":"2023-10-30T13:25:43.375075+0000","last_peered":"2023-10-30T13:25:43.375075+0000","last_clean":"2023-10-30T13:25:43.375075+0000","last_became_active":"2023-10-30T12:28:44.074866+0000","last_became_peered":"2023-10-30T12:28:44.074866+0000","last_unstale":"2023-10-30T13:25:43.375075+0000","last_undegraded":"2023-10-30T13:25:43.375075+0000","last_fullsized":"2023-10-30T13:25:43.375075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1282","last_scrub_stamp":"2023-10-30T13:25:43.374894+0000","last_deep_scrub":"85'1282","last_deep_scrub_stamp":"2023-10-30T13:25:43.374894+0000","last_clean_scrub_stamp":"2023-10-30T13:25:43.374894+0000","objects_scrubbed":59,"log_size":1282,"log_dups_size":0,"ondisk_log_size":1282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:09:32.582441+0000","scrub_duration":0.20001206899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19774519,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":862,"num_read_kb":221068,"num_write":1454,"num_write_kb":51536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":50,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":878727,"num_omap_keys":1229,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"85'2127","reported_seq":4170,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:30.367224+0000","last_change":"2023-10-30T13:26:30.367270+0000","last_active":"2023-10-30T13:26:30.367224+0000","last_peered":"2023-10-30T13:26:30.367224+0000","last_clean":"2023-10-30T13:26:30.367224+0000","last_became_active":"2023-10-30T12:28:44.950583+0000","last_became_peered":"2023-10-30T12:28:44.950583+0000","last_unstale":"2023-10-30T13:26:30.367224+0000","last_undegraded":"2023-10-30T13:26:30.367224+0000","last_fullsized":"2023-10-30T13:26:30.367224+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2127","last_scrub_stamp":"2023-10-30T13:26:30.367149+0000","last_deep_scrub":"85'2127","last_deep_scrub_stamp":"2023-10-30T13:26:30.367149+0000","last_clean_scrub_stamp":"2023-10-30T13:26:30.367149+0000","objects_scrubbed":612,"log_size":2127,"log_dups_size":0,"ondisk_log_size":2127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:03:49.227794+0000","scrub_duration":0.73352292799999996,"objects_trimmed":0,"snaptrim_duration":0.0080610860000000003,"stat_sum":{"num_bytes":22077043,"num_objects":612,"num_object_clones":281,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":230,"num_read":141,"num_read_kb":18856,"num_write":1294,"num_write_kb":28337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"85'1397","reported_seq":1954,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:41.415232+0000","last_change":"2023-10-30T13:25:41.415318+0000","last_active":"2023-10-30T13:25:41.415232+0000","last_peered":"2023-10-30T13:25:41.415232+0000","last_clean":"2023-10-30T13:25:41.415232+0000","last_became_active":"2023-10-30T12:28:44.062725+0000","last_became_peered":"2023-10-30T12:28:44.062725+0000","last_unstale":"2023-10-30T13:25:41.415232+0000","last_undegraded":"2023-10-30T13:25:41.415232+0000","last_fullsized":"2023-10-30T13:25:41.415232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1397","last_scrub_stamp":"2023-10-30T13:25:41.415138+0000","last_deep_scrub":"85'1397","last_deep_scrub_stamp":"2023-10-30T13:25:41.415138+0000","last_clean_scrub_stamp":"2023-10-30T13:25:41.415138+0000","objects_scrubbed":67,"log_size":1397,"log_dups_size":0,"ondisk_log_size":1397,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:34:14.524619+0000","scrub_duration":0.28929933099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37749408,"num_objects":67,"num_object_clones":0,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":865,"num_read_kb":420163,"num_write":1566,"num_write_kb":89397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":54,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":814706,"num_omap_keys":1307,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"85'2141","reported_seq":4173,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:01.200009+0000","last_change":"2023-10-30T13:26:01.200009+0000","last_active":"2023-10-30T13:26:01.200009+0000","last_peered":"2023-10-30T13:26:01.200009+0000","last_clean":"2023-10-30T13:26:01.200009+0000","last_became_active":"2023-10-30T12:28:44.971786+0000","last_became_peered":"2023-10-30T12:28:44.971786+0000","last_unstale":"2023-10-30T13:26:01.200009+0000","last_undegraded":"2023-10-30T13:26:01.200009+0000","last_fullsized":"2023-10-30T13:26:01.200009+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2141","last_scrub_stamp":"2023-10-30T13:26:01.199816+0000","last_deep_scrub":"85'2141","last_deep_scrub_stamp":"2023-10-30T13:26:01.199816+0000","last_clean_scrub_stamp":"2023-10-30T13:26:01.199816+0000","objects_scrubbed":606,"log_size":2141,"log_dups_size":0,"ondisk_log_size":2141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:21:03.156283+0000","scrub_duration":1.132667525,"objects_trimmed":0,"snaptrim_duration":0.008640567,"stat_sum":{"num_bytes":23022369,"num_objects":606,"num_object_clones":279,"num_object_copies":1212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":606,"num_whiteouts":220,"num_read":129,"num_read_kb":18941,"num_write":1340,"num_write_kb":24878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"85'1164","reported_seq":1635,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.276943+0000","last_change":"2023-10-30T13:25:48.277044+0000","last_active":"2023-10-30T13:25:48.276943+0000","last_peered":"2023-10-30T13:25:48.276943+0000","last_clean":"2023-10-30T13:25:48.276943+0000","last_became_active":"2023-10-30T12:28:44.064744+0000","last_became_peered":"2023-10-30T12:28:44.064744+0000","last_unstale":"2023-10-30T13:25:48.276943+0000","last_undegraded":"2023-10-30T13:25:48.276943+0000","last_fullsized":"2023-10-30T13:25:48.276943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1164","last_scrub_stamp":"2023-10-30T13:25:48.276852+0000","last_deep_scrub":"85'1164","last_deep_scrub_stamp":"2023-10-30T13:25:48.276852+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.276852+0000","objects_scrubbed":66,"log_size":1164,"log_dups_size":0,"ondisk_log_size":1164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:14:06.856285+0000","scrub_duration":0.19954839099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554466,"num_objects":66,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":701,"num_read_kb":318483,"num_write":1303,"num_write_kb":68223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":676667,"num_omap_keys":1245,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"85'2197","reported_seq":4231,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:52.017102+0000","last_change":"2023-10-30T13:25:52.017193+0000","last_active":"2023-10-30T13:25:52.017102+0000","last_peered":"2023-10-30T13:25:52.017102+0000","last_clean":"2023-10-30T13:25:52.017102+0000","last_became_active":"2023-10-30T12:28:44.960515+0000","last_became_peered":"2023-10-30T12:28:44.960515+0000","last_unstale":"2023-10-30T13:25:52.017102+0000","last_undegraded":"2023-10-30T13:25:52.017102+0000","last_fullsized":"2023-10-30T13:25:52.017102+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2197","last_scrub_stamp":"2023-10-30T13:25:52.016976+0000","last_deep_scrub":"85'2197","last_deep_scrub_stamp":"2023-10-30T13:25:52.016976+0000","last_clean_scrub_stamp":"2023-10-30T13:25:52.016976+0000","objects_scrubbed":613,"log_size":2197,"log_dups_size":0,"ondisk_log_size":2197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:09:13.519760+0000","scrub_duration":1.011428851,"objects_trimmed":0,"snaptrim_duration":0.0073009930000000004,"stat_sum":{"num_bytes":10084838,"num_objects":613,"num_object_clones":288,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":217,"num_read":139,"num_read_kb":5686,"num_write":1383,"num_write_kb":20147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"85'729","reported_seq":1132,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:54.578886+0000","last_change":"2023-10-30T13:25:54.578930+0000","last_active":"2023-10-30T13:25:54.578886+0000","last_peered":"2023-10-30T13:25:54.578886+0000","last_clean":"2023-10-30T13:25:54.578886+0000","last_became_active":"2023-10-30T12:28:44.061585+0000","last_became_peered":"2023-10-30T12:28:44.061585+0000","last_unstale":"2023-10-30T13:25:54.578886+0000","last_undegraded":"2023-10-30T13:25:54.578886+0000","last_fullsized":"2023-10-30T13:25:54.578886+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'729","last_scrub_stamp":"2023-10-30T13:25:54.578800+0000","last_deep_scrub":"85'729","last_deep_scrub_stamp":"2023-10-30T13:25:54.578800+0000","last_clean_scrub_stamp":"2023-10-30T13:25:54.578800+0000","objects_scrubbed":53,"log_size":729,"log_dups_size":0,"ondisk_log_size":729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:06:15.394870+0000","scrub_duration":0.12999461300000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":600,"num_read_kb":208389,"num_write":899,"num_write_kb":51587,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":569345,"num_omap_keys":982,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"85'2297","reported_seq":4342,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:22.388451+0000","last_change":"2023-10-30T13:26:22.388507+0000","last_active":"2023-10-30T13:26:22.388451+0000","last_peered":"2023-10-30T13:26:22.388451+0000","last_clean":"2023-10-30T13:26:22.388451+0000","last_became_active":"2023-10-30T12:28:44.957946+0000","last_became_peered":"2023-10-30T12:28:44.957946+0000","last_unstale":"2023-10-30T13:26:22.388451+0000","last_undegraded":"2023-10-30T13:26:22.388451+0000","last_fullsized":"2023-10-30T13:26:22.388451+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2297","last_scrub_stamp":"2023-10-30T13:26:22.388371+0000","last_deep_scrub":"85'2297","last_deep_scrub_stamp":"2023-10-30T13:26:22.388371+0000","last_clean_scrub_stamp":"2023-10-30T13:26:22.388371+0000","objects_scrubbed":655,"log_size":2297,"log_dups_size":0,"ondisk_log_size":2297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:53:56.894212+0000","scrub_duration":0.80626637899999998,"objects_trimmed":0,"snaptrim_duration":0.0091258150000000007,"stat_sum":{"num_bytes":19392609,"num_objects":655,"num_object_clones":299,"num_object_copies":1310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":655,"num_whiteouts":233,"num_read":161,"num_read_kb":7475,"num_write":1440,"num_write_kb":20643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.11","version":"85'1416","reported_seq":1810,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:40.394192+0000","last_change":"2023-10-30T13:25:40.394262+0000","last_active":"2023-10-30T13:25:40.394192+0000","last_peered":"2023-10-30T13:25:40.394192+0000","last_clean":"2023-10-30T13:25:40.394192+0000","last_became_active":"2023-10-30T12:28:44.059508+0000","last_became_peered":"2023-10-30T12:28:44.059508+0000","last_unstale":"2023-10-30T13:25:40.394192+0000","last_undegraded":"2023-10-30T13:25:40.394192+0000","last_fullsized":"2023-10-30T13:25:40.394192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1416","last_scrub_stamp":"2023-10-30T13:25:40.394118+0000","last_deep_scrub":"85'1416","last_deep_scrub_stamp":"2023-10-30T13:25:40.394118+0000","last_clean_scrub_stamp":"2023-10-30T13:25:40.394118+0000","objects_scrubbed":45,"log_size":1416,"log_dups_size":0,"ondisk_log_size":1416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:50:35.732748+0000","scrub_duration":0.10710792099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":539,"num_read_kb":266844,"num_write":1512,"num_write_kb":63647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":566794,"num_omap_keys":1066,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"85'2071","reported_seq":4113,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:58.521640+0000","last_change":"2023-10-30T13:25:58.521719+0000","last_active":"2023-10-30T13:25:58.521640+0000","last_peered":"2023-10-30T13:25:58.521640+0000","last_clean":"2023-10-30T13:25:58.521640+0000","last_became_active":"2023-10-30T12:28:44.980497+0000","last_became_peered":"2023-10-30T12:28:44.980497+0000","last_unstale":"2023-10-30T13:25:58.521640+0000","last_undegraded":"2023-10-30T13:25:58.521640+0000","last_fullsized":"2023-10-30T13:25:58.521640+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2071","last_scrub_stamp":"2023-10-30T13:25:58.521516+0000","last_deep_scrub":"85'2071","last_deep_scrub_stamp":"2023-10-30T13:25:58.521516+0000","last_clean_scrub_stamp":"2023-10-30T13:25:58.521516+0000","objects_scrubbed":590,"log_size":2071,"log_dups_size":0,"ondisk_log_size":2071,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:24:10.505945+0000","scrub_duration":1.0833768969999999,"objects_trimmed":0,"snaptrim_duration":0.001646376,"stat_sum":{"num_bytes":13778361,"num_objects":590,"num_object_clones":278,"num_object_copies":1180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":590,"num_whiteouts":227,"num_read":137,"num_read_kb":5723,"num_write":1258,"num_write_kb":14976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"85'1020","reported_seq":1516,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:45.278920+0000","last_change":"2023-10-30T13:25:45.278968+0000","last_active":"2023-10-30T13:25:45.278920+0000","last_peered":"2023-10-30T13:25:45.278920+0000","last_clean":"2023-10-30T13:25:45.278920+0000","last_became_active":"2023-10-30T12:28:44.064403+0000","last_became_peered":"2023-10-30T12:28:44.064403+0000","last_unstale":"2023-10-30T13:25:45.278920+0000","last_undegraded":"2023-10-30T13:25:45.278920+0000","last_fullsized":"2023-10-30T13:25:45.278920+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1020","last_scrub_stamp":"2023-10-30T13:25:45.278835+0000","last_deep_scrub":"85'1020","last_deep_scrub_stamp":"2023-10-30T13:25:45.278835+0000","last_clean_scrub_stamp":"2023-10-30T13:25:45.278835+0000","objects_scrubbed":50,"log_size":1020,"log_dups_size":0,"ondisk_log_size":1020,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:18:17.671466+0000","scrub_duration":0.27418579599999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554544,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":863,"num_read_kb":272228,"num_write":1253,"num_write_kb":56723,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":534875,"num_omap_keys":741,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"85'2238","reported_seq":4293,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:18.465655+0000","last_change":"2023-10-30T13:26:18.465733+0000","last_active":"2023-10-30T13:26:18.465655+0000","last_peered":"2023-10-30T13:26:18.465655+0000","last_clean":"2023-10-30T13:26:18.465655+0000","last_became_active":"2023-10-30T12:28:44.952190+0000","last_became_peered":"2023-10-30T12:28:44.952190+0000","last_unstale":"2023-10-30T13:26:18.465655+0000","last_undegraded":"2023-10-30T13:26:18.465655+0000","last_fullsized":"2023-10-30T13:26:18.465655+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2238","last_scrub_stamp":"2023-10-30T13:26:18.465564+0000","last_deep_scrub":"85'2238","last_deep_scrub_stamp":"2023-10-30T13:26:18.465564+0000","last_clean_scrub_stamp":"2023-10-30T13:26:18.465564+0000","objects_scrubbed":622,"log_size":2238,"log_dups_size":0,"ondisk_log_size":2238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:04.508589+0000","scrub_duration":0.99352104299999999,"objects_trimmed":0,"snaptrim_duration":0.0098326820000000006,"stat_sum":{"num_bytes":10901701,"num_objects":622,"num_object_clones":292,"num_object_copies":1244,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":622,"num_whiteouts":232,"num_read":163,"num_read_kb":14238,"num_write":1406,"num_write_kb":21652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"85'696","reported_seq":1065,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:42.423508+0000","last_change":"2023-10-30T13:25:42.423609+0000","last_active":"2023-10-30T13:25:42.423508+0000","last_peered":"2023-10-30T13:25:42.423508+0000","last_clean":"2023-10-30T13:25:42.423508+0000","last_became_active":"2023-10-30T12:28:44.079677+0000","last_became_peered":"2023-10-30T12:28:44.079677+0000","last_unstale":"2023-10-30T13:25:42.423508+0000","last_undegraded":"2023-10-30T13:25:42.423508+0000","last_fullsized":"2023-10-30T13:25:42.423508+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'696","last_scrub_stamp":"2023-10-30T13:25:42.423407+0000","last_deep_scrub":"85'696","last_deep_scrub_stamp":"2023-10-30T13:25:42.423407+0000","last_clean_scrub_stamp":"2023-10-30T13:25:42.423407+0000","objects_scrubbed":41,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:12:00.329817+0000","scrub_duration":0.18306038499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360150,"num_objects":41,"num_object_clones":0,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":443,"num_read_kb":261670,"num_write":858,"num_write_kb":58986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":31,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":305625,"num_omap_keys":524,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"85'2220","reported_seq":4221,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.104935+0000","last_change":"2023-10-30T13:25:53.105023+0000","last_active":"2023-10-30T13:25:53.104935+0000","last_peered":"2023-10-30T13:25:53.104935+0000","last_clean":"2023-10-30T13:25:53.104935+0000","last_became_active":"2023-10-30T12:28:44.959561+0000","last_became_peered":"2023-10-30T12:28:44.959561+0000","last_unstale":"2023-10-30T13:25:53.104935+0000","last_undegraded":"2023-10-30T13:25:53.104935+0000","last_fullsized":"2023-10-30T13:25:53.104935+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2220","last_scrub_stamp":"2023-10-30T13:25:53.104840+0000","last_deep_scrub":"85'2220","last_deep_scrub_stamp":"2023-10-30T13:25:53.104840+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.104840+0000","objects_scrubbed":647,"log_size":2220,"log_dups_size":0,"ondisk_log_size":2220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:47:46.541296+0000","scrub_duration":1.013459932,"objects_trimmed":0,"snaptrim_duration":0.004602437,"stat_sum":{"num_bytes":20428330,"num_objects":647,"num_object_clones":304,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":239,"num_read":119,"num_read_kb":13097,"num_write":1360,"num_write_kb":26395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.e","version":"85'779","reported_seq":1239,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.328717+0000","last_change":"2023-10-30T13:25:48.328798+0000","last_active":"2023-10-30T13:25:48.328717+0000","last_peered":"2023-10-30T13:25:48.328717+0000","last_clean":"2023-10-30T13:25:48.328717+0000","last_became_active":"2023-10-30T12:28:44.075614+0000","last_became_peered":"2023-10-30T12:28:44.075614+0000","last_unstale":"2023-10-30T13:25:48.328717+0000","last_undegraded":"2023-10-30T13:25:48.328717+0000","last_fullsized":"2023-10-30T13:25:48.328717+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'779","last_scrub_stamp":"2023-10-30T13:25:48.328628+0000","last_deep_scrub":"85'779","last_deep_scrub_stamp":"2023-10-30T13:25:48.328628+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.328628+0000","objects_scrubbed":62,"log_size":779,"log_dups_size":0,"ondisk_log_size":779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:35:51.114462+0000","scrub_duration":0.22182811399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":62,"num_object_clones":0,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":635,"num_read_kb":434171,"num_write":906,"num_write_kb":88121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":51,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":675117,"num_omap_keys":1207,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"85'2216","reported_seq":4237,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:59.936763+0000","last_change":"2023-10-30T13:25:59.936829+0000","last_active":"2023-10-30T13:25:59.936763+0000","last_peered":"2023-10-30T13:25:59.936763+0000","last_clean":"2023-10-30T13:25:59.936763+0000","last_became_active":"2023-10-30T12:29:27.340504+0000","last_became_peered":"2023-10-30T12:29:27.340504+0000","last_unstale":"2023-10-30T13:25:59.936763+0000","last_undegraded":"2023-10-30T13:25:59.936763+0000","last_fullsized":"2023-10-30T13:25:59.936763+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2216","last_scrub_stamp":"2023-10-30T13:25:59.936683+0000","last_deep_scrub":"85'2216","last_deep_scrub_stamp":"2023-10-30T13:25:59.936683+0000","last_clean_scrub_stamp":"2023-10-30T13:25:59.936683+0000","objects_scrubbed":641,"log_size":2216,"log_dups_size":0,"ondisk_log_size":2216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:57:07.443321+0000","scrub_duration":1.7189143169999999,"objects_trimmed":0,"snaptrim_duration":0.00653923,"stat_sum":{"num_bytes":23737065,"num_objects":641,"num_object_clones":299,"num_object_copies":1282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":641,"num_whiteouts":238,"num_read":138,"num_read_kb":28255,"num_write":1351,"num_write_kb":33550,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'766","reported_seq":1157,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:51.439924+0000","last_change":"2023-10-30T13:25:51.439991+0000","last_active":"2023-10-30T13:25:51.439924+0000","last_peered":"2023-10-30T13:25:51.439924+0000","last_clean":"2023-10-30T13:25:51.439924+0000","last_became_active":"2023-10-30T12:28:44.074774+0000","last_became_peered":"2023-10-30T12:28:44.074774+0000","last_unstale":"2023-10-30T13:25:51.439924+0000","last_undegraded":"2023-10-30T13:25:51.439924+0000","last_fullsized":"2023-10-30T13:25:51.439924+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'766","last_scrub_stamp":"2023-10-30T13:25:51.439810+0000","last_deep_scrub":"85'766","last_deep_scrub_stamp":"2023-10-30T13:25:51.439810+0000","last_clean_scrub_stamp":"2023-10-30T13:25:51.439810+0000","objects_scrubbed":50,"log_size":766,"log_dups_size":0,"ondisk_log_size":766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:50:56.709508+0000","scrub_duration":0.13294180999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29364369,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":493,"num_read_kb":320778,"num_write":891,"num_write_kb":71265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":522473,"num_omap_keys":1010,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'2210","reported_seq":4257,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:08.440439+0000","last_change":"2023-10-30T13:26:08.440557+0000","last_active":"2023-10-30T13:26:08.440439+0000","last_peered":"2023-10-30T13:26:08.440439+0000","last_clean":"2023-10-30T13:26:08.440439+0000","last_became_active":"2023-10-30T12:28:44.955780+0000","last_became_peered":"2023-10-30T12:28:44.955780+0000","last_unstale":"2023-10-30T13:26:08.440439+0000","last_undegraded":"2023-10-30T13:26:08.440439+0000","last_fullsized":"2023-10-30T13:26:08.440439+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2210","last_scrub_stamp":"2023-10-30T13:26:08.440312+0000","last_deep_scrub":"85'2210","last_deep_scrub_stamp":"2023-10-30T13:26:08.440312+0000","last_clean_scrub_stamp":"2023-10-30T13:26:08.440312+0000","objects_scrubbed":611,"log_size":2210,"log_dups_size":0,"ondisk_log_size":2210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:58:49.728570+0000","scrub_duration":1.28748733,"objects_trimmed":0,"snaptrim_duration":0.0076673150000000001,"stat_sum":{"num_bytes":18256447,"num_objects":611,"num_object_clones":286,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":229,"num_read":149,"num_read_kb":24381,"num_write":1383,"num_write_kb":20195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.c","version":"85'645","reported_seq":1007,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:48.161404+0000","last_change":"2023-10-30T13:25:48.161534+0000","last_active":"2023-10-30T13:25:48.161404+0000","last_peered":"2023-10-30T13:25:48.161404+0000","last_clean":"2023-10-30T13:25:48.161404+0000","last_became_active":"2023-10-30T12:28:44.059868+0000","last_became_peered":"2023-10-30T12:28:44.059868+0000","last_unstale":"2023-10-30T13:25:48.161404+0000","last_undegraded":"2023-10-30T13:25:48.161404+0000","last_fullsized":"2023-10-30T13:25:48.161404+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'645","last_scrub_stamp":"2023-10-30T13:25:48.161279+0000","last_deep_scrub":"85'645","last_deep_scrub_stamp":"2023-10-30T13:25:48.161279+0000","last_clean_scrub_stamp":"2023-10-30T13:25:48.161279+0000","objects_scrubbed":38,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:28:29.280017+0000","scrub_duration":0.15056981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360710,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":462,"num_read_kb":342669,"num_write":689,"num_write_kb":66904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":29,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":355436,"num_omap_keys":678,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"85'2132","reported_seq":4147,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:36.536448+0000","last_change":"2023-10-30T13:26:36.536557+0000","last_active":"2023-10-30T13:26:36.536448+0000","last_peered":"2023-10-30T13:26:36.536448+0000","last_clean":"2023-10-30T13:26:36.536448+0000","last_became_active":"2023-10-30T12:28:44.946482+0000","last_became_peered":"2023-10-30T12:28:44.946482+0000","last_unstale":"2023-10-30T13:26:36.536448+0000","last_undegraded":"2023-10-30T13:26:36.536448+0000","last_fullsized":"2023-10-30T13:26:36.536448+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2132","last_scrub_stamp":"2023-10-30T13:26:36.536359+0000","last_deep_scrub":"85'2132","last_deep_scrub_stamp":"2023-10-30T13:26:36.536359+0000","last_clean_scrub_stamp":"2023-10-30T13:26:36.536359+0000","objects_scrubbed":612,"log_size":2132,"log_dups_size":0,"ondisk_log_size":2132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:41:52.662853+0000","scrub_duration":0.86036190300000004,"objects_trimmed":0,"snaptrim_duration":0.010286791999999999,"stat_sum":{"num_bytes":26944901,"num_objects":612,"num_object_clones":287,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":236,"num_read":119,"num_read_kb":34683,"num_write":1299,"num_write_kb":33129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"85'119","reported_seq":268,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:46.486986+0000","last_change":"2023-10-30T13:25:46.486986+0000","last_active":"2023-10-30T13:25:46.486986+0000","last_peered":"2023-10-30T13:25:46.486986+0000","last_clean":"2023-10-30T13:25:46.486986+0000","last_became_active":"2023-10-30T12:28:33.432754+0000","last_became_peered":"2023-10-30T12:28:33.432754+0000","last_unstale":"2023-10-30T13:25:46.486986+0000","last_undegraded":"2023-10-30T13:25:46.486986+0000","last_fullsized":"2023-10-30T13:25:46.486986+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'119","last_scrub_stamp":"2023-10-30T13:25:46.486855+0000","last_deep_scrub":"85'119","last_deep_scrub_stamp":"2023-10-30T13:25:46.486855+0000","last_clean_scrub_stamp":"2023-10-30T13:25:46.486855+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:52:13.395892+0000","scrub_duration":0.0083870119999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"85'1371","reported_seq":1882,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:10.600505+0000","last_change":"2023-10-30T13:26:10.600590+0000","last_active":"2023-10-30T13:26:10.600505+0000","last_peered":"2023-10-30T13:26:10.600505+0000","last_clean":"2023-10-30T13:26:10.600505+0000","last_became_active":"2023-10-30T12:28:44.058809+0000","last_became_peered":"2023-10-30T12:28:44.058809+0000","last_unstale":"2023-10-30T13:26:10.600505+0000","last_undegraded":"2023-10-30T13:26:10.600505+0000","last_fullsized":"2023-10-30T13:26:10.600505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1371","last_scrub_stamp":"2023-10-30T13:26:10.600409+0000","last_deep_scrub":"85'1371","last_deep_scrub_stamp":"2023-10-30T13:26:10.600409+0000","last_clean_scrub_stamp":"2023-10-30T13:26:10.600409+0000","objects_scrubbed":53,"log_size":1371,"log_dups_size":0,"ondisk_log_size":1371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:34:15.701205+0000","scrub_duration":0.14720427899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165890,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":877,"num_read_kb":231219,"num_write":1489,"num_write_kb":63137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":475860,"num_omap_keys":919,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2","version":"85'2200","reported_seq":4242,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.149503+0000","last_change":"2023-10-30T13:25:57.149588+0000","last_active":"2023-10-30T13:25:57.149503+0000","last_peered":"2023-10-30T13:25:57.149503+0000","last_clean":"2023-10-30T13:25:57.149503+0000","last_became_active":"2023-10-30T12:28:44.964374+0000","last_became_peered":"2023-10-30T12:28:44.964374+0000","last_unstale":"2023-10-30T13:25:57.149503+0000","last_undegraded":"2023-10-30T13:25:57.149503+0000","last_fullsized":"2023-10-30T13:25:57.149503+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2200","last_scrub_stamp":"2023-10-30T13:25:57.149395+0000","last_deep_scrub":"85'2200","last_deep_scrub_stamp":"2023-10-30T13:25:57.149395+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.149395+0000","objects_scrubbed":626,"log_size":2200,"log_dups_size":0,"ondisk_log_size":2200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:01:04.364636+0000","scrub_duration":1.0072585730000001,"objects_trimmed":0,"snaptrim_duration":0.0068470270000000003,"stat_sum":{"num_bytes":8097312,"num_objects":626,"num_object_clones":290,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":225,"num_read":146,"num_read_kb":2079,"num_write":1425,"num_write_kb":13192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"85'2234","reported_seq":2616,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:57.726488+0000","last_change":"2023-10-30T13:25:57.726488+0000","last_active":"2023-10-30T13:25:57.726488+0000","last_peered":"2023-10-30T13:25:57.726488+0000","last_clean":"2023-10-30T13:25:57.726488+0000","last_became_active":"2023-10-30T12:28:44.079949+0000","last_became_peered":"2023-10-30T12:28:44.079949+0000","last_unstale":"2023-10-30T13:25:57.726488+0000","last_undegraded":"2023-10-30T13:25:57.726488+0000","last_fullsized":"2023-10-30T13:25:57.726488+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2234","last_scrub_stamp":"2023-10-30T13:25:57.726328+0000","last_deep_scrub":"85'2234","last_deep_scrub_stamp":"2023-10-30T13:25:57.726328+0000","last_clean_scrub_stamp":"2023-10-30T13:25:57.726328+0000","objects_scrubbed":42,"log_size":2234,"log_dups_size":0,"ondisk_log_size":2234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:34:51.655678+0000","scrub_duration":0.194142807,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":516,"num_read_kb":283293,"num_write":2369,"num_write_kb":72030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":392036,"num_omap_keys":649,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"85'2035","reported_seq":4089,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:05.024953+0000","last_change":"2023-10-30T13:26:05.025106+0000","last_active":"2023-10-30T13:26:05.024953+0000","last_peered":"2023-10-30T13:26:05.024953+0000","last_clean":"2023-10-30T13:26:05.024953+0000","last_became_active":"2023-10-30T12:28:44.960467+0000","last_became_peered":"2023-10-30T12:28:44.960467+0000","last_unstale":"2023-10-30T13:26:05.024953+0000","last_undegraded":"2023-10-30T13:26:05.024953+0000","last_fullsized":"2023-10-30T13:26:05.024953+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2035","last_scrub_stamp":"2023-10-30T13:26:05.024740+0000","last_deep_scrub":"85'2035","last_deep_scrub_stamp":"2023-10-30T13:26:05.024740+0000","last_clean_scrub_stamp":"2023-10-30T13:26:05.024740+0000","objects_scrubbed":570,"log_size":2035,"log_dups_size":0,"ondisk_log_size":2035,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:48:20.536984+0000","scrub_duration":1.046857548,"objects_trimmed":0,"snaptrim_duration":0.0057755250000000001,"stat_sum":{"num_bytes":14269585,"num_objects":570,"num_object_clones":262,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":207,"num_read":132,"num_read_kb":18022,"num_write":1264,"num_write_kb":17917,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"85'1898","reported_seq":3194,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:52.380541+0000","last_change":"2023-10-30T13:25:52.380604+0000","last_active":"2023-10-30T13:25:52.380541+0000","last_peered":"2023-10-30T13:25:52.380541+0000","last_clean":"2023-10-30T13:25:52.380541+0000","last_became_active":"2023-10-30T12:28:44.081837+0000","last_became_peered":"2023-10-30T12:28:44.081837+0000","last_unstale":"2023-10-30T13:25:52.380541+0000","last_undegraded":"2023-10-30T13:25:52.380541+0000","last_fullsized":"2023-10-30T13:25:52.380541+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1898","last_scrub_stamp":"2023-10-30T13:25:52.380459+0000","last_deep_scrub":"85'1898","last_deep_scrub_stamp":"2023-10-30T13:25:52.380459+0000","last_clean_scrub_stamp":"2023-10-30T13:25:52.380459+0000","objects_scrubbed":61,"log_size":1898,"log_dups_size":0,"ondisk_log_size":1898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:44:04.920583+0000","scrub_duration":0.247546554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49235352,"num_objects":61,"num_object_clones":0,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":0,"num_read":1514,"num_read_kb":360508,"num_write":2035,"num_write_kb":92558,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":705593,"num_omap_keys":1067,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"85'2175","reported_seq":4221,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:20.787360+0000","last_change":"2023-10-30T13:26:20.787360+0000","last_active":"2023-10-30T13:26:20.787360+0000","last_peered":"2023-10-30T13:26:20.787360+0000","last_clean":"2023-10-30T13:26:20.787360+0000","last_became_active":"2023-10-30T12:28:44.973264+0000","last_became_peered":"2023-10-30T12:28:44.973264+0000","last_unstale":"2023-10-30T13:26:20.787360+0000","last_undegraded":"2023-10-30T13:26:20.787360+0000","last_fullsized":"2023-10-30T13:26:20.787360+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2175","last_scrub_stamp":"2023-10-30T13:26:20.787197+0000","last_deep_scrub":"85'2175","last_deep_scrub_stamp":"2023-10-30T13:26:20.787197+0000","last_clean_scrub_stamp":"2023-10-30T13:26:20.787197+0000","objects_scrubbed":629,"log_size":2175,"log_dups_size":0,"ondisk_log_size":2175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T18:25:46.322449+0000","scrub_duration":1.0662295260000001,"objects_trimmed":0,"snaptrim_duration":0.015331312,"stat_sum":{"num_bytes":28178512,"num_objects":629,"num_object_clones":288,"num_object_copies":1258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":629,"num_whiteouts":231,"num_read":148,"num_read_kb":41809,"num_write":1347,"num_write_kb":29520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"85'1249","reported_seq":1669,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:09.605267+0000","last_change":"2023-10-30T13:26:09.605318+0000","last_active":"2023-10-30T13:26:09.605267+0000","last_peered":"2023-10-30T13:26:09.605267+0000","last_clean":"2023-10-30T13:26:09.605267+0000","last_became_active":"2023-10-30T12:28:44.058456+0000","last_became_peered":"2023-10-30T12:28:44.058456+0000","last_unstale":"2023-10-30T13:26:09.605267+0000","last_undegraded":"2023-10-30T13:26:09.605267+0000","last_fullsized":"2023-10-30T13:26:09.605267+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1249","last_scrub_stamp":"2023-10-30T13:26:09.605188+0000","last_deep_scrub":"85'1249","last_deep_scrub_stamp":"2023-10-30T13:26:09.605188+0000","last_clean_scrub_stamp":"2023-10-30T13:26:09.605188+0000","objects_scrubbed":53,"log_size":1249,"log_dups_size":0,"ondisk_log_size":1249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:14:52.759600+0000","scrub_duration":0.149468351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777216,"num_objects":53,"num_object_clones":0,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":0,"num_read":573,"num_read_kb":334248,"num_write":1338,"num_write_kb":75895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":744856,"num_omap_keys":1335,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.3","version":"85'2208","reported_seq":4242,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:02.512036+0000","last_change":"2023-10-30T13:26:02.512109+0000","last_active":"2023-10-30T13:26:02.512036+0000","last_peered":"2023-10-30T13:26:02.512036+0000","last_clean":"2023-10-30T13:26:02.512036+0000","last_became_active":"2023-10-30T12:28:44.954190+0000","last_became_peered":"2023-10-30T12:28:44.954190+0000","last_unstale":"2023-10-30T13:26:02.512036+0000","last_undegraded":"2023-10-30T13:26:02.512036+0000","last_fullsized":"2023-10-30T13:26:02.512036+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2208","last_scrub_stamp":"2023-10-30T13:26:02.511965+0000","last_deep_scrub":"85'2208","last_deep_scrub_stamp":"2023-10-30T13:26:02.511965+0000","last_clean_scrub_stamp":"2023-10-30T13:26:02.511965+0000","objects_scrubbed":627,"log_size":2208,"log_dups_size":0,"ondisk_log_size":2208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T15:59:27.984589+0000","scrub_duration":1.373745343,"objects_trimmed":0,"snaptrim_duration":0.0051193749999999998,"stat_sum":{"num_bytes":23443395,"num_objects":627,"num_object_clones":293,"num_object_copies":1254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":236,"num_read":141,"num_read_kb":26943,"num_write":1376,"num_write_kb":26009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"85'1517","reported_seq":1988,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.476829+0000","last_change":"2023-10-30T13:25:44.476908+0000","last_active":"2023-10-30T13:25:44.476829+0000","last_peered":"2023-10-30T13:25:44.476829+0000","last_clean":"2023-10-30T13:25:44.476829+0000","last_became_active":"2023-10-30T12:28:44.070969+0000","last_became_peered":"2023-10-30T12:28:44.070969+0000","last_unstale":"2023-10-30T13:25:44.476829+0000","last_undegraded":"2023-10-30T13:25:44.476829+0000","last_fullsized":"2023-10-30T13:25:44.476829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1517","last_scrub_stamp":"2023-10-30T13:25:44.476706+0000","last_deep_scrub":"85'1517","last_deep_scrub_stamp":"2023-10-30T13:25:44.476706+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.476706+0000","objects_scrubbed":50,"log_size":1517,"log_dups_size":0,"ondisk_log_size":1517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:43:12.189799+0000","scrub_duration":0.194542509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":50,"num_object_clones":0,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":0,"num_read":699,"num_read_kb":404466,"num_write":1620,"num_write_kb":92575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":728614,"num_omap_keys":1399,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"85'2249","reported_seq":4286,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:50.476286+0000","last_change":"2023-10-30T13:25:50.476388+0000","last_active":"2023-10-30T13:25:50.476286+0000","last_peered":"2023-10-30T13:25:50.476286+0000","last_clean":"2023-10-30T13:25:50.476286+0000","last_became_active":"2023-10-30T12:28:44.973110+0000","last_became_peered":"2023-10-30T12:28:44.973110+0000","last_unstale":"2023-10-30T13:25:50.476286+0000","last_undegraded":"2023-10-30T13:25:50.476286+0000","last_fullsized":"2023-10-30T13:25:50.476286+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2249","last_scrub_stamp":"2023-10-30T13:25:50.476194+0000","last_deep_scrub":"85'2249","last_deep_scrub_stamp":"2023-10-30T13:25:50.476194+0000","last_clean_scrub_stamp":"2023-10-30T13:25:50.476194+0000","objects_scrubbed":637,"log_size":2249,"log_dups_size":0,"ondisk_log_size":2249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:30:58.348787+0000","scrub_duration":1.2919135900000001,"objects_trimmed":0,"snaptrim_duration":0.0039865179999999997,"stat_sum":{"num_bytes":21812744,"num_objects":637,"num_object_clones":298,"num_object_copies":1274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":637,"num_whiteouts":235,"num_read":149,"num_read_kb":24572,"num_write":1399,"num_write_kb":31699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.5","version":"85'1056","reported_seq":1475,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:05.605392+0000","last_change":"2023-10-30T13:26:05.605477+0000","last_active":"2023-10-30T13:26:05.605392+0000","last_peered":"2023-10-30T13:26:05.605392+0000","last_clean":"2023-10-30T13:26:05.605392+0000","last_became_active":"2023-10-30T12:28:44.062962+0000","last_became_peered":"2023-10-30T12:28:44.062962+0000","last_unstale":"2023-10-30T13:26:05.605392+0000","last_undegraded":"2023-10-30T13:26:05.605392+0000","last_fullsized":"2023-10-30T13:26:05.605392+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1056","last_scrub_stamp":"2023-10-30T13:26:05.605294+0000","last_deep_scrub":"85'1056","last_deep_scrub_stamp":"2023-10-30T13:26:05.605294+0000","last_clean_scrub_stamp":"2023-10-30T13:26:05.605294+0000","objects_scrubbed":43,"log_size":1056,"log_dups_size":0,"ondisk_log_size":1056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:08:51.834930+0000","scrub_duration":0.16079274700000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37748758,"num_objects":43,"num_object_clones":0,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":0,"num_read":534,"num_read_kb":318720,"num_write":1164,"num_write_kb":75805,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":33,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":590943,"num_omap_keys":951,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"85'2157","reported_seq":4215,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:04.198525+0000","last_change":"2023-10-30T13:26:04.198592+0000","last_active":"2023-10-30T13:26:04.198525+0000","last_peered":"2023-10-30T13:26:04.198525+0000","last_clean":"2023-10-30T13:26:04.198525+0000","last_became_active":"2023-10-30T12:28:44.974773+0000","last_became_peered":"2023-10-30T12:28:44.974773+0000","last_unstale":"2023-10-30T13:26:04.198525+0000","last_undegraded":"2023-10-30T13:26:04.198525+0000","last_fullsized":"2023-10-30T13:26:04.198525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2157","last_scrub_stamp":"2023-10-30T13:26:04.198444+0000","last_deep_scrub":"85'2157","last_deep_scrub_stamp":"2023-10-30T13:26:04.198444+0000","last_clean_scrub_stamp":"2023-10-30T13:26:04.198444+0000","objects_scrubbed":615,"log_size":2157,"log_dups_size":0,"ondisk_log_size":2157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T20:17:59.418030+0000","scrub_duration":1.2099730950000001,"objects_trimmed":0,"snaptrim_duration":0.0070372949999999998,"stat_sum":{"num_bytes":38977850,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":222,"num_read":156,"num_read_kb":47793,"num_write":1351,"num_write_kb":41194,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"85'1184","reported_seq":1628,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:50.553950+0000","last_change":"2023-10-30T13:25:50.554036+0000","last_active":"2023-10-30T13:25:50.553950+0000","last_peered":"2023-10-30T13:25:50.553950+0000","last_clean":"2023-10-30T13:25:50.553950+0000","last_became_active":"2023-10-30T12:28:44.054548+0000","last_became_peered":"2023-10-30T12:28:44.054548+0000","last_unstale":"2023-10-30T13:25:50.553950+0000","last_undegraded":"2023-10-30T13:25:50.553950+0000","last_fullsized":"2023-10-30T13:25:50.553950+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1184","last_scrub_stamp":"2023-10-30T13:25:50.553862+0000","last_deep_scrub":"85'1184","last_deep_scrub_stamp":"2023-10-30T13:25:50.553862+0000","last_clean_scrub_stamp":"2023-10-30T13:25:50.553862+0000","objects_scrubbed":47,"log_size":1184,"log_dups_size":0,"ondisk_log_size":1184,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:07:34.636484+0000","scrub_duration":0.480914603,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29360128,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":644,"num_read_kb":249397,"num_write":1860,"num_write_kb":63729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":781147,"num_omap_keys":1113,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"85'2052","reported_seq":4082,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:03.318813+0000","last_change":"2023-10-30T13:26:03.318913+0000","last_active":"2023-10-30T13:26:03.318813+0000","last_peered":"2023-10-30T13:26:03.318813+0000","last_clean":"2023-10-30T13:26:03.318813+0000","last_became_active":"2023-10-30T12:28:44.973700+0000","last_became_peered":"2023-10-30T12:28:44.973700+0000","last_unstale":"2023-10-30T13:26:03.318813+0000","last_undegraded":"2023-10-30T13:26:03.318813+0000","last_fullsized":"2023-10-30T13:26:03.318813+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2052","last_scrub_stamp":"2023-10-30T13:26:03.318707+0000","last_deep_scrub":"85'2052","last_deep_scrub_stamp":"2023-10-30T13:26:03.318707+0000","last_clean_scrub_stamp":"2023-10-30T13:26:03.318707+0000","objects_scrubbed":603,"log_size":2052,"log_dups_size":0,"ondisk_log_size":2052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:04:58.304469+0000","scrub_duration":1.115999714,"objects_trimmed":0,"snaptrim_duration":0.0074996079999999996,"stat_sum":{"num_bytes":14238343,"num_objects":603,"num_object_clones":280,"num_object_copies":1206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":603,"num_whiteouts":229,"num_read":126,"num_read_kb":9383,"num_write":1239,"num_write_kb":19514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"85'541","reported_seq":901,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:08.512407+0000","last_change":"2023-10-30T13:26:08.512451+0000","last_active":"2023-10-30T13:26:08.512407+0000","last_peered":"2023-10-30T13:26:08.512407+0000","last_clean":"2023-10-30T13:26:08.512407+0000","last_became_active":"2023-10-30T12:28:44.058358+0000","last_became_peered":"2023-10-30T12:28:44.058358+0000","last_unstale":"2023-10-30T13:26:08.512407+0000","last_undegraded":"2023-10-30T13:26:08.512407+0000","last_fullsized":"2023-10-30T13:26:08.512407+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'541","last_scrub_stamp":"2023-10-30T13:26:08.512333+0000","last_deep_scrub":"85'541","last_deep_scrub_stamp":"2023-10-30T13:26:08.512333+0000","last_clean_scrub_stamp":"2023-10-30T13:26:08.512333+0000","objects_scrubbed":51,"log_size":541,"log_dups_size":0,"ondisk_log_size":541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:40:06.032449+0000","scrub_duration":0.092969488000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21146631,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":514,"num_read_kb":197148,"num_write":651,"num_write_kb":42826,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":495891,"num_omap_keys":618,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"85'2296","reported_seq":4306,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:58.407563+0000","last_change":"2023-10-30T13:25:58.407621+0000","last_active":"2023-10-30T13:25:58.407563+0000","last_peered":"2023-10-30T13:25:58.407563+0000","last_clean":"2023-10-30T13:25:58.407563+0000","last_became_active":"2023-10-30T12:28:44.956541+0000","last_became_peered":"2023-10-30T12:28:44.956541+0000","last_unstale":"2023-10-30T13:25:58.407563+0000","last_undegraded":"2023-10-30T13:25:58.407563+0000","last_fullsized":"2023-10-30T13:25:58.407563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2296","last_scrub_stamp":"2023-10-30T13:25:58.407437+0000","last_deep_scrub":"85'2296","last_deep_scrub_stamp":"2023-10-30T13:25:58.407437+0000","last_clean_scrub_stamp":"2023-10-30T13:25:58.407437+0000","objects_scrubbed":647,"log_size":2296,"log_dups_size":0,"ondisk_log_size":2296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T17:17:12.972219+0000","scrub_duration":1.296777326,"objects_trimmed":0,"snaptrim_duration":0.0085509929999999998,"stat_sum":{"num_bytes":14746405,"num_objects":647,"num_object_clones":344,"num_object_copies":1294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":647,"num_whiteouts":203,"num_read":167,"num_read_kb":6583,"num_write":1369,"num_write_kb":17013,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'916","reported_seq":1240,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:51.295504+0000","last_change":"2023-10-30T13:25:51.295571+0000","last_active":"2023-10-30T13:25:51.295504+0000","last_peered":"2023-10-30T13:25:51.295504+0000","last_clean":"2023-10-30T13:25:51.295504+0000","last_became_active":"2023-10-30T12:28:44.053659+0000","last_became_peered":"2023-10-30T12:28:44.053659+0000","last_unstale":"2023-10-30T13:25:51.295504+0000","last_undegraded":"2023-10-30T13:25:51.295504+0000","last_fullsized":"2023-10-30T13:25:51.295504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'916","last_scrub_stamp":"2023-10-30T13:25:51.295393+0000","last_deep_scrub":"85'916","last_deep_scrub_stamp":"2023-10-30T13:25:51.295393+0000","last_clean_scrub_stamp":"2023-10-30T13:25:51.295393+0000","objects_scrubbed":46,"log_size":916,"log_dups_size":0,"ondisk_log_size":916,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:13:46.299260+0000","scrub_duration":0.1008542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12582912,"num_objects":46,"num_object_clones":0,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":472,"num_read_kb":149352,"num_write":1030,"num_write_kb":38491,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":548946,"num_omap_keys":1000,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.9","version":"85'2069","reported_seq":4117,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.461766+0000","last_change":"2023-10-30T13:25:53.461883+0000","last_active":"2023-10-30T13:25:53.461766+0000","last_peered":"2023-10-30T13:25:53.461766+0000","last_clean":"2023-10-30T13:25:53.461766+0000","last_became_active":"2023-10-30T12:28:44.965505+0000","last_became_peered":"2023-10-30T12:28:44.965505+0000","last_unstale":"2023-10-30T13:25:53.461766+0000","last_undegraded":"2023-10-30T13:25:53.461766+0000","last_fullsized":"2023-10-30T13:25:53.461766+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2069","last_scrub_stamp":"2023-10-30T13:25:53.461661+0000","last_deep_scrub":"85'2069","last_deep_scrub_stamp":"2023-10-30T13:25:53.461661+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.461661+0000","objects_scrubbed":584,"log_size":2069,"log_dups_size":0,"ondisk_log_size":2069,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:15:51.529626+0000","scrub_duration":1.0309153659999999,"objects_trimmed":0,"snaptrim_duration":0.0096369869999999996,"stat_sum":{"num_bytes":18113290,"num_objects":584,"num_object_clones":270,"num_object_copies":1168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":584,"num_whiteouts":219,"num_read":134,"num_read_kb":20299,"num_write":1266,"num_write_kb":19768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"85'826","reported_seq":1199,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:49.233063+0000","last_change":"2023-10-30T13:25:49.233269+0000","last_active":"2023-10-30T13:25:49.233063+0000","last_peered":"2023-10-30T13:25:49.233063+0000","last_clean":"2023-10-30T13:25:49.233063+0000","last_became_active":"2023-10-30T12:28:44.057572+0000","last_became_peered":"2023-10-30T12:28:44.057572+0000","last_unstale":"2023-10-30T13:25:49.233063+0000","last_undegraded":"2023-10-30T13:25:49.233063+0000","last_fullsized":"2023-10-30T13:25:49.233063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'826","last_scrub_stamp":"2023-10-30T13:25:49.232843+0000","last_deep_scrub":"85'826","last_deep_scrub_stamp":"2023-10-30T13:25:49.232843+0000","last_clean_scrub_stamp":"2023-10-30T13:25:49.232843+0000","objects_scrubbed":49,"log_size":826,"log_dups_size":0,"ondisk_log_size":826,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:29:29.495169+0000","scrub_duration":0.14346297999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":16777306,"num_objects":49,"num_object_clones":0,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":0,"num_read":495,"num_read_kb":170685,"num_write":965,"num_write_kb":43434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":641331,"num_omap_keys":1102,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"85'2182","reported_seq":4222,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:53.248973+0000","last_change":"2023-10-30T13:25:53.249016+0000","last_active":"2023-10-30T13:25:53.248973+0000","last_peered":"2023-10-30T13:25:53.248973+0000","last_clean":"2023-10-30T13:25:53.248973+0000","last_became_active":"2023-10-30T12:28:44.956513+0000","last_became_peered":"2023-10-30T12:28:44.956513+0000","last_unstale":"2023-10-30T13:25:53.248973+0000","last_undegraded":"2023-10-30T13:25:53.248973+0000","last_fullsized":"2023-10-30T13:25:53.248973+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2182","last_scrub_stamp":"2023-10-30T13:25:53.248885+0000","last_deep_scrub":"85'2182","last_deep_scrub_stamp":"2023-10-30T13:25:53.248885+0000","last_clean_scrub_stamp":"2023-10-30T13:25:53.248885+0000","objects_scrubbed":615,"log_size":2182,"log_dups_size":0,"ondisk_log_size":2182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T00:04:59.062499+0000","scrub_duration":1.1234317469999999,"objects_trimmed":0,"snaptrim_duration":0.007081922,"stat_sum":{"num_bytes":14751960,"num_objects":615,"num_object_clones":283,"num_object_copies":1230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":615,"num_whiteouts":228,"num_read":137,"num_read_kb":2516,"num_write":1376,"num_write_kb":21368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1177","reported_seq":1676,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:47.247326+0000","last_change":"2023-10-30T13:25:47.247437+0000","last_active":"2023-10-30T13:25:47.247326+0000","last_peered":"2023-10-30T13:25:47.247326+0000","last_clean":"2023-10-30T13:25:47.247326+0000","last_became_active":"2023-10-30T12:28:44.072037+0000","last_became_peered":"2023-10-30T12:28:44.072037+0000","last_unstale":"2023-10-30T13:25:47.247326+0000","last_undegraded":"2023-10-30T13:25:47.247326+0000","last_fullsized":"2023-10-30T13:25:47.247326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1177","last_scrub_stamp":"2023-10-30T13:25:47.247175+0000","last_deep_scrub":"85'1177","last_deep_scrub_stamp":"2023-10-30T13:25:47.247175+0000","last_clean_scrub_stamp":"2023-10-30T13:25:47.247175+0000","objects_scrubbed":52,"log_size":1177,"log_dups_size":0,"ondisk_log_size":1177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:31:36.256311+0000","scrub_duration":0.26537599099999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33554432,"num_objects":52,"num_object_clones":0,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":0,"num_read":763,"num_read_kb":345628,"num_write":1535,"num_write_kb":76701,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":665341,"num_omap_keys":1170,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"85'2030","reported_seq":4076,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:56.194710+0000","last_change":"2023-10-30T13:25:56.194771+0000","last_active":"2023-10-30T13:25:56.194710+0000","last_peered":"2023-10-30T13:25:56.194710+0000","last_clean":"2023-10-30T13:25:56.194710+0000","last_became_active":"2023-10-30T12:28:44.977445+0000","last_became_peered":"2023-10-30T12:28:44.977445+0000","last_unstale":"2023-10-30T13:25:56.194710+0000","last_undegraded":"2023-10-30T13:25:56.194710+0000","last_fullsized":"2023-10-30T13:25:56.194710+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2030","last_scrub_stamp":"2023-10-30T13:25:56.194609+0000","last_deep_scrub":"85'2030","last_deep_scrub_stamp":"2023-10-30T13:25:56.194609+0000","last_clean_scrub_stamp":"2023-10-30T13:25:56.194609+0000","objects_scrubbed":570,"log_size":2030,"log_dups_size":0,"ondisk_log_size":2030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:43:38.396542+0000","scrub_duration":0.95094084300000004,"objects_trimmed":0,"snaptrim_duration":0.0060337460000000004,"stat_sum":{"num_bytes":23618154,"num_objects":570,"num_object_clones":263,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":570,"num_whiteouts":209,"num_read":127,"num_read_kb":29654,"num_write":1265,"num_write_kb":27771,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.b","version":"85'1241","reported_seq":1655,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:07.560122+0000","last_change":"2023-10-30T13:26:07.560178+0000","last_active":"2023-10-30T13:26:07.560122+0000","last_peered":"2023-10-30T13:26:07.560122+0000","last_clean":"2023-10-30T13:26:07.560122+0000","last_became_active":"2023-10-30T12:28:44.057728+0000","last_became_peered":"2023-10-30T12:28:44.057728+0000","last_unstale":"2023-10-30T13:26:07.560122+0000","last_undegraded":"2023-10-30T13:26:07.560122+0000","last_fullsized":"2023-10-30T13:26:07.560122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1241","last_scrub_stamp":"2023-10-30T13:26:07.560043+0000","last_deep_scrub":"85'1241","last_deep_scrub_stamp":"2023-10-30T13:26:07.560043+0000","last_clean_scrub_stamp":"2023-10-30T13:26:07.560043+0000","objects_scrubbed":59,"log_size":1241,"log_dups_size":0,"ondisk_log_size":1241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:25:34.609715+0000","scrub_duration":0.163669707,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41943040,"num_objects":59,"num_object_clones":0,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":0,"num_read":538,"num_read_kb":321476,"num_write":1330,"num_write_kb":79987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":46,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":506551,"num_omap_keys":794,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.a","version":"85'2158","reported_seq":4196,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:19.353150+0000","last_change":"2023-10-30T13:26:19.353226+0000","last_active":"2023-10-30T13:26:19.353150+0000","last_peered":"2023-10-30T13:26:19.353150+0000","last_clean":"2023-10-30T13:26:19.353150+0000","last_became_active":"2023-10-30T12:28:44.942985+0000","last_became_peered":"2023-10-30T12:28:44.942985+0000","last_unstale":"2023-10-30T13:26:19.353150+0000","last_undegraded":"2023-10-30T13:26:19.353150+0000","last_fullsized":"2023-10-30T13:26:19.353150+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2158","last_scrub_stamp":"2023-10-30T13:26:19.353076+0000","last_deep_scrub":"85'2158","last_deep_scrub_stamp":"2023-10-30T13:26:19.353076+0000","last_clean_scrub_stamp":"2023-10-30T13:26:19.353076+0000","objects_scrubbed":612,"log_size":2158,"log_dups_size":0,"ondisk_log_size":2158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T13:55:44.600734+0000","scrub_duration":0.83190967000000005,"objects_trimmed":0,"snaptrim_duration":0.0091395910000000007,"stat_sum":{"num_bytes":24394000,"num_objects":612,"num_object_clones":279,"num_object_copies":1224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":612,"num_whiteouts":219,"num_read":133,"num_read_kb":28881,"num_write":1364,"num_write_kb":25892,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"85'2109","reported_seq":4166,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:38.326148+0000","last_change":"2023-10-30T13:26:38.326202+0000","last_active":"2023-10-30T13:26:38.326148+0000","last_peered":"2023-10-30T13:26:38.326148+0000","last_clean":"2023-10-30T13:26:38.326148+0000","last_became_active":"2023-10-30T12:28:44.958220+0000","last_became_peered":"2023-10-30T12:28:44.958220+0000","last_unstale":"2023-10-30T13:26:38.326148+0000","last_undegraded":"2023-10-30T13:26:38.326148+0000","last_fullsized":"2023-10-30T13:26:38.326148+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2109","last_scrub_stamp":"2023-10-30T13:26:38.326074+0000","last_deep_scrub":"85'2109","last_deep_scrub_stamp":"2023-10-30T13:26:38.326074+0000","last_clean_scrub_stamp":"2023-10-30T13:26:38.326074+0000","objects_scrubbed":582,"log_size":2109,"log_dups_size":0,"ondisk_log_size":2109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T14:29:32.668066+0000","scrub_duration":0.71843828300000001,"objects_trimmed":0,"snaptrim_duration":0.010120377999999999,"stat_sum":{"num_bytes":19538275,"num_objects":582,"num_object_clones":275,"num_object_copies":1164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":582,"num_whiteouts":200,"num_read":151,"num_read_kb":26132,"num_write":1354,"num_write_kb":27822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"85'931","reported_seq":1390,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:44.318568+0000","last_change":"2023-10-30T13:25:44.318568+0000","last_active":"2023-10-30T13:25:44.318568+0000","last_peered":"2023-10-30T13:25:44.318568+0000","last_clean":"2023-10-30T13:25:44.318568+0000","last_became_active":"2023-10-30T12:28:44.054349+0000","last_became_peered":"2023-10-30T12:28:44.054349+0000","last_unstale":"2023-10-30T13:25:44.318568+0000","last_undegraded":"2023-10-30T13:25:44.318568+0000","last_fullsized":"2023-10-30T13:25:44.318568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'931","last_scrub_stamp":"2023-10-30T13:25:44.318392+0000","last_deep_scrub":"85'931","last_deep_scrub_stamp":"2023-10-30T13:25:44.318392+0000","last_clean_scrub_stamp":"2023-10-30T13:25:44.318392+0000","objects_scrubbed":51,"log_size":931,"log_dups_size":0,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T21:56:06.786185+0000","scrub_duration":0.105541015,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8388608,"num_objects":51,"num_object_clones":0,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":0,"num_read":739,"num_read_kb":197443,"num_write":1077,"num_write_kb":51357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":45,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":687759,"num_omap_keys":1211,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"85'2328","reported_seq":4360,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:02.827670+0000","last_change":"2023-10-30T13:26:02.827670+0000","last_active":"2023-10-30T13:26:02.827670+0000","last_peered":"2023-10-30T13:26:02.827670+0000","last_clean":"2023-10-30T13:26:02.827670+0000","last_became_active":"2023-10-30T12:28:44.957984+0000","last_became_peered":"2023-10-30T12:28:44.957984+0000","last_unstale":"2023-10-30T13:26:02.827670+0000","last_undegraded":"2023-10-30T13:26:02.827670+0000","last_fullsized":"2023-10-30T13:26:02.827670+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2328","last_scrub_stamp":"2023-10-30T13:26:02.827523+0000","last_deep_scrub":"85'2328","last_deep_scrub_stamp":"2023-10-30T13:26:02.827523+0000","last_clean_scrub_stamp":"2023-10-30T13:26:02.827523+0000","objects_scrubbed":686,"log_size":2328,"log_dups_size":0,"ondisk_log_size":2328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:54:56.339329+0000","scrub_duration":1.3553331850000001,"objects_trimmed":0,"snaptrim_duration":0.0011628750000000001,"stat_sum":{"num_bytes":22561655,"num_objects":686,"num_object_clones":314,"num_object_copies":1372,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":686,"num_whiteouts":247,"num_read":162,"num_read_kb":29557,"num_write":1441,"num_write_kb":28808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1d","version":"85'1103","reported_seq":1655,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:55.242833+0000","last_change":"2023-10-30T13:25:55.242833+0000","last_active":"2023-10-30T13:25:55.242833+0000","last_peered":"2023-10-30T13:25:55.242833+0000","last_clean":"2023-10-30T13:25:55.242833+0000","last_became_active":"2023-10-30T12:28:44.052258+0000","last_became_peered":"2023-10-30T12:28:44.052258+0000","last_unstale":"2023-10-30T13:25:55.242833+0000","last_undegraded":"2023-10-30T13:25:55.242833+0000","last_fullsized":"2023-10-30T13:25:55.242833+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'1103","last_scrub_stamp":"2023-10-30T13:25:55.242662+0000","last_deep_scrub":"85'1103","last_deep_scrub_stamp":"2023-10-30T13:25:55.242662+0000","last_clean_scrub_stamp":"2023-10-30T13:25:55.242662+0000","objects_scrubbed":55,"log_size":1103,"log_dups_size":0,"ondisk_log_size":1103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T16:05:46.724096+0000","scrub_duration":0.11943021299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25165824,"num_objects":55,"num_object_clones":0,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":0,"num_read":939,"num_read_kb":433966,"num_write":1214,"num_write_kb":88022,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":602554,"num_omap_keys":1058,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1f","version":"85'2157","reported_seq":4173,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:23.727155+0000","last_change":"2023-10-30T13:26:23.727243+0000","last_active":"2023-10-30T13:26:23.727155+0000","last_peered":"2023-10-30T13:26:23.727155+0000","last_clean":"2023-10-30T13:26:23.727155+0000","last_became_active":"2023-10-30T12:28:44.972999+0000","last_became_peered":"2023-10-30T12:28:44.972999+0000","last_unstale":"2023-10-30T13:26:23.727155+0000","last_undegraded":"2023-10-30T13:26:23.727155+0000","last_fullsized":"2023-10-30T13:26:23.727155+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2157","last_scrub_stamp":"2023-10-30T13:26:23.727045+0000","last_deep_scrub":"85'2157","last_deep_scrub_stamp":"2023-10-30T13:26:23.727045+0000","last_clean_scrub_stamp":"2023-10-30T13:26:23.727045+0000","objects_scrubbed":626,"log_size":2157,"log_dups_size":0,"ondisk_log_size":2157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-11-01T01:22:59.201600+0000","scrub_duration":1.0286007049999999,"objects_trimmed":0,"snaptrim_duration":0.014632805,"stat_sum":{"num_bytes":14611265,"num_objects":626,"num_object_clones":291,"num_object_copies":1252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":626,"num_whiteouts":226,"num_read":123,"num_read_kb":17907,"num_write":1326,"num_write_kb":16683,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"85'807","reported_seq":1196,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:50.519195+0000","last_change":"2023-10-30T13:25:50.519195+0000","last_active":"2023-10-30T13:25:50.519195+0000","last_peered":"2023-10-30T13:25:50.519195+0000","last_clean":"2023-10-30T13:25:50.519195+0000","last_became_active":"2023-10-30T12:28:44.074127+0000","last_became_peered":"2023-10-30T12:28:44.074127+0000","last_unstale":"2023-10-30T13:25:50.519195+0000","last_undegraded":"2023-10-30T13:25:50.519195+0000","last_fullsized":"2023-10-30T13:25:50.519195+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'807","last_scrub_stamp":"2023-10-30T13:25:50.519025+0000","last_deep_scrub":"85'807","last_deep_scrub_stamp":"2023-10-30T13:25:50.519025+0000","last_clean_scrub_stamp":"2023-10-30T13:25:50.519025+0000","objects_scrubbed":54,"log_size":807,"log_dups_size":0,"ondisk_log_size":807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T23:52:13.369063+0000","scrub_duration":0.17624699299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137344,"num_objects":54,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":488,"num_read_kb":379559,"num_write":896,"num_write_kb":83880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":513377,"num_omap_keys":969,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1e","version":"85'2062","reported_seq":4111,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:26:16.979624+0000","last_change":"2023-10-30T13:26:16.979757+0000","last_active":"2023-10-30T13:26:16.979624+0000","last_peered":"2023-10-30T13:26:16.979624+0000","last_clean":"2023-10-30T13:26:16.979624+0000","last_became_active":"2023-10-30T12:28:44.973539+0000","last_became_peered":"2023-10-30T12:28:44.973539+0000","last_unstale":"2023-10-30T13:26:16.979624+0000","last_undegraded":"2023-10-30T13:26:16.979624+0000","last_fullsized":"2023-10-30T13:26:16.979624+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'2062","last_scrub_stamp":"2023-10-30T13:26:16.979419+0000","last_deep_scrub":"85'2062","last_deep_scrub_stamp":"2023-10-30T13:26:16.979419+0000","last_clean_scrub_stamp":"2023-10-30T13:26:16.979419+0000","objects_scrubbed":581,"log_size":2062,"log_dups_size":0,"ondisk_log_size":2062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T19:26:59.407523+0000","scrub_duration":1.2670243729999999,"objects_trimmed":0,"snaptrim_duration":0.015871881000000001,"stat_sum":{"num_bytes":15563160,"num_objects":581,"num_object_clones":268,"num_object_copies":1162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":581,"num_whiteouts":208,"num_read":133,"num_read_kb":9744,"num_write":1268,"num_write_kb":20904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1f","version":"85'885","reported_seq":1253,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-10-30T13:25:51.286953+0000","last_change":"2023-10-30T13:25:51.287081+0000","last_active":"2023-10-30T13:25:51.286953+0000","last_peered":"2023-10-30T13:25:51.286953+0000","last_clean":"2023-10-30T13:25:51.286953+0000","last_became_active":"2023-10-30T12:28:44.064610+0000","last_became_peered":"2023-10-30T12:28:44.064610+0000","last_unstale":"2023-10-30T13:25:51.286953+0000","last_undegraded":"2023-10-30T13:25:51.286953+0000","last_fullsized":"2023-10-30T13:25:51.286953+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'885","last_scrub_stamp":"2023-10-30T13:25:51.286817+0000","last_deep_scrub":"85'885","last_deep_scrub_stamp":"2023-10-30T13:25:51.286817+0000","last_clean_scrub_stamp":"2023-10-30T13:25:51.286817+0000","objects_scrubbed":47,"log_size":885,"log_dups_size":0,"ondisk_log_size":885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-31T22:41:21.615461+0000","scrub_duration":0.19579813099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33555014,"num_objects":47,"num_object_clones":0,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":517,"num_read_kb":175570,"num_write":978,"num_write_kb":51325,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":37,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":553400,"num_omap_keys":904,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":1165214918,"num_objects":39301,"num_object_clones":18317,"num_object_copies":78602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39301,"num_whiteouts":14249,"num_read":9267,"num_read_kb":1285569,"num_write":86732,"num_write_kb":1459764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2424127488,"data_stored":2332445068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138818,"ondisk_log_size":138818,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":903126807,"num_objects":1657,"num_object_clones":0,"num_object_copies":3314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1657,"num_whiteouts":0,"num_read":21550,"num_read_kb":9552846,"num_write":41306,"num_write_kb":2210416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1366,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":19724250,"num_omap_keys":33398,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1806409728,"data_stored":1806253614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36411,"ondisk_log_size":36411,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":16,"seq":68719477443,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":865084,"kb_used_data":639604,"kb_used_omap":1,"kb_used_meta":225470,"kb_avail":93506756,"statfs":{"total":96636764160,"available":95750918144,"internally_reserved":0,"allocated":654954496,"data_stored":640799640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":230881736},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.41799999999999998,"15min":0.436},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.63200000000000001,"5min":0.66900000000000004,"15min":0.94599999999999995},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.441,"5min":0.46500000000000002,"15min":0.47399999999999998},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.27700000000000002},"max":{"1min":0.63900000000000001,"5min":0.75800000000000001,"15min":0.81899999999999995},"last":0.50900000000000001}]},{"osd":1,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.49199999999999999,"15min":0.496},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.58299999999999996,"5min":0.86099999999999999,"15min":1.0209999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.50600000000000001,"15min":0.52700000000000002},"min":{"1min":0.33000000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.60599999999999998,"5min":0.76600000000000001,"15min":0.99299999999999999},"last":0.73099999999999998}]},{"osd":2,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.501,"15min":0.504},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.28599999999999998},"max":{"1min":0.56399999999999995,"5min":0.85099999999999998,"15min":0.90800000000000003},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.55100000000000005,"15min":0.55400000000000005},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.74399999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.68500000000000005}]},{"osd":4,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.75700000000000001,"15min":0.78600000000000003},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.008,"5min":1.05,"15min":1.548},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.80400000000000005,"15min":0.81100000000000005},"min":{"1min":0.56699999999999995,"5min":0.497,"15min":0.497},"max":{"1min":1.1200000000000001,"5min":1.139,"15min":1.6950000000000001},"last":1.0449999999999999}]},{"osd":5,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.76700000000000002,"15min":0.80700000000000005},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.95899999999999996,"5min":1.1160000000000001,"15min":1.194},"last":0.995},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.77600000000000002,"15min":0.80600000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.157,"5min":1.1659999999999999,"15min":1.603},"last":1.107}]},{"osd":6,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.85599999999999998,"15min":0.86399999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.436},"max":{"1min":1.081,"5min":1.198,"15min":1.2270000000000001},"last":1.22},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.91800000000000004,"15min":0.92300000000000004},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0329999999999999,"5min":1.1639999999999999,"15min":1.3759999999999999},"last":1.2529999999999999}]},{"osd":7,"last update":"Mon Oct 30 13:26:15 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.84099999999999997,"15min":0.84199999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55300000000000005},"max":{"1min":1.137,"5min":1.137,"15min":1.1919999999999999},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.79100000000000004,"5min":0.89400000000000002,"15min":0.88},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.47699999999999998},"max":{"1min":0.96599999999999997,"5min":1.1990000000000001,"15min":1.2050000000000001},"last":1.143}]}]},{"osd":2,"up_from":16,"seq":68719477444,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":552124,"kb_used_data":412724,"kb_used_omap":1,"kb_used_meta":139390,"kb_avail":93819716,"statfs":{"total":96636764160,"available":96071389184,"internally_reserved":0,"allocated":422629376,"data_stored":409295067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":142735814},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:26:29 2023","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.437,"15min":0.436},"min":{"1min":0.254,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.47799999999999998,"15min":0.47199999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.66900000000000004,"5min":1.554,"15min":1.554},"last":0.435}]},{"osd":1,"last update":"Mon Oct 30 13:26:29 2023","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47599999999999998,"15min":0.48199999999999998},"min":{"1min":0.311,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.65500000000000003,"5min":1.071,"15min":1.071},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.52600000000000002,"15min":0.52900000000000003},"min":{"1min":0.39200000000000002,"5min":0.32100000000000001,"15min":0.28399999999999997},"max":{"1min":0.74099999999999999,"5min":1.198,"15min":1.198},"last":0.51300000000000001}]},{"osd":3,"last update":"Mon Oct 30 13:26:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.50900000000000001,"15min":0.51300000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.314},"max":{"1min":0.80300000000000005,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56200000000000006,"15min":0.57199999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.379},"max":{"1min":0.83699999999999997,"5min":1.4430000000000001,"15min":1.4430000000000001},"last":0.59499999999999997}]},{"osd":4,"last update":"Mon Oct 30 13:26:29 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.78600000000000003,"15min":0.79100000000000004},"min":{"1min":0.59199999999999997,"5min":0.48299999999999998,"15min":0.47999999999999998},"max":{"1min":1.083,"5min":1.1719999999999999,"15min":1.224},"last":0.746},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.752,"15min":0.78500000000000003},"min":{"1min":0.54900000000000004,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.0009999999999999,"5min":1.262,"15min":1.262},"last":0.93400000000000005}]},{"osd":5,"last update":"Mon Oct 30 13:26:29 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.86699999999999999,"15min":0.84699999999999998},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.014,"5min":1.494,"15min":1.494},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.83499999999999996,"15min":0.86699999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.0720000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.97299999999999998}]},{"osd":6,"last update":"Mon Oct 30 13:26:29 2023","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":0.81999999999999995,"15min":0.84299999999999997},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.238,"5min":1.238,"15min":1.294},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.82699999999999996,"15min":0.81699999999999995},"min":{"1min":0.54800000000000004,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0620000000000001,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.79000000000000004}]},{"osd":7,"last update":"Mon Oct 30 13:26:29 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84299999999999997,"15min":0.84399999999999997},"min":{"1min":0.63300000000000001,"5min":0.60699999999999998,"15min":0.51100000000000001},"max":{"1min":1.107,"5min":1.9139999999999999,"15min":1.9139999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.88300000000000001,"15min":0.88900000000000001},"min":{"1min":0.69099999999999995,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.212,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.95299999999999996}]}]},{"osd":1,"up_from":16,"seq":68719477443,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":583400,"kb_used_data":431520,"kb_used_omap":1,"kb_used_meta":151870,"kb_avail":93788440,"statfs":{"total":96636764160,"available":96039362560,"internally_reserved":0,"allocated":441876480,"data_stored":431229668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":155515334},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46100000000000002,"15min":0.45300000000000001},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.64900000000000002,"5min":0.70699999999999996,"15min":0.80400000000000005},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.53500000000000003,"15min":0.53100000000000003},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.65800000000000003,"5min":0.91100000000000003,"15min":0.92300000000000004},"last":0.61699999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48399999999999999,"15min":0.47199999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.76700000000000002,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56399999999999995,"15min":0.55200000000000005},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.78600000000000003,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.48199999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.47799999999999998,"15min":0.46999999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67300000000000004,"5min":0.80900000000000005,"15min":0.84999999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.55000000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.34999999999999998},"max":{"1min":0.75700000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.51100000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.81399999999999995,"15min":0.80400000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.073,"5min":1.133,"15min":1.133},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.877,"15min":0.84999999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1080000000000001,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.89700000000000002}]},{"osd":5,"last update":"Mon Oct 30 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.82199999999999995,"15min":0.81899999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.038,"5min":1.212,"15min":1.212},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.89300000000000002,"15min":0.89400000000000002},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.089,"5min":1.2350000000000001,"15min":1.2350000000000001},"last":0.78800000000000003}]},{"osd":6,"last update":"Mon Oct 30 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.79000000000000004,"15min":0.78100000000000003},"min":{"1min":0.56499999999999995,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.046,"5min":1.177,"15min":1.23},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.80900000000000005,"15min":0.82299999999999995},"min":{"1min":0.54500000000000004,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0669999999999999,"5min":1.1140000000000001,"15min":1.5449999999999999},"last":1.016}]},{"osd":7,"last update":"Mon Oct 30 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"min":{"1min":0.58799999999999997,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.105,"5min":1.2050000000000001,"15min":1.206},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.85599999999999998,"15min":0.86299999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.94899999999999995,"5min":1.1870000000000001,"15min":2.8359999999999999},"last":0.67400000000000004}]}]},{"osd":0,"up_from":16,"seq":68719477444,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":947656,"kb_used_data":695808,"kb_used_omap":1,"kb_used_meta":251838,"kb_avail":93424184,"statfs":{"total":96636764160,"available":95666364416,"internally_reserved":0,"allocated":712507392,"data_stored":696810667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":257882566},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Oct 30 13:26:30 2023","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.39900000000000002,"15min":0.40300000000000002},"min":{"1min":0.29199999999999998,"5min":0.23899999999999999,"15min":0.22700000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.50700000000000001,"15min":0.51600000000000001},"min":{"1min":0.40600000000000003,"5min":0.36199999999999999,"15min":0.309},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.57899999999999996}]},{"osd":2,"last update":"Mon Oct 30 13:26:30 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.42699999999999999,"15min":0.433},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.27700000000000002},"max":{"1min":0.58899999999999997,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.51700000000000002,"15min":0.51400000000000001},"min":{"1min":0.371,"5min":0.36099999999999999,"15min":0.30199999999999999},"max":{"1min":0.82799999999999996,"5min":0.878,"15min":0.878},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Oct 30 13:26:30 2023","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.46899999999999997,"15min":0.47099999999999997},"min":{"1min":0.309,"5min":0.29299999999999998,"15min":0.28299999999999997},"max":{"1min":0.59899999999999998,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.442},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.38600000000000001},"max":{"1min":0.84099999999999997,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.59499999999999997}]},{"osd":4,"last update":"Mon Oct 30 13:26:30 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.78000000000000003,"15min":0.78800000000000003},"min":{"1min":0.48699999999999999,"5min":0.46700000000000003,"15min":0.377},"max":{"1min":1.139,"5min":1.139,"15min":1.2609999999999999},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.77600000000000002,"15min":0.77800000000000002},"min":{"1min":0.55200000000000005,"5min":0.55000000000000004,"15min":0.48099999999999998},"max":{"1min":1.153,"5min":1.153,"15min":1.8420000000000001},"last":0.90600000000000003}]},{"osd":5,"last update":"Mon Oct 30 13:26:30 2023","interfaces":[{"interface":"back","average":{"1min":0.71799999999999997,"5min":0.80600000000000005,"15min":0.81200000000000006},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.91100000000000003,"5min":1.246,"15min":1.3340000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.76600000000000001,"5min":0.85499999999999998,"15min":0.84499999999999997},"min":{"1min":0.504,"5min":0.504,"15min":0.47099999999999997},"max":{"1min":1.0529999999999999,"5min":1.204,"15min":1.762},"last":0.96599999999999997}]},{"osd":6,"last update":"Mon Oct 30 13:26:30 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.80600000000000005,"15min":0.82099999999999995},"min":{"1min":0.59899999999999998,"5min":0.57599999999999996,"15min":0.495},"max":{"1min":0.95199999999999996,"5min":1.1319999999999999,"15min":1.446},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.746,"5min":0.85099999999999998,"15min":0.875},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.56699999999999995},"max":{"1min":1.1060000000000001,"5min":1.1699999999999999,"15min":1.8169999999999999},"last":0.754}]},{"osd":7,"last update":"Mon Oct 30 13:26:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.85299999999999998,"15min":0.84999999999999998},"min":{"1min":0.60999999999999999,"5min":0.56299999999999994,"15min":0.55700000000000005},"max":{"1min":1.012,"5min":1.206,"15min":1.7949999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.85899999999999999,"15min":0.88900000000000001},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.121,"5min":1.121,"15min":1.274},"last":1}]}]},{"osd":6,"up_from":15,"seq":64424510146,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":556972,"kb_used_data":409252,"kb_used_omap":1,"kb_used_meta":147710,"kb_avail":93814868,"statfs":{"total":96636764160,"available":96066424832,"internally_reserved":0,"allocated":419074048,"data_stored":408471323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":151255495},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.83199999999999996,"15min":0.85699999999999998},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.1359999999999999,"5min":1.2470000000000001,"15min":1.4730000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.82099999999999995,"15min":0.93799999999999994},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.2120000000000002,"5min":2.2120000000000002,"15min":2.2120000000000002},"last":0.77700000000000002}]},{"osd":1,"last update":"Mon Oct 30 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.78000000000000003,"15min":0.79400000000000004},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":1.175,"5min":1.262,"15min":1.399},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.85199999999999998,"15min":0.85599999999999998},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.125,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":1.034}]},{"osd":2,"last update":"Mon Oct 30 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.97799999999999998,"15min":0.97499999999999998},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.3280000000000001,"5min":1.496,"15min":1.496},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.89100000000000001,"15min":0.93200000000000005},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.246,"5min":1.3839999999999999,"15min":1.431},"last":0.98099999999999998}]},{"osd":3,"last update":"Mon Oct 30 13:26:47 2023","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.878,"15min":0.88400000000000001},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.222,"5min":1.2589999999999999,"15min":1.4530000000000001},"last":0.876},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.91500000000000004,"15min":0.89600000000000002},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.089,"5min":1.347,"15min":1.478},"last":1.149}]},{"osd":4,"last update":"Mon Oct 30 13:26:19 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.97699999999999998,"15min":0.93999999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0860000000000001,"15min":1.0509999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.484,"5min":1.546,"15min":1.546},"last":1.1990000000000001}]},{"osd":5,"last update":"Mon Oct 30 13:26:19 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1499999999999999,"15min":1.123},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.5640000000000001,"5min":1.5640000000000001,"15min":1.583},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0580000000000001,"15min":1.091},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.9410000000000001,"5min":1.9410000000000001,"15min":1.9410000000000001},"last":1.2410000000000001}]},{"osd":7,"last update":"Mon Oct 30 13:26:19 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.095,"15min":1.077},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.268},{"interface":"front","average":{"1min":1.212,"5min":1.2210000000000001,"15min":1.2110000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.4060000000000001,"5min":2.4060000000000001,"15min":2.4060000000000001},"last":1.3129999999999999}]}]},{"osd":4,"up_from":15,"seq":64424510148,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":673568,"kb_used_data":493912,"kb_used_omap":1,"kb_used_meta":179646,"kb_avail":93698272,"statfs":{"total":96636764160,"available":95947030528,"internally_reserved":0,"allocated":505765888,"data_stored":496228056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":183957959},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:26:22 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.82999999999999996,"15min":0.83699999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":1.9079999999999999,"5min":1.9079999999999999,"15min":1.9079999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.83199999999999996,"15min":0.84599999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.4570000000000001,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":1.141}]},{"osd":1,"last update":"Mon Oct 30 13:26:22 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.78100000000000003,"15min":0.78600000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.732,"5min":1.732,"15min":1.732},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.82599999999999996,"15min":0.83199999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.8220000000000001},"last":0.88100000000000001}]},{"osd":2,"last update":"Mon Oct 30 13:26:22 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.85299999999999998,"15min":0.89400000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.76800000000000002,"5min":0.91200000000000003,"15min":0.93200000000000005},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.3540000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.74299999999999999}]},{"osd":3,"last update":"Mon Oct 30 13:26:22 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.97299999999999998,"15min":0.95599999999999996},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.94199999999999995,"15min":0.95199999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":2.5209999999999999,"5min":2.5209999999999999,"15min":2.5209999999999999},"last":1.21}]},{"osd":5,"last update":"Mon Oct 30 13:26:22 2023","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.98199999999999998,"15min":0.99199999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.4870000000000001,"5min":1.4870000000000001,"15min":1.766},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.998,"5min":1.089,"15min":1.0580000000000001},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.534,"5min":1.5529999999999999,"15min":1.6100000000000001},"last":1.089}]},{"osd":6,"last update":"Mon Oct 30 13:26:22 2023","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.127,"15min":1.143},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.7509999999999999},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.2609999999999999,"15min":1.1859999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":1.9610000000000001},"last":1.258}]},{"osd":7,"last update":"Mon Oct 30 13:26:22 2023","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.044,"15min":1.1220000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":1.292},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.196,"15min":1.256},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":2.6000000000000001,"5min":2.6000000000000001,"15min":2.6000000000000001},"last":1.3500000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542849,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":564588,"kb_used_data":416932,"kb_used_omap":1,"kb_used_meta":147646,"kb_avail":93807252,"statfs":{"total":96636764160,"available":96058626048,"internally_reserved":0,"allocated":426938368,"data_stored":418861204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":151189963},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.82899999999999996,"15min":0.83099999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":1.1060000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.85099999999999998,"15min":0.86899999999999999},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.1120000000000001,"5min":1.333,"15min":1.3919999999999999},"last":0.73099999999999998}]},{"osd":1,"last update":"Mon Oct 30 13:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.76900000000000002,"15min":0.85799999999999998},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0229999999999999,"5min":1.244,"15min":1.4790000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.81899999999999995,"5min":0.90700000000000003,"15min":0.83299999999999996},"min":{"1min":0.5,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0329999999999999,"5min":1.524,"15min":1.524},"last":1.147}]},{"osd":2,"last update":"Mon Oct 30 13:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.032,"15min":1.0269999999999999},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":1.333,"5min":1.833,"15min":1.833},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1699999999999999,"15min":1.153},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.4079999999999999,"5min":1.8300000000000001,"15min":1.8300000000000001},"last":0.92600000000000005}]},{"osd":3,"last update":"Mon Oct 30 13:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.0680000000000001,"15min":1.077},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":1.446,"5min":1.673,"15min":1.673},"last":1.099},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.2250000000000001,"15min":1.2150000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.526,"5min":1.8109999999999999,"15min":1.8480000000000001},"last":1.04}]},{"osd":4,"last update":"Mon Oct 30 13:26:21 2023","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.84499999999999997,"15min":0.84299999999999997},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.97899999999999998,"5min":1.226,"15min":1.4970000000000001},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.93999999999999995,"15min":0.94799999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.073,"5min":1.4970000000000001,"15min":1.77},"last":0.999}]},{"osd":5,"last update":"Mon Oct 30 13:26:25 2023","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.93200000000000005,"15min":0.96799999999999997},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.3560000000000001,"5min":1.3799999999999999,"15min":2},"last":0.95799999999999996},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.117,"15min":1.042},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.4850000000000001,"5min":1.7150000000000001,"15min":2.056},"last":1.121}]},{"osd":6,"last update":"Mon Oct 30 13:26:21 2023","interfaces":[{"interface":"back","average":{"1min":1.0820000000000001,"5min":1.0549999999999999,"15min":1.01},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.4490000000000001,"5min":1.536,"15min":1.571},"last":1.069},{"interface":"front","average":{"1min":1.153,"5min":1.1499999999999999,"15min":1.1060000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":1.175}]}]},{"osd":5,"up_from":14,"seq":60129542848,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":824416,"kb_used_data":640600,"kb_used_omap":1,"kb_used_meta":183806,"kb_avail":93547424,"statfs":{"total":96636764160,"available":95792562176,"internally_reserved":0,"allocated":655974400,"data_stored":642425281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":188217798},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Oct 30 13:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.76500000000000001,"15min":0.80500000000000005},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.94599999999999995,"5min":1.276,"15min":1.343},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.77300000000000002,"15min":0.85999999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.0840000000000001,"5min":1.262,"15min":1.627},"last":0.85899999999999999}]},{"osd":1,"last update":"Mon Oct 30 13:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.77300000000000002,"15min":0.81999999999999995},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.78800000000000003,"5min":1.22,"15min":1.3220000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.85499999999999998,"15min":0.91000000000000003},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1719999999999999,"5min":1.274,"15min":1.389},"last":1.3049999999999999}]},{"osd":2,"last update":"Mon Oct 30 13:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.82699999999999996,"15min":0.84699999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.0620000000000001,"5min":1.3220000000000001,"15min":1.486},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.90200000000000002,"15min":0.877},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.879},"last":0.91100000000000003}]},{"osd":3,"last update":"Mon Oct 30 13:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.97299999999999998,"15min":0.89200000000000002},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.3480000000000001,"5min":1.361,"15min":2.097},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.83199999999999996,"5min":1.02,"15min":0.98899999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.1479999999999999,"5min":1.5169999999999999,"15min":1.9650000000000001},"last":1.2110000000000001}]},{"osd":4,"last update":"Mon Oct 30 13:26:24 2023","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.96199999999999997,"15min":0.97799999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6719999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.95799999999999996,"5min":1.016,"15min":1.081},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.1819999999999999}]},{"osd":6,"last update":"Mon Oct 30 13:26:24 2023","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.117,"15min":1.091},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.944,"5min":1.944,"15min":2.1509999999999998},"last":1.268},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.1120000000000001,"15min":1.1120000000000001},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.4850000000000001,"5min":1.625,"15min":1.821},"last":1.246}]},{"osd":7,"last update":"Mon Oct 30 13:26:24 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.177,"15min":1.129},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.875,"5min":1.875,"15min":3.3069999999999999},"last":1.3400000000000001},{"interface":"front","average":{"1min":1.052,"5min":1.137,"15min":1.113},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.5569999999999999,"5min":1.754,"15min":2.036},"last":1.0649999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":309846016,"data_stored":309823069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":142618624,"data_stored":142610667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":117452800,"data_stored":117441250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":294043648,"data_stored":294028670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":275668992,"data_stored":275628631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":300916736,"data_stored":300897937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":154218496,"data_stored":154207037,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":211644416,"data_stored":211616353,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":401661952,"data_stored":386457412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":297664512,"data_stored":287498447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":304177152,"data_stored":291323631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":359911424,"data_stored":346240784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":229097472,"data_stored":220069239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":353464320,"data_stored":340406790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":263856128,"data_stored":253734100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":214294528,"data_stored":206714665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-30T13:27:04.577 DEBUG:teuthology.orchestra.run.smithi125:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-30T13:27:04.749 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.744+0000 7fdb4070e640 1 Processor -- start 2023-10-30T13:27:04.750 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.744+0000 7fdb4070e640 1 -- start start 2023-10-30T13:27:04.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.744+0000 7fdb4070e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb3806a980 0x7fdb38162f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.744+0000 7fdb4070e640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb380c6cb0 con 0x7fdb3806f880 2023-10-30T13:27:04.751 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.744+0000 7fdb4070e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb380c6620 con 0x7fdb380c5810 2023-10-30T13:27:04.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.744+0000 7fdb4070e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb38071d90 con 0x7fdb3806a980 2023-10-30T13:27:04.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.744+0000 7fdb3dca9640 1 --1- >> v1:172.21.15.125:6789/0 conn(0x7fdb3806f880 0x7fdb3806a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.125:6789/0 says I am v1:172.21.15.125:37546/0 (socket says 172.21.15.125:37546) 2023-10-30T13:27:04.752 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.744+0000 7fdb3dca9640 1 -- 172.21.15.125:0/1516147869 learned_addr learned my addr 172.21.15.125:0/1516147869 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:27:04.753 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.744+0000 7fdb3ecab640 1 --2- 172.21.15.125:0/1516147869 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb3806a980 0x7fdb38162f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.754 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.744+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431454919 0 0) 0x7fdb380c6cb0 con 0x7fdb3806f880 2023-10-30T13:27:04.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.744+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 --> v1:172.21.15.125:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb24003530 con 0x7fdb3806f880 2023-10-30T13:27:04.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864468086 0 0) 0x7fdb380c6620 con 0x7fdb380c5810 2023-10-30T13:27:04.755 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb380c6cb0 con 0x7fdb380c5810 2023-10-30T13:27:04.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 <== mon.0 v1:172.21.15.125:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1713617395 0 0) 0x7fdb24003530 con 0x7fdb3806f880 2023-10-30T13:27:04.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb3806a980 msgr2=0x7fdb38162f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:04.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 --2- 172.21.15.125:0/1516147869 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb3806a980 0x7fdb38162f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.756 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 >> v1:172.21.15.136:6789/0 conn(0x7fdb380c5810 legacy=0x7fdb380c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 --> v1:172.21.15.125:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb381582f0 con 0x7fdb3806f880 2023-10-30T13:27:04.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 <== mon.0 v1:172.21.15.125:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1009937980 0 0) 0x7fdb28001680 con 0x7fdb3806f880 2023-10-30T13:27:04.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 >> v1:172.21.15.125:6789/0 conn(0x7fdb3806f880 legacy=0x7fdb3806a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 --2- 172.21.15.125:0/1516147869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb24003b40 0x7fdb24003f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.757 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 --2- 172.21.15.125:0/1516147869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb240047d0 0x7fdb24004c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 --2- 172.21.15.125:0/1516147869 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb3806a980 0x7fdb24005240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2400bb20 con 0x7fdb240047d0 2023-10-30T13:27:04.758 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2400bc90 con 0x7fdb24003b40 2023-10-30T13:27:04.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb2400be50 con 0x7fdb3806a980 2023-10-30T13:27:04.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3dca9640 1 --2- 172.21.15.125:0/1516147869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb240047d0 0x7fdb24004c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3e4aa640 1 --2- 172.21.15.125:0/1516147869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb24003b40 0x7fdb24003f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.759 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3ecab640 1 --2- 172.21.15.125:0/1516147869 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb3806a980 0x7fdb24005240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3dca9640 1 -- 172.21.15.125:0/1516147869 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb3806a980 msgr2=0x7fdb24005240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:04.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3dca9640 1 --2- 172.21.15.125:0/1516147869 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb3806a980 0x7fdb24005240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3dca9640 1 -- 172.21.15.125:0/1516147869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb24003b40 msgr2=0x7fdb24003f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:04.760 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3dca9640 1 --2- 172.21.15.125:0/1516147869 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb24003b40 0x7fdb24003f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.761 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3dca9640 1 -- 172.21.15.125:0/1516147869 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb24003530 con 0x7fdb240047d0 2023-10-30T13:27:04.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.748+0000 7fdb3dca9640 1 --2- 172.21.15.125:0/1516147869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb240047d0 0x7fdb24004c50 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800c160 tx=0x7fdb2801cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9b191ed0b03412c server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:04.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb280058e0 con 0x7fdb240047d0 2023-10-30T13:27:04.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb28005a40 con 0x7fdb240047d0 2023-10-30T13:27:04.764 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb3d4a8640 1 -- 172.21.15.125:0/1516147869 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28005e10 con 0x7fdb240047d0 2023-10-30T13:27:04.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb4070e640 1 -- 172.21.15.125:0/1516147869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb240047d0 msgr2=0x7fdb24004c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb4070e640 1 --2- 172.21.15.125:0/1516147869 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb240047d0 0x7fdb24004c50 secure :-1 s=READY pgs=2217 cs=0 l=1 rev1=1 crypto rx=0x7fdb2800c160 tx=0x7fdb2801cba0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb3ecab640 1 -- 172.21.15.125:0/1516147869 reap_dead start 2023-10-30T13:27:04.765 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb4070e640 1 -- 172.21.15.125:0/1516147869 shutdown_connections 2023-10-30T13:27:04.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb4070e640 1 -- 172.21.15.125:0/1516147869 >> 172.21.15.125:0/1516147869 conn(0x7fdb380c7910 msgr2=0x7fdb380c7cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:04.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb4070e640 1 -- 172.21.15.125:0/1516147869 shutdown_connections 2023-10-30T13:27:04.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb4070e640 1 -- 172.21.15.125:0/1516147869 wait complete. 2023-10-30T13:27:04.766 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb4070e640 1 Processor -- start 2023-10-30T13:27:04.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb4070e640 1 -- start start 2023-10-30T13:27:04.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb4070e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb240047d0 0x7fdb380c2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb4070e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb38151e40 0x7fdb38148b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.767 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb4070e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb381f0260 0x7fdb38156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb4070e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb38149080 con 0x7fdb38151e40 2023-10-30T13:27:04.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb4070e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb380c3fa0 con 0x7fdb381f0260 2023-10-30T13:27:04.768 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb4070e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb381efa80 con 0x7fdb240047d0 2023-10-30T13:27:04.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb3dca9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb38151e40 0x7fdb38148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb3dca9640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb38151e40 0x7fdb38148b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53768/0 (socket says 172.21.15.125:53768) 2023-10-30T13:27:04.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb3dca9640 1 -- 172.21.15.125:0/1684147007 learned_addr learned my addr 172.21.15.125:0/1684147007 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:04.769 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb3e4aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb240047d0 0x7fdb380c2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.752+0000 7fdb3ecab640 1 --2- 172.21.15.125:0/1684147007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb381f0260 0x7fdb38156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb3dca9640 1 -- 172.21.15.125:0/1684147007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb240047d0 msgr2=0x7fdb380c2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.770 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb3dca9640 1 --2- 172.21.15.125:0/1684147007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb240047d0 0x7fdb380c2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb3dca9640 1 -- 172.21.15.125:0/1684147007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb381f0260 msgr2=0x7fdb38156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb3dca9640 1 --2- 172.21.15.125:0/1684147007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb381f0260 0x7fdb38156e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb3dca9640 1 -- 172.21.15.125:0/1684147007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb38153770 con 0x7fdb38151e40 2023-10-30T13:27:04.771 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb3dca9640 1 --2- 172.21.15.125:0/1684147007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb38151e40 0x7fdb38148b40 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fdb280ab040 tx=0x7fdb24005800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:04.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb2effd640 1 -- 172.21.15.125:0/1684147007 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb280047f0 con 0x7fdb38151e40 2023-10-30T13:27:04.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb2effd640 1 -- 172.21.15.125:0/1684147007 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb28004970 con 0x7fdb38151e40 2023-10-30T13:27:04.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb2effd640 1 -- 172.21.15.125:0/1684147007 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb28004d60 con 0x7fdb38151e40 2023-10-30T13:27:04.772 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb4070e640 1 -- 172.21.15.125:0/1684147007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb380c4240 con 0x7fdb38151e40 2023-10-30T13:27:04.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb4070e640 1 -- 172.21.15.125:0/1684147007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb380c4800 con 0x7fdb38151e40 2023-10-30T13:27:04.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb4070e640 1 -- 172.21.15.125:0/1684147007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3804e170 con 0x7fdb38151e40 2023-10-30T13:27:04.773 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb2effd640 1 -- 172.21.15.125:0/1684147007 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdb28007070 con 0x7fdb38151e40 2023-10-30T13:27:04.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb2effd640 1 --2- 172.21.15.125:0/1684147007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb1c0664a0 0x7fdb1c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:04.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.756+0000 7fdb2effd640 1 -- 172.21.15.125:0/1684147007 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fdb2811e220 con 0x7fdb38151e40 2023-10-30T13:27:04.774 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.760+0000 7fdb3e4aa640 1 --2- 172.21.15.125:0/1684147007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb1c0664a0 0x7fdb1c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:04.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.760+0000 7fdb3e4aa640 1 --2- 172.21.15.125:0/1684147007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb1c0664a0 0x7fdb1c068960 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fdb34001810 tx=0x7fdb34009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:04.775 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.760+0000 7fdb2effd640 1 -- 172.21.15.125:0/1684147007 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdb280eab90 con 0x7fdb38151e40 2023-10-30T13:27:04.916 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.912+0000 7fdb4070e640 1 -- 172.21.15.125:0/1684147007 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fdb38073e10 con 0x7fdb38151e40 2023-10-30T13:27:04.920 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.916+0000 7fdb2effd640 1 -- 172.21.15.125:0/1684147007 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) v1 ==== 123+0+0 (secure 0 0 0) 0x7fdb280be0c0 con 0x7fdb38151e40 2023-10-30T13:27:04.922 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.916+0000 7fdb2effd640 1 -- 172.21.15.125:0/1684147007 <== mon.0 v2:172.21.15.125:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fdb280eab90 con 0x7fdb38151e40 2023-10-30T13:27:04.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.916+0000 7fdb4070e640 1 -- 172.21.15.125:0/1684147007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb1c0664a0 msgr2=0x7fdb1c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.916+0000 7fdb4070e640 1 --2- 172.21.15.125:0/1684147007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb1c0664a0 0x7fdb1c068960 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fdb34001810 tx=0x7fdb34009040 comp rx=0 tx=0).stop 2023-10-30T13:27:04.924 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.916+0000 7fdb4070e640 1 -- 172.21.15.125:0/1684147007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb38151e40 msgr2=0x7fdb38148b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:04.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.916+0000 7fdb4070e640 1 --2- 172.21.15.125:0/1684147007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb38151e40 0x7fdb38148b40 secure :-1 s=READY pgs=2218 cs=0 l=1 rev1=1 crypto rx=0x7fdb280ab040 tx=0x7fdb24005800 comp rx=0 tx=0).stop 2023-10-30T13:27:04.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.916+0000 7fdb4070e640 1 -- 172.21.15.125:0/1684147007 shutdown_connections 2023-10-30T13:27:04.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.916+0000 7fdb4070e640 1 --2- 172.21.15.125:0/1684147007 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdb1c0664a0 0x7fdb1c068960 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.925 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.916+0000 7fdb4070e640 1 --2- 172.21.15.125:0/1684147007 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdb381f0260 0x7fdb38156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.916+0000 7fdb4070e640 1 --2- 172.21.15.125:0/1684147007 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdb38151e40 0x7fdb38148b40 unknown :-1 s=CLOSED pgs=2218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.916+0000 7fdb4070e640 1 --2- 172.21.15.125:0/1684147007 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdb240047d0 0x7fdb380c2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:04.926 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.916+0000 7fdb4070e640 1 -- 172.21.15.125:0/1684147007 >> 172.21.15.125:0/1684147007 conn(0x7fdb380c7910 msgr2=0x7fdb38150930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:04.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.920+0000 7fdb4070e640 1 -- 172.21.15.125:0/1684147007 shutdown_connections 2023-10-30T13:27:04.927 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:04.920+0000 7fdb4070e640 1 -- 172.21.15.125:0/1684147007 wait complete. 2023-10-30T13:27:04.942 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-30T13:27:04.943 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:27:05.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.076+0000 7fb28b72a640 1 Processor -- start 2023-10-30T13:27:05.081 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.076+0000 7fb28b72a640 1 -- start start 2023-10-30T13:27:05.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.076+0000 7fb28b72a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb2840c93d0 0x7fb28406ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.076+0000 7fb28b72a640 1 -- --> v1:172.21.15.125:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb28406afa0 con 0x7fb28406e470 2023-10-30T13:27:05.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.076+0000 7fb28b72a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb284154150 con 0x7fb284153d70 2023-10-30T13:27:05.083 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.076+0000 7fb28b72a640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb28406d780 con 0x7fb2840c93d0 2023-10-30T13:27:05.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.076+0000 7fb289cc7640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb2840c93d0 0x7fb28406ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.076+0000 7fb289cc7640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fb2840c93d0 0x7fb28406ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53782/0 (socket says 172.21.15.125:53782) 2023-10-30T13:27:05.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.076+0000 7fb289cc7640 1 -- 172.21.15.125:0/1058660450 learned_addr learned my addr 172.21.15.125:0/1058660450 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:05.084 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.076+0000 7fb289cc7640 1 -- 172.21.15.125:0/1058660450 >> v1:172.21.15.136:6789/0 conn(0x7fb284153d70 legacy=0x7fb2840cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.076+0000 7fb289cc7640 1 -- 172.21.15.125:0/1058660450 >> v1:172.21.15.125:6789/0 conn(0x7fb28406e470 legacy=0x7fb2840c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.076+0000 7fb289cc7640 1 -- 172.21.15.125:0/1058660450 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28406e030 con 0x7fb2840c93d0 2023-10-30T13:27:05.085 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.076+0000 7fb27bfff640 1 -- 172.21.15.125:0/1058660450 <== mon.0 v1:172.21.15.125:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1252745640 0 0) 0x7fb28406afa0 con 0x7fb28406e470 2023-10-30T13:27:05.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.080+0000 7fb289cc7640 1 --2- 172.21.15.125:0/1058660450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2840c93d0 0x7fb28406ca80 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fb27c009e50 tx=0x7fb27c01c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fbba8e8d3afc935 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:05.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.080+0000 7fb27bfff640 1 -- 172.21.15.125:0/1058660450 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c0a58a0 con 0x7fb2840c93d0 2023-10-30T13:27:05.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.080+0000 7fb27bfff640 1 -- 172.21.15.125:0/1058660450 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fb27c0a5a00 con 0x7fb2840c93d0 2023-10-30T13:27:05.087 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.080+0000 7fb27bfff640 1 -- 172.21.15.125:0/1058660450 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c0a5db0 con 0x7fb2840c93d0 2023-10-30T13:27:05.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.080+0000 7fb28b72a640 1 -- 172.21.15.125:0/1058660450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2840c93d0 msgr2=0x7fb28406ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.080+0000 7fb28b72a640 1 --2- 172.21.15.125:0/1058660450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2840c93d0 0x7fb28406ca80 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7fb27c009e50 tx=0x7fb27c01c9e0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.088 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.080+0000 7fb28b72a640 1 -- 172.21.15.125:0/1058660450 shutdown_connections 2023-10-30T13:27:05.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.080+0000 7fb28b72a640 1 --2- 172.21.15.125:0/1058660450 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb2840c93d0 0x7fb28406ca80 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.080+0000 7fb28b72a640 1 -- 172.21.15.125:0/1058660450 >> 172.21.15.125:0/1058660450 conn(0x7fb284071ea0 msgr2=0x7fb284065a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:05.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.080+0000 7fb28b72a640 1 -- 172.21.15.125:0/1058660450 shutdown_connections 2023-10-30T13:27:05.089 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.080+0000 7fb28b72a640 1 -- 172.21.15.125:0/1058660450 wait complete. 2023-10-30T13:27:05.091 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb28b72a640 1 Processor -- start 2023-10-30T13:27:05.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb28b72a640 1 -- start start 2023-10-30T13:27:05.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb28b72a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb28406e470 0x7fb284154b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.092 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb28b72a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2840c93d0 0x7fb284145af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb28b72a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb284153d70 0x7fb284146030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb28b72a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2841f6a10 con 0x7fb284153d70 2023-10-30T13:27:05.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb28b72a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2841456d0 con 0x7fb2840c93d0 2023-10-30T13:27:05.093 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb28b72a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb28413ff70 con 0x7fb28406e470 2023-10-30T13:27:05.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb289cc7640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb284153d70 0x7fb284146030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb289cc7640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb284153d70 0x7fb284146030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53784/0 (socket says 172.21.15.125:53784) 2023-10-30T13:27:05.094 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb289cc7640 1 -- 172.21.15.125:0/4093486571 learned_addr learned my addr 172.21.15.125:0/4093486571 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:05.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb2894c6640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb28406e470 0x7fb284154b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb288cc5640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2840c93d0 0x7fb284145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb289cc7640 1 -- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb28406e470 msgr2=0x7fb284154b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.095 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb289cc7640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb28406e470 0x7fb284154b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb289cc7640 1 -- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2840c93d0 msgr2=0x7fb284145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb289cc7640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2840c93d0 0x7fb284145af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb289cc7640 1 -- 172.21.15.125:0/4093486571 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb27c009b30 con 0x7fb284153d70 2023-10-30T13:27:05.096 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb2894c6640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb28406e470 0x7fb284154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:27:05.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb289cc7640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb284153d70 0x7fb284146030 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fb27c0af040 tx=0x7fb27c019780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:05.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb279ffb640 1 -- 172.21.15.125:0/4093486571 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c0b8880 con 0x7fb284153d70 2023-10-30T13:27:05.097 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb279ffb640 1 -- 172.21.15.125:0/4093486571 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fb27c0b8a00 con 0x7fb284153d70 2023-10-30T13:27:05.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb279ffb640 1 -- 172.21.15.125:0/4093486571 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb27c0b8dd0 con 0x7fb284153d70 2023-10-30T13:27:05.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb28b72a640 1 -- 172.21.15.125:0/4093486571 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb284140460 con 0x7fb284153d70 2023-10-30T13:27:05.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb28b72a640 1 -- 172.21.15.125:0/4093486571 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb284145010 con 0x7fb284153d70 2023-10-30T13:27:05.098 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.084+0000 7fb28b72a640 1 -- 172.21.15.125:0/4093486571 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb284145810 con 0x7fb284153d70 2023-10-30T13:27:05.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.088+0000 7fb279ffb640 1 -- 172.21.15.125:0/4093486571 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fb27c0c3460 con 0x7fb284153d70 2023-10-30T13:27:05.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.088+0000 7fb279ffb640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2680664a0 0x7fb268068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.088+0000 7fb279ffb640 1 -- 172.21.15.125:0/4093486571 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fb27c120370 con 0x7fb284153d70 2023-10-30T13:27:05.100 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.088+0000 7fb2894c6640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2680664a0 0x7fb268068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.088+0000 7fb2894c6640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2680664a0 0x7fb268068960 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fb280000f70 tx=0x7fb280009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:05.101 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.092+0000 7fb279ffb640 1 -- 172.21.15.125:0/4093486571 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fb284145810 con 0x7fb284153d70 2023-10-30T13:27:05.277 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.272+0000 7fb28b72a640 1 -- 172.21.15.125:0/4093486571 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb284145810 con 0x7fb284153d70 2023-10-30T13:27:05.278 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.272+0000 7fb279ffb640 1 -- 172.21.15.125:0/4093486571 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 268 v268) v1 ==== 95+0+6554 (secure 0 0 0) 0x7fb284145810 con 0x7fb284153d70 2023-10-30T13:27:05.278 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:27:05.278 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":268,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2118690741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2118690741},{"type":"v1","addr":"172.21.15.136:6835","nonce":2118690741}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":264}],"filesystems":[{"mdsmap":{"epoch":268,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:25:49.238446+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:27:05.279 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 268 2023-10-30T13:27:05.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.272+0000 7fb28b72a640 1 -- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2680664a0 msgr2=0x7fb268068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.272+0000 7fb28b72a640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2680664a0 0x7fb268068960 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fb280000f70 tx=0x7fb280009040 comp rx=0 tx=0).stop 2023-10-30T13:27:05.280 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.272+0000 7fb28b72a640 1 -- 172.21.15.125:0/4093486571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb284153d70 msgr2=0x7fb284146030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.272+0000 7fb28b72a640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb284153d70 0x7fb284146030 secure :-1 s=READY pgs=2221 cs=0 l=1 rev1=1 crypto rx=0x7fb27c0af040 tx=0x7fb27c019780 comp rx=0 tx=0).stop 2023-10-30T13:27:05.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.272+0000 7fb28b72a640 1 -- 172.21.15.125:0/4093486571 shutdown_connections 2023-10-30T13:27:05.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.272+0000 7fb28b72a640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fb2680664a0 0x7fb268068960 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.281 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.272+0000 7fb28b72a640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fb284153d70 0x7fb284146030 unknown :-1 s=CLOSED pgs=2221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.272+0000 7fb28b72a640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fb2840c93d0 0x7fb284145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.272+0000 7fb28b72a640 1 --2- 172.21.15.125:0/4093486571 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fb28406e470 0x7fb284154b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.282 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.272+0000 7fb28b72a640 1 -- 172.21.15.125:0/4093486571 >> 172.21.15.125:0/4093486571 conn(0x7fb284071ea0 msgr2=0x7fb28414e980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:05.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.276+0000 7fb28b72a640 1 -- 172.21.15.125:0/4093486571 shutdown_connections 2023-10-30T13:27:05.283 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.276+0000 7fb28b72a640 1 -- 172.21.15.125:0/4093486571 wait complete. 2023-10-30T13:27:05.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-30T13:27:05.438 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.432+0000 7fa549980640 1 Processor -- start 2023-10-30T13:27:05.439 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.432+0000 7fa549980640 1 -- start start 2023-10-30T13:27:05.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.432+0000 7fa549980640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa54406f7a0 0x7fa54406fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.440 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.432+0000 7fa549980640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5440c6bd0 0x7fa544069e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.432+0000 7fa549980640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa54406a3d0 con 0x7fa544071110 2023-10-30T13:27:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.432+0000 7fa549980640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5440700c0 con 0x7fa54406f7a0 2023-10-30T13:27:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.432+0000 7fa549980640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa544066cf0 con 0x7fa5440c6bd0 2023-10-30T13:27:05.441 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.436+0000 7fa542ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa54406f7a0 0x7fa54406fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.436+0000 7fa542ffd640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fa54406f7a0 0x7fa54406fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53788/0 (socket says 172.21.15.125:53788) 2023-10-30T13:27:05.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.436+0000 7fa542ffd640 1 -- 172.21.15.125:0/3902496563 learned_addr learned my addr 172.21.15.125:0/3902496563 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:05.442 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.436+0000 7fa5427fc640 1 --2- 172.21.15.125:0/3902496563 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5440c6bd0 0x7fa544069e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.436+0000 7fa542ffd640 1 -- 172.21.15.125:0/3902496563 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5440c6bd0 msgr2=0x7fa544069e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:05.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.436+0000 7fa542ffd640 1 --2- 172.21.15.125:0/3902496563 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5440c6bd0 0x7fa544069e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.436+0000 7fa542ffd640 1 -- 172.21.15.125:0/3902496563 >> v1:172.21.15.136:6789/0 conn(0x7fa544071110 legacy=0x7fa544158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:05.443 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.436+0000 7fa542ffd640 1 -- 172.21.15.125:0/3902496563 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5440c70d0 con 0x7fa54406f7a0 2023-10-30T13:27:05.444 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.436+0000 7fa542ffd640 1 --2- 172.21.15.125:0/3902496563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa54406f7a0 0x7fa54406fb80 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fa534009870 tx=0x7fa53401ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7545a9448bc2bbd6 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:05.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.436+0000 7fa541ffb640 1 -- 172.21.15.125:0/3902496563 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5340a89b0 con 0x7fa54406f7a0 2023-10-30T13:27:05.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.436+0000 7fa541ffb640 1 -- 172.21.15.125:0/3902496563 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fa5340a8b10 con 0x7fa54406f7a0 2023-10-30T13:27:05.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.436+0000 7fa541ffb640 1 -- 172.21.15.125:0/3902496563 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5340b0440 con 0x7fa54406f7a0 2023-10-30T13:27:05.445 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 -- 172.21.15.125:0/3902496563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa54406f7a0 msgr2=0x7fa54406fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 --2- 172.21.15.125:0/3902496563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa54406f7a0 0x7fa54406fb80 secure :-1 s=READY pgs=2222 cs=0 l=1 rev1=1 crypto rx=0x7fa534009870 tx=0x7fa53401ecf0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 -- 172.21.15.125:0/3902496563 shutdown_connections 2023-10-30T13:27:05.446 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 --2- 172.21.15.125:0/3902496563 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5440c6bd0 0x7fa544069e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 --2- 172.21.15.125:0/3902496563 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa54406f7a0 0x7fa54406fb80 unknown :-1 s=CLOSED pgs=2222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 -- 172.21.15.125:0/3902496563 >> 172.21.15.125:0/3902496563 conn(0x7fa5440c7910 msgr2=0x7fa5440c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:05.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 -- 172.21.15.125:0/3902496563 shutdown_connections 2023-10-30T13:27:05.447 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 -- 172.21.15.125:0/3902496563 wait complete. 2023-10-30T13:27:05.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 Processor -- start 2023-10-30T13:27:05.449 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 -- start start 2023-10-30T13:27:05.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa54406f7a0 0x7fa5441e6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa544071110 0x7fa544157770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.450 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5440c6bd0 0x7fa5441eea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5441e2f20 con 0x7fa54406f7a0 2023-10-30T13:27:05.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5440c67e0 con 0x7fa544071110 2023-10-30T13:27:05.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa549980640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5440af180 con 0x7fa5440c6bd0 2023-10-30T13:27:05.451 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa542ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa54406f7a0 0x7fa5441e6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa542ffd640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa54406f7a0 0x7fa5441e6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53794/0 (socket says 172.21.15.125:53794) 2023-10-30T13:27:05.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa542ffd640 1 -- 172.21.15.125:0/1157421203 learned_addr learned my addr 172.21.15.125:0/1157421203 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:05.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa5427fc640 1 --2- 172.21.15.125:0/1157421203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa544071110 0x7fa544157770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.452 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa5437fe640 1 --2- 172.21.15.125:0/1157421203 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5440c6bd0 0x7fa5441eea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa542ffd640 1 -- 172.21.15.125:0/1157421203 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5440c6bd0 msgr2=0x7fa5441eea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa542ffd640 1 --2- 172.21.15.125:0/1157421203 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5440c6bd0 0x7fa5441eea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa542ffd640 1 -- 172.21.15.125:0/1157421203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa544071110 msgr2=0x7fa544157770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.453 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa542ffd640 1 --2- 172.21.15.125:0/1157421203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa544071110 0x7fa544157770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.440+0000 7fa542ffd640 1 -- 172.21.15.125:0/1157421203 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5340094d0 con 0x7fa54406f7a0 2023-10-30T13:27:05.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.444+0000 7fa542ffd640 1 --2- 172.21.15.125:0/1157421203 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa54406f7a0 0x7fa5441e6690 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fa5340b3040 tx=0x7fa53401ba90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:05.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.444+0000 7fa52bfff640 1 -- 172.21.15.125:0/1157421203 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5340a89b0 con 0x7fa54406f7a0 2023-10-30T13:27:05.454 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.444+0000 7fa52bfff640 1 -- 172.21.15.125:0/1157421203 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fa5340bb480 con 0x7fa54406f7a0 2023-10-30T13:27:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.444+0000 7fa549980640 1 -- 172.21.15.125:0/1157421203 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5440af460 con 0x7fa54406f7a0 2023-10-30T13:27:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.444+0000 7fa52bfff640 1 -- 172.21.15.125:0/1157421203 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5340bb850 con 0x7fa54406f7a0 2023-10-30T13:27:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.444+0000 7fa549980640 1 -- 172.21.15.125:0/1157421203 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5441f2460 con 0x7fa54406f7a0 2023-10-30T13:27:05.455 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.444+0000 7fa549980640 1 -- 172.21.15.125:0/1157421203 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5441f1fe0 con 0x7fa54406f7a0 2023-10-30T13:27:05.456 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.444+0000 7fa52bfff640 1 -- 172.21.15.125:0/1157421203 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fa5340bb9b0 con 0x7fa54406f7a0 2023-10-30T13:27:05.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.444+0000 7fa52bfff640 1 --2- 172.21.15.125:0/1157421203 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa5300664d0 0x7fa530068990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.444+0000 7fa52bfff640 1 -- 172.21.15.125:0/1157421203 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7fa5341230e0 con 0x7fa54406f7a0 2023-10-30T13:27:05.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.444+0000 7fa5427fc640 1 --2- 172.21.15.125:0/1157421203 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa5300664d0 0x7fa530068990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.457 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.448+0000 7fa5427fc640 1 --2- 172.21.15.125:0/1157421203 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa5300664d0 0x7fa530068990 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fa544150e20 tx=0x7fa538008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:05.458 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.448+0000 7fa52bfff640 1 -- 172.21.15.125:0/1157421203 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fa5441f1fe0 con 0x7fa54406f7a0 2023-10-30T13:27:05.636 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.632+0000 7fa549980640 1 -- 172.21.15.125:0/1157421203 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5441f1fe0 con 0x7fa54406f7a0 2023-10-30T13:27:05.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.632+0000 7fa52bfff640 1 -- 172.21.15.125:0/1157421203 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 269 v269) v1 ==== 95+0+6552 (secure 0 0 0) 0x7fa5441f1fe0 con 0x7fa54406f7a0 2023-10-30T13:27:05.637 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:27:05.638 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":269,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4256,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6837/2534930361","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6836","nonce":2534930361},{"type":"v1","addr":"172.21.15.136:6837","nonce":2534930361}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4262,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.125:6835/2903453732","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6832","nonce":2903453732},{"type":"v1","addr":"172.21.15.125:6835","nonce":2903453732}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":8273,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.136:6835/2118690741","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6834","nonce":2118690741},{"type":"v1","addr":"172.21.15.136:6835","nonce":2118690741}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":264}],"filesystems":[{"mdsmap":{"epoch":269,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-30T12:28:44.410746+0000","modified":"2023-10-30T13:27:04.837662+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":9409,"mds_1":9150,"mds_2":9600},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9150":{"gid":9150,"name":"c","rank":1,"incarnation":214,"state":"up:active","state_seq":36,"addr":"172.21.15.125:6837/796512544","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6834","nonce":796512544},{"type":"v1","addr":"172.21.15.125:6837","nonce":796512544}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9409":{"gid":9409,"name":"f","rank":0,"incarnation":259,"state":"up:active","state_seq":39,"addr":"172.21.15.136:6839/2056603754","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6838","nonce":2056603754},{"type":"v1","addr":"172.21.15.136:6839","nonce":2056603754}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_9600":{"gid":9600,"name":"e","rank":2,"incarnation":241,"state":"up:active","state_seq":42,"addr":"172.21.15.125:6836/2077416857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6833","nonce":2077416857},{"type":"v1","addr":"172.21.15.125:6836","nonce":2077416857}]},"join_fscid":-1,"export_targets":[0],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-30T13:27:05.639 INFO:teuthology.orchestra.run.smithi125.stderr:dumped fsmap epoch 269 2023-10-30T13:27:05.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.632+0000 7fa549980640 1 -- 172.21.15.125:0/1157421203 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa5300664d0 msgr2=0x7fa530068990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.632+0000 7fa549980640 1 --2- 172.21.15.125:0/1157421203 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa5300664d0 0x7fa530068990 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fa544150e20 tx=0x7fa538008040 comp rx=0 tx=0).stop 2023-10-30T13:27:05.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.632+0000 7fa549980640 1 -- 172.21.15.125:0/1157421203 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa54406f7a0 msgr2=0x7fa5441e6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.632+0000 7fa549980640 1 --2- 172.21.15.125:0/1157421203 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa54406f7a0 0x7fa5441e6690 secure :-1 s=READY pgs=2223 cs=0 l=1 rev1=1 crypto rx=0x7fa5340b3040 tx=0x7fa53401ba90 comp rx=0 tx=0).stop 2023-10-30T13:27:05.640 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.632+0000 7fa549980640 1 -- 172.21.15.125:0/1157421203 shutdown_connections 2023-10-30T13:27:05.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.632+0000 7fa549980640 1 --2- 172.21.15.125:0/1157421203 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fa5300664d0 0x7fa530068990 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.632+0000 7fa549980640 1 --2- 172.21.15.125:0/1157421203 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fa5440c6bd0 0x7fa5441eea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.632+0000 7fa549980640 1 --2- 172.21.15.125:0/1157421203 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fa544071110 0x7fa544157770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.632+0000 7fa549980640 1 --2- 172.21.15.125:0/1157421203 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fa54406f7a0 0x7fa5441e6690 unknown :-1 s=CLOSED pgs=2223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.632+0000 7fa549980640 1 -- 172.21.15.125:0/1157421203 >> 172.21.15.125:0/1157421203 conn(0x7fa5440c7910 msgr2=0x7fa5440c2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:05.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.632+0000 7fa549980640 1 -- 172.21.15.125:0/1157421203 shutdown_connections 2023-10-30T13:27:05.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.632+0000 7fa549980640 1 -- 172.21.15.125:0/1157421203 wait complete. 2023-10-30T13:27:05.658 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-30T13:27:05.794 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.788+0000 7f917ae46640 1 Processor -- start 2023-10-30T13:27:05.795 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.788+0000 7f917ae46640 1 -- start start 2023-10-30T13:27:05.796 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f917ae46640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f91740c6fe0 0x7f9174071260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f917ae46640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f91740c5f60 0x7f91740cb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f917ae46640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91740700b0 con 0x7f9174153bf0 2023-10-30T13:27:05.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f917ae46640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91740c6810 con 0x7f91740c6fe0 2023-10-30T13:27:05.797 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f917ae46640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9174159900 con 0x7f91740c5f60 2023-10-30T13:27:05.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f916bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f91740c6fe0 0x7f9174071260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f916bfff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f91740c6fe0 0x7f9174071260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53800/0 (socket says 172.21.15.125:53800) 2023-10-30T13:27:05.798 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f916bfff640 1 -- 172.21.15.125:0/2383102931 learned_addr learned my addr 172.21.15.125:0/2383102931 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:05.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f91793e3640 1 --2- 172.21.15.125:0/2383102931 >> v2:172.21.15.136:3300/0 conn(0x7f91740c5f60 0x7f91740cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f916bfff640 1 -- 172.21.15.125:0/2383102931 >> v2:172.21.15.136:3300/0 conn(0x7f91740c5f60 msgr2=0x7f91740cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:05.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f916bfff640 1 --2- 172.21.15.125:0/2383102931 >> v2:172.21.15.136:3300/0 conn(0x7f91740c5f60 0x7f91740cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.799 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f916bfff640 1 -- 172.21.15.125:0/2383102931 >> v1:172.21.15.136:6789/0 conn(0x7f9174153bf0 legacy=0x7f91740c74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:05.800 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f916bfff640 1 -- 172.21.15.125:0/2383102931 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91740c6530 con 0x7f91740c6fe0 2023-10-30T13:27:05.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f916bfff640 1 --2- 172.21.15.125:0/2383102931 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91740c6fe0 0x7f9174071260 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f917406ce90 tx=0x7f916401ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b7c37ff22c6ba2 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:05.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f916b7fe640 1 -- 172.21.15.125:0/2383102931 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91640a89b0 con 0x7f91740c6fe0 2023-10-30T13:27:05.801 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f916b7fe640 1 -- 172.21.15.125:0/2383102931 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f91640a8b10 con 0x7f91740c6fe0 2023-10-30T13:27:05.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.792+0000 7f916b7fe640 1 -- 172.21.15.125:0/2383102931 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91640b0440 con 0x7f91740c6fe0 2023-10-30T13:27:05.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 -- 172.21.15.125:0/2383102931 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91740c6fe0 msgr2=0x7f9174071260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 --2- 172.21.15.125:0/2383102931 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91740c6fe0 0x7f9174071260 secure :-1 s=READY pgs=2224 cs=0 l=1 rev1=1 crypto rx=0x7f917406ce90 tx=0x7f916401ecb0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.802 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 -- 172.21.15.125:0/2383102931 shutdown_connections 2023-10-30T13:27:05.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 --2- 172.21.15.125:0/2383102931 >> v2:172.21.15.136:3300/0 conn(0x7f91740c5f60 0x7f91740cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 --2- 172.21.15.125:0/2383102931 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91740c6fe0 0x7f9174071260 unknown :-1 s=CLOSED pgs=2224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.803 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 -- 172.21.15.125:0/2383102931 >> 172.21.15.125:0/2383102931 conn(0x7f917406c6e0 msgr2=0x7f9174069e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:05.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 -- 172.21.15.125:0/2383102931 shutdown_connections 2023-10-30T13:27:05.804 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 -- 172.21.15.125:0/2383102931 wait complete. 2023-10-30T13:27:05.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 Processor -- start 2023-10-30T13:27:05.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 -- start start 2023-10-30T13:27:05.806 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91740c5f60 0x7f91741f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91740c6fe0 0x7f91740c3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9174153bf0 0x7f91740c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.807 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91741f3220 con 0x7f91740c5f60 2023-10-30T13:27:05.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9174156ca0 con 0x7f91740c6fe0 2023-10-30T13:27:05.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f917ae46640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9174156ef0 con 0x7f9174153bf0 2023-10-30T13:27:05.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f9178be2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91740c5f60 0x7f91741f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.808 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f9178be2640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91740c5f60 0x7f91741f2ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53808/0 (socket says 172.21.15.125:53808) 2023-10-30T13:27:05.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f9178be2640 1 -- 172.21.15.125:0/1006205848 learned_addr learned my addr 172.21.15.125:0/1006205848 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:05.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f916bfff640 1 --2- 172.21.15.125:0/1006205848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91740c6fe0 0x7f91740c3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.796+0000 7f91793e3640 1 --2- 172.21.15.125:0/1006205848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9174153bf0 0x7f91740c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.809 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f9178be2640 1 -- 172.21.15.125:0/1006205848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9174153bf0 msgr2=0x7f91740c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f9178be2640 1 --2- 172.21.15.125:0/1006205848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9174153bf0 0x7f91740c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f9178be2640 1 -- 172.21.15.125:0/1006205848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91740c6fe0 msgr2=0x7f91740c3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.810 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f9178be2640 1 --2- 172.21.15.125:0/1006205848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91740c6fe0 0x7f91740c3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f9178be2640 1 -- 172.21.15.125:0/1006205848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91640094d0 con 0x7f91740c5f60 2023-10-30T13:27:05.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f9178be2640 1 --2- 172.21.15.125:0/1006205848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91740c5f60 0x7f91741f2ce0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f917000b410 tx=0x7f917000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:05.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f91697fa640 1 -- 172.21.15.125:0/1006205848 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9170004020 con 0x7f91740c5f60 2023-10-30T13:27:05.811 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f91697fa640 1 -- 172.21.15.125:0/1006205848 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f9170004180 con 0x7f91740c5f60 2023-10-30T13:27:05.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f91697fa640 1 -- 172.21.15.125:0/1006205848 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9170011630 con 0x7f91740c5f60 2023-10-30T13:27:05.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f917ae46640 1 -- 172.21.15.125:0/1006205848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9174157190 con 0x7f91740c5f60 2023-10-30T13:27:05.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f917ae46640 1 -- 172.21.15.125:0/1006205848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91740c4750 con 0x7f91740c5f60 2023-10-30T13:27:05.812 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f917ae46640 1 -- 172.21.15.125:0/1006205848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9174159af0 con 0x7f91740c5f60 2023-10-30T13:27:05.813 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f91697fa640 1 -- 172.21.15.125:0/1006205848 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f9170011790 con 0x7f91740c5f60 2023-10-30T13:27:05.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f91697fa640 1 --2- 172.21.15.125:0/1006205848 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91600664a0 0x7f9160068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:05.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.800+0000 7f91697fa640 1 -- 172.21.15.125:0/1006205848 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f9170084820 con 0x7f91740c5f60 2023-10-30T13:27:05.814 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.804+0000 7f916bfff640 1 --2- 172.21.15.125:0/1006205848 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91600664a0 0x7f9160068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:05.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.804+0000 7f916bfff640 1 --2- 172.21.15.125:0/1006205848 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91600664a0 0x7f9160068960 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f91640b2000 tx=0x7f9164008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:05.815 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.804+0000 7f91697fa640 1 -- 172.21.15.125:0/1006205848 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f9174159af0 con 0x7f91740c5f60 2023-10-30T13:27:05.960 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.956+0000 7f917ae46640 1 -- 172.21.15.125:0/1006205848 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9174159af0 con 0x7f91740c5f60 2023-10-30T13:27:05.961 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.956+0000 7f91697fa640 1 -- 172.21.15.125:0/1006205848 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+20714 (secure 0 0 0) 0x7f9174159af0 con 0x7f91740c5f60 2023-10-30T13:27:05.962 INFO:teuthology.orchestra.run.smithi125.stdout: 2023-10-30T13:27:05.962 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":85,"fsid":"714f08a3-69cf-4c1b-8fab-64b3ba5e42e9","created":"2023-10-30T12:28:20.111285+0000","modified":"2023-10-30T13:24:15.942081+0000","last_up_change":"2023-10-30T12:28:32.412295+0000","last_in_change":"2023-10-30T12:28:22.401009+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-30T12:28:31.063403+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-30T12:28:43.007849+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-30T12:28:43.401231+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":505,"snap_epoch":55,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2da5aeef-d0c2-42d6-b876-15bf6cdfb2bf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6801","nonce":1302652123}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6803","nonce":1302652123}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6807","nonce":1302652123}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":1302652123},{"type":"v1","addr":"172.21.15.125:6805","nonce":1302652123}]},"public_addr":"172.21.15.125:6801/1302652123","cluster_addr":"172.21.15.125:6803/1302652123","heartbeat_back_addr":"172.21.15.125:6807/1302652123","heartbeat_front_addr":"172.21.15.125:6805/1302652123","state":["exists","up"]},{"osd":1,"uuid":"5470ed31-d4ff-483d-a06f-82dc6ac62cce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6817","nonce":3694357726}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6819","nonce":3694357726}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6823","nonce":3694357726}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":3694357726},{"type":"v1","addr":"172.21.15.125:6821","nonce":3694357726}]},"public_addr":"172.21.15.125:6817/3694357726","cluster_addr":"172.21.15.125:6819/3694357726","heartbeat_back_addr":"172.21.15.125:6823/3694357726","heartbeat_front_addr":"172.21.15.125:6821/3694357726","state":["exists","up"]},{"osd":2,"uuid":"74491459-06a1-488c-8a93-b04224b25521","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6809","nonce":456280330}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6811","nonce":456280330}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6815","nonce":456280330}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":456280330},{"type":"v1","addr":"172.21.15.125:6813","nonce":456280330}]},"public_addr":"172.21.15.125:6809/456280330","cluster_addr":"172.21.15.125:6811/456280330","heartbeat_back_addr":"172.21.15.125:6815/456280330","heartbeat_front_addr":"172.21.15.125:6813/456280330","state":["exists","up"]},{"osd":3,"uuid":"0290df24-0fff-4968-a73f-df7f903afb35","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":16,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6825","nonce":3730905694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6827","nonce":3730905694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6831","nonce":3730905694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":3730905694},{"type":"v1","addr":"172.21.15.125:6829","nonce":3730905694}]},"public_addr":"172.21.15.125:6825/3730905694","cluster_addr":"172.21.15.125:6827/3730905694","heartbeat_back_addr":"172.21.15.125:6831/3730905694","heartbeat_front_addr":"172.21.15.125:6829/3730905694","state":["exists","up"]},{"osd":4,"uuid":"be856cf6-9dc7-479a-83b5-14f05dd3de02","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6817","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6819","nonce":4208991530}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6821","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6823","nonce":4208991530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6829","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6831","nonce":4208991530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6825","nonce":4208991530},{"type":"v1","addr":"172.21.15.136:6827","nonce":4208991530}]},"public_addr":"172.21.15.136:6819/4208991530","cluster_addr":"172.21.15.136:6823/4208991530","heartbeat_back_addr":"172.21.15.136:6831/4208991530","heartbeat_front_addr":"172.21.15.136:6827/4208991530","state":["exists","up"]},{"osd":5,"uuid":"906dc735-5893-48ac-9c97-f02ae207d1fb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6801","nonce":1849732357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6802","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6803","nonce":1849732357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6807","nonce":1849732357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6804","nonce":1849732357},{"type":"v1","addr":"172.21.15.136:6805","nonce":1849732357}]},"public_addr":"172.21.15.136:6801/1849732357","cluster_addr":"172.21.15.136:6803/1849732357","heartbeat_back_addr":"172.21.15.136:6807/1849732357","heartbeat_front_addr":"172.21.15.136:6805/1849732357","state":["exists","up"]},{"osd":6,"uuid":"69ecbffa-5402-4e0e-8833-f4334bd95935","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6818","nonce":3758986733}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6822","nonce":3758986733}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6830","nonce":3758986733}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":3758986733},{"type":"v1","addr":"172.21.15.136:6826","nonce":3758986733}]},"public_addr":"172.21.15.136:6818/3758986733","cluster_addr":"172.21.15.136:6822/3758986733","heartbeat_back_addr":"172.21.15.136:6830/3758986733","heartbeat_front_addr":"172.21.15.136:6826/3758986733","state":["exists","up"]},{"osd":7,"uuid":"ff4daea0-1315-43b4-83ed-d32535b951e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6808","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6809","nonce":3315858521}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6811","nonce":3315858521}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6815","nonce":3315858521}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6812","nonce":3315858521},{"type":"v1","addr":"172.21.15.136:6813","nonce":3315858521}]},"public_addr":"172.21.15.136:6809/3315858521","cluster_addr":"172.21.15.136:6811/3315858521","heartbeat_back_addr":"172.21.15.136:6815/3315858521","heartbeat_front_addr":"172.21.15.136:6813/3315858521","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.049774+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.967612+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.957150+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.136262+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.035736+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.885647+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:28.004540+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-30T12:28:27.868323+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.f","mappings":[{"from":0,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.136:6835/3480968625":"2023-10-31T13:24:15.942040+0000","172.21.15.136:6839/4284345684":"2023-10-31T13:21:50.903213+0000","172.21.15.125:6833/739564283":"2023-10-31T12:57:50.441726+0000","172.21.15.136:6839/1454326577":"2023-10-31T12:40:50.129310+0000","172.21.15.125:6837/2408701650":"2023-10-31T12:52:30.335181+0000","172.21.15.125:6833/1163622496":"2023-10-31T12:55:15.400773+0000","172.21.15.125:6834/1624923396":"2023-10-31T12:42:15.151597+0000","172.21.15.125:6834/2408701650":"2023-10-31T12:52:30.335181+0000","172.21.15.125:6837/3906359913":"2023-10-31T12:59:15.462699+0000","172.21.15.125:6836/1721514617":"2023-10-31T12:51:15.314344+0000","172.21.15.125:6833/2595438444":"2023-10-31T12:46:30.228008+0000","172.21.15.125:6833/1721514617":"2023-10-31T12:51:15.314344+0000","172.21.15.136:6838/1510290691":"2023-10-31T12:47:25.242471+0000","172.21.15.125:6836/2595438444":"2023-10-31T12:46:30.228008+0000","172.21.15.125:6833/2017382874":"2023-10-31T12:36:30.049259+0000","172.21.15.125:6834/2540063352":"2023-10-31T13:12:50.713238+0000","172.21.15.125:6836/3950044434":"2023-10-31T13:17:55.811938+0000","172.21.15.125:6834/2291877604":"2023-10-31T12:32:34.980679+0000","172.21.15.136:6839/3922559697":"2023-10-31T12:44:00.178262+0000","172.21.15.125:6836/1163622496":"2023-10-31T12:55:15.400773+0000","172.21.15.136:6834/3480968625":"2023-10-31T13:24:15.942040+0000","172.21.15.136:6835/3346501886":"2023-10-31T12:30:24.948059+0000","172.21.15.125:6834/3906359913":"2023-10-31T12:59:15.462699+0000","172.21.15.125:6834/2196617378":"2023-10-31T13:05:55.585769+0000","172.21.15.125:6833/3711687505":"2023-10-31T13:06:55.602060+0000","172.21.15.136:6839/2250748765":"2023-10-31T12:35:15.030179+0000","172.21.15.125:6837/2291877604":"2023-10-31T12:32:34.980679+0000","172.21.15.136:6835/3526499585":"2023-10-31T13:11:30.689863+0000","172.21.15.136:6838/1454326577":"2023-10-31T12:40:50.129310+0000","172.21.15.136:6835/695089547":"2023-10-31T13:15:05.766729+0000","172.21.15.136:6839/223464319":"2023-10-31T13:08:55.644039+0000","172.21.15.125:6837/1624923396":"2023-10-31T12:42:15.151597+0000","172.21.15.125:6834/1348687922":"2023-10-31T12:56:35.422955+0000","172.21.15.125:6836/2017382874":"2023-10-31T12:36:30.049259+0000","172.21.15.136:6838/3922559697":"2023-10-31T12:44:00.178262+0000","172.21.15.125:6834/3156415951":"2023-10-31T12:48:25.270273+0000","172.21.15.125:6833/1944683138":"2023-10-31T13:00:40.486003+0000","172.21.15.125:6836/1944683138":"2023-10-31T13:00:40.486003+0000","172.21.15.125:6834/3762807620":"2023-10-31T13:02:35.533843+0000","172.21.15.125:6837/1348687922":"2023-10-31T12:56:35.422955+0000","172.21.15.125:6837/3762807620":"2023-10-31T13:02:35.533843+0000","172.21.15.125:6833/1301336591":"2023-10-31T13:04:45.567511+0000","172.21.15.125:6836/1301336591":"2023-10-31T13:04:45.567511+0000","172.21.15.136:6835/2194892271":"2023-10-31T13:08:00.628452+0000","172.21.15.125:6837/2196617378":"2023-10-31T13:05:55.585769+0000","172.21.15.125:6836/3711687505":"2023-10-31T13:06:55.602060+0000","172.21.15.136:6838/3776186190":"2023-10-31T13:16:00.781626+0000","172.21.15.136:6834/2194892271":"2023-10-31T13:08:00.628452+0000","172.21.15.136:6834/3526499585":"2023-10-31T13:11:30.689863+0000","172.21.15.136:6838/4284345684":"2023-10-31T13:21:50.903213+0000","172.21.15.136:6838/2250748765":"2023-10-31T12:35:15.030179+0000","172.21.15.136:6838/223464319":"2023-10-31T13:08:55.644039+0000","172.21.15.125:6837/3156415951":"2023-10-31T12:48:25.270273+0000","172.21.15.136:6834/695089547":"2023-10-31T13:15:05.766729+0000","172.21.15.125:6837/2540063352":"2023-10-31T13:12:50.713238+0000","172.21.15.136:6834/3346501886":"2023-10-31T12:30:24.948059+0000","172.21.15.136:6839/3776186190":"2023-10-31T13:16:00.781626+0000","172.21.15.125:6833/3950044434":"2023-10-31T13:17:55.811938+0000","172.21.15.136:6839/1510290691":"2023-10-31T12:47:25.242471+0000","172.21.15.136:6834/3864637420":"2023-10-31T13:20:45.886166+0000","172.21.15.125:6836/739564283":"2023-10-31T12:57:50.441726+0000","172.21.15.136:6835/3864637420":"2023-10-31T13:20:45.886166+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-30T13:27:05.963 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.956+0000 7f917ae46640 1 -- 172.21.15.125:0/1006205848 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91600664a0 msgr2=0x7f9160068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.956+0000 7f917ae46640 1 --2- 172.21.15.125:0/1006205848 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91600664a0 0x7f9160068960 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f91640b2000 tx=0x7f9164008040 comp rx=0 tx=0).stop 2023-10-30T13:27:05.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.956+0000 7f917ae46640 1 -- 172.21.15.125:0/1006205848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91740c5f60 msgr2=0x7f91741f2ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:05.964 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.956+0000 7f917ae46640 1 --2- 172.21.15.125:0/1006205848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91740c5f60 0x7f91741f2ce0 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f917000b410 tx=0x7f917000b8d0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.956+0000 7f917ae46640 1 -- 172.21.15.125:0/1006205848 shutdown_connections 2023-10-30T13:27:05.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.956+0000 7f917ae46640 1 --2- 172.21.15.125:0/1006205848 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f91600664a0 0x7f9160068960 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.956+0000 7f917ae46640 1 --2- 172.21.15.125:0/1006205848 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f9174153bf0 0x7f91740c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.965 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.956+0000 7f917ae46640 1 --2- 172.21.15.125:0/1006205848 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f91740c6fe0 0x7f91740c3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.956+0000 7f917ae46640 1 --2- 172.21.15.125:0/1006205848 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f91740c5f60 0x7f91741f2ce0 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:05.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.956+0000 7f917ae46640 1 -- 172.21.15.125:0/1006205848 >> 172.21.15.125:0/1006205848 conn(0x7f917406c6e0 msgr2=0x7f9174158c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:05.966 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.956+0000 7f917ae46640 1 -- 172.21.15.125:0/1006205848 shutdown_connections 2023-10-30T13:27:05.967 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:05.956+0000 7f917ae46640 1 -- 172.21.15.125:0/1006205848 wait complete. 2023-10-30T13:27:05.983 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-30T13:27:06.127 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.120+0000 7f1fe671e640 1 Processor -- start 2023-10-30T13:27:06.128 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.124+0000 7f1fe671e640 1 -- start start 2023-10-30T13:27:06.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.124+0000 7f1fe671e640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1fe00c5550 0x7f1fe00cef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:06.129 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.124+0000 7f1fe671e640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f1fe00cb050 0x7f1fe00cf440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:06.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.124+0000 7f1fe671e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe00cba30 con 0x7f1fe00cab00 2023-10-30T13:27:06.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.124+0000 7f1fe671e640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe00ca290 con 0x7f1fe00c5550 2023-10-30T13:27:06.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.124+0000 7f1fe671e640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe00c9cf0 con 0x7f1fe00cb050 2023-10-30T13:27:06.130 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.124+0000 7f1fe571c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1fe00c5550 0x7f1fe00cef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:06.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.124+0000 7f1fe571c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f1fe00c5550 0x7f1fe00cef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53812/0 (socket says 172.21.15.125:53812) 2023-10-30T13:27:06.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.124+0000 7f1fe571c640 1 -- 172.21.15.125:0/912132003 learned_addr learned my addr 172.21.15.125:0/912132003 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:06.131 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.124+0000 7f1fe4f1b640 1 --2- 172.21.15.125:0/912132003 >> v2:172.21.15.136:3300/0 conn(0x7f1fe00cb050 0x7f1fe00cf440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:06.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.124+0000 7f1fe571c640 1 -- 172.21.15.125:0/912132003 >> v2:172.21.15.136:3300/0 conn(0x7f1fe00cb050 msgr2=0x7f1fe00cf440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:06.132 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.124+0000 7f1fe571c640 1 --2- 172.21.15.125:0/912132003 >> v2:172.21.15.136:3300/0 conn(0x7f1fe00cb050 0x7f1fe00cf440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.124+0000 7f1fe571c640 1 -- 172.21.15.125:0/912132003 >> v1:172.21.15.136:6789/0 conn(0x7f1fe00cab00 legacy=0x7f1fe00cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:06.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.124+0000 7f1fe571c640 1 -- 172.21.15.125:0/912132003 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe015d990 con 0x7f1fe00c5550 2023-10-30T13:27:06.133 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.128+0000 7f1fe571c640 1 --2- 172.21.15.125:0/912132003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00c5550 0x7f1fe00cef00 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00cffe0 tx=0x7f1fdc01c500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e5ba6552c9d3c9a server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:06.134 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.128+0000 7f1fd7fff640 1 -- 172.21.15.125:0/912132003 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0ab5f0 con 0x7f1fe00c5550 2023-10-30T13:27:06.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.128+0000 7f1fd7fff640 1 -- 172.21.15.125:0/912132003 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f1fdc0ab750 con 0x7f1fe00c5550 2023-10-30T13:27:06.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.128+0000 7f1fd7fff640 1 -- 172.21.15.125:0/912132003 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0abb00 con 0x7f1fe00c5550 2023-10-30T13:27:06.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.128+0000 7f1fe671e640 1 -- 172.21.15.125:0/912132003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00c5550 msgr2=0x7f1fe00cef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:06.135 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.128+0000 7f1fe671e640 1 --2- 172.21.15.125:0/912132003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00c5550 0x7f1fe00cef00 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f1fe00cffe0 tx=0x7f1fdc01c500 comp rx=0 tx=0).stop 2023-10-30T13:27:06.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.128+0000 7f1fe671e640 1 -- 172.21.15.125:0/912132003 shutdown_connections 2023-10-30T13:27:06.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.128+0000 7f1fe671e640 1 --2- 172.21.15.125:0/912132003 >> v2:172.21.15.136:3300/0 conn(0x7f1fe00cb050 0x7f1fe00cf440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.136 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.128+0000 7f1fe671e640 1 --2- 172.21.15.125:0/912132003 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00c5550 0x7f1fe00cef00 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.128+0000 7f1fe671e640 1 -- 172.21.15.125:0/912132003 >> 172.21.15.125:0/912132003 conn(0x7f1fe006bed0 msgr2=0x7f1fe006c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:06.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.128+0000 7f1fe671e640 1 -- 172.21.15.125:0/912132003 shutdown_connections 2023-10-30T13:27:06.137 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.128+0000 7f1fe671e640 1 -- 172.21.15.125:0/912132003 wait complete. 2023-10-30T13:27:06.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.128+0000 7f1fe671e640 1 Processor -- start 2023-10-30T13:27:06.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.128+0000 7f1fe671e640 1 -- start start 2023-10-30T13:27:06.143 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe671e640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00c5550 0x7f1fe015e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:06.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe671e640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1fe00cab00 0x7f1fe01585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:06.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe671e640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1fe00cb050 0x7f1fe01521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:06.144 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe671e640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe015f3a0 con 0x7f1fe00c5550 2023-10-30T13:27:06.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe671e640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe0157ef0 con 0x7f1fe00cb050 2023-10-30T13:27:06.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe671e640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe015e110 con 0x7f1fe00cab00 2023-10-30T13:27:06.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe571c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00c5550 0x7f1fe015e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:06.145 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe571c640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00c5550 0x7f1fe015e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53828/0 (socket says 172.21.15.125:53828) 2023-10-30T13:27:06.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe571c640 1 -- 172.21.15.125:0/1550437585 learned_addr learned my addr 172.21.15.125:0/1550437585 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:06.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe4f1b640 1 --2- 172.21.15.125:0/1550437585 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1fe00cab00 0x7f1fe01585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:06.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe5f1d640 1 --2- 172.21.15.125:0/1550437585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1fe00cb050 0x7f1fe01521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:06.146 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe571c640 1 -- 172.21.15.125:0/1550437585 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1fe00cab00 msgr2=0x7f1fe01585c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:06.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe571c640 1 --2- 172.21.15.125:0/1550437585 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1fe00cab00 0x7f1fe01585c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe571c640 1 -- 172.21.15.125:0/1550437585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1fe00cb050 msgr2=0x7f1fe01521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:06.147 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe571c640 1 --2- 172.21.15.125:0/1550437585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1fe00cb050 0x7f1fe01521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe571c640 1 -- 172.21.15.125:0/1550437585 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc0094d0 con 0x7f1fe00c5550 2023-10-30T13:27:06.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe571c640 1 --2- 172.21.15.125:0/1550437585 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00c5550 0x7f1fe015e9e0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc019110 tx=0x7f1fdc01ad40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:06.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fd5ffb640 1 -- 172.21.15.125:0/1550437585 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0b4800 con 0x7f1fe00c5550 2023-10-30T13:27:06.148 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fd5ffb640 1 -- 172.21.15.125:0/1550437585 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f1fdc0b4980 con 0x7f1fe00c5550 2023-10-30T13:27:06.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fd5ffb640 1 -- 172.21.15.125:0/1550437585 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc0b4d50 con 0x7f1fe00c5550 2023-10-30T13:27:06.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe671e640 1 -- 172.21.15.125:0/1550437585 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe015de30 con 0x7f1fe00c5550 2023-10-30T13:27:06.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe671e640 1 -- 172.21.15.125:0/1550437585 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe01578a0 con 0x7f1fe00c5550 2023-10-30T13:27:06.149 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.132+0000 7f1fe671e640 1 -- 172.21.15.125:0/1550437585 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe00c8610 con 0x7f1fe00c5550 2023-10-30T13:27:06.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.136+0000 7f1fd5ffb640 1 -- 172.21.15.125:0/1550437585 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f1fdc0b3380 con 0x7f1fe00c5550 2023-10-30T13:27:06.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.136+0000 7f1fd5ffb640 1 --2- 172.21.15.125:0/1550437585 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1fc40664a0 0x7f1fc4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:06.150 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.136+0000 7f1fd5ffb640 1 -- 172.21.15.125:0/1550437585 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8404+0+0 (secure 0 0 0) 0x7f1fdc0a8050 con 0x7f1fe00c5550 2023-10-30T13:27:06.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.136+0000 7f1fe4f1b640 1 --2- 172.21.15.125:0/1550437585 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1fc40664a0 0x7f1fc4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:06.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.136+0000 7f1fe4f1b640 1 --2- 172.21.15.125:0/1550437585 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1fc40664a0 0x7f1fc4068960 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0006fd0 tx=0x7f1fd0008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:06.151 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.140+0000 7f1fd5ffb640 1 -- 172.21.15.125:0/1550437585 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f1fdc0eba50 con 0x7f1fe00c5550 2023-10-30T13:27:06.326 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.320+0000 7f1fe671e640 1 -- 172.21.15.125:0/1550437585 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f1fe0157420 con 0x7f1fe00c5550 2023-10-30T13:27:06.329 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.324+0000 7f1fd5ffb640 1 -- 172.21.15.125:0/1550437585 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v270) v1 ==== 161+0+0 (secure 0 0 0) 0x7f1fe0157420 con 0x7f1fe00c5550 2023-10-30T13:27:06.329 INFO:teuthology.orchestra.run.smithi125.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-30T13:27:06.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.328+0000 7f1fe671e640 1 -- 172.21.15.125:0/1550437585 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1fc40664a0 msgr2=0x7f1fc4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:06.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.328+0000 7f1fe671e640 1 --2- 172.21.15.125:0/1550437585 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1fc40664a0 0x7f1fc4068960 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0006fd0 tx=0x7f1fd0008040 comp rx=0 tx=0).stop 2023-10-30T13:27:06.332 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.328+0000 7f1fe671e640 1 -- 172.21.15.125:0/1550437585 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00c5550 msgr2=0x7f1fe015e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:06.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.328+0000 7f1fe671e640 1 --2- 172.21.15.125:0/1550437585 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00c5550 0x7f1fe015e9e0 secure :-1 s=READY pgs=2227 cs=0 l=1 rev1=1 crypto rx=0x7f1fdc019110 tx=0x7f1fdc01ad40 comp rx=0 tx=0).stop 2023-10-30T13:27:06.333 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.328+0000 7f1fe671e640 1 -- 172.21.15.125:0/1550437585 shutdown_connections 2023-10-30T13:27:06.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.328+0000 7f1fe671e640 1 --2- 172.21.15.125:0/1550437585 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f1fc40664a0 0x7f1fc4068960 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.328+0000 7f1fe671e640 1 --2- 172.21.15.125:0/1550437585 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f1fe00cb050 0x7f1fe01521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.334 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.328+0000 7f1fe671e640 1 --2- 172.21.15.125:0/1550437585 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f1fe00cab00 0x7f1fe01585c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.328+0000 7f1fe671e640 1 --2- 172.21.15.125:0/1550437585 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f1fe00c5550 0x7f1fe015e9e0 unknown :-1 s=CLOSED pgs=2227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.328+0000 7f1fe671e640 1 -- 172.21.15.125:0/1550437585 >> 172.21.15.125:0/1550437585 conn(0x7f1fe006bed0 msgr2=0x7f1fe00ce5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:06.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.328+0000 7f1fe671e640 1 -- 172.21.15.125:0/1550437585 shutdown_connections 2023-10-30T13:27:06.335 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.328+0000 7f1fe671e640 1 -- 172.21.15.125:0/1550437585 wait complete. 2023-10-30T13:27:06.356 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-30T13:27:06.407 INFO:tasks.ceph.mds.c.smithi125.stdout:starting mds.c at 2023-10-30T13:27:06.407 INFO:tasks.ceph.mds.e.smithi125.stdout:starting mds.e at 2023-10-30T13:27:06.428 INFO:tasks.ceph.mds.f.smithi136.stdout:starting mds.f at 2023-10-30T13:27:06.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f586569a640 1 Processor -- start 2023-10-30T13:27:06.505 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f586569a640 1 -- start start 2023-10-30T13:27:06.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f586569a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5860151730 0x7f586006ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:06.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f586569a640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f586006df70 0x7f586006e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:06.507 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f586569a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f586015ad40 con 0x7f58600bf670 2023-10-30T13:27:06.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f586569a640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f586006e8c0 con 0x7f586006df70 2023-10-30T13:27:06.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f586569a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5860151b10 con 0x7f5860151730 2023-10-30T13:27:06.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f585ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f586006df70 0x7f586006e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:06.508 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f585ffff640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f586006df70 0x7f586006e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53872/0 (socket says 172.21.15.125:53872) 2023-10-30T13:27:06.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f585ffff640 1 -- 172.21.15.125:0/1840117975 learned_addr learned my addr 172.21.15.125:0/1840117975 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:06.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f585effd640 1 --2- 172.21.15.125:0/1840117975 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5860151730 0x7f586006ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:06.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f585ffff640 1 -- 172.21.15.125:0/1840117975 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5860151730 msgr2=0x7f586006ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:06.509 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f585ffff640 1 --2- 172.21.15.125:0/1840117975 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5860151730 0x7f586006ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f585ffff640 1 -- 172.21.15.125:0/1840117975 >> v1:172.21.15.136:6789/0 conn(0x7f58600bf670 legacy=0x7f586015b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:06.510 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.500+0000 7f585ffff640 1 -- 172.21.15.125:0/1840117975 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586006f6e0 con 0x7f586006df70 2023-10-30T13:27:06.511 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.504+0000 7f585ffff640 1 --2- 172.21.15.125:0/1840117975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f586006df70 0x7f586006e380 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f5858009e50 tx=0x7f585801f1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a67b525abaccd59e server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:06.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.504+0000 7f585e7fc640 1 -- 172.21.15.125:0/1840117975 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58580af400 con 0x7f586006df70 2023-10-30T13:27:06.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.504+0000 7f585e7fc640 1 -- 172.21.15.125:0/1840117975 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f58580ab050 con 0x7f586006df70 2023-10-30T13:27:06.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.504+0000 7f585e7fc640 1 -- 172.21.15.125:0/1840117975 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58580af7a0 con 0x7f586006df70 2023-10-30T13:27:06.512 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.504+0000 7f586569a640 1 -- 172.21.15.125:0/1840117975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f586006df70 msgr2=0x7f586006e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:06.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.504+0000 7f586569a640 1 --2- 172.21.15.125:0/1840117975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f586006df70 0x7f586006e380 secure :-1 s=READY pgs=2233 cs=0 l=1 rev1=1 crypto rx=0x7f5858009e50 tx=0x7f585801f1f0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.504+0000 7f586569a640 1 -- 172.21.15.125:0/1840117975 shutdown_connections 2023-10-30T13:27:06.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.504+0000 7f586569a640 1 --2- 172.21.15.125:0/1840117975 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f586006df70 0x7f586006e380 unknown :-1 s=CLOSED pgs=2233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.513 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.504+0000 7f586569a640 1 --2- 172.21.15.125:0/1840117975 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f5860151730 0x7f586006ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.504+0000 7f586569a640 1 -- 172.21.15.125:0/1840117975 >> 172.21.15.125:0/1840117975 conn(0x7f58600cc3e0 msgr2=0x7f58600cc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:06.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.504+0000 7f586569a640 1 -- 172.21.15.125:0/1840117975 shutdown_connections 2023-10-30T13:27:06.514 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.504+0000 7f586569a640 1 -- 172.21.15.125:0/1840117975 wait complete. 2023-10-30T13:27:06.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f586569a640 1 Processor -- start 2023-10-30T13:27:06.516 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f586569a640 1 -- start start 2023-10-30T13:27:06.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f586569a640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f586006df70 0x7f586014b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:06.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f586569a640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58600bf670 0x7f586006a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:06.517 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f586569a640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5860151730 0x7f5860156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:06.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f586569a640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5860152350 con 0x7f58600bf670 2023-10-30T13:27:06.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f586569a640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58600bef60 con 0x7f5860151730 2023-10-30T13:27:06.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f586569a640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5860152ef0 con 0x7f586006df70 2023-10-30T13:27:06.518 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f5857fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58600bf670 0x7f586006a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:06.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f5857fff640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58600bf670 0x7f586006a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53876/0 (socket says 172.21.15.125:53876) 2023-10-30T13:27:06.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f5857fff640 1 -- 172.21.15.125:0/236075314 learned_addr learned my addr 172.21.15.125:0/236075314 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:06.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f585effd640 1 --2- 172.21.15.125:0/236075314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f586006df70 0x7f586014b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:06.519 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f585ffff640 1 --2- 172.21.15.125:0/236075314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5860151730 0x7f5860156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:06.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f5857fff640 1 -- 172.21.15.125:0/236075314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f586006df70 msgr2=0x7f586014b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:06.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f5857fff640 1 --2- 172.21.15.125:0/236075314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f586006df70 0x7f586014b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f5857fff640 1 -- 172.21.15.125:0/236075314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5860151730 msgr2=0x7f5860156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:06.520 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f5857fff640 1 --2- 172.21.15.125:0/236075314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5860151730 0x7f5860156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:06.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f5857fff640 1 -- 172.21.15.125:0/236075314 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5858009b30 con 0x7f58600bf670 2023-10-30T13:27:06.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f5857fff640 1 --2- 172.21.15.125:0/236075314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58600bf670 0x7f586006a670 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f584800c500 tx=0x7f584800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:06.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f58577fe640 1 -- 172.21.15.125:0/236075314 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5848005220 con 0x7f58600bf670 2023-10-30T13:27:06.521 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f58577fe640 1 -- 172.21.15.125:0/236075314 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f5848005380 con 0x7f58600bf670 2023-10-30T13:27:06.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f58577fe640 1 -- 172.21.15.125:0/236075314 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5848007290 con 0x7f58600bf670 2023-10-30T13:27:06.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f586569a640 1 -- 172.21.15.125:0/236075314 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f586006a1e0 con 0x7f58600bf670 2023-10-30T13:27:06.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.508+0000 7f586569a640 1 -- 172.21.15.125:0/236075314 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58601f27e0 con 0x7f58600bf670 2023-10-30T13:27:06.522 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.512+0000 7f586569a640 1 -- 172.21.15.125:0/236075314 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f586004e170 con 0x7f58600bf670 2023-10-30T13:27:06.523 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.512+0000 7f58577fe640 1 -- 172.21.15.125:0/236075314 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f58480073f0 con 0x7f58600bf670 2023-10-30T13:27:06.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.512+0000 7f58577fe640 1 --2- 172.21.15.125:0/236075314 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58400664a0 0x7f5840068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:06.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.512+0000 7f58577fe640 1 -- 172.21.15.125:0/236075314 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8662+0+0 (secure 0 0 0) 0x7f58480842e0 con 0x7f58600bf670 2023-10-30T13:27:06.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.512+0000 7f585effd640 1 --2- 172.21.15.125:0/236075314 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58400664a0 0x7f5840068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:06.524 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.512+0000 7f585effd640 1 --2- 172.21.15.125:0/236075314 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58400664a0 0x7f5840068960 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f585000b9b0 tx=0x7f5850008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:06.525 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.516+0000 7f58577fe640 1 -- 172.21.15.125:0/236075314 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f5848050b90 con 0x7f58600bf670 2023-10-30T13:27:06.706 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:06.700+0000 7f586569a640 1 -- 172.21.15.125:0/236075314 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f586014c2a0 con 0x7f58600bf670 2023-10-30T13:27:07.382 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.376+0000 7f58577fe640 1 -- 172.21.15.125:0/236075314 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v271) v1 ==== 104+0+0 (secure 0 0 0) 0x7f58480541b0 con 0x7f58600bf670 2023-10-30T13:27:07.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.380+0000 7f586569a640 1 -- 172.21.15.125:0/236075314 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58400664a0 msgr2=0x7f5840068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:07.386 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.380+0000 7f586569a640 1 --2- 172.21.15.125:0/236075314 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58400664a0 0x7f5840068960 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f585000b9b0 tx=0x7f5850008040 comp rx=0 tx=0).stop 2023-10-30T13:27:07.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.380+0000 7f586569a640 1 -- 172.21.15.125:0/236075314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58600bf670 msgr2=0x7f586006a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:07.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.380+0000 7f586569a640 1 --2- 172.21.15.125:0/236075314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58600bf670 0x7f586006a670 secure :-1 s=READY pgs=2234 cs=0 l=1 rev1=1 crypto rx=0x7f584800c500 tx=0x7f584800c9c0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.380+0000 7f586569a640 1 -- 172.21.15.125:0/236075314 shutdown_connections 2023-10-30T13:27:07.387 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.380+0000 7f586569a640 1 --2- 172.21.15.125:0/236075314 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f58400664a0 0x7f5840068960 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.380+0000 7f586569a640 1 --2- 172.21.15.125:0/236075314 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f5860151730 0x7f5860156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.380+0000 7f586569a640 1 --2- 172.21.15.125:0/236075314 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f58600bf670 0x7f586006a670 unknown :-1 s=CLOSED pgs=2234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.388 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.380+0000 7f586569a640 1 --2- 172.21.15.125:0/236075314 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f586006df70 0x7f586014b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.380+0000 7f586569a640 1 -- 172.21.15.125:0/236075314 >> 172.21.15.125:0/236075314 conn(0x7f58600cc3e0 msgr2=0x7f5860150410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:07.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.380+0000 7f586569a640 1 -- 172.21.15.125:0/236075314 shutdown_connections 2023-10-30T13:27:07.389 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.380+0000 7f586569a640 1 -- 172.21.15.125:0/236075314 wait complete. 2023-10-30T13:27:07.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-30T13:27:07.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.548+0000 7f2e07380640 1 Processor -- start 2023-10-30T13:27:07.555 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.548+0000 7f2e07380640 1 -- start start 2023-10-30T13:27:07.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e07380640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2e001517f0 0x7f2e00157620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:07.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e07380640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e00158b10 0x7f2e000c6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:07.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e07380640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7f2e0015ac10 0x7f2e000c6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:07.557 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e07380640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e0015bbf0 con 0x7f2e001517f0 2023-10-30T13:27:07.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e07380640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e00071ed0 con 0x7f2e0015ac10 2023-10-30T13:27:07.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e07380640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e000c69d0 con 0x7f2e00158b10 2023-10-30T13:27:07.558 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e0511c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2e001517f0 0x7f2e00157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:07.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e0511c640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7f2e001517f0 0x7f2e00157620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53888/0 (socket says 172.21.15.125:53888) 2023-10-30T13:27:07.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e0511c640 1 -- 172.21.15.125:0/710361949 learned_addr learned my addr 172.21.15.125:0/710361949 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:07.559 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e0491b640 1 --2- 172.21.15.125:0/710361949 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e00158b10 0x7f2e000c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:07.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e0591d640 1 --2- 172.21.15.125:0/710361949 >> v2:172.21.15.136:3300/0 conn(0x7f2e0015ac10 0x7f2e000c6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:07.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e0511c640 1 -- 172.21.15.125:0/710361949 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e00158b10 msgr2=0x7f2e000c6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:07.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e0511c640 1 --2- 172.21.15.125:0/710361949 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e00158b10 0x7f2e000c6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.560 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e0511c640 1 -- 172.21.15.125:0/710361949 >> v2:172.21.15.136:3300/0 conn(0x7f2e0015ac10 msgr2=0x7f2e000c6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:07.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e0511c640 1 --2- 172.21.15.125:0/710361949 >> v2:172.21.15.136:3300/0 conn(0x7f2e0015ac10 0x7f2e000c6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.561 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.552+0000 7f2e0511c640 1 -- 172.21.15.125:0/710361949 --> v2:172.21.15.125:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0006c810 con 0x7f2e001517f0 2023-10-30T13:27:07.562 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2e0511c640 1 --2- 172.21.15.125:0/710361949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e001517f0 0x7f2e00157620 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f2e000b0760 tx=0x7f2dfc01c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=624ef877d3faa620 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:07.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2df7fff640 1 -- 172.21.15.125:0/710361949 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc0ad680 con 0x7f2e001517f0 2023-10-30T13:27:07.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2df7fff640 1 -- 172.21.15.125:0/710361949 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f2dfc0ad7e0 con 0x7f2e001517f0 2023-10-30T13:27:07.563 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2df7fff640 1 -- 172.21.15.125:0/710361949 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc0adb90 con 0x7f2e001517f0 2023-10-30T13:27:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2e07380640 1 -- 172.21.15.125:0/710361949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e001517f0 msgr2=0x7f2e00157620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2e07380640 1 --2- 172.21.15.125:0/710361949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e001517f0 0x7f2e00157620 secure :-1 s=READY pgs=2235 cs=0 l=1 rev1=1 crypto rx=0x7f2e000b0760 tx=0x7f2dfc01c4b0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2e07380640 1 -- 172.21.15.125:0/710361949 shutdown_connections 2023-10-30T13:27:07.564 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2e07380640 1 --2- 172.21.15.125:0/710361949 >> v2:172.21.15.136:3300/0 conn(0x7f2e0015ac10 0x7f2e000c6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2e07380640 1 --2- 172.21.15.125:0/710361949 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e00158b10 0x7f2e000c6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2e07380640 1 --2- 172.21.15.125:0/710361949 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e001517f0 0x7f2e00157620 unknown :-1 s=CLOSED pgs=2235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.565 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2e07380640 1 -- 172.21.15.125:0/710361949 >> 172.21.15.125:0/710361949 conn(0x7f2e000c7910 msgr2=0x7f2e000c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2e07380640 1 -- 172.21.15.125:0/710361949 shutdown_connections 2023-10-30T13:27:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2e07380640 1 -- 172.21.15.125:0/710361949 wait complete. 2023-10-30T13:27:07.566 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2e07380640 1 Processor -- start 2023-10-30T13:27:07.567 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2e07380640 1 -- start start 2023-10-30T13:27:07.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.556+0000 7f2e07380640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e001517f0 0x7f2e000c0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:07.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e07380640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e00158b10 0x7f2e001506d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:07.569 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e07380640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e0015ac10 0x7f2e00150c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:07.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e07380640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e000c16a0 con 0x7f2e00158b10 2023-10-30T13:27:07.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e07380640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e000c4b80 con 0x7f2e0015ac10 2023-10-30T13:27:07.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e07380640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e001f3450 con 0x7f2e001517f0 2023-10-30T13:27:07.570 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e0491b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e00158b10 0x7f2e001506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:07.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e0491b640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e00158b10 0x7f2e001506d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.125:3300/0 says I am v2:172.21.15.125:53904/0 (socket says 172.21.15.125:53904) 2023-10-30T13:27:07.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e0491b640 1 -- 172.21.15.125:0/1799647397 learned_addr learned my addr 172.21.15.125:0/1799647397 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:07.571 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e0511c640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e001517f0 0x7f2e000c0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e0591d640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e0015ac10 0x7f2e00150c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e0491b640 1 -- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e001517f0 msgr2=0x7f2e000c0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:07.572 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e0491b640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e001517f0 0x7f2e000c0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e0491b640 1 -- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e0015ac10 msgr2=0x7f2e00150c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e0491b640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e0015ac10 0x7f2e00150c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e0491b640 1 -- 172.21.15.125:0/1799647397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc0094d0 con 0x7f2e00158b10 2023-10-30T13:27:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e0511c640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e001517f0 0x7f2e000c0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-30T13:27:07.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e0491b640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e00158b10 0x7f2e001506d0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f2df000d830 tx=0x7f2df000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:07.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2df5ffb640 1 -- 172.21.15.125:0/1799647397 <== mon.0 v2:172.21.15.125:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df00096a0 con 0x7f2e00158b10 2023-10-30T13:27:07.574 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2df5ffb640 1 -- 172.21.15.125:0/1799647397 <== mon.0 v2:172.21.15.125:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f2df0009800 con 0x7f2e00158b10 2023-10-30T13:27:07.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e07380640 1 -- 172.21.15.125:0/1799647397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0014f620 con 0x7f2e00158b10 2023-10-30T13:27:07.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2df5ffb640 1 -- 172.21.15.125:0/1799647397 <== mon.0 v2:172.21.15.125:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0009b90 con 0x7f2e00158b10 2023-10-30T13:27:07.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e07380640 1 -- 172.21.15.125:0/1799647397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e0014ff60 con 0x7f2e00158b10 2023-10-30T13:27:07.575 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2e07380640 1 -- 172.21.15.125:0/1799647397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e0006f110 con 0x7f2e00158b10 2023-10-30T13:27:07.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.560+0000 7f2df5ffb640 1 -- 172.21.15.125:0/1799647397 <== mon.0 v2:172.21.15.125:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7f2df0014410 con 0x7f2e00158b10 2023-10-30T13:27:07.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.564+0000 7f2df5ffb640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2de40664a0 0x7f2de4068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:07.576 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.564+0000 7f2df5ffb640 1 -- 172.21.15.125:0/1799647397 <== mon.0 v2:172.21.15.125:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8662+0+0 (secure 0 0 0) 0x7f2df00853f0 con 0x7f2e00158b10 2023-10-30T13:27:07.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.564+0000 7f2e0511c640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2de40664a0 0x7f2de4068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:07.577 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.564+0000 7f2e0511c640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2de40664a0 0x7f2de4068960 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc0af000 tx=0x7f2dfc008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:07.578 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.568+0000 7f2df5ffb640 1 -- 172.21.15.125:0/1799647397 <== mon.0 v2:172.21.15.125:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7f2df0051c90 con 0x7f2e00158b10 2023-10-30T13:27:07.723 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:07.716+0000 7f2e07380640 1 -- 172.21.15.125:0/1799647397 --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2e0004e170 con 0x7f2e00158b10 2023-10-30T13:27:08.408 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.400+0000 7f2df5ffb640 1 -- 172.21.15.125:0/1799647397 <== mon.0 v2:172.21.15.125:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v87) v1 ==== 181+0+0 (secure 0 0 0) 0x7f2df000b440 con 0x7f2e00158b10 2023-10-30T13:27:08.408 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_metadata' removed 2023-10-30T13:27:08.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.404+0000 7f2e07380640 1 -- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2de40664a0 msgr2=0x7f2de4068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:08.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.404+0000 7f2e07380640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2de40664a0 0x7f2de4068960 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc0af000 tx=0x7f2dfc008040 comp rx=0 tx=0).stop 2023-10-30T13:27:08.411 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.404+0000 7f2e07380640 1 -- 172.21.15.125:0/1799647397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e00158b10 msgr2=0x7f2e001506d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:08.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.404+0000 7f2e07380640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e00158b10 0x7f2e001506d0 secure :-1 s=READY pgs=2236 cs=0 l=1 rev1=1 crypto rx=0x7f2df000d830 tx=0x7f2df000dcf0 comp rx=0 tx=0).stop 2023-10-30T13:27:08.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.404+0000 7f2e07380640 1 -- 172.21.15.125:0/1799647397 shutdown_connections 2023-10-30T13:27:08.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.404+0000 7f2e07380640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7f2de40664a0 0x7f2de4068960 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:08.412 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.404+0000 7f2e07380640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7f2e0015ac10 0x7f2e00150c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:08.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.404+0000 7f2e07380640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7f2e00158b10 0x7f2e001506d0 unknown :-1 s=CLOSED pgs=2236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:08.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.404+0000 7f2e07380640 1 --2- 172.21.15.125:0/1799647397 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7f2e001517f0 0x7f2e000c0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:08.413 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.404+0000 7f2e07380640 1 -- 172.21.15.125:0/1799647397 >> 172.21.15.125:0/1799647397 conn(0x7f2e000c7910 msgr2=0x7f2e001438c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:08.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.404+0000 7f2e07380640 1 -- 172.21.15.125:0/1799647397 shutdown_connections 2023-10-30T13:27:08.414 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.404+0000 7f2e07380640 1 -- 172.21.15.125:0/1799647397 wait complete. 2023-10-30T13:27:08.434 DEBUG:teuthology.orchestra.run.smithi125:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-30T13:27:08.617 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.608+0000 7fdda010f640 1 Processor -- start 2023-10-30T13:27:08.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.612+0000 7fdda010f640 1 -- start start 2023-10-30T13:27:08.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.612+0000 7fdda010f640 1 --2- >> v2:172.21.15.125:3300/0 conn(0x7fdd981536d0 0x7fdd980cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:08.620 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.612+0000 7fdda010f640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdd9815d9d0 0x7fdd980c4000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:08.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.616+0000 7fdda010f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd980c4540 con 0x7fdd98159760 2023-10-30T13:27:08.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.616+0000 7fdda010f640 1 -- --> v2:172.21.15.125:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd9815a210 con 0x7fdd981536d0 2023-10-30T13:27:08.621 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.616+0000 7fdda010f640 1 -- --> v2:172.21.15.136:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd9815a4f0 con 0x7fdd9815d9d0 2023-10-30T13:27:08.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.616+0000 7fdd9d6aa640 1 --2- >> v2:172.21.15.136:3300/0 conn(0x7fdd9815d9d0 0x7fdd980c4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:08.622 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.616+0000 7fdd9e6ac640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fdd98159760 0x7fdd980cf090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.125:44644/0 (socket says 172.21.15.125:44644) 2023-10-30T13:27:08.623 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.616+0000 7fdd9e6ac640 1 -- 172.21.15.125:0/1109742524 learned_addr learned my addr 172.21.15.125:0/1109742524 (peer_addr_for_me v1:172.21.15.125:0/0) 2023-10-30T13:27:08.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.616+0000 7fdd9cea9640 1 -- 172.21.15.125:0/1109742524 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152604094 0 0) 0x7fdd980c4540 con 0x7fdd98159760 2023-10-30T13:27:08.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.616+0000 7fdd9cea9640 1 -- 172.21.15.125:0/1109742524 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd84003530 con 0x7fdd98159760 2023-10-30T13:27:08.624 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.616+0000 7fdd9d6aa640 1 -- 172.21.15.125:0/1109742524 >> v2:172.21.15.125:3300/0 conn(0x7fdd981536d0 msgr2=0x7fdd980cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-30T13:27:08.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.616+0000 7fdd9d6aa640 1 --2- 172.21.15.125:0/1109742524 >> v2:172.21.15.125:3300/0 conn(0x7fdd981536d0 0x7fdd980cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:08.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.616+0000 7fdd9d6aa640 1 -- 172.21.15.125:0/1109742524 >> v1:172.21.15.136:6789/0 conn(0x7fdd98159760 legacy=0x7fdd980cf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:08.625 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.620+0000 7fdd9d6aa640 1 -- 172.21.15.125:0/1109742524 --> v2:172.21.15.136:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd9815cef0 con 0x7fdd9815d9d0 2023-10-30T13:27:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.624+0000 7fdd9d6aa640 1 --2- 172.21.15.125:0/1109742524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd9815d9d0 0x7fdd980c4000 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fdd980cfe30 tx=0x7fdd8801c2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af42970dbc8e535d server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:08.629 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.624+0000 7fdd9cea9640 1 -- 172.21.15.125:0/1109742524 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd90001a00 con 0x7fdd9815d9d0 2023-10-30T13:27:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.624+0000 7fdd9cea9640 1 -- 172.21.15.125:0/1109742524 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fdd880a7050 con 0x7fdd9815d9d0 2023-10-30T13:27:08.630 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.624+0000 7fdd9cea9640 1 -- 172.21.15.125:0/1109742524 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd880ab640 con 0x7fdd9815d9d0 2023-10-30T13:27:08.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.624+0000 7fdda010f640 1 -- 172.21.15.125:0/1109742524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd9815d9d0 msgr2=0x7fdd980c4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:08.631 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.624+0000 7fdda010f640 1 --2- 172.21.15.125:0/1109742524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd9815d9d0 0x7fdd980c4000 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fdd980cfe30 tx=0x7fdd8801c2a0 comp rx=0 tx=0).stop 2023-10-30T13:27:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.624+0000 7fdda010f640 1 -- 172.21.15.125:0/1109742524 shutdown_connections 2023-10-30T13:27:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.624+0000 7fdda010f640 1 --2- 172.21.15.125:0/1109742524 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd9815d9d0 0x7fdd980c4000 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.624+0000 7fdda010f640 1 --2- 172.21.15.125:0/1109742524 >> v2:172.21.15.125:3300/0 conn(0x7fdd981536d0 0x7fdd980cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:08.632 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.624+0000 7fdda010f640 1 -- 172.21.15.125:0/1109742524 >> 172.21.15.125:0/1109742524 conn(0x7fdd9806bf70 msgr2=0x7fdd9806c380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:08.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.628+0000 7fdda010f640 1 -- 172.21.15.125:0/1109742524 shutdown_connections 2023-10-30T13:27:08.633 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.628+0000 7fdda010f640 1 -- 172.21.15.125:0/1109742524 wait complete. 2023-10-30T13:27:08.635 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.628+0000 7fdda010f640 1 Processor -- start 2023-10-30T13:27:08.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.628+0000 7fdda010f640 1 -- start start 2023-10-30T13:27:08.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdda010f640 1 --2- >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd981536d0 0x7fdd980cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:08.637 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdda010f640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd98159760 0x7fdd980b3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:08.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdda010f640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd9815d9d0 0x7fdd981f6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:08.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdda010f640 1 -- --> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd980cefe0 con 0x7fdd981536d0 2023-10-30T13:27:08.638 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdda010f640 1 -- --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd981f99e0 con 0x7fdd9815d9d0 2023-10-30T13:27:08.639 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdda010f640 1 -- --> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd981fa990 con 0x7fdd98159760 2023-10-30T13:27:08.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdd9d6aa640 1 --2- >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd98159760 0x7fdd980b3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:08.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdd9e6ac640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd9815d9d0 0x7fdd981f6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:08.641 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdd9e6ac640 1 --2- >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd9815d9d0 0x7fdd981f6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.136:3300/0 says I am v2:172.21.15.125:39956/0 (socket says 172.21.15.125:39956) 2023-10-30T13:27:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdd9e6ac640 1 -- 172.21.15.125:0/3484341159 learned_addr learned my addr 172.21.15.125:0/3484341159 (peer_addr_for_me v2:172.21.15.125:0/0) 2023-10-30T13:27:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdd9deab640 1 --2- 172.21.15.125:0/3484341159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd981536d0 0x7fdd980cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:08.642 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdd9e6ac640 1 -- 172.21.15.125:0/3484341159 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd98159760 msgr2=0x7fdd980b3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdd9e6ac640 1 --2- 172.21.15.125:0/3484341159 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd98159760 0x7fdd980b3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdd9e6ac640 1 -- 172.21.15.125:0/3484341159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd981536d0 msgr2=0x7fdd980cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdd9e6ac640 1 --2- 172.21.15.125:0/3484341159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd981536d0 0x7fdd980cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:08.643 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdd9e6ac640 1 -- 172.21.15.125:0/3484341159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd88009420 con 0x7fdd9815d9d0 2023-10-30T13:27:08.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdd9deab640 1 --2- 172.21.15.125:0/3484341159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd981536d0 0x7fdd980cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-30T13:27:08.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdd9e6ac640 1 --2- 172.21.15.125:0/3484341159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd9815d9d0 0x7fdd981f6440 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000cb60 tx=0x7fdd9000a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:08.644 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdd8e7fc640 1 -- 172.21.15.125:0/3484341159 <== mon.1 v2:172.21.15.136:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9000ce30 con 0x7fdd9815d9d0 2023-10-30T13:27:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdd8e7fc640 1 -- 172.21.15.125:0/3484341159 <== mon.1 v2:172.21.15.136:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fdd9000ab40 con 0x7fdd9815d9d0 2023-10-30T13:27:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.632+0000 7fdda010f640 1 -- 172.21.15.125:0/3484341159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd981fa500 con 0x7fdd9815d9d0 2023-10-30T13:27:08.645 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.636+0000 7fdd8e7fc640 1 -- 172.21.15.125:0/3484341159 <== mon.1 v2:172.21.15.136:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd900070d0 con 0x7fdd9815d9d0 2023-10-30T13:27:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.636+0000 7fdda010f640 1 -- 172.21.15.125:0/3484341159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd980ced00 con 0x7fdd9815d9d0 2023-10-30T13:27:08.646 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.636+0000 7fdda010f640 1 -- 172.21.15.125:0/3484341159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd981fa6b0 con 0x7fdd9815d9d0 2023-10-30T13:27:08.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.636+0000 7fdd8e7fc640 1 -- 172.21.15.125:0/3484341159 <== mon.1 v2:172.21.15.136:3300/0 4 ==== mgrmap(e 5) v1 ==== 85850+0+0 (secure 0 0 0) 0x7fdd90007230 con 0x7fdd9815d9d0 2023-10-30T13:27:08.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.636+0000 7fdd8e7fc640 1 --2- 172.21.15.125:0/3484341159 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdd7c0664a0 0x7fdd7c068960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-30T13:27:08.647 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.636+0000 7fdd8e7fc640 1 -- 172.21.15.125:0/3484341159 <== mon.1 v2:172.21.15.136:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8224+0+0 (secure 0 0 0) 0x7fdd900843a0 con 0x7fdd9815d9d0 2023-10-30T13:27:08.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.636+0000 7fdd9deab640 1 --2- 172.21.15.125:0/3484341159 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdd7c0664a0 0x7fdd7c068960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-30T13:27:08.648 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.640+0000 7fdd9deab640 1 --2- 172.21.15.125:0/3484341159 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdd7c0664a0 0x7fdd7c068960 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fdd9400a8b0 tx=0x7fdd94008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-30T13:27:08.649 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.644+0000 7fdd8e7fc640 1 -- 172.21.15.125:0/3484341159 <== mon.1 v2:172.21.15.136:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+172159 (secure 0 0 0) 0x7fdd90050d20 con 0x7fdd9815d9d0 2023-10-30T13:27:08.896 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:08.892+0000 7fdda010f640 1 -- 172.21.15.125:0/3484341159 --> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdd981fa6b0 con 0x7fdd9815d9d0 2023-10-30T13:27:09.425 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:09.420+0000 7fdd8e7fc640 1 -- 172.21.15.125:0/3484341159 <== mon.1 v2:172.21.15.136:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdd981fa6b0 con 0x7fdd9815d9d0 2023-10-30T13:27:09.425 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'cephfs_data' removed 2023-10-30T13:27:09.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:09.420+0000 7fdda010f640 1 -- 172.21.15.125:0/3484341159 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdd7c0664a0 msgr2=0x7fdd7c068960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:09.427 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:09.420+0000 7fdda010f640 1 --2- 172.21.15.125:0/3484341159 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdd7c0664a0 0x7fdd7c068960 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fdd9400a8b0 tx=0x7fdd94008040 comp rx=0 tx=0).stop 2023-10-30T13:27:09.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:09.420+0000 7fdda010f640 1 -- 172.21.15.125:0/3484341159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd9815d9d0 msgr2=0x7fdd981f6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-30T13:27:09.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:09.420+0000 7fdda010f640 1 --2- 172.21.15.125:0/3484341159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd9815d9d0 0x7fdd981f6440 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fdd9000cb60 tx=0x7fdd9000a430 comp rx=0 tx=0).stop 2023-10-30T13:27:09.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:09.420+0000 7fdda010f640 1 -- 172.21.15.125:0/3484341159 shutdown_connections 2023-10-30T13:27:09.428 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:09.420+0000 7fdda010f640 1 --2- 172.21.15.125:0/3484341159 >> [v2:172.21.15.136:6832/999226929,v1:172.21.15.136:6833/999226929] conn(0x7fdd7c0664a0 0x7fdd7c068960 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:09.420+0000 7fdda010f640 1 --2- 172.21.15.125:0/3484341159 >> [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] conn(0x7fdd9815d9d0 0x7fdd981f6440 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:09.420+0000 7fdda010f640 1 --2- 172.21.15.125:0/3484341159 >> [v2:172.21.15.136:3301/0,v1:172.21.15.136:6790/0] conn(0x7fdd98159760 0x7fdd980b3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.429 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:09.420+0000 7fdda010f640 1 --2- 172.21.15.125:0/3484341159 >> [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] conn(0x7fdd981536d0 0x7fdd980cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-30T13:27:09.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:09.420+0000 7fdda010f640 1 -- 172.21.15.125:0/3484341159 >> 172.21.15.125:0/3484341159 conn(0x7fdd9806bf70 msgr2=0x7fdd980c9fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-30T13:27:09.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:09.420+0000 7fdda010f640 1 -- 172.21.15.125:0/3484341159 shutdown_connections 2023-10-30T13:27:09.430 INFO:teuthology.orchestra.run.smithi125.stderr:2023-10-30T13:27:09.420+0000 7fdda010f640 1 -- 172.21.15.125:0/3484341159 wait complete. 2023-10-30T13:27:09.447 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-30T13:27:09.448 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-30T13:27:09.448 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:09.461 INFO:tasks.ceph.mds.a:Stopped 2023-10-30T13:27:09.461 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-30T13:27:09.461 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:09.515 INFO:tasks.ceph.mds.c:Stopped 2023-10-30T13:27:09.516 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2023-10-30T13:27:09.516 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:09.571 INFO:tasks.ceph.mds.e:Stopped 2023-10-30T13:27:09.572 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-30T13:27:09.572 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:09.585 INFO:tasks.ceph.mds.b:Stopped 2023-10-30T13:27:09.585 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-30T13:27:09.586 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:09.638 INFO:tasks.ceph.mds.d:Stopped 2023-10-30T13:27:09.639 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2023-10-30T13:27:09.639 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:09.695 INFO:tasks.ceph.mds.f:Stopped 2023-10-30T13:27:09.696 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-30T13:27:09.696 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-30T13:27:09.697 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:15.799 INFO:tasks.ceph.osd.0:Stopped 2023-10-30T13:27:15.800 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-30T13:27:15.800 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:15.881 INFO:tasks.ceph.osd.1:Stopped 2023-10-30T13:27:15.882 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-30T13:27:15.882 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:16.797 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2023-10-30T13:27:19.004 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:19.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:21.984 INFO:tasks.ceph.osd.2:Stopped 2023-10-30T13:27:21.984 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-30T13:27:21.985 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:22.077 INFO:tasks.ceph.osd.3:Stopped 2023-10-30T13:27:22.078 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-30T13:27:22.078 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:22.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2023-10-30T13:27:22.806 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2023-10-30T13:27:28.180 INFO:tasks.ceph.osd.4:Stopped 2023-10-30T13:27:28.181 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-30T13:27:28.181 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:28.246 INFO:tasks.ceph.osd.5:Stopped 2023-10-30T13:27:28.246 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-30T13:27:28.247 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:28.327 INFO:tasks.ceph.osd.6:Stopped 2023-10-30T13:27:28.328 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-30T13:27:28.328 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:28.611 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2023-10-30T13:27:28.612 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2023-10-30T13:27:34.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2023-10-30T13:27:34.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2023-10-30T13:27:34.430 INFO:tasks.ceph.osd.7:Stopped 2023-10-30T13:27:34.431 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-30T13:27:34.431 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-30T13:27:34.432 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:34.484 INFO:tasks.ceph.mgr.y:Stopped 2023-10-30T13:27:34.484 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-30T13:27:34.485 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:34.515 INFO:tasks.ceph.mgr.x:Stopped 2023-10-30T13:27:34.515 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-30T13:27:34.516 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-30T13:27:34.516 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:34.540 INFO:tasks.ceph.mon.a:Stopped 2023-10-30T13:27:34.540 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-30T13:27:34.541 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:34.577 INFO:tasks.ceph.mon.b:Stopped 2023-10-30T13:27:34.578 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-30T13:27:34.578 INFO:teuthology.orchestra.run:waiting for 300 2023-10-30T13:27:34.635 INFO:tasks.ceph.mon.c:Stopped 2023-10-30T13:27:34.636 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-30T13:27:34.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v 'Replacing daemon mds' | head -n 1 2023-10-30T13:27:34.676 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi125.front.sepia.ceph.com 2023-10-30T13:27:34.676 DEBUG:teuthology.orchestra.run.smithi125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-30T13:27:35.221 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi125.front.sepia.ceph.com 2023-10-30T13:27:35.221 DEBUG:teuthology.orchestra.run.smithi125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-30T13:27:35.450 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi125.front.sepia.ceph.com 2023-10-30T13:27:35.451 DEBUG:teuthology.orchestra.run.smithi125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-30T13:27:35.693 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi125.front.sepia.ceph.com 2023-10-30T13:27:35.694 DEBUG:teuthology.orchestra.run.smithi125:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-30T13:27:35.893 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi136.front.sepia.ceph.com 2023-10-30T13:27:35.894 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-30T13:27:36.227 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi136.front.sepia.ceph.com 2023-10-30T13:27:36.227 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-30T13:27:36.452 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi136.front.sepia.ceph.com 2023-10-30T13:27:36.452 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-30T13:27:36.632 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi136.front.sepia.ceph.com 2023-10-30T13:27:36.633 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-30T13:27:36.797 INFO:tasks.ceph:Archiving mon data... 2023-10-30T13:27:36.798 DEBUG:teuthology.misc:Transferring archived files from smithi125:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019/data/mon.a.tgz 2023-10-30T13:27:36.799 DEBUG:teuthology.orchestra.run.smithi125:> mktemp 2023-10-30T13:27:36.808 INFO:teuthology.orchestra.run.smithi125.stdout:/tmp/tmp.cO2FqADlRh 2023-10-30T13:27:36.808 DEBUG:teuthology.orchestra.run.smithi125:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.cO2FqADlRh 2023-10-30T13:27:36.946 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 0666 /tmp/tmp.cO2FqADlRh 2023-10-30T13:27:37.024 DEBUG:teuthology.orchestra.remote:smithi125:/tmp/tmp.cO2FqADlRh is 397KB 2023-10-30T13:27:37.090 DEBUG:teuthology.orchestra.run.smithi125:> rm -fr /tmp/tmp.cO2FqADlRh 2023-10-30T13:27:37.099 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019/data/mon.b.tgz 2023-10-30T13:27:37.099 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2023-10-30T13:27:37.105 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.F6LJ06zPTb 2023-10-30T13:27:37.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.F6LJ06zPTb 2023-10-30T13:27:37.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.F6LJ06zPTb 2023-10-30T13:27:37.304 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.F6LJ06zPTb is 402KB 2023-10-30T13:27:37.362 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.F6LJ06zPTb 2023-10-30T13:27:37.368 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019/data/mon.c.tgz 2023-10-30T13:27:37.368 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2023-10-30T13:27:37.419 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.7gUbm2HTo9 2023-10-30T13:27:37.420 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.7gUbm2HTo9 2023-10-30T13:27:37.553 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.7gUbm2HTo9 2023-10-30T13:27:37.615 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.7gUbm2HTo9 is 402KB 2023-10-30T13:27:37.681 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.7gUbm2HTo9 2023-10-30T13:27:37.687 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-30T13:27:37.687 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T13:27:37.691 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-30T13:27:37.743 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-30T13:27:37.744 DEBUG:teuthology.orchestra.run.smithi125:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T13:27:37.754 DEBUG:teuthology.orchestra.run.smithi136:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-30T13:27:37.786 INFO:teuthology.orchestra.run.smithi125.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-30T13:27:37.808 INFO:teuthology.orchestra.run.smithi136.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-30T13:27:37.812 INFO:tasks.ceph:Archiving crash dumps... 2023-10-30T13:27:37.814 DEBUG:teuthology.misc:Transferring archived files from smithi125:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019/remote/smithi125/crash 2023-10-30T13:27:37.815 DEBUG:teuthology.orchestra.run.smithi125:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T13:27:37.859 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/crash to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019/remote/smithi136/crash 2023-10-30T13:27:37.860 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-30T13:27:37.871 INFO:tasks.ceph:Shutting down logrotate 2023-10-30T13:27:37.872 DEBUG:teuthology.orchestra.run.smithi125:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:37.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:37.959 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:37.980 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-30T13:27:38.016 INFO:tasks.ceph:Compressing logs... 2023-10-30T13:27:38.017 DEBUG:teuthology.orchestra.run.smithi125:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T13:27:38.026 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-30T13:27:38.043 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51014.log 2023-10-30T13:27:38.044 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67870.log 2023-10-30T13:27:38.045 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59924.log 2023-10-30T13:27:38.045 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51014.log.gz 2023-10-30T13:27:38.046 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67870.log: gzip -5 88.9% --verbose -- replaced with /var/log/ceph/ceph-client.admin.67870.log.gz 2023-10-30T13:27:38.046 INFO:teuthology.orchestra.run.smithi125.stderr: -- /var/log/ceph/ceph-client.admin.57157.log 2023-10-30T13:27:38.047 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70886.log 2023-10-30T13:27:38.048 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59924.log.gz 2023-10-30T13:27:38.049 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64540.log 2023-10-30T13:27:38.049 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57157.log.gz 2023-10-30T13:27:38.050 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63615.log 2023-10-30T13:27:38.050 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70886.log.gz 2023-10-30T13:27:38.052 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64540.log: gzip -5 --verbose 88.5% -- -- replaced with /var/log/ceph/ceph-client.admin.64540.log.gz /var/log/ceph/ceph-client.admin.62981.log 2023-10-30T13:27:38.052 INFO:teuthology.orchestra.run.smithi125.stderr: 2023-10-30T13:27:38.052 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58110.log 2023-10-30T13:27:38.053 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63615.log.gz 2023-10-30T13:27:38.053 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68696.log 2023-10-30T13:27:38.053 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62981.log.gz 2023-10-30T13:27:38.056 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58110.log: gzip -5 --verbose -- 88.4% /var/log/ceph/ceph-client.admin.44097.log 2023-10-30T13:27:38.056 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58110.log.gz 2023-10-30T13:27:38.056 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69613.log 2023-10-30T13:27:38.057 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.68696.log.gz 2023-10-30T13:27:38.057 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53397.log 2023-10-30T13:27:38.057 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44097.log.gz 2023-10-30T13:27:38.057 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52838.log 2023-10-30T13:27:38.058 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69613.log.gz 2023-10-30T13:27:38.060 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53397.log: gzip -5 89.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.53397.log.gz 2023-10-30T13:27:38.060 INFO:teuthology.orchestra.run.smithi125.stderr: /var/log/ceph/ceph-client.admin.58776.log 2023-10-30T13:27:38.061 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44688.log 2023-10-30T13:27:38.061 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52838.log.gz 2023-10-30T13:27:38.061 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51645.log 2023-10-30T13:27:38.062 INFO:teuthology.orchestra.run.smithi125.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.58776.log.gz 2023-10-30T13:27:38.062 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52602.log 2023-10-30T13:27:38.062 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44688.log.gz 2023-10-30T13:27:38.065 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51645.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51645.log.gz 2023-10-30T13:27:38.066 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58084.log 2023-10-30T13:27:38.066 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62409.log 2023-10-30T13:27:38.066 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52602.log.gz 2023-10-30T13:27:38.067 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66333.log 2023-10-30T13:27:38.067 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58084.log.gz 2023-10-30T13:27:38.067 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55362.log 2023-10-30T13:27:38.068 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62409.log.gz 2023-10-30T13:27:38.068 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51172.log 2023-10-30T13:27:38.068 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66333.log.gz 2023-10-30T13:27:38.069 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48064.log 2023-10-30T13:27:38.069 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55362.log.gz 2023-10-30T13:27:38.073 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67449.log 2023-10-30T13:27:38.073 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51172.log.gz 2023-10-30T13:27:38.073 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57024.log 2023-10-30T13:27:38.074 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48064.log.gz 2023-10-30T13:27:38.074 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50288.log 2023-10-30T13:27:38.075 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67449.log.gz 2023-10-30T13:27:38.075 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59681.log 2023-10-30T13:27:38.075 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57024.log.gz 2023-10-30T13:27:38.076 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51066.log 2023-10-30T13:27:38.076 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50288.log.gz 2023-10-30T13:27:38.076 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70411.log 2023-10-30T13:27:38.077 INFO:teuthology.orchestra.run.smithi125.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.59681.log.gz 2023-10-30T13:27:38.077 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60329.log 2023-10-30T13:27:38.077 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51066.log.gz 2023-10-30T13:27:38.086 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38715.log 2023-10-30T13:27:38.086 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-30T13:27:38.087 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-30T13:27:38.087 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.38715.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38787.log 2023-10-30T13:27:38.087 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38811.log 2023-10-30T13:27:38.087 INFO:teuthology.orchestra.run.smithi136.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38715.log.gz 2023-10-30T13:27:38.088 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.38787.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38667.log 2023-10-30T13:27:38.088 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.38811.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38787.log.gz 2023-10-30T13:27:38.088 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-30T13:27:38.089 INFO:teuthology.orchestra.run.smithi136.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38811.log.gz 2023-10-30T13:27:38.089 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.38667.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-30T13:27:38.089 INFO:teuthology.orchestra.run.smithi136.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38667.log.gz 2023-10-30T13:27:38.090 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2023-10-30T13:27:38.090 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T13:27:38.090 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-30T13:27:38.091 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38763.log 2023-10-30T13:27:38.091 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T13:27:38.091 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.38763.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.38763.log.gz 2023-10-30T13:27:38.092 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38739.log 2023-10-30T13:27:38.098 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.audit.log: 94.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T13:27:38.098 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-30T13:27:38.098 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.38739.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38739.log.gz 2023-10-30T13:27:38.099 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38643.log 2023-10-30T13:27:38.099 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mds.d.log: 93.0% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T13:27:38.099 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-30T13:27:38.100 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.38643.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38643.log.gz 2023-10-30T13:27:38.100 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-30T13:27:38.104 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50235.log 2023-10-30T13:27:38.104 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70411.log.gz 2023-10-30T13:27:38.104 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56229.log 2023-10-30T13:27:38.105 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60329.log.gz 2023-10-30T13:27:38.105 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50235.log: gzip -5 --verbose -- 87.8% /var/log/ceph/ceph-client.admin.57534.log -- replaced with /var/log/ceph/ceph-client.admin.50235.log.gz 2023-10-30T13:27:38.105 INFO:teuthology.orchestra.run.smithi125.stderr: 2023-10-30T13:27:38.105 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56229.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65559.log 2023-10-30T13:27:38.106 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56229.log.gz 2023-10-30T13:27:38.106 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70673.log 2023-10-30T13:27:38.106 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57534.log.gz 2023-10-30T13:27:38.107 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65559.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-30T13:27:38.107 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65559.log.gz 2023-10-30T13:27:38.107 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54968.log 2023-10-30T13:27:38.108 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70673.log.gz 2023-10-30T13:27:38.108 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54810.log 2023-10-30T13:27:38.108 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54968.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.54968.log.gz 2023-10-30T13:27:38.109 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45442.log 2023-10-30T13:27:38.109 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54676.log 2023-10-30T13:27:38.109 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54810.log.gz 2023-10-30T13:27:38.109 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48511.log 2023-10-30T13:27:38.110 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45442.log.gz 2023-10-30T13:27:38.110 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61707.log 2023-10-30T13:27:38.110 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54676.log.gz 2023-10-30T13:27:38.111 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48511.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48511.log.gz 2023-10-30T13:27:38.111 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57838.log 2023-10-30T13:27:38.111 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67529.log 2023-10-30T13:27:38.112 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.61707.log.gz 2023-10-30T13:27:38.112 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57838.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.57838.log.gz 2023-10-30T13:27:38.112 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61163.log 2023-10-30T13:27:38.112 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68444.log 2023-10-30T13:27:38.113 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67529.log.gz 2023-10-30T13:27:38.113 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61163.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61163.log.gz 2023-10-30T13:27:38.113 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59478.log 2023-10-30T13:27:38.114 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44892.log 2023-10-30T13:27:38.114 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68444.log.gz 2023-10-30T13:27:38.114 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59478.log: gzip 89.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.59478.log.gz 2023-10-30T13:27:38.115 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.63219.log 2023-10-30T13:27:38.115 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44892.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44892.log.gz 2023-10-30T13:27:38.115 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56610.log 2023-10-30T13:27:38.115 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63219.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63219.log.gz 2023-10-30T13:27:38.116 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55888.log 2023-10-30T13:27:38.116 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56610.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56610.log.gz 2023-10-30T13:27:38.116 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68774.log 2023-10-30T13:27:38.117 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55888.log: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.55888.log.gz 2023-10-30T13:27:38.117 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.67083.log 2023-10-30T13:27:38.117 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68774.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68774.log.gz 2023-10-30T13:27:38.118 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61733.log 2023-10-30T13:27:38.118 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67083.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.67083.log.gz 2023-10-30T13:27:38.118 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58972.log 2023-10-30T13:27:38.118 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61733.log: gzip -5 --verbose 87.7% -- -- replaced with /var/log/ceph/ceph-client.admin.61733.log.gz 2023-10-30T13:27:38.119 INFO:teuthology.orchestra.run.smithi125.stderr: /var/log/ceph/ceph-client.admin.65205.log 2023-10-30T13:27:38.119 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50078.log 2023-10-30T13:27:38.119 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58972.log.gz 2023-10-30T13:27:38.119 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46857.log 2023-10-30T13:27:38.120 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65205.log.gz 2023-10-30T13:27:38.120 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50078.log: gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.63689.log -- replaced with /var/log/ceph/ceph-client.admin.50078.log.gz 2023-10-30T13:27:38.120 INFO:teuthology.orchestra.run.smithi125.stderr: 2023-10-30T13:27:38.121 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66252.log 2023-10-30T13:27:38.121 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46857.log.gz 2023-10-30T13:27:38.121 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44879.log 2023-10-30T13:27:38.122 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63689.log.gz 2023-10-30T13:27:38.122 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50050.log 2023-10-30T13:27:38.122 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66252.log.gz 2023-10-30T13:27:38.124 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mds.b.log: 94.5% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-30T13:27:38.125 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-30T13:27:38.125 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38691.log 2023-10-30T13:27:38.136 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49612.log 2023-10-30T13:27:38.136 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44879.log.gz 2023-10-30T13:27:38.137 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58885.log 2023-10-30T13:27:38.137 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50050.log.gz 2023-10-30T13:27:38.137 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49612.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49612.log.gz 2023-10-30T13:27:38.137 INFO:teuthology.orchestra.run.smithi125.stderr: 95.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-30T13:27:38.138 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56281.log 2023-10-30T13:27:38.138 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55808.log 2023-10-30T13:27:38.138 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58885.log.gz 2023-10-30T13:27:38.139 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56281.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56281.log.gz 2023-10-30T13:27:38.139 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56584.log 2023-10-30T13:27:38.139 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68548.log 2023-10-30T13:27:38.141 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55808.log.gz 2023-10-30T13:27:38.142 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57508.log 2023-10-30T13:27:38.142 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56584.log.gz 2023-10-30T13:27:38.142 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68548.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.68548.log.gz 2023-10-30T13:27:38.142 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60216.log 2023-10-30T13:27:38.143 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57508.log: gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.57508.log.gz --verbose 2023-10-30T13:27:38.143 INFO:teuthology.orchestra.run.smithi125.stderr: -- /var/log/ceph/ceph-client.admin.56852.log 2023-10-30T13:27:38.143 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59896.log 2023-10-30T13:27:38.144 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60216.log.gz 2023-10-30T13:27:38.144 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56852.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70255.log 2023-10-30T13:27:38.144 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56852.log.gz 2023-10-30T13:27:38.145 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59896.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59896.log.gz 2023-10-30T13:27:38.145 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-30T13:27:38.145 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62679.log 2023-10-30T13:27:38.145 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70255.log.gz 2023-10-30T13:27:38.146 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58296.log 2023-10-30T13:27:38.146 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55126.log 2023-10-30T13:27:38.146 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62679.log.gz 2023-10-30T13:27:38.147 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68093.log 2023-10-30T13:27:38.147 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58296.log.gz 2023-10-30T13:27:38.147 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59296.log 2023-10-30T13:27:38.147 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55126.log.gz 2023-10-30T13:27:38.148 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43895.log 2023-10-30T13:27:38.148 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.68093.log.gz 2023-10-30T13:27:38.148 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59296.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59296.log.gz 2023-10-30T13:27:38.149 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65178.log 2023-10-30T13:27:38.149 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67217.log 2023-10-30T13:27:38.149 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43895.log.gz 2023-10-30T13:27:38.149 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52890.log 2023-10-30T13:27:38.150 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65178.log.gz 2023-10-30T13:27:38.150 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62627.log 2023-10-30T13:27:38.150 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67217.log.gz 2023-10-30T13:27:38.151 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52890.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52890.log.gz 2023-10-30T13:27:38.151 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62012.log 2023-10-30T13:27:38.151 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45909.log 2023-10-30T13:27:38.151 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62627.log.gz 2023-10-30T13:27:38.152 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66743.log 2023-10-30T13:27:38.152 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62012.log.gz 2023-10-30T13:27:38.152 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45909.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53128.log 2023-10-30T13:27:38.153 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45909.log.gz 2023-10-30T13:27:38.153 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66743.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.66743.log.gz 2023-10-30T13:27:38.153 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46647.log 2023-10-30T13:27:38.153 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48459.log 2023-10-30T13:27:38.154 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53128.log.gz 2023-10-30T13:27:38.154 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46647.log: gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.60540.log -- replaced with /var/log/ceph/ceph-client.admin.46647.log.gz 2023-10-30T13:27:38.154 INFO:teuthology.orchestra.run.smithi125.stderr: 2023-10-30T13:27:38.155 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46418.log 2023-10-30T13:27:38.155 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48459.log.gz 2023-10-30T13:27:38.155 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60540.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.60540.log.gz 2023-10-30T13:27:38.155 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53208.log 2023-10-30T13:27:38.156 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46366.log 2023-10-30T13:27:38.156 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46418.log.gz 2023-10-30T13:27:38.158 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.38691.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.38691.log.gz 2023-10-30T13:27:38.169 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70912.log 2023-10-30T13:27:38.169 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53208.log.gz 2023-10-30T13:27:38.169 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57269.log 2023-10-30T13:27:38.170 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46366.log.gz 2023-10-30T13:27:38.170 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70912.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70912.log.gz 2023-10-30T13:27:38.170 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52864.log 2023-10-30T13:27:38.171 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64171.log 2023-10-30T13:27:38.171 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57269.log.gz 2023-10-30T13:27:38.171 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52248.log 2023-10-30T13:27:38.171 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52864.log.gz 2023-10-30T13:27:38.172 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64356.log 2023-10-30T13:27:38.172 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64171.log.gz 2023-10-30T13:27:38.172 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56019.log 2023-10-30T13:27:38.173 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52248.log.gz 2023-10-30T13:27:38.173 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61435.log 2023-10-30T13:27:38.173 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.64356.log.gz 2023-10-30T13:27:38.173 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56019.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49745.log 2023-10-30T13:27:38.174 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56019.log.gz 2023-10-30T13:27:38.174 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61435.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61435.log.gz 2023-10-30T13:27:38.174 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61222.log 2023-10-30T13:27:38.175 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66605.log 2023-10-30T13:27:38.175 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49745.log.gz 2023-10-30T13:27:38.175 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55020.log 2023-10-30T13:27:38.176 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61222.log.gz 2023-10-30T13:27:38.176 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60383.log 2023-10-30T13:27:38.176 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66605.log.gz 2023-10-30T13:27:38.176 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55020.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55020.log.gz 2023-10-30T13:27:38.177 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48485.log 2023-10-30T13:27:38.177 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46182.log 2023-10-30T13:27:38.177 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60383.log.gz 2023-10-30T13:27:38.178 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52654.log 2023-10-30T13:27:38.178 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48485.log.gz 2023-10-30T13:27:38.178 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46182.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46182.log.gz 2023-10-30T13:27:38.178 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55258.log 2023-10-30T13:27:38.179 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46621.log 2023-10-30T13:27:38.179 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52654.log.gz 2023-10-30T13:27:38.179 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61956.log 2023-10-30T13:27:38.180 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55258.log.gz 2023-10-30T13:27:38.180 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65439.log 2023-10-30T13:27:38.180 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46621.log.gz 2023-10-30T13:27:38.181 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61956.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61956.log.gz 2023-10-30T13:27:38.181 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55519.log 2023-10-30T13:27:38.181 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54624.log 2023-10-30T13:27:38.181 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65439.log.gz 2023-10-30T13:27:38.182 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55232.log 2023-10-30T13:27:38.182 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55519.log.gz 2023-10-30T13:27:38.182 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56721.log 2023-10-30T13:27:38.183 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54624.log.gz 2023-10-30T13:27:38.183 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55232.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55232.log.gz 2023-10-30T13:27:38.183 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59506.log 2023-10-30T13:27:38.184 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68984.log 2023-10-30T13:27:38.184 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56721.log.gz 2023-10-30T13:27:38.184 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59506.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59506.log.gz 2023-10-30T13:27:38.185 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49551.log 2023-10-30T13:27:38.185 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54485.log 2023-10-30T13:27:38.185 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68984.log.gz 2023-10-30T13:27:38.185 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49551.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49551.log.gz 2023-10-30T13:27:38.186 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46496.log 2023-10-30T13:27:38.186 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52576.log 2023-10-30T13:27:38.186 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54485.log.gz 2023-10-30T13:27:38.187 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52434.log 88.7% 2023-10-30T13:27:38.187 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46496.log.gz 2023-10-30T13:27:38.187 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49667.log 2023-10-30T13:27:38.187 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52576.log.gz 2023-10-30T13:27:38.188 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62523.log 2023-10-30T13:27:38.188 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52434.log.gz 2023-10-30T13:27:38.199 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50208.log 2023-10-30T13:27:38.199 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49667.log.gz 2023-10-30T13:27:38.199 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66657.log 2023-10-30T13:27:38.200 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62523.log.gz 2023-10-30T13:27:38.200 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50208.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50208.log.gz 2023-10-30T13:27:38.200 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66278.log 2023-10-30T13:27:38.200 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66657.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.66657.log.gz 2023-10-30T13:27:38.201 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.58246.log 2023-10-30T13:27:38.201 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56878.log 2023-10-30T13:27:38.201 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66278.log.gz 2023-10-30T13:27:38.202 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65283.log 2023-10-30T13:27:38.202 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58246.log.gz 2023-10-30T13:27:38.202 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56878.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56878.log.gz 2023-10-30T13:27:38.203 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46104.log 2023-10-30T13:27:38.203 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65283.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65283.log.gz 2023-10-30T13:27:38.203 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50261.log 2023-10-30T13:27:38.203 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46104.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46104.log.gz 2023-10-30T13:27:38.204 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44373.log 2023-10-30T13:27:38.204 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50261.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50261.log.gz 2023-10-30T13:27:38.204 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62734.log 2023-10-30T13:27:38.205 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44373.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44373.log.gz 2023-10-30T13:27:38.205 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52274.log 2023-10-30T13:27:38.205 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55753.log 2023-10-30T13:27:38.206 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62734.log.gz 2023-10-30T13:27:38.206 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57637.log 2023-10-30T13:27:38.206 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52274.log.gz 2023-10-30T13:27:38.206 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53887.log 2023-10-30T13:27:38.207 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55753.log.gz 2023-10-30T13:27:38.207 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57637.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57637.log.gz 2023-10-30T13:27:38.207 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66688.log 2023-10-30T13:27:38.208 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44530.log 2023-10-30T13:27:38.208 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53887.log.gz 2023-10-30T13:27:38.208 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62494.log 2023-10-30T13:27:38.209 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66688.log.gz 2023-10-30T13:27:38.209 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57402.log 2023-10-30T13:27:38.209 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44530.log.gz 2023-10-30T13:27:38.209 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55649.log 2023-10-30T13:27:38.210 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62494.log.gz 2023-10-30T13:27:38.210 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57402.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57402.log.gz 2023-10-30T13:27:38.210 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52030.log 2023-10-30T13:27:38.211 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52995.log 2023-10-30T13:27:38.211 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55649.log.gz 2023-10-30T13:27:38.211 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54539.log 2023-10-30T13:27:38.211 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52030.log.gz 2023-10-30T13:27:38.212 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58750.log 2023-10-30T13:27:38.212 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52995.log.gz 2023-10-30T13:27:38.212 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54539.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.54539.log.gz 2023-10-30T13:27:38.213 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66444.log 2023-10-30T13:27:38.213 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58750.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48381.log 2023-10-30T13:27:38.213 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58750.log.gz 2023-10-30T13:27:38.214 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58189.log 2023-10-30T13:27:38.214 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66444.log.gz 2023-10-30T13:27:38.214 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48381.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48381.log.gz 2023-10-30T13:27:38.215 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49798.log 2023-10-30T13:27:38.215 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58189.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58189.log.gz 2023-10-30T13:27:38.215 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53861.log 2023-10-30T13:27:38.216 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45546.log 2023-10-30T13:27:38.216 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49798.log.gz 2023-10-30T13:27:38.226 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56826.log 2023-10-30T13:27:38.226 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53861.log.gz 2023-10-30T13:27:38.226 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61540.log 2023-10-30T13:27:38.227 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45546.log.gz 2023-10-30T13:27:38.227 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56826.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56826.log.gz 2023-10-30T13:27:38.227 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56151.log 2023-10-30T13:27:38.227 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64994.log 2023-10-30T13:27:38.228 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61540.log.gz 2023-10-30T13:27:38.228 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58397.log 2023-10-30T13:27:38.228 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56151.log.gz 2023-10-30T13:27:38.229 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53312.log 2023-10-30T13:27:38.229 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64994.log.gz 2023-10-30T13:27:38.229 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68800.log 2023-10-30T13:27:38.230 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58397.log.gz 2023-10-30T13:27:38.230 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53312.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53312.log.gz 2023-10-30T13:27:38.230 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59270.log 2023-10-30T13:27:38.231 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48301.log 2023-10-30T13:27:38.231 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68800.log.gz 2023-10-30T13:27:38.231 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56177.log 2023-10-30T13:27:38.232 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59270.log.gz 2023-10-30T13:27:38.232 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50022.log 2023-10-30T13:27:38.232 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48301.log.gz 2023-10-30T13:27:38.233 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70701.log 2023-10-30T13:27:38.233 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56177.log.gz 2023-10-30T13:27:38.233 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50692.log 2023-10-30T13:27:38.233 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50022.log.gz 2023-10-30T13:27:38.234 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47016.log 2023-10-30T13:27:38.234 INFO:teuthology.orchestra.run.smithi125.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70701.log.gz 2023-10-30T13:27:38.234 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50692.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64408.log 2023-10-30T13:27:38.235 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50692.log.gz 2023-10-30T13:27:38.235 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47016.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47016.log.gz 2023-10-30T13:27:38.235 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56640.log 2023-10-30T13:27:38.236 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64408.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-30T13:27:38.236 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64408.log.gz 2023-10-30T13:27:38.236 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55178.log 2023-10-30T13:27:38.237 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56640.log.gz 2023-10-30T13:27:38.237 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45211.log 2023-10-30T13:27:38.237 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50130.log 2023-10-30T13:27:38.238 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55178.log.gz 2023-10-30T13:27:38.238 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52089.log 2023-10-30T13:27:38.238 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45211.log.gz 2023-10-30T13:27:38.239 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45220.log 2023-10-30T13:27:38.239 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50130.log.gz 2023-10-30T13:27:38.239 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52089.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52089.log.gz 2023-10-30T13:27:38.240 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59955.log 2023-10-30T13:27:38.240 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45220.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45220.log.gz 2023-10-30T13:27:38.240 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67896.log 2023-10-30T13:27:38.240 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59955.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59955.log.gz 2023-10-30T13:27:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59244.log 2023-10-30T13:27:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67896.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67896.log.gz 2023-10-30T13:27:38.241 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44347.log 2023-10-30T13:27:38.242 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59244.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59244.log.gz 2023-10-30T13:27:38.242 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51225.log 2023-10-30T13:27:38.250 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44347.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44347.log.gz 2023-10-30T13:27:38.251 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45088.log 2023-10-30T13:27:38.251 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51225.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.51225.log.gz 2023-10-30T13:27:38.251 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57051.log 2023-10-30T13:27:38.252 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45088.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45088.log.gz 2023-10-30T13:27:38.252 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67189.log 2023-10-30T13:27:38.252 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57051.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57051.log.gz 2023-10-30T13:27:38.253 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57864.log 2023-10-30T13:27:38.253 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67189.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.67189.log.gz 2023-10-30T13:27:38.253 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63585.log 2023-10-30T13:27:38.254 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57864.log: gzip -5 --verbose -- 89.0% /var/log/ceph/ceph-client.admin.59581.log -- replaced with /var/log/ceph/ceph-client.admin.57864.log.gz 2023-10-30T13:27:38.254 INFO:teuthology.orchestra.run.smithi125.stderr: 2023-10-30T13:27:38.254 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63585.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63585.log.gz 2023-10-30T13:27:38.255 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64382.log 2023-10-30T13:27:38.255 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59581.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59581.log.gz 2023-10-30T13:27:38.255 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63715.log 2023-10-30T13:27:38.255 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64382.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64382.log.gz 2023-10-30T13:27:38.256 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61904.log 2023-10-30T13:27:38.256 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63715.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63715.log.gz 2023-10-30T13:27:38.256 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65126.log 2023-10-30T13:27:38.257 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61904.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.61904.log.gz 2023-10-30T13:27:38.257 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68250.log 2023-10-30T13:27:38.257 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65126.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65126.log.gz 2023-10-30T13:27:38.257 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49473.log 2023-10-30T13:27:38.258 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68250.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68250.log.gz 2023-10-30T13:27:38.258 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61136.log 2023-10-30T13:27:38.258 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62900.log 2023-10-30T13:27:38.259 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49473.log.gz 2023-10-30T13:27:38.259 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61136.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61136.log.gz 2023-10-30T13:27:38.259 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57131.log 2023-10-30T13:27:38.260 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62900.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62900.log.gz 2023-10-30T13:27:38.260 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64197.log 2023-10-30T13:27:38.260 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57131.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57131.log.gz 2023-10-30T13:27:38.260 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65689.log 2023-10-30T13:27:38.261 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64197.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64197.log.gz 2023-10-30T13:27:38.261 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55336.log 2023-10-30T13:27:38.261 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65689.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.65689.log.gz 2023-10-30T13:27:38.262 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51751.log 2023-10-30T13:27:38.262 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55336.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55336.log.gz 2023-10-30T13:27:38.262 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57734.log 2023-10-30T13:27:38.262 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51751.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51751.log.gz 2023-10-30T13:27:38.263 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52550.log 2023-10-30T13:27:38.263 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57734.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57734.log.gz 2023-10-30T13:27:38.263 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70004.log 2023-10-30T13:27:38.264 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52550.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52550.log.gz 2023-10-30T13:27:38.264 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45390.log 2023-10-30T13:27:38.272 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70004.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70004.log.gz 2023-10-30T13:27:38.272 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53741.log 2023-10-30T13:27:38.272 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45390.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45390.log.gz 2023-10-30T13:27:38.272 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65048.log 2023-10-30T13:27:38.273 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53741.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53741.log.gz 2023-10-30T13:27:38.273 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61032.log 2023-10-30T13:27:38.273 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65048.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65048.log.gz 2023-10-30T13:27:38.274 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59128.log 2023-10-30T13:27:38.274 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61032.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61032.log.gz 2023-10-30T13:27:38.274 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52460.log 2023-10-30T13:27:38.275 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59128.log: gzip -5 89.5% --verbose -- replaced with /var/log/ceph/ceph-client.admin.59128.log.gz -- 2023-10-30T13:27:38.275 INFO:teuthology.orchestra.run.smithi125.stderr: /var/log/ceph/ceph-client.admin.56447.log 2023-10-30T13:27:38.275 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52460.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52460.log.gz 2023-10-30T13:27:38.275 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51619.log 2023-10-30T13:27:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56447.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56447.log.gz 2023-10-30T13:27:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52943.log 2023-10-30T13:27:38.276 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53234.log 2023-10-30T13:27:38.277 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51619.log.gz 2023-10-30T13:27:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52943.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52943.log.gz 2023-10-30T13:27:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65847.log 2023-10-30T13:27:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53234.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53234.log.gz 2023-10-30T13:27:38.278 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48683.log 2023-10-30T13:27:38.278 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65847.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65847.log.gz 2023-10-30T13:27:38.278 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46052.log 2023-10-30T13:27:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54758.log 2023-10-30T13:27:38.279 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48683.log.gz 2023-10-30T13:27:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46052.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46052.log.gz 2023-10-30T13:27:38.279 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67477.log 2023-10-30T13:27:38.280 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54758.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54758.log.gz 2023-10-30T13:27:38.280 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45092.log 2023-10-30T13:27:38.280 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67477.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67477.log.gz 2023-10-30T13:27:38.281 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55152.log 2023-10-30T13:27:38.281 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45092.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45092.log.gz 2023-10-30T13:27:38.281 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64091.log 2023-10-30T13:27:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55152.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55152.log.gz 2023-10-30T13:27:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46776.log 2023-10-30T13:27:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64091.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64091.log.gz 2023-10-30T13:27:38.282 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61681.log 2023-10-30T13:27:38.283 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46776.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46776.log.gz 2023-10-30T13:27:38.283 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65637.log 2023-10-30T13:27:38.288 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61681.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61681.log.gz 2023-10-30T13:27:38.289 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-30T13:27:38.289 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65637.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.65637.log.gz 2023-10-30T13:27:38.289 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54862.log 2023-10-30T13:27:38.289 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57298.log 2023-10-30T13:27:38.290 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54862.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54862.log.gz 2023-10-30T13:27:38.290 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68906.log 2023-10-30T13:27:38.290 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57298.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57298.log.gz 2023-10-30T13:27:38.291 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65585.log 2023-10-30T13:27:38.291 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45650.log 2023-10-30T13:27:38.291 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68906.log.gz 2023-10-30T13:27:38.292 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65585.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.65585.log.gz 2023-10-30T13:27:38.292 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51277.log 2023-10-30T13:27:38.292 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45650.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45650.log.gz 2023-10-30T13:27:38.292 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51976.log 2023-10-30T13:27:38.293 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69978.log 2023-10-30T13:27:38.293 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51277.log.gz 2023-10-30T13:27:38.293 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51976.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51976.log.gz 2023-10-30T13:27:38.294 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70834.log 2023-10-30T13:27:38.294 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69978.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.69978.log.gz 2023-10-30T13:27:38.294 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44671.log 2023-10-30T13:27:38.294 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62381.log 2023-10-30T13:27:38.295 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70834.log.gz 2023-10-30T13:27:38.295 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44671.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44671.log.gz 2023-10-30T13:27:38.295 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67790.log 2023-10-30T13:27:38.299 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62381.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62381.log.gz 2023-10-30T13:27:38.299 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68470.log 2023-10-30T13:27:38.299 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54254.log 2023-10-30T13:27:38.300 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67790.log.gz 2023-10-30T13:27:38.300 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68470.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.68470.log.gz 2023-10-30T13:27:38.300 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66953.log 2023-10-30T13:27:38.301 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54254.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54254.log.gz 2023-10-30T13:27:38.301 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62468.log 2023-10-30T13:27:38.301 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48327.log 2023-10-30T13:27:38.302 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66953.log.gz 2023-10-30T13:27:38.302 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62468.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.62468.log.gz 2023-10-30T13:27:38.302 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69194.log 2023-10-30T13:27:38.302 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48327.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48327.log.gz 2023-10-30T13:27:38.303 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52115.log 2023-10-30T13:27:38.303 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46208.log 2023-10-30T13:27:38.303 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69194.log.gz 2023-10-30T13:27:38.306 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52115.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52115.log.gz 2023-10-30T13:27:38.306 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46595.log 2023-10-30T13:27:38.306 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46208.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46208.log.gz 2023-10-30T13:27:38.307 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46828.log 2023-10-30T13:27:38.307 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46595.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46595.log.gz 2023-10-30T13:27:38.307 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62329.log 2023-10-30T13:27:38.308 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46828.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46828.log.gz 2023-10-30T13:27:38.308 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49366.log 2023-10-30T13:27:38.308 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68932.log 2023-10-30T13:27:38.309 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62329.log.gz 2023-10-30T13:27:38.310 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49366.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49366.log.gz 2023-10-30T13:27:38.311 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59189.log 2023-10-30T13:27:38.311 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68932.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68932.log.gz 2023-10-30T13:27:38.311 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45212.log 2023-10-30T13:27:38.311 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59189.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59189.log.gz 2023-10-30T13:27:38.312 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59050.log 2023-10-30T13:27:38.313 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63009.log 2023-10-30T13:27:38.314 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45212.log.gz 2023-10-30T13:27:38.314 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59050.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59050.log.gz 2023-10-30T13:27:38.314 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67503.log 2023-10-30T13:27:38.315 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63009.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.63009.log.gz 2023-10-30T13:27:38.316 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46032.log 2023-10-30T13:27:38.316 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67503.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67503.log.gz 2023-10-30T13:27:38.316 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57430.log 2023-10-30T13:27:38.317 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.0.46032.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.46032.log.gz 2023-10-30T13:27:38.317 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64304.log 2023-10-30T13:27:38.318 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57430.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57430.log.gz 2023-10-30T13:27:38.319 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62814.log 2023-10-30T13:27:38.319 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44582.log 2023-10-30T13:27:38.320 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.64304.log.gz 2023-10-30T13:27:38.321 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62814.log: gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.62814.log.gz --verbose 2023-10-30T13:27:38.321 INFO:teuthology.orchestra.run.smithi125.stderr: -- /var/log/ceph/ceph-client.admin.55048.log 2023-10-30T13:27:38.322 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44582.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44582.log.gz 2023-10-30T13:27:38.323 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60164.log 2023-10-30T13:27:38.324 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54836.log 2023-10-30T13:27:38.324 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55048.log.gz 2023-10-30T13:27:38.325 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60164.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60164.log.gz 2023-10-30T13:27:38.326 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44321.log 2023-10-30T13:27:38.327 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44608.log 2023-10-30T13:27:38.327 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54836.log.gz 2023-10-30T13:27:38.328 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44321.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44321.log.gz 2023-10-30T13:27:38.328 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68880.log 2023-10-30T13:27:38.329 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44608.log: gzip 88.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.44608.log.gz -- 2023-10-30T13:27:38.330 INFO:teuthology.orchestra.run.smithi125.stderr: /var/log/ceph/ceph-client.admin.54148.log 2023-10-30T13:27:38.330 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68880.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.68880.log.gz 2023-10-30T13:27:38.331 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62143.log 2023-10-30T13:27:38.332 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49392.log 2023-10-30T13:27:38.333 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62143.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.54148.log.gz 2023-10-30T13:27:38.333 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.62143.log.gz 2023-10-30T13:27:38.334 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56800.log 2023-10-30T13:27:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49392.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49392.log.gz 2023-10-30T13:27:38.335 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58163.log 2023-10-30T13:27:38.336 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56800.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56800.log.gz 2023-10-30T13:27:38.337 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62064.log 2023-10-30T13:27:38.338 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58163.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58163.log.gz 2023-10-30T13:27:38.338 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50392.log 2023-10-30T13:27:38.339 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62064.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62064.log.gz 2023-10-30T13:27:38.340 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51898.log 2023-10-30T13:27:38.341 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50392.log: gzip 88.3% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.50392.log.gz -- 2023-10-30T13:27:38.341 INFO:teuthology.orchestra.run.smithi125.stderr: /var/log/ceph/ceph-client.admin.55547.log 2023-10-30T13:27:38.342 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51898.log: gzip -5 --verbose -- 88.9% /var/log/ceph/ceph-client.admin.66473.log 2023-10-30T13:27:38.343 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51898.log.gz 2023-10-30T13:27:38.344 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55547.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.55547.log.gz 2023-10-30T13:27:38.345 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45728.log 2023-10-30T13:27:38.345 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66473.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66473.log.gz 2023-10-30T13:27:38.346 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67926.log 2023-10-30T13:27:38.347 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45728.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45728.log.gz 2023-10-30T13:27:38.347 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51725.log 2023-10-30T13:27:38.348 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67926.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67926.log.gz 2023-10-30T13:27:38.349 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66527.log 2023-10-30T13:27:38.350 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65663.log 2023-10-30T13:27:38.351 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51725.log.gz 2023-10-30T13:27:38.351 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66527.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.66527.log.gz 2023-10-30T13:27:38.352 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64486.log 2023-10-30T13:27:38.352 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65663.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65663.log.gz 2023-10-30T13:27:38.353 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70125.log 2023-10-30T13:27:38.354 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64486.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64486.log.gz 2023-10-30T13:27:38.355 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62355.log 2023-10-30T13:27:38.355 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61602.log 2023-10-30T13:27:38.356 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70125.log.gz 2023-10-30T13:27:38.357 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62355.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.62355.log.gz 2023-10-30T13:27:38.357 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67659.log 2023-10-30T13:27:38.358 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61602.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61602.log.gz 2023-10-30T13:27:38.359 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59322.log 2023-10-30T13:27:38.360 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46392.log 2023-10-30T13:27:38.360 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67659.log.gz 2023-10-30T13:27:38.361 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59322.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59322.log.gz 2023-10-30T13:27:38.362 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70083.log 2023-10-30T13:27:38.362 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61329.log 2023-10-30T13:27:38.363 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46392.log.gz 2023-10-30T13:27:38.364 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70083.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70083.log.gz 2023-10-30T13:27:38.364 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59981.log 2023-10-30T13:27:38.365 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61329.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61329.log.gz 2023-10-30T13:27:38.366 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51671.log 2023-10-30T13:27:38.367 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59981.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59981.log.gz 2023-10-30T13:27:38.367 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63349.log 2023-10-30T13:27:38.368 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51671.log: gzip -5 --verbose 89.3% -- /var/log/ceph/ceph-client.admin.66769.log -- replaced with /var/log/ceph/ceph-client.admin.51671.log.gz 2023-10-30T13:27:38.369 INFO:teuthology.orchestra.run.smithi125.stderr: 2023-10-30T13:27:38.370 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61629.log 2023-10-30T13:27:38.370 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63349.log.gz 2023-10-30T13:27:38.371 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59733.log 2023-10-30T13:27:38.372 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66769.log.gz 2023-10-30T13:27:38.373 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61629.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.61629.log.gz 2023-10-30T13:27:38.373 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64826.log 2023-10-30T13:27:38.374 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59733.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59733.log.gz 2023-10-30T13:27:38.375 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64249.log 2023-10-30T13:27:38.375 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57949.log 2023-10-30T13:27:38.376 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64826.log.gz 2023-10-30T13:27:38.377 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48170.log 2023-10-30T13:27:38.377 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64249.log.gz 2023-10-30T13:27:38.378 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57949.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57949.log.gz 2023-10-30T13:27:38.378 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51331.log 2023-10-30T13:27:38.379 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44288.log 2023-10-30T13:27:38.380 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48170.log.gz 2023-10-30T13:27:38.381 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51331.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51331.log.gz 2023-10-30T13:27:38.381 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63847.log 2023-10-30T13:27:38.382 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44288.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44288.log.gz 2023-10-30T13:27:38.383 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64065.log 2023-10-30T13:27:38.384 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63847.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63847.log.gz 2023-10-30T13:27:38.384 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64748.log 2023-10-30T13:27:38.385 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64065.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64065.log.gz 2023-10-30T13:27:38.386 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69691.log 2023-10-30T13:27:38.387 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64748.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64748.log.gz 2023-10-30T13:27:38.387 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54565.log 2023-10-30T13:27:38.388 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69691.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69691.log.gz 2023-10-30T13:27:38.389 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68283.log 2023-10-30T13:27:38.390 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54565.log: gzip 89.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.54565.log.gz -- 2023-10-30T13:27:38.390 INFO:teuthology.orchestra.run.smithi125.stderr: /var/log/ceph/ceph-client.admin.53530.log 2023-10-30T13:27:38.391 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46444.log 2023-10-30T13:27:38.392 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.68283.log.gz 2023-10-30T13:27:38.393 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64434.log 2023-10-30T13:27:38.393 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53530.log.gz 2023-10-30T13:27:38.394 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46444.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.46444.log.gz 2023-10-30T13:27:38.394 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63906.log 2023-10-30T13:27:38.395 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69088.log 2023-10-30T13:27:38.396 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.64434.log.gz 2023-10-30T13:27:38.397 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65532.log 2023-10-30T13:27:38.397 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63906.log.gz 2023-10-30T13:27:38.398 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69088.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69088.log.gz 2023-10-30T13:27:38.398 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68146.log 2023-10-30T13:27:38.399 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57591.log 2023-10-30T13:27:38.400 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65532.log.gz 2023-10-30T13:27:38.401 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68146.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68146.log.gz 2023-10-30T13:27:38.401 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48118.log 2023-10-30T13:27:38.402 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57591.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57591.log.gz 2023-10-30T13:27:38.403 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61409.log 2023-10-30T13:27:38.404 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48118.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48118.log.gz 2023-10-30T13:27:38.405 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52408.log 2023-10-30T13:27:38.406 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61409.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61409.log.gz 2023-10-30T13:27:38.406 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62251.log 2023-10-30T13:27:38.408 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52408.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52408.log.gz 2023-10-30T13:27:38.408 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69140.log 2023-10-30T13:27:38.409 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62251.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62251.log.gz 2023-10-30T13:27:38.409 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52786.log 2023-10-30T13:27:38.410 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54306.log 2023-10-30T13:27:38.411 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.69140.log.gz 2023-10-30T13:27:38.412 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63113.log 2023-10-30T13:27:38.412 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52786.log.gz 2023-10-30T13:27:38.414 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66797.log 2023-10-30T13:27:38.414 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54306.log.gz 2023-10-30T13:27:38.415 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51515.log 2023-10-30T13:27:38.415 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63113.log.gz 2023-10-30T13:27:38.417 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62782.log 2023-10-30T13:27:38.417 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66797.log.gz 2023-10-30T13:27:38.418 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51515.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46314.log 2023-10-30T13:27:38.419 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51515.log.gz 2023-10-30T13:27:38.419 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62782.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62782.log.gz 2023-10-30T13:27:38.420 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55574.log 2023-10-30T13:27:38.421 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46314.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46314.log.gz 2023-10-30T13:27:38.421 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44556.log 2023-10-30T13:27:38.422 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55574.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55574.log.gz 2023-10-30T13:27:38.423 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61766.log 2023-10-30T13:27:38.424 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44556.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44556.log.gzgzip 2023-10-30T13:27:38.424 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53939.log 2023-10-30T13:27:38.425 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61766.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61766.log.gz 2023-10-30T13:27:38.426 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56313.log 2023-10-30T13:27:38.427 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53939.log: 87.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.53939.log.gz 2023-10-30T13:27:38.427 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52628.log 2023-10-30T13:27:38.428 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70177.log 2023-10-30T13:27:38.429 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56313.log.gz 2023-10-30T13:27:38.430 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52628.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52628.log.gz 2023-10-30T13:27:38.430 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45086.log 2023-10-30T13:27:38.431 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52524.log 2023-10-30T13:27:38.431 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70177.log.gz 2023-10-30T13:27:38.432 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45086.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45086.log.gz 2023-10-30T13:27:38.432 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70619.log 2023-10-30T13:27:38.434 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52524.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52524.log.gz 2023-10-30T13:27:38.434 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49719.log 2023-10-30T13:27:38.435 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70619.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70619.log.gz 2023-10-30T13:27:38.436 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70203.log 2023-10-30T13:27:38.437 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49719.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.49719.log.gz 2023-10-30T13:27:38.437 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50472.log 2023-10-30T13:27:38.438 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67270.log 2023-10-30T13:27:38.438 INFO:teuthology.orchestra.run.smithi125.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.70203.log.gz 2023-10-30T13:27:38.439 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50472.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50472.log.gz 2023-10-30T13:27:38.440 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58589.log 2023-10-30T13:27:38.441 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53662.log 2023-10-30T13:27:38.441 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67270.log.gz 2023-10-30T13:27:38.442 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58589.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58589.log.gz 2023-10-30T13:27:38.443 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44399.log 2023-10-30T13:27:38.443 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47042.log 2023-10-30T13:27:38.444 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53662.log.gz 2023-10-30T13:27:38.445 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44399.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44399.log.gz 2023-10-30T13:27:38.445 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65020.log 2023-10-30T13:27:38.446 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47042.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47042.log.gz 2023-10-30T13:27:38.447 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50156.log 2023-10-30T13:27:38.448 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66927.log 2023-10-30T13:27:38.448 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.65020.log.gz 2023-10-30T13:27:38.449 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53478.log 2023-10-30T13:27:38.450 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50156.log.gz 2023-10-30T13:27:38.451 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51487.log 2023-10-30T13:27:38.451 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66927.log.gz 2023-10-30T13:27:38.452 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53478.log: gzip -5 --verbose -- 88.9% /var/log/ceph/ceph-client.admin.58670.log -- replaced with /var/log/ceph/ceph-client.admin.53478.log.gz 2023-10-30T13:27:38.453 INFO:teuthology.orchestra.run.smithi125.stderr: 2023-10-30T13:27:38.454 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60007.log 2023-10-30T13:27:38.454 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51487.log.gz 2023-10-30T13:27:38.455 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58670.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58670.log.gz 2023-10-30T13:27:38.456 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58501.log 2023-10-30T13:27:38.456 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60007.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60007.log.gz 2023-10-30T13:27:38.457 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69849.log 2023-10-30T13:27:38.458 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64566.log 2023-10-30T13:27:38.458 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58501.log.gz 2023-10-30T13:27:38.459 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69849.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.69849.log.gz 2023-10-30T13:27:38.460 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52144.log 2023-10-30T13:27:38.461 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64566.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64566.log.gz 2023-10-30T13:27:38.461 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-30T13:27:38.462 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52144.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.52144.log.gz 2023-10-30T13:27:38.463 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57760.log 2023-10-30T13:27:38.464 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51461.log 2023-10-30T13:27:38.466 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48144.log 2023-10-30T13:27:38.466 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57760.log.gz 2023-10-30T13:27:38.467 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55779.log 2023-10-30T13:27:38.468 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51461.log.gz 2023-10-30T13:27:38.469 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54017.log 2023-10-30T13:27:38.469 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48144.log.gz 2023-10-30T13:27:38.470 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64278.log 2023-10-30T13:27:38.470 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55779.log.gz 2023-10-30T13:27:38.472 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44685.log 2023-10-30T13:27:38.472 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54017.log.gz 2023-10-30T13:27:38.473 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64278.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.64278.log.gz 2023-10-30T13:27:38.474 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54043.log 2023-10-30T13:27:38.475 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44685.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44685.log.gz -5 2023-10-30T13:27:38.476 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.64670.log 2023-10-30T13:27:38.477 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54043.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54043.log.gz 2023-10-30T13:27:38.477 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60625.log 2023-10-30T13:27:38.478 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52196.log 2023-10-30T13:27:38.479 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64670.log.gz 2023-10-30T13:27:38.480 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62653.log 2023-10-30T13:27:38.480 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60625.log.gz 2023-10-30T13:27:38.481 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69481.log 2023-10-30T13:27:38.482 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52196.log.gz 2023-10-30T13:27:38.482 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62653.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66499.log 2023-10-30T13:27:38.483 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62653.log.gz 2023-10-30T13:27:38.484 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63401.log 2023-10-30T13:27:38.485 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.69481.log.gz 2023-10-30T13:27:38.485 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58475.log 87.6% 2023-10-30T13:27:38.486 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.66499.log.gz 2023-10-30T13:27:38.487 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58802.log 2023-10-30T13:27:38.487 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63401.log.gz 2023-10-30T13:27:38.489 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68392.log 2023-10-30T13:27:38.489 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58475.log.gz 2023-10-30T13:27:38.490 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70758.log 2023-10-30T13:27:38.490 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58802.log.gz 2023-10-30T13:27:38.491 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51541.log 2023-10-30T13:27:38.491 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.68392.log.gz 2023-10-30T13:27:38.493 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57923.log 2023-10-30T13:27:38.494 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70758.log.gz 2023-10-30T13:27:38.494 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44896.log 2023-10-30T13:27:38.495 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51541.log.gz 2023-10-30T13:27:38.496 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57923.log: gzip -5 --verbose -- 88.7% /var/log/ceph/ceph-client.admin.53831.log 2023-10-30T13:27:38.496 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57923.log.gz 2023-10-30T13:27:38.497 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69717.log 2023-10-30T13:27:38.498 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44896.log.gz 2023-10-30T13:27:38.498 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68172.log 2023-10-30T13:27:38.499 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53831.log.gz 2023-10-30T13:27:38.500 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69717.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69717.log.gz 2023-10-30T13:27:38.500 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66226.log 2023-10-30T13:27:38.501 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59837.log 2023-10-30T13:27:38.502 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.68172.log.gz 2023-10-30T13:27:38.504 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45702.log 2023-10-30T13:27:38.504 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66226.log.gz 2023-10-30T13:27:38.505 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59837.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59837.log.gz 2023-10-30T13:27:38.505 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50446.log 2023-10-30T13:27:38.506 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45702.log: gzip -5 --verbose -- 89.5% /var/log/ceph/ceph-client.admin.62549.log -- replaced with /var/log/ceph/ceph-client.admin.45702.log.gz 2023-10-30T13:27:38.506 INFO:teuthology.orchestra.run.smithi125.stderr: 2023-10-30T13:27:38.507 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69744.log 2023-10-30T13:27:38.508 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50446.log.gz 2023-10-30T13:27:38.509 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48822.log 2023-10-30T13:27:38.510 INFO:teuthology.orchestra.run.smithi125.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.62549.log.gz 2023-10-30T13:27:38.511 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45494.log 2023-10-30T13:27:38.511 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69744.log.gz 2023-10-30T13:27:38.512 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48822.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48822.log.gz 2023-10-30T13:27:38.512 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70359.log 2023-10-30T13:27:38.514 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65335.log 2023-10-30T13:27:38.514 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45494.log.gz 2023-10-30T13:27:38.515 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61110.log 2023-10-30T13:27:38.515 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70359.log.gz 2023-10-30T13:27:38.516 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58722.log 2023-10-30T13:27:38.517 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.65335.log.gz 2023-10-30T13:27:38.518 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59759.log 2023-10-30T13:27:38.518 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.61110.log.gz 2023-10-30T13:27:38.519 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58722.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.58722.log.gz 2023-10-30T13:27:38.519 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.64968.log 2023-10-30T13:27:38.521 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49447.log 2023-10-30T13:27:38.521 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59759.log.gz 2023-10-30T13:27:38.522 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63663.log 2023-10-30T13:27:38.523 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64968.log.gz 2023-10-30T13:27:38.524 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53913.log 2023-10-30T13:27:38.524 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49447.log.gz 2023-10-30T13:27:38.525 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68119.log 2023-10-30T13:27:38.525 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63663.log.gz 2023-10-30T13:27:38.526 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53913.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53913.log.gz 2023-10-30T13:27:38.527 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58369.log 2023-10-30T13:27:38.528 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63061.log 2023-10-30T13:27:38.529 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.68119.log.gz 2023-10-30T13:27:38.529 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58369.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65361.log 2023-10-30T13:27:38.530 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58369.log.gz 2023-10-30T13:27:38.531 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66979.log 2023-10-30T13:27:38.531 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63061.log.gz 2023-10-30T13:27:38.532 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56525.log 2023-10-30T13:27:38.533 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65361.log.gz 2023-10-30T13:27:38.534 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66979.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50498.log 2023-10-30T13:27:38.534 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66979.log.gz 2023-10-30T13:27:38.535 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46260.log 2023-10-30T13:27:38.536 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56525.log.gz 2023-10-30T13:27:38.537 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59374.log 2023-10-30T13:27:38.537 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50498.log.gz 2023-10-30T13:27:38.538 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61492.log 2023-10-30T13:27:38.538 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46260.log.gz 2023-10-30T13:27:38.540 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46567.log 2023-10-30T13:27:38.540 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59374.log.gz 2023-10-30T13:27:38.541 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69587.log 2023-10-30T13:27:38.542 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61492.log.gz 2023-10-30T13:27:38.542 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63507.log 2023-10-30T13:27:38.543 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46567.log.gz 2023-10-30T13:27:38.544 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69587.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.69587.log.gz -5 2023-10-30T13:27:38.544 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51777.log 2023-10-30T13:27:38.546 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38187.log 2023-10-30T13:27:38.546 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63507.log.gz 2023-10-30T13:27:38.547 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62575.log 2023-10-30T13:27:38.547 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51777.log.gz 2023-10-30T13:27:38.548 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.38187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51872.log 2023-10-30T13:27:38.549 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.38187.log.gz 2023-10-30T13:27:38.550 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45090.log 2023-10-30T13:27:38.550 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.62575.log.gz 2023-10-30T13:27:38.551 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56473.log 2023-10-30T13:27:38.551 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51872.log.gz 2023-10-30T13:27:38.553 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57376.log 2023-10-30T13:27:38.554 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45090.log.gz 2023-10-30T13:27:38.554 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51251.log 2023-10-30T13:27:38.555 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56473.log.gz 2023-10-30T13:27:38.555 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57376.log: gzip 88.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.57376.log.gz 2023-10-30T13:27:38.556 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.45804.log 2023-10-30T13:27:38.557 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51251.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.51251.log.gz 2023-10-30T13:27:38.558 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45754.log 2023-10-30T13:27:38.559 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66359.log 2023-10-30T13:27:38.559 INFO:teuthology.orchestra.run.smithi125.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.45804.log.gz 2023-10-30T13:27:38.560 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45754.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-30T13:27:38.560 INFO:teuthology.orchestra.run.smithi125.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.45754.log.gz 2023-10-30T13:27:38.561 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66359.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66359.log.gz 2023-10-30T13:27:38.562 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53584.log 2023-10-30T13:27:38.563 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55967.log 2023-10-30T13:27:38.564 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65231.log 2023-10-30T13:27:38.565 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53584.log.gz 2023-10-30T13:27:38.566 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55967.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55967.log.gz 2023-10-30T13:27:38.566 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58449.log 2023-10-30T13:27:38.567 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65231.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.65231.log.gz 2023-10-30T13:27:38.568 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68198.log 2023-10-30T13:27:38.569 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55603.log 2023-10-30T13:27:38.570 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58449.log.gz 2023-10-30T13:27:38.571 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66823.log 2023-10-30T13:27:38.571 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68198.log.gz 2023-10-30T13:27:38.572 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66714.log 2023-10-30T13:27:38.573 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55603.log.gz 2023-10-30T13:27:38.574 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66823.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62117.log 2023-10-30T13:27:38.574 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66823.log.gz 2023-10-30T13:27:38.575 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63479.log 2023-10-30T13:27:38.576 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66714.log.gz 2023-10-30T13:27:38.577 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52300.log 2023-10-30T13:27:38.577 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62117.log.gz 2023-10-30T13:27:38.578 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49992.log 2023-10-30T13:27:38.578 INFO:teuthology.orchestra.run.smithi125.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63479.log.gz 2023-10-30T13:27:38.580 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54096.log 2023-10-30T13:27:38.580 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52300.log.gz 2023-10-30T13:27:38.581 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49992.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49992.log.gz 2023-10-30T13:27:38.581 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53182.log 2023-10-30T13:27:38.583 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66029.log 2023-10-30T13:27:38.583 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54096.log.gz 2023-10-30T13:27:38.584 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65899.log 2023-10-30T13:27:38.585 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53182.log.gz 2023-10-30T13:27:38.586 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66029.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66029.log.gz 2023-10-30T13:27:38.587 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65899.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65899.log.gz 2023-10-30T13:27:38.588 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57786.log 2023-10-30T13:27:38.589 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62277.log 2023-10-30T13:27:38.591 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57786.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57786.log.gzgzip 2023-10-30T13:27:38.591 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.62442.log 2023-10-30T13:27:38.592 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62038.log 2023-10-30T13:27:38.592 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62277.log.gz 2023-10-30T13:27:38.593 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62442.log: gzip -5 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62442.log.gz 2023-10-30T13:27:38.594 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57216.log 2023-10-30T13:27:38.595 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62038.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.62038.log.gz 2023-10-30T13:27:38.595 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68828.log 2023-10-30T13:27:38.597 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55834.log 2023-10-30T13:27:38.597 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57216.log.gz 2023-10-30T13:27:38.598 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68828.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68828.log.gz 2023-10-30T13:27:38.599 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55834.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55834.log.gz 2023-10-30T13:27:38.600 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55310.log 2023-10-30T13:27:38.601 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70489.log 2023-10-30T13:27:38.603 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65257.log 2023-10-30T13:27:38.603 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55310.log.gz 2023-10-30T13:27:38.604 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70489.log: gzip -5 --verbose -- 88.9% /var/log/ceph/ceph-client.admin.67555.log 2023-10-30T13:27:38.605 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.70489.log.gz 2023-10-30T13:27:38.606 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65257.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.65257.log.gz 2023-10-30T13:27:38.606 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67163.log 2023-10-30T13:27:38.607 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67555.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67555.log.gz 2023-10-30T13:27:38.608 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67423.log 2023-10-30T13:27:38.609 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67163.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.67163.log.gz 2023-10-30T13:27:38.609 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67816.log 2023-10-30T13:27:38.611 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63271.log 2023-10-30T13:27:38.611 INFO:teuthology.orchestra.run.smithi125.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.67423.log.gz 2023-10-30T13:27:38.612 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67816.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.67816.log.gz 2023-10-30T13:27:38.612 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69062.log 2023-10-30T13:27:38.614 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53714.log 2023-10-30T13:27:38.615 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63271.log.gz 2023-10-30T13:27:38.615 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56097.log 2023-10-30T13:27:38.616 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.69062.log.gz 2023-10-30T13:27:38.617 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64039.log 2023-10-30T13:27:38.617 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53714.log.gz 2023-10-30T13:27:38.618 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56097.log: gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.56097.log.gz --verbose 2023-10-30T13:27:38.618 INFO:teuthology.orchestra.run.smithi125.stderr: -- /var/log/ceph/ceph-client.admin.58058.log 2023-10-30T13:27:38.620 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67711.log 2023-10-30T13:27:38.620 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64039.log.gz 2023-10-30T13:27:38.621 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58058.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58058.log.gz -5 2023-10-30T13:27:38.621 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.60138.log 2023-10-30T13:27:38.623 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53286.log 2023-10-30T13:27:38.623 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.67711.log.gz 2023-10-30T13:27:38.624 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60138.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64013.log 2023-10-30T13:27:38.624 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60138.log.gz 2023-10-30T13:27:38.625 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57482.log 2023-10-30T13:27:38.626 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53286.log.gz 2023-10-30T13:27:38.627 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64013.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.64013.log.gz -5 2023-10-30T13:27:38.627 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.67057.log 2023-10-30T13:27:38.628 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44886.log 88.8% 2023-10-30T13:27:38.629 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.57482.log.gz 2023-10-30T13:27:38.629 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64117.log 2023-10-30T13:27:38.630 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67057.log.gz 2023-10-30T13:27:38.631 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69114.log 2023-10-30T13:27:38.631 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44886.log.gz 2023-10-30T13:27:38.633 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64117.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64117.log.gz 2023-10-30T13:27:38.633 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45216.log 2023-10-30T13:27:38.634 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69114.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69114.log.gz 2023-10-30T13:27:38.634 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44877.log 2023-10-30T13:27:38.636 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52382.log 2023-10-30T13:27:38.636 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45216.log.gz 2023-10-30T13:27:38.637 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58527.log 2023-10-30T13:27:38.638 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44877.log.gz 2023-10-30T13:27:38.639 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52382.log: gzip 89.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.52382.log.gz --verbose 2023-10-30T13:27:38.639 INFO:teuthology.orchestra.run.smithi125.stderr: -- /var/log/ceph/ceph-client.admin.52969.log 2023-10-30T13:27:38.640 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58527.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58527.log.gz 2023-10-30T13:27:38.641 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47108.log 2023-10-30T13:27:38.642 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65100.log 2023-10-30T13:27:38.642 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52969.log.gz 2023-10-30T13:27:38.643 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51409.log 2023-10-30T13:27:38.644 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47108.log.gz 2023-10-30T13:27:38.645 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65100.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65100.log.gz 2023-10-30T13:27:38.646 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51409.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51409.log.gz 2023-10-30T13:27:38.647 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60086.log 2023-10-30T13:27:38.648 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68748.log 2023-10-30T13:27:38.650 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60086.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54122.log 2023-10-30T13:27:38.650 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60086.log.gz 2023-10-30T13:27:38.651 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60654.log 2023-10-30T13:27:38.651 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68748.log.gz 2023-10-30T13:27:38.652 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54122.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54122.log.gz 2023-10-30T13:27:38.653 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70385.log 2023-10-30T13:27:38.654 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68854.log 2023-10-30T13:27:38.655 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60654.log.gz 2023-10-30T13:27:38.656 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63533.log 2023-10-30T13:27:38.656 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.70385.log.gz 2023-10-30T13:27:38.657 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57105.log 2023-10-30T13:27:38.658 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.68854.log.gz 2023-10-30T13:27:38.659 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63533.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.63533.log.gz 2023-10-30T13:27:38.659 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48433.log 2023-10-30T13:27:38.660 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57105.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57105.log.gz 2023-10-30T13:27:38.661 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58556.log 2023-10-30T13:27:38.662 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48433.log: gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48433.log.gz -5 2023-10-30T13:27:38.662 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44895.log 2023-10-30T13:27:38.664 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58556.log: gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.58032.log 2023-10-30T13:27:38.664 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.58556.log.gz 2023-10-30T13:27:38.665 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44895.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44895.log.gz 2023-10-30T13:27:38.665 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55862.log 2023-10-30T13:27:38.667 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56421.log 2023-10-30T13:27:38.667 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58032.log.gz 2023-10-30T13:27:38.668 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51040.log 2023-10-30T13:27:38.668 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55862.log.gz 2023-10-30T13:27:38.670 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56421.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56421.log.gz 2023-10-30T13:27:38.671 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51040.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51040.log.gz 2023-10-30T13:27:38.671 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49771.log 2023-10-30T13:27:38.673 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64722.log 2023-10-30T13:27:38.674 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52760.log 2023-10-30T13:27:38.675 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49771.log.gz 2023-10-30T13:27:38.676 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49638.log 2023-10-30T13:27:38.676 INFO:teuthology.orchestra.run.smithi125.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64722.log.gz 2023-10-30T13:27:38.677 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56255.log 2023-10-30T13:27:38.678 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52760.log.gz 2023-10-30T13:27:38.679 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56071.log 2023-10-30T13:27:38.679 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49638.log.gz 2023-10-30T13:27:38.680 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56255.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56255.log.gz 2023-10-30T13:27:38.680 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70463.log 2023-10-30T13:27:38.681 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64774.log 2023-10-30T13:27:38.682 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56071.log.gz 2023-10-30T13:27:38.683 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70463.log: 88.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.70463.log.gz -5 2023-10-30T13:27:38.683 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53154.log 2023-10-30T13:27:38.684 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53345.log 2023-10-30T13:27:38.685 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64774.log.gz 2023-10-30T13:27:38.686 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49499.log 2023-10-30T13:27:38.686 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53154.log.gz 2023-10-30T13:27:38.687 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53345.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.53345.log.gz -5 2023-10-30T13:27:38.687 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44888.log 2023-10-30T13:27:38.689 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56368.log 2023-10-30T13:27:38.689 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49499.log.gz 2023-10-30T13:27:38.690 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44888.log: gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.44888.log.gz --verbose 2023-10-30T13:27:38.691 INFO:teuthology.orchestra.run.smithi125.stderr: -- /var/log/ceph/ceph-client.admin.70645.log 2023-10-30T13:27:38.692 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66003.log 2023-10-30T13:27:38.692 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.56368.log.gz 2023-10-30T13:27:38.693 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70645.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.70645.log.gz 2023-10-30T13:27:38.693 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69351.log 2023-10-30T13:27:38.694 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66003.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66003.log.gz 2023-10-30T13:27:38.695 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63987.log 2023-10-30T13:27:38.696 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68366.log 2023-10-30T13:27:38.697 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.69351.log.gz 2023-10-30T13:27:38.698 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63987.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.63987.log.gz 2023-10-30T13:27:38.699 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68366.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68366.log.gz 2023-10-30T13:27:38.700 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61985.log 2023-10-30T13:27:38.701 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69010.log 2023-10-30T13:27:38.703 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63821.log 2023-10-30T13:27:38.703 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61985.log.gz 2023-10-30T13:27:38.704 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57890.log 2023-10-30T13:27:38.705 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69010.log.gz 2023-10-30T13:27:38.706 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70229.log 2023-10-30T13:27:38.706 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63821.log.gz 2023-10-30T13:27:38.707 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57890.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57890.log.gz 2023-10-30T13:27:38.708 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69168.log 2023-10-30T13:27:38.709 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70229.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70229.log.gz 2023-10-30T13:27:38.710 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44425.log 2023-10-30T13:27:38.710 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69168.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.69168.log.gz 2023-10-30T13:27:38.711 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54890.log 2023-10-30T13:27:38.713 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44425.log: gzip 88.2% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.44425.log.gz -- 2023-10-30T13:27:38.713 INFO:teuthology.orchestra.run.smithi125.stderr: /var/log/ceph/ceph-client.admin.60461.log 2023-10-30T13:27:38.714 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45676.log 2023-10-30T13:27:38.714 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54890.log.gz 2023-10-30T13:27:38.716 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60461.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60461.log.gz 2023-10-30T13:27:38.717 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45676.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45676.log.gz 2023-10-30T13:27:38.717 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-30T13:27:38.719 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61355.log 2023-10-30T13:27:38.722 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63087.log 2023-10-30T13:27:38.722 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61355.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61355.log.gz 2023-10-30T13:27:38.723 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63087.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63087.log.gz 2023-10-30T13:27:38.724 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55993.log 2023-10-30T13:27:38.725 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65977.log 2023-10-30T13:27:38.727 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55993.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55993.log.gz 2023-10-30T13:27:38.728 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65977.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65977.log.gz 2023-10-30T13:27:38.728 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44683.log 2023-10-30T13:27:38.730 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-30T13:27:38.730 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66901.log 2023-10-30T13:27:38.731 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46962.log 2023-10-30T13:27:38.732 INFO:teuthology.orchestra.run.smithi125.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44683.log.gz 2023-10-30T13:27:38.733 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46541.log 2023-10-30T13:27:38.733 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66901.log.gz 2023-10-30T13:27:38.734 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46962.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.46962.log.gz -5 2023-10-30T13:27:38.734 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.60599.log 2023-10-30T13:27:38.736 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60869.log 2023-10-30T13:27:38.736 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46541.log.gz 2023-10-30T13:27:38.737 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65491.log 2023-10-30T13:27:38.737 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60599.log.gz 2023-10-30T13:27:38.739 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60869.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50553.log 2023-10-30T13:27:38.739 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60869.log.gz 2023-10-30T13:27:38.740 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44669.log 2023-10-30T13:27:38.741 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65491.log.gz 2023-10-30T13:27:38.742 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49421.log 2023-10-30T13:27:38.742 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50553.log.gz 2023-10-30T13:27:38.743 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65925.log 2023-10-30T13:27:38.743 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.44669.log.gz 2023-10-30T13:27:38.745 INFO:teuthology.orchestra.run.smithi136.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-30T13:27:38.745 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70567.log 2023-10-30T13:27:38.746 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49421.log.gz 2023-10-30T13:27:38.746 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65925.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65925.log.gz 2023-10-30T13:27:38.747 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63245.log 2023-10-30T13:27:38.748 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63167.log 2023-10-30T13:27:38.748 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.70567.log.gz 2023-10-30T13:27:38.749 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63245.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63245.log.gzgzip 2023-10-30T13:27:38.749 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53260.log 2023-10-30T13:27:38.751 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44675.log 2023-10-30T13:27:38.751 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63167.log.gz 2023-10-30T13:27:38.752 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54070.log 2023-10-30T13:27:38.752 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53260.log.gz 2023-10-30T13:27:38.754 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69909.log 2023-10-30T13:27:38.754 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44675.log.gz 2023-10-30T13:27:38.755 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54070.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54070.log.gz 2023-10-30T13:27:38.756 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44178.log 2023-10-30T13:27:38.756 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69909.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.69909.log.gz 2023-10-30T13:27:38.757 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48275.log 2023-10-30T13:27:38.758 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51383.log 2023-10-30T13:27:38.759 INFO:teuthology.orchestra.run.smithi125.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44178.log.gz 2023-10-30T13:27:38.760 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59215.log 2023-10-30T13:27:38.760 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48275.log.gz 2023-10-30T13:27:38.761 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51383.log: gzip -5 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51383.log.gz 2023-10-30T13:27:38.762 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57079.log 2023-10-30T13:27:38.763 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59215.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59215.log.gz 2023-10-30T13:27:38.763 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60734.log 2023-10-30T13:27:38.765 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56747.log 2023-10-30T13:27:38.765 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.57079.log.gz 2023-10-30T13:27:38.766 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60734.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60734.log.gz 2023-10-30T13:27:38.767 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53047.log 2023-10-30T13:27:38.767 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56747.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56747.log.gz 2023-10-30T13:27:38.768 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63323.log 2023-10-30T13:27:38.769 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60112.log 2023-10-30T13:27:38.770 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53047.log.gz 2023-10-30T13:27:38.771 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51846.log 2023-10-30T13:27:38.771 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63323.log.gz 2023-10-30T13:27:38.772 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45264.log 2023-10-30T13:27:38.773 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60112.log.gz 2023-10-30T13:27:38.774 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59811.log 2023-10-30T13:27:38.774 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.51846.log.gz 2023-10-30T13:27:38.776 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60978.log 2023-10-30T13:27:38.776 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45264.log.gz 2023-10-30T13:27:38.777 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47071.log 2023-10-30T13:27:38.777 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59811.log.gz 2023-10-30T13:27:38.778 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44260.log 89.2% 2023-10-30T13:27:38.779 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60978.log.gz 2023-10-30T13:27:38.779 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47071.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47071.log.gz 2023-10-30T13:27:38.780 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60573.log 2023-10-30T13:27:38.782 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51146.log 2023-10-30T13:27:38.782 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44260.log.gz 2023-10-30T13:27:38.783 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44451.log 2023-10-30T13:27:38.783 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60573.log.gz 2023-10-30T13:27:38.785 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51146.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50420.log 2023-10-30T13:27:38.785 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51146.log.gz 2023-10-30T13:27:38.786 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64223.log 2023-10-30T13:27:38.786 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44451.log.gz 2023-10-30T13:27:38.787 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65743.log 2023-10-30T13:27:38.788 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50420.log.gz 2023-10-30T13:27:38.789 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49850.log 2023-10-30T13:27:38.789 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64223.log.gz 2023-10-30T13:27:38.790 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65743.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65743.log.gz 2023-10-30T13:27:38.791 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53688.log 2023-10-30T13:27:38.792 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69220.log 2023-10-30T13:27:38.792 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.49850.log.gz 2023-10-30T13:27:38.793 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53688.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69771.log 2023-10-30T13:27:38.794 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53688.log.gz 2023-10-30T13:27:38.795 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69220.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69220.log.gz 2023-10-30T13:27:38.795 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67243.log 2023-10-30T13:27:38.797 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69771.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69771.log.gz 2023-10-30T13:27:38.797 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65611.log 2023-10-30T13:27:38.798 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68722.log 2023-10-30T13:27:38.798 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67243.log.gz 2023-10-30T13:27:38.800 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51567.log 2023-10-30T13:27:38.800 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65611.log.gz 2023-10-30T13:27:38.801 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59707.log 2023-10-30T13:27:38.801 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68722.log.gz 2023-10-30T13:27:38.803 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62873.log 2023-10-30T13:27:38.803 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51567.log.gz 2023-10-30T13:27:38.804 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60680.log 2023-10-30T13:27:38.804 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59707.log.gz 2023-10-30T13:27:38.805 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62873.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62873.log.gz 2023-10-30T13:27:38.807 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60680.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60680.log.gz 2023-10-30T13:27:38.807 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63141.log 2023-10-30T13:27:38.808 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64330.log 2023-10-30T13:27:38.810 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63141.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57456.log 2023-10-30T13:27:38.811 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63141.log.gz 2023-10-30T13:27:38.812 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64330.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.64330.log.gz 2023-10-30T13:27:38.812 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.57977.log 2023-10-30T13:27:38.813 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57456.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58828.log 2023-10-30T13:27:38.814 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57456.log.gz 2023-10-30T13:27:38.815 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57977.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57977.log.gz 2023-10-30T13:27:38.816 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58828.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58828.log.gz 2023-10-30T13:27:38.817 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49525.log 2023-10-30T13:27:38.818 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44634.log 2023-10-30T13:27:38.820 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68670.log 2023-10-30T13:27:38.820 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49525.log.gz 2023-10-30T13:27:38.821 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44634.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44634.log.gz 2023-10-30T13:27:38.821 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48223.log 2023-10-30T13:27:38.823 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68670.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.68670.log.gz -5 2023-10-30T13:27:38.823 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.61461.log 2023-10-30T13:27:38.824 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57183.log 2023-10-30T13:27:38.825 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48223.log.gz 2023-10-30T13:27:38.826 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58998.log 2023-10-30T13:27:38.826 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61461.log.gz 2023-10-30T13:27:38.827 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62225.log 2023-10-30T13:27:38.827 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57183.log.gz 2023-10-30T13:27:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70030.log 2023-10-30T13:27:38.829 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58998.log.gz 2023-10-30T13:27:38.830 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38077.log 2023-10-30T13:27:38.830 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62225.log.gz 2023-10-30T13:27:38.831 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53504.log 2023-10-30T13:27:38.832 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70030.log.gz 2023-10-30T13:27:38.833 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.38077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61850.log 2023-10-30T13:27:38.833 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38077.log.gz 2023-10-30T13:27:38.834 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48090.log 2023-10-30T13:27:38.835 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53504.log.gz 2023-10-30T13:27:38.836 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61850.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61850.log.gz 2023-10-30T13:27:38.837 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68309.log 2023-10-30T13:27:38.837 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48090.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48090.log.gz 2023-10-30T13:27:38.838 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66200.log 2023-10-30T13:27:38.839 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63193.log 2023-10-30T13:27:38.840 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68309.log.gz 2023-10-30T13:27:38.841 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68067.log 2023-10-30T13:27:38.841 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66200.log.gz 2023-10-30T13:27:38.842 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63193.log: gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.63193.log.gz 2023-10-30T13:27:38.843 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45364.log 2023-10-30T13:27:38.844 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68067.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68067.log.gz 2023-10-30T13:27:38.844 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65821.log 2023-10-30T13:27:38.845 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45364.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45364.log.gz 2023-10-30T13:27:38.846 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63559.log 2023-10-30T13:27:38.847 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52917.log 2023-10-30T13:27:38.847 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65821.log.gz 2023-10-30T13:27:38.849 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58339.log 2023-10-30T13:27:38.849 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.63559.log.gz 2023-10-30T13:27:38.850 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70437.log 2023-10-30T13:27:38.850 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52917.log.gz 2023-10-30T13:27:38.852 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69429.log 2023-10-30T13:27:38.852 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.58339.log.gz 2023-10-30T13:27:38.853 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70437.log: gzip -5 --verbose 88.5% -- -- replaced with /var/log/ceph/ceph-client.admin.70437.log.gz /var/log/ceph/ceph-client.admin.51357.log 2023-10-30T13:27:38.853 INFO:teuthology.orchestra.run.smithi125.stderr: 2023-10-30T13:27:38.854 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66875.log 2023-10-30T13:27:38.855 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69429.log.gz 2023-10-30T13:27:38.855 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51357.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51357.log.gz 2023-10-30T13:27:38.856 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61655.log 2023-10-30T13:27:38.857 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66875.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66875.log.gz 2023-10-30T13:27:38.858 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46936.log 2023-10-30T13:27:38.859 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69665.log 2023-10-30T13:27:38.859 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61655.log.gz 2023-10-30T13:27:38.860 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46936.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46936.log.gz 2023-10-30T13:27:38.862 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69665.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.69665.log.gz 2023-10-30T13:27:38.862 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71002.log 2023-10-30T13:27:38.864 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67607.log 2023-10-30T13:27:38.865 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.71002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59400.log 2023-10-30T13:27:38.866 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.71002.log.gz 2023-10-30T13:27:38.867 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67607.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50314.log 2023-10-30T13:27:38.867 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67607.log.gz 2023-10-30T13:27:38.869 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46674.log 2023-10-30T13:27:38.869 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.59400.log.gz 2023-10-30T13:27:38.870 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60357.log 2023-10-30T13:27:38.870 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50314.log.gz 2023-10-30T13:27:38.871 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55074.log 2023-10-30T13:27:38.872 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46674.log.gz 2023-10-30T13:27:38.873 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68522.log 2023-10-30T13:27:38.873 INFO:teuthology.orchestra.run.smithi125.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60357.log.gz 2023-10-30T13:27:38.874 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55074.log: gzip -5 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55074.log.gz 2023-10-30T13:27:38.942 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.46130.log 2023-10-30T13:27:38.942 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68522.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68522.log.gz 2023-10-30T13:27:38.943 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61248.log 2023-10-30T13:27:38.943 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46130.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46130.log.gz 2023-10-30T13:27:38.943 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54994.log 2023-10-30T13:27:38.944 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61248.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61248.log.gz 2023-10-30T13:27:38.944 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60917.log 2023-10-30T13:27:38.944 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54335.log 2023-10-30T13:27:38.944 INFO:teuthology.orchestra.run.smithi125.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.54994.log.gz 2023-10-30T13:27:38.945 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44204.log 2023-10-30T13:27:38.945 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60917.log.gz 2023-10-30T13:27:38.945 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54335.log: gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.69377.log 2023-10-30T13:27:38.945 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54335.log.gz 2023-10-30T13:27:38.946 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70808.log 2023-10-30T13:27:38.946 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44204.log.gz 2023-10-30T13:27:38.946 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69377.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69377.log.gz 2023-10-30T13:27:38.947 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63035.log 2023-10-30T13:27:38.947 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70808.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70808.log.gz 2023-10-30T13:27:38.947 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45520.log 2023-10-30T13:27:38.947 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63035.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.63035.log.gz -5 2023-10-30T13:27:38.948 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55441.log 2023-10-30T13:27:38.948 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45520.log: 88.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.45520.log.gz -5 2023-10-30T13:27:38.948 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.55389.log 2023-10-30T13:27:38.949 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62705.log 2023-10-30T13:27:38.949 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.55441.log.gz 2023-10-30T13:27:38.949 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51699.log 2023-10-30T13:27:38.949 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55389.log.gz 2023-10-30T13:27:38.950 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62303.log 2023-10-30T13:27:38.950 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.62705.log.gz 2023-10-30T13:27:38.950 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57812.log 2023-10-30T13:27:38.951 INFO:teuthology.orchestra.run.smithi125.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51699.log.gz 2023-10-30T13:27:38.951 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62303.log: gzip -5 87.9% -- replaced with /var/log/ceph/ceph-client.admin.62303.log.gz 2023-10-30T13:27:38.951 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.62090.log 2023-10-30T13:27:38.951 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57812.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57812.log.gz 2023-10-30T13:27:38.952 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69325.log 2023-10-30T13:27:38.952 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67371.log 2023-10-30T13:27:38.952 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62090.log.gz 2023-10-30T13:27:38.953 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69325.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69325.log.gz 2023-10-30T13:27:38.953 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54280.log 2023-10-30T13:27:38.953 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57324.log 2023-10-30T13:27:38.953 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67371.log.gz 2023-10-30T13:27:38.954 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54280.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54280.log.gzgzip 2023-10-30T13:27:38.954 INFO:teuthology.orchestra.run.smithi125.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.61383.log 2023-10-30T13:27:38.954 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57324.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.57324.log.gz -5 2023-10-30T13:27:38.955 INFO:teuthology.orchestra.run.smithi125.stderr: --verbose -- /var/log/ceph/ceph-client.admin.44071.log 2023-10-30T13:27:38.955 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61383.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61383.log.gz 2023-10-30T13:27:38.955 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56773.log 2023-10-30T13:27:38.955 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55727.log 2023-10-30T13:27:38.956 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44071.log.gz 2023-10-30T13:27:38.956 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64852.log 2023-10-30T13:27:38.956 INFO:teuthology.orchestra.run.smithi125.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.56773.log.gz 2023-10-30T13:27:38.956 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55100.log 2023-10-30T13:27:38.957 INFO:teuthology.orchestra.run.smithi125.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55727.log.gz 2023-10-30T13:27:38.957 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64852.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64852.log.gz 2023-10-30T13:27:38.957 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66579.log 2023-10-30T13:27:38.957 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54228.log 2023-10-30T13:27:38.958 INFO:teuthology.orchestra.run.smithi125.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55100.log.gz 2023-10-30T13:27:38.958 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66579.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-30T13:27:38.958 INFO:teuthology.orchestra.run.smithi125.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66579.log.gz 2023-10-30T13:27:38.961 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48407.log 2023-10-30T13:27:38.961 INFO:teuthology.orchestra.run.smithi125.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.54228.log.gz 2023-10-30T13:27:38.962 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61930.log 2023-10-30T13:27:38.962 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48407.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.48407.log.gz 2023-10-30T13:27:38.962 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61930.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61930.log.gz 2023-10-30T13:27:38.963 INFO:teuthology.orchestra.run.smithi125.stderr: 89.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-30T13:27:38.963 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60033.log 2023-10-30T13:27:38.963 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-30T13:27:38.964 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56395.log 2023-10-30T13:27:38.964 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60033.log.gz 2023-10-30T13:27:38.964 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67397.log 2023-10-30T13:27:38.964 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56395.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56395.log.gz 2023-10-30T13:27:38.965 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67397.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67397.log.gz 2023-10-30T13:27:38.965 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65769.log 2023-10-30T13:27:38.965 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45468.log 2023-10-30T13:27:38.966 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65769.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65769.log.gz 2023-10-30T13:27:38.966 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65387.log 2023-10-30T13:27:38.966 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45468.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.45468.log.gz 2023-10-30T13:27:38.966 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56125.log 2023-10-30T13:27:38.967 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65387.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65387.log.gz 2023-10-30T13:27:38.967 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56125.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56125.log.gz 2023-10-30T13:27:38.967 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68224.log 2023-10-30T13:27:38.972 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-30T13:27:38.973 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68224.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.68224.log.gz 2023-10-30T13:27:38.973 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65309.log 2023-10-30T13:27:38.973 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61004.log 2023-10-30T13:27:38.974 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65309.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65309.log.gz 2023-10-30T13:27:38.974 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59102.log 2023-10-30T13:27:38.974 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69533.log 2023-10-30T13:27:38.975 INFO:teuthology.orchestra.run.smithi125.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.61004.log.gz 2023-10-30T13:27:38.975 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69533.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69533.log.gz 2023-10-30T13:27:38.975 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43988.log 2023-10-30T13:27:38.975 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49824.log 2023-10-30T13:27:38.976 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59102.log.gz 2023-10-30T13:27:38.976 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63767.log 2023-10-30T13:27:38.976 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49824.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49824.log.gz 2023-10-30T13:27:38.977 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45572.log 2023-10-30T13:27:38.977 INFO:teuthology.orchestra.run.smithi125.stderr: 94.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59076.log 2023-10-30T13:27:38.977 INFO:teuthology.orchestra.run.smithi125.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43988.log.gz 2023-10-30T13:27:38.977 INFO:teuthology.orchestra.run.smithi125.stderr: -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-30T13:27:38.978 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45572.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45572.log.gz 2023-10-30T13:27:38.978 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63767.log: /var/log/ceph/ceph-client.admin.59076.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59076.log.gz 2023-10-30T13:27:38.978 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63767.log.gz 2023-10-30T13:27:38.978 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52326.log 2023-10-30T13:27:38.979 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58136.log 2023-10-30T13:27:38.979 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52326.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52326.log.gz 2023-10-30T13:27:38.979 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60812.log 2023-10-30T13:27:38.980 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58136.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58136.log.gz 2023-10-30T13:27:38.982 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45598.log 2023-10-30T13:27:38.982 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60812.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60812.log.gz 2023-10-30T13:27:38.982 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51950.log 2023-10-30T13:27:38.983 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45598.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.45598.log.gz 2023-10-30T13:27:38.983 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66063.log 2023-10-30T13:27:38.983 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51950.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51950.log.gz 2023-10-30T13:27:38.984 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60708.log 2023-10-30T13:27:38.984 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66063.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66063.log.gz 2023-10-30T13:27:38.984 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52498.log 2023-10-30T13:27:38.985 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60708.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.60708.log.gz 2023-10-30T13:27:38.986 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47139.log 2023-10-30T13:27:38.986 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52498.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52498.log.gz 2023-10-30T13:27:38.986 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61084.log 2023-10-30T13:27:38.987 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.47139.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47139.log.gz 2023-10-30T13:27:38.987 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55467.log 2023-10-30T13:27:38.988 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61084.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61084.log.gz 2023-10-30T13:27:38.989 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70730.log 2023-10-30T13:27:38.990 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55467.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55467.log.gz 2023-10-30T13:27:38.991 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53991.log 2023-10-30T13:27:38.992 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70730.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.70730.log.gz 2023-10-30T13:27:38.993 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68600.log 2023-10-30T13:27:38.994 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53991.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53991.log.gz 2023-10-30T13:27:38.995 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68418.log 2023-10-30T13:27:38.995 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68600.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68600.log.gz 2023-10-30T13:27:38.997 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56695.log 2023-10-30T13:27:38.998 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68418.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68418.log.gz 2023-10-30T13:27:38.999 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61277.log 2023-10-30T13:27:38.999 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56695.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56695.log.gz 2023-10-30T13:27:39.000 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63427.log 2023-10-30T13:27:39.002 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61277.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61277.log.gz 2023-10-30T13:27:39.002 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51199.log 2023-10-30T13:27:39.003 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63427.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.63427.log.gz 2023-10-30T13:27:39.004 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67737.log 2023-10-30T13:27:39.006 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51199.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51199.log.gz 2023-10-30T13:27:39.006 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60760.log 2023-10-30T13:27:39.007 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67737.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67737.log.gz 2023-10-30T13:27:39.008 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54784.log 2023-10-30T13:27:39.010 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60760.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.60760.log.gz 2023-10-30T13:27:39.010 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46156.log 2023-10-30T13:27:39.011 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54784.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.54784.log.gz 2023-10-30T13:27:39.012 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70056.log 2023-10-30T13:27:39.013 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46156.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46156.log.gz 2023-10-30T13:27:39.014 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59785.log 2023-10-30T13:27:39.015 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70056.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70056.log.gz 2023-10-30T13:27:39.016 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.33227.log 2023-10-30T13:27:39.017 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59785.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59785.log.gz 2023-10-30T13:27:39.018 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69455.log 2023-10-30T13:27:39.019 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph.tmp-client.admin.33227.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.33227.log.gz 2023-10-30T13:27:39.019 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61792.log 2023-10-30T13:27:39.021 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69455.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69455.log.gz 2023-10-30T13:27:39.022 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69559.log 2023-10-30T13:27:39.022 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61792.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61792.log.gz 2023-10-30T13:27:39.027 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56904.log 2023-10-30T13:27:39.027 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69559.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69559.log.gz 2023-10-30T13:27:39.027 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45854.log 2023-10-30T13:27:39.028 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56904.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56904.log.gz 2023-10-30T13:27:39.028 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56939.log 2023-10-30T13:27:39.029 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45854.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.45854.log.gz 2023-10-30T13:27:39.030 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46234.log 2023-10-30T13:27:39.031 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56939.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56939.log.gz 2023-10-30T13:27:39.031 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65951.log 2023-10-30T13:27:39.033 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46234.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46234.log.gz 2023-10-30T13:27:39.033 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49586.log 2023-10-30T13:27:39.034 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65951.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65951.log.gz 2023-10-30T13:27:39.035 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64618.log 2023-10-30T13:27:39.036 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49586.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49586.log.gz 2023-10-30T13:27:39.037 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62840.log 2023-10-30T13:27:39.038 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64618.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64618.log.gz 2023-10-30T13:27:39.039 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63741.log 2023-10-30T13:27:39.040 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62840.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62840.log.gz 2023-10-30T13:27:39.041 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60786.log 2023-10-30T13:27:39.042 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63741.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63741.log.gz 2023-10-30T13:27:39.042 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70593.log 2023-10-30T13:27:39.044 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60786.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60786.log.gz 2023-10-30T13:27:39.045 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69797.log 2023-10-30T13:27:39.045 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70593.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70593.log.gz 2023-10-30T13:27:39.046 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51593.log 2023-10-30T13:27:39.048 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69797.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.69797.log.gz 2023-10-30T13:27:39.049 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50366.log 2023-10-30T13:27:39.050 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51593.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51593.log.gz 2023-10-30T13:27:39.050 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58006.log 2023-10-30T13:27:39.052 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50366.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50366.log.gz 2023-10-30T13:27:39.052 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60838.log 2023-10-30T13:27:39.053 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58006.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58006.log.gz 2023-10-30T13:27:39.055 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67764.log 2023-10-30T13:27:39.055 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60838.log.gz 2023-10-30T13:27:39.056 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54174.log 2023-10-30T13:27:39.058 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67764.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67764.log.gz 2023-10-30T13:27:39.058 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54650.log 2023-10-30T13:27:39.059 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54174.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.54174.log.gz 2023-10-30T13:27:39.060 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63297.log 2023-10-30T13:27:39.061 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54650.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54650.log.gz 2023-10-30T13:27:39.062 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65413.log 2023-10-30T13:27:39.063 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63297.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63297.log.gz 2023-10-30T13:27:39.064 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54459.log 2023-10-30T13:27:39.065 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65413.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65413.log.gz 2023-10-30T13:27:39.066 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53636.log 2023-10-30T13:27:39.067 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54459.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54459.log.gz 2023-10-30T13:27:39.068 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43952.log 2023-10-30T13:27:39.069 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53636.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53636.log.gz 2023-10-30T13:27:39.070 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54916.log 2023-10-30T13:27:39.071 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.43952.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43952.log.gz 2023-10-30T13:27:39.072 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54200.log 2023-10-30T13:27:39.073 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54916.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54916.log.gz 2023-10-30T13:27:39.074 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64696.log 2023-10-30T13:27:39.075 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54200.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54200.log.gz 2023-10-30T13:27:39.076 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64514.log 2023-10-30T13:27:39.077 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64696.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64696.log.gz 2023-10-30T13:27:39.077 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54591.log 2023-10-30T13:27:39.079 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64514.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64514.log.gz 2023-10-30T13:27:39.080 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63961.log 2023-10-30T13:27:39.080 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54591.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54591.log.gz 2023-10-30T13:27:39.082 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44504.log 2023-10-30T13:27:39.083 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63961.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63961.log.gz 2023-10-30T13:27:39.083 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67973.log 2023-10-30T13:27:39.085 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44504.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44504.log.gz 2023-10-30T13:27:39.086 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54381.log 2023-10-30T13:27:39.086 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67973.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67973.log.gz 2023-10-30T13:27:39.087 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46910.log 2023-10-30T13:27:39.089 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54381.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54381.log.gz 2023-10-30T13:27:39.090 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68496.log 2023-10-30T13:27:39.090 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46910.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46910.log.gz 2023-10-30T13:27:39.091 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56499.log 2023-10-30T13:27:39.093 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68496.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68496.log.gz 2023-10-30T13:27:39.093 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59655.log 2023-10-30T13:27:39.094 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56499.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56499.log.gz 2023-10-30T13:27:39.095 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44889.log 2023-10-30T13:27:39.097 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59655.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.59655.log.gz 2023-10-30T13:27:39.097 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55415.log 2023-10-30T13:27:39.098 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44889.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44889.log.gz 2023-10-30T13:27:39.099 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67685.log 2023-10-30T13:27:39.101 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55415.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55415.log.gz 2023-10-30T13:27:39.101 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54513.log 2023-10-30T13:27:39.102 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67685.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67685.log.gz 2023-10-30T13:27:39.103 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60250.log 2023-10-30T13:27:39.104 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54513.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54513.log.gz 2023-10-30T13:27:39.105 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67109.log 2023-10-30T13:27:39.106 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60250.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60250.log.gz 2023-10-30T13:27:39.107 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44230.log 2023-10-30T13:27:39.108 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67109.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.67109.log.gz 2023-10-30T13:27:39.109 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45338.log 2023-10-30T13:27:39.110 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44230.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44230.log.gz 2023-10-30T13:27:39.111 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52004.log 2023-10-30T13:27:39.112 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45338.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45338.log.gz 2023-10-30T13:27:39.113 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56669.log 2023-10-30T13:27:39.114 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52004.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.52004.log.gz 2023-10-30T13:27:39.115 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52352.log 2023-10-30T13:27:39.116 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56669.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56669.log.gz 2023-10-30T13:27:39.117 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44126.log 2023-10-30T13:27:39.117 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2023-10-30T13:27:39.118 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52352.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52352.log.gz 2023-10-30T13:27:39.119 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59024.log 2023-10-30T13:27:39.120 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44126.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44126.log.gz 2023-10-30T13:27:39.121 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50104.log 2023-10-30T13:27:39.122 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59024.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.59024.log.gz 2023-10-30T13:27:39.123 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50340.log 2023-10-30T13:27:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50104.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50104.log.gz 2023-10-30T13:27:39.125 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61058.log 2023-10-30T13:27:39.126 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50340.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50340.log.gz 2023-10-30T13:27:39.127 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70333.log 2023-10-30T13:27:39.128 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61058.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.61058.log.gz 2023-10-30T13:27:39.128 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59627.log 2023-10-30T13:27:39.130 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70333.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70333.log.gz 2023-10-30T13:27:39.131 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69823.log 2023-10-30T13:27:39.132 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59627.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59627.log.gz 2023-10-30T13:27:39.132 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66304.log 2023-10-30T13:27:39.134 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69823.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.69823.log.gz 2023-10-30T13:27:39.135 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69639.log 2023-10-30T13:27:39.135 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66304.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66304.log.gz 2023-10-30T13:27:39.136 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64800.log 2023-10-30T13:27:39.138 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69639.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69639.log.gz 2023-10-30T13:27:39.138 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53965.log 2023-10-30T13:27:39.139 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64800.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64800.log.gz 2023-10-30T13:27:39.140 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54942.log 2023-10-30T13:27:39.142 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53965.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53965.log.gz 2023-10-30T13:27:39.143 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58644.log 2023-10-30T13:27:39.143 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54942.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54942.log.gz 2023-10-30T13:27:39.144 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59870.log 2023-10-30T13:27:39.146 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58644.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58644.log.gz 2023-10-30T13:27:39.146 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59426.log 2023-10-30T13:27:39.147 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59870.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59870.log.gz 2023-10-30T13:27:39.149 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53452.log 2023-10-30T13:27:39.149 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59426.log.gz 2023-10-30T13:27:39.150 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55206.log 2023-10-30T13:27:39.152 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53452.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53452.log.gz 2023-10-30T13:27:39.152 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.46023.log 2023-10-30T13:27:39.153 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55206.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55206.log.gz 2023-10-30T13:27:39.154 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56551.log 2023-10-30T13:27:39.155 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.0.46023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.46023.log.gz 2023-10-30T13:27:39.156 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65717.log 2023-10-30T13:27:39.157 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56551.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56551.log.gz 2023-10-30T13:27:39.158 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64460.log 2023-10-30T13:27:39.159 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65717.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.65717.log.gz 2023-10-30T13:27:39.160 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67296.log 2023-10-30T13:27:39.161 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64460.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.64460.log.gz 2023-10-30T13:27:39.162 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52222.log 2023-10-30T13:27:39.163 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67296.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67296.log.gz 2023-10-30T13:27:39.164 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55493.log 2023-10-30T13:27:39.165 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52222.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52222.log.gz 2023-10-30T13:27:39.166 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60514.log 2023-10-30T13:27:39.170 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55493.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55493.log.gz 2023-10-30T13:27:39.171 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50182.log 2023-10-30T13:27:39.171 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60514.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60514.log.gz 2023-10-30T13:27:39.171 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63375.log 2023-10-30T13:27:39.172 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.50182.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50182.log.gz 2023-10-30T13:27:39.173 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54732.log 2023-10-30T13:27:39.173 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63375.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.63375.log.gz 2023-10-30T13:27:39.174 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58854.log 2023-10-30T13:27:39.175 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54732.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54732.log.gz 2023-10-30T13:27:39.176 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45076.log 2023-10-30T13:27:39.177 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58854.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58854.log.gz 2023-10-30T13:27:39.178 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55940.log 2023-10-30T13:27:39.179 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45076.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45076.log.gz 2023-10-30T13:27:39.179 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67633.log 2023-10-30T13:27:39.181 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55940.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55940.log.gz 2023-10-30T13:27:39.182 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69507.log 2023-10-30T13:27:39.182 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67633.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67633.log.gz 2023-10-30T13:27:39.183 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38312.log 2023-10-30T13:27:39.185 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69507.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69507.log.gz 2023-10-30T13:27:39.185 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64145.log 2023-10-30T13:27:39.186 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.38312.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.38312.log.gz 2023-10-30T13:27:39.187 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58696.log 2023-10-30T13:27:39.189 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64145.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64145.log.gz 2023-10-30T13:27:39.189 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58931.log 2023-10-30T13:27:39.190 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58696.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58696.log.gz 2023-10-30T13:27:39.192 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62601.log 2023-10-30T13:27:39.192 INFO:teuthology.orchestra.run.smithi125.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.58931.log.gz 2023-10-30T13:27:39.193 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62955.log 2023-10-30T13:27:39.194 INFO:teuthology.orchestra.run.smithi136.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-30T13:27:39.195 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62601.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62601.log.gz 2023-10-30T13:27:39.196 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61303.log 2023-10-30T13:27:39.197 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62955.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.62955.log.gz 2023-10-30T13:27:39.197 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68958.log 2023-10-30T13:27:39.199 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61303.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.61303.log.gz 2023-10-30T13:27:39.199 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66392.log 2023-10-30T13:27:39.200 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68958.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.68958.log.gz 2023-10-30T13:27:39.201 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70976.log 2023-10-30T13:27:39.203 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66392.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.66392.log.gz 2023-10-30T13:27:39.203 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44152.log 2023-10-30T13:27:39.204 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70976.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70976.log.gz 2023-10-30T13:27:39.205 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53078.log 2023-10-30T13:27:39.206 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44152.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44152.log.gz 2023-10-30T13:27:39.207 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56045.log 2023-10-30T13:27:39.208 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53078.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53078.log.gz 2023-10-30T13:27:39.209 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64592.log 2023-10-30T13:27:39.210 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56045.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56045.log.gz 2023-10-30T13:27:39.211 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67844.log 2023-10-30T13:27:39.212 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64592.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.64592.log.gz 2023-10-30T13:27:39.213 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68627.log 2023-10-30T13:27:39.214 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67844.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67844.log.gz 2023-10-30T13:27:39.215 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62176.log 2023-10-30T13:27:39.216 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68627.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68627.log.gz 2023-10-30T13:27:39.217 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49876.log 2023-10-30T13:27:39.218 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62176.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.62176.log.gz 2023-10-30T13:27:39.219 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69875.log 2023-10-30T13:27:39.220 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49876.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49876.log.gz 2023-10-30T13:27:39.220 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65465.log 2023-10-30T13:27:39.222 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69875.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.69875.log.gz 2023-10-30T13:27:39.223 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44478.log 2023-10-30T13:27:39.224 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65465.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65465.log.gz 2023-10-30T13:27:39.224 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67031.log 2023-10-30T13:27:39.226 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44478.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44478.log.gz 2023-10-30T13:27:39.227 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56985.log 2023-10-30T13:27:39.228 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67031.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67031.log.gz 2023-10-30T13:27:39.229 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46078.log 2023-10-30T13:27:39.230 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56985.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.56985.log.gz 2023-10-30T13:27:39.230 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53021.log 2023-10-30T13:27:39.232 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46078.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46078.log.gz 2023-10-30T13:27:39.232 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66109.log 2023-10-30T13:27:39.233 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53021.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53021.log.gz 2023-10-30T13:27:39.234 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46288.log 2023-10-30T13:27:39.236 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66109.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66109.log.gz 2023-10-30T13:27:39.236 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51093.log 2023-10-30T13:27:39.237 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46288.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46288.log.gz 2023-10-30T13:27:39.238 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53805.log 2023-10-30T13:27:39.239 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51093.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.51093.log.gz 2023-10-30T13:27:39.240 INFO:teuthology.orchestra.run.smithi125.stderr:gzip/var/log/ceph/ceph-client.admin.53805.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.59452.log 2023-10-30T13:27:39.241 INFO:teuthology.orchestra.run.smithi125.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53805.log.gz 2023-10-30T13:27:39.242 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70515.log 2023-10-30T13:27:39.243 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59452.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59452.log.gz 2023-10-30T13:27:39.244 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51924.log 2023-10-30T13:27:39.245 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70515.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.70515.log.gz 2023-10-30T13:27:39.246 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58423.log 2023-10-30T13:27:39.247 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51924.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51924.log.gz 2023-10-30T13:27:39.248 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61196.log 2023-10-30T13:27:39.249 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58423.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58423.log.gz 2023-10-30T13:27:39.250 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55675.log 2023-10-30T13:27:39.251 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61196.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.61196.log.gz 2023-10-30T13:27:39.252 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60409.log 2023-10-30T13:27:39.253 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55675.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.55675.log.gz 2023-10-30T13:27:39.254 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58615.log 2023-10-30T13:27:39.255 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60409.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60409.log.gz 2023-10-30T13:27:39.256 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70782.log 2023-10-30T13:27:39.257 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58615.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.58615.log.gz 2023-10-30T13:27:39.258 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61878.log 2023-10-30T13:27:39.259 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70782.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70782.log.gz 2023-10-30T13:27:39.260 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57350.log 2023-10-30T13:27:39.261 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61878.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.61878.log.gz 2023-10-30T13:27:39.262 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57560.log 2023-10-30T13:27:39.263 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57350.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57350.log.gz 2023-10-30T13:27:39.264 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53371.log 2023-10-30T13:27:39.265 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.57560.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57560.log.gz 2023-10-30T13:27:39.265 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53426.log 2023-10-30T13:27:39.267 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53371.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53371.log.gz 2023-10-30T13:27:39.268 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63453.log 2023-10-30T13:27:39.268 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53426.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53426.log.gz 2023-10-30T13:27:39.269 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65152.log 2023-10-30T13:27:39.271 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63453.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.63453.log.gz 2023-10-30T13:27:39.271 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49693.log 2023-10-30T13:27:39.273 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65152.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65152.log.gz 2023-10-30T13:27:39.274 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48249.log 2023-10-30T13:27:39.274 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.49693.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49693.log.gz 2023-10-30T13:27:39.275 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51803.log 2023-10-30T13:27:39.277 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48249.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48249.log.gz 2023-10-30T13:27:39.277 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48355.log 2023-10-30T13:27:39.279 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51803.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51803.log.gz 2023-10-30T13:27:39.279 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59163.log 2023-10-30T13:27:39.280 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48355.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48355.log.gz 2023-10-30T13:27:39.281 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70281.log 2023-10-30T13:27:39.283 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59163.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59163.log.gz 2023-10-30T13:27:39.283 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44681.log 2023-10-30T13:27:39.284 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70281.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.70281.log.gz 2023-10-30T13:27:39.285 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62926.log 2023-10-30T13:27:39.286 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44681.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44681.log.gz 2023-10-30T13:27:39.287 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68337.log 2023-10-30T13:27:39.288 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.62926.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.62926.log.gz 2023-10-30T13:27:39.289 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53558.log 2023-10-30T13:27:39.290 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68337.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.68337.log.gz 2023-10-30T13:27:39.291 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67581.log 2023-10-30T13:27:39.292 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53558.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53558.log.gz 2023-10-30T13:27:39.293 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70151.log 2023-10-30T13:27:39.294 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67581.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67581.log.gz 2023-10-30T13:27:39.295 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53610.log 2023-10-30T13:27:39.296 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70151.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70151.log.gz 2023-10-30T13:27:39.297 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52708.log 2023-10-30T13:27:39.298 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53610.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53610.log.gz 2023-10-30T13:27:39.299 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66849.log 2023-10-30T13:27:39.300 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52708.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52708.log.gz 2023-10-30T13:27:39.300 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68574.log 2023-10-30T13:27:39.302 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66849.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66849.log.gz 2023-10-30T13:27:39.303 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45624.log 2023-10-30T13:27:39.303 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.68574.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.68574.log.gz 2023-10-30T13:27:39.304 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59535.log 2023-10-30T13:27:39.306 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45624.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45624.log.gz 2023-10-30T13:27:39.306 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60303.log 2023-10-30T13:27:39.307 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.59535.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59535.log.gz 2023-10-30T13:27:39.308 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70307.log 2023-10-30T13:27:39.310 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60303.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60303.log.gz 2023-10-30T13:27:39.310 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69403.log 2023-10-30T13:27:39.311 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70307.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70307.log.gz 2023-10-30T13:27:39.312 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66174.log 2023-10-30T13:27:39.314 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69403.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69403.log.gz 2023-10-30T13:27:39.314 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46988.log 2023-10-30T13:27:39.315 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66174.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.66174.log.gz 2023-10-30T13:27:39.316 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44687.log 2023-10-30T13:27:39.317 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46988.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46988.log.gz 2023-10-30T13:27:39.318 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51303.log 2023-10-30T13:27:39.319 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.44687.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44687.log.gz 2023-10-30T13:27:39.320 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66553.log 2023-10-30T13:27:39.321 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51303.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51303.log.gz 2023-10-30T13:27:39.322 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52063.log 2023-10-30T13:27:39.323 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66553.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.66553.log.gz 2023-10-30T13:27:39.324 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60488.log 2023-10-30T13:27:39.325 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52063.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52063.log.gz 2023-10-30T13:27:39.326 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46470.log 2023-10-30T13:27:39.327 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60488.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60488.log.gz 2023-10-30T13:27:39.328 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54407.log 2023-10-30T13:27:39.329 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46470.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46470.log.gz 2023-10-30T13:27:39.330 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52734.log 2023-10-30T13:27:39.331 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54407.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54407.log.gz 2023-10-30T13:27:39.332 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57242.log 2023-10-30T13:27:39.334 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52734.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52734.log.gz 2023-10-30T13:27:39.334 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5/var/log/ceph/ceph-client.admin.57242.log: --verbose -- /var/log/ceph/ceph-client.admin.46884.log 2023-10-30T13:27:39.335 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57242.log.gz 2023-10-30T13:27:39.336 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2023-10-30T13:27:39.337 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46884.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46884.log.gz 2023-10-30T13:27:39.338 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59348.log 2023-10-30T13:27:39.341 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-mds.e.log: /var/log/ceph/ceph-client.admin.59348.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.59348.log.gz 2023-10-30T13:27:39.341 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53779.log 2023-10-30T13:27:39.344 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.53779.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53779.log.gz 2023-10-30T13:27:39.345 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55701.log 2023-10-30T13:27:39.348 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55701.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55701.log.gz 2023-10-30T13:27:39.348 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60059.log 2023-10-30T13:27:39.351 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60059.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60059.log.gz 2023-10-30T13:27:39.351 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60435.log 2023-10-30T13:27:39.355 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60435.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60435.log.gz 2023-10-30T13:27:39.356 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48197.log 2023-10-30T13:27:39.357 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.48197.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48197.log.gz 2023-10-30T13:27:39.358 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65074.log 2023-10-30T13:27:39.360 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46340.log 2023-10-30T13:27:39.361 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65074.log.gz 2023-10-30T13:27:39.364 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70860.log 2023-10-30T13:27:39.365 INFO:teuthology.orchestra.run.smithi125.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46340.log.gz 2023-10-30T13:27:39.368 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67135.log 2023-10-30T13:27:39.369 INFO:teuthology.orchestra.run.smithi125.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70860.log.gz 2023-10-30T13:27:39.371 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45416.log 2023-10-30T13:27:39.373 INFO:teuthology.orchestra.run.smithi125.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67135.log.gz 2023-10-30T13:27:39.374 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52170.log 2023-10-30T13:27:39.377 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.45416.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45416.log.gz 2023-10-30T13:27:39.377 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58215.log 2023-10-30T13:27:39.380 INFO:teuthology.orchestra.run.smithi125.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52170.log.gz 2023-10-30T13:27:39.381 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.58215.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58215.log.gz 2023-10-30T13:27:39.381 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51435.log 2023-10-30T13:27:39.384 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51435.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51435.log.gz 2023-10-30T13:27:39.385 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65873.log 2023-10-30T13:27:39.387 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65873.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.65873.log.gz 2023-10-30T13:27:39.388 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54705.log 2023-10-30T13:27:39.391 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54705.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54705.log.gz 2023-10-30T13:27:39.391 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66631.log 2023-10-30T13:27:39.394 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66631.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66631.log.gz 2023-10-30T13:27:39.394 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65795.log 2023-10-30T13:27:39.398 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.65795.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.65795.log.gz 2023-10-30T13:27:39.398 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55914.log 2023-10-30T13:27:39.401 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55914.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55914.log.gz 2023-10-30T13:27:39.401 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63932.log 2023-10-30T13:27:39.405 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63932.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.63932.log.gz 2023-10-30T13:27:39.405 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66418.log 2023-10-30T13:27:39.408 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.66418.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66418.log.gz 2023-10-30T13:27:39.408 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64644.log 2023-10-30T13:27:39.412 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.64644.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64644.log.gz 2023-10-30T13:27:39.412 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63880.log 2023-10-30T13:27:39.415 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63880.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.63880.log.gz 2023-10-30T13:27:39.415 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61821.log 2023-10-30T13:27:39.418 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.61821.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61821.log.gz 2023-10-30T13:27:39.419 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52680.log 2023-10-30T13:27:39.422 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52680.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52680.log.gz 2023-10-30T13:27:39.422 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55284.log 2023-10-30T13:27:39.424 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70541.log 2023-10-30T13:27:39.425 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.55284.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55284.log.gz 2023-10-30T13:27:39.427 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52812.log 2023-10-30T13:27:39.430 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.70541.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.70541.log.gz 2023-10-30T13:27:39.430 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.52812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60190.log 2023-10-30T13:27:39.430 INFO:teuthology.orchestra.run.smithi125.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52812.log.gz 2023-10-30T13:27:39.433 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.60190.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60190.log.gz 2023-10-30T13:27:39.433 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51120.log 2023-10-30T13:27:39.437 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.51120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56203.log 2023-10-30T13:27:39.437 INFO:teuthology.orchestra.run.smithi125.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51120.log.gz 2023-10-30T13:27:39.439 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54433.log 2023-10-30T13:27:39.442 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.56203.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56203.log.gz 2023-10-30T13:27:39.442 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.54433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46802.log 2023-10-30T13:27:39.442 INFO:teuthology.orchestra.run.smithi125.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54433.log.gz 2023-10-30T13:27:39.445 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.46802.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46802.log.gz 2023-10-30T13:27:39.445 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67005.log 2023-10-30T13:27:39.449 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.67005.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67005.log.gz 2023-10-30T13:27:39.449 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63795.log 2023-10-30T13:27:39.451 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.63795.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63795.log.gz 2023-10-30T13:27:39.452 INFO:teuthology.orchestra.run.smithi125.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69036.log 2023-10-30T13:27:39.455 INFO:teuthology.orchestra.run.smithi125.stderr:/var/log/ceph/ceph-client.admin.69036.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69036.log.gz 2023-10-30T13:27:39.525 INFO:teuthology.orchestra.run.smithi136.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-30T13:27:41.299 INFO:teuthology.orchestra.run.smithi125.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-30T13:27:43.903 INFO:teuthology.orchestra.run.smithi136.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-30T13:27:47.485 INFO:teuthology.orchestra.run.smithi125.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-30T13:27:48.349 INFO:teuthology.orchestra.run.smithi125.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-30T13:27:48.394 INFO:teuthology.orchestra.run.smithi136.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-30T13:27:48.661 INFO:teuthology.orchestra.run.smithi136.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-30T13:27:49.739 INFO:teuthology.orchestra.run.smithi136.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-30T13:27:50.927 INFO:teuthology.orchestra.run.smithi125.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-30T13:27:52.023 INFO:teuthology.orchestra.run.smithi125.stderr: 94.2% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-30T13:27:53.161 INFO:teuthology.orchestra.run.smithi125.stderr: 95.0% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2023-10-30T13:28:02.084 INFO:teuthology.orchestra.run.smithi136.stderr: 94.1% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-30T13:28:17.166 INFO:teuthology.orchestra.run.smithi136.stderr: 94.2% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2023-10-30T13:28:17.168 INFO:teuthology.orchestra.run.smithi136.stderr: 2023-10-30T13:28:17.168 INFO:teuthology.orchestra.run.smithi136.stderr:real 0m39.102s 2023-10-30T13:28:17.168 INFO:teuthology.orchestra.run.smithi136.stderr:user 1m40.673s 2023-10-30T13:28:17.169 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m3.357s 2023-10-30T13:28:55.207 INFO:teuthology.orchestra.run.smithi125.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-30T13:28:55.208 INFO:teuthology.orchestra.run.smithi125.stderr: 2023-10-30T13:28:55.209 INFO:teuthology.orchestra.run.smithi125.stderr:real 1m17.179s 2023-10-30T13:28:55.209 INFO:teuthology.orchestra.run.smithi125.stderr:user 2m9.441s 2023-10-30T13:28:55.209 INFO:teuthology.orchestra.run.smithi125.stderr:sys 0m11.766s 2023-10-30T13:28:55.210 INFO:tasks.ceph:Archiving logs... 2023-10-30T13:28:55.211 DEBUG:teuthology.misc:Transferring archived files from smithi125:/var/log/ceph to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019/remote/smithi125/log 2023-10-30T13:28:55.212 DEBUG:teuthology.orchestra.run.smithi125:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T13:29:20.388 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019/remote/smithi136/log 2023-10-30T13:29:20.389 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-30T13:29:30.959 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-30T13:29:30.969 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-30T13:29:30.969 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T13:29:30.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-30T13:29:31.050 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-30T13:29:31.059 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-30T13:29:31.060 DEBUG:teuthology.orchestra.run.smithi125:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T13:29:31.064 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-30T13:29:31.080 INFO:teuthology.orchestra.run.smithi125.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T13:29:31.081 INFO:teuthology.orchestra.run.smithi125.stdout:============================================================================== 2023-10-30T13:29:31.081 INFO:teuthology.orchestra.run.smithi125.stdout:*hv01.front.sepi 44.4.53.2 2 u 56 128 377 0.184 -2.764 0.497 2023-10-30T13:29:31.081 INFO:teuthology.orchestra.run.smithi125.stdout:+hv02.front.sepi 63.231.80.2 3 u 109 128 377 0.154 +0.870 0.447 2023-10-30T13:29:31.082 INFO:teuthology.orchestra.run.smithi125.stdout:+hv03.front.sepi 74.6.168.72 3 u 64 128 377 0.175 -1.111 0.298 2023-10-30T13:29:31.082 INFO:teuthology.orchestra.run.smithi125.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-30T13:29:31.084 INFO:teuthology.orchestra.run.smithi136.stdout: remote refid st t when poll reach delay offset jitter 2023-10-30T13:29:31.084 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================== 2023-10-30T13:29:31.084 INFO:teuthology.orchestra.run.smithi136.stdout:*hv01.front.sepi 44.4.53.2 2 u 127 128 377 0.087 -3.388 0.382 2023-10-30T13:29:31.085 INFO:teuthology.orchestra.run.smithi136.stdout:+hv02.front.sepi 63.231.80.2 3 u 2 128 377 0.073 +0.123 0.258 2023-10-30T13:29:31.085 INFO:teuthology.orchestra.run.smithi136.stdout:+hv03.front.sepi 74.6.168.72 3 u 1 128 377 0.088 -1.423 0.311 2023-10-30T13:29:31.085 INFO:teuthology.orchestra.run.smithi136.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-10-30T13:29:31.086 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-30T13:29:31.094 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-30T13:29:31.095 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-30T13:29:31.104 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-30T13:29:31.113 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-30T13:29:31.124 INFO:teuthology.task.internal:Duration was 4388.385241 seconds 2023-10-30T13:29:31.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-30T13:29:31.133 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-30T13:29:31.134 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T13:29:31.137 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-30T13:29:31.244 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-30T13:29:31.245 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi125.front.sepia.ceph.com 2023-10-30T13:29:31.245 DEBUG:teuthology.orchestra.run.smithi125:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T13:29:31.269 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2023-10-30T13:29:31.270 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-30T13:29:31.279 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-30T13:29:31.280 DEBUG:teuthology.orchestra.run.smithi125:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T13:29:31.319 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-30T13:29:31.357 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-30T13:29:31.358 DEBUG:teuthology.orchestra.run.smithi125:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T13:29:31.402 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-30T13:29:31.495 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-30T13:29:31.505 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-30T13:29:31.506 DEBUG:teuthology.orchestra.run.smithi125:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T13:29:31.527 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-30T13:29:31.543 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-30T13:29:31.558 DEBUG:teuthology.orchestra.run.smithi125:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T13:29:31.576 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-30T13:29:31.591 INFO:teuthology.orchestra.run.smithi125.stdout:kernel.core_pattern = core 2023-10-30T13:29:31.600 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2023-10-30T13:29:31.632 DEBUG:teuthology.orchestra.run.smithi125:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T13:29:31.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:29:31.661 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-30T13:29:31.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-30T13:29:31.683 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-30T13:29:31.739 INFO:teuthology.task.internal:Transferring archived files... 2023-10-30T13:29:31.740 DEBUG:teuthology.misc:Transferring archived files from smithi125:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019/remote/smithi125 2023-10-30T13:29:31.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T13:29:31.809 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019/remote/smithi136 2023-10-30T13:29:31.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-30T13:29:31.854 INFO:teuthology.task.internal:Removing archive directory... 2023-10-30T13:29:31.855 DEBUG:teuthology.orchestra.run.smithi125:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T13:29:31.860 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-30T13:29:31.901 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-30T13:29:31.920 INFO:teuthology.task.internal:Not uploading archives. 2023-10-30T13:29:31.921 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-30T13:29:32.054 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-30T13:29:32.054 DEBUG:teuthology.orchestra.run.smithi125:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T13:29:32.059 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-30T13:29:32.064 INFO:teuthology.orchestra.run.smithi125.stdout: 657978 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 13:29 /home/ubuntu/cephtest 2023-10-30T13:29:32.066 INFO:teuthology.orchestra.run.smithi136.stdout: 524293 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 30 13:29 /home/ubuntu/cephtest 2023-10-30T13:29:32.068 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-30T13:29:32.130 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-30T13:29:32.159 INFO:teuthology.nuke:Checking targets against current locks 2023-10-30T13:29:32.187 DEBUG:teuthology.nuke:shortname: smithi125 2023-10-30T13:29:32.188 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T13:29:32.216 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi125.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 12:05:37.069272', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAiFxaPW/fAIFYfAZ7S0QaSr7et4p16btxGCug/BeJSMAcBjUFWpEFIiQexh7R1wVQLHrWxl1wi22tuK+iNge+g='} 2023-10-30T13:29:32.218 DEBUG:teuthology.nuke:shortname: smithi136 2023-10-30T13:29:32.218 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-30T13:29:32.246 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2023-10-30_06:34:16-fs-wip-lusov-balance-after-stop-distro-default-smithi/7441019', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-30 12:05:37.071199', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHFv/5e/H1nftak/kNE7BpioBtAEeujqPjwXW0WI6M9KclRCZ1nNUcD4HU42sPUWOO2qfq1wGIW0kzp5YjMoVs4='} 2023-10-30T13:29:32.277 INFO:teuthology.orchestra.console.smithi125:Power off 2023-10-30T13:29:32.279 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T13:29:32.301 DEBUG:teuthology.orchestra.console.smithi125:power off output: Chassis Power Control: Down/Off 2023-10-30T13:29:32.302 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T13:29:32.318 INFO:teuthology.orchestra.console.smithi136:Power off 2023-10-30T13:29:32.318 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-30T13:29:32.328 DEBUG:teuthology.orchestra.console.smithi125:check power output: Chassis Power is on 2023-10-30T13:29:32.341 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2023-10-30T13:29:32.341 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T13:29:32.362 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2023-10-30T13:29:36.329 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T13:29:36.364 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T13:29:36.441 DEBUG:teuthology.orchestra.console.smithi125:check power output: Chassis Power is on 2023-10-30T13:29:36.475 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2023-10-30T13:29:40.444 DEBUG:teuthology.orchestra.console.smithi125:pexpect command: ipmitool -H smithi125.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T13:29:40.477 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-30T13:29:40.554 DEBUG:teuthology.orchestra.console.smithi125:check power output: Chassis Power is off 2023-10-30T13:29:40.588 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2023-10-30T13:29:40.656 INFO:teuthology.orchestra.console.smithi125:Power off completed 2023-10-30T13:29:40.689 INFO:teuthology.orchestra.console.smithi136:Power off completed 2023-10-30T13:29:40.796 INFO:teuthology.run:Summary data: client.0-kernel-sha1: eacf00cc51e204f401b4d5d4f84825a68f373c74 description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate} clusters/1a5s-mds-1c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down prefetch_dirfrags/yes prefetch_entire_dirfrags/no races session_timeout thrashosds-health} ranks/3 tasks/{1-thrash/mds 2-workunit/fs/snaps}} duration: 4388.385241031647 failure_reason: 'Command failed (workunit test fs/snaps/snaptest-git-ceph.sh) on smithi125 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=00c1c13cea21a686b046584596fbeac5315a7d71 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh''' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=16ad8708e9a84843ae2d25784cf9a1ab status: fail success: false 2023-10-30T13:29:40.797 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-30T13:29:40.881 INFO:teuthology.run:FAIL